2024-05-12T21:45:05.211 INFO:root:teuthology version: 0.0.1.dev271+g1ae7ad8 2024-05-12T21:45:05.211 DEBUG:teuthology.run:Teuthology command: teuthology --name teuthology-2024-05-12_21:24:02-fs-squid-distro-default-smithi --archive /home/teuthworker/archive/teuthology-2024-05-12_21:24:02-fs-squid-distro-default-smithi/7703941 --verbose --owner scheduled_teuthology@teuthology --description fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/fsx}} -- /home/teuthworker/archive/teuthology-2024-05-12_21:24:02-fs-squid-distro-default-smithi/7703941/orig.config.yaml 2024-05-12T21:45:05.274 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-05-12T21:45:05.370 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-05-12_21:24:02-fs-squid-distro-default-smithi/7703941 branch: squid description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/fsx}} email: ceph-qa@ceph.com first_in_suite: false job_id: '7703941' kernel: client: branch: testing kdb: 1 sha1: distro ktype: upstream last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules - exec: mon.a: - ceph mgr module enable snap_schedule - ceph config set mgr mgr/snap_schedule/allow_m_granularity true - ceph config set mgr mgr/snap_schedule/dump_on_update true name: teuthology-2024-05-12_21:24:02-fs-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: cephfs: max_mds: 3 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true mds_bal_replicate_threshold: 1 mds_export_ephemeral_random_max: 0.1 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug cephsqlite: 20 debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10000 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - MON_DOWN - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - slow metadata IO - SLOW_OPS - slow request sha1: f72fecff68e1d400c4568684327c900485c20d6a subvols: create: 2 subvol_options: --size 25000000000 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - min: 3 name: mds_cache.dir_update - min: 3 name: mds_cache.dir_update_receipt - mds.root_rsnaps - mds_server.req_mksnap_latency.avgcount - mds_server.req_rmsnap_latency.avgcount - mds.dir_split install: ceph: flavor: default sha1: f72fecff68e1d400c4568684327c900485c20d6a kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=legacy - wsync syntax: v2 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: squid sha1: c50db29d375ad9ca1a200449f9347b89e25fb278 owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 521 sha1: f72fecff68e1d400c4568684327c900485c20d6a sleep_before_teardown: 0 subset: 295/512 suite: fs suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_c50db29d375ad9ca1a200449f9347b89e25fb278/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: c50db29d375ad9ca1a200449f9347b89e25fb278 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi028.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDo+zcEat3FoejyP1Xh22qYVt0nyVtxgcdLMqfh03Mdzze/iDt4FpewLerXH3e4my7vCG+Ao6QZBuJsQF5Kt6HA= smithi086.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKe91EP00mtpT0bhWgWPA1pmY+P/QDBNmgZRLVNpx6BW4OgCPc/hZK6pSgc/C5+VBSOnxwa99MLSwzPW2Y6ng8I= smithi186.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBX7vLlnMm7oYwlDmyuB3GJBNIVtLzrjtokbCrB+aW/HHlnpxPaROl2qRZfyN7/XEmH6eknXHzThFehR4kxyjWA= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - sequential: - mgrmodules - kclient: null - exec: mon.a: - ceph fs set cephfs balance_automate false - ceph fs subvolumegroup pin cephfs qa random 0.10 - check-counter: {} - fwd_scrub: scrub_timeout: 900 sleep_between_iterations: 1 - exec: mon.a: - ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m - ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - date +%s > START_TIME - full_sequential_finally: - exec: mon.a: - date +%s > END_TIME - START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - workunit: clients: all: - suites/fsx.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 1ae7ad82388e92a475afff437d49054826c019a1 timestamp: 2024-05-12_21:24:02 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-05-12T21:45:05.370 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_c50db29d375ad9ca1a200449f9347b89e25fb278/qa; will attempt to use it 2024-05-12T21:45:05.371 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_c50db29d375ad9ca1a200449f9347b89e25fb278/qa/tasks 2024-05-12T21:45:05.371 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-05-12T21:45:05.373 INFO:teuthology.task.internal:Checking packages... 2024-05-12T21:45:05.396 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'f72fecff68e1d400c4568684327c900485c20d6a' 2024-05-12T21:45:05.396 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-05-12T21:45:05.396 INFO:teuthology.packaging:ref: None 2024-05-12T21:45:05.396 INFO:teuthology.packaging:tag: None 2024-05-12T21:45:05.396 INFO:teuthology.packaging:branch: squid 2024-05-12T21:45:05.396 INFO:teuthology.packaging:sha1: f72fecff68e1d400c4568684327c900485c20d6a 2024-05-12T21:45:05.396 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-05-12T21:45:05.592 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2555.gf72fecff 2024-05-12T21:45:05.594 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-05-12T21:45:05.600 INFO:teuthology.task.internal:no buildpackages task found 2024-05-12T21:45:05.600 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-05-12T21:45:05.621 INFO:teuthology.task.internal:Saving configuration 2024-05-12T21:45:05.644 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-05-12T21:45:05.650 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-05-12T21:45:05.673 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi028.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-05-12_21:24:02-fs-squid-distro-default-smithi/7703941', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-12 21:39:47.310876', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDo+zcEat3FoejyP1Xh22qYVt0nyVtxgcdLMqfh03Mdzze/iDt4FpewLerXH3e4my7vCG+Ao6QZBuJsQF5Kt6HA='} 2024-05-12T21:45:05.696 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi086.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-05-12_21:24:02-fs-squid-distro-default-smithi/7703941', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-12 21:39:47.312120', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKe91EP00mtpT0bhWgWPA1pmY+P/QDBNmgZRLVNpx6BW4OgCPc/hZK6pSgc/C5+VBSOnxwa99MLSwzPW2Y6ng8I='} 2024-05-12T21:45:05.717 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi186.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-05-12_21:24:02-fs-squid-distro-default-smithi/7703941', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-12 21:39:47.309124', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBX7vLlnMm7oYwlDmyuB3GJBNIVtLzrjtokbCrB+aW/HHlnpxPaROl2qRZfyN7/XEmH6eknXHzThFehR4kxyjWA='} 2024-05-12T21:45:05.718 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-05-12T21:45:05.724 INFO:teuthology.task.internal:roles: ubuntu@smithi028.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-05-12T21:45:05.724 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-05-12T21:45:05.724 INFO:teuthology.task.internal:roles: ubuntu@smithi186.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-05-12T21:45:05.724 INFO:teuthology.run_tasks:Running task console_log... 2024-05-12T21:45:05.833 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fa5a2104160>, signals=[15]) 2024-05-12T21:45:05.833 INFO:teuthology.run_tasks:Running task internal.connect... 2024-05-12T21:45:05.839 INFO:teuthology.task.internal:Opening connections... 2024-05-12T21:45:05.840 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi028.front.sepia.ceph.com 2024-05-12T21:45:05.841 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi028.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-12T21:45:05.928 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi086.front.sepia.ceph.com 2024-05-12T21:45:05.929 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-12T21:45:06.054 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi186.front.sepia.ceph.com 2024-05-12T21:45:06.055 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-12T21:45:06.140 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-05-12T21:45:06.147 DEBUG:teuthology.orchestra.run.smithi028:> uname -m 2024-05-12T21:45:06.166 INFO:teuthology.orchestra.run.smithi028.stdout:x86_64 2024-05-12T21:45:06.167 DEBUG:teuthology.orchestra.run.smithi028:> cat /etc/os-release 2024-05-12T21:45:06.225 INFO:teuthology.orchestra.run.smithi028.stdout:NAME="CentOS Stream" 2024-05-12T21:45:06.226 INFO:teuthology.orchestra.run.smithi028.stdout:VERSION="9" 2024-05-12T21:45:06.226 INFO:teuthology.orchestra.run.smithi028.stdout:ID="centos" 2024-05-12T21:45:06.226 INFO:teuthology.orchestra.run.smithi028.stdout:ID_LIKE="rhel fedora" 2024-05-12T21:45:06.226 INFO:teuthology.orchestra.run.smithi028.stdout:VERSION_ID="9" 2024-05-12T21:45:06.226 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM_ID="platform:el9" 2024-05-12T21:45:06.226 INFO:teuthology.orchestra.run.smithi028.stdout:PRETTY_NAME="CentOS Stream 9" 2024-05-12T21:45:06.227 INFO:teuthology.orchestra.run.smithi028.stdout:ANSI_COLOR="0;31" 2024-05-12T21:45:06.227 INFO:teuthology.orchestra.run.smithi028.stdout:LOGO="fedora-logo-icon" 2024-05-12T21:45:06.227 INFO:teuthology.orchestra.run.smithi028.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-05-12T21:45:06.227 INFO:teuthology.orchestra.run.smithi028.stdout:HOME_URL="https://centos.org/" 2024-05-12T21:45:06.227 INFO:teuthology.orchestra.run.smithi028.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-05-12T21:45:06.227 INFO:teuthology.orchestra.run.smithi028.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-05-12T21:45:06.228 INFO:teuthology.orchestra.run.smithi028.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-05-12T21:45:06.229 INFO:teuthology.lock.ops:Updating smithi028.front.sepia.ceph.com on lock server 2024-05-12T21:45:06.252 DEBUG:teuthology.orchestra.run.smithi086:> uname -m 2024-05-12T21:45:06.270 INFO:teuthology.orchestra.run.smithi086.stdout:x86_64 2024-05-12T21:45:06.270 DEBUG:teuthology.orchestra.run.smithi086:> cat /etc/os-release 2024-05-12T21:45:06.327 INFO:teuthology.orchestra.run.smithi086.stdout:NAME="CentOS Stream" 2024-05-12T21:45:06.328 INFO:teuthology.orchestra.run.smithi086.stdout:VERSION="9" 2024-05-12T21:45:06.328 INFO:teuthology.orchestra.run.smithi086.stdout:ID="centos" 2024-05-12T21:45:06.328 INFO:teuthology.orchestra.run.smithi086.stdout:ID_LIKE="rhel fedora" 2024-05-12T21:45:06.328 INFO:teuthology.orchestra.run.smithi086.stdout:VERSION_ID="9" 2024-05-12T21:45:06.328 INFO:teuthology.orchestra.run.smithi086.stdout:PLATFORM_ID="platform:el9" 2024-05-12T21:45:06.328 INFO:teuthology.orchestra.run.smithi086.stdout:PRETTY_NAME="CentOS Stream 9" 2024-05-12T21:45:06.328 INFO:teuthology.orchestra.run.smithi086.stdout:ANSI_COLOR="0;31" 2024-05-12T21:45:06.328 INFO:teuthology.orchestra.run.smithi086.stdout:LOGO="fedora-logo-icon" 2024-05-12T21:45:06.328 INFO:teuthology.orchestra.run.smithi086.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-05-12T21:45:06.328 INFO:teuthology.orchestra.run.smithi086.stdout:HOME_URL="https://centos.org/" 2024-05-12T21:45:06.328 INFO:teuthology.orchestra.run.smithi086.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-05-12T21:45:06.328 INFO:teuthology.orchestra.run.smithi086.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-05-12T21:45:06.328 INFO:teuthology.orchestra.run.smithi086.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-05-12T21:45:06.329 INFO:teuthology.lock.ops:Updating smithi086.front.sepia.ceph.com on lock server 2024-05-12T21:45:06.352 DEBUG:teuthology.orchestra.run.smithi186:> uname -m 2024-05-12T21:45:06.370 INFO:teuthology.orchestra.run.smithi186.stdout:x86_64 2024-05-12T21:45:06.371 DEBUG:teuthology.orchestra.run.smithi186:> cat /etc/os-release 2024-05-12T21:45:06.428 INFO:teuthology.orchestra.run.smithi186.stdout:NAME="CentOS Stream" 2024-05-12T21:45:06.428 INFO:teuthology.orchestra.run.smithi186.stdout:VERSION="9" 2024-05-12T21:45:06.428 INFO:teuthology.orchestra.run.smithi186.stdout:ID="centos" 2024-05-12T21:45:06.428 INFO:teuthology.orchestra.run.smithi186.stdout:ID_LIKE="rhel fedora" 2024-05-12T21:45:06.428 INFO:teuthology.orchestra.run.smithi186.stdout:VERSION_ID="9" 2024-05-12T21:45:06.429 INFO:teuthology.orchestra.run.smithi186.stdout:PLATFORM_ID="platform:el9" 2024-05-12T21:45:06.429 INFO:teuthology.orchestra.run.smithi186.stdout:PRETTY_NAME="CentOS Stream 9" 2024-05-12T21:45:06.429 INFO:teuthology.orchestra.run.smithi186.stdout:ANSI_COLOR="0;31" 2024-05-12T21:45:06.429 INFO:teuthology.orchestra.run.smithi186.stdout:LOGO="fedora-logo-icon" 2024-05-12T21:45:06.429 INFO:teuthology.orchestra.run.smithi186.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-05-12T21:45:06.429 INFO:teuthology.orchestra.run.smithi186.stdout:HOME_URL="https://centos.org/" 2024-05-12T21:45:06.429 INFO:teuthology.orchestra.run.smithi186.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-05-12T21:45:06.429 INFO:teuthology.orchestra.run.smithi186.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-05-12T21:45:06.429 INFO:teuthology.orchestra.run.smithi186.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-05-12T21:45:06.429 INFO:teuthology.lock.ops:Updating smithi186.front.sepia.ceph.com on lock server 2024-05-12T21:45:06.452 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-05-12T21:45:06.462 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-05-12T21:45:06.562 INFO:teuthology.task.internal:Checking for old test directory... 2024-05-12T21:45:06.563 DEBUG:teuthology.orchestra.run.smithi028:> test '!' -e /home/ubuntu/cephtest 2024-05-12T21:45:06.565 DEBUG:teuthology.orchestra.run.smithi086:> test '!' -e /home/ubuntu/cephtest 2024-05-12T21:45:06.567 DEBUG:teuthology.orchestra.run.smithi186:> test '!' -e /home/ubuntu/cephtest 2024-05-12T21:45:06.583 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-05-12T21:45:06.589 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-05-12T21:45:06.589 DEBUG:teuthology.orchestra.run.smithi028:> test -z $(ls -A /var/lib/ceph) 2024-05-12T21:45:06.622 DEBUG:teuthology.orchestra.run.smithi086:> test -z $(ls -A /var/lib/ceph) 2024-05-12T21:45:06.625 DEBUG:teuthology.orchestra.run.smithi186:> test -z $(ls -A /var/lib/ceph) 2024-05-12T21:45:06.731 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-05-12T21:45:06.810 INFO:teuthology.run_tasks:Running task kernel... 2024-05-12T21:45:06.827 INFO:teuthology.task.kernel:normalize config orig: {'client': {'branch': 'testing'}, 'kdb': 1, 'sha1': 'distro'} 2024-05-12T21:45:06.827 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'branch': 'testing'}} 2024-05-12T21:45:06.827 INFO:teuthology.task.kernel:config {'client.0': {'branch': 'testing'}}, timeout 300 2024-05-12T21:45:06.828 DEBUG:teuthology.orchestra.run.smithi028:> test -f /run/.containerenv -o -f /.dockerenv 2024-05-12T21:45:06.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-12T21:45:06.846 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&ref=testing 2024-05-12T21:45:07.006 DEBUG:teuthology.task.kernel:sha1 for client.0 is fb9a6a44107e0b9f9ca2d9a78f2308f5dd49911e 2024-05-12T21:45:07.006 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "fb9a6a44107e0b9f9ca2d9a78f2308f5dd49911e"... 2024-05-12T21:45:07.006 DEBUG:teuthology.orchestra.run.smithi028:> uname -r 2024-05-12T21:45:07.024 INFO:teuthology.orchestra.run.smithi028.stdout:5.14.0-437.el9.x86_64 2024-05-12T21:45:07.024 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-437.el9.x86_64 vs fb9a6a44107e0b9f9ca2d9a78f2308f5dd49911e 2024-05-12T21:45:07.024 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-05-12T21:45:07.025 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum upgrade -y linux-firmware 2024-05-12T21:45:09.323 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - BaseOS 10 MB/s | 8.1 MB 00:00 2024-05-12T21:45:11.833 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - AppStream 14 MB/s | 19 MB 00:01 2024-05-12T21:45:16.133 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - CRB 10 MB/s | 6.2 MB 00:00 2024-05-12T21:45:17.742 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - Extras packages 45 kB/s | 16 kB 00:00 2024-05-12T21:45:19.571 INFO:teuthology.orchestra.run.smithi028.stdout:Extra Packages for Enterprise Linux 12 MB/s | 21 MB 00:01 2024-05-12T21:45:24.935 INFO:teuthology.orchestra.run.smithi028.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-05-12T21:45:27.123 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-05-12T21:45:27.124 INFO:teuthology.orchestra.run.smithi028.stdout:Nothing to do. 2024-05-12T21:45:27.124 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-05-12T21:45:27.219 INFO:teuthology.task.kernel:Downloading kernel fb9a6a44107e0b9f9ca2d9a78f2308f5dd49911e on client.0... 2024-05-12T21:45:27.220 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=fb9a6a44107e0b9f9ca2d9a78f2308f5dd49911e 2024-05-12T21:45:27.362 INFO:teuthology.task.kernel:fetching, builder baseurl is https://2.chacra.ceph.com/r/kernel/testing/fb9a6a44107e0b9f9ca2d9a78f2308f5dd49911e/centos/9/flavors/default/x86_64/ 2024-05-12T21:45:27.362 DEBUG:teuthology.orchestra.run.smithi028:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.9.0_rc3_gfb9a6a44107e-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://2.chacra.ceph.com/r/kernel/testing/fb9a6a44107e0b9f9ca2d9a78f2308f5dd49911e/centos/9/flavors/default/x86_64/ --input-file=- 2024-05-12T21:45:27.364 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi028.front.sepia.ceph.com to complete... 2024-05-12T21:45:39.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12 21:45:39 URL:https://2.chacra.ceph.com/r/kernel/testing/fb9a6a44107e0b9f9ca2d9a78f2308f5dd49911e/centos/9/flavors/default/x86_64/kernel-6.9.0_rc3_gfb9a6a44107e-1.x86_64.rpm [357387737/357387737] -> "/tmp/kernel.x86_64.rpm" [1] 2024-05-12T21:45:39.500 INFO:teuthology.orchestra.run.smithi028.stderr:FINISHED --2024-05-12 21:45:39-- 2024-05-12T21:45:39.500 INFO:teuthology.orchestra.run.smithi028.stderr:Total wall clock time: 12s 2024-05-12T21:45:39.500 INFO:teuthology.orchestra.run.smithi028.stderr:Downloaded: 1 files, 341M in 12s (28.9 MB/s) 2024-05-12T21:45:39.503 INFO:teuthology.task.kernel:Installing kernel fb9a6a44107e0b9f9ca2d9a78f2308f5dd49911e on client.0... 2024-05-12T21:45:39.504 DEBUG:teuthology.orchestra.run.smithi028:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-05-12T21:45:41.736 INFO:teuthology.orchestra.run.smithi028.stdout:Verifying... ######################################## 2024-05-12T21:45:41.871 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing... ######################################## 2024-05-12T21:45:41.988 INFO:teuthology.orchestra.run.smithi028.stdout:Updating / installing... 2024-05-12T21:45:50.468 INFO:teuthology.orchestra.run.smithi028.stdout:kernel-6.9.0_rc3_gfb9a6a44107e-1 ######################################## 2024-05-12T21:46:29.834 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi028.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-05-12T21:46:29.834 DEBUG:teuthology.orchestra.run.smithi028:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-05-12T21:46:29.922 INFO:teuthology.orchestra.run.smithi028.stdout:/boot/System.map-6.9.0-rc3-gfb9a6a44107e 2024-05-12T21:46:29.922 INFO:teuthology.orchestra.run.smithi028.stdout:/boot/config-6.9.0-rc3-gfb9a6a44107e 2024-05-12T21:46:29.922 INFO:teuthology.orchestra.run.smithi028.stdout:/boot/vmlinuz-6.9.0-rc3-gfb9a6a44107e 2024-05-12T21:46:29.922 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/System.map 2024-05-12T21:46:29.922 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/config 2024-05-12T21:46:29.922 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel 2024-05-12T21:46:29.923 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/arch 2024-05-12T21:46:29.923 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/arch/x86 2024-05-12T21:46:29.923 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/arch/x86/crypto 2024-05-12T21:46:29.923 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-05-12T21:46:29.923 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-05-12T21:46:29.923 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-05-12T21:46:29.923 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-05-12T21:46:29.923 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-05-12T21:46:29.923 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-05-12T21:46:29.923 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-05-12T21:46:29.923 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/arch/x86/crypto/crc32c-intel.ko 2024-05-12T21:46:29.923 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-05-12T21:46:29.923 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-05-12T21:46:29.923 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-05-12T21:46:29.924 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-05-12T21:46:29.924 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/arch/x86/crypto/serpent-avx2.ko 2024-05-12T21:46:29.924 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-05-12T21:46:29.924 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-05-12T21:46:29.924 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-05-12T21:46:29.924 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-05-12T21:46:29.924 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-05-12T21:46:29.924 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-05-12T21:46:29.924 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-05-12T21:46:29.924 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/arch/x86/kernel 2024-05-12T21:46:29.924 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/arch/x86/kernel/cpu 2024-05-12T21:46:29.924 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/arch/x86/kernel/cpu/mce 2024-05-12T21:46:29.924 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-05-12T21:46:29.925 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/arch/x86/kvm 2024-05-12T21:46:29.925 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/arch/x86/kvm/kvm-amd.ko 2024-05-12T21:46:29.925 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/arch/x86/kvm/kvm-intel.ko 2024-05-12T21:46:29.925 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/arch/x86/kvm/kvm.ko 2024-05-12T21:46:29.925 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/crypto 2024-05-12T21:46:29.925 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/crypto/ansi_cprng.ko 2024-05-12T21:46:29.925 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/crypto/anubis.ko 2024-05-12T21:46:29.925 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/crypto/arc4.ko 2024-05-12T21:46:29.925 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/crypto/async_tx 2024-05-12T21:46:29.925 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/crypto/async_tx/async_memcpy.ko 2024-05-12T21:46:29.925 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/crypto/async_tx/async_pq.ko 2024-05-12T21:46:29.925 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/crypto/async_tx/async_raid6_recov.ko 2024-05-12T21:46:29.925 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/crypto/async_tx/async_tx.ko 2024-05-12T21:46:29.926 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/crypto/async_tx/async_xor.ko 2024-05-12T21:46:29.926 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/crypto/async_tx/raid6test.ko 2024-05-12T21:46:29.926 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/crypto/authenc.ko 2024-05-12T21:46:29.926 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/crypto/authencesn.ko 2024-05-12T21:46:29.926 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/crypto/blake2b_generic.ko 2024-05-12T21:46:29.926 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/crypto/blowfish_common.ko 2024-05-12T21:46:29.926 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/crypto/blowfish_generic.ko 2024-05-12T21:46:29.926 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/crypto/camellia_generic.ko 2024-05-12T21:46:29.926 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/crypto/cast5_generic.ko 2024-05-12T21:46:29.926 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/crypto/cast6_generic.ko 2024-05-12T21:46:29.926 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/crypto/cast_common.ko 2024-05-12T21:46:29.926 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/crypto/ccm.ko 2024-05-12T21:46:29.926 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/crypto/cmac.ko 2024-05-12T21:46:29.927 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/crypto/crc32_generic.ko 2024-05-12T21:46:29.927 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/crypto/crypto_engine.ko 2024-05-12T21:46:29.927 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/crypto/crypto_user.ko 2024-05-12T21:46:29.927 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/crypto/cts.ko 2024-05-12T21:46:29.927 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/crypto/des_generic.ko 2024-05-12T21:46:29.927 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/crypto/dh_generic.ko 2024-05-12T21:46:29.927 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/crypto/echainiv.ko 2024-05-12T21:46:29.927 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/crypto/essiv.ko 2024-05-12T21:46:29.927 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/crypto/fcrypt.ko 2024-05-12T21:46:29.927 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/crypto/khazad.ko 2024-05-12T21:46:29.927 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/crypto/lz4.ko 2024-05-12T21:46:29.927 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/crypto/lz4hc.ko 2024-05-12T21:46:29.927 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/crypto/md4.ko 2024-05-12T21:46:29.928 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/crypto/michael_mic.ko 2024-05-12T21:46:29.928 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/crypto/pcbc.ko 2024-05-12T21:46:29.928 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/crypto/pcrypt.ko 2024-05-12T21:46:29.928 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/crypto/rmd160.ko 2024-05-12T21:46:29.928 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/crypto/seed.ko 2024-05-12T21:46:29.928 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/crypto/serpent_generic.ko 2024-05-12T21:46:29.928 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/crypto/tcrypt.ko 2024-05-12T21:46:29.928 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/crypto/tea.ko 2024-05-12T21:46:29.930 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/crypto/twofish_common.ko 2024-05-12T21:46:29.930 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/crypto/twofish_generic.ko 2024-05-12T21:46:29.931 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/crypto/vmac.ko 2024-05-12T21:46:29.931 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/crypto/wp512.ko 2024-05-12T21:46:29.931 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/crypto/xcbc.ko 2024-05-12T21:46:29.931 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/crypto/xor.ko 2024-05-12T21:46:29.931 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/crypto/xxhash_generic.ko 2024-05-12T21:46:29.931 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers 2024-05-12T21:46:29.931 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/acpi 2024-05-12T21:46:29.931 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/acpi/acpi_ipmi.ko 2024-05-12T21:46:29.931 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/acpi/acpi_pad.ko 2024-05-12T21:46:29.931 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/acpi/ec_sys.ko 2024-05-12T21:46:29.931 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/acpi/platform_profile.ko 2024-05-12T21:46:29.931 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/acpi/sbs.ko 2024-05-12T21:46:29.931 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/acpi/sbshc.ko 2024-05-12T21:46:29.931 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/acpi/video.ko 2024-05-12T21:46:29.931 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ata 2024-05-12T21:46:29.932 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ata/acard-ahci.ko 2024-05-12T21:46:29.932 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ata/ahci_platform.ko 2024-05-12T21:46:29.932 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ata/ata_generic.ko 2024-05-12T21:46:29.932 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ata/libahci_platform.ko 2024-05-12T21:46:29.932 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ata/pata_acpi.ko 2024-05-12T21:46:29.932 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ata/pata_ali.ko 2024-05-12T21:46:29.932 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ata/pata_amd.ko 2024-05-12T21:46:29.932 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ata/pata_artop.ko 2024-05-12T21:46:29.932 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ata/pata_atiixp.ko 2024-05-12T21:46:29.932 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ata/pata_atp867x.ko 2024-05-12T21:46:29.932 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ata/pata_cmd640.ko 2024-05-12T21:46:29.932 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ata/pata_cmd64x.ko 2024-05-12T21:46:29.932 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ata/pata_cypress.ko 2024-05-12T21:46:29.932 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ata/pata_efar.ko 2024-05-12T21:46:29.932 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ata/pata_hpt366.ko 2024-05-12T21:46:29.933 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ata/pata_hpt37x.ko 2024-05-12T21:46:29.933 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ata/pata_hpt3x2n.ko 2024-05-12T21:46:29.933 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ata/pata_hpt3x3.ko 2024-05-12T21:46:29.933 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ata/pata_it8213.ko 2024-05-12T21:46:29.933 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ata/pata_it821x.ko 2024-05-12T21:46:29.933 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ata/pata_jmicron.ko 2024-05-12T21:46:29.933 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ata/pata_marvell.ko 2024-05-12T21:46:29.933 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ata/pata_mpiix.ko 2024-05-12T21:46:29.933 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ata/pata_netcell.ko 2024-05-12T21:46:29.933 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ata/pata_ninja32.ko 2024-05-12T21:46:29.933 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ata/pata_ns87410.ko 2024-05-12T21:46:29.933 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ata/pata_ns87415.ko 2024-05-12T21:46:29.933 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ata/pata_oldpiix.ko 2024-05-12T21:46:29.933 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ata/pata_opti.ko 2024-05-12T21:46:29.934 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ata/pata_optidma.ko 2024-05-12T21:46:29.934 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ata/pata_pcmcia.ko 2024-05-12T21:46:29.934 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ata/pata_pdc2027x.ko 2024-05-12T21:46:29.934 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-05-12T21:46:29.934 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ata/pata_piccolo.ko 2024-05-12T21:46:29.934 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ata/pata_rdc.ko 2024-05-12T21:46:29.934 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ata/pata_sch.ko 2024-05-12T21:46:29.934 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ata/pata_serverworks.ko 2024-05-12T21:46:29.934 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ata/pata_sil680.ko 2024-05-12T21:46:29.934 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ata/pata_sis.ko 2024-05-12T21:46:29.935 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ata/pata_sl82c105.ko 2024-05-12T21:46:29.935 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ata/pata_triflex.ko 2024-05-12T21:46:29.935 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ata/pata_via.ko 2024-05-12T21:46:29.935 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ata/pdc_adma.ko 2024-05-12T21:46:29.935 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ata/sata_inic162x.ko 2024-05-12T21:46:29.935 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ata/sata_mv.ko 2024-05-12T21:46:29.935 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ata/sata_nv.ko 2024-05-12T21:46:29.935 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ata/sata_promise.ko 2024-05-12T21:46:29.935 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ata/sata_qstor.ko 2024-05-12T21:46:29.935 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ata/sata_sil.ko 2024-05-12T21:46:29.935 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ata/sata_sil24.ko 2024-05-12T21:46:29.935 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ata/sata_sis.ko 2024-05-12T21:46:29.935 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ata/sata_svw.ko 2024-05-12T21:46:29.935 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ata/sata_sx4.ko 2024-05-12T21:46:29.935 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ata/sata_uli.ko 2024-05-12T21:46:29.935 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ata/sata_via.ko 2024-05-12T21:46:29.936 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ata/sata_vsc.ko 2024-05-12T21:46:29.936 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/auxdisplay 2024-05-12T21:46:29.936 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/auxdisplay/ks0108.ko 2024-05-12T21:46:29.936 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/base 2024-05-12T21:46:29.936 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/base/regmap 2024-05-12T21:46:29.936 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/base/regmap/regmap-i2c.ko 2024-05-12T21:46:29.936 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/base/regmap/regmap-mmio.ko 2024-05-12T21:46:29.936 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/bcma 2024-05-12T21:46:29.936 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/bcma/bcma.ko 2024-05-12T21:46:29.936 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/block 2024-05-12T21:46:29.936 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/block/aoe 2024-05-12T21:46:29.936 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/block/aoe/aoe.ko 2024-05-12T21:46:29.936 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/block/brd.ko 2024-05-12T21:46:29.936 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/block/drbd 2024-05-12T21:46:29.936 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/block/drbd/drbd.ko 2024-05-12T21:46:29.937 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/block/floppy.ko 2024-05-12T21:46:29.937 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/block/loop.ko 2024-05-12T21:46:29.937 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/block/mtip32xx 2024-05-12T21:46:29.937 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-05-12T21:46:29.937 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/block/nbd.ko 2024-05-12T21:46:29.937 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/block/null_blk 2024-05-12T21:46:29.937 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/block/null_blk/null_blk.ko 2024-05-12T21:46:29.937 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/block/pktcdvd.ko 2024-05-12T21:46:29.937 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/block/rbd.ko 2024-05-12T21:46:29.937 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/block/virtio_blk.ko 2024-05-12T21:46:29.938 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/block/xen-blkback 2024-05-12T21:46:29.938 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-05-12T21:46:29.938 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/block/xen-blkfront.ko 2024-05-12T21:46:29.938 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/block/zram 2024-05-12T21:46:29.938 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/block/zram/zram.ko 2024-05-12T21:46:29.938 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/char 2024-05-12T21:46:29.938 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/char/hangcheck-timer.ko 2024-05-12T21:46:29.938 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/char/hw_random 2024-05-12T21:46:29.938 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/char/hw_random/amd-rng.ko 2024-05-12T21:46:29.938 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/char/hw_random/intel-rng.ko 2024-05-12T21:46:29.938 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-05-12T21:46:29.938 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/char/hw_random/via-rng.ko 2024-05-12T21:46:29.938 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/char/hw_random/virtio-rng.ko 2024-05-12T21:46:29.938 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/char/ipmi 2024-05-12T21:46:29.939 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-05-12T21:46:29.939 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-05-12T21:46:29.939 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-05-12T21:46:29.939 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/char/ipmi/ipmi_si.ko 2024-05-12T21:46:29.939 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-05-12T21:46:29.939 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-05-12T21:46:29.939 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/char/lp.ko 2024-05-12T21:46:29.939 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/char/mwave 2024-05-12T21:46:29.939 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/char/mwave/mwave.ko 2024-05-12T21:46:29.940 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/char/ppdev.ko 2024-05-12T21:46:29.940 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/char/tlclk.ko 2024-05-12T21:46:29.940 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/char/tpm 2024-05-12T21:46:29.940 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/char/tpm/tpm.ko 2024-05-12T21:46:29.940 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/char/tpm/tpm_atmel.ko 2024-05-12T21:46:29.941 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/char/tpm/tpm_infineon.ko 2024-05-12T21:46:29.941 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/char/tpm/tpm_nsc.ko 2024-05-12T21:46:29.941 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/char/tpm/tpm_tis.ko 2024-05-12T21:46:29.941 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-05-12T21:46:29.941 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/char/uv_mmtimer.ko 2024-05-12T21:46:29.941 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/char/virtio_console.ko 2024-05-12T21:46:29.941 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/cpufreq 2024-05-12T21:46:29.941 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-05-12T21:46:29.941 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-05-12T21:46:29.941 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/cpufreq/p4-clockmod.ko 2024-05-12T21:46:29.942 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-05-12T21:46:29.942 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/cpufreq/powernow-k8.ko 2024-05-12T21:46:29.942 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/cpufreq/speedstep-lib.ko 2024-05-12T21:46:29.942 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/crypto 2024-05-12T21:46:29.942 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/crypto/ccp 2024-05-12T21:46:29.942 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-05-12T21:46:29.942 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/crypto/ccp/ccp.ko 2024-05-12T21:46:29.942 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/crypto/intel 2024-05-12T21:46:29.942 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/crypto/intel/qat 2024-05-12T21:46:29.942 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/crypto/intel/qat/qat_common 2024-05-12T21:46:29.942 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-05-12T21:46:29.942 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-05-12T21:46:29.942 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-05-12T21:46:29.942 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/crypto/padlock-aes.ko 2024-05-12T21:46:29.943 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/crypto/padlock-sha.ko 2024-05-12T21:46:29.943 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/crypto/virtio 2024-05-12T21:46:29.943 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-05-12T21:46:29.943 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/dca 2024-05-12T21:46:29.943 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/dca/dca.ko 2024-05-12T21:46:29.943 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/devfreq 2024-05-12T21:46:29.943 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-05-12T21:46:29.943 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/dma 2024-05-12T21:46:29.943 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/dma/dw 2024-05-12T21:46:29.943 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/dma/dw/dw_dmac.ko 2024-05-12T21:46:29.943 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/dma/ioat 2024-05-12T21:46:29.943 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/dma/ioat/ioatdma.ko 2024-05-12T21:46:29.943 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/edac 2024-05-12T21:46:29.944 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/edac/amd64_edac.ko 2024-05-12T21:46:29.944 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/edac/e752x_edac.ko 2024-05-12T21:46:29.944 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/edac/edac_mce_amd.ko 2024-05-12T21:46:29.944 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/edac/i3000_edac.ko 2024-05-12T21:46:29.944 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/edac/i3200_edac.ko 2024-05-12T21:46:29.944 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/edac/i5100_edac.ko 2024-05-12T21:46:29.944 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/edac/i5400_edac.ko 2024-05-12T21:46:29.944 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/edac/i7300_edac.ko 2024-05-12T21:46:29.944 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/edac/i7core_edac.ko 2024-05-12T21:46:29.944 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/edac/i82975x_edac.ko 2024-05-12T21:46:29.944 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/edac/ie31200_edac.ko 2024-05-12T21:46:29.944 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/edac/sb_edac.ko 2024-05-12T21:46:29.944 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/edac/x38_edac.ko 2024-05-12T21:46:29.944 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/firmware 2024-05-12T21:46:29.944 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/firmware/edd.ko 2024-05-12T21:46:29.945 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/firmware/iscsi_ibft.ko 2024-05-12T21:46:29.945 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/gpio 2024-05-12T21:46:29.945 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/gpio/gpio-ich.ko 2024-05-12T21:46:29.945 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/gpio/gpio-viperboard.ko 2024-05-12T21:46:29.945 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid 2024-05-12T21:46:29.945 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/hid-appleir.ko 2024-05-12T21:46:29.945 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/hid-aureal.ko 2024-05-12T21:46:29.945 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/hid-axff.ko 2024-05-12T21:46:29.945 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/hid-dr.ko 2024-05-12T21:46:29.945 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/hid-elecom.ko 2024-05-12T21:46:29.945 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/hid-elo.ko 2024-05-12T21:46:29.945 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/hid-emsff.ko 2024-05-12T21:46:29.945 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/hid-gaff.ko 2024-05-12T21:46:29.945 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/hid-gt683r.ko 2024-05-12T21:46:29.946 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/hid-gyration.ko 2024-05-12T21:46:29.946 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/hid-holtek-kbd.ko 2024-05-12T21:46:29.946 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/hid-holtek-mouse.ko 2024-05-12T21:46:29.946 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/hid-holtekff.ko 2024-05-12T21:46:29.946 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/hid-hyperv.ko 2024-05-12T21:46:29.946 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/hid-icade.ko 2024-05-12T21:46:29.946 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/hid-keytouch.ko 2024-05-12T21:46:29.946 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/hid-kye.ko 2024-05-12T21:46:29.946 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/hid-lcpower.ko 2024-05-12T21:46:29.946 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/hid-led.ko 2024-05-12T21:46:29.946 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/hid-lenovo.ko 2024-05-12T21:46:29.946 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/hid-logitech-dj.ko 2024-05-12T21:46:29.946 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-05-12T21:46:29.946 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/hid-multitouch.ko 2024-05-12T21:46:29.946 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/hid-ortek.ko 2024-05-12T21:46:29.947 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/hid-penmount.ko 2024-05-12T21:46:29.947 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/hid-petalynx.ko 2024-05-12T21:46:29.947 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/hid-picolcd.ko 2024-05-12T21:46:29.947 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/hid-pl.ko 2024-05-12T21:46:29.947 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/hid-plantronics.ko 2024-05-12T21:46:29.947 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/hid-primax.ko 2024-05-12T21:46:29.947 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/hid-rmi.ko 2024-05-12T21:46:29.947 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/hid-roccat-arvo.ko 2024-05-12T21:46:29.947 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/hid-roccat-common.ko 2024-05-12T21:46:29.947 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/hid-roccat-isku.ko 2024-05-12T21:46:29.947 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/hid-roccat-kone.ko 2024-05-12T21:46:29.947 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-05-12T21:46:29.947 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/hid-roccat-konepure.ko 2024-05-12T21:46:29.947 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-05-12T21:46:29.947 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/hid-roccat-lua.ko 2024-05-12T21:46:29.948 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/hid-roccat-pyra.ko 2024-05-12T21:46:29.948 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/hid-roccat-ryos.ko 2024-05-12T21:46:29.948 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/hid-roccat-savu.ko 2024-05-12T21:46:29.948 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/hid-roccat.ko 2024-05-12T21:46:29.948 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/hid-saitek.ko 2024-05-12T21:46:29.948 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/hid-samsung.ko 2024-05-12T21:46:29.948 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/hid-sensor-hub.ko 2024-05-12T21:46:29.948 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/hid-sjoy.ko 2024-05-12T21:46:29.948 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/hid-sony.ko 2024-05-12T21:46:29.948 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/hid-speedlink.ko 2024-05-12T21:46:29.948 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/hid-steelseries.ko 2024-05-12T21:46:29.948 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/hid-sunplus.ko 2024-05-12T21:46:29.948 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/hid-thrustmaster.ko 2024-05-12T21:46:29.948 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/hid-tivo.ko 2024-05-12T21:46:29.948 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/hid-tmff.ko 2024-05-12T21:46:29.949 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/hid-topseed.ko 2024-05-12T21:46:29.949 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/hid-twinhan.ko 2024-05-12T21:46:29.949 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/hid-uclogic.ko 2024-05-12T21:46:29.949 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/hid-waltop.ko 2024-05-12T21:46:29.949 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/hid-wiimote.ko 2024-05-12T21:46:29.949 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/hid-xinmo.ko 2024-05-12T21:46:29.949 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/hid-zpff.ko 2024-05-12T21:46:29.949 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/hid-zydacron.ko 2024-05-12T21:46:29.949 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/uhid.ko 2024-05-12T21:46:29.949 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hid/wacom.ko 2024-05-12T21:46:29.949 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hv 2024-05-12T21:46:29.949 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hv/hv_balloon.ko 2024-05-12T21:46:29.949 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hv/hv_utils.ko 2024-05-12T21:46:29.949 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hv/hv_vmbus.ko 2024-05-12T21:46:29.950 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon 2024-05-12T21:46:29.950 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/abituguru.ko 2024-05-12T21:46:29.950 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/abituguru3.ko 2024-05-12T21:46:29.950 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/acpi_power_meter.ko 2024-05-12T21:46:29.950 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/ad7414.ko 2024-05-12T21:46:29.950 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/ad7418.ko 2024-05-12T21:46:29.950 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/adc128d818.ko 2024-05-12T21:46:29.950 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/adm1025.ko 2024-05-12T21:46:29.951 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/adm1026.ko 2024-05-12T21:46:29.951 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/adm1029.ko 2024-05-12T21:46:29.951 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/adm1031.ko 2024-05-12T21:46:29.951 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/adm9240.ko 2024-05-12T21:46:29.951 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/ads7828.ko 2024-05-12T21:46:29.951 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/adt7410.ko 2024-05-12T21:46:29.951 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/adt7411.ko 2024-05-12T21:46:29.951 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/adt7462.ko 2024-05-12T21:46:29.951 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/adt7470.ko 2024-05-12T21:46:29.951 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/adt7475.ko 2024-05-12T21:46:29.951 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/adt7x10.ko 2024-05-12T21:46:29.951 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/amc6821.ko 2024-05-12T21:46:29.953 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/applesmc.ko 2024-05-12T21:46:29.953 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/asb100.ko 2024-05-12T21:46:29.953 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/asc7621.ko 2024-05-12T21:46:29.953 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/asus_atk0110.ko 2024-05-12T21:46:29.953 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/atxp1.ko 2024-05-12T21:46:29.953 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/coretemp.ko 2024-05-12T21:46:29.954 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-05-12T21:46:29.954 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/dme1737.ko 2024-05-12T21:46:29.954 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/ds1621.ko 2024-05-12T21:46:29.954 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/ds620.ko 2024-05-12T21:46:29.954 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/emc1403.ko 2024-05-12T21:46:29.954 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/emc6w201.ko 2024-05-12T21:46:29.954 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/f71805f.ko 2024-05-12T21:46:29.954 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/f71882fg.ko 2024-05-12T21:46:29.954 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/f75375s.ko 2024-05-12T21:46:29.954 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/fam15h_power.ko 2024-05-12T21:46:29.954 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/fschmd.ko 2024-05-12T21:46:29.954 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/g760a.ko 2024-05-12T21:46:29.954 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/g762.ko 2024-05-12T21:46:29.954 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/gl518sm.ko 2024-05-12T21:46:29.955 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/gl520sm.ko 2024-05-12T21:46:29.955 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/hwmon-vid.ko 2024-05-12T21:46:29.955 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/i5500_temp.ko 2024-05-12T21:46:29.955 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/i5k_amb.ko 2024-05-12T21:46:29.955 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/ibmaem.ko 2024-05-12T21:46:29.955 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/ibmpex.ko 2024-05-12T21:46:29.955 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/ina209.ko 2024-05-12T21:46:29.955 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/ina2xx.ko 2024-05-12T21:46:29.955 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/it87.ko 2024-05-12T21:46:29.955 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/k10temp.ko 2024-05-12T21:46:29.955 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/k8temp.ko 2024-05-12T21:46:29.955 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/lineage-pem.ko 2024-05-12T21:46:29.955 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/lm63.ko 2024-05-12T21:46:29.955 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/lm73.ko 2024-05-12T21:46:29.955 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/lm75.ko 2024-05-12T21:46:29.956 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/lm77.ko 2024-05-12T21:46:29.956 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/lm78.ko 2024-05-12T21:46:29.956 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/lm80.ko 2024-05-12T21:46:29.956 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/lm83.ko 2024-05-12T21:46:29.956 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/lm85.ko 2024-05-12T21:46:29.956 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/lm87.ko 2024-05-12T21:46:29.956 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/lm90.ko 2024-05-12T21:46:29.956 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/lm92.ko 2024-05-12T21:46:29.956 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/lm93.ko 2024-05-12T21:46:29.956 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/lm95234.ko 2024-05-12T21:46:29.956 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/lm95241.ko 2024-05-12T21:46:29.956 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/lm95245.ko 2024-05-12T21:46:29.956 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/ltc2945.ko 2024-05-12T21:46:29.956 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/ltc4151.ko 2024-05-12T21:46:29.956 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/ltc4215.ko 2024-05-12T21:46:29.957 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/ltc4222.ko 2024-05-12T21:46:29.957 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/ltc4245.ko 2024-05-12T21:46:29.957 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/ltc4260.ko 2024-05-12T21:46:29.957 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/ltc4261.ko 2024-05-12T21:46:29.957 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/max16065.ko 2024-05-12T21:46:29.957 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/max1619.ko 2024-05-12T21:46:29.957 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/max1668.ko 2024-05-12T21:46:29.957 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/max197.ko 2024-05-12T21:46:29.957 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/max6639.ko 2024-05-12T21:46:29.957 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/max6650.ko 2024-05-12T21:46:29.957 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/max6697.ko 2024-05-12T21:46:29.957 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/mcp3021.ko 2024-05-12T21:46:29.957 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/nct6683.ko 2024-05-12T21:46:29.957 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/nct6775-core.ko 2024-05-12T21:46:29.957 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/nct6775.ko 2024-05-12T21:46:29.958 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/nct7802.ko 2024-05-12T21:46:29.958 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/pc87360.ko 2024-05-12T21:46:29.958 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/pc87427.ko 2024-05-12T21:46:29.958 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/pcf8591.ko 2024-05-12T21:46:29.958 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/pmbus 2024-05-12T21:46:29.958 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-05-12T21:46:29.958 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-05-12T21:46:29.958 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-05-12T21:46:29.958 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/pmbus/max16064.ko 2024-05-12T21:46:29.958 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/pmbus/max34440.ko 2024-05-12T21:46:29.958 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/pmbus/max8688.ko 2024-05-12T21:46:29.958 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-05-12T21:46:29.958 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-05-12T21:46:29.958 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-05-12T21:46:29.959 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-05-12T21:46:29.959 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-05-12T21:46:29.959 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-05-12T21:46:29.959 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/powr1220.ko 2024-05-12T21:46:29.959 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/sch5627.ko 2024-05-12T21:46:29.959 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/sch5636.ko 2024-05-12T21:46:29.959 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/sch56xx-common.ko 2024-05-12T21:46:29.959 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/sht15.ko 2024-05-12T21:46:29.959 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/sht21.ko 2024-05-12T21:46:29.959 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/shtc1.ko 2024-05-12T21:46:29.959 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/sis5595.ko 2024-05-12T21:46:29.959 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/smsc47b397.ko 2024-05-12T21:46:29.959 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/smsc47m1.ko 2024-05-12T21:46:29.959 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/smsc47m192.ko 2024-05-12T21:46:29.959 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/thmc50.ko 2024-05-12T21:46:29.960 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/tmp102.ko 2024-05-12T21:46:29.960 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/tmp103.ko 2024-05-12T21:46:29.960 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/tmp401.ko 2024-05-12T21:46:29.960 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/tmp421.ko 2024-05-12T21:46:29.960 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/via-cputemp.ko 2024-05-12T21:46:29.960 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/via686a.ko 2024-05-12T21:46:29.960 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/vt1211.ko 2024-05-12T21:46:29.960 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/vt8231.ko 2024-05-12T21:46:29.960 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/w83627ehf.ko 2024-05-12T21:46:29.961 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/w83627hf.ko 2024-05-12T21:46:29.961 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/w83781d.ko 2024-05-12T21:46:29.961 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/w83791d.ko 2024-05-12T21:46:29.961 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/w83792d.ko 2024-05-12T21:46:29.961 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/w83793.ko 2024-05-12T21:46:29.961 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/w83795.ko 2024-05-12T21:46:29.961 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/w83l785ts.ko 2024-05-12T21:46:29.961 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/hwmon/w83l786ng.ko 2024-05-12T21:46:29.961 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/i2c 2024-05-12T21:46:29.961 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/i2c/algos 2024-05-12T21:46:29.961 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-05-12T21:46:29.961 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-05-12T21:46:29.962 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/i2c/busses 2024-05-12T21:46:29.962 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-05-12T21:46:29.962 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-05-12T21:46:29.962 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-05-12T21:46:29.962 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-05-12T21:46:29.962 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-05-12T21:46:29.962 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-05-12T21:46:29.962 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-05-12T21:46:29.962 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-05-12T21:46:29.962 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/i2c/busses/i2c-i801.ko 2024-05-12T21:46:29.962 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/i2c/busses/i2c-isch.ko 2024-05-12T21:46:29.962 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-05-12T21:46:29.962 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-05-12T21:46:29.963 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-05-12T21:46:29.963 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/i2c/busses/i2c-parport.ko 2024-05-12T21:46:29.963 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-05-12T21:46:29.963 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-05-12T21:46:29.963 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-05-12T21:46:29.963 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-05-12T21:46:29.963 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-05-12T21:46:29.963 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-05-12T21:46:29.963 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/i2c/busses/i2c-via.ko 2024-05-12T21:46:29.963 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-05-12T21:46:29.963 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-05-12T21:46:29.963 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/i2c/i2c-dev.ko 2024-05-12T21:46:29.963 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/i2c/i2c-mux.ko 2024-05-12T21:46:29.963 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-05-12T21:46:29.963 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/i2c/i2c-smbus.ko 2024-05-12T21:46:29.964 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/i2c/i2c-stub.ko 2024-05-12T21:46:29.964 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/infiniband 2024-05-12T21:46:29.964 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/infiniband/core 2024-05-12T21:46:29.964 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/infiniband/core/ib_cm.ko 2024-05-12T21:46:29.964 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/infiniband/core/ib_core.ko 2024-05-12T21:46:29.964 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/infiniband/core/ib_umad.ko 2024-05-12T21:46:29.964 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-05-12T21:46:29.964 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/infiniband/core/iw_cm.ko 2024-05-12T21:46:29.964 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/infiniband/core/rdma_cm.ko 2024-05-12T21:46:29.964 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-05-12T21:46:29.964 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/infiniband/hw 2024-05-12T21:46:29.964 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/infiniband/hw/cxgb4 2024-05-12T21:46:29.964 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-05-12T21:46:29.964 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/infiniband/hw/mlx4 2024-05-12T21:46:29.964 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-05-12T21:46:29.965 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/infiniband/hw/mlx5 2024-05-12T21:46:29.965 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-05-12T21:46:29.965 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/infiniband/hw/mthca 2024-05-12T21:46:29.965 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-05-12T21:46:29.965 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/infiniband/ulp 2024-05-12T21:46:29.965 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/infiniband/ulp/ipoib 2024-05-12T21:46:29.965 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-05-12T21:46:29.965 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/infiniband/ulp/iser 2024-05-12T21:46:29.965 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-05-12T21:46:29.965 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/infiniband/ulp/isert 2024-05-12T21:46:29.965 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-05-12T21:46:29.965 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/infiniband/ulp/srp 2024-05-12T21:46:29.965 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-05-12T21:46:29.965 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/infiniband/ulp/srpt 2024-05-12T21:46:29.965 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-05-12T21:46:29.966 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/input 2024-05-12T21:46:29.966 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/input/gameport 2024-05-12T21:46:29.966 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-05-12T21:46:29.966 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/input/gameport/fm801-gp.ko 2024-05-12T21:46:29.966 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/input/gameport/gameport.ko 2024-05-12T21:46:29.966 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/input/joydev.ko 2024-05-12T21:46:29.966 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/input/keyboard 2024-05-12T21:46:29.966 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/input/keyboard/gpio_keys.ko 2024-05-12T21:46:29.966 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/input/mouse 2024-05-12T21:46:29.966 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/input/mouse/appletouch.ko 2024-05-12T21:46:29.966 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/input/mouse/bcm5974.ko 2024-05-12T21:46:29.966 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/input/mouse/cyapatp.ko 2024-05-12T21:46:29.966 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/input/mouse/elan_i2c.ko 2024-05-12T21:46:29.966 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/input/mouse/sermouse.ko 2024-05-12T21:46:29.966 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-05-12T21:46:29.967 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/input/mouse/synaptics_usb.ko 2024-05-12T21:46:29.967 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/input/mouse/vsxxxaa.ko 2024-05-12T21:46:29.967 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/input/rmi4 2024-05-12T21:46:29.967 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/input/rmi4/rmi_core.ko 2024-05-12T21:46:29.967 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/input/serio 2024-05-12T21:46:29.967 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/input/serio/altera_ps2.ko 2024-05-12T21:46:29.967 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/input/serio/arc_ps2.ko 2024-05-12T21:46:29.967 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-05-12T21:46:29.967 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/input/serio/serio_raw.ko 2024-05-12T21:46:29.967 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/input/sparse-keymap.ko 2024-05-12T21:46:29.967 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/leds 2024-05-12T21:46:29.967 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/leds/leds-blinkm.ko 2024-05-12T21:46:29.967 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/leds/leds-lm3530.ko 2024-05-12T21:46:29.967 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/leds/leds-lp3944.ko 2024-05-12T21:46:29.967 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/leds/leds-ss4200.ko 2024-05-12T21:46:29.968 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/leds/trigger 2024-05-12T21:46:29.968 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-05-12T21:46:29.968 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-05-12T21:46:29.968 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-05-12T21:46:29.968 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-05-12T21:46:29.968 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-05-12T21:46:29.968 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-05-12T21:46:29.968 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-05-12T21:46:29.968 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-05-12T21:46:29.968 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-05-12T21:46:29.968 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/md 2024-05-12T21:46:29.968 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/md/bcache 2024-05-12T21:46:29.968 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/md/bcache/bcache.ko 2024-05-12T21:46:29.968 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/md/dm-bio-prison.ko 2024-05-12T21:46:29.968 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/md/dm-cache-smq.ko 2024-05-12T21:46:29.968 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/md/dm-cache.ko 2024-05-12T21:46:29.969 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/md/dm-crypt.ko 2024-05-12T21:46:29.969 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/md/dm-delay.ko 2024-05-12T21:46:29.969 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/md/dm-flakey.ko 2024-05-12T21:46:29.969 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/md/dm-log-userspace.ko 2024-05-12T21:46:29.969 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/md/dm-multipath.ko 2024-05-12T21:46:29.969 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/md/dm-queue-length.ko 2024-05-12T21:46:29.969 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/md/dm-raid.ko 2024-05-12T21:46:29.969 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/md/dm-round-robin.ko 2024-05-12T21:46:29.969 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/md/dm-service-time.ko 2024-05-12T21:46:29.969 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/md/dm-switch.ko 2024-05-12T21:46:29.969 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/md/dm-thin-pool.ko 2024-05-12T21:46:29.969 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/md/dm-verity.ko 2024-05-12T21:46:29.969 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/md/persistent-data 2024-05-12T21:46:29.969 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-05-12T21:46:29.970 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/md/raid0.ko 2024-05-12T21:46:29.970 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/md/raid1.ko 2024-05-12T21:46:29.970 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/md/raid10.ko 2024-05-12T21:46:29.970 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/md/raid456.ko 2024-05-12T21:46:29.970 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/message 2024-05-12T21:46:29.970 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/message/fusion 2024-05-12T21:46:29.970 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/message/fusion/mptbase.ko 2024-05-12T21:46:29.970 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/message/fusion/mptctl.ko 2024-05-12T21:46:29.970 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/message/fusion/mptfc.ko 2024-05-12T21:46:29.970 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/message/fusion/mptlan.ko 2024-05-12T21:46:29.970 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/message/fusion/mptsas.ko 2024-05-12T21:46:29.970 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/message/fusion/mptscsih.ko 2024-05-12T21:46:29.970 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/message/fusion/mptspi.ko 2024-05-12T21:46:29.970 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/mfd 2024-05-12T21:46:29.970 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/mfd/lpc_ich.ko 2024-05-12T21:46:29.971 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/mfd/lpc_sch.ko 2024-05-12T21:46:29.971 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/mfd/mfd-core.ko 2024-05-12T21:46:29.971 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/mfd/sm501.ko 2024-05-12T21:46:29.971 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/mfd/viperboard.ko 2024-05-12T21:46:29.971 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/mfd/vx855.ko 2024-05-12T21:46:29.971 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/mfd/wl1273-core.ko 2024-05-12T21:46:29.971 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/misc 2024-05-12T21:46:29.971 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/misc/altera-stapl 2024-05-12T21:46:29.971 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-05-12T21:46:29.972 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/misc/apds9802als.ko 2024-05-12T21:46:29.972 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/misc/apds990x.ko 2024-05-12T21:46:29.972 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/misc/bh1770glc.ko 2024-05-12T21:46:29.972 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/misc/cb710 2024-05-12T21:46:29.972 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/misc/cb710/cb710.ko 2024-05-12T21:46:29.972 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/misc/eeprom 2024-05-12T21:46:29.972 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/misc/eeprom/at24.ko 2024-05-12T21:46:29.972 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-05-12T21:46:29.972 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/misc/eeprom/max6875.ko 2024-05-12T21:46:29.972 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/misc/enclosure.ko 2024-05-12T21:46:29.972 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/misc/hpilo.ko 2024-05-12T21:46:29.972 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/misc/isl29003.ko 2024-05-12T21:46:29.972 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/misc/isl29020.ko 2024-05-12T21:46:29.972 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/misc/lis3lv02d 2024-05-12T21:46:29.972 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-05-12T21:46:29.973 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-05-12T21:46:29.973 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/misc/mei 2024-05-12T21:46:29.973 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/misc/mei/mei-me.ko 2024-05-12T21:46:29.973 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/misc/mei/mei-txe.ko 2024-05-12T21:46:29.973 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/misc/mei/mei.ko 2024-05-12T21:46:29.973 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/misc/sgi-gru 2024-05-12T21:46:29.973 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/misc/sgi-gru/gru.ko 2024-05-12T21:46:29.973 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/misc/sgi-xp 2024-05-12T21:46:29.973 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/misc/sgi-xp/xp.ko 2024-05-12T21:46:29.973 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/misc/sgi-xp/xpc.ko 2024-05-12T21:46:29.973 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-05-12T21:46:29.973 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/misc/tifm_7xx1.ko 2024-05-12T21:46:29.973 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/misc/tifm_core.ko 2024-05-12T21:46:29.973 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/misc/tsl2550.ko 2024-05-12T21:46:29.974 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/misc/vmw_balloon.ko 2024-05-12T21:46:29.974 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/misc/vmw_vmci 2024-05-12T21:46:29.974 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-05-12T21:46:29.974 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net 2024-05-12T21:46:29.974 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/bonding 2024-05-12T21:46:29.974 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/bonding/bonding.ko 2024-05-12T21:46:29.974 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/dsa 2024-05-12T21:46:29.974 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/dsa/b53 2024-05-12T21:46:29.974 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/dsa/b53/b53_common.ko 2024-05-12T21:46:29.974 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/dsa/bcm-sf2.ko 2024-05-12T21:46:29.974 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/dsa/mv88e6060.ko 2024-05-12T21:46:29.974 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/dsa/mv88e6xxx 2024-05-12T21:46:29.974 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-05-12T21:46:29.975 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/dummy.ko 2024-05-12T21:46:29.975 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/eql.ko 2024-05-12T21:46:29.975 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet 2024-05-12T21:46:29.975 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/3com 2024-05-12T21:46:29.975 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-05-12T21:46:29.975 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-05-12T21:46:29.975 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-05-12T21:46:29.975 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-05-12T21:46:29.975 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/8390 2024-05-12T21:46:29.975 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/8390/8390.ko 2024-05-12T21:46:29.975 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-05-12T21:46:29.975 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-05-12T21:46:29.975 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-05-12T21:46:29.976 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/adaptec 2024-05-12T21:46:29.976 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-05-12T21:46:29.976 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/agere 2024-05-12T21:46:29.976 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/agere/et131x.ko 2024-05-12T21:46:29.976 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/alteon 2024-05-12T21:46:29.976 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-05-12T21:46:29.976 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/altera 2024-05-12T21:46:29.976 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-05-12T21:46:29.976 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/amd 2024-05-12T21:46:29.976 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-05-12T21:46:29.976 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-05-12T21:46:29.977 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-05-12T21:46:29.977 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/atheros 2024-05-12T21:46:29.977 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/atheros/alx 2024-05-12T21:46:29.977 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-05-12T21:46:29.977 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/atheros/atl1c 2024-05-12T21:46:29.977 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-05-12T21:46:29.977 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/atheros/atl1e 2024-05-12T21:46:29.977 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-05-12T21:46:29.978 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/atheros/atlx 2024-05-12T21:46:29.978 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-05-12T21:46:29.978 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-05-12T21:46:29.978 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/broadcom 2024-05-12T21:46:29.978 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-05-12T21:46:29.978 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-05-12T21:46:29.978 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-05-12T21:46:29.978 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-05-12T21:46:29.978 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-05-12T21:46:29.978 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/broadcom/genet 2024-05-12T21:46:29.978 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-05-12T21:46:29.978 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-05-12T21:46:29.979 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/brocade 2024-05-12T21:46:29.979 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/brocade/bna 2024-05-12T21:46:29.979 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-05-12T21:46:29.979 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/chelsio 2024-05-12T21:46:29.979 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/chelsio/cxgb 2024-05-12T21:46:29.979 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-05-12T21:46:29.979 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-05-12T21:46:29.979 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-05-12T21:46:29.979 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-05-12T21:46:29.979 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-05-12T21:46:29.979 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-05-12T21:46:29.979 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-05-12T21:46:29.980 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-05-12T21:46:29.980 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-05-12T21:46:29.980 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/cisco 2024-05-12T21:46:29.980 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/cisco/enic 2024-05-12T21:46:29.980 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-05-12T21:46:29.980 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/dec 2024-05-12T21:46:29.980 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/dec/tulip 2024-05-12T21:46:29.980 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-05-12T21:46:29.980 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-05-12T21:46:29.980 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-05-12T21:46:29.981 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-05-12T21:46:29.981 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-05-12T21:46:29.981 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-05-12T21:46:29.981 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/dlink 2024-05-12T21:46:29.981 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-05-12T21:46:29.981 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-05-12T21:46:29.981 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/dnet.ko 2024-05-12T21:46:29.981 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/emulex 2024-05-12T21:46:29.981 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/emulex/benet 2024-05-12T21:46:29.982 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-05-12T21:46:29.982 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/ethoc.ko 2024-05-12T21:46:29.982 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/fealnx.ko 2024-05-12T21:46:29.982 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/intel 2024-05-12T21:46:29.982 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/intel/e100.ko 2024-05-12T21:46:29.982 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/intel/e1000 2024-05-12T21:46:29.982 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-05-12T21:46:29.982 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/intel/e1000e 2024-05-12T21:46:29.982 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-05-12T21:46:29.982 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/intel/fm10k 2024-05-12T21:46:29.982 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-05-12T21:46:29.983 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/intel/i40e 2024-05-12T21:46:29.983 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-05-12T21:46:29.983 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/intel/iavf 2024-05-12T21:46:29.983 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-05-12T21:46:29.983 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/intel/igb 2024-05-12T21:46:29.983 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-05-12T21:46:29.983 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/intel/igbvf 2024-05-12T21:46:29.983 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-05-12T21:46:29.983 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/intel/ixgbe 2024-05-12T21:46:29.983 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-05-12T21:46:29.983 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/intel/ixgbevf 2024-05-12T21:46:29.984 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-05-12T21:46:29.984 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/jme.ko 2024-05-12T21:46:29.984 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/marvell 2024-05-12T21:46:29.984 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-05-12T21:46:29.984 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/marvell/skge.ko 2024-05-12T21:46:29.984 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-05-12T21:46:29.984 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/mellanox 2024-05-12T21:46:29.984 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/mellanox/mlx4 2024-05-12T21:46:29.984 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-05-12T21:46:29.984 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-05-12T21:46:29.984 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/mellanox/mlx5 2024-05-12T21:46:29.984 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-05-12T21:46:29.984 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-05-12T21:46:29.984 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/micrel 2024-05-12T21:46:29.984 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-05-12T21:46:29.985 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/myricom 2024-05-12T21:46:29.985 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/myricom/myri10ge 2024-05-12T21:46:29.985 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-05-12T21:46:29.985 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/natsemi 2024-05-12T21:46:29.985 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-05-12T21:46:29.985 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-05-12T21:46:29.985 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/neterion 2024-05-12T21:46:29.985 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-05-12T21:46:29.985 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/nvidia 2024-05-12T21:46:29.985 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-05-12T21:46:29.985 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/packetengines 2024-05-12T21:46:29.985 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-05-12T21:46:29.985 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-05-12T21:46:29.985 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/qlogic 2024-05-12T21:46:29.986 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/qlogic/netxen 2024-05-12T21:46:29.986 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-05-12T21:46:29.986 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-05-12T21:46:29.986 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-05-12T21:46:29.986 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-05-12T21:46:29.986 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/rdc 2024-05-12T21:46:29.986 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-05-12T21:46:29.986 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/realtek 2024-05-12T21:46:29.986 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-05-12T21:46:29.986 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-05-12T21:46:29.986 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/realtek/atp.ko 2024-05-12T21:46:29.986 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-05-12T21:46:29.986 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/rocker 2024-05-12T21:46:29.986 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-05-12T21:46:29.987 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/sfc 2024-05-12T21:46:29.987 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-05-12T21:46:29.987 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/silan 2024-05-12T21:46:29.987 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-05-12T21:46:29.987 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/sis 2024-05-12T21:46:29.987 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/sis/sis190.ko 2024-05-12T21:46:29.987 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/sis/sis900.ko 2024-05-12T21:46:29.987 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/smsc 2024-05-12T21:46:29.987 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-05-12T21:46:29.987 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-05-12T21:46:29.988 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-05-12T21:46:29.988 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-05-12T21:46:29.988 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/stmicro 2024-05-12T21:46:29.988 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/stmicro/stmmac 2024-05-12T21:46:29.988 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-05-12T21:46:29.988 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-05-12T21:46:29.988 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/sun 2024-05-12T21:46:29.988 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/sun/cassini.ko 2024-05-12T21:46:29.988 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/sun/niu.ko 2024-05-12T21:46:29.989 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/sun/sungem.ko 2024-05-12T21:46:29.989 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-05-12T21:46:29.989 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/tehuti 2024-05-12T21:46:29.989 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-05-12T21:46:29.989 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/ti 2024-05-12T21:46:29.989 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/ti/tlan.ko 2024-05-12T21:46:29.989 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/via 2024-05-12T21:46:29.989 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-05-12T21:46:29.989 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-05-12T21:46:29.989 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/wiznet 2024-05-12T21:46:29.989 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-05-12T21:46:29.989 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-05-12T21:46:29.990 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/xircom 2024-05-12T21:46:29.990 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-05-12T21:46:29.990 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/geneve.ko 2024-05-12T21:46:29.990 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/hyperv 2024-05-12T21:46:29.990 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-05-12T21:46:29.990 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ieee802154 2024-05-12T21:46:29.990 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ieee802154/fakelb.ko 2024-05-12T21:46:29.990 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ifb.ko 2024-05-12T21:46:29.990 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ipvlan 2024-05-12T21:46:29.990 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ipvlan/ipvlan.ko 2024-05-12T21:46:29.990 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/macvlan.ko 2024-05-12T21:46:29.991 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/macvtap.ko 2024-05-12T21:46:29.991 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/mdio 2024-05-12T21:46:29.991 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/mdio.ko 2024-05-12T21:46:29.991 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-05-12T21:46:29.991 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-05-12T21:46:29.991 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/mdio/mdio-regmap.ko 2024-05-12T21:46:29.991 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/mii.ko 2024-05-12T21:46:29.991 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/net_failover.ko 2024-05-12T21:46:29.991 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/netconsole.ko 2024-05-12T21:46:29.991 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/nlmon.ko 2024-05-12T21:46:29.992 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/pcs 2024-05-12T21:46:29.992 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/pcs/pcs-lynx.ko 2024-05-12T21:46:29.992 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-05-12T21:46:29.992 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/phy 2024-05-12T21:46:29.992 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/phy/amd.ko 2024-05-12T21:46:29.992 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-05-12T21:46:29.992 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-05-12T21:46:29.992 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/phy/bcm7xxx.ko 2024-05-12T21:46:29.992 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/phy/bcm87xx.ko 2024-05-12T21:46:29.992 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/phy/broadcom.ko 2024-05-12T21:46:29.992 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/phy/cicada.ko 2024-05-12T21:46:29.993 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/phy/davicom.ko 2024-05-12T21:46:29.993 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/phy/dp83640.ko 2024-05-12T21:46:29.993 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/phy/et1011c.ko 2024-05-12T21:46:29.993 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/phy/icplus.ko 2024-05-12T21:46:29.993 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/phy/lxt.ko 2024-05-12T21:46:29.993 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/phy/marvell.ko 2024-05-12T21:46:29.993 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/phy/micrel.ko 2024-05-12T21:46:29.993 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/phy/national.ko 2024-05-12T21:46:29.993 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/phy/phylink.ko 2024-05-12T21:46:29.993 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/phy/qsemi.ko 2024-05-12T21:46:29.993 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/phy/realtek.ko 2024-05-12T21:46:29.993 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/phy/smsc.ko 2024-05-12T21:46:29.994 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/phy/ste10Xp.ko 2024-05-12T21:46:29.994 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/phy/vitesse.ko 2024-05-12T21:46:29.994 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ppp 2024-05-12T21:46:29.994 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ppp/bsd_comp.ko 2024-05-12T21:46:29.994 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ppp/ppp_async.ko 2024-05-12T21:46:29.994 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ppp/ppp_deflate.ko 2024-05-12T21:46:29.994 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ppp/ppp_generic.ko 2024-05-12T21:46:29.994 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ppp/ppp_mppe.ko 2024-05-12T21:46:29.994 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ppp/ppp_synctty.ko 2024-05-12T21:46:29.994 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ppp/pppoe.ko 2024-05-12T21:46:29.994 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ppp/pppox.ko 2024-05-12T21:46:29.994 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/ppp/pptp.ko 2024-05-12T21:46:29.995 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/slip 2024-05-12T21:46:29.995 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/slip/slhc.ko 2024-05-12T21:46:29.995 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/slip/slip.ko 2024-05-12T21:46:29.995 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/sungem_phy.ko 2024-05-12T21:46:29.995 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/tap.ko 2024-05-12T21:46:29.995 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/team 2024-05-12T21:46:29.995 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/team/team.ko 2024-05-12T21:46:29.995 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/team/team_mode_activebackup.ko 2024-05-12T21:46:29.995 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/team/team_mode_broadcast.ko 2024-05-12T21:46:29.995 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-05-12T21:46:29.995 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/team/team_mode_random.ko 2024-05-12T21:46:29.995 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-05-12T21:46:29.995 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/tun.ko 2024-05-12T21:46:29.996 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/veth.ko 2024-05-12T21:46:29.996 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/virtio_net.ko 2024-05-12T21:46:29.996 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/vmxnet3 2024-05-12T21:46:29.996 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-05-12T21:46:29.996 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/vxlan 2024-05-12T21:46:29.996 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/vxlan/vxlan.ko 2024-05-12T21:46:29.996 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/xen-netback 2024-05-12T21:46:29.996 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/xen-netback/xen-netback.ko 2024-05-12T21:46:29.996 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/net/xen-netfront.ko 2024-05-12T21:46:29.996 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ntb 2024-05-12T21:46:29.996 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ntb/ntb.ko 2024-05-12T21:46:29.996 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/nvme 2024-05-12T21:46:29.996 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/nvme/host 2024-05-12T21:46:29.996 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/nvme/host/nvme-core.ko 2024-05-12T21:46:29.996 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/nvme/host/nvme.ko 2024-05-12T21:46:29.996 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/parport 2024-05-12T21:46:29.997 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/parport/parport.ko 2024-05-12T21:46:29.997 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/parport/parport_cs.ko 2024-05-12T21:46:29.997 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/parport/parport_pc.ko 2024-05-12T21:46:29.997 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/parport/parport_serial.ko 2024-05-12T21:46:29.997 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/pci 2024-05-12T21:46:29.997 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/pci/hotplug 2024-05-12T21:46:29.997 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-05-12T21:46:29.997 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/pci/pcie 2024-05-12T21:46:29.997 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/pci/pcie/aer_inject.ko 2024-05-12T21:46:29.997 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/pci/xen-pcifront.ko 2024-05-12T21:46:29.997 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/pcmcia 2024-05-12T21:46:29.997 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/pcmcia/i82092.ko 2024-05-12T21:46:29.997 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/pcmcia/pd6729.ko 2024-05-12T21:46:29.997 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/pcmcia/yenta_socket.ko 2024-05-12T21:46:29.998 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/pinctrl 2024-05-12T21:46:29.998 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/pinctrl/intel 2024-05-12T21:46:29.998 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-05-12T21:46:29.998 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/platform 2024-05-12T21:46:29.998 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/platform/x86 2024-05-12T21:46:29.998 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/platform/x86/acer-wmi.ko 2024-05-12T21:46:29.998 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/platform/x86/acerhdf.ko 2024-05-12T21:46:29.998 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/platform/x86/apple-gmux.ko 2024-05-12T21:46:29.998 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/platform/x86/asus-laptop.ko 2024-05-12T21:46:29.998 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-05-12T21:46:29.998 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/platform/x86/asus-wmi.ko 2024-05-12T21:46:29.998 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/platform/x86/classmate-laptop.ko 2024-05-12T21:46:29.998 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-05-12T21:46:29.998 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-05-12T21:46:29.999 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-05-12T21:46:29.999 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-05-12T21:46:29.999 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/platform/x86/hdaps.ko 2024-05-12T21:46:29.999 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/platform/x86/intel 2024-05-12T21:46:29.999 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-05-12T21:46:29.999 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/platform/x86/intel_ips.ko 2024-05-12T21:46:29.999 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/platform/x86/msi-wmi.ko 2024-05-12T21:46:29.999 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/platform/x86/mxm-wmi.ko 2024-05-12T21:46:29.999 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-05-12T21:46:29.999 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/platform/x86/samsung-laptop.ko 2024-05-12T21:46:29.999 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/platform/x86/samsung-q10.ko 2024-05-12T21:46:29.999 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/platform/x86/topstar-laptop.ko 2024-05-12T21:46:29.999 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-05-12T21:46:29.999 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/platform/x86/toshiba_haps.ko 2024-05-12T21:46:29.999 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/platform/x86/wmi-bmof.ko 2024-05-12T21:46:30.000 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/platform/x86/wmi.ko 2024-05-12T21:46:30.000 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/powercap 2024-05-12T21:46:30.000 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/powercap/intel_rapl_common.ko 2024-05-12T21:46:30.000 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/powercap/intel_rapl_msr.ko 2024-05-12T21:46:30.000 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/pps 2024-05-12T21:46:30.000 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/pps/clients 2024-05-12T21:46:30.000 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/pps/clients/pps-gpio.ko 2024-05-12T21:46:30.000 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/pps/clients/pps-ldisc.ko 2024-05-12T21:46:30.000 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/pps/clients/pps_parport.ko 2024-05-12T21:46:30.000 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/pps/pps_core.ko 2024-05-12T21:46:30.000 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ptp 2024-05-12T21:46:30.000 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ptp/ptp.ko 2024-05-12T21:46:30.000 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ptp/ptp_kvm.ko 2024-05-12T21:46:30.001 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ras 2024-05-12T21:46:30.001 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ras/amd 2024-05-12T21:46:30.001 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ras/amd/atl 2024-05-12T21:46:30.001 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ras/amd/atl/amd_atl.ko 2024-05-12T21:46:30.001 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ras/amd/fmpm.ko 2024-05-12T21:46:30.001 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/rtc 2024-05-12T21:46:30.001 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/rtc/rtc-bq32k.ko 2024-05-12T21:46:30.001 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/rtc/rtc-ds1286.ko 2024-05-12T21:46:30.001 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/rtc/rtc-ds1307.ko 2024-05-12T21:46:30.001 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/rtc/rtc-ds1374.ko 2024-05-12T21:46:30.001 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/rtc/rtc-ds1511.ko 2024-05-12T21:46:30.001 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/rtc/rtc-ds1553.ko 2024-05-12T21:46:30.001 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/rtc/rtc-ds1672.ko 2024-05-12T21:46:30.001 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/rtc/rtc-ds1742.ko 2024-05-12T21:46:30.001 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/rtc/rtc-ds2404.ko 2024-05-12T21:46:30.002 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/rtc/rtc-ds3232.ko 2024-05-12T21:46:30.002 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/rtc/rtc-em3027.ko 2024-05-12T21:46:30.002 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/rtc/rtc-fm3130.ko 2024-05-12T21:46:30.002 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/rtc/rtc-isl12022.ko 2024-05-12T21:46:30.002 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/rtc/rtc-isl1208.ko 2024-05-12T21:46:30.002 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/rtc/rtc-m41t80.ko 2024-05-12T21:46:30.002 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/rtc/rtc-m48t35.ko 2024-05-12T21:46:30.002 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/rtc/rtc-m48t59.ko 2024-05-12T21:46:30.002 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/rtc/rtc-max6900.ko 2024-05-12T21:46:30.002 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/rtc/rtc-msm6242.ko 2024-05-12T21:46:30.002 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/rtc/rtc-pcf2127.ko 2024-05-12T21:46:30.002 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/rtc/rtc-pcf85063.ko 2024-05-12T21:46:30.002 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/rtc/rtc-pcf8523.ko 2024-05-12T21:46:30.002 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/rtc/rtc-pcf8563.ko 2024-05-12T21:46:30.003 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/rtc/rtc-pcf8583.ko 2024-05-12T21:46:30.003 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/rtc/rtc-rp5c01.ko 2024-05-12T21:46:30.003 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/rtc/rtc-rs5c372.ko 2024-05-12T21:46:30.003 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-05-12T21:46:30.003 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/rtc/rtc-rx8025.ko 2024-05-12T21:46:30.003 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/rtc/rtc-rx8581.ko 2024-05-12T21:46:30.003 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-05-12T21:46:30.003 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/rtc/rtc-x1205.ko 2024-05-12T21:46:30.003 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi 2024-05-12T21:46:30.003 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/3w-9xxx.ko 2024-05-12T21:46:30.003 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/3w-sas.ko 2024-05-12T21:46:30.003 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/3w-xxxx.ko 2024-05-12T21:46:30.003 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/BusLogic.ko 2024-05-12T21:46:30.003 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/a100u2w.ko 2024-05-12T21:46:30.003 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/aacraid 2024-05-12T21:46:30.003 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/aacraid/aacraid.ko 2024-05-12T21:46:30.004 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/advansys.ko 2024-05-12T21:46:30.004 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/aic7xxx 2024-05-12T21:46:30.004 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-05-12T21:46:30.004 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-05-12T21:46:30.004 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/am53c974.ko 2024-05-12T21:46:30.004 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/arcmsr 2024-05-12T21:46:30.004 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-05-12T21:46:30.004 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/atp870u.ko 2024-05-12T21:46:30.004 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/be2iscsi 2024-05-12T21:46:30.004 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-05-12T21:46:30.004 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/bfa 2024-05-12T21:46:30.004 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/bfa/bfa.ko 2024-05-12T21:46:30.004 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/bnx2fc 2024-05-12T21:46:30.004 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-05-12T21:46:30.004 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/bnx2i 2024-05-12T21:46:30.004 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-05-12T21:46:30.005 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/ch.ko 2024-05-12T21:46:30.005 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/csiostor 2024-05-12T21:46:30.005 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/csiostor/csiostor.ko 2024-05-12T21:46:30.005 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/cxgbi 2024-05-12T21:46:30.005 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/cxgbi/cxgb3i 2024-05-12T21:46:30.005 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-05-12T21:46:30.005 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/cxgbi/cxgb4i 2024-05-12T21:46:30.005 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-05-12T21:46:30.005 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-05-12T21:46:30.005 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/dc395x.ko 2024-05-12T21:46:30.005 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/device_handler 2024-05-12T21:46:30.006 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-05-12T21:46:30.006 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-05-12T21:46:30.006 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-05-12T21:46:30.006 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-05-12T21:46:30.006 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/dmx3191d.ko 2024-05-12T21:46:30.006 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/esas2r 2024-05-12T21:46:30.006 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/esas2r/esas2r.ko 2024-05-12T21:46:30.006 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/esp_scsi.ko 2024-05-12T21:46:30.006 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/fcoe 2024-05-12T21:46:30.006 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/fcoe/fcoe.ko 2024-05-12T21:46:30.006 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-05-12T21:46:30.006 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/fnic 2024-05-12T21:46:30.006 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/fnic/fnic.ko 2024-05-12T21:46:30.007 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/hpsa.ko 2024-05-12T21:46:30.007 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/hptiop.ko 2024-05-12T21:46:30.007 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/hv_storvsc.ko 2024-05-12T21:46:30.007 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/initio.ko 2024-05-12T21:46:30.007 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/ipr.ko 2024-05-12T21:46:30.007 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/ips.ko 2024-05-12T21:46:30.007 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/isci 2024-05-12T21:46:30.007 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/isci/isci.ko 2024-05-12T21:46:30.007 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-05-12T21:46:30.007 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/iscsi_tcp.ko 2024-05-12T21:46:30.007 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/libfc 2024-05-12T21:46:30.007 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/libfc/libfc.ko 2024-05-12T21:46:30.007 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/libiscsi.ko 2024-05-12T21:46:30.008 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/libiscsi_tcp.ko 2024-05-12T21:46:30.008 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/libsas 2024-05-12T21:46:30.008 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/libsas/libsas.ko 2024-05-12T21:46:30.008 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/lpfc 2024-05-12T21:46:30.008 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/lpfc/lpfc.ko 2024-05-12T21:46:30.008 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/megaraid 2024-05-12T21:46:30.008 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/megaraid.ko 2024-05-12T21:46:30.008 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-05-12T21:46:30.008 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-05-12T21:46:30.008 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-05-12T21:46:30.008 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/mpt3sas 2024-05-12T21:46:30.008 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-05-12T21:46:30.008 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/mvsas 2024-05-12T21:46:30.009 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/mvsas/mvsas.ko 2024-05-12T21:46:30.009 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/mvumi.ko 2024-05-12T21:46:30.009 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/pm8001 2024-05-12T21:46:30.009 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-05-12T21:46:30.009 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/pmcraid.ko 2024-05-12T21:46:30.009 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/qla1280.ko 2024-05-12T21:46:30.009 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/qla2xxx 2024-05-12T21:46:30.009 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-05-12T21:46:30.009 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-05-12T21:46:30.009 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/qla4xxx 2024-05-12T21:46:30.009 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-05-12T21:46:30.009 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/raid_class.ko 2024-05-12T21:46:30.009 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/scsi_debug.ko 2024-05-12T21:46:30.010 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/scsi_transport_fc.ko 2024-05-12T21:46:30.010 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-05-12T21:46:30.010 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/scsi_transport_sas.ko 2024-05-12T21:46:30.010 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/scsi_transport_spi.ko 2024-05-12T21:46:30.010 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/scsi_transport_srp.ko 2024-05-12T21:46:30.010 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/ses.ko 2024-05-12T21:46:30.010 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/st.ko 2024-05-12T21:46:30.010 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/stex.ko 2024-05-12T21:46:30.010 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/sym53c8xx_2 2024-05-12T21:46:30.010 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-05-12T21:46:30.010 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/virtio_scsi.ko 2024-05-12T21:46:30.010 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/vmw_pvscsi.ko 2024-05-12T21:46:30.010 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/scsi/wd719x.ko 2024-05-12T21:46:30.011 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ssb 2024-05-12T21:46:30.011 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ssb/ssb.ko 2024-05-12T21:46:30.011 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/target 2024-05-12T21:46:30.011 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/target/iscsi 2024-05-12T21:46:30.011 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-05-12T21:46:30.011 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/target/loopback 2024-05-12T21:46:30.011 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/target/loopback/tcm_loop.ko 2024-05-12T21:46:30.011 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/target/target_core_file.ko 2024-05-12T21:46:30.011 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/target/target_core_iblock.ko 2024-05-12T21:46:30.011 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/target/target_core_mod.ko 2024-05-12T21:46:30.011 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/target/target_core_pscsi.ko 2024-05-12T21:46:30.011 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/target/target_core_user.ko 2024-05-12T21:46:30.011 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/target/tcm_fc 2024-05-12T21:46:30.012 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-05-12T21:46:30.012 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/thermal 2024-05-12T21:46:30.012 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/thermal/intel 2024-05-12T21:46:30.012 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/thermal/intel/int340x_thermal 2024-05-12T21:46:30.012 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-05-12T21:46:30.012 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-05-12T21:46:30.012 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-05-12T21:46:30.012 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-05-12T21:46:30.012 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-05-12T21:46:30.012 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-05-12T21:46:30.013 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-05-12T21:46:30.013 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-05-12T21:46:30.013 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-05-12T21:46:30.013 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-05-12T21:46:30.013 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-05-12T21:46:30.013 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-05-12T21:46:30.013 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-05-12T21:46:30.013 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-05-12T21:46:30.013 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-05-12T21:46:30.013 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-05-12T21:46:30.013 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-05-12T21:46:30.014 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-05-12T21:46:30.014 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/tty 2024-05-12T21:46:30.014 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/tty/ipwireless 2024-05-12T21:46:30.014 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-05-12T21:46:30.014 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/tty/n_gsm.ko 2024-05-12T21:46:30.014 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/tty/n_hdlc.ko 2024-05-12T21:46:30.014 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/tty/nozomi.ko 2024-05-12T21:46:30.014 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/tty/serial 2024-05-12T21:46:30.014 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/tty/serial/8250 2024-05-12T21:46:30.014 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-05-12T21:46:30.014 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/tty/serial/arc_uart.ko 2024-05-12T21:46:30.014 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/tty/serial/jsm 2024-05-12T21:46:30.015 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/tty/serial/jsm/jsm.ko 2024-05-12T21:46:30.015 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ufs 2024-05-12T21:46:30.015 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ufs/core 2024-05-12T21:46:30.015 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ufs/core/ufshcd-core.ko 2024-05-12T21:46:30.015 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ufs/host 2024-05-12T21:46:30.015 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-05-12T21:46:30.015 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/uio 2024-05-12T21:46:30.015 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/uio/uio.ko 2024-05-12T21:46:30.015 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/uio/uio_aec.ko 2024-05-12T21:46:30.015 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/uio/uio_cif.ko 2024-05-12T21:46:30.015 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/uio/uio_pci_generic.ko 2024-05-12T21:46:30.015 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/uio/uio_sercos3.ko 2024-05-12T21:46:30.015 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb 2024-05-12T21:46:30.016 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/class 2024-05-12T21:46:30.016 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/class/cdc-acm.ko 2024-05-12T21:46:30.016 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/class/cdc-wdm.ko 2024-05-12T21:46:30.016 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/class/usblp.ko 2024-05-12T21:46:30.016 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/class/usbtmc.ko 2024-05-12T21:46:30.016 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/host 2024-05-12T21:46:30.016 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/host/sl811-hcd.ko 2024-05-12T21:46:30.016 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/image 2024-05-12T21:46:30.016 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/image/mdc800.ko 2024-05-12T21:46:30.016 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/image/microtek.ko 2024-05-12T21:46:30.016 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/misc 2024-05-12T21:46:30.016 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/misc/adutux.ko 2024-05-12T21:46:30.016 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/misc/appledisplay.ko 2024-05-12T21:46:30.017 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/misc/emi26.ko 2024-05-12T21:46:30.017 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/misc/emi62.ko 2024-05-12T21:46:30.017 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/misc/ezusb.ko 2024-05-12T21:46:30.017 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/misc/idmouse.ko 2024-05-12T21:46:30.017 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/misc/iowarrior.ko 2024-05-12T21:46:30.017 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/misc/isight_firmware.ko 2024-05-12T21:46:30.017 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/misc/ldusb.ko 2024-05-12T21:46:30.017 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/misc/legousbtower.ko 2024-05-12T21:46:30.017 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/misc/sisusbvga 2024-05-12T21:46:30.017 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-05-12T21:46:30.017 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/misc/trancevibrator.ko 2024-05-12T21:46:30.017 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/misc/usb3503.ko 2024-05-12T21:46:30.018 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/misc/usblcd.ko 2024-05-12T21:46:30.018 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/misc/usbsevseg.ko 2024-05-12T21:46:30.018 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/misc/uss720.ko 2024-05-12T21:46:30.018 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/misc/yurex.ko 2024-05-12T21:46:30.018 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/phy 2024-05-12T21:46:30.018 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/phy/phy-generic.ko 2024-05-12T21:46:30.018 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/serial 2024-05-12T21:46:30.018 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/serial/aircable.ko 2024-05-12T21:46:30.018 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/serial/ark3116.ko 2024-05-12T21:46:30.018 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/serial/belkin_sa.ko 2024-05-12T21:46:30.018 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/serial/ch341.ko 2024-05-12T21:46:30.019 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/serial/cp210x.ko 2024-05-12T21:46:30.019 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/serial/cyberjack.ko 2024-05-12T21:46:30.019 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/serial/cypress_m8.ko 2024-05-12T21:46:30.019 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/serial/digi_acceleport.ko 2024-05-12T21:46:30.019 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/serial/empeg.ko 2024-05-12T21:46:30.019 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/serial/ftdi_sio.ko 2024-05-12T21:46:30.019 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/serial/garmin_gps.ko 2024-05-12T21:46:30.019 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/serial/io_edgeport.ko 2024-05-12T21:46:30.019 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/serial/io_ti.ko 2024-05-12T21:46:30.019 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/serial/ipaq.ko 2024-05-12T21:46:30.019 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/serial/ipw.ko 2024-05-12T21:46:30.019 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/serial/ir-usb.ko 2024-05-12T21:46:30.020 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-05-12T21:46:30.020 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/serial/keyspan.ko 2024-05-12T21:46:30.020 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/serial/keyspan_pda.ko 2024-05-12T21:46:30.020 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/serial/kl5kusb105.ko 2024-05-12T21:46:30.020 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/serial/kobil_sct.ko 2024-05-12T21:46:30.020 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/serial/mct_u232.ko 2024-05-12T21:46:30.020 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/serial/mos7720.ko 2024-05-12T21:46:30.020 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/serial/mos7840.ko 2024-05-12T21:46:30.020 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/serial/navman.ko 2024-05-12T21:46:30.020 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/serial/omninet.ko 2024-05-12T21:46:30.020 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/serial/opticon.ko 2024-05-12T21:46:30.020 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/serial/option.ko 2024-05-12T21:46:30.020 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/serial/oti6858.ko 2024-05-12T21:46:30.021 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/serial/pl2303.ko 2024-05-12T21:46:30.021 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/serial/qcaux.ko 2024-05-12T21:46:30.021 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/serial/qcserial.ko 2024-05-12T21:46:30.021 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/serial/quatech2.ko 2024-05-12T21:46:30.021 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/serial/safe_serial.ko 2024-05-12T21:46:30.021 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/serial/sierra.ko 2024-05-12T21:46:30.022 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/serial/spcp8x5.ko 2024-05-12T21:46:30.022 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/serial/ssu100.ko 2024-05-12T21:46:30.022 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/serial/symbolserial.ko 2024-05-12T21:46:30.022 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-05-12T21:46:30.022 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-05-12T21:46:30.022 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/serial/usb_debug.ko 2024-05-12T21:46:30.022 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/serial/usb_wwan.ko 2024-05-12T21:46:30.022 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/serial/visor.ko 2024-05-12T21:46:30.022 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/serial/whiteheat.ko 2024-05-12T21:46:30.023 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/serial/xsens_mt.ko 2024-05-12T21:46:30.023 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/storage 2024-05-12T21:46:30.023 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/storage/uas.ko 2024-05-12T21:46:30.023 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/storage/ums-alauda.ko 2024-05-12T21:46:30.023 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/storage/ums-cypress.ko 2024-05-12T21:46:30.023 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/storage/ums-datafab.ko 2024-05-12T21:46:30.023 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-05-12T21:46:30.023 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/storage/ums-freecom.ko 2024-05-12T21:46:30.023 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/storage/ums-isd200.ko 2024-05-12T21:46:30.024 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-05-12T21:46:30.024 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/storage/ums-karma.ko 2024-05-12T21:46:30.024 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/storage/ums-onetouch.ko 2024-05-12T21:46:30.024 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/storage/ums-realtek.ko 2024-05-12T21:46:30.024 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/storage/ums-sddr09.ko 2024-05-12T21:46:30.024 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/storage/ums-sddr55.ko 2024-05-12T21:46:30.024 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/storage/ums-usbat.ko 2024-05-12T21:46:30.025 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/storage/usb-storage.ko 2024-05-12T21:46:30.025 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/usbip 2024-05-12T21:46:30.025 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/usbip/usbip-core.ko 2024-05-12T21:46:30.025 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/usbip/usbip-host.ko 2024-05-12T21:46:30.025 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-05-12T21:46:30.025 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/vfio 2024-05-12T21:46:30.025 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/vfio/pci 2024-05-12T21:46:30.025 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-05-12T21:46:30.025 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/vfio/pci/vfio-pci.ko 2024-05-12T21:46:30.025 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/vfio/vfio.ko 2024-05-12T21:46:30.025 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-05-12T21:46:30.025 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/vhost 2024-05-12T21:46:30.025 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/vhost/vhost.ko 2024-05-12T21:46:30.025 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/vhost/vhost_iotlb.ko 2024-05-12T21:46:30.025 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/vhost/vhost_net.ko 2024-05-12T21:46:30.026 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/vhost/vhost_scsi.ko 2024-05-12T21:46:30.026 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/video 2024-05-12T21:46:30.026 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/video/backlight 2024-05-12T21:46:30.026 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/video/backlight/apple_bl.ko 2024-05-12T21:46:30.026 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/video/backlight/lcd.ko 2024-05-12T21:46:30.026 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/video/backlight/platform_lcd.ko 2024-05-12T21:46:30.026 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/virtio 2024-05-12T21:46:30.026 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/virtio/virtio.ko 2024-05-12T21:46:30.026 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/virtio/virtio_balloon.ko 2024-05-12T21:46:30.026 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/virtio/virtio_mmio.ko 2024-05-12T21:46:30.026 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/virtio/virtio_pci.ko 2024-05-12T21:46:30.026 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-05-12T21:46:30.026 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-05-12T21:46:30.026 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/virtio/virtio_ring.ko 2024-05-12T21:46:30.027 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/w1 2024-05-12T21:46:30.027 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/w1/masters 2024-05-12T21:46:30.027 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/w1/masters/ds2482.ko 2024-05-12T21:46:30.027 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/w1/masters/ds2490.ko 2024-05-12T21:46:30.027 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/w1/slaves 2024-05-12T21:46:30.027 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-05-12T21:46:30.027 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-05-12T21:46:30.027 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-05-12T21:46:30.027 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-05-12T21:46:30.027 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-05-12T21:46:30.027 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-05-12T21:46:30.027 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-05-12T21:46:30.027 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-05-12T21:46:30.027 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-05-12T21:46:30.027 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/w1/slaves/w1_smem.ko 2024-05-12T21:46:30.027 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/w1/slaves/w1_therm.ko 2024-05-12T21:46:30.028 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/w1/wire.ko 2024-05-12T21:46:30.028 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/watchdog 2024-05-12T21:46:30.028 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/watchdog/alim1535_wdt.ko 2024-05-12T21:46:30.028 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/watchdog/alim7101_wdt.ko 2024-05-12T21:46:30.028 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/watchdog/f71808e_wdt.ko 2024-05-12T21:46:30.028 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/watchdog/hpwdt.ko 2024-05-12T21:46:30.028 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/watchdog/i6300esb.ko 2024-05-12T21:46:30.028 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-05-12T21:46:30.028 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/watchdog/iTCO_wdt.ko 2024-05-12T21:46:30.028 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/watchdog/ib700wdt.ko 2024-05-12T21:46:30.028 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/watchdog/ibmasr.ko 2024-05-12T21:46:30.028 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-05-12T21:46:30.028 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/watchdog/it8712f_wdt.ko 2024-05-12T21:46:30.028 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/watchdog/it87_wdt.ko 2024-05-12T21:46:30.029 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/watchdog/machzwd.ko 2024-05-12T21:46:30.029 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/watchdog/nv_tco.ko 2024-05-12T21:46:30.029 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/watchdog/pcwd_pci.ko 2024-05-12T21:46:30.029 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/watchdog/pcwd_usb.ko 2024-05-12T21:46:30.029 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-05-12T21:46:30.029 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/watchdog/sch311x_wdt.ko 2024-05-12T21:46:30.029 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/watchdog/softdog.ko 2024-05-12T21:46:30.029 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/watchdog/sp5100_tco.ko 2024-05-12T21:46:30.029 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/watchdog/via_wdt.ko 2024-05-12T21:46:30.029 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-05-12T21:46:30.029 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/watchdog/w83877f_wdt.ko 2024-05-12T21:46:30.029 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/watchdog/w83977f_wdt.ko 2024-05-12T21:46:30.029 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/watchdog/wdt_pci.ko 2024-05-12T21:46:30.029 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/watchdog/xen_wdt.ko 2024-05-12T21:46:30.030 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/xen 2024-05-12T21:46:30.030 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/xen/xen-acpi-processor.ko 2024-05-12T21:46:30.030 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/xen/xen-evtchn.ko 2024-05-12T21:46:30.030 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/xen/xen-gntalloc.ko 2024-05-12T21:46:30.030 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/xen/xen-gntdev.ko 2024-05-12T21:46:30.030 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/xen/xen-pciback 2024-05-12T21:46:30.030 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-05-12T21:46:30.030 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/xen/xen-privcmd.ko 2024-05-12T21:46:30.030 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/xen/xenfs 2024-05-12T21:46:30.030 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/drivers/xen/xenfs/xenfs.ko 2024-05-12T21:46:30.030 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs 2024-05-12T21:46:30.031 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/9p 2024-05-12T21:46:30.031 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/9p/9p.ko 2024-05-12T21:46:30.031 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/affs 2024-05-12T21:46:30.031 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/affs/affs.ko 2024-05-12T21:46:30.031 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/befs 2024-05-12T21:46:30.031 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/befs/befs.ko 2024-05-12T21:46:30.031 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/binfmt_misc.ko 2024-05-12T21:46:30.031 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/btrfs 2024-05-12T21:46:30.031 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/btrfs/btrfs.ko 2024-05-12T21:46:30.031 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/cachefiles 2024-05-12T21:46:30.031 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/cachefiles/cachefiles.ko 2024-05-12T21:46:30.031 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/ceph 2024-05-12T21:46:30.031 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/ceph/ceph.ko 2024-05-12T21:46:30.031 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/coda 2024-05-12T21:46:30.031 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/coda/coda.ko 2024-05-12T21:46:30.032 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/cramfs 2024-05-12T21:46:30.032 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/cramfs/cramfs.ko 2024-05-12T21:46:30.032 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/dlm 2024-05-12T21:46:30.032 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/dlm/dlm.ko 2024-05-12T21:46:30.032 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/ecryptfs 2024-05-12T21:46:30.032 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/ecryptfs/ecryptfs.ko 2024-05-12T21:46:30.032 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/f2fs 2024-05-12T21:46:30.032 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/f2fs/f2fs.ko 2024-05-12T21:46:30.032 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/fat 2024-05-12T21:46:30.032 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/fat/fat.ko 2024-05-12T21:46:30.032 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/fat/msdos.ko 2024-05-12T21:46:30.032 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/fat/vfat.ko 2024-05-12T21:46:30.032 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/fuse 2024-05-12T21:46:30.033 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/fuse/cuse.ko 2024-05-12T21:46:30.033 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/fuse/fuse.ko 2024-05-12T21:46:30.033 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/gfs2 2024-05-12T21:46:30.033 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/gfs2/gfs2.ko 2024-05-12T21:46:30.033 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/hfs 2024-05-12T21:46:30.033 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/hfs/hfs.ko 2024-05-12T21:46:30.033 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/hfsplus 2024-05-12T21:46:30.033 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/hfsplus/hfsplus.ko 2024-05-12T21:46:30.033 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/isofs 2024-05-12T21:46:30.033 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/isofs/isofs.ko 2024-05-12T21:46:30.033 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/jfs 2024-05-12T21:46:30.035 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/jfs/jfs.ko 2024-05-12T21:46:30.035 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/lockd 2024-05-12T21:46:30.035 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/lockd/lockd.ko 2024-05-12T21:46:30.035 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/minix 2024-05-12T21:46:30.035 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/minix/minix.ko 2024-05-12T21:46:30.035 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/netfs 2024-05-12T21:46:30.035 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/netfs/netfs.ko 2024-05-12T21:46:30.035 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nfs 2024-05-12T21:46:30.035 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nfs/blocklayout 2024-05-12T21:46:30.035 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-05-12T21:46:30.035 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nfs/filelayout 2024-05-12T21:46:30.036 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-05-12T21:46:30.036 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nfs/flexfilelayout 2024-05-12T21:46:30.036 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-05-12T21:46:30.036 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nfs/nfs.ko 2024-05-12T21:46:30.036 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nfs/nfsv3.ko 2024-05-12T21:46:30.036 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nfs/nfsv4.ko 2024-05-12T21:46:30.037 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nfs_common 2024-05-12T21:46:30.037 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nfs_common/grace.ko 2024-05-12T21:46:30.037 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nfs_common/nfs_acl.ko 2024-05-12T21:46:30.037 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nfsd 2024-05-12T21:46:30.037 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nfsd/nfsd.ko 2024-05-12T21:46:30.037 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nilfs2 2024-05-12T21:46:30.037 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nilfs2/nilfs2.ko 2024-05-12T21:46:30.037 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nls 2024-05-12T21:46:30.037 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nls/mac-celtic.ko 2024-05-12T21:46:30.037 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nls/mac-centeuro.ko 2024-05-12T21:46:30.037 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nls/mac-croatian.ko 2024-05-12T21:46:30.037 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nls/mac-cyrillic.ko 2024-05-12T21:46:30.037 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nls/mac-gaelic.ko 2024-05-12T21:46:30.037 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nls/mac-greek.ko 2024-05-12T21:46:30.038 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nls/mac-iceland.ko 2024-05-12T21:46:30.038 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nls/mac-inuit.ko 2024-05-12T21:46:30.038 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nls/mac-roman.ko 2024-05-12T21:46:30.038 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nls/mac-romanian.ko 2024-05-12T21:46:30.038 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nls/mac-turkish.ko 2024-05-12T21:46:30.038 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nls/nls_cp1250.ko 2024-05-12T21:46:30.038 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nls/nls_cp1251.ko 2024-05-12T21:46:30.038 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nls/nls_cp1255.ko 2024-05-12T21:46:30.038 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nls/nls_cp737.ko 2024-05-12T21:46:30.039 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nls/nls_cp775.ko 2024-05-12T21:46:30.039 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nls/nls_cp850.ko 2024-05-12T21:46:30.039 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nls/nls_cp852.ko 2024-05-12T21:46:30.039 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nls/nls_cp855.ko 2024-05-12T21:46:30.039 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nls/nls_cp857.ko 2024-05-12T21:46:30.039 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nls/nls_cp860.ko 2024-05-12T21:46:30.039 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nls/nls_cp861.ko 2024-05-12T21:46:30.039 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nls/nls_cp862.ko 2024-05-12T21:46:30.039 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nls/nls_cp863.ko 2024-05-12T21:46:30.039 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nls/nls_cp864.ko 2024-05-12T21:46:30.039 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nls/nls_cp865.ko 2024-05-12T21:46:30.039 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nls/nls_cp866.ko 2024-05-12T21:46:30.039 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nls/nls_cp869.ko 2024-05-12T21:46:30.040 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nls/nls_cp874.ko 2024-05-12T21:46:30.040 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nls/nls_cp932.ko 2024-05-12T21:46:30.040 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nls/nls_cp936.ko 2024-05-12T21:46:30.040 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nls/nls_cp949.ko 2024-05-12T21:46:30.040 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nls/nls_cp950.ko 2024-05-12T21:46:30.040 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nls/nls_euc-jp.ko 2024-05-12T21:46:30.040 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nls/nls_iso8859-1.ko 2024-05-12T21:46:30.040 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nls/nls_iso8859-13.ko 2024-05-12T21:46:30.040 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nls/nls_iso8859-14.ko 2024-05-12T21:46:30.040 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nls/nls_iso8859-15.ko 2024-05-12T21:46:30.040 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nls/nls_iso8859-2.ko 2024-05-12T21:46:30.041 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nls/nls_iso8859-3.ko 2024-05-12T21:46:30.041 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nls/nls_iso8859-4.ko 2024-05-12T21:46:30.041 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nls/nls_iso8859-5.ko 2024-05-12T21:46:30.041 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nls/nls_iso8859-6.ko 2024-05-12T21:46:30.041 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nls/nls_iso8859-7.ko 2024-05-12T21:46:30.041 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nls/nls_iso8859-9.ko 2024-05-12T21:46:30.041 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nls/nls_koi8-r.ko 2024-05-12T21:46:30.041 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nls/nls_koi8-ru.ko 2024-05-12T21:46:30.041 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nls/nls_koi8-u.ko 2024-05-12T21:46:30.041 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nls/nls_ucs2_utils.ko 2024-05-12T21:46:30.041 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/nls/nls_utf8.ko 2024-05-12T21:46:30.041 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/ocfs2 2024-05-12T21:46:30.041 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/ocfs2/cluster 2024-05-12T21:46:30.042 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-05-12T21:46:30.042 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/ocfs2/dlm 2024-05-12T21:46:30.042 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-05-12T21:46:30.042 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/ocfs2/dlmfs 2024-05-12T21:46:30.042 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-05-12T21:46:30.042 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/ocfs2/ocfs2.ko 2024-05-12T21:46:30.042 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-05-12T21:46:30.042 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-05-12T21:46:30.042 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-05-12T21:46:30.042 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/overlayfs 2024-05-12T21:46:30.042 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/overlayfs/overlay.ko 2024-05-12T21:46:30.042 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/pstore 2024-05-12T21:46:30.043 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/pstore/ramoops.ko 2024-05-12T21:46:30.043 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/reiserfs 2024-05-12T21:46:30.043 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/reiserfs/reiserfs.ko 2024-05-12T21:46:30.043 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/romfs 2024-05-12T21:46:30.043 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/romfs/romfs.ko 2024-05-12T21:46:30.043 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/smb 2024-05-12T21:46:30.043 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/smb/client 2024-05-12T21:46:30.043 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/smb/client/cifs.ko 2024-05-12T21:46:30.043 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/smb/common 2024-05-12T21:46:30.043 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/smb/common/cifs_arc4.ko 2024-05-12T21:46:30.043 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/smb/common/cifs_md4.ko 2024-05-12T21:46:30.043 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/squashfs 2024-05-12T21:46:30.044 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/squashfs/squashfs.ko 2024-05-12T21:46:30.044 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/sysv 2024-05-12T21:46:30.044 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/sysv/sysv.ko 2024-05-12T21:46:30.044 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/udf 2024-05-12T21:46:30.044 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/udf/udf.ko 2024-05-12T21:46:30.044 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/ufs 2024-05-12T21:46:30.044 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/ufs/ufs.ko 2024-05-12T21:46:30.044 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/xfs 2024-05-12T21:46:30.044 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/fs/xfs/xfs.ko 2024-05-12T21:46:30.044 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/kernel 2024-05-12T21:46:30.044 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/kernel/rcu 2024-05-12T21:46:30.044 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/kernel/rcu/rcutorture.ko 2024-05-12T21:46:30.044 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/kernel/torture.ko 2024-05-12T21:46:30.045 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/kernel/trace 2024-05-12T21:46:30.045 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-05-12T21:46:30.045 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/lib 2024-05-12T21:46:30.045 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/lib/asn1_encoder.ko 2024-05-12T21:46:30.045 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/lib/crc-itu-t.ko 2024-05-12T21:46:30.045 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/lib/crc8.ko 2024-05-12T21:46:30.045 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/lib/crypto 2024-05-12T21:46:30.045 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/lib/crypto/libarc4.ko 2024-05-12T21:46:30.045 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/lib/crypto/libdes.ko 2024-05-12T21:46:30.045 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/lib/libcrc32c.ko 2024-05-12T21:46:30.045 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/lib/lru_cache.ko 2024-05-12T21:46:30.045 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/lib/lz4 2024-05-12T21:46:30.046 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/lib/lz4/lz4_compress.ko 2024-05-12T21:46:30.046 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/lib/lz4/lz4hc_compress.ko 2024-05-12T21:46:30.046 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/lib/math 2024-05-12T21:46:30.046 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/lib/math/cordic.ko 2024-05-12T21:46:30.046 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/lib/raid6 2024-05-12T21:46:30.046 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/lib/raid6/raid6_pq.ko 2024-05-12T21:46:30.046 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/lib/reed_solomon 2024-05-12T21:46:30.046 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/lib/reed_solomon/reed_solomon.ko 2024-05-12T21:46:30.046 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/lib/ts_bm.ko 2024-05-12T21:46:30.046 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/lib/ts_fsm.ko 2024-05-12T21:46:30.046 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/lib/ts_kmp.ko 2024-05-12T21:46:30.046 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/lib/zstd 2024-05-12T21:46:30.046 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/lib/zstd/zstd_compress.ko 2024-05-12T21:46:30.047 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/mm 2024-05-12T21:46:30.047 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/mm/hwpoison-inject.ko 2024-05-12T21:46:30.047 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net 2024-05-12T21:46:30.047 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/6lowpan 2024-05-12T21:46:30.047 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/6lowpan/6lowpan.ko 2024-05-12T21:46:30.047 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/6lowpan/nhc_dest.ko 2024-05-12T21:46:30.047 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/6lowpan/nhc_fragment.ko 2024-05-12T21:46:30.047 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/6lowpan/nhc_hop.ko 2024-05-12T21:46:30.047 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/6lowpan/nhc_ipv6.ko 2024-05-12T21:46:30.047 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/6lowpan/nhc_mobility.ko 2024-05-12T21:46:30.047 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/6lowpan/nhc_routing.ko 2024-05-12T21:46:30.047 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/6lowpan/nhc_udp.ko 2024-05-12T21:46:30.048 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/802 2024-05-12T21:46:30.048 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/802/garp.ko 2024-05-12T21:46:30.048 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/802/mrp.ko 2024-05-12T21:46:30.048 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/802/p8022.ko 2024-05-12T21:46:30.048 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/802/psnap.ko 2024-05-12T21:46:30.048 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/802/stp.ko 2024-05-12T21:46:30.048 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/8021q 2024-05-12T21:46:30.048 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/8021q/8021q.ko 2024-05-12T21:46:30.048 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/9p 2024-05-12T21:46:30.048 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/9p/9pnet.ko 2024-05-12T21:46:30.048 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/9p/9pnet_fd.ko 2024-05-12T21:46:30.048 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/9p/9pnet_rdma.ko 2024-05-12T21:46:30.048 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/9p/9pnet_virtio.ko 2024-05-12T21:46:30.049 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/batman-adv 2024-05-12T21:46:30.049 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/batman-adv/batman-adv.ko 2024-05-12T21:46:30.049 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/bridge 2024-05-12T21:46:30.049 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/bridge/br_netfilter.ko 2024-05-12T21:46:30.049 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/bridge/bridge.ko 2024-05-12T21:46:30.049 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/bridge/netfilter 2024-05-12T21:46:30.049 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-05-12T21:46:30.049 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/bridge/netfilter/ebt_among.ko 2024-05-12T21:46:30.049 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/bridge/netfilter/ebt_arp.ko 2024-05-12T21:46:30.049 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-05-12T21:46:30.049 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-05-12T21:46:30.049 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/bridge/netfilter/ebt_ip.ko 2024-05-12T21:46:30.050 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-05-12T21:46:30.050 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/bridge/netfilter/ebt_limit.ko 2024-05-12T21:46:30.050 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/bridge/netfilter/ebt_log.ko 2024-05-12T21:46:30.050 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/bridge/netfilter/ebt_mark.ko 2024-05-12T21:46:30.050 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-05-12T21:46:30.050 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-05-12T21:46:30.050 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-05-12T21:46:30.050 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-05-12T21:46:30.050 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/bridge/netfilter/ebt_snat.ko 2024-05-12T21:46:30.051 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/bridge/netfilter/ebt_stp.ko 2024-05-12T21:46:30.051 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-05-12T21:46:30.051 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-05-12T21:46:30.051 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-05-12T21:46:30.051 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-05-12T21:46:30.051 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/bridge/netfilter/ebtables.ko 2024-05-12T21:46:30.051 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-05-12T21:46:30.052 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-05-12T21:46:30.052 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-05-12T21:46:30.052 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ceph 2024-05-12T21:46:30.052 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ceph/libceph.ko 2024-05-12T21:46:30.052 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/core 2024-05-12T21:46:30.052 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/core/failover.ko 2024-05-12T21:46:30.052 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/core/pktgen.ko 2024-05-12T21:46:30.052 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/dccp 2024-05-12T21:46:30.052 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/dccp/dccp.ko 2024-05-12T21:46:30.052 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/dccp/dccp_diag.ko 2024-05-12T21:46:30.053 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/dccp/dccp_ipv4.ko 2024-05-12T21:46:30.053 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/dccp/dccp_ipv6.ko 2024-05-12T21:46:30.053 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/dns_resolver 2024-05-12T21:46:30.053 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/dns_resolver/dns_resolver.ko 2024-05-12T21:46:30.053 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/dsa 2024-05-12T21:46:30.053 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/dsa/dsa_core.ko 2024-05-12T21:46:30.053 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/dsa/tag_brcm.ko 2024-05-12T21:46:30.053 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/dsa/tag_dsa.ko 2024-05-12T21:46:30.053 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/dsa/tag_none.ko 2024-05-12T21:46:30.053 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/dsa/tag_trailer.ko 2024-05-12T21:46:30.053 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ieee802154 2024-05-12T21:46:30.053 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ieee802154/6lowpan 2024-05-12T21:46:30.053 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-05-12T21:46:30.053 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ieee802154/ieee802154.ko 2024-05-12T21:46:30.053 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ieee802154/ieee802154_socket.ko 2024-05-12T21:46:30.053 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv4 2024-05-12T21:46:30.054 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv4/ah4.ko 2024-05-12T21:46:30.054 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv4/esp4.ko 2024-05-12T21:46:30.054 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv4/fou.ko 2024-05-12T21:46:30.054 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv4/gre.ko 2024-05-12T21:46:30.054 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv4/inet_diag.ko 2024-05-12T21:46:30.054 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv4/ip_gre.ko 2024-05-12T21:46:30.054 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv4/ip_tunnel.ko 2024-05-12T21:46:30.054 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv4/ip_vti.ko 2024-05-12T21:46:30.054 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv4/ipcomp.ko 2024-05-12T21:46:30.054 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv4/ipip.ko 2024-05-12T21:46:30.054 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv4/netfilter 2024-05-12T21:46:30.055 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv4/netfilter/arp_tables.ko 2024-05-12T21:46:30.055 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-05-12T21:46:30.055 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-05-12T21:46:30.055 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-05-12T21:46:30.055 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-05-12T21:46:30.055 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-05-12T21:46:30.055 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-05-12T21:46:30.055 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-05-12T21:46:30.055 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-05-12T21:46:30.055 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-05-12T21:46:30.055 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv4/netfilter/iptable_security.ko 2024-05-12T21:46:30.055 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-05-12T21:46:30.055 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-05-12T21:46:30.055 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-05-12T21:46:30.056 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-05-12T21:46:30.056 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-05-12T21:46:30.056 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-05-12T21:46:30.056 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-05-12T21:46:30.056 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-05-12T21:46:30.056 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-05-12T21:46:30.056 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-05-12T21:46:30.056 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv4/tcp_bic.ko 2024-05-12T21:46:30.056 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv4/tcp_dctcp.ko 2024-05-12T21:46:30.056 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv4/tcp_diag.ko 2024-05-12T21:46:30.056 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv4/tcp_highspeed.ko 2024-05-12T21:46:30.056 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv4/tcp_htcp.ko 2024-05-12T21:46:30.056 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv4/tcp_hybla.ko 2024-05-12T21:46:30.056 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv4/tcp_illinois.ko 2024-05-12T21:46:30.056 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv4/tcp_lp.ko 2024-05-12T21:46:30.056 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv4/tcp_scalable.ko 2024-05-12T21:46:30.057 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv4/tcp_vegas.ko 2024-05-12T21:46:30.057 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv4/tcp_veno.ko 2024-05-12T21:46:30.057 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv4/tcp_westwood.ko 2024-05-12T21:46:30.057 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv4/tcp_yeah.ko 2024-05-12T21:46:30.057 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv4/tunnel4.ko 2024-05-12T21:46:30.057 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv4/udp_diag.ko 2024-05-12T21:46:30.057 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv4/udp_tunnel.ko 2024-05-12T21:46:30.057 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv4/xfrm4_tunnel.ko 2024-05-12T21:46:30.057 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv6 2024-05-12T21:46:30.057 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv6/ah6.ko 2024-05-12T21:46:30.057 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv6/esp6.ko 2024-05-12T21:46:30.057 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv6/fou6.ko 2024-05-12T21:46:30.057 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv6/ip6_tunnel.ko 2024-05-12T21:46:30.057 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-05-12T21:46:30.057 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv6/ip6_vti.ko 2024-05-12T21:46:30.058 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv6/ipcomp6.ko 2024-05-12T21:46:30.058 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv6/netfilter 2024-05-12T21:46:30.058 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-05-12T21:46:30.058 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-05-12T21:46:30.058 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-05-12T21:46:30.058 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-05-12T21:46:30.058 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-05-12T21:46:30.058 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-05-12T21:46:30.058 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-05-12T21:46:30.058 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-05-12T21:46:30.058 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-05-12T21:46:30.058 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-05-12T21:46:30.058 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-05-12T21:46:30.059 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-05-12T21:46:30.059 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-05-12T21:46:30.059 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-05-12T21:46:30.059 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-05-12T21:46:30.059 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-05-12T21:46:30.059 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-05-12T21:46:30.059 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-05-12T21:46:30.059 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-05-12T21:46:30.059 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-05-12T21:46:30.059 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-05-12T21:46:30.059 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-05-12T21:46:30.059 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-05-12T21:46:30.059 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-05-12T21:46:30.059 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv6/sit.ko 2024-05-12T21:46:30.060 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv6/tunnel6.ko 2024-05-12T21:46:30.060 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/ipv6/xfrm6_tunnel.ko 2024-05-12T21:46:30.060 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/key 2024-05-12T21:46:30.060 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/key/af_key.ko 2024-05-12T21:46:30.060 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/l2tp 2024-05-12T21:46:30.060 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/l2tp/l2tp_core.ko 2024-05-12T21:46:30.060 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/l2tp/l2tp_debugfs.ko 2024-05-12T21:46:30.060 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/l2tp/l2tp_eth.ko 2024-05-12T21:46:30.060 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/l2tp/l2tp_ip.ko 2024-05-12T21:46:30.060 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/l2tp/l2tp_ip6.ko 2024-05-12T21:46:30.060 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/l2tp/l2tp_netlink.ko 2024-05-12T21:46:30.060 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/l2tp/l2tp_ppp.ko 2024-05-12T21:46:30.060 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/llc 2024-05-12T21:46:30.061 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/llc/llc.ko 2024-05-12T21:46:30.061 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/mac802154 2024-05-12T21:46:30.061 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/mac802154/mac802154.ko 2024-05-12T21:46:30.061 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/mpls 2024-05-12T21:46:30.061 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/mpls/mpls_gso.ko 2024-05-12T21:46:30.061 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter 2024-05-12T21:46:30.061 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/ipset 2024-05-12T21:46:30.061 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/ipset/ip_set.ko 2024-05-12T21:46:30.061 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-05-12T21:46:30.061 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-05-12T21:46:30.061 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-05-12T21:46:30.061 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-05-12T21:46:30.061 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-05-12T21:46:30.061 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-05-12T21:46:30.062 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-05-12T21:46:30.062 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-05-12T21:46:30.062 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-05-12T21:46:30.062 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-05-12T21:46:30.062 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-05-12T21:46:30.062 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-05-12T21:46:30.062 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-05-12T21:46:30.062 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-05-12T21:46:30.062 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-05-12T21:46:30.062 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/ipvs 2024-05-12T21:46:30.062 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/ipvs/ip_vs.ko 2024-05-12T21:46:30.063 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-05-12T21:46:30.063 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-05-12T21:46:30.063 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-05-12T21:46:30.063 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-05-12T21:46:30.063 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-05-12T21:46:30.063 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-05-12T21:46:30.063 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-05-12T21:46:30.063 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-05-12T21:46:30.063 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-05-12T21:46:30.063 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-05-12T21:46:30.063 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-05-12T21:46:30.063 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-05-12T21:46:30.063 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-05-12T21:46:30.063 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/nf_conncount.ko 2024-05-12T21:46:30.064 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/nf_conntrack.ko 2024-05-12T21:46:30.064 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-05-12T21:46:30.064 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-05-12T21:46:30.064 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-05-12T21:46:30.064 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/nf_conntrack_h323.ko 2024-05-12T21:46:30.064 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/nf_conntrack_irc.ko 2024-05-12T21:46:30.064 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-05-12T21:46:30.064 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-05-12T21:46:30.064 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-05-12T21:46:30.064 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/nf_conntrack_sane.ko 2024-05-12T21:46:30.064 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/nf_conntrack_sip.ko 2024-05-12T21:46:30.064 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-05-12T21:46:30.064 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-05-12T21:46:30.065 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/nf_dup_netdev.ko 2024-05-12T21:46:30.065 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/nf_flow_table.ko 2024-05-12T21:46:30.065 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/nf_flow_table_inet.ko 2024-05-12T21:46:30.065 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/nf_log_syslog.ko 2024-05-12T21:46:30.065 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/nf_nat.ko 2024-05-12T21:46:30.065 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/nf_nat_amanda.ko 2024-05-12T21:46:30.065 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/nf_nat_ftp.ko 2024-05-12T21:46:30.065 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/nf_nat_irc.ko 2024-05-12T21:46:30.065 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/nf_nat_sip.ko 2024-05-12T21:46:30.066 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/nf_nat_tftp.ko 2024-05-12T21:46:30.066 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/nf_synproxy_core.ko 2024-05-12T21:46:30.066 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/nf_tables.ko 2024-05-12T21:46:30.066 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/nfnetlink.ko 2024-05-12T21:46:30.066 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/nfnetlink_acct.ko 2024-05-12T21:46:30.066 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/nfnetlink_log.ko 2024-05-12T21:46:30.066 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/nfnetlink_osf.ko 2024-05-12T21:46:30.066 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/nfnetlink_queue.ko 2024-05-12T21:46:30.066 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/nft_chain_nat.ko 2024-05-12T21:46:30.066 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/nft_compat.ko 2024-05-12T21:46:30.066 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/nft_ct.ko 2024-05-12T21:46:30.066 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/nft_dup_netdev.ko 2024-05-12T21:46:30.067 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/nft_fib.ko 2024-05-12T21:46:30.067 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/nft_fwd_netdev.ko 2024-05-12T21:46:30.067 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/nft_hash.ko 2024-05-12T21:46:30.067 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/nft_limit.ko 2024-05-12T21:46:30.067 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/nft_log.ko 2024-05-12T21:46:30.067 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/nft_masq.ko 2024-05-12T21:46:30.067 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/nft_nat.ko 2024-05-12T21:46:30.067 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/nft_numgen.ko 2024-05-12T21:46:30.067 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/nft_queue.ko 2024-05-12T21:46:30.067 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/nft_quota.ko 2024-05-12T21:46:30.067 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/nft_redir.ko 2024-05-12T21:46:30.067 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/nft_reject.ko 2024-05-12T21:46:30.067 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/nft_reject_inet.ko 2024-05-12T21:46:30.068 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/nft_socket.ko 2024-05-12T21:46:30.068 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/nft_tunnel.ko 2024-05-12T21:46:30.068 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_AUDIT.ko 2024-05-12T21:46:30.068 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_CHECKSUM.ko 2024-05-12T21:46:30.068 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_CLASSIFY.ko 2024-05-12T21:46:30.068 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-05-12T21:46:30.068 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_CT.ko 2024-05-12T21:46:30.068 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_DSCP.ko 2024-05-12T21:46:30.068 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_HL.ko 2024-05-12T21:46:30.068 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_HMARK.ko 2024-05-12T21:46:30.068 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_IDLETIMER.ko 2024-05-12T21:46:30.068 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_LED.ko 2024-05-12T21:46:30.068 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_LOG.ko 2024-05-12T21:46:30.068 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_MASQUERADE.ko 2024-05-12T21:46:30.069 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_NETMAP.ko 2024-05-12T21:46:30.069 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_NFLOG.ko 2024-05-12T21:46:30.069 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_NFQUEUE.ko 2024-05-12T21:46:30.069 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_RATEEST.ko 2024-05-12T21:46:30.069 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_REDIRECT.ko 2024-05-12T21:46:30.069 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_SECMARK.ko 2024-05-12T21:46:30.069 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_TCPMSS.ko 2024-05-12T21:46:30.069 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-05-12T21:46:30.069 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_TEE.ko 2024-05-12T21:46:30.069 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_TPROXY.ko 2024-05-12T21:46:30.069 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_TRACE.ko 2024-05-12T21:46:30.069 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_addrtype.ko 2024-05-12T21:46:30.070 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_bpf.ko 2024-05-12T21:46:30.070 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_cgroup.ko 2024-05-12T21:46:30.070 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_cluster.ko 2024-05-12T21:46:30.070 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_comment.ko 2024-05-12T21:46:30.070 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_connbytes.ko 2024-05-12T21:46:30.070 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_connlabel.ko 2024-05-12T21:46:30.070 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_connlimit.ko 2024-05-12T21:46:30.070 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_connmark.ko 2024-05-12T21:46:30.070 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_conntrack.ko 2024-05-12T21:46:30.070 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_cpu.ko 2024-05-12T21:46:30.070 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_dccp.ko 2024-05-12T21:46:30.071 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_devgroup.ko 2024-05-12T21:46:30.071 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_dscp.ko 2024-05-12T21:46:30.071 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_ecn.ko 2024-05-12T21:46:30.071 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_esp.ko 2024-05-12T21:46:30.071 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_hashlimit.ko 2024-05-12T21:46:30.071 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_helper.ko 2024-05-12T21:46:30.071 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_hl.ko 2024-05-12T21:46:30.071 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_ipcomp.ko 2024-05-12T21:46:30.071 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_iprange.ko 2024-05-12T21:46:30.071 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_ipvs.ko 2024-05-12T21:46:30.071 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_l2tp.ko 2024-05-12T21:46:30.071 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_length.ko 2024-05-12T21:46:30.071 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_limit.ko 2024-05-12T21:46:30.072 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_mac.ko 2024-05-12T21:46:30.072 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_mark.ko 2024-05-12T21:46:30.072 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_multiport.ko 2024-05-12T21:46:30.072 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_nat.ko 2024-05-12T21:46:30.072 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_nfacct.ko 2024-05-12T21:46:30.072 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_osf.ko 2024-05-12T21:46:30.072 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_owner.ko 2024-05-12T21:46:30.072 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_physdev.ko 2024-05-12T21:46:30.072 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_pkttype.ko 2024-05-12T21:46:30.072 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_policy.ko 2024-05-12T21:46:30.072 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_quota.ko 2024-05-12T21:46:30.072 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_rateest.ko 2024-05-12T21:46:30.073 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_realm.ko 2024-05-12T21:46:30.073 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_recent.ko 2024-05-12T21:46:30.073 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_sctp.ko 2024-05-12T21:46:30.073 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_set.ko 2024-05-12T21:46:30.073 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_socket.ko 2024-05-12T21:46:30.073 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_state.ko 2024-05-12T21:46:30.073 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_statistic.ko 2024-05-12T21:46:30.073 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_string.ko 2024-05-12T21:46:30.073 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_tcpmss.ko 2024-05-12T21:46:30.073 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_time.ko 2024-05-12T21:46:30.073 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netfilter/xt_u32.ko 2024-05-12T21:46:30.073 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netlink 2024-05-12T21:46:30.073 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/netlink/netlink_diag.ko 2024-05-12T21:46:30.074 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/nsh 2024-05-12T21:46:30.074 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/nsh/nsh.ko 2024-05-12T21:46:30.074 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/openvswitch 2024-05-12T21:46:30.074 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/openvswitch/openvswitch.ko 2024-05-12T21:46:30.074 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/openvswitch/vport-geneve.ko 2024-05-12T21:46:30.074 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/openvswitch/vport-gre.ko 2024-05-12T21:46:30.074 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/openvswitch/vport-vxlan.ko 2024-05-12T21:46:30.074 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/packet 2024-05-12T21:46:30.074 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/packet/af_packet_diag.ko 2024-05-12T21:46:30.074 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/rds 2024-05-12T21:46:30.074 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/rds/rds.ko 2024-05-12T21:46:30.074 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/rds/rds_rdma.ko 2024-05-12T21:46:30.075 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/rds/rds_tcp.ko 2024-05-12T21:46:30.075 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/sched 2024-05-12T21:46:30.075 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/sched/act_csum.ko 2024-05-12T21:46:30.075 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/sched/act_gact.ko 2024-05-12T21:46:30.075 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/sched/act_mirred.ko 2024-05-12T21:46:30.075 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/sched/act_nat.ko 2024-05-12T21:46:30.075 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/sched/act_pedit.ko 2024-05-12T21:46:30.075 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/sched/act_police.ko 2024-05-12T21:46:30.075 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/sched/act_simple.ko 2024-05-12T21:46:30.075 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/sched/act_skbedit.ko 2024-05-12T21:46:30.075 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/sched/act_vlan.ko 2024-05-12T21:46:30.075 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/sched/cls_basic.ko 2024-05-12T21:46:30.075 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/sched/cls_bpf.ko 2024-05-12T21:46:30.075 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/sched/cls_flow.ko 2024-05-12T21:46:30.076 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/sched/cls_fw.ko 2024-05-12T21:46:30.076 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/sched/cls_route.ko 2024-05-12T21:46:30.076 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/sched/cls_u32.ko 2024-05-12T21:46:30.076 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/sched/em_cmp.ko 2024-05-12T21:46:30.076 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/sched/em_ipset.ko 2024-05-12T21:46:30.076 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/sched/em_meta.ko 2024-05-12T21:46:30.076 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/sched/em_nbyte.ko 2024-05-12T21:46:30.076 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/sched/em_text.ko 2024-05-12T21:46:30.076 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/sched/em_u32.ko 2024-05-12T21:46:30.076 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/sched/sch_choke.ko 2024-05-12T21:46:30.077 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/sched/sch_codel.ko 2024-05-12T21:46:30.077 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/sched/sch_drr.ko 2024-05-12T21:46:30.077 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/sched/sch_fq.ko 2024-05-12T21:46:30.077 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/sched/sch_gred.ko 2024-05-12T21:46:30.077 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/sched/sch_hfsc.ko 2024-05-12T21:46:30.077 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/sched/sch_hhf.ko 2024-05-12T21:46:30.077 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/sched/sch_htb.ko 2024-05-12T21:46:30.078 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/sched/sch_ingress.ko 2024-05-12T21:46:30.078 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/sched/sch_mqprio.ko 2024-05-12T21:46:30.078 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/sched/sch_mqprio_lib.ko 2024-05-12T21:46:30.078 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/sched/sch_multiq.ko 2024-05-12T21:46:30.078 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/sched/sch_netem.ko 2024-05-12T21:46:30.078 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/sched/sch_pie.ko 2024-05-12T21:46:30.078 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/sched/sch_plug.ko 2024-05-12T21:46:30.078 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/sched/sch_prio.ko 2024-05-12T21:46:30.078 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/sched/sch_qfq.ko 2024-05-12T21:46:30.078 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/sched/sch_red.ko 2024-05-12T21:46:30.079 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/sched/sch_sfb.ko 2024-05-12T21:46:30.079 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/sched/sch_sfq.ko 2024-05-12T21:46:30.079 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/sched/sch_tbf.ko 2024-05-12T21:46:30.079 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/sched/sch_teql.ko 2024-05-12T21:46:30.079 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/sctp 2024-05-12T21:46:30.079 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/sctp/sctp.ko 2024-05-12T21:46:30.079 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/sctp/sctp_diag.ko 2024-05-12T21:46:30.079 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/sunrpc 2024-05-12T21:46:30.079 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/sunrpc/auth_gss 2024-05-12T21:46:30.079 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-05-12T21:46:30.079 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-05-12T21:46:30.079 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/sunrpc/sunrpc.ko 2024-05-12T21:46:30.079 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/sunrpc/xprtrdma 2024-05-12T21:46:30.080 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-05-12T21:46:30.080 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/tipc 2024-05-12T21:46:30.081 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/tipc/tipc.ko 2024-05-12T21:46:30.081 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/tipc/tipc_diag.ko 2024-05-12T21:46:30.082 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/unix 2024-05-12T21:46:30.082 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/unix/unix_diag.ko 2024-05-12T21:46:30.082 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/vmw_vsock 2024-05-12T21:46:30.082 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-05-12T21:46:30.082 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-05-12T21:46:30.082 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/vmw_vsock/vsock.ko 2024-05-12T21:46:30.082 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/vmw_vsock/vsock_diag.ko 2024-05-12T21:46:30.082 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/vmw_vsock/vsock_loopback.ko 2024-05-12T21:46:30.082 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/xfrm 2024-05-12T21:46:30.082 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/net/xfrm/xfrm_ipcomp.ko 2024-05-12T21:46:30.082 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/security 2024-05-12T21:46:30.082 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/security/keys 2024-05-12T21:46:30.083 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/security/keys/encrypted-keys 2024-05-12T21:46:30.083 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-05-12T21:46:30.083 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/security/keys/trusted-keys 2024-05-12T21:46:30.083 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/kernel/security/keys/trusted-keys/trusted.ko 2024-05-12T21:46:30.083 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/modules.alias 2024-05-12T21:46:30.083 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/modules.alias.bin 2024-05-12T21:46:30.083 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/modules.builtin 2024-05-12T21:46:30.083 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/modules.builtin.alias.bin 2024-05-12T21:46:30.083 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/modules.builtin.bin 2024-05-12T21:46:30.083 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/modules.builtin.modinfo 2024-05-12T21:46:30.083 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/modules.dep 2024-05-12T21:46:30.083 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/modules.dep.bin 2024-05-12T21:46:30.083 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/modules.devname 2024-05-12T21:46:30.084 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/modules.order 2024-05-12T21:46:30.084 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/modules.softdep 2024-05-12T21:46:30.084 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/modules.symbols 2024-05-12T21:46:30.084 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/modules.symbols.bin 2024-05-12T21:46:30.084 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-gfb9a6a44107e/vmlinuz 2024-05-12T21:46:30.089 DEBUG:teuthology.task.kernel:get_image_version: 6.9.0-rc3-gfb9a6a44107e 2024-05-12T21:46:30.089 DEBUG:teuthology.orchestra.run.smithi028:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-05-12T21:46:30.113 INFO:teuthology.orchestra.run.smithi028.stdout:postinstall scriptlet (using /bin/sh): 2024-05-12T21:46:30.113 INFO:teuthology.orchestra.run.smithi028.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-05-12T21:46:30.113 INFO:teuthology.orchestra.run.smithi028.stdout: /usr/bin/kernel-install add 6.9.0-rc3-gfb9a6a44107e /lib/modules/6.9.0-rc3-gfb9a6a44107e/vmlinuz 2024-05-12T21:46:30.113 INFO:teuthology.orchestra.run.smithi028.stdout:fi 2024-05-12T21:46:30.113 INFO:teuthology.orchestra.run.smithi028.stdout:for file in vmlinuz System.map config; do 2024-05-12T21:46:30.113 INFO:teuthology.orchestra.run.smithi028.stdout: if ! cmp --silent "/lib/modules/6.9.0-rc3-gfb9a6a44107e/${file}" "/boot/${file}-6.9.0-rc3-gfb9a6a44107e"; then 2024-05-12T21:46:30.113 INFO:teuthology.orchestra.run.smithi028.stdout: cp "/lib/modules/6.9.0-rc3-gfb9a6a44107e/${file}" "/boot/${file}-6.9.0-rc3-gfb9a6a44107e" 2024-05-12T21:46:30.113 INFO:teuthology.orchestra.run.smithi028.stdout: fi 2024-05-12T21:46:30.113 INFO:teuthology.orchestra.run.smithi028.stdout:done 2024-05-12T21:46:30.114 INFO:teuthology.orchestra.run.smithi028.stdout:if [ -d "/lib/modules/6.9.0-rc3-gfb9a6a44107e/dtb" ] && \ 2024-05-12T21:46:30.114 INFO:teuthology.orchestra.run.smithi028.stdout: ! diff -rq "/lib/modules/6.9.0-rc3-gfb9a6a44107e/dtb" "/boot/dtb-6.9.0-rc3-gfb9a6a44107e" >/dev/null 2>&1; then 2024-05-12T21:46:30.114 INFO:teuthology.orchestra.run.smithi028.stdout: rm -rf "/boot/dtb-6.9.0-rc3-gfb9a6a44107e" 2024-05-12T21:46:30.114 INFO:teuthology.orchestra.run.smithi028.stdout: cp -r "/lib/modules/6.9.0-rc3-gfb9a6a44107e/dtb" "/boot/dtb-6.9.0-rc3-gfb9a6a44107e" 2024-05-12T21:46:30.114 INFO:teuthology.orchestra.run.smithi028.stdout:fi 2024-05-12T21:46:30.114 INFO:teuthology.orchestra.run.smithi028.stdout:if [ ! -e "/lib/modules/6.9.0-rc3-gfb9a6a44107e/modules.dep" ]; then 2024-05-12T21:46:30.114 INFO:teuthology.orchestra.run.smithi028.stdout: /usr/sbin/depmod 6.9.0-rc3-gfb9a6a44107e 2024-05-12T21:46:30.114 INFO:teuthology.orchestra.run.smithi028.stdout:fi 2024-05-12T21:46:30.114 INFO:teuthology.orchestra.run.smithi028.stdout:preuninstall scriptlet (using /bin/sh): 2024-05-12T21:46:30.114 INFO:teuthology.orchestra.run.smithi028.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-05-12T21:46:30.114 INFO:teuthology.orchestra.run.smithi028.stdout:kernel-install remove 6.9.0-rc3-gfb9a6a44107e 2024-05-12T21:46:30.115 INFO:teuthology.orchestra.run.smithi028.stdout:fi 2024-05-12T21:46:30.116 DEBUG:teuthology.orchestra.run.smithi028:> sudo rpm -qi grub2-tools 2024-05-12T21:46:30.190 INFO:teuthology.orchestra.run.smithi028.stdout:Name : grub2-tools 2024-05-12T21:46:30.190 INFO:teuthology.orchestra.run.smithi028.stdout:Epoch : 1 2024-05-12T21:46:30.190 INFO:teuthology.orchestra.run.smithi028.stdout:Version : 2.06 2024-05-12T21:46:30.190 INFO:teuthology.orchestra.run.smithi028.stdout:Release : 68.el9 2024-05-12T21:46:30.190 INFO:teuthology.orchestra.run.smithi028.stdout:Architecture: x86_64 2024-05-12T21:46:30.190 INFO:teuthology.orchestra.run.smithi028.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-05-12T21:46:30.190 INFO:teuthology.orchestra.run.smithi028.stdout:Group : Unspecified 2024-05-12T21:46:30.191 INFO:teuthology.orchestra.run.smithi028.stdout:Size : 8264254 2024-05-12T21:46:30.191 INFO:teuthology.orchestra.run.smithi028.stdout:License : GPLv3+ 2024-05-12T21:46:30.191 INFO:teuthology.orchestra.run.smithi028.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-05-12T21:46:30.191 INFO:teuthology.orchestra.run.smithi028.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-05-12T21:46:30.191 INFO:teuthology.orchestra.run.smithi028.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-05-12T21:46:30.191 INFO:teuthology.orchestra.run.smithi028.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-05-12T21:46:30.191 INFO:teuthology.orchestra.run.smithi028.stdout:Packager : builder@centos.org 2024-05-12T21:46:30.191 INFO:teuthology.orchestra.run.smithi028.stdout:Vendor : CentOS 2024-05-12T21:46:30.191 INFO:teuthology.orchestra.run.smithi028.stdout:URL : http://www.gnu.org/software/grub/ 2024-05-12T21:46:30.191 INFO:teuthology.orchestra.run.smithi028.stdout:Summary : Support tools for GRUB. 2024-05-12T21:46:30.192 INFO:teuthology.orchestra.run.smithi028.stdout:Description : 2024-05-12T21:46:30.192 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-12T21:46:30.192 INFO:teuthology.orchestra.run.smithi028.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-05-12T21:46:30.192 INFO:teuthology.orchestra.run.smithi028.stdout:customizable bootloader with modular architecture. It supports a rich 2024-05-12T21:46:30.192 INFO:teuthology.orchestra.run.smithi028.stdout:variety of kernel formats, file systems, computer architectures and 2024-05-12T21:46:30.192 INFO:teuthology.orchestra.run.smithi028.stdout:hardware devices. 2024-05-12T21:46:30.192 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-12T21:46:30.192 INFO:teuthology.orchestra.run.smithi028.stdout:This subpackage provides tools for support of all platforms. 2024-05-12T21:46:30.194 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-05-12T21:46:30.194 INFO:teuthology.task.kernel:Updating grub on smithi028 to boot 6.9.0-rc3-gfb9a6a44107e 2024-05-12T21:46:30.194 DEBUG:teuthology.orchestra.run.smithi028:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-05-12T21:46:30.731 INFO:teuthology.orchestra.run.smithi028.stderr:Generating grub configuration file ... 2024-05-12T21:46:32.152 INFO:teuthology.orchestra.run.smithi028.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-05-12T21:46:32.183 INFO:teuthology.orchestra.run.smithi028.stderr:done 2024-05-12T21:46:32.185 DEBUG:teuthology.orchestra.run.smithi028:> mktemp 2024-05-12T21:46:32.213 INFO:teuthology.orchestra.run.smithi028.stdout:/tmp/tmp.5mM2mY2qvr 2024-05-12T21:46:32.213 DEBUG:teuthology.orchestra.run.smithi028:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.5mM2mY2qvr 2024-05-12T21:46:32.289 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 0666 /tmp/tmp.5mM2mY2qvr 2024-05-12T21:46:32.438 DEBUG:teuthology.orchestra.remote:smithi028:/tmp/tmp.5mM2mY2qvr is 6KB 2024-05-12T21:46:32.486 DEBUG:teuthology.orchestra.run.smithi028:> rm -fr /tmp/tmp.5mM2mY2qvr 2024-05-12T21:46:32.502 DEBUG:teuthology.orchestra.run.smithi028:> sudo /bin/ls /boot/loader/entries || true 2024-05-12T21:46:32.564 INFO:teuthology.orchestra.run.smithi028.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-05-12T21:46:32.564 INFO:teuthology.orchestra.run.smithi028.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-05-12T21:46:32.564 INFO:teuthology.orchestra.run.smithi028.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-05-12T21:46:32.565 INFO:teuthology.orchestra.run.smithi028.stdout:7fc8828e06d0482694bcf4a447d13cb6-6.9.0-rc3-gfb9a6a44107e.conf 2024-05-12T21:46:32.566 DEBUG:teuthology.orchestra.run.smithi028:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-6.9.0-rc3-gfb9a6a44107e 2024-05-12T21:46:32.742 DEBUG:teuthology.orchestra.run.smithi028:> sudo shutdown -r now 2024-05-12T21:47:02.745 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-05-12T21:47:02.745 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-05-12T21:47:02.746 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi028.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-12T21:47:21.246 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.28 2024-05-12T21:47:30.247 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-05-12T21:47:30.248 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi028.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-12T21:47:48.642 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.28 2024-05-12T21:48:00.644 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-05-12T21:48:00.645 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi028.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-12T21:48:01.314 DEBUG:teuthology.orchestra.run.smithi028:> true 2024-05-12T21:48:02.047 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-05-12T21:48:02.047 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "6.9.0-rc3-gfb9a6a44107e"... 2024-05-12T21:48:02.048 DEBUG:teuthology.orchestra.run.smithi028:> uname -r 2024-05-12T21:48:02.106 INFO:teuthology.orchestra.run.smithi028.stdout:6.9.0-rc3-gfb9a6a44107e 2024-05-12T21:48:02.107 DEBUG:teuthology.task.kernel:current kernel version is 6.9.0-rc3-gfb9a6a44107e vs 6.9.0-rc3-gfb9a6a44107e 2024-05-12T21:48:02.107 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-05-12T21:48:02.107 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-05-12T21:48:03.108 DEBUG:teuthology.parallel:result is None 2024-05-12T21:48:03.109 INFO:teuthology.run_tasks:Running task internal.base... 2024-05-12T21:48:03.118 INFO:teuthology.task.internal:Creating test directory... 2024-05-12T21:48:03.118 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-05-12T21:48:03.125 DEBUG:teuthology.orchestra.run.smithi086:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-05-12T21:48:03.128 DEBUG:teuthology.orchestra.run.smithi186:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-05-12T21:48:03.168 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-05-12T21:48:03.175 INFO:teuthology.run_tasks:Running task internal.archive... 2024-05-12T21:48:03.183 INFO:teuthology.task.internal:Creating archive directory... 2024-05-12T21:48:03.183 DEBUG:teuthology.orchestra.run.smithi028:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-05-12T21:48:03.214 DEBUG:teuthology.orchestra.run.smithi086:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-05-12T21:48:03.217 DEBUG:teuthology.orchestra.run.smithi186:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-05-12T21:48:03.313 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-05-12T21:48:03.320 INFO:teuthology.task.internal:Enabling coredump saving... 2024-05-12T21:48:03.320 DEBUG:teuthology.orchestra.run.smithi086:> test -f /run/.containerenv -o -f /.dockerenv 2024-05-12T21:48:03.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-12T21:48:03.338 DEBUG:teuthology.orchestra.run.smithi186:> test -f /run/.containerenv -o -f /.dockerenv 2024-05-12T21:48:03.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-12T21:48:03.356 DEBUG:teuthology.orchestra.run.smithi028:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-05-12T21:48:03.362 DEBUG:teuthology.orchestra.run.smithi086:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-05-12T21:48:03.382 DEBUG:teuthology.orchestra.run.smithi186:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-05-12T21:48:03.414 INFO:teuthology.orchestra.run.smithi086.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-12T21:48:03.431 INFO:teuthology.orchestra.run.smithi186.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-12T21:48:03.433 INFO:teuthology.orchestra.run.smithi086.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-12T21:48:03.454 INFO:teuthology.orchestra.run.smithi186.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-12T21:48:03.524 INFO:teuthology.orchestra.run.smithi028.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-12T21:48:03.597 INFO:teuthology.orchestra.run.smithi028.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-12T21:48:03.601 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-05-12T21:48:03.612 INFO:teuthology.task.internal:Configuring sudo... 2024-05-12T21:48:03.612 DEBUG:teuthology.orchestra.run.smithi028:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-05-12T21:48:03.618 DEBUG:teuthology.orchestra.run.smithi086:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-05-12T21:48:03.620 DEBUG:teuthology.orchestra.run.smithi186:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-05-12T21:48:03.688 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-05-12T21:48:03.697 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-05-12T21:48:03.697 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-05-12T21:48:03.703 DEBUG:teuthology.orchestra.run.smithi086:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-05-12T21:48:03.706 DEBUG:teuthology.orchestra.run.smithi186:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-05-12T21:48:03.739 DEBUG:teuthology.orchestra.run.smithi028:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-12T21:48:03.858 DEBUG:teuthology.orchestra.run.smithi028:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-12T21:48:03.933 DEBUG:teuthology.orchestra.run.smithi028:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-12T21:48:04.017 DEBUG:teuthology.orchestra.run.smithi028:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-12T21:48:04.096 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-05-12T21:48:04.096 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-05-12T21:48:04.240 DEBUG:teuthology.orchestra.run.smithi086:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-12T21:48:04.283 DEBUG:teuthology.orchestra.run.smithi086:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-12T21:48:04.336 DEBUG:teuthology.orchestra.run.smithi086:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-12T21:48:04.418 DEBUG:teuthology.orchestra.run.smithi086:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-12T21:48:04.462 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-05-12T21:48:04.462 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-05-12T21:48:04.529 DEBUG:teuthology.orchestra.run.smithi186:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-12T21:48:04.574 DEBUG:teuthology.orchestra.run.smithi186:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-12T21:48:04.632 DEBUG:teuthology.orchestra.run.smithi186:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-12T21:48:04.715 DEBUG:teuthology.orchestra.run.smithi186:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-12T21:48:04.758 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-05-12T21:48:04.758 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-05-12T21:48:04.824 DEBUG:teuthology.orchestra.run.smithi028:> sudo service rsyslog restart 2024-05-12T21:48:04.832 DEBUG:teuthology.orchestra.run.smithi086:> sudo service rsyslog restart 2024-05-12T21:48:04.835 DEBUG:teuthology.orchestra.run.smithi186:> sudo service rsyslog restart 2024-05-12T21:48:04.875 INFO:teuthology.orchestra.run.smithi086.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-12T21:48:04.900 INFO:teuthology.orchestra.run.smithi186.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-12T21:48:04.945 INFO:teuthology.orchestra.run.smithi028.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-12T21:48:05.370 INFO:teuthology.run_tasks:Running task internal.timer... 2024-05-12T21:48:05.378 INFO:teuthology.task.internal:Starting timer... 2024-05-12T21:48:05.378 INFO:teuthology.run_tasks:Running task pcp... 2024-05-12T21:48:05.389 INFO:teuthology.run_tasks:Running task selinux... 2024-05-12T21:48:05.403 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-05-12T21:48:05.403 DEBUG:teuthology.orchestra.run.smithi028:> sudo service auditd rotate 2024-05-12T21:48:05.536 INFO:teuthology.orchestra.run.smithi028.stdout:Rotating logs: 2024-05-12T21:48:05.541 DEBUG:teuthology.orchestra.run.smithi086:> sudo service auditd rotate 2024-05-12T21:48:05.611 INFO:teuthology.orchestra.run.smithi086.stdout:Rotating logs: 2024-05-12T21:48:05.612 DEBUG:teuthology.orchestra.run.smithi186:> sudo service auditd rotate 2024-05-12T21:48:05.685 INFO:teuthology.orchestra.run.smithi186.stdout:Rotating logs: 2024-05-12T21:48:05.688 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-05-12T21:48:05.688 DEBUG:teuthology.orchestra.run.smithi028:> /usr/sbin/getenforce 2024-05-12T21:48:05.751 INFO:teuthology.orchestra.run.smithi028.stdout:Permissive 2024-05-12T21:48:05.752 DEBUG:teuthology.orchestra.run.smithi086:> /usr/sbin/getenforce 2024-05-12T21:48:05.799 INFO:teuthology.orchestra.run.smithi086.stdout:Permissive 2024-05-12T21:48:05.800 DEBUG:teuthology.orchestra.run.smithi186:> /usr/sbin/getenforce 2024-05-12T21:48:05.821 INFO:teuthology.orchestra.run.smithi186.stdout:Permissive 2024-05-12T21:48:05.822 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi028.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi086.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi186.front.sepia.ceph.com': 'permissive'} 2024-05-12T21:48:05.822 DEBUG:teuthology.orchestra.run.smithi028:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-12T21:48:05.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-12T21:48:05.896 DEBUG:teuthology.orchestra.run.smithi086:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-12T21:48:05.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-12T21:48:05.923 DEBUG:teuthology.orchestra.run.smithi186:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-12T21:48:05.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-12T21:48:05.950 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-05-12T21:48:05.950 DEBUG:teuthology.orchestra.run.smithi028:> sudo /usr/sbin/setenforce permissive 2024-05-12T21:48:06.104 DEBUG:teuthology.orchestra.run.smithi086:> sudo /usr/sbin/setenforce permissive 2024-05-12T21:48:06.143 DEBUG:teuthology.orchestra.run.smithi186:> sudo /usr/sbin/setenforce permissive 2024-05-12T21:48:06.169 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-05-12T21:48:06.179 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-05-12T21:48:06.198 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-05-12T21:48:06.199 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi028.front.sepia.ceph.com,smithi086.front.sepia.ceph.com,smithi186.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-05-12T21:53:53.299 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi028.front.sepia.ceph.com'), Remote(name='ubuntu@smithi086.front.sepia.ceph.com'), Remote(name='ubuntu@smithi186.front.sepia.ceph.com')] 2024-05-12T21:53:53.300 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-05-12T21:53:53.302 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi028.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-12T21:53:53.412 DEBUG:teuthology.orchestra.run.smithi028:> true 2024-05-12T21:53:53.580 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-05-12T21:53:53.580 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-05-12T21:53:53.580 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-12T21:53:53.652 DEBUG:teuthology.orchestra.run.smithi086:> true 2024-05-12T21:53:53.724 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-05-12T21:53:53.725 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-05-12T21:53:53.725 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-12T21:53:53.799 DEBUG:teuthology.orchestra.run.smithi186:> true 2024-05-12T21:53:53.891 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-05-12T21:53:53.892 INFO:teuthology.run_tasks:Running task clock... 2024-05-12T21:53:53.902 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-05-12T21:53:53.903 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-05-12T21:53:53.903 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-12T21:53:53.911 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-05-12T21:53:53.911 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-12T21:53:53.913 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-05-12T21:53:53.913 DEBUG:teuthology.orchestra.run.smithi186:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-12T21:53:53.938 INFO:teuthology.orchestra.run.smithi086.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-05-12T21:53:53.951 INFO:teuthology.orchestra.run.smithi086.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-05-12T21:53:53.962 INFO:teuthology.orchestra.run.smithi186.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-05-12T21:53:53.974 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: ntpd: command not found 2024-05-12T21:53:53.976 INFO:teuthology.orchestra.run.smithi186.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-05-12T21:53:53.984 INFO:teuthology.orchestra.run.smithi086.stdout:506 Cannot talk to daemon 2024-05-12T21:53:53.997 INFO:teuthology.orchestra.run.smithi086.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-05-12T21:53:54.000 INFO:teuthology.orchestra.run.smithi186.stderr:sudo: ntpd: command not found 2024-05-12T21:53:54.005 INFO:teuthology.orchestra.run.smithi028.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-05-12T21:53:54.010 INFO:teuthology.orchestra.run.smithi086.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-05-12T21:53:54.011 INFO:teuthology.orchestra.run.smithi186.stdout:506 Cannot talk to daemon 2024-05-12T21:53:54.025 INFO:teuthology.orchestra.run.smithi186.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-05-12T21:53:54.038 INFO:teuthology.orchestra.run.smithi186.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-05-12T21:53:54.047 INFO:teuthology.orchestra.run.smithi028.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-05-12T21:53:54.074 INFO:teuthology.orchestra.run.smithi086.stderr:bash: line 1: ntpq: command not found 2024-05-12T21:53:54.077 INFO:teuthology.orchestra.run.smithi086.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-12T21:53:54.077 INFO:teuthology.orchestra.run.smithi086.stdout:=============================================================================== 2024-05-12T21:53:54.078 INFO:teuthology.orchestra.run.smithi086.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-12T21:53:54.078 INFO:teuthology.orchestra.run.smithi086.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-12T21:53:54.078 INFO:teuthology.orchestra.run.smithi086.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-12T21:53:54.078 INFO:teuthology.orchestra.run.smithi086.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-12T21:53:54.096 INFO:teuthology.orchestra.run.smithi186.stderr:bash: line 1: ntpq: command not found 2024-05-12T21:53:54.100 INFO:teuthology.orchestra.run.smithi186.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-12T21:53:54.100 INFO:teuthology.orchestra.run.smithi186.stdout:=============================================================================== 2024-05-12T21:53:54.100 INFO:teuthology.orchestra.run.smithi186.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-12T21:53:54.100 INFO:teuthology.orchestra.run.smithi186.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-12T21:53:54.100 INFO:teuthology.orchestra.run.smithi186.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-12T21:53:54.100 INFO:teuthology.orchestra.run.smithi186.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-12T21:53:54.124 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: ntpd: command not found 2024-05-12T21:53:54.159 INFO:teuthology.orchestra.run.smithi028.stdout:506 Cannot talk to daemon 2024-05-12T21:53:54.205 INFO:teuthology.orchestra.run.smithi028.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-05-12T21:53:54.250 INFO:teuthology.orchestra.run.smithi028.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-05-12T21:53:54.402 INFO:teuthology.orchestra.run.smithi028.stderr:bash: line 1: ntpq: command not found 2024-05-12T21:53:54.410 INFO:teuthology.orchestra.run.smithi028.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-12T21:53:54.410 INFO:teuthology.orchestra.run.smithi028.stdout:=============================================================================== 2024-05-12T21:53:54.410 INFO:teuthology.orchestra.run.smithi028.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-12T21:53:54.410 INFO:teuthology.orchestra.run.smithi028.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-12T21:53:54.411 INFO:teuthology.orchestra.run.smithi028.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-12T21:53:54.411 INFO:teuthology.orchestra.run.smithi028.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-12T21:53:54.414 INFO:teuthology.run_tasks:Running task install... 2024-05-12T21:53:54.422 DEBUG:teuthology.task.install:project ceph 2024-05-12T21:53:54.422 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'f72fecff68e1d400c4568684327c900485c20d6a'}} 2024-05-12T21:53:54.423 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'f72fecff68e1d400c4568684327c900485c20d6a'} 2024-05-12T21:53:54.423 INFO:teuthology.task.install:Using flavor: default 2024-05-12T21:53:54.428 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-05-12T21:53:54.428 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-05-12T21:53:54.429 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'f72fecff68e1d400c4568684327c900485c20d6a', 'tag': None, 'wait_for_package': False} 2024-05-12T21:53:54.429 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=f72fecff68e1d400c4568684327c900485c20d6a 2024-05-12T21:53:54.431 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'f72fecff68e1d400c4568684327c900485c20d6a', 'tag': None, 'wait_for_package': False} 2024-05-12T21:53:54.431 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=f72fecff68e1d400c4568684327c900485c20d6a 2024-05-12T21:53:54.432 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'f72fecff68e1d400c4568684327c900485c20d6a', 'tag': None, 'wait_for_package': False} 2024-05-12T21:53:54.432 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=f72fecff68e1d400c4568684327c900485c20d6a 2024-05-12T21:53:54.573 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/squid/f72fecff68e1d400c4568684327c900485c20d6a/centos/9/flavors/default/ 2024-05-12T21:53:54.573 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2555.gf72fecff 2024-05-12T21:53:54.638 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/squid/f72fecff68e1d400c4568684327c900485c20d6a/centos/9/flavors/default/ 2024-05-12T21:53:54.638 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2555.gf72fecff 2024-05-12T21:53:54.654 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/squid/f72fecff68e1d400c4568684327c900485c20d6a/centos/9/flavors/default/ 2024-05-12T21:53:54.654 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2555.gf72fecff 2024-05-12T21:53:54.806 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/squid/f72fecff68e1d400c4568684327c900485c20d6a/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/f72fecff68e1d400c4568684327c900485c20d6a/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/f72fecff68e1d400c4568684327c900485c20d6a/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-05-12T21:53:54.806 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-05-12T21:53:54.807 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-05-12T21:53:54.820 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/squid/f72fecff68e1d400c4568684327c900485c20d6a/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/f72fecff68e1d400c4568684327c900485c20d6a/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/f72fecff68e1d400c4568684327c900485c20d6a/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-05-12T21:53:54.821 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-05-12T21:53:54.821 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-05-12T21:53:54.833 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-05-12T21:53:54.833 DEBUG:teuthology.orchestra.run.smithi086:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/f72fecff68e1d400c4568684327c900485c20d6a/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-05-12T21:53:54.879 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/squid/f72fecff68e1d400c4568684327c900485c20d6a/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/f72fecff68e1d400c4568684327c900485c20d6a/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/f72fecff68e1d400c4568684327c900485c20d6a/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-05-12T21:53:54.880 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-05-12T21:53:54.880 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-05-12T21:53:54.898 DEBUG:teuthology.orchestra.run.smithi086:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-05-12T21:53:54.908 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-05-12T21:53:54.908 DEBUG:teuthology.orchestra.run.smithi186:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/f72fecff68e1d400c4568684327c900485c20d6a/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-05-12T21:53:54.917 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-05-12T21:53:54.917 DEBUG:teuthology.orchestra.run.smithi028:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/f72fecff68e1d400c4568684327c900485c20d6a/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-05-12T21:53:54.972 DEBUG:teuthology.orchestra.run.smithi086:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-05-12T21:53:54.977 DEBUG:teuthology.orchestra.run.smithi186:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-05-12T21:53:55.035 INFO:teuthology.orchestra.run.smithi086.stdout:check_obsoletes = 1 2024-05-12T21:53:55.036 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum clean all 2024-05-12T21:53:55.052 DEBUG:teuthology.orchestra.run.smithi028:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-05-12T21:53:55.055 DEBUG:teuthology.orchestra.run.smithi186:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-05-12T21:53:55.119 INFO:teuthology.orchestra.run.smithi186.stdout:check_obsoletes = 1 2024-05-12T21:53:55.121 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum clean all 2024-05-12T21:53:55.163 DEBUG:teuthology.orchestra.run.smithi028:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-05-12T21:53:55.241 INFO:teuthology.orchestra.run.smithi028.stdout:check_obsoletes = 1 2024-05-12T21:53:55.246 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum clean all 2024-05-12T21:53:55.261 INFO:teuthology.orchestra.run.smithi086.stdout:54 files removed 2024-05-12T21:53:55.290 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-05-12T21:53:55.354 INFO:teuthology.orchestra.run.smithi186.stdout:54 files removed 2024-05-12T21:53:55.381 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-05-12T21:53:55.586 INFO:teuthology.orchestra.run.smithi028.stdout:54 files removed 2024-05-12T21:53:55.625 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-05-12T21:53:56.339 INFO:teuthology.orchestra.run.smithi086.stdout:ceph packages for x86_64 106 kB/s | 83 kB 00:00 2024-05-12T21:53:56.478 INFO:teuthology.orchestra.run.smithi186.stdout:ceph packages for x86_64 100 kB/s | 83 kB 00:00 2024-05-12T21:53:56.946 INFO:teuthology.orchestra.run.smithi028.stdout:ceph packages for x86_64 95 kB/s | 83 kB 00:00 2024-05-12T21:53:57.078 INFO:teuthology.orchestra.run.smithi086.stdout:ceph noarch packages 16 kB/s | 11 kB 00:00 2024-05-12T21:53:57.162 INFO:teuthology.orchestra.run.smithi186.stdout:ceph noarch packages 18 kB/s | 11 kB 00:00 2024-05-12T21:53:57.695 INFO:teuthology.orchestra.run.smithi186.stdout:ceph source packages 3.8 kB/s | 1.9 kB 00:00 2024-05-12T21:53:57.712 INFO:teuthology.orchestra.run.smithi086.stdout:ceph source packages 3.2 kB/s | 1.9 kB 00:00 2024-05-12T21:53:57.797 INFO:teuthology.orchestra.run.smithi028.stdout:ceph noarch packages 15 kB/s | 11 kB 00:00 2024-05-12T21:53:58.612 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 9 - BaseOS 9.3 MB/s | 8.1 MB 00:00 2024-05-12T21:53:58.773 INFO:teuthology.orchestra.run.smithi028.stdout:ceph source packages 2.1 kB/s | 1.9 kB 00:00 2024-05-12T21:53:59.729 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS Stream 9 - BaseOS 4.0 MB/s | 8.1 MB 00:01 2024-05-12T21:54:00.679 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - BaseOS 4.4 MB/s | 8.1 MB 00:01 2024-05-12T21:54:02.688 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS Stream 9 - AppStream 10 MB/s | 19 MB 00:01 2024-05-12T21:54:03.279 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 9 - AppStream 5.3 MB/s | 19 MB 00:03 2024-05-12T21:54:04.503 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - AppStream 7.5 MB/s | 19 MB 00:02 2024-05-12T21:54:08.222 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 9 - CRB 4.4 MB/s | 6.2 MB 00:01 2024-05-12T21:54:08.238 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS Stream 9 - CRB 3.1 MB/s | 6.2 MB 00:01 2024-05-12T21:54:09.098 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.2 MB 00:00 2024-05-12T21:54:09.623 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS Stream 9 - Extras packages 74 kB/s | 16 kB 00:00 2024-05-12T21:54:09.847 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 9 - Extras packages 34 kB/s | 16 kB 00:00 2024-05-12T21:54:10.416 INFO:teuthology.orchestra.run.smithi186.stdout:Extra Packages for Enterprise Linux 32 MB/s | 21 MB 00:00 2024-05-12T21:54:11.790 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - Extras packages 12 kB/s | 16 kB 00:01 2024-05-12T21:54:12.132 INFO:teuthology.orchestra.run.smithi086.stdout:Extra Packages for Enterprise Linux 9.8 MB/s | 21 MB 00:02 2024-05-12T21:54:13.531 INFO:teuthology.orchestra.run.smithi028.stdout:Extra Packages for Enterprise Linux 14 MB/s | 21 MB 00:01 2024-05-12T21:54:15.699 INFO:teuthology.orchestra.run.smithi186.stdout:lab-extras 16 kB/s | 1.7 kB 00:00 2024-05-12T21:54:17.324 INFO:teuthology.orchestra.run.smithi086.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-05-12T21:54:17.976 INFO:teuthology.orchestra.run.smithi186.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-05-12T21:54:17.976 INFO:teuthology.orchestra.run.smithi186.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-05-12T21:54:17.987 INFO:teuthology.orchestra.run.smithi186.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-05-12T21:54:17.988 INFO:teuthology.orchestra.run.smithi186.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-05-12T21:54:17.989 INFO:teuthology.orchestra.run.smithi186.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-05-12T21:54:17.990 INFO:teuthology.orchestra.run.smithi186.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-05-12T21:54:17.991 INFO:teuthology.orchestra.run.smithi186.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-05-12T21:54:17.993 INFO:teuthology.orchestra.run.smithi186.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-05-12T21:54:17.994 INFO:teuthology.orchestra.run.smithi186.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-05-12T21:54:18.055 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-05-12T21:54:18.074 INFO:teuthology.orchestra.run.smithi186.stdout:======================================================================================= 2024-05-12T21:54:18.074 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repository Size 2024-05-12T21:54:18.075 INFO:teuthology.orchestra.run.smithi186.stdout:======================================================================================= 2024-05-12T21:54:18.075 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-05-12T21:54:18.075 INFO:teuthology.orchestra.run.smithi186.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-05-12T21:54:18.075 INFO:teuthology.orchestra.run.smithi186.stdout: ceph x86_64 2:19.0.0-2555.gf72fecff.el9 ceph 6.6 k 2024-05-12T21:54:18.075 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-base x86_64 2:19.0.0-2555.gf72fecff.el9 ceph 5.4 M 2024-05-12T21:54:18.075 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-fuse x86_64 2:19.0.0-2555.gf72fecff.el9 ceph 889 k 2024-05-12T21:54:18.075 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2555.gf72fecff.el9 ceph 145 k 2024-05-12T21:54:18.075 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr x86_64 2:19.0.0-2555.gf72fecff.el9 ceph 1.5 M 2024-05-12T21:54:18.075 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2555.gf72fecff.el9 ceph-noarch 142 k 2024-05-12T21:54:18.075 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2555.gf72fecff.el9 ceph-noarch 3.5 M 2024-05-12T21:54:18.075 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2555.gf72fecff.el9 ceph-noarch 7.4 M 2024-05-12T21:54:18.075 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-rook noarch 2:19.0.0-2555.gf72fecff.el9 ceph-noarch 49 k 2024-05-12T21:54:18.075 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-radosgw x86_64 2:19.0.0-2555.gf72fecff.el9 ceph 10 M 2024-05-12T21:54:18.075 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-test x86_64 2:19.0.0-2555.gf72fecff.el9 ceph 48 M 2024-05-12T21:54:18.075 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-volume noarch 2:19.0.0-2555.gf72fecff.el9 ceph-noarch 264 k 2024-05-12T21:54:18.075 INFO:teuthology.orchestra.run.smithi186.stdout: cephadm noarch 2:19.0.0-2555.gf72fecff.el9 ceph-noarch 763 k 2024-05-12T21:54:18.076 INFO:teuthology.orchestra.run.smithi186.stdout: cephfs-mirror x86_64 2:19.0.0-2555.gf72fecff.el9 ceph 221 k 2024-05-12T21:54:18.076 INFO:teuthology.orchestra.run.smithi186.stdout: cephfs-shell x86_64 2:19.0.0-2555.gf72fecff.el9 ceph 27 k 2024-05-12T21:54:18.076 INFO:teuthology.orchestra.run.smithi186.stdout: cephfs-top noarch 2:19.0.0-2555.gf72fecff.el9 ceph-noarch 18 k 2024-05-12T21:54:18.076 INFO:teuthology.orchestra.run.smithi186.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-05-12T21:54:18.076 INFO:teuthology.orchestra.run.smithi186.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-05-12T21:54:18.076 INFO:teuthology.orchestra.run.smithi186.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-05-12T21:54:18.076 INFO:teuthology.orchestra.run.smithi186.stdout: libcephfs-devel x86_64 2:19.0.0-2555.gf72fecff.el9 ceph 32 k 2024-05-12T21:54:18.076 INFO:teuthology.orchestra.run.smithi186.stdout: libcephfs2 x86_64 2:19.0.0-2555.gf72fecff.el9 ceph 745 k 2024-05-12T21:54:18.076 INFO:teuthology.orchestra.run.smithi186.stdout: librados-devel x86_64 2:19.0.0-2555.gf72fecff.el9 ceph 127 k 2024-05-12T21:54:18.076 INFO:teuthology.orchestra.run.smithi186.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-05-12T21:54:18.076 INFO:teuthology.orchestra.run.smithi186.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-05-12T21:54:18.076 INFO:teuthology.orchestra.run.smithi186.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-05-12T21:54:18.076 INFO:teuthology.orchestra.run.smithi186.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-05-12T21:54:18.076 INFO:teuthology.orchestra.run.smithi186.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-05-12T21:54:18.077 INFO:teuthology.orchestra.run.smithi186.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-05-12T21:54:18.077 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cephfs x86_64 2:19.0.0-2555.gf72fecff.el9 ceph 161 k 2024-05-12T21:54:18.077 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rados x86_64 2:19.0.0-2555.gf72fecff.el9 ceph 321 k 2024-05-12T21:54:18.077 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rbd x86_64 2:19.0.0-2555.gf72fecff.el9 ceph 301 k 2024-05-12T21:54:18.077 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rgw x86_64 2:19.0.0-2555.gf72fecff.el9 ceph 99 k 2024-05-12T21:54:18.077 INFO:teuthology.orchestra.run.smithi186.stdout: rbd-fuse x86_64 2:19.0.0-2555.gf72fecff.el9 ceph 85 k 2024-05-12T21:54:18.077 INFO:teuthology.orchestra.run.smithi186.stdout: rbd-mirror x86_64 2:19.0.0-2555.gf72fecff.el9 ceph 3.1 M 2024-05-12T21:54:18.077 INFO:teuthology.orchestra.run.smithi186.stdout: rbd-nbd x86_64 2:19.0.0-2555.gf72fecff.el9 ceph 171 k 2024-05-12T21:54:18.077 INFO:teuthology.orchestra.run.smithi186.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-05-12T21:54:18.077 INFO:teuthology.orchestra.run.smithi186.stdout:Upgrading: 2024-05-12T21:54:18.077 INFO:teuthology.orchestra.run.smithi186.stdout: NetworkManager x86_64 1:1.47.5-1.el9 baseos 2.3 M 2024-05-12T21:54:18.077 INFO:teuthology.orchestra.run.smithi186.stdout: NetworkManager-libnm x86_64 1:1.47.5-1.el9 baseos 1.8 M 2024-05-12T21:54:18.077 INFO:teuthology.orchestra.run.smithi186.stdout: NetworkManager-team x86_64 1:1.47.5-1.el9 baseos 42 k 2024-05-12T21:54:18.077 INFO:teuthology.orchestra.run.smithi186.stdout: NetworkManager-tui x86_64 1:1.47.5-1.el9 baseos 248 k 2024-05-12T21:54:18.078 INFO:teuthology.orchestra.run.smithi186.stdout: librados2 x86_64 2:19.0.0-2555.gf72fecff.el9 ceph 3.4 M 2024-05-12T21:54:18.078 INFO:teuthology.orchestra.run.smithi186.stdout: librbd1 x86_64 2:19.0.0-2555.gf72fecff.el9 ceph 3.1 M 2024-05-12T21:54:18.078 INFO:teuthology.orchestra.run.smithi186.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-05-12T21:54:18.078 INFO:teuthology.orchestra.run.smithi186.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-05-12T21:54:18.078 INFO:teuthology.orchestra.run.smithi186.stdout:Installing dependencies: 2024-05-12T21:54:18.078 INFO:teuthology.orchestra.run.smithi186.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-05-12T21:54:18.078 INFO:teuthology.orchestra.run.smithi186.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-05-12T21:54:18.078 INFO:teuthology.orchestra.run.smithi186.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-05-12T21:54:18.079 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-common x86_64 2:19.0.0-2555.gf72fecff.el9 ceph 21 M 2024-05-12T21:54:18.079 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2555.gf72fecff.el9 ceph-noarch 32 k 2024-05-12T21:54:18.079 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mds x86_64 2:19.0.0-2555.gf72fecff.el9 ceph 2.4 M 2024-05-12T21:54:18.079 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2555.gf72fecff.el9 ceph-noarch 249 k 2024-05-12T21:54:18.079 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mon x86_64 2:19.0.0-2555.gf72fecff.el9 ceph 4.7 M 2024-05-12T21:54:18.079 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-osd x86_64 2:19.0.0-2555.gf72fecff.el9 ceph 17 M 2024-05-12T21:54:18.079 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2555.gf72fecff.el9 ceph-noarch 16 k 2024-05-12T21:54:18.079 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-selinux x86_64 2:19.0.0-2555.gf72fecff.el9 ceph 25 k 2024-05-12T21:54:18.079 INFO:teuthology.orchestra.run.smithi186.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-05-12T21:54:18.079 INFO:teuthology.orchestra.run.smithi186.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-05-12T21:54:18.079 INFO:teuthology.orchestra.run.smithi186.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-05-12T21:54:18.080 INFO:teuthology.orchestra.run.smithi186.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-05-12T21:54:18.080 INFO:teuthology.orchestra.run.smithi186.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-05-12T21:54:18.080 INFO:teuthology.orchestra.run.smithi186.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-05-12T21:54:18.080 INFO:teuthology.orchestra.run.smithi186.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-05-12T21:54:18.080 INFO:teuthology.orchestra.run.smithi186.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-05-12T21:54:18.080 INFO:teuthology.orchestra.run.smithi186.stdout: libcephsqlite x86_64 2:19.0.0-2555.gf72fecff.el9 ceph 163 k 2024-05-12T21:54:18.080 INFO:teuthology.orchestra.run.smithi186.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-05-12T21:54:18.080 INFO:teuthology.orchestra.run.smithi186.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-05-12T21:54:18.080 INFO:teuthology.orchestra.run.smithi186.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-05-12T21:54:18.080 INFO:teuthology.orchestra.run.smithi186.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-05-12T21:54:18.080 INFO:teuthology.orchestra.run.smithi186.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-05-12T21:54:18.080 INFO:teuthology.orchestra.run.smithi186.stdout: libradosstriper1 x86_64 2:19.0.0-2555.gf72fecff.el9 ceph 500 k 2024-05-12T21:54:18.080 INFO:teuthology.orchestra.run.smithi186.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-05-12T21:54:18.081 INFO:teuthology.orchestra.run.smithi186.stdout: librgw2 x86_64 2:19.0.0-2555.gf72fecff.el9 ceph 5.3 M 2024-05-12T21:54:18.081 INFO:teuthology.orchestra.run.smithi186.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-05-12T21:54:18.081 INFO:teuthology.orchestra.run.smithi186.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-05-12T21:54:18.081 INFO:teuthology.orchestra.run.smithi186.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-05-12T21:54:18.081 INFO:teuthology.orchestra.run.smithi186.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-05-12T21:54:18.081 INFO:teuthology.orchestra.run.smithi186.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-05-12T21:54:18.081 INFO:teuthology.orchestra.run.smithi186.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-05-12T21:54:18.081 INFO:teuthology.orchestra.run.smithi186.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-05-12T21:54:18.081 INFO:teuthology.orchestra.run.smithi186.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-05-12T21:54:18.081 INFO:teuthology.orchestra.run.smithi186.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-05-12T21:54:18.081 INFO:teuthology.orchestra.run.smithi186.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-05-12T21:54:18.081 INFO:teuthology.orchestra.run.smithi186.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-05-12T21:54:18.081 INFO:teuthology.orchestra.run.smithi186.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-05-12T21:54:18.082 INFO:teuthology.orchestra.run.smithi186.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-05-12T21:54:18.082 INFO:teuthology.orchestra.run.smithi186.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-05-12T21:54:18.082 INFO:teuthology.orchestra.run.smithi186.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-05-12T21:54:18.082 INFO:teuthology.orchestra.run.smithi186.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-05-12T21:54:18.082 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-05-12T21:54:18.082 INFO:teuthology.orchestra.run.smithi186.stdout: python3-ceph-argparse x86_64 2:19.0.0-2555.gf72fecff.el9 ceph 45 k 2024-05-12T21:54:18.082 INFO:teuthology.orchestra.run.smithi186.stdout: python3-ceph-common x86_64 2:19.0.0-2555.gf72fecff.el9 ceph 135 k 2024-05-12T21:54:18.082 INFO:teuthology.orchestra.run.smithi186.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-05-12T21:54:18.082 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-05-12T21:54:18.082 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-05-12T21:54:18.082 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-05-12T21:54:18.083 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-05-12T21:54:18.083 INFO:teuthology.orchestra.run.smithi186.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-05-12T21:54:18.083 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-05-12T21:54:18.083 INFO:teuthology.orchestra.run.smithi186.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-05-12T21:54:18.083 INFO:teuthology.orchestra.run.smithi186.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-05-12T21:54:18.083 INFO:teuthology.orchestra.run.smithi186.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-05-12T21:54:18.083 INFO:teuthology.orchestra.run.smithi186.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-05-12T21:54:18.083 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-05-12T21:54:18.083 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-05-12T21:54:18.083 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-05-12T21:54:18.083 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-05-12T21:54:18.083 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-05-12T21:54:18.083 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-05-12T21:54:18.084 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-05-12T21:54:18.084 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-05-12T21:54:18.084 INFO:teuthology.orchestra.run.smithi186.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-05-12T21:54:18.084 INFO:teuthology.orchestra.run.smithi186.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-05-12T21:54:18.084 INFO:teuthology.orchestra.run.smithi186.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-05-12T21:54:18.084 INFO:teuthology.orchestra.run.smithi186.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-05-12T21:54:18.084 INFO:teuthology.orchestra.run.smithi186.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-05-12T21:54:18.084 INFO:teuthology.orchestra.run.smithi186.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-05-12T21:54:18.084 INFO:teuthology.orchestra.run.smithi186.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-05-12T21:54:18.084 INFO:teuthology.orchestra.run.smithi186.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-05-12T21:54:18.084 INFO:teuthology.orchestra.run.smithi186.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-05-12T21:54:18.084 INFO:teuthology.orchestra.run.smithi186.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-05-12T21:54:18.084 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-05-12T21:54:18.085 INFO:teuthology.orchestra.run.smithi186.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-05-12T21:54:18.085 INFO:teuthology.orchestra.run.smithi186.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-05-12T21:54:18.085 INFO:teuthology.orchestra.run.smithi186.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-05-12T21:54:18.085 INFO:teuthology.orchestra.run.smithi186.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-05-12T21:54:18.085 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-05-12T21:54:18.085 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-05-12T21:54:18.085 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-05-12T21:54:18.085 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-05-12T21:54:18.085 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-05-12T21:54:18.085 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-05-12T21:54:18.085 INFO:teuthology.orchestra.run.smithi186.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-05-12T21:54:18.085 INFO:teuthology.orchestra.run.smithi186.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-05-12T21:54:18.085 INFO:teuthology.orchestra.run.smithi186.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-05-12T21:54:18.085 INFO:teuthology.orchestra.run.smithi186.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-05-12T21:54:18.086 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-05-12T21:54:18.086 INFO:teuthology.orchestra.run.smithi186.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-05-12T21:54:18.086 INFO:teuthology.orchestra.run.smithi186.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-05-12T21:54:18.086 INFO:teuthology.orchestra.run.smithi186.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-05-12T21:54:18.086 INFO:teuthology.orchestra.run.smithi186.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-05-12T21:54:18.086 INFO:teuthology.orchestra.run.smithi186.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-05-12T21:54:18.086 INFO:teuthology.orchestra.run.smithi186.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-05-12T21:54:18.086 INFO:teuthology.orchestra.run.smithi186.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-05-12T21:54:18.086 INFO:teuthology.orchestra.run.smithi186.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-05-12T21:54:18.086 INFO:teuthology.orchestra.run.smithi186.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-05-12T21:54:18.086 INFO:teuthology.orchestra.run.smithi186.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-05-12T21:54:18.086 INFO:teuthology.orchestra.run.smithi186.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-05-12T21:54:18.086 INFO:teuthology.orchestra.run.smithi186.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-05-12T21:54:18.087 INFO:teuthology.orchestra.run.smithi186.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-05-12T21:54:18.087 INFO:teuthology.orchestra.run.smithi186.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-05-12T21:54:18.087 INFO:teuthology.orchestra.run.smithi186.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-05-12T21:54:18.087 INFO:teuthology.orchestra.run.smithi186.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-05-12T21:54:18.087 INFO:teuthology.orchestra.run.smithi186.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-05-12T21:54:18.087 INFO:teuthology.orchestra.run.smithi186.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-05-12T21:54:18.087 INFO:teuthology.orchestra.run.smithi186.stdout:Installing weak dependencies: 2024-05-12T21:54:18.087 INFO:teuthology.orchestra.run.smithi186.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-05-12T21:54:18.087 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-12T21:54:18.087 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-05-12T21:54:18.087 INFO:teuthology.orchestra.run.smithi186.stdout:======================================================================================= 2024-05-12T21:54:18.087 INFO:teuthology.orchestra.run.smithi186.stdout:Install 148 Packages 2024-05-12T21:54:18.087 INFO:teuthology.orchestra.run.smithi186.stdout:Upgrade 8 Packages 2024-05-12T21:54:18.088 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-12T21:54:18.088 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 225 M 2024-05-12T21:54:18.088 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-05-12T21:54:19.026 INFO:teuthology.orchestra.run.smithi186.stdout:(1/156): ceph-19.0.0-2555.gf72fecff.el9.x86_64. 29 kB/s | 6.6 kB 00:00 2024-05-12T21:54:19.359 INFO:teuthology.orchestra.run.smithi186.stdout:(2/156): ceph-fuse-19.0.0-2555.gf72fecff.el9.x8 2.6 MB/s | 889 kB 00:00 2024-05-12T21:54:19.460 INFO:teuthology.orchestra.run.smithi186.stdout:(3/156): ceph-immutable-object-cache-19.0.0-255 1.4 MB/s | 145 kB 00:00 2024-05-12T21:54:19.592 INFO:teuthology.orchestra.run.smithi086.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-05-12T21:54:19.593 INFO:teuthology.orchestra.run.smithi086.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-05-12T21:54:19.604 INFO:teuthology.orchestra.run.smithi086.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-05-12T21:54:19.605 INFO:teuthology.orchestra.run.smithi086.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-05-12T21:54:19.605 INFO:teuthology.orchestra.run.smithi086.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-05-12T21:54:19.606 INFO:teuthology.orchestra.run.smithi086.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-05-12T21:54:19.608 INFO:teuthology.orchestra.run.smithi086.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-05-12T21:54:19.610 INFO:teuthology.orchestra.run.smithi086.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-05-12T21:54:19.611 INFO:teuthology.orchestra.run.smithi086.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-05-12T21:54:19.672 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-05-12T21:54:19.685 INFO:teuthology.orchestra.run.smithi086.stdout:======================================================================================= 2024-05-12T21:54:19.686 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2024-05-12T21:54:19.686 INFO:teuthology.orchestra.run.smithi086.stdout:======================================================================================= 2024-05-12T21:54:19.686 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-05-12T21:54:19.686 INFO:teuthology.orchestra.run.smithi086.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-05-12T21:54:19.686 INFO:teuthology.orchestra.run.smithi086.stdout: ceph x86_64 2:19.0.0-2555.gf72fecff.el9 ceph 6.6 k 2024-05-12T21:54:19.686 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-base x86_64 2:19.0.0-2555.gf72fecff.el9 ceph 5.4 M 2024-05-12T21:54:19.686 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-fuse x86_64 2:19.0.0-2555.gf72fecff.el9 ceph 889 k 2024-05-12T21:54:19.686 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2555.gf72fecff.el9 ceph 145 k 2024-05-12T21:54:19.687 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr x86_64 2:19.0.0-2555.gf72fecff.el9 ceph 1.5 M 2024-05-12T21:54:19.687 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2555.gf72fecff.el9 ceph-noarch 142 k 2024-05-12T21:54:19.687 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2555.gf72fecff.el9 ceph-noarch 3.5 M 2024-05-12T21:54:19.687 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2555.gf72fecff.el9 ceph-noarch 7.4 M 2024-05-12T21:54:19.687 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-rook noarch 2:19.0.0-2555.gf72fecff.el9 ceph-noarch 49 k 2024-05-12T21:54:19.687 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-radosgw x86_64 2:19.0.0-2555.gf72fecff.el9 ceph 10 M 2024-05-12T21:54:19.687 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-test x86_64 2:19.0.0-2555.gf72fecff.el9 ceph 48 M 2024-05-12T21:54:19.688 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-volume noarch 2:19.0.0-2555.gf72fecff.el9 ceph-noarch 264 k 2024-05-12T21:54:19.688 INFO:teuthology.orchestra.run.smithi086.stdout: cephadm noarch 2:19.0.0-2555.gf72fecff.el9 ceph-noarch 763 k 2024-05-12T21:54:19.688 INFO:teuthology.orchestra.run.smithi086.stdout: cephfs-mirror x86_64 2:19.0.0-2555.gf72fecff.el9 ceph 221 k 2024-05-12T21:54:19.688 INFO:teuthology.orchestra.run.smithi086.stdout: cephfs-shell x86_64 2:19.0.0-2555.gf72fecff.el9 ceph 27 k 2024-05-12T21:54:19.688 INFO:teuthology.orchestra.run.smithi086.stdout: cephfs-top noarch 2:19.0.0-2555.gf72fecff.el9 ceph-noarch 18 k 2024-05-12T21:54:19.688 INFO:teuthology.orchestra.run.smithi086.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-05-12T21:54:19.688 INFO:teuthology.orchestra.run.smithi086.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-05-12T21:54:19.688 INFO:teuthology.orchestra.run.smithi086.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-05-12T21:54:19.688 INFO:teuthology.orchestra.run.smithi086.stdout: libcephfs-devel x86_64 2:19.0.0-2555.gf72fecff.el9 ceph 32 k 2024-05-12T21:54:19.688 INFO:teuthology.orchestra.run.smithi086.stdout: libcephfs2 x86_64 2:19.0.0-2555.gf72fecff.el9 ceph 745 k 2024-05-12T21:54:19.688 INFO:teuthology.orchestra.run.smithi086.stdout: librados-devel x86_64 2:19.0.0-2555.gf72fecff.el9 ceph 127 k 2024-05-12T21:54:19.688 INFO:teuthology.orchestra.run.smithi086.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-05-12T21:54:19.688 INFO:teuthology.orchestra.run.smithi086.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-05-12T21:54:19.688 INFO:teuthology.orchestra.run.smithi086.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-05-12T21:54:19.688 INFO:teuthology.orchestra.run.smithi086.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-05-12T21:54:19.689 INFO:teuthology.orchestra.run.smithi086.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-05-12T21:54:19.689 INFO:teuthology.orchestra.run.smithi086.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-05-12T21:54:19.689 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cephfs x86_64 2:19.0.0-2555.gf72fecff.el9 ceph 161 k 2024-05-12T21:54:19.689 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rados x86_64 2:19.0.0-2555.gf72fecff.el9 ceph 321 k 2024-05-12T21:54:19.689 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rbd x86_64 2:19.0.0-2555.gf72fecff.el9 ceph 301 k 2024-05-12T21:54:19.689 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rgw x86_64 2:19.0.0-2555.gf72fecff.el9 ceph 99 k 2024-05-12T21:54:19.689 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-fuse x86_64 2:19.0.0-2555.gf72fecff.el9 ceph 85 k 2024-05-12T21:54:19.689 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-mirror x86_64 2:19.0.0-2555.gf72fecff.el9 ceph 3.1 M 2024-05-12T21:54:19.689 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-nbd x86_64 2:19.0.0-2555.gf72fecff.el9 ceph 171 k 2024-05-12T21:54:19.689 INFO:teuthology.orchestra.run.smithi086.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-05-12T21:54:19.689 INFO:teuthology.orchestra.run.smithi086.stdout:Upgrading: 2024-05-12T21:54:19.689 INFO:teuthology.orchestra.run.smithi086.stdout: NetworkManager x86_64 1:1.47.5-1.el9 baseos 2.3 M 2024-05-12T21:54:19.689 INFO:teuthology.orchestra.run.smithi086.stdout: NetworkManager-libnm x86_64 1:1.47.5-1.el9 baseos 1.8 M 2024-05-12T21:54:19.689 INFO:teuthology.orchestra.run.smithi086.stdout: NetworkManager-team x86_64 1:1.47.5-1.el9 baseos 42 k 2024-05-12T21:54:19.690 INFO:teuthology.orchestra.run.smithi086.stdout: NetworkManager-tui x86_64 1:1.47.5-1.el9 baseos 248 k 2024-05-12T21:54:19.690 INFO:teuthology.orchestra.run.smithi086.stdout: librados2 x86_64 2:19.0.0-2555.gf72fecff.el9 ceph 3.4 M 2024-05-12T21:54:19.690 INFO:teuthology.orchestra.run.smithi086.stdout: librbd1 x86_64 2:19.0.0-2555.gf72fecff.el9 ceph 3.1 M 2024-05-12T21:54:19.690 INFO:teuthology.orchestra.run.smithi086.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-05-12T21:54:19.690 INFO:teuthology.orchestra.run.smithi086.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-05-12T21:54:19.690 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2024-05-12T21:54:19.690 INFO:teuthology.orchestra.run.smithi086.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-05-12T21:54:19.690 INFO:teuthology.orchestra.run.smithi086.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-05-12T21:54:19.690 INFO:teuthology.orchestra.run.smithi086.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-05-12T21:54:19.690 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-common x86_64 2:19.0.0-2555.gf72fecff.el9 ceph 21 M 2024-05-12T21:54:19.691 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2555.gf72fecff.el9 ceph-noarch 32 k 2024-05-12T21:54:19.691 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mds x86_64 2:19.0.0-2555.gf72fecff.el9 ceph 2.4 M 2024-05-12T21:54:19.691 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2555.gf72fecff.el9 ceph-noarch 249 k 2024-05-12T21:54:19.691 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mon x86_64 2:19.0.0-2555.gf72fecff.el9 ceph 4.7 M 2024-05-12T21:54:19.691 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-osd x86_64 2:19.0.0-2555.gf72fecff.el9 ceph 17 M 2024-05-12T21:54:19.691 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2555.gf72fecff.el9 ceph-noarch 16 k 2024-05-12T21:54:19.691 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-selinux x86_64 2:19.0.0-2555.gf72fecff.el9 ceph 25 k 2024-05-12T21:54:19.691 INFO:teuthology.orchestra.run.smithi086.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-05-12T21:54:19.691 INFO:teuthology.orchestra.run.smithi086.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-05-12T21:54:19.691 INFO:teuthology.orchestra.run.smithi086.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-05-12T21:54:19.691 INFO:teuthology.orchestra.run.smithi086.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-05-12T21:54:19.691 INFO:teuthology.orchestra.run.smithi086.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-05-12T21:54:19.691 INFO:teuthology.orchestra.run.smithi086.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-05-12T21:54:19.692 INFO:teuthology.orchestra.run.smithi086.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-05-12T21:54:19.692 INFO:teuthology.orchestra.run.smithi086.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-05-12T21:54:19.692 INFO:teuthology.orchestra.run.smithi086.stdout: libcephsqlite x86_64 2:19.0.0-2555.gf72fecff.el9 ceph 163 k 2024-05-12T21:54:19.692 INFO:teuthology.orchestra.run.smithi086.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-05-12T21:54:19.692 INFO:teuthology.orchestra.run.smithi086.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-05-12T21:54:19.692 INFO:teuthology.orchestra.run.smithi086.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-05-12T21:54:19.692 INFO:teuthology.orchestra.run.smithi086.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-05-12T21:54:19.692 INFO:teuthology.orchestra.run.smithi086.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-05-12T21:54:19.692 INFO:teuthology.orchestra.run.smithi086.stdout: libradosstriper1 x86_64 2:19.0.0-2555.gf72fecff.el9 ceph 500 k 2024-05-12T21:54:19.692 INFO:teuthology.orchestra.run.smithi086.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-05-12T21:54:19.692 INFO:teuthology.orchestra.run.smithi086.stdout: librgw2 x86_64 2:19.0.0-2555.gf72fecff.el9 ceph 5.3 M 2024-05-12T21:54:19.692 INFO:teuthology.orchestra.run.smithi086.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-05-12T21:54:19.692 INFO:teuthology.orchestra.run.smithi086.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-05-12T21:54:19.692 INFO:teuthology.orchestra.run.smithi086.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-05-12T21:54:19.692 INFO:teuthology.orchestra.run.smithi086.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-05-12T21:54:19.693 INFO:teuthology.orchestra.run.smithi086.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-05-12T21:54:19.693 INFO:teuthology.orchestra.run.smithi086.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-05-12T21:54:19.693 INFO:teuthology.orchestra.run.smithi086.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-05-12T21:54:19.693 INFO:teuthology.orchestra.run.smithi086.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-05-12T21:54:19.693 INFO:teuthology.orchestra.run.smithi086.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-05-12T21:54:19.693 INFO:teuthology.orchestra.run.smithi086.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-05-12T21:54:19.693 INFO:teuthology.orchestra.run.smithi086.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-05-12T21:54:19.693 INFO:teuthology.orchestra.run.smithi086.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-05-12T21:54:19.693 INFO:teuthology.orchestra.run.smithi086.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-05-12T21:54:19.693 INFO:teuthology.orchestra.run.smithi086.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-05-12T21:54:19.693 INFO:teuthology.orchestra.run.smithi086.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-05-12T21:54:19.693 INFO:teuthology.orchestra.run.smithi086.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-05-12T21:54:19.694 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-05-12T21:54:19.694 INFO:teuthology.orchestra.run.smithi086.stdout: python3-ceph-argparse x86_64 2:19.0.0-2555.gf72fecff.el9 ceph 45 k 2024-05-12T21:54:19.694 INFO:teuthology.orchestra.run.smithi086.stdout: python3-ceph-common x86_64 2:19.0.0-2555.gf72fecff.el9 ceph 135 k 2024-05-12T21:54:19.694 INFO:teuthology.orchestra.run.smithi086.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-05-12T21:54:19.694 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-05-12T21:54:19.694 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-05-12T21:54:19.694 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-05-12T21:54:19.694 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-05-12T21:54:19.694 INFO:teuthology.orchestra.run.smithi086.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-05-12T21:54:19.694 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-05-12T21:54:19.694 INFO:teuthology.orchestra.run.smithi086.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-05-12T21:54:19.694 INFO:teuthology.orchestra.run.smithi086.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-05-12T21:54:19.694 INFO:teuthology.orchestra.run.smithi086.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-05-12T21:54:19.695 INFO:teuthology.orchestra.run.smithi086.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-05-12T21:54:19.695 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-05-12T21:54:19.695 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-05-12T21:54:19.695 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-05-12T21:54:19.695 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-05-12T21:54:19.695 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-05-12T21:54:19.695 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-05-12T21:54:19.695 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-05-12T21:54:19.695 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-05-12T21:54:19.695 INFO:teuthology.orchestra.run.smithi086.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-05-12T21:54:19.695 INFO:teuthology.orchestra.run.smithi086.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-05-12T21:54:19.695 INFO:teuthology.orchestra.run.smithi086.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-05-12T21:54:19.695 INFO:teuthology.orchestra.run.smithi086.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-05-12T21:54:19.695 INFO:teuthology.orchestra.run.smithi086.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-05-12T21:54:19.695 INFO:teuthology.orchestra.run.smithi086.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-05-12T21:54:19.696 INFO:teuthology.orchestra.run.smithi086.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-05-12T21:54:19.696 INFO:teuthology.orchestra.run.smithi086.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-05-12T21:54:19.696 INFO:teuthology.orchestra.run.smithi086.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-05-12T21:54:19.696 INFO:teuthology.orchestra.run.smithi086.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-05-12T21:54:19.696 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-05-12T21:54:19.696 INFO:teuthology.orchestra.run.smithi086.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-05-12T21:54:19.696 INFO:teuthology.orchestra.run.smithi086.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-05-12T21:54:19.696 INFO:teuthology.orchestra.run.smithi086.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-05-12T21:54:19.696 INFO:teuthology.orchestra.run.smithi086.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-05-12T21:54:19.696 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-05-12T21:54:19.697 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-05-12T21:54:19.697 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-05-12T21:54:19.697 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-05-12T21:54:19.697 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-05-12T21:54:19.697 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-05-12T21:54:19.697 INFO:teuthology.orchestra.run.smithi086.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-05-12T21:54:19.697 INFO:teuthology.orchestra.run.smithi086.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-05-12T21:54:19.697 INFO:teuthology.orchestra.run.smithi086.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-05-12T21:54:19.697 INFO:teuthology.orchestra.run.smithi086.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-05-12T21:54:19.697 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-05-12T21:54:19.697 INFO:teuthology.orchestra.run.smithi086.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-05-12T21:54:19.697 INFO:teuthology.orchestra.run.smithi086.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-05-12T21:54:19.697 INFO:teuthology.orchestra.run.smithi086.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-05-12T21:54:19.698 INFO:teuthology.orchestra.run.smithi086.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-05-12T21:54:19.698 INFO:teuthology.orchestra.run.smithi086.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-05-12T21:54:19.698 INFO:teuthology.orchestra.run.smithi086.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-05-12T21:54:19.698 INFO:teuthology.orchestra.run.smithi086.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-05-12T21:54:19.698 INFO:teuthology.orchestra.run.smithi086.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-05-12T21:54:19.698 INFO:teuthology.orchestra.run.smithi086.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-05-12T21:54:19.698 INFO:teuthology.orchestra.run.smithi086.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-05-12T21:54:19.698 INFO:teuthology.orchestra.run.smithi086.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-05-12T21:54:19.698 INFO:teuthology.orchestra.run.smithi086.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-05-12T21:54:19.698 INFO:teuthology.orchestra.run.smithi086.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-05-12T21:54:19.699 INFO:teuthology.orchestra.run.smithi086.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-05-12T21:54:19.699 INFO:teuthology.orchestra.run.smithi086.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-05-12T21:54:19.699 INFO:teuthology.orchestra.run.smithi086.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-05-12T21:54:19.699 INFO:teuthology.orchestra.run.smithi086.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-05-12T21:54:19.699 INFO:teuthology.orchestra.run.smithi086.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-05-12T21:54:19.699 INFO:teuthology.orchestra.run.smithi086.stdout:Installing weak dependencies: 2024-05-12T21:54:19.699 INFO:teuthology.orchestra.run.smithi086.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-05-12T21:54:19.699 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-05-12T21:54:19.699 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-05-12T21:54:19.699 INFO:teuthology.orchestra.run.smithi086.stdout:======================================================================================= 2024-05-12T21:54:19.699 INFO:teuthology.orchestra.run.smithi086.stdout:Install 148 Packages 2024-05-12T21:54:19.699 INFO:teuthology.orchestra.run.smithi086.stdout:Upgrade 8 Packages 2024-05-12T21:54:19.699 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-05-12T21:54:19.700 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 225 M 2024-05-12T21:54:19.700 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-05-12T21:54:19.860 INFO:teuthology.orchestra.run.smithi186.stdout:(4/156): ceph-base-19.0.0-2555.gf72fecff.el9.x8 5.1 MB/s | 5.4 MB 00:01 2024-05-12T21:54:19.919 INFO:teuthology.orchestra.run.smithi186.stdout:(5/156): ceph-mds-19.0.0-2555.gf72fecff.el9.x86 5.2 MB/s | 2.4 MB 00:00 2024-05-12T21:54:20.142 INFO:teuthology.orchestra.run.smithi028.stdout:lab-extras 28 kB/s | 1.7 kB 00:00 2024-05-12T21:54:20.384 INFO:teuthology.orchestra.run.smithi086.stdout:(1/156): ceph-19.0.0-2555.gf72fecff.el9.x86_64. 34 kB/s | 6.6 kB 00:00 2024-05-12T21:54:20.494 INFO:teuthology.orchestra.run.smithi186.stdout:(6/156): ceph-common-19.0.0-2555.gf72fecff.el9. 12 MB/s | 21 MB 00:01 2024-05-12T21:54:20.527 INFO:teuthology.orchestra.run.smithi186.stdout:(7/156): ceph-mgr-19.0.0-2555.gf72fecff.el9.x86 2.2 MB/s | 1.5 MB 00:00 2024-05-12T21:54:20.594 INFO:teuthology.orchestra.run.smithi186.stdout:(8/156): ceph-mon-19.0.0-2555.gf72fecff.el9.x86 6.9 MB/s | 4.7 MB 00:00 2024-05-12T21:54:20.711 INFO:teuthology.orchestra.run.smithi186.stdout:(9/156): ceph-selinux-19.0.0-2555.gf72fecff.el9 216 kB/s | 25 kB 00:00 2024-05-12T21:54:21.243 INFO:teuthology.orchestra.run.smithi086.stdout:(2/156): ceph-fuse-19.0.0-2555.gf72fecff.el9.x8 1.0 MB/s | 889 kB 00:00 2024-05-12T21:54:21.318 INFO:teuthology.orchestra.run.smithi086.stdout:(3/156): ceph-base-19.0.0-2555.gf72fecff.el9.x8 4.8 MB/s | 5.4 MB 00:01 2024-05-12T21:54:21.343 INFO:teuthology.orchestra.run.smithi086.stdout:(4/156): ceph-immutable-object-cache-19.0.0-255 1.4 MB/s | 145 kB 00:00 2024-05-12T21:54:21.768 INFO:teuthology.orchestra.run.smithi086.stdout:(5/156): ceph-mgr-19.0.0-2555.gf72fecff.el9.x86 3.5 MB/s | 1.5 MB 00:00 2024-05-12T21:54:21.852 INFO:teuthology.orchestra.run.smithi086.stdout:(6/156): ceph-mds-19.0.0-2555.gf72fecff.el9.x86 4.4 MB/s | 2.4 MB 00:00 2024-05-12T21:54:22.778 INFO:teuthology.orchestra.run.smithi028.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-05-12T21:54:22.780 INFO:teuthology.orchestra.run.smithi028.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-05-12T21:54:22.802 INFO:teuthology.orchestra.run.smithi028.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-05-12T21:54:22.804 INFO:teuthology.orchestra.run.smithi028.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-05-12T21:54:22.805 INFO:teuthology.orchestra.run.smithi028.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-05-12T21:54:22.806 INFO:teuthology.orchestra.run.smithi028.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-05-12T21:54:22.809 INFO:teuthology.orchestra.run.smithi028.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-05-12T21:54:22.812 INFO:teuthology.orchestra.run.smithi028.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-05-12T21:54:22.814 INFO:teuthology.orchestra.run.smithi028.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-05-12T21:54:22.890 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-05-12T21:54:22.906 INFO:teuthology.orchestra.run.smithi028.stdout:======================================================================================= 2024-05-12T21:54:22.906 INFO:teuthology.orchestra.run.smithi028.stdout: Package Arch Version Repository Size 2024-05-12T21:54:22.907 INFO:teuthology.orchestra.run.smithi028.stdout:======================================================================================= 2024-05-12T21:54:22.907 INFO:teuthology.orchestra.run.smithi028.stdout:Installing: 2024-05-12T21:54:22.907 INFO:teuthology.orchestra.run.smithi028.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-05-12T21:54:22.907 INFO:teuthology.orchestra.run.smithi028.stdout: ceph x86_64 2:19.0.0-2555.gf72fecff.el9 ceph 6.6 k 2024-05-12T21:54:22.907 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-base x86_64 2:19.0.0-2555.gf72fecff.el9 ceph 5.4 M 2024-05-12T21:54:22.907 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-fuse x86_64 2:19.0.0-2555.gf72fecff.el9 ceph 889 k 2024-05-12T21:54:22.907 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2555.gf72fecff.el9 ceph 145 k 2024-05-12T21:54:22.907 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr x86_64 2:19.0.0-2555.gf72fecff.el9 ceph 1.5 M 2024-05-12T21:54:22.907 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2555.gf72fecff.el9 ceph-noarch 142 k 2024-05-12T21:54:22.907 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2555.gf72fecff.el9 ceph-noarch 3.5 M 2024-05-12T21:54:22.907 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2555.gf72fecff.el9 ceph-noarch 7.4 M 2024-05-12T21:54:22.907 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-rook noarch 2:19.0.0-2555.gf72fecff.el9 ceph-noarch 49 k 2024-05-12T21:54:22.907 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-radosgw x86_64 2:19.0.0-2555.gf72fecff.el9 ceph 10 M 2024-05-12T21:54:22.907 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-test x86_64 2:19.0.0-2555.gf72fecff.el9 ceph 48 M 2024-05-12T21:54:22.907 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-volume noarch 2:19.0.0-2555.gf72fecff.el9 ceph-noarch 264 k 2024-05-12T21:54:22.908 INFO:teuthology.orchestra.run.smithi028.stdout: cephadm noarch 2:19.0.0-2555.gf72fecff.el9 ceph-noarch 763 k 2024-05-12T21:54:22.908 INFO:teuthology.orchestra.run.smithi028.stdout: cephfs-mirror x86_64 2:19.0.0-2555.gf72fecff.el9 ceph 221 k 2024-05-12T21:54:22.908 INFO:teuthology.orchestra.run.smithi028.stdout: cephfs-shell x86_64 2:19.0.0-2555.gf72fecff.el9 ceph 27 k 2024-05-12T21:54:22.908 INFO:teuthology.orchestra.run.smithi028.stdout: cephfs-top noarch 2:19.0.0-2555.gf72fecff.el9 ceph-noarch 18 k 2024-05-12T21:54:22.908 INFO:teuthology.orchestra.run.smithi028.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-05-12T21:54:22.908 INFO:teuthology.orchestra.run.smithi028.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-05-12T21:54:22.908 INFO:teuthology.orchestra.run.smithi028.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-05-12T21:54:22.908 INFO:teuthology.orchestra.run.smithi028.stdout: libcephfs-devel x86_64 2:19.0.0-2555.gf72fecff.el9 ceph 32 k 2024-05-12T21:54:22.908 INFO:teuthology.orchestra.run.smithi028.stdout: libcephfs2 x86_64 2:19.0.0-2555.gf72fecff.el9 ceph 745 k 2024-05-12T21:54:22.908 INFO:teuthology.orchestra.run.smithi028.stdout: librados-devel x86_64 2:19.0.0-2555.gf72fecff.el9 ceph 127 k 2024-05-12T21:54:22.908 INFO:teuthology.orchestra.run.smithi028.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-05-12T21:54:22.908 INFO:teuthology.orchestra.run.smithi028.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-05-12T21:54:22.908 INFO:teuthology.orchestra.run.smithi028.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-05-12T21:54:22.908 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-05-12T21:54:22.909 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-05-12T21:54:22.909 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-05-12T21:54:22.909 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cephfs x86_64 2:19.0.0-2555.gf72fecff.el9 ceph 161 k 2024-05-12T21:54:22.909 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rados x86_64 2:19.0.0-2555.gf72fecff.el9 ceph 321 k 2024-05-12T21:54:22.909 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rbd x86_64 2:19.0.0-2555.gf72fecff.el9 ceph 301 k 2024-05-12T21:54:22.909 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rgw x86_64 2:19.0.0-2555.gf72fecff.el9 ceph 99 k 2024-05-12T21:54:22.909 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-fuse x86_64 2:19.0.0-2555.gf72fecff.el9 ceph 85 k 2024-05-12T21:54:22.909 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-mirror x86_64 2:19.0.0-2555.gf72fecff.el9 ceph 3.1 M 2024-05-12T21:54:22.909 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-nbd x86_64 2:19.0.0-2555.gf72fecff.el9 ceph 171 k 2024-05-12T21:54:22.909 INFO:teuthology.orchestra.run.smithi028.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-05-12T21:54:22.909 INFO:teuthology.orchestra.run.smithi028.stdout:Upgrading: 2024-05-12T21:54:22.909 INFO:teuthology.orchestra.run.smithi028.stdout: NetworkManager x86_64 1:1.47.5-1.el9 baseos 2.3 M 2024-05-12T21:54:22.909 INFO:teuthology.orchestra.run.smithi028.stdout: NetworkManager-libnm x86_64 1:1.47.5-1.el9 baseos 1.8 M 2024-05-12T21:54:22.909 INFO:teuthology.orchestra.run.smithi028.stdout: NetworkManager-team x86_64 1:1.47.5-1.el9 baseos 42 k 2024-05-12T21:54:22.910 INFO:teuthology.orchestra.run.smithi028.stdout: NetworkManager-tui x86_64 1:1.47.5-1.el9 baseos 248 k 2024-05-12T21:54:22.910 INFO:teuthology.orchestra.run.smithi028.stdout: librados2 x86_64 2:19.0.0-2555.gf72fecff.el9 ceph 3.4 M 2024-05-12T21:54:22.910 INFO:teuthology.orchestra.run.smithi028.stdout: librbd1 x86_64 2:19.0.0-2555.gf72fecff.el9 ceph 3.1 M 2024-05-12T21:54:22.910 INFO:teuthology.orchestra.run.smithi028.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-05-12T21:54:22.910 INFO:teuthology.orchestra.run.smithi028.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-05-12T21:54:22.910 INFO:teuthology.orchestra.run.smithi028.stdout:Installing dependencies: 2024-05-12T21:54:22.910 INFO:teuthology.orchestra.run.smithi028.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-05-12T21:54:22.910 INFO:teuthology.orchestra.run.smithi028.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-05-12T21:54:22.910 INFO:teuthology.orchestra.run.smithi028.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-05-12T21:54:22.910 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-common x86_64 2:19.0.0-2555.gf72fecff.el9 ceph 21 M 2024-05-12T21:54:22.910 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2555.gf72fecff.el9 ceph-noarch 32 k 2024-05-12T21:54:22.910 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mds x86_64 2:19.0.0-2555.gf72fecff.el9 ceph 2.4 M 2024-05-12T21:54:22.910 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2555.gf72fecff.el9 ceph-noarch 249 k 2024-05-12T21:54:22.910 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mon x86_64 2:19.0.0-2555.gf72fecff.el9 ceph 4.7 M 2024-05-12T21:54:22.911 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-osd x86_64 2:19.0.0-2555.gf72fecff.el9 ceph 17 M 2024-05-12T21:54:22.911 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2555.gf72fecff.el9 ceph-noarch 16 k 2024-05-12T21:54:22.911 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-selinux x86_64 2:19.0.0-2555.gf72fecff.el9 ceph 25 k 2024-05-12T21:54:22.911 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-05-12T21:54:22.911 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-05-12T21:54:22.911 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-05-12T21:54:22.911 INFO:teuthology.orchestra.run.smithi028.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-05-12T21:54:22.911 INFO:teuthology.orchestra.run.smithi028.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-05-12T21:54:22.911 INFO:teuthology.orchestra.run.smithi028.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-05-12T21:54:22.911 INFO:teuthology.orchestra.run.smithi028.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-05-12T21:54:22.911 INFO:teuthology.orchestra.run.smithi028.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-05-12T21:54:22.911 INFO:teuthology.orchestra.run.smithi028.stdout: libcephsqlite x86_64 2:19.0.0-2555.gf72fecff.el9 ceph 163 k 2024-05-12T21:54:22.911 INFO:teuthology.orchestra.run.smithi028.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-05-12T21:54:22.911 INFO:teuthology.orchestra.run.smithi028.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-05-12T21:54:22.911 INFO:teuthology.orchestra.run.smithi028.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-05-12T21:54:22.911 INFO:teuthology.orchestra.run.smithi028.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-05-12T21:54:22.912 INFO:teuthology.orchestra.run.smithi028.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-05-12T21:54:22.912 INFO:teuthology.orchestra.run.smithi028.stdout: libradosstriper1 x86_64 2:19.0.0-2555.gf72fecff.el9 ceph 500 k 2024-05-12T21:54:22.912 INFO:teuthology.orchestra.run.smithi028.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-05-12T21:54:22.912 INFO:teuthology.orchestra.run.smithi028.stdout: librgw2 x86_64 2:19.0.0-2555.gf72fecff.el9 ceph 5.3 M 2024-05-12T21:54:22.912 INFO:teuthology.orchestra.run.smithi028.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-05-12T21:54:22.912 INFO:teuthology.orchestra.run.smithi028.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-05-12T21:54:22.912 INFO:teuthology.orchestra.run.smithi028.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-05-12T21:54:22.912 INFO:teuthology.orchestra.run.smithi028.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-05-12T21:54:22.912 INFO:teuthology.orchestra.run.smithi028.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-05-12T21:54:22.912 INFO:teuthology.orchestra.run.smithi028.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-05-12T21:54:22.912 INFO:teuthology.orchestra.run.smithi028.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-05-12T21:54:22.912 INFO:teuthology.orchestra.run.smithi028.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-05-12T21:54:22.912 INFO:teuthology.orchestra.run.smithi028.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-05-12T21:54:22.912 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-05-12T21:54:22.913 INFO:teuthology.orchestra.run.smithi028.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-05-12T21:54:22.913 INFO:teuthology.orchestra.run.smithi028.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-05-12T21:54:22.913 INFO:teuthology.orchestra.run.smithi028.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-05-12T21:54:22.913 INFO:teuthology.orchestra.run.smithi028.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-05-12T21:54:22.913 INFO:teuthology.orchestra.run.smithi028.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-05-12T21:54:22.913 INFO:teuthology.orchestra.run.smithi028.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-05-12T21:54:22.913 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-05-12T21:54:22.913 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ceph-argparse x86_64 2:19.0.0-2555.gf72fecff.el9 ceph 45 k 2024-05-12T21:54:22.913 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ceph-common x86_64 2:19.0.0-2555.gf72fecff.el9 ceph 135 k 2024-05-12T21:54:22.913 INFO:teuthology.orchestra.run.smithi028.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-05-12T21:54:22.913 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-05-12T21:54:22.913 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-05-12T21:54:22.913 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-05-12T21:54:22.913 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-05-12T21:54:22.913 INFO:teuthology.orchestra.run.smithi028.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-05-12T21:54:22.914 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-05-12T21:54:22.914 INFO:teuthology.orchestra.run.smithi028.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-05-12T21:54:22.914 INFO:teuthology.orchestra.run.smithi028.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-05-12T21:54:22.914 INFO:teuthology.orchestra.run.smithi028.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-05-12T21:54:22.914 INFO:teuthology.orchestra.run.smithi028.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-05-12T21:54:22.914 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-05-12T21:54:22.914 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-05-12T21:54:22.914 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-05-12T21:54:22.914 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-05-12T21:54:22.914 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-05-12T21:54:22.914 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-05-12T21:54:22.915 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-05-12T21:54:22.915 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-05-12T21:54:22.915 INFO:teuthology.orchestra.run.smithi028.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-05-12T21:54:22.915 INFO:teuthology.orchestra.run.smithi028.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-05-12T21:54:22.915 INFO:teuthology.orchestra.run.smithi028.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-05-12T21:54:22.915 INFO:teuthology.orchestra.run.smithi028.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-05-12T21:54:22.915 INFO:teuthology.orchestra.run.smithi028.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-05-12T21:54:22.915 INFO:teuthology.orchestra.run.smithi028.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-05-12T21:54:22.915 INFO:teuthology.orchestra.run.smithi028.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-05-12T21:54:22.915 INFO:teuthology.orchestra.run.smithi028.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-05-12T21:54:22.915 INFO:teuthology.orchestra.run.smithi028.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-05-12T21:54:22.915 INFO:teuthology.orchestra.run.smithi028.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-05-12T21:54:22.915 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-05-12T21:54:22.916 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-05-12T21:54:22.916 INFO:teuthology.orchestra.run.smithi028.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-05-12T21:54:22.916 INFO:teuthology.orchestra.run.smithi028.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-05-12T21:54:22.916 INFO:teuthology.orchestra.run.smithi028.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-05-12T21:54:22.916 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-05-12T21:54:22.916 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-05-12T21:54:22.916 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-05-12T21:54:22.916 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-05-12T21:54:22.916 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-05-12T21:54:22.916 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-05-12T21:54:22.916 INFO:teuthology.orchestra.run.smithi028.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-05-12T21:54:22.916 INFO:teuthology.orchestra.run.smithi028.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-05-12T21:54:22.916 INFO:teuthology.orchestra.run.smithi028.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-05-12T21:54:22.916 INFO:teuthology.orchestra.run.smithi028.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-05-12T21:54:22.916 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-05-12T21:54:22.917 INFO:teuthology.orchestra.run.smithi028.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-05-12T21:54:22.917 INFO:teuthology.orchestra.run.smithi028.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-05-12T21:54:22.917 INFO:teuthology.orchestra.run.smithi028.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-05-12T21:54:22.917 INFO:teuthology.orchestra.run.smithi028.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-05-12T21:54:22.917 INFO:teuthology.orchestra.run.smithi028.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-05-12T21:54:22.917 INFO:teuthology.orchestra.run.smithi028.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-05-12T21:54:22.917 INFO:teuthology.orchestra.run.smithi028.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-05-12T21:54:22.917 INFO:teuthology.orchestra.run.smithi028.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-05-12T21:54:22.917 INFO:teuthology.orchestra.run.smithi028.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-05-12T21:54:22.917 INFO:teuthology.orchestra.run.smithi028.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-05-12T21:54:22.917 INFO:teuthology.orchestra.run.smithi028.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-05-12T21:54:22.917 INFO:teuthology.orchestra.run.smithi028.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-05-12T21:54:22.917 INFO:teuthology.orchestra.run.smithi028.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-05-12T21:54:22.917 INFO:teuthology.orchestra.run.smithi028.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-05-12T21:54:22.917 INFO:teuthology.orchestra.run.smithi028.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-05-12T21:54:22.918 INFO:teuthology.orchestra.run.smithi028.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-05-12T21:54:22.918 INFO:teuthology.orchestra.run.smithi028.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-05-12T21:54:22.918 INFO:teuthology.orchestra.run.smithi028.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-05-12T21:54:22.918 INFO:teuthology.orchestra.run.smithi028.stdout:Installing weak dependencies: 2024-05-12T21:54:22.918 INFO:teuthology.orchestra.run.smithi028.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-05-12T21:54:22.918 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-12T21:54:22.918 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2024-05-12T21:54:22.918 INFO:teuthology.orchestra.run.smithi028.stdout:======================================================================================= 2024-05-12T21:54:22.918 INFO:teuthology.orchestra.run.smithi028.stdout:Install 148 Packages 2024-05-12T21:54:22.918 INFO:teuthology.orchestra.run.smithi028.stdout:Upgrade 8 Packages 2024-05-12T21:54:22.918 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-12T21:54:22.919 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 225 M 2024-05-12T21:54:22.919 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2024-05-12T21:54:23.195 INFO:teuthology.orchestra.run.smithi186.stdout:(10/156): ceph-osd-19.0.0-2555.gf72fecff.el9.x8 6.2 MB/s | 17 MB 00:02 2024-05-12T21:54:23.219 INFO:teuthology.orchestra.run.smithi086.stdout:(7/156): ceph-mon-19.0.0-2555.gf72fecff.el9.x86 3.2 MB/s | 4.7 MB 00:01 2024-05-12T21:54:23.312 INFO:teuthology.orchestra.run.smithi186.stdout:(11/156): cephfs-mirror-19.0.0-2555.gf72fecff.e 1.9 MB/s | 221 kB 00:00 2024-05-12T21:54:23.378 INFO:teuthology.orchestra.run.smithi186.stdout:(12/156): cephfs-shell-19.0.0-2555.gf72fecff.el 408 kB/s | 27 kB 00:00 2024-05-12T21:54:23.487 INFO:teuthology.orchestra.run.smithi186.stdout:(13/156): ceph-radosgw-19.0.0-2555.gf72fecff.el 3.5 MB/s | 10 MB 00:02 2024-05-12T21:54:23.504 INFO:teuthology.orchestra.run.smithi186.stdout:(14/156): libcephfs-devel-19.0.0-2555.gf72fecff 253 kB/s | 32 kB 00:00 2024-05-12T21:54:23.587 INFO:teuthology.orchestra.run.smithi186.stdout:(15/156): libcephsqlite-19.0.0-2555.gf72fecff.e 1.9 MB/s | 163 kB 00:00 2024-05-12T21:54:23.671 INFO:teuthology.orchestra.run.smithi186.stdout:(16/156): librados-devel-19.0.0-2555.gf72fecff. 1.5 MB/s | 127 kB 00:00 2024-05-12T21:54:23.730 INFO:teuthology.orchestra.run.smithi186.stdout:(17/156): libcephfs2-19.0.0-2555.gf72fecff.el9. 3.0 MB/s | 745 kB 00:00 2024-05-12T21:54:23.805 INFO:teuthology.orchestra.run.smithi186.stdout:(18/156): libradosstriper1-19.0.0-2555.gf72fecf 3.7 MB/s | 500 kB 00:00 2024-05-12T21:54:23.881 INFO:teuthology.orchestra.run.smithi186.stdout:(19/156): python3-ceph-argparse-19.0.0-2555.gf7 603 kB/s | 45 kB 00:00 2024-05-12T21:54:23.964 INFO:teuthology.orchestra.run.smithi186.stdout:(20/156): python3-ceph-common-19.0.0-2555.gf72f 1.6 MB/s | 135 kB 00:00 2024-05-12T21:54:24.048 INFO:teuthology.orchestra.run.smithi186.stdout:(21/156): python3-cephfs-19.0.0-2555.gf72fecff. 1.9 MB/s | 161 kB 00:00 2024-05-12T21:54:24.103 INFO:teuthology.orchestra.run.smithi028.stdout:(1/156): ceph-19.0.0-2555.gf72fecff.el9.x86_64. 15 kB/s | 6.6 kB 00:00 2024-05-12T21:54:24.140 INFO:teuthology.orchestra.run.smithi186.stdout:(22/156): python3-rados-19.0.0-2555.gf72fecff.e 3.4 MB/s | 321 kB 00:00 2024-05-12T21:54:24.240 INFO:teuthology.orchestra.run.smithi186.stdout:(23/156): python3-rbd-19.0.0-2555.gf72fecff.el9 2.9 MB/s | 301 kB 00:00 2024-05-12T21:54:24.332 INFO:teuthology.orchestra.run.smithi186.stdout:(24/156): python3-rgw-19.0.0-2555.gf72fecff.el9 1.1 MB/s | 99 kB 00:00 2024-05-12T21:54:24.424 INFO:teuthology.orchestra.run.smithi186.stdout:(25/156): rbd-fuse-19.0.0-2555.gf72fecff.el9.x8 931 kB/s | 85 kB 00:00 2024-05-12T21:54:24.852 INFO:teuthology.orchestra.run.smithi086.stdout:(8/156): ceph-osd-19.0.0-2555.gf72fecff.el9.x86 5.6 MB/s | 17 MB 00:02 2024-05-12T21:54:24.999 INFO:teuthology.orchestra.run.smithi186.stdout:(26/156): rbd-mirror-19.0.0-2555.gf72fecff.el9. 5.4 MB/s | 3.1 MB 00:00 2024-05-12T21:54:25.049 INFO:teuthology.orchestra.run.smithi186.stdout:(27/156): rbd-nbd-19.0.0-2555.gf72fecff.el9.x86 3.4 MB/s | 171 kB 00:00 2024-05-12T21:54:25.061 INFO:teuthology.orchestra.run.smithi086.stdout:(9/156): ceph-common-19.0.0-2555.gf72fecff.el9. 4.3 MB/s | 21 MB 00:04 2024-05-12T21:54:25.086 INFO:teuthology.orchestra.run.smithi086.stdout:(10/156): ceph-selinux-19.0.0-2555.gf72fecff.el 108 kB/s | 25 kB 00:00 2024-05-12T21:54:25.108 INFO:teuthology.orchestra.run.smithi186.stdout:(28/156): ceph-grafana-dashboards-19.0.0-2555.g 544 kB/s | 32 kB 00:00 2024-05-12T21:54:25.145 INFO:teuthology.orchestra.run.smithi086.stdout:(11/156): cephfs-mirror-19.0.0-2555.gf72fecff.e 3.7 MB/s | 221 kB 00:00 2024-05-12T21:54:25.161 INFO:teuthology.orchestra.run.smithi028.stdout:(2/156): ceph-fuse-19.0.0-2555.gf72fecff.el9.x8 841 kB/s | 889 kB 00:01 2024-05-12T21:54:25.212 INFO:teuthology.orchestra.run.smithi086.stdout:(12/156): cephfs-shell-19.0.0-2555.gf72fecff.el 407 kB/s | 27 kB 00:00 2024-05-12T21:54:25.225 INFO:teuthology.orchestra.run.smithi186.stdout:(29/156): ceph-mgr-cephadm-19.0.0-2555.gf72fecf 1.2 MB/s | 142 kB 00:00 2024-05-12T21:54:25.278 INFO:teuthology.orchestra.run.smithi086.stdout:(13/156): libcephfs-devel-19.0.0-2555.gf72fecff 477 kB/s | 32 kB 00:00 2024-05-12T21:54:25.361 INFO:teuthology.orchestra.run.smithi028.stdout:(3/156): ceph-immutable-object-cache-19.0.0-255 727 kB/s | 145 kB 00:00 2024-05-12T21:54:25.476 INFO:teuthology.orchestra.run.smithi186.stdout:(30/156): librgw2-19.0.0-2555.gf72fecff.el9.x86 3.0 MB/s | 5.3 MB 00:01 2024-05-12T21:54:25.504 INFO:teuthology.orchestra.run.smithi086.stdout:(14/156): libcephfs2-19.0.0-2555.gf72fecff.el9. 3.2 MB/s | 745 kB 00:00 2024-05-12T21:54:25.596 INFO:teuthology.orchestra.run.smithi086.stdout:(15/156): libcephsqlite-19.0.0-2555.gf72fecff.e 1.7 MB/s | 163 kB 00:00 2024-05-12T21:54:25.679 INFO:teuthology.orchestra.run.smithi086.stdout:(16/156): librados-devel-19.0.0-2555.gf72fecff. 1.5 MB/s | 127 kB 00:00 2024-05-12T21:54:25.838 INFO:teuthology.orchestra.run.smithi086.stdout:(17/156): libradosstriper1-19.0.0-2555.gf72fecf 3.1 MB/s | 500 kB 00:00 2024-05-12T21:54:25.892 INFO:teuthology.orchestra.run.smithi186.stdout:(31/156): ceph-mgr-dashboard-19.0.0-2555.gf72fe 5.3 MB/s | 3.5 MB 00:00 2024-05-12T21:54:25.984 INFO:teuthology.orchestra.run.smithi186.stdout:(32/156): ceph-mgr-modules-core-19.0.0-2555.gf7 2.7 MB/s | 249 kB 00:00 2024-05-12T21:54:26.005 INFO:teuthology.orchestra.run.smithi086.stdout:(18/156): ceph-radosgw-19.0.0-2555.gf72fecff.el 3.7 MB/s | 10 MB 00:02 2024-05-12T21:54:26.051 INFO:teuthology.orchestra.run.smithi186.stdout:(33/156): ceph-mgr-rook-19.0.0-2555.gf72fecff.e 741 kB/s | 49 kB 00:00 2024-05-12T21:54:26.080 INFO:teuthology.orchestra.run.smithi086.stdout:(19/156): python3-ceph-argparse-19.0.0-2555.gf7 605 kB/s | 45 kB 00:00 2024-05-12T21:54:26.118 INFO:teuthology.orchestra.run.smithi186.stdout:(34/156): ceph-prometheus-alerts-19.0.0-2555.gf 247 kB/s | 16 kB 00:00 2024-05-12T21:54:26.119 INFO:teuthology.orchestra.run.smithi028.stdout:(4/156): ceph-base-19.0.0-2555.gf72fecff.el9.x8 2.2 MB/s | 5.4 MB 00:02 2024-05-12T21:54:26.164 INFO:teuthology.orchestra.run.smithi086.stdout:(20/156): python3-ceph-common-19.0.0-2555.gf72f 1.6 MB/s | 135 kB 00:00 2024-05-12T21:54:26.210 INFO:teuthology.orchestra.run.smithi186.stdout:(35/156): ceph-volume-19.0.0-2555.gf72fecff.el9 2.8 MB/s | 264 kB 00:00 2024-05-12T21:54:26.248 INFO:teuthology.orchestra.run.smithi086.stdout:(21/156): python3-cephfs-19.0.0-2555.gf72fecff. 1.9 MB/s | 161 kB 00:00 2024-05-12T21:54:26.394 INFO:teuthology.orchestra.run.smithi186.stdout:(36/156): cephadm-19.0.0-2555.gf72fecff.el9.noa 4.1 MB/s | 763 kB 00:00 2024-05-12T21:54:26.415 INFO:teuthology.orchestra.run.smithi086.stdout:(22/156): python3-rados-19.0.0-2555.gf72fecff.e 1.9 MB/s | 321 kB 00:00 2024-05-12T21:54:26.494 INFO:teuthology.orchestra.run.smithi186.stdout:(37/156): cephfs-top-19.0.0-2555.gf72fecff.el9. 183 kB/s | 18 kB 00:00 2024-05-12T21:54:26.548 INFO:teuthology.orchestra.run.smithi086.stdout:(23/156): python3-rbd-19.0.0-2555.gf72fecff.el9 2.2 MB/s | 301 kB 00:00 2024-05-12T21:54:26.552 INFO:teuthology.orchestra.run.smithi186.stdout:(38/156): fuse-2.9.9-15.el9.x86_64.rpm 1.3 MB/s | 80 kB 00:00 2024-05-12T21:54:26.615 INFO:teuthology.orchestra.run.smithi086.stdout:(24/156): python3-rgw-19.0.0-2555.gf72fecff.el9 1.5 MB/s | 99 kB 00:00 2024-05-12T21:54:26.653 INFO:teuthology.orchestra.run.smithi186.stdout:(39/156): libgfortran-11.4.1-3.el9.x86_64.rpm 7.9 MB/s | 803 kB 00:00 2024-05-12T21:54:26.690 INFO:teuthology.orchestra.run.smithi086.stdout:(25/156): rbd-fuse-19.0.0-2555.gf72fecff.el9.x8 1.1 MB/s | 85 kB 00:00 2024-05-12T21:54:26.720 INFO:teuthology.orchestra.run.smithi186.stdout:(40/156): libquadmath-11.4.1-3.el9.x86_64.rpm 2.9 MB/s | 195 kB 00:00 2024-05-12T21:54:26.728 INFO:teuthology.orchestra.run.smithi028.stdout:(5/156): ceph-mgr-19.0.0-2555.gf72fecff.el9.x86 2.5 MB/s | 1.5 MB 00:00 2024-05-12T21:54:26.795 INFO:teuthology.orchestra.run.smithi186.stdout:(41/156): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.3 MB/s | 253 kB 00:00 2024-05-12T21:54:26.807 INFO:teuthology.orchestra.run.smithi086.stdout:(26/156): librgw2-19.0.0-2555.gf72fecff.el9.x86 5.5 MB/s | 5.3 MB 00:00 2024-05-12T21:54:26.891 INFO:teuthology.orchestra.run.smithi086.stdout:(27/156): rbd-nbd-19.0.0-2555.gf72fecff.el9.x86 2.0 MB/s | 171 kB 00:00 2024-05-12T21:54:26.895 INFO:teuthology.orchestra.run.smithi186.stdout:(42/156): python3-cryptography-36.0.1-4.el9.x86 12 MB/s | 1.2 MB 00:00 2024-05-12T21:54:26.958 INFO:teuthology.orchestra.run.smithi086.stdout:(28/156): ceph-grafana-dashboards-19.0.0-2555.g 476 kB/s | 32 kB 00:00 2024-05-12T21:54:26.962 INFO:teuthology.orchestra.run.smithi186.stdout:(43/156): python3-ply-3.11-14.el9.noarch.rpm 1.6 MB/s | 106 kB 00:00 2024-05-12T21:54:27.030 INFO:teuthology.orchestra.run.smithi186.stdout:(44/156): python3-pycparser-2.20-6.el9.noarch.r 2.0 MB/s | 135 kB 00:00 2024-05-12T21:54:27.042 INFO:teuthology.orchestra.run.smithi086.stdout:(29/156): ceph-mgr-cephadm-19.0.0-2555.gf72fecf 1.7 MB/s | 142 kB 00:00 2024-05-12T21:54:27.088 INFO:teuthology.orchestra.run.smithi186.stdout:(45/156): python3-requests-2.25.1-8.el9.noarch. 2.1 MB/s | 125 kB 00:00 2024-05-12T21:54:27.163 INFO:teuthology.orchestra.run.smithi186.stdout:(46/156): python3-urllib3-1.26.5-5.el9.noarch.r 2.8 MB/s | 215 kB 00:00 2024-05-12T21:54:27.364 INFO:teuthology.orchestra.run.smithi186.stdout:(47/156): automake-1.16.2-8.el9.noarch.rpm 3.4 MB/s | 689 kB 00:00 2024-05-12T21:54:27.497 INFO:teuthology.orchestra.run.smithi186.stdout:(48/156): bison-3.7.4-5.el9.x86_64.rpm 6.9 MB/s | 943 kB 00:00 2024-05-12T21:54:27.539 INFO:teuthology.orchestra.run.smithi186.stdout:(49/156): boost-program-options-1.75.0-8.el9.x8 2.5 MB/s | 107 kB 00:00 2024-05-12T21:54:27.581 INFO:teuthology.orchestra.run.smithi186.stdout:(50/156): elfutils-libelf-devel-0.190-2.el9.x86 519 kB/s | 21 kB 00:00 2024-05-12T21:54:27.634 INFO:teuthology.orchestra.run.smithi086.stdout:(30/156): rbd-mirror-19.0.0-2555.gf72fecff.el9. 3.3 MB/s | 3.1 MB 00:00 2024-05-12T21:54:27.640 INFO:teuthology.orchestra.run.smithi186.stdout:(51/156): flex-2.6.4-9.el9.x86_64.rpm 5.3 MB/s | 313 kB 00:00 2024-05-12T21:54:27.673 INFO:teuthology.orchestra.run.smithi186.stdout:(52/156): flexiblas-3.0.4-8.el9.x86_64.rpm 983 kB/s | 33 kB 00:00 2024-05-12T21:54:27.695 INFO:teuthology.orchestra.run.smithi028.stdout:(6/156): ceph-mds-19.0.0-2555.gf72fecff.el9.x86 1.0 MB/s | 2.4 MB 00:02 2024-05-12T21:54:27.717 INFO:teuthology.orchestra.run.smithi086.stdout:(31/156): ceph-mgr-dashboard-19.0.0-2555.gf72fe 5.2 MB/s | 3.5 MB 00:00 2024-05-12T21:54:27.810 INFO:teuthology.orchestra.run.smithi086.stdout:(32/156): ceph-mgr-modules-core-19.0.0-2555.gf7 2.7 MB/s | 249 kB 00:00 2024-05-12T21:54:27.901 INFO:teuthology.orchestra.run.smithi086.stdout:(33/156): ceph-mgr-rook-19.0.0-2555.gf72fecff.e 538 kB/s | 49 kB 00:00 2024-05-12T21:54:27.915 INFO:teuthology.orchestra.run.smithi186.stdout:(53/156): flexiblas-netlib-3.0.4-8.el9.x86_64.r 12 MB/s | 3.0 MB 00:00 2024-05-12T21:54:27.959 INFO:teuthology.orchestra.run.smithi086.stdout:(34/156): ceph-prometheus-alerts-19.0.0-2555.gf 283 kB/s | 16 kB 00:00 2024-05-12T21:54:27.999 INFO:teuthology.orchestra.run.smithi186.stdout:(54/156): ceph-mgr-diskprediction-local-19.0.0- 2.9 MB/s | 7.4 MB 00:02 2024-05-12T21:54:28.016 INFO:teuthology.orchestra.run.smithi186.stdout:(55/156): flexiblas-openblas-openmp-3.0.4-8.el9 181 kB/s | 18 kB 00:00 2024-05-12T21:54:28.026 INFO:teuthology.orchestra.run.smithi086.stdout:(35/156): ceph-volume-19.0.0-2555.gf72fecff.el9 3.9 MB/s | 264 kB 00:00 2024-05-12T21:54:28.058 INFO:teuthology.orchestra.run.smithi186.stdout:(56/156): libaio-devel-0.3.111-13.el9.x86_64.rp 190 kB/s | 11 kB 00:00 2024-05-12T21:54:28.083 INFO:teuthology.orchestra.run.smithi186.stdout:(57/156): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.3 MB/s | 160 kB 00:00 2024-05-12T21:54:28.125 INFO:teuthology.orchestra.run.smithi186.stdout:(58/156): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.1 MB/s | 45 kB 00:00 2024-05-12T21:54:28.192 INFO:teuthology.orchestra.run.smithi186.stdout:(59/156): libtool-2.4.6-45.el9.x86_64.rpm 8.5 MB/s | 582 kB 00:00 2024-05-12T21:54:28.210 INFO:teuthology.orchestra.run.smithi086.stdout:(36/156): cephadm-19.0.0-2555.gf72fecff.el9.noa 4.1 MB/s | 763 kB 00:00 2024-05-12T21:54:28.217 INFO:teuthology.orchestra.run.smithi186.stdout:(60/156): librdkafka-1.6.1-102.el9.x86_64.rpm 4.8 MB/s | 662 kB 00:00 2024-05-12T21:54:28.242 INFO:teuthology.orchestra.run.smithi186.stdout:(61/156): libuuid-devel-2.37.4-18.el9.x86_64.rp 506 kB/s | 25 kB 00:00 2024-05-12T21:54:28.260 INFO:teuthology.orchestra.run.smithi086.stdout:(37/156): cephfs-top-19.0.0-2555.gf72fecff.el9. 365 kB/s | 18 kB 00:00 2024-05-12T21:54:28.268 INFO:teuthology.orchestra.run.smithi186.stdout:(62/156): libzstd-devel-1.5.1-2.el9.x86_64.rpm 947 kB/s | 47 kB 00:00 2024-05-12T21:54:28.287 INFO:teuthology.orchestra.run.smithi028.stdout:(7/156): ceph-mon-19.0.0-2555.gf72fecff.el9.x86 3.0 MB/s | 4.7 MB 00:01 2024-05-12T21:54:28.293 INFO:teuthology.orchestra.run.smithi186.stdout:(63/156): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.7 MB/s | 292 kB 00:00 2024-05-12T21:54:28.318 INFO:teuthology.orchestra.run.smithi186.stdout:(64/156): lua-5.4.4-4.el9.x86_64.rpm 3.6 MB/s | 188 kB 00:00 2024-05-12T21:54:28.344 INFO:teuthology.orchestra.run.smithi186.stdout:(65/156): openblas-0.3.21-2.el9.x86_64.rpm 699 kB/s | 35 kB 00:00 2024-05-12T21:54:28.403 INFO:teuthology.orchestra.run.smithi086.stdout:(38/156): fuse-2.9.9-15.el9.x86_64.rpm 568 kB/s | 80 kB 00:00 2024-05-12T21:54:28.611 INFO:teuthology.orchestra.run.smithi086.stdout:(39/156): libgfortran-11.4.1-3.el9.x86_64.rpm 3.8 MB/s | 803 kB 00:00 2024-05-12T21:54:28.695 INFO:teuthology.orchestra.run.smithi086.stdout:(40/156): libquadmath-11.4.1-3.el9.x86_64.rpm 2.3 MB/s | 195 kB 00:00 2024-05-12T21:54:28.761 INFO:teuthology.orchestra.run.smithi186.stdout:(66/156): ceph-test-19.0.0-2555.gf72fecff.el9.x 6.0 MB/s | 48 MB 00:08 2024-05-12T21:54:28.778 INFO:teuthology.orchestra.run.smithi086.stdout:(41/156): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.0 MB/s | 253 kB 00:00 2024-05-12T21:54:28.869 INFO:teuthology.orchestra.run.smithi186.stdout:(67/156): openblas-openmp-0.3.21-2.el9.x86_64.r 8.6 MB/s | 4.7 MB 00:00 2024-05-12T21:54:28.936 INFO:teuthology.orchestra.run.smithi186.stdout:(68/156): postgresql-contrib-13.14-1.el9.x86_64 13 MB/s | 911 kB 00:00 2024-05-12T21:54:28.978 INFO:teuthology.orchestra.run.smithi186.stdout:(69/156): postgresql-private-libs-13.14-1.el9.x 3.1 MB/s | 133 kB 00:00 2024-05-12T21:54:29.045 INFO:teuthology.orchestra.run.smithi186.stdout:(70/156): postgresql-13.14-1.el9.x86_64.rpm 5.6 MB/s | 1.6 MB 00:00 2024-05-12T21:54:29.062 INFO:teuthology.orchestra.run.smithi086.stdout:(42/156): ceph-mgr-diskprediction-local-19.0.0- 5.2 MB/s | 7.4 MB 00:01 2024-05-12T21:54:29.096 INFO:teuthology.orchestra.run.smithi086.stdout:(43/156): python3-cryptography-36.0.1-4.el9.x86 3.9 MB/s | 1.2 MB 00:00 2024-05-12T21:54:29.103 INFO:teuthology.orchestra.run.smithi186.stdout:(71/156): openssl-devel-3.2.1-1.el9.x86_64.rpm 5.8 MB/s | 4.4 MB 00:00 2024-05-12T21:54:29.137 INFO:teuthology.orchestra.run.smithi186.stdout:(72/156): protobuf-3.14.0-13.el9.x86_64.rpm 11 MB/s | 1.0 MB 00:00 2024-05-12T21:54:29.163 INFO:teuthology.orchestra.run.smithi186.stdout:(73/156): python3-attrs-20.3.0-7.el9.noarch.rpm 1.5 MB/s | 87 kB 00:00 2024-05-12T21:54:29.180 INFO:teuthology.orchestra.run.smithi086.stdout:(44/156): python3-pycparser-2.20-6.el9.noarch.r 1.6 MB/s | 135 kB 00:00 2024-05-12T21:54:29.213 INFO:teuthology.orchestra.run.smithi186.stdout:(74/156): python3-devel-3.9.18-3.el9.x86_64.rpm 4.8 MB/s | 244 kB 00:00 2024-05-12T21:54:29.221 INFO:teuthology.orchestra.run.smithi086.stdout:(45/156): python3-requests-2.25.1-8.el9.noarch. 2.9 MB/s | 125 kB 00:00 2024-05-12T21:54:29.238 INFO:teuthology.orchestra.run.smithi086.stdout:(46/156): python3-ply-3.11-14.el9.noarch.rpm 606 kB/s | 106 kB 00:00 2024-05-12T21:54:29.280 INFO:teuthology.orchestra.run.smithi186.stdout:(75/156): postgresql-server-13.14-1.el9.x86_64. 19 MB/s | 5.8 MB 00:00 2024-05-12T21:54:29.305 INFO:teuthology.orchestra.run.smithi186.stdout:(76/156): python3-jinja2-2.11.3-5.el9.noarch.rp 2.6 MB/s | 248 kB 00:00 2024-05-12T21:54:29.305 INFO:teuthology.orchestra.run.smithi086.stdout:(47/156): python3-urllib3-1.26.5-5.el9.noarch.r 2.5 MB/s | 215 kB 00:00 2024-05-12T21:54:29.330 INFO:teuthology.orchestra.run.smithi186.stdout:(77/156): python3-jsonpatch-1.21-16.el9.noarch. 510 kB/s | 26 kB 00:00 2024-05-12T21:54:29.356 INFO:teuthology.orchestra.run.smithi186.stdout:(78/156): python3-jsonpointer-2.0-4.el9.noarch. 380 kB/s | 19 kB 00:00 2024-05-12T21:54:29.381 INFO:teuthology.orchestra.run.smithi186.stdout:(79/156): python3-mako-1.1.4-6.el9.noarch.rpm 3.3 MB/s | 172 kB 00:00 2024-05-12T21:54:29.407 INFO:teuthology.orchestra.run.smithi186.stdout:(80/156): python3-markupsafe-1.1.1-12.el9.x86_6 694 kB/s | 35 kB 00:00 2024-05-12T21:54:29.456 INFO:teuthology.orchestra.run.smithi086.stdout:(48/156): automake-1.16.2-8.el9.noarch.rpm 3.1 MB/s | 689 kB 00:00 2024-05-12T21:54:29.490 INFO:teuthology.orchestra.run.smithi186.stdout:(81/156): python3-babel-2.9.1-2.el9.noarch.rpm 17 MB/s | 6.0 MB 00:00 2024-05-12T21:54:29.506 INFO:teuthology.orchestra.run.smithi086.stdout:(49/156): boost-program-options-1.75.0-8.el9.x8 2.1 MB/s | 107 kB 00:00 2024-05-12T21:54:29.515 INFO:teuthology.orchestra.run.smithi186.stdout:(82/156): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.3 MB/s | 250 kB 00:00 2024-05-12T21:54:29.540 INFO:teuthology.orchestra.run.smithi186.stdout:(83/156): python3-oauthlib-3.1.1-5.el9.noarch.r 4.3 MB/s | 222 kB 00:00 2024-05-12T21:54:29.556 INFO:teuthology.orchestra.run.smithi086.stdout:(50/156): elfutils-libelf-devel-0.190-2.el9.x86 432 kB/s | 21 kB 00:00 2024-05-12T21:54:29.566 INFO:teuthology.orchestra.run.smithi186.stdout:(84/156): python3-packaging-20.9-5.el9.noarch.r 1.5 MB/s | 77 kB 00:00 2024-05-12T21:54:29.591 INFO:teuthology.orchestra.run.smithi186.stdout:(85/156): python3-prettytable-0.7.2-27.el9.noar 839 kB/s | 42 kB 00:00 2024-05-12T21:54:29.616 INFO:teuthology.orchestra.run.smithi186.stdout:(86/156): python3-protobuf-3.14.0-13.el9.noarch 5.2 MB/s | 269 kB 00:00 2024-05-12T21:54:29.632 INFO:teuthology.orchestra.run.smithi086.stdout:(51/156): flex-2.6.4-9.el9.x86_64.rpm 4.1 MB/s | 313 kB 00:00 2024-05-12T21:54:29.642 INFO:teuthology.orchestra.run.smithi186.stdout:(87/156): python3-pyasn1-0.4.8-6.el9.noarch.rpm 3.1 MB/s | 159 kB 00:00 2024-05-12T21:54:29.665 INFO:teuthology.orchestra.run.smithi086.stdout:(52/156): bison-3.7.4-5.el9.x86_64.rpm 2.6 MB/s | 943 kB 00:00 2024-05-12T21:54:29.667 INFO:teuthology.orchestra.run.smithi186.stdout:(88/156): python3-pyasn1-modules-0.4.8-6.el9.no 5.4 MB/s | 279 kB 00:00 2024-05-12T21:54:29.684 INFO:teuthology.orchestra.run.smithi186.stdout:(89/156): python3-pytz-2021.1-5.el9.noarch.rpm 1.2 MB/s | 51 kB 00:00 2024-05-12T21:54:29.690 INFO:teuthology.orchestra.run.smithi086.stdout:(53/156): flexiblas-3.0.4-8.el9.x86_64.rpm 560 kB/s | 33 kB 00:00 2024-05-12T21:54:29.741 INFO:teuthology.orchestra.run.smithi086.stdout:(54/156): flexiblas-openblas-openmp-3.0.4-8.el9 365 kB/s | 18 kB 00:00 2024-05-12T21:54:29.751 INFO:teuthology.orchestra.run.smithi186.stdout:(90/156): python3-numpy-1.20.1-5.el9.x86_64.rpm 14 MB/s | 5.1 MB 00:00 2024-05-12T21:54:29.776 INFO:teuthology.orchestra.run.smithi186.stdout:(91/156): python3-requests-oauthlib-1.3.0-12.el 492 kB/s | 54 kB 00:00 2024-05-12T21:54:29.799 INFO:teuthology.orchestra.run.smithi086.stdout:(55/156): libaio-devel-0.3.111-13.el9.x86_64.rp 191 kB/s | 11 kB 00:00 2024-05-12T21:54:29.810 INFO:teuthology.orchestra.run.smithi186.stdout:(92/156): python3-toml-0.10.2-6.el9.noarch.rpm 712 kB/s | 42 kB 00:00 2024-05-12T21:54:29.835 INFO:teuthology.orchestra.run.smithi186.stdout:(93/156): python3-wcwidth-0.2.5-8.el9.noarch.rp 753 kB/s | 44 kB 00:00 2024-05-12T21:54:29.846 INFO:teuthology.orchestra.run.smithi028.stdout:(8/156): ceph-radosgw-19.0.0-2555.gf72fecff.el9 6.6 MB/s | 10 MB 00:01 2024-05-12T21:54:29.850 INFO:teuthology.orchestra.run.smithi086.stdout:(56/156): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 160 kB 00:00 2024-05-12T21:54:29.860 INFO:teuthology.orchestra.run.smithi186.stdout:(94/156): qatlib-24.02.0-1.el9.x86_64.rpm 4.3 MB/s | 221 kB 00:00 2024-05-12T21:54:29.894 INFO:teuthology.orchestra.run.smithi186.stdout:(95/156): qatlib-service-24.02.0-1.el9.x86_64.r 612 kB/s | 36 kB 00:00 2024-05-12T21:54:29.896 INFO:teuthology.orchestra.run.smithi028.stdout:(9/156): ceph-selinux-19.0.0-2555.gf72fecff.el9 513 kB/s | 25 kB 00:00 2024-05-12T21:54:29.917 INFO:teuthology.orchestra.run.smithi086.stdout:(57/156): librabbitmq-0.11.0-7.el9.x86_64.rpm 680 kB/s | 45 kB 00:00 2024-05-12T21:54:29.919 INFO:teuthology.orchestra.run.smithi186.stdout:(96/156): qatzip-libs-1.2.0-1.el9.x86_64.rpm 809 kB/s | 47 kB 00:00 2024-05-12T21:54:29.961 INFO:teuthology.orchestra.run.smithi186.stdout:(97/156): socat-1.7.4.1-5.el9.x86_64.rpm 4.4 MB/s | 305 kB 00:00 2024-05-12T21:54:30.003 INFO:teuthology.orchestra.run.smithi186.stdout:(98/156): uuid-1.6.2-55.el9.x86_64.rpm 691 kB/s | 58 kB 00:00 2024-05-12T21:54:30.028 INFO:teuthology.orchestra.run.smithi186.stdout:(99/156): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.4 MB/s | 94 kB 00:00 2024-05-12T21:54:30.059 INFO:teuthology.orchestra.run.smithi086.stdout:(58/156): librdkafka-1.6.1-102.el9.x86_64.rpm 4.6 MB/s | 662 kB 00:00 2024-05-12T21:54:30.062 INFO:teuthology.orchestra.run.smithi186.stdout:(100/156): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.1 MB/s | 64 kB 00:00 2024-05-12T21:54:30.079 INFO:teuthology.orchestra.run.smithi186.stdout:(101/156): zlib-devel-1.2.11-41.el9.x86_64.rpm 893 kB/s | 45 kB 00:00 2024-05-12T21:54:30.175 INFO:teuthology.orchestra.run.smithi086.stdout:(59/156): libtool-2.4.6-45.el9.x86_64.rpm 4.9 MB/s | 582 kB 00:00 2024-05-12T21:54:30.226 INFO:teuthology.orchestra.run.smithi086.stdout:(60/156): libuuid-devel-2.37.4-18.el9.x86_64.rp 509 kB/s | 25 kB 00:00 2024-05-12T21:54:30.276 INFO:teuthology.orchestra.run.smithi086.stdout:(61/156): libzstd-devel-1.5.1-2.el9.x86_64.rpm 952 kB/s | 47 kB 00:00 2024-05-12T21:54:30.279 INFO:teuthology.orchestra.run.smithi186.stdout:(102/156): python3-scipy-1.6.2-8.el9.x86_64.rpm 27 MB/s | 16 MB 00:00 2024-05-12T21:54:30.326 INFO:teuthology.orchestra.run.smithi086.stdout:(62/156): flexiblas-netlib-3.0.4-8.el9.x86_64.r 4.5 MB/s | 3.0 MB 00:00 2024-05-12T21:54:30.352 INFO:teuthology.orchestra.run.smithi086.stdout:(63/156): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.8 MB/s | 292 kB 00:00 2024-05-12T21:54:30.363 INFO:teuthology.orchestra.run.smithi186.stdout:(103/156): lua-devel-5.4.4-4.el9.x86_64.rpm 74 kB/s | 22 kB 00:00 2024-05-12T21:54:30.377 INFO:teuthology.orchestra.run.smithi086.stdout:(64/156): lua-5.4.4-4.el9.x86_64.rpm 3.7 MB/s | 188 kB 00:00 2024-05-12T21:54:30.402 INFO:teuthology.orchestra.run.smithi086.stdout:(65/156): openblas-0.3.21-2.el9.x86_64.rpm 700 kB/s | 35 kB 00:00 2024-05-12T21:54:30.447 INFO:teuthology.orchestra.run.smithi186.stdout:(104/156): abseil-cpp-20211102.0-3.el9.x86_64.r 3.2 MB/s | 548 kB 00:00 2024-05-12T21:54:30.489 INFO:teuthology.orchestra.run.smithi186.stdout:(105/156): grpc-data-1.46.7-10.el9.noarch.rpm 466 kB/s | 19 kB 00:00 2024-05-12T21:54:30.530 INFO:teuthology.orchestra.run.smithi186.stdout:(106/156): gperftools-libs-2.9.1-2.el9.x86_64.r 1.8 MB/s | 309 kB 00:00 2024-05-12T21:54:30.572 INFO:teuthology.orchestra.run.smithi186.stdout:(107/156): libarrow-doc-9.0.0-11.el9.noarch.rpm 625 kB/s | 26 kB 00:00 2024-05-12T21:54:30.648 INFO:teuthology.orchestra.run.smithi186.stdout:(108/156): libarrow-9.0.0-11.el9.x86_64.rpm 28 MB/s | 4.4 MB 00:00 2024-05-12T21:54:30.725 INFO:teuthology.orchestra.run.smithi186.stdout:(109/156): protobuf-compiler-3.14.0-13.el9.x86_ 1.3 MB/s | 863 kB 00:00 2024-05-12T21:54:30.728 INFO:teuthology.orchestra.run.smithi086.stdout:(66/156): openssl-devel-3.2.1-1.el9.x86_64.rpm 14 MB/s | 4.4 MB 00:00 2024-05-12T21:54:30.748 INFO:teuthology.orchestra.run.smithi186.stdout:(110/156): liboath-2.6.7-2.el9.x86_64.rpm 276 kB/s | 49 kB 00:00 2024-05-12T21:54:30.774 INFO:teuthology.orchestra.run.smithi186.stdout:(111/156): libunwind-1.6.2-1.el9.x86_64.rpm 536 kB/s | 67 kB 00:00 2024-05-12T21:54:30.824 INFO:teuthology.orchestra.run.smithi186.stdout:(112/156): python3-asyncssh-2.13.2-5.el9.noarch 11 MB/s | 548 kB 00:00 2024-05-12T21:54:30.858 INFO:teuthology.orchestra.run.smithi186.stdout:(113/156): parquet-libs-9.0.0-11.el9.x86_64.rpm 7.5 MB/s | 839 kB 00:00 2024-05-12T21:54:30.883 INFO:teuthology.orchestra.run.smithi186.stdout:(114/156): python3-bcrypt-3.2.2-1.el9.x86_64.rp 741 kB/s | 43 kB 00:00 2024-05-12T21:54:30.908 INFO:teuthology.orchestra.run.smithi186.stdout:(115/156): luarocks-3.9.2-1.el9.noarch.rpm 827 kB/s | 151 kB 00:00 2024-05-12T21:54:30.934 INFO:teuthology.orchestra.run.smithi186.stdout:(116/156): python3-cachetools-4.2.4-1.el9.noarc 427 kB/s | 32 kB 00:00 2024-05-12T21:54:30.950 INFO:teuthology.orchestra.run.smithi186.stdout:(117/156): python3-certifi-2021.10.8-2.el9.noar 224 kB/s | 15 kB 00:00 2024-05-12T21:54:30.976 INFO:teuthology.orchestra.run.smithi186.stdout:(118/156): python3-cheroot-8.6.0-4.el9.noarch.r 2.5 MB/s | 172 kB 00:00 2024-05-12T21:54:31.001 INFO:teuthology.orchestra.run.smithi186.stdout:(119/156): python3-cherrypy-18.6.1-2.el9.noarch 7.0 MB/s | 358 kB 00:00 2024-05-12T21:54:31.026 INFO:teuthology.orchestra.run.smithi186.stdout:(120/156): python3-colorama-0.4.4-12.el9.noarch 704 kB/s | 35 kB 00:00 2024-05-12T21:54:31.054 INFO:teuthology.orchestra.run.smithi186.stdout:(121/156): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.9 MB/s | 303 kB 00:00 2024-05-12T21:54:31.068 INFO:teuthology.orchestra.run.smithi186.stdout:(122/156): python3-google-auth-2.29.0-1.el9.noa 3.0 MB/s | 206 kB 00:00 2024-05-12T21:54:31.102 INFO:teuthology.orchestra.run.smithi186.stdout:(123/156): python3-grpcio-tools-1.46.7-10.el9.x 2.8 MB/s | 144 kB 00:00 2024-05-12T21:54:31.127 INFO:teuthology.orchestra.run.smithi186.stdout:(124/156): python3-jaraco-8.2.1-3.el9.noarch.rp 182 kB/s | 11 kB 00:00 2024-05-12T21:54:31.152 INFO:teuthology.orchestra.run.smithi186.stdout:(125/156): python3-jaraco-classes-3.2.1-5.el9.n 353 kB/s | 18 kB 00:00 2024-05-12T21:54:31.178 INFO:teuthology.orchestra.run.smithi186.stdout:(126/156): python3-jaraco-collections-3.0.0-8.e 462 kB/s | 23 kB 00:00 2024-05-12T21:54:31.195 INFO:teuthology.orchestra.run.smithi086.stdout:(67/156): ceph-test-19.0.0-2555.gf72fecff.el9.x 7.9 MB/s | 48 MB 00:06 2024-05-12T21:54:31.203 INFO:teuthology.orchestra.run.smithi186.stdout:(127/156): python3-jaraco-functools-3.5.0-2.el9 386 kB/s | 19 kB 00:00 2024-05-12T21:54:31.229 INFO:teuthology.orchestra.run.smithi186.stdout:(128/156): python3-jaraco-text-3.2.0-6.el9.noar 390 kB/s | 20 kB 00:00 2024-05-12T21:54:31.246 INFO:teuthology.orchestra.run.smithi028.stdout:(10/156): ceph-common-19.0.0-2555.gf72fecff.el9 2.8 MB/s | 21 MB 00:07 2024-05-12T21:54:31.253 INFO:teuthology.orchestra.run.smithi086.stdout:(68/156): openblas-openmp-0.3.21-2.el9.x86_64.r 5.4 MB/s | 4.7 MB 00:00 2024-05-12T21:54:31.279 INFO:teuthology.orchestra.run.smithi186.stdout:(129/156): python3-grpcio-1.46.7-10.el9.x86_64. 8.1 MB/s | 2.0 MB 00:00 2024-05-12T21:54:31.295 INFO:teuthology.orchestra.run.smithi086.stdout:(69/156): postgresql-13.14-1.el9.x86_64.rpm 2.8 MB/s | 1.6 MB 00:00 2024-05-12T21:54:31.304 INFO:teuthology.orchestra.run.smithi186.stdout:(130/156): python3-logutils-0.3.5-21.el9.noarch 616 kB/s | 46 kB 00:00 2024-05-12T21:54:31.321 INFO:teuthology.orchestra.run.smithi086.stdout:(70/156): postgresql-private-libs-13.14-1.el9.x 1.9 MB/s | 133 kB 00:00 2024-05-12T21:54:31.338 INFO:teuthology.orchestra.run.smithi186.stdout:(131/156): python3-kubernetes-26.1.0-2.el9.noar 7.6 MB/s | 1.0 MB 00:00 2024-05-12T21:54:31.355 INFO:teuthology.orchestra.run.smithi028.stdout:(11/156): cephfs-mirror-19.0.0-2555.gf72fecff.e 2.0 MB/s | 221 kB 00:00 2024-05-12T21:54:31.363 INFO:teuthology.orchestra.run.smithi186.stdout:(132/156): python3-more-itertools-8.12.0-2.el9. 941 kB/s | 79 kB 00:00 2024-05-12T21:54:31.388 INFO:teuthology.orchestra.run.smithi186.stdout:(133/156): python3-natsort-7.1.1-5.el9.noarch.r 688 kB/s | 58 kB 00:00 2024-05-12T21:54:31.405 INFO:teuthology.orchestra.run.smithi186.stdout:(134/156): python3-portend-3.1.0-2.el9.noarch.r 390 kB/s | 16 kB 00:00 2024-05-12T21:54:31.422 INFO:teuthology.orchestra.run.smithi028.stdout:(12/156): cephfs-shell-19.0.0-2555.gf72fecff.el 415 kB/s | 27 kB 00:00 2024-05-12T21:54:31.430 INFO:teuthology.orchestra.run.smithi086.stdout:(71/156): protobuf-3.14.0-13.el9.x86_64.rpm 9.3 MB/s | 1.0 MB 00:00 2024-05-12T21:54:31.430 INFO:teuthology.orchestra.run.smithi186.stdout:(135/156): python3-pecan-1.4.2-3.el9.noarch.rpm 2.9 MB/s | 272 kB 00:00 2024-05-12T21:54:31.448 INFO:teuthology.orchestra.run.smithi186.stdout:(136/156): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.5 MB/s | 90 kB 00:00 2024-05-12T21:54:31.464 INFO:teuthology.orchestra.run.smithi186.stdout:(137/156): python3-pyperclip-1.8.2-2.el9.noarch 482 kB/s | 28 kB 00:00 2024-05-12T21:54:31.480 INFO:teuthology.orchestra.run.smithi086.stdout:(72/156): postgresql-contrib-13.14-1.el9.x86_64 3.1 MB/s | 911 kB 00:00 2024-05-12T21:54:31.481 INFO:teuthology.orchestra.run.smithi186.stdout:(138/156): python3-repoze-lru-0.7-16.el9.noarch 613 kB/s | 31 kB 00:00 2024-05-12T21:54:31.488 INFO:teuthology.orchestra.run.smithi028.stdout:(13/156): libcephfs-devel-19.0.0-2555.gf72fecff 481 kB/s | 32 kB 00:00 2024-05-12T21:54:31.498 INFO:teuthology.orchestra.run.smithi186.stdout:(139/156): python3-rsa-4.9-2.el9.noarch.rpm 1.7 MB/s | 59 kB 00:00 2024-05-12T21:54:31.505 INFO:teuthology.orchestra.run.smithi086.stdout:(73/156): python3-attrs-20.3.0-7.el9.noarch.rpm 1.1 MB/s | 87 kB 00:00 2024-05-12T21:54:31.515 INFO:teuthology.orchestra.run.smithi186.stdout:(140/156): python3-tempora-5.0.0-2.el9.noarch.r 1.0 MB/s | 36 kB 00:00 2024-05-12T21:54:31.540 INFO:teuthology.orchestra.run.smithi186.stdout:(141/156): python3-routes-2.5.1-5.el9.noarch.rp 2.4 MB/s | 188 kB 00:00 2024-05-12T21:54:31.555 INFO:teuthology.orchestra.run.smithi086.stdout:(74/156): python3-devel-3.9.18-3.el9.x86_64.rpm 4.8 MB/s | 244 kB 00:00 2024-05-12T21:54:31.557 INFO:teuthology.orchestra.run.smithi186.stdout:(142/156): python3-typing-extensions-4.4.0-2.el 871 kB/s | 51 kB 00:00 2024-05-12T21:54:31.582 INFO:teuthology.orchestra.run.smithi186.stdout:(143/156): python3-webob-1.8.7-6.el9.noarch.rpm 3.3 MB/s | 230 kB 00:00 2024-05-12T21:54:31.608 INFO:teuthology.orchestra.run.smithi186.stdout:(144/156): python3-websocket-client-1.2.3-2.el9 1.8 MB/s | 90 kB 00:00 2024-05-12T21:54:31.633 INFO:teuthology.orchestra.run.smithi186.stdout:(145/156): python3-zc-lockfile-2.0-10.el9.noarc 398 kB/s | 20 kB 00:00 2024-05-12T21:54:31.656 INFO:teuthology.orchestra.run.smithi086.stdout:(75/156): postgresql-server-13.14-1.el9.x86_64. 16 MB/s | 5.8 MB 00:00 2024-05-12T21:54:31.658 INFO:teuthology.orchestra.run.smithi186.stdout:(146/156): re2-20211101-3.el9.x86_64.rpm 3.7 MB/s | 192 kB 00:00 2024-05-12T21:54:31.681 INFO:teuthology.orchestra.run.smithi086.stdout:(76/156): python3-jinja2-2.11.3-5.el9.noarch.rp 1.9 MB/s | 248 kB 00:00 2024-05-12T21:54:31.684 INFO:teuthology.orchestra.run.smithi186.stdout:(147/156): python3-werkzeug-2.0.3-3.el9.1.noarc 3.3 MB/s | 427 kB 00:00 2024-05-12T21:54:31.689 INFO:teuthology.orchestra.run.smithi028.stdout:(14/156): libcephfs2-19.0.0-2555.gf72fecff.el9. 3.7 MB/s | 745 kB 00:00 2024-05-12T21:54:31.706 INFO:teuthology.orchestra.run.smithi086.stdout:(77/156): python3-jsonpatch-1.21-16.el9.noarch. 513 kB/s | 26 kB 00:00 2024-05-12T21:54:31.725 INFO:teuthology.orchestra.run.smithi186.stdout:(148/156): thrift-0.15.0-2.el9.x86_64.rpm 17 MB/s | 1.6 MB 00:00 2024-05-12T21:54:31.731 INFO:teuthology.orchestra.run.smithi086.stdout:(78/156): python3-jsonpointer-2.0-4.el9.noarch. 381 kB/s | 19 kB 00:00 2024-05-12T21:54:31.757 INFO:teuthology.orchestra.run.smithi086.stdout:(79/156): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 172 kB 00:00 2024-05-12T21:54:31.764 INFO:teuthology.orchestra.run.smithi028.stdout:(15/156): libcephsqlite-19.0.0-2555.gf72fecff.e 2.2 MB/s | 163 kB 00:00 2024-05-12T21:54:31.782 INFO:teuthology.orchestra.run.smithi086.stdout:(80/156): python3-markupsafe-1.1.1-12.el9.x86_6 693 kB/s | 35 kB 00:00 2024-05-12T21:54:31.839 INFO:teuthology.orchestra.run.smithi028.stdout:(16/156): librados-devel-19.0.0-2555.gf72fecff. 1.7 MB/s | 127 kB 00:00 2024-05-12T21:54:31.859 INFO:teuthology.orchestra.run.smithi186.stdout:(149/156): NetworkManager-1.47.5-1.el9.x86_64.r 17 MB/s | 2.3 MB 00:00 2024-05-12T21:54:31.865 INFO:teuthology.orchestra.run.smithi086.stdout:(81/156): python3-babel-2.9.1-2.el9.noarch.rpm 15 MB/s | 6.0 MB 00:00 2024-05-12T21:54:31.891 INFO:teuthology.orchestra.run.smithi086.stdout:(82/156): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.3 MB/s | 250 kB 00:00 2024-05-12T21:54:31.916 INFO:teuthology.orchestra.run.smithi086.stdout:(83/156): python3-oauthlib-3.1.1-5.el9.noarch.r 4.3 MB/s | 222 kB 00:00 2024-05-12T21:54:31.941 INFO:teuthology.orchestra.run.smithi086.stdout:(84/156): python3-packaging-20.9-5.el9.noarch.r 1.5 MB/s | 77 kB 00:00 2024-05-12T21:54:31.967 INFO:teuthology.orchestra.run.smithi086.stdout:(85/156): python3-prettytable-0.7.2-27.el9.noar 841 kB/s | 42 kB 00:00 2024-05-12T21:54:31.976 INFO:teuthology.orchestra.run.smithi186.stdout:(150/156): NetworkManager-libnm-1.47.5-1.el9.x8 16 MB/s | 1.8 MB 00:00 2024-05-12T21:54:32.000 INFO:teuthology.orchestra.run.smithi086.stdout:(86/156): python3-protobuf-3.14.0-13.el9.noarch 4.5 MB/s | 269 kB 00:00 2024-05-12T21:54:32.015 INFO:teuthology.orchestra.run.smithi028.stdout:(17/156): libradosstriper1-19.0.0-2555.gf72fecf 2.8 MB/s | 500 kB 00:00 2024-05-12T21:54:32.025 INFO:teuthology.orchestra.run.smithi086.stdout:(87/156): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.6 MB/s | 159 kB 00:00 2024-05-12T21:54:32.034 INFO:teuthology.orchestra.run.smithi186.stdout:(151/156): NetworkManager-team-1.47.5-1.el9.x86 717 kB/s | 42 kB 00:00 2024-05-12T21:54:32.051 INFO:teuthology.orchestra.run.smithi086.stdout:(88/156): python3-pyasn1-modules-0.4.8-6.el9.no 5.4 MB/s | 279 kB 00:00 2024-05-12T21:54:32.076 INFO:teuthology.orchestra.run.smithi086.stdout:(89/156): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-05-12T21:54:32.101 INFO:teuthology.orchestra.run.smithi086.stdout:(90/156): python3-requests-oauthlib-1.3.0-12.el 1.0 MB/s | 54 kB 00:00 2024-05-12T21:54:32.110 INFO:teuthology.orchestra.run.smithi186.stdout:(152/156): NetworkManager-tui-1.47.5-1.el9.x86_ 3.2 MB/s | 248 kB 00:00 2024-05-12T21:54:32.168 INFO:teuthology.orchestra.run.smithi086.stdout:(91/156): python3-numpy-1.20.1-5.el9.x86_64.rpm 12 MB/s | 5.1 MB 00:00 2024-05-12T21:54:32.193 INFO:teuthology.orchestra.run.smithi086.stdout:(92/156): python3-toml-0.10.2-6.el9.noarch.rpm 453 kB/s | 42 kB 00:00 2024-05-12T21:54:32.218 INFO:teuthology.orchestra.run.smithi186.stdout:(153/156): openssl-3.2.1-1.el9.x86_64.rpm 12 MB/s | 1.3 MB 00:00 2024-05-12T21:54:32.219 INFO:teuthology.orchestra.run.smithi086.stdout:(93/156): python3-wcwidth-0.2.5-8.el9.noarch.rp 880 kB/s | 44 kB 00:00 2024-05-12T21:54:32.244 INFO:teuthology.orchestra.run.smithi086.stdout:(94/156): qatlib-24.02.0-1.el9.x86_64.rpm 4.3 MB/s | 221 kB 00:00 2024-05-12T21:54:32.270 INFO:teuthology.orchestra.run.smithi086.stdout:(95/156): qatlib-service-24.02.0-1.el9.x86_64.r 716 kB/s | 36 kB 00:00 2024-05-12T21:54:32.295 INFO:teuthology.orchestra.run.smithi086.stdout:(96/156): qatzip-libs-1.2.0-1.el9.x86_64.rpm 944 kB/s | 47 kB 00:00 2024-05-12T21:54:32.319 INFO:teuthology.orchestra.run.smithi186.stdout:(154/156): librados2-19.0.0-2555.gf72fecff.el9. 5.1 MB/s | 3.4 MB 00:00 2024-05-12T21:54:32.320 INFO:teuthology.orchestra.run.smithi086.stdout:(97/156): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 305 kB 00:00 2024-05-12T21:54:32.345 INFO:teuthology.orchestra.run.smithi086.stdout:(98/156): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 58 kB 00:00 2024-05-12T21:54:32.370 INFO:teuthology.orchestra.run.smithi086.stdout:(99/156): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.8 MB/s | 94 kB 00:00 2024-05-12T21:54:32.394 INFO:teuthology.orchestra.run.smithi186.stdout:(155/156): openssl-libs-3.2.1-1.el9.x86_64.rpm 14 MB/s | 2.4 MB 00:00 2024-05-12T21:54:32.396 INFO:teuthology.orchestra.run.smithi086.stdout:(100/156): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2024-05-12T21:54:32.438 INFO:teuthology.orchestra.run.smithi086.stdout:(101/156): zlib-devel-1.2.11-41.el9.x86_64.rpm 666 kB/s | 45 kB 00:00 2024-05-12T21:54:32.480 INFO:teuthology.orchestra.run.smithi086.stdout:(102/156): lua-devel-5.4.4-4.el9.x86_64.rpm 266 kB/s | 22 kB 00:00 2024-05-12T21:54:32.769 INFO:teuthology.orchestra.run.smithi186.stdout:(156/156): librbd1-19.0.0-2555.gf72fecff.el9.x8 2.9 MB/s | 3.1 MB 00:01 2024-05-12T21:54:32.771 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-05-12T21:54:32.772 INFO:teuthology.orchestra.run.smithi186.stdout:Total 15 MB/s | 225 MB 00:14 2024-05-12T21:54:32.856 INFO:teuthology.orchestra.run.smithi086.stdout:(103/156): python3-scipy-1.6.2-8.el9.x86_64.rpm 21 MB/s | 16 MB 00:00 2024-05-12T21:54:32.897 INFO:teuthology.orchestra.run.smithi086.stdout:(104/156): protobuf-compiler-3.14.0-13.el9.x86_ 1.8 MB/s | 863 kB 00:00 2024-05-12T21:54:32.973 INFO:teuthology.orchestra.run.smithi086.stdout:(105/156): abseil-cpp-20211102.0-3.el9.x86_64.r 1.1 MB/s | 548 kB 00:00 2024-05-12T21:54:33.056 INFO:teuthology.orchestra.run.smithi086.stdout:(106/156): grpc-data-1.46.7-10.el9.noarch.rpm 123 kB/s | 19 kB 00:00 2024-05-12T21:54:33.115 INFO:teuthology.orchestra.run.smithi086.stdout:(107/156): libarrow-doc-9.0.0-11.el9.noarch.rpm 445 kB/s | 26 kB 00:00 2024-05-12T21:54:33.157 INFO:teuthology.orchestra.run.smithi086.stdout:(108/156): gperftools-libs-2.9.1-2.el9.x86_64.r 1.0 MB/s | 309 kB 00:00 2024-05-12T21:54:33.232 INFO:teuthology.orchestra.run.smithi086.stdout:(109/156): libarrow-9.0.0-11.el9.x86_64.rpm 17 MB/s | 4.4 MB 00:00 2024-05-12T21:54:33.265 INFO:teuthology.orchestra.run.smithi086.stdout:(110/156): liboath-2.6.7-2.el9.x86_64.rpm 322 kB/s | 49 kB 00:00 2024-05-12T21:54:33.291 INFO:teuthology.orchestra.run.smithi086.stdout:(111/156): libunwind-1.6.2-1.el9.x86_64.rpm 503 kB/s | 67 kB 00:00 2024-05-12T21:54:33.318 INFO:teuthology.orchestra.run.smithi086.stdout:(112/156): luarocks-3.9.2-1.el9.noarch.rpm 1.8 MB/s | 151 kB 00:00 2024-05-12T21:54:33.383 INFO:teuthology.orchestra.run.smithi086.stdout:(113/156): python3-bcrypt-3.2.2-1.el9.x86_64.rp 652 kB/s | 43 kB 00:00 2024-05-12T21:54:33.424 INFO:teuthology.orchestra.run.smithi028.stdout:(18/156): librgw2-19.0.0-2555.gf72fecff.el9.x86 3.8 MB/s | 5.3 MB 00:01 2024-05-12T21:54:33.425 INFO:teuthology.orchestra.run.smithi086.stdout:(114/156): python3-asyncssh-2.13.2-5.el9.noarch 4.0 MB/s | 548 kB 00:00 2024-05-12T21:54:33.450 INFO:teuthology.orchestra.run.smithi086.stdout:(115/156): python3-cachetools-4.2.4-1.el9.noarc 482 kB/s | 32 kB 00:00 2024-05-12T21:54:33.492 INFO:teuthology.orchestra.run.smithi086.stdout:(116/156): python3-certifi-2021.10.8-2.el9.noar 224 kB/s | 15 kB 00:00 2024-05-12T21:54:33.499 INFO:teuthology.orchestra.run.smithi028.stdout:(19/156): python3-ceph-argparse-19.0.0-2555.gf7 610 kB/s | 45 kB 00:00 2024-05-12T21:54:33.526 INFO:teuthology.orchestra.run.smithi086.stdout:(117/156): python3-cheroot-8.6.0-4.el9.noarch.r 2.2 MB/s | 172 kB 00:00 2024-05-12T21:54:33.568 INFO:teuthology.orchestra.run.smithi086.stdout:(118/156): python3-cherrypy-18.6.1-2.el9.noarch 4.6 MB/s | 358 kB 00:00 2024-05-12T21:54:33.574 INFO:teuthology.orchestra.run.smithi028.stdout:(20/156): python3-ceph-common-19.0.0-2555.gf72f 1.8 MB/s | 135 kB 00:00 2024-05-12T21:54:33.601 INFO:teuthology.orchestra.run.smithi086.stdout:(119/156): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.9 MB/s | 303 kB 00:00 2024-05-12T21:54:33.635 INFO:teuthology.orchestra.run.smithi086.stdout:(120/156): python3-colorama-0.4.4-12.el9.noarch 528 kB/s | 35 kB 00:00 2024-05-12T21:54:33.649 INFO:teuthology.orchestra.run.smithi028.stdout:(21/156): python3-cephfs-19.0.0-2555.gf72fecff. 2.1 MB/s | 161 kB 00:00 2024-05-12T21:54:33.677 INFO:teuthology.orchestra.run.smithi086.stdout:(121/156): python3-google-auth-2.29.0-1.el9.noa 2.7 MB/s | 206 kB 00:00 2024-05-12T21:54:33.719 INFO:teuthology.orchestra.run.smithi086.stdout:(122/156): parquet-libs-9.0.0-11.el9.x86_64.rpm 1.8 MB/s | 839 kB 00:00 2024-05-12T21:54:33.752 INFO:teuthology.orchestra.run.smithi086.stdout:(123/156): python3-grpcio-tools-1.46.7-10.el9.x 1.9 MB/s | 144 kB 00:00 2024-05-12T21:54:33.766 INFO:teuthology.orchestra.run.smithi028.stdout:(22/156): python3-rados-19.0.0-2555.gf72fecff.e 2.7 MB/s | 321 kB 00:00 2024-05-12T21:54:33.794 INFO:teuthology.orchestra.run.smithi086.stdout:(124/156): python3-grpcio-1.46.7-10.el9.x86_64. 13 MB/s | 2.0 MB 00:00 2024-05-12T21:54:33.820 INFO:teuthology.orchestra.run.smithi086.stdout:(125/156): python3-jaraco-8.2.1-3.el9.noarch.rp 106 kB/s | 11 kB 00:00 2024-05-12T21:54:33.862 INFO:teuthology.orchestra.run.smithi086.stdout:(126/156): python3-jaraco-classes-3.2.1-5.el9.n 162 kB/s | 18 kB 00:00 2024-05-12T21:54:33.875 INFO:teuthology.orchestra.run.smithi028.stdout:(23/156): python3-rbd-19.0.0-2555.gf72fecff.el9 2.7 MB/s | 301 kB 00:00 2024-05-12T21:54:33.887 INFO:teuthology.orchestra.run.smithi086.stdout:(127/156): python3-jaraco-collections-3.0.0-8.e 251 kB/s | 23 kB 00:00 2024-05-12T21:54:33.902 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-05-12T21:54:33.912 INFO:teuthology.orchestra.run.smithi086.stdout:(128/156): python3-jaraco-functools-3.5.0-2.el9 211 kB/s | 19 kB 00:00 2024-05-12T21:54:33.937 INFO:teuthology.orchestra.run.smithi086.stdout:(129/156): python3-jaraco-text-3.2.0-6.el9.noar 261 kB/s | 20 kB 00:00 2024-05-12T21:54:33.950 INFO:teuthology.orchestra.run.smithi028.stdout:(24/156): python3-rgw-19.0.0-2555.gf72fecff.el9 1.3 MB/s | 99 kB 00:00 2024-05-12T21:54:33.988 INFO:teuthology.orchestra.run.smithi086.stdout:(130/156): python3-kubernetes-26.1.0-2.el9.noar 10 MB/s | 1.0 MB 00:00 2024-05-12T21:54:34.013 INFO:teuthology.orchestra.run.smithi086.stdout:(131/156): python3-logutils-0.3.5-21.el9.noarch 460 kB/s | 46 kB 00:00 2024-05-12T21:54:34.026 INFO:teuthology.orchestra.run.smithi028.stdout:(25/156): rbd-fuse-19.0.0-2555.gf72fecff.el9.x8 1.1 MB/s | 85 kB 00:00 2024-05-12T21:54:34.039 INFO:teuthology.orchestra.run.smithi086.stdout:(132/156): python3-more-itertools-8.12.0-2.el9. 781 kB/s | 79 kB 00:00 2024-05-12T21:54:34.052 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-05-12T21:54:34.052 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-05-12T21:54:34.064 INFO:teuthology.orchestra.run.smithi086.stdout:(133/156): python3-natsort-7.1.1-5.el9.noarch.r 767 kB/s | 58 kB 00:00 2024-05-12T21:54:34.106 INFO:teuthology.orchestra.run.smithi086.stdout:(134/156): python3-portend-3.1.0-2.el9.noarch.r 246 kB/s | 16 kB 00:00 2024-05-12T21:54:34.139 INFO:teuthology.orchestra.run.smithi086.stdout:(135/156): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.2 MB/s | 90 kB 00:00 2024-05-12T21:54:34.190 INFO:teuthology.orchestra.run.smithi086.stdout:(136/156): python3-pyperclip-1.8.2-2.el9.noarch 338 kB/s | 28 kB 00:00 2024-05-12T21:54:34.215 INFO:teuthology.orchestra.run.smithi086.stdout:(137/156): python3-pecan-1.4.2-3.el9.noarch.rpm 1.3 MB/s | 272 kB 00:00 2024-05-12T21:54:34.240 INFO:teuthology.orchestra.run.smithi086.stdout:(138/156): python3-repoze-lru-0.7-16.el9.noarch 306 kB/s | 31 kB 00:00 2024-05-12T21:54:34.243 INFO:teuthology.orchestra.run.smithi028.stdout:(26/156): ceph-osd-19.0.0-2555.gf72fecff.el9.x8 2.6 MB/s | 17 MB 00:06 2024-05-12T21:54:34.265 INFO:teuthology.orchestra.run.smithi086.stdout:(139/156): python3-routes-2.5.1-5.el9.noarch.rp 2.4 MB/s | 188 kB 00:00 2024-05-12T21:54:34.316 INFO:teuthology.orchestra.run.smithi086.stdout:(140/156): python3-tempora-5.0.0-2.el9.noarch.r 476 kB/s | 36 kB 00:00 2024-05-12T21:54:34.341 INFO:teuthology.orchestra.run.smithi086.stdout:(141/156): python3-rsa-4.9-2.el9.noarch.rpm 589 kB/s | 59 kB 00:00 2024-05-12T21:54:34.366 INFO:teuthology.orchestra.run.smithi086.stdout:(142/156): python3-typing-extensions-4.4.0-2.el 509 kB/s | 51 kB 00:00 2024-05-12T21:54:34.368 INFO:teuthology.orchestra.run.smithi028.stdout:(27/156): rbd-nbd-19.0.0-2555.gf72fecff.el9.x86 1.4 MB/s | 171 kB 00:00 2024-05-12T21:54:34.441 INFO:teuthology.orchestra.run.smithi086.stdout:(143/156): python3-websocket-client-1.2.3-2.el9 1.2 MB/s | 90 kB 00:00 2024-05-12T21:54:34.443 INFO:teuthology.orchestra.run.smithi028.stdout:(28/156): ceph-grafana-dashboards-19.0.0-2555.g 430 kB/s | 32 kB 00:00 2024-05-12T21:54:34.466 INFO:teuthology.orchestra.run.smithi086.stdout:(144/156): python3-webob-1.8.7-6.el9.noarch.rpm 1.5 MB/s | 230 kB 00:00 2024-05-12T21:54:34.492 INFO:teuthology.orchestra.run.smithi086.stdout:(145/156): python3-werkzeug-2.0.3-3.el9.1.noarc 3.3 MB/s | 427 kB 00:00 2024-05-12T21:54:34.509 INFO:teuthology.orchestra.run.smithi086.stdout:(146/156): python3-zc-lockfile-2.0-10.el9.noarc 297 kB/s | 20 kB 00:00 2024-05-12T21:54:34.543 INFO:teuthology.orchestra.run.smithi028.stdout:(29/156): ceph-mgr-cephadm-19.0.0-2555.gf72fecf 1.4 MB/s | 142 kB 00:00 2024-05-12T21:54:34.593 INFO:teuthology.orchestra.run.smithi086.stdout:(147/156): thrift-0.15.0-2.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-05-12T21:54:34.618 INFO:teuthology.orchestra.run.smithi086.stdout:(148/156): re2-20211101-3.el9.x86_64.rpm 1.2 MB/s | 192 kB 00:00 2024-05-12T21:54:34.951 INFO:teuthology.orchestra.run.smithi086.stdout:(149/156): librbd1-19.0.0-2555.gf72fecff.el9.x8 8.7 MB/s | 3.1 MB 00:00 2024-05-12T21:54:35.001 INFO:teuthology.orchestra.run.smithi086.stdout:(150/156): librados2-19.0.0-2555.gf72fecff.el9. 6.9 MB/s | 3.4 MB 00:00 2024-05-12T21:54:35.044 INFO:teuthology.orchestra.run.smithi028.stdout:(30/156): ceph-test-19.0.0-2555.gf72fecff.el9.x 9.4 MB/s | 48 MB 00:05 2024-05-12T21:54:35.110 INFO:teuthology.orchestra.run.smithi086.stdout:(151/156): NetworkManager-1.47.5-1.el9.x86_64.r 4.7 MB/s | 2.3 MB 00:00 2024-05-12T21:54:35.135 INFO:teuthology.orchestra.run.smithi086.stdout:(152/156): NetworkManager-team-1.47.5-1.el9.x86 313 kB/s | 42 kB 00:00 2024-05-12T21:54:35.144 INFO:teuthology.orchestra.run.smithi028.stdout:(31/156): rbd-mirror-19.0.0-2555.gf72fecff.el9. 2.8 MB/s | 3.1 MB 00:01 2024-05-12T21:54:35.169 INFO:teuthology.orchestra.run.smithi086.stdout:(153/156): NetworkManager-tui-1.47.5-1.el9.x86_ 4.1 MB/s | 248 kB 00:00 2024-05-12T21:54:35.286 INFO:teuthology.orchestra.run.smithi028.stdout:(32/156): ceph-mgr-modules-core-19.0.0-2555.gf7 1.7 MB/s | 249 kB 00:00 2024-05-12T21:54:35.386 INFO:teuthology.orchestra.run.smithi028.stdout:(33/156): ceph-mgr-rook-19.0.0-2555.gf72fecff.e 497 kB/s | 49 kB 00:00 2024-05-12T21:54:35.487 INFO:teuthology.orchestra.run.smithi028.stdout:(34/156): ceph-prometheus-alerts-19.0.0-2555.gf 166 kB/s | 16 kB 00:00 2024-05-12T21:54:35.528 INFO:teuthology.orchestra.run.smithi086.stdout:(154/156): NetworkManager-libnm-1.47.5-1.el9.x8 3.2 MB/s | 1.8 MB 00:00 2024-05-12T21:54:35.644 INFO:teuthology.orchestra.run.smithi086.stdout:(155/156): openssl-libs-3.2.1-1.el9.x86_64.rpm 5.0 MB/s | 2.4 MB 00:00 2024-05-12T21:54:35.645 INFO:teuthology.orchestra.run.smithi028.stdout:(35/156): ceph-volume-19.0.0-2555.gf72fecff.el9 1.6 MB/s | 264 kB 00:00 2024-05-12T21:54:35.771 INFO:teuthology.orchestra.run.smithi028.stdout:(36/156): ceph-mgr-diskprediction-local-19.0.0- 10 MB/s | 7.4 MB 00:00 2024-05-12T21:54:35.803 INFO:teuthology.orchestra.run.smithi086.stdout:(156/156): openssl-3.2.1-1.el9.x86_64.rpm 2.0 MB/s | 1.3 MB 00:00 2024-05-12T21:54:35.806 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-05-12T21:54:35.806 INFO:teuthology.orchestra.run.smithi086.stdout:Total 14 MB/s | 225 MB 00:16 2024-05-12T21:54:35.821 INFO:teuthology.orchestra.run.smithi028.stdout:(37/156): cephfs-top-19.0.0-2555.gf72fecff.el9. 374 kB/s | 18 kB 00:00 2024-05-12T21:54:35.881 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-05-12T21:54:35.922 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-05-12T21:54:35.954 INFO:teuthology.orchestra.run.smithi028.stdout:(38/156): cephadm-19.0.0-2555.gf72fecff.el9.noa 2.4 MB/s | 763 kB 00:00 2024-05-12T21:54:36.255 INFO:teuthology.orchestra.run.smithi028.stdout:(39/156): ceph-mgr-dashboard-19.0.0-2555.gf72fe 2.1 MB/s | 3.5 MB 00:01 2024-05-12T21:54:36.288 INFO:teuthology.orchestra.run.smithi028.stdout:(40/156): fuse-2.9.9-15.el9.x86_64.rpm 172 kB/s | 80 kB 00:00 2024-05-12T21:54:36.647 INFO:teuthology.orchestra.run.smithi028.stdout:(41/156): python3-cffi-1.14.5-5.el9.x86_64.rpm 708 kB/s | 253 kB 00:00 2024-05-12T21:54:36.814 INFO:teuthology.orchestra.run.smithi028.stdout:(42/156): libgfortran-11.4.1-3.el9.x86_64.rpm 936 kB/s | 803 kB 00:00 2024-05-12T21:54:36.839 INFO:teuthology.orchestra.run.smithi028.stdout:(43/156): libquadmath-11.4.1-3.el9.x86_64.rpm 335 kB/s | 195 kB 00:00 2024-05-12T21:54:36.923 INFO:teuthology.orchestra.run.smithi028.stdout:(44/156): python3-ply-3.11-14.el9.noarch.rpm 987 kB/s | 106 kB 00:00 2024-05-12T21:54:36.927 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-05-12T21:54:37.031 INFO:teuthology.orchestra.run.smithi028.stdout:(45/156): python3-pycparser-2.20-6.el9.noarch.r 707 kB/s | 135 kB 00:00 2024-05-12T21:54:37.056 INFO:teuthology.orchestra.run.smithi028.stdout:(46/156): python3-requests-2.25.1-8.el9.noarch. 946 kB/s | 125 kB 00:00 2024-05-12T21:54:37.075 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-05-12T21:54:37.075 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-05-12T21:54:37.115 INFO:teuthology.orchestra.run.smithi028.stdout:(47/156): python3-cryptography-36.0.1-4.el9.x86 2.6 MB/s | 1.2 MB 00:00 2024-05-12T21:54:37.149 INFO:teuthology.orchestra.run.smithi028.stdout:(48/156): python3-urllib3-1.26.5-5.el9.noarch.r 1.8 MB/s | 215 kB 00:00 2024-05-12T21:54:37.282 INFO:teuthology.orchestra.run.smithi028.stdout:(49/156): boost-program-options-1.75.0-8.el9.x8 806 kB/s | 107 kB 00:00 2024-05-12T21:54:37.325 INFO:teuthology.orchestra.run.smithi028.stdout:(50/156): automake-1.16.2-8.el9.noarch.rpm 2.5 MB/s | 689 kB 00:00 2024-05-12T21:54:37.350 INFO:teuthology.orchestra.run.smithi028.stdout:(51/156): elfutils-libelf-devel-0.190-2.el9.x86 325 kB/s | 21 kB 00:00 2024-05-12T21:54:37.418 INFO:teuthology.orchestra.run.smithi028.stdout:(52/156): flex-2.6.4-9.el9.x86_64.rpm 3.4 MB/s | 313 kB 00:00 2024-05-12T21:54:37.442 INFO:teuthology.orchestra.run.smithi028.stdout:(53/156): flexiblas-3.0.4-8.el9.x86_64.rpm 360 kB/s | 33 kB 00:00 2024-05-12T21:54:37.484 INFO:teuthology.orchestra.run.smithi028.stdout:(54/156): bison-3.7.4-5.el9.x86_64.rpm 2.5 MB/s | 943 kB 00:00 2024-05-12T21:54:37.509 INFO:teuthology.orchestra.run.smithi028.stdout:(55/156): flexiblas-openblas-openmp-3.0.4-8.el9 275 kB/s | 18 kB 00:00 2024-05-12T21:54:37.534 INFO:teuthology.orchestra.run.smithi028.stdout:(56/156): libaio-devel-0.3.111-13.el9.x86_64.rp 225 kB/s | 11 kB 00:00 2024-05-12T21:54:37.568 INFO:teuthology.orchestra.run.smithi028.stdout:(57/156): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.7 MB/s | 160 kB 00:00 2024-05-12T21:54:37.594 INFO:teuthology.orchestra.run.smithi028.stdout:(58/156): librabbitmq-0.11.0-7.el9.x86_64.rpm 783 kB/s | 45 kB 00:00 2024-05-12T21:54:37.652 INFO:teuthology.orchestra.run.smithi028.stdout:(59/156): librdkafka-1.6.1-102.el9.x86_64.rpm 7.8 MB/s | 662 kB 00:00 2024-05-12T21:54:37.702 INFO:teuthology.orchestra.run.smithi028.stdout:(60/156): libuuid-devel-2.37.4-18.el9.x86_64.rp 519 kB/s | 25 kB 00:00 2024-05-12T21:54:37.744 INFO:teuthology.orchestra.run.smithi028.stdout:(61/156): libtool-2.4.6-45.el9.x86_64.rpm 3.8 MB/s | 582 kB 00:00 2024-05-12T21:54:37.820 INFO:teuthology.orchestra.run.smithi028.stdout:(62/156): flexiblas-netlib-3.0.4-8.el9.x86_64.r 7.4 MB/s | 3.0 MB 00:00 2024-05-12T21:54:38.003 INFO:teuthology.orchestra.run.smithi028.stdout:(63/156): libzstd-devel-1.5.1-2.el9.x86_64.rpm 158 kB/s | 47 kB 00:00 2024-05-12T21:54:38.037 INFO:teuthology.orchestra.run.smithi028.stdout:(64/156): lua-5.4.4-4.el9.x86_64.rpm 871 kB/s | 188 kB 00:00 2024-05-12T21:54:38.063 INFO:teuthology.orchestra.run.smithi028.stdout:(65/156): openblas-0.3.21-2.el9.x86_64.rpm 606 kB/s | 35 kB 00:00 2024-05-12T21:54:38.096 INFO:teuthology.orchestra.run.smithi028.stdout:(66/156): lttng-ust-2.12.0-6.el9.x86_64.rpm 835 kB/s | 292 kB 00:00 2024-05-12T21:54:38.230 INFO:teuthology.orchestra.run.smithi028.stdout:(67/156): postgresql-13.14-1.el9.x86_64.rpm 12 MB/s | 1.6 MB 00:00 2024-05-12T21:54:38.313 INFO:teuthology.orchestra.run.smithi028.stdout:(68/156): openssl-devel-3.2.1-1.el9.x86_64.rpm 18 MB/s | 4.4 MB 00:00 2024-05-12T21:54:38.430 INFO:teuthology.orchestra.run.smithi028.stdout:(69/156): openblas-openmp-0.3.21-2.el9.x86_64.r 12 MB/s | 4.7 MB 00:00 2024-05-12T21:54:38.455 INFO:teuthology.orchestra.run.smithi028.stdout:(70/156): postgresql-private-libs-13.14-1.el9.x 945 kB/s | 133 kB 00:00 2024-05-12T21:54:38.514 INFO:teuthology.orchestra.run.smithi028.stdout:(71/156): postgresql-contrib-13.14-1.el9.x86_64 3.1 MB/s | 911 kB 00:00 2024-05-12T21:54:38.581 INFO:teuthology.orchestra.run.smithi028.stdout:(72/156): protobuf-3.14.0-13.el9.x86_64.rpm 8.1 MB/s | 1.0 MB 00:00 2024-05-12T21:54:38.606 INFO:teuthology.orchestra.run.smithi028.stdout:(73/156): python3-attrs-20.3.0-7.el9.noarch.rpm 961 kB/s | 87 kB 00:00 2024-05-12T21:54:38.673 INFO:teuthology.orchestra.run.smithi028.stdout:(74/156): python3-devel-3.9.18-3.el9.x86_64.rpm 3.6 MB/s | 244 kB 00:00 2024-05-12T21:54:38.732 INFO:teuthology.orchestra.run.smithi028.stdout:(75/156): python3-jinja2-2.11.3-5.el9.noarch.rp 4.2 MB/s | 248 kB 00:00 2024-05-12T21:54:38.782 INFO:teuthology.orchestra.run.smithi028.stdout:(76/156): python3-jsonpatch-1.21-16.el9.noarch. 523 kB/s | 26 kB 00:00 2024-05-12T21:54:38.891 INFO:teuthology.orchestra.run.smithi028.stdout:(77/156): postgresql-server-13.14-1.el9.x86_64. 13 MB/s | 5.8 MB 00:00 2024-05-12T21:54:38.916 INFO:teuthology.orchestra.run.smithi028.stdout:(78/156): python3-jsonpointer-2.0-4.el9.noarch. 144 kB/s | 19 kB 00:00 2024-05-12T21:54:38.950 INFO:teuthology.orchestra.run.smithi028.stdout:(79/156): python3-mako-1.1.4-6.el9.noarch.rpm 2.9 MB/s | 172 kB 00:00 2024-05-12T21:54:38.975 INFO:teuthology.orchestra.run.smithi028.stdout:(80/156): python3-markupsafe-1.1.1-12.el9.x86_6 599 kB/s | 35 kB 00:00 2024-05-12T21:54:39.042 INFO:teuthology.orchestra.run.smithi028.stdout:(81/156): python3-numpy-f2py-1.20.1-5.el9.x86_6 3.8 MB/s | 250 kB 00:00 2024-05-12T21:54:39.149 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-05-12T21:54:39.149 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-05-12T21:54:39.159 INFO:teuthology.orchestra.run.smithi028.stdout:(82/156): python3-babel-2.9.1-2.el9.noarch.rpm 10 MB/s | 6.0 MB 00:00 2024-05-12T21:54:39.193 INFO:teuthology.orchestra.run.smithi028.stdout:(83/156): python3-oauthlib-3.1.1-5.el9.noarch.r 1.5 MB/s | 222 kB 00:00 2024-05-12T21:54:39.218 INFO:teuthology.orchestra.run.smithi028.stdout:(84/156): python3-packaging-20.9-5.el9.noarch.r 1.3 MB/s | 77 kB 00:00 2024-05-12T21:54:39.244 INFO:teuthology.orchestra.run.smithi028.stdout:(85/156): python3-prettytable-0.7.2-27.el9.noar 855 kB/s | 42 kB 00:00 2024-05-12T21:54:39.277 INFO:teuthology.orchestra.run.smithi028.stdout:(86/156): python3-protobuf-3.14.0-13.el9.noarch 4.6 MB/s | 269 kB 00:00 2024-05-12T21:54:39.310 INFO:teuthology.orchestra.run.smithi028.stdout:(87/156): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.3 MB/s | 159 kB 00:00 2024-05-12T21:54:39.344 INFO:teuthology.orchestra.run.smithi028.stdout:(88/156): python3-pyasn1-modules-0.4.8-6.el9.no 4.1 MB/s | 279 kB 00:00 2024-05-12T21:54:39.370 INFO:teuthology.orchestra.run.smithi028.stdout:(89/156): python3-pytz-2021.1-5.el9.noarch.rpm 883 kB/s | 51 kB 00:00 2024-05-12T21:54:39.395 INFO:teuthology.orchestra.run.smithi028.stdout:(90/156): python3-requests-oauthlib-1.3.0-12.el 1.1 MB/s | 54 kB 00:00 2024-05-12T21:54:39.462 INFO:teuthology.orchestra.run.smithi028.stdout:(91/156): python3-toml-0.10.2-6.el9.noarch.rpm 631 kB/s | 42 kB 00:00 2024-05-12T21:54:39.571 INFO:teuthology.orchestra.run.smithi028.stdout:(92/156): python3-numpy-1.20.1-5.el9.x86_64.rpm 8.2 MB/s | 5.1 MB 00:00 2024-05-12T21:54:39.595 INFO:teuthology.orchestra.run.smithi028.stdout:(93/156): python3-wcwidth-0.2.5-8.el9.noarch.rp 333 kB/s | 44 kB 00:00 2024-05-12T21:54:39.629 INFO:teuthology.orchestra.run.smithi028.stdout:(94/156): qatlib-24.02.0-1.el9.x86_64.rpm 3.8 MB/s | 221 kB 00:00 2024-05-12T21:54:39.654 INFO:teuthology.orchestra.run.smithi028.stdout:(95/156): qatlib-service-24.02.0-1.el9.x86_64.r 621 kB/s | 36 kB 00:00 2024-05-12T21:54:39.688 INFO:teuthology.orchestra.run.smithi028.stdout:(96/156): qatzip-libs-1.2.0-1.el9.x86_64.rpm 817 kB/s | 47 kB 00:00 2024-05-12T21:54:39.722 INFO:teuthology.orchestra.run.smithi028.stdout:(97/156): socat-1.7.4.1-5.el9.x86_64.rpm 4.5 MB/s | 305 kB 00:00 2024-05-12T21:54:39.747 INFO:teuthology.orchestra.run.smithi028.stdout:(98/156): uuid-1.6.2-55.el9.x86_64.rpm 1.0 MB/s | 58 kB 00:00 2024-05-12T21:54:39.772 INFO:teuthology.orchestra.run.smithi028.stdout:(99/156): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.9 MB/s | 94 kB 00:00 2024-05-12T21:54:39.798 INFO:teuthology.orchestra.run.smithi028.stdout:(100/156): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.3 MB/s | 64 kB 00:00 2024-05-12T21:54:39.823 INFO:teuthology.orchestra.run.smithi028.stdout:(101/156): zlib-devel-1.2.11-41.el9.x86_64.rpm 903 kB/s | 45 kB 00:00 2024-05-12T21:54:39.907 INFO:teuthology.orchestra.run.smithi028.stdout:(102/156): lua-devel-5.4.4-4.el9.x86_64.rpm 207 kB/s | 22 kB 00:00 2024-05-12T21:54:40.041 INFO:teuthology.orchestra.run.smithi028.stdout:(103/156): protobuf-compiler-3.14.0-13.el9.x86_ 3.9 MB/s | 863 kB 00:00 2024-05-12T21:54:40.233 INFO:teuthology.orchestra.run.smithi028.stdout:(104/156): python3-scipy-1.6.2-8.el9.x86_64.rpm 19 MB/s | 16 MB 00:00 2024-05-12T21:54:40.324 INFO:teuthology.orchestra.run.smithi028.stdout:(105/156): abseil-cpp-20211102.0-3.el9.x86_64.r 1.3 MB/s | 548 kB 00:00 2024-05-12T21:54:40.350 INFO:teuthology.orchestra.run.smithi028.stdout:(106/156): grpc-data-1.46.7-10.el9.noarch.rpm 169 kB/s | 19 kB 00:00 2024-05-12T21:54:40.400 INFO:teuthology.orchestra.run.smithi028.stdout:(107/156): libarrow-doc-9.0.0-11.el9.noarch.rpm 532 kB/s | 26 kB 00:00 2024-05-12T21:54:40.484 INFO:teuthology.orchestra.run.smithi028.stdout:(108/156): gperftools-libs-2.9.1-2.el9.x86_64.r 698 kB/s | 309 kB 00:00 2024-05-12T21:54:40.509 INFO:teuthology.orchestra.run.smithi028.stdout:(109/156): liboath-2.6.7-2.el9.x86_64.rpm 450 kB/s | 49 kB 00:00 2024-05-12T21:54:40.534 INFO:teuthology.orchestra.run.smithi028.stdout:(110/156): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-05-12T21:54:40.584 INFO:teuthology.orchestra.run.smithi028.stdout:(111/156): luarocks-3.9.2-1.el9.noarch.rpm 2.0 MB/s | 151 kB 00:00 2024-05-12T21:54:40.693 INFO:teuthology.orchestra.run.smithi028.stdout:(112/156): libarrow-9.0.0-11.el9.x86_64.rpm 12 MB/s | 4.4 MB 00:00 2024-05-12T21:54:40.735 INFO:teuthology.orchestra.run.smithi028.stdout:(113/156): parquet-libs-9.0.0-11.el9.x86_64.rpm 4.1 MB/s | 839 kB 00:00 2024-05-12T21:54:40.777 INFO:teuthology.orchestra.run.smithi028.stdout:(114/156): python3-asyncssh-2.13.2-5.el9.noarch 2.8 MB/s | 548 kB 00:00 2024-05-12T21:54:40.802 INFO:teuthology.orchestra.run.smithi028.stdout:(115/156): python3-cachetools-4.2.4-1.el9.noarc 486 kB/s | 32 kB 00:00 2024-05-12T21:54:40.827 INFO:teuthology.orchestra.run.smithi028.stdout:(116/156): python3-certifi-2021.10.8-2.el9.noar 303 kB/s | 15 kB 00:00 2024-05-12T21:54:40.886 INFO:teuthology.orchestra.run.smithi028.stdout:(117/156): python3-cherrypy-18.6.1-2.el9.noarch 6.1 MB/s | 358 kB 00:00 2024-05-12T21:54:40.928 INFO:teuthology.orchestra.run.smithi028.stdout:(118/156): python3-bcrypt-3.2.2-1.el9.x86_64.rp 186 kB/s | 43 kB 00:00 2024-05-12T21:54:40.962 INFO:teuthology.orchestra.run.smithi028.stdout:(119/156): python3-cheroot-8.6.0-4.el9.noarch.r 1.1 MB/s | 172 kB 00:00 2024-05-12T21:54:40.987 INFO:teuthology.orchestra.run.smithi028.stdout:(120/156): python3-colorama-0.4.4-12.el9.noarch 611 kB/s | 35 kB 00:00 2024-05-12T21:54:41.020 INFO:teuthology.orchestra.run.smithi028.stdout:(121/156): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.2 MB/s | 303 kB 00:00 2024-05-12T21:54:41.054 INFO:teuthology.orchestra.run.smithi028.stdout:(122/156): python3-google-auth-2.29.0-1.el9.noa 2.2 MB/s | 206 kB 00:00 2024-05-12T21:54:41.088 INFO:teuthology.orchestra.run.smithi028.stdout:(123/156): python3-grpcio-tools-1.46.7-10.el9.x 2.1 MB/s | 144 kB 00:00 2024-05-12T21:54:41.104 INFO:teuthology.orchestra.run.smithi028.stdout:(124/156): python3-jaraco-8.2.1-3.el9.noarch.rp 215 kB/s | 11 kB 00:00 2024-05-12T21:54:41.130 INFO:teuthology.orchestra.run.smithi028.stdout:(125/156): python3-jaraco-classes-3.2.1-5.el9.n 431 kB/s | 18 kB 00:00 2024-05-12T21:54:41.147 INFO:teuthology.orchestra.run.smithi028.stdout:(126/156): python3-jaraco-collections-3.0.0-8.e 561 kB/s | 23 kB 00:00 2024-05-12T21:54:41.180 INFO:teuthology.orchestra.run.smithi028.stdout:(127/156): python3-jaraco-functools-3.5.0-2.el9 393 kB/s | 19 kB 00:00 2024-05-12T21:54:41.206 INFO:teuthology.orchestra.run.smithi028.stdout:(128/156): python3-jaraco-text-3.2.0-6.el9.noar 339 kB/s | 20 kB 00:00 2024-05-12T21:54:41.273 INFO:teuthology.orchestra.run.smithi028.stdout:(129/156): python3-kubernetes-26.1.0-2.el9.noar 11 MB/s | 1.0 MB 00:00 2024-05-12T21:54:41.298 INFO:teuthology.orchestra.run.smithi028.stdout:(130/156): python3-logutils-0.3.5-21.el9.noarch 508 kB/s | 46 kB 00:00 2024-05-12T21:54:41.323 INFO:teuthology.orchestra.run.smithi028.stdout:(131/156): python3-more-itertools-8.12.0-2.el9. 1.6 MB/s | 79 kB 00:00 2024-05-12T21:54:41.390 INFO:teuthology.orchestra.run.smithi028.stdout:(132/156): python3-grpcio-1.46.7-10.el9.x86_64. 5.1 MB/s | 2.0 MB 00:00 2024-05-12T21:54:41.416 INFO:teuthology.orchestra.run.smithi028.stdout:(133/156): python3-natsort-7.1.1-5.el9.noarch.r 495 kB/s | 58 kB 00:00 2024-05-12T21:54:41.449 INFO:teuthology.orchestra.run.smithi028.stdout:(134/156): python3-pecan-1.4.2-3.el9.noarch.rpm 2.1 MB/s | 272 kB 00:00 2024-05-12T21:54:41.492 INFO:teuthology.orchestra.run.smithi028.stdout:(135/156): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.2 MB/s | 90 kB 00:00 2024-05-12T21:54:41.525 INFO:teuthology.orchestra.run.smithi028.stdout:(136/156): python3-pyperclip-1.8.2-2.el9.noarch 379 kB/s | 28 kB 00:00 2024-05-12T21:54:41.550 INFO:teuthology.orchestra.run.smithi028.stdout:(137/156): python3-repoze-lru-0.7-16.el9.noarch 537 kB/s | 31 kB 00:00 2024-05-12T21:54:41.584 INFO:teuthology.orchestra.run.smithi028.stdout:(138/156): python3-routes-2.5.1-5.el9.noarch.rp 3.2 MB/s | 188 kB 00:00 2024-05-12T21:54:41.609 INFO:teuthology.orchestra.run.smithi028.stdout:(139/156): python3-portend-3.1.0-2.el9.noarch.r 75 kB/s | 16 kB 00:00 2024-05-12T21:54:41.634 INFO:teuthology.orchestra.run.smithi028.stdout:(140/156): python3-rsa-4.9-2.el9.noarch.rpm 711 kB/s | 59 kB 00:00 2024-05-12T21:54:41.660 INFO:teuthology.orchestra.run.smithi028.stdout:(141/156): python3-tempora-5.0.0-2.el9.noarch.r 479 kB/s | 36 kB 00:00 2024-05-12T21:54:41.685 INFO:teuthology.orchestra.run.smithi028.stdout:(142/156): python3-typing-extensions-4.4.0-2.el 683 kB/s | 51 kB 00:00 2024-05-12T21:54:41.719 INFO:teuthology.orchestra.run.smithi028.stdout:(143/156): python3-webob-1.8.7-6.el9.noarch.rpm 2.7 MB/s | 230 kB 00:00 2024-05-12T21:54:41.761 INFO:teuthology.orchestra.run.smithi028.stdout:(144/156): python3-websocket-client-1.2.3-2.el9 900 kB/s | 90 kB 00:00 2024-05-12T21:54:41.794 INFO:teuthology.orchestra.run.smithi028.stdout:(145/156): python3-werkzeug-2.0.3-3.el9.1.noarc 3.9 MB/s | 427 kB 00:00 2024-05-12T21:54:41.819 INFO:teuthology.orchestra.run.smithi028.stdout:(146/156): python3-zc-lockfile-2.0-10.el9.noarc 200 kB/s | 20 kB 00:00 2024-05-12T21:54:41.861 INFO:teuthology.orchestra.run.smithi028.stdout:(147/156): re2-20211101-3.el9.x86_64.rpm 1.9 MB/s | 192 kB 00:00 2024-05-12T21:54:42.004 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-05-12T21:54:42.020 INFO:teuthology.orchestra.run.smithi028.stdout:(148/156): thrift-0.15.0-2.el9.x86_64.rpm 7.9 MB/s | 1.6 MB 00:00 2024-05-12T21:54:42.090 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/164 2024-05-12T21:54:42.128 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/164 2024-05-12T21:54:42.182 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/164 2024-05-12T21:54:42.216 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/164 2024-05-12T21:54:42.241 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/164 2024-05-12T21:54:42.283 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/164 2024-05-12T21:54:42.605 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/164 2024-05-12T21:54:42.612 INFO:teuthology.orchestra.run.smithi028.stdout:(149/156): librbd1-19.0.0-2555.gf72fecff.el9.x8 4.2 MB/s | 3.1 MB 00:00 2024-05-12T21:54:42.717 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : librados2-2:19.0.0-2555.gf72fecff.el9.x86_64 8/164 2024-05-12T21:54:42.830 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: librados2-2:19.0.0-2555.gf72fecff.el9.x86_64 8/164 2024-05-12T21:54:42.837 INFO:teuthology.orchestra.run.smithi028.stdout:(150/156): librados2-19.0.0-2555.gf72fecff.el9. 3.3 MB/s | 3.4 MB 00:01 2024-05-12T21:54:42.860 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libcephfs2-2:19.0.0-2555.gf72fecff.el9.x86_64 9/164 2024-05-12T21:54:42.951 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: libcephfs2-2:19.0.0-2555.gf72fecff.el9.x86_64 9/164 2024-05-12T21:54:42.981 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-rados-2:19.0.0-2555.gf72fecff.el9.x86_64 10/164 2024-05-12T21:54:43.038 INFO:teuthology.orchestra.run.smithi028.stdout:(151/156): NetworkManager-1.47.5-1.el9.x86_64.r 2.3 MB/s | 2.3 MB 00:01 2024-05-12T21:54:43.132 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/164 2024-05-12T21:54:43.154 INFO:teuthology.orchestra.run.smithi028.stdout:(152/156): NetworkManager-tui-1.47.5-1.el9.x86_ 2.1 MB/s | 248 kB 00:00 2024-05-12T21:54:43.178 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/164 2024-05-12T21:54:43.179 INFO:teuthology.orchestra.run.smithi028.stdout:(153/156): NetworkManager-team-1.47.5-1.el9.x86 122 kB/s | 42 kB 00:00 2024-05-12T21:54:43.205 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libcephsqlite-2:19.0.0-2555.gf72fecff.el9.x86_64 13/164 2024-05-12T21:54:43.309 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2555.gf72fecff.el9.x86_64 13/164 2024-05-12T21:54:43.331 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libradosstriper1-2:19.0.0-2555.gf72fecff.el9.x86 14/164 2024-05-12T21:54:43.372 INFO:teuthology.orchestra.run.smithi028.stdout:(154/156): openssl-3.2.1-1.el9.x86_64.rpm 6.2 MB/s | 1.3 MB 00:00 2024-05-12T21:54:43.394 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2555.gf72fecff.el9.x86 14/164 2024-05-12T21:54:43.416 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/164 2024-05-12T21:54:43.437 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/164 2024-05-12T21:54:43.563 INFO:teuthology.orchestra.run.smithi028.stdout:(155/156): NetworkManager-libnm-1.47.5-1.el9.x8 1.9 MB/s | 1.8 MB 00:00 2024-05-12T21:54:43.607 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/164 2024-05-12T21:54:43.734 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/164 2024-05-12T21:54:43.871 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : NetworkManager-libnm-1:1.47.5-1.el9.x86_64 17/164 2024-05-12T21:54:43.913 INFO:teuthology.orchestra.run.smithi028.stdout:(156/156): openssl-libs-3.2.1-1.el9.x86_64.rpm 3.3 MB/s | 2.4 MB 00:00 2024-05-12T21:54:43.920 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-05-12T21:54:43.921 INFO:teuthology.orchestra.run.smithi028.stdout:Total 11 MB/s | 225 MB 00:21 2024-05-12T21:54:43.932 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.5-1.el9.x86_64 17/164 2024-05-12T21:54:44.045 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: NetworkManager-1:1.47.5-1.el9.x86_64 18/164 2024-05-12T21:54:44.142 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : NetworkManager-1:1.47.5-1.el9.x86_64 18/164 2024-05-12T21:54:44.339 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: NetworkManager-1:1.47.5-1.el9.x86_64 18/164 2024-05-12T21:54:44.449 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : re2-1:20211101-3.el9.x86_64 19/164 2024-05-12T21:54:44.542 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 20/164 2024-05-12T21:54:44.582 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 21/164 2024-05-12T21:54:44.632 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : liboath-2.6.7-2.el9.x86_64 22/164 2024-05-12T21:54:44.724 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 23/164 2024-05-12T21:54:44.773 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 24/164 2024-05-12T21:54:44.816 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/164 2024-05-12T21:54:44.848 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 26/164 2024-05-12T21:54:44.876 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : lua-5.4.4-4.el9.x86_64 27/164 2024-05-12T21:54:44.945 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 28/164 2024-05-12T21:54:44.998 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 29/164 2024-05-12T21:54:45.029 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 30/164 2024-05-12T21:54:45.064 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 31/164 2024-05-12T21:54:45.142 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 32/164 2024-05-12T21:54:45.175 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-ceph-common-2:19.0.0-2555.gf72fecff.el9. 33/164 2024-05-12T21:54:45.218 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-ceph-argparse-2:19.0.0-2555.gf72fecff.el 34/164 2024-05-12T21:54:45.247 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-cephfs-2:19.0.0-2555.gf72fecff.el9.x86_6 35/164 2024-05-12T21:54:45.318 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 36/164 2024-05-12T21:54:45.362 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : luarocks-3.9.2-1.el9.noarch 37/164 2024-05-12T21:54:45.434 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-05-12T21:54:45.436 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 38/164 2024-05-12T21:54:45.524 INFO:teuthology.orchestra.run.smithi086.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/164 2024-05-12T21:54:45.562 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/164 2024-05-12T21:54:45.580 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 39/164 2024-05-12T21:54:45.624 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/164 2024-05-12T21:54:45.631 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 40/164 2024-05-12T21:54:45.666 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/164 2024-05-12T21:54:45.693 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/164 2024-05-12T21:54:45.735 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/164 2024-05-12T21:54:46.133 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/164 2024-05-12T21:54:46.164 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-rsa-4.9-2.el9.noarch 41/164 2024-05-12T21:54:46.218 INFO:teuthology.orchestra.run.smithi086.stdout: Upgrading : librados2-2:19.0.0-2555.gf72fecff.el9.x86_64 8/164 2024-05-12T21:54:46.251 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 42/164 2024-05-12T21:54:46.280 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 43/164 2024-05-12T21:54:46.323 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 44/164 2024-05-12T21:54:46.339 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librados2-2:19.0.0-2555.gf72fecff.el9.x86_64 8/164 2024-05-12T21:54:46.361 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : librados-devel-2:19.0.0-2555.gf72fecff.el9.x86_6 45/164 2024-05-12T21:54:46.369 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libcephfs2-2:19.0.0-2555.gf72fecff.el9.x86_64 9/164 2024-05-12T21:54:46.402 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 46/164 2024-05-12T21:54:46.459 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: libcephfs2-2:19.0.0-2555.gf72fecff.el9.x86_64 9/164 2024-05-12T21:54:46.490 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-rados-2:19.0.0-2555.gf72fecff.el9.x86_64 10/164 2024-05-12T21:54:46.564 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 47/164 2024-05-12T21:54:46.614 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 48/164 2024-05-12T21:54:46.637 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/164 2024-05-12T21:54:46.669 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 49/164 2024-05-12T21:54:46.695 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/164 2024-05-12T21:54:46.723 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libcephsqlite-2:19.0.0-2555.gf72fecff.el9.x86_64 13/164 2024-05-12T21:54:46.742 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 50/164 2024-05-12T21:54:46.773 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 51/164 2024-05-12T21:54:46.809 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 52/164 2024-05-12T21:54:46.817 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2555.gf72fecff.el9.x86_64 13/164 2024-05-12T21:54:46.841 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libradosstriper1-2:19.0.0-2555.gf72fecff.el9.x86 14/164 2024-05-12T21:54:46.856 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 53/164 2024-05-12T21:54:46.886 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2555.gf72fecff.el9.x86 14/164 2024-05-12T21:54:46.892 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 54/164 2024-05-12T21:54:46.909 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/164 2024-05-12T21:54:46.929 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/164 2024-05-12T21:54:46.934 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 55/164 2024-05-12T21:54:46.976 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 56/164 2024-05-12T21:54:47.105 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2024-05-12T21:54:47.112 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 57/164 2024-05-12T21:54:47.116 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/164 2024-05-12T21:54:47.166 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 58/164 2024-05-12T21:54:47.196 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 59/164 2024-05-12T21:54:47.237 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 60/164 2024-05-12T21:54:47.273 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/164 2024-05-12T21:54:47.332 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 61/164 2024-05-12T21:54:47.370 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2024-05-12T21:54:47.370 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2024-05-12T21:54:47.371 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 62/164 2024-05-12T21:54:47.410 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 63/164 2024-05-12T21:54:47.431 INFO:teuthology.orchestra.run.smithi086.stdout: Upgrading : NetworkManager-libnm-1:1.47.5-1.el9.x86_64 17/164 2024-05-12T21:54:47.436 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 64/164 2024-05-12T21:54:47.460 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 65/164 2024-05-12T21:54:47.491 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.5-1.el9.x86_64 17/164 2024-05-12T21:54:47.536 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 66/164 2024-05-12T21:54:47.619 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: NetworkManager-1:1.47.5-1.el9.x86_64 18/164 2024-05-12T21:54:47.693 INFO:teuthology.orchestra.run.smithi086.stdout: Upgrading : NetworkManager-1:1.47.5-1.el9.x86_64 18/164 2024-05-12T21:54:47.750 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 67/164 2024-05-12T21:54:47.876 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 68/164 2024-05-12T21:54:47.900 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: NetworkManager-1:1.47.5-1.el9.x86_64 18/164 2024-05-12T21:54:47.904 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 69/164 2024-05-12T21:54:47.938 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 70/164 2024-05-12T21:54:47.977 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : uuid-1.6.2-55.el9.x86_64 71/164 2024-05-12T21:54:48.009 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 72/164 2024-05-12T21:54:48.028 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : re2-1:20211101-3.el9.x86_64 19/164 2024-05-12T21:54:48.039 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 73/164 2024-05-12T21:54:48.073 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 74/164 2024-05-12T21:54:48.107 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 75/164 2024-05-12T21:54:48.118 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 20/164 2024-05-12T21:54:48.148 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 76/164 2024-05-12T21:54:48.149 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 21/164 2024-05-12T21:54:48.182 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 77/164 2024-05-12T21:54:48.183 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : liboath-2.6.7-2.el9.x86_64 22/164 2024-05-12T21:54:48.248 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 23/164 2024-05-12T21:54:48.270 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 78/164 2024-05-12T21:54:48.282 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 24/164 2024-05-12T21:54:48.313 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 79/164 2024-05-12T21:54:48.326 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/164 2024-05-12T21:54:48.344 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 80/164 2024-05-12T21:54:48.358 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 26/164 2024-05-12T21:54:48.386 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : lua-5.4.4-4.el9.x86_64 27/164 2024-05-12T21:54:48.391 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 81/164 2024-05-12T21:54:48.454 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 28/164 2024-05-12T21:54:48.490 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-packaging-20.9-5.el9.noarch 82/164 2024-05-12T21:54:48.508 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 29/164 2024-05-12T21:54:48.537 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 83/164 2024-05-12T21:54:48.539 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 30/164 2024-05-12T21:54:48.574 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 31/164 2024-05-12T21:54:48.642 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 32/164 2024-05-12T21:54:48.677 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-ceph-common-2:19.0.0-2555.gf72fecff.el9. 33/164 2024-05-12T21:54:48.720 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-ceph-argparse-2:19.0.0-2555.gf72fecff.el 34/164 2024-05-12T21:54:48.757 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-cephfs-2:19.0.0-2555.gf72fecff.el9.x86_6 35/164 2024-05-12T21:54:48.827 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 36/164 2024-05-12T21:54:48.872 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : luarocks-3.9.2-1.el9.noarch 37/164 2024-05-12T21:54:48.946 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 38/164 2024-05-12T21:54:49.078 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 39/164 2024-05-12T21:54:49.133 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 40/164 2024-05-12T21:54:49.268 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 84/164 2024-05-12T21:54:49.318 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 85/164 2024-05-12T21:54:49.351 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 86/164 2024-05-12T21:54:49.452 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 87/164 2024-05-12T21:54:49.510 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 88/164 2024-05-12T21:54:49.598 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 89/164 2024-05-12T21:54:49.629 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 90/164 2024-05-12T21:54:49.640 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-rsa-4.9-2.el9.noarch 41/164 2024-05-12T21:54:49.719 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 42/164 2024-05-12T21:54:49.749 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 43/164 2024-05-12T21:54:49.792 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 44/164 2024-05-12T21:54:49.808 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : openblas-0.3.21-2.el9.x86_64 91/164 2024-05-12T21:54:49.835 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 92/164 2024-05-12T21:54:49.838 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : librados-devel-2:19.0.0-2555.gf72fecff.el9.x86_6 45/164 2024-05-12T21:54:49.880 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 46/164 2024-05-12T21:54:49.904 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 93/164 2024-05-12T21:54:50.027 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 47/164 2024-05-12T21:54:50.082 INFO:teuthology.orchestra.run.smithi086.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 48/164 2024-05-12T21:54:50.088 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2024-05-12T21:54:50.089 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2024-05-12T21:54:50.241 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 49/164 2024-05-12T21:54:50.486 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 50/164 2024-05-12T21:54:50.525 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 51/164 2024-05-12T21:54:50.561 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 52/164 2024-05-12T21:54:50.565 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 94/164 2024-05-12T21:54:50.608 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 53/164 2024-05-12T21:54:50.644 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 54/164 2024-05-12T21:54:50.687 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 55/164 2024-05-12T21:54:50.701 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 95/164 2024-05-12T21:54:50.729 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 56/164 2024-05-12T21:54:50.874 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 57/164 2024-05-12T21:54:50.927 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 58/164 2024-05-12T21:54:50.956 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 59/164 2024-05-12T21:54:50.998 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 60/164 2024-05-12T21:54:51.092 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 61/164 2024-05-12T21:54:51.131 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 62/164 2024-05-12T21:54:51.171 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 63/164 2024-05-12T21:54:51.197 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 64/164 2024-05-12T21:54:51.221 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 65/164 2024-05-12T21:54:51.296 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 66/164 2024-05-12T21:54:51.423 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 67/164 2024-05-12T21:54:51.462 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 68/164 2024-05-12T21:54:51.490 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 69/164 2024-05-12T21:54:51.524 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 70/164 2024-05-12T21:54:51.564 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : uuid-1.6.2-55.el9.x86_64 71/164 2024-05-12T21:54:51.595 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 72/164 2024-05-12T21:54:51.625 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 73/164 2024-05-12T21:54:51.751 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 74/164 2024-05-12T21:54:51.785 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 75/164 2024-05-12T21:54:51.835 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 76/164 2024-05-12T21:54:51.869 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 77/164 2024-05-12T21:54:51.943 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 96/164 2024-05-12T21:54:51.955 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 78/164 2024-05-12T21:54:52.008 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 79/164 2024-05-12T21:54:52.022 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 97/164 2024-05-12T21:54:52.039 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 80/164 2024-05-12T21:54:52.050 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 98/164 2024-05-12T21:54:52.082 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 99/164 2024-05-12T21:54:52.085 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 81/164 2024-05-12T21:54:52.183 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-packaging-20.9-5.el9.noarch 82/164 2024-05-12T21:54:52.231 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 83/164 2024-05-12T21:54:52.391 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 100/164 2024-05-12T21:54:52.429 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : librbd1-2:19.0.0-2555.gf72fecff.el9.x86_64 101/164 2024-05-12T21:54:52.519 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: librbd1-2:19.0.0-2555.gf72fecff.el9.x86_64 101/164 2024-05-12T21:54:52.559 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-rbd-2:19.0.0-2555.gf72fecff.el9.x86_64 102/164 2024-05-12T21:54:52.594 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 103/164 2024-05-12T21:54:52.945 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 84/164 2024-05-12T21:54:52.988 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 85/164 2024-05-12T21:54:53.021 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 86/164 2024-05-12T21:54:53.068 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 104/164 2024-05-12T21:54:53.090 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : librgw2-2:19.0.0-2555.gf72fecff.el9.x86_64 105/164 2024-05-12T21:54:53.124 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 87/164 2024-05-12T21:54:53.160 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: librgw2-2:19.0.0-2555.gf72fecff.el9.x86_64 105/164 2024-05-12T21:54:53.179 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 88/164 2024-05-12T21:54:53.183 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-rgw-2:19.0.0-2555.gf72fecff.el9.x86_64 106/164 2024-05-12T21:54:53.264 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 89/164 2024-05-12T21:54:53.298 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 90/164 2024-05-12T21:54:53.475 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : openblas-0.3.21-2.el9.x86_64 91/164 2024-05-12T21:54:53.504 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 92/164 2024-05-12T21:54:53.581 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 93/164 2024-05-12T21:54:54.241 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 94/164 2024-05-12T21:54:54.360 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 95/164 2024-05-12T21:54:55.436 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-common-2:19.0.0-2555.gf72fecff.el9.x86_64 107/164 2024-05-12T21:54:55.471 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-common-2:19.0.0-2555.gf72fecff.el9.x86_64 107/164 2024-05-12T21:54:55.565 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 96/164 2024-05-12T21:54:55.649 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 97/164 2024-05-12T21:54:55.702 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 98/164 2024-05-12T21:54:55.735 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 99/164 2024-05-12T21:54:56.014 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-common-2:19.0.0-2555.gf72fecff.el9.x86_64 107/164 2024-05-12T21:54:56.041 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 100/164 2024-05-12T21:54:56.060 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-base-2:19.0.0-2555.gf72fecff.el9.x86_64 108/164 2024-05-12T21:54:56.082 INFO:teuthology.orchestra.run.smithi086.stdout: Upgrading : librbd1-2:19.0.0-2555.gf72fecff.el9.x86_64 101/164 2024-05-12T21:54:56.150 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-base-2:19.0.0-2555.gf72fecff.el9.x86_64 108/164 2024-05-12T21:54:56.150 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-05-12T21:54:56.150 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-05-12T21:54:56.151 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-12T21:54:56.155 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librbd1-2:19.0.0-2555.gf72fecff.el9.x86_64 101/164 2024-05-12T21:54:56.177 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-selinux-2:19.0.0-2555.gf72fecff.el9.x86_64 109/164 2024-05-12T21:54:56.186 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-rbd-2:19.0.0-2555.gf72fecff.el9.x86_64 102/164 2024-05-12T21:54:56.223 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 103/164 2024-05-12T21:54:56.692 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 104/164 2024-05-12T21:54:56.718 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : librgw2-2:19.0.0-2555.gf72fecff.el9.x86_64 105/164 2024-05-12T21:54:56.789 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librgw2-2:19.0.0-2555.gf72fecff.el9.x86_64 105/164 2024-05-12T21:54:56.811 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-rgw-2:19.0.0-2555.gf72fecff.el9.x86_64 106/164 2024-05-12T21:54:56.951 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2024-05-12T21:54:57.054 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/164 2024-05-12T21:54:57.146 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/164 2024-05-12T21:54:57.228 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/164 2024-05-12T21:54:57.278 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/164 2024-05-12T21:54:57.314 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/164 2024-05-12T21:54:57.408 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/164 2024-05-12T21:54:57.863 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/164 2024-05-12T21:54:57.971 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : librados2-2:19.0.0-2555.gf72fecff.el9.x86_64 8/164 2024-05-12T21:54:58.159 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: librados2-2:19.0.0-2555.gf72fecff.el9.x86_64 8/164 2024-05-12T21:54:58.205 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libcephfs2-2:19.0.0-2555.gf72fecff.el9.x86_64 9/164 2024-05-12T21:54:58.337 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: libcephfs2-2:19.0.0-2555.gf72fecff.el9.x86_64 9/164 2024-05-12T21:54:58.373 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-rados-2:19.0.0-2555.gf72fecff.el9.x86_64 10/164 2024-05-12T21:54:58.711 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/164 2024-05-12T21:54:58.776 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/164 2024-05-12T21:54:58.801 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libcephsqlite-2:19.0.0-2555.gf72fecff.el9.x86_64 13/164 2024-05-12T21:54:58.949 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2555.gf72fecff.el9.x86_64 13/164 2024-05-12T21:54:58.977 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libradosstriper1-2:19.0.0-2555.gf72fecff.el9.x86 14/164 2024-05-12T21:54:59.038 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-common-2:19.0.0-2555.gf72fecff.el9.x86_64 107/164 2024-05-12T21:54:59.072 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2555.gf72fecff.el9.x86 14/164 2024-05-12T21:54:59.083 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-common-2:19.0.0-2555.gf72fecff.el9.x86_64 107/164 2024-05-12T21:54:59.103 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/164 2024-05-12T21:54:59.190 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/164 2024-05-12T21:54:59.503 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/164 2024-05-12T21:54:59.615 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-common-2:19.0.0-2555.gf72fecff.el9.x86_64 107/164 2024-05-12T21:54:59.652 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-base-2:19.0.0-2555.gf72fecff.el9.x86_64 108/164 2024-05-12T21:54:59.743 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-base-2:19.0.0-2555.gf72fecff.el9.x86_64 108/164 2024-05-12T21:54:59.744 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-05-12T21:54:59.744 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-05-12T21:54:59.744 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-05-12T21:54:59.770 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-selinux-2:19.0.0-2555.gf72fecff.el9.x86_64 109/164 2024-05-12T21:54:59.843 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/164 2024-05-12T21:54:59.985 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : NetworkManager-libnm-1:1.47.5-1.el9.x86_64 17/164 2024-05-12T21:55:00.073 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.5-1.el9.x86_64 17/164 2024-05-12T21:55:00.269 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: NetworkManager-1:1.47.5-1.el9.x86_64 18/164 2024-05-12T21:55:00.371 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : NetworkManager-1:1.47.5-1.el9.x86_64 18/164 2024-05-12T21:55:00.703 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: NetworkManager-1:1.47.5-1.el9.x86_64 18/164 2024-05-12T21:55:00.849 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : re2-1:20211101-3.el9.x86_64 19/164 2024-05-12T21:55:00.983 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 20/164 2024-05-12T21:55:01.025 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 21/164 2024-05-12T21:55:01.076 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : liboath-2.6.7-2.el9.x86_64 22/164 2024-05-12T21:55:01.184 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 23/164 2024-05-12T21:55:01.233 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 24/164 2024-05-12T21:55:01.297 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/164 2024-05-12T21:55:01.345 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 26/164 2024-05-12T21:55:01.385 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : lua-5.4.4-4.el9.x86_64 27/164 2024-05-12T21:55:01.505 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 28/164 2024-05-12T21:55:01.574 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 29/164 2024-05-12T21:55:01.611 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 30/164 2024-05-12T21:55:01.667 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 31/164 2024-05-12T21:55:01.780 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 32/164 2024-05-12T21:55:01.816 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-ceph-common-2:19.0.0-2555.gf72fecff.el9. 33/164 2024-05-12T21:55:01.866 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-ceph-argparse-2:19.0.0-2555.gf72fecff.el 34/164 2024-05-12T21:55:01.897 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cephfs-2:19.0.0-2555.gf72fecff.el9.x86_6 35/164 2024-05-12T21:55:02.017 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 36/164 2024-05-12T21:55:02.086 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : luarocks-3.9.2-1.el9.noarch 37/164 2024-05-12T21:55:02.201 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 38/164 2024-05-12T21:55:02.402 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 39/164 2024-05-12T21:55:02.471 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 40/164 2024-05-12T21:55:03.163 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-rsa-4.9-2.el9.noarch 41/164 2024-05-12T21:55:03.270 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 42/164 2024-05-12T21:55:03.312 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 43/164 2024-05-12T21:55:03.358 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 44/164 2024-05-12T21:55:03.398 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : librados-devel-2:19.0.0-2555.gf72fecff.el9.x86_6 45/164 2024-05-12T21:55:03.469 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 46/164 2024-05-12T21:55:03.686 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 47/164 2024-05-12T21:55:03.764 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 48/164 2024-05-12T21:55:03.845 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 49/164 2024-05-12T21:55:03.950 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 50/164 2024-05-12T21:55:03.990 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 51/164 2024-05-12T21:55:04.030 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 52/164 2024-05-12T21:55:04.102 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 53/164 2024-05-12T21:55:04.154 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 54/164 2024-05-12T21:55:04.219 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 55/164 2024-05-12T21:55:04.311 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 56/164 2024-05-12T21:55:04.526 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 57/164 2024-05-12T21:55:04.584 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 58/164 2024-05-12T21:55:04.623 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 59/164 2024-05-12T21:55:04.681 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 60/164 2024-05-12T21:55:04.824 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 61/164 2024-05-12T21:55:04.871 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 62/164 2024-05-12T21:55:04.919 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 63/164 2024-05-12T21:55:04.954 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 64/164 2024-05-12T21:55:04.985 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 65/164 2024-05-12T21:55:05.158 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 66/164 2024-05-12T21:55:05.347 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 67/164 2024-05-12T21:55:05.401 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 68/164 2024-05-12T21:55:05.448 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 69/164 2024-05-12T21:55:05.505 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 70/164 2024-05-12T21:55:05.575 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : uuid-1.6.2-55.el9.x86_64 71/164 2024-05-12T21:55:05.633 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 72/164 2024-05-12T21:55:05.690 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 73/164 2024-05-12T21:55:05.746 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 74/164 2024-05-12T21:55:05.795 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 75/164 2024-05-12T21:55:05.873 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 76/164 2024-05-12T21:55:05.916 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 77/164 2024-05-12T21:55:06.064 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 78/164 2024-05-12T21:55:06.138 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 79/164 2024-05-12T21:55:06.182 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 80/164 2024-05-12T21:55:06.256 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 81/164 2024-05-12T21:55:06.416 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-packaging-20.9-5.el9.noarch 82/164 2024-05-12T21:55:06.478 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 83/164 2024-05-12T21:55:07.493 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 84/164 2024-05-12T21:55:07.552 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 85/164 2024-05-12T21:55:07.585 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 86/164 2024-05-12T21:55:07.717 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 87/164 2024-05-12T21:55:07.737 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2555.gf72fecff.el9.x86_64 109/164 2024-05-12T21:55:07.737 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /sys 2024-05-12T21:55:07.737 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /proc 2024-05-12T21:55:07.737 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /mnt 2024-05-12T21:55:07.737 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /var/tmp 2024-05-12T21:55:07.737 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /home 2024-05-12T21:55:07.737 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /root 2024-05-12T21:55:07.737 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /tmp 2024-05-12T21:55:07.737 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-12T21:55:07.782 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 88/164 2024-05-12T21:55:07.891 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 89/164 2024-05-12T21:55:07.909 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-volume-2:19.0.0-2555.gf72fecff.el9.noarch 110/164 2024-05-12T21:55:07.925 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-volume-2:19.0.0-2555.gf72fecff.el9.noarch 110/164 2024-05-12T21:55:07.925 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-12T21:55:07.925 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-05-12T21:55:07.925 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-12T21:55:07.928 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 90/164 2024-05-12T21:55:08.171 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : openblas-0.3.21-2.el9.x86_64 91/164 2024-05-12T21:55:08.192 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 92/164 2024-05-12T21:55:08.296 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 93/164 2024-05-12T21:55:09.187 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 94/164 2024-05-12T21:55:09.346 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 95/164 2024-05-12T21:55:09.412 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-osd-2:19.0.0-2555.gf72fecff.el9.x86_64 111/164 2024-05-12T21:55:09.438 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-osd-2:19.0.0-2555.gf72fecff.el9.x86_64 111/164 2024-05-12T21:55:09.438 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-12T21:55:09.438 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-05-12T21:55:09.438 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-12T21:55:09.438 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-12T21:55:09.438 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-12T21:55:09.671 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mds-2:19.0.0-2555.gf72fecff.el9.x86_64 112/164 2024-05-12T21:55:09.693 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mds-2:19.0.0-2555.gf72fecff.el9.x86_64 112/164 2024-05-12T21:55:09.694 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-12T21:55:09.694 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-05-12T21:55:09.694 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-12T21:55:09.694 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-12T21:55:09.694 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-12T21:55:10.138 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mon-2:19.0.0-2555.gf72fecff.el9.x86_64 113/164 2024-05-12T21:55:10.160 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mon-2:19.0.0-2555.gf72fecff.el9.x86_64 113/164 2024-05-12T21:55:10.160 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-12T21:55:10.161 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-05-12T21:55:10.161 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-12T21:55:10.161 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-12T21:55:10.161 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-12T21:55:10.269 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : automake-1.16.2-8.el9.noarch 114/164 2024-05-12T21:55:10.326 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-ply-3.11-14.el9.noarch 115/164 2024-05-12T21:55:10.382 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 116/164 2024-05-12T21:55:10.577 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 117/164 2024-05-12T21:55:10.623 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 118/164 2024-05-12T21:55:10.702 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 119/164 2024-05-12T21:55:10.791 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 120/164 2024-05-12T21:55:10.940 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 121/164 2024-05-12T21:55:10.981 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 122/164 2024-05-12T21:55:11.022 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 123/164 2024-05-12T21:55:11.049 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : fuse-2.9.9-15.el9.x86_64 124/164 2024-05-12T21:55:11.091 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 96/164 2024-05-12T21:55:11.199 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 97/164 2024-05-12T21:55:11.236 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 98/164 2024-05-12T21:55:11.270 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 99/164 2024-05-12T21:55:11.443 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2555.gf72fecff.el9.x86_64 109/164 2024-05-12T21:55:11.443 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /sys 2024-05-12T21:55:11.443 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /proc 2024-05-12T21:55:11.443 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /mnt 2024-05-12T21:55:11.443 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /var/tmp 2024-05-12T21:55:11.443 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /home 2024-05-12T21:55:11.443 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /root 2024-05-12T21:55:11.443 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /tmp 2024-05-12T21:55:11.443 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-05-12T21:55:11.502 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: cephadm-2:19.0.0-2555.gf72fecff.el9.noarch 125/164 2024-05-12T21:55:11.531 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : cephadm-2:19.0.0-2555.gf72fecff.el9.noarch 125/164 2024-05-12T21:55:11.571 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2555.gf72fecff.e 126/164 2024-05-12T21:55:11.610 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-volume-2:19.0.0-2555.gf72fecff.el9.noarch 110/164 2024-05-12T21:55:11.625 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-volume-2:19.0.0-2555.gf72fecff.el9.noarch 110/164 2024-05-12T21:55:11.625 INFO:teuthology.orchestra.run.smithi086.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-12T21:55:11.625 INFO:teuthology.orchestra.run.smithi086.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-05-12T21:55:11.625 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-05-12T21:55:11.634 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2555.gf72fecff. 127/164 2024-05-12T21:55:11.659 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 100/164 2024-05-12T21:55:11.671 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2555.gf72fecff.el9.noa 128/164 2024-05-12T21:55:11.703 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : librbd1-2:19.0.0-2555.gf72fecff.el9.x86_64 101/164 2024-05-12T21:55:11.827 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: librbd1-2:19.0.0-2555.gf72fecff.el9.x86_64 101/164 2024-05-12T21:55:11.860 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-rbd-2:19.0.0-2555.gf72fecff.el9.x86_64 102/164 2024-05-12T21:55:11.923 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 103/164 2024-05-12T21:55:12.498 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 104/164 2024-05-12T21:55:12.530 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : librgw2-2:19.0.0-2555.gf72fecff.el9.x86_64 105/164 2024-05-12T21:55:12.648 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: librgw2-2:19.0.0-2555.gf72fecff.el9.x86_64 105/164 2024-05-12T21:55:12.674 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-rgw-2:19.0.0-2555.gf72fecff.el9.x86_64 106/164 2024-05-12T21:55:12.714 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2555.gf72fecff.el9.noa 128/164 2024-05-12T21:55:12.756 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2555.gf72fecff.el9.n 129/164 2024-05-12T21:55:13.087 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-osd-2:19.0.0-2555.gf72fecff.el9.x86_64 111/164 2024-05-12T21:55:13.112 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-osd-2:19.0.0-2555.gf72fecff.el9.x86_64 111/164 2024-05-12T21:55:13.113 INFO:teuthology.orchestra.run.smithi086.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-12T21:55:13.113 INFO:teuthology.orchestra.run.smithi086.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-05-12T21:55:13.113 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-12T21:55:13.113 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-12T21:55:13.113 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-05-12T21:55:13.338 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mds-2:19.0.0-2555.gf72fecff.el9.x86_64 112/164 2024-05-12T21:55:13.361 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mds-2:19.0.0-2555.gf72fecff.el9.x86_64 112/164 2024-05-12T21:55:13.361 INFO:teuthology.orchestra.run.smithi086.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-12T21:55:13.361 INFO:teuthology.orchestra.run.smithi086.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-05-12T21:55:13.361 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-12T21:55:13.362 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-12T21:55:13.362 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-05-12T21:55:13.728 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2555.gf72fecff.el9.n 129/164 2024-05-12T21:55:13.816 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mon-2:19.0.0-2555.gf72fecff.el9.x86_64 113/164 2024-05-12T21:55:13.838 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mon-2:19.0.0-2555.gf72fecff.el9.x86_64 113/164 2024-05-12T21:55:13.839 INFO:teuthology.orchestra.run.smithi086.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-12T21:55:13.839 INFO:teuthology.orchestra.run.smithi086.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-05-12T21:55:13.839 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-12T21:55:13.839 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-12T21:55:13.839 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-05-12T21:55:13.937 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : automake-1.16.2-8.el9.noarch 114/164 2024-05-12T21:55:13.941 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2555.gf72 130/164 2024-05-12T21:55:13.986 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-ply-3.11-14.el9.noarch 115/164 2024-05-12T21:55:14.045 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2555.gf72 130/164 2024-05-12T21:55:14.049 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 116/164 2024-05-12T21:55:14.213 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2555.gf72fecff.el 131/164 2024-05-12T21:55:14.231 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 117/164 2024-05-12T21:55:14.242 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mgr-2:19.0.0-2555.gf72fecff.el9.x86_64 132/164 2024-05-12T21:55:14.265 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2555.gf72fecff.el9.x86_64 132/164 2024-05-12T21:55:14.265 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-12T21:55:14.265 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-05-12T21:55:14.265 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-12T21:55:14.266 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-12T21:55:14.266 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-12T21:55:14.283 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 118/164 2024-05-12T21:55:14.309 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mgr-rook-2:19.0.0-2555.gf72fecff.el9.noarch 133/164 2024-05-12T21:55:14.321 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2555.gf72fecff.el9.noarch 133/164 2024-05-12T21:55:14.351 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 119/164 2024-05-12T21:55:14.418 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-2:19.0.0-2555.gf72fecff.el9.x86_64 134/164 2024-05-12T21:55:14.439 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 120/164 2024-05-12T21:55:14.466 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-fuse-2:19.0.0-2555.gf72fecff.el9.x86_64 135/164 2024-05-12T21:55:14.570 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 121/164 2024-05-12T21:55:14.608 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 122/164 2024-05-12T21:55:14.648 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 123/164 2024-05-12T21:55:14.678 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : fuse-2.9.9-15.el9.x86_64 124/164 2024-05-12T21:55:15.070 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: cephadm-2:19.0.0-2555.gf72fecff.el9.noarch 125/164 2024-05-12T21:55:15.100 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : cephadm-2:19.0.0-2555.gf72fecff.el9.noarch 125/164 2024-05-12T21:55:15.131 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2555.gf72fecff.e 126/164 2024-05-12T21:55:15.193 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2555.gf72fecff. 127/164 2024-05-12T21:55:15.223 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2555.gf72fecff.el9.noa 128/164 2024-05-12T21:55:15.381 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libtool-2.4.6-45.el9.x86_64 136/164 2024-05-12T21:55:15.420 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-radosgw-2:19.0.0-2555.gf72fecff.el9.x86_64 137/164 2024-05-12T21:55:15.443 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2555.gf72fecff.el9.x86_64 137/164 2024-05-12T21:55:15.444 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-12T21:55:15.444 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-05-12T21:55:15.444 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-12T21:55:15.444 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-12T21:55:15.444 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-12T21:55:15.472 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-common-2:19.0.0-2555.gf72fecff.el9.x86_64 107/164 2024-05-12T21:55:15.487 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2555.gf72fe 138/164 2024-05-12T21:55:15.509 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2555.gf72fe 138/164 2024-05-12T21:55:15.510 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-12T21:55:15.510 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-05-12T21:55:15.510 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-12T21:55:15.512 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-common-2:19.0.0-2555.gf72fecff.el9.x86_64 107/164 2024-05-12T21:55:15.554 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : cephfs-mirror-2:19.0.0-2555.gf72fecff.el9.x86_64 139/164 2024-05-12T21:55:15.577 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2555.gf72fecff.el9.x86_64 139/164 2024-05-12T21:55:15.577 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-12T21:55:15.577 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-05-12T21:55:15.577 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-05-12T21:55:15.577 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-05-12T21:55:15.578 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-12T21:55:15.880 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : rbd-mirror-2:19.0.0-2555.gf72fecff.el9.x86_64 140/164 2024-05-12T21:55:15.903 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2555.gf72fecff.el9.x86_64 140/164 2024-05-12T21:55:15.903 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-12T21:55:15.903 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-05-12T21:55:15.903 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-12T21:55:15.903 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-12T21:55:15.903 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-12T21:55:16.186 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-common-2:19.0.0-2555.gf72fecff.el9.x86_64 107/164 2024-05-12T21:55:16.232 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-base-2:19.0.0-2555.gf72fecff.el9.x86_64 108/164 2024-05-12T21:55:16.243 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2555.gf72fecff.el9.noa 128/164 2024-05-12T21:55:16.275 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2555.gf72fecff.el9.n 129/164 2024-05-12T21:55:16.406 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-base-2:19.0.0-2555.gf72fecff.el9.x86_64 108/164 2024-05-12T21:55:16.406 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-05-12T21:55:16.406 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-05-12T21:55:16.406 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-12T21:55:16.450 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-selinux-2:19.0.0-2555.gf72fecff.el9.x86_64 109/164 2024-05-12T21:55:17.235 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2555.gf72fecff.el9.n 129/164 2024-05-12T21:55:17.258 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2555.gf72 130/164 2024-05-12T21:55:17.359 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2555.gf72 130/164 2024-05-12T21:55:17.529 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2555.gf72fecff.el 131/164 2024-05-12T21:55:17.560 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mgr-2:19.0.0-2555.gf72fecff.el9.x86_64 132/164 2024-05-12T21:55:17.583 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2555.gf72fecff.el9.x86_64 132/164 2024-05-12T21:55:17.583 INFO:teuthology.orchestra.run.smithi086.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-12T21:55:17.583 INFO:teuthology.orchestra.run.smithi086.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-05-12T21:55:17.583 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-12T21:55:17.583 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-12T21:55:17.583 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-05-12T21:55:17.627 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mgr-rook-2:19.0.0-2555.gf72fecff.el9.noarch 133/164 2024-05-12T21:55:17.638 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2555.gf72fecff.el9.noarch 133/164 2024-05-12T21:55:17.736 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-2:19.0.0-2555.gf72fecff.el9.x86_64 134/164 2024-05-12T21:55:17.784 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-fuse-2:19.0.0-2555.gf72fecff.el9.x86_64 135/164 2024-05-12T21:55:18.682 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libtool-2.4.6-45.el9.x86_64 136/164 2024-05-12T21:55:18.713 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-radosgw-2:19.0.0-2555.gf72fecff.el9.x86_64 137/164 2024-05-12T21:55:18.735 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2555.gf72fecff.el9.x86_64 137/164 2024-05-12T21:55:18.735 INFO:teuthology.orchestra.run.smithi086.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-12T21:55:18.736 INFO:teuthology.orchestra.run.smithi086.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-05-12T21:55:18.736 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-12T21:55:18.736 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-12T21:55:18.736 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-05-12T21:55:18.781 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2555.gf72fe 138/164 2024-05-12T21:55:18.802 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2555.gf72fe 138/164 2024-05-12T21:55:18.802 INFO:teuthology.orchestra.run.smithi086.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-12T21:55:18.802 INFO:teuthology.orchestra.run.smithi086.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-05-12T21:55:18.802 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-05-12T21:55:18.848 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : cephfs-mirror-2:19.0.0-2555.gf72fecff.el9.x86_64 139/164 2024-05-12T21:55:18.870 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2555.gf72fecff.el9.x86_64 139/164 2024-05-12T21:55:18.870 INFO:teuthology.orchestra.run.smithi086.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-12T21:55:18.870 INFO:teuthology.orchestra.run.smithi086.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-05-12T21:55:18.870 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-05-12T21:55:18.870 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-05-12T21:55:18.870 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-05-12T21:55:19.174 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : rbd-mirror-2:19.0.0-2555.gf72fecff.el9.x86_64 140/164 2024-05-12T21:55:19.196 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2555.gf72fecff.el9.x86_64 140/164 2024-05-12T21:55:19.196 INFO:teuthology.orchestra.run.smithi086.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-12T21:55:19.196 INFO:teuthology.orchestra.run.smithi086.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-05-12T21:55:19.196 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-12T21:55:19.196 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-12T21:55:19.196 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-05-12T21:55:20.426 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-test-2:19.0.0-2555.gf72fecff.el9.x86_64 141/164 2024-05-12T21:55:20.467 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : rbd-fuse-2:19.0.0-2555.gf72fecff.el9.x86_64 142/164 2024-05-12T21:55:20.508 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : rbd-nbd-2:19.0.0-2555.gf72fecff.el9.x86_64 143/164 2024-05-12T21:55:20.549 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 144/164 2024-05-12T21:55:20.576 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 145/164 2024-05-12T21:55:20.763 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : cephfs-shell-2:19.0.0-2555.gf72fecff.el9.x86_64 146/164 2024-05-12T21:55:20.805 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 147/164 2024-05-12T21:55:20.836 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libcephfs-devel-2:19.0.0-2555.gf72fecff.el9.x86_ 148/164 2024-05-12T21:55:20.865 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : NetworkManager-team-1:1.47.5-1.el9.x86_64 149/164 2024-05-12T21:55:20.911 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : NetworkManager-tui-1:1.47.5-1.el9.x86_64 150/164 2024-05-12T21:55:21.521 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 151/164 2024-05-12T21:55:21.559 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 151/164 2024-05-12T21:55:21.579 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 151/164 2024-05-12T21:55:22.417 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : cephfs-top-2:19.0.0-2555.gf72fecff.el9.noarch 152/164 2024-05-12T21:55:22.506 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 153/164 2024-05-12T21:55:22.547 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 154/164 2024-05-12T21:55:22.630 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : flex-2.6.4-9.el9.x86_64 155/164 2024-05-12T21:55:22.659 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : bison-3.7.4-5.el9.x86_64 156/164 2024-05-12T21:55:22.685 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : NetworkManager-tui-1:1.47.4-1.el9.x86_64 157/164 2024-05-12T21:55:22.685 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 158/164 2024-05-12T21:55:22.769 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 158/164 2024-05-12T21:55:22.802 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 159/164 2024-05-12T21:55:22.802 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 160/164 2024-05-12T21:55:22.904 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 160/164 2024-05-12T21:55:22.928 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : NetworkManager-team-1:1.47.4-1.el9.x86_64 161/164 2024-05-12T21:55:22.933 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 162/164 2024-05-12T21:55:22.933 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : NetworkManager-1:1.47.4-1.el9.x86_64 162/164 2024-05-12T21:55:23.122 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 162/164 2024-05-12T21:55:23.122 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 163/164 2024-05-12T21:55:23.214 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 163/164 2024-05-12T21:55:23.277 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 164/164 2024-05-12T21:55:23.677 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-test-2:19.0.0-2555.gf72fecff.el9.x86_64 141/164 2024-05-12T21:55:23.718 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : rbd-fuse-2:19.0.0-2555.gf72fecff.el9.x86_64 142/164 2024-05-12T21:55:23.771 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : rbd-nbd-2:19.0.0-2555.gf72fecff.el9.x86_64 143/164 2024-05-12T21:55:23.800 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 144/164 2024-05-12T21:55:23.827 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 145/164 2024-05-12T21:55:24.006 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : cephfs-shell-2:19.0.0-2555.gf72fecff.el9.x86_64 146/164 2024-05-12T21:55:24.048 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 147/164 2024-05-12T21:55:24.103 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libcephfs-devel-2:19.0.0-2555.gf72fecff.el9.x86_ 148/164 2024-05-12T21:55:24.133 INFO:teuthology.orchestra.run.smithi086.stdout: Upgrading : NetworkManager-team-1:1.47.5-1.el9.x86_64 149/164 2024-05-12T21:55:24.171 INFO:teuthology.orchestra.run.smithi086.stdout: Upgrading : NetworkManager-tui-1:1.47.5-1.el9.x86_64 150/164 2024-05-12T21:55:24.777 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 151/164 2024-05-12T21:55:24.827 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 151/164 2024-05-12T21:55:24.846 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 151/164 2024-05-12T21:55:25.678 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : cephfs-top-2:19.0.0-2555.gf72fecff.el9.noarch 152/164 2024-05-12T21:55:25.756 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 153/164 2024-05-12T21:55:25.797 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 154/164 2024-05-12T21:55:25.895 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : flex-2.6.4-9.el9.x86_64 155/164 2024-05-12T21:55:25.935 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : bison-3.7.4-5.el9.x86_64 156/164 2024-05-12T21:55:25.960 INFO:teuthology.orchestra.run.smithi086.stdout: Cleanup : NetworkManager-tui-1:1.47.4-1.el9.x86_64 157/164 2024-05-12T21:55:25.961 INFO:teuthology.orchestra.run.smithi086.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 158/164 2024-05-12T21:55:26.045 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 158/164 2024-05-12T21:55:26.086 INFO:teuthology.orchestra.run.smithi086.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 159/164 2024-05-12T21:55:26.086 INFO:teuthology.orchestra.run.smithi086.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 160/164 2024-05-12T21:55:26.154 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 160/164 2024-05-12T21:55:26.179 INFO:teuthology.orchestra.run.smithi086.stdout: Cleanup : NetworkManager-team-1:1.47.4-1.el9.x86_64 161/164 2024-05-12T21:55:26.184 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 162/164 2024-05-12T21:55:26.184 INFO:teuthology.orchestra.run.smithi086.stdout: Cleanup : NetworkManager-1:1.47.4-1.el9.x86_64 162/164 2024-05-12T21:55:26.364 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 162/164 2024-05-12T21:55:26.364 INFO:teuthology.orchestra.run.smithi086.stdout: Cleanup : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 163/164 2024-05-12T21:55:26.448 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 163/164 2024-05-12T21:55:26.511 INFO:teuthology.orchestra.run.smithi086.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 164/164 2024-05-12T21:55:28.782 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 164/164 2024-05-12T21:55:28.782 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-2:19.0.0-2555.gf72fecff.el9.x86_64 1/164 2024-05-12T21:55:28.782 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-base-2:19.0.0-2555.gf72fecff.el9.x86_64 2/164 2024-05-12T21:55:28.782 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-common-2:19.0.0-2555.gf72fecff.el9.x86_64 3/164 2024-05-12T21:55:28.782 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-fuse-2:19.0.0-2555.gf72fecff.el9.x86_64 4/164 2024-05-12T21:55:28.782 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2555.gf72fe 5/164 2024-05-12T21:55:28.783 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mds-2:19.0.0-2555.gf72fecff.el9.x86_64 6/164 2024-05-12T21:55:28.783 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mgr-2:19.0.0-2555.gf72fecff.el9.x86_64 7/164 2024-05-12T21:55:28.783 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mon-2:19.0.0-2555.gf72fecff.el9.x86_64 8/164 2024-05-12T21:55:28.783 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-osd-2:19.0.0-2555.gf72fecff.el9.x86_64 9/164 2024-05-12T21:55:28.783 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-radosgw-2:19.0.0-2555.gf72fecff.el9.x86_64 10/164 2024-05-12T21:55:28.783 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-selinux-2:19.0.0-2555.gf72fecff.el9.x86_64 11/164 2024-05-12T21:55:28.783 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-test-2:19.0.0-2555.gf72fecff.el9.x86_64 12/164 2024-05-12T21:55:28.783 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : cephfs-mirror-2:19.0.0-2555.gf72fecff.el9.x86_64 13/164 2024-05-12T21:55:28.783 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : cephfs-shell-2:19.0.0-2555.gf72fecff.el9.x86_64 14/164 2024-05-12T21:55:28.783 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libcephfs-devel-2:19.0.0-2555.gf72fecff.el9.x86_ 15/164 2024-05-12T21:55:28.783 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libcephfs2-2:19.0.0-2555.gf72fecff.el9.x86_64 16/164 2024-05-12T21:55:28.783 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libcephsqlite-2:19.0.0-2555.gf72fecff.el9.x86_64 17/164 2024-05-12T21:55:28.783 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librados-devel-2:19.0.0-2555.gf72fecff.el9.x86_6 18/164 2024-05-12T21:55:28.783 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libradosstriper1-2:19.0.0-2555.gf72fecff.el9.x86 19/164 2024-05-12T21:55:28.784 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librgw2-2:19.0.0-2555.gf72fecff.el9.x86_64 20/164 2024-05-12T21:55:28.784 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2555.gf72fecff.el 21/164 2024-05-12T21:55:28.784 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-ceph-common-2:19.0.0-2555.gf72fecff.el9. 22/164 2024-05-12T21:55:28.784 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-cephfs-2:19.0.0-2555.gf72fecff.el9.x86_6 23/164 2024-05-12T21:55:28.784 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-rados-2:19.0.0-2555.gf72fecff.el9.x86_64 24/164 2024-05-12T21:55:28.784 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-rbd-2:19.0.0-2555.gf72fecff.el9.x86_64 25/164 2024-05-12T21:55:28.784 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-rgw-2:19.0.0-2555.gf72fecff.el9.x86_64 26/164 2024-05-12T21:55:28.784 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : rbd-fuse-2:19.0.0-2555.gf72fecff.el9.x86_64 27/164 2024-05-12T21:55:28.784 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : rbd-mirror-2:19.0.0-2555.gf72fecff.el9.x86_64 28/164 2024-05-12T21:55:28.784 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : rbd-nbd-2:19.0.0-2555.gf72fecff.el9.x86_64 29/164 2024-05-12T21:55:28.784 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2555.gf72fecff. 30/164 2024-05-12T21:55:28.784 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2555.gf72fecff.el9.noa 31/164 2024-05-12T21:55:28.784 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2555.gf72fecff.el9.n 32/164 2024-05-12T21:55:28.785 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2555.gf72 33/164 2024-05-12T21:55:28.785 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2555.gf72fecff.el 34/164 2024-05-12T21:55:28.785 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2555.gf72fecff.el9.noarch 35/164 2024-05-12T21:55:28.785 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2555.gf72fecff.e 36/164 2024-05-12T21:55:28.785 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-volume-2:19.0.0-2555.gf72fecff.el9.noarch 37/164 2024-05-12T21:55:28.785 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : cephadm-2:19.0.0-2555.gf72fecff.el9.noarch 38/164 2024-05-12T21:55:28.785 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : cephfs-top-2:19.0.0-2555.gf72fecff.el9.noarch 39/164 2024-05-12T21:55:28.785 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/164 2024-05-12T21:55:28.785 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/164 2024-05-12T21:55:28.785 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/164 2024-05-12T21:55:28.785 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/164 2024-05-12T21:55:28.785 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/164 2024-05-12T21:55:28.785 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/164 2024-05-12T21:55:28.785 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/164 2024-05-12T21:55:28.786 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/164 2024-05-12T21:55:28.786 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/164 2024-05-12T21:55:28.786 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/164 2024-05-12T21:55:28.786 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/164 2024-05-12T21:55:28.786 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/164 2024-05-12T21:55:28.786 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/164 2024-05-12T21:55:28.786 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/164 2024-05-12T21:55:28.786 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/164 2024-05-12T21:55:28.786 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/164 2024-05-12T21:55:28.786 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/164 2024-05-12T21:55:28.786 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/164 2024-05-12T21:55:28.786 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/164 2024-05-12T21:55:28.786 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/164 2024-05-12T21:55:28.786 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/164 2024-05-12T21:55:28.786 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/164 2024-05-12T21:55:28.787 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/164 2024-05-12T21:55:28.787 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/164 2024-05-12T21:55:28.787 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/164 2024-05-12T21:55:28.787 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/164 2024-05-12T21:55:28.787 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/164 2024-05-12T21:55:28.787 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/164 2024-05-12T21:55:28.787 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/164 2024-05-12T21:55:28.787 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/164 2024-05-12T21:55:28.787 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/164 2024-05-12T21:55:28.787 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/164 2024-05-12T21:55:28.787 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/164 2024-05-12T21:55:28.787 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/164 2024-05-12T21:55:28.787 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/164 2024-05-12T21:55:28.787 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/164 2024-05-12T21:55:28.788 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/164 2024-05-12T21:55:28.788 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/164 2024-05-12T21:55:28.788 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/164 2024-05-12T21:55:28.788 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/164 2024-05-12T21:55:28.788 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/164 2024-05-12T21:55:28.788 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/164 2024-05-12T21:55:28.788 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/164 2024-05-12T21:55:28.788 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/164 2024-05-12T21:55:28.788 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/164 2024-05-12T21:55:28.788 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/164 2024-05-12T21:55:28.788 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/164 2024-05-12T21:55:28.788 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/164 2024-05-12T21:55:28.788 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/164 2024-05-12T21:55:28.788 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/164 2024-05-12T21:55:28.788 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/164 2024-05-12T21:55:28.789 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/164 2024-05-12T21:55:28.789 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/164 2024-05-12T21:55:28.789 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/164 2024-05-12T21:55:28.789 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/164 2024-05-12T21:55:28.789 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/164 2024-05-12T21:55:28.789 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/164 2024-05-12T21:55:28.789 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/164 2024-05-12T21:55:28.789 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/164 2024-05-12T21:55:28.789 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/164 2024-05-12T21:55:28.789 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/164 2024-05-12T21:55:28.789 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/164 2024-05-12T21:55:28.789 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/164 2024-05-12T21:55:28.789 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/164 2024-05-12T21:55:28.789 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/164 2024-05-12T21:55:28.789 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/164 2024-05-12T21:55:28.790 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/164 2024-05-12T21:55:28.790 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/164 2024-05-12T21:55:28.790 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 108/164 2024-05-12T21:55:28.790 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 109/164 2024-05-12T21:55:28.790 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/164 2024-05-12T21:55:28.790 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/164 2024-05-12T21:55:28.790 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/164 2024-05-12T21:55:28.790 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 113/164 2024-05-12T21:55:28.790 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/164 2024-05-12T21:55:28.790 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/164 2024-05-12T21:55:28.790 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/164 2024-05-12T21:55:28.790 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/164 2024-05-12T21:55:28.790 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/164 2024-05-12T21:55:28.790 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/164 2024-05-12T21:55:28.790 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/164 2024-05-12T21:55:28.791 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/164 2024-05-12T21:55:28.791 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/164 2024-05-12T21:55:28.791 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/164 2024-05-12T21:55:28.791 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/164 2024-05-12T21:55:28.791 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/164 2024-05-12T21:55:28.791 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/164 2024-05-12T21:55:28.791 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/164 2024-05-12T21:55:28.791 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/164 2024-05-12T21:55:28.791 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/164 2024-05-12T21:55:28.791 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/164 2024-05-12T21:55:28.791 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/164 2024-05-12T21:55:28.791 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/164 2024-05-12T21:55:28.791 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/164 2024-05-12T21:55:28.792 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/164 2024-05-12T21:55:28.792 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/164 2024-05-12T21:55:28.792 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/164 2024-05-12T21:55:28.792 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/164 2024-05-12T21:55:28.792 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/164 2024-05-12T21:55:28.792 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/164 2024-05-12T21:55:28.792 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/164 2024-05-12T21:55:28.792 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/164 2024-05-12T21:55:28.792 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/164 2024-05-12T21:55:28.792 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/164 2024-05-12T21:55:28.792 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/164 2024-05-12T21:55:28.792 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/164 2024-05-12T21:55:28.792 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/164 2024-05-12T21:55:28.792 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/164 2024-05-12T21:55:28.792 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/164 2024-05-12T21:55:28.792 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librados2-2:19.0.0-2555.gf72fecff.el9.x86_64 149/164 2024-05-12T21:55:28.793 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/164 2024-05-12T21:55:28.793 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librbd1-2:19.0.0-2555.gf72fecff.el9.x86_64 151/164 2024-05-12T21:55:28.793 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/164 2024-05-12T21:55:28.793 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : NetworkManager-1:1.47.5-1.el9.x86_64 153/164 2024-05-12T21:55:28.793 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : NetworkManager-1:1.47.4-1.el9.x86_64 154/164 2024-05-12T21:55:28.793 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : NetworkManager-libnm-1:1.47.5-1.el9.x86_64 155/164 2024-05-12T21:55:28.793 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 156/164 2024-05-12T21:55:28.793 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : NetworkManager-team-1:1.47.5-1.el9.x86_64 157/164 2024-05-12T21:55:28.793 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : NetworkManager-team-1:1.47.4-1.el9.x86_64 158/164 2024-05-12T21:55:28.793 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : NetworkManager-tui-1:1.47.5-1.el9.x86_64 159/164 2024-05-12T21:55:28.793 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : NetworkManager-tui-1:1.47.4-1.el9.x86_64 160/164 2024-05-12T21:55:28.793 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 161/164 2024-05-12T21:55:28.793 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 162/164 2024-05-12T21:55:28.793 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 163/164 2024-05-12T21:55:29.584 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2555.gf72fecff.el9.x86_64 109/164 2024-05-12T21:55:29.584 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /sys 2024-05-12T21:55:29.584 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /proc 2024-05-12T21:55:29.584 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /mnt 2024-05-12T21:55:29.584 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /var/tmp 2024-05-12T21:55:29.584 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /home 2024-05-12T21:55:29.584 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /root 2024-05-12T21:55:29.585 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /tmp 2024-05-12T21:55:29.585 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-12T21:55:29.833 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-volume-2:19.0.0-2555.gf72fecff.el9.noarch 110/164 2024-05-12T21:55:29.892 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-volume-2:19.0.0-2555.gf72fecff.el9.noarch 110/164 2024-05-12T21:55:29.892 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-12T21:55:29.892 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-05-12T21:55:29.892 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-12T21:55:31.585 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-osd-2:19.0.0-2555.gf72fecff.el9.x86_64 111/164 2024-05-12T21:55:31.589 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 164/164 2024-05-12T21:55:31.589 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-12T21:55:31.589 INFO:teuthology.orchestra.run.smithi186.stdout:Upgraded: 2024-05-12T21:55:31.590 INFO:teuthology.orchestra.run.smithi186.stdout: NetworkManager-1:1.47.5-1.el9.x86_64 2024-05-12T21:55:31.590 INFO:teuthology.orchestra.run.smithi186.stdout: NetworkManager-libnm-1:1.47.5-1.el9.x86_64 2024-05-12T21:55:31.590 INFO:teuthology.orchestra.run.smithi186.stdout: NetworkManager-team-1:1.47.5-1.el9.x86_64 2024-05-12T21:55:31.590 INFO:teuthology.orchestra.run.smithi186.stdout: NetworkManager-tui-1:1.47.5-1.el9.x86_64 2024-05-12T21:55:31.590 INFO:teuthology.orchestra.run.smithi186.stdout: librados2-2:19.0.0-2555.gf72fecff.el9.x86_64 2024-05-12T21:55:31.590 INFO:teuthology.orchestra.run.smithi186.stdout: librbd1-2:19.0.0-2555.gf72fecff.el9.x86_64 2024-05-12T21:55:31.590 INFO:teuthology.orchestra.run.smithi186.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-05-12T21:55:31.590 INFO:teuthology.orchestra.run.smithi186.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-05-12T21:55:31.590 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-05-12T21:55:31.591 INFO:teuthology.orchestra.run.smithi186.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-05-12T21:55:31.591 INFO:teuthology.orchestra.run.smithi186.stdout: automake-1.16.2-8.el9.noarch 2024-05-12T21:55:31.591 INFO:teuthology.orchestra.run.smithi186.stdout: bison-3.7.4-5.el9.x86_64 2024-05-12T21:55:31.591 INFO:teuthology.orchestra.run.smithi186.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-05-12T21:55:31.591 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-2:19.0.0-2555.gf72fecff.el9.x86_64 2024-05-12T21:55:31.591 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-base-2:19.0.0-2555.gf72fecff.el9.x86_64 2024-05-12T21:55:31.591 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-common-2:19.0.0-2555.gf72fecff.el9.x86_64 2024-05-12T21:55:31.591 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-fuse-2:19.0.0-2555.gf72fecff.el9.x86_64 2024-05-12T21:55:31.591 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-grafana-dashboards-2:19.0.0-2555.gf72fecff.el9.noarch 2024-05-12T21:55:31.591 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-immutable-object-cache-2:19.0.0-2555.gf72fecff.el9.x86_64 2024-05-12T21:55:31.591 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mds-2:19.0.0-2555.gf72fecff.el9.x86_64 2024-05-12T21:55:31.592 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-2:19.0.0-2555.gf72fecff.el9.x86_64 2024-05-12T21:55:31.592 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-cephadm-2:19.0.0-2555.gf72fecff.el9.noarch 2024-05-12T21:55:31.592 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-dashboard-2:19.0.0-2555.gf72fecff.el9.noarch 2024-05-12T21:55:31.592 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2555.gf72fecff.el9.noarch 2024-05-12T21:55:31.592 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-modules-core-2:19.0.0-2555.gf72fecff.el9.noarch 2024-05-12T21:55:31.592 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-rook-2:19.0.0-2555.gf72fecff.el9.noarch 2024-05-12T21:55:31.592 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mon-2:19.0.0-2555.gf72fecff.el9.x86_64 2024-05-12T21:55:31.592 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-osd-2:19.0.0-2555.gf72fecff.el9.x86_64 2024-05-12T21:55:31.592 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-prometheus-alerts-2:19.0.0-2555.gf72fecff.el9.noarch 2024-05-12T21:55:31.592 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-radosgw-2:19.0.0-2555.gf72fecff.el9.x86_64 2024-05-12T21:55:31.592 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-selinux-2:19.0.0-2555.gf72fecff.el9.x86_64 2024-05-12T21:55:31.592 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-test-2:19.0.0-2555.gf72fecff.el9.x86_64 2024-05-12T21:55:31.592 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-volume-2:19.0.0-2555.gf72fecff.el9.noarch 2024-05-12T21:55:31.592 INFO:teuthology.orchestra.run.smithi186.stdout: cephadm-2:19.0.0-2555.gf72fecff.el9.noarch 2024-05-12T21:55:31.593 INFO:teuthology.orchestra.run.smithi186.stdout: cephfs-mirror-2:19.0.0-2555.gf72fecff.el9.x86_64 2024-05-12T21:55:31.593 INFO:teuthology.orchestra.run.smithi186.stdout: cephfs-shell-2:19.0.0-2555.gf72fecff.el9.x86_64 2024-05-12T21:55:31.593 INFO:teuthology.orchestra.run.smithi186.stdout: cephfs-top-2:19.0.0-2555.gf72fecff.el9.noarch 2024-05-12T21:55:31.593 INFO:teuthology.orchestra.run.smithi186.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-05-12T21:55:31.593 INFO:teuthology.orchestra.run.smithi186.stdout: flex-2.6.4-9.el9.x86_64 2024-05-12T21:55:31.593 INFO:teuthology.orchestra.run.smithi186.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-05-12T21:55:31.593 INFO:teuthology.orchestra.run.smithi186.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-05-12T21:55:31.593 INFO:teuthology.orchestra.run.smithi186.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-05-12T21:55:31.593 INFO:teuthology.orchestra.run.smithi186.stdout: fuse-2.9.9-15.el9.x86_64 2024-05-12T21:55:31.593 INFO:teuthology.orchestra.run.smithi186.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-05-12T21:55:31.593 INFO:teuthology.orchestra.run.smithi186.stdout: grpc-data-1.46.7-10.el9.noarch 2024-05-12T21:55:31.593 INFO:teuthology.orchestra.run.smithi186.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-05-12T21:55:31.593 INFO:teuthology.orchestra.run.smithi186.stdout: libarrow-9.0.0-11.el9.x86_64 2024-05-12T21:55:31.593 INFO:teuthology.orchestra.run.smithi186.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-05-12T21:55:31.594 INFO:teuthology.orchestra.run.smithi186.stdout: libcephfs-devel-2:19.0.0-2555.gf72fecff.el9.x86_64 2024-05-12T21:55:31.594 INFO:teuthology.orchestra.run.smithi186.stdout: libcephfs2-2:19.0.0-2555.gf72fecff.el9.x86_64 2024-05-12T21:55:31.594 INFO:teuthology.orchestra.run.smithi186.stdout: libcephsqlite-2:19.0.0-2555.gf72fecff.el9.x86_64 2024-05-12T21:55:31.594 INFO:teuthology.orchestra.run.smithi186.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-05-12T21:55:31.594 INFO:teuthology.orchestra.run.smithi186.stdout: liboath-2.6.7-2.el9.x86_64 2024-05-12T21:55:31.594 INFO:teuthology.orchestra.run.smithi186.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-05-12T21:55:31.594 INFO:teuthology.orchestra.run.smithi186.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-05-12T21:55:31.594 INFO:teuthology.orchestra.run.smithi186.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-05-12T21:55:31.594 INFO:teuthology.orchestra.run.smithi186.stdout: librados-devel-2:19.0.0-2555.gf72fecff.el9.x86_64 2024-05-12T21:55:31.595 INFO:teuthology.orchestra.run.smithi186.stdout: libradosstriper1-2:19.0.0-2555.gf72fecff.el9.x86_64 2024-05-12T21:55:31.595 INFO:teuthology.orchestra.run.smithi186.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-05-12T21:55:31.595 INFO:teuthology.orchestra.run.smithi186.stdout: librgw2-2:19.0.0-2555.gf72fecff.el9.x86_64 2024-05-12T21:55:31.595 INFO:teuthology.orchestra.run.smithi186.stdout: libtool-2.4.6-45.el9.x86_64 2024-05-12T21:55:31.595 INFO:teuthology.orchestra.run.smithi186.stdout: libunwind-1.6.2-1.el9.x86_64 2024-05-12T21:55:31.595 INFO:teuthology.orchestra.run.smithi186.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-05-12T21:55:31.595 INFO:teuthology.orchestra.run.smithi186.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-05-12T21:55:31.595 INFO:teuthology.orchestra.run.smithi186.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-05-12T21:55:31.595 INFO:teuthology.orchestra.run.smithi186.stdout: lua-5.4.4-4.el9.x86_64 2024-05-12T21:55:31.595 INFO:teuthology.orchestra.run.smithi186.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-05-12T21:55:31.595 INFO:teuthology.orchestra.run.smithi186.stdout: luarocks-3.9.2-1.el9.noarch 2024-05-12T21:55:31.595 INFO:teuthology.orchestra.run.smithi186.stdout: openblas-0.3.21-2.el9.x86_64 2024-05-12T21:55:31.595 INFO:teuthology.orchestra.run.smithi186.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-05-12T21:55:31.595 INFO:teuthology.orchestra.run.smithi186.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-05-12T21:55:31.595 INFO:teuthology.orchestra.run.smithi186.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-05-12T21:55:31.596 INFO:teuthology.orchestra.run.smithi186.stdout: postgresql-13.14-1.el9.x86_64 2024-05-12T21:55:31.596 INFO:teuthology.orchestra.run.smithi186.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-05-12T21:55:31.596 INFO:teuthology.orchestra.run.smithi186.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-05-12T21:55:31.596 INFO:teuthology.orchestra.run.smithi186.stdout: postgresql-server-13.14-1.el9.x86_64 2024-05-12T21:55:31.596 INFO:teuthology.orchestra.run.smithi186.stdout: protobuf-3.14.0-13.el9.x86_64 2024-05-12T21:55:31.596 INFO:teuthology.orchestra.run.smithi186.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-05-12T21:55:31.596 INFO:teuthology.orchestra.run.smithi186.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-05-12T21:55:31.596 INFO:teuthology.orchestra.run.smithi186.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-05-12T21:55:31.596 INFO:teuthology.orchestra.run.smithi186.stdout: python3-babel-2.9.1-2.el9.noarch 2024-05-12T21:55:31.597 INFO:teuthology.orchestra.run.smithi186.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-05-12T21:55:31.597 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-05-12T21:55:31.597 INFO:teuthology.orchestra.run.smithi186.stdout: python3-ceph-argparse-2:19.0.0-2555.gf72fecff.el9.x86_64 2024-05-12T21:55:31.597 INFO:teuthology.orchestra.run.smithi186.stdout: python3-ceph-common-2:19.0.0-2555.gf72fecff.el9.x86_64 2024-05-12T21:55:31.597 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cephfs-2:19.0.0-2555.gf72fecff.el9.x86_64 2024-05-12T21:55:31.597 INFO:teuthology.orchestra.run.smithi186.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-05-12T21:55:31.597 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-05-12T21:55:31.597 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-05-12T21:55:31.597 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-05-12T21:55:31.597 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-05-12T21:55:31.597 INFO:teuthology.orchestra.run.smithi186.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-05-12T21:55:31.597 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-05-12T21:55:31.597 INFO:teuthology.orchestra.run.smithi186.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-05-12T21:55:31.598 INFO:teuthology.orchestra.run.smithi186.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-05-12T21:55:31.598 INFO:teuthology.orchestra.run.smithi186.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-05-12T21:55:31.598 INFO:teuthology.orchestra.run.smithi186.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-05-12T21:55:31.598 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-05-12T21:55:31.598 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-05-12T21:55:31.598 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-05-12T21:55:31.598 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-05-12T21:55:31.598 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-05-12T21:55:31.598 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-05-12T21:55:31.598 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-05-12T21:55:31.598 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-05-12T21:55:31.598 INFO:teuthology.orchestra.run.smithi186.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-05-12T21:55:31.598 INFO:teuthology.orchestra.run.smithi186.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-05-12T21:55:31.598 INFO:teuthology.orchestra.run.smithi186.stdout: python3-mako-1.1.4-6.el9.noarch 2024-05-12T21:55:31.599 INFO:teuthology.orchestra.run.smithi186.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-05-12T21:55:31.599 INFO:teuthology.orchestra.run.smithi186.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-05-12T21:55:31.599 INFO:teuthology.orchestra.run.smithi186.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-05-12T21:55:31.599 INFO:teuthology.orchestra.run.smithi186.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-05-12T21:55:31.599 INFO:teuthology.orchestra.run.smithi186.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-05-12T21:55:31.599 INFO:teuthology.orchestra.run.smithi186.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-05-12T21:55:31.599 INFO:teuthology.orchestra.run.smithi186.stdout: python3-packaging-20.9-5.el9.noarch 2024-05-12T21:55:31.599 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-05-12T21:55:31.599 INFO:teuthology.orchestra.run.smithi186.stdout: python3-ply-3.11-14.el9.noarch 2024-05-12T21:55:31.599 INFO:teuthology.orchestra.run.smithi186.stdout: python3-portend-3.1.0-2.el9.noarch 2024-05-12T21:55:31.599 INFO:teuthology.orchestra.run.smithi186.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-05-12T21:55:31.600 INFO:teuthology.orchestra.run.smithi186.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-05-12T21:55:31.600 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-05-12T21:55:31.600 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-05-12T21:55:31.600 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-05-12T21:55:31.600 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pycparser-2.20-6.el9.noarch 2024-05-12T21:55:31.600 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-05-12T21:55:31.600 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pytz-2021.1-5.el9.noarch 2024-05-12T21:55:31.600 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rados-2:19.0.0-2555.gf72fecff.el9.x86_64 2024-05-12T21:55:31.600 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rbd-2:19.0.0-2555.gf72fecff.el9.x86_64 2024-05-12T21:55:31.600 INFO:teuthology.orchestra.run.smithi186.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-05-12T21:55:31.600 INFO:teuthology.orchestra.run.smithi186.stdout: python3-requests-2.25.1-8.el9.noarch 2024-05-12T21:55:31.600 INFO:teuthology.orchestra.run.smithi186.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-05-12T21:55:31.600 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rgw-2:19.0.0-2555.gf72fecff.el9.x86_64 2024-05-12T21:55:31.600 INFO:teuthology.orchestra.run.smithi186.stdout: python3-routes-2.5.1-5.el9.noarch 2024-05-12T21:55:31.600 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rsa-4.9-2.el9.noarch 2024-05-12T21:55:31.601 INFO:teuthology.orchestra.run.smithi186.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-05-12T21:55:31.601 INFO:teuthology.orchestra.run.smithi186.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-05-12T21:55:31.601 INFO:teuthology.orchestra.run.smithi186.stdout: python3-toml-0.10.2-6.el9.noarch 2024-05-12T21:55:31.601 INFO:teuthology.orchestra.run.smithi186.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-05-12T21:55:31.601 INFO:teuthology.orchestra.run.smithi186.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-05-12T21:55:31.601 INFO:teuthology.orchestra.run.smithi186.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-05-12T21:55:31.601 INFO:teuthology.orchestra.run.smithi186.stdout: python3-webob-1.8.7-6.el9.noarch 2024-05-12T21:55:31.601 INFO:teuthology.orchestra.run.smithi186.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-05-12T21:55:31.601 INFO:teuthology.orchestra.run.smithi186.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-05-12T21:55:31.601 INFO:teuthology.orchestra.run.smithi186.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-05-12T21:55:31.601 INFO:teuthology.orchestra.run.smithi186.stdout: qatlib-24.02.0-1.el9.x86_64 2024-05-12T21:55:31.601 INFO:teuthology.orchestra.run.smithi186.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-05-12T21:55:31.601 INFO:teuthology.orchestra.run.smithi186.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-05-12T21:55:31.601 INFO:teuthology.orchestra.run.smithi186.stdout: rbd-fuse-2:19.0.0-2555.gf72fecff.el9.x86_64 2024-05-12T21:55:31.602 INFO:teuthology.orchestra.run.smithi186.stdout: rbd-mirror-2:19.0.0-2555.gf72fecff.el9.x86_64 2024-05-12T21:55:31.602 INFO:teuthology.orchestra.run.smithi186.stdout: rbd-nbd-2:19.0.0-2555.gf72fecff.el9.x86_64 2024-05-12T21:55:31.602 INFO:teuthology.orchestra.run.smithi186.stdout: re2-1:20211101-3.el9.x86_64 2024-05-12T21:55:31.602 INFO:teuthology.orchestra.run.smithi186.stdout: socat-1.7.4.1-5.el9.x86_64 2024-05-12T21:55:31.602 INFO:teuthology.orchestra.run.smithi186.stdout: thrift-0.15.0-2.el9.x86_64 2024-05-12T21:55:31.602 INFO:teuthology.orchestra.run.smithi186.stdout: uuid-1.6.2-55.el9.x86_64 2024-05-12T21:55:31.602 INFO:teuthology.orchestra.run.smithi186.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-05-12T21:55:31.602 INFO:teuthology.orchestra.run.smithi186.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-05-12T21:55:31.603 INFO:teuthology.orchestra.run.smithi186.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-05-12T21:55:31.603 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-12T21:55:31.603 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-05-12T21:55:31.692 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-osd-2:19.0.0-2555.gf72fecff.el9.x86_64 111/164 2024-05-12T21:55:31.692 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-12T21:55:31.692 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-05-12T21:55:31.692 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-12T21:55:31.692 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-12T21:55:31.693 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-12T21:55:31.949 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 164/164 2024-05-12T21:55:31.950 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-2:19.0.0-2555.gf72fecff.el9.x86_64 1/164 2024-05-12T21:55:31.950 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-base-2:19.0.0-2555.gf72fecff.el9.x86_64 2/164 2024-05-12T21:55:31.950 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-common-2:19.0.0-2555.gf72fecff.el9.x86_64 3/164 2024-05-12T21:55:31.950 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-fuse-2:19.0.0-2555.gf72fecff.el9.x86_64 4/164 2024-05-12T21:55:31.950 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2555.gf72fe 5/164 2024-05-12T21:55:31.950 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mds-2:19.0.0-2555.gf72fecff.el9.x86_64 6/164 2024-05-12T21:55:31.950 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mgr-2:19.0.0-2555.gf72fecff.el9.x86_64 7/164 2024-05-12T21:55:31.950 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mon-2:19.0.0-2555.gf72fecff.el9.x86_64 8/164 2024-05-12T21:55:31.950 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-osd-2:19.0.0-2555.gf72fecff.el9.x86_64 9/164 2024-05-12T21:55:31.950 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-radosgw-2:19.0.0-2555.gf72fecff.el9.x86_64 10/164 2024-05-12T21:55:31.950 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-selinux-2:19.0.0-2555.gf72fecff.el9.x86_64 11/164 2024-05-12T21:55:31.950 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-test-2:19.0.0-2555.gf72fecff.el9.x86_64 12/164 2024-05-12T21:55:31.951 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : cephfs-mirror-2:19.0.0-2555.gf72fecff.el9.x86_64 13/164 2024-05-12T21:55:31.951 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : cephfs-shell-2:19.0.0-2555.gf72fecff.el9.x86_64 14/164 2024-05-12T21:55:31.951 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libcephfs-devel-2:19.0.0-2555.gf72fecff.el9.x86_ 15/164 2024-05-12T21:55:31.951 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libcephfs2-2:19.0.0-2555.gf72fecff.el9.x86_64 16/164 2024-05-12T21:55:31.951 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libcephsqlite-2:19.0.0-2555.gf72fecff.el9.x86_64 17/164 2024-05-12T21:55:31.951 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librados-devel-2:19.0.0-2555.gf72fecff.el9.x86_6 18/164 2024-05-12T21:55:31.951 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libradosstriper1-2:19.0.0-2555.gf72fecff.el9.x86 19/164 2024-05-12T21:55:31.951 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librgw2-2:19.0.0-2555.gf72fecff.el9.x86_64 20/164 2024-05-12T21:55:31.951 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2555.gf72fecff.el 21/164 2024-05-12T21:55:31.951 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-ceph-common-2:19.0.0-2555.gf72fecff.el9. 22/164 2024-05-12T21:55:31.951 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-cephfs-2:19.0.0-2555.gf72fecff.el9.x86_6 23/164 2024-05-12T21:55:31.951 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-rados-2:19.0.0-2555.gf72fecff.el9.x86_64 24/164 2024-05-12T21:55:31.951 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-rbd-2:19.0.0-2555.gf72fecff.el9.x86_64 25/164 2024-05-12T21:55:31.951 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-rgw-2:19.0.0-2555.gf72fecff.el9.x86_64 26/164 2024-05-12T21:55:31.951 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : rbd-fuse-2:19.0.0-2555.gf72fecff.el9.x86_64 27/164 2024-05-12T21:55:31.964 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : rbd-mirror-2:19.0.0-2555.gf72fecff.el9.x86_64 28/164 2024-05-12T21:55:31.964 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : rbd-nbd-2:19.0.0-2555.gf72fecff.el9.x86_64 29/164 2024-05-12T21:55:31.964 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2555.gf72fecff. 30/164 2024-05-12T21:55:31.964 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2555.gf72fecff.el9.noa 31/164 2024-05-12T21:55:31.964 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2555.gf72fecff.el9.n 32/164 2024-05-12T21:55:31.965 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2555.gf72 33/164 2024-05-12T21:55:31.965 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2555.gf72fecff.el 34/164 2024-05-12T21:55:31.965 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2555.gf72fecff.el9.noarch 35/164 2024-05-12T21:55:31.965 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2555.gf72fecff.e 36/164 2024-05-12T21:55:31.965 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-volume-2:19.0.0-2555.gf72fecff.el9.noarch 37/164 2024-05-12T21:55:31.965 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : cephadm-2:19.0.0-2555.gf72fecff.el9.noarch 38/164 2024-05-12T21:55:31.965 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : cephfs-top-2:19.0.0-2555.gf72fecff.el9.noarch 39/164 2024-05-12T21:55:31.965 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/164 2024-05-12T21:55:31.965 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/164 2024-05-12T21:55:31.965 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/164 2024-05-12T21:55:31.965 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/164 2024-05-12T21:55:31.965 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/164 2024-05-12T21:55:31.965 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/164 2024-05-12T21:55:31.965 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/164 2024-05-12T21:55:31.966 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/164 2024-05-12T21:55:31.966 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/164 2024-05-12T21:55:31.966 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/164 2024-05-12T21:55:31.966 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/164 2024-05-12T21:55:31.966 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/164 2024-05-12T21:55:31.966 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/164 2024-05-12T21:55:31.966 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/164 2024-05-12T21:55:31.966 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/164 2024-05-12T21:55:31.966 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/164 2024-05-12T21:55:31.966 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/164 2024-05-12T21:55:31.966 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/164 2024-05-12T21:55:31.966 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/164 2024-05-12T21:55:31.966 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/164 2024-05-12T21:55:31.967 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/164 2024-05-12T21:55:31.967 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/164 2024-05-12T21:55:31.967 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/164 2024-05-12T21:55:31.967 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/164 2024-05-12T21:55:31.967 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/164 2024-05-12T21:55:31.967 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/164 2024-05-12T21:55:31.967 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/164 2024-05-12T21:55:31.967 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/164 2024-05-12T21:55:31.967 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/164 2024-05-12T21:55:31.967 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/164 2024-05-12T21:55:31.967 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/164 2024-05-12T21:55:31.967 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/164 2024-05-12T21:55:31.967 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/164 2024-05-12T21:55:31.967 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/164 2024-05-12T21:55:31.967 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/164 2024-05-12T21:55:31.968 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/164 2024-05-12T21:55:31.968 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/164 2024-05-12T21:55:31.968 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/164 2024-05-12T21:55:31.968 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/164 2024-05-12T21:55:31.968 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/164 2024-05-12T21:55:31.968 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/164 2024-05-12T21:55:31.968 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/164 2024-05-12T21:55:31.968 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/164 2024-05-12T21:55:31.968 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/164 2024-05-12T21:55:31.968 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/164 2024-05-12T21:55:31.968 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/164 2024-05-12T21:55:31.968 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/164 2024-05-12T21:55:31.968 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/164 2024-05-12T21:55:31.968 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/164 2024-05-12T21:55:31.968 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/164 2024-05-12T21:55:31.969 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/164 2024-05-12T21:55:31.969 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/164 2024-05-12T21:55:31.969 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/164 2024-05-12T21:55:31.969 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/164 2024-05-12T21:55:31.969 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/164 2024-05-12T21:55:31.969 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/164 2024-05-12T21:55:31.969 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/164 2024-05-12T21:55:31.969 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/164 2024-05-12T21:55:31.969 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/164 2024-05-12T21:55:31.969 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/164 2024-05-12T21:55:31.969 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/164 2024-05-12T21:55:31.969 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/164 2024-05-12T21:55:31.969 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/164 2024-05-12T21:55:31.969 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/164 2024-05-12T21:55:31.969 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/164 2024-05-12T21:55:31.970 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/164 2024-05-12T21:55:31.970 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/164 2024-05-12T21:55:31.970 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/164 2024-05-12T21:55:31.970 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 108/164 2024-05-12T21:55:31.970 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 109/164 2024-05-12T21:55:31.970 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/164 2024-05-12T21:55:31.970 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/164 2024-05-12T21:55:31.970 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/164 2024-05-12T21:55:31.970 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 113/164 2024-05-12T21:55:31.970 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/164 2024-05-12T21:55:31.970 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/164 2024-05-12T21:55:31.970 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/164 2024-05-12T21:55:31.970 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/164 2024-05-12T21:55:31.970 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/164 2024-05-12T21:55:31.971 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/164 2024-05-12T21:55:31.971 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/164 2024-05-12T21:55:31.971 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/164 2024-05-12T21:55:31.971 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/164 2024-05-12T21:55:31.971 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/164 2024-05-12T21:55:31.971 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/164 2024-05-12T21:55:31.971 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/164 2024-05-12T21:55:31.971 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/164 2024-05-12T21:55:31.971 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/164 2024-05-12T21:55:31.971 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/164 2024-05-12T21:55:31.971 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/164 2024-05-12T21:55:31.971 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/164 2024-05-12T21:55:31.971 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/164 2024-05-12T21:55:31.971 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/164 2024-05-12T21:55:31.971 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/164 2024-05-12T21:55:31.972 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/164 2024-05-12T21:55:31.972 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/164 2024-05-12T21:55:31.972 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/164 2024-05-12T21:55:31.972 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/164 2024-05-12T21:55:31.972 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/164 2024-05-12T21:55:31.972 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/164 2024-05-12T21:55:31.972 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/164 2024-05-12T21:55:31.972 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/164 2024-05-12T21:55:31.972 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/164 2024-05-12T21:55:31.972 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/164 2024-05-12T21:55:31.972 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/164 2024-05-12T21:55:31.972 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/164 2024-05-12T21:55:31.972 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/164 2024-05-12T21:55:31.972 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/164 2024-05-12T21:55:31.973 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/164 2024-05-12T21:55:31.973 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librados2-2:19.0.0-2555.gf72fecff.el9.x86_64 149/164 2024-05-12T21:55:31.973 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/164 2024-05-12T21:55:31.973 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librbd1-2:19.0.0-2555.gf72fecff.el9.x86_64 151/164 2024-05-12T21:55:31.973 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/164 2024-05-12T21:55:31.973 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : NetworkManager-1:1.47.5-1.el9.x86_64 153/164 2024-05-12T21:55:31.973 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : NetworkManager-1:1.47.4-1.el9.x86_64 154/164 2024-05-12T21:55:31.973 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : NetworkManager-libnm-1:1.47.5-1.el9.x86_64 155/164 2024-05-12T21:55:31.973 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 156/164 2024-05-12T21:55:31.973 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : NetworkManager-team-1:1.47.5-1.el9.x86_64 157/164 2024-05-12T21:55:31.973 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : NetworkManager-team-1:1.47.4-1.el9.x86_64 158/164 2024-05-12T21:55:31.973 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : NetworkManager-tui-1:1.47.5-1.el9.x86_64 159/164 2024-05-12T21:55:31.973 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : NetworkManager-tui-1:1.47.4-1.el9.x86_64 160/164 2024-05-12T21:55:31.973 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 161/164 2024-05-12T21:55:31.973 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 162/164 2024-05-12T21:55:31.974 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 163/164 2024-05-12T21:55:31.974 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mds-2:19.0.0-2555.gf72fecff.el9.x86_64 112/164 2024-05-12T21:55:32.006 DEBUG:teuthology.parallel:result is None 2024-05-12T21:55:32.056 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mds-2:19.0.0-2555.gf72fecff.el9.x86_64 112/164 2024-05-12T21:55:32.056 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-12T21:55:32.056 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-05-12T21:55:32.057 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-12T21:55:32.057 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-12T21:55:32.057 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-12T21:55:32.569 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mon-2:19.0.0-2555.gf72fecff.el9.x86_64 113/164 2024-05-12T21:55:32.666 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mon-2:19.0.0-2555.gf72fecff.el9.x86_64 113/164 2024-05-12T21:55:32.666 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-12T21:55:32.666 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-05-12T21:55:32.666 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-12T21:55:32.666 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-12T21:55:32.666 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-12T21:55:32.812 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : automake-1.16.2-8.el9.noarch 114/164 2024-05-12T21:55:32.875 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-ply-3.11-14.el9.noarch 115/164 2024-05-12T21:55:32.952 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 116/164 2024-05-12T21:55:33.199 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 117/164 2024-05-12T21:55:33.258 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 118/164 2024-05-12T21:55:33.348 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 119/164 2024-05-12T21:55:33.459 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 120/164 2024-05-12T21:55:33.639 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 121/164 2024-05-12T21:55:33.694 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 122/164 2024-05-12T21:55:33.742 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 123/164 2024-05-12T21:55:33.771 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : fuse-2.9.9-15.el9.x86_64 124/164 2024-05-12T21:55:34.434 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: cephadm-2:19.0.0-2555.gf72fecff.el9.noarch 125/164 2024-05-12T21:55:34.468 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : cephadm-2:19.0.0-2555.gf72fecff.el9.noarch 125/164 2024-05-12T21:55:34.523 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2555.gf72fecff.e 126/164 2024-05-12T21:55:34.638 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2555.gf72fecff. 127/164 2024-05-12T21:55:34.669 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2555.gf72fecff.el9.noa 128/164 2024-05-12T21:55:34.762 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 164/164 2024-05-12T21:55:34.762 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-05-12T21:55:34.762 INFO:teuthology.orchestra.run.smithi086.stdout:Upgraded: 2024-05-12T21:55:34.762 INFO:teuthology.orchestra.run.smithi086.stdout: NetworkManager-1:1.47.5-1.el9.x86_64 2024-05-12T21:55:34.762 INFO:teuthology.orchestra.run.smithi086.stdout: NetworkManager-libnm-1:1.47.5-1.el9.x86_64 2024-05-12T21:55:34.763 INFO:teuthology.orchestra.run.smithi086.stdout: NetworkManager-team-1:1.47.5-1.el9.x86_64 2024-05-12T21:55:34.763 INFO:teuthology.orchestra.run.smithi086.stdout: NetworkManager-tui-1:1.47.5-1.el9.x86_64 2024-05-12T21:55:34.763 INFO:teuthology.orchestra.run.smithi086.stdout: librados2-2:19.0.0-2555.gf72fecff.el9.x86_64 2024-05-12T21:55:34.763 INFO:teuthology.orchestra.run.smithi086.stdout: librbd1-2:19.0.0-2555.gf72fecff.el9.x86_64 2024-05-12T21:55:34.763 INFO:teuthology.orchestra.run.smithi086.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-05-12T21:55:34.763 INFO:teuthology.orchestra.run.smithi086.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-05-12T21:55:34.763 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-05-12T21:55:34.763 INFO:teuthology.orchestra.run.smithi086.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-05-12T21:55:34.763 INFO:teuthology.orchestra.run.smithi086.stdout: automake-1.16.2-8.el9.noarch 2024-05-12T21:55:34.763 INFO:teuthology.orchestra.run.smithi086.stdout: bison-3.7.4-5.el9.x86_64 2024-05-12T21:55:34.763 INFO:teuthology.orchestra.run.smithi086.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-05-12T21:55:34.763 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-2:19.0.0-2555.gf72fecff.el9.x86_64 2024-05-12T21:55:34.763 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-base-2:19.0.0-2555.gf72fecff.el9.x86_64 2024-05-12T21:55:34.763 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-common-2:19.0.0-2555.gf72fecff.el9.x86_64 2024-05-12T21:55:34.763 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-fuse-2:19.0.0-2555.gf72fecff.el9.x86_64 2024-05-12T21:55:34.764 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-grafana-dashboards-2:19.0.0-2555.gf72fecff.el9.noarch 2024-05-12T21:55:34.764 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-immutable-object-cache-2:19.0.0-2555.gf72fecff.el9.x86_64 2024-05-12T21:55:34.764 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mds-2:19.0.0-2555.gf72fecff.el9.x86_64 2024-05-12T21:55:34.764 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-2:19.0.0-2555.gf72fecff.el9.x86_64 2024-05-12T21:55:34.764 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-cephadm-2:19.0.0-2555.gf72fecff.el9.noarch 2024-05-12T21:55:34.764 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-dashboard-2:19.0.0-2555.gf72fecff.el9.noarch 2024-05-12T21:55:34.764 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2555.gf72fecff.el9.noarch 2024-05-12T21:55:34.764 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-modules-core-2:19.0.0-2555.gf72fecff.el9.noarch 2024-05-12T21:55:34.764 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-rook-2:19.0.0-2555.gf72fecff.el9.noarch 2024-05-12T21:55:34.764 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mon-2:19.0.0-2555.gf72fecff.el9.x86_64 2024-05-12T21:55:34.764 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-osd-2:19.0.0-2555.gf72fecff.el9.x86_64 2024-05-12T21:55:34.764 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-prometheus-alerts-2:19.0.0-2555.gf72fecff.el9.noarch 2024-05-12T21:55:34.764 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-radosgw-2:19.0.0-2555.gf72fecff.el9.x86_64 2024-05-12T21:55:34.764 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-selinux-2:19.0.0-2555.gf72fecff.el9.x86_64 2024-05-12T21:55:34.765 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-test-2:19.0.0-2555.gf72fecff.el9.x86_64 2024-05-12T21:55:34.765 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-volume-2:19.0.0-2555.gf72fecff.el9.noarch 2024-05-12T21:55:34.765 INFO:teuthology.orchestra.run.smithi086.stdout: cephadm-2:19.0.0-2555.gf72fecff.el9.noarch 2024-05-12T21:55:34.765 INFO:teuthology.orchestra.run.smithi086.stdout: cephfs-mirror-2:19.0.0-2555.gf72fecff.el9.x86_64 2024-05-12T21:55:34.765 INFO:teuthology.orchestra.run.smithi086.stdout: cephfs-shell-2:19.0.0-2555.gf72fecff.el9.x86_64 2024-05-12T21:55:34.765 INFO:teuthology.orchestra.run.smithi086.stdout: cephfs-top-2:19.0.0-2555.gf72fecff.el9.noarch 2024-05-12T21:55:34.765 INFO:teuthology.orchestra.run.smithi086.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-05-12T21:55:34.765 INFO:teuthology.orchestra.run.smithi086.stdout: flex-2.6.4-9.el9.x86_64 2024-05-12T21:55:34.765 INFO:teuthology.orchestra.run.smithi086.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-05-12T21:55:34.765 INFO:teuthology.orchestra.run.smithi086.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-05-12T21:55:34.765 INFO:teuthology.orchestra.run.smithi086.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-05-12T21:55:34.765 INFO:teuthology.orchestra.run.smithi086.stdout: fuse-2.9.9-15.el9.x86_64 2024-05-12T21:55:34.765 INFO:teuthology.orchestra.run.smithi086.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-05-12T21:55:34.765 INFO:teuthology.orchestra.run.smithi086.stdout: grpc-data-1.46.7-10.el9.noarch 2024-05-12T21:55:34.766 INFO:teuthology.orchestra.run.smithi086.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-05-12T21:55:34.766 INFO:teuthology.orchestra.run.smithi086.stdout: libarrow-9.0.0-11.el9.x86_64 2024-05-12T21:55:34.766 INFO:teuthology.orchestra.run.smithi086.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-05-12T21:55:34.766 INFO:teuthology.orchestra.run.smithi086.stdout: libcephfs-devel-2:19.0.0-2555.gf72fecff.el9.x86_64 2024-05-12T21:55:34.766 INFO:teuthology.orchestra.run.smithi086.stdout: libcephfs2-2:19.0.0-2555.gf72fecff.el9.x86_64 2024-05-12T21:55:34.766 INFO:teuthology.orchestra.run.smithi086.stdout: libcephsqlite-2:19.0.0-2555.gf72fecff.el9.x86_64 2024-05-12T21:55:34.766 INFO:teuthology.orchestra.run.smithi086.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-05-12T21:55:34.766 INFO:teuthology.orchestra.run.smithi086.stdout: liboath-2.6.7-2.el9.x86_64 2024-05-12T21:55:34.766 INFO:teuthology.orchestra.run.smithi086.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-05-12T21:55:34.766 INFO:teuthology.orchestra.run.smithi086.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-05-12T21:55:34.766 INFO:teuthology.orchestra.run.smithi086.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-05-12T21:55:34.766 INFO:teuthology.orchestra.run.smithi086.stdout: librados-devel-2:19.0.0-2555.gf72fecff.el9.x86_64 2024-05-12T21:55:34.767 INFO:teuthology.orchestra.run.smithi086.stdout: libradosstriper1-2:19.0.0-2555.gf72fecff.el9.x86_64 2024-05-12T21:55:34.767 INFO:teuthology.orchestra.run.smithi086.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-05-12T21:55:34.767 INFO:teuthology.orchestra.run.smithi086.stdout: librgw2-2:19.0.0-2555.gf72fecff.el9.x86_64 2024-05-12T21:55:34.767 INFO:teuthology.orchestra.run.smithi086.stdout: libtool-2.4.6-45.el9.x86_64 2024-05-12T21:55:34.767 INFO:teuthology.orchestra.run.smithi086.stdout: libunwind-1.6.2-1.el9.x86_64 2024-05-12T21:55:34.767 INFO:teuthology.orchestra.run.smithi086.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-05-12T21:55:34.767 INFO:teuthology.orchestra.run.smithi086.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-05-12T21:55:34.767 INFO:teuthology.orchestra.run.smithi086.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-05-12T21:55:34.767 INFO:teuthology.orchestra.run.smithi086.stdout: lua-5.4.4-4.el9.x86_64 2024-05-12T21:55:34.767 INFO:teuthology.orchestra.run.smithi086.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-05-12T21:55:34.767 INFO:teuthology.orchestra.run.smithi086.stdout: luarocks-3.9.2-1.el9.noarch 2024-05-12T21:55:34.767 INFO:teuthology.orchestra.run.smithi086.stdout: openblas-0.3.21-2.el9.x86_64 2024-05-12T21:55:34.768 INFO:teuthology.orchestra.run.smithi086.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-05-12T21:55:34.768 INFO:teuthology.orchestra.run.smithi086.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-05-12T21:55:34.768 INFO:teuthology.orchestra.run.smithi086.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-05-12T21:55:34.768 INFO:teuthology.orchestra.run.smithi086.stdout: postgresql-13.14-1.el9.x86_64 2024-05-12T21:55:34.768 INFO:teuthology.orchestra.run.smithi086.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-05-12T21:55:34.768 INFO:teuthology.orchestra.run.smithi086.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-05-12T21:55:34.768 INFO:teuthology.orchestra.run.smithi086.stdout: postgresql-server-13.14-1.el9.x86_64 2024-05-12T21:55:34.768 INFO:teuthology.orchestra.run.smithi086.stdout: protobuf-3.14.0-13.el9.x86_64 2024-05-12T21:55:34.768 INFO:teuthology.orchestra.run.smithi086.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-05-12T21:55:34.768 INFO:teuthology.orchestra.run.smithi086.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-05-12T21:55:34.768 INFO:teuthology.orchestra.run.smithi086.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-05-12T21:55:34.769 INFO:teuthology.orchestra.run.smithi086.stdout: python3-babel-2.9.1-2.el9.noarch 2024-05-12T21:55:34.769 INFO:teuthology.orchestra.run.smithi086.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-05-12T21:55:34.769 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-05-12T21:55:34.769 INFO:teuthology.orchestra.run.smithi086.stdout: python3-ceph-argparse-2:19.0.0-2555.gf72fecff.el9.x86_64 2024-05-12T21:55:34.769 INFO:teuthology.orchestra.run.smithi086.stdout: python3-ceph-common-2:19.0.0-2555.gf72fecff.el9.x86_64 2024-05-12T21:55:34.769 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cephfs-2:19.0.0-2555.gf72fecff.el9.x86_64 2024-05-12T21:55:34.769 INFO:teuthology.orchestra.run.smithi086.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-05-12T21:55:34.769 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-05-12T21:55:34.769 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-05-12T21:55:34.769 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-05-12T21:55:34.769 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-05-12T21:55:34.770 INFO:teuthology.orchestra.run.smithi086.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-05-12T21:55:34.770 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-05-12T21:55:34.770 INFO:teuthology.orchestra.run.smithi086.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-05-12T21:55:34.770 INFO:teuthology.orchestra.run.smithi086.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-05-12T21:55:34.770 INFO:teuthology.orchestra.run.smithi086.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-05-12T21:55:34.770 INFO:teuthology.orchestra.run.smithi086.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-05-12T21:55:34.770 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-05-12T21:55:34.770 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-05-12T21:55:34.770 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-05-12T21:55:34.770 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-05-12T21:55:34.770 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-05-12T21:55:34.770 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-05-12T21:55:34.771 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-05-12T21:55:34.771 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-05-12T21:55:34.771 INFO:teuthology.orchestra.run.smithi086.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-05-12T21:55:34.771 INFO:teuthology.orchestra.run.smithi086.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-05-12T21:55:34.771 INFO:teuthology.orchestra.run.smithi086.stdout: python3-mako-1.1.4-6.el9.noarch 2024-05-12T21:55:34.771 INFO:teuthology.orchestra.run.smithi086.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-05-12T21:55:34.771 INFO:teuthology.orchestra.run.smithi086.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-05-12T21:55:34.771 INFO:teuthology.orchestra.run.smithi086.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-05-12T21:55:34.771 INFO:teuthology.orchestra.run.smithi086.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-05-12T21:55:34.771 INFO:teuthology.orchestra.run.smithi086.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-05-12T21:55:34.771 INFO:teuthology.orchestra.run.smithi086.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-05-12T21:55:34.771 INFO:teuthology.orchestra.run.smithi086.stdout: python3-packaging-20.9-5.el9.noarch 2024-05-12T21:55:34.771 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-05-12T21:55:34.771 INFO:teuthology.orchestra.run.smithi086.stdout: python3-ply-3.11-14.el9.noarch 2024-05-12T21:55:34.772 INFO:teuthology.orchestra.run.smithi086.stdout: python3-portend-3.1.0-2.el9.noarch 2024-05-12T21:55:34.772 INFO:teuthology.orchestra.run.smithi086.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-05-12T21:55:34.772 INFO:teuthology.orchestra.run.smithi086.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-05-12T21:55:34.772 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-05-12T21:55:34.772 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-05-12T21:55:34.772 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-05-12T21:55:34.772 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pycparser-2.20-6.el9.noarch 2024-05-12T21:55:34.772 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-05-12T21:55:34.772 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pytz-2021.1-5.el9.noarch 2024-05-12T21:55:34.772 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rados-2:19.0.0-2555.gf72fecff.el9.x86_64 2024-05-12T21:55:34.772 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rbd-2:19.0.0-2555.gf72fecff.el9.x86_64 2024-05-12T21:55:34.772 INFO:teuthology.orchestra.run.smithi086.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-05-12T21:55:34.772 INFO:teuthology.orchestra.run.smithi086.stdout: python3-requests-2.25.1-8.el9.noarch 2024-05-12T21:55:34.772 INFO:teuthology.orchestra.run.smithi086.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-05-12T21:55:34.773 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rgw-2:19.0.0-2555.gf72fecff.el9.x86_64 2024-05-12T21:55:34.773 INFO:teuthology.orchestra.run.smithi086.stdout: python3-routes-2.5.1-5.el9.noarch 2024-05-12T21:55:34.773 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rsa-4.9-2.el9.noarch 2024-05-12T21:55:34.773 INFO:teuthology.orchestra.run.smithi086.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-05-12T21:55:34.773 INFO:teuthology.orchestra.run.smithi086.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-05-12T21:55:34.773 INFO:teuthology.orchestra.run.smithi086.stdout: python3-toml-0.10.2-6.el9.noarch 2024-05-12T21:55:34.773 INFO:teuthology.orchestra.run.smithi086.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-05-12T21:55:34.773 INFO:teuthology.orchestra.run.smithi086.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-05-12T21:55:34.773 INFO:teuthology.orchestra.run.smithi086.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-05-12T21:55:34.773 INFO:teuthology.orchestra.run.smithi086.stdout: python3-webob-1.8.7-6.el9.noarch 2024-05-12T21:55:34.773 INFO:teuthology.orchestra.run.smithi086.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-05-12T21:55:34.773 INFO:teuthology.orchestra.run.smithi086.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-05-12T21:55:34.773 INFO:teuthology.orchestra.run.smithi086.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-05-12T21:55:34.774 INFO:teuthology.orchestra.run.smithi086.stdout: qatlib-24.02.0-1.el9.x86_64 2024-05-12T21:55:34.774 INFO:teuthology.orchestra.run.smithi086.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-05-12T21:55:34.774 INFO:teuthology.orchestra.run.smithi086.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-05-12T21:55:34.774 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-fuse-2:19.0.0-2555.gf72fecff.el9.x86_64 2024-05-12T21:55:34.774 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-mirror-2:19.0.0-2555.gf72fecff.el9.x86_64 2024-05-12T21:55:34.774 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-nbd-2:19.0.0-2555.gf72fecff.el9.x86_64 2024-05-12T21:55:34.774 INFO:teuthology.orchestra.run.smithi086.stdout: re2-1:20211101-3.el9.x86_64 2024-05-12T21:55:34.774 INFO:teuthology.orchestra.run.smithi086.stdout: socat-1.7.4.1-5.el9.x86_64 2024-05-12T21:55:34.774 INFO:teuthology.orchestra.run.smithi086.stdout: thrift-0.15.0-2.el9.x86_64 2024-05-12T21:55:34.774 INFO:teuthology.orchestra.run.smithi086.stdout: uuid-1.6.2-55.el9.x86_64 2024-05-12T21:55:34.774 INFO:teuthology.orchestra.run.smithi086.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-05-12T21:55:34.774 INFO:teuthology.orchestra.run.smithi086.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-05-12T21:55:34.775 INFO:teuthology.orchestra.run.smithi086.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-05-12T21:55:34.775 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-05-12T21:55:34.775 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-05-12T21:55:35.176 DEBUG:teuthology.parallel:result is None 2024-05-12T21:55:36.004 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2555.gf72fecff.el9.noa 128/164 2024-05-12T21:55:36.046 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2555.gf72fecff.el9.n 129/164 2024-05-12T21:55:37.217 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2555.gf72fecff.el9.n 129/164 2024-05-12T21:55:37.246 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2555.gf72 130/164 2024-05-12T21:55:37.402 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2555.gf72 130/164 2024-05-12T21:55:37.586 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2555.gf72fecff.el 131/164 2024-05-12T21:55:37.615 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-2:19.0.0-2555.gf72fecff.el9.x86_64 132/164 2024-05-12T21:55:37.711 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2555.gf72fecff.el9.x86_64 132/164 2024-05-12T21:55:37.711 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-12T21:55:37.711 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-05-12T21:55:37.711 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-12T21:55:37.711 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-12T21:55:37.712 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-12T21:55:37.774 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-rook-2:19.0.0-2555.gf72fecff.el9.noarch 133/164 2024-05-12T21:55:37.812 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2555.gf72fecff.el9.noarch 133/164 2024-05-12T21:55:37.938 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-2:19.0.0-2555.gf72fecff.el9.x86_64 134/164 2024-05-12T21:55:38.004 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-fuse-2:19.0.0-2555.gf72fecff.el9.x86_64 135/164 2024-05-12T21:55:39.055 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libtool-2.4.6-45.el9.x86_64 136/164 2024-05-12T21:55:39.085 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-radosgw-2:19.0.0-2555.gf72fecff.el9.x86_64 137/164 2024-05-12T21:55:39.156 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2555.gf72fecff.el9.x86_64 137/164 2024-05-12T21:55:39.156 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-12T21:55:39.156 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-05-12T21:55:39.156 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-12T21:55:39.156 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-12T21:55:39.156 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-12T21:55:39.202 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2555.gf72fe 138/164 2024-05-12T21:55:39.276 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2555.gf72fe 138/164 2024-05-12T21:55:39.276 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-12T21:55:39.276 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-05-12T21:55:39.276 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-12T21:55:39.353 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : cephfs-mirror-2:19.0.0-2555.gf72fecff.el9.x86_64 139/164 2024-05-12T21:55:39.434 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2555.gf72fecff.el9.x86_64 139/164 2024-05-12T21:55:39.434 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-12T21:55:39.434 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-05-12T21:55:39.434 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-05-12T21:55:39.435 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-05-12T21:55:39.435 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-12T21:55:39.828 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : rbd-mirror-2:19.0.0-2555.gf72fecff.el9.x86_64 140/164 2024-05-12T21:55:39.902 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2555.gf72fecff.el9.x86_64 140/164 2024-05-12T21:55:39.903 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-12T21:55:39.903 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-05-12T21:55:39.903 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-12T21:55:39.903 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-12T21:55:39.903 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-12T21:55:45.153 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-test-2:19.0.0-2555.gf72fecff.el9.x86_64 141/164 2024-05-12T21:55:45.210 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : rbd-fuse-2:19.0.0-2555.gf72fecff.el9.x86_64 142/164 2024-05-12T21:55:45.276 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : rbd-nbd-2:19.0.0-2555.gf72fecff.el9.x86_64 143/164 2024-05-12T21:55:45.319 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 144/164 2024-05-12T21:55:45.359 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 145/164 2024-05-12T21:55:45.678 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : cephfs-shell-2:19.0.0-2555.gf72fecff.el9.x86_64 146/164 2024-05-12T21:55:45.738 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 147/164 2024-05-12T21:55:45.767 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libcephfs-devel-2:19.0.0-2555.gf72fecff.el9.x86_ 148/164 2024-05-12T21:55:45.802 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : NetworkManager-team-1:1.47.5-1.el9.x86_64 149/164 2024-05-12T21:55:45.852 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : NetworkManager-tui-1:1.47.5-1.el9.x86_64 150/164 2024-05-12T21:55:46.788 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 151/164 2024-05-12T21:55:46.831 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 151/164 2024-05-12T21:55:46.902 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 151/164 2024-05-12T21:55:48.352 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : cephfs-top-2:19.0.0-2555.gf72fecff.el9.noarch 152/164 2024-05-12T21:55:48.471 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 153/164 2024-05-12T21:55:48.542 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 154/164 2024-05-12T21:55:48.693 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : flex-2.6.4-9.el9.x86_64 155/164 2024-05-12T21:55:48.730 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : bison-3.7.4-5.el9.x86_64 156/164 2024-05-12T21:55:48.768 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : NetworkManager-tui-1:1.47.4-1.el9.x86_64 157/164 2024-05-12T21:55:48.768 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 158/164 2024-05-12T21:55:48.898 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 158/164 2024-05-12T21:55:48.943 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 159/164 2024-05-12T21:55:48.943 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 160/164 2024-05-12T21:55:49.096 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 160/164 2024-05-12T21:55:49.136 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : NetworkManager-team-1:1.47.4-1.el9.x86_64 161/164 2024-05-12T21:55:49.171 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 162/164 2024-05-12T21:55:49.172 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : NetworkManager-1:1.47.4-1.el9.x86_64 162/164 2024-05-12T21:55:49.532 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 162/164 2024-05-12T21:55:49.532 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 163/164 2024-05-12T21:55:49.676 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 163/164 2024-05-12T21:55:49.813 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 164/164 2024-05-12T21:55:50.092 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 164/164 2024-05-12T21:55:50.092 INFO:teuthology.orchestra.run.smithi028.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-05-12T21:55:50.092 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-12T21:55:55.595 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-2:19.0.0-2555.gf72fecff.el9.x86_64 1/164 2024-05-12T21:55:55.595 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-base-2:19.0.0-2555.gf72fecff.el9.x86_64 2/164 2024-05-12T21:55:55.595 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-common-2:19.0.0-2555.gf72fecff.el9.x86_64 3/164 2024-05-12T21:55:55.595 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-fuse-2:19.0.0-2555.gf72fecff.el9.x86_64 4/164 2024-05-12T21:55:55.595 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2555.gf72fe 5/164 2024-05-12T21:55:55.595 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mds-2:19.0.0-2555.gf72fecff.el9.x86_64 6/164 2024-05-12T21:55:55.595 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-2:19.0.0-2555.gf72fecff.el9.x86_64 7/164 2024-05-12T21:55:55.596 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mon-2:19.0.0-2555.gf72fecff.el9.x86_64 8/164 2024-05-12T21:55:55.596 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-osd-2:19.0.0-2555.gf72fecff.el9.x86_64 9/164 2024-05-12T21:55:55.596 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-radosgw-2:19.0.0-2555.gf72fecff.el9.x86_64 10/164 2024-05-12T21:55:55.596 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-selinux-2:19.0.0-2555.gf72fecff.el9.x86_64 11/164 2024-05-12T21:55:55.596 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-test-2:19.0.0-2555.gf72fecff.el9.x86_64 12/164 2024-05-12T21:55:55.596 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : cephfs-mirror-2:19.0.0-2555.gf72fecff.el9.x86_64 13/164 2024-05-12T21:55:55.596 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : cephfs-shell-2:19.0.0-2555.gf72fecff.el9.x86_64 14/164 2024-05-12T21:55:55.596 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libcephfs-devel-2:19.0.0-2555.gf72fecff.el9.x86_ 15/164 2024-05-12T21:55:55.596 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libcephfs2-2:19.0.0-2555.gf72fecff.el9.x86_64 16/164 2024-05-12T21:55:55.596 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libcephsqlite-2:19.0.0-2555.gf72fecff.el9.x86_64 17/164 2024-05-12T21:55:55.596 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librados-devel-2:19.0.0-2555.gf72fecff.el9.x86_6 18/164 2024-05-12T21:55:55.596 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libradosstriper1-2:19.0.0-2555.gf72fecff.el9.x86 19/164 2024-05-12T21:55:55.596 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librgw2-2:19.0.0-2555.gf72fecff.el9.x86_64 20/164 2024-05-12T21:55:55.596 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2555.gf72fecff.el 21/164 2024-05-12T21:55:55.597 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-ceph-common-2:19.0.0-2555.gf72fecff.el9. 22/164 2024-05-12T21:55:55.597 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cephfs-2:19.0.0-2555.gf72fecff.el9.x86_6 23/164 2024-05-12T21:55:55.597 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-rados-2:19.0.0-2555.gf72fecff.el9.x86_64 24/164 2024-05-12T21:55:55.597 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-rbd-2:19.0.0-2555.gf72fecff.el9.x86_64 25/164 2024-05-12T21:55:55.597 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-rgw-2:19.0.0-2555.gf72fecff.el9.x86_64 26/164 2024-05-12T21:55:55.597 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : rbd-fuse-2:19.0.0-2555.gf72fecff.el9.x86_64 27/164 2024-05-12T21:55:55.597 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : rbd-mirror-2:19.0.0-2555.gf72fecff.el9.x86_64 28/164 2024-05-12T21:55:55.610 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : rbd-nbd-2:19.0.0-2555.gf72fecff.el9.x86_64 29/164 2024-05-12T21:55:55.610 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2555.gf72fecff. 30/164 2024-05-12T21:55:55.610 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2555.gf72fecff.el9.noa 31/164 2024-05-12T21:55:55.610 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2555.gf72fecff.el9.n 32/164 2024-05-12T21:55:55.610 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2555.gf72 33/164 2024-05-12T21:55:55.610 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2555.gf72fecff.el 34/164 2024-05-12T21:55:55.610 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2555.gf72fecff.el9.noarch 35/164 2024-05-12T21:55:55.611 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2555.gf72fecff.e 36/164 2024-05-12T21:55:55.611 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-volume-2:19.0.0-2555.gf72fecff.el9.noarch 37/164 2024-05-12T21:55:55.611 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : cephadm-2:19.0.0-2555.gf72fecff.el9.noarch 38/164 2024-05-12T21:55:55.611 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : cephfs-top-2:19.0.0-2555.gf72fecff.el9.noarch 39/164 2024-05-12T21:55:55.611 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/164 2024-05-12T21:55:55.611 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/164 2024-05-12T21:55:55.611 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/164 2024-05-12T21:55:55.611 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/164 2024-05-12T21:55:55.611 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/164 2024-05-12T21:55:55.611 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/164 2024-05-12T21:55:55.611 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/164 2024-05-12T21:55:55.611 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/164 2024-05-12T21:55:55.611 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/164 2024-05-12T21:55:55.612 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/164 2024-05-12T21:55:55.612 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/164 2024-05-12T21:55:55.612 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/164 2024-05-12T21:55:55.612 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/164 2024-05-12T21:55:55.612 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/164 2024-05-12T21:55:55.612 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/164 2024-05-12T21:55:55.612 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/164 2024-05-12T21:55:55.612 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/164 2024-05-12T21:55:55.612 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/164 2024-05-12T21:55:55.612 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/164 2024-05-12T21:55:55.612 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/164 2024-05-12T21:55:55.612 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/164 2024-05-12T21:55:55.612 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/164 2024-05-12T21:55:55.612 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/164 2024-05-12T21:55:55.612 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/164 2024-05-12T21:55:55.613 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/164 2024-05-12T21:55:55.613 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/164 2024-05-12T21:55:55.613 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/164 2024-05-12T21:55:55.613 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/164 2024-05-12T21:55:55.613 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/164 2024-05-12T21:55:55.613 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/164 2024-05-12T21:55:55.613 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/164 2024-05-12T21:55:55.613 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/164 2024-05-12T21:55:55.613 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/164 2024-05-12T21:55:55.613 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/164 2024-05-12T21:55:55.614 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/164 2024-05-12T21:55:55.614 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/164 2024-05-12T21:55:55.614 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/164 2024-05-12T21:55:55.614 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/164 2024-05-12T21:55:55.614 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/164 2024-05-12T21:55:55.614 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/164 2024-05-12T21:55:55.614 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/164 2024-05-12T21:55:55.614 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/164 2024-05-12T21:55:55.614 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/164 2024-05-12T21:55:55.614 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/164 2024-05-12T21:55:55.614 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/164 2024-05-12T21:55:55.614 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/164 2024-05-12T21:55:55.615 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/164 2024-05-12T21:55:55.615 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/164 2024-05-12T21:55:55.615 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/164 2024-05-12T21:55:55.615 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/164 2024-05-12T21:55:55.615 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/164 2024-05-12T21:55:55.615 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/164 2024-05-12T21:55:55.615 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/164 2024-05-12T21:55:55.615 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/164 2024-05-12T21:55:55.615 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/164 2024-05-12T21:55:55.615 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/164 2024-05-12T21:55:55.615 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/164 2024-05-12T21:55:55.615 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/164 2024-05-12T21:55:55.615 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/164 2024-05-12T21:55:55.615 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/164 2024-05-12T21:55:55.615 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/164 2024-05-12T21:55:55.615 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/164 2024-05-12T21:55:55.616 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/164 2024-05-12T21:55:55.616 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/164 2024-05-12T21:55:55.616 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/164 2024-05-12T21:55:55.616 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/164 2024-05-12T21:55:55.616 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/164 2024-05-12T21:55:55.616 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/164 2024-05-12T21:55:55.616 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 108/164 2024-05-12T21:55:55.616 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 109/164 2024-05-12T21:55:55.616 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/164 2024-05-12T21:55:55.616 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/164 2024-05-12T21:55:55.616 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/164 2024-05-12T21:55:55.616 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 113/164 2024-05-12T21:55:55.616 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/164 2024-05-12T21:55:55.616 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/164 2024-05-12T21:55:55.616 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/164 2024-05-12T21:55:55.616 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/164 2024-05-12T21:55:55.617 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/164 2024-05-12T21:55:55.617 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/164 2024-05-12T21:55:55.617 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/164 2024-05-12T21:55:55.617 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/164 2024-05-12T21:55:55.617 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/164 2024-05-12T21:55:55.617 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/164 2024-05-12T21:55:55.617 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/164 2024-05-12T21:55:55.617 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/164 2024-05-12T21:55:55.617 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/164 2024-05-12T21:55:55.617 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/164 2024-05-12T21:55:55.617 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/164 2024-05-12T21:55:55.618 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/164 2024-05-12T21:55:55.618 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/164 2024-05-12T21:55:55.618 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/164 2024-05-12T21:55:55.618 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/164 2024-05-12T21:55:55.618 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/164 2024-05-12T21:55:55.618 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/164 2024-05-12T21:55:55.618 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/164 2024-05-12T21:55:55.618 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/164 2024-05-12T21:55:55.618 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/164 2024-05-12T21:55:55.619 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/164 2024-05-12T21:55:55.619 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/164 2024-05-12T21:55:55.619 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/164 2024-05-12T21:55:55.619 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/164 2024-05-12T21:55:55.619 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/164 2024-05-12T21:55:55.619 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/164 2024-05-12T21:55:55.619 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/164 2024-05-12T21:55:55.619 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/164 2024-05-12T21:55:55.619 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/164 2024-05-12T21:55:55.619 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/164 2024-05-12T21:55:55.619 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/164 2024-05-12T21:55:55.619 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librados2-2:19.0.0-2555.gf72fecff.el9.x86_64 149/164 2024-05-12T21:55:55.619 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/164 2024-05-12T21:55:55.619 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librbd1-2:19.0.0-2555.gf72fecff.el9.x86_64 151/164 2024-05-12T21:55:55.619 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/164 2024-05-12T21:55:55.619 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : NetworkManager-1:1.47.5-1.el9.x86_64 153/164 2024-05-12T21:55:55.619 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : NetworkManager-1:1.47.4-1.el9.x86_64 154/164 2024-05-12T21:55:55.620 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : NetworkManager-libnm-1:1.47.5-1.el9.x86_64 155/164 2024-05-12T21:55:55.620 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 156/164 2024-05-12T21:55:55.620 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : NetworkManager-team-1:1.47.5-1.el9.x86_64 157/164 2024-05-12T21:55:55.620 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : NetworkManager-team-1:1.47.4-1.el9.x86_64 158/164 2024-05-12T21:55:55.620 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : NetworkManager-tui-1:1.47.5-1.el9.x86_64 159/164 2024-05-12T21:55:55.620 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : NetworkManager-tui-1:1.47.4-1.el9.x86_64 160/164 2024-05-12T21:55:55.620 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 161/164 2024-05-12T21:55:55.620 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 162/164 2024-05-12T21:55:55.620 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 163/164 2024-05-12T21:55:57.942 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 164/164 2024-05-12T21:55:57.942 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-12T21:55:57.943 INFO:teuthology.orchestra.run.smithi028.stdout:Upgraded: 2024-05-12T21:55:57.943 INFO:teuthology.orchestra.run.smithi028.stdout: NetworkManager-1:1.47.5-1.el9.x86_64 2024-05-12T21:55:57.943 INFO:teuthology.orchestra.run.smithi028.stdout: NetworkManager-libnm-1:1.47.5-1.el9.x86_64 2024-05-12T21:55:57.943 INFO:teuthology.orchestra.run.smithi028.stdout: NetworkManager-team-1:1.47.5-1.el9.x86_64 2024-05-12T21:55:57.943 INFO:teuthology.orchestra.run.smithi028.stdout: NetworkManager-tui-1:1.47.5-1.el9.x86_64 2024-05-12T21:55:57.943 INFO:teuthology.orchestra.run.smithi028.stdout: librados2-2:19.0.0-2555.gf72fecff.el9.x86_64 2024-05-12T21:55:57.943 INFO:teuthology.orchestra.run.smithi028.stdout: librbd1-2:19.0.0-2555.gf72fecff.el9.x86_64 2024-05-12T21:55:57.943 INFO:teuthology.orchestra.run.smithi028.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-05-12T21:55:57.943 INFO:teuthology.orchestra.run.smithi028.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-05-12T21:55:57.943 INFO:teuthology.orchestra.run.smithi028.stdout:Installed: 2024-05-12T21:55:57.943 INFO:teuthology.orchestra.run.smithi028.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-05-12T21:55:57.943 INFO:teuthology.orchestra.run.smithi028.stdout: automake-1.16.2-8.el9.noarch 2024-05-12T21:55:57.943 INFO:teuthology.orchestra.run.smithi028.stdout: bison-3.7.4-5.el9.x86_64 2024-05-12T21:55:57.943 INFO:teuthology.orchestra.run.smithi028.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-05-12T21:55:57.944 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-2:19.0.0-2555.gf72fecff.el9.x86_64 2024-05-12T21:55:57.944 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-base-2:19.0.0-2555.gf72fecff.el9.x86_64 2024-05-12T21:55:57.944 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-common-2:19.0.0-2555.gf72fecff.el9.x86_64 2024-05-12T21:55:57.944 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-fuse-2:19.0.0-2555.gf72fecff.el9.x86_64 2024-05-12T21:55:57.944 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-grafana-dashboards-2:19.0.0-2555.gf72fecff.el9.noarch 2024-05-12T21:55:57.944 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-immutable-object-cache-2:19.0.0-2555.gf72fecff.el9.x86_64 2024-05-12T21:55:57.944 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mds-2:19.0.0-2555.gf72fecff.el9.x86_64 2024-05-12T21:55:57.944 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-2:19.0.0-2555.gf72fecff.el9.x86_64 2024-05-12T21:55:57.944 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-cephadm-2:19.0.0-2555.gf72fecff.el9.noarch 2024-05-12T21:55:57.944 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-dashboard-2:19.0.0-2555.gf72fecff.el9.noarch 2024-05-12T21:55:57.944 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2555.gf72fecff.el9.noarch 2024-05-12T21:55:57.944 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-modules-core-2:19.0.0-2555.gf72fecff.el9.noarch 2024-05-12T21:55:57.944 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-rook-2:19.0.0-2555.gf72fecff.el9.noarch 2024-05-12T21:55:57.945 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mon-2:19.0.0-2555.gf72fecff.el9.x86_64 2024-05-12T21:55:57.945 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-osd-2:19.0.0-2555.gf72fecff.el9.x86_64 2024-05-12T21:55:57.945 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-prometheus-alerts-2:19.0.0-2555.gf72fecff.el9.noarch 2024-05-12T21:55:57.945 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-radosgw-2:19.0.0-2555.gf72fecff.el9.x86_64 2024-05-12T21:55:57.945 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-selinux-2:19.0.0-2555.gf72fecff.el9.x86_64 2024-05-12T21:55:57.945 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-test-2:19.0.0-2555.gf72fecff.el9.x86_64 2024-05-12T21:55:57.945 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-volume-2:19.0.0-2555.gf72fecff.el9.noarch 2024-05-12T21:55:57.945 INFO:teuthology.orchestra.run.smithi028.stdout: cephadm-2:19.0.0-2555.gf72fecff.el9.noarch 2024-05-12T21:55:57.945 INFO:teuthology.orchestra.run.smithi028.stdout: cephfs-mirror-2:19.0.0-2555.gf72fecff.el9.x86_64 2024-05-12T21:55:57.945 INFO:teuthology.orchestra.run.smithi028.stdout: cephfs-shell-2:19.0.0-2555.gf72fecff.el9.x86_64 2024-05-12T21:55:57.945 INFO:teuthology.orchestra.run.smithi028.stdout: cephfs-top-2:19.0.0-2555.gf72fecff.el9.noarch 2024-05-12T21:55:57.945 INFO:teuthology.orchestra.run.smithi028.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-05-12T21:55:57.945 INFO:teuthology.orchestra.run.smithi028.stdout: flex-2.6.4-9.el9.x86_64 2024-05-12T21:55:57.946 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-05-12T21:55:57.946 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-05-12T21:55:57.946 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-05-12T21:55:57.946 INFO:teuthology.orchestra.run.smithi028.stdout: fuse-2.9.9-15.el9.x86_64 2024-05-12T21:55:57.946 INFO:teuthology.orchestra.run.smithi028.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-05-12T21:55:57.946 INFO:teuthology.orchestra.run.smithi028.stdout: grpc-data-1.46.7-10.el9.noarch 2024-05-12T21:55:57.946 INFO:teuthology.orchestra.run.smithi028.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-05-12T21:55:57.946 INFO:teuthology.orchestra.run.smithi028.stdout: libarrow-9.0.0-11.el9.x86_64 2024-05-12T21:55:57.946 INFO:teuthology.orchestra.run.smithi028.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-05-12T21:55:57.947 INFO:teuthology.orchestra.run.smithi028.stdout: libcephfs-devel-2:19.0.0-2555.gf72fecff.el9.x86_64 2024-05-12T21:55:57.947 INFO:teuthology.orchestra.run.smithi028.stdout: libcephfs2-2:19.0.0-2555.gf72fecff.el9.x86_64 2024-05-12T21:55:57.947 INFO:teuthology.orchestra.run.smithi028.stdout: libcephsqlite-2:19.0.0-2555.gf72fecff.el9.x86_64 2024-05-12T21:55:57.947 INFO:teuthology.orchestra.run.smithi028.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-05-12T21:55:57.947 INFO:teuthology.orchestra.run.smithi028.stdout: liboath-2.6.7-2.el9.x86_64 2024-05-12T21:55:57.947 INFO:teuthology.orchestra.run.smithi028.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-05-12T21:55:57.947 INFO:teuthology.orchestra.run.smithi028.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-05-12T21:55:57.947 INFO:teuthology.orchestra.run.smithi028.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-05-12T21:55:57.947 INFO:teuthology.orchestra.run.smithi028.stdout: librados-devel-2:19.0.0-2555.gf72fecff.el9.x86_64 2024-05-12T21:55:57.947 INFO:teuthology.orchestra.run.smithi028.stdout: libradosstriper1-2:19.0.0-2555.gf72fecff.el9.x86_64 2024-05-12T21:55:57.947 INFO:teuthology.orchestra.run.smithi028.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-05-12T21:55:57.947 INFO:teuthology.orchestra.run.smithi028.stdout: librgw2-2:19.0.0-2555.gf72fecff.el9.x86_64 2024-05-12T21:55:57.947 INFO:teuthology.orchestra.run.smithi028.stdout: libtool-2.4.6-45.el9.x86_64 2024-05-12T21:55:57.947 INFO:teuthology.orchestra.run.smithi028.stdout: libunwind-1.6.2-1.el9.x86_64 2024-05-12T21:55:57.948 INFO:teuthology.orchestra.run.smithi028.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-05-12T21:55:57.948 INFO:teuthology.orchestra.run.smithi028.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-05-12T21:55:57.948 INFO:teuthology.orchestra.run.smithi028.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-05-12T21:55:57.948 INFO:teuthology.orchestra.run.smithi028.stdout: lua-5.4.4-4.el9.x86_64 2024-05-12T21:55:57.948 INFO:teuthology.orchestra.run.smithi028.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-05-12T21:55:57.948 INFO:teuthology.orchestra.run.smithi028.stdout: luarocks-3.9.2-1.el9.noarch 2024-05-12T21:55:57.948 INFO:teuthology.orchestra.run.smithi028.stdout: openblas-0.3.21-2.el9.x86_64 2024-05-12T21:55:57.948 INFO:teuthology.orchestra.run.smithi028.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-05-12T21:55:57.948 INFO:teuthology.orchestra.run.smithi028.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-05-12T21:55:57.948 INFO:teuthology.orchestra.run.smithi028.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-05-12T21:55:57.948 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql-13.14-1.el9.x86_64 2024-05-12T21:55:57.948 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-05-12T21:55:57.948 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-05-12T21:55:57.948 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql-server-13.14-1.el9.x86_64 2024-05-12T21:55:57.949 INFO:teuthology.orchestra.run.smithi028.stdout: protobuf-3.14.0-13.el9.x86_64 2024-05-12T21:55:57.949 INFO:teuthology.orchestra.run.smithi028.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-05-12T21:55:57.949 INFO:teuthology.orchestra.run.smithi028.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-05-12T21:55:57.949 INFO:teuthology.orchestra.run.smithi028.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-05-12T21:55:57.949 INFO:teuthology.orchestra.run.smithi028.stdout: python3-babel-2.9.1-2.el9.noarch 2024-05-12T21:55:57.949 INFO:teuthology.orchestra.run.smithi028.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-05-12T21:55:57.949 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-05-12T21:55:57.949 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ceph-argparse-2:19.0.0-2555.gf72fecff.el9.x86_64 2024-05-12T21:55:57.949 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ceph-common-2:19.0.0-2555.gf72fecff.el9.x86_64 2024-05-12T21:55:57.949 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cephfs-2:19.0.0-2555.gf72fecff.el9.x86_64 2024-05-12T21:55:57.949 INFO:teuthology.orchestra.run.smithi028.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-05-12T21:55:57.949 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-05-12T21:55:57.949 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-05-12T21:55:57.949 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-05-12T21:55:57.950 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-05-12T21:55:57.950 INFO:teuthology.orchestra.run.smithi028.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-05-12T21:55:57.950 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-05-12T21:55:57.950 INFO:teuthology.orchestra.run.smithi028.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-05-12T21:55:57.950 INFO:teuthology.orchestra.run.smithi028.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-05-12T21:55:57.950 INFO:teuthology.orchestra.run.smithi028.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-05-12T21:55:57.950 INFO:teuthology.orchestra.run.smithi028.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-05-12T21:55:57.950 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-05-12T21:55:57.950 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-05-12T21:55:57.950 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-05-12T21:55:57.950 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-05-12T21:55:57.951 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-05-12T21:55:57.951 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-05-12T21:55:57.951 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-05-12T21:55:57.951 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-05-12T21:55:57.951 INFO:teuthology.orchestra.run.smithi028.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-05-12T21:55:57.951 INFO:teuthology.orchestra.run.smithi028.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-05-12T21:55:57.951 INFO:teuthology.orchestra.run.smithi028.stdout: python3-mako-1.1.4-6.el9.noarch 2024-05-12T21:55:57.951 INFO:teuthology.orchestra.run.smithi028.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-05-12T21:55:57.951 INFO:teuthology.orchestra.run.smithi028.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-05-12T21:55:57.951 INFO:teuthology.orchestra.run.smithi028.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-05-12T21:55:57.951 INFO:teuthology.orchestra.run.smithi028.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-05-12T21:55:57.951 INFO:teuthology.orchestra.run.smithi028.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-05-12T21:55:57.951 INFO:teuthology.orchestra.run.smithi028.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-05-12T21:55:57.952 INFO:teuthology.orchestra.run.smithi028.stdout: python3-packaging-20.9-5.el9.noarch 2024-05-12T21:55:57.952 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-05-12T21:55:57.952 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ply-3.11-14.el9.noarch 2024-05-12T21:55:57.952 INFO:teuthology.orchestra.run.smithi028.stdout: python3-portend-3.1.0-2.el9.noarch 2024-05-12T21:55:57.952 INFO:teuthology.orchestra.run.smithi028.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-05-12T21:55:57.952 INFO:teuthology.orchestra.run.smithi028.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-05-12T21:55:57.952 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-05-12T21:55:57.952 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-05-12T21:55:57.952 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-05-12T21:55:57.952 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pycparser-2.20-6.el9.noarch 2024-05-12T21:55:57.952 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-05-12T21:55:57.953 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pytz-2021.1-5.el9.noarch 2024-05-12T21:55:57.953 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rados-2:19.0.0-2555.gf72fecff.el9.x86_64 2024-05-12T21:55:57.953 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rbd-2:19.0.0-2555.gf72fecff.el9.x86_64 2024-05-12T21:55:57.953 INFO:teuthology.orchestra.run.smithi028.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-05-12T21:55:57.953 INFO:teuthology.orchestra.run.smithi028.stdout: python3-requests-2.25.1-8.el9.noarch 2024-05-12T21:55:57.953 INFO:teuthology.orchestra.run.smithi028.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-05-12T21:55:57.953 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rgw-2:19.0.0-2555.gf72fecff.el9.x86_64 2024-05-12T21:55:57.953 INFO:teuthology.orchestra.run.smithi028.stdout: python3-routes-2.5.1-5.el9.noarch 2024-05-12T21:55:57.953 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rsa-4.9-2.el9.noarch 2024-05-12T21:55:57.953 INFO:teuthology.orchestra.run.smithi028.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-05-12T21:55:57.953 INFO:teuthology.orchestra.run.smithi028.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-05-12T21:55:57.953 INFO:teuthology.orchestra.run.smithi028.stdout: python3-toml-0.10.2-6.el9.noarch 2024-05-12T21:55:57.953 INFO:teuthology.orchestra.run.smithi028.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-05-12T21:55:57.953 INFO:teuthology.orchestra.run.smithi028.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-05-12T21:55:57.953 INFO:teuthology.orchestra.run.smithi028.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-05-12T21:55:57.954 INFO:teuthology.orchestra.run.smithi028.stdout: python3-webob-1.8.7-6.el9.noarch 2024-05-12T21:55:57.954 INFO:teuthology.orchestra.run.smithi028.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-05-12T21:55:57.954 INFO:teuthology.orchestra.run.smithi028.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-05-12T21:55:57.954 INFO:teuthology.orchestra.run.smithi028.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-05-12T21:55:57.954 INFO:teuthology.orchestra.run.smithi028.stdout: qatlib-24.02.0-1.el9.x86_64 2024-05-12T21:55:57.954 INFO:teuthology.orchestra.run.smithi028.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-05-12T21:55:57.954 INFO:teuthology.orchestra.run.smithi028.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-05-12T21:55:57.954 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-fuse-2:19.0.0-2555.gf72fecff.el9.x86_64 2024-05-12T21:55:57.954 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-mirror-2:19.0.0-2555.gf72fecff.el9.x86_64 2024-05-12T21:55:57.954 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-nbd-2:19.0.0-2555.gf72fecff.el9.x86_64 2024-05-12T21:55:57.954 INFO:teuthology.orchestra.run.smithi028.stdout: re2-1:20211101-3.el9.x86_64 2024-05-12T21:55:57.954 INFO:teuthology.orchestra.run.smithi028.stdout: socat-1.7.4.1-5.el9.x86_64 2024-05-12T21:55:57.954 INFO:teuthology.orchestra.run.smithi028.stdout: thrift-0.15.0-2.el9.x86_64 2024-05-12T21:55:57.955 INFO:teuthology.orchestra.run.smithi028.stdout: uuid-1.6.2-55.el9.x86_64 2024-05-12T21:55:57.955 INFO:teuthology.orchestra.run.smithi028.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-05-12T21:55:57.955 INFO:teuthology.orchestra.run.smithi028.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-05-12T21:55:57.955 INFO:teuthology.orchestra.run.smithi028.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-05-12T21:55:57.955 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-12T21:55:57.955 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-05-12T21:55:58.678 DEBUG:teuthology.parallel:result is None 2024-05-12T21:55:58.678 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=f72fecff68e1d400c4568684327c900485c20d6a 2024-05-12T21:55:58.819 DEBUG:teuthology.orchestra.run.smithi028:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-05-12T21:55:58.875 INFO:teuthology.orchestra.run.smithi028.stdout:19.0.0-2555.gf72fecff.el9 2024-05-12T21:55:58.876 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2555.gf72fecff.el9 2024-05-12T21:55:58.876 INFO:teuthology.task.install:The correct ceph version 19.0.0-2555.gf72fecff is installed. 2024-05-12T21:55:58.878 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=f72fecff68e1d400c4568684327c900485c20d6a 2024-05-12T21:55:59.087 DEBUG:teuthology.orchestra.run.smithi086:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-05-12T21:55:59.113 INFO:teuthology.orchestra.run.smithi086.stdout:19.0.0-2555.gf72fecff.el9 2024-05-12T21:55:59.113 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2555.gf72fecff.el9 2024-05-12T21:55:59.113 INFO:teuthology.task.install:The correct ceph version 19.0.0-2555.gf72fecff is installed. 2024-05-12T21:55:59.115 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=f72fecff68e1d400c4568684327c900485c20d6a 2024-05-12T21:55:59.331 DEBUG:teuthology.orchestra.run.smithi186:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-05-12T21:55:59.357 INFO:teuthology.orchestra.run.smithi186.stdout:19.0.0-2555.gf72fecff.el9 2024-05-12T21:55:59.357 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2555.gf72fecff.el9 2024-05-12T21:55:59.357 INFO:teuthology.task.install:The correct ceph version 19.0.0-2555.gf72fecff is installed. 2024-05-12T21:55:59.359 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-05-12T21:55:59.359 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-05-12T21:55:59.359 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-05-12T21:55:59.437 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-05-12T21:55:59.437 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-05-12T21:55:59.468 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-05-12T21:55:59.468 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-05-12T21:55:59.498 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-05-12T21:55:59.499 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-05-12T21:55:59.499 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/usr/bin/daemon-helper 2024-05-12T21:55:59.564 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-05-12T21:55:59.678 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-05-12T21:55:59.678 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/usr/bin/daemon-helper 2024-05-12T21:55:59.707 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-05-12T21:55:59.776 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-05-12T21:55:59.776 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/usr/bin/daemon-helper 2024-05-12T21:55:59.806 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-05-12T21:55:59.873 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-05-12T21:55:59.874 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-05-12T21:55:59.874 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/usr/bin/adjust-ulimits 2024-05-12T21:55:59.942 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-05-12T21:56:00.090 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-05-12T21:56:00.090 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/usr/bin/adjust-ulimits 2024-05-12T21:56:00.114 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-05-12T21:56:00.180 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-05-12T21:56:00.180 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/usr/bin/adjust-ulimits 2024-05-12T21:56:00.203 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-05-12T21:56:00.267 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-05-12T21:56:00.267 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-05-12T21:56:00.267 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/usr/bin/stdin-killer 2024-05-12T21:56:00.332 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-05-12T21:56:00.432 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-05-12T21:56:00.432 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/usr/bin/stdin-killer 2024-05-12T21:56:00.455 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-05-12T21:56:00.518 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-05-12T21:56:00.518 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/usr/bin/stdin-killer 2024-05-12T21:56:00.542 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-05-12T21:56:00.606 INFO:teuthology.run_tasks:Running task cephadm... 2024-05-12T21:56:00.704 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'max_mds': 3, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_bal_replicate_threshold': 1, 'mds_export_ephemeral_random_max': 0.1, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug cephsqlite': 20, 'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'slow metadata IO', 'SLOW_OPS', 'slow request'], 'sha1': 'f72fecff68e1d400c4568684327c900485c20d6a', 'subvols': {'create': 2, 'subvol_options': '--size 25000000000'}} 2024-05-12T21:56:00.704 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f72fecff68e1d400c4568684327c900485c20d6a 2024-05-12T21:56:00.705 INFO:tasks.cephadm:Cluster fsid is 6b92e7b4-10aa-11ef-bc98-c7b262605968 2024-05-12T21:56:00.705 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-05-12T21:56:00.705 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.28', 'mon.b': '172.21.15.86', 'mon.c': '172.21.15.186'} 2024-05-12T21:56:00.705 INFO:tasks.cephadm:First mon is mon.a on smithi028 2024-05-12T21:56:00.705 INFO:tasks.cephadm:First mgr is x 2024-05-12T21:56:00.705 INFO:tasks.cephadm:Normalizing hostnames... 2024-05-12T21:56:00.705 DEBUG:teuthology.orchestra.run.smithi028:> sudo hostname $(hostname -s) 2024-05-12T21:56:00.775 DEBUG:teuthology.orchestra.run.smithi086:> sudo hostname $(hostname -s) 2024-05-12T21:56:00.799 DEBUG:teuthology.orchestra.run.smithi186:> sudo hostname $(hostname -s) 2024-05-12T21:56:00.823 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-05-12T21:56:00.823 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=f72fecff68e1d400c4568684327c900485c20d6a 2024-05-12T21:56:00.960 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': 'f72fecff68e1d400c4568684327c900485c20d6a', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/55120/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-2555-gf72fecff', 'node_name': '172.21.5.39+adami09', 'job_name': 'ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-2555.gf72fecff'}, 'url': 'https://3.chacra.ceph.com/r/ceph/squid/f72fecff68e1d400c4568684327c900485c20d6a/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-05-10 16:26:33.694063', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'squid', 'chacra_url': 'https://3.chacra.ceph.com/repos/ceph/squid/f72fecff68e1d400c4568684327c900485c20d6a/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}] 2024-05-12T21:56:01.176 INFO:tasks.util.chacra:got chacra host 3.chacra.ceph.com, ref squid, sha1 f72fecff68e1d400c4568684327c900485c20d6a from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=f72fecff68e1d400c4568684327c900485c20d6a 2024-05-12T21:56:01.178 INFO:tasks.cephadm:Discovered cachra url: https://3.chacra.ceph.com/binaries/ceph/squid/f72fecff68e1d400c4568684327c900485c20d6a/centos/9/x86_64/flavors/default/cephadm 2024-05-12T21:56:01.179 DEBUG:teuthology.orchestra.run.smithi028:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/squid/f72fecff68e1d400c4568684327c900485c20d6a/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-05-12T21:56:01.678 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r--. 1 ubuntu ubuntu 781960 May 12 21:56 /home/ubuntu/cephtest/cephadm 2024-05-12T21:56:01.678 DEBUG:teuthology.orchestra.run.smithi086:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/squid/f72fecff68e1d400c4568684327c900485c20d6a/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-05-12T21:56:02.131 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r--. 1 ubuntu ubuntu 781960 May 12 21:56 /home/ubuntu/cephtest/cephadm 2024-05-12T21:56:02.131 DEBUG:teuthology.orchestra.run.smithi186:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/squid/f72fecff68e1d400c4568684327c900485c20d6a/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-05-12T21:56:02.800 INFO:teuthology.orchestra.run.smithi186.stdout:-rw-r--r--. 1 ubuntu ubuntu 781960 May 12 21:56 /home/ubuntu/cephtest/cephadm 2024-05-12T21:56:02.801 DEBUG:teuthology.orchestra.run.smithi028:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-05-12T21:56:02.853 DEBUG:teuthology.orchestra.run.smithi086:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-05-12T21:56:02.871 DEBUG:teuthology.orchestra.run.smithi186:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-05-12T21:56:02.906 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f72fecff68e1d400c4568684327c900485c20d6a on all hosts... 2024-05-12T21:56:02.907 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f72fecff68e1d400c4568684327c900485c20d6a pull 2024-05-12T21:56:02.915 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f72fecff68e1d400c4568684327c900485c20d6a pull 2024-05-12T21:56:02.917 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f72fecff68e1d400c4568684327c900485c20d6a pull 2024-05-12T21:56:04.472 INFO:teuthology.orchestra.run.smithi186.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f72fecff68e1d400c4568684327c900485c20d6a... 2024-05-12T21:56:04.512 INFO:teuthology.orchestra.run.smithi086.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f72fecff68e1d400c4568684327c900485c20d6a... 2024-05-12T21:56:05.884 INFO:teuthology.orchestra.run.smithi028.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f72fecff68e1d400c4568684327c900485c20d6a... 2024-05-12T21:57:05.382 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-05-12T21:57:05.383 INFO:teuthology.orchestra.run.smithi086.stdout: "ceph_version": "ceph version 19.0.0-2555-gf72fecff (f72fecff68e1d400c4568684327c900485c20d6a) squid (dev)", 2024-05-12T21:57:05.383 INFO:teuthology.orchestra.run.smithi086.stdout: "image_id": "8252b7944d3b5c188c0ff632647ba6cc4e9a19588ee097b8a3dba27b42779c25", 2024-05-12T21:57:05.383 INFO:teuthology.orchestra.run.smithi086.stdout: "repo_digests": [ 2024-05-12T21:57:05.383 INFO:teuthology.orchestra.run.smithi086.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc" 2024-05-12T21:57:05.383 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-05-12T21:57:05.383 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-05-12T21:57:05.734 INFO:teuthology.orchestra.run.smithi186.stdout:{ 2024-05-12T21:57:05.734 INFO:teuthology.orchestra.run.smithi186.stdout: "ceph_version": "ceph version 19.0.0-2555-gf72fecff (f72fecff68e1d400c4568684327c900485c20d6a) squid (dev)", 2024-05-12T21:57:05.734 INFO:teuthology.orchestra.run.smithi186.stdout: "image_id": "8252b7944d3b5c188c0ff632647ba6cc4e9a19588ee097b8a3dba27b42779c25", 2024-05-12T21:57:05.734 INFO:teuthology.orchestra.run.smithi186.stdout: "repo_digests": [ 2024-05-12T21:57:05.734 INFO:teuthology.orchestra.run.smithi186.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc" 2024-05-12T21:57:05.734 INFO:teuthology.orchestra.run.smithi186.stdout: ] 2024-05-12T21:57:05.734 INFO:teuthology.orchestra.run.smithi186.stdout:} 2024-05-12T21:57:16.642 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-05-12T21:57:16.643 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_version": "ceph version 19.0.0-2555-gf72fecff (f72fecff68e1d400c4568684327c900485c20d6a) squid (dev)", 2024-05-12T21:57:16.643 INFO:teuthology.orchestra.run.smithi028.stdout: "image_id": "8252b7944d3b5c188c0ff632647ba6cc4e9a19588ee097b8a3dba27b42779c25", 2024-05-12T21:57:16.643 INFO:teuthology.orchestra.run.smithi028.stdout: "repo_digests": [ 2024-05-12T21:57:16.643 INFO:teuthology.orchestra.run.smithi028.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc" 2024-05-12T21:57:16.643 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-05-12T21:57:16.643 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-05-12T21:57:16.691 DEBUG:teuthology.orchestra.run.smithi028:> sudo mkdir -p /etc/ceph 2024-05-12T21:57:16.786 DEBUG:teuthology.orchestra.run.smithi086:> sudo mkdir -p /etc/ceph 2024-05-12T21:57:16.817 DEBUG:teuthology.orchestra.run.smithi186:> sudo mkdir -p /etc/ceph 2024-05-12T21:57:16.850 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 777 /etc/ceph 2024-05-12T21:57:16.943 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod 777 /etc/ceph 2024-05-12T21:57:16.971 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod 777 /etc/ceph 2024-05-12T21:57:16.998 INFO:tasks.cephadm:Writing seed config... 2024-05-12T21:57:16.999 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-05-12T21:57:16.999 INFO:tasks.cephadm: override: [client] debug client = 20 2024-05-12T21:57:16.999 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-05-12T21:57:16.999 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-05-12T21:57:17.000 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-05-12T21:57:17.000 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-05-12T21:57:17.000 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-05-12T21:57:17.000 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-05-12T21:57:17.000 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-05-12T21:57:17.000 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-05-12T21:57:17.000 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-05-12T21:57:17.000 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-05-12T21:57:17.000 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-05-12T21:57:17.000 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-05-12T21:57:17.000 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-05-12T21:57:17.000 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-05-12T21:57:17.000 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-05-12T21:57:17.000 INFO:tasks.cephadm: override: [mds] mds_bal_replicate_threshold = 1 2024-05-12T21:57:17.000 INFO:tasks.cephadm: override: [mds] mds_export_ephemeral_random_max = 0.1 2024-05-12T21:57:17.000 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-05-12T21:57:17.001 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-05-12T21:57:17.001 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-05-12T21:57:17.001 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-05-12T21:57:17.001 INFO:tasks.cephadm: override: [mgr] debug cephsqlite = 20 2024-05-12T21:57:17.001 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-05-12T21:57:17.001 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-05-12T21:57:17.001 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-05-12T21:57:17.001 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-05-12T21:57:17.001 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-05-12T21:57:17.001 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-05-12T21:57:17.001 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-05-12T21:57:17.001 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-05-12T21:57:17.001 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2024-05-12T21:57:17.001 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2024-05-12T21:57:17.002 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2024-05-12T21:57:17.002 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-05-12T21:57:17.002 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-05-12T21:57:17.002 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2024-05-12T21:57:17.002 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2024-05-12T21:57:17.002 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-05-12T21:57:17.002 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-05-12T21:57:17.002 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2024-05-12T21:57:17.002 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-05-12T21:57:17.002 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-05-12T21:57:17.002 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-05-12T21:57:17.002 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-05-12T21:57:17.002 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-05-12T21:57:17.002 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-05-12T21:57:17.003 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-05-12T21:57:17.003 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10000 2024-05-12T21:57:17.003 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-05-12T21:57:17.003 DEBUG:teuthology.orchestra.run.smithi028:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-05-12T21:57:17.064 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 6b92e7b4-10aa-11ef-bc98-c7b262605968 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10000 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug cephsqlite = 20 debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True mds_bal_replicate_threshold = 1 mds_export_ephemeral_random_max = 0.1 osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-05-12T21:57:17.065 DEBUG:teuthology.orchestra.run.smithi028:mon.a> sudo journalctl -f -n 0 -u ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968@mon.a.service 2024-05-12T21:57:17.110 DEBUG:teuthology.orchestra.run.smithi028:mgr.x> sudo journalctl -f -n 0 -u ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968@mgr.x.service 2024-05-12T21:57:17.157 INFO:tasks.cephadm:Bootstrapping... 2024-05-12T21:57:17.157 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f72fecff68e1d400c4568684327c900485c20d6a -v bootstrap --fsid 6b92e7b4-10aa-11ef-bc98-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.28 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-05-12T21:57:17.435 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-05-12T21:57:17.436 INFO:teuthology.orchestra.run.smithi028.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f72fecff68e1d400c4568684327c900485c20d6a', '-v', 'bootstrap', '--fsid', '6b92e7b4-10aa-11ef-bc98-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.28', '--skip-admin-label'] 2024-05-12T21:57:17.436 INFO:teuthology.orchestra.run.smithi028.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-05-12T21:57:17.436 INFO:teuthology.orchestra.run.smithi028.stdout:Verifying podman|docker is present... 2024-05-12T21:57:17.497 INFO:teuthology.orchestra.run.smithi028.stdout:/bin/podman: stdout 4.9.4-dev 2024-05-12T21:57:17.498 INFO:teuthology.orchestra.run.smithi028.stdout:Verifying lvm2 is present... 2024-05-12T21:57:17.498 INFO:teuthology.orchestra.run.smithi028.stdout:Verifying time synchronization is in place... 2024-05-12T21:57:17.517 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-05-12T21:57:17.518 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-05-12T21:57:17.535 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-05-12T21:57:17.535 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout inactive 2024-05-12T21:57:17.554 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout enabled 2024-05-12T21:57:17.573 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout active 2024-05-12T21:57:17.574 INFO:teuthology.orchestra.run.smithi028.stdout:Unit chronyd.service is enabled and running 2024-05-12T21:57:17.574 INFO:teuthology.orchestra.run.smithi028.stdout:Repeating the final host check... 2024-05-12T21:57:17.636 INFO:teuthology.orchestra.run.smithi028.stdout:/bin/podman: stdout 4.9.4-dev 2024-05-12T21:57:17.637 INFO:teuthology.orchestra.run.smithi028.stdout:podman (/bin/podman) version 4.9.4 is present 2024-05-12T21:57:17.637 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl is present 2024-05-12T21:57:17.637 INFO:teuthology.orchestra.run.smithi028.stdout:lvcreate is present 2024-05-12T21:57:17.651 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-05-12T21:57:17.651 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-05-12T21:57:17.667 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-05-12T21:57:17.667 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout inactive 2024-05-12T21:57:17.687 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout enabled 2024-05-12T21:57:17.707 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout active 2024-05-12T21:57:17.708 INFO:teuthology.orchestra.run.smithi028.stdout:Unit chronyd.service is enabled and running 2024-05-12T21:57:17.708 INFO:teuthology.orchestra.run.smithi028.stdout:Host looks OK 2024-05-12T21:57:17.708 INFO:teuthology.orchestra.run.smithi028.stdout:Cluster fsid: 6b92e7b4-10aa-11ef-bc98-c7b262605968 2024-05-12T21:57:17.709 INFO:teuthology.orchestra.run.smithi028.stdout:Acquiring lock 139874454663280 on /run/cephadm/6b92e7b4-10aa-11ef-bc98-c7b262605968.lock 2024-05-12T21:57:17.709 INFO:teuthology.orchestra.run.smithi028.stdout:Lock 139874454663280 acquired on /run/cephadm/6b92e7b4-10aa-11ef-bc98-c7b262605968.lock 2024-05-12T21:57:17.711 INFO:teuthology.orchestra.run.smithi028.stdout:Verifying IP 172.21.15.28 port 3300 ... 2024-05-12T21:57:17.712 INFO:teuthology.orchestra.run.smithi028.stdout:Verifying IP 172.21.15.28 port 6789 ... 2024-05-12T21:57:17.712 INFO:teuthology.orchestra.run.smithi028.stdout:Base mon IP(s) is [172.21.15.28:3300, 172.21.15.28:6789], mon addrv is [v2:172.21.15.28:3300,v1:172.21.15.28:6789] 2024-05-12T21:57:17.727 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.28 metric 100 2024-05-12T21:57:17.728 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.28 metric 100 2024-05-12T21:57:17.741 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-05-12T21:57:17.741 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-05-12T21:57:17.741 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-05-12T21:57:17.752 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-05-12T21:57:17.753 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-05-12T21:57:17.753 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-05-12T21:57:17.753 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-05-12T21:57:17.753 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout inet6 fe80::ae1f:6bff:fea5:ca3c/64 scope link noprefixroute 2024-05-12T21:57:17.754 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-05-12T21:57:17.756 INFO:teuthology.orchestra.run.smithi028.stdout:Mon IP `172.21.15.28` is in CIDR network `172.21.0.0/20` 2024-05-12T21:57:17.756 INFO:teuthology.orchestra.run.smithi028.stdout:Mon IP `172.21.15.28` is in CIDR network `172.21.0.0/20` 2024-05-12T21:57:17.757 INFO:teuthology.orchestra.run.smithi028.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-05-12T21:57:17.760 INFO:teuthology.orchestra.run.smithi028.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-05-12T21:57:17.762 INFO:teuthology.orchestra.run.smithi028.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f72fecff68e1d400c4568684327c900485c20d6a... 2024-05-12T21:57:18.487 INFO:teuthology.orchestra.run.smithi028.stdout:/bin/podman: stdout 8252b7944d3b5c188c0ff632647ba6cc4e9a19588ee097b8a3dba27b42779c25 2024-05-12T21:57:18.487 INFO:teuthology.orchestra.run.smithi028.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f72fecff68e1d400c4568684327c900485c20d6a... 2024-05-12T21:57:18.487 INFO:teuthology.orchestra.run.smithi028.stdout:/bin/podman: stderr Getting image source signatures 2024-05-12T21:57:18.487 INFO:teuthology.orchestra.run.smithi028.stdout:/bin/podman: stderr Copying blob sha256:bf5b95bd68251da39c6cd2eb96c6ba4138bb37a62bdc39287908b25e92047d7e 2024-05-12T21:57:18.487 INFO:teuthology.orchestra.run.smithi028.stdout:/bin/podman: stderr Copying blob sha256:affb849b679ff538e1c9ed3d6ce866901e4b22d50526d9b556580893583381c4 2024-05-12T21:57:18.487 INFO:teuthology.orchestra.run.smithi028.stdout:/bin/podman: stderr Copying config sha256:8252b7944d3b5c188c0ff632647ba6cc4e9a19588ee097b8a3dba27b42779c25 2024-05-12T21:57:18.487 INFO:teuthology.orchestra.run.smithi028.stdout:/bin/podman: stderr Writing manifest to image destination 2024-05-12T21:57:19.776 INFO:teuthology.orchestra.run.smithi028.stdout:ceph: stdout ceph version 19.0.0-2555-gf72fecff (f72fecff68e1d400c4568684327c900485c20d6a) squid (dev) 2024-05-12T21:57:19.776 INFO:teuthology.orchestra.run.smithi028.stdout:Ceph version: ceph version 19.0.0-2555-gf72fecff (f72fecff68e1d400c4568684327c900485c20d6a) squid (dev) 2024-05-12T21:57:19.776 INFO:teuthology.orchestra.run.smithi028.stdout:Extracting ceph user uid/gid from container image... 2024-05-12T21:57:20.970 INFO:teuthology.orchestra.run.smithi028.stdout:stat: stdout 167 167 2024-05-12T21:57:20.970 INFO:teuthology.orchestra.run.smithi028.stdout:Creating initial keys... 2024-05-12T21:57:22.269 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph-authtool: stdout AQBBO0FmLODUGhAAT8lhZLR4hJX5XwAxR3mQRw== 2024-05-12T21:57:28.497 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph-authtool: stdout AQBCO0FmbcoJLRAAqzlP7nULl/3PjocJlls7uw== 2024-05-12T21:57:32.628 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph-authtool: stdout AQBLO0Fm02DRMhAACq6Pg6kZ7IjHmctwAkI6Uw== 2024-05-12T21:57:32.630 INFO:teuthology.orchestra.run.smithi028.stdout:Creating initial monmap... 2024-05-12T21:57:33.694 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-05-12T21:57:33.695 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-05-12T21:57:33.695 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 6b92e7b4-10aa-11ef-bc98-c7b262605968 2024-05-12T21:57:33.695 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-05-12T21:57:33.695 INFO:teuthology.orchestra.run.smithi028.stdout:monmaptool for a [v2:172.21.15.28:3300,v1:172.21.15.28:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-05-12T21:57:33.695 INFO:teuthology.orchestra.run.smithi028.stdout:setting min_mon_release = quincy 2024-05-12T21:57:33.695 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/monmaptool: set fsid to 6b92e7b4-10aa-11ef-bc98-c7b262605968 2024-05-12T21:57:33.695 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-05-12T21:57:33.695 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-12T21:57:33.695 INFO:teuthology.orchestra.run.smithi028.stdout:Creating mon... 2024-05-12T21:57:34.923 INFO:teuthology.orchestra.run.smithi028.stdout:create mon.a on 2024-05-12T21:57:35.351 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-05-12T21:57:35.587 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-05-12T21:57:35.867 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968.target → /etc/systemd/system/ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968.target. 2024-05-12T21:57:35.867 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968.target → /etc/systemd/system/ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968.target. 2024-05-12T21:57:36.202 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968@mon.a 2024-05-12T21:57:36.202 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Failed to reset failed state of unit ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968@mon.a.service: Unit ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968@mon.a.service not loaded. 2024-05-12T21:57:36.515 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968.target.wants/ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968@mon.a.service → /etc/systemd/system/ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968@.service. 2024-05-12T21:57:37.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:36 smithi028 podman[29461]: 2024-05-12 21:57:36.930823522 +0000 UTC m=+0.149858236 container create 93e59980d7d09518e3aca9216d0d9f33866de3a74bbd72a0301dccb8d8cc226e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f72fecff68e1d400c4568684327c900485c20d6a, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mon-a, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=squid-f72fecf, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240507) 2024-05-12T21:57:37.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:37 smithi028 podman[29461]: 2024-05-12 21:57:37.18447551 +0000 UTC m=+0.403510231 container init 93e59980d7d09518e3aca9216d0d9f33866de3a74bbd72a0301dccb8d8cc226e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f72fecff68e1d400c4568684327c900485c20d6a, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mon-a, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240507, ceph=True, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.license=GPLv2, RELEASE=squid-f72fecf, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-12T21:57:37.307 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-05-12T21:57:37.307 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout disabled 2024-05-12T21:57:37.322 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-05-12T21:57:37.322 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout inactive 2024-05-12T21:57:37.322 INFO:teuthology.orchestra.run.smithi028.stdout:firewalld.service is not enabled 2024-05-12T21:57:37.322 INFO:teuthology.orchestra.run.smithi028.stdout:Not possible to enable service . firewalld.service is not available 2024-05-12T21:57:37.323 INFO:teuthology.orchestra.run.smithi028.stdout:Waiting for mon to start... 2024-05-12T21:57:37.323 INFO:teuthology.orchestra.run.smithi028.stdout:Waiting for mon... 2024-05-12T21:57:37.470 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:37 smithi028 podman[29461]: 2024-05-12 21:57:37.196364537 +0000 UTC m=+0.415399257 container start 93e59980d7d09518e3aca9216d0d9f33866de3a74bbd72a0301dccb8d8cc226e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f72fecff68e1d400c4568684327c900485c20d6a, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mon-a, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240507, RELEASE=squid-f72fecf, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD) 2024-05-12T21:57:37.470 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:37 smithi028 bash[29461]: 93e59980d7d09518e3aca9216d0d9f33866de3a74bbd72a0301dccb8d8cc226e 2024-05-12T21:57:37.471 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:37 smithi028 ceph-mon[29479]: mkfs 6b92e7b4-10aa-11ef-bc98-c7b262605968 2024-05-12T21:57:37.471 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:37 smithi028 systemd[1]: Started Ceph mon.a for 6b92e7b4-10aa-11ef-bc98-c7b262605968. 2024-05-12T21:57:37.471 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:37 smithi028 ceph-mon[29479]: mon.a is new leader, mons a in quorum (ranks 0) 2024-05-12T21:57:38.581 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:38 smithi028 ceph-mon[29479]: mon.a is new leader, mons a in quorum (ranks 0) 2024-05-12T21:57:38.581 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:38 smithi028 ceph-mon[29479]: monmap epoch 1 2024-05-12T21:57:38.581 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:38 smithi028 ceph-mon[29479]: fsid 6b92e7b4-10aa-11ef-bc98-c7b262605968 2024-05-12T21:57:38.581 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:38 smithi028 ceph-mon[29479]: last_changed 2024-05-12T21:57:33.130972+0000 2024-05-12T21:57:38.581 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:38 smithi028 ceph-mon[29479]: created 2024-05-12T21:57:33.130972+0000 2024-05-12T21:57:38.581 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:38 smithi028 ceph-mon[29479]: min_mon_release 19 (squid) 2024-05-12T21:57:38.581 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:38 smithi028 ceph-mon[29479]: election_strategy: 1 2024-05-12T21:57:38.582 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:38 smithi028 ceph-mon[29479]: 0: [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] mon.a 2024-05-12T21:57:38.582 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:38 smithi028 ceph-mon[29479]: fsmap 2024-05-12T21:57:38.582 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:38 smithi028 ceph-mon[29479]: osdmap e1: 0 total, 0 up, 0 in 2024-05-12T21:57:38.582 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:38 smithi028 ceph-mon[29479]: mgrmap e1: no daemons active 2024-05-12T21:57:38.582 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:38 smithi028 ceph-mon[29479]: from='client.? 172.21.15.28:0/3781035653' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-05-12T21:57:38.603 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout cluster: 2024-05-12T21:57:38.604 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout id: 6b92e7b4-10aa-11ef-bc98-c7b262605968 2024-05-12T21:57:38.604 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-05-12T21:57:38.604 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-05-12T21:57:38.604 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout services: 2024-05-12T21:57:38.604 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.744683s) 2024-05-12T21:57:38.604 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-05-12T21:57:38.604 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-05-12T21:57:38.604 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-05-12T21:57:38.610 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout data: 2024-05-12T21:57:38.610 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-05-12T21:57:38.610 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-05-12T21:57:38.610 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-05-12T21:57:38.610 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout pgs: 2024-05-12T21:57:38.610 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-05-12T21:57:38.610 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:37.924+0000 7f877ae00700 1 Processor -- start 2024-05-12T21:57:38.610 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:37.925+0000 7f877ae00700 1 -- start start 2024-05-12T21:57:38.610 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:37.925+0000 7f877ae00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8774107e50 0x7f8774108270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T21:57:38.611 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:37.925+0000 7f877ae00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f87741087b0 con 0x7f8774107e50 2024-05-12T21:57:38.611 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:37.926+0000 7f8779a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8774107e50 0x7f8774108270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T21:57:38.611 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:37.926+0000 7f8779a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8774107e50 0x7f8774108270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35212/0 (socket says 172.21.15.28:35212) 2024-05-12T21:57:38.611 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:37.926+0000 7f8779a00700 1 -- 172.21.15.28:0/2506509182 learned_addr learned my addr 172.21.15.28:0/2506509182 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T21:57:38.611 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:37.927+0000 7f8779a00700 1 -- 172.21.15.28:0/2506509182 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8774108fb0 con 0x7f8774107e50 2024-05-12T21:57:38.611 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:37.927+0000 7f8779a00700 1 --2- 172.21.15.28:0/2506509182 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8774107e50 0x7f8774108270 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f87680098a0 tx=0x7f8768009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a4b088f7f1451ca server_cookie=0 in_seq=0 out_seq=0 2024-05-12T21:57:38.611 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:37.927+0000 7f8773e00700 1 -- 172.21.15.28:0/2506509182 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8768004160 con 0x7f8774107e50 2024-05-12T21:57:38.611 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:37.927+0000 7f8773e00700 1 -- 172.21.15.28:0/2506509182 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8768004300 con 0x7f8774107e50 2024-05-12T21:57:38.611 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:37.927+0000 7f8773e00700 1 -- 172.21.15.28:0/2506509182 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f87680036a0 con 0x7f8774107e50 2024-05-12T21:57:38.611 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:37.928+0000 7f877ae00700 1 -- 172.21.15.28:0/2506509182 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8774107e50 msgr2=0x7f8774108270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T21:57:38.611 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:37.928+0000 7f877ae00700 1 --2- 172.21.15.28:0/2506509182 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8774107e50 0x7f8774108270 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f87680098a0 tx=0x7f8768009bb0 comp rx=0 tx=0).stop 2024-05-12T21:57:38.611 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:37.928+0000 7f877ae00700 1 -- 172.21.15.28:0/2506509182 shutdown_connections 2024-05-12T21:57:38.611 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:37.928+0000 7f877ae00700 1 --2- 172.21.15.28:0/2506509182 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8774107e50 0x7f8774108270 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:57:38.612 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:37.928+0000 7f877ae00700 1 -- 172.21.15.28:0/2506509182 >> 172.21.15.28:0/2506509182 conn(0x7f8774103410 msgr2=0x7f8774105830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T21:57:38.612 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:37.928+0000 7f877ae00700 1 -- 172.21.15.28:0/2506509182 shutdown_connections 2024-05-12T21:57:38.612 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:37.928+0000 7f877ae00700 1 -- 172.21.15.28:0/2506509182 wait complete. 2024-05-12T21:57:38.612 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:37.929+0000 7f877ae00700 1 Processor -- start 2024-05-12T21:57:38.612 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:37.929+0000 7f877ae00700 1 -- start start 2024-05-12T21:57:38.612 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:37.930+0000 7f877ae00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8774107e50 0x7f877419d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T21:57:38.612 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:37.930+0000 7f877ae00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8774109480 con 0x7f8774107e50 2024-05-12T21:57:38.612 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:37.930+0000 7f8779a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8774107e50 0x7f877419d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T21:57:38.612 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:37.930+0000 7f8779a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8774107e50 0x7f877419d6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35222/0 (socket says 172.21.15.28:35222) 2024-05-12T21:57:38.612 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:37.930+0000 7f8779a00700 1 -- 172.21.15.28:0/3781035653 learned_addr learned my addr 172.21.15.28:0/3781035653 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T21:57:38.612 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:37.931+0000 7f8779a00700 1 -- 172.21.15.28:0/3781035653 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f877419dc10 con 0x7f8774107e50 2024-05-12T21:57:38.612 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:37.931+0000 7f8779a00700 1 --2- 172.21.15.28:0/3781035653 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8774107e50 0x7f877419d6d0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f8768000c00 tx=0x7f8768004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T21:57:38.612 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:37.931+0000 7f8772000700 1 -- 172.21.15.28:0/3781035653 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8768003db0 con 0x7f8774107e50 2024-05-12T21:57:38.612 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:37.932+0000 7f8772000700 1 -- 172.21.15.28:0/3781035653 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8768011630 con 0x7f8774107e50 2024-05-12T21:57:38.612 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:37.932+0000 7f877ae00700 1 -- 172.21.15.28:0/3781035653 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f877419dea0 con 0x7f8774107e50 2024-05-12T21:57:38.613 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:37.932+0000 7f8772000700 1 -- 172.21.15.28:0/3781035653 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8768011950 con 0x7f8774107e50 2024-05-12T21:57:38.613 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:37.932+0000 7f877ae00700 1 -- 172.21.15.28:0/3781035653 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87741a0b90 con 0x7f8774107e50 2024-05-12T21:57:38.613 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:37.932+0000 7f8772000700 1 -- 172.21.15.28:0/3781035653 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f8768011af0 con 0x7f8774107e50 2024-05-12T21:57:38.613 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:37.932+0000 7f8772000700 1 -- 172.21.15.28:0/3781035653 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f876801c450 con 0x7f8774107e50 2024-05-12T21:57:38.613 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:37.933+0000 7f877ae00700 1 -- 172.21.15.28:0/3781035653 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f87740694a0 con 0x7f8774107e50 2024-05-12T21:57:38.613 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:37.936+0000 7f8772000700 1 -- 172.21.15.28:0/3781035653 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f8768032020 con 0x7f8774107e50 2024-05-12T21:57:38.613 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:38.018+0000 7f877ae00700 1 -- 172.21.15.28:0/3781035653 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7f87741a0e00 con 0x7f8774107e50 2024-05-12T21:57:38.613 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:38.019+0000 7f8772000700 1 -- 172.21.15.28:0/3781035653 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+317 (secure 0 0 0) 0x7f8768004260 con 0x7f8774107e50 2024-05-12T21:57:38.613 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:38.021+0000 7f877ae00700 1 -- 172.21.15.28:0/3781035653 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8774107e50 msgr2=0x7f877419d6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T21:57:38.613 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:38.021+0000 7f877ae00700 1 --2- 172.21.15.28:0/3781035653 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8774107e50 0x7f877419d6d0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f8768000c00 tx=0x7f8768004750 comp rx=0 tx=0).stop 2024-05-12T21:57:38.613 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:38.021+0000 7f877ae00700 1 -- 172.21.15.28:0/3781035653 shutdown_connections 2024-05-12T21:57:38.613 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:38.021+0000 7f877ae00700 1 --2- 172.21.15.28:0/3781035653 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8774107e50 0x7f877419d6d0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:57:38.613 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:38.021+0000 7f877ae00700 1 -- 172.21.15.28:0/3781035653 >> 172.21.15.28:0/3781035653 conn(0x7f8774103410 msgr2=0x7f877407c960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T21:57:38.613 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:38.022+0000 7f877ae00700 1 -- 172.21.15.28:0/3781035653 shutdown_connections 2024-05-12T21:57:38.613 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:38.022+0000 7f877ae00700 1 -- 172.21.15.28:0/3781035653 wait complete. 2024-05-12T21:57:38.614 INFO:teuthology.orchestra.run.smithi028.stdout:mon is available 2024-05-12T21:57:38.614 INFO:teuthology.orchestra.run.smithi028.stdout:Assimilating anything we can from ceph.conf... 2024-05-12T21:57:39.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:39 smithi028 ceph-mon[29479]: from='client.? 172.21.15.28:0/1473831999' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-05-12T21:57:39.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:39 smithi028 ceph-mon[29479]: from='client.? 172.21.15.28:0/1473831999' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-05-12T21:57:39.938 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-05-12T21:57:39.938 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout [global] 2024-05-12T21:57:39.938 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout fsid = 6b92e7b4-10aa-11ef-bc98-c7b262605968 2024-05-12T21:57:39.938 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.28:3300,v1:172.21.15.28:6789] 2024-05-12T21:57:39.938 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-05-12T21:57:39.938 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-05-12T21:57:39.938 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-05-12T21:57:39.938 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-05-12T21:57:39.939 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-05-12T21:57:39.939 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout [mgr] 2024-05-12T21:57:39.939 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-05-12T21:57:39.939 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-05-12T21:57:39.939 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout [osd] 2024-05-12T21:57:39.939 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-05-12T21:57:39.939 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-05-12T21:57:39.939 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-05-12T21:57:39.939 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-05-12T21:57:39.939 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-05-12T21:57:39.939 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:39.245+0000 7fe64a800700 1 Processor -- start 2024-05-12T21:57:39.939 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:39.246+0000 7fe64a800700 1 -- start start 2024-05-12T21:57:39.939 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:39.246+0000 7fe64a800700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe644107e60 0x7fe644108280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T21:57:39.939 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:39.246+0000 7fe64a800700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe6441087c0 con 0x7fe644107e60 2024-05-12T21:57:39.939 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:39.247+0000 7fe649400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe644107e60 0x7fe644108280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T21:57:39.940 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:39.247+0000 7fe649400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe644107e60 0x7fe644108280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35230/0 (socket says 172.21.15.28:35230) 2024-05-12T21:57:39.940 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:39.247+0000 7fe649400700 1 -- 172.21.15.28:0/4265371693 learned_addr learned my addr 172.21.15.28:0/4265371693 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T21:57:39.940 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:39.247+0000 7fe649400700 1 -- 172.21.15.28:0/4265371693 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe644108fc0 con 0x7fe644107e60 2024-05-12T21:57:39.940 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:39.248+0000 7fe649400700 1 --2- 172.21.15.28:0/4265371693 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe644107e60 0x7fe644108280 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fe6380098a0 tx=0x7fe638009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f0fc9a28ec120d2 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T21:57:39.940 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:39.248+0000 7fe643e00700 1 -- 172.21.15.28:0/4265371693 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe638004160 con 0x7fe644107e60 2024-05-12T21:57:39.940 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:39.248+0000 7fe643e00700 1 -- 172.21.15.28:0/4265371693 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe638004300 con 0x7fe644107e60 2024-05-12T21:57:39.940 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:39.248+0000 7fe643e00700 1 -- 172.21.15.28:0/4265371693 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe6380036a0 con 0x7fe644107e60 2024-05-12T21:57:39.940 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:39.249+0000 7fe64a800700 1 -- 172.21.15.28:0/4265371693 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe644107e60 msgr2=0x7fe644108280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T21:57:39.940 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:39.249+0000 7fe64a800700 1 --2- 172.21.15.28:0/4265371693 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe644107e60 0x7fe644108280 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fe6380098a0 tx=0x7fe638009bb0 comp rx=0 tx=0).stop 2024-05-12T21:57:39.940 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:39.249+0000 7fe64a800700 1 -- 172.21.15.28:0/4265371693 shutdown_connections 2024-05-12T21:57:39.940 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:39.249+0000 7fe64a800700 1 --2- 172.21.15.28:0/4265371693 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe644107e60 0x7fe644108280 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:57:39.940 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:39.249+0000 7fe64a800700 1 -- 172.21.15.28:0/4265371693 >> 172.21.15.28:0/4265371693 conn(0x7fe644103400 msgr2=0x7fe644105840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T21:57:39.940 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:39.249+0000 7fe64a800700 1 -- 172.21.15.28:0/4265371693 shutdown_connections 2024-05-12T21:57:39.940 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:39.249+0000 7fe64a800700 1 -- 172.21.15.28:0/4265371693 wait complete. 2024-05-12T21:57:39.940 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:39.250+0000 7fe64a800700 1 Processor -- start 2024-05-12T21:57:39.941 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:39.251+0000 7fe64a800700 1 -- start start 2024-05-12T21:57:39.941 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:39.251+0000 7fe64a800700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe644107e60 0x7fe64419b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T21:57:39.942 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:39.251+0000 7fe64a800700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe644109490 con 0x7fe644107e60 2024-05-12T21:57:39.943 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:39.251+0000 7fe649400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe644107e60 0x7fe64419b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T21:57:39.943 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:39.251+0000 7fe649400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe644107e60 0x7fe64419b4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35246/0 (socket says 172.21.15.28:35246) 2024-05-12T21:57:39.943 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:39.251+0000 7fe649400700 1 -- 172.21.15.28:0/1473831999 learned_addr learned my addr 172.21.15.28:0/1473831999 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T21:57:39.943 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:39.251+0000 7fe649400700 1 -- 172.21.15.28:0/1473831999 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe64419b9f0 con 0x7fe644107e60 2024-05-12T21:57:39.943 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:39.252+0000 7fe649400700 1 --2- 172.21.15.28:0/1473831999 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe644107e60 0x7fe64419b4b0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fe638000c00 tx=0x7fe638004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T21:57:39.943 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:39.252+0000 7fe642000700 1 -- 172.21.15.28:0/1473831999 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe638003db0 con 0x7fe644107e60 2024-05-12T21:57:39.943 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:39.252+0000 7fe642000700 1 -- 172.21.15.28:0/1473831999 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe638011630 con 0x7fe644107e60 2024-05-12T21:57:39.943 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:39.252+0000 7fe64a800700 1 -- 172.21.15.28:0/1473831999 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe64419bc80 con 0x7fe644107e60 2024-05-12T21:57:39.943 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:39.252+0000 7fe642000700 1 -- 172.21.15.28:0/1473831999 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe638011950 con 0x7fe644107e60 2024-05-12T21:57:39.943 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:39.252+0000 7fe64a800700 1 -- 172.21.15.28:0/1473831999 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe64419e970 con 0x7fe644107e60 2024-05-12T21:57:39.943 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:39.253+0000 7fe642000700 1 -- 172.21.15.28:0/1473831999 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fe638011af0 con 0x7fe644107e60 2024-05-12T21:57:39.943 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:39.253+0000 7fe642000700 1 -- 172.21.15.28:0/1473831999 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fe63801c450 con 0x7fe644107e60 2024-05-12T21:57:39.943 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:39.254+0000 7fe64a800700 1 -- 172.21.15.28:0/1473831999 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe6440694a0 con 0x7fe644107e60 2024-05-12T21:57:39.943 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:39.256+0000 7fe642000700 1 -- 172.21.15.28:0/1473831999 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fe638032020 con 0x7fe644107e60 2024-05-12T21:57:39.943 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:39.335+0000 7fe64a800700 1 -- 172.21.15.28:0/1473831999 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7fe64402dd00 con 0x7fe644107e60 2024-05-12T21:57:39.944 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:39.346+0000 7fe642000700 1 -- 172.21.15.28:0/1473831999 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) ==== 70+0+429 (secure 0 0 0) 0x7fe638004260 con 0x7fe644107e60 2024-05-12T21:57:39.944 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:39.347+0000 7fe642000700 1 -- 172.21.15.28:0/1473831999 <== mon.0 v2:172.21.15.28:3300/0 8 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fe63801b450 con 0x7fe644107e60 2024-05-12T21:57:39.944 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:39.348+0000 7fe64a800700 1 -- 172.21.15.28:0/1473831999 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe644107e60 msgr2=0x7fe64419b4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T21:57:39.944 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:39.348+0000 7fe64a800700 1 --2- 172.21.15.28:0/1473831999 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe644107e60 0x7fe64419b4b0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fe638000c00 tx=0x7fe638004750 comp rx=0 tx=0).stop 2024-05-12T21:57:39.944 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:39.348+0000 7fe64a800700 1 -- 172.21.15.28:0/1473831999 shutdown_connections 2024-05-12T21:57:39.944 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:39.348+0000 7fe64a800700 1 --2- 172.21.15.28:0/1473831999 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe644107e60 0x7fe64419b4b0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:57:39.944 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:39.348+0000 7fe64a800700 1 -- 172.21.15.28:0/1473831999 >> 172.21.15.28:0/1473831999 conn(0x7fe644103400 msgr2=0x7fe644192110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T21:57:39.944 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:39.348+0000 7fe64a800700 1 -- 172.21.15.28:0/1473831999 shutdown_connections 2024-05-12T21:57:39.944 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:39.349+0000 7fe64a800700 1 -- 172.21.15.28:0/1473831999 wait complete. 2024-05-12T21:57:39.944 INFO:teuthology.orchestra.run.smithi028.stdout:Generating new minimal ceph.conf... 2024-05-12T21:57:40.953 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:40 smithi028 ceph-mon[29479]: from='client.? 172.21.15.28:0/4253508957' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T21:57:41.286 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:40.547+0000 7f4955000700 1 Processor -- start 2024-05-12T21:57:41.287 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:40.548+0000 7f4955000700 1 -- start start 2024-05-12T21:57:41.287 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:40.548+0000 7f4955000700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4950107f80 0x7f49501083a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T21:57:41.287 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:40.548+0000 7f4955000700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4950108970 con 0x7f4950107f80 2024-05-12T21:57:41.287 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:40.549+0000 7f494de00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4950107f80 0x7f49501083a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T21:57:41.287 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:40.549+0000 7f494de00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4950107f80 0x7f49501083a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35248/0 (socket says 172.21.15.28:35248) 2024-05-12T21:57:41.287 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:40.549+0000 7f494de00700 1 -- 172.21.15.28:0/640834971 learned_addr learned my addr 172.21.15.28:0/640834971 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T21:57:41.287 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:40.549+0000 7f494de00700 1 -- 172.21.15.28:0/640834971 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4950109180 con 0x7f4950107f80 2024-05-12T21:57:41.287 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:40.549+0000 7f494de00700 1 --2- 172.21.15.28:0/640834971 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4950107f80 0x7f49501083a0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f49400098a0 tx=0x7f4940009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed96eec9e9d62a3c server_cookie=0 in_seq=0 out_seq=0 2024-05-12T21:57:41.287 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:40.550+0000 7f494ca00700 1 -- 172.21.15.28:0/640834971 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4940004160 con 0x7f4950107f80 2024-05-12T21:57:41.287 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:40.550+0000 7f494ca00700 1 -- 172.21.15.28:0/640834971 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f4940004300 con 0x7f4950107f80 2024-05-12T21:57:41.291 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:40.550+0000 7f494ca00700 1 -- 172.21.15.28:0/640834971 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4940011b50 con 0x7f4950107f80 2024-05-12T21:57:41.291 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:40.550+0000 7f4955000700 1 -- 172.21.15.28:0/640834971 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4950107f80 msgr2=0x7f49501083a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T21:57:41.291 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:40.550+0000 7f4955000700 1 --2- 172.21.15.28:0/640834971 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4950107f80 0x7f49501083a0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f49400098a0 tx=0x7f4940009bb0 comp rx=0 tx=0).stop 2024-05-12T21:57:41.291 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:40.551+0000 7f4955000700 1 -- 172.21.15.28:0/640834971 shutdown_connections 2024-05-12T21:57:41.291 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:40.551+0000 7f4955000700 1 --2- 172.21.15.28:0/640834971 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4950107f80 0x7f49501083a0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:57:41.291 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:40.551+0000 7f4955000700 1 -- 172.21.15.28:0/640834971 >> 172.21.15.28:0/640834971 conn(0x7f4950103540 msgr2=0x7f4950105960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T21:57:41.291 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:40.551+0000 7f4955000700 1 -- 172.21.15.28:0/640834971 shutdown_connections 2024-05-12T21:57:41.291 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:40.551+0000 7f4955000700 1 -- 172.21.15.28:0/640834971 wait complete. 2024-05-12T21:57:41.291 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:40.552+0000 7f4955000700 1 Processor -- start 2024-05-12T21:57:41.292 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:40.552+0000 7f4955000700 1 -- start start 2024-05-12T21:57:41.292 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:40.552+0000 7f4955000700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4950107f80 0x7f495019e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T21:57:41.292 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:40.552+0000 7f4955000700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f495010c530 con 0x7f4950107f80 2024-05-12T21:57:41.292 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:40.553+0000 7f494de00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4950107f80 0x7f495019e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T21:57:41.292 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:40.553+0000 7f494de00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4950107f80 0x7f495019e140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35262/0 (socket says 172.21.15.28:35262) 2024-05-12T21:57:41.292 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:40.553+0000 7f494de00700 1 -- 172.21.15.28:0/4253508957 learned_addr learned my addr 172.21.15.28:0/4253508957 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T21:57:41.292 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:40.553+0000 7f494de00700 1 -- 172.21.15.28:0/4253508957 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f495019e680 con 0x7f4950107f80 2024-05-12T21:57:41.292 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:40.553+0000 7f494de00700 1 --2- 172.21.15.28:0/4253508957 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4950107f80 0x7f495019e140 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f4940000c00 tx=0x7f4940011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T21:57:41.292 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:40.554+0000 7f4946a00700 1 -- 172.21.15.28:0/4253508957 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4940003b20 con 0x7f4950107f80 2024-05-12T21:57:41.292 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:40.554+0000 7f4946a00700 1 -- 172.21.15.28:0/4253508957 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f4940003cc0 con 0x7f4950107f80 2024-05-12T21:57:41.292 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:40.554+0000 7f4946a00700 1 -- 172.21.15.28:0/4253508957 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4940024910 con 0x7f4950107f80 2024-05-12T21:57:41.292 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:40.554+0000 7f4955000700 1 -- 172.21.15.28:0/4253508957 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f495019e910 con 0x7f4950107f80 2024-05-12T21:57:41.292 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:40.554+0000 7f4955000700 1 -- 172.21.15.28:0/4253508957 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49501a1600 con 0x7f4950107f80 2024-05-12T21:57:41.292 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:40.554+0000 7f4946a00700 1 -- 172.21.15.28:0/4253508957 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f4940022020 con 0x7f4950107f80 2024-05-12T21:57:41.292 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:40.554+0000 7f4946a00700 1 -- 172.21.15.28:0/4253508957 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f4940011e00 con 0x7f4950107f80 2024-05-12T21:57:41.293 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:40.555+0000 7f4955000700 1 -- 172.21.15.28:0/4253508957 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4950063980 con 0x7f4950107f80 2024-05-12T21:57:41.293 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:40.558+0000 7f4946a00700 1 -- 172.21.15.28:0/4253508957 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f494002ddf0 con 0x7f4950107f80 2024-05-12T21:57:41.293 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:40.637+0000 7f4955000700 1 -- 172.21.15.28:0/4253508957 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f49501a17d0 con 0x7f4950107f80 2024-05-12T21:57:41.293 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:40.638+0000 7f4946a00700 1 -- 172.21.15.28:0/4253508957 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) ==== 76+0+175 (secure 0 0 0) 0x7f4940032030 con 0x7f4950107f80 2024-05-12T21:57:41.293 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:40.639+0000 7f4955000700 1 -- 172.21.15.28:0/4253508957 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4950107f80 msgr2=0x7f495019e140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T21:57:41.293 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:40.639+0000 7f4955000700 1 --2- 172.21.15.28:0/4253508957 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4950107f80 0x7f495019e140 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f4940000c00 tx=0x7f4940011c10 comp rx=0 tx=0).stop 2024-05-12T21:57:41.293 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:40.639+0000 7f4955000700 1 -- 172.21.15.28:0/4253508957 shutdown_connections 2024-05-12T21:57:41.293 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:40.639+0000 7f4955000700 1 --2- 172.21.15.28:0/4253508957 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4950107f80 0x7f495019e140 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:57:41.293 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:40.639+0000 7f4955000700 1 -- 172.21.15.28:0/4253508957 >> 172.21.15.28:0/4253508957 conn(0x7f4950103540 msgr2=0x7f4950104070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T21:57:41.293 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:40.639+0000 7f4955000700 1 -- 172.21.15.28:0/4253508957 shutdown_connections 2024-05-12T21:57:41.293 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:40.640+0000 7f4955000700 1 -- 172.21.15.28:0/4253508957 wait complete. 2024-05-12T21:57:41.294 INFO:teuthology.orchestra.run.smithi028.stdout:Restarting the monitor... 2024-05-12T21:57:41.534 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:41 smithi028 systemd[1]: Stopping Ceph mon.a for 6b92e7b4-10aa-11ef-bc98-c7b262605968... 2024-05-12T21:57:42.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:41 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mon-a[29475]: 2024-05-12T21:57:41.718+0000 7f7532400700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-05-12T21:57:42.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:41 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mon-a[29475]: 2024-05-12T21:57:41.718+0000 7f7532400700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-05-12T21:57:42.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:41 smithi028 podman[29721]: 2024-05-12 21:57:41.832760738 +0000 UTC m=+0.234227114 container died 93e59980d7d09518e3aca9216d0d9f33866de3a74bbd72a0301dccb8d8cc226e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f72fecff68e1d400c4568684327c900485c20d6a, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mon-a, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-f72fecf, ceph=True, org.label-schema.build-date=20240507, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-05-12T21:57:42.395 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:42 smithi028 podman[29721]: 2024-05-12 21:57:42.227543243 +0000 UTC m=+0.629009629 container cleanup 93e59980d7d09518e3aca9216d0d9f33866de3a74bbd72a0301dccb8d8cc226e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f72fecff68e1d400c4568684327c900485c20d6a, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mon-a, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240507, io.buildah.version=1.33.5, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.schema-version=1.0, ceph=True, RELEASE=squid-f72fecf, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-05-12T21:57:42.395 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:42 smithi028 bash[29721]: ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mon-a 2024-05-12T21:57:42.767 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:42 smithi028 podman[29735]: 2024-05-12 21:57:42.394740655 +0000 UTC m=+0.544733711 container remove 93e59980d7d09518e3aca9216d0d9f33866de3a74bbd72a0301dccb8d8cc226e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f72fecff68e1d400c4568684327c900485c20d6a, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mon-a, io.buildah.version=1.33.5, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-f72fecf, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240507, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-05-12T21:57:43.166 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:42 smithi028 systemd[1]: ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968@mon.a.service: Deactivated successfully. 2024-05-12T21:57:43.166 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:42 smithi028 systemd[1]: Stopped Ceph mon.a for 6b92e7b4-10aa-11ef-bc98-c7b262605968. 2024-05-12T21:57:43.166 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:42 smithi028 systemd[1]: ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968@mon.a.service: Consumed 1.799s CPU time. 2024-05-12T21:57:43.166 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:42 smithi028 systemd[1]: Starting Ceph mon.a for 6b92e7b4-10aa-11ef-bc98-c7b262605968... 2024-05-12T21:57:43.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 podman[29829]: 2024-05-12 21:57:43.072643043 +0000 UTC m=+0.035920869 image pull 8252b7944d3b5c188c0ff632647ba6cc4e9a19588ee097b8a3dba27b42779c25 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f72fecff68e1d400c4568684327c900485c20d6a 2024-05-12T21:57:43.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 podman[29829]: 2024-05-12 21:57:43.190431247 +0000 UTC m=+0.153709105 container create d414f473c6992788c451a5cddeb3df439654f5c4c6067e5b77723e23aab44fc9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f72fecff68e1d400c4568684327c900485c20d6a, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, RELEASE=squid-f72fecf, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240507, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-05-12T21:57:43.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 podman[29829]: 2024-05-12 21:57:43.434502892 +0000 UTC m=+0.397780756 container init d414f473c6992788c451a5cddeb3df439654f5c4c6067e5b77723e23aab44fc9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f72fecff68e1d400c4568684327c900485c20d6a, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mon-a, io.buildah.version=1.33.5, org.label-schema.build-date=20240507, RELEASE=squid-f72fecf, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-05-12T21:57:43.539 INFO:teuthology.orchestra.run.smithi028.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-05-12T21:57:43.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 podman[29829]: 2024-05-12 21:57:43.446000446 +0000 UTC m=+0.409278311 container start d414f473c6992788c451a5cddeb3df439654f5c4c6067e5b77723e23aab44fc9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f72fecff68e1d400c4568684327c900485c20d6a, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240507, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, RELEASE=squid-f72fecf, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-05-12T21:57:43.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: set uid:gid to 167:167 (ceph:ceph) 2024-05-12T21:57:43.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: ceph version 19.0.0-2555-gf72fecff (f72fecff68e1d400c4568684327c900485c20d6a) squid (dev), process ceph-mon, pid 2 2024-05-12T21:57:43.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: pidfile_write: ignore empty --pid-file 2024-05-12T21:57:43.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: load: jerasure load: lrc 2024-05-12T21:57:43.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: RocksDB version: 7.9.2 2024-05-12T21:57:43.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Git sha 0 2024-05-12T21:57:43.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Compile date 2024-05-10 15:23:03 2024-05-12T21:57:43.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: DB SUMMARY 2024-05-12T21:57:43.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: DB Session ID: 30G5HV4U0V4TDX0R5S8M 2024-05-12T21:57:43.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: CURRENT file: CURRENT 2024-05-12T21:57:43.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: IDENTITY file: IDENTITY 2024-05-12T21:57:43.689 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-05-12T21:57:43.689 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-05-12T21:57:43.689 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 106708 ; 2024-05-12T21:57:43.689 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.error_if_exists: 0 2024-05-12T21:57:43.689 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.create_if_missing: 0 2024-05-12T21:57:43.689 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.paranoid_checks: 1 2024-05-12T21:57:43.689 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.flush_verify_memtable_count: 1 2024-05-12T21:57:43.689 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-05-12T21:57:43.689 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-05-12T21:57:43.689 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.env: 0x557d455e3c60 2024-05-12T21:57:43.689 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.fs: PosixFileSystem 2024-05-12T21:57:43.689 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.info_log: 0x557d5b7c7b00 2024-05-12T21:57:43.689 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.max_file_opening_threads: 16 2024-05-12T21:57:43.689 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.statistics: (nil) 2024-05-12T21:57:43.689 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.use_fsync: 0 2024-05-12T21:57:43.689 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.max_log_file_size: 0 2024-05-12T21:57:43.690 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-05-12T21:57:43.690 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.log_file_time_to_roll: 0 2024-05-12T21:57:43.690 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.keep_log_file_num: 1000 2024-05-12T21:57:43.690 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.recycle_log_file_num: 0 2024-05-12T21:57:43.690 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.allow_fallocate: 1 2024-05-12T21:57:43.690 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.allow_mmap_reads: 0 2024-05-12T21:57:43.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.allow_mmap_writes: 0 2024-05-12T21:57:43.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.use_direct_reads: 0 2024-05-12T21:57:43.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-05-12T21:57:43.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.create_missing_column_families: 0 2024-05-12T21:57:43.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.db_log_dir: 2024-05-12T21:57:43.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.wal_dir: 2024-05-12T21:57:43.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.table_cache_numshardbits: 6 2024-05-12T21:57:43.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.WAL_ttl_seconds: 0 2024-05-12T21:57:43.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.WAL_size_limit_MB: 0 2024-05-12T21:57:43.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-05-12T21:57:43.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-05-12T21:57:43.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.is_fd_close_on_exec: 1 2024-05-12T21:57:43.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.advise_random_on_open: 1 2024-05-12T21:57:43.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.db_write_buffer_size: 0 2024-05-12T21:57:43.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.write_buffer_manager: 0x557d5b79a1e0 2024-05-12T21:57:43.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-05-12T21:57:43.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-05-12T21:57:43.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.use_adaptive_mutex: 0 2024-05-12T21:57:43.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.rate_limiter: (nil) 2024-05-12T21:57:43.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-05-12T21:57:43.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.wal_recovery_mode: 2 2024-05-12T21:57:43.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.enable_thread_tracking: 0 2024-05-12T21:57:43.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.enable_pipelined_write: 0 2024-05-12T21:57:43.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.unordered_write: 0 2024-05-12T21:57:43.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-05-12T21:57:43.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-05-12T21:57:43.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-05-12T21:57:43.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-05-12T21:57:43.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.row_cache: None 2024-05-12T21:57:43.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.wal_filter: None 2024-05-12T21:57:43.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-05-12T21:57:43.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.allow_ingest_behind: 0 2024-05-12T21:57:43.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.two_write_queues: 0 2024-05-12T21:57:43.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.manual_wal_flush: 0 2024-05-12T21:57:43.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.wal_compression: 0 2024-05-12T21:57:43.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.atomic_flush: 0 2024-05-12T21:57:43.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-05-12T21:57:43.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.persist_stats_to_disk: 0 2024-05-12T21:57:43.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.write_dbid_to_manifest: 0 2024-05-12T21:57:43.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.log_readahead_size: 0 2024-05-12T21:57:43.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-05-12T21:57:43.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.best_efforts_recovery: 0 2024-05-12T21:57:43.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-05-12T21:57:43.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-05-12T21:57:43.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.allow_data_in_errors: 0 2024-05-12T21:57:43.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.db_host_id: __hostname__ 2024-05-12T21:57:43.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.enforce_single_del_contracts: true 2024-05-12T21:57:43.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.max_background_jobs: 2 2024-05-12T21:57:43.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.max_background_compactions: -1 2024-05-12T21:57:43.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.max_subcompactions: 1 2024-05-12T21:57:43.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-05-12T21:57:43.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-05-12T21:57:43.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.delayed_write_rate : 16777216 2024-05-12T21:57:43.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.max_total_wal_size: 0 2024-05-12T21:57:43.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-05-12T21:57:43.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.stats_dump_period_sec: 600 2024-05-12T21:57:43.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.stats_persist_period_sec: 600 2024-05-12T21:57:43.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-05-12T21:57:43.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.max_open_files: -1 2024-05-12T21:57:43.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.bytes_per_sync: 0 2024-05-12T21:57:43.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.wal_bytes_per_sync: 0 2024-05-12T21:57:43.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.strict_bytes_per_sync: 0 2024-05-12T21:57:43.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.compaction_readahead_size: 0 2024-05-12T21:57:43.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.max_background_flushes: -1 2024-05-12T21:57:43.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Compression algorithms supported: 2024-05-12T21:57:43.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-05-12T21:57:43.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: kZSTD supported: 0 2024-05-12T21:57:43.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: kXpressCompression supported: 0 2024-05-12T21:57:43.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: kLZ4HCCompression supported: 1 2024-05-12T21:57:43.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: kZlibCompression supported: 1 2024-05-12T21:57:43.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: kSnappyCompression supported: 1 2024-05-12T21:57:43.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: kLZ4Compression supported: 1 2024-05-12T21:57:43.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: kBZip2Compression supported: 0 2024-05-12T21:57:43.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Fast CRC32 supported: Supported on x86 2024-05-12T21:57:43.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: DMutex implementation: pthread_mutex_t 2024-05-12T21:57:43.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-05-12T21:57:43.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-05-12T21:57:43.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-05-12T21:57:43.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.merge_operator: 2024-05-12T21:57:43.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.compaction_filter: None 2024-05-12T21:57:43.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.compaction_filter_factory: None 2024-05-12T21:57:43.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.sst_partitioner_factory: None 2024-05-12T21:57:43.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.memtable_factory: SkipListFactory 2024-05-12T21:57:43.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.table_factory: BlockBasedTable 2024-05-12T21:57:43.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x557d5b7c7c20) 2024-05-12T21:57:43.697 INFO:journalctl@ceph.mon.a.smithi028.stdout: cache_index_and_filter_blocks: 1 2024-05-12T21:57:43.697 INFO:journalctl@ceph.mon.a.smithi028.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-05-12T21:57:43.697 INFO:journalctl@ceph.mon.a.smithi028.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-05-12T21:57:43.697 INFO:journalctl@ceph.mon.a.smithi028.stdout: pin_top_level_index_and_filter: 1 2024-05-12T21:57:43.697 INFO:journalctl@ceph.mon.a.smithi028.stdout: index_type: 0 2024-05-12T21:57:43.697 INFO:journalctl@ceph.mon.a.smithi028.stdout: data_block_index_type: 0 2024-05-12T21:57:43.697 INFO:journalctl@ceph.mon.a.smithi028.stdout: index_shortening: 1 2024-05-12T21:57:43.698 INFO:journalctl@ceph.mon.a.smithi028.stdout: data_block_hash_table_util_ratio: 0.750000 2024-05-12T21:57:43.698 INFO:journalctl@ceph.mon.a.smithi028.stdout: checksum: 4 2024-05-12T21:57:43.698 INFO:journalctl@ceph.mon.a.smithi028.stdout: no_block_cache: 0 2024-05-12T21:57:43.698 INFO:journalctl@ceph.mon.a.smithi028.stdout: block_cache: 0x557d5b7cb090 2024-05-12T21:57:43.698 INFO:journalctl@ceph.mon.a.smithi028.stdout: block_cache_name: BinnedLRUCache 2024-05-12T21:57:43.698 INFO:journalctl@ceph.mon.a.smithi028.stdout: block_cache_options: 2024-05-12T21:57:43.698 INFO:journalctl@ceph.mon.a.smithi028.stdout: capacity : 536870912 2024-05-12T21:57:43.698 INFO:journalctl@ceph.mon.a.smithi028.stdout: num_shard_bits : 4 2024-05-12T21:57:43.698 INFO:journalctl@ceph.mon.a.smithi028.stdout: strict_capacity_limit : 0 2024-05-12T21:57:43.698 INFO:journalctl@ceph.mon.a.smithi028.stdout: high_pri_pool_ratio: 0.000 2024-05-12T21:57:43.698 INFO:journalctl@ceph.mon.a.smithi028.stdout: block_cache_compressed: (nil) 2024-05-12T21:57:43.699 INFO:journalctl@ceph.mon.a.smithi028.stdout: persistent_cache: (nil) 2024-05-12T21:57:43.699 INFO:journalctl@ceph.mon.a.smithi028.stdout: block_size: 4096 2024-05-12T21:57:43.699 INFO:journalctl@ceph.mon.a.smithi028.stdout: block_size_deviation: 10 2024-05-12T21:57:43.699 INFO:journalctl@ceph.mon.a.smithi028.stdout: block_restart_interval: 16 2024-05-12T21:57:43.699 INFO:journalctl@ceph.mon.a.smithi028.stdout: index_block_restart_interval: 1 2024-05-12T21:57:43.699 INFO:journalctl@ceph.mon.a.smithi028.stdout: metadata_block_size: 4096 2024-05-12T21:57:43.699 INFO:journalctl@ceph.mon.a.smithi028.stdout: partition_filters: 0 2024-05-12T21:57:43.699 INFO:journalctl@ceph.mon.a.smithi028.stdout: use_delta_encoding: 1 2024-05-12T21:57:43.699 INFO:journalctl@ceph.mon.a.smithi028.stdout: filter_policy: bloomfilter 2024-05-12T21:57:43.699 INFO:journalctl@ceph.mon.a.smithi028.stdout: whole_key_filtering: 1 2024-05-12T21:57:43.699 INFO:journalctl@ceph.mon.a.smithi028.stdout: verify_compression: 0 2024-05-12T21:57:43.699 INFO:journalctl@ceph.mon.a.smithi028.stdout: read_amp_bytes_per_bit: 0 2024-05-12T21:57:43.699 INFO:journalctl@ceph.mon.a.smithi028.stdout: format_version: 5 2024-05-12T21:57:43.699 INFO:journalctl@ceph.mon.a.smithi028.stdout: enable_index_compression: 1 2024-05-12T21:57:43.699 INFO:journalctl@ceph.mon.a.smithi028.stdout: block_align: 0 2024-05-12T21:57:43.700 INFO:journalctl@ceph.mon.a.smithi028.stdout: max_auto_readahead_size: 262144 2024-05-12T21:57:43.700 INFO:journalctl@ceph.mon.a.smithi028.stdout: prepopulate_block_cache: 0 2024-05-12T21:57:43.700 INFO:journalctl@ceph.mon.a.smithi028.stdout: initial_auto_readahead_size: 8192 2024-05-12T21:57:43.700 INFO:journalctl@ceph.mon.a.smithi028.stdout: num_file_reads_for_auto_readahead: 2 2024-05-12T21:57:43.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.write_buffer_size: 33554432 2024-05-12T21:57:43.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.max_write_buffer_number: 2 2024-05-12T21:57:43.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.compression: NoCompression 2024-05-12T21:57:43.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.bottommost_compression: Disabled 2024-05-12T21:57:43.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.prefix_extractor: nullptr 2024-05-12T21:57:43.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-05-12T21:57:43.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.num_levels: 7 2024-05-12T21:57:43.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-05-12T21:57:43.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-05-12T21:57:43.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-05-12T21:57:43.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-05-12T21:57:43.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-05-12T21:57:43.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-05-12T21:57:43.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-05-12T21:57:43.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-05-12T21:57:43.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-05-12T21:57:43.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-05-12T21:57:43.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-05-12T21:57:43.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-05-12T21:57:43.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.compression_opts.window_bits: -14 2024-05-12T21:57:43.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.compression_opts.level: 32767 2024-05-12T21:57:43.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.compression_opts.strategy: 0 2024-05-12T21:57:43.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-05-12T21:57:43.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-05-12T21:57:43.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-05-12T21:57:43.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-05-12T21:57:43.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.compression_opts.enabled: false 2024-05-12T21:57:43.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-05-12T21:57:43.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-05-12T21:57:43.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-05-12T21:57:43.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-05-12T21:57:43.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.target_file_size_base: 67108864 2024-05-12T21:57:43.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.target_file_size_multiplier: 1 2024-05-12T21:57:43.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-05-12T21:57:43.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-05-12T21:57:43.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-05-12T21:57:43.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-05-12T21:57:43.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-05-12T21:57:43.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-05-12T21:57:43.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-05-12T21:57:43.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-05-12T21:57:43.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-05-12T21:57:43.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-05-12T21:57:43.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-05-12T21:57:43.703 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-05-12T21:57:43.703 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-05-12T21:57:43.703 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.arena_block_size: 1048576 2024-05-12T21:57:43.703 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-05-12T21:57:43.703 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-05-12T21:57:43.703 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.disable_auto_compactions: 0 2024-05-12T21:57:43.703 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-05-12T21:57:43.703 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-05-12T21:57:43.703 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-05-12T21:57:43.703 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-05-12T21:57:43.703 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-05-12T21:57:43.703 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-05-12T21:57:43.703 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-05-12T21:57:43.703 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-05-12T21:57:43.703 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-05-12T21:57:43.704 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-05-12T21:57:43.704 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.table_properties_collectors: 2024-05-12T21:57:43.704 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.inplace_update_support: 0 2024-05-12T21:57:43.704 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.inplace_update_num_locks: 10000 2024-05-12T21:57:43.704 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-05-12T21:57:43.704 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-05-12T21:57:43.704 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.memtable_huge_page_size: 0 2024-05-12T21:57:43.704 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.bloom_locality: 0 2024-05-12T21:57:43.704 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.max_successive_merges: 0 2024-05-12T21:57:43.704 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.optimize_filters_for_hits: 0 2024-05-12T21:57:43.704 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.paranoid_file_checks: 0 2024-05-12T21:57:43.704 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.force_consistency_checks: 1 2024-05-12T21:57:43.704 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.report_bg_io_stats: 0 2024-05-12T21:57:43.704 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.ttl: 2592000 2024-05-12T21:57:43.704 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.periodic_compaction_seconds: 0 2024-05-12T21:57:43.704 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-05-12T21:57:43.705 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-05-12T21:57:43.705 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.enable_blob_files: false 2024-05-12T21:57:43.705 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.min_blob_size: 0 2024-05-12T21:57:43.705 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.blob_file_size: 268435456 2024-05-12T21:57:43.705 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.blob_compression_type: NoCompression 2024-05-12T21:57:43.705 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.enable_blob_garbage_collection: false 2024-05-12T21:57:43.705 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-05-12T21:57:43.705 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-05-12T21:57:43.705 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-05-12T21:57:43.705 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.blob_file_starting_level: 0 2024-05-12T21:57:43.705 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-05-12T21:57:43.705 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-05-12T21:57:43.705 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-05-12T21:57:43.705 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: a55df28b-ad47-440d-9e4e-f7ff92a9bb27 2024-05-12T21:57:43.705 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715551063504558, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-05-12T21:57:43.705 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-05-12T21:57:43.706 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715551063506262, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 101582, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 334, "table_properties": {"data_size": 99493, "index_size": 307, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 773, "raw_key_size": 14657, "raw_average_key_size": 50, "raw_value_size": 91596, "raw_average_value_size": 312, "num_data_blocks": 12, "num_entries": 293, "num_filter_entries": 293, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1715551063, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "a55df28b-ad47-440d-9e4e-f7ff92a9bb27", "db_session_id": "30G5HV4U0V4TDX0R5S8M", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-05-12T21:57:43.706 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715551063506555, "job": 1, "event": "recovery_finished"} 2024-05-12T21:57:43.706 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-05-12T21:57:43.706 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-05-12T21:57:43.706 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x557d5b8bc000 2024-05-12T21:57:43.706 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: DB pointer 0x557d5b8a6000 2024-05-12T21:57:43.706 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-05-12T21:57:43.706 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-05-12T21:57:43.706 INFO:journalctl@ceph.mon.a.smithi028.stdout: ** DB Stats ** 2024-05-12T21:57:43.706 INFO:journalctl@ceph.mon.a.smithi028.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-12T21:57:43.706 INFO:journalctl@ceph.mon.a.smithi028.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-05-12T21:57:43.706 INFO:journalctl@ceph.mon.a.smithi028.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-05-12T21:57:43.706 INFO:journalctl@ceph.mon.a.smithi028.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-05-12T21:57:43.706 INFO:journalctl@ceph.mon.a.smithi028.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-05-12T21:57:43.706 INFO:journalctl@ceph.mon.a.smithi028.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-05-12T21:57:43.707 INFO:journalctl@ceph.mon.a.smithi028.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-05-12T21:57:43.707 INFO:journalctl@ceph.mon.a.smithi028.stdout: 2024-05-12T21:57:43.707 INFO:journalctl@ceph.mon.a.smithi028.stdout: ** Compaction Stats [default] ** 2024-05-12T21:57:43.707 INFO:journalctl@ceph.mon.a.smithi028.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-05-12T21:57:43.707 INFO:journalctl@ceph.mon.a.smithi028.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-05-12T21:57:43.707 INFO:journalctl@ceph.mon.a.smithi028.stdout: L0 2/0 101.04 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 92.6 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-05-12T21:57:43.707 INFO:journalctl@ceph.mon.a.smithi028.stdout: Sum 2/0 101.04 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 92.6 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-05-12T21:57:43.707 INFO:journalctl@ceph.mon.a.smithi028.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 92.6 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-05-12T21:57:43.707 INFO:journalctl@ceph.mon.a.smithi028.stdout: 2024-05-12T21:57:43.707 INFO:journalctl@ceph.mon.a.smithi028.stdout: ** Compaction Stats [default] ** 2024-05-12T21:57:43.707 INFO:journalctl@ceph.mon.a.smithi028.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-05-12T21:57:43.707 INFO:journalctl@ceph.mon.a.smithi028.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-05-12T21:57:43.707 INFO:journalctl@ceph.mon.a.smithi028.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 92.6 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-05-12T21:57:43.707 INFO:journalctl@ceph.mon.a.smithi028.stdout: 2024-05-12T21:57:43.707 INFO:journalctl@ceph.mon.a.smithi028.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-05-12T21:57:43.708 INFO:journalctl@ceph.mon.a.smithi028.stdout: 2024-05-12T21:57:43.708 INFO:journalctl@ceph.mon.a.smithi028.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-12T21:57:43.708 INFO:journalctl@ceph.mon.a.smithi028.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-05-12T21:57:43.708 INFO:journalctl@ceph.mon.a.smithi028.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-05-12T21:57:43.708 INFO:journalctl@ceph.mon.a.smithi028.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-05-12T21:57:43.708 INFO:journalctl@ceph.mon.a.smithi028.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-05-12T21:57:43.708 INFO:journalctl@ceph.mon.a.smithi028.stdout: AddFile(Keys): cumulative 0, interval 0 2024-05-12T21:57:43.708 INFO:journalctl@ceph.mon.a.smithi028.stdout: Cumulative compaction: 0.00 GB write, 14.82 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-12T21:57:43.708 INFO:journalctl@ceph.mon.a.smithi028.stdout: Interval compaction: 0.00 GB write, 14.82 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-12T21:57:43.708 INFO:journalctl@ceph.mon.a.smithi028.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-05-12T21:57:43.708 INFO:journalctl@ceph.mon.a.smithi028.stdout: Block cache BinnedLRUCache@0x557d5b7cb090#2 capacity: 512.00 MB usage: 1.52 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.1e-05 secs_since: 0 2024-05-12T21:57:43.708 INFO:journalctl@ceph.mon.a.smithi028.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,1.02 KB,0.000193715%) IndexBlock(2,0.50 KB,9.53674e-05%) Misc(1,0.00 KB,0%) 2024-05-12T21:57:43.708 INFO:journalctl@ceph.mon.a.smithi028.stdout: 2024-05-12T21:57:43.709 INFO:journalctl@ceph.mon.a.smithi028.stdout: ** File Read Latency Histogram By Level [default] ** 2024-05-12T21:57:43.709 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: starting mon.a rank 0 at public addrs [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] at bind addrs [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 6b92e7b4-10aa-11ef-bc98-c7b262605968 2024-05-12T21:57:43.709 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: mon.a@-1(???) e1 preinit fsid 6b92e7b4-10aa-11ef-bc98-c7b262605968 2024-05-12T21:57:43.709 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: mon.a@-1(???).mds e1 new map 2024-05-12T21:57:43.709 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: mon.a@-1(???).mds e1 print_map 2024-05-12T21:57:43.709 INFO:journalctl@ceph.mon.a.smithi028.stdout: e1 2024-05-12T21:57:43.709 INFO:journalctl@ceph.mon.a.smithi028.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-05-12T21:57:43.709 INFO:journalctl@ceph.mon.a.smithi028.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-05-12T21:57:43.709 INFO:journalctl@ceph.mon.a.smithi028.stdout: legacy client fscid: -1 2024-05-12T21:57:43.709 INFO:journalctl@ceph.mon.a.smithi028.stdout: 2024-05-12T21:57:43.709 INFO:journalctl@ceph.mon.a.smithi028.stdout: No filesystems configured 2024-05-12T21:57:43.709 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-05-12T21:57:43.709 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-05-12T21:57:43.709 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-05-12T21:57:43.710 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-05-12T21:57:43.710 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-05-12T21:57:43.710 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 bash[29829]: d414f473c6992788c451a5cddeb3df439654f5c4c6067e5b77723e23aab44fc9 2024-05-12T21:57:43.710 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 systemd[1]: Started Ceph mon.a for 6b92e7b4-10aa-11ef-bc98-c7b262605968. 2024-05-12T21:57:43.710 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: mon.a is new leader, mons a in quorum (ranks 0) 2024-05-12T21:57:43.710 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: monmap epoch 1 2024-05-12T21:57:43.710 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: fsid 6b92e7b4-10aa-11ef-bc98-c7b262605968 2024-05-12T21:57:43.710 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: last_changed 2024-05-12T21:57:33.130972+0000 2024-05-12T21:57:43.710 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: created 2024-05-12T21:57:33.130972+0000 2024-05-12T21:57:43.710 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: min_mon_release 19 (squid) 2024-05-12T21:57:43.710 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: election_strategy: 1 2024-05-12T21:57:43.710 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: 0: [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] mon.a 2024-05-12T21:57:43.710 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: fsmap 2024-05-12T21:57:43.710 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: osdmap e1: 0 total, 0 up, 0 in 2024-05-12T21:57:43.710 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:43 smithi028 ceph-mon[29847]: mgrmap e1: no daemons active 2024-05-12T21:57:44.956 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:44.220+0000 7fbbbf200700 1 Processor -- start 2024-05-12T21:57:44.957 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:44.221+0000 7fbbbf200700 1 -- start start 2024-05-12T21:57:44.957 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:44.221+0000 7fbbbf200700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbbb8106ce0 0x7fbbb8107100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T21:57:44.957 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:44.221+0000 7fbbbf200700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbbb81076d0 con 0x7fbbb8106ce0 2024-05-12T21:57:44.957 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:44.222+0000 7fbbbde00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbbb8106ce0 0x7fbbb8107100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T21:57:44.957 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:44.222+0000 7fbbbde00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbbb8106ce0 0x7fbbb8107100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35270/0 (socket says 172.21.15.28:35270) 2024-05-12T21:57:44.957 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:44.222+0000 7fbbbde00700 1 -- 172.21.15.28:0/3647796963 learned_addr learned my addr 172.21.15.28:0/3647796963 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T21:57:44.957 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:44.223+0000 7fbbbde00700 1 -- 172.21.15.28:0/3647796963 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbb8107ee0 con 0x7fbbb8106ce0 2024-05-12T21:57:44.957 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:44.223+0000 7fbbbde00700 1 --2- 172.21.15.28:0/3647796963 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbbb8106ce0 0x7fbbb8107100 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fbbac0098a0 tx=0x7fbbac009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74aced80b7876a37 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T21:57:44.957 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:44.224+0000 7fbbbca00700 1 -- 172.21.15.28:0/3647796963 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbbac004160 con 0x7fbbb8106ce0 2024-05-12T21:57:44.958 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:44.224+0000 7fbbbca00700 1 -- 172.21.15.28:0/3647796963 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fbbac004300 con 0x7fbbb8106ce0 2024-05-12T21:57:44.958 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:44.224+0000 7fbbbca00700 1 -- 172.21.15.28:0/3647796963 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbbac00fbc0 con 0x7fbbb8106ce0 2024-05-12T21:57:44.958 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:44.224+0000 7fbbbf200700 1 -- 172.21.15.28:0/3647796963 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbbb8106ce0 msgr2=0x7fbbb8107100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T21:57:44.958 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:44.224+0000 7fbbbf200700 1 --2- 172.21.15.28:0/3647796963 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbbb8106ce0 0x7fbbb8107100 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fbbac0098a0 tx=0x7fbbac009bb0 comp rx=0 tx=0).stop 2024-05-12T21:57:44.958 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:44.224+0000 7fbbbf200700 1 -- 172.21.15.28:0/3647796963 shutdown_connections 2024-05-12T21:57:44.958 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:44.225+0000 7fbbbf200700 1 --2- 172.21.15.28:0/3647796963 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbbb8106ce0 0x7fbbb8107100 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:57:44.958 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:44.225+0000 7fbbbf200700 1 -- 172.21.15.28:0/3647796963 >> 172.21.15.28:0/3647796963 conn(0x7fbbb8102280 msgr2=0x7fbbb81046c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T21:57:44.958 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:44.225+0000 7fbbbf200700 1 -- 172.21.15.28:0/3647796963 shutdown_connections 2024-05-12T21:57:44.958 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:44.225+0000 7fbbbf200700 1 -- 172.21.15.28:0/3647796963 wait complete. 2024-05-12T21:57:44.958 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:44.226+0000 7fbbbf200700 1 Processor -- start 2024-05-12T21:57:44.958 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:44.226+0000 7fbbbf200700 1 -- start start 2024-05-12T21:57:44.958 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:44.226+0000 7fbbbf200700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbbb8106ce0 0x7fbbb819dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T21:57:44.959 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:44.226+0000 7fbbbf200700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbbb8062760 con 0x7fbbb8106ce0 2024-05-12T21:57:44.959 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:44.227+0000 7fbbbde00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbbb8106ce0 0x7fbbb819dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T21:57:44.959 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:44.227+0000 7fbbbde00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbbb8106ce0 0x7fbbb819dff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35284/0 (socket says 172.21.15.28:35284) 2024-05-12T21:57:44.959 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:44.227+0000 7fbbbde00700 1 -- 172.21.15.28:0/564493431 learned_addr learned my addr 172.21.15.28:0/564493431 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T21:57:44.959 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:44.227+0000 7fbbbde00700 1 -- 172.21.15.28:0/564493431 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbb819e530 con 0x7fbbb8106ce0 2024-05-12T21:57:44.959 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:44.227+0000 7fbbbde00700 1 --2- 172.21.15.28:0/564493431 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbbb8106ce0 0x7fbbb819dff0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fbbac00f8c0 tx=0x7fbbac00ffb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T21:57:44.959 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:44.228+0000 7fbbb2a00700 1 -- 172.21.15.28:0/564493431 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbbac003ad0 con 0x7fbbb8106ce0 2024-05-12T21:57:44.959 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:44.228+0000 7fbbbf200700 1 -- 172.21.15.28:0/564493431 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbbb819e7c0 con 0x7fbbb8106ce0 2024-05-12T21:57:44.959 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:44.228+0000 7fbbb2a00700 1 -- 172.21.15.28:0/564493431 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fbbac01d070 con 0x7fbbb8106ce0 2024-05-12T21:57:44.959 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:44.228+0000 7fbbbf200700 1 -- 172.21.15.28:0/564493431 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbbb81a14b0 con 0x7fbbb8106ce0 2024-05-12T21:57:44.960 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:44.228+0000 7fbbb2a00700 1 -- 172.21.15.28:0/564493431 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbbac021540 con 0x7fbbb8106ce0 2024-05-12T21:57:44.960 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:44.228+0000 7fbbb2a00700 1 -- 172.21.15.28:0/564493431 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fbbac028060 con 0x7fbbb8106ce0 2024-05-12T21:57:44.960 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:44.228+0000 7fbbb2a00700 1 -- 172.21.15.28:0/564493431 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fbbac02aae0 con 0x7fbbb8106ce0 2024-05-12T21:57:44.960 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:44.229+0000 7fbbbf200700 1 -- 172.21.15.28:0/564493431 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbbb80694a0 con 0x7fbbb8106ce0 2024-05-12T21:57:44.960 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:44.232+0000 7fbbb2a00700 1 -- 172.21.15.28:0/564493431 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fbbac015070 con 0x7fbbb8106ce0 2024-05-12T21:57:44.960 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:44.310+0000 7fbbbf200700 1 -- 172.21.15.28:0/564493431 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) -- 0x7fbbb8193140 con 0x7fbbb8106ce0 2024-05-12T21:57:44.960 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:44.314+0000 7fbbb2a00700 1 -- 172.21.15.28:0/564493431 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) ==== 124+0+0 (secure 0 0 0) 0x7fbbac0042a0 con 0x7fbbb8106ce0 2024-05-12T21:57:44.960 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:44.315+0000 7fbbbf200700 1 -- 172.21.15.28:0/564493431 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbbb8106ce0 msgr2=0x7fbbb819dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T21:57:44.960 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:44.315+0000 7fbbbf200700 1 --2- 172.21.15.28:0/564493431 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbbb8106ce0 0x7fbbb819dff0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fbbac00f8c0 tx=0x7fbbac00ffb0 comp rx=0 tx=0).stop 2024-05-12T21:57:44.960 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:44.315+0000 7fbbbf200700 1 -- 172.21.15.28:0/564493431 shutdown_connections 2024-05-12T21:57:44.960 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:44.315+0000 7fbbbf200700 1 --2- 172.21.15.28:0/564493431 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbbb8106ce0 0x7fbbb819dff0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:57:44.961 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:44.315+0000 7fbbbf200700 1 -- 172.21.15.28:0/564493431 >> 172.21.15.28:0/564493431 conn(0x7fbbb8102280 msgr2=0x7fbbb8190200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T21:57:44.961 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:44.316+0000 7fbbbf200700 1 -- 172.21.15.28:0/564493431 shutdown_connections 2024-05-12T21:57:44.961 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:44.316+0000 7fbbbf200700 1 -- 172.21.15.28:0/564493431 wait complete. 2024-05-12T21:57:44.961 INFO:teuthology.orchestra.run.smithi028.stdout:Wrote config to /etc/ceph/ceph.conf 2024-05-12T21:57:44.978 INFO:teuthology.orchestra.run.smithi028.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-05-12T21:57:44.978 INFO:teuthology.orchestra.run.smithi028.stdout:Creating mgr... 2024-05-12T21:57:44.978 INFO:teuthology.orchestra.run.smithi028.stdout:Verifying port 0.0.0.0:9283 ... 2024-05-12T21:57:44.979 INFO:teuthology.orchestra.run.smithi028.stdout:Verifying port 0.0.0.0:8765 ... 2024-05-12T21:57:45.424 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968@mgr.x 2024-05-12T21:57:45.424 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Failed to reset failed state of unit ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968@mgr.x.service: Unit ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968@mgr.x.service not loaded. 2024-05-12T21:57:45.697 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968.target.wants/ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968@mgr.x.service → /etc/systemd/system/ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968@.service. 2024-05-12T21:57:45.732 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:45 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/564493431' entity='client.admin' 2024-05-12T21:57:46.035 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 21:57:45 smithi028 systemd[1]: Starting Ceph mgr.x for 6b92e7b4-10aa-11ef-bc98-c7b262605968... 2024-05-12T21:57:46.384 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 21:57:46 smithi028 podman[30063]: 2024-05-12 21:57:46.105800408 +0000 UTC m=+0.126062211 container create 1cf9bf8977739000e8e79d4d9f12daf54a215193bfc5820e3a728e0af49d0def (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f72fecff68e1d400c4568684327c900485c20d6a, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240507, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-f72fecf, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e) 2024-05-12T21:57:46.385 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 21:57:46 smithi028 podman[30063]: 2024-05-12 21:57:46.016372655 +0000 UTC m=+0.036634427 image pull 8252b7944d3b5c188c0ff632647ba6cc4e9a19588ee097b8a3dba27b42779c25 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f72fecff68e1d400c4568684327c900485c20d6a 2024-05-12T21:57:46.385 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 21:57:46 smithi028 podman[30063]: 2024-05-12 21:57:46.384309467 +0000 UTC m=+0.404571286 container init 1cf9bf8977739000e8e79d4d9f12daf54a215193bfc5820e3a728e0af49d0def (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f72fecff68e1d400c4568684327c900485c20d6a, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x, org.label-schema.build-date=20240507, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=squid-f72fecf, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, GIT_BRANCH=HEAD) 2024-05-12T21:57:46.531 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-05-12T21:57:46.532 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout disabled 2024-05-12T21:57:46.549 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-05-12T21:57:46.549 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout inactive 2024-05-12T21:57:46.549 INFO:teuthology.orchestra.run.smithi028.stdout:firewalld.service is not enabled 2024-05-12T21:57:46.549 INFO:teuthology.orchestra.run.smithi028.stdout:Not possible to enable service . firewalld.service is not available 2024-05-12T21:57:46.572 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-05-12T21:57:46.572 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout disabled 2024-05-12T21:57:46.589 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-05-12T21:57:46.589 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout inactive 2024-05-12T21:57:46.589 INFO:teuthology.orchestra.run.smithi028.stdout:firewalld.service is not enabled 2024-05-12T21:57:46.589 INFO:teuthology.orchestra.run.smithi028.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-05-12T21:57:46.590 INFO:teuthology.orchestra.run.smithi028.stdout:Waiting for mgr to start... 2024-05-12T21:57:46.590 INFO:teuthology.orchestra.run.smithi028.stdout:Waiting for mgr... 2024-05-12T21:57:46.653 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 21:57:46 smithi028 podman[30063]: 2024-05-12 21:57:46.394401868 +0000 UTC m=+0.414663644 container start 1cf9bf8977739000e8e79d4d9f12daf54a215193bfc5820e3a728e0af49d0def (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f72fecff68e1d400c4568684327c900485c20d6a, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x, RELEASE=squid-f72fecf, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240507, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-05-12T21:57:46.654 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 21:57:46 smithi028 bash[30063]: 1cf9bf8977739000e8e79d4d9f12daf54a215193bfc5820e3a728e0af49d0def 2024-05-12T21:57:46.654 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 21:57:46 smithi028 systemd[1]: Started Ceph mgr.x for 6b92e7b4-10aa-11ef-bc98-c7b262605968. 2024-05-12T21:57:46.931 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 21:57:46 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T21:57:46.652+0000 7ff3179cd200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-12T21:57:46.931 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 21:57:46 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T21:57:46.930+0000 7ff3179cd200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-12T21:57:47.244 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 21:57:47 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T21:57:47.085+0000 7ff3179cd200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-12T21:57:47.528 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:47 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/2839903927' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-05-12T21:57:47.785 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 21:57:47 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T21:57:47.526+0000 7ff3179cd200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-12T21:57:47.785 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 21:57:47 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T21:57:47.653+0000 7ff3179cd200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-12T21:57:48.006 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-05-12T21:57:48.008 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout { 2024-05-12T21:57:48.008 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "fsid": "6b92e7b4-10aa-11ef-bc98-c7b262605968", 2024-05-12T21:57:48.008 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "health": { 2024-05-12T21:57:48.008 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-05-12T21:57:48.008 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-05-12T21:57:48.009 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-05-12T21:57:48.009 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-05-12T21:57:48.009 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-05-12T21:57:48.009 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-05-12T21:57:48.009 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 0 2024-05-12T21:57:48.009 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-05-12T21:57:48.009 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-05-12T21:57:48.009 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "a" 2024-05-12T21:57:48.009 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-05-12T21:57:48.010 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum_age": 3, 2024-05-12T21:57:48.010 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "monmap": { 2024-05-12T21:57:48.011 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-12T21:57:48.011 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-05-12T21:57:48.011 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-05-12T21:57:48.011 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-05-12T21:57:48.011 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-05-12T21:57:48.011 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-12T21:57:48.011 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-05-12T21:57:48.011 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-05-12T21:57:48.011 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-05-12T21:57:48.020 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-05-12T21:57:48.020 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-05-12T21:57:48.020 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-05-12T21:57:48.020 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-05-12T21:57:48.020 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-05-12T21:57:48.021 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-05-12T21:57:48.021 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-05-12T21:57:48.021 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-05-12T21:57:48.021 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-05-12T21:57:48.021 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-05-12T21:57:48.021 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-05-12T21:57:48.021 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-05-12T21:57:48.021 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-05-12T21:57:48.021 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-05-12T21:57:48.021 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-05-12T21:57:48.021 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-12T21:57:48.021 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-05-12T21:57:48.021 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-05-12T21:57:48.021 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-05-12T21:57:48.021 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-05-12T21:57:48.022 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "available": false, 2024-05-12T21:57:48.022 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-05-12T21:57:48.022 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "modules": [ 2024-05-12T21:57:48.022 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "iostat", 2024-05-12T21:57:48.022 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "nfs", 2024-05-12T21:57:48.022 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "restful" 2024-05-12T21:57:48.022 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-05-12T21:57:48.022 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-12T21:57:48.022 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-05-12T21:57:48.022 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-05-12T21:57:48.022 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-12T21:57:48.022 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "modified": "2024-05-12T21:57:37.277650+0000", 2024-05-12T21:57:48.022 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-12T21:57:48.022 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-05-12T21:57:48.022 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-05-12T21:57:48.023 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout } 2024-05-12T21:57:48.023 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:47.279+0000 7f66f1e00700 1 Processor -- start 2024-05-12T21:57:48.023 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:47.279+0000 7f66f1e00700 1 -- start start 2024-05-12T21:57:48.023 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:47.279+0000 7f66f1e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66ec1067b0 0x7f66ec106bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T21:57:48.023 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:47.279+0000 7f66f1e00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f66ec063640 con 0x7f66ec1067b0 2024-05-12T21:57:48.023 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:47.280+0000 7f66eaa00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66ec1067b0 0x7f66ec106bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T21:57:48.023 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:47.280+0000 7f66eaa00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66ec1067b0 0x7f66ec106bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39646/0 (socket says 172.21.15.28:39646) 2024-05-12T21:57:48.023 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:47.280+0000 7f66eaa00700 1 -- 172.21.15.28:0/1981337776 learned_addr learned my addr 172.21.15.28:0/1981337776 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T21:57:48.023 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:47.280+0000 7f66eaa00700 1 -- 172.21.15.28:0/1981337776 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66ec107110 con 0x7f66ec1067b0 2024-05-12T21:57:48.023 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:47.281+0000 7f66eaa00700 1 --2- 172.21.15.28:0/1981337776 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66ec1067b0 0x7f66ec106bd0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f66dc0098a0 tx=0x7f66dc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c726f1c6fefe154b server_cookie=0 in_seq=0 out_seq=0 2024-05-12T21:57:48.023 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:47.281+0000 7f66e9600700 1 -- 172.21.15.28:0/1981337776 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f66dc004160 con 0x7f66ec1067b0 2024-05-12T21:57:48.023 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:47.281+0000 7f66e9600700 1 -- 172.21.15.28:0/1981337776 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f66dc004300 con 0x7f66ec1067b0 2024-05-12T21:57:48.023 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:47.281+0000 7f66e9600700 1 -- 172.21.15.28:0/1981337776 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f66dc011b50 con 0x7f66ec1067b0 2024-05-12T21:57:48.023 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:47.282+0000 7f66f1e00700 1 -- 172.21.15.28:0/1981337776 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66ec1067b0 msgr2=0x7f66ec106bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T21:57:48.024 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:47.282+0000 7f66f1e00700 1 --2- 172.21.15.28:0/1981337776 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66ec1067b0 0x7f66ec106bd0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f66dc0098a0 tx=0x7f66dc009bb0 comp rx=0 tx=0).stop 2024-05-12T21:57:48.024 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:47.282+0000 7f66f1e00700 1 -- 172.21.15.28:0/1981337776 shutdown_connections 2024-05-12T21:57:48.024 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:47.282+0000 7f66f1e00700 1 --2- 172.21.15.28:0/1981337776 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66ec1067b0 0x7f66ec106bd0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:57:48.024 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:47.282+0000 7f66f1e00700 1 -- 172.21.15.28:0/1981337776 >> 172.21.15.28:0/1981337776 conn(0x7f66ec102410 msgr2=0x7f66ec104830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T21:57:48.024 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:47.282+0000 7f66f1e00700 1 -- 172.21.15.28:0/1981337776 shutdown_connections 2024-05-12T21:57:48.024 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:47.283+0000 7f66f1e00700 1 -- 172.21.15.28:0/1981337776 wait complete. 2024-05-12T21:57:48.024 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:47.284+0000 7f66f1e00700 1 Processor -- start 2024-05-12T21:57:48.024 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:47.284+0000 7f66f1e00700 1 -- start start 2024-05-12T21:57:48.024 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:47.284+0000 7f66f1e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66ec1067b0 0x7f66ec19e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T21:57:48.024 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:47.284+0000 7f66f1e00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f66ec1081a0 con 0x7f66ec1067b0 2024-05-12T21:57:48.024 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:47.285+0000 7f66eaa00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66ec1067b0 0x7f66ec19e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T21:57:48.024 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:47.285+0000 7f66eaa00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66ec1067b0 0x7f66ec19e0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39654/0 (socket says 172.21.15.28:39654) 2024-05-12T21:57:48.024 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:47.285+0000 7f66eaa00700 1 -- 172.21.15.28:0/2839903927 learned_addr learned my addr 172.21.15.28:0/2839903927 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T21:57:48.024 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:47.285+0000 7f66eaa00700 1 -- 172.21.15.28:0/2839903927 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66ec19e5e0 con 0x7f66ec1067b0 2024-05-12T21:57:48.024 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:47.285+0000 7f66eaa00700 1 --2- 172.21.15.28:0/2839903927 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66ec1067b0 0x7f66ec19e0a0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f66dc011b60 tx=0x7f66dc011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T21:57:48.025 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:47.286+0000 7f66f0a00700 1 -- 172.21.15.28:0/2839903927 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f66dc003b20 con 0x7f66ec1067b0 2024-05-12T21:57:48.025 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:47.286+0000 7f66f0a00700 1 -- 172.21.15.28:0/2839903927 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f66dc003cc0 con 0x7f66ec1067b0 2024-05-12T21:57:48.025 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:47.286+0000 7f66f0a00700 1 -- 172.21.15.28:0/2839903927 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f66dc024840 con 0x7f66ec1067b0 2024-05-12T21:57:48.025 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:47.286+0000 7f66f1e00700 1 -- 172.21.15.28:0/2839903927 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66ec19e870 con 0x7f66ec1067b0 2024-05-12T21:57:48.025 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:47.286+0000 7f66f1e00700 1 -- 172.21.15.28:0/2839903927 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66ec1a1560 con 0x7f66ec1067b0 2024-05-12T21:57:48.025 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:47.286+0000 7f66f0a00700 1 -- 172.21.15.28:0/2839903927 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f66dc022020 con 0x7f66ec1067b0 2024-05-12T21:57:48.025 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:47.286+0000 7f66f0a00700 1 -- 172.21.15.28:0/2839903927 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f66dc011e00 con 0x7f66ec1067b0 2024-05-12T21:57:48.025 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:47.287+0000 7f66f1e00700 1 -- 172.21.15.28:0/2839903927 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f66ec063980 con 0x7f66ec1067b0 2024-05-12T21:57:48.025 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:47.290+0000 7f66f0a00700 1 -- 172.21.15.28:0/2839903927 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f66dc043dd0 con 0x7f66ec1067b0 2024-05-12T21:57:48.025 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:47.370+0000 7f66f1e00700 1 -- 172.21.15.28:0/2839903927 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f66ec02dd80 con 0x7f66ec1067b0 2024-05-12T21:57:48.025 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:47.371+0000 7f66f0a00700 1 -- 172.21.15.28:0/2839903927 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f66dc032020 con 0x7f66ec1067b0 2024-05-12T21:57:48.025 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:47.372+0000 7f66f1e00700 1 -- 172.21.15.28:0/2839903927 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66ec1067b0 msgr2=0x7f66ec19e0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T21:57:48.025 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:47.372+0000 7f66f1e00700 1 --2- 172.21.15.28:0/2839903927 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66ec1067b0 0x7f66ec19e0a0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f66dc011b60 tx=0x7f66dc011c40 comp rx=0 tx=0).stop 2024-05-12T21:57:48.025 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:47.372+0000 7f66f1e00700 1 -- 172.21.15.28:0/2839903927 shutdown_connections 2024-05-12T21:57:48.026 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:47.372+0000 7f66f1e00700 1 --2- 172.21.15.28:0/2839903927 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66ec1067b0 0x7f66ec19e0a0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:57:48.026 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:47.372+0000 7f66f1e00700 1 -- 172.21.15.28:0/2839903927 >> 172.21.15.28:0/2839903927 conn(0x7f66ec102410 msgr2=0x7f66ec194d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T21:57:48.026 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:47.373+0000 7f66f1e00700 1 -- 172.21.15.28:0/2839903927 shutdown_connections 2024-05-12T21:57:48.026 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:47.373+0000 7f66f1e00700 1 -- 172.21.15.28:0/2839903927 wait complete. 2024-05-12T21:57:48.026 INFO:teuthology.orchestra.run.smithi028.stdout:mgr not available, waiting (1/15)... 2024-05-12T21:57:49.775 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 21:57:49 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T21:57:49.324+0000 7ff3179cd200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-12T21:57:49.775 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 21:57:49 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T21:57:49.557+0000 7ff3179cd200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-12T21:57:49.776 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 21:57:49 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T21:57:49.773+0000 7ff3179cd200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-12T21:57:50.475 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 21:57:50 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T21:57:50.168+0000 7ff3179cd200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-12T21:57:50.475 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 21:57:50 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T21:57:50.296+0000 7ff3179cd200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-12T21:57:50.725 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 21:57:50 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T21:57:50.561+0000 7ff3179cd200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-12T21:57:51.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:50 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/3877319927' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-05-12T21:57:51.035 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 21:57:50 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T21:57:50.827+0000 7ff3179cd200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-12T21:57:51.404 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-05-12T21:57:51.404 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout { 2024-05-12T21:57:51.404 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "fsid": "6b92e7b4-10aa-11ef-bc98-c7b262605968", 2024-05-12T21:57:51.404 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "health": { 2024-05-12T21:57:51.404 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-05-12T21:57:51.404 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-05-12T21:57:51.404 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-05-12T21:57:51.404 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-05-12T21:57:51.404 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-05-12T21:57:51.404 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-05-12T21:57:51.404 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 0 2024-05-12T21:57:51.409 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-05-12T21:57:51.409 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-05-12T21:57:51.410 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "a" 2024-05-12T21:57:51.410 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-05-12T21:57:51.410 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum_age": 7, 2024-05-12T21:57:51.410 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "monmap": { 2024-05-12T21:57:51.410 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-12T21:57:51.410 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-05-12T21:57:51.410 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-05-12T21:57:51.410 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-05-12T21:57:51.410 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-05-12T21:57:51.410 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-12T21:57:51.410 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-05-12T21:57:51.410 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-05-12T21:57:51.410 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-05-12T21:57:51.410 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-05-12T21:57:51.410 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-05-12T21:57:51.411 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-05-12T21:57:51.411 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-05-12T21:57:51.411 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-05-12T21:57:51.411 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-05-12T21:57:51.411 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-05-12T21:57:51.411 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-05-12T21:57:51.411 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-05-12T21:57:51.411 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-05-12T21:57:51.411 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-05-12T21:57:51.411 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-05-12T21:57:51.411 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-05-12T21:57:51.413 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-05-12T21:57:51.413 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-05-12T21:57:51.413 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-12T21:57:51.413 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-05-12T21:57:51.413 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-05-12T21:57:51.413 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-05-12T21:57:51.413 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-05-12T21:57:51.413 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "available": false, 2024-05-12T21:57:51.413 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-05-12T21:57:51.413 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "modules": [ 2024-05-12T21:57:51.413 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "iostat", 2024-05-12T21:57:51.413 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "nfs", 2024-05-12T21:57:51.413 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "restful" 2024-05-12T21:57:51.414 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-05-12T21:57:51.414 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-12T21:57:51.414 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-05-12T21:57:51.414 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-05-12T21:57:51.414 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-12T21:57:51.414 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "modified": "2024-05-12T21:57:37.277650+0000", 2024-05-12T21:57:51.414 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-12T21:57:51.420 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-05-12T21:57:51.420 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-05-12T21:57:51.420 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout } 2024-05-12T21:57:51.420 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:50.633+0000 7f0b24800700 1 Processor -- start 2024-05-12T21:57:51.420 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:50.633+0000 7f0b24800700 1 -- start start 2024-05-12T21:57:51.420 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:50.633+0000 7f0b24800700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0b1c106e50 0x7f0b1c107270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T21:57:51.420 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:50.633+0000 7f0b24800700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0b1c1077b0 con 0x7f0b1c106e50 2024-05-12T21:57:51.420 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:50.634+0000 7f0b21e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0b1c106e50 0x7f0b1c107270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T21:57:51.420 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:50.634+0000 7f0b21e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0b1c106e50 0x7f0b1c107270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39666/0 (socket says 172.21.15.28:39666) 2024-05-12T21:57:51.421 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:50.634+0000 7f0b21e00700 1 -- 172.21.15.28:0/1148995160 learned_addr learned my addr 172.21.15.28:0/1148995160 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T21:57:51.421 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:50.635+0000 7f0b21e00700 1 -- 172.21.15.28:0/1148995160 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b1c107fb0 con 0x7f0b1c106e50 2024-05-12T21:57:51.421 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:50.635+0000 7f0b21e00700 1 --2- 172.21.15.28:0/1148995160 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0b1c106e50 0x7f0b1c107270 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f0b100098a0 tx=0x7f0b10009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36be6bc4c431b166 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T21:57:51.421 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:50.635+0000 7f0b20a00700 1 -- 172.21.15.28:0/1148995160 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0b10004160 con 0x7f0b1c106e50 2024-05-12T21:57:51.421 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:50.635+0000 7f0b20a00700 1 -- 172.21.15.28:0/1148995160 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f0b10004300 con 0x7f0b1c106e50 2024-05-12T21:57:51.421 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:50.635+0000 7f0b20a00700 1 -- 172.21.15.28:0/1148995160 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0b10011b50 con 0x7f0b1c106e50 2024-05-12T21:57:51.421 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:50.636+0000 7f0b24800700 1 -- 172.21.15.28:0/1148995160 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0b1c106e50 msgr2=0x7f0b1c107270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T21:57:51.421 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:50.636+0000 7f0b24800700 1 --2- 172.21.15.28:0/1148995160 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0b1c106e50 0x7f0b1c107270 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f0b100098a0 tx=0x7f0b10009bb0 comp rx=0 tx=0).stop 2024-05-12T21:57:51.421 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:50.636+0000 7f0b24800700 1 -- 172.21.15.28:0/1148995160 shutdown_connections 2024-05-12T21:57:51.421 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:50.636+0000 7f0b24800700 1 --2- 172.21.15.28:0/1148995160 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0b1c106e50 0x7f0b1c107270 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:57:51.421 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:50.636+0000 7f0b24800700 1 -- 172.21.15.28:0/1148995160 >> 172.21.15.28:0/1148995160 conn(0x7f0b1c102410 msgr2=0x7f0b1c104830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T21:57:51.421 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:50.636+0000 7f0b24800700 1 -- 172.21.15.28:0/1148995160 shutdown_connections 2024-05-12T21:57:51.421 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:50.636+0000 7f0b24800700 1 -- 172.21.15.28:0/1148995160 wait complete. 2024-05-12T21:57:51.421 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:50.637+0000 7f0b24800700 1 Processor -- start 2024-05-12T21:57:51.421 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:50.637+0000 7f0b24800700 1 -- start start 2024-05-12T21:57:51.422 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:50.638+0000 7f0b24800700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0b1c106e50 0x7f0b1c191ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T21:57:51.422 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:50.638+0000 7f0b24800700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0b1c10b3f0 con 0x7f0b1c106e50 2024-05-12T21:57:51.422 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:50.638+0000 7f0b21e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0b1c106e50 0x7f0b1c191ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T21:57:51.422 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:50.638+0000 7f0b21e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0b1c106e50 0x7f0b1c191ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39680/0 (socket says 172.21.15.28:39680) 2024-05-12T21:57:51.422 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:50.638+0000 7f0b21e00700 1 -- 172.21.15.28:0/3877319927 learned_addr learned my addr 172.21.15.28:0/3877319927 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T21:57:51.422 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:50.638+0000 7f0b21e00700 1 -- 172.21.15.28:0/3877319927 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b1c192530 con 0x7f0b1c106e50 2024-05-12T21:57:51.422 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:50.639+0000 7f0b21e00700 1 --2- 172.21.15.28:0/3877319927 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0b1c106e50 0x7f0b1c191ff0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f0b10011b60 tx=0x7f0b10011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T21:57:51.422 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:50.639+0000 7f0b16a00700 1 -- 172.21.15.28:0/3877319927 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0b10003b20 con 0x7f0b1c106e50 2024-05-12T21:57:51.422 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:50.639+0000 7f0b16a00700 1 -- 172.21.15.28:0/3877319927 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f0b10003cc0 con 0x7f0b1c106e50 2024-05-12T21:57:51.422 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:50.639+0000 7f0b16a00700 1 -- 172.21.15.28:0/3877319927 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0b10024840 con 0x7f0b1c106e50 2024-05-12T21:57:51.422 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:50.639+0000 7f0b24800700 1 -- 172.21.15.28:0/3877319927 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b1c190720 con 0x7f0b1c106e50 2024-05-12T21:57:51.422 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:50.639+0000 7f0b24800700 1 -- 172.21.15.28:0/3877319927 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b1c190c60 con 0x7f0b1c106e50 2024-05-12T21:57:51.422 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:50.640+0000 7f0b16a00700 1 -- 172.21.15.28:0/3877319927 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f0b10022020 con 0x7f0b1c106e50 2024-05-12T21:57:51.422 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:50.640+0000 7f0b16a00700 1 -- 172.21.15.28:0/3877319927 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f0b10011e00 con 0x7f0b1c106e50 2024-05-12T21:57:51.423 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:50.640+0000 7f0b24800700 1 -- 172.21.15.28:0/3877319927 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b1c063980 con 0x7f0b1c106e50 2024-05-12T21:57:51.423 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:50.643+0000 7f0b16a00700 1 -- 172.21.15.28:0/3877319927 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f0b10043dd0 con 0x7f0b1c106e50 2024-05-12T21:57:51.423 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:50.723+0000 7f0b24800700 1 -- 172.21.15.28:0/3877319927 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f0b1c195c80 con 0x7f0b1c106e50 2024-05-12T21:57:51.423 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:50.723+0000 7f0b16a00700 1 -- 172.21.15.28:0/3877319927 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f0b10032030 con 0x7f0b1c106e50 2024-05-12T21:57:51.423 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:50.725+0000 7f0b24800700 1 -- 172.21.15.28:0/3877319927 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0b1c106e50 msgr2=0x7f0b1c191ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T21:57:51.423 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:50.725+0000 7f0b24800700 1 --2- 172.21.15.28:0/3877319927 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0b1c106e50 0x7f0b1c191ff0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f0b10011b60 tx=0x7f0b10011c40 comp rx=0 tx=0).stop 2024-05-12T21:57:51.423 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:50.725+0000 7f0b24800700 1 -- 172.21.15.28:0/3877319927 shutdown_connections 2024-05-12T21:57:51.423 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:50.725+0000 7f0b24800700 1 --2- 172.21.15.28:0/3877319927 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0b1c106e50 0x7f0b1c191ff0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:57:51.423 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:50.725+0000 7f0b24800700 1 -- 172.21.15.28:0/3877319927 >> 172.21.15.28:0/3877319927 conn(0x7f0b1c102410 msgr2=0x7f0b1c1032a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T21:57:51.423 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:50.725+0000 7f0b24800700 1 -- 172.21.15.28:0/3877319927 shutdown_connections 2024-05-12T21:57:51.423 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:50.725+0000 7f0b24800700 1 -- 172.21.15.28:0/3877319927 wait complete. 2024-05-12T21:57:51.423 INFO:teuthology.orchestra.run.smithi028.stdout:mgr not available, waiting (2/15)... 2024-05-12T21:57:52.035 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 21:57:51 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T21:57:51.649+0000 7ff3179cd200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-12T21:57:52.535 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 21:57:52 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T21:57:52.223+0000 7ff3179cd200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-12T21:57:52.535 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 21:57:52 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T21:57:52.361+0000 7ff3179cd200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-12T21:57:53.035 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 21:57:52 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T21:57:52.634+0000 7ff3179cd200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-12T21:57:53.572 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 21:57:53 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T21:57:53.298+0000 7ff3179cd200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-12T21:57:53.572 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 21:57:53 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T21:57:53.425+0000 7ff3179cd200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-12T21:57:53.873 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 21:57:53 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T21:57:53.775+0000 7ff3179cd200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-12T21:57:54.123 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 21:57:53 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T21:57:53.962+0000 7ff3179cd200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-12T21:57:54.123 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 21:57:54 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T21:57:54.093+0000 7ff3179cd200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-12T21:57:54.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:54 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/1981846782' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-05-12T21:57:54.535 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 21:57:54 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T21:57:54.263+0000 7ff3179cd200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-12T21:57:54.916 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 21:57:54 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T21:57:54.637+0000 7ff3179cd200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-12T21:57:54.935 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-05-12T21:57:54.935 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout { 2024-05-12T21:57:54.936 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "fsid": "6b92e7b4-10aa-11ef-bc98-c7b262605968", 2024-05-12T21:57:54.936 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "health": { 2024-05-12T21:57:54.936 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-05-12T21:57:54.936 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-05-12T21:57:54.936 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-05-12T21:57:54.936 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-05-12T21:57:54.936 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-05-12T21:57:54.936 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-05-12T21:57:54.936 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 0 2024-05-12T21:57:54.936 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-05-12T21:57:54.936 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-05-12T21:57:54.936 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "a" 2024-05-12T21:57:54.936 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-05-12T21:57:54.937 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum_age": 10, 2024-05-12T21:57:54.937 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "monmap": { 2024-05-12T21:57:54.937 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-12T21:57:54.937 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-05-12T21:57:54.937 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-05-12T21:57:54.937 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-05-12T21:57:54.937 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-05-12T21:57:54.937 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-12T21:57:54.937 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-05-12T21:57:54.937 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-05-12T21:57:54.937 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-05-12T21:57:54.937 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-05-12T21:57:54.937 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-05-12T21:57:54.937 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-05-12T21:57:54.937 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-05-12T21:57:54.937 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-05-12T21:57:54.938 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-05-12T21:57:54.938 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-05-12T21:57:54.938 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-05-12T21:57:54.938 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-05-12T21:57:54.938 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-05-12T21:57:54.938 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-05-12T21:57:54.938 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-05-12T21:57:54.938 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-05-12T21:57:54.938 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-05-12T21:57:54.938 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-05-12T21:57:54.938 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-12T21:57:54.938 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-05-12T21:57:54.938 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-05-12T21:57:54.938 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-05-12T21:57:54.939 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-05-12T21:57:54.939 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "available": false, 2024-05-12T21:57:54.939 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-05-12T21:57:54.939 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "modules": [ 2024-05-12T21:57:54.939 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "iostat", 2024-05-12T21:57:54.939 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "nfs", 2024-05-12T21:57:54.939 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "restful" 2024-05-12T21:57:54.939 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-05-12T21:57:54.939 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-12T21:57:54.939 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-05-12T21:57:54.939 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-05-12T21:57:54.939 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-12T21:57:54.939 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "modified": "2024-05-12T21:57:37.277650+0000", 2024-05-12T21:57:54.939 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-12T21:57:54.939 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-05-12T21:57:54.940 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-05-12T21:57:54.940 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout } 2024-05-12T21:57:54.940 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:54.028+0000 7f6a52600700 1 Processor -- start 2024-05-12T21:57:54.940 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:54.028+0000 7f6a52600700 1 -- start start 2024-05-12T21:57:54.940 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:54.029+0000 7f6a52600700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a4c1089c0 0x7f6a4c10adb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T21:57:54.940 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:54.029+0000 7f6a52600700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6a4c063640 con 0x7f6a4c1089c0 2024-05-12T21:57:54.940 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:54.029+0000 7f6a4b400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a4c1089c0 0x7f6a4c10adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T21:57:54.940 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:54.029+0000 7f6a4b400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a4c1089c0 0x7f6a4c10adb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39682/0 (socket says 172.21.15.28:39682) 2024-05-12T21:57:54.940 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:54.029+0000 7f6a4b400700 1 -- 172.21.15.28:0/1489463433 learned_addr learned my addr 172.21.15.28:0/1489463433 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T21:57:54.940 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:54.030+0000 7f6a4b400700 1 -- 172.21.15.28:0/1489463433 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a4c10b2f0 con 0x7f6a4c1089c0 2024-05-12T21:57:54.940 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:54.030+0000 7f6a4b400700 1 --2- 172.21.15.28:0/1489463433 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a4c1089c0 0x7f6a4c10adb0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f6a3c0098a0 tx=0x7f6a3c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de3603e9dfa15410 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T21:57:54.940 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:54.030+0000 7f6a4a000700 1 -- 172.21.15.28:0/1489463433 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6a3c004160 con 0x7f6a4c1089c0 2024-05-12T21:57:54.940 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:54.030+0000 7f6a4a000700 1 -- 172.21.15.28:0/1489463433 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f6a3c004300 con 0x7f6a4c1089c0 2024-05-12T21:57:54.940 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:54.030+0000 7f6a4a000700 1 -- 172.21.15.28:0/1489463433 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6a3c011b50 con 0x7f6a4c1089c0 2024-05-12T21:57:54.940 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:54.031+0000 7f6a52600700 1 -- 172.21.15.28:0/1489463433 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a4c1089c0 msgr2=0x7f6a4c10adb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T21:57:54.941 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:54.031+0000 7f6a52600700 1 --2- 172.21.15.28:0/1489463433 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a4c1089c0 0x7f6a4c10adb0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f6a3c0098a0 tx=0x7f6a3c009bb0 comp rx=0 tx=0).stop 2024-05-12T21:57:54.941 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:54.031+0000 7f6a52600700 1 -- 172.21.15.28:0/1489463433 shutdown_connections 2024-05-12T21:57:54.941 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:54.031+0000 7f6a52600700 1 --2- 172.21.15.28:0/1489463433 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a4c1089c0 0x7f6a4c10adb0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:57:54.941 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:54.031+0000 7f6a52600700 1 -- 172.21.15.28:0/1489463433 >> 172.21.15.28:0/1489463433 conn(0x7f6a4c102410 msgr2=0x7f6a4c104830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T21:57:54.941 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:54.031+0000 7f6a52600700 1 -- 172.21.15.28:0/1489463433 shutdown_connections 2024-05-12T21:57:54.941 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:54.031+0000 7f6a52600700 1 -- 172.21.15.28:0/1489463433 wait complete. 2024-05-12T21:57:54.941 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:54.032+0000 7f6a52600700 1 Processor -- start 2024-05-12T21:57:54.941 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:54.032+0000 7f6a52600700 1 -- start start 2024-05-12T21:57:54.941 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:54.033+0000 7f6a52600700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a4c1089c0 0x7f6a4c19bee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T21:57:54.941 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:54.033+0000 7f6a52600700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6a4c10c320 con 0x7f6a4c1089c0 2024-05-12T21:57:54.941 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:54.033+0000 7f6a4b400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a4c1089c0 0x7f6a4c19bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T21:57:54.941 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:54.033+0000 7f6a4b400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a4c1089c0 0x7f6a4c19bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39686/0 (socket says 172.21.15.28:39686) 2024-05-12T21:57:54.941 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:54.033+0000 7f6a4b400700 1 -- 172.21.15.28:0/1981846782 learned_addr learned my addr 172.21.15.28:0/1981846782 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T21:57:54.941 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:54.033+0000 7f6a4b400700 1 -- 172.21.15.28:0/1981846782 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a4c19c420 con 0x7f6a4c1089c0 2024-05-12T21:57:54.941 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:54.034+0000 7f6a4b400700 1 --2- 172.21.15.28:0/1981846782 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a4c1089c0 0x7f6a4c19bee0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f6a3c011b60 tx=0x7f6a3c011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T21:57:54.942 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:54.034+0000 7f6a43e00700 1 -- 172.21.15.28:0/1981846782 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6a3c003b20 con 0x7f6a4c1089c0 2024-05-12T21:57:54.942 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:54.034+0000 7f6a43e00700 1 -- 172.21.15.28:0/1981846782 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f6a3c003cc0 con 0x7f6a4c1089c0 2024-05-12T21:57:54.942 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:54.034+0000 7f6a43e00700 1 -- 172.21.15.28:0/1981846782 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6a3c024840 con 0x7f6a4c1089c0 2024-05-12T21:57:54.942 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:54.034+0000 7f6a52600700 1 -- 172.21.15.28:0/1981846782 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a4c19c6b0 con 0x7f6a4c1089c0 2024-05-12T21:57:54.942 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:54.034+0000 7f6a52600700 1 -- 172.21.15.28:0/1981846782 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a4c19f3a0 con 0x7f6a4c1089c0 2024-05-12T21:57:54.942 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:54.035+0000 7f6a43e00700 1 -- 172.21.15.28:0/1981846782 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f6a3c022020 con 0x7f6a4c1089c0 2024-05-12T21:57:54.942 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:54.035+0000 7f6a43e00700 1 -- 172.21.15.28:0/1981846782 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f6a3c011e00 con 0x7f6a4c1089c0 2024-05-12T21:57:54.942 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:54.035+0000 7f6a52600700 1 -- 172.21.15.28:0/1981846782 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a4c063980 con 0x7f6a4c1089c0 2024-05-12T21:57:54.942 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:54.038+0000 7f6a43e00700 1 -- 172.21.15.28:0/1981846782 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f6a3c043dd0 con 0x7f6a4c1089c0 2024-05-12T21:57:54.942 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:54.118+0000 7f6a52600700 1 -- 172.21.15.28:0/1981846782 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f6a4c19f570 con 0x7f6a4c1089c0 2024-05-12T21:57:54.942 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:54.119+0000 7f6a43e00700 1 -- 172.21.15.28:0/1981846782 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1240 (secure 0 0 0) 0x7f6a3c032030 con 0x7f6a4c1089c0 2024-05-12T21:57:54.942 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:54.121+0000 7f6a52600700 1 -- 172.21.15.28:0/1981846782 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a4c1089c0 msgr2=0x7f6a4c19bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T21:57:54.942 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:54.121+0000 7f6a52600700 1 --2- 172.21.15.28:0/1981846782 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a4c1089c0 0x7f6a4c19bee0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f6a3c011b60 tx=0x7f6a3c011c40 comp rx=0 tx=0).stop 2024-05-12T21:57:54.942 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:54.121+0000 7f6a52600700 1 -- 172.21.15.28:0/1981846782 shutdown_connections 2024-05-12T21:57:54.943 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:54.121+0000 7f6a52600700 1 --2- 172.21.15.28:0/1981846782 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a4c1089c0 0x7f6a4c19bee0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:57:54.943 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:54.121+0000 7f6a52600700 1 -- 172.21.15.28:0/1981846782 >> 172.21.15.28:0/1981846782 conn(0x7f6a4c102410 msgr2=0x7f6a4c102f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T21:57:54.943 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:54.121+0000 7f6a52600700 1 -- 172.21.15.28:0/1981846782 shutdown_connections 2024-05-12T21:57:54.943 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:54.121+0000 7f6a52600700 1 -- 172.21.15.28:0/1981846782 wait complete. 2024-05-12T21:57:54.943 INFO:teuthology.orchestra.run.smithi028.stdout:mgr not available, waiting (3/15)... 2024-05-12T21:57:55.743 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 21:57:55 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T21:57:55.448+0000 7ff3179cd200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-12T21:57:56.035 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 21:57:55 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T21:57:55.741+0000 7ff3179cd200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-12T21:57:56.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:55 smithi028 ceph-mon[29847]: Activating manager daemon x 2024-05-12T21:57:56.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:55 smithi028 ceph-mon[29847]: mgrmap e2: x(active, starting, since 0.0111136s) 2024-05-12T21:57:56.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:55 smithi028 ceph-mon[29847]: from='mgr.14100 172.21.15.28:0/3214405358' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-12T21:57:56.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:55 smithi028 ceph-mon[29847]: from='mgr.14100 172.21.15.28:0/3214405358' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-12T21:57:56.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:55 smithi028 ceph-mon[29847]: from='mgr.14100 172.21.15.28:0/3214405358' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-12T21:57:56.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:55 smithi028 ceph-mon[29847]: from='mgr.14100 172.21.15.28:0/3214405358' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-12T21:57:56.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:55 smithi028 ceph-mon[29847]: from='mgr.14100 172.21.15.28:0/3214405358' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-12T21:57:56.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:55 smithi028 ceph-mon[29847]: Manager daemon x is now available 2024-05-12T21:57:56.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:55 smithi028 ceph-mon[29847]: from='mgr.14100 172.21.15.28:0/3214405358' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-05-12T21:57:56.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:55 smithi028 ceph-mon[29847]: from='mgr.14100 172.21.15.28:0/3214405358' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-05-12T21:57:56.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:55 smithi028 ceph-mon[29847]: from='mgr.14100 172.21.15.28:0/3214405358' entity='mgr.x' 2024-05-12T21:57:56.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:55 smithi028 ceph-mon[29847]: from='mgr.14100 172.21.15.28:0/3214405358' entity='mgr.x' 2024-05-12T21:57:57.097 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:56 smithi028 ceph-mon[29847]: from='mgr.14100 172.21.15.28:0/3214405358' entity='mgr.x' 2024-05-12T21:57:57.097 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:56 smithi028 ceph-mon[29847]: mgrmap e3: x(active, since 1.02284s) 2024-05-12T21:57:58.600 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-05-12T21:57:58.600 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout { 2024-05-12T21:57:58.600 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "fsid": "6b92e7b4-10aa-11ef-bc98-c7b262605968", 2024-05-12T21:57:58.602 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "health": { 2024-05-12T21:57:58.602 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-05-12T21:57:58.602 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-05-12T21:57:58.602 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-05-12T21:57:58.602 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-05-12T21:57:58.602 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-05-12T21:57:58.602 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-05-12T21:57:58.602 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 0 2024-05-12T21:57:58.602 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-05-12T21:57:58.602 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-05-12T21:57:58.602 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "a" 2024-05-12T21:57:58.602 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-05-12T21:57:58.602 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum_age": 14, 2024-05-12T21:57:58.603 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "monmap": { 2024-05-12T21:57:58.603 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-12T21:57:58.604 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-05-12T21:57:58.604 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-05-12T21:57:58.604 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-05-12T21:57:58.604 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-05-12T21:57:58.604 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-12T21:57:58.604 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-05-12T21:57:58.604 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-05-12T21:57:58.604 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-05-12T21:57:58.604 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-05-12T21:57:58.604 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-05-12T21:57:58.608 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-05-12T21:57:58.608 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-05-12T21:57:58.608 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-05-12T21:57:58.608 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-05-12T21:57:58.608 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-05-12T21:57:58.608 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-05-12T21:57:58.608 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-05-12T21:57:58.608 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-05-12T21:57:58.608 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-05-12T21:57:58.608 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-05-12T21:57:58.609 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-05-12T21:57:58.609 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-05-12T21:57:58.609 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-05-12T21:57:58.609 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-12T21:57:58.609 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-05-12T21:57:58.609 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-05-12T21:57:58.609 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-05-12T21:57:58.609 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-05-12T21:57:58.609 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "available": true, 2024-05-12T21:57:58.609 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-05-12T21:57:58.609 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "modules": [ 2024-05-12T21:57:58.609 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "iostat", 2024-05-12T21:57:58.609 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "nfs", 2024-05-12T21:57:58.609 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "restful" 2024-05-12T21:57:58.609 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-05-12T21:57:58.610 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-12T21:57:58.610 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-05-12T21:57:58.610 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-05-12T21:57:58.610 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-12T21:57:58.610 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "modified": "2024-05-12T21:57:37.277650+0000", 2024-05-12T21:57:58.616 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-12T21:57:58.616 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-05-12T21:57:58.616 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-05-12T21:57:58.616 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout } 2024-05-12T21:57:58.616 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:57.546+0000 7fb414000700 1 Processor -- start 2024-05-12T21:57:58.616 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:57.546+0000 7fb414000700 1 -- start start 2024-05-12T21:57:58.616 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:57.547+0000 7fb414000700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb40c107f70 0x7fb40c108390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T21:57:58.616 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:57.547+0000 7fb414000700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb40c108960 con 0x7fb40c107f70 2024-05-12T21:57:58.616 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:57.547+0000 7fb411600700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb40c107f70 0x7fb40c108390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T21:57:58.616 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:57.548+0000 7fb411600700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb40c107f70 0x7fb40c108390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47528/0 (socket says 172.21.15.28:47528) 2024-05-12T21:57:58.616 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:57.548+0000 7fb411600700 1 -- 172.21.15.28:0/767797558 learned_addr learned my addr 172.21.15.28:0/767797558 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T21:57:58.616 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:57.548+0000 7fb411600700 1 -- 172.21.15.28:0/767797558 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb40c109170 con 0x7fb40c107f70 2024-05-12T21:57:58.616 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:57.548+0000 7fb411600700 1 --2- 172.21.15.28:0/767797558 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb40c107f70 0x7fb40c108390 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fb400009e80 tx=0x7fb40000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7e7d5fd8b8801bb server_cookie=0 in_seq=0 out_seq=0 2024-05-12T21:57:58.617 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:57.549+0000 7fb40be00700 1 -- 172.21.15.28:0/767797558 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb400017070 con 0x7fb40c107f70 2024-05-12T21:57:58.617 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:57.549+0000 7fb40be00700 1 -- 172.21.15.28:0/767797558 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fb40000b8c0 con 0x7fb40c107f70 2024-05-12T21:57:58.617 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:57.549+0000 7fb40be00700 1 -- 172.21.15.28:0/767797558 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb4000045c0 con 0x7fb40c107f70 2024-05-12T21:57:58.617 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:57.549+0000 7fb414000700 1 -- 172.21.15.28:0/767797558 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb40c107f70 msgr2=0x7fb40c108390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T21:57:58.617 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:57.549+0000 7fb414000700 1 --2- 172.21.15.28:0/767797558 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb40c107f70 0x7fb40c108390 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fb400009e80 tx=0x7fb40000b330 comp rx=0 tx=0).stop 2024-05-12T21:57:58.617 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:57.549+0000 7fb414000700 1 -- 172.21.15.28:0/767797558 shutdown_connections 2024-05-12T21:57:58.617 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:57.549+0000 7fb414000700 1 --2- 172.21.15.28:0/767797558 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb40c107f70 0x7fb40c108390 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:57:58.617 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:57.549+0000 7fb414000700 1 -- 172.21.15.28:0/767797558 >> 172.21.15.28:0/767797558 conn(0x7fb40c103530 msgr2=0x7fb40c105950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T21:57:58.617 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:57.550+0000 7fb414000700 1 -- 172.21.15.28:0/767797558 shutdown_connections 2024-05-12T21:57:58.617 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:57.550+0000 7fb414000700 1 -- 172.21.15.28:0/767797558 wait complete. 2024-05-12T21:57:58.617 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:57.551+0000 7fb414000700 1 Processor -- start 2024-05-12T21:57:58.617 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:57.551+0000 7fb414000700 1 -- start start 2024-05-12T21:57:58.617 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:57.551+0000 7fb414000700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb40c107f70 0x7fb40c19e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T21:57:58.617 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:57.551+0000 7fb414000700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb40c10c520 con 0x7fb40c107f70 2024-05-12T21:57:58.618 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:57.551+0000 7fb411600700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb40c107f70 0x7fb40c19e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T21:57:58.618 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:57.551+0000 7fb411600700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb40c107f70 0x7fb40c19e1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47534/0 (socket says 172.21.15.28:47534) 2024-05-12T21:57:58.618 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:57.551+0000 7fb411600700 1 -- 172.21.15.28:0/3992322178 learned_addr learned my addr 172.21.15.28:0/3992322178 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T21:57:58.618 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:57.552+0000 7fb411600700 1 -- 172.21.15.28:0/3992322178 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb40c19e700 con 0x7fb40c107f70 2024-05-12T21:57:58.618 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:57.552+0000 7fb411600700 1 --2- 172.21.15.28:0/3992322178 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb40c107f70 0x7fb40c19e1c0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fb400004210 tx=0x7fb40000bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T21:57:58.618 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:57.552+0000 7fb40a000700 1 -- 172.21.15.28:0/3992322178 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb400020020 con 0x7fb40c107f70 2024-05-12T21:57:58.618 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:57.552+0000 7fb40a000700 1 -- 172.21.15.28:0/3992322178 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fb400003b50 con 0x7fb40c107f70 2024-05-12T21:57:58.618 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:57.552+0000 7fb40a000700 1 -- 172.21.15.28:0/3992322178 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb400017040 con 0x7fb40c107f70 2024-05-12T21:57:58.618 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:57.553+0000 7fb414000700 1 -- 172.21.15.28:0/3992322178 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb40c19e990 con 0x7fb40c107f70 2024-05-12T21:57:58.618 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:57.553+0000 7fb414000700 1 -- 172.21.15.28:0/3992322178 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb40c1a1680 con 0x7fb40c107f70 2024-05-12T21:57:58.618 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:57.554+0000 7fb414000700 1 -- 172.21.15.28:0/3992322178 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb40c063980 con 0x7fb40c107f70 2024-05-12T21:57:58.618 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:57.554+0000 7fb40a000700 1 -- 172.21.15.28:0/3992322178 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 3) ==== 46170+0+0 (secure 0 0 0) 0x7fb400023020 con 0x7fb40c107f70 2024-05-12T21:57:58.618 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:57.554+0000 7fb40a000700 1 --2- 172.21.15.28:0/3992322178 >> [v2:172.21.15.28:6800/3783742728,v1:172.21.15.28:6801/3783742728] conn(0x7fb3f4038f30 0x7fb3f403b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T21:57:58.618 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:57.554+0000 7fb40a000700 1 -- 172.21.15.28:0/3992322178 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fb40000f050 con 0x7fb40c107f70 2024-05-12T21:57:58.619 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:57.555+0000 7fb410c00700 1 --2- 172.21.15.28:0/3992322178 >> [v2:172.21.15.28:6800/3783742728,v1:172.21.15.28:6801/3783742728] conn(0x7fb3f4038f30 0x7fb3f403b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T21:57:58.619 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:57.555+0000 7fb410c00700 1 --2- 172.21.15.28:0/3992322178 >> [v2:172.21.15.28:6800/3783742728,v1:172.21.15.28:6801/3783742728] conn(0x7fb3f4038f30 0x7fb3f403b3f0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fb3f8006fd0 tx=0x7fb3f8006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T21:57:58.619 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:57.561+0000 7fb40a000700 1 -- 172.21.15.28:0/3992322178 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fb40002a020 con 0x7fb40c107f70 2024-05-12T21:57:58.619 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:57.808+0000 7fb40a000700 1 -- 172.21.15.28:0/3992322178 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mgrmap(e 4) ==== 46276+0+0 (secure 0 0 0) 0x7fb400012950 con 0x7fb40c107f70 2024-05-12T21:57:58.619 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:57.883+0000 7fb414000700 1 -- 172.21.15.28:0/3992322178 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fb40c1a1850 con 0x7fb40c107f70 2024-05-12T21:57:58.619 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:57.883+0000 7fb40a000700 1 -- 172.21.15.28:0/3992322178 <== mon.0 v2:172.21.15.28:3300/0 8 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7fb40002a200 con 0x7fb40c107f70 2024-05-12T21:57:58.619 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:57.886+0000 7fb414000700 1 -- 172.21.15.28:0/3992322178 >> [v2:172.21.15.28:6800/3783742728,v1:172.21.15.28:6801/3783742728] conn(0x7fb3f4038f30 msgr2=0x7fb3f403b3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T21:57:58.619 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:57.886+0000 7fb414000700 1 --2- 172.21.15.28:0/3992322178 >> [v2:172.21.15.28:6800/3783742728,v1:172.21.15.28:6801/3783742728] conn(0x7fb3f4038f30 0x7fb3f403b3f0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fb3f8006fd0 tx=0x7fb3f8006e40 comp rx=0 tx=0).stop 2024-05-12T21:57:58.619 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:57.886+0000 7fb414000700 1 -- 172.21.15.28:0/3992322178 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb40c107f70 msgr2=0x7fb40c19e1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T21:57:58.619 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:57.886+0000 7fb414000700 1 --2- 172.21.15.28:0/3992322178 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb40c107f70 0x7fb40c19e1c0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fb400004210 tx=0x7fb40000bd50 comp rx=0 tx=0).stop 2024-05-12T21:57:58.619 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:57.886+0000 7fb414000700 1 -- 172.21.15.28:0/3992322178 shutdown_connections 2024-05-12T21:57:58.619 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:57.886+0000 7fb414000700 1 --2- 172.21.15.28:0/3992322178 >> [v2:172.21.15.28:6800/3783742728,v1:172.21.15.28:6801/3783742728] conn(0x7fb3f4038f30 0x7fb3f403b3f0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:57:58.620 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:57.886+0000 7fb414000700 1 --2- 172.21.15.28:0/3992322178 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb40c107f70 0x7fb40c19e1c0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:57:58.620 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:57.886+0000 7fb414000700 1 -- 172.21.15.28:0/3992322178 >> 172.21.15.28:0/3992322178 conn(0x7fb40c103530 msgr2=0x7fb40c1040a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T21:57:58.620 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:57.887+0000 7fb414000700 1 -- 172.21.15.28:0/3992322178 shutdown_connections 2024-05-12T21:57:58.620 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:57.887+0000 7fb414000700 1 -- 172.21.15.28:0/3992322178 wait complete. 2024-05-12T21:57:58.620 INFO:teuthology.orchestra.run.smithi028.stdout:mgr is available 2024-05-12T21:57:58.924 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:58 smithi028 ceph-mon[29847]: mgrmap e4: x(active, since 2s) 2024-05-12T21:57:58.924 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:58 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/3992322178' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-05-12T21:58:00.203 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:57:59 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/2046033275' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-05-12T21:58:00.216 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-05-12T21:58:00.217 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout [global] 2024-05-12T21:58:00.217 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout fsid = 6b92e7b4-10aa-11ef-bc98-c7b262605968 2024-05-12T21:58:00.217 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-05-12T21:58:00.217 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-05-12T21:58:00.217 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-05-12T21:58:00.217 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-05-12T21:58:00.217 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-05-12T21:58:00.217 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout [mgr] 2024-05-12T21:58:00.217 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-05-12T21:58:00.217 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-05-12T21:58:00.217 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout [osd] 2024-05-12T21:58:00.217 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-05-12T21:58:00.217 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-05-12T21:58:00.217 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-05-12T21:58:00.217 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-05-12T21:58:00.217 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-05-12T21:58:00.218 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:59.211+0000 7fac07200700 1 Processor -- start 2024-05-12T21:58:00.218 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:59.212+0000 7fac07200700 1 -- start start 2024-05-12T21:58:00.218 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:59.212+0000 7fac07200700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fac0807e7f0 0x7fac0807ec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T21:58:00.218 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:59.212+0000 7fac07200700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fac0807f1e0 con 0x7fac0807e7f0 2024-05-12T21:58:00.218 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:59.213+0000 7fac05e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fac0807e7f0 0x7fac0807ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T21:58:00.218 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:59.213+0000 7fac05e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fac0807e7f0 0x7fac0807ec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47550/0 (socket says 172.21.15.28:47550) 2024-05-12T21:58:00.218 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:59.213+0000 7fac05e00700 1 -- 172.21.15.28:0/1679481560 learned_addr learned my addr 172.21.15.28:0/1679481560 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T21:58:00.218 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:59.213+0000 7fac05e00700 1 -- 172.21.15.28:0/1679481560 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac0807fa40 con 0x7fac0807e7f0 2024-05-12T21:58:00.218 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:59.214+0000 7fac05e00700 1 --2- 172.21.15.28:0/1679481560 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fac0807e7f0 0x7fac0807ec10 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fabf80098a0 tx=0x7fabf8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91536aa331b09cba server_cookie=0 in_seq=0 out_seq=0 2024-05-12T21:58:00.218 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:59.214+0000 7fac05400700 1 -- 172.21.15.28:0/1679481560 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fabf8004160 con 0x7fac0807e7f0 2024-05-12T21:58:00.218 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:59.214+0000 7fac05400700 1 -- 172.21.15.28:0/1679481560 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fabf8004300 con 0x7fac0807e7f0 2024-05-12T21:58:00.218 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:59.214+0000 7fac05400700 1 -- 172.21.15.28:0/1679481560 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fabf8011b50 con 0x7fac0807e7f0 2024-05-12T21:58:00.218 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:59.215+0000 7fac07200700 1 -- 172.21.15.28:0/1679481560 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fac0807e7f0 msgr2=0x7fac0807ec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T21:58:00.218 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:59.215+0000 7fac07200700 1 --2- 172.21.15.28:0/1679481560 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fac0807e7f0 0x7fac0807ec10 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fabf80098a0 tx=0x7fabf8009bb0 comp rx=0 tx=0).stop 2024-05-12T21:58:00.218 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:59.215+0000 7fac07200700 1 -- 172.21.15.28:0/1679481560 shutdown_connections 2024-05-12T21:58:00.219 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:59.215+0000 7fac07200700 1 --2- 172.21.15.28:0/1679481560 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fac0807e7f0 0x7fac0807ec10 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:58:00.219 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:59.215+0000 7fac07200700 1 -- 172.21.15.28:0/1679481560 >> 172.21.15.28:0/1679481560 conn(0x7fac0807d3d0 msgr2=0x7fac0807d7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T21:58:00.219 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:59.215+0000 7fac07200700 1 -- 172.21.15.28:0/1679481560 shutdown_connections 2024-05-12T21:58:00.219 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:59.215+0000 7fac07200700 1 -- 172.21.15.28:0/1679481560 wait complete. 2024-05-12T21:58:00.219 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:59.216+0000 7fac07200700 1 Processor -- start 2024-05-12T21:58:00.219 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:59.216+0000 7fac07200700 1 -- start start 2024-05-12T21:58:00.219 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:59.216+0000 7fac07200700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fac0807e7f0 0x7fac081a23d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T21:58:00.219 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:59.216+0000 7fac07200700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fac08080e00 con 0x7fac0807e7f0 2024-05-12T21:58:00.219 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:59.217+0000 7fac05e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fac0807e7f0 0x7fac081a23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T21:58:00.219 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:59.217+0000 7fac05e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fac0807e7f0 0x7fac081a23d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47558/0 (socket says 172.21.15.28:47558) 2024-05-12T21:58:00.219 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:59.217+0000 7fac05e00700 1 -- 172.21.15.28:0/2046033275 learned_addr learned my addr 172.21.15.28:0/2046033275 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T21:58:00.219 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:59.217+0000 7fac05e00700 1 -- 172.21.15.28:0/2046033275 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac081a2910 con 0x7fac0807e7f0 2024-05-12T21:58:00.219 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:59.217+0000 7fac05e00700 1 --2- 172.21.15.28:0/2046033275 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fac0807e7f0 0x7fac081a23d0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fabf8011b60 tx=0x7fabf8011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T21:58:00.219 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:59.218+0000 7fabfea00700 1 -- 172.21.15.28:0/2046033275 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fabf8003970 con 0x7fac0807e7f0 2024-05-12T21:58:00.220 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:59.218+0000 7fabfea00700 1 -- 172.21.15.28:0/2046033275 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fabf8003b10 con 0x7fac0807e7f0 2024-05-12T21:58:00.220 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:59.218+0000 7fabfea00700 1 -- 172.21.15.28:0/2046033275 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fabf8024610 con 0x7fac0807e7f0 2024-05-12T21:58:00.220 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:59.218+0000 7fac07200700 1 -- 172.21.15.28:0/2046033275 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac081a2ba0 con 0x7fac0807e7f0 2024-05-12T21:58:00.220 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:59.218+0000 7fac07200700 1 -- 172.21.15.28:0/2046033275 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac081a5890 con 0x7fac0807e7f0 2024-05-12T21:58:00.220 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:59.219+0000 7fabfea00700 1 -- 172.21.15.28:0/2046033275 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 4) ==== 46276+0+0 (secure 0 0 0) 0x7fabf801a9f0 con 0x7fac0807e7f0 2024-05-12T21:58:00.220 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:59.219+0000 7fac07200700 1 -- 172.21.15.28:0/2046033275 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac080694a0 con 0x7fac0807e7f0 2024-05-12T21:58:00.220 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:59.220+0000 7fabfea00700 1 --2- 172.21.15.28:0/2046033275 >> [v2:172.21.15.28:6800/3783742728,v1:172.21.15.28:6801/3783742728] conn(0x7fabec039050 0x7fabec03b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T21:58:00.220 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:59.220+0000 7fabfea00700 1 -- 172.21.15.28:0/2046033275 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fabf8011e00 con 0x7fac0807e7f0 2024-05-12T21:58:00.220 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:59.220+0000 7fabffe00700 1 --2- 172.21.15.28:0/2046033275 >> [v2:172.21.15.28:6800/3783742728,v1:172.21.15.28:6801/3783742728] conn(0x7fabec039050 0x7fabec03b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T21:58:00.220 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:59.221+0000 7fabffe00700 1 --2- 172.21.15.28:0/2046033275 >> [v2:172.21.15.28:6800/3783742728,v1:172.21.15.28:6801/3783742728] conn(0x7fabec039050 0x7fabec03b510 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fabf0006fd0 tx=0x7fabf0006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T21:58:00.220 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:59.226+0000 7fabfea00700 1 -- 172.21.15.28:0/2046033275 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fabf800b3e0 con 0x7fac0807e7f0 2024-05-12T21:58:00.220 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:59.451+0000 7fac07200700 1 -- 172.21.15.28:0/2046033275 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7fac080008d0 con 0x7fac0807e7f0 2024-05-12T21:58:00.220 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:59.452+0000 7fabfea00700 1 -- 172.21.15.28:0/2046033275 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) ==== 70+0+373 (secure 0 0 0) 0x7fabf804cbb0 con 0x7fac0807e7f0 2024-05-12T21:58:00.220 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:59.455+0000 7fac07200700 1 -- 172.21.15.28:0/2046033275 >> [v2:172.21.15.28:6800/3783742728,v1:172.21.15.28:6801/3783742728] conn(0x7fabec039050 msgr2=0x7fabec03b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T21:58:00.220 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:59.455+0000 7fac07200700 1 --2- 172.21.15.28:0/2046033275 >> [v2:172.21.15.28:6800/3783742728,v1:172.21.15.28:6801/3783742728] conn(0x7fabec039050 0x7fabec03b510 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fabf0006fd0 tx=0x7fabf0006e40 comp rx=0 tx=0).stop 2024-05-12T21:58:00.221 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:59.455+0000 7fac07200700 1 -- 172.21.15.28:0/2046033275 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fac0807e7f0 msgr2=0x7fac081a23d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T21:58:00.221 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:59.455+0000 7fac07200700 1 --2- 172.21.15.28:0/2046033275 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fac0807e7f0 0x7fac081a23d0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fabf8011b60 tx=0x7fabf8011c40 comp rx=0 tx=0).stop 2024-05-12T21:58:00.221 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:59.455+0000 7fac07200700 1 -- 172.21.15.28:0/2046033275 shutdown_connections 2024-05-12T21:58:00.221 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:59.455+0000 7fac07200700 1 --2- 172.21.15.28:0/2046033275 >> [v2:172.21.15.28:6800/3783742728,v1:172.21.15.28:6801/3783742728] conn(0x7fabec039050 0x7fabec03b510 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:58:00.221 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:59.455+0000 7fac07200700 1 --2- 172.21.15.28:0/2046033275 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fac0807e7f0 0x7fac081a23d0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:58:00.221 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:59.455+0000 7fac07200700 1 -- 172.21.15.28:0/2046033275 >> 172.21.15.28:0/2046033275 conn(0x7fac0807d3d0 msgr2=0x7fac08198fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T21:58:00.221 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:59.456+0000 7fac07200700 1 -- 172.21.15.28:0/2046033275 shutdown_connections 2024-05-12T21:58:00.221 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:57:59.456+0000 7fac07200700 1 -- 172.21.15.28:0/2046033275 wait complete. 2024-05-12T21:58:00.221 INFO:teuthology.orchestra.run.smithi028.stdout:Enabling cephadm module... 2024-05-12T21:58:01.516 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:58:01 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/571768510' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-05-12T21:58:01.516 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 21:58:01 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: ignoring --setuser ceph since I am not root 2024-05-12T21:58:01.516 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 21:58:01 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: ignoring --setgroup ceph since I am not root 2024-05-12T21:58:01.516 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 21:58:01 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T21:58:01.514+0000 7f1e8436f200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-12T21:58:01.949 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:00.838+0000 7f19a0200700 1 Processor -- start 2024-05-12T21:58:01.950 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:00.839+0000 7f19a0200700 1 -- start start 2024-05-12T21:58:01.950 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:00.839+0000 7f19a0200700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f199810a1b0 0x7f199810a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T21:58:01.950 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:00.839+0000 7f19a0200700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f199810aba0 con 0x7f199810a1b0 2024-05-12T21:58:01.950 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:00.840+0000 7f199d800700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f199810a1b0 0x7f199810a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T21:58:01.950 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:00.840+0000 7f199d800700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f199810a1b0 0x7f199810a5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47560/0 (socket says 172.21.15.28:47560) 2024-05-12T21:58:01.950 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:00.840+0000 7f199d800700 1 -- 172.21.15.28:0/1296499486 learned_addr learned my addr 172.21.15.28:0/1296499486 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T21:58:01.950 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:00.840+0000 7f199d800700 1 -- 172.21.15.28:0/1296499486 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f199810b360 con 0x7f199810a1b0 2024-05-12T21:58:01.950 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:00.841+0000 7f199d800700 1 --2- 172.21.15.28:0/1296499486 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f199810a1b0 0x7f199810a5d0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f198c0098a0 tx=0x7f198c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d786cae046caf3f6 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T21:58:01.950 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:00.841+0000 7f1997e00700 1 -- 172.21.15.28:0/1296499486 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f198c004160 con 0x7f199810a1b0 2024-05-12T21:58:01.950 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:00.841+0000 7f1997e00700 1 -- 172.21.15.28:0/1296499486 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f198c004300 con 0x7f199810a1b0 2024-05-12T21:58:01.950 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:00.841+0000 7f1997e00700 1 -- 172.21.15.28:0/1296499486 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f198c011b50 con 0x7f199810a1b0 2024-05-12T21:58:01.950 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:00.842+0000 7f19a0200700 1 -- 172.21.15.28:0/1296499486 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f199810a1b0 msgr2=0x7f199810a5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T21:58:01.950 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:00.842+0000 7f19a0200700 1 --2- 172.21.15.28:0/1296499486 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f199810a1b0 0x7f199810a5d0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f198c0098a0 tx=0x7f198c009bb0 comp rx=0 tx=0).stop 2024-05-12T21:58:01.951 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:00.842+0000 7f19a0200700 1 -- 172.21.15.28:0/1296499486 shutdown_connections 2024-05-12T21:58:01.951 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:00.842+0000 7f19a0200700 1 --2- 172.21.15.28:0/1296499486 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f199810a1b0 0x7f199810a5d0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:58:01.951 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:00.842+0000 7f19a0200700 1 -- 172.21.15.28:0/1296499486 >> 172.21.15.28:0/1296499486 conn(0x7f199807d650 msgr2=0x7f199807da80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T21:58:01.951 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:00.842+0000 7f19a0200700 1 -- 172.21.15.28:0/1296499486 shutdown_connections 2024-05-12T21:58:01.951 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:00.842+0000 7f19a0200700 1 -- 172.21.15.28:0/1296499486 wait complete. 2024-05-12T21:58:01.951 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:00.843+0000 7f19a0200700 1 Processor -- start 2024-05-12T21:58:01.951 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:00.843+0000 7f19a0200700 1 -- start start 2024-05-12T21:58:01.951 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:00.844+0000 7f19a0200700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f199810a1b0 0x7f199819e150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T21:58:01.951 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:00.844+0000 7f19a0200700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f199810e710 con 0x7f199810a1b0 2024-05-12T21:58:01.951 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:00.844+0000 7f199d800700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f199810a1b0 0x7f199819e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T21:58:01.951 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:00.844+0000 7f199d800700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f199810a1b0 0x7f199819e150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47576/0 (socket says 172.21.15.28:47576) 2024-05-12T21:58:01.951 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:00.844+0000 7f199d800700 1 -- 172.21.15.28:0/571768510 learned_addr learned my addr 172.21.15.28:0/571768510 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T21:58:01.951 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:00.844+0000 7f199d800700 1 -- 172.21.15.28:0/571768510 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f199819e690 con 0x7f199810a1b0 2024-05-12T21:58:01.951 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:00.845+0000 7f199d800700 1 --2- 172.21.15.28:0/571768510 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f199810a1b0 0x7f199819e150 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f198c000c00 tx=0x7f198c011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T21:58:01.951 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:00.845+0000 7f1996000700 1 -- 172.21.15.28:0/571768510 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f198c003b20 con 0x7f199810a1b0 2024-05-12T21:58:01.952 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:00.845+0000 7f1996000700 1 -- 172.21.15.28:0/571768510 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f198c003cc0 con 0x7f199810a1b0 2024-05-12T21:58:01.952 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:00.845+0000 7f1996000700 1 -- 172.21.15.28:0/571768510 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f198c024910 con 0x7f199810a1b0 2024-05-12T21:58:01.952 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:00.845+0000 7f19a0200700 1 -- 172.21.15.28:0/571768510 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f199819e920 con 0x7f199810a1b0 2024-05-12T21:58:01.952 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:00.845+0000 7f19a0200700 1 -- 172.21.15.28:0/571768510 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19981a1610 con 0x7f199810a1b0 2024-05-12T21:58:01.952 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:00.847+0000 7f19a0200700 1 -- 172.21.15.28:0/571768510 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1998063980 con 0x7f199810a1b0 2024-05-12T21:58:01.952 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:00.847+0000 7f1996000700 1 -- 172.21.15.28:0/571768510 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 4) ==== 46276+0+0 (secure 0 0 0) 0x7f198c022020 con 0x7f199810a1b0 2024-05-12T21:58:01.952 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:00.847+0000 7f1996000700 1 --2- 172.21.15.28:0/571768510 >> [v2:172.21.15.28:6800/3783742728,v1:172.21.15.28:6801/3783742728] conn(0x7f1980039050 0x7f198003b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T21:58:01.952 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:00.847+0000 7f1996000700 1 -- 172.21.15.28:0/571768510 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f198c011e00 con 0x7f199810a1b0 2024-05-12T21:58:01.952 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:00.848+0000 7f199ce00700 1 --2- 172.21.15.28:0/571768510 >> [v2:172.21.15.28:6800/3783742728,v1:172.21.15.28:6801/3783742728] conn(0x7f1980039050 0x7f198003b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T21:58:01.952 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:00.849+0000 7f199ce00700 1 --2- 172.21.15.28:0/571768510 >> [v2:172.21.15.28:6800/3783742728,v1:172.21.15.28:6801/3783742728] conn(0x7f1980039050 0x7f198003b510 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f1984006fd0 tx=0x7f1984006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T21:58:01.952 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:00.853+0000 7f1996000700 1 -- 172.21.15.28:0/571768510 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f198c01ad80 con 0x7f199810a1b0 2024-05-12T21:58:01.952 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:01.148+0000 7f19a0200700 1 -- 172.21.15.28:0/571768510 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) -- 0x7f19981a17e0 con 0x7f199810a1b0 2024-05-12T21:58:01.952 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:01.202+0000 7f1996000700 1 -- 172.21.15.28:0/571768510 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) ==== 86+0+0 (secure 0 0 0) 0x7f19981a17e0 con 0x7f199810a1b0 2024-05-12T21:58:01.952 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:01.206+0000 7f1996000700 1 -- 172.21.15.28:0/571768510 <== mon.0 v2:172.21.15.28:3300/0 8 ==== mgrmap(e 5) ==== 46287+0+0 (secure 0 0 0) 0x7f198c01a950 con 0x7f199810a1b0 2024-05-12T21:58:01.952 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:01.207+0000 7f19a0200700 1 -- 172.21.15.28:0/571768510 >> [v2:172.21.15.28:6800/3783742728,v1:172.21.15.28:6801/3783742728] conn(0x7f1980039050 msgr2=0x7f198003b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T21:58:01.953 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:01.207+0000 7f19a0200700 1 --2- 172.21.15.28:0/571768510 >> [v2:172.21.15.28:6800/3783742728,v1:172.21.15.28:6801/3783742728] conn(0x7f1980039050 0x7f198003b510 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f1984006fd0 tx=0x7f1984006e40 comp rx=0 tx=0).stop 2024-05-12T21:58:01.953 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:01.207+0000 7f19a0200700 1 -- 172.21.15.28:0/571768510 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f199810a1b0 msgr2=0x7f199819e150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T21:58:01.953 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:01.207+0000 7f19a0200700 1 --2- 172.21.15.28:0/571768510 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f199810a1b0 0x7f199819e150 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f198c000c00 tx=0x7f198c011c10 comp rx=0 tx=0).stop 2024-05-12T21:58:01.953 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:01.208+0000 7f19a0200700 1 -- 172.21.15.28:0/571768510 shutdown_connections 2024-05-12T21:58:01.953 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:01.208+0000 7f19a0200700 1 --2- 172.21.15.28:0/571768510 >> [v2:172.21.15.28:6800/3783742728,v1:172.21.15.28:6801/3783742728] conn(0x7f1980039050 0x7f198003b510 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:58:01.953 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:01.208+0000 7f19a0200700 1 --2- 172.21.15.28:0/571768510 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f199810a1b0 0x7f199819e150 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:58:01.953 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:01.208+0000 7f19a0200700 1 -- 172.21.15.28:0/571768510 >> 172.21.15.28:0/571768510 conn(0x7f199807d650 msgr2=0x7f19981074c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T21:58:01.953 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:01.208+0000 7f19a0200700 1 -- 172.21.15.28:0/571768510 shutdown_connections 2024-05-12T21:58:01.953 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:01.208+0000 7f19a0200700 1 -- 172.21.15.28:0/571768510 wait complete. 2024-05-12T21:58:02.034 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 21:58:01 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T21:58:01.779+0000 7f1e8436f200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-12T21:58:02.035 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 21:58:01 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T21:58:01.922+0000 7f1e8436f200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-12T21:58:02.396 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:58:02 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/571768510' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-05-12T21:58:02.396 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:58:02 smithi028 ceph-mon[29847]: mgrmap e5: x(active, since 5s) 2024-05-12T21:58:02.396 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 21:58:02 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T21:58:02.346+0000 7f1e8436f200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-12T21:58:02.762 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 21:58:02 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T21:58:02.465+0000 7f1e8436f200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-12T21:58:03.489 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:58:03 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/3656564902' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-05-12T21:58:03.504 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout { 2024-05-12T21:58:03.504 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-05-12T21:58:03.504 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "available": true, 2024-05-12T21:58:03.504 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-05-12T21:58:03.504 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-05-12T21:58:03.504 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout } 2024-05-12T21:58:03.504 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:02.550+0000 7f193e400700 1 Processor -- start 2024-05-12T21:58:03.504 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:02.550+0000 7f193e400700 1 -- start start 2024-05-12T21:58:03.504 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:02.551+0000 7f193e400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19381067b0 0x7f1938106bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T21:58:03.504 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:02.551+0000 7f193e400700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1938063640 con 0x7f19381067b0 2024-05-12T21:58:03.505 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:02.551+0000 7f1937400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19381067b0 0x7f1938106bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T21:58:03.505 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:02.551+0000 7f1937400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19381067b0 0x7f1938106bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47612/0 (socket says 172.21.15.28:47612) 2024-05-12T21:58:03.505 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:02.551+0000 7f1937400700 1 -- 172.21.15.28:0/2016560439 learned_addr learned my addr 172.21.15.28:0/2016560439 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T21:58:03.505 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:02.552+0000 7f1937400700 1 -- 172.21.15.28:0/2016560439 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1938107110 con 0x7f19381067b0 2024-05-12T21:58:03.505 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:02.552+0000 7f1937400700 1 --2- 172.21.15.28:0/2016560439 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19381067b0 0x7f1938106bd0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f19280098a0 tx=0x7f1928009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90744beac47992a6 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T21:58:03.505 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:02.553+0000 7f1936000700 1 -- 172.21.15.28:0/2016560439 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1928004160 con 0x7f19381067b0 2024-05-12T21:58:03.505 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:02.553+0000 7f1936000700 1 -- 172.21.15.28:0/2016560439 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f1928004300 con 0x7f19381067b0 2024-05-12T21:58:03.505 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:02.553+0000 7f1936000700 1 -- 172.21.15.28:0/2016560439 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1928011b50 con 0x7f19381067b0 2024-05-12T21:58:03.505 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:02.553+0000 7f193e400700 1 -- 172.21.15.28:0/2016560439 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19381067b0 msgr2=0x7f1938106bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T21:58:03.505 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:02.553+0000 7f193e400700 1 --2- 172.21.15.28:0/2016560439 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19381067b0 0x7f1938106bd0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f19280098a0 tx=0x7f1928009bb0 comp rx=0 tx=0).stop 2024-05-12T21:58:03.505 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:02.554+0000 7f193e400700 1 -- 172.21.15.28:0/2016560439 shutdown_connections 2024-05-12T21:58:03.505 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:02.554+0000 7f193e400700 1 --2- 172.21.15.28:0/2016560439 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19381067b0 0x7f1938106bd0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:58:03.505 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:02.554+0000 7f193e400700 1 -- 172.21.15.28:0/2016560439 >> 172.21.15.28:0/2016560439 conn(0x7f1938102410 msgr2=0x7f1938104830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T21:58:03.505 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:02.554+0000 7f193e400700 1 -- 172.21.15.28:0/2016560439 shutdown_connections 2024-05-12T21:58:03.506 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:02.554+0000 7f193e400700 1 -- 172.21.15.28:0/2016560439 wait complete. 2024-05-12T21:58:03.506 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:02.555+0000 7f193e400700 1 Processor -- start 2024-05-12T21:58:03.506 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:02.555+0000 7f193e400700 1 -- start start 2024-05-12T21:58:03.506 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:02.556+0000 7f193e400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19381067b0 0x7f193819e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T21:58:03.506 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:02.556+0000 7f193e400700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f19381081a0 con 0x7f19381067b0 2024-05-12T21:58:03.506 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:02.556+0000 7f1937400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19381067b0 0x7f193819e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T21:58:03.506 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:02.556+0000 7f1937400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19381067b0 0x7f193819e0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47622/0 (socket says 172.21.15.28:47622) 2024-05-12T21:58:03.506 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:02.556+0000 7f1937400700 1 -- 172.21.15.28:0/3656564902 learned_addr learned my addr 172.21.15.28:0/3656564902 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T21:58:03.506 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:02.557+0000 7f1937400700 1 -- 172.21.15.28:0/3656564902 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f193819e610 con 0x7f19381067b0 2024-05-12T21:58:03.506 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:02.557+0000 7f1937400700 1 --2- 172.21.15.28:0/3656564902 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19381067b0 0x7f193819e0d0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f1928011670 tx=0x7f1928011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T21:58:03.506 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:02.557+0000 7f192fe00700 1 -- 172.21.15.28:0/3656564902 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f19280039c0 con 0x7f19381067b0 2024-05-12T21:58:03.506 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:02.557+0000 7f192fe00700 1 -- 172.21.15.28:0/3656564902 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f1928011e50 con 0x7f19381067b0 2024-05-12T21:58:03.506 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:02.557+0000 7f192fe00700 1 -- 172.21.15.28:0/3656564902 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f192801a5c0 con 0x7f19381067b0 2024-05-12T21:58:03.506 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:02.557+0000 7f193e400700 1 -- 172.21.15.28:0/3656564902 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f193819e8a0 con 0x7f19381067b0 2024-05-12T21:58:03.507 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:02.558+0000 7f193e400700 1 -- 172.21.15.28:0/3656564902 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19381a1590 con 0x7f19381067b0 2024-05-12T21:58:03.507 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:02.559+0000 7f192fe00700 1 -- 172.21.15.28:0/3656564902 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 46287+0+0 (secure 0 0 0) 0x7f1928024430 con 0x7f19381067b0 2024-05-12T21:58:03.507 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:02.559+0000 7f193e400700 1 -- 172.21.15.28:0/3656564902 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1938063980 con 0x7f19381067b0 2024-05-12T21:58:03.507 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:02.559+0000 7f192fe00700 1 --2- 172.21.15.28:0/3656564902 >> [v2:172.21.15.28:6800/3783742728,v1:172.21.15.28:6801/3783742728] conn(0x7f191c039050 0x7f191c03b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T21:58:03.509 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:02.559+0000 7f192fe00700 1 -- 172.21.15.28:0/3656564902 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f192804d420 con 0x7f19381067b0 2024-05-12T21:58:03.509 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:02.560+0000 7f1936a00700 1 -- 172.21.15.28:0/3656564902 >> [v2:172.21.15.28:6800/3783742728,v1:172.21.15.28:6801/3783742728] conn(0x7f191c039050 msgr2=0x7f191c03b510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/3783742728 2024-05-12T21:58:03.509 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:02.560+0000 7f1936a00700 1 --2- 172.21.15.28:0/3656564902 >> [v2:172.21.15.28:6800/3783742728,v1:172.21.15.28:6801/3783742728] conn(0x7f191c039050 0x7f191c03b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-12T21:58:03.509 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:02.565+0000 7f192fe00700 1 -- 172.21.15.28:0/3656564902 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f192804ce40 con 0x7f19381067b0 2024-05-12T21:58:03.509 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:02.761+0000 7f1936a00700 1 -- 172.21.15.28:0/3656564902 >> [v2:172.21.15.28:6800/3783742728,v1:172.21.15.28:6801/3783742728] conn(0x7f191c039050 msgr2=0x7f191c03b510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/3783742728 2024-05-12T21:58:03.509 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:02.761+0000 7f1936a00700 1 --2- 172.21.15.28:0/3656564902 >> [v2:172.21.15.28:6800/3783742728,v1:172.21.15.28:6801/3783742728] conn(0x7f191c039050 0x7f191c03b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-12T21:58:03.509 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:02.848+0000 7f193e400700 1 -- 172.21.15.28:0/3656564902 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f19380008d0 con 0x7f19381067b0 2024-05-12T21:58:03.509 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:02.849+0000 7f192fe00700 1 -- 172.21.15.28:0/3656564902 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) ==== 56+0+88 (secure 0 0 0) 0x7f192802d3f0 con 0x7f19381067b0 2024-05-12T21:58:03.509 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:02.852+0000 7f193e400700 1 -- 172.21.15.28:0/3656564902 >> [v2:172.21.15.28:6800/3783742728,v1:172.21.15.28:6801/3783742728] conn(0x7f191c039050 msgr2=0x7f191c03b510 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-12T21:58:03.510 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:02.852+0000 7f193e400700 1 --2- 172.21.15.28:0/3656564902 >> [v2:172.21.15.28:6800/3783742728,v1:172.21.15.28:6801/3783742728] conn(0x7f191c039050 0x7f191c03b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:58:03.510 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:02.852+0000 7f193e400700 1 -- 172.21.15.28:0/3656564902 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19381067b0 msgr2=0x7f193819e0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T21:58:03.510 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:02.852+0000 7f193e400700 1 --2- 172.21.15.28:0/3656564902 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19381067b0 0x7f193819e0d0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f1928011670 tx=0x7f1928011b60 comp rx=0 tx=0).stop 2024-05-12T21:58:03.510 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:02.852+0000 7f193e400700 1 -- 172.21.15.28:0/3656564902 shutdown_connections 2024-05-12T21:58:03.510 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:02.852+0000 7f193e400700 1 --2- 172.21.15.28:0/3656564902 >> [v2:172.21.15.28:6800/3783742728,v1:172.21.15.28:6801/3783742728] conn(0x7f191c039050 0x7f191c03b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:58:03.510 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:02.852+0000 7f193e400700 1 --2- 172.21.15.28:0/3656564902 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19381067b0 0x7f193819e0d0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:58:03.510 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:02.852+0000 7f193e400700 1 -- 172.21.15.28:0/3656564902 >> 172.21.15.28:0/3656564902 conn(0x7f1938102410 msgr2=0x7f1938194d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T21:58:03.510 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:02.852+0000 7f193e400700 1 -- 172.21.15.28:0/3656564902 shutdown_connections 2024-05-12T21:58:03.510 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:02.852+0000 7f193e400700 1 -- 172.21.15.28:0/3656564902 wait complete. 2024-05-12T21:58:03.510 INFO:teuthology.orchestra.run.smithi028.stdout:Waiting for the mgr to restart... 2024-05-12T21:58:03.510 INFO:teuthology.orchestra.run.smithi028.stdout:Waiting for mgr epoch 5... 2024-05-12T21:58:04.528 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 21:58:04 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T21:58:04.074+0000 7f1e8436f200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-12T21:58:04.528 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 21:58:04 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T21:58:04.310+0000 7f1e8436f200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-12T21:58:04.528 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 21:58:04 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T21:58:04.528+0000 7f1e8436f200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-12T21:58:05.285 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 21:58:04 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T21:58:04.919+0000 7f1e8436f200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-12T21:58:05.285 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 21:58:05 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T21:58:05.046+0000 7f1e8436f200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-12T21:58:05.569 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 21:58:05 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T21:58:05.304+0000 7f1e8436f200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-12T21:58:05.569 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 21:58:05 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T21:58:05.568+0000 7f1e8436f200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-12T21:58:06.785 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 21:58:06 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T21:58:06.382+0000 7f1e8436f200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-12T21:58:07.285 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 21:58:06 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T21:58:06.954+0000 7f1e8436f200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-12T21:58:07.285 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 21:58:07 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T21:58:07.086+0000 7f1e8436f200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-12T21:58:07.785 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 21:58:07 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T21:58:07.360+0000 7f1e8436f200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-12T21:58:08.285 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 21:58:08 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T21:58:08.028+0000 7f1e8436f200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-12T21:58:08.285 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 21:58:08 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T21:58:08.154+0000 7f1e8436f200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-12T21:58:08.785 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 21:58:08 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T21:58:08.486+0000 7f1e8436f200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-12T21:58:08.785 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 21:58:08 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T21:58:08.672+0000 7f1e8436f200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-12T21:58:09.285 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 21:58:08 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T21:58:08.803+0000 7f1e8436f200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-12T21:58:09.285 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 21:58:08 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T21:58:08.962+0000 7f1e8436f200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-12T21:58:09.785 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 21:58:09 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T21:58:09.336+0000 7f1e8436f200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-12T21:58:10.418 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 21:58:10 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T21:58:10.121+0000 7f1e8436f200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-12T21:58:10.785 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 21:58:10 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T21:58:10.418+0000 7f1e8436f200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-12T21:58:10.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:58:10 smithi028 ceph-mon[29847]: Active manager daemon x restarted 2024-05-12T21:58:10.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:58:10 smithi028 ceph-mon[29847]: Activating manager daemon x 2024-05-12T21:58:10.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:58:10 smithi028 ceph-mon[29847]: osdmap e2: 0 total, 0 up, 0 in 2024-05-12T21:58:10.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:58:10 smithi028 ceph-mon[29847]: mgrmap e6: x(active, starting, since 0.00810797s) 2024-05-12T21:58:10.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:58:10 smithi028 ceph-mon[29847]: from='mgr.14120 172.21.15.28:0/1562014889' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-12T21:58:10.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:58:10 smithi028 ceph-mon[29847]: from='mgr.14120 172.21.15.28:0/1562014889' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-12T21:58:10.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:58:10 smithi028 ceph-mon[29847]: from='mgr.14120 172.21.15.28:0/1562014889' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-12T21:58:10.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:58:10 smithi028 ceph-mon[29847]: from='mgr.14120 172.21.15.28:0/1562014889' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-12T21:58:10.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:58:10 smithi028 ceph-mon[29847]: from='mgr.14120 172.21.15.28:0/1562014889' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-12T21:58:10.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:58:10 smithi028 ceph-mon[29847]: Manager daemon x is now available 2024-05-12T21:58:11.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:58:11 smithi028 ceph-mon[29847]: Found migration_current of "None". Setting to last migration. 2024-05-12T21:58:11.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:58:11 smithi028 ceph-mon[29847]: from='mgr.14120 172.21.15.28:0/1562014889' entity='mgr.x' 2024-05-12T21:58:11.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:58:11 smithi028 ceph-mon[29847]: from='mgr.14120 172.21.15.28:0/1562014889' entity='mgr.x' 2024-05-12T21:58:11.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:58:11 smithi028 ceph-mon[29847]: from='mgr.14120 172.21.15.28:0/1562014889' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T21:58:11.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:58:11 smithi028 ceph-mon[29847]: from='mgr.14120 172.21.15.28:0/1562014889' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T21:58:11.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:58:11 smithi028 ceph-mon[29847]: from='mgr.14120 172.21.15.28:0/1562014889' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-05-12T21:58:11.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:58:11 smithi028 ceph-mon[29847]: from='mgr.14120 172.21.15.28:0/1562014889' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-05-12T21:58:11.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:58:11 smithi028 ceph-mon[29847]: from='mgr.14120 172.21.15.28:0/1562014889' entity='mgr.x' 2024-05-12T21:58:11.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:58:11 smithi028 ceph-mon[29847]: from='mgr.14120 172.21.15.28:0/1562014889' entity='mgr.x' 2024-05-12T21:58:11.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:58:11 smithi028 ceph-mon[29847]: mgrmap e7: x(active, since 1.01543s) 2024-05-12T21:58:12.153 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout { 2024-05-12T21:58:12.154 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-05-12T21:58:12.154 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "initialized": true 2024-05-12T21:58:12.154 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout } 2024-05-12T21:58:12.154 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:04.080+0000 7f6daa400700 1 Processor -- start 2024-05-12T21:58:12.154 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:04.080+0000 7f6daa400700 1 -- start start 2024-05-12T21:58:12.154 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:04.080+0000 7f6daa400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6da407cd20 0x7f6da407b180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T21:58:12.154 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:04.080+0000 7f6daa400700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6da407b750 con 0x7f6da407cd20 2024-05-12T21:58:12.154 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:04.081+0000 7f6da9000700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6da407cd20 0x7f6da407b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T21:58:12.154 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:04.081+0000 7f6da9000700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6da407cd20 0x7f6da407b180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47626/0 (socket says 172.21.15.28:47626) 2024-05-12T21:58:12.154 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:04.081+0000 7f6da9000700 1 -- 172.21.15.28:0/2748895361 learned_addr learned my addr 172.21.15.28:0/2748895361 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T21:58:12.154 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:04.081+0000 7f6da9000700 1 -- 172.21.15.28:0/2748895361 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6da407b8d0 con 0x7f6da407cd20 2024-05-12T21:58:12.154 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:04.082+0000 7f6da9000700 1 --2- 172.21.15.28:0/2748895361 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6da407cd20 0x7f6da407b180 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f6d980098a0 tx=0x7f6d98009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2508b3dca88e7c86 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T21:58:12.154 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:04.082+0000 7f6da3400700 1 -- 172.21.15.28:0/2748895361 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6d98004160 con 0x7f6da407cd20 2024-05-12T21:58:12.154 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:04.082+0000 7f6da3400700 1 -- 172.21.15.28:0/2748895361 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f6d98004300 con 0x7f6da407cd20 2024-05-12T21:58:12.155 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:04.082+0000 7f6da3400700 1 -- 172.21.15.28:0/2748895361 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6d98011b50 con 0x7f6da407cd20 2024-05-12T21:58:12.155 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:04.083+0000 7f6daa400700 1 -- 172.21.15.28:0/2748895361 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6da407cd20 msgr2=0x7f6da407b180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T21:58:12.155 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:04.083+0000 7f6daa400700 1 --2- 172.21.15.28:0/2748895361 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6da407cd20 0x7f6da407b180 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f6d980098a0 tx=0x7f6d98009bb0 comp rx=0 tx=0).stop 2024-05-12T21:58:12.155 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:04.083+0000 7f6daa400700 1 -- 172.21.15.28:0/2748895361 shutdown_connections 2024-05-12T21:58:12.155 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:04.083+0000 7f6daa400700 1 --2- 172.21.15.28:0/2748895361 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6da407cd20 0x7f6da407b180 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:58:12.155 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:04.083+0000 7f6daa400700 1 -- 172.21.15.28:0/2748895361 >> 172.21.15.28:0/2748895361 conn(0x7f6da4103390 msgr2=0x7f6da41057d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T21:58:12.155 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:04.083+0000 7f6daa400700 1 -- 172.21.15.28:0/2748895361 shutdown_connections 2024-05-12T21:58:12.155 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:04.083+0000 7f6daa400700 1 -- 172.21.15.28:0/2748895361 wait complete. 2024-05-12T21:58:12.155 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:04.084+0000 7f6daa400700 1 Processor -- start 2024-05-12T21:58:12.155 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:04.085+0000 7f6daa400700 1 -- start start 2024-05-12T21:58:12.155 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:04.085+0000 7f6daa400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6da407cd20 0x7f6da419dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T21:58:12.155 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:04.085+0000 7f6daa400700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6da410e400 con 0x7f6da407cd20 2024-05-12T21:58:12.155 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:04.085+0000 7f6da9000700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6da407cd20 0x7f6da419dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T21:58:12.155 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:04.085+0000 7f6da9000700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6da407cd20 0x7f6da419dff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47636/0 (socket says 172.21.15.28:47636) 2024-05-12T21:58:12.156 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:04.085+0000 7f6da9000700 1 -- 172.21.15.28:0/3338602647 learned_addr learned my addr 172.21.15.28:0/3338602647 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T21:58:12.156 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:04.086+0000 7f6da9000700 1 -- 172.21.15.28:0/3338602647 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6da419e530 con 0x7f6da407cd20 2024-05-12T21:58:12.156 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:04.086+0000 7f6da9000700 1 --2- 172.21.15.28:0/3338602647 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6da407cd20 0x7f6da419dff0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f6d98011850 tx=0x7f6d98011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T21:58:12.156 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:04.086+0000 7f6da1600700 1 -- 172.21.15.28:0/3338602647 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6d98003af0 con 0x7f6da407cd20 2024-05-12T21:58:12.156 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:04.086+0000 7f6da1600700 1 -- 172.21.15.28:0/3338602647 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f6d98020070 con 0x7f6da407cd20 2024-05-12T21:58:12.156 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:04.086+0000 7f6da1600700 1 -- 172.21.15.28:0/3338602647 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6d980195a0 con 0x7f6da407cd20 2024-05-12T21:58:12.156 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:04.086+0000 7f6daa400700 1 -- 172.21.15.28:0/3338602647 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6da419e7c0 con 0x7f6da407cd20 2024-05-12T21:58:12.156 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:04.086+0000 7f6daa400700 1 -- 172.21.15.28:0/3338602647 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6da41a14b0 con 0x7f6da407cd20 2024-05-12T21:58:12.156 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:04.088+0000 7f6da1600700 1 -- 172.21.15.28:0/3338602647 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 46287+0+0 (secure 0 0 0) 0x7f6d9801a430 con 0x7f6da407cd20 2024-05-12T21:58:12.156 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:04.088+0000 7f6da1600700 1 --2- 172.21.15.28:0/3338602647 >> [v2:172.21.15.28:6800/3783742728,v1:172.21.15.28:6801/3783742728] conn(0x7f6d8c0390a0 0x7f6d8c03b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T21:58:12.156 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:04.088+0000 7f6da1600700 1 -- 172.21.15.28:0/3338602647 --> [v2:172.21.15.28:6800/3783742728,v1:172.21.15.28:6801/3783742728] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f6d8c03bc70 con 0x7f6d8c0390a0 2024-05-12T21:58:12.156 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:04.088+0000 7f6da1600700 1 -- 172.21.15.28:0/3338602647 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f6d98011e00 con 0x7f6da407cd20 2024-05-12T21:58:12.156 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:04.088+0000 7f6da3e00700 1 -- 172.21.15.28:0/3338602647 >> [v2:172.21.15.28:6800/3783742728,v1:172.21.15.28:6801/3783742728] conn(0x7f6d8c0390a0 msgr2=0x7f6d8c03b560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/3783742728 2024-05-12T21:58:12.156 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:04.088+0000 7f6da3e00700 1 --2- 172.21.15.28:0/3338602647 >> [v2:172.21.15.28:6800/3783742728,v1:172.21.15.28:6801/3783742728] conn(0x7f6d8c0390a0 0x7f6d8c03b560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-12T21:58:12.156 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:04.289+0000 7f6da3e00700 1 -- 172.21.15.28:0/3338602647 >> [v2:172.21.15.28:6800/3783742728,v1:172.21.15.28:6801/3783742728] conn(0x7f6d8c0390a0 msgr2=0x7f6d8c03b560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/3783742728 2024-05-12T21:58:12.157 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:04.289+0000 7f6da3e00700 1 --2- 172.21.15.28:0/3338602647 >> [v2:172.21.15.28:6800/3783742728,v1:172.21.15.28:6801/3783742728] conn(0x7f6d8c0390a0 0x7f6d8c03b560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-12T21:58:12.157 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:04.690+0000 7f6da3e00700 1 -- 172.21.15.28:0/3338602647 >> [v2:172.21.15.28:6800/3783742728,v1:172.21.15.28:6801/3783742728] conn(0x7f6d8c0390a0 msgr2=0x7f6d8c03b560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/3783742728 2024-05-12T21:58:12.157 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:04.690+0000 7f6da3e00700 1 --2- 172.21.15.28:0/3338602647 >> [v2:172.21.15.28:6800/3783742728,v1:172.21.15.28:6801/3783742728] conn(0x7f6d8c0390a0 0x7f6d8c03b560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-12T21:58:12.157 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:05.491+0000 7f6da3e00700 1 -- 172.21.15.28:0/3338602647 >> [v2:172.21.15.28:6800/3783742728,v1:172.21.15.28:6801/3783742728] conn(0x7f6d8c0390a0 msgr2=0x7f6d8c03b560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/3783742728 2024-05-12T21:58:12.157 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:05.491+0000 7f6da3e00700 1 --2- 172.21.15.28:0/3338602647 >> [v2:172.21.15.28:6800/3783742728,v1:172.21.15.28:6801/3783742728] conn(0x7f6d8c0390a0 0x7f6d8c03b560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-12T21:58:12.157 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:07.093+0000 7f6da3e00700 1 -- 172.21.15.28:0/3338602647 >> [v2:172.21.15.28:6800/3783742728,v1:172.21.15.28:6801/3783742728] conn(0x7f6d8c0390a0 msgr2=0x7f6d8c03b560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/3783742728 2024-05-12T21:58:12.157 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:07.093+0000 7f6da3e00700 1 --2- 172.21.15.28:0/3338602647 >> [v2:172.21.15.28:6800/3783742728,v1:172.21.15.28:6801/3783742728] conn(0x7f6d8c0390a0 0x7f6d8c03b560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-05-12T21:58:12.157 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:10.297+0000 7f6da3e00700 1 -- 172.21.15.28:0/3338602647 >> [v2:172.21.15.28:6800/3783742728,v1:172.21.15.28:6801/3783742728] conn(0x7f6d8c0390a0 msgr2=0x7f6d8c03b560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/3783742728 2024-05-12T21:58:12.157 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:10.297+0000 7f6da3e00700 1 --2- 172.21.15.28:0/3338602647 >> [v2:172.21.15.28:6800/3783742728,v1:172.21.15.28:6801/3783742728] conn(0x7f6d8c0390a0 0x7f6d8c03b560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-05-12T21:58:12.157 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:10.427+0000 7f6da1600700 1 -- 172.21.15.28:0/3338602647 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mgrmap(e 6) ==== 46054+0+0 (secure 0 0 0) 0x7f6d9801a770 con 0x7f6da407cd20 2024-05-12T21:58:12.157 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:10.427+0000 7f6da1600700 1 -- 172.21.15.28:0/3338602647 >> [v2:172.21.15.28:6800/3783742728,v1:172.21.15.28:6801/3783742728] conn(0x7f6d8c0390a0 msgr2=0x7f6d8c03b560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-12T21:58:12.157 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:10.427+0000 7f6da1600700 1 --2- 172.21.15.28:0/3338602647 >> [v2:172.21.15.28:6800/3783742728,v1:172.21.15.28:6801/3783742728] conn(0x7f6d8c0390a0 0x7f6d8c03b560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:58:12.157 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:11.430+0000 7f6da1600700 1 -- 172.21.15.28:0/3338602647 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mgrmap(e 7) ==== 46181+0+0 (secure 0 0 0) 0x7f6d98025ce0 con 0x7f6da407cd20 2024-05-12T21:58:12.157 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:11.430+0000 7f6da1600700 1 --2- 172.21.15.28:0/3338602647 >> [v2:172.21.15.28:6800/3760499540,v1:172.21.15.28:6801/3760499540] conn(0x7f6d8c03c9d0 0x7f6d8c03edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T21:58:12.157 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:11.430+0000 7f6da1600700 1 -- 172.21.15.28:0/3338602647 --> [v2:172.21.15.28:6800/3760499540,v1:172.21.15.28:6801/3760499540] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f6d8c03bc70 con 0x7f6d8c03c9d0 2024-05-12T21:58:12.157 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:11.432+0000 7f6da3e00700 1 --2- 172.21.15.28:0/3338602647 >> [v2:172.21.15.28:6800/3760499540,v1:172.21.15.28:6801/3760499540] conn(0x7f6d8c03c9d0 0x7f6d8c03edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T21:58:12.158 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:11.434+0000 7f6da3e00700 1 --2- 172.21.15.28:0/3338602647 >> [v2:172.21.15.28:6800/3760499540,v1:172.21.15.28:6801/3760499540] conn(0x7f6d8c03c9d0 0x7f6d8c03edc0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f6d90003a10 tx=0x7f6d900092b0 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T21:58:12.158 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:11.437+0000 7f6da1600700 1 -- 172.21.15.28:0/3338602647 <== mgr.14120 v2:172.21.15.28:6800/3760499540 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7f6d8c03bc70 con 0x7f6d8c03c9d0 2024-05-12T21:58:12.158 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:11.445+0000 7f6daa400700 1 -- 172.21.15.28:0/3338602647 --> [v2:172.21.15.28:6800/3760499540,v1:172.21.15.28:6801/3760499540] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f6da40694a0 con 0x7f6d8c03c9d0 2024-05-12T21:58:12.158 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:11.445+0000 7f6da1600700 1 -- 172.21.15.28:0/3338602647 <== mgr.14120 v2:172.21.15.28:6800/3760499540 2 ==== command_reply(tid 1: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7f6da40694a0 con 0x7f6d8c03c9d0 2024-05-12T21:58:12.158 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:11.446+0000 7f6daa400700 1 -- 172.21.15.28:0/3338602647 >> [v2:172.21.15.28:6800/3760499540,v1:172.21.15.28:6801/3760499540] conn(0x7f6d8c03c9d0 msgr2=0x7f6d8c03edc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T21:58:12.158 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:11.446+0000 7f6daa400700 1 --2- 172.21.15.28:0/3338602647 >> [v2:172.21.15.28:6800/3760499540,v1:172.21.15.28:6801/3760499540] conn(0x7f6d8c03c9d0 0x7f6d8c03edc0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f6d90003a10 tx=0x7f6d900092b0 comp rx=0 tx=0).stop 2024-05-12T21:58:12.158 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:11.446+0000 7f6daa400700 1 -- 172.21.15.28:0/3338602647 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6da407cd20 msgr2=0x7f6da419dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T21:58:12.158 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:11.446+0000 7f6daa400700 1 --2- 172.21.15.28:0/3338602647 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6da407cd20 0x7f6da419dff0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f6d98011850 tx=0x7f6d98011c10 comp rx=0 tx=0).stop 2024-05-12T21:58:12.158 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:11.446+0000 7f6daa400700 1 -- 172.21.15.28:0/3338602647 shutdown_connections 2024-05-12T21:58:12.158 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:11.446+0000 7f6daa400700 1 --2- 172.21.15.28:0/3338602647 >> [v2:172.21.15.28:6800/3760499540,v1:172.21.15.28:6801/3760499540] conn(0x7f6d8c03c9d0 0x7f6d8c03edc0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:58:12.158 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:11.446+0000 7f6daa400700 1 --2- 172.21.15.28:0/3338602647 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6da407cd20 0x7f6da419dff0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:58:12.158 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:11.446+0000 7f6daa400700 1 --2- 172.21.15.28:0/3338602647 >> [v2:172.21.15.28:6800/3783742728,v1:172.21.15.28:6801/3783742728] conn(0x7f6d8c0390a0 0x7f6d8c03b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:58:12.158 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:11.446+0000 7f6daa400700 1 -- 172.21.15.28:0/3338602647 >> 172.21.15.28:0/3338602647 conn(0x7f6da4103390 msgr2=0x7f6da4103f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T21:58:12.158 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:11.447+0000 7f6daa400700 1 -- 172.21.15.28:0/3338602647 shutdown_connections 2024-05-12T21:58:12.159 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:11.447+0000 7f6daa400700 1 -- 172.21.15.28:0/3338602647 wait complete. 2024-05-12T21:58:12.159 INFO:teuthology.orchestra.run.smithi028.stdout:mgr epoch 5 is available 2024-05-12T21:58:12.159 INFO:teuthology.orchestra.run.smithi028.stdout:Setting orchestrator backend to cephadm... 2024-05-12T21:58:12.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:58:12 smithi028 ceph-mon[29847]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-05-12T21:58:12.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:58:12 smithi028 ceph-mon[29847]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-05-12T21:58:12.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:58:12 smithi028 ceph-mon[29847]: [12/May/2024:21:58:12] ENGINE Bus STARTING 2024-05-12T21:58:12.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:58:12 smithi028 ceph-mon[29847]: from='mgr.14120 172.21.15.28:0/1562014889' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T21:58:13.760 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:58:13 smithi028 ceph-mon[29847]: [12/May/2024:21:58:12] ENGINE Serving on https://172.21.15.28:7150 2024-05-12T21:58:13.760 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:58:13 smithi028 ceph-mon[29847]: [12/May/2024:21:58:12] ENGINE Serving on http://172.21.15.28:8765 2024-05-12T21:58:13.760 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:58:13 smithi028 ceph-mon[29847]: [12/May/2024:21:58:12] ENGINE Bus STARTED 2024-05-12T21:58:13.760 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:58:13 smithi028 ceph-mon[29847]: mgrmap e8: x(active, since 2s) 2024-05-12T21:58:13.760 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:58:13 smithi028 ceph-mon[29847]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-05-12T21:58:13.760 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:58:13 smithi028 ceph-mon[29847]: from='mgr.14120 172.21.15.28:0/1562014889' entity='mgr.x' 2024-05-12T21:58:13.760 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:58:13 smithi028 ceph-mon[29847]: from='mgr.14120 172.21.15.28:0/1562014889' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T21:58:13.772 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:12.790+0000 7f396aa00700 1 Processor -- start 2024-05-12T21:58:13.772 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:12.790+0000 7f396aa00700 1 -- start start 2024-05-12T21:58:13.776 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:12.790+0000 7f396aa00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3964107e60 0x7f3964108280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T21:58:13.776 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:12.790+0000 7f396aa00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f39641087c0 con 0x7f3964107e60 2024-05-12T21:58:13.776 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:12.791+0000 7f3969600700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3964107e60 0x7f3964108280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T21:58:13.776 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:12.791+0000 7f3969600700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3964107e60 0x7f3964108280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37488/0 (socket says 172.21.15.28:37488) 2024-05-12T21:58:13.776 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:12.791+0000 7f3969600700 1 -- 172.21.15.28:0/225019310 learned_addr learned my addr 172.21.15.28:0/225019310 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T21:58:13.776 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:12.791+0000 7f3969600700 1 -- 172.21.15.28:0/225019310 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3964108fc0 con 0x7f3964107e60 2024-05-12T21:58:13.776 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:12.792+0000 7f3969600700 1 --2- 172.21.15.28:0/225019310 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3964107e60 0x7f3964108280 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f39580098a0 tx=0x7f3958009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33d7a5b7d623c7aa server_cookie=0 in_seq=0 out_seq=0 2024-05-12T21:58:13.776 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:12.792+0000 7f3963e00700 1 -- 172.21.15.28:0/225019310 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3958004160 con 0x7f3964107e60 2024-05-12T21:58:13.776 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:12.792+0000 7f3963e00700 1 -- 172.21.15.28:0/225019310 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f3958004300 con 0x7f3964107e60 2024-05-12T21:58:13.776 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:12.792+0000 7f3963e00700 1 -- 172.21.15.28:0/225019310 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3958011b50 con 0x7f3964107e60 2024-05-12T21:58:13.776 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:12.792+0000 7f396aa00700 1 -- 172.21.15.28:0/225019310 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3964107e60 msgr2=0x7f3964108280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T21:58:13.776 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:12.792+0000 7f396aa00700 1 --2- 172.21.15.28:0/225019310 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3964107e60 0x7f3964108280 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f39580098a0 tx=0x7f3958009bb0 comp rx=0 tx=0).stop 2024-05-12T21:58:13.776 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:12.793+0000 7f396aa00700 1 -- 172.21.15.28:0/225019310 shutdown_connections 2024-05-12T21:58:13.777 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:12.793+0000 7f396aa00700 1 --2- 172.21.15.28:0/225019310 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3964107e60 0x7f3964108280 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:58:13.777 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:12.793+0000 7f396aa00700 1 -- 172.21.15.28:0/225019310 >> 172.21.15.28:0/225019310 conn(0x7f3964103400 msgr2=0x7f3964105840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T21:58:13.777 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:12.793+0000 7f396aa00700 1 -- 172.21.15.28:0/225019310 shutdown_connections 2024-05-12T21:58:13.777 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:12.793+0000 7f396aa00700 1 -- 172.21.15.28:0/225019310 wait complete. 2024-05-12T21:58:13.777 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:12.794+0000 7f396aa00700 1 Processor -- start 2024-05-12T21:58:13.777 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:12.794+0000 7f396aa00700 1 -- start start 2024-05-12T21:58:13.777 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:12.794+0000 7f396aa00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3964107e60 0x7f396419bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T21:58:13.777 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:12.794+0000 7f396aa00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f396410c280 con 0x7f3964107e60 2024-05-12T21:58:13.777 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:12.795+0000 7f3969600700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3964107e60 0x7f396419bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T21:58:13.777 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:12.795+0000 7f3969600700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3964107e60 0x7f396419bdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37494/0 (socket says 172.21.15.28:37494) 2024-05-12T21:58:13.777 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:12.795+0000 7f3969600700 1 -- 172.21.15.28:0/1332138371 learned_addr learned my addr 172.21.15.28:0/1332138371 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T21:58:13.777 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:12.795+0000 7f3969600700 1 -- 172.21.15.28:0/1332138371 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f396419c2f0 con 0x7f3964107e60 2024-05-12T21:58:13.777 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:12.795+0000 7f3969600700 1 --2- 172.21.15.28:0/1332138371 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3964107e60 0x7f396419bdb0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f3958011b60 tx=0x7f3958011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T21:58:13.778 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:12.796+0000 7f3962000700 1 -- 172.21.15.28:0/1332138371 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3958003970 con 0x7f3964107e60 2024-05-12T21:58:13.778 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:12.796+0000 7f3962000700 1 -- 172.21.15.28:0/1332138371 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f3958003b10 con 0x7f3964107e60 2024-05-12T21:58:13.778 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:12.796+0000 7f396aa00700 1 -- 172.21.15.28:0/1332138371 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f396419c580 con 0x7f3964107e60 2024-05-12T21:58:13.778 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:12.796+0000 7f3962000700 1 -- 172.21.15.28:0/1332138371 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3958024610 con 0x7f3964107e60 2024-05-12T21:58:13.778 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:12.796+0000 7f396aa00700 1 -- 172.21.15.28:0/1332138371 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f396419f230 con 0x7f3964107e60 2024-05-12T21:58:13.778 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:12.797+0000 7f3962000700 1 -- 172.21.15.28:0/1332138371 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f395801a430 con 0x7f3964107e60 2024-05-12T21:58:13.778 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:12.797+0000 7f396aa00700 1 -- 172.21.15.28:0/1332138371 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f396410c400 con 0x7f3964107e60 2024-05-12T21:58:13.778 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:12.797+0000 7f3962000700 1 --2- 172.21.15.28:0/1332138371 >> [v2:172.21.15.28:6800/3760499540,v1:172.21.15.28:6801/3760499540] conn(0x7f394c0390a0 0x7f394c03b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T21:58:13.780 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:12.797+0000 7f3962000700 1 -- 172.21.15.28:0/1332138371 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f3958011e00 con 0x7f3964107e60 2024-05-12T21:58:13.780 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:12.798+0000 7f3968c00700 1 --2- 172.21.15.28:0/1332138371 >> [v2:172.21.15.28:6800/3760499540,v1:172.21.15.28:6801/3760499540] conn(0x7f394c0390a0 0x7f394c03b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T21:58:13.780 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:12.798+0000 7f3968c00700 1 --2- 172.21.15.28:0/1332138371 >> [v2:172.21.15.28:6800/3760499540,v1:172.21.15.28:6801/3760499540] conn(0x7f394c0390a0 0x7f394c03b560 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f3950006fd0 tx=0x7f3950006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T21:58:13.780 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:12.804+0000 7f3962000700 1 -- 172.21.15.28:0/1332138371 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f3958016260 con 0x7f3964107e60 2024-05-12T21:58:13.780 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:13.048+0000 7f396aa00700 1 -- 172.21.15.28:0/1332138371 --> [v2:172.21.15.28:6800/3760499540,v1:172.21.15.28:6801/3760499540] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) -- 0x7f39641046b0 con 0x7f394c0390a0 2024-05-12T21:58:13.781 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:13.059+0000 7f3962000700 1 -- 172.21.15.28:0/1332138371 <== mgr.14120 v2:172.21.15.28:6800/3760499540 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f39641046b0 con 0x7f394c0390a0 2024-05-12T21:58:13.781 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:13.062+0000 7f396aa00700 1 -- 172.21.15.28:0/1332138371 >> [v2:172.21.15.28:6800/3760499540,v1:172.21.15.28:6801/3760499540] conn(0x7f394c0390a0 msgr2=0x7f394c03b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T21:58:13.781 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:13.062+0000 7f396aa00700 1 --2- 172.21.15.28:0/1332138371 >> [v2:172.21.15.28:6800/3760499540,v1:172.21.15.28:6801/3760499540] conn(0x7f394c0390a0 0x7f394c03b560 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f3950006fd0 tx=0x7f3950006e40 comp rx=0 tx=0).stop 2024-05-12T21:58:13.781 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:13.062+0000 7f396aa00700 1 -- 172.21.15.28:0/1332138371 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3964107e60 msgr2=0x7f396419bdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T21:58:13.781 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:13.062+0000 7f396aa00700 1 --2- 172.21.15.28:0/1332138371 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3964107e60 0x7f396419bdb0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f3958011b60 tx=0x7f3958011c40 comp rx=0 tx=0).stop 2024-05-12T21:58:13.781 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:13.062+0000 7f396aa00700 1 -- 172.21.15.28:0/1332138371 shutdown_connections 2024-05-12T21:58:13.781 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:13.062+0000 7f396aa00700 1 --2- 172.21.15.28:0/1332138371 >> [v2:172.21.15.28:6800/3760499540,v1:172.21.15.28:6801/3760499540] conn(0x7f394c0390a0 0x7f394c03b560 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:58:13.781 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:13.062+0000 7f396aa00700 1 --2- 172.21.15.28:0/1332138371 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3964107e60 0x7f396419bdb0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:58:13.781 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:13.062+0000 7f396aa00700 1 -- 172.21.15.28:0/1332138371 >> 172.21.15.28:0/1332138371 conn(0x7f3964103400 msgr2=0x7f3964103fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T21:58:13.781 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:13.063+0000 7f396aa00700 1 -- 172.21.15.28:0/1332138371 shutdown_connections 2024-05-12T21:58:13.781 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:13.063+0000 7f396aa00700 1 -- 172.21.15.28:0/1332138371 wait complete. 2024-05-12T21:58:15.300 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout value unchanged 2024-05-12T21:58:15.301 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:14.416+0000 7fee5b400700 1 Processor -- start 2024-05-12T21:58:15.301 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:14.416+0000 7fee5b400700 1 -- start start 2024-05-12T21:58:15.302 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:14.416+0000 7fee5b400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee5410ee30 0x7fee54111220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T21:58:15.302 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:14.417+0000 7fee5b400700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fee54111760 con 0x7fee5410ee30 2024-05-12T21:58:15.302 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:14.417+0000 7fee58a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee5410ee30 0x7fee54111220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T21:58:15.302 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:14.417+0000 7fee58a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee5410ee30 0x7fee54111220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37498/0 (socket says 172.21.15.28:37498) 2024-05-12T21:58:15.302 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:14.417+0000 7fee58a00700 1 -- 172.21.15.28:0/2473554061 learned_addr learned my addr 172.21.15.28:0/2473554061 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T21:58:15.302 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:14.417+0000 7fee58a00700 1 -- 172.21.15.28:0/2473554061 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee541118e0 con 0x7fee5410ee30 2024-05-12T21:58:15.302 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:14.418+0000 7fee58a00700 1 --2- 172.21.15.28:0/2473554061 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee5410ee30 0x7fee54111220 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fee480098a0 tx=0x7fee48009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e26f61abf330b879 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T21:58:15.302 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:14.418+0000 7fee53400700 1 -- 172.21.15.28:0/2473554061 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fee48004160 con 0x7fee5410ee30 2024-05-12T21:58:15.302 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:14.418+0000 7fee53400700 1 -- 172.21.15.28:0/2473554061 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fee48004300 con 0x7fee5410ee30 2024-05-12T21:58:15.302 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:14.418+0000 7fee53400700 1 -- 172.21.15.28:0/2473554061 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fee48011b50 con 0x7fee5410ee30 2024-05-12T21:58:15.302 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:14.419+0000 7fee5b400700 1 -- 172.21.15.28:0/2473554061 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee5410ee30 msgr2=0x7fee54111220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T21:58:15.308 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:14.419+0000 7fee5b400700 1 --2- 172.21.15.28:0/2473554061 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee5410ee30 0x7fee54111220 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fee480098a0 tx=0x7fee48009bb0 comp rx=0 tx=0).stop 2024-05-12T21:58:15.308 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:14.419+0000 7fee5b400700 1 -- 172.21.15.28:0/2473554061 shutdown_connections 2024-05-12T21:58:15.308 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:14.419+0000 7fee5b400700 1 --2- 172.21.15.28:0/2473554061 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee5410ee30 0x7fee54111220 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:58:15.308 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:14.419+0000 7fee5b400700 1 -- 172.21.15.28:0/2473554061 >> 172.21.15.28:0/2473554061 conn(0x7fee5407d650 msgr2=0x7fee5407fa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T21:58:15.308 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:14.419+0000 7fee5b400700 1 -- 172.21.15.28:0/2473554061 shutdown_connections 2024-05-12T21:58:15.308 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:14.419+0000 7fee5b400700 1 -- 172.21.15.28:0/2473554061 wait complete. 2024-05-12T21:58:15.308 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:14.420+0000 7fee5b400700 1 Processor -- start 2024-05-12T21:58:15.308 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:14.420+0000 7fee5b400700 1 -- start start 2024-05-12T21:58:15.308 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:14.421+0000 7fee5b400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee5410ee30 0x7fee541a2540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T21:58:15.309 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:14.421+0000 7fee5b400700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fee54112960 con 0x7fee5410ee30 2024-05-12T21:58:15.309 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:14.421+0000 7fee58a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee5410ee30 0x7fee541a2540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T21:58:15.309 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:14.421+0000 7fee58a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee5410ee30 0x7fee541a2540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37508/0 (socket says 172.21.15.28:37508) 2024-05-12T21:58:15.309 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:14.421+0000 7fee58a00700 1 -- 172.21.15.28:0/3688361641 learned_addr learned my addr 172.21.15.28:0/3688361641 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T21:58:15.309 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:14.421+0000 7fee58a00700 1 -- 172.21.15.28:0/3688361641 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee541a2a80 con 0x7fee5410ee30 2024-05-12T21:58:15.309 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:14.421+0000 7fee58a00700 1 --2- 172.21.15.28:0/3688361641 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee5410ee30 0x7fee541a2540 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fee48011b60 tx=0x7fee48011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T21:58:15.309 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:14.422+0000 7fee51600700 1 -- 172.21.15.28:0/3688361641 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fee48003970 con 0x7fee5410ee30 2024-05-12T21:58:15.309 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:14.422+0000 7fee51600700 1 -- 172.21.15.28:0/3688361641 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fee48003b10 con 0x7fee5410ee30 2024-05-12T21:58:15.310 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:14.422+0000 7fee5b400700 1 -- 172.21.15.28:0/3688361641 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee541a2d10 con 0x7fee5410ee30 2024-05-12T21:58:15.310 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:14.422+0000 7fee51600700 1 -- 172.21.15.28:0/3688361641 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fee48024610 con 0x7fee5410ee30 2024-05-12T21:58:15.310 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:14.422+0000 7fee5b400700 1 -- 172.21.15.28:0/3688361641 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee541a5a00 con 0x7fee5410ee30 2024-05-12T21:58:15.310 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:14.423+0000 7fee51600700 1 -- 172.21.15.28:0/3688361641 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7fee4801a430 con 0x7fee5410ee30 2024-05-12T21:58:15.310 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:14.423+0000 7fee5b400700 1 -- 172.21.15.28:0/3688361641 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee540694a0 con 0x7fee5410ee30 2024-05-12T21:58:15.310 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:14.424+0000 7fee51600700 1 --2- 172.21.15.28:0/3688361641 >> [v2:172.21.15.28:6800/3760499540,v1:172.21.15.28:6801/3760499540] conn(0x7fee3c0390a0 0x7fee3c03b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T21:58:15.310 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:14.424+0000 7fee51600700 1 -- 172.21.15.28:0/3688361641 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fee48011e00 con 0x7fee5410ee30 2024-05-12T21:58:15.310 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:14.424+0000 7fee53e00700 1 --2- 172.21.15.28:0/3688361641 >> [v2:172.21.15.28:6800/3760499540,v1:172.21.15.28:6801/3760499540] conn(0x7fee3c0390a0 0x7fee3c03b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T21:58:15.311 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:14.425+0000 7fee53e00700 1 --2- 172.21.15.28:0/3688361641 >> [v2:172.21.15.28:6800/3760499540,v1:172.21.15.28:6801/3760499540] conn(0x7fee3c0390a0 0x7fee3c03b560 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fee40006fd0 tx=0x7fee40006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T21:58:15.311 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:14.430+0000 7fee51600700 1 -- 172.21.15.28:0/3688361641 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fee48016260 con 0x7fee5410ee30 2024-05-12T21:58:15.311 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:14.660+0000 7fee5b400700 1 -- 172.21.15.28:0/3688361641 --> [v2:172.21.15.28:6800/3760499540,v1:172.21.15.28:6801/3760499540] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) -- 0x7fee5407f8e0 con 0x7fee3c0390a0 2024-05-12T21:58:15.311 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:14.662+0000 7fee51600700 1 -- 172.21.15.28:0/3688361641 <== mgr.14120 v2:172.21.15.28:6800/3760499540 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+16 (secure 0 0 0) 0x7fee5407f8e0 con 0x7fee3c0390a0 2024-05-12T21:58:15.311 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:14.665+0000 7fee5b400700 1 -- 172.21.15.28:0/3688361641 >> [v2:172.21.15.28:6800/3760499540,v1:172.21.15.28:6801/3760499540] conn(0x7fee3c0390a0 msgr2=0x7fee3c03b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T21:58:15.311 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:14.665+0000 7fee5b400700 1 --2- 172.21.15.28:0/3688361641 >> [v2:172.21.15.28:6800/3760499540,v1:172.21.15.28:6801/3760499540] conn(0x7fee3c0390a0 0x7fee3c03b560 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fee40006fd0 tx=0x7fee40006e40 comp rx=0 tx=0).stop 2024-05-12T21:58:15.311 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:14.665+0000 7fee5b400700 1 -- 172.21.15.28:0/3688361641 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee5410ee30 msgr2=0x7fee541a2540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T21:58:15.311 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:14.665+0000 7fee5b400700 1 --2- 172.21.15.28:0/3688361641 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee5410ee30 0x7fee541a2540 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fee48011b60 tx=0x7fee48011c40 comp rx=0 tx=0).stop 2024-05-12T21:58:15.312 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:14.665+0000 7fee5b400700 1 -- 172.21.15.28:0/3688361641 shutdown_connections 2024-05-12T21:58:15.312 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:14.665+0000 7fee5b400700 1 --2- 172.21.15.28:0/3688361641 >> [v2:172.21.15.28:6800/3760499540,v1:172.21.15.28:6801/3760499540] conn(0x7fee3c0390a0 0x7fee3c03b560 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:58:15.312 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:14.665+0000 7fee5b400700 1 --2- 172.21.15.28:0/3688361641 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee5410ee30 0x7fee541a2540 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:58:15.312 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:14.665+0000 7fee5b400700 1 -- 172.21.15.28:0/3688361641 >> 172.21.15.28:0/3688361641 conn(0x7fee5407d650 msgr2=0x7fee5407f1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T21:58:15.312 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:14.665+0000 7fee5b400700 1 -- 172.21.15.28:0/3688361641 shutdown_connections 2024-05-12T21:58:15.312 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:14.665+0000 7fee5b400700 1 -- 172.21.15.28:0/3688361641 wait complete. 2024-05-12T21:58:15.312 INFO:teuthology.orchestra.run.smithi028.stdout:Generating ssh key... 2024-05-12T21:58:15.601 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:58:15 smithi028 ceph-mon[29847]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-05-12T21:58:16.647 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:58:16 smithi028 ceph-mon[29847]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-05-12T21:58:16.690 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:58:16 smithi028 ceph-mon[29847]: Generating ssh key... 2024-05-12T21:58:16.690 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 21:58:16 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: Generating public/private rsa key pair. 2024-05-12T21:58:16.690 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 21:58:16 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: Your identification has been saved in /tmp/tmpe15yyo2c/key. 2024-05-12T21:58:16.690 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 21:58:16 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: Your public key has been saved in /tmp/tmpe15yyo2c/key.pub. 2024-05-12T21:58:16.691 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 21:58:16 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: The key fingerprint is: 2024-05-12T21:58:16.691 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 21:58:16 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: SHA256:F41WKxyUv1VmoQD7XBgi8HVYoWZrP+S2CNFk0qUQ1SM ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968 2024-05-12T21:58:16.691 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 21:58:16 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: The key's randomart image is: 2024-05-12T21:58:16.691 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 21:58:16 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: +---[RSA 3072]----+ 2024-05-12T21:58:16.691 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 21:58:16 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: | ..+o*BOo ..| 2024-05-12T21:58:16.691 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 21:58:16 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: | . =EOB+.. +| 2024-05-12T21:58:16.691 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 21:58:16 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: | o XBo+o + | 2024-05-12T21:58:16.691 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 21:58:16 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: | B.+oo . | 2024-05-12T21:58:16.691 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 21:58:16 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: | .S+.+ o | 2024-05-12T21:58:16.691 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 21:58:16 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: | o.+ . | 2024-05-12T21:58:16.691 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 21:58:16 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: | . = | 2024-05-12T21:58:16.691 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 21:58:16 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: | . o o | 2024-05-12T21:58:16.691 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 21:58:16 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: | . . | 2024-05-12T21:58:16.692 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 21:58:16 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: +----[SHA256]-----+ 2024-05-12T21:58:17.367 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:15.913+0000 7f3b9a200700 1 Processor -- start 2024-05-12T21:58:17.367 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:15.913+0000 7f3b9a200700 1 -- start start 2024-05-12T21:58:17.367 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:15.914+0000 7f3b9a200700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b9407de70 0x7f3b9407c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T21:58:17.367 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:15.914+0000 7f3b9a200700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3b9407c8a0 con 0x7f3b9407de70 2024-05-12T21:58:17.367 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:15.914+0000 7f3b98e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b9407de70 0x7f3b9407c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T21:58:17.368 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:15.914+0000 7f3b98e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b9407de70 0x7f3b9407c2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38676/0 (socket says 172.21.15.28:38676) 2024-05-12T21:58:17.368 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:15.914+0000 7f3b98e00700 1 -- 172.21.15.28:0/2877389054 learned_addr learned my addr 172.21.15.28:0/2877389054 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T21:58:17.368 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:15.915+0000 7f3b98e00700 1 -- 172.21.15.28:0/2877389054 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b9407ca20 con 0x7f3b9407de70 2024-05-12T21:58:17.368 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:15.915+0000 7f3b98e00700 1 --2- 172.21.15.28:0/2877389054 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b9407de70 0x7f3b9407c2d0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f3b880098a0 tx=0x7f3b88009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a4a733f5280427b server_cookie=0 in_seq=0 out_seq=0 2024-05-12T21:58:17.368 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:15.916+0000 7f3b93400700 1 -- 172.21.15.28:0/2877389054 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3b88004160 con 0x7f3b9407de70 2024-05-12T21:58:17.368 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:15.916+0000 7f3b93400700 1 -- 172.21.15.28:0/2877389054 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f3b88004300 con 0x7f3b9407de70 2024-05-12T21:58:17.368 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:15.916+0000 7f3b93400700 1 -- 172.21.15.28:0/2877389054 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3b88011b50 con 0x7f3b9407de70 2024-05-12T21:58:17.368 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:15.916+0000 7f3b9a200700 1 -- 172.21.15.28:0/2877389054 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b9407de70 msgr2=0x7f3b9407c2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T21:58:17.368 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:15.916+0000 7f3b9a200700 1 --2- 172.21.15.28:0/2877389054 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b9407de70 0x7f3b9407c2d0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f3b880098a0 tx=0x7f3b88009bb0 comp rx=0 tx=0).stop 2024-05-12T21:58:17.368 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:15.916+0000 7f3b9a200700 1 -- 172.21.15.28:0/2877389054 shutdown_connections 2024-05-12T21:58:17.368 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:15.916+0000 7f3b9a200700 1 --2- 172.21.15.28:0/2877389054 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b9407de70 0x7f3b9407c2d0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:58:17.368 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:15.916+0000 7f3b9a200700 1 -- 172.21.15.28:0/2877389054 >> 172.21.15.28:0/2877389054 conn(0x7f3b94103400 msgr2=0x7f3b94105840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T21:58:17.368 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:15.916+0000 7f3b9a200700 1 -- 172.21.15.28:0/2877389054 shutdown_connections 2024-05-12T21:58:17.368 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:15.916+0000 7f3b9a200700 1 -- 172.21.15.28:0/2877389054 wait complete. 2024-05-12T21:58:17.368 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:15.917+0000 7f3b9a200700 1 Processor -- start 2024-05-12T21:58:17.369 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:15.918+0000 7f3b9a200700 1 -- start start 2024-05-12T21:58:17.369 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:15.918+0000 7f3b9a200700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b9407de70 0x7f3b941a2450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T21:58:17.369 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:15.918+0000 7f3b9a200700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3b9410a010 con 0x7f3b9407de70 2024-05-12T21:58:17.369 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:15.918+0000 7f3b98e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b9407de70 0x7f3b941a2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T21:58:17.369 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:15.918+0000 7f3b98e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b9407de70 0x7f3b941a2450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38682/0 (socket says 172.21.15.28:38682) 2024-05-12T21:58:17.369 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:15.918+0000 7f3b98e00700 1 -- 172.21.15.28:0/1395128542 learned_addr learned my addr 172.21.15.28:0/1395128542 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T21:58:17.369 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:15.919+0000 7f3b98e00700 1 -- 172.21.15.28:0/1395128542 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b941a2990 con 0x7f3b9407de70 2024-05-12T21:58:17.369 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:15.919+0000 7f3b98e00700 1 --2- 172.21.15.28:0/1395128542 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b9407de70 0x7f3b941a2450 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f3b88009fb0 tx=0x7f3b88011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T21:58:17.369 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:15.919+0000 7f3b91600700 1 -- 172.21.15.28:0/1395128542 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3b880038b0 con 0x7f3b9407de70 2024-05-12T21:58:17.369 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:15.919+0000 7f3b91600700 1 -- 172.21.15.28:0/1395128542 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f3b88011e50 con 0x7f3b9407de70 2024-05-12T21:58:17.369 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:15.919+0000 7f3b91600700 1 -- 172.21.15.28:0/1395128542 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3b880193c0 con 0x7f3b9407de70 2024-05-12T21:58:17.369 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:15.919+0000 7f3b9a200700 1 -- 172.21.15.28:0/1395128542 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b941a2c20 con 0x7f3b9407de70 2024-05-12T21:58:17.369 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:15.920+0000 7f3b9a200700 1 -- 172.21.15.28:0/1395128542 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b941a5910 con 0x7f3b9407de70 2024-05-12T21:58:17.369 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:15.921+0000 7f3b91600700 1 -- 172.21.15.28:0/1395128542 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f3b8801a710 con 0x7f3b9407de70 2024-05-12T21:58:17.369 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:15.921+0000 7f3b9a200700 1 -- 172.21.15.28:0/1395128542 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b940694a0 con 0x7f3b9407de70 2024-05-12T21:58:17.370 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:15.921+0000 7f3b91600700 1 --2- 172.21.15.28:0/1395128542 >> [v2:172.21.15.28:6800/3760499540,v1:172.21.15.28:6801/3760499540] conn(0x7f3b7c039050 0x7f3b7c03b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T21:58:17.370 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:15.921+0000 7f3b91600700 1 -- 172.21.15.28:0/1395128542 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f3b8804d200 con 0x7f3b9407de70 2024-05-12T21:58:17.370 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:15.921+0000 7f3b93e00700 1 --2- 172.21.15.28:0/1395128542 >> [v2:172.21.15.28:6800/3760499540,v1:172.21.15.28:6801/3760499540] conn(0x7f3b7c039050 0x7f3b7c03b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T21:58:17.370 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:15.922+0000 7f3b93e00700 1 --2- 172.21.15.28:0/1395128542 >> [v2:172.21.15.28:6800/3760499540,v1:172.21.15.28:6801/3760499540] conn(0x7f3b7c039050 0x7f3b7c03b510 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f3b80006fd0 tx=0x7f3b80006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T21:58:17.370 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:15.927+0000 7f3b91600700 1 -- 172.21.15.28:0/1395128542 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f3b8801a4e0 con 0x7f3b9407de70 2024-05-12T21:58:17.370 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:16.156+0000 7f3b9a200700 1 -- 172.21.15.28:0/1395128542 --> [v2:172.21.15.28:6800/3760499540,v1:172.21.15.28:6801/3760499540] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) -- 0x7f3b940008d0 con 0x7f3b7c039050 2024-05-12T21:58:17.370 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:16.645+0000 7f3b91600700 1 -- 172.21.15.28:0/1395128542 <== mgr.14120 v2:172.21.15.28:6800/3760499540 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f3b940008d0 con 0x7f3b7c039050 2024-05-12T21:58:17.370 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:16.649+0000 7f3b9a200700 1 -- 172.21.15.28:0/1395128542 >> [v2:172.21.15.28:6800/3760499540,v1:172.21.15.28:6801/3760499540] conn(0x7f3b7c039050 msgr2=0x7f3b7c03b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T21:58:17.370 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:16.649+0000 7f3b9a200700 1 --2- 172.21.15.28:0/1395128542 >> [v2:172.21.15.28:6800/3760499540,v1:172.21.15.28:6801/3760499540] conn(0x7f3b7c039050 0x7f3b7c03b510 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f3b80006fd0 tx=0x7f3b80006e40 comp rx=0 tx=0).stop 2024-05-12T21:58:17.370 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:16.649+0000 7f3b9a200700 1 -- 172.21.15.28:0/1395128542 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b9407de70 msgr2=0x7f3b941a2450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T21:58:17.370 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:16.649+0000 7f3b9a200700 1 --2- 172.21.15.28:0/1395128542 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b9407de70 0x7f3b941a2450 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f3b88009fb0 tx=0x7f3b88011bf0 comp rx=0 tx=0).stop 2024-05-12T21:58:17.370 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:16.649+0000 7f3b9a200700 1 -- 172.21.15.28:0/1395128542 shutdown_connections 2024-05-12T21:58:17.370 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:16.649+0000 7f3b9a200700 1 --2- 172.21.15.28:0/1395128542 >> [v2:172.21.15.28:6800/3760499540,v1:172.21.15.28:6801/3760499540] conn(0x7f3b7c039050 0x7f3b7c03b510 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:58:17.370 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:16.649+0000 7f3b9a200700 1 --2- 172.21.15.28:0/1395128542 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b9407de70 0x7f3b941a2450 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:58:17.370 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:16.649+0000 7f3b9a200700 1 -- 172.21.15.28:0/1395128542 >> 172.21.15.28:0/1395128542 conn(0x7f3b94103400 msgr2=0x7f3b94199150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T21:58:17.371 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:16.650+0000 7f3b9a200700 1 -- 172.21.15.28:0/1395128542 shutdown_connections 2024-05-12T21:58:17.371 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:16.650+0000 7f3b9a200700 1 -- 172.21.15.28:0/1395128542 wait complete. 2024-05-12T21:58:17.841 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:58:17 smithi028 ceph-mon[29847]: from='mgr.14120 172.21.15.28:0/1562014889' entity='mgr.x' 2024-05-12T21:58:17.841 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:58:17 smithi028 ceph-mon[29847]: from='mgr.14120 172.21.15.28:0/1562014889' entity='mgr.x' 2024-05-12T21:58:18.893 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:58:18 smithi028 ceph-mon[29847]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-05-12T21:58:18.908 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC655sRXRKBWw7UC94HKv3vOnjQ7x3I+Fg+CKyLM4otPCS/L+e9W4IJ+bjby77xzjF16TkWqDxtzNBFU4bifqwzBytknKoSxS+G/lhWBRlxuPRPbf9gjAGsCDfmzZKOPdPxELNwnQS23//2j44YpShSdkCCabywLxPOLsEY0CAzzfakm7ts0Qh2tIDOBMeGpAN1jUe8cFu9M9rM/tCwE9Q7JoUHRC9BChJn+Ule+ifwrdDYIiLi0rg4Z6NIl8DxNJYHSPIZb+/VVJ/HfLb6OeCFkOr03342+QLVUVSlTPyX+2VHS05N0SGi/xa66y3uZWLMvp7oAOVULnV/ntWWkUW3JxBSRDHaUqAllPKfczHIDPm7mxL4TOOyTWf++BOUpOcwQnzK1ih7uKw5d/CIRk9dIWggg1Zg9njsQZ3qywyzTOXX98/DG8JZFCEL2iFcc77pCgKx6+QxRPIbkjUMf7q++cAftCCIz5BaEZpllXTwme31W19rE3mDItZz2GTgFlU= ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968 2024-05-12T21:58:18.909 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:18.007+0000 7f4c32c00700 1 Processor -- start 2024-05-12T21:58:18.909 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:18.008+0000 7f4c32c00700 1 -- start start 2024-05-12T21:58:18.909 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:18.008+0000 7f4c32c00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c2c109fb0 0x7f4c2c10a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T21:58:18.910 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:18.008+0000 7f4c32c00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4c2c10a9a0 con 0x7f4c2c109fb0 2024-05-12T21:58:18.911 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:18.009+0000 7f4c31800700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c2c109fb0 0x7f4c2c10a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T21:58:18.911 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:18.009+0000 7f4c31800700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c2c109fb0 0x7f4c2c10a3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38690/0 (socket says 172.21.15.28:38690) 2024-05-12T21:58:18.911 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:18.009+0000 7f4c31800700 1 -- 172.21.15.28:0/3135116644 learned_addr learned my addr 172.21.15.28:0/3135116644 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T21:58:18.911 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:18.009+0000 7f4c31800700 1 -- 172.21.15.28:0/3135116644 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c2c10b1b0 con 0x7f4c2c109fb0 2024-05-12T21:58:18.911 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:18.010+0000 7f4c31800700 1 --2- 172.21.15.28:0/3135116644 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c2c109fb0 0x7f4c2c10a3d0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f4c200098a0 tx=0x7f4c20009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0ea8333a4f110a4 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T21:58:18.911 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:18.010+0000 7f4c2be00700 1 -- 172.21.15.28:0/3135116644 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4c20004160 con 0x7f4c2c109fb0 2024-05-12T21:58:18.911 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:18.010+0000 7f4c2be00700 1 -- 172.21.15.28:0/3135116644 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f4c20004300 con 0x7f4c2c109fb0 2024-05-12T21:58:18.911 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:18.010+0000 7f4c2be00700 1 -- 172.21.15.28:0/3135116644 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4c20011b50 con 0x7f4c2c109fb0 2024-05-12T21:58:18.911 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:18.011+0000 7f4c32c00700 1 -- 172.21.15.28:0/3135116644 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c2c109fb0 msgr2=0x7f4c2c10a3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T21:58:18.911 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:18.011+0000 7f4c32c00700 1 --2- 172.21.15.28:0/3135116644 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c2c109fb0 0x7f4c2c10a3d0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f4c200098a0 tx=0x7f4c20009bb0 comp rx=0 tx=0).stop 2024-05-12T21:58:18.911 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:18.011+0000 7f4c32c00700 1 -- 172.21.15.28:0/3135116644 shutdown_connections 2024-05-12T21:58:18.911 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:18.011+0000 7f4c32c00700 1 --2- 172.21.15.28:0/3135116644 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c2c109fb0 0x7f4c2c10a3d0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:58:18.911 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:18.011+0000 7f4c32c00700 1 -- 172.21.15.28:0/3135116644 >> 172.21.15.28:0/3135116644 conn(0x7f4c2c07d380 msgr2=0x7f4c2c07d7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T21:58:18.915 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:18.011+0000 7f4c32c00700 1 -- 172.21.15.28:0/3135116644 shutdown_connections 2024-05-12T21:58:18.915 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:18.011+0000 7f4c32c00700 1 -- 172.21.15.28:0/3135116644 wait complete. 2024-05-12T21:58:18.915 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:18.012+0000 7f4c32c00700 1 Processor -- start 2024-05-12T21:58:18.916 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:18.012+0000 7f4c32c00700 1 -- start start 2024-05-12T21:58:18.916 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:18.013+0000 7f4c32c00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c2c109fb0 0x7f4c2c19dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T21:58:18.916 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:18.013+0000 7f4c32c00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4c2c10e560 con 0x7f4c2c109fb0 2024-05-12T21:58:18.916 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:18.013+0000 7f4c31800700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c2c109fb0 0x7f4c2c19dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T21:58:18.916 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:18.013+0000 7f4c31800700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c2c109fb0 0x7f4c2c19dff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38696/0 (socket says 172.21.15.28:38696) 2024-05-12T21:58:18.916 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:18.013+0000 7f4c31800700 1 -- 172.21.15.28:0/1378287903 learned_addr learned my addr 172.21.15.28:0/1378287903 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T21:58:18.916 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:18.013+0000 7f4c31800700 1 -- 172.21.15.28:0/1378287903 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c2c19e530 con 0x7f4c2c109fb0 2024-05-12T21:58:18.916 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:18.014+0000 7f4c31800700 1 --2- 172.21.15.28:0/1378287903 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c2c109fb0 0x7f4c2c19dff0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f4c20000c00 tx=0x7f4c20011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T21:58:18.916 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:18.014+0000 7f4c2a000700 1 -- 172.21.15.28:0/1378287903 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4c20003b20 con 0x7f4c2c109fb0 2024-05-12T21:58:18.916 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:18.014+0000 7f4c2a000700 1 -- 172.21.15.28:0/1378287903 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f4c20003cc0 con 0x7f4c2c109fb0 2024-05-12T21:58:18.916 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:18.014+0000 7f4c32c00700 1 -- 172.21.15.28:0/1378287903 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c2c19e7c0 con 0x7f4c2c109fb0 2024-05-12T21:58:18.916 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:18.014+0000 7f4c32c00700 1 -- 172.21.15.28:0/1378287903 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c2c1a14b0 con 0x7f4c2c109fb0 2024-05-12T21:58:18.916 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:18.014+0000 7f4c2a000700 1 -- 172.21.15.28:0/1378287903 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4c20024910 con 0x7f4c2c109fb0 2024-05-12T21:58:18.916 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:18.015+0000 7f4c32c00700 1 -- 172.21.15.28:0/1378287903 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c0c005360 con 0x7f4c2c109fb0 2024-05-12T21:58:18.916 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:18.016+0000 7f4c2a000700 1 -- 172.21.15.28:0/1378287903 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f4c20022020 con 0x7f4c2c109fb0 2024-05-12T21:58:18.917 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:18.016+0000 7f4c2a000700 1 --2- 172.21.15.28:0/1378287903 >> [v2:172.21.15.28:6800/3760499540,v1:172.21.15.28:6801/3760499540] conn(0x7f4c140390a0 0x7f4c1403b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T21:58:18.917 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:18.016+0000 7f4c2a000700 1 -- 172.21.15.28:0/1378287903 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f4c20011e00 con 0x7f4c2c109fb0 2024-05-12T21:58:18.917 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:18.017+0000 7f4c30e00700 1 --2- 172.21.15.28:0/1378287903 >> [v2:172.21.15.28:6800/3760499540,v1:172.21.15.28:6801/3760499540] conn(0x7f4c140390a0 0x7f4c1403b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T21:58:18.917 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:18.018+0000 7f4c30e00700 1 --2- 172.21.15.28:0/1378287903 >> [v2:172.21.15.28:6800/3760499540,v1:172.21.15.28:6801/3760499540] conn(0x7f4c140390a0 0x7f4c1403b560 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f4c18006fd0 tx=0x7f4c18006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T21:58:18.917 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:18.022+0000 7f4c2a000700 1 -- 172.21.15.28:0/1378287903 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f4c2004cbe0 con 0x7f4c2c109fb0 2024-05-12T21:58:18.917 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:18.251+0000 7f4c32c00700 1 -- 172.21.15.28:0/1378287903 --> [v2:172.21.15.28:6800/3760499540,v1:172.21.15.28:6801/3760499540] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) -- 0x7f4c0c000bf0 con 0x7f4c140390a0 2024-05-12T21:58:18.917 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:18.253+0000 7f4c2a000700 1 -- 172.21.15.28:0/1378287903 <== mgr.14120 v2:172.21.15.28:6800/3760499540 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+595 (secure 0 0 0) 0x7f4c0c000bf0 con 0x7f4c140390a0 2024-05-12T21:58:18.917 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:18.256+0000 7f4c32c00700 1 -- 172.21.15.28:0/1378287903 >> [v2:172.21.15.28:6800/3760499540,v1:172.21.15.28:6801/3760499540] conn(0x7f4c140390a0 msgr2=0x7f4c1403b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T21:58:18.917 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:18.256+0000 7f4c32c00700 1 --2- 172.21.15.28:0/1378287903 >> [v2:172.21.15.28:6800/3760499540,v1:172.21.15.28:6801/3760499540] conn(0x7f4c140390a0 0x7f4c1403b560 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f4c18006fd0 tx=0x7f4c18006e40 comp rx=0 tx=0).stop 2024-05-12T21:58:18.917 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:18.256+0000 7f4c32c00700 1 -- 172.21.15.28:0/1378287903 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c2c109fb0 msgr2=0x7f4c2c19dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T21:58:18.917 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:18.256+0000 7f4c32c00700 1 --2- 172.21.15.28:0/1378287903 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c2c109fb0 0x7f4c2c19dff0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f4c20000c00 tx=0x7f4c20011c10 comp rx=0 tx=0).stop 2024-05-12T21:58:18.917 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:18.256+0000 7f4c32c00700 1 -- 172.21.15.28:0/1378287903 shutdown_connections 2024-05-12T21:58:18.917 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:18.256+0000 7f4c32c00700 1 --2- 172.21.15.28:0/1378287903 >> [v2:172.21.15.28:6800/3760499540,v1:172.21.15.28:6801/3760499540] conn(0x7f4c140390a0 0x7f4c1403b560 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:58:18.917 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:18.256+0000 7f4c32c00700 1 --2- 172.21.15.28:0/1378287903 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c2c109fb0 0x7f4c2c19dff0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:58:18.917 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:18.256+0000 7f4c32c00700 1 -- 172.21.15.28:0/1378287903 >> 172.21.15.28:0/1378287903 conn(0x7f4c2c07d380 msgr2=0x7f4c2c107530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T21:58:18.918 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:18.257+0000 7f4c32c00700 1 -- 172.21.15.28:0/1378287903 shutdown_connections 2024-05-12T21:58:18.918 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:18.257+0000 7f4c32c00700 1 -- 172.21.15.28:0/1378287903 wait complete. 2024-05-12T21:58:18.918 INFO:teuthology.orchestra.run.smithi028.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-05-12T21:58:18.918 INFO:teuthology.orchestra.run.smithi028.stdout:Adding key to root@localhost authorized_keys... 2024-05-12T21:58:18.918 INFO:teuthology.orchestra.run.smithi028.stdout:Adding host smithi028... 2024-05-12T21:58:20.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:58:20 smithi028 ceph-mon[29847]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi028", "addr": "172.21.15.28", "target": ["mon-mgr", ""]}]: dispatch 2024-05-12T21:58:21.498 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:58:21 smithi028 ceph-mon[29847]: Deploying cephadm binary to smithi028 2024-05-12T21:58:26.835 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout Added host 'smithi028' with addr '172.21.15.28' 2024-05-12T21:58:26.835 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:19.548+0000 7f4c63a00700 1 Processor -- start 2024-05-12T21:58:26.835 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:19.548+0000 7f4c63a00700 1 -- start start 2024-05-12T21:58:26.835 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:19.549+0000 7f4c63a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c5c107f70 0x7f4c5c108390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T21:58:26.835 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:19.549+0000 7f4c63a00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4c5c108960 con 0x7f4c5c107f70 2024-05-12T21:58:26.835 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:19.549+0000 7f4c61000700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c5c107f70 0x7f4c5c108390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T21:58:26.835 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:19.549+0000 7f4c61000700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c5c107f70 0x7f4c5c108390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38706/0 (socket says 172.21.15.28:38706) 2024-05-12T21:58:26.835 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:19.549+0000 7f4c61000700 1 -- 172.21.15.28:0/401459051 learned_addr learned my addr 172.21.15.28:0/401459051 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T21:58:26.835 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:19.550+0000 7f4c61000700 1 -- 172.21.15.28:0/401459051 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c5c109170 con 0x7f4c5c107f70 2024-05-12T21:58:26.836 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:19.550+0000 7f4c61000700 1 --2- 172.21.15.28:0/401459051 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c5c107f70 0x7f4c5c108390 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f4c500098a0 tx=0x7f4c50009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4160a5b35af4b66 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T21:58:26.836 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:19.550+0000 7f4c5b400700 1 -- 172.21.15.28:0/401459051 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4c50004160 con 0x7f4c5c107f70 2024-05-12T21:58:26.836 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:19.551+0000 7f4c5b400700 1 -- 172.21.15.28:0/401459051 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f4c50004300 con 0x7f4c5c107f70 2024-05-12T21:58:26.836 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:19.551+0000 7f4c5b400700 1 -- 172.21.15.28:0/401459051 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4c50011b50 con 0x7f4c5c107f70 2024-05-12T21:58:26.836 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:19.551+0000 7f4c63a00700 1 -- 172.21.15.28:0/401459051 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c5c107f70 msgr2=0x7f4c5c108390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T21:58:26.836 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:19.551+0000 7f4c63a00700 1 --2- 172.21.15.28:0/401459051 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c5c107f70 0x7f4c5c108390 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f4c500098a0 tx=0x7f4c50009bb0 comp rx=0 tx=0).stop 2024-05-12T21:58:26.836 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:19.551+0000 7f4c63a00700 1 -- 172.21.15.28:0/401459051 shutdown_connections 2024-05-12T21:58:26.836 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:19.551+0000 7f4c63a00700 1 --2- 172.21.15.28:0/401459051 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c5c107f70 0x7f4c5c108390 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:58:26.836 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:19.551+0000 7f4c63a00700 1 -- 172.21.15.28:0/401459051 >> 172.21.15.28:0/401459051 conn(0x7f4c5c103530 msgr2=0x7f4c5c105950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T21:58:26.836 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:19.552+0000 7f4c63a00700 1 -- 172.21.15.28:0/401459051 shutdown_connections 2024-05-12T21:58:26.836 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:19.552+0000 7f4c63a00700 1 -- 172.21.15.28:0/401459051 wait complete. 2024-05-12T21:58:26.836 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:19.553+0000 7f4c63a00700 1 Processor -- start 2024-05-12T21:58:26.836 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:19.553+0000 7f4c63a00700 1 -- start start 2024-05-12T21:58:26.836 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:19.553+0000 7f4c63a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c5c107f70 0x7f4c5c07c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T21:58:26.836 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:19.553+0000 7f4c63a00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4c5c10c520 con 0x7f4c5c107f70 2024-05-12T21:58:26.837 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:19.553+0000 7f4c61000700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c5c107f70 0x7f4c5c07c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T21:58:26.837 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:19.553+0000 7f4c61000700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c5c107f70 0x7f4c5c07c9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38722/0 (socket says 172.21.15.28:38722) 2024-05-12T21:58:26.837 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:19.554+0000 7f4c61000700 1 -- 172.21.15.28:0/1623708288 learned_addr learned my addr 172.21.15.28:0/1623708288 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T21:58:26.837 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:19.554+0000 7f4c61000700 1 -- 172.21.15.28:0/1623708288 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c5c07cf30 con 0x7f4c5c107f70 2024-05-12T21:58:26.837 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:19.554+0000 7f4c61000700 1 --2- 172.21.15.28:0/1623708288 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c5c107f70 0x7f4c5c07c9f0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f4c50009fb0 tx=0x7f4c50011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T21:58:26.837 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:19.554+0000 7f4c59600700 1 -- 172.21.15.28:0/1623708288 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4c500038b0 con 0x7f4c5c107f70 2024-05-12T21:58:26.837 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:19.555+0000 7f4c59600700 1 -- 172.21.15.28:0/1623708288 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f4c50011e50 con 0x7f4c5c107f70 2024-05-12T21:58:26.837 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:19.555+0000 7f4c63a00700 1 -- 172.21.15.28:0/1623708288 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c5c07b120 con 0x7f4c5c107f70 2024-05-12T21:58:26.837 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:19.555+0000 7f4c59600700 1 -- 172.21.15.28:0/1623708288 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4c50019550 con 0x7f4c5c107f70 2024-05-12T21:58:26.837 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:19.555+0000 7f4c63a00700 1 -- 172.21.15.28:0/1623708288 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c5c07b660 con 0x7f4c5c107f70 2024-05-12T21:58:26.837 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:19.556+0000 7f4c63a00700 1 -- 172.21.15.28:0/1623708288 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c5c063980 con 0x7f4c5c107f70 2024-05-12T21:58:26.837 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:19.556+0000 7f4c59600700 1 -- 172.21.15.28:0/1623708288 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f4c50003a50 con 0x7f4c5c107f70 2024-05-12T21:58:26.837 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:19.556+0000 7f4c59600700 1 --2- 172.21.15.28:0/1623708288 >> [v2:172.21.15.28:6800/3760499540,v1:172.21.15.28:6801/3760499540] conn(0x7f4c44039050 0x7f4c4403b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T21:58:26.837 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:19.556+0000 7f4c59600700 1 -- 172.21.15.28:0/1623708288 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f4c5004d2c0 con 0x7f4c5c107f70 2024-05-12T21:58:26.838 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:19.557+0000 7f4c5be00700 1 --2- 172.21.15.28:0/1623708288 >> [v2:172.21.15.28:6800/3760499540,v1:172.21.15.28:6801/3760499540] conn(0x7f4c44039050 0x7f4c4403b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T21:58:26.838 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:19.557+0000 7f4c5be00700 1 --2- 172.21.15.28:0/1623708288 >> [v2:172.21.15.28:6800/3760499540,v1:172.21.15.28:6801/3760499540] conn(0x7f4c44039050 0x7f4c4403b510 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c006fd0 tx=0x7f4c4c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T21:58:26.838 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:19.562+0000 7f4c59600700 1 -- 172.21.15.28:0/1623708288 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f4c5000fda0 con 0x7f4c5c107f70 2024-05-12T21:58:26.838 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:19.796+0000 7f4c63a00700 1 -- 172.21.15.28:0/1623708288 --> [v2:172.21.15.28:6800/3760499540,v1:172.21.15.28:6801/3760499540] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi028", "addr": "172.21.15.28", "target": ["mon-mgr", ""]}) -- 0x7f4c5c104d40 con 0x7f4c44039050 2024-05-12T21:58:26.838 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:26.030+0000 7f4c59600700 1 -- 172.21.15.28:0/1623708288 <== mgr.14120 v2:172.21.15.28:6800/3760499540 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7f4c5c104d40 con 0x7f4c44039050 2024-05-12T21:58:26.838 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:26.033+0000 7f4c63a00700 1 -- 172.21.15.28:0/1623708288 >> [v2:172.21.15.28:6800/3760499540,v1:172.21.15.28:6801/3760499540] conn(0x7f4c44039050 msgr2=0x7f4c4403b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T21:58:26.838 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:26.034+0000 7f4c63a00700 1 --2- 172.21.15.28:0/1623708288 >> [v2:172.21.15.28:6800/3760499540,v1:172.21.15.28:6801/3760499540] conn(0x7f4c44039050 0x7f4c4403b510 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c006fd0 tx=0x7f4c4c006e40 comp rx=0 tx=0).stop 2024-05-12T21:58:26.838 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:26.034+0000 7f4c63a00700 1 -- 172.21.15.28:0/1623708288 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c5c107f70 msgr2=0x7f4c5c07c9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T21:58:26.838 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:26.034+0000 7f4c63a00700 1 --2- 172.21.15.28:0/1623708288 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c5c107f70 0x7f4c5c07c9f0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f4c50009fb0 tx=0x7f4c50011bf0 comp rx=0 tx=0).stop 2024-05-12T21:58:26.838 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:26.034+0000 7f4c63a00700 1 -- 172.21.15.28:0/1623708288 shutdown_connections 2024-05-12T21:58:26.838 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:26.034+0000 7f4c63a00700 1 --2- 172.21.15.28:0/1623708288 >> [v2:172.21.15.28:6800/3760499540,v1:172.21.15.28:6801/3760499540] conn(0x7f4c44039050 0x7f4c4403b510 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:58:26.838 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:26.034+0000 7f4c63a00700 1 --2- 172.21.15.28:0/1623708288 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c5c107f70 0x7f4c5c07c9f0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:58:26.838 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:26.034+0000 7f4c63a00700 1 -- 172.21.15.28:0/1623708288 >> 172.21.15.28:0/1623708288 conn(0x7f4c5c103530 msgr2=0x7f4c5c104110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T21:58:26.838 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:26.034+0000 7f4c63a00700 1 -- 172.21.15.28:0/1623708288 shutdown_connections 2024-05-12T21:58:26.838 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:26.034+0000 7f4c63a00700 1 -- 172.21.15.28:0/1623708288 wait complete. 2024-05-12T21:58:26.839 INFO:teuthology.orchestra.run.smithi028.stdout:Deploying unmanaged mon service... 2024-05-12T21:58:27.113 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:58:27 smithi028 ceph-mon[29847]: from='mgr.14120 172.21.15.28:0/1562014889' entity='mgr.x' 2024-05-12T21:58:27.113 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:58:27 smithi028 ceph-mon[29847]: Added host smithi028 2024-05-12T21:58:27.113 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:58:27 smithi028 ceph-mon[29847]: from='mgr.14120 172.21.15.28:0/1562014889' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T21:58:28.469 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-05-12T21:58:28.469 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:27.446+0000 7fbabbe00700 1 Processor -- start 2024-05-12T21:58:28.469 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:27.447+0000 7fbabbe00700 1 -- start start 2024-05-12T21:58:28.469 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:27.447+0000 7fbabbe00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbabc10a010 0x7fbabc10a430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T21:58:28.469 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:27.447+0000 7fbabbe00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbabc10aa00 con 0x7fbabc10a010 2024-05-12T21:58:28.469 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:27.448+0000 7fbabaa00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbabc10a010 0x7fbabc10a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T21:58:28.469 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:27.448+0000 7fbabaa00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbabc10a010 0x7fbabc10a430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40772/0 (socket says 172.21.15.28:40772) 2024-05-12T21:58:28.469 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:27.448+0000 7fbabaa00700 1 -- 172.21.15.28:0/1855844989 learned_addr learned my addr 172.21.15.28:0/1855844989 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T21:58:28.470 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:27.448+0000 7fbabaa00700 1 -- 172.21.15.28:0/1855844989 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbabc10b1c0 con 0x7fbabc10a010 2024-05-12T21:58:28.470 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:27.449+0000 7fbabaa00700 1 --2- 172.21.15.28:0/1855844989 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbabc10a010 0x7fbabc10a430 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fbaac009e80 tx=0x7fbaac00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65a527966500e3dd server_cookie=0 in_seq=0 out_seq=0 2024-05-12T21:58:28.470 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:27.449+0000 7fbaba000700 1 -- 172.21.15.28:0/1855844989 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbaac017070 con 0x7fbabc10a010 2024-05-12T21:58:28.470 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:27.449+0000 7fbaba000700 1 -- 172.21.15.28:0/1855844989 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fbaac00b8c0 con 0x7fbabc10a010 2024-05-12T21:58:28.470 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:27.449+0000 7fbaba000700 1 -- 172.21.15.28:0/1855844989 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbaac0045c0 con 0x7fbabc10a010 2024-05-12T21:58:28.470 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:27.450+0000 7fbabbe00700 1 -- 172.21.15.28:0/1855844989 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbabc10a010 msgr2=0x7fbabc10a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T21:58:28.470 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:27.450+0000 7fbabbe00700 1 --2- 172.21.15.28:0/1855844989 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbabc10a010 0x7fbabc10a430 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fbaac009e80 tx=0x7fbaac00b330 comp rx=0 tx=0).stop 2024-05-12T21:58:28.470 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:27.450+0000 7fbabbe00700 1 -- 172.21.15.28:0/1855844989 shutdown_connections 2024-05-12T21:58:28.470 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:27.450+0000 7fbabbe00700 1 --2- 172.21.15.28:0/1855844989 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbabc10a010 0x7fbabc10a430 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:58:28.470 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:27.450+0000 7fbabbe00700 1 -- 172.21.15.28:0/1855844989 >> 172.21.15.28:0/1855844989 conn(0x7fbabc07d3d0 msgr2=0x7fbabc07d800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T21:58:28.470 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:27.450+0000 7fbabbe00700 1 -- 172.21.15.28:0/1855844989 shutdown_connections 2024-05-12T21:58:28.470 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:27.450+0000 7fbabbe00700 1 -- 172.21.15.28:0/1855844989 wait complete. 2024-05-12T21:58:28.470 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:27.451+0000 7fbabbe00700 1 Processor -- start 2024-05-12T21:58:28.474 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:27.451+0000 7fbabbe00700 1 -- start start 2024-05-12T21:58:28.474 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:27.452+0000 7fbabbe00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbabc10a010 0x7fbabc19dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T21:58:28.474 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:27.452+0000 7fbabbe00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbabc10e570 con 0x7fbabc10a010 2024-05-12T21:58:28.474 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:27.452+0000 7fbabaa00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbabc10a010 0x7fbabc19dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T21:58:28.474 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:27.452+0000 7fbabaa00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbabc10a010 0x7fbabc19dfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40780/0 (socket says 172.21.15.28:40780) 2024-05-12T21:58:28.474 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:27.452+0000 7fbabaa00700 1 -- 172.21.15.28:0/2746434857 learned_addr learned my addr 172.21.15.28:0/2746434857 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T21:58:28.474 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:27.452+0000 7fbabaa00700 1 -- 172.21.15.28:0/2746434857 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbabc19e520 con 0x7fbabc10a010 2024-05-12T21:58:28.474 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:27.453+0000 7fbabaa00700 1 --2- 172.21.15.28:0/2746434857 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbabc10a010 0x7fbabc19dfe0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fbaac004210 tx=0x7fbaac00bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T21:58:28.474 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:27.453+0000 7fbab3e00700 1 -- 172.21.15.28:0/2746434857 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbaac020020 con 0x7fbabc10a010 2024-05-12T21:58:28.474 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:27.453+0000 7fbab3e00700 1 -- 172.21.15.28:0/2746434857 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fbaac003b50 con 0x7fbabc10a010 2024-05-12T21:58:28.475 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:27.453+0000 7fbabbe00700 1 -- 172.21.15.28:0/2746434857 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbabc19e7b0 con 0x7fbabc10a010 2024-05-12T21:58:28.475 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:27.453+0000 7fbab3e00700 1 -- 172.21.15.28:0/2746434857 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbaac017040 con 0x7fbabc10a010 2024-05-12T21:58:28.475 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:27.453+0000 7fbabbe00700 1 -- 172.21.15.28:0/2746434857 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbabc1a14a0 con 0x7fbabc10a010 2024-05-12T21:58:28.475 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:27.454+0000 7fbab3e00700 1 -- 172.21.15.28:0/2746434857 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7fbaac023020 con 0x7fbabc10a010 2024-05-12T21:58:28.475 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:27.454+0000 7fbabbe00700 1 -- 172.21.15.28:0/2746434857 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbabc0694a0 con 0x7fbabc10a010 2024-05-12T21:58:28.475 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:27.455+0000 7fbab3e00700 1 --2- 172.21.15.28:0/2746434857 >> [v2:172.21.15.28:6800/3760499540,v1:172.21.15.28:6801/3760499540] conn(0x7fbaa00390a0 0x7fbaa003b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T21:58:28.475 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:27.455+0000 7fbab3e00700 1 -- 172.21.15.28:0/2746434857 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fbaac00f050 con 0x7fbabc10a010 2024-05-12T21:58:28.475 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:27.455+0000 7fbab2000700 1 --2- 172.21.15.28:0/2746434857 >> [v2:172.21.15.28:6800/3760499540,v1:172.21.15.28:6801/3760499540] conn(0x7fbaa00390a0 0x7fbaa003b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T21:58:28.475 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:27.456+0000 7fbab2000700 1 --2- 172.21.15.28:0/2746434857 >> [v2:172.21.15.28:6800/3760499540,v1:172.21.15.28:6801/3760499540] conn(0x7fbaa00390a0 0x7fbaa003b560 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fbaa4006fd0 tx=0x7fbaa4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T21:58:28.475 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:27.460+0000 7fbab3e00700 1 -- 172.21.15.28:0/2746434857 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fbaac02a020 con 0x7fbabc10a010 2024-05-12T21:58:28.475 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:27.711+0000 7fbabbe00700 1 -- 172.21.15.28:0/2746434857 --> [v2:172.21.15.28:6800/3760499540,v1:172.21.15.28:6801/3760499540] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7fbabc107bb0 con 0x7fbaa00390a0 2024-05-12T21:58:28.475 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:27.717+0000 7fbab3e00700 1 -- 172.21.15.28:0/2746434857 <== mgr.14120 v2:172.21.15.28:6800/3760499540 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fbabc107bb0 con 0x7fbaa00390a0 2024-05-12T21:58:28.475 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:27.720+0000 7fbabbe00700 1 -- 172.21.15.28:0/2746434857 >> [v2:172.21.15.28:6800/3760499540,v1:172.21.15.28:6801/3760499540] conn(0x7fbaa00390a0 msgr2=0x7fbaa003b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T21:58:28.475 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:27.720+0000 7fbabbe00700 1 --2- 172.21.15.28:0/2746434857 >> [v2:172.21.15.28:6800/3760499540,v1:172.21.15.28:6801/3760499540] conn(0x7fbaa00390a0 0x7fbaa003b560 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fbaa4006fd0 tx=0x7fbaa4006e40 comp rx=0 tx=0).stop 2024-05-12T21:58:28.475 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:27.720+0000 7fbabbe00700 1 -- 172.21.15.28:0/2746434857 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbabc10a010 msgr2=0x7fbabc19dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T21:58:28.476 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:27.720+0000 7fbabbe00700 1 --2- 172.21.15.28:0/2746434857 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbabc10a010 0x7fbabc19dfe0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fbaac004210 tx=0x7fbaac00bd50 comp rx=0 tx=0).stop 2024-05-12T21:58:28.476 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:27.720+0000 7fbabbe00700 1 -- 172.21.15.28:0/2746434857 shutdown_connections 2024-05-12T21:58:28.476 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:27.720+0000 7fbabbe00700 1 --2- 172.21.15.28:0/2746434857 >> [v2:172.21.15.28:6800/3760499540,v1:172.21.15.28:6801/3760499540] conn(0x7fbaa00390a0 0x7fbaa003b560 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:58:28.476 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:27.720+0000 7fbabbe00700 1 --2- 172.21.15.28:0/2746434857 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbabc10a010 0x7fbabc19dfe0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:58:28.476 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:27.720+0000 7fbabbe00700 1 -- 172.21.15.28:0/2746434857 >> 172.21.15.28:0/2746434857 conn(0x7fbabc07d3d0 msgr2=0x7fbabc1074a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T21:58:28.476 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:27.720+0000 7fbabbe00700 1 -- 172.21.15.28:0/2746434857 shutdown_connections 2024-05-12T21:58:28.476 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:27.720+0000 7fbabbe00700 1 -- 172.21.15.28:0/2746434857 wait complete. 2024-05-12T21:58:28.476 INFO:teuthology.orchestra.run.smithi028.stdout:Deploying unmanaged mgr service... 2024-05-12T21:58:28.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:58:28 smithi028 ceph-mon[29847]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-12T21:58:28.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:58:28 smithi028 ceph-mon[29847]: Saving service mon spec with placement count:5 2024-05-12T21:58:28.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:58:28 smithi028 ceph-mon[29847]: from='mgr.14120 172.21.15.28:0/1562014889' entity='mgr.x' 2024-05-12T21:58:30.970 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:58:30 smithi028 ceph-mon[29847]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-12T21:58:30.970 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:58:30 smithi028 ceph-mon[29847]: Saving service mgr spec with placement count:2 2024-05-12T21:58:30.971 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:58:30 smithi028 ceph-mon[29847]: from='mgr.14120 172.21.15.28:0/1562014889' entity='mgr.x' 2024-05-12T21:58:30.971 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:58:30 smithi028 ceph-mon[29847]: from='mgr.14120 172.21.15.28:0/1562014889' entity='mgr.x' 2024-05-12T21:58:30.988 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-05-12T21:58:30.989 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:29.441+0000 7f1c20a00700 1 Processor -- start 2024-05-12T21:58:30.989 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:29.442+0000 7f1c20a00700 1 -- start start 2024-05-12T21:58:30.989 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:29.442+0000 7f1c20a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c1c10ee30 0x7f1c1c111220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T21:58:30.989 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:29.442+0000 7f1c20a00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1c1c111760 con 0x7f1c1c10ee30 2024-05-12T21:58:30.989 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:29.443+0000 7f1c19e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c1c10ee30 0x7f1c1c111220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T21:58:30.989 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:29.443+0000 7f1c19e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c1c10ee30 0x7f1c1c111220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40782/0 (socket says 172.21.15.28:40782) 2024-05-12T21:58:30.989 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:29.443+0000 7f1c19e00700 1 -- 172.21.15.28:0/2324153223 learned_addr learned my addr 172.21.15.28:0/2324153223 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T21:58:30.989 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:29.443+0000 7f1c19e00700 1 -- 172.21.15.28:0/2324153223 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c1c1118e0 con 0x7f1c1c10ee30 2024-05-12T21:58:30.989 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:29.443+0000 7f1c19e00700 1 --2- 172.21.15.28:0/2324153223 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c1c10ee30 0x7f1c1c111220 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f1c0c0098a0 tx=0x7f1c0c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95c045f72d644532 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T21:58:30.989 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:29.444+0000 7f1c18a00700 1 -- 172.21.15.28:0/2324153223 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1c0c004160 con 0x7f1c1c10ee30 2024-05-12T21:58:30.990 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:29.444+0000 7f1c18a00700 1 -- 172.21.15.28:0/2324153223 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f1c0c004300 con 0x7f1c1c10ee30 2024-05-12T21:58:30.990 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:29.444+0000 7f1c18a00700 1 -- 172.21.15.28:0/2324153223 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1c0c011b50 con 0x7f1c1c10ee30 2024-05-12T21:58:30.990 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:29.444+0000 7f1c20a00700 1 -- 172.21.15.28:0/2324153223 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c1c10ee30 msgr2=0x7f1c1c111220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T21:58:30.990 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:29.444+0000 7f1c20a00700 1 --2- 172.21.15.28:0/2324153223 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c1c10ee30 0x7f1c1c111220 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f1c0c0098a0 tx=0x7f1c0c009bb0 comp rx=0 tx=0).stop 2024-05-12T21:58:30.990 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:29.444+0000 7f1c20a00700 1 -- 172.21.15.28:0/2324153223 shutdown_connections 2024-05-12T21:58:30.990 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:29.444+0000 7f1c20a00700 1 --2- 172.21.15.28:0/2324153223 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c1c10ee30 0x7f1c1c111220 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:58:30.990 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:29.444+0000 7f1c20a00700 1 -- 172.21.15.28:0/2324153223 >> 172.21.15.28:0/2324153223 conn(0x7f1c1c07d6a0 msgr2=0x7f1c1c07fac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T21:58:30.990 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:29.445+0000 7f1c20a00700 1 -- 172.21.15.28:0/2324153223 shutdown_connections 2024-05-12T21:58:30.990 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:29.445+0000 7f1c20a00700 1 -- 172.21.15.28:0/2324153223 wait complete. 2024-05-12T21:58:30.990 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:29.446+0000 7f1c20a00700 1 Processor -- start 2024-05-12T21:58:30.990 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:29.446+0000 7f1c20a00700 1 -- start start 2024-05-12T21:58:30.990 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:29.446+0000 7f1c20a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c1c10ee30 0x7f1c1c1a2530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T21:58:30.990 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:29.446+0000 7f1c20a00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1c1c112960 con 0x7f1c1c10ee30 2024-05-12T21:58:30.991 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:29.447+0000 7f1c19e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c1c10ee30 0x7f1c1c1a2530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T21:58:30.991 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:29.447+0000 7f1c19e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c1c10ee30 0x7f1c1c1a2530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40790/0 (socket says 172.21.15.28:40790) 2024-05-12T21:58:30.991 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:29.447+0000 7f1c19e00700 1 -- 172.21.15.28:0/3449508692 learned_addr learned my addr 172.21.15.28:0/3449508692 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T21:58:30.991 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:29.447+0000 7f1c19e00700 1 -- 172.21.15.28:0/3449508692 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c1c1a2a70 con 0x7f1c1c10ee30 2024-05-12T21:58:30.991 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:29.447+0000 7f1c19e00700 1 --2- 172.21.15.28:0/3449508692 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c1c10ee30 0x7f1c1c1a2530 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f1c0c011b60 tx=0x7f1c0c011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T21:58:30.991 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:29.448+0000 7f1c12a00700 1 -- 172.21.15.28:0/3449508692 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1c0c003970 con 0x7f1c1c10ee30 2024-05-12T21:58:30.991 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:29.448+0000 7f1c12a00700 1 -- 172.21.15.28:0/3449508692 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f1c0c003b10 con 0x7f1c1c10ee30 2024-05-12T21:58:30.991 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:29.448+0000 7f1c12a00700 1 -- 172.21.15.28:0/3449508692 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1c0c024610 con 0x7f1c1c10ee30 2024-05-12T21:58:30.991 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:29.448+0000 7f1c20a00700 1 -- 172.21.15.28:0/3449508692 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c1c1a2d00 con 0x7f1c1c10ee30 2024-05-12T21:58:30.991 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:29.448+0000 7f1c20a00700 1 -- 172.21.15.28:0/3449508692 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c1c1a59f0 con 0x7f1c1c10ee30 2024-05-12T21:58:30.991 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:29.449+0000 7f1c12a00700 1 -- 172.21.15.28:0/3449508692 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f1c0c01a9f0 con 0x7f1c1c10ee30 2024-05-12T21:58:30.991 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:29.449+0000 7f1c20a00700 1 -- 172.21.15.28:0/3449508692 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c1c063980 con 0x7f1c1c10ee30 2024-05-12T21:58:30.992 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:29.449+0000 7f1c12a00700 1 --2- 172.21.15.28:0/3449508692 >> [v2:172.21.15.28:6800/3760499540,v1:172.21.15.28:6801/3760499540] conn(0x7f1c000390a0 0x7f1c0003b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T21:58:30.992 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:29.450+0000 7f1c12a00700 1 -- 172.21.15.28:0/3449508692 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f1c0c011e00 con 0x7f1c1c10ee30 2024-05-12T21:58:30.992 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:29.450+0000 7f1c19400700 1 --2- 172.21.15.28:0/3449508692 >> [v2:172.21.15.28:6800/3760499540,v1:172.21.15.28:6801/3760499540] conn(0x7f1c000390a0 0x7f1c0003b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T21:58:30.992 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:29.450+0000 7f1c19400700 1 --2- 172.21.15.28:0/3449508692 >> [v2:172.21.15.28:6800/3760499540,v1:172.21.15.28:6801/3760499540] conn(0x7f1c000390a0 0x7f1c0003b560 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f1c08006fd0 tx=0x7f1c08006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T21:58:30.992 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:29.455+0000 7f1c12a00700 1 -- 172.21.15.28:0/3449508692 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f1c0c016260 con 0x7f1c1c10ee30 2024-05-12T21:58:30.992 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:29.700+0000 7f1c20a00700 1 -- 172.21.15.28:0/3449508692 --> [v2:172.21.15.28:6800/3760499540,v1:172.21.15.28:6801/3760499540] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f1c1c07f9c0 con 0x7f1c000390a0 2024-05-12T21:58:30.992 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:29.708+0000 7f1c12a00700 1 -- 172.21.15.28:0/3449508692 <== mgr.14120 v2:172.21.15.28:6800/3760499540 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f1c1c07f9c0 con 0x7f1c000390a0 2024-05-12T21:58:30.992 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:29.711+0000 7f1c20a00700 1 -- 172.21.15.28:0/3449508692 >> [v2:172.21.15.28:6800/3760499540,v1:172.21.15.28:6801/3760499540] conn(0x7f1c000390a0 msgr2=0x7f1c0003b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T21:58:30.992 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:29.711+0000 7f1c20a00700 1 --2- 172.21.15.28:0/3449508692 >> [v2:172.21.15.28:6800/3760499540,v1:172.21.15.28:6801/3760499540] conn(0x7f1c000390a0 0x7f1c0003b560 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f1c08006fd0 tx=0x7f1c08006e40 comp rx=0 tx=0).stop 2024-05-12T21:58:30.992 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:29.711+0000 7f1c20a00700 1 -- 172.21.15.28:0/3449508692 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c1c10ee30 msgr2=0x7f1c1c1a2530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T21:58:30.992 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:29.711+0000 7f1c20a00700 1 --2- 172.21.15.28:0/3449508692 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c1c10ee30 0x7f1c1c1a2530 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f1c0c011b60 tx=0x7f1c0c011c40 comp rx=0 tx=0).stop 2024-05-12T21:58:30.992 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:29.711+0000 7f1c20a00700 1 -- 172.21.15.28:0/3449508692 shutdown_connections 2024-05-12T21:58:30.992 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:29.711+0000 7f1c20a00700 1 --2- 172.21.15.28:0/3449508692 >> [v2:172.21.15.28:6800/3760499540,v1:172.21.15.28:6801/3760499540] conn(0x7f1c000390a0 0x7f1c0003b560 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:58:30.992 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:29.711+0000 7f1c20a00700 1 --2- 172.21.15.28:0/3449508692 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c1c10ee30 0x7f1c1c1a2530 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:58:30.993 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:29.711+0000 7f1c20a00700 1 -- 172.21.15.28:0/3449508692 >> 172.21.15.28:0/3449508692 conn(0x7f1c1c07d6a0 msgr2=0x7f1c1c07f2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T21:58:30.993 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:29.711+0000 7f1c20a00700 1 -- 172.21.15.28:0/3449508692 shutdown_connections 2024-05-12T21:58:30.993 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:29.711+0000 7f1c20a00700 1 -- 172.21.15.28:0/3449508692 wait complete. 2024-05-12T21:58:31.799 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:58:31 smithi028 ceph-mon[29847]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T21:58:31.799 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:58:31 smithi028 ceph-mon[29847]: from='mgr.14120 172.21.15.28:0/1562014889' entity='mgr.x' 2024-05-12T21:58:32.412 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:31.546+0000 7f15b7200700 1 Processor -- start 2024-05-12T21:58:32.412 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:31.546+0000 7f15b7200700 1 -- start start 2024-05-12T21:58:32.412 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:31.546+0000 7f15b7200700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15b0108a20 0x7f15b010ae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T21:58:32.413 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:31.546+0000 7f15b7200700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f15b00635e0 con 0x7f15b0108a20 2024-05-12T21:58:32.413 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:31.547+0000 7f15afe00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15b0108a20 0x7f15b010ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T21:58:32.413 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:31.547+0000 7f15afe00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15b0108a20 0x7f15b010ae10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40796/0 (socket says 172.21.15.28:40796) 2024-05-12T21:58:32.413 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:31.547+0000 7f15afe00700 1 -- 172.21.15.28:0/1960034213 learned_addr learned my addr 172.21.15.28:0/1960034213 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T21:58:32.414 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:31.547+0000 7f15afe00700 1 -- 172.21.15.28:0/1960034213 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15b010b350 con 0x7f15b0108a20 2024-05-12T21:58:32.414 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:31.548+0000 7f15afe00700 1 --2- 172.21.15.28:0/1960034213 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15b0108a20 0x7f15b010ae10 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f15a00098a0 tx=0x7f15a0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f93f8b8310c62eab server_cookie=0 in_seq=0 out_seq=0 2024-05-12T21:58:32.415 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:31.548+0000 7f15af400700 1 -- 172.21.15.28:0/1960034213 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f15a0004160 con 0x7f15b0108a20 2024-05-12T21:58:32.415 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:31.548+0000 7f15af400700 1 -- 172.21.15.28:0/1960034213 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f15a0004300 con 0x7f15b0108a20 2024-05-12T21:58:32.415 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:31.548+0000 7f15af400700 1 -- 172.21.15.28:0/1960034213 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f15a0011b20 con 0x7f15b0108a20 2024-05-12T21:58:32.415 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:31.549+0000 7f15b7200700 1 -- 172.21.15.28:0/1960034213 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15b0108a20 msgr2=0x7f15b010ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T21:58:32.415 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:31.549+0000 7f15b7200700 1 --2- 172.21.15.28:0/1960034213 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15b0108a20 0x7f15b010ae10 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f15a00098a0 tx=0x7f15a0009bb0 comp rx=0 tx=0).stop 2024-05-12T21:58:32.416 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:31.549+0000 7f15b7200700 1 -- 172.21.15.28:0/1960034213 shutdown_connections 2024-05-12T21:58:32.416 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:31.549+0000 7f15b7200700 1 --2- 172.21.15.28:0/1960034213 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15b0108a20 0x7f15b010ae10 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:58:32.417 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:31.549+0000 7f15b7200700 1 -- 172.21.15.28:0/1960034213 >> 172.21.15.28:0/1960034213 conn(0x7f15b0102410 msgr2=0x7f15b0104830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T21:58:32.417 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:31.549+0000 7f15b7200700 1 -- 172.21.15.28:0/1960034213 shutdown_connections 2024-05-12T21:58:32.417 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:31.549+0000 7f15b7200700 1 -- 172.21.15.28:0/1960034213 wait complete. 2024-05-12T21:58:32.417 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:31.550+0000 7f15b7200700 1 Processor -- start 2024-05-12T21:58:32.417 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:31.550+0000 7f15b7200700 1 -- start start 2024-05-12T21:58:32.417 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:31.551+0000 7f15b7200700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15b0108a20 0x7f15b019e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T21:58:32.418 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:31.551+0000 7f15b7200700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f15b010c360 con 0x7f15b0108a20 2024-05-12T21:58:32.418 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:31.551+0000 7f15afe00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15b0108a20 0x7f15b019e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T21:58:32.418 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:31.551+0000 7f15afe00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15b0108a20 0x7f15b019e060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40798/0 (socket says 172.21.15.28:40798) 2024-05-12T21:58:32.418 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:31.551+0000 7f15afe00700 1 -- 172.21.15.28:0/3159163545 learned_addr learned my addr 172.21.15.28:0/3159163545 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T21:58:32.418 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:31.551+0000 7f15afe00700 1 -- 172.21.15.28:0/3159163545 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15b019e5a0 con 0x7f15b0108a20 2024-05-12T21:58:32.420 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:31.552+0000 7f15afe00700 1 --2- 172.21.15.28:0/3159163545 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15b0108a20 0x7f15b019e060 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f15a00117e0 tx=0x7f15a0011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T21:58:32.420 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:31.552+0000 7f15ad600700 1 -- 172.21.15.28:0/3159163545 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f15a0003ba0 con 0x7f15b0108a20 2024-05-12T21:58:32.420 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:31.552+0000 7f15ad600700 1 -- 172.21.15.28:0/3159163545 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f15a001a910 con 0x7f15b0108a20 2024-05-12T21:58:32.420 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:31.552+0000 7f15ad600700 1 -- 172.21.15.28:0/3159163545 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f15a00248f0 con 0x7f15b0108a20 2024-05-12T21:58:32.420 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:31.552+0000 7f15b7200700 1 -- 172.21.15.28:0/3159163545 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15b019e830 con 0x7f15b0108a20 2024-05-12T21:58:32.420 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:31.552+0000 7f15b7200700 1 -- 172.21.15.28:0/3159163545 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15b019ec90 con 0x7f15b0108a20 2024-05-12T21:58:32.420 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:31.553+0000 7f15ad600700 1 -- 172.21.15.28:0/3159163545 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f15a0022020 con 0x7f15b0108a20 2024-05-12T21:58:32.420 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:31.553+0000 7f15b7200700 1 -- 172.21.15.28:0/3159163545 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15b0063980 con 0x7f15b0108a20 2024-05-12T21:58:32.420 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:31.554+0000 7f15ad600700 1 --2- 172.21.15.28:0/3159163545 >> [v2:172.21.15.28:6800/3760499540,v1:172.21.15.28:6801/3760499540] conn(0x7f15940390a0 0x7f159403b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T21:58:32.420 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:31.554+0000 7f15ad600700 1 -- 172.21.15.28:0/3159163545 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f15a004d540 con 0x7f15b0108a20 2024-05-12T21:58:32.420 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:31.554+0000 7f15a7400700 1 --2- 172.21.15.28:0/3159163545 >> [v2:172.21.15.28:6800/3760499540,v1:172.21.15.28:6801/3760499540] conn(0x7f15940390a0 0x7f159403b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T21:58:32.420 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:31.555+0000 7f15a7400700 1 --2- 172.21.15.28:0/3159163545 >> [v2:172.21.15.28:6800/3760499540,v1:172.21.15.28:6801/3760499540] conn(0x7f15940390a0 0x7f159403b560 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f1598006fd0 tx=0x7f1598006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T21:58:32.420 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:31.559+0000 7f15ad600700 1 -- 172.21.15.28:0/3159163545 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f15a001ab70 con 0x7f15b0108a20 2024-05-12T21:58:32.420 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:31.791+0000 7f15b7200700 1 -- 172.21.15.28:0/3159163545 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) -- 0x7f15b01a15e0 con 0x7f15b0108a20 2024-05-12T21:58:32.421 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:31.795+0000 7f15ad600700 1 -- 172.21.15.28:0/3159163545 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) ==== 142+0+0 (secure 0 0 0) 0x7f15a00212c0 con 0x7f15b0108a20 2024-05-12T21:58:32.422 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:31.798+0000 7f15b7200700 1 -- 172.21.15.28:0/3159163545 >> [v2:172.21.15.28:6800/3760499540,v1:172.21.15.28:6801/3760499540] conn(0x7f15940390a0 msgr2=0x7f159403b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T21:58:32.422 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:31.798+0000 7f15b7200700 1 --2- 172.21.15.28:0/3159163545 >> [v2:172.21.15.28:6800/3760499540,v1:172.21.15.28:6801/3760499540] conn(0x7f15940390a0 0x7f159403b560 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f1598006fd0 tx=0x7f1598006e40 comp rx=0 tx=0).stop 2024-05-12T21:58:32.422 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:31.798+0000 7f15b7200700 1 -- 172.21.15.28:0/3159163545 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15b0108a20 msgr2=0x7f15b019e060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T21:58:32.422 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:31.798+0000 7f15b7200700 1 --2- 172.21.15.28:0/3159163545 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15b0108a20 0x7f15b019e060 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f15a00117e0 tx=0x7f15a0011ba0 comp rx=0 tx=0).stop 2024-05-12T21:58:32.422 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:31.798+0000 7f15b7200700 1 -- 172.21.15.28:0/3159163545 shutdown_connections 2024-05-12T21:58:32.422 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:31.798+0000 7f15b7200700 1 --2- 172.21.15.28:0/3159163545 >> [v2:172.21.15.28:6800/3760499540,v1:172.21.15.28:6801/3760499540] conn(0x7f15940390a0 0x7f159403b560 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:58:32.422 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:31.798+0000 7f15b7200700 1 --2- 172.21.15.28:0/3159163545 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15b0108a20 0x7f15b019e060 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:58:32.422 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:31.799+0000 7f15b7200700 1 -- 172.21.15.28:0/3159163545 >> 172.21.15.28:0/3159163545 conn(0x7f15b0102410 msgr2=0x7f15b0102e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T21:58:32.422 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:31.799+0000 7f15b7200700 1 -- 172.21.15.28:0/3159163545 shutdown_connections 2024-05-12T21:58:32.423 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:31.799+0000 7f15b7200700 1 -- 172.21.15.28:0/3159163545 wait complete. 2024-05-12T21:58:32.939 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:58:32 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/3159163545' entity='client.admin' 2024-05-12T21:58:34.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:58:33 smithi028 ceph-mon[29847]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T21:58:34.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:58:33 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/2640276319' entity='client.admin' 2024-05-12T21:58:34.390 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:33.106+0000 7fe7fbe00700 1 Processor -- start 2024-05-12T21:58:34.390 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:33.106+0000 7fe7fbe00700 1 -- start start 2024-05-12T21:58:34.390 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:33.106+0000 7fe7fbe00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe7f41089c0 0x7fe7f410adb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T21:58:34.390 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:33.106+0000 7fe7fbe00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe7f4063640 con 0x7fe7f41089c0 2024-05-12T21:58:34.390 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:33.107+0000 7fe7f9400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe7f41089c0 0x7fe7f410adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T21:58:34.390 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:33.107+0000 7fe7f9400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe7f41089c0 0x7fe7f410adb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40802/0 (socket says 172.21.15.28:40802) 2024-05-12T21:58:34.390 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:33.107+0000 7fe7f9400700 1 -- 172.21.15.28:0/3984091639 learned_addr learned my addr 172.21.15.28:0/3984091639 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T21:58:34.391 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:33.107+0000 7fe7f9400700 1 -- 172.21.15.28:0/3984091639 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7f410b2f0 con 0x7fe7f41089c0 2024-05-12T21:58:34.391 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:33.108+0000 7fe7f9400700 1 --2- 172.21.15.28:0/3984091639 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe7f41089c0 0x7fe7f410adb0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fe7e8009e80 tx=0x7fe7e800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb39162974db0571 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T21:58:34.391 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:33.108+0000 7fe7f3e00700 1 -- 172.21.15.28:0/3984091639 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe7e8017070 con 0x7fe7f41089c0 2024-05-12T21:58:34.391 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:33.108+0000 7fe7f3e00700 1 -- 172.21.15.28:0/3984091639 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe7e800b8c0 con 0x7fe7f41089c0 2024-05-12T21:58:34.391 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:33.108+0000 7fe7f3e00700 1 -- 172.21.15.28:0/3984091639 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe7e80045c0 con 0x7fe7f41089c0 2024-05-12T21:58:34.391 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:33.109+0000 7fe7fbe00700 1 -- 172.21.15.28:0/3984091639 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe7f41089c0 msgr2=0x7fe7f410adb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T21:58:34.391 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:33.109+0000 7fe7fbe00700 1 --2- 172.21.15.28:0/3984091639 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe7f41089c0 0x7fe7f410adb0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fe7e8009e80 tx=0x7fe7e800b330 comp rx=0 tx=0).stop 2024-05-12T21:58:34.391 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:33.109+0000 7fe7fbe00700 1 -- 172.21.15.28:0/3984091639 shutdown_connections 2024-05-12T21:58:34.391 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:33.109+0000 7fe7fbe00700 1 --2- 172.21.15.28:0/3984091639 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe7f41089c0 0x7fe7f410adb0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:58:34.392 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:33.109+0000 7fe7fbe00700 1 -- 172.21.15.28:0/3984091639 >> 172.21.15.28:0/3984091639 conn(0x7fe7f4102410 msgr2=0x7fe7f4104830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T21:58:34.392 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:33.109+0000 7fe7fbe00700 1 -- 172.21.15.28:0/3984091639 shutdown_connections 2024-05-12T21:58:34.392 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:33.109+0000 7fe7fbe00700 1 -- 172.21.15.28:0/3984091639 wait complete. 2024-05-12T21:58:34.392 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:33.110+0000 7fe7fbe00700 1 Processor -- start 2024-05-12T21:58:34.392 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:33.110+0000 7fe7fbe00700 1 -- start start 2024-05-12T21:58:34.392 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:33.111+0000 7fe7fbe00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe7f41089c0 0x7fe7f4108240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T21:58:34.392 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:33.111+0000 7fe7fbe00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe7f410c360 con 0x7fe7f41089c0 2024-05-12T21:58:34.392 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:33.111+0000 7fe7f9400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe7f41089c0 0x7fe7f4108240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T21:58:34.392 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:33.111+0000 7fe7f9400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe7f41089c0 0x7fe7f4108240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40804/0 (socket says 172.21.15.28:40804) 2024-05-12T21:58:34.392 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:33.111+0000 7fe7f9400700 1 -- 172.21.15.28:0/2640276319 learned_addr learned my addr 172.21.15.28:0/2640276319 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T21:58:34.393 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:33.112+0000 7fe7f9400700 1 -- 172.21.15.28:0/2640276319 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7f4108780 con 0x7fe7f41089c0 2024-05-12T21:58:34.393 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:33.112+0000 7fe7f9400700 1 --2- 172.21.15.28:0/2640276319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe7f41089c0 0x7fe7f4108240 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fe7e8009fd0 tx=0x7fe7e8003d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T21:58:34.393 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:33.112+0000 7fe7f2000700 1 -- 172.21.15.28:0/2640276319 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe7e8020020 con 0x7fe7f41089c0 2024-05-12T21:58:34.393 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:33.112+0000 7fe7f2000700 1 -- 172.21.15.28:0/2640276319 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe7e800bba0 con 0x7fe7f41089c0 2024-05-12T21:58:34.393 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:33.112+0000 7fe7f2000700 1 -- 172.21.15.28:0/2640276319 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe7e8017040 con 0x7fe7f41089c0 2024-05-12T21:58:34.393 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:33.112+0000 7fe7fbe00700 1 -- 172.21.15.28:0/2640276319 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7f4106970 con 0x7fe7f41089c0 2024-05-12T21:58:34.393 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:33.112+0000 7fe7fbe00700 1 -- 172.21.15.28:0/2640276319 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7f4106eb0 con 0x7fe7f41089c0 2024-05-12T21:58:34.393 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:33.114+0000 7fe7fbe00700 1 -- 172.21.15.28:0/2640276319 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe7f4063980 con 0x7fe7f41089c0 2024-05-12T21:58:34.393 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:33.114+0000 7fe7f2000700 1 -- 172.21.15.28:0/2640276319 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7fe7e8023020 con 0x7fe7f41089c0 2024-05-12T21:58:34.393 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:33.114+0000 7fe7f2000700 1 --2- 172.21.15.28:0/2640276319 >> [v2:172.21.15.28:6800/3760499540,v1:172.21.15.28:6801/3760499540] conn(0x7fe7dc0390a0 0x7fe7dc03b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T21:58:34.393 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:33.114+0000 7fe7f2000700 1 -- 172.21.15.28:0/2640276319 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fe7e800f050 con 0x7fe7f41089c0 2024-05-12T21:58:34.393 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:33.115+0000 7fe7f8a00700 1 --2- 172.21.15.28:0/2640276319 >> [v2:172.21.15.28:6800/3760499540,v1:172.21.15.28:6801/3760499540] conn(0x7fe7dc0390a0 0x7fe7dc03b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T21:58:34.394 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:33.115+0000 7fe7f8a00700 1 --2- 172.21.15.28:0/2640276319 >> [v2:172.21.15.28:6800/3760499540,v1:172.21.15.28:6801/3760499540] conn(0x7fe7dc0390a0 0x7fe7dc03b560 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fe7e0006fd0 tx=0x7fe7e0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T21:58:34.394 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:33.120+0000 7fe7f2000700 1 -- 172.21.15.28:0/2640276319 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fe7e802a020 con 0x7fe7f41089c0 2024-05-12T21:58:34.394 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:33.350+0000 7fe7fbe00700 1 -- 172.21.15.28:0/2640276319 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) -- 0x7fe7f41071f0 con 0x7fe7f41089c0 2024-05-12T21:58:34.394 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:33.354+0000 7fe7f2000700 1 -- 172.21.15.28:0/2640276319 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) ==== 130+0+0 (secure 0 0 0) 0x7fe7f41071f0 con 0x7fe7f41089c0 2024-05-12T21:58:34.394 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:33.357+0000 7fe7fbe00700 1 -- 172.21.15.28:0/2640276319 >> [v2:172.21.15.28:6800/3760499540,v1:172.21.15.28:6801/3760499540] conn(0x7fe7dc0390a0 msgr2=0x7fe7dc03b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T21:58:34.394 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:33.357+0000 7fe7fbe00700 1 --2- 172.21.15.28:0/2640276319 >> [v2:172.21.15.28:6800/3760499540,v1:172.21.15.28:6801/3760499540] conn(0x7fe7dc0390a0 0x7fe7dc03b560 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fe7e0006fd0 tx=0x7fe7e0006e40 comp rx=0 tx=0).stop 2024-05-12T21:58:34.394 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:33.357+0000 7fe7fbe00700 1 -- 172.21.15.28:0/2640276319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe7f41089c0 msgr2=0x7fe7f4108240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T21:58:34.394 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:33.357+0000 7fe7fbe00700 1 --2- 172.21.15.28:0/2640276319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe7f41089c0 0x7fe7f4108240 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fe7e8009fd0 tx=0x7fe7e8003d00 comp rx=0 tx=0).stop 2024-05-12T21:58:34.394 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:33.358+0000 7fe7fbe00700 1 -- 172.21.15.28:0/2640276319 shutdown_connections 2024-05-12T21:58:34.394 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:33.358+0000 7fe7fbe00700 1 --2- 172.21.15.28:0/2640276319 >> [v2:172.21.15.28:6800/3760499540,v1:172.21.15.28:6801/3760499540] conn(0x7fe7dc0390a0 0x7fe7dc03b560 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:58:34.395 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:33.358+0000 7fe7fbe00700 1 --2- 172.21.15.28:0/2640276319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe7f41089c0 0x7fe7f4108240 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:58:34.395 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:33.358+0000 7fe7fbe00700 1 -- 172.21.15.28:0/2640276319 >> 172.21.15.28:0/2640276319 conn(0x7fe7f4102410 msgr2=0x7fe7f4102f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T21:58:34.395 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:33.358+0000 7fe7fbe00700 1 -- 172.21.15.28:0/2640276319 shutdown_connections 2024-05-12T21:58:34.395 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:33.358+0000 7fe7fbe00700 1 -- 172.21.15.28:0/2640276319 wait complete. 2024-05-12T21:58:34.395 INFO:teuthology.orchestra.run.smithi028.stdout:Enabling the dashboard module... 2024-05-12T21:58:35.534 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:58:35 smithi028 ceph-mon[29847]: from='mgr.14120 172.21.15.28:0/1562014889' entity='mgr.x' 2024-05-12T21:58:36.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:58:36 smithi028 ceph-mon[29847]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T21:58:36.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:58:36 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/1436827742' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-05-12T21:58:36.535 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 21:58:36 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: ignoring --setuser ceph since I am not root 2024-05-12T21:58:36.535 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 21:58:36 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: ignoring --setgroup ceph since I am not root 2024-05-12T21:58:36.811 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 21:58:36 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T21:58:36.550+0000 7f6304123200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-12T21:58:36.811 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 21:58:36 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T21:58:36.810+0000 7f6304123200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-12T21:58:37.067 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 21:58:36 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T21:58:36.952+0000 7f6304123200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-12T21:58:37.083 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:34.981+0000 7f11d8c00700 1 Processor -- start 2024-05-12T21:58:37.083 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:34.982+0000 7f11d8c00700 1 -- start start 2024-05-12T21:58:37.083 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:34.982+0000 7f11d8c00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11d407cce0 0x7f11d407b140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T21:58:37.083 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:34.982+0000 7f11d8c00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f11d407b710 con 0x7f11d407cce0 2024-05-12T21:58:37.083 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:34.983+0000 7f11d1e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11d407cce0 0x7f11d407b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T21:58:37.083 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:34.983+0000 7f11d1e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11d407cce0 0x7f11d407b140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40820/0 (socket says 172.21.15.28:40820) 2024-05-12T21:58:37.084 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:34.983+0000 7f11d1e00700 1 -- 172.21.15.28:0/3178987277 learned_addr learned my addr 172.21.15.28:0/3178987277 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T21:58:37.084 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:34.983+0000 7f11d1e00700 1 -- 172.21.15.28:0/3178987277 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11d407b890 con 0x7f11d407cce0 2024-05-12T21:58:37.084 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:34.983+0000 7f11d1e00700 1 --2- 172.21.15.28:0/3178987277 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11d407cce0 0x7f11d407b140 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f11c40098a0 tx=0x7f11c4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c38b3142c08a69f server_cookie=0 in_seq=0 out_seq=0 2024-05-12T21:58:37.084 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:34.984+0000 7f11d0a00700 1 -- 172.21.15.28:0/3178987277 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f11c4004160 con 0x7f11d407cce0 2024-05-12T21:58:37.084 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:34.984+0000 7f11d0a00700 1 -- 172.21.15.28:0/3178987277 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f11c4004300 con 0x7f11d407cce0 2024-05-12T21:58:37.084 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:34.984+0000 7f11d0a00700 1 -- 172.21.15.28:0/3178987277 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f11c4011b20 con 0x7f11d407cce0 2024-05-12T21:58:37.084 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:34.984+0000 7f11d8c00700 1 -- 172.21.15.28:0/3178987277 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11d407cce0 msgr2=0x7f11d407b140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T21:58:37.088 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:34.984+0000 7f11d8c00700 1 --2- 172.21.15.28:0/3178987277 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11d407cce0 0x7f11d407b140 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f11c40098a0 tx=0x7f11c4009bb0 comp rx=0 tx=0).stop 2024-05-12T21:58:37.088 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:34.985+0000 7f11d8c00700 1 -- 172.21.15.28:0/3178987277 shutdown_connections 2024-05-12T21:58:37.088 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:34.985+0000 7f11d8c00700 1 --2- 172.21.15.28:0/3178987277 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11d407cce0 0x7f11d407b140 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:58:37.088 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:34.985+0000 7f11d8c00700 1 -- 172.21.15.28:0/3178987277 >> 172.21.15.28:0/3178987277 conn(0x7f11d4103530 msgr2=0x7f11d4105950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T21:58:37.088 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:34.985+0000 7f11d8c00700 1 -- 172.21.15.28:0/3178987277 shutdown_connections 2024-05-12T21:58:37.088 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:34.985+0000 7f11d8c00700 1 -- 172.21.15.28:0/3178987277 wait complete. 2024-05-12T21:58:37.089 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:34.986+0000 7f11d8c00700 1 Processor -- start 2024-05-12T21:58:37.089 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:34.986+0000 7f11d8c00700 1 -- start start 2024-05-12T21:58:37.089 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:34.987+0000 7f11d8c00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11d407cce0 0x7f11d419e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T21:58:37.089 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:34.987+0000 7f11d8c00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f11d410e5b0 con 0x7f11d407cce0 2024-05-12T21:58:37.089 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:34.987+0000 7f11d1e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11d407cce0 0x7f11d419e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T21:58:37.089 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:34.987+0000 7f11d1e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11d407cce0 0x7f11d419e140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40826/0 (socket says 172.21.15.28:40826) 2024-05-12T21:58:37.089 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:34.987+0000 7f11d1e00700 1 -- 172.21.15.28:0/1436827742 learned_addr learned my addr 172.21.15.28:0/1436827742 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T21:58:37.089 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:34.987+0000 7f11d1e00700 1 -- 172.21.15.28:0/1436827742 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11d419e680 con 0x7f11d407cce0 2024-05-12T21:58:37.089 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:34.988+0000 7f11d1e00700 1 --2- 172.21.15.28:0/1436827742 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11d407cce0 0x7f11d419e140 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f11c4011b60 tx=0x7f11c4011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T21:58:37.089 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:34.990+0000 7f11caa00700 1 -- 172.21.15.28:0/1436827742 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f11c4003b60 con 0x7f11d407cce0 2024-05-12T21:58:37.089 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:34.990+0000 7f11caa00700 1 -- 172.21.15.28:0/1436827742 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f11c401a910 con 0x7f11d407cce0 2024-05-12T21:58:37.089 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:34.990+0000 7f11caa00700 1 -- 172.21.15.28:0/1436827742 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f11c4024930 con 0x7f11d407cce0 2024-05-12T21:58:37.089 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:34.991+0000 7f11d8c00700 1 -- 172.21.15.28:0/1436827742 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11d419e910 con 0x7f11d407cce0 2024-05-12T21:58:37.089 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:34.991+0000 7f11d8c00700 1 -- 172.21.15.28:0/1436827742 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11d419ed70 con 0x7f11d407cce0 2024-05-12T21:58:37.089 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:34.992+0000 7f11d8c00700 1 -- 172.21.15.28:0/1436827742 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11d4063980 con 0x7f11d407cce0 2024-05-12T21:58:37.090 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:34.992+0000 7f11caa00700 1 -- 172.21.15.28:0/1436827742 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f11c4022020 con 0x7f11d407cce0 2024-05-12T21:58:37.090 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:34.993+0000 7f11caa00700 1 --2- 172.21.15.28:0/1436827742 >> [v2:172.21.15.28:6800/3760499540,v1:172.21.15.28:6801/3760499540] conn(0x7f11b80390a0 0x7f11b803b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T21:58:37.090 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:34.993+0000 7f11caa00700 1 -- 172.21.15.28:0/1436827742 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f11c404d510 con 0x7f11d407cce0 2024-05-12T21:58:37.090 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:34.993+0000 7f11d1400700 1 --2- 172.21.15.28:0/1436827742 >> [v2:172.21.15.28:6800/3760499540,v1:172.21.15.28:6801/3760499540] conn(0x7f11b80390a0 0x7f11b803b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T21:58:37.090 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:34.994+0000 7f11d1400700 1 --2- 172.21.15.28:0/1436827742 >> [v2:172.21.15.28:6800/3760499540,v1:172.21.15.28:6801/3760499540] conn(0x7f11b80390a0 0x7f11b803b560 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f11c0006fd0 tx=0x7f11c0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T21:58:37.090 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:34.999+0000 7f11caa00700 1 -- 172.21.15.28:0/1436827742 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f11c4029060 con 0x7f11d407cce0 2024-05-12T21:58:37.090 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:35.290+0000 7f11d8c00700 1 -- 172.21.15.28:0/1436827742 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) -- 0x7f11d41a1740 con 0x7f11d407cce0 2024-05-12T21:58:37.090 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:36.224+0000 7f11caa00700 1 -- 172.21.15.28:0/1436827742 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) ==== 88+0+0 (secure 0 0 0) 0x7f11c400b3e0 con 0x7f11d407cce0 2024-05-12T21:58:37.090 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:36.227+0000 7f11caa00700 1 -- 172.21.15.28:0/1436827742 <== mon.0 v2:172.21.15.28:3300/0 8 ==== mgrmap(e 9) ==== 46300+0+0 (secure 0 0 0) 0x7f11c4024ad0 con 0x7f11d407cce0 2024-05-12T21:58:37.090 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:36.228+0000 7f11d8c00700 1 -- 172.21.15.28:0/1436827742 >> [v2:172.21.15.28:6800/3760499540,v1:172.21.15.28:6801/3760499540] conn(0x7f11b80390a0 msgr2=0x7f11b803b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T21:58:37.090 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:36.228+0000 7f11d8c00700 1 --2- 172.21.15.28:0/1436827742 >> [v2:172.21.15.28:6800/3760499540,v1:172.21.15.28:6801/3760499540] conn(0x7f11b80390a0 0x7f11b803b560 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f11c0006fd0 tx=0x7f11c0006e40 comp rx=0 tx=0).stop 2024-05-12T21:58:37.090 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:36.228+0000 7f11d8c00700 1 -- 172.21.15.28:0/1436827742 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11d407cce0 msgr2=0x7f11d419e140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T21:58:37.091 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:36.228+0000 7f11d8c00700 1 --2- 172.21.15.28:0/1436827742 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11d407cce0 0x7f11d419e140 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f11c4011b60 tx=0x7f11c4011c40 comp rx=0 tx=0).stop 2024-05-12T21:58:37.091 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:36.229+0000 7f11d8c00700 1 -- 172.21.15.28:0/1436827742 shutdown_connections 2024-05-12T21:58:37.091 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:36.229+0000 7f11d8c00700 1 --2- 172.21.15.28:0/1436827742 >> [v2:172.21.15.28:6800/3760499540,v1:172.21.15.28:6801/3760499540] conn(0x7f11b80390a0 0x7f11b803b560 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:58:37.091 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:36.229+0000 7f11d8c00700 1 --2- 172.21.15.28:0/1436827742 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11d407cce0 0x7f11d419e140 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:58:37.091 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:36.229+0000 7f11d8c00700 1 -- 172.21.15.28:0/1436827742 >> 172.21.15.28:0/1436827742 conn(0x7f11d4103530 msgr2=0x7f11d41040c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T21:58:37.091 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:36.229+0000 7f11d8c00700 1 -- 172.21.15.28:0/1436827742 shutdown_connections 2024-05-12T21:58:37.091 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:36.229+0000 7f11d8c00700 1 -- 172.21.15.28:0/1436827742 wait complete. 2024-05-12T21:58:37.375 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:58:37 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/1436827742' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-05-12T21:58:37.375 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:58:37 smithi028 ceph-mon[29847]: mgrmap e9: x(active, since 25s) 2024-05-12T21:58:37.675 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 21:58:37 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T21:58:37.400+0000 7f6304123200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-12T21:58:37.676 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 21:58:37 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T21:58:37.526+0000 7f6304123200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-12T21:58:38.534 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:58:38 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/91271404' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-05-12T21:58:38.696 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout { 2024-05-12T21:58:38.696 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-05-12T21:58:38.697 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "available": true, 2024-05-12T21:58:38.697 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-05-12T21:58:38.697 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-05-12T21:58:38.697 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout } 2024-05-12T21:58:38.697 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:37.673+0000 7fa90fe00700 1 Processor -- start 2024-05-12T21:58:38.697 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:37.674+0000 7fa90fe00700 1 -- start start 2024-05-12T21:58:38.697 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:37.674+0000 7fa90fe00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa908107f70 0x7fa908108390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T21:58:38.697 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:37.674+0000 7fa90fe00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa908108960 con 0x7fa908107f70 2024-05-12T21:58:38.697 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:37.675+0000 7fa90d400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa908107f70 0x7fa908108390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T21:58:38.697 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:37.675+0000 7fa90d400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa908107f70 0x7fa908108390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45546/0 (socket says 172.21.15.28:45546) 2024-05-12T21:58:38.697 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:37.675+0000 7fa90d400700 1 -- 172.21.15.28:0/294896248 learned_addr learned my addr 172.21.15.28:0/294896248 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T21:58:38.697 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:37.675+0000 7fa90d400700 1 -- 172.21.15.28:0/294896248 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa908109170 con 0x7fa908107f70 2024-05-12T21:58:38.697 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:37.676+0000 7fa90d400700 1 --2- 172.21.15.28:0/294896248 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa908107f70 0x7fa908108390 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fa8fc0098a0 tx=0x7fa8fc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=894432e513c43d8a server_cookie=0 in_seq=0 out_seq=0 2024-05-12T21:58:38.697 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:37.676+0000 7fa907e00700 1 -- 172.21.15.28:0/294896248 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa8fc004160 con 0x7fa908107f70 2024-05-12T21:58:38.698 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:37.676+0000 7fa907e00700 1 -- 172.21.15.28:0/294896248 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa8fc004300 con 0x7fa908107f70 2024-05-12T21:58:38.698 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:37.676+0000 7fa907e00700 1 -- 172.21.15.28:0/294896248 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa8fc011b20 con 0x7fa908107f70 2024-05-12T21:58:38.698 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:37.677+0000 7fa90fe00700 1 -- 172.21.15.28:0/294896248 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa908107f70 msgr2=0x7fa908108390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T21:58:38.698 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:37.677+0000 7fa90fe00700 1 --2- 172.21.15.28:0/294896248 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa908107f70 0x7fa908108390 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fa8fc0098a0 tx=0x7fa8fc009bb0 comp rx=0 tx=0).stop 2024-05-12T21:58:38.698 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:37.677+0000 7fa90fe00700 1 -- 172.21.15.28:0/294896248 shutdown_connections 2024-05-12T21:58:38.698 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:37.677+0000 7fa90fe00700 1 --2- 172.21.15.28:0/294896248 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa908107f70 0x7fa908108390 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:58:38.698 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:37.677+0000 7fa90fe00700 1 -- 172.21.15.28:0/294896248 >> 172.21.15.28:0/294896248 conn(0x7fa908103530 msgr2=0x7fa908105950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T21:58:38.698 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:37.677+0000 7fa90fe00700 1 -- 172.21.15.28:0/294896248 shutdown_connections 2024-05-12T21:58:38.698 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:37.677+0000 7fa90fe00700 1 -- 172.21.15.28:0/294896248 wait complete. 2024-05-12T21:58:38.698 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:37.678+0000 7fa90fe00700 1 Processor -- start 2024-05-12T21:58:38.698 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:37.678+0000 7fa90fe00700 1 -- start start 2024-05-12T21:58:38.698 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:37.679+0000 7fa90fe00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa908107f70 0x7fa90819e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T21:58:38.698 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:37.679+0000 7fa90fe00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa90810c520 con 0x7fa908107f70 2024-05-12T21:58:38.698 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:37.679+0000 7fa90d400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa908107f70 0x7fa90819e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T21:58:38.699 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:37.679+0000 7fa90d400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa908107f70 0x7fa90819e160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45548/0 (socket says 172.21.15.28:45548) 2024-05-12T21:58:38.699 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:37.679+0000 7fa90d400700 1 -- 172.21.15.28:0/91271404 learned_addr learned my addr 172.21.15.28:0/91271404 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T21:58:38.699 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:37.679+0000 7fa90d400700 1 -- 172.21.15.28:0/91271404 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa90819e6a0 con 0x7fa908107f70 2024-05-12T21:58:38.699 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:37.679+0000 7fa90d400700 1 --2- 172.21.15.28:0/91271404 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa908107f70 0x7fa90819e160 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fa8fc000c00 tx=0x7fa8fc011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T21:58:38.699 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:37.680+0000 7fa906000700 1 -- 172.21.15.28:0/91271404 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa8fc003b60 con 0x7fa908107f70 2024-05-12T21:58:38.699 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:37.680+0000 7fa906000700 1 -- 172.21.15.28:0/91271404 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa8fc01a910 con 0x7fa908107f70 2024-05-12T21:58:38.699 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:37.680+0000 7fa906000700 1 -- 172.21.15.28:0/91271404 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa8fc024930 con 0x7fa908107f70 2024-05-12T21:58:38.699 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:37.680+0000 7fa90fe00700 1 -- 172.21.15.28:0/91271404 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa90819e930 con 0x7fa908107f70 2024-05-12T21:58:38.699 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:37.680+0000 7fa90fe00700 1 -- 172.21.15.28:0/91271404 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa90819ed90 con 0x7fa908107f70 2024-05-12T21:58:38.699 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:37.681+0000 7fa90fe00700 1 -- 172.21.15.28:0/91271404 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa908063980 con 0x7fa908107f70 2024-05-12T21:58:38.699 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:37.681+0000 7fa906000700 1 -- 172.21.15.28:0/91271404 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 9) ==== 46300+0+0 (secure 0 0 0) 0x7fa8fc022020 con 0x7fa908107f70 2024-05-12T21:58:38.699 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:37.682+0000 7fa906000700 1 --2- 172.21.15.28:0/91271404 >> [v2:172.21.15.28:6800/3760499540,v1:172.21.15.28:6801/3760499540] conn(0x7fa8f0039100 0x7fa8f003b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T21:58:38.699 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:37.682+0000 7fa906000700 1 -- 172.21.15.28:0/91271404 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fa8fc04d1b0 con 0x7fa908107f70 2024-05-12T21:58:38.699 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:37.682+0000 7fa90ca00700 1 -- 172.21.15.28:0/91271404 >> [v2:172.21.15.28:6800/3760499540,v1:172.21.15.28:6801/3760499540] conn(0x7fa8f0039100 msgr2=0x7fa8f003b5c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/3760499540 2024-05-12T21:58:38.700 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:37.682+0000 7fa90ca00700 1 --2- 172.21.15.28:0/91271404 >> [v2:172.21.15.28:6800/3760499540,v1:172.21.15.28:6801/3760499540] conn(0x7fa8f0039100 0x7fa8f003b5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-12T21:58:38.700 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:37.688+0000 7fa906000700 1 -- 172.21.15.28:0/91271404 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fa8fc02ddb0 con 0x7fa908107f70 2024-05-12T21:58:38.700 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:37.883+0000 7fa90ca00700 1 -- 172.21.15.28:0/91271404 >> [v2:172.21.15.28:6800/3760499540,v1:172.21.15.28:6801/3760499540] conn(0x7fa8f0039100 msgr2=0x7fa8f003b5c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/3760499540 2024-05-12T21:58:38.700 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:37.883+0000 7fa90ca00700 1 --2- 172.21.15.28:0/91271404 >> [v2:172.21.15.28:6800/3760499540,v1:172.21.15.28:6801/3760499540] conn(0x7fa8f0039100 0x7fa8f003b5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-12T21:58:38.700 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:37.973+0000 7fa90fe00700 1 -- 172.21.15.28:0/91271404 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7fa9081a1800 con 0x7fa908107f70 2024-05-12T21:58:38.700 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:37.974+0000 7fa906000700 1 -- 172.21.15.28:0/91271404 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) ==== 56+0+88 (secure 0 0 0) 0x7fa8fc00b3e0 con 0x7fa908107f70 2024-05-12T21:58:38.700 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:37.977+0000 7fa90fe00700 1 -- 172.21.15.28:0/91271404 >> [v2:172.21.15.28:6800/3760499540,v1:172.21.15.28:6801/3760499540] conn(0x7fa8f0039100 msgr2=0x7fa8f003b5c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-12T21:58:38.700 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:37.977+0000 7fa90fe00700 1 --2- 172.21.15.28:0/91271404 >> [v2:172.21.15.28:6800/3760499540,v1:172.21.15.28:6801/3760499540] conn(0x7fa8f0039100 0x7fa8f003b5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:58:38.700 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:37.978+0000 7fa90fe00700 1 -- 172.21.15.28:0/91271404 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa908107f70 msgr2=0x7fa90819e160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T21:58:38.700 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:37.978+0000 7fa90fe00700 1 --2- 172.21.15.28:0/91271404 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa908107f70 0x7fa90819e160 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fa8fc000c00 tx=0x7fa8fc011c10 comp rx=0 tx=0).stop 2024-05-12T21:58:38.700 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:37.978+0000 7fa90fe00700 1 -- 172.21.15.28:0/91271404 shutdown_connections 2024-05-12T21:58:38.700 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:37.978+0000 7fa90fe00700 1 --2- 172.21.15.28:0/91271404 >> [v2:172.21.15.28:6800/3760499540,v1:172.21.15.28:6801/3760499540] conn(0x7fa8f0039100 0x7fa8f003b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:58:38.700 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:37.978+0000 7fa90fe00700 1 --2- 172.21.15.28:0/91271404 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa908107f70 0x7fa90819e160 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:58:38.700 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:37.978+0000 7fa90fe00700 1 -- 172.21.15.28:0/91271404 >> 172.21.15.28:0/91271404 conn(0x7fa908103530 msgr2=0x7fa9081040e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T21:58:38.701 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:37.978+0000 7fa90fe00700 1 -- 172.21.15.28:0/91271404 shutdown_connections 2024-05-12T21:58:38.701 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:37.978+0000 7fa90fe00700 1 -- 172.21.15.28:0/91271404 wait complete. 2024-05-12T21:58:38.701 INFO:teuthology.orchestra.run.smithi028.stdout:Waiting for the mgr to restart... 2024-05-12T21:58:38.701 INFO:teuthology.orchestra.run.smithi028.stdout:Waiting for mgr epoch 9... 2024-05-12T21:58:39.242 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 21:58:39 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T21:58:39.160+0000 7f6304123200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-12T21:58:39.501 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 21:58:39 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T21:58:39.397+0000 7f6304123200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-12T21:58:39.785 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 21:58:39 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T21:58:39.618+0000 7f6304123200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-12T21:58:40.285 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 21:58:40 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T21:58:40.013+0000 7f6304123200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-12T21:58:40.285 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 21:58:40 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T21:58:40.140+0000 7f6304123200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-12T21:58:40.661 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 21:58:40 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T21:58:40.399+0000 7f6304123200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-12T21:58:40.662 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 21:58:40 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T21:58:40.661+0000 7f6304123200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-12T21:58:41.785 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 21:58:41 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T21:58:41.482+0000 7f6304123200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-12T21:58:42.305 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 21:58:42 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T21:58:42.053+0000 7f6304123200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-12T21:58:42.306 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 21:58:42 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T21:58:42.186+0000 7f6304123200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-12T21:58:42.785 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 21:58:42 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T21:58:42.461+0000 7f6304123200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-12T21:58:43.535 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 21:58:43 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T21:58:43.131+0000 7f6304123200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-12T21:58:43.535 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 21:58:43 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T21:58:43.259+0000 7f6304123200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-12T21:58:43.916 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 21:58:43 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T21:58:43.596+0000 7f6304123200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-12T21:58:43.916 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 21:58:43 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T21:58:43.784+0000 7f6304123200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-12T21:58:43.916 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 21:58:43 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T21:58:43.916+0000 7f6304123200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-12T21:58:44.285 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 21:58:44 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T21:58:44.075+0000 7f6304123200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-12T21:58:44.785 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 21:58:44 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T21:58:44.450+0000 7f6304123200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-12T21:58:45.509 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 21:58:45 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T21:58:45.241+0000 7f6304123200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-12T21:58:45.785 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 21:58:45 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T21:58:45.547+0000 7f6304123200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-12T21:58:45.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:58:45 smithi028 ceph-mon[29847]: Active manager daemon x restarted 2024-05-12T21:58:45.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:58:45 smithi028 ceph-mon[29847]: Activating manager daemon x 2024-05-12T21:58:45.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:58:45 smithi028 ceph-mon[29847]: osdmap e3: 0 total, 0 up, 0 in 2024-05-12T21:58:45.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:58:45 smithi028 ceph-mon[29847]: mgrmap e10: x(active, starting, since 0.0092135s) 2024-05-12T21:58:45.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:58:45 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-12T21:58:45.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:58:45 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-12T21:58:45.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:58:45 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-12T21:58:45.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:58:45 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-12T21:58:45.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:58:45 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-12T21:58:45.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:58:45 smithi028 ceph-mon[29847]: Manager daemon x is now available 2024-05-12T21:58:46.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:58:46 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T21:58:46.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:58:46 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-05-12T21:58:46.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:58:46 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-05-12T21:58:46.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:58:46 smithi028 ceph-mon[29847]: [12/May/2024:21:58:46] ENGINE Bus STARTING 2024-05-12T21:58:46.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:58:46 smithi028 ceph-mon[29847]: [12/May/2024:21:58:46] ENGINE Serving on http://172.21.15.28:8765 2024-05-12T21:58:46.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:58:46 smithi028 ceph-mon[29847]: mgrmap e11: x(active, since 1.01085s) 2024-05-12T21:58:47.237 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout { 2024-05-12T21:58:47.246 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-05-12T21:58:47.246 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "initialized": true 2024-05-12T21:58:47.246 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout } 2024-05-12T21:58:47.246 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:39.292+0000 7f6d54c00700 1 Processor -- start 2024-05-12T21:58:47.246 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:39.292+0000 7f6d54c00700 1 -- start start 2024-05-12T21:58:47.246 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:39.292+0000 7f6d54c00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d5007cd00 0x7f6d5007b160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T21:58:47.246 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:39.292+0000 7f6d54c00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6d5007b730 con 0x7f6d5007cd00 2024-05-12T21:58:47.246 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:39.293+0000 7f6d4de00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d5007cd00 0x7f6d5007b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T21:58:47.246 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:39.293+0000 7f6d4de00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d5007cd00 0x7f6d5007b160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45562/0 (socket says 172.21.15.28:45562) 2024-05-12T21:58:47.246 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:39.293+0000 7f6d4de00700 1 -- 172.21.15.28:0/1351829479 learned_addr learned my addr 172.21.15.28:0/1351829479 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T21:58:47.246 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:39.293+0000 7f6d4de00700 1 -- 172.21.15.28:0/1351829479 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d5007b8b0 con 0x7f6d5007cd00 2024-05-12T21:58:47.246 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:39.294+0000 7f6d4de00700 1 --2- 172.21.15.28:0/1351829479 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d5007cd00 0x7f6d5007b160 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f6d400098a0 tx=0x7f6d40009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a17a4dfc434b594 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T21:58:47.247 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:39.294+0000 7f6d4ca00700 1 -- 172.21.15.28:0/1351829479 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6d40004160 con 0x7f6d5007cd00 2024-05-12T21:58:47.247 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:39.294+0000 7f6d4ca00700 1 -- 172.21.15.28:0/1351829479 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6d40004300 con 0x7f6d5007cd00 2024-05-12T21:58:47.247 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:39.294+0000 7f6d4ca00700 1 -- 172.21.15.28:0/1351829479 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6d40011b20 con 0x7f6d5007cd00 2024-05-12T21:58:47.247 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:39.295+0000 7f6d54c00700 1 -- 172.21.15.28:0/1351829479 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d5007cd00 msgr2=0x7f6d5007b160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T21:58:47.247 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:39.295+0000 7f6d54c00700 1 --2- 172.21.15.28:0/1351829479 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d5007cd00 0x7f6d5007b160 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f6d400098a0 tx=0x7f6d40009bb0 comp rx=0 tx=0).stop 2024-05-12T21:58:47.247 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:39.295+0000 7f6d54c00700 1 -- 172.21.15.28:0/1351829479 shutdown_connections 2024-05-12T21:58:47.247 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:39.295+0000 7f6d54c00700 1 --2- 172.21.15.28:0/1351829479 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d5007cd00 0x7f6d5007b160 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:58:47.247 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:39.295+0000 7f6d54c00700 1 -- 172.21.15.28:0/1351829479 >> 172.21.15.28:0/1351829479 conn(0x7f6d501034e0 msgr2=0x7f6d50105920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T21:58:47.247 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:39.295+0000 7f6d54c00700 1 -- 172.21.15.28:0/1351829479 shutdown_connections 2024-05-12T21:58:47.247 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:39.295+0000 7f6d54c00700 1 -- 172.21.15.28:0/1351829479 wait complete. 2024-05-12T21:58:47.247 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:39.296+0000 7f6d54c00700 1 Processor -- start 2024-05-12T21:58:47.247 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:39.296+0000 7f6d54c00700 1 -- start start 2024-05-12T21:58:47.247 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:39.297+0000 7f6d54c00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d5007cd00 0x7f6d5019e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T21:58:47.247 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:39.297+0000 7f6d54c00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6d5010e5c0 con 0x7f6d5007cd00 2024-05-12T21:58:47.248 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:39.297+0000 7f6d4de00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d5007cd00 0x7f6d5019e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T21:58:47.248 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:39.297+0000 7f6d4de00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d5007cd00 0x7f6d5019e0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45572/0 (socket says 172.21.15.28:45572) 2024-05-12T21:58:47.248 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:39.297+0000 7f6d4de00700 1 -- 172.21.15.28:0/3647413092 learned_addr learned my addr 172.21.15.28:0/3647413092 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T21:58:47.248 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:39.297+0000 7f6d4de00700 1 -- 172.21.15.28:0/3647413092 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d5019e5e0 con 0x7f6d5007cd00 2024-05-12T21:58:47.248 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:39.298+0000 7f6d4de00700 1 --2- 172.21.15.28:0/3647413092 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d5007cd00 0x7f6d5019e0a0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f6d400117e0 tx=0x7f6d40011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T21:58:47.248 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:39.298+0000 7f6d46a00700 1 -- 172.21.15.28:0/3647413092 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6d40003af0 con 0x7f6d5007cd00 2024-05-12T21:58:47.248 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:39.298+0000 7f6d46a00700 1 -- 172.21.15.28:0/3647413092 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6d40003c90 con 0x7f6d5007cd00 2024-05-12T21:58:47.248 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:39.298+0000 7f6d46a00700 1 -- 172.21.15.28:0/3647413092 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6d40024800 con 0x7f6d5007cd00 2024-05-12T21:58:47.248 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:39.298+0000 7f6d54c00700 1 -- 172.21.15.28:0/3647413092 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d5019e870 con 0x7f6d5007cd00 2024-05-12T21:58:47.248 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:39.298+0000 7f6d54c00700 1 -- 172.21.15.28:0/3647413092 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d5019ecd0 con 0x7f6d5007cd00 2024-05-12T21:58:47.248 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:39.300+0000 7f6d46a00700 1 -- 172.21.15.28:0/3647413092 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 9) ==== 46300+0+0 (secure 0 0 0) 0x7f6d40022020 con 0x7f6d5007cd00 2024-05-12T21:58:47.248 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:39.300+0000 7f6d46a00700 1 --2- 172.21.15.28:0/3647413092 >> [v2:172.21.15.28:6800/3760499540,v1:172.21.15.28:6801/3760499540] conn(0x7f6d34039100 0x7f6d3403b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T21:58:47.248 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:39.300+0000 7f6d46a00700 1 -- 172.21.15.28:0/3647413092 --> [v2:172.21.15.28:6800/3760499540,v1:172.21.15.28:6801/3760499540] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f6d3403bcd0 con 0x7f6d34039100 2024-05-12T21:58:47.248 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:39.300+0000 7f6d46a00700 1 -- 172.21.15.28:0/3647413092 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f6d4004d320 con 0x7f6d5007cd00 2024-05-12T21:58:47.248 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:39.300+0000 7f6d4d400700 1 -- 172.21.15.28:0/3647413092 >> [v2:172.21.15.28:6800/3760499540,v1:172.21.15.28:6801/3760499540] conn(0x7f6d34039100 msgr2=0x7f6d3403b5c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/3760499540 2024-05-12T21:58:47.249 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:39.300+0000 7f6d4d400700 1 --2- 172.21.15.28:0/3647413092 >> [v2:172.21.15.28:6800/3760499540,v1:172.21.15.28:6801/3760499540] conn(0x7f6d34039100 0x7f6d3403b5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-12T21:58:47.249 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:39.501+0000 7f6d4d400700 1 -- 172.21.15.28:0/3647413092 >> [v2:172.21.15.28:6800/3760499540,v1:172.21.15.28:6801/3760499540] conn(0x7f6d34039100 msgr2=0x7f6d3403b5c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/3760499540 2024-05-12T21:58:47.249 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:39.501+0000 7f6d4d400700 1 --2- 172.21.15.28:0/3647413092 >> [v2:172.21.15.28:6800/3760499540,v1:172.21.15.28:6801/3760499540] conn(0x7f6d34039100 0x7f6d3403b5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-12T21:58:47.249 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:39.902+0000 7f6d4d400700 1 -- 172.21.15.28:0/3647413092 >> [v2:172.21.15.28:6800/3760499540,v1:172.21.15.28:6801/3760499540] conn(0x7f6d34039100 msgr2=0x7f6d3403b5c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/3760499540 2024-05-12T21:58:47.249 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:39.902+0000 7f6d4d400700 1 --2- 172.21.15.28:0/3647413092 >> [v2:172.21.15.28:6800/3760499540,v1:172.21.15.28:6801/3760499540] conn(0x7f6d34039100 0x7f6d3403b5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-12T21:58:47.249 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:40.703+0000 7f6d4d400700 1 -- 172.21.15.28:0/3647413092 >> [v2:172.21.15.28:6800/3760499540,v1:172.21.15.28:6801/3760499540] conn(0x7f6d34039100 msgr2=0x7f6d3403b5c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/3760499540 2024-05-12T21:58:47.249 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:40.703+0000 7f6d4d400700 1 --2- 172.21.15.28:0/3647413092 >> [v2:172.21.15.28:6800/3760499540,v1:172.21.15.28:6801/3760499540] conn(0x7f6d34039100 0x7f6d3403b5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-12T21:58:47.249 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:42.305+0000 7f6d4d400700 1 -- 172.21.15.28:0/3647413092 >> [v2:172.21.15.28:6800/3760499540,v1:172.21.15.28:6801/3760499540] conn(0x7f6d34039100 msgr2=0x7f6d3403b5c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/3760499540 2024-05-12T21:58:47.249 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:42.305+0000 7f6d4d400700 1 --2- 172.21.15.28:0/3647413092 >> [v2:172.21.15.28:6800/3760499540,v1:172.21.15.28:6801/3760499540] conn(0x7f6d34039100 0x7f6d3403b5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-05-12T21:58:47.249 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:45.509+0000 7f6d4d400700 1 -- 172.21.15.28:0/3647413092 >> [v2:172.21.15.28:6800/3760499540,v1:172.21.15.28:6801/3760499540] conn(0x7f6d34039100 msgr2=0x7f6d3403b5c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/3760499540 2024-05-12T21:58:47.249 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:45.509+0000 7f6d4d400700 1 --2- 172.21.15.28:0/3647413092 >> [v2:172.21.15.28:6800/3760499540,v1:172.21.15.28:6801/3760499540] conn(0x7f6d34039100 0x7f6d3403b5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-05-12T21:58:47.249 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:45.556+0000 7f6d46a00700 1 -- 172.21.15.28:0/3647413092 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mgrmap(e 10) ==== 46067+0+0 (secure 0 0 0) 0x7f6d4001aba0 con 0x7f6d5007cd00 2024-05-12T21:58:47.249 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:45.556+0000 7f6d46a00700 1 -- 172.21.15.28:0/3647413092 >> [v2:172.21.15.28:6800/3760499540,v1:172.21.15.28:6801/3760499540] conn(0x7f6d34039100 msgr2=0x7f6d3403b5c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-12T21:58:47.249 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:45.556+0000 7f6d46a00700 1 --2- 172.21.15.28:0/3647413092 >> [v2:172.21.15.28:6800/3760499540,v1:172.21.15.28:6801/3760499540] conn(0x7f6d34039100 0x7f6d3403b5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:58:47.249 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:46.558+0000 7f6d46a00700 1 -- 172.21.15.28:0/3647413092 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mgrmap(e 11) ==== 46194+0+0 (secure 0 0 0) 0x7f6d4004f1f0 con 0x7f6d5007cd00 2024-05-12T21:58:47.250 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:46.558+0000 7f6d46a00700 1 --2- 172.21.15.28:0/3647413092 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f6d3403cbc0 0x7f6d3403efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T21:58:47.250 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:46.558+0000 7f6d46a00700 1 -- 172.21.15.28:0/3647413092 --> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f6d400068a0 con 0x7f6d3403cbc0 2024-05-12T21:58:47.250 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:46.559+0000 7f6d4d400700 1 --2- 172.21.15.28:0/3647413092 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f6d3403cbc0 0x7f6d3403efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T21:58:47.250 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:46.561+0000 7f6d4d400700 1 --2- 172.21.15.28:0/3647413092 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f6d3403cbc0 0x7f6d3403efb0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f6d3c003a10 tx=0x7f6d3c009270 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T21:58:47.250 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:46.562+0000 7f6d46a00700 1 -- 172.21.15.28:0/3647413092 <== mgr.14152 v2:172.21.15.28:6800/3123328651 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7f6d400068a0 con 0x7f6d3403cbc0 2024-05-12T21:58:47.250 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:46.571+0000 7f6d54c00700 1 -- 172.21.15.28:0/3647413092 --> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f6d500694a0 con 0x7f6d3403cbc0 2024-05-12T21:58:47.250 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:46.572+0000 7f6d46a00700 1 -- 172.21.15.28:0/3647413092 <== mgr.14152 v2:172.21.15.28:6800/3123328651 2 ==== command_reply(tid 1: 0 ) ==== 8+0+52 (secure 0 0 0) 0x7f6d500694a0 con 0x7f6d3403cbc0 2024-05-12T21:58:47.250 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:46.572+0000 7f6d54c00700 1 -- 172.21.15.28:0/3647413092 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f6d3403cbc0 msgr2=0x7f6d3403efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T21:58:47.250 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:46.572+0000 7f6d54c00700 1 --2- 172.21.15.28:0/3647413092 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f6d3403cbc0 0x7f6d3403efb0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f6d3c003a10 tx=0x7f6d3c009270 comp rx=0 tx=0).stop 2024-05-12T21:58:47.250 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:46.572+0000 7f6d54c00700 1 -- 172.21.15.28:0/3647413092 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d5007cd00 msgr2=0x7f6d5019e0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T21:58:47.250 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:46.572+0000 7f6d54c00700 1 --2- 172.21.15.28:0/3647413092 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d5007cd00 0x7f6d5019e0a0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f6d400117e0 tx=0x7f6d40011ba0 comp rx=0 tx=0).stop 2024-05-12T21:58:47.250 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:46.573+0000 7f6d54c00700 1 -- 172.21.15.28:0/3647413092 shutdown_connections 2024-05-12T21:58:47.250 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:46.573+0000 7f6d54c00700 1 --2- 172.21.15.28:0/3647413092 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f6d3403cbc0 0x7f6d3403efb0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:58:47.250 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:46.573+0000 7f6d54c00700 1 --2- 172.21.15.28:0/3647413092 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d5007cd00 0x7f6d5019e0a0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:58:47.251 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:46.573+0000 7f6d54c00700 1 --2- 172.21.15.28:0/3647413092 >> [v2:172.21.15.28:6800/3760499540,v1:172.21.15.28:6801/3760499540] conn(0x7f6d34039100 0x7f6d3403b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:58:47.251 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:46.573+0000 7f6d54c00700 1 -- 172.21.15.28:0/3647413092 >> 172.21.15.28:0/3647413092 conn(0x7f6d501034e0 msgr2=0x7f6d50104000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T21:58:47.251 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:46.573+0000 7f6d54c00700 1 -- 172.21.15.28:0/3647413092 shutdown_connections 2024-05-12T21:58:47.251 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:46.573+0000 7f6d54c00700 1 -- 172.21.15.28:0/3647413092 wait complete. 2024-05-12T21:58:47.251 INFO:teuthology.orchestra.run.smithi028.stdout:mgr epoch 9 is available 2024-05-12T21:58:47.251 INFO:teuthology.orchestra.run.smithi028.stdout:Generating a dashboard self-signed certificate... 2024-05-12T21:58:47.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:58:47 smithi028 ceph-mon[29847]: [12/May/2024:21:58:46] ENGINE Serving on https://172.21.15.28:7150 2024-05-12T21:58:47.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:58:47 smithi028 ceph-mon[29847]: [12/May/2024:21:58:46] ENGINE Bus STARTED 2024-05-12T21:58:47.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:58:47 smithi028 ceph-mon[29847]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-05-12T21:58:47.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:58:47 smithi028 ceph-mon[29847]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-05-12T21:58:48.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:58:48 smithi028 ceph-mon[29847]: mgrmap e12: x(active, since 2s) 2024-05-12T21:58:48.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:58:48 smithi028 ceph-mon[29847]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-05-12T21:58:48.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:58:48 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:58:48.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:58:48 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:58:49.192 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-05-12T21:58:49.192 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:48.020+0000 7f7509600700 1 Processor -- start 2024-05-12T21:58:49.192 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:48.020+0000 7f7509600700 1 -- start start 2024-05-12T21:58:49.192 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:48.020+0000 7f7509600700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7504107f50 0x7f7504108390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T21:58:49.192 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:48.020+0000 7f7509600700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f75041088d0 con 0x7f7504107f50 2024-05-12T21:58:49.192 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:48.021+0000 7f7502a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7504107f50 0x7f7504108390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T21:58:49.192 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:48.021+0000 7f7502a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7504107f50 0x7f7504108390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57116/0 (socket says 172.21.15.28:57116) 2024-05-12T21:58:49.192 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:48.021+0000 7f7502a00700 1 -- 172.21.15.28:0/4082311225 learned_addr learned my addr 172.21.15.28:0/4082311225 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T21:58:49.192 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:48.021+0000 7f7502a00700 1 -- 172.21.15.28:0/4082311225 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75041090e0 con 0x7f7504107f50 2024-05-12T21:58:49.192 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:48.022+0000 7f7502a00700 1 --2- 172.21.15.28:0/4082311225 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7504107f50 0x7f7504108390 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f74f40098a0 tx=0x7f74f4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17be7cfcd705f224 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T21:58:49.192 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:48.022+0000 7f7501600700 1 -- 172.21.15.28:0/4082311225 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f74f4004160 con 0x7f7504107f50 2024-05-12T21:58:49.193 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:48.022+0000 7f7501600700 1 -- 172.21.15.28:0/4082311225 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f74f4004300 con 0x7f7504107f50 2024-05-12T21:58:49.193 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:48.022+0000 7f7501600700 1 -- 172.21.15.28:0/4082311225 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f74f4011b20 con 0x7f7504107f50 2024-05-12T21:58:49.193 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:48.023+0000 7f7509600700 1 -- 172.21.15.28:0/4082311225 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7504107f50 msgr2=0x7f7504108390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T21:58:49.193 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:48.023+0000 7f7509600700 1 --2- 172.21.15.28:0/4082311225 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7504107f50 0x7f7504108390 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f74f40098a0 tx=0x7f74f4009bb0 comp rx=0 tx=0).stop 2024-05-12T21:58:49.193 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:48.023+0000 7f7509600700 1 -- 172.21.15.28:0/4082311225 shutdown_connections 2024-05-12T21:58:49.193 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:48.023+0000 7f7509600700 1 --2- 172.21.15.28:0/4082311225 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7504107f50 0x7f7504108390 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:58:49.193 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:48.023+0000 7f7509600700 1 -- 172.21.15.28:0/4082311225 >> 172.21.15.28:0/4082311225 conn(0x7f7504103560 msgr2=0x7f7504105980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T21:58:49.193 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:48.023+0000 7f7509600700 1 -- 172.21.15.28:0/4082311225 shutdown_connections 2024-05-12T21:58:49.193 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:48.023+0000 7f7509600700 1 -- 172.21.15.28:0/4082311225 wait complete. 2024-05-12T21:58:49.193 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:48.024+0000 7f7509600700 1 Processor -- start 2024-05-12T21:58:49.193 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:48.024+0000 7f7509600700 1 -- start start 2024-05-12T21:58:49.193 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:48.024+0000 7f7509600700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7504107f50 0x7f750419e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T21:58:49.193 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:48.024+0000 7f7509600700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f750410c3a0 con 0x7f7504107f50 2024-05-12T21:58:49.193 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:48.025+0000 7f7502a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7504107f50 0x7f750419e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T21:58:49.193 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:48.025+0000 7f7502a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7504107f50 0x7f750419e0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57128/0 (socket says 172.21.15.28:57128) 2024-05-12T21:58:49.194 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:48.025+0000 7f7502a00700 1 -- 172.21.15.28:0/2528207872 learned_addr learned my addr 172.21.15.28:0/2528207872 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T21:58:49.194 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:48.025+0000 7f7502a00700 1 -- 172.21.15.28:0/2528207872 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f750419e610 con 0x7f7504107f50 2024-05-12T21:58:49.194 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:48.025+0000 7f7502a00700 1 --2- 172.21.15.28:0/2528207872 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7504107f50 0x7f750419e0d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f74f40117e0 tx=0x7f74f4011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T21:58:49.194 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:48.026+0000 7f74fb400700 1 -- 172.21.15.28:0/2528207872 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f74f4003bf0 con 0x7f7504107f50 2024-05-12T21:58:49.194 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:48.026+0000 7f74fb400700 1 -- 172.21.15.28:0/2528207872 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f74f401a910 con 0x7f7504107f50 2024-05-12T21:58:49.194 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:48.026+0000 7f74fb400700 1 -- 172.21.15.28:0/2528207872 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f74f4024970 con 0x7f7504107f50 2024-05-12T21:58:49.194 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:48.026+0000 7f7509600700 1 -- 172.21.15.28:0/2528207872 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f750419e8a0 con 0x7f7504107f50 2024-05-12T21:58:49.194 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:48.026+0000 7f7509600700 1 -- 172.21.15.28:0/2528207872 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f750419ed00 con 0x7f7504107f50 2024-05-12T21:58:49.194 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:48.027+0000 7f7509600700 1 -- 172.21.15.28:0/2528207872 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7504063980 con 0x7f7504107f50 2024-05-12T21:58:49.194 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:48.027+0000 7f74fb400700 1 -- 172.21.15.28:0/2528207872 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 12) ==== 46300+0+0 (secure 0 0 0) 0x7f74f4022020 con 0x7f7504107f50 2024-05-12T21:58:49.195 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:48.028+0000 7f74fb400700 1 --2- 172.21.15.28:0/2528207872 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f74e8039100 0x7f74e803b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T21:58:49.195 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:48.028+0000 7f74fb400700 1 -- 172.21.15.28:0/2528207872 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f74f404d580 con 0x7f7504107f50 2024-05-12T21:58:49.195 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:48.028+0000 7f7502000700 1 --2- 172.21.15.28:0/2528207872 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f74e8039100 0x7f74e803b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T21:58:49.195 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:48.029+0000 7f7502000700 1 --2- 172.21.15.28:0/2528207872 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f74e8039100 0x7f74e803b5c0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f74f0006fd0 tx=0x7f74f0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T21:58:49.195 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:48.033+0000 7f74fb400700 1 -- 172.21.15.28:0/2528207872 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f74f40212c0 con 0x7f7504107f50 2024-05-12T21:58:49.195 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:48.259+0000 7f7509600700 1 -- 172.21.15.28:0/2528207872 --> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) -- 0x7f75041046d0 con 0x7f74e8039100 2024-05-12T21:58:49.195 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:48.449+0000 7f74fb400700 1 -- 172.21.15.28:0/2528207872 <== mgr.14152 v2:172.21.15.28:6800/3123328651 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+32 (secure 0 0 0) 0x7f75041046d0 con 0x7f74e8039100 2024-05-12T21:58:49.195 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:48.452+0000 7f7509600700 1 -- 172.21.15.28:0/2528207872 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f74e8039100 msgr2=0x7f74e803b5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T21:58:49.195 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:48.452+0000 7f7509600700 1 --2- 172.21.15.28:0/2528207872 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f74e8039100 0x7f74e803b5c0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f74f0006fd0 tx=0x7f74f0006e40 comp rx=0 tx=0).stop 2024-05-12T21:58:49.195 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:48.452+0000 7f7509600700 1 -- 172.21.15.28:0/2528207872 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7504107f50 msgr2=0x7f750419e0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T21:58:49.195 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:48.452+0000 7f7509600700 1 --2- 172.21.15.28:0/2528207872 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7504107f50 0x7f750419e0d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f74f40117e0 tx=0x7f74f4011ba0 comp rx=0 tx=0).stop 2024-05-12T21:58:49.195 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:48.453+0000 7f7509600700 1 -- 172.21.15.28:0/2528207872 shutdown_connections 2024-05-12T21:58:49.195 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:48.453+0000 7f7509600700 1 --2- 172.21.15.28:0/2528207872 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f74e8039100 0x7f74e803b5c0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:58:49.195 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:48.453+0000 7f7509600700 1 --2- 172.21.15.28:0/2528207872 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7504107f50 0x7f750419e0d0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:58:49.195 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:48.453+0000 7f7509600700 1 -- 172.21.15.28:0/2528207872 >> 172.21.15.28:0/2528207872 conn(0x7f7504103560 msgr2=0x7f7504103fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T21:58:49.196 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:48.453+0000 7f7509600700 1 -- 172.21.15.28:0/2528207872 shutdown_connections 2024-05-12T21:58:49.196 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:48.453+0000 7f7509600700 1 -- 172.21.15.28:0/2528207872 wait complete. 2024-05-12T21:58:49.196 INFO:teuthology.orchestra.run.smithi028.stdout:Creating initial admin user... 2024-05-12T21:58:50.092 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:58:49 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:58:51.112 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$Hnncy/5WyrdOWTfPdNnBueODqkqSxmEVUBFSG9GQsUC0Vd/njY8P6", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1715551130, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-05-12T21:58:51.112 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:49.816+0000 7f35b7e00700 1 Processor -- start 2024-05-12T21:58:51.112 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:49.817+0000 7f35b7e00700 1 -- start start 2024-05-12T21:58:51.112 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:49.817+0000 7f35b7e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35b8108850 0x7f35b810ac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T21:58:51.112 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:49.817+0000 7f35b7e00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f35b807be40 con 0x7f35b8108850 2024-05-12T21:58:51.112 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:49.817+0000 7f35b6a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35b8108850 0x7f35b810ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T21:58:51.112 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:49.817+0000 7f35b6a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35b8108850 0x7f35b810ac40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57136/0 (socket says 172.21.15.28:57136) 2024-05-12T21:58:51.112 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:49.817+0000 7f35b6a00700 1 -- 172.21.15.28:0/1845463093 learned_addr learned my addr 172.21.15.28:0/1845463093 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T21:58:51.112 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:49.818+0000 7f35b6a00700 1 -- 172.21.15.28:0/1845463093 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35b810b180 con 0x7f35b8108850 2024-05-12T21:58:51.112 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:49.818+0000 7f35b6a00700 1 --2- 172.21.15.28:0/1845463093 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35b8108850 0x7f35b810ac40 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f35a80098a0 tx=0x7f35a8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7aa3dd6b4deb554 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T21:58:51.112 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:49.818+0000 7f35b5600700 1 -- 172.21.15.28:0/1845463093 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f35a8004160 con 0x7f35b8108850 2024-05-12T21:58:51.113 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:49.819+0000 7f35b5600700 1 -- 172.21.15.28:0/1845463093 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f35a8004300 con 0x7f35b8108850 2024-05-12T21:58:51.113 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:49.819+0000 7f35b5600700 1 -- 172.21.15.28:0/1845463093 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f35a8011b20 con 0x7f35b8108850 2024-05-12T21:58:51.113 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:49.819+0000 7f35b7e00700 1 -- 172.21.15.28:0/1845463093 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35b8108850 msgr2=0x7f35b810ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T21:58:51.113 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:49.819+0000 7f35b7e00700 1 --2- 172.21.15.28:0/1845463093 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35b8108850 0x7f35b810ac40 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f35a80098a0 tx=0x7f35a8009bb0 comp rx=0 tx=0).stop 2024-05-12T21:58:51.113 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:49.819+0000 7f35b7e00700 1 -- 172.21.15.28:0/1845463093 shutdown_connections 2024-05-12T21:58:51.113 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:49.819+0000 7f35b7e00700 1 --2- 172.21.15.28:0/1845463093 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35b8108850 0x7f35b810ac40 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:58:51.113 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:49.819+0000 7f35b7e00700 1 -- 172.21.15.28:0/1845463093 >> 172.21.15.28:0/1845463093 conn(0x7f35b8102280 msgr2=0x7f35b81046c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T21:58:51.113 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:49.819+0000 7f35b7e00700 1 -- 172.21.15.28:0/1845463093 shutdown_connections 2024-05-12T21:58:51.113 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:49.820+0000 7f35b7e00700 1 -- 172.21.15.28:0/1845463093 wait complete. 2024-05-12T21:58:51.113 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:49.821+0000 7f35b7e00700 1 Processor -- start 2024-05-12T21:58:51.113 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:49.821+0000 7f35b7e00700 1 -- start start 2024-05-12T21:58:51.113 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:49.822+0000 7f35b7e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35b8108850 0x7f35b81080d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T21:58:51.113 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:49.822+0000 7f35b7e00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f35b810c1f0 con 0x7f35b8108850 2024-05-12T21:58:51.113 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:49.822+0000 7f35b6a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35b8108850 0x7f35b81080d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T21:58:51.113 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:49.822+0000 7f35b6a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35b8108850 0x7f35b81080d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57152/0 (socket says 172.21.15.28:57152) 2024-05-12T21:58:51.114 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:49.822+0000 7f35b6a00700 1 -- 172.21.15.28:0/740851383 learned_addr learned my addr 172.21.15.28:0/740851383 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T21:58:51.114 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:49.822+0000 7f35b6a00700 1 -- 172.21.15.28:0/740851383 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35b8108610 con 0x7f35b8108850 2024-05-12T21:58:51.114 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:49.823+0000 7f35b6a00700 1 --2- 172.21.15.28:0/740851383 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35b8108850 0x7f35b81080d0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f35a80117e0 tx=0x7f35a8011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T21:58:51.114 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:49.823+0000 7f35af400700 1 -- 172.21.15.28:0/740851383 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f35a8003bf0 con 0x7f35b8108850 2024-05-12T21:58:51.114 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:49.823+0000 7f35af400700 1 -- 172.21.15.28:0/740851383 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f35a801a910 con 0x7f35b8108850 2024-05-12T21:58:51.114 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:49.823+0000 7f35af400700 1 -- 172.21.15.28:0/740851383 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f35a8024970 con 0x7f35b8108850 2024-05-12T21:58:51.114 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:49.823+0000 7f35b7e00700 1 -- 172.21.15.28:0/740851383 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35b8106800 con 0x7f35b8108850 2024-05-12T21:58:51.114 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:49.823+0000 7f35b7e00700 1 -- 172.21.15.28:0/740851383 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35b8106cc0 con 0x7f35b8108850 2024-05-12T21:58:51.114 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:49.824+0000 7f35af400700 1 -- 172.21.15.28:0/740851383 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 12) ==== 46300+0+0 (secure 0 0 0) 0x7f35a8022020 con 0x7f35b8108850 2024-05-12T21:58:51.114 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:49.825+0000 7f35af400700 1 --2- 172.21.15.28:0/740851383 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f359c039050 0x7f359c03b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T21:58:51.114 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:49.825+0000 7f35af400700 1 -- 172.21.15.28:0/740851383 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f35a804d580 con 0x7f35b8108850 2024-05-12T21:58:51.114 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:49.825+0000 7f35b6000700 1 --2- 172.21.15.28:0/740851383 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f359c039050 0x7f359c03b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T21:58:51.114 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:49.825+0000 7f35b7e00700 1 -- 172.21.15.28:0/740851383 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f35b80694a0 con 0x7f35b8108850 2024-05-12T21:58:51.115 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:49.826+0000 7f35b6000700 1 --2- 172.21.15.28:0/740851383 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f359c039050 0x7f359c03b510 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f35a4006fd0 tx=0x7f35a4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T21:58:51.115 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:49.831+0000 7f35af400700 1 -- 172.21.15.28:0/740851383 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f35a8011e20 con 0x7f35b8108850 2024-05-12T21:58:51.115 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:50.091+0000 7f35b7e00700 1 -- 172.21.15.28:0/740851383 --> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) -- 0x7f35b8107340 con 0x7f359c039050 2024-05-12T21:58:51.115 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:50.360+0000 7f35af400700 1 -- 172.21.15.28:0/740851383 <== mgr.14152 v2:172.21.15.28:6800/3123328651 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+252 (secure 0 0 0) 0x7f35b8107340 con 0x7f359c039050 2024-05-12T21:58:51.115 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:50.363+0000 7f35b7e00700 1 -- 172.21.15.28:0/740851383 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f359c039050 msgr2=0x7f359c03b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T21:58:51.115 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:50.363+0000 7f35b7e00700 1 --2- 172.21.15.28:0/740851383 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f359c039050 0x7f359c03b510 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f35a4006fd0 tx=0x7f35a4006e40 comp rx=0 tx=0).stop 2024-05-12T21:58:51.115 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:50.363+0000 7f35b7e00700 1 -- 172.21.15.28:0/740851383 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35b8108850 msgr2=0x7f35b81080d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T21:58:51.115 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:50.363+0000 7f35b7e00700 1 --2- 172.21.15.28:0/740851383 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35b8108850 0x7f35b81080d0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f35a80117e0 tx=0x7f35a8011c10 comp rx=0 tx=0).stop 2024-05-12T21:58:51.115 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:50.363+0000 7f35b7e00700 1 -- 172.21.15.28:0/740851383 shutdown_connections 2024-05-12T21:58:51.115 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:50.363+0000 7f35b7e00700 1 --2- 172.21.15.28:0/740851383 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f359c039050 0x7f359c03b510 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:58:51.115 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:50.363+0000 7f35b7e00700 1 --2- 172.21.15.28:0/740851383 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35b8108850 0x7f35b81080d0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:58:51.115 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:50.363+0000 7f35b7e00700 1 -- 172.21.15.28:0/740851383 >> 172.21.15.28:0/740851383 conn(0x7f35b8102280 msgr2=0x7f35b8102de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T21:58:51.115 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:50.364+0000 7f35b7e00700 1 -- 172.21.15.28:0/740851383 shutdown_connections 2024-05-12T21:58:51.115 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:50.364+0000 7f35b7e00700 1 -- 172.21.15.28:0/740851383 wait complete. 2024-05-12T21:58:51.116 INFO:teuthology.orchestra.run.smithi028.stdout:Fetching dashboard port number... 2024-05-12T21:58:51.639 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:58:51 smithi028 ceph-mon[29847]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-12T21:58:51.639 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:58:51 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:58:51.639 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:58:51 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:58:52.668 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:58:52 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/124854366' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-05-12T21:58:52.678 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 8443 2024-05-12T21:58:52.679 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:51.712+0000 7f31e2600700 1 Processor -- start 2024-05-12T21:58:52.679 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:51.712+0000 7f31e2600700 1 -- start start 2024-05-12T21:58:52.679 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:51.713+0000 7f31e2600700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31dc108a00 0x7f31dc10adf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T21:58:52.680 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:51.713+0000 7f31e2600700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f31dc07be40 con 0x7f31dc108a00 2024-05-12T21:58:52.680 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:51.713+0000 7f31db400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31dc108a00 0x7f31dc10adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T21:58:52.680 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:51.713+0000 7f31db400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31dc108a00 0x7f31dc10adf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57164/0 (socket says 172.21.15.28:57164) 2024-05-12T21:58:52.680 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:51.713+0000 7f31db400700 1 -- 172.21.15.28:0/1456030105 learned_addr learned my addr 172.21.15.28:0/1456030105 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T21:58:52.681 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:51.714+0000 7f31db400700 1 -- 172.21.15.28:0/1456030105 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31dc10b330 con 0x7f31dc108a00 2024-05-12T21:58:52.681 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:51.714+0000 7f31db400700 1 --2- 172.21.15.28:0/1456030105 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31dc108a00 0x7f31dc10adf0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f31cc009e80 tx=0x7f31cc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3b2b8fee657c2d2 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T21:58:52.681 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:51.714+0000 7f31da000700 1 -- 172.21.15.28:0/1456030105 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f31cc017070 con 0x7f31dc108a00 2024-05-12T21:58:52.681 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:51.714+0000 7f31da000700 1 -- 172.21.15.28:0/1456030105 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f31cc00b8c0 con 0x7f31dc108a00 2024-05-12T21:58:52.682 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:51.714+0000 7f31da000700 1 -- 172.21.15.28:0/1456030105 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f31cc0045c0 con 0x7f31dc108a00 2024-05-12T21:58:52.682 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:51.715+0000 7f31e2600700 1 -- 172.21.15.28:0/1456030105 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31dc108a00 msgr2=0x7f31dc10adf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T21:58:52.682 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:51.715+0000 7f31e2600700 1 --2- 172.21.15.28:0/1456030105 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31dc108a00 0x7f31dc10adf0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f31cc009e80 tx=0x7f31cc00b330 comp rx=0 tx=0).stop 2024-05-12T21:58:52.682 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:51.716+0000 7f31e2600700 1 -- 172.21.15.28:0/1456030105 shutdown_connections 2024-05-12T21:58:52.683 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:51.716+0000 7f31e2600700 1 --2- 172.21.15.28:0/1456030105 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31dc108a00 0x7f31dc10adf0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:58:52.683 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:51.716+0000 7f31e2600700 1 -- 172.21.15.28:0/1456030105 >> 172.21.15.28:0/1456030105 conn(0x7f31dc102450 msgr2=0x7f31dc104870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T21:58:52.683 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:51.716+0000 7f31e2600700 1 -- 172.21.15.28:0/1456030105 shutdown_connections 2024-05-12T21:58:52.683 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:51.716+0000 7f31e2600700 1 -- 172.21.15.28:0/1456030105 wait complete. 2024-05-12T21:58:52.683 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:51.717+0000 7f31e2600700 1 Processor -- start 2024-05-12T21:58:52.683 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:51.718+0000 7f31e2600700 1 -- start start 2024-05-12T21:58:52.684 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:51.718+0000 7f31e2600700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31dc108a00 0x7f31dc19e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T21:58:52.684 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:51.718+0000 7f31e2600700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f31dc10c3a0 con 0x7f31dc108a00 2024-05-12T21:58:52.684 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:51.718+0000 7f31db400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31dc108a00 0x7f31dc19e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T21:58:52.685 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:51.718+0000 7f31db400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31dc108a00 0x7f31dc19e180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57176/0 (socket says 172.21.15.28:57176) 2024-05-12T21:58:52.685 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:51.718+0000 7f31db400700 1 -- 172.21.15.28:0/124854366 learned_addr learned my addr 172.21.15.28:0/124854366 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T21:58:52.685 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:51.719+0000 7f31db400700 1 -- 172.21.15.28:0/124854366 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31dc19e6c0 con 0x7f31dc108a00 2024-05-12T21:58:52.686 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:51.719+0000 7f31db400700 1 --2- 172.21.15.28:0/124854366 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31dc108a00 0x7f31dc19e180 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f31cc004350 tx=0x7f31cc00bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T21:58:52.686 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:51.719+0000 7f31d3e00700 1 -- 172.21.15.28:0/124854366 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f31cc020020 con 0x7f31dc108a00 2024-05-12T21:58:52.686 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:51.719+0000 7f31d3e00700 1 -- 172.21.15.28:0/124854366 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f31cc003d90 con 0x7f31dc108a00 2024-05-12T21:58:52.686 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:51.719+0000 7f31d3e00700 1 -- 172.21.15.28:0/124854366 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f31cc017040 con 0x7f31dc108a00 2024-05-12T21:58:52.686 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:51.719+0000 7f31e2600700 1 -- 172.21.15.28:0/124854366 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31dc19e950 con 0x7f31dc108a00 2024-05-12T21:58:52.687 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:51.720+0000 7f31e2600700 1 -- 172.21.15.28:0/124854366 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31dc1a1640 con 0x7f31dc108a00 2024-05-12T21:58:52.687 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:51.721+0000 7f31d3e00700 1 -- 172.21.15.28:0/124854366 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 12) ==== 46300+0+0 (secure 0 0 0) 0x7f31cc023020 con 0x7f31dc108a00 2024-05-12T21:58:52.687 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:51.721+0000 7f31e2600700 1 -- 172.21.15.28:0/124854366 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f31dc063980 con 0x7f31dc108a00 2024-05-12T21:58:52.687 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:51.721+0000 7f31d3e00700 1 --2- 172.21.15.28:0/124854366 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f31c00390b0 0x7f31c003b570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T21:58:52.687 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:51.721+0000 7f31d3e00700 1 -- 172.21.15.28:0/124854366 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f31cc00f050 con 0x7f31dc108a00 2024-05-12T21:58:52.688 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:51.722+0000 7f31daa00700 1 --2- 172.21.15.28:0/124854366 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f31c00390b0 0x7f31c003b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T21:58:52.689 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:51.722+0000 7f31daa00700 1 --2- 172.21.15.28:0/124854366 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f31c00390b0 0x7f31c003b570 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f31c8006fd0 tx=0x7f31c8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T21:58:52.690 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:51.732+0000 7f31d3e00700 1 -- 172.21.15.28:0/124854366 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f31cc02a020 con 0x7f31dc108a00 2024-05-12T21:58:52.690 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:51.974+0000 7f31e2600700 1 -- 172.21.15.28:0/124854366 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) -- 0x7f31dc1a1c10 con 0x7f31dc108a00 2024-05-12T21:58:52.690 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:51.974+0000 7f31d3e00700 1 -- 172.21.15.28:0/124854366 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) ==== 112+0+5 (secure 0 0 0) 0x7f31cc0124e0 con 0x7f31dc108a00 2024-05-12T21:58:52.690 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:51.978+0000 7f31e2600700 1 -- 172.21.15.28:0/124854366 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f31c00390b0 msgr2=0x7f31c003b570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T21:58:52.690 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:51.978+0000 7f31e2600700 1 --2- 172.21.15.28:0/124854366 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f31c00390b0 0x7f31c003b570 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f31c8006fd0 tx=0x7f31c8006e40 comp rx=0 tx=0).stop 2024-05-12T21:58:52.690 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:51.978+0000 7f31e2600700 1 -- 172.21.15.28:0/124854366 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31dc108a00 msgr2=0x7f31dc19e180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T21:58:52.690 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:51.978+0000 7f31e2600700 1 --2- 172.21.15.28:0/124854366 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31dc108a00 0x7f31dc19e180 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f31cc004350 tx=0x7f31cc00bd50 comp rx=0 tx=0).stop 2024-05-12T21:58:52.690 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:51.978+0000 7f31e2600700 1 -- 172.21.15.28:0/124854366 shutdown_connections 2024-05-12T21:58:52.690 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:51.978+0000 7f31e2600700 1 --2- 172.21.15.28:0/124854366 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f31c00390b0 0x7f31c003b570 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:58:52.690 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:51.978+0000 7f31e2600700 1 --2- 172.21.15.28:0/124854366 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31dc108a00 0x7f31dc19e180 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:58:52.690 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:51.978+0000 7f31e2600700 1 -- 172.21.15.28:0/124854366 >> 172.21.15.28:0/124854366 conn(0x7f31dc102450 msgr2=0x7f31dc102f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T21:58:52.690 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:51.979+0000 7f31e2600700 1 -- 172.21.15.28:0/124854366 shutdown_connections 2024-05-12T21:58:52.690 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:51.979+0000 7f31e2600700 1 -- 172.21.15.28:0/124854366 wait complete. 2024-05-12T21:58:52.720 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-05-12T21:58:52.721 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout disabled 2024-05-12T21:58:52.740 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-05-12T21:58:52.741 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout inactive 2024-05-12T21:58:52.741 INFO:teuthology.orchestra.run.smithi028.stdout:firewalld.service is not enabled 2024-05-12T21:58:52.741 INFO:teuthology.orchestra.run.smithi028.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-05-12T21:58:52.743 INFO:teuthology.orchestra.run.smithi028.stdout:Ceph Dashboard is now available at: 2024-05-12T21:58:52.743 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-12T21:58:52.743 INFO:teuthology.orchestra.run.smithi028.stdout: URL: https://smithi028.front.sepia.ceph.com:8443/ 2024-05-12T21:58:52.744 INFO:teuthology.orchestra.run.smithi028.stdout: User: admin 2024-05-12T21:58:52.744 INFO:teuthology.orchestra.run.smithi028.stdout: Password: 14qtdtkia7 2024-05-12T21:58:52.744 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-12T21:58:52.744 INFO:teuthology.orchestra.run.smithi028.stdout:Saving cluster configuration to /var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/config directory 2024-05-12T21:58:52.745 INFO:teuthology.orchestra.run.smithi028.stdout:Enabling autotune for osd_memory_target 2024-05-12T21:58:53.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:58:53 smithi028 ceph-mon[29847]: mgrmap e13: x(active, since 6s) 2024-05-12T21:58:54.936 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:53.914+0000 7f964b200700 1 Processor -- start 2024-05-12T21:58:54.937 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:53.914+0000 7f964b200700 1 -- start start 2024-05-12T21:58:54.937 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:53.914+0000 7f964b200700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f964407de10 0x7f964407c270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T21:58:54.937 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:53.915+0000 7f964b200700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f964407c840 con 0x7f964407de10 2024-05-12T21:58:54.937 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:53.915+0000 7f9649e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f964407de10 0x7f964407c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T21:58:54.937 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:53.915+0000 7f9649e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f964407de10 0x7f964407c270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57184/0 (socket says 172.21.15.28:57184) 2024-05-12T21:58:54.937 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:53.915+0000 7f9649e00700 1 -- 172.21.15.28:0/481136670 learned_addr learned my addr 172.21.15.28:0/481136670 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T21:58:54.937 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:53.916+0000 7f9649e00700 1 -- 172.21.15.28:0/481136670 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f964407c9c0 con 0x7f964407de10 2024-05-12T21:58:54.937 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:53.916+0000 7f9649e00700 1 --2- 172.21.15.28:0/481136670 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f964407de10 0x7f964407c270 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f96380098a0 tx=0x7f9638009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93a89a73f5c86489 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T21:58:54.937 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:53.916+0000 7f9648a00700 1 -- 172.21.15.28:0/481136670 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9638004160 con 0x7f964407de10 2024-05-12T21:58:54.937 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:53.916+0000 7f9648a00700 1 -- 172.21.15.28:0/481136670 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9638004300 con 0x7f964407de10 2024-05-12T21:58:54.937 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:53.916+0000 7f9648a00700 1 -- 172.21.15.28:0/481136670 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9638011b20 con 0x7f964407de10 2024-05-12T21:58:54.937 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:53.917+0000 7f964b200700 1 -- 172.21.15.28:0/481136670 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f964407de10 msgr2=0x7f964407c270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T21:58:54.937 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:53.917+0000 7f964b200700 1 --2- 172.21.15.28:0/481136670 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f964407de10 0x7f964407c270 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f96380098a0 tx=0x7f9638009bb0 comp rx=0 tx=0).stop 2024-05-12T21:58:54.938 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:53.917+0000 7f964b200700 1 -- 172.21.15.28:0/481136670 shutdown_connections 2024-05-12T21:58:54.938 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:53.917+0000 7f964b200700 1 --2- 172.21.15.28:0/481136670 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f964407de10 0x7f964407c270 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:58:54.942 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:53.917+0000 7f964b200700 1 -- 172.21.15.28:0/481136670 >> 172.21.15.28:0/481136670 conn(0x7f9644103400 msgr2=0x7f9644105840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T21:58:54.942 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:53.917+0000 7f964b200700 1 -- 172.21.15.28:0/481136670 shutdown_connections 2024-05-12T21:58:54.942 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:53.917+0000 7f964b200700 1 -- 172.21.15.28:0/481136670 wait complete. 2024-05-12T21:58:54.942 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:53.918+0000 7f964b200700 1 Processor -- start 2024-05-12T21:58:54.942 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:53.919+0000 7f964b200700 1 -- start start 2024-05-12T21:58:54.942 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:53.919+0000 7f964b200700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f964407de10 0x7f964419df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T21:58:54.942 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:53.919+0000 7f964b200700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f964410e4e0 con 0x7f964407de10 2024-05-12T21:58:54.942 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:53.919+0000 7f9649e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f964407de10 0x7f964419df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T21:58:54.942 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:53.919+0000 7f9649e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f964407de10 0x7f964419df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57188/0 (socket says 172.21.15.28:57188) 2024-05-12T21:58:54.942 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:53.919+0000 7f9649e00700 1 -- 172.21.15.28:0/2145641442 learned_addr learned my addr 172.21.15.28:0/2145641442 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T21:58:54.943 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:53.920+0000 7f9649e00700 1 -- 172.21.15.28:0/2145641442 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f964419e4c0 con 0x7f964407de10 2024-05-12T21:58:54.943 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:53.920+0000 7f9649e00700 1 --2- 172.21.15.28:0/2145641442 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f964407de10 0x7f964419df80 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f9638011b60 tx=0x7f9638011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T21:58:54.943 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:53.920+0000 7f963ea00700 1 -- 172.21.15.28:0/2145641442 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9638003b60 con 0x7f964407de10 2024-05-12T21:58:54.943 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:53.920+0000 7f963ea00700 1 -- 172.21.15.28:0/2145641442 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f963801a910 con 0x7f964407de10 2024-05-12T21:58:54.943 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:53.920+0000 7f963ea00700 1 -- 172.21.15.28:0/2145641442 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9638024930 con 0x7f964407de10 2024-05-12T21:58:54.943 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:53.920+0000 7f964b200700 1 -- 172.21.15.28:0/2145641442 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f964419e750 con 0x7f964407de10 2024-05-12T21:58:54.943 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:53.921+0000 7f964b200700 1 -- 172.21.15.28:0/2145641442 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f964419ebb0 con 0x7f964407de10 2024-05-12T21:58:54.943 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:53.922+0000 7f964b200700 1 -- 172.21.15.28:0/2145641442 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f96440694a0 con 0x7f964407de10 2024-05-12T21:58:54.943 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:53.922+0000 7f963ea00700 1 -- 172.21.15.28:0/2145641442 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f9638022020 con 0x7f964407de10 2024-05-12T21:58:54.943 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:53.922+0000 7f963ea00700 1 --2- 172.21.15.28:0/2145641442 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f962c0391a0 0x7f962c03b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T21:58:54.943 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:53.922+0000 7f963ea00700 1 -- 172.21.15.28:0/2145641442 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f963804d500 con 0x7f964407de10 2024-05-12T21:58:54.943 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:53.923+0000 7f9649400700 1 --2- 172.21.15.28:0/2145641442 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f962c0391a0 0x7f962c03b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T21:58:54.943 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:53.923+0000 7f9649400700 1 --2- 172.21.15.28:0/2145641442 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f962c0391a0 0x7f962c03b660 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f9634006fd0 tx=0x7f9634006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T21:58:54.943 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:53.928+0000 7f963ea00700 1 -- 172.21.15.28:0/2145641442 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f963800b3e0 con 0x7f964407de10 2024-05-12T21:58:54.943 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:54.157+0000 7f964b200700 1 -- 172.21.15.28:0/2145641442 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) -- 0x7f96441a1640 con 0x7f964407de10 2024-05-12T21:58:54.944 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:54.158+0000 7f963ea00700 1 -- 172.21.15.28:0/2145641442 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) ==== 127+0+0 (secure 0 0 0) 0x7f963802dde0 con 0x7f964407de10 2024-05-12T21:58:54.944 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:54.160+0000 7f964b200700 1 -- 172.21.15.28:0/2145641442 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f962c0391a0 msgr2=0x7f962c03b660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T21:58:54.944 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:54.160+0000 7f964b200700 1 --2- 172.21.15.28:0/2145641442 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f962c0391a0 0x7f962c03b660 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f9634006fd0 tx=0x7f9634006e40 comp rx=0 tx=0).stop 2024-05-12T21:58:54.944 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:54.160+0000 7f964b200700 1 -- 172.21.15.28:0/2145641442 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f964407de10 msgr2=0x7f964419df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T21:58:54.944 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:54.160+0000 7f964b200700 1 --2- 172.21.15.28:0/2145641442 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f964407de10 0x7f964419df80 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f9638011b60 tx=0x7f9638011c40 comp rx=0 tx=0).stop 2024-05-12T21:58:54.944 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:54.160+0000 7f964b200700 1 -- 172.21.15.28:0/2145641442 shutdown_connections 2024-05-12T21:58:54.944 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:54.160+0000 7f964b200700 1 --2- 172.21.15.28:0/2145641442 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f962c0391a0 0x7f962c03b660 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:58:54.945 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:54.160+0000 7f964b200700 1 --2- 172.21.15.28:0/2145641442 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f964407de10 0x7f964419df80 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:58:54.945 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:54.160+0000 7f964b200700 1 -- 172.21.15.28:0/2145641442 >> 172.21.15.28:0/2145641442 conn(0x7f9644103400 msgr2=0x7f9644103f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T21:58:54.945 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:54.161+0000 7f964b200700 1 -- 172.21.15.28:0/2145641442 shutdown_connections 2024-05-12T21:58:54.945 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:54.161+0000 7f964b200700 1 -- 172.21.15.28:0/2145641442 wait complete. 2024-05-12T21:58:56.625 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:55.615+0000 7f4c59600700 1 Processor -- start 2024-05-12T21:58:56.625 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:55.615+0000 7f4c59600700 1 -- start start 2024-05-12T21:58:56.625 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:55.616+0000 7f4c59600700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c54106ce0 0x7f4c54107100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T21:58:56.627 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:55.616+0000 7f4c59600700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4c541076d0 con 0x7f4c54106ce0 2024-05-12T21:58:56.627 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:55.616+0000 7f4c53e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c54106ce0 0x7f4c54107100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T21:58:56.627 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:55.616+0000 7f4c53e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c54106ce0 0x7f4c54107100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57196/0 (socket says 172.21.15.28:57196) 2024-05-12T21:58:56.627 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:55.616+0000 7f4c53e00700 1 -- 172.21.15.28:0/1343482030 learned_addr learned my addr 172.21.15.28:0/1343482030 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T21:58:56.627 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:55.617+0000 7f4c53e00700 1 -- 172.21.15.28:0/1343482030 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c54107ee0 con 0x7f4c54106ce0 2024-05-12T21:58:56.627 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:55.617+0000 7f4c53e00700 1 --2- 172.21.15.28:0/1343482030 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c54106ce0 0x7f4c54107100 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f4c440098a0 tx=0x7f4c44009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a61906bc51b0dde8 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T21:58:56.627 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:55.618+0000 7f4c52a00700 1 -- 172.21.15.28:0/1343482030 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4c44004160 con 0x7f4c54106ce0 2024-05-12T21:58:56.628 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:55.618+0000 7f4c52a00700 1 -- 172.21.15.28:0/1343482030 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4c44004300 con 0x7f4c54106ce0 2024-05-12T21:58:56.629 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:55.618+0000 7f4c52a00700 1 -- 172.21.15.28:0/1343482030 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4c44011b20 con 0x7f4c54106ce0 2024-05-12T21:58:56.629 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:55.618+0000 7f4c59600700 1 -- 172.21.15.28:0/1343482030 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c54106ce0 msgr2=0x7f4c54107100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T21:58:56.629 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:55.618+0000 7f4c59600700 1 --2- 172.21.15.28:0/1343482030 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c54106ce0 0x7f4c54107100 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f4c440098a0 tx=0x7f4c44009bb0 comp rx=0 tx=0).stop 2024-05-12T21:58:56.629 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:55.618+0000 7f4c59600700 1 -- 172.21.15.28:0/1343482030 shutdown_connections 2024-05-12T21:58:56.629 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:55.618+0000 7f4c59600700 1 --2- 172.21.15.28:0/1343482030 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c54106ce0 0x7f4c54107100 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:58:56.629 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:55.618+0000 7f4c59600700 1 -- 172.21.15.28:0/1343482030 >> 172.21.15.28:0/1343482030 conn(0x7f4c54102280 msgr2=0x7f4c541046c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T21:58:56.629 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:55.619+0000 7f4c59600700 1 -- 172.21.15.28:0/1343482030 shutdown_connections 2024-05-12T21:58:56.629 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:55.619+0000 7f4c59600700 1 -- 172.21.15.28:0/1343482030 wait complete. 2024-05-12T21:58:56.629 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:55.620+0000 7f4c59600700 1 Processor -- start 2024-05-12T21:58:56.629 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:55.620+0000 7f4c59600700 1 -- start start 2024-05-12T21:58:56.629 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:55.620+0000 7f4c59600700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c54106ce0 0x7f4c5419df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T21:58:56.633 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:55.620+0000 7f4c59600700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4c5410b290 con 0x7f4c54106ce0 2024-05-12T21:58:56.633 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:55.621+0000 7f4c53e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c54106ce0 0x7f4c5419df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T21:58:56.633 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:55.621+0000 7f4c53e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c54106ce0 0x7f4c5419df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57204/0 (socket says 172.21.15.28:57204) 2024-05-12T21:58:56.633 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:55.621+0000 7f4c53e00700 1 -- 172.21.15.28:0/780311130 learned_addr learned my addr 172.21.15.28:0/780311130 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T21:58:56.633 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:55.621+0000 7f4c53e00700 1 -- 172.21.15.28:0/780311130 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c5419e4c0 con 0x7f4c54106ce0 2024-05-12T21:58:56.633 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:55.621+0000 7f4c53e00700 1 --2- 172.21.15.28:0/780311130 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c54106ce0 0x7f4c5419df80 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f4c44000c00 tx=0x7f4c44011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T21:58:56.633 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:55.622+0000 7f4c50c00700 1 -- 172.21.15.28:0/780311130 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4c44003b60 con 0x7f4c54106ce0 2024-05-12T21:58:56.633 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:55.622+0000 7f4c50c00700 1 -- 172.21.15.28:0/780311130 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4c4401a910 con 0x7f4c54106ce0 2024-05-12T21:58:56.633 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:55.622+0000 7f4c59600700 1 -- 172.21.15.28:0/780311130 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c5419e750 con 0x7f4c54106ce0 2024-05-12T21:58:56.633 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:55.622+0000 7f4c50c00700 1 -- 172.21.15.28:0/780311130 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4c44024930 con 0x7f4c54106ce0 2024-05-12T21:58:56.633 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:55.622+0000 7f4c59600700 1 -- 172.21.15.28:0/780311130 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c5419ebb0 con 0x7f4c54106ce0 2024-05-12T21:58:56.634 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:55.623+0000 7f4c50c00700 1 -- 172.21.15.28:0/780311130 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f4c44022020 con 0x7f4c54106ce0 2024-05-12T21:58:56.634 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:55.623+0000 7f4c59600700 1 -- 172.21.15.28:0/780311130 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c540694a0 con 0x7f4c54106ce0 2024-05-12T21:58:56.634 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:55.624+0000 7f4c50c00700 1 --2- 172.21.15.28:0/780311130 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f4c380391a0 0x7f4c3803b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T21:58:56.634 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:55.624+0000 7f4c50c00700 1 -- 172.21.15.28:0/780311130 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f4c4404d490 con 0x7f4c54106ce0 2024-05-12T21:58:56.634 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:55.624+0000 7f4c53400700 1 --2- 172.21.15.28:0/780311130 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f4c380391a0 0x7f4c3803b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T21:58:56.634 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:55.625+0000 7f4c53400700 1 --2- 172.21.15.28:0/780311130 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f4c380391a0 0x7f4c3803b660 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f4c40006fd0 tx=0x7f4c40006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T21:58:56.634 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:55.629+0000 7f4c50c00700 1 -- 172.21.15.28:0/780311130 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f4c4400b3e0 con 0x7f4c54106ce0 2024-05-12T21:58:56.634 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:55.975+0000 7f4c59600700 1 -- 172.21.15.28:0/780311130 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) -- 0x7f4c54075570 con 0x7f4c54106ce0 2024-05-12T21:58:56.634 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:55.978+0000 7f4c50c00700 1 -- 172.21.15.28:0/780311130 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v27)=0 set mgr/dashboard/cluster/status v27) ==== 153+0+0 (secure 0 0 0) 0x7f4c44016260 con 0x7f4c54106ce0 2024-05-12T21:58:56.634 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:55.981+0000 7f4c59600700 1 -- 172.21.15.28:0/780311130 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f4c380391a0 msgr2=0x7f4c3803b660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T21:58:56.634 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:55.981+0000 7f4c59600700 1 --2- 172.21.15.28:0/780311130 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f4c380391a0 0x7f4c3803b660 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f4c40006fd0 tx=0x7f4c40006e40 comp rx=0 tx=0).stop 2024-05-12T21:58:56.634 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:55.981+0000 7f4c59600700 1 -- 172.21.15.28:0/780311130 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c54106ce0 msgr2=0x7f4c5419df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T21:58:56.634 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:55.981+0000 7f4c59600700 1 --2- 172.21.15.28:0/780311130 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c54106ce0 0x7f4c5419df80 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f4c44000c00 tx=0x7f4c44011c10 comp rx=0 tx=0).stop 2024-05-12T21:58:56.634 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:55.981+0000 7f4c59600700 1 -- 172.21.15.28:0/780311130 shutdown_connections 2024-05-12T21:58:56.635 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:55.981+0000 7f4c59600700 1 --2- 172.21.15.28:0/780311130 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f4c380391a0 0x7f4c3803b660 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:58:56.635 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:55.981+0000 7f4c59600700 1 --2- 172.21.15.28:0/780311130 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c54106ce0 0x7f4c5419df80 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:58:56.636 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:55.981+0000 7f4c59600700 1 -- 172.21.15.28:0/780311130 >> 172.21.15.28:0/780311130 conn(0x7f4c54102280 msgr2=0x7f4c54103120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T21:58:56.636 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:55.982+0000 7f4c59600700 1 -- 172.21.15.28:0/780311130 shutdown_connections 2024-05-12T21:58:56.636 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-05-12T21:58:55.982+0000 7f4c59600700 1 -- 172.21.15.28:0/780311130 wait complete. 2024-05-12T21:58:56.636 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-05-12T21:58:56.636 INFO:teuthology.orchestra.run.smithi028.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-05-12T21:58:56.636 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-12T21:58:56.637 INFO:teuthology.orchestra.run.smithi028.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 6b92e7b4-10aa-11ef-bc98-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-05-12T21:58:56.637 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-12T21:58:56.637 INFO:teuthology.orchestra.run.smithi028.stdout:Or, if you are only running a single cluster on this host: 2024-05-12T21:58:56.637 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-12T21:58:56.637 INFO:teuthology.orchestra.run.smithi028.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-05-12T21:58:56.637 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-12T21:58:56.637 INFO:teuthology.orchestra.run.smithi028.stdout:Please consider enabling telemetry to help improve Ceph: 2024-05-12T21:58:56.637 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-12T21:58:56.637 INFO:teuthology.orchestra.run.smithi028.stdout: ceph telemetry on 2024-05-12T21:58:56.637 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-12T21:58:56.637 INFO:teuthology.orchestra.run.smithi028.stdout:For more information see: 2024-05-12T21:58:56.637 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-12T21:58:56.637 INFO:teuthology.orchestra.run.smithi028.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-05-12T21:58:56.637 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-12T21:58:56.637 INFO:teuthology.orchestra.run.smithi028.stdout:Bootstrap complete. 2024-05-12T21:58:56.729 INFO:tasks.cephadm:Fetching config... 2024-05-12T21:58:56.729 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-05-12T21:58:56.729 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-05-12T21:58:56.780 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-05-12T21:58:56.780 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-05-12T21:58:56.780 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-05-12T21:58:56.866 INFO:tasks.cephadm:Fetching mon keyring... 2024-05-12T21:58:56.866 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-05-12T21:58:56.866 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/mon.a/keyring of=/dev/stdout 2024-05-12T21:58:56.998 INFO:tasks.cephadm:Fetching pub ssh key... 2024-05-12T21:58:56.998 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-05-12T21:58:56.998 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-05-12T21:58:57.053 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-05-12T21:58:57.053 DEBUG:teuthology.orchestra.run.smithi028:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC655sRXRKBWw7UC94HKv3vOnjQ7x3I+Fg+CKyLM4otPCS/L+e9W4IJ+bjby77xzjF16TkWqDxtzNBFU4bifqwzBytknKoSxS+G/lhWBRlxuPRPbf9gjAGsCDfmzZKOPdPxELNwnQS23//2j44YpShSdkCCabywLxPOLsEY0CAzzfakm7ts0Qh2tIDOBMeGpAN1jUe8cFu9M9rM/tCwE9Q7JoUHRC9BChJn+Ule+ifwrdDYIiLi0rg4Z6NIl8DxNJYHSPIZb+/VVJ/HfLb6OeCFkOr03342+QLVUVSlTPyX+2VHS05N0SGi/xa66y3uZWLMvp7oAOVULnV/ntWWkUW3JxBSRDHaUqAllPKfczHIDPm7mxL4TOOyTWf++BOUpOcwQnzK1ih7uKw5d/CIRk9dIWggg1Zg9njsQZ3qywyzTOXX98/DG8JZFCEL2iFcc77pCgKx6+QxRPIbkjUMf7q++cAftCCIz5BaEZpllXTwme31W19rE3mDItZz2GTgFlU= ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-05-12T21:58:57.168 INFO:teuthology.orchestra.run.smithi028.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC655sRXRKBWw7UC94HKv3vOnjQ7x3I+Fg+CKyLM4otPCS/L+e9W4IJ+bjby77xzjF16TkWqDxtzNBFU4bifqwzBytknKoSxS+G/lhWBRlxuPRPbf9gjAGsCDfmzZKOPdPxELNwnQS23//2j44YpShSdkCCabywLxPOLsEY0CAzzfakm7ts0Qh2tIDOBMeGpAN1jUe8cFu9M9rM/tCwE9Q7JoUHRC9BChJn+Ule+ifwrdDYIiLi0rg4Z6NIl8DxNJYHSPIZb+/VVJ/HfLb6OeCFkOr03342+QLVUVSlTPyX+2VHS05N0SGi/xa66y3uZWLMvp7oAOVULnV/ntWWkUW3JxBSRDHaUqAllPKfczHIDPm7mxL4TOOyTWf++BOUpOcwQnzK1ih7uKw5d/CIRk9dIWggg1Zg9njsQZ3qywyzTOXX98/DG8JZFCEL2iFcc77pCgKx6+QxRPIbkjUMf7q++cAftCCIz5BaEZpllXTwme31W19rE3mDItZz2GTgFlU= ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968 2024-05-12T21:58:57.211 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:58:56 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/780311130' entity='client.admin' 2024-05-12T21:58:57.227 DEBUG:teuthology.orchestra.run.smithi086:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC655sRXRKBWw7UC94HKv3vOnjQ7x3I+Fg+CKyLM4otPCS/L+e9W4IJ+bjby77xzjF16TkWqDxtzNBFU4bifqwzBytknKoSxS+G/lhWBRlxuPRPbf9gjAGsCDfmzZKOPdPxELNwnQS23//2j44YpShSdkCCabywLxPOLsEY0CAzzfakm7ts0Qh2tIDOBMeGpAN1jUe8cFu9M9rM/tCwE9Q7JoUHRC9BChJn+Ule+ifwrdDYIiLi0rg4Z6NIl8DxNJYHSPIZb+/VVJ/HfLb6OeCFkOr03342+QLVUVSlTPyX+2VHS05N0SGi/xa66y3uZWLMvp7oAOVULnV/ntWWkUW3JxBSRDHaUqAllPKfczHIDPm7mxL4TOOyTWf++BOUpOcwQnzK1ih7uKw5d/CIRk9dIWggg1Zg9njsQZ3qywyzTOXX98/DG8JZFCEL2iFcc77pCgKx6+QxRPIbkjUMf7q++cAftCCIz5BaEZpllXTwme31W19rE3mDItZz2GTgFlU= ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-05-12T21:58:57.267 INFO:teuthology.orchestra.run.smithi086.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC655sRXRKBWw7UC94HKv3vOnjQ7x3I+Fg+CKyLM4otPCS/L+e9W4IJ+bjby77xzjF16TkWqDxtzNBFU4bifqwzBytknKoSxS+G/lhWBRlxuPRPbf9gjAGsCDfmzZKOPdPxELNwnQS23//2j44YpShSdkCCabywLxPOLsEY0CAzzfakm7ts0Qh2tIDOBMeGpAN1jUe8cFu9M9rM/tCwE9Q7JoUHRC9BChJn+Ule+ifwrdDYIiLi0rg4Z6NIl8DxNJYHSPIZb+/VVJ/HfLb6OeCFkOr03342+QLVUVSlTPyX+2VHS05N0SGi/xa66y3uZWLMvp7oAOVULnV/ntWWkUW3JxBSRDHaUqAllPKfczHIDPm7mxL4TOOyTWf++BOUpOcwQnzK1ih7uKw5d/CIRk9dIWggg1Zg9njsQZ3qywyzTOXX98/DG8JZFCEL2iFcc77pCgKx6+QxRPIbkjUMf7q++cAftCCIz5BaEZpllXTwme31W19rE3mDItZz2GTgFlU= ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968 2024-05-12T21:58:57.280 DEBUG:teuthology.orchestra.run.smithi186:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC655sRXRKBWw7UC94HKv3vOnjQ7x3I+Fg+CKyLM4otPCS/L+e9W4IJ+bjby77xzjF16TkWqDxtzNBFU4bifqwzBytknKoSxS+G/lhWBRlxuPRPbf9gjAGsCDfmzZKOPdPxELNwnQS23//2j44YpShSdkCCabywLxPOLsEY0CAzzfakm7ts0Qh2tIDOBMeGpAN1jUe8cFu9M9rM/tCwE9Q7JoUHRC9BChJn+Ule+ifwrdDYIiLi0rg4Z6NIl8DxNJYHSPIZb+/VVJ/HfLb6OeCFkOr03342+QLVUVSlTPyX+2VHS05N0SGi/xa66y3uZWLMvp7oAOVULnV/ntWWkUW3JxBSRDHaUqAllPKfczHIDPm7mxL4TOOyTWf++BOUpOcwQnzK1ih7uKw5d/CIRk9dIWggg1Zg9njsQZ3qywyzTOXX98/DG8JZFCEL2iFcc77pCgKx6+QxRPIbkjUMf7q++cAftCCIz5BaEZpllXTwme31W19rE3mDItZz2GTgFlU= ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-05-12T21:58:57.319 INFO:teuthology.orchestra.run.smithi186.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC655sRXRKBWw7UC94HKv3vOnjQ7x3I+Fg+CKyLM4otPCS/L+e9W4IJ+bjby77xzjF16TkWqDxtzNBFU4bifqwzBytknKoSxS+G/lhWBRlxuPRPbf9gjAGsCDfmzZKOPdPxELNwnQS23//2j44YpShSdkCCabywLxPOLsEY0CAzzfakm7ts0Qh2tIDOBMeGpAN1jUe8cFu9M9rM/tCwE9Q7JoUHRC9BChJn+Ule+ifwrdDYIiLi0rg4Z6NIl8DxNJYHSPIZb+/VVJ/HfLb6OeCFkOr03342+QLVUVSlTPyX+2VHS05N0SGi/xa66y3uZWLMvp7oAOVULnV/ntWWkUW3JxBSRDHaUqAllPKfczHIDPm7mxL4TOOyTWf++BOUpOcwQnzK1ih7uKw5d/CIRk9dIWggg1Zg9njsQZ3qywyzTOXX98/DG8JZFCEL2iFcc77pCgKx6+QxRPIbkjUMf7q++cAftCCIz5BaEZpllXTwme31W19rE3mDItZz2GTgFlU= ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968 2024-05-12T21:58:57.331 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f72fecff68e1d400c4568684327c900485c20d6a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6b92e7b4-10aa-11ef-bc98-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-05-12T21:58:57.776 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/mon.a/config 2024-05-12T21:59:00.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:00.357+0000 7fe5e2800700 1 -- 172.21.15.28:0/3372183914 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5dc103e00 msgr2=0x7fe5dc104220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T21:59:00.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:00.357+0000 7fe5e2800700 1 --2- 172.21.15.28:0/3372183914 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5dc103e00 0x7fe5dc104220 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fe5d000b0a0 tx=0x7fe5d000b3b0 comp rx=0 tx=0).stop 2024-05-12T21:59:00.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:00.357+0000 7fe5e2800700 1 -- 172.21.15.28:0/3372183914 shutdown_connections 2024-05-12T21:59:00.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:00.357+0000 7fe5e2800700 1 --2- 172.21.15.28:0/3372183914 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5dc103e00 0x7fe5dc104220 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:59:00.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:00.357+0000 7fe5e2800700 1 -- 172.21.15.28:0/3372183914 >> 172.21.15.28:0/3372183914 conn(0x7fe5dc0ff380 msgr2=0x7fe5dc1017e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T21:59:00.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:00.357+0000 7fe5e2800700 1 -- 172.21.15.28:0/3372183914 shutdown_connections 2024-05-12T21:59:00.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:00.357+0000 7fe5e2800700 1 -- 172.21.15.28:0/3372183914 wait complete. 2024-05-12T21:59:00.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:00.359+0000 7fe5e2800700 1 Processor -- start 2024-05-12T21:59:00.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:00.359+0000 7fe5e2800700 1 -- start start 2024-05-12T21:59:00.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:00.359+0000 7fe5e2800700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5dc103e00 0x7fe5dc199430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T21:59:00.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:00.359+0000 7fe5e2800700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe5dc109e10 con 0x7fe5dc103e00 2024-05-12T21:59:00.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:00.360+0000 7fe5e1400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5dc103e00 0x7fe5dc199430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T21:59:00.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:00.360+0000 7fe5e1400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5dc103e00 0x7fe5dc199430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35100/0 (socket says 172.21.15.28:35100) 2024-05-12T21:59:00.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:00.360+0000 7fe5e1400700 1 -- 172.21.15.28:0/664075661 learned_addr learned my addr 172.21.15.28:0/664075661 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T21:59:00.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:00.360+0000 7fe5e1400700 1 -- 172.21.15.28:0/664075661 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5dc199970 con 0x7fe5dc103e00 2024-05-12T21:59:00.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:00.360+0000 7fe5e1400700 1 --2- 172.21.15.28:0/664075661 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5dc103e00 0x7fe5dc199430 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fe5d0015040 tx=0x7fe5d0009790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T21:59:00.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:00.361+0000 7fe5da000700 1 -- 172.21.15.28:0/664075661 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe5d0017040 con 0x7fe5dc103e00 2024-05-12T21:59:00.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:00.361+0000 7fe5da000700 1 -- 172.21.15.28:0/664075661 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe5d0022070 con 0x7fe5dc103e00 2024-05-12T21:59:00.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:00.361+0000 7fe5da000700 1 -- 172.21.15.28:0/664075661 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe5d001d5d0 con 0x7fe5dc103e00 2024-05-12T21:59:00.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:00.361+0000 7fe5e2800700 1 -- 172.21.15.28:0/664075661 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5dc199c00 con 0x7fe5dc103e00 2024-05-12T21:59:00.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:00.361+0000 7fe5e2800700 1 -- 172.21.15.28:0/664075661 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5dc19c8f0 con 0x7fe5dc103e00 2024-05-12T21:59:00.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:00.362+0000 7fe5e2800700 1 -- 172.21.15.28:0/664075661 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe5dc10a590 con 0x7fe5dc103e00 2024-05-12T21:59:00.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:00.362+0000 7fe5da000700 1 -- 172.21.15.28:0/664075661 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7fe5d0004000 con 0x7fe5dc103e00 2024-05-12T21:59:00.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:00.363+0000 7fe5da000700 1 --2- 172.21.15.28:0/664075661 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fe5c4039190 0x7fe5c403b650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T21:59:00.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:00.363+0000 7fe5da000700 1 -- 172.21.15.28:0/664075661 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fe5d000f050 con 0x7fe5dc103e00 2024-05-12T21:59:00.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:00.363+0000 7fe5e0a00700 1 --2- 172.21.15.28:0/664075661 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fe5c4039190 0x7fe5c403b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T21:59:00.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:00.364+0000 7fe5e0a00700 1 --2- 172.21.15.28:0/664075661 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fe5c4039190 0x7fe5c403b650 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fe5c8006fd0 tx=0x7fe5c8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T21:59:00.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:00.369+0000 7fe5da000700 1 -- 172.21.15.28:0/664075661 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fe5d0007e20 con 0x7fe5dc103e00 2024-05-12T21:59:00.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:00.603+0000 7fe5e2800700 1 -- 172.21.15.28:0/664075661 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) -- 0x7fe5dc02dc90 con 0x7fe5dc103e00 2024-05-12T21:59:00.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:00.606+0000 7fe5da000700 1 -- 172.21.15.28:0/664075661 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) ==== 125+0+0 (secure 0 0 0) 0x7fe5d001c3a0 con 0x7fe5dc103e00 2024-05-12T21:59:00.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:00.610+0000 7fe5e2800700 1 -- 172.21.15.28:0/664075661 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fe5c4039190 msgr2=0x7fe5c403b650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T21:59:00.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:00.610+0000 7fe5e2800700 1 --2- 172.21.15.28:0/664075661 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fe5c4039190 0x7fe5c403b650 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fe5c8006fd0 tx=0x7fe5c8006e40 comp rx=0 tx=0).stop 2024-05-12T21:59:00.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:00.610+0000 7fe5e2800700 1 -- 172.21.15.28:0/664075661 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5dc103e00 msgr2=0x7fe5dc199430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T21:59:00.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:00.610+0000 7fe5e2800700 1 --2- 172.21.15.28:0/664075661 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5dc103e00 0x7fe5dc199430 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fe5d0015040 tx=0x7fe5d0009790 comp rx=0 tx=0).stop 2024-05-12T21:59:00.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:00.610+0000 7fe5e2800700 1 -- 172.21.15.28:0/664075661 shutdown_connections 2024-05-12T21:59:00.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:00.610+0000 7fe5e2800700 1 --2- 172.21.15.28:0/664075661 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fe5c4039190 0x7fe5c403b650 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:59:00.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:00.610+0000 7fe5e2800700 1 --2- 172.21.15.28:0/664075661 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5dc103e00 0x7fe5dc199430 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:59:00.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:00.610+0000 7fe5e2800700 1 -- 172.21.15.28:0/664075661 >> 172.21.15.28:0/664075661 conn(0x7fe5dc0ff380 msgr2=0x7fe5dc0ffd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T21:59:00.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:00.611+0000 7fe5e2800700 1 -- 172.21.15.28:0/664075661 shutdown_connections 2024-05-12T21:59:00.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:00.611+0000 7fe5e2800700 1 -- 172.21.15.28:0/664075661 wait complete. 2024-05-12T21:59:01.338 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-05-12T21:59:01.338 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f72fecff68e1d400c4568684327c900485c20d6a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6b92e7b4-10aa-11ef-bc98-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-05-12T21:59:01.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:01 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:01.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:01 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:01.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:01 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi028", "name": "osd_memory_target"}]: dispatch 2024-05-12T21:59:01.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:01 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:01.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:01 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T21:59:01.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:01 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:01.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:01 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/664075661' entity='client.admin' 2024-05-12T21:59:01.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:01 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T21:59:01.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:01 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:01.789 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/mon.a/config 2024-05-12T21:59:03.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:03.756+0000 7f873c200700 1 -- 172.21.15.28:0/3975815468 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8734101d80 msgr2=0x7f87341021a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T21:59:03.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:03.756+0000 7f873c200700 1 --2- 172.21.15.28:0/3975815468 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8734101d80 0x7f87341021a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f8728009910 tx=0x7f8728009c20 comp rx=0 tx=0).stop 2024-05-12T21:59:03.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:03.756+0000 7f873c200700 1 -- 172.21.15.28:0/3975815468 shutdown_connections 2024-05-12T21:59:03.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:03.756+0000 7f873c200700 1 --2- 172.21.15.28:0/3975815468 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8734101d80 0x7f87341021a0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:59:03.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:03.756+0000 7f873c200700 1 -- 172.21.15.28:0/3975815468 >> 172.21.15.28:0/3975815468 conn(0x7f87340fd320 msgr2=0x7f87340ff760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T21:59:03.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:03.757+0000 7f873c200700 1 -- 172.21.15.28:0/3975815468 shutdown_connections 2024-05-12T21:59:03.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:03.757+0000 7f873c200700 1 -- 172.21.15.28:0/3975815468 wait complete. 2024-05-12T21:59:03.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:03.758+0000 7f873c200700 1 Processor -- start 2024-05-12T21:59:03.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:03.758+0000 7f873c200700 1 -- start start 2024-05-12T21:59:03.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:03.758+0000 7f873c200700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8734101d80 0x7f8734197490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T21:59:03.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:03.758+0000 7f873c200700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8734107e10 con 0x7f8734101d80 2024-05-12T21:59:03.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:03.759+0000 7f8739800700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8734101d80 0x7f8734197490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T21:59:03.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:03.759+0000 7f8739800700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8734101d80 0x7f8734197490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35110/0 (socket says 172.21.15.28:35110) 2024-05-12T21:59:03.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:03.759+0000 7f8739800700 1 -- 172.21.15.28:0/2999818970 learned_addr learned my addr 172.21.15.28:0/2999818970 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T21:59:03.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:03.759+0000 7f8739800700 1 -- 172.21.15.28:0/2999818970 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87341979d0 con 0x7f8734101d80 2024-05-12T21:59:03.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:03.759+0000 7f8739800700 1 --2- 172.21.15.28:0/2999818970 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8734101d80 0x7f8734197490 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f8728005850 tx=0x7f8728005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T21:59:03.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:03.760+0000 7f8732000700 1 -- 172.21.15.28:0/2999818970 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8728026070 con 0x7f8734101d80 2024-05-12T21:59:03.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:03.760+0000 7f8732000700 1 -- 172.21.15.28:0/2999818970 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f87280044a0 con 0x7f8734101d80 2024-05-12T21:59:03.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:03.760+0000 7f873c200700 1 -- 172.21.15.28:0/2999818970 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8734197c60 con 0x7f8734101d80 2024-05-12T21:59:03.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:03.760+0000 7f8732000700 1 -- 172.21.15.28:0/2999818970 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8728003ab0 con 0x7f8734101d80 2024-05-12T21:59:03.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:03.760+0000 7f873c200700 1 -- 172.21.15.28:0/2999818970 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f873419a950 con 0x7f8734101d80 2024-05-12T21:59:03.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:03.761+0000 7f8732000700 1 -- 172.21.15.28:0/2999818970 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f8728029070 con 0x7f8734101d80 2024-05-12T21:59:03.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:03.761+0000 7f873c200700 1 -- 172.21.15.28:0/2999818970 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8734108590 con 0x7f8734101d80 2024-05-12T21:59:03.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:03.762+0000 7f8732000700 1 --2- 172.21.15.28:0/2999818970 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f871c039150 0x7f871c03b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T21:59:03.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:03.762+0000 7f8732000700 1 -- 172.21.15.28:0/2999818970 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f872801e070 con 0x7f8734101d80 2024-05-12T21:59:03.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:03.762+0000 7f8738e00700 1 --2- 172.21.15.28:0/2999818970 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f871c039150 0x7f871c03b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T21:59:03.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:03.762+0000 7f8738e00700 1 --2- 172.21.15.28:0/2999818970 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f871c039150 0x7f871c03b610 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f8720006fd0 tx=0x7f8720006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T21:59:03.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:03.767+0000 7f8732000700 1 -- 172.21.15.28:0/2999818970 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f8728013070 con 0x7f8734101d80 2024-05-12T21:59:04.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:04.004+0000 7f873c200700 1 -- 172.21.15.28:0/2999818970 --> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) -- 0x7f873402dc90 con 0x7f871c039150 2024-05-12T21:59:04.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:04.013+0000 7f8732000700 1 -- 172.21.15.28:0/2999818970 <== mgr.14152 v2:172.21.15.28:6800/3123328651 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f873402dc90 con 0x7f871c039150 2024-05-12T21:59:04.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:04.017+0000 7f873c200700 1 -- 172.21.15.28:0/2999818970 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f871c039150 msgr2=0x7f871c03b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T21:59:04.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:04.017+0000 7f873c200700 1 --2- 172.21.15.28:0/2999818970 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f871c039150 0x7f871c03b610 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f8720006fd0 tx=0x7f8720006e40 comp rx=0 tx=0).stop 2024-05-12T21:59:04.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:04.017+0000 7f873c200700 1 -- 172.21.15.28:0/2999818970 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8734101d80 msgr2=0x7f8734197490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T21:59:04.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:04.017+0000 7f873c200700 1 --2- 172.21.15.28:0/2999818970 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8734101d80 0x7f8734197490 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f8728005850 tx=0x7f8728005030 comp rx=0 tx=0).stop 2024-05-12T21:59:04.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:04.017+0000 7f873c200700 1 -- 172.21.15.28:0/2999818970 shutdown_connections 2024-05-12T21:59:04.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:04.017+0000 7f873c200700 1 --2- 172.21.15.28:0/2999818970 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f871c039150 0x7f871c03b610 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:59:04.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:04.017+0000 7f873c200700 1 --2- 172.21.15.28:0/2999818970 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8734101d80 0x7f8734197490 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:59:04.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:04.017+0000 7f873c200700 1 -- 172.21.15.28:0/2999818970 >> 172.21.15.28:0/2999818970 conn(0x7f87340fd320 msgr2=0x7f87340fddd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T21:59:04.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:04.017+0000 7f873c200700 1 -- 172.21.15.28:0/2999818970 shutdown_connections 2024-05-12T21:59:04.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:04.017+0000 7f873c200700 1 -- 172.21.15.28:0/2999818970 wait complete. 2024-05-12T21:59:04.829 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi086 2024-05-12T21:59:04.829 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-05-12T21:59:04.829 DEBUG:teuthology.orchestra.run.smithi086:> dd of=/etc/ceph/ceph.conf 2024-05-12T21:59:04.849 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-05-12T21:59:04.849 DEBUG:teuthology.orchestra.run.smithi086:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-05-12T21:59:04.906 INFO:tasks.cephadm:Adding host smithi086 to orchestrator... 2024-05-12T21:59:04.906 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f72fecff68e1d400c4568684327c900485c20d6a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6b92e7b4-10aa-11ef-bc98-c7b262605968 -- ceph orch host add smithi086 2024-05-12T21:59:05.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:05 smithi028 ceph-mon[29847]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-05-12T21:59:05.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:05 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:05.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:05 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T21:59:05.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:05 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T21:59:05.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:05 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T21:59:05.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:05 smithi028 ceph-mon[29847]: Updating smithi028:/etc/ceph/ceph.conf 2024-05-12T21:59:05.327 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/mon.a/config 2024-05-12T21:59:06.034 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:06 smithi028 ceph-mon[29847]: Updating smithi028:/var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-12T21:59:07.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:07 smithi028 ceph-mon[29847]: Updating smithi028:/etc/ceph/ceph.client.admin.keyring 2024-05-12T21:59:07.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:07 smithi028 ceph-mon[29847]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T21:59:07.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:07 smithi028 ceph-mon[29847]: Updating smithi028:/var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/config/ceph.client.admin.keyring 2024-05-12T21:59:07.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:07 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:07.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:07 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:07.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:07 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:07.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:07.319+0000 7f2bdf200700 1 -- 172.21.15.28:0/2791493492 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2bd80746c0 msgr2=0x7f2bd8074aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T21:59:07.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:07.319+0000 7f2bdf200700 1 --2- 172.21.15.28:0/2791493492 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2bd80746c0 0x7f2bd8074aa0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f2bcc009910 tx=0x7f2bcc009c20 comp rx=0 tx=0).stop 2024-05-12T21:59:07.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:07.319+0000 7f2bdf200700 1 -- 172.21.15.28:0/2791493492 shutdown_connections 2024-05-12T21:59:07.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:07.319+0000 7f2bdf200700 1 --2- 172.21.15.28:0/2791493492 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2bd80746c0 0x7f2bd8074aa0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:59:07.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:07.319+0000 7f2bdf200700 1 -- 172.21.15.28:0/2791493492 >> 172.21.15.28:0/2791493492 conn(0x7f2bd80fd680 msgr2=0x7f2bd80ffaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T21:59:07.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:07.319+0000 7f2bdf200700 1 -- 172.21.15.28:0/2791493492 shutdown_connections 2024-05-12T21:59:07.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:07.319+0000 7f2bdf200700 1 -- 172.21.15.28:0/2791493492 wait complete. 2024-05-12T21:59:07.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:07.320+0000 7f2bdf200700 1 Processor -- start 2024-05-12T21:59:07.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:07.321+0000 7f2bdf200700 1 -- start start 2024-05-12T21:59:07.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:07.321+0000 7f2bdf200700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2bd80746c0 0x7f2bd81a3940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T21:59:07.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:07.321+0000 7f2bdf200700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2bd8112810 con 0x7f2bd80746c0 2024-05-12T21:59:07.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:07.321+0000 7f2bdde00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2bd80746c0 0x7f2bd81a3940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T21:59:07.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:07.321+0000 7f2bdde00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2bd80746c0 0x7f2bd81a3940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50322/0 (socket says 172.21.15.28:50322) 2024-05-12T21:59:07.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:07.321+0000 7f2bdde00700 1 -- 172.21.15.28:0/726522193 learned_addr learned my addr 172.21.15.28:0/726522193 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T21:59:07.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:07.322+0000 7f2bdde00700 1 -- 172.21.15.28:0/726522193 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2bd81a3e80 con 0x7f2bd80746c0 2024-05-12T21:59:07.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:07.322+0000 7f2bdde00700 1 --2- 172.21.15.28:0/726522193 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2bd80746c0 0x7f2bd81a3940 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f2bcc006010 tx=0x7f2bcc005050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T21:59:07.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:07.322+0000 7f2bd2a00700 1 -- 172.21.15.28:0/726522193 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2bcc026070 con 0x7f2bd80746c0 2024-05-12T21:59:07.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:07.322+0000 7f2bdf200700 1 -- 172.21.15.28:0/726522193 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2bd81a4110 con 0x7f2bd80746c0 2024-05-12T21:59:07.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:07.322+0000 7f2bd2a00700 1 -- 172.21.15.28:0/726522193 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2bcc00bc30 con 0x7f2bd80746c0 2024-05-12T21:59:07.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:07.322+0000 7f2bd2a00700 1 -- 172.21.15.28:0/726522193 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2bcc003ab0 con 0x7f2bd80746c0 2024-05-12T21:59:07.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:07.323+0000 7f2bdf200700 1 -- 172.21.15.28:0/726522193 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2bd819db00 con 0x7f2bd80746c0 2024-05-12T21:59:07.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:07.324+0000 7f2bdf200700 1 -- 172.21.15.28:0/726522193 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2bd8075e00 con 0x7f2bd80746c0 2024-05-12T21:59:07.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:07.324+0000 7f2bd2a00700 1 -- 172.21.15.28:0/726522193 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f2bcc029070 con 0x7f2bd80746c0 2024-05-12T21:59:07.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:07.324+0000 7f2bd2a00700 1 --2- 172.21.15.28:0/726522193 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f2bc00390f0 0x7f2bc003b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T21:59:07.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:07.324+0000 7f2bd2a00700 1 -- 172.21.15.28:0/726522193 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f2bcc01e070 con 0x7f2bd80746c0 2024-05-12T21:59:07.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:07.325+0000 7f2bdd400700 1 --2- 172.21.15.28:0/726522193 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f2bc00390f0 0x7f2bc003b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T21:59:07.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:07.325+0000 7f2bdd400700 1 --2- 172.21.15.28:0/726522193 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f2bc00390f0 0x7f2bc003b5b0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f2bc8006fd0 tx=0x7f2bc8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T21:59:07.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:07.330+0000 7f2bd2a00700 1 -- 172.21.15.28:0/726522193 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f2bcc013070 con 0x7f2bd80746c0 2024-05-12T21:59:07.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:07.563+0000 7f2bdf200700 1 -- 172.21.15.28:0/726522193 --> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi086", "target": ["mon-mgr", ""]}) -- 0x7f2bd819eff0 con 0x7f2bc00390f0 2024-05-12T21:59:09.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:09 smithi028 ceph-mon[29847]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi086", "target": ["mon-mgr", ""]}]: dispatch 2024-05-12T21:59:09.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:09 smithi028 ceph-mon[29847]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T21:59:09.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:09 smithi028 ceph-mon[29847]: Deploying cephadm binary to smithi086 2024-05-12T21:59:11.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:11 smithi028 ceph-mon[29847]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T21:59:12.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:12.568+0000 7f2bd2a00700 1 -- 172.21.15.28:0/726522193 <== mgr.14152 v2:172.21.15.28:6800/3123328651 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7f2bd819eff0 con 0x7f2bc00390f0 2024-05-12T21:59:12.570 INFO:teuthology.orchestra.run.smithi028.stdout:Added host 'smithi086' with addr '172.21.15.86' 2024-05-12T21:59:12.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:12.572+0000 7f2bdf200700 1 -- 172.21.15.28:0/726522193 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f2bc00390f0 msgr2=0x7f2bc003b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T21:59:12.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:12.572+0000 7f2bdf200700 1 --2- 172.21.15.28:0/726522193 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f2bc00390f0 0x7f2bc003b5b0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f2bc8006fd0 tx=0x7f2bc8006e40 comp rx=0 tx=0).stop 2024-05-12T21:59:12.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:12.572+0000 7f2bdf200700 1 -- 172.21.15.28:0/726522193 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2bd80746c0 msgr2=0x7f2bd81a3940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T21:59:12.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:12.572+0000 7f2bdf200700 1 --2- 172.21.15.28:0/726522193 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2bd80746c0 0x7f2bd81a3940 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f2bcc006010 tx=0x7f2bcc005050 comp rx=0 tx=0).stop 2024-05-12T21:59:12.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:12.572+0000 7f2bdf200700 1 -- 172.21.15.28:0/726522193 shutdown_connections 2024-05-12T21:59:12.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:12.572+0000 7f2bdf200700 1 --2- 172.21.15.28:0/726522193 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f2bc00390f0 0x7f2bc003b5b0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:59:12.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:12.572+0000 7f2bdf200700 1 --2- 172.21.15.28:0/726522193 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2bd80746c0 0x7f2bd81a3940 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:59:12.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:12.572+0000 7f2bdf200700 1 -- 172.21.15.28:0/726522193 >> 172.21.15.28:0/726522193 conn(0x7f2bd80fd680 msgr2=0x7f2bd81038c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T21:59:12.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:12.573+0000 7f2bdf200700 1 -- 172.21.15.28:0/726522193 shutdown_connections 2024-05-12T21:59:12.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:12.573+0000 7f2bdf200700 1 -- 172.21.15.28:0/726522193 wait complete. 2024-05-12T21:59:13.279 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:13 smithi028 ceph-mon[29847]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T21:59:13.279 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:13 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:13.279 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:13 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T21:59:13.344 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f72fecff68e1d400c4568684327c900485c20d6a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6b92e7b4-10aa-11ef-bc98-c7b262605968 -- ceph orch host ls --format=json 2024-05-12T21:59:13.794 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/mon.a/config 2024-05-12T21:59:14.216 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:14 smithi028 ceph-mon[29847]: Added host smithi086 2024-05-12T21:59:14.216 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:14 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:14.216 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:14 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:15.270 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:15 smithi028 ceph-mon[29847]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T21:59:15.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:15.770+0000 7f89f2c00700 1 -- 172.21.15.28:0/133530765 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89ec101f90 msgr2=0x7f89ec102370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T21:59:15.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:15.770+0000 7f89f2c00700 1 --2- 172.21.15.28:0/133530765 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89ec101f90 0x7f89ec102370 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f89dc009910 tx=0x7f89dc009c20 comp rx=0 tx=0).stop 2024-05-12T21:59:15.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:15.770+0000 7f89f2c00700 1 -- 172.21.15.28:0/133530765 shutdown_connections 2024-05-12T21:59:15.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:15.770+0000 7f89f2c00700 1 --2- 172.21.15.28:0/133530765 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89ec101f90 0x7f89ec102370 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:59:15.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:15.770+0000 7f89f2c00700 1 -- 172.21.15.28:0/133530765 >> 172.21.15.28:0/133530765 conn(0x7f89ec0fd860 msgr2=0x7f89ec0ffc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T21:59:15.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:15.771+0000 7f89f2c00700 1 -- 172.21.15.28:0/133530765 shutdown_connections 2024-05-12T21:59:15.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:15.771+0000 7f89f2c00700 1 -- 172.21.15.28:0/133530765 wait complete. 2024-05-12T21:59:15.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:15.772+0000 7f89f2c00700 1 Processor -- start 2024-05-12T21:59:15.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:15.772+0000 7f89f2c00700 1 -- start start 2024-05-12T21:59:15.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:15.773+0000 7f89f2c00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89ec101f90 0x7f89ec19f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T21:59:15.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:15.773+0000 7f89f2c00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f89ec10c360 con 0x7f89ec101f90 2024-05-12T21:59:15.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:15.773+0000 7f89ebe00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89ec101f90 0x7f89ec19f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T21:59:15.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:15.773+0000 7f89ebe00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89ec101f90 0x7f89ec19f6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50344/0 (socket says 172.21.15.28:50344) 2024-05-12T21:59:15.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:15.773+0000 7f89ebe00700 1 -- 172.21.15.28:0/905243839 learned_addr learned my addr 172.21.15.28:0/905243839 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T21:59:15.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:15.773+0000 7f89ebe00700 1 -- 172.21.15.28:0/905243839 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89ec19fc00 con 0x7f89ec101f90 2024-05-12T21:59:15.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:15.774+0000 7f89ebe00700 1 --2- 172.21.15.28:0/905243839 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89ec101f90 0x7f89ec19f6c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f89dc005850 tx=0x7f89dc005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T21:59:15.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:15.774+0000 7f89e8c00700 1 -- 172.21.15.28:0/905243839 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f89dc026070 con 0x7f89ec101f90 2024-05-12T21:59:15.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:15.774+0000 7f89e8c00700 1 -- 172.21.15.28:0/905243839 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89dc00bbd0 con 0x7f89ec101f90 2024-05-12T21:59:15.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:15.774+0000 7f89e8c00700 1 -- 172.21.15.28:0/905243839 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f89dc003970 con 0x7f89ec101f90 2024-05-12T21:59:15.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:15.774+0000 7f89f2c00700 1 -- 172.21.15.28:0/905243839 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89ec19fef0 con 0x7f89ec101f90 2024-05-12T21:59:15.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:15.774+0000 7f89f2c00700 1 -- 172.21.15.28:0/905243839 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89ec199880 con 0x7f89ec101f90 2024-05-12T21:59:15.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:15.775+0000 7f89e8c00700 1 -- 172.21.15.28:0/905243839 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f89dc029070 con 0x7f89ec101f90 2024-05-12T21:59:15.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:15.776+0000 7f89f2c00700 1 -- 172.21.15.28:0/905243839 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89ec103470 con 0x7f89ec101f90 2024-05-12T21:59:15.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:15.776+0000 7f89e8c00700 1 --2- 172.21.15.28:0/905243839 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f89d0039150 0x7f89d003b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T21:59:15.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:15.776+0000 7f89e8c00700 1 -- 172.21.15.28:0/905243839 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f89dc01e070 con 0x7f89ec101f90 2024-05-12T21:59:15.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:15.776+0000 7f89eb400700 1 --2- 172.21.15.28:0/905243839 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f89d0039150 0x7f89d003b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T21:59:15.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:15.777+0000 7f89eb400700 1 --2- 172.21.15.28:0/905243839 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f89d0039150 0x7f89d003b610 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f89d8006fd0 tx=0x7f89d8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T21:59:15.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:15.782+0000 7f89e8c00700 1 -- 172.21.15.28:0/905243839 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f89dc050030 con 0x7f89ec101f90 2024-05-12T21:59:16.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:16.017+0000 7f89f2c00700 1 -- 172.21.15.28:0/905243839 --> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f89ec19ae20 con 0x7f89d0039150 2024-05-12T21:59:16.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:16.020+0000 7f89e8c00700 1 -- 172.21.15.28:0/905243839 <== mgr.14152 v2:172.21.15.28:6800/3123328651 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+159 (secure 0 0 0) 0x7f89ec19ae20 con 0x7f89d0039150 2024-05-12T21:59:16.020 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-12T21:59:16.020 INFO:teuthology.orchestra.run.smithi028.stdout:[{"addr": "172.21.15.28", "hostname": "smithi028", "labels": [], "status": ""}, {"addr": "172.21.15.86", "hostname": "smithi086", "labels": [], "status": ""}] 2024-05-12T21:59:16.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:16.023+0000 7f89f2c00700 1 -- 172.21.15.28:0/905243839 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f89d0039150 msgr2=0x7f89d003b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T21:59:16.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:16.023+0000 7f89f2c00700 1 --2- 172.21.15.28:0/905243839 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f89d0039150 0x7f89d003b610 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f89d8006fd0 tx=0x7f89d8006e40 comp rx=0 tx=0).stop 2024-05-12T21:59:16.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:16.023+0000 7f89f2c00700 1 -- 172.21.15.28:0/905243839 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89ec101f90 msgr2=0x7f89ec19f6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T21:59:16.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:16.023+0000 7f89f2c00700 1 --2- 172.21.15.28:0/905243839 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89ec101f90 0x7f89ec19f6c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f89dc005850 tx=0x7f89dc005030 comp rx=0 tx=0).stop 2024-05-12T21:59:16.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:16.023+0000 7f89f2c00700 1 -- 172.21.15.28:0/905243839 shutdown_connections 2024-05-12T21:59:16.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:16.023+0000 7f89f2c00700 1 --2- 172.21.15.28:0/905243839 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f89d0039150 0x7f89d003b610 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:59:16.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:16.023+0000 7f89f2c00700 1 --2- 172.21.15.28:0/905243839 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89ec101f90 0x7f89ec19f6c0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:59:16.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:16.023+0000 7f89f2c00700 1 -- 172.21.15.28:0/905243839 >> 172.21.15.28:0/905243839 conn(0x7f89ec0fd860 msgr2=0x7f89ec074550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T21:59:16.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:16.024+0000 7f89f2c00700 1 -- 172.21.15.28:0/905243839 shutdown_connections 2024-05-12T21:59:16.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:16.024+0000 7f89f2c00700 1 -- 172.21.15.28:0/905243839 wait complete. 2024-05-12T21:59:16.675 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:16 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:16.743 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi186 2024-05-12T21:59:16.743 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-05-12T21:59:16.743 DEBUG:teuthology.orchestra.run.smithi186:> dd of=/etc/ceph/ceph.conf 2024-05-12T21:59:16.762 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-05-12T21:59:16.762 DEBUG:teuthology.orchestra.run.smithi186:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-05-12T21:59:16.819 INFO:tasks.cephadm:Adding host smithi186 to orchestrator... 2024-05-12T21:59:16.820 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f72fecff68e1d400c4568684327c900485c20d6a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6b92e7b4-10aa-11ef-bc98-c7b262605968 -- ceph orch host add smithi186 2024-05-12T21:59:17.279 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/mon.a/config 2024-05-12T21:59:17.598 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:17 smithi028 ceph-mon[29847]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T21:59:17.598 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:17 smithi028 ceph-mon[29847]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-12T21:59:19.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:19.314+0000 7f1676400700 1 -- 172.21.15.28:0/1656503009 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16700746c0 msgr2=0x7f1670074aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T21:59:19.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:19.314+0000 7f1676400700 1 --2- 172.21.15.28:0/1656503009 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16700746c0 0x7f1670074aa0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f1660009910 tx=0x7f1660009c20 comp rx=0 tx=0).stop 2024-05-12T21:59:19.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:19.314+0000 7f1676400700 1 -- 172.21.15.28:0/1656503009 shutdown_connections 2024-05-12T21:59:19.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:19.314+0000 7f1676400700 1 --2- 172.21.15.28:0/1656503009 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16700746c0 0x7f1670074aa0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:59:19.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:19.314+0000 7f1676400700 1 -- 172.21.15.28:0/1656503009 >> 172.21.15.28:0/1656503009 conn(0x7f16700fd860 msgr2=0x7f16700ffc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T21:59:19.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:19.314+0000 7f1676400700 1 -- 172.21.15.28:0/1656503009 shutdown_connections 2024-05-12T21:59:19.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:19.314+0000 7f1676400700 1 -- 172.21.15.28:0/1656503009 wait complete. 2024-05-12T21:59:19.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:19.315+0000 7f1676400700 1 Processor -- start 2024-05-12T21:59:19.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:19.316+0000 7f1676400700 1 -- start start 2024-05-12T21:59:19.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:19.316+0000 7f1676400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16700746c0 0x7f167019dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T21:59:19.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:19.316+0000 7f1676400700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f16701129d0 con 0x7f16700746c0 2024-05-12T21:59:19.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:19.316+0000 7f166f400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16700746c0 0x7f167019dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T21:59:19.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:19.317+0000 7f166f400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16700746c0 0x7f167019dea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55068/0 (socket says 172.21.15.28:55068) 2024-05-12T21:59:19.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:19.317+0000 7f166f400700 1 -- 172.21.15.28:0/2692971380 learned_addr learned my addr 172.21.15.28:0/2692971380 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T21:59:19.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:19.317+0000 7f166f400700 1 -- 172.21.15.28:0/2692971380 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f167019e3e0 con 0x7f16700746c0 2024-05-12T21:59:19.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:19.317+0000 7f166f400700 1 --2- 172.21.15.28:0/2692971380 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16700746c0 0x7f167019dea0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f1660006010 tx=0x7f1660005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T21:59:19.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:19.319+0000 7f166cc00700 1 -- 172.21.15.28:0/2692971380 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1660026070 con 0x7f16700746c0 2024-05-12T21:59:19.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:19.319+0000 7f166cc00700 1 -- 172.21.15.28:0/2692971380 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f166000bc30 con 0x7f16700746c0 2024-05-12T21:59:19.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:19.319+0000 7f166cc00700 1 -- 172.21.15.28:0/2692971380 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f16600039a0 con 0x7f16700746c0 2024-05-12T21:59:19.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:19.319+0000 7f1676400700 1 -- 172.21.15.28:0/2692971380 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f167019e670 con 0x7f16700746c0 2024-05-12T21:59:19.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:19.320+0000 7f1676400700 1 -- 172.21.15.28:0/2692971380 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f167019ead0 con 0x7f16700746c0 2024-05-12T21:59:19.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:19.321+0000 7f166cc00700 1 -- 172.21.15.28:0/2692971380 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f1660029070 con 0x7f16700746c0 2024-05-12T21:59:19.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:19.321+0000 7f1676400700 1 -- 172.21.15.28:0/2692971380 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1670075e00 con 0x7f16700746c0 2024-05-12T21:59:19.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:19.321+0000 7f166cc00700 1 --2- 172.21.15.28:0/2692971380 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f1654039150 0x7f165403b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T21:59:19.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:19.321+0000 7f166cc00700 1 -- 172.21.15.28:0/2692971380 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f166001e070 con 0x7f16700746c0 2024-05-12T21:59:19.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:19.322+0000 7f1666a00700 1 --2- 172.21.15.28:0/2692971380 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f1654039150 0x7f165403b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T21:59:19.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:19.322+0000 7f1666a00700 1 --2- 172.21.15.28:0/2692971380 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f1654039150 0x7f165403b610 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f1658006fd0 tx=0x7f1658006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T21:59:19.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:19.330+0000 7f166cc00700 1 -- 172.21.15.28:0/2692971380 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f1660013070 con 0x7f16700746c0 2024-05-12T21:59:19.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:19 smithi028 ceph-mon[29847]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T21:59:19.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:19 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:19.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:19 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:19.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:19 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:19.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:19 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:19.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:19 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi086", "name": "osd_memory_target"}]: dispatch 2024-05-12T21:59:19.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:19 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T21:59:19.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:19 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T21:59:19.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:19 smithi028 ceph-mon[29847]: Updating smithi086:/etc/ceph/ceph.conf 2024-05-12T21:59:19.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:19.568+0000 7f1676400700 1 -- 172.21.15.28:0/2692971380 --> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi186", "target": ["mon-mgr", ""]}) -- 0x7f167019f270 con 0x7f1654039150 2024-05-12T21:59:20.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:20 smithi028 ceph-mon[29847]: Updating smithi086:/var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-12T21:59:20.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:20 smithi028 ceph-mon[29847]: Updating smithi086:/etc/ceph/ceph.client.admin.keyring 2024-05-12T21:59:20.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:20 smithi028 ceph-mon[29847]: Updating smithi086:/var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/config/ceph.client.admin.keyring 2024-05-12T21:59:20.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:20 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:20.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:20 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:20.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:20 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:21.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:21 smithi028 ceph-mon[29847]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi186", "target": ["mon-mgr", ""]}]: dispatch 2024-05-12T21:59:21.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:21 smithi028 ceph-mon[29847]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T21:59:21.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:21 smithi028 ceph-mon[29847]: Deploying cephadm binary to smithi186 2024-05-12T21:59:23.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:23 smithi028 ceph-mon[29847]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T21:59:24.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:24.417+0000 7f166cc00700 1 -- 172.21.15.28:0/2692971380 <== mgr.14152 v2:172.21.15.28:6800/3123328651 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f167019f270 con 0x7f1654039150 2024-05-12T21:59:24.418 INFO:teuthology.orchestra.run.smithi028.stdout:Added host 'smithi186' with addr '172.21.15.186' 2024-05-12T21:59:24.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:24.421+0000 7f1676400700 1 -- 172.21.15.28:0/2692971380 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f1654039150 msgr2=0x7f165403b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T21:59:24.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:24.421+0000 7f1676400700 1 --2- 172.21.15.28:0/2692971380 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f1654039150 0x7f165403b610 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f1658006fd0 tx=0x7f1658006e40 comp rx=0 tx=0).stop 2024-05-12T21:59:24.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:24.421+0000 7f1676400700 1 -- 172.21.15.28:0/2692971380 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16700746c0 msgr2=0x7f167019dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T21:59:24.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:24.421+0000 7f1676400700 1 --2- 172.21.15.28:0/2692971380 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16700746c0 0x7f167019dea0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f1660006010 tx=0x7f1660005030 comp rx=0 tx=0).stop 2024-05-12T21:59:24.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:24.421+0000 7f1676400700 1 -- 172.21.15.28:0/2692971380 shutdown_connections 2024-05-12T21:59:24.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:24.421+0000 7f1676400700 1 --2- 172.21.15.28:0/2692971380 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f1654039150 0x7f165403b610 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:59:24.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:24.421+0000 7f1676400700 1 --2- 172.21.15.28:0/2692971380 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16700746c0 0x7f167019dea0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:59:24.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:24.421+0000 7f1676400700 1 -- 172.21.15.28:0/2692971380 >> 172.21.15.28:0/2692971380 conn(0x7f16700fd860 msgr2=0x7f16700ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T21:59:24.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:24.422+0000 7f1676400700 1 -- 172.21.15.28:0/2692971380 shutdown_connections 2024-05-12T21:59:24.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:24.422+0000 7f1676400700 1 -- 172.21.15.28:0/2692971380 wait complete. 2024-05-12T21:59:25.176 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f72fecff68e1d400c4568684327c900485c20d6a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6b92e7b4-10aa-11ef-bc98-c7b262605968 -- ceph orch host ls --format=json 2024-05-12T21:59:25.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:25 smithi028 ceph-mon[29847]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T21:59:25.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:25 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:25.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:25 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T21:59:25.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:25 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:25.649 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/mon.a/config 2024-05-12T21:59:26.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:26 smithi028 ceph-mon[29847]: Added host smithi186 2024-05-12T21:59:26.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:26 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:27.428 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:27 smithi028 ceph-mon[29847]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T21:59:27.429 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:27 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:27.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:27.602+0000 7f3f7aa00700 1 -- 172.21.15.28:0/1473849768 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f74102d30 msgr2=0x7f3f74103110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T21:59:27.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:27.602+0000 7f3f7aa00700 1 --2- 172.21.15.28:0/1473849768 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f74102d30 0x7f3f74103110 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f3f68009910 tx=0x7f3f68009c20 comp rx=0 tx=0).stop 2024-05-12T21:59:27.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:27.602+0000 7f3f7aa00700 1 -- 172.21.15.28:0/1473849768 shutdown_connections 2024-05-12T21:59:27.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:27.602+0000 7f3f7aa00700 1 --2- 172.21.15.28:0/1473849768 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f74102d30 0x7f3f74103110 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:59:27.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:27.602+0000 7f3f7aa00700 1 -- 172.21.15.28:0/1473849768 >> 172.21.15.28:0/1473849768 conn(0x7f3f740fc580 msgr2=0x7f3f740fe9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T21:59:27.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:27.602+0000 7f3f7aa00700 1 -- 172.21.15.28:0/1473849768 shutdown_connections 2024-05-12T21:59:27.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:27.602+0000 7f3f7aa00700 1 -- 172.21.15.28:0/1473849768 wait complete. 2024-05-12T21:59:27.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:27.603+0000 7f3f7aa00700 1 Processor -- start 2024-05-12T21:59:27.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:27.603+0000 7f3f7aa00700 1 -- start start 2024-05-12T21:59:27.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:27.604+0000 7f3f7aa00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f74102d30 0x7f3f7419d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T21:59:27.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:27.604+0000 7f3f7aa00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3f7410c220 con 0x7f3f74102d30 2024-05-12T21:59:27.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:27.604+0000 7f3f79600700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f74102d30 0x7f3f7419d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T21:59:27.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:27.604+0000 7f3f79600700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f74102d30 0x7f3f7419d440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59950/0 (socket says 172.21.15.28:59950) 2024-05-12T21:59:27.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:27.604+0000 7f3f79600700 1 -- 172.21.15.28:0/1180899133 learned_addr learned my addr 172.21.15.28:0/1180899133 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T21:59:27.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:27.605+0000 7f3f79600700 1 -- 172.21.15.28:0/1180899133 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f7419d980 con 0x7f3f74102d30 2024-05-12T21:59:27.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:27.605+0000 7f3f79600700 1 --2- 172.21.15.28:0/1180899133 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f74102d30 0x7f3f7419d440 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f3f68006010 tx=0x7f3f68005050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T21:59:27.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:27.605+0000 7f3f72000700 1 -- 172.21.15.28:0/1180899133 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3f68026070 con 0x7f3f74102d30 2024-05-12T21:59:27.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:27.605+0000 7f3f72000700 1 -- 172.21.15.28:0/1180899133 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f6800bc30 con 0x7f3f74102d30 2024-05-12T21:59:27.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:27.605+0000 7f3f72000700 1 -- 172.21.15.28:0/1180899133 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3f68003ab0 con 0x7f3f74102d30 2024-05-12T21:59:27.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:27.605+0000 7f3f7aa00700 1 -- 172.21.15.28:0/1180899133 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f7419dc10 con 0x7f3f74102d30 2024-05-12T21:59:27.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:27.606+0000 7f3f7aa00700 1 -- 172.21.15.28:0/1180899133 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f74197650 con 0x7f3f74102d30 2024-05-12T21:59:27.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:27.607+0000 7f3f72000700 1 -- 172.21.15.28:0/1180899133 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f3f68029070 con 0x7f3f74102d30 2024-05-12T21:59:27.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:27.607+0000 7f3f7aa00700 1 -- 172.21.15.28:0/1180899133 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f7406a8b0 con 0x7f3f74102d30 2024-05-12T21:59:27.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:27.607+0000 7f3f72000700 1 --2- 172.21.15.28:0/1180899133 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f3f5c0390f0 0x7f3f5c03b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T21:59:27.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:27.607+0000 7f3f72000700 1 -- 172.21.15.28:0/1180899133 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f3f6801e070 con 0x7f3f74102d30 2024-05-12T21:59:27.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:27.607+0000 7f3f78c00700 1 --2- 172.21.15.28:0/1180899133 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f3f5c0390f0 0x7f3f5c03b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T21:59:27.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:27.608+0000 7f3f78c00700 1 --2- 172.21.15.28:0/1180899133 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f3f5c0390f0 0x7f3f5c03b5b0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f3f60006fd0 tx=0x7f3f60006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T21:59:27.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:27.613+0000 7f3f72000700 1 -- 172.21.15.28:0/1180899133 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f3f68013070 con 0x7f3f74102d30 2024-05-12T21:59:27.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:27.870+0000 7f3f7aa00700 1 -- 172.21.15.28:0/1180899133 --> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f3f74063980 con 0x7f3f5c0390f0 2024-05-12T21:59:27.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:27.872+0000 7f3f72000700 1 -- 172.21.15.28:0/1180899133 <== mgr.14152 v2:172.21.15.28:6800/3123328651 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+239 (secure 0 0 0) 0x7f3f74063980 con 0x7f3f5c0390f0 2024-05-12T21:59:27.873 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-12T21:59:27.873 INFO:teuthology.orchestra.run.smithi028.stdout:[{"addr": "172.21.15.28", "hostname": "smithi028", "labels": [], "status": ""}, {"addr": "172.21.15.86", "hostname": "smithi086", "labels": [], "status": ""}, {"addr": "172.21.15.186", "hostname": "smithi186", "labels": [], "status": ""}] 2024-05-12T21:59:27.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:27.876+0000 7f3f7aa00700 1 -- 172.21.15.28:0/1180899133 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f3f5c0390f0 msgr2=0x7f3f5c03b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T21:59:27.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:27.876+0000 7f3f7aa00700 1 --2- 172.21.15.28:0/1180899133 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f3f5c0390f0 0x7f3f5c03b5b0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f3f60006fd0 tx=0x7f3f60006e40 comp rx=0 tx=0).stop 2024-05-12T21:59:27.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:27.876+0000 7f3f7aa00700 1 -- 172.21.15.28:0/1180899133 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f74102d30 msgr2=0x7f3f7419d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T21:59:27.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:27.876+0000 7f3f7aa00700 1 --2- 172.21.15.28:0/1180899133 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f74102d30 0x7f3f7419d440 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f3f68006010 tx=0x7f3f68005050 comp rx=0 tx=0).stop 2024-05-12T21:59:27.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:27.876+0000 7f3f7aa00700 1 -- 172.21.15.28:0/1180899133 shutdown_connections 2024-05-12T21:59:27.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:27.876+0000 7f3f7aa00700 1 --2- 172.21.15.28:0/1180899133 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f3f5c0390f0 0x7f3f5c03b5b0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:59:27.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:27.876+0000 7f3f7aa00700 1 --2- 172.21.15.28:0/1180899133 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f74102d30 0x7f3f7419d440 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:59:27.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:27.876+0000 7f3f7aa00700 1 -- 172.21.15.28:0/1180899133 >> 172.21.15.28:0/1180899133 conn(0x7f3f740fc580 msgr2=0x7f3f74103ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T21:59:27.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:27.876+0000 7f3f7aa00700 1 -- 172.21.15.28:0/1180899133 shutdown_connections 2024-05-12T21:59:27.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:27.876+0000 7f3f7aa00700 1 -- 172.21.15.28:0/1180899133 wait complete. 2024-05-12T21:59:28.647 INFO:tasks.cephadm:Setting crush tunables to default 2024-05-12T21:59:28.647 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f72fecff68e1d400c4568684327c900485c20d6a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6b92e7b4-10aa-11ef-bc98-c7b262605968 -- ceph osd crush tunables default 2024-05-12T21:59:29.103 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/mon.a/config 2024-05-12T21:59:29.527 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:29 smithi028 ceph-mon[29847]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T21:59:29.527 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:29 smithi028 ceph-mon[29847]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-12T21:59:31.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:31.176+0000 7f7701000700 1 -- 172.21.15.28:0/191413140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76fc074750 msgr2=0x7f76fc074b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T21:59:31.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:31.176+0000 7f7701000700 1 --2- 172.21.15.28:0/191413140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76fc074750 0x7f76fc074b30 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f76ec009910 tx=0x7f76ec009c20 comp rx=0 tx=0).stop 2024-05-12T21:59:31.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:31.177+0000 7f7701000700 1 -- 172.21.15.28:0/191413140 shutdown_connections 2024-05-12T21:59:31.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:31.177+0000 7f7701000700 1 --2- 172.21.15.28:0/191413140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76fc074750 0x7f76fc074b30 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:59:31.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:31.177+0000 7f7701000700 1 -- 172.21.15.28:0/191413140 >> 172.21.15.28:0/191413140 conn(0x7f76fc0fd860 msgr2=0x7f76fc0ffc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T21:59:31.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:31.177+0000 7f7701000700 1 -- 172.21.15.28:0/191413140 shutdown_connections 2024-05-12T21:59:31.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:31.177+0000 7f7701000700 1 -- 172.21.15.28:0/191413140 wait complete. 2024-05-12T21:59:31.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:31.178+0000 7f7701000700 1 Processor -- start 2024-05-12T21:59:31.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:31.178+0000 7f7701000700 1 -- start start 2024-05-12T21:59:31.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:31.179+0000 7f7701000700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76fc199d50 0x7f76fc19a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T21:59:31.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:31.179+0000 7f7701000700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f76fc10e5e0 con 0x7f76fc199d50 2024-05-12T21:59:31.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:31.179+0000 7f76f9e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76fc199d50 0x7f76fc19a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T21:59:31.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:31.179+0000 7f76f9e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76fc199d50 0x7f76fc19a130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59970/0 (socket says 172.21.15.28:59970) 2024-05-12T21:59:31.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:31.179+0000 7f76f9e00700 1 -- 172.21.15.28:0/1881656745 learned_addr learned my addr 172.21.15.28:0/1881656745 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T21:59:31.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:31.180+0000 7f76f9e00700 1 -- 172.21.15.28:0/1881656745 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76fc19a670 con 0x7f76fc199d50 2024-05-12T21:59:31.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:31.180+0000 7f76f9e00700 1 --2- 172.21.15.28:0/1881656745 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76fc199d50 0x7f76fc19a130 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f76ec005950 tx=0x7f76ec005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T21:59:31.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:31.181+0000 7f76f2a00700 1 -- 172.21.15.28:0/1881656745 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f76ec026070 con 0x7f76fc199d50 2024-05-12T21:59:31.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:31.181+0000 7f76f2a00700 1 -- 172.21.15.28:0/1881656745 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f76ec00bc30 con 0x7f76fc199d50 2024-05-12T21:59:31.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:31.181+0000 7f7701000700 1 -- 172.21.15.28:0/1881656745 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76fc19a900 con 0x7f76fc199d50 2024-05-12T21:59:31.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:31.181+0000 7f76f2a00700 1 -- 172.21.15.28:0/1881656745 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f76ec003970 con 0x7f76fc199d50 2024-05-12T21:59:31.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:31.181+0000 7f7701000700 1 -- 172.21.15.28:0/1881656745 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76fc19e7f0 con 0x7f76fc199d50 2024-05-12T21:59:31.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:31.182+0000 7f7701000700 1 -- 172.21.15.28:0/1881656745 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f76fc075e00 con 0x7f76fc199d50 2024-05-12T21:59:31.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:31.182+0000 7f76f2a00700 1 -- 172.21.15.28:0/1881656745 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f76ec029070 con 0x7f76fc199d50 2024-05-12T21:59:31.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:31.183+0000 7f76f2a00700 1 --2- 172.21.15.28:0/1881656745 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f76e0039150 0x7f76e003b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T21:59:31.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:31.183+0000 7f76f2a00700 1 -- 172.21.15.28:0/1881656745 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f76ec01e070 con 0x7f76fc199d50 2024-05-12T21:59:31.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:31.183+0000 7f76f9400700 1 --2- 172.21.15.28:0/1881656745 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f76e0039150 0x7f76e003b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T21:59:31.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:31.184+0000 7f76f9400700 1 --2- 172.21.15.28:0/1881656745 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f76e0039150 0x7f76e003b610 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f76e8006fd0 tx=0x7f76e8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T21:59:31.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:31.188+0000 7f76f2a00700 1 -- 172.21.15.28:0/1881656745 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f76ec013070 con 0x7f76fc199d50 2024-05-12T21:59:31.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:31.423+0000 7f7701000700 1 -- 172.21.15.28:0/1881656745 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f76fc19b0b0 con 0x7f76fc199d50 2024-05-12T21:59:31.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:31 smithi028 ceph-mon[29847]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T21:59:31.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:31 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:31.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:31 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:31.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:31 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:31.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:31 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:31.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:31 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi186", "name": "osd_memory_target"}]: dispatch 2024-05-12T21:59:31.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:31 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T21:59:31.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:31 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T21:59:31.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:31 smithi028 ceph-mon[29847]: Updating smithi186:/etc/ceph/ceph.conf 2024-05-12T21:59:32.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:32.333+0000 7f76f2a00700 1 -- 172.21.15.28:0/1881656745 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) ==== 124+0+0 (secure 0 0 0) 0x7f76ec0047d0 con 0x7f76fc199d50 2024-05-12T21:59:32.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:32.337+0000 7f7701000700 1 -- 172.21.15.28:0/1881656745 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f76e0039150 msgr2=0x7f76e003b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T21:59:32.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:32.337+0000 7f7701000700 1 --2- 172.21.15.28:0/1881656745 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f76e0039150 0x7f76e003b610 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f76e8006fd0 tx=0x7f76e8006e40 comp rx=0 tx=0).stop 2024-05-12T21:59:32.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:32.337+0000 7f7701000700 1 -- 172.21.15.28:0/1881656745 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76fc199d50 msgr2=0x7f76fc19a130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T21:59:32.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:32.337+0000 7f7701000700 1 --2- 172.21.15.28:0/1881656745 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76fc199d50 0x7f76fc19a130 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f76ec005950 tx=0x7f76ec005030 comp rx=0 tx=0).stop 2024-05-12T21:59:32.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:32.337+0000 7f7701000700 1 -- 172.21.15.28:0/1881656745 shutdown_connections 2024-05-12T21:59:32.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:32.337+0000 7f7701000700 1 --2- 172.21.15.28:0/1881656745 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f76e0039150 0x7f76e003b610 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:59:32.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:32.337+0000 7f7701000700 1 --2- 172.21.15.28:0/1881656745 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76fc199d50 0x7f76fc19a130 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:59:32.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:32.337+0000 7f7701000700 1 -- 172.21.15.28:0/1881656745 >> 172.21.15.28:0/1881656745 conn(0x7f76fc0fd860 msgr2=0x7f76fc0fdc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T21:59:32.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:32.338+0000 7f7701000700 1 -- 172.21.15.28:0/1881656745 shutdown_connections 2024-05-12T21:59:32.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:32.338+0000 7f7701000700 1 -- 172.21.15.28:0/1881656745 wait complete. 2024-05-12T21:59:32.340 INFO:teuthology.orchestra.run.smithi028.stderr:adjusted tunables profile to default 2024-05-12T21:59:32.713 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:32 smithi028 ceph-mon[29847]: Updating smithi186:/var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-12T21:59:32.714 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:32 smithi028 ceph-mon[29847]: Updating smithi186:/etc/ceph/ceph.client.admin.keyring 2024-05-12T21:59:32.714 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:32 smithi028 ceph-mon[29847]: Updating smithi186:/var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/config/ceph.client.admin.keyring 2024-05-12T21:59:32.714 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:32 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/1881656745' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-05-12T21:59:32.714 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:32 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:32.714 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:32 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:32.714 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:32 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:33.087 INFO:tasks.cephadm:Adding mon.a on smithi028 2024-05-12T21:59:33.087 INFO:tasks.cephadm:Adding mon.b on smithi086 2024-05-12T21:59:33.088 INFO:tasks.cephadm:Adding mon.c on smithi186 2024-05-12T21:59:33.088 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f72fecff68e1d400c4568684327c900485c20d6a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6b92e7b4-10aa-11ef-bc98-c7b262605968 -- ceph orch apply mon '3;smithi028:172.21.15.28=a;smithi086:172.21.15.86=b;smithi186:172.21.15.186=c' 2024-05-12T21:59:33.346 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-12T21:59:33.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:33 smithi028 ceph-mon[29847]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T21:59:33.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:33 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/1881656745' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-05-12T21:59:33.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:33 smithi028 ceph-mon[29847]: osdmap e4: 0 total, 0 up, 0 in 2024-05-12T21:59:34.693 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:34.692+0000 7f92f3e80700 1 -- 172.21.15.186:0/1426404969 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92ec101f90 msgr2=0x7f92ec102370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T21:59:34.693 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:34.692+0000 7f92f3e80700 1 --2- 172.21.15.186:0/1426404969 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92ec101f90 0x7f92ec102370 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f92dc009910 tx=0x7f92dc009c20 comp rx=0 tx=0).stop 2024-05-12T21:59:34.693 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:34.692+0000 7f92f3e80700 1 -- 172.21.15.186:0/1426404969 shutdown_connections 2024-05-12T21:59:34.693 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:34.692+0000 7f92f3e80700 1 --2- 172.21.15.186:0/1426404969 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92ec101f90 0x7f92ec102370 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:59:34.693 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:34.692+0000 7f92f3e80700 1 -- 172.21.15.186:0/1426404969 >> 172.21.15.186:0/1426404969 conn(0x7f92ec0fd860 msgr2=0x7f92ec0ffc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T21:59:34.693 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:34.692+0000 7f92f3e80700 1 -- 172.21.15.186:0/1426404969 shutdown_connections 2024-05-12T21:59:34.694 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:34.692+0000 7f92f3e80700 1 -- 172.21.15.186:0/1426404969 wait complete. 2024-05-12T21:59:34.695 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:34.692+0000 7f92f3e80700 1 Processor -- start 2024-05-12T21:59:34.695 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:34.693+0000 7f92f3e80700 1 -- start start 2024-05-12T21:59:34.695 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:34.693+0000 7f92f3e80700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92ec101f90 0x7f92ec19d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T21:59:34.695 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:34.693+0000 7f92f3e80700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f92ec10c360 con 0x7f92ec101f90 2024-05-12T21:59:34.695 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:34.693+0000 7f92f1c1c700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92ec101f90 0x7f92ec19d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T21:59:34.696 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:34.694+0000 7f92f1c1c700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92ec101f90 0x7f92ec19d5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.186:54862/0 (socket says 172.21.15.186:54862) 2024-05-12T21:59:34.696 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:34.694+0000 7f92f1c1c700 1 -- 172.21.15.186:0/3587652206 learned_addr learned my addr 172.21.15.186:0/3587652206 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-05-12T21:59:34.696 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:34.694+0000 7f92f1c1c700 1 -- 172.21.15.186:0/3587652206 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92ec19db00 con 0x7f92ec101f90 2024-05-12T21:59:34.696 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:34.695+0000 7f92f1c1c700 1 --2- 172.21.15.186:0/3587652206 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92ec101f90 0x7f92ec19d5c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f92dc005850 tx=0x7f92dc005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T21:59:34.697 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:34.695+0000 7f92e2ffd700 1 -- 172.21.15.186:0/3587652206 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f92dc026070 con 0x7f92ec101f90 2024-05-12T21:59:34.697 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:34.695+0000 7f92f3e80700 1 -- 172.21.15.186:0/3587652206 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92ec19ddf0 con 0x7f92ec101f90 2024-05-12T21:59:34.697 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:34.695+0000 7f92e2ffd700 1 -- 172.21.15.186:0/3587652206 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f92dc00bbd0 con 0x7f92ec101f90 2024-05-12T21:59:34.697 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:34.695+0000 7f92e2ffd700 1 -- 172.21.15.186:0/3587652206 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f92dc003970 con 0x7f92ec101f90 2024-05-12T21:59:34.697 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:34.695+0000 7f92f3e80700 1 -- 172.21.15.186:0/3587652206 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92ec197860 con 0x7f92ec101f90 2024-05-12T21:59:34.697 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:34.696+0000 7f92f3e80700 1 -- 172.21.15.186:0/3587652206 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f92ec103470 con 0x7f92ec101f90 2024-05-12T21:59:34.698 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:34.697+0000 7f92e2ffd700 1 -- 172.21.15.186:0/3587652206 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f92dc029070 con 0x7f92ec101f90 2024-05-12T21:59:34.698 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:34.697+0000 7f92e2ffd700 1 --2- 172.21.15.186:0/3587652206 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f92d8039150 0x7f92d803b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T21:59:34.698 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:34.697+0000 7f92e2ffd700 1 -- 172.21.15.186:0/3587652206 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f92dc01e070 con 0x7f92ec101f90 2024-05-12T21:59:34.698 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:34.697+0000 7f92f141b700 1 --2- 172.21.15.186:0/3587652206 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f92d8039150 0x7f92d803b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T21:59:34.699 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:34.698+0000 7f92f141b700 1 --2- 172.21.15.186:0/3587652206 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f92d8039150 0x7f92d803b610 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f92e8006fd0 tx=0x7f92e8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T21:59:34.713 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:34.712+0000 7f92e2ffd700 1 -- 172.21.15.186:0/3587652206 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f92dc013070 con 0x7f92ec101f90 2024-05-12T21:59:34.920 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:34.919+0000 7f92f3e80700 1 -- 172.21.15.186:0/3587652206 --> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi028:172.21.15.28=a;smithi086:172.21.15.86=b;smithi186:172.21.15.186=c", "target": ["mon-mgr", ""]}) -- 0x7f92ec0008d0 con 0x7f92d8039150 2024-05-12T21:59:34.935 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:34.934+0000 7f92e2ffd700 1 -- 172.21.15.186:0/3587652206 <== mgr.14152 v2:172.21.15.28:6800/3123328651 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f92ec0008d0 con 0x7f92d8039150 2024-05-12T21:59:34.935 INFO:teuthology.orchestra.run.smithi186.stdout:Scheduled mon update... 2024-05-12T21:59:34.937 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:34.936+0000 7f92f3e80700 1 -- 172.21.15.186:0/3587652206 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f92d8039150 msgr2=0x7f92d803b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T21:59:34.937 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:34.936+0000 7f92f3e80700 1 --2- 172.21.15.186:0/3587652206 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f92d8039150 0x7f92d803b610 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f92e8006fd0 tx=0x7f92e8006e40 comp rx=0 tx=0).stop 2024-05-12T21:59:34.937 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:34.936+0000 7f92f3e80700 1 -- 172.21.15.186:0/3587652206 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92ec101f90 msgr2=0x7f92ec19d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T21:59:34.937 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:34.936+0000 7f92f3e80700 1 --2- 172.21.15.186:0/3587652206 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92ec101f90 0x7f92ec19d5c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f92dc005850 tx=0x7f92dc005030 comp rx=0 tx=0).stop 2024-05-12T21:59:34.937 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:34.936+0000 7f92f3e80700 1 -- 172.21.15.186:0/3587652206 shutdown_connections 2024-05-12T21:59:34.937 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:34.936+0000 7f92f3e80700 1 --2- 172.21.15.186:0/3587652206 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f92d8039150 0x7f92d803b610 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:59:34.937 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:34.936+0000 7f92f3e80700 1 --2- 172.21.15.186:0/3587652206 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92ec101f90 0x7f92ec19d5c0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:59:34.938 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:34.936+0000 7f92f3e80700 1 -- 172.21.15.186:0/3587652206 >> 172.21.15.186:0/3587652206 conn(0x7f92ec0fd860 msgr2=0x7f92ec103960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T21:59:34.938 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:34.936+0000 7f92f3e80700 1 -- 172.21.15.186:0/3587652206 shutdown_connections 2024-05-12T21:59:34.938 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:34.936+0000 7f92f3e80700 1 -- 172.21.15.186:0/3587652206 wait complete. 2024-05-12T21:59:35.506 DEBUG:teuthology.orchestra.run.smithi086:mon.b> sudo journalctl -f -n 0 -u ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968@mon.b.service 2024-05-12T21:59:35.509 DEBUG:teuthology.orchestra.run.smithi186:mon.c> sudo journalctl -f -n 0 -u ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968@mon.c.service 2024-05-12T21:59:35.511 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-05-12T21:59:35.511 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f72fecff68e1d400c4568684327c900485c20d6a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6b92e7b4-10aa-11ef-bc98-c7b262605968 -- ceph mon dump -f json 2024-05-12T21:59:35.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:35 smithi028 ceph-mon[29847]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T21:59:35.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:35 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:35.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:35 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T21:59:35.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:35 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T21:59:35.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:35 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T21:59:35.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:35 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:35.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:35 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-12T21:59:35.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:35 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T21:59:36.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:36 smithi028 ceph-mon[29847]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi028:172.21.15.28=a;smithi086:172.21.15.86=b;smithi186:172.21.15.186=c", "target": ["mon-mgr", ""]}]: dispatch 2024-05-12T21:59:36.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:36 smithi028 ceph-mon[29847]: Saving service mon spec with placement smithi028:172.21.15.28=a;smithi086:172.21.15.86=b;smithi186:172.21.15.186=c;count:3 2024-05-12T21:59:36.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:36 smithi028 ceph-mon[29847]: Deploying daemon mon.c on smithi186 2024-05-12T21:59:37.206 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-12T21:59:37.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:37 smithi028 ceph-mon[29847]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T21:59:39.468 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:39.467+0000 7f7290b3f700 1 -- 172.21.15.186:0/29992210 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f728c107dc0 msgr2=0x7f728c1081a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T21:59:39.468 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:39.467+0000 7f7290b3f700 1 --2- 172.21.15.186:0/29992210 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f728c107dc0 0x7f728c1081a0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f728400b0a0 tx=0x7f728400b3b0 comp rx=0 tx=0).stop 2024-05-12T21:59:39.468 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:39.467+0000 7f7290b3f700 1 -- 172.21.15.186:0/29992210 shutdown_connections 2024-05-12T21:59:39.468 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:39.467+0000 7f7290b3f700 1 --2- 172.21.15.186:0/29992210 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f728c107dc0 0x7f728c1081a0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:59:39.468 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:39.467+0000 7f7290b3f700 1 -- 172.21.15.186:0/29992210 >> 172.21.15.186:0/29992210 conn(0x7f728c076780 msgr2=0x7f728c076b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T21:59:39.469 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:39.467+0000 7f7290b3f700 1 -- 172.21.15.186:0/29992210 shutdown_connections 2024-05-12T21:59:39.469 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:39.467+0000 7f7290b3f700 1 -- 172.21.15.186:0/29992210 wait complete. 2024-05-12T21:59:39.469 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:39.468+0000 7f7290b3f700 1 Processor -- start 2024-05-12T21:59:39.469 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:39.468+0000 7f7290b3f700 1 -- start start 2024-05-12T21:59:39.469 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:39.468+0000 7f7290b3f700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f728c107dc0 0x7f728c100d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T21:59:39.469 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:39.468+0000 7f7290b3f700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f728c10c350 con 0x7f728c107dc0 2024-05-12T21:59:39.470 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:39.469+0000 7f728a59c700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f728c107dc0 0x7f728c100d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T21:59:39.470 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:39.469+0000 7f728a59c700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f728c107dc0 0x7f728c100d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.186:46466/0 (socket says 172.21.15.186:46466) 2024-05-12T21:59:39.470 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:39.469+0000 7f728a59c700 1 -- 172.21.15.186:0/2124248858 learned_addr learned my addr 172.21.15.186:0/2124248858 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-05-12T21:59:39.470 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:39.469+0000 7f728a59c700 1 -- 172.21.15.186:0/2124248858 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f728c102be0 con 0x7f728c107dc0 2024-05-12T21:59:39.470 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:39.470+0000 7f728a59c700 1 --2- 172.21.15.186:0/2124248858 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f728c107dc0 0x7f728c100d40 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f7284015040 tx=0x7f7284003d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T21:59:39.470 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:39.470+0000 7f727b7fe700 1 -- 172.21.15.186:0/2124248858 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7284017040 con 0x7f728c107dc0 2024-05-12T21:59:39.470 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:39.470+0000 7f7290b3f700 1 -- 172.21.15.186:0/2124248858 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f728c101280 con 0x7f728c107dc0 2024-05-12T21:59:39.471 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:39.470+0000 7f7290b3f700 1 -- 172.21.15.186:0/2124248858 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f728c101740 con 0x7f728c107dc0 2024-05-12T21:59:39.471 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:39.470+0000 7f727b7fe700 1 -- 172.21.15.186:0/2124248858 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7284022070 con 0x7f728c107dc0 2024-05-12T21:59:39.471 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:39.470+0000 7f727b7fe700 1 -- 172.21.15.186:0/2124248858 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7284027ba0 con 0x7f728c107dc0 2024-05-12T21:59:39.471 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:39.471+0000 7f7290b3f700 1 -- 172.21.15.186:0/2124248858 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f726c005330 con 0x7f728c107dc0 2024-05-12T21:59:39.472 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:39.472+0000 7f727b7fe700 1 -- 172.21.15.186:0/2124248858 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f7284027460 con 0x7f728c107dc0 2024-05-12T21:59:39.472 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:39.472+0000 7f727b7fe700 1 --2- 172.21.15.186:0/2124248858 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f72740390f0 0x7f727403b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T21:59:39.472 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:39.472+0000 7f727b7fe700 1 -- 172.21.15.186:0/2124248858 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f728400f050 con 0x7f728c107dc0 2024-05-12T21:59:39.472 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:39.472+0000 7f7289d9b700 1 --2- 172.21.15.186:0/2124248858 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f72740390f0 0x7f727403b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T21:59:39.473 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:39.474+0000 7f7289d9b700 1 --2- 172.21.15.186:0/2124248858 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f72740390f0 0x7f727403b5b0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f7280006fd0 tx=0x7f7280006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T21:59:39.487 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:39.488+0000 7f727b7fe700 1 -- 172.21.15.186:0/2124248858 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f7284007e20 con 0x7f728c107dc0 2024-05-12T21:59:39.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:39 smithi028 ceph-mon[29847]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T21:59:39.801 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:39.801+0000 7f7290b3f700 1 -- 172.21.15.186:0/2124248858 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f726c0058f0 con 0x7f728c107dc0 2024-05-12T21:59:39.802 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:39.802+0000 7f727b7fe700 1 -- 172.21.15.186:0/2124248858 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) ==== 95+0+745 (secure 0 0 0) 0x7f728400bc10 con 0x7f728c107dc0 2024-05-12T21:59:39.802 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-12T21:59:39.802 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":1,"fsid":"6b92e7b4-10aa-11ef-bc98-c7b262605968","modified":"2024-05-12T21:57:33.130972Z","created":"2024-05-12T21:57:33.130972Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:3300","nonce":0},{"type":"v1","addr":"172.21.15.28:6789","nonce":0}]},"addr":"172.21.15.28:6789/0","public_addr":"172.21.15.28:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-05-12T21:59:39.804 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:39.804+0000 7f7290b3f700 1 -- 172.21.15.186:0/2124248858 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f72740390f0 msgr2=0x7f727403b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T21:59:39.804 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:39.804+0000 7f7290b3f700 1 --2- 172.21.15.186:0/2124248858 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f72740390f0 0x7f727403b5b0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f7280006fd0 tx=0x7f7280006e40 comp rx=0 tx=0).stop 2024-05-12T21:59:39.804 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:39.804+0000 7f7290b3f700 1 -- 172.21.15.186:0/2124248858 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f728c107dc0 msgr2=0x7f728c100d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T21:59:39.804 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:39.804+0000 7f7290b3f700 1 --2- 172.21.15.186:0/2124248858 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f728c107dc0 0x7f728c100d40 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f7284015040 tx=0x7f7284003d50 comp rx=0 tx=0).stop 2024-05-12T21:59:39.804 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:39.804+0000 7f7290b3f700 1 -- 172.21.15.186:0/2124248858 shutdown_connections 2024-05-12T21:59:39.804 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:39.804+0000 7f7290b3f700 1 --2- 172.21.15.186:0/2124248858 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f72740390f0 0x7f727403b5b0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:59:39.804 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:39.804+0000 7f7290b3f700 1 --2- 172.21.15.186:0/2124248858 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f728c107dc0 0x7f728c100d40 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:59:39.805 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:39.804+0000 7f7290b3f700 1 -- 172.21.15.186:0/2124248858 >> 172.21.15.186:0/2124248858 conn(0x7f728c076780 msgr2=0x7f728c0fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T21:59:39.805 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:39.804+0000 7f7290b3f700 1 -- 172.21.15.186:0/2124248858 shutdown_connections 2024-05-12T21:59:39.805 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:39.804+0000 7f7290b3f700 1 -- 172.21.15.186:0/2124248858 wait complete. 2024-05-12T21:59:39.805 INFO:teuthology.orchestra.run.smithi186.stderr:dumped monmap epoch 1 2024-05-12T21:59:39.975 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:39 smithi186 ceph-mon[34785]: mon.c@-1(synchronizing).mds e1 new map 2024-05-12T21:59:39.975 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:39 smithi186 ceph-mon[34785]: mon.c@-1(synchronizing).mds e1 print_map 2024-05-12T21:59:39.975 INFO:journalctl@ceph.mon.c.smithi186.stdout: e1 2024-05-12T21:59:39.975 INFO:journalctl@ceph.mon.c.smithi186.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-05-12T21:59:39.975 INFO:journalctl@ceph.mon.c.smithi186.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-05-12T21:59:39.975 INFO:journalctl@ceph.mon.c.smithi186.stdout: legacy client fscid: -1 2024-05-12T21:59:39.975 INFO:journalctl@ceph.mon.c.smithi186.stdout: 2024-05-12T21:59:39.975 INFO:journalctl@ceph.mon.c.smithi186.stdout: No filesystems configured 2024-05-12T21:59:39.975 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:39 smithi186 ceph-mon[34785]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-05-12T21:59:39.975 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:39 smithi186 ceph-mon[34785]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-05-12T21:59:39.975 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:39 smithi186 ceph-mon[34785]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-05-12T21:59:39.975 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:39 smithi186 ceph-mon[34785]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-05-12T21:59:39.976 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:39 smithi186 ceph-mon[34785]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-05-12T21:59:39.976 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:39 smithi186 ceph-mon[34785]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-05-12T21:59:39.976 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:39 smithi186 ceph-mon[34785]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-05-12T21:59:39.976 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:39 smithi186 ceph-mon[34785]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-05-12T21:59:39.976 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:39 smithi186 ceph-mon[34785]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-05-12T21:59:39.976 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:39 smithi186 ceph-mon[34785]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-05-12T21:59:39.976 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:39 smithi186 ceph-mon[34785]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T21:59:39.976 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:39 smithi186 ceph-mon[34785]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T21:59:39.976 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:39 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:39.976 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:39 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T21:59:39.976 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:39 smithi186 ceph-mon[34785]: Added host smithi086 2024-05-12T21:59:39.976 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:39 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:39.976 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:39 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:39.976 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:39 smithi186 ceph-mon[34785]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T21:59:39.976 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:39 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:39.977 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:39 smithi186 ceph-mon[34785]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T21:59:39.977 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:39 smithi186 ceph-mon[34785]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-12T21:59:39.977 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:39 smithi186 ceph-mon[34785]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T21:59:39.977 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:39 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:39.977 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:39 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:39.977 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:39 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:39.977 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:39 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:39.977 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:39 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi086", "name": "osd_memory_target"}]: dispatch 2024-05-12T21:59:39.977 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:39 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T21:59:39.977 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:39 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T21:59:39.977 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:39 smithi186 ceph-mon[34785]: Updating smithi086:/etc/ceph/ceph.conf 2024-05-12T21:59:39.977 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:39 smithi186 ceph-mon[34785]: Updating smithi086:/var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-12T21:59:39.977 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:39 smithi186 ceph-mon[34785]: Updating smithi086:/etc/ceph/ceph.client.admin.keyring 2024-05-12T21:59:39.977 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:39 smithi186 ceph-mon[34785]: Updating smithi086:/var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/config/ceph.client.admin.keyring 2024-05-12T21:59:40.365 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:39 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:40.365 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:39 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:40.365 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:39 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:40.365 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:39 smithi186 ceph-mon[34785]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi186", "target": ["mon-mgr", ""]}]: dispatch 2024-05-12T21:59:40.365 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:39 smithi186 ceph-mon[34785]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T21:59:40.365 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:39 smithi186 ceph-mon[34785]: Deploying cephadm binary to smithi186 2024-05-12T21:59:40.365 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:39 smithi186 ceph-mon[34785]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T21:59:40.365 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:39 smithi186 ceph-mon[34785]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T21:59:40.366 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:39 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:40.366 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:39 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T21:59:40.366 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:39 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:40.366 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:39 smithi186 ceph-mon[34785]: Added host smithi186 2024-05-12T21:59:40.366 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:39 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:40.366 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:39 smithi186 ceph-mon[34785]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T21:59:40.366 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:39 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:40.366 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:39 smithi186 ceph-mon[34785]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T21:59:40.366 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:39 smithi186 ceph-mon[34785]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-12T21:59:40.366 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:39 smithi186 ceph-mon[34785]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T21:59:40.366 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:39 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:40.366 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:39 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:40.366 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:39 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:40.367 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:39 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:40.367 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:39 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi186", "name": "osd_memory_target"}]: dispatch 2024-05-12T21:59:40.367 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:39 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T21:59:40.367 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:39 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T21:59:40.367 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:39 smithi186 ceph-mon[34785]: Updating smithi186:/etc/ceph/ceph.conf 2024-05-12T21:59:40.367 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:39 smithi186 ceph-mon[34785]: Updating smithi186:/var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-12T21:59:40.367 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:39 smithi186 ceph-mon[34785]: Updating smithi186:/etc/ceph/ceph.client.admin.keyring 2024-05-12T21:59:40.367 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:39 smithi186 ceph-mon[34785]: Updating smithi186:/var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/config/ceph.client.admin.keyring 2024-05-12T21:59:40.367 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:39 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/1881656745' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-05-12T21:59:40.367 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:39 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:40.367 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:39 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:40.367 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:39 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:40.367 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:39 smithi186 ceph-mon[34785]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T21:59:40.367 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:39 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/1881656745' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-05-12T21:59:40.368 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:39 smithi186 ceph-mon[34785]: osdmap e4: 0 total, 0 up, 0 in 2024-05-12T21:59:40.368 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:39 smithi186 ceph-mon[34785]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T21:59:40.368 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:39 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:40.368 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:39 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T21:59:40.368 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:39 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T21:59:40.368 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:39 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T21:59:40.368 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:39 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:40.368 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:39 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-12T21:59:40.368 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:39 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T21:59:40.368 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:39 smithi186 ceph-mon[34785]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi028:172.21.15.28=a;smithi086:172.21.15.86=b;smithi186:172.21.15.186=c", "target": ["mon-mgr", ""]}]: dispatch 2024-05-12T21:59:40.368 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:39 smithi186 ceph-mon[34785]: Saving service mon spec with placement smithi028:172.21.15.28=a;smithi086:172.21.15.86=b;smithi186:172.21.15.186=c;count:3 2024-05-12T21:59:40.368 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:39 smithi186 ceph-mon[34785]: Deploying daemon mon.c on smithi186 2024-05-12T21:59:40.368 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:39 smithi186 ceph-mon[34785]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T21:59:40.368 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:39 smithi186 ceph-mon[34785]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T21:59:40.369 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:39 smithi186 ceph-mon[34785]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-05-12T21:59:40.369 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:39 smithi186 bash[34752]: 5a2c2778badf3aa40a49a55c171003adbc3e3f4884e15380caa84eba0cd00c82 2024-05-12T21:59:40.369 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:39 smithi186 systemd[1]: Started Ceph mon.c for 6b92e7b4-10aa-11ef-bc98-c7b262605968. 2024-05-12T21:59:41.638 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-05-12T21:59:41.638 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f72fecff68e1d400c4568684327c900485c20d6a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6b92e7b4-10aa-11ef-bc98-c7b262605968 -- ceph mon dump -f json 2024-05-12T21:59:41.882 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/mon.c/config 2024-05-12T21:59:45.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:45 smithi186 ceph-mon[34785]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T21:59:45.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:45 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-12T21:59:45.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:45 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-12T21:59:45.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:45 smithi186 ceph-mon[34785]: mon.a calling monitor election 2024-05-12T21:59:45.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:45 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-12T21:59:45.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:45 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-12T21:59:45.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:45 smithi186 ceph-mon[34785]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T21:59:45.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:45 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-12T21:59:45.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:45 smithi186 ceph-mon[34785]: mon.c calling monitor election 2024-05-12T21:59:45.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:45 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-12T21:59:45.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:45 smithi186 ceph-mon[34785]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T21:59:45.431 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:45 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-12T21:59:45.431 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:45 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-12T21:59:45.431 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:45 smithi186 ceph-mon[34785]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-05-12T21:59:45.431 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:45 smithi186 ceph-mon[34785]: monmap epoch 2 2024-05-12T21:59:45.431 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:45 smithi186 ceph-mon[34785]: fsid 6b92e7b4-10aa-11ef-bc98-c7b262605968 2024-05-12T21:59:45.431 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:45 smithi186 ceph-mon[34785]: last_changed 2024-05-12T21:59:39.980086+0000 2024-05-12T21:59:45.431 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:45 smithi186 ceph-mon[34785]: created 2024-05-12T21:57:33.130972+0000 2024-05-12T21:59:45.431 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:45 smithi186 ceph-mon[34785]: min_mon_release 19 (squid) 2024-05-12T21:59:45.431 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:45 smithi186 ceph-mon[34785]: election_strategy: 1 2024-05-12T21:59:45.431 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:45 smithi186 ceph-mon[34785]: 0: [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] mon.a 2024-05-12T21:59:45.431 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:45 smithi186 ceph-mon[34785]: 1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.c 2024-05-12T21:59:45.431 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:45 smithi186 ceph-mon[34785]: fsmap 2024-05-12T21:59:45.431 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:45 smithi186 ceph-mon[34785]: osdmap e4: 0 total, 0 up, 0 in 2024-05-12T21:59:45.431 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:45 smithi186 ceph-mon[34785]: mgrmap e13: x(active, since 59s) 2024-05-12T21:59:45.431 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:45 smithi186 ceph-mon[34785]: overall HEALTH_OK 2024-05-12T21:59:45.431 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:45 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:45.432 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:45 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:45.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:45 smithi028 ceph-mon[29847]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T21:59:45.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:45 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-12T21:59:45.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:45 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-12T21:59:45.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:45 smithi028 ceph-mon[29847]: mon.a calling monitor election 2024-05-12T21:59:45.537 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:45 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-12T21:59:45.537 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:45 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-12T21:59:45.537 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:45 smithi028 ceph-mon[29847]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T21:59:45.537 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:45 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-12T21:59:45.537 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:45 smithi028 ceph-mon[29847]: mon.c calling monitor election 2024-05-12T21:59:45.537 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:45 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-12T21:59:45.537 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:45 smithi028 ceph-mon[29847]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T21:59:45.537 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:45 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-12T21:59:45.537 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:45 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-12T21:59:45.537 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:45 smithi028 ceph-mon[29847]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-05-12T21:59:45.537 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:45 smithi028 ceph-mon[29847]: monmap epoch 2 2024-05-12T21:59:45.537 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:45 smithi028 ceph-mon[29847]: fsid 6b92e7b4-10aa-11ef-bc98-c7b262605968 2024-05-12T21:59:45.537 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:45 smithi028 ceph-mon[29847]: last_changed 2024-05-12T21:59:39.980086+0000 2024-05-12T21:59:45.537 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:45 smithi028 ceph-mon[29847]: created 2024-05-12T21:57:33.130972+0000 2024-05-12T21:59:45.538 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:45 smithi028 ceph-mon[29847]: min_mon_release 19 (squid) 2024-05-12T21:59:45.538 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:45 smithi028 ceph-mon[29847]: election_strategy: 1 2024-05-12T21:59:45.538 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:45 smithi028 ceph-mon[29847]: 0: [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] mon.a 2024-05-12T21:59:45.538 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:45 smithi028 ceph-mon[29847]: 1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.c 2024-05-12T21:59:45.538 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:45 smithi028 ceph-mon[29847]: fsmap 2024-05-12T21:59:45.538 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:45 smithi028 ceph-mon[29847]: osdmap e4: 0 total, 0 up, 0 in 2024-05-12T21:59:45.538 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:45 smithi028 ceph-mon[29847]: mgrmap e13: x(active, since 59s) 2024-05-12T21:59:45.538 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:45 smithi028 ceph-mon[29847]: overall HEALTH_OK 2024-05-12T21:59:45.538 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:45 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:45.538 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:45 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:46.166 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:46.166+0000 7f9adf906700 1 -- 172.21.15.186:0/2793140830 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ac40035c0 msgr2=0x7f9ac4005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T21:59:46.166 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:46.166+0000 7f9adf906700 1 --2- 172.21.15.186:0/2793140830 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ac40035c0 0x7f9ac4005a50 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f9ad8101d70 tx=0x7f9ad400feb0 comp rx=0 tx=0).stop 2024-05-12T21:59:46.166 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:46.166+0000 7f9adf906700 1 -- 172.21.15.186:0/2793140830 shutdown_connections 2024-05-12T21:59:46.166 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:46.166+0000 7f9adf906700 1 --2- 172.21.15.186:0/2793140830 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ac40035c0 0x7f9ac4005a50 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:59:46.166 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:46.166+0000 7f9adf906700 1 -- 172.21.15.186:0/2793140830 >> 172.21.15.186:0/2793140830 conn(0x7f9ad80fd930 msgr2=0x7f9ad80ffd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T21:59:46.166 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:46.166+0000 7f9adf906700 1 -- 172.21.15.186:0/2793140830 shutdown_connections 2024-05-12T21:59:46.166 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:46.166+0000 7f9adf906700 1 -- 172.21.15.186:0/2793140830 wait complete. 2024-05-12T21:59:46.167 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:46.167+0000 7f9adf906700 1 Processor -- start 2024-05-12T21:59:46.167 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:46.167+0000 7f9adf906700 1 -- start start 2024-05-12T21:59:46.168 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:46.167+0000 7f9adf906700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9ac40035c0 0x7f9ad819f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T21:59:46.168 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:46.167+0000 7f9adf906700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ad819fdb0 0x7f9ad8199a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T21:59:46.168 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:46.167+0000 7f9adf906700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9ad8109ed0 con 0x7f9ad819fdb0 2024-05-12T21:59:46.168 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:46.167+0000 7f9adf906700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f9ad8109d50 con 0x7f9ac40035c0 2024-05-12T21:59:46.168 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:46.167+0000 7f9add6a2700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9ac40035c0 0x7f9ad819f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T21:59:46.168 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:46.167+0000 7f9add6a2700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9ac40035c0 0x7f9ad819f870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:46834/0 (socket says 172.21.15.186:46834) 2024-05-12T21:59:46.168 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:46.167+0000 7f9add6a2700 1 -- 172.21.15.186:0/2370675227 learned_addr learned my addr 172.21.15.186:0/2370675227 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-05-12T21:59:46.168 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:46.167+0000 7f9add6a2700 1 -- 172.21.15.186:0/2370675227 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9ac40035c0 msgr2=0x7f9ad819f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-05-12T21:59:46.168 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:46.167+0000 7f9add6a2700 1 -- 172.21.15.186:0/2370675227 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9ac40035c0 msgr2=0x7f9ad819f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-05-12T21:59:46.168 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:46.167+0000 7f9add6a2700 1 --2- 172.21.15.186:0/2370675227 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9ac40035c0 0x7f9ad819f870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-12T21:59:46.168 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:46.167+0000 7f9add6a2700 1 --2- 172.21.15.186:0/2370675227 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9ac40035c0 0x7f9ad819f870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-12T21:59:46.168 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:46.168+0000 7f9adcea1700 1 --2- 172.21.15.186:0/2370675227 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ad819fdb0 0x7f9ad8199a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T21:59:46.169 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:46.168+0000 7f9adcea1700 1 -- 172.21.15.186:0/2370675227 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9ac40035c0 msgr2=0x7f9ad819f870 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-12T21:59:46.169 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:46.168+0000 7f9adcea1700 1 --2- 172.21.15.186:0/2370675227 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9ac40035c0 0x7f9ad819f870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:59:46.169 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:46.168+0000 7f9adcea1700 1 -- 172.21.15.186:0/2370675227 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ad81a03f0 con 0x7f9ad819fdb0 2024-05-12T21:59:46.169 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:46.169+0000 7f9adcea1700 1 --2- 172.21.15.186:0/2370675227 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ad819fdb0 0x7f9ad8199a30 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f9ac800b4d0 tx=0x7f9ac800b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T21:59:46.170 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:46.169+0000 7f9ace7fc700 1 -- 172.21.15.186:0/2370675227 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f9ac8011630 con 0x7f9ad819fdb0 2024-05-12T21:59:46.170 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:46.169+0000 7f9adf906700 1 -- 172.21.15.186:0/2370675227 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ad8199f70 con 0x7f9ad819fdb0 2024-05-12T21:59:46.170 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:46.169+0000 7f9adf906700 1 -- 172.21.15.186:0/2370675227 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ad819a500 con 0x7f9ad819fdb0 2024-05-12T21:59:46.170 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:46.169+0000 7f9ace7fc700 1 -- 172.21.15.186:0/2370675227 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9ac8011cb0 con 0x7f9ad819fdb0 2024-05-12T21:59:46.170 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:46.169+0000 7f9ace7fc700 1 -- 172.21.15.186:0/2370675227 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f9ac80193b0 con 0x7f9ad819fdb0 2024-05-12T21:59:46.171 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:46.170+0000 7f9adf906700 1 -- 172.21.15.186:0/2370675227 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ad8103580 con 0x7f9ad819fdb0 2024-05-12T21:59:46.171 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:46.171+0000 7f9ace7fc700 1 -- 172.21.15.186:0/2370675227 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f9ac8018020 con 0x7f9ad819fdb0 2024-05-12T21:59:46.171 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:46.171+0000 7f9ace7fc700 1 --2- 172.21.15.186:0/2370675227 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f9abc038e90 0x7f9abc03b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T21:59:46.171 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:46.171+0000 7f9ace7fc700 1 -- 172.21.15.186:0/2370675227 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f9ac804bb70 con 0x7f9ad819fdb0 2024-05-12T21:59:46.171 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:46.172+0000 7f9add6a2700 1 --2- 172.21.15.186:0/2370675227 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f9abc038e90 0x7f9abc03b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T21:59:46.172 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:46.173+0000 7f9add6a2700 1 --2- 172.21.15.186:0/2370675227 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f9abc038e90 0x7f9abc03b350 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f9ad400fa60 tx=0x7f9ad4005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T21:59:46.185 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:46.185+0000 7f9ace7fc700 1 -- 172.21.15.186:0/2370675227 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f9ac801f020 con 0x7f9ad819fdb0 2024-05-12T21:59:46.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:46 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:46.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:46 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-12T21:59:46.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:46 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T21:59:46.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:46 smithi028 ceph-mon[29847]: Deploying daemon mon.b on smithi086 2024-05-12T21:59:46.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:46 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-12T21:59:46.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:46 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:46.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:46 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-12T21:59:46.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:46 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T21:59:46.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:46 smithi186 ceph-mon[34785]: Deploying daemon mon.b on smithi086 2024-05-12T21:59:46.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:46 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-12T21:59:46.461 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:46.461+0000 7f9adf906700 1 -- 172.21.15.186:0/2370675227 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f9ad819af60 con 0x7f9ad819fdb0 2024-05-12T21:59:46.462 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:46.463+0000 7f9ace7fc700 1 -- 172.21.15.186:0/2370675227 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1015 (secure 0 0 0) 0x7f9ac8018570 con 0x7f9ad819fdb0 2024-05-12T21:59:46.462 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-12T21:59:46.463 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":2,"fsid":"6b92e7b4-10aa-11ef-bc98-c7b262605968","modified":"2024-05-12T21:59:39.980086Z","created":"2024-05-12T21:57:33.130972Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:3300","nonce":0},{"type":"v1","addr":"172.21.15.28:6789","nonce":0}]},"addr":"172.21.15.28:6789/0","public_addr":"172.21.15.28:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:3300","nonce":0},{"type":"v1","addr":"172.21.15.186:6789","nonce":0}]},"addr":"172.21.15.186:6789/0","public_addr":"172.21.15.186:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-05-12T21:59:46.465 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:46.465+0000 7f9adf906700 1 -- 172.21.15.186:0/2370675227 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f9abc038e90 msgr2=0x7f9abc03b350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T21:59:46.465 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:46.465+0000 7f9adf906700 1 --2- 172.21.15.186:0/2370675227 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f9abc038e90 0x7f9abc03b350 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f9ad400fa60 tx=0x7f9ad4005960 comp rx=0 tx=0).stop 2024-05-12T21:59:46.465 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:46.465+0000 7f9adf906700 1 -- 172.21.15.186:0/2370675227 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ad819fdb0 msgr2=0x7f9ad8199a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T21:59:46.465 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:46.465+0000 7f9adf906700 1 --2- 172.21.15.186:0/2370675227 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ad819fdb0 0x7f9ad8199a30 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f9ac800b4d0 tx=0x7f9ac800b890 comp rx=0 tx=0).stop 2024-05-12T21:59:46.465 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:46.465+0000 7f9adf906700 1 -- 172.21.15.186:0/2370675227 shutdown_connections 2024-05-12T21:59:46.465 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:46.465+0000 7f9adf906700 1 --2- 172.21.15.186:0/2370675227 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f9abc038e90 0x7f9abc03b350 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:59:46.465 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:46.465+0000 7f9adf906700 1 --2- 172.21.15.186:0/2370675227 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9ac40035c0 0x7f9ad819f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:59:46.465 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:46.465+0000 7f9adf906700 1 --2- 172.21.15.186:0/2370675227 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ad819fdb0 0x7f9ad8199a30 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:59:46.465 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:46.465+0000 7f9adf906700 1 -- 172.21.15.186:0/2370675227 >> 172.21.15.186:0/2370675227 conn(0x7f9ad80fd930 msgr2=0x7f9ad8074890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T21:59:46.465 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:46.465+0000 7f9adf906700 1 -- 172.21.15.186:0/2370675227 shutdown_connections 2024-05-12T21:59:46.465 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:46.465+0000 7f9adf906700 1 -- 172.21.15.186:0/2370675227 wait complete. 2024-05-12T21:59:46.466 INFO:teuthology.orchestra.run.smithi186.stderr:dumped monmap epoch 2 2024-05-12T21:59:47.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:47 smithi028 ceph-mon[29847]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T21:59:47.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:47 smithi028 ceph-mon[29847]: from='client.? 172.21.15.186:0/2370675227' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-12T21:59:47.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:47 smithi186 ceph-mon[34785]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T21:59:47.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:47 smithi186 ceph-mon[34785]: from='client.? 172.21.15.186:0/2370675227' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-12T21:59:47.983 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-05-12T21:59:47.983 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f72fecff68e1d400c4568684327c900485c20d6a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6b92e7b4-10aa-11ef-bc98-c7b262605968 -- ceph mon dump -f json 2024-05-12T21:59:48.226 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/mon.c/config 2024-05-12T21:59:49.090 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 podman[34277]: 2024-05-12 21:59:49.067662281 +0000 UTC m=+0.358701591 container start d12e520b6c5ccd7d5e17136e3f52defceaf31c757111ebdcdffcb1bff67132fc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mon-b, org.label-schema.build-date=20240507, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, RELEASE=squid-f72fecf, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-05-12T21:59:49.091 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: set uid:gid to 167:167 (ceph:ceph) 2024-05-12T21:59:49.091 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: ceph version 19.0.0-2555-gf72fecff (f72fecff68e1d400c4568684327c900485c20d6a) squid (dev), process ceph-mon, pid 2 2024-05-12T21:59:49.091 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: pidfile_write: ignore empty --pid-file 2024-05-12T21:59:49.355 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:49 smithi186 ceph-mon[34785]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T21:59:49.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: load: jerasure load: lrc 2024-05-12T21:59:49.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: RocksDB version: 7.9.2 2024-05-12T21:59:49.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Git sha 0 2024-05-12T21:59:49.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Compile date 2024-05-10 15:23:03 2024-05-12T21:59:49.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: DB SUMMARY 2024-05-12T21:59:49.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: DB Session ID: 4JC7U5LZWKMNBY7QIN0P 2024-05-12T21:59:49.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: CURRENT file: CURRENT 2024-05-12T21:59:49.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: IDENTITY file: IDENTITY 2024-05-12T21:59:49.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-05-12T21:59:49.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-05-12T21:59:49.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 636 ; 2024-05-12T21:59:49.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.error_if_exists: 0 2024-05-12T21:59:49.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.create_if_missing: 0 2024-05-12T21:59:49.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.paranoid_checks: 1 2024-05-12T21:59:49.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.flush_verify_memtable_count: 1 2024-05-12T21:59:49.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-05-12T21:59:49.494 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-05-12T21:59:49.494 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.env: 0x557147f50c60 2024-05-12T21:59:49.494 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.fs: PosixFileSystem 2024-05-12T21:59:49.494 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.info_log: 0x557148b49ac0 2024-05-12T21:59:49.494 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.max_file_opening_threads: 16 2024-05-12T21:59:49.494 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.statistics: (nil) 2024-05-12T21:59:49.494 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.use_fsync: 0 2024-05-12T21:59:49.494 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.max_log_file_size: 0 2024-05-12T21:59:49.494 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-05-12T21:59:49.494 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.log_file_time_to_roll: 0 2024-05-12T21:59:49.494 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.keep_log_file_num: 1000 2024-05-12T21:59:49.494 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.recycle_log_file_num: 0 2024-05-12T21:59:49.494 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.allow_fallocate: 1 2024-05-12T21:59:49.494 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.allow_mmap_reads: 0 2024-05-12T21:59:49.494 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.allow_mmap_writes: 0 2024-05-12T21:59:49.495 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.use_direct_reads: 0 2024-05-12T21:59:49.495 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-05-12T21:59:49.495 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.create_missing_column_families: 0 2024-05-12T21:59:49.495 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.db_log_dir: 2024-05-12T21:59:49.495 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.wal_dir: 2024-05-12T21:59:49.495 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.table_cache_numshardbits: 6 2024-05-12T21:59:49.495 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.WAL_ttl_seconds: 0 2024-05-12T21:59:49.495 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.WAL_size_limit_MB: 0 2024-05-12T21:59:49.495 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-05-12T21:59:49.497 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-05-12T21:59:49.497 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.is_fd_close_on_exec: 1 2024-05-12T21:59:49.498 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.advise_random_on_open: 1 2024-05-12T21:59:49.498 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.db_write_buffer_size: 0 2024-05-12T21:59:49.498 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.write_buffer_manager: 0x557148b1c1e0 2024-05-12T21:59:49.498 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-05-12T21:59:49.498 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-05-12T21:59:49.498 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.use_adaptive_mutex: 0 2024-05-12T21:59:49.498 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.rate_limiter: (nil) 2024-05-12T21:59:49.498 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-05-12T21:59:49.498 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.wal_recovery_mode: 2 2024-05-12T21:59:49.498 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.enable_thread_tracking: 0 2024-05-12T21:59:49.498 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.enable_pipelined_write: 0 2024-05-12T21:59:49.498 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.unordered_write: 0 2024-05-12T21:59:49.498 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-05-12T21:59:49.499 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-05-12T21:59:49.499 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-05-12T21:59:49.499 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-05-12T21:59:49.499 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.row_cache: None 2024-05-12T21:59:49.499 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.wal_filter: None 2024-05-12T21:59:49.499 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-05-12T21:59:49.499 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.allow_ingest_behind: 0 2024-05-12T21:59:49.500 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.two_write_queues: 0 2024-05-12T21:59:49.500 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.manual_wal_flush: 0 2024-05-12T21:59:49.500 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.wal_compression: 0 2024-05-12T21:59:49.500 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.atomic_flush: 0 2024-05-12T21:59:49.500 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-05-12T21:59:49.500 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.persist_stats_to_disk: 0 2024-05-12T21:59:49.500 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.write_dbid_to_manifest: 0 2024-05-12T21:59:49.500 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.log_readahead_size: 0 2024-05-12T21:59:49.500 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-05-12T21:59:49.500 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.best_efforts_recovery: 0 2024-05-12T21:59:49.500 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-05-12T21:59:49.500 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-05-12T21:59:49.500 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.allow_data_in_errors: 0 2024-05-12T21:59:49.501 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.db_host_id: __hostname__ 2024-05-12T21:59:49.501 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.enforce_single_del_contracts: true 2024-05-12T21:59:49.501 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.max_background_jobs: 2 2024-05-12T21:59:49.501 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.max_background_compactions: -1 2024-05-12T21:59:49.501 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.max_subcompactions: 1 2024-05-12T21:59:49.501 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-05-12T21:59:49.501 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-05-12T21:59:49.501 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.delayed_write_rate : 16777216 2024-05-12T21:59:49.501 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.max_total_wal_size: 0 2024-05-12T21:59:49.501 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-05-12T21:59:49.501 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.stats_dump_period_sec: 600 2024-05-12T21:59:49.501 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.stats_persist_period_sec: 600 2024-05-12T21:59:49.501 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-05-12T21:59:49.501 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.max_open_files: -1 2024-05-12T21:59:49.502 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.bytes_per_sync: 0 2024-05-12T21:59:49.502 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.wal_bytes_per_sync: 0 2024-05-12T21:59:49.502 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.strict_bytes_per_sync: 0 2024-05-12T21:59:49.502 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.compaction_readahead_size: 0 2024-05-12T21:59:49.502 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.max_background_flushes: -1 2024-05-12T21:59:49.502 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Compression algorithms supported: 2024-05-12T21:59:49.502 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-05-12T21:59:49.502 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: kZSTD supported: 0 2024-05-12T21:59:49.502 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: kXpressCompression supported: 0 2024-05-12T21:59:49.502 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: kLZ4HCCompression supported: 1 2024-05-12T21:59:49.502 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: kZlibCompression supported: 1 2024-05-12T21:59:49.502 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: kSnappyCompression supported: 1 2024-05-12T21:59:49.502 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: kLZ4Compression supported: 1 2024-05-12T21:59:49.502 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: kBZip2Compression supported: 0 2024-05-12T21:59:49.502 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Fast CRC32 supported: Supported on x86 2024-05-12T21:59:49.503 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: DMutex implementation: pthread_mutex_t 2024-05-12T21:59:49.503 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-05-12T21:59:49.503 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-05-12T21:59:49.503 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-05-12T21:59:49.503 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.merge_operator: 2024-05-12T21:59:49.503 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.compaction_filter: None 2024-05-12T21:59:49.503 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.compaction_filter_factory: None 2024-05-12T21:59:49.503 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.sst_partitioner_factory: None 2024-05-12T21:59:49.503 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.memtable_factory: SkipListFactory 2024-05-12T21:59:49.503 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.table_factory: BlockBasedTable 2024-05-12T21:59:49.503 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x557148b49bc0) 2024-05-12T21:59:49.503 INFO:journalctl@ceph.mon.b.smithi086.stdout: cache_index_and_filter_blocks: 1 2024-05-12T21:59:49.503 INFO:journalctl@ceph.mon.b.smithi086.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-05-12T21:59:49.503 INFO:journalctl@ceph.mon.b.smithi086.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-05-12T21:59:49.503 INFO:journalctl@ceph.mon.b.smithi086.stdout: pin_top_level_index_and_filter: 1 2024-05-12T21:59:49.504 INFO:journalctl@ceph.mon.b.smithi086.stdout: index_type: 0 2024-05-12T21:59:49.504 INFO:journalctl@ceph.mon.b.smithi086.stdout: data_block_index_type: 0 2024-05-12T21:59:49.504 INFO:journalctl@ceph.mon.b.smithi086.stdout: index_shortening: 1 2024-05-12T21:59:49.504 INFO:journalctl@ceph.mon.b.smithi086.stdout: data_block_hash_table_util_ratio: 0.750000 2024-05-12T21:59:49.504 INFO:journalctl@ceph.mon.b.smithi086.stdout: checksum: 4 2024-05-12T21:59:49.504 INFO:journalctl@ceph.mon.b.smithi086.stdout: no_block_cache: 0 2024-05-12T21:59:49.504 INFO:journalctl@ceph.mon.b.smithi086.stdout: block_cache: 0x557148b4d090 2024-05-12T21:59:49.504 INFO:journalctl@ceph.mon.b.smithi086.stdout: block_cache_name: BinnedLRUCache 2024-05-12T21:59:49.505 INFO:journalctl@ceph.mon.b.smithi086.stdout: block_cache_options: 2024-05-12T21:59:49.505 INFO:journalctl@ceph.mon.b.smithi086.stdout: capacity : 536870912 2024-05-12T21:59:49.505 INFO:journalctl@ceph.mon.b.smithi086.stdout: num_shard_bits : 4 2024-05-12T21:59:49.505 INFO:journalctl@ceph.mon.b.smithi086.stdout: strict_capacity_limit : 0 2024-05-12T21:59:49.505 INFO:journalctl@ceph.mon.b.smithi086.stdout: high_pri_pool_ratio: 0.000 2024-05-12T21:59:49.505 INFO:journalctl@ceph.mon.b.smithi086.stdout: block_cache_compressed: (nil) 2024-05-12T21:59:49.505 INFO:journalctl@ceph.mon.b.smithi086.stdout: persistent_cache: (nil) 2024-05-12T21:59:49.505 INFO:journalctl@ceph.mon.b.smithi086.stdout: block_size: 4096 2024-05-12T21:59:49.505 INFO:journalctl@ceph.mon.b.smithi086.stdout: block_size_deviation: 10 2024-05-12T21:59:49.505 INFO:journalctl@ceph.mon.b.smithi086.stdout: block_restart_interval: 16 2024-05-12T21:59:49.506 INFO:journalctl@ceph.mon.b.smithi086.stdout: index_block_restart_interval: 1 2024-05-12T21:59:49.506 INFO:journalctl@ceph.mon.b.smithi086.stdout: metadata_block_size: 4096 2024-05-12T21:59:49.506 INFO:journalctl@ceph.mon.b.smithi086.stdout: partition_filters: 0 2024-05-12T21:59:49.506 INFO:journalctl@ceph.mon.b.smithi086.stdout: use_delta_encoding: 1 2024-05-12T21:59:49.506 INFO:journalctl@ceph.mon.b.smithi086.stdout: filter_policy: bloomfilter 2024-05-12T21:59:49.506 INFO:journalctl@ceph.mon.b.smithi086.stdout: whole_key_filtering: 1 2024-05-12T21:59:49.506 INFO:journalctl@ceph.mon.b.smithi086.stdout: verify_compression: 0 2024-05-12T21:59:49.506 INFO:journalctl@ceph.mon.b.smithi086.stdout: read_amp_bytes_per_bit: 0 2024-05-12T21:59:49.506 INFO:journalctl@ceph.mon.b.smithi086.stdout: format_version: 5 2024-05-12T21:59:49.506 INFO:journalctl@ceph.mon.b.smithi086.stdout: enable_index_compression: 1 2024-05-12T21:59:49.506 INFO:journalctl@ceph.mon.b.smithi086.stdout: block_align: 0 2024-05-12T21:59:49.506 INFO:journalctl@ceph.mon.b.smithi086.stdout: max_auto_readahead_size: 262144 2024-05-12T21:59:49.506 INFO:journalctl@ceph.mon.b.smithi086.stdout: prepopulate_block_cache: 0 2024-05-12T21:59:49.506 INFO:journalctl@ceph.mon.b.smithi086.stdout: initial_auto_readahead_size: 8192 2024-05-12T21:59:49.507 INFO:journalctl@ceph.mon.b.smithi086.stdout: num_file_reads_for_auto_readahead: 2 2024-05-12T21:59:49.507 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.write_buffer_size: 33554432 2024-05-12T21:59:49.507 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.max_write_buffer_number: 2 2024-05-12T21:59:49.507 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.compression: NoCompression 2024-05-12T21:59:49.507 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.bottommost_compression: Disabled 2024-05-12T21:59:49.507 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.prefix_extractor: nullptr 2024-05-12T21:59:49.507 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-05-12T21:59:49.507 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.num_levels: 7 2024-05-12T21:59:49.507 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-05-12T21:59:49.507 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-05-12T21:59:49.507 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-05-12T21:59:49.507 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-05-12T21:59:49.507 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-05-12T21:59:49.507 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-05-12T21:59:49.507 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-05-12T21:59:49.508 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-05-12T21:59:49.508 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-05-12T21:59:49.508 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-05-12T21:59:49.508 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-05-12T21:59:49.508 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-05-12T21:59:49.508 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.compression_opts.window_bits: -14 2024-05-12T21:59:49.508 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.compression_opts.level: 32767 2024-05-12T21:59:49.508 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.compression_opts.strategy: 0 2024-05-12T21:59:49.508 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-05-12T21:59:49.508 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-05-12T21:59:49.508 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-05-12T21:59:49.508 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-05-12T21:59:49.509 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.compression_opts.enabled: false 2024-05-12T21:59:49.509 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-05-12T21:59:49.509 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-05-12T21:59:49.509 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-05-12T21:59:49.509 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-05-12T21:59:49.509 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.target_file_size_base: 67108864 2024-05-12T21:59:49.509 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.target_file_size_multiplier: 1 2024-05-12T21:59:49.509 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-05-12T21:59:49.509 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-05-12T21:59:49.509 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-05-12T21:59:49.509 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-05-12T21:59:49.509 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-05-12T21:59:49.509 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-05-12T21:59:49.509 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-05-12T21:59:49.510 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-05-12T21:59:49.510 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-05-12T21:59:49.510 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-05-12T21:59:49.510 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-05-12T21:59:49.510 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-05-12T21:59:49.510 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-05-12T21:59:49.510 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.arena_block_size: 1048576 2024-05-12T21:59:49.510 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-05-12T21:59:49.510 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-05-12T21:59:49.510 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.disable_auto_compactions: 0 2024-05-12T21:59:49.510 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-05-12T21:59:49.510 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-05-12T21:59:49.510 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-05-12T21:59:49.510 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-05-12T21:59:49.510 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-05-12T21:59:49.511 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-05-12T21:59:49.511 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-05-12T21:59:49.511 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-05-12T21:59:49.511 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-05-12T21:59:49.512 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-05-12T21:59:49.512 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.table_properties_collectors: 2024-05-12T21:59:49.512 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.inplace_update_support: 0 2024-05-12T21:59:49.512 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.inplace_update_num_locks: 10000 2024-05-12T21:59:49.512 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-05-12T21:59:49.512 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-05-12T21:59:49.512 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.memtable_huge_page_size: 0 2024-05-12T21:59:49.512 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.bloom_locality: 0 2024-05-12T21:59:49.512 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.max_successive_merges: 0 2024-05-12T21:59:49.512 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.optimize_filters_for_hits: 0 2024-05-12T21:59:49.512 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.paranoid_file_checks: 0 2024-05-12T21:59:49.512 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.force_consistency_checks: 1 2024-05-12T21:59:49.512 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.report_bg_io_stats: 0 2024-05-12T21:59:49.512 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.ttl: 2592000 2024-05-12T21:59:49.512 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.periodic_compaction_seconds: 0 2024-05-12T21:59:49.513 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-05-12T21:59:49.513 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-05-12T21:59:49.513 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.enable_blob_files: false 2024-05-12T21:59:49.513 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.min_blob_size: 0 2024-05-12T21:59:49.513 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.blob_file_size: 268435456 2024-05-12T21:59:49.513 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.blob_compression_type: NoCompression 2024-05-12T21:59:49.513 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.enable_blob_garbage_collection: false 2024-05-12T21:59:49.513 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-05-12T21:59:49.513 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-05-12T21:59:49.513 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-05-12T21:59:49.513 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.blob_file_starting_level: 0 2024-05-12T21:59:49.513 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-05-12T21:59:49.513 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-05-12T21:59:49.513 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-05-12T21:59:49.513 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: bcf229f9-11c8-412e-8536-a774e2d74a7f 2024-05-12T21:59:49.514 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715551189096969, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-05-12T21:59:49.514 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-05-12T21:59:49.514 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715551189097400, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1747, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1715551189, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "bcf229f9-11c8-412e-8536-a774e2d74a7f", "db_session_id": "4JC7U5LZWKMNBY7QIN0P", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-05-12T21:59:49.514 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715551189097480, "job": 1, "event": "recovery_finished"} 2024-05-12T21:59:49.514 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-05-12T21:59:49.514 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-05-12T21:59:49.514 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x557148c3e000 2024-05-12T21:59:49.514 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: DB pointer 0x557148c28000 2024-05-12T21:59:49.514 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-05-12T21:59:49.514 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-05-12T21:59:49.514 INFO:journalctl@ceph.mon.b.smithi086.stdout: ** DB Stats ** 2024-05-12T21:59:49.514 INFO:journalctl@ceph.mon.b.smithi086.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-12T21:59:49.514 INFO:journalctl@ceph.mon.b.smithi086.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-05-12T21:59:49.514 INFO:journalctl@ceph.mon.b.smithi086.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-05-12T21:59:49.515 INFO:journalctl@ceph.mon.b.smithi086.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-05-12T21:59:49.515 INFO:journalctl@ceph.mon.b.smithi086.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-05-12T21:59:49.515 INFO:journalctl@ceph.mon.b.smithi086.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-05-12T21:59:49.515 INFO:journalctl@ceph.mon.b.smithi086.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-05-12T21:59:49.515 INFO:journalctl@ceph.mon.b.smithi086.stdout: 2024-05-12T21:59:49.515 INFO:journalctl@ceph.mon.b.smithi086.stdout: ** Compaction Stats [default] ** 2024-05-12T21:59:49.515 INFO:journalctl@ceph.mon.b.smithi086.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-05-12T21:59:49.515 INFO:journalctl@ceph.mon.b.smithi086.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-05-12T21:59:49.515 INFO:journalctl@ceph.mon.b.smithi086.stdout: L0 1/0 1.71 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.1 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-05-12T21:59:49.515 INFO:journalctl@ceph.mon.b.smithi086.stdout: Sum 1/0 1.71 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.1 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-05-12T21:59:49.515 INFO:journalctl@ceph.mon.b.smithi086.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.1 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-05-12T21:59:49.515 INFO:journalctl@ceph.mon.b.smithi086.stdout: 2024-05-12T21:59:49.515 INFO:journalctl@ceph.mon.b.smithi086.stdout: ** Compaction Stats [default] ** 2024-05-12T21:59:49.515 INFO:journalctl@ceph.mon.b.smithi086.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-05-12T21:59:49.515 INFO:journalctl@ceph.mon.b.smithi086.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-05-12T21:59:49.516 INFO:journalctl@ceph.mon.b.smithi086.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.1 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-05-12T21:59:49.516 INFO:journalctl@ceph.mon.b.smithi086.stdout: 2024-05-12T21:59:49.516 INFO:journalctl@ceph.mon.b.smithi086.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-05-12T21:59:49.516 INFO:journalctl@ceph.mon.b.smithi086.stdout: 2024-05-12T21:59:49.516 INFO:journalctl@ceph.mon.b.smithi086.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-12T21:59:49.516 INFO:journalctl@ceph.mon.b.smithi086.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-05-12T21:59:49.516 INFO:journalctl@ceph.mon.b.smithi086.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-05-12T21:59:49.516 INFO:journalctl@ceph.mon.b.smithi086.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-05-12T21:59:49.516 INFO:journalctl@ceph.mon.b.smithi086.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-05-12T21:59:49.516 INFO:journalctl@ceph.mon.b.smithi086.stdout: AddFile(Keys): cumulative 0, interval 0 2024-05-12T21:59:49.516 INFO:journalctl@ceph.mon.b.smithi086.stdout: Cumulative compaction: 0.00 GB write, 0.68 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-12T21:59:49.516 INFO:journalctl@ceph.mon.b.smithi086.stdout: Interval compaction: 0.00 GB write, 0.68 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-12T21:59:49.516 INFO:journalctl@ceph.mon.b.smithi086.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-05-12T21:59:49.516 INFO:journalctl@ceph.mon.b.smithi086.stdout: Block cache BinnedLRUCache@0x557148b4d090#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-05-12T21:59:49.516 INFO:journalctl@ceph.mon.b.smithi086.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-05-12T21:59:49.517 INFO:journalctl@ceph.mon.b.smithi086.stdout: 2024-05-12T21:59:49.517 INFO:journalctl@ceph.mon.b.smithi086.stdout: ** File Read Latency Histogram By Level [default] ** 2024-05-12T21:59:49.517 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-05-12T21:59:49.517 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: using public_addr v2:172.21.15.86:0/0 -> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-05-12T21:59:49.517 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: starting mon.b rank -1 at public addrs [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] at bind addrs [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 6b92e7b4-10aa-11ef-bc98-c7b262605968 2024-05-12T21:59:49.517 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: mon.b@-1(???) e0 preinit fsid 6b92e7b4-10aa-11ef-bc98-c7b262605968 2024-05-12T21:59:49.517 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: mon.b@-1(synchronizing).mds e1 new map 2024-05-12T21:59:49.517 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: mon.b@-1(synchronizing).mds e1 print_map 2024-05-12T21:59:49.517 INFO:journalctl@ceph.mon.b.smithi086.stdout: e1 2024-05-12T21:59:49.517 INFO:journalctl@ceph.mon.b.smithi086.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-05-12T21:59:49.517 INFO:journalctl@ceph.mon.b.smithi086.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-05-12T21:59:49.517 INFO:journalctl@ceph.mon.b.smithi086.stdout: legacy client fscid: -1 2024-05-12T21:59:49.517 INFO:journalctl@ceph.mon.b.smithi086.stdout: 2024-05-12T21:59:49.517 INFO:journalctl@ceph.mon.b.smithi086.stdout: No filesystems configured 2024-05-12T21:59:49.517 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-05-12T21:59:49.518 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-05-12T21:59:49.518 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-05-12T21:59:49.518 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-05-12T21:59:49.518 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-05-12T21:59:49.518 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-05-12T21:59:49.518 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-05-12T21:59:49.518 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-05-12T21:59:49.518 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-05-12T21:59:49.518 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-05-12T21:59:49.518 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T21:59:49.519 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T21:59:49.519 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:49.519 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T21:59:49.519 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: Added host smithi086 2024-05-12T21:59:49.519 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:49.519 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:49.519 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T21:59:49.519 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:49.519 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T21:59:49.519 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-12T21:59:49.519 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T21:59:49.519 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:49.519 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:49.519 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:49.519 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:49.519 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi086", "name": "osd_memory_target"}]: dispatch 2024-05-12T21:59:49.520 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T21:59:49.520 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T21:59:49.520 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: Updating smithi086:/etc/ceph/ceph.conf 2024-05-12T21:59:49.520 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: Updating smithi086:/var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-12T21:59:49.520 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: Updating smithi086:/etc/ceph/ceph.client.admin.keyring 2024-05-12T21:59:49.520 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: Updating smithi086:/var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/config/ceph.client.admin.keyring 2024-05-12T21:59:49.520 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:49.520 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:49.520 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:49.520 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi186", "target": ["mon-mgr", ""]}]: dispatch 2024-05-12T21:59:49.520 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T21:59:49.520 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: Deploying cephadm binary to smithi186 2024-05-12T21:59:49.520 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T21:59:49.520 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T21:59:49.520 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:49.521 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T21:59:49.521 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:49.521 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: Added host smithi186 2024-05-12T21:59:49.521 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:49.521 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T21:59:49.521 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:49.521 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T21:59:49.521 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-12T21:59:49.521 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T21:59:49.521 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:49.521 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:49.521 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:49.522 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:49.522 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi186", "name": "osd_memory_target"}]: dispatch 2024-05-12T21:59:49.522 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T21:59:49.522 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T21:59:49.522 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: Updating smithi186:/etc/ceph/ceph.conf 2024-05-12T21:59:49.522 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: Updating smithi186:/var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-12T21:59:49.522 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: Updating smithi186:/etc/ceph/ceph.client.admin.keyring 2024-05-12T21:59:49.522 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: Updating smithi186:/var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/config/ceph.client.admin.keyring 2024-05-12T21:59:49.522 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/1881656745' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-05-12T21:59:49.522 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:49.522 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:49.522 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:49.523 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T21:59:49.523 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/1881656745' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-05-12T21:59:49.523 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: osdmap e4: 0 total, 0 up, 0 in 2024-05-12T21:59:49.523 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T21:59:49.523 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:49.523 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T21:59:49.523 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T21:59:49.523 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T21:59:49.523 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:49.523 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-12T21:59:49.523 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T21:59:49.523 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi028:172.21.15.28=a;smithi086:172.21.15.86=b;smithi186:172.21.15.186=c", "target": ["mon-mgr", ""]}]: dispatch 2024-05-12T21:59:49.523 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: Saving service mon spec with placement smithi028:172.21.15.28=a;smithi086:172.21.15.86=b;smithi186:172.21.15.186=c;count:3 2024-05-12T21:59:49.523 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: Deploying daemon mon.c on smithi186 2024-05-12T21:59:49.523 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T21:59:49.523 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T21:59:49.524 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T21:59:49.524 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-12T21:59:49.524 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-12T21:59:49.524 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: mon.a calling monitor election 2024-05-12T21:59:49.524 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-12T21:59:49.524 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-12T21:59:49.524 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T21:59:49.524 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-12T21:59:49.524 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: mon.c calling monitor election 2024-05-12T21:59:49.524 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-12T21:59:49.524 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T21:59:49.524 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-12T21:59:49.524 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-12T21:59:49.524 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-05-12T21:59:49.524 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: monmap epoch 2 2024-05-12T21:59:49.525 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: fsid 6b92e7b4-10aa-11ef-bc98-c7b262605968 2024-05-12T21:59:49.525 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: last_changed 2024-05-12T21:59:39.980086+0000 2024-05-12T21:59:49.525 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: created 2024-05-12T21:57:33.130972+0000 2024-05-12T21:59:49.525 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: min_mon_release 19 (squid) 2024-05-12T21:59:49.525 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: election_strategy: 1 2024-05-12T21:59:49.525 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: 0: [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] mon.a 2024-05-12T21:59:49.525 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: 1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.c 2024-05-12T21:59:49.525 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: fsmap 2024-05-12T21:59:49.525 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: osdmap e4: 0 total, 0 up, 0 in 2024-05-12T21:59:49.525 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: mgrmap e13: x(active, since 59s) 2024-05-12T21:59:49.525 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: overall HEALTH_OK 2024-05-12T21:59:49.525 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:49.525 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:49.525 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:49.525 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-12T21:59:49.525 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T21:59:49.526 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: Deploying daemon mon.b on smithi086 2024-05-12T21:59:49.526 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-12T21:59:49.526 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T21:59:49.526 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: from='client.? 172.21.15.186:0/2370675227' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-12T21:59:49.526 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T21:59:49.526 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 ceph-mon[34297]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-05-12T21:59:49.526 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 bash[34277]: d12e520b6c5ccd7d5e17136e3f52defceaf31c757111ebdcdffcb1bff67132fc 2024-05-12T21:59:49.526 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:49 smithi086 systemd[1]: Started Ceph mon.b for 6b92e7b4-10aa-11ef-bc98-c7b262605968. 2024-05-12T21:59:49.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:49 smithi028 ceph-mon[29847]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T21:59:54.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:54 smithi028 ceph-mon[29847]: mon.c calling monitor election 2024-05-12T21:59:54.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:54 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-12T21:59:54.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:54 smithi028 ceph-mon[29847]: mon.a calling monitor election 2024-05-12T21:59:54.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:54 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-12T21:59:54.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:54 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-12T21:59:54.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:54 smithi028 ceph-mon[29847]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T21:59:54.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:54 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-12T21:59:54.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:54 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-12T21:59:54.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:54 smithi028 ceph-mon[29847]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T21:59:54.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:54 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-12T21:59:54.537 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:54 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-12T21:59:54.537 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:54 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-12T21:59:54.537 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:54 smithi028 ceph-mon[29847]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-05-12T21:59:54.537 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:54 smithi028 ceph-mon[29847]: monmap epoch 3 2024-05-12T21:59:54.537 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:54 smithi028 ceph-mon[29847]: fsid 6b92e7b4-10aa-11ef-bc98-c7b262605968 2024-05-12T21:59:54.537 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:54 smithi028 ceph-mon[29847]: last_changed 2024-05-12T21:59:49.125381+0000 2024-05-12T21:59:54.537 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:54 smithi028 ceph-mon[29847]: created 2024-05-12T21:57:33.130972+0000 2024-05-12T21:59:54.537 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:54 smithi028 ceph-mon[29847]: min_mon_release 19 (squid) 2024-05-12T21:59:54.537 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:54 smithi028 ceph-mon[29847]: election_strategy: 1 2024-05-12T21:59:54.537 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:54 smithi028 ceph-mon[29847]: 0: [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] mon.a 2024-05-12T21:59:54.537 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:54 smithi028 ceph-mon[29847]: 1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.c 2024-05-12T21:59:54.537 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:54 smithi028 ceph-mon[29847]: 2: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.b 2024-05-12T21:59:54.538 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:54 smithi028 ceph-mon[29847]: fsmap 2024-05-12T21:59:54.538 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:54 smithi028 ceph-mon[29847]: osdmap e4: 0 total, 0 up, 0 in 2024-05-12T21:59:54.538 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:54 smithi028 ceph-mon[29847]: mgrmap e13: x(active, since 68s) 2024-05-12T21:59:54.538 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:54 smithi028 ceph-mon[29847]: overall HEALTH_OK 2024-05-12T21:59:54.538 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:54 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:54.538 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:54 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:54.538 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:54 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:54.538 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:54 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:54.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:54 smithi186 ceph-mon[34785]: mon.c calling monitor election 2024-05-12T21:59:54.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:54 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-12T21:59:54.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:54 smithi186 ceph-mon[34785]: mon.a calling monitor election 2024-05-12T21:59:54.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:54 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-12T21:59:54.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:54 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-12T21:59:54.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:54 smithi186 ceph-mon[34785]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T21:59:54.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:54 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-12T21:59:54.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:54 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-12T21:59:54.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:54 smithi186 ceph-mon[34785]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T21:59:54.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:54 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-12T21:59:54.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:54 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-12T21:59:54.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:54 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-12T21:59:54.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:54 smithi186 ceph-mon[34785]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-05-12T21:59:54.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:54 smithi186 ceph-mon[34785]: monmap epoch 3 2024-05-12T21:59:54.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:54 smithi186 ceph-mon[34785]: fsid 6b92e7b4-10aa-11ef-bc98-c7b262605968 2024-05-12T21:59:54.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:54 smithi186 ceph-mon[34785]: last_changed 2024-05-12T21:59:49.125381+0000 2024-05-12T21:59:54.681 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:54 smithi186 ceph-mon[34785]: created 2024-05-12T21:57:33.130972+0000 2024-05-12T21:59:54.681 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:54 smithi186 ceph-mon[34785]: min_mon_release 19 (squid) 2024-05-12T21:59:54.681 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:54 smithi186 ceph-mon[34785]: election_strategy: 1 2024-05-12T21:59:54.681 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:54 smithi186 ceph-mon[34785]: 0: [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] mon.a 2024-05-12T21:59:54.681 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:54 smithi186 ceph-mon[34785]: 1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.c 2024-05-12T21:59:54.681 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:54 smithi186 ceph-mon[34785]: 2: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.b 2024-05-12T21:59:54.681 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:54 smithi186 ceph-mon[34785]: fsmap 2024-05-12T21:59:54.681 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:54 smithi186 ceph-mon[34785]: osdmap e4: 0 total, 0 up, 0 in 2024-05-12T21:59:54.681 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:54 smithi186 ceph-mon[34785]: mgrmap e13: x(active, since 68s) 2024-05-12T21:59:54.681 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:54 smithi186 ceph-mon[34785]: overall HEALTH_OK 2024-05-12T21:59:54.681 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:54 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:54.681 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:54 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:54.681 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:54 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:54.681 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:54 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:55.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:55 smithi028 ceph-mon[29847]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T21:59:55.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:55 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T21:59:55.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:55 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-12T21:59:55.560 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:55.560+0000 7f139ed2a700 1 -- 172.21.15.186:0/1782173316 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13880035c0 msgr2=0x7f1388005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T21:59:55.560 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:55.560+0000 7f139ed2a700 1 --2- 172.21.15.186:0/1782173316 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13880035c0 0x7f1388005a50 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f1398101d60 tx=0x7f138c00ff50 comp rx=0 tx=0).stop 2024-05-12T21:59:55.560 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:55.560+0000 7f139ed2a700 1 -- 172.21.15.186:0/1782173316 shutdown_connections 2024-05-12T21:59:55.560 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:55.560+0000 7f139ed2a700 1 --2- 172.21.15.186:0/1782173316 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13880035c0 0x7f1388005a50 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:59:55.560 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:55.560+0000 7f139ed2a700 1 -- 172.21.15.186:0/1782173316 >> 172.21.15.186:0/1782173316 conn(0x7f13980fd920 msgr2=0x7f13980ffd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T21:59:55.560 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:55.560+0000 7f139ed2a700 1 -- 172.21.15.186:0/1782173316 shutdown_connections 2024-05-12T21:59:55.560 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:55.560+0000 7f139ed2a700 1 -- 172.21.15.186:0/1782173316 wait complete. 2024-05-12T21:59:55.561 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:55 smithi186 ceph-mon[34785]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T21:59:55.561 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:55 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T21:59:55.561 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:55 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-12T21:59:55.563 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:55.560+0000 7f139ed2a700 1 Processor -- start 2024-05-12T21:59:55.563 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:55.561+0000 7f139ed2a700 1 -- start start 2024-05-12T21:59:55.563 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:55.561+0000 7f139ed2a700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f13880035c0 0x7f139819d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T21:59:55.563 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:55.561+0000 7f139ed2a700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f139819dd00 0x7f1398197980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T21:59:55.564 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:55.561+0000 7f139ed2a700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1398197ec0 0x7f1398198370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T21:59:55.564 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:55.561+0000 7f139ed2a700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1398109e50 con 0x7f1398197ec0 2024-05-12T21:59:55.564 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:55.561+0000 7f139ed2a700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f1398109cd0 con 0x7f139819dd00 2024-05-12T21:59:55.564 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:55.561+0000 7f139ed2a700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f1398109fd0 con 0x7f13880035c0 2024-05-12T21:59:55.564 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:55.561+0000 7f139cac6700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f13880035c0 0x7f139819d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T21:59:55.564 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:55.561+0000 7f139cac6700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f13880035c0 0x7f139819d7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:36668/0 (socket says 172.21.15.186:36668) 2024-05-12T21:59:55.564 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:55.561+0000 7f139cac6700 1 -- 172.21.15.186:0/712776077 learned_addr learned my addr 172.21.15.186:0/712776077 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-05-12T21:59:55.564 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:55.561+0000 7f1397fff700 1 --2- 172.21.15.186:0/712776077 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f139819dd00 0x7f1398197980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T21:59:55.564 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:55.561+0000 7f139d2c7700 1 --2- 172.21.15.186:0/712776077 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1398197ec0 0x7f1398198370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T21:59:55.564 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:55.561+0000 7f1397fff700 1 -- 172.21.15.186:0/712776077 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f139819dd00 msgr2=0x7f1398197980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-05-12T21:59:55.564 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:55.561+0000 7f1397fff700 1 -- 172.21.15.186:0/712776077 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f139819dd00 msgr2=0x7f1398197980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-05-12T21:59:55.565 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:55.561+0000 7f1397fff700 1 --2- 172.21.15.186:0/712776077 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f139819dd00 0x7f1398197980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-12T21:59:55.565 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:55.561+0000 7f1397fff700 1 --2- 172.21.15.186:0/712776077 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f139819dd00 0x7f1398197980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-12T21:59:55.565 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:55.561+0000 7f139cac6700 1 -- 172.21.15.186:0/712776077 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f139819dd00 msgr2=0x7f1398197980 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-12T21:59:55.565 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:55.561+0000 7f139cac6700 1 --2- 172.21.15.186:0/712776077 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f139819dd00 0x7f1398197980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:59:55.565 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:55.561+0000 7f139cac6700 1 -- 172.21.15.186:0/712776077 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1398197ec0 msgr2=0x7f1398198370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T21:59:55.565 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:55.561+0000 7f139cac6700 1 --2- 172.21.15.186:0/712776077 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1398197ec0 0x7f1398198370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:59:55.565 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:55.562+0000 7f139cac6700 1 -- 172.21.15.186:0/712776077 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13981a4690 con 0x7f13880035c0 2024-05-12T21:59:55.565 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:55.562+0000 7f139d2c7700 1 --2- 172.21.15.186:0/712776077 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1398197ec0 0x7f1398198370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T21:59:55.565 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:55.562+0000 7f139cac6700 1 --2- 172.21.15.186:0/712776077 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f13880035c0 0x7f139819d7c0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f138c0059d0 tx=0x7f138c003d70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T21:59:55.565 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:55.562+0000 7f1395ffb700 1 -- 172.21.15.186:0/712776077 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f138c02a030 con 0x7f13880035c0 2024-05-12T21:59:55.565 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:55.562+0000 7f139ed2a700 1 -- 172.21.15.186:0/712776077 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13981a4860 con 0x7f13880035c0 2024-05-12T21:59:55.565 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:55.562+0000 7f1395ffb700 1 -- 172.21.15.186:0/712776077 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f138c012bf0 con 0x7f13880035c0 2024-05-12T21:59:55.565 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:55.562+0000 7f1395ffb700 1 -- 172.21.15.186:0/712776077 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f138c026600 con 0x7f13880035c0 2024-05-12T21:59:55.565 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:55.562+0000 7f139ed2a700 1 -- 172.21.15.186:0/712776077 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13981a4d40 con 0x7f13880035c0 2024-05-12T21:59:55.565 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:55.563+0000 7f1395ffb700 1 -- 172.21.15.186:0/712776077 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f138c02d030 con 0x7f13880035c0 2024-05-12T21:59:55.566 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:55.563+0000 7f139ed2a700 1 -- 172.21.15.186:0/712776077 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1398063980 con 0x7f13880035c0 2024-05-12T21:59:55.566 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:55.563+0000 7f1395ffb700 1 --2- 172.21.15.186:0/712776077 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f137c039580 0x7f137c03ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T21:59:55.566 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:55.563+0000 7f1395ffb700 1 -- 172.21.15.186:0/712776077 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f138c04eb90 con 0x7f13880035c0 2024-05-12T21:59:55.566 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:55.564+0000 7f1397fff700 1 --2- 172.21.15.186:0/712776077 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f137c039580 0x7f137c03ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T21:59:55.566 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:55.565+0000 7f1397fff700 1 --2- 172.21.15.186:0/712776077 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f137c039580 0x7f137c03ba40 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f1384009850 tx=0x7f1384008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T21:59:55.568 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:55.568+0000 7f1395ffb700 1 -- 172.21.15.186:0/712776077 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f138c012da0 con 0x7f13880035c0 2024-05-12T21:59:55.848 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:55.848+0000 7f139ed2a700 1 -- 172.21.15.186:0/712776077 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f139802db80 con 0x7f13880035c0 2024-05-12T21:59:55.848 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:55.848+0000 7f1395ffb700 1 -- 172.21.15.186:0/712776077 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) ==== 95+0+1279 (secure 0 0 0) 0x7f138c02d320 con 0x7f13880035c0 2024-05-12T21:59:55.848 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-12T21:59:55.848 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":3,"fsid":"6b92e7b4-10aa-11ef-bc98-c7b262605968","modified":"2024-05-12T21:59:49.125381Z","created":"2024-05-12T21:57:33.130972Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:3300","nonce":0},{"type":"v1","addr":"172.21.15.28:6789","nonce":0}]},"addr":"172.21.15.28:6789/0","public_addr":"172.21.15.28:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:3300","nonce":0},{"type":"v1","addr":"172.21.15.186:6789","nonce":0}]},"addr":"172.21.15.186:6789/0","public_addr":"172.21.15.186:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-05-12T21:59:55.851 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:55.851+0000 7f139ed2a700 1 -- 172.21.15.186:0/712776077 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f137c039580 msgr2=0x7f137c03ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T21:59:55.851 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:55.851+0000 7f139ed2a700 1 --2- 172.21.15.186:0/712776077 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f137c039580 0x7f137c03ba40 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f1384009850 tx=0x7f1384008040 comp rx=0 tx=0).stop 2024-05-12T21:59:55.851 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:55.851+0000 7f139ed2a700 1 -- 172.21.15.186:0/712776077 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f13880035c0 msgr2=0x7f139819d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T21:59:55.851 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:55.851+0000 7f139ed2a700 1 --2- 172.21.15.186:0/712776077 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f13880035c0 0x7f139819d7c0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f138c0059d0 tx=0x7f138c003d70 comp rx=0 tx=0).stop 2024-05-12T21:59:55.851 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:55.851+0000 7f139ed2a700 1 -- 172.21.15.186:0/712776077 shutdown_connections 2024-05-12T21:59:55.851 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:55.851+0000 7f139ed2a700 1 --2- 172.21.15.186:0/712776077 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f137c039580 0x7f137c03ba40 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:59:55.851 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:55.851+0000 7f139ed2a700 1 --2- 172.21.15.186:0/712776077 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f13880035c0 0x7f139819d7c0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:59:55.851 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:55.851+0000 7f139ed2a700 1 --2- 172.21.15.186:0/712776077 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1398197ec0 0x7f1398198370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:59:55.851 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:55.851+0000 7f139ed2a700 1 --2- 172.21.15.186:0/712776077 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f139819dd00 0x7f1398197980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:59:55.851 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:55.851+0000 7f139ed2a700 1 -- 172.21.15.186:0/712776077 >> 172.21.15.186:0/712776077 conn(0x7f13980fd920 msgr2=0x7f13980fee80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T21:59:55.851 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:55.851+0000 7f139ed2a700 1 -- 172.21.15.186:0/712776077 shutdown_connections 2024-05-12T21:59:55.851 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T21:59:55.851+0000 7f139ed2a700 1 -- 172.21.15.186:0/712776077 wait complete. 2024-05-12T21:59:55.852 INFO:teuthology.orchestra.run.smithi186.stderr:dumped monmap epoch 3 2024-05-12T21:59:56.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:56 smithi086 ceph-mon[34297]: mon.c calling monitor election 2024-05-12T21:59:56.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:56 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-12T21:59:56.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:56 smithi086 ceph-mon[34297]: mon.a calling monitor election 2024-05-12T21:59:56.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:56 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-12T21:59:56.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:56 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-12T21:59:56.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:56 smithi086 ceph-mon[34297]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T21:59:56.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:56 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-12T21:59:56.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:56 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-12T21:59:56.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:56 smithi086 ceph-mon[34297]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T21:59:56.494 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:56 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-12T21:59:56.494 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:56 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-12T21:59:56.494 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:56 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-12T21:59:56.494 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:56 smithi086 ceph-mon[34297]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-05-12T21:59:56.494 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:56 smithi086 ceph-mon[34297]: monmap epoch 3 2024-05-12T21:59:56.494 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:56 smithi086 ceph-mon[34297]: fsid 6b92e7b4-10aa-11ef-bc98-c7b262605968 2024-05-12T21:59:56.494 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:56 smithi086 ceph-mon[34297]: last_changed 2024-05-12T21:59:49.125381+0000 2024-05-12T21:59:56.494 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:56 smithi086 ceph-mon[34297]: created 2024-05-12T21:57:33.130972+0000 2024-05-12T21:59:56.494 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:56 smithi086 ceph-mon[34297]: min_mon_release 19 (squid) 2024-05-12T21:59:56.494 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:56 smithi086 ceph-mon[34297]: election_strategy: 1 2024-05-12T21:59:56.495 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:56 smithi086 ceph-mon[34297]: 0: [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] mon.a 2024-05-12T21:59:56.495 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:56 smithi086 ceph-mon[34297]: 1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.c 2024-05-12T21:59:56.495 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:56 smithi086 ceph-mon[34297]: 2: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.b 2024-05-12T21:59:56.495 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:56 smithi086 ceph-mon[34297]: fsmap 2024-05-12T21:59:56.495 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:56 smithi086 ceph-mon[34297]: osdmap e4: 0 total, 0 up, 0 in 2024-05-12T21:59:56.495 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:56 smithi086 ceph-mon[34297]: mgrmap e13: x(active, since 68s) 2024-05-12T21:59:56.495 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:56 smithi086 ceph-mon[34297]: overall HEALTH_OK 2024-05-12T21:59:56.495 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:56 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:56.495 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:56 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:56.495 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:56 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:56.495 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:56 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:56.496 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:56 smithi086 ceph-mon[34297]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T21:59:56.496 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:56 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T21:59:56.496 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:56 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-12T21:59:56.560 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-05-12T21:59:56.561 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f72fecff68e1d400c4568684327c900485c20d6a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6b92e7b4-10aa-11ef-bc98-c7b262605968 -- ceph config generate-minimal-conf 2024-05-12T21:59:56.983 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/mon.a/config 2024-05-12T21:59:57.008 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:56 smithi028 ceph-mon[29847]: mon.b calling monitor election 2024-05-12T21:59:57.008 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:56 smithi028 ceph-mon[29847]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T21:59:57.008 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:56 smithi028 ceph-mon[29847]: Updating smithi028:/etc/ceph/ceph.conf 2024-05-12T21:59:57.008 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:56 smithi028 ceph-mon[29847]: Updating smithi086:/etc/ceph/ceph.conf 2024-05-12T21:59:57.008 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:56 smithi028 ceph-mon[29847]: Updating smithi186:/etc/ceph/ceph.conf 2024-05-12T21:59:57.008 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:56 smithi028 ceph-mon[29847]: mon.b calling monitor election 2024-05-12T21:59:57.008 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:56 smithi028 ceph-mon[29847]: mon.c calling monitor election 2024-05-12T21:59:57.008 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:56 smithi028 ceph-mon[29847]: mon.a calling monitor election 2024-05-12T21:59:57.008 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:56 smithi028 ceph-mon[29847]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-05-12T21:59:57.009 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:56 smithi028 ceph-mon[29847]: monmap epoch 3 2024-05-12T21:59:57.009 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:56 smithi028 ceph-mon[29847]: fsid 6b92e7b4-10aa-11ef-bc98-c7b262605968 2024-05-12T21:59:57.009 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:56 smithi028 ceph-mon[29847]: last_changed 2024-05-12T21:59:49.125381+0000 2024-05-12T21:59:57.009 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:56 smithi028 ceph-mon[29847]: created 2024-05-12T21:57:33.130972+0000 2024-05-12T21:59:57.009 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:56 smithi028 ceph-mon[29847]: min_mon_release 19 (squid) 2024-05-12T21:59:57.009 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:56 smithi028 ceph-mon[29847]: election_strategy: 1 2024-05-12T21:59:57.009 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:56 smithi028 ceph-mon[29847]: 0: [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] mon.a 2024-05-12T21:59:57.009 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:56 smithi028 ceph-mon[29847]: 1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.c 2024-05-12T21:59:57.009 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:56 smithi028 ceph-mon[29847]: 2: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.b 2024-05-12T21:59:57.009 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:56 smithi028 ceph-mon[29847]: fsmap 2024-05-12T21:59:57.009 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:56 smithi028 ceph-mon[29847]: osdmap e4: 0 total, 0 up, 0 in 2024-05-12T21:59:57.009 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:56 smithi028 ceph-mon[29847]: mgrmap e13: x(active, since 70s) 2024-05-12T21:59:57.009 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:56 smithi028 ceph-mon[29847]: overall HEALTH_OK 2024-05-12T21:59:57.009 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:56 smithi028 ceph-mon[29847]: Updating smithi086:/var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-12T21:59:57.009 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:56 smithi028 ceph-mon[29847]: Updating smithi186:/var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-12T21:59:57.010 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:56 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:57.010 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:56 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:57.010 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:56 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:57.010 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:56 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:57.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:56 smithi186 ceph-mon[34785]: mon.b calling monitor election 2024-05-12T21:59:57.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:56 smithi186 ceph-mon[34785]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T21:59:57.180 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:56 smithi186 ceph-mon[34785]: Updating smithi028:/etc/ceph/ceph.conf 2024-05-12T21:59:57.180 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:56 smithi186 ceph-mon[34785]: Updating smithi086:/etc/ceph/ceph.conf 2024-05-12T21:59:57.180 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:56 smithi186 ceph-mon[34785]: Updating smithi186:/etc/ceph/ceph.conf 2024-05-12T21:59:57.180 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:56 smithi186 ceph-mon[34785]: mon.b calling monitor election 2024-05-12T21:59:57.180 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:56 smithi186 ceph-mon[34785]: mon.c calling monitor election 2024-05-12T21:59:57.180 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:56 smithi186 ceph-mon[34785]: mon.a calling monitor election 2024-05-12T21:59:57.180 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:56 smithi186 ceph-mon[34785]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-05-12T21:59:57.180 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:56 smithi186 ceph-mon[34785]: monmap epoch 3 2024-05-12T21:59:57.180 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:56 smithi186 ceph-mon[34785]: fsid 6b92e7b4-10aa-11ef-bc98-c7b262605968 2024-05-12T21:59:57.180 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:56 smithi186 ceph-mon[34785]: last_changed 2024-05-12T21:59:49.125381+0000 2024-05-12T21:59:57.180 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:56 smithi186 ceph-mon[34785]: created 2024-05-12T21:57:33.130972+0000 2024-05-12T21:59:57.180 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:56 smithi186 ceph-mon[34785]: min_mon_release 19 (squid) 2024-05-12T21:59:57.180 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:56 smithi186 ceph-mon[34785]: election_strategy: 1 2024-05-12T21:59:57.180 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:56 smithi186 ceph-mon[34785]: 0: [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] mon.a 2024-05-12T21:59:57.180 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:56 smithi186 ceph-mon[34785]: 1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.c 2024-05-12T21:59:57.181 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:56 smithi186 ceph-mon[34785]: 2: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.b 2024-05-12T21:59:57.181 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:56 smithi186 ceph-mon[34785]: fsmap 2024-05-12T21:59:57.181 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:56 smithi186 ceph-mon[34785]: osdmap e4: 0 total, 0 up, 0 in 2024-05-12T21:59:57.181 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:56 smithi186 ceph-mon[34785]: mgrmap e13: x(active, since 70s) 2024-05-12T21:59:57.181 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:56 smithi186 ceph-mon[34785]: overall HEALTH_OK 2024-05-12T21:59:57.181 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:56 smithi186 ceph-mon[34785]: Updating smithi086:/var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-12T21:59:57.181 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:56 smithi186 ceph-mon[34785]: Updating smithi186:/var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-12T21:59:57.181 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:56 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:57.181 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:56 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:57.181 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:56 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:57.182 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:56 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:57.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:56 smithi086 ceph-mon[34297]: mon.b calling monitor election 2024-05-12T21:59:57.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:56 smithi086 ceph-mon[34297]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T21:59:57.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:56 smithi086 ceph-mon[34297]: Updating smithi028:/etc/ceph/ceph.conf 2024-05-12T21:59:57.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:56 smithi086 ceph-mon[34297]: Updating smithi086:/etc/ceph/ceph.conf 2024-05-12T21:59:57.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:56 smithi086 ceph-mon[34297]: Updating smithi186:/etc/ceph/ceph.conf 2024-05-12T21:59:57.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:56 smithi086 ceph-mon[34297]: mon.b calling monitor election 2024-05-12T21:59:57.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:56 smithi086 ceph-mon[34297]: mon.c calling monitor election 2024-05-12T21:59:57.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:56 smithi086 ceph-mon[34297]: mon.a calling monitor election 2024-05-12T21:59:57.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:56 smithi086 ceph-mon[34297]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-05-12T21:59:57.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:56 smithi086 ceph-mon[34297]: monmap epoch 3 2024-05-12T21:59:57.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:56 smithi086 ceph-mon[34297]: fsid 6b92e7b4-10aa-11ef-bc98-c7b262605968 2024-05-12T21:59:57.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:56 smithi086 ceph-mon[34297]: last_changed 2024-05-12T21:59:49.125381+0000 2024-05-12T21:59:57.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:56 smithi086 ceph-mon[34297]: created 2024-05-12T21:57:33.130972+0000 2024-05-12T21:59:57.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:56 smithi086 ceph-mon[34297]: min_mon_release 19 (squid) 2024-05-12T21:59:57.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:56 smithi086 ceph-mon[34297]: election_strategy: 1 2024-05-12T21:59:57.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:56 smithi086 ceph-mon[34297]: 0: [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] mon.a 2024-05-12T21:59:57.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:56 smithi086 ceph-mon[34297]: 1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.c 2024-05-12T21:59:57.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:56 smithi086 ceph-mon[34297]: 2: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.b 2024-05-12T21:59:57.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:56 smithi086 ceph-mon[34297]: fsmap 2024-05-12T21:59:57.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:56 smithi086 ceph-mon[34297]: osdmap e4: 0 total, 0 up, 0 in 2024-05-12T21:59:57.244 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:56 smithi086 ceph-mon[34297]: mgrmap e13: x(active, since 70s) 2024-05-12T21:59:57.244 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:56 smithi086 ceph-mon[34297]: overall HEALTH_OK 2024-05-12T21:59:57.244 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:56 smithi086 ceph-mon[34297]: Updating smithi086:/var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-12T21:59:57.244 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:56 smithi086 ceph-mon[34297]: Updating smithi186:/var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-12T21:59:57.244 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:56 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:57.244 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:56 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:57.244 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:56 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:57.244 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:56 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:57.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:57 smithi086 ceph-mon[34297]: Updating smithi028:/var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-12T21:59:57.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:57 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-12T21:59:57.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:57 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:57.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:57 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:57.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:57 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:57.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:57 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:57.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:57 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:57.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:57 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:57.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:57 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:57.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:57 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-12T21:59:57.993 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:57 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-12T21:59:57.993 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:57 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T21:59:58.165 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:57 smithi028 ceph-mon[29847]: Updating smithi028:/var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-12T21:59:58.165 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:57 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-12T21:59:58.165 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:57 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:58.165 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:57 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:58.165 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:57 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:58.165 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:57 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:58.165 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:57 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:58.165 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:57 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:58.165 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:57 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:58.165 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:57 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-12T21:59:58.165 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:57 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-12T21:59:58.165 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:57 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T21:59:58.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:57 smithi186 ceph-mon[34785]: Updating smithi028:/var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-12T21:59:58.180 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:57 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-12T21:59:58.180 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:57 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:58.180 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:57 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:58.180 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:57 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:58.180 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:57 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:58.180 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:57 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:58.180 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:57 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:58.180 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:57 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T21:59:58.180 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:57 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-12T21:59:58.180 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:57 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-12T21:59:58.180 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:57 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T21:59:59.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:58 smithi028 ceph-mon[29847]: Reconfiguring mon.a (unknown last config time)... 2024-05-12T21:59:59.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:58 smithi028 ceph-mon[29847]: Reconfiguring daemon mon.a on smithi028 2024-05-12T21:59:59.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:58 smithi028 ceph-mon[29847]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T21:59:59.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:58 smithi186 ceph-mon[34785]: Reconfiguring mon.a (unknown last config time)... 2024-05-12T21:59:59.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:58 smithi186 ceph-mon[34785]: Reconfiguring daemon mon.a on smithi028 2024-05-12T21:59:59.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:58 smithi186 ceph-mon[34785]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T21:59:59.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:59.234+0000 7f922be00700 1 -- 172.21.15.28:0/3926165427 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9224101f90 msgr2=0x7f9224102370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T21:59:59.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:59.234+0000 7f922be00700 1 --2- 172.21.15.28:0/3926165427 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9224101f90 0x7f9224102370 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f9218009910 tx=0x7f9218009c20 comp rx=0 tx=0).stop 2024-05-12T21:59:59.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:59.234+0000 7f922be00700 1 -- 172.21.15.28:0/3926165427 shutdown_connections 2024-05-12T21:59:59.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:59.234+0000 7f922be00700 1 --2- 172.21.15.28:0/3926165427 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9224101f90 0x7f9224102370 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:59:59.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:59.234+0000 7f922be00700 1 -- 172.21.15.28:0/3926165427 >> 172.21.15.28:0/3926165427 conn(0x7f92240fd860 msgr2=0x7f92240ffc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T21:59:59.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:59.235+0000 7f922be00700 1 -- 172.21.15.28:0/3926165427 shutdown_connections 2024-05-12T21:59:59.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:59.235+0000 7f922be00700 1 -- 172.21.15.28:0/3926165427 wait complete. 2024-05-12T21:59:59.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:59.236+0000 7f922be00700 1 Processor -- start 2024-05-12T21:59:59.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:59.237+0000 7f922be00700 1 -- start start 2024-05-12T21:59:59.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:59.237+0000 7f922be00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9224101f90 0x7f922419d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T21:59:59.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:59.237+0000 7f922be00700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f922419dc80 0x7f9224197900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T21:59:59.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:59.237+0000 7f922be00700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9224197e40 0x7f92241982f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T21:59:59.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:59.237+0000 7f922be00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f922410c360 con 0x7f9224101f90 2024-05-12T21:59:59.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:59.237+0000 7f922be00700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f922410c1e0 con 0x7f9224197e40 2024-05-12T21:59:59.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:59.237+0000 7f922be00700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f922410c4e0 con 0x7f922419dc80 2024-05-12T21:59:59.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:59.237+0000 7f9229400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9224101f90 0x7f922419d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T21:59:59.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:59.237+0000 7f9229400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9224101f90 0x7f922419d740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43910/0 (socket says 172.21.15.28:43910) 2024-05-12T21:59:59.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:59.237+0000 7f9229400700 1 -- 172.21.15.28:0/268983875 learned_addr learned my addr 172.21.15.28:0/268983875 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T21:59:59.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:59.238+0000 7f9229e00700 1 --2- 172.21.15.28:0/268983875 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9224197e40 0x7f92241982f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T21:59:59.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:59.238+0000 7f9228a00700 1 --2- 172.21.15.28:0/268983875 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f922419dc80 0x7f9224197900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T21:59:59.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:59.238+0000 7f9229400700 1 -- 172.21.15.28:0/268983875 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f922419dc80 msgr2=0x7f9224197900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T21:59:59.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:59.238+0000 7f9229400700 1 --2- 172.21.15.28:0/268983875 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f922419dc80 0x7f9224197900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:59:59.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:59.238+0000 7f9229400700 1 -- 172.21.15.28:0/268983875 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9224197e40 msgr2=0x7f92241982f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T21:59:59.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:59.238+0000 7f9229400700 1 --2- 172.21.15.28:0/268983875 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9224197e40 0x7f92241982f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:59:59.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:59.238+0000 7f9229400700 1 -- 172.21.15.28:0/268983875 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92241a4610 con 0x7f9224101f90 2024-05-12T21:59:59.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:59.238+0000 7f9228a00700 1 --2- 172.21.15.28:0/268983875 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f922419dc80 0x7f9224197900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-12T21:59:59.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:59.239+0000 7f9229400700 1 --2- 172.21.15.28:0/268983875 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9224101f90 0x7f922419d740 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f9218005850 tx=0x7f9218004f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T21:59:59.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:59.239+0000 7f9222000700 1 -- 172.21.15.28:0/268983875 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9218026070 con 0x7f9224101f90 2024-05-12T21:59:59.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:59.239+0000 7f9222000700 1 -- 172.21.15.28:0/268983875 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f921800b6f0 con 0x7f9224101f90 2024-05-12T21:59:59.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:59.239+0000 7f922be00700 1 -- 172.21.15.28:0/268983875 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92241a4840 con 0x7f9224101f90 2024-05-12T21:59:59.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:59.239+0000 7f9222000700 1 -- 172.21.15.28:0/268983875 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92180038e0 con 0x7f9224101f90 2024-05-12T21:59:59.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:59.239+0000 7f922be00700 1 -- 172.21.15.28:0/268983875 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92241a4dd0 con 0x7f9224101f90 2024-05-12T21:59:59.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:59.240+0000 7f922be00700 1 -- 172.21.15.28:0/268983875 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9224063980 con 0x7f9224101f90 2024-05-12T21:59:59.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:59.241+0000 7f9222000700 1 -- 172.21.15.28:0/268983875 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f9218029070 con 0x7f9224101f90 2024-05-12T21:59:59.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:58 smithi086 ceph-mon[34297]: Reconfiguring mon.a (unknown last config time)... 2024-05-12T21:59:59.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:58 smithi086 ceph-mon[34297]: Reconfiguring daemon mon.a on smithi028 2024-05-12T21:59:59.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:58 smithi086 ceph-mon[34297]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T21:59:59.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:59.241+0000 7f9222000700 1 --2- 172.21.15.28:0/268983875 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f920c039580 0x7f920c03ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T21:59:59.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:59.241+0000 7f9222000700 1 -- 172.21.15.28:0/268983875 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f921804ce40 con 0x7f9224101f90 2024-05-12T21:59:59.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:59.241+0000 7f9228a00700 1 --2- 172.21.15.28:0/268983875 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f920c039580 0x7f920c03ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T21:59:59.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:59.242+0000 7f9228a00700 1 --2- 172.21.15.28:0/268983875 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f920c039580 0x7f920c03ba40 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f92100079b0 tx=0x7f9210008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T21:59:59.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:59.247+0000 7f9222000700 1 -- 172.21.15.28:0/268983875 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f9218013070 con 0x7f9224101f90 2024-05-12T21:59:59.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:59.482+0000 7f922be00700 1 -- 172.21.15.28:0/268983875 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f92241a4fd0 con 0x7f9224101f90 2024-05-12T21:59:59.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:59.483+0000 7f9222000700 1 -- 172.21.15.28:0/268983875 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) ==== 76+0+273 (secure 0 0 0) 0x7f9218030020 con 0x7f9224101f90 2024-05-12T21:59:59.484 INFO:teuthology.orchestra.run.smithi028.stdout:# minimal ceph.conf for 6b92e7b4-10aa-11ef-bc98-c7b262605968 2024-05-12T21:59:59.484 INFO:teuthology.orchestra.run.smithi028.stdout:[global] 2024-05-12T21:59:59.484 INFO:teuthology.orchestra.run.smithi028.stdout: fsid = 6b92e7b4-10aa-11ef-bc98-c7b262605968 2024-05-12T21:59:59.484 INFO:teuthology.orchestra.run.smithi028.stdout: mon_host = [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-05-12T21:59:59.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:59.486+0000 7f922be00700 1 -- 172.21.15.28:0/268983875 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f920c039580 msgr2=0x7f920c03ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T21:59:59.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:59.486+0000 7f922be00700 1 --2- 172.21.15.28:0/268983875 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f920c039580 0x7f920c03ba40 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f92100079b0 tx=0x7f9210008040 comp rx=0 tx=0).stop 2024-05-12T21:59:59.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:59.486+0000 7f922be00700 1 -- 172.21.15.28:0/268983875 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9224101f90 msgr2=0x7f922419d740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T21:59:59.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:59.486+0000 7f922be00700 1 --2- 172.21.15.28:0/268983875 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9224101f90 0x7f922419d740 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f9218005850 tx=0x7f9218004f80 comp rx=0 tx=0).stop 2024-05-12T21:59:59.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:59.486+0000 7f922be00700 1 -- 172.21.15.28:0/268983875 shutdown_connections 2024-05-12T21:59:59.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:59.486+0000 7f922be00700 1 --2- 172.21.15.28:0/268983875 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f920c039580 0x7f920c03ba40 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:59:59.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:59.486+0000 7f922be00700 1 --2- 172.21.15.28:0/268983875 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9224197e40 0x7f92241982f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:59:59.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:59.486+0000 7f922be00700 1 --2- 172.21.15.28:0/268983875 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9224101f90 0x7f922419d740 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:59:59.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:59.486+0000 7f922be00700 1 --2- 172.21.15.28:0/268983875 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f922419dc80 0x7f9224197900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T21:59:59.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:59.486+0000 7f922be00700 1 -- 172.21.15.28:0/268983875 >> 172.21.15.28:0/268983875 conn(0x7f92240fd860 msgr2=0x7f9224103960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T21:59:59.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:59.486+0000 7f922be00700 1 -- 172.21.15.28:0/268983875 shutdown_connections 2024-05-12T21:59:59.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T21:59:59.486+0000 7f922be00700 1 -- 172.21.15.28:0/268983875 wait complete. 2024-05-12T21:59:59.991 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 21:59:59 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/268983875' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:00:00.180 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 21:59:59 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/268983875' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:00:00.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 21:59:59 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/268983875' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:00:00.696 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-05-12T22:00:00.696 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-05-12T22:00:00.696 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/etc/ceph/ceph.conf 2024-05-12T22:00:00.765 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-05-12T22:00:00.765 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-05-12T22:00:00.891 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-05-12T22:00:00.891 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/etc/ceph/ceph.conf 2024-05-12T22:00:00.914 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:00 smithi086 ceph-mon[34297]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T22:00:00.914 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:00 smithi086 ceph-mon[34297]: overall HEALTH_OK 2024-05-12T22:00:00.914 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:00 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:00:00.914 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:00 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:00:00.914 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:00 smithi086 ceph-mon[34297]: Reconfiguring mon.b (monmap changed)... 2024-05-12T22:00:00.914 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:00 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-12T22:00:00.914 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:00 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-12T22:00:00.914 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:00 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:00:00.914 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:00 smithi086 ceph-mon[34297]: Reconfiguring daemon mon.b on smithi086 2024-05-12T22:00:00.918 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-05-12T22:00:00.918 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-05-12T22:00:00.984 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-05-12T22:00:00.984 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/etc/ceph/ceph.conf 2024-05-12T22:00:01.017 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-05-12T22:00:01.017 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-05-12T22:00:01.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:00 smithi028 ceph-mon[29847]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T22:00:01.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:00 smithi028 ceph-mon[29847]: overall HEALTH_OK 2024-05-12T22:00:01.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:00 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:00:01.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:00 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:00:01.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:00 smithi028 ceph-mon[29847]: Reconfiguring mon.b (monmap changed)... 2024-05-12T22:00:01.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:00 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-12T22:00:01.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:00 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-12T22:00:01.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:00 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:00:01.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:00 smithi028 ceph-mon[29847]: Reconfiguring daemon mon.b on smithi086 2024-05-12T22:00:01.085 INFO:tasks.cephadm:Adding mgr.x on smithi028 2024-05-12T22:00:01.085 INFO:tasks.cephadm:Adding mgr.y on smithi086 2024-05-12T22:00:01.086 INFO:tasks.cephadm:Adding mgr.z on smithi186 2024-05-12T22:00:01.086 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f72fecff68e1d400c4568684327c900485c20d6a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6b92e7b4-10aa-11ef-bc98-c7b262605968 -- ceph orch apply mgr '3;smithi028=x;smithi086=y;smithi186=z' 2024-05-12T22:00:01.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:00 smithi186 ceph-mon[34785]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T22:00:01.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:00 smithi186 ceph-mon[34785]: overall HEALTH_OK 2024-05-12T22:00:01.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:00 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:00:01.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:00 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:00:01.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:00 smithi186 ceph-mon[34785]: Reconfiguring mon.b (monmap changed)... 2024-05-12T22:00:01.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:00 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-12T22:00:01.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:00 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-12T22:00:01.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:00 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:00:01.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:00 smithi186 ceph-mon[34785]: Reconfiguring daemon mon.b on smithi086 2024-05-12T22:00:01.358 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/mon.c/config 2024-05-12T22:00:02.670 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:02 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:00:02.670 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:02 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:00:02.670 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:02 smithi186 ceph-mon[34785]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T22:00:02.670 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:02 smithi186 ceph-mon[34785]: Reconfiguring mon.c (monmap changed)... 2024-05-12T22:00:02.670 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:02 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-12T22:00:02.671 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:02 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-12T22:00:02.671 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:02 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:00:02.671 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:02 smithi186 ceph-mon[34785]: Reconfiguring daemon mon.c on smithi186 2024-05-12T22:00:02.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:02 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:00:02.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:02 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:00:02.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:02 smithi086 ceph-mon[34297]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T22:00:02.993 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:02 smithi086 ceph-mon[34297]: Reconfiguring mon.c (monmap changed)... 2024-05-12T22:00:02.993 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:02 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-12T22:00:02.993 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:02 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-12T22:00:02.993 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:02 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:00:02.993 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:02 smithi086 ceph-mon[34297]: Reconfiguring daemon mon.c on smithi186 2024-05-12T22:00:03.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:02 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:00:03.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:02 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:00:03.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:02 smithi028 ceph-mon[29847]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T22:00:03.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:02 smithi028 ceph-mon[29847]: Reconfiguring mon.c (monmap changed)... 2024-05-12T22:00:03.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:02 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-12T22:00:03.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:02 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-12T22:00:03.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:02 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:00:03.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:02 smithi028 ceph-mon[29847]: Reconfiguring daemon mon.c on smithi186 2024-05-12T22:00:03.816 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:00:03.812+0000 7f04aaeba700 1 -- 172.21.15.186:0/3173858917 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f04a4103fb0 msgr2=0x7f04a410a860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:00:03.817 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:00:03.812+0000 7f04aaeba700 1 --2- 172.21.15.186:0/3173858917 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f04a4103fb0 0x7f04a410a860 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f049c00b0a0 tx=0x7f049c00b3b0 comp rx=0 tx=0).stop 2024-05-12T22:00:03.817 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:00:03.812+0000 7f04aaeba700 1 -- 172.21.15.186:0/3173858917 shutdown_connections 2024-05-12T22:00:03.817 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:00:03.812+0000 7f04aaeba700 1 --2- 172.21.15.186:0/3173858917 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f04a4103fb0 0x7f04a410a860 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:00:03.817 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:00:03.812+0000 7f04aaeba700 1 --2- 172.21.15.186:0/3173858917 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f04a4100630 0x7f04a4100a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:00:03.817 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:00:03.812+0000 7f04aaeba700 1 --2- 172.21.15.186:0/3173858917 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f04a4103610 0x7f04a4103a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:00:03.817 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:00:03.812+0000 7f04aaeba700 1 -- 172.21.15.186:0/3173858917 >> 172.21.15.186:0/3173858917 conn(0x7f04a40fbfb0 msgr2=0x7f04a40fe3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:00:03.817 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:00:03.812+0000 7f04aaeba700 1 -- 172.21.15.186:0/3173858917 shutdown_connections 2024-05-12T22:00:03.817 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:00:03.812+0000 7f04aaeba700 1 -- 172.21.15.186:0/3173858917 wait complete. 2024-05-12T22:00:03.817 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:00:03.813+0000 7f04aaeba700 1 Processor -- start 2024-05-12T22:00:03.818 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:00:03.813+0000 7f04aaeba700 1 -- start start 2024-05-12T22:00:03.818 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:00:03.813+0000 7f04aaeba700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f04a4100630 0x7f04a41985d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:00:03.818 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:00:03.813+0000 7f04aaeba700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f04a4103610 0x7f04a4198b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:00:03.818 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:00:03.813+0000 7f04aaeba700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f04a4103fb0 0x7f04a419fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:00:03.818 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:00:03.813+0000 7f04aaeba700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f04a410dff0 con 0x7f04a4103fb0 2024-05-12T22:00:03.818 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:00:03.813+0000 7f04aaeba700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f04a410de70 con 0x7f04a4100630 2024-05-12T22:00:03.818 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:00:03.813+0000 7f04aaeba700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f04a410e170 con 0x7f04a4103610 2024-05-12T22:00:03.818 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:00:03.813+0000 7f04a3fff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f04a4103610 0x7f04a4198b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:00:03.818 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:00:03.813+0000 7f04a3fff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f04a4103610 0x7f04a4198b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:55520/0 (socket says 172.21.15.186:55520) 2024-05-12T22:00:03.818 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:00:03.813+0000 7f04a3fff700 1 -- 172.21.15.186:0/531039183 learned_addr learned my addr 172.21.15.186:0/531039183 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-05-12T22:00:03.819 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:00:03.813+0000 7f04a8c56700 1 --2- 172.21.15.186:0/531039183 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f04a4100630 0x7f04a41985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:00:03.819 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:00:03.813+0000 7f04a3fff700 1 -- 172.21.15.186:0/531039183 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f04a4100630 msgr2=0x7f04a41985d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:00:03.819 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:00:03.813+0000 7f04a3fff700 1 --2- 172.21.15.186:0/531039183 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f04a4100630 0x7f04a41985d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:00:03.819 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:00:03.813+0000 7f04a3fff700 1 -- 172.21.15.186:0/531039183 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f04a4103fb0 msgr2=0x7f04a419fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:00:03.819 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:00:03.813+0000 7f04a3fff700 1 --2- 172.21.15.186:0/531039183 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f04a4103fb0 0x7f04a419fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:00:03.819 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:00:03.813+0000 7f04a3fff700 1 -- 172.21.15.186:0/531039183 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04a41a0230 con 0x7f04a4103610 2024-05-12T22:00:03.819 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:00:03.814+0000 7f04a3fff700 1 --2- 172.21.15.186:0/531039183 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f04a4103610 0x7f04a4198b10 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f049000c9a0 tx=0x7f049000cd60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:00:03.819 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:00:03.814+0000 7f04a1ffb700 1 -- 172.21.15.186:0/531039183 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0490007a60 con 0x7f04a4103610 2024-05-12T22:00:03.819 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:00:03.814+0000 7f04a1ffb700 1 -- 172.21.15.186:0/531039183 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0490004d10 con 0x7f04a4103610 2024-05-12T22:00:03.819 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:00:03.814+0000 7f04aaeba700 1 -- 172.21.15.186:0/531039183 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04a41a0520 con 0x7f04a4103610 2024-05-12T22:00:03.820 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:00:03.814+0000 7f04a1ffb700 1 -- 172.21.15.186:0/531039183 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f049000f450 con 0x7f04a4103610 2024-05-12T22:00:03.820 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:00:03.814+0000 7f04aaeba700 1 -- 172.21.15.186:0/531039183 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04a41a0a10 con 0x7f04a4103610 2024-05-12T22:00:03.820 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:00:03.815+0000 7f04a1ffb700 1 -- 172.21.15.186:0/531039183 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f049000f5f0 con 0x7f04a4103610 2024-05-12T22:00:03.820 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:00:03.815+0000 7f04a1ffb700 1 --2- 172.21.15.186:0/531039183 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f0494039330 0x7f049403b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:00:03.820 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:00:03.815+0000 7f04a1ffb700 1 -- 172.21.15.186:0/531039183 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f049004c210 con 0x7f04a4103610 2024-05-12T22:00:03.820 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:00:03.815+0000 7f04aaeba700 1 -- 172.21.15.186:0/531039183 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04a402db80 con 0x7f04a4103610 2024-05-12T22:00:03.820 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:00:03.815+0000 7f04a8c56700 1 --2- 172.21.15.186:0/531039183 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f0494039330 0x7f049403b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:00:03.820 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:00:03.817+0000 7f04a8c56700 1 --2- 172.21.15.186:0/531039183 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f0494039330 0x7f049403b7f0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f0498006fd0 tx=0x7f0498008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:00:03.821 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:00:03.820+0000 7f04a1ffb700 1 -- 172.21.15.186:0/531039183 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f04900169d0 con 0x7f04a4103610 2024-05-12T22:00:04.026 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:00:04.026+0000 7f04aaeba700 1 -- 172.21.15.186:0/531039183 --> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi028=x;smithi086=y;smithi186=z", "target": ["mon-mgr", ""]}) -- 0x7f04a4062760 con 0x7f0494039330 2024-05-12T22:00:04.041 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:00:04.040+0000 7f04a1ffb700 1 -- 172.21.15.186:0/531039183 <== mgr.14152 v2:172.21.15.28:6800/3123328651 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f04a4062760 con 0x7f0494039330 2024-05-12T22:00:04.041 INFO:teuthology.orchestra.run.smithi186.stdout:Scheduled mgr update... 2024-05-12T22:00:04.043 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:00:04.043+0000 7f04aaeba700 1 -- 172.21.15.186:0/531039183 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f0494039330 msgr2=0x7f049403b7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:00:04.043 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:00:04.043+0000 7f04aaeba700 1 --2- 172.21.15.186:0/531039183 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f0494039330 0x7f049403b7f0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f0498006fd0 tx=0x7f0498008040 comp rx=0 tx=0).stop 2024-05-12T22:00:04.043 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:00:04.043+0000 7f04aaeba700 1 -- 172.21.15.186:0/531039183 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f04a4103610 msgr2=0x7f04a4198b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:00:04.043 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:00:04.043+0000 7f04aaeba700 1 --2- 172.21.15.186:0/531039183 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f04a4103610 0x7f04a4198b10 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f049000c9a0 tx=0x7f049000cd60 comp rx=0 tx=0).stop 2024-05-12T22:00:04.043 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:00:04.043+0000 7f04aaeba700 1 -- 172.21.15.186:0/531039183 shutdown_connections 2024-05-12T22:00:04.043 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:00:04.043+0000 7f04aaeba700 1 --2- 172.21.15.186:0/531039183 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f0494039330 0x7f049403b7f0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:00:04.043 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:00:04.043+0000 7f04aaeba700 1 --2- 172.21.15.186:0/531039183 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f04a4103fb0 0x7f04a419fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:00:04.043 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:00:04.043+0000 7f04aaeba700 1 --2- 172.21.15.186:0/531039183 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f04a4100630 0x7f04a41985d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:00:04.043 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:00:04.043+0000 7f04aaeba700 1 --2- 172.21.15.186:0/531039183 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f04a4103610 0x7f04a4198b10 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:00:04.043 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:00:04.043+0000 7f04aaeba700 1 -- 172.21.15.186:0/531039183 >> 172.21.15.186:0/531039183 conn(0x7f04a40fbfb0 msgr2=0x7f04a41068b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:00:04.043 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:00:04.043+0000 7f04aaeba700 1 -- 172.21.15.186:0/531039183 shutdown_connections 2024-05-12T22:00:04.043 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:00:04.043+0000 7f04aaeba700 1 -- 172.21.15.186:0/531039183 wait complete. 2024-05-12T22:00:04.657 DEBUG:teuthology.orchestra.run.smithi086:mgr.y> sudo journalctl -f -n 0 -u ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968@mgr.y.service 2024-05-12T22:00:04.660 DEBUG:teuthology.orchestra.run.smithi186:mgr.z> sudo journalctl -f -n 0 -u ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968@mgr.z.service 2024-05-12T22:00:04.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:04 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:00:04.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:04 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:00:04.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:04 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:00:04.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:04 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:00:04.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:04 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:00:04.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:04 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:00:04.681 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:04 smithi186 ceph-mon[34785]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T22:00:04.681 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:04 smithi186 ceph-mon[34785]: from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi028=x;smithi086=y;smithi186=z", "target": ["mon-mgr", ""]}]: dispatch 2024-05-12T22:00:04.681 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:04 smithi186 ceph-mon[34785]: Saving service mgr spec with placement smithi028=x;smithi086=y;smithi186=z;count:3 2024-05-12T22:00:04.681 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:04 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:00:04.681 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:04 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:00:04.681 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:04 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:00:04.682 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:04 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:00:04.682 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:04 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:00:04.682 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:04 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-12T22:00:04.682 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:04 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-05-12T22:00:04.682 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:04 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-12T22:00:04.682 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:04 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:00:04.683 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:04 smithi186 ceph-mon[34785]: Deploying daemon mgr.z on smithi186 2024-05-12T22:00:04.683 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:04 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:00:04.684 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:04 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:00:04.684 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:04 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:00:04.684 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:04 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:00:04.684 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:04 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:00:04.684 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:04 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:00:04.685 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:04 smithi086 ceph-mon[34297]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T22:00:04.685 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:04 smithi086 ceph-mon[34297]: from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi028=x;smithi086=y;smithi186=z", "target": ["mon-mgr", ""]}]: dispatch 2024-05-12T22:00:04.685 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:04 smithi086 ceph-mon[34297]: Saving service mgr spec with placement smithi028=x;smithi086=y;smithi186=z;count:3 2024-05-12T22:00:04.685 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:04 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:00:04.685 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:04 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:00:04.686 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:04 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:00:04.686 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:04 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:00:04.686 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:04 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:00:04.686 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:04 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-12T22:00:04.686 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:04 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-05-12T22:00:04.686 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:04 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-12T22:00:04.687 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:04 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:00:04.687 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:04 smithi086 ceph-mon[34297]: Deploying daemon mgr.z on smithi186 2024-05-12T22:00:04.688 INFO:tasks.cephadm:Deploying OSDs... 2024-05-12T22:00:04.688 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-05-12T22:00:04.688 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/scratch_devs of=/dev/stdout 2024-05-12T22:00:04.750 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-05-12T22:00:04.750 DEBUG:teuthology.orchestra.run.smithi028:> stat /dev/vg_nvme/lv_1 2024-05-12T22:00:04.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:04 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:00:04.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:04 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:00:04.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:04 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:00:04.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:04 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:00:04.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:04 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:00:04.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:04 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:00:04.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:04 smithi028 ceph-mon[29847]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T22:00:04.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:04 smithi028 ceph-mon[29847]: from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi028=x;smithi086=y;smithi186=z", "target": ["mon-mgr", ""]}]: dispatch 2024-05-12T22:00:04.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:04 smithi028 ceph-mon[29847]: Saving service mgr spec with placement smithi028=x;smithi086=y;smithi186=z;count:3 2024-05-12T22:00:04.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:04 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:00:04.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:04 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:00:04.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:04 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:00:04.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:04 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:00:04.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:04 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:00:04.787 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:04 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-12T22:00:04.787 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:04 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-05-12T22:00:04.787 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:04 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-12T22:00:04.787 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:04 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:00:04.787 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:04 smithi028 ceph-mon[29847]: Deploying daemon mgr.z on smithi186 2024-05-12T22:00:04.806 INFO:teuthology.orchestra.run.smithi028.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-05-12T22:00:04.806 INFO:teuthology.orchestra.run.smithi028.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-12T22:00:04.807 INFO:teuthology.orchestra.run.smithi028.stdout:Device: 6h/6d Inode: 632 Links: 1 2024-05-12T22:00:04.807 INFO:teuthology.orchestra.run.smithi028.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-12T22:00:04.807 INFO:teuthology.orchestra.run.smithi028.stdout:Context: system_u:object_r:device_t:s0 2024-05-12T22:00:04.807 INFO:teuthology.orchestra.run.smithi028.stdout:Access: 2024-05-12 21:58:58.463703045 +0000 2024-05-12T22:00:04.807 INFO:teuthology.orchestra.run.smithi028.stdout:Modify: 2024-05-12 21:53:25.509960540 +0000 2024-05-12T22:00:04.807 INFO:teuthology.orchestra.run.smithi028.stdout:Change: 2024-05-12 21:53:25.509960540 +0000 2024-05-12T22:00:04.807 INFO:teuthology.orchestra.run.smithi028.stdout: Birth: 2024-05-12 21:53:25.509960540 +0000 2024-05-12T22:00:04.807 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-05-12T22:00:04.931 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-05-12T22:00:04.931 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-05-12T22:00:04.931 INFO:teuthology.orchestra.run.smithi028.stderr:512 bytes copied, 0.000300346 s, 1.7 MB/s 2024-05-12T22:00:04.935 DEBUG:teuthology.orchestra.run.smithi028:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-05-12T22:00:04.996 DEBUG:teuthology.orchestra.run.smithi028:> stat /dev/vg_nvme/lv_2 2024-05-12T22:00:05.050 INFO:teuthology.orchestra.run.smithi028.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-05-12T22:00:05.050 INFO:teuthology.orchestra.run.smithi028.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-12T22:00:05.050 INFO:teuthology.orchestra.run.smithi028.stdout:Device: 6h/6d Inode: 653 Links: 1 2024-05-12T22:00:05.050 INFO:teuthology.orchestra.run.smithi028.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-12T22:00:05.050 INFO:teuthology.orchestra.run.smithi028.stdout:Context: system_u:object_r:device_t:s0 2024-05-12T22:00:05.051 INFO:teuthology.orchestra.run.smithi028.stdout:Access: 2024-05-12 21:58:58.544700728 +0000 2024-05-12T22:00:05.051 INFO:teuthology.orchestra.run.smithi028.stdout:Modify: 2024-05-12 21:53:26.123942992 +0000 2024-05-12T22:00:05.051 INFO:teuthology.orchestra.run.smithi028.stdout:Change: 2024-05-12 21:53:26.123942992 +0000 2024-05-12T22:00:05.051 INFO:teuthology.orchestra.run.smithi028.stdout: Birth: 2024-05-12 21:53:26.123942992 +0000 2024-05-12T22:00:05.051 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-05-12T22:00:05.137 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-05-12T22:00:05.138 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-05-12T22:00:05.138 INFO:teuthology.orchestra.run.smithi028.stderr:512 bytes copied, 0.000282048 s, 1.8 MB/s 2024-05-12T22:00:05.142 DEBUG:teuthology.orchestra.run.smithi028:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-05-12T22:00:05.203 DEBUG:teuthology.orchestra.run.smithi028:> stat /dev/vg_nvme/lv_3 2024-05-12T22:00:05.255 INFO:teuthology.orchestra.run.smithi028.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-05-12T22:00:05.255 INFO:teuthology.orchestra.run.smithi028.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-12T22:00:05.255 INFO:teuthology.orchestra.run.smithi028.stdout:Device: 6h/6d Inode: 671 Links: 1 2024-05-12T22:00:05.255 INFO:teuthology.orchestra.run.smithi028.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-12T22:00:05.255 INFO:teuthology.orchestra.run.smithi028.stdout:Context: system_u:object_r:device_t:s0 2024-05-12T22:00:05.255 INFO:teuthology.orchestra.run.smithi028.stdout:Access: 2024-05-12 21:58:58.618698610 +0000 2024-05-12T22:00:05.255 INFO:teuthology.orchestra.run.smithi028.stdout:Modify: 2024-05-12 21:53:26.715926072 +0000 2024-05-12T22:00:05.255 INFO:teuthology.orchestra.run.smithi028.stdout:Change: 2024-05-12 21:53:26.715926072 +0000 2024-05-12T22:00:05.255 INFO:teuthology.orchestra.run.smithi028.stdout: Birth: 2024-05-12 21:53:26.715926072 +0000 2024-05-12T22:00:05.255 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-05-12T22:00:05.339 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-05-12T22:00:05.339 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-05-12T22:00:05.339 INFO:teuthology.orchestra.run.smithi028.stderr:512 bytes copied, 0.000282228 s, 1.8 MB/s 2024-05-12T22:00:05.343 DEBUG:teuthology.orchestra.run.smithi028:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-05-12T22:00:05.403 DEBUG:teuthology.orchestra.run.smithi028:> stat /dev/vg_nvme/lv_4 2024-05-12T22:00:05.458 INFO:teuthology.orchestra.run.smithi028.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-05-12T22:00:05.493 INFO:teuthology.orchestra.run.smithi028.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-12T22:00:05.493 INFO:teuthology.orchestra.run.smithi028.stdout:Device: 6h/6d Inode: 689 Links: 1 2024-05-12T22:00:05.493 INFO:teuthology.orchestra.run.smithi028.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-12T22:00:05.493 INFO:teuthology.orchestra.run.smithi028.stdout:Context: system_u:object_r:device_t:s0 2024-05-12T22:00:05.493 INFO:teuthology.orchestra.run.smithi028.stdout:Access: 2024-05-12 21:58:58.694696436 +0000 2024-05-12T22:00:05.493 INFO:teuthology.orchestra.run.smithi028.stdout:Modify: 2024-05-12 21:53:27.334908381 +0000 2024-05-12T22:00:05.493 INFO:teuthology.orchestra.run.smithi028.stdout:Change: 2024-05-12 21:53:27.334908381 +0000 2024-05-12T22:00:05.493 INFO:teuthology.orchestra.run.smithi028.stdout: Birth: 2024-05-12 21:53:27.334908381 +0000 2024-05-12T22:00:05.494 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-05-12T22:00:05.590 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-05-12T22:00:05.591 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-05-12T22:00:05.591 INFO:teuthology.orchestra.run.smithi028.stderr:512 bytes copied, 0.000285935 s, 1.8 MB/s 2024-05-12T22:00:05.593 DEBUG:teuthology.orchestra.run.smithi028:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-05-12T22:00:05.663 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-05-12T22:00:05.663 DEBUG:teuthology.orchestra.run.smithi086:> dd if=/scratch_devs of=/dev/stdout 2024-05-12T22:00:05.682 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-05-12T22:00:05.682 DEBUG:teuthology.orchestra.run.smithi086:> stat /dev/vg_nvme/lv_1 2024-05-12T22:00:05.740 INFO:teuthology.orchestra.run.smithi086.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-05-12T22:00:05.740 INFO:teuthology.orchestra.run.smithi086.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-12T22:00:05.740 INFO:teuthology.orchestra.run.smithi086.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-05-12T22:00:05.740 INFO:teuthology.orchestra.run.smithi086.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-12T22:00:05.740 INFO:teuthology.orchestra.run.smithi086.stdout:Context: system_u:object_r:device_t:s0 2024-05-12T22:00:05.740 INFO:teuthology.orchestra.run.smithi086.stdout:Access: 2024-05-12 21:59:48.519531467 +0000 2024-05-12T22:00:05.740 INFO:teuthology.orchestra.run.smithi086.stdout:Modify: 2024-05-12 21:52:01.975418888 +0000 2024-05-12T22:00:05.740 INFO:teuthology.orchestra.run.smithi086.stdout:Change: 2024-05-12 21:52:01.975418888 +0000 2024-05-12T22:00:05.740 INFO:teuthology.orchestra.run.smithi086.stdout: Birth: 2024-05-12 21:52:01.975418888 +0000 2024-05-12T22:00:05.741 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-05-12T22:00:05.805 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records in 2024-05-12T22:00:05.805 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records out 2024-05-12T22:00:05.805 INFO:teuthology.orchestra.run.smithi086.stderr:512 bytes copied, 0.000228499 s, 2.2 MB/s 2024-05-12T22:00:05.806 DEBUG:teuthology.orchestra.run.smithi086:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-05-12T22:00:05.864 DEBUG:teuthology.orchestra.run.smithi086:> stat /dev/vg_nvme/lv_2 2024-05-12T22:00:05.921 INFO:teuthology.orchestra.run.smithi086.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-05-12T22:00:05.921 INFO:teuthology.orchestra.run.smithi086.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-12T22:00:05.921 INFO:teuthology.orchestra.run.smithi086.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-05-12T22:00:05.921 INFO:teuthology.orchestra.run.smithi086.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-12T22:00:05.921 INFO:teuthology.orchestra.run.smithi086.stdout:Context: system_u:object_r:device_t:s0 2024-05-12T22:00:05.921 INFO:teuthology.orchestra.run.smithi086.stdout:Access: 2024-05-12 21:59:48.519531467 +0000 2024-05-12T22:00:05.922 INFO:teuthology.orchestra.run.smithi086.stdout:Modify: 2024-05-12 21:52:02.320411403 +0000 2024-05-12T22:00:05.922 INFO:teuthology.orchestra.run.smithi086.stdout:Change: 2024-05-12 21:52:02.320411403 +0000 2024-05-12T22:00:05.922 INFO:teuthology.orchestra.run.smithi086.stdout: Birth: 2024-05-12 21:52:02.320411403 +0000 2024-05-12T22:00:05.922 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-05-12T22:00:05.984 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records in 2024-05-12T22:00:05.984 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records out 2024-05-12T22:00:05.984 INFO:teuthology.orchestra.run.smithi086.stderr:512 bytes copied, 0.000209676 s, 2.4 MB/s 2024-05-12T22:00:05.985 DEBUG:teuthology.orchestra.run.smithi086:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-05-12T22:00:06.042 DEBUG:teuthology.orchestra.run.smithi086:> stat /dev/vg_nvme/lv_3 2024-05-12T22:00:06.097 INFO:teuthology.orchestra.run.smithi086.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-05-12T22:00:06.097 INFO:teuthology.orchestra.run.smithi086.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-12T22:00:06.097 INFO:teuthology.orchestra.run.smithi086.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-05-12T22:00:06.097 INFO:teuthology.orchestra.run.smithi086.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-12T22:00:06.097 INFO:teuthology.orchestra.run.smithi086.stdout:Context: system_u:object_r:device_t:s0 2024-05-12T22:00:06.097 INFO:teuthology.orchestra.run.smithi086.stdout:Access: 2024-05-12 21:59:48.519531467 +0000 2024-05-12T22:00:06.097 INFO:teuthology.orchestra.run.smithi086.stdout:Modify: 2024-05-12 21:52:02.651404221 +0000 2024-05-12T22:00:06.097 INFO:teuthology.orchestra.run.smithi086.stdout:Change: 2024-05-12 21:52:02.651404221 +0000 2024-05-12T22:00:06.097 INFO:teuthology.orchestra.run.smithi086.stdout: Birth: 2024-05-12 21:52:02.651404221 +0000 2024-05-12T22:00:06.097 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-05-12T22:00:06.159 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records in 2024-05-12T22:00:06.159 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records out 2024-05-12T22:00:06.159 INFO:teuthology.orchestra.run.smithi086.stderr:512 bytes copied, 0.00019159 s, 2.7 MB/s 2024-05-12T22:00:06.160 DEBUG:teuthology.orchestra.run.smithi086:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-05-12T22:00:06.179 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 12 22:00:05 smithi186 systemd[1]: Starting Ceph mgr.z for 6b92e7b4-10aa-11ef-bc98-c7b262605968... 2024-05-12T22:00:06.217 DEBUG:teuthology.orchestra.run.smithi086:> stat /dev/vg_nvme/lv_4 2024-05-12T22:00:06.273 INFO:teuthology.orchestra.run.smithi086.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-05-12T22:00:06.273 INFO:teuthology.orchestra.run.smithi086.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-12T22:00:06.273 INFO:teuthology.orchestra.run.smithi086.stdout:Device: 5h/5d Inode: 721 Links: 1 2024-05-12T22:00:06.273 INFO:teuthology.orchestra.run.smithi086.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-12T22:00:06.273 INFO:teuthology.orchestra.run.smithi086.stdout:Context: system_u:object_r:device_t:s0 2024-05-12T22:00:06.273 INFO:teuthology.orchestra.run.smithi086.stdout:Access: 2024-05-12 21:59:48.520531446 +0000 2024-05-12T22:00:06.273 INFO:teuthology.orchestra.run.smithi086.stdout:Modify: 2024-05-12 21:52:02.991396844 +0000 2024-05-12T22:00:06.273 INFO:teuthology.orchestra.run.smithi086.stdout:Change: 2024-05-12 21:52:02.991396844 +0000 2024-05-12T22:00:06.273 INFO:teuthology.orchestra.run.smithi086.stdout: Birth: 2024-05-12 21:52:02.991396844 +0000 2024-05-12T22:00:06.273 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-05-12T22:00:06.336 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records in 2024-05-12T22:00:06.336 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records out 2024-05-12T22:00:06.336 INFO:teuthology.orchestra.run.smithi086.stderr:512 bytes copied, 0.000244517 s, 2.1 MB/s 2024-05-12T22:00:06.337 DEBUG:teuthology.orchestra.run.smithi086:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-05-12T22:00:06.393 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-05-12T22:00:06.394 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/scratch_devs of=/dev/stdout 2024-05-12T22:00:06.410 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-05-12T22:00:06.410 DEBUG:teuthology.orchestra.run.smithi186:> stat /dev/vg_nvme/lv_1 2024-05-12T22:00:06.464 INFO:teuthology.orchestra.run.smithi186.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-05-12T22:00:06.464 INFO:teuthology.orchestra.run.smithi186.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-12T22:00:06.464 INFO:teuthology.orchestra.run.smithi186.stdout:Device: 5h/5d Inode: 664 Links: 1 2024-05-12T22:00:06.465 INFO:teuthology.orchestra.run.smithi186.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-12T22:00:06.465 INFO:teuthology.orchestra.run.smithi186.stdout:Context: system_u:object_r:device_t:s0 2024-05-12T22:00:06.465 INFO:teuthology.orchestra.run.smithi186.stdout:Access: 2024-05-12 22:00:05.923095334 +0000 2024-05-12T22:00:06.465 INFO:teuthology.orchestra.run.smithi186.stdout:Modify: 2024-05-12 21:52:03.635179757 +0000 2024-05-12T22:00:06.465 INFO:teuthology.orchestra.run.smithi186.stdout:Change: 2024-05-12 21:52:03.635179757 +0000 2024-05-12T22:00:06.465 INFO:teuthology.orchestra.run.smithi186.stdout: Birth: 2024-05-12 21:52:03.635179757 +0000 2024-05-12T22:00:06.465 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-05-12T22:00:06.475 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 12 22:00:06 smithi186 podman[36444]: 2024-05-12 22:00:06.104478027 +0000 UTC m=+0.018122175 image pull 8252b7944d3b5c188c0ff632647ba6cc4e9a19588ee097b8a3dba27b42779c25 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc 2024-05-12T22:00:06.475 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 12 22:00:06 smithi186 podman[36444]: 2024-05-12 22:00:06.228681474 +0000 UTC m=+0.142325597 container create 893dc82d49b9126afe509635c6e6b3b1352b943e8cf19a8b03126128249e2a54 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-z, org.label-schema.build-date=20240507, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=squid-f72fecf, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e) 2024-05-12T22:00:06.475 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 12 22:00:06 smithi186 podman[36444]: 2024-05-12 22:00:06.44781922 +0000 UTC m=+0.361463347 container init 893dc82d49b9126afe509635c6e6b3b1352b943e8cf19a8b03126128249e2a54 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-z, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=squid-f72fecf, GIT_CLEAN=True, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.build-date=20240507, ceph=True, io.buildah.version=1.33.5) 2024-05-12T22:00:06.475 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 12 22:00:06 smithi186 podman[36444]: 2024-05-12 22:00:06.450561259 +0000 UTC m=+0.364205386 container start 893dc82d49b9126afe509635c6e6b3b1352b943e8cf19a8b03126128249e2a54 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-z, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.build-date=20240507, RELEASE=squid-f72fecf, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-05-12T22:00:06.500 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records in 2024-05-12T22:00:06.500 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records out 2024-05-12T22:00:06.500 INFO:teuthology.orchestra.run.smithi186.stderr:512 bytes copied, 0.00025257 s, 2.0 MB/s 2024-05-12T22:00:06.502 DEBUG:teuthology.orchestra.run.smithi186:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-05-12T22:00:06.557 DEBUG:teuthology.orchestra.run.smithi186:> stat /dev/vg_nvme/lv_2 2024-05-12T22:00:06.613 INFO:teuthology.orchestra.run.smithi186.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-05-12T22:00:06.613 INFO:teuthology.orchestra.run.smithi186.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-12T22:00:06.613 INFO:teuthology.orchestra.run.smithi186.stdout:Device: 5h/5d Inode: 683 Links: 1 2024-05-12T22:00:06.613 INFO:teuthology.orchestra.run.smithi186.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-12T22:00:06.613 INFO:teuthology.orchestra.run.smithi186.stdout:Context: system_u:object_r:device_t:s0 2024-05-12T22:00:06.613 INFO:teuthology.orchestra.run.smithi186.stdout:Access: 2024-05-12 22:00:05.923095334 +0000 2024-05-12T22:00:06.613 INFO:teuthology.orchestra.run.smithi186.stdout:Modify: 2024-05-12 21:52:03.989173254 +0000 2024-05-12T22:00:06.613 INFO:teuthology.orchestra.run.smithi186.stdout:Change: 2024-05-12 21:52:03.989173254 +0000 2024-05-12T22:00:06.613 INFO:teuthology.orchestra.run.smithi186.stdout: Birth: 2024-05-12 21:52:03.989173254 +0000 2024-05-12T22:00:06.614 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-05-12T22:00:06.677 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records in 2024-05-12T22:00:06.677 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records out 2024-05-12T22:00:06.677 INFO:teuthology.orchestra.run.smithi186.stderr:512 bytes copied, 0.000189242 s, 2.7 MB/s 2024-05-12T22:00:06.678 DEBUG:teuthology.orchestra.run.smithi186:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-05-12T22:00:06.734 DEBUG:teuthology.orchestra.run.smithi186:> stat /dev/vg_nvme/lv_3 2024-05-12T22:00:06.789 INFO:teuthology.orchestra.run.smithi186.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-05-12T22:00:06.789 INFO:teuthology.orchestra.run.smithi186.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-12T22:00:06.789 INFO:teuthology.orchestra.run.smithi186.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-05-12T22:00:06.789 INFO:teuthology.orchestra.run.smithi186.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-12T22:00:06.789 INFO:teuthology.orchestra.run.smithi186.stdout:Context: system_u:object_r:device_t:s0 2024-05-12T22:00:06.789 INFO:teuthology.orchestra.run.smithi186.stdout:Access: 2024-05-12 22:00:05.924095316 +0000 2024-05-12T22:00:06.789 INFO:teuthology.orchestra.run.smithi186.stdout:Modify: 2024-05-12 21:52:04.330166991 +0000 2024-05-12T22:00:06.789 INFO:teuthology.orchestra.run.smithi186.stdout:Change: 2024-05-12 21:52:04.330166991 +0000 2024-05-12T22:00:06.789 INFO:teuthology.orchestra.run.smithi186.stdout: Birth: 2024-05-12 21:52:04.330166991 +0000 2024-05-12T22:00:06.789 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-05-12T22:00:06.802 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:06 smithi186 ceph-mon[34785]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T22:00:06.802 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 12 22:00:06 smithi186 ceph-mgr[36480]: -- 172.21.15.186:0/3379040491 <== mon.1 v2:172.21.15.186:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x564cda6165a0 con 0x564cda5eb000 2024-05-12T22:00:06.802 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 12 22:00:06 smithi186 bash[36444]: 893dc82d49b9126afe509635c6e6b3b1352b943e8cf19a8b03126128249e2a54 2024-05-12T22:00:06.802 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 12 22:00:06 smithi186 systemd[1]: Started Ceph mgr.z for 6b92e7b4-10aa-11ef-bc98-c7b262605968. 2024-05-12T22:00:06.802 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 12 22:00:06 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-z[36460]: 2024-05-12T22:00:06.603+0000 7f4a9fdce200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-12T22:00:06.802 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 12 22:00:06 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-z[36460]: 2024-05-12T22:00:06.801+0000 7f4a9fdce200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-12T22:00:06.824 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records in 2024-05-12T22:00:06.824 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records out 2024-05-12T22:00:06.824 INFO:teuthology.orchestra.run.smithi186.stderr:512 bytes copied, 0.00020531 s, 2.5 MB/s 2024-05-12T22:00:06.825 DEBUG:teuthology.orchestra.run.smithi186:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-05-12T22:00:06.881 DEBUG:teuthology.orchestra.run.smithi186:> stat /dev/vg_nvme/lv_4 2024-05-12T22:00:06.936 INFO:teuthology.orchestra.run.smithi186.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-05-12T22:00:06.936 INFO:teuthology.orchestra.run.smithi186.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-12T22:00:06.936 INFO:teuthology.orchestra.run.smithi186.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-05-12T22:00:06.937 INFO:teuthology.orchestra.run.smithi186.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-12T22:00:06.937 INFO:teuthology.orchestra.run.smithi186.stdout:Context: system_u:object_r:device_t:s0 2024-05-12T22:00:06.937 INFO:teuthology.orchestra.run.smithi186.stdout:Access: 2024-05-12 22:00:05.924095316 +0000 2024-05-12T22:00:06.937 INFO:teuthology.orchestra.run.smithi186.stdout:Modify: 2024-05-12 21:52:04.670160745 +0000 2024-05-12T22:00:06.937 INFO:teuthology.orchestra.run.smithi186.stdout:Change: 2024-05-12 21:52:04.670160745 +0000 2024-05-12T22:00:06.937 INFO:teuthology.orchestra.run.smithi186.stdout: Birth: 2024-05-12 21:52:04.670160745 +0000 2024-05-12T22:00:06.937 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-05-12T22:00:06.991 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:06 smithi086 ceph-mon[34297]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T22:00:06.998 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records in 2024-05-12T22:00:06.999 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records out 2024-05-12T22:00:06.999 INFO:teuthology.orchestra.run.smithi186.stderr:512 bytes copied, 0.000196381 s, 2.6 MB/s 2024-05-12T22:00:07.000 DEBUG:teuthology.orchestra.run.smithi186:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-05-12T22:00:07.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:06 smithi028 ceph-mon[29847]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T22:00:07.056 INFO:tasks.cephadm:Deploying osd.0 on smithi028 with /dev/vg_nvme/lv_4... 2024-05-12T22:00:07.057 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f72fecff68e1d400c4568684327c900485c20d6a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6b92e7b4-10aa-11ef-bc98-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-05-12T22:00:07.179 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 12 22:00:06 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-z[36460]: 2024-05-12T22:00:06.909+0000 7f4a9fdce200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-12T22:00:07.509 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/mon.a/config 2024-05-12T22:00:07.674 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 12 22:00:07 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-z[36460]: 2024-05-12T22:00:07.228+0000 7f4a9fdce200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-12T22:00:07.674 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 12 22:00:07 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-z[36460]: 2024-05-12T22:00:07.312+0000 7f4a9fdce200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-12T22:00:07.914 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:07 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:00:07.914 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:07 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:00:07.915 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:07 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:00:07.915 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:07 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-12T22:00:07.915 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:07 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-05-12T22:00:07.915 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:07 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-12T22:00:07.915 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:07 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:00:07.915 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:07 smithi028 ceph-mon[29847]: Deploying daemon mgr.y on smithi086 2024-05-12T22:00:07.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:07 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:00:07.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:07 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:00:07.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:07 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:00:07.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:07 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-12T22:00:07.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:07 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-05-12T22:00:07.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:07 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-12T22:00:07.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:07 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:00:07.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:07 smithi186 ceph-mon[34785]: Deploying daemon mgr.y on smithi086 2024-05-12T22:00:07.933 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:07 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:00:07.933 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:07 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:00:07.933 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:07 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:00:07.934 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:07 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-12T22:00:07.934 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:07 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-05-12T22:00:07.934 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:07 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-12T22:00:07.934 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:07 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:00:07.934 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:07 smithi086 ceph-mon[34297]: Deploying daemon mgr.y on smithi086 2024-05-12T22:00:08.851 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:08 smithi186 ceph-mon[34785]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T22:00:08.851 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 12 22:00:08 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-z[36460]: 2024-05-12T22:00:08.514+0000 7f4a9fdce200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-12T22:00:08.851 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 12 22:00:08 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-z[36460]: 2024-05-12T22:00:08.682+0000 7f4a9fdce200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-12T22:00:08.851 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 12 22:00:08 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-z[36460]: 2024-05-12T22:00:08.851+0000 7f4a9fdce200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-12T22:00:08.936 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:08 smithi086 ceph-mon[34297]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T22:00:09.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:08 smithi028 ceph-mon[29847]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T22:00:09.242 INFO:journalctl@ceph.mgr.y.smithi086.stdout:May 12 22:00:09 smithi086 podman[35708]: 2024-05-12 22:00:09.053855981 +0000 UTC m=+0.017539543 image pull 8252b7944d3b5c188c0ff632647ba6cc4e9a19588ee097b8a3dba27b42779c25 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc 2024-05-12T22:00:09.425 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 12 22:00:09 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-z[36460]: 2024-05-12T22:00:09.132+0000 7f4a9fdce200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-12T22:00:09.425 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 12 22:00:09 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-z[36460]: 2024-05-12T22:00:09.222+0000 7f4a9fdce200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-12T22:00:09.425 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 12 22:00:09 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-z[36460]: 2024-05-12T22:00:09.425+0000 7f4a9fdce200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-12T22:00:09.504 INFO:journalctl@ceph.mgr.y.smithi086.stdout:May 12 22:00:09 smithi086 podman[35708]: 2024-05-12 22:00:09.349320137 +0000 UTC m=+0.313003690 container init 529d155079c6b0e621975eb64a9629b69079d706d81014410167dec2b7046e37 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-y, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.license=GPLv2, RELEASE=squid-f72fecf, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240507) 2024-05-12T22:00:09.505 INFO:journalctl@ceph.mgr.y.smithi086.stdout:May 12 22:00:09 smithi086 podman[35708]: 2024-05-12 22:00:09.351821051 +0000 UTC m=+0.315504608 container start 529d155079c6b0e621975eb64a9629b69079d706d81014410167dec2b7046e37 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-y, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240507, RELEASE=squid-f72fecf, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-05-12T22:00:09.505 INFO:journalctl@ceph.mgr.y.smithi086.stdout:May 12 22:00:09 smithi086 ceph-mgr[35727]: -- 172.21.15.86:0/3716420673 <== mon.2 v2:172.21.15.86:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5612520485a0 con 0x56125201cc00 2024-05-12T22:00:09.505 INFO:journalctl@ceph.mgr.y.smithi086.stdout:May 12 22:00:09 smithi086 bash[35708]: 529d155079c6b0e621975eb64a9629b69079d706d81014410167dec2b7046e37 2024-05-12T22:00:09.505 INFO:journalctl@ceph.mgr.y.smithi086.stdout:May 12 22:00:09 smithi086 systemd[1]: Started Ceph mgr.y for 6b92e7b4-10aa-11ef-bc98-c7b262605968. 2024-05-12T22:00:09.505 INFO:journalctl@ceph.mgr.y.smithi086.stdout:May 12 22:00:09 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-y[35723]: 2024-05-12T22:00:09.504+0000 7f8a18874200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-12T22:00:09.679 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 12 22:00:09 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-z[36460]: 2024-05-12T22:00:09.615+0000 7f4a9fdce200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-12T22:00:09.793 INFO:journalctl@ceph.mgr.y.smithi086.stdout:May 12 22:00:09 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-y[35723]: 2024-05-12T22:00:09.690+0000 7f8a18874200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-12T22:00:09.794 INFO:journalctl@ceph.mgr.y.smithi086.stdout:May 12 22:00:09 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-y[35723]: 2024-05-12T22:00:09.793+0000 7f8a18874200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-12T22:00:10.345 INFO:teuthology.orchestra.run.smithi028.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-05-12T22:00:10.345 INFO:teuthology.orchestra.run.smithi028.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-05-12T22:00:10.345 INFO:teuthology.orchestra.run.smithi028.stderr: stderr: 10+0 records in 2024-05-12T22:00:10.345 INFO:teuthology.orchestra.run.smithi028.stderr:10+0 records out 2024-05-12T22:00:10.345 INFO:teuthology.orchestra.run.smithi028.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0359412 s, 292 MB/s 2024-05-12T22:00:10.345 INFO:teuthology.orchestra.run.smithi028.stderr:--> Zapping successful for: 2024-05-12T22:00:10.492 INFO:journalctl@ceph.mgr.y.smithi086.stdout:May 12 22:00:10 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-y[35723]: 2024-05-12T22:00:10.118+0000 7f8a18874200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-12T22:00:10.492 INFO:journalctl@ceph.mgr.y.smithi086.stdout:May 12 22:00:10 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-y[35723]: 2024-05-12T22:00:10.207+0000 7f8a18874200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-12T22:00:10.496 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 12 22:00:10 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-z[36460]: 2024-05-12T22:00:10.232+0000 7f4a9fdce200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-12T22:00:10.750 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:10 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:00:10.750 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:10 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:00:10.750 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:10 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:00:10.750 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:10 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:00:10.750 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:10 smithi028 ceph-mon[29847]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T22:00:10.750 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:10 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:00:10.751 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:10 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:00:10.751 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:10 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:00:10.751 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:10 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:00:10.752 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:10 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:00:10.752 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:10 smithi086 ceph-mon[34297]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T22:00:10.752 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:10 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:00:10.779 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 12 22:00:10 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-z[36460]: 2024-05-12T22:00:10.679+0000 7f4a9fdce200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-12T22:00:10.780 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:10 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:00:10.780 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:10 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:00:10.780 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:10 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:00:10.780 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:10 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:00:10.780 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:10 smithi186 ceph-mon[34785]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T22:00:10.780 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:10 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:00:11.179 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 12 22:00:10 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-z[36460]: 2024-05-12T22:00:10.782+0000 7f4a9fdce200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-12T22:00:11.179 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 12 22:00:10 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-z[36460]: 2024-05-12T22:00:10.981+0000 7f4a9fdce200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-12T22:00:11.202 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f72fecff68e1d400c4568684327c900485c20d6a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6b92e7b4-10aa-11ef-bc98-c7b262605968 -- ceph orch daemon add osd smithi028:vg_nvme/lv_4 2024-05-12T22:00:11.644 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/mon.a/config 2024-05-12T22:00:11.742 INFO:journalctl@ceph.mgr.y.smithi086.stdout:May 12 22:00:11 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-y[35723]: 2024-05-12T22:00:11.451+0000 7f8a18874200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-12T22:00:11.742 INFO:journalctl@ceph.mgr.y.smithi086.stdout:May 12 22:00:11 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-y[35723]: 2024-05-12T22:00:11.618+0000 7f8a18874200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-12T22:00:11.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:11 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:00:11.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:11 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:00:11.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:11 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:00:11.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:11 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:00:11.743 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:11 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:00:11.743 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:11 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:00:11.743 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:11 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:00:11.743 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:11 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:00:11.743 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:11 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-12T22:00:11.743 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:11 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-12T22:00:11.743 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:11 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:00:11.841 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 12 22:00:11 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-z[36460]: 2024-05-12T22:00:11.515+0000 7f4a9fdce200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-12T22:00:11.841 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 12 22:00:11 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-z[36460]: 2024-05-12T22:00:11.603+0000 7f4a9fdce200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-12T22:00:11.842 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:11 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:00:11.842 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:11 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:00:11.842 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:11 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:00:11.842 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:11 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:00:11.842 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:11 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:00:11.842 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:11 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:00:11.842 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:11 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:00:11.842 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:11 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:00:11.842 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:11 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-12T22:00:11.842 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:11 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-12T22:00:11.843 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:11 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:00:11.919 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:11 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:00:11.919 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:11 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:00:11.919 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:11 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:00:11.919 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:11 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:00:11.919 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:11 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:00:11.919 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:11 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:00:11.919 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:11 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:00:11.920 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:11 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:00:11.920 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:11 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-12T22:00:11.920 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:11 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-12T22:00:11.920 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:11 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:00:12.074 INFO:journalctl@ceph.mgr.y.smithi086.stdout:May 12 22:00:11 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-y[35723]: 2024-05-12T22:00:11.787+0000 7f8a18874200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-12T22:00:12.074 INFO:journalctl@ceph.mgr.y.smithi086.stdout:May 12 22:00:12 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-y[35723]: 2024-05-12T22:00:12.074+0000 7f8a18874200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-12T22:00:12.178 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 12 22:00:11 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-z[36460]: 2024-05-12T22:00:11.840+0000 7f4a9fdce200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-12T22:00:12.179 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 12 22:00:11 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-z[36460]: 2024-05-12T22:00:11.970+0000 7f4a9fdce200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-12T22:00:12.179 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 12 22:00:12 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-z[36460]: 2024-05-12T22:00:12.064+0000 7f4a9fdce200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-12T22:00:12.179 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 12 22:00:12 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-z[36460]: 2024-05-12T22:00:12.177+0000 7f4a9fdce200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-12T22:00:12.366 INFO:journalctl@ceph.mgr.y.smithi086.stdout:May 12 22:00:12 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-y[35723]: 2024-05-12T22:00:12.165+0000 7f8a18874200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-12T22:00:12.367 INFO:journalctl@ceph.mgr.y.smithi086.stdout:May 12 22:00:12 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-y[35723]: 2024-05-12T22:00:12.366+0000 7f8a18874200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-12T22:00:12.742 INFO:journalctl@ceph.mgr.y.smithi086.stdout:May 12 22:00:12 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-y[35723]: 2024-05-12T22:00:12.556+0000 7f8a18874200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-12T22:00:12.929 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 12 22:00:12 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-z[36460]: 2024-05-12T22:00:12.478+0000 7f4a9fdce200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-12T22:00:13.331 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:13 smithi028 ceph-mon[29847]: Reconfiguring mgr.x (unknown last config time)... 2024-05-12T22:00:13.331 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:13 smithi028 ceph-mon[29847]: Reconfiguring daemon mgr.x on smithi028 2024-05-12T22:00:13.331 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:13 smithi028 ceph-mon[29847]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T22:00:13.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:13 smithi186 ceph-mon[34785]: Reconfiguring mgr.x (unknown last config time)... 2024-05-12T22:00:13.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:13 smithi186 ceph-mon[34785]: Reconfiguring daemon mgr.x on smithi028 2024-05-12T22:00:13.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:13 smithi186 ceph-mon[34785]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T22:00:13.430 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 12 22:00:13 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-z[36460]: 2024-05-12T22:00:13.067+0000 7f4a9fdce200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-12T22:00:13.430 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 12 22:00:13 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-z[36460]: 2024-05-12T22:00:13.279+0000 7f4a9fdce200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-12T22:00:13.492 INFO:journalctl@ceph.mgr.y.smithi086.stdout:May 12 22:00:13 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-y[35723]: 2024-05-12T22:00:13.150+0000 7f8a18874200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-12T22:00:13.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:13 smithi086 ceph-mon[34297]: Reconfiguring mgr.x (unknown last config time)... 2024-05-12T22:00:13.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:13 smithi086 ceph-mon[34297]: Reconfiguring daemon mgr.x on smithi028 2024-05-12T22:00:13.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:13 smithi086 ceph-mon[34297]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T22:00:13.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:00:13.652+0000 7fe04d600700 1 -- 172.21.15.28:0/2387478246 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe0481089c0 msgr2=0x7fe048108de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:00:13.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:00:13.652+0000 7fe04d600700 1 --2- 172.21.15.28:0/2387478246 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe0481089c0 0x7fe048108de0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fe0380099c0 tx=0x7fe038009cd0 comp rx=0 tx=0).stop 2024-05-12T22:00:13.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:00:13.652+0000 7fe04d600700 1 -- 172.21.15.28:0/2387478246 shutdown_connections 2024-05-12T22:00:13.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:00:13.652+0000 7fe04d600700 1 --2- 172.21.15.28:0/2387478246 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe04810a560 0x7fe048110e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:00:13.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:00:13.652+0000 7fe04d600700 1 --2- 172.21.15.28:0/2387478246 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe0481089c0 0x7fe048108de0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:00:13.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:00:13.652+0000 7fe04d600700 1 --2- 172.21.15.28:0/2387478246 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe048109bc0 0x7fe04810a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:00:13.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:00:13.652+0000 7fe04d600700 1 -- 172.21.15.28:0/2387478246 >> 172.21.15.28:0/2387478246 conn(0x7fe048077990 msgr2=0x7fe048079dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:00:13.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:00:13.652+0000 7fe04d600700 1 -- 172.21.15.28:0/2387478246 shutdown_connections 2024-05-12T22:00:13.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:00:13.652+0000 7fe04d600700 1 -- 172.21.15.28:0/2387478246 wait complete. 2024-05-12T22:00:13.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:00:13.654+0000 7fe04d600700 1 Processor -- start 2024-05-12T22:00:13.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:00:13.654+0000 7fe04d600700 1 -- start start 2024-05-12T22:00:13.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:00:13.654+0000 7fe04d600700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0481089c0 0x7fe04819ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:00:13.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:00:13.654+0000 7fe04d600700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe048109bc0 0x7fe04819f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:00:13.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:00:13.654+0000 7fe04d600700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe04810a560 0x7fe0481a61a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:00:13.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:00:13.655+0000 7fe04d600700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe0481145f0 con 0x7fe0481089c0 2024-05-12T22:00:13.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:00:13.655+0000 7fe04d600700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fe048114470 con 0x7fe048109bc0 2024-05-12T22:00:13.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:00:13.655+0000 7fe04d600700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe048114770 con 0x7fe04810a560 2024-05-12T22:00:13.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:00:13.655+0000 7fe046a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0481089c0 0x7fe04819ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:00:13.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:00:13.655+0000 7fe03e000700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe048109bc0 0x7fe04819f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:00:13.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:00:13.655+0000 7fe03e000700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe048109bc0 0x7fe04819f120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:44096/0 (socket says 172.21.15.28:44096) 2024-05-12T22:00:13.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:00:13.655+0000 7fe03e000700 1 -- 172.21.15.28:0/2242681655 learned_addr learned my addr 172.21.15.28:0/2242681655 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:00:13.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:00:13.655+0000 7fe047400700 1 --2- 172.21.15.28:0/2242681655 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe04810a560 0x7fe0481a61a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:00:13.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:00:13.655+0000 7fe03e000700 1 -- 172.21.15.28:0/2242681655 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe04810a560 msgr2=0x7fe0481a61a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:00:13.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:00:13.655+0000 7fe03e000700 1 --2- 172.21.15.28:0/2242681655 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe04810a560 0x7fe0481a61a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:00:13.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:00:13.655+0000 7fe03e000700 1 -- 172.21.15.28:0/2242681655 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0481089c0 msgr2=0x7fe04819ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:00:13.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:00:13.655+0000 7fe03e000700 1 --2- 172.21.15.28:0/2242681655 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0481089c0 0x7fe04819ebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:00:13.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:00:13.655+0000 7fe03e000700 1 -- 172.21.15.28:0/2242681655 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0481a6840 con 0x7fe048109bc0 2024-05-12T22:00:13.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:00:13.655+0000 7fe047400700 1 --2- 172.21.15.28:0/2242681655 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe04810a560 0x7fe0481a61a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:00:13.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:00:13.656+0000 7fe03e000700 1 --2- 172.21.15.28:0/2242681655 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe048109bc0 0x7fe04819f120 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fe03000e9c0 tx=0x7fe03000ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:00:13.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:00:13.656+0000 7fe03fe00700 1 -- 172.21.15.28:0/2242681655 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe03000cbd0 con 0x7fe048109bc0 2024-05-12T22:00:13.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:00:13.656+0000 7fe03fe00700 1 -- 172.21.15.28:0/2242681655 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe03000cd70 con 0x7fe048109bc0 2024-05-12T22:00:13.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:00:13.656+0000 7fe04d600700 1 -- 172.21.15.28:0/2242681655 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0481a6b30 con 0x7fe048109bc0 2024-05-12T22:00:13.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:00:13.656+0000 7fe03fe00700 1 -- 172.21.15.28:0/2242681655 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe030010430 con 0x7fe048109bc0 2024-05-12T22:00:13.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:00:13.656+0000 7fe04d600700 1 -- 172.21.15.28:0/2242681655 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0481a7070 con 0x7fe048109bc0 2024-05-12T22:00:13.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:00:13.657+0000 7fe03fe00700 1 -- 172.21.15.28:0/2242681655 <== mon.2 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7fe0300105d0 con 0x7fe048109bc0 2024-05-12T22:00:13.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:00:13.657+0000 7fe04d600700 1 -- 172.21.15.28:0/2242681655 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe048069010 con 0x7fe048109bc0 2024-05-12T22:00:13.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:00:13.658+0000 7fe03fe00700 1 --2- 172.21.15.28:0/2242681655 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fe02c039330 0x7fe02c03b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:00:13.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:00:13.658+0000 7fe03fe00700 1 -- 172.21.15.28:0/2242681655 <== mon.2 v2:172.21.15.86:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fe03004c640 con 0x7fe048109bc0 2024-05-12T22:00:13.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:00:13.658+0000 7fe046a00700 1 --2- 172.21.15.28:0/2242681655 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fe02c039330 0x7fe02c03b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:00:13.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:00:13.659+0000 7fe046a00700 1 --2- 172.21.15.28:0/2242681655 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fe02c039330 0x7fe02c03b7f0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fe03800b580 tx=0x7fe038005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:00:13.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:00:13.663+0000 7fe03fe00700 1 -- 172.21.15.28:0/2242681655 <== mon.2 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fe03000a250 con 0x7fe048109bc0 2024-05-12T22:00:13.863 INFO:journalctl@ceph.mgr.y.smithi086.stdout:May 12 22:00:13 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-y[35723]: 2024-05-12T22:00:13.568+0000 7f8a18874200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-12T22:00:13.863 INFO:journalctl@ceph.mgr.y.smithi086.stdout:May 12 22:00:13 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-y[35723]: 2024-05-12T22:00:13.664+0000 7f8a18874200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-12T22:00:13.863 INFO:journalctl@ceph.mgr.y.smithi086.stdout:May 12 22:00:13 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-y[35723]: 2024-05-12T22:00:13.861+0000 7f8a18874200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-12T22:00:13.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:00:13.912+0000 7fe04d600700 1 -- 172.21.15.28:0/2242681655 --> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7fe048062760 con 0x7fe02c039330 2024-05-12T22:00:14.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:00:14.258+0000 7fe03fe00700 1 -- 172.21.15.28:0/2242681655 <== mon.2 v2:172.21.15.86:3300/0 7 ==== mgrmap(e 14) ==== 92060+0+0 (secure 0 0 0) 0x7fe030004da0 con 0x7fe048109bc0 2024-05-12T22:00:14.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:14 smithi028 ceph-mon[29847]: from='mgr.? 172.21.15.186:0/3226425156' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-05-12T22:00:14.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:14 smithi028 ceph-mon[29847]: from='mgr.? 172.21.15.186:0/3226425156' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-12T22:00:14.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:14 smithi028 ceph-mon[29847]: Standby manager daemon z started 2024-05-12T22:00:14.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:14 smithi028 ceph-mon[29847]: from='mgr.? 172.21.15.186:0/3226425156' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-05-12T22:00:14.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:14 smithi028 ceph-mon[29847]: from='mgr.? 172.21.15.186:0/3226425156' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-12T22:00:14.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:14 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-12T22:00:14.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:14 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-12T22:00:14.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:14 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:00:14.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:14 smithi186 ceph-mon[34785]: from='mgr.? 172.21.15.186:0/3226425156' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-05-12T22:00:14.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:14 smithi186 ceph-mon[34785]: from='mgr.? 172.21.15.186:0/3226425156' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-12T22:00:14.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:14 smithi186 ceph-mon[34785]: Standby manager daemon z started 2024-05-12T22:00:14.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:14 smithi186 ceph-mon[34785]: from='mgr.? 172.21.15.186:0/3226425156' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-05-12T22:00:14.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:14 smithi186 ceph-mon[34785]: from='mgr.? 172.21.15.186:0/3226425156' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-12T22:00:14.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:14 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-12T22:00:14.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:14 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-12T22:00:14.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:14 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:00:14.711 INFO:journalctl@ceph.mgr.y.smithi086.stdout:May 12 22:00:14 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-y[35723]: 2024-05-12T22:00:14.373+0000 7f8a18874200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-12T22:00:14.711 INFO:journalctl@ceph.mgr.y.smithi086.stdout:May 12 22:00:14 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-y[35723]: 2024-05-12T22:00:14.466+0000 7f8a18874200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-12T22:00:14.711 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:14 smithi086 ceph-mon[34297]: from='mgr.? 172.21.15.186:0/3226425156' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-05-12T22:00:14.712 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:14 smithi086 ceph-mon[34297]: from='mgr.? 172.21.15.186:0/3226425156' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-12T22:00:14.712 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:14 smithi086 ceph-mon[34297]: Standby manager daemon z started 2024-05-12T22:00:14.712 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:14 smithi086 ceph-mon[34297]: from='mgr.? 172.21.15.186:0/3226425156' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-05-12T22:00:14.712 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:14 smithi086 ceph-mon[34297]: from='mgr.? 172.21.15.186:0/3226425156' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-12T22:00:14.712 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:14 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-12T22:00:14.712 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:14 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-12T22:00:14.712 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:14 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:00:14.992 INFO:journalctl@ceph.mgr.y.smithi086.stdout:May 12 22:00:14 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-y[35723]: 2024-05-12T22:00:14.710+0000 7f8a18874200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-12T22:00:14.992 INFO:journalctl@ceph.mgr.y.smithi086.stdout:May 12 22:00:14 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-y[35723]: 2024-05-12T22:00:14.842+0000 7f8a18874200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-12T22:00:14.992 INFO:journalctl@ceph.mgr.y.smithi086.stdout:May 12 22:00:14 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-y[35723]: 2024-05-12T22:00:14.935+0000 7f8a18874200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-12T22:00:15.346 INFO:journalctl@ceph.mgr.y.smithi086.stdout:May 12 22:00:15 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-y[35723]: 2024-05-12T22:00:15.049+0000 7f8a18874200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-12T22:00:15.346 INFO:journalctl@ceph.mgr.y.smithi086.stdout:May 12 22:00:15 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-y[35723]: 2024-05-12T22:00:15.345+0000 7f8a18874200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-12T22:00:15.347 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:15 smithi086 ceph-mon[34297]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T22:00:15.347 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:15 smithi086 ceph-mon[34297]: from='client.24122 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-12T22:00:15.347 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:15 smithi086 ceph-mon[34297]: mgrmap e14: x(active, since 88s), standbys: z 2024-05-12T22:00:15.347 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:15 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-05-12T22:00:15.347 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:15 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:00:15.347 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:15 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:00:15.347 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:15 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:00:15.347 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:15 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:00:15.347 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:15 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:00:15.347 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:15 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:00:15.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:15 smithi028 ceph-mon[29847]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T22:00:15.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:15 smithi028 ceph-mon[29847]: from='client.24122 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-12T22:00:15.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:15 smithi028 ceph-mon[29847]: mgrmap e14: x(active, since 88s), standbys: z 2024-05-12T22:00:15.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:15 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-05-12T22:00:15.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:15 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:00:15.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:15 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:00:15.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:15 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:00:15.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:15 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:00:15.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:15 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:00:15.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:15 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:00:15.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:15 smithi186 ceph-mon[34785]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T22:00:15.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:15 smithi186 ceph-mon[34785]: from='client.24122 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-12T22:00:15.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:15 smithi186 ceph-mon[34785]: mgrmap e14: x(active, since 88s), standbys: z 2024-05-12T22:00:15.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:15 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-05-12T22:00:15.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:15 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:00:15.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:15 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:00:15.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:15 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:00:15.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:15 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:00:15.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:15 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:00:15.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:15 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:00:16.242 INFO:journalctl@ceph.mgr.y.smithi086.stdout:May 12 22:00:15 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-y[35723]: 2024-05-12T22:00:15.941+0000 7f8a18874200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-12T22:00:16.242 INFO:journalctl@ceph.mgr.y.smithi086.stdout:May 12 22:00:16 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-y[35723]: 2024-05-12T22:00:16.156+0000 7f8a18874200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-12T22:00:16.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:00:16.271+0000 7fe03fe00700 1 -- 172.21.15.28:0/2242681655 <== mon.2 v2:172.21.15.86:3300/0 8 ==== mgrmap(e 15) ==== 137777+0+0 (secure 0 0 0) 0x7fe03000fcc0 con 0x7fe048109bc0 2024-05-12T22:00:16.516 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:16 smithi028 ceph-mon[29847]: from='mgr.? 172.21.15.86:0/979193946' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-05-12T22:00:16.516 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:16 smithi028 ceph-mon[29847]: Standby manager daemon y started 2024-05-12T22:00:16.516 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:16 smithi028 ceph-mon[29847]: from='mgr.? 172.21.15.86:0/979193946' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-12T22:00:16.516 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:16 smithi028 ceph-mon[29847]: from='mgr.? 172.21.15.86:0/979193946' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-05-12T22:00:16.516 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:16 smithi028 ceph-mon[29847]: from='mgr.? 172.21.15.86:0/979193946' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-12T22:00:16.592 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:16 smithi086 ceph-mon[34297]: from='mgr.? 172.21.15.86:0/979193946' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-05-12T22:00:16.593 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:16 smithi086 ceph-mon[34297]: Standby manager daemon y started 2024-05-12T22:00:16.593 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:16 smithi086 ceph-mon[34297]: from='mgr.? 172.21.15.86:0/979193946' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-12T22:00:16.593 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:16 smithi086 ceph-mon[34297]: from='mgr.? 172.21.15.86:0/979193946' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-05-12T22:00:16.593 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:16 smithi086 ceph-mon[34297]: from='mgr.? 172.21.15.86:0/979193946' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-12T22:00:16.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:16 smithi186 ceph-mon[34785]: from='mgr.? 172.21.15.86:0/979193946' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-05-12T22:00:16.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:16 smithi186 ceph-mon[34785]: Standby manager daemon y started 2024-05-12T22:00:16.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:16 smithi186 ceph-mon[34785]: from='mgr.? 172.21.15.86:0/979193946' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-12T22:00:16.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:16 smithi186 ceph-mon[34785]: from='mgr.? 172.21.15.86:0/979193946' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-05-12T22:00:16.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:16 smithi186 ceph-mon[34785]: from='mgr.? 172.21.15.86:0/979193946' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-12T22:00:17.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:17 smithi186 ceph-mon[34785]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T22:00:17.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:17 smithi186 ceph-mon[34785]: mgrmap e15: x(active, since 90s), standbys: z, y 2024-05-12T22:00:17.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:17 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-12T22:00:17.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:17 smithi086 ceph-mon[34297]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T22:00:17.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:17 smithi086 ceph-mon[34297]: mgrmap e15: x(active, since 90s), standbys: z, y 2024-05-12T22:00:17.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:17 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-12T22:00:17.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:17 smithi028 ceph-mon[29847]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T22:00:17.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:17 smithi028 ceph-mon[29847]: mgrmap e15: x(active, since 90s), standbys: z, y 2024-05-12T22:00:17.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:17 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-12T22:00:18.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:18 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/2348928810' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6992ce0a-844b-40e3-8ed1-181c8474dccf"}]: dispatch 2024-05-12T22:00:18.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:18 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/2348928810' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6992ce0a-844b-40e3-8ed1-181c8474dccf"}]': finished 2024-05-12T22:00:18.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:18 smithi028 ceph-mon[29847]: osdmap e5: 1 total, 0 up, 1 in 2024-05-12T22:00:18.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:18 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-12T22:00:18.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:18 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/2348928810' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6992ce0a-844b-40e3-8ed1-181c8474dccf"}]: dispatch 2024-05-12T22:00:18.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:18 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/2348928810' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6992ce0a-844b-40e3-8ed1-181c8474dccf"}]': finished 2024-05-12T22:00:18.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:18 smithi186 ceph-mon[34785]: osdmap e5: 1 total, 0 up, 1 in 2024-05-12T22:00:18.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:18 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-12T22:00:18.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:18 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/2348928810' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6992ce0a-844b-40e3-8ed1-181c8474dccf"}]: dispatch 2024-05-12T22:00:18.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:18 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/2348928810' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6992ce0a-844b-40e3-8ed1-181c8474dccf"}]': finished 2024-05-12T22:00:18.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:18 smithi086 ceph-mon[34297]: osdmap e5: 1 total, 0 up, 1 in 2024-05-12T22:00:18.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:18 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-12T22:00:19.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:19 smithi028 ceph-mon[29847]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T22:00:19.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:19 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/3278751295' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-12T22:00:19.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:19 smithi186 ceph-mon[34785]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T22:00:19.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:19 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/3278751295' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-12T22:00:19.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:19 smithi086 ceph-mon[34297]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T22:00:19.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:19 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/3278751295' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-12T22:00:21.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:21 smithi028 ceph-mon[29847]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T22:00:21.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:21 smithi186 ceph-mon[34785]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T22:00:21.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:21 smithi086 ceph-mon[34297]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T22:00:23.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:23 smithi186 ceph-mon[34785]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T22:00:23.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:23 smithi086 ceph-mon[34297]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T22:00:23.784 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:23 smithi028 ceph-mon[29847]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T22:00:25.553 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:25 smithi028 ceph-mon[29847]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T22:00:25.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:25 smithi186 ceph-mon[34785]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T22:00:25.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:25 smithi086 ceph-mon[34297]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T22:00:27.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:27 smithi186 ceph-mon[34785]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T22:00:27.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:27 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-05-12T22:00:27.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:27 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:00:27.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:27 smithi086 ceph-mon[34297]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T22:00:27.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:27 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-05-12T22:00:27.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:27 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:00:27.784 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:27 smithi028 ceph-mon[29847]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T22:00:27.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:27 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-05-12T22:00:27.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:27 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:00:28.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:28 smithi028 ceph-mon[29847]: Deploying daemon osd.0 on smithi028 2024-05-12T22:00:28.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:28 smithi186 ceph-mon[34785]: Deploying daemon osd.0 on smithi028 2024-05-12T22:00:28.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:28 smithi086 ceph-mon[34297]: Deploying daemon osd.0 on smithi028 2024-05-12T22:00:29.567 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:29 smithi028 ceph-mon[29847]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T22:00:29.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:29 smithi186 ceph-mon[34785]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T22:00:29.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:29 smithi086 ceph-mon[34297]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T22:00:31.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:31 smithi186 ceph-mon[34785]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T22:00:31.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:31 smithi086 ceph-mon[34297]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T22:00:31.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:31 smithi028 ceph-mon[29847]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T22:00:33.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:33 smithi186 ceph-mon[34785]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T22:00:33.736 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:33 smithi028 ceph-mon[29847]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T22:00:33.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:33 smithi086 ceph-mon[34297]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T22:00:35.534 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:35 smithi028 ceph-mon[29847]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T22:00:35.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:35 smithi186 ceph-mon[34785]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T22:00:35.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:35 smithi086 ceph-mon[34297]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T22:00:36.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:36 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:00:36.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:36 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:00:36.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:36 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:00:36.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:36 smithi028 ceph-mon[29847]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T22:00:36.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:36 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:00:36.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:36 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:00:36.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:36 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:00:36.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:36 smithi086 ceph-mon[34297]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T22:00:36.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:36 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:00:36.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:36 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:00:36.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:36 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:00:36.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:36 smithi186 ceph-mon[34785]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T22:00:38.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:38 smithi186 ceph-mon[34785]: pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T22:00:38.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:38 smithi086 ceph-mon[34297]: pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T22:00:39.034 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:38 smithi028 ceph-mon[29847]: pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T22:00:39.996 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:39 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:00:39.996 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:39 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:00:39.996 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:39 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:00:39.996 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:39 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:00:39.996 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:39 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:00:39.996 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:39 smithi086 ceph-mon[34297]: from='osd.0 [v2:172.21.15.28:6802/906003882,v1:172.21.15.28:6803/906003882]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-05-12T22:00:39.996 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:39 smithi086 ceph-mon[34297]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-05-12T22:00:40.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:39 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:00:40.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:39 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:00:40.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:39 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:00:40.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:39 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:00:40.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:39 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:00:40.180 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:39 smithi186 ceph-mon[34785]: from='osd.0 [v2:172.21.15.28:6802/906003882,v1:172.21.15.28:6803/906003882]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-05-12T22:00:40.180 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:39 smithi186 ceph-mon[34785]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-05-12T22:00:40.237 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:39 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:00:40.237 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:39 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:00:40.238 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:39 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:00:40.238 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:39 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:00:40.238 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:39 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:00:40.238 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:39 smithi028 ceph-mon[29847]: from='osd.0 [v2:172.21.15.28:6802/906003882,v1:172.21.15.28:6803/906003882]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-05-12T22:00:40.238 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:39 smithi028 ceph-mon[29847]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-05-12T22:00:40.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:00:40.361+0000 7fe03fe00700 1 -- 172.21.15.28:0/2242681655 <== mgr.14152 v2:172.21.15.28:6800/3123328651 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fe048062760 con 0x7fe02c039330 2024-05-12T22:00:40.363 INFO:teuthology.orchestra.run.smithi028.stdout:Created osd(s) 0 on host 'smithi028' 2024-05-12T22:00:40.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:00:40.365+0000 7fe04d600700 1 -- 172.21.15.28:0/2242681655 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fe02c039330 msgr2=0x7fe02c03b7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:00:40.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:00:40.365+0000 7fe04d600700 1 --2- 172.21.15.28:0/2242681655 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fe02c039330 0x7fe02c03b7f0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fe03800b580 tx=0x7fe038005fb0 comp rx=0 tx=0).stop 2024-05-12T22:00:40.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:00:40.365+0000 7fe04d600700 1 -- 172.21.15.28:0/2242681655 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe048109bc0 msgr2=0x7fe04819f120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:00:40.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:00:40.365+0000 7fe04d600700 1 --2- 172.21.15.28:0/2242681655 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe048109bc0 0x7fe04819f120 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fe03000e9c0 tx=0x7fe03000ecd0 comp rx=0 tx=0).stop 2024-05-12T22:00:40.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:00:40.365+0000 7fe04d600700 1 -- 172.21.15.28:0/2242681655 shutdown_connections 2024-05-12T22:00:40.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:00:40.365+0000 7fe04d600700 1 --2- 172.21.15.28:0/2242681655 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fe02c039330 0x7fe02c03b7f0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:00:40.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:00:40.365+0000 7fe04d600700 1 --2- 172.21.15.28:0/2242681655 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe04810a560 0x7fe0481a61a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:00:40.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:00:40.365+0000 7fe04d600700 1 --2- 172.21.15.28:0/2242681655 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0481089c0 0x7fe04819ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:00:40.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:00:40.365+0000 7fe04d600700 1 --2- 172.21.15.28:0/2242681655 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe048109bc0 0x7fe04819f120 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:00:40.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:00:40.365+0000 7fe04d600700 1 -- 172.21.15.28:0/2242681655 >> 172.21.15.28:0/2242681655 conn(0x7fe048077990 msgr2=0x7fe04810cec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:00:40.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:00:40.366+0000 7fe04d600700 1 -- 172.21.15.28:0/2242681655 shutdown_connections 2024-05-12T22:00:40.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:00:40.366+0000 7fe04d600700 1 -- 172.21.15.28:0/2242681655 wait complete. 2024-05-12T22:00:41.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:40 smithi028 ceph-mon[29847]: pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T22:00:41.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:40 smithi028 ceph-mon[29847]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-05-12T22:00:41.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:40 smithi028 ceph-mon[29847]: osdmap e6: 1 total, 0 up, 1 in 2024-05-12T22:00:41.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:40 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-12T22:00:41.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:40 smithi028 ceph-mon[29847]: from='osd.0 [v2:172.21.15.28:6802/906003882,v1:172.21.15.28:6803/906003882]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-05-12T22:00:41.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:40 smithi028 ceph-mon[29847]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-05-12T22:00:41.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:40 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:00:41.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:40 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:00:41.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:40 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:00:41.132 DEBUG:teuthology.orchestra.run.smithi028:osd.0> sudo journalctl -f -n 0 -u ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968@osd.0.service 2024-05-12T22:00:41.136 INFO:tasks.cephadm:Deploying osd.1 on smithi086 with /dev/vg_nvme/lv_4... 2024-05-12T22:00:41.136 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f72fecff68e1d400c4568684327c900485c20d6a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6b92e7b4-10aa-11ef-bc98-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-05-12T22:00:41.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:40 smithi086 ceph-mon[34297]: pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T22:00:41.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:40 smithi086 ceph-mon[34297]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-05-12T22:00:41.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:40 smithi086 ceph-mon[34297]: osdmap e6: 1 total, 0 up, 1 in 2024-05-12T22:00:41.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:40 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-12T22:00:41.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:40 smithi086 ceph-mon[34297]: from='osd.0 [v2:172.21.15.28:6802/906003882,v1:172.21.15.28:6803/906003882]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-05-12T22:00:41.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:40 smithi086 ceph-mon[34297]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-05-12T22:00:41.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:40 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:00:41.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:40 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:00:41.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:40 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:00:41.379 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/mon.b/config 2024-05-12T22:00:41.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:40 smithi186 ceph-mon[34785]: pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T22:00:41.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:40 smithi186 ceph-mon[34785]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-05-12T22:00:41.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:40 smithi186 ceph-mon[34785]: osdmap e6: 1 total, 0 up, 1 in 2024-05-12T22:00:41.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:40 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-12T22:00:41.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:40 smithi186 ceph-mon[34785]: from='osd.0 [v2:172.21.15.28:6802/906003882,v1:172.21.15.28:6803/906003882]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-05-12T22:00:41.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:40 smithi186 ceph-mon[34785]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-05-12T22:00:41.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:40 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:00:41.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:40 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:00:41.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:40 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:00:42.030 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:41 smithi086 ceph-mon[34297]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-05-12T22:00:42.030 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:41 smithi086 ceph-mon[34297]: osdmap e7: 1 total, 0 up, 1 in 2024-05-12T22:00:42.030 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:41 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-12T22:00:42.030 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:41 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-12T22:00:42.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:41 smithi028 ceph-mon[29847]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-05-12T22:00:42.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:41 smithi028 ceph-mon[29847]: osdmap e7: 1 total, 0 up, 1 in 2024-05-12T22:00:42.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:41 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-12T22:00:42.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:41 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-12T22:00:42.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:41 smithi186 ceph-mon[34785]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-05-12T22:00:42.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:41 smithi186 ceph-mon[34785]: osdmap e7: 1 total, 0 up, 1 in 2024-05-12T22:00:42.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:41 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-12T22:00:42.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:41 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-12T22:00:42.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:42 smithi086 ceph-mon[34297]: purged_snaps scrub starts 2024-05-12T22:00:42.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:42 smithi086 ceph-mon[34297]: purged_snaps scrub ok 2024-05-12T22:00:42.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:42 smithi086 ceph-mon[34297]: pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T22:00:42.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:42 smithi086 ceph-mon[34297]: osd.0 [v2:172.21.15.28:6802/906003882,v1:172.21.15.28:6803/906003882] boot 2024-05-12T22:00:42.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:42 smithi086 ceph-mon[34297]: osdmap e8: 1 total, 1 up, 1 in 2024-05-12T22:00:42.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:42 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-12T22:00:43.086 INFO:teuthology.orchestra.run.smithi086.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-05-12T22:00:43.086 INFO:teuthology.orchestra.run.smithi086.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-05-12T22:00:43.086 INFO:teuthology.orchestra.run.smithi086.stderr: stderr: 10+0 records in 2024-05-12T22:00:43.086 INFO:teuthology.orchestra.run.smithi086.stderr:10+0 records out 2024-05-12T22:00:43.086 INFO:teuthology.orchestra.run.smithi086.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112947 s, 928 MB/s 2024-05-12T22:00:43.086 INFO:teuthology.orchestra.run.smithi086.stderr:--> Zapping successful for: 2024-05-12T22:00:43.235 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:42 smithi028 ceph-mon[29847]: purged_snaps scrub starts 2024-05-12T22:00:43.236 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:42 smithi028 ceph-mon[29847]: purged_snaps scrub ok 2024-05-12T22:00:43.236 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:42 smithi028 ceph-mon[29847]: pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T22:00:43.236 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:42 smithi028 ceph-mon[29847]: osd.0 [v2:172.21.15.28:6802/906003882,v1:172.21.15.28:6803/906003882] boot 2024-05-12T22:00:43.236 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:42 smithi028 ceph-mon[29847]: osdmap e8: 1 total, 1 up, 1 in 2024-05-12T22:00:43.236 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:42 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-12T22:00:43.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:42 smithi186 ceph-mon[34785]: purged_snaps scrub starts 2024-05-12T22:00:43.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:42 smithi186 ceph-mon[34785]: purged_snaps scrub ok 2024-05-12T22:00:43.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:42 smithi186 ceph-mon[34785]: pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-12T22:00:43.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:42 smithi186 ceph-mon[34785]: osd.0 [v2:172.21.15.28:6802/906003882,v1:172.21.15.28:6803/906003882] boot 2024-05-12T22:00:43.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:42 smithi186 ceph-mon[34785]: osdmap e8: 1 total, 1 up, 1 in 2024-05-12T22:00:43.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:42 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-12T22:00:43.698 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f72fecff68e1d400c4568684327c900485c20d6a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6b92e7b4-10aa-11ef-bc98-c7b262605968 -- ceph orch daemon add osd smithi086:vg_nvme/lv_4 2024-05-12T22:00:43.932 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/mon.b/config 2024-05-12T22:00:44.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:43 smithi086 ceph-mon[34297]: osdmap e9: 1 total, 1 up, 1 in 2024-05-12T22:00:44.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:43 smithi028 ceph-mon[29847]: osdmap e9: 1 total, 1 up, 1 in 2024-05-12T22:00:44.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:43 smithi186 ceph-mon[34785]: osdmap e9: 1 total, 1 up, 1 in 2024-05-12T22:00:45.219 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:44 smithi086 ceph-mon[34297]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-12T22:00:45.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:00:45.220+0000 7f483acdd700 1 -- 172.21.15.86:0/3796574362 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4834100650 msgr2=0x7f4834100a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:00:45.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:00:45.220+0000 7f483acdd700 1 --2- 172.21.15.86:0/3796574362 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4834100650 0x7f4834100a70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f4828009990 tx=0x7f4828009ca0 comp rx=0 tx=0).stop 2024-05-12T22:00:45.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:00:45.220+0000 7f483acdd700 1 -- 172.21.15.86:0/3796574362 shutdown_connections 2024-05-12T22:00:45.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:00:45.220+0000 7f483acdd700 1 --2- 172.21.15.86:0/3796574362 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4834103fd0 0x7f483410a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:00:45.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:00:45.220+0000 7f483acdd700 1 --2- 172.21.15.86:0/3796574362 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4834100650 0x7f4834100a70 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:00:45.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:00:45.220+0000 7f483acdd700 1 --2- 172.21.15.86:0/3796574362 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4834103630 0x7f4834103a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:00:45.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:00:45.221+0000 7f483acdd700 1 -- 172.21.15.86:0/3796574362 >> 172.21.15.86:0/3796574362 conn(0x7f48340fbf90 msgr2=0x7f48340fe3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:00:45.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:00:45.221+0000 7f483acdd700 1 -- 172.21.15.86:0/3796574362 shutdown_connections 2024-05-12T22:00:45.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:00:45.221+0000 7f483acdd700 1 -- 172.21.15.86:0/3796574362 wait complete. 2024-05-12T22:00:45.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:00:45.221+0000 7f483acdd700 1 Processor -- start 2024-05-12T22:00:45.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:00:45.221+0000 7f483acdd700 1 -- start start 2024-05-12T22:00:45.224 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:00:45.221+0000 7f483acdd700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4834100650 0x7f4834102f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:00:45.224 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:00:45.221+0000 7f483acdd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4834103630 0x7f48341015e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:00:45.224 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:00:45.221+0000 7f483acdd700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4834103fd0 0x7f4834101b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:00:45.224 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:00:45.221+0000 7f483acdd700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f483410dfc0 con 0x7f4834100650 2024-05-12T22:00:45.224 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:00:45.221+0000 7f483acdd700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f483410de40 con 0x7f4834103630 2024-05-12T22:00:45.224 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:00:45.221+0000 7f483acdd700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f483410e140 con 0x7f4834103fd0 2024-05-12T22:00:45.224 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:00:45.221+0000 7f4833fff700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4834103630 0x7f48341015e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:00:45.224 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:00:45.221+0000 7f4833fff700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4834103630 0x7f48341015e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:46288/0 (socket says 172.21.15.86:46288) 2024-05-12T22:00:45.224 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:00:45.221+0000 7f4833fff700 1 -- 172.21.15.86:0/4180565609 learned_addr learned my addr 172.21.15.86:0/4180565609 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-05-12T22:00:45.224 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:00:45.222+0000 7f4833fff700 1 -- 172.21.15.86:0/4180565609 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4834103fd0 msgr2=0x7f4834101b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:00:45.224 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:00:45.222+0000 7f483927a700 1 --2- 172.21.15.86:0/4180565609 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4834103fd0 0x7f4834101b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:00:45.224 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:00:45.222+0000 7f4833fff700 1 --2- 172.21.15.86:0/4180565609 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4834103fd0 0x7f4834101b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:00:45.225 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:00:45.222+0000 7f4833fff700 1 -- 172.21.15.86:0/4180565609 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4834100650 msgr2=0x7f4834102f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:00:45.225 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:00:45.222+0000 7f4833fff700 1 --2- 172.21.15.86:0/4180565609 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4834100650 0x7f4834102f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:00:45.225 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:00:45.222+0000 7f4833fff700 1 -- 172.21.15.86:0/4180565609 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48341023d0 con 0x7f4834103630 2024-05-12T22:00:45.225 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:00:45.222+0000 7f483927a700 1 --2- 172.21.15.86:0/4180565609 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4834103fd0 0x7f4834101b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-12T22:00:45.225 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:00:45.222+0000 7f4833fff700 1 --2- 172.21.15.86:0/4180565609 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4834103630 0x7f48341015e0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f482000d980 tx=0x7f482000dd40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:00:45.225 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:00:45.222+0000 7f4831ffb700 1 -- 172.21.15.86:0/4180565609 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4820016070 con 0x7f4834103630 2024-05-12T22:00:45.225 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:00:45.222+0000 7f4831ffb700 1 -- 172.21.15.86:0/4180565609 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4820004500 con 0x7f4834103630 2024-05-12T22:00:45.225 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:00:45.222+0000 7f483acdd700 1 -- 172.21.15.86:0/4180565609 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48341a2790 con 0x7f4834103630 2024-05-12T22:00:45.225 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:00:45.222+0000 7f4831ffb700 1 -- 172.21.15.86:0/4180565609 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4820004d00 con 0x7f4834103630 2024-05-12T22:00:45.225 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:00:45.222+0000 7f483acdd700 1 -- 172.21.15.86:0/4180565609 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48341a2cd0 con 0x7f4834103630 2024-05-12T22:00:45.225 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:00:45.223+0000 7f483acdd700 1 -- 172.21.15.86:0/4180565609 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4834069010 con 0x7f4834103630 2024-05-12T22:00:45.226 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:00:45.224+0000 7f4831ffb700 1 -- 172.21.15.86:0/4180565609 <== mon.2 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 15) ==== 137777+0+0 (secure 0 0 0) 0x7f482000bbf0 con 0x7f4834103630 2024-05-12T22:00:45.226 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:00:45.225+0000 7f4831ffb700 1 --2- 172.21.15.86:0/4180565609 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f48240a2c10 0x7f48240a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:00:45.226 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:00:45.225+0000 7f4831ffb700 1 -- 172.21.15.86:0/4180565609 <== mon.2 v2:172.21.15.86:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1767+0+0 (secure 0 0 0) 0x7f48200cd3d0 con 0x7f4834103630 2024-05-12T22:00:45.226 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:00:45.225+0000 7f4838a79700 1 --2- 172.21.15.86:0/4180565609 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f48240a2c10 0x7f48240a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:00:45.226 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:00:45.227+0000 7f4838a79700 1 --2- 172.21.15.86:0/4180565609 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f48240a2c10 0x7f48240a50d0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f48280098e0 tx=0x7f4828005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:00:45.229 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:00:45.229+0000 7f4831ffb700 1 -- 172.21.15.86:0/4180565609 <== mon.2 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f482001f020 con 0x7f4834103630 2024-05-12T22:00:45.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:44 smithi028 ceph-mon[29847]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-12T22:00:45.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:44 smithi186 ceph-mon[34785]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-12T22:00:45.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:00:45.429+0000 7f483acdd700 1 -- 172.21.15.86:0/4180565609 --> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f4834062760 con 0x7f48240a2c10 2024-05-12T22:00:46.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:45 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-12T22:00:46.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:45 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-12T22:00:46.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:45 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:00:46.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:45 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:00:46.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:45 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:00:46.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:45 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:00:46.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:45 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:00:46.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:45 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:00:46.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:45 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:00:46.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:45 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-12T22:00:46.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:45 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-12T22:00:46.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:45 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:00:46.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:45 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:00:46.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:45 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:00:46.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:45 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:00:46.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:45 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:00:46.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:45 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:00:46.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:45 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:00:46.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:45 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-12T22:00:46.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:45 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-12T22:00:46.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:45 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:00:46.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:45 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:00:46.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:45 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:00:46.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:45 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:00:46.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:45 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:00:46.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:45 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:00:46.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:45 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:00:47.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:46 smithi086 ceph-mon[34297]: from='client.24149 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-12T22:00:47.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:46 smithi086 ceph-mon[34297]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-12T22:00:47.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:46 smithi086 ceph-mon[34297]: Detected new or changed devices on smithi028 2024-05-12T22:00:47.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:46 smithi086 ceph-mon[34297]: Adjusting osd_memory_target on smithi028 to 17094M 2024-05-12T22:00:47.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:46 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:00:47.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:46 smithi028 ceph-mon[29847]: from='client.24149 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-12T22:00:47.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:46 smithi028 ceph-mon[29847]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-12T22:00:47.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:46 smithi028 ceph-mon[29847]: Detected new or changed devices on smithi028 2024-05-12T22:00:47.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:46 smithi028 ceph-mon[29847]: Adjusting osd_memory_target on smithi028 to 17094M 2024-05-12T22:00:47.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:46 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:00:47.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:46 smithi186 ceph-mon[34785]: from='client.24149 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-12T22:00:47.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:46 smithi186 ceph-mon[34785]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-12T22:00:47.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:46 smithi186 ceph-mon[34785]: Detected new or changed devices on smithi028 2024-05-12T22:00:47.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:46 smithi186 ceph-mon[34785]: Adjusting osd_memory_target on smithi028 to 17094M 2024-05-12T22:00:47.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:46 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:00:48.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:47 smithi028 ceph-mon[29847]: from='client.? 172.21.15.86:0/3422041172' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ced20c96-c481-4d48-90d1-6b43228959ce"}]: dispatch 2024-05-12T22:00:48.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:47 smithi028 ceph-mon[29847]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ced20c96-c481-4d48-90d1-6b43228959ce"}]: dispatch 2024-05-12T22:00:48.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:47 smithi028 ceph-mon[29847]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ced20c96-c481-4d48-90d1-6b43228959ce"}]': finished 2024-05-12T22:00:48.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:47 smithi028 ceph-mon[29847]: osdmap e10: 2 total, 1 up, 2 in 2024-05-12T22:00:48.314 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:47 smithi086 ceph-mon[34297]: from='client.? 172.21.15.86:0/3422041172' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ced20c96-c481-4d48-90d1-6b43228959ce"}]: dispatch 2024-05-12T22:00:48.314 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:47 smithi086 ceph-mon[34297]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ced20c96-c481-4d48-90d1-6b43228959ce"}]: dispatch 2024-05-12T22:00:48.314 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:47 smithi086 ceph-mon[34297]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ced20c96-c481-4d48-90d1-6b43228959ce"}]': finished 2024-05-12T22:00:48.314 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:47 smithi086 ceph-mon[34297]: osdmap e10: 2 total, 1 up, 2 in 2024-05-12T22:00:48.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:47 smithi186 ceph-mon[34785]: from='client.? 172.21.15.86:0/3422041172' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ced20c96-c481-4d48-90d1-6b43228959ce"}]: dispatch 2024-05-12T22:00:48.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:47 smithi186 ceph-mon[34785]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ced20c96-c481-4d48-90d1-6b43228959ce"}]: dispatch 2024-05-12T22:00:48.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:47 smithi186 ceph-mon[34785]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ced20c96-c481-4d48-90d1-6b43228959ce"}]': finished 2024-05-12T22:00:48.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:47 smithi186 ceph-mon[34785]: osdmap e10: 2 total, 1 up, 2 in 2024-05-12T22:00:49.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:49 smithi028 ceph-mon[29847]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-12T22:00:49.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:49 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-12T22:00:49.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:49 smithi028 ceph-mon[29847]: from='client.? 172.21.15.86:0/1574043931' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-12T22:00:49.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:49 smithi186 ceph-mon[34785]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-12T22:00:49.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:49 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-12T22:00:49.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:49 smithi186 ceph-mon[34785]: from='client.? 172.21.15.86:0/1574043931' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-12T22:00:49.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:49 smithi086 ceph-mon[34297]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-12T22:00:49.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:49 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-12T22:00:49.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:49 smithi086 ceph-mon[34297]: from='client.? 172.21.15.86:0/1574043931' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-12T22:00:51.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:51 smithi028 ceph-mon[29847]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-12T22:00:51.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:51 smithi186 ceph-mon[34785]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-12T22:00:51.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:51 smithi086 ceph-mon[34297]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-12T22:00:53.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:53 smithi028 ceph-mon[29847]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-12T22:00:53.343 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:53 smithi086 ceph-mon[34297]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-12T22:00:53.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:53 smithi186 ceph-mon[34785]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-12T22:00:55.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:55 smithi028 ceph-mon[29847]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-12T22:00:55.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:55 smithi186 ceph-mon[34785]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-12T22:00:55.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:55 smithi086 ceph-mon[34297]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-12T22:00:56.172 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:56 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-05-12T22:00:56.172 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:56 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:00:56.172 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:56 smithi086 ceph-mon[34297]: Deploying daemon osd.1 on smithi086 2024-05-12T22:00:56.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:56 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-05-12T22:00:56.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:56 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:00:56.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:56 smithi028 ceph-mon[29847]: Deploying daemon osd.1 on smithi086 2024-05-12T22:00:56.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:56 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-05-12T22:00:56.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:56 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:00:56.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:56 smithi186 ceph-mon[34785]: Deploying daemon osd.1 on smithi086 2024-05-12T22:00:57.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:57 smithi086 ceph-mon[34297]: pgmap v65: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-12T22:00:57.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:57 smithi028 ceph-mon[29847]: pgmap v65: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-12T22:00:57.330 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:57 smithi186 ceph-mon[34785]: pgmap v65: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-12T22:00:59.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:00:59 smithi086 ceph-mon[34297]: pgmap v66: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-12T22:00:59.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:00:59 smithi028 ceph-mon[29847]: pgmap v66: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-12T22:00:59.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:00:59 smithi186 ceph-mon[34785]: pgmap v66: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-12T22:01:01.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:01 smithi186 ceph-mon[34785]: pgmap v67: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-12T22:01:01.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:01 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:01:01.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:01 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:01:01.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:01 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:01:01.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:01 smithi086 ceph-mon[34297]: pgmap v67: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-12T22:01:01.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:01 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:01:01.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:01 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:01:01.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:01 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:01:01.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:01 smithi028 ceph-mon[29847]: pgmap v67: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-12T22:01:01.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:01 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:01:01.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:01 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:01:01.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:01 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:01:03.614 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:03 smithi086 ceph-mon[34297]: pgmap v68: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-12T22:01:03.668 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:01:03.667+0000 7f4831ffb700 1 -- 172.21.15.86:0/4180565609 <== mgr.14152 v2:172.21.15.28:6800/3123328651 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f4834062760 con 0x7f48240a2c10 2024-05-12T22:01:03.668 INFO:teuthology.orchestra.run.smithi086.stdout:Created osd(s) 1 on host 'smithi086' 2024-05-12T22:01:03.671 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:01:03.670+0000 7f483acdd700 1 -- 172.21.15.86:0/4180565609 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f48240a2c10 msgr2=0x7f48240a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:01:03.671 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:01:03.670+0000 7f483acdd700 1 --2- 172.21.15.86:0/4180565609 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f48240a2c10 0x7f48240a50d0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f48280098e0 tx=0x7f4828005fb0 comp rx=0 tx=0).stop 2024-05-12T22:01:03.671 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:01:03.670+0000 7f483acdd700 1 -- 172.21.15.86:0/4180565609 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4834103630 msgr2=0x7f48341015e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:01:03.671 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:01:03.670+0000 7f483acdd700 1 --2- 172.21.15.86:0/4180565609 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4834103630 0x7f48341015e0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f482000d980 tx=0x7f482000dd40 comp rx=0 tx=0).stop 2024-05-12T22:01:03.671 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:01:03.670+0000 7f483acdd700 1 -- 172.21.15.86:0/4180565609 shutdown_connections 2024-05-12T22:01:03.671 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:01:03.670+0000 7f483acdd700 1 --2- 172.21.15.86:0/4180565609 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f48240a2c10 0x7f48240a50d0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:01:03.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:01:03.670+0000 7f483acdd700 1 --2- 172.21.15.86:0/4180565609 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4834103fd0 0x7f4834101b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:01:03.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:01:03.670+0000 7f483acdd700 1 --2- 172.21.15.86:0/4180565609 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4834100650 0x7f4834102f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:01:03.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:01:03.670+0000 7f483acdd700 1 --2- 172.21.15.86:0/4180565609 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4834103630 0x7f48341015e0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:01:03.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:01:03.670+0000 7f483acdd700 1 -- 172.21.15.86:0/4180565609 >> 172.21.15.86:0/4180565609 conn(0x7f48340fbf90 msgr2=0x7f4834106a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:01:03.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:01:03.670+0000 7f483acdd700 1 -- 172.21.15.86:0/4180565609 shutdown_connections 2024-05-12T22:01:03.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:01:03.670+0000 7f483acdd700 1 -- 172.21.15.86:0/4180565609 wait complete. 2024-05-12T22:01:03.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:03 smithi186 ceph-mon[34785]: pgmap v68: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-12T22:01:03.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:03 smithi028 ceph-mon[29847]: pgmap v68: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-12T22:01:04.262 DEBUG:teuthology.orchestra.run.smithi086:osd.1> sudo journalctl -f -n 0 -u ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968@osd.1.service 2024-05-12T22:01:04.264 INFO:tasks.cephadm:Deploying osd.2 on smithi186 with /dev/vg_nvme/lv_4... 2024-05-12T22:01:04.264 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f72fecff68e1d400c4568684327c900485c20d6a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6b92e7b4-10aa-11ef-bc98-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-05-12T22:01:04.492 INFO:journalctl@ceph.osd.1.smithi086.stdout:May 12 22:01:04 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-1[38925]: 2024-05-12T22:01:04.302+0000 7efc70e2b7c0 -1 osd.1 0 log_to_monitors true 2024-05-12T22:01:04.514 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/mon.c/config 2024-05-12T22:01:04.926 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:04 smithi186 ceph-mon[34785]: pgmap v69: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-12T22:01:04.926 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:04 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:01:04.926 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:04 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:01:04.926 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:04 smithi186 ceph-mon[34785]: from='osd.1 [v2:172.21.15.86:6800/3354141819,v1:172.21.15.86:6801/3354141819]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-12T22:01:04.926 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:04 smithi186 ceph-mon[34785]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-12T22:01:04.926 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:04 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:01:04.926 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:04 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:01:04.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:04 smithi086 ceph-mon[34297]: pgmap v69: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-12T22:01:04.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:04 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:01:04.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:04 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:01:04.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:04 smithi086 ceph-mon[34297]: from='osd.1 [v2:172.21.15.86:6800/3354141819,v1:172.21.15.86:6801/3354141819]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-12T22:01:04.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:04 smithi086 ceph-mon[34297]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-12T22:01:04.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:04 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:01:04.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:04 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:01:05.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:04 smithi028 ceph-mon[29847]: pgmap v69: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-12T22:01:05.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:04 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:01:05.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:04 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:01:05.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:04 smithi028 ceph-mon[29847]: from='osd.1 [v2:172.21.15.86:6800/3354141819,v1:172.21.15.86:6801/3354141819]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-12T22:01:05.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:04 smithi028 ceph-mon[29847]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-12T22:01:05.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:04 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:01:05.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:04 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:01:05.780 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:05 smithi186 ceph-mon[34785]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-05-12T22:01:05.780 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:05 smithi186 ceph-mon[34785]: from='osd.1 [v2:172.21.15.86:6800/3354141819,v1:172.21.15.86:6801/3354141819]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-05-12T22:01:05.780 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:05 smithi186 ceph-mon[34785]: osdmap e11: 2 total, 1 up, 2 in 2024-05-12T22:01:05.780 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:05 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-12T22:01:05.780 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:05 smithi186 ceph-mon[34785]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-05-12T22:01:05.973 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:05 smithi086 ceph-mon[34297]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-05-12T22:01:05.973 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:05 smithi086 ceph-mon[34297]: from='osd.1 [v2:172.21.15.86:6800/3354141819,v1:172.21.15.86:6801/3354141819]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-05-12T22:01:05.973 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:05 smithi086 ceph-mon[34297]: osdmap e11: 2 total, 1 up, 2 in 2024-05-12T22:01:05.973 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:05 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-12T22:01:05.973 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:05 smithi086 ceph-mon[34297]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-05-12T22:01:06.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:05 smithi028 ceph-mon[29847]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-05-12T22:01:06.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:05 smithi028 ceph-mon[29847]: from='osd.1 [v2:172.21.15.86:6800/3354141819,v1:172.21.15.86:6801/3354141819]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-05-12T22:01:06.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:05 smithi028 ceph-mon[29847]: osdmap e11: 2 total, 1 up, 2 in 2024-05-12T22:01:06.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:05 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-12T22:01:06.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:05 smithi028 ceph-mon[29847]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-05-12T22:01:06.340 INFO:teuthology.orchestra.run.smithi186.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-05-12T22:01:06.340 INFO:teuthology.orchestra.run.smithi186.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-05-12T22:01:06.340 INFO:teuthology.orchestra.run.smithi186.stderr: stderr: 10+0 records in 2024-05-12T22:01:06.340 INFO:teuthology.orchestra.run.smithi186.stderr:10+0 records out 2024-05-12T22:01:06.340 INFO:teuthology.orchestra.run.smithi186.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114333 s, 917 MB/s 2024-05-12T22:01:06.340 INFO:teuthology.orchestra.run.smithi186.stderr:--> Zapping successful for: 2024-05-12T22:01:06.881 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f72fecff68e1d400c4568684327c900485c20d6a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6b92e7b4-10aa-11ef-bc98-c7b262605968 -- ceph orch daemon add osd smithi186:vg_nvme/lv_4 2024-05-12T22:01:06.904 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:06 smithi186 ceph-mon[34785]: pgmap v71: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-12T22:01:06.905 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:06 smithi186 ceph-mon[34785]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]': finished 2024-05-12T22:01:06.905 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:06 smithi186 ceph-mon[34785]: osdmap e12: 2 total, 1 up, 2 in 2024-05-12T22:01:06.905 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:06 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-12T22:01:06.905 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:06 smithi186 ceph-mon[34785]: from='osd.1 ' entity='osd.1' 2024-05-12T22:01:06.905 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:06 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-12T22:01:06.992 INFO:journalctl@ceph.osd.1.smithi086.stdout:May 12 22:01:06 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-1[38925]: 2024-05-12T22:01:06.510+0000 7efc65c8e700 -1 osd.1 0 waiting for initial osdmap 2024-05-12T22:01:06.992 INFO:journalctl@ceph.osd.1.smithi086.stdout:May 12 22:01:06 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-1[38925]: 2024-05-12T22:01:06.515+0000 7efc5e27a700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-12T22:01:06.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:06 smithi086 ceph-mon[34297]: pgmap v71: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-12T22:01:06.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:06 smithi086 ceph-mon[34297]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]': finished 2024-05-12T22:01:06.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:06 smithi086 ceph-mon[34297]: osdmap e12: 2 total, 1 up, 2 in 2024-05-12T22:01:06.993 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:06 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-12T22:01:06.993 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:06 smithi086 ceph-mon[34297]: from='osd.1 ' entity='osd.1' 2024-05-12T22:01:06.993 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:06 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-12T22:01:07.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:06 smithi028 ceph-mon[29847]: pgmap v71: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-12T22:01:07.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:06 smithi028 ceph-mon[29847]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]': finished 2024-05-12T22:01:07.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:06 smithi028 ceph-mon[29847]: osdmap e12: 2 total, 1 up, 2 in 2024-05-12T22:01:07.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:06 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-12T22:01:07.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:06 smithi028 ceph-mon[29847]: from='osd.1 ' entity='osd.1' 2024-05-12T22:01:07.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:06 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-12T22:01:07.114 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/mon.c/config 2024-05-12T22:01:07.901 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:07 smithi186 ceph-mon[34785]: purged_snaps scrub starts 2024-05-12T22:01:07.901 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:07 smithi186 ceph-mon[34785]: purged_snaps scrub ok 2024-05-12T22:01:07.901 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:07 smithi186 ceph-mon[34785]: osd.1 [v2:172.21.15.86:6800/3354141819,v1:172.21.15.86:6801/3354141819] boot 2024-05-12T22:01:07.901 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:07 smithi186 ceph-mon[34785]: osdmap e13: 2 total, 2 up, 2 in 2024-05-12T22:01:07.901 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:07 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-12T22:01:07.901 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:07 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:01:07.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:07 smithi086 ceph-mon[34297]: purged_snaps scrub starts 2024-05-12T22:01:07.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:07 smithi086 ceph-mon[34297]: purged_snaps scrub ok 2024-05-12T22:01:07.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:07 smithi086 ceph-mon[34297]: osd.1 [v2:172.21.15.86:6800/3354141819,v1:172.21.15.86:6801/3354141819] boot 2024-05-12T22:01:07.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:07 smithi086 ceph-mon[34297]: osdmap e13: 2 total, 2 up, 2 in 2024-05-12T22:01:07.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:07 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-12T22:01:07.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:07 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:01:08.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:07 smithi028 ceph-mon[29847]: purged_snaps scrub starts 2024-05-12T22:01:08.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:07 smithi028 ceph-mon[29847]: purged_snaps scrub ok 2024-05-12T22:01:08.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:07 smithi028 ceph-mon[29847]: osd.1 [v2:172.21.15.86:6800/3354141819,v1:172.21.15.86:6801/3354141819] boot 2024-05-12T22:01:08.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:07 smithi028 ceph-mon[29847]: osdmap e13: 2 total, 2 up, 2 in 2024-05-12T22:01:08.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:07 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-12T22:01:08.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:07 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:01:08.618 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:01:08.615+0000 7f05949ad700 1 -- 172.21.15.186:0/1306200312 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f05901035b0 msgr2=0x7f0590103a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:01:08.619 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:01:08.615+0000 7f05949ad700 1 --2- 172.21.15.186:0/1306200312 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f05901035b0 0x7f0590103a10 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f0580009990 tx=0x7f0580009ca0 comp rx=0 tx=0).stop 2024-05-12T22:01:08.619 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:01:08.615+0000 7f05949ad700 1 -- 172.21.15.186:0/1306200312 shutdown_connections 2024-05-12T22:01:08.619 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:01:08.615+0000 7f05949ad700 1 --2- 172.21.15.186:0/1306200312 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0590103f50 0x7f059010a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:01:08.619 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:01:08.615+0000 7f05949ad700 1 --2- 172.21.15.186:0/1306200312 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f05901023b0 0x7f05901027d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:01:08.619 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:01:08.615+0000 7f05949ad700 1 --2- 172.21.15.186:0/1306200312 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f05901035b0 0x7f0590103a10 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:01:08.619 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:01:08.615+0000 7f05949ad700 1 -- 172.21.15.186:0/1306200312 >> 172.21.15.186:0/1306200312 conn(0x7f05900fd970 msgr2=0x7f05900ffd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:01:08.619 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:01:08.615+0000 7f05949ad700 1 -- 172.21.15.186:0/1306200312 shutdown_connections 2024-05-12T22:01:08.619 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:01:08.615+0000 7f05949ad700 1 -- 172.21.15.186:0/1306200312 wait complete. 2024-05-12T22:01:08.619 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:01:08.616+0000 7f05949ad700 1 Processor -- start 2024-05-12T22:01:08.619 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:01:08.616+0000 7f05949ad700 1 -- start start 2024-05-12T22:01:08.619 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:01:08.616+0000 7f05949ad700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f05901023b0 0x7f0590198610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:01:08.620 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:01:08.616+0000 7f05949ad700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05901035b0 0x7f0590198b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:01:08.620 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:01:08.616+0000 7f05949ad700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0590103f50 0x7f059019fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:01:08.620 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:01:08.616+0000 7f05949ad700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f059010e060 con 0x7f05901035b0 2024-05-12T22:01:08.620 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:01:08.616+0000 7f05949ad700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f059010dee0 con 0x7f05901023b0 2024-05-12T22:01:08.620 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:01:08.616+0000 7f05949ad700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f059010e1e0 con 0x7f0590103f50 2024-05-12T22:01:08.620 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:01:08.616+0000 7f058ed9d700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0590103f50 0x7f059019fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:01:08.620 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:01:08.616+0000 7f058ed9d700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0590103f50 0x7f059019fbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:43386/0 (socket says 172.21.15.186:43386) 2024-05-12T22:01:08.620 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:01:08.616+0000 7f058ed9d700 1 -- 172.21.15.186:0/46253408 learned_addr learned my addr 172.21.15.186:0/46253408 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-05-12T22:01:08.620 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:01:08.616+0000 7f058e59c700 1 --2- 172.21.15.186:0/46253408 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f05901023b0 0x7f0590198610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:01:08.621 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:01:08.617+0000 7f058ed9d700 1 -- 172.21.15.186:0/46253408 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f05901023b0 msgr2=0x7f0590198610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:01:08.621 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:01:08.617+0000 7f058ed9d700 1 --2- 172.21.15.186:0/46253408 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f05901023b0 0x7f0590198610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:01:08.621 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:01:08.617+0000 7f058ed9d700 1 -- 172.21.15.186:0/46253408 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05901035b0 msgr2=0x7f0590198b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:01:08.621 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:01:08.617+0000 7f058ed9d700 1 --2- 172.21.15.186:0/46253408 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05901035b0 0x7f0590198b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:01:08.621 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:01:08.617+0000 7f058ed9d700 1 -- 172.21.15.186:0/46253408 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05901a0270 con 0x7f0590103f50 2024-05-12T22:01:08.621 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:01:08.617+0000 7f058ed9d700 1 --2- 172.21.15.186:0/46253408 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0590103f50 0x7f059019fbd0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f058800b9a0 tx=0x7f058800bcb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:01:08.621 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:01:08.617+0000 7f057f7fe700 1 -- 172.21.15.186:0/46253408 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0588019070 con 0x7f0590103f50 2024-05-12T22:01:08.621 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:01:08.617+0000 7f05949ad700 1 -- 172.21.15.186:0/46253408 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05901a0560 con 0x7f0590103f50 2024-05-12T22:01:08.621 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:01:08.617+0000 7f057f7fe700 1 -- 172.21.15.186:0/46253408 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f058800ce10 con 0x7f0590103f50 2024-05-12T22:01:08.621 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:01:08.617+0000 7f05949ad700 1 -- 172.21.15.186:0/46253408 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05901a0b20 con 0x7f0590103f50 2024-05-12T22:01:08.621 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:01:08.617+0000 7f057f7fe700 1 -- 172.21.15.186:0/46253408 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05880073d0 con 0x7f0590103f50 2024-05-12T22:01:08.621 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:01:08.618+0000 7f05949ad700 1 -- 172.21.15.186:0/46253408 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0590069010 con 0x7f0590103f50 2024-05-12T22:01:08.622 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:01:08.619+0000 7f057f7fe700 1 -- 172.21.15.186:0/46253408 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 15) ==== 137777+0+0 (secure 0 0 0) 0x7f058801c070 con 0x7f0590103f50 2024-05-12T22:01:08.622 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:01:08.620+0000 7f057f7fe700 1 --2- 172.21.15.186:0/46253408 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f05780a2b90 0x7f05780a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:01:08.622 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:01:08.620+0000 7f057f7fe700 1 -- 172.21.15.186:0/46253408 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(13..13 src has 1..13) ==== 2309+0+0 (secure 0 0 0) 0x7f05880cd010 con 0x7f0590103f50 2024-05-12T22:01:08.623 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:01:08.621+0000 7f058e59c700 1 --2- 172.21.15.186:0/46253408 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f05780a2b90 0x7f05780a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:01:08.623 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:01:08.622+0000 7f058e59c700 1 --2- 172.21.15.186:0/46253408 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f05780a2b90 0x7f05780a5050 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f0590103410 tx=0x7f058400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:01:08.624 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:01:08.624+0000 7f057f7fe700 1 -- 172.21.15.186:0/46253408 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f058809b7c0 con 0x7f0590103f50 2024-05-12T22:01:08.831 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:01:08.830+0000 7f05949ad700 1 -- 172.21.15.186:0/46253408 --> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f0590062760 con 0x7f05780a2b90 2024-05-12T22:01:08.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:08 smithi086 ceph-mon[34297]: pgmap v74: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-12T22:01:08.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:08 smithi086 ceph-mon[34297]: Detected new or changed devices on smithi086 2024-05-12T22:01:08.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:08 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:01:08.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:08 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:01:08.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:08 smithi086 ceph-mon[34297]: Adjusting osd_memory_target on smithi086 to 17088M 2024-05-12T22:01:08.993 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:08 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:01:08.993 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:08 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:01:08.993 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:08 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:01:08.993 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:08 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:01:08.993 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:08 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:01:08.993 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:08 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:01:08.993 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:08 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:01:08.993 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:08 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:01:09.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:08 smithi028 ceph-mon[29847]: pgmap v74: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-12T22:01:09.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:08 smithi028 ceph-mon[29847]: Detected new or changed devices on smithi086 2024-05-12T22:01:09.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:08 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:01:09.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:08 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:01:09.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:08 smithi028 ceph-mon[29847]: Adjusting osd_memory_target on smithi086 to 17088M 2024-05-12T22:01:09.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:08 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:01:09.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:08 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:01:09.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:08 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:01:09.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:08 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:01:09.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:08 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:01:09.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:08 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:01:09.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:08 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:01:09.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:08 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:01:09.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:08 smithi186 ceph-mon[34785]: pgmap v74: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-12T22:01:09.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:08 smithi186 ceph-mon[34785]: Detected new or changed devices on smithi086 2024-05-12T22:01:09.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:08 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:01:09.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:08 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:01:09.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:08 smithi186 ceph-mon[34785]: Adjusting osd_memory_target on smithi086 to 17088M 2024-05-12T22:01:09.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:08 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:01:09.180 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:08 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:01:09.180 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:08 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:01:09.180 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:08 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:01:09.180 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:08 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:01:09.180 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:08 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:01:09.180 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:08 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:01:09.180 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:08 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:01:09.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:09 smithi186 ceph-mon[34785]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-12T22:01:09.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:09 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-12T22:01:09.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:09 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-12T22:01:09.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:09 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:01:09.767 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:09 smithi186 ceph-mon[34785]: osdmap e14: 2 total, 2 up, 2 in 2024-05-12T22:01:09.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:09 smithi086 ceph-mon[34297]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-12T22:01:09.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:09 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-12T22:01:09.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:09 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-12T22:01:09.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:09 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:01:09.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:09 smithi086 ceph-mon[34297]: osdmap e14: 2 total, 2 up, 2 in 2024-05-12T22:01:10.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:09 smithi028 ceph-mon[29847]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-12T22:01:10.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:09 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-12T22:01:10.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:09 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-12T22:01:10.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:09 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:01:10.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:09 smithi028 ceph-mon[29847]: osdmap e14: 2 total, 2 up, 2 in 2024-05-12T22:01:11.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:11 smithi186 ceph-mon[34785]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-12T22:01:11.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:11 smithi086 ceph-mon[34297]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-12T22:01:11.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:11 smithi028 ceph-mon[29847]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-12T22:01:12.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:12 smithi186 ceph-mon[34785]: from='client.? 172.21.15.186:0/2844747122' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "54a2f221-fa65-4795-9f5e-e660cc7c0098"}]: dispatch 2024-05-12T22:01:12.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:12 smithi186 ceph-mon[34785]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "54a2f221-fa65-4795-9f5e-e660cc7c0098"}]: dispatch 2024-05-12T22:01:12.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:12 smithi186 ceph-mon[34785]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "54a2f221-fa65-4795-9f5e-e660cc7c0098"}]': finished 2024-05-12T22:01:12.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:12 smithi186 ceph-mon[34785]: osdmap e15: 3 total, 2 up, 3 in 2024-05-12T22:01:12.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:12 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-12T22:01:12.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:12 smithi086 ceph-mon[34297]: from='client.? 172.21.15.186:0/2844747122' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "54a2f221-fa65-4795-9f5e-e660cc7c0098"}]: dispatch 2024-05-12T22:01:12.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:12 smithi086 ceph-mon[34297]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "54a2f221-fa65-4795-9f5e-e660cc7c0098"}]: dispatch 2024-05-12T22:01:12.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:12 smithi086 ceph-mon[34297]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "54a2f221-fa65-4795-9f5e-e660cc7c0098"}]': finished 2024-05-12T22:01:12.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:12 smithi086 ceph-mon[34297]: osdmap e15: 3 total, 2 up, 3 in 2024-05-12T22:01:12.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:12 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-12T22:01:12.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:12 smithi028 ceph-mon[29847]: from='client.? 172.21.15.186:0/2844747122' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "54a2f221-fa65-4795-9f5e-e660cc7c0098"}]: dispatch 2024-05-12T22:01:12.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:12 smithi028 ceph-mon[29847]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "54a2f221-fa65-4795-9f5e-e660cc7c0098"}]: dispatch 2024-05-12T22:01:12.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:12 smithi028 ceph-mon[29847]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "54a2f221-fa65-4795-9f5e-e660cc7c0098"}]': finished 2024-05-12T22:01:12.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:12 smithi028 ceph-mon[29847]: osdmap e15: 3 total, 2 up, 3 in 2024-05-12T22:01:12.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:12 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-12T22:01:13.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:13 smithi186 ceph-mon[34785]: pgmap v78: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-12T22:01:13.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:13 smithi186 ceph-mon[34785]: from='client.? 172.21.15.186:0/4202411859' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-12T22:01:13.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:13 smithi086 ceph-mon[34297]: pgmap v78: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-12T22:01:13.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:13 smithi086 ceph-mon[34297]: from='client.? 172.21.15.186:0/4202411859' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-12T22:01:13.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:13 smithi028 ceph-mon[29847]: pgmap v78: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-12T22:01:13.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:13 smithi028 ceph-mon[29847]: from='client.? 172.21.15.186:0/4202411859' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-12T22:01:15.325 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:15 smithi186 ceph-mon[34785]: pgmap v79: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-12T22:01:15.338 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:15 smithi028 ceph-mon[29847]: pgmap v79: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-12T22:01:15.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:15 smithi086 ceph-mon[34297]: pgmap v79: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-12T22:01:17.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:17 smithi186 ceph-mon[34785]: pgmap v80: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-12T22:01:17.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:17 smithi086 ceph-mon[34297]: pgmap v80: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-12T22:01:17.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:17 smithi028 ceph-mon[29847]: pgmap v80: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-12T22:01:19.343 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:19 smithi186 ceph-mon[34785]: pgmap v81: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-12T22:01:19.343 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:19 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-05-12T22:01:19.343 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:19 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:01:19.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:19 smithi086 ceph-mon[34297]: pgmap v81: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-12T22:01:19.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:19 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-05-12T22:01:19.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:19 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:01:19.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:19 smithi028 ceph-mon[29847]: pgmap v81: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-12T22:01:19.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:19 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-05-12T22:01:19.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:19 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:01:20.334 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:20 smithi186 ceph-mon[34785]: Deploying daemon osd.2 on smithi186 2024-05-12T22:01:20.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:20 smithi086 ceph-mon[34297]: Deploying daemon osd.2 on smithi186 2024-05-12T22:01:20.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:20 smithi028 ceph-mon[29847]: Deploying daemon osd.2 on smithi186 2024-05-12T22:01:21.344 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:21 smithi186 ceph-mon[34785]: pgmap v82: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-12T22:01:21.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:21 smithi086 ceph-mon[34297]: pgmap v82: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-12T22:01:21.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:21 smithi028 ceph-mon[29847]: pgmap v82: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-12T22:01:23.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:23 smithi186 ceph-mon[34785]: pgmap v83: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-12T22:01:23.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:23 smithi086 ceph-mon[34297]: pgmap v83: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-12T22:01:23.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:23 smithi028 ceph-mon[29847]: pgmap v83: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-12T22:01:24.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:24 smithi086 ceph-mon[34297]: pgmap v84: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-12T22:01:24.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:24 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:01:24.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:24 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:01:24.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:24 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:01:25.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:24 smithi028 ceph-mon[29847]: pgmap v84: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-12T22:01:25.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:24 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:01:25.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:24 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:01:25.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:24 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:01:25.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:24 smithi186 ceph-mon[34785]: pgmap v84: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-12T22:01:25.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:24 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:01:25.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:24 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:01:25.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:24 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:01:26.939 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:26 smithi186 ceph-mon[34785]: pgmap v85: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-12T22:01:26.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:26 smithi086 ceph-mon[34297]: pgmap v85: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-12T22:01:27.014 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:01:27.013+0000 7f057f7fe700 1 -- 172.21.15.186:0/46253408 <== mgr.14152 v2:172.21.15.28:6800/3123328651 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f0590062760 con 0x7f05780a2b90 2024-05-12T22:01:27.014 INFO:teuthology.orchestra.run.smithi186.stdout:Created osd(s) 2 on host 'smithi186' 2024-05-12T22:01:27.016 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:01:27.015+0000 7f05949ad700 1 -- 172.21.15.186:0/46253408 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f05780a2b90 msgr2=0x7f05780a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:01:27.016 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:01:27.015+0000 7f05949ad700 1 --2- 172.21.15.186:0/46253408 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f05780a2b90 0x7f05780a5050 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f0590103410 tx=0x7f058400b410 comp rx=0 tx=0).stop 2024-05-12T22:01:27.016 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:01:27.015+0000 7f05949ad700 1 -- 172.21.15.186:0/46253408 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0590103f50 msgr2=0x7f059019fbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:01:27.016 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:01:27.015+0000 7f05949ad700 1 --2- 172.21.15.186:0/46253408 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0590103f50 0x7f059019fbd0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f058800b9a0 tx=0x7f058800bcb0 comp rx=0 tx=0).stop 2024-05-12T22:01:27.016 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:01:27.015+0000 7f05949ad700 1 -- 172.21.15.186:0/46253408 shutdown_connections 2024-05-12T22:01:27.016 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:01:27.015+0000 7f05949ad700 1 --2- 172.21.15.186:0/46253408 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f05780a2b90 0x7f05780a5050 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:01:27.016 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:01:27.015+0000 7f05949ad700 1 --2- 172.21.15.186:0/46253408 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0590103f50 0x7f059019fbd0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:01:27.016 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:01:27.015+0000 7f05949ad700 1 --2- 172.21.15.186:0/46253408 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f05901023b0 0x7f0590198610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:01:27.017 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:01:27.015+0000 7f05949ad700 1 --2- 172.21.15.186:0/46253408 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05901035b0 0x7f0590198b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:01:27.017 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:01:27.015+0000 7f05949ad700 1 -- 172.21.15.186:0/46253408 >> 172.21.15.186:0/46253408 conn(0x7f05900fd970 msgr2=0x7f0590106850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:01:27.017 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:01:27.015+0000 7f05949ad700 1 -- 172.21.15.186:0/46253408 shutdown_connections 2024-05-12T22:01:27.017 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:01:27.015+0000 7f05949ad700 1 -- 172.21.15.186:0/46253408 wait complete. 2024-05-12T22:01:27.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:26 smithi028 ceph-mon[29847]: pgmap v85: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-12T22:01:27.602 DEBUG:teuthology.orchestra.run.smithi186:osd.2> sudo journalctl -f -n 0 -u ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968@osd.2.service 2024-05-12T22:01:27.644 INFO:tasks.cephadm:Deploying osd.3 on smithi028 with /dev/vg_nvme/lv_3... 2024-05-12T22:01:27.644 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f72fecff68e1d400c4568684327c900485c20d6a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6b92e7b4-10aa-11ef-bc98-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-05-12T22:01:27.977 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:27 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:01:27.977 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:27 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:01:27.977 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:27 smithi028 ceph-mon[29847]: from='osd.2 [v2:172.21.15.186:6800/906459365,v1:172.21.15.186:6801/906459365]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-12T22:01:27.977 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:27 smithi028 ceph-mon[29847]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-12T22:01:27.977 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:27 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:01:27.977 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:27 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:01:28.108 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/mon.a/config 2024-05-12T22:01:28.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:27 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:01:28.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:27 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:01:28.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:27 smithi086 ceph-mon[34297]: from='osd.2 [v2:172.21.15.186:6800/906459365,v1:172.21.15.186:6801/906459365]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-12T22:01:28.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:27 smithi086 ceph-mon[34297]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-12T22:01:28.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:27 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:01:28.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:27 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:01:28.377 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:27 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:01:28.377 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:27 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:01:28.377 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:27 smithi186 ceph-mon[34785]: from='osd.2 [v2:172.21.15.186:6800/906459365,v1:172.21.15.186:6801/906459365]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-12T22:01:28.377 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:27 smithi186 ceph-mon[34785]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-12T22:01:28.377 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:27 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:01:28.377 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:27 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:01:29.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:29 smithi028 ceph-mon[29847]: pgmap v86: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-12T22:01:29.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:29 smithi028 ceph-mon[29847]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-05-12T22:01:29.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:29 smithi028 ceph-mon[29847]: from='osd.2 [v2:172.21.15.186:6800/906459365,v1:172.21.15.186:6801/906459365]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-05-12T22:01:29.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:29 smithi028 ceph-mon[29847]: osdmap e16: 3 total, 2 up, 3 in 2024-05-12T22:01:29.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:29 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-12T22:01:29.282 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:29 smithi028 ceph-mon[29847]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-05-12T22:01:29.380 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:29 smithi186 ceph-mon[34785]: pgmap v86: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-12T22:01:29.380 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:29 smithi186 ceph-mon[34785]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-05-12T22:01:29.380 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:29 smithi186 ceph-mon[34785]: from='osd.2 [v2:172.21.15.186:6800/906459365,v1:172.21.15.186:6801/906459365]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-05-12T22:01:29.380 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:29 smithi186 ceph-mon[34785]: osdmap e16: 3 total, 2 up, 3 in 2024-05-12T22:01:29.380 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:29 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-12T22:01:29.380 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:29 smithi186 ceph-mon[34785]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-05-12T22:01:29.381 INFO:journalctl@ceph.osd.2.smithi186.stdout:May 12 22:01:29 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-2[39983]: 2024-05-12T22:01:29.001+0000 7fee17589700 -1 osd.2 0 waiting for initial osdmap 2024-05-12T22:01:29.381 INFO:journalctl@ceph.osd.2.smithi186.stdout:May 12 22:01:29 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-2[39983]: 2024-05-12T22:01:29.014+0000 7fee13b7d700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-12T22:01:29.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:29 smithi086 ceph-mon[34297]: pgmap v86: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-12T22:01:29.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:29 smithi086 ceph-mon[34297]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-05-12T22:01:29.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:29 smithi086 ceph-mon[34297]: from='osd.2 [v2:172.21.15.186:6800/906459365,v1:172.21.15.186:6801/906459365]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-05-12T22:01:29.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:29 smithi086 ceph-mon[34297]: osdmap e16: 3 total, 2 up, 3 in 2024-05-12T22:01:29.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:29 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-12T22:01:29.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:29 smithi086 ceph-mon[34297]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-05-12T22:01:30.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:30 smithi028 ceph-mon[29847]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]': finished 2024-05-12T22:01:30.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:30 smithi028 ceph-mon[29847]: osdmap e17: 3 total, 2 up, 3 in 2024-05-12T22:01:30.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:30 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-12T22:01:30.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:30 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-12T22:01:30.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:30 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-12T22:01:30.364 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:30 smithi186 ceph-mon[34785]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]': finished 2024-05-12T22:01:30.364 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:30 smithi186 ceph-mon[34785]: osdmap e17: 3 total, 2 up, 3 in 2024-05-12T22:01:30.364 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:30 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-12T22:01:30.365 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:30 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-12T22:01:30.365 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:30 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-12T22:01:30.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:30 smithi086 ceph-mon[34297]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]': finished 2024-05-12T22:01:30.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:30 smithi086 ceph-mon[34297]: osdmap e17: 3 total, 2 up, 3 in 2024-05-12T22:01:30.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:30 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-12T22:01:30.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:30 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-12T22:01:30.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:30 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-12T22:01:31.036 INFO:teuthology.orchestra.run.smithi028.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-05-12T22:01:31.036 INFO:teuthology.orchestra.run.smithi028.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-05-12T22:01:31.036 INFO:teuthology.orchestra.run.smithi028.stderr: stderr: 10+0 records in 2024-05-12T22:01:31.036 INFO:teuthology.orchestra.run.smithi028.stderr:10+0 records out 2024-05-12T22:01:31.036 INFO:teuthology.orchestra.run.smithi028.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0422278 s, 248 MB/s 2024-05-12T22:01:31.036 INFO:teuthology.orchestra.run.smithi028.stderr:--> Zapping successful for: 2024-05-12T22:01:31.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:31 smithi028 ceph-mon[29847]: purged_snaps scrub starts 2024-05-12T22:01:31.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:31 smithi028 ceph-mon[29847]: purged_snaps scrub ok 2024-05-12T22:01:31.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:31 smithi028 ceph-mon[29847]: pgmap v89: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-12T22:01:31.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:31 smithi028 ceph-mon[29847]: osd.2 [v2:172.21.15.186:6800/906459365,v1:172.21.15.186:6801/906459365] boot 2024-05-12T22:01:31.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:31 smithi028 ceph-mon[29847]: osdmap e18: 3 total, 3 up, 3 in 2024-05-12T22:01:31.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:31 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-12T22:01:31.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:31 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:01:31.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:31 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:01:31.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:31 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:01:31.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:31 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:01:31.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:31 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:01:31.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:31 smithi186 ceph-mon[34785]: purged_snaps scrub starts 2024-05-12T22:01:31.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:31 smithi186 ceph-mon[34785]: purged_snaps scrub ok 2024-05-12T22:01:31.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:31 smithi186 ceph-mon[34785]: pgmap v89: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-12T22:01:31.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:31 smithi186 ceph-mon[34785]: osd.2 [v2:172.21.15.186:6800/906459365,v1:172.21.15.186:6801/906459365] boot 2024-05-12T22:01:31.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:31 smithi186 ceph-mon[34785]: osdmap e18: 3 total, 3 up, 3 in 2024-05-12T22:01:31.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:31 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-12T22:01:31.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:31 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:01:31.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:31 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:01:31.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:31 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:01:31.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:31 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:01:31.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:31 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:01:31.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:31 smithi086 ceph-mon[34297]: purged_snaps scrub starts 2024-05-12T22:01:31.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:31 smithi086 ceph-mon[34297]: purged_snaps scrub ok 2024-05-12T22:01:31.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:31 smithi086 ceph-mon[34297]: pgmap v89: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-12T22:01:31.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:31 smithi086 ceph-mon[34297]: osd.2 [v2:172.21.15.186:6800/906459365,v1:172.21.15.186:6801/906459365] boot 2024-05-12T22:01:31.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:31 smithi086 ceph-mon[34297]: osdmap e18: 3 total, 3 up, 3 in 2024-05-12T22:01:31.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:31 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-12T22:01:31.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:31 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:01:31.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:31 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:01:31.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:31 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:01:31.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:31 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:01:31.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:31 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:01:31.987 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f72fecff68e1d400c4568684327c900485c20d6a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6b92e7b4-10aa-11ef-bc98-c7b262605968 -- ceph orch daemon add osd smithi028:vg_nvme/lv_3 2024-05-12T22:01:32.415 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/mon.a/config 2024-05-12T22:01:32.564 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:32 smithi028 ceph-mon[29847]: Detected new or changed devices on smithi186 2024-05-12T22:01:32.564 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:32 smithi028 ceph-mon[29847]: Adjusting osd_memory_target on smithi186 to 17088M 2024-05-12T22:01:32.564 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:32 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:01:32.564 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:32 smithi028 ceph-mon[29847]: osdmap e19: 3 total, 3 up, 3 in 2024-05-12T22:01:32.565 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:32 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:01:32.565 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:32 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:01:32.565 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:32 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:01:32.565 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:32 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:01:32.565 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:32 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:01:32.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:32 smithi186 ceph-mon[34785]: Detected new or changed devices on smithi186 2024-05-12T22:01:32.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:32 smithi186 ceph-mon[34785]: Adjusting osd_memory_target on smithi186 to 17088M 2024-05-12T22:01:32.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:32 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:01:32.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:32 smithi186 ceph-mon[34785]: osdmap e19: 3 total, 3 up, 3 in 2024-05-12T22:01:32.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:32 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:01:32.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:32 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:01:32.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:32 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:01:32.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:32 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:01:32.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:32 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:01:32.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:32 smithi086 ceph-mon[34297]: Detected new or changed devices on smithi186 2024-05-12T22:01:32.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:32 smithi086 ceph-mon[34297]: Adjusting osd_memory_target on smithi186 to 17088M 2024-05-12T22:01:32.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:32 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:01:32.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:32 smithi086 ceph-mon[34297]: osdmap e19: 3 total, 3 up, 3 in 2024-05-12T22:01:32.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:32 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:01:32.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:32 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:01:32.743 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:32 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:01:32.743 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:32 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:01:32.743 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:32 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:01:33.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:33 smithi028 ceph-mon[29847]: pgmap v92: 0 pgs: ; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-12T22:01:33.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:33 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-05-12T22:01:33.573 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:33 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-05-12T22:01:33.573 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:33 smithi028 ceph-mon[29847]: osdmap e20: 3 total, 3 up, 3 in 2024-05-12T22:01:33.573 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:33 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-05-12T22:01:33.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:33 smithi186 ceph-mon[34785]: pgmap v92: 0 pgs: ; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-12T22:01:33.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:33 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-05-12T22:01:33.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:33 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-05-12T22:01:33.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:33 smithi186 ceph-mon[34785]: osdmap e20: 3 total, 3 up, 3 in 2024-05-12T22:01:33.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:33 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-05-12T22:01:33.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:33 smithi086 ceph-mon[34297]: pgmap v92: 0 pgs: ; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-12T22:01:33.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:33 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-05-12T22:01:33.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:33 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-05-12T22:01:33.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:33 smithi086 ceph-mon[34297]: osdmap e20: 3 total, 3 up, 3 in 2024-05-12T22:01:33.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:33 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-05-12T22:01:34.236 INFO:journalctl@ceph.osd.0.smithi028.stdout:May 12 22:01:34 smithi028 sudo[46155]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-12T22:01:34.236 INFO:journalctl@ceph.osd.0.smithi028.stdout:May 12 22:01:34 smithi028 sudo[46155]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-12T22:01:34.236 INFO:journalctl@ceph.osd.0.smithi028.stdout:May 12 22:01:34 smithi028 sudo[46155]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-12T22:01:34.236 INFO:journalctl@ceph.osd.0.smithi028.stdout:May 12 22:01:34 smithi028 sudo[46155]: pam_unix(sudo:session): session closed for user root 2024-05-12T22:01:34.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:34 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-05-12T22:01:34.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:34 smithi028 ceph-mon[29847]: osdmap e21: 3 total, 3 up, 3 in 2024-05-12T22:01:34.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:34 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-05-12T22:01:34.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:34 smithi186 ceph-mon[34785]: osdmap e21: 3 total, 3 up, 3 in 2024-05-12T22:01:34.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:01:34.713+0000 7f1a94000700 1 -- 172.21.15.28:0/3254711698 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1a8c108070 msgr2=0x7f1a8c10a4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:01:34.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:01:34.713+0000 7f1a94000700 1 --2- 172.21.15.28:0/3254711698 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1a8c108070 0x7f1a8c10a4e0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f1a8400b0a0 tx=0x7f1a8400b3b0 comp rx=0 tx=0).stop 2024-05-12T22:01:34.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:01:34.713+0000 7f1a94000700 1 -- 172.21.15.28:0/3254711698 shutdown_connections 2024-05-12T22:01:34.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:01:34.714+0000 7f1a94000700 1 --2- 172.21.15.28:0/3254711698 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1a8c102e10 0x7f1a8c105200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:01:34.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:01:34.714+0000 7f1a94000700 1 --2- 172.21.15.28:0/3254711698 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1a8c108070 0x7f1a8c10a4e0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:01:34.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:01:34.714+0000 7f1a94000700 1 --2- 172.21.15.28:0/3254711698 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a8c105740 0x7f1a8c107b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:01:34.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:01:34.714+0000 7f1a94000700 1 -- 172.21.15.28:0/3254711698 >> 172.21.15.28:0/3254711698 conn(0x7f1a8c0fc820 msgr2=0x7f1a8c0fec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:01:34.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:01:34.714+0000 7f1a94000700 1 -- 172.21.15.28:0/3254711698 shutdown_connections 2024-05-12T22:01:34.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:01:34.714+0000 7f1a94000700 1 -- 172.21.15.28:0/3254711698 wait complete. 2024-05-12T22:01:34.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:01:34.715+0000 7f1a94000700 1 Processor -- start 2024-05-12T22:01:34.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:01:34.715+0000 7f1a94000700 1 -- start start 2024-05-12T22:01:34.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:01:34.716+0000 7f1a94000700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1a8c102e10 0x7f1a8c102770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:01:34.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:01:34.716+0000 7f1a94000700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1a8c105740 0x7f1a8c100dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:01:34.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:01:34.716+0000 7f1a94000700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a8c108070 0x7f1a8c101300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:01:34.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:01:34.716+0000 7f1a94000700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1a8c10dfe0 con 0x7f1a8c108070 2024-05-12T22:01:34.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:01:34.716+0000 7f1a94000700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f1a8c10de60 con 0x7f1a8c102e10 2024-05-12T22:01:34.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:01:34.716+0000 7f1a94000700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f1a8c10e160 con 0x7f1a8c105740 2024-05-12T22:01:34.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:01:34.716+0000 7f1a90c00700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1a8c105740 0x7f1a8c100dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:01:34.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:01:34.716+0000 7f1a91600700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1a8c102e10 0x7f1a8c102770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:01:34.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:01:34.716+0000 7f1a90c00700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1a8c105740 0x7f1a8c100dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:48786/0 (socket says 172.21.15.28:48786) 2024-05-12T22:01:34.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:01:34.716+0000 7f1a90c00700 1 -- 172.21.15.28:0/4192785124 learned_addr learned my addr 172.21.15.28:0/4192785124 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:01:34.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:01:34.716+0000 7f1a92000700 1 --2- 172.21.15.28:0/4192785124 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a8c108070 0x7f1a8c101300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:01:34.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:01:34.717+0000 7f1a91600700 1 -- 172.21.15.28:0/4192785124 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1a8c105740 msgr2=0x7f1a8c100dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:01:34.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:01:34.717+0000 7f1a91600700 1 --2- 172.21.15.28:0/4192785124 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1a8c105740 0x7f1a8c100dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:01:34.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:01:34.717+0000 7f1a91600700 1 -- 172.21.15.28:0/4192785124 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a8c108070 msgr2=0x7f1a8c101300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:01:34.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:01:34.717+0000 7f1a91600700 1 --2- 172.21.15.28:0/4192785124 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a8c108070 0x7f1a8c101300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:01:34.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:01:34.717+0000 7f1a91600700 1 -- 172.21.15.28:0/4192785124 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a8c101bb0 con 0x7f1a8c102e10 2024-05-12T22:01:34.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:01:34.717+0000 7f1a92000700 1 --2- 172.21.15.28:0/4192785124 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a8c108070 0x7f1a8c101300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:01:34.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:01:34.717+0000 7f1a90c00700 1 --2- 172.21.15.28:0/4192785124 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1a8c105740 0x7f1a8c100dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:01:34.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:01:34.717+0000 7f1a91600700 1 --2- 172.21.15.28:0/4192785124 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1a8c102e10 0x7f1a8c102770 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f1a8000d750 tx=0x7f1a8000da60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:01:34.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:01:34.717+0000 7f1a8a000700 1 -- 172.21.15.28:0/4192785124 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a80016070 con 0x7f1a8c102e10 2024-05-12T22:01:34.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:01:34.718+0000 7f1a94000700 1 -- 172.21.15.28:0/4192785124 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a8c1a2780 con 0x7f1a8c102e10 2024-05-12T22:01:34.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:01:34.718+0000 7f1a8a000700 1 -- 172.21.15.28:0/4192785124 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1a80004d10 con 0x7f1a8c102e10 2024-05-12T22:01:34.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:01:34.718+0000 7f1a8a000700 1 -- 172.21.15.28:0/4192785124 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a8000f690 con 0x7f1a8c102e10 2024-05-12T22:01:34.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:01:34.718+0000 7f1a94000700 1 -- 172.21.15.28:0/4192785124 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a8c1a2cc0 con 0x7f1a8c102e10 2024-05-12T22:01:34.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:01:34.719+0000 7f1a94000700 1 -- 172.21.15.28:0/4192785124 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a8c069010 con 0x7f1a8c102e10 2024-05-12T22:01:34.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:01:34.722+0000 7f1a8a000700 1 -- 172.21.15.28:0/4192785124 <== mon.2 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f1a80004750 con 0x7f1a8c102e10 2024-05-12T22:01:34.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:01:34.723+0000 7f1a8a000700 1 --2- 172.21.15.28:0/4192785124 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f1a740a2c20 0x7f1a740a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:01:34.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:01:34.724+0000 7f1a8a000700 1 -- 172.21.15.28:0/4192785124 <== mon.2 v2:172.21.15.86:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 3237+0+0 (secure 0 0 0) 0x7f1a800cdaa0 con 0x7f1a8c102e10 2024-05-12T22:01:34.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:01:34.724+0000 7f1a90c00700 1 --2- 172.21.15.28:0/4192785124 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f1a740a2c20 0x7f1a740a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:01:34.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:01:34.725+0000 7f1a90c00700 1 --2- 172.21.15.28:0/4192785124 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f1a740a2c20 0x7f1a740a50e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f1a78009680 tx=0x7f1a78009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:01:34.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:01:34.731+0000 7f1a8a000700 1 -- 172.21.15.28:0/4192785124 <== mon.2 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f1a8001f020 con 0x7f1a8c102e10 2024-05-12T22:01:34.741 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:34 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-05-12T22:01:34.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:34 smithi086 ceph-mon[34297]: osdmap e21: 3 total, 3 up, 3 in 2024-05-12T22:01:34.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:01:34.967+0000 7f1a94000700 1 -- 172.21.15.28:0/4192785124 --> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f1a8c062760 con 0x7f1a740a2c20 2024-05-12T22:01:34.968 INFO:journalctl@ceph.osd.0.smithi028.stdout:May 12 22:01:34 smithi028 sudo[46183]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-12T22:01:34.968 INFO:journalctl@ceph.osd.0.smithi028.stdout:May 12 22:01:34 smithi028 sudo[46183]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-12T22:01:34.968 INFO:journalctl@ceph.osd.0.smithi028.stdout:May 12 22:01:34 smithi028 sudo[46183]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-12T22:01:34.968 INFO:journalctl@ceph.osd.0.smithi028.stdout:May 12 22:01:34 smithi028 sudo[46183]: pam_unix(sudo:session): session closed for user root 2024-05-12T22:01:35.231 INFO:journalctl@ceph.osd.1.smithi086.stdout:May 12 22:01:34 smithi086 sudo[42793]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-12T22:01:35.231 INFO:journalctl@ceph.osd.1.smithi086.stdout:May 12 22:01:34 smithi086 sudo[42793]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-12T22:01:35.231 INFO:journalctl@ceph.osd.1.smithi086.stdout:May 12 22:01:34 smithi086 sudo[42793]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-12T22:01:35.231 INFO:journalctl@ceph.osd.1.smithi086.stdout:May 12 22:01:34 smithi086 sudo[42793]: pam_unix(sudo:session): session closed for user root 2024-05-12T22:01:35.492 INFO:journalctl@ceph.osd.1.smithi086.stdout:May 12 22:01:35 smithi086 sudo[42796]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-12T22:01:35.492 INFO:journalctl@ceph.osd.1.smithi086.stdout:May 12 22:01:35 smithi086 sudo[42796]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-12T22:01:35.492 INFO:journalctl@ceph.osd.1.smithi086.stdout:May 12 22:01:35 smithi086 sudo[42796]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-12T22:01:35.492 INFO:journalctl@ceph.osd.1.smithi086.stdout:May 12 22:01:35 smithi086 sudo[42796]: pam_unix(sudo:session): session closed for user root 2024-05-12T22:01:35.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:35 smithi086 ceph-mon[34297]: pgmap v95: 1 pgs: 1 creating+peering; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-12T22:01:35.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:35 smithi086 ceph-mon[34297]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-05-12T22:01:35.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:35 smithi086 ceph-mon[34297]: osdmap e22: 3 total, 3 up, 3 in 2024-05-12T22:01:35.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:35 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:01:35.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:35 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-12T22:01:35.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:35 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-12T22:01:35.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:35 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:01:35.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:35 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:01:35.679 INFO:journalctl@ceph.osd.2.smithi186.stdout:May 12 22:01:35 smithi186 sudo[44065]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-12T22:01:35.679 INFO:journalctl@ceph.osd.2.smithi186.stdout:May 12 22:01:35 smithi186 sudo[44065]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-12T22:01:35.679 INFO:journalctl@ceph.osd.2.smithi186.stdout:May 12 22:01:35 smithi186 sudo[44065]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-12T22:01:35.679 INFO:journalctl@ceph.osd.2.smithi186.stdout:May 12 22:01:35 smithi186 sudo[44065]: pam_unix(sudo:session): session closed for user root 2024-05-12T22:01:35.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:35 smithi186 ceph-mon[34785]: pgmap v95: 1 pgs: 1 creating+peering; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-12T22:01:35.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:35 smithi186 ceph-mon[34785]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-05-12T22:01:35.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:35 smithi186 ceph-mon[34785]: osdmap e22: 3 total, 3 up, 3 in 2024-05-12T22:01:35.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:35 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:01:35.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:35 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-12T22:01:35.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:35 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-12T22:01:35.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:35 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:01:35.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:35 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:01:35.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:35 smithi028 ceph-mon[29847]: pgmap v95: 1 pgs: 1 creating+peering; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-12T22:01:35.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:35 smithi028 ceph-mon[29847]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-05-12T22:01:35.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:35 smithi028 ceph-mon[29847]: osdmap e22: 3 total, 3 up, 3 in 2024-05-12T22:01:35.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:35 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:01:35.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:35 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-12T22:01:35.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:35 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-12T22:01:35.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:35 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:01:35.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:35 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:01:36.179 INFO:journalctl@ceph.osd.2.smithi186.stdout:May 12 22:01:35 smithi186 sudo[44068]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-12T22:01:36.179 INFO:journalctl@ceph.osd.2.smithi186.stdout:May 12 22:01:35 smithi186 sudo[44068]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-12T22:01:36.179 INFO:journalctl@ceph.osd.2.smithi186.stdout:May 12 22:01:35 smithi186 sudo[44068]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-12T22:01:36.179 INFO:journalctl@ceph.osd.2.smithi186.stdout:May 12 22:01:35 smithi186 sudo[44068]: pam_unix(sudo:session): session closed for user root 2024-05-12T22:01:36.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:36 smithi028 sudo[46344]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-12T22:01:36.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:36 smithi028 sudo[46344]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-12T22:01:36.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:36 smithi028 sudo[46344]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-12T22:01:36.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:36 smithi028 sudo[46344]: pam_unix(sudo:session): session closed for user root 2024-05-12T22:01:37.256 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:37 smithi186 ceph-mon[34785]: from='client.24194 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-12T22:01:37.256 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:37 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:01:37.256 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:37 smithi186 ceph-mon[34785]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-12T22:01:37.256 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:37 smithi186 sudo[44071]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-12T22:01:37.257 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:37 smithi186 sudo[44071]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-12T22:01:37.257 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:37 smithi186 sudo[44071]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-12T22:01:37.257 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:37 smithi186 sudo[44071]: pam_unix(sudo:session): session closed for user root 2024-05-12T22:01:37.263 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:36 smithi028 sudo[46350]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-12T22:01:37.264 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:36 smithi028 sudo[46350]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-12T22:01:37.264 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:36 smithi028 sudo[46350]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-12T22:01:37.264 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:36 smithi028 sudo[46350]: pam_unix(sudo:session): session closed for user root 2024-05-12T22:01:37.264 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:37 smithi028 ceph-mon[29847]: from='client.24194 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-12T22:01:37.264 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:37 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:01:37.264 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:37 smithi028 ceph-mon[29847]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-12T22:01:37.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:37 smithi086 ceph-mon[34297]: from='client.24194 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-12T22:01:37.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:37 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:01:37.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:37 smithi086 ceph-mon[34297]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-12T22:01:37.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:37 smithi186 sudo[44074]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-12T22:01:37.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:37 smithi186 sudo[44074]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-12T22:01:37.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:37 smithi186 sudo[44074]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-12T22:01:37.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:37 smithi186 sudo[44074]: pam_unix(sudo:session): session closed for user root 2024-05-12T22:01:37.995 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:37 smithi086 sudo[42811]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-12T22:01:37.995 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:37 smithi086 sudo[42811]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-12T22:01:37.995 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:37 smithi086 sudo[42811]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-12T22:01:37.995 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:37 smithi086 sudo[42811]: pam_unix(sudo:session): session closed for user root 2024-05-12T22:01:38.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:38 smithi028 ceph-mon[29847]: pgmap v97: 1 pgs: 1 creating+peering; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-12T22:01:38.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:38 smithi028 ceph-mon[29847]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-12T22:01:38.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:38 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-12T22:01:38.287 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:38 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-12T22:01:38.287 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:38 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-12T22:01:38.287 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:38 smithi028 ceph-mon[29847]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-12T22:01:38.287 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:38 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-12T22:01:38.287 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:38 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-12T22:01:38.287 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:38 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-12T22:01:38.287 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:38 smithi028 ceph-mon[29847]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-12T22:01:38.287 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:38 smithi028 ceph-mon[29847]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-12T22:01:38.287 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:38 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-12T22:01:38.287 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:38 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-12T22:01:38.287 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:38 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-12T22:01:38.288 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:38 smithi028 ceph-mon[29847]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-12T22:01:38.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:38 smithi186 ceph-mon[34785]: pgmap v97: 1 pgs: 1 creating+peering; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-12T22:01:38.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:38 smithi186 ceph-mon[34785]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-12T22:01:38.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:38 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-12T22:01:38.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:38 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-12T22:01:38.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:38 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-12T22:01:38.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:38 smithi186 ceph-mon[34785]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-12T22:01:38.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:38 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-12T22:01:38.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:38 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-12T22:01:38.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:38 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-12T22:01:38.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:38 smithi186 ceph-mon[34785]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-12T22:01:38.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:38 smithi186 ceph-mon[34785]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-12T22:01:38.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:38 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-12T22:01:38.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:38 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-12T22:01:38.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:38 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-12T22:01:38.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:38 smithi186 ceph-mon[34785]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-12T22:01:38.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:37 smithi086 sudo[42814]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-12T22:01:38.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:37 smithi086 sudo[42814]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-12T22:01:38.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:37 smithi086 sudo[42814]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-12T22:01:38.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:37 smithi086 sudo[42814]: pam_unix(sudo:session): session closed for user root 2024-05-12T22:01:38.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:38 smithi086 ceph-mon[34297]: pgmap v97: 1 pgs: 1 creating+peering; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-12T22:01:38.494 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:38 smithi086 ceph-mon[34297]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-12T22:01:38.494 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:38 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-12T22:01:38.494 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:38 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-12T22:01:38.494 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:38 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-12T22:01:38.494 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:38 smithi086 ceph-mon[34297]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-12T22:01:38.494 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:38 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-12T22:01:38.494 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:38 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-12T22:01:38.494 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:38 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-12T22:01:38.494 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:38 smithi086 ceph-mon[34297]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-12T22:01:38.494 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:38 smithi086 ceph-mon[34297]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-12T22:01:38.495 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:38 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-12T22:01:38.495 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:38 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-12T22:01:38.495 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:38 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-12T22:01:38.495 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:38 smithi086 ceph-mon[34297]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-12T22:01:39.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:39 smithi028 ceph-mon[29847]: pgmap v98: 1 pgs: 1 creating+peering; 0 B data, 85 MiB used, 268 GiB / 268 GiB avail 2024-05-12T22:01:39.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:39 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/3949677463' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "86836737-ac9a-487b-93f2-7fcd3c406960"}]: dispatch 2024-05-12T22:01:39.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:39 smithi028 ceph-mon[29847]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "86836737-ac9a-487b-93f2-7fcd3c406960"}]: dispatch 2024-05-12T22:01:39.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:39 smithi028 ceph-mon[29847]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "86836737-ac9a-487b-93f2-7fcd3c406960"}]': finished 2024-05-12T22:01:39.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:39 smithi028 ceph-mon[29847]: osdmap e23: 4 total, 3 up, 4 in 2024-05-12T22:01:39.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:39 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-12T22:01:39.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:39 smithi186 ceph-mon[34785]: pgmap v98: 1 pgs: 1 creating+peering; 0 B data, 85 MiB used, 268 GiB / 268 GiB avail 2024-05-12T22:01:39.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:39 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/3949677463' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "86836737-ac9a-487b-93f2-7fcd3c406960"}]: dispatch 2024-05-12T22:01:39.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:39 smithi186 ceph-mon[34785]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "86836737-ac9a-487b-93f2-7fcd3c406960"}]: dispatch 2024-05-12T22:01:39.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:39 smithi186 ceph-mon[34785]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "86836737-ac9a-487b-93f2-7fcd3c406960"}]': finished 2024-05-12T22:01:39.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:39 smithi186 ceph-mon[34785]: osdmap e23: 4 total, 3 up, 4 in 2024-05-12T22:01:39.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:39 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-12T22:01:39.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:39 smithi086 ceph-mon[34297]: pgmap v98: 1 pgs: 1 creating+peering; 0 B data, 85 MiB used, 268 GiB / 268 GiB avail 2024-05-12T22:01:39.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:39 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/3949677463' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "86836737-ac9a-487b-93f2-7fcd3c406960"}]: dispatch 2024-05-12T22:01:39.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:39 smithi086 ceph-mon[34297]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "86836737-ac9a-487b-93f2-7fcd3c406960"}]: dispatch 2024-05-12T22:01:39.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:39 smithi086 ceph-mon[34297]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "86836737-ac9a-487b-93f2-7fcd3c406960"}]': finished 2024-05-12T22:01:39.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:39 smithi086 ceph-mon[34297]: osdmap e23: 4 total, 3 up, 4 in 2024-05-12T22:01:39.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:39 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-12T22:01:40.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:40 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/1647931585' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-12T22:01:40.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:40 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/1647931585' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-12T22:01:40.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:40 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/1647931585' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-12T22:01:41.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:41 smithi186 ceph-mon[34785]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail 2024-05-12T22:01:41.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:41 smithi028 ceph-mon[29847]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail 2024-05-12T22:01:41.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:41 smithi086 ceph-mon[34297]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail 2024-05-12T22:01:43.067 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:43 smithi028 ceph-mon[29847]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-05-12T22:01:43.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:43 smithi186 ceph-mon[34785]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-05-12T22:01:43.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:43 smithi086 ceph-mon[34297]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-05-12T22:01:45.132 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:45 smithi028 ceph-mon[29847]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 2.1 KiB/s rd, 54 KiB/s wr, 5 op/s 2024-05-12T22:01:45.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:45 smithi186 ceph-mon[34785]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 2.1 KiB/s rd, 54 KiB/s wr, 5 op/s 2024-05-12T22:01:45.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:45 smithi086 ceph-mon[34297]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 2.1 KiB/s rd, 54 KiB/s wr, 5 op/s 2024-05-12T22:01:47.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:47 smithi186 ceph-mon[34785]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 38 KiB/s wr, 3 op/s 2024-05-12T22:01:47.486 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:47 smithi028 ceph-mon[29847]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 38 KiB/s wr, 3 op/s 2024-05-12T22:01:47.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:47 smithi086 ceph-mon[34297]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 38 KiB/s wr, 3 op/s 2024-05-12T22:01:48.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:48 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-05-12T22:01:48.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:48 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:01:48.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:48 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-05-12T22:01:48.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:48 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:01:48.534 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:48 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-05-12T22:01:48.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:48 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:01:49.183 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:49 smithi028 ceph-mon[29847]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 1.1 KiB/s rd, 30 KiB/s wr, 2 op/s 2024-05-12T22:01:49.183 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:49 smithi028 ceph-mon[29847]: Deploying daemon osd.3 on smithi028 2024-05-12T22:01:49.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:49 smithi186 ceph-mon[34785]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 1.1 KiB/s rd, 30 KiB/s wr, 2 op/s 2024-05-12T22:01:49.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:49 smithi186 ceph-mon[34785]: Deploying daemon osd.3 on smithi028 2024-05-12T22:01:49.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:49 smithi086 ceph-mon[34297]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 1.1 KiB/s rd, 30 KiB/s wr, 2 op/s 2024-05-12T22:01:49.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:49 smithi086 ceph-mon[34297]: Deploying daemon osd.3 on smithi028 2024-05-12T22:01:51.165 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:51 smithi028 ceph-mon[29847]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 946 B/s rd, 24 KiB/s wr, 2 op/s 2024-05-12T22:01:51.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:51 smithi186 ceph-mon[34785]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 946 B/s rd, 24 KiB/s wr, 2 op/s 2024-05-12T22:01:51.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:51 smithi086 ceph-mon[34297]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 946 B/s rd, 24 KiB/s wr, 2 op/s 2024-05-12T22:01:53.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:53 smithi028 ceph-mon[29847]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2024-05-12T22:01:53.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:53 smithi186 ceph-mon[34785]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2024-05-12T22:01:53.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:53 smithi086 ceph-mon[34297]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2024-05-12T22:01:55.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:55 smithi186 ceph-mon[34785]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2024-05-12T22:01:55.466 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:55 smithi028 ceph-mon[29847]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2024-05-12T22:01:55.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:55 smithi086 ceph-mon[34297]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2024-05-12T22:01:57.677 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:57 smithi028 ceph-mon[29847]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-05-12T22:01:57.677 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:57 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:01:57.677 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:57 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:01:57.678 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:57 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:01:57.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:57 smithi186 ceph-mon[34785]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-05-12T22:01:57.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:57 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:01:57.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:57 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:01:57.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:57 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:01:57.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:57 smithi086 ceph-mon[34297]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-05-12T22:01:57.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:57 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:01:57.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:57 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:01:57.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:57 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:01:59.455 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:01:59 smithi028 ceph-mon[29847]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-05-12T22:01:59.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:01:59 smithi186 ceph-mon[34785]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-05-12T22:01:59.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:01:59 smithi086 ceph-mon[34297]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-05-12T22:02:01.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:00 smithi028 ceph-mon[29847]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-05-12T22:02:01.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:00 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:02:01.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:00 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:02:01.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:00 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:02:01.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:00 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:02:01.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:00 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:02:01.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:00 smithi028 ceph-mon[29847]: from='osd.3 [v2:172.21.15.28:6810/4182884436,v1:172.21.15.28:6811/4182884436]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-05-12T22:02:01.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:02:01.163+0000 7f1a8a000700 1 -- 172.21.15.28:0/4192785124 <== mgr.14152 v2:172.21.15.28:6800/3123328651 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f1a8c062760 con 0x7f1a740a2c20 2024-05-12T22:02:01.164 INFO:teuthology.orchestra.run.smithi028.stdout:Created osd(s) 3 on host 'smithi028' 2024-05-12T22:02:01.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:02:01.167+0000 7f1a94000700 1 -- 172.21.15.28:0/4192785124 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f1a740a2c20 msgr2=0x7f1a740a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:02:01.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:02:01.167+0000 7f1a94000700 1 --2- 172.21.15.28:0/4192785124 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f1a740a2c20 0x7f1a740a50e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f1a78009680 tx=0x7f1a78009260 comp rx=0 tx=0).stop 2024-05-12T22:02:01.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:02:01.167+0000 7f1a94000700 1 -- 172.21.15.28:0/4192785124 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1a8c102e10 msgr2=0x7f1a8c102770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:02:01.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:02:01.167+0000 7f1a94000700 1 --2- 172.21.15.28:0/4192785124 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1a8c102e10 0x7f1a8c102770 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f1a8000d750 tx=0x7f1a8000da60 comp rx=0 tx=0).stop 2024-05-12T22:02:01.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:02:01.167+0000 7f1a94000700 1 -- 172.21.15.28:0/4192785124 shutdown_connections 2024-05-12T22:02:01.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:02:01.167+0000 7f1a94000700 1 --2- 172.21.15.28:0/4192785124 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f1a740a2c20 0x7f1a740a50e0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:02:01.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:02:01.167+0000 7f1a94000700 1 --2- 172.21.15.28:0/4192785124 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a8c108070 0x7f1a8c101300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:02:01.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:02:01.167+0000 7f1a94000700 1 --2- 172.21.15.28:0/4192785124 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1a8c102e10 0x7f1a8c102770 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:02:01.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:02:01.167+0000 7f1a94000700 1 --2- 172.21.15.28:0/4192785124 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1a8c105740 0x7f1a8c100dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:02:01.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:02:01.167+0000 7f1a94000700 1 -- 172.21.15.28:0/4192785124 >> 172.21.15.28:0/4192785124 conn(0x7f1a8c0fc820 msgr2=0x7f1a8c1036b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:02:01.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:02:01.167+0000 7f1a94000700 1 -- 172.21.15.28:0/4192785124 shutdown_connections 2024-05-12T22:02:01.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:02:01.167+0000 7f1a94000700 1 -- 172.21.15.28:0/4192785124 wait complete. 2024-05-12T22:02:01.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:00 smithi186 ceph-mon[34785]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-05-12T22:02:01.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:00 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:02:01.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:00 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:02:01.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:00 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:02:01.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:00 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:02:01.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:00 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:02:01.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:00 smithi186 ceph-mon[34785]: from='osd.3 [v2:172.21.15.28:6810/4182884436,v1:172.21.15.28:6811/4182884436]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-05-12T22:02:01.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:00 smithi086 ceph-mon[34297]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-05-12T22:02:01.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:00 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:02:01.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:00 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:02:01.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:00 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:02:01.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:00 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:02:01.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:00 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:02:01.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:00 smithi086 ceph-mon[34297]: from='osd.3 [v2:172.21.15.28:6810/4182884436,v1:172.21.15.28:6811/4182884436]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-05-12T22:02:01.908 DEBUG:teuthology.orchestra.run.smithi028:osd.3> sudo journalctl -f -n 0 -u ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968@osd.3.service 2024-05-12T22:02:01.913 INFO:tasks.cephadm:Deploying osd.4 on smithi086 with /dev/vg_nvme/lv_3... 2024-05-12T22:02:01.913 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f72fecff68e1d400c4568684327c900485c20d6a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6b92e7b4-10aa-11ef-bc98-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-05-12T22:02:02.141 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:01 smithi028 ceph-mon[29847]: from='osd.3 [v2:172.21.15.28:6810/4182884436,v1:172.21.15.28:6811/4182884436]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-05-12T22:02:02.142 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:01 smithi028 ceph-mon[29847]: osdmap e24: 4 total, 3 up, 4 in 2024-05-12T22:02:02.142 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:01 smithi028 ceph-mon[29847]: from='osd.3 [v2:172.21.15.28:6810/4182884436,v1:172.21.15.28:6811/4182884436]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-05-12T22:02:02.142 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:01 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-12T22:02:02.142 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:01 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:02:02.142 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:01 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:02:02.142 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:01 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:02:02.157 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/mon.b/config 2024-05-12T22:02:02.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:01 smithi186 ceph-mon[34785]: from='osd.3 [v2:172.21.15.28:6810/4182884436,v1:172.21.15.28:6811/4182884436]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-05-12T22:02:02.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:01 smithi186 ceph-mon[34785]: osdmap e24: 4 total, 3 up, 4 in 2024-05-12T22:02:02.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:01 smithi186 ceph-mon[34785]: from='osd.3 [v2:172.21.15.28:6810/4182884436,v1:172.21.15.28:6811/4182884436]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-05-12T22:02:02.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:01 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-12T22:02:02.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:01 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:02:02.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:01 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:02:02.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:01 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:02:02.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:01 smithi086 ceph-mon[34297]: from='osd.3 [v2:172.21.15.28:6810/4182884436,v1:172.21.15.28:6811/4182884436]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-05-12T22:02:02.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:01 smithi086 ceph-mon[34297]: osdmap e24: 4 total, 3 up, 4 in 2024-05-12T22:02:02.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:01 smithi086 ceph-mon[34297]: from='osd.3 [v2:172.21.15.28:6810/4182884436,v1:172.21.15.28:6811/4182884436]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-05-12T22:02:02.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:01 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-12T22:02:02.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:01 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:02:02.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:01 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:02:02.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:01 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:02:02.989 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:02 smithi028 ceph-mon[29847]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-05-12T22:02:02.989 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:02 smithi028 ceph-mon[29847]: from='osd.3 [v2:172.21.15.28:6810/4182884436,v1:172.21.15.28:6811/4182884436]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-05-12T22:02:02.989 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:02 smithi028 ceph-mon[29847]: osdmap e25: 4 total, 3 up, 4 in 2024-05-12T22:02:02.989 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:02 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-12T22:02:02.989 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:02 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-12T22:02:02.989 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:02 smithi028 ceph-mon[29847]: osd.3 [v2:172.21.15.28:6810/4182884436,v1:172.21.15.28:6811/4182884436] boot 2024-05-12T22:02:02.989 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:02 smithi028 ceph-mon[29847]: osdmap e26: 4 total, 4 up, 4 in 2024-05-12T22:02:02.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:02 smithi086 ceph-mon[34297]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-05-12T22:02:02.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:02 smithi086 ceph-mon[34297]: from='osd.3 [v2:172.21.15.28:6810/4182884436,v1:172.21.15.28:6811/4182884436]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-05-12T22:02:02.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:02 smithi086 ceph-mon[34297]: osdmap e25: 4 total, 3 up, 4 in 2024-05-12T22:02:02.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:02 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-12T22:02:02.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:02 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-12T22:02:02.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:02 smithi086 ceph-mon[34297]: osd.3 [v2:172.21.15.28:6810/4182884436,v1:172.21.15.28:6811/4182884436] boot 2024-05-12T22:02:02.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:02 smithi086 ceph-mon[34297]: osdmap e26: 4 total, 4 up, 4 in 2024-05-12T22:02:03.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:02 smithi186 ceph-mon[34785]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-05-12T22:02:03.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:02 smithi186 ceph-mon[34785]: from='osd.3 [v2:172.21.15.28:6810/4182884436,v1:172.21.15.28:6811/4182884436]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-05-12T22:02:03.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:02 smithi186 ceph-mon[34785]: osdmap e25: 4 total, 3 up, 4 in 2024-05-12T22:02:03.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:02 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-12T22:02:03.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:02 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-12T22:02:03.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:02 smithi186 ceph-mon[34785]: osd.3 [v2:172.21.15.28:6810/4182884436,v1:172.21.15.28:6811/4182884436] boot 2024-05-12T22:02:03.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:02 smithi186 ceph-mon[34785]: osdmap e26: 4 total, 4 up, 4 in 2024-05-12T22:02:03.993 INFO:teuthology.orchestra.run.smithi086.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-05-12T22:02:03.993 INFO:teuthology.orchestra.run.smithi086.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-05-12T22:02:03.993 INFO:teuthology.orchestra.run.smithi086.stderr: stderr: 10+0 records in 2024-05-12T22:02:03.993 INFO:teuthology.orchestra.run.smithi086.stderr:10+0 records out 2024-05-12T22:02:03.993 INFO:teuthology.orchestra.run.smithi086.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113281 s, 926 MB/s 2024-05-12T22:02:03.993 INFO:teuthology.orchestra.run.smithi086.stderr:--> Zapping successful for: 2024-05-12T22:02:03.993 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:03 smithi086 ceph-mon[34297]: purged_snaps scrub starts 2024-05-12T22:02:03.994 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:03 smithi086 ceph-mon[34297]: purged_snaps scrub ok 2024-05-12T22:02:03.994 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:03 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-12T22:02:03.994 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:03 smithi086 ceph-mon[34297]: osdmap e27: 4 total, 4 up, 4 in 2024-05-12T22:02:04.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:03 smithi186 ceph-mon[34785]: purged_snaps scrub starts 2024-05-12T22:02:04.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:03 smithi186 ceph-mon[34785]: purged_snaps scrub ok 2024-05-12T22:02:04.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:03 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-12T22:02:04.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:03 smithi186 ceph-mon[34785]: osdmap e27: 4 total, 4 up, 4 in 2024-05-12T22:02:04.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:03 smithi028 ceph-mon[29847]: purged_snaps scrub starts 2024-05-12T22:02:04.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:03 smithi028 ceph-mon[29847]: purged_snaps scrub ok 2024-05-12T22:02:04.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:03 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-12T22:02:04.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:03 smithi028 ceph-mon[29847]: osdmap e27: 4 total, 4 up, 4 in 2024-05-12T22:02:04.658 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f72fecff68e1d400c4568684327c900485c20d6a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6b92e7b4-10aa-11ef-bc98-c7b262605968 -- ceph orch daemon add osd smithi086:vg_nvme/lv_3 2024-05-12T22:02:04.897 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/mon.b/config 2024-05-12T22:02:04.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:04 smithi086 ceph-mon[34297]: pgmap v116: 1 pgs: 1 peering; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-05-12T22:02:04.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:04 smithi086 ceph-mon[34297]: osdmap e28: 4 total, 4 up, 4 in 2024-05-12T22:02:05.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:04 smithi186 ceph-mon[34785]: pgmap v116: 1 pgs: 1 peering; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-05-12T22:02:05.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:04 smithi186 ceph-mon[34785]: osdmap e28: 4 total, 4 up, 4 in 2024-05-12T22:02:05.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:04 smithi028 ceph-mon[29847]: pgmap v116: 1 pgs: 1 peering; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-05-12T22:02:05.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:04 smithi028 ceph-mon[29847]: osdmap e28: 4 total, 4 up, 4 in 2024-05-12T22:02:06.411 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:02:06.408+0000 7f03905bc700 1 -- 172.21.15.86:0/3141002908 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0388102380 msgr2=0x7f03881027a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:02:06.411 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:02:06.408+0000 7f03905bc700 1 --2- 172.21.15.86:0/3141002908 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0388102380 0x7f03881027a0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f03780099c0 tx=0x7f0378009cd0 comp rx=0 tx=0).stop 2024-05-12T22:02:06.411 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:02:06.409+0000 7f03905bc700 1 -- 172.21.15.86:0/3141002908 shutdown_connections 2024-05-12T22:02:06.411 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:02:06.409+0000 7f03905bc700 1 --2- 172.21.15.86:0/3141002908 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0388103f20 0x7f038810a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:02:06.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:02:06.409+0000 7f03905bc700 1 --2- 172.21.15.86:0/3141002908 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0388102380 0x7f03881027a0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:02:06.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:02:06.409+0000 7f03905bc700 1 --2- 172.21.15.86:0/3141002908 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0388103580 0x7f03881039e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:02:06.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:02:06.409+0000 7f03905bc700 1 -- 172.21.15.86:0/3141002908 >> 172.21.15.86:0/3141002908 conn(0x7f03880fd920 msgr2=0x7f03880ffd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:02:06.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:02:06.409+0000 7f03905bc700 1 -- 172.21.15.86:0/3141002908 shutdown_connections 2024-05-12T22:02:06.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:02:06.409+0000 7f03905bc700 1 -- 172.21.15.86:0/3141002908 wait complete. 2024-05-12T22:02:06.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:02:06.409+0000 7f03905bc700 1 Processor -- start 2024-05-12T22:02:06.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:02:06.409+0000 7f03905bc700 1 -- start start 2024-05-12T22:02:06.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:02:06.409+0000 7f03905bc700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0388102380 0x7f03880760c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:02:06.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:02:06.409+0000 7f03905bc700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0388103580 0x7f0388074710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:02:06.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:02:06.409+0000 7f03905bc700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0388103f20 0x7f0388074c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:02:06.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:02:06.409+0000 7f03905bc700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f038810e040 con 0x7f0388102380 2024-05-12T22:02:06.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:02:06.409+0000 7f03905bc700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f038810dec0 con 0x7f0388103580 2024-05-12T22:02:06.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:02:06.409+0000 7f03905bc700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f038810e1c0 con 0x7f0388103f20 2024-05-12T22:02:06.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:02:06.410+0000 7f038db57700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0388103580 0x7f0388074710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:02:06.413 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:02:06.410+0000 7f038db57700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0388103580 0x7f0388074710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:37784/0 (socket says 172.21.15.86:37784) 2024-05-12T22:02:06.413 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:02:06.410+0000 7f038db57700 1 -- 172.21.15.86:0/6910101 learned_addr learned my addr 172.21.15.86:0/6910101 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-05-12T22:02:06.413 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:02:06.410+0000 7f038eb59700 1 --2- 172.21.15.86:0/6910101 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0388103f20 0x7f0388074c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:02:06.413 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:02:06.410+0000 7f038db57700 1 -- 172.21.15.86:0/6910101 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0388103f20 msgr2=0x7f0388074c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:02:06.413 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:02:06.410+0000 7f038db57700 1 --2- 172.21.15.86:0/6910101 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0388103f20 0x7f0388074c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:02:06.413 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:02:06.410+0000 7f038db57700 1 -- 172.21.15.86:0/6910101 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0388102380 msgr2=0x7f03880760c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:02:06.413 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:02:06.410+0000 7f038db57700 1 --2- 172.21.15.86:0/6910101 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0388102380 0x7f03880760c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:02:06.413 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:02:06.410+0000 7f038db57700 1 -- 172.21.15.86:0/6910101 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0388075500 con 0x7f0388103580 2024-05-12T22:02:06.413 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:02:06.410+0000 7f038eb59700 1 --2- 172.21.15.86:0/6910101 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0388103f20 0x7f0388074c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-12T22:02:06.413 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:02:06.410+0000 7f038db57700 1 --2- 172.21.15.86:0/6910101 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0388103580 0x7f0388074710 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f038400b530 tx=0x7f038400b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:02:06.413 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:02:06.410+0000 7f037f7fe700 1 -- 172.21.15.86:0/6910101 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0384015020 con 0x7f0388103580 2024-05-12T22:02:06.413 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:02:06.410+0000 7f037f7fe700 1 -- 172.21.15.86:0/6910101 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0384011d10 con 0x7f0388103580 2024-05-12T22:02:06.413 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:02:06.410+0000 7f03905bc700 1 -- 172.21.15.86:0/6910101 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03881a28a0 con 0x7f0388103580 2024-05-12T22:02:06.413 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:02:06.410+0000 7f037f7fe700 1 -- 172.21.15.86:0/6910101 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0384010e50 con 0x7f0388103580 2024-05-12T22:02:06.413 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:02:06.410+0000 7f03905bc700 1 -- 172.21.15.86:0/6910101 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03881a2de0 con 0x7f0388103580 2024-05-12T22:02:06.413 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:02:06.411+0000 7f03905bc700 1 -- 172.21.15.86:0/6910101 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0388069010 con 0x7f0388103580 2024-05-12T22:02:06.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:02:06.412+0000 7f037f7fe700 1 -- 172.21.15.86:0/6910101 <== mon.2 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f0384018020 con 0x7f0388103580 2024-05-12T22:02:06.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:02:06.413+0000 7f037f7fe700 1 --2- 172.21.15.86:0/6910101 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f03740a2dc0 0x7f03740a5280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:02:06.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:02:06.413+0000 7f037f7fe700 1 -- 172.21.15.86:0/6910101 <== mon.2 v2:172.21.15.86:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 3669+0+0 (secure 0 0 0) 0x7f03840cdc40 con 0x7f0388103580 2024-05-12T22:02:06.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:02:06.413+0000 7f038e358700 1 --2- 172.21.15.86:0/6910101 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f03740a2dc0 0x7f03740a5280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:02:06.415 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:02:06.414+0000 7f038e358700 1 --2- 172.21.15.86:0/6910101 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f03740a2dc0 0x7f03740a5280 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f0378003890 tx=0x7f0378000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:02:06.416 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:02:06.416+0000 7f037f7fe700 1 -- 172.21.15.86:0/6910101 <== mon.2 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f038401f020 con 0x7f0388103580 2024-05-12T22:02:06.623 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:02:06.623+0000 7f03905bc700 1 -- 172.21.15.86:0/6910101 --> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f0388062760 con 0x7f03740a2dc0 2024-05-12T22:02:06.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:06 smithi186 ceph-mon[34785]: osdmap e29: 4 total, 4 up, 4 in 2024-05-12T22:02:06.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:06 smithi186 ceph-mon[34785]: pgmap v119: 1 pgs: 1 peering; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-05-12T22:02:06.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:06 smithi086 ceph-mon[34297]: osdmap e29: 4 total, 4 up, 4 in 2024-05-12T22:02:06.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:06 smithi086 ceph-mon[34297]: pgmap v119: 1 pgs: 1 peering; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-05-12T22:02:07.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:06 smithi028 ceph-mon[29847]: osdmap e29: 4 total, 4 up, 4 in 2024-05-12T22:02:07.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:06 smithi028 ceph-mon[29847]: pgmap v119: 1 pgs: 1 peering; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-05-12T22:02:07.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:07 smithi186 ceph-mon[34785]: osdmap e30: 4 total, 4 up, 4 in 2024-05-12T22:02:07.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:07 smithi186 ceph-mon[34785]: from='client.24224 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-12T22:02:07.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:07 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-12T22:02:07.930 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:07 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-12T22:02:07.930 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:07 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:02:07.930 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:07 smithi186 ceph-mon[34785]: Detected new or changed devices on smithi028 2024-05-12T22:02:07.930 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:07 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:02:07.930 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:07 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:02:07.930 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:07 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:02:07.930 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:07 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:02:07.930 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:07 smithi186 ceph-mon[34785]: Adjusting osd_memory_target on smithi028 to 8547M 2024-05-12T22:02:07.930 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:07 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:02:07.930 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:07 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:02:07.930 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:07 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:02:07.931 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:07 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:02:07.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:07 smithi086 ceph-mon[34297]: osdmap e30: 4 total, 4 up, 4 in 2024-05-12T22:02:07.993 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:07 smithi086 ceph-mon[34297]: from='client.24224 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-12T22:02:07.993 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:07 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-12T22:02:07.993 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:07 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-12T22:02:07.993 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:07 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:02:07.993 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:07 smithi086 ceph-mon[34297]: Detected new or changed devices on smithi028 2024-05-12T22:02:07.993 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:07 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:02:07.993 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:07 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:02:07.993 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:07 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:02:07.993 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:07 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:02:07.993 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:07 smithi086 ceph-mon[34297]: Adjusting osd_memory_target on smithi028 to 8547M 2024-05-12T22:02:07.993 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:07 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:02:07.993 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:07 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:02:07.994 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:07 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:02:07.994 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:07 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:02:08.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:07 smithi028 ceph-mon[29847]: osdmap e30: 4 total, 4 up, 4 in 2024-05-12T22:02:08.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:07 smithi028 ceph-mon[29847]: from='client.24224 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-12T22:02:08.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:07 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-12T22:02:08.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:07 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-12T22:02:08.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:07 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:02:08.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:07 smithi028 ceph-mon[29847]: Detected new or changed devices on smithi028 2024-05-12T22:02:08.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:07 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:02:08.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:07 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:02:08.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:07 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:02:08.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:07 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:02:08.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:07 smithi028 ceph-mon[29847]: Adjusting osd_memory_target on smithi028 to 8547M 2024-05-12T22:02:08.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:07 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:02:08.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:07 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:02:08.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:07 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:02:08.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:07 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:02:08.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:08 smithi086 ceph-mon[34297]: pgmap v121: 1 pgs: 1 peering; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-05-12T22:02:09.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:08 smithi028 ceph-mon[29847]: pgmap v121: 1 pgs: 1 peering; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-05-12T22:02:09.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:08 smithi186 ceph-mon[34785]: pgmap v121: 1 pgs: 1 peering; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-05-12T22:02:09.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:09 smithi086 ceph-mon[34297]: from='client.? 172.21.15.86:0/4208355206' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d162ebcd-83c0-4fd5-9925-28b2cecea935"}]: dispatch 2024-05-12T22:02:09.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:09 smithi086 ceph-mon[34297]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d162ebcd-83c0-4fd5-9925-28b2cecea935"}]: dispatch 2024-05-12T22:02:09.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:09 smithi086 ceph-mon[34297]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d162ebcd-83c0-4fd5-9925-28b2cecea935"}]': finished 2024-05-12T22:02:09.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:09 smithi086 ceph-mon[34297]: osdmap e31: 5 total, 4 up, 5 in 2024-05-12T22:02:09.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:09 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-12T22:02:10.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:09 smithi028 ceph-mon[29847]: from='client.? 172.21.15.86:0/4208355206' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d162ebcd-83c0-4fd5-9925-28b2cecea935"}]: dispatch 2024-05-12T22:02:10.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:09 smithi028 ceph-mon[29847]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d162ebcd-83c0-4fd5-9925-28b2cecea935"}]: dispatch 2024-05-12T22:02:10.037 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:09 smithi028 ceph-mon[29847]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d162ebcd-83c0-4fd5-9925-28b2cecea935"}]': finished 2024-05-12T22:02:10.038 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:09 smithi028 ceph-mon[29847]: osdmap e31: 5 total, 4 up, 5 in 2024-05-12T22:02:10.038 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:09 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-12T22:02:10.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:09 smithi186 ceph-mon[34785]: from='client.? 172.21.15.86:0/4208355206' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d162ebcd-83c0-4fd5-9925-28b2cecea935"}]: dispatch 2024-05-12T22:02:10.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:09 smithi186 ceph-mon[34785]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d162ebcd-83c0-4fd5-9925-28b2cecea935"}]: dispatch 2024-05-12T22:02:10.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:09 smithi186 ceph-mon[34785]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d162ebcd-83c0-4fd5-9925-28b2cecea935"}]': finished 2024-05-12T22:02:10.180 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:09 smithi186 ceph-mon[34785]: osdmap e31: 5 total, 4 up, 5 in 2024-05-12T22:02:10.180 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:09 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-12T22:02:10.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:10 smithi086 ceph-mon[34297]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 115 KiB/s, 0 objects/s recovering 2024-05-12T22:02:10.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:10 smithi086 ceph-mon[34297]: from='client.? 172.21.15.86:0/3165837504' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-12T22:02:11.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:10 smithi028 ceph-mon[29847]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 115 KiB/s, 0 objects/s recovering 2024-05-12T22:02:11.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:10 smithi028 ceph-mon[29847]: from='client.? 172.21.15.86:0/3165837504' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-12T22:02:11.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:10 smithi186 ceph-mon[34785]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 115 KiB/s, 0 objects/s recovering 2024-05-12T22:02:11.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:10 smithi186 ceph-mon[34785]: from='client.? 172.21.15.86:0/3165837504' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-12T22:02:12.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:12 smithi086 ceph-mon[34297]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-05-12T22:02:13.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:12 smithi028 ceph-mon[29847]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-05-12T22:02:13.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:12 smithi186 ceph-mon[34785]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-05-12T22:02:14.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:14 smithi086 ceph-mon[34297]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-05-12T22:02:15.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:14 smithi028 ceph-mon[29847]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-05-12T22:02:15.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:14 smithi186 ceph-mon[34785]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-05-12T22:02:16.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:16 smithi086 ceph-mon[34297]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-05-12T22:02:16.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:16 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-05-12T22:02:16.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:16 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:02:17.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:16 smithi028 ceph-mon[29847]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-05-12T22:02:17.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:16 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-05-12T22:02:17.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:16 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:02:17.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:16 smithi186 ceph-mon[34785]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-05-12T22:02:17.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:16 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-05-12T22:02:17.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:16 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:02:17.991 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:17 smithi086 ceph-mon[34297]: Deploying daemon osd.4 on smithi086 2024-05-12T22:02:18.037 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:17 smithi028 ceph-mon[29847]: Deploying daemon osd.4 on smithi086 2024-05-12T22:02:18.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:17 smithi186 ceph-mon[34785]: Deploying daemon osd.4 on smithi086 2024-05-12T22:02:18.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:18 smithi086 ceph-mon[34297]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-12T22:02:19.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:18 smithi028 ceph-mon[29847]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-12T22:02:19.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:18 smithi186 ceph-mon[34785]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-12T22:02:21.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:20 smithi028 ceph-mon[29847]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-05-12T22:02:21.037 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:20 smithi086 ceph-mon[34297]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-05-12T22:02:21.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:20 smithi186 ceph-mon[34785]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-05-12T22:02:22.020 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:21 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:02:22.020 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:21 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:02:22.020 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:21 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:02:22.034 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:21 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:02:22.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:21 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:02:22.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:21 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:02:22.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:21 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:02:22.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:21 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:02:22.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:21 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:02:22.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:22 smithi086 ceph-mon[34297]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-05-12T22:02:23.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:22 smithi028 ceph-mon[29847]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-05-12T22:02:23.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:22 smithi186 ceph-mon[34785]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-05-12T22:02:25.224 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:02:25.224+0000 7f037f7fe700 1 -- 172.21.15.86:0/6910101 <== mgr.14152 v2:172.21.15.28:6800/3123328651 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f0388062760 con 0x7f03740a2dc0 2024-05-12T22:02:25.225 INFO:teuthology.orchestra.run.smithi086.stdout:Created osd(s) 4 on host 'smithi086' 2024-05-12T22:02:25.227 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:02:25.226+0000 7f03905bc700 1 -- 172.21.15.86:0/6910101 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f03740a2dc0 msgr2=0x7f03740a5280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:02:25.227 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:02:25.226+0000 7f03905bc700 1 --2- 172.21.15.86:0/6910101 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f03740a2dc0 0x7f03740a5280 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f0378003890 tx=0x7f0378000bc0 comp rx=0 tx=0).stop 2024-05-12T22:02:25.227 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:02:25.226+0000 7f03905bc700 1 -- 172.21.15.86:0/6910101 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0388103580 msgr2=0x7f0388074710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:02:25.227 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:02:25.226+0000 7f03905bc700 1 --2- 172.21.15.86:0/6910101 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0388103580 0x7f0388074710 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f038400b530 tx=0x7f038400b8f0 comp rx=0 tx=0).stop 2024-05-12T22:02:25.227 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:02:25.227+0000 7f03905bc700 1 -- 172.21.15.86:0/6910101 shutdown_connections 2024-05-12T22:02:25.227 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:02:25.227+0000 7f03905bc700 1 --2- 172.21.15.86:0/6910101 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f03740a2dc0 0x7f03740a5280 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:02:25.227 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:02:25.227+0000 7f03905bc700 1 --2- 172.21.15.86:0/6910101 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0388103f20 0x7f0388074c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:02:25.227 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:02:25.227+0000 7f03905bc700 1 --2- 172.21.15.86:0/6910101 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0388102380 0x7f03880760c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:02:25.227 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:02:25.227+0000 7f03905bc700 1 --2- 172.21.15.86:0/6910101 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0388103580 0x7f0388074710 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:02:25.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:02:25.227+0000 7f03905bc700 1 -- 172.21.15.86:0/6910101 >> 172.21.15.86:0/6910101 conn(0x7f03880fd920 msgr2=0x7f03881069c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:02:25.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:02:25.227+0000 7f03905bc700 1 -- 172.21.15.86:0/6910101 shutdown_connections 2024-05-12T22:02:25.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:02:25.227+0000 7f03905bc700 1 -- 172.21.15.86:0/6910101 wait complete. 2024-05-12T22:02:25.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:25 smithi086 ceph-mon[34297]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-05-12T22:02:25.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:25 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:02:25.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:25 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:02:25.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:25 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:02:25.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:25 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:02:25.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:25 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:02:25.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:25 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:02:25.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:25 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:02:25.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:25 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:02:25.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:25 smithi186 ceph-mon[34785]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-05-12T22:02:25.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:25 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:02:25.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:25 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:02:25.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:25 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:02:25.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:25 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:02:25.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:25 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:02:25.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:25 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:02:25.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:25 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:02:25.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:25 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:02:25.784 DEBUG:teuthology.orchestra.run.smithi086:osd.4> sudo journalctl -f -n 0 -u ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968@osd.4.service 2024-05-12T22:02:25.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:25 smithi028 ceph-mon[29847]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-05-12T22:02:25.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:25 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:02:25.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:25 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:02:25.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:25 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:02:25.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:25 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:02:25.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:25 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:02:25.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:25 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:02:25.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:25 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:02:25.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:25 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:02:25.787 INFO:tasks.cephadm:Deploying osd.5 on smithi186 with /dev/vg_nvme/lv_3... 2024-05-12T22:02:25.787 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f72fecff68e1d400c4568684327c900485c20d6a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6b92e7b4-10aa-11ef-bc98-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-05-12T22:02:26.035 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/mon.c/config 2024-05-12T22:02:26.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:26 smithi186 ceph-mon[34785]: from='osd.4 [v2:172.21.15.86:6808/2596998708,v1:172.21.15.86:6809/2596998708]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-12T22:02:26.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:26 smithi186 ceph-mon[34785]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-12T22:02:26.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:26 smithi028 ceph-mon[29847]: from='osd.4 [v2:172.21.15.86:6808/2596998708,v1:172.21.15.86:6809/2596998708]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-12T22:02:26.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:26 smithi028 ceph-mon[29847]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-12T22:02:26.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:26 smithi086 ceph-mon[34297]: from='osd.4 [v2:172.21.15.86:6808/2596998708,v1:172.21.15.86:6809/2596998708]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-12T22:02:26.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:26 smithi086 ceph-mon[34297]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-12T22:02:27.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:27 smithi186 ceph-mon[34785]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-05-12T22:02:27.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:27 smithi186 ceph-mon[34785]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-05-12T22:02:27.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:27 smithi186 ceph-mon[34785]: from='osd.4 [v2:172.21.15.86:6808/2596998708,v1:172.21.15.86:6809/2596998708]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-05-12T22:02:27.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:27 smithi186 ceph-mon[34785]: osdmap e32: 5 total, 4 up, 5 in 2024-05-12T22:02:27.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:27 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-12T22:02:27.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:27 smithi186 ceph-mon[34785]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-05-12T22:02:27.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:27 smithi086 ceph-mon[34297]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-05-12T22:02:27.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:27 smithi086 ceph-mon[34297]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-05-12T22:02:27.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:27 smithi086 ceph-mon[34297]: from='osd.4 [v2:172.21.15.86:6808/2596998708,v1:172.21.15.86:6809/2596998708]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-05-12T22:02:27.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:27 smithi086 ceph-mon[34297]: osdmap e32: 5 total, 4 up, 5 in 2024-05-12T22:02:27.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:27 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-12T22:02:27.743 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:27 smithi086 ceph-mon[34297]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-05-12T22:02:27.743 INFO:journalctl@ceph.osd.4.smithi086.stdout:May 12 22:02:27 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-4[45634]: 2024-05-12T22:02:27.393+0000 7f74ec306700 -1 osd.4 0 waiting for initial osdmap 2024-05-12T22:02:27.743 INFO:journalctl@ceph.osd.4.smithi086.stdout:May 12 22:02:27 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-4[45634]: 2024-05-12T22:02:27.401+0000 7f74e88fa700 -1 osd.4 33 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-12T22:02:27.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:27 smithi028 ceph-mon[29847]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-05-12T22:02:27.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:27 smithi028 ceph-mon[29847]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-05-12T22:02:27.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:27 smithi028 ceph-mon[29847]: from='osd.4 [v2:172.21.15.86:6808/2596998708,v1:172.21.15.86:6809/2596998708]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-05-12T22:02:27.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:27 smithi028 ceph-mon[29847]: osdmap e32: 5 total, 4 up, 5 in 2024-05-12T22:02:27.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:27 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-12T22:02:27.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:27 smithi028 ceph-mon[29847]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-05-12T22:02:27.907 INFO:teuthology.orchestra.run.smithi186.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-05-12T22:02:27.907 INFO:teuthology.orchestra.run.smithi186.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-05-12T22:02:27.907 INFO:teuthology.orchestra.run.smithi186.stderr: stderr: 10+0 records in 2024-05-12T22:02:27.908 INFO:teuthology.orchestra.run.smithi186.stderr:10+0 records out 2024-05-12T22:02:27.908 INFO:teuthology.orchestra.run.smithi186.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0115781 s, 906 MB/s 2024-05-12T22:02:27.908 INFO:teuthology.orchestra.run.smithi186.stderr:--> Zapping successful for: 2024-05-12T22:02:28.438 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:28 smithi086 ceph-mon[34297]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]': finished 2024-05-12T22:02:28.438 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:28 smithi086 ceph-mon[34297]: osdmap e33: 5 total, 4 up, 5 in 2024-05-12T22:02:28.438 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:28 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-12T22:02:28.457 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f72fecff68e1d400c4568684327c900485c20d6a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6b92e7b4-10aa-11ef-bc98-c7b262605968 -- ceph orch daemon add osd smithi186:vg_nvme/lv_3 2024-05-12T22:02:28.470 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:28 smithi186 ceph-mon[34785]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]': finished 2024-05-12T22:02:28.470 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:28 smithi186 ceph-mon[34785]: osdmap e33: 5 total, 4 up, 5 in 2024-05-12T22:02:28.470 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:28 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-12T22:02:28.698 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/mon.c/config 2024-05-12T22:02:28.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:28 smithi028 ceph-mon[29847]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]': finished 2024-05-12T22:02:28.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:28 smithi028 ceph-mon[29847]: osdmap e33: 5 total, 4 up, 5 in 2024-05-12T22:02:28.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:28 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-12T22:02:29.663 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:29 smithi186 ceph-mon[34785]: purged_snaps scrub starts 2024-05-12T22:02:29.663 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:29 smithi186 ceph-mon[34785]: purged_snaps scrub ok 2024-05-12T22:02:29.663 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:29 smithi186 ceph-mon[34785]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-05-12T22:02:29.663 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:29 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-12T22:02:29.663 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:29 smithi186 ceph-mon[34785]: osd.4 [v2:172.21.15.86:6808/2596998708,v1:172.21.15.86:6809/2596998708] boot 2024-05-12T22:02:29.663 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:29 smithi186 ceph-mon[34785]: osdmap e34: 5 total, 5 up, 5 in 2024-05-12T22:02:29.663 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:29 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-12T22:02:29.663 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:29 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:02:29.663 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:29 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:02:29.663 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:29 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:02:29.663 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:29 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:02:29.663 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:29 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:02:29.663 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:29 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:02:29.664 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:29 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:02:29.664 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:29 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:02:29.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:29 smithi086 ceph-mon[34297]: purged_snaps scrub starts 2024-05-12T22:02:29.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:29 smithi086 ceph-mon[34297]: purged_snaps scrub ok 2024-05-12T22:02:29.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:29 smithi086 ceph-mon[34297]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-05-12T22:02:29.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:29 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-12T22:02:29.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:29 smithi086 ceph-mon[34297]: osd.4 [v2:172.21.15.86:6808/2596998708,v1:172.21.15.86:6809/2596998708] boot 2024-05-12T22:02:29.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:29 smithi086 ceph-mon[34297]: osdmap e34: 5 total, 5 up, 5 in 2024-05-12T22:02:29.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:29 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-12T22:02:29.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:29 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:02:29.743 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:29 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:02:29.743 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:29 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:02:29.743 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:29 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:02:29.743 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:29 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:02:29.743 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:29 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:02:29.743 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:29 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:02:29.743 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:29 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:02:29.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:29 smithi028 ceph-mon[29847]: purged_snaps scrub starts 2024-05-12T22:02:29.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:29 smithi028 ceph-mon[29847]: purged_snaps scrub ok 2024-05-12T22:02:29.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:29 smithi028 ceph-mon[29847]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-05-12T22:02:29.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:29 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-12T22:02:29.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:29 smithi028 ceph-mon[29847]: osd.4 [v2:172.21.15.86:6808/2596998708,v1:172.21.15.86:6809/2596998708] boot 2024-05-12T22:02:29.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:29 smithi028 ceph-mon[29847]: osdmap e34: 5 total, 5 up, 5 in 2024-05-12T22:02:29.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:29 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-12T22:02:29.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:29 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:02:29.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:29 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:02:29.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:29 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:02:29.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:29 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:02:29.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:29 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:02:29.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:29 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:02:29.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:29 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:02:29.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:29 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:02:30.171 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:02:30.169+0000 7f86a83a7700 1 -- 172.21.15.186:0/3881075756 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86a0103f40 msgr2=0x7f86a010a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:02:30.171 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:02:30.169+0000 7f86a83a7700 1 --2- 172.21.15.186:0/3881075756 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86a0103f40 0x7f86a010a7f0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f869800b0a0 tx=0x7f869800b3b0 comp rx=0 tx=0).stop 2024-05-12T22:02:30.171 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:02:30.169+0000 7f86a83a7700 1 -- 172.21.15.186:0/3881075756 shutdown_connections 2024-05-12T22:02:30.171 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:02:30.169+0000 7f86a83a7700 1 --2- 172.21.15.186:0/3881075756 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86a0103f40 0x7f86a010a7f0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:02:30.171 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:02:30.169+0000 7f86a83a7700 1 --2- 172.21.15.186:0/3881075756 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f86a01023a0 0x7f86a01027c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:02:30.171 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:02:30.169+0000 7f86a83a7700 1 --2- 172.21.15.186:0/3881075756 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86a01035a0 0x7f86a0103a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:02:30.171 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:02:30.169+0000 7f86a83a7700 1 -- 172.21.15.186:0/3881075756 >> 172.21.15.186:0/3881075756 conn(0x7f86a00fd940 msgr2=0x7f86a00ffd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:02:30.171 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:02:30.169+0000 7f86a83a7700 1 -- 172.21.15.186:0/3881075756 shutdown_connections 2024-05-12T22:02:30.172 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:02:30.169+0000 7f86a83a7700 1 -- 172.21.15.186:0/3881075756 wait complete. 2024-05-12T22:02:30.172 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:02:30.170+0000 7f86a83a7700 1 Processor -- start 2024-05-12T22:02:30.172 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:02:30.170+0000 7f86a83a7700 1 -- start start 2024-05-12T22:02:30.172 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:02:30.170+0000 7f86a83a7700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f86a01023a0 0x7f86a00760c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:02:30.172 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:02:30.170+0000 7f86a83a7700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86a01035a0 0x7f86a0074710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:02:30.172 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:02:30.170+0000 7f86a83a7700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86a0103f40 0x7f86a0074c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:02:30.172 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:02:30.170+0000 7f86a83a7700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f86a010e080 con 0x7f86a01035a0 2024-05-12T22:02:30.172 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:02:30.170+0000 7f86a83a7700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f86a010df00 con 0x7f86a01023a0 2024-05-12T22:02:30.172 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:02:30.170+0000 7f86a83a7700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f86a010e200 con 0x7f86a0103f40 2024-05-12T22:02:30.172 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:02:30.170+0000 7f86a6944700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86a0103f40 0x7f86a0074c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:02:30.172 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:02:30.170+0000 7f86a6944700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86a0103f40 0x7f86a0074c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:34644/0 (socket says 172.21.15.186:34644) 2024-05-12T22:02:30.172 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:02:30.170+0000 7f86a6944700 1 -- 172.21.15.186:0/1592643311 learned_addr learned my addr 172.21.15.186:0/1592643311 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-05-12T22:02:30.172 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:02:30.171+0000 7f86a6143700 1 --2- 172.21.15.186:0/1592643311 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f86a01023a0 0x7f86a00760c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:02:30.172 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:02:30.171+0000 7f86a6944700 1 -- 172.21.15.186:0/1592643311 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f86a01023a0 msgr2=0x7f86a00760c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:02:30.173 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:02:30.171+0000 7f86a6944700 1 --2- 172.21.15.186:0/1592643311 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f86a01023a0 0x7f86a00760c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:02:30.173 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:02:30.171+0000 7f86a6944700 1 -- 172.21.15.186:0/1592643311 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86a01035a0 msgr2=0x7f86a0074710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:02:30.173 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:02:30.171+0000 7f86a6944700 1 --2- 172.21.15.186:0/1592643311 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86a01035a0 0x7f86a0074710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:02:30.173 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:02:30.171+0000 7f86a6944700 1 -- 172.21.15.186:0/1592643311 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86a0075190 con 0x7f86a0103f40 2024-05-12T22:02:30.173 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:02:30.171+0000 7f86a6944700 1 --2- 172.21.15.186:0/1592643311 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86a0103f40 0x7f86a0074c50 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f869800b070 tx=0x7f8698007770 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:02:30.173 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:02:30.171+0000 7f86977fe700 1 -- 172.21.15.186:0/1592643311 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8698021020 con 0x7f86a0103f40 2024-05-12T22:02:30.173 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:02:30.171+0000 7f86977fe700 1 -- 172.21.15.186:0/1592643311 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8698005520 con 0x7f86a0103f40 2024-05-12T22:02:30.173 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:02:30.171+0000 7f86a83a7700 1 -- 172.21.15.186:0/1592643311 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86a0075420 con 0x7f86a0103f40 2024-05-12T22:02:30.173 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:02:30.171+0000 7f86977fe700 1 -- 172.21.15.186:0/1592643311 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f869801ba70 con 0x7f86a0103f40 2024-05-12T22:02:30.173 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:02:30.171+0000 7f86a83a7700 1 -- 172.21.15.186:0/1592643311 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86a01a7140 con 0x7f86a0103f40 2024-05-12T22:02:30.174 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:02:30.172+0000 7f86a83a7700 1 -- 172.21.15.186:0/1592643311 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f86a002db80 con 0x7f86a0103f40 2024-05-12T22:02:30.174 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:02:30.173+0000 7f86977fe700 1 -- 172.21.15.186:0/1592643311 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f8698024020 con 0x7f86a0103f40 2024-05-12T22:02:30.174 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:02:30.174+0000 7f86977fe700 1 --2- 172.21.15.186:0/1592643311 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f868c0a2ca0 0x7f868c0a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:02:30.174 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:02:30.174+0000 7f86977fe700 1 -- 172.21.15.186:0/1592643311 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 4101+0+0 (secure 0 0 0) 0x7f869800f020 con 0x7f86a0103f40 2024-05-12T22:02:30.174 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:02:30.174+0000 7f86a6143700 1 --2- 172.21.15.186:0/1592643311 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f868c0a2ca0 0x7f868c0a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:02:30.175 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:02:30.175+0000 7f86a6143700 1 --2- 172.21.15.186:0/1592643311 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f868c0a2ca0 0x7f868c0a5160 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f8690006fd0 tx=0x7f8690008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:02:30.178 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:02:30.178+0000 7f86977fe700 1 -- 172.21.15.186:0/1592643311 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f869809d240 con 0x7f86a0103f40 2024-05-12T22:02:30.379 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:02:30.380+0000 7f86a83a7700 1 -- 172.21.15.186:0/1592643311 --> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f86a0062760 con 0x7f868c0a2ca0 2024-05-12T22:02:30.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:30 smithi186 ceph-mon[34785]: Detected new or changed devices on smithi086 2024-05-12T22:02:30.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:30 smithi186 ceph-mon[34785]: Adjusting osd_memory_target on smithi086 to 8544M 2024-05-12T22:02:30.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:30 smithi186 ceph-mon[34785]: osdmap e35: 5 total, 5 up, 5 in 2024-05-12T22:02:30.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:30 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-12T22:02:30.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:30 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-12T22:02:30.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:30 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:02:30.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:30 smithi086 ceph-mon[34297]: Detected new or changed devices on smithi086 2024-05-12T22:02:30.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:30 smithi086 ceph-mon[34297]: Adjusting osd_memory_target on smithi086 to 8544M 2024-05-12T22:02:30.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:30 smithi086 ceph-mon[34297]: osdmap e35: 5 total, 5 up, 5 in 2024-05-12T22:02:30.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:30 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-12T22:02:30.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:30 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-12T22:02:30.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:30 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:02:30.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:30 smithi028 ceph-mon[29847]: Detected new or changed devices on smithi086 2024-05-12T22:02:30.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:30 smithi028 ceph-mon[29847]: Adjusting osd_memory_target on smithi086 to 8544M 2024-05-12T22:02:30.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:30 smithi028 ceph-mon[29847]: osdmap e35: 5 total, 5 up, 5 in 2024-05-12T22:02:30.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:30 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-12T22:02:30.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:30 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-12T22:02:30.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:30 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:02:31.655 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:31 smithi186 ceph-mon[34785]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-05-12T22:02:31.655 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:31 smithi186 ceph-mon[34785]: from='client.24235 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-12T22:02:31.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:31 smithi086 ceph-mon[34297]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-05-12T22:02:31.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:31 smithi086 ceph-mon[34297]: from='client.24235 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-12T22:02:31.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:31 smithi028 ceph-mon[29847]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-05-12T22:02:31.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:31 smithi028 ceph-mon[29847]: from='client.24235 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-12T22:02:33.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:33 smithi186 ceph-mon[34785]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-05-12T22:02:33.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:33 smithi186 ceph-mon[34785]: from='client.? 172.21.15.186:0/4169935683' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c794336a-a105-425c-b6c7-a32b90c05bb8"}]: dispatch 2024-05-12T22:02:33.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:33 smithi186 ceph-mon[34785]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c794336a-a105-425c-b6c7-a32b90c05bb8"}]: dispatch 2024-05-12T22:02:33.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:33 smithi186 ceph-mon[34785]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c794336a-a105-425c-b6c7-a32b90c05bb8"}]': finished 2024-05-12T22:02:33.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:33 smithi186 ceph-mon[34785]: osdmap e36: 6 total, 5 up, 6 in 2024-05-12T22:02:33.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:33 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-12T22:02:33.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:33 smithi086 ceph-mon[34297]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-05-12T22:02:33.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:33 smithi086 ceph-mon[34297]: from='client.? 172.21.15.186:0/4169935683' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c794336a-a105-425c-b6c7-a32b90c05bb8"}]: dispatch 2024-05-12T22:02:33.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:33 smithi086 ceph-mon[34297]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c794336a-a105-425c-b6c7-a32b90c05bb8"}]: dispatch 2024-05-12T22:02:33.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:33 smithi086 ceph-mon[34297]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c794336a-a105-425c-b6c7-a32b90c05bb8"}]': finished 2024-05-12T22:02:33.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:33 smithi086 ceph-mon[34297]: osdmap e36: 6 total, 5 up, 6 in 2024-05-12T22:02:33.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:33 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-12T22:02:33.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:33 smithi028 ceph-mon[29847]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-05-12T22:02:33.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:33 smithi028 ceph-mon[29847]: from='client.? 172.21.15.186:0/4169935683' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c794336a-a105-425c-b6c7-a32b90c05bb8"}]: dispatch 2024-05-12T22:02:33.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:33 smithi028 ceph-mon[29847]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c794336a-a105-425c-b6c7-a32b90c05bb8"}]: dispatch 2024-05-12T22:02:33.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:33 smithi028 ceph-mon[29847]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c794336a-a105-425c-b6c7-a32b90c05bb8"}]': finished 2024-05-12T22:02:33.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:33 smithi028 ceph-mon[29847]: osdmap e36: 6 total, 5 up, 6 in 2024-05-12T22:02:33.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:33 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-12T22:02:34.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:34 smithi086 ceph-mon[34297]: from='client.? 172.21.15.186:0/2014208866' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-12T22:02:34.784 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:34 smithi028 ceph-mon[29847]: from='client.? 172.21.15.186:0/2014208866' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-12T22:02:34.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:34 smithi186 ceph-mon[34785]: from='client.? 172.21.15.186:0/2014208866' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-12T22:02:35.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:35 smithi086 ceph-mon[34297]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-05-12T22:02:35.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:35 smithi028 ceph-mon[29847]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-05-12T22:02:35.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:35 smithi186 ceph-mon[34785]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-05-12T22:02:36.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:36 smithi086 ceph-mon[34297]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-05-12T22:02:36.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:36 smithi028 ceph-mon[29847]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-05-12T22:02:36.881 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:36 smithi186 ceph-mon[34785]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-05-12T22:02:38.925 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:38 smithi186 ceph-mon[34785]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-05-12T22:02:38.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:38 smithi086 ceph-mon[34297]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-05-12T22:02:39.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:38 smithi028 ceph-mon[29847]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-05-12T22:02:40.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:40 smithi186 ceph-mon[34785]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-05-12T22:02:40.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:40 smithi086 ceph-mon[34297]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-05-12T22:02:41.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:40 smithi028 ceph-mon[29847]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-05-12T22:02:41.871 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:41 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-05-12T22:02:41.871 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:41 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:02:41.872 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:41 smithi186 ceph-mon[34785]: Deploying daemon osd.5 on smithi186 2024-05-12T22:02:41.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:41 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-05-12T22:02:41.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:41 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:02:41.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:41 smithi086 ceph-mon[34297]: Deploying daemon osd.5 on smithi186 2024-05-12T22:02:42.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:41 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-05-12T22:02:42.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:41 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:02:42.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:41 smithi028 ceph-mon[29847]: Deploying daemon osd.5 on smithi186 2024-05-12T22:02:42.911 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:42 smithi186 ceph-mon[34785]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-05-12T22:02:42.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:42 smithi086 ceph-mon[34297]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-05-12T22:02:43.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:42 smithi028 ceph-mon[29847]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-05-12T22:02:44.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:44 smithi086 ceph-mon[34297]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-05-12T22:02:45.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:44 smithi028 ceph-mon[29847]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-05-12T22:02:45.129 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:44 smithi186 ceph-mon[34785]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-05-12T22:02:47.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:46 smithi186 ceph-mon[34785]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-05-12T22:02:47.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:46 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:02:47.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:46 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:02:47.180 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:46 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:02:47.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:46 smithi086 ceph-mon[34297]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-05-12T22:02:47.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:46 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:02:47.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:46 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:02:47.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:46 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:02:47.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:46 smithi028 ceph-mon[29847]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-05-12T22:02:47.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:46 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:02:47.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:46 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:02:47.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:46 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:02:49.008 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:48 smithi186 ceph-mon[34785]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-05-12T22:02:49.009 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:48 smithi028 ceph-mon[29847]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-05-12T22:02:49.113 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:02:49.113+0000 7f86977fe700 1 -- 172.21.15.186:0/1592643311 <== mgr.14152 v2:172.21.15.28:6800/3123328651 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f86a0062760 con 0x7f868c0a2ca0 2024-05-12T22:02:49.113 INFO:teuthology.orchestra.run.smithi186.stdout:Created osd(s) 5 on host 'smithi186' 2024-05-12T22:02:49.116 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:02:49.116+0000 7f86a83a7700 1 -- 172.21.15.186:0/1592643311 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f868c0a2ca0 msgr2=0x7f868c0a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:02:49.116 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:02:49.116+0000 7f86a83a7700 1 --2- 172.21.15.186:0/1592643311 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f868c0a2ca0 0x7f868c0a5160 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f8690006fd0 tx=0x7f8690008040 comp rx=0 tx=0).stop 2024-05-12T22:02:49.116 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:02:49.116+0000 7f86a83a7700 1 -- 172.21.15.186:0/1592643311 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86a0103f40 msgr2=0x7f86a0074c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:02:49.116 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:02:49.116+0000 7f86a83a7700 1 --2- 172.21.15.186:0/1592643311 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86a0103f40 0x7f86a0074c50 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f869800b070 tx=0x7f8698007770 comp rx=0 tx=0).stop 2024-05-12T22:02:49.116 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:02:49.116+0000 7f86a83a7700 1 -- 172.21.15.186:0/1592643311 shutdown_connections 2024-05-12T22:02:49.116 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:02:49.116+0000 7f86a83a7700 1 --2- 172.21.15.186:0/1592643311 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f868c0a2ca0 0x7f868c0a5160 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:02:49.116 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:02:49.116+0000 7f86a83a7700 1 --2- 172.21.15.186:0/1592643311 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86a0103f40 0x7f86a0074c50 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:02:49.116 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:02:49.116+0000 7f86a83a7700 1 --2- 172.21.15.186:0/1592643311 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f86a01023a0 0x7f86a00760c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:02:49.116 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:02:49.116+0000 7f86a83a7700 1 --2- 172.21.15.186:0/1592643311 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86a01035a0 0x7f86a0074710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:02:49.116 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:02:49.116+0000 7f86a83a7700 1 -- 172.21.15.186:0/1592643311 >> 172.21.15.186:0/1592643311 conn(0x7f86a00fd940 msgr2=0x7f86a01068a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:02:49.116 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:02:49.116+0000 7f86a83a7700 1 -- 172.21.15.186:0/1592643311 shutdown_connections 2024-05-12T22:02:49.116 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:02:49.116+0000 7f86a83a7700 1 -- 172.21.15.186:0/1592643311 wait complete. 2024-05-12T22:02:49.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:48 smithi086 ceph-mon[34297]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-05-12T22:02:49.722 DEBUG:teuthology.orchestra.run.smithi186:osd.5> sudo journalctl -f -n 0 -u ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968@osd.5.service 2024-05-12T22:02:49.765 INFO:tasks.cephadm:Deploying osd.6 on smithi028 with /dev/vg_nvme/lv_2... 2024-05-12T22:02:49.765 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f72fecff68e1d400c4568684327c900485c20d6a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6b92e7b4-10aa-11ef-bc98-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-05-12T22:02:50.016 INFO:journalctl@ceph.osd.5.smithi186.stdout:May 12 22:02:49 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-5[46962]: 2024-05-12T22:02:49.803+0000 7f82f09157c0 -1 osd.5 0 log_to_monitors true 2024-05-12T22:02:50.178 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/mon.a/config 2024-05-12T22:02:50.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:50 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:02:50.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:50 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:02:50.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:50 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:02:50.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:50 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:02:50.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:50 smithi028 ceph-mon[29847]: from='osd.5 [v2:172.21.15.186:6808/1944205756,v1:172.21.15.186:6809/1944205756]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-12T22:02:50.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:50 smithi028 ceph-mon[29847]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-12T22:02:50.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:50 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:02:50.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:50 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:02:50.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:50 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:02:50.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:50 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:02:50.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:50 smithi186 ceph-mon[34785]: from='osd.5 [v2:172.21.15.186:6808/1944205756,v1:172.21.15.186:6809/1944205756]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-12T22:02:50.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:50 smithi186 ceph-mon[34785]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-12T22:02:50.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:50 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:02:50.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:50 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:02:50.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:50 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:02:50.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:50 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:02:50.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:50 smithi086 ceph-mon[34297]: from='osd.5 [v2:172.21.15.186:6808/1944205756,v1:172.21.15.186:6809/1944205756]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-12T22:02:50.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:50 smithi086 ceph-mon[34297]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-12T22:02:51.429 INFO:journalctl@ceph.osd.5.smithi186.stdout:May 12 22:02:51 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-5[46962]: 2024-05-12T22:02:51.130+0000 7f82e5778700 -1 osd.5 0 waiting for initial osdmap 2024-05-12T22:02:51.430 INFO:journalctl@ceph.osd.5.smithi186.stdout:May 12 22:02:51 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-5[46962]: 2024-05-12T22:02:51.137+0000 7f82e1d6c700 -1 osd.5 38 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-12T22:02:51.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:51 smithi186 ceph-mon[34785]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-05-12T22:02:51.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:51 smithi186 ceph-mon[34785]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-05-12T22:02:51.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:51 smithi186 ceph-mon[34785]: osdmap e37: 6 total, 5 up, 6 in 2024-05-12T22:02:51.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:51 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-12T22:02:51.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:51 smithi186 ceph-mon[34785]: from='osd.5 [v2:172.21.15.186:6808/1944205756,v1:172.21.15.186:6809/1944205756]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-05-12T22:02:51.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:51 smithi186 ceph-mon[34785]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-05-12T22:02:51.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:51 smithi086 ceph-mon[34297]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-05-12T22:02:51.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:51 smithi086 ceph-mon[34297]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-05-12T22:02:51.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:51 smithi086 ceph-mon[34297]: osdmap e37: 6 total, 5 up, 6 in 2024-05-12T22:02:51.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:51 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-12T22:02:51.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:51 smithi086 ceph-mon[34297]: from='osd.5 [v2:172.21.15.186:6808/1944205756,v1:172.21.15.186:6809/1944205756]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-05-12T22:02:51.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:51 smithi086 ceph-mon[34297]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-05-12T22:02:51.520 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:51 smithi028 ceph-mon[29847]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-05-12T22:02:51.520 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:51 smithi028 ceph-mon[29847]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-05-12T22:02:51.520 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:51 smithi028 ceph-mon[29847]: osdmap e37: 6 total, 5 up, 6 in 2024-05-12T22:02:51.520 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:51 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-12T22:02:51.520 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:51 smithi028 ceph-mon[29847]: from='osd.5 [v2:172.21.15.186:6808/1944205756,v1:172.21.15.186:6809/1944205756]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-05-12T22:02:51.520 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:51 smithi028 ceph-mon[29847]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-05-12T22:02:52.341 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:52 smithi186 ceph-mon[34785]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]': finished 2024-05-12T22:02:52.341 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:52 smithi186 ceph-mon[34785]: osdmap e38: 6 total, 5 up, 6 in 2024-05-12T22:02:52.341 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:52 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-12T22:02:52.342 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:52 smithi186 ceph-mon[34785]: osd.5 [v2:172.21.15.186:6808/1944205756,v1:172.21.15.186:6809/1944205756] boot 2024-05-12T22:02:52.342 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:52 smithi186 ceph-mon[34785]: osdmap e39: 6 total, 6 up, 6 in 2024-05-12T22:02:52.342 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:52 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-12T22:02:52.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:52 smithi086 ceph-mon[34297]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]': finished 2024-05-12T22:02:52.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:52 smithi086 ceph-mon[34297]: osdmap e38: 6 total, 5 up, 6 in 2024-05-12T22:02:52.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:52 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-12T22:02:52.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:52 smithi086 ceph-mon[34297]: osd.5 [v2:172.21.15.186:6808/1944205756,v1:172.21.15.186:6809/1944205756] boot 2024-05-12T22:02:52.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:52 smithi086 ceph-mon[34297]: osdmap e39: 6 total, 6 up, 6 in 2024-05-12T22:02:52.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:52 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-12T22:02:52.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:52 smithi028 ceph-mon[29847]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]': finished 2024-05-12T22:02:52.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:52 smithi028 ceph-mon[29847]: osdmap e38: 6 total, 5 up, 6 in 2024-05-12T22:02:52.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:52 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-12T22:02:52.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:52 smithi028 ceph-mon[29847]: osd.5 [v2:172.21.15.186:6808/1944205756,v1:172.21.15.186:6809/1944205756] boot 2024-05-12T22:02:52.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:52 smithi028 ceph-mon[29847]: osdmap e39: 6 total, 6 up, 6 in 2024-05-12T22:02:52.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:52 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-12T22:02:53.140 INFO:teuthology.orchestra.run.smithi028.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-05-12T22:02:53.141 INFO:teuthology.orchestra.run.smithi028.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-05-12T22:02:53.141 INFO:teuthology.orchestra.run.smithi028.stderr: stderr: 10+0 records in 2024-05-12T22:02:53.141 INFO:teuthology.orchestra.run.smithi028.stderr:10+0 records out 2024-05-12T22:02:53.141 INFO:teuthology.orchestra.run.smithi028.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0354919 s, 295 MB/s 2024-05-12T22:02:53.141 INFO:teuthology.orchestra.run.smithi028.stderr:--> Zapping successful for: 2024-05-12T22:02:53.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:53 smithi186 ceph-mon[34785]: purged_snaps scrub starts 2024-05-12T22:02:53.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:53 smithi186 ceph-mon[34785]: purged_snaps scrub ok 2024-05-12T22:02:53.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:53 smithi186 ceph-mon[34785]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-05-12T22:02:53.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:53 smithi186 ceph-mon[34785]: Detected new or changed devices on smithi186 2024-05-12T22:02:53.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:53 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:02:53.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:53 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:02:53.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:53 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:02:53.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:53 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:02:53.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:53 smithi186 ceph-mon[34785]: Adjusting osd_memory_target on smithi186 to 8544M 2024-05-12T22:02:53.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:53 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:02:53.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:53 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:02:53.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:53 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:02:53.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:53 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:02:53.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:53 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:02:53.681 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:53 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:02:53.681 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:53 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:02:53.681 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:53 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:02:53.681 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:53 smithi186 ceph-mon[34785]: osdmap e40: 6 total, 6 up, 6 in 2024-05-12T22:02:53.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:53 smithi086 ceph-mon[34297]: purged_snaps scrub starts 2024-05-12T22:02:53.743 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:53 smithi086 ceph-mon[34297]: purged_snaps scrub ok 2024-05-12T22:02:53.743 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:53 smithi086 ceph-mon[34297]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-05-12T22:02:53.743 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:53 smithi086 ceph-mon[34297]: Detected new or changed devices on smithi186 2024-05-12T22:02:53.743 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:53 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:02:53.743 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:53 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:02:53.743 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:53 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:02:53.743 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:53 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:02:53.743 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:53 smithi086 ceph-mon[34297]: Adjusting osd_memory_target on smithi186 to 8544M 2024-05-12T22:02:53.743 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:53 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:02:53.743 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:53 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:02:53.743 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:53 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:02:53.743 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:53 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:02:53.743 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:53 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:02:53.743 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:53 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:02:53.744 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:53 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:02:53.744 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:53 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:02:53.744 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:53 smithi086 ceph-mon[34297]: osdmap e40: 6 total, 6 up, 6 in 2024-05-12T22:02:53.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:53 smithi028 ceph-mon[29847]: purged_snaps scrub starts 2024-05-12T22:02:53.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:53 smithi028 ceph-mon[29847]: purged_snaps scrub ok 2024-05-12T22:02:53.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:53 smithi028 ceph-mon[29847]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-05-12T22:02:53.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:53 smithi028 ceph-mon[29847]: Detected new or changed devices on smithi186 2024-05-12T22:02:53.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:53 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:02:53.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:53 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:02:53.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:53 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:02:53.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:53 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:02:53.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:53 smithi028 ceph-mon[29847]: Adjusting osd_memory_target on smithi186 to 8544M 2024-05-12T22:02:53.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:53 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:02:53.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:53 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:02:53.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:53 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:02:53.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:53 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:02:53.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:53 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:02:53.787 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:53 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:02:53.787 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:53 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:02:53.787 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:53 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:02:53.787 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:53 smithi028 ceph-mon[29847]: osdmap e40: 6 total, 6 up, 6 in 2024-05-12T22:02:53.905 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f72fecff68e1d400c4568684327c900485c20d6a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6b92e7b4-10aa-11ef-bc98-c7b262605968 -- ceph orch daemon add osd smithi028:vg_nvme/lv_2 2024-05-12T22:02:54.366 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/mon.a/config 2024-05-12T22:02:55.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:55 smithi186 ceph-mon[34785]: pgmap v154: 1 pgs: 1 remapped+peering; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-05-12T22:02:55.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:55 smithi186 ceph-mon[34785]: osdmap e41: 6 total, 6 up, 6 in 2024-05-12T22:02:55.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:55 smithi086 ceph-mon[34297]: pgmap v154: 1 pgs: 1 remapped+peering; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-05-12T22:02:55.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:55 smithi086 ceph-mon[34297]: osdmap e41: 6 total, 6 up, 6 in 2024-05-12T22:02:55.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:55 smithi028 ceph-mon[29847]: pgmap v154: 1 pgs: 1 remapped+peering; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-05-12T22:02:55.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:55 smithi028 ceph-mon[29847]: osdmap e41: 6 total, 6 up, 6 in 2024-05-12T22:02:56.366 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:56 smithi028 ceph-mon[29847]: osdmap e42: 6 total, 6 up, 6 in 2024-05-12T22:02:56.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:56 smithi186 ceph-mon[34785]: osdmap e42: 6 total, 6 up, 6 in 2024-05-12T22:02:56.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:56 smithi086 ceph-mon[34297]: osdmap e42: 6 total, 6 up, 6 in 2024-05-12T22:02:56.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:02:56.627+0000 7f4760600700 1 -- 172.21.15.28:0/4029180189 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4758102390 msgr2=0x7f47581027b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:02:56.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:02:56.627+0000 7f4760600700 1 --2- 172.21.15.28:0/4029180189 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4758102390 0x7f47581027b0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f474c009920 tx=0x7f474c009c30 comp rx=0 tx=0).stop 2024-05-12T22:02:56.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:02:56.627+0000 7f4760600700 1 -- 172.21.15.28:0/4029180189 shutdown_connections 2024-05-12T22:02:56.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:02:56.627+0000 7f4760600700 1 --2- 172.21.15.28:0/4029180189 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4758103f30 0x7f475810a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:02:56.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:02:56.627+0000 7f4760600700 1 --2- 172.21.15.28:0/4029180189 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4758102390 0x7f47581027b0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:02:56.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:02:56.627+0000 7f4760600700 1 --2- 172.21.15.28:0/4029180189 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4758103590 0x7f47581039f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:02:56.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:02:56.627+0000 7f4760600700 1 -- 172.21.15.28:0/4029180189 >> 172.21.15.28:0/4029180189 conn(0x7f47580fd8f0 msgr2=0x7f47580ffd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:02:56.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:02:56.627+0000 7f4760600700 1 -- 172.21.15.28:0/4029180189 shutdown_connections 2024-05-12T22:02:56.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:02:56.627+0000 7f4760600700 1 -- 172.21.15.28:0/4029180189 wait complete. 2024-05-12T22:02:56.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:02:56.629+0000 7f4760600700 1 Processor -- start 2024-05-12T22:02:56.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:02:56.629+0000 7f4760600700 1 -- start start 2024-05-12T22:02:56.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:02:56.630+0000 7f4760600700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4758102390 0x7f475819a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:02:56.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:02:56.630+0000 7f4760600700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4758103590 0x7f475819ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:02:56.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:02:56.630+0000 7f4760600700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4758103f30 0x7f47581a1df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:02:56.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:02:56.630+0000 7f4760600700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f475810df70 con 0x7f4758103f30 2024-05-12T22:02:56.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:02:56.630+0000 7f4760600700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f475810ddf0 con 0x7f4758102390 2024-05-12T22:02:56.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:02:56.630+0000 7f4760600700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f475810e0f0 con 0x7f4758103590 2024-05-12T22:02:56.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:02:56.630+0000 7f475dc00700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4758102390 0x7f475819a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:02:56.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:02:56.630+0000 7f475dc00700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4758102390 0x7f475819a830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:39454/0 (socket says 172.21.15.28:39454) 2024-05-12T22:02:56.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:02:56.630+0000 7f475dc00700 1 -- 172.21.15.28:0/3717444827 learned_addr learned my addr 172.21.15.28:0/3717444827 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:02:56.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:02:56.630+0000 7f475e600700 1 --2- 172.21.15.28:0/3717444827 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4758103f30 0x7f47581a1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:02:56.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:02:56.630+0000 7f475d200700 1 --2- 172.21.15.28:0/3717444827 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4758103590 0x7f475819ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:02:56.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:02:56.630+0000 7f475d200700 1 -- 172.21.15.28:0/3717444827 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4758102390 msgr2=0x7f475819a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:02:56.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:02:56.630+0000 7f475d200700 1 --2- 172.21.15.28:0/3717444827 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4758102390 0x7f475819a830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:02:56.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:02:56.631+0000 7f475d200700 1 -- 172.21.15.28:0/3717444827 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4758103f30 msgr2=0x7f47581a1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:02:56.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:02:56.631+0000 7f475d200700 1 --2- 172.21.15.28:0/3717444827 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4758103f30 0x7f47581a1df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:02:56.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:02:56.631+0000 7f475d200700 1 -- 172.21.15.28:0/3717444827 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47581a2490 con 0x7f4758103590 2024-05-12T22:02:56.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:02:56.631+0000 7f475d200700 1 --2- 172.21.15.28:0/3717444827 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4758103590 0x7f475819ad70 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f474400ea20 tx=0x7f474400ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:02:56.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:02:56.631+0000 7f4756000700 1 -- 172.21.15.28:0/3717444827 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f474400cbd0 con 0x7f4758103590 2024-05-12T22:02:56.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:02:56.631+0000 7f4756000700 1 -- 172.21.15.28:0/3717444827 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4744004500 con 0x7f4758103590 2024-05-12T22:02:56.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:02:56.631+0000 7f4756000700 1 -- 172.21.15.28:0/3717444827 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4744005020 con 0x7f4758103590 2024-05-12T22:02:56.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:02:56.631+0000 7f4760600700 1 -- 172.21.15.28:0/3717444827 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47581a2780 con 0x7f4758103590 2024-05-12T22:02:56.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:02:56.631+0000 7f4760600700 1 -- 172.21.15.28:0/3717444827 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47581a2cc0 con 0x7f4758103590 2024-05-12T22:02:56.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:02:56.632+0000 7f4760600700 1 -- 172.21.15.28:0/3717444827 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4758069010 con 0x7f4758103590 2024-05-12T22:02:56.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:02:56.634+0000 7f4756000700 1 -- 172.21.15.28:0/3717444827 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f4744005200 con 0x7f4758103590 2024-05-12T22:02:56.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:02:56.636+0000 7f4756000700 1 --2- 172.21.15.28:0/3717444827 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f47400a2ca0 0x7f47400a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:02:56.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:02:56.636+0000 7f4756000700 1 -- 172.21.15.28:0/3717444827 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 4533+0+0 (secure 0 0 0) 0x7f47440ce060 con 0x7f4758103590 2024-05-12T22:02:56.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:02:56.636+0000 7f475dc00700 1 --2- 172.21.15.28:0/3717444827 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f47400a2ca0 0x7f47400a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:02:56.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:02:56.637+0000 7f475dc00700 1 --2- 172.21.15.28:0/3717444827 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f47400a2ca0 0x7f47400a5160 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f474c00fca0 tx=0x7f474c019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:02:56.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:02:56.640+0000 7f4756000700 1 -- 172.21.15.28:0/3717444827 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f474409bde0 con 0x7f4758103590 2024-05-12T22:02:56.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:02:56.886+0000 7f4760600700 1 -- 172.21.15.28:0/3717444827 --> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f4758075c70 con 0x7f47400a2ca0 2024-05-12T22:02:57.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:57 smithi186 ceph-mon[34785]: pgmap v157: 1 pgs: 1 remapped+peering; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-05-12T22:02:57.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:57 smithi186 ceph-mon[34785]: osdmap e43: 6 total, 6 up, 6 in 2024-05-12T22:02:57.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:57 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-12T22:02:57.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:57 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-12T22:02:57.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:57 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:02:57.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:57 smithi086 ceph-mon[34297]: pgmap v157: 1 pgs: 1 remapped+peering; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-05-12T22:02:57.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:57 smithi086 ceph-mon[34297]: osdmap e43: 6 total, 6 up, 6 in 2024-05-12T22:02:57.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:57 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-12T22:02:57.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:57 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-12T22:02:57.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:57 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:02:57.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:57 smithi028 ceph-mon[29847]: pgmap v157: 1 pgs: 1 remapped+peering; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-05-12T22:02:57.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:57 smithi028 ceph-mon[29847]: osdmap e43: 6 total, 6 up, 6 in 2024-05-12T22:02:57.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:57 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-12T22:02:57.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:57 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-12T22:02:57.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:57 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:02:58.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:58 smithi186 ceph-mon[34785]: from='client.24262 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-12T22:02:58.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:58 smithi086 ceph-mon[34297]: from='client.24262 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-12T22:02:58.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:58 smithi028 ceph-mon[29847]: from='client.24262 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-12T22:02:59.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:02:59 smithi186 ceph-mon[34785]: pgmap v159: 1 pgs: 1 remapped+peering; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-05-12T22:02:59.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:02:59 smithi086 ceph-mon[34297]: pgmap v159: 1 pgs: 1 remapped+peering; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-05-12T22:02:59.534 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:02:59 smithi028 ceph-mon[29847]: pgmap v159: 1 pgs: 1 remapped+peering; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-05-12T22:03:01.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:01 smithi186 ceph-mon[34785]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-05-12T22:03:01.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:01 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/2708554198' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "516932ac-af00-45b7-ad3c-d6dec7db442f"}]: dispatch 2024-05-12T22:03:01.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:01 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/2708554198' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "516932ac-af00-45b7-ad3c-d6dec7db442f"}]': finished 2024-05-12T22:03:01.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:01 smithi186 ceph-mon[34785]: osdmap e44: 7 total, 6 up, 7 in 2024-05-12T22:03:01.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:01 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-12T22:03:01.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:01 smithi086 ceph-mon[34297]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-05-12T22:03:01.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:01 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/2708554198' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "516932ac-af00-45b7-ad3c-d6dec7db442f"}]: dispatch 2024-05-12T22:03:01.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:01 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/2708554198' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "516932ac-af00-45b7-ad3c-d6dec7db442f"}]': finished 2024-05-12T22:03:01.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:01 smithi086 ceph-mon[34297]: osdmap e44: 7 total, 6 up, 7 in 2024-05-12T22:03:01.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:01 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-12T22:03:01.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:01 smithi028 ceph-mon[29847]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-05-12T22:03:01.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:01 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/2708554198' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "516932ac-af00-45b7-ad3c-d6dec7db442f"}]: dispatch 2024-05-12T22:03:01.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:01 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/2708554198' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "516932ac-af00-45b7-ad3c-d6dec7db442f"}]': finished 2024-05-12T22:03:01.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:01 smithi028 ceph-mon[29847]: osdmap e44: 7 total, 6 up, 7 in 2024-05-12T22:03:01.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:01 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-12T22:03:02.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:02 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/438742142' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-12T22:03:02.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:02 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/438742142' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-12T22:03:02.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:02 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/438742142' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-12T22:03:03.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:03 smithi186 ceph-mon[34785]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-05-12T22:03:03.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:03 smithi086 ceph-mon[34297]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-05-12T22:03:03.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:03 smithi028 ceph-mon[29847]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-05-12T22:03:05.429 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:05 smithi028 ceph-mon[29847]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-05-12T22:03:05.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:05 smithi086 ceph-mon[34297]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-05-12T22:03:05.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:05 smithi186 ceph-mon[34785]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-05-12T22:03:07.488 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:07 smithi028 ceph-mon[29847]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-05-12T22:03:07.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:07 smithi086 ceph-mon[34297]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-05-12T22:03:07.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:07 smithi186 ceph-mon[34785]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-05-12T22:03:09.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:09 smithi028 ceph-mon[29847]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-12T22:03:09.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:09 smithi086 ceph-mon[34297]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-12T22:03:09.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:09 smithi186 ceph-mon[34785]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-12T22:03:11.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:11 smithi086 ceph-mon[34297]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-05-12T22:03:11.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:11 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-05-12T22:03:11.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:11 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:03:11.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:11 smithi028 ceph-mon[29847]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-05-12T22:03:11.566 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:11 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-05-12T22:03:11.566 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:11 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:03:11.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:11 smithi186 ceph-mon[34785]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-05-12T22:03:11.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:11 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-05-12T22:03:11.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:11 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:03:12.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:12 smithi086 ceph-mon[34297]: Deploying daemon osd.6 on smithi028 2024-05-12T22:03:12.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:12 smithi028 ceph-mon[29847]: Deploying daemon osd.6 on smithi028 2024-05-12T22:03:12.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:12 smithi186 ceph-mon[34785]: Deploying daemon osd.6 on smithi028 2024-05-12T22:03:13.284 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:13 smithi028 ceph-mon[29847]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-05-12T22:03:13.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:13 smithi086 ceph-mon[34297]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-05-12T22:03:13.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:13 smithi186 ceph-mon[34785]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-05-12T22:03:15.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:15 smithi028 ceph-mon[29847]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-05-12T22:03:15.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:15 smithi086 ceph-mon[34297]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-05-12T22:03:15.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:15 smithi186 ceph-mon[34785]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-05-12T22:03:17.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:17 smithi086 ceph-mon[34297]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-05-12T22:03:17.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:17 smithi028 ceph-mon[29847]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-05-12T22:03:17.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:17 smithi186 ceph-mon[34785]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-05-12T22:03:19.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:19 smithi086 ceph-mon[34297]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-05-12T22:03:19.534 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:19 smithi028 ceph-mon[29847]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-05-12T22:03:19.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:19 smithi186 ceph-mon[34785]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-05-12T22:03:20.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:20 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:03:20.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:20 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:03:20.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:20 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:03:20.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:20 smithi028 ceph-mon[29847]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-05-12T22:03:20.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:20 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:03:20.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:20 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:03:20.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:20 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:03:20.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:20 smithi186 ceph-mon[34785]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-05-12T22:03:20.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:20 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:03:20.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:20 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:03:20.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:20 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:03:20.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:20 smithi086 ceph-mon[34297]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-05-12T22:03:22.962 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:22 smithi028 ceph-mon[29847]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-05-12T22:03:22.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:22 smithi086 ceph-mon[34297]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-05-12T22:03:23.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:22 smithi186 ceph-mon[34785]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-05-12T22:03:24.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:24 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:03:24.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:24 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:03:24.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:24 smithi028 ceph-mon[29847]: from='osd.6 [v2:172.21.15.28:6818/1642308925,v1:172.21.15.28:6819/1642308925]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-05-12T22:03:24.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:24 smithi028 ceph-mon[29847]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-05-12T22:03:24.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:24 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:03:24.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:24 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:03:24.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:24 smithi186 ceph-mon[34785]: from='osd.6 [v2:172.21.15.28:6818/1642308925,v1:172.21.15.28:6819/1642308925]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-05-12T22:03:24.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:24 smithi186 ceph-mon[34785]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-05-12T22:03:24.700 INFO:teuthology.orchestra.run.smithi028.stdout:Created osd(s) 6 on host 'smithi028' 2024-05-12T22:03:24.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:03:24.699+0000 7f4756000700 1 -- 172.21.15.28:0/3717444827 <== mgr.14152 v2:172.21.15.28:6800/3123328651 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f4758075c70 con 0x7f47400a2ca0 2024-05-12T22:03:24.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:03:24.703+0000 7f4760600700 1 -- 172.21.15.28:0/3717444827 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f47400a2ca0 msgr2=0x7f47400a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:03:24.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:03:24.703+0000 7f4760600700 1 --2- 172.21.15.28:0/3717444827 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f47400a2ca0 0x7f47400a5160 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f474c00fca0 tx=0x7f474c019040 comp rx=0 tx=0).stop 2024-05-12T22:03:24.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:03:24.703+0000 7f4760600700 1 -- 172.21.15.28:0/3717444827 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4758103590 msgr2=0x7f475819ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:03:24.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:03:24.703+0000 7f4760600700 1 --2- 172.21.15.28:0/3717444827 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4758103590 0x7f475819ad70 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f474400ea20 tx=0x7f474400ed30 comp rx=0 tx=0).stop 2024-05-12T22:03:24.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:03:24.703+0000 7f4760600700 1 -- 172.21.15.28:0/3717444827 shutdown_connections 2024-05-12T22:03:24.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:03:24.703+0000 7f4760600700 1 --2- 172.21.15.28:0/3717444827 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f47400a2ca0 0x7f47400a5160 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:03:24.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:03:24.703+0000 7f4760600700 1 --2- 172.21.15.28:0/3717444827 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4758103f30 0x7f47581a1df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:03:24.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:03:24.703+0000 7f4760600700 1 --2- 172.21.15.28:0/3717444827 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4758102390 0x7f475819a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:03:24.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:03:24.703+0000 7f4760600700 1 --2- 172.21.15.28:0/3717444827 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4758103590 0x7f475819ad70 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:03:24.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:03:24.703+0000 7f4760600700 1 -- 172.21.15.28:0/3717444827 >> 172.21.15.28:0/3717444827 conn(0x7f47580fd8f0 msgr2=0x7f4758074550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:03:24.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:03:24.703+0000 7f4760600700 1 -- 172.21.15.28:0/3717444827 shutdown_connections 2024-05-12T22:03:24.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:03:24.704+0000 7f4760600700 1 -- 172.21.15.28:0/3717444827 wait complete. 2024-05-12T22:03:24.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:24 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:03:24.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:24 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:03:24.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:24 smithi086 ceph-mon[34297]: from='osd.6 [v2:172.21.15.28:6818/1642308925,v1:172.21.15.28:6819/1642308925]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-05-12T22:03:24.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:24 smithi086 ceph-mon[34297]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-05-12T22:03:25.517 DEBUG:teuthology.orchestra.run.smithi028:osd.6> sudo journalctl -f -n 0 -u ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968@osd.6.service 2024-05-12T22:03:25.525 INFO:tasks.cephadm:Deploying osd.7 on smithi086 with /dev/vg_nvme/lv_2... 2024-05-12T22:03:25.525 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f72fecff68e1d400c4568684327c900485c20d6a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6b92e7b4-10aa-11ef-bc98-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-05-12T22:03:25.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:25 smithi186 ceph-mon[34785]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-05-12T22:03:25.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:25 smithi186 ceph-mon[34785]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-05-12T22:03:25.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:25 smithi186 ceph-mon[34785]: osdmap e45: 7 total, 6 up, 7 in 2024-05-12T22:03:25.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:25 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-12T22:03:25.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:25 smithi186 ceph-mon[34785]: from='osd.6 [v2:172.21.15.28:6818/1642308925,v1:172.21.15.28:6819/1642308925]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-05-12T22:03:25.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:25 smithi186 ceph-mon[34785]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-05-12T22:03:25.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:25 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:03:25.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:25 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:03:25.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:25 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:03:25.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:25 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:03:25.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:25 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:03:25.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:25 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:03:25.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:25 smithi086 ceph-mon[34297]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-05-12T22:03:25.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:25 smithi086 ceph-mon[34297]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-05-12T22:03:25.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:25 smithi086 ceph-mon[34297]: osdmap e45: 7 total, 6 up, 7 in 2024-05-12T22:03:25.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:25 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-12T22:03:25.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:25 smithi086 ceph-mon[34297]: from='osd.6 [v2:172.21.15.28:6818/1642308925,v1:172.21.15.28:6819/1642308925]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-05-12T22:03:25.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:25 smithi086 ceph-mon[34297]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-05-12T22:03:25.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:25 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:03:25.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:25 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:03:25.743 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:25 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:03:25.743 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:25 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:03:25.743 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:25 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:03:25.743 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:25 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:03:25.769 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/mon.b/config 2024-05-12T22:03:25.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:25 smithi028 ceph-mon[29847]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-05-12T22:03:25.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:25 smithi028 ceph-mon[29847]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-05-12T22:03:25.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:25 smithi028 ceph-mon[29847]: osdmap e45: 7 total, 6 up, 7 in 2024-05-12T22:03:25.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:25 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-12T22:03:25.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:25 smithi028 ceph-mon[29847]: from='osd.6 [v2:172.21.15.28:6818/1642308925,v1:172.21.15.28:6819/1642308925]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-05-12T22:03:25.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:25 smithi028 ceph-mon[29847]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-05-12T22:03:25.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:25 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:03:25.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:25 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:03:25.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:25 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:03:25.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:25 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:03:25.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:25 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:03:25.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:25 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:03:26.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:26 smithi028 ceph-mon[29847]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-05-12T22:03:26.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:26 smithi028 ceph-mon[29847]: osdmap e46: 7 total, 6 up, 7 in 2024-05-12T22:03:26.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:26 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-12T22:03:26.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:26 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-12T22:03:26.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:26 smithi186 ceph-mon[34785]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-05-12T22:03:26.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:26 smithi186 ceph-mon[34785]: osdmap e46: 7 total, 6 up, 7 in 2024-05-12T22:03:26.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:26 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-12T22:03:26.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:26 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-12T22:03:26.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:26 smithi086 ceph-mon[34297]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-05-12T22:03:26.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:26 smithi086 ceph-mon[34297]: osdmap e46: 7 total, 6 up, 7 in 2024-05-12T22:03:26.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:26 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-12T22:03:26.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:26 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-12T22:03:27.558 INFO:teuthology.orchestra.run.smithi086.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-05-12T22:03:27.558 INFO:teuthology.orchestra.run.smithi086.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-05-12T22:03:27.558 INFO:teuthology.orchestra.run.smithi086.stderr: stderr: 10+0 records in 2024-05-12T22:03:27.558 INFO:teuthology.orchestra.run.smithi086.stderr:10+0 records out 2024-05-12T22:03:27.558 INFO:teuthology.orchestra.run.smithi086.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.011357 s, 923 MB/s 2024-05-12T22:03:27.558 INFO:teuthology.orchestra.run.smithi086.stderr:--> Zapping successful for: 2024-05-12T22:03:27.638 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:27 smithi028 ceph-mon[29847]: purged_snaps scrub starts 2024-05-12T22:03:27.639 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:27 smithi028 ceph-mon[29847]: purged_snaps scrub ok 2024-05-12T22:03:27.639 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:27 smithi028 ceph-mon[29847]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-05-12T22:03:27.639 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:27 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-12T22:03:27.639 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:27 smithi028 ceph-mon[29847]: osd.6 [v2:172.21.15.28:6818/1642308925,v1:172.21.15.28:6819/1642308925] boot 2024-05-12T22:03:27.639 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:27 smithi028 ceph-mon[29847]: osdmap e47: 7 total, 7 up, 7 in 2024-05-12T22:03:27.639 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:27 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-12T22:03:27.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:27 smithi186 ceph-mon[34785]: purged_snaps scrub starts 2024-05-12T22:03:27.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:27 smithi186 ceph-mon[34785]: purged_snaps scrub ok 2024-05-12T22:03:27.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:27 smithi186 ceph-mon[34785]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-05-12T22:03:27.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:27 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-12T22:03:27.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:27 smithi186 ceph-mon[34785]: osd.6 [v2:172.21.15.28:6818/1642308925,v1:172.21.15.28:6819/1642308925] boot 2024-05-12T22:03:27.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:27 smithi186 ceph-mon[34785]: osdmap e47: 7 total, 7 up, 7 in 2024-05-12T22:03:27.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:27 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-12T22:03:27.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:27 smithi086 ceph-mon[34297]: purged_snaps scrub starts 2024-05-12T22:03:27.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:27 smithi086 ceph-mon[34297]: purged_snaps scrub ok 2024-05-12T22:03:27.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:27 smithi086 ceph-mon[34297]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-05-12T22:03:27.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:27 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-12T22:03:27.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:27 smithi086 ceph-mon[34297]: osd.6 [v2:172.21.15.28:6818/1642308925,v1:172.21.15.28:6819/1642308925] boot 2024-05-12T22:03:27.743 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:27 smithi086 ceph-mon[34297]: osdmap e47: 7 total, 7 up, 7 in 2024-05-12T22:03:27.743 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:27 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-12T22:03:28.127 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f72fecff68e1d400c4568684327c900485c20d6a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6b92e7b4-10aa-11ef-bc98-c7b262605968 -- ceph orch daemon add osd smithi086:vg_nvme/lv_2 2024-05-12T22:03:28.368 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/mon.b/config 2024-05-12T22:03:28.668 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:28 smithi086 ceph-mon[34297]: osdmap e48: 7 total, 7 up, 7 in 2024-05-12T22:03:28.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:28 smithi028 ceph-mon[29847]: osdmap e48: 7 total, 7 up, 7 in 2024-05-12T22:03:28.855 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:28 smithi186 ceph-mon[34785]: osdmap e48: 7 total, 7 up, 7 in 2024-05-12T22:03:29.547 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:29 smithi086 ceph-mon[34297]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-05-12T22:03:29.760 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:03:29.759+0000 7f6cee0f6700 1 -- 172.21.15.86:0/712586606 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6ce8102e10 msgr2=0x7f6ce81096c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:03:29.760 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:03:29.759+0000 7f6cee0f6700 1 --2- 172.21.15.86:0/712586606 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6ce8102e10 0x7f6ce81096c0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6ce0009fc0 tx=0x7f6ce000b330 comp rx=0 tx=0).stop 2024-05-12T22:03:29.760 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:03:29.759+0000 7f6cee0f6700 1 -- 172.21.15.86:0/712586606 shutdown_connections 2024-05-12T22:03:29.760 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:03:29.759+0000 7f6cee0f6700 1 --2- 172.21.15.86:0/712586606 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6ce8102e10 0x7f6ce81096c0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:03:29.760 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:03:29.759+0000 7f6cee0f6700 1 --2- 172.21.15.86:0/712586606 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ce8101270 0x7f6ce8101690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:03:29.760 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:03:29.759+0000 7f6cee0f6700 1 --2- 172.21.15.86:0/712586606 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6ce8102470 0x7f6ce81028d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:03:29.760 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:03:29.759+0000 7f6cee0f6700 1 -- 172.21.15.86:0/712586606 >> 172.21.15.86:0/712586606 conn(0x7f6ce80fc810 msgr2=0x7f6ce80fec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:03:29.760 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:03:29.759+0000 7f6cee0f6700 1 -- 172.21.15.86:0/712586606 shutdown_connections 2024-05-12T22:03:29.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:03:29.759+0000 7f6cee0f6700 1 -- 172.21.15.86:0/712586606 wait complete. 2024-05-12T22:03:29.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:03:29.760+0000 7f6cee0f6700 1 Processor -- start 2024-05-12T22:03:29.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:03:29.760+0000 7f6cee0f6700 1 -- start start 2024-05-12T22:03:29.763 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:03:29.760+0000 7f6cee0f6700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6ce8101270 0x7f6ce81974c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:03:29.763 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:03:29.760+0000 7f6cee0f6700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6ce8102470 0x7f6ce8197a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:03:29.763 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:03:29.760+0000 7f6cee0f6700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ce8102e10 0x7f6ce819ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:03:29.763 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:03:29.760+0000 7f6cee0f6700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6ce810ced0 con 0x7f6ce8102e10 2024-05-12T22:03:29.763 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:03:29.760+0000 7f6cee0f6700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f6ce810cd50 con 0x7f6ce8102470 2024-05-12T22:03:29.763 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:03:29.760+0000 7f6cee0f6700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f6ce810d050 con 0x7f6ce8101270 2024-05-12T22:03:29.763 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:03:29.760+0000 7f6ce6ffd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6ce8102470 0x7f6ce8197a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:03:29.763 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:03:29.760+0000 7f6ce6ffd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6ce8102470 0x7f6ce8197a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:57892/0 (socket says 172.21.15.86:57892) 2024-05-12T22:03:29.763 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:03:29.760+0000 7f6ce6ffd700 1 -- 172.21.15.86:0/516402366 learned_addr learned my addr 172.21.15.86:0/516402366 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-05-12T22:03:29.763 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:03:29.760+0000 7f6ce77fe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6ce8101270 0x7f6ce81974c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:03:29.763 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:03:29.761+0000 7f6ce77fe700 1 -- 172.21.15.86:0/516402366 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6ce8102470 msgr2=0x7f6ce8197a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:03:29.764 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:03:29.761+0000 7f6ce77fe700 1 --2- 172.21.15.86:0/516402366 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6ce8102470 0x7f6ce8197a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:03:29.764 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:03:29.761+0000 7f6ce77fe700 1 -- 172.21.15.86:0/516402366 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ce8102e10 msgr2=0x7f6ce819ea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:03:29.764 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:03:29.761+0000 7f6ce77fe700 1 --2- 172.21.15.86:0/516402366 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ce8102e10 0x7f6ce819ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:03:29.764 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:03:29.761+0000 7f6ce77fe700 1 -- 172.21.15.86:0/516402366 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ce819f120 con 0x7f6ce8101270 2024-05-12T22:03:29.764 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:03:29.761+0000 7f6ce6ffd700 1 --2- 172.21.15.86:0/516402366 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6ce8102470 0x7f6ce8197a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:03:29.764 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:03:29.761+0000 7f6ce77fe700 1 --2- 172.21.15.86:0/516402366 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6ce8101270 0x7f6ce81974c0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f6cd800b450 tx=0x7f6cd800b760 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:03:29.764 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:03:29.761+0000 7f6ce4ff9700 1 -- 172.21.15.86:0/516402366 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cd8015020 con 0x7f6ce8101270 2024-05-12T22:03:29.764 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:03:29.761+0000 7f6cee0f6700 1 -- 172.21.15.86:0/516402366 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ce806a8f0 con 0x7f6ce8101270 2024-05-12T22:03:29.764 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:03:29.761+0000 7f6ce4ff9700 1 -- 172.21.15.86:0/516402366 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6cd8011d10 con 0x7f6ce8101270 2024-05-12T22:03:29.764 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:03:29.761+0000 7f6ce4ff9700 1 -- 172.21.15.86:0/516402366 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cd8010d80 con 0x7f6ce8101270 2024-05-12T22:03:29.765 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:03:29.761+0000 7f6cee0f6700 1 -- 172.21.15.86:0/516402366 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ce806ae30 con 0x7f6ce8101270 2024-05-12T22:03:29.765 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:03:29.762+0000 7f6cee0f6700 1 -- 172.21.15.86:0/516402366 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ce8069010 con 0x7f6ce8101270 2024-05-12T22:03:29.765 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:03:29.763+0000 7f6ce4ff9700 1 -- 172.21.15.86:0/516402366 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f6cd8018020 con 0x7f6ce8101270 2024-05-12T22:03:29.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:03:29.764+0000 7f6ce4ff9700 1 --2- 172.21.15.86:0/516402366 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f6cd00a2c20 0x7f6cd00a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:03:29.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:03:29.764+0000 7f6ce4ff9700 1 -- 172.21.15.86:0/516402366 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 4965+0+0 (secure 0 0 0) 0x7f6cd80cdcc0 con 0x7f6ce8101270 2024-05-12T22:03:29.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:03:29.764+0000 7f6ce6ffd700 1 --2- 172.21.15.86:0/516402366 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f6cd00a2c20 0x7f6cd00a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:03:29.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:03:29.765+0000 7f6ce6ffd700 1 --2- 172.21.15.86:0/516402366 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f6cd00a2c20 0x7f6cd00a50e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f6cdc009750 tx=0x7f6cdc010040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:03:29.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:03:29.767+0000 7f6ce4ff9700 1 -- 172.21.15.86:0/516402366 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f6cd801f020 con 0x7f6ce8101270 2024-05-12T22:03:29.768 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:29 smithi186 ceph-mon[34785]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-05-12T22:03:29.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:29 smithi028 ceph-mon[29847]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-05-12T22:03:29.973 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:03:29.973+0000 7f6cee0f6700 1 -- 172.21.15.86:0/516402366 --> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f6ce806b0f0 con 0x7f6cd00a2c20 2024-05-12T22:03:30.706 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:30 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-12T22:03:30.706 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:30 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-12T22:03:30.707 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:30 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:03:30.707 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:30 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:03:30.707 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:30 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:03:30.707 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:30 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:03:30.707 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:30 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:03:30.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:30 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-12T22:03:30.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:30 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-12T22:03:30.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:30 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:03:30.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:30 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:03:30.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:30 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:03:30.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:30 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:03:30.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:30 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:03:30.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:30 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-12T22:03:30.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:30 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-12T22:03:30.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:30 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:03:30.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:30 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:03:30.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:30 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:03:30.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:30 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:03:30.930 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:30 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:03:31.658 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:31 smithi086 ceph-mon[34297]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-05-12T22:03:31.658 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:31 smithi086 ceph-mon[34297]: from='client.24283 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-12T22:03:31.658 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:31 smithi086 ceph-mon[34297]: Detected new or changed devices on smithi028 2024-05-12T22:03:31.658 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:31 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:03:31.659 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:31 smithi086 ceph-mon[34297]: Adjusting osd_memory_target on smithi028 to 5698M 2024-05-12T22:03:31.659 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:31 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:03:31.659 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:31 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:03:31.659 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:31 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:03:31.659 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:31 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:03:31.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:31 smithi028 ceph-mon[29847]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-05-12T22:03:31.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:31 smithi028 ceph-mon[29847]: from='client.24283 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-12T22:03:31.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:31 smithi028 ceph-mon[29847]: Detected new or changed devices on smithi028 2024-05-12T22:03:31.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:31 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:03:31.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:31 smithi028 ceph-mon[29847]: Adjusting osd_memory_target on smithi028 to 5698M 2024-05-12T22:03:31.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:31 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:03:31.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:31 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:03:31.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:31 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:03:31.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:31 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:03:31.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:31 smithi186 ceph-mon[34785]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-05-12T22:03:31.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:31 smithi186 ceph-mon[34785]: from='client.24283 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-12T22:03:31.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:31 smithi186 ceph-mon[34785]: Detected new or changed devices on smithi028 2024-05-12T22:03:31.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:31 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:03:31.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:31 smithi186 ceph-mon[34785]: Adjusting osd_memory_target on smithi028 to 5698M 2024-05-12T22:03:31.930 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:31 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:03:31.930 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:31 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:03:31.930 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:31 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:03:31.930 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:31 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:03:32.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:32 smithi086 ceph-mon[34297]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-05-12T22:03:32.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:32 smithi028 ceph-mon[29847]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-05-12T22:03:32.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:32 smithi186 ceph-mon[34785]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-05-12T22:03:33.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:33 smithi086 ceph-mon[34297]: from='client.? 172.21.15.86:0/1491042080' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ca7a059f-0414-438e-b2d1-a10f348036be"}]: dispatch 2024-05-12T22:03:33.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:33 smithi086 ceph-mon[34297]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ca7a059f-0414-438e-b2d1-a10f348036be"}]: dispatch 2024-05-12T22:03:33.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:33 smithi086 ceph-mon[34297]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ca7a059f-0414-438e-b2d1-a10f348036be"}]': finished 2024-05-12T22:03:33.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:33 smithi086 ceph-mon[34297]: osdmap e49: 8 total, 7 up, 8 in 2024-05-12T22:03:33.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:33 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-12T22:03:33.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:33 smithi086 ceph-mon[34297]: from='client.? 172.21.15.86:0/1147159916' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-12T22:03:33.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:33 smithi028 ceph-mon[29847]: from='client.? 172.21.15.86:0/1491042080' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ca7a059f-0414-438e-b2d1-a10f348036be"}]: dispatch 2024-05-12T22:03:33.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:33 smithi028 ceph-mon[29847]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ca7a059f-0414-438e-b2d1-a10f348036be"}]: dispatch 2024-05-12T22:03:33.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:33 smithi028 ceph-mon[29847]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ca7a059f-0414-438e-b2d1-a10f348036be"}]': finished 2024-05-12T22:03:33.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:33 smithi028 ceph-mon[29847]: osdmap e49: 8 total, 7 up, 8 in 2024-05-12T22:03:33.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:33 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-12T22:03:33.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:33 smithi028 ceph-mon[29847]: from='client.? 172.21.15.86:0/1147159916' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-12T22:03:33.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:33 smithi186 ceph-mon[34785]: from='client.? 172.21.15.86:0/1491042080' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ca7a059f-0414-438e-b2d1-a10f348036be"}]: dispatch 2024-05-12T22:03:33.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:33 smithi186 ceph-mon[34785]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ca7a059f-0414-438e-b2d1-a10f348036be"}]: dispatch 2024-05-12T22:03:33.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:33 smithi186 ceph-mon[34785]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ca7a059f-0414-438e-b2d1-a10f348036be"}]': finished 2024-05-12T22:03:33.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:33 smithi186 ceph-mon[34785]: osdmap e49: 8 total, 7 up, 8 in 2024-05-12T22:03:33.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:33 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-12T22:03:33.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:33 smithi186 ceph-mon[34785]: from='client.? 172.21.15.86:0/1147159916' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-12T22:03:34.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:34 smithi086 ceph-mon[34297]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-05-12T22:03:34.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:34 smithi028 ceph-mon[29847]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-05-12T22:03:34.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:34 smithi186 ceph-mon[34785]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-05-12T22:03:36.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:36 smithi086 ceph-mon[34297]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-05-12T22:03:37.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:36 smithi028 ceph-mon[29847]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-05-12T22:03:37.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:36 smithi186 ceph-mon[34785]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-05-12T22:03:38.811 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:38 smithi086 ceph-mon[34297]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-05-12T22:03:39.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:38 smithi028 ceph-mon[29847]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-05-12T22:03:39.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:38 smithi186 ceph-mon[34785]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-05-12T22:03:40.737 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:40 smithi086 ceph-mon[34297]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-05-12T22:03:40.737 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:40 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-05-12T22:03:40.737 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:40 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:03:40.737 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:40 smithi086 ceph-mon[34297]: Deploying daemon osd.7 on smithi086 2024-05-12T22:03:41.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:40 smithi028 ceph-mon[29847]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-05-12T22:03:41.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:40 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-05-12T22:03:41.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:40 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:03:41.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:40 smithi028 ceph-mon[29847]: Deploying daemon osd.7 on smithi086 2024-05-12T22:03:41.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:40 smithi186 ceph-mon[34785]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-05-12T22:03:41.180 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:40 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-05-12T22:03:41.180 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:40 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:03:41.180 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:40 smithi186 ceph-mon[34785]: Deploying daemon osd.7 on smithi086 2024-05-12T22:03:42.938 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:42 smithi086 ceph-mon[34297]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-05-12T22:03:43.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:42 smithi028 ceph-mon[29847]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-05-12T22:03:43.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:42 smithi186 ceph-mon[34785]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-05-12T22:03:44.974 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:44 smithi086 ceph-mon[34297]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-05-12T22:03:45.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:44 smithi028 ceph-mon[29847]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-05-12T22:03:45.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:44 smithi186 ceph-mon[34785]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-05-12T22:03:46.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:46 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:03:46.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:46 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:03:46.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:46 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:03:46.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:46 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:03:46.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:46 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:03:46.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:46 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:03:46.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:46 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:03:46.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:46 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:03:46.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:46 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:03:47.197 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:47 smithi086 ceph-mon[34297]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-05-12T22:03:47.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:47 smithi186 ceph-mon[34785]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-05-12T22:03:47.437 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:47 smithi028 ceph-mon[29847]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-05-12T22:03:48.664 INFO:teuthology.orchestra.run.smithi086.stdout:Created osd(s) 7 on host 'smithi086' 2024-05-12T22:03:48.665 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:03:48.664+0000 7f6ce4ff9700 1 -- 172.21.15.86:0/516402366 <== mgr.14152 v2:172.21.15.28:6800/3123328651 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f6ce806b0f0 con 0x7f6cd00a2c20 2024-05-12T22:03:48.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:03:48.666+0000 7f6cee0f6700 1 -- 172.21.15.86:0/516402366 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f6cd00a2c20 msgr2=0x7f6cd00a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:03:48.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:03:48.666+0000 7f6cee0f6700 1 --2- 172.21.15.86:0/516402366 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f6cd00a2c20 0x7f6cd00a50e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f6cdc009750 tx=0x7f6cdc010040 comp rx=0 tx=0).stop 2024-05-12T22:03:48.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:03:48.666+0000 7f6cee0f6700 1 -- 172.21.15.86:0/516402366 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6ce8101270 msgr2=0x7f6ce81974c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:03:48.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:03:48.666+0000 7f6cee0f6700 1 --2- 172.21.15.86:0/516402366 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6ce8101270 0x7f6ce81974c0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f6cd800b450 tx=0x7f6cd800b760 comp rx=0 tx=0).stop 2024-05-12T22:03:48.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:03:48.666+0000 7f6cee0f6700 1 -- 172.21.15.86:0/516402366 shutdown_connections 2024-05-12T22:03:48.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:03:48.666+0000 7f6cee0f6700 1 --2- 172.21.15.86:0/516402366 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f6cd00a2c20 0x7f6cd00a50e0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:03:48.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:03:48.666+0000 7f6cee0f6700 1 --2- 172.21.15.86:0/516402366 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6ce8101270 0x7f6ce81974c0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:03:48.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:03:48.666+0000 7f6cee0f6700 1 --2- 172.21.15.86:0/516402366 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ce8102e10 0x7f6ce819ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:03:48.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:03:48.666+0000 7f6cee0f6700 1 --2- 172.21.15.86:0/516402366 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6ce8102470 0x7f6ce8197a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:03:48.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:03:48.666+0000 7f6cee0f6700 1 -- 172.21.15.86:0/516402366 >> 172.21.15.86:0/516402366 conn(0x7f6ce80fc810 msgr2=0x7f6ce8105710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:03:48.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:03:48.666+0000 7f6cee0f6700 1 -- 172.21.15.86:0/516402366 shutdown_connections 2024-05-12T22:03:48.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:03:48.666+0000 7f6cee0f6700 1 -- 172.21.15.86:0/516402366 wait complete. 2024-05-12T22:03:49.101 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:48 smithi086 ceph-mon[34297]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-05-12T22:03:49.101 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:48 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:03:49.101 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:48 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:03:49.101 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:48 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:03:49.101 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:48 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:03:49.101 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:48 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:03:49.101 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:48 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:03:49.101 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:48 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:03:49.101 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:48 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:03:49.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:48 smithi028 ceph-mon[29847]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-05-12T22:03:49.106 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:48 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:03:49.106 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:48 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:03:49.106 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:48 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:03:49.106 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:48 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:03:49.106 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:48 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:03:49.106 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:48 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:03:49.106 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:48 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:03:49.106 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:48 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:03:49.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:48 smithi186 ceph-mon[34785]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-05-12T22:03:49.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:48 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:03:49.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:48 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:03:49.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:48 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:03:49.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:48 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:03:49.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:48 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:03:49.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:48 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:03:49.180 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:48 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:03:49.180 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:48 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:03:49.283 DEBUG:teuthology.orchestra.run.smithi086:osd.7> sudo journalctl -f -n 0 -u ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968@osd.7.service 2024-05-12T22:03:49.285 INFO:tasks.cephadm:Deploying osd.8 on smithi186 with /dev/vg_nvme/lv_2... 2024-05-12T22:03:49.286 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f72fecff68e1d400c4568684327c900485c20d6a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6b92e7b4-10aa-11ef-bc98-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-05-12T22:03:49.532 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/mon.c/config 2024-05-12T22:03:49.905 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:49 smithi186 ceph-mon[34785]: from='osd.7 [v2:172.21.15.86:6816/726062711,v1:172.21.15.86:6817/726062711]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-12T22:03:49.905 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:49 smithi186 ceph-mon[34785]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-12T22:03:49.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:49 smithi086 ceph-mon[34297]: from='osd.7 [v2:172.21.15.86:6816/726062711,v1:172.21.15.86:6817/726062711]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-12T22:03:49.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:49 smithi086 ceph-mon[34297]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-12T22:03:50.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:49 smithi028 ceph-mon[29847]: from='osd.7 [v2:172.21.15.86:6816/726062711,v1:172.21.15.86:6817/726062711]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-12T22:03:50.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:49 smithi028 ceph-mon[29847]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-12T22:03:50.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:50 smithi086 ceph-mon[34297]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-05-12T22:03:50.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:50 smithi086 ceph-mon[34297]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-05-12T22:03:50.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:50 smithi086 ceph-mon[34297]: from='osd.7 [v2:172.21.15.86:6816/726062711,v1:172.21.15.86:6817/726062711]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-05-12T22:03:50.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:50 smithi086 ceph-mon[34297]: osdmap e50: 8 total, 7 up, 8 in 2024-05-12T22:03:50.993 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:50 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-12T22:03:50.993 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:50 smithi086 ceph-mon[34297]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-05-12T22:03:50.993 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 12 22:03:50 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-7[52481]: 2024-05-12T22:03:50.856+0000 7f1b79ad8700 -1 osd.7 0 waiting for initial osdmap 2024-05-12T22:03:50.993 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 12 22:03:50 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-7[52481]: 2024-05-12T22:03:50.867+0000 7f1b760cc700 -1 osd.7 51 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-12T22:03:51.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:50 smithi186 ceph-mon[34785]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-05-12T22:03:51.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:50 smithi186 ceph-mon[34785]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-05-12T22:03:51.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:50 smithi186 ceph-mon[34785]: from='osd.7 [v2:172.21.15.86:6816/726062711,v1:172.21.15.86:6817/726062711]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-05-12T22:03:51.180 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:50 smithi186 ceph-mon[34785]: osdmap e50: 8 total, 7 up, 8 in 2024-05-12T22:03:51.180 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:50 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-12T22:03:51.180 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:50 smithi186 ceph-mon[34785]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-05-12T22:03:51.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:50 smithi028 ceph-mon[29847]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-05-12T22:03:51.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:50 smithi028 ceph-mon[29847]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-05-12T22:03:51.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:50 smithi028 ceph-mon[29847]: from='osd.7 [v2:172.21.15.86:6816/726062711,v1:172.21.15.86:6817/726062711]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-05-12T22:03:51.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:50 smithi028 ceph-mon[29847]: osdmap e50: 8 total, 7 up, 8 in 2024-05-12T22:03:51.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:50 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-12T22:03:51.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:50 smithi028 ceph-mon[29847]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-05-12T22:03:51.498 INFO:teuthology.orchestra.run.smithi186.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-05-12T22:03:51.498 INFO:teuthology.orchestra.run.smithi186.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-05-12T22:03:51.499 INFO:teuthology.orchestra.run.smithi186.stderr: stderr: 10+0 records in 2024-05-12T22:03:51.499 INFO:teuthology.orchestra.run.smithi186.stderr:10+0 records out 2024-05-12T22:03:51.499 INFO:teuthology.orchestra.run.smithi186.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0120734 s, 869 MB/s 2024-05-12T22:03:51.499 INFO:teuthology.orchestra.run.smithi186.stderr:--> Zapping successful for: 2024-05-12T22:03:51.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:51 smithi086 ceph-mon[34297]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]': finished 2024-05-12T22:03:51.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:51 smithi086 ceph-mon[34297]: osdmap e51: 8 total, 7 up, 8 in 2024-05-12T22:03:51.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:51 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-12T22:03:51.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:51 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-12T22:03:52.051 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:51 smithi186 ceph-mon[34785]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]': finished 2024-05-12T22:03:52.051 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:51 smithi186 ceph-mon[34785]: osdmap e51: 8 total, 7 up, 8 in 2024-05-12T22:03:52.051 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:51 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-12T22:03:52.051 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:51 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-12T22:03:52.053 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f72fecff68e1d400c4568684327c900485c20d6a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6b92e7b4-10aa-11ef-bc98-c7b262605968 -- ceph orch daemon add osd smithi186:vg_nvme/lv_2 2024-05-12T22:03:52.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:51 smithi028 ceph-mon[29847]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]': finished 2024-05-12T22:03:52.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:51 smithi028 ceph-mon[29847]: osdmap e51: 8 total, 7 up, 8 in 2024-05-12T22:03:52.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:51 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-12T22:03:52.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:51 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-12T22:03:52.305 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/mon.c/config 2024-05-12T22:03:52.965 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:52 smithi186 ceph-mon[34785]: purged_snaps scrub starts 2024-05-12T22:03:52.965 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:52 smithi186 ceph-mon[34785]: purged_snaps scrub ok 2024-05-12T22:03:52.966 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:52 smithi186 ceph-mon[34785]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-05-12T22:03:52.966 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:52 smithi186 ceph-mon[34785]: osd.7 [v2:172.21.15.86:6816/726062711,v1:172.21.15.86:6817/726062711] boot 2024-05-12T22:03:52.966 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:52 smithi186 ceph-mon[34785]: osdmap e52: 8 total, 8 up, 8 in 2024-05-12T22:03:52.966 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:52 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-12T22:03:52.966 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:52 smithi186 ceph-mon[34785]: Detected new or changed devices on smithi086 2024-05-12T22:03:52.966 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:52 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:03:52.967 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:52 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:03:52.967 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:52 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:03:52.967 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:52 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:03:52.967 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:52 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:03:52.967 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:52 smithi186 ceph-mon[34785]: Adjusting osd_memory_target on smithi086 to 5696M 2024-05-12T22:03:52.967 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:52 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:03:52.967 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:52 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:03:52.968 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:52 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:03:52.968 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:52 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:03:52.968 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:52 smithi186 ceph-mon[34785]: osdmap e53: 8 total, 8 up, 8 in 2024-05-12T22:03:53.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:52 smithi086 ceph-mon[34297]: purged_snaps scrub starts 2024-05-12T22:03:53.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:52 smithi086 ceph-mon[34297]: purged_snaps scrub ok 2024-05-12T22:03:53.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:52 smithi086 ceph-mon[34297]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-05-12T22:03:53.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:52 smithi086 ceph-mon[34297]: osd.7 [v2:172.21.15.86:6816/726062711,v1:172.21.15.86:6817/726062711] boot 2024-05-12T22:03:53.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:52 smithi086 ceph-mon[34297]: osdmap e52: 8 total, 8 up, 8 in 2024-05-12T22:03:53.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:52 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-12T22:03:53.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:52 smithi086 ceph-mon[34297]: Detected new or changed devices on smithi086 2024-05-12T22:03:53.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:52 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:03:53.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:52 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:03:53.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:52 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:03:53.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:52 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:03:53.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:52 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:03:53.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:52 smithi086 ceph-mon[34297]: Adjusting osd_memory_target on smithi086 to 5696M 2024-05-12T22:03:53.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:52 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:03:53.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:52 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:03:53.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:52 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:03:53.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:52 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:03:53.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:52 smithi086 ceph-mon[34297]: osdmap e53: 8 total, 8 up, 8 in 2024-05-12T22:03:53.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:52 smithi028 ceph-mon[29847]: purged_snaps scrub starts 2024-05-12T22:03:53.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:52 smithi028 ceph-mon[29847]: purged_snaps scrub ok 2024-05-12T22:03:53.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:52 smithi028 ceph-mon[29847]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-05-12T22:03:53.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:52 smithi028 ceph-mon[29847]: osd.7 [v2:172.21.15.86:6816/726062711,v1:172.21.15.86:6817/726062711] boot 2024-05-12T22:03:53.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:52 smithi028 ceph-mon[29847]: osdmap e52: 8 total, 8 up, 8 in 2024-05-12T22:03:53.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:52 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-12T22:03:53.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:52 smithi028 ceph-mon[29847]: Detected new or changed devices on smithi086 2024-05-12T22:03:53.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:52 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:03:53.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:52 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:03:53.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:52 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:03:53.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:52 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:03:53.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:52 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:03:53.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:52 smithi028 ceph-mon[29847]: Adjusting osd_memory_target on smithi086 to 5696M 2024-05-12T22:03:53.287 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:52 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:03:53.287 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:52 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:03:53.287 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:52 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:03:53.287 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:52 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:03:53.287 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:52 smithi028 ceph-mon[29847]: osdmap e53: 8 total, 8 up, 8 in 2024-05-12T22:03:53.704 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:03:53.703+0000 7f07bceee700 1 -- 172.21.15.186:0/3042298667 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f07b8102470 msgr2=0x7f07b81028d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:03:53.704 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:03:53.703+0000 7f07bceee700 1 --2- 172.21.15.186:0/3042298667 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f07b8102470 0x7f07b81028d0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f07ac009990 tx=0x7f07ac009ca0 comp rx=0 tx=0).stop 2024-05-12T22:03:53.705 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:03:53.703+0000 7f07bceee700 1 -- 172.21.15.186:0/3042298667 shutdown_connections 2024-05-12T22:03:53.705 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:03:53.703+0000 7f07bceee700 1 --2- 172.21.15.186:0/3042298667 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f07b8102e10 0x7f07b81096c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:03:53.705 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:03:53.703+0000 7f07bceee700 1 --2- 172.21.15.186:0/3042298667 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07b8101270 0x7f07b8101690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:03:53.705 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:03:53.703+0000 7f07bceee700 1 --2- 172.21.15.186:0/3042298667 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f07b8102470 0x7f07b81028d0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:03:53.705 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:03:53.703+0000 7f07bceee700 1 -- 172.21.15.186:0/3042298667 >> 172.21.15.186:0/3042298667 conn(0x7f07b80fc810 msgr2=0x7f07b80fec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:03:53.705 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:03:53.703+0000 7f07bceee700 1 -- 172.21.15.186:0/3042298667 shutdown_connections 2024-05-12T22:03:53.705 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:03:53.703+0000 7f07bceee700 1 -- 172.21.15.186:0/3042298667 wait complete. 2024-05-12T22:03:53.707 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:03:53.704+0000 7f07bceee700 1 Processor -- start 2024-05-12T22:03:53.707 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:03:53.704+0000 7f07bceee700 1 -- start start 2024-05-12T22:03:53.707 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:03:53.704+0000 7f07bceee700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f07b8101270 0x7f07b8197500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:03:53.707 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:03:53.704+0000 7f07bceee700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07b8102470 0x7f07b8197a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:03:53.707 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:03:53.704+0000 7f07bceee700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f07b8102e10 0x7f07b819eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:03:53.707 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:03:53.704+0000 7f07bceee700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f07b810cef0 con 0x7f07b8102470 2024-05-12T22:03:53.708 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:03:53.704+0000 7f07bceee700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f07b810cd70 con 0x7f07b8102e10 2024-05-12T22:03:53.708 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:03:53.704+0000 7f07bceee700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f07b810d070 con 0x7f07b8101270 2024-05-12T22:03:53.708 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:03:53.705+0000 7f07b659c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f07b8101270 0x7f07b8197500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:03:53.708 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:03:53.705+0000 7f07b659c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f07b8101270 0x7f07b8197500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:57540/0 (socket says 172.21.15.186:57540) 2024-05-12T22:03:53.708 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:03:53.705+0000 7f07b659c700 1 -- 172.21.15.186:0/4021469318 learned_addr learned my addr 172.21.15.186:0/4021469318 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-05-12T22:03:53.708 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:03:53.705+0000 7f07b6d9d700 1 --2- 172.21.15.186:0/4021469318 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f07b8102e10 0x7f07b819eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:03:53.708 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:03:53.705+0000 7f07b659c700 1 -- 172.21.15.186:0/4021469318 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f07b8102e10 msgr2=0x7f07b819eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:03:53.708 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:03:53.705+0000 7f07b659c700 1 --2- 172.21.15.186:0/4021469318 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f07b8102e10 0x7f07b819eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:03:53.708 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:03:53.705+0000 7f07b659c700 1 -- 172.21.15.186:0/4021469318 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07b8102470 msgr2=0x7f07b8197a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:03:53.708 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:03:53.705+0000 7f07b659c700 1 --2- 172.21.15.186:0/4021469318 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07b8102470 0x7f07b8197a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:03:53.708 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:03:53.705+0000 7f07b659c700 1 -- 172.21.15.186:0/4021469318 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07b819f160 con 0x7f07b8101270 2024-05-12T22:03:53.708 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:03:53.705+0000 7f07b659c700 1 --2- 172.21.15.186:0/4021469318 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f07b8101270 0x7f07b8197500 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f07a800b530 tx=0x7f07a800b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:03:53.708 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:03:53.705+0000 7f07a77fe700 1 -- 172.21.15.186:0/4021469318 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07a8015020 con 0x7f07b8101270 2024-05-12T22:03:53.708 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:03:53.705+0000 7f07bceee700 1 -- 172.21.15.186:0/4021469318 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07b819f450 con 0x7f07b8101270 2024-05-12T22:03:53.709 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:03:53.705+0000 7f07a77fe700 1 -- 172.21.15.186:0/4021469318 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f07a8011d10 con 0x7f07b8101270 2024-05-12T22:03:53.709 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:03:53.705+0000 7f07a77fe700 1 -- 172.21.15.186:0/4021469318 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07a800f2e0 con 0x7f07b8101270 2024-05-12T22:03:53.709 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:03:53.705+0000 7f07bceee700 1 -- 172.21.15.186:0/4021469318 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07b819fa60 con 0x7f07b8101270 2024-05-12T22:03:53.709 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:03:53.706+0000 7f07bceee700 1 -- 172.21.15.186:0/4021469318 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f07b8069010 con 0x7f07b8101270 2024-05-12T22:03:53.709 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:03:53.707+0000 7f07a77fe700 1 -- 172.21.15.186:0/4021469318 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f07a8018020 con 0x7f07b8101270 2024-05-12T22:03:53.710 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:03:53.708+0000 7f07a77fe700 1 --2- 172.21.15.186:0/4021469318 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f07a00a2c20 0x7f07a00a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:03:53.710 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:03:53.708+0000 7f07a77fe700 1 -- 172.21.15.186:0/4021469318 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 5426+0+0 (secure 0 0 0) 0x7f07a80cdc60 con 0x7f07b8101270 2024-05-12T22:03:53.710 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:03:53.709+0000 7f07b5d9b700 1 --2- 172.21.15.186:0/4021469318 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f07a00a2c20 0x7f07a00a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:03:53.710 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:03:53.710+0000 7f07b5d9b700 1 --2- 172.21.15.186:0/4021469318 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f07a00a2c20 0x7f07a00a50e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f07ac005f50 tx=0x7f07ac005e80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:03:53.713 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:03:53.713+0000 7f07a77fe700 1 -- 172.21.15.186:0/4021469318 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f07a801f020 con 0x7f07b8101270 2024-05-12T22:03:53.921 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:03:53.920+0000 7f07bceee700 1 -- 172.21.15.186:0/4021469318 --> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f07b8062760 con 0x7f07a00a2c20 2024-05-12T22:03:55.140 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:54 smithi186 ceph-mon[34785]: pgmap v197: 1 pgs: 1 remapped; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-05-12T22:03:55.141 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:54 smithi186 ceph-mon[34785]: osdmap e54: 8 total, 8 up, 8 in 2024-05-12T22:03:55.141 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:54 smithi186 ceph-mon[34785]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-05-12T22:03:55.141 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:54 smithi186 ceph-mon[34785]: from='client.24295 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-12T22:03:55.141 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:54 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-12T22:03:55.141 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:54 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-12T22:03:55.141 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:54 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:03:55.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:54 smithi086 ceph-mon[34297]: pgmap v197: 1 pgs: 1 remapped; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-05-12T22:03:55.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:54 smithi086 ceph-mon[34297]: osdmap e54: 8 total, 8 up, 8 in 2024-05-12T22:03:55.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:54 smithi086 ceph-mon[34297]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-05-12T22:03:55.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:54 smithi086 ceph-mon[34297]: from='client.24295 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-12T22:03:55.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:54 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-12T22:03:55.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:54 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-12T22:03:55.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:54 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:03:55.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:54 smithi028 ceph-mon[29847]: pgmap v197: 1 pgs: 1 remapped; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-05-12T22:03:55.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:54 smithi028 ceph-mon[29847]: osdmap e54: 8 total, 8 up, 8 in 2024-05-12T22:03:55.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:54 smithi028 ceph-mon[29847]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-05-12T22:03:55.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:54 smithi028 ceph-mon[29847]: from='client.24295 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-12T22:03:55.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:54 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-12T22:03:55.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:54 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-12T22:03:55.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:54 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:03:55.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:55 smithi186 ceph-mon[34785]: osdmap e55: 8 total, 8 up, 8 in 2024-05-12T22:03:56.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:55 smithi086 ceph-mon[34297]: osdmap e55: 8 total, 8 up, 8 in 2024-05-12T22:03:56.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:55 smithi028 ceph-mon[29847]: osdmap e55: 8 total, 8 up, 8 in 2024-05-12T22:03:57.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:56 smithi186 ceph-mon[34785]: pgmap v200: 1 pgs: 1 remapped; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-05-12T22:03:57.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:56 smithi186 ceph-mon[34785]: osdmap e56: 8 total, 8 up, 8 in 2024-05-12T22:03:57.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:56 smithi186 ceph-mon[34785]: from='client.? 172.21.15.186:0/826182445' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "92e3f41f-aebd-422e-a99e-555f7ba4aa53"}]: dispatch 2024-05-12T22:03:57.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:56 smithi186 ceph-mon[34785]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "92e3f41f-aebd-422e-a99e-555f7ba4aa53"}]: dispatch 2024-05-12T22:03:57.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:56 smithi186 ceph-mon[34785]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "92e3f41f-aebd-422e-a99e-555f7ba4aa53"}]': finished 2024-05-12T22:03:57.180 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:56 smithi186 ceph-mon[34785]: osdmap e57: 9 total, 8 up, 9 in 2024-05-12T22:03:57.180 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:56 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-12T22:03:57.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:56 smithi086 ceph-mon[34297]: pgmap v200: 1 pgs: 1 remapped; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-05-12T22:03:57.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:56 smithi086 ceph-mon[34297]: osdmap e56: 8 total, 8 up, 8 in 2024-05-12T22:03:57.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:56 smithi086 ceph-mon[34297]: from='client.? 172.21.15.186:0/826182445' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "92e3f41f-aebd-422e-a99e-555f7ba4aa53"}]: dispatch 2024-05-12T22:03:57.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:56 smithi086 ceph-mon[34297]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "92e3f41f-aebd-422e-a99e-555f7ba4aa53"}]: dispatch 2024-05-12T22:03:57.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:56 smithi086 ceph-mon[34297]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "92e3f41f-aebd-422e-a99e-555f7ba4aa53"}]': finished 2024-05-12T22:03:57.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:56 smithi086 ceph-mon[34297]: osdmap e57: 9 total, 8 up, 9 in 2024-05-12T22:03:57.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:56 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-12T22:03:57.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:56 smithi028 ceph-mon[29847]: pgmap v200: 1 pgs: 1 remapped; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-05-12T22:03:57.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:56 smithi028 ceph-mon[29847]: osdmap e56: 8 total, 8 up, 8 in 2024-05-12T22:03:57.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:56 smithi028 ceph-mon[29847]: from='client.? 172.21.15.186:0/826182445' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "92e3f41f-aebd-422e-a99e-555f7ba4aa53"}]: dispatch 2024-05-12T22:03:57.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:56 smithi028 ceph-mon[29847]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "92e3f41f-aebd-422e-a99e-555f7ba4aa53"}]: dispatch 2024-05-12T22:03:57.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:56 smithi028 ceph-mon[29847]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "92e3f41f-aebd-422e-a99e-555f7ba4aa53"}]': finished 2024-05-12T22:03:57.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:56 smithi028 ceph-mon[29847]: osdmap e57: 9 total, 8 up, 9 in 2024-05-12T22:03:57.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:56 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-12T22:03:58.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:57 smithi186 ceph-mon[34785]: from='client.? 172.21.15.186:0/3877479696' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-12T22:03:58.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:57 smithi086 ceph-mon[34297]: from='client.? 172.21.15.186:0/3877479696' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-12T22:03:58.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:57 smithi028 ceph-mon[29847]: from='client.? 172.21.15.186:0/3877479696' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-12T22:03:59.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:58 smithi186 ceph-mon[34785]: pgmap v203: 1 pgs: 1 remapped; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-05-12T22:03:59.213 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:58 smithi028 ceph-mon[29847]: pgmap v203: 1 pgs: 1 remapped; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-05-12T22:03:59.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:58 smithi086 ceph-mon[34297]: pgmap v203: 1 pgs: 1 remapped; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-05-12T22:04:00.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:59 smithi086 ceph-mon[34297]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-05-12T22:04:00.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:03:59 smithi086 ceph-mon[34297]: Cluster is now healthy 2024-05-12T22:04:00.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:59 smithi028 ceph-mon[29847]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-05-12T22:04:00.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:03:59 smithi028 ceph-mon[29847]: Cluster is now healthy 2024-05-12T22:04:00.379 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:59 smithi186 ceph-mon[34785]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-05-12T22:04:00.379 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:03:59 smithi186 ceph-mon[34785]: Cluster is now healthy 2024-05-12T22:04:01.061 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:00 smithi186 ceph-mon[34785]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-05-12T22:04:01.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:00 smithi086 ceph-mon[34297]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-05-12T22:04:01.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:00 smithi028 ceph-mon[29847]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-05-12T22:04:03.170 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:02 smithi186 ceph-mon[34785]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-05-12T22:04:03.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:02 smithi086 ceph-mon[34297]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-05-12T22:04:03.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:02 smithi028 ceph-mon[29847]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-05-12T22:04:04.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:03 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-05-12T22:04:04.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:03 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:04:04.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:03 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-05-12T22:04:04.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:03 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:04:04.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:03 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-05-12T22:04:04.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:03 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:04:05.219 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:04 smithi186 ceph-mon[34785]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-05-12T22:04:05.220 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:04 smithi186 ceph-mon[34785]: Deploying daemon osd.8 on smithi186 2024-05-12T22:04:05.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:04 smithi086 ceph-mon[34297]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-05-12T22:04:05.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:04 smithi086 ceph-mon[34297]: Deploying daemon osd.8 on smithi186 2024-05-12T22:04:05.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:04 smithi028 ceph-mon[29847]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-05-12T22:04:05.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:04 smithi028 ceph-mon[29847]: Deploying daemon osd.8 on smithi186 2024-05-12T22:04:07.145 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:06 smithi186 ceph-mon[34785]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-05-12T22:04:07.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:06 smithi086 ceph-mon[34297]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-05-12T22:04:07.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:06 smithi028 ceph-mon[29847]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-05-12T22:04:09.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:08 smithi186 ceph-mon[34785]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-05-12T22:04:09.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:08 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:04:09.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:08 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:04:09.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:08 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:04:09.239 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:08 smithi086 ceph-mon[34297]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-05-12T22:04:09.240 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:08 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:04:09.240 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:08 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:04:09.240 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:08 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:04:09.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:08 smithi028 ceph-mon[29847]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-05-12T22:04:09.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:08 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:04:09.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:08 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:04:09.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:08 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:04:11.143 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:10 smithi186 ceph-mon[34785]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-05-12T22:04:11.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:10 smithi086 ceph-mon[34297]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-05-12T22:04:11.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:10 smithi028 ceph-mon[29847]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-05-12T22:04:11.858 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:04:11.857+0000 7f07a77fe700 1 -- 172.21.15.186:0/4021469318 <== mgr.14152 v2:172.21.15.28:6800/3123328651 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f07b8062760 con 0x7f07a00a2c20 2024-05-12T22:04:11.858 INFO:teuthology.orchestra.run.smithi186.stdout:Created osd(s) 8 on host 'smithi186' 2024-05-12T22:04:11.860 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:04:11.859+0000 7f07bceee700 1 -- 172.21.15.186:0/4021469318 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f07a00a2c20 msgr2=0x7f07a00a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:04:11.860 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:04:11.859+0000 7f07bceee700 1 --2- 172.21.15.186:0/4021469318 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f07a00a2c20 0x7f07a00a50e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f07ac005f50 tx=0x7f07ac005e80 comp rx=0 tx=0).stop 2024-05-12T22:04:11.860 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:04:11.859+0000 7f07bceee700 1 -- 172.21.15.186:0/4021469318 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f07b8101270 msgr2=0x7f07b8197500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:04:11.860 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:04:11.859+0000 7f07bceee700 1 --2- 172.21.15.186:0/4021469318 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f07b8101270 0x7f07b8197500 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f07a800b530 tx=0x7f07a800b840 comp rx=0 tx=0).stop 2024-05-12T22:04:11.860 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:04:11.859+0000 7f07bceee700 1 -- 172.21.15.186:0/4021469318 shutdown_connections 2024-05-12T22:04:11.860 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:04:11.859+0000 7f07bceee700 1 --2- 172.21.15.186:0/4021469318 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f07a00a2c20 0x7f07a00a50e0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:04:11.861 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:04:11.859+0000 7f07bceee700 1 --2- 172.21.15.186:0/4021469318 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f07b8101270 0x7f07b8197500 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:04:11.861 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:04:11.859+0000 7f07bceee700 1 --2- 172.21.15.186:0/4021469318 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f07b8102e10 0x7f07b819eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:04:11.861 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:04:11.859+0000 7f07bceee700 1 --2- 172.21.15.186:0/4021469318 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07b8102470 0x7f07b8197a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:04:11.861 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:04:11.859+0000 7f07bceee700 1 -- 172.21.15.186:0/4021469318 >> 172.21.15.186:0/4021469318 conn(0x7f07b80fc810 msgr2=0x7f07b8105770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:04:11.861 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:04:11.859+0000 7f07bceee700 1 -- 172.21.15.186:0/4021469318 shutdown_connections 2024-05-12T22:04:11.861 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:04:11.859+0000 7f07bceee700 1 -- 172.21.15.186:0/4021469318 wait complete. 2024-05-12T22:04:12.485 DEBUG:teuthology.orchestra.run.smithi186:osd.8> sudo journalctl -f -n 0 -u ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968@osd.8.service 2024-05-12T22:04:12.487 INFO:tasks.cephadm:Deploying osd.9 on smithi028 with /dev/vg_nvme/lv_1... 2024-05-12T22:04:12.487 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f72fecff68e1d400c4568684327c900485c20d6a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6b92e7b4-10aa-11ef-bc98-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-05-12T22:04:12.958 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/mon.a/config 2024-05-12T22:04:13.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:12 smithi028 ceph-mon[29847]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-05-12T22:04:13.041 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:12 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:04:13.042 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:12 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:04:13.042 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:12 smithi028 ceph-mon[29847]: from='osd.8 [v2:172.21.15.186:6816/3007687277,v1:172.21.15.186:6817/3007687277]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-05-12T22:04:13.042 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:12 smithi028 ceph-mon[29847]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-05-12T22:04:13.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:12 smithi186 ceph-mon[34785]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-05-12T22:04:13.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:12 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:04:13.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:12 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:04:13.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:12 smithi186 ceph-mon[34785]: from='osd.8 [v2:172.21.15.186:6816/3007687277,v1:172.21.15.186:6817/3007687277]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-05-12T22:04:13.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:12 smithi186 ceph-mon[34785]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-05-12T22:04:13.179 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 12 22:04:12 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-8[53963]: 2024-05-12T22:04:12.789+0000 7f50091277c0 -1 osd.8 0 log_to_monitors true 2024-05-12T22:04:13.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:12 smithi086 ceph-mon[34297]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-05-12T22:04:13.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:12 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:04:13.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:12 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:04:13.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:12 smithi086 ceph-mon[34297]: from='osd.8 [v2:172.21.15.186:6816/3007687277,v1:172.21.15.186:6817/3007687277]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-05-12T22:04:13.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:12 smithi086 ceph-mon[34297]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-05-12T22:04:14.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:13 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:04:14.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:13 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:04:14.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:13 smithi186 ceph-mon[34785]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-05-12T22:04:14.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:13 smithi186 ceph-mon[34785]: osdmap e58: 9 total, 8 up, 9 in 2024-05-12T22:04:14.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:13 smithi186 ceph-mon[34785]: from='osd.8 [v2:172.21.15.186:6816/3007687277,v1:172.21.15.186:6817/3007687277]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-05-12T22:04:14.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:13 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-12T22:04:14.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:13 smithi186 ceph-mon[34785]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-05-12T22:04:14.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:13 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:04:14.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:13 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:04:14.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:13 smithi086 ceph-mon[34297]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-05-12T22:04:14.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:13 smithi086 ceph-mon[34297]: osdmap e58: 9 total, 8 up, 9 in 2024-05-12T22:04:14.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:13 smithi086 ceph-mon[34297]: from='osd.8 [v2:172.21.15.186:6816/3007687277,v1:172.21.15.186:6817/3007687277]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-05-12T22:04:14.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:13 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-12T22:04:14.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:13 smithi086 ceph-mon[34297]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-05-12T22:04:14.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:13 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:04:14.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:13 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:04:14.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:13 smithi028 ceph-mon[29847]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-05-12T22:04:14.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:13 smithi028 ceph-mon[29847]: osdmap e58: 9 total, 8 up, 9 in 2024-05-12T22:04:14.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:13 smithi028 ceph-mon[29847]: from='osd.8 [v2:172.21.15.186:6816/3007687277,v1:172.21.15.186:6817/3007687277]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-05-12T22:04:14.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:13 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-12T22:04:14.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:13 smithi028 ceph-mon[29847]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-05-12T22:04:14.929 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 12 22:04:14 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-8[53963]: 2024-05-12T22:04:14.808+0000 7f4ffdf8a700 -1 osd.8 0 waiting for initial osdmap 2024-05-12T22:04:14.929 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 12 22:04:14 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-8[53963]: 2024-05-12T22:04:14.824+0000 7f4ff6576700 -1 osd.8 59 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-12T22:04:15.046 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:15 smithi028 ceph-mon[29847]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-05-12T22:04:15.047 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:15 smithi028 ceph-mon[29847]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]': finished 2024-05-12T22:04:15.047 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:15 smithi028 ceph-mon[29847]: osdmap e59: 9 total, 8 up, 9 in 2024-05-12T22:04:15.047 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:15 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-12T22:04:15.047 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:15 smithi028 ceph-mon[29847]: from='osd.8 ' entity='osd.8' 2024-05-12T22:04:15.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:15 smithi186 ceph-mon[34785]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-05-12T22:04:15.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:15 smithi186 ceph-mon[34785]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]': finished 2024-05-12T22:04:15.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:15 smithi186 ceph-mon[34785]: osdmap e59: 9 total, 8 up, 9 in 2024-05-12T22:04:15.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:15 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-12T22:04:15.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:15 smithi186 ceph-mon[34785]: from='osd.8 ' entity='osd.8' 2024-05-12T22:04:15.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:15 smithi086 ceph-mon[34297]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-05-12T22:04:15.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:15 smithi086 ceph-mon[34297]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]': finished 2024-05-12T22:04:15.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:15 smithi086 ceph-mon[34297]: osdmap e59: 9 total, 8 up, 9 in 2024-05-12T22:04:15.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:15 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-12T22:04:15.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:15 smithi086 ceph-mon[34297]: from='osd.8 ' entity='osd.8' 2024-05-12T22:04:16.113 INFO:teuthology.orchestra.run.smithi028.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-05-12T22:04:16.113 INFO:teuthology.orchestra.run.smithi028.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-05-12T22:04:16.113 INFO:teuthology.orchestra.run.smithi028.stderr: stderr: 10+0 records in 2024-05-12T22:04:16.114 INFO:teuthology.orchestra.run.smithi028.stderr:10+0 records out 2024-05-12T22:04:16.114 INFO:teuthology.orchestra.run.smithi028.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0653477 s, 160 MB/s 2024-05-12T22:04:16.114 INFO:teuthology.orchestra.run.smithi028.stderr:--> Zapping successful for: 2024-05-12T22:04:16.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:16 smithi028 ceph-mon[29847]: purged_snaps scrub starts 2024-05-12T22:04:16.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:16 smithi028 ceph-mon[29847]: purged_snaps scrub ok 2024-05-12T22:04:16.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:16 smithi028 ceph-mon[29847]: osd.8 [v2:172.21.15.186:6816/3007687277,v1:172.21.15.186:6817/3007687277] boot 2024-05-12T22:04:16.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:16 smithi028 ceph-mon[29847]: osdmap e60: 9 total, 9 up, 9 in 2024-05-12T22:04:16.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:16 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-12T22:04:16.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:16 smithi028 ceph-mon[29847]: osdmap e61: 9 total, 9 up, 9 in 2024-05-12T22:04:16.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:16 smithi186 ceph-mon[34785]: purged_snaps scrub starts 2024-05-12T22:04:16.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:16 smithi186 ceph-mon[34785]: purged_snaps scrub ok 2024-05-12T22:04:16.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:16 smithi186 ceph-mon[34785]: osd.8 [v2:172.21.15.186:6816/3007687277,v1:172.21.15.186:6817/3007687277] boot 2024-05-12T22:04:16.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:16 smithi186 ceph-mon[34785]: osdmap e60: 9 total, 9 up, 9 in 2024-05-12T22:04:16.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:16 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-12T22:04:16.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:16 smithi186 ceph-mon[34785]: osdmap e61: 9 total, 9 up, 9 in 2024-05-12T22:04:16.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:16 smithi086 ceph-mon[34297]: purged_snaps scrub starts 2024-05-12T22:04:16.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:16 smithi086 ceph-mon[34297]: purged_snaps scrub ok 2024-05-12T22:04:16.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:16 smithi086 ceph-mon[34297]: osd.8 [v2:172.21.15.186:6816/3007687277,v1:172.21.15.186:6817/3007687277] boot 2024-05-12T22:04:16.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:16 smithi086 ceph-mon[34297]: osdmap e60: 9 total, 9 up, 9 in 2024-05-12T22:04:16.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:16 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-12T22:04:16.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:16 smithi086 ceph-mon[34297]: osdmap e61: 9 total, 9 up, 9 in 2024-05-12T22:04:17.016 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f72fecff68e1d400c4568684327c900485c20d6a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6b92e7b4-10aa-11ef-bc98-c7b262605968 -- ceph orch daemon add osd smithi028:vg_nvme/lv_1 2024-05-12T22:04:17.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:17 smithi186 ceph-mon[34785]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 652 MiB used, 804 GiB / 805 GiB avail 2024-05-12T22:04:17.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:17 smithi186 ceph-mon[34785]: Detected new or changed devices on smithi186 2024-05-12T22:04:17.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:17 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:04:17.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:17 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:04:17.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:17 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:04:17.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:17 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:04:17.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:17 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:04:17.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:17 smithi186 ceph-mon[34785]: Adjusting osd_memory_target on smithi186 to 5696M 2024-05-12T22:04:17.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:17 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:04:17.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:17 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:04:17.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:17 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:04:17.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:17 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:04:17.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:17 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:04:17.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:17 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:04:17.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:17 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:04:17.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:17 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:04:17.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:17 smithi186 ceph-mon[34785]: osdmap e62: 9 total, 9 up, 9 in 2024-05-12T22:04:17.475 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/mon.a/config 2024-05-12T22:04:17.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:17 smithi086 ceph-mon[34297]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 652 MiB used, 804 GiB / 805 GiB avail 2024-05-12T22:04:17.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:17 smithi086 ceph-mon[34297]: Detected new or changed devices on smithi186 2024-05-12T22:04:17.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:17 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:04:17.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:17 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:04:17.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:17 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:04:17.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:17 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:04:17.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:17 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:04:17.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:17 smithi086 ceph-mon[34297]: Adjusting osd_memory_target on smithi186 to 5696M 2024-05-12T22:04:17.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:17 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:04:17.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:17 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:04:17.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:17 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:04:17.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:17 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:04:17.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:17 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:04:17.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:17 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:04:17.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:17 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:04:17.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:17 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:04:17.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:17 smithi086 ceph-mon[34297]: osdmap e62: 9 total, 9 up, 9 in 2024-05-12T22:04:17.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:17 smithi028 ceph-mon[29847]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 652 MiB used, 804 GiB / 805 GiB avail 2024-05-12T22:04:17.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:17 smithi028 ceph-mon[29847]: Detected new or changed devices on smithi186 2024-05-12T22:04:17.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:17 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:04:17.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:17 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:04:17.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:17 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:04:17.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:17 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:04:17.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:17 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:04:17.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:17 smithi028 ceph-mon[29847]: Adjusting osd_memory_target on smithi186 to 5696M 2024-05-12T22:04:17.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:17 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:04:17.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:17 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:04:17.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:17 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:04:17.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:17 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:04:17.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:17 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:04:17.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:17 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:04:17.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:17 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:04:17.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:17 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:04:17.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:17 smithi028 ceph-mon[29847]: osdmap e62: 9 total, 9 up, 9 in 2024-05-12T22:04:19.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:19 smithi028 ceph-mon[29847]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 652 MiB used, 804 GiB / 805 GiB avail 2024-05-12T22:04:19.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:19 smithi028 ceph-mon[29847]: osdmap e63: 9 total, 9 up, 9 in 2024-05-12T22:04:19.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:19 smithi186 ceph-mon[34785]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 652 MiB used, 804 GiB / 805 GiB avail 2024-05-12T22:04:19.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:19 smithi186 ceph-mon[34785]: osdmap e63: 9 total, 9 up, 9 in 2024-05-12T22:04:19.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:19 smithi086 ceph-mon[34297]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 652 MiB used, 804 GiB / 805 GiB avail 2024-05-12T22:04:19.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:19 smithi086 ceph-mon[34297]: osdmap e63: 9 total, 9 up, 9 in 2024-05-12T22:04:19.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:04:19.662+0000 7f9647c00700 1 -- 172.21.15.28:0/1627278439 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9640103f20 msgr2=0x7f964010a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:04:19.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:04:19.662+0000 7f9647c00700 1 --2- 172.21.15.28:0/1627278439 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9640103f20 0x7f964010a7d0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f963800b0a0 tx=0x7f963800b3b0 comp rx=0 tx=0).stop 2024-05-12T22:04:19.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:04:19.662+0000 7f9647c00700 1 -- 172.21.15.28:0/1627278439 shutdown_connections 2024-05-12T22:04:19.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:04:19.662+0000 7f9647c00700 1 --2- 172.21.15.28:0/1627278439 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9640103f20 0x7f964010a7d0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:04:19.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:04:19.662+0000 7f9647c00700 1 --2- 172.21.15.28:0/1627278439 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9640102380 0x7f96401027a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:04:19.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:04:19.662+0000 7f9647c00700 1 --2- 172.21.15.28:0/1627278439 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9640103580 0x7f96401039e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:04:19.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:04:19.662+0000 7f9647c00700 1 -- 172.21.15.28:0/1627278439 >> 172.21.15.28:0/1627278439 conn(0x7f96400fd920 msgr2=0x7f96400ffd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:04:19.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:04:19.662+0000 7f9647c00700 1 -- 172.21.15.28:0/1627278439 shutdown_connections 2024-05-12T22:04:19.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:04:19.662+0000 7f9647c00700 1 -- 172.21.15.28:0/1627278439 wait complete. 2024-05-12T22:04:19.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:04:19.664+0000 7f9647c00700 1 Processor -- start 2024-05-12T22:04:19.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:04:19.664+0000 7f9647c00700 1 -- start start 2024-05-12T22:04:19.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:04:19.664+0000 7f9647c00700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9640102380 0x7f96400760c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:04:19.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:04:19.665+0000 7f9647c00700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9640103580 0x7f9640074710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:04:19.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:04:19.665+0000 7f9647c00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9640103f20 0x7f9640074c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:04:19.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:04:19.665+0000 7f9647c00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f964010dfb0 con 0x7f9640103f20 2024-05-12T22:04:19.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:04:19.665+0000 7f9647c00700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f964010de30 con 0x7f9640102380 2024-05-12T22:04:19.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:04:19.665+0000 7f9647c00700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f964010e130 con 0x7f9640103580 2024-05-12T22:04:19.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:04:19.665+0000 7f9645c00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9640103f20 0x7f9640074c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:04:19.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:04:19.665+0000 7f9645200700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9640102380 0x7f96400760c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:04:19.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:04:19.665+0000 7f963fe00700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9640103580 0x7f9640074710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:04:19.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:04:19.665+0000 7f9645c00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9640103f20 0x7f9640074c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54280/0 (socket says 172.21.15.28:54280) 2024-05-12T22:04:19.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:04:19.665+0000 7f9645200700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9640102380 0x7f96400760c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:59092/0 (socket says 172.21.15.28:59092) 2024-05-12T22:04:19.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:04:19.665+0000 7f9645c00700 1 -- 172.21.15.28:0/1338207042 learned_addr learned my addr 172.21.15.28:0/1338207042 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:04:19.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:04:19.665+0000 7f963fe00700 1 -- 172.21.15.28:0/1338207042 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9640102380 msgr2=0x7f96400760c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:04:19.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:04:19.665+0000 7f963fe00700 1 --2- 172.21.15.28:0/1338207042 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9640102380 0x7f96400760c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:04:19.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:04:19.665+0000 7f963fe00700 1 -- 172.21.15.28:0/1338207042 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9640103f20 msgr2=0x7f9640074c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:04:19.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:04:19.665+0000 7f963fe00700 1 --2- 172.21.15.28:0/1338207042 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9640103f20 0x7f9640074c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:04:19.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:04:19.666+0000 7f963fe00700 1 -- 172.21.15.28:0/1338207042 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9640075500 con 0x7f9640103580 2024-05-12T22:04:19.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:04:19.666+0000 7f9645c00700 1 --2- 172.21.15.28:0/1338207042 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9640103f20 0x7f9640074c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:04:19.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:04:19.666+0000 7f963fe00700 1 --2- 172.21.15.28:0/1338207042 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9640103580 0x7f9640074710 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f963000e9f0 tx=0x7f963000ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:04:19.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:04:19.666+0000 7f963d600700 1 -- 172.21.15.28:0/1338207042 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f963000c9c0 con 0x7f9640103580 2024-05-12T22:04:19.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:04:19.666+0000 7f963d600700 1 -- 172.21.15.28:0/1338207042 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f963000cb60 con 0x7f9640103580 2024-05-12T22:04:19.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:04:19.666+0000 7f963d600700 1 -- 172.21.15.28:0/1338207042 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9630018680 con 0x7f9640103580 2024-05-12T22:04:19.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:04:19.666+0000 7f9647c00700 1 -- 172.21.15.28:0/1338207042 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96401a2700 con 0x7f9640103580 2024-05-12T22:04:19.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:04:19.666+0000 7f9647c00700 1 -- 172.21.15.28:0/1338207042 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96401a2c40 con 0x7f9640103580 2024-05-12T22:04:19.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:04:19.667+0000 7f9647c00700 1 -- 172.21.15.28:0/1338207042 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9640069010 con 0x7f9640103580 2024-05-12T22:04:19.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:04:19.670+0000 7f963d600700 1 -- 172.21.15.28:0/1338207042 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f9630010430 con 0x7f9640103580 2024-05-12T22:04:19.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:04:19.671+0000 7f963d600700 1 --2- 172.21.15.28:0/1338207042 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f96280a2ca0 0x7f96280a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:04:19.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:04:19.671+0000 7f963d600700 1 -- 172.21.15.28:0/1338207042 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 5829+0+0 (secure 0 0 0) 0x7f96300ceac0 con 0x7f9640103580 2024-05-12T22:04:19.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:04:19.671+0000 7f9645200700 1 --2- 172.21.15.28:0/1338207042 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f96280a2ca0 0x7f96280a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:04:19.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:04:19.672+0000 7f9645200700 1 --2- 172.21.15.28:0/1338207042 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f96280a2ca0 0x7f96280a5160 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f96401033e0 tx=0x7f9634009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:04:19.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:04:19.673+0000 7f963d600700 1 -- 172.21.15.28:0/1338207042 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f963009c330 con 0x7f9640103580 2024-05-12T22:04:19.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:04:19.913+0000 7f9647c00700 1 -- 172.21.15.28:0/1338207042 --> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f9640062760 con 0x7f96280a2ca0 2024-05-12T22:04:20.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:20 smithi028 ceph-mon[29847]: osdmap e64: 9 total, 9 up, 9 in 2024-05-12T22:04:20.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:20 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-12T22:04:20.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:20 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-12T22:04:20.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:20 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:04:20.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:20 smithi186 ceph-mon[34785]: osdmap e64: 9 total, 9 up, 9 in 2024-05-12T22:04:20.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:20 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-12T22:04:20.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:20 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-12T22:04:20.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:20 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:04:20.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:20 smithi086 ceph-mon[34297]: osdmap e64: 9 total, 9 up, 9 in 2024-05-12T22:04:20.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:20 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-12T22:04:20.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:20 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-12T22:04:20.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:20 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:04:21.220 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:21 smithi028 ceph-mon[29847]: pgmap v221: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 652 MiB used, 804 GiB / 805 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-05-12T22:04:21.221 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:21 smithi028 ceph-mon[29847]: from='client.24322 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-12T22:04:21.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:21 smithi186 ceph-mon[34785]: pgmap v221: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 652 MiB used, 804 GiB / 805 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-05-12T22:04:21.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:21 smithi186 ceph-mon[34785]: from='client.24322 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-12T22:04:21.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:21 smithi086 ceph-mon[34297]: pgmap v221: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 652 MiB used, 804 GiB / 805 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-05-12T22:04:21.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:21 smithi086 ceph-mon[34297]: from='client.24322 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-12T22:04:23.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:23 smithi186 ceph-mon[34785]: pgmap v222: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-05-12T22:04:23.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:23 smithi086 ceph-mon[34297]: pgmap v222: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-05-12T22:04:23.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:23 smithi028 ceph-mon[29847]: pgmap v222: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-05-12T22:04:24.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:24 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/1813613280' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "38ff7d9d-ee85-404c-8c59-adc327e1ccfa"}]: dispatch 2024-05-12T22:04:24.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:24 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/1813613280' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "38ff7d9d-ee85-404c-8c59-adc327e1ccfa"}]': finished 2024-05-12T22:04:24.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:24 smithi186 ceph-mon[34785]: osdmap e65: 10 total, 9 up, 10 in 2024-05-12T22:04:24.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:24 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-12T22:04:24.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:24 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/1813613280' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "38ff7d9d-ee85-404c-8c59-adc327e1ccfa"}]: dispatch 2024-05-12T22:04:24.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:24 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/1813613280' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "38ff7d9d-ee85-404c-8c59-adc327e1ccfa"}]': finished 2024-05-12T22:04:24.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:24 smithi086 ceph-mon[34297]: osdmap e65: 10 total, 9 up, 10 in 2024-05-12T22:04:24.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:24 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-12T22:04:24.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:24 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/1813613280' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "38ff7d9d-ee85-404c-8c59-adc327e1ccfa"}]: dispatch 2024-05-12T22:04:24.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:24 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/1813613280' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "38ff7d9d-ee85-404c-8c59-adc327e1ccfa"}]': finished 2024-05-12T22:04:24.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:24 smithi028 ceph-mon[29847]: osdmap e65: 10 total, 9 up, 10 in 2024-05-12T22:04:24.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:24 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-12T22:04:25.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:25 smithi186 ceph-mon[34785]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-05-12T22:04:25.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:25 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/3442998647' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-12T22:04:25.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:25 smithi086 ceph-mon[34297]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-05-12T22:04:25.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:25 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/3442998647' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-12T22:04:25.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:25 smithi028 ceph-mon[29847]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-05-12T22:04:25.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:25 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/3442998647' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-12T22:04:27.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:27 smithi186 ceph-mon[34785]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-05-12T22:04:27.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:27 smithi086 ceph-mon[34297]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-05-12T22:04:27.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:27 smithi028 ceph-mon[29847]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-05-12T22:04:29.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:29 smithi186 ceph-mon[34785]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-05-12T22:04:29.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:29 smithi086 ceph-mon[34297]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-05-12T22:04:29.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:29 smithi028 ceph-mon[29847]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-05-12T22:04:31.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:31 smithi028 ceph-mon[29847]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-05-12T22:04:31.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:31 smithi186 ceph-mon[34785]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-05-12T22:04:31.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:31 smithi086 ceph-mon[34297]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-05-12T22:04:33.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:33 smithi028 ceph-mon[29847]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-05-12T22:04:33.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:33 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-05-12T22:04:33.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:33 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:04:33.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:33 smithi186 ceph-mon[34785]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-05-12T22:04:33.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:33 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-05-12T22:04:33.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:33 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:04:33.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:33 smithi086 ceph-mon[34297]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-05-12T22:04:33.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:33 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-05-12T22:04:33.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:33 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:04:34.419 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:34 smithi028 ceph-mon[29847]: Deploying daemon osd.9 on smithi028 2024-05-12T22:04:34.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:34 smithi186 ceph-mon[34785]: Deploying daemon osd.9 on smithi028 2024-05-12T22:04:34.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:34 smithi086 ceph-mon[34297]: Deploying daemon osd.9 on smithi028 2024-05-12T22:04:35.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:35 smithi186 ceph-mon[34785]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-05-12T22:04:35.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:35 smithi086 ceph-mon[34297]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-05-12T22:04:35.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:35 smithi028 ceph-mon[29847]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-05-12T22:04:37.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:37 smithi186 ceph-mon[34785]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-05-12T22:04:37.491 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:37 smithi028 ceph-mon[29847]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-05-12T22:04:37.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:37 smithi086 ceph-mon[34297]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-05-12T22:04:39.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:39 smithi186 ceph-mon[34785]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-05-12T22:04:39.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:39 smithi086 ceph-mon[34297]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-05-12T22:04:39.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:39 smithi028 ceph-mon[29847]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-05-12T22:04:41.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:41 smithi186 ceph-mon[34785]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-05-12T22:04:41.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:41 smithi086 ceph-mon[34297]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-05-12T22:04:41.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:41 smithi028 ceph-mon[29847]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-05-12T22:04:42.663 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:42 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:04:42.663 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:42 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:04:42.663 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:42 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:04:42.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:42 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:04:42.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:42 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:04:42.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:42 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:04:42.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:42 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:04:42.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:42 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:04:42.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:42 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:04:43.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:43 smithi028 ceph-mon[29847]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-05-12T22:04:43.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:43 smithi186 ceph-mon[34785]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-05-12T22:04:43.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:43 smithi086 ceph-mon[34297]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-05-12T22:04:45.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:45 smithi186 ceph-mon[34785]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-05-12T22:04:45.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:45 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:04:45.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:45 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:04:45.709 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:45 smithi028 ceph-mon[29847]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-05-12T22:04:45.709 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:45 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:04:45.709 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:45 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:04:45.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:45 smithi086 ceph-mon[34297]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-05-12T22:04:45.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:45 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:04:45.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:45 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:04:46.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:04:46.275+0000 7f963d600700 1 -- 172.21.15.28:0/1338207042 <== mgr.14152 v2:172.21.15.28:6800/3123328651 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f9640062760 con 0x7f96280a2ca0 2024-05-12T22:04:46.277 INFO:teuthology.orchestra.run.smithi028.stdout:Created osd(s) 9 on host 'smithi028' 2024-05-12T22:04:46.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:04:46.279+0000 7f9647c00700 1 -- 172.21.15.28:0/1338207042 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f96280a2ca0 msgr2=0x7f96280a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:04:46.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:04:46.279+0000 7f9647c00700 1 --2- 172.21.15.28:0/1338207042 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f96280a2ca0 0x7f96280a5160 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f96401033e0 tx=0x7f9634009260 comp rx=0 tx=0).stop 2024-05-12T22:04:46.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:04:46.279+0000 7f9647c00700 1 -- 172.21.15.28:0/1338207042 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9640103580 msgr2=0x7f9640074710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:04:46.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:04:46.279+0000 7f9647c00700 1 --2- 172.21.15.28:0/1338207042 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9640103580 0x7f9640074710 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f963000e9f0 tx=0x7f963000ed00 comp rx=0 tx=0).stop 2024-05-12T22:04:46.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:04:46.279+0000 7f9647c00700 1 -- 172.21.15.28:0/1338207042 shutdown_connections 2024-05-12T22:04:46.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:04:46.279+0000 7f9647c00700 1 --2- 172.21.15.28:0/1338207042 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f96280a2ca0 0x7f96280a5160 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:04:46.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:04:46.279+0000 7f9647c00700 1 --2- 172.21.15.28:0/1338207042 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9640103f20 0x7f9640074c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:04:46.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:04:46.279+0000 7f9647c00700 1 --2- 172.21.15.28:0/1338207042 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9640102380 0x7f96400760c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:04:46.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:04:46.279+0000 7f9647c00700 1 --2- 172.21.15.28:0/1338207042 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9640103580 0x7f9640074710 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:04:46.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:04:46.279+0000 7f9647c00700 1 -- 172.21.15.28:0/1338207042 >> 172.21.15.28:0/1338207042 conn(0x7f96400fd920 msgr2=0x7f9640106880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:04:46.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:04:46.279+0000 7f9647c00700 1 -- 172.21.15.28:0/1338207042 shutdown_connections 2024-05-12T22:04:46.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:04:46.279+0000 7f9647c00700 1 -- 172.21.15.28:0/1338207042 wait complete. 2024-05-12T22:04:46.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:46 smithi028 ceph-mon[29847]: from='osd.9 [v2:172.21.15.28:6826/136220472,v1:172.21.15.28:6827/136220472]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-05-12T22:04:46.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:46 smithi028 ceph-mon[29847]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-05-12T22:04:46.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:46 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:04:46.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:46 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:04:46.681 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:46 smithi186 ceph-mon[34785]: from='osd.9 [v2:172.21.15.28:6826/136220472,v1:172.21.15.28:6827/136220472]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-05-12T22:04:46.681 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:46 smithi186 ceph-mon[34785]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-05-12T22:04:46.681 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:46 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:04:46.681 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:46 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:04:46.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:46 smithi086 ceph-mon[34297]: from='osd.9 [v2:172.21.15.28:6826/136220472,v1:172.21.15.28:6827/136220472]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-05-12T22:04:46.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:46 smithi086 ceph-mon[34297]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-05-12T22:04:46.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:46 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:04:46.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:46 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:04:47.030 DEBUG:teuthology.orchestra.run.smithi028:osd.9> sudo journalctl -f -n 0 -u ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968@osd.9.service 2024-05-12T22:04:47.036 INFO:tasks.cephadm:Deploying osd.10 on smithi086 with /dev/vg_nvme/lv_1... 2024-05-12T22:04:47.036 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f72fecff68e1d400c4568684327c900485c20d6a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6b92e7b4-10aa-11ef-bc98-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-05-12T22:04:47.281 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/mon.b/config 2024-05-12T22:04:47.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:47 smithi186 ceph-mon[34785]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-05-12T22:04:47.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:47 smithi186 ceph-mon[34785]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-05-12T22:04:47.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:47 smithi186 ceph-mon[34785]: from='osd.9 [v2:172.21.15.28:6826/136220472,v1:172.21.15.28:6827/136220472]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-05-12T22:04:47.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:47 smithi186 ceph-mon[34785]: osdmap e66: 10 total, 9 up, 10 in 2024-05-12T22:04:47.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:47 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-12T22:04:47.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:47 smithi186 ceph-mon[34785]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-05-12T22:04:47.729 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:47 smithi028 ceph-mon[29847]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-05-12T22:04:47.729 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:47 smithi028 ceph-mon[29847]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-05-12T22:04:47.729 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:47 smithi028 ceph-mon[29847]: from='osd.9 [v2:172.21.15.28:6826/136220472,v1:172.21.15.28:6827/136220472]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-05-12T22:04:47.729 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:47 smithi028 ceph-mon[29847]: osdmap e66: 10 total, 9 up, 10 in 2024-05-12T22:04:47.730 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:47 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-12T22:04:47.730 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:47 smithi028 ceph-mon[29847]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-05-12T22:04:47.730 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:04:47 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:04:47.444+0000 7f0879600700 -1 osd.9 0 waiting for initial osdmap 2024-05-12T22:04:47.730 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:04:47 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:04:47.465+0000 7f0874e00700 -1 osd.9 67 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-12T22:04:47.732 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:47 smithi086 ceph-mon[34297]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-05-12T22:04:47.732 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:47 smithi086 ceph-mon[34297]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-05-12T22:04:47.732 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:47 smithi086 ceph-mon[34297]: from='osd.9 [v2:172.21.15.28:6826/136220472,v1:172.21.15.28:6827/136220472]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-05-12T22:04:47.732 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:47 smithi086 ceph-mon[34297]: osdmap e66: 10 total, 9 up, 10 in 2024-05-12T22:04:47.732 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:47 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-12T22:04:47.732 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:47 smithi086 ceph-mon[34297]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-05-12T22:04:48.665 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:48 smithi086 ceph-mon[34297]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-05-12T22:04:48.665 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:48 smithi086 ceph-mon[34297]: osdmap e67: 10 total, 9 up, 10 in 2024-05-12T22:04:48.665 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:48 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-12T22:04:48.665 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:48 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-12T22:04:48.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:48 smithi028 ceph-mon[29847]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-05-12T22:04:48.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:48 smithi028 ceph-mon[29847]: osdmap e67: 10 total, 9 up, 10 in 2024-05-12T22:04:48.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:48 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-12T22:04:48.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:48 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-12T22:04:48.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:48 smithi186 ceph-mon[34785]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-05-12T22:04:48.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:48 smithi186 ceph-mon[34785]: osdmap e67: 10 total, 9 up, 10 in 2024-05-12T22:04:48.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:48 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-12T22:04:48.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:48 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-12T22:04:49.232 INFO:teuthology.orchestra.run.smithi086.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-05-12T22:04:49.232 INFO:teuthology.orchestra.run.smithi086.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-05-12T22:04:49.232 INFO:teuthology.orchestra.run.smithi086.stderr: stderr: 10+0 records in 2024-05-12T22:04:49.232 INFO:teuthology.orchestra.run.smithi086.stderr:10+0 records out 2024-05-12T22:04:49.232 INFO:teuthology.orchestra.run.smithi086.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0129593 s, 809 MB/s 2024-05-12T22:04:49.232 INFO:teuthology.orchestra.run.smithi086.stderr:--> Zapping successful for: 2024-05-12T22:04:49.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:49 smithi028 ceph-mon[29847]: purged_snaps scrub starts 2024-05-12T22:04:49.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:49 smithi028 ceph-mon[29847]: purged_snaps scrub ok 2024-05-12T22:04:49.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:49 smithi028 ceph-mon[29847]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-05-12T22:04:49.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:49 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-12T22:04:49.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:49 smithi028 ceph-mon[29847]: osd.9 [v2:172.21.15.28:6826/136220472,v1:172.21.15.28:6827/136220472] boot 2024-05-12T22:04:49.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:49 smithi028 ceph-mon[29847]: osdmap e68: 10 total, 10 up, 10 in 2024-05-12T22:04:49.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:49 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-12T22:04:49.537 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:49 smithi086 ceph-mon[34297]: purged_snaps scrub starts 2024-05-12T22:04:49.538 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:49 smithi086 ceph-mon[34297]: purged_snaps scrub ok 2024-05-12T22:04:49.538 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:49 smithi086 ceph-mon[34297]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-05-12T22:04:49.538 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:49 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-12T22:04:49.538 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:49 smithi086 ceph-mon[34297]: osd.9 [v2:172.21.15.28:6826/136220472,v1:172.21.15.28:6827/136220472] boot 2024-05-12T22:04:49.538 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:49 smithi086 ceph-mon[34297]: osdmap e68: 10 total, 10 up, 10 in 2024-05-12T22:04:49.538 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:49 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-12T22:04:49.862 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f72fecff68e1d400c4568684327c900485c20d6a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6b92e7b4-10aa-11ef-bc98-c7b262605968 -- ceph orch daemon add osd smithi086:vg_nvme/lv_1 2024-05-12T22:04:49.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:49 smithi186 ceph-mon[34785]: purged_snaps scrub starts 2024-05-12T22:04:49.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:49 smithi186 ceph-mon[34785]: purged_snaps scrub ok 2024-05-12T22:04:49.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:49 smithi186 ceph-mon[34785]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-05-12T22:04:49.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:49 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-12T22:04:49.930 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:49 smithi186 ceph-mon[34785]: osd.9 [v2:172.21.15.28:6826/136220472,v1:172.21.15.28:6827/136220472] boot 2024-05-12T22:04:49.930 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:49 smithi186 ceph-mon[34785]: osdmap e68: 10 total, 10 up, 10 in 2024-05-12T22:04:49.930 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:49 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-12T22:04:50.114 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/mon.b/config 2024-05-12T22:04:50.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:50 smithi086 ceph-mon[34297]: osdmap e69: 10 total, 10 up, 10 in 2024-05-12T22:04:50.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:50 smithi028 ceph-mon[29847]: osdmap e69: 10 total, 10 up, 10 in 2024-05-12T22:04:50.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:50 smithi186 ceph-mon[34785]: osdmap e69: 10 total, 10 up, 10 in 2024-05-12T22:04:51.641 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:04:51.639+0000 7fb81e39d700 1 -- 172.21.15.86:0/82789298 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb818102480 msgr2=0x7fb8181028e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:04:51.642 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:04:51.639+0000 7fb81e39d700 1 --2- 172.21.15.86:0/82789298 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb818102480 0x7fb8181028e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fb80c0099c0 tx=0x7fb80c009cd0 comp rx=0 tx=0).stop 2024-05-12T22:04:51.642 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:04:51.639+0000 7fb81e39d700 1 -- 172.21.15.86:0/82789298 shutdown_connections 2024-05-12T22:04:51.642 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:04:51.639+0000 7fb81e39d700 1 --2- 172.21.15.86:0/82789298 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb818101280 0x7fb8181016a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:04:51.642 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:04:51.639+0000 7fb81e39d700 1 --2- 172.21.15.86:0/82789298 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb818102e20 0x7fb8181096d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:04:51.642 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:04:51.639+0000 7fb81e39d700 1 --2- 172.21.15.86:0/82789298 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb818102480 0x7fb8181028e0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:04:51.642 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:04:51.639+0000 7fb81e39d700 1 -- 172.21.15.86:0/82789298 >> 172.21.15.86:0/82789298 conn(0x7fb8180fc840 msgr2=0x7fb8180fec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:04:51.642 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:04:51.639+0000 7fb81e39d700 1 -- 172.21.15.86:0/82789298 shutdown_connections 2024-05-12T22:04:51.642 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:04:51.639+0000 7fb81e39d700 1 -- 172.21.15.86:0/82789298 wait complete. 2024-05-12T22:04:51.643 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:04:51.639+0000 7fb81e39d700 1 Processor -- start 2024-05-12T22:04:51.643 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:04:51.639+0000 7fb81e39d700 1 -- start start 2024-05-12T22:04:51.643 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:04:51.640+0000 7fb81e39d700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb818101280 0x7fb8181985d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:04:51.643 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:04:51.640+0000 7fb81e39d700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb818102480 0x7fb818198b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:04:51.643 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:04:51.640+0000 7fb81e39d700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb818102e20 0x7fb81819fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:04:51.643 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:04:51.640+0000 7fb81e39d700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb81810cec0 con 0x7fb818102e20 2024-05-12T22:04:51.643 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:04:51.640+0000 7fb81e39d700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fb81810cd40 con 0x7fb818102480 2024-05-12T22:04:51.643 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:04:51.640+0000 7fb81e39d700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb81810d040 con 0x7fb818101280 2024-05-12T22:04:51.643 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:04:51.640+0000 7fb8177fe700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb818102480 0x7fb818198b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:04:51.643 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:04:51.640+0000 7fb8177fe700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb818102480 0x7fb818198b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:38774/0 (socket says 172.21.15.86:38774) 2024-05-12T22:04:51.643 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:04:51.640+0000 7fb8177fe700 1 -- 172.21.15.86:0/3079459989 learned_addr learned my addr 172.21.15.86:0/3079459989 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-05-12T22:04:51.643 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:04:51.640+0000 7fb817fff700 1 --2- 172.21.15.86:0/3079459989 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb818101280 0x7fb8181985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:04:51.643 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:04:51.640+0000 7fb8177fe700 1 -- 172.21.15.86:0/3079459989 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb818101280 msgr2=0x7fb8181985d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:04:51.643 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:04:51.640+0000 7fb8177fe700 1 --2- 172.21.15.86:0/3079459989 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb818101280 0x7fb8181985d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:04:51.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:04:51.640+0000 7fb8177fe700 1 -- 172.21.15.86:0/3079459989 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb818102e20 msgr2=0x7fb81819fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:04:51.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:04:51.640+0000 7fb8177fe700 1 --2- 172.21.15.86:0/3079459989 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb818102e20 0x7fb81819fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:04:51.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:04:51.640+0000 7fb8177fe700 1 -- 172.21.15.86:0/3079459989 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8181a0230 con 0x7fb818102480 2024-05-12T22:04:51.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:04:51.640+0000 7fb817fff700 1 --2- 172.21.15.86:0/3079459989 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb818101280 0x7fb8181985d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-12T22:04:51.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:04:51.640+0000 7fb8177fe700 1 --2- 172.21.15.86:0/3079459989 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb818102480 0x7fb818198b10 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fb80c003890 tx=0x7fb80c003730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:04:51.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:04:51.640+0000 7fb8157fa700 1 -- 172.21.15.86:0/3079459989 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb80c027070 con 0x7fb818102480 2024-05-12T22:04:51.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:04:51.641+0000 7fb8157fa700 1 -- 172.21.15.86:0/3079459989 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb80c022470 con 0x7fb818102480 2024-05-12T22:04:51.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:04:51.641+0000 7fb81e39d700 1 -- 172.21.15.86:0/3079459989 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8181a04c0 con 0x7fb818102480 2024-05-12T22:04:51.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:04:51.641+0000 7fb8157fa700 1 -- 172.21.15.86:0/3079459989 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb80c00f6a0 con 0x7fb818102480 2024-05-12T22:04:51.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:04:51.641+0000 7fb81e39d700 1 -- 172.21.15.86:0/3079459989 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8181a0990 con 0x7fb818102480 2024-05-12T22:04:51.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:04:51.641+0000 7fb81e39d700 1 -- 172.21.15.86:0/3079459989 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb818069010 con 0x7fb818102480 2024-05-12T22:04:51.645 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:04:51.642+0000 7fb8157fa700 1 -- 172.21.15.86:0/3079459989 <== mon.2 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fb80c02a070 con 0x7fb818102480 2024-05-12T22:04:51.645 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:04:51.643+0000 7fb8157fa700 1 --2- 172.21.15.86:0/3079459989 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fb8000a2c20 0x7fb8000a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:04:51.645 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:04:51.643+0000 7fb8157fa700 1 -- 172.21.15.86:0/3079459989 <== mon.2 v2:172.21.15.86:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 6261+0+0 (secure 0 0 0) 0x7fb80c0cf790 con 0x7fb818102480 2024-05-12T22:04:51.645 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:04:51.644+0000 7fb817fff700 1 --2- 172.21.15.86:0/3079459989 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fb8000a2c20 0x7fb8000a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:04:51.646 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:04:51.644+0000 7fb817fff700 1 --2- 172.21.15.86:0/3079459989 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fb8000a2c20 0x7fb8000a50e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fb808005fd0 tx=0x7fb808005dc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:04:51.647 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:04:51.647+0000 7fb8157fa700 1 -- 172.21.15.86:0/3079459989 <== mon.2 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fb80c013070 con 0x7fb818102480 2024-05-12T22:04:51.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:51 smithi086 ceph-mon[34297]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-05-12T22:04:51.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:51 smithi028 ceph-mon[29847]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-05-12T22:04:51.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:04:51.854+0000 7fb81e39d700 1 -- 172.21.15.86:0/3079459989 --> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7fb8180fea30 con 0x7fb8000a2c20 2024-05-12T22:04:51.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:51 smithi186 ceph-mon[34785]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-05-12T22:04:52.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:52 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-12T22:04:52.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:52 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-12T22:04:52.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:52 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:04:52.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:52 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:04:52.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:52 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:04:52.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:52 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:04:52.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:52 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:04:52.743 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:52 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:04:52.743 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:52 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:04:52.743 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:52 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:04:52.743 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:52 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:04:52.743 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:52 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:04:52.743 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:52 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:04:52.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:52 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-12T22:04:52.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:52 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-12T22:04:52.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:52 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:04:52.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:52 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:04:52.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:52 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:04:52.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:52 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:04:52.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:52 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:04:52.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:52 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:04:52.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:52 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:04:52.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:52 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:04:52.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:52 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:04:52.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:52 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:04:52.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:52 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:04:52.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:52 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-12T22:04:52.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:52 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-12T22:04:52.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:52 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:04:52.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:52 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:04:52.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:52 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:04:52.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:52 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:04:52.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:52 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:04:52.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:52 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:04:52.930 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:52 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:04:52.930 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:52 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:04:52.930 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:52 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:04:52.930 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:52 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:04:52.930 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:52 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:04:53.682 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:53 smithi086 ceph-mon[34297]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-05-12T22:04:53.682 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:53 smithi086 ceph-mon[34297]: from='client.24356 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-12T22:04:53.683 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:53 smithi086 ceph-mon[34297]: Detected new or changed devices on smithi028 2024-05-12T22:04:53.683 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:53 smithi086 ceph-mon[34297]: Adjusting osd_memory_target on smithi028 to 4273M 2024-05-12T22:04:53.683 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:53 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:04:53.683 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:53 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:04:53.683 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:53 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:04:53.683 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:53 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:04:53.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:53 smithi028 ceph-mon[29847]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-05-12T22:04:53.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:53 smithi028 ceph-mon[29847]: from='client.24356 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-12T22:04:53.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:53 smithi028 ceph-mon[29847]: Detected new or changed devices on smithi028 2024-05-12T22:04:53.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:53 smithi028 ceph-mon[29847]: Adjusting osd_memory_target on smithi028 to 4273M 2024-05-12T22:04:53.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:53 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:04:53.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:53 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:04:53.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:53 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:04:53.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:53 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:04:53.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:53 smithi186 ceph-mon[34785]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-05-12T22:04:53.982 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:53 smithi186 ceph-mon[34785]: from='client.24356 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-12T22:04:53.983 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:53 smithi186 ceph-mon[34785]: Detected new or changed devices on smithi028 2024-05-12T22:04:53.983 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:53 smithi186 ceph-mon[34785]: Adjusting osd_memory_target on smithi028 to 4273M 2024-05-12T22:04:53.983 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:53 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:04:53.983 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:53 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:04:53.984 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:53 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:04:53.984 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:53 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:04:54.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:54 smithi086 ceph-mon[34297]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-05-12T22:04:54.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:54 smithi028 ceph-mon[29847]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-05-12T22:04:54.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:54 smithi186 ceph-mon[34785]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-05-12T22:04:55.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:55 smithi028 ceph-mon[29847]: from='client.? 172.21.15.86:0/3280789808' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5e9ff657-afa8-4bab-8491-f209038392b4"}]: dispatch 2024-05-12T22:04:55.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:55 smithi028 ceph-mon[29847]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5e9ff657-afa8-4bab-8491-f209038392b4"}]: dispatch 2024-05-12T22:04:55.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:55 smithi028 ceph-mon[29847]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5e9ff657-afa8-4bab-8491-f209038392b4"}]': finished 2024-05-12T22:04:55.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:55 smithi028 ceph-mon[29847]: osdmap e70: 11 total, 10 up, 11 in 2024-05-12T22:04:55.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:55 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-12T22:04:55.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:55 smithi028 ceph-mon[29847]: from='client.? 172.21.15.86:0/4189741794' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-12T22:04:55.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:55 smithi186 ceph-mon[34785]: from='client.? 172.21.15.86:0/3280789808' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5e9ff657-afa8-4bab-8491-f209038392b4"}]: dispatch 2024-05-12T22:04:55.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:55 smithi186 ceph-mon[34785]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5e9ff657-afa8-4bab-8491-f209038392b4"}]: dispatch 2024-05-12T22:04:55.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:55 smithi186 ceph-mon[34785]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5e9ff657-afa8-4bab-8491-f209038392b4"}]': finished 2024-05-12T22:04:55.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:55 smithi186 ceph-mon[34785]: osdmap e70: 11 total, 10 up, 11 in 2024-05-12T22:04:55.930 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:55 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-12T22:04:55.930 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:55 smithi186 ceph-mon[34785]: from='client.? 172.21.15.86:0/4189741794' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-12T22:04:55.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:55 smithi086 ceph-mon[34297]: from='client.? 172.21.15.86:0/3280789808' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5e9ff657-afa8-4bab-8491-f209038392b4"}]: dispatch 2024-05-12T22:04:55.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:55 smithi086 ceph-mon[34297]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5e9ff657-afa8-4bab-8491-f209038392b4"}]: dispatch 2024-05-12T22:04:55.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:55 smithi086 ceph-mon[34297]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5e9ff657-afa8-4bab-8491-f209038392b4"}]': finished 2024-05-12T22:04:55.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:55 smithi086 ceph-mon[34297]: osdmap e70: 11 total, 10 up, 11 in 2024-05-12T22:04:55.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:55 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-12T22:04:55.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:55 smithi086 ceph-mon[34297]: from='client.? 172.21.15.86:0/4189741794' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-12T22:04:56.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:56 smithi028 ceph-mon[29847]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-05-12T22:04:56.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:56 smithi186 ceph-mon[34785]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-05-12T22:04:56.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:56 smithi086 ceph-mon[34297]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-05-12T22:04:58.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:04:58 smithi086 ceph-mon[34297]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-05-12T22:04:59.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:04:58 smithi028 ceph-mon[29847]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-05-12T22:04:59.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:04:58 smithi186 ceph-mon[34785]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-05-12T22:05:00.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:00 smithi086 ceph-mon[34297]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-05-12T22:05:01.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:00 smithi028 ceph-mon[29847]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-05-12T22:05:01.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:00 smithi186 ceph-mon[34785]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-05-12T22:05:02.978 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:02 smithi086 ceph-mon[34297]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-05-12T22:05:02.979 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:02 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-05-12T22:05:02.979 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:02 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:05:02.979 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:02 smithi086 ceph-mon[34297]: Deploying daemon osd.10 on smithi086 2024-05-12T22:05:03.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:02 smithi028 ceph-mon[29847]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-05-12T22:05:03.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:02 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-05-12T22:05:03.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:02 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:05:03.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:02 smithi028 ceph-mon[29847]: Deploying daemon osd.10 on smithi086 2024-05-12T22:05:03.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:02 smithi186 ceph-mon[34785]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-05-12T22:05:03.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:02 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-05-12T22:05:03.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:02 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:05:03.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:02 smithi186 ceph-mon[34785]: Deploying daemon osd.10 on smithi086 2024-05-12T22:05:04.987 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:04 smithi086 ceph-mon[34297]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-05-12T22:05:05.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:04 smithi028 ceph-mon[29847]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-05-12T22:05:05.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:04 smithi186 ceph-mon[34785]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-05-12T22:05:06.843 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:06 smithi086 ceph-mon[34297]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-05-12T22:05:07.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:06 smithi028 ceph-mon[29847]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-05-12T22:05:07.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:06 smithi186 ceph-mon[34785]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-05-12T22:05:08.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:08 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:05:08.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:08 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:05:08.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:08 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:05:08.688 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:08 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:05:08.688 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:08 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:05:08.688 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:08 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:05:08.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:08 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:05:08.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:08 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:05:08.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:08 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:05:09.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:09 smithi086 ceph-mon[34297]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-05-12T22:05:09.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:09 smithi028 ceph-mon[29847]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-05-12T22:05:09.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:09 smithi186 ceph-mon[34785]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-05-12T22:05:10.991 INFO:teuthology.orchestra.run.smithi086.stdout:Created osd(s) 10 on host 'smithi086' 2024-05-12T22:05:10.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:05:10.990+0000 7fb8157fa700 1 -- 172.21.15.86:0/3079459989 <== mgr.14152 v2:172.21.15.28:6800/3123328651 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7fb8180fea30 con 0x7fb8000a2c20 2024-05-12T22:05:10.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:05:10.992+0000 7fb81e39d700 1 -- 172.21.15.86:0/3079459989 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fb8000a2c20 msgr2=0x7fb8000a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:05:10.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:05:10.992+0000 7fb81e39d700 1 --2- 172.21.15.86:0/3079459989 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fb8000a2c20 0x7fb8000a50e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fb808005fd0 tx=0x7fb808005dc0 comp rx=0 tx=0).stop 2024-05-12T22:05:10.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:05:10.992+0000 7fb81e39d700 1 -- 172.21.15.86:0/3079459989 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb818102480 msgr2=0x7fb818198b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:05:10.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:05:10.992+0000 7fb81e39d700 1 --2- 172.21.15.86:0/3079459989 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb818102480 0x7fb818198b10 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fb80c003890 tx=0x7fb80c003730 comp rx=0 tx=0).stop 2024-05-12T22:05:10.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:05:10.992+0000 7fb81e39d700 1 -- 172.21.15.86:0/3079459989 shutdown_connections 2024-05-12T22:05:10.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:05:10.992+0000 7fb81e39d700 1 --2- 172.21.15.86:0/3079459989 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fb8000a2c20 0x7fb8000a50e0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:05:10.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:05:10.992+0000 7fb81e39d700 1 --2- 172.21.15.86:0/3079459989 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb818101280 0x7fb8181985d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:05:10.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:05:10.992+0000 7fb81e39d700 1 --2- 172.21.15.86:0/3079459989 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb818102e20 0x7fb81819fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:05:10.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:05:10.992+0000 7fb81e39d700 1 --2- 172.21.15.86:0/3079459989 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb818102480 0x7fb818198b10 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:05:10.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:05:10.992+0000 7fb81e39d700 1 -- 172.21.15.86:0/3079459989 >> 172.21.15.86:0/3079459989 conn(0x7fb8180fc840 msgr2=0x7fb8180fe320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:05:10.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:05:10.992+0000 7fb81e39d700 1 -- 172.21.15.86:0/3079459989 shutdown_connections 2024-05-12T22:05:10.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-05-12T22:05:10.993+0000 7fb81e39d700 1 -- 172.21.15.86:0/3079459989 wait complete. 2024-05-12T22:05:11.552 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:11 smithi086 ceph-mon[34297]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-05-12T22:05:11.553 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:11 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:05:11.553 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:11 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:05:11.553 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:11 smithi086 ceph-mon[34297]: from='osd.10 [v2:172.21.15.86:6824/1189968310,v1:172.21.15.86:6825/1189968310]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-05-12T22:05:11.553 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:11 smithi086 ceph-mon[34297]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-05-12T22:05:11.576 DEBUG:teuthology.orchestra.run.smithi086:osd.10> sudo journalctl -f -n 0 -u ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968@osd.10.service 2024-05-12T22:05:11.578 INFO:tasks.cephadm:Deploying osd.11 on smithi186 with /dev/vg_nvme/lv_1... 2024-05-12T22:05:11.578 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f72fecff68e1d400c4568684327c900485c20d6a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6b92e7b4-10aa-11ef-bc98-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-05-12T22:05:11.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:11 smithi028 ceph-mon[29847]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-05-12T22:05:11.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:11 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:05:11.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:11 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:05:11.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:11 smithi028 ceph-mon[29847]: from='osd.10 [v2:172.21.15.86:6824/1189968310,v1:172.21.15.86:6825/1189968310]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-05-12T22:05:11.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:11 smithi028 ceph-mon[29847]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-05-12T22:05:11.828 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/mon.c/config 2024-05-12T22:05:11.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:11 smithi186 ceph-mon[34785]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-05-12T22:05:11.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:11 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:05:11.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:11 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:05:11.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:11 smithi186 ceph-mon[34785]: from='osd.10 [v2:172.21.15.86:6824/1189968310,v1:172.21.15.86:6825/1189968310]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-05-12T22:05:11.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:11 smithi186 ceph-mon[34785]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-05-12T22:05:12.967 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:12 smithi086 ceph-mon[34297]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-05-12T22:05:12.967 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:12 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:05:12.967 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:12 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:05:12.967 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:12 smithi086 ceph-mon[34297]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-05-12T22:05:12.967 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:12 smithi086 ceph-mon[34297]: from='osd.10 [v2:172.21.15.86:6824/1189968310,v1:172.21.15.86:6825/1189968310]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-05-12T22:05:12.967 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:12 smithi086 ceph-mon[34297]: osdmap e71: 11 total, 10 up, 11 in 2024-05-12T22:05:12.967 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:12 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-12T22:05:12.968 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:12 smithi086 ceph-mon[34297]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-05-12T22:05:13.003 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:12 smithi186 ceph-mon[34785]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-05-12T22:05:13.003 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:12 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:05:13.003 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:12 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:05:13.003 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:12 smithi186 ceph-mon[34785]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-05-12T22:05:13.003 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:12 smithi186 ceph-mon[34785]: from='osd.10 [v2:172.21.15.86:6824/1189968310,v1:172.21.15.86:6825/1189968310]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-05-12T22:05:13.003 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:12 smithi186 ceph-mon[34785]: osdmap e71: 11 total, 10 up, 11 in 2024-05-12T22:05:13.003 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:12 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-12T22:05:13.003 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:12 smithi186 ceph-mon[34785]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-05-12T22:05:13.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:12 smithi028 ceph-mon[29847]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-05-12T22:05:13.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:12 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:05:13.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:12 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:05:13.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:12 smithi028 ceph-mon[29847]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-05-12T22:05:13.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:12 smithi028 ceph-mon[29847]: from='osd.10 [v2:172.21.15.86:6824/1189968310,v1:172.21.15.86:6825/1189968310]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-05-12T22:05:13.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:12 smithi028 ceph-mon[29847]: osdmap e71: 11 total, 10 up, 11 in 2024-05-12T22:05:13.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:12 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-12T22:05:13.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:12 smithi028 ceph-mon[29847]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-05-12T22:05:13.242 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:05:12 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:05:12.966+0000 7f113d8dd700 -1 osd.10 0 waiting for initial osdmap 2024-05-12T22:05:13.242 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:05:12 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:05:12.979+0000 7f1139ed1700 -1 osd.10 72 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-12T22:05:13.638 INFO:teuthology.orchestra.run.smithi186.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-05-12T22:05:13.638 INFO:teuthology.orchestra.run.smithi186.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-05-12T22:05:13.639 INFO:teuthology.orchestra.run.smithi186.stderr: stderr: 10+0 records in 2024-05-12T22:05:13.639 INFO:teuthology.orchestra.run.smithi186.stderr:10+0 records out 2024-05-12T22:05:13.639 INFO:teuthology.orchestra.run.smithi186.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114191 s, 918 MB/s 2024-05-12T22:05:13.639 INFO:teuthology.orchestra.run.smithi186.stderr:--> Zapping successful for: 2024-05-12T22:05:14.225 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:13 smithi186 ceph-mon[34785]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]': finished 2024-05-12T22:05:14.226 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:13 smithi186 ceph-mon[34785]: osdmap e72: 11 total, 10 up, 11 in 2024-05-12T22:05:14.226 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:13 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-12T22:05:14.226 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:13 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-12T22:05:14.226 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:13 smithi186 ceph-mon[34785]: osd.10 [v2:172.21.15.86:6824/1189968310,v1:172.21.15.86:6825/1189968310] boot 2024-05-12T22:05:14.226 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:13 smithi186 ceph-mon[34785]: osdmap e73: 11 total, 11 up, 11 in 2024-05-12T22:05:14.226 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:13 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-12T22:05:14.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:13 smithi086 ceph-mon[34297]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]': finished 2024-05-12T22:05:14.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:13 smithi086 ceph-mon[34297]: osdmap e72: 11 total, 10 up, 11 in 2024-05-12T22:05:14.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:13 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-12T22:05:14.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:13 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-12T22:05:14.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:13 smithi086 ceph-mon[34297]: osd.10 [v2:172.21.15.86:6824/1189968310,v1:172.21.15.86:6825/1189968310] boot 2024-05-12T22:05:14.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:13 smithi086 ceph-mon[34297]: osdmap e73: 11 total, 11 up, 11 in 2024-05-12T22:05:14.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:13 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-12T22:05:14.253 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f72fecff68e1d400c4568684327c900485c20d6a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6b92e7b4-10aa-11ef-bc98-c7b262605968 -- ceph orch daemon add osd smithi186:vg_nvme/lv_1 2024-05-12T22:05:14.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:13 smithi028 ceph-mon[29847]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]': finished 2024-05-12T22:05:14.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:13 smithi028 ceph-mon[29847]: osdmap e72: 11 total, 10 up, 11 in 2024-05-12T22:05:14.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:13 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-12T22:05:14.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:13 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-12T22:05:14.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:13 smithi028 ceph-mon[29847]: osd.10 [v2:172.21.15.86:6824/1189968310,v1:172.21.15.86:6825/1189968310] boot 2024-05-12T22:05:14.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:13 smithi028 ceph-mon[29847]: osdmap e73: 11 total, 11 up, 11 in 2024-05-12T22:05:14.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:13 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-12T22:05:14.498 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/mon.c/config 2024-05-12T22:05:15.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:14 smithi186 ceph-mon[34785]: purged_snaps scrub starts 2024-05-12T22:05:15.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:14 smithi186 ceph-mon[34785]: purged_snaps scrub ok 2024-05-12T22:05:15.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:14 smithi186 ceph-mon[34785]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-05-12T22:05:15.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:14 smithi186 ceph-mon[34785]: osdmap e74: 11 total, 11 up, 11 in 2024-05-12T22:05:15.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:14 smithi086 ceph-mon[34297]: purged_snaps scrub starts 2024-05-12T22:05:15.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:14 smithi086 ceph-mon[34297]: purged_snaps scrub ok 2024-05-12T22:05:15.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:14 smithi086 ceph-mon[34297]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-05-12T22:05:15.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:14 smithi086 ceph-mon[34297]: osdmap e74: 11 total, 11 up, 11 in 2024-05-12T22:05:15.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:14 smithi028 ceph-mon[29847]: purged_snaps scrub starts 2024-05-12T22:05:15.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:14 smithi028 ceph-mon[29847]: purged_snaps scrub ok 2024-05-12T22:05:15.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:14 smithi028 ceph-mon[29847]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-05-12T22:05:15.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:14 smithi028 ceph-mon[29847]: osdmap e74: 11 total, 11 up, 11 in 2024-05-12T22:05:15.932 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:05:15.930+0000 7fba7d7ba700 1 -- 172.21.15.186:0/1610385590 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fba78102e20 msgr2=0x7fba781096d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:05:15.932 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:05:15.930+0000 7fba7d7ba700 1 --2- 172.21.15.186:0/1610385590 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fba78102e20 0x7fba781096d0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fba7000b0a0 tx=0x7fba7000b3b0 comp rx=0 tx=0).stop 2024-05-12T22:05:15.932 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:05:15.930+0000 7fba7d7ba700 1 -- 172.21.15.186:0/1610385590 shutdown_connections 2024-05-12T22:05:15.933 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:05:15.930+0000 7fba7d7ba700 1 --2- 172.21.15.186:0/1610385590 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fba78102e20 0x7fba781096d0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:05:15.933 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:05:15.930+0000 7fba7d7ba700 1 --2- 172.21.15.186:0/1610385590 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fba78101280 0x7fba781016a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:05:15.933 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:05:15.930+0000 7fba7d7ba700 1 --2- 172.21.15.186:0/1610385590 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba78102480 0x7fba781028e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:05:15.933 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:05:15.930+0000 7fba7d7ba700 1 -- 172.21.15.186:0/1610385590 >> 172.21.15.186:0/1610385590 conn(0x7fba780fc820 msgr2=0x7fba780fec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:05:15.933 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:05:15.930+0000 7fba7d7ba700 1 -- 172.21.15.186:0/1610385590 shutdown_connections 2024-05-12T22:05:15.933 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:05:15.930+0000 7fba7d7ba700 1 -- 172.21.15.186:0/1610385590 wait complete. 2024-05-12T22:05:15.933 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:05:15.931+0000 7fba7d7ba700 1 Processor -- start 2024-05-12T22:05:15.933 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:05:15.931+0000 7fba7d7ba700 1 -- start start 2024-05-12T22:05:15.933 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:05:15.931+0000 7fba7d7ba700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fba78101280 0x7fba7806ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:05:15.933 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:05:15.931+0000 7fba7d7ba700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba78102480 0x7fba7819d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:05:15.933 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:05:15.931+0000 7fba7d7ba700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fba78102e20 0x7fba7819f930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:05:15.933 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:05:15.931+0000 7fba7d7ba700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fba7810cec0 con 0x7fba78102480 2024-05-12T22:05:15.933 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:05:15.931+0000 7fba7d7ba700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fba7810cd40 con 0x7fba78101280 2024-05-12T22:05:15.933 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:05:15.931+0000 7fba7d7ba700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fba7810d040 con 0x7fba78102e20 2024-05-12T22:05:15.934 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:05:15.931+0000 7fba777fe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fba78102e20 0x7fba7819f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:05:15.934 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:05:15.931+0000 7fba777fe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fba78102e20 0x7fba7819f930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:41114/0 (socket says 172.21.15.186:41114) 2024-05-12T22:05:15.934 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:05:15.931+0000 7fba777fe700 1 -- 172.21.15.186:0/2220788989 learned_addr learned my addr 172.21.15.186:0/2220788989 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-05-12T22:05:15.934 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:05:15.931+0000 7fba76ffd700 1 --2- 172.21.15.186:0/2220788989 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fba78101280 0x7fba7806ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:05:15.934 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:05:15.932+0000 7fba777fe700 1 -- 172.21.15.186:0/2220788989 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fba78101280 msgr2=0x7fba7806ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:05:15.934 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:05:15.932+0000 7fba777fe700 1 --2- 172.21.15.186:0/2220788989 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fba78101280 0x7fba7806ab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:05:15.934 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:05:15.932+0000 7fba777fe700 1 -- 172.21.15.186:0/2220788989 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba78102480 msgr2=0x7fba7819d2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:05:15.934 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:05:15.932+0000 7fba777fe700 1 --2- 172.21.15.186:0/2220788989 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba78102480 0x7fba7819d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:05:15.934 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:05:15.932+0000 7fba777fe700 1 -- 172.21.15.186:0/2220788989 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba781a0200 con 0x7fba78102e20 2024-05-12T22:05:15.934 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:05:15.932+0000 7fba76ffd700 1 --2- 172.21.15.186:0/2220788989 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fba78101280 0x7fba7806ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:05:15.934 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:05:15.932+0000 7fba777fe700 1 --2- 172.21.15.186:0/2220788989 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fba78102e20 0x7fba7819f930 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fba70007d90 tx=0x7fba70007dc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:05:15.934 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:05:15.932+0000 7fba5ffff700 1 -- 172.21.15.186:0/2220788989 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba70021020 con 0x7fba78102e20 2024-05-12T22:05:15.934 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:05:15.932+0000 7fba5ffff700 1 -- 172.21.15.186:0/2220788989 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fba7001d950 con 0x7fba78102e20 2024-05-12T22:05:15.934 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:05:15.932+0000 7fba7d7ba700 1 -- 172.21.15.186:0/2220788989 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba781a0490 con 0x7fba78102e20 2024-05-12T22:05:15.935 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:05:15.932+0000 7fba5ffff700 1 -- 172.21.15.186:0/2220788989 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba700053b0 con 0x7fba78102e20 2024-05-12T22:05:15.935 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:05:15.932+0000 7fba7d7ba700 1 -- 172.21.15.186:0/2220788989 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba781a0a10 con 0x7fba78102e20 2024-05-12T22:05:15.936 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:05:15.933+0000 7fba7d7ba700 1 -- 172.21.15.186:0/2220788989 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba78069010 con 0x7fba78102e20 2024-05-12T22:05:15.936 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:05:15.934+0000 7fba5ffff700 1 -- 172.21.15.186:0/2220788989 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fba70024020 con 0x7fba78102e20 2024-05-12T22:05:15.936 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:05:15.934+0000 7fba5ffff700 1 --2- 172.21.15.186:0/2220788989 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fba600a2ca0 0x7fba600a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:05:15.936 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:05:15.935+0000 7fba5ffff700 1 -- 172.21.15.186:0/2220788989 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 6723+0+0 (secure 0 0 0) 0x7fba7000f020 con 0x7fba78102e20 2024-05-12T22:05:15.936 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:05:15.935+0000 7fba76ffd700 1 --2- 172.21.15.186:0/2220788989 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fba600a2ca0 0x7fba600a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:05:15.936 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:05:15.936+0000 7fba76ffd700 1 --2- 172.21.15.186:0/2220788989 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fba600a2ca0 0x7fba600a5160 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fba781022e0 tx=0x7fba6c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:05:15.939 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:05:15.939+0000 7fba5ffff700 1 -- 172.21.15.186:0/2220788989 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fba7009d300 con 0x7fba78102e20 2024-05-12T22:05:16.144 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:05:16.144+0000 7fba7d7ba700 1 -- 172.21.15.186:0/2220788989 --> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7fba78062760 con 0x7fba600a2ca0 2024-05-12T22:05:16.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:16 smithi186 ceph-mon[34785]: Detected new or changed devices on smithi086 2024-05-12T22:05:16.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:16 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:05:16.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:16 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:05:16.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:16 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:05:16.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:16 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:05:16.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:16 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:05:16.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:16 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:05:16.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:16 smithi186 ceph-mon[34785]: Adjusting osd_memory_target on smithi086 to 4272M 2024-05-12T22:05:16.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:16 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:05:16.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:16 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:05:16.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:16 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:05:16.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:16 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:05:16.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:16 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:05:16.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:16 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:05:16.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:16 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:05:16.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:16 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:05:16.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:16 smithi186 ceph-mon[34785]: osdmap e75: 11 total, 11 up, 11 in 2024-05-12T22:05:16.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:16 smithi086 ceph-mon[34297]: Detected new or changed devices on smithi086 2024-05-12T22:05:16.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:16 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:05:16.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:16 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:05:16.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:16 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:05:16.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:16 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:05:16.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:16 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:05:16.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:16 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:05:16.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:16 smithi086 ceph-mon[34297]: Adjusting osd_memory_target on smithi086 to 4272M 2024-05-12T22:05:16.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:16 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:05:16.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:16 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:05:16.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:16 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:05:16.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:16 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:05:16.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:16 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:05:16.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:16 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:05:16.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:16 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:05:16.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:16 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:05:16.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:16 smithi086 ceph-mon[34297]: osdmap e75: 11 total, 11 up, 11 in 2024-05-12T22:05:16.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:16 smithi028 ceph-mon[29847]: Detected new or changed devices on smithi086 2024-05-12T22:05:16.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:16 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:05:16.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:16 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:05:16.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:16 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:05:16.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:16 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:05:16.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:16 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:05:16.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:16 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:05:16.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:16 smithi028 ceph-mon[29847]: Adjusting osd_memory_target on smithi086 to 4272M 2024-05-12T22:05:16.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:16 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:05:16.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:16 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:05:16.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:16 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:05:16.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:16 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:05:16.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:16 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:05:16.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:16 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:05:16.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:16 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:05:16.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:16 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:05:16.537 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:16 smithi028 ceph-mon[29847]: osdmap e75: 11 total, 11 up, 11 in 2024-05-12T22:05:17.324 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:17 smithi186 ceph-mon[34785]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 308 MiB used, 983 GiB / 983 GiB avail 2024-05-12T22:05:17.324 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:17 smithi186 ceph-mon[34785]: from='client.24358 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-12T22:05:17.324 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:17 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-12T22:05:17.324 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:17 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-12T22:05:17.324 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:17 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:05:17.324 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:17 smithi186 ceph-mon[34785]: osdmap e76: 11 total, 11 up, 11 in 2024-05-12T22:05:17.436 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:17 smithi028 ceph-mon[29847]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 308 MiB used, 983 GiB / 983 GiB avail 2024-05-12T22:05:17.437 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:17 smithi028 ceph-mon[29847]: from='client.24358 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-12T22:05:17.437 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:17 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-12T22:05:17.437 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:17 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-12T22:05:17.437 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:17 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:05:17.437 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:17 smithi028 ceph-mon[29847]: osdmap e76: 11 total, 11 up, 11 in 2024-05-12T22:05:17.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:17 smithi086 ceph-mon[34297]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 308 MiB used, 983 GiB / 983 GiB avail 2024-05-12T22:05:17.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:17 smithi086 ceph-mon[34297]: from='client.24358 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-12T22:05:17.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:17 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-12T22:05:17.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:17 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-12T22:05:17.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:17 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:05:17.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:17 smithi086 ceph-mon[34297]: osdmap e76: 11 total, 11 up, 11 in 2024-05-12T22:05:18.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:18 smithi028 ceph-mon[29847]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 308 MiB used, 983 GiB / 983 GiB avail 2024-05-12T22:05:18.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:18 smithi028 ceph-mon[29847]: osdmap e77: 11 total, 11 up, 11 in 2024-05-12T22:05:18.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:18 smithi186 ceph-mon[34785]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 308 MiB used, 983 GiB / 983 GiB avail 2024-05-12T22:05:18.930 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:18 smithi186 ceph-mon[34785]: osdmap e77: 11 total, 11 up, 11 in 2024-05-12T22:05:18.964 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:18 smithi086 ceph-mon[34297]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 308 MiB used, 983 GiB / 983 GiB avail 2024-05-12T22:05:18.964 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:18 smithi086 ceph-mon[34297]: osdmap e77: 11 total, 11 up, 11 in 2024-05-12T22:05:19.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:19 smithi086 ceph-mon[34297]: from='client.? 172.21.15.186:0/4095062254' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f112e3c6-65b6-4253-a966-62001db7d069"}]: dispatch 2024-05-12T22:05:19.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:19 smithi086 ceph-mon[34297]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f112e3c6-65b6-4253-a966-62001db7d069"}]: dispatch 2024-05-12T22:05:19.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:19 smithi086 ceph-mon[34297]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f112e3c6-65b6-4253-a966-62001db7d069"}]': finished 2024-05-12T22:05:19.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:19 smithi086 ceph-mon[34297]: osdmap e78: 12 total, 11 up, 12 in 2024-05-12T22:05:19.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:19 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-12T22:05:19.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:19 smithi086 ceph-mon[34297]: from='client.? 172.21.15.186:0/3824224325' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-12T22:05:20.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:19 smithi028 ceph-mon[29847]: from='client.? 172.21.15.186:0/4095062254' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f112e3c6-65b6-4253-a966-62001db7d069"}]: dispatch 2024-05-12T22:05:20.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:19 smithi028 ceph-mon[29847]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f112e3c6-65b6-4253-a966-62001db7d069"}]: dispatch 2024-05-12T22:05:20.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:19 smithi028 ceph-mon[29847]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f112e3c6-65b6-4253-a966-62001db7d069"}]': finished 2024-05-12T22:05:20.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:19 smithi028 ceph-mon[29847]: osdmap e78: 12 total, 11 up, 12 in 2024-05-12T22:05:20.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:19 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-12T22:05:20.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:19 smithi028 ceph-mon[29847]: from='client.? 172.21.15.186:0/3824224325' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-12T22:05:20.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:19 smithi186 ceph-mon[34785]: from='client.? 172.21.15.186:0/4095062254' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f112e3c6-65b6-4253-a966-62001db7d069"}]: dispatch 2024-05-12T22:05:20.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:19 smithi186 ceph-mon[34785]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f112e3c6-65b6-4253-a966-62001db7d069"}]: dispatch 2024-05-12T22:05:20.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:19 smithi186 ceph-mon[34785]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f112e3c6-65b6-4253-a966-62001db7d069"}]': finished 2024-05-12T22:05:20.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:19 smithi186 ceph-mon[34785]: osdmap e78: 12 total, 11 up, 12 in 2024-05-12T22:05:20.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:19 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-12T22:05:20.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:19 smithi186 ceph-mon[34785]: from='client.? 172.21.15.186:0/3824224325' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-12T22:05:21.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:20 smithi028 ceph-mon[29847]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-05-12T22:05:21.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:20 smithi186 ceph-mon[34785]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-05-12T22:05:21.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:20 smithi086 ceph-mon[34297]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-05-12T22:05:23.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:22 smithi028 ceph-mon[29847]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-05-12T22:05:23.042 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:22 smithi186 ceph-mon[34785]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-05-12T22:05:23.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:22 smithi086 ceph-mon[34297]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-05-12T22:05:25.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:24 smithi028 ceph-mon[29847]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-05-12T22:05:25.123 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:24 smithi186 ceph-mon[34785]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-05-12T22:05:25.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:24 smithi086 ceph-mon[34297]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-05-12T22:05:27.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:26 smithi028 ceph-mon[29847]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-05-12T22:05:27.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:26 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-05-12T22:05:27.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:26 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:05:27.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:26 smithi028 ceph-mon[29847]: Deploying daemon osd.11 on smithi186 2024-05-12T22:05:27.124 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:26 smithi186 ceph-mon[34785]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-05-12T22:05:27.124 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:26 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-05-12T22:05:27.124 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:26 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:05:27.124 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:26 smithi186 ceph-mon[34785]: Deploying daemon osd.11 on smithi186 2024-05-12T22:05:27.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:26 smithi086 ceph-mon[34297]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-05-12T22:05:27.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:26 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-05-12T22:05:27.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:26 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:05:27.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:26 smithi086 ceph-mon[34297]: Deploying daemon osd.11 on smithi186 2024-05-12T22:05:29.107 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:28 smithi186 ceph-mon[34785]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-12T22:05:29.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:28 smithi086 ceph-mon[34297]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-12T22:05:29.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:28 smithi028 ceph-mon[29847]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-12T22:05:31.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:30 smithi186 ceph-mon[34785]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-05-12T22:05:31.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:30 smithi086 ceph-mon[34297]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-05-12T22:05:31.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:30 smithi028 ceph-mon[29847]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-05-12T22:05:32.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:32 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:05:32.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:32 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:05:32.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:32 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:05:32.930 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:32 smithi186 ceph-mon[34785]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-05-12T22:05:32.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:32 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:05:32.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:32 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:05:32.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:32 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:05:32.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:32 smithi086 ceph-mon[34297]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-05-12T22:05:33.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:32 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:05:33.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:32 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:05:33.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:32 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:05:33.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:32 smithi028 ceph-mon[29847]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-05-12T22:05:34.861 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:34 smithi186 ceph-mon[34785]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-05-12T22:05:34.932 INFO:teuthology.orchestra.run.smithi186.stdout:Created osd(s) 11 on host 'smithi186' 2024-05-12T22:05:34.933 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:05:34.932+0000 7fba5ffff700 1 -- 172.21.15.186:0/2220788989 <== mgr.14152 v2:172.21.15.28:6800/3123328651 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7fba78062760 con 0x7fba600a2ca0 2024-05-12T22:05:34.934 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:05:34.934+0000 7fba7d7ba700 1 -- 172.21.15.186:0/2220788989 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fba600a2ca0 msgr2=0x7fba600a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:05:34.934 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:05:34.934+0000 7fba7d7ba700 1 --2- 172.21.15.186:0/2220788989 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fba600a2ca0 0x7fba600a5160 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fba781022e0 tx=0x7fba6c00b410 comp rx=0 tx=0).stop 2024-05-12T22:05:34.934 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:05:34.934+0000 7fba7d7ba700 1 -- 172.21.15.186:0/2220788989 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fba78102e20 msgr2=0x7fba7819f930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:05:34.935 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:05:34.934+0000 7fba7d7ba700 1 --2- 172.21.15.186:0/2220788989 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fba78102e20 0x7fba7819f930 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fba70007d90 tx=0x7fba70007dc0 comp rx=0 tx=0).stop 2024-05-12T22:05:34.935 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:05:34.934+0000 7fba7d7ba700 1 -- 172.21.15.186:0/2220788989 shutdown_connections 2024-05-12T22:05:34.935 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:05:34.934+0000 7fba7d7ba700 1 --2- 172.21.15.186:0/2220788989 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fba600a2ca0 0x7fba600a5160 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:05:34.935 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:05:34.934+0000 7fba7d7ba700 1 --2- 172.21.15.186:0/2220788989 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fba78102e20 0x7fba7819f930 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:05:34.935 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:05:34.934+0000 7fba7d7ba700 1 --2- 172.21.15.186:0/2220788989 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fba78101280 0x7fba7806ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:05:34.935 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:05:34.934+0000 7fba7d7ba700 1 --2- 172.21.15.186:0/2220788989 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba78102480 0x7fba7819d2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:05:34.935 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:05:34.934+0000 7fba7d7ba700 1 -- 172.21.15.186:0/2220788989 >> 172.21.15.186:0/2220788989 conn(0x7fba780fc820 msgr2=0x7fba78105780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:05:34.935 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:05:34.935+0000 7fba7d7ba700 1 -- 172.21.15.186:0/2220788989 shutdown_connections 2024-05-12T22:05:34.935 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:05:34.935+0000 7fba7d7ba700 1 -- 172.21.15.186:0/2220788989 wait complete. 2024-05-12T22:05:34.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:34 smithi086 ceph-mon[34297]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-05-12T22:05:35.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:34 smithi028 ceph-mon[29847]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-05-12T22:05:35.515 DEBUG:teuthology.orchestra.run.smithi186:osd.11> sudo journalctl -f -n 0 -u ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968@osd.11.service 2024-05-12T22:05:35.517 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-05-12T22:05:35.518 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f72fecff68e1d400c4568684327c900485c20d6a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6b92e7b4-10aa-11ef-bc98-c7b262605968 -- ceph osd stat -f json 2024-05-12T22:05:35.998 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/mon.a/config 2024-05-12T22:05:36.157 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:35 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:05:36.158 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:35 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:05:36.158 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:35 smithi028 ceph-mon[29847]: from='osd.11 [v2:172.21.15.186:6824/3801668110,v1:172.21.15.186:6825/3801668110]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-05-12T22:05:36.158 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:35 smithi028 ceph-mon[29847]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-05-12T22:05:36.158 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:35 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:05:36.158 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:35 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:05:36.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:35 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:05:36.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:35 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:05:36.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:35 smithi186 ceph-mon[34785]: from='osd.11 [v2:172.21.15.186:6824/3801668110,v1:172.21.15.186:6825/3801668110]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-05-12T22:05:36.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:35 smithi186 ceph-mon[34785]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-05-12T22:05:36.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:35 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:05:36.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:35 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:05:36.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:35 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:05:36.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:35 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:05:36.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:35 smithi086 ceph-mon[34297]: from='osd.11 [v2:172.21.15.186:6824/3801668110,v1:172.21.15.186:6825/3801668110]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-05-12T22:05:36.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:35 smithi086 ceph-mon[34297]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-05-12T22:05:36.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:35 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:05:36.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:35 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:05:37.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:36 smithi186 ceph-mon[34785]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-05-12T22:05:37.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:36 smithi186 ceph-mon[34785]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-05-12T22:05:37.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:36 smithi186 ceph-mon[34785]: osdmap e79: 12 total, 11 up, 12 in 2024-05-12T22:05:37.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:36 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-12T22:05:37.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:36 smithi186 ceph-mon[34785]: from='osd.11 [v2:172.21.15.186:6824/3801668110,v1:172.21.15.186:6825/3801668110]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-05-12T22:05:37.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:36 smithi186 ceph-mon[34785]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-05-12T22:05:37.180 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:05:36 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:05:36.925+0000 7f9705c8f700 -1 osd.11 0 waiting for initial osdmap 2024-05-12T22:05:37.180 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:05:36 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:05:36.941+0000 7f9702283700 -1 osd.11 80 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-12T22:05:37.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:36 smithi028 ceph-mon[29847]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-05-12T22:05:37.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:36 smithi028 ceph-mon[29847]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-05-12T22:05:37.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:36 smithi028 ceph-mon[29847]: osdmap e79: 12 total, 11 up, 12 in 2024-05-12T22:05:37.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:36 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-12T22:05:37.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:36 smithi028 ceph-mon[29847]: from='osd.11 [v2:172.21.15.186:6824/3801668110,v1:172.21.15.186:6825/3801668110]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-05-12T22:05:37.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:36 smithi028 ceph-mon[29847]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-05-12T22:05:37.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:36 smithi086 ceph-mon[34297]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-05-12T22:05:37.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:36 smithi086 ceph-mon[34297]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-05-12T22:05:37.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:36 smithi086 ceph-mon[34297]: osdmap e79: 12 total, 11 up, 12 in 2024-05-12T22:05:37.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:36 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-12T22:05:37.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:36 smithi086 ceph-mon[34297]: from='osd.11 [v2:172.21.15.186:6824/3801668110,v1:172.21.15.186:6825/3801668110]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-05-12T22:05:37.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:36 smithi086 ceph-mon[34297]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-05-12T22:05:37.992 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:37 smithi028 ceph-mon[29847]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]': finished 2024-05-12T22:05:37.992 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:37 smithi028 ceph-mon[29847]: osdmap e80: 12 total, 11 up, 12 in 2024-05-12T22:05:37.992 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:37 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-12T22:05:37.992 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:37 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-12T22:05:38.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:37 smithi086 ceph-mon[34297]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]': finished 2024-05-12T22:05:38.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:37 smithi086 ceph-mon[34297]: osdmap e80: 12 total, 11 up, 12 in 2024-05-12T22:05:38.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:37 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-12T22:05:38.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:37 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-12T22:05:38.315 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:37 smithi186 ceph-mon[34785]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]': finished 2024-05-12T22:05:38.316 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:37 smithi186 ceph-mon[34785]: osdmap e80: 12 total, 11 up, 12 in 2024-05-12T22:05:38.316 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:37 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-12T22:05:38.316 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:37 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-12T22:05:38.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:38.333+0000 7f2b75e00700 1 -- 172.21.15.28:0/822516038 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2b70074d80 msgr2=0x7f2b70075200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:05:38.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:38.333+0000 7f2b75e00700 1 --2- 172.21.15.28:0/822516038 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2b70074d80 0x7f2b70075200 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f2b580098f0 tx=0x7f2b58009c00 comp rx=0 tx=0).stop 2024-05-12T22:05:38.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:38.333+0000 7f2b75e00700 1 -- 172.21.15.28:0/822516038 shutdown_connections 2024-05-12T22:05:38.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:38.333+0000 7f2b75e00700 1 --2- 172.21.15.28:0/822516038 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2b70076350 0x7f2b700747b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:05:38.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:38.333+0000 7f2b75e00700 1 --2- 172.21.15.28:0/822516038 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b7010e970 0x7f2b70110d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:05:38.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:38.333+0000 7f2b75e00700 1 --2- 172.21.15.28:0/822516038 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2b70074d80 0x7f2b70075200 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:05:38.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:38.333+0000 7f2b75e00700 1 -- 172.21.15.28:0/822516038 >> 172.21.15.28:0/822516038 conn(0x7f2b700fd930 msgr2=0x7f2b700ffd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:05:38.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:38.333+0000 7f2b75e00700 1 -- 172.21.15.28:0/822516038 shutdown_connections 2024-05-12T22:05:38.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:38.333+0000 7f2b75e00700 1 -- 172.21.15.28:0/822516038 wait complete. 2024-05-12T22:05:38.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:38.335+0000 7f2b75e00700 1 Processor -- start 2024-05-12T22:05:38.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:38.335+0000 7f2b75e00700 1 -- start start 2024-05-12T22:05:38.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:38.335+0000 7f2b75e00700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2b70074d80 0x7f2b7019ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:05:38.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:38.335+0000 7f2b75e00700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2b70076350 0x7f2b7019f180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:05:38.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:38.335+0000 7f2b75e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b7010e970 0x7f2b701a61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:05:38.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:38.335+0000 7f2b75e00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2b701145c0 con 0x7f2b7010e970 2024-05-12T22:05:38.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:38.335+0000 7f2b75e00700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f2b70114440 con 0x7f2b70074d80 2024-05-12T22:05:38.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:38.335+0000 7f2b75e00700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f2b70114740 con 0x7f2b70076350 2024-05-12T22:05:38.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:38.336+0000 7f2b6ea00700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2b70074d80 0x7f2b7019ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:05:38.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:38.336+0000 7f2b6ea00700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2b70074d80 0x7f2b7019ec40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:60816/0 (socket says 172.21.15.28:60816) 2024-05-12T22:05:38.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:38.336+0000 7f2b6ea00700 1 -- 172.21.15.28:0/1631433799 learned_addr learned my addr 172.21.15.28:0/1631433799 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:05:38.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:38.336+0000 7f2b67e00700 1 --2- 172.21.15.28:0/1631433799 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2b70076350 0x7f2b7019f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:05:38.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:38.336+0000 7f2b6f400700 1 --2- 172.21.15.28:0/1631433799 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b7010e970 0x7f2b701a61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:05:38.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:38.336+0000 7f2b6ea00700 1 -- 172.21.15.28:0/1631433799 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2b70076350 msgr2=0x7f2b7019f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:05:38.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:38.336+0000 7f2b6ea00700 1 --2- 172.21.15.28:0/1631433799 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2b70076350 0x7f2b7019f180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:05:38.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:38.336+0000 7f2b6ea00700 1 -- 172.21.15.28:0/1631433799 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b7010e970 msgr2=0x7f2b701a61c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:05:38.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:38.336+0000 7f2b6ea00700 1 --2- 172.21.15.28:0/1631433799 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b7010e970 0x7f2b701a61c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:05:38.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:38.336+0000 7f2b6ea00700 1 -- 172.21.15.28:0/1631433799 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b701a6890 con 0x7f2b70074d80 2024-05-12T22:05:38.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:38.336+0000 7f2b6f400700 1 --2- 172.21.15.28:0/1631433799 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b7010e970 0x7f2b701a61c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:05:38.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:38.336+0000 7f2b6ea00700 1 --2- 172.21.15.28:0/1631433799 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2b70074d80 0x7f2b7019ec40 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f2b6000ec70 tx=0x7f2b6000ef80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:05:38.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:38.337+0000 7f2b74a00700 1 -- 172.21.15.28:0/1631433799 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b6000cd40 con 0x7f2b70074d80 2024-05-12T22:05:38.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:38.337+0000 7f2b74a00700 1 -- 172.21.15.28:0/1631433799 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2b60004500 con 0x7f2b70074d80 2024-05-12T22:05:38.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:38.337+0000 7f2b74a00700 1 -- 172.21.15.28:0/1631433799 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b60010690 con 0x7f2b70074d80 2024-05-12T22:05:38.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:38.337+0000 7f2b75e00700 1 -- 172.21.15.28:0/1631433799 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b701a6b80 con 0x7f2b70074d80 2024-05-12T22:05:38.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:38.337+0000 7f2b75e00700 1 -- 172.21.15.28:0/1631433799 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b701a7110 con 0x7f2b70074d80 2024-05-12T22:05:38.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:38.338+0000 7f2b75e00700 1 -- 172.21.15.28:0/1631433799 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b70069010 con 0x7f2b70074d80 2024-05-12T22:05:38.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:38.340+0000 7f2b74a00700 1 -- 172.21.15.28:0/1631433799 <== mon.2 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f2b60003ee0 con 0x7f2b70074d80 2024-05-12T22:05:38.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:38.341+0000 7f2b74a00700 1 --2- 172.21.15.28:0/1631433799 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f2b540a2ca0 0x7f2b540a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:05:38.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:38.341+0000 7f2b74a00700 1 -- 172.21.15.28:0/1631433799 <== mon.2 v2:172.21.15.86:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f2b600cef50 con 0x7f2b70074d80 2024-05-12T22:05:38.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:38.341+0000 7f2b67e00700 1 --2- 172.21.15.28:0/1631433799 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f2b540a2ca0 0x7f2b540a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:05:38.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:38.342+0000 7f2b67e00700 1 --2- 172.21.15.28:0/1631433799 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f2b540a2ca0 0x7f2b540a5160 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f2b580098c0 tx=0x7f2b58019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:05:38.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:38.345+0000 7f2b74a00700 1 -- 172.21.15.28:0/1631433799 <== mon.2 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f2b6009c2b0 con 0x7f2b70074d80 2024-05-12T22:05:38.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:38.574+0000 7f2b75e00700 1 -- 172.21.15.28:0/1631433799 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f2b70062f30 con 0x7f2b70074d80 2024-05-12T22:05:38.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:38.575+0000 7f2b74a00700 1 -- 172.21.15.28:0/1631433799 <== mon.2 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v81) ==== 74+0+133 (secure 0 0 0) 0x7f2b6009bbb0 con 0x7f2b70074d80 2024-05-12T22:05:38.574 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-12T22:05:38.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:38.578+0000 7f2b75e00700 1 -- 172.21.15.28:0/1631433799 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f2b540a2ca0 msgr2=0x7f2b540a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:05:38.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:38.578+0000 7f2b75e00700 1 --2- 172.21.15.28:0/1631433799 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f2b540a2ca0 0x7f2b540a5160 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f2b580098c0 tx=0x7f2b58019040 comp rx=0 tx=0).stop 2024-05-12T22:05:38.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:38.578+0000 7f2b75e00700 1 -- 172.21.15.28:0/1631433799 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2b70074d80 msgr2=0x7f2b7019ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:05:38.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:38.578+0000 7f2b75e00700 1 --2- 172.21.15.28:0/1631433799 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2b70074d80 0x7f2b7019ec40 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f2b6000ec70 tx=0x7f2b6000ef80 comp rx=0 tx=0).stop 2024-05-12T22:05:38.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:38.578+0000 7f2b75e00700 1 -- 172.21.15.28:0/1631433799 shutdown_connections 2024-05-12T22:05:38.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:38.578+0000 7f2b75e00700 1 --2- 172.21.15.28:0/1631433799 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f2b540a2ca0 0x7f2b540a5160 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:05:38.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:38.578+0000 7f2b75e00700 1 --2- 172.21.15.28:0/1631433799 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b7010e970 0x7f2b701a61c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:05:38.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:38.578+0000 7f2b75e00700 1 --2- 172.21.15.28:0/1631433799 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2b70074d80 0x7f2b7019ec40 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:05:38.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:38.578+0000 7f2b75e00700 1 --2- 172.21.15.28:0/1631433799 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2b70076350 0x7f2b7019f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:05:38.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:38.578+0000 7f2b75e00700 1 -- 172.21.15.28:0/1631433799 >> 172.21.15.28:0/1631433799 conn(0x7f2b700fd930 msgr2=0x7f2b70101d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:05:38.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:38.578+0000 7f2b75e00700 1 -- 172.21.15.28:0/1631433799 shutdown_connections 2024-05-12T22:05:38.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:38.578+0000 7f2b75e00700 1 -- 172.21.15.28:0/1631433799 wait complete. 2024-05-12T22:05:39.053 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:38 smithi186 ceph-mon[34785]: purged_snaps scrub starts 2024-05-12T22:05:39.053 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:38 smithi186 ceph-mon[34785]: purged_snaps scrub ok 2024-05-12T22:05:39.053 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:38 smithi186 ceph-mon[34785]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-05-12T22:05:39.053 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:38 smithi186 ceph-mon[34785]: osd.11 [v2:172.21.15.186:6824/3801668110,v1:172.21.15.186:6825/3801668110] boot 2024-05-12T22:05:39.053 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:38 smithi186 ceph-mon[34785]: osdmap e81: 12 total, 12 up, 12 in 2024-05-12T22:05:39.053 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:38 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-12T22:05:39.053 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:38 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/1631433799' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-05-12T22:05:39.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:38 smithi086 ceph-mon[34297]: purged_snaps scrub starts 2024-05-12T22:05:39.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:38 smithi086 ceph-mon[34297]: purged_snaps scrub ok 2024-05-12T22:05:39.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:38 smithi086 ceph-mon[34297]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-05-12T22:05:39.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:38 smithi086 ceph-mon[34297]: osd.11 [v2:172.21.15.186:6824/3801668110,v1:172.21.15.186:6825/3801668110] boot 2024-05-12T22:05:39.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:38 smithi086 ceph-mon[34297]: osdmap e81: 12 total, 12 up, 12 in 2024-05-12T22:05:39.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:38 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-12T22:05:39.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:38 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/1631433799' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-05-12T22:05:39.284 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:38 smithi028 ceph-mon[29847]: purged_snaps scrub starts 2024-05-12T22:05:39.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:38 smithi028 ceph-mon[29847]: purged_snaps scrub ok 2024-05-12T22:05:39.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:38 smithi028 ceph-mon[29847]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-05-12T22:05:39.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:38 smithi028 ceph-mon[29847]: osd.11 [v2:172.21.15.186:6824/3801668110,v1:172.21.15.186:6825/3801668110] boot 2024-05-12T22:05:39.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:38 smithi028 ceph-mon[29847]: osdmap e81: 12 total, 12 up, 12 in 2024-05-12T22:05:39.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:38 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-12T22:05:39.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:38 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/1631433799' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-05-12T22:05:39.493 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":81,"num_osds":12,"num_up_osds":12,"osd_up_since":1715551537,"num_in_osds":12,"osd_in_since":1715551518,"num_remapped_pgs":0} 2024-05-12T22:05:39.494 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f72fecff68e1d400c4568684327c900485c20d6a shell --fsid 6b92e7b4-10aa-11ef-bc98-c7b262605968 -- ceph osd dump --format=json 2024-05-12T22:05:39.924 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/mon.a/config 2024-05-12T22:05:40.235 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:39 smithi028 ceph-mon[29847]: osdmap e82: 12 total, 12 up, 12 in 2024-05-12T22:05:40.235 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:39 smithi028 ceph-mon[29847]: Detected new or changed devices on smithi186 2024-05-12T22:05:40.235 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:39 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:05:40.235 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:39 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:05:40.235 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:39 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:05:40.236 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:39 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:05:40.236 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:39 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:05:40.236 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:39 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:05:40.236 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:39 smithi028 ceph-mon[29847]: Adjusting osd_memory_target on smithi186 to 4272M 2024-05-12T22:05:40.236 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:39 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:05:40.236 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:39 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:05:40.236 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:39 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:05:40.236 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:39 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:05:40.236 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:39 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:05:40.236 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:39 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:05:40.236 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:39 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:05:40.236 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:39 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:05:40.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:39 smithi086 ceph-mon[34297]: osdmap e82: 12 total, 12 up, 12 in 2024-05-12T22:05:40.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:39 smithi086 ceph-mon[34297]: Detected new or changed devices on smithi186 2024-05-12T22:05:40.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:39 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:05:40.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:39 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:05:40.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:39 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:05:40.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:39 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:05:40.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:39 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:05:40.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:39 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:05:40.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:39 smithi086 ceph-mon[34297]: Adjusting osd_memory_target on smithi186 to 4272M 2024-05-12T22:05:40.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:39 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:05:40.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:39 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:05:40.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:39 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:05:40.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:39 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:05:40.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:39 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:05:40.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:39 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:05:40.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:39 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:05:40.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:39 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:05:40.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:39 smithi186 ceph-mon[34785]: osdmap e82: 12 total, 12 up, 12 in 2024-05-12T22:05:40.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:39 smithi186 ceph-mon[34785]: Detected new or changed devices on smithi186 2024-05-12T22:05:40.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:39 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:05:40.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:39 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:05:40.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:39 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:05:40.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:39 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:05:40.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:39 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:05:40.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:39 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:05:40.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:39 smithi186 ceph-mon[34785]: Adjusting osd_memory_target on smithi186 to 4272M 2024-05-12T22:05:40.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:39 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:05:40.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:39 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:05:40.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:39 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:05:40.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:39 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:05:40.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:39 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:05:40.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:39 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:05:40.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:39 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:05:40.431 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:39 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:05:41.674 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:41 smithi028 ceph-mon[29847]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-12T22:05:41.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:41 smithi186 ceph-mon[34785]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-12T22:05:41.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:41 smithi086 ceph-mon[34297]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-12T22:05:41.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:41.949+0000 7f946f800700 1 -- 172.21.15.28:0/1987507655 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9468104f20 msgr2=0x7f94681113e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:05:41.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:41.949+0000 7f946f800700 1 --2- 172.21.15.28:0/1987507655 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9468104f20 0x7f94681113e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f9460009fc0 tx=0x7f946000b330 comp rx=0 tx=0).stop 2024-05-12T22:05:41.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:41.950+0000 7f946f800700 1 -- 172.21.15.28:0/1987507655 shutdown_connections 2024-05-12T22:05:41.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:41.950+0000 7f946f800700 1 --2- 172.21.15.28:0/1987507655 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9468104f20 0x7f94681113e0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:05:41.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:41.950+0000 7f946f800700 1 --2- 172.21.15.28:0/1987507655 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f946810a610 0x7f946810a9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:05:41.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:41.950+0000 7f946f800700 1 --2- 172.21.15.28:0/1987507655 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9468104560 0x7f94681049e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:05:41.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:41.950+0000 7f946f800700 1 -- 172.21.15.28:0/1987507655 >> 172.21.15.28:0/1987507655 conn(0x7f9468100030 msgr2=0x7f9468102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:05:41.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:41.950+0000 7f946f800700 1 -- 172.21.15.28:0/1987507655 shutdown_connections 2024-05-12T22:05:41.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:41.950+0000 7f946f800700 1 -- 172.21.15.28:0/1987507655 wait complete. 2024-05-12T22:05:41.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:41.951+0000 7f946f800700 1 Processor -- start 2024-05-12T22:05:41.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:41.951+0000 7f946f800700 1 -- start start 2024-05-12T22:05:41.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:41.952+0000 7f946f800700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9468104560 0x7f94681a08a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:05:41.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:41.952+0000 7f946f800700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9468104f20 0x7f94681a0de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:05:41.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:41.952+0000 7f946f800700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f946810a610 0x7f946819aab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:05:41.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:41.952+0000 7f946f800700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f94681148d0 con 0x7f9468104560 2024-05-12T22:05:41.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:41.952+0000 7f946f800700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f9468114750 con 0x7f9468104f20 2024-05-12T22:05:41.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:41.952+0000 7f946f800700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f9468114a50 con 0x7f946810a610 2024-05-12T22:05:41.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:41.952+0000 7f946ce00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9468104560 0x7f94681a08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:05:41.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:41.952+0000 7f9467e00700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9468104f20 0x7f94681a0de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:05:41.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:41.952+0000 7f946ce00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9468104560 0x7f94681a08a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34526/0 (socket says 172.21.15.28:34526) 2024-05-12T22:05:41.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:41.952+0000 7f946ce00700 1 -- 172.21.15.28:0/1047409412 learned_addr learned my addr 172.21.15.28:0/1047409412 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:05:41.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:41.952+0000 7f946d800700 1 --2- 172.21.15.28:0/1047409412 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f946810a610 0x7f946819aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:05:41.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:41.952+0000 7f9467e00700 1 -- 172.21.15.28:0/1047409412 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f946810a610 msgr2=0x7f946819aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:05:41.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:41.952+0000 7f9467e00700 1 --2- 172.21.15.28:0/1047409412 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f946810a610 0x7f946819aab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:05:41.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:41.952+0000 7f9467e00700 1 -- 172.21.15.28:0/1047409412 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9468104560 msgr2=0x7f94681a08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:05:41.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:41.952+0000 7f9467e00700 1 --2- 172.21.15.28:0/1047409412 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9468104560 0x7f94681a08a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:05:41.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:41.952+0000 7f9467e00700 1 -- 172.21.15.28:0/1047409412 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f946819b210 con 0x7f9468104f20 2024-05-12T22:05:41.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:41.952+0000 7f946d800700 1 --2- 172.21.15.28:0/1047409412 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f946810a610 0x7f946819aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:05:41.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:41.953+0000 7f9467e00700 1 --2- 172.21.15.28:0/1047409412 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9468104f20 0x7f94681a0de0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f945400e9c0 tx=0x7f945400ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:05:41.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:41.953+0000 7f9465600700 1 -- 172.21.15.28:0/1047409412 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9454004220 con 0x7f9468104f20 2024-05-12T22:05:41.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:41.953+0000 7f946f800700 1 -- 172.21.15.28:0/1047409412 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f946819b500 con 0x7f9468104f20 2024-05-12T22:05:41.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:41.953+0000 7f9465600700 1 -- 172.21.15.28:0/1047409412 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9454004d10 con 0x7f9468104f20 2024-05-12T22:05:41.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:41.953+0000 7f9465600700 1 -- 172.21.15.28:0/1047409412 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f945400ce60 con 0x7f9468104f20 2024-05-12T22:05:41.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:41.953+0000 7f946f800700 1 -- 172.21.15.28:0/1047409412 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94681a7770 con 0x7f9468104f20 2024-05-12T22:05:41.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:41.954+0000 7f946f800700 1 -- 172.21.15.28:0/1047409412 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9468069010 con 0x7f9468104f20 2024-05-12T22:05:41.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:41.956+0000 7f9465600700 1 -- 172.21.15.28:0/1047409412 <== mon.2 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f9454004750 con 0x7f9468104f20 2024-05-12T22:05:41.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:41.958+0000 7f9465600700 1 --2- 172.21.15.28:0/1047409412 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f94500a2c20 0x7f94500a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:05:41.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:41.958+0000 7f9465600700 1 -- 172.21.15.28:0/1047409412 <== mon.2 v2:172.21.15.86:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f94540ce9b0 con 0x7f9468104f20 2024-05-12T22:05:41.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:41.958+0000 7f946ce00700 1 --2- 172.21.15.28:0/1047409412 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f94500a2c20 0x7f94500a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:05:41.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:41.959+0000 7f946ce00700 1 --2- 172.21.15.28:0/1047409412 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f94500a2c20 0x7f94500a50e0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f94681095d0 tx=0x7f945c007480 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:05:41.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:41.962+0000 7f9465600700 1 -- 172.21.15.28:0/1047409412 <== mon.2 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f945409bd10 con 0x7f9468104f20 2024-05-12T22:05:42.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:42.191+0000 7f946f800700 1 -- 172.21.15.28:0/1047409412 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f946802db80 con 0x7f9468104f20 2024-05-12T22:05:42.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:42.193+0000 7f9465600700 1 -- 172.21.15.28:0/1047409412 <== mon.2 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) ==== 74+0+18397 (secure 0 0 0) 0x7f945409b610 con 0x7f9468104f20 2024-05-12T22:05:42.193 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-12T22:05:42.193 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":82,"fsid":"6b92e7b4-10aa-11ef-bc98-c7b262605968","created":"2024-05-12T21:57:37.276782+0000","modified":"2024-05-12T22:05:38.939521+0000","last_up_change":"2024-05-12T22:05:37.929562+0000","last_in_change":"2024-05-12T22:05:18.750768+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-12T22:01:32.292826+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6992ce0a-844b-40e3-8ed1-181c8474dccf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":906003882},{"type":"v1","addr":"172.21.15.28:6803","nonce":906003882}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":906003882},{"type":"v1","addr":"172.21.15.28:6805","nonce":906003882}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":906003882},{"type":"v1","addr":"172.21.15.28:6809","nonce":906003882}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":906003882},{"type":"v1","addr":"172.21.15.28:6807","nonce":906003882}]},"public_addr":"172.21.15.28:6803/906003882","cluster_addr":"172.21.15.28:6805/906003882","heartbeat_back_addr":"172.21.15.28:6809/906003882","heartbeat_front_addr":"172.21.15.28:6807/906003882","state":["exists","up"]},{"osd":1,"uuid":"ced20c96-c481-4d48-90d1-6b43228959ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":3354141819},{"type":"v1","addr":"172.21.15.86:6801","nonce":3354141819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":3354141819},{"type":"v1","addr":"172.21.15.86:6803","nonce":3354141819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":3354141819},{"type":"v1","addr":"172.21.15.86:6807","nonce":3354141819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":3354141819},{"type":"v1","addr":"172.21.15.86:6805","nonce":3354141819}]},"public_addr":"172.21.15.86:6801/3354141819","cluster_addr":"172.21.15.86:6803/3354141819","heartbeat_back_addr":"172.21.15.86:6807/3354141819","heartbeat_front_addr":"172.21.15.86:6805/3354141819","state":["exists","up"]},{"osd":2,"uuid":"54a2f221-fa65-4795-9f5e-e660cc7c0098","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":906459365},{"type":"v1","addr":"172.21.15.186:6801","nonce":906459365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":906459365},{"type":"v1","addr":"172.21.15.186:6803","nonce":906459365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":906459365},{"type":"v1","addr":"172.21.15.186:6807","nonce":906459365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":906459365},{"type":"v1","addr":"172.21.15.186:6805","nonce":906459365}]},"public_addr":"172.21.15.186:6801/906459365","cluster_addr":"172.21.15.186:6803/906459365","heartbeat_back_addr":"172.21.15.186:6807/906459365","heartbeat_front_addr":"172.21.15.186:6805/906459365","state":["exists","up"]},{"osd":3,"uuid":"86836737-ac9a-487b-93f2-7fcd3c406960","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":4182884436},{"type":"v1","addr":"172.21.15.28:6811","nonce":4182884436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":4182884436},{"type":"v1","addr":"172.21.15.28:6813","nonce":4182884436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":4182884436},{"type":"v1","addr":"172.21.15.28:6817","nonce":4182884436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":4182884436},{"type":"v1","addr":"172.21.15.28:6815","nonce":4182884436}]},"public_addr":"172.21.15.28:6811/4182884436","cluster_addr":"172.21.15.28:6813/4182884436","heartbeat_back_addr":"172.21.15.28:6817/4182884436","heartbeat_front_addr":"172.21.15.28:6815/4182884436","state":["exists","up"]},{"osd":4,"uuid":"d162ebcd-83c0-4fd5-9925-28b2cecea935","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2596998708},{"type":"v1","addr":"172.21.15.86:6809","nonce":2596998708}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2596998708},{"type":"v1","addr":"172.21.15.86:6811","nonce":2596998708}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2596998708},{"type":"v1","addr":"172.21.15.86:6815","nonce":2596998708}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2596998708},{"type":"v1","addr":"172.21.15.86:6813","nonce":2596998708}]},"public_addr":"172.21.15.86:6809/2596998708","cluster_addr":"172.21.15.86:6811/2596998708","heartbeat_back_addr":"172.21.15.86:6815/2596998708","heartbeat_front_addr":"172.21.15.86:6813/2596998708","state":["exists","up"]},{"osd":5,"uuid":"c794336a-a105-425c-b6c7-a32b90c05bb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":1944205756},{"type":"v1","addr":"172.21.15.186:6809","nonce":1944205756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":1944205756},{"type":"v1","addr":"172.21.15.186:6811","nonce":1944205756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":1944205756},{"type":"v1","addr":"172.21.15.186:6815","nonce":1944205756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":1944205756},{"type":"v1","addr":"172.21.15.186:6813","nonce":1944205756}]},"public_addr":"172.21.15.186:6809/1944205756","cluster_addr":"172.21.15.186:6811/1944205756","heartbeat_back_addr":"172.21.15.186:6815/1944205756","heartbeat_front_addr":"172.21.15.186:6813/1944205756","state":["exists","up"]},{"osd":6,"uuid":"516932ac-af00-45b7-ad3c-d6dec7db442f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":1642308925},{"type":"v1","addr":"172.21.15.28:6819","nonce":1642308925}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":1642308925},{"type":"v1","addr":"172.21.15.28:6821","nonce":1642308925}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1642308925},{"type":"v1","addr":"172.21.15.28:6825","nonce":1642308925}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":1642308925},{"type":"v1","addr":"172.21.15.28:6823","nonce":1642308925}]},"public_addr":"172.21.15.28:6819/1642308925","cluster_addr":"172.21.15.28:6821/1642308925","heartbeat_back_addr":"172.21.15.28:6825/1642308925","heartbeat_front_addr":"172.21.15.28:6823/1642308925","state":["exists","up"]},{"osd":7,"uuid":"ca7a059f-0414-438e-b2d1-a10f348036be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":726062711},{"type":"v1","addr":"172.21.15.86:6817","nonce":726062711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":726062711},{"type":"v1","addr":"172.21.15.86:6819","nonce":726062711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":726062711},{"type":"v1","addr":"172.21.15.86:6823","nonce":726062711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":726062711},{"type":"v1","addr":"172.21.15.86:6821","nonce":726062711}]},"public_addr":"172.21.15.86:6817/726062711","cluster_addr":"172.21.15.86:6819/726062711","heartbeat_back_addr":"172.21.15.86:6823/726062711","heartbeat_front_addr":"172.21.15.86:6821/726062711","state":["exists","up"]},{"osd":8,"uuid":"92e3f41f-aebd-422e-a99e-555f7ba4aa53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":3007687277},{"type":"v1","addr":"172.21.15.186:6817","nonce":3007687277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":3007687277},{"type":"v1","addr":"172.21.15.186:6819","nonce":3007687277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":3007687277},{"type":"v1","addr":"172.21.15.186:6823","nonce":3007687277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":3007687277},{"type":"v1","addr":"172.21.15.186:6821","nonce":3007687277}]},"public_addr":"172.21.15.186:6817/3007687277","cluster_addr":"172.21.15.186:6819/3007687277","heartbeat_back_addr":"172.21.15.186:6823/3007687277","heartbeat_front_addr":"172.21.15.186:6821/3007687277","state":["exists","up"]},{"osd":9,"uuid":"38ff7d9d-ee85-404c-8c59-adc327e1ccfa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":136220472},{"type":"v1","addr":"172.21.15.28:6827","nonce":136220472}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":136220472},{"type":"v1","addr":"172.21.15.28:6829","nonce":136220472}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":136220472},{"type":"v1","addr":"172.21.15.28:6833","nonce":136220472}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":136220472},{"type":"v1","addr":"172.21.15.28:6831","nonce":136220472}]},"public_addr":"172.21.15.28:6827/136220472","cluster_addr":"172.21.15.28:6829/136220472","heartbeat_back_addr":"172.21.15.28:6833/136220472","heartbeat_front_addr":"172.21.15.28:6831/136220472","state":["exists","up"]},{"osd":10,"uuid":"5e9ff657-afa8-4bab-8491-f209038392b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1189968310},{"type":"v1","addr":"172.21.15.86:6825","nonce":1189968310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":1189968310},{"type":"v1","addr":"172.21.15.86:6827","nonce":1189968310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":1189968310},{"type":"v1","addr":"172.21.15.86:6831","nonce":1189968310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":1189968310},{"type":"v1","addr":"172.21.15.86:6829","nonce":1189968310}]},"public_addr":"172.21.15.86:6825/1189968310","cluster_addr":"172.21.15.86:6827/1189968310","heartbeat_back_addr":"172.21.15.86:6831/1189968310","heartbeat_front_addr":"172.21.15.86:6829/1189968310","state":["exists","up"]},{"osd":11,"uuid":"f112e3c6-65b6-4253-a966-62001db7d069","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3801668110},{"type":"v1","addr":"172.21.15.186:6825","nonce":3801668110}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3801668110},{"type":"v1","addr":"172.21.15.186:6827","nonce":3801668110}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3801668110},{"type":"v1","addr":"172.21.15.186:6831","nonce":3801668110}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3801668110},{"type":"v1","addr":"172.21.15.186:6829","nonce":3801668110}]},"public_addr":"172.21.15.186:6825/3801668110","cluster_addr":"172.21.15.186:6827/3801668110","heartbeat_back_addr":"172.21.15.186:6831/3801668110","heartbeat_front_addr":"172.21.15.186:6829/3801668110","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:00:40.483623+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:01:05.284385+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:01:28.645716+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:02:01.295092+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:02:26.486601+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:02:50.807478+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:03:24.756169+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:03:49.975173+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:04:13.772233+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:04:46.758555+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:05:12.331685+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:05:36.462548+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/1177009533":"2024-05-13T21:58:45.550459+0000","172.21.15.28:6800/3760499540":"2024-05-13T21:58:45.550459+0000","172.21.15.28:6801/3783742728":"2024-05-13T21:58:10.421564+0000","172.21.15.28:6800/3783742728":"2024-05-13T21:58:10.421564+0000","172.21.15.28:0/2229156260":"2024-05-13T21:58:45.550459+0000","172.21.15.28:0/1202140963":"2024-05-13T21:58:10.421564+0000","172.21.15.28:0/2924223784":"2024-05-13T21:58:45.550459+0000","172.21.15.28:6801/3760499540":"2024-05-13T21:58:45.550459+0000","172.21.15.28:0/1240027206":"2024-05-13T21:58:10.421564+0000","172.21.15.28:0/171417728":"2024-05-13T21:58:10.421564+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-12T22:05:42.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:42.196+0000 7f946f800700 1 -- 172.21.15.28:0/1047409412 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f94500a2c20 msgr2=0x7f94500a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:05:42.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:42.196+0000 7f946f800700 1 --2- 172.21.15.28:0/1047409412 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f94500a2c20 0x7f94500a50e0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f94681095d0 tx=0x7f945c007480 comp rx=0 tx=0).stop 2024-05-12T22:05:42.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:42.196+0000 7f946f800700 1 -- 172.21.15.28:0/1047409412 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9468104f20 msgr2=0x7f94681a0de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:05:42.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:42.196+0000 7f946f800700 1 --2- 172.21.15.28:0/1047409412 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9468104f20 0x7f94681a0de0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f945400e9c0 tx=0x7f945400ecd0 comp rx=0 tx=0).stop 2024-05-12T22:05:42.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:42.196+0000 7f946f800700 1 -- 172.21.15.28:0/1047409412 shutdown_connections 2024-05-12T22:05:42.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:42.196+0000 7f946f800700 1 --2- 172.21.15.28:0/1047409412 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f94500a2c20 0x7f94500a50e0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:05:42.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:42.196+0000 7f946f800700 1 --2- 172.21.15.28:0/1047409412 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f946810a610 0x7f946819aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:05:42.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:42.196+0000 7f946f800700 1 --2- 172.21.15.28:0/1047409412 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9468104560 0x7f94681a08a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:05:42.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:42.196+0000 7f946f800700 1 --2- 172.21.15.28:0/1047409412 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9468104f20 0x7f94681a0de0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:05:42.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:42.196+0000 7f946f800700 1 -- 172.21.15.28:0/1047409412 >> 172.21.15.28:0/1047409412 conn(0x7f9468100030 msgr2=0x7f94681015e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:05:42.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:42.196+0000 7f946f800700 1 -- 172.21.15.28:0/1047409412 shutdown_connections 2024-05-12T22:05:42.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:42.196+0000 7f946f800700 1 -- 172.21.15.28:0/1047409412 wait complete. 2024-05-12T22:05:42.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:42 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/1047409412' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-12T22:05:42.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:42 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/1047409412' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-12T22:05:42.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:42 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/1047409412' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-12T22:05:43.036 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-12T22:01:32.292826+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-12T22:05:43.036 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f72fecff68e1d400c4568684327c900485c20d6a shell --fsid 6b92e7b4-10aa-11ef-bc98-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-05-12T22:05:43.499 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/mon.a/config 2024-05-12T22:05:43.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:43 smithi086 ceph-mon[34297]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-12T22:05:43.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:43 smithi028 ceph-mon[29847]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-12T22:05:43.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:43 smithi186 ceph-mon[34785]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-12T22:05:45.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:45.606+0000 7f0aa9600700 1 -- 172.21.15.28:0/342342191 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0aa4104f20 msgr2=0x7f0aa41113e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:05:45.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:45.606+0000 7f0aa9600700 1 --2- 172.21.15.28:0/342342191 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0aa4104f20 0x7f0aa41113e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f0a9c00b0d0 tx=0x7f0a9c00b3e0 comp rx=0 tx=0).stop 2024-05-12T22:05:45.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:45.606+0000 7f0aa9600700 1 -- 172.21.15.28:0/342342191 shutdown_connections 2024-05-12T22:05:45.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:45.606+0000 7f0aa9600700 1 --2- 172.21.15.28:0/342342191 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0aa4104f20 0x7f0aa41113e0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:05:45.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:45.606+0000 7f0aa9600700 1 --2- 172.21.15.28:0/342342191 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0aa410a610 0x7f0aa410a9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:05:45.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:45.606+0000 7f0aa9600700 1 --2- 172.21.15.28:0/342342191 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0aa4104560 0x7f0aa41049e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:05:45.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:45.606+0000 7f0aa9600700 1 -- 172.21.15.28:0/342342191 >> 172.21.15.28:0/342342191 conn(0x7f0aa4100030 msgr2=0x7f0aa4102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:05:45.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:45.607+0000 7f0aa9600700 1 -- 172.21.15.28:0/342342191 shutdown_connections 2024-05-12T22:05:45.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:45.607+0000 7f0aa9600700 1 -- 172.21.15.28:0/342342191 wait complete. 2024-05-12T22:05:45.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:45.608+0000 7f0aa9600700 1 Processor -- start 2024-05-12T22:05:45.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:45.608+0000 7f0aa9600700 1 -- start start 2024-05-12T22:05:45.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:45.608+0000 7f0aa9600700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0aa4104560 0x7f0aa41a08b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:05:45.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:45.608+0000 7f0aa9600700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0aa4104f20 0x7f0aa41a0df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:05:45.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:45.608+0000 7f0aa9600700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0aa410a610 0x7f0aa419aa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:05:45.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:45.608+0000 7f0aa9600700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0aa41148b0 con 0x7f0aa4104f20 2024-05-12T22:05:45.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:45.608+0000 7f0aa9600700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f0aa4114730 con 0x7f0aa410a610 2024-05-12T22:05:45.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:45.608+0000 7f0aa9600700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f0aa4114a30 con 0x7f0aa4104560 2024-05-12T22:05:45.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:45.609+0000 7f0aa2a00700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0aa4104560 0x7f0aa41a08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:05:45.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:45.609+0000 7f0aa2a00700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0aa4104560 0x7f0aa41a08b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:36690/0 (socket says 172.21.15.28:36690) 2024-05-12T22:05:45.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:45.609+0000 7f0aa2a00700 1 -- 172.21.15.28:0/45154726 learned_addr learned my addr 172.21.15.28:0/45154726 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:05:45.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:45.609+0000 7f0aa2000700 1 --2- 172.21.15.28:0/45154726 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0aa4104f20 0x7f0aa41a0df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:05:45.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:45.609+0000 7f0aa3400700 1 --2- 172.21.15.28:0/45154726 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0aa410a610 0x7f0aa419aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:05:45.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:45.609+0000 7f0aa2a00700 1 -- 172.21.15.28:0/45154726 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0aa410a610 msgr2=0x7f0aa419aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:05:45.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:45.609+0000 7f0aa2a00700 1 --2- 172.21.15.28:0/45154726 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0aa410a610 0x7f0aa419aa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:05:45.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:45.609+0000 7f0aa2a00700 1 -- 172.21.15.28:0/45154726 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0aa4104f20 msgr2=0x7f0aa41a0df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:05:45.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:45.609+0000 7f0aa2a00700 1 --2- 172.21.15.28:0/45154726 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0aa4104f20 0x7f0aa41a0df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:05:45.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:45.609+0000 7f0aa2a00700 1 -- 172.21.15.28:0/45154726 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0aa419b1d0 con 0x7f0aa4104560 2024-05-12T22:05:45.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:45.609+0000 7f0aa2000700 1 --2- 172.21.15.28:0/45154726 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0aa4104f20 0x7f0aa41a0df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:05:45.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:45.610+0000 7f0aa2a00700 1 --2- 172.21.15.28:0/45154726 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0aa4104560 0x7f0aa41a08b0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f0a9400e9a0 tx=0x7f0a9400ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:05:45.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:45.610+0000 7f0a9b400700 1 -- 172.21.15.28:0/45154726 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a9400cbd0 con 0x7f0aa4104560 2024-05-12T22:05:45.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:45.610+0000 7f0aa9600700 1 -- 172.21.15.28:0/45154726 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0aa419b4c0 con 0x7f0aa4104560 2024-05-12T22:05:45.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:45.610+0000 7f0a9b400700 1 -- 172.21.15.28:0/45154726 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0a9400cd70 con 0x7f0aa4104560 2024-05-12T22:05:45.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:45.610+0000 7f0aa9600700 1 -- 172.21.15.28:0/45154726 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0aa41a7780 con 0x7f0aa4104560 2024-05-12T22:05:45.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:45.610+0000 7f0a9b400700 1 -- 172.21.15.28:0/45154726 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a94018750 con 0x7f0aa4104560 2024-05-12T22:05:45.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:45.611+0000 7f0aa9600700 1 -- 172.21.15.28:0/45154726 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a84005330 con 0x7f0aa4104560 2024-05-12T22:05:45.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:45.614+0000 7f0a9b400700 1 -- 172.21.15.28:0/45154726 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f0a940188f0 con 0x7f0aa4104560 2024-05-12T22:05:45.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:45.615+0000 7f0a9b400700 1 --2- 172.21.15.28:0/45154726 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f0a880a2c20 0x7f0a880a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:05:45.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:45.615+0000 7f0a9b400700 1 -- 172.21.15.28:0/45154726 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f0a940cf330 con 0x7f0aa4104560 2024-05-12T22:05:45.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:45.615+0000 7f0aa2000700 1 --2- 172.21.15.28:0/45154726 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f0a880a2c20 0x7f0a880a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:05:45.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:45.616+0000 7f0aa2000700 1 --2- 172.21.15.28:0/45154726 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f0a880a2c20 0x7f0a880a50e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f0aa4109b90 tx=0x7f0a90009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:05:45.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:45.617+0000 7f0a9b400700 1 -- 172.21.15.28:0/45154726 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f0a9409c690 con 0x7f0aa4104560 2024-05-12T22:05:45.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:45 smithi086 ceph-mon[34297]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-12T22:05:45.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:45 smithi028 ceph-mon[29847]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-12T22:05:45.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:45.848+0000 7f0aa9600700 1 -- 172.21.15.28:0/45154726 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f0a840058f0 con 0x7f0aa4104560 2024-05-12T22:05:45.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:45.849+0000 7f0a9b400700 1 -- 172.21.15.28:0/45154726 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) ==== 93+0+10 (secure 0 0 0) 0x7f0a9409bf90 con 0x7f0aa4104560 2024-05-12T22:05:45.849 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-05-12T22:05:45.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:45.852+0000 7f0aa9600700 1 -- 172.21.15.28:0/45154726 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f0a880a2c20 msgr2=0x7f0a880a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:05:45.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:45.852+0000 7f0aa9600700 1 --2- 172.21.15.28:0/45154726 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f0a880a2c20 0x7f0a880a50e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f0aa4109b90 tx=0x7f0a90009260 comp rx=0 tx=0).stop 2024-05-12T22:05:45.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:45.852+0000 7f0aa9600700 1 -- 172.21.15.28:0/45154726 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0aa4104560 msgr2=0x7f0aa41a08b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:05:45.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:45.852+0000 7f0aa9600700 1 --2- 172.21.15.28:0/45154726 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0aa4104560 0x7f0aa41a08b0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f0a9400e9a0 tx=0x7f0a9400ecb0 comp rx=0 tx=0).stop 2024-05-12T22:05:45.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:45.852+0000 7f0aa9600700 1 -- 172.21.15.28:0/45154726 shutdown_connections 2024-05-12T22:05:45.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:45.852+0000 7f0aa9600700 1 --2- 172.21.15.28:0/45154726 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f0a880a2c20 0x7f0a880a50e0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:05:45.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:45.852+0000 7f0aa9600700 1 --2- 172.21.15.28:0/45154726 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0aa4104560 0x7f0aa41a08b0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:05:45.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:45.852+0000 7f0aa9600700 1 --2- 172.21.15.28:0/45154726 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0aa410a610 0x7f0aa419aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:05:45.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:45.852+0000 7f0aa9600700 1 --2- 172.21.15.28:0/45154726 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0aa4104f20 0x7f0aa41a0df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:05:45.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:45.852+0000 7f0aa9600700 1 -- 172.21.15.28:0/45154726 >> 172.21.15.28:0/45154726 conn(0x7f0aa4100030 msgr2=0x7f0aa4101570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:05:45.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:45.852+0000 7f0aa9600700 1 -- 172.21.15.28:0/45154726 shutdown_connections 2024-05-12T22:05:45.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:45.852+0000 7f0aa9600700 1 -- 172.21.15.28:0/45154726 wait complete. 2024-05-12T22:05:45.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:45 smithi186 ceph-mon[34785]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-12T22:05:46.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:46 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/45154726' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-12T22:05:46.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:46 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/45154726' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-12T22:05:46.759 INFO:tasks.cephadm:Adding mds.a on smithi028 2024-05-12T22:05:46.760 INFO:tasks.cephadm:Adding mds.d on smithi028 2024-05-12T22:05:46.760 INFO:tasks.cephadm:Adding mds.g on smithi028 2024-05-12T22:05:46.760 INFO:tasks.cephadm:Adding mds.j on smithi028 2024-05-12T22:05:46.760 INFO:tasks.cephadm:Adding mds.b on smithi086 2024-05-12T22:05:46.760 INFO:tasks.cephadm:Adding mds.e on smithi086 2024-05-12T22:05:46.760 INFO:tasks.cephadm:Adding mds.h on smithi086 2024-05-12T22:05:46.760 INFO:tasks.cephadm:Adding mds.k on smithi086 2024-05-12T22:05:46.760 INFO:tasks.cephadm:Adding mds.c on smithi186 2024-05-12T22:05:46.760 INFO:tasks.cephadm:Adding mds.f on smithi186 2024-05-12T22:05:46.760 INFO:tasks.cephadm:Adding mds.i on smithi186 2024-05-12T22:05:46.760 INFO:tasks.cephadm:Adding mds.l on smithi186 2024-05-12T22:05:46.761 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f72fecff68e1d400c4568684327c900485c20d6a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6b92e7b4-10aa-11ef-bc98-c7b262605968 -- ceph orch apply mds all '12;smithi028=a;smithi028=d;smithi028=g;smithi028=j;smithi086=b;smithi086=e;smithi086=h;smithi086=k;smithi186=c;smithi186=f;smithi186=i;smithi186=l' 2024-05-12T22:05:46.788 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:46 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/45154726' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-12T22:05:47.013 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/mon.c/config 2024-05-12T22:05:47.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:47 smithi186 ceph-mon[34785]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-12T22:05:47.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:47 smithi086 ceph-mon[34297]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-12T22:05:47.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:47 smithi028 ceph-mon[29847]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-12T22:05:48.527 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:05:48.524+0000 7f86115e6700 1 -- 172.21.15.186:0/2250952201 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f860c1057e0 msgr2=0x7f860c107bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:05:48.527 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:05:48.524+0000 7f86115e6700 1 --2- 172.21.15.186:0/2250952201 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f860c1057e0 0x7f860c107bd0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f85f4009990 tx=0x7f85f4009ca0 comp rx=0 tx=0).stop 2024-05-12T22:05:48.527 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:05:48.524+0000 7f86115e6700 1 -- 172.21.15.186:0/2250952201 shutdown_connections 2024-05-12T22:05:48.527 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:05:48.524+0000 7f86115e6700 1 --2- 172.21.15.186:0/2250952201 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f860c108110 0x7f860c10a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:05:48.527 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:05:48.524+0000 7f86115e6700 1 --2- 172.21.15.186:0/2250952201 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f860c102eb0 0x7f860c1052a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:05:48.527 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:05:48.524+0000 7f86115e6700 1 --2- 172.21.15.186:0/2250952201 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f860c1057e0 0x7f860c107bd0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:05:48.527 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:05:48.524+0000 7f86115e6700 1 -- 172.21.15.186:0/2250952201 >> 172.21.15.186:0/2250952201 conn(0x7f860c0fc820 msgr2=0x7f860c0fec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:05:48.527 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:05:48.524+0000 7f86115e6700 1 -- 172.21.15.186:0/2250952201 shutdown_connections 2024-05-12T22:05:48.527 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:05:48.524+0000 7f86115e6700 1 -- 172.21.15.186:0/2250952201 wait complete. 2024-05-12T22:05:48.527 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:05:48.524+0000 7f86115e6700 1 Processor -- start 2024-05-12T22:05:48.527 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:05:48.525+0000 7f86115e6700 1 -- start start 2024-05-12T22:05:48.527 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:05:48.525+0000 7f86115e6700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f860c102eb0 0x7f860c198610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:05:48.528 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:05:48.525+0000 7f86115e6700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f860c1057e0 0x7f860c198b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:05:48.528 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:05:48.525+0000 7f86115e6700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f860c108110 0x7f860c19fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:05:48.528 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:05:48.525+0000 7f86115e6700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f860c10dff0 con 0x7f860c102eb0 2024-05-12T22:05:48.528 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:05:48.525+0000 7f86115e6700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f860c10de70 con 0x7f860c108110 2024-05-12T22:05:48.528 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:05:48.525+0000 7f86115e6700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f860c10e170 con 0x7f860c1057e0 2024-05-12T22:05:48.528 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:05:48.525+0000 7f860a7fc700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f860c1057e0 0x7f860c198b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:05:48.528 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:05:48.525+0000 7f860a7fc700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f860c1057e0 0x7f860c198b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:56228/0 (socket says 172.21.15.186:56228) 2024-05-12T22:05:48.528 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:05:48.525+0000 7f860a7fc700 1 -- 172.21.15.186:0/1654777141 learned_addr learned my addr 172.21.15.186:0/1654777141 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-05-12T22:05:48.528 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:05:48.525+0000 7f860b7fe700 1 --2- 172.21.15.186:0/1654777141 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f860c108110 0x7f860c19fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:05:48.528 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:05:48.525+0000 7f860a7fc700 1 -- 172.21.15.186:0/1654777141 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f860c108110 msgr2=0x7f860c19fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:05:48.528 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:05:48.525+0000 7f860a7fc700 1 --2- 172.21.15.186:0/1654777141 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f860c108110 0x7f860c19fbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:05:48.528 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:05:48.525+0000 7f860a7fc700 1 -- 172.21.15.186:0/1654777141 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f860c102eb0 msgr2=0x7f860c198610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:05:48.528 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:05:48.525+0000 7f860a7fc700 1 --2- 172.21.15.186:0/1654777141 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f860c102eb0 0x7f860c198610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:05:48.528 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:05:48.525+0000 7f860a7fc700 1 -- 172.21.15.186:0/1654777141 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f860c1a0270 con 0x7f860c1057e0 2024-05-12T22:05:48.528 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:05:48.525+0000 7f860a7fc700 1 --2- 172.21.15.186:0/1654777141 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f860c1057e0 0x7f860c198b50 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f85f4000c00 tx=0x7f85f4005e80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:05:48.529 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:05:48.526+0000 7f8603fff700 1 -- 172.21.15.186:0/1654777141 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85f4027070 con 0x7f860c1057e0 2024-05-12T22:05:48.529 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:05:48.526+0000 7f86115e6700 1 -- 172.21.15.186:0/1654777141 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f860c1a0500 con 0x7f860c1057e0 2024-05-12T22:05:48.529 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:05:48.526+0000 7f8603fff700 1 -- 172.21.15.186:0/1654777141 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f85f40059c0 con 0x7f860c1057e0 2024-05-12T22:05:48.529 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:05:48.526+0000 7f8603fff700 1 -- 172.21.15.186:0/1654777141 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85f4022e30 con 0x7f860c1057e0 2024-05-12T22:05:48.529 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:05:48.526+0000 7f86115e6700 1 -- 172.21.15.186:0/1654777141 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f860c1a09e0 con 0x7f860c1057e0 2024-05-12T22:05:48.530 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:05:48.527+0000 7f86115e6700 1 -- 172.21.15.186:0/1654777141 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f860c069010 con 0x7f860c1057e0 2024-05-12T22:05:48.530 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:05:48.528+0000 7f8603fff700 1 -- 172.21.15.186:0/1654777141 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f85f402a070 con 0x7f860c1057e0 2024-05-12T22:05:48.530 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:05:48.528+0000 7f8603fff700 1 --2- 172.21.15.186:0/1654777141 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f85f80a2c20 0x7f85f80a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:05:48.530 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:05:48.529+0000 7f8603fff700 1 -- 172.21.15.186:0/1654777141 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f85f40cf850 con 0x7f860c1057e0 2024-05-12T22:05:48.530 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:05:48.529+0000 7f860affd700 1 --2- 172.21.15.186:0/1654777141 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f85f80a2c20 0x7f85f80a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:05:48.530 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:05:48.530+0000 7f860affd700 1 --2- 172.21.15.186:0/1654777141 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f85f80a2c20 0x7f85f80a50e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f85fc0060b0 tx=0x7f85fc006040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:05:48.533 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:05:48.533+0000 7f8603fff700 1 -- 172.21.15.186:0/1654777141 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f85f4013070 con 0x7f860c1057e0 2024-05-12T22:05:48.744 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:05:48.743+0000 7f86115e6700 1 -- 172.21.15.186:0/1654777141 --> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi028=a;smithi028=d;smithi028=g;smithi028=j;smithi086=b;smithi086=e;smithi086=h;smithi086=k;smithi186=c;smithi186=f;smithi186=i;smithi186=l", "target": ["mon-mgr", ""]}) -- 0x7f860c062860 con 0x7f85f80a2c20 2024-05-12T22:05:48.761 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:05:48.760+0000 7f8603fff700 1 -- 172.21.15.186:0/1654777141 <== mgr.14152 v2:172.21.15.28:6800/3123328651 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (secure 0 0 0) 0x7f860c062860 con 0x7f85f80a2c20 2024-05-12T22:05:48.761 INFO:teuthology.orchestra.run.smithi186.stdout:Scheduled mds.all update... 2024-05-12T22:05:48.763 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:05:48.762+0000 7f86115e6700 1 -- 172.21.15.186:0/1654777141 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f85f80a2c20 msgr2=0x7f85f80a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:05:48.763 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:05:48.762+0000 7f86115e6700 1 --2- 172.21.15.186:0/1654777141 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f85f80a2c20 0x7f85f80a50e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f85fc0060b0 tx=0x7f85fc006040 comp rx=0 tx=0).stop 2024-05-12T22:05:48.763 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:05:48.762+0000 7f86115e6700 1 -- 172.21.15.186:0/1654777141 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f860c1057e0 msgr2=0x7f860c198b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:05:48.763 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:05:48.762+0000 7f86115e6700 1 --2- 172.21.15.186:0/1654777141 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f860c1057e0 0x7f860c198b50 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f85f4000c00 tx=0x7f85f4005e80 comp rx=0 tx=0).stop 2024-05-12T22:05:48.764 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:05:48.762+0000 7f86115e6700 1 -- 172.21.15.186:0/1654777141 shutdown_connections 2024-05-12T22:05:48.764 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:05:48.762+0000 7f86115e6700 1 --2- 172.21.15.186:0/1654777141 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f85f80a2c20 0x7f85f80a50e0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:05:48.764 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:05:48.762+0000 7f86115e6700 1 --2- 172.21.15.186:0/1654777141 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f860c108110 0x7f860c19fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:05:48.764 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:05:48.762+0000 7f86115e6700 1 --2- 172.21.15.186:0/1654777141 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f860c102eb0 0x7f860c198610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:05:48.764 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:05:48.762+0000 7f86115e6700 1 --2- 172.21.15.186:0/1654777141 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f860c1057e0 0x7f860c198b50 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:05:48.764 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:05:48.762+0000 7f86115e6700 1 -- 172.21.15.186:0/1654777141 >> 172.21.15.186:0/1654777141 conn(0x7f860c0fc820 msgr2=0x7f860c1038d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:05:48.764 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:05:48.763+0000 7f86115e6700 1 -- 172.21.15.186:0/1654777141 shutdown_connections 2024-05-12T22:05:48.764 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-12T22:05:48.763+0000 7f86115e6700 1 -- 172.21.15.186:0/1654777141 wait complete. 2024-05-12T22:05:49.409 DEBUG:teuthology.orchestra.run.smithi028:mds.a> sudo journalctl -f -n 0 -u ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968@mds.a.service 2024-05-12T22:05:49.417 DEBUG:teuthology.orchestra.run.smithi028:mds.d> sudo journalctl -f -n 0 -u ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968@mds.d.service 2024-05-12T22:05:49.462 DEBUG:teuthology.orchestra.run.smithi028:mds.g> sudo journalctl -f -n 0 -u ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968@mds.g.service 2024-05-12T22:05:49.509 DEBUG:teuthology.orchestra.run.smithi028:mds.j> sudo journalctl -f -n 0 -u ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968@mds.j.service 2024-05-12T22:05:49.555 DEBUG:teuthology.orchestra.run.smithi086:mds.b> sudo journalctl -f -n 0 -u ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968@mds.b.service 2024-05-12T22:05:49.557 DEBUG:teuthology.orchestra.run.smithi086:mds.e> sudo journalctl -f -n 0 -u ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968@mds.e.service 2024-05-12T22:05:49.600 DEBUG:teuthology.orchestra.run.smithi086:mds.h> sudo journalctl -f -n 0 -u ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968@mds.h.service 2024-05-12T22:05:49.643 DEBUG:teuthology.orchestra.run.smithi086:mds.k> sudo journalctl -f -n 0 -u ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968@mds.k.service 2024-05-12T22:05:49.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:49 smithi186 ceph-mon[34785]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-12T22:05:49.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:49 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:05:49.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:49 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:05:49.685 DEBUG:teuthology.orchestra.run.smithi186:mds.c> sudo journalctl -f -n 0 -u ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968@mds.c.service 2024-05-12T22:05:49.689 DEBUG:teuthology.orchestra.run.smithi186:mds.f> sudo journalctl -f -n 0 -u ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968@mds.f.service 2024-05-12T22:05:49.703 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:49 smithi086 ceph-mon[34297]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-12T22:05:49.703 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:49 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:05:49.703 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:49 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:05:49.731 DEBUG:teuthology.orchestra.run.smithi186:mds.i> sudo journalctl -f -n 0 -u ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968@mds.i.service 2024-05-12T22:05:49.774 DEBUG:teuthology.orchestra.run.smithi186:mds.l> sudo journalctl -f -n 0 -u ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968@mds.l.service 2024-05-12T22:05:49.784 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:49 smithi028 ceph-mon[29847]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-12T22:05:49.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:49 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:05:49.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:49 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:05:49.818 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-05-12T22:05:49.818 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-12T22:05:50.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:50.048+0000 7f1603000640 1 -- 172.21.15.28:0/3272820311 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f15fc1038b0 msgr2=0x7f15fc103cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:05:50.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:50.048+0000 7f1603000640 1 --2- 172.21.15.28:0/3272820311 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f15fc1038b0 0x7f15fc103cd0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f15f0009ca0 tx=0x7f15f002a8c0 comp rx=0 tx=0).stop 2024-05-12T22:05:50.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:50.048+0000 7f1603000640 1 -- 172.21.15.28:0/3272820311 shutdown_connections 2024-05-12T22:05:50.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:50.048+0000 7f1603000640 1 --2- 172.21.15.28:0/3272820311 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f15fc105460 0x7f15fc10bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:05:50.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:50.048+0000 7f1603000640 1 --2- 172.21.15.28:0/3272820311 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15fc104aa0 0x7f15fc104f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:05:50.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:50.048+0000 7f1603000640 1 --2- 172.21.15.28:0/3272820311 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f15fc1038b0 0x7f15fc103cd0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:05:50.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:50.048+0000 7f1603000640 1 -- 172.21.15.28:0/3272820311 >> 172.21.15.28:0/3272820311 conn(0x7f15fc0fee40 msgr2=0x7f15fc101260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:05:50.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:50.048+0000 7f1603000640 1 -- 172.21.15.28:0/3272820311 shutdown_connections 2024-05-12T22:05:50.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:50.048+0000 7f1603000640 1 -- 172.21.15.28:0/3272820311 wait complete. 2024-05-12T22:05:50.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:50.049+0000 7f1603000640 1 Processor -- start 2024-05-12T22:05:50.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:50.049+0000 7f1603000640 1 -- start start 2024-05-12T22:05:50.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:50.050+0000 7f1603000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f15fc1038b0 0x7f15fc0795e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:05:50.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:50.050+0000 7f1603000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f15fc104aa0 0x7f15fc079b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:05:50.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:50.050+0000 7f1603000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15fc105460 0x7f15fc076150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:05:50.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:50.050+0000 7f1603000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f15fc10f580 con 0x7f15fc105460 2024-05-12T22:05:50.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:50.050+0000 7f1603000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f15fc10f400 con 0x7f15fc1038b0 2024-05-12T22:05:50.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:50.050+0000 7f1603000640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f15fc10f700 con 0x7f15fc104aa0 2024-05-12T22:05:50.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:50.050+0000 7f1601c00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f15fc1038b0 0x7f15fc0795e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:05:50.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:50.050+0000 7f1601200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f15fc104aa0 0x7f15fc079b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:05:50.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:50.050+0000 7f1601c00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f15fc1038b0 0x7f15fc0795e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:60126/0 (socket says 172.21.15.28:60126) 2024-05-12T22:05:50.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:50.050+0000 7f1601c00640 1 -- 172.21.15.28:0/1196540447 learned_addr learned my addr 172.21.15.28:0/1196540447 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:05:50.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:50.050+0000 7f1602600640 1 --2- 172.21.15.28:0/1196540447 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15fc105460 0x7f15fc076150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:05:50.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:50.050+0000 7f1601200640 1 -- 172.21.15.28:0/1196540447 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f15fc1038b0 msgr2=0x7f15fc0795e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:05:50.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:50.050+0000 7f1601200640 1 --2- 172.21.15.28:0/1196540447 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f15fc1038b0 0x7f15fc0795e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:05:50.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:50.050+0000 7f1601200640 1 -- 172.21.15.28:0/1196540447 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15fc105460 msgr2=0x7f15fc076150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:05:50.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:50.050+0000 7f1601200640 1 --2- 172.21.15.28:0/1196540447 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15fc105460 0x7f15fc076150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:05:50.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:50.050+0000 7f1601200640 1 -- 172.21.15.28:0/1196540447 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15fc076a10 con 0x7f15fc104aa0 2024-05-12T22:05:50.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:50.050+0000 7f1601c00640 1 --2- 172.21.15.28:0/1196540447 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f15fc1038b0 0x7f15fc0795e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:05:50.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:50.051+0000 7f1601200640 1 --2- 172.21.15.28:0/1196540447 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f15fc104aa0 0x7f15fc079b20 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f15e800e9f0 tx=0x7f15e800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:05:50.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:50.051+0000 7f15fa000640 1 -- 172.21.15.28:0/1196540447 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15e800ccc0 con 0x7f15fc104aa0 2024-05-12T22:05:50.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:50.051+0000 7f15fa000640 1 -- 172.21.15.28:0/1196540447 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f15e8010910 con 0x7f15fc104aa0 2024-05-12T22:05:50.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:50.051+0000 7f15fa000640 1 -- 172.21.15.28:0/1196540447 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15e8018960 con 0x7f15fc104aa0 2024-05-12T22:05:50.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:50.051+0000 7f1603000640 1 -- 172.21.15.28:0/1196540447 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15fc1a6010 con 0x7f15fc104aa0 2024-05-12T22:05:50.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:50.051+0000 7f1603000640 1 -- 172.21.15.28:0/1196540447 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15fc1a6420 con 0x7f15fc104aa0 2024-05-12T22:05:50.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:50.052+0000 7f1603000640 1 -- 172.21.15.28:0/1196540447 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15fc10ab70 con 0x7f15fc104aa0 2024-05-12T22:05:50.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:50.055+0000 7f15fa000640 1 -- 172.21.15.28:0/1196540447 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f15e8010ac0 con 0x7f15fc104aa0 2024-05-12T22:05:50.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:50.057+0000 7f15fa000640 1 --2- 172.21.15.28:0/1196540447 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f15e40a2c10 0x7f15e40a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:05:50.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:50.057+0000 7f15fa000640 1 -- 172.21.15.28:0/1196540447 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f15e80cef30 con 0x7f15fc104aa0 2024-05-12T22:05:50.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:50.057+0000 7f1601c00640 1 --2- 172.21.15.28:0/1196540447 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f15e40a2c10 0x7f15e40a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:05:50.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:50.058+0000 7f1601c00640 1 --2- 172.21.15.28:0/1196540447 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f15e40a2c10 0x7f15e40a50d0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f15f0004ab0 tx=0x7f15f0004a40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:05:50.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:50.061+0000 7f15fa000640 1 -- 172.21.15.28:0/1196540447 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f15e809bc40 con 0x7f15fc104aa0 2024-05-12T22:05:50.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:50.266+0000 7f1603000640 1 -- 172.21.15.28:0/1196540447 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f15fc062530 con 0x7f15fc104aa0 2024-05-12T22:05:50.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:50.267+0000 7f15fa000640 1 -- 172.21.15.28:0/1196540447 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) ==== 74+0+18397 (secure 0 0 0) 0x7f15e809ff40 con 0x7f15fc104aa0 2024-05-12T22:05:50.296 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-12T22:05:50.297 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":82,"fsid":"6b92e7b4-10aa-11ef-bc98-c7b262605968","created":"2024-05-12T21:57:37.276782+0000","modified":"2024-05-12T22:05:38.939521+0000","last_up_change":"2024-05-12T22:05:37.929562+0000","last_in_change":"2024-05-12T22:05:18.750768+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-12T22:01:32.292826+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6992ce0a-844b-40e3-8ed1-181c8474dccf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":906003882},{"type":"v1","addr":"172.21.15.28:6803","nonce":906003882}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":906003882},{"type":"v1","addr":"172.21.15.28:6805","nonce":906003882}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":906003882},{"type":"v1","addr":"172.21.15.28:6809","nonce":906003882}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":906003882},{"type":"v1","addr":"172.21.15.28:6807","nonce":906003882}]},"public_addr":"172.21.15.28:6803/906003882","cluster_addr":"172.21.15.28:6805/906003882","heartbeat_back_addr":"172.21.15.28:6809/906003882","heartbeat_front_addr":"172.21.15.28:6807/906003882","state":["exists","up"]},{"osd":1,"uuid":"ced20c96-c481-4d48-90d1-6b43228959ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":3354141819},{"type":"v1","addr":"172.21.15.86:6801","nonce":3354141819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":3354141819},{"type":"v1","addr":"172.21.15.86:6803","nonce":3354141819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":3354141819},{"type":"v1","addr":"172.21.15.86:6807","nonce":3354141819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":3354141819},{"type":"v1","addr":"172.21.15.86:6805","nonce":3354141819}]},"public_addr":"172.21.15.86:6801/3354141819","cluster_addr":"172.21.15.86:6803/3354141819","heartbeat_back_addr":"172.21.15.86:6807/3354141819","heartbeat_front_addr":"172.21.15.86:6805/3354141819","state":["exists","up"]},{"osd":2,"uuid":"54a2f221-fa65-4795-9f5e-e660cc7c0098","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":906459365},{"type":"v1","addr":"172.21.15.186:6801","nonce":906459365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":906459365},{"type":"v1","addr":"172.21.15.186:6803","nonce":906459365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":906459365},{"type":"v1","addr":"172.21.15.186:6807","nonce":906459365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":906459365},{"type":"v1","addr":"172.21.15.186:6805","nonce":906459365}]},"public_addr":"172.21.15.186:6801/906459365","cluster_addr":"172.21.15.186:6803/906459365","heartbeat_back_addr":"172.21.15.186:6807/906459365","heartbeat_front_addr":"172.21.15.186:6805/906459365","state":["exists","up"]},{"osd":3,"uuid":"86836737-ac9a-487b-93f2-7fcd3c406960","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":4182884436},{"type":"v1","addr":"172.21.15.28:6811","nonce":4182884436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":4182884436},{"type":"v1","addr":"172.21.15.28:6813","nonce":4182884436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":4182884436},{"type":"v1","addr":"172.21.15.28:6817","nonce":4182884436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":4182884436},{"type":"v1","addr":"172.21.15.28:6815","nonce":4182884436}]},"public_addr":"172.21.15.28:6811/4182884436","cluster_addr":"172.21.15.28:6813/4182884436","heartbeat_back_addr":"172.21.15.28:6817/4182884436","heartbeat_front_addr":"172.21.15.28:6815/4182884436","state":["exists","up"]},{"osd":4,"uuid":"d162ebcd-83c0-4fd5-9925-28b2cecea935","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2596998708},{"type":"v1","addr":"172.21.15.86:6809","nonce":2596998708}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2596998708},{"type":"v1","addr":"172.21.15.86:6811","nonce":2596998708}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2596998708},{"type":"v1","addr":"172.21.15.86:6815","nonce":2596998708}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2596998708},{"type":"v1","addr":"172.21.15.86:6813","nonce":2596998708}]},"public_addr":"172.21.15.86:6809/2596998708","cluster_addr":"172.21.15.86:6811/2596998708","heartbeat_back_addr":"172.21.15.86:6815/2596998708","heartbeat_front_addr":"172.21.15.86:6813/2596998708","state":["exists","up"]},{"osd":5,"uuid":"c794336a-a105-425c-b6c7-a32b90c05bb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":1944205756},{"type":"v1","addr":"172.21.15.186:6809","nonce":1944205756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":1944205756},{"type":"v1","addr":"172.21.15.186:6811","nonce":1944205756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":1944205756},{"type":"v1","addr":"172.21.15.186:6815","nonce":1944205756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":1944205756},{"type":"v1","addr":"172.21.15.186:6813","nonce":1944205756}]},"public_addr":"172.21.15.186:6809/1944205756","cluster_addr":"172.21.15.186:6811/1944205756","heartbeat_back_addr":"172.21.15.186:6815/1944205756","heartbeat_front_addr":"172.21.15.186:6813/1944205756","state":["exists","up"]},{"osd":6,"uuid":"516932ac-af00-45b7-ad3c-d6dec7db442f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":1642308925},{"type":"v1","addr":"172.21.15.28:6819","nonce":1642308925}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":1642308925},{"type":"v1","addr":"172.21.15.28:6821","nonce":1642308925}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1642308925},{"type":"v1","addr":"172.21.15.28:6825","nonce":1642308925}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":1642308925},{"type":"v1","addr":"172.21.15.28:6823","nonce":1642308925}]},"public_addr":"172.21.15.28:6819/1642308925","cluster_addr":"172.21.15.28:6821/1642308925","heartbeat_back_addr":"172.21.15.28:6825/1642308925","heartbeat_front_addr":"172.21.15.28:6823/1642308925","state":["exists","up"]},{"osd":7,"uuid":"ca7a059f-0414-438e-b2d1-a10f348036be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":726062711},{"type":"v1","addr":"172.21.15.86:6817","nonce":726062711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":726062711},{"type":"v1","addr":"172.21.15.86:6819","nonce":726062711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":726062711},{"type":"v1","addr":"172.21.15.86:6823","nonce":726062711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":726062711},{"type":"v1","addr":"172.21.15.86:6821","nonce":726062711}]},"public_addr":"172.21.15.86:6817/726062711","cluster_addr":"172.21.15.86:6819/726062711","heartbeat_back_addr":"172.21.15.86:6823/726062711","heartbeat_front_addr":"172.21.15.86:6821/726062711","state":["exists","up"]},{"osd":8,"uuid":"92e3f41f-aebd-422e-a99e-555f7ba4aa53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":3007687277},{"type":"v1","addr":"172.21.15.186:6817","nonce":3007687277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":3007687277},{"type":"v1","addr":"172.21.15.186:6819","nonce":3007687277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":3007687277},{"type":"v1","addr":"172.21.15.186:6823","nonce":3007687277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":3007687277},{"type":"v1","addr":"172.21.15.186:6821","nonce":3007687277}]},"public_addr":"172.21.15.186:6817/3007687277","cluster_addr":"172.21.15.186:6819/3007687277","heartbeat_back_addr":"172.21.15.186:6823/3007687277","heartbeat_front_addr":"172.21.15.186:6821/3007687277","state":["exists","up"]},{"osd":9,"uuid":"38ff7d9d-ee85-404c-8c59-adc327e1ccfa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":136220472},{"type":"v1","addr":"172.21.15.28:6827","nonce":136220472}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":136220472},{"type":"v1","addr":"172.21.15.28:6829","nonce":136220472}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":136220472},{"type":"v1","addr":"172.21.15.28:6833","nonce":136220472}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":136220472},{"type":"v1","addr":"172.21.15.28:6831","nonce":136220472}]},"public_addr":"172.21.15.28:6827/136220472","cluster_addr":"172.21.15.28:6829/136220472","heartbeat_back_addr":"172.21.15.28:6833/136220472","heartbeat_front_addr":"172.21.15.28:6831/136220472","state":["exists","up"]},{"osd":10,"uuid":"5e9ff657-afa8-4bab-8491-f209038392b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1189968310},{"type":"v1","addr":"172.21.15.86:6825","nonce":1189968310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":1189968310},{"type":"v1","addr":"172.21.15.86:6827","nonce":1189968310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":1189968310},{"type":"v1","addr":"172.21.15.86:6831","nonce":1189968310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":1189968310},{"type":"v1","addr":"172.21.15.86:6829","nonce":1189968310}]},"public_addr":"172.21.15.86:6825/1189968310","cluster_addr":"172.21.15.86:6827/1189968310","heartbeat_back_addr":"172.21.15.86:6831/1189968310","heartbeat_front_addr":"172.21.15.86:6829/1189968310","state":["exists","up"]},{"osd":11,"uuid":"f112e3c6-65b6-4253-a966-62001db7d069","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3801668110},{"type":"v1","addr":"172.21.15.186:6825","nonce":3801668110}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3801668110},{"type":"v1","addr":"172.21.15.186:6827","nonce":3801668110}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3801668110},{"type":"v1","addr":"172.21.15.186:6831","nonce":3801668110}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3801668110},{"type":"v1","addr":"172.21.15.186:6829","nonce":3801668110}]},"public_addr":"172.21.15.186:6825/3801668110","cluster_addr":"172.21.15.186:6827/3801668110","heartbeat_back_addr":"172.21.15.186:6831/3801668110","heartbeat_front_addr":"172.21.15.186:6829/3801668110","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:00:40.483623+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:01:05.284385+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:01:28.645716+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:02:01.295092+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:02:26.486601+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:02:50.807478+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:03:24.756169+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:03:49.975173+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:04:13.772233+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:04:46.758555+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:05:12.331685+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:05:36.462548+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/1177009533":"2024-05-13T21:58:45.550459+0000","172.21.15.28:6800/3760499540":"2024-05-13T21:58:45.550459+0000","172.21.15.28:6801/3783742728":"2024-05-13T21:58:10.421564+0000","172.21.15.28:6800/3783742728":"2024-05-13T21:58:10.421564+0000","172.21.15.28:0/2229156260":"2024-05-13T21:58:45.550459+0000","172.21.15.28:0/1202140963":"2024-05-13T21:58:10.421564+0000","172.21.15.28:0/2924223784":"2024-05-13T21:58:45.550459+0000","172.21.15.28:6801/3760499540":"2024-05-13T21:58:45.550459+0000","172.21.15.28:0/1240027206":"2024-05-13T21:58:10.421564+0000","172.21.15.28:0/171417728":"2024-05-13T21:58:10.421564+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-12T22:05:50.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:50.270+0000 7f1603000640 1 -- 172.21.15.28:0/1196540447 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f15e40a2c10 msgr2=0x7f15e40a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:05:50.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:50.270+0000 7f1603000640 1 --2- 172.21.15.28:0/1196540447 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f15e40a2c10 0x7f15e40a50d0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f15f0004ab0 tx=0x7f15f0004a40 comp rx=0 tx=0).stop 2024-05-12T22:05:50.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:50.270+0000 7f1603000640 1 -- 172.21.15.28:0/1196540447 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f15fc104aa0 msgr2=0x7f15fc079b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:05:50.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:50.270+0000 7f1603000640 1 --2- 172.21.15.28:0/1196540447 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f15fc104aa0 0x7f15fc079b20 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f15e800e9f0 tx=0x7f15e800eeb0 comp rx=0 tx=0).stop 2024-05-12T22:05:50.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:50.270+0000 7f1603000640 1 -- 172.21.15.28:0/1196540447 shutdown_connections 2024-05-12T22:05:50.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:50.270+0000 7f1603000640 1 --2- 172.21.15.28:0/1196540447 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f15e40a2c10 0x7f15e40a50d0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:05:50.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:50.270+0000 7f1603000640 1 --2- 172.21.15.28:0/1196540447 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15fc105460 0x7f15fc076150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:05:50.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:50.270+0000 7f1603000640 1 --2- 172.21.15.28:0/1196540447 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f15fc104aa0 0x7f15fc079b20 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:05:50.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:50.270+0000 7f1603000640 1 --2- 172.21.15.28:0/1196540447 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f15fc1038b0 0x7f15fc0795e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:05:50.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:50.270+0000 7f1603000640 1 -- 172.21.15.28:0/1196540447 >> 172.21.15.28:0/1196540447 conn(0x7f15fc0fee40 msgr2=0x7f15fc107d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:05:50.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:50.270+0000 7f1603000640 1 -- 172.21.15.28:0/1196540447 shutdown_connections 2024-05-12T22:05:50.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:50.270+0000 7f1603000640 1 -- 172.21.15.28:0/1196540447 wait complete. 2024-05-12T22:05:50.299 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-12T22:01:32.292826+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-12T22:05:50.300 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-12T22:05:50.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:50.517+0000 7f7918c00640 1 -- 172.21.15.28:0/94336182 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7914101830 msgr2=0x7f7914101cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:05:50.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:50.517+0000 7f7918c00640 1 --2- 172.21.15.28:0/94336182 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7914101830 0x7f7914101cb0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f78fc009900 tx=0x7f78fc02a3f0 comp rx=0 tx=0).stop 2024-05-12T22:05:50.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:50.517+0000 7f7918c00640 1 -- 172.21.15.28:0/94336182 shutdown_connections 2024-05-12T22:05:50.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:50.517+0000 7f7918c00640 1 --2- 172.21.15.28:0/94336182 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f79141021f0 0x7f7914108a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:05:50.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:50.517+0000 7f7918c00640 1 --2- 172.21.15.28:0/94336182 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7914101830 0x7f7914101cb0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:05:50.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:50.517+0000 7f7918c00640 1 --2- 172.21.15.28:0/94336182 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7914100640 0x7f7914100a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:05:50.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:50.517+0000 7f7918c00640 1 -- 172.21.15.28:0/94336182 >> 172.21.15.28:0/94336182 conn(0x7f79140fbbd0 msgr2=0x7f79140fdff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:05:50.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:50.517+0000 7f7918c00640 1 -- 172.21.15.28:0/94336182 shutdown_connections 2024-05-12T22:05:50.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:50.517+0000 7f7918c00640 1 -- 172.21.15.28:0/94336182 wait complete. 2024-05-12T22:05:50.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:50.518+0000 7f7918c00640 1 Processor -- start 2024-05-12T22:05:50.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:50.519+0000 7f7918c00640 1 -- start start 2024-05-12T22:05:50.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:50.519+0000 7f7918c00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7914100640 0x7f7914199c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:05:50.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:50.519+0000 7f7918c00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7914101830 0x7f791419a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:05:50.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:50.519+0000 7f7918c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79141021f0 0x7f79141a1230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:05:50.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:50.519+0000 7f7918c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f791410c2f0 con 0x7f79141021f0 2024-05-12T22:05:50.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:50.519+0000 7f7918c00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f791410c170 con 0x7f7914100640 2024-05-12T22:05:50.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:50.519+0000 7f7918c00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f791410c470 con 0x7f7914101830 2024-05-12T22:05:50.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:50.519+0000 7f7912a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7914101830 0x7f791419a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:05:50.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:50.519+0000 7f7912a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7914101830 0x7f791419a1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:33250/0 (socket says 172.21.15.28:33250) 2024-05-12T22:05:50.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:50.519+0000 7f7913400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7914100640 0x7f7914199c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:05:50.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:50.519+0000 7f7912a00640 1 -- 172.21.15.28:0/3567243855 learned_addr learned my addr 172.21.15.28:0/3567243855 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:05:50.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:50.519+0000 7f7913e00640 1 --2- 172.21.15.28:0/3567243855 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79141021f0 0x7f79141a1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:05:50.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:50.520+0000 7f7913400640 1 -- 172.21.15.28:0/3567243855 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7914101830 msgr2=0x7f791419a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:05:50.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:50.520+0000 7f7913400640 1 --2- 172.21.15.28:0/3567243855 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7914101830 0x7f791419a1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:05:50.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:50.520+0000 7f7913400640 1 -- 172.21.15.28:0/3567243855 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79141021f0 msgr2=0x7f79141a1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:05:50.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:50.520+0000 7f7913400640 1 --2- 172.21.15.28:0/3567243855 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79141021f0 0x7f79141a1230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:05:50.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:50.520+0000 7f7913400640 1 -- 172.21.15.28:0/3567243855 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79141a18d0 con 0x7f7914100640 2024-05-12T22:05:50.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:50.520+0000 7f7912a00640 1 --2- 172.21.15.28:0/3567243855 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7914101830 0x7f791419a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:05:50.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:50.520+0000 7f7913400640 1 --2- 172.21.15.28:0/3567243855 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7914100640 0x7f7914199c90 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f7904005fd0 tx=0x7f790400c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:05:50.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:50.520+0000 7f790be00640 1 -- 172.21.15.28:0/3567243855 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7904010630 con 0x7f7914100640 2024-05-12T22:05:50.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:50.520+0000 7f7918c00640 1 -- 172.21.15.28:0/3567243855 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79141a1bc0 con 0x7f7914100640 2024-05-12T22:05:50.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:50.520+0000 7f790be00640 1 -- 172.21.15.28:0/3567243855 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f790400ee90 con 0x7f7914100640 2024-05-12T22:05:50.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:50.520+0000 7f790be00640 1 -- 172.21.15.28:0/3567243855 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7904018cb0 con 0x7f7914100640 2024-05-12T22:05:50.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:50.521+0000 7f7918c00640 1 -- 172.21.15.28:0/3567243855 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79141a2110 con 0x7f7914100640 2024-05-12T22:05:50.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:50.522+0000 7f7918c00640 1 -- 172.21.15.28:0/3567243855 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79141a22e0 con 0x7f7914100640 2024-05-12T22:05:50.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:50.524+0000 7f790be00640 1 -- 172.21.15.28:0/3567243855 <== mon.2 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f790401f020 con 0x7f7914100640 2024-05-12T22:05:50.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:50.525+0000 7f790be00640 1 --2- 172.21.15.28:0/3567243855 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f78f80a2b90 0x7f78f80a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:05:50.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:50.525+0000 7f790be00640 1 -- 172.21.15.28:0/3567243855 <== mon.2 v2:172.21.15.86:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f79040cf070 con 0x7f7914100640 2024-05-12T22:05:50.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:50.526+0000 7f7912a00640 1 --2- 172.21.15.28:0/3567243855 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f78f80a2b90 0x7f78f80a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:05:50.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:50.526+0000 7f7912a00640 1 --2- 172.21.15.28:0/3567243855 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f78f80a2b90 0x7f78f80a5050 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f791419b190 tx=0x7f78fc0315b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:05:50.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:50.530+0000 7f790be00640 1 -- 172.21.15.28:0/3567243855 <== mon.2 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f7904053850 con 0x7f7914100640 2024-05-12T22:05:50.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:50.736+0000 7f7918c00640 1 -- 172.21.15.28:0/3567243855 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f79141a2670 con 0x7f7914100640 2024-05-12T22:05:50.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:50.737+0000 7f790be00640 1 -- 172.21.15.28:0/3567243855 <== mon.2 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) ==== 93+0+10 (secure 0 0 0) 0x7f790409c3d0 con 0x7f7914100640 2024-05-12T22:05:50.738 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-05-12T22:05:50.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:50.740+0000 7f7918c00640 1 -- 172.21.15.28:0/3567243855 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f78f80a2b90 msgr2=0x7f78f80a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:05:50.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:50.740+0000 7f7918c00640 1 --2- 172.21.15.28:0/3567243855 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f78f80a2b90 0x7f78f80a5050 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f791419b190 tx=0x7f78fc0315b0 comp rx=0 tx=0).stop 2024-05-12T22:05:50.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:50.740+0000 7f7918c00640 1 -- 172.21.15.28:0/3567243855 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7914100640 msgr2=0x7f7914199c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:05:50.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:50.740+0000 7f7918c00640 1 --2- 172.21.15.28:0/3567243855 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7914100640 0x7f7914199c90 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f7904005fd0 tx=0x7f790400c6a0 comp rx=0 tx=0).stop 2024-05-12T22:05:50.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:50.740+0000 7f7918c00640 1 -- 172.21.15.28:0/3567243855 shutdown_connections 2024-05-12T22:05:50.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:50.740+0000 7f7918c00640 1 --2- 172.21.15.28:0/3567243855 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f78f80a2b90 0x7f78f80a5050 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:05:50.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:50.740+0000 7f7918c00640 1 --2- 172.21.15.28:0/3567243855 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79141021f0 0x7f79141a1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:05:50.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:50.740+0000 7f7918c00640 1 --2- 172.21.15.28:0/3567243855 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7914101830 0x7f791419a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:05:50.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:50.740+0000 7f7918c00640 1 --2- 172.21.15.28:0/3567243855 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7914100640 0x7f7914199c90 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:05:50.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:50.740+0000 7f7918c00640 1 -- 172.21.15.28:0/3567243855 >> 172.21.15.28:0/3567243855 conn(0x7f79140fbbd0 msgr2=0x7f79140fd660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:05:50.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:50.740+0000 7f7918c00640 1 -- 172.21.15.28:0/3567243855 shutdown_connections 2024-05-12T22:05:50.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:50.741+0000 7f7918c00640 1 -- 172.21.15.28:0/3567243855 wait complete. 2024-05-12T22:05:50.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:50 smithi086 ceph-mon[34297]: from='client.24403 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi028=a;smithi028=d;smithi028=g;smithi028=j;smithi086=b;smithi086=e;smithi086=h;smithi086=k;smithi186=c;smithi186=f;smithi186=i;smithi186=l", "target": ["mon-mgr", ""]}]: dispatch 2024-05-12T22:05:50.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:50 smithi086 ceph-mon[34297]: Saving service mds.all spec with placement smithi028=a;smithi028=d;smithi028=g;smithi028=j;smithi086=b;smithi086=e;smithi086=h;smithi086=k;smithi186=c;smithi186=f;smithi186=i;smithi186=l;count:12 2024-05-12T22:05:50.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:50 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:05:50.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:50 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:05:50.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:50 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:05:50.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:50 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-12T22:05:50.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:50 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-12T22:05:50.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:50 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:05:50.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:50 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/1196540447' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-12T22:05:50.754 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:50 smithi186 ceph-mon[34785]: from='client.24403 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi028=a;smithi028=d;smithi028=g;smithi028=j;smithi086=b;smithi086=e;smithi086=h;smithi086=k;smithi186=c;smithi186=f;smithi186=i;smithi186=l", "target": ["mon-mgr", ""]}]: dispatch 2024-05-12T22:05:50.754 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:50 smithi186 ceph-mon[34785]: Saving service mds.all spec with placement smithi028=a;smithi028=d;smithi028=g;smithi028=j;smithi086=b;smithi086=e;smithi086=h;smithi086=k;smithi186=c;smithi186=f;smithi186=i;smithi186=l;count:12 2024-05-12T22:05:50.754 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:50 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:05:50.754 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:50 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:05:50.754 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:50 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:05:50.754 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:50 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-12T22:05:50.754 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:50 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-12T22:05:50.754 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:50 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:05:50.755 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:50 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/1196540447' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-12T22:05:50.759 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:50 smithi028 ceph-mon[29847]: from='client.24403 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi028=a;smithi028=d;smithi028=g;smithi028=j;smithi086=b;smithi086=e;smithi086=h;smithi086=k;smithi186=c;smithi186=f;smithi186=i;smithi186=l", "target": ["mon-mgr", ""]}]: dispatch 2024-05-12T22:05:50.759 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:50 smithi028 ceph-mon[29847]: Saving service mds.all spec with placement smithi028=a;smithi028=d;smithi028=g;smithi028=j;smithi086=b;smithi086=e;smithi086=h;smithi086=k;smithi186=c;smithi186=f;smithi186=i;smithi186=l;count:12 2024-05-12T22:05:50.759 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:50 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:05:50.759 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:50 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:05:50.759 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:50 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:05:50.759 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:50 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-12T22:05:50.759 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:50 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-12T22:05:50.759 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:50 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:05:50.759 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:50 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/1196540447' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-12T22:05:50.761 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-12T22:05:51.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:51.021+0000 7f8d3b400640 1 -- 172.21.15.28:0/2724943135 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d34077430 msgr2=0x7f8d34077890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:05:51.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:51.021+0000 7f8d3b400640 1 --2- 172.21.15.28:0/2724943135 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d34077430 0x7f8d34077890 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f8d20009900 tx=0x7f8d2002a400 comp rx=0 tx=0).stop 2024-05-12T22:05:51.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:51.021+0000 7f8d3b400640 1 -- 172.21.15.28:0/2724943135 shutdown_connections 2024-05-12T22:05:51.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:51.021+0000 7f8d3b400640 1 --2- 172.21.15.28:0/2724943135 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8d34077dd0 0x7f8d34078670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:05:51.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:51.021+0000 7f8d3b400640 1 --2- 172.21.15.28:0/2724943135 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d34077430 0x7f8d34077890 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:05:51.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:51.021+0000 7f8d3b400640 1 --2- 172.21.15.28:0/2724943135 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8d340761f0 0x7f8d34076610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:05:51.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:51.021+0000 7f8d3b400640 1 -- 172.21.15.28:0/2724943135 >> 172.21.15.28:0/2724943135 conn(0x7f8d340ff020 msgr2=0x7f8d34101460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:05:51.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:51.021+0000 7f8d3b400640 1 -- 172.21.15.28:0/2724943135 shutdown_connections 2024-05-12T22:05:51.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:51.021+0000 7f8d3b400640 1 -- 172.21.15.28:0/2724943135 wait complete. 2024-05-12T22:05:51.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:51.022+0000 7f8d3b400640 1 Processor -- start 2024-05-12T22:05:51.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:51.023+0000 7f8d3b400640 1 -- start start 2024-05-12T22:05:51.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:51.023+0000 7f8d3b400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8d340761f0 0x7f8d3419e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:05:51.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:51.023+0000 7f8d3b400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8d34077430 0x7f8d3419e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:05:51.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:51.023+0000 7f8d3b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d34077dd0 0x7f8d341a5750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:05:51.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:51.023+0000 7f8d3b400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8d34113b40 con 0x7f8d34077dd0 2024-05-12T22:05:51.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:51.023+0000 7f8d3b400640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f8d341139c0 con 0x7f8d34077430 2024-05-12T22:05:51.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:51.023+0000 7f8d3b400640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f8d34113cc0 con 0x7f8d340761f0 2024-05-12T22:05:51.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:51.023+0000 7f8d39400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d34077dd0 0x7f8d341a5750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:05:51.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:51.023+0000 7f8d38a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8d340761f0 0x7f8d3419e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:05:51.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:51.023+0000 7f8d38a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8d340761f0 0x7f8d3419e1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:33264/0 (socket says 172.21.15.28:33264) 2024-05-12T22:05:51.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:51.023+0000 7f8d38a00640 1 -- 172.21.15.28:0/468157790 learned_addr learned my addr 172.21.15.28:0/468157790 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:05:51.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:51.023+0000 7f8d33e00640 1 --2- 172.21.15.28:0/468157790 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8d34077430 0x7f8d3419e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:05:51.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:51.024+0000 7f8d39400640 1 -- 172.21.15.28:0/468157790 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8d340761f0 msgr2=0x7f8d3419e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:05:51.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:51.024+0000 7f8d39400640 1 --2- 172.21.15.28:0/468157790 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8d340761f0 0x7f8d3419e1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:05:51.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:51.024+0000 7f8d39400640 1 -- 172.21.15.28:0/468157790 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8d34077430 msgr2=0x7f8d3419e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:05:51.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:51.024+0000 7f8d39400640 1 --2- 172.21.15.28:0/468157790 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8d34077430 0x7f8d3419e6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:05:51.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:51.024+0000 7f8d39400640 1 -- 172.21.15.28:0/468157790 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d341a5dc0 con 0x7f8d34077dd0 2024-05-12T22:05:51.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:51.024+0000 7f8d38a00640 1 --2- 172.21.15.28:0/468157790 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8d340761f0 0x7f8d3419e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:05:51.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:51.024+0000 7f8d39400640 1 --2- 172.21.15.28:0/468157790 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d34077dd0 0x7f8d341a5750 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f8d2c009fd0 tx=0x7f8d2c00ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:05:51.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:51.024+0000 7f8d31600640 1 -- 172.21.15.28:0/468157790 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d2c00ce20 con 0x7f8d34077dd0 2024-05-12T22:05:51.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:51.025+0000 7f8d31600640 1 -- 172.21.15.28:0/468157790 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8d2c005020 con 0x7f8d34077dd0 2024-05-12T22:05:51.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:51.025+0000 7f8d3b400640 1 -- 172.21.15.28:0/468157790 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d341a60b0 con 0x7f8d34077dd0 2024-05-12T22:05:51.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:51.025+0000 7f8d31600640 1 -- 172.21.15.28:0/468157790 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d2c010720 con 0x7f8d34077dd0 2024-05-12T22:05:51.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:51.025+0000 7f8d3b400640 1 -- 172.21.15.28:0/468157790 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d341a65d0 con 0x7f8d34077dd0 2024-05-12T22:05:51.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:51.027+0000 7f8d3b400640 1 -- 172.21.15.28:0/468157790 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d34110600 con 0x7f8d34077dd0 2024-05-12T22:05:51.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:51.029+0000 7f8d31600640 1 -- 172.21.15.28:0/468157790 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f8d2c005540 con 0x7f8d34077dd0 2024-05-12T22:05:51.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:51.031+0000 7f8d31600640 1 --2- 172.21.15.28:0/468157790 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f8d1c0a2c10 0x7f8d1c0a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:05:51.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:51.031+0000 7f8d31600640 1 -- 172.21.15.28:0/468157790 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f8d2c0cf2c0 con 0x7f8d34077dd0 2024-05-12T22:05:51.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:51.031+0000 7f8d38a00640 1 --2- 172.21.15.28:0/468157790 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f8d1c0a2c10 0x7f8d1c0a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:05:51.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:51.032+0000 7f8d38a00640 1 --2- 172.21.15.28:0/468157790 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f8d1c0a2c10 0x7f8d1c0a50d0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f8d28011550 tx=0x7f8d28005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:05:51.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:51.036+0000 7f8d31600640 1 -- 172.21.15.28:0/468157790 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f8d2c09c620 con 0x7f8d34077dd0 2024-05-12T22:05:51.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:51.285+0000 7f8d3b400640 1 -- 172.21.15.28:0/468157790 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8d3402d1c0 con 0x7f8d34077dd0 2024-05-12T22:05:51.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:51.285+0000 7f8d31600640 1 -- 172.21.15.28:0/468157790 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) ==== 93+0+569 (secure 0 0 0) 0x7f8d2c09bf20 con 0x7f8d34077dd0 2024-05-12T22:05:51.286 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-12T22:05:51.286 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-05-12T22:05:51.287 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 1 2024-05-12T22:05:51.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:51.288+0000 7f8d3b400640 1 -- 172.21.15.28:0/468157790 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f8d1c0a2c10 msgr2=0x7f8d1c0a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:05:51.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:51.288+0000 7f8d3b400640 1 --2- 172.21.15.28:0/468157790 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f8d1c0a2c10 0x7f8d1c0a50d0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f8d28011550 tx=0x7f8d28005f90 comp rx=0 tx=0).stop 2024-05-12T22:05:51.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:51.288+0000 7f8d3b400640 1 -- 172.21.15.28:0/468157790 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d34077dd0 msgr2=0x7f8d341a5750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:05:51.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:51.288+0000 7f8d3b400640 1 --2- 172.21.15.28:0/468157790 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d34077dd0 0x7f8d341a5750 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f8d2c009fd0 tx=0x7f8d2c00ef10 comp rx=0 tx=0).stop 2024-05-12T22:05:51.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:51.289+0000 7f8d3b400640 1 -- 172.21.15.28:0/468157790 shutdown_connections 2024-05-12T22:05:51.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:51.289+0000 7f8d3b400640 1 --2- 172.21.15.28:0/468157790 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f8d1c0a2c10 0x7f8d1c0a50d0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:05:51.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:51.289+0000 7f8d3b400640 1 --2- 172.21.15.28:0/468157790 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d34077dd0 0x7f8d341a5750 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:05:51.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:51.289+0000 7f8d3b400640 1 --2- 172.21.15.28:0/468157790 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8d34077430 0x7f8d3419e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:05:51.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:51.289+0000 7f8d3b400640 1 --2- 172.21.15.28:0/468157790 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8d340761f0 0x7f8d3419e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:05:51.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:51.289+0000 7f8d3b400640 1 -- 172.21.15.28:0/468157790 >> 172.21.15.28:0/468157790 conn(0x7f8d340ff020 msgr2=0x7f8d34100730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:05:51.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:51.289+0000 7f8d3b400640 1 -- 172.21.15.28:0/468157790 shutdown_connections 2024-05-12T22:05:51.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:51.289+0000 7f8d3b400640 1 -- 172.21.15.28:0/468157790 wait complete. 2024-05-12T22:05:51.554 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:51 smithi186 ceph-mon[34785]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-12T22:05:51.555 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:51 smithi186 ceph-mon[34785]: Deploying daemon mds.l on smithi186 2024-05-12T22:05:51.555 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:51 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/3567243855' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-12T22:05:51.555 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:51 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/468157790' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:05:51.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:51 smithi086 ceph-mon[34297]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-12T22:05:51.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:51 smithi086 ceph-mon[34297]: Deploying daemon mds.l on smithi186 2024-05-12T22:05:51.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:51 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/3567243855' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-12T22:05:51.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:51 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/468157790' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:05:51.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:51 smithi028 ceph-mon[29847]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-12T22:05:51.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:51 smithi028 ceph-mon[29847]: Deploying daemon mds.l on smithi186 2024-05-12T22:05:51.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:51 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/3567243855' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-12T22:05:51.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:51 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/468157790' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:05:52.179 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 12 22:05:52 smithi186 podman[64999]: 2024-05-12 22:05:52.159086649 +0000 UTC m=+0.139178319 container create c639841ebac2e8ba315fd23434a49a83d9354149f555fa3a8ccd3bb2e49e8ac4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-l, GIT_BRANCH=HEAD, org.label-schema.build-date=20240507, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=squid-f72fecf, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-05-12T22:05:52.679 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 12 22:05:52 smithi186 podman[64999]: 2024-05-12 22:05:52.36925148 +0000 UTC m=+0.349343150 container init c639841ebac2e8ba315fd23434a49a83d9354149f555fa3a8ccd3bb2e49e8ac4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-l, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240507, org.label-schema.schema-version=1.0, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, GIT_BRANCH=HEAD, RELEASE=squid-f72fecf) 2024-05-12T22:05:52.679 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 12 22:05:52 smithi186 podman[64999]: 2024-05-12 22:05:52.371744917 +0000 UTC m=+0.351836591 container start c639841ebac2e8ba315fd23434a49a83d9354149f555fa3a8ccd3bb2e49e8ac4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-l, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240507, RELEASE=squid-f72fecf, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True) 2024-05-12T22:05:52.679 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 12 22:05:52 smithi186 ceph-mds[65018]: -- 172.21.15.186:0/3413860606 <== mon.1 v2:172.21.15.186:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55f262d5ba40 con 0x55f262d2ec00 2024-05-12T22:05:52.680 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 12 22:05:52 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-l[65014]: starting mds.l at 2024-05-12T22:05:52.680 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 12 22:05:52 smithi186 bash[64999]: c639841ebac2e8ba315fd23434a49a83d9354149f555fa3a8ccd3bb2e49e8ac4 2024-05-12T22:05:52.680 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 12 22:05:52 smithi186 systemd[1]: Started Ceph mds.l for 6b92e7b4-10aa-11ef-bc98-c7b262605968. 2024-05-12T22:05:53.312 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-12T22:05:53.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:53 smithi086 ceph-mon[34297]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-12T22:05:53.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:53 smithi086 ceph-mon[34297]: mds.? [v2:172.21.15.186:6832/1844141745,v1:172.21.15.186:6833/1844141745] up:boot 2024-05-12T22:05:53.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:53 smithi086 ceph-mon[34297]: fsmap 1 up:standby 2024-05-12T22:05:53.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:53 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-12T22:05:53.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:53 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-12T22:05:53.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:53 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:05:53.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:53 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:05:53.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:53 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:05:53.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:53 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-12T22:05:53.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:53 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-12T22:05:53.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:53 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:05:53.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:53.542+0000 7fc63f800640 1 -- 172.21.15.28:0/2349765247 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc638102a70 msgr2=0x7fc638102ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:05:53.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:53.542+0000 7fc63f800640 1 --2- 172.21.15.28:0/2349765247 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc638102a70 0x7fc638102ef0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fc628009900 tx=0x7fc62802a3f0 comp rx=0 tx=0).stop 2024-05-12T22:05:53.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:53.542+0000 7fc63f800640 1 -- 172.21.15.28:0/2349765247 shutdown_connections 2024-05-12T22:05:53.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:53.542+0000 7fc63f800640 1 --2- 172.21.15.28:0/2349765247 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc638103430 0x7fc638109cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:05:53.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:53.542+0000 7fc63f800640 1 --2- 172.21.15.28:0/2349765247 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc638102a70 0x7fc638102ef0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:05:53.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:53.542+0000 7fc63f800640 1 --2- 172.21.15.28:0/2349765247 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc638101880 0x7fc638101ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:05:53.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:53.542+0000 7fc63f800640 1 -- 172.21.15.28:0/2349765247 >> 172.21.15.28:0/2349765247 conn(0x7fc6380fce10 msgr2=0x7fc6380ff230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:05:53.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:53.542+0000 7fc63f800640 1 -- 172.21.15.28:0/2349765247 shutdown_connections 2024-05-12T22:05:53.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:53.543+0000 7fc63f800640 1 -- 172.21.15.28:0/2349765247 wait complete. 2024-05-12T22:05:53.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:53.543+0000 7fc63f800640 1 Processor -- start 2024-05-12T22:05:53.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:53.544+0000 7fc63f800640 1 -- start start 2024-05-12T22:05:53.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:53.544+0000 7fc63f800640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc638101880 0x7fc638197bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:05:53.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:53.544+0000 7fc63f800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc638102a70 0x7fc638198130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:05:53.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:53.544+0000 7fc63f800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc638103430 0x7fc63819f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:05:53.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:53.544+0000 7fc63f800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc63810d550 con 0x7fc638102a70 2024-05-12T22:05:53.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:53.544+0000 7fc63f800640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fc63810d3d0 con 0x7fc638103430 2024-05-12T22:05:53.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:53.544+0000 7fc63f800640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc63810d6d0 con 0x7fc638101880 2024-05-12T22:05:53.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:53.544+0000 7fc62fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc638102a70 0x7fc638198130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:05:53.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:53.544+0000 7fc63d800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc638103430 0x7fc63819f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:05:53.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:53.544+0000 7fc62fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc638102a70 0x7fc638198130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42518/0 (socket says 172.21.15.28:42518) 2024-05-12T22:05:53.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:53.544+0000 7fc62fe00640 1 -- 172.21.15.28:0/3263542811 learned_addr learned my addr 172.21.15.28:0/3263542811 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:05:53.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:53.544+0000 7fc63ce00640 1 --2- 172.21.15.28:0/3263542811 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc638101880 0x7fc638197bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:05:53.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:53.545+0000 7fc62fe00640 1 -- 172.21.15.28:0/3263542811 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc638101880 msgr2=0x7fc638197bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:05:53.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:53.545+0000 7fc62fe00640 1 --2- 172.21.15.28:0/3263542811 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc638101880 0x7fc638197bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:05:53.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:53.545+0000 7fc62fe00640 1 -- 172.21.15.28:0/3263542811 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc638103430 msgr2=0x7fc63819f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:05:53.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:53.545+0000 7fc62fe00640 1 --2- 172.21.15.28:0/3263542811 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc638103430 0x7fc63819f1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:05:53.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:53.545+0000 7fc62fe00640 1 -- 172.21.15.28:0/3263542811 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc63819f810 con 0x7fc638102a70 2024-05-12T22:05:53.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:53.545+0000 7fc63d800640 1 --2- 172.21.15.28:0/3263542811 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc638103430 0x7fc63819f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:05:53.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:53.545+0000 7fc62fe00640 1 --2- 172.21.15.28:0/3263542811 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc638102a70 0x7fc638198130 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fc628031440 tx=0x7fc628032ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:05:53.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:53.545+0000 7fc636000640 1 -- 172.21.15.28:0/3263542811 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc62803f030 con 0x7fc638102a70 2024-05-12T22:05:53.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:53.545+0000 7fc636000640 1 -- 172.21.15.28:0/3263542811 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc628003ce0 con 0x7fc638102a70 2024-05-12T22:05:53.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:53.545+0000 7fc63f800640 1 -- 172.21.15.28:0/3263542811 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc63819faa0 con 0x7fc638102a70 2024-05-12T22:05:53.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:53.545+0000 7fc636000640 1 -- 172.21.15.28:0/3263542811 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc628035070 con 0x7fc638102a70 2024-05-12T22:05:53.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:53.546+0000 7fc63f800640 1 -- 172.21.15.28:0/3263542811 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc63819ffe0 con 0x7fc638102a70 2024-05-12T22:05:53.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:53.547+0000 7fc63f800640 1 -- 172.21.15.28:0/3263542811 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc638108c20 con 0x7fc638102a70 2024-05-12T22:05:53.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:53.549+0000 7fc636000640 1 -- 172.21.15.28:0/3263542811 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fc628042030 con 0x7fc638102a70 2024-05-12T22:05:53.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:53.550+0000 7fc636000640 1 --2- 172.21.15.28:0/3263542811 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fc6180a2c10 0x7fc6180a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:05:53.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:53.550+0000 7fc636000640 1 -- 172.21.15.28:0/3263542811 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fc6280ef550 con 0x7fc638102a70 2024-05-12T22:05:53.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:53.550+0000 7fc63ce00640 1 --2- 172.21.15.28:0/3263542811 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fc6180a2c10 0x7fc6180a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:05:53.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:53.551+0000 7fc63ce00640 1 --2- 172.21.15.28:0/3263542811 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fc6180a2c10 0x7fc6180a50d0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fc620005fd0 tx=0x7fc620005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:05:53.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:53.554+0000 7fc636000640 1 -- 172.21.15.28:0/3263542811 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fc6280bc8b0 con 0x7fc638102a70 2024-05-12T22:05:53.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:53 smithi186 ceph-mon[34785]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-12T22:05:53.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:53 smithi186 ceph-mon[34785]: mds.? [v2:172.21.15.186:6832/1844141745,v1:172.21.15.186:6833/1844141745] up:boot 2024-05-12T22:05:53.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:53 smithi186 ceph-mon[34785]: fsmap 1 up:standby 2024-05-12T22:05:53.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:53 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-12T22:05:53.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:53 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-12T22:05:53.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:53 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:05:53.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:53 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:05:53.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:53 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:05:53.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:53 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-12T22:05:53.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:53 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-12T22:05:53.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:53 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:05:53.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:53 smithi028 ceph-mon[29847]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-12T22:05:53.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:53 smithi028 ceph-mon[29847]: mds.? [v2:172.21.15.186:6832/1844141745,v1:172.21.15.186:6833/1844141745] up:boot 2024-05-12T22:05:53.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:53 smithi028 ceph-mon[29847]: fsmap 1 up:standby 2024-05-12T22:05:53.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:53 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-12T22:05:53.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:53 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-12T22:05:53.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:53 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:05:53.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:53 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:05:53.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:53 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:05:53.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:53 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-12T22:05:53.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:53 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-12T22:05:53.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:53 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:05:53.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:53.799+0000 7fc63f800640 1 -- 172.21.15.28:0/3263542811 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc638062430 con 0x7fc638102a70 2024-05-12T22:05:53.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:53.800+0000 7fc636000640 1 -- 172.21.15.28:0/3263542811 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1396 (secure 0 0 0) 0x7fc6280bc1b0 con 0x7fc638102a70 2024-05-12T22:05:53.800 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-12T22:05:53.800 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1844141745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1844141745},{"type":"v1","addr":"172.21.15.186:6833","nonce":1844141745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-05-12T22:05:53.801 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 2 2024-05-12T22:05:53.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:53.802+0000 7fc63f800640 1 -- 172.21.15.28:0/3263542811 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fc6180a2c10 msgr2=0x7fc6180a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:05:53.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:53.802+0000 7fc63f800640 1 --2- 172.21.15.28:0/3263542811 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fc6180a2c10 0x7fc6180a50d0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fc620005fd0 tx=0x7fc620005850 comp rx=0 tx=0).stop 2024-05-12T22:05:53.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:53.802+0000 7fc63f800640 1 -- 172.21.15.28:0/3263542811 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc638102a70 msgr2=0x7fc638198130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:05:53.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:53.802+0000 7fc63f800640 1 --2- 172.21.15.28:0/3263542811 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc638102a70 0x7fc638198130 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fc628031440 tx=0x7fc628032ce0 comp rx=0 tx=0).stop 2024-05-12T22:05:53.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:53.803+0000 7fc63f800640 1 -- 172.21.15.28:0/3263542811 shutdown_connections 2024-05-12T22:05:53.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:53.803+0000 7fc63f800640 1 --2- 172.21.15.28:0/3263542811 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fc6180a2c10 0x7fc6180a50d0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:05:53.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:53.803+0000 7fc63f800640 1 --2- 172.21.15.28:0/3263542811 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc638103430 0x7fc63819f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:05:53.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:53.803+0000 7fc63f800640 1 --2- 172.21.15.28:0/3263542811 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc638102a70 0x7fc638198130 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:05:53.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:53.803+0000 7fc63f800640 1 --2- 172.21.15.28:0/3263542811 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc638101880 0x7fc638197bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:05:53.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:53.803+0000 7fc63f800640 1 -- 172.21.15.28:0/3263542811 >> 172.21.15.28:0/3263542811 conn(0x7fc6380fce10 msgr2=0x7fc638105df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:05:53.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:53.803+0000 7fc63f800640 1 -- 172.21.15.28:0/3263542811 shutdown_connections 2024-05-12T22:05:53.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:53.803+0000 7fc63f800640 1 -- 172.21.15.28:0/3263542811 wait complete. 2024-05-12T22:05:54.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:54 smithi186 ceph-mon[34785]: Deploying daemon mds.k on smithi086 2024-05-12T22:05:54.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:54 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/3263542811' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:05:54.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:54 smithi086 ceph-mon[34297]: Deploying daemon mds.k on smithi086 2024-05-12T22:05:54.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:54 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/3263542811' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:05:54.742 INFO:journalctl@ceph.mds.k.smithi086.stdout:May 12 22:05:54 smithi086 systemd[1]: Starting Ceph mds.k for 6b92e7b4-10aa-11ef-bc98-c7b262605968... 2024-05-12T22:05:54.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:54 smithi028 ceph-mon[29847]: Deploying daemon mds.k on smithi086 2024-05-12T22:05:54.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:54 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/3263542811' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:05:55.140 INFO:journalctl@ceph.mds.k.smithi086.stdout:May 12 22:05:54 smithi086 podman[63840]: 2024-05-12 22:05:54.778366188 +0000 UTC m=+0.018055343 image pull 8252b7944d3b5c188c0ff632647ba6cc4e9a19588ee097b8a3dba27b42779c25 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc 2024-05-12T22:05:55.140 INFO:journalctl@ceph.mds.k.smithi086.stdout:May 12 22:05:54 smithi086 podman[63840]: 2024-05-12 22:05:54.8796478 +0000 UTC m=+0.119353595 container create 5c29b95217929e50456847bf58a8c8ed942f33ca1399c25e306d35cc776f6eaf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-k, RELEASE=squid-f72fecf, GIT_CLEAN=True, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.build-date=20240507, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-05-12T22:05:55.140 INFO:journalctl@ceph.mds.k.smithi086.stdout:May 12 22:05:55 smithi086 podman[63840]: 2024-05-12 22:05:55.04884547 +0000 UTC m=+0.288534613 container init 5c29b95217929e50456847bf58a8c8ed942f33ca1399c25e306d35cc776f6eaf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-k, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-f72fecf, GIT_CLEAN=True, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.build-date=20240507, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-05-12T22:05:55.141 INFO:journalctl@ceph.mds.k.smithi086.stdout:May 12 22:05:55 smithi086 podman[63840]: 2024-05-12 22:05:55.051277423 +0000 UTC m=+0.290966561 container start 5c29b95217929e50456847bf58a8c8ed942f33ca1399c25e306d35cc776f6eaf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-k, org.label-schema.build-date=20240507, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, RELEASE=squid-f72fecf, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.vendor=CentOS) 2024-05-12T22:05:55.141 INFO:journalctl@ceph.mds.k.smithi086.stdout:May 12 22:05:55 smithi086 ceph-mds[63859]: -- 172.21.15.86:0/2029925221 <== mon.2 v2:172.21.15.86:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55cb8efaba40 con 0x55cb8ef7ec00 2024-05-12T22:05:55.141 INFO:journalctl@ceph.mds.k.smithi086.stdout:May 12 22:05:55 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-k[63855]: starting mds.k at 2024-05-12T22:05:55.141 INFO:journalctl@ceph.mds.k.smithi086.stdout:May 12 22:05:55 smithi086 bash[63840]: 5c29b95217929e50456847bf58a8c8ed942f33ca1399c25e306d35cc776f6eaf 2024-05-12T22:05:55.422 INFO:journalctl@ceph.mds.k.smithi086.stdout:May 12 22:05:55 smithi086 systemd[1]: Started Ceph mds.k for 6b92e7b4-10aa-11ef-bc98-c7b262605968. 2024-05-12T22:05:55.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:55 smithi186 ceph-mon[34785]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-12T22:05:55.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:55 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:05:55.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:55 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:05:55.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:55 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:05:55.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:55 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-12T22:05:55.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:55 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-12T22:05:55.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:55 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:05:55.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:55 smithi086 ceph-mon[34297]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-12T22:05:55.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:55 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:05:55.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:55 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:05:55.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:55 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:05:55.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:55 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-12T22:05:55.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:55 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-12T22:05:55.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:55 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:05:55.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:55 smithi028 ceph-mon[29847]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-12T22:05:55.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:55 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:05:55.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:55 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:05:55.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:55 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:05:55.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:55 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-12T22:05:55.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:55 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-12T22:05:55.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:55 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:05:55.823 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-12T22:05:56.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:56.036+0000 7fa944000640 1 -- 172.21.15.28:0/3256399065 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa93c107e90 msgr2=0x7fa93c1082b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:05:56.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:56.036+0000 7fa944000640 1 --2- 172.21.15.28:0/3256399065 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa93c107e90 0x7fa93c1082b0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fa930009900 tx=0x7fa93002a410 comp rx=0 tx=0).stop 2024-05-12T22:05:56.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:56.036+0000 7fa944000640 1 -- 172.21.15.28:0/3256399065 shutdown_connections 2024-05-12T22:05:56.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:56.036+0000 7fa944000640 1 --2- 172.21.15.28:0/3256399065 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa93c1099a0 0x7fa93c1102c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:05:56.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:56.036+0000 7fa944000640 1 --2- 172.21.15.28:0/3256399065 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa93c108fe0 0x7fa93c109460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:05:56.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:56.036+0000 7fa944000640 1 --2- 172.21.15.28:0/3256399065 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa93c107e90 0x7fa93c1082b0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:05:56.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:56.036+0000 7fa944000640 1 -- 172.21.15.28:0/3256399065 >> 172.21.15.28:0/3256399065 conn(0x7fa93c076e50 msgr2=0x7fa93c079270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:05:56.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:56.036+0000 7fa944000640 1 -- 172.21.15.28:0/3256399065 shutdown_connections 2024-05-12T22:05:56.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:56.037+0000 7fa944000640 1 -- 172.21.15.28:0/3256399065 wait complete. 2024-05-12T22:05:56.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:56.038+0000 7fa944000640 1 Processor -- start 2024-05-12T22:05:56.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:56.038+0000 7fa944000640 1 -- start start 2024-05-12T22:05:56.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:56.038+0000 7fa944000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa93c107e90 0x7fa93c19e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:05:56.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:56.038+0000 7fa944000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa93c108fe0 0x7fa93c19e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:05:56.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:56.038+0000 7fa944000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa93c1099a0 0x7fa93c1a5750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:05:56.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:56.038+0000 7fa944000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa93c113b80 con 0x7fa93c1099a0 2024-05-12T22:05:56.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:56.038+0000 7fa944000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fa93c113a00 con 0x7fa93c108fe0 2024-05-12T22:05:56.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:56.038+0000 7fa944000640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa93c113d00 con 0x7fa93c107e90 2024-05-12T22:05:56.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:56.039+0000 7fa942000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa93c1099a0 0x7fa93c1a5750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:05:56.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:56.039+0000 7fa942000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa93c1099a0 0x7fa93c1a5750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49482/0 (socket says 172.21.15.28:49482) 2024-05-12T22:05:56.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:56.039+0000 7fa942000640 1 -- 172.21.15.28:0/3040176911 learned_addr learned my addr 172.21.15.28:0/3040176911 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:05:56.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:56.039+0000 7fa941600640 1 --2- 172.21.15.28:0/3040176911 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa93c107e90 0x7fa93c19e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:05:56.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:56.039+0000 7fa940c00640 1 --2- 172.21.15.28:0/3040176911 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa93c108fe0 0x7fa93c19e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:05:56.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:56.039+0000 7fa942000640 1 -- 172.21.15.28:0/3040176911 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa93c107e90 msgr2=0x7fa93c19e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:05:56.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:56.039+0000 7fa942000640 1 --2- 172.21.15.28:0/3040176911 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa93c107e90 0x7fa93c19e1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:05:56.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:56.039+0000 7fa942000640 1 -- 172.21.15.28:0/3040176911 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa93c108fe0 msgr2=0x7fa93c19e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:05:56.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:56.039+0000 7fa942000640 1 --2- 172.21.15.28:0/3040176911 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa93c108fe0 0x7fa93c19e6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:05:56.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:56.039+0000 7fa942000640 1 -- 172.21.15.28:0/3040176911 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa93c1a5dc0 con 0x7fa93c1099a0 2024-05-12T22:05:56.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:56.039+0000 7fa941600640 1 --2- 172.21.15.28:0/3040176911 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa93c107e90 0x7fa93c19e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:05:56.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:56.039+0000 7fa940c00640 1 --2- 172.21.15.28:0/3040176911 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa93c108fe0 0x7fa93c19e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:05:56.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:56.040+0000 7fa942000640 1 --2- 172.21.15.28:0/3040176911 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa93c1099a0 0x7fa93c1a5750 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fa93400eb10 tx=0x7fa93400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:05:56.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:56.040+0000 7fa93a000640 1 -- 172.21.15.28:0/3040176911 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa934004b20 con 0x7fa93c1099a0 2024-05-12T22:05:56.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:56.040+0000 7fa944000640 1 -- 172.21.15.28:0/3040176911 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa93c1a60b0 con 0x7fa93c1099a0 2024-05-12T22:05:56.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:56.040+0000 7fa93a000640 1 -- 172.21.15.28:0/3040176911 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa934004cc0 con 0x7fa93c1099a0 2024-05-12T22:05:56.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:56.040+0000 7fa93a000640 1 -- 172.21.15.28:0/3040176911 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa934010760 con 0x7fa93c1099a0 2024-05-12T22:05:56.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:56.040+0000 7fa944000640 1 -- 172.21.15.28:0/3040176911 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa93c1a6650 con 0x7fa93c1099a0 2024-05-12T22:05:56.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:56.041+0000 7fa944000640 1 -- 172.21.15.28:0/3040176911 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa93c10f1a0 con 0x7fa93c1099a0 2024-05-12T22:05:56.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:56.044+0000 7fa93a000640 1 -- 172.21.15.28:0/3040176911 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fa934005000 con 0x7fa93c1099a0 2024-05-12T22:05:56.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:56.045+0000 7fa93a000640 1 --2- 172.21.15.28:0/3040176911 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fa9240a2b90 0x7fa9240a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:05:56.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:56.045+0000 7fa93a000640 1 -- 172.21.15.28:0/3040176911 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fa9340cf8f0 con 0x7fa93c1099a0 2024-05-12T22:05:56.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:56.045+0000 7fa941600640 1 --2- 172.21.15.28:0/3040176911 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fa9240a2b90 0x7fa9240a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:05:56.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:56.046+0000 7fa941600640 1 --2- 172.21.15.28:0/3040176911 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fa9240a2b90 0x7fa9240a5050 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fa930031440 tx=0x7fa9300315b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:05:56.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:56.048+0000 7fa93a000640 1 -- 172.21.15.28:0/3040176911 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fa93409cc50 con 0x7fa93c1099a0 2024-05-12T22:05:56.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:56.313+0000 7fa944000640 1 -- 172.21.15.28:0/3040176911 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa93c062430 con 0x7fa93c1099a0 2024-05-12T22:05:56.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:56.314+0000 7fa93a000640 1 -- 172.21.15.28:0/3040176911 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2221 (secure 0 0 0) 0x7fa93409c550 con 0x7fa93c1099a0 2024-05-12T22:05:56.315 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-12T22:05:56.315 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1844141745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1844141745},{"type":"v1","addr":"172.21.15.186:6833","nonce":1844141745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6833/2960763758","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":2960763758},{"type":"v1","addr":"172.21.15.86:6833","nonce":2960763758}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-05-12T22:05:56.315 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 3 2024-05-12T22:05:56.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:56.317+0000 7fa944000640 1 -- 172.21.15.28:0/3040176911 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fa9240a2b90 msgr2=0x7fa9240a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:05:56.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:56.317+0000 7fa944000640 1 --2- 172.21.15.28:0/3040176911 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fa9240a2b90 0x7fa9240a5050 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fa930031440 tx=0x7fa9300315b0 comp rx=0 tx=0).stop 2024-05-12T22:05:56.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:56.317+0000 7fa944000640 1 -- 172.21.15.28:0/3040176911 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa93c1099a0 msgr2=0x7fa93c1a5750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:05:56.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:56.317+0000 7fa944000640 1 --2- 172.21.15.28:0/3040176911 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa93c1099a0 0x7fa93c1a5750 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fa93400eb10 tx=0x7fa93400c2d0 comp rx=0 tx=0).stop 2024-05-12T22:05:56.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:56.317+0000 7fa944000640 1 -- 172.21.15.28:0/3040176911 shutdown_connections 2024-05-12T22:05:56.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:56.317+0000 7fa944000640 1 --2- 172.21.15.28:0/3040176911 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fa9240a2b90 0x7fa9240a5050 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:05:56.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:56.317+0000 7fa944000640 1 --2- 172.21.15.28:0/3040176911 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa93c1099a0 0x7fa93c1a5750 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:05:56.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:56.317+0000 7fa944000640 1 --2- 172.21.15.28:0/3040176911 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa93c108fe0 0x7fa93c19e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:05:56.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:56.317+0000 7fa944000640 1 --2- 172.21.15.28:0/3040176911 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa93c107e90 0x7fa93c19e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:05:56.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:56.317+0000 7fa944000640 1 -- 172.21.15.28:0/3040176911 >> 172.21.15.28:0/3040176911 conn(0x7fa93c076e50 msgr2=0x7fa93c10c370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:05:56.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:56.317+0000 7fa944000640 1 -- 172.21.15.28:0/3040176911 shutdown_connections 2024-05-12T22:05:56.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:56.317+0000 7fa944000640 1 -- 172.21.15.28:0/3040176911 wait complete. 2024-05-12T22:05:56.467 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:56 smithi086 ceph-mon[34297]: Deploying daemon mds.j on smithi028 2024-05-12T22:05:56.467 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:56 smithi086 ceph-mon[34297]: mds.? [v2:172.21.15.86:6832/2960763758,v1:172.21.15.86:6833/2960763758] up:boot 2024-05-12T22:05:56.467 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:56 smithi086 ceph-mon[34297]: fsmap 2 up:standby 2024-05-12T22:05:56.467 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:56 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-12T22:05:56.467 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:56 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/3040176911' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:05:56.619 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:56 smithi028 ceph-mon[29847]: Deploying daemon mds.j on smithi028 2024-05-12T22:05:56.619 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:56 smithi028 ceph-mon[29847]: mds.? [v2:172.21.15.86:6832/2960763758,v1:172.21.15.86:6833/2960763758] up:boot 2024-05-12T22:05:56.619 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:56 smithi028 ceph-mon[29847]: fsmap 2 up:standby 2024-05-12T22:05:56.619 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:56 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-12T22:05:56.619 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:56 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/3040176911' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:05:56.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:56 smithi186 ceph-mon[34785]: Deploying daemon mds.j on smithi028 2024-05-12T22:05:56.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:56 smithi186 ceph-mon[34785]: mds.? [v2:172.21.15.86:6832/2960763758,v1:172.21.15.86:6833/2960763758] up:boot 2024-05-12T22:05:56.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:56 smithi186 ceph-mon[34785]: fsmap 2 up:standby 2024-05-12T22:05:56.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:56 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-12T22:05:56.930 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:56 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/3040176911' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:05:57.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:57 smithi086 ceph-mon[34297]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-12T22:05:57.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:57 smithi028 ceph-mon[29847]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-12T22:05:57.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:57 smithi186 ceph-mon[34785]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-12T22:05:58.340 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-12T22:05:58.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:58.545+0000 7f55b7e00640 1 -- 172.21.15.28:0/2118402219 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55b0103ab0 msgr2=0x7f55b0103ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:05:58.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:58.545+0000 7f55b7e00640 1 --2- 172.21.15.28:0/2118402219 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55b0103ab0 0x7f55b0103ed0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f55a4009900 tx=0x7f55a402a3f0 comp rx=0 tx=0).stop 2024-05-12T22:05:58.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:58.545+0000 7f55b7e00640 1 -- 172.21.15.28:0/2118402219 shutdown_connections 2024-05-12T22:05:58.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:58.545+0000 7f55b7e00640 1 --2- 172.21.15.28:0/2118402219 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f55b0105660 0x7f55b010bef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:05:58.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:58.545+0000 7f55b7e00640 1 --2- 172.21.15.28:0/2118402219 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f55b0104ca0 0x7f55b0105120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:05:58.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:58.545+0000 7f55b7e00640 1 --2- 172.21.15.28:0/2118402219 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55b0103ab0 0x7f55b0103ed0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:05:58.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:58.545+0000 7f55b7e00640 1 -- 172.21.15.28:0/2118402219 >> 172.21.15.28:0/2118402219 conn(0x7f55b00ff020 msgr2=0x7f55b0101460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:05:58.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:58.545+0000 7f55b7e00640 1 -- 172.21.15.28:0/2118402219 shutdown_connections 2024-05-12T22:05:58.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:58.545+0000 7f55b7e00640 1 -- 172.21.15.28:0/2118402219 wait complete. 2024-05-12T22:05:58.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:58.546+0000 7f55b7e00640 1 Processor -- start 2024-05-12T22:05:58.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:58.546+0000 7f55b7e00640 1 -- start start 2024-05-12T22:05:58.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:58.546+0000 7f55b7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55b0103ab0 0x7f55b0199dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:05:58.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:58.546+0000 7f55b7e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f55b0104ca0 0x7f55b019a300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:05:58.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:58.546+0000 7f55b7e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f55b0105660 0x7f55b01a1370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:05:58.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:58.546+0000 7f55b7e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f55b010f750 con 0x7f55b0103ab0 2024-05-12T22:05:58.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:58.546+0000 7f55b7e00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f55b010f5d0 con 0x7f55b0104ca0 2024-05-12T22:05:58.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:58.546+0000 7f55b7e00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f55b010f8d0 con 0x7f55b0105660 2024-05-12T22:05:58.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:58.547+0000 7f55b5400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55b0103ab0 0x7f55b0199dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:05:58.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:58.547+0000 7f55b5400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55b0103ab0 0x7f55b0199dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49498/0 (socket says 172.21.15.28:49498) 2024-05-12T22:05:58.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:58.547+0000 7f55b5400640 1 -- 172.21.15.28:0/4111378480 learned_addr learned my addr 172.21.15.28:0/4111378480 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:05:58.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:58.547+0000 7f55b4a00640 1 --2- 172.21.15.28:0/4111378480 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f55b0104ca0 0x7f55b019a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:05:58.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:58.547+0000 7f55b5e00640 1 --2- 172.21.15.28:0/4111378480 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f55b0105660 0x7f55b01a1370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:05:58.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:58.547+0000 7f55b5400640 1 -- 172.21.15.28:0/4111378480 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f55b0105660 msgr2=0x7f55b01a1370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:05:58.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:58.547+0000 7f55b5400640 1 --2- 172.21.15.28:0/4111378480 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f55b0105660 0x7f55b01a1370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:05:58.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:58.547+0000 7f55b5400640 1 -- 172.21.15.28:0/4111378480 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f55b0104ca0 msgr2=0x7f55b019a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:05:58.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:58.547+0000 7f55b5400640 1 --2- 172.21.15.28:0/4111378480 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f55b0104ca0 0x7f55b019a300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:05:58.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:58.547+0000 7f55b5400640 1 -- 172.21.15.28:0/4111378480 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55b01a19e0 con 0x7f55b0103ab0 2024-05-12T22:05:58.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:58.547+0000 7f55b5e00640 1 --2- 172.21.15.28:0/4111378480 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f55b0105660 0x7f55b01a1370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:05:58.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:58.548+0000 7f55b5400640 1 --2- 172.21.15.28:0/4111378480 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55b0103ab0 0x7f55b0199dc0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f55a4031440 tx=0x7f55a4032ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:05:58.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:58.548+0000 7f55ae000640 1 -- 172.21.15.28:0/4111378480 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55a403f030 con 0x7f55b0103ab0 2024-05-12T22:05:58.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:58.548+0000 7f55ae000640 1 -- 172.21.15.28:0/4111378480 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f55a4003ce0 con 0x7f55b0103ab0 2024-05-12T22:05:58.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:58.548+0000 7f55ae000640 1 -- 172.21.15.28:0/4111378480 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55a4035070 con 0x7f55b0103ab0 2024-05-12T22:05:58.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:58.548+0000 7f55b7e00640 1 -- 172.21.15.28:0/4111378480 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55b01a1c70 con 0x7f55b0103ab0 2024-05-12T22:05:58.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:58.548+0000 7f55b7e00640 1 -- 172.21.15.28:0/4111378480 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55b01a2110 con 0x7f55b0103ab0 2024-05-12T22:05:58.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:58.550+0000 7f55b7e00640 1 -- 172.21.15.28:0/4111378480 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f55b010add0 con 0x7f55b0103ab0 2024-05-12T22:05:58.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:58.553+0000 7f55ae000640 1 -- 172.21.15.28:0/4111378480 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f55a4042030 con 0x7f55b0103ab0 2024-05-12T22:05:58.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:58.554+0000 7f55ae000640 1 --2- 172.21.15.28:0/4111378480 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f55980a2c10 0x7f55980a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:05:58.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:58.555+0000 7f55ae000640 1 -- 172.21.15.28:0/4111378480 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f55a40ef1d0 con 0x7f55b0103ab0 2024-05-12T22:05:58.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:58.555+0000 7f55b4a00640 1 --2- 172.21.15.28:0/4111378480 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f55980a2c10 0x7f55980a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:05:58.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:58.556+0000 7f55b4a00640 1 --2- 172.21.15.28:0/4111378480 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f55980a2c10 0x7f55980a50d0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f55b019b2d0 tx=0x7f559c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:05:58.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:58.559+0000 7f55ae000640 1 -- 172.21.15.28:0/4111378480 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f55a40bc530 con 0x7f55b0103ab0 2024-05-12T22:05:58.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:58.815+0000 7f55b7e00640 1 -- 172.21.15.28:0/4111378480 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f55b0062530 con 0x7f55b0103ab0 2024-05-12T22:05:58.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:58.816+0000 7f55ae000640 1 -- 172.21.15.28:0/4111378480 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2221 (secure 0 0 0) 0x7f55a40bbe30 con 0x7f55b0103ab0 2024-05-12T22:05:58.816 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-12T22:05:58.816 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1844141745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1844141745},{"type":"v1","addr":"172.21.15.186:6833","nonce":1844141745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6833/2960763758","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":2960763758},{"type":"v1","addr":"172.21.15.86:6833","nonce":2960763758}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-05-12T22:05:58.817 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 3 2024-05-12T22:05:58.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:58.819+0000 7f55b7e00640 1 -- 172.21.15.28:0/4111378480 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f55980a2c10 msgr2=0x7f55980a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:05:58.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:58.819+0000 7f55b7e00640 1 --2- 172.21.15.28:0/4111378480 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f55980a2c10 0x7f55980a50d0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f55b019b2d0 tx=0x7f559c00b410 comp rx=0 tx=0).stop 2024-05-12T22:05:58.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:58.819+0000 7f55b7e00640 1 -- 172.21.15.28:0/4111378480 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55b0103ab0 msgr2=0x7f55b0199dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:05:58.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:58.819+0000 7f55b7e00640 1 --2- 172.21.15.28:0/4111378480 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55b0103ab0 0x7f55b0199dc0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f55a4031440 tx=0x7f55a4032ce0 comp rx=0 tx=0).stop 2024-05-12T22:05:58.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:58.819+0000 7f55b7e00640 1 -- 172.21.15.28:0/4111378480 shutdown_connections 2024-05-12T22:05:58.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:58.819+0000 7f55b7e00640 1 --2- 172.21.15.28:0/4111378480 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f55980a2c10 0x7f55980a50d0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:05:58.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:58.819+0000 7f55b7e00640 1 --2- 172.21.15.28:0/4111378480 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f55b0105660 0x7f55b01a1370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:05:58.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:58.819+0000 7f55b7e00640 1 --2- 172.21.15.28:0/4111378480 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f55b0104ca0 0x7f55b019a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:05:58.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:58.819+0000 7f55b7e00640 1 --2- 172.21.15.28:0/4111378480 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55b0103ab0 0x7f55b0199dc0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:05:58.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:58.819+0000 7f55b7e00640 1 -- 172.21.15.28:0/4111378480 >> 172.21.15.28:0/4111378480 conn(0x7f55b00ff020 msgr2=0x7f55b0107fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:05:58.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:58.819+0000 7f55b7e00640 1 -- 172.21.15.28:0/4111378480 shutdown_connections 2024-05-12T22:05:58.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:05:58.819+0000 7f55b7e00640 1 -- 172.21.15.28:0/4111378480 wait complete. 2024-05-12T22:05:59.445 INFO:journalctl@ceph.mds.j.smithi028.stdout:May 12 22:05:59 smithi028 systemd[1]: Started Ceph mds.j for 6b92e7b4-10aa-11ef-bc98-c7b262605968. 2024-05-12T22:05:59.731 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:59 smithi186 ceph-mon[34785]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-12T22:05:59.732 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:59 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/4111378480' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:05:59.732 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:59 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:05:59.732 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:59 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:05:59.732 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:59 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:05:59.732 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:59 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-12T22:05:59.732 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:59 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-12T22:05:59.732 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:05:59 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:05:59.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:59 smithi086 ceph-mon[34297]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-12T22:05:59.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:59 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/4111378480' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:05:59.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:59 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:05:59.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:59 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:05:59.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:59 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:05:59.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:59 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-12T22:05:59.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:59 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-12T22:05:59.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:05:59 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:05:59.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:59 smithi028 ceph-mon[29847]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-12T22:05:59.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:59 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/4111378480' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:05:59.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:59 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:05:59.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:59 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:05:59.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:59 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:05:59.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:59 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-12T22:05:59.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:59 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-12T22:05:59.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:05:59 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:06:00.655 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:00 smithi028 ceph-mon[29847]: Deploying daemon mds.d on smithi028 2024-05-12T22:06:00.655 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:00 smithi028 ceph-mon[29847]: mds.? [v2:172.21.15.28:6834/4131134219,v1:172.21.15.28:6835/4131134219] up:boot 2024-05-12T22:06:00.655 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:00 smithi028 ceph-mon[29847]: fsmap 3 up:standby 2024-05-12T22:06:00.655 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:00 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-12T22:06:00.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:00 smithi086 ceph-mon[34297]: Deploying daemon mds.d on smithi028 2024-05-12T22:06:00.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:00 smithi086 ceph-mon[34297]: mds.? [v2:172.21.15.28:6834/4131134219,v1:172.21.15.28:6835/4131134219] up:boot 2024-05-12T22:06:00.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:00 smithi086 ceph-mon[34297]: fsmap 3 up:standby 2024-05-12T22:06:00.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:00 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-12T22:06:00.840 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-12T22:06:00.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:00 smithi186 ceph-mon[34785]: Deploying daemon mds.d on smithi028 2024-05-12T22:06:00.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:00 smithi186 ceph-mon[34785]: mds.? [v2:172.21.15.28:6834/4131134219,v1:172.21.15.28:6835/4131134219] up:boot 2024-05-12T22:06:00.930 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:00 smithi186 ceph-mon[34785]: fsmap 3 up:standby 2024-05-12T22:06:00.930 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:00 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-12T22:06:01.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:01.066+0000 7f4d3ca00640 1 -- 172.21.15.28:0/3304513700 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d38074210 msgr2=0x7f4d38074670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:01.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:01.066+0000 7f4d3ca00640 1 --2- 172.21.15.28:0/3304513700 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d38074210 0x7f4d38074670 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f4d24009900 tx=0x7f4d2402a480 comp rx=0 tx=0).stop 2024-05-12T22:06:01.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:01.067+0000 7f4d3ca00640 1 -- 172.21.15.28:0/3304513700 shutdown_connections 2024-05-12T22:06:01.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:01.067+0000 7f4d3ca00640 1 --2- 172.21.15.28:0/3304513700 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4d38109580 0x7f4d3810ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:01.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:01.067+0000 7f4d3ca00640 1 --2- 172.21.15.28:0/3304513700 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d38074210 0x7f4d38074670 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:01.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:01.067+0000 7f4d3ca00640 1 --2- 172.21.15.28:0/3304513700 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4d380757f0 0x7f4d38073c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:01.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:01.067+0000 7f4d3ca00640 1 -- 172.21.15.28:0/3304513700 >> 172.21.15.28:0/3304513700 conn(0x7f4d380fcc40 msgr2=0x7f4d380ff080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:06:01.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:01.067+0000 7f4d3ca00640 1 -- 172.21.15.28:0/3304513700 shutdown_connections 2024-05-12T22:06:01.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:01.067+0000 7f4d3ca00640 1 -- 172.21.15.28:0/3304513700 wait complete. 2024-05-12T22:06:01.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:01.068+0000 7f4d3ca00640 1 Processor -- start 2024-05-12T22:06:01.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:01.068+0000 7f4d3ca00640 1 -- start start 2024-05-12T22:06:01.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:01.069+0000 7f4d3ca00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d38074210 0x7f4d38199bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:01.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:01.069+0000 7f4d3ca00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4d380757f0 0x7f4d3819a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:01.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:01.069+0000 7f4d3ca00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4d38109580 0x7f4d381a1180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:01.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:01.069+0000 7f4d3ca00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4d3810f580 con 0x7f4d38074210 2024-05-12T22:06:01.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:01.069+0000 7f4d3ca00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f4d3810f400 con 0x7f4d38109580 2024-05-12T22:06:01.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:01.069+0000 7f4d3ca00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f4d3810f700 con 0x7f4d380757f0 2024-05-12T22:06:01.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:01.069+0000 7f4d36800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d38074210 0x7f4d38199bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:01.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:01.069+0000 7f4d36800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d38074210 0x7f4d38199bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49536/0 (socket says 172.21.15.28:49536) 2024-05-12T22:06:01.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:01.069+0000 7f4d36800640 1 -- 172.21.15.28:0/4169465770 learned_addr learned my addr 172.21.15.28:0/4169465770 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:06:01.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:01.069+0000 7f4d35e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4d380757f0 0x7f4d3819a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:01.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:01.069+0000 7f4d37200640 1 --2- 172.21.15.28:0/4169465770 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4d38109580 0x7f4d381a1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:01.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:01.070+0000 7f4d35e00640 1 -- 172.21.15.28:0/4169465770 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4d38109580 msgr2=0x7f4d381a1180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:01.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:01.070+0000 7f4d35e00640 1 --2- 172.21.15.28:0/4169465770 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4d38109580 0x7f4d381a1180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:01.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:01.070+0000 7f4d35e00640 1 -- 172.21.15.28:0/4169465770 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d38074210 msgr2=0x7f4d38199bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:01.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:01.070+0000 7f4d35e00640 1 --2- 172.21.15.28:0/4169465770 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d38074210 0x7f4d38199bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:01.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:01.070+0000 7f4d35e00640 1 -- 172.21.15.28:0/4169465770 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d381a17f0 con 0x7f4d380757f0 2024-05-12T22:06:01.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:01.070+0000 7f4d37200640 1 --2- 172.21.15.28:0/4169465770 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4d38109580 0x7f4d381a1180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:06:01.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:01.070+0000 7f4d36800640 1 --2- 172.21.15.28:0/4169465770 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d38074210 0x7f4d38199bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:06:01.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:01.070+0000 7f4d35e00640 1 --2- 172.21.15.28:0/4169465770 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4d380757f0 0x7f4d3819a110 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f4d2402a9a0 tx=0x7f4d24032ce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:06:01.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:01.071+0000 7f4d2f400640 1 -- 172.21.15.28:0/4169465770 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d2403f030 con 0x7f4d380757f0 2024-05-12T22:06:01.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:01.071+0000 7f4d2f400640 1 -- 172.21.15.28:0/4169465770 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4d24003ce0 con 0x7f4d380757f0 2024-05-12T22:06:01.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:01.071+0000 7f4d3ca00640 1 -- 172.21.15.28:0/4169465770 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d381a1a80 con 0x7f4d380757f0 2024-05-12T22:06:01.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:01.071+0000 7f4d2f400640 1 -- 172.21.15.28:0/4169465770 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d24035070 con 0x7f4d380757f0 2024-05-12T22:06:01.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:01.071+0000 7f4d3ca00640 1 -- 172.21.15.28:0/4169465770 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d381a1f40 con 0x7f4d380757f0 2024-05-12T22:06:01.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:01.072+0000 7f4d3ca00640 1 -- 172.21.15.28:0/4169465770 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d38108340 con 0x7f4d380757f0 2024-05-12T22:06:01.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:01.074+0000 7f4d2f400640 1 -- 172.21.15.28:0/4169465770 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f4d24042030 con 0x7f4d380757f0 2024-05-12T22:06:01.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:01.076+0000 7f4d2f400640 1 --2- 172.21.15.28:0/4169465770 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f4d1c0a2b90 0x7f4d1c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:01.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:01.076+0000 7f4d2f400640 1 -- 172.21.15.28:0/4169465770 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f4d240ef620 con 0x7f4d380757f0 2024-05-12T22:06:01.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:01.076+0000 7f4d36800640 1 --2- 172.21.15.28:0/4169465770 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f4d1c0a2b90 0x7f4d1c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:01.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:01.077+0000 7f4d36800640 1 --2- 172.21.15.28:0/4169465770 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f4d1c0a2b90 0x7f4d1c0a5050 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f4d28011550 tx=0x7f4d28005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:06:01.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:01.078+0000 7f4d2f400640 1 -- 172.21.15.28:0/4169465770 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f4d240bc980 con 0x7f4d380757f0 2024-05-12T22:06:01.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:01.326+0000 7f4d3ca00640 1 -- 172.21.15.28:0/4169465770 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4d3802d1c0 con 0x7f4d380757f0 2024-05-12T22:06:01.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:01.327+0000 7f4d2f400640 1 -- 172.21.15.28:0/4169465770 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) ==== 93+0+3046 (secure 0 0 0) 0x7f4d240bc280 con 0x7f4d380757f0 2024-05-12T22:06:01.328 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-12T22:06:01.328 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1844141745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1844141745},{"type":"v1","addr":"172.21.15.186:6833","nonce":1844141745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6833/2960763758","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":2960763758},{"type":"v1","addr":"172.21.15.86:6833","nonce":2960763758}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24470,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/4131134219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":4131134219},{"type":"v1","addr":"172.21.15.28:6835","nonce":4131134219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4}],"filesystems":[]} 2024-05-12T22:06:01.328 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 4 2024-05-12T22:06:01.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:01.329+0000 7f4d3ca00640 1 -- 172.21.15.28:0/4169465770 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f4d1c0a2b90 msgr2=0x7f4d1c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:01.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:01.329+0000 7f4d3ca00640 1 --2- 172.21.15.28:0/4169465770 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f4d1c0a2b90 0x7f4d1c0a5050 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f4d28011550 tx=0x7f4d28005f90 comp rx=0 tx=0).stop 2024-05-12T22:06:01.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:01.329+0000 7f4d3ca00640 1 -- 172.21.15.28:0/4169465770 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4d380757f0 msgr2=0x7f4d3819a110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:01.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:01.329+0000 7f4d3ca00640 1 --2- 172.21.15.28:0/4169465770 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4d380757f0 0x7f4d3819a110 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f4d2402a9a0 tx=0x7f4d24032ce0 comp rx=0 tx=0).stop 2024-05-12T22:06:01.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:01.329+0000 7f4d3ca00640 1 -- 172.21.15.28:0/4169465770 shutdown_connections 2024-05-12T22:06:01.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:01.329+0000 7f4d3ca00640 1 --2- 172.21.15.28:0/4169465770 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f4d1c0a2b90 0x7f4d1c0a5050 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:01.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:01.329+0000 7f4d3ca00640 1 --2- 172.21.15.28:0/4169465770 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4d38109580 0x7f4d381a1180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:01.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:01.329+0000 7f4d3ca00640 1 --2- 172.21.15.28:0/4169465770 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4d380757f0 0x7f4d3819a110 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:01.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:01.329+0000 7f4d3ca00640 1 --2- 172.21.15.28:0/4169465770 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d38074210 0x7f4d38199bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:01.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:01.329+0000 7f4d3ca00640 1 -- 172.21.15.28:0/4169465770 >> 172.21.15.28:0/4169465770 conn(0x7f4d380fcc40 msgr2=0x7f4d38105510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:06:01.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:01.330+0000 7f4d3ca00640 1 -- 172.21.15.28:0/4169465770 shutdown_connections 2024-05-12T22:06:01.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:01.330+0000 7f4d3ca00640 1 -- 172.21.15.28:0/4169465770 wait complete. 2024-05-12T22:06:01.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:01 smithi086 ceph-mon[34297]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-12T22:06:01.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:01 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/4169465770' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:06:01.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:01 smithi028 ceph-mon[29847]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-12T22:06:01.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:01 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/4169465770' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:06:01.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:01 smithi186 ceph-mon[34785]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-12T22:06:01.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:01 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/4169465770' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:06:02.667 INFO:journalctl@ceph.mds.d.smithi028.stdout:May 12 22:06:02 smithi028 systemd[1]: Starting Ceph mds.d for 6b92e7b4-10aa-11ef-bc98-c7b262605968... 2024-05-12T22:06:02.946 INFO:journalctl@ceph.mds.d.smithi028.stdout:May 12 22:06:02 smithi028 podman[67143]: 2024-05-12 22:06:02.570932401 +0000 UTC m=+0.037756145 image pull 8252b7944d3b5c188c0ff632647ba6cc4e9a19588ee097b8a3dba27b42779c25 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc 2024-05-12T22:06:02.947 INFO:journalctl@ceph.mds.d.smithi028.stdout:May 12 22:06:02 smithi028 podman[67143]: 2024-05-12 22:06:02.667019475 +0000 UTC m=+0.133843323 container create a971d49e34e43d0c0b3bfaff3d4abeaa13df5ea39822fda4351d28b45665d568 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-d, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240507, GIT_BRANCH=HEAD, RELEASE=squid-f72fecf, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True) 2024-05-12T22:06:02.947 INFO:journalctl@ceph.mds.d.smithi028.stdout:May 12 22:06:02 smithi028 podman[67143]: 2024-05-12 22:06:02.945663947 +0000 UTC m=+0.412487735 container init a971d49e34e43d0c0b3bfaff3d4abeaa13df5ea39822fda4351d28b45665d568 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-d, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=squid-f72fecf, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240507, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.schema-version=1.0, ceph=True) 2024-05-12T22:06:03.285 INFO:journalctl@ceph.mds.d.smithi028.stdout:May 12 22:06:02 smithi028 podman[67143]: 2024-05-12 22:06:02.957533641 +0000 UTC m=+0.424357403 container start a971d49e34e43d0c0b3bfaff3d4abeaa13df5ea39822fda4351d28b45665d568 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-d, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, RELEASE=squid-f72fecf, org.label-schema.build-date=20240507, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-12T22:06:03.286 INFO:journalctl@ceph.mds.d.smithi028.stdout:May 12 22:06:03 smithi028 ceph-mds[67162]: -- 172.21.15.28:0/1203291669 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x56206d57a4e0 con 0x56206d551000 2024-05-12T22:06:03.286 INFO:journalctl@ceph.mds.d.smithi028.stdout:May 12 22:06:03 smithi028 ceph-mds[67162]: -- 172.21.15.28:0/1203291669 <== mon.0 v2:172.21.15.28:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56206d57da40 con 0x56206d551000 2024-05-12T22:06:03.286 INFO:journalctl@ceph.mds.d.smithi028.stdout:May 12 22:06:03 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-d[67158]: starting mds.d at 2024-05-12T22:06:03.286 INFO:journalctl@ceph.mds.d.smithi028.stdout:May 12 22:06:03 smithi028 bash[67143]: a971d49e34e43d0c0b3bfaff3d4abeaa13df5ea39822fda4351d28b45665d568 2024-05-12T22:06:03.286 INFO:journalctl@ceph.mds.d.smithi028.stdout:May 12 22:06:03 smithi028 systemd[1]: Started Ceph mds.d for 6b92e7b4-10aa-11ef-bc98-c7b262605968. 2024-05-12T22:06:03.351 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-12T22:06:03.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:03.581+0000 7ff1cbe00640 1 -- 172.21.15.28:0/3008977020 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff1cc1038b0 msgr2=0x7ff1cc103cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:03.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:03.581+0000 7ff1cbe00640 1 --2- 172.21.15.28:0/3008977020 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff1cc1038b0 0x7ff1cc103cd0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff1bc009900 tx=0x7ff1bc02a410 comp rx=0 tx=0).stop 2024-05-12T22:06:03.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:03.581+0000 7ff1cbe00640 1 -- 172.21.15.28:0/3008977020 shutdown_connections 2024-05-12T22:06:03.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:03.581+0000 7ff1cbe00640 1 --2- 172.21.15.28:0/3008977020 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1cc105460 0x7ff1cc10bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:03.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:03.581+0000 7ff1cbe00640 1 --2- 172.21.15.28:0/3008977020 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1cc104aa0 0x7ff1cc104f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:03.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:03.581+0000 7ff1cbe00640 1 --2- 172.21.15.28:0/3008977020 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff1cc1038b0 0x7ff1cc103cd0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:03.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:03.581+0000 7ff1cbe00640 1 -- 172.21.15.28:0/3008977020 >> 172.21.15.28:0/3008977020 conn(0x7ff1cc0fee40 msgr2=0x7ff1cc101260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:06:03.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:03.582+0000 7ff1cbe00640 1 -- 172.21.15.28:0/3008977020 shutdown_connections 2024-05-12T22:06:03.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:03.582+0000 7ff1cbe00640 1 -- 172.21.15.28:0/3008977020 wait complete. 2024-05-12T22:06:03.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:03.583+0000 7ff1cbe00640 1 Processor -- start 2024-05-12T22:06:03.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:03.583+0000 7ff1cbe00640 1 -- start start 2024-05-12T22:06:03.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:03.583+0000 7ff1cbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1cc1038b0 0x7ff1cc199b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:03.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:03.583+0000 7ff1cbe00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1cc104aa0 0x7ff1cc19a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:03.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:03.583+0000 7ff1cbe00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff1cc105460 0x7ff1cc1a1140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:03.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:03.583+0000 7ff1cbe00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff1cc10f550 con 0x7ff1cc1038b0 2024-05-12T22:06:03.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:03.583+0000 7ff1cbe00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7ff1cc10f3d0 con 0x7ff1cc105460 2024-05-12T22:06:03.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:03.583+0000 7ff1cbe00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff1cc10f6d0 con 0x7ff1cc104aa0 2024-05-12T22:06:03.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:03.583+0000 7ff1cb400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff1cc105460 0x7ff1cc1a1140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:03.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:03.583+0000 7ff1caa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1cc1038b0 0x7ff1cc199b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:03.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:03.583+0000 7ff1cb400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff1cc105460 0x7ff1cc1a1140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:34232/0 (socket says 172.21.15.28:34232) 2024-05-12T22:06:03.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:03.583+0000 7ff1cb400640 1 -- 172.21.15.28:0/827454840 learned_addr learned my addr 172.21.15.28:0/827454840 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:06:03.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:03.583+0000 7ff1ca000640 1 --2- 172.21.15.28:0/827454840 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1cc104aa0 0x7ff1cc19a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:03.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:03.584+0000 7ff1cb400640 1 -- 172.21.15.28:0/827454840 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1cc104aa0 msgr2=0x7ff1cc19a0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:03.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:03.584+0000 7ff1cb400640 1 --2- 172.21.15.28:0/827454840 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1cc104aa0 0x7ff1cc19a0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:03.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:03.584+0000 7ff1cb400640 1 -- 172.21.15.28:0/827454840 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1cc1038b0 msgr2=0x7ff1cc199b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:03.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:03.584+0000 7ff1cb400640 1 --2- 172.21.15.28:0/827454840 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1cc1038b0 0x7ff1cc199b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:03.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:03.584+0000 7ff1cb400640 1 -- 172.21.15.28:0/827454840 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1cc1a17b0 con 0x7ff1cc105460 2024-05-12T22:06:03.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:03.584+0000 7ff1cb400640 1 --2- 172.21.15.28:0/827454840 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff1cc105460 0x7ff1cc1a1140 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff1c400eaa0 tx=0x7ff1c400ef60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:06:03.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:03.584+0000 7ff1c3400640 1 -- 172.21.15.28:0/827454840 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1c400cdb0 con 0x7ff1cc105460 2024-05-12T22:06:03.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:03.584+0000 7ff1c3400640 1 -- 172.21.15.28:0/827454840 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff1c4005020 con 0x7ff1cc105460 2024-05-12T22:06:03.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:03.584+0000 7ff1c3400640 1 -- 172.21.15.28:0/827454840 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1c40116b0 con 0x7ff1cc105460 2024-05-12T22:06:03.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:03.584+0000 7ff1cbe00640 1 -- 172.21.15.28:0/827454840 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1cc1a1aa0 con 0x7ff1cc105460 2024-05-12T22:06:03.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:03.585+0000 7ff1cbe00640 1 -- 172.21.15.28:0/827454840 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1cc1a1ff0 con 0x7ff1cc105460 2024-05-12T22:06:03.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:03.586+0000 7ff1cbe00640 1 -- 172.21.15.28:0/827454840 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1cc10abd0 con 0x7ff1cc105460 2024-05-12T22:06:03.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:03.587+0000 7ff1c3400640 1 -- 172.21.15.28:0/827454840 <== mon.2 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7ff1c4005560 con 0x7ff1cc105460 2024-05-12T22:06:03.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:03.589+0000 7ff1c3400640 1 --2- 172.21.15.28:0/827454840 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7ff1b00a2c10 0x7ff1b00a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:03.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:03.589+0000 7ff1c3400640 1 -- 172.21.15.28:0/827454840 <== mon.2 v2:172.21.15.86:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7ff1c40cedb0 con 0x7ff1cc105460 2024-05-12T22:06:03.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:03.589+0000 7ff1caa00640 1 --2- 172.21.15.28:0/827454840 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7ff1b00a2c10 0x7ff1b00a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:03.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:03.590+0000 7ff1caa00640 1 --2- 172.21.15.28:0/827454840 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7ff1b00a2c10 0x7ff1b00a50d0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7ff1bc02aa00 tx=0x7ff1bc0315b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:06:03.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:03.593+0000 7ff1c3400640 1 -- 172.21.15.28:0/827454840 <== mon.2 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7ff1c40d3070 con 0x7ff1cc105460 2024-05-12T22:06:03.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:03 smithi086 ceph-mon[34297]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-12T22:06:03.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:03 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:03.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:03 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:03.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:03 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:03.743 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:03 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-12T22:06:03.743 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:03 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-12T22:06:03.743 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:03 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:06:03.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:03 smithi028 ceph-mon[29847]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-12T22:06:03.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:03 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:03.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:03 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:03.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:03 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:03.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:03 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-12T22:06:03.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:03 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-12T22:06:03.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:03 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:06:03.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:03.839+0000 7ff1cbe00640 1 -- 172.21.15.28:0/827454840 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff1cc02d1c0 con 0x7ff1cc105460 2024-05-12T22:06:03.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:03.840+0000 7ff1c3400640 1 -- 172.21.15.28:0/827454840 <== mon.2 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+3871 (secure 0 0 0) 0x7ff1c4016030 con 0x7ff1cc105460 2024-05-12T22:06:03.840 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-12T22:06:03.840 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1844141745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1844141745},{"type":"v1","addr":"172.21.15.186:6833","nonce":1844141745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6833/2960763758","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":2960763758},{"type":"v1","addr":"172.21.15.86:6833","nonce":2960763758}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24470,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/4131134219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":4131134219},{"type":"v1","addr":"172.21.15.28:6835","nonce":4131134219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24482,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2938260704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2938260704},{"type":"v1","addr":"172.21.15.28:6837","nonce":2938260704}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[]} 2024-05-12T22:06:03.841 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 5 2024-05-12T22:06:03.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:03.843+0000 7ff1cbe00640 1 -- 172.21.15.28:0/827454840 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7ff1b00a2c10 msgr2=0x7ff1b00a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:03.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:03.843+0000 7ff1cbe00640 1 --2- 172.21.15.28:0/827454840 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7ff1b00a2c10 0x7ff1b00a50d0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7ff1bc02aa00 tx=0x7ff1bc0315b0 comp rx=0 tx=0).stop 2024-05-12T22:06:03.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:03.843+0000 7ff1cbe00640 1 -- 172.21.15.28:0/827454840 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff1cc105460 msgr2=0x7ff1cc1a1140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:03.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:03.843+0000 7ff1cbe00640 1 --2- 172.21.15.28:0/827454840 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff1cc105460 0x7ff1cc1a1140 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff1c400eaa0 tx=0x7ff1c400ef60 comp rx=0 tx=0).stop 2024-05-12T22:06:03.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:03.843+0000 7ff1cbe00640 1 -- 172.21.15.28:0/827454840 shutdown_connections 2024-05-12T22:06:03.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:03.843+0000 7ff1cbe00640 1 --2- 172.21.15.28:0/827454840 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7ff1b00a2c10 0x7ff1b00a50d0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:03.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:03.843+0000 7ff1cbe00640 1 --2- 172.21.15.28:0/827454840 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff1cc105460 0x7ff1cc1a1140 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:03.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:03.843+0000 7ff1cbe00640 1 --2- 172.21.15.28:0/827454840 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1cc104aa0 0x7ff1cc19a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:03.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:03.843+0000 7ff1cbe00640 1 --2- 172.21.15.28:0/827454840 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1cc1038b0 0x7ff1cc199b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:03.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:03.843+0000 7ff1cbe00640 1 -- 172.21.15.28:0/827454840 >> 172.21.15.28:0/827454840 conn(0x7ff1cc0fee40 msgr2=0x7ff1cc107da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:06:03.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:03.843+0000 7ff1cbe00640 1 -- 172.21.15.28:0/827454840 shutdown_connections 2024-05-12T22:06:03.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:03.843+0000 7ff1cbe00640 1 -- 172.21.15.28:0/827454840 wait complete. 2024-05-12T22:06:03.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:03 smithi186 ceph-mon[34785]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-12T22:06:03.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:03 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:03.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:03 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:03.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:03 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:03.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:03 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-12T22:06:03.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:03 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-12T22:06:03.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:03 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:06:04.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:04 smithi086 ceph-mon[34297]: Deploying daemon mds.e on smithi086 2024-05-12T22:06:04.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:04 smithi086 ceph-mon[34297]: mds.? [v2:172.21.15.28:6836/2938260704,v1:172.21.15.28:6837/2938260704] up:boot 2024-05-12T22:06:04.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:04 smithi086 ceph-mon[34297]: fsmap 4 up:standby 2024-05-12T22:06:04.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:04 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-12T22:06:04.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:04 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/827454840' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:06:04.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:04 smithi028 ceph-mon[29847]: Deploying daemon mds.e on smithi086 2024-05-12T22:06:04.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:04 smithi028 ceph-mon[29847]: mds.? [v2:172.21.15.28:6836/2938260704,v1:172.21.15.28:6837/2938260704] up:boot 2024-05-12T22:06:04.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:04 smithi028 ceph-mon[29847]: fsmap 4 up:standby 2024-05-12T22:06:04.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:04 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-12T22:06:04.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:04 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/827454840' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:06:04.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:04 smithi186 ceph-mon[34785]: Deploying daemon mds.e on smithi086 2024-05-12T22:06:04.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:04 smithi186 ceph-mon[34785]: mds.? [v2:172.21.15.28:6836/2938260704,v1:172.21.15.28:6837/2938260704] up:boot 2024-05-12T22:06:04.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:04 smithi186 ceph-mon[34785]: fsmap 4 up:standby 2024-05-12T22:06:04.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:04 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-12T22:06:04.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:04 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/827454840' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:06:05.471 INFO:journalctl@ceph.mds.e.smithi086.stdout:May 12 22:06:05 smithi086 systemd[1]: Starting Ceph mds.e for 6b92e7b4-10aa-11ef-bc98-c7b262605968... 2024-05-12T22:06:05.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:05 smithi086 ceph-mon[34297]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-12T22:06:05.742 INFO:journalctl@ceph.mds.e.smithi086.stdout:May 12 22:06:05 smithi086 podman[64159]: 2024-05-12 22:06:05.467111925 +0000 UTC m=+0.018027484 image pull 8252b7944d3b5c188c0ff632647ba6cc4e9a19588ee097b8a3dba27b42779c25 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc 2024-05-12T22:06:05.742 INFO:journalctl@ceph.mds.e.smithi086.stdout:May 12 22:06:05 smithi086 podman[64159]: 2024-05-12 22:06:05.598632729 +0000 UTC m=+0.149548324 container create d1d854c7ed0bdecf637489fdc37ec7118d6d395acf840a27ea55c05a62ca1305 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-e, RELEASE=squid-f72fecf, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240507, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e) 2024-05-12T22:06:05.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:05 smithi028 ceph-mon[29847]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-12T22:06:05.865 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-12T22:06:05.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:05 smithi186 ceph-mon[34785]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-12T22:06:06.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:06.092+0000 7f3070a00640 1 -- 172.21.15.28:0/2822868739 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f306c0761f0 msgr2=0x7f306c076610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:06.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:06.092+0000 7f3070a00640 1 --2- 172.21.15.28:0/2822868739 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f306c0761f0 0x7f306c076610 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f305c009900 tx=0x7f305c02a410 comp rx=0 tx=0).stop 2024-05-12T22:06:06.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:06.093+0000 7f3070a00640 1 -- 172.21.15.28:0/2822868739 shutdown_connections 2024-05-12T22:06:06.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:06.093+0000 7f3070a00640 1 --2- 172.21.15.28:0/2822868739 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f306c077dd0 0x7f306c078670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:06.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:06.093+0000 7f3070a00640 1 --2- 172.21.15.28:0/2822868739 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f306c077430 0x7f306c077890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:06.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:06.093+0000 7f3070a00640 1 --2- 172.21.15.28:0/2822868739 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f306c0761f0 0x7f306c076610 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:06.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:06.093+0000 7f3070a00640 1 -- 172.21.15.28:0/2822868739 >> 172.21.15.28:0/2822868739 conn(0x7f306c0ff020 msgr2=0x7f306c101460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:06:06.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:06.093+0000 7f3070a00640 1 -- 172.21.15.28:0/2822868739 shutdown_connections 2024-05-12T22:06:06.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:06.093+0000 7f3070a00640 1 -- 172.21.15.28:0/2822868739 wait complete. 2024-05-12T22:06:06.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:06.094+0000 7f3070a00640 1 Processor -- start 2024-05-12T22:06:06.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:06.094+0000 7f3070a00640 1 -- start start 2024-05-12T22:06:06.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:06.094+0000 7f3070a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f306c0761f0 0x7f306c19e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:06.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:06.094+0000 7f3070a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f306c077430 0x7f306c19e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:06.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:06.094+0000 7f3070a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f306c077dd0 0x7f306c1a5730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:06.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:06.094+0000 7f3070a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f306c113b80 con 0x7f306c077dd0 2024-05-12T22:06:06.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:06.094+0000 7f3070a00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f306c113a00 con 0x7f306c0761f0 2024-05-12T22:06:06.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:06.094+0000 7f3070a00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f306c113d00 con 0x7f306c077430 2024-05-12T22:06:06.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:06.094+0000 7f3069e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f306c0761f0 0x7f306c19e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:06.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:06.094+0000 7f3069e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f306c0761f0 0x7f306c19e180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:35304/0 (socket says 172.21.15.28:35304) 2024-05-12T22:06:06.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:06.094+0000 7f3069e00640 1 -- 172.21.15.28:0/1577806957 learned_addr learned my addr 172.21.15.28:0/1577806957 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:06:06.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:06.095+0000 7f306a800640 1 --2- 172.21.15.28:0/1577806957 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f306c077dd0 0x7f306c1a5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:06.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:06.095+0000 7f3069400640 1 --2- 172.21.15.28:0/1577806957 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f306c077430 0x7f306c19e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:06.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:06.095+0000 7f3069e00640 1 -- 172.21.15.28:0/1577806957 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f306c077430 msgr2=0x7f306c19e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:06.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:06.095+0000 7f3069e00640 1 --2- 172.21.15.28:0/1577806957 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f306c077430 0x7f306c19e6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:06.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:06.095+0000 7f3069e00640 1 -- 172.21.15.28:0/1577806957 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f306c077dd0 msgr2=0x7f306c1a5730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:06.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:06.095+0000 7f3069e00640 1 --2- 172.21.15.28:0/1577806957 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f306c077dd0 0x7f306c1a5730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:06.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:06.095+0000 7f3069e00640 1 -- 172.21.15.28:0/1577806957 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f306c1a5da0 con 0x7f306c0761f0 2024-05-12T22:06:06.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:06.095+0000 7f3069e00640 1 --2- 172.21.15.28:0/1577806957 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f306c0761f0 0x7f306c19e180 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f305c031440 tx=0x7f305c032ce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:06:06.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:06.096+0000 7f3062a00640 1 -- 172.21.15.28:0/1577806957 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f305c03f030 con 0x7f306c0761f0 2024-05-12T22:06:06.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:06.096+0000 7f3070a00640 1 -- 172.21.15.28:0/1577806957 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f306c1a6030 con 0x7f306c0761f0 2024-05-12T22:06:06.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:06.096+0000 7f3062a00640 1 -- 172.21.15.28:0/1577806957 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f305c003ce0 con 0x7f306c0761f0 2024-05-12T22:06:06.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:06.096+0000 7f3062a00640 1 -- 172.21.15.28:0/1577806957 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f305c035070 con 0x7f306c0761f0 2024-05-12T22:06:06.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:06.096+0000 7f3070a00640 1 -- 172.21.15.28:0/1577806957 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f306c1a6570 con 0x7f306c0761f0 2024-05-12T22:06:06.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:06.097+0000 7f3070a00640 1 -- 172.21.15.28:0/1577806957 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f306c110590 con 0x7f306c0761f0 2024-05-12T22:06:06.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:06.099+0000 7f3062a00640 1 -- 172.21.15.28:0/1577806957 <== mon.2 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f305c042030 con 0x7f306c0761f0 2024-05-12T22:06:06.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:06.100+0000 7f3062a00640 1 --2- 172.21.15.28:0/1577806957 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f30500a2c10 0x7f30500a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:06.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:06.100+0000 7f3062a00640 1 -- 172.21.15.28:0/1577806957 <== mon.2 v2:172.21.15.86:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f305c0ef0f0 con 0x7f306c0761f0 2024-05-12T22:06:06.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:06.100+0000 7f3069400640 1 --2- 172.21.15.28:0/1577806957 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f30500a2c10 0x7f30500a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:06.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:06.101+0000 7f3069400640 1 --2- 172.21.15.28:0/1577806957 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f30500a2c10 0x7f30500a50d0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f306c19f690 tx=0x7f3058005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:06:06.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:06.104+0000 7f3062a00640 1 -- 172.21.15.28:0/1577806957 <== mon.2 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f305c0bbe00 con 0x7f306c0761f0 2024-05-12T22:06:06.242 INFO:journalctl@ceph.mds.e.smithi086.stdout:May 12 22:06:05 smithi086 podman[64159]: 2024-05-12 22:06:05.801342919 +0000 UTC m=+0.352258471 container init d1d854c7ed0bdecf637489fdc37ec7118d6d395acf840a27ea55c05a62ca1305 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-e, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-f72fecf, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.build-date=20240507, org.label-schema.vendor=CentOS) 2024-05-12T22:06:06.242 INFO:journalctl@ceph.mds.e.smithi086.stdout:May 12 22:06:05 smithi086 podman[64159]: 2024-05-12 22:06:05.804180223 +0000 UTC m=+0.355095779 container start d1d854c7ed0bdecf637489fdc37ec7118d6d395acf840a27ea55c05a62ca1305 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-e, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=squid-f72fecf, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240507, maintainer=Guillaume Abrioux , GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-05-12T22:06:06.242 INFO:journalctl@ceph.mds.e.smithi086.stdout:May 12 22:06:05 smithi086 ceph-mds[64178]: -- 172.21.15.86:0/2549732951 <== mon.2 v2:172.21.15.86:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5605b8ba5a40 con 0x5605b8b78800 2024-05-12T22:06:06.242 INFO:journalctl@ceph.mds.e.smithi086.stdout:May 12 22:06:05 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-e[64174]: starting mds.e at 2024-05-12T22:06:06.242 INFO:journalctl@ceph.mds.e.smithi086.stdout:May 12 22:06:05 smithi086 bash[64159]: d1d854c7ed0bdecf637489fdc37ec7118d6d395acf840a27ea55c05a62ca1305 2024-05-12T22:06:06.242 INFO:journalctl@ceph.mds.e.smithi086.stdout:May 12 22:06:05 smithi086 systemd[1]: Started Ceph mds.e for 6b92e7b4-10aa-11ef-bc98-c7b262605968. 2024-05-12T22:06:06.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:06.349+0000 7f3070a00640 1 -- 172.21.15.28:0/1577806957 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f306c02d1c0 con 0x7f306c0761f0 2024-05-12T22:06:06.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:06.350+0000 7f3062a00640 1 -- 172.21.15.28:0/1577806957 <== mon.2 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+3871 (secure 0 0 0) 0x7f305c0c0100 con 0x7f306c0761f0 2024-05-12T22:06:06.351 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-12T22:06:06.351 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1844141745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1844141745},{"type":"v1","addr":"172.21.15.186:6833","nonce":1844141745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6833/2960763758","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":2960763758},{"type":"v1","addr":"172.21.15.86:6833","nonce":2960763758}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24470,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/4131134219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":4131134219},{"type":"v1","addr":"172.21.15.28:6835","nonce":4131134219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24482,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2938260704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2938260704},{"type":"v1","addr":"172.21.15.28:6837","nonce":2938260704}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[]} 2024-05-12T22:06:06.351 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 5 2024-05-12T22:06:06.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:06.353+0000 7f3070a00640 1 -- 172.21.15.28:0/1577806957 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f30500a2c10 msgr2=0x7f30500a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:06.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:06.353+0000 7f3070a00640 1 --2- 172.21.15.28:0/1577806957 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f30500a2c10 0x7f30500a50d0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f306c19f690 tx=0x7f3058005f90 comp rx=0 tx=0).stop 2024-05-12T22:06:06.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:06.353+0000 7f3070a00640 1 -- 172.21.15.28:0/1577806957 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f306c0761f0 msgr2=0x7f306c19e180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:06.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:06.353+0000 7f3070a00640 1 --2- 172.21.15.28:0/1577806957 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f306c0761f0 0x7f306c19e180 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f305c031440 tx=0x7f305c032ce0 comp rx=0 tx=0).stop 2024-05-12T22:06:06.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:06.353+0000 7f3070a00640 1 -- 172.21.15.28:0/1577806957 shutdown_connections 2024-05-12T22:06:06.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:06.353+0000 7f3070a00640 1 --2- 172.21.15.28:0/1577806957 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f30500a2c10 0x7f30500a50d0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:06.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:06.353+0000 7f3070a00640 1 --2- 172.21.15.28:0/1577806957 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f306c077dd0 0x7f306c1a5730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:06.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:06.353+0000 7f3070a00640 1 --2- 172.21.15.28:0/1577806957 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f306c077430 0x7f306c19e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:06.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:06.353+0000 7f3070a00640 1 --2- 172.21.15.28:0/1577806957 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f306c0761f0 0x7f306c19e180 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:06.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:06.353+0000 7f3070a00640 1 -- 172.21.15.28:0/1577806957 >> 172.21.15.28:0/1577806957 conn(0x7f306c0ff020 msgr2=0x7f306c100760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:06:06.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:06.353+0000 7f3070a00640 1 -- 172.21.15.28:0/1577806957 shutdown_connections 2024-05-12T22:06:06.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:06.353+0000 7f3070a00640 1 -- 172.21.15.28:0/1577806957 wait complete. 2024-05-12T22:06:07.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:06 smithi186 ceph-mon[34785]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-12T22:06:07.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:06 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:07.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:06 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:07.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:06 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:07.180 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:06 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-12T22:06:07.180 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:06 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-12T22:06:07.180 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:06 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:06:07.180 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:06 smithi186 ceph-mon[34785]: Deploying daemon mds.f on smithi186 2024-05-12T22:06:07.180 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:06 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/1577806957' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:06:07.180 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:06 smithi186 ceph-mon[34785]: mds.? [v2:172.21.15.86:6834/3473700910,v1:172.21.15.86:6835/3473700910] up:boot 2024-05-12T22:06:07.180 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:06 smithi186 ceph-mon[34785]: fsmap 5 up:standby 2024-05-12T22:06:07.180 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:06 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-12T22:06:07.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:06 smithi086 ceph-mon[34297]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-12T22:06:07.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:06 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:07.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:06 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:07.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:06 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:07.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:06 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-12T22:06:07.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:06 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-12T22:06:07.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:06 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:06:07.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:06 smithi086 ceph-mon[34297]: Deploying daemon mds.f on smithi186 2024-05-12T22:06:07.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:06 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/1577806957' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:06:07.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:06 smithi086 ceph-mon[34297]: mds.? [v2:172.21.15.86:6834/3473700910,v1:172.21.15.86:6835/3473700910] up:boot 2024-05-12T22:06:07.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:06 smithi086 ceph-mon[34297]: fsmap 5 up:standby 2024-05-12T22:06:07.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:06 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-12T22:06:07.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:06 smithi028 ceph-mon[29847]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-12T22:06:07.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:06 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:07.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:06 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:07.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:06 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:07.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:06 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-12T22:06:07.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:06 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-12T22:06:07.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:06 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:06:07.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:06 smithi028 ceph-mon[29847]: Deploying daemon mds.f on smithi186 2024-05-12T22:06:07.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:06 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/1577806957' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:06:07.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:06 smithi028 ceph-mon[29847]: mds.? [v2:172.21.15.86:6834/3473700910,v1:172.21.15.86:6835/3473700910] up:boot 2024-05-12T22:06:07.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:06 smithi028 ceph-mon[29847]: fsmap 5 up:standby 2024-05-12T22:06:07.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:06 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-12T22:06:08.179 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 12 22:06:08 smithi186 systemd[1]: Starting Ceph mds.f for 6b92e7b4-10aa-11ef-bc98-c7b262605968... 2024-05-12T22:06:08.374 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-12T22:06:08.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:08.601+0000 7f2e1e800640 1 -- 172.21.15.28:0/259826457 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e18077430 msgr2=0x7f2e18077890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:08.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:08.601+0000 7f2e1e800640 1 --2- 172.21.15.28:0/259826457 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e18077430 0x7f2e18077890 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f2e00009900 tx=0x7f2e0002a3f0 comp rx=0 tx=0).stop 2024-05-12T22:06:08.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:08.601+0000 7f2e1e800640 1 -- 172.21.15.28:0/259826457 shutdown_connections 2024-05-12T22:06:08.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:08.601+0000 7f2e1e800640 1 --2- 172.21.15.28:0/259826457 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2e18077dd0 0x7f2e18078670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:08.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:08.601+0000 7f2e1e800640 1 --2- 172.21.15.28:0/259826457 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e18077430 0x7f2e18077890 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:08.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:08.601+0000 7f2e1e800640 1 --2- 172.21.15.28:0/259826457 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e180761f0 0x7f2e18076610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:08.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:08.601+0000 7f2e1e800640 1 -- 172.21.15.28:0/259826457 >> 172.21.15.28:0/259826457 conn(0x7f2e180ff020 msgr2=0x7f2e18101460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:06:08.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:08.601+0000 7f2e1e800640 1 -- 172.21.15.28:0/259826457 shutdown_connections 2024-05-12T22:06:08.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:08.601+0000 7f2e1e800640 1 -- 172.21.15.28:0/259826457 wait complete. 2024-05-12T22:06:08.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:08.602+0000 7f2e1e800640 1 Processor -- start 2024-05-12T22:06:08.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:08.602+0000 7f2e1e800640 1 -- start start 2024-05-12T22:06:08.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:08.603+0000 7f2e1e800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e180761f0 0x7f2e1819e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:08.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:08.603+0000 7f2e1e800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e18077430 0x7f2e1819e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:08.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:08.603+0000 7f2e1e800640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2e18077dd0 0x7f2e181a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:08.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:08.603+0000 7f2e1e800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2e18113b80 con 0x7f2e180761f0 2024-05-12T22:06:08.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:08.603+0000 7f2e1e800640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f2e18113a00 con 0x7f2e18077430 2024-05-12T22:06:08.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:08.603+0000 7f2e1e800640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f2e18113d00 con 0x7f2e18077dd0 2024-05-12T22:06:08.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:08.603+0000 7f2e17400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e180761f0 0x7f2e1819e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:08.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:08.603+0000 7f2e17400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e180761f0 0x7f2e1819e1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39034/0 (socket says 172.21.15.28:39034) 2024-05-12T22:06:08.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:08.603+0000 7f2e17400640 1 -- 172.21.15.28:0/1949006008 learned_addr learned my addr 172.21.15.28:0/1949006008 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:06:08.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:08.603+0000 7f2e0fe00640 1 --2- 172.21.15.28:0/1949006008 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e18077430 0x7f2e1819e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:08.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:08.603+0000 7f2e17e00640 1 --2- 172.21.15.28:0/1949006008 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2e18077dd0 0x7f2e181a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:08.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:08.603+0000 7f2e0fe00640 1 -- 172.21.15.28:0/1949006008 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2e18077dd0 msgr2=0x7f2e181a5780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:08.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:08.603+0000 7f2e0fe00640 1 --2- 172.21.15.28:0/1949006008 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2e18077dd0 0x7f2e181a5780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:08.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:08.603+0000 7f2e0fe00640 1 -- 172.21.15.28:0/1949006008 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e180761f0 msgr2=0x7f2e1819e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:08.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:08.603+0000 7f2e0fe00640 1 --2- 172.21.15.28:0/1949006008 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e180761f0 0x7f2e1819e1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:08.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:08.603+0000 7f2e0fe00640 1 -- 172.21.15.28:0/1949006008 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e181a5df0 con 0x7f2e18077430 2024-05-12T22:06:08.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:08.604+0000 7f2e17400640 1 --2- 172.21.15.28:0/1949006008 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e180761f0 0x7f2e1819e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:06:08.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:08.604+0000 7f2e0fe00640 1 --2- 172.21.15.28:0/1949006008 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e18077430 0x7f2e1819e710 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f2e0002a910 tx=0x7f2e00032ce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:06:08.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:08.604+0000 7f2e14c00640 1 -- 172.21.15.28:0/1949006008 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e0003f030 con 0x7f2e18077430 2024-05-12T22:06:08.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:08.604+0000 7f2e14c00640 1 -- 172.21.15.28:0/1949006008 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2e00003ce0 con 0x7f2e18077430 2024-05-12T22:06:08.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:08.604+0000 7f2e1e800640 1 -- 172.21.15.28:0/1949006008 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e181a6080 con 0x7f2e18077430 2024-05-12T22:06:08.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:08.604+0000 7f2e14c00640 1 -- 172.21.15.28:0/1949006008 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e00035070 con 0x7f2e18077430 2024-05-12T22:06:08.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:08.604+0000 7f2e1e800640 1 -- 172.21.15.28:0/1949006008 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e181a6590 con 0x7f2e18077430 2024-05-12T22:06:08.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:08.605+0000 7f2e1e800640 1 -- 172.21.15.28:0/1949006008 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e181105b0 con 0x7f2e18077430 2024-05-12T22:06:08.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:08.607+0000 7f2e14c00640 1 -- 172.21.15.28:0/1949006008 <== mon.2 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f2e00042030 con 0x7f2e18077430 2024-05-12T22:06:08.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:08.608+0000 7f2e14c00640 1 --2- 172.21.15.28:0/1949006008 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f2dfc0a2c10 0x7f2dfc0a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:08.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:08.609+0000 7f2e14c00640 1 -- 172.21.15.28:0/1949006008 <== mon.2 v2:172.21.15.86:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f2e000eeff0 con 0x7f2e18077430 2024-05-12T22:06:08.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:08.609+0000 7f2e17400640 1 --2- 172.21.15.28:0/1949006008 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f2dfc0a2c10 0x7f2dfc0a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:08.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:08.610+0000 7f2e17400640 1 --2- 172.21.15.28:0/1949006008 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f2dfc0a2c10 0x7f2dfc0a50d0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f2e08005fd0 tx=0x7f2e0800b4a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:06:08.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:08.612+0000 7f2e14c00640 1 -- 172.21.15.28:0/1949006008 <== mon.2 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f2e000bbd20 con 0x7f2e18077430 2024-05-12T22:06:08.633 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 12 22:06:08 smithi186 podman[65304]: 2024-05-12 22:06:08.32287056 +0000 UTC m=+0.106179548 container create 24c6a9b76babfc0e4f1a04c0488d4a710839519ce5481eef28d7db9f6af9516b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-f, ceph=True, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-f72fecf, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240507, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-05-12T22:06:08.633 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 12 22:06:08 smithi186 podman[65304]: 2024-05-12 22:06:08.234021122 +0000 UTC m=+0.017330121 image pull 8252b7944d3b5c188c0ff632647ba6cc4e9a19588ee097b8a3dba27b42779c25 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc 2024-05-12T22:06:08.633 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 12 22:06:08 smithi186 podman[65304]: 2024-05-12 22:06:08.542470458 +0000 UTC m=+0.325779457 container init 24c6a9b76babfc0e4f1a04c0488d4a710839519ce5481eef28d7db9f6af9516b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-f, org.label-schema.build-date=20240507, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=squid-f72fecf, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-12T22:06:08.633 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 12 22:06:08 smithi186 podman[65304]: 2024-05-12 22:06:08.545488271 +0000 UTC m=+0.328797267 container start 24c6a9b76babfc0e4f1a04c0488d4a710839519ce5481eef28d7db9f6af9516b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-f, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.build-date=20240507, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=squid-f72fecf, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-12T22:06:08.633 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 12 22:06:08 smithi186 ceph-mds[65322]: -- 172.21.15.186:0/415415112 <== mon.1 v2:172.21.15.186:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5617e82bba40 con 0x5617e828ec00 2024-05-12T22:06:08.633 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 12 22:06:08 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-f[65318]: starting mds.f at 2024-05-12T22:06:08.633 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 12 22:06:08 smithi186 bash[65304]: 24c6a9b76babfc0e4f1a04c0488d4a710839519ce5481eef28d7db9f6af9516b 2024-05-12T22:06:08.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:08.861+0000 7f2e1e800640 1 -- 172.21.15.28:0/1949006008 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2e18062430 con 0x7f2e18077430 2024-05-12T22:06:08.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:08.862+0000 7f2e14c00640 1 -- 172.21.15.28:0/1949006008 <== mon.2 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+4696 (secure 0 0 0) 0x7f2e000c0020 con 0x7f2e18077430 2024-05-12T22:06:08.863 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-12T22:06:08.863 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1844141745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1844141745},{"type":"v1","addr":"172.21.15.186:6833","nonce":1844141745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6833/2960763758","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":2960763758},{"type":"v1","addr":"172.21.15.86:6833","nonce":2960763758}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24470,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/4131134219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":4131134219},{"type":"v1","addr":"172.21.15.28:6835","nonce":4131134219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24482,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2938260704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2938260704},{"type":"v1","addr":"172.21.15.28:6837","nonce":2938260704}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24494,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3473700910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3473700910},{"type":"v1","addr":"172.21.15.86:6835","nonce":3473700910}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-05-12T22:06:08.863 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 6 2024-05-12T22:06:08.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:08.865+0000 7f2e1e800640 1 -- 172.21.15.28:0/1949006008 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f2dfc0a2c10 msgr2=0x7f2dfc0a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:08.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:08.865+0000 7f2e1e800640 1 --2- 172.21.15.28:0/1949006008 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f2dfc0a2c10 0x7f2dfc0a50d0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f2e08005fd0 tx=0x7f2e0800b4a0 comp rx=0 tx=0).stop 2024-05-12T22:06:08.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:08.865+0000 7f2e1e800640 1 -- 172.21.15.28:0/1949006008 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e18077430 msgr2=0x7f2e1819e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:08.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:08.865+0000 7f2e1e800640 1 --2- 172.21.15.28:0/1949006008 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e18077430 0x7f2e1819e710 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f2e0002a910 tx=0x7f2e00032ce0 comp rx=0 tx=0).stop 2024-05-12T22:06:08.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:08.865+0000 7f2e1e800640 1 -- 172.21.15.28:0/1949006008 shutdown_connections 2024-05-12T22:06:08.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:08.865+0000 7f2e1e800640 1 --2- 172.21.15.28:0/1949006008 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f2dfc0a2c10 0x7f2dfc0a50d0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:08.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:08.865+0000 7f2e1e800640 1 --2- 172.21.15.28:0/1949006008 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2e18077dd0 0x7f2e181a5780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:08.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:08.865+0000 7f2e1e800640 1 --2- 172.21.15.28:0/1949006008 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e18077430 0x7f2e1819e710 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:08.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:08.865+0000 7f2e1e800640 1 --2- 172.21.15.28:0/1949006008 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e180761f0 0x7f2e1819e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:08.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:08.865+0000 7f2e1e800640 1 -- 172.21.15.28:0/1949006008 >> 172.21.15.28:0/1949006008 conn(0x7f2e180ff020 msgr2=0x7f2e18100700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:06:08.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:08.865+0000 7f2e1e800640 1 -- 172.21.15.28:0/1949006008 shutdown_connections 2024-05-12T22:06:08.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:08.865+0000 7f2e1e800640 1 -- 172.21.15.28:0/1949006008 wait complete. 2024-05-12T22:06:08.929 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 12 22:06:08 smithi186 systemd[1]: Started Ceph mds.f for 6b92e7b4-10aa-11ef-bc98-c7b262605968. 2024-05-12T22:06:09.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:08 smithi086 ceph-mon[34297]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-12T22:06:09.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:08 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:09.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:08 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:09.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:08 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:09.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:08 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-12T22:06:09.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:08 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-12T22:06:09.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:08 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:06:09.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:08 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/1949006008' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:06:09.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:08 smithi028 ceph-mon[29847]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-12T22:06:09.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:08 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:09.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:08 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:09.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:08 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:09.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:08 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-12T22:06:09.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:08 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-12T22:06:09.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:08 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:06:09.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:08 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/1949006008' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:06:09.375 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:08 smithi186 ceph-mon[34785]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-12T22:06:09.375 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:08 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:09.375 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:08 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:09.375 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:08 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:09.375 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:08 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-12T22:06:09.375 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:08 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-12T22:06:09.376 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:08 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:06:09.376 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:08 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/1949006008' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:06:10.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:09 smithi086 ceph-mon[34297]: Deploying daemon mds.c on smithi186 2024-05-12T22:06:10.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:09 smithi086 ceph-mon[34297]: mds.? [v2:172.21.15.186:6834/1727869064,v1:172.21.15.186:6835/1727869064] up:boot 2024-05-12T22:06:10.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:09 smithi086 ceph-mon[34297]: fsmap 6 up:standby 2024-05-12T22:06:10.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:09 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-12T22:06:10.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:09 smithi028 ceph-mon[29847]: Deploying daemon mds.c on smithi186 2024-05-12T22:06:10.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:09 smithi028 ceph-mon[29847]: mds.? [v2:172.21.15.186:6834/1727869064,v1:172.21.15.186:6835/1727869064] up:boot 2024-05-12T22:06:10.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:09 smithi028 ceph-mon[29847]: fsmap 6 up:standby 2024-05-12T22:06:10.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:09 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-12T22:06:10.357 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:09 smithi186 ceph-mon[34785]: Deploying daemon mds.c on smithi186 2024-05-12T22:06:10.357 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:09 smithi186 ceph-mon[34785]: mds.? [v2:172.21.15.186:6834/1727869064,v1:172.21.15.186:6835/1727869064] up:boot 2024-05-12T22:06:10.357 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:09 smithi186 ceph-mon[34785]: fsmap 6 up:standby 2024-05-12T22:06:10.357 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:09 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-12T22:06:10.888 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-12T22:06:10.929 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 12 22:06:10 smithi186 systemd[1]: Starting Ceph mds.c for 6b92e7b4-10aa-11ef-bc98-c7b262605968... 2024-05-12T22:06:11.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:11.113+0000 7f01abe00640 1 -- 172.21.15.28:0/1283001296 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f01ac107cb0 msgr2=0x7f01ac1080d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:11.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:11.113+0000 7f01abe00640 1 --2- 172.21.15.28:0/1283001296 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f01ac107cb0 0x7f01ac1080d0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f019c009860 tx=0x7f019c02a340 comp rx=0 tx=0).stop 2024-05-12T22:06:11.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:11.113+0000 7f01abe00640 1 -- 172.21.15.28:0/1283001296 shutdown_connections 2024-05-12T22:06:11.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:11.113+0000 7f01abe00640 1 --2- 172.21.15.28:0/1283001296 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f01ac109860 0x7f01ac1100f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:11.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:11.113+0000 7f01abe00640 1 --2- 172.21.15.28:0/1283001296 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01ac108ea0 0x7f01ac109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:11.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:11.113+0000 7f01abe00640 1 --2- 172.21.15.28:0/1283001296 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f01ac107cb0 0x7f01ac1080d0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:11.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:11.113+0000 7f01abe00640 1 -- 172.21.15.28:0/1283001296 >> 172.21.15.28:0/1283001296 conn(0x7f01ac076c70 msgr2=0x7f01ac079090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:06:11.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:11.113+0000 7f01abe00640 1 -- 172.21.15.28:0/1283001296 shutdown_connections 2024-05-12T22:06:11.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:11.114+0000 7f01abe00640 1 -- 172.21.15.28:0/1283001296 wait complete. 2024-05-12T22:06:11.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:11.114+0000 7f01abe00640 1 Processor -- start 2024-05-12T22:06:11.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:11.115+0000 7f01abe00640 1 -- start start 2024-05-12T22:06:11.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:11.115+0000 7f01abe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01ac107cb0 0x7f01ac19dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:11.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:11.115+0000 7f01abe00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f01ac108ea0 0x7f01ac19e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:11.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:11.115+0000 7f01abe00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f01ac109860 0x7f01ac1a5590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:11.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:11.115+0000 7f01abe00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f01ac113930 con 0x7f01ac107cb0 2024-05-12T22:06:11.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:11.115+0000 7f01abe00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f01ac1137b0 con 0x7f01ac108ea0 2024-05-12T22:06:11.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:11.115+0000 7f01abe00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f01ac113ab0 con 0x7f01ac109860 2024-05-12T22:06:11.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:11.115+0000 7f01aaa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01ac107cb0 0x7f01ac19dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:11.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:11.115+0000 7f01a3e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f01ac108ea0 0x7f01ac19e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:11.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:11.115+0000 7f01aaa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01ac107cb0 0x7f01ac19dfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39054/0 (socket says 172.21.15.28:39054) 2024-05-12T22:06:11.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:11.115+0000 7f01aaa00640 1 -- 172.21.15.28:0/802963968 learned_addr learned my addr 172.21.15.28:0/802963968 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:06:11.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:11.115+0000 7f01ab400640 1 --2- 172.21.15.28:0/802963968 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f01ac109860 0x7f01ac1a5590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:11.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:11.116+0000 7f01aaa00640 1 -- 172.21.15.28:0/802963968 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f01ac109860 msgr2=0x7f01ac1a5590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:11.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:11.116+0000 7f01aaa00640 1 --2- 172.21.15.28:0/802963968 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f01ac109860 0x7f01ac1a5590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:11.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:11.116+0000 7f01aaa00640 1 -- 172.21.15.28:0/802963968 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f01ac108ea0 msgr2=0x7f01ac19e520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:11.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:11.116+0000 7f01aaa00640 1 --2- 172.21.15.28:0/802963968 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f01ac108ea0 0x7f01ac19e520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:11.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:11.116+0000 7f01aaa00640 1 -- 172.21.15.28:0/802963968 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01ac1a5c00 con 0x7f01ac107cb0 2024-05-12T22:06:11.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:11.116+0000 7f01a3e00640 1 --2- 172.21.15.28:0/802963968 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f01ac108ea0 0x7f01ac19e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:06:11.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:11.116+0000 7f01aaa00640 1 --2- 172.21.15.28:0/802963968 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01ac107cb0 0x7f01ac19dfe0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f019c035040 tx=0x7f019c033d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:06:11.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:11.116+0000 7f01a3400640 1 -- 172.21.15.28:0/802963968 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f019c037040 con 0x7f01ac107cb0 2024-05-12T22:06:11.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:11.116+0000 7f01a3400640 1 -- 172.21.15.28:0/802963968 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f019c030850 con 0x7f01ac107cb0 2024-05-12T22:06:11.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:11.116+0000 7f01abe00640 1 -- 172.21.15.28:0/802963968 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01ac1a5e90 con 0x7f01ac107cb0 2024-05-12T22:06:11.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:11.116+0000 7f01a3400640 1 -- 172.21.15.28:0/802963968 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f019c004920 con 0x7f01ac107cb0 2024-05-12T22:06:11.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:11.117+0000 7f01abe00640 1 -- 172.21.15.28:0/802963968 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01ac1a62a0 con 0x7f01ac107cb0 2024-05-12T22:06:11.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:11.118+0000 7f01abe00640 1 -- 172.21.15.28:0/802963968 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01ac10f050 con 0x7f01ac107cb0 2024-05-12T22:06:11.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:11.120+0000 7f01a3400640 1 -- 172.21.15.28:0/802963968 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f019c004050 con 0x7f01ac107cb0 2024-05-12T22:06:11.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:11.121+0000 7f01a3400640 1 --2- 172.21.15.28:0/802963968 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f01900a2b90 0x7f01900a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:11.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:11.121+0000 7f01a3400640 1 -- 172.21.15.28:0/802963968 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f019c0eea90 con 0x7f01ac107cb0 2024-05-12T22:06:11.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:11.122+0000 7f01a3e00640 1 --2- 172.21.15.28:0/802963968 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f01900a2b90 0x7f01900a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:11.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:11.122+0000 7f01a3e00640 1 --2- 172.21.15.28:0/802963968 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f01900a2b90 0x7f01900a5050 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f01940096c0 tx=0x7f0194009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:06:11.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:11.126+0000 7f01a3400640 1 -- 172.21.15.28:0/802963968 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f019c0bbdf0 con 0x7f01ac107cb0 2024-05-12T22:06:11.206 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:10 smithi186 ceph-mon[34785]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-12T22:06:11.207 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 12 22:06:11 smithi186 podman[65611]: 2024-05-12 22:06:10.958269932 +0000 UTC m=+0.018315603 image pull 8252b7944d3b5c188c0ff632647ba6cc4e9a19588ee097b8a3dba27b42779c25 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc 2024-05-12T22:06:11.207 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 12 22:06:11 smithi186 podman[65611]: 2024-05-12 22:06:11.076673039 +0000 UTC m=+0.136718708 container create bfbd5c1aad9994d6b1f812db50681402c28a6f8e213766acd4baffe5c077565d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-c, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=squid-f72fecf, org.label-schema.build-date=20240507, GIT_BRANCH=HEAD, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0) 2024-05-12T22:06:11.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:10 smithi086 ceph-mon[34297]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-12T22:06:11.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:10 smithi028 ceph-mon[29847]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-12T22:06:11.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:11.372+0000 7f01abe00640 1 -- 172.21.15.28:0/802963968 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f01ac062570 con 0x7f01ac107cb0 2024-05-12T22:06:11.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:11.374+0000 7f01a3400640 1 -- 172.21.15.28:0/802963968 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5524 (secure 0 0 0) 0x7f019c0bb6f0 con 0x7f01ac107cb0 2024-05-12T22:06:11.375 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-12T22:06:11.375 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1844141745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1844141745},{"type":"v1","addr":"172.21.15.186:6833","nonce":1844141745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6833/2960763758","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":2960763758},{"type":"v1","addr":"172.21.15.86:6833","nonce":2960763758}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24470,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/4131134219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":4131134219},{"type":"v1","addr":"172.21.15.28:6835","nonce":4131134219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24481,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/1727869064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1727869064},{"type":"v1","addr":"172.21.15.186:6835","nonce":1727869064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24482,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2938260704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2938260704},{"type":"v1","addr":"172.21.15.28:6837","nonce":2938260704}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24494,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3473700910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3473700910},{"type":"v1","addr":"172.21.15.86:6835","nonce":3473700910}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-05-12T22:06:11.376 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 7 2024-05-12T22:06:11.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:11.377+0000 7f01abe00640 1 -- 172.21.15.28:0/802963968 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f01900a2b90 msgr2=0x7f01900a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:11.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:11.377+0000 7f01abe00640 1 --2- 172.21.15.28:0/802963968 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f01900a2b90 0x7f01900a5050 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f01940096c0 tx=0x7f0194009260 comp rx=0 tx=0).stop 2024-05-12T22:06:11.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:11.377+0000 7f01abe00640 1 -- 172.21.15.28:0/802963968 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01ac107cb0 msgr2=0x7f01ac19dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:11.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:11.377+0000 7f01abe00640 1 --2- 172.21.15.28:0/802963968 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01ac107cb0 0x7f01ac19dfe0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f019c035040 tx=0x7f019c033d90 comp rx=0 tx=0).stop 2024-05-12T22:06:11.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:11.377+0000 7f01abe00640 1 -- 172.21.15.28:0/802963968 shutdown_connections 2024-05-12T22:06:11.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:11.377+0000 7f01abe00640 1 --2- 172.21.15.28:0/802963968 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f01900a2b90 0x7f01900a5050 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:11.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:11.377+0000 7f01abe00640 1 --2- 172.21.15.28:0/802963968 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f01ac109860 0x7f01ac1a5590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:11.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:11.377+0000 7f01abe00640 1 --2- 172.21.15.28:0/802963968 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f01ac108ea0 0x7f01ac19e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:11.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:11.377+0000 7f01abe00640 1 --2- 172.21.15.28:0/802963968 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01ac107cb0 0x7f01ac19dfe0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:11.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:11.377+0000 7f01abe00640 1 -- 172.21.15.28:0/802963968 >> 172.21.15.28:0/802963968 conn(0x7f01ac076c70 msgr2=0x7f01ac10c220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:06:11.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:11.377+0000 7f01abe00640 1 -- 172.21.15.28:0/802963968 shutdown_connections 2024-05-12T22:06:11.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:11.377+0000 7f01abe00640 1 -- 172.21.15.28:0/802963968 wait complete. 2024-05-12T22:06:11.679 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 12 22:06:11 smithi186 podman[65611]: 2024-05-12 22:06:11.313750223 +0000 UTC m=+0.373795892 container init bfbd5c1aad9994d6b1f812db50681402c28a6f8e213766acd4baffe5c077565d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-c, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240507, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=squid-f72fecf, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-05-12T22:06:11.680 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 12 22:06:11 smithi186 podman[65611]: 2024-05-12 22:06:11.316484047 +0000 UTC m=+0.376529717 container start bfbd5c1aad9994d6b1f812db50681402c28a6f8e213766acd4baffe5c077565d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-c, org.label-schema.build-date=20240507, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=squid-f72fecf, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-05-12T22:06:11.680 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 12 22:06:11 smithi186 ceph-mds[65630]: -- 172.21.15.186:0/2672936173 <== mon.1 v2:172.21.15.186:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x564b645e3a40 con 0x564b645b7000 2024-05-12T22:06:11.680 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 12 22:06:11 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-c[65626]: starting mds.c at 2024-05-12T22:06:11.680 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 12 22:06:11 smithi186 bash[65611]: bfbd5c1aad9994d6b1f812db50681402c28a6f8e213766acd4baffe5c077565d 2024-05-12T22:06:11.680 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 12 22:06:11 smithi186 systemd[1]: Started Ceph mds.c for 6b92e7b4-10aa-11ef-bc98-c7b262605968. 2024-05-12T22:06:12.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:11 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/802963968' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:06:12.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:11 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:12.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:11 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:12.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:11 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:12.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:11 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-12T22:06:12.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:11 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-12T22:06:12.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:11 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:06:12.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:11 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/802963968' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:06:12.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:11 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:12.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:11 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:12.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:11 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:12.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:11 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-12T22:06:12.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:11 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-12T22:06:12.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:11 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:06:12.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:11 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/802963968' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:06:12.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:11 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:12.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:11 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:12.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:11 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:12.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:11 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-12T22:06:12.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:11 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-12T22:06:12.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:11 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:06:13.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:12 smithi086 ceph-mon[34297]: Deploying daemon mds.h on smithi086 2024-05-12T22:06:13.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:12 smithi086 ceph-mon[34297]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-12T22:06:13.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:12 smithi086 ceph-mon[34297]: mds.? [v2:172.21.15.186:6836/3600809235,v1:172.21.15.186:6837/3600809235] up:boot 2024-05-12T22:06:13.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:12 smithi086 ceph-mon[34297]: fsmap 7 up:standby 2024-05-12T22:06:13.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:12 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-12T22:06:13.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:12 smithi028 ceph-mon[29847]: Deploying daemon mds.h on smithi086 2024-05-12T22:06:13.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:12 smithi028 ceph-mon[29847]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-12T22:06:13.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:12 smithi028 ceph-mon[29847]: mds.? [v2:172.21.15.186:6836/3600809235,v1:172.21.15.186:6837/3600809235] up:boot 2024-05-12T22:06:13.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:12 smithi028 ceph-mon[29847]: fsmap 7 up:standby 2024-05-12T22:06:13.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:12 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-12T22:06:13.399 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-12T22:06:13.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:12 smithi186 ceph-mon[34785]: Deploying daemon mds.h on smithi086 2024-05-12T22:06:13.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:12 smithi186 ceph-mon[34785]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-12T22:06:13.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:12 smithi186 ceph-mon[34785]: mds.? [v2:172.21.15.186:6836/3600809235,v1:172.21.15.186:6837/3600809235] up:boot 2024-05-12T22:06:13.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:12 smithi186 ceph-mon[34785]: fsmap 7 up:standby 2024-05-12T22:06:13.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:12 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-12T22:06:13.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:13.634+0000 7f4c9b200640 1 -- 172.21.15.28:0/910479010 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c9c1016a0 msgr2=0x7f4c9c101ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:13.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:13.634+0000 7f4c9b200640 1 --2- 172.21.15.28:0/910479010 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c9c1016a0 0x7f4c9c101ac0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f4c8c009900 tx=0x7f4c8c02a3f0 comp rx=0 tx=0).stop 2024-05-12T22:06:13.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:13.634+0000 7f4c9b200640 1 -- 172.21.15.28:0/910479010 shutdown_connections 2024-05-12T22:06:13.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:13.634+0000 7f4c9b200640 1 --2- 172.21.15.28:0/910479010 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4c9c103250 0x7f4c9c109ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:13.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:13.634+0000 7f4c9b200640 1 --2- 172.21.15.28:0/910479010 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4c9c102890 0x7f4c9c102d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:13.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:13.634+0000 7f4c9b200640 1 --2- 172.21.15.28:0/910479010 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c9c1016a0 0x7f4c9c101ac0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:13.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:13.634+0000 7f4c9b200640 1 -- 172.21.15.28:0/910479010 >> 172.21.15.28:0/910479010 conn(0x7f4c9c0fcc10 msgr2=0x7f4c9c0ff050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:06:13.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:13.634+0000 7f4c9b200640 1 -- 172.21.15.28:0/910479010 shutdown_connections 2024-05-12T22:06:13.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:13.634+0000 7f4c9b200640 1 -- 172.21.15.28:0/910479010 wait complete. 2024-05-12T22:06:13.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:13.635+0000 7f4c9b200640 1 Processor -- start 2024-05-12T22:06:13.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:13.635+0000 7f4c9b200640 1 -- start start 2024-05-12T22:06:13.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:13.635+0000 7f4c9b200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c9c1016a0 0x7f4c9c199c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:13.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:13.635+0000 7f4c9b200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4c9c102890 0x7f4c9c19a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:13.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:13.635+0000 7f4c9b200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4c9c103250 0x7f4c9c1a11d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:13.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:13.635+0000 7f4c9b200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4c9c10d360 con 0x7f4c9c1016a0 2024-05-12T22:06:13.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:13.635+0000 7f4c9b200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f4c9c10d1e0 con 0x7f4c9c102890 2024-05-12T22:06:13.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:13.635+0000 7f4c9b200640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f4c9c10d4e0 con 0x7f4c9c103250 2024-05-12T22:06:13.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:13.636+0000 7f4c99400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4c9c102890 0x7f4c9c19a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:13.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:13.636+0000 7f4c99e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c9c1016a0 0x7f4c9c199c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:13.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:13.636+0000 7f4c99400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4c9c102890 0x7f4c9c19a1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:35352/0 (socket says 172.21.15.28:35352) 2024-05-12T22:06:13.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:13.636+0000 7f4c9a800640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4c9c103250 0x7f4c9c1a11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:13.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:13.636+0000 7f4c99400640 1 -- 172.21.15.28:0/1937925718 learned_addr learned my addr 172.21.15.28:0/1937925718 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:06:13.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:13.636+0000 7f4c99400640 1 -- 172.21.15.28:0/1937925718 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4c9c103250 msgr2=0x7f4c9c1a11d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:13.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:13.636+0000 7f4c99400640 1 --2- 172.21.15.28:0/1937925718 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4c9c103250 0x7f4c9c1a11d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:13.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:13.636+0000 7f4c99400640 1 -- 172.21.15.28:0/1937925718 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c9c1016a0 msgr2=0x7f4c9c199c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:13.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:13.636+0000 7f4c99400640 1 --2- 172.21.15.28:0/1937925718 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c9c1016a0 0x7f4c9c199c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:13.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:13.636+0000 7f4c99400640 1 -- 172.21.15.28:0/1937925718 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c9c1a1870 con 0x7f4c9c102890 2024-05-12T22:06:13.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:13.636+0000 7f4c9a800640 1 --2- 172.21.15.28:0/1937925718 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4c9c103250 0x7f4c9c1a11d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:06:13.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:13.637+0000 7f4c99400640 1 --2- 172.21.15.28:0/1937925718 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4c9c102890 0x7f4c9c19a1a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f4c8800e9f0 tx=0x7f4c8800edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:06:13.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:13.637+0000 7f4c92a00640 1 -- 172.21.15.28:0/1937925718 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c8800ccc0 con 0x7f4c9c102890 2024-05-12T22:06:13.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:13.637+0000 7f4c92a00640 1 -- 172.21.15.28:0/1937925718 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4c8800ce60 con 0x7f4c9c102890 2024-05-12T22:06:13.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:13.637+0000 7f4c92a00640 1 -- 172.21.15.28:0/1937925718 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c88010690 con 0x7f4c9c102890 2024-05-12T22:06:13.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:13.637+0000 7f4c9b200640 1 -- 172.21.15.28:0/1937925718 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c9c1a1b60 con 0x7f4c9c102890 2024-05-12T22:06:13.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:13.637+0000 7f4c9b200640 1 -- 172.21.15.28:0/1937925718 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c9c1a2030 con 0x7f4c9c102890 2024-05-12T22:06:13.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:13.638+0000 7f4c9b200640 1 -- 172.21.15.28:0/1937925718 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c9c1a2260 con 0x7f4c9c102890 2024-05-12T22:06:13.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:13.641+0000 7f4c92a00640 1 -- 172.21.15.28:0/1937925718 <== mon.2 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f4c880046c0 con 0x7f4c9c102890 2024-05-12T22:06:13.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:13.642+0000 7f4c92a00640 1 --2- 172.21.15.28:0/1937925718 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f4c800a2b90 0x7f4c800a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:13.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:13.642+0000 7f4c92a00640 1 -- 172.21.15.28:0/1937925718 <== mon.2 v2:172.21.15.86:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f4c880cf0f0 con 0x7f4c9c102890 2024-05-12T22:06:13.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:13.643+0000 7f4c99e00640 1 --2- 172.21.15.28:0/1937925718 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f4c800a2b90 0x7f4c800a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:13.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:13.643+0000 7f4c99e00640 1 --2- 172.21.15.28:0/1937925718 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f4c800a2b90 0x7f4c800a5050 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f4c8c02a9e0 tx=0x7f4c8c0315b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:06:13.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:13.647+0000 7f4c92a00640 1 -- 172.21.15.28:0/1937925718 <== mon.2 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f4c8809c450 con 0x7f4c9c102890 2024-05-12T22:06:13.742 INFO:journalctl@ceph.mds.h.smithi086.stdout:May 12 22:06:13 smithi086 systemd[1]: Starting Ceph mds.h for 6b92e7b4-10aa-11ef-bc98-c7b262605968... 2024-05-12T22:06:13.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:13.893+0000 7f4c9b200640 1 -- 172.21.15.28:0/1937925718 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4c9c1a2430 con 0x7f4c9c102890 2024-05-12T22:06:13.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:13.894+0000 7f4c92a00640 1 -- 172.21.15.28:0/1937925718 <== mon.2 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6352 (secure 0 0 0) 0x7f4c8809bd50 con 0x7f4c9c102890 2024-05-12T22:06:13.894 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-12T22:06:13.894 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1844141745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1844141745},{"type":"v1","addr":"172.21.15.186:6833","nonce":1844141745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6833/2960763758","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":2960763758},{"type":"v1","addr":"172.21.15.86:6833","nonce":2960763758}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24470,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/4131134219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":4131134219},{"type":"v1","addr":"172.21.15.28:6835","nonce":4131134219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24481,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/1727869064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1727869064},{"type":"v1","addr":"172.21.15.186:6835","nonce":1727869064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24482,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2938260704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2938260704},{"type":"v1","addr":"172.21.15.28:6837","nonce":2938260704}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24494,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3473700910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3473700910},{"type":"v1","addr":"172.21.15.86:6835","nonce":3473700910}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24499,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3600809235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":3600809235},{"type":"v1","addr":"172.21.15.186:6837","nonce":3600809235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-05-12T22:06:13.894 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 8 2024-05-12T22:06:13.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:13.897+0000 7f4c9b200640 1 -- 172.21.15.28:0/1937925718 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f4c800a2b90 msgr2=0x7f4c800a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:13.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:13.897+0000 7f4c9b200640 1 --2- 172.21.15.28:0/1937925718 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f4c800a2b90 0x7f4c800a5050 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f4c8c02a9e0 tx=0x7f4c8c0315b0 comp rx=0 tx=0).stop 2024-05-12T22:06:13.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:13.897+0000 7f4c9b200640 1 -- 172.21.15.28:0/1937925718 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4c9c102890 msgr2=0x7f4c9c19a1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:13.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:13.897+0000 7f4c9b200640 1 --2- 172.21.15.28:0/1937925718 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4c9c102890 0x7f4c9c19a1a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f4c8800e9f0 tx=0x7f4c8800edf0 comp rx=0 tx=0).stop 2024-05-12T22:06:13.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:13.897+0000 7f4c9b200640 1 -- 172.21.15.28:0/1937925718 shutdown_connections 2024-05-12T22:06:13.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:13.897+0000 7f4c9b200640 1 --2- 172.21.15.28:0/1937925718 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f4c800a2b90 0x7f4c800a5050 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:13.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:13.897+0000 7f4c9b200640 1 --2- 172.21.15.28:0/1937925718 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4c9c103250 0x7f4c9c1a11d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:13.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:13.897+0000 7f4c9b200640 1 --2- 172.21.15.28:0/1937925718 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4c9c102890 0x7f4c9c19a1a0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:13.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:13.897+0000 7f4c9b200640 1 --2- 172.21.15.28:0/1937925718 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c9c1016a0 0x7f4c9c199c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:13.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:13.897+0000 7f4c9b200640 1 -- 172.21.15.28:0/1937925718 >> 172.21.15.28:0/1937925718 conn(0x7f4c9c0fcc10 msgr2=0x7f4c9c073a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:06:13.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:13.897+0000 7f4c9b200640 1 -- 172.21.15.28:0/1937925718 shutdown_connections 2024-05-12T22:06:13.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:13.898+0000 7f4c9b200640 1 -- 172.21.15.28:0/1937925718 wait complete. 2024-05-12T22:06:14.095 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:13 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/1937925718' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:06:14.095 INFO:journalctl@ceph.mds.h.smithi086.stdout:May 12 22:06:13 smithi086 podman[64481]: 2024-05-12 22:06:13.711058162 +0000 UTC m=+0.017984639 image pull 8252b7944d3b5c188c0ff632647ba6cc4e9a19588ee097b8a3dba27b42779c25 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc 2024-05-12T22:06:14.095 INFO:journalctl@ceph.mds.h.smithi086.stdout:May 12 22:06:13 smithi086 podman[64481]: 2024-05-12 22:06:13.835037404 +0000 UTC m=+0.141963853 container create 8e70c957f69d5998c4d23321e0389c5f741ed4027daa456b71c01cdeb6435ea0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-h, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=squid-f72fecf, org.label-schema.build-date=20240507) 2024-05-12T22:06:14.095 INFO:journalctl@ceph.mds.h.smithi086.stdout:May 12 22:06:14 smithi086 podman[64481]: 2024-05-12 22:06:14.096111289 +0000 UTC m=+0.403037742 container init 8e70c957f69d5998c4d23321e0389c5f741ed4027daa456b71c01cdeb6435ea0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-h, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.build-date=20240507, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, RELEASE=squid-f72fecf, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-05-12T22:06:14.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:13 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/1937925718' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:06:14.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:13 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/1937925718' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:06:14.492 INFO:journalctl@ceph.mds.h.smithi086.stdout:May 12 22:06:14 smithi086 podman[64481]: 2024-05-12 22:06:14.098806854 +0000 UTC m=+0.405733311 container start 8e70c957f69d5998c4d23321e0389c5f741ed4027daa456b71c01cdeb6435ea0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-h, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240507, GIT_CLEAN=True, RELEASE=squid-f72fecf, maintainer=Guillaume Abrioux , GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-05-12T22:06:14.492 INFO:journalctl@ceph.mds.h.smithi086.stdout:May 12 22:06:14 smithi086 ceph-mds[64500]: -- 172.21.15.86:0/2828804973 <== mon.2 v2:172.21.15.86:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x564e6303fa40 con 0x564e63012800 2024-05-12T22:06:14.492 INFO:journalctl@ceph.mds.h.smithi086.stdout:May 12 22:06:14 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-h[64496]: starting mds.h at 2024-05-12T22:06:14.492 INFO:journalctl@ceph.mds.h.smithi086.stdout:May 12 22:06:14 smithi086 bash[64481]: 8e70c957f69d5998c4d23321e0389c5f741ed4027daa456b71c01cdeb6435ea0 2024-05-12T22:06:14.492 INFO:journalctl@ceph.mds.h.smithi086.stdout:May 12 22:06:14 smithi086 systemd[1]: Started Ceph mds.h for 6b92e7b4-10aa-11ef-bc98-c7b262605968. 2024-05-12T22:06:15.528 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:15 smithi186 ceph-mon[34785]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-12T22:06:15.528 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:15 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:15.528 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:15 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:15.528 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:15 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:15.528 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:15 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-12T22:06:15.528 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:15 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-12T22:06:15.528 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:15 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:06:15.529 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:15 smithi186 ceph-mon[34785]: Deploying daemon mds.i on smithi186 2024-05-12T22:06:15.529 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:15 smithi186 ceph-mon[34785]: mds.? [v2:172.21.15.86:6836/3782448884,v1:172.21.15.86:6837/3782448884] up:boot 2024-05-12T22:06:15.529 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:15 smithi186 ceph-mon[34785]: fsmap 8 up:standby 2024-05-12T22:06:15.529 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:15 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-12T22:06:15.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:15 smithi028 ceph-mon[29847]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-12T22:06:15.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:15 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:15.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:15 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:15.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:15 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:15.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:15 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-12T22:06:15.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:15 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-12T22:06:15.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:15 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:06:15.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:15 smithi028 ceph-mon[29847]: Deploying daemon mds.i on smithi186 2024-05-12T22:06:15.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:15 smithi028 ceph-mon[29847]: mds.? [v2:172.21.15.86:6836/3782448884,v1:172.21.15.86:6837/3782448884] up:boot 2024-05-12T22:06:15.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:15 smithi028 ceph-mon[29847]: fsmap 8 up:standby 2024-05-12T22:06:15.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:15 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-12T22:06:15.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:15 smithi086 ceph-mon[34297]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-12T22:06:15.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:15 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:15.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:15 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:15.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:15 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:15.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:15 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-12T22:06:15.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:15 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-12T22:06:15.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:15 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:06:15.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:15 smithi086 ceph-mon[34297]: Deploying daemon mds.i on smithi186 2024-05-12T22:06:15.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:15 smithi086 ceph-mon[34297]: mds.? [v2:172.21.15.86:6836/3782448884,v1:172.21.15.86:6837/3782448884] up:boot 2024-05-12T22:06:15.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:15 smithi086 ceph-mon[34297]: fsmap 8 up:standby 2024-05-12T22:06:15.743 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:15 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-12T22:06:15.920 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-12T22:06:16.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:16.145+0000 7fbf2ca00640 1 -- 172.21.15.28:0/1699386407 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbf28109860 msgr2=0x7fbf281100f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:16.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:16.145+0000 7fbf2ca00640 1 --2- 172.21.15.28:0/1699386407 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbf28109860 0x7fbf281100f0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fbf20009be0 tx=0x7fbf2002a6d0 comp rx=0 tx=0).stop 2024-05-12T22:06:16.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:16.145+0000 7fbf2ca00640 1 -- 172.21.15.28:0/1699386407 shutdown_connections 2024-05-12T22:06:16.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:16.145+0000 7fbf2ca00640 1 --2- 172.21.15.28:0/1699386407 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbf28109860 0x7fbf281100f0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:16.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:16.145+0000 7fbf2ca00640 1 --2- 172.21.15.28:0/1699386407 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf28108ea0 0x7fbf28109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:16.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:16.145+0000 7fbf2ca00640 1 --2- 172.21.15.28:0/1699386407 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbf28107cb0 0x7fbf281080d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:16.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:16.145+0000 7fbf2ca00640 1 -- 172.21.15.28:0/1699386407 >> 172.21.15.28:0/1699386407 conn(0x7fbf28076c70 msgr2=0x7fbf28079090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:06:16.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:16.145+0000 7fbf2ca00640 1 -- 172.21.15.28:0/1699386407 shutdown_connections 2024-05-12T22:06:16.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:16.145+0000 7fbf2ca00640 1 -- 172.21.15.28:0/1699386407 wait complete. 2024-05-12T22:06:16.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:16.146+0000 7fbf2ca00640 1 Processor -- start 2024-05-12T22:06:16.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:16.146+0000 7fbf2ca00640 1 -- start start 2024-05-12T22:06:16.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:16.146+0000 7fbf2ca00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf28107cb0 0x7fbf2819dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:16.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:16.146+0000 7fbf2ca00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbf28108ea0 0x7fbf2819e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:16.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:16.146+0000 7fbf2ca00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbf28109860 0x7fbf281a5510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:16.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:16.147+0000 7fbf2ca00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbf281138c0 con 0x7fbf28107cb0 2024-05-12T22:06:16.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:16.147+0000 7fbf2ca00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fbf28113740 con 0x7fbf28109860 2024-05-12T22:06:16.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:16.147+0000 7fbf2ca00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fbf28113a40 con 0x7fbf28108ea0 2024-05-12T22:06:16.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:16.147+0000 7fbf27400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf28107cb0 0x7fbf2819dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:16.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:16.147+0000 7fbf1ea00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbf28108ea0 0x7fbf2819e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:16.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:16.147+0000 7fbf27400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf28107cb0 0x7fbf2819dfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58010/0 (socket says 172.21.15.28:58010) 2024-05-12T22:06:16.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:16.147+0000 7fbf27400640 1 -- 172.21.15.28:0/2708009179 learned_addr learned my addr 172.21.15.28:0/2708009179 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:06:16.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:16.147+0000 7fbf27e00640 1 --2- 172.21.15.28:0/2708009179 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbf28109860 0x7fbf281a5510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:16.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:16.147+0000 7fbf1ea00640 1 -- 172.21.15.28:0/2708009179 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbf28109860 msgr2=0x7fbf281a5510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:16.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:16.147+0000 7fbf1ea00640 1 --2- 172.21.15.28:0/2708009179 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbf28109860 0x7fbf281a5510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:16.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:16.147+0000 7fbf1ea00640 1 -- 172.21.15.28:0/2708009179 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf28107cb0 msgr2=0x7fbf2819dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:16.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:16.147+0000 7fbf1ea00640 1 --2- 172.21.15.28:0/2708009179 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf28107cb0 0x7fbf2819dfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:16.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:16.147+0000 7fbf1ea00640 1 -- 172.21.15.28:0/2708009179 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf281a5b80 con 0x7fbf28108ea0 2024-05-12T22:06:16.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:16.148+0000 7fbf1ea00640 1 --2- 172.21.15.28:0/2708009179 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbf28108ea0 0x7fbf2819e4f0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fbf1000e790 tx=0x7fbf1000ec50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:06:16.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:16.148+0000 7fbf24c00640 1 -- 172.21.15.28:0/2708009179 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf1000eea0 con 0x7fbf28108ea0 2024-05-12T22:06:16.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:16.148+0000 7fbf2ca00640 1 -- 172.21.15.28:0/2708009179 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf281a5e70 con 0x7fbf28108ea0 2024-05-12T22:06:16.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:16.148+0000 7fbf24c00640 1 -- 172.21.15.28:0/2708009179 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbf1000ccc0 con 0x7fbf28108ea0 2024-05-12T22:06:16.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:16.148+0000 7fbf24c00640 1 -- 172.21.15.28:0/2708009179 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf10018760 con 0x7fbf28108ea0 2024-05-12T22:06:16.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:16.148+0000 7fbf2ca00640 1 -- 172.21.15.28:0/2708009179 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf281a6380 con 0x7fbf28108ea0 2024-05-12T22:06:16.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:16.149+0000 7fbf2ca00640 1 -- 172.21.15.28:0/2708009179 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf2810efd0 con 0x7fbf28108ea0 2024-05-12T22:06:16.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:16.151+0000 7fbf24c00640 1 -- 172.21.15.28:0/2708009179 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fbf10010430 con 0x7fbf28108ea0 2024-05-12T22:06:16.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:16.152+0000 7fbf24c00640 1 --2- 172.21.15.28:0/2708009179 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fbf0c0a2c10 0x7fbf0c0a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:16.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:16.153+0000 7fbf24c00640 1 -- 172.21.15.28:0/2708009179 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fbf100ced20 con 0x7fbf28108ea0 2024-05-12T22:06:16.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:16.153+0000 7fbf27400640 1 --2- 172.21.15.28:0/2708009179 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fbf0c0a2c10 0x7fbf0c0a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:16.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:16.153+0000 7fbf27400640 1 --2- 172.21.15.28:0/2708009179 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fbf0c0a2c10 0x7fbf0c0a50d0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fbf180058c0 tx=0x7fbf18005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:06:16.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:16.156+0000 7fbf24c00640 1 -- 172.21.15.28:0/2708009179 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fbf1009ba30 con 0x7fbf28108ea0 2024-05-12T22:06:16.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:16.403+0000 7fbf2ca00640 1 -- 172.21.15.28:0/2708009179 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbf280624d0 con 0x7fbf28108ea0 2024-05-12T22:06:16.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:16.404+0000 7fbf24c00640 1 -- 172.21.15.28:0/2708009179 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+7177 (secure 0 0 0) 0x7fbf1009fd30 con 0x7fbf28108ea0 2024-05-12T22:06:16.403 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-12T22:06:16.404 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1844141745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1844141745},{"type":"v1","addr":"172.21.15.186:6833","nonce":1844141745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6833/2960763758","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":2960763758},{"type":"v1","addr":"172.21.15.86:6833","nonce":2960763758}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24470,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/4131134219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":4131134219},{"type":"v1","addr":"172.21.15.28:6835","nonce":4131134219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24481,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/1727869064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1727869064},{"type":"v1","addr":"172.21.15.186:6835","nonce":1727869064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24482,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2938260704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2938260704},{"type":"v1","addr":"172.21.15.28:6837","nonce":2938260704}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24494,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3473700910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3473700910},{"type":"v1","addr":"172.21.15.86:6835","nonce":3473700910}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24499,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3600809235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":3600809235},{"type":"v1","addr":"172.21.15.186:6837","nonce":3600809235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24530,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/3782448884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":3782448884},{"type":"v1","addr":"172.21.15.86:6837","nonce":3782448884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-05-12T22:06:16.404 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 9 2024-05-12T22:06:16.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:16.406+0000 7fbf2ca00640 1 -- 172.21.15.28:0/2708009179 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fbf0c0a2c10 msgr2=0x7fbf0c0a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:16.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:16.406+0000 7fbf2ca00640 1 --2- 172.21.15.28:0/2708009179 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fbf0c0a2c10 0x7fbf0c0a50d0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fbf180058c0 tx=0x7fbf18005f90 comp rx=0 tx=0).stop 2024-05-12T22:06:16.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:16.406+0000 7fbf2ca00640 1 -- 172.21.15.28:0/2708009179 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbf28108ea0 msgr2=0x7fbf2819e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:16.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:16.406+0000 7fbf2ca00640 1 --2- 172.21.15.28:0/2708009179 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbf28108ea0 0x7fbf2819e4f0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fbf1000e790 tx=0x7fbf1000ec50 comp rx=0 tx=0).stop 2024-05-12T22:06:16.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:16.407+0000 7fbf2ca00640 1 -- 172.21.15.28:0/2708009179 shutdown_connections 2024-05-12T22:06:16.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:16.407+0000 7fbf2ca00640 1 --2- 172.21.15.28:0/2708009179 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fbf0c0a2c10 0x7fbf0c0a50d0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:16.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:16.407+0000 7fbf2ca00640 1 --2- 172.21.15.28:0/2708009179 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbf28109860 0x7fbf281a5510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:16.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:16.407+0000 7fbf2ca00640 1 --2- 172.21.15.28:0/2708009179 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbf28108ea0 0x7fbf2819e4f0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:16.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:16.407+0000 7fbf2ca00640 1 --2- 172.21.15.28:0/2708009179 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf28107cb0 0x7fbf2819dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:16.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:16.407+0000 7fbf2ca00640 1 -- 172.21.15.28:0/2708009179 >> 172.21.15.28:0/2708009179 conn(0x7fbf28076c70 msgr2=0x7fbf2810c1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:06:16.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:16.407+0000 7fbf2ca00640 1 -- 172.21.15.28:0/2708009179 shutdown_connections 2024-05-12T22:06:16.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:16.407+0000 7fbf2ca00640 1 -- 172.21.15.28:0/2708009179 wait complete. 2024-05-12T22:06:16.429 INFO:journalctl@ceph.mds.i.smithi186.stdout:May 12 22:06:16 smithi186 systemd[1]: Starting Ceph mds.i for 6b92e7b4-10aa-11ef-bc98-c7b262605968... 2024-05-12T22:06:16.810 INFO:journalctl@ceph.mds.i.smithi186.stdout:May 12 22:06:16 smithi186 podman[65914]: 2024-05-12 22:06:16.507900014 +0000 UTC m=+0.086544100 container create ce0d3c9e0bae4476845de5f52188e87dd13cc52bbb60a0b9773bb701f8736c34 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-i, org.label-schema.schema-version=1.0, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240507, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=squid-f72fecf, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-05-12T22:06:16.811 INFO:journalctl@ceph.mds.i.smithi186.stdout:May 12 22:06:16 smithi186 podman[65914]: 2024-05-12 22:06:16.439696483 +0000 UTC m=+0.018340540 image pull 8252b7944d3b5c188c0ff632647ba6cc4e9a19588ee097b8a3dba27b42779c25 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc 2024-05-12T22:06:16.811 INFO:journalctl@ceph.mds.i.smithi186.stdout:May 12 22:06:16 smithi186 podman[65914]: 2024-05-12 22:06:16.710079274 +0000 UTC m=+0.288723322 container init ce0d3c9e0bae4476845de5f52188e87dd13cc52bbb60a0b9773bb701f8736c34 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-i, maintainer=Guillaume Abrioux , GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240507, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=squid-f72fecf, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-05-12T22:06:16.811 INFO:journalctl@ceph.mds.i.smithi186.stdout:May 12 22:06:16 smithi186 podman[65914]: 2024-05-12 22:06:16.713328513 +0000 UTC m=+0.291972558 container start ce0d3c9e0bae4476845de5f52188e87dd13cc52bbb60a0b9773bb701f8736c34 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-i, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240507, org.label-schema.schema-version=1.0, RELEASE=squid-f72fecf, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux ) 2024-05-12T22:06:16.811 INFO:journalctl@ceph.mds.i.smithi186.stdout:May 12 22:06:16 smithi186 ceph-mds[65934]: -- 172.21.15.186:0/1829307228 <== mon.1 v2:172.21.15.186:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x561f43b81a40 con 0x561f43b54800 2024-05-12T22:06:16.811 INFO:journalctl@ceph.mds.i.smithi186.stdout:May 12 22:06:16 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-i[65930]: starting mds.i at 2024-05-12T22:06:16.811 INFO:journalctl@ceph.mds.i.smithi186.stdout:May 12 22:06:16 smithi186 bash[65914]: ce0d3c9e0bae4476845de5f52188e87dd13cc52bbb60a0b9773bb701f8736c34 2024-05-12T22:06:17.179 INFO:journalctl@ceph.mds.i.smithi186.stdout:May 12 22:06:16 smithi186 systemd[1]: Started Ceph mds.i for 6b92e7b4-10aa-11ef-bc98-c7b262605968. 2024-05-12T22:06:17.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:17 smithi028 ceph-mon[29847]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-12T22:06:17.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:17 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/2708009179' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:06:17.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:17 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:17.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:17 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:17.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:17 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:17.537 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:17 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-12T22:06:17.537 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:17 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-12T22:06:17.537 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:17 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:06:17.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:17 smithi186 ceph-mon[34785]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-12T22:06:17.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:17 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/2708009179' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:06:17.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:17 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:17.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:17 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:17.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:17 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:17.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:17 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-12T22:06:17.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:17 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-12T22:06:17.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:17 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:06:17.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:17 smithi086 ceph-mon[34297]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-12T22:06:17.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:17 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/2708009179' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:06:17.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:17 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:17.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:17 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:17.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:17 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:17.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:17 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-12T22:06:17.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:17 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-12T22:06:17.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:17 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:06:18.427 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-12T22:06:18.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:18 smithi086 ceph-mon[34297]: Deploying daemon mds.g on smithi028 2024-05-12T22:06:18.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:18 smithi086 ceph-mon[34297]: mds.? [v2:172.21.15.186:6838/2399610807,v1:172.21.15.186:6839/2399610807] up:boot 2024-05-12T22:06:18.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:18 smithi086 ceph-mon[34297]: fsmap 9 up:standby 2024-05-12T22:06:18.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:18 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-12T22:06:18.534 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:18 smithi028 ceph-mon[29847]: Deploying daemon mds.g on smithi028 2024-05-12T22:06:18.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:18 smithi028 ceph-mon[29847]: mds.? [v2:172.21.15.186:6838/2399610807,v1:172.21.15.186:6839/2399610807] up:boot 2024-05-12T22:06:18.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:18 smithi028 ceph-mon[29847]: fsmap 9 up:standby 2024-05-12T22:06:18.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:18 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-12T22:06:18.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:18.626+0000 7fd866e00640 1 -- 172.21.15.28:0/886470343 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd860104a60 msgr2=0x7fd860106e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:18.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:18.626+0000 7fd866e00640 1 --2- 172.21.15.28:0/886470343 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd860104a60 0x7fd860106e50 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fd84c009860 tx=0x7fd84c02a340 comp rx=0 tx=0).stop 2024-05-12T22:06:18.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:18.626+0000 7fd866e00640 1 -- 172.21.15.28:0/886470343 shutdown_connections 2024-05-12T22:06:18.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:18.626+0000 7fd866e00640 1 --2- 172.21.15.28:0/886470343 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd860107390 0x7fd8601097e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:18.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:18.626+0000 7fd866e00640 1 --2- 172.21.15.28:0/886470343 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd860104a60 0x7fd860106e50 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:18.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:18.626+0000 7fd866e00640 1 --2- 172.21.15.28:0/886470343 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd860102130 0x7fd860104520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:18.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:18.626+0000 7fd866e00640 1 -- 172.21.15.28:0/886470343 >> 172.21.15.28:0/886470343 conn(0x7fd8600fbb00 msgr2=0x7fd8600fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:06:18.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:18.626+0000 7fd866e00640 1 -- 172.21.15.28:0/886470343 shutdown_connections 2024-05-12T22:06:18.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:18.626+0000 7fd866e00640 1 -- 172.21.15.28:0/886470343 wait complete. 2024-05-12T22:06:18.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:18.627+0000 7fd866e00640 1 Processor -- start 2024-05-12T22:06:18.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:18.627+0000 7fd866e00640 1 -- start start 2024-05-12T22:06:18.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:18.628+0000 7fd866e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd860102130 0x7fd8601979f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:18.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:18.628+0000 7fd866e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd860104a60 0x7fd860197f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:18.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:18.628+0000 7fd866e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd860107390 0x7fd86019efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:18.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:18.628+0000 7fd866e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd86010d320 con 0x7fd860107390 2024-05-12T22:06:18.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:18.628+0000 7fd866e00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fd86010d1a0 con 0x7fd860104a60 2024-05-12T22:06:18.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:18.628+0000 7fd866e00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd86010d4a0 con 0x7fd860102130 2024-05-12T22:06:18.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:18.628+0000 7fd865a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd860102130 0x7fd8601979f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:18.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:18.628+0000 7fd866400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd860107390 0x7fd86019efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:18.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:18.628+0000 7fd865a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd860102130 0x7fd8601979f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:42526/0 (socket says 172.21.15.28:42526) 2024-05-12T22:06:18.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:18.628+0000 7fd865a00640 1 -- 172.21.15.28:0/2826061492 learned_addr learned my addr 172.21.15.28:0/2826061492 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:06:18.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:18.628+0000 7fd865000640 1 --2- 172.21.15.28:0/2826061492 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd860104a60 0x7fd860197f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:18.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:18.629+0000 7fd865a00640 1 -- 172.21.15.28:0/2826061492 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd860104a60 msgr2=0x7fd860197f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:18.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:18.629+0000 7fd865a00640 1 --2- 172.21.15.28:0/2826061492 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd860104a60 0x7fd860197f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:18.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:18.629+0000 7fd865a00640 1 -- 172.21.15.28:0/2826061492 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd860107390 msgr2=0x7fd86019efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:18.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:18.629+0000 7fd865a00640 1 --2- 172.21.15.28:0/2826061492 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd860107390 0x7fd86019efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:18.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:18.629+0000 7fd865a00640 1 -- 172.21.15.28:0/2826061492 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd86019f610 con 0x7fd860102130 2024-05-12T22:06:18.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:18.629+0000 7fd865000640 1 --2- 172.21.15.28:0/2826061492 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd860104a60 0x7fd860197f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:06:18.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:18.629+0000 7fd866400640 1 --2- 172.21.15.28:0/2826061492 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd860107390 0x7fd86019efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:06:18.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:18.629+0000 7fd865a00640 1 --2- 172.21.15.28:0/2826061492 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd860102130 0x7fd8601979f0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fd85400e9f0 tx=0x7fd85400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:06:18.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:18.629+0000 7fd85e000640 1 -- 172.21.15.28:0/2826061492 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd85400cd80 con 0x7fd860102130 2024-05-12T22:06:18.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:18.629+0000 7fd866e00640 1 -- 172.21.15.28:0/2826061492 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd86019f900 con 0x7fd860102130 2024-05-12T22:06:18.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:18.629+0000 7fd85e000640 1 -- 172.21.15.28:0/2826061492 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd854010910 con 0x7fd860102130 2024-05-12T22:06:18.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:18.629+0000 7fd85e000640 1 -- 172.21.15.28:0/2826061492 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8540188f0 con 0x7fd860102130 2024-05-12T22:06:18.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:18.629+0000 7fd866e00640 1 -- 172.21.15.28:0/2826061492 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd86019fe70 con 0x7fd860102130 2024-05-12T22:06:18.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:18.631+0000 7fd866e00640 1 -- 172.21.15.28:0/2826061492 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8600fd5f0 con 0x7fd860102130 2024-05-12T22:06:18.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:18.633+0000 7fd85e000640 1 -- 172.21.15.28:0/2826061492 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fd854010ac0 con 0x7fd860102130 2024-05-12T22:06:18.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:18.634+0000 7fd85e000640 1 --2- 172.21.15.28:0/2826061492 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fd8480a2b90 0x7fd8480a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:18.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:18.634+0000 7fd85e000640 1 -- 172.21.15.28:0/2826061492 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fd8540cefd0 con 0x7fd860102130 2024-05-12T22:06:18.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:18.634+0000 7fd865000640 1 --2- 172.21.15.28:0/2826061492 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fd8480a2b90 0x7fd8480a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:18.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:18.635+0000 7fd865000640 1 --2- 172.21.15.28:0/2826061492 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fd8480a2b90 0x7fd8480a5050 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fd84c009830 tx=0x7fd84c030570 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:06:18.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:18.638+0000 7fd85e000640 1 -- 172.21.15.28:0/2826061492 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fd85409bce0 con 0x7fd860102130 2024-05-12T22:06:18.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:18 smithi186 ceph-mon[34785]: Deploying daemon mds.g on smithi028 2024-05-12T22:06:18.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:18 smithi186 ceph-mon[34785]: mds.? [v2:172.21.15.186:6838/2399610807,v1:172.21.15.186:6839/2399610807] up:boot 2024-05-12T22:06:18.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:18 smithi186 ceph-mon[34785]: fsmap 9 up:standby 2024-05-12T22:06:18.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:18 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-12T22:06:18.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:18.889+0000 7fd866e00640 1 -- 172.21.15.28:0/2826061492 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd86002d1c0 con 0x7fd860102130 2024-05-12T22:06:18.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:18.890+0000 7fd85e000640 1 -- 172.21.15.28:0/2826061492 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+8007 (secure 0 0 0) 0x7fd85409ffe0 con 0x7fd860102130 2024-05-12T22:06:18.890 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-12T22:06:18.890 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1844141745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1844141745},{"type":"v1","addr":"172.21.15.186:6833","nonce":1844141745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6833/2960763758","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":2960763758},{"type":"v1","addr":"172.21.15.86:6833","nonce":2960763758}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24470,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/4131134219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":4131134219},{"type":"v1","addr":"172.21.15.28:6835","nonce":4131134219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24481,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/1727869064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1727869064},{"type":"v1","addr":"172.21.15.186:6835","nonce":1727869064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24482,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2938260704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2938260704},{"type":"v1","addr":"172.21.15.28:6837","nonce":2938260704}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24494,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3473700910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3473700910},{"type":"v1","addr":"172.21.15.86:6835","nonce":3473700910}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24499,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3600809235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":3600809235},{"type":"v1","addr":"172.21.15.186:6837","nonce":3600809235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24520,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2399610807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":2399610807},{"type":"v1","addr":"172.21.15.186:6839","nonce":2399610807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24530,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/3782448884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":3782448884},{"type":"v1","addr":"172.21.15.86:6837","nonce":3782448884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-05-12T22:06:18.890 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 10 2024-05-12T22:06:18.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:18.893+0000 7fd866e00640 1 -- 172.21.15.28:0/2826061492 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fd8480a2b90 msgr2=0x7fd8480a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:18.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:18.893+0000 7fd866e00640 1 --2- 172.21.15.28:0/2826061492 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fd8480a2b90 0x7fd8480a5050 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fd84c009830 tx=0x7fd84c030570 comp rx=0 tx=0).stop 2024-05-12T22:06:18.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:18.893+0000 7fd866e00640 1 -- 172.21.15.28:0/2826061492 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd860102130 msgr2=0x7fd8601979f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:18.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:18.893+0000 7fd866e00640 1 --2- 172.21.15.28:0/2826061492 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd860102130 0x7fd8601979f0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fd85400e9f0 tx=0x7fd85400eeb0 comp rx=0 tx=0).stop 2024-05-12T22:06:18.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:18.893+0000 7fd866e00640 1 -- 172.21.15.28:0/2826061492 shutdown_connections 2024-05-12T22:06:18.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:18.893+0000 7fd866e00640 1 --2- 172.21.15.28:0/2826061492 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fd8480a2b90 0x7fd8480a5050 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:18.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:18.893+0000 7fd866e00640 1 --2- 172.21.15.28:0/2826061492 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd860107390 0x7fd86019efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:18.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:18.893+0000 7fd866e00640 1 --2- 172.21.15.28:0/2826061492 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd860104a60 0x7fd860197f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:18.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:18.893+0000 7fd866e00640 1 --2- 172.21.15.28:0/2826061492 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd860102130 0x7fd8601979f0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:18.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:18.893+0000 7fd866e00640 1 -- 172.21.15.28:0/2826061492 >> 172.21.15.28:0/2826061492 conn(0x7fd8600fbb00 msgr2=0x7fd860102a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:06:18.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:18.893+0000 7fd866e00640 1 -- 172.21.15.28:0/2826061492 shutdown_connections 2024-05-12T22:06:18.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:18.893+0000 7fd866e00640 1 -- 172.21.15.28:0/2826061492 wait complete. 2024-05-12T22:06:19.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:19 smithi028 ceph-mon[29847]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-12T22:06:19.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:19 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/2826061492' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:06:19.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:19 smithi186 ceph-mon[34785]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-12T22:06:19.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:19 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/2826061492' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:06:19.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:19 smithi086 ceph-mon[34297]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-12T22:06:19.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:19 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/2826061492' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:06:20.285 INFO:journalctl@ceph.mds.g.smithi028.stdout:May 12 22:06:20 smithi028 systemd[1]: Starting Ceph mds.g for 6b92e7b4-10aa-11ef-bc98-c7b262605968... 2024-05-12T22:06:20.683 INFO:journalctl@ceph.mds.g.smithi028.stdout:May 12 22:06:20 smithi028 podman[67743]: 2024-05-12 22:06:20.320282046 +0000 UTC m=+0.043282864 image pull 8252b7944d3b5c188c0ff632647ba6cc4e9a19588ee097b8a3dba27b42779c25 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc 2024-05-12T22:06:20.683 INFO:journalctl@ceph.mds.g.smithi028.stdout:May 12 22:06:20 smithi028 podman[67743]: 2024-05-12 22:06:20.419648631 +0000 UTC m=+0.142649475 container create 82c3e6745e1fcc4a660b4369bfb61045ce21d45d1662f2c141c8002762f74fd1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-g, org.label-schema.license=GPLv2, org.label-schema.build-date=20240507, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=squid-f72fecf, maintainer=Guillaume Abrioux , GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True) 2024-05-12T22:06:20.683 INFO:journalctl@ceph.mds.g.smithi028.stdout:May 12 22:06:20 smithi028 podman[67743]: 2024-05-12 22:06:20.681802595 +0000 UTC m=+0.404803445 container init 82c3e6745e1fcc4a660b4369bfb61045ce21d45d1662f2c141c8002762f74fd1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-g, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240507, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, RELEASE=squid-f72fecf, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-05-12T22:06:20.912 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-12T22:06:20.993 INFO:journalctl@ceph.mds.g.smithi028.stdout:May 12 22:06:20 smithi028 podman[67743]: 2024-05-12 22:06:20.692890457 +0000 UTC m=+0.415891285 container start 82c3e6745e1fcc4a660b4369bfb61045ce21d45d1662f2c141c8002762f74fd1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-g, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, RELEASE=squid-f72fecf, org.label-schema.build-date=20240507, org.label-schema.vendor=CentOS) 2024-05-12T22:06:20.994 INFO:journalctl@ceph.mds.g.smithi028.stdout:May 12 22:06:20 smithi028 ceph-mds[67762]: -- 172.21.15.28:0/2843892875 <== mon.0 v2:172.21.15.28:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55f8048bda40 con 0x55f804890c00 2024-05-12T22:06:20.994 INFO:journalctl@ceph.mds.g.smithi028.stdout:May 12 22:06:20 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-g[67758]: starting mds.g at 2024-05-12T22:06:20.994 INFO:journalctl@ceph.mds.g.smithi028.stdout:May 12 22:06:20 smithi028 bash[67743]: 82c3e6745e1fcc4a660b4369bfb61045ce21d45d1662f2c141c8002762f74fd1 2024-05-12T22:06:20.994 INFO:journalctl@ceph.mds.g.smithi028.stdout:May 12 22:06:20 smithi028 systemd[1]: Started Ceph mds.g for 6b92e7b4-10aa-11ef-bc98-c7b262605968. 2024-05-12T22:06:21.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:21.125+0000 7fb6c1e00640 1 -- 172.21.15.28:0/3083057297 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb6bc1099a0 msgr2=0x7fb6bc1102c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:21.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:21.125+0000 7fb6c1e00640 1 --2- 172.21.15.28:0/3083057297 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb6bc1099a0 0x7fb6bc1102c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fb6b4009860 tx=0x7fb6b402a340 comp rx=0 tx=0).stop 2024-05-12T22:06:21.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:21.126+0000 7fb6c1e00640 1 -- 172.21.15.28:0/3083057297 shutdown_connections 2024-05-12T22:06:21.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:21.126+0000 7fb6c1e00640 1 --2- 172.21.15.28:0/3083057297 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb6bc1099a0 0x7fb6bc1102c0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:21.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:21.126+0000 7fb6c1e00640 1 --2- 172.21.15.28:0/3083057297 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6bc108fe0 0x7fb6bc109460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:21.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:21.126+0000 7fb6c1e00640 1 --2- 172.21.15.28:0/3083057297 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb6bc107e90 0x7fb6bc1082b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:21.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:21.126+0000 7fb6c1e00640 1 -- 172.21.15.28:0/3083057297 >> 172.21.15.28:0/3083057297 conn(0x7fb6bc076e50 msgr2=0x7fb6bc079270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:06:21.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:21.126+0000 7fb6c1e00640 1 -- 172.21.15.28:0/3083057297 shutdown_connections 2024-05-12T22:06:21.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:21.126+0000 7fb6c1e00640 1 -- 172.21.15.28:0/3083057297 wait complete. 2024-05-12T22:06:21.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:21.127+0000 7fb6c1e00640 1 Processor -- start 2024-05-12T22:06:21.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:21.127+0000 7fb6c1e00640 1 -- start start 2024-05-12T22:06:21.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:21.127+0000 7fb6c1e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb6bc107e90 0x7fb6bc19e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:21.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:21.127+0000 7fb6c1e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6bc108fe0 0x7fb6bc19e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:21.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:21.127+0000 7fb6c1e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb6bc1099a0 0x7fb6bc1a5730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:21.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:21.127+0000 7fb6c1e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb6bc113b20 con 0x7fb6bc108fe0 2024-05-12T22:06:21.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:21.127+0000 7fb6c1e00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fb6bc1139a0 con 0x7fb6bc107e90 2024-05-12T22:06:21.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:21.127+0000 7fb6c1e00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb6bc113ca0 con 0x7fb6bc1099a0 2024-05-12T22:06:21.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:21.127+0000 7fb6b2000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6bc108fe0 0x7fb6bc19e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:21.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:21.127+0000 7fb6b2000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6bc108fe0 0x7fb6bc19e6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58058/0 (socket says 172.21.15.28:58058) 2024-05-12T22:06:21.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:21.127+0000 7fb6b2000640 1 -- 172.21.15.28:0/1432282983 learned_addr learned my addr 172.21.15.28:0/1432282983 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:06:21.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:21.127+0000 7fb6baa00640 1 --2- 172.21.15.28:0/1432282983 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb6bc107e90 0x7fb6bc19e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:21.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:21.128+0000 7fb6bb400640 1 --2- 172.21.15.28:0/1432282983 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb6bc1099a0 0x7fb6bc1a5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:21.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:21.128+0000 7fb6b2000640 1 -- 172.21.15.28:0/1432282983 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb6bc1099a0 msgr2=0x7fb6bc1a5730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:21.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:21.128+0000 7fb6b2000640 1 --2- 172.21.15.28:0/1432282983 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb6bc1099a0 0x7fb6bc1a5730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:21.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:21.128+0000 7fb6b2000640 1 -- 172.21.15.28:0/1432282983 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb6bc107e90 msgr2=0x7fb6bc19e180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:21.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:21.128+0000 7fb6b2000640 1 --2- 172.21.15.28:0/1432282983 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb6bc107e90 0x7fb6bc19e180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:21.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:21.128+0000 7fb6b2000640 1 -- 172.21.15.28:0/1432282983 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6bc1a5da0 con 0x7fb6bc108fe0 2024-05-12T22:06:21.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:21.128+0000 7fb6baa00640 1 --2- 172.21.15.28:0/1432282983 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb6bc107e90 0x7fb6bc19e180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:06:21.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:21.128+0000 7fb6b2000640 1 --2- 172.21.15.28:0/1432282983 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6bc108fe0 0x7fb6bc19e6c0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fb6a400e9f0 tx=0x7fb6a400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:06:21.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:21.128+0000 7fb6c0a00640 1 -- 172.21.15.28:0/1432282983 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6a400ccc0 con 0x7fb6bc108fe0 2024-05-12T22:06:21.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:21.129+0000 7fb6c0a00640 1 -- 172.21.15.28:0/1432282983 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb6a40035f0 con 0x7fb6bc108fe0 2024-05-12T22:06:21.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:21.129+0000 7fb6c1e00640 1 -- 172.21.15.28:0/1432282983 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6bc1a6090 con 0x7fb6bc108fe0 2024-05-12T22:06:21.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:21.129+0000 7fb6c0a00640 1 -- 172.21.15.28:0/1432282983 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6a400ce60 con 0x7fb6bc108fe0 2024-05-12T22:06:21.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:21.129+0000 7fb6c1e00640 1 -- 172.21.15.28:0/1432282983 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6bc1a65e0 con 0x7fb6bc108fe0 2024-05-12T22:06:21.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:21.130+0000 7fb6c1e00640 1 -- 172.21.15.28:0/1432282983 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6bc10f140 con 0x7fb6bc108fe0 2024-05-12T22:06:21.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:21.133+0000 7fb6c0a00640 1 -- 172.21.15.28:0/1432282983 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fb6a40037a0 con 0x7fb6bc108fe0 2024-05-12T22:06:21.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:21.134+0000 7fb6c0a00640 1 --2- 172.21.15.28:0/1432282983 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fb6a00a2b90 0x7fb6a00a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:21.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:21.134+0000 7fb6c0a00640 1 -- 172.21.15.28:0/1432282983 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fb6a40cee80 con 0x7fb6bc108fe0 2024-05-12T22:06:21.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:21.135+0000 7fb6baa00640 1 --2- 172.21.15.28:0/1432282983 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fb6a00a2b90 0x7fb6a00a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:21.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:21.135+0000 7fb6baa00640 1 --2- 172.21.15.28:0/1432282983 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fb6a00a2b90 0x7fb6a00a5050 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fb6ac005fd0 tx=0x7fb6ac009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:06:21.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:21.140+0000 7fb6c0a00640 1 -- 172.21.15.28:0/1432282983 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fb6a409c1e0 con 0x7fb6bc108fe0 2024-05-12T22:06:21.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:21.392+0000 7fb6c1e00640 1 -- 172.21.15.28:0/1432282983 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb6bc02d1c0 con 0x7fb6bc108fe0 2024-05-12T22:06:21.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:21.394+0000 7fb6c0a00640 1 -- 172.21.15.28:0/1432282983 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8830 (secure 0 0 0) 0x7fb6a409bae0 con 0x7fb6bc108fe0 2024-05-12T22:06:21.393 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-12T22:06:21.393 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1844141745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1844141745},{"type":"v1","addr":"172.21.15.186:6833","nonce":1844141745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6833/2960763758","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":2960763758},{"type":"v1","addr":"172.21.15.86:6833","nonce":2960763758}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24470,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/4131134219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":4131134219},{"type":"v1","addr":"172.21.15.28:6835","nonce":4131134219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24481,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/1727869064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1727869064},{"type":"v1","addr":"172.21.15.186:6835","nonce":1727869064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24482,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2938260704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2938260704},{"type":"v1","addr":"172.21.15.28:6837","nonce":2938260704}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24494,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3473700910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3473700910},{"type":"v1","addr":"172.21.15.86:6835","nonce":3473700910}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24499,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3600809235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":3600809235},{"type":"v1","addr":"172.21.15.186:6837","nonce":3600809235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24520,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2399610807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":2399610807},{"type":"v1","addr":"172.21.15.186:6839","nonce":2399610807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24530,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/3782448884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":3782448884},{"type":"v1","addr":"172.21.15.86:6837","nonce":3782448884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24532,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6839/123892158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":123892158},{"type":"v1","addr":"172.21.15.28:6839","nonce":123892158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[]} 2024-05-12T22:06:21.394 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 11 2024-05-12T22:06:21.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:21.397+0000 7fb6c1e00640 1 -- 172.21.15.28:0/1432282983 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fb6a00a2b90 msgr2=0x7fb6a00a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:21.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:21.397+0000 7fb6c1e00640 1 --2- 172.21.15.28:0/1432282983 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fb6a00a2b90 0x7fb6a00a5050 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fb6ac005fd0 tx=0x7fb6ac009500 comp rx=0 tx=0).stop 2024-05-12T22:06:21.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:21.397+0000 7fb6c1e00640 1 -- 172.21.15.28:0/1432282983 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6bc108fe0 msgr2=0x7fb6bc19e6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:21.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:21.397+0000 7fb6c1e00640 1 --2- 172.21.15.28:0/1432282983 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6bc108fe0 0x7fb6bc19e6c0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fb6a400e9f0 tx=0x7fb6a400eeb0 comp rx=0 tx=0).stop 2024-05-12T22:06:21.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:21.397+0000 7fb6c1e00640 1 -- 172.21.15.28:0/1432282983 shutdown_connections 2024-05-12T22:06:21.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:21.397+0000 7fb6c1e00640 1 --2- 172.21.15.28:0/1432282983 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fb6a00a2b90 0x7fb6a00a5050 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:21.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:21.397+0000 7fb6c1e00640 1 --2- 172.21.15.28:0/1432282983 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb6bc1099a0 0x7fb6bc1a5730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:21.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:21.397+0000 7fb6c1e00640 1 --2- 172.21.15.28:0/1432282983 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6bc108fe0 0x7fb6bc19e6c0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:21.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:21.397+0000 7fb6c1e00640 1 --2- 172.21.15.28:0/1432282983 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb6bc107e90 0x7fb6bc19e180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:21.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:21.397+0000 7fb6c1e00640 1 -- 172.21.15.28:0/1432282983 >> 172.21.15.28:0/1432282983 conn(0x7fb6bc076e50 msgr2=0x7fb6bc10c310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:06:21.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:21.397+0000 7fb6c1e00640 1 -- 172.21.15.28:0/1432282983 shutdown_connections 2024-05-12T22:06:21.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:21.397+0000 7fb6c1e00640 1 -- 172.21.15.28:0/1432282983 wait complete. 2024-05-12T22:06:21.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:21 smithi186 ceph-mon[34785]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-12T22:06:21.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:21 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:21.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:21 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:21.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:21 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:21.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:21 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-12T22:06:21.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:21 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-12T22:06:21.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:21 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:06:21.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:21 smithi086 ceph-mon[34297]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-12T22:06:21.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:21 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:21.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:21 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:21.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:21 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:21.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:21 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-12T22:06:21.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:21 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-12T22:06:21.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:21 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:06:21.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:21 smithi028 ceph-mon[29847]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-12T22:06:21.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:21 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:21.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:21 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:21.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:21 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:21.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:21 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-12T22:06:21.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:21 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-12T22:06:21.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:21 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:06:22.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:22 smithi028 ceph-mon[29847]: Deploying daemon mds.a on smithi028 2024-05-12T22:06:22.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:22 smithi028 ceph-mon[29847]: mds.? [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] up:boot 2024-05-12T22:06:22.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:22 smithi028 ceph-mon[29847]: fsmap 10 up:standby 2024-05-12T22:06:22.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:22 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-12T22:06:22.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:22 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/1432282983' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:06:22.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:22 smithi186 ceph-mon[34785]: Deploying daemon mds.a on smithi028 2024-05-12T22:06:22.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:22 smithi186 ceph-mon[34785]: mds.? [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] up:boot 2024-05-12T22:06:22.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:22 smithi186 ceph-mon[34785]: fsmap 10 up:standby 2024-05-12T22:06:22.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:22 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-12T22:06:22.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:22 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/1432282983' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:06:22.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:22 smithi086 ceph-mon[34297]: Deploying daemon mds.a on smithi028 2024-05-12T22:06:22.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:22 smithi086 ceph-mon[34297]: mds.? [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] up:boot 2024-05-12T22:06:22.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:22 smithi086 ceph-mon[34297]: fsmap 10 up:standby 2024-05-12T22:06:22.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:22 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-12T22:06:22.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:22 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/1432282983' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:06:23.417 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-12T22:06:23.494 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:23 smithi028 ceph-mon[29847]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-12T22:06:23.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:23 smithi186 ceph-mon[34785]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-12T22:06:23.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:23 smithi086 ceph-mon[34297]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-12T22:06:23.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:23.803+0000 7f1c19400640 1 -- 172.21.15.28:0/2889607509 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1c14108ec0 msgr2=0x7f1c14109340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:23.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:23.803+0000 7f1c19400640 1 --2- 172.21.15.28:0/2889607509 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1c14108ec0 0x7f1c14109340 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f1c0c009ca0 tx=0x7f1c0c02a8e0 comp rx=0 tx=0).stop 2024-05-12T22:06:23.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:23.803+0000 7f1c19400640 1 -- 172.21.15.28:0/2889607509 shutdown_connections 2024-05-12T22:06:23.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:23.803+0000 7f1c19400640 1 --2- 172.21.15.28:0/2889607509 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1c14109880 0x7f1c14110110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:23.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:23.803+0000 7f1c19400640 1 --2- 172.21.15.28:0/2889607509 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1c14108ec0 0x7f1c14109340 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:23.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:23.803+0000 7f1c19400640 1 --2- 172.21.15.28:0/2889607509 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c14107cd0 0x7f1c141080f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:23.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:23.803+0000 7f1c19400640 1 -- 172.21.15.28:0/2889607509 >> 172.21.15.28:0/2889607509 conn(0x7f1c14076c70 msgr2=0x7f1c140790b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:06:23.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:23.804+0000 7f1c19400640 1 -- 172.21.15.28:0/2889607509 shutdown_connections 2024-05-12T22:06:23.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:23.804+0000 7f1c19400640 1 -- 172.21.15.28:0/2889607509 wait complete. 2024-05-12T22:06:23.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:23.807+0000 7f1c19400640 1 Processor -- start 2024-05-12T22:06:23.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:23.807+0000 7f1c19400640 1 -- start start 2024-05-12T22:06:23.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:23.807+0000 7f1c19400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c14107cd0 0x7f1c140701c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:23.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:23.808+0000 7f1c19400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1c14108ec0 0x7f1c14070700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:23.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:23.808+0000 7f1c19400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1c14109880 0x7f1c141ae0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:23.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:23.808+0000 7f1c19400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1c141139a0 con 0x7f1c14107cd0 2024-05-12T22:06:23.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:23.808+0000 7f1c19400640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f1c14113820 con 0x7f1c14108ec0 2024-05-12T22:06:23.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:23.808+0000 7f1c19400640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f1c14113b20 con 0x7f1c14109880 2024-05-12T22:06:23.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:23.808+0000 7f1c13e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c14107cd0 0x7f1c140701c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:23.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:23.808+0000 7f1c13400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1c14108ec0 0x7f1c14070700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:23.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:23.809+0000 7f1c18a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1c14109880 0x7f1c141ae0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:23.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:23.809+0000 7f1c18a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1c14109880 0x7f1c141ae0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:42608/0 (socket says 172.21.15.28:42608) 2024-05-12T22:06:23.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:23.809+0000 7f1c18a00640 1 -- 172.21.15.28:0/3531152384 learned_addr learned my addr 172.21.15.28:0/3531152384 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:06:23.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:23.809+0000 7f1c18a00640 1 -- 172.21.15.28:0/3531152384 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1c14108ec0 msgr2=0x7f1c14070700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:23.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:23.809+0000 7f1c18a00640 1 --2- 172.21.15.28:0/3531152384 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1c14108ec0 0x7f1c14070700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:23.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:23.809+0000 7f1c18a00640 1 -- 172.21.15.28:0/3531152384 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c14107cd0 msgr2=0x7f1c140701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:23.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:23.809+0000 7f1c18a00640 1 --2- 172.21.15.28:0/3531152384 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c14107cd0 0x7f1c140701c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:23.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:23.809+0000 7f1c18a00640 1 -- 172.21.15.28:0/3531152384 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c141ae5f0 con 0x7f1c14109880 2024-05-12T22:06:23.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:23.810+0000 7f1c18a00640 1 --2- 172.21.15.28:0/3531152384 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1c14109880 0x7f1c141ae0b0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f1c04009f70 tx=0x7f1c0400edd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:06:23.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:23.810+0000 7f1c10c00640 1 -- 172.21.15.28:0/3531152384 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c0400cd20 con 0x7f1c14109880 2024-05-12T22:06:23.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:23.811+0000 7f1c10c00640 1 -- 172.21.15.28:0/3531152384 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1c04005500 con 0x7f1c14109880 2024-05-12T22:06:23.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:23.811+0000 7f1c19400640 1 -- 172.21.15.28:0/3531152384 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c14114370 con 0x7f1c14109880 2024-05-12T22:06:23.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:23.811+0000 7f1c10c00640 1 -- 172.21.15.28:0/3531152384 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c04032440 con 0x7f1c14109880 2024-05-12T22:06:23.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:23.811+0000 7f1c19400640 1 -- 172.21.15.28:0/3531152384 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c141148c0 con 0x7f1c14109880 2024-05-12T22:06:23.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:23.812+0000 7f1c19400640 1 -- 172.21.15.28:0/3531152384 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c14068570 con 0x7f1c14109880 2024-05-12T22:06:23.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:23.821+0000 7f1c10c00640 1 -- 172.21.15.28:0/3531152384 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f1c04066b30 con 0x7f1c14109880 2024-05-12T22:06:23.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:23.825+0000 7f1c10c00640 1 --2- 172.21.15.28:0/3531152384 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f1bf80a2c60 0x7f1bf80a5120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:23.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:23.825+0000 7f1c10c00640 1 -- 172.21.15.28:0/3531152384 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f1c040f0fa0 con 0x7f1c14109880 2024-05-12T22:06:23.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:23.825+0000 7f1c10c00640 1 -- 172.21.15.28:0/3531152384 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f1c040f14e0 con 0x7f1c14109880 2024-05-12T22:06:23.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:23.825+0000 7f1c13e00640 1 --2- 172.21.15.28:0/3531152384 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f1bf80a2c60 0x7f1bf80a5120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:23.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:23.827+0000 7f1c13e00640 1 --2- 172.21.15.28:0/3531152384 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f1bf80a2c60 0x7f1bf80a5120 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f1c00005db0 tx=0x7f1c00005d40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:06:24.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:24.092+0000 7f1c19400640 1 -- 172.21.15.28:0/3531152384 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1c1410ef90 con 0x7f1c14109880 2024-05-12T22:06:24.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:24.094+0000 7f1c10c00640 1 -- 172.21.15.28:0/3531152384 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8830 (secure 0 0 0) 0x7f1c040be380 con 0x7f1c14109880 2024-05-12T22:06:24.095 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-12T22:06:24.095 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1844141745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1844141745},{"type":"v1","addr":"172.21.15.186:6833","nonce":1844141745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6833/2960763758","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":2960763758},{"type":"v1","addr":"172.21.15.86:6833","nonce":2960763758}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24470,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/4131134219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":4131134219},{"type":"v1","addr":"172.21.15.28:6835","nonce":4131134219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24481,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/1727869064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1727869064},{"type":"v1","addr":"172.21.15.186:6835","nonce":1727869064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24482,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2938260704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2938260704},{"type":"v1","addr":"172.21.15.28:6837","nonce":2938260704}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24494,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3473700910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3473700910},{"type":"v1","addr":"172.21.15.86:6835","nonce":3473700910}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24499,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3600809235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":3600809235},{"type":"v1","addr":"172.21.15.186:6837","nonce":3600809235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24520,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2399610807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":2399610807},{"type":"v1","addr":"172.21.15.186:6839","nonce":2399610807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24530,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/3782448884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":3782448884},{"type":"v1","addr":"172.21.15.86:6837","nonce":3782448884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24532,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6839/123892158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":123892158},{"type":"v1","addr":"172.21.15.28:6839","nonce":123892158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[]} 2024-05-12T22:06:24.095 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 11 2024-05-12T22:06:24.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:24.096+0000 7f1c19400640 1 -- 172.21.15.28:0/3531152384 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f1bf80a2c60 msgr2=0x7f1bf80a5120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:24.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:24.096+0000 7f1c19400640 1 --2- 172.21.15.28:0/3531152384 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f1bf80a2c60 0x7f1bf80a5120 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f1c00005db0 tx=0x7f1c00005d40 comp rx=0 tx=0).stop 2024-05-12T22:06:24.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:24.096+0000 7f1c19400640 1 -- 172.21.15.28:0/3531152384 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1c14109880 msgr2=0x7f1c141ae0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:24.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:24.096+0000 7f1c19400640 1 --2- 172.21.15.28:0/3531152384 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1c14109880 0x7f1c141ae0b0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f1c04009f70 tx=0x7f1c0400edd0 comp rx=0 tx=0).stop 2024-05-12T22:06:24.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:24.097+0000 7f1c19400640 1 -- 172.21.15.28:0/3531152384 shutdown_connections 2024-05-12T22:06:24.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:24.097+0000 7f1c19400640 1 --2- 172.21.15.28:0/3531152384 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f1bf80a2c60 0x7f1bf80a5120 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:24.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:24.097+0000 7f1c19400640 1 --2- 172.21.15.28:0/3531152384 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1c14109880 0x7f1c141ae0b0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:24.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:24.097+0000 7f1c19400640 1 --2- 172.21.15.28:0/3531152384 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1c14108ec0 0x7f1c14070700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:24.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:24.097+0000 7f1c19400640 1 --2- 172.21.15.28:0/3531152384 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c14107cd0 0x7f1c140701c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:24.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:24.097+0000 7f1c19400640 1 -- 172.21.15.28:0/3531152384 >> 172.21.15.28:0/3531152384 conn(0x7f1c14076c70 msgr2=0x7f1c1410c160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:06:24.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:24.097+0000 7f1c19400640 1 -- 172.21.15.28:0/3531152384 shutdown_connections 2024-05-12T22:06:24.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:24.097+0000 7f1c19400640 1 -- 172.21.15.28:0/3531152384 wait complete. 2024-05-12T22:06:24.456 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:24 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/3531152384' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:06:24.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:24 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/3531152384' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:06:24.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:24 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/3531152384' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:06:24.747 INFO:journalctl@ceph.mds.a.smithi028.stdout:May 12 22:06:24 smithi028 podman[68127]: 2024-05-12 22:06:24.490001678 +0000 UTC m=+0.164900499 container create 8d25c544008053ac556b084585611198c114f05ad1d14ca0a241d8e0e03af57a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-a, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240507, RELEASE=squid-f72fecf, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.schema-version=1.0) 2024-05-12T22:06:24.747 INFO:journalctl@ceph.mds.a.smithi028.stdout:May 12 22:06:24 smithi028 podman[68127]: 2024-05-12 22:06:24.744995085 +0000 UTC m=+0.419893967 container init 8d25c544008053ac556b084585611198c114f05ad1d14ca0a241d8e0e03af57a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-a, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-f72fecf, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240507, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-05-12T22:06:25.036 INFO:journalctl@ceph.mds.a.smithi028.stdout:May 12 22:06:24 smithi028 podman[68127]: 2024-05-12 22:06:24.75778338 +0000 UTC m=+0.432682228 container start 8d25c544008053ac556b084585611198c114f05ad1d14ca0a241d8e0e03af57a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-a, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=squid-f72fecf, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240507, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-05-12T22:06:25.036 INFO:journalctl@ceph.mds.a.smithi028.stdout:May 12 22:06:24 smithi028 ceph-mds[68146]: -- 172.21.15.28:0/751260504 <== mon.0 v2:172.21.15.28:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56521c477a40 con 0x56521c44a800 2024-05-12T22:06:25.036 INFO:journalctl@ceph.mds.a.smithi028.stdout:May 12 22:06:24 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-a[68142]: starting mds.a at 2024-05-12T22:06:25.036 INFO:journalctl@ceph.mds.a.smithi028.stdout:May 12 22:06:24 smithi028 bash[68127]: 8d25c544008053ac556b084585611198c114f05ad1d14ca0a241d8e0e03af57a 2024-05-12T22:06:25.036 INFO:journalctl@ceph.mds.a.smithi028.stdout:May 12 22:06:24 smithi028 systemd[1]: Started Ceph mds.a for 6b92e7b4-10aa-11ef-bc98-c7b262605968. 2024-05-12T22:06:25.664 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:25 smithi086 ceph-mon[34297]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-12T22:06:25.664 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:25 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:25.664 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:25 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:25.664 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:25 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:25.664 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:25 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-12T22:06:25.664 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:25 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-12T22:06:25.664 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:25 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:06:25.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:25 smithi186 ceph-mon[34785]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-12T22:06:25.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:25 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:25.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:25 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:25.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:25 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:25.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:25 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-12T22:06:25.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:25 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-12T22:06:25.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:25 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:06:25.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:25 smithi028 ceph-mon[29847]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-12T22:06:25.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:25 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:25.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:25 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:25.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:25 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:25.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:25 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-12T22:06:25.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:25 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-12T22:06:25.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:25 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:06:26.117 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-12T22:06:26.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:26.350+0000 7f78d8400640 1 -- 172.21.15.28:0/3067119928 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f78d0103a90 msgr2=0x7f78d0103eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:26.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:26.350+0000 7f78d8400640 1 --2- 172.21.15.28:0/3067119928 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f78d0103a90 0x7f78d0103eb0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f78c0009900 tx=0x7f78c002a3f0 comp rx=0 tx=0).stop 2024-05-12T22:06:26.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:26.351+0000 7f78d8400640 1 -- 172.21.15.28:0/3067119928 shutdown_connections 2024-05-12T22:06:26.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:26.351+0000 7f78d8400640 1 --2- 172.21.15.28:0/3067119928 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f78d0105640 0x7f78d010bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:26.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:26.351+0000 7f78d8400640 1 --2- 172.21.15.28:0/3067119928 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78d0104c80 0x7f78d0105100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:26.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:26.351+0000 7f78d8400640 1 --2- 172.21.15.28:0/3067119928 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f78d0103a90 0x7f78d0103eb0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:26.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:26.351+0000 7f78d8400640 1 -- 172.21.15.28:0/3067119928 >> 172.21.15.28:0/3067119928 conn(0x7f78d00ff020 msgr2=0x7f78d0101440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:06:26.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:26.351+0000 7f78d8400640 1 -- 172.21.15.28:0/3067119928 shutdown_connections 2024-05-12T22:06:26.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:26.351+0000 7f78d8400640 1 -- 172.21.15.28:0/3067119928 wait complete. 2024-05-12T22:06:26.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:26.352+0000 7f78d8400640 1 Processor -- start 2024-05-12T22:06:26.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:26.352+0000 7f78d8400640 1 -- start start 2024-05-12T22:06:26.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:26.352+0000 7f78d8400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f78d0103a90 0x7f78d0199db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:26.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:26.353+0000 7f78d8400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78d0104c80 0x7f78d019a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:26.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:26.353+0000 7f78d8400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f78d0105640 0x7f78d01a1320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:26.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:26.353+0000 7f78d8400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f78d010f6e0 con 0x7f78d0104c80 2024-05-12T22:06:26.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:26.353+0000 7f78d8400640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f78d010f560 con 0x7f78d0105640 2024-05-12T22:06:26.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:26.353+0000 7f78d8400640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f78d010f860 con 0x7f78d0103a90 2024-05-12T22:06:26.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:26.353+0000 7f78d5a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f78d0103a90 0x7f78d0199db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:26.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:26.353+0000 7f78d5000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78d0104c80 0x7f78d019a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:26.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:26.353+0000 7f78d5a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f78d0103a90 0x7f78d0199db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:45242/0 (socket says 172.21.15.28:45242) 2024-05-12T22:06:26.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:26.353+0000 7f78d5a00640 1 -- 172.21.15.28:0/3975408618 learned_addr learned my addr 172.21.15.28:0/3975408618 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:06:26.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:26.353+0000 7f78d6400640 1 --2- 172.21.15.28:0/3975408618 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f78d0105640 0x7f78d01a1320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:26.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:26.353+0000 7f78d5a00640 1 -- 172.21.15.28:0/3975408618 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f78d0105640 msgr2=0x7f78d01a1320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:26.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:26.353+0000 7f78d5a00640 1 --2- 172.21.15.28:0/3975408618 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f78d0105640 0x7f78d01a1320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:26.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:26.353+0000 7f78d5a00640 1 -- 172.21.15.28:0/3975408618 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78d0104c80 msgr2=0x7f78d019a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:26.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:26.353+0000 7f78d5a00640 1 --2- 172.21.15.28:0/3975408618 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78d0104c80 0x7f78d019a2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:26.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:26.353+0000 7f78d5a00640 1 -- 172.21.15.28:0/3975408618 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78d01a19c0 con 0x7f78d0103a90 2024-05-12T22:06:26.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:26.353+0000 7f78d6400640 1 --2- 172.21.15.28:0/3975408618 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f78d0105640 0x7f78d01a1320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:06:26.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:26.353+0000 7f78d5000640 1 --2- 172.21.15.28:0/3975408618 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78d0104c80 0x7f78d019a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:06:26.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:26.354+0000 7f78d5a00640 1 --2- 172.21.15.28:0/3975408618 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f78d0103a90 0x7f78d0199db0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f78c0031610 tx=0x7f78c0004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:06:26.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:26.354+0000 7f78ce000640 1 -- 172.21.15.28:0/3975408618 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78c003f030 con 0x7f78d0103a90 2024-05-12T22:06:26.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:26.354+0000 7f78ce000640 1 -- 172.21.15.28:0/3975408618 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f78c0003ad0 con 0x7f78d0103a90 2024-05-12T22:06:26.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:26.354+0000 7f78d8400640 1 -- 172.21.15.28:0/3975408618 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78d01a1c50 con 0x7f78d0103a90 2024-05-12T22:06:26.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:26.354+0000 7f78ce000640 1 -- 172.21.15.28:0/3975408618 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78c0035070 con 0x7f78d0103a90 2024-05-12T22:06:26.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:26.354+0000 7f78d8400640 1 -- 172.21.15.28:0/3975408618 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78d01a2140 con 0x7f78d0103a90 2024-05-12T22:06:26.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:26.355+0000 7f78d8400640 1 -- 172.21.15.28:0/3975408618 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78d010adb0 con 0x7f78d0103a90 2024-05-12T22:06:26.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:26.357+0000 7f78ce000640 1 -- 172.21.15.28:0/3975408618 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f78c0042030 con 0x7f78d0103a90 2024-05-12T22:06:26.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:26.359+0000 7f78ce000640 1 --2- 172.21.15.28:0/3975408618 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f78b80a2b90 0x7f78b80a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:26.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:26.359+0000 7f78ce000640 1 -- 172.21.15.28:0/3975408618 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f78c00ef260 con 0x7f78d0103a90 2024-05-12T22:06:26.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:26.359+0000 7f78d5000640 1 --2- 172.21.15.28:0/3975408618 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f78b80a2b90 0x7f78b80a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:26.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:26.360+0000 7f78d5000640 1 --2- 172.21.15.28:0/3975408618 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f78b80a2b90 0x7f78b80a5050 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f78c40099a0 tx=0x7f78c4009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:06:26.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:26.363+0000 7f78ce000640 1 -- 172.21.15.28:0/3975408618 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f78c00bbf70 con 0x7f78d0103a90 2024-05-12T22:06:26.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:26 smithi028 ceph-mon[29847]: Deploying daemon mds.b on smithi086 2024-05-12T22:06:26.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:26 smithi028 ceph-mon[29847]: mds.? [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] up:boot 2024-05-12T22:06:26.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:26 smithi028 ceph-mon[29847]: fsmap 11 up:standby 2024-05-12T22:06:26.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:26 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-12T22:06:26.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:26.609+0000 7f78d8400640 1 -- 172.21.15.28:0/3975408618 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f78d0062460 con 0x7f78d0103a90 2024-05-12T22:06:26.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:26.611+0000 7f78ce000640 1 -- 172.21.15.28:0/3975408618 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9653 (secure 0 0 0) 0x7f78c00c0270 con 0x7f78d0103a90 2024-05-12T22:06:26.612 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-12T22:06:26.612 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1844141745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1844141745},{"type":"v1","addr":"172.21.15.186:6833","nonce":1844141745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6833/2960763758","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":2960763758},{"type":"v1","addr":"172.21.15.86:6833","nonce":2960763758}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24470,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/4131134219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":4131134219},{"type":"v1","addr":"172.21.15.28:6835","nonce":4131134219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24481,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/1727869064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1727869064},{"type":"v1","addr":"172.21.15.186:6835","nonce":1727869064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24482,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2938260704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2938260704},{"type":"v1","addr":"172.21.15.28:6837","nonce":2938260704}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24494,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3473700910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3473700910},{"type":"v1","addr":"172.21.15.86:6835","nonce":3473700910}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24499,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3600809235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":3600809235},{"type":"v1","addr":"172.21.15.186:6837","nonce":3600809235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24520,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2399610807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":2399610807},{"type":"v1","addr":"172.21.15.186:6839","nonce":2399610807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24530,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/3782448884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":3782448884},{"type":"v1","addr":"172.21.15.86:6837","nonce":3782448884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24532,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6839/123892158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":123892158},{"type":"v1","addr":"172.21.15.28:6839","nonce":123892158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24550,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/761183870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":761183870},{"type":"v1","addr":"172.21.15.28:6841","nonce":761183870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[]} 2024-05-12T22:06:26.612 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 12 2024-05-12T22:06:26.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:26.613+0000 7f78d8400640 1 -- 172.21.15.28:0/3975408618 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f78b80a2b90 msgr2=0x7f78b80a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:26.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:26.613+0000 7f78d8400640 1 --2- 172.21.15.28:0/3975408618 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f78b80a2b90 0x7f78b80a5050 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f78c40099a0 tx=0x7f78c4009500 comp rx=0 tx=0).stop 2024-05-12T22:06:26.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:26.613+0000 7f78d8400640 1 -- 172.21.15.28:0/3975408618 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f78d0103a90 msgr2=0x7f78d0199db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:26.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:26.613+0000 7f78d8400640 1 --2- 172.21.15.28:0/3975408618 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f78d0103a90 0x7f78d0199db0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f78c0031610 tx=0x7f78c0004270 comp rx=0 tx=0).stop 2024-05-12T22:06:26.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:26.613+0000 7f78d8400640 1 -- 172.21.15.28:0/3975408618 shutdown_connections 2024-05-12T22:06:26.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:26.613+0000 7f78d8400640 1 --2- 172.21.15.28:0/3975408618 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f78b80a2b90 0x7f78b80a5050 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:26.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:26.613+0000 7f78d8400640 1 --2- 172.21.15.28:0/3975408618 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f78d0105640 0x7f78d01a1320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:26.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:26.613+0000 7f78d8400640 1 --2- 172.21.15.28:0/3975408618 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78d0104c80 0x7f78d019a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:26.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:26.613+0000 7f78d8400640 1 --2- 172.21.15.28:0/3975408618 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f78d0103a90 0x7f78d0199db0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:26.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:26.613+0000 7f78d8400640 1 -- 172.21.15.28:0/3975408618 >> 172.21.15.28:0/3975408618 conn(0x7f78d00ff020 msgr2=0x7f78d0107f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:06:26.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:26.614+0000 7f78d8400640 1 -- 172.21.15.28:0/3975408618 shutdown_connections 2024-05-12T22:06:26.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:26.614+0000 7f78d8400640 1 -- 172.21.15.28:0/3975408618 wait complete. 2024-05-12T22:06:26.642 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:26 smithi086 ceph-mon[34297]: Deploying daemon mds.b on smithi086 2024-05-12T22:06:26.642 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:26 smithi086 ceph-mon[34297]: mds.? [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] up:boot 2024-05-12T22:06:26.643 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:26 smithi086 ceph-mon[34297]: fsmap 11 up:standby 2024-05-12T22:06:26.643 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:26 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-12T22:06:26.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:26 smithi186 ceph-mon[34785]: Deploying daemon mds.b on smithi086 2024-05-12T22:06:26.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:26 smithi186 ceph-mon[34785]: mds.? [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] up:boot 2024-05-12T22:06:26.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:26 smithi186 ceph-mon[34785]: fsmap 11 up:standby 2024-05-12T22:06:26.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:26 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-12T22:06:27.332 INFO:journalctl@ceph.mds.b.smithi086.stdout:May 12 22:06:27 smithi086 podman[64786]: 2024-05-12 22:06:27.137116042 +0000 UTC m=+0.018256305 image pull 8252b7944d3b5c188c0ff632647ba6cc4e9a19588ee097b8a3dba27b42779c25 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc 2024-05-12T22:06:27.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:27 smithi186 ceph-mon[34785]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-12T22:06:27.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:27 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/3975408618' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:06:27.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:27 smithi086 ceph-mon[34297]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-12T22:06:27.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:27 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/3975408618' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:06:27.742 INFO:journalctl@ceph.mds.b.smithi086.stdout:May 12 22:06:27 smithi086 podman[64786]: 2024-05-12 22:06:27.457299945 +0000 UTC m=+0.338440204 container init c6618c6944eb2b460403c7fa3458d4db60d8b0a046be4fac7c80d55607c7fb2c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-b, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240507, CEPH_POINT_RELEASE=, ceph=True, RELEASE=squid-f72fecf, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-05-12T22:06:27.742 INFO:journalctl@ceph.mds.b.smithi086.stdout:May 12 22:06:27 smithi086 podman[64786]: 2024-05-12 22:06:27.460667217 +0000 UTC m=+0.341807476 container start c6618c6944eb2b460403c7fa3458d4db60d8b0a046be4fac7c80d55607c7fb2c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-b, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, GIT_CLEAN=True, ceph=True, RELEASE=squid-f72fecf, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240507, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-12T22:06:27.742 INFO:journalctl@ceph.mds.b.smithi086.stdout:May 12 22:06:27 smithi086 ceph-mds[64806]: -- 172.21.15.86:0/873881812 <== mon.2 v2:172.21.15.86:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5607fd3cda40 con 0x5607fd3a0800 2024-05-12T22:06:27.742 INFO:journalctl@ceph.mds.b.smithi086.stdout:May 12 22:06:27 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-b[64802]: starting mds.b at 2024-05-12T22:06:27.743 INFO:journalctl@ceph.mds.b.smithi086.stdout:May 12 22:06:27 smithi086 bash[64786]: c6618c6944eb2b460403c7fa3458d4db60d8b0a046be4fac7c80d55607c7fb2c 2024-05-12T22:06:27.743 INFO:journalctl@ceph.mds.b.smithi086.stdout:May 12 22:06:27 smithi086 systemd[1]: Started Ceph mds.b for 6b92e7b4-10aa-11ef-bc98-c7b262605968. 2024-05-12T22:06:27.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:27 smithi028 ceph-mon[29847]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-12T22:06:27.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:27 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/3975408618' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:06:28.635 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-12T22:06:28.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:28.836+0000 7ffa7e800640 1 -- 172.21.15.28:0/368115314 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa78102130 msgr2=0x7ffa78104520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:28.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:28.836+0000 7ffa7e800640 1 --2- 172.21.15.28:0/368115314 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa78102130 0x7ffa78104520 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7ffa6c009860 tx=0x7ffa6c02a340 comp rx=0 tx=0).stop 2024-05-12T22:06:28.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:28.837+0000 7ffa7e800640 1 -- 172.21.15.28:0/368115314 shutdown_connections 2024-05-12T22:06:28.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:28.837+0000 7ffa7e800640 1 --2- 172.21.15.28:0/368115314 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffa78107390 0x7ffa781097e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:28.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:28.837+0000 7ffa7e800640 1 --2- 172.21.15.28:0/368115314 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffa78104a60 0x7ffa78106e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:28.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:28.837+0000 7ffa7e800640 1 --2- 172.21.15.28:0/368115314 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa78102130 0x7ffa78104520 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:28.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:28.837+0000 7ffa7e800640 1 -- 172.21.15.28:0/368115314 >> 172.21.15.28:0/368115314 conn(0x7ffa780fbb00 msgr2=0x7ffa780fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:06:28.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:28.837+0000 7ffa7e800640 1 -- 172.21.15.28:0/368115314 shutdown_connections 2024-05-12T22:06:28.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:28.837+0000 7ffa7e800640 1 -- 172.21.15.28:0/368115314 wait complete. 2024-05-12T22:06:28.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:28.838+0000 7ffa7e800640 1 Processor -- start 2024-05-12T22:06:28.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:28.838+0000 7ffa7e800640 1 -- start start 2024-05-12T22:06:28.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:28.838+0000 7ffa7e800640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffa78102130 0x7ffa78197a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:28.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:28.838+0000 7ffa7e800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa78104a60 0x7ffa78197f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:28.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:28.838+0000 7ffa7e800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffa78107390 0x7ffa7819ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:28.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:28.838+0000 7ffa7e800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ffa7810d360 con 0x7ffa78104a60 2024-05-12T22:06:28.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:28.838+0000 7ffa7e800640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7ffa7810d1e0 con 0x7ffa78107390 2024-05-12T22:06:28.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:28.838+0000 7ffa7e800640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ffa7810d4e0 con 0x7ffa78102130 2024-05-12T22:06:28.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:28.839+0000 7ffa7d400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffa78102130 0x7ffa78197a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:28.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:28.839+0000 7ffa7de00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffa78107390 0x7ffa7819ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:28.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:28.839+0000 7ffa7d400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffa78102130 0x7ffa78197a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:45266/0 (socket says 172.21.15.28:45266) 2024-05-12T22:06:28.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:28.839+0000 7ffa7d400640 1 -- 172.21.15.28:0/1325054623 learned_addr learned my addr 172.21.15.28:0/1325054623 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:06:28.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:28.839+0000 7ffa7ca00640 1 --2- 172.21.15.28:0/1325054623 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa78104a60 0x7ffa78197f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:28.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:28.839+0000 7ffa7d400640 1 -- 172.21.15.28:0/1325054623 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffa78107390 msgr2=0x7ffa7819ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:28.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:28.839+0000 7ffa7d400640 1 --2- 172.21.15.28:0/1325054623 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffa78107390 0x7ffa7819ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:28.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:28.839+0000 7ffa7d400640 1 -- 172.21.15.28:0/1325054623 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa78104a60 msgr2=0x7ffa78197f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:28.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:28.839+0000 7ffa7d400640 1 --2- 172.21.15.28:0/1325054623 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa78104a60 0x7ffa78197f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:28.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:28.839+0000 7ffa7d400640 1 -- 172.21.15.28:0/1325054623 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa78100050 con 0x7ffa78102130 2024-05-12T22:06:28.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:28.839+0000 7ffa7d400640 1 --2- 172.21.15.28:0/1325054623 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffa78102130 0x7ffa78197a00 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7ffa6c035040 tx=0x7ffa6c033d90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:06:28.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:28.840+0000 7ffa76000640 1 -- 172.21.15.28:0/1325054623 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa6c037040 con 0x7ffa78102130 2024-05-12T22:06:28.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:28.840+0000 7ffa76000640 1 -- 172.21.15.28:0/1325054623 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa6c030850 con 0x7ffa78102130 2024-05-12T22:06:28.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:28.840+0000 7ffa76000640 1 -- 172.21.15.28:0/1325054623 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa6c004920 con 0x7ffa78102130 2024-05-12T22:06:28.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:28.840+0000 7ffa7e800640 1 -- 172.21.15.28:0/1325054623 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa781002e0 con 0x7ffa78102130 2024-05-12T22:06:28.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:28.840+0000 7ffa7e800640 1 -- 172.21.15.28:0/1325054623 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa78100820 con 0x7ffa78102130 2024-05-12T22:06:28.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:28.841+0000 7ffa7e800640 1 -- 172.21.15.28:0/1325054623 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa78101c00 con 0x7ffa78102130 2024-05-12T22:06:28.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:28.843+0000 7ffa76000640 1 -- 172.21.15.28:0/1325054623 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7ffa6c004050 con 0x7ffa78102130 2024-05-12T22:06:28.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:28.844+0000 7ffa76000640 1 --2- 172.21.15.28:0/1325054623 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7ffa600a2c10 0x7ffa600a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:28.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:28.845+0000 7ffa76000640 1 -- 172.21.15.28:0/1325054623 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7ffa6c0eeac0 con 0x7ffa78102130 2024-05-12T22:06:28.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:28.845+0000 7ffa7ca00640 1 --2- 172.21.15.28:0/1325054623 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7ffa600a2c10 0x7ffa600a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:28.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:28.845+0000 7ffa7ca00640 1 --2- 172.21.15.28:0/1325054623 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7ffa600a2c10 0x7ffa600a50d0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7ffa78198ec0 tx=0x7ffa64009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:06:28.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:28.849+0000 7ffa76000640 1 -- 172.21.15.28:0/1325054623 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7ffa6c0bbe20 con 0x7ffa78102130 2024-05-12T22:06:28.900 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:28 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:28.901 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:28 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:28.901 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:28 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:28.901 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:28 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:28.901 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:28 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:28.901 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:28 smithi186 ceph-mon[34785]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-12T22:06:28.901 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:28 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:06:28.901 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:28 smithi186 ceph-mon[34785]: mds.? [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] up:boot 2024-05-12T22:06:28.901 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:28 smithi186 ceph-mon[34785]: fsmap 12 up:standby 2024-05-12T22:06:28.901 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:28 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-12T22:06:28.923 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:28 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:28.923 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:28 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:28.924 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:28 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:28.924 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:28 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:28.924 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:28 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:28.924 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:28 smithi086 ceph-mon[34297]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-12T22:06:28.924 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:28 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:06:28.924 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:28 smithi086 ceph-mon[34297]: mds.? [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] up:boot 2024-05-12T22:06:28.924 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:28 smithi086 ceph-mon[34297]: fsmap 12 up:standby 2024-05-12T22:06:28.924 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:28 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-12T22:06:29.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:28 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:29.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:28 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:29.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:28 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:29.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:28 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:29.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:28 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:29.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:28 smithi028 ceph-mon[29847]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-12T22:06:29.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:28 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:06:29.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:28 smithi028 ceph-mon[29847]: mds.? [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] up:boot 2024-05-12T22:06:29.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:28 smithi028 ceph-mon[29847]: fsmap 12 up:standby 2024-05-12T22:06:29.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:28 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-12T22:06:29.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:29.150+0000 7ffa7e800640 1 -- 172.21.15.28:0/1325054623 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ffa78101e20 con 0x7ffa78102130 2024-05-12T22:06:29.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:29.151+0000 7ffa76000640 1 -- 172.21.15.28:0/1325054623 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+10479 (secure 0 0 0) 0x7ffa6c0bb720 con 0x7ffa78102130 2024-05-12T22:06:29.152 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-12T22:06:29.152 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1844141745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1844141745},{"type":"v1","addr":"172.21.15.186:6833","nonce":1844141745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6833/2960763758","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":2960763758},{"type":"v1","addr":"172.21.15.86:6833","nonce":2960763758}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24470,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/4131134219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":4131134219},{"type":"v1","addr":"172.21.15.28:6835","nonce":4131134219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24481,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/1727869064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1727869064},{"type":"v1","addr":"172.21.15.186:6835","nonce":1727869064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24482,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2938260704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2938260704},{"type":"v1","addr":"172.21.15.28:6837","nonce":2938260704}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24494,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3473700910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3473700910},{"type":"v1","addr":"172.21.15.86:6835","nonce":3473700910}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24499,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3600809235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":3600809235},{"type":"v1","addr":"172.21.15.186:6837","nonce":3600809235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24520,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2399610807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":2399610807},{"type":"v1","addr":"172.21.15.186:6839","nonce":2399610807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24530,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/3782448884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":3782448884},{"type":"v1","addr":"172.21.15.86:6837","nonce":3782448884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24532,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6839/123892158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":123892158},{"type":"v1","addr":"172.21.15.28:6839","nonce":123892158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24550,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/761183870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":761183870},{"type":"v1","addr":"172.21.15.28:6841","nonce":761183870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24581,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6839/1251999130","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6838","nonce":1251999130},{"type":"v1","addr":"172.21.15.86:6839","nonce":1251999130}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[]} 2024-05-12T22:06:29.153 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 13 2024-05-12T22:06:29.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:29.154+0000 7ffa7e800640 1 -- 172.21.15.28:0/1325054623 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7ffa600a2c10 msgr2=0x7ffa600a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:29.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:29.154+0000 7ffa7e800640 1 --2- 172.21.15.28:0/1325054623 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7ffa600a2c10 0x7ffa600a50d0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7ffa78198ec0 tx=0x7ffa64009260 comp rx=0 tx=0).stop 2024-05-12T22:06:29.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:29.154+0000 7ffa7e800640 1 -- 172.21.15.28:0/1325054623 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffa78102130 msgr2=0x7ffa78197a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:29.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:29.154+0000 7ffa7e800640 1 --2- 172.21.15.28:0/1325054623 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffa78102130 0x7ffa78197a00 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7ffa6c035040 tx=0x7ffa6c033d90 comp rx=0 tx=0).stop 2024-05-12T22:06:29.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:29.154+0000 7ffa7e800640 1 -- 172.21.15.28:0/1325054623 shutdown_connections 2024-05-12T22:06:29.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:29.154+0000 7ffa7e800640 1 --2- 172.21.15.28:0/1325054623 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7ffa600a2c10 0x7ffa600a50d0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:29.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:29.154+0000 7ffa7e800640 1 --2- 172.21.15.28:0/1325054623 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffa78107390 0x7ffa7819ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:29.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:29.154+0000 7ffa7e800640 1 --2- 172.21.15.28:0/1325054623 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa78104a60 0x7ffa78197f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:29.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:29.154+0000 7ffa7e800640 1 --2- 172.21.15.28:0/1325054623 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffa78102130 0x7ffa78197a00 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:29.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:29.154+0000 7ffa7e800640 1 -- 172.21.15.28:0/1325054623 >> 172.21.15.28:0/1325054623 conn(0x7ffa780fbb00 msgr2=0x7ffa78102b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:06:29.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:29.154+0000 7ffa7e800640 1 -- 172.21.15.28:0/1325054623 shutdown_connections 2024-05-12T22:06:29.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:29.154+0000 7ffa7e800640 1 -- 172.21.15.28:0/1325054623 wait complete. 2024-05-12T22:06:29.173 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-12T22:06:29.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:29.362+0000 7fa71ca00640 1 -- 172.21.15.28:0/4143053654 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa718103a90 msgr2=0x7fa718103eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:29.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:29.362+0000 7fa71ca00640 1 --2- 172.21.15.28:0/4143053654 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa718103a90 0x7fa718103eb0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fa708009900 tx=0x7fa70802a3f0 comp rx=0 tx=0).stop 2024-05-12T22:06:29.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:29.362+0000 7fa71ca00640 1 -- 172.21.15.28:0/4143053654 shutdown_connections 2024-05-12T22:06:29.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:29.362+0000 7fa71ca00640 1 --2- 172.21.15.28:0/4143053654 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa718105640 0x7fa71810bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:29.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:29.362+0000 7fa71ca00640 1 --2- 172.21.15.28:0/4143053654 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa718104c80 0x7fa718105100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:29.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:29.362+0000 7fa71ca00640 1 --2- 172.21.15.28:0/4143053654 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa718103a90 0x7fa718103eb0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:29.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:29.362+0000 7fa71ca00640 1 -- 172.21.15.28:0/4143053654 >> 172.21.15.28:0/4143053654 conn(0x7fa7180ff020 msgr2=0x7fa718101440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:06:29.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:29.362+0000 7fa71ca00640 1 -- 172.21.15.28:0/4143053654 shutdown_connections 2024-05-12T22:06:29.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:29.362+0000 7fa71ca00640 1 -- 172.21.15.28:0/4143053654 wait complete. 2024-05-12T22:06:29.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:29.363+0000 7fa71ca00640 1 Processor -- start 2024-05-12T22:06:29.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:29.363+0000 7fa71ca00640 1 -- start start 2024-05-12T22:06:29.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:29.363+0000 7fa71ca00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa718103a90 0x7fa718199dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:29.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:29.363+0000 7fa71ca00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa718104c80 0x7fa71819a300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:29.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:29.363+0000 7fa71ca00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa718105640 0x7fa7181a1370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:29.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:29.363+0000 7fa71ca00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa71810f750 con 0x7fa718104c80 2024-05-12T22:06:29.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:29.363+0000 7fa71ca00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fa71810f5d0 con 0x7fa718103a90 2024-05-12T22:06:29.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:29.363+0000 7fa71ca00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa71810f8d0 con 0x7fa718105640 2024-05-12T22:06:29.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:29.364+0000 7fa715400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa718104c80 0x7fa71819a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:29.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:29.364+0000 7fa715e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa718103a90 0x7fa718199dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:29.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:29.364+0000 7fa716800640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa718105640 0x7fa7181a1370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:29.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:29.364+0000 7fa715400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa718104c80 0x7fa71819a300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55336/0 (socket says 172.21.15.28:55336) 2024-05-12T22:06:29.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:29.364+0000 7fa715400640 1 -- 172.21.15.28:0/122701065 learned_addr learned my addr 172.21.15.28:0/122701065 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:06:29.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:29.364+0000 7fa715400640 1 -- 172.21.15.28:0/122701065 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa718105640 msgr2=0x7fa7181a1370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:29.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:29.364+0000 7fa715400640 1 --2- 172.21.15.28:0/122701065 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa718105640 0x7fa7181a1370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:29.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:29.364+0000 7fa715400640 1 -- 172.21.15.28:0/122701065 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa718103a90 msgr2=0x7fa718199dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:29.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:29.364+0000 7fa715400640 1 --2- 172.21.15.28:0/122701065 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa718103a90 0x7fa718199dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:29.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:29.364+0000 7fa715400640 1 -- 172.21.15.28:0/122701065 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7181a19e0 con 0x7fa718104c80 2024-05-12T22:06:29.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:29.364+0000 7fa715e00640 1 --2- 172.21.15.28:0/122701065 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa718103a90 0x7fa718199dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:06:29.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:29.364+0000 7fa715400640 1 --2- 172.21.15.28:0/122701065 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa718104c80 0x7fa71819a300 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fa70400e9f0 tx=0x7fa70400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:06:29.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:29.365+0000 7fa70ea00640 1 -- 172.21.15.28:0/122701065 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa70400cd80 con 0x7fa718104c80 2024-05-12T22:06:29.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:29.365+0000 7fa70ea00640 1 -- 172.21.15.28:0/122701065 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa704005500 con 0x7fa718104c80 2024-05-12T22:06:29.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:29.365+0000 7fa70ea00640 1 -- 172.21.15.28:0/122701065 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa704010690 con 0x7fa718104c80 2024-05-12T22:06:29.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:29.365+0000 7fa71ca00640 1 -- 172.21.15.28:0/122701065 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7181a1cd0 con 0x7fa718104c80 2024-05-12T22:06:29.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:29.365+0000 7fa71ca00640 1 -- 172.21.15.28:0/122701065 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7181a21d0 con 0x7fa718104c80 2024-05-12T22:06:29.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:29.366+0000 7fa71ca00640 1 -- 172.21.15.28:0/122701065 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa71810adb0 con 0x7fa718104c80 2024-05-12T22:06:29.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:29.368+0000 7fa70ea00640 1 -- 172.21.15.28:0/122701065 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fa7040046c0 con 0x7fa718104c80 2024-05-12T22:06:29.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:29.370+0000 7fa70ea00640 1 --2- 172.21.15.28:0/122701065 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fa6fc0a2b90 0x7fa6fc0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:29.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:29.370+0000 7fa70ea00640 1 -- 172.21.15.28:0/122701065 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fa7040cf060 con 0x7fa718104c80 2024-05-12T22:06:29.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:29.371+0000 7fa715e00640 1 --2- 172.21.15.28:0/122701065 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fa6fc0a2b90 0x7fa6fc0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:29.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:29.371+0000 7fa715e00640 1 --2- 172.21.15.28:0/122701065 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fa6fc0a2b90 0x7fa6fc0a5050 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fa70802a9e0 tx=0x7fa7080315b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:06:29.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:29.374+0000 7fa70ea00640 1 -- 172.21.15.28:0/122701065 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fa70409c3c0 con 0x7fa718104c80 2024-05-12T22:06:29.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:29.583+0000 7fa71ca00640 1 -- 172.21.15.28:0/122701065 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa718062430 con 0x7fa718104c80 2024-05-12T22:06:29.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:29.584+0000 7fa70ea00640 1 -- 172.21.15.28:0/122701065 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) ==== 74+0+18397 (secure 0 0 0) 0x7fa70409bcc0 con 0x7fa718104c80 2024-05-12T22:06:29.585 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-12T22:06:29.585 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":82,"fsid":"6b92e7b4-10aa-11ef-bc98-c7b262605968","created":"2024-05-12T21:57:37.276782+0000","modified":"2024-05-12T22:05:38.939521+0000","last_up_change":"2024-05-12T22:05:37.929562+0000","last_in_change":"2024-05-12T22:05:18.750768+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-12T22:01:32.292826+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6992ce0a-844b-40e3-8ed1-181c8474dccf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":906003882},{"type":"v1","addr":"172.21.15.28:6803","nonce":906003882}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":906003882},{"type":"v1","addr":"172.21.15.28:6805","nonce":906003882}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":906003882},{"type":"v1","addr":"172.21.15.28:6809","nonce":906003882}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":906003882},{"type":"v1","addr":"172.21.15.28:6807","nonce":906003882}]},"public_addr":"172.21.15.28:6803/906003882","cluster_addr":"172.21.15.28:6805/906003882","heartbeat_back_addr":"172.21.15.28:6809/906003882","heartbeat_front_addr":"172.21.15.28:6807/906003882","state":["exists","up"]},{"osd":1,"uuid":"ced20c96-c481-4d48-90d1-6b43228959ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":3354141819},{"type":"v1","addr":"172.21.15.86:6801","nonce":3354141819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":3354141819},{"type":"v1","addr":"172.21.15.86:6803","nonce":3354141819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":3354141819},{"type":"v1","addr":"172.21.15.86:6807","nonce":3354141819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":3354141819},{"type":"v1","addr":"172.21.15.86:6805","nonce":3354141819}]},"public_addr":"172.21.15.86:6801/3354141819","cluster_addr":"172.21.15.86:6803/3354141819","heartbeat_back_addr":"172.21.15.86:6807/3354141819","heartbeat_front_addr":"172.21.15.86:6805/3354141819","state":["exists","up"]},{"osd":2,"uuid":"54a2f221-fa65-4795-9f5e-e660cc7c0098","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":906459365},{"type":"v1","addr":"172.21.15.186:6801","nonce":906459365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":906459365},{"type":"v1","addr":"172.21.15.186:6803","nonce":906459365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":906459365},{"type":"v1","addr":"172.21.15.186:6807","nonce":906459365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":906459365},{"type":"v1","addr":"172.21.15.186:6805","nonce":906459365}]},"public_addr":"172.21.15.186:6801/906459365","cluster_addr":"172.21.15.186:6803/906459365","heartbeat_back_addr":"172.21.15.186:6807/906459365","heartbeat_front_addr":"172.21.15.186:6805/906459365","state":["exists","up"]},{"osd":3,"uuid":"86836737-ac9a-487b-93f2-7fcd3c406960","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":4182884436},{"type":"v1","addr":"172.21.15.28:6811","nonce":4182884436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":4182884436},{"type":"v1","addr":"172.21.15.28:6813","nonce":4182884436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":4182884436},{"type":"v1","addr":"172.21.15.28:6817","nonce":4182884436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":4182884436},{"type":"v1","addr":"172.21.15.28:6815","nonce":4182884436}]},"public_addr":"172.21.15.28:6811/4182884436","cluster_addr":"172.21.15.28:6813/4182884436","heartbeat_back_addr":"172.21.15.28:6817/4182884436","heartbeat_front_addr":"172.21.15.28:6815/4182884436","state":["exists","up"]},{"osd":4,"uuid":"d162ebcd-83c0-4fd5-9925-28b2cecea935","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2596998708},{"type":"v1","addr":"172.21.15.86:6809","nonce":2596998708}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2596998708},{"type":"v1","addr":"172.21.15.86:6811","nonce":2596998708}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2596998708},{"type":"v1","addr":"172.21.15.86:6815","nonce":2596998708}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2596998708},{"type":"v1","addr":"172.21.15.86:6813","nonce":2596998708}]},"public_addr":"172.21.15.86:6809/2596998708","cluster_addr":"172.21.15.86:6811/2596998708","heartbeat_back_addr":"172.21.15.86:6815/2596998708","heartbeat_front_addr":"172.21.15.86:6813/2596998708","state":["exists","up"]},{"osd":5,"uuid":"c794336a-a105-425c-b6c7-a32b90c05bb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":1944205756},{"type":"v1","addr":"172.21.15.186:6809","nonce":1944205756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":1944205756},{"type":"v1","addr":"172.21.15.186:6811","nonce":1944205756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":1944205756},{"type":"v1","addr":"172.21.15.186:6815","nonce":1944205756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":1944205756},{"type":"v1","addr":"172.21.15.186:6813","nonce":1944205756}]},"public_addr":"172.21.15.186:6809/1944205756","cluster_addr":"172.21.15.186:6811/1944205756","heartbeat_back_addr":"172.21.15.186:6815/1944205756","heartbeat_front_addr":"172.21.15.186:6813/1944205756","state":["exists","up"]},{"osd":6,"uuid":"516932ac-af00-45b7-ad3c-d6dec7db442f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":1642308925},{"type":"v1","addr":"172.21.15.28:6819","nonce":1642308925}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":1642308925},{"type":"v1","addr":"172.21.15.28:6821","nonce":1642308925}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1642308925},{"type":"v1","addr":"172.21.15.28:6825","nonce":1642308925}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":1642308925},{"type":"v1","addr":"172.21.15.28:6823","nonce":1642308925}]},"public_addr":"172.21.15.28:6819/1642308925","cluster_addr":"172.21.15.28:6821/1642308925","heartbeat_back_addr":"172.21.15.28:6825/1642308925","heartbeat_front_addr":"172.21.15.28:6823/1642308925","state":["exists","up"]},{"osd":7,"uuid":"ca7a059f-0414-438e-b2d1-a10f348036be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":726062711},{"type":"v1","addr":"172.21.15.86:6817","nonce":726062711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":726062711},{"type":"v1","addr":"172.21.15.86:6819","nonce":726062711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":726062711},{"type":"v1","addr":"172.21.15.86:6823","nonce":726062711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":726062711},{"type":"v1","addr":"172.21.15.86:6821","nonce":726062711}]},"public_addr":"172.21.15.86:6817/726062711","cluster_addr":"172.21.15.86:6819/726062711","heartbeat_back_addr":"172.21.15.86:6823/726062711","heartbeat_front_addr":"172.21.15.86:6821/726062711","state":["exists","up"]},{"osd":8,"uuid":"92e3f41f-aebd-422e-a99e-555f7ba4aa53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":3007687277},{"type":"v1","addr":"172.21.15.186:6817","nonce":3007687277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":3007687277},{"type":"v1","addr":"172.21.15.186:6819","nonce":3007687277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":3007687277},{"type":"v1","addr":"172.21.15.186:6823","nonce":3007687277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":3007687277},{"type":"v1","addr":"172.21.15.186:6821","nonce":3007687277}]},"public_addr":"172.21.15.186:6817/3007687277","cluster_addr":"172.21.15.186:6819/3007687277","heartbeat_back_addr":"172.21.15.186:6823/3007687277","heartbeat_front_addr":"172.21.15.186:6821/3007687277","state":["exists","up"]},{"osd":9,"uuid":"38ff7d9d-ee85-404c-8c59-adc327e1ccfa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":136220472},{"type":"v1","addr":"172.21.15.28:6827","nonce":136220472}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":136220472},{"type":"v1","addr":"172.21.15.28:6829","nonce":136220472}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":136220472},{"type":"v1","addr":"172.21.15.28:6833","nonce":136220472}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":136220472},{"type":"v1","addr":"172.21.15.28:6831","nonce":136220472}]},"public_addr":"172.21.15.28:6827/136220472","cluster_addr":"172.21.15.28:6829/136220472","heartbeat_back_addr":"172.21.15.28:6833/136220472","heartbeat_front_addr":"172.21.15.28:6831/136220472","state":["exists","up"]},{"osd":10,"uuid":"5e9ff657-afa8-4bab-8491-f209038392b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1189968310},{"type":"v1","addr":"172.21.15.86:6825","nonce":1189968310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":1189968310},{"type":"v1","addr":"172.21.15.86:6827","nonce":1189968310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":1189968310},{"type":"v1","addr":"172.21.15.86:6831","nonce":1189968310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":1189968310},{"type":"v1","addr":"172.21.15.86:6829","nonce":1189968310}]},"public_addr":"172.21.15.86:6825/1189968310","cluster_addr":"172.21.15.86:6827/1189968310","heartbeat_back_addr":"172.21.15.86:6831/1189968310","heartbeat_front_addr":"172.21.15.86:6829/1189968310","state":["exists","up"]},{"osd":11,"uuid":"f112e3c6-65b6-4253-a966-62001db7d069","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3801668110},{"type":"v1","addr":"172.21.15.186:6825","nonce":3801668110}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3801668110},{"type":"v1","addr":"172.21.15.186:6827","nonce":3801668110}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3801668110},{"type":"v1","addr":"172.21.15.186:6831","nonce":3801668110}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3801668110},{"type":"v1","addr":"172.21.15.186:6829","nonce":3801668110}]},"public_addr":"172.21.15.186:6825/3801668110","cluster_addr":"172.21.15.186:6827/3801668110","heartbeat_back_addr":"172.21.15.186:6831/3801668110","heartbeat_front_addr":"172.21.15.186:6829/3801668110","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:00:40.483623+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:01:05.284385+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:01:28.645716+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:02:01.295092+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:02:26.486601+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:02:50.807478+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:03:24.756169+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:03:49.975173+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:04:13.772233+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:04:46.758555+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:05:12.331685+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:05:36.462548+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/1177009533":"2024-05-13T21:58:45.550459+0000","172.21.15.28:6800/3760499540":"2024-05-13T21:58:45.550459+0000","172.21.15.28:6801/3783742728":"2024-05-13T21:58:10.421564+0000","172.21.15.28:6800/3783742728":"2024-05-13T21:58:10.421564+0000","172.21.15.28:0/2229156260":"2024-05-13T21:58:45.550459+0000","172.21.15.28:0/1202140963":"2024-05-13T21:58:10.421564+0000","172.21.15.28:0/2924223784":"2024-05-13T21:58:45.550459+0000","172.21.15.28:6801/3760499540":"2024-05-13T21:58:45.550459+0000","172.21.15.28:0/1240027206":"2024-05-13T21:58:10.421564+0000","172.21.15.28:0/171417728":"2024-05-13T21:58:10.421564+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-12T22:06:29.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:29.587+0000 7fa71ca00640 1 -- 172.21.15.28:0/122701065 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fa6fc0a2b90 msgr2=0x7fa6fc0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:29.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:29.587+0000 7fa71ca00640 1 --2- 172.21.15.28:0/122701065 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fa6fc0a2b90 0x7fa6fc0a5050 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fa70802a9e0 tx=0x7fa7080315b0 comp rx=0 tx=0).stop 2024-05-12T22:06:29.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:29.587+0000 7fa71ca00640 1 -- 172.21.15.28:0/122701065 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa718104c80 msgr2=0x7fa71819a300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:29.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:29.587+0000 7fa71ca00640 1 --2- 172.21.15.28:0/122701065 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa718104c80 0x7fa71819a300 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fa70400e9f0 tx=0x7fa70400edf0 comp rx=0 tx=0).stop 2024-05-12T22:06:29.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:29.587+0000 7fa71ca00640 1 -- 172.21.15.28:0/122701065 shutdown_connections 2024-05-12T22:06:29.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:29.587+0000 7fa71ca00640 1 --2- 172.21.15.28:0/122701065 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fa6fc0a2b90 0x7fa6fc0a5050 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:29.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:29.587+0000 7fa71ca00640 1 --2- 172.21.15.28:0/122701065 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa718105640 0x7fa7181a1370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:29.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:29.587+0000 7fa71ca00640 1 --2- 172.21.15.28:0/122701065 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa718104c80 0x7fa71819a300 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:29.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:29.587+0000 7fa71ca00640 1 --2- 172.21.15.28:0/122701065 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa718103a90 0x7fa718199dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:29.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:29.587+0000 7fa71ca00640 1 -- 172.21.15.28:0/122701065 >> 172.21.15.28:0/122701065 conn(0x7fa7180ff020 msgr2=0x7fa718107f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:06:29.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:29.587+0000 7fa71ca00640 1 -- 172.21.15.28:0/122701065 shutdown_connections 2024-05-12T22:06:29.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:29.587+0000 7fa71ca00640 1 -- 172.21.15.28:0/122701065 wait complete. 2024-05-12T22:06:29.605 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-12T22:01:32.292826+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-12T22:06:29.605 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-12T22:06:29.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:29 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/1325054623' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:06:29.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:29 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/122701065' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-12T22:06:29.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:29 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:29.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:29 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:29.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:29.809+0000 7fe024c00640 1 -- 172.21.15.28:0/1759838917 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe02010dd90 msgr2=0x7fe020110180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:29.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:29.809+0000 7fe024c00640 1 --2- 172.21.15.28:0/1759838917 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe02010dd90 0x7fe020110180 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fe018009860 tx=0x7fe01802a340 comp rx=0 tx=0).stop 2024-05-12T22:06:29.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:29.809+0000 7fe024c00640 1 -- 172.21.15.28:0/1759838917 shutdown_connections 2024-05-12T22:06:29.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:29.809+0000 7fe024c00640 1 --2- 172.21.15.28:0/1759838917 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe02010dd90 0x7fe020110180 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:29.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:29.809+0000 7fe024c00640 1 --2- 172.21.15.28:0/1759838917 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe020074290 0x7fe020074710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:29.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:29.809+0000 7fe024c00640 1 --2- 172.21.15.28:0/1759838917 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe020075870 0x7fe020073cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:29.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:29.809+0000 7fe024c00640 1 -- 172.21.15.28:0/1759838917 >> 172.21.15.28:0/1759838917 conn(0x7fe0200fcd30 msgr2=0x7fe0200ff170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:06:29.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:29.809+0000 7fe024c00640 1 -- 172.21.15.28:0/1759838917 shutdown_connections 2024-05-12T22:06:29.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:29.809+0000 7fe024c00640 1 -- 172.21.15.28:0/1759838917 wait complete. 2024-05-12T22:06:29.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:29.810+0000 7fe024c00640 1 Processor -- start 2024-05-12T22:06:29.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:29.810+0000 7fe024c00640 1 -- start start 2024-05-12T22:06:29.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:29.810+0000 7fe024c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe020074290 0x7fe02019dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:29.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:29.810+0000 7fe024c00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe020075870 0x7fe02019e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:29.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:29.810+0000 7fe024c00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe02010dd90 0x7fe0201a5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:29.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:29.810+0000 7fe024c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe0201139a0 con 0x7fe020074290 2024-05-12T22:06:29.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:29.810+0000 7fe024c00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fe020113820 con 0x7fe02010dd90 2024-05-12T22:06:29.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:29.810+0000 7fe024c00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe020113b20 con 0x7fe020075870 2024-05-12T22:06:29.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:29.810+0000 7fe01f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe020074290 0x7fe02019dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:29.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:29.811+0000 7fe01f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe020074290 0x7fe02019dfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55358/0 (socket says 172.21.15.28:55358) 2024-05-12T22:06:29.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:29.811+0000 7fe01f400640 1 -- 172.21.15.28:0/3343768427 learned_addr learned my addr 172.21.15.28:0/3343768427 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:06:29.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:29.811+0000 7fe016a00640 1 --2- 172.21.15.28:0/3343768427 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe020075870 0x7fe02019e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:29.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:29.811+0000 7fe01fe00640 1 --2- 172.21.15.28:0/3343768427 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe02010dd90 0x7fe0201a5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:29.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:29.811+0000 7fe016a00640 1 -- 172.21.15.28:0/3343768427 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe02010dd90 msgr2=0x7fe0201a5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:29.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:29.811+0000 7fe016a00640 1 --2- 172.21.15.28:0/3343768427 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe02010dd90 0x7fe0201a5560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:29.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:29.811+0000 7fe016a00640 1 -- 172.21.15.28:0/3343768427 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe020074290 msgr2=0x7fe02019dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:29.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:29.811+0000 7fe016a00640 1 --2- 172.21.15.28:0/3343768427 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe020074290 0x7fe02019dfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:29.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:29.811+0000 7fe016a00640 1 -- 172.21.15.28:0/3343768427 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0201a5bd0 con 0x7fe020075870 2024-05-12T22:06:29.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:29.811+0000 7fe01f400640 1 --2- 172.21.15.28:0/3343768427 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe020074290 0x7fe02019dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:06:29.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:29.811+0000 7fe01fe00640 1 --2- 172.21.15.28:0/3343768427 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe02010dd90 0x7fe0201a5560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:06:29.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:29.811+0000 7fe016a00640 1 --2- 172.21.15.28:0/3343768427 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe020075870 0x7fe02019e4f0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe00800e9f0 tx=0x7fe00800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:06:29.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:29.811+0000 7fe01cc00640 1 -- 172.21.15.28:0/3343768427 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe00800ccc0 con 0x7fe020075870 2024-05-12T22:06:29.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:29.812+0000 7fe01cc00640 1 -- 172.21.15.28:0/3343768427 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe008010910 con 0x7fe020075870 2024-05-12T22:06:29.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:29.812+0000 7fe024c00640 1 -- 172.21.15.28:0/3343768427 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0201a5ec0 con 0x7fe020075870 2024-05-12T22:06:29.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:29.812+0000 7fe01cc00640 1 -- 172.21.15.28:0/3343768427 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe008018960 con 0x7fe020075870 2024-05-12T22:06:29.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:29.812+0000 7fe024c00640 1 -- 172.21.15.28:0/3343768427 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0201a6380 con 0x7fe020075870 2024-05-12T22:06:29.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:29.813+0000 7fe024c00640 1 -- 172.21.15.28:0/3343768427 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe020103f60 con 0x7fe020075870 2024-05-12T22:06:29.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:29.815+0000 7fe01cc00640 1 -- 172.21.15.28:0/3343768427 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fe008010ac0 con 0x7fe020075870 2024-05-12T22:06:29.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:29.816+0000 7fe01cc00640 1 --2- 172.21.15.28:0/3343768427 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fe0040a2b90 0x7fe0040a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:29.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:29.816+0000 7fe01cc00640 1 -- 172.21.15.28:0/3343768427 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fe0080cef90 con 0x7fe020075870 2024-05-12T22:06:29.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:29.816+0000 7fe01f400640 1 --2- 172.21.15.28:0/3343768427 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fe0040a2b90 0x7fe0040a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:29.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:29.817+0000 7fe01f400640 1 --2- 172.21.15.28:0/3343768427 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fe0040a2b90 0x7fe0040a5050 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fe01000e8b0 tx=0x7fe0100095c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:06:29.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:29.820+0000 7fe01cc00640 1 -- 172.21.15.28:0/3343768427 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fe00809bca0 con 0x7fe020075870 2024-05-12T22:06:29.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:29 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/1325054623' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:06:29.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:29 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/122701065' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-12T22:06:29.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:29 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:29.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:29 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:29.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:29 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/1325054623' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:06:29.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:29 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/122701065' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-12T22:06:29.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:29 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:29.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:29 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:30.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:30.030+0000 7fe024c00640 1 -- 172.21.15.28:0/3343768427 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fe020062430 con 0x7fe020075870 2024-05-12T22:06:30.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:30.031+0000 7fe01cc00640 1 -- 172.21.15.28:0/3343768427 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) ==== 93+0+10 (secure 0 0 0) 0x7fe00809ffa0 con 0x7fe020075870 2024-05-12T22:06:30.031 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-05-12T22:06:30.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:30.034+0000 7fe024c00640 1 -- 172.21.15.28:0/3343768427 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fe0040a2b90 msgr2=0x7fe0040a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:30.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:30.034+0000 7fe024c00640 1 --2- 172.21.15.28:0/3343768427 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fe0040a2b90 0x7fe0040a5050 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fe01000e8b0 tx=0x7fe0100095c0 comp rx=0 tx=0).stop 2024-05-12T22:06:30.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:30.034+0000 7fe024c00640 1 -- 172.21.15.28:0/3343768427 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe020075870 msgr2=0x7fe02019e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:30.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:30.034+0000 7fe024c00640 1 --2- 172.21.15.28:0/3343768427 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe020075870 0x7fe02019e4f0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe00800e9f0 tx=0x7fe00800eeb0 comp rx=0 tx=0).stop 2024-05-12T22:06:30.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:30.034+0000 7fe024c00640 1 -- 172.21.15.28:0/3343768427 shutdown_connections 2024-05-12T22:06:30.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:30.034+0000 7fe024c00640 1 --2- 172.21.15.28:0/3343768427 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fe0040a2b90 0x7fe0040a5050 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:30.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:30.034+0000 7fe024c00640 1 --2- 172.21.15.28:0/3343768427 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe02010dd90 0x7fe0201a5560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:30.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:30.034+0000 7fe024c00640 1 --2- 172.21.15.28:0/3343768427 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe020075870 0x7fe02019e4f0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:30.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:30.034+0000 7fe024c00640 1 --2- 172.21.15.28:0/3343768427 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe020074290 0x7fe02019dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:30.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:30.034+0000 7fe024c00640 1 -- 172.21.15.28:0/3343768427 >> 172.21.15.28:0/3343768427 conn(0x7fe0200fcd30 msgr2=0x7fe020101130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:06:30.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:30.034+0000 7fe024c00640 1 -- 172.21.15.28:0/3343768427 shutdown_connections 2024-05-12T22:06:30.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:30.034+0000 7fe024c00640 1 -- 172.21.15.28:0/3343768427 wait complete. 2024-05-12T22:06:30.053 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-05-12T22:06:30.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:30.252+0000 7fa711a00640 1 -- 172.21.15.28:0/3665735587 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa70c101880 msgr2=0x7fa70c101ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:30.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:30.252+0000 7fa711a00640 1 --2- 172.21.15.28:0/3665735587 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa70c101880 0x7fa70c101ca0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fa6fc009900 tx=0x7fa6fc02a410 comp rx=0 tx=0).stop 2024-05-12T22:06:30.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:30.252+0000 7fa711a00640 1 -- 172.21.15.28:0/3665735587 shutdown_connections 2024-05-12T22:06:30.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:30.252+0000 7fa711a00640 1 --2- 172.21.15.28:0/3665735587 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa70c103430 0x7fa70c109cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:30.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:30.252+0000 7fa711a00640 1 --2- 172.21.15.28:0/3665735587 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa70c102a70 0x7fa70c102ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:30.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:30.252+0000 7fa711a00640 1 --2- 172.21.15.28:0/3665735587 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa70c101880 0x7fa70c101ca0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:30.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:30.252+0000 7fa711a00640 1 -- 172.21.15.28:0/3665735587 >> 172.21.15.28:0/3665735587 conn(0x7fa70c0fce10 msgr2=0x7fa70c0ff230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:06:30.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:30.252+0000 7fa711a00640 1 -- 172.21.15.28:0/3665735587 shutdown_connections 2024-05-12T22:06:30.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:30.252+0000 7fa711a00640 1 -- 172.21.15.28:0/3665735587 wait complete. 2024-05-12T22:06:30.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:30.254+0000 7fa711a00640 1 Processor -- start 2024-05-12T22:06:30.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:30.254+0000 7fa711a00640 1 -- start start 2024-05-12T22:06:30.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:30.254+0000 7fa711a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa70c101880 0x7fa70c199de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:30.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:30.254+0000 7fa711a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa70c102a70 0x7fa70c19a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:30.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:30.254+0000 7fa711a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa70c103430 0x7fa70c1a1390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:30.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:30.254+0000 7fa711a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa70c10d520 con 0x7fa70c103430 2024-05-12T22:06:30.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:30.254+0000 7fa711a00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fa70c10d3a0 con 0x7fa70c102a70 2024-05-12T22:06:30.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:30.254+0000 7fa711a00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa70c10d6a0 con 0x7fa70c101880 2024-05-12T22:06:30.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:30.254+0000 7fa70aa00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa70c101880 0x7fa70c199de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:30.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:30.255+0000 7fa70aa00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa70c101880 0x7fa70c199de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:45316/0 (socket says 172.21.15.28:45316) 2024-05-12T22:06:30.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:30.255+0000 7fa70aa00640 1 -- 172.21.15.28:0/1639179262 learned_addr learned my addr 172.21.15.28:0/1639179262 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:06:30.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:30.255+0000 7fa70a000640 1 --2- 172.21.15.28:0/1639179262 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa70c102a70 0x7fa70c19a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:30.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:30.255+0000 7fa70b400640 1 --2- 172.21.15.28:0/1639179262 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa70c103430 0x7fa70c1a1390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:30.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:30.255+0000 7fa70aa00640 1 -- 172.21.15.28:0/1639179262 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa70c102a70 msgr2=0x7fa70c19a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:30.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:30.255+0000 7fa70aa00640 1 --2- 172.21.15.28:0/1639179262 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa70c102a70 0x7fa70c19a320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:30.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:30.255+0000 7fa70aa00640 1 -- 172.21.15.28:0/1639179262 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa70c103430 msgr2=0x7fa70c1a1390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:30.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:30.255+0000 7fa70aa00640 1 --2- 172.21.15.28:0/1639179262 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa70c103430 0x7fa70c1a1390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:30.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:30.255+0000 7fa70aa00640 1 -- 172.21.15.28:0/1639179262 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa70c1a1a00 con 0x7fa70c101880 2024-05-12T22:06:30.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:30.255+0000 7fa70aa00640 1 --2- 172.21.15.28:0/1639179262 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa70c101880 0x7fa70c199de0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fa6fc031440 tx=0x7fa6fc032ce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:06:30.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:30.256+0000 7fa703400640 1 -- 172.21.15.28:0/1639179262 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6fc03f030 con 0x7fa70c101880 2024-05-12T22:06:30.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:30.256+0000 7fa703400640 1 -- 172.21.15.28:0/1639179262 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa6fc003ce0 con 0x7fa70c101880 2024-05-12T22:06:30.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:30.256+0000 7fa703400640 1 -- 172.21.15.28:0/1639179262 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6fc035070 con 0x7fa70c101880 2024-05-12T22:06:30.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:30.256+0000 7fa711a00640 1 -- 172.21.15.28:0/1639179262 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa70c1a1c90 con 0x7fa70c101880 2024-05-12T22:06:30.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:30.256+0000 7fa711a00640 1 -- 172.21.15.28:0/1639179262 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa70c1a2130 con 0x7fa70c101880 2024-05-12T22:06:30.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:30.257+0000 7fa711a00640 1 -- 172.21.15.28:0/1639179262 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa70c1a2300 con 0x7fa70c101880 2024-05-12T22:06:30.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:30.259+0000 7fa703400640 1 -- 172.21.15.28:0/1639179262 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fa6fc042030 con 0x7fa70c101880 2024-05-12T22:06:30.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:30.261+0000 7fa703400640 1 --2- 172.21.15.28:0/1639179262 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fa6f00a2c10 0x7fa6f00a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:30.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:30.261+0000 7fa703400640 1 -- 172.21.15.28:0/1639179262 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fa6fc0ef1d0 con 0x7fa70c101880 2024-05-12T22:06:30.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:30.261+0000 7fa70a000640 1 --2- 172.21.15.28:0/1639179262 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fa6f00a2c10 0x7fa6f00a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:30.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:30.262+0000 7fa70a000640 1 --2- 172.21.15.28:0/1639179262 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fa6f00a2c10 0x7fa6f00a50d0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fa70c19b2f0 tx=0x7fa6f8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:06:30.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:30.265+0000 7fa703400640 1 -- 172.21.15.28:0/1639179262 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fa6fc0bc530 con 0x7fa70c101880 2024-05-12T22:06:30.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:30.504+0000 7fa711a00640 1 -- 172.21.15.28:0/1639179262 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fa70c1a2690 con 0x7fa70c101880 2024-05-12T22:06:30.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:30.504+0000 7fa703400640 1 -- 172.21.15.28:0/1639179262 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v82) ==== 84+0+65 (secure 0 0 0) 0x7fa6fc0bbe30 con 0x7fa70c101880 2024-05-12T22:06:30.505 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-12T22:06:30.505 INFO:teuthology.orchestra.run.smithi028.stdout:[ 2024-05-12T22:06:30.505 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-12T22:06:30.505 INFO:teuthology.orchestra.run.smithi028.stdout: "poolnum": 1, 2024-05-12T22:06:30.505 INFO:teuthology.orchestra.run.smithi028.stdout: "poolname": ".mgr" 2024-05-12T22:06:30.505 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:06:30.506 INFO:teuthology.orchestra.run.smithi028.stdout:] 2024-05-12T22:06:30.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:30.507+0000 7fa711a00640 1 -- 172.21.15.28:0/1639179262 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fa6f00a2c10 msgr2=0x7fa6f00a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:30.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:30.507+0000 7fa711a00640 1 --2- 172.21.15.28:0/1639179262 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fa6f00a2c10 0x7fa6f00a50d0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fa70c19b2f0 tx=0x7fa6f8005f90 comp rx=0 tx=0).stop 2024-05-12T22:06:30.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:30.507+0000 7fa711a00640 1 -- 172.21.15.28:0/1639179262 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa70c101880 msgr2=0x7fa70c199de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:30.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:30.507+0000 7fa711a00640 1 --2- 172.21.15.28:0/1639179262 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa70c101880 0x7fa70c199de0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fa6fc031440 tx=0x7fa6fc032ce0 comp rx=0 tx=0).stop 2024-05-12T22:06:30.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:30.507+0000 7fa711a00640 1 -- 172.21.15.28:0/1639179262 shutdown_connections 2024-05-12T22:06:30.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:30.507+0000 7fa711a00640 1 --2- 172.21.15.28:0/1639179262 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fa6f00a2c10 0x7fa6f00a50d0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:30.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:30.507+0000 7fa711a00640 1 --2- 172.21.15.28:0/1639179262 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa70c103430 0x7fa70c1a1390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:30.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:30.507+0000 7fa711a00640 1 --2- 172.21.15.28:0/1639179262 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa70c102a70 0x7fa70c19a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:30.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:30.507+0000 7fa711a00640 1 --2- 172.21.15.28:0/1639179262 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa70c101880 0x7fa70c199de0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:30.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:30.507+0000 7fa711a00640 1 -- 172.21.15.28:0/1639179262 >> 172.21.15.28:0/1639179262 conn(0x7fa70c0fce10 msgr2=0x7fa70c073a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:06:30.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:30.507+0000 7fa711a00640 1 -- 172.21.15.28:0/1639179262 shutdown_connections 2024-05-12T22:06:30.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:30.507+0000 7fa711a00640 1 -- 172.21.15.28:0/1639179262 wait complete. 2024-05-12T22:06:30.528 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-05-12T22:06:30.528 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-05-12T22:06:30.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:30.733+0000 7f76fb200640 1 -- 172.21.15.28:0/3377871994 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76fc101760 msgr2=0x7f76fc101b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:30.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:30.733+0000 7f76fb200640 1 --2- 172.21.15.28:0/3377871994 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76fc101760 0x7f76fc101b80 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f76ec009900 tx=0x7f76ec02a3f0 comp rx=0 tx=0).stop 2024-05-12T22:06:30.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:30.733+0000 7f76fb200640 1 -- 172.21.15.28:0/3377871994 shutdown_connections 2024-05-12T22:06:30.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:30.733+0000 7f76fb200640 1 --2- 172.21.15.28:0/3377871994 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f76fc103310 0x7f76fc109ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:30.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:30.733+0000 7f76fb200640 1 --2- 172.21.15.28:0/3377871994 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f76fc102950 0x7f76fc102dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:30.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:30.733+0000 7f76fb200640 1 --2- 172.21.15.28:0/3377871994 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76fc101760 0x7f76fc101b80 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:30.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:30.733+0000 7f76fb200640 1 -- 172.21.15.28:0/3377871994 >> 172.21.15.28:0/3377871994 conn(0x7f76fc0fccd0 msgr2=0x7f76fc0ff110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:06:30.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:30.733+0000 7f76fb200640 1 -- 172.21.15.28:0/3377871994 shutdown_connections 2024-05-12T22:06:30.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:30.733+0000 7f76fb200640 1 -- 172.21.15.28:0/3377871994 wait complete. 2024-05-12T22:06:30.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:30.734+0000 7f76fb200640 1 Processor -- start 2024-05-12T22:06:30.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:30.734+0000 7f76fb200640 1 -- start start 2024-05-12T22:06:30.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:30.735+0000 7f76fb200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76fc101760 0x7f76fc199cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:30.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:30.735+0000 7f76fb200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f76fc102950 0x7f76fc19a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:30.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:30.735+0000 7f76fb200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f76fc103310 0x7f76fc1a1260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:30.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:30.735+0000 7f76fb200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f76fc10d400 con 0x7f76fc101760 2024-05-12T22:06:30.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:30.735+0000 7f76fb200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f76fc10d280 con 0x7f76fc103310 2024-05-12T22:06:30.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:30.735+0000 7f76fb200640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f76fc10d580 con 0x7f76fc102950 2024-05-12T22:06:30.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:30.735+0000 7f76f9e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76fc101760 0x7f76fc199cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:30.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:30.735+0000 7f76f9e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76fc101760 0x7f76fc199cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55394/0 (socket says 172.21.15.28:55394) 2024-05-12T22:06:30.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:30.735+0000 7f76f9e00640 1 -- 172.21.15.28:0/200002419 learned_addr learned my addr 172.21.15.28:0/200002419 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:06:30.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:30.735+0000 7f76f9400640 1 --2- 172.21.15.28:0/200002419 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f76fc102950 0x7f76fc19a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:30.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:30.735+0000 7f76fa800640 1 --2- 172.21.15.28:0/200002419 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f76fc103310 0x7f76fc1a1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:30.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:30.735+0000 7f76f9400640 1 -- 172.21.15.28:0/200002419 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f76fc103310 msgr2=0x7f76fc1a1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:30.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:30.735+0000 7f76f9400640 1 --2- 172.21.15.28:0/200002419 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f76fc103310 0x7f76fc1a1260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:30.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:30.735+0000 7f76f9400640 1 -- 172.21.15.28:0/200002419 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76fc101760 msgr2=0x7f76fc199cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:30.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:30.735+0000 7f76f9400640 1 --2- 172.21.15.28:0/200002419 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76fc101760 0x7f76fc199cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:30.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:30.735+0000 7f76f9400640 1 -- 172.21.15.28:0/200002419 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76fc1a1900 con 0x7f76fc102950 2024-05-12T22:06:30.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:30.735+0000 7f76f9e00640 1 --2- 172.21.15.28:0/200002419 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76fc101760 0x7f76fc199cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:06:30.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:30.736+0000 7f76f9400640 1 --2- 172.21.15.28:0/200002419 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f76fc102950 0x7f76fc19a200 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f76e800e9f0 tx=0x7f76e800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:06:30.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:30.736+0000 7f76f2a00640 1 -- 172.21.15.28:0/200002419 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76e800ccc0 con 0x7f76fc102950 2024-05-12T22:06:30.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:30.736+0000 7f76f2a00640 1 -- 172.21.15.28:0/200002419 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f76e8010910 con 0x7f76fc102950 2024-05-12T22:06:30.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:30.736+0000 7f76fb200640 1 -- 172.21.15.28:0/200002419 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76fc1a1bf0 con 0x7f76fc102950 2024-05-12T22:06:30.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:30.736+0000 7f76f2a00640 1 -- 172.21.15.28:0/200002419 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76e8018960 con 0x7f76fc102950 2024-05-12T22:06:30.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:30.736+0000 7f76fb200640 1 -- 172.21.15.28:0/200002419 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76fc1a2140 con 0x7f76fc102950 2024-05-12T22:06:30.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:30.738+0000 7f76fb200640 1 -- 172.21.15.28:0/200002419 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f76fc1a2310 con 0x7f76fc102950 2024-05-12T22:06:30.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:30.739+0000 7f76f2a00640 1 -- 172.21.15.28:0/200002419 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f76e8010ac0 con 0x7f76fc102950 2024-05-12T22:06:30.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:30.741+0000 7f76f2a00640 1 --2- 172.21.15.28:0/200002419 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f76e00a2b90 0x7f76e00a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:30.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:30.741+0000 7f76f2a00640 1 -- 172.21.15.28:0/200002419 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f76e80cefb0 con 0x7f76fc102950 2024-05-12T22:06:30.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:30.742+0000 7f76f9e00640 1 --2- 172.21.15.28:0/200002419 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f76e00a2b90 0x7f76e00a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:30.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:30.742+0000 7f76f9e00640 1 --2- 172.21.15.28:0/200002419 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f76e00a2b90 0x7f76e00a5050 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f76ec006010 tx=0x7f76ec0315b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:06:30.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:30.745+0000 7f76f2a00640 1 -- 172.21.15.28:0/200002419 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f76e809bcc0 con 0x7f76fc102950 2024-05-12T22:06:30.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:30 smithi028 ceph-mon[29847]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-12T22:06:30.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:30 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:30.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:30 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:30.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:30 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/3343768427' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-12T22:06:30.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:30 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/1639179262' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-05-12T22:06:30.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:30 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:30.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:30 smithi186 ceph-mon[34785]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-12T22:06:30.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:30 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:30.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:30 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:30.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:30 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/3343768427' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-12T22:06:30.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:30 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/1639179262' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-05-12T22:06:30.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:30 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:30.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:30.979+0000 7f76fb200640 1 -- 172.21.15.28:0/200002419 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f76fc1a2600 con 0x7f76fc102950 2024-05-12T22:06:30.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:30 smithi086 ceph-mon[34297]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-12T22:06:30.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:30 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:30.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:30 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:30.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:30 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/3343768427' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-12T22:06:30.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:30 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/1639179262' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-05-12T22:06:30.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:30 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:31.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:31.884+0000 7f76f2a00640 1 -- 172.21.15.28:0/200002419 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v83) ==== 138+0+0 (secure 0 0 0) 0x7f76e809ffc0 con 0x7f76fc102950 2024-05-12T22:06:31.885 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_metadata' created 2024-05-12T22:06:31.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:31.890+0000 7f76fb200640 1 -- 172.21.15.28:0/200002419 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f76e00a2b90 msgr2=0x7f76e00a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:31.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:31.890+0000 7f76fb200640 1 --2- 172.21.15.28:0/200002419 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f76e00a2b90 0x7f76e00a5050 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f76ec006010 tx=0x7f76ec0315b0 comp rx=0 tx=0).stop 2024-05-12T22:06:31.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:31.890+0000 7f76fb200640 1 -- 172.21.15.28:0/200002419 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f76fc102950 msgr2=0x7f76fc19a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:31.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:31.890+0000 7f76fb200640 1 --2- 172.21.15.28:0/200002419 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f76fc102950 0x7f76fc19a200 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f76e800e9f0 tx=0x7f76e800eeb0 comp rx=0 tx=0).stop 2024-05-12T22:06:31.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:31.891+0000 7f76fb200640 1 -- 172.21.15.28:0/200002419 shutdown_connections 2024-05-12T22:06:31.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:31.891+0000 7f76fb200640 1 --2- 172.21.15.28:0/200002419 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f76e00a2b90 0x7f76e00a5050 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:31.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:31.891+0000 7f76fb200640 1 --2- 172.21.15.28:0/200002419 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f76fc103310 0x7f76fc1a1260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:31.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:31.891+0000 7f76fb200640 1 --2- 172.21.15.28:0/200002419 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f76fc102950 0x7f76fc19a200 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:31.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:31.891+0000 7f76fb200640 1 --2- 172.21.15.28:0/200002419 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76fc101760 0x7f76fc199cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:31.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:31.891+0000 7f76fb200640 1 -- 172.21.15.28:0/200002419 >> 172.21.15.28:0/200002419 conn(0x7f76fc0fccd0 msgr2=0x7f76fc073a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:06:31.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:31.891+0000 7f76fb200640 1 -- 172.21.15.28:0/200002419 shutdown_connections 2024-05-12T22:06:31.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:31.893+0000 7f76fb200640 1 -- 172.21.15.28:0/200002419 wait complete. 2024-05-12T22:06:31.913 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-05-12T22:06:32.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:32.106+0000 7f3772c00640 1 -- 172.21.15.28:0/1508851251 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f376c1021c0 msgr2=0x7f376c1045b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:32.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:32.106+0000 7f3772c00640 1 --2- 172.21.15.28:0/1508851251 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f376c1021c0 0x7f376c1045b0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f375c009900 tx=0x7f375c02a410 comp rx=0 tx=0).stop 2024-05-12T22:06:32.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:32.106+0000 7f3772c00640 1 -- 172.21.15.28:0/1508851251 shutdown_connections 2024-05-12T22:06:32.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:32.106+0000 7f3772c00640 1 --2- 172.21.15.28:0/1508851251 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f376c107420 0x7f376c109870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:32.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:32.106+0000 7f3772c00640 1 --2- 172.21.15.28:0/1508851251 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f376c104af0 0x7f376c106ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:32.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:32.106+0000 7f3772c00640 1 --2- 172.21.15.28:0/1508851251 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f376c1021c0 0x7f376c1045b0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:32.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:32.106+0000 7f3772c00640 1 -- 172.21.15.28:0/1508851251 >> 172.21.15.28:0/1508851251 conn(0x7f376c0fbbd0 msgr2=0x7f376c0fdff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:06:32.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:32.106+0000 7f3772c00640 1 -- 172.21.15.28:0/1508851251 shutdown_connections 2024-05-12T22:06:32.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:32.106+0000 7f3772c00640 1 -- 172.21.15.28:0/1508851251 wait complete. 2024-05-12T22:06:32.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:32.107+0000 7f3772c00640 1 Processor -- start 2024-05-12T22:06:32.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:32.107+0000 7f3772c00640 1 -- start start 2024-05-12T22:06:32.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:32.107+0000 7f3772c00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f376c1021c0 0x7f376c197a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:32.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:32.107+0000 7f3772c00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f376c104af0 0x7f376c197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:32.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:32.107+0000 7f3772c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f376c107420 0x7f376c19f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:32.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:32.107+0000 7f3772c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f376c10d410 con 0x7f376c107420 2024-05-12T22:06:32.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:32.107+0000 7f3772c00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f376c10d290 con 0x7f376c1021c0 2024-05-12T22:06:32.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:32.107+0000 7f3772c00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f376c10d590 con 0x7f376c104af0 2024-05-12T22:06:32.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:32.108+0000 7f3770e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f376c104af0 0x7f376c197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:32.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:32.108+0000 7f3771800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f376c1021c0 0x7f376c197a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:32.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:32.108+0000 7f3771800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f376c1021c0 0x7f376c197a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:41906/0 (socket says 172.21.15.28:41906) 2024-05-12T22:06:32.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:32.108+0000 7f3771800640 1 -- 172.21.15.28:0/311435184 learned_addr learned my addr 172.21.15.28:0/311435184 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:06:32.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:32.108+0000 7f3772200640 1 --2- 172.21.15.28:0/311435184 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f376c107420 0x7f376c19f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:32.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:32.108+0000 7f3770e00640 1 -- 172.21.15.28:0/311435184 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f376c1021c0 msgr2=0x7f376c197a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:32.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:32.108+0000 7f3770e00640 1 --2- 172.21.15.28:0/311435184 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f376c1021c0 0x7f376c197a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:32.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:32.108+0000 7f3770e00640 1 -- 172.21.15.28:0/311435184 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f376c107420 msgr2=0x7f376c19f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:32.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:32.108+0000 7f3770e00640 1 --2- 172.21.15.28:0/311435184 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f376c107420 0x7f376c19f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:32.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:32.108+0000 7f3770e00640 1 -- 172.21.15.28:0/311435184 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f376c19f670 con 0x7f376c104af0 2024-05-12T22:06:32.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:32.108+0000 7f3770e00640 1 --2- 172.21.15.28:0/311435184 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f376c104af0 0x7f376c197f90 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f375800e9c0 tx=0x7f375800ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:06:32.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:32.109+0000 7f376a000640 1 -- 172.21.15.28:0/311435184 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f375800ccc0 con 0x7f376c104af0 2024-05-12T22:06:32.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:32.109+0000 7f376a000640 1 -- 172.21.15.28:0/311435184 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3758010910 con 0x7f376c104af0 2024-05-12T22:06:32.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:32.109+0000 7f3772c00640 1 -- 172.21.15.28:0/311435184 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f376c19f960 con 0x7f376c104af0 2024-05-12T22:06:32.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:32.109+0000 7f376a000640 1 -- 172.21.15.28:0/311435184 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37580188c0 con 0x7f376c104af0 2024-05-12T22:06:32.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:32.109+0000 7f3772c00640 1 -- 172.21.15.28:0/311435184 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f376c19feb0 con 0x7f376c104af0 2024-05-12T22:06:32.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:32.110+0000 7f3772c00640 1 -- 172.21.15.28:0/311435184 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f376c0fd6c0 con 0x7f376c104af0 2024-05-12T22:06:32.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:32.112+0000 7f376a000640 1 -- 172.21.15.28:0/311435184 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f3758010ac0 con 0x7f376c104af0 2024-05-12T22:06:32.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:32.114+0000 7f376a000640 1 --2- 172.21.15.28:0/311435184 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f37500a2b90 0x7f37500a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:32.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:32.114+0000 7f376a000640 1 -- 172.21.15.28:0/311435184 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7497+0+0 (secure 0 0 0) 0x7f37580cf120 con 0x7f376c104af0 2024-05-12T22:06:32.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:32.114+0000 7f3771800640 1 --2- 172.21.15.28:0/311435184 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f37500a2b90 0x7f37500a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:32.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:32.115+0000 7f3771800640 1 --2- 172.21.15.28:0/311435184 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f37500a2b90 0x7f37500a5050 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f375c0096c0 tx=0x7f375c0315b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:06:32.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:32.118+0000 7f376a000640 1 -- 172.21.15.28:0/311435184 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f375809bcc0 con 0x7f376c104af0 2024-05-12T22:06:32.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:31 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:32.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:31 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:32.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:31 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:06:32.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:31 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:06:32.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:31 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:32.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:31 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/200002419' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-05-12T22:06:32.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:31 smithi186 ceph-mon[34785]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-05-12T22:06:32.180 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:31 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:06:32.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:31 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:32.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:31 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:32.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:31 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:06:32.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:31 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:06:32.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:31 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:32.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:31 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/200002419' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-05-12T22:06:32.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:31 smithi086 ceph-mon[34297]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-05-12T22:06:32.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:31 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:06:32.284 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:31 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:32.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:31 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:32.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:31 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:06:32.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:31 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:06:32.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:31 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:32.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:31 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/200002419' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-05-12T22:06:32.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:31 smithi028 ceph-mon[29847]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-05-12T22:06:32.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:31 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:06:32.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:32.350+0000 7f3772c00640 1 -- 172.21.15.28:0/311435184 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f376c02d1c0 con 0x7f376c104af0 2024-05-12T22:06:32.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:32.916+0000 7f376a000640 1 -- 172.21.15.28:0/311435184 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v84) ==== 170+0+0 (secure 0 0 0) 0x7f375809ffc0 con 0x7f376c104af0 2024-05-12T22:06:32.917 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data' created 2024-05-12T22:06:32.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:32.922+0000 7f3772c00640 1 -- 172.21.15.28:0/311435184 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f37500a2b90 msgr2=0x7f37500a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:32.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:32.922+0000 7f3772c00640 1 --2- 172.21.15.28:0/311435184 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f37500a2b90 0x7f37500a5050 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f375c0096c0 tx=0x7f375c0315b0 comp rx=0 tx=0).stop 2024-05-12T22:06:32.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:32.922+0000 7f3772c00640 1 -- 172.21.15.28:0/311435184 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f376c104af0 msgr2=0x7f376c197f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:32.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:32.922+0000 7f3772c00640 1 --2- 172.21.15.28:0/311435184 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f376c104af0 0x7f376c197f90 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f375800e9c0 tx=0x7f375800ee80 comp rx=0 tx=0).stop 2024-05-12T22:06:32.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:32.923+0000 7f3772c00640 1 -- 172.21.15.28:0/311435184 shutdown_connections 2024-05-12T22:06:32.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:32.923+0000 7f3772c00640 1 --2- 172.21.15.28:0/311435184 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f37500a2b90 0x7f37500a5050 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:32.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:32.923+0000 7f3772c00640 1 --2- 172.21.15.28:0/311435184 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f376c107420 0x7f376c19f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:32.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:32.923+0000 7f3772c00640 1 --2- 172.21.15.28:0/311435184 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f376c104af0 0x7f376c197f90 secure :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f375800e9c0 tx=0x7f375800ee80 comp rx=0 tx=0).stop 2024-05-12T22:06:32.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:32.923+0000 7f3772c00640 1 --2- 172.21.15.28:0/311435184 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f376c1021c0 0x7f376c197a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:32.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:32.923+0000 7f3772c00640 1 -- 172.21.15.28:0/311435184 >> 172.21.15.28:0/311435184 conn(0x7f376c0fbbd0 msgr2=0x7f376c102b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:06:32.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:32.923+0000 7f3772c00640 1 -- 172.21.15.28:0/311435184 shutdown_connections 2024-05-12T22:06:32.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:32.923+0000 7f3772c00640 1 -- 172.21.15.28:0/311435184 wait complete. 2024-05-12T22:06:32.955 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-05-12T22:06:33.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:33.133+0000 7f929d200640 1 -- 172.21.15.28:0/4191792597 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9298105640 msgr2=0x7f929810bed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:33.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:33.133+0000 7f929d200640 1 --2- 172.21.15.28:0/4191792597 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9298105640 0x7f929810bed0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f9290009be0 tx=0x7f929002a6c0 comp rx=0 tx=0).stop 2024-05-12T22:06:33.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:33.133+0000 7f929d200640 1 -- 172.21.15.28:0/4191792597 shutdown_connections 2024-05-12T22:06:33.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:33.133+0000 7f929d200640 1 --2- 172.21.15.28:0/4191792597 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9298105640 0x7f929810bed0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:33.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:33.133+0000 7f929d200640 1 --2- 172.21.15.28:0/4191792597 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9298104c80 0x7f9298105100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:33.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:33.133+0000 7f929d200640 1 --2- 172.21.15.28:0/4191792597 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9298103a90 0x7f9298103eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:33.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:33.133+0000 7f929d200640 1 -- 172.21.15.28:0/4191792597 >> 172.21.15.28:0/4191792597 conn(0x7f92980ff020 msgr2=0x7f9298101440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:06:33.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:33.133+0000 7f929d200640 1 -- 172.21.15.28:0/4191792597 shutdown_connections 2024-05-12T22:06:33.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:33.133+0000 7f929d200640 1 -- 172.21.15.28:0/4191792597 wait complete. 2024-05-12T22:06:33.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:33.134+0000 7f929d200640 1 Processor -- start 2024-05-12T22:06:33.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:33.134+0000 7f929d200640 1 -- start start 2024-05-12T22:06:33.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:33.135+0000 7f929d200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9298103a90 0x7f9298199d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:33.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:33.135+0000 7f929d200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9298104c80 0x7f929819a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:33.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:33.135+0000 7f929d200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9298105640 0x7f92981a12f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:33.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:33.135+0000 7f929d200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f929810f6a0 con 0x7f9298103a90 2024-05-12T22:06:33.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:33.135+0000 7f929d200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f929810f520 con 0x7f9298104c80 2024-05-12T22:06:33.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:33.135+0000 7f929d200640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f929810f820 con 0x7f9298105640 2024-05-12T22:06:33.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:33.135+0000 7f928fe00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9298104c80 0x7f929819a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:33.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:33.135+0000 7f928fe00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9298104c80 0x7f929819a280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:41924/0 (socket says 172.21.15.28:41924) 2024-05-12T22:06:33.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:33.135+0000 7f928fe00640 1 -- 172.21.15.28:0/1115148596 learned_addr learned my addr 172.21.15.28:0/1115148596 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:06:33.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:33.135+0000 7f9295e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9298103a90 0x7f9298199d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:33.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:33.135+0000 7f9296800640 1 --2- 172.21.15.28:0/1115148596 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9298105640 0x7f92981a12f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:33.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:33.135+0000 7f928fe00640 1 -- 172.21.15.28:0/1115148596 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9298105640 msgr2=0x7f92981a12f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:33.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:33.135+0000 7f928fe00640 1 --2- 172.21.15.28:0/1115148596 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9298105640 0x7f92981a12f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:33.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:33.135+0000 7f928fe00640 1 -- 172.21.15.28:0/1115148596 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9298103a90 msgr2=0x7f9298199d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:33.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:33.135+0000 7f928fe00640 1 --2- 172.21.15.28:0/1115148596 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9298103a90 0x7f9298199d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:33.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:33.135+0000 7f928fe00640 1 -- 172.21.15.28:0/1115148596 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92981a1960 con 0x7f9298104c80 2024-05-12T22:06:33.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:33.135+0000 7f9295e00640 1 --2- 172.21.15.28:0/1115148596 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9298103a90 0x7f9298199d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:06:33.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:33.136+0000 7f928fe00640 1 --2- 172.21.15.28:0/1115148596 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9298104c80 0x7f929819a280 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f928000e9f0 tx=0x7f928000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:06:33.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:33.136+0000 7f928ea00640 1 -- 172.21.15.28:0/1115148596 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f928000cd80 con 0x7f9298104c80 2024-05-12T22:06:33.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:33.136+0000 7f928ea00640 1 -- 172.21.15.28:0/1115148596 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9280010910 con 0x7f9298104c80 2024-05-12T22:06:33.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:33.136+0000 7f928ea00640 1 -- 172.21.15.28:0/1115148596 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9280018990 con 0x7f9298104c80 2024-05-12T22:06:33.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:33.136+0000 7f929d200640 1 -- 172.21.15.28:0/1115148596 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92981a1c50 con 0x7f9298104c80 2024-05-12T22:06:33.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:33.136+0000 7f929d200640 1 -- 172.21.15.28:0/1115148596 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92981a21a0 con 0x7f9298104c80 2024-05-12T22:06:33.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:33.137+0000 7f929d200640 1 -- 172.21.15.28:0/1115148596 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f929810adb0 con 0x7f9298104c80 2024-05-12T22:06:33.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:33.139+0000 7f928ea00640 1 -- 172.21.15.28:0/1115148596 <== mon.2 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f9280010ac0 con 0x7f9298104c80 2024-05-12T22:06:33.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:33.140+0000 7f928ea00640 1 --2- 172.21.15.28:0/1115148596 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f927c0a2b90 0x7f927c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:33.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:33.140+0000 7f928ea00640 1 -- 172.21.15.28:0/1115148596 <== mon.2 v2:172.21.15.86:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7879+0+0 (secure 0 0 0) 0x7f92800cf2d0 con 0x7f9298104c80 2024-05-12T22:06:33.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:33.141+0000 7f9295e00640 1 --2- 172.21.15.28:0/1115148596 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f927c0a2b90 0x7f927c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:33.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:33.141+0000 7f9295e00640 1 --2- 172.21.15.28:0/1115148596 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f927c0a2b90 0x7f927c0a5050 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f9298104ae0 tx=0x7f9288009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:06:33.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:33.145+0000 7f928ea00640 1 -- 172.21.15.28:0/1115148596 <== mon.2 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f928009c340 con 0x7f9298104c80 2024-05-12T22:06:33.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:32 smithi086 ceph-mon[34297]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-05-12T22:06:33.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:32 smithi086 ceph-mon[34297]: osdmap e83: 12 total, 12 up, 12 in 2024-05-12T22:06:33.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:32 smithi028 ceph-mon[29847]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-05-12T22:06:33.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:32 smithi028 ceph-mon[29847]: osdmap e83: 12 total, 12 up, 12 in 2024-05-12T22:06:33.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:33.424+0000 7f929d200640 1 -- 172.21.15.28:0/1115148596 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f929802d1c0 con 0x7f9298104c80 2024-05-12T22:06:33.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:32 smithi186 ceph-mon[34785]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-05-12T22:06:33.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:32 smithi186 ceph-mon[34785]: osdmap e83: 12 total, 12 up, 12 in 2024-05-12T22:06:34.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:33 smithi086 ceph-mon[34297]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-12T22:06:34.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:33 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/311435184' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-05-12T22:06:34.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:33 smithi086 ceph-mon[34297]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-05-12T22:06:34.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:33 smithi086 ceph-mon[34297]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-05-12T22:06:34.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:33 smithi086 ceph-mon[34297]: osdmap e84: 12 total, 12 up, 12 in 2024-05-12T22:06:34.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:33 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:34.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:33 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:34.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:33 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:34.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:33 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:34.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:33 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/1115148596' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-05-12T22:06:34.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:33 smithi086 ceph-mon[34297]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-05-12T22:06:34.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:33 smithi086 ceph-mon[34297]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-05-12T22:06:34.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:34.264+0000 7f928ea00640 1 -- 172.21.15.28:0/1115148596 <== mon.2 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) ==== 172+0+0 (secure 0 0 0) 0x7f928009bc40 con 0x7f9298104c80 2024-05-12T22:06:34.264 INFO:teuthology.orchestra.run.smithi028.stderr:new fs with metadata pool 2 and data pool 3 2024-05-12T22:06:34.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:34.268+0000 7f929d200640 1 -- 172.21.15.28:0/1115148596 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f927c0a2b90 msgr2=0x7f927c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:34.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:34.268+0000 7f929d200640 1 --2- 172.21.15.28:0/1115148596 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f927c0a2b90 0x7f927c0a5050 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f9298104ae0 tx=0x7f9288009260 comp rx=0 tx=0).stop 2024-05-12T22:06:34.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:34.268+0000 7f929d200640 1 -- 172.21.15.28:0/1115148596 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9298104c80 msgr2=0x7f929819a280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:34.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:34.268+0000 7f929d200640 1 --2- 172.21.15.28:0/1115148596 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9298104c80 0x7f929819a280 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f928000e9f0 tx=0x7f928000eeb0 comp rx=0 tx=0).stop 2024-05-12T22:06:34.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:34.268+0000 7f929d200640 1 -- 172.21.15.28:0/1115148596 shutdown_connections 2024-05-12T22:06:34.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:34.269+0000 7f929d200640 1 --2- 172.21.15.28:0/1115148596 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f927c0a2b90 0x7f927c0a5050 secure :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f9298104ae0 tx=0x7f9288009260 comp rx=0 tx=0).stop 2024-05-12T22:06:34.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:34.269+0000 7f929d200640 1 --2- 172.21.15.28:0/1115148596 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9298105640 0x7f92981a12f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:34.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:34.269+0000 7f929d200640 1 --2- 172.21.15.28:0/1115148596 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9298104c80 0x7f929819a280 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:34.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:34.269+0000 7f929d200640 1 --2- 172.21.15.28:0/1115148596 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9298103a90 0x7f9298199d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:34.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:34.269+0000 7f929d200640 1 -- 172.21.15.28:0/1115148596 >> 172.21.15.28:0/1115148596 conn(0x7f92980ff020 msgr2=0x7f9298107f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:06:34.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:34.270+0000 7f929d200640 1 -- 172.21.15.28:0/1115148596 shutdown_connections 2024-05-12T22:06:34.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:34.271+0000 7f929d200640 1 -- 172.21.15.28:0/1115148596 wait complete. 2024-05-12T22:06:34.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:33 smithi028 ceph-mon[29847]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-12T22:06:34.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:33 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/311435184' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-05-12T22:06:34.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:33 smithi028 ceph-mon[29847]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-05-12T22:06:34.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:33 smithi028 ceph-mon[29847]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-05-12T22:06:34.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:33 smithi028 ceph-mon[29847]: osdmap e84: 12 total, 12 up, 12 in 2024-05-12T22:06:34.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:33 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:34.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:33 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:34.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:33 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:34.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:33 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:34.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:33 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/1115148596' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-05-12T22:06:34.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:33 smithi028 ceph-mon[29847]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-05-12T22:06:34.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:33 smithi028 ceph-mon[29847]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-05-12T22:06:34.295 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-12T22:06:34.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:33 smithi186 ceph-mon[34785]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-12T22:06:34.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:33 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/311435184' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-05-12T22:06:34.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:33 smithi186 ceph-mon[34785]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-05-12T22:06:34.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:33 smithi186 ceph-mon[34785]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-05-12T22:06:34.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:33 smithi186 ceph-mon[34785]: osdmap e84: 12 total, 12 up, 12 in 2024-05-12T22:06:34.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:33 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:34.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:33 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:34.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:33 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:34.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:33 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:34.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:33 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/1115148596' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-05-12T22:06:34.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:33 smithi186 ceph-mon[34785]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-05-12T22:06:34.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:33 smithi186 ceph-mon[34785]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-05-12T22:06:34.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:34.485+0000 7f5b96c00640 1 -- 172.21.15.28:0/393308056 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5b901028c0 msgr2=0x7f5b90102d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:34.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:34.485+0000 7f5b96c00640 1 --2- 172.21.15.28:0/393308056 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5b901028c0 0x7f5b90102d40 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f5b7c009900 tx=0x7f5b7c02a3f0 comp rx=0 tx=0).stop 2024-05-12T22:06:34.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:34.485+0000 7f5b96c00640 1 -- 172.21.15.28:0/393308056 shutdown_connections 2024-05-12T22:06:34.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:34.485+0000 7f5b96c00640 1 --2- 172.21.15.28:0/393308056 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b90103280 0x7f5b90109b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:34.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:34.485+0000 7f5b96c00640 1 --2- 172.21.15.28:0/393308056 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5b901028c0 0x7f5b90102d40 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:34.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:34.485+0000 7f5b96c00640 1 --2- 172.21.15.28:0/393308056 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5b901016d0 0x7f5b90101af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:34.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:34.485+0000 7f5b96c00640 1 -- 172.21.15.28:0/393308056 >> 172.21.15.28:0/393308056 conn(0x7f5b900fcc40 msgr2=0x7f5b900ff080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:06:34.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:34.485+0000 7f5b96c00640 1 -- 172.21.15.28:0/393308056 shutdown_connections 2024-05-12T22:06:34.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:34.486+0000 7f5b96c00640 1 -- 172.21.15.28:0/393308056 wait complete. 2024-05-12T22:06:34.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:34.486+0000 7f5b96c00640 1 Processor -- start 2024-05-12T22:06:34.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:34.487+0000 7f5b96c00640 1 -- start start 2024-05-12T22:06:34.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:34.487+0000 7f5b96c00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5b901016d0 0x7f5b901979b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:34.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:34.487+0000 7f5b96c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b901028c0 0x7f5b90197ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:34.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:34.487+0000 7f5b96c00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5b90103280 0x7f5b9019ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:34.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:34.487+0000 7f5b96c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5b9010d3a0 con 0x7f5b901028c0 2024-05-12T22:06:34.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:34.487+0000 7f5b96c00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f5b9010d220 con 0x7f5b901016d0 2024-05-12T22:06:34.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:34.487+0000 7f5b96c00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f5b9010d520 con 0x7f5b90103280 2024-05-12T22:06:34.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:34.487+0000 7f5b94e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b901028c0 0x7f5b90197ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:34.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:34.487+0000 7f5b94e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b901028c0 0x7f5b90197ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55442/0 (socket says 172.21.15.28:55442) 2024-05-12T22:06:34.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:34.487+0000 7f5b94e00640 1 -- 172.21.15.28:0/1808503876 learned_addr learned my addr 172.21.15.28:0/1808503876 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:06:34.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:34.487+0000 7f5b95800640 1 --2- 172.21.15.28:0/1808503876 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5b901016d0 0x7f5b901979b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:34.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:34.487+0000 7f5b96200640 1 --2- 172.21.15.28:0/1808503876 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5b90103280 0x7f5b9019ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:34.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:34.488+0000 7f5b95800640 1 -- 172.21.15.28:0/1808503876 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5b90103280 msgr2=0x7f5b9019ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:34.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:34.488+0000 7f5b95800640 1 --2- 172.21.15.28:0/1808503876 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5b90103280 0x7f5b9019ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:34.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:34.488+0000 7f5b95800640 1 -- 172.21.15.28:0/1808503876 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b901028c0 msgr2=0x7f5b90197ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:34.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:34.488+0000 7f5b95800640 1 --2- 172.21.15.28:0/1808503876 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b901028c0 0x7f5b90197ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:34.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:34.488+0000 7f5b95800640 1 -- 172.21.15.28:0/1808503876 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b9019f5d0 con 0x7f5b901016d0 2024-05-12T22:06:34.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:34.488+0000 7f5b94e00640 1 --2- 172.21.15.28:0/1808503876 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b901028c0 0x7f5b90197ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:06:34.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:34.488+0000 7f5b95800640 1 --2- 172.21.15.28:0/1808503876 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5b901016d0 0x7f5b901979b0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f5b8400e9c0 tx=0x7f5b8400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:06:34.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:34.488+0000 7f5b8e000640 1 -- 172.21.15.28:0/1808503876 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b8400ccc0 con 0x7f5b901016d0 2024-05-12T22:06:34.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:34.488+0000 7f5b8e000640 1 -- 172.21.15.28:0/1808503876 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5b840035f0 con 0x7f5b901016d0 2024-05-12T22:06:34.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:34.488+0000 7f5b8e000640 1 -- 172.21.15.28:0/1808503876 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b8400ce60 con 0x7f5b901016d0 2024-05-12T22:06:34.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:34.488+0000 7f5b96c00640 1 -- 172.21.15.28:0/1808503876 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b9019f8c0 con 0x7f5b901016d0 2024-05-12T22:06:34.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:34.489+0000 7f5b96c00640 1 -- 172.21.15.28:0/1808503876 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b9019fdc0 con 0x7f5b901016d0 2024-05-12T22:06:34.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:34.490+0000 7f5b96c00640 1 -- 172.21.15.28:0/1808503876 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b901089f0 con 0x7f5b901016d0 2024-05-12T22:06:34.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:34.492+0000 7f5b8e000640 1 -- 172.21.15.28:0/1808503876 <== mon.2 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f5b840037a0 con 0x7f5b901016d0 2024-05-12T22:06:34.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:34.493+0000 7f5b8e000640 1 --2- 172.21.15.28:0/1808503876 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f5b780a2b90 0x7f5b780a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:34.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:34.494+0000 7f5b8e000640 1 -- 172.21.15.28:0/1808503876 <== mon.2 v2:172.21.15.86:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f5b840cf490 con 0x7f5b901016d0 2024-05-12T22:06:34.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:34.494+0000 7f5b94e00640 1 --2- 172.21.15.28:0/1808503876 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f5b780a2b90 0x7f5b780a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:34.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:34.495+0000 7f5b94e00640 1 --2- 172.21.15.28:0/1808503876 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f5b780a2b90 0x7f5b780a5050 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f5b90198ec0 tx=0x7f5b7c0315b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:06:34.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:34.498+0000 7f5b8e000640 1 -- 172.21.15.28:0/1808503876 <== mon.2 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f5b8409c4a0 con 0x7f5b901016d0 2024-05-12T22:06:34.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:34.703+0000 7f5b96c00640 1 -- 172.21.15.28:0/1808503876 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5b9002d1c0 con 0x7f5b901016d0 2024-05-12T22:06:34.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:34.705+0000 7f5b8e000640 1 -- 172.21.15.28:0/1808503876 <== mon.2 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) ==== 74+0+22173 (secure 0 0 0) 0x7f5b8409bda0 con 0x7f5b901016d0 2024-05-12T22:06:34.707 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-12T22:06:34.707 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":85,"fsid":"6b92e7b4-10aa-11ef-bc98-c7b262605968","created":"2024-05-12T21:57:37.276782+0000","modified":"2024-05-12T22:06:33.426586+0000","last_up_change":"2024-05-12T22:05:37.929562+0000","last_in_change":"2024-05-12T22:05:18.750768+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-12T22:01:32.292826+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-12T22:06:31.540263+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-12T22:06:32.906118+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6992ce0a-844b-40e3-8ed1-181c8474dccf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":906003882},{"type":"v1","addr":"172.21.15.28:6803","nonce":906003882}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":906003882},{"type":"v1","addr":"172.21.15.28:6805","nonce":906003882}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":906003882},{"type":"v1","addr":"172.21.15.28:6809","nonce":906003882}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":906003882},{"type":"v1","addr":"172.21.15.28:6807","nonce":906003882}]},"public_addr":"172.21.15.28:6803/906003882","cluster_addr":"172.21.15.28:6805/906003882","heartbeat_back_addr":"172.21.15.28:6809/906003882","heartbeat_front_addr":"172.21.15.28:6807/906003882","state":["exists","up"]},{"osd":1,"uuid":"ced20c96-c481-4d48-90d1-6b43228959ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":3354141819},{"type":"v1","addr":"172.21.15.86:6801","nonce":3354141819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":3354141819},{"type":"v1","addr":"172.21.15.86:6803","nonce":3354141819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":3354141819},{"type":"v1","addr":"172.21.15.86:6807","nonce":3354141819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":3354141819},{"type":"v1","addr":"172.21.15.86:6805","nonce":3354141819}]},"public_addr":"172.21.15.86:6801/3354141819","cluster_addr":"172.21.15.86:6803/3354141819","heartbeat_back_addr":"172.21.15.86:6807/3354141819","heartbeat_front_addr":"172.21.15.86:6805/3354141819","state":["exists","up"]},{"osd":2,"uuid":"54a2f221-fa65-4795-9f5e-e660cc7c0098","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":906459365},{"type":"v1","addr":"172.21.15.186:6801","nonce":906459365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":906459365},{"type":"v1","addr":"172.21.15.186:6803","nonce":906459365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":906459365},{"type":"v1","addr":"172.21.15.186:6807","nonce":906459365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":906459365},{"type":"v1","addr":"172.21.15.186:6805","nonce":906459365}]},"public_addr":"172.21.15.186:6801/906459365","cluster_addr":"172.21.15.186:6803/906459365","heartbeat_back_addr":"172.21.15.186:6807/906459365","heartbeat_front_addr":"172.21.15.186:6805/906459365","state":["exists","up"]},{"osd":3,"uuid":"86836737-ac9a-487b-93f2-7fcd3c406960","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":4182884436},{"type":"v1","addr":"172.21.15.28:6811","nonce":4182884436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":4182884436},{"type":"v1","addr":"172.21.15.28:6813","nonce":4182884436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":4182884436},{"type":"v1","addr":"172.21.15.28:6817","nonce":4182884436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":4182884436},{"type":"v1","addr":"172.21.15.28:6815","nonce":4182884436}]},"public_addr":"172.21.15.28:6811/4182884436","cluster_addr":"172.21.15.28:6813/4182884436","heartbeat_back_addr":"172.21.15.28:6817/4182884436","heartbeat_front_addr":"172.21.15.28:6815/4182884436","state":["exists","up"]},{"osd":4,"uuid":"d162ebcd-83c0-4fd5-9925-28b2cecea935","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2596998708},{"type":"v1","addr":"172.21.15.86:6809","nonce":2596998708}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2596998708},{"type":"v1","addr":"172.21.15.86:6811","nonce":2596998708}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2596998708},{"type":"v1","addr":"172.21.15.86:6815","nonce":2596998708}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2596998708},{"type":"v1","addr":"172.21.15.86:6813","nonce":2596998708}]},"public_addr":"172.21.15.86:6809/2596998708","cluster_addr":"172.21.15.86:6811/2596998708","heartbeat_back_addr":"172.21.15.86:6815/2596998708","heartbeat_front_addr":"172.21.15.86:6813/2596998708","state":["exists","up"]},{"osd":5,"uuid":"c794336a-a105-425c-b6c7-a32b90c05bb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":1944205756},{"type":"v1","addr":"172.21.15.186:6809","nonce":1944205756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":1944205756},{"type":"v1","addr":"172.21.15.186:6811","nonce":1944205756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":1944205756},{"type":"v1","addr":"172.21.15.186:6815","nonce":1944205756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":1944205756},{"type":"v1","addr":"172.21.15.186:6813","nonce":1944205756}]},"public_addr":"172.21.15.186:6809/1944205756","cluster_addr":"172.21.15.186:6811/1944205756","heartbeat_back_addr":"172.21.15.186:6815/1944205756","heartbeat_front_addr":"172.21.15.186:6813/1944205756","state":["exists","up"]},{"osd":6,"uuid":"516932ac-af00-45b7-ad3c-d6dec7db442f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":1642308925},{"type":"v1","addr":"172.21.15.28:6819","nonce":1642308925}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":1642308925},{"type":"v1","addr":"172.21.15.28:6821","nonce":1642308925}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1642308925},{"type":"v1","addr":"172.21.15.28:6825","nonce":1642308925}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":1642308925},{"type":"v1","addr":"172.21.15.28:6823","nonce":1642308925}]},"public_addr":"172.21.15.28:6819/1642308925","cluster_addr":"172.21.15.28:6821/1642308925","heartbeat_back_addr":"172.21.15.28:6825/1642308925","heartbeat_front_addr":"172.21.15.28:6823/1642308925","state":["exists","up"]},{"osd":7,"uuid":"ca7a059f-0414-438e-b2d1-a10f348036be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":726062711},{"type":"v1","addr":"172.21.15.86:6817","nonce":726062711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":726062711},{"type":"v1","addr":"172.21.15.86:6819","nonce":726062711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":726062711},{"type":"v1","addr":"172.21.15.86:6823","nonce":726062711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":726062711},{"type":"v1","addr":"172.21.15.86:6821","nonce":726062711}]},"public_addr":"172.21.15.86:6817/726062711","cluster_addr":"172.21.15.86:6819/726062711","heartbeat_back_addr":"172.21.15.86:6823/726062711","heartbeat_front_addr":"172.21.15.86:6821/726062711","state":["exists","up"]},{"osd":8,"uuid":"92e3f41f-aebd-422e-a99e-555f7ba4aa53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":3007687277},{"type":"v1","addr":"172.21.15.186:6817","nonce":3007687277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":3007687277},{"type":"v1","addr":"172.21.15.186:6819","nonce":3007687277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":3007687277},{"type":"v1","addr":"172.21.15.186:6823","nonce":3007687277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":3007687277},{"type":"v1","addr":"172.21.15.186:6821","nonce":3007687277}]},"public_addr":"172.21.15.186:6817/3007687277","cluster_addr":"172.21.15.186:6819/3007687277","heartbeat_back_addr":"172.21.15.186:6823/3007687277","heartbeat_front_addr":"172.21.15.186:6821/3007687277","state":["exists","up"]},{"osd":9,"uuid":"38ff7d9d-ee85-404c-8c59-adc327e1ccfa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":136220472},{"type":"v1","addr":"172.21.15.28:6827","nonce":136220472}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":136220472},{"type":"v1","addr":"172.21.15.28:6829","nonce":136220472}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":136220472},{"type":"v1","addr":"172.21.15.28:6833","nonce":136220472}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":136220472},{"type":"v1","addr":"172.21.15.28:6831","nonce":136220472}]},"public_addr":"172.21.15.28:6827/136220472","cluster_addr":"172.21.15.28:6829/136220472","heartbeat_back_addr":"172.21.15.28:6833/136220472","heartbeat_front_addr":"172.21.15.28:6831/136220472","state":["exists","up"]},{"osd":10,"uuid":"5e9ff657-afa8-4bab-8491-f209038392b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1189968310},{"type":"v1","addr":"172.21.15.86:6825","nonce":1189968310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":1189968310},{"type":"v1","addr":"172.21.15.86:6827","nonce":1189968310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":1189968310},{"type":"v1","addr":"172.21.15.86:6831","nonce":1189968310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":1189968310},{"type":"v1","addr":"172.21.15.86:6829","nonce":1189968310}]},"public_addr":"172.21.15.86:6825/1189968310","cluster_addr":"172.21.15.86:6827/1189968310","heartbeat_back_addr":"172.21.15.86:6831/1189968310","heartbeat_front_addr":"172.21.15.86:6829/1189968310","state":["exists","up"]},{"osd":11,"uuid":"f112e3c6-65b6-4253-a966-62001db7d069","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3801668110},{"type":"v1","addr":"172.21.15.186:6825","nonce":3801668110}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3801668110},{"type":"v1","addr":"172.21.15.186:6827","nonce":3801668110}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3801668110},{"type":"v1","addr":"172.21.15.186:6831","nonce":3801668110}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3801668110},{"type":"v1","addr":"172.21.15.186:6829","nonce":3801668110}]},"public_addr":"172.21.15.186:6825/3801668110","cluster_addr":"172.21.15.186:6827/3801668110","heartbeat_back_addr":"172.21.15.186:6831/3801668110","heartbeat_front_addr":"172.21.15.186:6829/3801668110","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:00:40.483623+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:01:05.284385+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:01:28.645716+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:02:01.295092+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:02:26.486601+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:02:50.807478+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:03:24.756169+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:03:49.975173+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:04:13.772233+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:04:46.758555+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:05:12.331685+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:05:36.462548+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/1177009533":"2024-05-13T21:58:45.550459+0000","172.21.15.28:6800/3760499540":"2024-05-13T21:58:45.550459+0000","172.21.15.28:6801/3783742728":"2024-05-13T21:58:10.421564+0000","172.21.15.28:6800/3783742728":"2024-05-13T21:58:10.421564+0000","172.21.15.28:0/2229156260":"2024-05-13T21:58:45.550459+0000","172.21.15.28:0/1202140963":"2024-05-13T21:58:10.421564+0000","172.21.15.28:0/2924223784":"2024-05-13T21:58:45.550459+0000","172.21.15.28:6801/3760499540":"2024-05-13T21:58:45.550459+0000","172.21.15.28:0/1240027206":"2024-05-13T21:58:10.421564+0000","172.21.15.28:0/171417728":"2024-05-13T21:58:10.421564+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-12T22:06:34.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:34.708+0000 7f5b96c00640 1 -- 172.21.15.28:0/1808503876 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f5b780a2b90 msgr2=0x7f5b780a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:34.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:34.708+0000 7f5b96c00640 1 --2- 172.21.15.28:0/1808503876 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f5b780a2b90 0x7f5b780a5050 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f5b90198ec0 tx=0x7f5b7c0315b0 comp rx=0 tx=0).stop 2024-05-12T22:06:34.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:34.708+0000 7f5b96c00640 1 -- 172.21.15.28:0/1808503876 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5b901016d0 msgr2=0x7f5b901979b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:34.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:34.708+0000 7f5b96c00640 1 --2- 172.21.15.28:0/1808503876 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5b901016d0 0x7f5b901979b0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f5b8400e9c0 tx=0x7f5b8400ee80 comp rx=0 tx=0).stop 2024-05-12T22:06:34.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:34.708+0000 7f5b96c00640 1 -- 172.21.15.28:0/1808503876 shutdown_connections 2024-05-12T22:06:34.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:34.708+0000 7f5b96c00640 1 --2- 172.21.15.28:0/1808503876 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f5b780a2b90 0x7f5b780a5050 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:34.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:34.708+0000 7f5b96c00640 1 --2- 172.21.15.28:0/1808503876 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5b90103280 0x7f5b9019ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:34.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:34.708+0000 7f5b96c00640 1 --2- 172.21.15.28:0/1808503876 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b901028c0 0x7f5b90197ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:34.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:34.708+0000 7f5b96c00640 1 --2- 172.21.15.28:0/1808503876 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5b901016d0 0x7f5b901979b0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:34.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:34.708+0000 7f5b96c00640 1 -- 172.21.15.28:0/1808503876 >> 172.21.15.28:0/1808503876 conn(0x7f5b900fcc40 msgr2=0x7f5b90105bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:06:34.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:34.708+0000 7f5b96c00640 1 -- 172.21.15.28:0/1808503876 shutdown_connections 2024-05-12T22:06:34.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:34.708+0000 7f5b96c00640 1 -- 172.21.15.28:0/1808503876 wait complete. 2024-05-12T22:06:34.729 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-12T22:06:34.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:34.931+0000 7fed43600640 1 -- 172.21.15.28:0/2834531005 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed3c0761f0 msgr2=0x7fed3c076610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:34.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:34.931+0000 7fed43600640 1 --2- 172.21.15.28:0/2834531005 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed3c0761f0 0x7fed3c076610 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fed30009900 tx=0x7fed3002a3f0 comp rx=0 tx=0).stop 2024-05-12T22:06:34.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:34.932+0000 7fed43600640 1 -- 172.21.15.28:0/2834531005 shutdown_connections 2024-05-12T22:06:34.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:34.932+0000 7fed43600640 1 --2- 172.21.15.28:0/2834531005 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fed3c077dd0 0x7fed3c078670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:34.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:34.932+0000 7fed43600640 1 --2- 172.21.15.28:0/2834531005 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fed3c077430 0x7fed3c077890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:34.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:34.932+0000 7fed43600640 1 --2- 172.21.15.28:0/2834531005 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed3c0761f0 0x7fed3c076610 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:34.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:34.932+0000 7fed43600640 1 -- 172.21.15.28:0/2834531005 >> 172.21.15.28:0/2834531005 conn(0x7fed3c0ff020 msgr2=0x7fed3c101460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:06:34.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:34.932+0000 7fed43600640 1 -- 172.21.15.28:0/2834531005 shutdown_connections 2024-05-12T22:06:34.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:34.932+0000 7fed43600640 1 -- 172.21.15.28:0/2834531005 wait complete. 2024-05-12T22:06:34.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:34.933+0000 7fed43600640 1 Processor -- start 2024-05-12T22:06:34.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:34.933+0000 7fed43600640 1 -- start start 2024-05-12T22:06:34.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:34.933+0000 7fed43600640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fed3c0761f0 0x7fed3c19e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:34.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:34.933+0000 7fed43600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fed3c077430 0x7fed3c19e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:34.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:34.933+0000 7fed43600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed3c077dd0 0x7fed3c1a5730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:34.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:34.933+0000 7fed43600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fed3c113b80 con 0x7fed3c077dd0 2024-05-12T22:06:34.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:34.933+0000 7fed43600640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fed3c113a00 con 0x7fed3c077430 2024-05-12T22:06:34.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:34.933+0000 7fed43600640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fed3c113d00 con 0x7fed3c0761f0 2024-05-12T22:06:34.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:34.933+0000 7fed41600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed3c077dd0 0x7fed3c1a5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:34.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:34.933+0000 7fed41600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed3c077dd0 0x7fed3c1a5730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55468/0 (socket says 172.21.15.28:55468) 2024-05-12T22:06:34.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:34.933+0000 7fed41600640 1 -- 172.21.15.28:0/81648135 learned_addr learned my addr 172.21.15.28:0/81648135 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:06:34.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:34.933+0000 7fed3be00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fed3c077430 0x7fed3c19e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:34.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:34.933+0000 7fed40c00640 1 --2- 172.21.15.28:0/81648135 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fed3c0761f0 0x7fed3c19e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:34.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:34.934+0000 7fed41600640 1 -- 172.21.15.28:0/81648135 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fed3c0761f0 msgr2=0x7fed3c19e180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:34.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:34.934+0000 7fed41600640 1 --2- 172.21.15.28:0/81648135 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fed3c0761f0 0x7fed3c19e180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:34.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:34.934+0000 7fed41600640 1 -- 172.21.15.28:0/81648135 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fed3c077430 msgr2=0x7fed3c19e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:34.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:34.934+0000 7fed41600640 1 --2- 172.21.15.28:0/81648135 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fed3c077430 0x7fed3c19e6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:34.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:34.934+0000 7fed41600640 1 -- 172.21.15.28:0/81648135 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed3c1a5da0 con 0x7fed3c077dd0 2024-05-12T22:06:34.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:34.934+0000 7fed41600640 1 --2- 172.21.15.28:0/81648135 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed3c077dd0 0x7fed3c1a5730 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fed3400eaa0 tx=0x7fed3400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:06:34.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:34.934+0000 7fed39600640 1 -- 172.21.15.28:0/81648135 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed3400cdb0 con 0x7fed3c077dd0 2024-05-12T22:06:34.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:34.934+0000 7fed39600640 1 -- 172.21.15.28:0/81648135 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fed34005020 con 0x7fed3c077dd0 2024-05-12T22:06:34.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:34.934+0000 7fed43600640 1 -- 172.21.15.28:0/81648135 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed3c1a6090 con 0x7fed3c077dd0 2024-05-12T22:06:34.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:34.935+0000 7fed39600640 1 -- 172.21.15.28:0/81648135 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed340116b0 con 0x7fed3c077dd0 2024-05-12T22:06:34.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:34.935+0000 7fed43600640 1 -- 172.21.15.28:0/81648135 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed3c1a65e0 con 0x7fed3c077dd0 2024-05-12T22:06:34.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:34.936+0000 7fed43600640 1 -- 172.21.15.28:0/81648135 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed3c110570 con 0x7fed3c077dd0 2024-05-12T22:06:34.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:34.938+0000 7fed39600640 1 -- 172.21.15.28:0/81648135 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fed34005560 con 0x7fed3c077dd0 2024-05-12T22:06:34.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:34.939+0000 7fed39600640 1 --2- 172.21.15.28:0/81648135 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fed240a2b90 0x7fed240a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:34.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:34.940+0000 7fed39600640 1 -- 172.21.15.28:0/81648135 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7fed340cf150 con 0x7fed3c077dd0 2024-05-12T22:06:34.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:34.940+0000 7fed40c00640 1 --2- 172.21.15.28:0/81648135 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fed240a2b90 0x7fed240a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:34.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:34.941+0000 7fed40c00640 1 --2- 172.21.15.28:0/81648135 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fed240a2b90 0x7fed240a5050 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fed30031440 tx=0x7fed300315b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:06:34.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:34.944+0000 7fed39600640 1 -- 172.21.15.28:0/81648135 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fed34016030 con 0x7fed3c077dd0 2024-05-12T22:06:35.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:34 smithi028 ceph-mon[29847]: pgmap v308: 97 pgs: 22 creating+peering, 74 unknown, 1 active+clean; 577 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-12T22:06:35.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:34 smithi028 ceph-mon[29847]: osdmap e85: 12 total, 12 up, 12 in 2024-05-12T22:06:35.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:34 smithi028 ceph-mon[29847]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-12T22:06:35.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:34 smithi028 ceph-mon[29847]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-05-12T22:06:35.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:34 smithi028 ceph-mon[29847]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-05-12T22:06:35.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:34 smithi028 ceph-mon[29847]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-05-12T22:06:35.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:34 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:35.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:34 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:35.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:34 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:06:35.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:34 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:06:35.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:34 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:35.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:34 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/1808503876' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-12T22:06:35.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:35.153+0000 7fed43600640 1 -- 172.21.15.28:0/81648135 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fed3c062430 con 0x7fed3c077dd0 2024-05-12T22:06:35.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:35.157+0000 7fed39600640 1 -- 172.21.15.28:0/81648135 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) ==== 74+0+22160 (secure 0 0 0) 0x7fed3409bb10 con 0x7fed3c077dd0 2024-05-12T22:06:35.159 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-12T22:06:35.159 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":86,"fsid":"6b92e7b4-10aa-11ef-bc98-c7b262605968","created":"2024-05-12T21:57:37.276782+0000","modified":"2024-05-12T22:06:34.945595+0000","last_up_change":"2024-05-12T22:05:37.929562+0000","last_in_change":"2024-05-12T22:05:18.750768+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-12T22:01:32.292826+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-12T22:06:31.540263+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-12T22:06:32.906118+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6992ce0a-844b-40e3-8ed1-181c8474dccf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":906003882},{"type":"v1","addr":"172.21.15.28:6803","nonce":906003882}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":906003882},{"type":"v1","addr":"172.21.15.28:6805","nonce":906003882}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":906003882},{"type":"v1","addr":"172.21.15.28:6809","nonce":906003882}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":906003882},{"type":"v1","addr":"172.21.15.28:6807","nonce":906003882}]},"public_addr":"172.21.15.28:6803/906003882","cluster_addr":"172.21.15.28:6805/906003882","heartbeat_back_addr":"172.21.15.28:6809/906003882","heartbeat_front_addr":"172.21.15.28:6807/906003882","state":["exists","up"]},{"osd":1,"uuid":"ced20c96-c481-4d48-90d1-6b43228959ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":3354141819},{"type":"v1","addr":"172.21.15.86:6801","nonce":3354141819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":3354141819},{"type":"v1","addr":"172.21.15.86:6803","nonce":3354141819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":3354141819},{"type":"v1","addr":"172.21.15.86:6807","nonce":3354141819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":3354141819},{"type":"v1","addr":"172.21.15.86:6805","nonce":3354141819}]},"public_addr":"172.21.15.86:6801/3354141819","cluster_addr":"172.21.15.86:6803/3354141819","heartbeat_back_addr":"172.21.15.86:6807/3354141819","heartbeat_front_addr":"172.21.15.86:6805/3354141819","state":["exists","up"]},{"osd":2,"uuid":"54a2f221-fa65-4795-9f5e-e660cc7c0098","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":906459365},{"type":"v1","addr":"172.21.15.186:6801","nonce":906459365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":906459365},{"type":"v1","addr":"172.21.15.186:6803","nonce":906459365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":906459365},{"type":"v1","addr":"172.21.15.186:6807","nonce":906459365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":906459365},{"type":"v1","addr":"172.21.15.186:6805","nonce":906459365}]},"public_addr":"172.21.15.186:6801/906459365","cluster_addr":"172.21.15.186:6803/906459365","heartbeat_back_addr":"172.21.15.186:6807/906459365","heartbeat_front_addr":"172.21.15.186:6805/906459365","state":["exists","up"]},{"osd":3,"uuid":"86836737-ac9a-487b-93f2-7fcd3c406960","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":4182884436},{"type":"v1","addr":"172.21.15.28:6811","nonce":4182884436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":4182884436},{"type":"v1","addr":"172.21.15.28:6813","nonce":4182884436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":4182884436},{"type":"v1","addr":"172.21.15.28:6817","nonce":4182884436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":4182884436},{"type":"v1","addr":"172.21.15.28:6815","nonce":4182884436}]},"public_addr":"172.21.15.28:6811/4182884436","cluster_addr":"172.21.15.28:6813/4182884436","heartbeat_back_addr":"172.21.15.28:6817/4182884436","heartbeat_front_addr":"172.21.15.28:6815/4182884436","state":["exists","up"]},{"osd":4,"uuid":"d162ebcd-83c0-4fd5-9925-28b2cecea935","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2596998708},{"type":"v1","addr":"172.21.15.86:6809","nonce":2596998708}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2596998708},{"type":"v1","addr":"172.21.15.86:6811","nonce":2596998708}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2596998708},{"type":"v1","addr":"172.21.15.86:6815","nonce":2596998708}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2596998708},{"type":"v1","addr":"172.21.15.86:6813","nonce":2596998708}]},"public_addr":"172.21.15.86:6809/2596998708","cluster_addr":"172.21.15.86:6811/2596998708","heartbeat_back_addr":"172.21.15.86:6815/2596998708","heartbeat_front_addr":"172.21.15.86:6813/2596998708","state":["exists","up"]},{"osd":5,"uuid":"c794336a-a105-425c-b6c7-a32b90c05bb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":1944205756},{"type":"v1","addr":"172.21.15.186:6809","nonce":1944205756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":1944205756},{"type":"v1","addr":"172.21.15.186:6811","nonce":1944205756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":1944205756},{"type":"v1","addr":"172.21.15.186:6815","nonce":1944205756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":1944205756},{"type":"v1","addr":"172.21.15.186:6813","nonce":1944205756}]},"public_addr":"172.21.15.186:6809/1944205756","cluster_addr":"172.21.15.186:6811/1944205756","heartbeat_back_addr":"172.21.15.186:6815/1944205756","heartbeat_front_addr":"172.21.15.186:6813/1944205756","state":["exists","up"]},{"osd":6,"uuid":"516932ac-af00-45b7-ad3c-d6dec7db442f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":1642308925},{"type":"v1","addr":"172.21.15.28:6819","nonce":1642308925}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":1642308925},{"type":"v1","addr":"172.21.15.28:6821","nonce":1642308925}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1642308925},{"type":"v1","addr":"172.21.15.28:6825","nonce":1642308925}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":1642308925},{"type":"v1","addr":"172.21.15.28:6823","nonce":1642308925}]},"public_addr":"172.21.15.28:6819/1642308925","cluster_addr":"172.21.15.28:6821/1642308925","heartbeat_back_addr":"172.21.15.28:6825/1642308925","heartbeat_front_addr":"172.21.15.28:6823/1642308925","state":["exists","up"]},{"osd":7,"uuid":"ca7a059f-0414-438e-b2d1-a10f348036be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":726062711},{"type":"v1","addr":"172.21.15.86:6817","nonce":726062711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":726062711},{"type":"v1","addr":"172.21.15.86:6819","nonce":726062711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":726062711},{"type":"v1","addr":"172.21.15.86:6823","nonce":726062711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":726062711},{"type":"v1","addr":"172.21.15.86:6821","nonce":726062711}]},"public_addr":"172.21.15.86:6817/726062711","cluster_addr":"172.21.15.86:6819/726062711","heartbeat_back_addr":"172.21.15.86:6823/726062711","heartbeat_front_addr":"172.21.15.86:6821/726062711","state":["exists","up"]},{"osd":8,"uuid":"92e3f41f-aebd-422e-a99e-555f7ba4aa53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":3007687277},{"type":"v1","addr":"172.21.15.186:6817","nonce":3007687277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":3007687277},{"type":"v1","addr":"172.21.15.186:6819","nonce":3007687277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":3007687277},{"type":"v1","addr":"172.21.15.186:6823","nonce":3007687277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":3007687277},{"type":"v1","addr":"172.21.15.186:6821","nonce":3007687277}]},"public_addr":"172.21.15.186:6817/3007687277","cluster_addr":"172.21.15.186:6819/3007687277","heartbeat_back_addr":"172.21.15.186:6823/3007687277","heartbeat_front_addr":"172.21.15.186:6821/3007687277","state":["exists","up"]},{"osd":9,"uuid":"38ff7d9d-ee85-404c-8c59-adc327e1ccfa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":136220472},{"type":"v1","addr":"172.21.15.28:6827","nonce":136220472}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":136220472},{"type":"v1","addr":"172.21.15.28:6829","nonce":136220472}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":136220472},{"type":"v1","addr":"172.21.15.28:6833","nonce":136220472}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":136220472},{"type":"v1","addr":"172.21.15.28:6831","nonce":136220472}]},"public_addr":"172.21.15.28:6827/136220472","cluster_addr":"172.21.15.28:6829/136220472","heartbeat_back_addr":"172.21.15.28:6833/136220472","heartbeat_front_addr":"172.21.15.28:6831/136220472","state":["exists","up"]},{"osd":10,"uuid":"5e9ff657-afa8-4bab-8491-f209038392b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1189968310},{"type":"v1","addr":"172.21.15.86:6825","nonce":1189968310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":1189968310},{"type":"v1","addr":"172.21.15.86:6827","nonce":1189968310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":1189968310},{"type":"v1","addr":"172.21.15.86:6831","nonce":1189968310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":1189968310},{"type":"v1","addr":"172.21.15.86:6829","nonce":1189968310}]},"public_addr":"172.21.15.86:6825/1189968310","cluster_addr":"172.21.15.86:6827/1189968310","heartbeat_back_addr":"172.21.15.86:6831/1189968310","heartbeat_front_addr":"172.21.15.86:6829/1189968310","state":["exists","up"]},{"osd":11,"uuid":"f112e3c6-65b6-4253-a966-62001db7d069","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3801668110},{"type":"v1","addr":"172.21.15.186:6825","nonce":3801668110}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3801668110},{"type":"v1","addr":"172.21.15.186:6827","nonce":3801668110}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3801668110},{"type":"v1","addr":"172.21.15.186:6831","nonce":3801668110}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3801668110},{"type":"v1","addr":"172.21.15.186:6829","nonce":3801668110}]},"public_addr":"172.21.15.186:6825/3801668110","cluster_addr":"172.21.15.186:6827/3801668110","heartbeat_back_addr":"172.21.15.186:6831/3801668110","heartbeat_front_addr":"172.21.15.186:6829/3801668110","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:00:40.483623+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:01:05.284385+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:01:28.645716+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:02:01.295092+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:02:26.486601+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:02:50.807478+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:03:24.756169+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:03:49.975173+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:04:13.772233+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:04:46.758555+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:05:12.331685+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:05:36.462548+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/1177009533":"2024-05-13T21:58:45.550459+0000","172.21.15.28:6800/3760499540":"2024-05-13T21:58:45.550459+0000","172.21.15.28:6801/3783742728":"2024-05-13T21:58:10.421564+0000","172.21.15.28:6800/3783742728":"2024-05-13T21:58:10.421564+0000","172.21.15.28:0/2229156260":"2024-05-13T21:58:45.550459+0000","172.21.15.28:0/1202140963":"2024-05-13T21:58:10.421564+0000","172.21.15.28:0/2924223784":"2024-05-13T21:58:45.550459+0000","172.21.15.28:6801/3760499540":"2024-05-13T21:58:45.550459+0000","172.21.15.28:0/1240027206":"2024-05-13T21:58:10.421564+0000","172.21.15.28:0/171417728":"2024-05-13T21:58:10.421564+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-12T22:06:35.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:35.160+0000 7fed43600640 1 -- 172.21.15.28:0/81648135 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fed240a2b90 msgr2=0x7fed240a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:35.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:35.160+0000 7fed43600640 1 --2- 172.21.15.28:0/81648135 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fed240a2b90 0x7fed240a5050 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fed30031440 tx=0x7fed300315b0 comp rx=0 tx=0).stop 2024-05-12T22:06:35.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:35.160+0000 7fed43600640 1 -- 172.21.15.28:0/81648135 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed3c077dd0 msgr2=0x7fed3c1a5730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:35.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:35.160+0000 7fed43600640 1 --2- 172.21.15.28:0/81648135 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed3c077dd0 0x7fed3c1a5730 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fed3400eaa0 tx=0x7fed3400ef60 comp rx=0 tx=0).stop 2024-05-12T22:06:35.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:35.160+0000 7fed43600640 1 -- 172.21.15.28:0/81648135 shutdown_connections 2024-05-12T22:06:35.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:35.160+0000 7fed43600640 1 --2- 172.21.15.28:0/81648135 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fed240a2b90 0x7fed240a5050 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:35.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:35.160+0000 7fed43600640 1 --2- 172.21.15.28:0/81648135 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed3c077dd0 0x7fed3c1a5730 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:35.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:35.160+0000 7fed43600640 1 --2- 172.21.15.28:0/81648135 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fed3c077430 0x7fed3c19e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:35.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:35.160+0000 7fed43600640 1 --2- 172.21.15.28:0/81648135 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fed3c0761f0 0x7fed3c19e180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:35.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:35.160+0000 7fed43600640 1 -- 172.21.15.28:0/81648135 >> 172.21.15.28:0/81648135 conn(0x7fed3c0ff020 msgr2=0x7fed3c100740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:06:35.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:35.160+0000 7fed43600640 1 -- 172.21.15.28:0/81648135 shutdown_connections 2024-05-12T22:06:35.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:35.160+0000 7fed43600640 1 -- 172.21.15.28:0/81648135 wait complete. 2024-05-12T22:06:35.181 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-05-12T22:06:35.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:34 smithi086 ceph-mon[34297]: pgmap v308: 97 pgs: 22 creating+peering, 74 unknown, 1 active+clean; 577 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-12T22:06:35.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:34 smithi086 ceph-mon[34297]: osdmap e85: 12 total, 12 up, 12 in 2024-05-12T22:06:35.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:34 smithi086 ceph-mon[34297]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-12T22:06:35.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:34 smithi086 ceph-mon[34297]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-05-12T22:06:35.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:34 smithi086 ceph-mon[34297]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-05-12T22:06:35.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:34 smithi086 ceph-mon[34297]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-05-12T22:06:35.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:34 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:35.244 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:34 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:35.244 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:34 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:06:35.244 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:34 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:06:35.244 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:34 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:35.244 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:34 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/1808503876' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-12T22:06:35.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:35.404+0000 7ff0b5200640 1 -- 172.21.15.28:0/2613274083 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff0b0103420 msgr2=0x7ff0b0109cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:35.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:35.404+0000 7ff0b5200640 1 --2- 172.21.15.28:0/2613274083 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff0b0103420 0x7ff0b0109cb0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7ff0a8009be0 tx=0x7ff0a802a6c0 comp rx=0 tx=0).stop 2024-05-12T22:06:35.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:35.404+0000 7ff0b5200640 1 -- 172.21.15.28:0/2613274083 shutdown_connections 2024-05-12T22:06:35.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:35.404+0000 7ff0b5200640 1 --2- 172.21.15.28:0/2613274083 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff0b0103420 0x7ff0b0109cb0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:35.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:35.404+0000 7ff0b5200640 1 --2- 172.21.15.28:0/2613274083 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff0b0102a60 0x7ff0b0102ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:35.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:35.404+0000 7ff0b5200640 1 --2- 172.21.15.28:0/2613274083 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0b0101870 0x7ff0b0101c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:35.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:35.404+0000 7ff0b5200640 1 -- 172.21.15.28:0/2613274083 >> 172.21.15.28:0/2613274083 conn(0x7ff0b00fcde0 msgr2=0x7ff0b00ff220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:06:35.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:35.404+0000 7ff0b5200640 1 -- 172.21.15.28:0/2613274083 shutdown_connections 2024-05-12T22:06:35.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:35.404+0000 7ff0b5200640 1 -- 172.21.15.28:0/2613274083 wait complete. 2024-05-12T22:06:35.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:35.405+0000 7ff0b5200640 1 Processor -- start 2024-05-12T22:06:35.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:35.406+0000 7ff0b5200640 1 -- start start 2024-05-12T22:06:35.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:35.406+0000 7ff0b5200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff0b0101870 0x7ff0b0197b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:35.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:35.406+0000 7ff0b5200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0b0102a60 0x7ff0b0198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:35.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:35.406+0000 7ff0b5200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff0b0103420 0x7ff0b019f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:35.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:35.406+0000 7ff0b5200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff0b010d4d0 con 0x7ff0b0102a60 2024-05-12T22:06:35.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:35.406+0000 7ff0b5200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7ff0b010d350 con 0x7ff0b0101870 2024-05-12T22:06:35.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:35.406+0000 7ff0b5200640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff0b010d650 con 0x7ff0b0103420 2024-05-12T22:06:35.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:35.406+0000 7ff0a7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0b0102a60 0x7ff0b0198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:35.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:35.406+0000 7ff0ae800640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff0b0103420 0x7ff0b019f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:35.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:35.406+0000 7ff0ae800640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff0b0103420 0x7ff0b019f0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:45428/0 (socket says 172.21.15.28:45428) 2024-05-12T22:06:35.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:35.406+0000 7ff0a7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0b0102a60 0x7ff0b0198060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55486/0 (socket says 172.21.15.28:55486) 2024-05-12T22:06:35.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:35.406+0000 7ff0ae800640 1 -- 172.21.15.28:0/386348958 learned_addr learned my addr 172.21.15.28:0/386348958 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:06:35.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:35.406+0000 7ff0ade00640 1 --2- 172.21.15.28:0/386348958 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff0b0101870 0x7ff0b0197b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:35.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:35.407+0000 7ff0ae800640 1 -- 172.21.15.28:0/386348958 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff0b0101870 msgr2=0x7ff0b0197b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:35.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:35.407+0000 7ff0ae800640 1 --2- 172.21.15.28:0/386348958 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff0b0101870 0x7ff0b0197b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:35.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:35.407+0000 7ff0ae800640 1 -- 172.21.15.28:0/386348958 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0b0102a60 msgr2=0x7ff0b0198060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:35.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:35.407+0000 7ff0ae800640 1 --2- 172.21.15.28:0/386348958 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0b0102a60 0x7ff0b0198060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:35.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:35.407+0000 7ff0ae800640 1 -- 172.21.15.28:0/386348958 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0b019f740 con 0x7ff0b0103420 2024-05-12T22:06:35.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:35.407+0000 7ff0ae800640 1 --2- 172.21.15.28:0/386348958 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff0b0103420 0x7ff0b019f0d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7ff0a8009b30 tx=0x7ff0a80305d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:06:35.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:35.407+0000 7ff0a6a00640 1 -- 172.21.15.28:0/386348958 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0a8041020 con 0x7ff0b0103420 2024-05-12T22:06:35.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:35.407+0000 7ff0a6a00640 1 -- 172.21.15.28:0/386348958 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff0a8030740 con 0x7ff0b0103420 2024-05-12T22:06:35.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:35.407+0000 7ff0a6a00640 1 -- 172.21.15.28:0/386348958 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0a8036040 con 0x7ff0b0103420 2024-05-12T22:06:35.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:35.407+0000 7ff0b5200640 1 -- 172.21.15.28:0/386348958 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0b0073a60 con 0x7ff0b0103420 2024-05-12T22:06:35.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:35.407+0000 7ff0b5200640 1 -- 172.21.15.28:0/386348958 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0b0073f50 con 0x7ff0b0103420 2024-05-12T22:06:35.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:35.408+0000 7ff0b5200640 1 -- 172.21.15.28:0/386348958 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0b0108b90 con 0x7ff0b0103420 2024-05-12T22:06:35.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:35.411+0000 7ff0a6a00640 1 -- 172.21.15.28:0/386348958 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7ff0a8044020 con 0x7ff0b0103420 2024-05-12T22:06:35.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:35.412+0000 7ff0a6a00640 1 --2- 172.21.15.28:0/386348958 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7ff0940a2b90 0x7ff0940a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:35.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:35.412+0000 7ff0a6a00640 1 -- 172.21.15.28:0/386348958 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7ff0a80efff0 con 0x7ff0b0103420 2024-05-12T22:06:35.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:35.412+0000 7ff0ade00640 1 --2- 172.21.15.28:0/386348958 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7ff0940a2b90 0x7ff0940a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:35.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:35.413+0000 7ff0ade00640 1 --2- 172.21.15.28:0/386348958 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7ff0940a2b90 0x7ff0940a5050 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7ff0b01028c0 tx=0x7ff0a0012040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:06:35.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:35.414+0000 7ff0a6a00640 1 -- 172.21.15.28:0/386348958 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7ff0a804b020 con 0x7ff0b0103420 2024-05-12T22:06:35.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:34 smithi186 ceph-mon[34785]: pgmap v308: 97 pgs: 22 creating+peering, 74 unknown, 1 active+clean; 577 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-12T22:06:35.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:34 smithi186 ceph-mon[34785]: osdmap e85: 12 total, 12 up, 12 in 2024-05-12T22:06:35.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:34 smithi186 ceph-mon[34785]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-12T22:06:35.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:34 smithi186 ceph-mon[34785]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-05-12T22:06:35.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:34 smithi186 ceph-mon[34785]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-05-12T22:06:35.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:34 smithi186 ceph-mon[34785]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-05-12T22:06:35.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:34 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:35.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:34 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:35.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:34 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:06:35.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:34 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:06:35.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:34 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:06:35.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:34 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/1808503876' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-12T22:06:35.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:35.664+0000 7ff0b5200640 1 -- 172.21.15.28:0/386348958 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7ff0b002d2e0 con 0x7ff0b0103420 2024-05-12T22:06:36.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:35 smithi086 ceph-mon[34297]: osdmap e86: 12 total, 12 up, 12 in 2024-05-12T22:06:36.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:35 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/81648135' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-12T22:06:36.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:35 smithi086 ceph-mon[34297]: mds.? [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] up:active 2024-05-12T22:06:36.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:35 smithi086 ceph-mon[34297]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-12T22:06:36.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:35 smithi086 ceph-mon[34297]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-12T22:06:36.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:35 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/386348958' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-05-12T22:06:36.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:35 smithi086 ceph-mon[34297]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-05-12T22:06:36.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:36.280+0000 7ff0a6a00640 1 -- 172.21.15.28:0/386348958 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v17) ==== 118+0+0 (secure 0 0 0) 0x7ff0a80bd040 con 0x7ff0b0103420 2024-05-12T22:06:36.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:36.284+0000 7ff0b5200640 1 -- 172.21.15.28:0/386348958 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7ff0940a2b90 msgr2=0x7ff0940a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:36.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:36.284+0000 7ff0b5200640 1 --2- 172.21.15.28:0/386348958 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7ff0940a2b90 0x7ff0940a5050 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7ff0b01028c0 tx=0x7ff0a0012040 comp rx=0 tx=0).stop 2024-05-12T22:06:36.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:36.284+0000 7ff0b5200640 1 -- 172.21.15.28:0/386348958 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff0b0103420 msgr2=0x7ff0b019f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:36.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:36.284+0000 7ff0b5200640 1 --2- 172.21.15.28:0/386348958 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff0b0103420 0x7ff0b019f0d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7ff0a8009b30 tx=0x7ff0a80305d0 comp rx=0 tx=0).stop 2024-05-12T22:06:36.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:36.284+0000 7ff0b5200640 1 -- 172.21.15.28:0/386348958 shutdown_connections 2024-05-12T22:06:36.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:36.284+0000 7ff0b5200640 1 --2- 172.21.15.28:0/386348958 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7ff0940a2b90 0x7ff0940a5050 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:36.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:36.284+0000 7ff0b5200640 1 --2- 172.21.15.28:0/386348958 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff0b0103420 0x7ff0b019f0d0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:36.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:36.284+0000 7ff0b5200640 1 --2- 172.21.15.28:0/386348958 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0b0102a60 0x7ff0b0198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:36.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:36.284+0000 7ff0b5200640 1 --2- 172.21.15.28:0/386348958 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff0b0101870 0x7ff0b0197b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:36.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:36.284+0000 7ff0b5200640 1 -- 172.21.15.28:0/386348958 >> 172.21.15.28:0/386348958 conn(0x7ff0b00fcde0 msgr2=0x7ff0b0105d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:06:36.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:35 smithi028 ceph-mon[29847]: osdmap e86: 12 total, 12 up, 12 in 2024-05-12T22:06:36.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:35 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/81648135' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-12T22:06:36.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:35 smithi028 ceph-mon[29847]: mds.? [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] up:active 2024-05-12T22:06:36.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:35 smithi028 ceph-mon[29847]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-12T22:06:36.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:35 smithi028 ceph-mon[29847]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-12T22:06:36.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:35 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/386348958' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-05-12T22:06:36.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:35 smithi028 ceph-mon[29847]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-05-12T22:06:36.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:36.285+0000 7ff0b5200640 1 -- 172.21.15.28:0/386348958 shutdown_connections 2024-05-12T22:06:36.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:36.285+0000 7ff0b5200640 1 -- 172.21.15.28:0/386348958 wait complete. 2024-05-12T22:06:36.310 DEBUG:tasks.cephfs.filesystem:fs_config: {'max_mds': 3, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2, 'subvol_options': '--size 25000000000'}} 2024-05-12T22:06:36.310 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2024-05-12T22:06:36.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:35 smithi186 ceph-mon[34785]: osdmap e86: 12 total, 12 up, 12 in 2024-05-12T22:06:36.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:35 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/81648135' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-12T22:06:36.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:35 smithi186 ceph-mon[34785]: mds.? [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] up:active 2024-05-12T22:06:36.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:35 smithi186 ceph-mon[34785]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-12T22:06:36.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:35 smithi186 ceph-mon[34785]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-12T22:06:36.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:35 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/386348958' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-05-12T22:06:36.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:35 smithi186 ceph-mon[34785]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-05-12T22:06:36.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:36.533+0000 7f617b200640 1 -- 172.21.15.28:0/1746333896 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f617c1021c0 msgr2=0x7f617c1045b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:36.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:36.533+0000 7f617b200640 1 --2- 172.21.15.28:0/1746333896 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f617c1021c0 0x7f617c1045b0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f616c009ca0 tx=0x7f616c02a8c0 comp rx=0 tx=0).stop 2024-05-12T22:06:36.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:36.534+0000 7f617b200640 1 -- 172.21.15.28:0/1746333896 shutdown_connections 2024-05-12T22:06:36.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:36.534+0000 7f617b200640 1 --2- 172.21.15.28:0/1746333896 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f617c107420 0x7f617c109870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:36.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:36.534+0000 7f617b200640 1 --2- 172.21.15.28:0/1746333896 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f617c104af0 0x7f617c106ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:36.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:36.534+0000 7f617b200640 1 --2- 172.21.15.28:0/1746333896 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f617c1021c0 0x7f617c1045b0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:36.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:36.534+0000 7f617b200640 1 -- 172.21.15.28:0/1746333896 >> 172.21.15.28:0/1746333896 conn(0x7f617c0fbbd0 msgr2=0x7f617c0fdff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:06:36.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:36.534+0000 7f617b200640 1 -- 172.21.15.28:0/1746333896 shutdown_connections 2024-05-12T22:06:36.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:36.534+0000 7f617b200640 1 -- 172.21.15.28:0/1746333896 wait complete. 2024-05-12T22:06:36.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:36.535+0000 7f617b200640 1 Processor -- start 2024-05-12T22:06:36.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:36.535+0000 7f617b200640 1 -- start start 2024-05-12T22:06:36.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:36.536+0000 7f617b200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f617c1021c0 0x7f617c199c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:36.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:36.536+0000 7f617b200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f617c104af0 0x7f617c19a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:36.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:36.536+0000 7f617b200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f617c107420 0x7f617c1a1220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:36.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:36.536+0000 7f617b200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f617c10d430 con 0x7f617c1021c0 2024-05-12T22:06:36.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:36.536+0000 7f617b200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f617c10d2b0 con 0x7f617c107420 2024-05-12T22:06:36.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:36.536+0000 7f617b200640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f617c10d5b0 con 0x7f617c104af0 2024-05-12T22:06:36.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:36.536+0000 7f6179e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f617c1021c0 0x7f617c199c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:36.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:36.536+0000 7f6179e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f617c1021c0 0x7f617c199c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36254/0 (socket says 172.21.15.28:36254) 2024-05-12T22:06:36.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:36.536+0000 7f6179e00640 1 -- 172.21.15.28:0/2803788930 learned_addr learned my addr 172.21.15.28:0/2803788930 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:06:36.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:36.536+0000 7f617a800640 1 --2- 172.21.15.28:0/2803788930 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f617c107420 0x7f617c1a1220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:36.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:36.536+0000 7f6179400640 1 --2- 172.21.15.28:0/2803788930 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f617c104af0 0x7f617c19a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:36.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:36.536+0000 7f6179e00640 1 -- 172.21.15.28:0/2803788930 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f617c104af0 msgr2=0x7f617c19a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:36.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:36.536+0000 7f6179e00640 1 --2- 172.21.15.28:0/2803788930 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f617c104af0 0x7f617c19a1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:36.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:36.536+0000 7f6179e00640 1 -- 172.21.15.28:0/2803788930 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f617c107420 msgr2=0x7f617c1a1220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:36.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:36.536+0000 7f6179e00640 1 --2- 172.21.15.28:0/2803788930 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f617c107420 0x7f617c1a1220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:36.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:36.536+0000 7f6179e00640 1 -- 172.21.15.28:0/2803788930 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f617c1a18c0 con 0x7f617c1021c0 2024-05-12T22:06:36.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:36.536+0000 7f617a800640 1 --2- 172.21.15.28:0/2803788930 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f617c107420 0x7f617c1a1220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:06:36.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:36.537+0000 7f6179e00640 1 --2- 172.21.15.28:0/2803788930 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f617c1021c0 0x7f617c199c80 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f616c000f80 tx=0x7f616c0319f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:06:36.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:36.537+0000 7f6172a00640 1 -- 172.21.15.28:0/2803788930 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f616c003b70 con 0x7f617c1021c0 2024-05-12T22:06:36.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:36.537+0000 7f6172a00640 1 -- 172.21.15.28:0/2803788930 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f616c003d10 con 0x7f617c1021c0 2024-05-12T22:06:36.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:36.537+0000 7f6172a00640 1 -- 172.21.15.28:0/2803788930 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f616c02fc00 con 0x7f617c1021c0 2024-05-12T22:06:36.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:36.537+0000 7f617b200640 1 -- 172.21.15.28:0/2803788930 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f617c1a1b50 con 0x7f617c1021c0 2024-05-12T22:06:36.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:36.537+0000 7f617b200640 1 -- 172.21.15.28:0/2803788930 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f617c1a20f0 con 0x7f617c1021c0 2024-05-12T22:06:36.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:36.539+0000 7f617b200640 1 -- 172.21.15.28:0/2803788930 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f617c108f70 con 0x7f617c1021c0 2024-05-12T22:06:36.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:36.541+0000 7f6172a00640 1 -- 172.21.15.28:0/2803788930 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f616c004040 con 0x7f617c1021c0 2024-05-12T22:06:36.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:36.544+0000 7f6172a00640 1 --2- 172.21.15.28:0/2803788930 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f61600a2c10 0x7f61600a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:36.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:36.544+0000 7f6172a00640 1 -- 172.21.15.28:0/2803788930 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f616c0f0490 con 0x7f617c1021c0 2024-05-12T22:06:36.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:36.544+0000 7f6179400640 1 --2- 172.21.15.28:0/2803788930 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f61600a2c10 0x7f61600a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:36.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:36.545+0000 7f6179400640 1 --2- 172.21.15.28:0/2803788930 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f61600a2c10 0x7f61600a50d0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f617c19b180 tx=0x7f6168005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:06:36.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:36.546+0000 7f6172a00640 1 -- 172.21.15.28:0/2803788930 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f616c0bd4a0 con 0x7f617c1021c0 2024-05-12T22:06:36.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:36.791+0000 7f617b200640 1 -- 172.21.15.28:0/2803788930 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"} v 0) -- 0x7f617c02d3c0 con 0x7f617c1021c0 2024-05-12T22:06:37.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:36 smithi086 ceph-mon[34297]: pgmap v311: 97 pgs: 1 active, 29 creating+peering, 40 unknown, 27 active+clean; 577 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 6 op/s 2024-05-12T22:06:37.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:36 smithi086 ceph-mon[34297]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-05-12T22:06:37.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:36 smithi086 ceph-mon[34297]: Cluster is now healthy 2024-05-12T22:06:37.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:36 smithi086 ceph-mon[34297]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-05-12T22:06:37.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:36 smithi086 ceph-mon[34297]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-12T22:06:37.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:36 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/2803788930' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-05-12T22:06:37.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:36 smithi028 ceph-mon[29847]: pgmap v311: 97 pgs: 1 active, 29 creating+peering, 40 unknown, 27 active+clean; 577 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 6 op/s 2024-05-12T22:06:37.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:36 smithi028 ceph-mon[29847]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-05-12T22:06:37.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:36 smithi028 ceph-mon[29847]: Cluster is now healthy 2024-05-12T22:06:37.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:36 smithi028 ceph-mon[29847]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-05-12T22:06:37.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:36 smithi028 ceph-mon[29847]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-12T22:06:37.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:36 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/2803788930' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-05-12T22:06:37.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:37.289+0000 7f6172a00640 1 -- 172.21.15.28:0/2803788930 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]=0 v18) ==== 105+0+0 (secure 0 0 0) 0x7f616c0bcda0 con 0x7f617c1021c0 2024-05-12T22:06:37.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:37.292+0000 7f617b200640 1 -- 172.21.15.28:0/2803788930 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f61600a2c10 msgr2=0x7f61600a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:37.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:37.292+0000 7f617b200640 1 --2- 172.21.15.28:0/2803788930 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f61600a2c10 0x7f61600a50d0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f617c19b180 tx=0x7f6168005f90 comp rx=0 tx=0).stop 2024-05-12T22:06:37.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:37.292+0000 7f617b200640 1 -- 172.21.15.28:0/2803788930 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f617c1021c0 msgr2=0x7f617c199c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:37.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:37.292+0000 7f617b200640 1 --2- 172.21.15.28:0/2803788930 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f617c1021c0 0x7f617c199c80 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f616c000f80 tx=0x7f616c0319f0 comp rx=0 tx=0).stop 2024-05-12T22:06:37.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:37.292+0000 7f617b200640 1 -- 172.21.15.28:0/2803788930 shutdown_connections 2024-05-12T22:06:37.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:37.292+0000 7f617b200640 1 --2- 172.21.15.28:0/2803788930 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f61600a2c10 0x7f61600a50d0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:37.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:37.292+0000 7f617b200640 1 --2- 172.21.15.28:0/2803788930 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f617c107420 0x7f617c1a1220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:37.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:37.292+0000 7f617b200640 1 --2- 172.21.15.28:0/2803788930 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f617c104af0 0x7f617c19a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:37.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:37.292+0000 7f617b200640 1 --2- 172.21.15.28:0/2803788930 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f617c1021c0 0x7f617c199c80 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:37.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:37.292+0000 7f617b200640 1 -- 172.21.15.28:0/2803788930 >> 172.21.15.28:0/2803788930 conn(0x7f617c0fbbd0 msgr2=0x7f617c1018b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:06:37.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:37.293+0000 7f617b200640 1 -- 172.21.15.28:0/2803788930 shutdown_connections 2024-05-12T22:06:37.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:37.293+0000 7f617b200640 1 -- 172.21.15.28:0/2803788930 wait complete. 2024-05-12T22:06:37.320 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2024-05-12T22:06:37.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:36 smithi186 ceph-mon[34785]: pgmap v311: 97 pgs: 1 active, 29 creating+peering, 40 unknown, 27 active+clean; 577 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 6 op/s 2024-05-12T22:06:37.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:36 smithi186 ceph-mon[34785]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-05-12T22:06:37.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:36 smithi186 ceph-mon[34785]: Cluster is now healthy 2024-05-12T22:06:37.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:36 smithi186 ceph-mon[34785]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-05-12T22:06:37.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:36 smithi186 ceph-mon[34785]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-12T22:06:37.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:36 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/2803788930' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-05-12T22:06:37.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:37.512+0000 7f45db000640 1 -- 172.21.15.28:0/230710246 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f45d4077430 msgr2=0x7f45d4077890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:37.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:37.512+0000 7f45db000640 1 --2- 172.21.15.28:0/230710246 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f45d4077430 0x7f45d4077890 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f45c0009900 tx=0x7f45c002a400 comp rx=0 tx=0).stop 2024-05-12T22:06:37.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:37.513+0000 7f45db000640 1 -- 172.21.15.28:0/230710246 shutdown_connections 2024-05-12T22:06:37.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:37.513+0000 7f45db000640 1 --2- 172.21.15.28:0/230710246 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45d4077dd0 0x7f45d4078670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:37.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:37.513+0000 7f45db000640 1 --2- 172.21.15.28:0/230710246 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f45d4077430 0x7f45d4077890 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:37.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:37.513+0000 7f45db000640 1 --2- 172.21.15.28:0/230710246 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f45d40761f0 0x7f45d4076610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:37.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:37.513+0000 7f45db000640 1 -- 172.21.15.28:0/230710246 >> 172.21.15.28:0/230710246 conn(0x7f45d40fefe0 msgr2=0x7f45d4101440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:06:37.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:37.513+0000 7f45db000640 1 -- 172.21.15.28:0/230710246 shutdown_connections 2024-05-12T22:06:37.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:37.513+0000 7f45db000640 1 -- 172.21.15.28:0/230710246 wait complete. 2024-05-12T22:06:37.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:37.514+0000 7f45db000640 1 Processor -- start 2024-05-12T22:06:37.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:37.514+0000 7f45db000640 1 -- start start 2024-05-12T22:06:37.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:37.514+0000 7f45db000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45d40761f0 0x7f45d419e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:37.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:37.514+0000 7f45db000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f45d4077430 0x7f45d419e720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:37.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:37.514+0000 7f45db000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f45d4077dd0 0x7f45d41a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:37.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:37.514+0000 7f45db000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f45d4113b60 con 0x7f45d40761f0 2024-05-12T22:06:37.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:37.514+0000 7f45db000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f45d41139e0 con 0x7f45d4077dd0 2024-05-12T22:06:37.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:37.514+0000 7f45db000640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f45d4113ce0 con 0x7f45d4077430 2024-05-12T22:06:37.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:37.515+0000 7f45d3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45d40761f0 0x7f45d419e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:37.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:37.515+0000 7f45d3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45d40761f0 0x7f45d419e1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36272/0 (socket says 172.21.15.28:36272) 2024-05-12T22:06:37.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:37.515+0000 7f45d3e00640 1 -- 172.21.15.28:0/2114195004 learned_addr learned my addr 172.21.15.28:0/2114195004 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:06:37.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:37.515+0000 7f45d9000640 1 --2- 172.21.15.28:0/2114195004 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f45d4077dd0 0x7f45d41a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:37.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:37.515+0000 7f45d3400640 1 --2- 172.21.15.28:0/2114195004 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f45d4077430 0x7f45d419e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:37.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:37.515+0000 7f45d3e00640 1 -- 172.21.15.28:0/2114195004 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f45d4077430 msgr2=0x7f45d419e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:37.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:37.515+0000 7f45d3e00640 1 --2- 172.21.15.28:0/2114195004 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f45d4077430 0x7f45d419e720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:37.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:37.515+0000 7f45d3e00640 1 -- 172.21.15.28:0/2114195004 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f45d4077dd0 msgr2=0x7f45d41a5790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:37.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:37.515+0000 7f45d3e00640 1 --2- 172.21.15.28:0/2114195004 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f45d4077dd0 0x7f45d41a5790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:37.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:37.515+0000 7f45d3e00640 1 -- 172.21.15.28:0/2114195004 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45d41a5e00 con 0x7f45d40761f0 2024-05-12T22:06:37.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:37.515+0000 7f45d9000640 1 --2- 172.21.15.28:0/2114195004 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f45d4077dd0 0x7f45d41a5790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:06:37.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:37.516+0000 7f45d3e00640 1 --2- 172.21.15.28:0/2114195004 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45d40761f0 0x7f45d419e1e0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f45c400e9f0 tx=0x7f45c400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:06:37.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:37.516+0000 7f45d0c00640 1 -- 172.21.15.28:0/2114195004 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45c400ccc0 con 0x7f45d40761f0 2024-05-12T22:06:37.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:37.516+0000 7f45d0c00640 1 -- 172.21.15.28:0/2114195004 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f45c400ce60 con 0x7f45d40761f0 2024-05-12T22:06:37.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:37.516+0000 7f45db000640 1 -- 172.21.15.28:0/2114195004 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45d41a60f0 con 0x7f45d40761f0 2024-05-12T22:06:37.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:37.516+0000 7f45d0c00640 1 -- 172.21.15.28:0/2114195004 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45c4010690 con 0x7f45d40761f0 2024-05-12T22:06:37.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:37.516+0000 7f45db000640 1 -- 172.21.15.28:0/2114195004 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45d41a65c0 con 0x7f45d40761f0 2024-05-12T22:06:37.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:37.517+0000 7f45db000640 1 -- 172.21.15.28:0/2114195004 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45d41105e0 con 0x7f45d40761f0 2024-05-12T22:06:37.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:37.520+0000 7f45d0c00640 1 -- 172.21.15.28:0/2114195004 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f45c40046c0 con 0x7f45d40761f0 2024-05-12T22:06:37.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:37.521+0000 7f45d0c00640 1 --2- 172.21.15.28:0/2114195004 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f45b80a2b90 0x7f45b80a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:37.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:37.521+0000 7f45d0c00640 1 -- 172.21.15.28:0/2114195004 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f45c40cf3b0 con 0x7f45d40761f0 2024-05-12T22:06:37.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:37.521+0000 7f45d3400640 1 --2- 172.21.15.28:0/2114195004 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f45b80a2b90 0x7f45b80a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:37.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:37.522+0000 7f45d3400640 1 --2- 172.21.15.28:0/2114195004 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f45b80a2b90 0x7f45b80a5050 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f45c0031440 tx=0x7f45c00315b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:06:37.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:37.526+0000 7f45d0c00640 1 -- 172.21.15.28:0/2114195004 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f45c409c3c0 con 0x7f45d40761f0 2024-05-12T22:06:37.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:37.773+0000 7f45db000640 1 -- 172.21.15.28:0/2114195004 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) -- 0x7f45d402d1c0 con 0x7f45d40761f0 2024-05-12T22:06:38.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:37 smithi086 ceph-mon[34297]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-05-12T22:06:38.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:37 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/2803788930' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-05-12T22:06:38.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:37 smithi086 ceph-mon[34297]: daemon mds.a assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-05-12T22:06:38.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:37 smithi086 ceph-mon[34297]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-12T22:06:38.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:37 smithi086 ceph-mon[34297]: fsmap cephfs:2 {0=b=up:active,1=a=up:creating} 10 up:standby 2024-05-12T22:06:38.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:37 smithi086 ceph-mon[34297]: daemon mds.a is now active in filesystem cephfs as rank 1 2024-05-12T22:06:38.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:37 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/2114195004' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-05-12T22:06:38.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:37 smithi028 ceph-mon[29847]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-05-12T22:06:38.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:37 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/2803788930' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-05-12T22:06:38.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:37 smithi028 ceph-mon[29847]: daemon mds.a assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-05-12T22:06:38.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:37 smithi028 ceph-mon[29847]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-12T22:06:38.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:37 smithi028 ceph-mon[29847]: fsmap cephfs:2 {0=b=up:active,1=a=up:creating} 10 up:standby 2024-05-12T22:06:38.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:37 smithi028 ceph-mon[29847]: daemon mds.a is now active in filesystem cephfs as rank 1 2024-05-12T22:06:38.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:37 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/2114195004' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-05-12T22:06:38.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:38.301+0000 7f45d0c00640 1 -- 172.21.15.28:0/2114195004 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v20) ==== 121+0+0 (secure 0 0 0) 0x7f45c409bcc0 con 0x7f45d40761f0 2024-05-12T22:06:38.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:38.306+0000 7f45db000640 1 -- 172.21.15.28:0/2114195004 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f45b80a2b90 msgr2=0x7f45b80a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:38.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:38.306+0000 7f45db000640 1 --2- 172.21.15.28:0/2114195004 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f45b80a2b90 0x7f45b80a5050 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f45c0031440 tx=0x7f45c00315b0 comp rx=0 tx=0).stop 2024-05-12T22:06:38.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:38.306+0000 7f45db000640 1 -- 172.21.15.28:0/2114195004 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45d40761f0 msgr2=0x7f45d419e1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:38.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:38.306+0000 7f45db000640 1 --2- 172.21.15.28:0/2114195004 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45d40761f0 0x7f45d419e1e0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f45c400e9f0 tx=0x7f45c400edf0 comp rx=0 tx=0).stop 2024-05-12T22:06:38.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:38.306+0000 7f45db000640 1 -- 172.21.15.28:0/2114195004 shutdown_connections 2024-05-12T22:06:38.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:38.306+0000 7f45db000640 1 --2- 172.21.15.28:0/2114195004 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f45b80a2b90 0x7f45b80a5050 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:38.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:38.306+0000 7f45db000640 1 --2- 172.21.15.28:0/2114195004 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f45d4077dd0 0x7f45d41a5790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:38.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:38.306+0000 7f45db000640 1 --2- 172.21.15.28:0/2114195004 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f45d4077430 0x7f45d419e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:38.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:38.306+0000 7f45db000640 1 --2- 172.21.15.28:0/2114195004 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45d40761f0 0x7f45d419e1e0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:38.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:38.306+0000 7f45db000640 1 -- 172.21.15.28:0/2114195004 >> 172.21.15.28:0/2114195004 conn(0x7f45d40fefe0 msgr2=0x7f45d4100710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:06:38.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:38.307+0000 7f45db000640 1 -- 172.21.15.28:0/2114195004 shutdown_connections 2024-05-12T22:06:38.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:38.307+0000 7f45db000640 1 -- 172.21.15.28:0/2114195004 wait complete. 2024-05-12T22:06:38.335 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-05-12T22:06:38.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:37 smithi186 ceph-mon[34785]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-05-12T22:06:38.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:37 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/2803788930' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-05-12T22:06:38.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:37 smithi186 ceph-mon[34785]: daemon mds.a assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-05-12T22:06:38.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:37 smithi186 ceph-mon[34785]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-12T22:06:38.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:37 smithi186 ceph-mon[34785]: fsmap cephfs:2 {0=b=up:active,1=a=up:creating} 10 up:standby 2024-05-12T22:06:38.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:37 smithi186 ceph-mon[34785]: daemon mds.a is now active in filesystem cephfs as rank 1 2024-05-12T22:06:38.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:37 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/2114195004' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-05-12T22:06:38.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:38.520+0000 7f411e000640 1 -- 172.21.15.28:0/3805460350 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4118074230 msgr2=0x7f4118074690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:38.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:38.520+0000 7f411e000640 1 --2- 172.21.15.28:0/3805460350 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4118074230 0x7f4118074690 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f4100009900 tx=0x7f410002a480 comp rx=0 tx=0).stop 2024-05-12T22:06:38.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:38.520+0000 7f411e000640 1 -- 172.21.15.28:0/3805460350 shutdown_connections 2024-05-12T22:06:38.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:38.520+0000 7f411e000640 1 --2- 172.21.15.28:0/3805460350 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41181095d0 0x7f411810ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:38.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:38.520+0000 7f411e000640 1 --2- 172.21.15.28:0/3805460350 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4118074230 0x7f4118074690 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:38.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:38.520+0000 7f411e000640 1 --2- 172.21.15.28:0/3805460350 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4118075810 0x7f4118073c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:38.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:38.520+0000 7f411e000640 1 -- 172.21.15.28:0/3805460350 >> 172.21.15.28:0/3805460350 conn(0x7f41180fccd0 msgr2=0x7f41180ff0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:06:38.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:38.520+0000 7f411e000640 1 -- 172.21.15.28:0/3805460350 shutdown_connections 2024-05-12T22:06:38.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:38.520+0000 7f411e000640 1 -- 172.21.15.28:0/3805460350 wait complete. 2024-05-12T22:06:38.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:38.521+0000 7f411e000640 1 Processor -- start 2024-05-12T22:06:38.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:38.522+0000 7f411e000640 1 -- start start 2024-05-12T22:06:38.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:38.522+0000 7f411e000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4118074230 0x7f4118199c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:38.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:38.522+0000 7f411e000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4118075810 0x7f411819a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:38.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:38.522+0000 7f411e000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f41181095d0 0x7f41181a11d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:38.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:38.522+0000 7f411e000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f411810f610 con 0x7f4118075810 2024-05-12T22:06:38.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:38.522+0000 7f411e000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f411810f490 con 0x7f4118074230 2024-05-12T22:06:38.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:38.522+0000 7f411e000640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f411810f790 con 0x7f41181095d0 2024-05-12T22:06:38.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:38.522+0000 7f411cc00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4118074230 0x7f4118199c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:38.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:38.522+0000 7f411cc00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4118074230 0x7f4118199c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:58178/0 (socket says 172.21.15.28:58178) 2024-05-12T22:06:38.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:38.522+0000 7f411cc00640 1 -- 172.21.15.28:0/3377855766 learned_addr learned my addr 172.21.15.28:0/3377855766 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:06:38.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:38.522+0000 7f4117e00640 1 --2- 172.21.15.28:0/3377855766 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4118075810 0x7f411819a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:38.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:38.523+0000 7f411cc00640 1 -- 172.21.15.28:0/3377855766 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f41181095d0 msgr2=0x7f41181a11d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-12T22:06:38.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:38.523+0000 7f411cc00640 1 --2- 172.21.15.28:0/3377855766 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f41181095d0 0x7f41181a11d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:38.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:38.523+0000 7f411cc00640 1 -- 172.21.15.28:0/3377855766 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4118075810 msgr2=0x7f411819a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:38.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:38.523+0000 7f411cc00640 1 --2- 172.21.15.28:0/3377855766 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4118075810 0x7f411819a1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:38.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:38.523+0000 7f411cc00640 1 -- 172.21.15.28:0/3377855766 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41181a1870 con 0x7f4118074230 2024-05-12T22:06:38.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:38.523+0000 7f411cc00640 1 --2- 172.21.15.28:0/3377855766 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4118074230 0x7f4118199c60 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f410800e9f0 tx=0x7f410800edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:06:38.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:38.523+0000 7f4115600640 1 -- 172.21.15.28:0/3377855766 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f410800cd80 con 0x7f4118074230 2024-05-12T22:06:38.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:38.523+0000 7f4115600640 1 -- 172.21.15.28:0/3377855766 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4108005500 con 0x7f4118074230 2024-05-12T22:06:38.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:38.523+0000 7f4115600640 1 -- 172.21.15.28:0/3377855766 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4108010690 con 0x7f4118074230 2024-05-12T22:06:38.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:38.523+0000 7f411e000640 1 -- 172.21.15.28:0/3377855766 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41181a1b60 con 0x7f4118074230 2024-05-12T22:06:38.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:38.523+0000 7f411e000640 1 -- 172.21.15.28:0/3377855766 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41181a2060 con 0x7f4118074230 2024-05-12T22:06:38.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:38.525+0000 7f411e000640 1 -- 172.21.15.28:0/3377855766 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4118108390 con 0x7f4118074230 2024-05-12T22:06:38.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:38.526+0000 7f4115600640 1 -- 172.21.15.28:0/3377855766 <== mon.2 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f41080046c0 con 0x7f4118074230 2024-05-12T22:06:38.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:38.528+0000 7f4115600640 1 --2- 172.21.15.28:0/3377855766 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f40fc0a2c10 0x7f40fc0a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:38.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:38.528+0000 7f4115600640 1 -- 172.21.15.28:0/3377855766 <== mon.2 v2:172.21.15.86:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f41080cf510 con 0x7f4118074230 2024-05-12T22:06:38.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:38.528+0000 7f4117e00640 1 --2- 172.21.15.28:0/3377855766 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f40fc0a2c10 0x7f40fc0a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:38.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:38.529+0000 7f4117e00640 1 --2- 172.21.15.28:0/3377855766 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f40fc0a2c10 0x7f40fc0a50d0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f4100006010 tx=0x7f4100000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:06:38.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:38.532+0000 7f4115600640 1 -- 172.21.15.28:0/3377855766 <== mon.2 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f410809c520 con 0x7f4118074230 2024-05-12T22:06:38.730 INFO:journalctl@ceph.mds.b.smithi086.stdout:May 12 22:06:38 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-b[64802]: 2024-05-12T22:06:38.342+0000 7f142c986700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-05-12T22:06:38.731 INFO:journalctl@ceph.mds.b.smithi086.stdout:May 12 22:06:38 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-b[64802]: 2024-05-12T22:06:38.342+0000 7f142c986700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-05-12T22:06:38.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:38.786+0000 7f411e000640 1 -- 172.21.15.28:0/3377855766 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7f411802d340 con 0x7f4118074230 2024-05-12T22:06:39.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:39 smithi186 ceph-mon[34785]: pgmap v312: 97 pgs: 1 active, 21 creating+peering, 22 unknown, 53 active+clean; 577 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s wr, 6 op/s 2024-05-12T22:06:39.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:39 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/2114195004' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-05-12T22:06:39.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:39 smithi186 ceph-mon[34785]: mds.? [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] up:active 2024-05-12T22:06:39.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:39 smithi186 ceph-mon[34785]: daemon mds.g assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-05-12T22:06:39.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:39 smithi186 ceph-mon[34785]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-05-12T22:06:39.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:39 smithi186 ceph-mon[34785]: Cluster is now healthy 2024-05-12T22:06:39.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:39 smithi186 ceph-mon[34785]: fsmap cephfs:2 {0=b=up:active,1=a=up:active} 10 up:standby 2024-05-12T22:06:39.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:39 smithi186 ceph-mon[34785]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=g=up:creating} 9 up:standby 2024-05-12T22:06:39.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:39 smithi186 ceph-mon[34785]: daemon mds.g is now active in filesystem cephfs as rank 2 2024-05-12T22:06:39.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:39 smithi186 ceph-mon[34785]: mds.? [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] up:active 2024-05-12T22:06:39.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:39 smithi186 ceph-mon[34785]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=g=up:active} 3 up:standby-replay 6 up:standby 2024-05-12T22:06:39.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:39 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/3377855766' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-05-12T22:06:39.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:39 smithi186 ceph-mon[34785]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-05-12T22:06:39.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:39.706+0000 7f4115600640 1 -- 172.21.15.28:0/3377855766 <== mon.2 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v23) ==== 115+0+0 (secure 0 0 0) 0x7f410809be20 con 0x7f4118074230 2024-05-12T22:06:39.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:39.709+0000 7f411e000640 1 -- 172.21.15.28:0/3377855766 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f40fc0a2c10 msgr2=0x7f40fc0a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:39.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:39.710+0000 7f411e000640 1 --2- 172.21.15.28:0/3377855766 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f40fc0a2c10 0x7f40fc0a50d0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f4100006010 tx=0x7f4100000bc0 comp rx=0 tx=0).stop 2024-05-12T22:06:39.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:39.710+0000 7f411e000640 1 -- 172.21.15.28:0/3377855766 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4118074230 msgr2=0x7f4118199c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:39.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:39.710+0000 7f411e000640 1 --2- 172.21.15.28:0/3377855766 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4118074230 0x7f4118199c60 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f410800e9f0 tx=0x7f410800edf0 comp rx=0 tx=0).stop 2024-05-12T22:06:39.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:39.710+0000 7f411e000640 1 -- 172.21.15.28:0/3377855766 shutdown_connections 2024-05-12T22:06:39.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:39.710+0000 7f411e000640 1 --2- 172.21.15.28:0/3377855766 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f40fc0a2c10 0x7f40fc0a50d0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:39.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:39.710+0000 7f411e000640 1 --2- 172.21.15.28:0/3377855766 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f41181095d0 0x7f41181a11d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:39.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:39.710+0000 7f411e000640 1 --2- 172.21.15.28:0/3377855766 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4118075810 0x7f411819a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:39.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:39.710+0000 7f411e000640 1 --2- 172.21.15.28:0/3377855766 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4118074230 0x7f4118199c60 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:39.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:39.710+0000 7f411e000640 1 -- 172.21.15.28:0/3377855766 >> 172.21.15.28:0/3377855766 conn(0x7f41180fccd0 msgr2=0x7f4118105560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:06:39.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:39.710+0000 7f411e000640 1 -- 172.21.15.28:0/3377855766 shutdown_connections 2024-05-12T22:06:39.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:39.710+0000 7f411e000640 1 -- 172.21.15.28:0/3377855766 wait complete. 2024-05-12T22:06:39.730 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:39 smithi028 ceph-mon[29847]: pgmap v312: 97 pgs: 1 active, 21 creating+peering, 22 unknown, 53 active+clean; 577 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s wr, 6 op/s 2024-05-12T22:06:39.730 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:39 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/2114195004' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-05-12T22:06:39.730 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:39 smithi028 ceph-mon[29847]: mds.? [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] up:active 2024-05-12T22:06:39.730 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:39 smithi028 ceph-mon[29847]: daemon mds.g assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-05-12T22:06:39.730 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:39 smithi028 ceph-mon[29847]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-05-12T22:06:39.730 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:39 smithi028 ceph-mon[29847]: Cluster is now healthy 2024-05-12T22:06:39.730 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:39 smithi028 ceph-mon[29847]: fsmap cephfs:2 {0=b=up:active,1=a=up:active} 10 up:standby 2024-05-12T22:06:39.731 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:39 smithi028 ceph-mon[29847]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=g=up:creating} 9 up:standby 2024-05-12T22:06:39.731 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:39 smithi028 ceph-mon[29847]: daemon mds.g is now active in filesystem cephfs as rank 2 2024-05-12T22:06:39.731 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:39 smithi028 ceph-mon[29847]: mds.? [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] up:active 2024-05-12T22:06:39.731 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:39 smithi028 ceph-mon[29847]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=g=up:active} 3 up:standby-replay 6 up:standby 2024-05-12T22:06:39.731 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:39 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/3377855766' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-05-12T22:06:39.731 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:39 smithi028 ceph-mon[29847]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-05-12T22:06:39.733 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2, 'subvol_options': '--size 25000000000'} subvols for filesystem 'cephfs' 2024-05-12T22:06:39.733 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create cephfs qa 2024-05-12T22:06:39.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:39 smithi086 ceph-mon[34297]: pgmap v312: 97 pgs: 1 active, 21 creating+peering, 22 unknown, 53 active+clean; 577 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s wr, 6 op/s 2024-05-12T22:06:39.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:39 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/2114195004' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-05-12T22:06:39.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:39 smithi086 ceph-mon[34297]: mds.? [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] up:active 2024-05-12T22:06:39.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:39 smithi086 ceph-mon[34297]: daemon mds.g assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-05-12T22:06:39.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:39 smithi086 ceph-mon[34297]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-05-12T22:06:39.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:39 smithi086 ceph-mon[34297]: Cluster is now healthy 2024-05-12T22:06:39.743 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:39 smithi086 ceph-mon[34297]: fsmap cephfs:2 {0=b=up:active,1=a=up:active} 10 up:standby 2024-05-12T22:06:39.743 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:39 smithi086 ceph-mon[34297]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=g=up:creating} 9 up:standby 2024-05-12T22:06:39.743 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:39 smithi086 ceph-mon[34297]: daemon mds.g is now active in filesystem cephfs as rank 2 2024-05-12T22:06:39.743 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:39 smithi086 ceph-mon[34297]: mds.? [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] up:active 2024-05-12T22:06:39.743 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:39 smithi086 ceph-mon[34297]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=g=up:active} 3 up:standby-replay 6 up:standby 2024-05-12T22:06:39.743 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:39 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/3377855766' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-05-12T22:06:39.743 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:39 smithi086 ceph-mon[34297]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-05-12T22:06:39.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:39.948+0000 7fb5df200640 1 -- 172.21.15.28:0/1953508920 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb5e0102930 msgr2=0x7fb5e0102db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:39.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:39.948+0000 7fb5df200640 1 --2- 172.21.15.28:0/1953508920 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb5e0102930 0x7fb5e0102db0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fb5cc009900 tx=0x7fb5cc02a3f0 comp rx=0 tx=0).stop 2024-05-12T22:06:39.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:39.948+0000 7fb5df200640 1 -- 172.21.15.28:0/1953508920 shutdown_connections 2024-05-12T22:06:39.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:39.948+0000 7fb5df200640 1 --2- 172.21.15.28:0/1953508920 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5e01032f0 0x7fb5e0109b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:39.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:39.948+0000 7fb5df200640 1 --2- 172.21.15.28:0/1953508920 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb5e0102930 0x7fb5e0102db0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:39.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:39.948+0000 7fb5df200640 1 --2- 172.21.15.28:0/1953508920 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb5e0101740 0x7fb5e0101b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:39.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:39.948+0000 7fb5df200640 1 -- 172.21.15.28:0/1953508920 >> 172.21.15.28:0/1953508920 conn(0x7fb5e00fccd0 msgr2=0x7fb5e00ff0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:06:39.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:39.948+0000 7fb5df200640 1 -- 172.21.15.28:0/1953508920 shutdown_connections 2024-05-12T22:06:39.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:39.948+0000 7fb5df200640 1 -- 172.21.15.28:0/1953508920 wait complete. 2024-05-12T22:06:39.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:39.950+0000 7fb5df200640 1 Processor -- start 2024-05-12T22:06:39.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:39.950+0000 7fb5df200640 1 -- start start 2024-05-12T22:06:39.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:39.950+0000 7fb5df200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb5e0101740 0x7fb5e0199c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:39.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:39.950+0000 7fb5df200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb5e0102930 0x7fb5e019a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:39.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:39.950+0000 7fb5df200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5e01032f0 0x7fb5e01a1210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:39.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:39.951+0000 7fb5df200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb5e010d410 con 0x7fb5e01032f0 2024-05-12T22:06:39.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:39.951+0000 7fb5df200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fb5e010d290 con 0x7fb5e0102930 2024-05-12T22:06:39.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:39.951+0000 7fb5df200640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb5e010d590 con 0x7fb5e0101740 2024-05-12T22:06:39.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:39.951+0000 7fb5dd400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb5e0102930 0x7fb5e019a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:39.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:39.951+0000 7fb5dd400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb5e0102930 0x7fb5e019a1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:58192/0 (socket says 172.21.15.28:58192) 2024-05-12T22:06:39.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:39.951+0000 7fb5dd400640 1 -- 172.21.15.28:0/4230818909 learned_addr learned my addr 172.21.15.28:0/4230818909 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:06:39.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:39.951+0000 7fb5dde00640 1 --2- 172.21.15.28:0/4230818909 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb5e0101740 0x7fb5e0199c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:39.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:39.951+0000 7fb5de800640 1 --2- 172.21.15.28:0/4230818909 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5e01032f0 0x7fb5e01a1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:39.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:39.951+0000 7fb5dd400640 1 -- 172.21.15.28:0/4230818909 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb5e0101740 msgr2=0x7fb5e0199c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:39.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:39.951+0000 7fb5dd400640 1 --2- 172.21.15.28:0/4230818909 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb5e0101740 0x7fb5e0199c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:39.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:39.951+0000 7fb5dd400640 1 -- 172.21.15.28:0/4230818909 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5e01032f0 msgr2=0x7fb5e01a1210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:39.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:39.951+0000 7fb5dd400640 1 --2- 172.21.15.28:0/4230818909 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5e01032f0 0x7fb5e01a1210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:39.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:39.951+0000 7fb5dd400640 1 -- 172.21.15.28:0/4230818909 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5e01a18b0 con 0x7fb5e0102930 2024-05-12T22:06:39.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:39.951+0000 7fb5dde00640 1 --2- 172.21.15.28:0/4230818909 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb5e0101740 0x7fb5e0199c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:06:39.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:39.951+0000 7fb5de800640 1 --2- 172.21.15.28:0/4230818909 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5e01032f0 0x7fb5e01a1210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:06:39.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:39.952+0000 7fb5dd400640 1 --2- 172.21.15.28:0/4230818909 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb5e0102930 0x7fb5e019a1b0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fb5cc031440 tx=0x7fb5cc032ce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:06:39.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:39.952+0000 7fb5d6a00640 1 -- 172.21.15.28:0/4230818909 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5cc03f030 con 0x7fb5e0102930 2024-05-12T22:06:39.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:39.952+0000 7fb5d6a00640 1 -- 172.21.15.28:0/4230818909 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb5cc003ce0 con 0x7fb5e0102930 2024-05-12T22:06:39.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:39.952+0000 7fb5d6a00640 1 -- 172.21.15.28:0/4230818909 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5cc035070 con 0x7fb5e0102930 2024-05-12T22:06:39.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:39.952+0000 7fb5df200640 1 -- 172.21.15.28:0/4230818909 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5e01a1b40 con 0x7fb5e0102930 2024-05-12T22:06:39.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:39.952+0000 7fb5df200640 1 -- 172.21.15.28:0/4230818909 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5e01a2080 con 0x7fb5e0102930 2024-05-12T22:06:39.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:39.953+0000 7fb5df200640 1 -- 172.21.15.28:0/4230818909 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb5e01a2250 con 0x7fb5e0102930 2024-05-12T22:06:39.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:39.955+0000 7fb5d6a00640 1 -- 172.21.15.28:0/4230818909 <== mon.2 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fb5cc042030 con 0x7fb5e0102930 2024-05-12T22:06:39.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:39.957+0000 7fb5d6a00640 1 --2- 172.21.15.28:0/4230818909 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fb5c40a2c10 0x7fb5c40a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:39.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:39.957+0000 7fb5d6a00640 1 -- 172.21.15.28:0/4230818909 <== mon.2 v2:172.21.15.86:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7fb5cc0ef870 con 0x7fb5e0102930 2024-05-12T22:06:39.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:39.957+0000 7fb5dde00640 1 --2- 172.21.15.28:0/4230818909 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fb5c40a2c10 0x7fb5c40a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:39.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:39.958+0000 7fb5dde00640 1 --2- 172.21.15.28:0/4230818909 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fb5c40a2c10 0x7fb5c40a50d0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fb5e019ab40 tx=0x7fb5d0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:06:39.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:39.961+0000 7fb5d6a00640 1 -- 172.21.15.28:0/4230818909 <== mon.2 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fb5cc0bc880 con 0x7fb5e0102930 2024-05-12T22:06:40.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:40.199+0000 7fb5df200640 1 -- 172.21.15.28:0/4230818909 --> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7fb5e01a24e0 con 0x7fb5c40a2c10 2024-05-12T22:06:40.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:40.241+0000 7fb5d6a00640 1 -- 172.21.15.28:0/4230818909 <== mgr.14152 v2:172.21.15.28:6800/3123328651 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fb5e01a24e0 con 0x7fb5c40a2c10 2024-05-12T22:06:40.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:40.244+0000 7fb5df200640 1 -- 172.21.15.28:0/4230818909 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fb5c40a2c10 msgr2=0x7fb5c40a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:40.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:40.244+0000 7fb5df200640 1 --2- 172.21.15.28:0/4230818909 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fb5c40a2c10 0x7fb5c40a50d0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fb5e019ab40 tx=0x7fb5d0005f90 comp rx=0 tx=0).stop 2024-05-12T22:06:40.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:40.244+0000 7fb5df200640 1 -- 172.21.15.28:0/4230818909 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb5e0102930 msgr2=0x7fb5e019a1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:40.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:40.244+0000 7fb5df200640 1 --2- 172.21.15.28:0/4230818909 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb5e0102930 0x7fb5e019a1b0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fb5cc031440 tx=0x7fb5cc032ce0 comp rx=0 tx=0).stop 2024-05-12T22:06:40.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:40.244+0000 7fb5df200640 1 -- 172.21.15.28:0/4230818909 shutdown_connections 2024-05-12T22:06:40.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:40.244+0000 7fb5df200640 1 --2- 172.21.15.28:0/4230818909 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fb5c40a2c10 0x7fb5c40a50d0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:40.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:40.244+0000 7fb5df200640 1 --2- 172.21.15.28:0/4230818909 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5e01032f0 0x7fb5e01a1210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:40.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:40.244+0000 7fb5df200640 1 --2- 172.21.15.28:0/4230818909 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb5e0102930 0x7fb5e019a1b0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:40.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:40.244+0000 7fb5df200640 1 --2- 172.21.15.28:0/4230818909 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb5e0101740 0x7fb5e0199c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:40.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:40.244+0000 7fb5df200640 1 -- 172.21.15.28:0/4230818909 >> 172.21.15.28:0/4230818909 conn(0x7fb5e00fccd0 msgr2=0x7fb5e0073a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:06:40.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:40.244+0000 7fb5df200640 1 -- 172.21.15.28:0/4230818909 shutdown_connections 2024-05-12T22:06:40.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:40.244+0000 7fb5df200640 1 -- 172.21.15.28:0/4230818909 wait complete. 2024-05-12T22:06:40.265 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 --group_name qa 2024-05-12T22:06:40.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:40.466+0000 7f2840a00640 1 -- 172.21.15.28:0/3597389992 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f283c108f60 msgr2=0x7f283c1093e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:40.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:40.466+0000 7f2840a00640 1 --2- 172.21.15.28:0/3597389992 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f283c108f60 0x7f283c1093e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f2828009900 tx=0x7f282802a3f0 comp rx=0 tx=0).stop 2024-05-12T22:06:40.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:40.466+0000 7f2840a00640 1 -- 172.21.15.28:0/3597389992 shutdown_connections 2024-05-12T22:06:40.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:40.466+0000 7f2840a00640 1 --2- 172.21.15.28:0/3597389992 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f283c109920 0x7f283c1101b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:40.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:40.466+0000 7f2840a00640 1 --2- 172.21.15.28:0/3597389992 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f283c108f60 0x7f283c1093e0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:40.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:40.466+0000 7f2840a00640 1 --2- 172.21.15.28:0/3597389992 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f283c107d70 0x7f283c108190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:40.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:40.466+0000 7f2840a00640 1 -- 172.21.15.28:0/3597389992 >> 172.21.15.28:0/3597389992 conn(0x7f283c076d00 msgr2=0x7f283c079140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:06:40.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:40.466+0000 7f2840a00640 1 -- 172.21.15.28:0/3597389992 shutdown_connections 2024-05-12T22:06:40.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:40.466+0000 7f2840a00640 1 -- 172.21.15.28:0/3597389992 wait complete. 2024-05-12T22:06:40.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:40.467+0000 7f2840a00640 1 Processor -- start 2024-05-12T22:06:40.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:40.467+0000 7f2840a00640 1 -- start start 2024-05-12T22:06:40.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:40.468+0000 7f2840a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f283c107d70 0x7f283c19e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:40.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:40.468+0000 7f2840a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f283c108f60 0x7f283c19e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:40.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:40.468+0000 7f2840a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f283c109920 0x7f283c1a55f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:40.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:40.468+0000 7f2840a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f283c1139d0 con 0x7f283c108f60 2024-05-12T22:06:40.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:40.468+0000 7f2840a00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f283c113850 con 0x7f283c107d70 2024-05-12T22:06:40.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:40.468+0000 7f2840a00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f283c113b50 con 0x7f283c109920 2024-05-12T22:06:40.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:40.468+0000 7f2839e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f283c108f60 0x7f283c19e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:40.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:40.468+0000 7f283a800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f283c107d70 0x7f283c19e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:40.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:40.468+0000 7f2839e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f283c108f60 0x7f283c19e580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36344/0 (socket says 172.21.15.28:36344) 2024-05-12T22:06:40.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:40.468+0000 7f2839e00640 1 -- 172.21.15.28:0/716666462 learned_addr learned my addr 172.21.15.28:0/716666462 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:06:40.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:40.468+0000 7f283b200640 1 --2- 172.21.15.28:0/716666462 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f283c109920 0x7f283c1a55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:40.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:40.468+0000 7f283a800640 1 -- 172.21.15.28:0/716666462 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f283c109920 msgr2=0x7f283c1a55f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:40.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:40.468+0000 7f283a800640 1 --2- 172.21.15.28:0/716666462 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f283c109920 0x7f283c1a55f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:40.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:40.468+0000 7f283a800640 1 -- 172.21.15.28:0/716666462 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f283c108f60 msgr2=0x7f283c19e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:40.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:40.468+0000 7f283a800640 1 --2- 172.21.15.28:0/716666462 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f283c108f60 0x7f283c19e580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:40.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:40.468+0000 7f283a800640 1 -- 172.21.15.28:0/716666462 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f283c1a5c60 con 0x7f283c107d70 2024-05-12T22:06:40.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:40.469+0000 7f283a800640 1 --2- 172.21.15.28:0/716666462 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f283c107d70 0x7f283c19e040 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f282c00e9c0 tx=0x7f282c00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:06:40.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:40.469+0000 7f2833400640 1 -- 172.21.15.28:0/716666462 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f282c00ccc0 con 0x7f283c107d70 2024-05-12T22:06:40.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:40.469+0000 7f2833400640 1 -- 172.21.15.28:0/716666462 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f282c0035f0 con 0x7f283c107d70 2024-05-12T22:06:40.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:40.469+0000 7f2840a00640 1 -- 172.21.15.28:0/716666462 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f283c1a5f50 con 0x7f283c107d70 2024-05-12T22:06:40.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:40.469+0000 7f2833400640 1 -- 172.21.15.28:0/716666462 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f282c00ce60 con 0x7f283c107d70 2024-05-12T22:06:40.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:40.469+0000 7f2840a00640 1 -- 172.21.15.28:0/716666462 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f283c1a63b0 con 0x7f283c107d70 2024-05-12T22:06:40.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:40.470+0000 7f2840a00640 1 -- 172.21.15.28:0/716666462 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f283c10f090 con 0x7f283c107d70 2024-05-12T22:06:40.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:40.472+0000 7f2833400640 1 -- 172.21.15.28:0/716666462 <== mon.2 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f282c0037a0 con 0x7f283c107d70 2024-05-12T22:06:40.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:40.473+0000 7f2833400640 1 --2- 172.21.15.28:0/716666462 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f28200a2b90 0x7f28200a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:40.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:40.474+0000 7f2833400640 1 -- 172.21.15.28:0/716666462 <== mon.2 v2:172.21.15.86:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f282c0cf3b0 con 0x7f283c107d70 2024-05-12T22:06:40.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:40.474+0000 7f2839e00640 1 --2- 172.21.15.28:0/716666462 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f28200a2b90 0x7f28200a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:40.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:40.474+0000 7f2839e00640 1 --2- 172.21.15.28:0/716666462 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f28200a2b90 0x7f28200a5050 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f2828031440 tx=0x7f28280315b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:06:40.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:40.478+0000 7f2833400640 1 -- 172.21.15.28:0/716666462 <== mon.2 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f282c09c3c0 con 0x7f283c107d70 2024-05-12T22:06:40.535 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 22:06:40 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T22:06:40.214+0000 7f6259a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-12T22:06:40.536 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 22:06:40 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T22:06:40.214+0000 7f6259a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-12T22:06:40.536 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 22:06:40 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T22:06:40.214+0000 7f6259a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-12T22:06:40.536 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 22:06:40 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T22:06:40.214+0000 7f6259a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-12T22:06:40.536 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 22:06:40 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T22:06:40.214+0000 7f6259a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-12T22:06:40.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:40.714+0000 7f2840a00640 1 -- 172.21.15.28:0/716666462 --> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f283c061c70 con 0x7f28200a2b90 2024-05-12T22:06:40.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:40.750+0000 7f2833400640 1 -- 172.21.15.28:0/716666462 <== mgr.14152 v2:172.21.15.28:6800/3123328651 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f283c061c70 con 0x7f28200a2b90 2024-05-12T22:06:40.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:40.753+0000 7f2840a00640 1 -- 172.21.15.28:0/716666462 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f28200a2b90 msgr2=0x7f28200a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:40.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:40.753+0000 7f2840a00640 1 --2- 172.21.15.28:0/716666462 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f28200a2b90 0x7f28200a5050 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f2828031440 tx=0x7f28280315b0 comp rx=0 tx=0).stop 2024-05-12T22:06:40.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:40.753+0000 7f2840a00640 1 -- 172.21.15.28:0/716666462 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f283c107d70 msgr2=0x7f283c19e040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:40.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:40.753+0000 7f2840a00640 1 --2- 172.21.15.28:0/716666462 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f283c107d70 0x7f283c19e040 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f282c00e9c0 tx=0x7f282c00ee80 comp rx=0 tx=0).stop 2024-05-12T22:06:40.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:40.753+0000 7f2840a00640 1 -- 172.21.15.28:0/716666462 shutdown_connections 2024-05-12T22:06:40.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:40.753+0000 7f2840a00640 1 --2- 172.21.15.28:0/716666462 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f28200a2b90 0x7f28200a5050 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:40.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:40.753+0000 7f2840a00640 1 --2- 172.21.15.28:0/716666462 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f283c109920 0x7f283c1a55f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:40.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:40.753+0000 7f2840a00640 1 --2- 172.21.15.28:0/716666462 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f283c108f60 0x7f283c19e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:40.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:40.753+0000 7f2840a00640 1 --2- 172.21.15.28:0/716666462 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f283c107d70 0x7f283c19e040 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:40.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:40.753+0000 7f2840a00640 1 -- 172.21.15.28:0/716666462 >> 172.21.15.28:0/716666462 conn(0x7f283c076d00 msgr2=0x7f283c10c260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:06:40.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:40.753+0000 7f2840a00640 1 -- 172.21.15.28:0/716666462 shutdown_connections 2024-05-12T22:06:40.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:40.754+0000 7f2840a00640 1 -- 172.21.15.28:0/716666462 wait complete. 2024-05-12T22:06:40.775 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-05-12T22:06:40.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:40 smithi186 ceph-mon[34785]: pgmap v313: 97 pgs: 97 active+clean; 580 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3.5 KiB/s wr, 14 op/s 2024-05-12T22:06:40.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:40 smithi186 ceph-mon[34785]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-05-12T22:06:40.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:40 smithi186 ceph-mon[34785]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=g=up:active} 3 up:standby-replay 6 up:standby 2024-05-12T22:06:40.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:40 smithi186 ceph-mon[34785]: from='client.24665 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-12T22:06:40.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:40 smithi086 ceph-mon[34297]: pgmap v313: 97 pgs: 97 active+clean; 580 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3.5 KiB/s wr, 14 op/s 2024-05-12T22:06:40.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:40 smithi086 ceph-mon[34297]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-05-12T22:06:40.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:40 smithi086 ceph-mon[34297]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=g=up:active} 3 up:standby-replay 6 up:standby 2024-05-12T22:06:40.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:40 smithi086 ceph-mon[34297]: from='client.24665 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-12T22:06:41.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:40.999+0000 7f037f200640 1 -- 172.21.15.28:0/3427109661 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0380102260 msgr2=0x7f0380104650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:41.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:40.999+0000 7f037f200640 1 --2- 172.21.15.28:0/3427109661 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0380102260 0x7f0380104650 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f0370009900 tx=0x7f037002a410 comp rx=0 tx=0).stop 2024-05-12T22:06:41.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:40.999+0000 7f037f200640 1 -- 172.21.15.28:0/3427109661 shutdown_connections 2024-05-12T22:06:41.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:40.999+0000 7f037f200640 1 --2- 172.21.15.28:0/3427109661 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f03801074c0 0x7f03801098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:41.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:40.999+0000 7f037f200640 1 --2- 172.21.15.28:0/3427109661 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0380104b90 0x7f0380106f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:41.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:40.999+0000 7f037f200640 1 --2- 172.21.15.28:0/3427109661 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0380102260 0x7f0380104650 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:41.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:40.999+0000 7f037f200640 1 -- 172.21.15.28:0/3427109661 >> 172.21.15.28:0/3427109661 conn(0x7f03800fbbd0 msgr2=0x7f03800fdff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:06:41.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:41.000+0000 7f037f200640 1 -- 172.21.15.28:0/3427109661 shutdown_connections 2024-05-12T22:06:41.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:41.000+0000 7f037f200640 1 -- 172.21.15.28:0/3427109661 wait complete. 2024-05-12T22:06:41.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:41.001+0000 7f037f200640 1 Processor -- start 2024-05-12T22:06:41.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:41.001+0000 7f037f200640 1 -- start start 2024-05-12T22:06:41.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:41.001+0000 7f037f200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0380102260 0x7f0380199c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:41.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:41.001+0000 7f037f200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0380104b90 0x7f038019a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:41.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:41.002+0000 7f037f200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f03801074c0 0x7f03801a1230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:41.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:41.002+0000 7f037f200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f038010d400 con 0x7f0380104b90 2024-05-12T22:06:41.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:41.002+0000 7f037f200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f038010d280 con 0x7f03801074c0 2024-05-12T22:06:41.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:41.002+0000 7f037f200640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f038010d580 con 0x7f0380102260 2024-05-12T22:06:41.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:41.002+0000 7f037d400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0380104b90 0x7f038019a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:41.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:41.002+0000 7f037d400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0380104b90 0x7f038019a1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36370/0 (socket says 172.21.15.28:36370) 2024-05-12T22:06:41.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:41.002+0000 7f037d400640 1 -- 172.21.15.28:0/2761572830 learned_addr learned my addr 172.21.15.28:0/2761572830 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:06:41.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:41.002+0000 7f037de00640 1 --2- 172.21.15.28:0/2761572830 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0380102260 0x7f0380199c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:41.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:41.002+0000 7f037e800640 1 --2- 172.21.15.28:0/2761572830 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f03801074c0 0x7f03801a1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:41.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:41.002+0000 7f037d400640 1 -- 172.21.15.28:0/2761572830 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0380102260 msgr2=0x7f0380199c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:41.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:41.002+0000 7f037d400640 1 --2- 172.21.15.28:0/2761572830 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0380102260 0x7f0380199c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:41.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:41.002+0000 7f037d400640 1 -- 172.21.15.28:0/2761572830 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f03801074c0 msgr2=0x7f03801a1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:41.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:41.002+0000 7f037d400640 1 --2- 172.21.15.28:0/2761572830 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f03801074c0 0x7f03801a1230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:41.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:41.002+0000 7f037d400640 1 -- 172.21.15.28:0/2761572830 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03801a18d0 con 0x7f0380104b90 2024-05-12T22:06:41.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:41.003+0000 7f037d400640 1 --2- 172.21.15.28:0/2761572830 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0380104b90 0x7f038019a1d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f036800e9c0 tx=0x7f036800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:06:41.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:41.003+0000 7f0376a00640 1 -- 172.21.15.28:0/2761572830 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f036800cd80 con 0x7f0380104b90 2024-05-12T22:06:41.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:41.003+0000 7f0376a00640 1 -- 172.21.15.28:0/2761572830 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0368005500 con 0x7f0380104b90 2024-05-12T22:06:41.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:41.003+0000 7f037f200640 1 -- 172.21.15.28:0/2761572830 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03801a1bc0 con 0x7f0380104b90 2024-05-12T22:06:41.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:41.003+0000 7f0376a00640 1 -- 172.21.15.28:0/2761572830 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0368018630 con 0x7f0380104b90 2024-05-12T22:06:41.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:41.003+0000 7f037f200640 1 -- 172.21.15.28:0/2761572830 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03801a20e0 con 0x7f0380104b90 2024-05-12T22:06:41.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:41.005+0000 7f037f200640 1 -- 172.21.15.28:0/2761572830 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0380108fb0 con 0x7f0380104b90 2024-05-12T22:06:41.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:41.007+0000 7f0376a00640 1 -- 172.21.15.28:0/2761572830 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f03680046c0 con 0x7f0380104b90 2024-05-12T22:06:41.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:41.009+0000 7f0376a00640 1 --2- 172.21.15.28:0/2761572830 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f03640a2b90 0x7f03640a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:41.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:41.009+0000 7f0376a00640 1 -- 172.21.15.28:0/2761572830 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f03680cf390 con 0x7f0380104b90 2024-05-12T22:06:41.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:41.009+0000 7f037de00640 1 --2- 172.21.15.28:0/2761572830 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f03640a2b90 0x7f03640a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:41.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:41.010+0000 7f037de00640 1 --2- 172.21.15.28:0/2761572830 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f03640a2b90 0x7f03640a5050 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f0370031440 tx=0x7f03700315b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:06:41.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:41.014+0000 7f0376a00640 1 -- 172.21.15.28:0/2761572830 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f036809c440 con 0x7f0380104b90 2024-05-12T22:06:41.034 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:40 smithi028 ceph-mon[29847]: pgmap v313: 97 pgs: 97 active+clean; 580 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3.5 KiB/s wr, 14 op/s 2024-05-12T22:06:41.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:40 smithi028 ceph-mon[29847]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-05-12T22:06:41.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:40 smithi028 ceph-mon[29847]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=g=up:active} 3 up:standby-replay 6 up:standby 2024-05-12T22:06:41.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:40 smithi028 ceph-mon[29847]: from='client.24665 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-12T22:06:41.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:41.249+0000 7f037f200640 1 -- 172.21.15.28:0/2761572830 --> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) -- 0x7f0380061c70 con 0x7f03640a2b90 2024-05-12T22:06:41.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:41.258+0000 7f0376a00640 1 -- 172.21.15.28:0/2761572830 <== mgr.14152 v2:172.21.15.28:6800/3123328651 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f0380061c70 con 0x7f03640a2b90 2024-05-12T22:06:41.259 INFO:teuthology.orchestra.run.smithi028.stdout:/volumes/qa/sv_0/604a69d7-0749-4617-8a9d-2c153087712c 2024-05-12T22:06:41.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:41.261+0000 7f037f200640 1 -- 172.21.15.28:0/2761572830 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f03640a2b90 msgr2=0x7f03640a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:41.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:41.261+0000 7f037f200640 1 --2- 172.21.15.28:0/2761572830 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f03640a2b90 0x7f03640a5050 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f0370031440 tx=0x7f03700315b0 comp rx=0 tx=0).stop 2024-05-12T22:06:41.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:41.261+0000 7f037f200640 1 -- 172.21.15.28:0/2761572830 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0380104b90 msgr2=0x7f038019a1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:41.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:41.261+0000 7f037f200640 1 --2- 172.21.15.28:0/2761572830 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0380104b90 0x7f038019a1d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f036800e9c0 tx=0x7f036800ee80 comp rx=0 tx=0).stop 2024-05-12T22:06:41.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:41.261+0000 7f037f200640 1 -- 172.21.15.28:0/2761572830 shutdown_connections 2024-05-12T22:06:41.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:41.261+0000 7f037f200640 1 --2- 172.21.15.28:0/2761572830 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f03640a2b90 0x7f03640a5050 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:41.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:41.261+0000 7f037f200640 1 --2- 172.21.15.28:0/2761572830 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f03801074c0 0x7f03801a1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:41.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:41.261+0000 7f037f200640 1 --2- 172.21.15.28:0/2761572830 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0380104b90 0x7f038019a1d0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:41.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:41.261+0000 7f037f200640 1 --2- 172.21.15.28:0/2761572830 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0380102260 0x7f0380199c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:41.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:41.261+0000 7f037f200640 1 -- 172.21.15.28:0/2761572830 >> 172.21.15.28:0/2761572830 conn(0x7f03800fbbd0 msgr2=0x7f0380101960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:06:41.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:41.261+0000 7f037f200640 1 -- 172.21.15.28:0/2761572830 shutdown_connections 2024-05-12T22:06:41.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:41.261+0000 7f037f200640 1 -- 172.21.15.28:0/2761572830 wait complete. 2024-05-12T22:06:41.282 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 --group_name qa 2024-05-12T22:06:41.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:41.495+0000 7fe070800640 1 -- 172.21.15.28:0/3675617447 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe068103430 msgr2=0x7fe068109cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:41.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:41.495+0000 7fe070800640 1 --2- 172.21.15.28:0/3675617447 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe068103430 0x7fe068109cc0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fe064009ca0 tx=0x7fe06402a8c0 comp rx=0 tx=0).stop 2024-05-12T22:06:41.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:41.495+0000 7fe070800640 1 -- 172.21.15.28:0/3675617447 shutdown_connections 2024-05-12T22:06:41.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:41.495+0000 7fe070800640 1 --2- 172.21.15.28:0/3675617447 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe068103430 0x7fe068109cc0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:41.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:41.495+0000 7fe070800640 1 --2- 172.21.15.28:0/3675617447 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe068102a70 0x7fe068102ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:41.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:41.495+0000 7fe070800640 1 --2- 172.21.15.28:0/3675617447 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe068101880 0x7fe068101ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:41.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:41.495+0000 7fe070800640 1 -- 172.21.15.28:0/3675617447 >> 172.21.15.28:0/3675617447 conn(0x7fe0680fce10 msgr2=0x7fe0680ff230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:06:41.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:41.496+0000 7fe070800640 1 -- 172.21.15.28:0/3675617447 shutdown_connections 2024-05-12T22:06:41.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:41.496+0000 7fe070800640 1 -- 172.21.15.28:0/3675617447 wait complete. 2024-05-12T22:06:41.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:41.497+0000 7fe070800640 1 Processor -- start 2024-05-12T22:06:41.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:41.497+0000 7fe070800640 1 -- start start 2024-05-12T22:06:41.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:41.497+0000 7fe070800640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe068101880 0x7fe0680755e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:41.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:41.497+0000 7fe070800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe068102a70 0x7fe068073c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:41.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:41.497+0000 7fe070800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe068103430 0x7fe068074160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:41.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:41.497+0000 7fe070800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe06810d500 con 0x7fe068102a70 2024-05-12T22:06:41.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:41.497+0000 7fe070800640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fe06810d380 con 0x7fe068103430 2024-05-12T22:06:41.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:41.497+0000 7fe070800640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe06810d680 con 0x7fe068101880 2024-05-12T22:06:41.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:41.497+0000 7fe06d400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe068102a70 0x7fe068073c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:41.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:41.497+0000 7fe06e800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe068103430 0x7fe068074160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:41.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:41.497+0000 7fe06de00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe068101880 0x7fe0680755e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:41.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:41.497+0000 7fe06e800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe068103430 0x7fe068074160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:58270/0 (socket says 172.21.15.28:58270) 2024-05-12T22:06:41.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:41.497+0000 7fe06e800640 1 -- 172.21.15.28:0/3779619377 learned_addr learned my addr 172.21.15.28:0/3779619377 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:06:41.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:41.498+0000 7fe06e800640 1 -- 172.21.15.28:0/3779619377 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe068101880 msgr2=0x7fe0680755e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:41.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:41.498+0000 7fe06e800640 1 --2- 172.21.15.28:0/3779619377 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe068101880 0x7fe0680755e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:41.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:41.498+0000 7fe06e800640 1 -- 172.21.15.28:0/3779619377 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe068102a70 msgr2=0x7fe068073c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:41.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:41.498+0000 7fe06e800640 1 --2- 172.21.15.28:0/3779619377 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe068102a70 0x7fe068073c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:41.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:41.498+0000 7fe06e800640 1 -- 172.21.15.28:0/3779619377 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0680749f0 con 0x7fe068103430 2024-05-12T22:06:41.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:41.498+0000 7fe06d400640 1 --2- 172.21.15.28:0/3779619377 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe068102a70 0x7fe068073c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:06:41.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:41.498+0000 7fe06e800640 1 --2- 172.21.15.28:0/3779619377 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe068103430 0x7fe068074160 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fe064031ae0 tx=0x7fe064031ee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:06:41.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:41.498+0000 7fe062a00640 1 -- 172.21.15.28:0/3779619377 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe064003b70 con 0x7fe068103430 2024-05-12T22:06:41.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:41.498+0000 7fe070800640 1 -- 172.21.15.28:0/3779619377 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0681a1c60 con 0x7fe068103430 2024-05-12T22:06:41.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:41.498+0000 7fe062a00640 1 -- 172.21.15.28:0/3779619377 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe064003d10 con 0x7fe068103430 2024-05-12T22:06:41.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:41.498+0000 7fe062a00640 1 -- 172.21.15.28:0/3779619377 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe06402fc40 con 0x7fe068103430 2024-05-12T22:06:41.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:41.499+0000 7fe070800640 1 -- 172.21.15.28:0/3779619377 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0681a21b0 con 0x7fe068103430 2024-05-12T22:06:41.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:41.500+0000 7fe070800640 1 -- 172.21.15.28:0/3779619377 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe068108ba0 con 0x7fe068103430 2024-05-12T22:06:41.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:41.502+0000 7fe062a00640 1 -- 172.21.15.28:0/3779619377 <== mon.2 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fe064004050 con 0x7fe068103430 2024-05-12T22:06:41.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:41.503+0000 7fe062a00640 1 --2- 172.21.15.28:0/3779619377 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fe0500a2b90 0x7fe0500a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:41.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:41.503+0000 7fe062a00640 1 -- 172.21.15.28:0/3779619377 <== mon.2 v2:172.21.15.86:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7fe0640f0060 con 0x7fe068103430 2024-05-12T22:06:41.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:41.503+0000 7fe06de00640 1 --2- 172.21.15.28:0/3779619377 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fe0500a2b90 0x7fe0500a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:41.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:41.504+0000 7fe06de00640 1 --2- 172.21.15.28:0/3779619377 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fe0500a2b90 0x7fe0500a5050 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fe0681028d0 tx=0x7fe05c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:06:41.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:41.507+0000 7fe062a00640 1 -- 172.21.15.28:0/3779619377 <== mon.2 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fe0640bd190 con 0x7fe068103430 2024-05-12T22:06:41.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:41.750+0000 7fe070800640 1 -- 172.21.15.28:0/3779619377 --> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7fe068061c70 con 0x7fe0500a2b90 2024-05-12T22:06:41.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:41.791+0000 7fe062a00640 1 -- 172.21.15.28:0/3779619377 <== mgr.14152 v2:172.21.15.28:6800/3123328651 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fe068061c70 con 0x7fe0500a2b90 2024-05-12T22:06:41.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:41.795+0000 7fe070800640 1 -- 172.21.15.28:0/3779619377 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fe0500a2b90 msgr2=0x7fe0500a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:41.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:41.795+0000 7fe070800640 1 --2- 172.21.15.28:0/3779619377 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fe0500a2b90 0x7fe0500a5050 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fe0681028d0 tx=0x7fe05c005f90 comp rx=0 tx=0).stop 2024-05-12T22:06:41.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:41.795+0000 7fe070800640 1 -- 172.21.15.28:0/3779619377 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe068103430 msgr2=0x7fe068074160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:41.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:41.795+0000 7fe070800640 1 --2- 172.21.15.28:0/3779619377 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe068103430 0x7fe068074160 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fe064031ae0 tx=0x7fe064031ee0 comp rx=0 tx=0).stop 2024-05-12T22:06:41.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:41.795+0000 7fe070800640 1 -- 172.21.15.28:0/3779619377 shutdown_connections 2024-05-12T22:06:41.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:41.795+0000 7fe070800640 1 --2- 172.21.15.28:0/3779619377 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fe0500a2b90 0x7fe0500a5050 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:41.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:41.795+0000 7fe070800640 1 --2- 172.21.15.28:0/3779619377 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe068103430 0x7fe068074160 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:41.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:41.795+0000 7fe070800640 1 --2- 172.21.15.28:0/3779619377 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe068102a70 0x7fe068073c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:41.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:41.795+0000 7fe070800640 1 --2- 172.21.15.28:0/3779619377 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe068101880 0x7fe0680755e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:41.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:41.795+0000 7fe070800640 1 -- 172.21.15.28:0/3779619377 >> 172.21.15.28:0/3779619377 conn(0x7fe0680fce10 msgr2=0x7fe068105d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:06:41.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:41.795+0000 7fe070800640 1 -- 172.21.15.28:0/3779619377 shutdown_connections 2024-05-12T22:06:41.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:41.795+0000 7fe070800640 1 -- 172.21.15.28:0/3779619377 wait complete. 2024-05-12T22:06:41.817 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-05-12T22:06:41.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:41 smithi086 ceph-mon[34297]: from='client.24677 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-12T22:06:41.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:41 smithi086 ceph-mon[34297]: from='client.14565 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-05-12T22:06:42.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:41 smithi028 ceph-mon[29847]: from='client.24677 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-12T22:06:42.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:41 smithi028 ceph-mon[29847]: from='client.14565 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-05-12T22:06:42.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:42.038+0000 7f947e600640 1 -- 172.21.15.28:0/2572514880 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9478104c40 msgr2=0x7f9478107030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:42.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:42.038+0000 7f947e600640 1 --2- 172.21.15.28:0/2572514880 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9478104c40 0x7f9478107030 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f9460009ca0 tx=0x7f946002a8c0 comp rx=0 tx=0).stop 2024-05-12T22:06:42.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:42.038+0000 7f947e600640 1 -- 172.21.15.28:0/2572514880 shutdown_connections 2024-05-12T22:06:42.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:42.038+0000 7f947e600640 1 --2- 172.21.15.28:0/2572514880 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9478107570 0x7f94781099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:42.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:42.038+0000 7f947e600640 1 --2- 172.21.15.28:0/2572514880 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9478104c40 0x7f9478107030 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:42.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:42.038+0000 7f947e600640 1 --2- 172.21.15.28:0/2572514880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9478102310 0x7f9478104700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:42.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:42.038+0000 7f947e600640 1 -- 172.21.15.28:0/2572514880 >> 172.21.15.28:0/2572514880 conn(0x7f94780fbd20 msgr2=0x7f94780fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:06:42.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:42.038+0000 7f947e600640 1 -- 172.21.15.28:0/2572514880 shutdown_connections 2024-05-12T22:06:42.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:42.038+0000 7f947e600640 1 -- 172.21.15.28:0/2572514880 wait complete. 2024-05-12T22:06:42.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:42.039+0000 7f947e600640 1 Processor -- start 2024-05-12T22:06:42.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:42.039+0000 7f947e600640 1 -- start start 2024-05-12T22:06:42.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:42.040+0000 7f947e600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9478102310 0x7f9478199df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:42.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:42.040+0000 7f947e600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9478104c40 0x7f947819a330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:42.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:42.040+0000 7f947e600640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9478107570 0x7f94781a13a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:42.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:42.040+0000 7f947e600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f947810d560 con 0x7f9478104c40 2024-05-12T22:06:42.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:42.040+0000 7f947e600640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f947810d3e0 con 0x7f9478102310 2024-05-12T22:06:42.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:42.040+0000 7f947e600640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f947810d6e0 con 0x7f9478107570 2024-05-12T22:06:42.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:42.040+0000 7f9476a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9478104c40 0x7f947819a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:42.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:42.040+0000 7f9477400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9478102310 0x7f9478199df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:42.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:42.040+0000 7f9476a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9478104c40 0x7f947819a330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36400/0 (socket says 172.21.15.28:36400) 2024-05-12T22:06:42.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:42.040+0000 7f9477400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9478102310 0x7f9478199df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:58298/0 (socket says 172.21.15.28:58298) 2024-05-12T22:06:42.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:42.040+0000 7f9476a00640 1 -- 172.21.15.28:0/2524244053 learned_addr learned my addr 172.21.15.28:0/2524244053 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:06:42.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:42.040+0000 7f9477e00640 1 --2- 172.21.15.28:0/2524244053 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9478107570 0x7f94781a13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:42.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:42.040+0000 7f9477400640 1 -- 172.21.15.28:0/2524244053 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9478107570 msgr2=0x7f94781a13a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:42.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:42.040+0000 7f9477400640 1 --2- 172.21.15.28:0/2524244053 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9478107570 0x7f94781a13a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:42.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:42.040+0000 7f9477400640 1 -- 172.21.15.28:0/2524244053 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9478104c40 msgr2=0x7f947819a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:42.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:42.040+0000 7f9477400640 1 --2- 172.21.15.28:0/2524244053 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9478104c40 0x7f947819a330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:42.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:42.040+0000 7f9477400640 1 -- 172.21.15.28:0/2524244053 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94781a1a10 con 0x7f9478102310 2024-05-12T22:06:42.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:42.041+0000 7f9477400640 1 --2- 172.21.15.28:0/2524244053 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9478102310 0x7f9478199df0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f947000e9f0 tx=0x7f947000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:06:42.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:42.041+0000 7f946fe00640 1 -- 172.21.15.28:0/2524244053 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f947000cd80 con 0x7f9478102310 2024-05-12T22:06:42.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:42.041+0000 7f946fe00640 1 -- 172.21.15.28:0/2524244053 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f94700035f0 con 0x7f9478102310 2024-05-12T22:06:42.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:42.041+0000 7f946fe00640 1 -- 172.21.15.28:0/2524244053 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9470010690 con 0x7f9478102310 2024-05-12T22:06:42.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:42.041+0000 7f947e600640 1 -- 172.21.15.28:0/2524244053 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94781a1d00 con 0x7f9478102310 2024-05-12T22:06:42.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:42.041+0000 7f947e600640 1 -- 172.21.15.28:0/2524244053 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94781a21a0 con 0x7f9478102310 2024-05-12T22:06:42.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:42.042+0000 7f947e600640 1 -- 172.21.15.28:0/2524244053 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94781090c0 con 0x7f9478102310 2024-05-12T22:06:42.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:42.044+0000 7f946fe00640 1 -- 172.21.15.28:0/2524244053 <== mon.2 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f9470003b30 con 0x7f9478102310 2024-05-12T22:06:42.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:42.046+0000 7f946fe00640 1 --2- 172.21.15.28:0/2524244053 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f945c0a2c10 0x7f945c0a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:42.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:42.046+0000 7f946fe00640 1 -- 172.21.15.28:0/2524244053 <== mon.2 v2:172.21.15.86:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f94700cf1e0 con 0x7f9478102310 2024-05-12T22:06:42.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:42.046+0000 7f9476a00640 1 --2- 172.21.15.28:0/2524244053 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f945c0a2c10 0x7f945c0a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:42.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:42.046+0000 7f9476a00640 1 --2- 172.21.15.28:0/2524244053 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f945c0a2c10 0x7f945c0a50d0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f947819b300 tx=0x7f9460005f70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:06:42.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:42.050+0000 7f946fe00640 1 -- 172.21.15.28:0/2524244053 <== mon.2 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f947009c1f0 con 0x7f9478102310 2024-05-12T22:06:42.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:41 smithi186 ceph-mon[34785]: from='client.24677 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-12T22:06:42.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:41 smithi186 ceph-mon[34785]: from='client.14565 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-05-12T22:06:42.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:42.286+0000 7f947e600640 1 -- 172.21.15.28:0/2524244053 --> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) -- 0x7f9478061c70 con 0x7f945c0a2c10 2024-05-12T22:06:42.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:42.293+0000 7f946fe00640 1 -- 172.21.15.28:0/2524244053 <== mgr.14152 v2:172.21.15.28:6800/3123328651 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f9478061c70 con 0x7f945c0a2c10 2024-05-12T22:06:42.293 INFO:teuthology.orchestra.run.smithi028.stdout:/volumes/qa/sv_1/6f9c1691-e25c-43e3-9e4e-205c2965b112 2024-05-12T22:06:42.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:42.295+0000 7f947e600640 1 -- 172.21.15.28:0/2524244053 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f945c0a2c10 msgr2=0x7f945c0a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:42.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:42.295+0000 7f947e600640 1 --2- 172.21.15.28:0/2524244053 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f945c0a2c10 0x7f945c0a50d0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f947819b300 tx=0x7f9460005f70 comp rx=0 tx=0).stop 2024-05-12T22:06:42.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:42.295+0000 7f947e600640 1 -- 172.21.15.28:0/2524244053 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9478102310 msgr2=0x7f9478199df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:42.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:42.295+0000 7f947e600640 1 --2- 172.21.15.28:0/2524244053 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9478102310 0x7f9478199df0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f947000e9f0 tx=0x7f947000eeb0 comp rx=0 tx=0).stop 2024-05-12T22:06:42.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:42.296+0000 7f947e600640 1 -- 172.21.15.28:0/2524244053 shutdown_connections 2024-05-12T22:06:42.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:42.296+0000 7f947e600640 1 --2- 172.21.15.28:0/2524244053 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f945c0a2c10 0x7f945c0a50d0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:42.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:42.296+0000 7f947e600640 1 --2- 172.21.15.28:0/2524244053 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9478107570 0x7f94781a13a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:42.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:42.296+0000 7f947e600640 1 --2- 172.21.15.28:0/2524244053 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9478104c40 0x7f947819a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:42.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:42.296+0000 7f947e600640 1 --2- 172.21.15.28:0/2524244053 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9478102310 0x7f9478199df0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:42.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:42.296+0000 7f947e600640 1 -- 172.21.15.28:0/2524244053 >> 172.21.15.28:0/2524244053 conn(0x7f94780fbd20 msgr2=0x7f9478101a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:06:42.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:42.296+0000 7f947e600640 1 -- 172.21.15.28:0/2524244053 shutdown_connections 2024-05-12T22:06:42.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:42.296+0000 7f947e600640 1 -- 172.21.15.28:0/2524244053 wait complete. 2024-05-12T22:06:42.316 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-12T22:06:42.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:42.519+0000 7fa81f200640 1 -- 172.21.15.28:0/3085525465 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa820077430 msgr2=0x7fa820077890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:42.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:42.519+0000 7fa81f200640 1 --2- 172.21.15.28:0/3085525465 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa820077430 0x7fa820077890 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fa808009900 tx=0x7fa80802a410 comp rx=0 tx=0).stop 2024-05-12T22:06:42.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:42.519+0000 7fa81f200640 1 -- 172.21.15.28:0/3085525465 shutdown_connections 2024-05-12T22:06:42.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:42.519+0000 7fa81f200640 1 --2- 172.21.15.28:0/3085525465 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa820077dd0 0x7fa820078670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:42.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:42.519+0000 7fa81f200640 1 --2- 172.21.15.28:0/3085525465 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa820077430 0x7fa820077890 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:42.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:42.519+0000 7fa81f200640 1 --2- 172.21.15.28:0/3085525465 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa8200761f0 0x7fa820076610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:42.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:42.519+0000 7fa81f200640 1 -- 172.21.15.28:0/3085525465 >> 172.21.15.28:0/3085525465 conn(0x7fa8200fee40 msgr2=0x7fa820101280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:06:42.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:42.519+0000 7fa81f200640 1 -- 172.21.15.28:0/3085525465 shutdown_connections 2024-05-12T22:06:42.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:42.519+0000 7fa81f200640 1 -- 172.21.15.28:0/3085525465 wait complete. 2024-05-12T22:06:42.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:42.520+0000 7fa81f200640 1 Processor -- start 2024-05-12T22:06:42.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:42.520+0000 7fa81f200640 1 -- start start 2024-05-12T22:06:42.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:42.520+0000 7fa81f200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa8200761f0 0x7fa82019dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:42.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:42.520+0000 7fa81f200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa820077430 0x7fa82019e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:42.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:42.520+0000 7fa81f200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa820077dd0 0x7fa8201a5590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:42.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:42.520+0000 7fa81f200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa820113960 con 0x7fa820077dd0 2024-05-12T22:06:42.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:42.520+0000 7fa81f200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fa8201137e0 con 0x7fa8200761f0 2024-05-12T22:06:42.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:42.520+0000 7fa81f200640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa820113ae0 con 0x7fa820077430 2024-05-12T22:06:42.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:42.521+0000 7fa81e800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa820077dd0 0x7fa8201a5590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:42.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:42.521+0000 7fa815400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa820077430 0x7fa82019e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:42.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:42.521+0000 7fa81de00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa8200761f0 0x7fa82019dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:42.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:42.521+0000 7fa815400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa820077430 0x7fa82019e520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:50658/0 (socket says 172.21.15.28:50658) 2024-05-12T22:06:42.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:42.521+0000 7fa815400640 1 -- 172.21.15.28:0/1837148121 learned_addr learned my addr 172.21.15.28:0/1837148121 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:06:42.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:42.521+0000 7fa81e800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa820077dd0 0x7fa8201a5590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36426/0 (socket says 172.21.15.28:36426) 2024-05-12T22:06:42.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:42.521+0000 7fa815400640 1 -- 172.21.15.28:0/1837148121 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa8200761f0 msgr2=0x7fa82019dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:42.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:42.521+0000 7fa815400640 1 --2- 172.21.15.28:0/1837148121 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa8200761f0 0x7fa82019dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:42.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:42.521+0000 7fa815400640 1 -- 172.21.15.28:0/1837148121 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa820077dd0 msgr2=0x7fa8201a5590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:42.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:42.521+0000 7fa815400640 1 --2- 172.21.15.28:0/1837148121 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa820077dd0 0x7fa8201a5590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:42.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:42.521+0000 7fa815400640 1 -- 172.21.15.28:0/1837148121 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8201a5c00 con 0x7fa820077430 2024-05-12T22:06:42.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:42.522+0000 7fa815400640 1 --2- 172.21.15.28:0/1837148121 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa820077430 0x7fa82019e520 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fa808009230 tx=0x7fa8080042f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:06:42.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:42.522+0000 7fa817400640 1 -- 172.21.15.28:0/1837148121 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa80803f030 con 0x7fa820077430 2024-05-12T22:06:42.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:42.522+0000 7fa817400640 1 -- 172.21.15.28:0/1837148121 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa808003ce0 con 0x7fa820077430 2024-05-12T22:06:42.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:42.522+0000 7fa817400640 1 -- 172.21.15.28:0/1837148121 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa808035070 con 0x7fa820077430 2024-05-12T22:06:42.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:42.522+0000 7fa81f200640 1 -- 172.21.15.28:0/1837148121 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8201a5e90 con 0x7fa820077430 2024-05-12T22:06:42.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:42.522+0000 7fa81f200640 1 -- 172.21.15.28:0/1837148121 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8201a62a0 con 0x7fa820077430 2024-05-12T22:06:42.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:42.523+0000 7fa81f200640 1 -- 172.21.15.28:0/1837148121 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa820110240 con 0x7fa820077430 2024-05-12T22:06:42.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:42.525+0000 7fa817400640 1 -- 172.21.15.28:0/1837148121 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fa808042030 con 0x7fa820077430 2024-05-12T22:06:42.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:42.527+0000 7fa817400640 1 --2- 172.21.15.28:0/1837148121 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fa8040a2c10 0x7fa8040a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:42.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:42.527+0000 7fa817400640 1 -- 172.21.15.28:0/1837148121 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7fa8080ef790 con 0x7fa820077430 2024-05-12T22:06:42.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:42.527+0000 7fa81de00640 1 --2- 172.21.15.28:0/1837148121 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fa8040a2c10 0x7fa8040a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:42.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:42.528+0000 7fa81de00640 1 --2- 172.21.15.28:0/1837148121 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fa8040a2c10 0x7fa8040a50d0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fa8100095f0 tx=0x7fa810005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:06:42.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:42.529+0000 7fa817400640 1 -- 172.21.15.28:0/1837148121 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fa8080bc150 con 0x7fa820077430 2024-05-12T22:06:42.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:42.737+0000 7fa817400640 1 -- 172.21.15.28:0/1837148121 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fa80803b460 con 0x7fa820077430 2024-05-12T22:06:42.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:42.783+0000 7fa81f200640 1 -- 172.21.15.28:0/1837148121 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa82002d1c0 con 0x7fa820077430 2024-05-12T22:06:42.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:42.785+0000 7fa817400640 1 -- 172.21.15.28:0/1837148121 <== mon.1 v2:172.21.15.186:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11864 (secure 0 0 0) 0x7fa8080e9bc0 con 0x7fa820077430 2024-05-12T22:06:42.785 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-12T22:06:42.786 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1844141745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1844141745},{"type":"v1","addr":"172.21.15.186:6833","nonce":1844141745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6833/2960763758","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":2960763758},{"type":"v1","addr":"172.21.15.86:6833","nonce":2960763758}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24470,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/4131134219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":4131134219},{"type":"v1","addr":"172.21.15.28:6835","nonce":4131134219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24481,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/1727869064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1727869064},{"type":"v1","addr":"172.21.15.186:6835","nonce":1727869064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24482,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2938260704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2938260704},{"type":"v1","addr":"172.21.15.28:6837","nonce":2938260704}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24494,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3473700910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3473700910},{"type":"v1","addr":"172.21.15.86:6835","nonce":3473700910}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-12T22:06:33.426986+0000","modified":"2024-05-12T22:06:38.789246+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24581,"mds_1":24550,"mds_2":24532},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24499":{"gid":24499,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6837/3600809235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":3600809235},{"type":"v1","addr":"172.21.15.186:6837","nonce":3600809235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"i","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6839/2399610807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":2399610807},{"type":"v1","addr":"172.21.15.186:6839","nonce":2399610807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24530":{"gid":24530,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.86:6837/3782448884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":3782448884},{"type":"v1","addr":"172.21.15.86:6837","nonce":3782448884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24532":{"gid":24532,"name":"g","rank":2,"incarnation":21,"state":"up:active","state_seq":6,"addr":"172.21.15.28:6839/123892158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":123892158},{"type":"v1","addr":"172.21.15.28:6839","nonce":123892158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24550":{"gid":24550,"name":"a","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.28:6841/761183870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":761183870},{"type":"v1","addr":"172.21.15.28:6841","nonce":761183870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24581":{"gid":24581,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6839/1251999130","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6838","nonce":1251999130},{"type":"v1","addr":"172.21.15.86:6839","nonce":1251999130}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24581,"qdb_cluster":[24550,24532,24581]},"id":1}]} 2024-05-12T22:06:42.786 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 23 2024-05-12T22:06:42.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:42.787+0000 7fa81f200640 1 -- 172.21.15.28:0/1837148121 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fa8040a2c10 msgr2=0x7fa8040a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:42.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:42.787+0000 7fa81f200640 1 --2- 172.21.15.28:0/1837148121 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fa8040a2c10 0x7fa8040a50d0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fa8100095f0 tx=0x7fa810005f90 comp rx=0 tx=0).stop 2024-05-12T22:06:42.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:42.787+0000 7fa81f200640 1 -- 172.21.15.28:0/1837148121 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa820077430 msgr2=0x7fa82019e520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:42.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:42.787+0000 7fa81f200640 1 --2- 172.21.15.28:0/1837148121 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa820077430 0x7fa82019e520 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fa808009230 tx=0x7fa8080042f0 comp rx=0 tx=0).stop 2024-05-12T22:06:42.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:42.787+0000 7fa81f200640 1 -- 172.21.15.28:0/1837148121 shutdown_connections 2024-05-12T22:06:42.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:42.787+0000 7fa81f200640 1 --2- 172.21.15.28:0/1837148121 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fa8040a2c10 0x7fa8040a50d0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:42.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:42.787+0000 7fa81f200640 1 --2- 172.21.15.28:0/1837148121 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa820077dd0 0x7fa8201a5590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:42.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:42.787+0000 7fa81f200640 1 --2- 172.21.15.28:0/1837148121 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa820077430 0x7fa82019e520 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:42.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:42.787+0000 7fa81f200640 1 --2- 172.21.15.28:0/1837148121 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa8200761f0 0x7fa82019dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:42.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:42.787+0000 7fa81f200640 1 -- 172.21.15.28:0/1837148121 >> 172.21.15.28:0/1837148121 conn(0x7fa8200fee40 msgr2=0x7fa820100940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:06:42.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:42.788+0000 7fa81f200640 1 -- 172.21.15.28:0/1837148121 shutdown_connections 2024-05-12T22:06:42.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:42.788+0000 7fa81f200640 1 -- 172.21.15.28:0/1837148121 wait complete. 2024-05-12T22:06:42.808 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-12T22:06:43.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:43.007+0000 7f652b000640 1 -- 172.21.15.28:0/2305853047 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6524102890 msgr2=0x7f6524104d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:43.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:43.007+0000 7f652b000640 1 --2- 172.21.15.28:0/2305853047 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6524102890 0x7f6524104d00 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f6510009900 tx=0x7f651002a400 comp rx=0 tx=0).stop 2024-05-12T22:06:43.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:43.007+0000 7f652b000640 1 -- 172.21.15.28:0/2305853047 shutdown_connections 2024-05-12T22:06:43.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:43.007+0000 7f652b000640 1 --2- 172.21.15.28:0/2305853047 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6524105240 0x7f65241056c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:43.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:43.007+0000 7f652b000640 1 --2- 172.21.15.28:0/2305853047 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6524102890 0x7f6524104d00 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:43.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:43.007+0000 7f652b000640 1 --2- 172.21.15.28:0/2305853047 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f65240fff20 0x7f6524102350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:43.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:43.007+0000 7f652b000640 1 -- 172.21.15.28:0/2305853047 >> 172.21.15.28:0/2305853047 conn(0x7f65240fbb00 msgr2=0x7f65240fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:06:43.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:43.008+0000 7f652b000640 1 -- 172.21.15.28:0/2305853047 shutdown_connections 2024-05-12T22:06:43.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:43.008+0000 7f652b000640 1 -- 172.21.15.28:0/2305853047 wait complete. 2024-05-12T22:06:43.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:43.009+0000 7f652b000640 1 Processor -- start 2024-05-12T22:06:43.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:43.009+0000 7f652b000640 1 -- start start 2024-05-12T22:06:43.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:43.010+0000 7f652b000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f65240fff20 0x7f652419df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:43.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:43.010+0000 7f652b000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6524102890 0x7f652419e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:43.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:43.010+0000 7f652b000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6524105240 0x7f65241a5530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:43.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:43.010+0000 7f652b000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6524113970 con 0x7f6524102890 2024-05-12T22:06:43.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:43.010+0000 7f652b000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f65241137f0 con 0x7f6524105240 2024-05-12T22:06:43.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:43.010+0000 7f652b000640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f6524113af0 con 0x7f65240fff20 2024-05-12T22:06:43.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:43.010+0000 7f6529200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6524102890 0x7f652419e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:43.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:43.010+0000 7f6529200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6524102890 0x7f652419e4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36450/0 (socket says 172.21.15.28:36450) 2024-05-12T22:06:43.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:43.010+0000 7f6529200640 1 -- 172.21.15.28:0/432021215 learned_addr learned my addr 172.21.15.28:0/432021215 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:06:43.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:43.010+0000 7f6529c00640 1 --2- 172.21.15.28:0/432021215 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f65240fff20 0x7f652419df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:43.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:43.010+0000 7f652a600640 1 --2- 172.21.15.28:0/432021215 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6524105240 0x7f65241a5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:43.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:43.010+0000 7f6529200640 1 -- 172.21.15.28:0/432021215 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f65240fff20 msgr2=0x7f652419df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:43.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:43.010+0000 7f6529200640 1 --2- 172.21.15.28:0/432021215 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f65240fff20 0x7f652419df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:43.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:43.010+0000 7f6529200640 1 -- 172.21.15.28:0/432021215 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6524105240 msgr2=0x7f65241a5530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:43.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:43.010+0000 7f6529200640 1 --2- 172.21.15.28:0/432021215 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6524105240 0x7f65241a5530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:43.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:43.010+0000 7f6529200640 1 -- 172.21.15.28:0/432021215 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65241a5ba0 con 0x7f6524102890 2024-05-12T22:06:43.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:43.011+0000 7f6529200640 1 --2- 172.21.15.28:0/432021215 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6524102890 0x7f652419e4c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f6510031440 tx=0x7f6510032ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:06:43.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:43.011+0000 7f6522000640 1 -- 172.21.15.28:0/432021215 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f651003f030 con 0x7f6524102890 2024-05-12T22:06:43.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:43.011+0000 7f6522000640 1 -- 172.21.15.28:0/432021215 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6510003ce0 con 0x7f6524102890 2024-05-12T22:06:43.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:43.011+0000 7f6522000640 1 -- 172.21.15.28:0/432021215 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6510035070 con 0x7f6524102890 2024-05-12T22:06:43.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:43.011+0000 7f652b000640 1 -- 172.21.15.28:0/432021215 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65241a5e30 con 0x7f6524102890 2024-05-12T22:06:43.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:43.011+0000 7f652b000640 1 -- 172.21.15.28:0/432021215 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65241a6240 con 0x7f6524102890 2024-05-12T22:06:43.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:43.013+0000 7f652b000640 1 -- 172.21.15.28:0/432021215 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65240fd5f0 con 0x7f6524102890 2024-05-12T22:06:43.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:43.014+0000 7f6522000640 1 -- 172.21.15.28:0/432021215 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f6510042030 con 0x7f6524102890 2024-05-12T22:06:43.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:43.016+0000 7f6522000640 1 --2- 172.21.15.28:0/432021215 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f650c0a2c20 0x7f650c0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:43.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:43.016+0000 7f6522000640 1 -- 172.21.15.28:0/432021215 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f65100ef910 con 0x7f6524102890 2024-05-12T22:06:43.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:43.016+0000 7f6529c00640 1 --2- 172.21.15.28:0/432021215 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f650c0a2c20 0x7f650c0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:43.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:43.017+0000 7f6529c00640 1 --2- 172.21.15.28:0/432021215 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f650c0a2c20 0x7f650c0a50e0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f6518005fd0 tx=0x7f6518009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:06:43.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:43.020+0000 7f6522000640 1 -- 172.21.15.28:0/432021215 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f65100bc170 con 0x7f6524102890 2024-05-12T22:06:43.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:42 smithi028 ceph-mon[29847]: pgmap v314: 97 pgs: 97 active+clean; 580 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4.2 KiB/s wr, 16 op/s 2024-05-12T22:06:43.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:42 smithi028 ceph-mon[29847]: from='client.24689 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-12T22:06:43.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:42 smithi028 ceph-mon[29847]: from='client.24695 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-12T22:06:43.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:42 smithi186 ceph-mon[34785]: pgmap v314: 97 pgs: 97 active+clean; 580 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4.2 KiB/s wr, 16 op/s 2024-05-12T22:06:43.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:42 smithi186 ceph-mon[34785]: from='client.24689 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-12T22:06:43.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:42 smithi186 ceph-mon[34785]: from='client.24695 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-12T22:06:43.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:43.227+0000 7f652b000640 1 -- 172.21.15.28:0/432021215 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f652402d410 con 0x7f6524102890 2024-05-12T22:06:43.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:43.230+0000 7f6522000640 1 -- 172.21.15.28:0/432021215 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) ==== 74+0+22160 (secure 0 0 0) 0x7f65100c0470 con 0x7f6524102890 2024-05-12T22:06:43.231 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-12T22:06:43.231 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":86,"fsid":"6b92e7b4-10aa-11ef-bc98-c7b262605968","created":"2024-05-12T21:57:37.276782+0000","modified":"2024-05-12T22:06:34.945595+0000","last_up_change":"2024-05-12T22:05:37.929562+0000","last_in_change":"2024-05-12T22:05:18.750768+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-12T22:01:32.292826+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-12T22:06:31.540263+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-12T22:06:32.906118+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6992ce0a-844b-40e3-8ed1-181c8474dccf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":906003882},{"type":"v1","addr":"172.21.15.28:6803","nonce":906003882}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":906003882},{"type":"v1","addr":"172.21.15.28:6805","nonce":906003882}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":906003882},{"type":"v1","addr":"172.21.15.28:6809","nonce":906003882}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":906003882},{"type":"v1","addr":"172.21.15.28:6807","nonce":906003882}]},"public_addr":"172.21.15.28:6803/906003882","cluster_addr":"172.21.15.28:6805/906003882","heartbeat_back_addr":"172.21.15.28:6809/906003882","heartbeat_front_addr":"172.21.15.28:6807/906003882","state":["exists","up"]},{"osd":1,"uuid":"ced20c96-c481-4d48-90d1-6b43228959ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":3354141819},{"type":"v1","addr":"172.21.15.86:6801","nonce":3354141819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":3354141819},{"type":"v1","addr":"172.21.15.86:6803","nonce":3354141819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":3354141819},{"type":"v1","addr":"172.21.15.86:6807","nonce":3354141819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":3354141819},{"type":"v1","addr":"172.21.15.86:6805","nonce":3354141819}]},"public_addr":"172.21.15.86:6801/3354141819","cluster_addr":"172.21.15.86:6803/3354141819","heartbeat_back_addr":"172.21.15.86:6807/3354141819","heartbeat_front_addr":"172.21.15.86:6805/3354141819","state":["exists","up"]},{"osd":2,"uuid":"54a2f221-fa65-4795-9f5e-e660cc7c0098","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":906459365},{"type":"v1","addr":"172.21.15.186:6801","nonce":906459365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":906459365},{"type":"v1","addr":"172.21.15.186:6803","nonce":906459365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":906459365},{"type":"v1","addr":"172.21.15.186:6807","nonce":906459365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":906459365},{"type":"v1","addr":"172.21.15.186:6805","nonce":906459365}]},"public_addr":"172.21.15.186:6801/906459365","cluster_addr":"172.21.15.186:6803/906459365","heartbeat_back_addr":"172.21.15.186:6807/906459365","heartbeat_front_addr":"172.21.15.186:6805/906459365","state":["exists","up"]},{"osd":3,"uuid":"86836737-ac9a-487b-93f2-7fcd3c406960","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":4182884436},{"type":"v1","addr":"172.21.15.28:6811","nonce":4182884436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":4182884436},{"type":"v1","addr":"172.21.15.28:6813","nonce":4182884436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":4182884436},{"type":"v1","addr":"172.21.15.28:6817","nonce":4182884436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":4182884436},{"type":"v1","addr":"172.21.15.28:6815","nonce":4182884436}]},"public_addr":"172.21.15.28:6811/4182884436","cluster_addr":"172.21.15.28:6813/4182884436","heartbeat_back_addr":"172.21.15.28:6817/4182884436","heartbeat_front_addr":"172.21.15.28:6815/4182884436","state":["exists","up"]},{"osd":4,"uuid":"d162ebcd-83c0-4fd5-9925-28b2cecea935","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2596998708},{"type":"v1","addr":"172.21.15.86:6809","nonce":2596998708}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2596998708},{"type":"v1","addr":"172.21.15.86:6811","nonce":2596998708}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2596998708},{"type":"v1","addr":"172.21.15.86:6815","nonce":2596998708}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2596998708},{"type":"v1","addr":"172.21.15.86:6813","nonce":2596998708}]},"public_addr":"172.21.15.86:6809/2596998708","cluster_addr":"172.21.15.86:6811/2596998708","heartbeat_back_addr":"172.21.15.86:6815/2596998708","heartbeat_front_addr":"172.21.15.86:6813/2596998708","state":["exists","up"]},{"osd":5,"uuid":"c794336a-a105-425c-b6c7-a32b90c05bb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":1944205756},{"type":"v1","addr":"172.21.15.186:6809","nonce":1944205756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":1944205756},{"type":"v1","addr":"172.21.15.186:6811","nonce":1944205756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":1944205756},{"type":"v1","addr":"172.21.15.186:6815","nonce":1944205756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":1944205756},{"type":"v1","addr":"172.21.15.186:6813","nonce":1944205756}]},"public_addr":"172.21.15.186:6809/1944205756","cluster_addr":"172.21.15.186:6811/1944205756","heartbeat_back_addr":"172.21.15.186:6815/1944205756","heartbeat_front_addr":"172.21.15.186:6813/1944205756","state":["exists","up"]},{"osd":6,"uuid":"516932ac-af00-45b7-ad3c-d6dec7db442f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":1642308925},{"type":"v1","addr":"172.21.15.28:6819","nonce":1642308925}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":1642308925},{"type":"v1","addr":"172.21.15.28:6821","nonce":1642308925}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1642308925},{"type":"v1","addr":"172.21.15.28:6825","nonce":1642308925}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":1642308925},{"type":"v1","addr":"172.21.15.28:6823","nonce":1642308925}]},"public_addr":"172.21.15.28:6819/1642308925","cluster_addr":"172.21.15.28:6821/1642308925","heartbeat_back_addr":"172.21.15.28:6825/1642308925","heartbeat_front_addr":"172.21.15.28:6823/1642308925","state":["exists","up"]},{"osd":7,"uuid":"ca7a059f-0414-438e-b2d1-a10f348036be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":726062711},{"type":"v1","addr":"172.21.15.86:6817","nonce":726062711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":726062711},{"type":"v1","addr":"172.21.15.86:6819","nonce":726062711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":726062711},{"type":"v1","addr":"172.21.15.86:6823","nonce":726062711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":726062711},{"type":"v1","addr":"172.21.15.86:6821","nonce":726062711}]},"public_addr":"172.21.15.86:6817/726062711","cluster_addr":"172.21.15.86:6819/726062711","heartbeat_back_addr":"172.21.15.86:6823/726062711","heartbeat_front_addr":"172.21.15.86:6821/726062711","state":["exists","up"]},{"osd":8,"uuid":"92e3f41f-aebd-422e-a99e-555f7ba4aa53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":3007687277},{"type":"v1","addr":"172.21.15.186:6817","nonce":3007687277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":3007687277},{"type":"v1","addr":"172.21.15.186:6819","nonce":3007687277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":3007687277},{"type":"v1","addr":"172.21.15.186:6823","nonce":3007687277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":3007687277},{"type":"v1","addr":"172.21.15.186:6821","nonce":3007687277}]},"public_addr":"172.21.15.186:6817/3007687277","cluster_addr":"172.21.15.186:6819/3007687277","heartbeat_back_addr":"172.21.15.186:6823/3007687277","heartbeat_front_addr":"172.21.15.186:6821/3007687277","state":["exists","up"]},{"osd":9,"uuid":"38ff7d9d-ee85-404c-8c59-adc327e1ccfa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":136220472},{"type":"v1","addr":"172.21.15.28:6827","nonce":136220472}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":136220472},{"type":"v1","addr":"172.21.15.28:6829","nonce":136220472}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":136220472},{"type":"v1","addr":"172.21.15.28:6833","nonce":136220472}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":136220472},{"type":"v1","addr":"172.21.15.28:6831","nonce":136220472}]},"public_addr":"172.21.15.28:6827/136220472","cluster_addr":"172.21.15.28:6829/136220472","heartbeat_back_addr":"172.21.15.28:6833/136220472","heartbeat_front_addr":"172.21.15.28:6831/136220472","state":["exists","up"]},{"osd":10,"uuid":"5e9ff657-afa8-4bab-8491-f209038392b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1189968310},{"type":"v1","addr":"172.21.15.86:6825","nonce":1189968310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":1189968310},{"type":"v1","addr":"172.21.15.86:6827","nonce":1189968310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":1189968310},{"type":"v1","addr":"172.21.15.86:6831","nonce":1189968310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":1189968310},{"type":"v1","addr":"172.21.15.86:6829","nonce":1189968310}]},"public_addr":"172.21.15.86:6825/1189968310","cluster_addr":"172.21.15.86:6827/1189968310","heartbeat_back_addr":"172.21.15.86:6831/1189968310","heartbeat_front_addr":"172.21.15.86:6829/1189968310","state":["exists","up"]},{"osd":11,"uuid":"f112e3c6-65b6-4253-a966-62001db7d069","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3801668110},{"type":"v1","addr":"172.21.15.186:6825","nonce":3801668110}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3801668110},{"type":"v1","addr":"172.21.15.186:6827","nonce":3801668110}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3801668110},{"type":"v1","addr":"172.21.15.186:6831","nonce":3801668110}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3801668110},{"type":"v1","addr":"172.21.15.186:6829","nonce":3801668110}]},"public_addr":"172.21.15.186:6825/3801668110","cluster_addr":"172.21.15.186:6827/3801668110","heartbeat_back_addr":"172.21.15.186:6831/3801668110","heartbeat_front_addr":"172.21.15.186:6829/3801668110","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:00:40.483623+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:01:05.284385+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:01:28.645716+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:02:01.295092+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:02:26.486601+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:02:50.807478+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:03:24.756169+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:03:49.975173+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:04:13.772233+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:04:46.758555+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:05:12.331685+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:05:36.462548+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/1177009533":"2024-05-13T21:58:45.550459+0000","172.21.15.28:6800/3760499540":"2024-05-13T21:58:45.550459+0000","172.21.15.28:6801/3783742728":"2024-05-13T21:58:10.421564+0000","172.21.15.28:6800/3783742728":"2024-05-13T21:58:10.421564+0000","172.21.15.28:0/2229156260":"2024-05-13T21:58:45.550459+0000","172.21.15.28:0/1202140963":"2024-05-13T21:58:10.421564+0000","172.21.15.28:0/2924223784":"2024-05-13T21:58:45.550459+0000","172.21.15.28:6801/3760499540":"2024-05-13T21:58:45.550459+0000","172.21.15.28:0/1240027206":"2024-05-13T21:58:10.421564+0000","172.21.15.28:0/171417728":"2024-05-13T21:58:10.421564+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-12T22:06:43.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:43.233+0000 7f652b000640 1 -- 172.21.15.28:0/432021215 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f650c0a2c20 msgr2=0x7f650c0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:43.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:43.233+0000 7f652b000640 1 --2- 172.21.15.28:0/432021215 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f650c0a2c20 0x7f650c0a50e0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f6518005fd0 tx=0x7f6518009500 comp rx=0 tx=0).stop 2024-05-12T22:06:43.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:43.233+0000 7f652b000640 1 -- 172.21.15.28:0/432021215 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6524102890 msgr2=0x7f652419e4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:43.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:43.233+0000 7f652b000640 1 --2- 172.21.15.28:0/432021215 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6524102890 0x7f652419e4c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f6510031440 tx=0x7f6510032ce0 comp rx=0 tx=0).stop 2024-05-12T22:06:43.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:43.233+0000 7f652b000640 1 -- 172.21.15.28:0/432021215 shutdown_connections 2024-05-12T22:06:43.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:43.233+0000 7f652b000640 1 --2- 172.21.15.28:0/432021215 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f650c0a2c20 0x7f650c0a50e0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:43.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:43.233+0000 7f652b000640 1 --2- 172.21.15.28:0/432021215 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6524105240 0x7f65241a5530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:43.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:43.233+0000 7f652b000640 1 --2- 172.21.15.28:0/432021215 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6524102890 0x7f652419e4c0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:43.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:43.233+0000 7f652b000640 1 --2- 172.21.15.28:0/432021215 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f65240fff20 0x7f652419df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:43.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:43.233+0000 7f652b000640 1 -- 172.21.15.28:0/432021215 >> 172.21.15.28:0/432021215 conn(0x7f65240fbb00 msgr2=0x7f6524100c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:06:43.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:43.233+0000 7f652b000640 1 -- 172.21.15.28:0/432021215 shutdown_connections 2024-05-12T22:06:43.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:43.234+0000 7f652b000640 1 -- 172.21.15.28:0/432021215 wait complete. 2024-05-12T22:06:43.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:42 smithi086 ceph-mon[34297]: pgmap v314: 97 pgs: 97 active+clean; 580 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4.2 KiB/s wr, 16 op/s 2024-05-12T22:06:43.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:42 smithi086 ceph-mon[34297]: from='client.24689 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-12T22:06:43.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:42 smithi086 ceph-mon[34297]: from='client.24695 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-12T22:06:43.253 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-05-12T22:06:43.254 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-12T22:06:43.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:43.468+0000 7ff739c00640 1 -- 172.21.15.28:0/1890562033 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff734104a60 msgr2=0x7ff734106e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:43.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:43.468+0000 7ff739c00640 1 --2- 172.21.15.28:0/1890562033 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff734104a60 0x7ff734106e50 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7ff71c009900 tx=0x7ff71c02a3f0 comp rx=0 tx=0).stop 2024-05-12T22:06:43.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:43.468+0000 7ff739c00640 1 -- 172.21.15.28:0/1890562033 shutdown_connections 2024-05-12T22:06:43.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:43.468+0000 7ff739c00640 1 --2- 172.21.15.28:0/1890562033 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff734107390 0x7ff7341097e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:43.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:43.468+0000 7ff739c00640 1 --2- 172.21.15.28:0/1890562033 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff734104a60 0x7ff734106e50 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:43.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:43.468+0000 7ff739c00640 1 --2- 172.21.15.28:0/1890562033 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff734102130 0x7ff734104520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:43.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:43.468+0000 7ff739c00640 1 -- 172.21.15.28:0/1890562033 >> 172.21.15.28:0/1890562033 conn(0x7ff7340fbb00 msgr2=0x7ff7340fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:06:43.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:43.468+0000 7ff739c00640 1 -- 172.21.15.28:0/1890562033 shutdown_connections 2024-05-12T22:06:43.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:43.468+0000 7ff739c00640 1 -- 172.21.15.28:0/1890562033 wait complete. 2024-05-12T22:06:43.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:43.469+0000 7ff739c00640 1 Processor -- start 2024-05-12T22:06:43.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:43.470+0000 7ff739c00640 1 -- start start 2024-05-12T22:06:43.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:43.470+0000 7ff739c00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff734102130 0x7ff734199c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:43.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:43.470+0000 7ff739c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff734104a60 0x7ff73419a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:43.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:43.470+0000 7ff739c00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff734107390 0x7ff7341a1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:43.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:43.470+0000 7ff739c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff73410d390 con 0x7ff734104a60 2024-05-12T22:06:43.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:43.470+0000 7ff739c00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7ff73410d210 con 0x7ff734102130 2024-05-12T22:06:43.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:43.470+0000 7ff739c00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff73410d510 con 0x7ff734107390 2024-05-12T22:06:43.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:43.470+0000 7ff733e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff734102130 0x7ff734199c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:43.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:43.470+0000 7ff72b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff734104a60 0x7ff73419a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:43.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:43.470+0000 7ff733e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff734102130 0x7ff734199c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:58372/0 (socket says 172.21.15.28:58372) 2024-05-12T22:06:43.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:43.470+0000 7ff733e00640 1 -- 172.21.15.28:0/3786021189 learned_addr learned my addr 172.21.15.28:0/3786021189 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:06:43.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:43.471+0000 7ff739200640 1 --2- 172.21.15.28:0/3786021189 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff734107390 0x7ff7341a1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:43.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:43.471+0000 7ff733e00640 1 -- 172.21.15.28:0/3786021189 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff734107390 msgr2=0x7ff7341a1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:43.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:43.471+0000 7ff733e00640 1 --2- 172.21.15.28:0/3786021189 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff734107390 0x7ff7341a1190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:43.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:43.471+0000 7ff733e00640 1 -- 172.21.15.28:0/3786021189 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff734104a60 msgr2=0x7ff73419a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:43.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:43.471+0000 7ff733e00640 1 --2- 172.21.15.28:0/3786021189 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff734104a60 0x7ff73419a160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:43.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:43.471+0000 7ff733e00640 1 -- 172.21.15.28:0/3786021189 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7341a1830 con 0x7ff734102130 2024-05-12T22:06:43.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:43.471+0000 7ff733e00640 1 --2- 172.21.15.28:0/3786021189 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff734102130 0x7ff734199c20 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7ff72400e9f0 tx=0x7ff72400edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:06:43.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:43.471+0000 7ff731600640 1 -- 172.21.15.28:0/3786021189 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff72400cd80 con 0x7ff734102130 2024-05-12T22:06:43.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:43.471+0000 7ff731600640 1 -- 172.21.15.28:0/3786021189 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff724005500 con 0x7ff734102130 2024-05-12T22:06:43.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:43.471+0000 7ff731600640 1 -- 172.21.15.28:0/3786021189 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff724010690 con 0x7ff734102130 2024-05-12T22:06:43.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:43.472+0000 7ff739c00640 1 -- 172.21.15.28:0/3786021189 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7341a1b20 con 0x7ff734102130 2024-05-12T22:06:43.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:43.472+0000 7ff739c00640 1 -- 172.21.15.28:0/3786021189 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7341a2070 con 0x7ff734102130 2024-05-12T22:06:43.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:43.473+0000 7ff739c00640 1 -- 172.21.15.28:0/3786021189 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff734108ee0 con 0x7ff734102130 2024-05-12T22:06:43.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:43.475+0000 7ff731600640 1 -- 172.21.15.28:0/3786021189 <== mon.2 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7ff7240046c0 con 0x7ff734102130 2024-05-12T22:06:43.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:43.476+0000 7ff731600640 1 --2- 172.21.15.28:0/3786021189 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7ff7180a2ca0 0x7ff7180a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:43.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:43.477+0000 7ff731600640 1 -- 172.21.15.28:0/3786021189 <== mon.2 v2:172.21.15.86:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7ff7240cf530 con 0x7ff734102130 2024-05-12T22:06:43.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:43.477+0000 7ff72b400640 1 --2- 172.21.15.28:0/3786021189 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7ff7180a2ca0 0x7ff7180a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:43.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:43.477+0000 7ff72b400640 1 --2- 172.21.15.28:0/3786021189 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7ff7180a2ca0 0x7ff7180a5160 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7ff71c02a9e0 tx=0x7ff71c0315b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:06:43.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:43.481+0000 7ff731600640 1 -- 172.21.15.28:0/3786021189 <== mon.2 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7ff72409c3e0 con 0x7ff734102130 2024-05-12T22:06:43.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:43.685+0000 7ff739c00640 1 -- 172.21.15.28:0/3786021189 --> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff734061c70 con 0x7ff7180a2ca0 2024-05-12T22:06:43.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:43.713+0000 7ff731600640 1 -- 172.21.15.28:0/3786021189 <== mgr.14152 v2:172.21.15.28:6800/3123328651 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258727 (secure 0 0 0) 0x7ff734061c70 con 0x7ff7180a2ca0 2024-05-12T22:06:43.718 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-12T22:06:43.720 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-05-12T22:06:43.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:43.720+0000 7ff739c00640 1 -- 172.21.15.28:0/3786021189 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7ff7180a2ca0 msgr2=0x7ff7180a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:43.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:43.720+0000 7ff739c00640 1 --2- 172.21.15.28:0/3786021189 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7ff7180a2ca0 0x7ff7180a5160 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7ff71c02a9e0 tx=0x7ff71c0315b0 comp rx=0 tx=0).stop 2024-05-12T22:06:43.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:43.720+0000 7ff739c00640 1 -- 172.21.15.28:0/3786021189 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff734102130 msgr2=0x7ff734199c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:43.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:43.720+0000 7ff739c00640 1 --2- 172.21.15.28:0/3786021189 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff734102130 0x7ff734199c20 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7ff72400e9f0 tx=0x7ff72400edf0 comp rx=0 tx=0).stop 2024-05-12T22:06:43.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:43.720+0000 7ff739c00640 1 -- 172.21.15.28:0/3786021189 shutdown_connections 2024-05-12T22:06:43.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:43.720+0000 7ff739c00640 1 --2- 172.21.15.28:0/3786021189 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7ff7180a2ca0 0x7ff7180a5160 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:43.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:43.720+0000 7ff739c00640 1 --2- 172.21.15.28:0/3786021189 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff734107390 0x7ff7341a1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:43.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:43.720+0000 7ff739c00640 1 --2- 172.21.15.28:0/3786021189 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff734104a60 0x7ff73419a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:43.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:43.720+0000 7ff739c00640 1 --2- 172.21.15.28:0/3786021189 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff734102130 0x7ff734199c20 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:43.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:43.720+0000 7ff739c00640 1 -- 172.21.15.28:0/3786021189 >> 172.21.15.28:0/3786021189 conn(0x7ff7340fbb00 msgr2=0x7ff734101870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:06:43.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:43.720+0000 7ff739c00640 1 -- 172.21.15.28:0/3786021189 shutdown_connections 2024-05-12T22:06:43.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:43.720+0000 7ff739c00640 1 -- 172.21.15.28:0/3786021189 wait complete. 2024-05-12T22:06:43.741 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":314,"stamp":"2024-05-12T22:06:41.690122+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594296,"num_objects":54,"num_object_clones":0,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":156,"num_read_kb":137,"num_write":290,"num_write_kb":2972,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":36,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":228,"ondisk_log_size":228,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":345276,"kb_used_data":12732,"kb_used_omap":18,"kb_used_meta":332397,"kb_avail":1124596548,"statfs":{"total":1151940427776,"available":1151586865152,"internally_reserved":0,"allocated":13037568,"data_stored":7082698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":340374841},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3928,"num_objects":52,"num_object_clones":0,"num_object_copies":156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":20,"num_read_kb":18,"num_write":109,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":36,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003394"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:35.520965+0000","last_change":"2024-05-12T22:06:33.982079+0000","last_active":"2024-05-12T22:06:35.520965+0000","last_peered":"2024-05-12T22:06:35.520965+0000","last_clean":"2024-05-12T22:06:35.520965+0000","last_became_active":"2024-05-12T22:06:33.981703+0000","last_became_peered":"2024-05-12T22:06:33.981703+0000","last_unstale":"2024-05-12T22:06:35.520965+0000","last_undegraded":"2024-05-12T22:06:35.520965+0000","last_fullsized":"2024-05-12T22:06:35.520965+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T08:52:47.157067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:35.340488+0000","last_change":"2024-05-12T22:06:33.984194+0000","last_active":"2024-05-12T22:06:35.340488+0000","last_peered":"2024-05-12T22:06:35.340488+0000","last_clean":"2024-05-12T22:06:35.340488+0000","last_became_active":"2024-05-12T22:06:33.979787+0000","last_became_peered":"2024-05-12T22:06:33.979787+0000","last_unstale":"2024-05-12T22:06:35.340488+0000","last_undegraded":"2024-05-12T22:06:35.340488+0000","last_fullsized":"2024-05-12T22:06:35.340488+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T07:36:46.588641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:35.340564+0000","last_change":"2024-05-12T22:06:33.982428+0000","last_active":"2024-05-12T22:06:35.340564+0000","last_peered":"2024-05-12T22:06:35.340564+0000","last_clean":"2024-05-12T22:06:35.340564+0000","last_became_active":"2024-05-12T22:06:33.979563+0000","last_became_peered":"2024-05-12T22:06:33.979563+0000","last_unstale":"2024-05-12T22:06:35.340564+0000","last_undegraded":"2024-05-12T22:06:35.340564+0000","last_fullsized":"2024-05-12T22:06:35.340564+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T06:26:12.586519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:35.339993+0000","last_change":"2024-05-12T22:06:33.964836+0000","last_active":"2024-05-12T22:06:35.339993+0000","last_peered":"2024-05-12T22:06:35.339993+0000","last_clean":"2024-05-12T22:06:35.339993+0000","last_became_active":"2024-05-12T22:06:33.964474+0000","last_became_peered":"2024-05-12T22:06:33.964474+0000","last_unstale":"2024-05-12T22:06:35.339993+0000","last_undegraded":"2024-05-12T22:06:35.339993+0000","last_fullsized":"2024-05-12T22:06:35.339993+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T03:41:49.802372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:35.339223+0000","last_change":"2024-05-12T22:06:33.967492+0000","last_active":"2024-05-12T22:06:35.339223+0000","last_peered":"2024-05-12T22:06:35.339223+0000","last_clean":"2024-05-12T22:06:35.339223+0000","last_became_active":"2024-05-12T22:06:33.967146+0000","last_became_peered":"2024-05-12T22:06:33.967146+0000","last_unstale":"2024-05-12T22:06:35.339223+0000","last_undegraded":"2024-05-12T22:06:35.339223+0000","last_fullsized":"2024-05-12T22:06:35.339223+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T07:31:05.946473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:35.340570+0000","last_change":"2024-05-12T22:06:33.973653+0000","last_active":"2024-05-12T22:06:35.340570+0000","last_peered":"2024-05-12T22:06:35.340570+0000","last_clean":"2024-05-12T22:06:35.340570+0000","last_became_active":"2024-05-12T22:06:33.973290+0000","last_became_peered":"2024-05-12T22:06:33.973290+0000","last_unstale":"2024-05-12T22:06:35.340570+0000","last_undegraded":"2024-05-12T22:06:35.340570+0000","last_fullsized":"2024-05-12T22:06:35.340570+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T03:07:21.933141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:34.954863+0000","last_change":"2024-05-12T22:06:33.973807+0000","last_active":"2024-05-12T22:06:34.954863+0000","last_peered":"2024-05-12T22:06:34.954863+0000","last_clean":"2024-05-12T22:06:34.954863+0000","last_became_active":"2024-05-12T22:06:33.973466+0000","last_became_peered":"2024-05-12T22:06:33.973466+0000","last_unstale":"2024-05-12T22:06:34.954863+0000","last_undegraded":"2024-05-12T22:06:34.954863+0000","last_fullsized":"2024-05-12T22:06:34.954863+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T02:31:13.608309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:34.954929+0000","last_change":"2024-05-12T22:06:33.967356+0000","last_active":"2024-05-12T22:06:34.954929+0000","last_peered":"2024-05-12T22:06:34.954929+0000","last_clean":"2024-05-12T22:06:34.954929+0000","last_became_active":"2024-05-12T22:06:33.966054+0000","last_became_peered":"2024-05-12T22:06:33.966054+0000","last_unstale":"2024-05-12T22:06:34.954929+0000","last_undegraded":"2024-05-12T22:06:34.954929+0000","last_fullsized":"2024-05-12T22:06:34.954929+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T23:54:08.285720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:35.340237+0000","last_change":"2024-05-12T22:06:33.968782+0000","last_active":"2024-05-12T22:06:35.340237+0000","last_peered":"2024-05-12T22:06:35.340237+0000","last_clean":"2024-05-12T22:06:35.340237+0000","last_became_active":"2024-05-12T22:06:33.968435+0000","last_became_peered":"2024-05-12T22:06:33.968435+0000","last_unstale":"2024-05-12T22:06:35.340237+0000","last_undegraded":"2024-05-12T22:06:35.340237+0000","last_fullsized":"2024-05-12T22:06:35.340237+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T07:01:21.056777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:35.340693+0000","last_change":"2024-05-12T22:06:33.962018+0000","last_active":"2024-05-12T22:06:35.340693+0000","last_peered":"2024-05-12T22:06:35.340693+0000","last_clean":"2024-05-12T22:06:35.340693+0000","last_became_active":"2024-05-12T22:06:33.961583+0000","last_became_peered":"2024-05-12T22:06:33.961583+0000","last_unstale":"2024-05-12T22:06:35.340693+0000","last_undegraded":"2024-05-12T22:06:35.340693+0000","last_fullsized":"2024-05-12T22:06:35.340693+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T00:16:40.483194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:34.955430+0000","last_change":"2024-05-12T22:06:33.965776+0000","last_active":"2024-05-12T22:06:34.955430+0000","last_peered":"2024-05-12T22:06:34.955430+0000","last_clean":"2024-05-12T22:06:34.955430+0000","last_became_active":"2024-05-12T22:06:33.965439+0000","last_became_peered":"2024-05-12T22:06:33.965439+0000","last_unstale":"2024-05-12T22:06:34.955430+0000","last_undegraded":"2024-05-12T22:06:34.955430+0000","last_fullsized":"2024-05-12T22:06:34.955430+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T23:59:56.925209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:35.339787+0000","last_change":"2024-05-12T22:06:33.967906+0000","last_active":"2024-05-12T22:06:35.339787+0000","last_peered":"2024-05-12T22:06:35.339787+0000","last_clean":"2024-05-12T22:06:35.339787+0000","last_became_active":"2024-05-12T22:06:33.967516+0000","last_became_peered":"2024-05-12T22:06:33.967516+0000","last_unstale":"2024-05-12T22:06:35.339787+0000","last_undegraded":"2024-05-12T22:06:35.339787+0000","last_fullsized":"2024-05-12T22:06:35.339787+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T08:16:56.892024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:35.343740+0000","last_change":"2024-05-12T22:06:33.971456+0000","last_active":"2024-05-12T22:06:35.343740+0000","last_peered":"2024-05-12T22:06:35.343740+0000","last_clean":"2024-05-12T22:06:35.343740+0000","last_became_active":"2024-05-12T22:06:33.971110+0000","last_became_peered":"2024-05-12T22:06:33.971110+0000","last_unstale":"2024-05-12T22:06:35.343740+0000","last_undegraded":"2024-05-12T22:06:35.343740+0000","last_fullsized":"2024-05-12T22:06:35.343740+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T09:48:56.275346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:35.340215+0000","last_change":"2024-05-12T22:06:33.983421+0000","last_active":"2024-05-12T22:06:35.340215+0000","last_peered":"2024-05-12T22:06:35.340215+0000","last_clean":"2024-05-12T22:06:35.340215+0000","last_became_active":"2024-05-12T22:06:33.978515+0000","last_became_peered":"2024-05-12T22:06:33.978515+0000","last_unstale":"2024-05-12T22:06:35.340215+0000","last_undegraded":"2024-05-12T22:06:35.340215+0000","last_fullsized":"2024-05-12T22:06:35.340215+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T23:09:48.264541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:35.340839+0000","last_change":"2024-05-12T22:06:33.980956+0000","last_active":"2024-05-12T22:06:35.340839+0000","last_peered":"2024-05-12T22:06:35.340839+0000","last_clean":"2024-05-12T22:06:35.340839+0000","last_became_active":"2024-05-12T22:06:33.979860+0000","last_became_peered":"2024-05-12T22:06:33.979860+0000","last_unstale":"2024-05-12T22:06:35.340839+0000","last_undegraded":"2024-05-12T22:06:35.340839+0000","last_fullsized":"2024-05-12T22:06:35.340839+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T02:58:29.255704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:35.339172+0000","last_change":"2024-05-12T22:06:33.978682+0000","last_active":"2024-05-12T22:06:35.339172+0000","last_peered":"2024-05-12T22:06:35.339172+0000","last_clean":"2024-05-12T22:06:35.339172+0000","last_became_active":"2024-05-12T22:06:33.978234+0000","last_became_peered":"2024-05-12T22:06:33.978234+0000","last_unstale":"2024-05-12T22:06:35.339172+0000","last_undegraded":"2024-05-12T22:06:35.339172+0000","last_fullsized":"2024-05-12T22:06:35.339172+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T07:40:48.540839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:35.521065+0000","last_change":"2024-05-12T22:06:33.986598+0000","last_active":"2024-05-12T22:06:35.521065+0000","last_peered":"2024-05-12T22:06:35.521065+0000","last_clean":"2024-05-12T22:06:35.521065+0000","last_became_active":"2024-05-12T22:06:33.986346+0000","last_became_peered":"2024-05-12T22:06:33.986346+0000","last_unstale":"2024-05-12T22:06:35.521065+0000","last_undegraded":"2024-05-12T22:06:35.521065+0000","last_fullsized":"2024-05-12T22:06:35.521065+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T05:52:05.473099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:35.344836+0000","last_change":"2024-05-12T22:06:33.978900+0000","last_active":"2024-05-12T22:06:35.344836+0000","last_peered":"2024-05-12T22:06:35.344836+0000","last_clean":"2024-05-12T22:06:35.344836+0000","last_became_active":"2024-05-12T22:06:33.978474+0000","last_became_peered":"2024-05-12T22:06:33.978474+0000","last_unstale":"2024-05-12T22:06:35.344836+0000","last_undegraded":"2024-05-12T22:06:35.344836+0000","last_fullsized":"2024-05-12T22:06:35.344836+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T03:47:53.014263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:35.339462+0000","last_change":"2024-05-12T22:06:33.966752+0000","last_active":"2024-05-12T22:06:35.339462+0000","last_peered":"2024-05-12T22:06:35.339462+0000","last_clean":"2024-05-12T22:06:35.339462+0000","last_became_active":"2024-05-12T22:06:33.966320+0000","last_became_peered":"2024-05-12T22:06:33.966320+0000","last_unstale":"2024-05-12T22:06:35.339462+0000","last_undegraded":"2024-05-12T22:06:35.339462+0000","last_fullsized":"2024-05-12T22:06:35.339462+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T22:11:48.356156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'9","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:38.332078+0000","last_change":"2024-05-12T22:06:32.931346+0000","last_active":"2024-05-12T22:06:38.332078+0000","last_peered":"2024-05-12T22:06:38.332078+0000","last_clean":"2024-05-12T22:06:38.332078+0000","last_became_active":"2024-05-12T22:06:32.930909+0000","last_became_peered":"2024-05-12T22:06:32.930909+0000","last_unstale":"2024-05-12T22:06:38.332078+0000","last_undegraded":"2024-05-12T22:06:38.332078+0000","last_fullsized":"2024-05-12T22:06:38.332078+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T01:16:34.444226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:34.954955+0000","last_change":"2024-05-12T22:06:33.963875+0000","last_active":"2024-05-12T22:06:34.954955+0000","last_peered":"2024-05-12T22:06:34.954955+0000","last_clean":"2024-05-12T22:06:34.954955+0000","last_became_active":"2024-05-12T22:06:33.963239+0000","last_became_peered":"2024-05-12T22:06:33.963239+0000","last_unstale":"2024-05-12T22:06:34.954955+0000","last_undegraded":"2024-05-12T22:06:34.954955+0000","last_fullsized":"2024-05-12T22:06:34.954955+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T23:15:01.832082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"85'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:34.955158+0000","last_change":"2024-05-12T22:06:32.950352+0000","last_active":"2024-05-12T22:06:34.955158+0000","last_peered":"2024-05-12T22:06:34.955158+0000","last_clean":"2024-05-12T22:06:34.955158+0000","last_became_active":"2024-05-12T22:06:32.949759+0000","last_became_peered":"2024-05-12T22:06:32.949759+0000","last_unstale":"2024-05-12T22:06:34.955158+0000","last_undegraded":"2024-05-12T22:06:34.955158+0000","last_fullsized":"2024-05-12T22:06:34.955158+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T06:28:07.613351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:35.341217+0000","last_change":"2024-05-12T22:06:33.990783+0000","last_active":"2024-05-12T22:06:35.341217+0000","last_peered":"2024-05-12T22:06:35.341217+0000","last_clean":"2024-05-12T22:06:35.341217+0000","last_became_active":"2024-05-12T22:06:33.990387+0000","last_became_peered":"2024-05-12T22:06:33.990387+0000","last_unstale":"2024-05-12T22:06:35.341217+0000","last_undegraded":"2024-05-12T22:06:35.341217+0000","last_fullsized":"2024-05-12T22:06:35.341217+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T05:52:52.247458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"86'4","reported_seq":19,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:38.324914+0000","last_change":"2024-05-12T22:06:32.931718+0000","last_active":"2024-05-12T22:06:38.324914+0000","last_peered":"2024-05-12T22:06:38.324914+0000","last_clean":"2024-05-12T22:06:38.324914+0000","last_became_active":"2024-05-12T22:06:32.931404+0000","last_became_peered":"2024-05-12T22:06:32.931404+0000","last_unstale":"2024-05-12T22:06:38.324914+0000","last_undegraded":"2024-05-12T22:06:38.324914+0000","last_fullsized":"2024-05-12T22:06:38.324914+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T02:55:57.770504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:35.339769+0000","last_change":"2024-05-12T22:06:33.963299+0000","last_active":"2024-05-12T22:06:35.339769+0000","last_peered":"2024-05-12T22:06:35.339769+0000","last_clean":"2024-05-12T22:06:35.339769+0000","last_became_active":"2024-05-12T22:06:33.962916+0000","last_became_peered":"2024-05-12T22:06:33.962916+0000","last_unstale":"2024-05-12T22:06:35.339769+0000","last_undegraded":"2024-05-12T22:06:35.339769+0000","last_fullsized":"2024-05-12T22:06:35.339769+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T05:46:32.131918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:35.339875+0000","last_change":"2024-05-12T22:06:32.932319+0000","last_active":"2024-05-12T22:06:35.339875+0000","last_peered":"2024-05-12T22:06:35.339875+0000","last_clean":"2024-05-12T22:06:35.339875+0000","last_became_active":"2024-05-12T22:06:32.931936+0000","last_became_peered":"2024-05-12T22:06:32.931936+0000","last_unstale":"2024-05-12T22:06:35.339875+0000","last_undegraded":"2024-05-12T22:06:35.339875+0000","last_fullsized":"2024-05-12T22:06:35.339875+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T23:09:24.075793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:34.955558+0000","last_change":"2024-05-12T22:06:33.967926+0000","last_active":"2024-05-12T22:06:34.955558+0000","last_peered":"2024-05-12T22:06:34.955558+0000","last_clean":"2024-05-12T22:06:34.955558+0000","last_became_active":"2024-05-12T22:06:33.967588+0000","last_became_peered":"2024-05-12T22:06:33.967588+0000","last_unstale":"2024-05-12T22:06:34.955558+0000","last_undegraded":"2024-05-12T22:06:34.955558+0000","last_fullsized":"2024-05-12T22:06:34.955558+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T22:39:54.434117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"85'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:34.955112+0000","last_change":"2024-05-12T22:06:32.930347+0000","last_active":"2024-05-12T22:06:34.955112+0000","last_peered":"2024-05-12T22:06:34.955112+0000","last_clean":"2024-05-12T22:06:34.955112+0000","last_became_active":"2024-05-12T22:06:32.930083+0000","last_became_peered":"2024-05-12T22:06:32.930083+0000","last_unstale":"2024-05-12T22:06:34.955112+0000","last_undegraded":"2024-05-12T22:06:34.955112+0000","last_fullsized":"2024-05-12T22:06:34.955112+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T07:10:08.898490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:35.339000+0000","last_change":"2024-05-12T22:06:33.973793+0000","last_active":"2024-05-12T22:06:35.339000+0000","last_peered":"2024-05-12T22:06:35.339000+0000","last_clean":"2024-05-12T22:06:35.339000+0000","last_became_active":"2024-05-12T22:06:33.973380+0000","last_became_peered":"2024-05-12T22:06:33.973380+0000","last_unstale":"2024-05-12T22:06:35.339000+0000","last_undegraded":"2024-05-12T22:06:35.339000+0000","last_fullsized":"2024-05-12T22:06:35.339000+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T04:32:01.817780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:38.327455+0000","last_change":"2024-05-12T22:06:32.938215+0000","last_active":"2024-05-12T22:06:38.327455+0000","last_peered":"2024-05-12T22:06:38.327455+0000","last_clean":"2024-05-12T22:06:38.327455+0000","last_became_active":"2024-05-12T22:06:32.937939+0000","last_became_peered":"2024-05-12T22:06:32.937939+0000","last_unstale":"2024-05-12T22:06:38.327455+0000","last_undegraded":"2024-05-12T22:06:38.327455+0000","last_fullsized":"2024-05-12T22:06:38.327455+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T04:15:28.821835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:34.955403+0000","last_change":"2024-05-12T22:06:33.977362+0000","last_active":"2024-05-12T22:06:34.955403+0000","last_peered":"2024-05-12T22:06:34.955403+0000","last_clean":"2024-05-12T22:06:34.955403+0000","last_became_active":"2024-05-12T22:06:33.977074+0000","last_became_peered":"2024-05-12T22:06:33.977074+0000","last_unstale":"2024-05-12T22:06:34.955403+0000","last_undegraded":"2024-05-12T22:06:34.955403+0000","last_fullsized":"2024-05-12T22:06:34.955403+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T22:21:00.062780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:35.342436+0000","last_change":"2024-05-12T22:06:32.935508+0000","last_active":"2024-05-12T22:06:35.342436+0000","last_peered":"2024-05-12T22:06:35.342436+0000","last_clean":"2024-05-12T22:06:35.342436+0000","last_became_active":"2024-05-12T22:06:32.934760+0000","last_became_peered":"2024-05-12T22:06:32.934760+0000","last_unstale":"2024-05-12T22:06:35.342436+0000","last_undegraded":"2024-05-12T22:06:35.342436+0000","last_fullsized":"2024-05-12T22:06:35.342436+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T01:58:09.450187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:34.954484+0000","last_change":"2024-05-12T22:06:33.966421+0000","last_active":"2024-05-12T22:06:34.954484+0000","last_peered":"2024-05-12T22:06:34.954484+0000","last_clean":"2024-05-12T22:06:34.954484+0000","last_became_active":"2024-05-12T22:06:33.965627+0000","last_became_peered":"2024-05-12T22:06:33.965627+0000","last_unstale":"2024-05-12T22:06:34.954484+0000","last_undegraded":"2024-05-12T22:06:34.954484+0000","last_fullsized":"2024-05-12T22:06:34.954484+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T09:25:03.836082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:34.954724+0000","last_change":"2024-05-12T22:06:32.928907+0000","last_active":"2024-05-12T22:06:34.954724+0000","last_peered":"2024-05-12T22:06:34.954724+0000","last_clean":"2024-05-12T22:06:34.954724+0000","last_became_active":"2024-05-12T22:06:32.928586+0000","last_became_peered":"2024-05-12T22:06:32.928586+0000","last_unstale":"2024-05-12T22:06:34.954724+0000","last_undegraded":"2024-05-12T22:06:34.954724+0000","last_fullsized":"2024-05-12T22:06:34.954724+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T04:24:21.658957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:35.344161+0000","last_change":"2024-05-12T22:06:33.982873+0000","last_active":"2024-05-12T22:06:35.344161+0000","last_peered":"2024-05-12T22:06:35.344161+0000","last_clean":"2024-05-12T22:06:35.344161+0000","last_became_active":"2024-05-12T22:06:33.982495+0000","last_became_peered":"2024-05-12T22:06:33.982495+0000","last_unstale":"2024-05-12T22:06:35.344161+0000","last_undegraded":"2024-05-12T22:06:35.344161+0000","last_fullsized":"2024-05-12T22:06:35.344161+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T05:45:39.404282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"86'3","reported_seq":19,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:37.318487+0000","last_change":"2024-05-12T22:06:32.937716+0000","last_active":"2024-05-12T22:06:37.318487+0000","last_peered":"2024-05-12T22:06:37.318487+0000","last_clean":"2024-05-12T22:06:37.318487+0000","last_became_active":"2024-05-12T22:06:32.937418+0000","last_became_peered":"2024-05-12T22:06:32.937418+0000","last_unstale":"2024-05-12T22:06:37.318487+0000","last_undegraded":"2024-05-12T22:06:37.318487+0000","last_fullsized":"2024-05-12T22:06:37.318487+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T08:20:13.639958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:35.340221+0000","last_change":"2024-05-12T22:06:33.974702+0000","last_active":"2024-05-12T22:06:35.340221+0000","last_peered":"2024-05-12T22:06:35.340221+0000","last_clean":"2024-05-12T22:06:35.340221+0000","last_became_active":"2024-05-12T22:06:33.974363+0000","last_became_peered":"2024-05-12T22:06:33.974363+0000","last_unstale":"2024-05-12T22:06:35.340221+0000","last_undegraded":"2024-05-12T22:06:35.340221+0000","last_fullsized":"2024-05-12T22:06:35.340221+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T09:16:33.620945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"86'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:37.316282+0000","last_change":"2024-05-12T22:06:32.932583+0000","last_active":"2024-05-12T22:06:37.316282+0000","last_peered":"2024-05-12T22:06:37.316282+0000","last_clean":"2024-05-12T22:06:37.316282+0000","last_became_active":"2024-05-12T22:06:32.932251+0000","last_became_peered":"2024-05-12T22:06:32.932251+0000","last_unstale":"2024-05-12T22:06:37.316282+0000","last_undegraded":"2024-05-12T22:06:37.316282+0000","last_fullsized":"2024-05-12T22:06:37.316282+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T04:59:28.152763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:35.341062+0000","last_change":"2024-05-12T22:06:33.980496+0000","last_active":"2024-05-12T22:06:35.341062+0000","last_peered":"2024-05-12T22:06:35.341062+0000","last_clean":"2024-05-12T22:06:35.341062+0000","last_became_active":"2024-05-12T22:06:33.980155+0000","last_became_peered":"2024-05-12T22:06:33.980155+0000","last_unstale":"2024-05-12T22:06:35.341062+0000","last_undegraded":"2024-05-12T22:06:35.341062+0000","last_fullsized":"2024-05-12T22:06:35.341062+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T01:54:03.831828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"85'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:34.955546+0000","last_change":"2024-05-12T22:06:32.951633+0000","last_active":"2024-05-12T22:06:34.955546+0000","last_peered":"2024-05-12T22:06:34.955546+0000","last_clean":"2024-05-12T22:06:34.955546+0000","last_became_active":"2024-05-12T22:06:32.951382+0000","last_became_peered":"2024-05-12T22:06:32.951382+0000","last_unstale":"2024-05-12T22:06:34.955546+0000","last_undegraded":"2024-05-12T22:06:34.955546+0000","last_fullsized":"2024-05-12T22:06:34.955546+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T02:06:21.408593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:35.340267+0000","last_change":"2024-05-12T22:06:33.963493+0000","last_active":"2024-05-12T22:06:35.340267+0000","last_peered":"2024-05-12T22:06:35.340267+0000","last_clean":"2024-05-12T22:06:35.340267+0000","last_became_active":"2024-05-12T22:06:33.963127+0000","last_became_peered":"2024-05-12T22:06:33.963127+0000","last_unstale":"2024-05-12T22:06:35.340267+0000","last_undegraded":"2024-05-12T22:06:35.340267+0000","last_fullsized":"2024-05-12T22:06:35.340267+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T05:39:31.038446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"86'8","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:38.703111+0000","last_change":"2024-05-12T22:06:32.932026+0000","last_active":"2024-05-12T22:06:38.703111+0000","last_peered":"2024-05-12T22:06:38.703111+0000","last_clean":"2024-05-12T22:06:38.703111+0000","last_became_active":"2024-05-12T22:06:32.931667+0000","last_became_peered":"2024-05-12T22:06:32.931667+0000","last_unstale":"2024-05-12T22:06:38.703111+0000","last_undegraded":"2024-05-12T22:06:38.703111+0000","last_fullsized":"2024-05-12T22:06:38.703111+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T06:16:28.259047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":9,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:35.340037+0000","last_change":"2024-05-12T22:06:33.962677+0000","last_active":"2024-05-12T22:06:35.340037+0000","last_peered":"2024-05-12T22:06:35.340037+0000","last_clean":"2024-05-12T22:06:35.340037+0000","last_became_active":"2024-05-12T22:06:33.962295+0000","last_became_peered":"2024-05-12T22:06:33.962295+0000","last_unstale":"2024-05-12T22:06:35.340037+0000","last_undegraded":"2024-05-12T22:06:35.340037+0000","last_fullsized":"2024-05-12T22:06:35.340037+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T01:31:04.587046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"86'6","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:38.331216+0000","last_change":"2024-05-12T22:06:32.936104+0000","last_active":"2024-05-12T22:06:38.331216+0000","last_peered":"2024-05-12T22:06:38.331216+0000","last_clean":"2024-05-12T22:06:38.331216+0000","last_became_active":"2024-05-12T22:06:32.935701+0000","last_became_peered":"2024-05-12T22:06:32.935701+0000","last_unstale":"2024-05-12T22:06:38.331216+0000","last_undegraded":"2024-05-12T22:06:38.331216+0000","last_fullsized":"2024-05-12T22:06:38.331216+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T07:04:10.115302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:35.342726+0000","last_change":"2024-05-12T22:06:33.977850+0000","last_active":"2024-05-12T22:06:35.342726+0000","last_peered":"2024-05-12T22:06:35.342726+0000","last_clean":"2024-05-12T22:06:35.342726+0000","last_became_active":"2024-05-12T22:06:33.977511+0000","last_became_peered":"2024-05-12T22:06:33.977511+0000","last_unstale":"2024-05-12T22:06:35.342726+0000","last_undegraded":"2024-05-12T22:06:35.342726+0000","last_fullsized":"2024-05-12T22:06:35.342726+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T23:10:17.267823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:34.955012+0000","last_change":"2024-05-12T22:06:32.929955+0000","last_active":"2024-05-12T22:06:34.955012+0000","last_peered":"2024-05-12T22:06:34.955012+0000","last_clean":"2024-05-12T22:06:34.955012+0000","last_became_active":"2024-05-12T22:06:32.929623+0000","last_became_peered":"2024-05-12T22:06:32.929623+0000","last_unstale":"2024-05-12T22:06:34.955012+0000","last_undegraded":"2024-05-12T22:06:34.955012+0000","last_fullsized":"2024-05-12T22:06:34.955012+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T08:15:22.622908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:34.955279+0000","last_change":"2024-05-12T22:06:33.969771+0000","last_active":"2024-05-12T22:06:34.955279+0000","last_peered":"2024-05-12T22:06:34.955279+0000","last_clean":"2024-05-12T22:06:34.955279+0000","last_became_active":"2024-05-12T22:06:33.969296+0000","last_became_peered":"2024-05-12T22:06:33.969296+0000","last_unstale":"2024-05-12T22:06:34.955279+0000","last_undegraded":"2024-05-12T22:06:34.955279+0000","last_fullsized":"2024-05-12T22:06:34.955279+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T08:51:46.398555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"86'3","reported_seq":21,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:38.704381+0000","last_change":"2024-05-12T22:06:32.927707+0000","last_active":"2024-05-12T22:06:38.704381+0000","last_peered":"2024-05-12T22:06:38.704381+0000","last_clean":"2024-05-12T22:06:38.704381+0000","last_became_active":"2024-05-12T22:06:32.927406+0000","last_became_peered":"2024-05-12T22:06:32.927406+0000","last_unstale":"2024-05-12T22:06:38.704381+0000","last_undegraded":"2024-05-12T22:06:38.704381+0000","last_fullsized":"2024-05-12T22:06:38.704381+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T05:14:34.268589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:34.954844+0000","last_change":"2024-05-12T22:06:33.964996+0000","last_active":"2024-05-12T22:06:34.954844+0000","last_peered":"2024-05-12T22:06:34.954844+0000","last_clean":"2024-05-12T22:06:34.954844+0000","last_became_active":"2024-05-12T22:06:33.964587+0000","last_became_peered":"2024-05-12T22:06:33.964587+0000","last_unstale":"2024-05-12T22:06:34.954844+0000","last_undegraded":"2024-05-12T22:06:34.954844+0000","last_fullsized":"2024-05-12T22:06:34.954844+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T22:27:28.316377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"86'15","reported_seq":31,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:38.330792+0000","last_change":"2024-05-12T22:06:32.935543+0000","last_active":"2024-05-12T22:06:38.330792+0000","last_peered":"2024-05-12T22:06:38.330792+0000","last_clean":"2024-05-12T22:06:38.330792+0000","last_became_active":"2024-05-12T22:06:32.935145+0000","last_became_peered":"2024-05-12T22:06:32.935145+0000","last_unstale":"2024-05-12T22:06:38.330792+0000","last_undegraded":"2024-05-12T22:06:38.330792+0000","last_fullsized":"2024-05-12T22:06:38.330792+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T01:13:23.289057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:34.955030+0000","last_change":"2024-05-12T22:06:33.964367+0000","last_active":"2024-05-12T22:06:34.955030+0000","last_peered":"2024-05-12T22:06:34.955030+0000","last_clean":"2024-05-12T22:06:34.955030+0000","last_became_active":"2024-05-12T22:06:33.963876+0000","last_became_peered":"2024-05-12T22:06:33.963876+0000","last_unstale":"2024-05-12T22:06:34.955030+0000","last_undegraded":"2024-05-12T22:06:34.955030+0000","last_fullsized":"2024-05-12T22:06:34.955030+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T07:09:20.962965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"86'2","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:38.699116+0000","last_change":"2024-05-12T22:06:32.927270+0000","last_active":"2024-05-12T22:06:38.699116+0000","last_peered":"2024-05-12T22:06:38.699116+0000","last_clean":"2024-05-12T22:06:38.699116+0000","last_became_active":"2024-05-12T22:06:32.926930+0000","last_became_peered":"2024-05-12T22:06:32.926930+0000","last_unstale":"2024-05-12T22:06:38.699116+0000","last_undegraded":"2024-05-12T22:06:38.699116+0000","last_fullsized":"2024-05-12T22:06:38.699116+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T07:04:48.336599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:35.520861+0000","last_change":"2024-05-12T22:06:33.966350+0000","last_active":"2024-05-12T22:06:35.520861+0000","last_peered":"2024-05-12T22:06:35.520861+0000","last_clean":"2024-05-12T22:06:35.520861+0000","last_became_active":"2024-05-12T22:06:33.965974+0000","last_became_peered":"2024-05-12T22:06:33.965974+0000","last_unstale":"2024-05-12T22:06:35.520861+0000","last_undegraded":"2024-05-12T22:06:35.520861+0000","last_fullsized":"2024-05-12T22:06:35.520861+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T02:08:16.889684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:35.343874+0000","last_change":"2024-05-12T22:06:32.944945+0000","last_active":"2024-05-12T22:06:35.343874+0000","last_peered":"2024-05-12T22:06:35.343874+0000","last_clean":"2024-05-12T22:06:35.343874+0000","last_became_active":"2024-05-12T22:06:32.944576+0000","last_became_peered":"2024-05-12T22:06:32.944576+0000","last_unstale":"2024-05-12T22:06:35.343874+0000","last_undegraded":"2024-05-12T22:06:35.343874+0000","last_fullsized":"2024-05-12T22:06:35.343874+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T07:29:14.544821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:35.340754+0000","last_change":"2024-05-12T22:06:33.966299+0000","last_active":"2024-05-12T22:06:35.340754+0000","last_peered":"2024-05-12T22:06:35.340754+0000","last_clean":"2024-05-12T22:06:35.340754+0000","last_became_active":"2024-05-12T22:06:33.965902+0000","last_became_peered":"2024-05-12T22:06:33.965902+0000","last_unstale":"2024-05-12T22:06:35.340754+0000","last_undegraded":"2024-05-12T22:06:35.340754+0000","last_fullsized":"2024-05-12T22:06:35.340754+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T22:37:51.476546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:34.954892+0000","last_change":"2024-05-12T22:06:32.937210+0000","last_active":"2024-05-12T22:06:34.954892+0000","last_peered":"2024-05-12T22:06:34.954892+0000","last_clean":"2024-05-12T22:06:34.954892+0000","last_became_active":"2024-05-12T22:06:32.936805+0000","last_became_peered":"2024-05-12T22:06:32.936805+0000","last_unstale":"2024-05-12T22:06:34.954892+0000","last_undegraded":"2024-05-12T22:06:34.954892+0000","last_fullsized":"2024-05-12T22:06:34.954892+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T06:14:20.825258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:35.340359+0000","last_change":"2024-05-12T22:06:33.983604+0000","last_active":"2024-05-12T22:06:35.340359+0000","last_peered":"2024-05-12T22:06:35.340359+0000","last_clean":"2024-05-12T22:06:35.340359+0000","last_became_active":"2024-05-12T22:06:33.983168+0000","last_became_peered":"2024-05-12T22:06:33.983168+0000","last_unstale":"2024-05-12T22:06:35.340359+0000","last_undegraded":"2024-05-12T22:06:35.340359+0000","last_fullsized":"2024-05-12T22:06:35.340359+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T05:54:55.854593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:34.955536+0000","last_change":"2024-05-12T22:06:33.968848+0000","last_active":"2024-05-12T22:06:34.955536+0000","last_peered":"2024-05-12T22:06:34.955536+0000","last_clean":"2024-05-12T22:06:34.955536+0000","last_became_active":"2024-05-12T22:06:33.968529+0000","last_became_peered":"2024-05-12T22:06:33.968529+0000","last_unstale":"2024-05-12T22:06:34.955536+0000","last_undegraded":"2024-05-12T22:06:34.955536+0000","last_fullsized":"2024-05-12T22:06:34.955536+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T03:00:41.129662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"86'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:35.342238+0000","last_change":"2024-05-12T22:06:32.946670+0000","last_active":"2024-05-12T22:06:35.342238+0000","last_peered":"2024-05-12T22:06:35.342238+0000","last_clean":"2024-05-12T22:06:35.342238+0000","last_became_active":"2024-05-12T22:06:32.946059+0000","last_became_peered":"2024-05-12T22:06:32.946059+0000","last_unstale":"2024-05-12T22:06:35.342238+0000","last_undegraded":"2024-05-12T22:06:35.342238+0000","last_fullsized":"2024-05-12T22:06:35.342238+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T22:35:56.116979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:34.954731+0000","last_change":"2024-05-12T22:06:33.962177+0000","last_active":"2024-05-12T22:06:34.954731+0000","last_peered":"2024-05-12T22:06:34.954731+0000","last_clean":"2024-05-12T22:06:34.954731+0000","last_became_active":"2024-05-12T22:06:33.960919+0000","last_became_peered":"2024-05-12T22:06:33.960919+0000","last_unstale":"2024-05-12T22:06:34.954731+0000","last_undegraded":"2024-05-12T22:06:34.954731+0000","last_fullsized":"2024-05-12T22:06:34.954731+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T00:39:54.566460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'3","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:38.700719+0000","last_change":"2024-05-12T22:06:32.927885+0000","last_active":"2024-05-12T22:06:38.700719+0000","last_peered":"2024-05-12T22:06:38.700719+0000","last_clean":"2024-05-12T22:06:38.700719+0000","last_became_active":"2024-05-12T22:06:32.927567+0000","last_became_peered":"2024-05-12T22:06:32.927567+0000","last_unstale":"2024-05-12T22:06:38.700719+0000","last_undegraded":"2024-05-12T22:06:38.700719+0000","last_fullsized":"2024-05-12T22:06:38.700719+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T07:13:50.811992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":348,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:34.954726+0000","last_change":"2024-05-12T22:05:17.683363+0000","last_active":"2024-05-12T22:06:34.954726+0000","last_peered":"2024-05-12T22:06:34.954726+0000","last_clean":"2024-05-12T22:06:34.954726+0000","last_became_active":"2024-05-12T22:05:17.682824+0000","last_became_peered":"2024-05-12T22:05:17.682824+0000","last_unstale":"2024-05-12T22:06:34.954726+0000","last_undegraded":"2024-05-12T22:06:34.954726+0000","last_fullsized":"2024-05-12T22:06:34.954726+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:01:32.304714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:01:32.304714+0000","last_clean_scrub_stamp":"2024-05-12T22:01:32.304714+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T06:35:22.474820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:35.342343+0000","last_change":"2024-05-12T22:06:33.965332+0000","last_active":"2024-05-12T22:06:35.342343+0000","last_peered":"2024-05-12T22:06:35.342343+0000","last_clean":"2024-05-12T22:06:35.342343+0000","last_became_active":"2024-05-12T22:06:33.964858+0000","last_became_peered":"2024-05-12T22:06:33.964858+0000","last_unstale":"2024-05-12T22:06:35.342343+0000","last_undegraded":"2024-05-12T22:06:35.342343+0000","last_fullsized":"2024-05-12T22:06:35.342343+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T05:14:09.825614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:34.954991+0000","last_change":"2024-05-12T22:06:32.929773+0000","last_active":"2024-05-12T22:06:34.954991+0000","last_peered":"2024-05-12T22:06:34.954991+0000","last_clean":"2024-05-12T22:06:34.954991+0000","last_became_active":"2024-05-12T22:06:32.929458+0000","last_became_peered":"2024-05-12T22:06:32.929458+0000","last_unstale":"2024-05-12T22:06:34.954991+0000","last_undegraded":"2024-05-12T22:06:34.954991+0000","last_fullsized":"2024-05-12T22:06:34.954991+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T01:30:42.853183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"86'1","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:37.316957+0000","last_change":"2024-05-12T22:06:32.947014+0000","last_active":"2024-05-12T22:06:37.316957+0000","last_peered":"2024-05-12T22:06:37.316957+0000","last_clean":"2024-05-12T22:06:37.316957+0000","last_became_active":"2024-05-12T22:06:32.946661+0000","last_became_peered":"2024-05-12T22:06:32.946661+0000","last_unstale":"2024-05-12T22:06:37.316957+0000","last_undegraded":"2024-05-12T22:06:37.316957+0000","last_fullsized":"2024-05-12T22:06:37.316957+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T23:09:19.378353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:35.342207+0000","last_change":"2024-05-12T22:06:33.980490+0000","last_active":"2024-05-12T22:06:35.342207+0000","last_peered":"2024-05-12T22:06:35.342207+0000","last_clean":"2024-05-12T22:06:35.342207+0000","last_became_active":"2024-05-12T22:06:33.980133+0000","last_became_peered":"2024-05-12T22:06:33.980133+0000","last_unstale":"2024-05-12T22:06:35.342207+0000","last_undegraded":"2024-05-12T22:06:35.342207+0000","last_fullsized":"2024-05-12T22:06:35.342207+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T00:28:10.461428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:38.329705+0000","last_change":"2024-05-12T22:06:32.943152+0000","last_active":"2024-05-12T22:06:38.329705+0000","last_peered":"2024-05-12T22:06:38.329705+0000","last_clean":"2024-05-12T22:06:38.329705+0000","last_became_active":"2024-05-12T22:06:32.942769+0000","last_became_peered":"2024-05-12T22:06:32.942769+0000","last_unstale":"2024-05-12T22:06:38.329705+0000","last_undegraded":"2024-05-12T22:06:38.329705+0000","last_fullsized":"2024-05-12T22:06:38.329705+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T01:23:24.955739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:34.954952+0000","last_change":"2024-05-12T22:06:33.964081+0000","last_active":"2024-05-12T22:06:34.954952+0000","last_peered":"2024-05-12T22:06:34.954952+0000","last_clean":"2024-05-12T22:06:34.954952+0000","last_became_active":"2024-05-12T22:06:33.963054+0000","last_became_peered":"2024-05-12T22:06:33.963054+0000","last_unstale":"2024-05-12T22:06:34.954952+0000","last_undegraded":"2024-05-12T22:06:34.954952+0000","last_fullsized":"2024-05-12T22:06:34.954952+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T09:40:03.396200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"86'4","reported_seq":19,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:37.318135+0000","last_change":"2024-05-12T22:06:32.936432+0000","last_active":"2024-05-12T22:06:37.318135+0000","last_peered":"2024-05-12T22:06:37.318135+0000","last_clean":"2024-05-12T22:06:37.318135+0000","last_became_active":"2024-05-12T22:06:32.936094+0000","last_became_peered":"2024-05-12T22:06:32.936094+0000","last_unstale":"2024-05-12T22:06:37.318135+0000","last_undegraded":"2024-05-12T22:06:37.318135+0000","last_fullsized":"2024-05-12T22:06:37.318135+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T02:40:31.346118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:35.340679+0000","last_change":"2024-05-12T22:06:33.985953+0000","last_active":"2024-05-12T22:06:35.340679+0000","last_peered":"2024-05-12T22:06:35.340679+0000","last_clean":"2024-05-12T22:06:35.340679+0000","last_became_active":"2024-05-12T22:06:33.984864+0000","last_became_peered":"2024-05-12T22:06:33.984864+0000","last_unstale":"2024-05-12T22:06:35.340679+0000","last_undegraded":"2024-05-12T22:06:35.340679+0000","last_fullsized":"2024-05-12T22:06:35.340679+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T07:28:22.357841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:37.316020+0000","last_change":"2024-05-12T22:06:32.934510+0000","last_active":"2024-05-12T22:06:37.316020+0000","last_peered":"2024-05-12T22:06:37.316020+0000","last_clean":"2024-05-12T22:06:37.316020+0000","last_became_active":"2024-05-12T22:06:32.934222+0000","last_became_peered":"2024-05-12T22:06:32.934222+0000","last_unstale":"2024-05-12T22:06:37.316020+0000","last_undegraded":"2024-05-12T22:06:37.316020+0000","last_fullsized":"2024-05-12T22:06:37.316020+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T00:57:44.321629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:35.338903+0000","last_change":"2024-05-12T22:06:33.963870+0000","last_active":"2024-05-12T22:06:35.338903+0000","last_peered":"2024-05-12T22:06:35.338903+0000","last_clean":"2024-05-12T22:06:35.338903+0000","last_became_active":"2024-05-12T22:06:33.963533+0000","last_became_peered":"2024-05-12T22:06:33.963533+0000","last_unstale":"2024-05-12T22:06:35.338903+0000","last_undegraded":"2024-05-12T22:06:35.338903+0000","last_fullsized":"2024-05-12T22:06:35.338903+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T08:56:59.495469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:37.319040+0000","last_change":"2024-05-12T22:06:32.939738+0000","last_active":"2024-05-12T22:06:37.319040+0000","last_peered":"2024-05-12T22:06:37.319040+0000","last_clean":"2024-05-12T22:06:37.319040+0000","last_became_active":"2024-05-12T22:06:32.939365+0000","last_became_peered":"2024-05-12T22:06:32.939365+0000","last_unstale":"2024-05-12T22:06:37.319040+0000","last_undegraded":"2024-05-12T22:06:37.319040+0000","last_fullsized":"2024-05-12T22:06:37.319040+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T02:19:31.180834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:35.520724+0000","last_change":"2024-05-12T22:06:33.975064+0000","last_active":"2024-05-12T22:06:35.520724+0000","last_peered":"2024-05-12T22:06:35.520724+0000","last_clean":"2024-05-12T22:06:35.520724+0000","last_became_active":"2024-05-12T22:06:33.974613+0000","last_became_peered":"2024-05-12T22:06:33.974613+0000","last_unstale":"2024-05-12T22:06:35.520724+0000","last_undegraded":"2024-05-12T22:06:35.520724+0000","last_fullsized":"2024-05-12T22:06:35.520724+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T08:46:27.343777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"86'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:36.031010+0000","last_change":"2024-05-12T22:06:32.931148+0000","last_active":"2024-05-12T22:06:36.031010+0000","last_peered":"2024-05-12T22:06:36.031010+0000","last_clean":"2024-05-12T22:06:36.031010+0000","last_became_active":"2024-05-12T22:06:32.930717+0000","last_became_peered":"2024-05-12T22:06:32.930717+0000","last_unstale":"2024-05-12T22:06:36.031010+0000","last_undegraded":"2024-05-12T22:06:36.031010+0000","last_fullsized":"2024-05-12T22:06:36.031010+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T07:26:20.915467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:34.954710+0000","last_change":"2024-05-12T22:06:33.988390+0000","last_active":"2024-05-12T22:06:34.954710+0000","last_peered":"2024-05-12T22:06:34.954710+0000","last_clean":"2024-05-12T22:06:34.954710+0000","last_became_active":"2024-05-12T22:06:33.986970+0000","last_became_peered":"2024-05-12T22:06:33.986970+0000","last_unstale":"2024-05-12T22:06:34.954710+0000","last_undegraded":"2024-05-12T22:06:34.954710+0000","last_fullsized":"2024-05-12T22:06:34.954710+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T05:03:31.930261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:37.315887+0000","last_change":"2024-05-12T22:06:32.931161+0000","last_active":"2024-05-12T22:06:37.315887+0000","last_peered":"2024-05-12T22:06:37.315887+0000","last_clean":"2024-05-12T22:06:37.315887+0000","last_became_active":"2024-05-12T22:06:32.930801+0000","last_became_peered":"2024-05-12T22:06:32.930801+0000","last_unstale":"2024-05-12T22:06:37.315887+0000","last_undegraded":"2024-05-12T22:06:37.315887+0000","last_fullsized":"2024-05-12T22:06:37.315887+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T05:58:48.226595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:34.955052+0000","last_change":"2024-05-12T22:06:33.968256+0000","last_active":"2024-05-12T22:06:34.955052+0000","last_peered":"2024-05-12T22:06:34.955052+0000","last_clean":"2024-05-12T22:06:34.955052+0000","last_became_active":"2024-05-12T22:06:33.967885+0000","last_became_peered":"2024-05-12T22:06:33.967885+0000","last_unstale":"2024-05-12T22:06:34.955052+0000","last_undegraded":"2024-05-12T22:06:34.955052+0000","last_fullsized":"2024-05-12T22:06:34.955052+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T09:12:40.687367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"86'3","reported_seq":19,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:38.331104+0000","last_change":"2024-05-12T22:06:32.944736+0000","last_active":"2024-05-12T22:06:38.331104+0000","last_peered":"2024-05-12T22:06:38.331104+0000","last_clean":"2024-05-12T22:06:38.331104+0000","last_became_active":"2024-05-12T22:06:32.944269+0000","last_became_peered":"2024-05-12T22:06:32.944269+0000","last_unstale":"2024-05-12T22:06:38.331104+0000","last_undegraded":"2024-05-12T22:06:38.331104+0000","last_fullsized":"2024-05-12T22:06:38.331104+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T22:56:24.391267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:34.955430+0000","last_change":"2024-05-12T22:06:33.968360+0000","last_active":"2024-05-12T22:06:34.955430+0000","last_peered":"2024-05-12T22:06:34.955430+0000","last_clean":"2024-05-12T22:06:34.955430+0000","last_became_active":"2024-05-12T22:06:33.967967+0000","last_became_peered":"2024-05-12T22:06:33.967967+0000","last_unstale":"2024-05-12T22:06:34.955430+0000","last_undegraded":"2024-05-12T22:06:34.955430+0000","last_fullsized":"2024-05-12T22:06:34.955430+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T00:41:12.693858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"85'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:34.955175+0000","last_change":"2024-05-12T22:06:32.935993+0000","last_active":"2024-05-12T22:06:34.955175+0000","last_peered":"2024-05-12T22:06:34.955175+0000","last_clean":"2024-05-12T22:06:34.955175+0000","last_became_active":"2024-05-12T22:06:32.935615+0000","last_became_peered":"2024-05-12T22:06:32.935615+0000","last_unstale":"2024-05-12T22:06:34.955175+0000","last_undegraded":"2024-05-12T22:06:34.955175+0000","last_fullsized":"2024-05-12T22:06:34.955175+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T00:43:39.254637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:34.955213+0000","last_change":"2024-05-12T22:06:33.966110+0000","last_active":"2024-05-12T22:06:34.955213+0000","last_peered":"2024-05-12T22:06:34.955213+0000","last_clean":"2024-05-12T22:06:34.955213+0000","last_became_active":"2024-05-12T22:06:33.965755+0000","last_became_peered":"2024-05-12T22:06:33.965755+0000","last_unstale":"2024-05-12T22:06:34.955213+0000","last_undegraded":"2024-05-12T22:06:34.955213+0000","last_fullsized":"2024-05-12T22:06:34.955213+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T22:18:53.941335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:35.343164+0000","last_change":"2024-05-12T22:06:32.942229+0000","last_active":"2024-05-12T22:06:35.343164+0000","last_peered":"2024-05-12T22:06:35.343164+0000","last_clean":"2024-05-12T22:06:35.343164+0000","last_became_active":"2024-05-12T22:06:32.941826+0000","last_became_peered":"2024-05-12T22:06:32.941826+0000","last_unstale":"2024-05-12T22:06:35.343164+0000","last_undegraded":"2024-05-12T22:06:35.343164+0000","last_fullsized":"2024-05-12T22:06:35.343164+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T03:52:34.886318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:34.955455+0000","last_change":"2024-05-12T22:06:33.963081+0000","last_active":"2024-05-12T22:06:34.955455+0000","last_peered":"2024-05-12T22:06:34.955455+0000","last_clean":"2024-05-12T22:06:34.955455+0000","last_became_active":"2024-05-12T22:06:33.962696+0000","last_became_peered":"2024-05-12T22:06:33.962696+0000","last_unstale":"2024-05-12T22:06:34.955455+0000","last_undegraded":"2024-05-12T22:06:34.955455+0000","last_fullsized":"2024-05-12T22:06:34.955455+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T05:55:45.581823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:35.520910+0000","last_change":"2024-05-12T22:06:33.968865+0000","last_active":"2024-05-12T22:06:35.520910+0000","last_peered":"2024-05-12T22:06:35.520910+0000","last_clean":"2024-05-12T22:06:35.520910+0000","last_became_active":"2024-05-12T22:06:33.968185+0000","last_became_peered":"2024-05-12T22:06:33.968185+0000","last_unstale":"2024-05-12T22:06:35.520910+0000","last_undegraded":"2024-05-12T22:06:35.520910+0000","last_fullsized":"2024-05-12T22:06:35.520910+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T01:23:41.083778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:35.339719+0000","last_change":"2024-05-12T22:06:33.964786+0000","last_active":"2024-05-12T22:06:35.339719+0000","last_peered":"2024-05-12T22:06:35.339719+0000","last_clean":"2024-05-12T22:06:35.339719+0000","last_became_active":"2024-05-12T22:06:33.964227+0000","last_became_peered":"2024-05-12T22:06:33.964227+0000","last_unstale":"2024-05-12T22:06:35.339719+0000","last_undegraded":"2024-05-12T22:06:35.339719+0000","last_fullsized":"2024-05-12T22:06:35.339719+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T02:53:42.739455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:35.339370+0000","last_change":"2024-05-12T22:06:33.978857+0000","last_active":"2024-05-12T22:06:35.339370+0000","last_peered":"2024-05-12T22:06:35.339370+0000","last_clean":"2024-05-12T22:06:35.339370+0000","last_became_active":"2024-05-12T22:06:33.978531+0000","last_became_peered":"2024-05-12T22:06:33.978531+0000","last_unstale":"2024-05-12T22:06:35.339370+0000","last_undegraded":"2024-05-12T22:06:35.339370+0000","last_fullsized":"2024-05-12T22:06:35.339370+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T02:51:05.609267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:35.340667+0000","last_change":"2024-05-12T22:06:33.980662+0000","last_active":"2024-05-12T22:06:35.340667+0000","last_peered":"2024-05-12T22:06:35.340667+0000","last_clean":"2024-05-12T22:06:35.340667+0000","last_became_active":"2024-05-12T22:06:33.980332+0000","last_became_peered":"2024-05-12T22:06:33.980332+0000","last_unstale":"2024-05-12T22:06:35.340667+0000","last_undegraded":"2024-05-12T22:06:35.340667+0000","last_fullsized":"2024-05-12T22:06:35.340667+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T06:21:59.715792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:35.342553+0000","last_change":"2024-05-12T22:06:33.987338+0000","last_active":"2024-05-12T22:06:35.342553+0000","last_peered":"2024-05-12T22:06:35.342553+0000","last_clean":"2024-05-12T22:06:35.342553+0000","last_became_active":"2024-05-12T22:06:33.986942+0000","last_became_peered":"2024-05-12T22:06:33.986942+0000","last_unstale":"2024-05-12T22:06:35.342553+0000","last_undegraded":"2024-05-12T22:06:35.342553+0000","last_fullsized":"2024-05-12T22:06:35.342553+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T01:02:05.137558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:35.520956+0000","last_change":"2024-05-12T22:06:33.966967+0000","last_active":"2024-05-12T22:06:35.520956+0000","last_peered":"2024-05-12T22:06:35.520956+0000","last_clean":"2024-05-12T22:06:35.520956+0000","last_became_active":"2024-05-12T22:06:33.965375+0000","last_became_peered":"2024-05-12T22:06:33.965375+0000","last_unstale":"2024-05-12T22:06:35.520956+0000","last_undegraded":"2024-05-12T22:06:35.520956+0000","last_fullsized":"2024-05-12T22:06:35.520956+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T04:57:36.341705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:35.339754+0000","last_change":"2024-05-12T22:06:33.963902+0000","last_active":"2024-05-12T22:06:35.339754+0000","last_peered":"2024-05-12T22:06:35.339754+0000","last_clean":"2024-05-12T22:06:35.339754+0000","last_became_active":"2024-05-12T22:06:33.963436+0000","last_became_peered":"2024-05-12T22:06:33.963436+0000","last_unstale":"2024-05-12T22:06:35.339754+0000","last_undegraded":"2024-05-12T22:06:35.339754+0000","last_fullsized":"2024-05-12T22:06:35.339754+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T02:51:56.415167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:34.954919+0000","last_change":"2024-05-12T22:06:33.969491+0000","last_active":"2024-05-12T22:06:34.954919+0000","last_peered":"2024-05-12T22:06:34.954919+0000","last_clean":"2024-05-12T22:06:34.954919+0000","last_became_active":"2024-05-12T22:06:33.969161+0000","last_became_peered":"2024-05-12T22:06:33.969161+0000","last_unstale":"2024-05-12T22:06:34.954919+0000","last_undegraded":"2024-05-12T22:06:34.954919+0000","last_fullsized":"2024-05-12T22:06:34.954919+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T06:04:21.847520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:34.954990+0000","last_change":"2024-05-12T22:06:33.973490+0000","last_active":"2024-05-12T22:06:34.954990+0000","last_peered":"2024-05-12T22:06:34.954990+0000","last_clean":"2024-05-12T22:06:34.954990+0000","last_became_active":"2024-05-12T22:06:33.973161+0000","last_became_peered":"2024-05-12T22:06:33.973161+0000","last_unstale":"2024-05-12T22:06:34.954990+0000","last_undegraded":"2024-05-12T22:06:34.954990+0000","last_fullsized":"2024-05-12T22:06:34.954990+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T03:29:36.972631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:34.954452+0000","last_change":"2024-05-12T22:06:33.957320+0000","last_active":"2024-05-12T22:06:34.954452+0000","last_peered":"2024-05-12T22:06:34.954452+0000","last_clean":"2024-05-12T22:06:34.954452+0000","last_became_active":"2024-05-12T22:06:33.956837+0000","last_became_peered":"2024-05-12T22:06:33.956837+0000","last_unstale":"2024-05-12T22:06:34.954452+0000","last_undegraded":"2024-05-12T22:06:34.954452+0000","last_fullsized":"2024-05-12T22:06:34.954452+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T01:20:51.018175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:35.339899+0000","last_change":"2024-05-12T22:06:33.968363+0000","last_active":"2024-05-12T22:06:35.339899+0000","last_peered":"2024-05-12T22:06:35.339899+0000","last_clean":"2024-05-12T22:06:35.339899+0000","last_became_active":"2024-05-12T22:06:33.968031+0000","last_became_peered":"2024-05-12T22:06:33.968031+0000","last_unstale":"2024-05-12T22:06:35.339899+0000","last_undegraded":"2024-05-12T22:06:35.339899+0000","last_fullsized":"2024-05-12T22:06:35.339899+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T00:14:07.124058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:35.343242+0000","last_change":"2024-05-12T22:06:33.984823+0000","last_active":"2024-05-12T22:06:35.343242+0000","last_peered":"2024-05-12T22:06:35.343242+0000","last_clean":"2024-05-12T22:06:35.343242+0000","last_became_active":"2024-05-12T22:06:33.984430+0000","last_became_peered":"2024-05-12T22:06:33.984430+0000","last_unstale":"2024-05-12T22:06:35.343242+0000","last_undegraded":"2024-05-12T22:06:35.343242+0000","last_fullsized":"2024-05-12T22:06:35.343242+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T01:30:18.109265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:35.520677+0000","last_change":"2024-05-12T22:06:33.962660+0000","last_active":"2024-05-12T22:06:35.520677+0000","last_peered":"2024-05-12T22:06:35.520677+0000","last_clean":"2024-05-12T22:06:35.520677+0000","last_became_active":"2024-05-12T22:06:33.962284+0000","last_became_peered":"2024-05-12T22:06:33.962284+0000","last_unstale":"2024-05-12T22:06:35.520677+0000","last_undegraded":"2024-05-12T22:06:35.520677+0000","last_fullsized":"2024-05-12T22:06:35.520677+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T01:08:53.888992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":3928,"num_objects":52,"num_object_clones":0,"num_object_copies":156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":20,"num_read_kb":18,"num_write":109,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":36,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":217088,"data_stored":32494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91,"ondisk_log_size":91,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892350990,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27920,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717232,"statfs":{"total":95995035648,"available":95966445568,"internally_reserved":0,"allocated":925696,"data_stored":442154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612626,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28540,"kb_used_data":1524,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716612,"statfs":{"total":95995035648,"available":95965810688,"internally_reserved":0,"allocated":1560576,"data_stored":1035130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776152,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27936,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717216,"statfs":{"total":95995035648,"available":95966429184,"internally_reserved":0,"allocated":942080,"data_stored":442390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037791,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32096,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713056,"statfs":{"total":95995035648,"available":95962169344,"internally_reserved":0,"allocated":942080,"data_stored":442310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299426,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28512,"kb_used_data":1492,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716640,"statfs":{"total":95995035648,"available":95965839360,"internally_reserved":0,"allocated":1527808,"data_stored":1033144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462953,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27924,"kb_used_data":900,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717228,"statfs":{"total":95995035648,"available":95966441472,"internally_reserved":0,"allocated":921600,"data_stored":441572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574916,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32092,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713060,"statfs":{"total":95995035648,"available":95962173440,"internally_reserved":0,"allocated":933888,"data_stored":443184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738441,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27928,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717224,"statfs":{"total":95995035648,"available":95966437376,"internally_reserved":0,"allocated":933888,"data_stored":442266,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411392,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27952,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717200,"statfs":{"total":95995035648,"available":95966412800,"internally_reserved":0,"allocated":958464,"data_stored":444534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149753,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27928,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717224,"statfs":{"total":95995035648,"available":95966437376,"internally_reserved":0,"allocated":929792,"data_stored":441676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888115,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27928,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717224,"statfs":{"total":95995035648,"available":95966437376,"internally_reserved":0,"allocated":933888,"data_stored":442216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724590,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28520,"kb_used_data":1492,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716632,"statfs":{"total":95995035648,"available":95965831168,"internally_reserved":0,"allocated":1527808,"data_stored":1032122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":22636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-12T22:06:43.745 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-12T22:06:44.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:44.007+0000 7fd332200640 1 -- 172.21.15.28:0/726383895 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd32c104ca0 msgr2=0x7fd32c105120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:44.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:44.007+0000 7fd332200640 1 --2- 172.21.15.28:0/726383895 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd32c104ca0 0x7fd32c105120 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fd318009860 tx=0x7fd31802a340 comp rx=0 tx=0).stop 2024-05-12T22:06:44.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:44.007+0000 7fd332200640 1 -- 172.21.15.28:0/726383895 shutdown_connections 2024-05-12T22:06:44.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:44.007+0000 7fd332200640 1 --2- 172.21.15.28:0/726383895 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd32c105660 0x7fd32c10bef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:44.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:44.007+0000 7fd332200640 1 --2- 172.21.15.28:0/726383895 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd32c104ca0 0x7fd32c105120 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:44.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:44.007+0000 7fd332200640 1 --2- 172.21.15.28:0/726383895 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd32c103ab0 0x7fd32c103ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:44.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:44.007+0000 7fd332200640 1 -- 172.21.15.28:0/726383895 >> 172.21.15.28:0/726383895 conn(0x7fd32c0ff020 msgr2=0x7fd32c101460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:06:44.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:44.007+0000 7fd332200640 1 -- 172.21.15.28:0/726383895 shutdown_connections 2024-05-12T22:06:44.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:44.007+0000 7fd332200640 1 -- 172.21.15.28:0/726383895 wait complete. 2024-05-12T22:06:44.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:44.008+0000 7fd332200640 1 Processor -- start 2024-05-12T22:06:44.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:44.008+0000 7fd332200640 1 -- start start 2024-05-12T22:06:44.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:44.008+0000 7fd332200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd32c103ab0 0x7fd32c199db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:44.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:44.009+0000 7fd332200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd32c104ca0 0x7fd32c19a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:44.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:44.009+0000 7fd332200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd32c105660 0x7fd32c1a1360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:44.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:44.009+0000 7fd332200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd32c10f730 con 0x7fd32c105660 2024-05-12T22:06:44.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:44.009+0000 7fd332200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fd32c10f5b0 con 0x7fd32c104ca0 2024-05-12T22:06:44.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:44.009+0000 7fd332200640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd32c10f8b0 con 0x7fd32c103ab0 2024-05-12T22:06:44.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:44.009+0000 7fd32be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd32c105660 0x7fd32c1a1360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:44.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:44.009+0000 7fd32be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd32c105660 0x7fd32c1a1360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36496/0 (socket says 172.21.15.28:36496) 2024-05-12T22:06:44.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:44.009+0000 7fd32be00640 1 -- 172.21.15.28:0/202319658 learned_addr learned my addr 172.21.15.28:0/202319658 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:06:44.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:44.009+0000 7fd32b400640 1 --2- 172.21.15.28:0/202319658 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd32c103ab0 0x7fd32c199db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:44.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:44.009+0000 7fd32aa00640 1 --2- 172.21.15.28:0/202319658 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd32c104ca0 0x7fd32c19a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:44.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:44.009+0000 7fd32aa00640 1 -- 172.21.15.28:0/202319658 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd32c103ab0 msgr2=0x7fd32c199db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:44.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:44.009+0000 7fd32aa00640 1 --2- 172.21.15.28:0/202319658 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd32c103ab0 0x7fd32c199db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:44.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:44.009+0000 7fd32aa00640 1 -- 172.21.15.28:0/202319658 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd32c105660 msgr2=0x7fd32c1a1360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:44.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:44.009+0000 7fd32aa00640 1 --2- 172.21.15.28:0/202319658 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd32c105660 0x7fd32c1a1360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:44.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:44.009+0000 7fd32aa00640 1 -- 172.21.15.28:0/202319658 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd32c1a19d0 con 0x7fd32c104ca0 2024-05-12T22:06:44.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:44.009+0000 7fd32be00640 1 --2- 172.21.15.28:0/202319658 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd32c105660 0x7fd32c1a1360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:06:44.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:44.010+0000 7fd32aa00640 1 --2- 172.21.15.28:0/202319658 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd32c104ca0 0x7fd32c19a2f0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fd318035040 tx=0x7fd318033d90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:06:44.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:44.010+0000 7fd323e00640 1 -- 172.21.15.28:0/202319658 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd318037040 con 0x7fd32c104ca0 2024-05-12T22:06:44.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:44.010+0000 7fd323e00640 1 -- 172.21.15.28:0/202319658 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd318030850 con 0x7fd32c104ca0 2024-05-12T22:06:44.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:44.010+0000 7fd323e00640 1 -- 172.21.15.28:0/202319658 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3180046c0 con 0x7fd32c104ca0 2024-05-12T22:06:44.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:44.010+0000 7fd332200640 1 -- 172.21.15.28:0/202319658 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd32c1a1c60 con 0x7fd32c104ca0 2024-05-12T22:06:44.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:44.010+0000 7fd332200640 1 -- 172.21.15.28:0/202319658 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd32c1a20d0 con 0x7fd32c104ca0 2024-05-12T22:06:44.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:44.012+0000 7fd332200640 1 -- 172.21.15.28:0/202319658 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd32c10ae50 con 0x7fd32c104ca0 2024-05-12T22:06:44.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:44.013+0000 7fd323e00640 1 -- 172.21.15.28:0/202319658 <== mon.2 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fd318004050 con 0x7fd32c104ca0 2024-05-12T22:06:44.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:44.015+0000 7fd323e00640 1 --2- 172.21.15.28:0/202319658 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fd3100a2c20 0x7fd3100a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:44.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:44.015+0000 7fd323e00640 1 -- 172.21.15.28:0/202319658 <== mon.2 v2:172.21.15.86:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7fd3180eedc0 con 0x7fd32c104ca0 2024-05-12T22:06:44.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:44.015+0000 7fd32b400640 1 --2- 172.21.15.28:0/202319658 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fd3100a2c20 0x7fd3100a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:44.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:44.016+0000 7fd32b400640 1 --2- 172.21.15.28:0/202319658 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fd3100a2c20 0x7fd3100a50e0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fd32c104b00 tx=0x7fd31c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:06:44.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:44.019+0000 7fd323e00640 1 -- 172.21.15.28:0/202319658 <== mon.2 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fd3180bbd80 con 0x7fd32c104ca0 2024-05-12T22:06:44.034 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:43 smithi028 ceph-mon[29847]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-05-12T22:06:44.034 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:43 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/1837148121' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:06:44.034 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:43 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/432021215' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-12T22:06:44.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:43 smithi186 ceph-mon[34785]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-05-12T22:06:44.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:43 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/1837148121' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:06:44.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:43 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/432021215' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-12T22:06:44.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:44.219+0000 7fd332200640 1 -- 172.21.15.28:0/202319658 --> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fd32c061c70 con 0x7fd3100a2c20 2024-05-12T22:06:44.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:43 smithi086 ceph-mon[34297]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-05-12T22:06:44.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:43 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/1837148121' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:06:44.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:43 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/432021215' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-12T22:06:44.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:44.243+0000 7fd323e00640 1 -- 172.21.15.28:0/202319658 <== mgr.14152 v2:172.21.15.28:6800/3123328651 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258748 (secure 0 0 0) 0x7fd32c061c70 con 0x7fd3100a2c20 2024-05-12T22:06:44.246 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-12T22:06:44.248 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-05-12T22:06:44.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:44.249+0000 7fd332200640 1 -- 172.21.15.28:0/202319658 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fd3100a2c20 msgr2=0x7fd3100a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:44.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:44.249+0000 7fd332200640 1 --2- 172.21.15.28:0/202319658 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fd3100a2c20 0x7fd3100a50e0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fd32c104b00 tx=0x7fd31c009260 comp rx=0 tx=0).stop 2024-05-12T22:06:44.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:44.249+0000 7fd332200640 1 -- 172.21.15.28:0/202319658 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd32c104ca0 msgr2=0x7fd32c19a2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:44.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:44.249+0000 7fd332200640 1 --2- 172.21.15.28:0/202319658 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd32c104ca0 0x7fd32c19a2f0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fd318035040 tx=0x7fd318033d90 comp rx=0 tx=0).stop 2024-05-12T22:06:44.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:44.249+0000 7fd332200640 1 -- 172.21.15.28:0/202319658 shutdown_connections 2024-05-12T22:06:44.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:44.249+0000 7fd332200640 1 --2- 172.21.15.28:0/202319658 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fd3100a2c20 0x7fd3100a50e0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:44.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:44.249+0000 7fd332200640 1 --2- 172.21.15.28:0/202319658 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd32c105660 0x7fd32c1a1360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:44.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:44.249+0000 7fd332200640 1 --2- 172.21.15.28:0/202319658 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd32c104ca0 0x7fd32c19a2f0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:44.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:44.249+0000 7fd332200640 1 --2- 172.21.15.28:0/202319658 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd32c103ab0 0x7fd32c199db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:44.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:44.249+0000 7fd332200640 1 -- 172.21.15.28:0/202319658 >> 172.21.15.28:0/202319658 conn(0x7fd32c0ff020 msgr2=0x7fd32c108020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:06:44.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:44.250+0000 7fd332200640 1 -- 172.21.15.28:0/202319658 shutdown_connections 2024-05-12T22:06:44.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:44.250+0000 7fd332200640 1 -- 172.21.15.28:0/202319658 wait complete. 2024-05-12T22:06:44.273 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":315,"stamp":"2024-05-12T22:06:43.712751+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596098,"num_objects":63,"num_object_clones":0,"num_object_copies":189,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":0,"num_read":169,"num_read_kb":146,"num_write":308,"num_write_kb":2978,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":248,"ondisk_log_size":248,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":345340,"kb_used_data":12828,"kb_used_omap":18,"kb_used_meta":332397,"kb_avail":1124596484,"statfs":{"total":1151940427776,"available":1151586799616,"internally_reserved":0,"allocated":13135872,"data_stored":7121208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":340374841},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5730,"num_objects":61,"num_object_clones":0,"num_object_copies":183,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":127,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.770084"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:35.520965+0000","last_change":"2024-05-12T22:06:33.982079+0000","last_active":"2024-05-12T22:06:35.520965+0000","last_peered":"2024-05-12T22:06:35.520965+0000","last_clean":"2024-05-12T22:06:35.520965+0000","last_became_active":"2024-05-12T22:06:33.981703+0000","last_became_peered":"2024-05-12T22:06:33.981703+0000","last_unstale":"2024-05-12T22:06:35.520965+0000","last_undegraded":"2024-05-12T22:06:35.520965+0000","last_fullsized":"2024-05-12T22:06:35.520965+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T08:52:47.157067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:35.340488+0000","last_change":"2024-05-12T22:06:33.984194+0000","last_active":"2024-05-12T22:06:35.340488+0000","last_peered":"2024-05-12T22:06:35.340488+0000","last_clean":"2024-05-12T22:06:35.340488+0000","last_became_active":"2024-05-12T22:06:33.979787+0000","last_became_peered":"2024-05-12T22:06:33.979787+0000","last_unstale":"2024-05-12T22:06:35.340488+0000","last_undegraded":"2024-05-12T22:06:35.340488+0000","last_fullsized":"2024-05-12T22:06:35.340488+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T07:36:46.588641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:35.340564+0000","last_change":"2024-05-12T22:06:33.982428+0000","last_active":"2024-05-12T22:06:35.340564+0000","last_peered":"2024-05-12T22:06:35.340564+0000","last_clean":"2024-05-12T22:06:35.340564+0000","last_became_active":"2024-05-12T22:06:33.979563+0000","last_became_peered":"2024-05-12T22:06:33.979563+0000","last_unstale":"2024-05-12T22:06:35.340564+0000","last_undegraded":"2024-05-12T22:06:35.340564+0000","last_fullsized":"2024-05-12T22:06:35.340564+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T06:26:12.586519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:35.339993+0000","last_change":"2024-05-12T22:06:33.964836+0000","last_active":"2024-05-12T22:06:35.339993+0000","last_peered":"2024-05-12T22:06:35.339993+0000","last_clean":"2024-05-12T22:06:35.339993+0000","last_became_active":"2024-05-12T22:06:33.964474+0000","last_became_peered":"2024-05-12T22:06:33.964474+0000","last_unstale":"2024-05-12T22:06:35.339993+0000","last_undegraded":"2024-05-12T22:06:35.339993+0000","last_fullsized":"2024-05-12T22:06:35.339993+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T03:41:49.802372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:35.339223+0000","last_change":"2024-05-12T22:06:33.967492+0000","last_active":"2024-05-12T22:06:35.339223+0000","last_peered":"2024-05-12T22:06:35.339223+0000","last_clean":"2024-05-12T22:06:35.339223+0000","last_became_active":"2024-05-12T22:06:33.967146+0000","last_became_peered":"2024-05-12T22:06:33.967146+0000","last_unstale":"2024-05-12T22:06:35.339223+0000","last_undegraded":"2024-05-12T22:06:35.339223+0000","last_fullsized":"2024-05-12T22:06:35.339223+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T07:31:05.946473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:35.340570+0000","last_change":"2024-05-12T22:06:33.973653+0000","last_active":"2024-05-12T22:06:35.340570+0000","last_peered":"2024-05-12T22:06:35.340570+0000","last_clean":"2024-05-12T22:06:35.340570+0000","last_became_active":"2024-05-12T22:06:33.973290+0000","last_became_peered":"2024-05-12T22:06:33.973290+0000","last_unstale":"2024-05-12T22:06:35.340570+0000","last_undegraded":"2024-05-12T22:06:35.340570+0000","last_fullsized":"2024-05-12T22:06:35.340570+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T03:07:21.933141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:34.954863+0000","last_change":"2024-05-12T22:06:33.973807+0000","last_active":"2024-05-12T22:06:34.954863+0000","last_peered":"2024-05-12T22:06:34.954863+0000","last_clean":"2024-05-12T22:06:34.954863+0000","last_became_active":"2024-05-12T22:06:33.973466+0000","last_became_peered":"2024-05-12T22:06:33.973466+0000","last_unstale":"2024-05-12T22:06:34.954863+0000","last_undegraded":"2024-05-12T22:06:34.954863+0000","last_fullsized":"2024-05-12T22:06:34.954863+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T02:31:13.608309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:34.954929+0000","last_change":"2024-05-12T22:06:33.967356+0000","last_active":"2024-05-12T22:06:34.954929+0000","last_peered":"2024-05-12T22:06:34.954929+0000","last_clean":"2024-05-12T22:06:34.954929+0000","last_became_active":"2024-05-12T22:06:33.966054+0000","last_became_peered":"2024-05-12T22:06:33.966054+0000","last_unstale":"2024-05-12T22:06:34.954929+0000","last_undegraded":"2024-05-12T22:06:34.954929+0000","last_fullsized":"2024-05-12T22:06:34.954929+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T23:54:08.285720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:35.340237+0000","last_change":"2024-05-12T22:06:33.968782+0000","last_active":"2024-05-12T22:06:35.340237+0000","last_peered":"2024-05-12T22:06:35.340237+0000","last_clean":"2024-05-12T22:06:35.340237+0000","last_became_active":"2024-05-12T22:06:33.968435+0000","last_became_peered":"2024-05-12T22:06:33.968435+0000","last_unstale":"2024-05-12T22:06:35.340237+0000","last_undegraded":"2024-05-12T22:06:35.340237+0000","last_fullsized":"2024-05-12T22:06:35.340237+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T07:01:21.056777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:35.340693+0000","last_change":"2024-05-12T22:06:33.962018+0000","last_active":"2024-05-12T22:06:35.340693+0000","last_peered":"2024-05-12T22:06:35.340693+0000","last_clean":"2024-05-12T22:06:35.340693+0000","last_became_active":"2024-05-12T22:06:33.961583+0000","last_became_peered":"2024-05-12T22:06:33.961583+0000","last_unstale":"2024-05-12T22:06:35.340693+0000","last_undegraded":"2024-05-12T22:06:35.340693+0000","last_fullsized":"2024-05-12T22:06:35.340693+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T00:16:40.483194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:34.955430+0000","last_change":"2024-05-12T22:06:33.965776+0000","last_active":"2024-05-12T22:06:34.955430+0000","last_peered":"2024-05-12T22:06:34.955430+0000","last_clean":"2024-05-12T22:06:34.955430+0000","last_became_active":"2024-05-12T22:06:33.965439+0000","last_became_peered":"2024-05-12T22:06:33.965439+0000","last_unstale":"2024-05-12T22:06:34.955430+0000","last_undegraded":"2024-05-12T22:06:34.955430+0000","last_fullsized":"2024-05-12T22:06:34.955430+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T23:59:56.925209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:35.339787+0000","last_change":"2024-05-12T22:06:33.967906+0000","last_active":"2024-05-12T22:06:35.339787+0000","last_peered":"2024-05-12T22:06:35.339787+0000","last_clean":"2024-05-12T22:06:35.339787+0000","last_became_active":"2024-05-12T22:06:33.967516+0000","last_became_peered":"2024-05-12T22:06:33.967516+0000","last_unstale":"2024-05-12T22:06:35.339787+0000","last_undegraded":"2024-05-12T22:06:35.339787+0000","last_fullsized":"2024-05-12T22:06:35.339787+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T08:16:56.892024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:35.343740+0000","last_change":"2024-05-12T22:06:33.971456+0000","last_active":"2024-05-12T22:06:35.343740+0000","last_peered":"2024-05-12T22:06:35.343740+0000","last_clean":"2024-05-12T22:06:35.343740+0000","last_became_active":"2024-05-12T22:06:33.971110+0000","last_became_peered":"2024-05-12T22:06:33.971110+0000","last_unstale":"2024-05-12T22:06:35.343740+0000","last_undegraded":"2024-05-12T22:06:35.343740+0000","last_fullsized":"2024-05-12T22:06:35.343740+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T09:48:56.275346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:35.340215+0000","last_change":"2024-05-12T22:06:33.983421+0000","last_active":"2024-05-12T22:06:35.340215+0000","last_peered":"2024-05-12T22:06:35.340215+0000","last_clean":"2024-05-12T22:06:35.340215+0000","last_became_active":"2024-05-12T22:06:33.978515+0000","last_became_peered":"2024-05-12T22:06:33.978515+0000","last_unstale":"2024-05-12T22:06:35.340215+0000","last_undegraded":"2024-05-12T22:06:35.340215+0000","last_fullsized":"2024-05-12T22:06:35.340215+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T23:09:48.264541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:35.340839+0000","last_change":"2024-05-12T22:06:33.980956+0000","last_active":"2024-05-12T22:06:35.340839+0000","last_peered":"2024-05-12T22:06:35.340839+0000","last_clean":"2024-05-12T22:06:35.340839+0000","last_became_active":"2024-05-12T22:06:33.979860+0000","last_became_peered":"2024-05-12T22:06:33.979860+0000","last_unstale":"2024-05-12T22:06:35.340839+0000","last_undegraded":"2024-05-12T22:06:35.340839+0000","last_fullsized":"2024-05-12T22:06:35.340839+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T02:58:29.255704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:35.339172+0000","last_change":"2024-05-12T22:06:33.978682+0000","last_active":"2024-05-12T22:06:35.339172+0000","last_peered":"2024-05-12T22:06:35.339172+0000","last_clean":"2024-05-12T22:06:35.339172+0000","last_became_active":"2024-05-12T22:06:33.978234+0000","last_became_peered":"2024-05-12T22:06:33.978234+0000","last_unstale":"2024-05-12T22:06:35.339172+0000","last_undegraded":"2024-05-12T22:06:35.339172+0000","last_fullsized":"2024-05-12T22:06:35.339172+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T07:40:48.540839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:35.521065+0000","last_change":"2024-05-12T22:06:33.986598+0000","last_active":"2024-05-12T22:06:35.521065+0000","last_peered":"2024-05-12T22:06:35.521065+0000","last_clean":"2024-05-12T22:06:35.521065+0000","last_became_active":"2024-05-12T22:06:33.986346+0000","last_became_peered":"2024-05-12T22:06:33.986346+0000","last_unstale":"2024-05-12T22:06:35.521065+0000","last_undegraded":"2024-05-12T22:06:35.521065+0000","last_fullsized":"2024-05-12T22:06:35.521065+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T05:52:05.473099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:35.344836+0000","last_change":"2024-05-12T22:06:33.978900+0000","last_active":"2024-05-12T22:06:35.344836+0000","last_peered":"2024-05-12T22:06:35.344836+0000","last_clean":"2024-05-12T22:06:35.344836+0000","last_became_active":"2024-05-12T22:06:33.978474+0000","last_became_peered":"2024-05-12T22:06:33.978474+0000","last_unstale":"2024-05-12T22:06:35.344836+0000","last_undegraded":"2024-05-12T22:06:35.344836+0000","last_fullsized":"2024-05-12T22:06:35.344836+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T03:47:53.014263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:35.339462+0000","last_change":"2024-05-12T22:06:33.966752+0000","last_active":"2024-05-12T22:06:35.339462+0000","last_peered":"2024-05-12T22:06:35.339462+0000","last_clean":"2024-05-12T22:06:35.339462+0000","last_became_active":"2024-05-12T22:06:33.966320+0000","last_became_peered":"2024-05-12T22:06:33.966320+0000","last_unstale":"2024-05-12T22:06:35.339462+0000","last_undegraded":"2024-05-12T22:06:35.339462+0000","last_fullsized":"2024-05-12T22:06:35.339462+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T22:11:48.356156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'9","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:38.332078+0000","last_change":"2024-05-12T22:06:32.931346+0000","last_active":"2024-05-12T22:06:38.332078+0000","last_peered":"2024-05-12T22:06:38.332078+0000","last_clean":"2024-05-12T22:06:38.332078+0000","last_became_active":"2024-05-12T22:06:32.930909+0000","last_became_peered":"2024-05-12T22:06:32.930909+0000","last_unstale":"2024-05-12T22:06:38.332078+0000","last_undegraded":"2024-05-12T22:06:38.332078+0000","last_fullsized":"2024-05-12T22:06:38.332078+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T01:16:34.444226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:34.954955+0000","last_change":"2024-05-12T22:06:33.963875+0000","last_active":"2024-05-12T22:06:34.954955+0000","last_peered":"2024-05-12T22:06:34.954955+0000","last_clean":"2024-05-12T22:06:34.954955+0000","last_became_active":"2024-05-12T22:06:33.963239+0000","last_became_peered":"2024-05-12T22:06:33.963239+0000","last_unstale":"2024-05-12T22:06:34.954955+0000","last_undegraded":"2024-05-12T22:06:34.954955+0000","last_fullsized":"2024-05-12T22:06:34.954955+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T23:15:01.832082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:34.955193+0000","last_change":"2024-05-12T22:06:32.950352+0000","last_active":"2024-05-12T22:06:34.955193+0000","last_peered":"2024-05-12T22:06:34.955193+0000","last_clean":"2024-05-12T22:06:34.955193+0000","last_became_active":"2024-05-12T22:06:32.949759+0000","last_became_peered":"2024-05-12T22:06:32.949759+0000","last_unstale":"2024-05-12T22:06:34.955193+0000","last_undegraded":"2024-05-12T22:06:34.955193+0000","last_fullsized":"2024-05-12T22:06:34.955193+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T06:28:07.613351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:35.341217+0000","last_change":"2024-05-12T22:06:33.990783+0000","last_active":"2024-05-12T22:06:35.341217+0000","last_peered":"2024-05-12T22:06:35.341217+0000","last_clean":"2024-05-12T22:06:35.341217+0000","last_became_active":"2024-05-12T22:06:33.990387+0000","last_became_peered":"2024-05-12T22:06:33.990387+0000","last_unstale":"2024-05-12T22:06:35.341217+0000","last_undegraded":"2024-05-12T22:06:35.341217+0000","last_fullsized":"2024-05-12T22:06:35.341217+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T05:52:52.247458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"86'4","reported_seq":19,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:38.324914+0000","last_change":"2024-05-12T22:06:32.931718+0000","last_active":"2024-05-12T22:06:38.324914+0000","last_peered":"2024-05-12T22:06:38.324914+0000","last_clean":"2024-05-12T22:06:38.324914+0000","last_became_active":"2024-05-12T22:06:32.931404+0000","last_became_peered":"2024-05-12T22:06:32.931404+0000","last_unstale":"2024-05-12T22:06:38.324914+0000","last_undegraded":"2024-05-12T22:06:38.324914+0000","last_fullsized":"2024-05-12T22:06:38.324914+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T02:55:57.770504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:35.339769+0000","last_change":"2024-05-12T22:06:33.963299+0000","last_active":"2024-05-12T22:06:35.339769+0000","last_peered":"2024-05-12T22:06:35.339769+0000","last_clean":"2024-05-12T22:06:35.339769+0000","last_became_active":"2024-05-12T22:06:33.962916+0000","last_became_peered":"2024-05-12T22:06:33.962916+0000","last_unstale":"2024-05-12T22:06:35.339769+0000","last_undegraded":"2024-05-12T22:06:35.339769+0000","last_fullsized":"2024-05-12T22:06:35.339769+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T05:46:32.131918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:35.339875+0000","last_change":"2024-05-12T22:06:32.932319+0000","last_active":"2024-05-12T22:06:35.339875+0000","last_peered":"2024-05-12T22:06:35.339875+0000","last_clean":"2024-05-12T22:06:35.339875+0000","last_became_active":"2024-05-12T22:06:32.931936+0000","last_became_peered":"2024-05-12T22:06:32.931936+0000","last_unstale":"2024-05-12T22:06:35.339875+0000","last_undegraded":"2024-05-12T22:06:35.339875+0000","last_fullsized":"2024-05-12T22:06:35.339875+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T23:09:24.075793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:34.955558+0000","last_change":"2024-05-12T22:06:33.967926+0000","last_active":"2024-05-12T22:06:34.955558+0000","last_peered":"2024-05-12T22:06:34.955558+0000","last_clean":"2024-05-12T22:06:34.955558+0000","last_became_active":"2024-05-12T22:06:33.967588+0000","last_became_peered":"2024-05-12T22:06:33.967588+0000","last_unstale":"2024-05-12T22:06:34.955558+0000","last_undegraded":"2024-05-12T22:06:34.955558+0000","last_fullsized":"2024-05-12T22:06:34.955558+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T22:39:54.434117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"86'6","reported_seq":28,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:40.707340+0000","last_change":"2024-05-12T22:06:32.930347+0000","last_active":"2024-05-12T22:06:40.707340+0000","last_peered":"2024-05-12T22:06:40.707340+0000","last_clean":"2024-05-12T22:06:40.707340+0000","last_became_active":"2024-05-12T22:06:32.930083+0000","last_became_peered":"2024-05-12T22:06:32.930083+0000","last_unstale":"2024-05-12T22:06:40.707340+0000","last_undegraded":"2024-05-12T22:06:40.707340+0000","last_fullsized":"2024-05-12T22:06:40.707340+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T07:10:08.898490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:35.339000+0000","last_change":"2024-05-12T22:06:33.973793+0000","last_active":"2024-05-12T22:06:35.339000+0000","last_peered":"2024-05-12T22:06:35.339000+0000","last_clean":"2024-05-12T22:06:35.339000+0000","last_became_active":"2024-05-12T22:06:33.973380+0000","last_became_peered":"2024-05-12T22:06:33.973380+0000","last_unstale":"2024-05-12T22:06:35.339000+0000","last_undegraded":"2024-05-12T22:06:35.339000+0000","last_fullsized":"2024-05-12T22:06:35.339000+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T04:32:01.817780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:38.327455+0000","last_change":"2024-05-12T22:06:32.938215+0000","last_active":"2024-05-12T22:06:38.327455+0000","last_peered":"2024-05-12T22:06:38.327455+0000","last_clean":"2024-05-12T22:06:38.327455+0000","last_became_active":"2024-05-12T22:06:32.937939+0000","last_became_peered":"2024-05-12T22:06:32.937939+0000","last_unstale":"2024-05-12T22:06:38.327455+0000","last_undegraded":"2024-05-12T22:06:38.327455+0000","last_fullsized":"2024-05-12T22:06:38.327455+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T04:15:28.821835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:34.955403+0000","last_change":"2024-05-12T22:06:33.977362+0000","last_active":"2024-05-12T22:06:34.955403+0000","last_peered":"2024-05-12T22:06:34.955403+0000","last_clean":"2024-05-12T22:06:34.955403+0000","last_became_active":"2024-05-12T22:06:33.977074+0000","last_became_peered":"2024-05-12T22:06:33.977074+0000","last_unstale":"2024-05-12T22:06:34.955403+0000","last_undegraded":"2024-05-12T22:06:34.955403+0000","last_fullsized":"2024-05-12T22:06:34.955403+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T22:21:00.062780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"86'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:38.329547+0000","last_change":"2024-05-12T22:06:32.935508+0000","last_active":"2024-05-12T22:06:38.329547+0000","last_peered":"2024-05-12T22:06:38.329547+0000","last_clean":"2024-05-12T22:06:38.329547+0000","last_became_active":"2024-05-12T22:06:32.934760+0000","last_became_peered":"2024-05-12T22:06:32.934760+0000","last_unstale":"2024-05-12T22:06:38.329547+0000","last_undegraded":"2024-05-12T22:06:38.329547+0000","last_fullsized":"2024-05-12T22:06:38.329547+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T01:58:09.450187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:34.954484+0000","last_change":"2024-05-12T22:06:33.966421+0000","last_active":"2024-05-12T22:06:34.954484+0000","last_peered":"2024-05-12T22:06:34.954484+0000","last_clean":"2024-05-12T22:06:34.954484+0000","last_became_active":"2024-05-12T22:06:33.965627+0000","last_became_peered":"2024-05-12T22:06:33.965627+0000","last_unstale":"2024-05-12T22:06:34.954484+0000","last_undegraded":"2024-05-12T22:06:34.954484+0000","last_fullsized":"2024-05-12T22:06:34.954484+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T09:25:03.836082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"86'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:34.954795+0000","last_change":"2024-05-12T22:06:32.928907+0000","last_active":"2024-05-12T22:06:34.954795+0000","last_peered":"2024-05-12T22:06:34.954795+0000","last_clean":"2024-05-12T22:06:34.954795+0000","last_became_active":"2024-05-12T22:06:32.928586+0000","last_became_peered":"2024-05-12T22:06:32.928586+0000","last_unstale":"2024-05-12T22:06:34.954795+0000","last_undegraded":"2024-05-12T22:06:34.954795+0000","last_fullsized":"2024-05-12T22:06:34.954795+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T04:24:21.658957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:35.344161+0000","last_change":"2024-05-12T22:06:33.982873+0000","last_active":"2024-05-12T22:06:35.344161+0000","last_peered":"2024-05-12T22:06:35.344161+0000","last_clean":"2024-05-12T22:06:35.344161+0000","last_became_active":"2024-05-12T22:06:33.982495+0000","last_became_peered":"2024-05-12T22:06:33.982495+0000","last_unstale":"2024-05-12T22:06:35.344161+0000","last_undegraded":"2024-05-12T22:06:35.344161+0000","last_fullsized":"2024-05-12T22:06:35.344161+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T05:45:39.404282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"86'3","reported_seq":19,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:37.318487+0000","last_change":"2024-05-12T22:06:32.937716+0000","last_active":"2024-05-12T22:06:37.318487+0000","last_peered":"2024-05-12T22:06:37.318487+0000","last_clean":"2024-05-12T22:06:37.318487+0000","last_became_active":"2024-05-12T22:06:32.937418+0000","last_became_peered":"2024-05-12T22:06:32.937418+0000","last_unstale":"2024-05-12T22:06:37.318487+0000","last_undegraded":"2024-05-12T22:06:37.318487+0000","last_fullsized":"2024-05-12T22:06:37.318487+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T08:20:13.639958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:35.340221+0000","last_change":"2024-05-12T22:06:33.974702+0000","last_active":"2024-05-12T22:06:35.340221+0000","last_peered":"2024-05-12T22:06:35.340221+0000","last_clean":"2024-05-12T22:06:35.340221+0000","last_became_active":"2024-05-12T22:06:33.974363+0000","last_became_peered":"2024-05-12T22:06:33.974363+0000","last_unstale":"2024-05-12T22:06:35.340221+0000","last_undegraded":"2024-05-12T22:06:35.340221+0000","last_fullsized":"2024-05-12T22:06:35.340221+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T09:16:33.620945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"86'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:37.316282+0000","last_change":"2024-05-12T22:06:32.932583+0000","last_active":"2024-05-12T22:06:37.316282+0000","last_peered":"2024-05-12T22:06:37.316282+0000","last_clean":"2024-05-12T22:06:37.316282+0000","last_became_active":"2024-05-12T22:06:32.932251+0000","last_became_peered":"2024-05-12T22:06:32.932251+0000","last_unstale":"2024-05-12T22:06:37.316282+0000","last_undegraded":"2024-05-12T22:06:37.316282+0000","last_fullsized":"2024-05-12T22:06:37.316282+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T04:59:28.152763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:35.341062+0000","last_change":"2024-05-12T22:06:33.980496+0000","last_active":"2024-05-12T22:06:35.341062+0000","last_peered":"2024-05-12T22:06:35.341062+0000","last_clean":"2024-05-12T22:06:35.341062+0000","last_became_active":"2024-05-12T22:06:33.980155+0000","last_became_peered":"2024-05-12T22:06:33.980155+0000","last_unstale":"2024-05-12T22:06:35.341062+0000","last_undegraded":"2024-05-12T22:06:35.341062+0000","last_fullsized":"2024-05-12T22:06:35.341062+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T01:54:03.831828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"86'4","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:37.317039+0000","last_change":"2024-05-12T22:06:32.951633+0000","last_active":"2024-05-12T22:06:37.317039+0000","last_peered":"2024-05-12T22:06:37.317039+0000","last_clean":"2024-05-12T22:06:37.317039+0000","last_became_active":"2024-05-12T22:06:32.951382+0000","last_became_peered":"2024-05-12T22:06:32.951382+0000","last_unstale":"2024-05-12T22:06:37.317039+0000","last_undegraded":"2024-05-12T22:06:37.317039+0000","last_fullsized":"2024-05-12T22:06:37.317039+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T02:06:21.408593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:35.340267+0000","last_change":"2024-05-12T22:06:33.963493+0000","last_active":"2024-05-12T22:06:35.340267+0000","last_peered":"2024-05-12T22:06:35.340267+0000","last_clean":"2024-05-12T22:06:35.340267+0000","last_became_active":"2024-05-12T22:06:33.963127+0000","last_became_peered":"2024-05-12T22:06:33.963127+0000","last_unstale":"2024-05-12T22:06:35.340267+0000","last_undegraded":"2024-05-12T22:06:35.340267+0000","last_fullsized":"2024-05-12T22:06:35.340267+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T05:39:31.038446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"86'8","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:38.703111+0000","last_change":"2024-05-12T22:06:32.932026+0000","last_active":"2024-05-12T22:06:38.703111+0000","last_peered":"2024-05-12T22:06:38.703111+0000","last_clean":"2024-05-12T22:06:38.703111+0000","last_became_active":"2024-05-12T22:06:32.931667+0000","last_became_peered":"2024-05-12T22:06:32.931667+0000","last_unstale":"2024-05-12T22:06:38.703111+0000","last_undegraded":"2024-05-12T22:06:38.703111+0000","last_fullsized":"2024-05-12T22:06:38.703111+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T06:16:28.259047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":9,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:35.340037+0000","last_change":"2024-05-12T22:06:33.962677+0000","last_active":"2024-05-12T22:06:35.340037+0000","last_peered":"2024-05-12T22:06:35.340037+0000","last_clean":"2024-05-12T22:06:35.340037+0000","last_became_active":"2024-05-12T22:06:33.962295+0000","last_became_peered":"2024-05-12T22:06:33.962295+0000","last_unstale":"2024-05-12T22:06:35.340037+0000","last_undegraded":"2024-05-12T22:06:35.340037+0000","last_fullsized":"2024-05-12T22:06:35.340037+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T01:31:04.587046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"86'6","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:38.331216+0000","last_change":"2024-05-12T22:06:32.936104+0000","last_active":"2024-05-12T22:06:38.331216+0000","last_peered":"2024-05-12T22:06:38.331216+0000","last_clean":"2024-05-12T22:06:38.331216+0000","last_became_active":"2024-05-12T22:06:32.935701+0000","last_became_peered":"2024-05-12T22:06:32.935701+0000","last_unstale":"2024-05-12T22:06:38.331216+0000","last_undegraded":"2024-05-12T22:06:38.331216+0000","last_fullsized":"2024-05-12T22:06:38.331216+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T07:04:10.115302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:35.342726+0000","last_change":"2024-05-12T22:06:33.977850+0000","last_active":"2024-05-12T22:06:35.342726+0000","last_peered":"2024-05-12T22:06:35.342726+0000","last_clean":"2024-05-12T22:06:35.342726+0000","last_became_active":"2024-05-12T22:06:33.977511+0000","last_became_peered":"2024-05-12T22:06:33.977511+0000","last_unstale":"2024-05-12T22:06:35.342726+0000","last_undegraded":"2024-05-12T22:06:35.342726+0000","last_fullsized":"2024-05-12T22:06:35.342726+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T23:10:17.267823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:37.320195+0000","last_change":"2024-05-12T22:06:32.929955+0000","last_active":"2024-05-12T22:06:37.320195+0000","last_peered":"2024-05-12T22:06:37.320195+0000","last_clean":"2024-05-12T22:06:37.320195+0000","last_became_active":"2024-05-12T22:06:32.929623+0000","last_became_peered":"2024-05-12T22:06:32.929623+0000","last_unstale":"2024-05-12T22:06:37.320195+0000","last_undegraded":"2024-05-12T22:06:37.320195+0000","last_fullsized":"2024-05-12T22:06:37.320195+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T08:15:22.622908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:34.955279+0000","last_change":"2024-05-12T22:06:33.969771+0000","last_active":"2024-05-12T22:06:34.955279+0000","last_peered":"2024-05-12T22:06:34.955279+0000","last_clean":"2024-05-12T22:06:34.955279+0000","last_became_active":"2024-05-12T22:06:33.969296+0000","last_became_peered":"2024-05-12T22:06:33.969296+0000","last_unstale":"2024-05-12T22:06:34.955279+0000","last_undegraded":"2024-05-12T22:06:34.955279+0000","last_fullsized":"2024-05-12T22:06:34.955279+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T08:51:46.398555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"86'3","reported_seq":21,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:38.704381+0000","last_change":"2024-05-12T22:06:32.927707+0000","last_active":"2024-05-12T22:06:38.704381+0000","last_peered":"2024-05-12T22:06:38.704381+0000","last_clean":"2024-05-12T22:06:38.704381+0000","last_became_active":"2024-05-12T22:06:32.927406+0000","last_became_peered":"2024-05-12T22:06:32.927406+0000","last_unstale":"2024-05-12T22:06:38.704381+0000","last_undegraded":"2024-05-12T22:06:38.704381+0000","last_fullsized":"2024-05-12T22:06:38.704381+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T05:14:34.268589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:34.954844+0000","last_change":"2024-05-12T22:06:33.964996+0000","last_active":"2024-05-12T22:06:34.954844+0000","last_peered":"2024-05-12T22:06:34.954844+0000","last_clean":"2024-05-12T22:06:34.954844+0000","last_became_active":"2024-05-12T22:06:33.964587+0000","last_became_peered":"2024-05-12T22:06:33.964587+0000","last_unstale":"2024-05-12T22:06:34.954844+0000","last_undegraded":"2024-05-12T22:06:34.954844+0000","last_fullsized":"2024-05-12T22:06:34.954844+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T22:27:28.316377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"86'15","reported_seq":31,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:38.330792+0000","last_change":"2024-05-12T22:06:32.935543+0000","last_active":"2024-05-12T22:06:38.330792+0000","last_peered":"2024-05-12T22:06:38.330792+0000","last_clean":"2024-05-12T22:06:38.330792+0000","last_became_active":"2024-05-12T22:06:32.935145+0000","last_became_peered":"2024-05-12T22:06:32.935145+0000","last_unstale":"2024-05-12T22:06:38.330792+0000","last_undegraded":"2024-05-12T22:06:38.330792+0000","last_fullsized":"2024-05-12T22:06:38.330792+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T01:13:23.289057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:34.955030+0000","last_change":"2024-05-12T22:06:33.964367+0000","last_active":"2024-05-12T22:06:34.955030+0000","last_peered":"2024-05-12T22:06:34.955030+0000","last_clean":"2024-05-12T22:06:34.955030+0000","last_became_active":"2024-05-12T22:06:33.963876+0000","last_became_peered":"2024-05-12T22:06:33.963876+0000","last_unstale":"2024-05-12T22:06:34.955030+0000","last_undegraded":"2024-05-12T22:06:34.955030+0000","last_fullsized":"2024-05-12T22:06:34.955030+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T07:09:20.962965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"86'2","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:38.699116+0000","last_change":"2024-05-12T22:06:32.927270+0000","last_active":"2024-05-12T22:06:38.699116+0000","last_peered":"2024-05-12T22:06:38.699116+0000","last_clean":"2024-05-12T22:06:38.699116+0000","last_became_active":"2024-05-12T22:06:32.926930+0000","last_became_peered":"2024-05-12T22:06:32.926930+0000","last_unstale":"2024-05-12T22:06:38.699116+0000","last_undegraded":"2024-05-12T22:06:38.699116+0000","last_fullsized":"2024-05-12T22:06:38.699116+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T07:04:48.336599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:35.520861+0000","last_change":"2024-05-12T22:06:33.966350+0000","last_active":"2024-05-12T22:06:35.520861+0000","last_peered":"2024-05-12T22:06:35.520861+0000","last_clean":"2024-05-12T22:06:35.520861+0000","last_became_active":"2024-05-12T22:06:33.965974+0000","last_became_peered":"2024-05-12T22:06:33.965974+0000","last_unstale":"2024-05-12T22:06:35.520861+0000","last_undegraded":"2024-05-12T22:06:35.520861+0000","last_fullsized":"2024-05-12T22:06:35.520861+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T02:08:16.889684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:35.343874+0000","last_change":"2024-05-12T22:06:32.944945+0000","last_active":"2024-05-12T22:06:35.343874+0000","last_peered":"2024-05-12T22:06:35.343874+0000","last_clean":"2024-05-12T22:06:35.343874+0000","last_became_active":"2024-05-12T22:06:32.944576+0000","last_became_peered":"2024-05-12T22:06:32.944576+0000","last_unstale":"2024-05-12T22:06:35.343874+0000","last_undegraded":"2024-05-12T22:06:35.343874+0000","last_fullsized":"2024-05-12T22:06:35.343874+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T07:29:14.544821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:35.340754+0000","last_change":"2024-05-12T22:06:33.966299+0000","last_active":"2024-05-12T22:06:35.340754+0000","last_peered":"2024-05-12T22:06:35.340754+0000","last_clean":"2024-05-12T22:06:35.340754+0000","last_became_active":"2024-05-12T22:06:33.965902+0000","last_became_peered":"2024-05-12T22:06:33.965902+0000","last_unstale":"2024-05-12T22:06:35.340754+0000","last_undegraded":"2024-05-12T22:06:35.340754+0000","last_fullsized":"2024-05-12T22:06:35.340754+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T22:37:51.476546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"86'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:34.954969+0000","last_change":"2024-05-12T22:06:32.937210+0000","last_active":"2024-05-12T22:06:34.954969+0000","last_peered":"2024-05-12T22:06:34.954969+0000","last_clean":"2024-05-12T22:06:34.954969+0000","last_became_active":"2024-05-12T22:06:32.936805+0000","last_became_peered":"2024-05-12T22:06:32.936805+0000","last_unstale":"2024-05-12T22:06:34.954969+0000","last_undegraded":"2024-05-12T22:06:34.954969+0000","last_fullsized":"2024-05-12T22:06:34.954969+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T06:14:20.825258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:35.340359+0000","last_change":"2024-05-12T22:06:33.983604+0000","last_active":"2024-05-12T22:06:35.340359+0000","last_peered":"2024-05-12T22:06:35.340359+0000","last_clean":"2024-05-12T22:06:35.340359+0000","last_became_active":"2024-05-12T22:06:33.983168+0000","last_became_peered":"2024-05-12T22:06:33.983168+0000","last_unstale":"2024-05-12T22:06:35.340359+0000","last_undegraded":"2024-05-12T22:06:35.340359+0000","last_fullsized":"2024-05-12T22:06:35.340359+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T05:54:55.854593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:34.955536+0000","last_change":"2024-05-12T22:06:33.968848+0000","last_active":"2024-05-12T22:06:34.955536+0000","last_peered":"2024-05-12T22:06:34.955536+0000","last_clean":"2024-05-12T22:06:34.955536+0000","last_became_active":"2024-05-12T22:06:33.968529+0000","last_became_peered":"2024-05-12T22:06:33.968529+0000","last_unstale":"2024-05-12T22:06:34.955536+0000","last_undegraded":"2024-05-12T22:06:34.955536+0000","last_fullsized":"2024-05-12T22:06:34.955536+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T03:00:41.129662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"86'4","reported_seq":23,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:40.708985+0000","last_change":"2024-05-12T22:06:32.946670+0000","last_active":"2024-05-12T22:06:40.708985+0000","last_peered":"2024-05-12T22:06:40.708985+0000","last_clean":"2024-05-12T22:06:40.708985+0000","last_became_active":"2024-05-12T22:06:32.946059+0000","last_became_peered":"2024-05-12T22:06:32.946059+0000","last_unstale":"2024-05-12T22:06:40.708985+0000","last_undegraded":"2024-05-12T22:06:40.708985+0000","last_fullsized":"2024-05-12T22:06:40.708985+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T22:35:56.116979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:34.954731+0000","last_change":"2024-05-12T22:06:33.962177+0000","last_active":"2024-05-12T22:06:34.954731+0000","last_peered":"2024-05-12T22:06:34.954731+0000","last_clean":"2024-05-12T22:06:34.954731+0000","last_became_active":"2024-05-12T22:06:33.960919+0000","last_became_peered":"2024-05-12T22:06:33.960919+0000","last_unstale":"2024-05-12T22:06:34.954731+0000","last_undegraded":"2024-05-12T22:06:34.954731+0000","last_fullsized":"2024-05-12T22:06:34.954731+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T00:39:54.566460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'3","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:38.700719+0000","last_change":"2024-05-12T22:06:32.927885+0000","last_active":"2024-05-12T22:06:38.700719+0000","last_peered":"2024-05-12T22:06:38.700719+0000","last_clean":"2024-05-12T22:06:38.700719+0000","last_became_active":"2024-05-12T22:06:32.927567+0000","last_became_peered":"2024-05-12T22:06:32.927567+0000","last_unstale":"2024-05-12T22:06:38.700719+0000","last_undegraded":"2024-05-12T22:06:38.700719+0000","last_fullsized":"2024-05-12T22:06:38.700719+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T07:13:50.811992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":348,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:34.954726+0000","last_change":"2024-05-12T22:05:17.683363+0000","last_active":"2024-05-12T22:06:34.954726+0000","last_peered":"2024-05-12T22:06:34.954726+0000","last_clean":"2024-05-12T22:06:34.954726+0000","last_became_active":"2024-05-12T22:05:17.682824+0000","last_became_peered":"2024-05-12T22:05:17.682824+0000","last_unstale":"2024-05-12T22:06:34.954726+0000","last_undegraded":"2024-05-12T22:06:34.954726+0000","last_fullsized":"2024-05-12T22:06:34.954726+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:01:32.304714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:01:32.304714+0000","last_clean_scrub_stamp":"2024-05-12T22:01:32.304714+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T06:35:22.474820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:35.342343+0000","last_change":"2024-05-12T22:06:33.965332+0000","last_active":"2024-05-12T22:06:35.342343+0000","last_peered":"2024-05-12T22:06:35.342343+0000","last_clean":"2024-05-12T22:06:35.342343+0000","last_became_active":"2024-05-12T22:06:33.964858+0000","last_became_peered":"2024-05-12T22:06:33.964858+0000","last_unstale":"2024-05-12T22:06:35.342343+0000","last_undegraded":"2024-05-12T22:06:35.342343+0000","last_fullsized":"2024-05-12T22:06:35.342343+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T05:14:09.825614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:38.324423+0000","last_change":"2024-05-12T22:06:32.929773+0000","last_active":"2024-05-12T22:06:38.324423+0000","last_peered":"2024-05-12T22:06:38.324423+0000","last_clean":"2024-05-12T22:06:38.324423+0000","last_became_active":"2024-05-12T22:06:32.929458+0000","last_became_peered":"2024-05-12T22:06:32.929458+0000","last_unstale":"2024-05-12T22:06:38.324423+0000","last_undegraded":"2024-05-12T22:06:38.324423+0000","last_fullsized":"2024-05-12T22:06:38.324423+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T01:30:42.853183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"86'1","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:37.316957+0000","last_change":"2024-05-12T22:06:32.947014+0000","last_active":"2024-05-12T22:06:37.316957+0000","last_peered":"2024-05-12T22:06:37.316957+0000","last_clean":"2024-05-12T22:06:37.316957+0000","last_became_active":"2024-05-12T22:06:32.946661+0000","last_became_peered":"2024-05-12T22:06:32.946661+0000","last_unstale":"2024-05-12T22:06:37.316957+0000","last_undegraded":"2024-05-12T22:06:37.316957+0000","last_fullsized":"2024-05-12T22:06:37.316957+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T23:09:19.378353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:35.342207+0000","last_change":"2024-05-12T22:06:33.980490+0000","last_active":"2024-05-12T22:06:35.342207+0000","last_peered":"2024-05-12T22:06:35.342207+0000","last_clean":"2024-05-12T22:06:35.342207+0000","last_became_active":"2024-05-12T22:06:33.980133+0000","last_became_peered":"2024-05-12T22:06:33.980133+0000","last_unstale":"2024-05-12T22:06:35.342207+0000","last_undegraded":"2024-05-12T22:06:35.342207+0000","last_fullsized":"2024-05-12T22:06:35.342207+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T00:28:10.461428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:38.329705+0000","last_change":"2024-05-12T22:06:32.943152+0000","last_active":"2024-05-12T22:06:38.329705+0000","last_peered":"2024-05-12T22:06:38.329705+0000","last_clean":"2024-05-12T22:06:38.329705+0000","last_became_active":"2024-05-12T22:06:32.942769+0000","last_became_peered":"2024-05-12T22:06:32.942769+0000","last_unstale":"2024-05-12T22:06:38.329705+0000","last_undegraded":"2024-05-12T22:06:38.329705+0000","last_fullsized":"2024-05-12T22:06:38.329705+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T01:23:24.955739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:34.954952+0000","last_change":"2024-05-12T22:06:33.964081+0000","last_active":"2024-05-12T22:06:34.954952+0000","last_peered":"2024-05-12T22:06:34.954952+0000","last_clean":"2024-05-12T22:06:34.954952+0000","last_became_active":"2024-05-12T22:06:33.963054+0000","last_became_peered":"2024-05-12T22:06:33.963054+0000","last_unstale":"2024-05-12T22:06:34.954952+0000","last_undegraded":"2024-05-12T22:06:34.954952+0000","last_fullsized":"2024-05-12T22:06:34.954952+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T09:40:03.396200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"86'4","reported_seq":19,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:37.318135+0000","last_change":"2024-05-12T22:06:32.936432+0000","last_active":"2024-05-12T22:06:37.318135+0000","last_peered":"2024-05-12T22:06:37.318135+0000","last_clean":"2024-05-12T22:06:37.318135+0000","last_became_active":"2024-05-12T22:06:32.936094+0000","last_became_peered":"2024-05-12T22:06:32.936094+0000","last_unstale":"2024-05-12T22:06:37.318135+0000","last_undegraded":"2024-05-12T22:06:37.318135+0000","last_fullsized":"2024-05-12T22:06:37.318135+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T02:40:31.346118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:35.340679+0000","last_change":"2024-05-12T22:06:33.985953+0000","last_active":"2024-05-12T22:06:35.340679+0000","last_peered":"2024-05-12T22:06:35.340679+0000","last_clean":"2024-05-12T22:06:35.340679+0000","last_became_active":"2024-05-12T22:06:33.984864+0000","last_became_peered":"2024-05-12T22:06:33.984864+0000","last_unstale":"2024-05-12T22:06:35.340679+0000","last_undegraded":"2024-05-12T22:06:35.340679+0000","last_fullsized":"2024-05-12T22:06:35.340679+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T07:28:22.357841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:37.316020+0000","last_change":"2024-05-12T22:06:32.934510+0000","last_active":"2024-05-12T22:06:37.316020+0000","last_peered":"2024-05-12T22:06:37.316020+0000","last_clean":"2024-05-12T22:06:37.316020+0000","last_became_active":"2024-05-12T22:06:32.934222+0000","last_became_peered":"2024-05-12T22:06:32.934222+0000","last_unstale":"2024-05-12T22:06:37.316020+0000","last_undegraded":"2024-05-12T22:06:37.316020+0000","last_fullsized":"2024-05-12T22:06:37.316020+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T00:57:44.321629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:35.338903+0000","last_change":"2024-05-12T22:06:33.963870+0000","last_active":"2024-05-12T22:06:35.338903+0000","last_peered":"2024-05-12T22:06:35.338903+0000","last_clean":"2024-05-12T22:06:35.338903+0000","last_became_active":"2024-05-12T22:06:33.963533+0000","last_became_peered":"2024-05-12T22:06:33.963533+0000","last_unstale":"2024-05-12T22:06:35.338903+0000","last_undegraded":"2024-05-12T22:06:35.338903+0000","last_fullsized":"2024-05-12T22:06:35.338903+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T08:56:59.495469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:37.319040+0000","last_change":"2024-05-12T22:06:32.939738+0000","last_active":"2024-05-12T22:06:37.319040+0000","last_peered":"2024-05-12T22:06:37.319040+0000","last_clean":"2024-05-12T22:06:37.319040+0000","last_became_active":"2024-05-12T22:06:32.939365+0000","last_became_peered":"2024-05-12T22:06:32.939365+0000","last_unstale":"2024-05-12T22:06:37.319040+0000","last_undegraded":"2024-05-12T22:06:37.319040+0000","last_fullsized":"2024-05-12T22:06:37.319040+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T02:19:31.180834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:35.520724+0000","last_change":"2024-05-12T22:06:33.975064+0000","last_active":"2024-05-12T22:06:35.520724+0000","last_peered":"2024-05-12T22:06:35.520724+0000","last_clean":"2024-05-12T22:06:35.520724+0000","last_became_active":"2024-05-12T22:06:33.974613+0000","last_became_peered":"2024-05-12T22:06:33.974613+0000","last_unstale":"2024-05-12T22:06:35.520724+0000","last_undegraded":"2024-05-12T22:06:35.520724+0000","last_fullsized":"2024-05-12T22:06:35.520724+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T08:46:27.343777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"86'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:36.031010+0000","last_change":"2024-05-12T22:06:32.931148+0000","last_active":"2024-05-12T22:06:36.031010+0000","last_peered":"2024-05-12T22:06:36.031010+0000","last_clean":"2024-05-12T22:06:36.031010+0000","last_became_active":"2024-05-12T22:06:32.930717+0000","last_became_peered":"2024-05-12T22:06:32.930717+0000","last_unstale":"2024-05-12T22:06:36.031010+0000","last_undegraded":"2024-05-12T22:06:36.031010+0000","last_fullsized":"2024-05-12T22:06:36.031010+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T07:26:20.915467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:34.954710+0000","last_change":"2024-05-12T22:06:33.988390+0000","last_active":"2024-05-12T22:06:34.954710+0000","last_peered":"2024-05-12T22:06:34.954710+0000","last_clean":"2024-05-12T22:06:34.954710+0000","last_became_active":"2024-05-12T22:06:33.986970+0000","last_became_peered":"2024-05-12T22:06:33.986970+0000","last_unstale":"2024-05-12T22:06:34.954710+0000","last_undegraded":"2024-05-12T22:06:34.954710+0000","last_fullsized":"2024-05-12T22:06:34.954710+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T05:03:31.930261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:37.315887+0000","last_change":"2024-05-12T22:06:32.931161+0000","last_active":"2024-05-12T22:06:37.315887+0000","last_peered":"2024-05-12T22:06:37.315887+0000","last_clean":"2024-05-12T22:06:37.315887+0000","last_became_active":"2024-05-12T22:06:32.930801+0000","last_became_peered":"2024-05-12T22:06:32.930801+0000","last_unstale":"2024-05-12T22:06:37.315887+0000","last_undegraded":"2024-05-12T22:06:37.315887+0000","last_fullsized":"2024-05-12T22:06:37.315887+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T05:58:48.226595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:34.955052+0000","last_change":"2024-05-12T22:06:33.968256+0000","last_active":"2024-05-12T22:06:34.955052+0000","last_peered":"2024-05-12T22:06:34.955052+0000","last_clean":"2024-05-12T22:06:34.955052+0000","last_became_active":"2024-05-12T22:06:33.967885+0000","last_became_peered":"2024-05-12T22:06:33.967885+0000","last_unstale":"2024-05-12T22:06:34.955052+0000","last_undegraded":"2024-05-12T22:06:34.955052+0000","last_fullsized":"2024-05-12T22:06:34.955052+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T09:12:40.687367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"86'3","reported_seq":19,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:38.331104+0000","last_change":"2024-05-12T22:06:32.944736+0000","last_active":"2024-05-12T22:06:38.331104+0000","last_peered":"2024-05-12T22:06:38.331104+0000","last_clean":"2024-05-12T22:06:38.331104+0000","last_became_active":"2024-05-12T22:06:32.944269+0000","last_became_peered":"2024-05-12T22:06:32.944269+0000","last_unstale":"2024-05-12T22:06:38.331104+0000","last_undegraded":"2024-05-12T22:06:38.331104+0000","last_fullsized":"2024-05-12T22:06:38.331104+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T22:56:24.391267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:34.955430+0000","last_change":"2024-05-12T22:06:33.968360+0000","last_active":"2024-05-12T22:06:34.955430+0000","last_peered":"2024-05-12T22:06:34.955430+0000","last_clean":"2024-05-12T22:06:34.955430+0000","last_became_active":"2024-05-12T22:06:33.967967+0000","last_became_peered":"2024-05-12T22:06:33.967967+0000","last_unstale":"2024-05-12T22:06:34.955430+0000","last_undegraded":"2024-05-12T22:06:34.955430+0000","last_fullsized":"2024-05-12T22:06:34.955430+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T00:41:12.693858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"85'1","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:34.955243+0000","last_change":"2024-05-12T22:06:32.935993+0000","last_active":"2024-05-12T22:06:34.955243+0000","last_peered":"2024-05-12T22:06:34.955243+0000","last_clean":"2024-05-12T22:06:34.955243+0000","last_became_active":"2024-05-12T22:06:32.935615+0000","last_became_peered":"2024-05-12T22:06:32.935615+0000","last_unstale":"2024-05-12T22:06:34.955243+0000","last_undegraded":"2024-05-12T22:06:34.955243+0000","last_fullsized":"2024-05-12T22:06:34.955243+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T00:43:39.254637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"86'1","reported_seq":14,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:37.874646+0000","last_change":"2024-05-12T22:06:33.966110+0000","last_active":"2024-05-12T22:06:37.874646+0000","last_peered":"2024-05-12T22:06:37.874646+0000","last_clean":"2024-05-12T22:06:37.874646+0000","last_became_active":"2024-05-12T22:06:33.965755+0000","last_became_peered":"2024-05-12T22:06:33.965755+0000","last_unstale":"2024-05-12T22:06:37.874646+0000","last_undegraded":"2024-05-12T22:06:37.874646+0000","last_fullsized":"2024-05-12T22:06:37.874646+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T22:18:53.941335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:35.343164+0000","last_change":"2024-05-12T22:06:32.942229+0000","last_active":"2024-05-12T22:06:35.343164+0000","last_peered":"2024-05-12T22:06:35.343164+0000","last_clean":"2024-05-12T22:06:35.343164+0000","last_became_active":"2024-05-12T22:06:32.941826+0000","last_became_peered":"2024-05-12T22:06:32.941826+0000","last_unstale":"2024-05-12T22:06:35.343164+0000","last_undegraded":"2024-05-12T22:06:35.343164+0000","last_fullsized":"2024-05-12T22:06:35.343164+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T03:52:34.886318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:34.955455+0000","last_change":"2024-05-12T22:06:33.963081+0000","last_active":"2024-05-12T22:06:34.955455+0000","last_peered":"2024-05-12T22:06:34.955455+0000","last_clean":"2024-05-12T22:06:34.955455+0000","last_became_active":"2024-05-12T22:06:33.962696+0000","last_became_peered":"2024-05-12T22:06:33.962696+0000","last_unstale":"2024-05-12T22:06:34.955455+0000","last_undegraded":"2024-05-12T22:06:34.955455+0000","last_fullsized":"2024-05-12T22:06:34.955455+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T05:55:45.581823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:35.520910+0000","last_change":"2024-05-12T22:06:33.968865+0000","last_active":"2024-05-12T22:06:35.520910+0000","last_peered":"2024-05-12T22:06:35.520910+0000","last_clean":"2024-05-12T22:06:35.520910+0000","last_became_active":"2024-05-12T22:06:33.968185+0000","last_became_peered":"2024-05-12T22:06:33.968185+0000","last_unstale":"2024-05-12T22:06:35.520910+0000","last_undegraded":"2024-05-12T22:06:35.520910+0000","last_fullsized":"2024-05-12T22:06:35.520910+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T01:23:41.083778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:35.339719+0000","last_change":"2024-05-12T22:06:33.964786+0000","last_active":"2024-05-12T22:06:35.339719+0000","last_peered":"2024-05-12T22:06:35.339719+0000","last_clean":"2024-05-12T22:06:35.339719+0000","last_became_active":"2024-05-12T22:06:33.964227+0000","last_became_peered":"2024-05-12T22:06:33.964227+0000","last_unstale":"2024-05-12T22:06:35.339719+0000","last_undegraded":"2024-05-12T22:06:35.339719+0000","last_fullsized":"2024-05-12T22:06:35.339719+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T02:53:42.739455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:35.339370+0000","last_change":"2024-05-12T22:06:33.978857+0000","last_active":"2024-05-12T22:06:35.339370+0000","last_peered":"2024-05-12T22:06:35.339370+0000","last_clean":"2024-05-12T22:06:35.339370+0000","last_became_active":"2024-05-12T22:06:33.978531+0000","last_became_peered":"2024-05-12T22:06:33.978531+0000","last_unstale":"2024-05-12T22:06:35.339370+0000","last_undegraded":"2024-05-12T22:06:35.339370+0000","last_fullsized":"2024-05-12T22:06:35.339370+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T02:51:05.609267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:35.340667+0000","last_change":"2024-05-12T22:06:33.980662+0000","last_active":"2024-05-12T22:06:35.340667+0000","last_peered":"2024-05-12T22:06:35.340667+0000","last_clean":"2024-05-12T22:06:35.340667+0000","last_became_active":"2024-05-12T22:06:33.980332+0000","last_became_peered":"2024-05-12T22:06:33.980332+0000","last_unstale":"2024-05-12T22:06:35.340667+0000","last_undegraded":"2024-05-12T22:06:35.340667+0000","last_fullsized":"2024-05-12T22:06:35.340667+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T06:21:59.715792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:35.342553+0000","last_change":"2024-05-12T22:06:33.987338+0000","last_active":"2024-05-12T22:06:35.342553+0000","last_peered":"2024-05-12T22:06:35.342553+0000","last_clean":"2024-05-12T22:06:35.342553+0000","last_became_active":"2024-05-12T22:06:33.986942+0000","last_became_peered":"2024-05-12T22:06:33.986942+0000","last_unstale":"2024-05-12T22:06:35.342553+0000","last_undegraded":"2024-05-12T22:06:35.342553+0000","last_fullsized":"2024-05-12T22:06:35.342553+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T01:02:05.137558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:35.520956+0000","last_change":"2024-05-12T22:06:33.966967+0000","last_active":"2024-05-12T22:06:35.520956+0000","last_peered":"2024-05-12T22:06:35.520956+0000","last_clean":"2024-05-12T22:06:35.520956+0000","last_became_active":"2024-05-12T22:06:33.965375+0000","last_became_peered":"2024-05-12T22:06:33.965375+0000","last_unstale":"2024-05-12T22:06:35.520956+0000","last_undegraded":"2024-05-12T22:06:35.520956+0000","last_fullsized":"2024-05-12T22:06:35.520956+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T04:57:36.341705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:35.339754+0000","last_change":"2024-05-12T22:06:33.963902+0000","last_active":"2024-05-12T22:06:35.339754+0000","last_peered":"2024-05-12T22:06:35.339754+0000","last_clean":"2024-05-12T22:06:35.339754+0000","last_became_active":"2024-05-12T22:06:33.963436+0000","last_became_peered":"2024-05-12T22:06:33.963436+0000","last_unstale":"2024-05-12T22:06:35.339754+0000","last_undegraded":"2024-05-12T22:06:35.339754+0000","last_fullsized":"2024-05-12T22:06:35.339754+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T02:51:56.415167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:34.954919+0000","last_change":"2024-05-12T22:06:33.969491+0000","last_active":"2024-05-12T22:06:34.954919+0000","last_peered":"2024-05-12T22:06:34.954919+0000","last_clean":"2024-05-12T22:06:34.954919+0000","last_became_active":"2024-05-12T22:06:33.969161+0000","last_became_peered":"2024-05-12T22:06:33.969161+0000","last_unstale":"2024-05-12T22:06:34.954919+0000","last_undegraded":"2024-05-12T22:06:34.954919+0000","last_fullsized":"2024-05-12T22:06:34.954919+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T06:04:21.847520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:34.954990+0000","last_change":"2024-05-12T22:06:33.973490+0000","last_active":"2024-05-12T22:06:34.954990+0000","last_peered":"2024-05-12T22:06:34.954990+0000","last_clean":"2024-05-12T22:06:34.954990+0000","last_became_active":"2024-05-12T22:06:33.973161+0000","last_became_peered":"2024-05-12T22:06:33.973161+0000","last_unstale":"2024-05-12T22:06:34.954990+0000","last_undegraded":"2024-05-12T22:06:34.954990+0000","last_fullsized":"2024-05-12T22:06:34.954990+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T03:29:36.972631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:34.954452+0000","last_change":"2024-05-12T22:06:33.957320+0000","last_active":"2024-05-12T22:06:34.954452+0000","last_peered":"2024-05-12T22:06:34.954452+0000","last_clean":"2024-05-12T22:06:34.954452+0000","last_became_active":"2024-05-12T22:06:33.956837+0000","last_became_peered":"2024-05-12T22:06:33.956837+0000","last_unstale":"2024-05-12T22:06:34.954452+0000","last_undegraded":"2024-05-12T22:06:34.954452+0000","last_fullsized":"2024-05-12T22:06:34.954452+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T01:20:51.018175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:35.339899+0000","last_change":"2024-05-12T22:06:33.968363+0000","last_active":"2024-05-12T22:06:35.339899+0000","last_peered":"2024-05-12T22:06:35.339899+0000","last_clean":"2024-05-12T22:06:35.339899+0000","last_became_active":"2024-05-12T22:06:33.968031+0000","last_became_peered":"2024-05-12T22:06:33.968031+0000","last_unstale":"2024-05-12T22:06:35.339899+0000","last_undegraded":"2024-05-12T22:06:35.339899+0000","last_fullsized":"2024-05-12T22:06:35.339899+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T00:14:07.124058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:35.343242+0000","last_change":"2024-05-12T22:06:33.984823+0000","last_active":"2024-05-12T22:06:35.343242+0000","last_peered":"2024-05-12T22:06:35.343242+0000","last_clean":"2024-05-12T22:06:35.343242+0000","last_became_active":"2024-05-12T22:06:33.984430+0000","last_became_peered":"2024-05-12T22:06:33.984430+0000","last_unstale":"2024-05-12T22:06:35.343242+0000","last_undegraded":"2024-05-12T22:06:35.343242+0000","last_fullsized":"2024-05-12T22:06:35.343242+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T01:30:18.109265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-05-12T22:06:35.520677+0000","last_change":"2024-05-12T22:06:33.962660+0000","last_active":"2024-05-12T22:06:35.520677+0000","last_peered":"2024-05-12T22:06:35.520677+0000","last_clean":"2024-05-12T22:06:35.520677+0000","last_became_active":"2024-05-12T22:06:33.962284+0000","last_became_peered":"2024-05-12T22:06:33.962284+0000","last_unstale":"2024-05-12T22:06:35.520677+0000","last_undegraded":"2024-05-12T22:06:35.520677+0000","last_fullsized":"2024-05-12T22:06:35.520677+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T01:08:53.888992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":5613,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":126,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":303104,"data_stored":79128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":110,"ondisk_log_size":110,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892350990,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27920,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717232,"statfs":{"total":95995035648,"available":95966445568,"internally_reserved":0,"allocated":925696,"data_stored":442154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612626,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28540,"kb_used_data":1524,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716612,"statfs":{"total":95995035648,"available":95965810688,"internally_reserved":0,"allocated":1560576,"data_stored":1035130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776152,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27936,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717216,"statfs":{"total":95995035648,"available":95966429184,"internally_reserved":0,"allocated":942080,"data_stored":442390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037791,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32096,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713056,"statfs":{"total":95995035648,"available":95962169344,"internally_reserved":0,"allocated":942080,"data_stored":442310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299427,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28516,"kb_used_data":1500,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716636,"statfs":{"total":95995035648,"available":95965835264,"internally_reserved":0,"allocated":1536000,"data_stored":1034631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462954,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27924,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717228,"statfs":{"total":95995035648,"available":95966441472,"internally_reserved":0,"allocated":929792,"data_stored":441779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574917,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32120,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713032,"statfs":{"total":95995035648,"available":95962144768,"internally_reserved":0,"allocated":966656,"data_stored":477117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738442,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27944,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717208,"statfs":{"total":95995035648,"available":95966420992,"internally_reserved":0,"allocated":950272,"data_stored":443050,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411392,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27952,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717200,"statfs":{"total":95995035648,"available":95966412800,"internally_reserved":0,"allocated":958464,"data_stored":444534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149754,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27940,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717212,"statfs":{"total":95995035648,"available":95966425088,"internally_reserved":0,"allocated":946176,"data_stored":442090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888115,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27928,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717224,"statfs":{"total":95995035648,"available":95966437376,"internally_reserved":0,"allocated":933888,"data_stored":442216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724591,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28524,"kb_used_data":1508,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716628,"statfs":{"total":95995035648,"available":95965827072,"internally_reserved":0,"allocated":1544192,"data_stored":1033807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":44227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":22636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-12T22:06:44.279 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-05-12T22:06:44.279 INFO:tasks.cephadm:Setting up client nodes... 2024-05-12T22:06:44.280 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f72fecff68e1d400c4568684327c900485c20d6a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6b92e7b4-10aa-11ef-bc98-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-05-12T22:06:44.790 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/mon.a/config 2024-05-12T22:06:45.018 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:44 smithi028 ceph-mon[29847]: from='client.24713 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-12T22:06:45.019 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:44 smithi028 ceph-mon[29847]: pgmap v315: 97 pgs: 97 active+clean; 582 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 4.1 KiB/s wr, 16 op/s 2024-05-12T22:06:45.019 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:44 smithi028 ceph-mon[29847]: from='client.24719 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-12T22:06:45.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:44 smithi186 ceph-mon[34785]: from='client.24713 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-12T22:06:45.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:44 smithi186 ceph-mon[34785]: pgmap v315: 97 pgs: 97 active+clean; 582 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 4.1 KiB/s wr, 16 op/s 2024-05-12T22:06:45.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:44 smithi186 ceph-mon[34785]: from='client.24719 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-12T22:06:45.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:44 smithi086 ceph-mon[34297]: from='client.24713 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-12T22:06:45.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:44 smithi086 ceph-mon[34297]: pgmap v315: 97 pgs: 97 active+clean; 582 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 4.1 KiB/s wr, 16 op/s 2024-05-12T22:06:45.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:44 smithi086 ceph-mon[34297]: from='client.24719 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-12T22:06:46.033 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:45 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 1]}]: dispatch 2024-05-12T22:06:46.033 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:45 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]: dispatch 2024-05-12T22:06:46.034 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:45 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [8, 11]}]: dispatch 2024-05-12T22:06:46.034 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:45 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 11]}]: dispatch 2024-05-12T22:06:46.034 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:45 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]: dispatch 2024-05-12T22:06:46.034 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:45 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.25", "id": [5, 9]}]: dispatch 2024-05-12T22:06:46.034 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:45 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [4, 8]}]: dispatch 2024-05-12T22:06:46.034 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:45 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]: dispatch 2024-05-12T22:06:46.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:45 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 1]}]: dispatch 2024-05-12T22:06:46.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:45 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]: dispatch 2024-05-12T22:06:46.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:45 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [8, 11]}]: dispatch 2024-05-12T22:06:46.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:45 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 11]}]: dispatch 2024-05-12T22:06:46.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:45 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]: dispatch 2024-05-12T22:06:46.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:45 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.25", "id": [5, 9]}]: dispatch 2024-05-12T22:06:46.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:45 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [4, 8]}]: dispatch 2024-05-12T22:06:46.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:45 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]: dispatch 2024-05-12T22:06:46.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:45 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 1]}]: dispatch 2024-05-12T22:06:46.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:45 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]: dispatch 2024-05-12T22:06:46.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:45 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [8, 11]}]: dispatch 2024-05-12T22:06:46.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:45 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 11]}]: dispatch 2024-05-12T22:06:46.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:45 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]: dispatch 2024-05-12T22:06:46.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:45 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.25", "id": [5, 9]}]: dispatch 2024-05-12T22:06:46.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:45 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [4, 8]}]: dispatch 2024-05-12T22:06:46.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:45 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]: dispatch 2024-05-12T22:06:46.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:46.943+0000 7fdf5b400700 1 -- 172.21.15.28:0/377579832 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdf54102e40 msgr2=0x7fdf541096f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:46.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:46.943+0000 7fdf5b400700 1 --2- 172.21.15.28:0/377579832 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdf54102e40 0x7fdf541096f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fdf4c00b0a0 tx=0x7fdf4c00b3b0 comp rx=0 tx=0).stop 2024-05-12T22:06:46.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:46.943+0000 7fdf5b400700 1 -- 172.21.15.28:0/377579832 shutdown_connections 2024-05-12T22:06:46.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:46.943+0000 7fdf5b400700 1 --2- 172.21.15.28:0/377579832 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdf54102e40 0x7fdf541096f0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:46.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:46.943+0000 7fdf5b400700 1 --2- 172.21.15.28:0/377579832 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf541012a0 0x7fdf541016c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:46.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:46.943+0000 7fdf5b400700 1 --2- 172.21.15.28:0/377579832 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdf541024a0 0x7fdf54102900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:46.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:46.943+0000 7fdf5b400700 1 -- 172.21.15.28:0/377579832 >> 172.21.15.28:0/377579832 conn(0x7fdf540fc840 msgr2=0x7fdf540fec80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:06:46.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:46.944+0000 7fdf5b400700 1 -- 172.21.15.28:0/377579832 shutdown_connections 2024-05-12T22:06:46.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:46.944+0000 7fdf5b400700 1 -- 172.21.15.28:0/377579832 wait complete. 2024-05-12T22:06:46.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:46.945+0000 7fdf5b400700 1 Processor -- start 2024-05-12T22:06:46.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:46.945+0000 7fdf5b400700 1 -- start start 2024-05-12T22:06:46.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:46.946+0000 7fdf5b400700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdf541012a0 0x7fdf5419a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:46.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:46.946+0000 7fdf5b400700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdf541024a0 0x7fdf5419abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:46.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:46.946+0000 7fdf5b400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf54102e40 0x7fdf541a1c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:46.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:46.946+0000 7fdf5b400700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdf5410ceb0 con 0x7fdf54102e40 2024-05-12T22:06:46.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:46.946+0000 7fdf5b400700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fdf5410cd30 con 0x7fdf541024a0 2024-05-12T22:06:46.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:46.946+0000 7fdf5b400700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fdf5410d030 con 0x7fdf541012a0 2024-05-12T22:06:46.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:46.946+0000 7fdf58a00700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdf541012a0 0x7fdf5419a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:46.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:46.946+0000 7fdf58a00700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdf541012a0 0x7fdf5419a690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:38188/0 (socket says 172.21.15.28:38188) 2024-05-12T22:06:46.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:46.946+0000 7fdf58a00700 1 -- 172.21.15.28:0/2220996336 learned_addr learned my addr 172.21.15.28:0/2220996336 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:06:46.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:46.946+0000 7fdf53e00700 1 --2- 172.21.15.28:0/2220996336 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdf541024a0 0x7fdf5419abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:46.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:46.946+0000 7fdf59400700 1 --2- 172.21.15.28:0/2220996336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf54102e40 0x7fdf541a1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:46.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:46.947+0000 7fdf58a00700 1 -- 172.21.15.28:0/2220996336 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdf541024a0 msgr2=0x7fdf5419abd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:46.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:46.947+0000 7fdf58a00700 1 --2- 172.21.15.28:0/2220996336 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdf541024a0 0x7fdf5419abd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:46.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:46.947+0000 7fdf58a00700 1 -- 172.21.15.28:0/2220996336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf54102e40 msgr2=0x7fdf541a1c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:46.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:46.947+0000 7fdf58a00700 1 --2- 172.21.15.28:0/2220996336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf54102e40 0x7fdf541a1c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:46.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:46.947+0000 7fdf58a00700 1 -- 172.21.15.28:0/2220996336 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf541a22f0 con 0x7fdf541012a0 2024-05-12T22:06:46.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:46.947+0000 7fdf58a00700 1 --2- 172.21.15.28:0/2220996336 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdf541012a0 0x7fdf5419a690 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fdf4800e9a0 tx=0x7fdf4800ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:06:46.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:46.947+0000 7fdf51600700 1 -- 172.21.15.28:0/2220996336 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf48004220 con 0x7fdf541012a0 2024-05-12T22:06:46.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:46.947+0000 7fdf51600700 1 -- 172.21.15.28:0/2220996336 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdf480043c0 con 0x7fdf541012a0 2024-05-12T22:06:46.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:46.947+0000 7fdf51600700 1 -- 172.21.15.28:0/2220996336 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf48003df0 con 0x7fdf541012a0 2024-05-12T22:06:46.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:46.948+0000 7fdf5b400700 1 -- 172.21.15.28:0/2220996336 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf541a25e0 con 0x7fdf541012a0 2024-05-12T22:06:46.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:46.948+0000 7fdf5b400700 1 -- 172.21.15.28:0/2220996336 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf541a2b70 con 0x7fdf541012a0 2024-05-12T22:06:46.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:46.949+0000 7fdf5b400700 1 -- 172.21.15.28:0/2220996336 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf54050890 con 0x7fdf541012a0 2024-05-12T22:06:46.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:46.951+0000 7fdf51600700 1 -- 172.21.15.28:0/2220996336 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fdf4800c810 con 0x7fdf541012a0 2024-05-12T22:06:46.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:46.952+0000 7fdf51600700 1 --2- 172.21.15.28:0/2220996336 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fdf3c0a2de0 0x7fdf3c0a52a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:46.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:46.953+0000 7fdf51600700 1 -- 172.21.15.28:0/2220996336 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7fdf480cf450 con 0x7fdf541012a0 2024-05-12T22:06:46.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:46.953+0000 7fdf53e00700 1 --2- 172.21.15.28:0/2220996336 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fdf3c0a2de0 0x7fdf3c0a52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:46.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:46.954+0000 7fdf53e00700 1 --2- 172.21.15.28:0/2220996336 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fdf3c0a2de0 0x7fdf3c0a52a0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fdf40005950 tx=0x7fdf4000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:06:46.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:46.955+0000 7fdf51600700 1 -- 172.21.15.28:0/2220996336 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fdf4809c290 con 0x7fdf541012a0 2024-05-12T22:06:47.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:46 smithi028 ceph-mon[29847]: pgmap v316: 97 pgs: 97 active+clean; 624 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 8.4 KiB/s wr, 18 op/s 2024-05-12T22:06:47.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:46 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 1]}]': finished 2024-05-12T22:06:47.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:46 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]': finished 2024-05-12T22:06:47.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:46 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [8, 11]}]': finished 2024-05-12T22:06:47.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:46 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 11]}]': finished 2024-05-12T22:06:47.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:46 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]': finished 2024-05-12T22:06:47.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:46 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.25", "id": [5, 9]}]': finished 2024-05-12T22:06:47.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:46 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [4, 8]}]': finished 2024-05-12T22:06:47.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:46 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]': finished 2024-05-12T22:06:47.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:46 smithi028 ceph-mon[29847]: osdmap e87: 12 total, 12 up, 12 in 2024-05-12T22:06:47.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:46 smithi186 ceph-mon[34785]: pgmap v316: 97 pgs: 97 active+clean; 624 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 8.4 KiB/s wr, 18 op/s 2024-05-12T22:06:47.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:46 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 1]}]': finished 2024-05-12T22:06:47.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:46 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]': finished 2024-05-12T22:06:47.180 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:46 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [8, 11]}]': finished 2024-05-12T22:06:47.180 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:46 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 11]}]': finished 2024-05-12T22:06:47.180 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:46 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]': finished 2024-05-12T22:06:47.180 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:46 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.25", "id": [5, 9]}]': finished 2024-05-12T22:06:47.180 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:46 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [4, 8]}]': finished 2024-05-12T22:06:47.180 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:46 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]': finished 2024-05-12T22:06:47.180 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:46 smithi186 ceph-mon[34785]: osdmap e87: 12 total, 12 up, 12 in 2024-05-12T22:06:47.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:46 smithi086 ceph-mon[34297]: pgmap v316: 97 pgs: 97 active+clean; 624 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 8.4 KiB/s wr, 18 op/s 2024-05-12T22:06:47.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:46 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 1]}]': finished 2024-05-12T22:06:47.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:46 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]': finished 2024-05-12T22:06:47.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:46 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [8, 11]}]': finished 2024-05-12T22:06:47.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:46 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 11]}]': finished 2024-05-12T22:06:47.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:46 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]': finished 2024-05-12T22:06:47.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:46 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.25", "id": [5, 9]}]': finished 2024-05-12T22:06:47.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:46 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [4, 8]}]': finished 2024-05-12T22:06:47.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:46 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]': finished 2024-05-12T22:06:47.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:46 smithi086 ceph-mon[34297]: osdmap e87: 12 total, 12 up, 12 in 2024-05-12T22:06:47.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:47.283+0000 7fdf5b400700 1 -- 172.21.15.28:0/2220996336 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7fdf541a2e30 con 0x7fdf541012a0 2024-05-12T22:06:47.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:47.292+0000 7fdf51600700 1 -- 172.21.15.28:0/2220996336 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) ==== 170+0+59 (secure 0 0 0) 0x7fdf4809bb90 con 0x7fdf541012a0 2024-05-12T22:06:47.293 INFO:teuthology.orchestra.run.smithi028.stdout:[client.0] 2024-05-12T22:06:47.294 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQB3PUFmyEQZERAAj8fHdhR4XT7+WdMfcvRyrA== 2024-05-12T22:06:47.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:47.296+0000 7fdf5b400700 1 -- 172.21.15.28:0/2220996336 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fdf3c0a2de0 msgr2=0x7fdf3c0a52a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:47.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:47.296+0000 7fdf5b400700 1 --2- 172.21.15.28:0/2220996336 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fdf3c0a2de0 0x7fdf3c0a52a0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fdf40005950 tx=0x7fdf4000b410 comp rx=0 tx=0).stop 2024-05-12T22:06:47.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:47.296+0000 7fdf5b400700 1 -- 172.21.15.28:0/2220996336 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdf541012a0 msgr2=0x7fdf5419a690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:47.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:47.296+0000 7fdf5b400700 1 --2- 172.21.15.28:0/2220996336 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdf541012a0 0x7fdf5419a690 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fdf4800e9a0 tx=0x7fdf4800ecb0 comp rx=0 tx=0).stop 2024-05-12T22:06:47.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:47.296+0000 7fdf5b400700 1 -- 172.21.15.28:0/2220996336 shutdown_connections 2024-05-12T22:06:47.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:47.296+0000 7fdf5b400700 1 --2- 172.21.15.28:0/2220996336 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fdf3c0a2de0 0x7fdf3c0a52a0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:47.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:47.296+0000 7fdf5b400700 1 --2- 172.21.15.28:0/2220996336 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdf541012a0 0x7fdf5419a690 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:47.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:47.296+0000 7fdf5b400700 1 --2- 172.21.15.28:0/2220996336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf54102e40 0x7fdf541a1c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:47.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:47.296+0000 7fdf5b400700 1 --2- 172.21.15.28:0/2220996336 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdf541024a0 0x7fdf5419abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:47.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:47.296+0000 7fdf5b400700 1 -- 172.21.15.28:0/2220996336 >> 172.21.15.28:0/2220996336 conn(0x7fdf540fc840 msgr2=0x7fdf540fe1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:06:47.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:47.296+0000 7fdf5b400700 1 -- 172.21.15.28:0/2220996336 shutdown_connections 2024-05-12T22:06:47.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:47.296+0000 7fdf5b400700 1 -- 172.21.15.28:0/2220996336 wait complete. 2024-05-12T22:06:48.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:47 smithi028 ceph-mon[29847]: osdmap e88: 12 total, 12 up, 12 in 2024-05-12T22:06:48.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:47 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/2220996336' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-12T22:06:48.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:47 smithi028 ceph-mon[29847]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-12T22:06:48.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:47 smithi028 ceph-mon[29847]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-05-12T22:06:48.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:47 smithi186 ceph-mon[34785]: osdmap e88: 12 total, 12 up, 12 in 2024-05-12T22:06:48.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:47 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/2220996336' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-12T22:06:48.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:47 smithi186 ceph-mon[34785]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-12T22:06:48.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:47 smithi186 ceph-mon[34785]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-05-12T22:06:48.186 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-05-12T22:06:48.186 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-05-12T22:06:48.186 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-05-12T22:06:48.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:47 smithi086 ceph-mon[34297]: osdmap e88: 12 total, 12 up, 12 in 2024-05-12T22:06:48.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:47 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/2220996336' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-12T22:06:48.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:47 smithi086 ceph-mon[34297]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-12T22:06:48.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:47 smithi086 ceph-mon[34297]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-05-12T22:06:48.301 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-05-12T22:06:48.301 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-05-12T22:06:48.302 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f72fecff68e1d400c4568684327c900485c20d6a shell --fsid 6b92e7b4-10aa-11ef-bc98-c7b262605968 -- ceph mgr dump --format=json 2024-05-12T22:06:48.756 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/mon.a/config 2024-05-12T22:06:49.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:49 smithi028 ceph-mon[29847]: pgmap v319: 97 pgs: 97 active+clean; 624 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 8.3 KiB/s wr, 14 op/s 2024-05-12T22:06:49.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:49 smithi186 ceph-mon[34785]: pgmap v319: 97 pgs: 97 active+clean; 624 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 8.3 KiB/s wr, 14 op/s 2024-05-12T22:06:49.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:49 smithi086 ceph-mon[34297]: pgmap v319: 97 pgs: 97 active+clean; 624 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 8.3 KiB/s wr, 14 op/s 2024-05-12T22:06:51.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:51.087+0000 7f3a4b200700 1 -- 172.21.15.28:0/2107278343 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3a4c102e60 msgr2=0x7f3a4c106eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:51.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:51.087+0000 7f3a4b200700 1 --2- 172.21.15.28:0/2107278343 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3a4c102e60 0x7f3a4c106eb0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f3a34009990 tx=0x7f3a34009ca0 comp rx=0 tx=0).stop 2024-05-12T22:06:51.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:51.087+0000 7f3a4b200700 1 -- 172.21.15.28:0/2107278343 shutdown_connections 2024-05-12T22:06:51.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:51.087+0000 7f3a4b200700 1 --2- 172.21.15.28:0/2107278343 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3a4c107590 0x7f3a4c10f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:51.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:51.087+0000 7f3a4b200700 1 --2- 172.21.15.28:0/2107278343 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a4c1024b0 0x7f3a4c102890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:51.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:51.087+0000 7f3a4b200700 1 --2- 172.21.15.28:0/2107278343 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3a4c102e60 0x7f3a4c106eb0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:51.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:51.087+0000 7f3a4b200700 1 -- 172.21.15.28:0/2107278343 >> 172.21.15.28:0/2107278343 conn(0x7f3a4c0fdda0 msgr2=0x7f3a4c1001c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:06:51.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:51.087+0000 7f3a4b200700 1 -- 172.21.15.28:0/2107278343 shutdown_connections 2024-05-12T22:06:51.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:51.087+0000 7f3a4b200700 1 -- 172.21.15.28:0/2107278343 wait complete. 2024-05-12T22:06:51.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:51.089+0000 7f3a4b200700 1 Processor -- start 2024-05-12T22:06:51.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:51.089+0000 7f3a4b200700 1 -- start start 2024-05-12T22:06:51.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:51.090+0000 7f3a4b200700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3a4c1024b0 0x7f3a4c19e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:51.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:51.090+0000 7f3a4b200700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a4c102e60 0x7f3a4c19ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:51.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:51.090+0000 7f3a4b200700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3a4c107590 0x7f3a4c1988a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:51.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:51.090+0000 7f3a4b200700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3a4c1125a0 con 0x7f3a4c102e60 2024-05-12T22:06:51.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:51.090+0000 7f3a4b200700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f3a4c112420 con 0x7f3a4c107590 2024-05-12T22:06:51.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:51.090+0000 7f3a4b200700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f3a4c112720 con 0x7f3a4c1024b0 2024-05-12T22:06:51.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:51.090+0000 7f3a49e00700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3a4c1024b0 0x7f3a4c19e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:51.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:51.090+0000 7f3a41400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a4c102e60 0x7f3a4c19ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:51.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:51.091+0000 7f3a41400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a4c102e60 0x7f3a4c19ec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53820/0 (socket says 172.21.15.28:53820) 2024-05-12T22:06:51.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:51.091+0000 7f3a41400700 1 -- 172.21.15.28:0/1586252187 learned_addr learned my addr 172.21.15.28:0/1586252187 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:06:51.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:51.091+0000 7f3a4a800700 1 --2- 172.21.15.28:0/1586252187 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3a4c107590 0x7f3a4c1988a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:51.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:51.091+0000 7f3a41400700 1 -- 172.21.15.28:0/1586252187 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3a4c1024b0 msgr2=0x7f3a4c19e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:51.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:51.091+0000 7f3a41400700 1 --2- 172.21.15.28:0/1586252187 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3a4c1024b0 0x7f3a4c19e6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:51.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:51.091+0000 7f3a41400700 1 -- 172.21.15.28:0/1586252187 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3a4c107590 msgr2=0x7f3a4c1988a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:51.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:51.091+0000 7f3a41400700 1 --2- 172.21.15.28:0/1586252187 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3a4c107590 0x7f3a4c1988a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:51.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:51.091+0000 7f3a41400700 1 -- 172.21.15.28:0/1586252187 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a4c199080 con 0x7f3a4c102e60 2024-05-12T22:06:51.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:51.091+0000 7f3a49e00700 1 --2- 172.21.15.28:0/1586252187 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3a4c1024b0 0x7f3a4c19e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:06:51.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:51.091+0000 7f3a41400700 1 --2- 172.21.15.28:0/1586252187 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a4c102e60 0x7f3a4c19ec10 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f3a3400b2e0 tx=0x7f3a340047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:06:51.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:51.092+0000 7f3a43400700 1 -- 172.21.15.28:0/1586252187 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a34026070 con 0x7f3a4c102e60 2024-05-12T22:06:51.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:51.092+0000 7f3a4b200700 1 -- 172.21.15.28:0/1586252187 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a4c199310 con 0x7f3a4c102e60 2024-05-12T22:06:51.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:51.092+0000 7f3a43400700 1 -- 172.21.15.28:0/1586252187 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a3400fde0 con 0x7f3a4c102e60 2024-05-12T22:06:51.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:51.092+0000 7f3a4b200700 1 -- 172.21.15.28:0/1586252187 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a4c1a5530 con 0x7f3a4c102e60 2024-05-12T22:06:51.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:51.092+0000 7f3a43400700 1 -- 172.21.15.28:0/1586252187 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a340173f0 con 0x7f3a4c102e60 2024-05-12T22:06:51.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:51.093+0000 7f3a4b200700 1 -- 172.21.15.28:0/1586252187 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a4c103990 con 0x7f3a4c102e60 2024-05-12T22:06:51.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:51.095+0000 7f3a43400700 1 -- 172.21.15.28:0/1586252187 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f3a34029070 con 0x7f3a4c102e60 2024-05-12T22:06:51.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:51.097+0000 7f3a43400700 1 --2- 172.21.15.28:0/1586252187 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f3a300a2cb0 0x7f3a300a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:51.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:51.097+0000 7f3a43400700 1 -- 172.21.15.28:0/1586252187 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f3a340cfde0 con 0x7f3a4c102e60 2024-05-12T22:06:51.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:51.097+0000 7f3a49e00700 1 --2- 172.21.15.28:0/1586252187 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f3a300a2cb0 0x7f3a300a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:51.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:51.098+0000 7f3a49e00700 1 --2- 172.21.15.28:0/1586252187 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f3a300a2cb0 0x7f3a300a5170 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f3a3c009680 tx=0x7f3a3c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:06:51.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:51.102+0000 7f3a43400700 1 -- 172.21.15.28:0/1586252187 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f3a34013070 con 0x7f3a4c102e60 2024-05-12T22:06:51.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:51.398+0000 7f3a4b200700 1 -- 172.21.15.28:0/1586252187 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f3a4c02db80 con 0x7f3a4c102e60 2024-05-12T22:06:51.398 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:51 smithi028 ceph-mon[29847]: pgmap v320: 97 pgs: 97 active+clean; 632 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.5 KiB/s wr, 14 op/s; 72 B/s, 0 objects/s recovering 2024-05-12T22:06:51.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:51.408+0000 7f3a43400700 1 -- 172.21.15.28:0/1586252187 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) ==== 74+0+263200 (secure 0 0 0) 0x7f3a3409cc20 con 0x7f3a4c102e60 2024-05-12T22:06:51.412 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-12T22:06:51.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:51.413+0000 7f3a4b200700 1 -- 172.21.15.28:0/1586252187 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f3a300a2cb0 msgr2=0x7f3a300a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:51.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:51.413+0000 7f3a4b200700 1 --2- 172.21.15.28:0/1586252187 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f3a300a2cb0 0x7f3a300a5170 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f3a3c009680 tx=0x7f3a3c009260 comp rx=0 tx=0).stop 2024-05-12T22:06:51.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:51.413+0000 7f3a4b200700 1 -- 172.21.15.28:0/1586252187 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a4c102e60 msgr2=0x7f3a4c19ec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:51.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:51.413+0000 7f3a4b200700 1 --2- 172.21.15.28:0/1586252187 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a4c102e60 0x7f3a4c19ec10 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f3a3400b2e0 tx=0x7f3a340047b0 comp rx=0 tx=0).stop 2024-05-12T22:06:51.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:51.413+0000 7f3a4b200700 1 -- 172.21.15.28:0/1586252187 shutdown_connections 2024-05-12T22:06:51.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:51.413+0000 7f3a4b200700 1 --2- 172.21.15.28:0/1586252187 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f3a300a2cb0 0x7f3a300a5170 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:51.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:51.413+0000 7f3a4b200700 1 --2- 172.21.15.28:0/1586252187 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3a4c1024b0 0x7f3a4c19e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:51.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:51.413+0000 7f3a4b200700 1 --2- 172.21.15.28:0/1586252187 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3a4c107590 0x7f3a4c1988a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:51.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:51.413+0000 7f3a4b200700 1 --2- 172.21.15.28:0/1586252187 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a4c102e60 0x7f3a4c19ec10 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:51.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:51.413+0000 7f3a4b200700 1 -- 172.21.15.28:0/1586252187 >> 172.21.15.28:0/1586252187 conn(0x7f3a4c0fdda0 msgr2=0x7f3a4c100140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:06:51.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:51.414+0000 7f3a4b200700 1 -- 172.21.15.28:0/1586252187 shutdown_connections 2024-05-12T22:06:51.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:51.414+0000 7f3a4b200700 1 -- 172.21.15.28:0/1586252187 wait complete. 2024-05-12T22:06:51.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:51 smithi186 ceph-mon[34785]: pgmap v320: 97 pgs: 97 active+clean; 632 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.5 KiB/s wr, 14 op/s; 72 B/s, 0 objects/s recovering 2024-05-12T22:06:51.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:51 smithi086 ceph-mon[34297]: pgmap v320: 97 pgs: 97 active+clean; 632 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.5 KiB/s wr, 14 op/s; 72 B/s, 0 objects/s recovering 2024-05-12T22:06:52.291 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":17,"flags":0,"active_gid":14152,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3123328651},{"type":"v1","addr":"172.21.15.28:6801","nonce":3123328651}]},"active_addr":"172.21.15.28:6801/3123328651","active_change":"2024-05-12T21:58:45.550689+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24112,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24116,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.28:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.28:0","nonce":820158394}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.28:0","nonce":2580087728}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.28:0","nonce":1978033727}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.28:0","nonce":1581755795}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.28:0","nonce":1285119774}]}]} 2024-05-12T22:06:52.296 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-05-12T22:06:52.296 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-05-12T22:06:52.297 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f72fecff68e1d400c4568684327c900485c20d6a shell --fsid 6b92e7b4-10aa-11ef-bc98-c7b262605968 -- ceph osd dump --format=json 2024-05-12T22:06:52.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:52 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/1586252187' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-05-12T22:06:52.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:52 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/1586252187' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-05-12T22:06:52.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:52 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/1586252187' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-05-12T22:06:52.774 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/mon.a/config 2024-05-12T22:06:53.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:53 smithi028 ceph-mon[29847]: pgmap v321: 97 pgs: 97 active+clean; 632 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 7.7 KiB/s wr, 10 op/s; 72 B/s, 0 objects/s recovering 2024-05-12T22:06:53.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:53 smithi186 ceph-mon[34785]: pgmap v321: 97 pgs: 97 active+clean; 632 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 7.7 KiB/s wr, 10 op/s; 72 B/s, 0 objects/s recovering 2024-05-12T22:06:53.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:53 smithi086 ceph-mon[34297]: pgmap v321: 97 pgs: 97 active+clean; 632 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 7.7 KiB/s wr, 10 op/s; 72 B/s, 0 objects/s recovering 2024-05-12T22:06:54.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:54.819+0000 7fbd13200700 1 -- 172.21.15.28:0/1466457169 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbd14104e10 msgr2=0x7fbd14111320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:54.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:54.819+0000 7fbd13200700 1 --2- 172.21.15.28:0/1466457169 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbd14104e10 0x7fbd14111320 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fbd0c009f90 tx=0x7fbd0c00b330 comp rx=0 tx=0).stop 2024-05-12T22:06:54.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:54.819+0000 7fbd13200700 1 -- 172.21.15.28:0/1466457169 shutdown_connections 2024-05-12T22:06:54.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:54.819+0000 7fbd13200700 1 --2- 172.21.15.28:0/1466457169 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbd14104e10 0x7fbd14111320 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:54.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:54.819+0000 7fbd13200700 1 --2- 172.21.15.28:0/1466457169 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd1410a4e0 0x7fbd1410a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:54.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:54.819+0000 7fbd13200700 1 --2- 172.21.15.28:0/1466457169 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd14104450 0x7fbd141048d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:54.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:54.819+0000 7fbd13200700 1 -- 172.21.15.28:0/1466457169 >> 172.21.15.28:0/1466457169 conn(0x7fbd140fff40 msgr2=0x7fbd14102360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:06:54.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:54.820+0000 7fbd13200700 1 -- 172.21.15.28:0/1466457169 shutdown_connections 2024-05-12T22:06:54.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:54.820+0000 7fbd13200700 1 -- 172.21.15.28:0/1466457169 wait complete. 2024-05-12T22:06:54.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:54.821+0000 7fbd13200700 1 Processor -- start 2024-05-12T22:06:54.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:54.821+0000 7fbd13200700 1 -- start start 2024-05-12T22:06:54.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:54.821+0000 7fbd13200700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd14104450 0x7fbd141a0770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:54.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:54.821+0000 7fbd13200700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd14104e10 0x7fbd141a0cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:54.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:54.821+0000 7fbd13200700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbd1410a4e0 0x7fbd1419a980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:54.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:54.821+0000 7fbd13200700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbd14114740 con 0x7fbd14104e10 2024-05-12T22:06:54.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:54.821+0000 7fbd13200700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fbd141145c0 con 0x7fbd14104450 2024-05-12T22:06:54.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:54.821+0000 7fbd13200700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fbd141148c0 con 0x7fbd1410a4e0 2024-05-12T22:06:54.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:54.822+0000 7fbd11e00700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd14104450 0x7fbd141a0770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:54.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:54.822+0000 7fbd11400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd14104e10 0x7fbd141a0cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:54.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:54.822+0000 7fbd11e00700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd14104450 0x7fbd141a0770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:42570/0 (socket says 172.21.15.28:42570) 2024-05-12T22:06:54.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:54.822+0000 7fbd11e00700 1 -- 172.21.15.28:0/3007862934 learned_addr learned my addr 172.21.15.28:0/3007862934 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:06:54.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:54.822+0000 7fbd12800700 1 --2- 172.21.15.28:0/3007862934 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbd1410a4e0 0x7fbd1419a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:54.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:54.822+0000 7fbd11e00700 1 -- 172.21.15.28:0/3007862934 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbd1410a4e0 msgr2=0x7fbd1419a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:54.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:54.822+0000 7fbd11e00700 1 --2- 172.21.15.28:0/3007862934 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbd1410a4e0 0x7fbd1419a980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:54.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:54.822+0000 7fbd11e00700 1 -- 172.21.15.28:0/3007862934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd14104e10 msgr2=0x7fbd141a0cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:54.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:54.822+0000 7fbd11e00700 1 --2- 172.21.15.28:0/3007862934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd14104e10 0x7fbd141a0cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:54.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:54.822+0000 7fbd11e00700 1 -- 172.21.15.28:0/3007862934 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd1419b0e0 con 0x7fbd14104450 2024-05-12T22:06:54.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:54.822+0000 7fbd11e00700 1 --2- 172.21.15.28:0/3007862934 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd14104450 0x7fbd141a0770 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fbd0400e9c0 tx=0x7fbd0400ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:06:54.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:54.823+0000 7fbd0aa00700 1 -- 172.21.15.28:0/3007862934 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd04004a30 con 0x7fbd14104450 2024-05-12T22:06:54.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:54.823+0000 7fbd0aa00700 1 -- 172.21.15.28:0/3007862934 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbd0400cca0 con 0x7fbd14104450 2024-05-12T22:06:54.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:54.823+0000 7fbd13200700 1 -- 172.21.15.28:0/3007862934 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd1419b3d0 con 0x7fbd14104450 2024-05-12T22:06:54.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:54.823+0000 7fbd0aa00700 1 -- 172.21.15.28:0/3007862934 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd04005610 con 0x7fbd14104450 2024-05-12T22:06:54.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:54.823+0000 7fbd13200700 1 -- 172.21.15.28:0/3007862934 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd141a7640 con 0x7fbd14104450 2024-05-12T22:06:54.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:54.824+0000 7fbd13200700 1 -- 172.21.15.28:0/3007862934 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd14069010 con 0x7fbd14104450 2024-05-12T22:06:54.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:54.826+0000 7fbd0aa00700 1 -- 172.21.15.28:0/3007862934 <== mon.2 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fbd04004cf0 con 0x7fbd14104450 2024-05-12T22:06:54.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:54.829+0000 7fbd0aa00700 1 --2- 172.21.15.28:0/3007862934 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fbcf80a2cb0 0x7fbcf80a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:54.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:54.829+0000 7fbd11400700 1 --2- 172.21.15.28:0/3007862934 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fbcf80a2cb0 0x7fbcf80a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:54.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:54.829+0000 7fbd0aa00700 1 -- 172.21.15.28:0/3007862934 <== mon.2 v2:172.21.15.86:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7fbd040cf2a0 con 0x7fbd14104450 2024-05-12T22:06:54.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:54.830+0000 7fbd11400700 1 --2- 172.21.15.28:0/3007862934 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fbcf80a2cb0 0x7fbcf80a5170 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fbd141099c0 tx=0x7fbd000073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:06:54.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:54.831+0000 7fbd0aa00700 1 -- 172.21.15.28:0/3007862934 <== mon.2 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fbd0409c050 con 0x7fbd14104450 2024-05-12T22:06:55.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:55.064+0000 7fbd13200700 1 -- 172.21.15.28:0/3007862934 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbd1419be60 con 0x7fbd14104450 2024-05-12T22:06:55.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:55.066+0000 7fbd0aa00700 1 -- 172.21.15.28:0/3007862934 <== mon.2 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22595 (secure 0 0 0) 0x7fbd0409b950 con 0x7fbd14104450 2024-05-12T22:06:55.066 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-12T22:06:55.067 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":88,"fsid":"6b92e7b4-10aa-11ef-bc98-c7b262605968","created":"2024-05-12T21:57:37.276782+0000","modified":"2024-05-12T22:06:46.768798+0000","last_up_change":"2024-05-12T22:05:37.929562+0000","last_in_change":"2024-05-12T22:05:18.750768+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-12T22:01:32.292826+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-12T22:06:31.540263+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-12T22:06:32.906118+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6992ce0a-844b-40e3-8ed1-181c8474dccf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":906003882},{"type":"v1","addr":"172.21.15.28:6803","nonce":906003882}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":906003882},{"type":"v1","addr":"172.21.15.28:6805","nonce":906003882}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":906003882},{"type":"v1","addr":"172.21.15.28:6809","nonce":906003882}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":906003882},{"type":"v1","addr":"172.21.15.28:6807","nonce":906003882}]},"public_addr":"172.21.15.28:6803/906003882","cluster_addr":"172.21.15.28:6805/906003882","heartbeat_back_addr":"172.21.15.28:6809/906003882","heartbeat_front_addr":"172.21.15.28:6807/906003882","state":["exists","up"]},{"osd":1,"uuid":"ced20c96-c481-4d48-90d1-6b43228959ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":3354141819},{"type":"v1","addr":"172.21.15.86:6801","nonce":3354141819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":3354141819},{"type":"v1","addr":"172.21.15.86:6803","nonce":3354141819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":3354141819},{"type":"v1","addr":"172.21.15.86:6807","nonce":3354141819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":3354141819},{"type":"v1","addr":"172.21.15.86:6805","nonce":3354141819}]},"public_addr":"172.21.15.86:6801/3354141819","cluster_addr":"172.21.15.86:6803/3354141819","heartbeat_back_addr":"172.21.15.86:6807/3354141819","heartbeat_front_addr":"172.21.15.86:6805/3354141819","state":["exists","up"]},{"osd":2,"uuid":"54a2f221-fa65-4795-9f5e-e660cc7c0098","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":906459365},{"type":"v1","addr":"172.21.15.186:6801","nonce":906459365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":906459365},{"type":"v1","addr":"172.21.15.186:6803","nonce":906459365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":906459365},{"type":"v1","addr":"172.21.15.186:6807","nonce":906459365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":906459365},{"type":"v1","addr":"172.21.15.186:6805","nonce":906459365}]},"public_addr":"172.21.15.186:6801/906459365","cluster_addr":"172.21.15.186:6803/906459365","heartbeat_back_addr":"172.21.15.186:6807/906459365","heartbeat_front_addr":"172.21.15.186:6805/906459365","state":["exists","up"]},{"osd":3,"uuid":"86836737-ac9a-487b-93f2-7fcd3c406960","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":4182884436},{"type":"v1","addr":"172.21.15.28:6811","nonce":4182884436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":4182884436},{"type":"v1","addr":"172.21.15.28:6813","nonce":4182884436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":4182884436},{"type":"v1","addr":"172.21.15.28:6817","nonce":4182884436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":4182884436},{"type":"v1","addr":"172.21.15.28:6815","nonce":4182884436}]},"public_addr":"172.21.15.28:6811/4182884436","cluster_addr":"172.21.15.28:6813/4182884436","heartbeat_back_addr":"172.21.15.28:6817/4182884436","heartbeat_front_addr":"172.21.15.28:6815/4182884436","state":["exists","up"]},{"osd":4,"uuid":"d162ebcd-83c0-4fd5-9925-28b2cecea935","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2596998708},{"type":"v1","addr":"172.21.15.86:6809","nonce":2596998708}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2596998708},{"type":"v1","addr":"172.21.15.86:6811","nonce":2596998708}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2596998708},{"type":"v1","addr":"172.21.15.86:6815","nonce":2596998708}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2596998708},{"type":"v1","addr":"172.21.15.86:6813","nonce":2596998708}]},"public_addr":"172.21.15.86:6809/2596998708","cluster_addr":"172.21.15.86:6811/2596998708","heartbeat_back_addr":"172.21.15.86:6815/2596998708","heartbeat_front_addr":"172.21.15.86:6813/2596998708","state":["exists","up"]},{"osd":5,"uuid":"c794336a-a105-425c-b6c7-a32b90c05bb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":1944205756},{"type":"v1","addr":"172.21.15.186:6809","nonce":1944205756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":1944205756},{"type":"v1","addr":"172.21.15.186:6811","nonce":1944205756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":1944205756},{"type":"v1","addr":"172.21.15.186:6815","nonce":1944205756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":1944205756},{"type":"v1","addr":"172.21.15.186:6813","nonce":1944205756}]},"public_addr":"172.21.15.186:6809/1944205756","cluster_addr":"172.21.15.186:6811/1944205756","heartbeat_back_addr":"172.21.15.186:6815/1944205756","heartbeat_front_addr":"172.21.15.186:6813/1944205756","state":["exists","up"]},{"osd":6,"uuid":"516932ac-af00-45b7-ad3c-d6dec7db442f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":1642308925},{"type":"v1","addr":"172.21.15.28:6819","nonce":1642308925}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":1642308925},{"type":"v1","addr":"172.21.15.28:6821","nonce":1642308925}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1642308925},{"type":"v1","addr":"172.21.15.28:6825","nonce":1642308925}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":1642308925},{"type":"v1","addr":"172.21.15.28:6823","nonce":1642308925}]},"public_addr":"172.21.15.28:6819/1642308925","cluster_addr":"172.21.15.28:6821/1642308925","heartbeat_back_addr":"172.21.15.28:6825/1642308925","heartbeat_front_addr":"172.21.15.28:6823/1642308925","state":["exists","up"]},{"osd":7,"uuid":"ca7a059f-0414-438e-b2d1-a10f348036be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":726062711},{"type":"v1","addr":"172.21.15.86:6817","nonce":726062711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":726062711},{"type":"v1","addr":"172.21.15.86:6819","nonce":726062711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":726062711},{"type":"v1","addr":"172.21.15.86:6823","nonce":726062711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":726062711},{"type":"v1","addr":"172.21.15.86:6821","nonce":726062711}]},"public_addr":"172.21.15.86:6817/726062711","cluster_addr":"172.21.15.86:6819/726062711","heartbeat_back_addr":"172.21.15.86:6823/726062711","heartbeat_front_addr":"172.21.15.86:6821/726062711","state":["exists","up"]},{"osd":8,"uuid":"92e3f41f-aebd-422e-a99e-555f7ba4aa53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":3007687277},{"type":"v1","addr":"172.21.15.186:6817","nonce":3007687277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":3007687277},{"type":"v1","addr":"172.21.15.186:6819","nonce":3007687277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":3007687277},{"type":"v1","addr":"172.21.15.186:6823","nonce":3007687277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":3007687277},{"type":"v1","addr":"172.21.15.186:6821","nonce":3007687277}]},"public_addr":"172.21.15.186:6817/3007687277","cluster_addr":"172.21.15.186:6819/3007687277","heartbeat_back_addr":"172.21.15.186:6823/3007687277","heartbeat_front_addr":"172.21.15.186:6821/3007687277","state":["exists","up"]},{"osd":9,"uuid":"38ff7d9d-ee85-404c-8c59-adc327e1ccfa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":136220472},{"type":"v1","addr":"172.21.15.28:6827","nonce":136220472}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":136220472},{"type":"v1","addr":"172.21.15.28:6829","nonce":136220472}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":136220472},{"type":"v1","addr":"172.21.15.28:6833","nonce":136220472}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":136220472},{"type":"v1","addr":"172.21.15.28:6831","nonce":136220472}]},"public_addr":"172.21.15.28:6827/136220472","cluster_addr":"172.21.15.28:6829/136220472","heartbeat_back_addr":"172.21.15.28:6833/136220472","heartbeat_front_addr":"172.21.15.28:6831/136220472","state":["exists","up"]},{"osd":10,"uuid":"5e9ff657-afa8-4bab-8491-f209038392b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1189968310},{"type":"v1","addr":"172.21.15.86:6825","nonce":1189968310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":1189968310},{"type":"v1","addr":"172.21.15.86:6827","nonce":1189968310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":1189968310},{"type":"v1","addr":"172.21.15.86:6831","nonce":1189968310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":1189968310},{"type":"v1","addr":"172.21.15.86:6829","nonce":1189968310}]},"public_addr":"172.21.15.86:6825/1189968310","cluster_addr":"172.21.15.86:6827/1189968310","heartbeat_back_addr":"172.21.15.86:6831/1189968310","heartbeat_front_addr":"172.21.15.86:6829/1189968310","state":["exists","up"]},{"osd":11,"uuid":"f112e3c6-65b6-4253-a966-62001db7d069","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3801668110},{"type":"v1","addr":"172.21.15.186:6825","nonce":3801668110}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3801668110},{"type":"v1","addr":"172.21.15.186:6827","nonce":3801668110}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3801668110},{"type":"v1","addr":"172.21.15.186:6831","nonce":3801668110}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3801668110},{"type":"v1","addr":"172.21.15.186:6829","nonce":3801668110}]},"public_addr":"172.21.15.186:6825/3801668110","cluster_addr":"172.21.15.186:6827/3801668110","heartbeat_back_addr":"172.21.15.186:6831/3801668110","heartbeat_front_addr":"172.21.15.186:6829/3801668110","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:00:40.483623+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:01:05.284385+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:01:28.645716+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:02:01.295092+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:02:26.486601+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:02:50.807478+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:03:24.756169+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:03:49.975173+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:04:13.772233+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:04:46.758555+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:05:12.331685+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:05:36.462548+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":2,"to":1}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.25","mappings":[{"from":5,"to":9}]},{"pgid":"3.27","mappings":[{"from":4,"to":8}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/1177009533":"2024-05-13T21:58:45.550459+0000","172.21.15.28:6800/3760499540":"2024-05-13T21:58:45.550459+0000","172.21.15.28:6801/3783742728":"2024-05-13T21:58:10.421564+0000","172.21.15.28:6800/3783742728":"2024-05-13T21:58:10.421564+0000","172.21.15.28:0/2229156260":"2024-05-13T21:58:45.550459+0000","172.21.15.28:0/1202140963":"2024-05-13T21:58:10.421564+0000","172.21.15.28:0/2924223784":"2024-05-13T21:58:45.550459+0000","172.21.15.28:6801/3760499540":"2024-05-13T21:58:45.550459+0000","172.21.15.28:0/1240027206":"2024-05-13T21:58:10.421564+0000","172.21.15.28:0/171417728":"2024-05-13T21:58:10.421564+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-12T22:06:55.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:55.069+0000 7fbd13200700 1 -- 172.21.15.28:0/3007862934 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fbcf80a2cb0 msgr2=0x7fbcf80a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:55.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:55.069+0000 7fbd13200700 1 --2- 172.21.15.28:0/3007862934 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fbcf80a2cb0 0x7fbcf80a5170 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fbd141099c0 tx=0x7fbd000073d0 comp rx=0 tx=0).stop 2024-05-12T22:06:55.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:55.069+0000 7fbd13200700 1 -- 172.21.15.28:0/3007862934 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd14104450 msgr2=0x7fbd141a0770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:55.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:55.069+0000 7fbd13200700 1 --2- 172.21.15.28:0/3007862934 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd14104450 0x7fbd141a0770 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fbd0400e9c0 tx=0x7fbd0400ecd0 comp rx=0 tx=0).stop 2024-05-12T22:06:55.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:55.069+0000 7fbd13200700 1 -- 172.21.15.28:0/3007862934 shutdown_connections 2024-05-12T22:06:55.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:55.069+0000 7fbd13200700 1 --2- 172.21.15.28:0/3007862934 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fbcf80a2cb0 0x7fbcf80a5170 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:55.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:55.069+0000 7fbd13200700 1 --2- 172.21.15.28:0/3007862934 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbd1410a4e0 0x7fbd1419a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:55.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:55.069+0000 7fbd13200700 1 --2- 172.21.15.28:0/3007862934 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd14104450 0x7fbd141a0770 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:55.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:55.069+0000 7fbd13200700 1 --2- 172.21.15.28:0/3007862934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd14104e10 0x7fbd141a0cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:55.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:55.069+0000 7fbd13200700 1 -- 172.21.15.28:0/3007862934 >> 172.21.15.28:0/3007862934 conn(0x7fbd140fff40 msgr2=0x7fbd14100460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:06:55.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:55.070+0000 7fbd13200700 1 -- 172.21.15.28:0/3007862934 shutdown_connections 2024-05-12T22:06:55.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:55.070+0000 7fbd13200700 1 -- 172.21.15.28:0/3007862934 wait complete. 2024-05-12T22:06:55.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:55 smithi186 ceph-mon[34785]: pgmap v322: 97 pgs: 97 active+clean; 632 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.4 KiB/s wr, 6 op/s; 72 B/s, 0 objects/s recovering 2024-05-12T22:06:55.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:55 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/3007862934' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-12T22:06:55.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:55 smithi086 ceph-mon[34297]: pgmap v322: 97 pgs: 97 active+clean; 632 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.4 KiB/s wr, 6 op/s; 72 B/s, 0 objects/s recovering 2024-05-12T22:06:55.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:55 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/3007862934' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-12T22:06:55.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:55 smithi028 ceph-mon[29847]: pgmap v322: 97 pgs: 97 active+clean; 632 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.4 KiB/s wr, 6 op/s; 72 B/s, 0 objects/s recovering 2024-05-12T22:06:55.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:55 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/3007862934' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-12T22:06:56.089 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-05-12T22:06:56.090 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f72fecff68e1d400c4568684327c900485c20d6a shell --fsid 6b92e7b4-10aa-11ef-bc98-c7b262605968 -- ceph osd dump --format=json 2024-05-12T22:06:56.566 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/mon.a/config 2024-05-12T22:06:57.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:57 smithi186 ceph-mon[34785]: pgmap v323: 97 pgs: 97 active+clean; 636 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 1.7 KiB/s wr, 7 op/s; 58 B/s, 0 objects/s recovering 2024-05-12T22:06:57.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:57 smithi086 ceph-mon[34297]: pgmap v323: 97 pgs: 97 active+clean; 636 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 1.7 KiB/s wr, 7 op/s; 58 B/s, 0 objects/s recovering 2024-05-12T22:06:57.743 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:57 smithi028 ceph-mon[29847]: pgmap v323: 97 pgs: 97 active+clean; 636 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 1.7 KiB/s wr, 7 op/s; 58 B/s, 0 objects/s recovering 2024-05-12T22:06:58.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:58.703+0000 7fa8a9200700 1 -- 172.21.15.28:0/3522770770 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa8a410d730 msgr2=0x7fa8a410db10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:58.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:58.703+0000 7fa8a9200700 1 --2- 172.21.15.28:0/3522770770 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa8a410d730 0x7fa8a410db10 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fa8940098f0 tx=0x7fa894009c00 comp rx=0 tx=0).stop 2024-05-12T22:06:58.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:58.703+0000 7fa8a9200700 1 -- 172.21.15.28:0/3522770770 shutdown_connections 2024-05-12T22:06:58.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:58.703+0000 7fa8a9200700 1 --2- 172.21.15.28:0/3522770770 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa8a4109220 0x7fa8a41156c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:58.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:58.703+0000 7fa8a9200700 1 --2- 172.21.15.28:0/3522770770 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa8a410d730 0x7fa8a410db10 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:58.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:58.703+0000 7fa8a9200700 1 --2- 172.21.15.28:0/3522770770 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa8a4108860 0x7fa8a4108ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:58.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:58.703+0000 7fa8a9200700 1 -- 172.21.15.28:0/3522770770 >> 172.21.15.28:0/3522770770 conn(0x7fa8a40780b0 msgr2=0x7fa8a4107140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:06:58.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:58.703+0000 7fa8a9200700 1 -- 172.21.15.28:0/3522770770 shutdown_connections 2024-05-12T22:06:58.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:58.703+0000 7fa8a9200700 1 -- 172.21.15.28:0/3522770770 wait complete. 2024-05-12T22:06:58.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:58.704+0000 7fa8a9200700 1 Processor -- start 2024-05-12T22:06:58.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:58.705+0000 7fa8a9200700 1 -- start start 2024-05-12T22:06:58.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:58.705+0000 7fa8a9200700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa8a4108860 0x7fa8a419f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:58.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:58.705+0000 7fa8a9200700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa8a4109220 0x7fa8a419f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:58.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:58.705+0000 7fa8a9200700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa8a410d730 0x7fa8a41a3900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:58.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:58.705+0000 7fa8a9200700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa8a4118b90 con 0x7fa8a410d730 2024-05-12T22:06:58.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:58.705+0000 7fa8a9200700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fa8a4118a10 con 0x7fa8a4109220 2024-05-12T22:06:58.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:58.705+0000 7fa8a9200700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa8a4118d10 con 0x7fa8a4108860 2024-05-12T22:06:58.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:58.705+0000 7fa8a3e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa8a410d730 0x7fa8a41a3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:58.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:58.705+0000 7fa89be00700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa8a4109220 0x7fa8a419f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:58.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:58.705+0000 7fa8a3e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa8a410d730 0x7fa8a41a3900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51812/0 (socket says 172.21.15.28:51812) 2024-05-12T22:06:58.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:58.705+0000 7fa8a3e00700 1 -- 172.21.15.28:0/2376450041 learned_addr learned my addr 172.21.15.28:0/2376450041 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:06:58.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:58.705+0000 7fa89be00700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa8a4109220 0x7fa8a419f5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:36844/0 (socket says 172.21.15.28:36844) 2024-05-12T22:06:58.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:58.706+0000 7fa8a3400700 1 --2- 172.21.15.28:0/2376450041 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa8a4108860 0x7fa8a419f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:58.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:58.706+0000 7fa89be00700 1 -- 172.21.15.28:0/2376450041 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa8a4108860 msgr2=0x7fa8a419f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:58.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:58.706+0000 7fa89be00700 1 --2- 172.21.15.28:0/2376450041 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa8a4108860 0x7fa8a419f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:58.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:58.706+0000 7fa89be00700 1 -- 172.21.15.28:0/2376450041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa8a410d730 msgr2=0x7fa8a41a3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:58.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:58.706+0000 7fa89be00700 1 --2- 172.21.15.28:0/2376450041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa8a410d730 0x7fa8a41a3900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:58.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:58.706+0000 7fa89be00700 1 -- 172.21.15.28:0/2376450041 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8a41a4040 con 0x7fa8a4109220 2024-05-12T22:06:58.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:58.706+0000 7fa8a3e00700 1 --2- 172.21.15.28:0/2376450041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa8a410d730 0x7fa8a41a3900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:06:58.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:58.706+0000 7fa89be00700 1 --2- 172.21.15.28:0/2376450041 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa8a4109220 0x7fa8a419f5c0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fa88c00eb90 tx=0x7fa88c00eea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:06:58.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:58.707+0000 7fa8a0c00700 1 -- 172.21.15.28:0/2376450041 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa88c00cc90 con 0x7fa8a4109220 2024-05-12T22:06:58.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:58.707+0000 7fa8a0c00700 1 -- 172.21.15.28:0/2376450041 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa88c00ce30 con 0x7fa8a4109220 2024-05-12T22:06:58.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:58.707+0000 7fa8a9200700 1 -- 172.21.15.28:0/2376450041 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8a41a4330 con 0x7fa8a4109220 2024-05-12T22:06:58.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:58.707+0000 7fa8a0c00700 1 -- 172.21.15.28:0/2376450041 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa88c010430 con 0x7fa8a4109220 2024-05-12T22:06:58.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:58.707+0000 7fa8a9200700 1 -- 172.21.15.28:0/2376450041 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8a41abbb0 con 0x7fa8a4109220 2024-05-12T22:06:58.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:58.708+0000 7fa8a9200700 1 -- 172.21.15.28:0/2376450041 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa8a4069010 con 0x7fa8a4109220 2024-05-12T22:06:58.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:58.710+0000 7fa8a0c00700 1 -- 172.21.15.28:0/2376450041 <== mon.2 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fa88c003680 con 0x7fa8a4109220 2024-05-12T22:06:58.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:58.711+0000 7fa8a0c00700 1 --2- 172.21.15.28:0/2376450041 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fa8880a2d30 0x7fa8880a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:06:58.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:58.711+0000 7fa8a0c00700 1 -- 172.21.15.28:0/2376450041 <== mon.2 v2:172.21.15.86:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7fa88c0cf460 con 0x7fa8a4109220 2024-05-12T22:06:58.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:58.711+0000 7fa8a3400700 1 --2- 172.21.15.28:0/2376450041 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fa8880a2d30 0x7fa8880a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:06:58.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:58.712+0000 7fa8a3400700 1 --2- 172.21.15.28:0/2376450041 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fa8880a2d30 0x7fa8880a51f0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fa894009810 tx=0x7fa894019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:06:58.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:58.716+0000 7fa8a0c00700 1 -- 172.21.15.28:0/2376450041 <== mon.2 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fa88c09c210 con 0x7fa8a4109220 2024-05-12T22:06:58.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:58.948+0000 7fa8a9200700 1 -- 172.21.15.28:0/2376450041 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa8a41a0390 con 0x7fa8a4109220 2024-05-12T22:06:58.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:58.950+0000 7fa8a0c00700 1 -- 172.21.15.28:0/2376450041 <== mon.2 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22595 (secure 0 0 0) 0x7fa88c09bb10 con 0x7fa8a4109220 2024-05-12T22:06:58.951 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-12T22:06:58.952 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":88,"fsid":"6b92e7b4-10aa-11ef-bc98-c7b262605968","created":"2024-05-12T21:57:37.276782+0000","modified":"2024-05-12T22:06:46.768798+0000","last_up_change":"2024-05-12T22:05:37.929562+0000","last_in_change":"2024-05-12T22:05:18.750768+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-12T22:01:32.292826+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-12T22:06:31.540263+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-12T22:06:32.906118+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6992ce0a-844b-40e3-8ed1-181c8474dccf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":906003882},{"type":"v1","addr":"172.21.15.28:6803","nonce":906003882}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":906003882},{"type":"v1","addr":"172.21.15.28:6805","nonce":906003882}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":906003882},{"type":"v1","addr":"172.21.15.28:6809","nonce":906003882}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":906003882},{"type":"v1","addr":"172.21.15.28:6807","nonce":906003882}]},"public_addr":"172.21.15.28:6803/906003882","cluster_addr":"172.21.15.28:6805/906003882","heartbeat_back_addr":"172.21.15.28:6809/906003882","heartbeat_front_addr":"172.21.15.28:6807/906003882","state":["exists","up"]},{"osd":1,"uuid":"ced20c96-c481-4d48-90d1-6b43228959ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":3354141819},{"type":"v1","addr":"172.21.15.86:6801","nonce":3354141819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":3354141819},{"type":"v1","addr":"172.21.15.86:6803","nonce":3354141819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":3354141819},{"type":"v1","addr":"172.21.15.86:6807","nonce":3354141819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":3354141819},{"type":"v1","addr":"172.21.15.86:6805","nonce":3354141819}]},"public_addr":"172.21.15.86:6801/3354141819","cluster_addr":"172.21.15.86:6803/3354141819","heartbeat_back_addr":"172.21.15.86:6807/3354141819","heartbeat_front_addr":"172.21.15.86:6805/3354141819","state":["exists","up"]},{"osd":2,"uuid":"54a2f221-fa65-4795-9f5e-e660cc7c0098","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":906459365},{"type":"v1","addr":"172.21.15.186:6801","nonce":906459365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":906459365},{"type":"v1","addr":"172.21.15.186:6803","nonce":906459365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":906459365},{"type":"v1","addr":"172.21.15.186:6807","nonce":906459365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":906459365},{"type":"v1","addr":"172.21.15.186:6805","nonce":906459365}]},"public_addr":"172.21.15.186:6801/906459365","cluster_addr":"172.21.15.186:6803/906459365","heartbeat_back_addr":"172.21.15.186:6807/906459365","heartbeat_front_addr":"172.21.15.186:6805/906459365","state":["exists","up"]},{"osd":3,"uuid":"86836737-ac9a-487b-93f2-7fcd3c406960","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":4182884436},{"type":"v1","addr":"172.21.15.28:6811","nonce":4182884436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":4182884436},{"type":"v1","addr":"172.21.15.28:6813","nonce":4182884436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":4182884436},{"type":"v1","addr":"172.21.15.28:6817","nonce":4182884436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":4182884436},{"type":"v1","addr":"172.21.15.28:6815","nonce":4182884436}]},"public_addr":"172.21.15.28:6811/4182884436","cluster_addr":"172.21.15.28:6813/4182884436","heartbeat_back_addr":"172.21.15.28:6817/4182884436","heartbeat_front_addr":"172.21.15.28:6815/4182884436","state":["exists","up"]},{"osd":4,"uuid":"d162ebcd-83c0-4fd5-9925-28b2cecea935","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2596998708},{"type":"v1","addr":"172.21.15.86:6809","nonce":2596998708}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2596998708},{"type":"v1","addr":"172.21.15.86:6811","nonce":2596998708}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2596998708},{"type":"v1","addr":"172.21.15.86:6815","nonce":2596998708}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2596998708},{"type":"v1","addr":"172.21.15.86:6813","nonce":2596998708}]},"public_addr":"172.21.15.86:6809/2596998708","cluster_addr":"172.21.15.86:6811/2596998708","heartbeat_back_addr":"172.21.15.86:6815/2596998708","heartbeat_front_addr":"172.21.15.86:6813/2596998708","state":["exists","up"]},{"osd":5,"uuid":"c794336a-a105-425c-b6c7-a32b90c05bb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":1944205756},{"type":"v1","addr":"172.21.15.186:6809","nonce":1944205756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":1944205756},{"type":"v1","addr":"172.21.15.186:6811","nonce":1944205756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":1944205756},{"type":"v1","addr":"172.21.15.186:6815","nonce":1944205756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":1944205756},{"type":"v1","addr":"172.21.15.186:6813","nonce":1944205756}]},"public_addr":"172.21.15.186:6809/1944205756","cluster_addr":"172.21.15.186:6811/1944205756","heartbeat_back_addr":"172.21.15.186:6815/1944205756","heartbeat_front_addr":"172.21.15.186:6813/1944205756","state":["exists","up"]},{"osd":6,"uuid":"516932ac-af00-45b7-ad3c-d6dec7db442f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":1642308925},{"type":"v1","addr":"172.21.15.28:6819","nonce":1642308925}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":1642308925},{"type":"v1","addr":"172.21.15.28:6821","nonce":1642308925}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1642308925},{"type":"v1","addr":"172.21.15.28:6825","nonce":1642308925}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":1642308925},{"type":"v1","addr":"172.21.15.28:6823","nonce":1642308925}]},"public_addr":"172.21.15.28:6819/1642308925","cluster_addr":"172.21.15.28:6821/1642308925","heartbeat_back_addr":"172.21.15.28:6825/1642308925","heartbeat_front_addr":"172.21.15.28:6823/1642308925","state":["exists","up"]},{"osd":7,"uuid":"ca7a059f-0414-438e-b2d1-a10f348036be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":726062711},{"type":"v1","addr":"172.21.15.86:6817","nonce":726062711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":726062711},{"type":"v1","addr":"172.21.15.86:6819","nonce":726062711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":726062711},{"type":"v1","addr":"172.21.15.86:6823","nonce":726062711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":726062711},{"type":"v1","addr":"172.21.15.86:6821","nonce":726062711}]},"public_addr":"172.21.15.86:6817/726062711","cluster_addr":"172.21.15.86:6819/726062711","heartbeat_back_addr":"172.21.15.86:6823/726062711","heartbeat_front_addr":"172.21.15.86:6821/726062711","state":["exists","up"]},{"osd":8,"uuid":"92e3f41f-aebd-422e-a99e-555f7ba4aa53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":3007687277},{"type":"v1","addr":"172.21.15.186:6817","nonce":3007687277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":3007687277},{"type":"v1","addr":"172.21.15.186:6819","nonce":3007687277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":3007687277},{"type":"v1","addr":"172.21.15.186:6823","nonce":3007687277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":3007687277},{"type":"v1","addr":"172.21.15.186:6821","nonce":3007687277}]},"public_addr":"172.21.15.186:6817/3007687277","cluster_addr":"172.21.15.186:6819/3007687277","heartbeat_back_addr":"172.21.15.186:6823/3007687277","heartbeat_front_addr":"172.21.15.186:6821/3007687277","state":["exists","up"]},{"osd":9,"uuid":"38ff7d9d-ee85-404c-8c59-adc327e1ccfa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":136220472},{"type":"v1","addr":"172.21.15.28:6827","nonce":136220472}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":136220472},{"type":"v1","addr":"172.21.15.28:6829","nonce":136220472}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":136220472},{"type":"v1","addr":"172.21.15.28:6833","nonce":136220472}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":136220472},{"type":"v1","addr":"172.21.15.28:6831","nonce":136220472}]},"public_addr":"172.21.15.28:6827/136220472","cluster_addr":"172.21.15.28:6829/136220472","heartbeat_back_addr":"172.21.15.28:6833/136220472","heartbeat_front_addr":"172.21.15.28:6831/136220472","state":["exists","up"]},{"osd":10,"uuid":"5e9ff657-afa8-4bab-8491-f209038392b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1189968310},{"type":"v1","addr":"172.21.15.86:6825","nonce":1189968310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":1189968310},{"type":"v1","addr":"172.21.15.86:6827","nonce":1189968310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":1189968310},{"type":"v1","addr":"172.21.15.86:6831","nonce":1189968310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":1189968310},{"type":"v1","addr":"172.21.15.86:6829","nonce":1189968310}]},"public_addr":"172.21.15.86:6825/1189968310","cluster_addr":"172.21.15.86:6827/1189968310","heartbeat_back_addr":"172.21.15.86:6831/1189968310","heartbeat_front_addr":"172.21.15.86:6829/1189968310","state":["exists","up"]},{"osd":11,"uuid":"f112e3c6-65b6-4253-a966-62001db7d069","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3801668110},{"type":"v1","addr":"172.21.15.186:6825","nonce":3801668110}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3801668110},{"type":"v1","addr":"172.21.15.186:6827","nonce":3801668110}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3801668110},{"type":"v1","addr":"172.21.15.186:6831","nonce":3801668110}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3801668110},{"type":"v1","addr":"172.21.15.186:6829","nonce":3801668110}]},"public_addr":"172.21.15.186:6825/3801668110","cluster_addr":"172.21.15.186:6827/3801668110","heartbeat_back_addr":"172.21.15.186:6831/3801668110","heartbeat_front_addr":"172.21.15.186:6829/3801668110","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:00:40.483623+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:01:05.284385+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:01:28.645716+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:02:01.295092+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:02:26.486601+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:02:50.807478+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:03:24.756169+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:03:49.975173+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:04:13.772233+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:04:46.758555+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:05:12.331685+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:05:36.462548+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":2,"to":1}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.25","mappings":[{"from":5,"to":9}]},{"pgid":"3.27","mappings":[{"from":4,"to":8}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/1177009533":"2024-05-13T21:58:45.550459+0000","172.21.15.28:6800/3760499540":"2024-05-13T21:58:45.550459+0000","172.21.15.28:6801/3783742728":"2024-05-13T21:58:10.421564+0000","172.21.15.28:6800/3783742728":"2024-05-13T21:58:10.421564+0000","172.21.15.28:0/2229156260":"2024-05-13T21:58:45.550459+0000","172.21.15.28:0/1202140963":"2024-05-13T21:58:10.421564+0000","172.21.15.28:0/2924223784":"2024-05-13T21:58:45.550459+0000","172.21.15.28:6801/3760499540":"2024-05-13T21:58:45.550459+0000","172.21.15.28:0/1240027206":"2024-05-13T21:58:10.421564+0000","172.21.15.28:0/171417728":"2024-05-13T21:58:10.421564+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-12T22:06:58.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:58.953+0000 7fa8a9200700 1 -- 172.21.15.28:0/2376450041 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fa8880a2d30 msgr2=0x7fa8880a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:58.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:58.954+0000 7fa8a9200700 1 --2- 172.21.15.28:0/2376450041 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fa8880a2d30 0x7fa8880a51f0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fa894009810 tx=0x7fa894019040 comp rx=0 tx=0).stop 2024-05-12T22:06:58.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:58.954+0000 7fa8a9200700 1 -- 172.21.15.28:0/2376450041 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa8a4109220 msgr2=0x7fa8a419f5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:06:58.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:58.954+0000 7fa8a9200700 1 --2- 172.21.15.28:0/2376450041 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa8a4109220 0x7fa8a419f5c0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fa88c00eb90 tx=0x7fa88c00eea0 comp rx=0 tx=0).stop 2024-05-12T22:06:58.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:58.954+0000 7fa8a9200700 1 -- 172.21.15.28:0/2376450041 shutdown_connections 2024-05-12T22:06:58.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:58.954+0000 7fa8a9200700 1 --2- 172.21.15.28:0/2376450041 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fa8880a2d30 0x7fa8880a51f0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:58.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:58.954+0000 7fa8a9200700 1 --2- 172.21.15.28:0/2376450041 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa8a4108860 0x7fa8a419f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:58.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:58.954+0000 7fa8a9200700 1 --2- 172.21.15.28:0/2376450041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa8a410d730 0x7fa8a41a3900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:58.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:58.954+0000 7fa8a9200700 1 --2- 172.21.15.28:0/2376450041 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa8a4109220 0x7fa8a419f5c0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:06:58.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:58.954+0000 7fa8a9200700 1 -- 172.21.15.28:0/2376450041 >> 172.21.15.28:0/2376450041 conn(0x7fa8a40780b0 msgr2=0x7fa8a4078710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:06:58.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:58.954+0000 7fa8a9200700 1 -- 172.21.15.28:0/2376450041 shutdown_connections 2024-05-12T22:06:58.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:06:58.954+0000 7fa8a9200700 1 -- 172.21.15.28:0/2376450041 wait complete. 2024-05-12T22:06:59.666 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:59 smithi028 ceph-mon[29847]: pgmap v324: 97 pgs: 97 active+clean; 636 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 1.6 KiB/s wr, 7 op/s; 53 B/s, 0 objects/s recovering 2024-05-12T22:06:59.667 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:06:59 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/2376450041' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-12T22:06:59.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:59 smithi186 ceph-mon[34785]: pgmap v324: 97 pgs: 97 active+clean; 636 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 1.6 KiB/s wr, 7 op/s; 53 B/s, 0 objects/s recovering 2024-05-12T22:06:59.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:06:59 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/2376450041' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-12T22:06:59.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:59 smithi086 ceph-mon[34297]: pgmap v324: 97 pgs: 97 active+clean; 636 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 1.6 KiB/s wr, 7 op/s; 53 B/s, 0 objects/s recovering 2024-05-12T22:06:59.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:06:59 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/2376450041' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-12T22:06:59.744 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f72fecff68e1d400c4568684327c900485c20d6a shell --fsid 6b92e7b4-10aa-11ef-bc98-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-05-12T22:06:59.744 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f72fecff68e1d400c4568684327c900485c20d6a shell --fsid 6b92e7b4-10aa-11ef-bc98-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-05-12T22:06:59.745 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f72fecff68e1d400c4568684327c900485c20d6a shell --fsid 6b92e7b4-10aa-11ef-bc98-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-05-12T22:06:59.745 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f72fecff68e1d400c4568684327c900485c20d6a shell --fsid 6b92e7b4-10aa-11ef-bc98-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-05-12T22:06:59.745 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f72fecff68e1d400c4568684327c900485c20d6a shell --fsid 6b92e7b4-10aa-11ef-bc98-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-05-12T22:06:59.746 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f72fecff68e1d400c4568684327c900485c20d6a shell --fsid 6b92e7b4-10aa-11ef-bc98-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-05-12T22:06:59.746 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f72fecff68e1d400c4568684327c900485c20d6a shell --fsid 6b92e7b4-10aa-11ef-bc98-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-05-12T22:06:59.746 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f72fecff68e1d400c4568684327c900485c20d6a shell --fsid 6b92e7b4-10aa-11ef-bc98-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-05-12T22:06:59.747 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f72fecff68e1d400c4568684327c900485c20d6a shell --fsid 6b92e7b4-10aa-11ef-bc98-c7b262605968 -- ceph tell osd.8 flush_pg_stats 2024-05-12T22:06:59.747 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f72fecff68e1d400c4568684327c900485c20d6a shell --fsid 6b92e7b4-10aa-11ef-bc98-c7b262605968 -- ceph tell osd.9 flush_pg_stats 2024-05-12T22:06:59.748 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f72fecff68e1d400c4568684327c900485c20d6a shell --fsid 6b92e7b4-10aa-11ef-bc98-c7b262605968 -- ceph tell osd.10 flush_pg_stats 2024-05-12T22:06:59.748 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f72fecff68e1d400c4568684327c900485c20d6a shell --fsid 6b92e7b4-10aa-11ef-bc98-c7b262605968 -- ceph tell osd.11 flush_pg_stats 2024-05-12T22:07:00.816 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/mon.a/config 2024-05-12T22:07:00.854 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/mon.a/config 2024-05-12T22:07:01.060 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/mon.a/config 2024-05-12T22:07:01.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:07:01 smithi028 ceph-mon[29847]: pgmap v325: 97 pgs: 97 active+clean; 639 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.7 KiB/s wr, 8 op/s; 48 B/s, 0 objects/s recovering 2024-05-12T22:07:01.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:07:01 smithi186 ceph-mon[34785]: pgmap v325: 97 pgs: 97 active+clean; 639 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.7 KiB/s wr, 8 op/s; 48 B/s, 0 objects/s recovering 2024-05-12T22:07:01.712 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/mon.a/config 2024-05-12T22:07:01.713 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/mon.a/config 2024-05-12T22:07:01.713 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/mon.a/config 2024-05-12T22:07:01.713 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/mon.a/config 2024-05-12T22:07:01.715 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/mon.a/config 2024-05-12T22:07:01.715 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/mon.a/config 2024-05-12T22:07:01.717 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/mon.a/config 2024-05-12T22:07:01.720 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/mon.a/config 2024-05-12T22:07:01.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:07:01 smithi086 ceph-mon[34297]: pgmap v325: 97 pgs: 97 active+clean; 639 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.7 KiB/s wr, 8 op/s; 48 B/s, 0 objects/s recovering 2024-05-12T22:07:01.829 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/mon.a/config 2024-05-12T22:07:03.414 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:07:03 smithi028 ceph-mon[29847]: pgmap v326: 97 pgs: 97 active+clean; 639 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 852 B/s wr, 5 op/s; 0 B/s, 0 objects/s recovering 2024-05-12T22:07:03.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:07:03 smithi186 ceph-mon[34785]: pgmap v326: 97 pgs: 97 active+clean; 639 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 852 B/s wr, 5 op/s; 0 B/s, 0 objects/s recovering 2024-05-12T22:07:03.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:07:03 smithi086 ceph-mon[34297]: pgmap v326: 97 pgs: 97 active+clean; 639 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 852 B/s wr, 5 op/s; 0 B/s, 0 objects/s recovering 2024-05-12T22:07:05.597 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:07:05 smithi028 ceph-mon[29847]: pgmap v327: 97 pgs: 97 active+clean; 639 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 852 B/s wr, 6 op/s 2024-05-12T22:07:05.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:07:05 smithi186 ceph-mon[34785]: pgmap v327: 97 pgs: 97 active+clean; 639 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 852 B/s wr, 6 op/s 2024-05-12T22:07:05.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:07:05 smithi086 ceph-mon[34297]: pgmap v327: 97 pgs: 97 active+clean; 639 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 852 B/s wr, 6 op/s 2024-05-12T22:07:07.463 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:07:07 smithi028 ceph-mon[29847]: pgmap v328: 97 pgs: 97 active+clean; 639 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 852 B/s wr, 7 op/s 2024-05-12T22:07:07.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:07:07 smithi186 ceph-mon[34785]: pgmap v328: 97 pgs: 97 active+clean; 639 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 852 B/s wr, 7 op/s 2024-05-12T22:07:07.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:07:07 smithi086 ceph-mon[34297]: pgmap v328: 97 pgs: 97 active+clean; 639 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 852 B/s wr, 7 op/s 2024-05-12T22:07:09.608 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:07:09 smithi028 ceph-mon[29847]: pgmap v329: 97 pgs: 97 active+clean; 639 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 341 B/s wr, 6 op/s 2024-05-12T22:07:09.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:07:09 smithi186 ceph-mon[34785]: pgmap v329: 97 pgs: 97 active+clean; 639 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 341 B/s wr, 6 op/s 2024-05-12T22:07:09.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:07:09 smithi086 ceph-mon[34297]: pgmap v329: 97 pgs: 97 active+clean; 639 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 341 B/s wr, 6 op/s 2024-05-12T22:07:11.664 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:07:11 smithi028 ceph-mon[29847]: pgmap v330: 97 pgs: 97 active+clean; 640 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 597 B/s wr, 7 op/s 2024-05-12T22:07:11.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:07:11 smithi186 ceph-mon[34785]: pgmap v330: 97 pgs: 97 active+clean; 640 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 597 B/s wr, 7 op/s 2024-05-12T22:07:11.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:07:11 smithi086 ceph-mon[34297]: pgmap v330: 97 pgs: 97 active+clean; 640 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 597 B/s wr, 7 op/s 2024-05-12T22:07:12.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:11.932+0000 7fd5e9e00700 1 -- 172.21.15.28:0/2670962270 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5e41083a0 msgr2=0x7fd5e4108780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:12.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:11.932+0000 7fd5e9e00700 1 --2- 172.21.15.28:0/2670962270 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5e41083a0 0x7fd5e4108780 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fd5d4009990 tx=0x7fd5d4009ca0 comp rx=0 tx=0).stop 2024-05-12T22:07:12.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:11.932+0000 7fd5e9e00700 1 -- 172.21.15.28:0/2670962270 shutdown_connections 2024-05-12T22:07:12.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:11.932+0000 7fd5e9e00700 1 --2- 172.21.15.28:0/2670962270 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd5e406a9d0 0x7fd5e410f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:12.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:11.932+0000 7fd5e9e00700 1 --2- 172.21.15.28:0/2670962270 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5e41083a0 0x7fd5e4108780 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:12.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:11.932+0000 7fd5e9e00700 1 --2- 172.21.15.28:0/2670962270 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd5e406a030 0x7fd5e406a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:12.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:11.932+0000 7fd5e9e00700 1 -- 172.21.15.28:0/2670962270 >> 172.21.15.28:0/2670962270 conn(0x7fd5e40781d0 msgr2=0x7fd5e40785e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:07:12.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:11.932+0000 7fd5e9e00700 1 -- 172.21.15.28:0/2670962270 shutdown_connections 2024-05-12T22:07:12.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:11.932+0000 7fd5e9e00700 1 -- 172.21.15.28:0/2670962270 wait complete. 2024-05-12T22:07:12.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:11.933+0000 7fd5e9e00700 1 Processor -- start 2024-05-12T22:07:12.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:11.934+0000 7fd5e9e00700 1 -- start start 2024-05-12T22:07:12.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:11.934+0000 7fd5e9e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5e406a030 0x7fd5e41a0a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:07:12.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:11.934+0000 7fd5e9e00700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd5e406a9d0 0x7fd5e41a0fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:07:12.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:11.934+0000 7fd5e9e00700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd5e41083a0 0x7fd5e419ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:07:12.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:11.934+0000 7fd5e9e00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd5e4112650 con 0x7fd5e406a030 2024-05-12T22:07:12.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:11.934+0000 7fd5e9e00700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fd5e41124d0 con 0x7fd5e41083a0 2024-05-12T22:07:12.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:11.934+0000 7fd5e9e00700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd5e41127d0 con 0x7fd5e406a9d0 2024-05-12T22:07:12.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:11.934+0000 7fd5e2a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5e406a030 0x7fd5e41a0a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:07:12.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:11.934+0000 7fd5da000700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd5e406a9d0 0x7fd5e41a0fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:07:12.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:11.934+0000 7fd5e3400700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd5e41083a0 0x7fd5e419ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:07:12.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:11.934+0000 7fd5e2a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5e406a030 0x7fd5e41a0a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49698/0 (socket says 172.21.15.28:49698) 2024-05-12T22:07:12.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:11.934+0000 7fd5e2a00700 1 -- 172.21.15.28:0/2277396536 learned_addr learned my addr 172.21.15.28:0/2277396536 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:07:12.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:11.935+0000 7fd5e2a00700 1 -- 172.21.15.28:0/2277396536 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd5e406a9d0 msgr2=0x7fd5e41a0fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:12.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:11.935+0000 7fd5e2a00700 1 --2- 172.21.15.28:0/2277396536 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd5e406a9d0 0x7fd5e41a0fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:12.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:11.935+0000 7fd5e2a00700 1 -- 172.21.15.28:0/2277396536 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd5e41083a0 msgr2=0x7fd5e419ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:12.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:11.935+0000 7fd5e2a00700 1 --2- 172.21.15.28:0/2277396536 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd5e41083a0 0x7fd5e419ab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:12.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:11.935+0000 7fd5e2a00700 1 -- 172.21.15.28:0/2277396536 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5e419b410 con 0x7fd5e406a030 2024-05-12T22:07:12.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:11.935+0000 7fd5e2a00700 1 --2- 172.21.15.28:0/2277396536 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5e406a030 0x7fd5e41a0a60 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fd5d4000c00 tx=0x7fd5d400bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:07:12.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:11.935+0000 7fd5e8a00700 1 -- 172.21.15.28:0/2277396536 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5d4027070 con 0x7fd5e406a030 2024-05-12T22:07:12.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:11.936+0000 7fd5e8a00700 1 -- 172.21.15.28:0/2277396536 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd5d40056d0 con 0x7fd5e406a030 2024-05-12T22:07:12.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:11.936+0000 7fd5e9e00700 1 -- 172.21.15.28:0/2277396536 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5e419b6a0 con 0x7fd5e406a030 2024-05-12T22:07:12.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:11.936+0000 7fd5e8a00700 1 -- 172.21.15.28:0/2277396536 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5d4022e10 con 0x7fd5e406a030 2024-05-12T22:07:12.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:11.936+0000 7fd5e9e00700 1 -- 172.21.15.28:0/2277396536 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5e41a79e0 con 0x7fd5e406a030 2024-05-12T22:07:12.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:11.937+0000 7fd5e9e00700 1 -- 172.21.15.28:0/2277396536 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fd5e410df50 con 0x7fd5e406a030 2024-05-12T22:07:12.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:11.939+0000 7fd5e8a00700 1 -- 172.21.15.28:0/2277396536 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fd5d402a070 con 0x7fd5e406a030 2024-05-12T22:07:12.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:11.940+0000 7fd5e8a00700 1 --2- 172.21.15.28:0/2277396536 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fd5c80a2d30 0x7fd5c80a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:07:12.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:11.940+0000 7fd5e8a00700 1 -- 172.21.15.28:0/2277396536 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7fd5d40d0690 con 0x7fd5e406a030 2024-05-12T22:07:12.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:11.940+0000 7fd5da000700 1 --2- 172.21.15.28:0/2277396536 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fd5c80a2d30 0x7fd5c80a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:07:12.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:11.941+0000 7fd5e8a00700 1 --2- 172.21.15.28:0/2277396536 >> [v2:172.21.15.28:6826/136220472,v1:172.21.15.28:6827/136220472] conn(0x7fd5c80ac870 0x7fd5c80aed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:07:12.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:11.941+0000 7fd5e8a00700 1 -- 172.21.15.28:0/2277396536 --> [v2:172.21.15.28:6826/136220472,v1:172.21.15.28:6827/136220472] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd5c80af3e0 con 0x7fd5c80ac870 2024-05-12T22:07:12.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:11.941+0000 7fd5e8a00700 1 -- 172.21.15.28:0/2277396536 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=88) ==== 24+0+0 (secure 0 0 0) 0x7fd5d40d0a40 con 0x7fd5e406a030 2024-05-12T22:07:12.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:11.941+0000 7fd5e3400700 1 --2- 172.21.15.28:0/2277396536 >> [v2:172.21.15.28:6826/136220472,v1:172.21.15.28:6827/136220472] conn(0x7fd5c80ac870 0x7fd5c80aed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:07:12.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:11.941+0000 7fd5da000700 1 --2- 172.21.15.28:0/2277396536 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fd5c80a2d30 0x7fd5c80a51f0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fd5cc009da0 tx=0x7fd5cc00b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:07:12.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:11.942+0000 7fd5e3400700 1 --2- 172.21.15.28:0/2277396536 >> [v2:172.21.15.28:6826/136220472,v1:172.21.15.28:6827/136220472] conn(0x7fd5c80ac870 0x7fd5c80aed10 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:07:12.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:11.943+0000 7fd5e8a00700 1 -- 172.21.15.28:0/2277396536 <== osd.9 v2:172.21.15.28:6826/136220472 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd5c80af3e0 con 0x7fd5c80ac870 2024-05-12T22:07:12.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:11.961+0000 7fd5e9e00700 1 -- 172.21.15.28:0/2277396536 --> [v2:172.21.15.28:6826/136220472,v1:172.21.15.28:6827/136220472] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fd5e4103a50 con 0x7fd5c80ac870 2024-05-12T22:07:12.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:11.962+0000 7fd5e8a00700 1 -- 172.21.15.28:0/2277396536 <== osd.9 v2:172.21.15.28:6826/136220472 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fd5e4103a50 con 0x7fd5c80ac870 2024-05-12T22:07:12.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:11.962+0000 7fd5e9e00700 1 -- 172.21.15.28:0/2277396536 >> [v2:172.21.15.28:6826/136220472,v1:172.21.15.28:6827/136220472] conn(0x7fd5c80ac870 msgr2=0x7fd5c80aed10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:12.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:11.962+0000 7fd5e9e00700 1 --2- 172.21.15.28:0/2277396536 >> [v2:172.21.15.28:6826/136220472,v1:172.21.15.28:6827/136220472] conn(0x7fd5c80ac870 0x7fd5c80aed10 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:12.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:11.963+0000 7fd5e9e00700 1 -- 172.21.15.28:0/2277396536 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fd5c80a2d30 msgr2=0x7fd5c80a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:12.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:11.963+0000 7fd5e9e00700 1 --2- 172.21.15.28:0/2277396536 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fd5c80a2d30 0x7fd5c80a51f0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fd5cc009da0 tx=0x7fd5cc00b040 comp rx=0 tx=0).stop 2024-05-12T22:07:12.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:11.963+0000 7fd5e9e00700 1 -- 172.21.15.28:0/2277396536 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5e406a030 msgr2=0x7fd5e41a0a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:12.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:11.963+0000 7fd5e9e00700 1 --2- 172.21.15.28:0/2277396536 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5e406a030 0x7fd5e41a0a60 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fd5d4000c00 tx=0x7fd5d400bab0 comp rx=0 tx=0).stop 2024-05-12T22:07:12.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:11.963+0000 7fd5e3400700 1 -- 172.21.15.28:0/2277396536 reap_dead start 2024-05-12T22:07:12.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:11.963+0000 7fd5e9e00700 1 -- 172.21.15.28:0/2277396536 shutdown_connections 2024-05-12T22:07:12.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:11.963+0000 7fd5e9e00700 1 -- 172.21.15.28:0/2277396536 >> 172.21.15.28:0/2277396536 conn(0x7fd5e40781d0 msgr2=0x7fd5e410d530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:07:12.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:11.963+0000 7fd5e9e00700 1 -- 172.21.15.28:0/2277396536 shutdown_connections 2024-05-12T22:07:12.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:11.963+0000 7fd5e9e00700 1 -- 172.21.15.28:0/2277396536 wait complete. 2024-05-12T22:07:13.471 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:07:13 smithi028 ceph-mon[29847]: pgmap v331: 97 pgs: 97 active+clean; 640 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2024-05-12T22:07:13.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:07:13 smithi186 ceph-mon[34785]: pgmap v331: 97 pgs: 97 active+clean; 640 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2024-05-12T22:07:13.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:07:13 smithi086 ceph-mon[34297]: pgmap v331: 97 pgs: 97 active+clean; 640 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2024-05-12T22:07:14.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:14.662+0000 7f76d1200700 1 -- 172.21.15.28:0/3049914442 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76cc104430 msgr2=0x7f76cc1048b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:14.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:14.662+0000 7f76d1200700 1 --2- 172.21.15.28:0/3049914442 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76cc104430 0x7f76cc1048b0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f76b80098f0 tx=0x7f76b8009c00 comp rx=0 tx=0).stop 2024-05-12T22:07:14.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:14.662+0000 7f76d1200700 1 -- 172.21.15.28:0/3049914442 shutdown_connections 2024-05-12T22:07:14.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:14.662+0000 7f76d1200700 1 --2- 172.21.15.28:0/3049914442 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f76cc10a4c0 0x7f76cc10a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:14.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:14.662+0000 7f76d1200700 1 --2- 172.21.15.28:0/3049914442 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f76cc104df0 0x7f76cc111300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:14.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:14.662+0000 7f76d1200700 1 --2- 172.21.15.28:0/3049914442 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76cc104430 0x7f76cc1048b0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:14.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:14.662+0000 7f76d1200700 1 -- 172.21.15.28:0/3049914442 >> 172.21.15.28:0/3049914442 conn(0x7f76cc0fff00 msgr2=0x7f76cc102320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:07:14.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:14.662+0000 7f76d1200700 1 -- 172.21.15.28:0/3049914442 shutdown_connections 2024-05-12T22:07:14.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:14.662+0000 7f76d1200700 1 -- 172.21.15.28:0/3049914442 wait complete. 2024-05-12T22:07:14.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:14.663+0000 7f76d1200700 1 Processor -- start 2024-05-12T22:07:14.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:14.663+0000 7f76d1200700 1 -- start start 2024-05-12T22:07:14.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:14.664+0000 7f76d1200700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f76cc104430 0x7f76cc1a08e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:07:14.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:14.664+0000 7f76d1200700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76cc104df0 0x7f76cc1a0e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:07:14.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:14.664+0000 7f76d1200700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f76cc10a4c0 0x7f76cc19aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:07:14.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:14.664+0000 7f76d1200700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f76cc114720 con 0x7f76cc104df0 2024-05-12T22:07:14.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:14.664+0000 7f76d1200700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f76cc1145a0 con 0x7f76cc10a4c0 2024-05-12T22:07:14.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:14.664+0000 7f76d1200700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f76cc1148a0 con 0x7f76cc104430 2024-05-12T22:07:14.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:14.664+0000 7f76cb400700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f76cc104430 0x7f76cc1a08e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:07:14.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:14.664+0000 7f76cb400700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f76cc104430 0x7f76cc1a08e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:37290/0 (socket says 172.21.15.28:37290) 2024-05-12T22:07:14.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:14.664+0000 7f76cb400700 1 -- 172.21.15.28:0/2846398801 learned_addr learned my addr 172.21.15.28:0/2846398801 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:07:14.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:14.664+0000 7f76cbe00700 1 --2- 172.21.15.28:0/2846398801 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f76cc10a4c0 0x7f76cc19aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:07:14.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:14.664+0000 7f76caa00700 1 --2- 172.21.15.28:0/2846398801 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76cc104df0 0x7f76cc1a0e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:07:14.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:14.665+0000 7f76cb400700 1 -- 172.21.15.28:0/2846398801 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f76cc10a4c0 msgr2=0x7f76cc19aaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:14.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:14.665+0000 7f76cb400700 1 --2- 172.21.15.28:0/2846398801 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f76cc10a4c0 0x7f76cc19aaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:14.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:14.665+0000 7f76cb400700 1 -- 172.21.15.28:0/2846398801 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76cc104df0 msgr2=0x7f76cc1a0e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:14.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:14.665+0000 7f76cb400700 1 --2- 172.21.15.28:0/2846398801 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76cc104df0 0x7f76cc1a0e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:14.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:14.665+0000 7f76cb400700 1 -- 172.21.15.28:0/2846398801 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76cc19b200 con 0x7f76cc104430 2024-05-12T22:07:14.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:14.665+0000 7f76cbe00700 1 --2- 172.21.15.28:0/2846398801 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f76cc10a4c0 0x7f76cc19aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:07:14.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:14.665+0000 7f76cb400700 1 --2- 172.21.15.28:0/2846398801 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f76cc104430 0x7f76cc1a08e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f76bc00ec10 tx=0x7f76bc00ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:07:14.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:14.665+0000 7f76c3e00700 1 -- 172.21.15.28:0/2846398801 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76bc00cc90 con 0x7f76cc104430 2024-05-12T22:07:14.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:14.665+0000 7f76d1200700 1 -- 172.21.15.28:0/2846398801 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76cc19b4f0 con 0x7f76cc104430 2024-05-12T22:07:14.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:14.666+0000 7f76c3e00700 1 -- 172.21.15.28:0/2846398801 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f76bc00ce30 con 0x7f76cc104430 2024-05-12T22:07:14.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:14.666+0000 7f76c3e00700 1 -- 172.21.15.28:0/2846398801 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76bc010690 con 0x7f76cc104430 2024-05-12T22:07:14.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:14.666+0000 7f76d1200700 1 -- 172.21.15.28:0/2846398801 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76cc1a77b0 con 0x7f76cc104430 2024-05-12T22:07:14.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:14.667+0000 7f76d1200700 1 -- 172.21.15.28:0/2846398801 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f76cc062a30 con 0x7f76cc104430 2024-05-12T22:07:14.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:14.670+0000 7f76c3e00700 1 -- 172.21.15.28:0/2846398801 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f76bc003680 con 0x7f76cc104430 2024-05-12T22:07:14.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:14.671+0000 7f76c3e00700 1 --2- 172.21.15.28:0/2846398801 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f76b00a2cb0 0x7f76b00a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:07:14.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:14.671+0000 7f76c3e00700 1 -- 172.21.15.28:0/2846398801 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f76bc0cf4a0 con 0x7f76cc104430 2024-05-12T22:07:14.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:14.671+0000 7f76caa00700 1 --2- 172.21.15.28:0/2846398801 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f76b00a2cb0 0x7f76b00a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:07:14.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:14.672+0000 7f76c3e00700 1 --2- 172.21.15.28:0/2846398801 >> [v2:172.21.15.86:6824/1189968310,v1:172.21.15.86:6825/1189968310] conn(0x7f76b00ac7f0 0x7f76b00aec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:07:14.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:14.672+0000 7f76c3e00700 1 -- 172.21.15.28:0/2846398801 --> [v2:172.21.15.86:6824/1189968310,v1:172.21.15.86:6825/1189968310] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f76bc00be50 con 0x7f76b00ac7f0 2024-05-12T22:07:14.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:14.672+0000 7f76c3e00700 1 -- 172.21.15.28:0/2846398801 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_get_version_reply(handle=1 version=88) ==== 24+0+0 (secure 0 0 0) 0x7f76bc0d6050 con 0x7f76cc104430 2024-05-12T22:07:14.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:14.672+0000 7f76cbe00700 1 --2- 172.21.15.28:0/2846398801 >> [v2:172.21.15.86:6824/1189968310,v1:172.21.15.86:6825/1189968310] conn(0x7f76b00ac7f0 0x7f76b00aec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:07:14.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:14.672+0000 7f76caa00700 1 --2- 172.21.15.28:0/2846398801 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f76b00a2cb0 0x7f76b00a5170 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f76b80098c0 tx=0x7f76b8019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:07:14.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:14.673+0000 7f76cbe00700 1 --2- 172.21.15.28:0/2846398801 >> [v2:172.21.15.86:6824/1189968310,v1:172.21.15.86:6825/1189968310] conn(0x7f76b00ac7f0 0x7f76b00aec90 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:07:14.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:14.674+0000 7f76c3e00700 1 -- 172.21.15.28:0/2846398801 <== osd.10 v2:172.21.15.86:6824/1189968310 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f76bc00be50 con 0x7f76b00ac7f0 2024-05-12T22:07:14.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:14.693+0000 7f76d1200700 1 -- 172.21.15.28:0/2846398801 --> [v2:172.21.15.86:6824/1189968310,v1:172.21.15.86:6825/1189968310] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f76cc062e30 con 0x7f76b00ac7f0 2024-05-12T22:07:14.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:14.694+0000 7f76c3e00700 1 -- 172.21.15.28:0/2846398801 <== osd.10 v2:172.21.15.86:6824/1189968310 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f76cc062e30 con 0x7f76b00ac7f0 2024-05-12T22:07:14.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:14.694+0000 7f76d1200700 1 -- 172.21.15.28:0/2846398801 >> [v2:172.21.15.86:6824/1189968310,v1:172.21.15.86:6825/1189968310] conn(0x7f76b00ac7f0 msgr2=0x7f76b00aec90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:14.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:14.694+0000 7f76d1200700 1 --2- 172.21.15.28:0/2846398801 >> [v2:172.21.15.86:6824/1189968310,v1:172.21.15.86:6825/1189968310] conn(0x7f76b00ac7f0 0x7f76b00aec90 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:14.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:14.694+0000 7f76d1200700 1 -- 172.21.15.28:0/2846398801 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f76b00a2cb0 msgr2=0x7f76b00a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:14.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:14.694+0000 7f76d1200700 1 --2- 172.21.15.28:0/2846398801 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f76b00a2cb0 0x7f76b00a5170 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f76b80098c0 tx=0x7f76b8019040 comp rx=0 tx=0).stop 2024-05-12T22:07:14.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:14.694+0000 7f76d1200700 1 -- 172.21.15.28:0/2846398801 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f76cc104430 msgr2=0x7f76cc1a08e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:14.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:14.694+0000 7f76d1200700 1 --2- 172.21.15.28:0/2846398801 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f76cc104430 0x7f76cc1a08e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f76bc00ec10 tx=0x7f76bc00ef20 comp rx=0 tx=0).stop 2024-05-12T22:07:14.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:14.694+0000 7f76cbe00700 1 -- 172.21.15.28:0/2846398801 reap_dead start 2024-05-12T22:07:14.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:14.695+0000 7f76d1200700 1 -- 172.21.15.28:0/2846398801 shutdown_connections 2024-05-12T22:07:14.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:14.695+0000 7f76d1200700 1 -- 172.21.15.28:0/2846398801 >> 172.21.15.28:0/2846398801 conn(0x7f76cc0fff00 msgr2=0x7f76cc1005a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:07:14.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:14.695+0000 7f76d1200700 1 -- 172.21.15.28:0/2846398801 shutdown_connections 2024-05-12T22:07:14.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:14.695+0000 7f76d1200700 1 -- 172.21.15.28:0/2846398801 wait complete. 2024-05-12T22:07:15.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:07:15 smithi028 ceph-mon[29847]: pgmap v332: 97 pgs: 97 active+clean; 640 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 255 B/s wr, 6 op/s 2024-05-12T22:07:15.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:07:15 smithi186 ceph-mon[34785]: pgmap v332: 97 pgs: 97 active+clean; 640 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 255 B/s wr, 6 op/s 2024-05-12T22:07:15.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:07:15 smithi086 ceph-mon[34297]: pgmap v332: 97 pgs: 97 active+clean; 640 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 255 B/s wr, 6 op/s 2024-05-12T22:07:16.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:16.056+0000 7f417d600700 1 -- 172.21.15.28:0/1556962628 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f417806a030 msgr2=0x7f417806a490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:16.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:16.056+0000 7f417d600700 1 --2- 172.21.15.28:0/1556962628 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f417806a030 0x7f417806a490 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f41640098f0 tx=0x7f4164009c00 comp rx=0 tx=0).stop 2024-05-12T22:07:16.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:16.057+0000 7f417d600700 1 -- 172.21.15.28:0/1556962628 shutdown_connections 2024-05-12T22:07:16.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:16.057+0000 7f417d600700 1 --2- 172.21.15.28:0/1556962628 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f41781083a0 0x7f4178108780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:16.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:16.057+0000 7f417d600700 1 --2- 172.21.15.28:0/1556962628 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f417806a9d0 0x7f417810f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:16.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:16.057+0000 7f417d600700 1 --2- 172.21.15.28:0/1556962628 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f417806a030 0x7f417806a490 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:16.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:16.057+0000 7f417d600700 1 -- 172.21.15.28:0/1556962628 >> 172.21.15.28:0/1556962628 conn(0x7f41780781d0 msgr2=0x7f41780785e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:07:16.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:16.057+0000 7f417d600700 1 -- 172.21.15.28:0/1556962628 shutdown_connections 2024-05-12T22:07:16.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:16.057+0000 7f417d600700 1 -- 172.21.15.28:0/1556962628 wait complete. 2024-05-12T22:07:16.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:16.058+0000 7f417d600700 1 Processor -- start 2024-05-12T22:07:16.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:16.058+0000 7f417d600700 1 -- start start 2024-05-12T22:07:16.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:16.058+0000 7f417d600700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f417806a030 0x7f417819e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:07:16.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:16.058+0000 7f417d600700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f417806a9d0 0x7f417819ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:07:16.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:16.058+0000 7f417d600700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f41781083a0 0x7f41781989b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:07:16.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:16.058+0000 7f417d600700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4178112650 con 0x7f417806a030 2024-05-12T22:07:16.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:16.058+0000 7f417d600700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f41781124d0 con 0x7f41781083a0 2024-05-12T22:07:16.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:16.058+0000 7f417d600700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f41781127d0 con 0x7f417806a9d0 2024-05-12T22:07:16.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:16.059+0000 7f4176a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f417806a030 0x7f417819e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:07:16.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:16.059+0000 7f4176a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f417806a030 0x7f417819e7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55068/0 (socket says 172.21.15.28:55068) 2024-05-12T22:07:16.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:16.059+0000 7f4176a00700 1 -- 172.21.15.28:0/1903375764 learned_addr learned my addr 172.21.15.28:0/1903375764 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:07:16.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:16.059+0000 7f4176000700 1 --2- 172.21.15.28:0/1903375764 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f417806a9d0 0x7f417819ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:07:16.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:16.059+0000 7f4177400700 1 --2- 172.21.15.28:0/1903375764 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f41781083a0 0x7f41781989b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:07:16.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:16.059+0000 7f4176000700 1 -- 172.21.15.28:0/1903375764 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f41781083a0 msgr2=0x7f41781989b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:16.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:16.059+0000 7f4176000700 1 --2- 172.21.15.28:0/1903375764 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f41781083a0 0x7f41781989b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:16.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:16.059+0000 7f4176000700 1 -- 172.21.15.28:0/1903375764 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f417806a030 msgr2=0x7f417819e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:16.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:16.059+0000 7f4176000700 1 --2- 172.21.15.28:0/1903375764 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f417806a030 0x7f417819e7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:16.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:16.059+0000 7f4176000700 1 -- 172.21.15.28:0/1903375764 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4178199110 con 0x7f417806a9d0 2024-05-12T22:07:16.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:16.060+0000 7f4176000700 1 --2- 172.21.15.28:0/1903375764 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f417806a9d0 0x7f417819ed30 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f4164000c00 tx=0x7f416400fa00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:07:16.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:16.060+0000 7f416f400700 1 -- 172.21.15.28:0/1903375764 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4164005040 con 0x7f417806a9d0 2024-05-12T22:07:16.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:16.060+0000 7f416f400700 1 -- 172.21.15.28:0/1903375764 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f41640056c0 con 0x7f417806a9d0 2024-05-12T22:07:16.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:16.060+0000 7f416f400700 1 -- 172.21.15.28:0/1903375764 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4164003a00 con 0x7f417806a9d0 2024-05-12T22:07:16.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:16.060+0000 7f417d600700 1 -- 172.21.15.28:0/1903375764 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41781993a0 con 0x7f417806a9d0 2024-05-12T22:07:16.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:16.060+0000 7f417d600700 1 -- 172.21.15.28:0/1903375764 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41781a56c0 con 0x7f417806a9d0 2024-05-12T22:07:16.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:16.061+0000 7f417d600700 1 -- 172.21.15.28:0/1903375764 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f4178062ab0 con 0x7f417806a9d0 2024-05-12T22:07:16.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:16.064+0000 7f416f400700 1 -- 172.21.15.28:0/1903375764 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f416400b560 con 0x7f417806a9d0 2024-05-12T22:07:16.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:16.065+0000 7f416f400700 1 --2- 172.21.15.28:0/1903375764 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f415c0a2d30 0x7f415c0a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:07:16.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:16.066+0000 7f416f400700 1 -- 172.21.15.28:0/1903375764 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f41640d04e0 con 0x7f417806a9d0 2024-05-12T22:07:16.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:16.066+0000 7f4176a00700 1 --2- 172.21.15.28:0/1903375764 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f415c0a2d30 0x7f415c0a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:07:16.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:16.066+0000 7f416f400700 1 --2- 172.21.15.28:0/1903375764 >> [v2:172.21.15.28:6802/906003882,v1:172.21.15.28:6803/906003882] conn(0x7f415c0ac870 0x7f415c0aed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:07:16.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:16.066+0000 7f416f400700 1 -- 172.21.15.28:0/1903375764 --> [v2:172.21.15.28:6802/906003882,v1:172.21.15.28:6803/906003882] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f416400bde0 con 0x7f415c0ac870 2024-05-12T22:07:16.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:16.066+0000 7f416f400700 1 -- 172.21.15.28:0/1903375764 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_get_version_reply(handle=1 version=88) ==== 24+0+0 (secure 0 0 0) 0x7f41640d0890 con 0x7f417806a9d0 2024-05-12T22:07:16.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:16.066+0000 7f4177400700 1 --2- 172.21.15.28:0/1903375764 >> [v2:172.21.15.28:6802/906003882,v1:172.21.15.28:6803/906003882] conn(0x7f415c0ac870 0x7f415c0aed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:07:16.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:16.066+0000 7f4176a00700 1 --2- 172.21.15.28:0/1903375764 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f415c0a2d30 0x7f415c0a51f0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f4178199af0 tx=0x7f4168009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:07:16.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:16.067+0000 7f4177400700 1 --2- 172.21.15.28:0/1903375764 >> [v2:172.21.15.28:6802/906003882,v1:172.21.15.28:6803/906003882] conn(0x7f415c0ac870 0x7f415c0aed10 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:07:16.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:16.068+0000 7f416f400700 1 -- 172.21.15.28:0/1903375764 <== osd.0 v2:172.21.15.28:6802/906003882 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f416400bde0 con 0x7f415c0ac870 2024-05-12T22:07:16.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:16.087+0000 7f417d600700 1 -- 172.21.15.28:0/1903375764 --> [v2:172.21.15.28:6802/906003882,v1:172.21.15.28:6803/906003882] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f4178062f80 con 0x7f415c0ac870 2024-05-12T22:07:16.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:16.088+0000 7f416f400700 1 -- 172.21.15.28:0/1903375764 <== osd.0 v2:172.21.15.28:6802/906003882 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f4178062f80 con 0x7f415c0ac870 2024-05-12T22:07:16.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:16.088+0000 7f417d600700 1 -- 172.21.15.28:0/1903375764 >> [v2:172.21.15.28:6802/906003882,v1:172.21.15.28:6803/906003882] conn(0x7f415c0ac870 msgr2=0x7f415c0aed10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:16.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:16.088+0000 7f417d600700 1 --2- 172.21.15.28:0/1903375764 >> [v2:172.21.15.28:6802/906003882,v1:172.21.15.28:6803/906003882] conn(0x7f415c0ac870 0x7f415c0aed10 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:16.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:16.088+0000 7f417d600700 1 -- 172.21.15.28:0/1903375764 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f415c0a2d30 msgr2=0x7f415c0a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:16.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:16.088+0000 7f417d600700 1 --2- 172.21.15.28:0/1903375764 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f415c0a2d30 0x7f415c0a51f0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f4178199af0 tx=0x7f4168009310 comp rx=0 tx=0).stop 2024-05-12T22:07:16.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:16.088+0000 7f417d600700 1 -- 172.21.15.28:0/1903375764 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f417806a9d0 msgr2=0x7f417819ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:16.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:16.088+0000 7f417d600700 1 --2- 172.21.15.28:0/1903375764 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f417806a9d0 0x7f417819ed30 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f4164000c00 tx=0x7f416400fa00 comp rx=0 tx=0).stop 2024-05-12T22:07:16.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:16.089+0000 7f4177400700 1 -- 172.21.15.28:0/1903375764 reap_dead start 2024-05-12T22:07:16.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:16.089+0000 7f417d600700 1 -- 172.21.15.28:0/1903375764 shutdown_connections 2024-05-12T22:07:16.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:16.089+0000 7f417d600700 1 -- 172.21.15.28:0/1903375764 >> 172.21.15.28:0/1903375764 conn(0x7f41780781d0 msgr2=0x7f41780ffeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:07:16.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:16.089+0000 7f417d600700 1 -- 172.21.15.28:0/1903375764 shutdown_connections 2024-05-12T22:07:16.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:16.089+0000 7f417d600700 1 -- 172.21.15.28:0/1903375764 wait complete. 2024-05-12T22:07:16.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:16.661+0000 7f3969200700 1 -- 172.21.15.28:0/1177103507 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f396406a030 msgr2=0x7f396406a490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:16.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:16.661+0000 7f3969200700 1 --2- 172.21.15.28:0/1177103507 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f396406a030 0x7f396406a490 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f39500098f0 tx=0x7f3950009c00 comp rx=0 tx=0).stop 2024-05-12T22:07:16.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:16.661+0000 7f3969200700 1 -- 172.21.15.28:0/1177103507 shutdown_connections 2024-05-12T22:07:16.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:16.661+0000 7f3969200700 1 --2- 172.21.15.28:0/1177103507 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f39641083a0 0x7f3964108780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:16.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:16.661+0000 7f3969200700 1 --2- 172.21.15.28:0/1177103507 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f396406a9d0 0x7f396410f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:16.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:16.661+0000 7f3969200700 1 --2- 172.21.15.28:0/1177103507 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f396406a030 0x7f396406a490 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:16.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:16.661+0000 7f3969200700 1 -- 172.21.15.28:0/1177103507 >> 172.21.15.28:0/1177103507 conn(0x7f39640781d0 msgr2=0x7f39640785e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:07:16.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:16.661+0000 7f3969200700 1 -- 172.21.15.28:0/1177103507 shutdown_connections 2024-05-12T22:07:16.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:16.661+0000 7f3969200700 1 -- 172.21.15.28:0/1177103507 wait complete. 2024-05-12T22:07:16.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:16.662+0000 7f3969200700 1 Processor -- start 2024-05-12T22:07:16.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:16.662+0000 7f3969200700 1 -- start start 2024-05-12T22:07:16.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:16.663+0000 7f3969200700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f396406a030 0x7f396419e820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:07:16.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:16.663+0000 7f3969200700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f396406a9d0 0x7f396419ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:07:16.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:16.663+0000 7f3969200700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39641083a0 0x7f39641989e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:07:16.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:16.663+0000 7f3969200700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3964112600 con 0x7f39641083a0 2024-05-12T22:07:16.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:16.663+0000 7f3969200700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f3964112480 con 0x7f396406a9d0 2024-05-12T22:07:16.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:16.663+0000 7f3969200700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f3964112780 con 0x7f396406a030 2024-05-12T22:07:16.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:16.663+0000 7f3961e00700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f396406a030 0x7f396419e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:07:16.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:16.663+0000 7f3961e00700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f396406a030 0x7f396419e820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:33572/0 (socket says 172.21.15.28:33572) 2024-05-12T22:07:16.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:16.663+0000 7f3961e00700 1 -- 172.21.15.28:0/707721958 learned_addr learned my addr 172.21.15.28:0/707721958 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:07:16.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:16.663+0000 7f3962800700 1 --2- 172.21.15.28:0/707721958 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39641083a0 0x7f39641989e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:07:16.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:16.663+0000 7f3961400700 1 --2- 172.21.15.28:0/707721958 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f396406a9d0 0x7f396419ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:07:16.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:16.664+0000 7f3961e00700 1 -- 172.21.15.28:0/707721958 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f396406a9d0 msgr2=0x7f396419ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:16.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:16.664+0000 7f3961e00700 1 --2- 172.21.15.28:0/707721958 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f396406a9d0 0x7f396419ed60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:16.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:16.664+0000 7f3961e00700 1 -- 172.21.15.28:0/707721958 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39641083a0 msgr2=0x7f39641989e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:16.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:16.664+0000 7f3961e00700 1 --2- 172.21.15.28:0/707721958 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39641083a0 0x7f39641989e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:16.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:16.664+0000 7f3961e00700 1 -- 172.21.15.28:0/707721958 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3964199140 con 0x7f396406a030 2024-05-12T22:07:16.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:16.664+0000 7f3961e00700 1 --2- 172.21.15.28:0/707721958 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f396406a030 0x7f396419e820 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f395400e9c0 tx=0x7f395400ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:07:16.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:16.664+0000 7f395aa00700 1 -- 172.21.15.28:0/707721958 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3954004a30 con 0x7f396406a030 2024-05-12T22:07:16.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:16.665+0000 7f395aa00700 1 -- 172.21.15.28:0/707721958 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f395400cca0 con 0x7f396406a030 2024-05-12T22:07:16.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:16.665+0000 7f3969200700 1 -- 172.21.15.28:0/707721958 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3964199430 con 0x7f396406a030 2024-05-12T22:07:16.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:16.665+0000 7f395aa00700 1 -- 172.21.15.28:0/707721958 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3954005610 con 0x7f396406a030 2024-05-12T22:07:16.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:16.665+0000 7f3969200700 1 -- 172.21.15.28:0/707721958 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39641a56f0 con 0x7f396406a030 2024-05-12T22:07:16.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:16.665+0000 7f3969200700 1 -- 172.21.15.28:0/707721958 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f3964062a30 con 0x7f396406a030 2024-05-12T22:07:16.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:16.668+0000 7f395aa00700 1 -- 172.21.15.28:0/707721958 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f3954004cf0 con 0x7f396406a030 2024-05-12T22:07:16.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:16.669+0000 7f395aa00700 1 --2- 172.21.15.28:0/707721958 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f39480a2cb0 0x7f39480a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:07:16.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:16.669+0000 7f395aa00700 1 -- 172.21.15.28:0/707721958 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f39540cf6f0 con 0x7f396406a030 2024-05-12T22:07:16.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:16.669+0000 7f3961400700 1 --2- 172.21.15.28:0/707721958 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f39480a2cb0 0x7f39480a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:07:16.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:16.669+0000 7f395aa00700 1 --2- 172.21.15.28:0/707721958 >> [v2:172.21.15.86:6800/3354141819,v1:172.21.15.86:6801/3354141819] conn(0x7f39480ac7f0 0x7f39480aec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:07:16.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:16.670+0000 7f395aa00700 1 -- 172.21.15.28:0/707721958 --> [v2:172.21.15.86:6800/3354141819,v1:172.21.15.86:6801/3354141819] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f395400ce50 con 0x7f39480ac7f0 2024-05-12T22:07:16.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:16.670+0000 7f395aa00700 1 -- 172.21.15.28:0/707721958 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_get_version_reply(handle=1 version=88) ==== 24+0+0 (secure 0 0 0) 0x7f39540cfaa0 con 0x7f396406a030 2024-05-12T22:07:16.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:16.670+0000 7f3962800700 1 --2- 172.21.15.28:0/707721958 >> [v2:172.21.15.86:6800/3354141819,v1:172.21.15.86:6801/3354141819] conn(0x7f39480ac7f0 0x7f39480aec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:07:16.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:16.670+0000 7f3961400700 1 --2- 172.21.15.28:0/707721958 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f39480a2cb0 0x7f39480a5170 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f39500098c0 tx=0x7f3950019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:07:16.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:16.671+0000 7f3962800700 1 --2- 172.21.15.28:0/707721958 >> [v2:172.21.15.86:6800/3354141819,v1:172.21.15.86:6801/3354141819] conn(0x7f39480ac7f0 0x7f39480aec90 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:07:16.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:16.672+0000 7f395aa00700 1 -- 172.21.15.28:0/707721958 <== osd.1 v2:172.21.15.86:6800/3354141819 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f395400ce50 con 0x7f39480ac7f0 2024-05-12T22:07:16.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:16.703+0000 7f3969200700 1 -- 172.21.15.28:0/707721958 --> [v2:172.21.15.86:6800/3354141819,v1:172.21.15.86:6801/3354141819] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f3964062eb0 con 0x7f39480ac7f0 2024-05-12T22:07:16.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:16.704+0000 7f395aa00700 1 -- 172.21.15.28:0/707721958 <== osd.1 v2:172.21.15.86:6800/3354141819 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f3964062eb0 con 0x7f39480ac7f0 2024-05-12T22:07:16.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:16.705+0000 7f3969200700 1 -- 172.21.15.28:0/707721958 >> [v2:172.21.15.86:6800/3354141819,v1:172.21.15.86:6801/3354141819] conn(0x7f39480ac7f0 msgr2=0x7f39480aec90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:16.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:16.705+0000 7f3969200700 1 --2- 172.21.15.28:0/707721958 >> [v2:172.21.15.86:6800/3354141819,v1:172.21.15.86:6801/3354141819] conn(0x7f39480ac7f0 0x7f39480aec90 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:16.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:16.705+0000 7f3969200700 1 -- 172.21.15.28:0/707721958 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f39480a2cb0 msgr2=0x7f39480a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:16.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:16.705+0000 7f3969200700 1 --2- 172.21.15.28:0/707721958 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f39480a2cb0 0x7f39480a5170 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f39500098c0 tx=0x7f3950019040 comp rx=0 tx=0).stop 2024-05-12T22:07:16.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:16.705+0000 7f3969200700 1 -- 172.21.15.28:0/707721958 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f396406a030 msgr2=0x7f396419e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:16.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:16.705+0000 7f3969200700 1 --2- 172.21.15.28:0/707721958 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f396406a030 0x7f396419e820 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f395400e9c0 tx=0x7f395400ecd0 comp rx=0 tx=0).stop 2024-05-12T22:07:16.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:16.705+0000 7f3962800700 1 -- 172.21.15.28:0/707721958 reap_dead start 2024-05-12T22:07:16.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:16.705+0000 7f3969200700 1 -- 172.21.15.28:0/707721958 shutdown_connections 2024-05-12T22:07:16.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:16.705+0000 7f3969200700 1 -- 172.21.15.28:0/707721958 >> 172.21.15.28:0/707721958 conn(0x7f39640781d0 msgr2=0x7f3964100050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:07:16.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:16.705+0000 7f3969200700 1 -- 172.21.15.28:0/707721958 shutdown_connections 2024-05-12T22:07:16.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:16.705+0000 7f3969200700 1 -- 172.21.15.28:0/707721958 wait complete. 2024-05-12T22:07:16.735 INFO:teuthology.orchestra.run.smithi028.stdout:292057776159 2024-05-12T22:07:16.735 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f72fecff68e1d400c4568684327c900485c20d6a shell --fsid 6b92e7b4-10aa-11ef-bc98-c7b262605968 -- ceph osd last-stat-seq osd.9 2024-05-12T22:07:17.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:07:17 smithi028 ceph-mon[29847]: pgmap v333: 97 pgs: 97 active+clean; 640 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 255 B/s wr, 6 op/s 2024-05-12T22:07:17.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:07:17 smithi186 ceph-mon[34785]: pgmap v333: 97 pgs: 97 active+clean; 640 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 255 B/s wr, 6 op/s 2024-05-12T22:07:17.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:07:17 smithi086 ceph-mon[34297]: pgmap v333: 97 pgs: 97 active+clean; 640 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 255 B/s wr, 6 op/s 2024-05-12T22:07:18.628 INFO:teuthology.orchestra.run.smithi028.stdout:34359738449 2024-05-12T22:07:18.628 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f72fecff68e1d400c4568684327c900485c20d6a shell --fsid 6b92e7b4-10aa-11ef-bc98-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-05-12T22:07:18.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:18.633+0000 7f6b46800700 1 -- 172.21.15.28:0/4146370188 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6b40107250 msgr2=0x7f6b4006a030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:18.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:18.633+0000 7f6b46800700 1 --2- 172.21.15.28:0/4146370188 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6b40107250 0x7f6b4006a030 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f6b30009990 tx=0x7f6b30009ca0 comp rx=0 tx=0).stop 2024-05-12T22:07:18.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:18.633+0000 7f6b46800700 1 -- 172.21.15.28:0/4146370188 shutdown_connections 2024-05-12T22:07:18.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:18.633+0000 7f6b46800700 1 --2- 172.21.15.28:0/4146370188 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6b40107250 0x7f6b4006a030 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:18.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:18.633+0000 7f6b46800700 1 --2- 172.21.15.28:0/4146370188 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b40104d60 0x7f6b401051f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:18.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:18.633+0000 7f6b46800700 1 --2- 172.21.15.28:0/4146370188 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6b4006a570 0x7f6b4006a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:18.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:18.633+0000 7f6b46800700 1 -- 172.21.15.28:0/4146370188 >> 172.21.15.28:0/4146370188 conn(0x7f6b400781d0 msgr2=0x7f6b400785e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:07:18.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:18.633+0000 7f6b46800700 1 -- 172.21.15.28:0/4146370188 shutdown_connections 2024-05-12T22:07:18.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:18.633+0000 7f6b46800700 1 -- 172.21.15.28:0/4146370188 wait complete. 2024-05-12T22:07:18.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:18.634+0000 7f6b46800700 1 Processor -- start 2024-05-12T22:07:18.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:18.634+0000 7f6b46800700 1 -- start start 2024-05-12T22:07:18.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:18.635+0000 7f6b46800700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b4006a570 0x7f6b4019f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:07:18.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:18.635+0000 7f6b46800700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6b40104d60 0x7f6b4019f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:07:18.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:18.635+0000 7f6b46800700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6b401a3c70 0x7f6b401a40f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:07:18.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:18.635+0000 7f6b46800700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6b40118c30 con 0x7f6b4006a570 2024-05-12T22:07:18.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:18.635+0000 7f6b46800700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f6b40118ab0 con 0x7f6b40104d60 2024-05-12T22:07:18.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:18.635+0000 7f6b46800700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f6b40118db0 con 0x7f6b401a3c70 2024-05-12T22:07:18.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:18.635+0000 7f6b3f400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b4006a570 0x7f6b4019f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:07:18.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:18.635+0000 7f6b3f400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b4006a570 0x7f6b4019f3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55122/0 (socket says 172.21.15.28:55122) 2024-05-12T22:07:18.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:18.635+0000 7f6b3f400700 1 -- 172.21.15.28:0/1715362530 learned_addr learned my addr 172.21.15.28:0/1715362530 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:07:18.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:18.635+0000 7f6b3ea00700 1 --2- 172.21.15.28:0/1715362530 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6b40104d60 0x7f6b4019f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:07:18.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:18.635+0000 7f6b3fe00700 1 --2- 172.21.15.28:0/1715362530 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6b401a3c70 0x7f6b401a40f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:07:18.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:18.636+0000 7f6b3f400700 1 -- 172.21.15.28:0/1715362530 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6b401a3c70 msgr2=0x7f6b401a40f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:18.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:18.636+0000 7f6b3f400700 1 --2- 172.21.15.28:0/1715362530 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6b401a3c70 0x7f6b401a40f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:18.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:18.636+0000 7f6b3f400700 1 -- 172.21.15.28:0/1715362530 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6b40104d60 msgr2=0x7f6b4019f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:18.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:18.636+0000 7f6b3f400700 1 --2- 172.21.15.28:0/1715362530 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6b40104d60 0x7f6b4019f8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:18.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:18.636+0000 7f6b3f400700 1 -- 172.21.15.28:0/1715362530 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b401a4790 con 0x7f6b4006a570 2024-05-12T22:07:18.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:18.636+0000 7f6b3ea00700 1 --2- 172.21.15.28:0/1715362530 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6b40104d60 0x7f6b4019f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:07:18.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:18.636+0000 7f6b3f400700 1 --2- 172.21.15.28:0/1715362530 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b4006a570 0x7f6b4019f3a0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f6b300098e0 tx=0x7f6b3000b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:07:18.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:18.636+0000 7f6b37e00700 1 -- 172.21.15.28:0/1715362530 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b30026070 con 0x7f6b4006a570 2024-05-12T22:07:18.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:18.636+0000 7f6b37e00700 1 -- 172.21.15.28:0/1715362530 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6b3000be60 con 0x7f6b4006a570 2024-05-12T22:07:18.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:18.636+0000 7f6b37e00700 1 -- 172.21.15.28:0/1715362530 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b3000fdf0 con 0x7f6b4006a570 2024-05-12T22:07:18.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:18.636+0000 7f6b46800700 1 -- 172.21.15.28:0/1715362530 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b401a4a20 con 0x7f6b4006a570 2024-05-12T22:07:18.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:18.637+0000 7f6b46800700 1 -- 172.21.15.28:0/1715362530 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b401ac0f0 con 0x7f6b4006a570 2024-05-12T22:07:18.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:18.638+0000 7f6b46800700 1 -- 172.21.15.28:0/1715362530 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f6b40100bf0 con 0x7f6b4006a570 2024-05-12T22:07:18.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:18.640+0000 7f6b37e00700 1 -- 172.21.15.28:0/1715362530 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f6b30029070 con 0x7f6b4006a570 2024-05-12T22:07:18.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:18.641+0000 7f6b37e00700 1 --2- 172.21.15.28:0/1715362530 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f6b240a2cb0 0x7f6b240a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:07:18.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:18.641+0000 7f6b37e00700 1 -- 172.21.15.28:0/1715362530 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f6b300d0370 con 0x7f6b4006a570 2024-05-12T22:07:18.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:18.641+0000 7f6b3ea00700 1 --2- 172.21.15.28:0/1715362530 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f6b240a2cb0 0x7f6b240a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:07:18.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:18.642+0000 7f6b37e00700 1 --2- 172.21.15.28:0/1715362530 >> [v2:172.21.15.186:6816/3007687277,v1:172.21.15.186:6817/3007687277] conn(0x7f6b240ac7f0 0x7f6b240aec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:07:18.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:18.642+0000 7f6b37e00700 1 -- 172.21.15.28:0/1715362530 --> [v2:172.21.15.186:6816/3007687277,v1:172.21.15.186:6817/3007687277] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6b3000b0b0 con 0x7f6b240ac7f0 2024-05-12T22:07:18.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:18.642+0000 7f6b37e00700 1 -- 172.21.15.28:0/1715362530 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=88) ==== 24+0+0 (secure 0 0 0) 0x7f6b300d0720 con 0x7f6b4006a570 2024-05-12T22:07:18.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:18.642+0000 7f6b3fe00700 1 --2- 172.21.15.28:0/1715362530 >> [v2:172.21.15.186:6816/3007687277,v1:172.21.15.186:6817/3007687277] conn(0x7f6b240ac7f0 0x7f6b240aec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:07:18.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:18.642+0000 7f6b3ea00700 1 --2- 172.21.15.28:0/1715362530 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f6b240a2cb0 0x7f6b240a5170 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f6b40078b10 tx=0x7f6b2c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:07:18.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:18.643+0000 7f6b3fe00700 1 --2- 172.21.15.28:0/1715362530 >> [v2:172.21.15.186:6816/3007687277,v1:172.21.15.186:6817/3007687277] conn(0x7f6b240ac7f0 0x7f6b240aec90 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:07:18.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:18.644+0000 7f6b37e00700 1 -- 172.21.15.28:0/1715362530 <== osd.8 v2:172.21.15.186:6816/3007687277 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6b3000b0b0 con 0x7f6b240ac7f0 2024-05-12T22:07:18.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:18.663+0000 7f6b46800700 1 -- 172.21.15.28:0/1715362530 --> [v2:172.21.15.186:6816/3007687277,v1:172.21.15.186:6817/3007687277] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f6b40101840 con 0x7f6b240ac7f0 2024-05-12T22:07:18.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:18.663+0000 7f6b37e00700 1 -- 172.21.15.28:0/1715362530 <== osd.8 v2:172.21.15.186:6816/3007687277 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f6b40101840 con 0x7f6b240ac7f0 2024-05-12T22:07:18.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:18.664+0000 7f6b46800700 1 -- 172.21.15.28:0/1715362530 >> [v2:172.21.15.186:6816/3007687277,v1:172.21.15.186:6817/3007687277] conn(0x7f6b240ac7f0 msgr2=0x7f6b240aec90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:18.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:18.664+0000 7f6b46800700 1 --2- 172.21.15.28:0/1715362530 >> [v2:172.21.15.186:6816/3007687277,v1:172.21.15.186:6817/3007687277] conn(0x7f6b240ac7f0 0x7f6b240aec90 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:18.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:18.664+0000 7f6b46800700 1 -- 172.21.15.28:0/1715362530 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f6b240a2cb0 msgr2=0x7f6b240a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:18.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:18.664+0000 7f6b46800700 1 --2- 172.21.15.28:0/1715362530 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f6b240a2cb0 0x7f6b240a5170 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f6b40078b10 tx=0x7f6b2c009260 comp rx=0 tx=0).stop 2024-05-12T22:07:18.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:18.664+0000 7f6b46800700 1 -- 172.21.15.28:0/1715362530 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b4006a570 msgr2=0x7f6b4019f3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:18.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:18.664+0000 7f6b46800700 1 --2- 172.21.15.28:0/1715362530 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b4006a570 0x7f6b4019f3a0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f6b300098e0 tx=0x7f6b3000b8a0 comp rx=0 tx=0).stop 2024-05-12T22:07:18.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:18.664+0000 7f6b3fe00700 1 -- 172.21.15.28:0/1715362530 reap_dead start 2024-05-12T22:07:18.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:18.664+0000 7f6b46800700 1 -- 172.21.15.28:0/1715362530 shutdown_connections 2024-05-12T22:07:18.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:18.664+0000 7f6b46800700 1 -- 172.21.15.28:0/1715362530 >> 172.21.15.28:0/1715362530 conn(0x7f6b400781d0 msgr2=0x7f6b40100210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:07:18.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:18.665+0000 7f6b46800700 1 -- 172.21.15.28:0/1715362530 shutdown_connections 2024-05-12T22:07:18.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:18.665+0000 7f6b46800700 1 -- 172.21.15.28:0/1715362530 wait complete. 2024-05-12T22:07:19.403 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:07:19 smithi028 ceph-mon[29847]: pgmap v334: 97 pgs: 97 active+clean; 640 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 255 B/s wr, 6 op/s 2024-05-12T22:07:19.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:19.421+0000 7f18ce200700 1 -- 172.21.15.28:0/2653929709 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f18c8101420 msgr2=0x7f18c8101800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:19.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:19.421+0000 7f18ce200700 1 --2- 172.21.15.28:0/2653929709 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f18c8101420 0x7f18c8101800 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f18bc009990 tx=0x7f18bc009ca0 comp rx=0 tx=0).stop 2024-05-12T22:07:19.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:19.421+0000 7f18ce200700 1 -- 172.21.15.28:0/2653929709 shutdown_connections 2024-05-12T22:07:19.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:19.421+0000 7f18ce200700 1 --2- 172.21.15.28:0/2653929709 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f18c8106500 0x7f18c810e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:19.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:19.421+0000 7f18ce200700 1 --2- 172.21.15.28:0/2653929709 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f18c8101420 0x7f18c8101800 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:19.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:19.421+0000 7f18ce200700 1 --2- 172.21.15.28:0/2653929709 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18c8101dd0 0x7f18c8105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:19.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:19.421+0000 7f18ce200700 1 -- 172.21.15.28:0/2653929709 >> 172.21.15.28:0/2653929709 conn(0x7f18c80fccd0 msgr2=0x7f18c80ff0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:07:19.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:19.421+0000 7f18ce200700 1 -- 172.21.15.28:0/2653929709 shutdown_connections 2024-05-12T22:07:19.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:19.422+0000 7f18ce200700 1 -- 172.21.15.28:0/2653929709 wait complete. 2024-05-12T22:07:19.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:19.423+0000 7f18ce200700 1 Processor -- start 2024-05-12T22:07:19.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:19.423+0000 7f18ce200700 1 -- start start 2024-05-12T22:07:19.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:19.423+0000 7f18ce200700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f18c8101420 0x7f18c819e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:07:19.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:19.423+0000 7f18ce200700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18c8101dd0 0x7f18c819ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:07:19.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:19.423+0000 7f18ce200700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f18c8106500 0x7f18c81988c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:07:19.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:19.423+0000 7f18ce200700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f18c8111560 con 0x7f18c8101dd0 2024-05-12T22:07:19.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:19.423+0000 7f18ce200700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f18c81113e0 con 0x7f18c8101420 2024-05-12T22:07:19.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:19.423+0000 7f18ce200700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f18c81116e0 con 0x7f18c8106500 2024-05-12T22:07:19.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:19.424+0000 7f18c7e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18c8101dd0 0x7f18c819ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:07:19.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:19.424+0000 7f18c7e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18c8101dd0 0x7f18c819ec40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55142/0 (socket says 172.21.15.28:55142) 2024-05-12T22:07:19.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:19.424+0000 7f18c7e00700 1 -- 172.21.15.28:0/4111268553 learned_addr learned my addr 172.21.15.28:0/4111268553 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:07:19.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:19.424+0000 7f18cce00700 1 --2- 172.21.15.28:0/4111268553 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f18c8101420 0x7f18c819e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:07:19.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:19.424+0000 7f18cd800700 1 --2- 172.21.15.28:0/4111268553 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f18c8106500 0x7f18c81988c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:07:19.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:19.424+0000 7f18cce00700 1 -- 172.21.15.28:0/4111268553 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f18c8106500 msgr2=0x7f18c81988c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:19.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:19.424+0000 7f18cce00700 1 --2- 172.21.15.28:0/4111268553 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f18c8106500 0x7f18c81988c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:19.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:19.424+0000 7f18cce00700 1 -- 172.21.15.28:0/4111268553 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18c8101dd0 msgr2=0x7f18c819ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:19.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:19.424+0000 7f18cce00700 1 --2- 172.21.15.28:0/4111268553 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18c8101dd0 0x7f18c819ec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:19.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:19.424+0000 7f18cce00700 1 -- 172.21.15.28:0/4111268553 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18c8199020 con 0x7f18c8101420 2024-05-12T22:07:19.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:19.424+0000 7f18c7e00700 1 --2- 172.21.15.28:0/4111268553 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18c8101dd0 0x7f18c819ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:07:19.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:19.425+0000 7f18cce00700 1 --2- 172.21.15.28:0/4111268553 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f18c8101420 0x7f18c819e700 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f18bc0098e0 tx=0x7f18bc00b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:07:19.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:19.425+0000 7f18c5600700 1 -- 172.21.15.28:0/4111268553 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18bc027070 con 0x7f18c8101420 2024-05-12T22:07:19.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:19.425+0000 7f18ce200700 1 -- 172.21.15.28:0/4111268553 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18c81992b0 con 0x7f18c8101420 2024-05-12T22:07:19.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:19.425+0000 7f18c5600700 1 -- 172.21.15.28:0/4111268553 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f18bc00fd10 con 0x7f18c8101420 2024-05-12T22:07:19.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:19.425+0000 7f18c5600700 1 -- 172.21.15.28:0/4111268553 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18bc0173f0 con 0x7f18c8101420 2024-05-12T22:07:19.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:19.425+0000 7f18ce200700 1 -- 172.21.15.28:0/4111268553 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18c81a55d0 con 0x7f18c8101420 2024-05-12T22:07:19.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:19.426+0000 7f18ce200700 1 -- 172.21.15.28:0/4111268553 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f18c80629f0 con 0x7f18c8101420 2024-05-12T22:07:19.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:19.428+0000 7f18c5600700 1 -- 172.21.15.28:0/4111268553 <== mon.2 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f18bc02a070 con 0x7f18c8101420 2024-05-12T22:07:19.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:19.429+0000 7f18c5600700 1 --2- 172.21.15.28:0/4111268553 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f18b00a2cb0 0x7f18b00a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:07:19.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:19.429+0000 7f18c5600700 1 -- 172.21.15.28:0/4111268553 <== mon.2 v2:172.21.15.86:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f18bc0d0510 con 0x7f18c8101420 2024-05-12T22:07:19.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:19.430+0000 7f18c7e00700 1 --2- 172.21.15.28:0/4111268553 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f18b00a2cb0 0x7f18b00a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:07:19.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:19.430+0000 7f18c5600700 1 --2- 172.21.15.28:0/4111268553 >> [v2:172.21.15.186:6800/906459365,v1:172.21.15.186:6801/906459365] conn(0x7f18b00ac7f0 0x7f18b00aec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:07:19.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:19.430+0000 7f18c5600700 1 -- 172.21.15.28:0/4111268553 --> [v2:172.21.15.186:6800/906459365,v1:172.21.15.186:6801/906459365] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f18bc010ea0 con 0x7f18b00ac7f0 2024-05-12T22:07:19.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:19.430+0000 7f18c5600700 1 -- 172.21.15.28:0/4111268553 <== mon.2 v2:172.21.15.86:3300/0 6 ==== mon_get_version_reply(handle=1 version=88) ==== 24+0+0 (secure 0 0 0) 0x7f18bc0d08c0 con 0x7f18c8101420 2024-05-12T22:07:19.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:19.430+0000 7f18cd800700 1 --2- 172.21.15.28:0/4111268553 >> [v2:172.21.15.186:6800/906459365,v1:172.21.15.186:6801/906459365] conn(0x7f18b00ac7f0 0x7f18b00aec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:07:19.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:19.430+0000 7f18c7e00700 1 --2- 172.21.15.28:0/4111268553 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f18b00a2cb0 0x7f18b00a5170 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f18c80fe2f0 tx=0x7f18b4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:07:19.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:19.431+0000 7f18cd800700 1 --2- 172.21.15.28:0/4111268553 >> [v2:172.21.15.186:6800/906459365,v1:172.21.15.186:6801/906459365] conn(0x7f18b00ac7f0 0x7f18b00aec90 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:07:19.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:19.432+0000 7f18c5600700 1 -- 172.21.15.28:0/4111268553 <== osd.2 v2:172.21.15.186:6800/906459365 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f18bc010ea0 con 0x7f18b00ac7f0 2024-05-12T22:07:19.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:19.449+0000 7f18ce200700 1 -- 172.21.15.28:0/4111268553 --> [v2:172.21.15.186:6800/906459365,v1:172.21.15.186:6801/906459365] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f18c8062e20 con 0x7f18b00ac7f0 2024-05-12T22:07:19.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:19.450+0000 7f18c5600700 1 -- 172.21.15.28:0/4111268553 <== osd.2 v2:172.21.15.186:6800/906459365 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f18c8062e20 con 0x7f18b00ac7f0 2024-05-12T22:07:19.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:19.451+0000 7f18ce200700 1 -- 172.21.15.28:0/4111268553 >> [v2:172.21.15.186:6800/906459365,v1:172.21.15.186:6801/906459365] conn(0x7f18b00ac7f0 msgr2=0x7f18b00aec90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:19.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:19.451+0000 7f18ce200700 1 --2- 172.21.15.28:0/4111268553 >> [v2:172.21.15.186:6800/906459365,v1:172.21.15.186:6801/906459365] conn(0x7f18b00ac7f0 0x7f18b00aec90 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:19.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:19.451+0000 7f18ce200700 1 -- 172.21.15.28:0/4111268553 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f18b00a2cb0 msgr2=0x7f18b00a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:19.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:19.451+0000 7f18ce200700 1 --2- 172.21.15.28:0/4111268553 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f18b00a2cb0 0x7f18b00a5170 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f18c80fe2f0 tx=0x7f18b4009260 comp rx=0 tx=0).stop 2024-05-12T22:07:19.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:19.451+0000 7f18ce200700 1 -- 172.21.15.28:0/4111268553 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f18c8101420 msgr2=0x7f18c819e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:19.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:19.451+0000 7f18ce200700 1 --2- 172.21.15.28:0/4111268553 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f18c8101420 0x7f18c819e700 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f18bc0098e0 tx=0x7f18bc00b840 comp rx=0 tx=0).stop 2024-05-12T22:07:19.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:19.451+0000 7f18cd800700 1 -- 172.21.15.28:0/4111268553 reap_dead start 2024-05-12T22:07:19.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:19.451+0000 7f18ce200700 1 -- 172.21.15.28:0/4111268553 shutdown_connections 2024-05-12T22:07:19.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:19.451+0000 7f18ce200700 1 -- 172.21.15.28:0/4111268553 >> 172.21.15.28:0/4111268553 conn(0x7f18c80fccd0 msgr2=0x7f18c81053a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:07:19.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:19.451+0000 7f18ce200700 1 -- 172.21.15.28:0/4111268553 shutdown_connections 2024-05-12T22:07:19.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:19.451+0000 7f18ce200700 1 -- 172.21.15.28:0/4111268553 wait complete. 2024-05-12T22:07:19.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:07:19 smithi186 ceph-mon[34785]: pgmap v334: 97 pgs: 97 active+clean; 640 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 255 B/s wr, 6 op/s 2024-05-12T22:07:19.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:07:19 smithi086 ceph-mon[34297]: pgmap v334: 97 pgs: 97 active+clean; 640 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 255 B/s wr, 6 op/s 2024-05-12T22:07:20.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:19.723+0000 7faba0400700 1 -- 172.21.15.28:0/1438191063 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fab98074550 msgr2=0x7fab98074930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:20.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:19.723+0000 7faba0400700 1 --2- 172.21.15.28:0/1438191063 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fab98074550 0x7fab98074930 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fab8c009990 tx=0x7fab8c009ca0 comp rx=0 tx=0).stop 2024-05-12T22:07:20.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:19.723+0000 7faba0400700 1 -- 172.21.15.28:0/1438191063 shutdown_connections 2024-05-12T22:07:20.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:19.723+0000 7faba0400700 1 --2- 172.21.15.28:0/1438191063 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab9810ed90 0x7fab98111180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:20.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:19.723+0000 7faba0400700 1 --2- 172.21.15.28:0/1438191063 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fab98074550 0x7fab98074930 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:20.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:19.723+0000 7faba0400700 1 --2- 172.21.15.28:0/1438191063 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fab98074e70 0x7fab9810e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:20.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:19.723+0000 7faba0400700 1 -- 172.21.15.28:0/1438191063 >> 172.21.15.28:0/1438191063 conn(0x7fab980794e0 msgr2=0x7fab980798f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:07:20.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:19.723+0000 7faba0400700 1 -- 172.21.15.28:0/1438191063 shutdown_connections 2024-05-12T22:07:20.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:19.723+0000 7faba0400700 1 -- 172.21.15.28:0/1438191063 wait complete. 2024-05-12T22:07:20.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:19.725+0000 7faba0400700 1 Processor -- start 2024-05-12T22:07:20.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:19.725+0000 7faba0400700 1 -- start start 2024-05-12T22:07:20.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:19.725+0000 7faba0400700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fab98074550 0x7fab981a0a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:07:20.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:19.725+0000 7faba0400700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fab98074e70 0x7fab981a0f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:07:20.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:19.725+0000 7faba0400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab9810ed90 0x7fab9819ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:07:20.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:19.725+0000 7faba0400700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fab98114860 con 0x7fab9810ed90 2024-05-12T22:07:20.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:19.725+0000 7faba0400700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fab981146e0 con 0x7fab98074550 2024-05-12T22:07:20.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:19.725+0000 7faba0400700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fab981149e0 con 0x7fab98074e70 2024-05-12T22:07:20.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:19.726+0000 7fab9da00700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fab98074550 0x7fab981a0a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:07:20.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:19.726+0000 7fab9da00700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fab98074550 0x7fab981a0a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:35294/0 (socket says 172.21.15.28:35294) 2024-05-12T22:07:20.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:19.726+0000 7fab9da00700 1 -- 172.21.15.28:0/53518926 learned_addr learned my addr 172.21.15.28:0/53518926 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:07:20.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:19.726+0000 7fab9e400700 1 --2- 172.21.15.28:0/53518926 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab9810ed90 0x7fab9819ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:07:20.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:19.726+0000 7fab9d000700 1 --2- 172.21.15.28:0/53518926 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fab98074e70 0x7fab981a0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:07:20.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:19.726+0000 7fab9da00700 1 -- 172.21.15.28:0/53518926 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fab98074e70 msgr2=0x7fab981a0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:20.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:19.726+0000 7fab9da00700 1 --2- 172.21.15.28:0/53518926 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fab98074e70 0x7fab981a0f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:20.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:19.726+0000 7fab9da00700 1 -- 172.21.15.28:0/53518926 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab9810ed90 msgr2=0x7fab9819ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:20.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:19.726+0000 7fab9da00700 1 --2- 172.21.15.28:0/53518926 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab9810ed90 0x7fab9819ab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:20.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:19.726+0000 7fab9da00700 1 -- 172.21.15.28:0/53518926 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab9819b400 con 0x7fab98074550 2024-05-12T22:07:20.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:19.726+0000 7fab9da00700 1 --2- 172.21.15.28:0/53518926 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fab98074550 0x7fab981a0a50 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fab8c000c00 tx=0x7fab8c00bab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:07:20.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:19.727+0000 7fab96000700 1 -- 172.21.15.28:0/53518926 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab8c027070 con 0x7fab98074550 2024-05-12T22:07:20.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:19.727+0000 7faba0400700 1 -- 172.21.15.28:0/53518926 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab9819b690 con 0x7fab98074550 2024-05-12T22:07:20.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:19.727+0000 7fab96000700 1 -- 172.21.15.28:0/53518926 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fab8c0056d0 con 0x7fab98074550 2024-05-12T22:07:20.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:19.727+0000 7fab96000700 1 -- 172.21.15.28:0/53518926 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab8c022e10 con 0x7fab98074550 2024-05-12T22:07:20.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:19.727+0000 7faba0400700 1 -- 172.21.15.28:0/53518926 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab981a79d0 con 0x7fab98074550 2024-05-12T22:07:20.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:19.727+0000 7faba0400700 1 -- 172.21.15.28:0/53518926 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fab980fff80 con 0x7fab98074550 2024-05-12T22:07:20.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:19.730+0000 7fab96000700 1 -- 172.21.15.28:0/53518926 <== mon.2 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fab8c02a070 con 0x7fab98074550 2024-05-12T22:07:20.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:19.731+0000 7fab96000700 1 --2- 172.21.15.28:0/53518926 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fab800a2cb0 0x7fab800a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:07:20.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:19.731+0000 7fab96000700 1 -- 172.21.15.28:0/53518926 <== mon.2 v2:172.21.15.86:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7fab8c0d0770 con 0x7fab98074550 2024-05-12T22:07:20.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:19.731+0000 7fab9d000700 1 --2- 172.21.15.28:0/53518926 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fab800a2cb0 0x7fab800a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:07:20.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:19.732+0000 7fab96000700 1 --2- 172.21.15.28:0/53518926 >> [v2:172.21.15.28:6810/4182884436,v1:172.21.15.28:6811/4182884436] conn(0x7fab800ac7f0 0x7fab800aec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:07:20.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:19.732+0000 7fab96000700 1 -- 172.21.15.28:0/53518926 --> [v2:172.21.15.28:6810/4182884436,v1:172.21.15.28:6811/4182884436] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fab8c001d30 con 0x7fab800ac7f0 2024-05-12T22:07:20.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:19.732+0000 7fab96000700 1 -- 172.21.15.28:0/53518926 <== mon.2 v2:172.21.15.86:3300/0 6 ==== mon_get_version_reply(handle=1 version=88) ==== 24+0+0 (secure 0 0 0) 0x7fab8c0d0b20 con 0x7fab98074550 2024-05-12T22:07:20.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:19.732+0000 7fab9e400700 1 --2- 172.21.15.28:0/53518926 >> [v2:172.21.15.28:6810/4182884436,v1:172.21.15.28:6811/4182884436] conn(0x7fab800ac7f0 0x7fab800aec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:07:20.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:19.732+0000 7fab9d000700 1 --2- 172.21.15.28:0/53518926 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fab800a2cb0 0x7fab800a5170 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fab9819c270 tx=0x7fab84009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:07:20.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:19.732+0000 7fab9e400700 1 --2- 172.21.15.28:0/53518926 >> [v2:172.21.15.28:6810/4182884436,v1:172.21.15.28:6811/4182884436] conn(0x7fab800ac7f0 0x7fab800aec90 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:07:20.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:19.733+0000 7fab96000700 1 -- 172.21.15.28:0/53518926 <== osd.3 v2:172.21.15.28:6810/4182884436 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fab8c001d30 con 0x7fab800ac7f0 2024-05-12T22:07:20.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:19.751+0000 7faba0400700 1 -- 172.21.15.28:0/53518926 --> [v2:172.21.15.28:6810/4182884436,v1:172.21.15.28:6811/4182884436] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fab98075ba0 con 0x7fab800ac7f0 2024-05-12T22:07:20.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:19.752+0000 7fab96000700 1 -- 172.21.15.28:0/53518926 <== osd.3 v2:172.21.15.28:6810/4182884436 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fab98075ba0 con 0x7fab800ac7f0 2024-05-12T22:07:20.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:19.752+0000 7faba0400700 1 -- 172.21.15.28:0/53518926 >> [v2:172.21.15.28:6810/4182884436,v1:172.21.15.28:6811/4182884436] conn(0x7fab800ac7f0 msgr2=0x7fab800aec90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:20.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:19.753+0000 7faba0400700 1 --2- 172.21.15.28:0/53518926 >> [v2:172.21.15.28:6810/4182884436,v1:172.21.15.28:6811/4182884436] conn(0x7fab800ac7f0 0x7fab800aec90 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:20.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:19.753+0000 7faba0400700 1 -- 172.21.15.28:0/53518926 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fab800a2cb0 msgr2=0x7fab800a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:20.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:19.753+0000 7faba0400700 1 --2- 172.21.15.28:0/53518926 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fab800a2cb0 0x7fab800a5170 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fab9819c270 tx=0x7fab84009260 comp rx=0 tx=0).stop 2024-05-12T22:07:20.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:19.753+0000 7faba0400700 1 -- 172.21.15.28:0/53518926 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fab98074550 msgr2=0x7fab981a0a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:20.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:19.753+0000 7faba0400700 1 --2- 172.21.15.28:0/53518926 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fab98074550 0x7fab981a0a50 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fab8c000c00 tx=0x7fab8c00bab0 comp rx=0 tx=0).stop 2024-05-12T22:07:20.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:19.753+0000 7fab9e400700 1 -- 172.21.15.28:0/53518926 reap_dead start 2024-05-12T22:07:20.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:19.753+0000 7faba0400700 1 -- 172.21.15.28:0/53518926 shutdown_connections 2024-05-12T22:07:20.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:19.753+0000 7faba0400700 1 -- 172.21.15.28:0/53518926 >> 172.21.15.28:0/53518926 conn(0x7fab980794e0 msgr2=0x7fab980ff560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:07:20.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:19.753+0000 7faba0400700 1 -- 172.21.15.28:0/53518926 shutdown_connections 2024-05-12T22:07:20.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:19.753+0000 7faba0400700 1 -- 172.21.15.28:0/53518926 wait complete. 2024-05-12T22:07:20.282 INFO:teuthology.orchestra.run.smithi028.stdout:257698037799 2024-05-12T22:07:20.282 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f72fecff68e1d400c4568684327c900485c20d6a shell --fsid 6b92e7b4-10aa-11ef-bc98-c7b262605968 -- ceph osd last-stat-seq osd.8 2024-05-12T22:07:21.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:20.903+0000 7fda1b000700 1 -- 172.21.15.28:0/2809744945 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fda141083a0 msgr2=0x7fda14108780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:21.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:20.903+0000 7fda1b000700 1 --2- 172.21.15.28:0/2809744945 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fda141083a0 0x7fda14108780 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fda040098f0 tx=0x7fda04009c00 comp rx=0 tx=0).stop 2024-05-12T22:07:21.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:20.904+0000 7fda1b000700 1 -- 172.21.15.28:0/2809744945 shutdown_connections 2024-05-12T22:07:21.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:20.904+0000 7fda1b000700 1 --2- 172.21.15.28:0/2809744945 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda1406a9d0 0x7fda1410f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:21.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:20.904+0000 7fda1b000700 1 --2- 172.21.15.28:0/2809744945 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fda141083a0 0x7fda14108780 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:21.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:20.904+0000 7fda1b000700 1 --2- 172.21.15.28:0/2809744945 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fda1406a030 0x7fda1406a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:21.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:20.904+0000 7fda1b000700 1 -- 172.21.15.28:0/2809744945 >> 172.21.15.28:0/2809744945 conn(0x7fda140781d0 msgr2=0x7fda140785e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:07:21.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:20.904+0000 7fda1b000700 1 -- 172.21.15.28:0/2809744945 shutdown_connections 2024-05-12T22:07:21.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:20.904+0000 7fda1b000700 1 -- 172.21.15.28:0/2809744945 wait complete. 2024-05-12T22:07:21.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:20.905+0000 7fda1b000700 1 Processor -- start 2024-05-12T22:07:21.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:20.905+0000 7fda1b000700 1 -- start start 2024-05-12T22:07:21.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:20.905+0000 7fda1b000700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fda1406a030 0x7fda1419ad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:07:21.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:20.905+0000 7fda1b000700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fda1406a9d0 0x7fda1419b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:07:21.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:20.905+0000 7fda1b000700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda141083a0 0x7fda1419f650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:07:21.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:20.905+0000 7fda1b000700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fda14112630 con 0x7fda141083a0 2024-05-12T22:07:21.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:20.905+0000 7fda1b000700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fda141124b0 con 0x7fda1406a030 2024-05-12T22:07:21.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:20.905+0000 7fda1b000700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fda141127b0 con 0x7fda1406a9d0 2024-05-12T22:07:21.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:20.906+0000 7fda19000700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda141083a0 0x7fda1419f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:07:21.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:20.906+0000 7fda19000700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda141083a0 0x7fda1419f650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55170/0 (socket says 172.21.15.28:55170) 2024-05-12T22:07:21.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:20.906+0000 7fda13e00700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fda1406a030 0x7fda1419ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:07:21.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:20.906+0000 7fda19000700 1 -- 172.21.15.28:0/4024634337 learned_addr learned my addr 172.21.15.28:0/4024634337 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:07:21.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:20.906+0000 7fda0be00700 1 --2- 172.21.15.28:0/4024634337 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fda1406a9d0 0x7fda1419b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:07:21.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:20.906+0000 7fda19000700 1 -- 172.21.15.28:0/4024634337 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fda1406a9d0 msgr2=0x7fda1419b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:21.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:20.906+0000 7fda19000700 1 --2- 172.21.15.28:0/4024634337 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fda1406a9d0 0x7fda1419b2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:21.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:20.906+0000 7fda19000700 1 -- 172.21.15.28:0/4024634337 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fda1406a030 msgr2=0x7fda1419ad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:21.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:20.906+0000 7fda19000700 1 --2- 172.21.15.28:0/4024634337 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fda1406a030 0x7fda1419ad80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:21.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:20.906+0000 7fda19000700 1 -- 172.21.15.28:0/4024634337 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda1419fd90 con 0x7fda141083a0 2024-05-12T22:07:21.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:20.906+0000 7fda13e00700 1 --2- 172.21.15.28:0/4024634337 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fda1406a030 0x7fda1419ad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:07:21.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:20.906+0000 7fda19000700 1 --2- 172.21.15.28:0/4024634337 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda141083a0 0x7fda1419f650 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fda0c00ef30 tx=0x7fda0c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:07:21.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:20.907+0000 7fda11600700 1 -- 172.21.15.28:0/4024634337 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda0c0092e0 con 0x7fda141083a0 2024-05-12T22:07:21.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:20.907+0000 7fda1b000700 1 -- 172.21.15.28:0/4024634337 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda141a0020 con 0x7fda141083a0 2024-05-12T22:07:21.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:20.907+0000 7fda11600700 1 -- 172.21.15.28:0/4024634337 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fda0c004bb0 con 0x7fda141083a0 2024-05-12T22:07:21.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:20.907+0000 7fda11600700 1 -- 172.21.15.28:0/4024634337 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda0c004280 con 0x7fda141083a0 2024-05-12T22:07:21.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:20.907+0000 7fda1b000700 1 -- 172.21.15.28:0/4024634337 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda141a7900 con 0x7fda141083a0 2024-05-12T22:07:21.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:20.908+0000 7fda1b000700 1 -- 172.21.15.28:0/4024634337 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fda1410df90 con 0x7fda141083a0 2024-05-12T22:07:21.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:20.910+0000 7fda11600700 1 -- 172.21.15.28:0/4024634337 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fda0c01b030 con 0x7fda141083a0 2024-05-12T22:07:21.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:20.911+0000 7fda11600700 1 --2- 172.21.15.28:0/4024634337 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fd9f80a2cb0 0x7fd9f80a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:07:21.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:20.912+0000 7fda11600700 1 -- 172.21.15.28:0/4024634337 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7fda0c0cf5d0 con 0x7fda141083a0 2024-05-12T22:07:21.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:20.912+0000 7fda13e00700 1 --2- 172.21.15.28:0/4024634337 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fd9f80a2cb0 0x7fd9f80a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:07:21.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:20.912+0000 7fda11600700 1 --2- 172.21.15.28:0/4024634337 >> [v2:172.21.15.28:6818/1642308925,v1:172.21.15.28:6819/1642308925] conn(0x7fd9f80ac7f0 0x7fd9f80aec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:07:21.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:20.912+0000 7fda11600700 1 -- 172.21.15.28:0/4024634337 --> [v2:172.21.15.28:6818/1642308925,v1:172.21.15.28:6819/1642308925] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fda0c01fe50 con 0x7fd9f80ac7f0 2024-05-12T22:07:21.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:20.912+0000 7fda11600700 1 -- 172.21.15.28:0/4024634337 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=88) ==== 24+0+0 (secure 0 0 0) 0x7fda0c0cf980 con 0x7fda141083a0 2024-05-12T22:07:21.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:20.912+0000 7fda0be00700 1 --2- 172.21.15.28:0/4024634337 >> [v2:172.21.15.28:6818/1642308925,v1:172.21.15.28:6819/1642308925] conn(0x7fd9f80ac7f0 0x7fd9f80aec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:07:21.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:20.913+0000 7fda13e00700 1 --2- 172.21.15.28:0/4024634337 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fd9f80a2cb0 0x7fd9f80a5170 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fda04009810 tx=0x7fda04019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:07:21.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:20.913+0000 7fda0be00700 1 --2- 172.21.15.28:0/4024634337 >> [v2:172.21.15.28:6818/1642308925,v1:172.21.15.28:6819/1642308925] conn(0x7fd9f80ac7f0 0x7fd9f80aec90 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:07:21.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:20.914+0000 7fda11600700 1 -- 172.21.15.28:0/4024634337 <== osd.6 v2:172.21.15.28:6818/1642308925 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fda0c01fe50 con 0x7fd9f80ac7f0 2024-05-12T22:07:21.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:20.933+0000 7fda1b000700 1 -- 172.21.15.28:0/4024634337 --> [v2:172.21.15.28:6818/1642308925,v1:172.21.15.28:6819/1642308925] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fda14103a50 con 0x7fd9f80ac7f0 2024-05-12T22:07:21.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:20.934+0000 7fda11600700 1 -- 172.21.15.28:0/4024634337 <== osd.6 v2:172.21.15.28:6818/1642308925 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fda14103a50 con 0x7fd9f80ac7f0 2024-05-12T22:07:21.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:20.934+0000 7fda1b000700 1 -- 172.21.15.28:0/4024634337 >> [v2:172.21.15.28:6818/1642308925,v1:172.21.15.28:6819/1642308925] conn(0x7fd9f80ac7f0 msgr2=0x7fd9f80aec90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:21.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:20.934+0000 7fda1b000700 1 --2- 172.21.15.28:0/4024634337 >> [v2:172.21.15.28:6818/1642308925,v1:172.21.15.28:6819/1642308925] conn(0x7fd9f80ac7f0 0x7fd9f80aec90 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:21.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:20.935+0000 7fda1b000700 1 -- 172.21.15.28:0/4024634337 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fd9f80a2cb0 msgr2=0x7fd9f80a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:21.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:20.935+0000 7fda1b000700 1 --2- 172.21.15.28:0/4024634337 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fd9f80a2cb0 0x7fd9f80a5170 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fda04009810 tx=0x7fda04019040 comp rx=0 tx=0).stop 2024-05-12T22:07:21.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:20.935+0000 7fda1b000700 1 -- 172.21.15.28:0/4024634337 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda141083a0 msgr2=0x7fda1419f650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:21.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:20.935+0000 7fda1b000700 1 --2- 172.21.15.28:0/4024634337 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda141083a0 0x7fda1419f650 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fda0c00ef30 tx=0x7fda0c00c3a0 comp rx=0 tx=0).stop 2024-05-12T22:07:21.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:20.935+0000 7fda19000700 1 -- 172.21.15.28:0/4024634337 reap_dead start 2024-05-12T22:07:21.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:20.935+0000 7fda1b000700 1 -- 172.21.15.28:0/4024634337 shutdown_connections 2024-05-12T22:07:21.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:20.935+0000 7fda1b000700 1 -- 172.21.15.28:0/4024634337 >> 172.21.15.28:0/4024634337 conn(0x7fda140781d0 msgr2=0x7fda1410d530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:07:21.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:20.935+0000 7fda1b000700 1 -- 172.21.15.28:0/4024634337 shutdown_connections 2024-05-12T22:07:21.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:20.936+0000 7fda1b000700 1 -- 172.21.15.28:0/4024634337 wait complete. 2024-05-12T22:07:21.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:07:21 smithi186 ceph-mon[34785]: pgmap v335: 97 pgs: 97 active+clean; 640 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 255 B/s wr, 6 op/s 2024-05-12T22:07:21.734 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:07:21 smithi028 ceph-mon[29847]: pgmap v335: 97 pgs: 97 active+clean; 640 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 255 B/s wr, 6 op/s 2024-05-12T22:07:21.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:07:21 smithi086 ceph-mon[34297]: pgmap v335: 97 pgs: 97 active+clean; 640 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 255 B/s wr, 6 op/s 2024-05-12T22:07:21.915 INFO:teuthology.orchestra.run.smithi028.stdout:55834574924 2024-05-12T22:07:21.915 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f72fecff68e1d400c4568684327c900485c20d6a shell --fsid 6b92e7b4-10aa-11ef-bc98-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-05-12T22:07:22.888 INFO:teuthology.orchestra.run.smithi028.stdout:111669149762 2024-05-12T22:07:22.889 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f72fecff68e1d400c4568684327c900485c20d6a shell --fsid 6b92e7b4-10aa-11ef-bc98-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-05-12T22:07:23.276 INFO:teuthology.orchestra.run.smithi028.stdout:313532612634 2024-05-12T22:07:23.277 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f72fecff68e1d400c4568684327c900485c20d6a shell --fsid 6b92e7b4-10aa-11ef-bc98-c7b262605968 -- ceph osd last-stat-seq osd.10 2024-05-12T22:07:23.511 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:07:23 smithi028 ceph-mon[29847]: pgmap v336: 97 pgs: 97 active+clean; 640 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-05-12T22:07:23.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:07:23 smithi186 ceph-mon[34785]: pgmap v336: 97 pgs: 97 active+clean; 640 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-05-12T22:07:23.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:07:23 smithi086 ceph-mon[34297]: pgmap v336: 97 pgs: 97 active+clean; 640 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-05-12T22:07:24.170 INFO:teuthology.orchestra.run.smithi028.stdout:77309411400 2024-05-12T22:07:24.171 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f72fecff68e1d400c4568684327c900485c20d6a shell --fsid 6b92e7b4-10aa-11ef-bc98-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-05-12T22:07:25.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:24.778+0000 7f83c8c00700 1 -- 172.21.15.28:0/1729392639 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83c406a9d0 msgr2=0x7f83c410f1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:25.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:24.778+0000 7f83c8c00700 1 --2- 172.21.15.28:0/1729392639 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83c406a9d0 0x7f83c410f1e0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f83bc009990 tx=0x7f83bc009ca0 comp rx=0 tx=0).stop 2024-05-12T22:07:25.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:24.778+0000 7f83c8c00700 1 -- 172.21.15.28:0/1729392639 shutdown_connections 2024-05-12T22:07:25.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:24.778+0000 7f83c8c00700 1 --2- 172.21.15.28:0/1729392639 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f83c41083a0 0x7f83c4108780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:25.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:24.778+0000 7f83c8c00700 1 --2- 172.21.15.28:0/1729392639 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83c406a9d0 0x7f83c410f1e0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:25.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:24.778+0000 7f83c8c00700 1 --2- 172.21.15.28:0/1729392639 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f83c406a030 0x7f83c406a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:25.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:24.778+0000 7f83c8c00700 1 -- 172.21.15.28:0/1729392639 >> 172.21.15.28:0/1729392639 conn(0x7f83c40781d0 msgr2=0x7f83c40785e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:07:25.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:24.778+0000 7f83c8c00700 1 -- 172.21.15.28:0/1729392639 shutdown_connections 2024-05-12T22:07:25.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:24.778+0000 7f83c8c00700 1 -- 172.21.15.28:0/1729392639 wait complete. 2024-05-12T22:07:25.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:24.779+0000 7f83c8c00700 1 Processor -- start 2024-05-12T22:07:25.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:24.779+0000 7f83c8c00700 1 -- start start 2024-05-12T22:07:25.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:24.779+0000 7f83c8c00700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f83c406a030 0x7f83c41012f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:07:25.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:24.779+0000 7f83c8c00700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f83c406a9d0 0x7f83c4101830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:07:25.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:24.780+0000 7f83c8c00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83c41083a0 0x7f83c4101d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:07:25.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:24.780+0000 7f83c8c00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f83c4112640 con 0x7f83c41083a0 2024-05-12T22:07:25.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:24.780+0000 7f83c8c00700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f83c41124c0 con 0x7f83c406a030 2024-05-12T22:07:25.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:24.780+0000 7f83c8c00700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f83c41127c0 con 0x7f83c406a9d0 2024-05-12T22:07:25.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:24.780+0000 7f83c1e00700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f83c406a030 0x7f83c41012f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:07:25.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:24.780+0000 7f83c1e00700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f83c406a030 0x7f83c41012f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:35380/0 (socket says 172.21.15.28:35380) 2024-05-12T22:07:25.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:24.780+0000 7f83b9400700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f83c406a9d0 0x7f83c4101830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:07:25.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:24.780+0000 7f83c1e00700 1 -- 172.21.15.28:0/3467840941 learned_addr learned my addr 172.21.15.28:0/3467840941 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:07:25.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:24.780+0000 7f83c2800700 1 --2- 172.21.15.28:0/3467840941 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83c41083a0 0x7f83c4101d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:07:25.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:24.780+0000 7f83b9400700 1 -- 172.21.15.28:0/3467840941 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f83c406a030 msgr2=0x7f83c41012f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:25.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:24.780+0000 7f83b9400700 1 --2- 172.21.15.28:0/3467840941 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f83c406a030 0x7f83c41012f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:25.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:24.780+0000 7f83b9400700 1 -- 172.21.15.28:0/3467840941 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83c41083a0 msgr2=0x7f83c4101d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:25.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:24.780+0000 7f83b9400700 1 --2- 172.21.15.28:0/3467840941 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83c41083a0 0x7f83c4101d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:25.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:24.780+0000 7f83b9400700 1 -- 172.21.15.28:0/3467840941 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83c41a7070 con 0x7f83c406a9d0 2024-05-12T22:07:25.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:24.780+0000 7f83c1e00700 1 --2- 172.21.15.28:0/3467840941 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f83c406a030 0x7f83c41012f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:07:25.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:24.781+0000 7f83b9400700 1 --2- 172.21.15.28:0/3467840941 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f83c406a9d0 0x7f83c4101830 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f83ac00ef10 tx=0x7f83ac00c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:07:25.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:24.781+0000 7f83bb400700 1 -- 172.21.15.28:0/3467840941 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83ac00c9c0 con 0x7f83c406a9d0 2024-05-12T22:07:25.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:24.781+0000 7f83c8c00700 1 -- 172.21.15.28:0/3467840941 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83c41a7300 con 0x7f83c406a9d0 2024-05-12T22:07:25.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:24.781+0000 7f83bb400700 1 -- 172.21.15.28:0/3467840941 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f83ac019070 con 0x7f83c406a9d0 2024-05-12T22:07:25.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:24.781+0000 7f83bb400700 1 -- 172.21.15.28:0/3467840941 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83ac0146e0 con 0x7f83c406a9d0 2024-05-12T22:07:25.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:24.781+0000 7f83c8c00700 1 -- 172.21.15.28:0/3467840941 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83c41a7890 con 0x7f83c406a9d0 2024-05-12T22:07:25.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:24.782+0000 7f83c8c00700 1 -- 172.21.15.28:0/3467840941 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f83c40629f0 con 0x7f83c406a9d0 2024-05-12T22:07:25.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:24.784+0000 7f83bb400700 1 -- 172.21.15.28:0/3467840941 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f83ac012050 con 0x7f83c406a9d0 2024-05-12T22:07:25.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:24.786+0000 7f83bb400700 1 --2- 172.21.15.28:0/3467840941 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f83a80a2cb0 0x7f83a80a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:07:25.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:24.786+0000 7f83bb400700 1 -- 172.21.15.28:0/3467840941 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f83ac0cfc50 con 0x7f83c406a9d0 2024-05-12T22:07:25.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:24.786+0000 7f83c1e00700 1 --2- 172.21.15.28:0/3467840941 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f83a80a2cb0 0x7f83a80a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:07:25.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:24.786+0000 7f83bb400700 1 --2- 172.21.15.28:0/3467840941 >> [v2:172.21.15.186:6808/1944205756,v1:172.21.15.186:6809/1944205756] conn(0x7f83a80ac7f0 0x7f83a80aec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:07:25.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:24.786+0000 7f83bb400700 1 -- 172.21.15.28:0/3467840941 --> [v2:172.21.15.186:6808/1944205756,v1:172.21.15.186:6809/1944205756] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f83ac00be50 con 0x7f83a80ac7f0 2024-05-12T22:07:25.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:24.786+0000 7f83bb400700 1 -- 172.21.15.28:0/3467840941 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_get_version_reply(handle=1 version=88) ==== 24+0+0 (secure 0 0 0) 0x7f83ac0d0000 con 0x7f83c406a9d0 2024-05-12T22:07:25.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:24.787+0000 7f83c2800700 1 --2- 172.21.15.28:0/3467840941 >> [v2:172.21.15.186:6808/1944205756,v1:172.21.15.186:6809/1944205756] conn(0x7f83a80ac7f0 0x7f83a80aec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:07:25.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:24.787+0000 7f83c1e00700 1 --2- 172.21.15.28:0/3467840941 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f83a80a2cb0 0x7f83a80a5170 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f83b4009680 tx=0x7f83b4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:07:25.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:24.787+0000 7f83c2800700 1 --2- 172.21.15.28:0/3467840941 >> [v2:172.21.15.186:6808/1944205756,v1:172.21.15.186:6809/1944205756] conn(0x7f83a80ac7f0 0x7f83a80aec90 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:07:25.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:24.788+0000 7f83bb400700 1 -- 172.21.15.28:0/3467840941 <== osd.5 v2:172.21.15.186:6808/1944205756 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f83ac00be50 con 0x7f83a80ac7f0 2024-05-12T22:07:25.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:24.806+0000 7f83c8c00700 1 -- 172.21.15.28:0/3467840941 --> [v2:172.21.15.186:6808/1944205756,v1:172.21.15.186:6809/1944205756] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f83c4062df0 con 0x7f83a80ac7f0 2024-05-12T22:07:25.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:24.807+0000 7f83bb400700 1 -- 172.21.15.28:0/3467840941 <== osd.5 v2:172.21.15.186:6808/1944205756 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f83c4062df0 con 0x7f83a80ac7f0 2024-05-12T22:07:25.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:24.808+0000 7f83c8c00700 1 -- 172.21.15.28:0/3467840941 >> [v2:172.21.15.186:6808/1944205756,v1:172.21.15.186:6809/1944205756] conn(0x7f83a80ac7f0 msgr2=0x7f83a80aec90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:25.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:24.808+0000 7f83c8c00700 1 --2- 172.21.15.28:0/3467840941 >> [v2:172.21.15.186:6808/1944205756,v1:172.21.15.186:6809/1944205756] conn(0x7f83a80ac7f0 0x7f83a80aec90 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:25.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:24.808+0000 7f83c8c00700 1 -- 172.21.15.28:0/3467840941 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f83a80a2cb0 msgr2=0x7f83a80a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:25.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:24.808+0000 7f83c8c00700 1 --2- 172.21.15.28:0/3467840941 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f83a80a2cb0 0x7f83a80a5170 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f83b4009680 tx=0x7f83b4009260 comp rx=0 tx=0).stop 2024-05-12T22:07:25.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:24.808+0000 7f83c8c00700 1 -- 172.21.15.28:0/3467840941 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f83c406a9d0 msgr2=0x7f83c4101830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:25.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:24.808+0000 7f83c8c00700 1 --2- 172.21.15.28:0/3467840941 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f83c406a9d0 0x7f83c4101830 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f83ac00ef10 tx=0x7f83ac00c3a0 comp rx=0 tx=0).stop 2024-05-12T22:07:25.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:24.808+0000 7f83c2800700 1 -- 172.21.15.28:0/3467840941 reap_dead start 2024-05-12T22:07:25.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:24.808+0000 7f83c8c00700 1 -- 172.21.15.28:0/3467840941 shutdown_connections 2024-05-12T22:07:25.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:24.808+0000 7f83c8c00700 1 -- 172.21.15.28:0/3467840941 >> 172.21.15.28:0/3467840941 conn(0x7f83c40781d0 msgr2=0x7f83c40ffda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:07:25.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:24.808+0000 7f83c8c00700 1 -- 172.21.15.28:0/3467840941 shutdown_connections 2024-05-12T22:07:25.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:24.808+0000 7f83c8c00700 1 -- 172.21.15.28:0/3467840941 wait complete. 2024-05-12T22:07:25.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:24.639+0000 7f9e61400700 1 -- 172.21.15.28:0/1012837118 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9e5c101220 msgr2=0x7f9e5c1016a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:25.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:24.639+0000 7f9e61400700 1 --2- 172.21.15.28:0/1012837118 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9e5c101220 0x7f9e5c1016a0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f9e480098f0 tx=0x7f9e48009c00 comp rx=0 tx=0).stop 2024-05-12T22:07:25.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:24.639+0000 7f9e61400700 1 -- 172.21.15.28:0/1012837118 shutdown_connections 2024-05-12T22:07:25.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:24.639+0000 7f9e61400700 1 --2- 172.21.15.28:0/1012837118 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9e5c101be0 0x7f9e5c10e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:25.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:24.639+0000 7f9e61400700 1 --2- 172.21.15.28:0/1012837118 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e5c107290 0x7f9e5c107670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:25.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:24.639+0000 7f9e61400700 1 --2- 172.21.15.28:0/1012837118 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9e5c101220 0x7f9e5c1016a0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:25.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:24.639+0000 7f9e61400700 1 -- 172.21.15.28:0/1012837118 >> 172.21.15.28:0/1012837118 conn(0x7f9e5c0781d0 msgr2=0x7f9e5c0785e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:07:25.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:24.639+0000 7f9e61400700 1 -- 172.21.15.28:0/1012837118 shutdown_connections 2024-05-12T22:07:25.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:24.639+0000 7f9e61400700 1 -- 172.21.15.28:0/1012837118 wait complete. 2024-05-12T22:07:25.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:24.641+0000 7f9e61400700 1 Processor -- start 2024-05-12T22:07:25.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:24.641+0000 7f9e61400700 1 -- start start 2024-05-12T22:07:25.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:24.641+0000 7f9e61400700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9e5c101220 0x7f9e5c19ad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:07:25.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:24.641+0000 7f9e61400700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9e5c101be0 0x7f9e5c19b270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:07:25.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:24.641+0000 7f9e61400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e5c107290 0x7f9e5c19f600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:07:25.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:24.641+0000 7f9e61400700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9e5c111550 con 0x7f9e5c107290 2024-05-12T22:07:25.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:24.641+0000 7f9e61400700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f9e5c1113d0 con 0x7f9e5c101220 2024-05-12T22:07:25.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:24.641+0000 7f9e61400700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f9e5c1116d0 con 0x7f9e5c101be0 2024-05-12T22:07:25.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:24.642+0000 7f9e59e00700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9e5c101be0 0x7f9e5c19b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:07:25.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:24.642+0000 7f9e5a800700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9e5c101220 0x7f9e5c19ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:07:25.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:24.642+0000 7f9e59e00700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9e5c101be0 0x7f9e5c19b270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:33690/0 (socket says 172.21.15.28:33690) 2024-05-12T22:07:25.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:24.642+0000 7f9e5a800700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9e5c101220 0x7f9e5c19ad30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:35358/0 (socket says 172.21.15.28:35358) 2024-05-12T22:07:25.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:24.642+0000 7f9e59e00700 1 -- 172.21.15.28:0/3410722944 learned_addr learned my addr 172.21.15.28:0/3410722944 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:07:25.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:24.642+0000 7f9e5b200700 1 --2- 172.21.15.28:0/3410722944 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e5c107290 0x7f9e5c19f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:07:25.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:24.642+0000 7f9e59e00700 1 -- 172.21.15.28:0/3410722944 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9e5c101220 msgr2=0x7f9e5c19ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:25.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:24.642+0000 7f9e59e00700 1 --2- 172.21.15.28:0/3410722944 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9e5c101220 0x7f9e5c19ad30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:25.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:24.642+0000 7f9e59e00700 1 -- 172.21.15.28:0/3410722944 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e5c107290 msgr2=0x7f9e5c19f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:25.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:24.642+0000 7f9e59e00700 1 --2- 172.21.15.28:0/3410722944 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e5c107290 0x7f9e5c19f600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:25.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:24.642+0000 7f9e59e00700 1 -- 172.21.15.28:0/3410722944 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e5c19fd40 con 0x7f9e5c101be0 2024-05-12T22:07:25.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:24.642+0000 7f9e5a800700 1 --2- 172.21.15.28:0/3410722944 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9e5c101220 0x7f9e5c19ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:07:25.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:24.643+0000 7f9e59e00700 1 --2- 172.21.15.28:0/3410722944 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9e5c101be0 0x7f9e5c19b270 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f9e48005950 tx=0x7f9e4800fb20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:07:25.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:24.643+0000 7f9e53400700 1 -- 172.21.15.28:0/3410722944 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e480051f0 con 0x7f9e5c101be0 2024-05-12T22:07:25.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:24.643+0000 7f9e53400700 1 -- 172.21.15.28:0/3410722944 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9e4800b560 con 0x7f9e5c101be0 2024-05-12T22:07:25.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:24.643+0000 7f9e53400700 1 -- 172.21.15.28:0/3410722944 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e48003a40 con 0x7f9e5c101be0 2024-05-12T22:07:25.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:24.643+0000 7f9e61400700 1 -- 172.21.15.28:0/3410722944 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e5c19ffd0 con 0x7f9e5c101be0 2024-05-12T22:07:25.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:24.643+0000 7f9e61400700 1 -- 172.21.15.28:0/3410722944 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e5c1a78b0 con 0x7f9e5c101be0 2024-05-12T22:07:25.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:24.644+0000 7f9e61400700 1 -- 172.21.15.28:0/3410722944 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f9e5c06aa60 con 0x7f9e5c101be0 2024-05-12T22:07:25.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:24.647+0000 7f9e53400700 1 -- 172.21.15.28:0/3410722944 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f9e4800ba80 con 0x7f9e5c101be0 2024-05-12T22:07:25.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:24.648+0000 7f9e53400700 1 --2- 172.21.15.28:0/3410722944 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f9e400a2d30 0x7f9e400a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:07:25.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:24.648+0000 7f9e53400700 1 -- 172.21.15.28:0/3410722944 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f9e480d0520 con 0x7f9e5c101be0 2024-05-12T22:07:25.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:24.648+0000 7f9e5a800700 1 --2- 172.21.15.28:0/3410722944 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f9e400a2d30 0x7f9e400a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:07:25.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:24.649+0000 7f9e53400700 1 --2- 172.21.15.28:0/3410722944 >> [v2:172.21.15.186:6824/3801668110,v1:172.21.15.186:6825/3801668110] conn(0x7f9e400ac740 0x7f9e400aebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:07:25.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:24.649+0000 7f9e53400700 1 -- 172.21.15.28:0/3410722944 --> [v2:172.21.15.186:6824/3801668110,v1:172.21.15.186:6825/3801668110] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9e48004e50 con 0x7f9e400ac740 2024-05-12T22:07:25.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:24.649+0000 7f9e53400700 1 -- 172.21.15.28:0/3410722944 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_get_version_reply(handle=1 version=88) ==== 24+0+0 (secure 0 0 0) 0x7f9e480d08d0 con 0x7f9e5c101be0 2024-05-12T22:07:25.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:24.649+0000 7f9e5b200700 1 --2- 172.21.15.28:0/3410722944 >> [v2:172.21.15.186:6824/3801668110,v1:172.21.15.186:6825/3801668110] conn(0x7f9e400ac740 0x7f9e400aebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:07:25.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:24.649+0000 7f9e5a800700 1 --2- 172.21.15.28:0/3410722944 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f9e400a2d30 0x7f9e400a51f0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f9e4c005fd0 tx=0x7f9e4c009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:07:25.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:24.650+0000 7f9e5b200700 1 --2- 172.21.15.28:0/3410722944 >> [v2:172.21.15.186:6824/3801668110,v1:172.21.15.186:6825/3801668110] conn(0x7f9e400ac740 0x7f9e400aebe0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:07:25.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:24.651+0000 7f9e53400700 1 -- 172.21.15.28:0/3410722944 <== osd.11 v2:172.21.15.186:6824/3801668110 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9e48004e50 con 0x7f9e400ac740 2024-05-12T22:07:25.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:24.687+0000 7f9e61400700 1 -- 172.21.15.28:0/3410722944 --> [v2:172.21.15.186:6824/3801668110,v1:172.21.15.186:6825/3801668110] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f9e5c102940 con 0x7f9e400ac740 2024-05-12T22:07:25.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:24.688+0000 7f9e53400700 1 -- 172.21.15.28:0/3410722944 <== osd.11 v2:172.21.15.186:6824/3801668110 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f9e5c102940 con 0x7f9e400ac740 2024-05-12T22:07:25.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:24.688+0000 7f9e61400700 1 -- 172.21.15.28:0/3410722944 >> [v2:172.21.15.186:6824/3801668110,v1:172.21.15.186:6825/3801668110] conn(0x7f9e400ac740 msgr2=0x7f9e400aebe0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:25.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:24.688+0000 7f9e61400700 1 --2- 172.21.15.28:0/3410722944 >> [v2:172.21.15.186:6824/3801668110,v1:172.21.15.186:6825/3801668110] conn(0x7f9e400ac740 0x7f9e400aebe0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:25.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:24.689+0000 7f9e61400700 1 -- 172.21.15.28:0/3410722944 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f9e400a2d30 msgr2=0x7f9e400a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:25.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:24.689+0000 7f9e61400700 1 --2- 172.21.15.28:0/3410722944 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f9e400a2d30 0x7f9e400a51f0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f9e4c005fd0 tx=0x7f9e4c009310 comp rx=0 tx=0).stop 2024-05-12T22:07:25.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:24.689+0000 7f9e61400700 1 -- 172.21.15.28:0/3410722944 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9e5c101be0 msgr2=0x7f9e5c19b270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:25.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:24.689+0000 7f9e61400700 1 --2- 172.21.15.28:0/3410722944 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9e5c101be0 0x7f9e5c19b270 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f9e48005950 tx=0x7f9e4800fb20 comp rx=0 tx=0).stop 2024-05-12T22:07:25.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:24.689+0000 7f9e5b200700 1 -- 172.21.15.28:0/3410722944 reap_dead start 2024-05-12T22:07:25.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:24.689+0000 7f9e61400700 1 -- 172.21.15.28:0/3410722944 shutdown_connections 2024-05-12T22:07:25.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:24.689+0000 7f9e61400700 1 -- 172.21.15.28:0/3410722944 >> 172.21.15.28:0/3410722944 conn(0x7f9e5c0781d0 msgr2=0x7f9e5c06a000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:07:25.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:24.689+0000 7f9e61400700 1 -- 172.21.15.28:0/3410722944 shutdown_connections 2024-05-12T22:07:25.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:24.689+0000 7f9e61400700 1 -- 172.21.15.28:0/3410722944 wait complete. 2024-05-12T22:07:25.683 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:07:25 smithi186 ceph-mon[34785]: pgmap v337: 97 pgs: 97 active+clean; 640 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-12T22:07:25.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:22.749+0000 7f6f89c00700 1 -- 172.21.15.28:0/3521647331 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f8410ea90 msgr2=0x7f6f84110f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:25.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:22.749+0000 7f6f89c00700 1 --2- 172.21.15.28:0/3521647331 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f8410ea90 0x7f6f84110f00 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f6f7c009f90 tx=0x7f6f7c00b330 comp rx=0 tx=0).stop 2024-05-12T22:07:25.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:22.749+0000 7f6f89c00700 1 -- 172.21.15.28:0/3521647331 shutdown_connections 2024-05-12T22:07:25.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:22.749+0000 7f6f89c00700 1 --2- 172.21.15.28:0/3521647331 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f8410ea90 0x7f6f84110f00 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:25.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:22.749+0000 7f6f89c00700 1 --2- 172.21.15.28:0/3521647331 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6f84074750 0x7f6f84074b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:25.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:22.749+0000 7f6f89c00700 1 --2- 172.21.15.28:0/3521647331 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6f84075070 0x7f6f8410e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:25.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:22.749+0000 7f6f89c00700 1 -- 172.21.15.28:0/3521647331 >> 172.21.15.28:0/3521647331 conn(0x7f6f840fdda0 msgr2=0x7f6f841001c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:07:25.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:22.749+0000 7f6f89c00700 1 -- 172.21.15.28:0/3521647331 shutdown_connections 2024-05-12T22:07:25.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:22.749+0000 7f6f89c00700 1 -- 172.21.15.28:0/3521647331 wait complete. 2024-05-12T22:07:25.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:22.749+0000 7f6f89c00700 1 Processor -- start 2024-05-12T22:07:25.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:22.749+0000 7f6f89c00700 1 -- start start 2024-05-12T22:07:25.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:22.750+0000 7f6f89c00700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6f84074750 0x7f6f8419ac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:07:25.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:22.750+0000 7f6f89c00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f84075070 0x7f6f8419b190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:07:25.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:22.750+0000 7f6f89c00700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6f8410ea90 0x7f6f8419f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:07:25.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:22.750+0000 7f6f89c00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6f84114760 con 0x7f6f84075070 2024-05-12T22:07:25.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:22.750+0000 7f6f89c00700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f6f841145e0 con 0x7f6f84074750 2024-05-12T22:07:25.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:22.750+0000 7f6f89c00700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f6f841148e0 con 0x7f6f8410ea90 2024-05-12T22:07:25.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:22.750+0000 7f6f83400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f84075070 0x7f6f8419b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:07:25.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:22.750+0000 7f6f89200700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6f8410ea90 0x7f6f8419f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:07:25.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:22.750+0000 7f6f83e00700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6f84074750 0x7f6f8419ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:07:25.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:22.750+0000 7f6f89200700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6f8410ea90 0x7f6f8419f550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:33670/0 (socket says 172.21.15.28:33670) 2024-05-12T22:07:25.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:22.750+0000 7f6f89200700 1 -- 172.21.15.28:0/1950552035 learned_addr learned my addr 172.21.15.28:0/1950552035 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:07:25.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:22.750+0000 7f6f83400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f84075070 0x7f6f8419b190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55192/0 (socket says 172.21.15.28:55192) 2024-05-12T22:07:25.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:22.750+0000 7f6f89200700 1 -- 172.21.15.28:0/1950552035 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6f84074750 msgr2=0x7f6f8419ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:25.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:22.750+0000 7f6f89200700 1 --2- 172.21.15.28:0/1950552035 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6f84074750 0x7f6f8419ac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:25.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:22.750+0000 7f6f89200700 1 -- 172.21.15.28:0/1950552035 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f84075070 msgr2=0x7f6f8419b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:25.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:22.750+0000 7f6f89200700 1 --2- 172.21.15.28:0/1950552035 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f84075070 0x7f6f8419b190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:25.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:22.751+0000 7f6f89200700 1 -- 172.21.15.28:0/1950552035 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f8419fc50 con 0x7f6f8410ea90 2024-05-12T22:07:25.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:22.751+0000 7f6f83e00700 1 --2- 172.21.15.28:0/1950552035 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6f84074750 0x7f6f8419ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:07:25.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:22.751+0000 7f6f89200700 1 --2- 172.21.15.28:0/1950552035 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6f8410ea90 0x7f6f8419f550 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f6f7c004f30 tx=0x7f6f7c004f60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:07:25.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:22.751+0000 7f6f80c00700 1 -- 172.21.15.28:0/1950552035 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f7c021070 con 0x7f6f8410ea90 2024-05-12T22:07:25.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:22.751+0000 7f6f80c00700 1 -- 172.21.15.28:0/1950552035 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6f7c007ac0 con 0x7f6f8410ea90 2024-05-12T22:07:25.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:22.751+0000 7f6f80c00700 1 -- 172.21.15.28:0/1950552035 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f7c0052a0 con 0x7f6f8410ea90 2024-05-12T22:07:25.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:22.751+0000 7f6f89c00700 1 -- 172.21.15.28:0/1950552035 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f8419fee0 con 0x7f6f8410ea90 2024-05-12T22:07:25.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:22.751+0000 7f6f89c00700 1 -- 172.21.15.28:0/1950552035 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f841a7800 con 0x7f6f8410ea90 2024-05-12T22:07:25.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:22.752+0000 7f6f89c00700 1 -- 172.21.15.28:0/1950552035 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f6f84062a30 con 0x7f6f8410ea90 2024-05-12T22:07:25.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:22.755+0000 7f6f80c00700 1 -- 172.21.15.28:0/1950552035 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f6f7c024070 con 0x7f6f8410ea90 2024-05-12T22:07:25.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:22.757+0000 7f6f80c00700 1 --2- 172.21.15.28:0/1950552035 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f6f680a2cb0 0x7f6f680a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:07:25.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:22.757+0000 7f6f80c00700 1 -- 172.21.15.28:0/1950552035 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f6f7c0d04a0 con 0x7f6f8410ea90 2024-05-12T22:07:25.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:22.757+0000 7f6f83e00700 1 --2- 172.21.15.28:0/1950552035 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f6f680a2cb0 0x7f6f680a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:07:25.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:22.757+0000 7f6f80c00700 1 --2- 172.21.15.28:0/1950552035 >> [v2:172.21.15.86:6816/726062711,v1:172.21.15.86:6817/726062711] conn(0x7f6f680ac7f0 0x7f6f680aec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:07:25.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:22.757+0000 7f6f80c00700 1 -- 172.21.15.28:0/1950552035 --> [v2:172.21.15.86:6816/726062711,v1:172.21.15.86:6817/726062711] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6f7c01da20 con 0x7f6f680ac7f0 2024-05-12T22:07:25.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:22.757+0000 7f6f80c00700 1 -- 172.21.15.28:0/1950552035 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_get_version_reply(handle=1 version=88) ==== 24+0+0 (secure 0 0 0) 0x7f6f7c0d0850 con 0x7f6f8410ea90 2024-05-12T22:07:25.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:22.757+0000 7f6f83e00700 1 --2- 172.21.15.28:0/1950552035 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f6f680a2cb0 0x7f6f680a5170 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f6f74011490 tx=0x7f6f74009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:07:25.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:22.757+0000 7f6f83400700 1 --2- 172.21.15.28:0/1950552035 >> [v2:172.21.15.86:6816/726062711,v1:172.21.15.86:6817/726062711] conn(0x7f6f680ac7f0 0x7f6f680aec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:07:25.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:22.757+0000 7f6f83400700 1 --2- 172.21.15.28:0/1950552035 >> [v2:172.21.15.86:6816/726062711,v1:172.21.15.86:6817/726062711] conn(0x7f6f680ac7f0 0x7f6f680aec90 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:07:25.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:22.758+0000 7f6f80c00700 1 -- 172.21.15.28:0/1950552035 <== osd.7 v2:172.21.15.86:6816/726062711 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6f7c01da20 con 0x7f6f680ac7f0 2024-05-12T22:07:25.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:22.777+0000 7f6f89c00700 1 -- 172.21.15.28:0/1950552035 --> [v2:172.21.15.86:6816/726062711,v1:172.21.15.86:6817/726062711] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f6f84062e30 con 0x7f6f680ac7f0 2024-05-12T22:07:25.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:22.777+0000 7f6f80c00700 1 -- 172.21.15.28:0/1950552035 <== osd.7 v2:172.21.15.86:6816/726062711 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f6f84062e30 con 0x7f6f680ac7f0 2024-05-12T22:07:25.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:22.779+0000 7f6f89c00700 1 -- 172.21.15.28:0/1950552035 >> [v2:172.21.15.86:6816/726062711,v1:172.21.15.86:6817/726062711] conn(0x7f6f680ac7f0 msgr2=0x7f6f680aec90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:25.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:22.779+0000 7f6f89c00700 1 --2- 172.21.15.28:0/1950552035 >> [v2:172.21.15.86:6816/726062711,v1:172.21.15.86:6817/726062711] conn(0x7f6f680ac7f0 0x7f6f680aec90 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:25.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:22.779+0000 7f6f89c00700 1 -- 172.21.15.28:0/1950552035 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f6f680a2cb0 msgr2=0x7f6f680a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:25.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:22.779+0000 7f6f89c00700 1 --2- 172.21.15.28:0/1950552035 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f6f680a2cb0 0x7f6f680a5170 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f6f74011490 tx=0x7f6f74009260 comp rx=0 tx=0).stop 2024-05-12T22:07:25.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:07:25 smithi028 ceph-mon[29847]: pgmap v337: 97 pgs: 97 active+clean; 640 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-12T22:07:25.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:22.779+0000 7f6f89c00700 1 -- 172.21.15.28:0/1950552035 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6f8410ea90 msgr2=0x7f6f8419f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:25.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:22.779+0000 7f6f89c00700 1 --2- 172.21.15.28:0/1950552035 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6f8410ea90 0x7f6f8419f550 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f6f7c004f30 tx=0x7f6f7c004f60 comp rx=0 tx=0).stop 2024-05-12T22:07:25.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:22.779+0000 7f6f89200700 1 -- 172.21.15.28:0/1950552035 reap_dead start 2024-05-12T22:07:25.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:22.779+0000 7f6f89c00700 1 -- 172.21.15.28:0/1950552035 shutdown_connections 2024-05-12T22:07:25.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:22.779+0000 7f6f89c00700 1 -- 172.21.15.28:0/1950552035 >> 172.21.15.28:0/1950552035 conn(0x7f6f840fdda0 msgr2=0x7f6f840fea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:07:25.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:22.779+0000 7f6f89c00700 1 -- 172.21.15.28:0/1950552035 shutdown_connections 2024-05-12T22:07:25.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:22.779+0000 7f6f89c00700 1 -- 172.21.15.28:0/1950552035 wait complete. 2024-05-12T22:07:25.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:07:25 smithi086 ceph-mon[34297]: pgmap v337: 97 pgs: 97 active+clean; 640 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-12T22:07:26.664 INFO:teuthology.orchestra.run.smithi028.stdout:167503724600 2024-05-12T22:07:26.664 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f72fecff68e1d400c4568684327c900485c20d6a shell --fsid 6b92e7b4-10aa-11ef-bc98-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-05-12T22:07:27.169 INFO:teuthology.orchestra.run.smithi028.stdout:223338299435 2024-05-12T22:07:27.169 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f72fecff68e1d400c4568684327c900485c20d6a shell --fsid 6b92e7b4-10aa-11ef-bc98-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-05-12T22:07:27.331 INFO:teuthology.orchestra.run.smithi028.stdout:347892351000 2024-05-12T22:07:27.331 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f72fecff68e1d400c4568684327c900485c20d6a shell --fsid 6b92e7b4-10aa-11ef-bc98-c7b262605968 -- ceph osd last-stat-seq osd.11 2024-05-12T22:07:27.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:07:27 smithi186 ceph-mon[34785]: pgmap v338: 97 pgs: 97 active+clean; 640 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-12T22:07:27.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:07:27 smithi086 ceph-mon[34297]: pgmap v338: 97 pgs: 97 active+clean; 640 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-12T22:07:27.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:07:27 smithi028 ceph-mon[29847]: pgmap v338: 97 pgs: 97 active+clean; 640 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-12T22:07:27.845 INFO:teuthology.orchestra.run.smithi028.stdout:201863462962 2024-05-12T22:07:27.846 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f72fecff68e1d400c4568684327c900485c20d6a shell --fsid 6b92e7b4-10aa-11ef-bc98-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-05-12T22:07:27.894 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/mon.a/config 2024-05-12T22:07:28.004 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/mon.a/config 2024-05-12T22:07:28.142 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/mon.a/config 2024-05-12T22:07:28.142 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/mon.a/config 2024-05-12T22:07:29.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:29.018+0000 7f57f9200700 1 -- 172.21.15.28:0/383008942 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57f41013a0 msgr2=0x7f57f410d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:29.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:29.018+0000 7f57f9200700 1 --2- 172.21.15.28:0/383008942 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57f41013a0 0x7f57f410d860 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f57ec00b0a0 tx=0x7f57ec00b3b0 comp rx=0 tx=0).stop 2024-05-12T22:07:29.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:29.018+0000 7f57f9200700 1 -- 172.21.15.28:0/383008942 shutdown_connections 2024-05-12T22:07:29.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:29.018+0000 7f57f9200700 1 --2- 172.21.15.28:0/383008942 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f57f4106990 0x7f57f4106d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:29.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:29.018+0000 7f57f9200700 1 --2- 172.21.15.28:0/383008942 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57f41013a0 0x7f57f410d860 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:29.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:29.018+0000 7f57f9200700 1 --2- 172.21.15.28:0/383008942 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f57f41009e0 0x7f57f4100e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:29.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:29.018+0000 7f57f9200700 1 -- 172.21.15.28:0/383008942 >> 172.21.15.28:0/383008942 conn(0x7f57f4077890 msgr2=0x7f57f40fec40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:07:29.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:29.018+0000 7f57f9200700 1 -- 172.21.15.28:0/383008942 shutdown_connections 2024-05-12T22:07:29.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:29.018+0000 7f57f9200700 1 -- 172.21.15.28:0/383008942 wait complete. 2024-05-12T22:07:29.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:29.020+0000 7f57f9200700 1 Processor -- start 2024-05-12T22:07:29.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:29.020+0000 7f57f9200700 1 -- start start 2024-05-12T22:07:29.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:29.020+0000 7f57f9200700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57f41009e0 0x7f57f40fcd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:07:29.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:29.020+0000 7f57f9200700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f57f41013a0 0x7f57f40fd240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:07:29.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:29.020+0000 7f57f9200700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f57f4106990 0x7f57f40fd780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:07:29.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:29.020+0000 7f57f9200700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f57f4111540 con 0x7f57f41009e0 2024-05-12T22:07:29.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:29.020+0000 7f57f9200700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f57f41113c0 con 0x7f57f41013a0 2024-05-12T22:07:29.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:29.020+0000 7f57f9200700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f57f41116c0 con 0x7f57f4106990 2024-05-12T22:07:29.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:29.021+0000 7f57f1e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57f41009e0 0x7f57f40fcd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:07:29.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:29.021+0000 7f57f1400700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f57f41013a0 0x7f57f40fd240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:07:29.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:29.021+0000 7f57f1e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57f41009e0 0x7f57f40fcd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43598/0 (socket says 172.21.15.28:43598) 2024-05-12T22:07:29.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:29.021+0000 7f57f1400700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f57f41013a0 0x7f57f40fd240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:50116/0 (socket says 172.21.15.28:50116) 2024-05-12T22:07:29.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:29.021+0000 7f57f1e00700 1 -- 172.21.15.28:0/1100108950 learned_addr learned my addr 172.21.15.28:0/1100108950 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:07:29.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:29.021+0000 7f57f2800700 1 --2- 172.21.15.28:0/1100108950 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f57f4106990 0x7f57f40fd780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:07:29.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:29.021+0000 7f57f1400700 1 -- 172.21.15.28:0/1100108950 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f57f4106990 msgr2=0x7f57f40fd780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:29.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:29.021+0000 7f57f1400700 1 --2- 172.21.15.28:0/1100108950 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f57f4106990 0x7f57f40fd780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:29.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:29.021+0000 7f57f1400700 1 -- 172.21.15.28:0/1100108950 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57f41009e0 msgr2=0x7f57f40fcd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:29.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:29.021+0000 7f57f1400700 1 --2- 172.21.15.28:0/1100108950 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57f41009e0 0x7f57f40fcd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:29.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:29.021+0000 7f57f1400700 1 -- 172.21.15.28:0/1100108950 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57f41a6ff0 con 0x7f57f41013a0 2024-05-12T22:07:29.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:29.021+0000 7f57f2800700 1 --2- 172.21.15.28:0/1100108950 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f57f4106990 0x7f57f40fd780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:07:29.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:29.021+0000 7f57f1e00700 1 --2- 172.21.15.28:0/1100108950 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57f41009e0 0x7f57f40fcd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:07:29.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:29.021+0000 7f57f1400700 1 --2- 172.21.15.28:0/1100108950 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f57f41013a0 0x7f57f40fd240 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f57dc00e9f0 tx=0x7f57dc00ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:07:29.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:29.022+0000 7f57eaa00700 1 -- 172.21.15.28:0/1100108950 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57dc00cbd0 con 0x7f57f41013a0 2024-05-12T22:07:29.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:29.022+0000 7f57eaa00700 1 -- 172.21.15.28:0/1100108950 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f57dc00cd70 con 0x7f57f41013a0 2024-05-12T22:07:29.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:29.022+0000 7f57f9200700 1 -- 172.21.15.28:0/1100108950 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57f41a72e0 con 0x7f57f41013a0 2024-05-12T22:07:29.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:29.022+0000 7f57eaa00700 1 -- 172.21.15.28:0/1100108950 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57dc018710 con 0x7f57f41013a0 2024-05-12T22:07:29.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:29.022+0000 7f57f9200700 1 -- 172.21.15.28:0/1100108950 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57f41a7820 con 0x7f57f41013a0 2024-05-12T22:07:29.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:29.023+0000 7f57f9200700 1 -- 172.21.15.28:0/1100108950 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f57f4062a30 con 0x7f57f41013a0 2024-05-12T22:07:29.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:29.026+0000 7f57eaa00700 1 -- 172.21.15.28:0/1100108950 <== mon.2 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f57dc010430 con 0x7f57f41013a0 2024-05-12T22:07:29.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:29.027+0000 7f57eaa00700 1 --2- 172.21.15.28:0/1100108950 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f57d80a2cb0 0x7f57d80a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:07:29.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:29.027+0000 7f57eaa00700 1 -- 172.21.15.28:0/1100108950 <== mon.2 v2:172.21.15.86:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f57dc0cfbd0 con 0x7f57f41013a0 2024-05-12T22:07:29.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:29.028+0000 7f57f1e00700 1 --2- 172.21.15.28:0/1100108950 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f57d80a2cb0 0x7f57d80a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:07:29.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:29.028+0000 7f57eaa00700 1 --2- 172.21.15.28:0/1100108950 >> [v2:172.21.15.86:6808/2596998708,v1:172.21.15.86:6809/2596998708] conn(0x7f57d80ac7f0 0x7f57d80aec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:07:29.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:29.028+0000 7f57eaa00700 1 -- 172.21.15.28:0/1100108950 --> [v2:172.21.15.86:6808/2596998708,v1:172.21.15.86:6809/2596998708] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f57d80af360 con 0x7f57d80ac7f0 2024-05-12T22:07:29.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:29.028+0000 7f57eaa00700 1 -- 172.21.15.28:0/1100108950 <== mon.2 v2:172.21.15.86:3300/0 6 ==== mon_get_version_reply(handle=1 version=88) ==== 24+0+0 (secure 0 0 0) 0x7f57dc0cff80 con 0x7f57f41013a0 2024-05-12T22:07:29.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:29.028+0000 7f57f2800700 1 --2- 172.21.15.28:0/1100108950 >> [v2:172.21.15.86:6808/2596998708,v1:172.21.15.86:6809/2596998708] conn(0x7f57d80ac7f0 0x7f57d80aec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:07:29.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:29.028+0000 7f57f1e00700 1 --2- 172.21.15.28:0/1100108950 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f57d80a2cb0 0x7f57d80a5170 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f57e4011550 tx=0x7f57e4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:07:29.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:29.029+0000 7f57f2800700 1 --2- 172.21.15.28:0/1100108950 >> [v2:172.21.15.86:6808/2596998708,v1:172.21.15.86:6809/2596998708] conn(0x7f57d80ac7f0 0x7f57d80aec90 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:07:29.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:29.030+0000 7f57eaa00700 1 -- 172.21.15.28:0/1100108950 <== osd.4 v2:172.21.15.86:6808/2596998708 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f57d80af360 con 0x7f57d80ac7f0 2024-05-12T22:07:29.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:29.048+0000 7f57f9200700 1 -- 172.21.15.28:0/1100108950 --> [v2:172.21.15.86:6808/2596998708,v1:172.21.15.86:6809/2596998708] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f57f4062eb0 con 0x7f57d80ac7f0 2024-05-12T22:07:29.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:29.049+0000 7f57eaa00700 1 -- 172.21.15.28:0/1100108950 <== osd.4 v2:172.21.15.86:6808/2596998708 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f57f4062eb0 con 0x7f57d80ac7f0 2024-05-12T22:07:29.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:29.049+0000 7f57f9200700 1 -- 172.21.15.28:0/1100108950 >> [v2:172.21.15.86:6808/2596998708,v1:172.21.15.86:6809/2596998708] conn(0x7f57d80ac7f0 msgr2=0x7f57d80aec90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:29.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:29.049+0000 7f57f9200700 1 --2- 172.21.15.28:0/1100108950 >> [v2:172.21.15.86:6808/2596998708,v1:172.21.15.86:6809/2596998708] conn(0x7f57d80ac7f0 0x7f57d80aec90 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:29.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:29.049+0000 7f57f9200700 1 -- 172.21.15.28:0/1100108950 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f57d80a2cb0 msgr2=0x7f57d80a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:29.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:29.049+0000 7f57f9200700 1 --2- 172.21.15.28:0/1100108950 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f57d80a2cb0 0x7f57d80a5170 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f57e4011550 tx=0x7f57e4009260 comp rx=0 tx=0).stop 2024-05-12T22:07:29.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:29.049+0000 7f57f9200700 1 -- 172.21.15.28:0/1100108950 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f57f41013a0 msgr2=0x7f57f40fd240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:29.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:29.049+0000 7f57f9200700 1 --2- 172.21.15.28:0/1100108950 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f57f41013a0 0x7f57f40fd240 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f57dc00e9f0 tx=0x7f57dc00ed00 comp rx=0 tx=0).stop 2024-05-12T22:07:29.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:29.049+0000 7f57f2800700 1 -- 172.21.15.28:0/1100108950 reap_dead start 2024-05-12T22:07:29.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:29.050+0000 7f57f9200700 1 -- 172.21.15.28:0/1100108950 shutdown_connections 2024-05-12T22:07:29.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:29.050+0000 7f57f9200700 1 -- 172.21.15.28:0/1100108950 >> 172.21.15.28:0/1100108950 conn(0x7f57f4077890 msgr2=0x7f57f410e500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:07:29.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:29.050+0000 7f57f9200700 1 -- 172.21.15.28:0/1100108950 shutdown_connections 2024-05-12T22:07:29.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:29.050+0000 7f57f9200700 1 -- 172.21.15.28:0/1100108950 wait complete. 2024-05-12T22:07:29.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:07:29 smithi186 ceph-mon[34785]: pgmap v339: 97 pgs: 97 active+clean; 640 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-12T22:07:29.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:07:29 smithi086 ceph-mon[34297]: pgmap v339: 97 pgs: 97 active+clean; 640 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-12T22:07:29.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:07:29 smithi028 ceph-mon[29847]: pgmap v339: 97 pgs: 97 active+clean; 640 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-12T22:07:30.982 INFO:teuthology.orchestra.run.smithi028.stdout:146028888125 2024-05-12T22:07:30.982 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f72fecff68e1d400c4568684327c900485c20d6a shell --fsid 6b92e7b4-10aa-11ef-bc98-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-05-12T22:07:31.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:07:31 smithi028 ceph-mon[29847]: pgmap v340: 97 pgs: 97 active+clean; 640 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-12T22:07:31.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:07:31 smithi186 ceph-mon[34785]: pgmap v340: 97 pgs: 97 active+clean; 640 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-12T22:07:31.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:07:31 smithi086 ceph-mon[34297]: pgmap v340: 97 pgs: 97 active+clean; 640 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-12T22:07:32.462 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/mon.a/config 2024-05-12T22:07:33.661 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:07:33 smithi028 ceph-mon[29847]: pgmap v341: 97 pgs: 97 active+clean; 640 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:07:33.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:07:33 smithi186 ceph-mon[34785]: pgmap v341: 97 pgs: 97 active+clean; 640 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:07:33.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:07:33 smithi086 ceph-mon[34297]: pgmap v341: 97 pgs: 97 active+clean; 640 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:07:34.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:34.162+0000 7fe85ae00700 1 -- 172.21.15.28:0/3212810232 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe854102e60 msgr2=0x7fe854106eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:34.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:34.162+0000 7fe85ae00700 1 --2- 172.21.15.28:0/3212810232 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe854102e60 0x7fe854106eb0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fe8400099c0 tx=0x7fe840009cd0 comp rx=0 tx=0).stop 2024-05-12T22:07:34.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:34.162+0000 7fe85ae00700 1 -- 172.21.15.28:0/3212810232 shutdown_connections 2024-05-12T22:07:34.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:34.162+0000 7fe85ae00700 1 --2- 172.21.15.28:0/3212810232 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe854107590 0x7fe85410f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:34.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:34.162+0000 7fe85ae00700 1 --2- 172.21.15.28:0/3212810232 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe8541024b0 0x7fe854102890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:34.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:34.162+0000 7fe85ae00700 1 --2- 172.21.15.28:0/3212810232 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe854102e60 0x7fe854106eb0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:34.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:34.162+0000 7fe85ae00700 1 -- 172.21.15.28:0/3212810232 >> 172.21.15.28:0/3212810232 conn(0x7fe8540fdda0 msgr2=0x7fe8541001c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:07:34.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:34.163+0000 7fe85ae00700 1 -- 172.21.15.28:0/3212810232 shutdown_connections 2024-05-12T22:07:34.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:34.163+0000 7fe85ae00700 1 -- 172.21.15.28:0/3212810232 wait complete. 2024-05-12T22:07:34.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:34.164+0000 7fe85ae00700 1 Processor -- start 2024-05-12T22:07:34.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:34.164+0000 7fe85ae00700 1 -- start start 2024-05-12T22:07:34.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:34.165+0000 7fe85ae00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8541024b0 0x7fe85410c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:07:34.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:34.165+0000 7fe85ae00700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe854102e60 0x7fe85410c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:07:34.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:34.165+0000 7fe85ae00700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe854107590 0x7fe8541a39c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:07:34.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:34.165+0000 7fe85ae00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe8541125c0 con 0x7fe8541024b0 2024-05-12T22:07:34.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:34.165+0000 7fe85ae00700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fe854112440 con 0x7fe854102e60 2024-05-12T22:07:34.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:34.165+0000 7fe85ae00700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe854112740 con 0x7fe854107590 2024-05-12T22:07:34.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:34.165+0000 7fe859a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8541024b0 0x7fe85410c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:07:34.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:34.165+0000 7fe859000700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe854102e60 0x7fe85410c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:07:34.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:34.165+0000 7fe859000700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe854102e60 0x7fe85410c8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:50134/0 (socket says 172.21.15.28:50134) 2024-05-12T22:07:34.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:34.165+0000 7fe859000700 1 -- 172.21.15.28:0/504752050 learned_addr learned my addr 172.21.15.28:0/504752050 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:07:34.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:34.165+0000 7fe85a400700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe854107590 0x7fe8541a39c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:07:34.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:34.166+0000 7fe859000700 1 -- 172.21.15.28:0/504752050 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe854107590 msgr2=0x7fe8541a39c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:34.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:34.166+0000 7fe859000700 1 --2- 172.21.15.28:0/504752050 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe854107590 0x7fe8541a39c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:34.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:34.166+0000 7fe859000700 1 -- 172.21.15.28:0/504752050 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8541024b0 msgr2=0x7fe85410c370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:34.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:34.166+0000 7fe859000700 1 --2- 172.21.15.28:0/504752050 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8541024b0 0x7fe85410c370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:34.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:34.166+0000 7fe859000700 1 -- 172.21.15.28:0/504752050 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8541a4100 con 0x7fe854102e60 2024-05-12T22:07:34.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:34.166+0000 7fe85a400700 1 --2- 172.21.15.28:0/504752050 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe854107590 0x7fe8541a39c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:07:34.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:34.166+0000 7fe859000700 1 --2- 172.21.15.28:0/504752050 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe854102e60 0x7fe85410c8b0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fe840005f50 tx=0x7fe840003a90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:07:34.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:34.166+0000 7fe852000700 1 -- 172.21.15.28:0/504752050 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe840003680 con 0x7fe854102e60 2024-05-12T22:07:34.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:34.166+0000 7fe852000700 1 -- 172.21.15.28:0/504752050 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe8400055a0 con 0x7fe854102e60 2024-05-12T22:07:34.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:34.167+0000 7fe85ae00700 1 -- 172.21.15.28:0/504752050 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8541a4390 con 0x7fe854102e60 2024-05-12T22:07:34.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:34.167+0000 7fe852000700 1 -- 172.21.15.28:0/504752050 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe84000fcc0 con 0x7fe854102e60 2024-05-12T22:07:34.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:34.167+0000 7fe85ae00700 1 -- 172.21.15.28:0/504752050 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8541abc70 con 0x7fe854102e60 2024-05-12T22:07:34.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:34.168+0000 7fe85ae00700 1 -- 172.21.15.28:0/504752050 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe854103990 con 0x7fe854102e60 2024-05-12T22:07:34.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:34.171+0000 7fe852000700 1 -- 172.21.15.28:0/504752050 <== mon.2 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fe84000f460 con 0x7fe854102e60 2024-05-12T22:07:34.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:34.173+0000 7fe852000700 1 --2- 172.21.15.28:0/504752050 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fe83c0a2e50 0x7fe83c0a5310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:07:34.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:34.173+0000 7fe852000700 1 -- 172.21.15.28:0/504752050 <== mon.2 v2:172.21.15.86:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7fe8400cfe90 con 0x7fe854102e60 2024-05-12T22:07:34.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:34.173+0000 7fe859a00700 1 --2- 172.21.15.28:0/504752050 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fe83c0a2e50 0x7fe83c0a5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:07:34.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:34.174+0000 7fe859a00700 1 --2- 172.21.15.28:0/504752050 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fe83c0a2e50 0x7fe83c0a5310 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fe848009da0 tx=0x7fe84800b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:07:34.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:34.177+0000 7fe852000700 1 -- 172.21.15.28:0/504752050 <== mon.2 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fe840013070 con 0x7fe854102e60 2024-05-12T22:07:34.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:34.245+0000 7f27f5400700 1 -- 172.21.15.28:0/129631285 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f27f01023a0 msgr2=0x7f27f0102800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:34.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:34.245+0000 7f27f5400700 1 --2- 172.21.15.28:0/129631285 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f27f01023a0 0x7f27f0102800 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f27dc009990 tx=0x7f27dc009ca0 comp rx=0 tx=0).stop 2024-05-12T22:07:34.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:34.245+0000 7f27f5400700 1 -- 172.21.15.28:0/129631285 shutdown_connections 2024-05-12T22:07:34.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:34.245+0000 7f27f5400700 1 --2- 172.21.15.28:0/129631285 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27f0102d40 0x7f27f010f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:34.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:34.245+0000 7f27f5400700 1 --2- 172.21.15.28:0/129631285 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27f01083e0 0x7f27f01087c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:34.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:34.245+0000 7f27f5400700 1 --2- 172.21.15.28:0/129631285 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f27f01023a0 0x7f27f0102800 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:34.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:34.245+0000 7f27f5400700 1 -- 172.21.15.28:0/129631285 >> 172.21.15.28:0/129631285 conn(0x7f27f00793f0 msgr2=0x7f27f0079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:07:34.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:34.245+0000 7f27f5400700 1 -- 172.21.15.28:0/129631285 shutdown_connections 2024-05-12T22:07:34.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:34.245+0000 7f27f5400700 1 -- 172.21.15.28:0/129631285 wait complete. 2024-05-12T22:07:34.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:34.246+0000 7f27f5400700 1 Processor -- start 2024-05-12T22:07:34.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:34.247+0000 7f27f5400700 1 -- start start 2024-05-12T22:07:34.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:34.247+0000 7f27f5400700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f27f01023a0 0x7f27f0074620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:07:34.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:34.247+0000 7f27f5400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27f0102d40 0x7f27f0074b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:07:34.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:34.247+0000 7f27f5400700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27f01083e0 0x7f27f00750a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:07:34.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:34.247+0000 7f27f5400700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f27f01126b0 con 0x7f27f0102d40 2024-05-12T22:07:34.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:34.247+0000 7f27f5400700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f27f0112530 con 0x7f27f01083e0 2024-05-12T22:07:34.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:34.247+0000 7f27f5400700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f27f0112830 con 0x7f27f01023a0 2024-05-12T22:07:34.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:34.247+0000 7f27ef200700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27f01083e0 0x7f27f00750a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:07:34.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:34.247+0000 7f27ee800700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f27f01023a0 0x7f27f0074620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:07:34.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:34.247+0000 7f27ede00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27f0102d40 0x7f27f0074b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:07:34.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:34.247+0000 7f27ef200700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27f01083e0 0x7f27f00750a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:50148/0 (socket says 172.21.15.28:50148) 2024-05-12T22:07:34.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:34.247+0000 7f27ef200700 1 -- 172.21.15.28:0/195323455 learned_addr learned my addr 172.21.15.28:0/195323455 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:07:34.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:34.248+0000 7f27ef200700 1 -- 172.21.15.28:0/195323455 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f27f01023a0 msgr2=0x7f27f0074620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:34.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:34.248+0000 7f27ef200700 1 --2- 172.21.15.28:0/195323455 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f27f01023a0 0x7f27f0074620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:34.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:34.248+0000 7f27ef200700 1 -- 172.21.15.28:0/195323455 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27f0102d40 msgr2=0x7f27f0074b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:34.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:34.248+0000 7f27ef200700 1 --2- 172.21.15.28:0/195323455 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27f0102d40 0x7f27f0074b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:34.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:34.248+0000 7f27ef200700 1 -- 172.21.15.28:0/195323455 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27f01a7030 con 0x7f27f01083e0 2024-05-12T22:07:34.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:34.248+0000 7f27ee800700 1 --2- 172.21.15.28:0/195323455 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f27f01023a0 0x7f27f0074620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:07:34.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:34.248+0000 7f27ef200700 1 --2- 172.21.15.28:0/195323455 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27f01083e0 0x7f27f00750a0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f27e800ef30 tx=0x7f27e800c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:07:34.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:34.249+0000 7f27e7400700 1 -- 172.21.15.28:0/195323455 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27e80092e0 con 0x7f27f01083e0 2024-05-12T22:07:34.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:34.249+0000 7f27e7400700 1 -- 172.21.15.28:0/195323455 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f27e8004bb0 con 0x7f27f01083e0 2024-05-12T22:07:34.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:34.249+0000 7f27f5400700 1 -- 172.21.15.28:0/195323455 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27f01a72c0 con 0x7f27f01083e0 2024-05-12T22:07:34.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:34.249+0000 7f27f5400700 1 -- 172.21.15.28:0/195323455 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27f01a7850 con 0x7f27f01083e0 2024-05-12T22:07:34.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:34.249+0000 7f27e7400700 1 -- 172.21.15.28:0/195323455 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27e8004280 con 0x7f27f01083e0 2024-05-12T22:07:34.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:34.250+0000 7f27f5400700 1 -- 172.21.15.28:0/195323455 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f27f0069010 con 0x7f27f01083e0 2024-05-12T22:07:34.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:34.252+0000 7f27e7400700 1 -- 172.21.15.28:0/195323455 <== mon.2 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f27e801b030 con 0x7f27f01083e0 2024-05-12T22:07:34.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:34.253+0000 7f27e7400700 1 --2- 172.21.15.28:0/195323455 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f27d40a2d30 0x7f27d40a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:07:34.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:34.254+0000 7f27e7400700 1 -- 172.21.15.28:0/195323455 <== mon.2 v2:172.21.15.86:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f27e80cec60 con 0x7f27f01083e0 2024-05-12T22:07:34.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:34.254+0000 7f27ee800700 1 --2- 172.21.15.28:0/195323455 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f27d40a2d30 0x7f27d40a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:07:34.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:34.255+0000 7f27ee800700 1 --2- 172.21.15.28:0/195323455 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f27d40a2d30 0x7f27d40a51f0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f27e0009680 tx=0x7f27e0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:07:34.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:34.258+0000 7f27e7400700 1 -- 172.21.15.28:0/195323455 <== mon.2 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f27e8022020 con 0x7f27f01083e0 2024-05-12T22:07:34.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:34.418+0000 7fe85ae00700 1 -- 172.21.15.28:0/504752050 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) -- 0x7fe8541a0450 con 0x7fe854102e60 2024-05-12T22:07:34.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:34.418+0000 7fe852000700 1 -- 172.21.15.28:0/504752050 <== mon.2 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7fe84009ccd0 con 0x7fe854102e60 2024-05-12T22:07:34.420 INFO:teuthology.orchestra.run.smithi028.stdout:313532612637 2024-05-12T22:07:34.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:34.422+0000 7fe85ae00700 1 -- 172.21.15.28:0/504752050 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fe83c0a2e50 msgr2=0x7fe83c0a5310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:34.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:34.422+0000 7fe85ae00700 1 --2- 172.21.15.28:0/504752050 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fe83c0a2e50 0x7fe83c0a5310 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fe848009da0 tx=0x7fe84800b040 comp rx=0 tx=0).stop 2024-05-12T22:07:34.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:34.422+0000 7fe85ae00700 1 -- 172.21.15.28:0/504752050 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe854102e60 msgr2=0x7fe85410c8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:34.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:34.422+0000 7fe85ae00700 1 --2- 172.21.15.28:0/504752050 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe854102e60 0x7fe85410c8b0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fe840005f50 tx=0x7fe840003a90 comp rx=0 tx=0).stop 2024-05-12T22:07:34.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:34.422+0000 7fe85ae00700 1 -- 172.21.15.28:0/504752050 shutdown_connections 2024-05-12T22:07:34.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:34.422+0000 7fe85ae00700 1 --2- 172.21.15.28:0/504752050 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fe83c0a2e50 0x7fe83c0a5310 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:34.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:34.422+0000 7fe85ae00700 1 --2- 172.21.15.28:0/504752050 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe854107590 0x7fe8541a39c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:34.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:34.422+0000 7fe85ae00700 1 --2- 172.21.15.28:0/504752050 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8541024b0 0x7fe85410c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:34.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:34.422+0000 7fe85ae00700 1 --2- 172.21.15.28:0/504752050 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe854102e60 0x7fe85410c8b0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:34.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:34.422+0000 7fe85ae00700 1 -- 172.21.15.28:0/504752050 >> 172.21.15.28:0/504752050 conn(0x7fe8540fdda0 msgr2=0x7fe85410ee30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:07:34.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:34.422+0000 7fe85ae00700 1 -- 172.21.15.28:0/504752050 shutdown_connections 2024-05-12T22:07:34.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:34.422+0000 7fe85ae00700 1 -- 172.21.15.28:0/504752050 wait complete. 2024-05-12T22:07:34.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:34.496+0000 7f27f5400700 1 -- 172.21.15.28:0/195323455 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) -- 0x7f27f002db80 con 0x7f27f01083e0 2024-05-12T22:07:34.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:34.497+0000 7f27e7400700 1 -- 172.21.15.28:0/195323455 <== mon.2 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f27e8010040 con 0x7f27f01083e0 2024-05-12T22:07:34.498 INFO:teuthology.orchestra.run.smithi028.stdout:292057776163 2024-05-12T22:07:34.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:34.500+0000 7f27f5400700 1 -- 172.21.15.28:0/195323455 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f27d40a2d30 msgr2=0x7f27d40a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:34.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:34.500+0000 7f27f5400700 1 --2- 172.21.15.28:0/195323455 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f27d40a2d30 0x7f27d40a51f0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f27e0009680 tx=0x7f27e0009260 comp rx=0 tx=0).stop 2024-05-12T22:07:34.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:34.500+0000 7f27f5400700 1 -- 172.21.15.28:0/195323455 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27f01083e0 msgr2=0x7f27f00750a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:34.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:34.500+0000 7f27f5400700 1 --2- 172.21.15.28:0/195323455 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27f01083e0 0x7f27f00750a0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f27e800ef30 tx=0x7f27e800c3a0 comp rx=0 tx=0).stop 2024-05-12T22:07:34.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:34.500+0000 7f27f5400700 1 -- 172.21.15.28:0/195323455 shutdown_connections 2024-05-12T22:07:34.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:34.500+0000 7f27f5400700 1 --2- 172.21.15.28:0/195323455 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f27d40a2d30 0x7f27d40a51f0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:34.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:34.500+0000 7f27f5400700 1 --2- 172.21.15.28:0/195323455 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f27f01023a0 0x7f27f0074620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:34.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:34.500+0000 7f27f5400700 1 --2- 172.21.15.28:0/195323455 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27f01083e0 0x7f27f00750a0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:34.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:34.500+0000 7f27f5400700 1 --2- 172.21.15.28:0/195323455 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27f0102d40 0x7f27f0074b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:34.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:34.500+0000 7f27f5400700 1 -- 172.21.15.28:0/195323455 >> 172.21.15.28:0/195323455 conn(0x7f27f00793f0 msgr2=0x7f27f0100fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:07:34.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:34.500+0000 7f27f5400700 1 -- 172.21.15.28:0/195323455 shutdown_connections 2024-05-12T22:07:34.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:34.500+0000 7f27f5400700 1 -- 172.21.15.28:0/195323455 wait complete. 2024-05-12T22:07:35.389 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/mon.a/config 2024-05-12T22:07:35.398 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/mon.a/config 2024-05-12T22:07:35.655 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:07:35 smithi028 ceph-mon[29847]: pgmap v342: 97 pgs: 97 active+clean; 640 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-05-12T22:07:35.656 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:07:35 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/504752050' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-05-12T22:07:35.656 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:07:35 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/195323455' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-05-12T22:07:35.656 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:07:35 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:07:35.673 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:07:35 smithi186 ceph-mon[34785]: pgmap v342: 97 pgs: 97 active+clean; 640 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-05-12T22:07:35.673 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:07:35 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/504752050' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-05-12T22:07:35.673 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:07:35 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/195323455' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-05-12T22:07:35.673 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:07:35 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:07:35.702 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:07:35 smithi086 ceph-mon[34297]: pgmap v342: 97 pgs: 97 active+clean; 640 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-05-12T22:07:35.702 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:07:35 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/504752050' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-05-12T22:07:35.702 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:07:35 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/195323455' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-05-12T22:07:35.703 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:07:35 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:07:36.305 INFO:tasks.cephadm.ceph_manager.ceph:need seq 292057776159 got 292057776163 for osd.9 2024-05-12T22:07:36.305 DEBUG:teuthology.parallel:result is None 2024-05-12T22:07:36.647 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/mon.a/config 2024-05-12T22:07:36.652 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/mon.a/config 2024-05-12T22:07:36.684 INFO:tasks.cephadm.ceph_manager.ceph:need seq 313532612634 got 313532612637 for osd.10 2024-05-12T22:07:36.684 DEBUG:teuthology.parallel:result is None 2024-05-12T22:07:37.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:07:37 smithi186 ceph-mon[34785]: pgmap v343: 97 pgs: 97 active+clean; 640 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-12T22:07:37.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:07:37 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:07:37.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:07:37 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:07:37.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:07:37 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:07:37.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:07:37 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:07:37.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:07:37 smithi086 ceph-mon[34297]: pgmap v343: 97 pgs: 97 active+clean; 640 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-12T22:07:37.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:07:37 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:07:37.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:07:37 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:07:37.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:07:37 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:07:37.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:07:37 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:07:37.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:07:37 smithi028 ceph-mon[29847]: pgmap v343: 97 pgs: 97 active+clean; 640 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-12T22:07:37.858 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:07:37 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:07:37.858 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:07:37 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:07:37.858 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:07:37 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:07:37.858 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:07:37 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:07:38.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:38.734+0000 7f2e29e00700 1 -- 172.21.15.28:0/287151191 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e24108450 msgr2=0x7f2e24108830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:38.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:38.734+0000 7f2e29e00700 1 --2- 172.21.15.28:0/287151191 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e24108450 0x7f2e24108830 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f2e14009990 tx=0x7f2e14009ca0 comp rx=0 tx=0).stop 2024-05-12T22:07:38.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:38.734+0000 7f2e29e00700 1 -- 172.21.15.28:0/287151191 shutdown_connections 2024-05-12T22:07:38.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:38.734+0000 7f2e29e00700 1 --2- 172.21.15.28:0/287151191 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e24102d70 0x7f2e2410f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:38.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:38.734+0000 7f2e29e00700 1 --2- 172.21.15.28:0/287151191 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e24108450 0x7f2e24108830 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:38.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:38.734+0000 7f2e29e00700 1 --2- 172.21.15.28:0/287151191 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2e241023b0 0x7f2e24102830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:38.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:38.734+0000 7f2e29e00700 1 -- 172.21.15.28:0/287151191 >> 172.21.15.28:0/287151191 conn(0x7f2e24079510 msgr2=0x7f2e24079920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:07:38.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:38.734+0000 7f2e29e00700 1 -- 172.21.15.28:0/287151191 shutdown_connections 2024-05-12T22:07:38.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:38.734+0000 7f2e29e00700 1 -- 172.21.15.28:0/287151191 wait complete. 2024-05-12T22:07:38.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:38.736+0000 7f2e29e00700 1 Processor -- start 2024-05-12T22:07:38.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:38.736+0000 7f2e29e00700 1 -- start start 2024-05-12T22:07:38.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:38.736+0000 7f2e29e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e241023b0 0x7f2e2419e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:07:38.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:38.736+0000 7f2e29e00700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2e24102d70 0x7f2e2419ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:07:38.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:38.736+0000 7f2e29e00700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e24108450 0x7f2e24198930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:07:38.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:38.736+0000 7f2e29e00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2e24112690 con 0x7f2e241023b0 2024-05-12T22:07:38.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:38.736+0000 7f2e29e00700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f2e24112510 con 0x7f2e24108450 2024-05-12T22:07:38.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:38.736+0000 7f2e29e00700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f2e24112810 con 0x7f2e24102d70 2024-05-12T22:07:38.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:38.737+0000 7f2e23400700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e24108450 0x7f2e24198930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:07:38.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:38.737+0000 7f2e22a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e241023b0 0x7f2e2419e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:07:38.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:38.737+0000 7f2e23400700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e24108450 0x7f2e24198930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:60072/0 (socket says 172.21.15.28:60072) 2024-05-12T22:07:38.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:38.737+0000 7f2e22a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e241023b0 0x7f2e2419e770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40758/0 (socket says 172.21.15.28:40758) 2024-05-12T22:07:38.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:38.737+0000 7f2e23400700 1 -- 172.21.15.28:0/1322351755 learned_addr learned my addr 172.21.15.28:0/1322351755 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:07:38.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:38.737+0000 7f2e22000700 1 --2- 172.21.15.28:0/1322351755 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2e24102d70 0x7f2e2419ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:07:38.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:38.737+0000 7f2e23400700 1 -- 172.21.15.28:0/1322351755 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2e24102d70 msgr2=0x7f2e2419ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:38.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:38.737+0000 7f2e23400700 1 --2- 172.21.15.28:0/1322351755 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2e24102d70 0x7f2e2419ecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:38.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:38.737+0000 7f2e23400700 1 -- 172.21.15.28:0/1322351755 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e241023b0 msgr2=0x7f2e2419e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:38.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:38.737+0000 7f2e23400700 1 --2- 172.21.15.28:0/1322351755 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e241023b0 0x7f2e2419e770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:38.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:38.737+0000 7f2e23400700 1 -- 172.21.15.28:0/1322351755 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e24199090 con 0x7f2e24108450 2024-05-12T22:07:38.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:38.737+0000 7f2e22a00700 1 --2- 172.21.15.28:0/1322351755 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e241023b0 0x7f2e2419e770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:07:38.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:38.737+0000 7f2e23400700 1 --2- 172.21.15.28:0/1322351755 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e24108450 0x7f2e24198930 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f2e1c00ef30 tx=0x7f2e1c00c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:07:38.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:38.738+0000 7f2e28a00700 1 -- 172.21.15.28:0/1322351755 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e1c0092e0 con 0x7f2e24108450 2024-05-12T22:07:38.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:38.738+0000 7f2e28a00700 1 -- 172.21.15.28:0/1322351755 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2e1c004bb0 con 0x7f2e24108450 2024-05-12T22:07:38.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:38.738+0000 7f2e29e00700 1 -- 172.21.15.28:0/1322351755 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e24199320 con 0x7f2e24108450 2024-05-12T22:07:38.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:38.738+0000 7f2e29e00700 1 -- 172.21.15.28:0/1322351755 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e24074550 con 0x7f2e24108450 2024-05-12T22:07:38.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:38.738+0000 7f2e28a00700 1 -- 172.21.15.28:0/1322351755 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e1c004280 con 0x7f2e24108450 2024-05-12T22:07:38.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:38.739+0000 7f2e29e00700 1 -- 172.21.15.28:0/1322351755 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e2410a910 con 0x7f2e24108450 2024-05-12T22:07:38.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:38.741+0000 7f2e28a00700 1 -- 172.21.15.28:0/1322351755 <== mon.2 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f2e1c01b030 con 0x7f2e24108450 2024-05-12T22:07:38.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:38.743+0000 7f2e28a00700 1 --2- 172.21.15.28:0/1322351755 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f2e080a2d30 0x7f2e080a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:07:38.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:38.743+0000 7f2e28a00700 1 -- 172.21.15.28:0/1322351755 <== mon.2 v2:172.21.15.86:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f2e1c0cec80 con 0x7f2e24108450 2024-05-12T22:07:38.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:38.743+0000 7f2e22a00700 1 --2- 172.21.15.28:0/1322351755 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f2e080a2d30 0x7f2e080a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:07:38.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:38.744+0000 7f2e22a00700 1 --2- 172.21.15.28:0/1322351755 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f2e080a2d30 0x7f2e080a51f0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f2e140098e0 tx=0x7f2e1401e040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:07:38.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:38.747+0000 7f2e28a00700 1 -- 172.21.15.28:0/1322351755 <== mon.2 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f2e1c022020 con 0x7f2e24108450 2024-05-12T22:07:38.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:38.983+0000 7f2e29e00700 1 -- 172.21.15.28:0/1322351755 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f2e24199e60 con 0x7f2e24108450 2024-05-12T22:07:38.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:38.984+0000 7f2e28a00700 1 -- 172.21.15.28:0/1322351755 <== mon.2 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f2e1c010040 con 0x7f2e24108450 2024-05-12T22:07:38.985 INFO:teuthology.orchestra.run.smithi028.stdout:34359738453 2024-05-12T22:07:38.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:38.987+0000 7f2e29e00700 1 -- 172.21.15.28:0/1322351755 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f2e080a2d30 msgr2=0x7f2e080a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:38.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:38.987+0000 7f2e29e00700 1 --2- 172.21.15.28:0/1322351755 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f2e080a2d30 0x7f2e080a51f0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f2e140098e0 tx=0x7f2e1401e040 comp rx=0 tx=0).stop 2024-05-12T22:07:38.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:38.987+0000 7f2e29e00700 1 -- 172.21.15.28:0/1322351755 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e24108450 msgr2=0x7f2e24198930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:38.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:38.987+0000 7f2e29e00700 1 --2- 172.21.15.28:0/1322351755 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e24108450 0x7f2e24198930 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f2e1c00ef30 tx=0x7f2e1c00c3a0 comp rx=0 tx=0).stop 2024-05-12T22:07:38.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:38.987+0000 7f2e29e00700 1 -- 172.21.15.28:0/1322351755 shutdown_connections 2024-05-12T22:07:38.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:38.987+0000 7f2e29e00700 1 --2- 172.21.15.28:0/1322351755 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f2e080a2d30 0x7f2e080a51f0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:38.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:38.987+0000 7f2e29e00700 1 --2- 172.21.15.28:0/1322351755 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e24108450 0x7f2e24198930 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:38.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:38.987+0000 7f2e29e00700 1 --2- 172.21.15.28:0/1322351755 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e241023b0 0x7f2e2419e770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:38.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:38.987+0000 7f2e29e00700 1 --2- 172.21.15.28:0/1322351755 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2e24102d70 0x7f2e2419ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:38.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:38.987+0000 7f2e29e00700 1 -- 172.21.15.28:0/1322351755 >> 172.21.15.28:0/1322351755 conn(0x7f2e24079510 msgr2=0x7f2e241011c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:07:38.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:38.987+0000 7f2e29e00700 1 -- 172.21.15.28:0/1322351755 shutdown_connections 2024-05-12T22:07:38.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:38.987+0000 7f2e29e00700 1 -- 172.21.15.28:0/1322351755 wait complete. 2024-05-12T22:07:39.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:07:39 smithi186 ceph-mon[34785]: pgmap v344: 97 pgs: 97 active+clean; 640 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-12T22:07:39.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:07:39 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/1322351755' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-05-12T22:07:39.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:07:39 smithi086 ceph-mon[34297]: pgmap v344: 97 pgs: 97 active+clean; 640 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-12T22:07:39.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:07:39 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/1322351755' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-05-12T22:07:39.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:07:39 smithi028 ceph-mon[29847]: pgmap v344: 97 pgs: 97 active+clean; 640 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-12T22:07:39.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:07:39 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/1322351755' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-05-12T22:07:41.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:07:41 smithi086 ceph-mon[34297]: pgmap v345: 97 pgs: 97 active+clean; 640 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-12T22:07:41.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:07:41 smithi028 ceph-mon[29847]: pgmap v345: 97 pgs: 97 active+clean; 640 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-12T22:07:41.866 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/mon.a/config 2024-05-12T22:07:41.868 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/mon.a/config 2024-05-12T22:07:41.873 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/mon.a/config 2024-05-12T22:07:41.906 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738449 got 34359738453 for osd.0 2024-05-12T22:07:41.906 DEBUG:teuthology.parallel:result is None 2024-05-12T22:07:41.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:07:41 smithi186 ceph-mon[34785]: pgmap v345: 97 pgs: 97 active+clean; 640 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-12T22:07:43.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:07:43 smithi086 ceph-mon[34297]: pgmap v346: 97 pgs: 97 active+clean; 640 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:07:43.745 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:07:43 smithi028 ceph-mon[29847]: pgmap v346: 97 pgs: 97 active+clean; 640 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:07:43.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:07:43 smithi186 ceph-mon[34785]: pgmap v346: 97 pgs: 97 active+clean; 640 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:07:44.584 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:07:44 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/3897567388' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-05-12T22:07:44.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:07:44 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/3897567388' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-05-12T22:07:44.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:07:44 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/3897567388' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-05-12T22:07:45.537 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:07:45 smithi028 ceph-mon[29847]: pgmap v347: 97 pgs: 97 active+clean; 640 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-12T22:07:45.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:07:45 smithi086 ceph-mon[34297]: pgmap v347: 97 pgs: 97 active+clean; 640 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-12T22:07:45.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:07:45 smithi186 ceph-mon[34785]: pgmap v347: 97 pgs: 97 active+clean; 640 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-12T22:07:46.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:45.936+0000 7f7c5a000700 1 -- 172.21.15.28:0/2461996492 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7c541025d0 msgr2=0x7f7c541029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:46.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:45.936+0000 7f7c5a000700 1 --2- 172.21.15.28:0/2461996492 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7c541025d0 0x7f7c541029b0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f7c44009990 tx=0x7f7c44009ca0 comp rx=0 tx=0).stop 2024-05-12T22:07:46.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:45.936+0000 7f7c5a000700 1 -- 172.21.15.28:0/2461996492 shutdown_connections 2024-05-12T22:07:46.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:45.936+0000 7f7c5a000700 1 --2- 172.21.15.28:0/2461996492 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c54107690 0x7f7c5410f260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:46.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:45.936+0000 7f7c5a000700 1 --2- 172.21.15.28:0/2461996492 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7c541025d0 0x7f7c541029b0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:46.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:45.936+0000 7f7c5a000700 1 --2- 172.21.15.28:0/2461996492 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7c54102f80 0x7f7c54106fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:46.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:45.936+0000 7f7c5a000700 1 -- 172.21.15.28:0/2461996492 >> 172.21.15.28:0/2461996492 conn(0x7f7c540794d0 msgr2=0x7f7c540798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:07:46.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:45.936+0000 7f7c5a000700 1 -- 172.21.15.28:0/2461996492 shutdown_connections 2024-05-12T22:07:46.346 INFO:teuthology.orchestra.run.smithi028.stdout:167503724604 2024-05-12T22:07:46.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:45.937+0000 7f7c5a000700 1 -- 172.21.15.28:0/2461996492 wait complete. 2024-05-12T22:07:46.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:45.938+0000 7f7c5a000700 1 Processor -- start 2024-05-12T22:07:46.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:45.938+0000 7f7c5a000700 1 -- start start 2024-05-12T22:07:46.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:45.938+0000 7f7c5a000700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7c541025d0 0x7f7c5419e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:07:46.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:45.938+0000 7f7c5a000700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c54102f80 0x7f7c5419ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:07:46.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:45.938+0000 7f7c5a000700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7c54107690 0x7f7c54198960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:07:46.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:45.938+0000 7f7c5a000700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7c541126c0 con 0x7f7c54102f80 2024-05-12T22:07:46.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:45.938+0000 7f7c5a000700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f7c54112540 con 0x7f7c541025d0 2024-05-12T22:07:46.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:45.938+0000 7f7c5a000700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f7c54112840 con 0x7f7c54107690 2024-05-12T22:07:46.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:45.939+0000 7f7c53400700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7c541025d0 0x7f7c5419e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:07:46.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:45.939+0000 7f7c52a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c54102f80 0x7f7c5419ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:07:46.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:45.939+0000 7f7c53400700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7c541025d0 0x7f7c5419e7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:57214/0 (socket says 172.21.15.28:57214) 2024-05-12T22:07:46.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:45.939+0000 7f7c53400700 1 -- 172.21.15.28:0/4215489072 learned_addr learned my addr 172.21.15.28:0/4215489072 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:07:46.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:45.939+0000 7f7c53e00700 1 --2- 172.21.15.28:0/4215489072 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7c54107690 0x7f7c54198960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:07:46.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:45.939+0000 7f7c53400700 1 -- 172.21.15.28:0/4215489072 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7c54107690 msgr2=0x7f7c54198960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:46.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:45.939+0000 7f7c53400700 1 --2- 172.21.15.28:0/4215489072 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7c54107690 0x7f7c54198960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:46.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:45.939+0000 7f7c53400700 1 -- 172.21.15.28:0/4215489072 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c54102f80 msgr2=0x7f7c5419ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:46.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:45.939+0000 7f7c53400700 1 --2- 172.21.15.28:0/4215489072 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c54102f80 0x7f7c5419ece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:46.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:45.939+0000 7f7c53400700 1 -- 172.21.15.28:0/4215489072 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c541990c0 con 0x7f7c541025d0 2024-05-12T22:07:46.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:45.939+0000 7f7c53400700 1 --2- 172.21.15.28:0/4215489072 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7c541025d0 0x7f7c5419e7a0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f7c44005910 tx=0x7f7c4400bab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:07:46.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:45.940+0000 7f7c4be00700 1 -- 172.21.15.28:0/4215489072 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c44027070 con 0x7f7c541025d0 2024-05-12T22:07:46.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:45.940+0000 7f7c4be00700 1 -- 172.21.15.28:0/4215489072 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7c44003d70 con 0x7f7c541025d0 2024-05-12T22:07:46.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:45.940+0000 7f7c5a000700 1 -- 172.21.15.28:0/4215489072 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c54199350 con 0x7f7c541025d0 2024-05-12T22:07:46.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:45.940+0000 7f7c4be00700 1 -- 172.21.15.28:0/4215489072 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c4400f460 con 0x7f7c541025d0 2024-05-12T22:07:46.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:45.940+0000 7f7c5a000700 1 -- 172.21.15.28:0/4215489072 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c541a5670 con 0x7f7c541025d0 2024-05-12T22:07:46.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:45.941+0000 7f7c5a000700 1 -- 172.21.15.28:0/4215489072 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c54069010 con 0x7f7c541025d0 2024-05-12T22:07:46.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:45.943+0000 7f7c4be00700 1 -- 172.21.15.28:0/4215489072 <== mon.2 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f7c4402a070 con 0x7f7c541025d0 2024-05-12T22:07:46.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:45.944+0000 7f7c4be00700 1 --2- 172.21.15.28:0/4215489072 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f7c380a2d30 0x7f7c380a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:07:46.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:45.944+0000 7f7c4be00700 1 -- 172.21.15.28:0/4215489072 <== mon.2 v2:172.21.15.86:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f7c440cff20 con 0x7f7c541025d0 2024-05-12T22:07:46.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:45.945+0000 7f7c52a00700 1 --2- 172.21.15.28:0/4215489072 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f7c380a2d30 0x7f7c380a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:07:46.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:45.945+0000 7f7c52a00700 1 --2- 172.21.15.28:0/4215489072 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f7c380a2d30 0x7f7c380a51f0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f7c54107170 tx=0x7f7c40009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:07:46.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:45.948+0000 7f7c4be00700 1 -- 172.21.15.28:0/4215489072 <== mon.2 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f7c44013070 con 0x7f7c541025d0 2024-05-12T22:07:46.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:46.192+0000 7f7c5a000700 1 -- 172.21.15.28:0/4215489072 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f7c5402db80 con 0x7f7c541025d0 2024-05-12T22:07:46.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:46.192+0000 7f7c4be00700 1 -- 172.21.15.28:0/4215489072 <== mon.2 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f7c4409cd60 con 0x7f7c541025d0 2024-05-12T22:07:46.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:46.196+0000 7f7c5a000700 1 -- 172.21.15.28:0/4215489072 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f7c380a2d30 msgr2=0x7f7c380a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:46.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:46.196+0000 7f7c5a000700 1 --2- 172.21.15.28:0/4215489072 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f7c380a2d30 0x7f7c380a51f0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f7c54107170 tx=0x7f7c40009260 comp rx=0 tx=0).stop 2024-05-12T22:07:46.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:46.196+0000 7f7c5a000700 1 -- 172.21.15.28:0/4215489072 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7c541025d0 msgr2=0x7f7c5419e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:46.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:46.196+0000 7f7c5a000700 1 --2- 172.21.15.28:0/4215489072 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7c541025d0 0x7f7c5419e7a0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f7c44005910 tx=0x7f7c4400bab0 comp rx=0 tx=0).stop 2024-05-12T22:07:46.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:46.196+0000 7f7c5a000700 1 -- 172.21.15.28:0/4215489072 shutdown_connections 2024-05-12T22:07:46.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:46.196+0000 7f7c5a000700 1 --2- 172.21.15.28:0/4215489072 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f7c380a2d30 0x7f7c380a51f0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:46.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:46.196+0000 7f7c5a000700 1 --2- 172.21.15.28:0/4215489072 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7c54107690 0x7f7c54198960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:46.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:46.196+0000 7f7c5a000700 1 --2- 172.21.15.28:0/4215489072 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7c541025d0 0x7f7c5419e7a0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:46.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:46.196+0000 7f7c5a000700 1 --2- 172.21.15.28:0/4215489072 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c54102f80 0x7f7c5419ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:46.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:46.196+0000 7f7c5a000700 1 -- 172.21.15.28:0/4215489072 >> 172.21.15.28:0/4215489072 conn(0x7f7c540794d0 msgr2=0x7f7c54101af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:07:46.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:46.196+0000 7f7c5a000700 1 -- 172.21.15.28:0/4215489072 shutdown_connections 2024-05-12T22:07:46.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:46.196+0000 7f7c5a000700 1 -- 172.21.15.28:0/4215489072 wait complete. 2024-05-12T22:07:46.715 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:07:46 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/4215489072' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-05-12T22:07:46.715 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:07:46 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/4247761325' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-05-12T22:07:46.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:07:46 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/4215489072' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-05-12T22:07:46.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:07:46 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/4247761325' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-05-12T22:07:46.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:07:46 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/4215489072' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-05-12T22:07:46.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:07:46 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/4247761325' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-05-12T22:07:47.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:43.744+0000 7fd7ff000700 1 -- 172.21.15.28:0/292214349 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd7f8104560 msgr2=0x7fd7f81049e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:47.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:43.744+0000 7fd7ff000700 1 --2- 172.21.15.28:0/292214349 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd7f8104560 0x7fd7f81049e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fd7e40099c0 tx=0x7fd7e4009cd0 comp rx=0 tx=0).stop 2024-05-12T22:07:47.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:43.744+0000 7fd7ff000700 1 -- 172.21.15.28:0/292214349 shutdown_connections 2024-05-12T22:07:47.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:43.744+0000 7fd7ff000700 1 --2- 172.21.15.28:0/292214349 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd7f8104f20 0x7fd7f81113e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:47.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:43.744+0000 7fd7ff000700 1 --2- 172.21.15.28:0/292214349 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7f810a610 0x7fd7f810a9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:47.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:43.744+0000 7fd7ff000700 1 --2- 172.21.15.28:0/292214349 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd7f8104560 0x7fd7f81049e0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:47.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:43.745+0000 7fd7ff000700 1 -- 172.21.15.28:0/292214349 >> 172.21.15.28:0/292214349 conn(0x7fd7f8100030 msgr2=0x7fd7f8102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:07:47.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:43.745+0000 7fd7ff000700 1 -- 172.21.15.28:0/292214349 shutdown_connections 2024-05-12T22:07:47.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:43.745+0000 7fd7ff000700 1 -- 172.21.15.28:0/292214349 wait complete. 2024-05-12T22:07:47.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:43.746+0000 7fd7ff000700 1 Processor -- start 2024-05-12T22:07:47.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:43.746+0000 7fd7ff000700 1 -- start start 2024-05-12T22:07:47.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:43.747+0000 7fd7ff000700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd7f8104560 0x7fd7f81a08c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:07:47.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:43.747+0000 7fd7ff000700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7f8104f20 0x7fd7f81a0e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:07:47.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:43.747+0000 7fd7ff000700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd7f810a610 0x7fd7f819aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:07:47.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:43.747+0000 7fd7ff000700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd7f81148b0 con 0x7fd7f8104f20 2024-05-12T22:07:47.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:43.747+0000 7fd7ff000700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fd7f8114730 con 0x7fd7f8104560 2024-05-12T22:07:47.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:43.747+0000 7fd7ff000700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd7f8114a30 con 0x7fd7f810a610 2024-05-12T22:07:47.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:43.747+0000 7fd7f7400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7f8104f20 0x7fd7f81a0e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:07:47.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:43.747+0000 7fd7f7e00700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd7f8104560 0x7fd7f81a08c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:07:47.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:43.747+0000 7fd7f7400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7f8104f20 0x7fd7f81a0e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40772/0 (socket says 172.21.15.28:40772) 2024-05-12T22:07:47.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:43.747+0000 7fd7f7400700 1 -- 172.21.15.28:0/3897567388 learned_addr learned my addr 172.21.15.28:0/3897567388 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:07:47.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:43.747+0000 7fd7fd000700 1 --2- 172.21.15.28:0/3897567388 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd7f810a610 0x7fd7f819aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:07:47.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:43.748+0000 7fd7f7e00700 1 -- 172.21.15.28:0/3897567388 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd7f810a610 msgr2=0x7fd7f819aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:47.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:43.748+0000 7fd7f7e00700 1 --2- 172.21.15.28:0/3897567388 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd7f810a610 0x7fd7f819aad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:47.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:43.748+0000 7fd7f7e00700 1 -- 172.21.15.28:0/3897567388 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7f8104f20 msgr2=0x7fd7f81a0e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:47.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:43.748+0000 7fd7f7e00700 1 --2- 172.21.15.28:0/3897567388 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7f8104f20 0x7fd7f81a0e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:47.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:43.748+0000 7fd7f7e00700 1 -- 172.21.15.28:0/3897567388 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7f819b230 con 0x7fd7f8104560 2024-05-12T22:07:47.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:43.748+0000 7fd7f7e00700 1 --2- 172.21.15.28:0/3897567388 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd7f8104560 0x7fd7f81a08c0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fd7e800e9f0 tx=0x7fd7e800ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:07:47.739 INFO:teuthology.orchestra.run.smithi028.stdout:77309411405 2024-05-12T22:07:47.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:43.748+0000 7fd7f4c00700 1 -- 172.21.15.28:0/3897567388 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7e800cb70 con 0x7fd7f8104560 2024-05-12T22:07:47.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:43.748+0000 7fd7f4c00700 1 -- 172.21.15.28:0/3897567388 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd7e800ee90 con 0x7fd7f8104560 2024-05-12T22:07:47.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:43.748+0000 7fd7f4c00700 1 -- 172.21.15.28:0/3897567388 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7e8003dc0 con 0x7fd7f8104560 2024-05-12T22:07:47.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:43.748+0000 7fd7ff000700 1 -- 172.21.15.28:0/3897567388 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7f819b520 con 0x7fd7f8104560 2024-05-12T22:07:47.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:43.748+0000 7fd7ff000700 1 -- 172.21.15.28:0/3897567388 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7f81a7790 con 0x7fd7f8104560 2024-05-12T22:07:47.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:43.750+0000 7fd7ff000700 1 -- 172.21.15.28:0/3897567388 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7f8069010 con 0x7fd7f8104560 2024-05-12T22:07:47.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:43.751+0000 7fd7f4c00700 1 -- 172.21.15.28:0/3897567388 <== mon.2 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fd7e8003fa0 con 0x7fd7f8104560 2024-05-12T22:07:47.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:43.753+0000 7fd7f4c00700 1 --2- 172.21.15.28:0/3897567388 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fd7dc0a2cb0 0x7fd7dc0a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:07:47.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:43.753+0000 7fd7f4c00700 1 -- 172.21.15.28:0/3897567388 <== mon.2 v2:172.21.15.86:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7fd7e80cf380 con 0x7fd7f8104560 2024-05-12T22:07:47.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:43.753+0000 7fd7f7400700 1 --2- 172.21.15.28:0/3897567388 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fd7dc0a2cb0 0x7fd7dc0a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:07:47.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:43.754+0000 7fd7f7400700 1 --2- 172.21.15.28:0/3897567388 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fd7dc0a2cb0 0x7fd7dc0a5170 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fd7f81092e0 tx=0x7fd7e4005ab0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:07:47.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:43.757+0000 7fd7f4c00700 1 -- 172.21.15.28:0/3897567388 <== mon.2 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fd7e809c130 con 0x7fd7f8104560 2024-05-12T22:07:47.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:43.997+0000 7fd7ff000700 1 -- 172.21.15.28:0/3897567388 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7fd7f802db80 con 0x7fd7f8104560 2024-05-12T22:07:47.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:43.998+0000 7fd7f4c00700 1 -- 172.21.15.28:0/3897567388 <== mon.2 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fd7e809ba30 con 0x7fd7f8104560 2024-05-12T22:07:47.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:44.001+0000 7fd7ff000700 1 -- 172.21.15.28:0/3897567388 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fd7dc0a2cb0 msgr2=0x7fd7dc0a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:47.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:44.001+0000 7fd7ff000700 1 --2- 172.21.15.28:0/3897567388 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fd7dc0a2cb0 0x7fd7dc0a5170 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fd7f81092e0 tx=0x7fd7e4005ab0 comp rx=0 tx=0).stop 2024-05-12T22:07:47.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:44.001+0000 7fd7ff000700 1 -- 172.21.15.28:0/3897567388 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd7f8104560 msgr2=0x7fd7f81a08c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:47.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:44.001+0000 7fd7ff000700 1 --2- 172.21.15.28:0/3897567388 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd7f8104560 0x7fd7f81a08c0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fd7e800e9f0 tx=0x7fd7e800ed00 comp rx=0 tx=0).stop 2024-05-12T22:07:47.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:44.001+0000 7fd7ff000700 1 -- 172.21.15.28:0/3897567388 shutdown_connections 2024-05-12T22:07:47.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:44.001+0000 7fd7ff000700 1 --2- 172.21.15.28:0/3897567388 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fd7dc0a2cb0 0x7fd7dc0a5170 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:47.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:44.001+0000 7fd7ff000700 1 --2- 172.21.15.28:0/3897567388 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd7f810a610 0x7fd7f819aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:47.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:44.001+0000 7fd7ff000700 1 --2- 172.21.15.28:0/3897567388 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd7f8104560 0x7fd7f81a08c0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:47.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:44.001+0000 7fd7ff000700 1 --2- 172.21.15.28:0/3897567388 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7f8104f20 0x7fd7f81a0e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:47.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:44.001+0000 7fd7ff000700 1 -- 172.21.15.28:0/3897567388 >> 172.21.15.28:0/3897567388 conn(0x7fd7f8100030 msgr2=0x7fd7f8101620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:07:47.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:44.002+0000 7fd7ff000700 1 -- 172.21.15.28:0/3897567388 shutdown_connections 2024-05-12T22:07:47.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:44.002+0000 7fd7ff000700 1 -- 172.21.15.28:0/3897567388 wait complete. 2024-05-12T22:07:47.741 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:07:47 smithi028 ceph-mon[29847]: pgmap v348: 97 pgs: 97 active+clean; 640 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-12T22:07:47.741 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:07:47 smithi028 ceph-mon[29847]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-05-12T22:07:47.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:07:47 smithi086 ceph-mon[34297]: pgmap v348: 97 pgs: 97 active+clean; 640 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-12T22:07:47.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:07:47 smithi086 ceph-mon[34297]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-05-12T22:07:47.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:46.064+0000 7f15d4800700 1 -- 172.21.15.28:0/1743309292 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f15cc06a030 msgr2=0x7f15cc06a410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:47.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:46.064+0000 7f15d4800700 1 --2- 172.21.15.28:0/1743309292 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f15cc06a030 0x7f15cc06a410 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f15c0009990 tx=0x7f15c0009ca0 comp rx=0 tx=0).stop 2024-05-12T22:07:47.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:46.064+0000 7f15d4800700 1 -- 172.21.15.28:0/1743309292 shutdown_connections 2024-05-12T22:07:47.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:46.064+0000 7f15d4800700 1 --2- 172.21.15.28:0/1743309292 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f15cc06a030 0x7f15cc06a410 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:47.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:46.064+0000 7f15d4800700 1 --2- 172.21.15.28:0/1743309292 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15cc10edd0 0x7f15cc1111c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:47.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:46.064+0000 7f15d4800700 1 --2- 172.21.15.28:0/1743309292 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f15cc06a950 0x7f15cc10e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:47.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:46.064+0000 7f15d4800700 1 -- 172.21.15.28:0/1743309292 >> 172.21.15.28:0/1743309292 conn(0x7f15cc0781d0 msgr2=0x7f15cc0785e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:07:47.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:46.064+0000 7f15d4800700 1 -- 172.21.15.28:0/1743309292 shutdown_connections 2024-05-12T22:07:47.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:46.065+0000 7f15d4800700 1 -- 172.21.15.28:0/1743309292 wait complete. 2024-05-12T22:07:47.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:46.066+0000 7f15d4800700 1 Processor -- start 2024-05-12T22:07:47.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:46.066+0000 7f15d4800700 1 -- start start 2024-05-12T22:07:47.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:46.066+0000 7f15d4800700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15cc06a030 0x7f15cc1a0940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:07:47.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:46.066+0000 7f15d4800700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f15cc06a950 0x7f15cc1a0e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:07:47.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:46.066+0000 7f15d4800700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f15cc10edd0 0x7f15cc19ab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:07:47.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:46.066+0000 7f15d4800700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f15cc114850 con 0x7f15cc06a030 2024-05-12T22:07:47.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:46.066+0000 7f15d4800700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f15cc1146d0 con 0x7f15cc06a950 2024-05-12T22:07:47.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:46.066+0000 7f15d4800700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f15cc1149d0 con 0x7f15cc10edd0 2024-05-12T22:07:47.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:46.067+0000 7f15d1400700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f15cc06a950 0x7f15cc1a0e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:07:47.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:46.067+0000 7f15d1400700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f15cc06a950 0x7f15cc1a0e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:57230/0 (socket says 172.21.15.28:57230) 2024-05-12T22:07:47.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:46.067+0000 7f15d1400700 1 -- 172.21.15.28:0/4247761325 learned_addr learned my addr 172.21.15.28:0/4247761325 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:07:47.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:46.067+0000 7f15d1e00700 1 --2- 172.21.15.28:0/4247761325 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15cc06a030 0x7f15cc1a0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:07:47.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:46.067+0000 7f15d2800700 1 --2- 172.21.15.28:0/4247761325 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f15cc10edd0 0x7f15cc19ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:07:47.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:46.067+0000 7f15d1400700 1 -- 172.21.15.28:0/4247761325 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f15cc10edd0 msgr2=0x7f15cc19ab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:47.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:46.067+0000 7f15d1400700 1 --2- 172.21.15.28:0/4247761325 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f15cc10edd0 0x7f15cc19ab50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:47.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:46.067+0000 7f15d1400700 1 -- 172.21.15.28:0/4247761325 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15cc06a030 msgr2=0x7f15cc1a0940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:47.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:46.067+0000 7f15d1400700 1 --2- 172.21.15.28:0/4247761325 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15cc06a030 0x7f15cc1a0940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:47.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:46.067+0000 7f15d1400700 1 -- 172.21.15.28:0/4247761325 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15cc19b2b0 con 0x7f15cc06a950 2024-05-12T22:07:47.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:46.067+0000 7f15d1400700 1 --2- 172.21.15.28:0/4247761325 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f15cc06a950 0x7f15cc1a0e80 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f15bc00e9c0 tx=0x7f15bc00ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:07:47.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:46.068+0000 7f15c6a00700 1 -- 172.21.15.28:0/4247761325 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15bc00cbd0 con 0x7f15cc06a950 2024-05-12T22:07:47.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:46.068+0000 7f15c6a00700 1 -- 172.21.15.28:0/4247761325 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f15bc00cd70 con 0x7f15cc06a950 2024-05-12T22:07:47.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:46.068+0000 7f15c6a00700 1 -- 172.21.15.28:0/4247761325 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15bc010430 con 0x7f15cc06a950 2024-05-12T22:07:47.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:46.068+0000 7f15d4800700 1 -- 172.21.15.28:0/4247761325 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15cc19b5a0 con 0x7f15cc06a950 2024-05-12T22:07:47.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:46.068+0000 7f15d4800700 1 -- 172.21.15.28:0/4247761325 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15cc1a7810 con 0x7f15cc06a950 2024-05-12T22:07:47.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:46.069+0000 7f15d4800700 1 -- 172.21.15.28:0/4247761325 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15cc069010 con 0x7f15cc06a950 2024-05-12T22:07:47.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:46.072+0000 7f15c6a00700 1 -- 172.21.15.28:0/4247761325 <== mon.2 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f15bc004750 con 0x7f15cc06a950 2024-05-12T22:07:47.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:46.073+0000 7f15c6a00700 1 --2- 172.21.15.28:0/4247761325 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f15b40a2cb0 0x7f15b40a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:07:47.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:46.073+0000 7f15c6a00700 1 -- 172.21.15.28:0/4247761325 <== mon.2 v2:172.21.15.86:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f15bc0cf4c0 con 0x7f15cc06a950 2024-05-12T22:07:47.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:46.073+0000 7f15d1e00700 1 --2- 172.21.15.28:0/4247761325 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f15b40a2cb0 0x7f15b40a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:07:47.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:46.074+0000 7f15d1e00700 1 --2- 172.21.15.28:0/4247761325 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f15b40a2cb0 0x7f15b40a5170 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f15c00098e0 tx=0x7f15c0005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:07:47.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:46.077+0000 7f15c6a00700 1 -- 172.21.15.28:0/4247761325 <== mon.2 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f15bc09c270 con 0x7f15cc06a950 2024-05-12T22:07:47.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:46.330+0000 7f15d4800700 1 -- 172.21.15.28:0/4247761325 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) -- 0x7f15cc02db80 con 0x7f15cc06a950 2024-05-12T22:07:47.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:46.331+0000 7f15c6a00700 1 -- 172.21.15.28:0/4247761325 <== mon.2 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f15bc09bb70 con 0x7f15cc06a950 2024-05-12T22:07:47.849 INFO:teuthology.orchestra.run.smithi028.stdout:347892351003 2024-05-12T22:07:47.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:46.333+0000 7f15d4800700 1 -- 172.21.15.28:0/4247761325 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f15b40a2cb0 msgr2=0x7f15b40a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:47.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:46.333+0000 7f15d4800700 1 --2- 172.21.15.28:0/4247761325 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f15b40a2cb0 0x7f15b40a5170 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f15c00098e0 tx=0x7f15c0005960 comp rx=0 tx=0).stop 2024-05-12T22:07:47.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:46.333+0000 7f15d4800700 1 -- 172.21.15.28:0/4247761325 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f15cc06a950 msgr2=0x7f15cc1a0e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:47.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:46.333+0000 7f15d4800700 1 --2- 172.21.15.28:0/4247761325 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f15cc06a950 0x7f15cc1a0e80 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f15bc00e9c0 tx=0x7f15bc00ecd0 comp rx=0 tx=0).stop 2024-05-12T22:07:47.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:46.334+0000 7f15d4800700 1 -- 172.21.15.28:0/4247761325 shutdown_connections 2024-05-12T22:07:47.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:46.334+0000 7f15d4800700 1 --2- 172.21.15.28:0/4247761325 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f15b40a2cb0 0x7f15b40a5170 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:47.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:46.334+0000 7f15d4800700 1 --2- 172.21.15.28:0/4247761325 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f15cc10edd0 0x7f15cc19ab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:47.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:46.334+0000 7f15d4800700 1 --2- 172.21.15.28:0/4247761325 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15cc06a030 0x7f15cc1a0940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:47.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:46.334+0000 7f15d4800700 1 --2- 172.21.15.28:0/4247761325 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f15cc06a950 0x7f15cc1a0e80 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:47.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:46.334+0000 7f15d4800700 1 -- 172.21.15.28:0/4247761325 >> 172.21.15.28:0/4247761325 conn(0x7f15cc0781d0 msgr2=0x7f15cc110220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:07:47.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:46.334+0000 7f15d4800700 1 -- 172.21.15.28:0/4247761325 shutdown_connections 2024-05-12T22:07:47.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:46.334+0000 7f15d4800700 1 -- 172.21.15.28:0/4247761325 wait complete. 2024-05-12T22:07:47.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:07:47 smithi186 ceph-mon[34785]: pgmap v348: 97 pgs: 97 active+clean; 640 KiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-12T22:07:47.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:07:47 smithi186 ceph-mon[34785]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-05-12T22:07:49.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:07:49 smithi086 ceph-mon[34297]: pgmap v349: 97 pgs: 97 active+clean; 641 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 170 B/s wr, 6 op/s 2024-05-12T22:07:49.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:07:49 smithi028 ceph-mon[29847]: pgmap v349: 97 pgs: 97 active+clean; 641 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 170 B/s wr, 6 op/s 2024-05-12T22:07:49.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:07:49 smithi186 ceph-mon[34785]: pgmap v349: 97 pgs: 97 active+clean; 641 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 170 B/s wr, 6 op/s 2024-05-12T22:07:50.665 INFO:tasks.cephadm.ceph_manager.ceph:need seq 347892351000 got 347892351003 for osd.11 2024-05-12T22:07:50.665 DEBUG:teuthology.parallel:result is None 2024-05-12T22:07:50.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:50.899+0000 7f7055c00700 1 -- 172.21.15.28:0/2900886665 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7050069320 msgr2=0x7f7050105120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:50.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:50.899+0000 7f7055c00700 1 --2- 172.21.15.28:0/2900886665 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7050069320 0x7f7050105120 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f704000aaa0 tx=0x7f704000adb0 comp rx=0 tx=0).stop 2024-05-12T22:07:50.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:50.899+0000 7f7055c00700 1 -- 172.21.15.28:0/2900886665 shutdown_connections 2024-05-12T22:07:50.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:50.899+0000 7f7055c00700 1 --2- 172.21.15.28:0/2900886665 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f705010b3c0 0x7f705010d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:50.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:50.899+0000 7f7055c00700 1 --2- 172.21.15.28:0/2900886665 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7050069320 0x7f7050105120 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:50.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:50.899+0000 7f7055c00700 1 --2- 172.21.15.28:0/2900886665 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7050101240 0x7f7050105660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:50.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:50.899+0000 7f7055c00700 1 -- 172.21.15.28:0/2900886665 >> 172.21.15.28:0/2900886665 conn(0x7f7050075fc0 msgr2=0x7f70500763d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:07:50.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:50.899+0000 7f7055c00700 1 -- 172.21.15.28:0/2900886665 shutdown_connections 2024-05-12T22:07:50.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:50.899+0000 7f7055c00700 1 -- 172.21.15.28:0/2900886665 wait complete. 2024-05-12T22:07:50.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:50.900+0000 7f7055c00700 1 Processor -- start 2024-05-12T22:07:50.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:50.900+0000 7f7055c00700 1 -- start start 2024-05-12T22:07:50.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:50.901+0000 7f7055c00700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7050069320 0x7f705019e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:07:50.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:50.901+0000 7f7055c00700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7050101240 0x7f705019eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:07:50.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:50.901+0000 7f7055c00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f705010b3c0 0x7f70501987c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:07:50.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:50.901+0000 7f7055c00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7050110390 con 0x7f705010b3c0 2024-05-12T22:07:50.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:50.901+0000 7f7055c00700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f7050110210 con 0x7f7050101240 2024-05-12T22:07:50.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:50.901+0000 7f7055c00700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f7050110510 con 0x7f7050069320 2024-05-12T22:07:50.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:50.901+0000 7f704fe00700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7050069320 0x7f705019e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:07:50.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:50.901+0000 7f7055200700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f705010b3c0 0x7f70501987c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:07:50.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:50.901+0000 7f704fe00700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7050069320 0x7f705019e640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:60342/0 (socket says 172.21.15.28:60342) 2024-05-12T22:07:50.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:50.901+0000 7f7055200700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f705010b3c0 0x7f70501987c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60414/0 (socket says 172.21.15.28:60414) 2024-05-12T22:07:50.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:50.901+0000 7f704fe00700 1 -- 172.21.15.28:0/1818793830 learned_addr learned my addr 172.21.15.28:0/1818793830 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:07:50.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:50.901+0000 7f704f400700 1 --2- 172.21.15.28:0/1818793830 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7050101240 0x7f705019eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:07:50.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:50.902+0000 7f704fe00700 1 -- 172.21.15.28:0/1818793830 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7050101240 msgr2=0x7f705019eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:50.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:50.902+0000 7f704fe00700 1 --2- 172.21.15.28:0/1818793830 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7050101240 0x7f705019eb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:50.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:50.902+0000 7f704fe00700 1 -- 172.21.15.28:0/1818793830 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f705010b3c0 msgr2=0x7f70501987c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:50.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:50.902+0000 7f704fe00700 1 --2- 172.21.15.28:0/1818793830 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f705010b3c0 0x7f70501987c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:50.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:50.902+0000 7f704fe00700 1 -- 172.21.15.28:0/1818793830 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7050198fa0 con 0x7f7050069320 2024-05-12T22:07:50.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:50.902+0000 7f704fe00700 1 --2- 172.21.15.28:0/1818793830 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7050069320 0x7f705019e640 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f7040005910 tx=0x7f70400040a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:07:50.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:50.902+0000 7f704cc00700 1 -- 172.21.15.28:0/1818793830 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7040023020 con 0x7f7050069320 2024-05-12T22:07:50.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:50.903+0000 7f704cc00700 1 -- 172.21.15.28:0/1818793830 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7040003b60 con 0x7f7050069320 2024-05-12T22:07:50.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:50.903+0000 7f704cc00700 1 -- 172.21.15.28:0/1818793830 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f704000ee10 con 0x7f7050069320 2024-05-12T22:07:50.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:50.903+0000 7f7055c00700 1 -- 172.21.15.28:0/1818793830 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7050199230 con 0x7f7050069320 2024-05-12T22:07:50.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:50.903+0000 7f7055c00700 1 -- 172.21.15.28:0/1818793830 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70501a54a0 con 0x7f7050069320 2024-05-12T22:07:50.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:50.904+0000 7f7055c00700 1 -- 172.21.15.28:0/1818793830 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7050101bd0 con 0x7f7050069320 2024-05-12T22:07:50.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:50.906+0000 7f704cc00700 1 -- 172.21.15.28:0/1818793830 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f7040026020 con 0x7f7050069320 2024-05-12T22:07:50.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:50.907+0000 7f704cc00700 1 --2- 172.21.15.28:0/1818793830 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f70340a2ca0 0x7f70340a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:07:50.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:50.907+0000 7f704cc00700 1 -- 172.21.15.28:0/1818793830 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f70400d07f0 con 0x7f7050069320 2024-05-12T22:07:50.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:50.908+0000 7f704f400700 1 --2- 172.21.15.28:0/1818793830 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f70340a2ca0 0x7f70340a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:07:50.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:50.908+0000 7f704f400700 1 --2- 172.21.15.28:0/1818793830 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f70340a2ca0 0x7f70340a5160 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f7050100620 tx=0x7f703c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:07:50.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:50.910+0000 7f704cc00700 1 -- 172.21.15.28:0/1818793830 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f704002d020 con 0x7f7050069320 2024-05-12T22:07:51.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:51.137+0000 7f7055c00700 1 -- 172.21.15.28:0/1818793830 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) -- 0x7f70501a5670 con 0x7f7050069320 2024-05-12T22:07:51.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:51.138+0000 7f704cc00700 1 -- 172.21.15.28:0/1818793830 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f704009d790 con 0x7f7050069320 2024-05-12T22:07:51.139 INFO:teuthology.orchestra.run.smithi028.stdout:257698037805 2024-05-12T22:07:51.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:51.141+0000 7f7055c00700 1 -- 172.21.15.28:0/1818793830 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f70340a2ca0 msgr2=0x7f70340a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:51.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:51.141+0000 7f7055c00700 1 --2- 172.21.15.28:0/1818793830 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f70340a2ca0 0x7f70340a5160 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f7050100620 tx=0x7f703c009260 comp rx=0 tx=0).stop 2024-05-12T22:07:51.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:51.141+0000 7f7055c00700 1 -- 172.21.15.28:0/1818793830 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7050069320 msgr2=0x7f705019e640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:51.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:51.141+0000 7f7055c00700 1 --2- 172.21.15.28:0/1818793830 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7050069320 0x7f705019e640 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f7040005910 tx=0x7f70400040a0 comp rx=0 tx=0).stop 2024-05-12T22:07:51.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:51.141+0000 7f7055c00700 1 -- 172.21.15.28:0/1818793830 shutdown_connections 2024-05-12T22:07:51.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:51.141+0000 7f7055c00700 1 --2- 172.21.15.28:0/1818793830 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f70340a2ca0 0x7f70340a5160 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:51.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:51.141+0000 7f7055c00700 1 --2- 172.21.15.28:0/1818793830 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7050069320 0x7f705019e640 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:51.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:51.141+0000 7f7055c00700 1 --2- 172.21.15.28:0/1818793830 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f705010b3c0 0x7f70501987c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:51.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:51.141+0000 7f7055c00700 1 --2- 172.21.15.28:0/1818793830 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7050101240 0x7f705019eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:51.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:51.141+0000 7f7055c00700 1 -- 172.21.15.28:0/1818793830 >> 172.21.15.28:0/1818793830 conn(0x7f7050075fc0 msgr2=0x7f705010c150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:07:51.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:51.141+0000 7f7055c00700 1 -- 172.21.15.28:0/1818793830 shutdown_connections 2024-05-12T22:07:51.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:51.141+0000 7f7055c00700 1 -- 172.21.15.28:0/1818793830 wait complete. 2024-05-12T22:07:51.243 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411400 got 77309411405 for osd.2 2024-05-12T22:07:51.243 DEBUG:teuthology.parallel:result is None 2024-05-12T22:07:51.727 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:07:51 smithi028 ceph-mon[29847]: pgmap v350: 97 pgs: 97 active+clean; 641 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 597 B/s wr, 7 op/s 2024-05-12T22:07:51.727 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:07:51 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/1818793830' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-05-12T22:07:51.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:07:51 smithi086 ceph-mon[34297]: pgmap v350: 97 pgs: 97 active+clean; 641 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 597 B/s wr, 7 op/s 2024-05-12T22:07:51.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:07:51 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/1818793830' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-05-12T22:07:51.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:07:51 smithi186 ceph-mon[34785]: pgmap v350: 97 pgs: 97 active+clean; 641 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 597 B/s wr, 7 op/s 2024-05-12T22:07:51.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:07:51 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/1818793830' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-05-12T22:07:51.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:51.947+0000 7fb7f2600700 1 -- 172.21.15.28:0/3221059283 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb7ec107690 msgr2=0x7fb7ec10f260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:51.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:51.947+0000 7fb7f2600700 1 --2- 172.21.15.28:0/3221059283 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb7ec107690 0x7fb7ec10f260 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fb7e4009f90 tx=0x7fb7e400b330 comp rx=0 tx=0).stop 2024-05-12T22:07:51.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:51.947+0000 7fb7f2600700 1 -- 172.21.15.28:0/3221059283 shutdown_connections 2024-05-12T22:07:51.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:51.947+0000 7fb7f2600700 1 --2- 172.21.15.28:0/3221059283 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb7ec107690 0x7fb7ec10f260 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:51.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:51.947+0000 7fb7f2600700 1 --2- 172.21.15.28:0/3221059283 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7ec1025d0 0x7fb7ec1029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:51.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:51.947+0000 7fb7f2600700 1 --2- 172.21.15.28:0/3221059283 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb7ec102f80 0x7fb7ec106fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:51.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:51.948+0000 7fb7f2600700 1 -- 172.21.15.28:0/3221059283 >> 172.21.15.28:0/3221059283 conn(0x7fb7ec0794d0 msgr2=0x7fb7ec0798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:07:51.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:51.948+0000 7fb7f2600700 1 -- 172.21.15.28:0/3221059283 shutdown_connections 2024-05-12T22:07:51.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:51.948+0000 7fb7f2600700 1 -- 172.21.15.28:0/3221059283 wait complete. 2024-05-12T22:07:51.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:51.949+0000 7fb7f2600700 1 Processor -- start 2024-05-12T22:07:51.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:51.950+0000 7fb7f2600700 1 -- start start 2024-05-12T22:07:51.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:51.950+0000 7fb7f2600700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb7ec1025d0 0x7fb7ec19ad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:07:51.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:51.950+0000 7fb7f2600700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7ec102f80 0x7fb7ec19b270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:07:51.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:51.950+0000 7fb7f2600700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb7ec107690 0x7fb7ec19f600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:07:51.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:51.950+0000 7fb7f2600700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb7ec1126c0 con 0x7fb7ec102f80 2024-05-12T22:07:51.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:51.950+0000 7fb7f2600700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fb7ec112540 con 0x7fb7ec107690 2024-05-12T22:07:51.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:51.950+0000 7fb7f2600700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb7ec112840 con 0x7fb7ec1025d0 2024-05-12T22:07:51.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:51.950+0000 7fb7e3e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7ec102f80 0x7fb7ec19b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:07:51.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:51.950+0000 7fb7ebe00700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb7ec107690 0x7fb7ec19f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:07:51.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:51.951+0000 7fb7eb400700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb7ec1025d0 0x7fb7ec19ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:07:51.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:51.951+0000 7fb7e3e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7ec102f80 0x7fb7ec19b270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60434/0 (socket says 172.21.15.28:60434) 2024-05-12T22:07:51.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:51.951+0000 7fb7ebe00700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb7ec107690 0x7fb7ec19f600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:57260/0 (socket says 172.21.15.28:57260) 2024-05-12T22:07:51.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:51.951+0000 7fb7e3e00700 1 -- 172.21.15.28:0/13825255 learned_addr learned my addr 172.21.15.28:0/13825255 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:07:51.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:51.951+0000 7fb7eb400700 1 -- 172.21.15.28:0/13825255 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb7ec107690 msgr2=0x7fb7ec19f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:51.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:51.951+0000 7fb7eb400700 1 --2- 172.21.15.28:0/13825255 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb7ec107690 0x7fb7ec19f600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:51.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:51.951+0000 7fb7eb400700 1 -- 172.21.15.28:0/13825255 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7ec102f80 msgr2=0x7fb7ec19b270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:51.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:51.951+0000 7fb7eb400700 1 --2- 172.21.15.28:0/13825255 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7ec102f80 0x7fb7ec19b270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:51.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:51.951+0000 7fb7eb400700 1 -- 172.21.15.28:0/13825255 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7ec19fd40 con 0x7fb7ec1025d0 2024-05-12T22:07:51.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:51.951+0000 7fb7eb400700 1 --2- 172.21.15.28:0/13825255 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb7ec1025d0 0x7fb7ec19ad30 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fb7dc00eb90 tx=0x7fb7dc00eea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:07:51.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:51.952+0000 7fb7e8c00700 1 -- 172.21.15.28:0/13825255 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7dc00cc90 con 0x7fb7ec1025d0 2024-05-12T22:07:51.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:51.952+0000 7fb7e8c00700 1 -- 172.21.15.28:0/13825255 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb7dc00ce30 con 0x7fb7ec1025d0 2024-05-12T22:07:51.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:51.952+0000 7fb7f2600700 1 -- 172.21.15.28:0/13825255 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7ec1a0030 con 0x7fb7ec1025d0 2024-05-12T22:07:51.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:51.952+0000 7fb7e8c00700 1 -- 172.21.15.28:0/13825255 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7dc010430 con 0x7fb7ec1025d0 2024-05-12T22:07:51.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:51.952+0000 7fb7f2600700 1 -- 172.21.15.28:0/13825255 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7ec1a78b0 con 0x7fb7ec1025d0 2024-05-12T22:07:51.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:51.953+0000 7fb7f2600700 1 -- 172.21.15.28:0/13825255 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb7ec069010 con 0x7fb7ec1025d0 2024-05-12T22:07:51.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:51.955+0000 7fb7e8c00700 1 -- 172.21.15.28:0/13825255 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fb7dc043ae0 con 0x7fb7ec1025d0 2024-05-12T22:07:51.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:51.956+0000 7fb7e8c00700 1 --2- 172.21.15.28:0/13825255 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fb7d00a2c20 0x7fb7d00a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:07:51.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:51.957+0000 7fb7e8c00700 1 -- 172.21.15.28:0/13825255 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7fb7dc0cf3f0 con 0x7fb7ec1025d0 2024-05-12T22:07:51.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:51.957+0000 7fb7e3e00700 1 --2- 172.21.15.28:0/13825255 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fb7d00a2c20 0x7fb7d00a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:07:51.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:51.958+0000 7fb7e3e00700 1 --2- 172.21.15.28:0/13825255 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fb7d00a2c20 0x7fb7d00a50e0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fb7ec108ca0 tx=0x7fb7d4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:07:51.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:51.959+0000 7fb7e8c00700 1 -- 172.21.15.28:0/13825255 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fb7dc09c300 con 0x7fb7ec1025d0 2024-05-12T22:07:52.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:52.182+0000 7f0294e00700 1 -- 172.21.15.28:0/448786455 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0290103d30 msgr2=0x7f029010c220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:52.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:52.182+0000 7f0294e00700 1 --2- 172.21.15.28:0/448786455 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0290103d30 0x7f029010c220 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f027c009990 tx=0x7f027c009ca0 comp rx=0 tx=0).stop 2024-05-12T22:07:52.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:52.182+0000 7f0294e00700 1 -- 172.21.15.28:0/448786455 shutdown_connections 2024-05-12T22:07:52.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:52.182+0000 7f0294e00700 1 --2- 172.21.15.28:0/448786455 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f029010c840 0x7f029010ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:52.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:52.182+0000 7f0294e00700 1 --2- 172.21.15.28:0/448786455 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0290103410 0x7f02901037f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:52.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:52.182+0000 7f0294e00700 1 --2- 172.21.15.28:0/448786455 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0290103d30 0x7f029010c220 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:52.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:52.182+0000 7f0294e00700 1 -- 172.21.15.28:0/448786455 >> 172.21.15.28:0/448786455 conn(0x7f02900fcc40 msgr2=0x7f02900ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:07:52.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:52.182+0000 7f0294e00700 1 -- 172.21.15.28:0/448786455 shutdown_connections 2024-05-12T22:07:52.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:52.182+0000 7f0294e00700 1 -- 172.21.15.28:0/448786455 wait complete. 2024-05-12T22:07:52.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:52.183+0000 7f0294e00700 1 Processor -- start 2024-05-12T22:07:52.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:52.184+0000 7f0294e00700 1 -- start start 2024-05-12T22:07:52.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:52.184+0000 7f0294e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0290103410 0x7f02901a0910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:07:52.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:52.184+0000 7f0294e00700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0290103d30 0x7f02901a0e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:07:52.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:52.184+0000 7f0294e00700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f029010c840 0x7f029019aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:07:52.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:52.184+0000 7f0294e00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f02901125e0 con 0x7f0290103410 2024-05-12T22:07:52.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:52.184+0000 7f0294e00700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f0290112460 con 0x7f029010c840 2024-05-12T22:07:52.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:52.184+0000 7f0294e00700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f0290112760 con 0x7f0290103d30 2024-05-12T22:07:52.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:52.185+0000 7f028ea00700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0290103d30 0x7f02901a0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:07:52.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:52.185+0000 7f028ea00700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0290103d30 0x7f02901a0e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:60392/0 (socket says 172.21.15.28:60392) 2024-05-12T22:07:52.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:52.185+0000 7f028ea00700 1 -- 172.21.15.28:0/748542514 learned_addr learned my addr 172.21.15.28:0/748542514 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:07:52.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:52.185+0000 7f028f400700 1 --2- 172.21.15.28:0/748542514 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0290103410 0x7f02901a0910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:07:52.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:52.185+0000 7f028fe00700 1 --2- 172.21.15.28:0/748542514 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f029010c840 0x7f029019aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:07:52.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:52.185+0000 7f028ea00700 1 -- 172.21.15.28:0/748542514 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f029010c840 msgr2=0x7f029019aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:52.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:52.185+0000 7f028ea00700 1 --2- 172.21.15.28:0/748542514 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f029010c840 0x7f029019aad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:52.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:52.185+0000 7f028ea00700 1 -- 172.21.15.28:0/748542514 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0290103410 msgr2=0x7f02901a0910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:52.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:52.185+0000 7f028ea00700 1 --2- 172.21.15.28:0/748542514 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0290103410 0x7f02901a0910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:52.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:52.185+0000 7f028ea00700 1 -- 172.21.15.28:0/748542514 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f029019b230 con 0x7f0290103d30 2024-05-12T22:07:52.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:52.185+0000 7f028ea00700 1 --2- 172.21.15.28:0/748542514 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0290103d30 0x7f02901a0e50 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f027c00b2e0 tx=0x7f027c004790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:07:52.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:52.186+0000 7f0287e00700 1 -- 172.21.15.28:0/748542514 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f027c004800 con 0x7f0290103d30 2024-05-12T22:07:52.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:52.186+0000 7f0294e00700 1 -- 172.21.15.28:0/748542514 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f029019b4c0 con 0x7f0290103d30 2024-05-12T22:07:52.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:52.186+0000 7f0287e00700 1 -- 172.21.15.28:0/748542514 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f027c005030 con 0x7f0290103d30 2024-05-12T22:07:52.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:52.186+0000 7f0287e00700 1 -- 172.21.15.28:0/748542514 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f027c003a40 con 0x7f0290103d30 2024-05-12T22:07:52.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:52.186+0000 7f0294e00700 1 -- 172.21.15.28:0/748542514 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02901a77e0 con 0x7f0290103d30 2024-05-12T22:07:52.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:52.187+0000 7f0294e00700 1 -- 172.21.15.28:0/748542514 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0290069010 con 0x7f0290103d30 2024-05-12T22:07:52.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:52.189+0000 7f0287e00700 1 -- 172.21.15.28:0/748542514 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f027c009d10 con 0x7f0290103d30 2024-05-12T22:07:52.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:52.190+0000 7f0287e00700 1 --2- 172.21.15.28:0/748542514 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f02740a2c20 0x7f02740a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:07:52.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:52.191+0000 7f0287e00700 1 -- 172.21.15.28:0/748542514 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f027c0d0070 con 0x7f0290103d30 2024-05-12T22:07:52.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:52.191+0000 7f028f400700 1 --2- 172.21.15.28:0/748542514 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f02740a2c20 0x7f02740a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:07:52.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:52.192+0000 7f028f400700 1 --2- 172.21.15.28:0/748542514 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f02740a2c20 0x7f02740a50e0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f0280009c60 tx=0x7f0280009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:07:52.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:52.193+0000 7f0287e00700 1 -- 172.21.15.28:0/748542514 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f027c013070 con 0x7f0290103d30 2024-05-12T22:07:52.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:52.201+0000 7fb7f2600700 1 -- 172.21.15.28:0/13825255 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7fb7ec19c090 con 0x7fb7ec1025d0 2024-05-12T22:07:52.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:52.201+0000 7fb7e8c00700 1 -- 172.21.15.28:0/13825255 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fb7dc09bc00 con 0x7fb7ec1025d0 2024-05-12T22:07:52.202 INFO:teuthology.orchestra.run.smithi028.stdout:146028888130 2024-05-12T22:07:52.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:52.204+0000 7fb7f2600700 1 -- 172.21.15.28:0/13825255 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fb7d00a2c20 msgr2=0x7fb7d00a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:52.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:52.204+0000 7fb7f2600700 1 --2- 172.21.15.28:0/13825255 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fb7d00a2c20 0x7fb7d00a50e0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fb7ec108ca0 tx=0x7fb7d4009260 comp rx=0 tx=0).stop 2024-05-12T22:07:52.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:52.204+0000 7fb7f2600700 1 -- 172.21.15.28:0/13825255 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb7ec1025d0 msgr2=0x7fb7ec19ad30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:52.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:52.205+0000 7fb7f2600700 1 --2- 172.21.15.28:0/13825255 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb7ec1025d0 0x7fb7ec19ad30 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fb7dc00eb90 tx=0x7fb7dc00eea0 comp rx=0 tx=0).stop 2024-05-12T22:07:52.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:52.205+0000 7fb7f2600700 1 -- 172.21.15.28:0/13825255 shutdown_connections 2024-05-12T22:07:52.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:52.205+0000 7fb7f2600700 1 --2- 172.21.15.28:0/13825255 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fb7d00a2c20 0x7fb7d00a50e0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:52.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:52.205+0000 7fb7f2600700 1 --2- 172.21.15.28:0/13825255 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb7ec1025d0 0x7fb7ec19ad30 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:52.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:52.205+0000 7fb7f2600700 1 --2- 172.21.15.28:0/13825255 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb7ec107690 0x7fb7ec19f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:52.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:52.205+0000 7fb7f2600700 1 --2- 172.21.15.28:0/13825255 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7ec102f80 0x7fb7ec19b270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:52.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:52.205+0000 7fb7f2600700 1 -- 172.21.15.28:0/13825255 >> 172.21.15.28:0/13825255 conn(0x7fb7ec0794d0 msgr2=0x7fb7ec100580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:07:52.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:52.205+0000 7fb7f2600700 1 -- 172.21.15.28:0/13825255 shutdown_connections 2024-05-12T22:07:52.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:52.205+0000 7fb7f2600700 1 -- 172.21.15.28:0/13825255 wait complete. 2024-05-12T22:07:52.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:52.433+0000 7f0294e00700 1 -- 172.21.15.28:0/748542514 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f02901a79b0 con 0x7f0290103d30 2024-05-12T22:07:52.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:52.434+0000 7f0287e00700 1 -- 172.21.15.28:0/748542514 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f027c09d070 con 0x7f0290103d30 2024-05-12T22:07:52.435 INFO:teuthology.orchestra.run.smithi028.stdout:223338299441 2024-05-12T22:07:52.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:52.437+0000 7f0294e00700 1 -- 172.21.15.28:0/748542514 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f02740a2c20 msgr2=0x7f02740a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:52.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:52.437+0000 7f0294e00700 1 --2- 172.21.15.28:0/748542514 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f02740a2c20 0x7f02740a50e0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f0280009c60 tx=0x7f0280009260 comp rx=0 tx=0).stop 2024-05-12T22:07:52.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:52.437+0000 7f0294e00700 1 -- 172.21.15.28:0/748542514 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0290103d30 msgr2=0x7f02901a0e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:52.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:52.437+0000 7f0294e00700 1 --2- 172.21.15.28:0/748542514 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0290103d30 0x7f02901a0e50 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f027c00b2e0 tx=0x7f027c004790 comp rx=0 tx=0).stop 2024-05-12T22:07:52.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:52.437+0000 7f0294e00700 1 -- 172.21.15.28:0/748542514 shutdown_connections 2024-05-12T22:07:52.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:52.437+0000 7f0294e00700 1 --2- 172.21.15.28:0/748542514 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f02740a2c20 0x7f02740a50e0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:52.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:52.437+0000 7f0294e00700 1 --2- 172.21.15.28:0/748542514 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f029010c840 0x7f029019aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:52.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:52.437+0000 7f0294e00700 1 --2- 172.21.15.28:0/748542514 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0290103410 0x7f02901a0910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:52.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:52.437+0000 7f0294e00700 1 --2- 172.21.15.28:0/748542514 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0290103d30 0x7f02901a0e50 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:52.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:52.437+0000 7f0294e00700 1 -- 172.21.15.28:0/748542514 >> 172.21.15.28:0/748542514 conn(0x7f02900fcc40 msgr2=0x7f02900ff030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:07:52.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:52.438+0000 7f0294e00700 1 -- 172.21.15.28:0/748542514 shutdown_connections 2024-05-12T22:07:52.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:52.438+0000 7f0294e00700 1 -- 172.21.15.28:0/748542514 wait complete. 2024-05-12T22:07:52.625 INFO:tasks.cephadm.ceph_manager.ceph:need seq 167503724600 got 167503724604 for osd.5 2024-05-12T22:07:52.625 DEBUG:teuthology.parallel:result is None 2024-05-12T22:07:52.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:07:52 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/13825255' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-05-12T22:07:52.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:07:52 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/748542514' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-05-12T22:07:52.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:07:52 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/13825255' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-05-12T22:07:52.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:07:52 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/748542514' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-05-12T22:07:52.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:07:52 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/13825255' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-05-12T22:07:52.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:07:52 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/748542514' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-05-12T22:07:53.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:53.224+0000 7f8404e00700 1 -- 172.21.15.28:0/1924229615 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8400075030 msgr2=0x7f840010e570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:53.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:53.224+0000 7f8404e00700 1 --2- 172.21.15.28:0/1924229615 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8400075030 0x7f840010e570 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f83ec009990 tx=0x7f83ec009ca0 comp rx=0 tx=0).stop 2024-05-12T22:07:53.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:53.225+0000 7f8404e00700 1 -- 172.21.15.28:0/1924229615 shutdown_connections 2024-05-12T22:07:53.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:53.225+0000 7f8404e00700 1 --2- 172.21.15.28:0/1924229615 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f840010eab0 0x7f8400110ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:53.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:53.225+0000 7f8404e00700 1 --2- 172.21.15.28:0/1924229615 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8400074710 0x7f8400074af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:53.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:53.225+0000 7f8404e00700 1 --2- 172.21.15.28:0/1924229615 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8400075030 0x7f840010e570 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:53.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:53.225+0000 7f8404e00700 1 -- 172.21.15.28:0/1924229615 >> 172.21.15.28:0/1924229615 conn(0x7f84000fdd40 msgr2=0x7f8400100160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:07:53.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:53.225+0000 7f8404e00700 1 -- 172.21.15.28:0/1924229615 shutdown_connections 2024-05-12T22:07:53.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:53.225+0000 7f8404e00700 1 -- 172.21.15.28:0/1924229615 wait complete. 2024-05-12T22:07:53.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:53.226+0000 7f8404e00700 1 Processor -- start 2024-05-12T22:07:53.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:53.226+0000 7f8404e00700 1 -- start start 2024-05-12T22:07:53.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:53.227+0000 7f8404e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8400074710 0x7f84001a08f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:07:53.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:53.227+0000 7f8404e00700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8400075030 0x7f84001a0e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:07:53.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:53.227+0000 7f8404e00700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f840010eab0 0x7f840019ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:07:53.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:53.227+0000 7f8404e00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f84001147e0 con 0x7f8400074710 2024-05-12T22:07:53.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:53.227+0000 7f8404e00700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f8400114660 con 0x7f8400075030 2024-05-12T22:07:53.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:53.227+0000 7f8404e00700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f8400114960 con 0x7f840010eab0 2024-05-12T22:07:53.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:53.227+0000 7f83ffe00700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f840010eab0 0x7f840019ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:07:53.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:53.227+0000 7f83ffe00700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f840010eab0 0x7f840019ab00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:60400/0 (socket says 172.21.15.28:60400) 2024-05-12T22:07:53.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:53.227+0000 7f83fea00700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8400075030 0x7f84001a0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:07:53.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:53.227+0000 7f83ff400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8400074710 0x7f84001a08f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:07:53.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:53.227+0000 7f83ffe00700 1 -- 172.21.15.28:0/138936835 learned_addr learned my addr 172.21.15.28:0/138936835 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:07:53.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:53.227+0000 7f83ffe00700 1 -- 172.21.15.28:0/138936835 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8400075030 msgr2=0x7f84001a0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:53.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:53.227+0000 7f83ffe00700 1 --2- 172.21.15.28:0/138936835 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8400075030 0x7f84001a0e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:53.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:53.228+0000 7f83ffe00700 1 -- 172.21.15.28:0/138936835 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8400074710 msgr2=0x7f84001a08f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:53.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:53.228+0000 7f83ffe00700 1 --2- 172.21.15.28:0/138936835 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8400074710 0x7f84001a08f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:53.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:53.228+0000 7f83ffe00700 1 -- 172.21.15.28:0/138936835 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f840019b260 con 0x7f840010eab0 2024-05-12T22:07:53.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:53.228+0000 7f83ffe00700 1 --2- 172.21.15.28:0/138936835 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f840010eab0 0x7f840019ab00 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f83f800ef90 tx=0x7f83f800c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:07:53.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:53.228+0000 7f83f7e00700 1 -- 172.21.15.28:0/138936835 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83f80092e0 con 0x7f840010eab0 2024-05-12T22:07:53.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:53.228+0000 7f83f7e00700 1 -- 172.21.15.28:0/138936835 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f83f8004bb0 con 0x7f840010eab0 2024-05-12T22:07:53.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:53.228+0000 7f8404e00700 1 -- 172.21.15.28:0/138936835 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f840019b4f0 con 0x7f840010eab0 2024-05-12T22:07:53.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:53.228+0000 7f83f7e00700 1 -- 172.21.15.28:0/138936835 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83f8004280 con 0x7f840010eab0 2024-05-12T22:07:53.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:53.229+0000 7f8404e00700 1 -- 172.21.15.28:0/138936835 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84001a77c0 con 0x7f840010eab0 2024-05-12T22:07:53.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:53.230+0000 7f8404e00700 1 -- 172.21.15.28:0/138936835 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8400069010 con 0x7f840010eab0 2024-05-12T22:07:53.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:53.233+0000 7f83f7e00700 1 -- 172.21.15.28:0/138936835 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f83f8044d10 con 0x7f840010eab0 2024-05-12T22:07:53.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:53.236+0000 7f83f7e00700 1 --2- 172.21.15.28:0/138936835 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f83e40a2ca0 0x7f83e40a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:07:53.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:53.236+0000 7f83f7e00700 1 -- 172.21.15.28:0/138936835 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f83f80cef30 con 0x7f840010eab0 2024-05-12T22:07:53.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:53.236+0000 7f83f7e00700 1 -- 172.21.15.28:0/138936835 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f83f8022020 con 0x7f840010eab0 2024-05-12T22:07:53.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:53.237+0000 7f83ff400700 1 --2- 172.21.15.28:0/138936835 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f83e40a2ca0 0x7f83e40a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:07:53.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:53.238+0000 7f83ff400700 1 --2- 172.21.15.28:0/138936835 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f83e40a2ca0 0x7f83e40a5160 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f8400111370 tx=0x7f83f0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:07:53.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:53.476+0000 7f8404e00700 1 -- 172.21.15.28:0/138936835 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f840002db80 con 0x7f840010eab0 2024-05-12T22:07:53.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:53.477+0000 7f83f7e00700 1 -- 172.21.15.28:0/138936835 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f83f8010040 con 0x7f840010eab0 2024-05-12T22:07:53.477 INFO:teuthology.orchestra.run.smithi028.stdout:111669149768 2024-05-12T22:07:53.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:53.481+0000 7f8404e00700 1 -- 172.21.15.28:0/138936835 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f83e40a2ca0 msgr2=0x7f83e40a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:53.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:53.481+0000 7f8404e00700 1 --2- 172.21.15.28:0/138936835 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f83e40a2ca0 0x7f83e40a5160 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f8400111370 tx=0x7f83f0009260 comp rx=0 tx=0).stop 2024-05-12T22:07:53.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:53.481+0000 7f8404e00700 1 -- 172.21.15.28:0/138936835 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f840010eab0 msgr2=0x7f840019ab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:53.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:53.481+0000 7f8404e00700 1 --2- 172.21.15.28:0/138936835 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f840010eab0 0x7f840019ab00 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f83f800ef90 tx=0x7f83f800c3a0 comp rx=0 tx=0).stop 2024-05-12T22:07:53.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:53.481+0000 7f8404e00700 1 -- 172.21.15.28:0/138936835 shutdown_connections 2024-05-12T22:07:53.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:53.481+0000 7f8404e00700 1 --2- 172.21.15.28:0/138936835 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f83e40a2ca0 0x7f83e40a5160 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:53.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:53.481+0000 7f8404e00700 1 --2- 172.21.15.28:0/138936835 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f840010eab0 0x7f840019ab00 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:53.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:53.481+0000 7f8404e00700 1 --2- 172.21.15.28:0/138936835 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8400074710 0x7f84001a08f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:53.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:53.481+0000 7f8404e00700 1 --2- 172.21.15.28:0/138936835 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8400075030 0x7f84001a0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:53.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:53.481+0000 7f8404e00700 1 -- 172.21.15.28:0/138936835 >> 172.21.15.28:0/138936835 conn(0x7f84000fdd40 msgr2=0x7f84000fea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:07:53.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:53.481+0000 7f8404e00700 1 -- 172.21.15.28:0/138936835 shutdown_connections 2024-05-12T22:07:53.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:53.481+0000 7f8404e00700 1 -- 172.21.15.28:0/138936835 wait complete. 2024-05-12T22:07:53.710 INFO:tasks.cephadm.ceph_manager.ceph:need seq 257698037799 got 257698037805 for osd.8 2024-05-12T22:07:53.710 DEBUG:teuthology.parallel:result is None 2024-05-12T22:07:53.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:07:53 smithi086 ceph-mon[34297]: pgmap v351: 97 pgs: 97 active+clean; 641 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 597 B/s wr, 5 op/s 2024-05-12T22:07:53.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:07:53 smithi028 ceph-mon[29847]: pgmap v351: 97 pgs: 97 active+clean; 641 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 597 B/s wr, 5 op/s 2024-05-12T22:07:53.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:07:53 smithi186 ceph-mon[34785]: pgmap v351: 97 pgs: 97 active+clean; 641 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 597 B/s wr, 5 op/s 2024-05-12T22:07:54.573 INFO:tasks.cephadm.ceph_manager.ceph:need seq 223338299435 got 223338299441 for osd.7 2024-05-12T22:07:54.573 DEBUG:teuthology.parallel:result is None 2024-05-12T22:07:54.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:07:54 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/138936835' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-05-12T22:07:54.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:07:54 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/138936835' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-05-12T22:07:54.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:07:54 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/138936835' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-05-12T22:07:55.489 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149762 got 111669149768 for osd.3 2024-05-12T22:07:55.490 DEBUG:teuthology.parallel:result is None 2024-05-12T22:07:55.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:07:55 smithi028 ceph-mon[29847]: pgmap v352: 97 pgs: 97 active+clean; 641 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 597 B/s wr, 6 op/s 2024-05-12T22:07:55.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:07:55 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/3104361094' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-05-12T22:07:55.838 INFO:tasks.cephadm.ceph_manager.ceph:need seq 146028888125 got 146028888130 for osd.4 2024-05-12T22:07:55.838 DEBUG:teuthology.parallel:result is None 2024-05-12T22:07:55.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:55.142+0000 7fb8b8a00700 1 -- 172.21.15.28:0/3305319991 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8b4102f80 msgr2=0x7fb8b4106f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:55.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:55.142+0000 7fb8b8a00700 1 --2- 172.21.15.28:0/3305319991 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8b4102f80 0x7fb8b4106f80 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fb89c0098f0 tx=0x7fb89c009c00 comp rx=0 tx=0).stop 2024-05-12T22:07:55.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:55.142+0000 7fb8b8a00700 1 -- 172.21.15.28:0/3305319991 shutdown_connections 2024-05-12T22:07:55.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:55.142+0000 7fb8b8a00700 1 --2- 172.21.15.28:0/3305319991 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb8b41025d0 0x7fb8b41029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:55.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:55.142+0000 7fb8b8a00700 1 --2- 172.21.15.28:0/3305319991 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb8b4107660 0x7fb8b410f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:55.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:55.142+0000 7fb8b8a00700 1 --2- 172.21.15.28:0/3305319991 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8b4102f80 0x7fb8b4106f80 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:55.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:55.142+0000 7fb8b8a00700 1 -- 172.21.15.28:0/3305319991 >> 172.21.15.28:0/3305319991 conn(0x7fb8b4079480 msgr2=0x7fb8b4079890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:07:55.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:55.142+0000 7fb8b8a00700 1 -- 172.21.15.28:0/3305319991 shutdown_connections 2024-05-12T22:07:55.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:55.142+0000 7fb8b8a00700 1 -- 172.21.15.28:0/3305319991 wait complete. 2024-05-12T22:07:55.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:55.143+0000 7fb8b8a00700 1 Processor -- start 2024-05-12T22:07:55.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:55.143+0000 7fb8b8a00700 1 -- start start 2024-05-12T22:07:55.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:55.144+0000 7fb8b8a00700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb8b41025d0 0x7fb8b419adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:07:55.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:55.144+0000 7fb8b8a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8b4102f80 0x7fb8b419b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:07:55.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:55.144+0000 7fb8b8a00700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb8b4107660 0x7fb8b419f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:07:55.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:55.144+0000 7fb8b8a00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb8b4112650 con 0x7fb8b4102f80 2024-05-12T22:07:55.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:55.144+0000 7fb8b8a00700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fb8b41124d0 con 0x7fb8b41025d0 2024-05-12T22:07:55.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:55.144+0000 7fb8b8a00700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb8b41127d0 con 0x7fb8b4107660 2024-05-12T22:07:55.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:55.144+0000 7fb8abe00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8b4102f80 0x7fb8b419b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:07:55.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:55.144+0000 7fb8abe00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8b4102f80 0x7fb8b419b300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60488/0 (socket says 172.21.15.28:60488) 2024-05-12T22:07:55.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:55.144+0000 7fb8abe00700 1 -- 172.21.15.28:0/3104361094 learned_addr learned my addr 172.21.15.28:0/3104361094 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:07:55.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:55.144+0000 7fb8b1e00700 1 --2- 172.21.15.28:0/3104361094 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb8b41025d0 0x7fb8b419adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:07:55.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:55.144+0000 7fb8b2800700 1 --2- 172.21.15.28:0/3104361094 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb8b4107660 0x7fb8b419f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:07:55.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:55.145+0000 7fb8abe00700 1 -- 172.21.15.28:0/3104361094 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb8b4107660 msgr2=0x7fb8b419f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:55.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:55.145+0000 7fb8abe00700 1 --2- 172.21.15.28:0/3104361094 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb8b4107660 0x7fb8b419f690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:55.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:55.145+0000 7fb8abe00700 1 -- 172.21.15.28:0/3104361094 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb8b41025d0 msgr2=0x7fb8b419adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:55.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:55.145+0000 7fb8abe00700 1 --2- 172.21.15.28:0/3104361094 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb8b41025d0 0x7fb8b419adc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:55.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:55.145+0000 7fb8abe00700 1 -- 172.21.15.28:0/3104361094 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8b419fdd0 con 0x7fb8b4102f80 2024-05-12T22:07:55.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:55.145+0000 7fb8abe00700 1 --2- 172.21.15.28:0/3104361094 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8b4102f80 0x7fb8b419b300 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fb89c0094f0 tx=0x7fb89c00fa00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:07:55.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:55.145+0000 7fb8aaa00700 1 -- 172.21.15.28:0/3104361094 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb89c027070 con 0x7fb8b4102f80 2024-05-12T22:07:55.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:55.145+0000 7fb8aaa00700 1 -- 172.21.15.28:0/3104361094 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb89c005620 con 0x7fb8b4102f80 2024-05-12T22:07:55.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:55.145+0000 7fb8b8a00700 1 -- 172.21.15.28:0/3104361094 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8b41a0060 con 0x7fb8b4102f80 2024-05-12T22:07:55.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:55.146+0000 7fb8aaa00700 1 -- 172.21.15.28:0/3104361094 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb89c004930 con 0x7fb8b4102f80 2024-05-12T22:07:55.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:55.146+0000 7fb8b8a00700 1 -- 172.21.15.28:0/3104361094 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8b41a7940 con 0x7fb8b4102f80 2024-05-12T22:07:55.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:55.147+0000 7fb8b8a00700 1 -- 172.21.15.28:0/3104361094 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb8b4103a60 con 0x7fb8b4102f80 2024-05-12T22:07:55.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:55.149+0000 7fb8aaa00700 1 -- 172.21.15.28:0/3104361094 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fb89c02a070 con 0x7fb8b4102f80 2024-05-12T22:07:55.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:55.150+0000 7fb8aaa00700 1 --2- 172.21.15.28:0/3104361094 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fb8980a2c20 0x7fb8980a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:07:55.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:55.151+0000 7fb8aaa00700 1 -- 172.21.15.28:0/3104361094 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7fb89c0cfdc0 con 0x7fb8b4102f80 2024-05-12T22:07:55.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:55.151+0000 7fb8b1e00700 1 --2- 172.21.15.28:0/3104361094 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fb8980a2c20 0x7fb8980a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:07:55.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:55.151+0000 7fb8b1e00700 1 --2- 172.21.15.28:0/3104361094 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fb8980a2c20 0x7fb8980a50e0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fb8a4005fd0 tx=0x7fb8a4009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:07:55.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:55.153+0000 7fb8aaa00700 1 -- 172.21.15.28:0/3104361094 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fb89c031020 con 0x7fb8b4102f80 2024-05-12T22:07:55.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:55.384+0000 7fb8b8a00700 1 -- 172.21.15.28:0/3104361094 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7fb8b41a7b10 con 0x7fb8b4102f80 2024-05-12T22:07:55.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:55.385+0000 7fb8aaa00700 1 -- 172.21.15.28:0/3104361094 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fb89c013030 con 0x7fb8b4102f80 2024-05-12T22:07:55.892 INFO:teuthology.orchestra.run.smithi028.stdout:55834574932 2024-05-12T22:07:55.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:55.388+0000 7fb8b8a00700 1 -- 172.21.15.28:0/3104361094 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fb8980a2c20 msgr2=0x7fb8980a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:55.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:55.388+0000 7fb8b8a00700 1 --2- 172.21.15.28:0/3104361094 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fb8980a2c20 0x7fb8980a50e0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fb8a4005fd0 tx=0x7fb8a4009310 comp rx=0 tx=0).stop 2024-05-12T22:07:55.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:55.388+0000 7fb8b8a00700 1 -- 172.21.15.28:0/3104361094 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8b4102f80 msgr2=0x7fb8b419b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:55.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:55.388+0000 7fb8b8a00700 1 --2- 172.21.15.28:0/3104361094 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8b4102f80 0x7fb8b419b300 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fb89c0094f0 tx=0x7fb89c00fa00 comp rx=0 tx=0).stop 2024-05-12T22:07:55.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:55.388+0000 7fb8b8a00700 1 -- 172.21.15.28:0/3104361094 shutdown_connections 2024-05-12T22:07:55.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:55.388+0000 7fb8b8a00700 1 --2- 172.21.15.28:0/3104361094 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fb8980a2c20 0x7fb8980a50e0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:55.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:55.388+0000 7fb8b8a00700 1 --2- 172.21.15.28:0/3104361094 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb8b4107660 0x7fb8b419f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:55.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:55.388+0000 7fb8b8a00700 1 --2- 172.21.15.28:0/3104361094 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb8b41025d0 0x7fb8b419adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:55.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:55.388+0000 7fb8b8a00700 1 --2- 172.21.15.28:0/3104361094 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8b4102f80 0x7fb8b419b300 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:55.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:55.388+0000 7fb8b8a00700 1 -- 172.21.15.28:0/3104361094 >> 172.21.15.28:0/3104361094 conn(0x7fb8b4079480 msgr2=0x7fb8b4100b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:07:55.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:55.389+0000 7fb8b8a00700 1 -- 172.21.15.28:0/3104361094 shutdown_connections 2024-05-12T22:07:55.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:55.389+0000 7fb8b8a00700 1 -- 172.21.15.28:0/3104361094 wait complete. 2024-05-12T22:07:55.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:07:55 smithi186 ceph-mon[34785]: pgmap v352: 97 pgs: 97 active+clean; 641 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 597 B/s wr, 6 op/s 2024-05-12T22:07:55.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:07:55 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/3104361094' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-05-12T22:07:55.994 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:07:55 smithi086 ceph-mon[34297]: pgmap v352: 97 pgs: 97 active+clean; 641 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 597 B/s wr, 6 op/s 2024-05-12T22:07:55.994 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:07:55 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/3104361094' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-05-12T22:07:56.745 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:07:56 smithi028 ceph-mon[29847]: pgmap v353: 97 pgs: 97 active+clean; 641 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 597 B/s wr, 7 op/s 2024-05-12T22:07:56.803 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574924 got 55834574932 for osd.1 2024-05-12T22:07:56.803 DEBUG:teuthology.parallel:result is None 2024-05-12T22:07:56.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:56.585+0000 7f1d52e00700 1 -- 172.21.15.28:0/1460987164 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d4c10e950 msgr2=0x7f1d4c10ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:56.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:56.585+0000 7f1d52e00700 1 --2- 172.21.15.28:0/1460987164 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d4c10e950 0x7f1d4c10ed30 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f1d3c009990 tx=0x7f1d3c009ca0 comp rx=0 tx=0).stop 2024-05-12T22:07:56.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:56.586+0000 7f1d52e00700 1 -- 172.21.15.28:0/1460987164 shutdown_connections 2024-05-12T22:07:56.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:56.586+0000 7f1d52e00700 1 --2- 172.21.15.28:0/1460987164 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1d4c109310 0x7f1d4c115820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:56.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:56.586+0000 7f1d52e00700 1 --2- 172.21.15.28:0/1460987164 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d4c10e950 0x7f1d4c10ed30 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:56.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:56.586+0000 7f1d52e00700 1 --2- 172.21.15.28:0/1460987164 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1d4c108950 0x7f1d4c108dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:56.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:56.586+0000 7f1d52e00700 1 -- 172.21.15.28:0/1460987164 >> 172.21.15.28:0/1460987164 conn(0x7f1d4c075af0 msgr2=0x7f1d4c107140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:07:56.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:56.586+0000 7f1d52e00700 1 -- 172.21.15.28:0/1460987164 shutdown_connections 2024-05-12T22:07:56.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:56.586+0000 7f1d52e00700 1 -- 172.21.15.28:0/1460987164 wait complete. 2024-05-12T22:07:56.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:56.587+0000 7f1d52e00700 1 Processor -- start 2024-05-12T22:07:56.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:56.587+0000 7f1d52e00700 1 -- start start 2024-05-12T22:07:56.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:56.587+0000 7f1d52e00700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1d4c108950 0x7f1d4c19f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:07:56.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:56.587+0000 7f1d52e00700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1d4c109310 0x7f1d4c19f630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:07:56.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:56.587+0000 7f1d52e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d4c10e950 0x7f1d4c1a39c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:07:56.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:56.587+0000 7f1d52e00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1d4c118d00 con 0x7f1d4c10e950 2024-05-12T22:07:56.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:56.587+0000 7f1d52e00700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f1d4c118b80 con 0x7f1d4c109310 2024-05-12T22:07:56.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:56.587+0000 7f1d52e00700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f1d4c118e80 con 0x7f1d4c108950 2024-05-12T22:07:56.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:56.588+0000 7f1d50e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d4c10e950 0x7f1d4c1a39c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:07:56.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:56.588+0000 7f1d4b400700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1d4c109310 0x7f1d4c19f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:07:56.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:56.588+0000 7f1d50e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d4c10e950 0x7f1d4c1a39c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48928/0 (socket says 172.21.15.28:48928) 2024-05-12T22:07:56.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:56.588+0000 7f1d50e00700 1 -- 172.21.15.28:0/3195005986 learned_addr learned my addr 172.21.15.28:0/3195005986 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:07:56.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:56.588+0000 7f1d4be00700 1 --2- 172.21.15.28:0/3195005986 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1d4c108950 0x7f1d4c19f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:07:56.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:56.588+0000 7f1d4b400700 1 -- 172.21.15.28:0/3195005986 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1d4c108950 msgr2=0x7f1d4c19f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:56.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:56.588+0000 7f1d4b400700 1 --2- 172.21.15.28:0/3195005986 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1d4c108950 0x7f1d4c19f0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:56.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:56.588+0000 7f1d4b400700 1 -- 172.21.15.28:0/3195005986 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d4c10e950 msgr2=0x7f1d4c1a39c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:56.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:56.588+0000 7f1d4b400700 1 --2- 172.21.15.28:0/3195005986 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d4c10e950 0x7f1d4c1a39c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:56.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:56.588+0000 7f1d4b400700 1 -- 172.21.15.28:0/3195005986 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d4c1a4100 con 0x7f1d4c109310 2024-05-12T22:07:56.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:56.588+0000 7f1d4b400700 1 --2- 172.21.15.28:0/3195005986 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1d4c109310 0x7f1d4c19f630 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f1d3800e9f0 tx=0x7f1d3800ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:07:56.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:56.589+0000 7f1d48c00700 1 -- 172.21.15.28:0/3195005986 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d38004220 con 0x7f1d4c109310 2024-05-12T22:07:56.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:56.589+0000 7f1d52e00700 1 -- 172.21.15.28:0/3195005986 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d4c1a43f0 con 0x7f1d4c109310 2024-05-12T22:07:56.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:56.589+0000 7f1d48c00700 1 -- 172.21.15.28:0/3195005986 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1d38004d10 con 0x7f1d4c109310 2024-05-12T22:07:56.856 INFO:teuthology.orchestra.run.smithi028.stdout:201863462969 2024-05-12T22:07:56.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:56.589+0000 7f1d48c00700 1 -- 172.21.15.28:0/3195005986 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d38005020 con 0x7f1d4c109310 2024-05-12T22:07:56.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:56.589+0000 7f1d52e00700 1 -- 172.21.15.28:0/3195005986 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d4c1abc70 con 0x7f1d4c109310 2024-05-12T22:07:56.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:56.590+0000 7f1d52e00700 1 -- 172.21.15.28:0/3195005986 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1d4c069010 con 0x7f1d4c109310 2024-05-12T22:07:56.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:56.592+0000 7f1d48c00700 1 -- 172.21.15.28:0/3195005986 <== mon.2 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f1d38004750 con 0x7f1d4c109310 2024-05-12T22:07:56.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:56.594+0000 7f1d48c00700 1 --2- 172.21.15.28:0/3195005986 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f1d300a2c20 0x7f1d300a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:07:56.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:56.594+0000 7f1d48c00700 1 -- 172.21.15.28:0/3195005986 <== mon.2 v2:172.21.15.86:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f1d380cef40 con 0x7f1d4c109310 2024-05-12T22:07:56.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:56.594+0000 7f1d4be00700 1 --2- 172.21.15.28:0/3195005986 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f1d300a2c20 0x7f1d300a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:07:56.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:56.595+0000 7f1d4be00700 1 --2- 172.21.15.28:0/3195005986 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f1d300a2c20 0x7f1d300a50e0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f1d3c00b2e0 tx=0x7f1d3c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:07:56.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:56.598+0000 7f1d48c00700 1 -- 172.21.15.28:0/3195005986 <== mon.2 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f1d3809be50 con 0x7f1d4c109310 2024-05-12T22:07:56.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:56.839+0000 7f1d52e00700 1 -- 172.21.15.28:0/3195005986 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f1d4c02db80 con 0x7f1d4c109310 2024-05-12T22:07:56.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:56.840+0000 7f1d48c00700 1 -- 172.21.15.28:0/3195005986 <== mon.2 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f1d3809b750 con 0x7f1d4c109310 2024-05-12T22:07:56.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:56.844+0000 7f1d52e00700 1 -- 172.21.15.28:0/3195005986 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f1d300a2c20 msgr2=0x7f1d300a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:56.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:56.844+0000 7f1d52e00700 1 --2- 172.21.15.28:0/3195005986 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f1d300a2c20 0x7f1d300a50e0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f1d3c00b2e0 tx=0x7f1d3c005fd0 comp rx=0 tx=0).stop 2024-05-12T22:07:56.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:56.844+0000 7f1d52e00700 1 -- 172.21.15.28:0/3195005986 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1d4c109310 msgr2=0x7f1d4c19f630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:07:56.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:56.844+0000 7f1d52e00700 1 --2- 172.21.15.28:0/3195005986 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1d4c109310 0x7f1d4c19f630 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f1d3800e9f0 tx=0x7f1d3800ed00 comp rx=0 tx=0).stop 2024-05-12T22:07:56.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:56.844+0000 7f1d52e00700 1 -- 172.21.15.28:0/3195005986 shutdown_connections 2024-05-12T22:07:56.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:56.844+0000 7f1d52e00700 1 --2- 172.21.15.28:0/3195005986 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f1d300a2c20 0x7f1d300a50e0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:56.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:56.844+0000 7f1d52e00700 1 --2- 172.21.15.28:0/3195005986 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1d4c108950 0x7f1d4c19f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:56.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:56.844+0000 7f1d52e00700 1 --2- 172.21.15.28:0/3195005986 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d4c10e950 0x7f1d4c1a39c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:56.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:56.844+0000 7f1d52e00700 1 --2- 172.21.15.28:0/3195005986 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1d4c109310 0x7f1d4c19f630 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:07:56.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:56.844+0000 7f1d52e00700 1 -- 172.21.15.28:0/3195005986 >> 172.21.15.28:0/3195005986 conn(0x7f1d4c075af0 msgr2=0x7f1d4c076030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:07:56.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:56.844+0000 7f1d52e00700 1 -- 172.21.15.28:0/3195005986 shutdown_connections 2024-05-12T22:07:56.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:07:56.844+0000 7f1d52e00700 1 -- 172.21.15.28:0/3195005986 wait complete. 2024-05-12T22:07:56.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:07:56 smithi186 ceph-mon[34785]: pgmap v353: 97 pgs: 97 active+clean; 641 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 597 B/s wr, 7 op/s 2024-05-12T22:07:56.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:07:56 smithi086 ceph-mon[34297]: pgmap v353: 97 pgs: 97 active+clean; 641 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 597 B/s wr, 7 op/s 2024-05-12T22:07:57.524 INFO:tasks.cephadm.ceph_manager.ceph:need seq 201863462962 got 201863462969 for osd.6 2024-05-12T22:07:57.524 DEBUG:teuthology.parallel:result is None 2024-05-12T22:07:57.524 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-05-12T22:07:57.525 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f72fecff68e1d400c4568684327c900485c20d6a shell --fsid 6b92e7b4-10aa-11ef-bc98-c7b262605968 -- ceph pg dump --format=json 2024-05-12T22:07:57.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:07:57 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/3195005986' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-05-12T22:07:57.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:07:57 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/3195005986' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-05-12T22:07:57.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:07:57 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/3195005986' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-05-12T22:07:58.398 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/mon.a/config 2024-05-12T22:07:58.881 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:07:58 smithi028 ceph-mon[29847]: pgmap v354: 97 pgs: 97 active+clean; 641 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 597 B/s wr, 6 op/s 2024-05-12T22:07:58.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:07:58 smithi186 ceph-mon[34785]: pgmap v354: 97 pgs: 97 active+clean; 641 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 597 B/s wr, 6 op/s 2024-05-12T22:07:58.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:07:58 smithi086 ceph-mon[34297]: pgmap v354: 97 pgs: 97 active+clean; 641 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 597 B/s wr, 6 op/s 2024-05-12T22:08:00.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:00.561+0000 7ff2a3e00700 1 -- 172.21.15.28:0/2050818826 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff29c10ed20 msgr2=0x7ff29c111110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:00.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:00.561+0000 7ff2a3e00700 1 --2- 172.21.15.28:0/2050818826 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff29c10ed20 0x7ff29c111110 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7ff29400b0a0 tx=0x7ff29400b3b0 comp rx=0 tx=0).stop 2024-05-12T22:08:00.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:00.561+0000 7ff2a3e00700 1 -- 172.21.15.28:0/2050818826 shutdown_connections 2024-05-12T22:08:00.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:00.561+0000 7ff2a3e00700 1 --2- 172.21.15.28:0/2050818826 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff29c074550 0x7ff29c074930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:00.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:00.561+0000 7ff2a3e00700 1 --2- 172.21.15.28:0/2050818826 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff29c10ed20 0x7ff29c111110 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:00.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:00.561+0000 7ff2a3e00700 1 --2- 172.21.15.28:0/2050818826 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff29c074e70 0x7ff29c10e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:00.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:00.561+0000 7ff2a3e00700 1 -- 172.21.15.28:0/2050818826 >> 172.21.15.28:0/2050818826 conn(0x7ff29c0794d0 msgr2=0x7ff29c0798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:08:00.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:00.562+0000 7ff2a3e00700 1 -- 172.21.15.28:0/2050818826 shutdown_connections 2024-05-12T22:08:00.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:00.562+0000 7ff2a3e00700 1 -- 172.21.15.28:0/2050818826 wait complete. 2024-05-12T22:08:00.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:00.563+0000 7ff2a3e00700 1 Processor -- start 2024-05-12T22:08:00.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:00.563+0000 7ff2a3e00700 1 -- start start 2024-05-12T22:08:00.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:00.563+0000 7ff2a3e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff29c074550 0x7ff29c102440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:00.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:00.563+0000 7ff2a3e00700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff29c074e70 0x7ff29c102980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:00.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:00.563+0000 7ff2a3e00700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff29c10ed20 0x7ff29c102ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:00.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:00.563+0000 7ff2a3e00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff29c1148c0 con 0x7ff29c074550 2024-05-12T22:08:00.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:00.563+0000 7ff2a3e00700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7ff29c114740 con 0x7ff29c10ed20 2024-05-12T22:08:00.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:00.563+0000 7ff2a3e00700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff29c114a40 con 0x7ff29c074e70 2024-05-12T22:08:00.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:00.564+0000 7ff2a1400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff29c074550 0x7ff29c102440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:00.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:00.564+0000 7ff2a0a00700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff29c074e70 0x7ff29c102980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:00.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:00.564+0000 7ff2a1400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff29c074550 0x7ff29c102440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48944/0 (socket says 172.21.15.28:48944) 2024-05-12T22:08:00.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:00.564+0000 7ff2a0a00700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff29c074e70 0x7ff29c102980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:33074/0 (socket says 172.21.15.28:33074) 2024-05-12T22:08:00.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:00.564+0000 7ff2a1400700 1 -- 172.21.15.28:0/3208625187 learned_addr learned my addr 172.21.15.28:0/3208625187 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:08:00.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:00.564+0000 7ff2a1e00700 1 --2- 172.21.15.28:0/3208625187 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff29c10ed20 0x7ff29c102ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:00.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:00.564+0000 7ff2a0a00700 1 -- 172.21.15.28:0/3208625187 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff29c10ed20 msgr2=0x7ff29c102ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:00.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:00.564+0000 7ff2a0a00700 1 --2- 172.21.15.28:0/3208625187 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff29c10ed20 0x7ff29c102ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:00.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:00.564+0000 7ff2a0a00700 1 -- 172.21.15.28:0/3208625187 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff29c074550 msgr2=0x7ff29c102440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:00.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:00.564+0000 7ff2a0a00700 1 --2- 172.21.15.28:0/3208625187 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff29c074550 0x7ff29c102440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:00.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:00.564+0000 7ff2a0a00700 1 -- 172.21.15.28:0/3208625187 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff29c1ab340 con 0x7ff29c074e70 2024-05-12T22:08:00.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:00.564+0000 7ff2a1e00700 1 --2- 172.21.15.28:0/3208625187 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff29c10ed20 0x7ff29c102ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:08:00.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:00.564+0000 7ff2a1400700 1 --2- 172.21.15.28:0/3208625187 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff29c074550 0x7ff29c102440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:08:00.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:00.565+0000 7ff2a0a00700 1 --2- 172.21.15.28:0/3208625187 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff29c074e70 0x7ff29c102980 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7ff28800c9a0 tx=0x7ff28800ccb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:08:00.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:00.565+0000 7ff29a000700 1 -- 172.21.15.28:0/3208625187 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff288014070 con 0x7ff29c074e70 2024-05-12T22:08:00.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:00.565+0000 7ff2a3e00700 1 -- 172.21.15.28:0/3208625187 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff29c1ab630 con 0x7ff29c074e70 2024-05-12T22:08:00.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:00.565+0000 7ff2a3e00700 1 -- 172.21.15.28:0/3208625187 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff29c1abbc0 con 0x7ff29c074e70 2024-05-12T22:08:00.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:00.565+0000 7ff29a000700 1 -- 172.21.15.28:0/3208625187 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff288004440 con 0x7ff29c074e70 2024-05-12T22:08:00.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:00.565+0000 7ff29a000700 1 -- 172.21.15.28:0/3208625187 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff28800eb70 con 0x7ff29c074e70 2024-05-12T22:08:00.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:00.566+0000 7ff2a3e00700 1 -- 172.21.15.28:0/3208625187 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff27c005330 con 0x7ff29c074e70 2024-05-12T22:08:00.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:00.569+0000 7ff29a000700 1 -- 172.21.15.28:0/3208625187 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7ff288017070 con 0x7ff29c074e70 2024-05-12T22:08:00.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:00.572+0000 7ff29a000700 1 --2- 172.21.15.28:0/3208625187 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7ff2840a2ca0 0x7ff2840a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:00.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:00.572+0000 7ff29a000700 1 -- 172.21.15.28:0/3208625187 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7ff2880cee10 con 0x7ff29c074e70 2024-05-12T22:08:00.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:00.572+0000 7ff2a1400700 1 --2- 172.21.15.28:0/3208625187 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7ff2840a2ca0 0x7ff2840a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:00.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:00.573+0000 7ff29a000700 1 -- 172.21.15.28:0/3208625187 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7ff28809bda0 con 0x7ff29c074e70 2024-05-12T22:08:00.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:00.573+0000 7ff2a1400700 1 --2- 172.21.15.28:0/3208625187 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7ff2840a2ca0 0x7ff2840a5160 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7ff2900105c0 tx=0x7ff29000b480 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:08:00.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:00.810+0000 7ff2a3e00700 1 -- 172.21.15.28:0/3208625187 --> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff27c000bc0 con 0x7ff2840a2ca0 2024-05-12T22:08:00.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:00.820+0000 7ff29a000700 1 -- 172.21.15.28:0/3208625187 <== mgr.14152 v2:172.21.15.28:6800/3123328651 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258833 (secure 0 0 0) 0x7ff27c000bc0 con 0x7ff2840a2ca0 2024-05-12T22:08:00.820 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:00 smithi028 ceph-mon[29847]: pgmap v355: 97 pgs: 97 active+clean; 641 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 426 B/s wr, 7 op/s 2024-05-12T22:08:00.824 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-12T22:08:00.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:00.827+0000 7ff2a3e00700 1 -- 172.21.15.28:0/3208625187 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7ff2840a2ca0 msgr2=0x7ff2840a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:00.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:00.827+0000 7ff2a3e00700 1 --2- 172.21.15.28:0/3208625187 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7ff2840a2ca0 0x7ff2840a5160 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7ff2900105c0 tx=0x7ff29000b480 comp rx=0 tx=0).stop 2024-05-12T22:08:00.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:00.827+0000 7ff2a3e00700 1 -- 172.21.15.28:0/3208625187 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff29c074e70 msgr2=0x7ff29c102980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:00.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:00.827+0000 7ff2a3e00700 1 --2- 172.21.15.28:0/3208625187 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff29c074e70 0x7ff29c102980 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7ff28800c9a0 tx=0x7ff28800ccb0 comp rx=0 tx=0).stop 2024-05-12T22:08:00.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:00.827+0000 7ff2a3e00700 1 -- 172.21.15.28:0/3208625187 shutdown_connections 2024-05-12T22:08:00.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:00.827+0000 7ff2a3e00700 1 --2- 172.21.15.28:0/3208625187 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7ff2840a2ca0 0x7ff2840a5160 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:00.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:00.827+0000 7ff2a3e00700 1 --2- 172.21.15.28:0/3208625187 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff29c10ed20 0x7ff29c102ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:00.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:00.827+0000 7ff2a3e00700 1 --2- 172.21.15.28:0/3208625187 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff29c074550 0x7ff29c102440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:00.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:00.827+0000 7ff2a3e00700 1 --2- 172.21.15.28:0/3208625187 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff29c074e70 0x7ff29c102980 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:00.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:00.827+0000 7ff2a3e00700 1 -- 172.21.15.28:0/3208625187 >> 172.21.15.28:0/3208625187 conn(0x7ff29c0794d0 msgr2=0x7ff29c0ff4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:08:00.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:00.827+0000 7ff2a3e00700 1 -- 172.21.15.28:0/3208625187 shutdown_connections 2024-05-12T22:08:00.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:00.827+0000 7ff2a3e00700 1 -- 172.21.15.28:0/3208625187 wait complete. 2024-05-12T22:08:00.828 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-05-12T22:08:01.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:00 smithi186 ceph-mon[34785]: pgmap v355: 97 pgs: 97 active+clean; 641 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 426 B/s wr, 7 op/s 2024-05-12T22:08:01.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:00 smithi086 ceph-mon[34297]: pgmap v355: 97 pgs: 97 active+clean; 641 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 426 B/s wr, 7 op/s 2024-05-12T22:08:01.669 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":355,"stamp":"2024-05-12T22:07:59.747765+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":656398,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":650,"num_read_kb":447,"num_write":355,"num_write_kb":3060,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":1181318,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":787,"ondisk_log_size":787,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":349932,"kb_used_data":13260,"kb_used_omap":18,"kb_used_meta":336557,"kb_avail":1124591892,"statfs":{"total":1151940427776,"available":1151582097408,"internally_reserved":0,"allocated":13578240,"data_stored":7473186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":344634681},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":390,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":82,"num_read_kb":42,"num_write":7,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005191"},"pg_stats":[{"pgid":"3.3f","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.739120+0000","last_change":"2024-05-12T22:06:33.982079+0000","last_active":"2024-05-12T22:07:45.739120+0000","last_peered":"2024-05-12T22:07:45.739120+0000","last_clean":"2024-05-12T22:07:45.739120+0000","last_became_active":"2024-05-12T22:06:33.981703+0000","last_became_peered":"2024-05-12T22:06:33.981703+0000","last_unstale":"2024-05-12T22:07:45.739120+0000","last_undegraded":"2024-05-12T22:07:45.739120+0000","last_fullsized":"2024-05-12T22:07:45.739120+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T08:52:47.157067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"88'4","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.735806+0000","last_change":"2024-05-12T22:06:33.984194+0000","last_active":"2024-05-12T22:07:45.735806+0000","last_peered":"2024-05-12T22:07:45.735806+0000","last_clean":"2024-05-12T22:07:45.735806+0000","last_became_active":"2024-05-12T22:06:33.979787+0000","last_became_peered":"2024-05-12T22:06:33.979787+0000","last_unstale":"2024-05-12T22:07:45.735806+0000","last_undegraded":"2024-05-12T22:07:45.735806+0000","last_fullsized":"2024-05-12T22:07:45.735806+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T07:36:46.588641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.729071+0000","last_change":"2024-05-12T22:06:33.982428+0000","last_active":"2024-05-12T22:07:45.729071+0000","last_peered":"2024-05-12T22:07:45.729071+0000","last_clean":"2024-05-12T22:07:45.729071+0000","last_became_active":"2024-05-12T22:06:33.979563+0000","last_became_peered":"2024-05-12T22:06:33.979563+0000","last_unstale":"2024-05-12T22:07:45.729071+0000","last_undegraded":"2024-05-12T22:07:45.729071+0000","last_fullsized":"2024-05-12T22:07:45.729071+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T06:26:12.586519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"88'8","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.712039+0000","last_change":"2024-05-12T22:06:46.796152+0000","last_active":"2024-05-12T22:07:45.712039+0000","last_peered":"2024-05-12T22:07:45.712039+0000","last_clean":"2024-05-12T22:07:45.712039+0000","last_became_active":"2024-05-12T22:06:46.795484+0000","last_became_peered":"2024-05-12T22:06:46.795484+0000","last_unstale":"2024-05-12T22:07:45.712039+0000","last_undegraded":"2024-05-12T22:07:45.712039+0000","last_fullsized":"2024-05-12T22:07:45.712039+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T03:13:45.901507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,2],"acting":[9,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.3b","version":"88'4","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.711043+0000","last_change":"2024-05-12T22:06:33.967492+0000","last_active":"2024-05-12T22:07:45.711043+0000","last_peered":"2024-05-12T22:07:45.711043+0000","last_clean":"2024-05-12T22:07:45.711043+0000","last_became_active":"2024-05-12T22:06:33.967146+0000","last_became_peered":"2024-05-12T22:06:33.967146+0000","last_unstale":"2024-05-12T22:07:45.711043+0000","last_undegraded":"2024-05-12T22:07:45.711043+0000","last_fullsized":"2024-05-12T22:07:45.711043+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T07:31:05.946473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.712943+0000","last_change":"2024-05-12T22:06:33.973653+0000","last_active":"2024-05-12T22:07:45.712943+0000","last_peered":"2024-05-12T22:07:45.712943+0000","last_clean":"2024-05-12T22:07:45.712943+0000","last_became_active":"2024-05-12T22:06:33.973290+0000","last_became_peered":"2024-05-12T22:06:33.973290+0000","last_unstale":"2024-05-12T22:07:45.712943+0000","last_undegraded":"2024-05-12T22:07:45.712943+0000","last_fullsized":"2024-05-12T22:07:45.712943+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T03:07:21.933141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.701187+0000","last_change":"2024-05-12T22:06:33.973807+0000","last_active":"2024-05-12T22:07:45.701187+0000","last_peered":"2024-05-12T22:07:45.701187+0000","last_clean":"2024-05-12T22:07:45.701187+0000","last_became_active":"2024-05-12T22:06:33.973466+0000","last_became_peered":"2024-05-12T22:06:33.973466+0000","last_unstale":"2024-05-12T22:07:45.701187+0000","last_undegraded":"2024-05-12T22:07:45.701187+0000","last_fullsized":"2024-05-12T22:07:45.701187+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T02:31:13.608309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.717205+0000","last_change":"2024-05-12T22:06:33.967356+0000","last_active":"2024-05-12T22:07:45.717205+0000","last_peered":"2024-05-12T22:07:45.717205+0000","last_clean":"2024-05-12T22:07:45.717205+0000","last_became_active":"2024-05-12T22:06:33.966054+0000","last_became_peered":"2024-05-12T22:06:33.966054+0000","last_unstale":"2024-05-12T22:07:45.717205+0000","last_undegraded":"2024-05-12T22:07:45.717205+0000","last_fullsized":"2024-05-12T22:07:45.717205+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T23:54:08.285720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"88'11","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.721194+0000","last_change":"2024-05-12T22:06:33.968782+0000","last_active":"2024-05-12T22:07:45.721194+0000","last_peered":"2024-05-12T22:07:45.721194+0000","last_clean":"2024-05-12T22:07:45.721194+0000","last_became_active":"2024-05-12T22:06:33.968435+0000","last_became_peered":"2024-05-12T22:06:33.968435+0000","last_unstale":"2024-05-12T22:07:45.721194+0000","last_undegraded":"2024-05-12T22:07:45.721194+0000","last_fullsized":"2024-05-12T22:07:45.721194+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T07:01:21.056777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"88'16","reported_seq":33,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.699709+0000","last_change":"2024-05-12T22:06:33.962018+0000","last_active":"2024-05-12T22:07:45.699709+0000","last_peered":"2024-05-12T22:07:45.699709+0000","last_clean":"2024-05-12T22:07:45.699709+0000","last_became_active":"2024-05-12T22:06:33.961583+0000","last_became_peered":"2024-05-12T22:06:33.961583+0000","last_unstale":"2024-05-12T22:07:45.699709+0000","last_undegraded":"2024-05-12T22:07:45.699709+0000","last_fullsized":"2024-05-12T22:07:45.699709+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T00:16:40.483194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.721027+0000","last_change":"2024-05-12T22:06:33.965776+0000","last_active":"2024-05-12T22:07:45.721027+0000","last_peered":"2024-05-12T22:07:45.721027+0000","last_clean":"2024-05-12T22:07:45.721027+0000","last_became_active":"2024-05-12T22:06:33.965439+0000","last_became_peered":"2024-05-12T22:06:33.965439+0000","last_unstale":"2024-05-12T22:07:45.721027+0000","last_undegraded":"2024-05-12T22:07:45.721027+0000","last_fullsized":"2024-05-12T22:07:45.721027+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T23:59:56.925209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.693930+0000","last_change":"2024-05-12T22:06:33.967906+0000","last_active":"2024-05-12T22:07:45.693930+0000","last_peered":"2024-05-12T22:07:45.693930+0000","last_clean":"2024-05-12T22:07:45.693930+0000","last_became_active":"2024-05-12T22:06:33.967516+0000","last_became_peered":"2024-05-12T22:06:33.967516+0000","last_unstale":"2024-05-12T22:07:45.693930+0000","last_undegraded":"2024-05-12T22:07:45.693930+0000","last_fullsized":"2024-05-12T22:07:45.693930+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T08:16:56.892024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"88'10","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.726321+0000","last_change":"2024-05-12T22:06:33.971456+0000","last_active":"2024-05-12T22:07:45.726321+0000","last_peered":"2024-05-12T22:07:45.726321+0000","last_clean":"2024-05-12T22:07:45.726321+0000","last_became_active":"2024-05-12T22:06:33.971110+0000","last_became_peered":"2024-05-12T22:06:33.971110+0000","last_unstale":"2024-05-12T22:07:45.726321+0000","last_undegraded":"2024-05-12T22:07:45.726321+0000","last_fullsized":"2024-05-12T22:07:45.726321+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T09:48:56.275346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.726703+0000","last_change":"2024-05-12T22:06:33.983421+0000","last_active":"2024-05-12T22:07:45.726703+0000","last_peered":"2024-05-12T22:07:45.726703+0000","last_clean":"2024-05-12T22:07:45.726703+0000","last_became_active":"2024-05-12T22:06:33.978515+0000","last_became_peered":"2024-05-12T22:06:33.978515+0000","last_unstale":"2024-05-12T22:07:45.726703+0000","last_undegraded":"2024-05-12T22:07:45.726703+0000","last_fullsized":"2024-05-12T22:07:45.726703+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T23:09:48.264541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.731547+0000","last_change":"2024-05-12T22:06:33.980956+0000","last_active":"2024-05-12T22:07:45.731547+0000","last_peered":"2024-05-12T22:07:45.731547+0000","last_clean":"2024-05-12T22:07:45.731547+0000","last_became_active":"2024-05-12T22:06:33.979860+0000","last_became_peered":"2024-05-12T22:06:33.979860+0000","last_unstale":"2024-05-12T22:07:45.731547+0000","last_undegraded":"2024-05-12T22:07:45.731547+0000","last_fullsized":"2024-05-12T22:07:45.731547+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T02:58:29.255704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.725416+0000","last_change":"2024-05-12T22:06:33.978682+0000","last_active":"2024-05-12T22:07:45.725416+0000","last_peered":"2024-05-12T22:07:45.725416+0000","last_clean":"2024-05-12T22:07:45.725416+0000","last_became_active":"2024-05-12T22:06:33.978234+0000","last_became_peered":"2024-05-12T22:06:33.978234+0000","last_unstale":"2024-05-12T22:07:45.725416+0000","last_undegraded":"2024-05-12T22:07:45.725416+0000","last_fullsized":"2024-05-12T22:07:45.725416+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T07:40:48.540839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.739297+0000","last_change":"2024-05-12T22:06:33.986598+0000","last_active":"2024-05-12T22:07:45.739297+0000","last_peered":"2024-05-12T22:07:45.739297+0000","last_clean":"2024-05-12T22:07:45.739297+0000","last_became_active":"2024-05-12T22:06:33.986346+0000","last_became_peered":"2024-05-12T22:06:33.986346+0000","last_unstale":"2024-05-12T22:07:45.739297+0000","last_undegraded":"2024-05-12T22:07:45.739297+0000","last_fullsized":"2024-05-12T22:07:45.739297+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T05:52:05.473099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"88'10","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.714677+0000","last_change":"2024-05-12T22:06:33.978900+0000","last_active":"2024-05-12T22:07:45.714677+0000","last_peered":"2024-05-12T22:07:45.714677+0000","last_clean":"2024-05-12T22:07:45.714677+0000","last_became_active":"2024-05-12T22:06:33.978474+0000","last_became_peered":"2024-05-12T22:06:33.978474+0000","last_unstale":"2024-05-12T22:07:45.714677+0000","last_undegraded":"2024-05-12T22:07:45.714677+0000","last_fullsized":"2024-05-12T22:07:45.714677+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T03:47:53.014263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.719258+0000","last_change":"2024-05-12T22:06:33.966752+0000","last_active":"2024-05-12T22:07:45.719258+0000","last_peered":"2024-05-12T22:07:45.719258+0000","last_clean":"2024-05-12T22:07:45.719258+0000","last_became_active":"2024-05-12T22:06:33.966320+0000","last_became_peered":"2024-05-12T22:06:33.966320+0000","last_unstale":"2024-05-12T22:07:45.719258+0000","last_undegraded":"2024-05-12T22:07:45.719258+0000","last_fullsized":"2024-05-12T22:07:45.719258+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T22:11:48.356156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'9","reported_seq":45,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:06:46.808696+0000","last_change":"2024-05-12T22:06:46.808696+0000","last_active":"2024-05-12T22:06:46.808696+0000","last_peered":"2024-05-12T22:06:46.808696+0000","last_clean":"2024-05-12T22:06:46.808696+0000","last_became_active":"2024-05-12T22:06:46.796654+0000","last_became_peered":"2024-05-12T22:06:46.796654+0000","last_unstale":"2024-05-12T22:06:46.808696+0000","last_undegraded":"2024-05-12T22:06:46.808696+0000","last_fullsized":"2024-05-12T22:06:46.808696+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T09:32:33.209844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,8],"acting":[9,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.715776+0000","last_change":"2024-05-12T22:06:33.963875+0000","last_active":"2024-05-12T22:07:45.715776+0000","last_peered":"2024-05-12T22:07:45.715776+0000","last_clean":"2024-05-12T22:07:45.715776+0000","last_became_active":"2024-05-12T22:06:33.963239+0000","last_became_peered":"2024-05-12T22:06:33.963239+0000","last_unstale":"2024-05-12T22:07:45.715776+0000","last_undegraded":"2024-05-12T22:07:45.715776+0000","last_fullsized":"2024-05-12T22:07:45.715776+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T23:15:01.832082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:06:46.788427+0000","last_change":"2024-05-12T22:06:32.950352+0000","last_active":"2024-05-12T22:06:46.788427+0000","last_peered":"2024-05-12T22:06:46.788427+0000","last_clean":"2024-05-12T22:06:46.788427+0000","last_became_active":"2024-05-12T22:06:32.949759+0000","last_became_peered":"2024-05-12T22:06:32.949759+0000","last_unstale":"2024-05-12T22:06:46.788427+0000","last_undegraded":"2024-05-12T22:06:46.788427+0000","last_fullsized":"2024-05-12T22:06:46.788427+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T06:28:07.613351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.737914+0000","last_change":"2024-05-12T22:06:33.990783+0000","last_active":"2024-05-12T22:07:45.737914+0000","last_peered":"2024-05-12T22:07:45.737914+0000","last_clean":"2024-05-12T22:07:45.737914+0000","last_became_active":"2024-05-12T22:06:33.990387+0000","last_became_peered":"2024-05-12T22:06:33.990387+0000","last_unstale":"2024-05-12T22:07:45.737914+0000","last_undegraded":"2024-05-12T22:07:45.737914+0000","last_fullsized":"2024-05-12T22:07:45.737914+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T05:52:52.247458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"86'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:06:46.788832+0000","last_change":"2024-05-12T22:06:32.931718+0000","last_active":"2024-05-12T22:06:46.788832+0000","last_peered":"2024-05-12T22:06:46.788832+0000","last_clean":"2024-05-12T22:06:46.788832+0000","last_became_active":"2024-05-12T22:06:32.931404+0000","last_became_peered":"2024-05-12T22:06:32.931404+0000","last_unstale":"2024-05-12T22:06:46.788832+0000","last_undegraded":"2024-05-12T22:06:46.788832+0000","last_fullsized":"2024-05-12T22:06:46.788832+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T02:55:57.770504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.720351+0000","last_change":"2024-05-12T22:06:33.963299+0000","last_active":"2024-05-12T22:07:45.720351+0000","last_peered":"2024-05-12T22:07:45.720351+0000","last_clean":"2024-05-12T22:07:45.720351+0000","last_became_active":"2024-05-12T22:06:33.962916+0000","last_became_peered":"2024-05-12T22:06:33.962916+0000","last_unstale":"2024-05-12T22:07:45.720351+0000","last_undegraded":"2024-05-12T22:07:45.720351+0000","last_fullsized":"2024-05-12T22:07:45.720351+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T05:46:32.131918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:06:46.801285+0000","last_change":"2024-05-12T22:06:46.800870+0000","last_active":"2024-05-12T22:06:46.801285+0000","last_peered":"2024-05-12T22:06:46.801285+0000","last_clean":"2024-05-12T22:06:46.801285+0000","last_became_active":"2024-05-12T22:06:46.797716+0000","last_became_peered":"2024-05-12T22:06:46.797716+0000","last_unstale":"2024-05-12T22:06:46.801285+0000","last_undegraded":"2024-05-12T22:06:46.801285+0000","last_fullsized":"2024-05-12T22:06:46.801285+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T08:31:57.557564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1d","version":"88'15","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.709001+0000","last_change":"2024-05-12T22:06:33.967926+0000","last_active":"2024-05-12T22:07:45.709001+0000","last_peered":"2024-05-12T22:07:45.709001+0000","last_clean":"2024-05-12T22:07:45.709001+0000","last_became_active":"2024-05-12T22:06:33.967588+0000","last_became_peered":"2024-05-12T22:06:33.967588+0000","last_unstale":"2024-05-12T22:07:45.709001+0000","last_undegraded":"2024-05-12T22:07:45.709001+0000","last_fullsized":"2024-05-12T22:07:45.709001+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T22:39:54.434117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"88'7","reported_seq":109,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:55.813998+0000","last_change":"2024-05-12T22:06:32.930347+0000","last_active":"2024-05-12T22:07:55.813998+0000","last_peered":"2024-05-12T22:07:55.813998+0000","last_clean":"2024-05-12T22:07:55.813998+0000","last_became_active":"2024-05-12T22:06:32.930083+0000","last_became_peered":"2024-05-12T22:06:32.930083+0000","last_unstale":"2024-05-12T22:07:55.813998+0000","last_undegraded":"2024-05-12T22:07:55.813998+0000","last_fullsized":"2024-05-12T22:07:55.813998+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T07:10:08.898490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1154,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":83,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.712891+0000","last_change":"2024-05-12T22:06:33.973793+0000","last_active":"2024-05-12T22:07:45.712891+0000","last_peered":"2024-05-12T22:07:45.712891+0000","last_clean":"2024-05-12T22:07:45.712891+0000","last_became_active":"2024-05-12T22:06:33.973380+0000","last_became_peered":"2024-05-12T22:06:33.973380+0000","last_unstale":"2024-05-12T22:07:45.712891+0000","last_undegraded":"2024-05-12T22:07:45.712891+0000","last_fullsized":"2024-05-12T22:07:45.712891+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T04:32:01.817780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:06:47.067177+0000","last_change":"2024-05-12T22:06:32.938215+0000","last_active":"2024-05-12T22:06:47.067177+0000","last_peered":"2024-05-12T22:06:47.067177+0000","last_clean":"2024-05-12T22:06:47.067177+0000","last_became_active":"2024-05-12T22:06:32.937939+0000","last_became_peered":"2024-05-12T22:06:32.937939+0000","last_unstale":"2024-05-12T22:06:47.067177+0000","last_undegraded":"2024-05-12T22:06:47.067177+0000","last_fullsized":"2024-05-12T22:06:47.067177+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T04:15:28.821835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.718624+0000","last_change":"2024-05-12T22:06:33.977362+0000","last_active":"2024-05-12T22:07:45.718624+0000","last_peered":"2024-05-12T22:07:45.718624+0000","last_clean":"2024-05-12T22:07:45.718624+0000","last_became_active":"2024-05-12T22:06:33.977074+0000","last_became_peered":"2024-05-12T22:06:33.977074+0000","last_unstale":"2024-05-12T22:07:45.718624+0000","last_undegraded":"2024-05-12T22:07:45.718624+0000","last_fullsized":"2024-05-12T22:07:45.718624+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T22:21:00.062780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:06:46.928947+0000","last_change":"2024-05-12T22:06:32.935508+0000","last_active":"2024-05-12T22:06:46.928947+0000","last_peered":"2024-05-12T22:06:46.928947+0000","last_clean":"2024-05-12T22:06:46.928947+0000","last_became_active":"2024-05-12T22:06:32.934760+0000","last_became_peered":"2024-05-12T22:06:32.934760+0000","last_unstale":"2024-05-12T22:06:46.928947+0000","last_undegraded":"2024-05-12T22:06:46.928947+0000","last_fullsized":"2024-05-12T22:06:46.928947+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T01:58:09.450187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.722016+0000","last_change":"2024-05-12T22:06:33.966421+0000","last_active":"2024-05-12T22:07:45.722016+0000","last_peered":"2024-05-12T22:07:45.722016+0000","last_clean":"2024-05-12T22:07:45.722016+0000","last_became_active":"2024-05-12T22:06:33.965627+0000","last_became_peered":"2024-05-12T22:06:33.965627+0000","last_unstale":"2024-05-12T22:07:45.722016+0000","last_undegraded":"2024-05-12T22:07:45.722016+0000","last_fullsized":"2024-05-12T22:07:45.722016+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T09:25:03.836082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"86'1","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:06:46.794429+0000","last_change":"2024-05-12T22:06:32.928907+0000","last_active":"2024-05-12T22:06:46.794429+0000","last_peered":"2024-05-12T22:06:46.794429+0000","last_clean":"2024-05-12T22:06:46.794429+0000","last_became_active":"2024-05-12T22:06:32.928586+0000","last_became_peered":"2024-05-12T22:06:32.928586+0000","last_unstale":"2024-05-12T22:06:46.794429+0000","last_undegraded":"2024-05-12T22:06:46.794429+0000","last_fullsized":"2024-05-12T22:06:46.794429+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T04:24:21.658957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"88'11","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.720453+0000","last_change":"2024-05-12T22:06:33.982873+0000","last_active":"2024-05-12T22:07:45.720453+0000","last_peered":"2024-05-12T22:07:45.720453+0000","last_clean":"2024-05-12T22:07:45.720453+0000","last_became_active":"2024-05-12T22:06:33.982495+0000","last_became_peered":"2024-05-12T22:06:33.982495+0000","last_unstale":"2024-05-12T22:07:45.720453+0000","last_undegraded":"2024-05-12T22:07:45.720453+0000","last_fullsized":"2024-05-12T22:07:45.720453+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T05:45:39.404282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"88'4","reported_seq":104,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:57.816041+0000","last_change":"2024-05-12T22:06:32.937716+0000","last_active":"2024-05-12T22:07:57.816041+0000","last_peered":"2024-05-12T22:07:57.816041+0000","last_clean":"2024-05-12T22:07:57.816041+0000","last_became_active":"2024-05-12T22:06:32.937418+0000","last_became_peered":"2024-05-12T22:06:32.937418+0000","last_unstale":"2024-05-12T22:07:57.816041+0000","last_undegraded":"2024-05-12T22:07:57.816041+0000","last_fullsized":"2024-05-12T22:07:57.816041+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T08:20:13.639958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":81,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"88'10","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.717249+0000","last_change":"2024-05-12T22:06:33.974702+0000","last_active":"2024-05-12T22:07:45.717249+0000","last_peered":"2024-05-12T22:07:45.717249+0000","last_clean":"2024-05-12T22:07:45.717249+0000","last_became_active":"2024-05-12T22:06:33.974363+0000","last_became_peered":"2024-05-12T22:06:33.974363+0000","last_unstale":"2024-05-12T22:07:45.717249+0000","last_undegraded":"2024-05-12T22:07:45.717249+0000","last_fullsized":"2024-05-12T22:07:45.717249+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T09:16:33.620945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"86'2","reported_seq":31,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:06:46.793331+0000","last_change":"2024-05-12T22:06:46.793331+0000","last_active":"2024-05-12T22:06:46.793331+0000","last_peered":"2024-05-12T22:06:46.793331+0000","last_clean":"2024-05-12T22:06:46.793331+0000","last_became_active":"2024-05-12T22:06:46.789984+0000","last_became_peered":"2024-05-12T22:06:46.789984+0000","last_unstale":"2024-05-12T22:06:46.793331+0000","last_undegraded":"2024-05-12T22:06:46.793331+0000","last_fullsized":"2024-05-12T22:06:46.793331+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T08:16:29.505145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.738121+0000","last_change":"2024-05-12T22:06:33.980496+0000","last_active":"2024-05-12T22:07:45.738121+0000","last_peered":"2024-05-12T22:07:45.738121+0000","last_clean":"2024-05-12T22:07:45.738121+0000","last_became_active":"2024-05-12T22:06:33.980155+0000","last_became_peered":"2024-05-12T22:06:33.980155+0000","last_unstale":"2024-05-12T22:07:45.738121+0000","last_undegraded":"2024-05-12T22:07:45.738121+0000","last_fullsized":"2024-05-12T22:07:45.738121+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T01:54:03.831828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"86'4","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:06:46.795752+0000","last_change":"2024-05-12T22:06:32.951633+0000","last_active":"2024-05-12T22:06:46.795752+0000","last_peered":"2024-05-12T22:06:46.795752+0000","last_clean":"2024-05-12T22:06:46.795752+0000","last_became_active":"2024-05-12T22:06:32.951382+0000","last_became_peered":"2024-05-12T22:06:32.951382+0000","last_unstale":"2024-05-12T22:06:46.795752+0000","last_undegraded":"2024-05-12T22:06:46.795752+0000","last_fullsized":"2024-05-12T22:06:46.795752+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T02:06:21.408593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"88'11","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.721698+0000","last_change":"2024-05-12T22:06:33.963493+0000","last_active":"2024-05-12T22:07:45.721698+0000","last_peered":"2024-05-12T22:07:45.721698+0000","last_clean":"2024-05-12T22:07:45.721698+0000","last_became_active":"2024-05-12T22:06:33.963127+0000","last_became_peered":"2024-05-12T22:06:33.963127+0000","last_unstale":"2024-05-12T22:07:45.721698+0000","last_undegraded":"2024-05-12T22:07:45.721698+0000","last_fullsized":"2024-05-12T22:07:45.721698+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T05:39:31.038446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"88'33","reported_seq":224,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:58.816916+0000","last_change":"2024-05-12T22:06:32.932026+0000","last_active":"2024-05-12T22:07:58.816916+0000","last_peered":"2024-05-12T22:07:58.816916+0000","last_clean":"2024-05-12T22:07:58.816916+0000","last_became_active":"2024-05-12T22:06:32.931667+0000","last_became_peered":"2024-05-12T22:06:32.931667+0000","last_unstale":"2024-05-12T22:07:58.816916+0000","last_undegraded":"2024-05-12T22:07:58.816916+0000","last_fullsized":"2024-05-12T22:07:58.816916+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T06:16:28.259047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61122,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":172,"num_read_kb":146,"num_write":34,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.693378+0000","last_change":"2024-05-12T22:06:33.962677+0000","last_active":"2024-05-12T22:07:45.693378+0000","last_peered":"2024-05-12T22:07:45.693378+0000","last_clean":"2024-05-12T22:07:45.693378+0000","last_became_active":"2024-05-12T22:06:33.962295+0000","last_became_peered":"2024-05-12T22:06:33.962295+0000","last_unstale":"2024-05-12T22:07:45.693378+0000","last_undegraded":"2024-05-12T22:07:45.693378+0000","last_fullsized":"2024-05-12T22:07:45.693378+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T01:31:04.587046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"86'6","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:06:46.789562+0000","last_change":"2024-05-12T22:06:32.936104+0000","last_active":"2024-05-12T22:06:46.789562+0000","last_peered":"2024-05-12T22:06:46.789562+0000","last_clean":"2024-05-12T22:06:46.789562+0000","last_became_active":"2024-05-12T22:06:32.935701+0000","last_became_peered":"2024-05-12T22:06:32.935701+0000","last_unstale":"2024-05-12T22:06:46.789562+0000","last_undegraded":"2024-05-12T22:06:46.789562+0000","last_fullsized":"2024-05-12T22:06:46.789562+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T07:04:10.115302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.723098+0000","last_change":"2024-05-12T22:06:33.977850+0000","last_active":"2024-05-12T22:07:45.723098+0000","last_peered":"2024-05-12T22:07:45.723098+0000","last_clean":"2024-05-12T22:07:45.723098+0000","last_became_active":"2024-05-12T22:06:33.977511+0000","last_became_peered":"2024-05-12T22:06:33.977511+0000","last_unstale":"2024-05-12T22:07:45.723098+0000","last_undegraded":"2024-05-12T22:07:45.723098+0000","last_fullsized":"2024-05-12T22:07:45.723098+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T23:10:17.267823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:06:46.795258+0000","last_change":"2024-05-12T22:06:32.929955+0000","last_active":"2024-05-12T22:06:46.795258+0000","last_peered":"2024-05-12T22:06:46.795258+0000","last_clean":"2024-05-12T22:06:46.795258+0000","last_became_active":"2024-05-12T22:06:32.929623+0000","last_became_peered":"2024-05-12T22:06:32.929623+0000","last_unstale":"2024-05-12T22:06:46.795258+0000","last_undegraded":"2024-05-12T22:06:46.795258+0000","last_fullsized":"2024-05-12T22:06:46.795258+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T08:15:22.622908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"88'15","reported_seq":38,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.725455+0000","last_change":"2024-05-12T22:06:46.931736+0000","last_active":"2024-05-12T22:07:45.725455+0000","last_peered":"2024-05-12T22:07:45.725455+0000","last_clean":"2024-05-12T22:07:45.725455+0000","last_became_active":"2024-05-12T22:06:46.931265+0000","last_became_peered":"2024-05-12T22:06:46.931265+0000","last_unstale":"2024-05-12T22:07:45.725455+0000","last_undegraded":"2024-05-12T22:07:45.725455+0000","last_fullsized":"2024-05-12T22:07:45.725455+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T02:25:34.002024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"88'5","reported_seq":109,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:57.877999+0000","last_change":"2024-05-12T22:06:32.927707+0000","last_active":"2024-05-12T22:07:57.877999+0000","last_peered":"2024-05-12T22:07:57.877999+0000","last_clean":"2024-05-12T22:07:57.877999+0000","last_became_active":"2024-05-12T22:06:32.927406+0000","last_became_peered":"2024-05-12T22:06:32.927406+0000","last_unstale":"2024-05-12T22:07:57.877999+0000","last_undegraded":"2024-05-12T22:07:57.877999+0000","last_fullsized":"2024-05-12T22:07:57.877999+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T05:14:34.268589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1120,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":5,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.691438+0000","last_change":"2024-05-12T22:06:33.964996+0000","last_active":"2024-05-12T22:07:45.691438+0000","last_peered":"2024-05-12T22:07:45.691438+0000","last_clean":"2024-05-12T22:07:45.691438+0000","last_became_active":"2024-05-12T22:06:33.964587+0000","last_became_peered":"2024-05-12T22:06:33.964587+0000","last_unstale":"2024-05-12T22:07:45.691438+0000","last_undegraded":"2024-05-12T22:07:45.691438+0000","last_fullsized":"2024-05-12T22:07:45.691438+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T22:27:28.316377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"86'15","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:06:46.788234+0000","last_change":"2024-05-12T22:06:32.935543+0000","last_active":"2024-05-12T22:06:46.788234+0000","last_peered":"2024-05-12T22:06:46.788234+0000","last_clean":"2024-05-12T22:06:46.788234+0000","last_became_active":"2024-05-12T22:06:32.935145+0000","last_became_peered":"2024-05-12T22:06:32.935145+0000","last_unstale":"2024-05-12T22:06:46.788234+0000","last_undegraded":"2024-05-12T22:06:46.788234+0000","last_fullsized":"2024-05-12T22:06:46.788234+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T01:13:23.289057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.702680+0000","last_change":"2024-05-12T22:06:33.964367+0000","last_active":"2024-05-12T22:07:45.702680+0000","last_peered":"2024-05-12T22:07:45.702680+0000","last_clean":"2024-05-12T22:07:45.702680+0000","last_became_active":"2024-05-12T22:06:33.963876+0000","last_became_peered":"2024-05-12T22:06:33.963876+0000","last_unstale":"2024-05-12T22:07:45.702680+0000","last_undegraded":"2024-05-12T22:07:45.702680+0000","last_fullsized":"2024-05-12T22:07:45.702680+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T07:09:20.962965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"86'2","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:06:46.788927+0000","last_change":"2024-05-12T22:06:32.927270+0000","last_active":"2024-05-12T22:06:46.788927+0000","last_peered":"2024-05-12T22:06:46.788927+0000","last_clean":"2024-05-12T22:06:46.788927+0000","last_became_active":"2024-05-12T22:06:32.926930+0000","last_became_peered":"2024-05-12T22:06:32.926930+0000","last_unstale":"2024-05-12T22:06:46.788927+0000","last_undegraded":"2024-05-12T22:06:46.788927+0000","last_fullsized":"2024-05-12T22:06:46.788927+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T07:04:48.336599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.695444+0000","last_change":"2024-05-12T22:06:33.966350+0000","last_active":"2024-05-12T22:07:45.695444+0000","last_peered":"2024-05-12T22:07:45.695444+0000","last_clean":"2024-05-12T22:07:45.695444+0000","last_became_active":"2024-05-12T22:06:33.965974+0000","last_became_peered":"2024-05-12T22:06:33.965974+0000","last_unstale":"2024-05-12T22:07:45.695444+0000","last_undegraded":"2024-05-12T22:07:45.695444+0000","last_fullsized":"2024-05-12T22:07:45.695444+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T02:08:16.889684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:06:46.792377+0000","last_change":"2024-05-12T22:06:32.944945+0000","last_active":"2024-05-12T22:06:46.792377+0000","last_peered":"2024-05-12T22:06:46.792377+0000","last_clean":"2024-05-12T22:06:46.792377+0000","last_became_active":"2024-05-12T22:06:32.944576+0000","last_became_peered":"2024-05-12T22:06:32.944576+0000","last_unstale":"2024-05-12T22:06:46.792377+0000","last_undegraded":"2024-05-12T22:06:46.792377+0000","last_fullsized":"2024-05-12T22:06:46.792377+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T07:29:14.544821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.720934+0000","last_change":"2024-05-12T22:06:33.966299+0000","last_active":"2024-05-12T22:07:45.720934+0000","last_peered":"2024-05-12T22:07:45.720934+0000","last_clean":"2024-05-12T22:07:45.720934+0000","last_became_active":"2024-05-12T22:06:33.965902+0000","last_became_peered":"2024-05-12T22:06:33.965902+0000","last_unstale":"2024-05-12T22:07:45.720934+0000","last_undegraded":"2024-05-12T22:07:45.720934+0000","last_fullsized":"2024-05-12T22:07:45.720934+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T22:37:51.476546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"86'1","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:06:46.786408+0000","last_change":"2024-05-12T22:06:32.937210+0000","last_active":"2024-05-12T22:06:46.786408+0000","last_peered":"2024-05-12T22:06:46.786408+0000","last_clean":"2024-05-12T22:06:46.786408+0000","last_became_active":"2024-05-12T22:06:32.936805+0000","last_became_peered":"2024-05-12T22:06:32.936805+0000","last_unstale":"2024-05-12T22:06:46.786408+0000","last_undegraded":"2024-05-12T22:06:46.786408+0000","last_fullsized":"2024-05-12T22:06:46.786408+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T06:14:20.825258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.692294+0000","last_change":"2024-05-12T22:06:33.983604+0000","last_active":"2024-05-12T22:07:45.692294+0000","last_peered":"2024-05-12T22:07:45.692294+0000","last_clean":"2024-05-12T22:07:45.692294+0000","last_became_active":"2024-05-12T22:06:33.983168+0000","last_became_peered":"2024-05-12T22:06:33.983168+0000","last_unstale":"2024-05-12T22:07:45.692294+0000","last_undegraded":"2024-05-12T22:07:45.692294+0000","last_fullsized":"2024-05-12T22:07:45.692294+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T05:54:55.854593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.704701+0000","last_change":"2024-05-12T22:06:33.968848+0000","last_active":"2024-05-12T22:07:45.704701+0000","last_peered":"2024-05-12T22:07:45.704701+0000","last_clean":"2024-05-12T22:07:45.704701+0000","last_became_active":"2024-05-12T22:06:33.968529+0000","last_became_peered":"2024-05-12T22:06:33.968529+0000","last_unstale":"2024-05-12T22:07:45.704701+0000","last_undegraded":"2024-05-12T22:07:45.704701+0000","last_fullsized":"2024-05-12T22:07:45.704701+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T03:00:41.129662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"88'5","reported_seq":103,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:55.872933+0000","last_change":"2024-05-12T22:06:32.946670+0000","last_active":"2024-05-12T22:07:55.872933+0000","last_peered":"2024-05-12T22:07:55.872933+0000","last_clean":"2024-05-12T22:07:55.872933+0000","last_became_active":"2024-05-12T22:06:32.946059+0000","last_became_peered":"2024-05-12T22:06:32.946059+0000","last_unstale":"2024-05-12T22:07:55.872933+0000","last_undegraded":"2024-05-12T22:07:55.872933+0000","last_fullsized":"2024-05-12T22:07:55.872933+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T22:35:56.116979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":79,"num_read_kb":79,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.720948+0000","last_change":"2024-05-12T22:06:33.962177+0000","last_active":"2024-05-12T22:07:45.720948+0000","last_peered":"2024-05-12T22:07:45.720948+0000","last_clean":"2024-05-12T22:07:45.720948+0000","last_became_active":"2024-05-12T22:06:33.960919+0000","last_became_peered":"2024-05-12T22:06:33.960919+0000","last_unstale":"2024-05-12T22:07:45.720948+0000","last_undegraded":"2024-05-12T22:07:45.720948+0000","last_fullsized":"2024-05-12T22:07:45.720948+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T00:39:54.566460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'3","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:06:46.784690+0000","last_change":"2024-05-12T22:06:32.927885+0000","last_active":"2024-05-12T22:06:46.784690+0000","last_peered":"2024-05-12T22:06:46.784690+0000","last_clean":"2024-05-12T22:06:46.784690+0000","last_became_active":"2024-05-12T22:06:32.927567+0000","last_became_peered":"2024-05-12T22:06:32.927567+0000","last_unstale":"2024-05-12T22:06:46.784690+0000","last_undegraded":"2024-05-12T22:06:46.784690+0000","last_fullsized":"2024-05-12T22:06:46.784690+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T07:13:50.811992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":353,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:06:46.786385+0000","last_change":"2024-05-12T22:05:17.683363+0000","last_active":"2024-05-12T22:06:46.786385+0000","last_peered":"2024-05-12T22:06:46.786385+0000","last_clean":"2024-05-12T22:06:46.786385+0000","last_became_active":"2024-05-12T22:05:17.682824+0000","last_became_peered":"2024-05-12T22:05:17.682824+0000","last_unstale":"2024-05-12T22:06:46.786385+0000","last_undegraded":"2024-05-12T22:06:46.786385+0000","last_fullsized":"2024-05-12T22:06:46.786385+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:01:32.304714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:01:32.304714+0000","last_clean_scrub_stamp":"2024-05-12T22:01:32.304714+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T06:35:22.474820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.714366+0000","last_change":"2024-05-12T22:06:33.965332+0000","last_active":"2024-05-12T22:07:45.714366+0000","last_peered":"2024-05-12T22:07:45.714366+0000","last_clean":"2024-05-12T22:07:45.714366+0000","last_became_active":"2024-05-12T22:06:33.964858+0000","last_became_peered":"2024-05-12T22:06:33.964858+0000","last_unstale":"2024-05-12T22:07:45.714366+0000","last_undegraded":"2024-05-12T22:07:45.714366+0000","last_fullsized":"2024-05-12T22:07:45.714366+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T05:14:09.825614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:06:46.786368+0000","last_change":"2024-05-12T22:06:32.929773+0000","last_active":"2024-05-12T22:06:46.786368+0000","last_peered":"2024-05-12T22:06:46.786368+0000","last_clean":"2024-05-12T22:06:46.786368+0000","last_became_active":"2024-05-12T22:06:32.929458+0000","last_became_peered":"2024-05-12T22:06:32.929458+0000","last_unstale":"2024-05-12T22:06:46.786368+0000","last_undegraded":"2024-05-12T22:06:46.786368+0000","last_fullsized":"2024-05-12T22:06:46.786368+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T01:30:42.853183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"86'1","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:06:46.788861+0000","last_change":"2024-05-12T22:06:32.947014+0000","last_active":"2024-05-12T22:06:46.788861+0000","last_peered":"2024-05-12T22:06:46.788861+0000","last_clean":"2024-05-12T22:06:46.788861+0000","last_became_active":"2024-05-12T22:06:32.946661+0000","last_became_peered":"2024-05-12T22:06:32.946661+0000","last_unstale":"2024-05-12T22:06:46.788861+0000","last_undegraded":"2024-05-12T22:06:46.788861+0000","last_fullsized":"2024-05-12T22:06:46.788861+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T23:09:19.378353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.715360+0000","last_change":"2024-05-12T22:06:33.980490+0000","last_active":"2024-05-12T22:07:45.715360+0000","last_peered":"2024-05-12T22:07:45.715360+0000","last_clean":"2024-05-12T22:07:45.715360+0000","last_became_active":"2024-05-12T22:06:33.980133+0000","last_became_peered":"2024-05-12T22:06:33.980133+0000","last_unstale":"2024-05-12T22:07:45.715360+0000","last_undegraded":"2024-05-12T22:07:45.715360+0000","last_fullsized":"2024-05-12T22:07:45.715360+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T00:28:10.461428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"88'11","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:06:48.270662+0000","last_change":"2024-05-12T22:06:32.943152+0000","last_active":"2024-05-12T22:06:48.270662+0000","last_peered":"2024-05-12T22:06:48.270662+0000","last_clean":"2024-05-12T22:06:48.270662+0000","last_became_active":"2024-05-12T22:06:32.942769+0000","last_became_peered":"2024-05-12T22:06:32.942769+0000","last_unstale":"2024-05-12T22:06:48.270662+0000","last_undegraded":"2024-05-12T22:06:48.270662+0000","last_fullsized":"2024-05-12T22:06:48.270662+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T01:23:24.955739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.719134+0000","last_change":"2024-05-12T22:06:33.964081+0000","last_active":"2024-05-12T22:07:45.719134+0000","last_peered":"2024-05-12T22:07:45.719134+0000","last_clean":"2024-05-12T22:07:45.719134+0000","last_became_active":"2024-05-12T22:06:33.963054+0000","last_became_peered":"2024-05-12T22:06:33.963054+0000","last_unstale":"2024-05-12T22:07:45.719134+0000","last_undegraded":"2024-05-12T22:07:45.719134+0000","last_fullsized":"2024-05-12T22:07:45.719134+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T09:40:03.396200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"86'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:06:46.785077+0000","last_change":"2024-05-12T22:06:32.936432+0000","last_active":"2024-05-12T22:06:46.785077+0000","last_peered":"2024-05-12T22:06:46.785077+0000","last_clean":"2024-05-12T22:06:46.785077+0000","last_became_active":"2024-05-12T22:06:32.936094+0000","last_became_peered":"2024-05-12T22:06:32.936094+0000","last_unstale":"2024-05-12T22:06:46.785077+0000","last_undegraded":"2024-05-12T22:06:46.785077+0000","last_fullsized":"2024-05-12T22:06:46.785077+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T02:40:31.346118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.733892+0000","last_change":"2024-05-12T22:06:33.985953+0000","last_active":"2024-05-12T22:07:45.733892+0000","last_peered":"2024-05-12T22:07:45.733892+0000","last_clean":"2024-05-12T22:07:45.733892+0000","last_became_active":"2024-05-12T22:06:33.984864+0000","last_became_peered":"2024-05-12T22:06:33.984864+0000","last_unstale":"2024-05-12T22:07:45.733892+0000","last_undegraded":"2024-05-12T22:07:45.733892+0000","last_fullsized":"2024-05-12T22:07:45.733892+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T07:28:22.357841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:06:46.788690+0000","last_change":"2024-05-12T22:06:32.934510+0000","last_active":"2024-05-12T22:06:46.788690+0000","last_peered":"2024-05-12T22:06:46.788690+0000","last_clean":"2024-05-12T22:06:46.788690+0000","last_became_active":"2024-05-12T22:06:32.934222+0000","last_became_peered":"2024-05-12T22:06:32.934222+0000","last_unstale":"2024-05-12T22:06:46.788690+0000","last_undegraded":"2024-05-12T22:06:46.788690+0000","last_fullsized":"2024-05-12T22:06:46.788690+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T00:57:44.321629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"88'2","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.670036+0000","last_change":"2024-05-12T22:06:33.963870+0000","last_active":"2024-05-12T22:07:45.670036+0000","last_peered":"2024-05-12T22:07:45.670036+0000","last_clean":"2024-05-12T22:07:45.670036+0000","last_became_active":"2024-05-12T22:06:33.963533+0000","last_became_peered":"2024-05-12T22:06:33.963533+0000","last_unstale":"2024-05-12T22:07:45.670036+0000","last_undegraded":"2024-05-12T22:07:45.670036+0000","last_fullsized":"2024-05-12T22:07:45.670036+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T08:56:59.495469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:06:46.788300+0000","last_change":"2024-05-12T22:06:32.939738+0000","last_active":"2024-05-12T22:06:46.788300+0000","last_peered":"2024-05-12T22:06:46.788300+0000","last_clean":"2024-05-12T22:06:46.788300+0000","last_became_active":"2024-05-12T22:06:32.939365+0000","last_became_peered":"2024-05-12T22:06:32.939365+0000","last_unstale":"2024-05-12T22:06:46.788300+0000","last_undegraded":"2024-05-12T22:06:46.788300+0000","last_fullsized":"2024-05-12T22:06:46.788300+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T02:19:31.180834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.736281+0000","last_change":"2024-05-12T22:06:33.975064+0000","last_active":"2024-05-12T22:07:45.736281+0000","last_peered":"2024-05-12T22:07:45.736281+0000","last_clean":"2024-05-12T22:07:45.736281+0000","last_became_active":"2024-05-12T22:06:33.974613+0000","last_became_peered":"2024-05-12T22:06:33.974613+0000","last_unstale":"2024-05-12T22:07:45.736281+0000","last_undegraded":"2024-05-12T22:07:45.736281+0000","last_fullsized":"2024-05-12T22:07:45.736281+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T08:46:27.343777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"86'1","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:06:46.789708+0000","last_change":"2024-05-12T22:06:32.931148+0000","last_active":"2024-05-12T22:06:46.789708+0000","last_peered":"2024-05-12T22:06:46.789708+0000","last_clean":"2024-05-12T22:06:46.789708+0000","last_became_active":"2024-05-12T22:06:32.930717+0000","last_became_peered":"2024-05-12T22:06:32.930717+0000","last_unstale":"2024-05-12T22:06:46.789708+0000","last_undegraded":"2024-05-12T22:06:46.789708+0000","last_fullsized":"2024-05-12T22:06:46.789708+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T07:26:20.915467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.736548+0000","last_change":"2024-05-12T22:06:33.988390+0000","last_active":"2024-05-12T22:07:45.736548+0000","last_peered":"2024-05-12T22:07:45.736548+0000","last_clean":"2024-05-12T22:07:45.736548+0000","last_became_active":"2024-05-12T22:06:33.986970+0000","last_became_peered":"2024-05-12T22:06:33.986970+0000","last_unstale":"2024-05-12T22:07:45.736548+0000","last_undegraded":"2024-05-12T22:07:45.736548+0000","last_fullsized":"2024-05-12T22:07:45.736548+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T05:03:31.930261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":36,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:06:46.798123+0000","last_change":"2024-05-12T22:06:46.798123+0000","last_active":"2024-05-12T22:06:46.798123+0000","last_peered":"2024-05-12T22:06:46.798123+0000","last_clean":"2024-05-12T22:06:46.798123+0000","last_became_active":"2024-05-12T22:06:46.791029+0000","last_became_peered":"2024-05-12T22:06:46.791029+0000","last_unstale":"2024-05-12T22:06:46.798123+0000","last_undegraded":"2024-05-12T22:06:46.798123+0000","last_fullsized":"2024-05-12T22:06:46.798123+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T06:31:25.108659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,7],"acting":[8,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.720460+0000","last_change":"2024-05-12T22:06:33.968256+0000","last_active":"2024-05-12T22:07:45.720460+0000","last_peered":"2024-05-12T22:07:45.720460+0000","last_clean":"2024-05-12T22:07:45.720460+0000","last_became_active":"2024-05-12T22:06:33.967885+0000","last_became_peered":"2024-05-12T22:06:33.967885+0000","last_unstale":"2024-05-12T22:07:45.720460+0000","last_undegraded":"2024-05-12T22:07:45.720460+0000","last_fullsized":"2024-05-12T22:07:45.720460+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T09:12:40.687367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"86'3","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:06:46.789849+0000","last_change":"2024-05-12T22:06:32.944736+0000","last_active":"2024-05-12T22:06:46.789849+0000","last_peered":"2024-05-12T22:06:46.789849+0000","last_clean":"2024-05-12T22:06:46.789849+0000","last_became_active":"2024-05-12T22:06:32.944269+0000","last_became_peered":"2024-05-12T22:06:32.944269+0000","last_unstale":"2024-05-12T22:06:46.789849+0000","last_undegraded":"2024-05-12T22:06:46.789849+0000","last_fullsized":"2024-05-12T22:06:46.789849+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T22:56:24.391267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.721036+0000","last_change":"2024-05-12T22:06:33.968360+0000","last_active":"2024-05-12T22:07:45.721036+0000","last_peered":"2024-05-12T22:07:45.721036+0000","last_clean":"2024-05-12T22:07:45.721036+0000","last_became_active":"2024-05-12T22:06:33.967967+0000","last_became_peered":"2024-05-12T22:06:33.967967+0000","last_unstale":"2024-05-12T22:07:45.721036+0000","last_undegraded":"2024-05-12T22:07:45.721036+0000","last_fullsized":"2024-05-12T22:07:45.721036+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T00:41:12.693858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"85'1","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:06:46.794591+0000","last_change":"2024-05-12T22:06:32.935993+0000","last_active":"2024-05-12T22:06:46.794591+0000","last_peered":"2024-05-12T22:06:46.794591+0000","last_clean":"2024-05-12T22:06:46.794591+0000","last_became_active":"2024-05-12T22:06:32.935615+0000","last_became_peered":"2024-05-12T22:06:32.935615+0000","last_unstale":"2024-05-12T22:06:46.794591+0000","last_undegraded":"2024-05-12T22:06:46.794591+0000","last_fullsized":"2024-05-12T22:06:46.794591+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T00:43:39.254637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.708035+0000","last_change":"2024-05-12T22:06:33.966110+0000","last_active":"2024-05-12T22:07:45.708035+0000","last_peered":"2024-05-12T22:07:45.708035+0000","last_clean":"2024-05-12T22:07:45.708035+0000","last_became_active":"2024-05-12T22:06:33.965755+0000","last_became_peered":"2024-05-12T22:06:33.965755+0000","last_unstale":"2024-05-12T22:07:45.708035+0000","last_undegraded":"2024-05-12T22:07:45.708035+0000","last_fullsized":"2024-05-12T22:07:45.708035+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T22:18:53.941335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:06:46.929070+0000","last_change":"2024-05-12T22:06:32.942229+0000","last_active":"2024-05-12T22:06:46.929070+0000","last_peered":"2024-05-12T22:06:46.929070+0000","last_clean":"2024-05-12T22:06:46.929070+0000","last_became_active":"2024-05-12T22:06:32.941826+0000","last_became_peered":"2024-05-12T22:06:32.941826+0000","last_unstale":"2024-05-12T22:06:46.929070+0000","last_undegraded":"2024-05-12T22:06:46.929070+0000","last_fullsized":"2024-05-12T22:06:46.929070+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T03:52:34.886318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.695139+0000","last_change":"2024-05-12T22:06:33.963081+0000","last_active":"2024-05-12T22:07:45.695139+0000","last_peered":"2024-05-12T22:07:45.695139+0000","last_clean":"2024-05-12T22:07:45.695139+0000","last_became_active":"2024-05-12T22:06:33.962696+0000","last_became_peered":"2024-05-12T22:06:33.962696+0000","last_unstale":"2024-05-12T22:07:45.695139+0000","last_undegraded":"2024-05-12T22:07:45.695139+0000","last_fullsized":"2024-05-12T22:07:45.695139+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T05:55:45.581823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.698920+0000","last_change":"2024-05-12T22:06:33.968865+0000","last_active":"2024-05-12T22:07:45.698920+0000","last_peered":"2024-05-12T22:07:45.698920+0000","last_clean":"2024-05-12T22:07:45.698920+0000","last_became_active":"2024-05-12T22:06:33.968185+0000","last_became_peered":"2024-05-12T22:06:33.968185+0000","last_unstale":"2024-05-12T22:07:45.698920+0000","last_undegraded":"2024-05-12T22:07:45.698920+0000","last_fullsized":"2024-05-12T22:07:45.698920+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T01:23:41.083778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"88'12","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.704125+0000","last_change":"2024-05-12T22:06:33.964786+0000","last_active":"2024-05-12T22:07:45.704125+0000","last_peered":"2024-05-12T22:07:45.704125+0000","last_clean":"2024-05-12T22:07:45.704125+0000","last_became_active":"2024-05-12T22:06:33.964227+0000","last_became_peered":"2024-05-12T22:06:33.964227+0000","last_unstale":"2024-05-12T22:07:45.704125+0000","last_undegraded":"2024-05-12T22:07:45.704125+0000","last_fullsized":"2024-05-12T22:07:45.704125+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T02:53:42.739455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.717643+0000","last_change":"2024-05-12T22:06:33.978857+0000","last_active":"2024-05-12T22:07:45.717643+0000","last_peered":"2024-05-12T22:07:45.717643+0000","last_clean":"2024-05-12T22:07:45.717643+0000","last_became_active":"2024-05-12T22:06:33.978531+0000","last_became_peered":"2024-05-12T22:06:33.978531+0000","last_unstale":"2024-05-12T22:07:45.717643+0000","last_undegraded":"2024-05-12T22:07:45.717643+0000","last_fullsized":"2024-05-12T22:07:45.717643+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T02:51:05.609267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.733111+0000","last_change":"2024-05-12T22:06:33.980662+0000","last_active":"2024-05-12T22:07:45.733111+0000","last_peered":"2024-05-12T22:07:45.733111+0000","last_clean":"2024-05-12T22:07:45.733111+0000","last_became_active":"2024-05-12T22:06:33.980332+0000","last_became_peered":"2024-05-12T22:06:33.980332+0000","last_unstale":"2024-05-12T22:07:45.733111+0000","last_undegraded":"2024-05-12T22:07:45.733111+0000","last_fullsized":"2024-05-12T22:07:45.733111+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T06:21:59.715792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"88'4","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.737927+0000","last_change":"2024-05-12T22:06:33.987338+0000","last_active":"2024-05-12T22:07:45.737927+0000","last_peered":"2024-05-12T22:07:45.737927+0000","last_clean":"2024-05-12T22:07:45.737927+0000","last_became_active":"2024-05-12T22:06:33.986942+0000","last_became_peered":"2024-05-12T22:06:33.986942+0000","last_unstale":"2024-05-12T22:07:45.737927+0000","last_undegraded":"2024-05-12T22:07:45.737927+0000","last_fullsized":"2024-05-12T22:07:45.737927+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T01:02:05.137558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"88'6","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.711410+0000","last_change":"2024-05-12T22:06:46.796752+0000","last_active":"2024-05-12T22:07:45.711410+0000","last_peered":"2024-05-12T22:07:45.711410+0000","last_clean":"2024-05-12T22:07:45.711410+0000","last_became_active":"2024-05-12T22:06:46.796369+0000","last_became_peered":"2024-05-12T22:06:46.796369+0000","last_unstale":"2024-05-12T22:07:45.711410+0000","last_undegraded":"2024-05-12T22:07:45.711410+0000","last_fullsized":"2024-05-12T22:07:45.711410+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T08:23:24.685495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,9],"acting":[11,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"88'15","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.714582+0000","last_change":"2024-05-12T22:06:33.963902+0000","last_active":"2024-05-12T22:07:45.714582+0000","last_peered":"2024-05-12T22:07:45.714582+0000","last_clean":"2024-05-12T22:07:45.714582+0000","last_became_active":"2024-05-12T22:06:33.963436+0000","last_became_peered":"2024-05-12T22:06:33.963436+0000","last_unstale":"2024-05-12T22:07:45.714582+0000","last_undegraded":"2024-05-12T22:07:45.714582+0000","last_fullsized":"2024-05-12T22:07:45.714582+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T02:51:56.415167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"88'2","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.707384+0000","last_change":"2024-05-12T22:06:46.794434+0000","last_active":"2024-05-12T22:07:45.707384+0000","last_peered":"2024-05-12T22:07:45.707384+0000","last_clean":"2024-05-12T22:07:45.707384+0000","last_became_active":"2024-05-12T22:06:46.793147+0000","last_became_peered":"2024-05-12T22:06:46.793147+0000","last_unstale":"2024-05-12T22:07:45.707384+0000","last_undegraded":"2024-05-12T22:07:45.707384+0000","last_fullsized":"2024-05-12T22:07:45.707384+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T23:10:39.529863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,8,2],"acting":[3,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.718420+0000","last_change":"2024-05-12T22:06:33.973490+0000","last_active":"2024-05-12T22:07:45.718420+0000","last_peered":"2024-05-12T22:07:45.718420+0000","last_clean":"2024-05-12T22:07:45.718420+0000","last_became_active":"2024-05-12T22:06:33.973161+0000","last_became_peered":"2024-05-12T22:06:33.973161+0000","last_unstale":"2024-05-12T22:07:45.718420+0000","last_undegraded":"2024-05-12T22:07:45.718420+0000","last_fullsized":"2024-05-12T22:07:45.718420+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T03:29:36.972631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.721084+0000","last_change":"2024-05-12T22:06:33.957320+0000","last_active":"2024-05-12T22:07:45.721084+0000","last_peered":"2024-05-12T22:07:45.721084+0000","last_clean":"2024-05-12T22:07:45.721084+0000","last_became_active":"2024-05-12T22:06:33.956837+0000","last_became_peered":"2024-05-12T22:06:33.956837+0000","last_unstale":"2024-05-12T22:07:45.721084+0000","last_undegraded":"2024-05-12T22:07:45.721084+0000","last_fullsized":"2024-05-12T22:07:45.721084+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T01:20:51.018175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"88'10","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.722857+0000","last_change":"2024-05-12T22:06:33.968363+0000","last_active":"2024-05-12T22:07:45.722857+0000","last_peered":"2024-05-12T22:07:45.722857+0000","last_clean":"2024-05-12T22:07:45.722857+0000","last_became_active":"2024-05-12T22:06:33.968031+0000","last_became_peered":"2024-05-12T22:06:33.968031+0000","last_unstale":"2024-05-12T22:07:45.722857+0000","last_undegraded":"2024-05-12T22:07:45.722857+0000","last_fullsized":"2024-05-12T22:07:45.722857+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T00:14:07.124058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"88'11","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.738418+0000","last_change":"2024-05-12T22:06:33.984823+0000","last_active":"2024-05-12T22:07:45.738418+0000","last_peered":"2024-05-12T22:07:45.738418+0000","last_clean":"2024-05-12T22:07:45.738418+0000","last_became_active":"2024-05-12T22:06:33.984430+0000","last_became_peered":"2024-05-12T22:06:33.984430+0000","last_unstale":"2024-05-12T22:07:45.738418+0000","last_undegraded":"2024-05-12T22:07:45.738418+0000","last_fullsized":"2024-05-12T22:07:45.738418+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T01:30:18.109265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.693058+0000","last_change":"2024-05-12T22:06:33.962660+0000","last_active":"2024-05-12T22:07:45.693058+0000","last_peered":"2024-05-12T22:07:45.693058+0000","last_clean":"2024-05-12T22:07:45.693058+0000","last_became_active":"2024-05-12T22:06:33.962284+0000","last_became_peered":"2024-05-12T22:06:33.962284+0000","last_unstale":"2024-05-12T22:07:45.693058+0000","last_undegraded":"2024-05-12T22:07:45.693058+0000","last_fullsized":"2024-05-12T22:07:45.693058+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T01:08:53.888992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":65796,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":514,"num_read_kb":328,"num_write":172,"num_write_kb":120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":197388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":148,"ondisk_log_size":148,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351007,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27948,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717204,"statfs":{"total":95995035648,"available":95966416896,"internally_reserved":0,"allocated":954368,"data_stored":459830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612643,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28624,"kb_used_data":1608,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716528,"statfs":{"total":95995035648,"available":95965724672,"internally_reserved":0,"allocated":1646592,"data_stored":1112383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776169,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":966656,"data_stored":459484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037807,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32120,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713032,"statfs":{"total":95995035648,"available":95962144768,"internally_reserved":0,"allocated":966656,"data_stored":459404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299443,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28536,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716616,"statfs":{"total":95995035648,"available":95965814784,"internally_reserved":0,"allocated":1556480,"data_stored":1051425,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462970,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27948,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717204,"statfs":{"total":95995035648,"available":95966416896,"internally_reserved":0,"allocated":954368,"data_stored":458873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574933,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36328,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708824,"statfs":{"total":95995035648,"available":95957835776,"internally_reserved":0,"allocated":1015808,"data_stored":519788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738458,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":974848,"data_stored":460144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411408,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28032,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717120,"statfs":{"total":95995035648,"available":95966330880,"internally_reserved":0,"allocated":1040384,"data_stored":521670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149770,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":970752,"data_stored":459184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888132,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27952,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717200,"statfs":{"total":95995035648,"available":95966412800,"internally_reserved":0,"allocated":958464,"data_stored":459842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724607,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28552,"kb_used_data":1536,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716600,"statfs":{"total":95995035648,"available":95965798400,"internally_reserved":0,"allocated":1572864,"data_stored":1051159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":61122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":63004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":63232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-12T22:08:01.674 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f72fecff68e1d400c4568684327c900485c20d6a shell --fsid 6b92e7b4-10aa-11ef-bc98-c7b262605968 -- ceph pg dump --format=json 2024-05-12T22:08:02.172 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/mon.a/config 2024-05-12T22:08:02.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:01 smithi186 ceph-mon[34785]: from='client.24868 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-12T22:08:02.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:01 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:08:02.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:01 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:08:02.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:01 smithi086 ceph-mon[34297]: from='client.24868 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-12T22:08:02.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:01 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:08:02.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:01 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:08:02.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:01 smithi028 ceph-mon[29847]: from='client.24868 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-12T22:08:02.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:01 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:08:02.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:01 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:08:03.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:02 smithi186 ceph-mon[34785]: pgmap v356: 97 pgs: 97 active+clean; 641 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:08:03.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:02 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:08:03.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:02 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:08:03.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:02 smithi186 ceph-mon[34785]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:08:03.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:02 smithi086 ceph-mon[34297]: pgmap v356: 97 pgs: 97 active+clean; 641 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:08:03.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:02 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:08:03.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:02 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:08:03.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:02 smithi086 ceph-mon[34297]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:08:03.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:02 smithi028 ceph-mon[29847]: pgmap v356: 97 pgs: 97 active+clean; 641 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:08:03.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:02 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:08:03.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:02 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:08:03.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:02 smithi028 ceph-mon[29847]: from='mgr.14152 172.21.15.28:0/1948263547' entity='mgr.x' 2024-05-12T22:08:04.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:04.145+0000 7f38c8c00700 1 -- 172.21.15.28:0/1058709427 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f38c4104f20 msgr2=0x7f38c41113e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:04.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:04.145+0000 7f38c8c00700 1 --2- 172.21.15.28:0/1058709427 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f38c4104f20 0x7f38c41113e0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f38bc00b0a0 tx=0x7f38bc00b3b0 comp rx=0 tx=0).stop 2024-05-12T22:08:04.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:04.145+0000 7f38c8c00700 1 -- 172.21.15.28:0/1058709427 shutdown_connections 2024-05-12T22:08:04.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:04.145+0000 7f38c8c00700 1 --2- 172.21.15.28:0/1058709427 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f38c4104f20 0x7f38c41113e0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:04.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:04.145+0000 7f38c8c00700 1 --2- 172.21.15.28:0/1058709427 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f38c410a610 0x7f38c410a9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:04.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:04.145+0000 7f38c8c00700 1 --2- 172.21.15.28:0/1058709427 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38c4104560 0x7f38c41049e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:04.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:04.145+0000 7f38c8c00700 1 -- 172.21.15.28:0/1058709427 >> 172.21.15.28:0/1058709427 conn(0x7f38c4100030 msgr2=0x7f38c4102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:08:04.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:04.145+0000 7f38c8c00700 1 -- 172.21.15.28:0/1058709427 shutdown_connections 2024-05-12T22:08:04.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:04.145+0000 7f38c8c00700 1 -- 172.21.15.28:0/1058709427 wait complete. 2024-05-12T22:08:04.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:04.146+0000 7f38c8c00700 1 Processor -- start 2024-05-12T22:08:04.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:04.147+0000 7f38c8c00700 1 -- start start 2024-05-12T22:08:04.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:04.147+0000 7f38c8c00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38c4104560 0x7f38c41a08d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:04.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:04.147+0000 7f38c8c00700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f38c4104f20 0x7f38c41a0e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:04.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:04.147+0000 7f38c8c00700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f38c410a610 0x7f38c419aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:04.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:04.147+0000 7f38c8c00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f38c4114910 con 0x7f38c4104560 2024-05-12T22:08:04.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:04.147+0000 7f38c8c00700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f38c4114790 con 0x7f38c410a610 2024-05-12T22:08:04.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:04.147+0000 7f38c8c00700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f38c4114a90 con 0x7f38c4104f20 2024-05-12T22:08:04.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:04.147+0000 7f38c1400700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f38c4104f20 0x7f38c41a0e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:04.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:04.147+0000 7f38c1e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38c4104560 0x7f38c41a08d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:04.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:04.147+0000 7f38c1400700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f38c4104f20 0x7f38c41a0e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:33094/0 (socket says 172.21.15.28:33094) 2024-05-12T22:08:04.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:04.147+0000 7f38c1400700 1 -- 172.21.15.28:0/2551303707 learned_addr learned my addr 172.21.15.28:0/2551303707 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:08:04.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:04.148+0000 7f38c2800700 1 --2- 172.21.15.28:0/2551303707 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f38c410a610 0x7f38c419aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:04.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:04.148+0000 7f38c1400700 1 -- 172.21.15.28:0/2551303707 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f38c410a610 msgr2=0x7f38c419aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:04.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:04.148+0000 7f38c1400700 1 --2- 172.21.15.28:0/2551303707 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f38c410a610 0x7f38c419aa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:04.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:04.148+0000 7f38c1400700 1 -- 172.21.15.28:0/2551303707 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38c4104560 msgr2=0x7f38c41a08d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:04.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:04.148+0000 7f38c1400700 1 --2- 172.21.15.28:0/2551303707 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38c4104560 0x7f38c41a08d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:04.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:04.148+0000 7f38c1400700 1 -- 172.21.15.28:0/2551303707 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38c419b1f0 con 0x7f38c4104f20 2024-05-12T22:08:04.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:04.148+0000 7f38c1400700 1 --2- 172.21.15.28:0/2551303707 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f38c4104f20 0x7f38c41a0e10 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f38b000e9a0 tx=0x7f38b000ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:08:04.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:04.148+0000 7f38baa00700 1 -- 172.21.15.28:0/2551303707 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38b000ca10 con 0x7f38c4104f20 2024-05-12T22:08:04.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:04.149+0000 7f38baa00700 1 -- 172.21.15.28:0/2551303707 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f38b000cbb0 con 0x7f38c4104f20 2024-05-12T22:08:04.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:04.149+0000 7f38c8c00700 1 -- 172.21.15.28:0/2551303707 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38c419b4e0 con 0x7f38c4104f20 2024-05-12T22:08:04.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:04.149+0000 7f38baa00700 1 -- 172.21.15.28:0/2551303707 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38b0018680 con 0x7f38c4104f20 2024-05-12T22:08:04.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:04.149+0000 7f38c8c00700 1 -- 172.21.15.28:0/2551303707 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38c41a77a0 con 0x7f38c4104f20 2024-05-12T22:08:04.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:04.150+0000 7f38c8c00700 1 -- 172.21.15.28:0/2551303707 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f38c4069010 con 0x7f38c4104f20 2024-05-12T22:08:04.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:04.152+0000 7f38baa00700 1 -- 172.21.15.28:0/2551303707 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f38b0010430 con 0x7f38c4104f20 2024-05-12T22:08:04.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:04.154+0000 7f38baa00700 1 --2- 172.21.15.28:0/2551303707 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f38a80a2ca0 0x7f38a80a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:04.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:04.154+0000 7f38baa00700 1 -- 172.21.15.28:0/2551303707 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f38b00cf5b0 con 0x7f38c4104f20 2024-05-12T22:08:04.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:04.154+0000 7f38c1e00700 1 --2- 172.21.15.28:0/2551303707 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f38a80a2ca0 0x7f38a80a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:04.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:04.155+0000 7f38c1e00700 1 --2- 172.21.15.28:0/2551303707 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f38a80a2ca0 0x7f38a80a5160 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f38c419bbd0 tx=0x7f38b4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:08:04.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:04.155+0000 7f38baa00700 1 -- 172.21.15.28:0/2551303707 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f38b009c4c0 con 0x7f38c4104f20 2024-05-12T22:08:04.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:04.380+0000 7f38c8c00700 1 -- 172.21.15.28:0/2551303707 --> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f38c4101c70 con 0x7f38a80a2ca0 2024-05-12T22:08:04.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:04.403+0000 7f38baa00700 1 -- 172.21.15.28:0/2551303707 <== mgr.14152 v2:172.21.15.28:6800/3123328651 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258831 (secure 0 0 0) 0x7f38c4101c70 con 0x7f38a80a2ca0 2024-05-12T22:08:04.405 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-12T22:08:04.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:04.409+0000 7f38c8c00700 1 -- 172.21.15.28:0/2551303707 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f38a80a2ca0 msgr2=0x7f38a80a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:04.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:04.409+0000 7f38c8c00700 1 --2- 172.21.15.28:0/2551303707 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f38a80a2ca0 0x7f38a80a5160 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f38c419bbd0 tx=0x7f38b4009260 comp rx=0 tx=0).stop 2024-05-12T22:08:04.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:04.409+0000 7f38c8c00700 1 -- 172.21.15.28:0/2551303707 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f38c4104f20 msgr2=0x7f38c41a0e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:04.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:04.409+0000 7f38c8c00700 1 --2- 172.21.15.28:0/2551303707 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f38c4104f20 0x7f38c41a0e10 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f38b000e9a0 tx=0x7f38b000ecb0 comp rx=0 tx=0).stop 2024-05-12T22:08:04.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:04.409+0000 7f38c8c00700 1 -- 172.21.15.28:0/2551303707 shutdown_connections 2024-05-12T22:08:04.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:04.409+0000 7f38c8c00700 1 --2- 172.21.15.28:0/2551303707 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f38a80a2ca0 0x7f38a80a5160 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:04.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:04.409+0000 7f38c8c00700 1 --2- 172.21.15.28:0/2551303707 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f38c410a610 0x7f38c419aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:04.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:04.409+0000 7f38c8c00700 1 --2- 172.21.15.28:0/2551303707 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38c4104560 0x7f38c41a08d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:04.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:04.409+0000 7f38c8c00700 1 --2- 172.21.15.28:0/2551303707 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f38c4104f20 0x7f38c41a0e10 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:04.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:04.409+0000 7f38c8c00700 1 -- 172.21.15.28:0/2551303707 >> 172.21.15.28:0/2551303707 conn(0x7f38c4100030 msgr2=0x7f38c4101560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:08:04.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:04.409+0000 7f38c8c00700 1 -- 172.21.15.28:0/2551303707 shutdown_connections 2024-05-12T22:08:04.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:04.409+0000 7f38c8c00700 1 -- 172.21.15.28:0/2551303707 wait complete. 2024-05-12T22:08:04.411 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-05-12T22:08:05.146 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:04 smithi028 ceph-mon[29847]: pgmap v357: 97 pgs: 97 active+clean; 641 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-12T22:08:05.147 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:04 smithi028 ceph-mon[29847]: from='client.24874 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-12T22:08:05.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:04 smithi186 ceph-mon[34785]: pgmap v357: 97 pgs: 97 active+clean; 641 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-12T22:08:05.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:04 smithi186 ceph-mon[34785]: from='client.24874 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-12T22:08:05.222 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":357,"stamp":"2024-05-12T22:08:03.749395+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":656398,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":660,"num_read_kb":452,"num_write":355,"num_write_kb":3060,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":1181318,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":787,"ondisk_log_size":787,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":349932,"kb_used_data":13260,"kb_used_omap":18,"kb_used_meta":336557,"kb_avail":1124591892,"statfs":{"total":1151940427776,"available":1151582097408,"internally_reserved":0,"allocated":13578240,"data_stored":7473186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":344634681},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":70,"num_read_kb":35,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005303"},"pg_stats":[{"pgid":"3.3f","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.739120+0000","last_change":"2024-05-12T22:06:33.982079+0000","last_active":"2024-05-12T22:07:45.739120+0000","last_peered":"2024-05-12T22:07:45.739120+0000","last_clean":"2024-05-12T22:07:45.739120+0000","last_became_active":"2024-05-12T22:06:33.981703+0000","last_became_peered":"2024-05-12T22:06:33.981703+0000","last_unstale":"2024-05-12T22:07:45.739120+0000","last_undegraded":"2024-05-12T22:07:45.739120+0000","last_fullsized":"2024-05-12T22:07:45.739120+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T08:52:47.157067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"88'4","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.735806+0000","last_change":"2024-05-12T22:06:33.984194+0000","last_active":"2024-05-12T22:07:45.735806+0000","last_peered":"2024-05-12T22:07:45.735806+0000","last_clean":"2024-05-12T22:07:45.735806+0000","last_became_active":"2024-05-12T22:06:33.979787+0000","last_became_peered":"2024-05-12T22:06:33.979787+0000","last_unstale":"2024-05-12T22:07:45.735806+0000","last_undegraded":"2024-05-12T22:07:45.735806+0000","last_fullsized":"2024-05-12T22:07:45.735806+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T07:36:46.588641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.729071+0000","last_change":"2024-05-12T22:06:33.982428+0000","last_active":"2024-05-12T22:07:45.729071+0000","last_peered":"2024-05-12T22:07:45.729071+0000","last_clean":"2024-05-12T22:07:45.729071+0000","last_became_active":"2024-05-12T22:06:33.979563+0000","last_became_peered":"2024-05-12T22:06:33.979563+0000","last_unstale":"2024-05-12T22:07:45.729071+0000","last_undegraded":"2024-05-12T22:07:45.729071+0000","last_fullsized":"2024-05-12T22:07:45.729071+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T06:26:12.586519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"88'8","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.712039+0000","last_change":"2024-05-12T22:06:46.796152+0000","last_active":"2024-05-12T22:07:45.712039+0000","last_peered":"2024-05-12T22:07:45.712039+0000","last_clean":"2024-05-12T22:07:45.712039+0000","last_became_active":"2024-05-12T22:06:46.795484+0000","last_became_peered":"2024-05-12T22:06:46.795484+0000","last_unstale":"2024-05-12T22:07:45.712039+0000","last_undegraded":"2024-05-12T22:07:45.712039+0000","last_fullsized":"2024-05-12T22:07:45.712039+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T03:13:45.901507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,2],"acting":[9,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.3b","version":"88'4","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.711043+0000","last_change":"2024-05-12T22:06:33.967492+0000","last_active":"2024-05-12T22:07:45.711043+0000","last_peered":"2024-05-12T22:07:45.711043+0000","last_clean":"2024-05-12T22:07:45.711043+0000","last_became_active":"2024-05-12T22:06:33.967146+0000","last_became_peered":"2024-05-12T22:06:33.967146+0000","last_unstale":"2024-05-12T22:07:45.711043+0000","last_undegraded":"2024-05-12T22:07:45.711043+0000","last_fullsized":"2024-05-12T22:07:45.711043+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T07:31:05.946473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.712943+0000","last_change":"2024-05-12T22:06:33.973653+0000","last_active":"2024-05-12T22:07:45.712943+0000","last_peered":"2024-05-12T22:07:45.712943+0000","last_clean":"2024-05-12T22:07:45.712943+0000","last_became_active":"2024-05-12T22:06:33.973290+0000","last_became_peered":"2024-05-12T22:06:33.973290+0000","last_unstale":"2024-05-12T22:07:45.712943+0000","last_undegraded":"2024-05-12T22:07:45.712943+0000","last_fullsized":"2024-05-12T22:07:45.712943+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T03:07:21.933141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.701187+0000","last_change":"2024-05-12T22:06:33.973807+0000","last_active":"2024-05-12T22:07:45.701187+0000","last_peered":"2024-05-12T22:07:45.701187+0000","last_clean":"2024-05-12T22:07:45.701187+0000","last_became_active":"2024-05-12T22:06:33.973466+0000","last_became_peered":"2024-05-12T22:06:33.973466+0000","last_unstale":"2024-05-12T22:07:45.701187+0000","last_undegraded":"2024-05-12T22:07:45.701187+0000","last_fullsized":"2024-05-12T22:07:45.701187+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T02:31:13.608309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.717205+0000","last_change":"2024-05-12T22:06:33.967356+0000","last_active":"2024-05-12T22:07:45.717205+0000","last_peered":"2024-05-12T22:07:45.717205+0000","last_clean":"2024-05-12T22:07:45.717205+0000","last_became_active":"2024-05-12T22:06:33.966054+0000","last_became_peered":"2024-05-12T22:06:33.966054+0000","last_unstale":"2024-05-12T22:07:45.717205+0000","last_undegraded":"2024-05-12T22:07:45.717205+0000","last_fullsized":"2024-05-12T22:07:45.717205+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T23:54:08.285720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"88'11","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.721194+0000","last_change":"2024-05-12T22:06:33.968782+0000","last_active":"2024-05-12T22:07:45.721194+0000","last_peered":"2024-05-12T22:07:45.721194+0000","last_clean":"2024-05-12T22:07:45.721194+0000","last_became_active":"2024-05-12T22:06:33.968435+0000","last_became_peered":"2024-05-12T22:06:33.968435+0000","last_unstale":"2024-05-12T22:07:45.721194+0000","last_undegraded":"2024-05-12T22:07:45.721194+0000","last_fullsized":"2024-05-12T22:07:45.721194+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T07:01:21.056777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"88'16","reported_seq":33,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.699709+0000","last_change":"2024-05-12T22:06:33.962018+0000","last_active":"2024-05-12T22:07:45.699709+0000","last_peered":"2024-05-12T22:07:45.699709+0000","last_clean":"2024-05-12T22:07:45.699709+0000","last_became_active":"2024-05-12T22:06:33.961583+0000","last_became_peered":"2024-05-12T22:06:33.961583+0000","last_unstale":"2024-05-12T22:07:45.699709+0000","last_undegraded":"2024-05-12T22:07:45.699709+0000","last_fullsized":"2024-05-12T22:07:45.699709+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T00:16:40.483194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.721027+0000","last_change":"2024-05-12T22:06:33.965776+0000","last_active":"2024-05-12T22:07:45.721027+0000","last_peered":"2024-05-12T22:07:45.721027+0000","last_clean":"2024-05-12T22:07:45.721027+0000","last_became_active":"2024-05-12T22:06:33.965439+0000","last_became_peered":"2024-05-12T22:06:33.965439+0000","last_unstale":"2024-05-12T22:07:45.721027+0000","last_undegraded":"2024-05-12T22:07:45.721027+0000","last_fullsized":"2024-05-12T22:07:45.721027+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T23:59:56.925209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.693930+0000","last_change":"2024-05-12T22:06:33.967906+0000","last_active":"2024-05-12T22:07:45.693930+0000","last_peered":"2024-05-12T22:07:45.693930+0000","last_clean":"2024-05-12T22:07:45.693930+0000","last_became_active":"2024-05-12T22:06:33.967516+0000","last_became_peered":"2024-05-12T22:06:33.967516+0000","last_unstale":"2024-05-12T22:07:45.693930+0000","last_undegraded":"2024-05-12T22:07:45.693930+0000","last_fullsized":"2024-05-12T22:07:45.693930+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T08:16:56.892024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"88'10","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.726321+0000","last_change":"2024-05-12T22:06:33.971456+0000","last_active":"2024-05-12T22:07:45.726321+0000","last_peered":"2024-05-12T22:07:45.726321+0000","last_clean":"2024-05-12T22:07:45.726321+0000","last_became_active":"2024-05-12T22:06:33.971110+0000","last_became_peered":"2024-05-12T22:06:33.971110+0000","last_unstale":"2024-05-12T22:07:45.726321+0000","last_undegraded":"2024-05-12T22:07:45.726321+0000","last_fullsized":"2024-05-12T22:07:45.726321+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T09:48:56.275346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.726703+0000","last_change":"2024-05-12T22:06:33.983421+0000","last_active":"2024-05-12T22:07:45.726703+0000","last_peered":"2024-05-12T22:07:45.726703+0000","last_clean":"2024-05-12T22:07:45.726703+0000","last_became_active":"2024-05-12T22:06:33.978515+0000","last_became_peered":"2024-05-12T22:06:33.978515+0000","last_unstale":"2024-05-12T22:07:45.726703+0000","last_undegraded":"2024-05-12T22:07:45.726703+0000","last_fullsized":"2024-05-12T22:07:45.726703+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T23:09:48.264541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.731547+0000","last_change":"2024-05-12T22:06:33.980956+0000","last_active":"2024-05-12T22:07:45.731547+0000","last_peered":"2024-05-12T22:07:45.731547+0000","last_clean":"2024-05-12T22:07:45.731547+0000","last_became_active":"2024-05-12T22:06:33.979860+0000","last_became_peered":"2024-05-12T22:06:33.979860+0000","last_unstale":"2024-05-12T22:07:45.731547+0000","last_undegraded":"2024-05-12T22:07:45.731547+0000","last_fullsized":"2024-05-12T22:07:45.731547+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T02:58:29.255704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.725416+0000","last_change":"2024-05-12T22:06:33.978682+0000","last_active":"2024-05-12T22:07:45.725416+0000","last_peered":"2024-05-12T22:07:45.725416+0000","last_clean":"2024-05-12T22:07:45.725416+0000","last_became_active":"2024-05-12T22:06:33.978234+0000","last_became_peered":"2024-05-12T22:06:33.978234+0000","last_unstale":"2024-05-12T22:07:45.725416+0000","last_undegraded":"2024-05-12T22:07:45.725416+0000","last_fullsized":"2024-05-12T22:07:45.725416+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T07:40:48.540839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.739297+0000","last_change":"2024-05-12T22:06:33.986598+0000","last_active":"2024-05-12T22:07:45.739297+0000","last_peered":"2024-05-12T22:07:45.739297+0000","last_clean":"2024-05-12T22:07:45.739297+0000","last_became_active":"2024-05-12T22:06:33.986346+0000","last_became_peered":"2024-05-12T22:06:33.986346+0000","last_unstale":"2024-05-12T22:07:45.739297+0000","last_undegraded":"2024-05-12T22:07:45.739297+0000","last_fullsized":"2024-05-12T22:07:45.739297+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T05:52:05.473099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"88'10","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.714677+0000","last_change":"2024-05-12T22:06:33.978900+0000","last_active":"2024-05-12T22:07:45.714677+0000","last_peered":"2024-05-12T22:07:45.714677+0000","last_clean":"2024-05-12T22:07:45.714677+0000","last_became_active":"2024-05-12T22:06:33.978474+0000","last_became_peered":"2024-05-12T22:06:33.978474+0000","last_unstale":"2024-05-12T22:07:45.714677+0000","last_undegraded":"2024-05-12T22:07:45.714677+0000","last_fullsized":"2024-05-12T22:07:45.714677+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T03:47:53.014263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.719258+0000","last_change":"2024-05-12T22:06:33.966752+0000","last_active":"2024-05-12T22:07:45.719258+0000","last_peered":"2024-05-12T22:07:45.719258+0000","last_clean":"2024-05-12T22:07:45.719258+0000","last_became_active":"2024-05-12T22:06:33.966320+0000","last_became_peered":"2024-05-12T22:06:33.966320+0000","last_unstale":"2024-05-12T22:07:45.719258+0000","last_undegraded":"2024-05-12T22:07:45.719258+0000","last_fullsized":"2024-05-12T22:07:45.719258+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T22:11:48.356156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'9","reported_seq":45,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:06:46.808696+0000","last_change":"2024-05-12T22:06:46.808696+0000","last_active":"2024-05-12T22:06:46.808696+0000","last_peered":"2024-05-12T22:06:46.808696+0000","last_clean":"2024-05-12T22:06:46.808696+0000","last_became_active":"2024-05-12T22:06:46.796654+0000","last_became_peered":"2024-05-12T22:06:46.796654+0000","last_unstale":"2024-05-12T22:06:46.808696+0000","last_undegraded":"2024-05-12T22:06:46.808696+0000","last_fullsized":"2024-05-12T22:06:46.808696+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T09:32:33.209844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,8],"acting":[9,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.715776+0000","last_change":"2024-05-12T22:06:33.963875+0000","last_active":"2024-05-12T22:07:45.715776+0000","last_peered":"2024-05-12T22:07:45.715776+0000","last_clean":"2024-05-12T22:07:45.715776+0000","last_became_active":"2024-05-12T22:06:33.963239+0000","last_became_peered":"2024-05-12T22:06:33.963239+0000","last_unstale":"2024-05-12T22:07:45.715776+0000","last_undegraded":"2024-05-12T22:07:45.715776+0000","last_fullsized":"2024-05-12T22:07:45.715776+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T23:15:01.832082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:06:46.788427+0000","last_change":"2024-05-12T22:06:32.950352+0000","last_active":"2024-05-12T22:06:46.788427+0000","last_peered":"2024-05-12T22:06:46.788427+0000","last_clean":"2024-05-12T22:06:46.788427+0000","last_became_active":"2024-05-12T22:06:32.949759+0000","last_became_peered":"2024-05-12T22:06:32.949759+0000","last_unstale":"2024-05-12T22:06:46.788427+0000","last_undegraded":"2024-05-12T22:06:46.788427+0000","last_fullsized":"2024-05-12T22:06:46.788427+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T06:28:07.613351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.737914+0000","last_change":"2024-05-12T22:06:33.990783+0000","last_active":"2024-05-12T22:07:45.737914+0000","last_peered":"2024-05-12T22:07:45.737914+0000","last_clean":"2024-05-12T22:07:45.737914+0000","last_became_active":"2024-05-12T22:06:33.990387+0000","last_became_peered":"2024-05-12T22:06:33.990387+0000","last_unstale":"2024-05-12T22:07:45.737914+0000","last_undegraded":"2024-05-12T22:07:45.737914+0000","last_fullsized":"2024-05-12T22:07:45.737914+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T05:52:52.247458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"86'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:06:46.788832+0000","last_change":"2024-05-12T22:06:32.931718+0000","last_active":"2024-05-12T22:06:46.788832+0000","last_peered":"2024-05-12T22:06:46.788832+0000","last_clean":"2024-05-12T22:06:46.788832+0000","last_became_active":"2024-05-12T22:06:32.931404+0000","last_became_peered":"2024-05-12T22:06:32.931404+0000","last_unstale":"2024-05-12T22:06:46.788832+0000","last_undegraded":"2024-05-12T22:06:46.788832+0000","last_fullsized":"2024-05-12T22:06:46.788832+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T02:55:57.770504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.720351+0000","last_change":"2024-05-12T22:06:33.963299+0000","last_active":"2024-05-12T22:07:45.720351+0000","last_peered":"2024-05-12T22:07:45.720351+0000","last_clean":"2024-05-12T22:07:45.720351+0000","last_became_active":"2024-05-12T22:06:33.962916+0000","last_became_peered":"2024-05-12T22:06:33.962916+0000","last_unstale":"2024-05-12T22:07:45.720351+0000","last_undegraded":"2024-05-12T22:07:45.720351+0000","last_fullsized":"2024-05-12T22:07:45.720351+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T05:46:32.131918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:06:46.801285+0000","last_change":"2024-05-12T22:06:46.800870+0000","last_active":"2024-05-12T22:06:46.801285+0000","last_peered":"2024-05-12T22:06:46.801285+0000","last_clean":"2024-05-12T22:06:46.801285+0000","last_became_active":"2024-05-12T22:06:46.797716+0000","last_became_peered":"2024-05-12T22:06:46.797716+0000","last_unstale":"2024-05-12T22:06:46.801285+0000","last_undegraded":"2024-05-12T22:06:46.801285+0000","last_fullsized":"2024-05-12T22:06:46.801285+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T08:31:57.557564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1d","version":"88'15","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.709001+0000","last_change":"2024-05-12T22:06:33.967926+0000","last_active":"2024-05-12T22:07:45.709001+0000","last_peered":"2024-05-12T22:07:45.709001+0000","last_clean":"2024-05-12T22:07:45.709001+0000","last_became_active":"2024-05-12T22:06:33.967588+0000","last_became_peered":"2024-05-12T22:06:33.967588+0000","last_unstale":"2024-05-12T22:07:45.709001+0000","last_undegraded":"2024-05-12T22:07:45.709001+0000","last_fullsized":"2024-05-12T22:07:45.709001+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T22:39:54.434117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"88'7","reported_seq":114,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:08:00.820450+0000","last_change":"2024-05-12T22:06:32.930347+0000","last_active":"2024-05-12T22:08:00.820450+0000","last_peered":"2024-05-12T22:08:00.820450+0000","last_clean":"2024-05-12T22:08:00.820450+0000","last_became_active":"2024-05-12T22:06:32.930083+0000","last_became_peered":"2024-05-12T22:06:32.930083+0000","last_unstale":"2024-05-12T22:08:00.820450+0000","last_undegraded":"2024-05-12T22:08:00.820450+0000","last_fullsized":"2024-05-12T22:08:00.820450+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T07:10:08.898490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1154,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":88,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.712891+0000","last_change":"2024-05-12T22:06:33.973793+0000","last_active":"2024-05-12T22:07:45.712891+0000","last_peered":"2024-05-12T22:07:45.712891+0000","last_clean":"2024-05-12T22:07:45.712891+0000","last_became_active":"2024-05-12T22:06:33.973380+0000","last_became_peered":"2024-05-12T22:06:33.973380+0000","last_unstale":"2024-05-12T22:07:45.712891+0000","last_undegraded":"2024-05-12T22:07:45.712891+0000","last_fullsized":"2024-05-12T22:07:45.712891+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T04:32:01.817780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:06:47.067177+0000","last_change":"2024-05-12T22:06:32.938215+0000","last_active":"2024-05-12T22:06:47.067177+0000","last_peered":"2024-05-12T22:06:47.067177+0000","last_clean":"2024-05-12T22:06:47.067177+0000","last_became_active":"2024-05-12T22:06:32.937939+0000","last_became_peered":"2024-05-12T22:06:32.937939+0000","last_unstale":"2024-05-12T22:06:47.067177+0000","last_undegraded":"2024-05-12T22:06:47.067177+0000","last_fullsized":"2024-05-12T22:06:47.067177+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T04:15:28.821835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.718624+0000","last_change":"2024-05-12T22:06:33.977362+0000","last_active":"2024-05-12T22:07:45.718624+0000","last_peered":"2024-05-12T22:07:45.718624+0000","last_clean":"2024-05-12T22:07:45.718624+0000","last_became_active":"2024-05-12T22:06:33.977074+0000","last_became_peered":"2024-05-12T22:06:33.977074+0000","last_unstale":"2024-05-12T22:07:45.718624+0000","last_undegraded":"2024-05-12T22:07:45.718624+0000","last_fullsized":"2024-05-12T22:07:45.718624+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T22:21:00.062780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:06:46.928947+0000","last_change":"2024-05-12T22:06:32.935508+0000","last_active":"2024-05-12T22:06:46.928947+0000","last_peered":"2024-05-12T22:06:46.928947+0000","last_clean":"2024-05-12T22:06:46.928947+0000","last_became_active":"2024-05-12T22:06:32.934760+0000","last_became_peered":"2024-05-12T22:06:32.934760+0000","last_unstale":"2024-05-12T22:06:46.928947+0000","last_undegraded":"2024-05-12T22:06:46.928947+0000","last_fullsized":"2024-05-12T22:06:46.928947+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T01:58:09.450187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.722016+0000","last_change":"2024-05-12T22:06:33.966421+0000","last_active":"2024-05-12T22:07:45.722016+0000","last_peered":"2024-05-12T22:07:45.722016+0000","last_clean":"2024-05-12T22:07:45.722016+0000","last_became_active":"2024-05-12T22:06:33.965627+0000","last_became_peered":"2024-05-12T22:06:33.965627+0000","last_unstale":"2024-05-12T22:07:45.722016+0000","last_undegraded":"2024-05-12T22:07:45.722016+0000","last_fullsized":"2024-05-12T22:07:45.722016+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T09:25:03.836082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"86'1","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:06:46.794429+0000","last_change":"2024-05-12T22:06:32.928907+0000","last_active":"2024-05-12T22:06:46.794429+0000","last_peered":"2024-05-12T22:06:46.794429+0000","last_clean":"2024-05-12T22:06:46.794429+0000","last_became_active":"2024-05-12T22:06:32.928586+0000","last_became_peered":"2024-05-12T22:06:32.928586+0000","last_unstale":"2024-05-12T22:06:46.794429+0000","last_undegraded":"2024-05-12T22:06:46.794429+0000","last_fullsized":"2024-05-12T22:06:46.794429+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T04:24:21.658957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"88'11","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.720453+0000","last_change":"2024-05-12T22:06:33.982873+0000","last_active":"2024-05-12T22:07:45.720453+0000","last_peered":"2024-05-12T22:07:45.720453+0000","last_clean":"2024-05-12T22:07:45.720453+0000","last_became_active":"2024-05-12T22:06:33.982495+0000","last_became_peered":"2024-05-12T22:06:33.982495+0000","last_unstale":"2024-05-12T22:07:45.720453+0000","last_undegraded":"2024-05-12T22:07:45.720453+0000","last_fullsized":"2024-05-12T22:07:45.720453+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T05:45:39.404282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"88'4","reported_seq":104,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:57.816041+0000","last_change":"2024-05-12T22:06:32.937716+0000","last_active":"2024-05-12T22:07:57.816041+0000","last_peered":"2024-05-12T22:07:57.816041+0000","last_clean":"2024-05-12T22:07:57.816041+0000","last_became_active":"2024-05-12T22:06:32.937418+0000","last_became_peered":"2024-05-12T22:06:32.937418+0000","last_unstale":"2024-05-12T22:07:57.816041+0000","last_undegraded":"2024-05-12T22:07:57.816041+0000","last_fullsized":"2024-05-12T22:07:57.816041+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T08:20:13.639958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":81,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"88'10","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.717249+0000","last_change":"2024-05-12T22:06:33.974702+0000","last_active":"2024-05-12T22:07:45.717249+0000","last_peered":"2024-05-12T22:07:45.717249+0000","last_clean":"2024-05-12T22:07:45.717249+0000","last_became_active":"2024-05-12T22:06:33.974363+0000","last_became_peered":"2024-05-12T22:06:33.974363+0000","last_unstale":"2024-05-12T22:07:45.717249+0000","last_undegraded":"2024-05-12T22:07:45.717249+0000","last_fullsized":"2024-05-12T22:07:45.717249+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T09:16:33.620945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"86'2","reported_seq":31,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:06:46.793331+0000","last_change":"2024-05-12T22:06:46.793331+0000","last_active":"2024-05-12T22:06:46.793331+0000","last_peered":"2024-05-12T22:06:46.793331+0000","last_clean":"2024-05-12T22:06:46.793331+0000","last_became_active":"2024-05-12T22:06:46.789984+0000","last_became_peered":"2024-05-12T22:06:46.789984+0000","last_unstale":"2024-05-12T22:06:46.793331+0000","last_undegraded":"2024-05-12T22:06:46.793331+0000","last_fullsized":"2024-05-12T22:06:46.793331+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T08:16:29.505145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.738121+0000","last_change":"2024-05-12T22:06:33.980496+0000","last_active":"2024-05-12T22:07:45.738121+0000","last_peered":"2024-05-12T22:07:45.738121+0000","last_clean":"2024-05-12T22:07:45.738121+0000","last_became_active":"2024-05-12T22:06:33.980155+0000","last_became_peered":"2024-05-12T22:06:33.980155+0000","last_unstale":"2024-05-12T22:07:45.738121+0000","last_undegraded":"2024-05-12T22:07:45.738121+0000","last_fullsized":"2024-05-12T22:07:45.738121+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T01:54:03.831828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"86'4","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:06:46.795752+0000","last_change":"2024-05-12T22:06:32.951633+0000","last_active":"2024-05-12T22:06:46.795752+0000","last_peered":"2024-05-12T22:06:46.795752+0000","last_clean":"2024-05-12T22:06:46.795752+0000","last_became_active":"2024-05-12T22:06:32.951382+0000","last_became_peered":"2024-05-12T22:06:32.951382+0000","last_unstale":"2024-05-12T22:06:46.795752+0000","last_undegraded":"2024-05-12T22:06:46.795752+0000","last_fullsized":"2024-05-12T22:06:46.795752+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T02:06:21.408593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"88'11","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.721698+0000","last_change":"2024-05-12T22:06:33.963493+0000","last_active":"2024-05-12T22:07:45.721698+0000","last_peered":"2024-05-12T22:07:45.721698+0000","last_clean":"2024-05-12T22:07:45.721698+0000","last_became_active":"2024-05-12T22:06:33.963127+0000","last_became_peered":"2024-05-12T22:06:33.963127+0000","last_unstale":"2024-05-12T22:07:45.721698+0000","last_undegraded":"2024-05-12T22:07:45.721698+0000","last_fullsized":"2024-05-12T22:07:45.721698+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T05:39:31.038446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"88'33","reported_seq":224,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:58.816916+0000","last_change":"2024-05-12T22:06:32.932026+0000","last_active":"2024-05-12T22:07:58.816916+0000","last_peered":"2024-05-12T22:07:58.816916+0000","last_clean":"2024-05-12T22:07:58.816916+0000","last_became_active":"2024-05-12T22:06:32.931667+0000","last_became_peered":"2024-05-12T22:06:32.931667+0000","last_unstale":"2024-05-12T22:07:58.816916+0000","last_undegraded":"2024-05-12T22:07:58.816916+0000","last_fullsized":"2024-05-12T22:07:58.816916+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T06:16:28.259047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61122,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":172,"num_read_kb":146,"num_write":34,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.693378+0000","last_change":"2024-05-12T22:06:33.962677+0000","last_active":"2024-05-12T22:07:45.693378+0000","last_peered":"2024-05-12T22:07:45.693378+0000","last_clean":"2024-05-12T22:07:45.693378+0000","last_became_active":"2024-05-12T22:06:33.962295+0000","last_became_peered":"2024-05-12T22:06:33.962295+0000","last_unstale":"2024-05-12T22:07:45.693378+0000","last_undegraded":"2024-05-12T22:07:45.693378+0000","last_fullsized":"2024-05-12T22:07:45.693378+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T01:31:04.587046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"86'6","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:06:46.789562+0000","last_change":"2024-05-12T22:06:32.936104+0000","last_active":"2024-05-12T22:06:46.789562+0000","last_peered":"2024-05-12T22:06:46.789562+0000","last_clean":"2024-05-12T22:06:46.789562+0000","last_became_active":"2024-05-12T22:06:32.935701+0000","last_became_peered":"2024-05-12T22:06:32.935701+0000","last_unstale":"2024-05-12T22:06:46.789562+0000","last_undegraded":"2024-05-12T22:06:46.789562+0000","last_fullsized":"2024-05-12T22:06:46.789562+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T07:04:10.115302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.723098+0000","last_change":"2024-05-12T22:06:33.977850+0000","last_active":"2024-05-12T22:07:45.723098+0000","last_peered":"2024-05-12T22:07:45.723098+0000","last_clean":"2024-05-12T22:07:45.723098+0000","last_became_active":"2024-05-12T22:06:33.977511+0000","last_became_peered":"2024-05-12T22:06:33.977511+0000","last_unstale":"2024-05-12T22:07:45.723098+0000","last_undegraded":"2024-05-12T22:07:45.723098+0000","last_fullsized":"2024-05-12T22:07:45.723098+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T23:10:17.267823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:06:46.795258+0000","last_change":"2024-05-12T22:06:32.929955+0000","last_active":"2024-05-12T22:06:46.795258+0000","last_peered":"2024-05-12T22:06:46.795258+0000","last_clean":"2024-05-12T22:06:46.795258+0000","last_became_active":"2024-05-12T22:06:32.929623+0000","last_became_peered":"2024-05-12T22:06:32.929623+0000","last_unstale":"2024-05-12T22:06:46.795258+0000","last_undegraded":"2024-05-12T22:06:46.795258+0000","last_fullsized":"2024-05-12T22:06:46.795258+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T08:15:22.622908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"88'15","reported_seq":38,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.725455+0000","last_change":"2024-05-12T22:06:46.931736+0000","last_active":"2024-05-12T22:07:45.725455+0000","last_peered":"2024-05-12T22:07:45.725455+0000","last_clean":"2024-05-12T22:07:45.725455+0000","last_became_active":"2024-05-12T22:06:46.931265+0000","last_became_peered":"2024-05-12T22:06:46.931265+0000","last_unstale":"2024-05-12T22:07:45.725455+0000","last_undegraded":"2024-05-12T22:07:45.725455+0000","last_fullsized":"2024-05-12T22:07:45.725455+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T02:25:34.002024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"88'5","reported_seq":109,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:57.877999+0000","last_change":"2024-05-12T22:06:32.927707+0000","last_active":"2024-05-12T22:07:57.877999+0000","last_peered":"2024-05-12T22:07:57.877999+0000","last_clean":"2024-05-12T22:07:57.877999+0000","last_became_active":"2024-05-12T22:06:32.927406+0000","last_became_peered":"2024-05-12T22:06:32.927406+0000","last_unstale":"2024-05-12T22:07:57.877999+0000","last_undegraded":"2024-05-12T22:07:57.877999+0000","last_fullsized":"2024-05-12T22:07:57.877999+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T05:14:34.268589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1120,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":5,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.691438+0000","last_change":"2024-05-12T22:06:33.964996+0000","last_active":"2024-05-12T22:07:45.691438+0000","last_peered":"2024-05-12T22:07:45.691438+0000","last_clean":"2024-05-12T22:07:45.691438+0000","last_became_active":"2024-05-12T22:06:33.964587+0000","last_became_peered":"2024-05-12T22:06:33.964587+0000","last_unstale":"2024-05-12T22:07:45.691438+0000","last_undegraded":"2024-05-12T22:07:45.691438+0000","last_fullsized":"2024-05-12T22:07:45.691438+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T22:27:28.316377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"86'15","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:06:46.788234+0000","last_change":"2024-05-12T22:06:32.935543+0000","last_active":"2024-05-12T22:06:46.788234+0000","last_peered":"2024-05-12T22:06:46.788234+0000","last_clean":"2024-05-12T22:06:46.788234+0000","last_became_active":"2024-05-12T22:06:32.935145+0000","last_became_peered":"2024-05-12T22:06:32.935145+0000","last_unstale":"2024-05-12T22:06:46.788234+0000","last_undegraded":"2024-05-12T22:06:46.788234+0000","last_fullsized":"2024-05-12T22:06:46.788234+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T01:13:23.289057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.702680+0000","last_change":"2024-05-12T22:06:33.964367+0000","last_active":"2024-05-12T22:07:45.702680+0000","last_peered":"2024-05-12T22:07:45.702680+0000","last_clean":"2024-05-12T22:07:45.702680+0000","last_became_active":"2024-05-12T22:06:33.963876+0000","last_became_peered":"2024-05-12T22:06:33.963876+0000","last_unstale":"2024-05-12T22:07:45.702680+0000","last_undegraded":"2024-05-12T22:07:45.702680+0000","last_fullsized":"2024-05-12T22:07:45.702680+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T07:09:20.962965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"86'2","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:06:46.788927+0000","last_change":"2024-05-12T22:06:32.927270+0000","last_active":"2024-05-12T22:06:46.788927+0000","last_peered":"2024-05-12T22:06:46.788927+0000","last_clean":"2024-05-12T22:06:46.788927+0000","last_became_active":"2024-05-12T22:06:32.926930+0000","last_became_peered":"2024-05-12T22:06:32.926930+0000","last_unstale":"2024-05-12T22:06:46.788927+0000","last_undegraded":"2024-05-12T22:06:46.788927+0000","last_fullsized":"2024-05-12T22:06:46.788927+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T07:04:48.336599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.695444+0000","last_change":"2024-05-12T22:06:33.966350+0000","last_active":"2024-05-12T22:07:45.695444+0000","last_peered":"2024-05-12T22:07:45.695444+0000","last_clean":"2024-05-12T22:07:45.695444+0000","last_became_active":"2024-05-12T22:06:33.965974+0000","last_became_peered":"2024-05-12T22:06:33.965974+0000","last_unstale":"2024-05-12T22:07:45.695444+0000","last_undegraded":"2024-05-12T22:07:45.695444+0000","last_fullsized":"2024-05-12T22:07:45.695444+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T02:08:16.889684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:06:46.792377+0000","last_change":"2024-05-12T22:06:32.944945+0000","last_active":"2024-05-12T22:06:46.792377+0000","last_peered":"2024-05-12T22:06:46.792377+0000","last_clean":"2024-05-12T22:06:46.792377+0000","last_became_active":"2024-05-12T22:06:32.944576+0000","last_became_peered":"2024-05-12T22:06:32.944576+0000","last_unstale":"2024-05-12T22:06:46.792377+0000","last_undegraded":"2024-05-12T22:06:46.792377+0000","last_fullsized":"2024-05-12T22:06:46.792377+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T07:29:14.544821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.720934+0000","last_change":"2024-05-12T22:06:33.966299+0000","last_active":"2024-05-12T22:07:45.720934+0000","last_peered":"2024-05-12T22:07:45.720934+0000","last_clean":"2024-05-12T22:07:45.720934+0000","last_became_active":"2024-05-12T22:06:33.965902+0000","last_became_peered":"2024-05-12T22:06:33.965902+0000","last_unstale":"2024-05-12T22:07:45.720934+0000","last_undegraded":"2024-05-12T22:07:45.720934+0000","last_fullsized":"2024-05-12T22:07:45.720934+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T22:37:51.476546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"86'1","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:06:46.786408+0000","last_change":"2024-05-12T22:06:32.937210+0000","last_active":"2024-05-12T22:06:46.786408+0000","last_peered":"2024-05-12T22:06:46.786408+0000","last_clean":"2024-05-12T22:06:46.786408+0000","last_became_active":"2024-05-12T22:06:32.936805+0000","last_became_peered":"2024-05-12T22:06:32.936805+0000","last_unstale":"2024-05-12T22:06:46.786408+0000","last_undegraded":"2024-05-12T22:06:46.786408+0000","last_fullsized":"2024-05-12T22:06:46.786408+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T06:14:20.825258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.692294+0000","last_change":"2024-05-12T22:06:33.983604+0000","last_active":"2024-05-12T22:07:45.692294+0000","last_peered":"2024-05-12T22:07:45.692294+0000","last_clean":"2024-05-12T22:07:45.692294+0000","last_became_active":"2024-05-12T22:06:33.983168+0000","last_became_peered":"2024-05-12T22:06:33.983168+0000","last_unstale":"2024-05-12T22:07:45.692294+0000","last_undegraded":"2024-05-12T22:07:45.692294+0000","last_fullsized":"2024-05-12T22:07:45.692294+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T05:54:55.854593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.704701+0000","last_change":"2024-05-12T22:06:33.968848+0000","last_active":"2024-05-12T22:07:45.704701+0000","last_peered":"2024-05-12T22:07:45.704701+0000","last_clean":"2024-05-12T22:07:45.704701+0000","last_became_active":"2024-05-12T22:06:33.968529+0000","last_became_peered":"2024-05-12T22:06:33.968529+0000","last_unstale":"2024-05-12T22:07:45.704701+0000","last_undegraded":"2024-05-12T22:07:45.704701+0000","last_fullsized":"2024-05-12T22:07:45.704701+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T03:00:41.129662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"88'5","reported_seq":108,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:08:00.883223+0000","last_change":"2024-05-12T22:06:32.946670+0000","last_active":"2024-05-12T22:08:00.883223+0000","last_peered":"2024-05-12T22:08:00.883223+0000","last_clean":"2024-05-12T22:08:00.883223+0000","last_became_active":"2024-05-12T22:06:32.946059+0000","last_became_peered":"2024-05-12T22:06:32.946059+0000","last_unstale":"2024-05-12T22:08:00.883223+0000","last_undegraded":"2024-05-12T22:08:00.883223+0000","last_fullsized":"2024-05-12T22:08:00.883223+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T22:35:56.116979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":84,"num_read_kb":84,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.720948+0000","last_change":"2024-05-12T22:06:33.962177+0000","last_active":"2024-05-12T22:07:45.720948+0000","last_peered":"2024-05-12T22:07:45.720948+0000","last_clean":"2024-05-12T22:07:45.720948+0000","last_became_active":"2024-05-12T22:06:33.960919+0000","last_became_peered":"2024-05-12T22:06:33.960919+0000","last_unstale":"2024-05-12T22:07:45.720948+0000","last_undegraded":"2024-05-12T22:07:45.720948+0000","last_fullsized":"2024-05-12T22:07:45.720948+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T00:39:54.566460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'3","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:06:46.784690+0000","last_change":"2024-05-12T22:06:32.927885+0000","last_active":"2024-05-12T22:06:46.784690+0000","last_peered":"2024-05-12T22:06:46.784690+0000","last_clean":"2024-05-12T22:06:46.784690+0000","last_became_active":"2024-05-12T22:06:32.927567+0000","last_became_peered":"2024-05-12T22:06:32.927567+0000","last_unstale":"2024-05-12T22:06:46.784690+0000","last_undegraded":"2024-05-12T22:06:46.784690+0000","last_fullsized":"2024-05-12T22:06:46.784690+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T07:13:50.811992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":353,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:06:46.786385+0000","last_change":"2024-05-12T22:05:17.683363+0000","last_active":"2024-05-12T22:06:46.786385+0000","last_peered":"2024-05-12T22:06:46.786385+0000","last_clean":"2024-05-12T22:06:46.786385+0000","last_became_active":"2024-05-12T22:05:17.682824+0000","last_became_peered":"2024-05-12T22:05:17.682824+0000","last_unstale":"2024-05-12T22:06:46.786385+0000","last_undegraded":"2024-05-12T22:06:46.786385+0000","last_fullsized":"2024-05-12T22:06:46.786385+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:01:32.304714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:01:32.304714+0000","last_clean_scrub_stamp":"2024-05-12T22:01:32.304714+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T06:35:22.474820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.714366+0000","last_change":"2024-05-12T22:06:33.965332+0000","last_active":"2024-05-12T22:07:45.714366+0000","last_peered":"2024-05-12T22:07:45.714366+0000","last_clean":"2024-05-12T22:07:45.714366+0000","last_became_active":"2024-05-12T22:06:33.964858+0000","last_became_peered":"2024-05-12T22:06:33.964858+0000","last_unstale":"2024-05-12T22:07:45.714366+0000","last_undegraded":"2024-05-12T22:07:45.714366+0000","last_fullsized":"2024-05-12T22:07:45.714366+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T05:14:09.825614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:06:46.786368+0000","last_change":"2024-05-12T22:06:32.929773+0000","last_active":"2024-05-12T22:06:46.786368+0000","last_peered":"2024-05-12T22:06:46.786368+0000","last_clean":"2024-05-12T22:06:46.786368+0000","last_became_active":"2024-05-12T22:06:32.929458+0000","last_became_peered":"2024-05-12T22:06:32.929458+0000","last_unstale":"2024-05-12T22:06:46.786368+0000","last_undegraded":"2024-05-12T22:06:46.786368+0000","last_fullsized":"2024-05-12T22:06:46.786368+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T01:30:42.853183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"86'1","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:06:46.788861+0000","last_change":"2024-05-12T22:06:32.947014+0000","last_active":"2024-05-12T22:06:46.788861+0000","last_peered":"2024-05-12T22:06:46.788861+0000","last_clean":"2024-05-12T22:06:46.788861+0000","last_became_active":"2024-05-12T22:06:32.946661+0000","last_became_peered":"2024-05-12T22:06:32.946661+0000","last_unstale":"2024-05-12T22:06:46.788861+0000","last_undegraded":"2024-05-12T22:06:46.788861+0000","last_fullsized":"2024-05-12T22:06:46.788861+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T23:09:19.378353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.715360+0000","last_change":"2024-05-12T22:06:33.980490+0000","last_active":"2024-05-12T22:07:45.715360+0000","last_peered":"2024-05-12T22:07:45.715360+0000","last_clean":"2024-05-12T22:07:45.715360+0000","last_became_active":"2024-05-12T22:06:33.980133+0000","last_became_peered":"2024-05-12T22:06:33.980133+0000","last_unstale":"2024-05-12T22:07:45.715360+0000","last_undegraded":"2024-05-12T22:07:45.715360+0000","last_fullsized":"2024-05-12T22:07:45.715360+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T00:28:10.461428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"88'11","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:06:48.270662+0000","last_change":"2024-05-12T22:06:32.943152+0000","last_active":"2024-05-12T22:06:48.270662+0000","last_peered":"2024-05-12T22:06:48.270662+0000","last_clean":"2024-05-12T22:06:48.270662+0000","last_became_active":"2024-05-12T22:06:32.942769+0000","last_became_peered":"2024-05-12T22:06:32.942769+0000","last_unstale":"2024-05-12T22:06:48.270662+0000","last_undegraded":"2024-05-12T22:06:48.270662+0000","last_fullsized":"2024-05-12T22:06:48.270662+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T01:23:24.955739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.719134+0000","last_change":"2024-05-12T22:06:33.964081+0000","last_active":"2024-05-12T22:07:45.719134+0000","last_peered":"2024-05-12T22:07:45.719134+0000","last_clean":"2024-05-12T22:07:45.719134+0000","last_became_active":"2024-05-12T22:06:33.963054+0000","last_became_peered":"2024-05-12T22:06:33.963054+0000","last_unstale":"2024-05-12T22:07:45.719134+0000","last_undegraded":"2024-05-12T22:07:45.719134+0000","last_fullsized":"2024-05-12T22:07:45.719134+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T09:40:03.396200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"86'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:06:46.785077+0000","last_change":"2024-05-12T22:06:32.936432+0000","last_active":"2024-05-12T22:06:46.785077+0000","last_peered":"2024-05-12T22:06:46.785077+0000","last_clean":"2024-05-12T22:06:46.785077+0000","last_became_active":"2024-05-12T22:06:32.936094+0000","last_became_peered":"2024-05-12T22:06:32.936094+0000","last_unstale":"2024-05-12T22:06:46.785077+0000","last_undegraded":"2024-05-12T22:06:46.785077+0000","last_fullsized":"2024-05-12T22:06:46.785077+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T02:40:31.346118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.733892+0000","last_change":"2024-05-12T22:06:33.985953+0000","last_active":"2024-05-12T22:07:45.733892+0000","last_peered":"2024-05-12T22:07:45.733892+0000","last_clean":"2024-05-12T22:07:45.733892+0000","last_became_active":"2024-05-12T22:06:33.984864+0000","last_became_peered":"2024-05-12T22:06:33.984864+0000","last_unstale":"2024-05-12T22:07:45.733892+0000","last_undegraded":"2024-05-12T22:07:45.733892+0000","last_fullsized":"2024-05-12T22:07:45.733892+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T07:28:22.357841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:06:46.788690+0000","last_change":"2024-05-12T22:06:32.934510+0000","last_active":"2024-05-12T22:06:46.788690+0000","last_peered":"2024-05-12T22:06:46.788690+0000","last_clean":"2024-05-12T22:06:46.788690+0000","last_became_active":"2024-05-12T22:06:32.934222+0000","last_became_peered":"2024-05-12T22:06:32.934222+0000","last_unstale":"2024-05-12T22:06:46.788690+0000","last_undegraded":"2024-05-12T22:06:46.788690+0000","last_fullsized":"2024-05-12T22:06:46.788690+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T00:57:44.321629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"88'2","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.670036+0000","last_change":"2024-05-12T22:06:33.963870+0000","last_active":"2024-05-12T22:07:45.670036+0000","last_peered":"2024-05-12T22:07:45.670036+0000","last_clean":"2024-05-12T22:07:45.670036+0000","last_became_active":"2024-05-12T22:06:33.963533+0000","last_became_peered":"2024-05-12T22:06:33.963533+0000","last_unstale":"2024-05-12T22:07:45.670036+0000","last_undegraded":"2024-05-12T22:07:45.670036+0000","last_fullsized":"2024-05-12T22:07:45.670036+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T08:56:59.495469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:06:46.788300+0000","last_change":"2024-05-12T22:06:32.939738+0000","last_active":"2024-05-12T22:06:46.788300+0000","last_peered":"2024-05-12T22:06:46.788300+0000","last_clean":"2024-05-12T22:06:46.788300+0000","last_became_active":"2024-05-12T22:06:32.939365+0000","last_became_peered":"2024-05-12T22:06:32.939365+0000","last_unstale":"2024-05-12T22:06:46.788300+0000","last_undegraded":"2024-05-12T22:06:46.788300+0000","last_fullsized":"2024-05-12T22:06:46.788300+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T02:19:31.180834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.736281+0000","last_change":"2024-05-12T22:06:33.975064+0000","last_active":"2024-05-12T22:07:45.736281+0000","last_peered":"2024-05-12T22:07:45.736281+0000","last_clean":"2024-05-12T22:07:45.736281+0000","last_became_active":"2024-05-12T22:06:33.974613+0000","last_became_peered":"2024-05-12T22:06:33.974613+0000","last_unstale":"2024-05-12T22:07:45.736281+0000","last_undegraded":"2024-05-12T22:07:45.736281+0000","last_fullsized":"2024-05-12T22:07:45.736281+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T08:46:27.343777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"86'1","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:06:46.789708+0000","last_change":"2024-05-12T22:06:32.931148+0000","last_active":"2024-05-12T22:06:46.789708+0000","last_peered":"2024-05-12T22:06:46.789708+0000","last_clean":"2024-05-12T22:06:46.789708+0000","last_became_active":"2024-05-12T22:06:32.930717+0000","last_became_peered":"2024-05-12T22:06:32.930717+0000","last_unstale":"2024-05-12T22:06:46.789708+0000","last_undegraded":"2024-05-12T22:06:46.789708+0000","last_fullsized":"2024-05-12T22:06:46.789708+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T07:26:20.915467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.736548+0000","last_change":"2024-05-12T22:06:33.988390+0000","last_active":"2024-05-12T22:07:45.736548+0000","last_peered":"2024-05-12T22:07:45.736548+0000","last_clean":"2024-05-12T22:07:45.736548+0000","last_became_active":"2024-05-12T22:06:33.986970+0000","last_became_peered":"2024-05-12T22:06:33.986970+0000","last_unstale":"2024-05-12T22:07:45.736548+0000","last_undegraded":"2024-05-12T22:07:45.736548+0000","last_fullsized":"2024-05-12T22:07:45.736548+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T05:03:31.930261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":36,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:06:46.798123+0000","last_change":"2024-05-12T22:06:46.798123+0000","last_active":"2024-05-12T22:06:46.798123+0000","last_peered":"2024-05-12T22:06:46.798123+0000","last_clean":"2024-05-12T22:06:46.798123+0000","last_became_active":"2024-05-12T22:06:46.791029+0000","last_became_peered":"2024-05-12T22:06:46.791029+0000","last_unstale":"2024-05-12T22:06:46.798123+0000","last_undegraded":"2024-05-12T22:06:46.798123+0000","last_fullsized":"2024-05-12T22:06:46.798123+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T06:31:25.108659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,7],"acting":[8,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.720460+0000","last_change":"2024-05-12T22:06:33.968256+0000","last_active":"2024-05-12T22:07:45.720460+0000","last_peered":"2024-05-12T22:07:45.720460+0000","last_clean":"2024-05-12T22:07:45.720460+0000","last_became_active":"2024-05-12T22:06:33.967885+0000","last_became_peered":"2024-05-12T22:06:33.967885+0000","last_unstale":"2024-05-12T22:07:45.720460+0000","last_undegraded":"2024-05-12T22:07:45.720460+0000","last_fullsized":"2024-05-12T22:07:45.720460+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T09:12:40.687367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"86'3","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:06:46.789849+0000","last_change":"2024-05-12T22:06:32.944736+0000","last_active":"2024-05-12T22:06:46.789849+0000","last_peered":"2024-05-12T22:06:46.789849+0000","last_clean":"2024-05-12T22:06:46.789849+0000","last_became_active":"2024-05-12T22:06:32.944269+0000","last_became_peered":"2024-05-12T22:06:32.944269+0000","last_unstale":"2024-05-12T22:06:46.789849+0000","last_undegraded":"2024-05-12T22:06:46.789849+0000","last_fullsized":"2024-05-12T22:06:46.789849+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T22:56:24.391267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.721036+0000","last_change":"2024-05-12T22:06:33.968360+0000","last_active":"2024-05-12T22:07:45.721036+0000","last_peered":"2024-05-12T22:07:45.721036+0000","last_clean":"2024-05-12T22:07:45.721036+0000","last_became_active":"2024-05-12T22:06:33.967967+0000","last_became_peered":"2024-05-12T22:06:33.967967+0000","last_unstale":"2024-05-12T22:07:45.721036+0000","last_undegraded":"2024-05-12T22:07:45.721036+0000","last_fullsized":"2024-05-12T22:07:45.721036+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T00:41:12.693858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"85'1","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:06:46.794591+0000","last_change":"2024-05-12T22:06:32.935993+0000","last_active":"2024-05-12T22:06:46.794591+0000","last_peered":"2024-05-12T22:06:46.794591+0000","last_clean":"2024-05-12T22:06:46.794591+0000","last_became_active":"2024-05-12T22:06:32.935615+0000","last_became_peered":"2024-05-12T22:06:32.935615+0000","last_unstale":"2024-05-12T22:06:46.794591+0000","last_undegraded":"2024-05-12T22:06:46.794591+0000","last_fullsized":"2024-05-12T22:06:46.794591+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T00:43:39.254637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.708035+0000","last_change":"2024-05-12T22:06:33.966110+0000","last_active":"2024-05-12T22:07:45.708035+0000","last_peered":"2024-05-12T22:07:45.708035+0000","last_clean":"2024-05-12T22:07:45.708035+0000","last_became_active":"2024-05-12T22:06:33.965755+0000","last_became_peered":"2024-05-12T22:06:33.965755+0000","last_unstale":"2024-05-12T22:07:45.708035+0000","last_undegraded":"2024-05-12T22:07:45.708035+0000","last_fullsized":"2024-05-12T22:07:45.708035+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T22:18:53.941335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:06:46.929070+0000","last_change":"2024-05-12T22:06:32.942229+0000","last_active":"2024-05-12T22:06:46.929070+0000","last_peered":"2024-05-12T22:06:46.929070+0000","last_clean":"2024-05-12T22:06:46.929070+0000","last_became_active":"2024-05-12T22:06:32.941826+0000","last_became_peered":"2024-05-12T22:06:32.941826+0000","last_unstale":"2024-05-12T22:06:46.929070+0000","last_undegraded":"2024-05-12T22:06:46.929070+0000","last_fullsized":"2024-05-12T22:06:46.929070+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T03:52:34.886318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.695139+0000","last_change":"2024-05-12T22:06:33.963081+0000","last_active":"2024-05-12T22:07:45.695139+0000","last_peered":"2024-05-12T22:07:45.695139+0000","last_clean":"2024-05-12T22:07:45.695139+0000","last_became_active":"2024-05-12T22:06:33.962696+0000","last_became_peered":"2024-05-12T22:06:33.962696+0000","last_unstale":"2024-05-12T22:07:45.695139+0000","last_undegraded":"2024-05-12T22:07:45.695139+0000","last_fullsized":"2024-05-12T22:07:45.695139+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T05:55:45.581823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.698920+0000","last_change":"2024-05-12T22:06:33.968865+0000","last_active":"2024-05-12T22:07:45.698920+0000","last_peered":"2024-05-12T22:07:45.698920+0000","last_clean":"2024-05-12T22:07:45.698920+0000","last_became_active":"2024-05-12T22:06:33.968185+0000","last_became_peered":"2024-05-12T22:06:33.968185+0000","last_unstale":"2024-05-12T22:07:45.698920+0000","last_undegraded":"2024-05-12T22:07:45.698920+0000","last_fullsized":"2024-05-12T22:07:45.698920+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T01:23:41.083778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"88'12","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.704125+0000","last_change":"2024-05-12T22:06:33.964786+0000","last_active":"2024-05-12T22:07:45.704125+0000","last_peered":"2024-05-12T22:07:45.704125+0000","last_clean":"2024-05-12T22:07:45.704125+0000","last_became_active":"2024-05-12T22:06:33.964227+0000","last_became_peered":"2024-05-12T22:06:33.964227+0000","last_unstale":"2024-05-12T22:07:45.704125+0000","last_undegraded":"2024-05-12T22:07:45.704125+0000","last_fullsized":"2024-05-12T22:07:45.704125+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T02:53:42.739455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.717643+0000","last_change":"2024-05-12T22:06:33.978857+0000","last_active":"2024-05-12T22:07:45.717643+0000","last_peered":"2024-05-12T22:07:45.717643+0000","last_clean":"2024-05-12T22:07:45.717643+0000","last_became_active":"2024-05-12T22:06:33.978531+0000","last_became_peered":"2024-05-12T22:06:33.978531+0000","last_unstale":"2024-05-12T22:07:45.717643+0000","last_undegraded":"2024-05-12T22:07:45.717643+0000","last_fullsized":"2024-05-12T22:07:45.717643+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T02:51:05.609267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.733111+0000","last_change":"2024-05-12T22:06:33.980662+0000","last_active":"2024-05-12T22:07:45.733111+0000","last_peered":"2024-05-12T22:07:45.733111+0000","last_clean":"2024-05-12T22:07:45.733111+0000","last_became_active":"2024-05-12T22:06:33.980332+0000","last_became_peered":"2024-05-12T22:06:33.980332+0000","last_unstale":"2024-05-12T22:07:45.733111+0000","last_undegraded":"2024-05-12T22:07:45.733111+0000","last_fullsized":"2024-05-12T22:07:45.733111+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T06:21:59.715792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"88'4","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.737927+0000","last_change":"2024-05-12T22:06:33.987338+0000","last_active":"2024-05-12T22:07:45.737927+0000","last_peered":"2024-05-12T22:07:45.737927+0000","last_clean":"2024-05-12T22:07:45.737927+0000","last_became_active":"2024-05-12T22:06:33.986942+0000","last_became_peered":"2024-05-12T22:06:33.986942+0000","last_unstale":"2024-05-12T22:07:45.737927+0000","last_undegraded":"2024-05-12T22:07:45.737927+0000","last_fullsized":"2024-05-12T22:07:45.737927+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T01:02:05.137558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"88'6","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.711410+0000","last_change":"2024-05-12T22:06:46.796752+0000","last_active":"2024-05-12T22:07:45.711410+0000","last_peered":"2024-05-12T22:07:45.711410+0000","last_clean":"2024-05-12T22:07:45.711410+0000","last_became_active":"2024-05-12T22:06:46.796369+0000","last_became_peered":"2024-05-12T22:06:46.796369+0000","last_unstale":"2024-05-12T22:07:45.711410+0000","last_undegraded":"2024-05-12T22:07:45.711410+0000","last_fullsized":"2024-05-12T22:07:45.711410+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T08:23:24.685495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,9],"acting":[11,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"88'15","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.714582+0000","last_change":"2024-05-12T22:06:33.963902+0000","last_active":"2024-05-12T22:07:45.714582+0000","last_peered":"2024-05-12T22:07:45.714582+0000","last_clean":"2024-05-12T22:07:45.714582+0000","last_became_active":"2024-05-12T22:06:33.963436+0000","last_became_peered":"2024-05-12T22:06:33.963436+0000","last_unstale":"2024-05-12T22:07:45.714582+0000","last_undegraded":"2024-05-12T22:07:45.714582+0000","last_fullsized":"2024-05-12T22:07:45.714582+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T02:51:56.415167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"88'2","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.707384+0000","last_change":"2024-05-12T22:06:46.794434+0000","last_active":"2024-05-12T22:07:45.707384+0000","last_peered":"2024-05-12T22:07:45.707384+0000","last_clean":"2024-05-12T22:07:45.707384+0000","last_became_active":"2024-05-12T22:06:46.793147+0000","last_became_peered":"2024-05-12T22:06:46.793147+0000","last_unstale":"2024-05-12T22:07:45.707384+0000","last_undegraded":"2024-05-12T22:07:45.707384+0000","last_fullsized":"2024-05-12T22:07:45.707384+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T23:10:39.529863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,8,2],"acting":[3,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.718420+0000","last_change":"2024-05-12T22:06:33.973490+0000","last_active":"2024-05-12T22:07:45.718420+0000","last_peered":"2024-05-12T22:07:45.718420+0000","last_clean":"2024-05-12T22:07:45.718420+0000","last_became_active":"2024-05-12T22:06:33.973161+0000","last_became_peered":"2024-05-12T22:06:33.973161+0000","last_unstale":"2024-05-12T22:07:45.718420+0000","last_undegraded":"2024-05-12T22:07:45.718420+0000","last_fullsized":"2024-05-12T22:07:45.718420+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T03:29:36.972631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.721084+0000","last_change":"2024-05-12T22:06:33.957320+0000","last_active":"2024-05-12T22:07:45.721084+0000","last_peered":"2024-05-12T22:07:45.721084+0000","last_clean":"2024-05-12T22:07:45.721084+0000","last_became_active":"2024-05-12T22:06:33.956837+0000","last_became_peered":"2024-05-12T22:06:33.956837+0000","last_unstale":"2024-05-12T22:07:45.721084+0000","last_undegraded":"2024-05-12T22:07:45.721084+0000","last_fullsized":"2024-05-12T22:07:45.721084+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T01:20:51.018175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"88'10","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.722857+0000","last_change":"2024-05-12T22:06:33.968363+0000","last_active":"2024-05-12T22:07:45.722857+0000","last_peered":"2024-05-12T22:07:45.722857+0000","last_clean":"2024-05-12T22:07:45.722857+0000","last_became_active":"2024-05-12T22:06:33.968031+0000","last_became_peered":"2024-05-12T22:06:33.968031+0000","last_unstale":"2024-05-12T22:07:45.722857+0000","last_undegraded":"2024-05-12T22:07:45.722857+0000","last_fullsized":"2024-05-12T22:07:45.722857+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T00:14:07.124058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"88'11","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.738418+0000","last_change":"2024-05-12T22:06:33.984823+0000","last_active":"2024-05-12T22:07:45.738418+0000","last_peered":"2024-05-12T22:07:45.738418+0000","last_clean":"2024-05-12T22:07:45.738418+0000","last_became_active":"2024-05-12T22:06:33.984430+0000","last_became_peered":"2024-05-12T22:06:33.984430+0000","last_unstale":"2024-05-12T22:07:45.738418+0000","last_undegraded":"2024-05-12T22:07:45.738418+0000","last_fullsized":"2024-05-12T22:07:45.738418+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T01:30:18.109265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-05-12T22:07:45.693058+0000","last_change":"2024-05-12T22:06:33.962660+0000","last_active":"2024-05-12T22:07:45.693058+0000","last_peered":"2024-05-12T22:07:45.693058+0000","last_clean":"2024-05-12T22:07:45.693058+0000","last_became_active":"2024-05-12T22:06:33.962284+0000","last_became_peered":"2024-05-12T22:06:33.962284+0000","last_unstale":"2024-05-12T22:07:45.693058+0000","last_undegraded":"2024-05-12T22:07:45.693058+0000","last_fullsized":"2024-05-12T22:07:45.693058+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T01:08:53.888992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":65796,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":524,"num_read_kb":333,"num_write":172,"num_write_kb":120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":197388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":148,"ondisk_log_size":148,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351007,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27948,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717204,"statfs":{"total":95995035648,"available":95966416896,"internally_reserved":0,"allocated":954368,"data_stored":459830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612643,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28624,"kb_used_data":1608,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716528,"statfs":{"total":95995035648,"available":95965724672,"internally_reserved":0,"allocated":1646592,"data_stored":1112383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776169,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":966656,"data_stored":459484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037808,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32120,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713032,"statfs":{"total":95995035648,"available":95962144768,"internally_reserved":0,"allocated":966656,"data_stored":459404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299444,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28536,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716616,"statfs":{"total":95995035648,"available":95965814784,"internally_reserved":0,"allocated":1556480,"data_stored":1051425,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462971,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27948,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717204,"statfs":{"total":95995035648,"available":95966416896,"internally_reserved":0,"allocated":954368,"data_stored":458873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574934,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36328,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708824,"statfs":{"total":95995035648,"available":95957835776,"internally_reserved":0,"allocated":1015808,"data_stored":519788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738459,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":974848,"data_stored":460144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411409,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28032,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717120,"statfs":{"total":95995035648,"available":95966330880,"internally_reserved":0,"allocated":1040384,"data_stored":521670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149771,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":970752,"data_stored":459184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888132,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27952,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717200,"statfs":{"total":95995035648,"available":95966412800,"internally_reserved":0,"allocated":958464,"data_stored":459842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724608,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28552,"kb_used_data":1536,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716600,"statfs":{"total":95995035648,"available":95965798400,"internally_reserved":0,"allocated":1572864,"data_stored":1051159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":61122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":63004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":63232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-12T22:08:05.226 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-05-12T22:08:05.226 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-05-12T22:08:05.226 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-05-12T22:08:05.226 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f72fecff68e1d400c4568684327c900485c20d6a shell --fsid 6b92e7b4-10aa-11ef-bc98-c7b262605968 -- ceph health --format=json 2024-05-12T22:08:05.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:04 smithi086 ceph-mon[34297]: pgmap v357: 97 pgs: 97 active+clean; 641 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-12T22:08:05.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:04 smithi086 ceph-mon[34297]: from='client.24874 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-12T22:08:05.745 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/mon.a/config 2024-05-12T22:08:07.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:06 smithi028 ceph-mon[29847]: pgmap v358: 97 pgs: 97 active+clean; 641 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-12T22:08:07.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:06 smithi186 ceph-mon[34785]: pgmap v358: 97 pgs: 97 active+clean; 641 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-12T22:08:07.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:06 smithi086 ceph-mon[34297]: pgmap v358: 97 pgs: 97 active+clean; 641 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-12T22:08:08.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:08.022+0000 7fb5b3200700 1 -- 172.21.15.28:0/871543100 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb5ac10e940 msgr2=0x7fb5ac10ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:08.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:08.022+0000 7fb5b3200700 1 --2- 172.21.15.28:0/871543100 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb5ac10e940 0x7fb5ac10ed20 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fb59c009990 tx=0x7fb59c009ca0 comp rx=0 tx=0).stop 2024-05-12T22:08:08.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:08.023+0000 7fb5b3200700 1 -- 172.21.15.28:0/871543100 shutdown_connections 2024-05-12T22:08:08.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:08.023+0000 7fb5b3200700 1 --2- 172.21.15.28:0/871543100 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5ac109350 0x7fb5ac115810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:08.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:08.023+0000 7fb5b3200700 1 --2- 172.21.15.28:0/871543100 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb5ac10e940 0x7fb5ac10ed20 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:08.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:08.023+0000 7fb5b3200700 1 --2- 172.21.15.28:0/871543100 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb5ac108990 0x7fb5ac108e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:08.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:08.023+0000 7fb5b3200700 1 -- 172.21.15.28:0/871543100 >> 172.21.15.28:0/871543100 conn(0x7fb5ac075af0 msgr2=0x7fb5ac077f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:08:08.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:08.023+0000 7fb5b3200700 1 -- 172.21.15.28:0/871543100 shutdown_connections 2024-05-12T22:08:08.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:08.023+0000 7fb5b3200700 1 -- 172.21.15.28:0/871543100 wait complete. 2024-05-12T22:08:08.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:08.024+0000 7fb5b3200700 1 Processor -- start 2024-05-12T22:08:08.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:08.024+0000 7fb5b3200700 1 -- start start 2024-05-12T22:08:08.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:08.025+0000 7fb5b3200700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5ac108990 0x7fb5ac19f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:08.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:08.025+0000 7fb5b3200700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb5ac109350 0x7fb5ac19f630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:08.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:08.025+0000 7fb5b3200700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb5ac10e940 0x7fb5ac1a39c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:08.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:08.025+0000 7fb5b3200700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb5ac118ca0 con 0x7fb5ac108990 2024-05-12T22:08:08.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:08.025+0000 7fb5b3200700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fb5ac118b20 con 0x7fb5ac109350 2024-05-12T22:08:08.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:08.025+0000 7fb5b3200700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb5ac118e20 con 0x7fb5ac10e940 2024-05-12T22:08:08.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:08.025+0000 7fb5b1200700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb5ac10e940 0x7fb5ac1a39c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:08.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:08.025+0000 7fb5abe00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5ac108990 0x7fb5ac19f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:08.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:08.025+0000 7fb5b1200700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb5ac10e940 0x7fb5ac1a39c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:34226/0 (socket says 172.21.15.28:34226) 2024-05-12T22:08:08.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:08.025+0000 7fb5b1200700 1 -- 172.21.15.28:0/2275235047 learned_addr learned my addr 172.21.15.28:0/2275235047 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:08:08.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:08.025+0000 7fb5ab400700 1 --2- 172.21.15.28:0/2275235047 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb5ac109350 0x7fb5ac19f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:08.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:08.026+0000 7fb5b1200700 1 -- 172.21.15.28:0/2275235047 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb5ac109350 msgr2=0x7fb5ac19f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:08.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:08.026+0000 7fb5b1200700 1 --2- 172.21.15.28:0/2275235047 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb5ac109350 0x7fb5ac19f630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:08.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:08.026+0000 7fb5b1200700 1 -- 172.21.15.28:0/2275235047 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5ac108990 msgr2=0x7fb5ac19f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:08.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:08.026+0000 7fb5b1200700 1 --2- 172.21.15.28:0/2275235047 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5ac108990 0x7fb5ac19f0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:08.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:08.026+0000 7fb5b1200700 1 -- 172.21.15.28:0/2275235047 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5ac1a4100 con 0x7fb5ac10e940 2024-05-12T22:08:08.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:08.026+0000 7fb5b1200700 1 --2- 172.21.15.28:0/2275235047 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb5ac10e940 0x7fb5ac1a39c0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fb5a400c370 tx=0x7fb5a400c680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:08:08.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:08.026+0000 7fb5a8c00700 1 -- 172.21.15.28:0/2275235047 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5a40092e0 con 0x7fb5ac10e940 2024-05-12T22:08:08.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:08.026+0000 7fb5b3200700 1 -- 172.21.15.28:0/2275235047 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5ac1a4390 con 0x7fb5ac10e940 2024-05-12T22:08:08.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:08.026+0000 7fb5a8c00700 1 -- 172.21.15.28:0/2275235047 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb5a4003a00 con 0x7fb5ac10e940 2024-05-12T22:08:08.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:08.026+0000 7fb5a8c00700 1 -- 172.21.15.28:0/2275235047 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5a40047c0 con 0x7fb5ac10e940 2024-05-12T22:08:08.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:08.027+0000 7fb5b3200700 1 -- 172.21.15.28:0/2275235047 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5ac1abc70 con 0x7fb5ac10e940 2024-05-12T22:08:08.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:08.028+0000 7fb5b3200700 1 -- 172.21.15.28:0/2275235047 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb5ac110ed0 con 0x7fb5ac10e940 2024-05-12T22:08:08.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:08.029+0000 7fb5a8c00700 1 -- 172.21.15.28:0/2275235047 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fb5a401a020 con 0x7fb5ac10e940 2024-05-12T22:08:08.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:08.031+0000 7fb5a8c00700 1 --2- 172.21.15.28:0/2275235047 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fb5900a2c20 0x7fb5900a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:08.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:08.031+0000 7fb5a8c00700 1 -- 172.21.15.28:0/2275235047 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7fb5a40ce910 con 0x7fb5ac10e940 2024-05-12T22:08:08.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:08.031+0000 7fb5abe00700 1 --2- 172.21.15.28:0/2275235047 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fb5900a2c20 0x7fb5900a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:08.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:08.032+0000 7fb5abe00700 1 --2- 172.21.15.28:0/2275235047 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fb5900a2c20 0x7fb5900a50e0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fb59c00b2e0 tx=0x7fb59c005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:08:08.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:08.035+0000 7fb5a8c00700 1 -- 172.21.15.28:0/2275235047 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fb5a4010040 con 0x7fb5ac10e940 2024-05-12T22:08:08.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:08.316+0000 7fb5b3200700 1 -- 172.21.15.28:0/2275235047 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fb5ac1108b0 con 0x7fb5ac10e940 2024-05-12T22:08:08.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:08.317+0000 7fb5a8c00700 1 -- 172.21.15.28:0/2275235047 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7fb5a409b820 con 0x7fb5ac10e940 2024-05-12T22:08:08.318 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-12T22:08:08.318 INFO:teuthology.orchestra.run.smithi028.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-05-12T22:08:08.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:08.320+0000 7fb5b3200700 1 -- 172.21.15.28:0/2275235047 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fb5900a2c20 msgr2=0x7fb5900a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:08.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:08.320+0000 7fb5b3200700 1 --2- 172.21.15.28:0/2275235047 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fb5900a2c20 0x7fb5900a50e0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fb59c00b2e0 tx=0x7fb59c005960 comp rx=0 tx=0).stop 2024-05-12T22:08:08.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:08.320+0000 7fb5b3200700 1 -- 172.21.15.28:0/2275235047 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb5ac10e940 msgr2=0x7fb5ac1a39c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:08.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:08.320+0000 7fb5b3200700 1 --2- 172.21.15.28:0/2275235047 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb5ac10e940 0x7fb5ac1a39c0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fb5a400c370 tx=0x7fb5a400c680 comp rx=0 tx=0).stop 2024-05-12T22:08:08.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:08.321+0000 7fb5b3200700 1 -- 172.21.15.28:0/2275235047 shutdown_connections 2024-05-12T22:08:08.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:08.321+0000 7fb5b3200700 1 --2- 172.21.15.28:0/2275235047 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fb5900a2c20 0x7fb5900a50e0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:08.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:08.321+0000 7fb5b3200700 1 --2- 172.21.15.28:0/2275235047 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb5ac10e940 0x7fb5ac1a39c0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:08.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:08.321+0000 7fb5b3200700 1 --2- 172.21.15.28:0/2275235047 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5ac108990 0x7fb5ac19f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:08.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:08.321+0000 7fb5b3200700 1 --2- 172.21.15.28:0/2275235047 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb5ac109350 0x7fb5ac19f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:08.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:08.321+0000 7fb5b3200700 1 -- 172.21.15.28:0/2275235047 >> 172.21.15.28:0/2275235047 conn(0x7fb5ac075af0 msgr2=0x7fb5ac076240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:08:08.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:08.321+0000 7fb5b3200700 1 -- 172.21.15.28:0/2275235047 shutdown_connections 2024-05-12T22:08:08.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:08.321+0000 7fb5b3200700 1 -- 172.21.15.28:0/2275235047 wait complete. 2024-05-12T22:08:09.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:08 smithi028 ceph-mon[29847]: pgmap v359: 97 pgs: 97 active+clean; 641 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-12T22:08:09.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:08 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/2275235047' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-05-12T22:08:09.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:08 smithi186 ceph-mon[34785]: pgmap v359: 97 pgs: 97 active+clean; 641 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-12T22:08:09.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:08 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/2275235047' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-05-12T22:08:09.216 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-05-12T22:08:09.217 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-12T22:08:09.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:08 smithi086 ceph-mon[34297]: pgmap v359: 97 pgs: 97 active+clean; 641 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-12T22:08:09.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:08 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/2275235047' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-05-12T22:08:09.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:09.444+0000 7f0b01600640 1 -- 172.21.15.28:0/457415867 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0afc109860 msgr2=0x7f0afc1100f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:09.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:09.444+0000 7f0b01600640 1 --2- 172.21.15.28:0/457415867 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0afc109860 0x7f0afc1100f0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f0af4009ca0 tx=0x7f0af402a8c0 comp rx=0 tx=0).stop 2024-05-12T22:08:09.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:09.444+0000 7f0b01600640 1 -- 172.21.15.28:0/457415867 shutdown_connections 2024-05-12T22:08:09.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:09.444+0000 7f0b01600640 1 --2- 172.21.15.28:0/457415867 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0afc109860 0x7f0afc1100f0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:09.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:09.444+0000 7f0b01600640 1 --2- 172.21.15.28:0/457415867 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0afc108ea0 0x7f0afc109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:09.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:09.444+0000 7f0b01600640 1 --2- 172.21.15.28:0/457415867 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0afc107cb0 0x7f0afc1080d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:09.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:09.444+0000 7f0b01600640 1 -- 172.21.15.28:0/457415867 >> 172.21.15.28:0/457415867 conn(0x7f0afc076c70 msgr2=0x7f0afc079090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:08:09.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:09.444+0000 7f0b01600640 1 -- 172.21.15.28:0/457415867 shutdown_connections 2024-05-12T22:08:09.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:09.444+0000 7f0b01600640 1 -- 172.21.15.28:0/457415867 wait complete. 2024-05-12T22:08:09.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:09.445+0000 7f0b01600640 1 Processor -- start 2024-05-12T22:08:09.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:09.446+0000 7f0b01600640 1 -- start start 2024-05-12T22:08:09.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:09.446+0000 7f0b01600640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0afc107cb0 0x7f0afc19df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:09.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:09.446+0000 7f0b01600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0afc108ea0 0x7f0afc19e490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:09.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:09.446+0000 7f0b01600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0afc109860 0x7f0afc1a5500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:09.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:09.446+0000 7f0b01600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0afc113960 con 0x7f0afc108ea0 2024-05-12T22:08:09.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:09.446+0000 7f0b01600640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f0afc1137e0 con 0x7f0afc109860 2024-05-12T22:08:09.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:09.446+0000 7f0b01600640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f0afc113ae0 con 0x7f0afc107cb0 2024-05-12T22:08:09.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:09.446+0000 7f0afbe00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0afc107cb0 0x7f0afc19df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:09.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:09.446+0000 7f0afbe00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0afc107cb0 0x7f0afc19df50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:34244/0 (socket says 172.21.15.28:34244) 2024-05-12T22:08:09.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:09.446+0000 7f0af3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0afc108ea0 0x7f0afc19e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:09.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:09.446+0000 7f0afbe00640 1 -- 172.21.15.28:0/1682830164 learned_addr learned my addr 172.21.15.28:0/1682830164 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:08:09.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:09.446+0000 7f0b00c00640 1 --2- 172.21.15.28:0/1682830164 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0afc109860 0x7f0afc1a5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:09.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:09.446+0000 7f0afbe00640 1 -- 172.21.15.28:0/1682830164 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0afc109860 msgr2=0x7f0afc1a5500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:09.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:09.446+0000 7f0afbe00640 1 --2- 172.21.15.28:0/1682830164 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0afc109860 0x7f0afc1a5500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:09.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:09.447+0000 7f0afbe00640 1 -- 172.21.15.28:0/1682830164 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0afc108ea0 msgr2=0x7f0afc19e490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:09.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:09.447+0000 7f0afbe00640 1 --2- 172.21.15.28:0/1682830164 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0afc108ea0 0x7f0afc19e490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:09.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:09.447+0000 7f0afbe00640 1 -- 172.21.15.28:0/1682830164 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0afc1a5b70 con 0x7f0afc107cb0 2024-05-12T22:08:09.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:09.447+0000 7f0af3400640 1 --2- 172.21.15.28:0/1682830164 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0afc108ea0 0x7f0afc19e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:08:09.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:09.447+0000 7f0afbe00640 1 --2- 172.21.15.28:0/1682830164 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0afc107cb0 0x7f0afc19df50 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f0aec00e9c0 tx=0x7f0aec00ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:08:09.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:09.447+0000 7f0af9600640 1 -- 172.21.15.28:0/1682830164 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0aec00cd80 con 0x7f0afc107cb0 2024-05-12T22:08:09.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:09.447+0000 7f0af9600640 1 -- 172.21.15.28:0/1682830164 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0aec0035f0 con 0x7f0afc107cb0 2024-05-12T22:08:09.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:09.447+0000 7f0af9600640 1 -- 172.21.15.28:0/1682830164 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0aec010690 con 0x7f0afc107cb0 2024-05-12T22:08:09.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:09.447+0000 7f0b01600640 1 -- 172.21.15.28:0/1682830164 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0afc1a5e60 con 0x7f0afc107cb0 2024-05-12T22:08:09.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:09.448+0000 7f0b01600640 1 -- 172.21.15.28:0/1682830164 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0afc1a6270 con 0x7f0afc107cb0 2024-05-12T22:08:09.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:09.449+0000 7f0b01600640 1 -- 172.21.15.28:0/1682830164 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0afc10ef70 con 0x7f0afc107cb0 2024-05-12T22:08:09.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:09.451+0000 7f0af9600640 1 -- 172.21.15.28:0/1682830164 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f0aec0037a0 con 0x7f0afc107cb0 2024-05-12T22:08:09.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:09.452+0000 7f0af9600640 1 --2- 172.21.15.28:0/1682830164 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f0ae00a2b90 0x7f0ae00a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:09.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:09.452+0000 7f0af9600640 1 -- 172.21.15.28:0/1682830164 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f0aec0cf4f0 con 0x7f0afc107cb0 2024-05-12T22:08:09.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:09.453+0000 7f0af3400640 1 --2- 172.21.15.28:0/1682830164 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f0ae00a2b90 0x7f0ae00a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:09.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:09.453+0000 7f0af3400640 1 --2- 172.21.15.28:0/1682830164 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f0ae00a2b90 0x7f0ae00a5050 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f0afc19f460 tx=0x7f0ae400b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:08:09.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:09.454+0000 7f0af9600640 1 -- 172.21.15.28:0/1682830164 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f0aec09c490 con 0x7f0afc107cb0 2024-05-12T22:08:09.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:09.664+0000 7f0b01600640 1 -- 172.21.15.28:0/1682830164 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0afc02d3b0 con 0x7f0afc107cb0 2024-05-12T22:08:09.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:09.666+0000 7f0af9600640 1 -- 172.21.15.28:0/1682830164 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22595 (secure 0 0 0) 0x7f0aec09bd90 con 0x7f0afc107cb0 2024-05-12T22:08:09.667 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-12T22:08:09.667 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":88,"fsid":"6b92e7b4-10aa-11ef-bc98-c7b262605968","created":"2024-05-12T21:57:37.276782+0000","modified":"2024-05-12T22:06:46.768798+0000","last_up_change":"2024-05-12T22:05:37.929562+0000","last_in_change":"2024-05-12T22:05:18.750768+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-12T22:01:32.292826+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-12T22:06:31.540263+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-12T22:06:32.906118+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6992ce0a-844b-40e3-8ed1-181c8474dccf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":906003882},{"type":"v1","addr":"172.21.15.28:6803","nonce":906003882}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":906003882},{"type":"v1","addr":"172.21.15.28:6805","nonce":906003882}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":906003882},{"type":"v1","addr":"172.21.15.28:6809","nonce":906003882}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":906003882},{"type":"v1","addr":"172.21.15.28:6807","nonce":906003882}]},"public_addr":"172.21.15.28:6803/906003882","cluster_addr":"172.21.15.28:6805/906003882","heartbeat_back_addr":"172.21.15.28:6809/906003882","heartbeat_front_addr":"172.21.15.28:6807/906003882","state":["exists","up"]},{"osd":1,"uuid":"ced20c96-c481-4d48-90d1-6b43228959ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":3354141819},{"type":"v1","addr":"172.21.15.86:6801","nonce":3354141819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":3354141819},{"type":"v1","addr":"172.21.15.86:6803","nonce":3354141819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":3354141819},{"type":"v1","addr":"172.21.15.86:6807","nonce":3354141819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":3354141819},{"type":"v1","addr":"172.21.15.86:6805","nonce":3354141819}]},"public_addr":"172.21.15.86:6801/3354141819","cluster_addr":"172.21.15.86:6803/3354141819","heartbeat_back_addr":"172.21.15.86:6807/3354141819","heartbeat_front_addr":"172.21.15.86:6805/3354141819","state":["exists","up"]},{"osd":2,"uuid":"54a2f221-fa65-4795-9f5e-e660cc7c0098","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":906459365},{"type":"v1","addr":"172.21.15.186:6801","nonce":906459365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":906459365},{"type":"v1","addr":"172.21.15.186:6803","nonce":906459365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":906459365},{"type":"v1","addr":"172.21.15.186:6807","nonce":906459365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":906459365},{"type":"v1","addr":"172.21.15.186:6805","nonce":906459365}]},"public_addr":"172.21.15.186:6801/906459365","cluster_addr":"172.21.15.186:6803/906459365","heartbeat_back_addr":"172.21.15.186:6807/906459365","heartbeat_front_addr":"172.21.15.186:6805/906459365","state":["exists","up"]},{"osd":3,"uuid":"86836737-ac9a-487b-93f2-7fcd3c406960","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":4182884436},{"type":"v1","addr":"172.21.15.28:6811","nonce":4182884436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":4182884436},{"type":"v1","addr":"172.21.15.28:6813","nonce":4182884436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":4182884436},{"type":"v1","addr":"172.21.15.28:6817","nonce":4182884436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":4182884436},{"type":"v1","addr":"172.21.15.28:6815","nonce":4182884436}]},"public_addr":"172.21.15.28:6811/4182884436","cluster_addr":"172.21.15.28:6813/4182884436","heartbeat_back_addr":"172.21.15.28:6817/4182884436","heartbeat_front_addr":"172.21.15.28:6815/4182884436","state":["exists","up"]},{"osd":4,"uuid":"d162ebcd-83c0-4fd5-9925-28b2cecea935","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2596998708},{"type":"v1","addr":"172.21.15.86:6809","nonce":2596998708}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2596998708},{"type":"v1","addr":"172.21.15.86:6811","nonce":2596998708}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2596998708},{"type":"v1","addr":"172.21.15.86:6815","nonce":2596998708}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2596998708},{"type":"v1","addr":"172.21.15.86:6813","nonce":2596998708}]},"public_addr":"172.21.15.86:6809/2596998708","cluster_addr":"172.21.15.86:6811/2596998708","heartbeat_back_addr":"172.21.15.86:6815/2596998708","heartbeat_front_addr":"172.21.15.86:6813/2596998708","state":["exists","up"]},{"osd":5,"uuid":"c794336a-a105-425c-b6c7-a32b90c05bb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":1944205756},{"type":"v1","addr":"172.21.15.186:6809","nonce":1944205756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":1944205756},{"type":"v1","addr":"172.21.15.186:6811","nonce":1944205756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":1944205756},{"type":"v1","addr":"172.21.15.186:6815","nonce":1944205756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":1944205756},{"type":"v1","addr":"172.21.15.186:6813","nonce":1944205756}]},"public_addr":"172.21.15.186:6809/1944205756","cluster_addr":"172.21.15.186:6811/1944205756","heartbeat_back_addr":"172.21.15.186:6815/1944205756","heartbeat_front_addr":"172.21.15.186:6813/1944205756","state":["exists","up"]},{"osd":6,"uuid":"516932ac-af00-45b7-ad3c-d6dec7db442f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":1642308925},{"type":"v1","addr":"172.21.15.28:6819","nonce":1642308925}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":1642308925},{"type":"v1","addr":"172.21.15.28:6821","nonce":1642308925}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1642308925},{"type":"v1","addr":"172.21.15.28:6825","nonce":1642308925}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":1642308925},{"type":"v1","addr":"172.21.15.28:6823","nonce":1642308925}]},"public_addr":"172.21.15.28:6819/1642308925","cluster_addr":"172.21.15.28:6821/1642308925","heartbeat_back_addr":"172.21.15.28:6825/1642308925","heartbeat_front_addr":"172.21.15.28:6823/1642308925","state":["exists","up"]},{"osd":7,"uuid":"ca7a059f-0414-438e-b2d1-a10f348036be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":726062711},{"type":"v1","addr":"172.21.15.86:6817","nonce":726062711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":726062711},{"type":"v1","addr":"172.21.15.86:6819","nonce":726062711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":726062711},{"type":"v1","addr":"172.21.15.86:6823","nonce":726062711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":726062711},{"type":"v1","addr":"172.21.15.86:6821","nonce":726062711}]},"public_addr":"172.21.15.86:6817/726062711","cluster_addr":"172.21.15.86:6819/726062711","heartbeat_back_addr":"172.21.15.86:6823/726062711","heartbeat_front_addr":"172.21.15.86:6821/726062711","state":["exists","up"]},{"osd":8,"uuid":"92e3f41f-aebd-422e-a99e-555f7ba4aa53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":3007687277},{"type":"v1","addr":"172.21.15.186:6817","nonce":3007687277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":3007687277},{"type":"v1","addr":"172.21.15.186:6819","nonce":3007687277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":3007687277},{"type":"v1","addr":"172.21.15.186:6823","nonce":3007687277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":3007687277},{"type":"v1","addr":"172.21.15.186:6821","nonce":3007687277}]},"public_addr":"172.21.15.186:6817/3007687277","cluster_addr":"172.21.15.186:6819/3007687277","heartbeat_back_addr":"172.21.15.186:6823/3007687277","heartbeat_front_addr":"172.21.15.186:6821/3007687277","state":["exists","up"]},{"osd":9,"uuid":"38ff7d9d-ee85-404c-8c59-adc327e1ccfa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":136220472},{"type":"v1","addr":"172.21.15.28:6827","nonce":136220472}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":136220472},{"type":"v1","addr":"172.21.15.28:6829","nonce":136220472}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":136220472},{"type":"v1","addr":"172.21.15.28:6833","nonce":136220472}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":136220472},{"type":"v1","addr":"172.21.15.28:6831","nonce":136220472}]},"public_addr":"172.21.15.28:6827/136220472","cluster_addr":"172.21.15.28:6829/136220472","heartbeat_back_addr":"172.21.15.28:6833/136220472","heartbeat_front_addr":"172.21.15.28:6831/136220472","state":["exists","up"]},{"osd":10,"uuid":"5e9ff657-afa8-4bab-8491-f209038392b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1189968310},{"type":"v1","addr":"172.21.15.86:6825","nonce":1189968310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":1189968310},{"type":"v1","addr":"172.21.15.86:6827","nonce":1189968310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":1189968310},{"type":"v1","addr":"172.21.15.86:6831","nonce":1189968310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":1189968310},{"type":"v1","addr":"172.21.15.86:6829","nonce":1189968310}]},"public_addr":"172.21.15.86:6825/1189968310","cluster_addr":"172.21.15.86:6827/1189968310","heartbeat_back_addr":"172.21.15.86:6831/1189968310","heartbeat_front_addr":"172.21.15.86:6829/1189968310","state":["exists","up"]},{"osd":11,"uuid":"f112e3c6-65b6-4253-a966-62001db7d069","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3801668110},{"type":"v1","addr":"172.21.15.186:6825","nonce":3801668110}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3801668110},{"type":"v1","addr":"172.21.15.186:6827","nonce":3801668110}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3801668110},{"type":"v1","addr":"172.21.15.186:6831","nonce":3801668110}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3801668110},{"type":"v1","addr":"172.21.15.186:6829","nonce":3801668110}]},"public_addr":"172.21.15.186:6825/3801668110","cluster_addr":"172.21.15.186:6827/3801668110","heartbeat_back_addr":"172.21.15.186:6831/3801668110","heartbeat_front_addr":"172.21.15.186:6829/3801668110","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:00:40.483623+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:01:05.284385+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:01:28.645716+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:02:01.295092+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:02:26.486601+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:02:50.807478+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:03:24.756169+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:03:49.975173+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:04:13.772233+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:04:46.758555+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:05:12.331685+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:05:36.462548+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":2,"to":1}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.25","mappings":[{"from":5,"to":9}]},{"pgid":"3.27","mappings":[{"from":4,"to":8}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/1177009533":"2024-05-13T21:58:45.550459+0000","172.21.15.28:6800/3760499540":"2024-05-13T21:58:45.550459+0000","172.21.15.28:6801/3783742728":"2024-05-13T21:58:10.421564+0000","172.21.15.28:6800/3783742728":"2024-05-13T21:58:10.421564+0000","172.21.15.28:0/2229156260":"2024-05-13T21:58:45.550459+0000","172.21.15.28:0/1202140963":"2024-05-13T21:58:10.421564+0000","172.21.15.28:0/2924223784":"2024-05-13T21:58:45.550459+0000","172.21.15.28:6801/3760499540":"2024-05-13T21:58:45.550459+0000","172.21.15.28:0/1240027206":"2024-05-13T21:58:10.421564+0000","172.21.15.28:0/171417728":"2024-05-13T21:58:10.421564+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-12T22:08:09.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:09.669+0000 7f0b01600640 1 -- 172.21.15.28:0/1682830164 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f0ae00a2b90 msgr2=0x7f0ae00a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:09.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:09.669+0000 7f0b01600640 1 --2- 172.21.15.28:0/1682830164 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f0ae00a2b90 0x7f0ae00a5050 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f0afc19f460 tx=0x7f0ae400b390 comp rx=0 tx=0).stop 2024-05-12T22:08:09.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:09.669+0000 7f0b01600640 1 -- 172.21.15.28:0/1682830164 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0afc107cb0 msgr2=0x7f0afc19df50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:09.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:09.669+0000 7f0b01600640 1 --2- 172.21.15.28:0/1682830164 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0afc107cb0 0x7f0afc19df50 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f0aec00e9c0 tx=0x7f0aec00ee80 comp rx=0 tx=0).stop 2024-05-12T22:08:09.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:09.669+0000 7f0b01600640 1 -- 172.21.15.28:0/1682830164 shutdown_connections 2024-05-12T22:08:09.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:09.669+0000 7f0b01600640 1 --2- 172.21.15.28:0/1682830164 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f0ae00a2b90 0x7f0ae00a5050 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:09.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:09.669+0000 7f0b01600640 1 --2- 172.21.15.28:0/1682830164 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0afc109860 0x7f0afc1a5500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:09.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:09.669+0000 7f0b01600640 1 --2- 172.21.15.28:0/1682830164 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0afc108ea0 0x7f0afc19e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:09.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:09.669+0000 7f0b01600640 1 --2- 172.21.15.28:0/1682830164 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0afc107cb0 0x7f0afc19df50 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:09.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:09.669+0000 7f0b01600640 1 -- 172.21.15.28:0/1682830164 >> 172.21.15.28:0/1682830164 conn(0x7f0afc076c70 msgr2=0x7f0afc10c140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:08:09.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:09.669+0000 7f0b01600640 1 -- 172.21.15.28:0/1682830164 shutdown_connections 2024-05-12T22:08:09.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:09.670+0000 7f0b01600640 1 -- 172.21.15.28:0/1682830164 wait complete. 2024-05-12T22:08:09.690 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-12T22:01:32.292826+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-12T22:06:31.540263+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-12T22:06:32.906118+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-12T22:08:09.691 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-12T22:08:09.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:09.910+0000 7f90e6e00640 1 -- 172.21.15.28:0/2659773727 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f90e01012b0 msgr2=0x7f90e0101730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:09.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:09.910+0000 7f90e6e00640 1 --2- 172.21.15.28:0/2659773727 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f90e01012b0 0x7f90e0101730 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f90d8009ca0 tx=0x7f90d802a8c0 comp rx=0 tx=0).stop 2024-05-12T22:08:09.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:09.910+0000 7f90e6e00640 1 -- 172.21.15.28:0/2659773727 shutdown_connections 2024-05-12T22:08:09.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:09.910+0000 7f90e6e00640 1 --2- 172.21.15.28:0/2659773727 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f90e01012b0 0x7f90e0101730 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:09.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:09.910+0000 7f90e6e00640 1 --2- 172.21.15.28:0/2659773727 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f90e0100910 0x7f90e0100d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:09.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:09.910+0000 7f90e6e00640 1 --2- 172.21.15.28:0/2659773727 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f90e00fffb0 0x7f90e01003d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:09.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:09.910+0000 7f90e6e00640 1 -- 172.21.15.28:0/2659773727 >> 172.21.15.28:0/2659773727 conn(0x7f90e00fbbd0 msgr2=0x7f90e00fdff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:08:09.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:09.910+0000 7f90e6e00640 1 -- 172.21.15.28:0/2659773727 shutdown_connections 2024-05-12T22:08:09.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:09.910+0000 7f90e6e00640 1 -- 172.21.15.28:0/2659773727 wait complete. 2024-05-12T22:08:09.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:09.912+0000 7f90e6e00640 1 Processor -- start 2024-05-12T22:08:09.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:09.912+0000 7f90e6e00640 1 -- start start 2024-05-12T22:08:09.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:09.912+0000 7f90e6e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f90e00fffb0 0x7f90e0199ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:09.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:09.912+0000 7f90e6e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f90e0100910 0x7f90e019a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:09.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:09.912+0000 7f90e6e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f90e01012b0 0x7f90e01a1150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:09.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:09.912+0000 7f90e6e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f90e010f590 con 0x7f90e00fffb0 2024-05-12T22:08:09.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:09.912+0000 7f90e6e00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f90e010f410 con 0x7f90e0100910 2024-05-12T22:08:09.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:09.912+0000 7f90e6e00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f90e010f710 con 0x7f90e01012b0 2024-05-12T22:08:09.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:09.913+0000 7f90e5a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f90e00fffb0 0x7f90e0199ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:09.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:09.913+0000 7f90e5000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f90e0100910 0x7f90e019a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:09.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:09.913+0000 7f90e5a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f90e00fffb0 0x7f90e0199ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38168/0 (socket says 172.21.15.28:38168) 2024-05-12T22:08:09.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:09.913+0000 7f90e5a00640 1 -- 172.21.15.28:0/3005760130 learned_addr learned my addr 172.21.15.28:0/3005760130 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:08:09.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:09.913+0000 7f90e6400640 1 --2- 172.21.15.28:0/3005760130 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f90e01012b0 0x7f90e01a1150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:09.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:09.913+0000 7f90e5000640 1 -- 172.21.15.28:0/3005760130 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f90e01012b0 msgr2=0x7f90e01a1150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:09.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:09.913+0000 7f90e5000640 1 --2- 172.21.15.28:0/3005760130 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f90e01012b0 0x7f90e01a1150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:09.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:09.913+0000 7f90e5000640 1 -- 172.21.15.28:0/3005760130 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f90e00fffb0 msgr2=0x7f90e0199ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:09.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:09.913+0000 7f90e5000640 1 --2- 172.21.15.28:0/3005760130 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f90e00fffb0 0x7f90e0199ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:09.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:09.913+0000 7f90e5000640 1 -- 172.21.15.28:0/3005760130 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90e01a17c0 con 0x7f90e0100910 2024-05-12T22:08:09.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:09.913+0000 7f90e5a00640 1 --2- 172.21.15.28:0/3005760130 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f90e00fffb0 0x7f90e0199ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:08:09.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:09.914+0000 7f90e5000640 1 --2- 172.21.15.28:0/3005760130 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f90e0100910 0x7f90e019a0e0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f90d000e9f0 tx=0x7f90d000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:08:09.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:09.916+0000 7f90de000640 1 -- 172.21.15.28:0/3005760130 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90d000cd80 con 0x7f90e0100910 2024-05-12T22:08:09.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:09.916+0000 7f90de000640 1 -- 172.21.15.28:0/3005760130 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f90d00035f0 con 0x7f90e0100910 2024-05-12T22:08:09.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:09.916+0000 7f90de000640 1 -- 172.21.15.28:0/3005760130 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90d0010690 con 0x7f90e0100910 2024-05-12T22:08:09.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:09.916+0000 7f90e6e00640 1 -- 172.21.15.28:0/3005760130 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90e01a1ab0 con 0x7f90e0100910 2024-05-12T22:08:09.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:09.916+0000 7f90e6e00640 1 -- 172.21.15.28:0/3005760130 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90e01a2050 con 0x7f90e0100910 2024-05-12T22:08:09.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:09.917+0000 7f90e6e00640 1 -- 172.21.15.28:0/3005760130 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f90e010ae40 con 0x7f90e0100910 2024-05-12T22:08:09.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:09.919+0000 7f90de000640 1 -- 172.21.15.28:0/3005760130 <== mon.2 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f90d0004020 con 0x7f90e0100910 2024-05-12T22:08:09.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:09.921+0000 7f90de000640 1 --2- 172.21.15.28:0/3005760130 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f90c80a2c10 0x7f90c80a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:09.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:09.921+0000 7f90de000640 1 -- 172.21.15.28:0/3005760130 <== mon.2 v2:172.21.15.86:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f90d00d1460 con 0x7f90e0100910 2024-05-12T22:08:09.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:09.921+0000 7f90e5a00640 1 --2- 172.21.15.28:0/3005760130 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f90c80a2c10 0x7f90c80a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:09.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:09.922+0000 7f90e5a00640 1 --2- 172.21.15.28:0/3005760130 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f90c80a2c10 0x7f90c80a50d0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f90d4011550 tx=0x7f90d4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:08:09.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:09.925+0000 7f90de000640 1 -- 172.21.15.28:0/3005760130 <== mon.2 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f90d009c230 con 0x7f90e0100910 2024-05-12T22:08:10.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:09 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/1682830164' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-12T22:08:10.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:10.136+0000 7f90e6e00640 1 -- 172.21.15.28:0/3005760130 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f90e002d1c0 con 0x7f90e0100910 2024-05-12T22:08:10.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:10.137+0000 7f90de000640 1 -- 172.21.15.28:0/3005760130 <== mon.2 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) ==== 93+0+10 (secure 0 0 0) 0x7f90d009bb30 con 0x7f90e0100910 2024-05-12T22:08:10.139 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-05-12T22:08:10.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:10.140+0000 7f90e6e00640 1 -- 172.21.15.28:0/3005760130 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f90c80a2c10 msgr2=0x7f90c80a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:10.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:10.140+0000 7f90e6e00640 1 --2- 172.21.15.28:0/3005760130 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f90c80a2c10 0x7f90c80a50d0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f90d4011550 tx=0x7f90d4005f90 comp rx=0 tx=0).stop 2024-05-12T22:08:10.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:10.140+0000 7f90e6e00640 1 -- 172.21.15.28:0/3005760130 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f90e0100910 msgr2=0x7f90e019a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:10.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:10.140+0000 7f90e6e00640 1 --2- 172.21.15.28:0/3005760130 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f90e0100910 0x7f90e019a0e0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f90d000e9f0 tx=0x7f90d000eeb0 comp rx=0 tx=0).stop 2024-05-12T22:08:10.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:10.140+0000 7f90e6e00640 1 -- 172.21.15.28:0/3005760130 shutdown_connections 2024-05-12T22:08:10.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:10.140+0000 7f90e6e00640 1 --2- 172.21.15.28:0/3005760130 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f90c80a2c10 0x7f90c80a50d0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:10.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:10.140+0000 7f90e6e00640 1 --2- 172.21.15.28:0/3005760130 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f90e01012b0 0x7f90e01a1150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:10.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:10.140+0000 7f90e6e00640 1 --2- 172.21.15.28:0/3005760130 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f90e0100910 0x7f90e019a0e0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:10.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:10.140+0000 7f90e6e00640 1 --2- 172.21.15.28:0/3005760130 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f90e00fffb0 0x7f90e0199ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:10.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:10.140+0000 7f90e6e00640 1 -- 172.21.15.28:0/3005760130 >> 172.21.15.28:0/3005760130 conn(0x7f90e00fbbd0 msgr2=0x7f90e00fd280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:08:10.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:10.140+0000 7f90e6e00640 1 -- 172.21.15.28:0/3005760130 shutdown_connections 2024-05-12T22:08:10.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:10.140+0000 7f90e6e00640 1 -- 172.21.15.28:0/3005760130 wait complete. 2024-05-12T22:08:10.161 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-12T22:08:10.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:09 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/1682830164' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-12T22:08:10.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:09 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/1682830164' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-12T22:08:10.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:10.373+0000 7f20ba800640 1 -- 172.21.15.28:0/3722788614 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f20b4074290 msgr2=0x7f20b4074710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:10.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:10.373+0000 7f20ba800640 1 --2- 172.21.15.28:0/3722788614 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f20b4074290 0x7f20b4074710 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f20a0009930 tx=0x7f20a002a420 comp rx=0 tx=0).stop 2024-05-12T22:08:10.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:10.374+0000 7f20ba800640 1 -- 172.21.15.28:0/3722788614 shutdown_connections 2024-05-12T22:08:10.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:10.374+0000 7f20ba800640 1 --2- 172.21.15.28:0/3722788614 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f20b410de40 0x7f20b4110230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:10.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:10.374+0000 7f20ba800640 1 --2- 172.21.15.28:0/3722788614 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f20b4074290 0x7f20b4074710 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:10.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:10.374+0000 7f20ba800640 1 --2- 172.21.15.28:0/3722788614 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f20b4075870 0x7f20b4073cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:10.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:10.374+0000 7f20ba800640 1 -- 172.21.15.28:0/3722788614 >> 172.21.15.28:0/3722788614 conn(0x7f20b40fcde0 msgr2=0x7f20b40ff220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:08:10.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:10.374+0000 7f20ba800640 1 -- 172.21.15.28:0/3722788614 shutdown_connections 2024-05-12T22:08:10.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:10.374+0000 7f20ba800640 1 -- 172.21.15.28:0/3722788614 wait complete. 2024-05-12T22:08:10.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:10.375+0000 7f20ba800640 1 Processor -- start 2024-05-12T22:08:10.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:10.375+0000 7f20ba800640 1 -- start start 2024-05-12T22:08:10.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:10.375+0000 7f20ba800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f20b4074290 0x7f20b419e210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:10.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:10.375+0000 7f20ba800640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f20b4075870 0x7f20b419e750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:10.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:10.375+0000 7f20ba800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f20b410de40 0x7f20b41a57c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:10.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:10.375+0000 7f20ba800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f20b4113b30 con 0x7f20b410de40 2024-05-12T22:08:10.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:10.375+0000 7f20ba800640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f20b41139b0 con 0x7f20b4074290 2024-05-12T22:08:10.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:10.375+0000 7f20ba800640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f20b4113cb0 con 0x7f20b4075870 2024-05-12T22:08:10.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:10.375+0000 7f20b3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f20b410de40 0x7f20b41a57c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:10.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:10.375+0000 7f20b3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f20b410de40 0x7f20b41a57c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38184/0 (socket says 172.21.15.28:38184) 2024-05-12T22:08:10.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:10.375+0000 7f20b3e00640 1 -- 172.21.15.28:0/3889933315 learned_addr learned my addr 172.21.15.28:0/3889933315 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:08:10.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:10.376+0000 7f20b3400640 1 --2- 172.21.15.28:0/3889933315 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f20b4074290 0x7f20b419e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:10.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:10.376+0000 7f20b2a00640 1 --2- 172.21.15.28:0/3889933315 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f20b4075870 0x7f20b419e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:10.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:10.376+0000 7f20b3e00640 1 -- 172.21.15.28:0/3889933315 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f20b4075870 msgr2=0x7f20b419e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:10.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:10.376+0000 7f20b3e00640 1 --2- 172.21.15.28:0/3889933315 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f20b4075870 0x7f20b419e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:10.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:10.376+0000 7f20b3e00640 1 -- 172.21.15.28:0/3889933315 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f20b4074290 msgr2=0x7f20b419e210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:10.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:10.376+0000 7f20b3e00640 1 --2- 172.21.15.28:0/3889933315 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f20b4074290 0x7f20b419e210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:10.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:10.376+0000 7f20b3e00640 1 -- 172.21.15.28:0/3889933315 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20b41a5e30 con 0x7f20b410de40 2024-05-12T22:08:10.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:10.376+0000 7f20b3e00640 1 --2- 172.21.15.28:0/3889933315 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f20b410de40 0x7f20b41a57c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f20ac00eb10 tx=0x7f20ac00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:08:10.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:10.376+0000 7f20abe00640 1 -- 172.21.15.28:0/3889933315 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20ac004b20 con 0x7f20b410de40 2024-05-12T22:08:10.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:10.376+0000 7f20abe00640 1 -- 172.21.15.28:0/3889933315 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f20ac004cc0 con 0x7f20b410de40 2024-05-12T22:08:10.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:10.376+0000 7f20ba800640 1 -- 172.21.15.28:0/3889933315 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20b41a6120 con 0x7f20b410de40 2024-05-12T22:08:10.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:10.377+0000 7f20abe00640 1 -- 172.21.15.28:0/3889933315 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20ac010700 con 0x7f20b410de40 2024-05-12T22:08:10.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:10.377+0000 7f20ba800640 1 -- 172.21.15.28:0/3889933315 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20b41a65f0 con 0x7f20b410de40 2024-05-12T22:08:10.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:10.378+0000 7f20ba800640 1 -- 172.21.15.28:0/3889933315 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20b4103f00 con 0x7f20b410de40 2024-05-12T22:08:10.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:10.380+0000 7f20abe00640 1 -- 172.21.15.28:0/3889933315 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f20ac005000 con 0x7f20b410de40 2024-05-12T22:08:10.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:10.382+0000 7f20abe00640 1 --2- 172.21.15.28:0/3889933315 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f20980a2b90 0x7f20980a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:10.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:10.382+0000 7f20abe00640 1 -- 172.21.15.28:0/3889933315 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f20ac0cf750 con 0x7f20b410de40 2024-05-12T22:08:10.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:10.382+0000 7f20b3400640 1 --2- 172.21.15.28:0/3889933315 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f20980a2b90 0x7f20980a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:10.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:10.383+0000 7f20b3400640 1 --2- 172.21.15.28:0/3889933315 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f20980a2b90 0x7f20980a5050 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f20a4009bd0 tx=0x7f20a4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:08:10.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:10.387+0000 7f20abe00640 1 -- 172.21.15.28:0/3889933315 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f20ac09c660 con 0x7f20b410de40 2024-05-12T22:08:10.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:10.591+0000 7f20ba800640 1 -- 172.21.15.28:0/3889933315 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f20b4062430 con 0x7f20b410de40 2024-05-12T22:08:10.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:10.592+0000 7f20abe00640 1 -- 172.21.15.28:0/3889933315 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) ==== 104+0+11 (secure 0 0 0) 0x7f20ac09bf60 con 0x7f20b410de40 2024-05-12T22:08:10.593 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-05-12T22:08:10.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:10.595+0000 7f20ba800640 1 -- 172.21.15.28:0/3889933315 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f20980a2b90 msgr2=0x7f20980a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:10.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:10.595+0000 7f20ba800640 1 --2- 172.21.15.28:0/3889933315 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f20980a2b90 0x7f20980a5050 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f20a4009bd0 tx=0x7f20a4005f90 comp rx=0 tx=0).stop 2024-05-12T22:08:10.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:10.595+0000 7f20ba800640 1 -- 172.21.15.28:0/3889933315 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f20b410de40 msgr2=0x7f20b41a57c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:10.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:10.595+0000 7f20ba800640 1 --2- 172.21.15.28:0/3889933315 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f20b410de40 0x7f20b41a57c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f20ac00eb10 tx=0x7f20ac00c2d0 comp rx=0 tx=0).stop 2024-05-12T22:08:10.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:10.595+0000 7f20ba800640 1 -- 172.21.15.28:0/3889933315 shutdown_connections 2024-05-12T22:08:10.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:10.595+0000 7f20ba800640 1 --2- 172.21.15.28:0/3889933315 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f20980a2b90 0x7f20980a5050 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:10.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:10.595+0000 7f20ba800640 1 --2- 172.21.15.28:0/3889933315 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f20b410de40 0x7f20b41a57c0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:10.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:10.595+0000 7f20ba800640 1 --2- 172.21.15.28:0/3889933315 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f20b4075870 0x7f20b419e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:10.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:10.595+0000 7f20ba800640 1 --2- 172.21.15.28:0/3889933315 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f20b4074290 0x7f20b419e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:10.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:10.595+0000 7f20ba800640 1 -- 172.21.15.28:0/3889933315 >> 172.21.15.28:0/3889933315 conn(0x7f20b40fcde0 msgr2=0x7f20b41010d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:08:10.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:10.595+0000 7f20ba800640 1 -- 172.21.15.28:0/3889933315 shutdown_connections 2024-05-12T22:08:10.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:10.595+0000 7f20ba800640 1 -- 172.21.15.28:0/3889933315 wait complete. 2024-05-12T22:08:10.616 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-12T22:08:10.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:10.833+0000 7fdaa2000640 1 -- 172.21.15.28:0/126083160 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda9c107d70 msgr2=0x7fda9c108190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:10.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:10.833+0000 7fdaa2000640 1 --2- 172.21.15.28:0/126083160 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda9c107d70 0x7fda9c108190 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fda90009900 tx=0x7fda9002a410 comp rx=0 tx=0).stop 2024-05-12T22:08:10.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:10.833+0000 7fdaa2000640 1 -- 172.21.15.28:0/126083160 shutdown_connections 2024-05-12T22:08:10.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:10.833+0000 7fdaa2000640 1 --2- 172.21.15.28:0/126083160 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fda9c109920 0x7fda9c1101b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:10.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:10.833+0000 7fdaa2000640 1 --2- 172.21.15.28:0/126083160 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fda9c108f60 0x7fda9c1093e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:10.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:10.833+0000 7fdaa2000640 1 --2- 172.21.15.28:0/126083160 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda9c107d70 0x7fda9c108190 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:10.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:10.833+0000 7fdaa2000640 1 -- 172.21.15.28:0/126083160 >> 172.21.15.28:0/126083160 conn(0x7fda9c076d00 msgr2=0x7fda9c079140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:08:10.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:10.833+0000 7fdaa2000640 1 -- 172.21.15.28:0/126083160 shutdown_connections 2024-05-12T22:08:10.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:10.833+0000 7fdaa2000640 1 -- 172.21.15.28:0/126083160 wait complete. 2024-05-12T22:08:10.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:10.834+0000 7fdaa2000640 1 Processor -- start 2024-05-12T22:08:10.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:10.834+0000 7fdaa2000640 1 -- start start 2024-05-12T22:08:10.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:10.834+0000 7fdaa2000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fda9c107d70 0x7fda9c19e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:10.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:10.834+0000 7fdaa2000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda9c108f60 0x7fda9c19e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:10.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:10.834+0000 7fdaa2000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fda9c109920 0x7fda9c1a5680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:10.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:10.834+0000 7fdaa2000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fda9c113a10 con 0x7fda9c108f60 2024-05-12T22:08:10.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:10.834+0000 7fdaa2000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fda9c113890 con 0x7fda9c109920 2024-05-12T22:08:10.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:10.834+0000 7fdaa2000640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fda9c113b90 con 0x7fda9c107d70 2024-05-12T22:08:10.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:10.834+0000 7fdaa0c00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fda9c107d70 0x7fda9c19e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:10.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:10.834+0000 7fdaa0c00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fda9c107d70 0x7fda9c19e0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:34304/0 (socket says 172.21.15.28:34304) 2024-05-12T22:08:10.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:10.834+0000 7fdaa0c00640 1 -- 172.21.15.28:0/3805044765 learned_addr learned my addr 172.21.15.28:0/3805044765 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:08:10.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:10.835+0000 7fda9be00640 1 --2- 172.21.15.28:0/3805044765 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda9c108f60 0x7fda9c19e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:10.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:10.835+0000 7fdaa1600640 1 --2- 172.21.15.28:0/3805044765 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fda9c109920 0x7fda9c1a5680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:10.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:10.835+0000 7fdaa0c00640 1 -- 172.21.15.28:0/3805044765 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fda9c109920 msgr2=0x7fda9c1a5680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:10.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:10.835+0000 7fdaa0c00640 1 --2- 172.21.15.28:0/3805044765 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fda9c109920 0x7fda9c1a5680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:10.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:10.835+0000 7fdaa0c00640 1 -- 172.21.15.28:0/3805044765 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda9c108f60 msgr2=0x7fda9c19e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:10.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:10.835+0000 7fdaa0c00640 1 --2- 172.21.15.28:0/3805044765 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda9c108f60 0x7fda9c19e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:10.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:10.835+0000 7fdaa0c00640 1 -- 172.21.15.28:0/3805044765 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda9c1a5cf0 con 0x7fda9c107d70 2024-05-12T22:08:10.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:10.835+0000 7fdaa0c00640 1 --2- 172.21.15.28:0/3805044765 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fda9c107d70 0x7fda9c19e0d0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fda90031440 tx=0x7fda90032ce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:08:10.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:10.836+0000 7fda99600640 1 -- 172.21.15.28:0/3805044765 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda9003f030 con 0x7fda9c107d70 2024-05-12T22:08:10.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:10.836+0000 7fda99600640 1 -- 172.21.15.28:0/3805044765 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fda90003ce0 con 0x7fda9c107d70 2024-05-12T22:08:10.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:10.836+0000 7fdaa2000640 1 -- 172.21.15.28:0/3805044765 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda9c1a5f80 con 0x7fda9c107d70 2024-05-12T22:08:10.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:10.836+0000 7fda99600640 1 -- 172.21.15.28:0/3805044765 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda90035070 con 0x7fda9c107d70 2024-05-12T22:08:10.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:10.836+0000 7fdaa2000640 1 -- 172.21.15.28:0/3805044765 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda9c1a6390 con 0x7fda9c107d70 2024-05-12T22:08:10.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:10.837+0000 7fdaa2000640 1 -- 172.21.15.28:0/3805044765 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda9c10f090 con 0x7fda9c107d70 2024-05-12T22:08:10.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:10.839+0000 7fda99600640 1 -- 172.21.15.28:0/3805044765 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fda90042030 con 0x7fda9c107d70 2024-05-12T22:08:10.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:10.840+0000 7fda99600640 1 --2- 172.21.15.28:0/3805044765 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fda840a2b90 0x7fda840a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:10.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:10.841+0000 7fda99600640 1 -- 172.21.15.28:0/3805044765 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7fda900ef510 con 0x7fda9c107d70 2024-05-12T22:08:10.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:10.841+0000 7fda9be00640 1 --2- 172.21.15.28:0/3805044765 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fda840a2b90 0x7fda840a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:10.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:10.841+0000 7fda9be00640 1 --2- 172.21.15.28:0/3805044765 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fda840a2b90 0x7fda840a5050 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fda9c19f5e0 tx=0x7fda8c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:08:10.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:10.845+0000 7fda99600640 1 -- 172.21.15.28:0/3805044765 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fda900bc420 con 0x7fda9c107d70 2024-05-12T22:08:11.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:11.054+0000 7fdaa2000640 1 -- 172.21.15.28:0/3805044765 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fda9c0624d0 con 0x7fda9c107d70 2024-05-12T22:08:11.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:11.055+0000 7fda99600640 1 -- 172.21.15.28:0/3805044765 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) ==== 100+0+11 (secure 0 0 0) 0x7fda900bbd20 con 0x7fda9c107d70 2024-05-12T22:08:11.055 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-05-12T22:08:11.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:11.057+0000 7fdaa2000640 1 -- 172.21.15.28:0/3805044765 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fda840a2b90 msgr2=0x7fda840a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:11.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:11.057+0000 7fdaa2000640 1 --2- 172.21.15.28:0/3805044765 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fda840a2b90 0x7fda840a5050 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fda9c19f5e0 tx=0x7fda8c005f90 comp rx=0 tx=0).stop 2024-05-12T22:08:11.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:11.057+0000 7fdaa2000640 1 -- 172.21.15.28:0/3805044765 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fda9c107d70 msgr2=0x7fda9c19e0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:11.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:11.057+0000 7fdaa2000640 1 --2- 172.21.15.28:0/3805044765 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fda9c107d70 0x7fda9c19e0d0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fda90031440 tx=0x7fda90032ce0 comp rx=0 tx=0).stop 2024-05-12T22:08:11.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:11.058+0000 7fdaa2000640 1 -- 172.21.15.28:0/3805044765 shutdown_connections 2024-05-12T22:08:11.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:11.058+0000 7fdaa2000640 1 --2- 172.21.15.28:0/3805044765 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fda840a2b90 0x7fda840a5050 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:11.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:11.058+0000 7fdaa2000640 1 --2- 172.21.15.28:0/3805044765 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fda9c109920 0x7fda9c1a5680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:11.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:11.058+0000 7fdaa2000640 1 --2- 172.21.15.28:0/3805044765 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda9c108f60 0x7fda9c19e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:11.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:11.058+0000 7fdaa2000640 1 --2- 172.21.15.28:0/3805044765 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fda9c107d70 0x7fda9c19e0d0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:11.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:11.058+0000 7fdaa2000640 1 -- 172.21.15.28:0/3805044765 >> 172.21.15.28:0/3805044765 conn(0x7fda9c076d00 msgr2=0x7fda9c10c260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:08:11.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:11.058+0000 7fdaa2000640 1 -- 172.21.15.28:0/3805044765 shutdown_connections 2024-05-12T22:08:11.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:11.058+0000 7fdaa2000640 1 -- 172.21.15.28:0/3805044765 wait complete. 2024-05-12T22:08:11.077 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-12T22:08:11.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:10 smithi186 ceph-mon[34785]: pgmap v360: 97 pgs: 97 active+clean; 641 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-12T22:08:11.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:10 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/3005760130' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-12T22:08:11.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:10 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/3889933315' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-12T22:08:11.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:10 smithi086 ceph-mon[34297]: pgmap v360: 97 pgs: 97 active+clean; 641 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-12T22:08:11.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:10 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/3005760130' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-12T22:08:11.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:10 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/3889933315' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-12T22:08:11.284 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:10 smithi028 ceph-mon[29847]: pgmap v360: 97 pgs: 97 active+clean; 641 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-12T22:08:11.284 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:10 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/3005760130' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-12T22:08:11.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:10 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/3889933315' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-12T22:08:11.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:11.290+0000 7f57bec00640 1 -- 172.21.15.28:0/3925069198 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f57b8105660 msgr2=0x7f57b810bef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:11.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:11.290+0000 7f57bec00640 1 --2- 172.21.15.28:0/3925069198 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f57b8105660 0x7f57b810bef0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f57b0009860 tx=0x7f57b002a340 comp rx=0 tx=0).stop 2024-05-12T22:08:11.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:11.291+0000 7f57bec00640 1 -- 172.21.15.28:0/3925069198 shutdown_connections 2024-05-12T22:08:11.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:11.291+0000 7f57bec00640 1 --2- 172.21.15.28:0/3925069198 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f57b8105660 0x7f57b810bef0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:11.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:11.291+0000 7f57bec00640 1 --2- 172.21.15.28:0/3925069198 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f57b8104ca0 0x7f57b8105120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:11.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:11.291+0000 7f57bec00640 1 --2- 172.21.15.28:0/3925069198 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57b8103ab0 0x7f57b8103ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:11.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:11.291+0000 7f57bec00640 1 -- 172.21.15.28:0/3925069198 >> 172.21.15.28:0/3925069198 conn(0x7f57b80ff020 msgr2=0x7f57b8101460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:08:11.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:11.291+0000 7f57bec00640 1 -- 172.21.15.28:0/3925069198 shutdown_connections 2024-05-12T22:08:11.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:11.291+0000 7f57bec00640 1 -- 172.21.15.28:0/3925069198 wait complete. 2024-05-12T22:08:11.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:11.292+0000 7f57bec00640 1 Processor -- start 2024-05-12T22:08:11.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:11.292+0000 7f57bec00640 1 -- start start 2024-05-12T22:08:11.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:11.292+0000 7f57bec00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f57b8103ab0 0x7f57b8199d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:11.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:11.292+0000 7f57bec00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57b8104ca0 0x7f57b819a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:11.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:11.292+0000 7f57bec00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f57b8105660 0x7f57b81a12c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:11.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:11.292+0000 7f57bec00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f57b810f710 con 0x7f57b8104ca0 2024-05-12T22:08:11.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:11.292+0000 7f57bec00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f57b810f590 con 0x7f57b8103ab0 2024-05-12T22:08:11.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:11.292+0000 7f57bec00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f57b810f890 con 0x7f57b8105660 2024-05-12T22:08:11.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:11.293+0000 7f57b7e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f57b8103ab0 0x7f57b8199d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:11.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:11.293+0000 7f57b7e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f57b8103ab0 0x7f57b8199d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:51062/0 (socket says 172.21.15.28:51062) 2024-05-12T22:08:11.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:11.293+0000 7f57b7e00640 1 -- 172.21.15.28:0/1558589318 learned_addr learned my addr 172.21.15.28:0/1558589318 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:08:11.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:11.293+0000 7f57bcc00640 1 --2- 172.21.15.28:0/1558589318 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f57b8105660 0x7f57b81a12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:11.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:11.293+0000 7f57b7400640 1 --2- 172.21.15.28:0/1558589318 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57b8104ca0 0x7f57b819a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:11.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:11.293+0000 7f57b7e00640 1 -- 172.21.15.28:0/1558589318 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f57b8105660 msgr2=0x7f57b81a12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:11.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:11.293+0000 7f57b7e00640 1 --2- 172.21.15.28:0/1558589318 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f57b8105660 0x7f57b81a12c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:11.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:11.293+0000 7f57b7e00640 1 -- 172.21.15.28:0/1558589318 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57b8104ca0 msgr2=0x7f57b819a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:11.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:11.293+0000 7f57b7e00640 1 --2- 172.21.15.28:0/1558589318 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57b8104ca0 0x7f57b819a290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:11.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:11.293+0000 7f57b7e00640 1 -- 172.21.15.28:0/1558589318 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57b81a1960 con 0x7f57b8103ab0 2024-05-12T22:08:11.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:11.293+0000 7f57b7e00640 1 --2- 172.21.15.28:0/1558589318 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f57b8103ab0 0x7f57b8199d50 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f57a800e9a0 tx=0x7f57a800ee60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:08:11.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:11.294+0000 7f57b4c00640 1 -- 172.21.15.28:0/1558589318 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57a800cd80 con 0x7f57b8103ab0 2024-05-12T22:08:11.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:11.294+0000 7f57bec00640 1 -- 172.21.15.28:0/1558589318 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57b81a1c50 con 0x7f57b8103ab0 2024-05-12T22:08:11.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:11.294+0000 7f57b4c00640 1 -- 172.21.15.28:0/1558589318 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f57a8010910 con 0x7f57b8103ab0 2024-05-12T22:08:11.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:11.294+0000 7f57b4c00640 1 -- 172.21.15.28:0/1558589318 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57a8018a40 con 0x7f57b8103ab0 2024-05-12T22:08:11.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:11.294+0000 7f57bec00640 1 -- 172.21.15.28:0/1558589318 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57b81a21a0 con 0x7f57b8103ab0 2024-05-12T22:08:11.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:11.295+0000 7f57bec00640 1 -- 172.21.15.28:0/1558589318 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f57b810ad70 con 0x7f57b8103ab0 2024-05-12T22:08:11.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:11.297+0000 7f57b4c00640 1 -- 172.21.15.28:0/1558589318 <== mon.2 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f57a8010ac0 con 0x7f57b8103ab0 2024-05-12T22:08:11.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:11.299+0000 7f57b4c00640 1 --2- 172.21.15.28:0/1558589318 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f579c0a2b90 0x7f579c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:11.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:11.299+0000 7f57b4c00640 1 -- 172.21.15.28:0/1558589318 <== mon.2 v2:172.21.15.86:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f57a80cfa80 con 0x7f57b8103ab0 2024-05-12T22:08:11.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:11.299+0000 7f57b7400640 1 --2- 172.21.15.28:0/1558589318 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f579c0a2b90 0x7f579c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:11.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:11.300+0000 7f57b7400640 1 --2- 172.21.15.28:0/1558589318 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f579c0a2b90 0x7f579c0a5050 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f57a0005fd0 tx=0x7f57a0009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:08:11.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:11.301+0000 7f57b4c00640 1 -- 172.21.15.28:0/1558589318 <== mon.2 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f57a809c990 con 0x7f57b8103ab0 2024-05-12T22:08:11.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:11.546+0000 7f57bec00640 1 -- 172.21.15.28:0/1558589318 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f57b802d3a0 con 0x7f57b8103ab0 2024-05-12T22:08:11.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:11.547+0000 7f57b4c00640 1 -- 172.21.15.28:0/1558589318 <== mon.2 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11864 (secure 0 0 0) 0x7f57a809c290 con 0x7f57b8103ab0 2024-05-12T22:08:11.548 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-12T22:08:11.548 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1844141745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1844141745},{"type":"v1","addr":"172.21.15.186:6833","nonce":1844141745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6833/2960763758","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":2960763758},{"type":"v1","addr":"172.21.15.86:6833","nonce":2960763758}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24470,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/4131134219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":4131134219},{"type":"v1","addr":"172.21.15.28:6835","nonce":4131134219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24481,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/1727869064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1727869064},{"type":"v1","addr":"172.21.15.186:6835","nonce":1727869064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24482,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2938260704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2938260704},{"type":"v1","addr":"172.21.15.28:6837","nonce":2938260704}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24494,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3473700910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3473700910},{"type":"v1","addr":"172.21.15.86:6835","nonce":3473700910}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-12T22:06:33.426986+0000","modified":"2024-05-12T22:06:38.789246+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24581,"mds_1":24550,"mds_2":24532},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24499":{"gid":24499,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6837/3600809235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":3600809235},{"type":"v1","addr":"172.21.15.186:6837","nonce":3600809235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"i","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6839/2399610807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":2399610807},{"type":"v1","addr":"172.21.15.186:6839","nonce":2399610807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24530":{"gid":24530,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.86:6837/3782448884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":3782448884},{"type":"v1","addr":"172.21.15.86:6837","nonce":3782448884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24532":{"gid":24532,"name":"g","rank":2,"incarnation":21,"state":"up:active","state_seq":6,"addr":"172.21.15.28:6839/123892158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":123892158},{"type":"v1","addr":"172.21.15.28:6839","nonce":123892158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24550":{"gid":24550,"name":"a","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.28:6841/761183870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":761183870},{"type":"v1","addr":"172.21.15.28:6841","nonce":761183870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24581":{"gid":24581,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6839/1251999130","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6838","nonce":1251999130},{"type":"v1","addr":"172.21.15.86:6839","nonce":1251999130}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24581,"qdb_cluster":[24550,24532,24581]},"id":1}]} 2024-05-12T22:08:11.548 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 23 2024-05-12T22:08:11.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:11.550+0000 7f57bec00640 1 -- 172.21.15.28:0/1558589318 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f579c0a2b90 msgr2=0x7f579c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:11.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:11.550+0000 7f57bec00640 1 --2- 172.21.15.28:0/1558589318 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f579c0a2b90 0x7f579c0a5050 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f57a0005fd0 tx=0x7f57a0009500 comp rx=0 tx=0).stop 2024-05-12T22:08:11.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:11.550+0000 7f57bec00640 1 -- 172.21.15.28:0/1558589318 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f57b8103ab0 msgr2=0x7f57b8199d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:11.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:11.550+0000 7f57bec00640 1 --2- 172.21.15.28:0/1558589318 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f57b8103ab0 0x7f57b8199d50 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f57a800e9a0 tx=0x7f57a800ee60 comp rx=0 tx=0).stop 2024-05-12T22:08:11.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:11.550+0000 7f57bec00640 1 -- 172.21.15.28:0/1558589318 shutdown_connections 2024-05-12T22:08:11.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:11.550+0000 7f57bec00640 1 --2- 172.21.15.28:0/1558589318 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f579c0a2b90 0x7f579c0a5050 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:11.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:11.550+0000 7f57bec00640 1 --2- 172.21.15.28:0/1558589318 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f57b8105660 0x7f57b81a12c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:11.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:11.550+0000 7f57bec00640 1 --2- 172.21.15.28:0/1558589318 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57b8104ca0 0x7f57b819a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:11.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:11.550+0000 7f57bec00640 1 --2- 172.21.15.28:0/1558589318 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f57b8103ab0 0x7f57b8199d50 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:11.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:11.550+0000 7f57bec00640 1 -- 172.21.15.28:0/1558589318 >> 172.21.15.28:0/1558589318 conn(0x7f57b80ff020 msgr2=0x7f57b8107f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:08:11.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:11.551+0000 7f57bec00640 1 -- 172.21.15.28:0/1558589318 shutdown_connections 2024-05-12T22:08:11.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:11.551+0000 7f57bec00640 1 -- 172.21.15.28:0/1558589318 wait complete. 2024-05-12T22:08:11.571 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-12T22:08:11.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:11.777+0000 7fe37b200640 1 -- 172.21.15.28:0/2987509269 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe37c107cb0 msgr2=0x7fe37c1080d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:11.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:11.777+0000 7fe37b200640 1 --2- 172.21.15.28:0/2987509269 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe37c107cb0 0x7fe37c1080d0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fe36c009860 tx=0x7fe36c02a340 comp rx=0 tx=0).stop 2024-05-12T22:08:11.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:11.777+0000 7fe37b200640 1 -- 172.21.15.28:0/2987509269 shutdown_connections 2024-05-12T22:08:11.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:11.777+0000 7fe37b200640 1 --2- 172.21.15.28:0/2987509269 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe37c109860 0x7fe37c1100f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:11.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:11.777+0000 7fe37b200640 1 --2- 172.21.15.28:0/2987509269 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe37c108ea0 0x7fe37c109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:11.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:11.777+0000 7fe37b200640 1 --2- 172.21.15.28:0/2987509269 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe37c107cb0 0x7fe37c1080d0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:11.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:11.777+0000 7fe37b200640 1 -- 172.21.15.28:0/2987509269 >> 172.21.15.28:0/2987509269 conn(0x7fe37c076c70 msgr2=0x7fe37c079090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:08:11.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:11.777+0000 7fe37b200640 1 -- 172.21.15.28:0/2987509269 shutdown_connections 2024-05-12T22:08:11.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:11.777+0000 7fe37b200640 1 -- 172.21.15.28:0/2987509269 wait complete. 2024-05-12T22:08:11.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:11.778+0000 7fe37b200640 1 Processor -- start 2024-05-12T22:08:11.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:11.779+0000 7fe37b200640 1 -- start start 2024-05-12T22:08:11.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:11.779+0000 7fe37b200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe37c107cb0 0x7fe37c19dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:11.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:11.779+0000 7fe37b200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe37c108ea0 0x7fe37c19e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:11.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:11.779+0000 7fe37b200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe37c109860 0x7fe37c1a5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:11.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:11.779+0000 7fe37b200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe37c113900 con 0x7fe37c109860 2024-05-12T22:08:11.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:11.779+0000 7fe37b200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fe37c113780 con 0x7fe37c108ea0 2024-05-12T22:08:11.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:11.779+0000 7fe37b200640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe37c113a80 con 0x7fe37c107cb0 2024-05-12T22:08:11.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:11.779+0000 7fe37a800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe37c109860 0x7fe37c1a5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:11.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:11.779+0000 7fe37a800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe37c109860 0x7fe37c1a5560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38240/0 (socket says 172.21.15.28:38240) 2024-05-12T22:08:11.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:11.779+0000 7fe379e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe37c107cb0 0x7fe37c19dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:11.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:11.779+0000 7fe379400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe37c108ea0 0x7fe37c19e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:11.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:11.779+0000 7fe37a800640 1 -- 172.21.15.28:0/921943428 learned_addr learned my addr 172.21.15.28:0/921943428 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:08:11.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:11.780+0000 7fe37a800640 1 -- 172.21.15.28:0/921943428 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe37c107cb0 msgr2=0x7fe37c19dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:11.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:11.780+0000 7fe37a800640 1 --2- 172.21.15.28:0/921943428 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe37c107cb0 0x7fe37c19dfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:11.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:11.780+0000 7fe37a800640 1 -- 172.21.15.28:0/921943428 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe37c108ea0 msgr2=0x7fe37c19e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:11.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:11.780+0000 7fe37a800640 1 --2- 172.21.15.28:0/921943428 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe37c108ea0 0x7fe37c19e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:11.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:11.780+0000 7fe37a800640 1 -- 172.21.15.28:0/921943428 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe37c1a5bd0 con 0x7fe37c109860 2024-05-12T22:08:11.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:11.780+0000 7fe379400640 1 --2- 172.21.15.28:0/921943428 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe37c108ea0 0x7fe37c19e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:08:11.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:11.780+0000 7fe37a800640 1 --2- 172.21.15.28:0/921943428 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe37c109860 0x7fe37c1a5560 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fe37400eb10 tx=0x7fe37400ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:08:11.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:11.780+0000 7fe372a00640 1 -- 172.21.15.28:0/921943428 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe37400ce20 con 0x7fe37c109860 2024-05-12T22:08:11.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:11.780+0000 7fe372a00640 1 -- 172.21.15.28:0/921943428 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe374005020 con 0x7fe37c109860 2024-05-12T22:08:11.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:11.781+0000 7fe37b200640 1 -- 172.21.15.28:0/921943428 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe37c1a5ec0 con 0x7fe37c109860 2024-05-12T22:08:11.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:11.781+0000 7fe372a00640 1 -- 172.21.15.28:0/921943428 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe374010690 con 0x7fe37c109860 2024-05-12T22:08:11.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:11.781+0000 7fe37b200640 1 -- 172.21.15.28:0/921943428 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe37c1a6380 con 0x7fe37c109860 2024-05-12T22:08:11.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:11.782+0000 7fe37b200640 1 -- 172.21.15.28:0/921943428 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe37c10f050 con 0x7fe37c109860 2024-05-12T22:08:11.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:11.784+0000 7fe372a00640 1 -- 172.21.15.28:0/921943428 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fe374005540 con 0x7fe37c109860 2024-05-12T22:08:11.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:11.785+0000 7fe372a00640 1 --2- 172.21.15.28:0/921943428 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fe3600a2c10 0x7fe3600a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:11.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:11.785+0000 7fe372a00640 1 -- 172.21.15.28:0/921943428 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7fe3740cf590 con 0x7fe37c109860 2024-05-12T22:08:11.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:11.786+0000 7fe379e00640 1 --2- 172.21.15.28:0/921943428 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fe3600a2c10 0x7fe3600a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:11.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:11.786+0000 7fe379e00640 1 --2- 172.21.15.28:0/921943428 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fe3600a2c10 0x7fe3600a50d0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fe36c030410 tx=0x7fe36c030550 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:08:11.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:11.790+0000 7fe372a00640 1 -- 172.21.15.28:0/921943428 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fe37409c4a0 con 0x7fe37c109860 2024-05-12T22:08:11.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:11.996+0000 7fe37b200640 1 -- 172.21.15.28:0/921943428 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe37c0624d0 con 0x7fe37c109860 2024-05-12T22:08:12.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:12.000+0000 7fe372a00640 1 -- 172.21.15.28:0/921943428 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22595 (secure 0 0 0) 0x7fe37409bda0 con 0x7fe37c109860 2024-05-12T22:08:12.001 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-12T22:08:12.001 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":88,"fsid":"6b92e7b4-10aa-11ef-bc98-c7b262605968","created":"2024-05-12T21:57:37.276782+0000","modified":"2024-05-12T22:06:46.768798+0000","last_up_change":"2024-05-12T22:05:37.929562+0000","last_in_change":"2024-05-12T22:05:18.750768+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-12T22:01:32.292826+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-12T22:06:31.540263+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-12T22:06:32.906118+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6992ce0a-844b-40e3-8ed1-181c8474dccf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":906003882},{"type":"v1","addr":"172.21.15.28:6803","nonce":906003882}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":906003882},{"type":"v1","addr":"172.21.15.28:6805","nonce":906003882}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":906003882},{"type":"v1","addr":"172.21.15.28:6809","nonce":906003882}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":906003882},{"type":"v1","addr":"172.21.15.28:6807","nonce":906003882}]},"public_addr":"172.21.15.28:6803/906003882","cluster_addr":"172.21.15.28:6805/906003882","heartbeat_back_addr":"172.21.15.28:6809/906003882","heartbeat_front_addr":"172.21.15.28:6807/906003882","state":["exists","up"]},{"osd":1,"uuid":"ced20c96-c481-4d48-90d1-6b43228959ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":3354141819},{"type":"v1","addr":"172.21.15.86:6801","nonce":3354141819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":3354141819},{"type":"v1","addr":"172.21.15.86:6803","nonce":3354141819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":3354141819},{"type":"v1","addr":"172.21.15.86:6807","nonce":3354141819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":3354141819},{"type":"v1","addr":"172.21.15.86:6805","nonce":3354141819}]},"public_addr":"172.21.15.86:6801/3354141819","cluster_addr":"172.21.15.86:6803/3354141819","heartbeat_back_addr":"172.21.15.86:6807/3354141819","heartbeat_front_addr":"172.21.15.86:6805/3354141819","state":["exists","up"]},{"osd":2,"uuid":"54a2f221-fa65-4795-9f5e-e660cc7c0098","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":906459365},{"type":"v1","addr":"172.21.15.186:6801","nonce":906459365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":906459365},{"type":"v1","addr":"172.21.15.186:6803","nonce":906459365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":906459365},{"type":"v1","addr":"172.21.15.186:6807","nonce":906459365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":906459365},{"type":"v1","addr":"172.21.15.186:6805","nonce":906459365}]},"public_addr":"172.21.15.186:6801/906459365","cluster_addr":"172.21.15.186:6803/906459365","heartbeat_back_addr":"172.21.15.186:6807/906459365","heartbeat_front_addr":"172.21.15.186:6805/906459365","state":["exists","up"]},{"osd":3,"uuid":"86836737-ac9a-487b-93f2-7fcd3c406960","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":4182884436},{"type":"v1","addr":"172.21.15.28:6811","nonce":4182884436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":4182884436},{"type":"v1","addr":"172.21.15.28:6813","nonce":4182884436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":4182884436},{"type":"v1","addr":"172.21.15.28:6817","nonce":4182884436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":4182884436},{"type":"v1","addr":"172.21.15.28:6815","nonce":4182884436}]},"public_addr":"172.21.15.28:6811/4182884436","cluster_addr":"172.21.15.28:6813/4182884436","heartbeat_back_addr":"172.21.15.28:6817/4182884436","heartbeat_front_addr":"172.21.15.28:6815/4182884436","state":["exists","up"]},{"osd":4,"uuid":"d162ebcd-83c0-4fd5-9925-28b2cecea935","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2596998708},{"type":"v1","addr":"172.21.15.86:6809","nonce":2596998708}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2596998708},{"type":"v1","addr":"172.21.15.86:6811","nonce":2596998708}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2596998708},{"type":"v1","addr":"172.21.15.86:6815","nonce":2596998708}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2596998708},{"type":"v1","addr":"172.21.15.86:6813","nonce":2596998708}]},"public_addr":"172.21.15.86:6809/2596998708","cluster_addr":"172.21.15.86:6811/2596998708","heartbeat_back_addr":"172.21.15.86:6815/2596998708","heartbeat_front_addr":"172.21.15.86:6813/2596998708","state":["exists","up"]},{"osd":5,"uuid":"c794336a-a105-425c-b6c7-a32b90c05bb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":1944205756},{"type":"v1","addr":"172.21.15.186:6809","nonce":1944205756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":1944205756},{"type":"v1","addr":"172.21.15.186:6811","nonce":1944205756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":1944205756},{"type":"v1","addr":"172.21.15.186:6815","nonce":1944205756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":1944205756},{"type":"v1","addr":"172.21.15.186:6813","nonce":1944205756}]},"public_addr":"172.21.15.186:6809/1944205756","cluster_addr":"172.21.15.186:6811/1944205756","heartbeat_back_addr":"172.21.15.186:6815/1944205756","heartbeat_front_addr":"172.21.15.186:6813/1944205756","state":["exists","up"]},{"osd":6,"uuid":"516932ac-af00-45b7-ad3c-d6dec7db442f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":1642308925},{"type":"v1","addr":"172.21.15.28:6819","nonce":1642308925}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":1642308925},{"type":"v1","addr":"172.21.15.28:6821","nonce":1642308925}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1642308925},{"type":"v1","addr":"172.21.15.28:6825","nonce":1642308925}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":1642308925},{"type":"v1","addr":"172.21.15.28:6823","nonce":1642308925}]},"public_addr":"172.21.15.28:6819/1642308925","cluster_addr":"172.21.15.28:6821/1642308925","heartbeat_back_addr":"172.21.15.28:6825/1642308925","heartbeat_front_addr":"172.21.15.28:6823/1642308925","state":["exists","up"]},{"osd":7,"uuid":"ca7a059f-0414-438e-b2d1-a10f348036be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":726062711},{"type":"v1","addr":"172.21.15.86:6817","nonce":726062711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":726062711},{"type":"v1","addr":"172.21.15.86:6819","nonce":726062711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":726062711},{"type":"v1","addr":"172.21.15.86:6823","nonce":726062711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":726062711},{"type":"v1","addr":"172.21.15.86:6821","nonce":726062711}]},"public_addr":"172.21.15.86:6817/726062711","cluster_addr":"172.21.15.86:6819/726062711","heartbeat_back_addr":"172.21.15.86:6823/726062711","heartbeat_front_addr":"172.21.15.86:6821/726062711","state":["exists","up"]},{"osd":8,"uuid":"92e3f41f-aebd-422e-a99e-555f7ba4aa53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":3007687277},{"type":"v1","addr":"172.21.15.186:6817","nonce":3007687277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":3007687277},{"type":"v1","addr":"172.21.15.186:6819","nonce":3007687277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":3007687277},{"type":"v1","addr":"172.21.15.186:6823","nonce":3007687277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":3007687277},{"type":"v1","addr":"172.21.15.186:6821","nonce":3007687277}]},"public_addr":"172.21.15.186:6817/3007687277","cluster_addr":"172.21.15.186:6819/3007687277","heartbeat_back_addr":"172.21.15.186:6823/3007687277","heartbeat_front_addr":"172.21.15.186:6821/3007687277","state":["exists","up"]},{"osd":9,"uuid":"38ff7d9d-ee85-404c-8c59-adc327e1ccfa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":136220472},{"type":"v1","addr":"172.21.15.28:6827","nonce":136220472}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":136220472},{"type":"v1","addr":"172.21.15.28:6829","nonce":136220472}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":136220472},{"type":"v1","addr":"172.21.15.28:6833","nonce":136220472}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":136220472},{"type":"v1","addr":"172.21.15.28:6831","nonce":136220472}]},"public_addr":"172.21.15.28:6827/136220472","cluster_addr":"172.21.15.28:6829/136220472","heartbeat_back_addr":"172.21.15.28:6833/136220472","heartbeat_front_addr":"172.21.15.28:6831/136220472","state":["exists","up"]},{"osd":10,"uuid":"5e9ff657-afa8-4bab-8491-f209038392b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1189968310},{"type":"v1","addr":"172.21.15.86:6825","nonce":1189968310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":1189968310},{"type":"v1","addr":"172.21.15.86:6827","nonce":1189968310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":1189968310},{"type":"v1","addr":"172.21.15.86:6831","nonce":1189968310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":1189968310},{"type":"v1","addr":"172.21.15.86:6829","nonce":1189968310}]},"public_addr":"172.21.15.86:6825/1189968310","cluster_addr":"172.21.15.86:6827/1189968310","heartbeat_back_addr":"172.21.15.86:6831/1189968310","heartbeat_front_addr":"172.21.15.86:6829/1189968310","state":["exists","up"]},{"osd":11,"uuid":"f112e3c6-65b6-4253-a966-62001db7d069","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3801668110},{"type":"v1","addr":"172.21.15.186:6825","nonce":3801668110}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3801668110},{"type":"v1","addr":"172.21.15.186:6827","nonce":3801668110}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3801668110},{"type":"v1","addr":"172.21.15.186:6831","nonce":3801668110}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3801668110},{"type":"v1","addr":"172.21.15.186:6829","nonce":3801668110}]},"public_addr":"172.21.15.186:6825/3801668110","cluster_addr":"172.21.15.186:6827/3801668110","heartbeat_back_addr":"172.21.15.186:6831/3801668110","heartbeat_front_addr":"172.21.15.186:6829/3801668110","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:00:40.483623+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:01:05.284385+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:01:28.645716+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:02:01.295092+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:02:26.486601+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:02:50.807478+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:03:24.756169+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:03:49.975173+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:04:13.772233+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:04:46.758555+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:05:12.331685+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:05:36.462548+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":2,"to":1}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.25","mappings":[{"from":5,"to":9}]},{"pgid":"3.27","mappings":[{"from":4,"to":8}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/1177009533":"2024-05-13T21:58:45.550459+0000","172.21.15.28:6800/3760499540":"2024-05-13T21:58:45.550459+0000","172.21.15.28:6801/3783742728":"2024-05-13T21:58:10.421564+0000","172.21.15.28:6800/3783742728":"2024-05-13T21:58:10.421564+0000","172.21.15.28:0/2229156260":"2024-05-13T21:58:45.550459+0000","172.21.15.28:0/1202140963":"2024-05-13T21:58:10.421564+0000","172.21.15.28:0/2924223784":"2024-05-13T21:58:45.550459+0000","172.21.15.28:6801/3760499540":"2024-05-13T21:58:45.550459+0000","172.21.15.28:0/1240027206":"2024-05-13T21:58:10.421564+0000","172.21.15.28:0/171417728":"2024-05-13T21:58:10.421564+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-12T22:08:12.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:12.002+0000 7fe37b200640 1 -- 172.21.15.28:0/921943428 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fe3600a2c10 msgr2=0x7fe3600a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:12.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:12.002+0000 7fe37b200640 1 --2- 172.21.15.28:0/921943428 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fe3600a2c10 0x7fe3600a50d0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fe36c030410 tx=0x7fe36c030550 comp rx=0 tx=0).stop 2024-05-12T22:08:12.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:12.002+0000 7fe37b200640 1 -- 172.21.15.28:0/921943428 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe37c109860 msgr2=0x7fe37c1a5560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:12.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:12.002+0000 7fe37b200640 1 --2- 172.21.15.28:0/921943428 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe37c109860 0x7fe37c1a5560 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fe37400eb10 tx=0x7fe37400ef10 comp rx=0 tx=0).stop 2024-05-12T22:08:12.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:12.003+0000 7fe37b200640 1 -- 172.21.15.28:0/921943428 shutdown_connections 2024-05-12T22:08:12.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:12.003+0000 7fe37b200640 1 --2- 172.21.15.28:0/921943428 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fe3600a2c10 0x7fe3600a50d0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:12.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:12.003+0000 7fe37b200640 1 --2- 172.21.15.28:0/921943428 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe37c109860 0x7fe37c1a5560 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:12.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:12.003+0000 7fe37b200640 1 --2- 172.21.15.28:0/921943428 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe37c108ea0 0x7fe37c19e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:12.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:12.003+0000 7fe37b200640 1 --2- 172.21.15.28:0/921943428 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe37c107cb0 0x7fe37c19dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:12.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:12.003+0000 7fe37b200640 1 -- 172.21.15.28:0/921943428 >> 172.21.15.28:0/921943428 conn(0x7fe37c076c70 msgr2=0x7fe37c10c220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:08:12.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:12.003+0000 7fe37b200640 1 -- 172.21.15.28:0/921943428 shutdown_connections 2024-05-12T22:08:12.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:12.003+0000 7fe37b200640 1 -- 172.21.15.28:0/921943428 wait complete. 2024-05-12T22:08:12.023 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-12T22:01:32.292826+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-12T22:06:31.540263+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-12T22:06:32.906118+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-12T22:08:12.024 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-12T22:08:12.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:11 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/3805044765' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-12T22:08:12.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:11 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/1558589318' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:08:12.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:12.239+0000 7f27c7c00640 1 -- 172.21.15.28:0/1823002628 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27c006a670 msgr2=0x7f27c010b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:12.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:12.239+0000 7f27c7c00640 1 --2- 172.21.15.28:0/1823002628 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27c006a670 0x7f27c010b340 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f27b0009ca0 tx=0x7f27b002a8c0 comp rx=0 tx=0).stop 2024-05-12T22:08:12.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:12.239+0000 7f27c7c00640 1 -- 172.21.15.28:0/1823002628 shutdown_connections 2024-05-12T22:08:12.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:12.239+0000 7f27c7c00640 1 --2- 172.21.15.28:0/1823002628 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27c010e370 0x7f27c0110830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:12.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:12.239+0000 7f27c7c00640 1 --2- 172.21.15.28:0/1823002628 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f27c010b910 0x7f27c010dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:12.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:12.239+0000 7f27c7c00640 1 --2- 172.21.15.28:0/1823002628 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27c006a670 0x7f27c010b340 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:12.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:12.239+0000 7f27c7c00640 1 -- 172.21.15.28:0/1823002628 >> 172.21.15.28:0/1823002628 conn(0x7f27c006ea60 msgr2=0x7f27c0070e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:08:12.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:12.239+0000 7f27c7c00640 1 -- 172.21.15.28:0/1823002628 shutdown_connections 2024-05-12T22:08:12.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:12.239+0000 7f27c7c00640 1 -- 172.21.15.28:0/1823002628 wait complete. 2024-05-12T22:08:12.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:12.240+0000 7f27c7c00640 1 Processor -- start 2024-05-12T22:08:12.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:12.240+0000 7f27c7c00640 1 -- start start 2024-05-12T22:08:12.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:12.241+0000 7f27c7c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27c010b910 0x7f27c01a28a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:12.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:12.241+0000 7f27c7c00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27c010e370 0x7f27c01a2de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:12.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:12.241+0000 7f27c7c00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f27c01a7e40 0x7f27c01aa230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:12.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:12.241+0000 7f27c7c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f27c0113b50 con 0x7f27c010b910 2024-05-12T22:08:12.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:12.241+0000 7f27c7c00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f27c01139d0 con 0x7f27c010e370 2024-05-12T22:08:12.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:12.241+0000 7f27c7c00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f27c0113cd0 con 0x7f27c01a7e40 2024-05-12T22:08:12.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:12.241+0000 7f27c5200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27c010b910 0x7f27c01a28a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:12.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:12.241+0000 7f27bfe00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27c010e370 0x7f27c01a2de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:12.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:12.241+0000 7f27c5200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27c010b910 0x7f27c01a28a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38258/0 (socket says 172.21.15.28:38258) 2024-05-12T22:08:12.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:12.241+0000 7f27c5200640 1 -- 172.21.15.28:0/1628334792 learned_addr learned my addr 172.21.15.28:0/1628334792 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:08:12.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:12.241+0000 7f27c5c00640 1 --2- 172.21.15.28:0/1628334792 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f27c01a7e40 0x7f27c01aa230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:12.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:12.241+0000 7f27c5200640 1 -- 172.21.15.28:0/1628334792 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f27c01a7e40 msgr2=0x7f27c01aa230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:12.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:12.241+0000 7f27c5200640 1 --2- 172.21.15.28:0/1628334792 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f27c01a7e40 0x7f27c01aa230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:12.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:12.241+0000 7f27c5200640 1 -- 172.21.15.28:0/1628334792 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27c010e370 msgr2=0x7f27c01a2de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:12.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:12.241+0000 7f27c5200640 1 --2- 172.21.15.28:0/1628334792 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27c010e370 0x7f27c01a2de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:12.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:12.241+0000 7f27c5200640 1 -- 172.21.15.28:0/1628334792 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27c01aa8a0 con 0x7f27c010b910 2024-05-12T22:08:12.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:12.242+0000 7f27c5200640 1 --2- 172.21.15.28:0/1628334792 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27c010b910 0x7f27c01a28a0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f27b002aeb0 tx=0x7f27b00319f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:08:12.244 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:11 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/3805044765' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-12T22:08:12.244 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:11 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/1558589318' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:08:12.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:12.242+0000 7f27bd600640 1 -- 172.21.15.28:0/1628334792 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27b0003b70 con 0x7f27c010b910 2024-05-12T22:08:12.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:12.242+0000 7f27bd600640 1 -- 172.21.15.28:0/1628334792 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f27b0003d10 con 0x7f27c010b910 2024-05-12T22:08:12.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:12.242+0000 7f27c7c00640 1 -- 172.21.15.28:0/1628334792 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27c01aab30 con 0x7f27c010b910 2024-05-12T22:08:12.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:12.242+0000 7f27bd600640 1 -- 172.21.15.28:0/1628334792 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27b002fae0 con 0x7f27c010b910 2024-05-12T22:08:12.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:12.242+0000 7f27c7c00640 1 -- 172.21.15.28:0/1628334792 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27c01ab080 con 0x7f27c010b910 2024-05-12T22:08:12.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:12.243+0000 7f27c7c00640 1 -- 172.21.15.28:0/1628334792 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f27c0070550 con 0x7f27c010b910 2024-05-12T22:08:12.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:12.245+0000 7f27bd600640 1 -- 172.21.15.28:0/1628334792 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f27b002fc80 con 0x7f27c010b910 2024-05-12T22:08:12.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:12.246+0000 7f27bd600640 1 --2- 172.21.15.28:0/1628334792 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f27a40a2c10 0x7f27a40a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:12.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:12.247+0000 7f27bd600640 1 -- 172.21.15.28:0/1628334792 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f27b00f0200 con 0x7f27c010b910 2024-05-12T22:08:12.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:12.247+0000 7f27bfe00640 1 --2- 172.21.15.28:0/1628334792 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f27a40a2c10 0x7f27a40a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:12.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:12.248+0000 7f27bfe00640 1 --2- 172.21.15.28:0/1628334792 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f27a40a2c10 0x7f27a40a50d0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f27c01a3db0 tx=0x7f27ac010040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:08:12.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:12.251+0000 7f27bd600640 1 -- 172.21.15.28:0/1628334792 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f27b00bd110 con 0x7f27c010b910 2024-05-12T22:08:12.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:11 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/3805044765' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-12T22:08:12.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:11 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/1558589318' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:08:12.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:12.457+0000 7f27c7c00640 1 -- 172.21.15.28:0/1628334792 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f27c0062430 con 0x7f27c010b910 2024-05-12T22:08:12.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:12.458+0000 7f27bd600640 1 -- 172.21.15.28:0/1628334792 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) ==== 93+0+10 (secure 0 0 0) 0x7f27b00bca10 con 0x7f27c010b910 2024-05-12T22:08:12.459 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-05-12T22:08:12.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:12.461+0000 7f27c7c00640 1 -- 172.21.15.28:0/1628334792 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f27a40a2c10 msgr2=0x7f27a40a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:12.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:12.461+0000 7f27c7c00640 1 --2- 172.21.15.28:0/1628334792 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f27a40a2c10 0x7f27a40a50d0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f27c01a3db0 tx=0x7f27ac010040 comp rx=0 tx=0).stop 2024-05-12T22:08:12.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:12.461+0000 7f27c7c00640 1 -- 172.21.15.28:0/1628334792 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27c010b910 msgr2=0x7f27c01a28a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:12.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:12.461+0000 7f27c7c00640 1 --2- 172.21.15.28:0/1628334792 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27c010b910 0x7f27c01a28a0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f27b002aeb0 tx=0x7f27b00319f0 comp rx=0 tx=0).stop 2024-05-12T22:08:12.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:12.461+0000 7f27c7c00640 1 -- 172.21.15.28:0/1628334792 shutdown_connections 2024-05-12T22:08:12.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:12.461+0000 7f27c7c00640 1 --2- 172.21.15.28:0/1628334792 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f27a40a2c10 0x7f27a40a50d0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:12.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:12.461+0000 7f27c7c00640 1 --2- 172.21.15.28:0/1628334792 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f27c01a7e40 0x7f27c01aa230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:12.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:12.461+0000 7f27c7c00640 1 --2- 172.21.15.28:0/1628334792 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27c010e370 0x7f27c01a2de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:12.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:12.461+0000 7f27c7c00640 1 --2- 172.21.15.28:0/1628334792 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27c010b910 0x7f27c01a28a0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:12.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:12.461+0000 7f27c7c00640 1 -- 172.21.15.28:0/1628334792 >> 172.21.15.28:0/1628334792 conn(0x7f27c006ea60 msgr2=0x7f27c010ae40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:08:12.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:12.461+0000 7f27c7c00640 1 -- 172.21.15.28:0/1628334792 shutdown_connections 2024-05-12T22:08:12.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:12.461+0000 7f27c7c00640 1 -- 172.21.15.28:0/1628334792 wait complete. 2024-05-12T22:08:12.482 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-12T22:08:12.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:12.694+0000 7f2af6200640 1 -- 172.21.15.28:0/2091453526 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2af0107d70 msgr2=0x7f2af0108190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:12.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:12.694+0000 7f2af6200640 1 --2- 172.21.15.28:0/2091453526 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2af0107d70 0x7f2af0108190 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f2ae4009900 tx=0x7f2ae402a3f0 comp rx=0 tx=0).stop 2024-05-12T22:08:12.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:12.694+0000 7f2af6200640 1 -- 172.21.15.28:0/2091453526 shutdown_connections 2024-05-12T22:08:12.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:12.694+0000 7f2af6200640 1 --2- 172.21.15.28:0/2091453526 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2af0109920 0x7f2af01101b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:12.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:12.694+0000 7f2af6200640 1 --2- 172.21.15.28:0/2091453526 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2af0108f60 0x7f2af01093e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:12.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:12.694+0000 7f2af6200640 1 --2- 172.21.15.28:0/2091453526 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2af0107d70 0x7f2af0108190 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:12.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:12.694+0000 7f2af6200640 1 -- 172.21.15.28:0/2091453526 >> 172.21.15.28:0/2091453526 conn(0x7f2af0076d00 msgr2=0x7f2af0079140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:08:12.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:12.695+0000 7f2af6200640 1 -- 172.21.15.28:0/2091453526 shutdown_connections 2024-05-12T22:08:12.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:12.695+0000 7f2af6200640 1 -- 172.21.15.28:0/2091453526 wait complete. 2024-05-12T22:08:12.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:12.696+0000 7f2af6200640 1 Processor -- start 2024-05-12T22:08:12.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:12.696+0000 7f2af6200640 1 -- start start 2024-05-12T22:08:12.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:12.697+0000 7f2af6200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2af0107d70 0x7f2af019e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:12.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:12.697+0000 7f2af6200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2af0108f60 0x7f2af019e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:12.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:12.697+0000 7f2af6200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2af0109920 0x7f2af01a5640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:12.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:12.697+0000 7f2af6200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2af01139c0 con 0x7f2af0108f60 2024-05-12T22:08:12.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:12.697+0000 7f2af6200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f2af0113840 con 0x7f2af0107d70 2024-05-12T22:08:12.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:12.697+0000 7f2af6200640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f2af0113b40 con 0x7f2af0109920 2024-05-12T22:08:12.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:12.697+0000 7f2aefe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2af0108f60 0x7f2af019e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:12.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:12.697+0000 7f2af5800640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2af0109920 0x7f2af01a5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:12.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:12.697+0000 7f2af5800640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2af0109920 0x7f2af01a5640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:34392/0 (socket says 172.21.15.28:34392) 2024-05-12T22:08:12.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:12.697+0000 7f2aefe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2af0108f60 0x7f2af019e5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38270/0 (socket says 172.21.15.28:38270) 2024-05-12T22:08:12.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:12.697+0000 7f2af5800640 1 -- 172.21.15.28:0/2367283631 learned_addr learned my addr 172.21.15.28:0/2367283631 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:08:12.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:12.697+0000 7f2af4e00640 1 --2- 172.21.15.28:0/2367283631 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2af0107d70 0x7f2af019e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:12.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:12.697+0000 7f2af5800640 1 -- 172.21.15.28:0/2367283631 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2af0107d70 msgr2=0x7f2af019e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:12.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:12.697+0000 7f2af5800640 1 --2- 172.21.15.28:0/2367283631 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2af0107d70 0x7f2af019e090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:12.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:12.698+0000 7f2af5800640 1 -- 172.21.15.28:0/2367283631 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2af0108f60 msgr2=0x7f2af019e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:12.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:12.698+0000 7f2af5800640 1 --2- 172.21.15.28:0/2367283631 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2af0108f60 0x7f2af019e5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:12.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:12.698+0000 7f2af5800640 1 -- 172.21.15.28:0/2367283631 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2af01a5cb0 con 0x7f2af0109920 2024-05-12T22:08:12.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:12.698+0000 7f2aefe00640 1 --2- 172.21.15.28:0/2367283631 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2af0108f60 0x7f2af019e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:08:12.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:12.698+0000 7f2af5800640 1 --2- 172.21.15.28:0/2367283631 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2af0109920 0x7f2af01a5640 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f2ae800efd0 tx=0x7f2ae800c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:08:12.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:12.698+0000 7f2aed600640 1 -- 172.21.15.28:0/2367283631 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ae8003b70 con 0x7f2af0109920 2024-05-12T22:08:12.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:12.698+0000 7f2aed600640 1 -- 172.21.15.28:0/2367283631 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2ae8003d10 con 0x7f2af0109920 2024-05-12T22:08:12.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:12.698+0000 7f2af6200640 1 -- 172.21.15.28:0/2367283631 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2af01a5fa0 con 0x7f2af0109920 2024-05-12T22:08:12.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:12.698+0000 7f2aed600640 1 -- 172.21.15.28:0/2367283631 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ae80109f0 con 0x7f2af0109920 2024-05-12T22:08:12.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:12.698+0000 7f2af6200640 1 -- 172.21.15.28:0/2367283631 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2af01a6410 con 0x7f2af0109920 2024-05-12T22:08:12.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:12.699+0000 7f2af6200640 1 -- 172.21.15.28:0/2367283631 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2af010f110 con 0x7f2af0109920 2024-05-12T22:08:12.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:12.701+0000 7f2aed600640 1 -- 172.21.15.28:0/2367283631 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f2ae8004020 con 0x7f2af0109920 2024-05-12T22:08:12.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:12.703+0000 7f2aed600640 1 --2- 172.21.15.28:0/2367283631 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f2ad80a2b90 0x7f2ad80a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:12.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:12.703+0000 7f2aed600640 1 -- 172.21.15.28:0/2367283631 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f2ae80cf4b0 con 0x7f2af0109920 2024-05-12T22:08:12.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:12.703+0000 7f2af4e00640 1 --2- 172.21.15.28:0/2367283631 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f2ad80a2b90 0x7f2ad80a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:12.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:12.704+0000 7f2af4e00640 1 --2- 172.21.15.28:0/2367283631 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f2ad80a2b90 0x7f2ad80a5050 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f2ae4005ab0 tx=0x7f2ae4000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:08:12.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:12.707+0000 7f2aed600640 1 -- 172.21.15.28:0/2367283631 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f2ae809c3c0 con 0x7f2af0109920 2024-05-12T22:08:12.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:12.901+0000 7f2af6200640 1 -- 172.21.15.28:0/2367283631 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f2af0062430 con 0x7f2af0109920 2024-05-12T22:08:12.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:12.902+0000 7f2aed600640 1 -- 172.21.15.28:0/2367283631 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) ==== 104+0+11 (secure 0 0 0) 0x7f2ae809bcc0 con 0x7f2af0109920 2024-05-12T22:08:12.902 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-05-12T22:08:12.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:12.904+0000 7f2af6200640 1 -- 172.21.15.28:0/2367283631 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f2ad80a2b90 msgr2=0x7f2ad80a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:12.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:12.904+0000 7f2af6200640 1 --2- 172.21.15.28:0/2367283631 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f2ad80a2b90 0x7f2ad80a5050 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f2ae4005ab0 tx=0x7f2ae4000bc0 comp rx=0 tx=0).stop 2024-05-12T22:08:12.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:12.904+0000 7f2af6200640 1 -- 172.21.15.28:0/2367283631 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2af0109920 msgr2=0x7f2af01a5640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:12.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:12.904+0000 7f2af6200640 1 --2- 172.21.15.28:0/2367283631 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2af0109920 0x7f2af01a5640 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f2ae800efd0 tx=0x7f2ae800c6a0 comp rx=0 tx=0).stop 2024-05-12T22:08:12.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:12.904+0000 7f2af6200640 1 -- 172.21.15.28:0/2367283631 shutdown_connections 2024-05-12T22:08:12.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:12.904+0000 7f2af6200640 1 --2- 172.21.15.28:0/2367283631 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f2ad80a2b90 0x7f2ad80a5050 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:12.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:12.904+0000 7f2af6200640 1 --2- 172.21.15.28:0/2367283631 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2af0109920 0x7f2af01a5640 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:12.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:12.904+0000 7f2af6200640 1 --2- 172.21.15.28:0/2367283631 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2af0108f60 0x7f2af019e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:12.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:12.904+0000 7f2af6200640 1 --2- 172.21.15.28:0/2367283631 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2af0107d70 0x7f2af019e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:12.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:12.904+0000 7f2af6200640 1 -- 172.21.15.28:0/2367283631 >> 172.21.15.28:0/2367283631 conn(0x7f2af0076d00 msgr2=0x7f2af010c2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:08:12.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:12.904+0000 7f2af6200640 1 -- 172.21.15.28:0/2367283631 shutdown_connections 2024-05-12T22:08:12.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:12.904+0000 7f2af6200640 1 -- 172.21.15.28:0/2367283631 wait complete. 2024-05-12T22:08:12.925 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-12T22:08:13.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:13.139+0000 7f81ad800640 1 -- 172.21.15.28:0/1906442345 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f81a8105520 msgr2=0x7f81a810bdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:13.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:13.139+0000 7f81ad800640 1 --2- 172.21.15.28:0/1906442345 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f81a8105520 0x7f81a810bdb0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f81a0009ca0 tx=0x7f81a002a8c0 comp rx=0 tx=0).stop 2024-05-12T22:08:13.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:13.139+0000 7f81ad800640 1 -- 172.21.15.28:0/1906442345 shutdown_connections 2024-05-12T22:08:13.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:13.139+0000 7f81ad800640 1 --2- 172.21.15.28:0/1906442345 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f81a8105520 0x7f81a810bdb0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:13.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:13.139+0000 7f81ad800640 1 --2- 172.21.15.28:0/1906442345 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f81a8104b60 0x7f81a8104fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:13.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:13.139+0000 7f81ad800640 1 --2- 172.21.15.28:0/1906442345 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f81a8103970 0x7f81a8103d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:13.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:13.139+0000 7f81ad800640 1 -- 172.21.15.28:0/1906442345 >> 172.21.15.28:0/1906442345 conn(0x7f81a80feee0 msgr2=0x7f81a8101320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:08:13.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:13.139+0000 7f81ad800640 1 -- 172.21.15.28:0/1906442345 shutdown_connections 2024-05-12T22:08:13.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:13.139+0000 7f81ad800640 1 -- 172.21.15.28:0/1906442345 wait complete. 2024-05-12T22:08:13.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:13.141+0000 7f81ad800640 1 Processor -- start 2024-05-12T22:08:13.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:13.141+0000 7f81ad800640 1 -- start start 2024-05-12T22:08:13.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:13.141+0000 7f81ad800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f81a8103970 0x7f81a8199bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:13.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:13.141+0000 7f81ad800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f81a8104b60 0x7f81a819a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:13.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:13.141+0000 7f81ad800640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f81a8105520 0x7f81a81a11a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:13.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:13.141+0000 7f81ad800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f81a810f5d0 con 0x7f81a8104b60 2024-05-12T22:08:13.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:13.141+0000 7f81ad800640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f81a810f450 con 0x7f81a8103970 2024-05-12T22:08:13.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:13.141+0000 7f81ad800640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f81a810f750 con 0x7f81a8105520 2024-05-12T22:08:13.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:13.142+0000 7f81a7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f81a8104b60 0x7f81a819a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:13.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:13.142+0000 7f81a7e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f81a8103970 0x7f81a8199bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:13.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:13.142+0000 7f81a7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f81a8104b60 0x7f81a819a130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38302/0 (socket says 172.21.15.28:38302) 2024-05-12T22:08:13.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:13.142+0000 7f81ace00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f81a8105520 0x7f81a81a11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:13.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:13.142+0000 7f81a7e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f81a8103970 0x7f81a8199bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:51130/0 (socket says 172.21.15.28:51130) 2024-05-12T22:08:13.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:13.142+0000 7f81a7400640 1 -- 172.21.15.28:0/3936178429 learned_addr learned my addr 172.21.15.28:0/3936178429 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:08:13.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:13.142+0000 7f81a7400640 1 -- 172.21.15.28:0/3936178429 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f81a8105520 msgr2=0x7f81a81a11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:13.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:13.142+0000 7f81a7400640 1 --2- 172.21.15.28:0/3936178429 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f81a8105520 0x7f81a81a11a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:13.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:13.142+0000 7f81a7400640 1 -- 172.21.15.28:0/3936178429 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f81a8103970 msgr2=0x7f81a8199bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:13.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:13.142+0000 7f81a7400640 1 --2- 172.21.15.28:0/3936178429 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f81a8103970 0x7f81a8199bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:13.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:13.142+0000 7f81a7400640 1 -- 172.21.15.28:0/3936178429 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81a81a1810 con 0x7f81a8104b60 2024-05-12T22:08:13.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:13.142+0000 7f81ace00640 1 --2- 172.21.15.28:0/3936178429 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f81a8105520 0x7f81a81a11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:08:13.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:13.142+0000 7f81a7e00640 1 --2- 172.21.15.28:0/3936178429 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f81a8103970 0x7f81a8199bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:08:13.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:13.142+0000 7f81a7400640 1 --2- 172.21.15.28:0/3936178429 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f81a8104b60 0x7f81a819a130 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f819400e9f0 tx=0x7f819400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:08:13.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:13.143+0000 7f81a4c00640 1 -- 172.21.15.28:0/3936178429 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f819400cd80 con 0x7f81a8104b60 2024-05-12T22:08:13.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:13.143+0000 7f81ad800640 1 -- 172.21.15.28:0/3936178429 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81a81a1b00 con 0x7f81a8104b60 2024-05-12T22:08:13.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:13.143+0000 7f81a4c00640 1 -- 172.21.15.28:0/3936178429 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8194010910 con 0x7f81a8104b60 2024-05-12T22:08:13.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:13.143+0000 7f81a4c00640 1 -- 172.21.15.28:0/3936178429 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81940188f0 con 0x7f81a8104b60 2024-05-12T22:08:13.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:13.143+0000 7f81ad800640 1 -- 172.21.15.28:0/3936178429 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81a81a20a0 con 0x7f81a8104b60 2024-05-12T22:08:13.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:13.144+0000 7f81ad800640 1 -- 172.21.15.28:0/3936178429 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f81a810ac90 con 0x7f81a8104b60 2024-05-12T22:08:13.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:13.147+0000 7f81a4c00640 1 -- 172.21.15.28:0/3936178429 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f8194010ac0 con 0x7f81a8104b60 2024-05-12T22:08:13.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:13.148+0000 7f81a4c00640 1 --2- 172.21.15.28:0/3936178429 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f818c0a2c10 0x7f818c0a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:13.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:13.148+0000 7f81a4c00640 1 -- 172.21.15.28:0/3936178429 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f81940cf3c0 con 0x7f81a8104b60 2024-05-12T22:08:13.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:13.149+0000 7f81a7e00640 1 --2- 172.21.15.28:0/3936178429 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f818c0a2c10 0x7f818c0a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:13.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:13.149+0000 7f81a7e00640 1 --2- 172.21.15.28:0/3936178429 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f818c0a2c10 0x7f818c0a50d0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f81a81049c0 tx=0x7f8198005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:08:13.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:13.153+0000 7f81a4c00640 1 -- 172.21.15.28:0/3936178429 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f819409bc80 con 0x7f81a8104b60 2024-05-12T22:08:13.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:12 smithi086 ceph-mon[34297]: pgmap v361: 97 pgs: 97 active+clean; 641 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:08:13.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:12 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/921943428' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-12T22:08:13.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:12 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/1628334792' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-12T22:08:13.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:12 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/2367283631' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-12T22:08:13.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:12 smithi028 ceph-mon[29847]: pgmap v361: 97 pgs: 97 active+clean; 641 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:08:13.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:12 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/921943428' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-12T22:08:13.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:12 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/1628334792' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-12T22:08:13.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:12 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/2367283631' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-12T22:08:13.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:13.356+0000 7f81ad800640 1 -- 172.21.15.28:0/3936178429 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f81a8062430 con 0x7f81a8104b60 2024-05-12T22:08:13.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:13.357+0000 7f81a4c00640 1 -- 172.21.15.28:0/3936178429 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) ==== 100+0+11 (secure 0 0 0) 0x7f819409ff80 con 0x7f81a8104b60 2024-05-12T22:08:13.357 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-05-12T22:08:13.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:13.359+0000 7f81ad800640 1 -- 172.21.15.28:0/3936178429 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f818c0a2c10 msgr2=0x7f818c0a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:13.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:13.359+0000 7f81ad800640 1 --2- 172.21.15.28:0/3936178429 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f818c0a2c10 0x7f818c0a50d0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f81a81049c0 tx=0x7f8198005f90 comp rx=0 tx=0).stop 2024-05-12T22:08:13.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:13.359+0000 7f81ad800640 1 -- 172.21.15.28:0/3936178429 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f81a8104b60 msgr2=0x7f81a819a130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:13.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:13.359+0000 7f81ad800640 1 --2- 172.21.15.28:0/3936178429 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f81a8104b60 0x7f81a819a130 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f819400e9f0 tx=0x7f819400eeb0 comp rx=0 tx=0).stop 2024-05-12T22:08:13.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:13.359+0000 7f81ad800640 1 -- 172.21.15.28:0/3936178429 shutdown_connections 2024-05-12T22:08:13.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:13.359+0000 7f81ad800640 1 --2- 172.21.15.28:0/3936178429 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f818c0a2c10 0x7f818c0a50d0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:13.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:13.360+0000 7f81ad800640 1 --2- 172.21.15.28:0/3936178429 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f81a8105520 0x7f81a81a11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:13.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:13.360+0000 7f81ad800640 1 --2- 172.21.15.28:0/3936178429 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f81a8104b60 0x7f81a819a130 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:13.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:13.360+0000 7f81ad800640 1 --2- 172.21.15.28:0/3936178429 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f81a8103970 0x7f81a8199bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:13.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:13.360+0000 7f81ad800640 1 -- 172.21.15.28:0/3936178429 >> 172.21.15.28:0/3936178429 conn(0x7f81a80feee0 msgr2=0x7f81a8107e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:08:13.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:13.360+0000 7f81ad800640 1 -- 172.21.15.28:0/3936178429 shutdown_connections 2024-05-12T22:08:13.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:13.360+0000 7f81ad800640 1 -- 172.21.15.28:0/3936178429 wait complete. 2024-05-12T22:08:13.379 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-12T22:08:13.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:12 smithi186 ceph-mon[34785]: pgmap v361: 97 pgs: 97 active+clean; 641 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:08:13.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:12 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/921943428' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-12T22:08:13.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:12 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/1628334792' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-12T22:08:13.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:12 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/2367283631' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-12T22:08:13.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:13.590+0000 7f6112e00640 1 -- 172.21.15.28:0/523957738 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f610c107cb0 msgr2=0x7f610c1080d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:13.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:13.590+0000 7f6112e00640 1 --2- 172.21.15.28:0/523957738 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f610c107cb0 0x7f610c1080d0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f6100009900 tx=0x7f610002a3f0 comp rx=0 tx=0).stop 2024-05-12T22:08:13.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:13.590+0000 7f6112e00640 1 -- 172.21.15.28:0/523957738 shutdown_connections 2024-05-12T22:08:13.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:13.590+0000 7f6112e00640 1 --2- 172.21.15.28:0/523957738 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f610c109860 0x7f610c1100f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:13.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:13.590+0000 7f6112e00640 1 --2- 172.21.15.28:0/523957738 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f610c108ea0 0x7f610c109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:13.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:13.590+0000 7f6112e00640 1 --2- 172.21.15.28:0/523957738 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f610c107cb0 0x7f610c1080d0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:13.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:13.590+0000 7f6112e00640 1 -- 172.21.15.28:0/523957738 >> 172.21.15.28:0/523957738 conn(0x7f610c076c70 msgr2=0x7f610c079090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:08:13.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:13.590+0000 7f6112e00640 1 -- 172.21.15.28:0/523957738 shutdown_connections 2024-05-12T22:08:13.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:13.590+0000 7f6112e00640 1 -- 172.21.15.28:0/523957738 wait complete. 2024-05-12T22:08:13.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:13.591+0000 7f6112e00640 1 Processor -- start 2024-05-12T22:08:13.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:13.591+0000 7f6112e00640 1 -- start start 2024-05-12T22:08:13.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:13.592+0000 7f6112e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f610c107cb0 0x7f610c19dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:13.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:13.592+0000 7f6112e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f610c108ea0 0x7f610c19e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:13.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:13.592+0000 7f6112e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f610c109860 0x7f610c1a5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:13.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:13.592+0000 7f6112e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f610c113900 con 0x7f610c109860 2024-05-12T22:08:13.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:13.592+0000 7f6112e00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f610c113780 con 0x7f610c108ea0 2024-05-12T22:08:13.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:13.592+0000 7f6112e00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f610c113a80 con 0x7f610c107cb0 2024-05-12T22:08:13.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:13.592+0000 7f6111a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f610c107cb0 0x7f610c19dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:13.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:13.592+0000 7f6111000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f610c108ea0 0x7f610c19e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:13.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:13.592+0000 7f6112400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f610c109860 0x7f610c1a5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:13.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:13.592+0000 7f6111a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f610c107cb0 0x7f610c19dfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:34416/0 (socket says 172.21.15.28:34416) 2024-05-12T22:08:13.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:13.592+0000 7f6111a00640 1 -- 172.21.15.28:0/1942267693 learned_addr learned my addr 172.21.15.28:0/1942267693 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:08:13.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:13.592+0000 7f6111000640 1 --2- 172.21.15.28:0/1942267693 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f610c108ea0 0x7f610c19e4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:51150/0 (socket says 172.21.15.28:51150) 2024-05-12T22:08:13.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:13.593+0000 7f6111000640 1 -- 172.21.15.28:0/1942267693 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f610c107cb0 msgr2=0x7f610c19dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:13.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:13.593+0000 7f6111000640 1 --2- 172.21.15.28:0/1942267693 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f610c107cb0 0x7f610c19dfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:13.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:13.593+0000 7f6111000640 1 -- 172.21.15.28:0/1942267693 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f610c109860 msgr2=0x7f610c1a5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:13.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:13.593+0000 7f6111000640 1 --2- 172.21.15.28:0/1942267693 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f610c109860 0x7f610c1a5560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:13.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:13.593+0000 7f6111000640 1 -- 172.21.15.28:0/1942267693 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f610c1a5bd0 con 0x7f610c108ea0 2024-05-12T22:08:13.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:13.593+0000 7f6111000640 1 --2- 172.21.15.28:0/1942267693 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f610c108ea0 0x7f610c19e4f0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f60f800e9f0 tx=0x7f60f800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:08:13.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:13.593+0000 7f610a000640 1 -- 172.21.15.28:0/1942267693 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60f800ccc0 con 0x7f610c108ea0 2024-05-12T22:08:13.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:13.593+0000 7f610a000640 1 -- 172.21.15.28:0/1942267693 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f60f8010910 con 0x7f610c108ea0 2024-05-12T22:08:13.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:13.593+0000 7f610a000640 1 -- 172.21.15.28:0/1942267693 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60f8018960 con 0x7f610c108ea0 2024-05-12T22:08:13.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:13.593+0000 7f6112e00640 1 -- 172.21.15.28:0/1942267693 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f610c1a5ec0 con 0x7f610c108ea0 2024-05-12T22:08:13.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:13.594+0000 7f6112e00640 1 -- 172.21.15.28:0/1942267693 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f610c1a6330 con 0x7f610c108ea0 2024-05-12T22:08:13.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:13.595+0000 7f6112e00640 1 -- 172.21.15.28:0/1942267693 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f610c10f050 con 0x7f610c108ea0 2024-05-12T22:08:13.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:13.597+0000 7f610a000640 1 -- 172.21.15.28:0/1942267693 <== mon.2 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f60f8010ac0 con 0x7f610c108ea0 2024-05-12T22:08:13.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:13.598+0000 7f610a000640 1 --2- 172.21.15.28:0/1942267693 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f60f40a2b90 0x7f60f40a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:13.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:13.598+0000 7f610a000640 1 -- 172.21.15.28:0/1942267693 <== mon.2 v2:172.21.15.86:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f60f80cf3e0 con 0x7f610c108ea0 2024-05-12T22:08:13.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:13.598+0000 7f6111a00640 1 --2- 172.21.15.28:0/1942267693 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f60f40a2b90 0x7f60f40a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:13.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:13.599+0000 7f6111a00640 1 --2- 172.21.15.28:0/1942267693 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f60f40a2b90 0x7f60f40a5050 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f6100000c00 tx=0x7f61000046c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:08:13.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:13.602+0000 7f610a000640 1 -- 172.21.15.28:0/1942267693 <== mon.2 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f60f809bca0 con 0x7f610c108ea0 2024-05-12T22:08:13.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:13.843+0000 7f6112e00640 1 -- 172.21.15.28:0/1942267693 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f610c0624d0 con 0x7f610c108ea0 2024-05-12T22:08:13.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:13.845+0000 7f610a000640 1 -- 172.21.15.28:0/1942267693 <== mon.2 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11864 (secure 0 0 0) 0x7f60f809ffa0 con 0x7f610c108ea0 2024-05-12T22:08:13.846 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-12T22:08:13.846 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1844141745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1844141745},{"type":"v1","addr":"172.21.15.186:6833","nonce":1844141745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6833/2960763758","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":2960763758},{"type":"v1","addr":"172.21.15.86:6833","nonce":2960763758}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24470,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/4131134219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":4131134219},{"type":"v1","addr":"172.21.15.28:6835","nonce":4131134219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24481,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/1727869064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1727869064},{"type":"v1","addr":"172.21.15.186:6835","nonce":1727869064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24482,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2938260704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2938260704},{"type":"v1","addr":"172.21.15.28:6837","nonce":2938260704}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24494,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3473700910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3473700910},{"type":"v1","addr":"172.21.15.86:6835","nonce":3473700910}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-12T22:06:33.426986+0000","modified":"2024-05-12T22:06:38.789246+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24581,"mds_1":24550,"mds_2":24532},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24499":{"gid":24499,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6837/3600809235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":3600809235},{"type":"v1","addr":"172.21.15.186:6837","nonce":3600809235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"i","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6839/2399610807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":2399610807},{"type":"v1","addr":"172.21.15.186:6839","nonce":2399610807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24530":{"gid":24530,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.86:6837/3782448884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":3782448884},{"type":"v1","addr":"172.21.15.86:6837","nonce":3782448884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24532":{"gid":24532,"name":"g","rank":2,"incarnation":21,"state":"up:active","state_seq":6,"addr":"172.21.15.28:6839/123892158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":123892158},{"type":"v1","addr":"172.21.15.28:6839","nonce":123892158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24550":{"gid":24550,"name":"a","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.28:6841/761183870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":761183870},{"type":"v1","addr":"172.21.15.28:6841","nonce":761183870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24581":{"gid":24581,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6839/1251999130","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6838","nonce":1251999130},{"type":"v1","addr":"172.21.15.86:6839","nonce":1251999130}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24581,"qdb_cluster":[24550,24532,24581]},"id":1}]} 2024-05-12T22:08:13.846 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 23 2024-05-12T22:08:13.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:13.847+0000 7f6112e00640 1 -- 172.21.15.28:0/1942267693 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f60f40a2b90 msgr2=0x7f60f40a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:13.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:13.847+0000 7f6112e00640 1 --2- 172.21.15.28:0/1942267693 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f60f40a2b90 0x7f60f40a5050 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f6100000c00 tx=0x7f61000046c0 comp rx=0 tx=0).stop 2024-05-12T22:08:13.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:13.847+0000 7f6112e00640 1 -- 172.21.15.28:0/1942267693 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f610c108ea0 msgr2=0x7f610c19e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:13.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:13.847+0000 7f6112e00640 1 --2- 172.21.15.28:0/1942267693 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f610c108ea0 0x7f610c19e4f0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f60f800e9f0 tx=0x7f60f800eeb0 comp rx=0 tx=0).stop 2024-05-12T22:08:13.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:13.848+0000 7f6112e00640 1 -- 172.21.15.28:0/1942267693 shutdown_connections 2024-05-12T22:08:13.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:13.848+0000 7f6112e00640 1 --2- 172.21.15.28:0/1942267693 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f60f40a2b90 0x7f60f40a5050 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:13.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:13.848+0000 7f6112e00640 1 --2- 172.21.15.28:0/1942267693 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f610c109860 0x7f610c1a5560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:13.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:13.848+0000 7f6112e00640 1 --2- 172.21.15.28:0/1942267693 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f610c108ea0 0x7f610c19e4f0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:13.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:13.848+0000 7f6112e00640 1 --2- 172.21.15.28:0/1942267693 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f610c107cb0 0x7f610c19dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:13.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:13.848+0000 7f6112e00640 1 -- 172.21.15.28:0/1942267693 >> 172.21.15.28:0/1942267693 conn(0x7f610c076c70 msgr2=0x7f610c10c220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:08:13.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:13.848+0000 7f6112e00640 1 -- 172.21.15.28:0/1942267693 shutdown_connections 2024-05-12T22:08:13.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:13.848+0000 7f6112e00640 1 -- 172.21.15.28:0/1942267693 wait complete. 2024-05-12T22:08:13.867 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-12T22:08:14.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:14.077+0000 7f7571800640 1 -- 172.21.15.28:0/937401874 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f756c108ea0 msgr2=0x7f756c109320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:14.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:14.077+0000 7f7571800640 1 --2- 172.21.15.28:0/937401874 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f756c108ea0 0x7f756c109320 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f7558009900 tx=0x7f755802a400 comp rx=0 tx=0).stop 2024-05-12T22:08:14.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:14.077+0000 7f7571800640 1 -- 172.21.15.28:0/937401874 shutdown_connections 2024-05-12T22:08:14.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:14.077+0000 7f7571800640 1 --2- 172.21.15.28:0/937401874 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f756c109860 0x7f756c1100f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:14.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:14.077+0000 7f7571800640 1 --2- 172.21.15.28:0/937401874 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f756c108ea0 0x7f756c109320 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:14.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:14.077+0000 7f7571800640 1 --2- 172.21.15.28:0/937401874 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f756c107cb0 0x7f756c1080d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:14.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:14.077+0000 7f7571800640 1 -- 172.21.15.28:0/937401874 >> 172.21.15.28:0/937401874 conn(0x7f756c076c70 msgr2=0x7f756c079090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:08:14.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:14.077+0000 7f7571800640 1 -- 172.21.15.28:0/937401874 shutdown_connections 2024-05-12T22:08:14.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:14.077+0000 7f7571800640 1 -- 172.21.15.28:0/937401874 wait complete. 2024-05-12T22:08:14.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:14.078+0000 7f7571800640 1 Processor -- start 2024-05-12T22:08:14.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:14.078+0000 7f7571800640 1 -- start start 2024-05-12T22:08:14.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:14.078+0000 7f7571800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f756c107cb0 0x7f756c19dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:14.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:14.078+0000 7f7571800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f756c108ea0 0x7f756c19e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:14.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:14.079+0000 7f7571800640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f756c109860 0x7f756c1a5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:14.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:14.079+0000 7f7571800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f756c113930 con 0x7f756c108ea0 2024-05-12T22:08:14.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:14.079+0000 7f7571800640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f756c1137b0 con 0x7f756c107cb0 2024-05-12T22:08:14.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:14.079+0000 7f7571800640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f756c113ab0 con 0x7f756c109860 2024-05-12T22:08:14.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:14.079+0000 7f756b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f756c108ea0 0x7f756c19e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:14.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:14.079+0000 7f756be00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f756c107cb0 0x7f756c19dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:14.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:14.079+0000 7f756b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f756c108ea0 0x7f756c19e4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38346/0 (socket says 172.21.15.28:38346) 2024-05-12T22:08:14.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:14.079+0000 7f756b400640 1 -- 172.21.15.28:0/1590771468 learned_addr learned my addr 172.21.15.28:0/1590771468 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:08:14.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:14.079+0000 7f7570e00640 1 --2- 172.21.15.28:0/1590771468 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f756c109860 0x7f756c1a5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:14.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:14.079+0000 7f756be00640 1 -- 172.21.15.28:0/1590771468 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f756c109860 msgr2=0x7f756c1a5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:14.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:14.079+0000 7f756be00640 1 --2- 172.21.15.28:0/1590771468 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f756c109860 0x7f756c1a5560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:14.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:14.079+0000 7f756be00640 1 -- 172.21.15.28:0/1590771468 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f756c108ea0 msgr2=0x7f756c19e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:14.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:14.079+0000 7f756be00640 1 --2- 172.21.15.28:0/1590771468 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f756c108ea0 0x7f756c19e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:14.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:14.079+0000 7f756be00640 1 -- 172.21.15.28:0/1590771468 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f756c1a5bd0 con 0x7f756c107cb0 2024-05-12T22:08:14.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:14.080+0000 7f756be00640 1 --2- 172.21.15.28:0/1590771468 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f756c107cb0 0x7f756c19dfb0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f755c00e9c0 tx=0x7f755c00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:08:14.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:14.080+0000 7f7568c00640 1 -- 172.21.15.28:0/1590771468 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f755c00ccc0 con 0x7f756c107cb0 2024-05-12T22:08:14.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:14.080+0000 7f7571800640 1 -- 172.21.15.28:0/1590771468 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f756c1a5ec0 con 0x7f756c107cb0 2024-05-12T22:08:14.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:14.080+0000 7f7568c00640 1 -- 172.21.15.28:0/1590771468 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f755c0035f0 con 0x7f756c107cb0 2024-05-12T22:08:14.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:14.080+0000 7f7568c00640 1 -- 172.21.15.28:0/1590771468 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f755c00ce60 con 0x7f756c107cb0 2024-05-12T22:08:14.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:14.080+0000 7f7571800640 1 -- 172.21.15.28:0/1590771468 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f756c1a6330 con 0x7f756c107cb0 2024-05-12T22:08:14.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:14.081+0000 7f7571800640 1 -- 172.21.15.28:0/1590771468 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f756c10f050 con 0x7f756c107cb0 2024-05-12T22:08:14.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:14.083+0000 7f7568c00640 1 -- 172.21.15.28:0/1590771468 <== mon.2 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f755c0037a0 con 0x7f756c107cb0 2024-05-12T22:08:14.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:14.084+0000 7f7568c00640 1 --2- 172.21.15.28:0/1590771468 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f75500a2c10 0x7f75500a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:14.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:14.084+0000 7f7568c00640 1 -- 172.21.15.28:0/1590771468 <== mon.2 v2:172.21.15.86:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f755c0cf450 con 0x7f756c107cb0 2024-05-12T22:08:14.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:14.085+0000 7f756b400640 1 --2- 172.21.15.28:0/1590771468 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f75500a2c10 0x7f75500a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:14.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:14.085+0000 7f756b400640 1 --2- 172.21.15.28:0/1590771468 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f75500a2c10 0x7f75500a50d0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f7558031440 tx=0x7f75580315b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:08:14.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:14.089+0000 7f7568c00640 1 -- 172.21.15.28:0/1590771468 <== mon.2 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f755c09c360 con 0x7f756c107cb0 2024-05-12T22:08:14.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:13 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/3936178429' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-12T22:08:14.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:13 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/1942267693' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:08:14.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:13 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/3936178429' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-12T22:08:14.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:13 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/1942267693' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:08:14.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:14.295+0000 7f7571800640 1 -- 172.21.15.28:0/1590771468 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f756c0624d0 con 0x7f756c107cb0 2024-05-12T22:08:14.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:14.297+0000 7f7568c00640 1 -- 172.21.15.28:0/1590771468 <== mon.2 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22595 (secure 0 0 0) 0x7f755c09bc60 con 0x7f756c107cb0 2024-05-12T22:08:14.298 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-12T22:08:14.298 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":88,"fsid":"6b92e7b4-10aa-11ef-bc98-c7b262605968","created":"2024-05-12T21:57:37.276782+0000","modified":"2024-05-12T22:06:46.768798+0000","last_up_change":"2024-05-12T22:05:37.929562+0000","last_in_change":"2024-05-12T22:05:18.750768+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-12T22:01:32.292826+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-12T22:06:31.540263+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-12T22:06:32.906118+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6992ce0a-844b-40e3-8ed1-181c8474dccf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":906003882},{"type":"v1","addr":"172.21.15.28:6803","nonce":906003882}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":906003882},{"type":"v1","addr":"172.21.15.28:6805","nonce":906003882}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":906003882},{"type":"v1","addr":"172.21.15.28:6809","nonce":906003882}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":906003882},{"type":"v1","addr":"172.21.15.28:6807","nonce":906003882}]},"public_addr":"172.21.15.28:6803/906003882","cluster_addr":"172.21.15.28:6805/906003882","heartbeat_back_addr":"172.21.15.28:6809/906003882","heartbeat_front_addr":"172.21.15.28:6807/906003882","state":["exists","up"]},{"osd":1,"uuid":"ced20c96-c481-4d48-90d1-6b43228959ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":3354141819},{"type":"v1","addr":"172.21.15.86:6801","nonce":3354141819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":3354141819},{"type":"v1","addr":"172.21.15.86:6803","nonce":3354141819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":3354141819},{"type":"v1","addr":"172.21.15.86:6807","nonce":3354141819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":3354141819},{"type":"v1","addr":"172.21.15.86:6805","nonce":3354141819}]},"public_addr":"172.21.15.86:6801/3354141819","cluster_addr":"172.21.15.86:6803/3354141819","heartbeat_back_addr":"172.21.15.86:6807/3354141819","heartbeat_front_addr":"172.21.15.86:6805/3354141819","state":["exists","up"]},{"osd":2,"uuid":"54a2f221-fa65-4795-9f5e-e660cc7c0098","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":906459365},{"type":"v1","addr":"172.21.15.186:6801","nonce":906459365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":906459365},{"type":"v1","addr":"172.21.15.186:6803","nonce":906459365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":906459365},{"type":"v1","addr":"172.21.15.186:6807","nonce":906459365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":906459365},{"type":"v1","addr":"172.21.15.186:6805","nonce":906459365}]},"public_addr":"172.21.15.186:6801/906459365","cluster_addr":"172.21.15.186:6803/906459365","heartbeat_back_addr":"172.21.15.186:6807/906459365","heartbeat_front_addr":"172.21.15.186:6805/906459365","state":["exists","up"]},{"osd":3,"uuid":"86836737-ac9a-487b-93f2-7fcd3c406960","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":4182884436},{"type":"v1","addr":"172.21.15.28:6811","nonce":4182884436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":4182884436},{"type":"v1","addr":"172.21.15.28:6813","nonce":4182884436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":4182884436},{"type":"v1","addr":"172.21.15.28:6817","nonce":4182884436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":4182884436},{"type":"v1","addr":"172.21.15.28:6815","nonce":4182884436}]},"public_addr":"172.21.15.28:6811/4182884436","cluster_addr":"172.21.15.28:6813/4182884436","heartbeat_back_addr":"172.21.15.28:6817/4182884436","heartbeat_front_addr":"172.21.15.28:6815/4182884436","state":["exists","up"]},{"osd":4,"uuid":"d162ebcd-83c0-4fd5-9925-28b2cecea935","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2596998708},{"type":"v1","addr":"172.21.15.86:6809","nonce":2596998708}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2596998708},{"type":"v1","addr":"172.21.15.86:6811","nonce":2596998708}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2596998708},{"type":"v1","addr":"172.21.15.86:6815","nonce":2596998708}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2596998708},{"type":"v1","addr":"172.21.15.86:6813","nonce":2596998708}]},"public_addr":"172.21.15.86:6809/2596998708","cluster_addr":"172.21.15.86:6811/2596998708","heartbeat_back_addr":"172.21.15.86:6815/2596998708","heartbeat_front_addr":"172.21.15.86:6813/2596998708","state":["exists","up"]},{"osd":5,"uuid":"c794336a-a105-425c-b6c7-a32b90c05bb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":1944205756},{"type":"v1","addr":"172.21.15.186:6809","nonce":1944205756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":1944205756},{"type":"v1","addr":"172.21.15.186:6811","nonce":1944205756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":1944205756},{"type":"v1","addr":"172.21.15.186:6815","nonce":1944205756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":1944205756},{"type":"v1","addr":"172.21.15.186:6813","nonce":1944205756}]},"public_addr":"172.21.15.186:6809/1944205756","cluster_addr":"172.21.15.186:6811/1944205756","heartbeat_back_addr":"172.21.15.186:6815/1944205756","heartbeat_front_addr":"172.21.15.186:6813/1944205756","state":["exists","up"]},{"osd":6,"uuid":"516932ac-af00-45b7-ad3c-d6dec7db442f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":1642308925},{"type":"v1","addr":"172.21.15.28:6819","nonce":1642308925}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":1642308925},{"type":"v1","addr":"172.21.15.28:6821","nonce":1642308925}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1642308925},{"type":"v1","addr":"172.21.15.28:6825","nonce":1642308925}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":1642308925},{"type":"v1","addr":"172.21.15.28:6823","nonce":1642308925}]},"public_addr":"172.21.15.28:6819/1642308925","cluster_addr":"172.21.15.28:6821/1642308925","heartbeat_back_addr":"172.21.15.28:6825/1642308925","heartbeat_front_addr":"172.21.15.28:6823/1642308925","state":["exists","up"]},{"osd":7,"uuid":"ca7a059f-0414-438e-b2d1-a10f348036be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":726062711},{"type":"v1","addr":"172.21.15.86:6817","nonce":726062711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":726062711},{"type":"v1","addr":"172.21.15.86:6819","nonce":726062711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":726062711},{"type":"v1","addr":"172.21.15.86:6823","nonce":726062711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":726062711},{"type":"v1","addr":"172.21.15.86:6821","nonce":726062711}]},"public_addr":"172.21.15.86:6817/726062711","cluster_addr":"172.21.15.86:6819/726062711","heartbeat_back_addr":"172.21.15.86:6823/726062711","heartbeat_front_addr":"172.21.15.86:6821/726062711","state":["exists","up"]},{"osd":8,"uuid":"92e3f41f-aebd-422e-a99e-555f7ba4aa53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":3007687277},{"type":"v1","addr":"172.21.15.186:6817","nonce":3007687277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":3007687277},{"type":"v1","addr":"172.21.15.186:6819","nonce":3007687277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":3007687277},{"type":"v1","addr":"172.21.15.186:6823","nonce":3007687277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":3007687277},{"type":"v1","addr":"172.21.15.186:6821","nonce":3007687277}]},"public_addr":"172.21.15.186:6817/3007687277","cluster_addr":"172.21.15.186:6819/3007687277","heartbeat_back_addr":"172.21.15.186:6823/3007687277","heartbeat_front_addr":"172.21.15.186:6821/3007687277","state":["exists","up"]},{"osd":9,"uuid":"38ff7d9d-ee85-404c-8c59-adc327e1ccfa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":136220472},{"type":"v1","addr":"172.21.15.28:6827","nonce":136220472}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":136220472},{"type":"v1","addr":"172.21.15.28:6829","nonce":136220472}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":136220472},{"type":"v1","addr":"172.21.15.28:6833","nonce":136220472}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":136220472},{"type":"v1","addr":"172.21.15.28:6831","nonce":136220472}]},"public_addr":"172.21.15.28:6827/136220472","cluster_addr":"172.21.15.28:6829/136220472","heartbeat_back_addr":"172.21.15.28:6833/136220472","heartbeat_front_addr":"172.21.15.28:6831/136220472","state":["exists","up"]},{"osd":10,"uuid":"5e9ff657-afa8-4bab-8491-f209038392b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1189968310},{"type":"v1","addr":"172.21.15.86:6825","nonce":1189968310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":1189968310},{"type":"v1","addr":"172.21.15.86:6827","nonce":1189968310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":1189968310},{"type":"v1","addr":"172.21.15.86:6831","nonce":1189968310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":1189968310},{"type":"v1","addr":"172.21.15.86:6829","nonce":1189968310}]},"public_addr":"172.21.15.86:6825/1189968310","cluster_addr":"172.21.15.86:6827/1189968310","heartbeat_back_addr":"172.21.15.86:6831/1189968310","heartbeat_front_addr":"172.21.15.86:6829/1189968310","state":["exists","up"]},{"osd":11,"uuid":"f112e3c6-65b6-4253-a966-62001db7d069","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3801668110},{"type":"v1","addr":"172.21.15.186:6825","nonce":3801668110}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3801668110},{"type":"v1","addr":"172.21.15.186:6827","nonce":3801668110}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3801668110},{"type":"v1","addr":"172.21.15.186:6831","nonce":3801668110}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3801668110},{"type":"v1","addr":"172.21.15.186:6829","nonce":3801668110}]},"public_addr":"172.21.15.186:6825/3801668110","cluster_addr":"172.21.15.186:6827/3801668110","heartbeat_back_addr":"172.21.15.186:6831/3801668110","heartbeat_front_addr":"172.21.15.186:6829/3801668110","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:00:40.483623+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:01:05.284385+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:01:28.645716+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:02:01.295092+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:02:26.486601+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:02:50.807478+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:03:24.756169+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:03:49.975173+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:04:13.772233+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:04:46.758555+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:05:12.331685+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:05:36.462548+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":2,"to":1}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.25","mappings":[{"from":5,"to":9}]},{"pgid":"3.27","mappings":[{"from":4,"to":8}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/1177009533":"2024-05-13T21:58:45.550459+0000","172.21.15.28:6800/3760499540":"2024-05-13T21:58:45.550459+0000","172.21.15.28:6801/3783742728":"2024-05-13T21:58:10.421564+0000","172.21.15.28:6800/3783742728":"2024-05-13T21:58:10.421564+0000","172.21.15.28:0/2229156260":"2024-05-13T21:58:45.550459+0000","172.21.15.28:0/1202140963":"2024-05-13T21:58:10.421564+0000","172.21.15.28:0/2924223784":"2024-05-13T21:58:45.550459+0000","172.21.15.28:6801/3760499540":"2024-05-13T21:58:45.550459+0000","172.21.15.28:0/1240027206":"2024-05-13T21:58:10.421564+0000","172.21.15.28:0/171417728":"2024-05-13T21:58:10.421564+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-12T22:08:14.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:14.299+0000 7f7571800640 1 -- 172.21.15.28:0/1590771468 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f75500a2c10 msgr2=0x7f75500a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:14.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:14.299+0000 7f7571800640 1 --2- 172.21.15.28:0/1590771468 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f75500a2c10 0x7f75500a50d0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f7558031440 tx=0x7f75580315b0 comp rx=0 tx=0).stop 2024-05-12T22:08:14.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:14.299+0000 7f7571800640 1 -- 172.21.15.28:0/1590771468 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f756c107cb0 msgr2=0x7f756c19dfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:14.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:14.299+0000 7f7571800640 1 --2- 172.21.15.28:0/1590771468 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f756c107cb0 0x7f756c19dfb0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f755c00e9c0 tx=0x7f755c00ee80 comp rx=0 tx=0).stop 2024-05-12T22:08:14.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:14.300+0000 7f7571800640 1 -- 172.21.15.28:0/1590771468 shutdown_connections 2024-05-12T22:08:14.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:14.300+0000 7f7571800640 1 --2- 172.21.15.28:0/1590771468 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f75500a2c10 0x7f75500a50d0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:14.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:14.300+0000 7f7571800640 1 --2- 172.21.15.28:0/1590771468 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f756c109860 0x7f756c1a5560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:14.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:14.300+0000 7f7571800640 1 --2- 172.21.15.28:0/1590771468 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f756c108ea0 0x7f756c19e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:14.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:14.300+0000 7f7571800640 1 --2- 172.21.15.28:0/1590771468 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f756c107cb0 0x7f756c19dfb0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:14.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:14.300+0000 7f7571800640 1 -- 172.21.15.28:0/1590771468 >> 172.21.15.28:0/1590771468 conn(0x7f756c076c70 msgr2=0x7f756c10c220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:08:14.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:14.300+0000 7f7571800640 1 -- 172.21.15.28:0/1590771468 shutdown_connections 2024-05-12T22:08:14.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:14.300+0000 7f7571800640 1 -- 172.21.15.28:0/1590771468 wait complete. 2024-05-12T22:08:14.320 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-12T22:08:14.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:13 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/3936178429' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-12T22:08:14.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:13 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/1942267693' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:08:14.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:14.523+0000 7fc667e00640 1 -- 172.21.15.28:0/2068503726 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc668104aa0 msgr2=0x7fc668104f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:14.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:14.523+0000 7fc667e00640 1 --2- 172.21.15.28:0/2068503726 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc668104aa0 0x7fc668104f20 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fc650009900 tx=0x7fc65002a3f0 comp rx=0 tx=0).stop 2024-05-12T22:08:14.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:14.523+0000 7fc667e00640 1 -- 172.21.15.28:0/2068503726 shutdown_connections 2024-05-12T22:08:14.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:14.523+0000 7fc667e00640 1 --2- 172.21.15.28:0/2068503726 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc668105460 0x7fc66810bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:14.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:14.523+0000 7fc667e00640 1 --2- 172.21.15.28:0/2068503726 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc668104aa0 0x7fc668104f20 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:14.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:14.523+0000 7fc667e00640 1 --2- 172.21.15.28:0/2068503726 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6681038b0 0x7fc668103cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:14.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:14.523+0000 7fc667e00640 1 -- 172.21.15.28:0/2068503726 >> 172.21.15.28:0/2068503726 conn(0x7fc6680fee40 msgr2=0x7fc668101260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:08:14.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:14.523+0000 7fc667e00640 1 -- 172.21.15.28:0/2068503726 shutdown_connections 2024-05-12T22:08:14.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:14.523+0000 7fc667e00640 1 -- 172.21.15.28:0/2068503726 wait complete. 2024-05-12T22:08:14.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:14.525+0000 7fc667e00640 1 Processor -- start 2024-05-12T22:08:14.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:14.525+0000 7fc667e00640 1 -- start start 2024-05-12T22:08:14.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:14.525+0000 7fc667e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc6681038b0 0x7fc668199bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:14.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:14.525+0000 7fc667e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc668104aa0 0x7fc66819a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:14.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:14.525+0000 7fc667e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc668105460 0x7fc6681a11a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:14.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:14.525+0000 7fc667e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc66810f580 con 0x7fc668104aa0 2024-05-12T22:08:14.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:14.525+0000 7fc667e00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fc66810f400 con 0x7fc668105460 2024-05-12T22:08:14.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:14.525+0000 7fc667e00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc66810f700 con 0x7fc6681038b0 2024-05-12T22:08:14.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:14.526+0000 7fc65e000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc668104aa0 0x7fc66819a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:14.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:14.526+0000 7fc666a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc6681038b0 0x7fc668199bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:14.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:14.526+0000 7fc65e000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc668104aa0 0x7fc66819a130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38368/0 (socket says 172.21.15.28:38368) 2024-05-12T22:08:14.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:14.526+0000 7fc667400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc668105460 0x7fc6681a11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:14.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:14.526+0000 7fc65e000640 1 -- 172.21.15.28:0/1696090119 learned_addr learned my addr 172.21.15.28:0/1696090119 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:08:14.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:14.526+0000 7fc666a00640 1 -- 172.21.15.28:0/1696090119 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc668105460 msgr2=0x7fc6681a11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:14.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:14.526+0000 7fc666a00640 1 --2- 172.21.15.28:0/1696090119 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc668105460 0x7fc6681a11a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:14.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:14.526+0000 7fc666a00640 1 -- 172.21.15.28:0/1696090119 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc668104aa0 msgr2=0x7fc66819a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:14.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:14.526+0000 7fc666a00640 1 --2- 172.21.15.28:0/1696090119 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc668104aa0 0x7fc66819a130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:14.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:14.526+0000 7fc666a00640 1 -- 172.21.15.28:0/1696090119 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6681a1810 con 0x7fc6681038b0 2024-05-12T22:08:14.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:14.527+0000 7fc666a00640 1 --2- 172.21.15.28:0/1696090119 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc6681038b0 0x7fc668199bf0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fc65800e9c0 tx=0x7fc65800ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:08:14.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:14.527+0000 7fc66ca00640 1 -- 172.21.15.28:0/1696090119 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc65800ccc0 con 0x7fc6681038b0 2024-05-12T22:08:14.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:14.527+0000 7fc66ca00640 1 -- 172.21.15.28:0/1696090119 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc6580035f0 con 0x7fc6681038b0 2024-05-12T22:08:14.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:14.527+0000 7fc66ca00640 1 -- 172.21.15.28:0/1696090119 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc65800ce60 con 0x7fc6681038b0 2024-05-12T22:08:14.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:14.527+0000 7fc667e00640 1 -- 172.21.15.28:0/1696090119 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6681a1b00 con 0x7fc6681038b0 2024-05-12T22:08:14.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:14.527+0000 7fc667e00640 1 -- 172.21.15.28:0/1696090119 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6681a2000 con 0x7fc6681038b0 2024-05-12T22:08:14.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:14.528+0000 7fc667e00640 1 -- 172.21.15.28:0/1696090119 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc66810abd0 con 0x7fc6681038b0 2024-05-12T22:08:14.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:14.530+0000 7fc66ca00640 1 -- 172.21.15.28:0/1696090119 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fc6580037a0 con 0x7fc6681038b0 2024-05-12T22:08:14.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:14.532+0000 7fc66ca00640 1 --2- 172.21.15.28:0/1696090119 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fc64c0a2c90 0x7fc64c0a5150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:14.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:14.533+0000 7fc65e000640 1 --2- 172.21.15.28:0/1696090119 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fc64c0a2c90 0x7fc64c0a5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:14.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:14.533+0000 7fc66ca00640 1 -- 172.21.15.28:0/1696090119 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7fc6580cf350 con 0x7fc6681038b0 2024-05-12T22:08:14.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:14.534+0000 7fc65e000640 1 --2- 172.21.15.28:0/1696090119 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fc64c0a2c90 0x7fc64c0a5150 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fc66819b100 tx=0x7fc6500315b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:08:14.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:14.536+0000 7fc66ca00640 1 -- 172.21.15.28:0/1696090119 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fc65809c260 con 0x7fc6681038b0 2024-05-12T22:08:14.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:14.782+0000 7fc667e00640 1 -- 172.21.15.28:0/1696090119 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc66802d1c0 con 0x7fc6681038b0 2024-05-12T22:08:14.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:14.783+0000 7fc66ca00640 1 -- 172.21.15.28:0/1696090119 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11864 (secure 0 0 0) 0x7fc65809bb60 con 0x7fc6681038b0 2024-05-12T22:08:14.784 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-12T22:08:14.784 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1844141745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1844141745},{"type":"v1","addr":"172.21.15.186:6833","nonce":1844141745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6833/2960763758","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":2960763758},{"type":"v1","addr":"172.21.15.86:6833","nonce":2960763758}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24470,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/4131134219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":4131134219},{"type":"v1","addr":"172.21.15.28:6835","nonce":4131134219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24481,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/1727869064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1727869064},{"type":"v1","addr":"172.21.15.186:6835","nonce":1727869064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24482,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2938260704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2938260704},{"type":"v1","addr":"172.21.15.28:6837","nonce":2938260704}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24494,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3473700910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3473700910},{"type":"v1","addr":"172.21.15.86:6835","nonce":3473700910}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-12T22:06:33.426986+0000","modified":"2024-05-12T22:06:38.789246+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24581,"mds_1":24550,"mds_2":24532},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24499":{"gid":24499,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6837/3600809235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":3600809235},{"type":"v1","addr":"172.21.15.186:6837","nonce":3600809235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"i","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6839/2399610807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":2399610807},{"type":"v1","addr":"172.21.15.186:6839","nonce":2399610807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24530":{"gid":24530,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.86:6837/3782448884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":3782448884},{"type":"v1","addr":"172.21.15.86:6837","nonce":3782448884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24532":{"gid":24532,"name":"g","rank":2,"incarnation":21,"state":"up:active","state_seq":6,"addr":"172.21.15.28:6839/123892158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":123892158},{"type":"v1","addr":"172.21.15.28:6839","nonce":123892158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24550":{"gid":24550,"name":"a","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.28:6841/761183870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":761183870},{"type":"v1","addr":"172.21.15.28:6841","nonce":761183870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24581":{"gid":24581,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6839/1251999130","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6838","nonce":1251999130},{"type":"v1","addr":"172.21.15.86:6839","nonce":1251999130}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24581,"qdb_cluster":[24550,24532,24581]},"id":1}]} 2024-05-12T22:08:14.784 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 23 2024-05-12T22:08:14.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:14.786+0000 7fc667e00640 1 -- 172.21.15.28:0/1696090119 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fc64c0a2c90 msgr2=0x7fc64c0a5150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:14.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:14.786+0000 7fc667e00640 1 --2- 172.21.15.28:0/1696090119 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fc64c0a2c90 0x7fc64c0a5150 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fc66819b100 tx=0x7fc6500315b0 comp rx=0 tx=0).stop 2024-05-12T22:08:14.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:14.786+0000 7fc667e00640 1 -- 172.21.15.28:0/1696090119 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc6681038b0 msgr2=0x7fc668199bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:14.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:14.786+0000 7fc667e00640 1 --2- 172.21.15.28:0/1696090119 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc6681038b0 0x7fc668199bf0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fc65800e9c0 tx=0x7fc65800ee80 comp rx=0 tx=0).stop 2024-05-12T22:08:14.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:14.786+0000 7fc667e00640 1 -- 172.21.15.28:0/1696090119 shutdown_connections 2024-05-12T22:08:14.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:14.786+0000 7fc667e00640 1 --2- 172.21.15.28:0/1696090119 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fc64c0a2c90 0x7fc64c0a5150 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:14.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:14.786+0000 7fc667e00640 1 --2- 172.21.15.28:0/1696090119 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc668105460 0x7fc6681a11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:14.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:14.786+0000 7fc667e00640 1 --2- 172.21.15.28:0/1696090119 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc668104aa0 0x7fc66819a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:14.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:14.786+0000 7fc667e00640 1 --2- 172.21.15.28:0/1696090119 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc6681038b0 0x7fc668199bf0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:14.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:14.786+0000 7fc667e00640 1 -- 172.21.15.28:0/1696090119 >> 172.21.15.28:0/1696090119 conn(0x7fc6680fee40 msgr2=0x7fc668107da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:08:14.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:14.786+0000 7fc667e00640 1 -- 172.21.15.28:0/1696090119 shutdown_connections 2024-05-12T22:08:14.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:14.786+0000 7fc667e00640 1 -- 172.21.15.28:0/1696090119 wait complete. 2024-05-12T22:08:14.808 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-05-12T22:06:33.426986+0000', 'modified': '2024-05-12T22:06:38.789246+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24581, 'mds_1': 24550, 'mds_2': 24532}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24499': {'gid': 24499, 'name': 'c', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.186:6837/3600809235', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.186:6836', 'nonce': 3600809235}, {'type': 'v1', 'addr': '172.21.15.186:6837', 'nonce': 3600809235}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24520': {'gid': 24520, 'name': 'i', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.186:6839/2399610807', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.186:6838', 'nonce': 2399610807}, {'type': 'v1', 'addr': '172.21.15.186:6839', 'nonce': 2399610807}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24530': {'gid': 24530, 'name': 'h', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.86:6837/3782448884', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6836', 'nonce': 3782448884}, {'type': 'v1', 'addr': '172.21.15.86:6837', 'nonce': 3782448884}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24532': {'gid': 24532, 'name': 'g', 'rank': 2, 'incarnation': 21, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.28:6839/123892158', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6838', 'nonce': 123892158}, {'type': 'v1', 'addr': '172.21.15.28:6839', 'nonce': 123892158}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24550': {'gid': 24550, 'name': 'a', 'rank': 1, 'incarnation': 19, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.28:6841/761183870', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6840', 'nonce': 761183870}, {'type': 'v1', 'addr': '172.21.15.28:6841', 'nonce': 761183870}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24581': {'gid': 24581, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.86:6839/1251999130', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6838', 'nonce': 1251999130}, {'type': 'v1', 'addr': '172.21.15.86:6839', 'nonce': 1251999130}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24581, 'qdb_cluster': [24550, 24532, 24581]} 2024-05-12T22:08:14.808 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-05-12T22:08:14.808 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.g status 2024-05-12T22:08:15.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.014+0000 7f095e200640 1 -- 172.21.15.28:0/681626669 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0958108f60 msgr2=0x7f09581093e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:15.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.014+0000 7f095e200640 1 --2- 172.21.15.28:0/681626669 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0958108f60 0x7f09581093e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f0944009900 tx=0x7f094402a410 comp rx=0 tx=0).stop 2024-05-12T22:08:15.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.015+0000 7f095e200640 1 -- 172.21.15.28:0/681626669 shutdown_connections 2024-05-12T22:08:15.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.015+0000 7f095e200640 1 --2- 172.21.15.28:0/681626669 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0958109920 0x7f09581101b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:15.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.015+0000 7f095e200640 1 --2- 172.21.15.28:0/681626669 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0958108f60 0x7f09581093e0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:15.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.015+0000 7f095e200640 1 --2- 172.21.15.28:0/681626669 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0958107d70 0x7f0958108190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:15.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.015+0000 7f095e200640 1 -- 172.21.15.28:0/681626669 >> 172.21.15.28:0/681626669 conn(0x7f0958076d00 msgr2=0x7f0958079140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:08:15.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.015+0000 7f095e200640 1 -- 172.21.15.28:0/681626669 shutdown_connections 2024-05-12T22:08:15.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.015+0000 7f095e200640 1 -- 172.21.15.28:0/681626669 wait complete. 2024-05-12T22:08:15.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.016+0000 7f095e200640 1 Processor -- start 2024-05-12T22:08:15.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.016+0000 7f095e200640 1 -- start start 2024-05-12T22:08:15.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.016+0000 7f095e200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0958107d70 0x7f095819e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:15.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.016+0000 7f095e200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0958108f60 0x7f095819e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:15.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.016+0000 7f095e200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0958109920 0x7f09581a55e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:15.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.016+0000 7f095e200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0958113a40 con 0x7f0958109920 2024-05-12T22:08:15.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.016+0000 7f095e200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f09581138c0 con 0x7f0958107d70 2024-05-12T22:08:15.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.016+0000 7f095e200640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f0958113bc0 con 0x7f0958108f60 2024-05-12T22:08:15.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.016+0000 7f095d800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0958109920 0x7f09581a55e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:15.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.016+0000 7f095d800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0958109920 0x7f09581a55e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38384/0 (socket says 172.21.15.28:38384) 2024-05-12T22:08:15.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.016+0000 7f095d800640 1 -- 172.21.15.28:0/2867452509 learned_addr learned my addr 172.21.15.28:0/2867452509 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:08:15.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.016+0000 7f095ce00640 1 --2- 172.21.15.28:0/2867452509 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0958107d70 0x7f095819e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:15.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.017+0000 7f0957e00640 1 --2- 172.21.15.28:0/2867452509 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0958108f60 0x7f095819e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:15.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.017+0000 7f095d800640 1 -- 172.21.15.28:0/2867452509 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0958108f60 msgr2=0x7f095819e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:15.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.017+0000 7f095d800640 1 --2- 172.21.15.28:0/2867452509 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0958108f60 0x7f095819e570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:15.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.017+0000 7f095d800640 1 -- 172.21.15.28:0/2867452509 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0958107d70 msgr2=0x7f095819e030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:15.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.017+0000 7f095d800640 1 --2- 172.21.15.28:0/2867452509 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0958107d70 0x7f095819e030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:15.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.017+0000 7f095d800640 1 -- 172.21.15.28:0/2867452509 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09581a5c50 con 0x7f0958109920 2024-05-12T22:08:15.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.017+0000 7f095d800640 1 --2- 172.21.15.28:0/2867452509 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0958109920 0x7f09581a55e0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f095000eb10 tx=0x7f095000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:08:15.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.017+0000 7f0955600640 1 -- 172.21.15.28:0/2867452509 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0950004b20 con 0x7f0958109920 2024-05-12T22:08:15.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.018+0000 7f0955600640 1 -- 172.21.15.28:0/2867452509 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0950004cc0 con 0x7f0958109920 2024-05-12T22:08:15.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.018+0000 7f095e200640 1 -- 172.21.15.28:0/2867452509 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09581a5f40 con 0x7f0958109920 2024-05-12T22:08:15.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.018+0000 7f0955600640 1 -- 172.21.15.28:0/2867452509 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0950010760 con 0x7f0958109920 2024-05-12T22:08:15.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.018+0000 7f095e200640 1 -- 172.21.15.28:0/2867452509 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09581a6400 con 0x7f0958109920 2024-05-12T22:08:15.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.021+0000 7f096261e740 1 Processor -- start 2024-05-12T22:08:15.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.021+0000 7f0955600640 1 -- 172.21.15.28:0/2867452509 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f0950005020 con 0x7f0958109920 2024-05-12T22:08:15.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.021+0000 7f096261e740 1 -- start start 2024-05-12T22:08:15.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.022+0000 7f096261e740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5620c0edd190 0x5620c0edd5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:15.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.022+0000 7f096261e740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5620c0eddb80 0x5620c0ee2680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:15.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.022+0000 7f096261e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5620c0ee2cf0 0x5620c0ee5180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:15.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.022+0000 7f096261e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5620c0e40480 con 0x5620c0ee2cf0 2024-05-12T22:08:15.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.022+0000 7f096261e740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x5620c0de85c0 con 0x5620c0eddb80 2024-05-12T22:08:15.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.022+0000 7f096261e740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5620c0e3c440 con 0x5620c0edd190 2024-05-12T22:08:15.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.022+0000 7f0957e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5620c0edd190 0x5620c0edd5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:15.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.022+0000 7f095ce00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5620c0eddb80 0x5620c0ee2680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:15.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.022+0000 7f0957e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5620c0edd190 0x5620c0edd5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:34486/0 (socket says 172.21.15.28:34486) 2024-05-12T22:08:15.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.022+0000 7f0957e00640 1 -- 172.21.15.28:0/1376921007 learned_addr learned my addr 172.21.15.28:0/1376921007 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:08:15.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.022+0000 7f0957e00640 1 --2- 172.21.15.28:0/1376921007 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5620c0ee2cf0 0x5620c0ee5180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:15.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.022+0000 7f095ce00640 1 -- 172.21.15.28:0/1376921007 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5620c0edd190 msgr2=0x5620c0edd5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:08:15.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.022+0000 7f095ce00640 1 --2- 172.21.15.28:0/1376921007 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5620c0edd190 0x5620c0edd5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:15.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.022+0000 7f095ce00640 1 -- 172.21.15.28:0/1376921007 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5620c0ee2cf0 msgr2=0x5620c0ee5180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:08:15.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.022+0000 7f095ce00640 1 --2- 172.21.15.28:0/1376921007 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5620c0ee2cf0 0x5620c0ee5180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:15.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.022+0000 7f095ce00640 1 -- 172.21.15.28:0/1376921007 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5620c0b542d0 con 0x5620c0eddb80 2024-05-12T22:08:15.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.022+0000 7f0957e00640 1 --2- 172.21.15.28:0/1376921007 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5620c0edd190 0x5620c0edd5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:08:15.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.023+0000 7f0955600640 1 --2- 172.21.15.28:0/2867452509 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f09400a2b90 0x7f09400a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:15.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.023+0000 7f095ce00640 1 --2- 172.21.15.28:0/1376921007 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5620c0eddb80 0x5620c0ee2680 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f094c009bf0 tx=0x7f094c011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cb71f1cc1daeaa24 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:08:15.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.023+0000 7f0955600640 1 -- 172.21.15.28:0/2867452509 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f09500cf730 con 0x7f0958109920 2024-05-12T22:08:15.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.023+0000 7f0957e00640 1 --2- 172.21.15.28:0/2867452509 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f09400a2b90 0x7f09400a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:15.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.023+0000 7f094a000640 1 -- 172.21.15.28:0/1376921007 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f094c009da0 con 0x5620c0eddb80 2024-05-12T22:08:15.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.023+0000 7f094a000640 1 -- 172.21.15.28:0/1376921007 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f094c005020 con 0x5620c0eddb80 2024-05-12T22:08:15.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.023+0000 7f094a000640 1 -- 172.21.15.28:0/1376921007 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f094c004bc0 con 0x5620c0eddb80 2024-05-12T22:08:15.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.024+0000 7f096261e740 1 -- 172.21.15.28:0/1376921007 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5620c0eddb80 msgr2=0x5620c0ee2680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:15.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.024+0000 7f0957e00640 1 --2- 172.21.15.28:0/2867452509 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f09400a2b90 0x7f09400a5050 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f0944000c00 tx=0x7f09440315b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:08:15.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.024+0000 7f096261e740 1 --2- 172.21.15.28:0/1376921007 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5620c0eddb80 0x5620c0ee2680 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f094c009bf0 tx=0x7f094c011810 comp rx=0 tx=0).stop 2024-05-12T22:08:15.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.024+0000 7f096261e740 1 -- 172.21.15.28:0/1376921007 shutdown_connections 2024-05-12T22:08:15.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.024+0000 7f096261e740 1 --2- 172.21.15.28:0/1376921007 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5620c0ee2cf0 0x5620c0ee5180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:15.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.024+0000 7f096261e740 1 --2- 172.21.15.28:0/1376921007 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5620c0eddb80 0x5620c0ee2680 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:15.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.024+0000 7f096261e740 1 --2- 172.21.15.28:0/1376921007 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5620c0edd190 0x5620c0edd5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:15.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.024+0000 7f096261e740 1 -- 172.21.15.28:0/1376921007 >> 172.21.15.28:0/1376921007 conn(0x5620c0ea48a0 msgr2=0x5620c0ed8c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:08:15.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.024+0000 7f096261e740 1 -- 172.21.15.28:0/1376921007 shutdown_connections 2024-05-12T22:08:15.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.024+0000 7f096261e740 1 -- 172.21.15.28:0/1376921007 wait complete. 2024-05-12T22:08:15.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.024+0000 7f096261e740 1 -- start start 2024-05-12T22:08:15.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.025+0000 7f096261e740 1 Processor -- start 2024-05-12T22:08:15.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.025+0000 7f096261e740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5620c0edd190 0x5620c0ee72b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:15.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.025+0000 7f096261e740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5620c0eddb80 0x5620c0ee77f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:15.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.025+0000 7f096261e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5620c0ee2cf0 0x5620c0eee830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:15.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.025+0000 7f096261e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5620c0ee5ca0 con 0x5620c0ee2cf0 2024-05-12T22:08:15.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.025+0000 7f096261e740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x5620c0dfb980 con 0x5620c0eddb80 2024-05-12T22:08:15.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.025+0000 7f096261e740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5620c0ee5e20 con 0x5620c0edd190 2024-05-12T22:08:15.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.025+0000 7f096261e740 10 client.0 _ll_register_callbacks cb 0x5620c0e36fc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:08:15.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.026+0000 7f0957e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5620c0eddb80 0x5620c0ee77f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:15.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.025+0000 7f095d800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5620c0ee2cf0 0x5620c0eee830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:15.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.026+0000 7f0957e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5620c0eddb80 0x5620c0ee77f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:51234/0 (socket says 172.21.15.28:51234) 2024-05-12T22:08:15.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.026+0000 7f0957e00640 1 -- 172.21.15.28:0/3907746062 learned_addr learned my addr 172.21.15.28:0/3907746062 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:08:15.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.026+0000 7f095ce00640 1 --2- 172.21.15.28:0/3907746062 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5620c0edd190 0x5620c0ee72b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:15.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.026+0000 7f0957e00640 1 -- 172.21.15.28:0/3907746062 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5620c0edd190 msgr2=0x5620c0ee72b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:08:15.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.026+0000 7f0957e00640 1 --2- 172.21.15.28:0/3907746062 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5620c0edd190 0x5620c0ee72b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:15.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.026+0000 7f0957e00640 1 -- 172.21.15.28:0/3907746062 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5620c0ee2cf0 msgr2=0x5620c0eee830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:08:15.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.026+0000 7f0957e00640 1 --2- 172.21.15.28:0/3907746062 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5620c0ee2cf0 0x5620c0eee830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:15.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.026+0000 7f0957e00640 1 -- 172.21.15.28:0/3907746062 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5620c0c1e6b0 con 0x5620c0eddb80 2024-05-12T22:08:15.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.026+0000 7f095d800640 1 --2- 172.21.15.28:0/3907746062 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5620c0ee2cf0 0x5620c0eee830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:08:15.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.026+0000 7f0957e00640 1 --2- 172.21.15.28:0/3907746062 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5620c0eddb80 0x5620c0ee77f0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f0944031440 tx=0x7f0944005780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9efb336ffff19cfe server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:08:15.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.026+0000 7f093ea00640 10 client.0 ms_handle_connect on v2:172.21.15.86:3300/0 2024-05-12T22:08:15.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.027+0000 7f093ea00640 1 -- 172.21.15.28:0/3907746062 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09440035f0 con 0x5620c0eddb80 2024-05-12T22:08:15.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.027+0000 7f096261e740 1 -- 172.21.15.28:0/3907746062 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f094c005020 con 0x5620c0eddb80 2024-05-12T22:08:15.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.027+0000 7f093ea00640 1 -- 172.21.15.28:0/3907746062 <== mon.2 v2:172.21.15.86:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f0944003790 con 0x5620c0eddb80 2024-05-12T22:08:15.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.027+0000 7f093ea00640 1 -- 172.21.15.28:0/3907746062 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f0944003990 con 0x5620c0eddb80 2024-05-12T22:08:15.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.027+0000 7f096261e740 10 client.24986 fetch_fsmap learned FSMap version 23 2024-05-12T22:08:15.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.027+0000 7f096261e740 1 -- 172.21.15.28:0/3907746062 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5620c0ef18d0 con 0x5620c0eddb80 2024-05-12T22:08:15.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.027+0000 7f093ea00640 1 -- 172.21.15.28:0/3907746062 <== mon.2 v2:172.21.15.86:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f094c005020 con 0x5620c0eddb80 2024-05-12T22:08:15.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.028+0000 7f093ea00640 1 -- 172.21.15.28:0/3907746062 <== mon.2 v2:172.21.15.86:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f0944044030 con 0x5620c0eddb80 2024-05-12T22:08:15.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.028+0000 7f096261e740 10 client.24986 fetch_fsmap finished waiting for FSMap version 23 2024-05-12T22:08:15.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.028+0000 7f096261e740 10 client.24986 resolve_mds: resolved name 'g' to daemon mds.g 2024-05-12T22:08:15.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.028+0000 7f096261e740 20 client.24986 populate_metadata read hostname 'smithi028' 2024-05-12T22:08:15.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.028+0000 7f096261e740 1 --2- 172.21.15.28:0/3907746062 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x5620c0ef3e80 0x5620c0ef52f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:15.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.028+0000 7f096261e740 4 client.24986 mds_command: new command op to 24532 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-12T22:08:15.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.028+0000 7f096261e740 1 -- 172.21.15.28:0/3907746062 --> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5620c0e54060 con 0x5620c0ef3e80 2024-05-12T22:08:15.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.028+0000 7f095ce00640 1 --2- 172.21.15.28:0/3907746062 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x5620c0ef3e80 0x5620c0ef52f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:15.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.029+0000 7f095ce00640 1 --2- 172.21.15.28:0/3907746062 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x5620c0ef3e80 0x5620c0ef52f0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=2c9fde92b51bfeeb server_cookie=43f91e6220903aa6 in_seq=0 out_seq=0 2024-05-12T22:08:15.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.029+0000 7f093ea00640 10 client.24986 ms_handle_connect on v2:172.21.15.28:6838/123892158 2024-05-12T22:08:15.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.030+0000 7f093ea00640 1 -- 172.21.15.28:0/3907746062 <== mds.2 v2:172.21.15.28:6838/123892158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5620c0e54060 con 0x5620c0ef3e80 2024-05-12T22:08:15.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.030+0000 7f093ea00640 10 client.24986 handle_command_reply: tid=0 2024-05-12T22:08:15.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.030+0000 7f096261e740 1 client.24986 shutdown 2024-05-12T22:08:15.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.030+0000 7f096261e740 1 -- 172.21.15.28:0/3907746062 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5620c0eddb80 msgr2=0x5620c0ee77f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:15.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.030+0000 7f096261e740 1 --2- 172.21.15.28:0/3907746062 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5620c0eddb80 0x5620c0ee77f0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f0944031440 tx=0x7f0944005780 comp rx=0 tx=0).stop 2024-05-12T22:08:15.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.030+0000 7f096261e740 1 -- 172.21.15.28:0/3907746062 shutdown_connections 2024-05-12T22:08:15.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.030+0000 7f096261e740 1 --2- 172.21.15.28:0/3907746062 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5620c0ee2cf0 0x5620c0eee830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:15.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.030+0000 7f096261e740 1 --2- 172.21.15.28:0/3907746062 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x5620c0ef3e80 0x5620c0ef52f0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:15.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.030+0000 7f096261e740 1 --2- 172.21.15.28:0/3907746062 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5620c0eddb80 0x5620c0ee77f0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:15.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.030+0000 7f096261e740 1 --2- 172.21.15.28:0/3907746062 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5620c0edd190 0x5620c0ee72b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:15.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.030+0000 7f093ea00640 0 client.24986 ms_handle_reset on v2:172.21.15.28:6838/123892158 2024-05-12T22:08:15.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.030+0000 7f096261e740 1 -- 172.21.15.28:0/3907746062 >> 172.21.15.28:0/3907746062 conn(0x5620c0ea48a0 msgr2=0x5620c0ed7c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:08:15.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.030+0000 7f096261e740 1 -- 172.21.15.28:0/3907746062 shutdown_connections 2024-05-12T22:08:15.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.030+0000 7f096261e740 1 -- 172.21.15.28:0/3907746062 wait complete. 2024-05-12T22:08:15.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.031+0000 7f096261e740 20 client.24986 trim_cache size 0 max 16384 2024-05-12T22:08:15.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.051+0000 7f096261e740 1 Processor -- start 2024-05-12T22:08:15.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.051+0000 7f096261e740 1 -- start start 2024-05-12T22:08:15.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.051+0000 7f096261e740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5620c0edd190 0x5620c0edc210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:15.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.051+0000 7f096261e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5620c0eddb80 0x5620c0eee7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:15.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.051+0000 7f096261e740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5620c0ee2cf0 0x5620c0ed7390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:15.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.051+0000 7f096261e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5620c0ef3630 con 0x5620c0eddb80 2024-05-12T22:08:15.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.051+0000 7f096261e740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x5620c0ef3d00 con 0x5620c0edd190 2024-05-12T22:08:15.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.051+0000 7f096261e740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5620c0ef43d0 con 0x5620c0ee2cf0 2024-05-12T22:08:15.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.052+0000 7f0957e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5620c0eddb80 0x5620c0eee7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:15.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.052+0000 7f095ce00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5620c0edd190 0x5620c0edc210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:15.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.052+0000 7f0957e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5620c0eddb80 0x5620c0eee7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38418/0 (socket says 172.21.15.28:38418) 2024-05-12T22:08:15.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.052+0000 7f0957e00640 1 -- 172.21.15.28:0/1980460422 learned_addr learned my addr 172.21.15.28:0/1980460422 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:08:15.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.052+0000 7f095d800640 1 --2- 172.21.15.28:0/1980460422 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5620c0ee2cf0 0x5620c0ed7390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:15.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.052+0000 7f0957e00640 1 -- 172.21.15.28:0/1980460422 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5620c0ee2cf0 msgr2=0x5620c0ed7390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:08:15.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.052+0000 7f0957e00640 1 --2- 172.21.15.28:0/1980460422 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5620c0ee2cf0 0x5620c0ed7390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:15.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.052+0000 7f0957e00640 1 -- 172.21.15.28:0/1980460422 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5620c0edd190 msgr2=0x5620c0edc210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:08:15.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.052+0000 7f0957e00640 1 --2- 172.21.15.28:0/1980460422 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5620c0edd190 0x5620c0edc210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:15.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.052+0000 7f0957e00640 1 -- 172.21.15.28:0/1980460422 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5620c0ede370 con 0x5620c0eddb80 2024-05-12T22:08:15.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.052+0000 7f095ce00640 1 --2- 172.21.15.28:0/1980460422 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5620c0edd190 0x5620c0edc210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:08:15.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.052+0000 7f0957e00640 1 --2- 172.21.15.28:0/1980460422 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5620c0eddb80 0x5620c0eee7f0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f0944051f40 tx=0x7f0944031af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d79503f9921f403e server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:08:15.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.053+0000 7f0949600640 1 -- 172.21.15.28:0/1980460422 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09440035f0 con 0x5620c0eddb80 2024-05-12T22:08:15.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.053+0000 7f0949600640 1 -- 172.21.15.28:0/1980460422 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0944057020 con 0x5620c0eddb80 2024-05-12T22:08:15.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.053+0000 7f0949600640 1 -- 172.21.15.28:0/1980460422 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0944003cb0 con 0x5620c0eddb80 2024-05-12T22:08:15.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.053+0000 7f096261e740 1 -- 172.21.15.28:0/1980460422 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5620c0eddb80 msgr2=0x5620c0eee7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:15.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.053+0000 7f096261e740 1 --2- 172.21.15.28:0/1980460422 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5620c0eddb80 0x5620c0eee7f0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f0944051f40 tx=0x7f0944031af0 comp rx=0 tx=0).stop 2024-05-12T22:08:15.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.053+0000 7f096261e740 1 -- 172.21.15.28:0/1980460422 shutdown_connections 2024-05-12T22:08:15.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.053+0000 7f096261e740 1 --2- 172.21.15.28:0/1980460422 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5620c0ee2cf0 0x5620c0ed7390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:15.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.053+0000 7f096261e740 1 --2- 172.21.15.28:0/1980460422 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5620c0eddb80 0x5620c0eee7f0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:15.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.053+0000 7f096261e740 1 --2- 172.21.15.28:0/1980460422 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5620c0edd190 0x5620c0edc210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:15.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.053+0000 7f096261e740 1 -- 172.21.15.28:0/1980460422 >> 172.21.15.28:0/1980460422 conn(0x5620c0ea48a0 msgr2=0x5620c0edbb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:08:15.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.054+0000 7f096261e740 1 -- 172.21.15.28:0/1980460422 shutdown_connections 2024-05-12T22:08:15.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.054+0000 7f096261e740 1 -- 172.21.15.28:0/1980460422 wait complete. 2024-05-12T22:08:15.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.054+0000 7f096261e740 1 -- start start 2024-05-12T22:08:15.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.054+0000 7f096261e740 1 Processor -- start 2024-05-12T22:08:15.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.055+0000 7f096261e740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5620c0edd190 0x5620c0ee0d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:15.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.055+0000 7f096261e740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5620c0eddb80 0x5620c0ee1260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:15.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.055+0000 7f096261e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5620c0ee2cf0 0x5620c0f57e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:15.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.055+0000 7f096261e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5620c0ef5170 con 0x5620c0ee2cf0 2024-05-12T22:08:15.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.055+0000 7f096261e740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x5620c0ef4aa0 con 0x5620c0edd190 2024-05-12T22:08:15.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.055+0000 7f096261e740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5620c0ef5840 con 0x5620c0eddb80 2024-05-12T22:08:15.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.055+0000 7f096261e740 10 client.0 _ll_register_callbacks cb 0x7f09500093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:08:15.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.055+0000 7f095d800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5620c0ee2cf0 0x5620c0f57e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:15.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.055+0000 7f095ce00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5620c0edd190 0x5620c0ee0d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:15.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.055+0000 7f095d800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5620c0ee2cf0 0x5620c0f57e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38422/0 (socket says 172.21.15.28:38422) 2024-05-12T22:08:15.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.055+0000 7f095d800640 1 -- 172.21.15.28:0/1511330816 learned_addr learned my addr 172.21.15.28:0/1511330816 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:08:15.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.055+0000 7f0957e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5620c0eddb80 0x5620c0ee1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:15.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.055+0000 7f095d800640 1 -- 172.21.15.28:0/1511330816 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5620c0eddb80 msgr2=0x5620c0ee1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:08:15.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.055+0000 7f095d800640 1 --2- 172.21.15.28:0/1511330816 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5620c0eddb80 0x5620c0ee1260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:15.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.055+0000 7f095d800640 1 -- 172.21.15.28:0/1511330816 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5620c0edd190 msgr2=0x5620c0ee0d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:08:15.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.055+0000 7f095d800640 1 --2- 172.21.15.28:0/1511330816 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5620c0edd190 0x5620c0ee0d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:15.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.055+0000 7f095d800640 1 -- 172.21.15.28:0/1511330816 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5620c0f58390 con 0x5620c0ee2cf0 2024-05-12T22:08:15.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.055+0000 7f095ce00640 1 --2- 172.21.15.28:0/1511330816 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5620c0edd190 0x5620c0ee0d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:08:15.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.055+0000 7f0957e00640 1 --2- 172.21.15.28:0/1511330816 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5620c0eddb80 0x5620c0ee1260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:08:15.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.056+0000 7f095d800640 1 --2- 172.21.15.28:0/1511330816 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5620c0ee2cf0 0x5620c0f57e20 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f09500983a0 tx=0x7f095009c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=921e4279b8c606b1 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:08:15.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.056+0000 7f093ea00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-12T22:08:15.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.056+0000 7f093ea00640 1 -- 172.21.15.28:0/1511330816 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0950005510 con 0x5620c0ee2cf0 2024-05-12T22:08:15.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.056+0000 7f093ea00640 1 -- 172.21.15.28:0/1511330816 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f095009b390 con 0x5620c0ee2cf0 2024-05-12T22:08:15.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.056+0000 7f096261e740 1 -- 172.21.15.28:0/1511330816 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0944057020 con 0x5620c0ee2cf0 2024-05-12T22:08:15.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.056+0000 7f093ea00640 1 -- 172.21.15.28:0/1511330816 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f095009bf40 con 0x5620c0ee2cf0 2024-05-12T22:08:15.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.056+0000 7f096261e740 10 client.14877 fetch_fsmap learned FSMap version 23 2024-05-12T22:08:15.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.056+0000 7f096261e740 1 -- 172.21.15.28:0/1511330816 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5620c0f59760 con 0x5620c0ee2cf0 2024-05-12T22:08:15.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.056+0000 7f093ea00640 1 -- 172.21.15.28:0/1511330816 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0944057020 con 0x5620c0ee2cf0 2024-05-12T22:08:15.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.057+0000 7f093ea00640 1 -- 172.21.15.28:0/1511330816 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f0950096b40 con 0x5620c0ee2cf0 2024-05-12T22:08:15.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.057+0000 7f096261e740 10 client.14877 fetch_fsmap finished waiting for FSMap version 23 2024-05-12T22:08:15.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.057+0000 7f096261e740 10 client.14877 resolve_mds: resolved name 'g' to daemon mds.g 2024-05-12T22:08:15.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.057+0000 7f096261e740 20 client.14877 populate_metadata read hostname 'smithi028' 2024-05-12T22:08:15.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.057+0000 7f096261e740 1 --2- 172.21.15.28:0/1511330816 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x5620c0f5e370 0x5620c0f5d080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:15.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.057+0000 7f096261e740 4 client.14877 mds_command: new command op to 24532 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-12T22:08:15.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.057+0000 7f096261e740 1 -- 172.21.15.28:0/1511330816 --> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] -- command(tid 0: {"prefix": "status"}) -- 0x5620c0e4dbb0 con 0x5620c0f5e370 2024-05-12T22:08:15.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.057+0000 7f095ce00640 1 --2- 172.21.15.28:0/1511330816 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x5620c0f5e370 0x5620c0f5d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:15.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.058+0000 7f095ce00640 1 --2- 172.21.15.28:0/1511330816 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x5620c0f5e370 0x5620c0f5d080 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f9070699146c04cd server_cookie=ab6e537d82a017d0 in_seq=0 out_seq=0 2024-05-12T22:08:15.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.058+0000 7f093ea00640 10 client.14877 ms_handle_connect on v2:172.21.15.28:6838/123892158 2024-05-12T22:08:15.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.058+0000 7f093ea00640 1 -- 172.21.15.28:0/1511330816 <== mds.2 v2:172.21.15.28:6838/123892158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x5620c0e4dbb0 con 0x5620c0f5e370 2024-05-12T22:08:15.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.058+0000 7f093ea00640 10 client.14877 handle_command_reply: tid=0 2024-05-12T22:08:15.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.058+0000 7f096261e740 1 client.14877 shutdown 2024-05-12T22:08:15.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.058+0000 7f096261e740 1 -- 172.21.15.28:0/1511330816 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5620c0ee2cf0 msgr2=0x5620c0f57e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:15.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.058+0000 7f096261e740 1 --2- 172.21.15.28:0/1511330816 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5620c0ee2cf0 0x5620c0f57e20 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f09500983a0 tx=0x7f095009c540 comp rx=0 tx=0).stop 2024-05-12T22:08:15.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.058+0000 7f096261e740 1 -- 172.21.15.28:0/1511330816 shutdown_connections 2024-05-12T22:08:15.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.058+0000 7f096261e740 1 --2- 172.21.15.28:0/1511330816 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5620c0ee2cf0 0x5620c0f57e20 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:15.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.058+0000 7f096261e740 1 --2- 172.21.15.28:0/1511330816 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5620c0eddb80 0x5620c0ee1260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:15.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.058+0000 7f096261e740 1 --2- 172.21.15.28:0/1511330816 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x5620c0f5e370 0x5620c0f5d080 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:15.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.058+0000 7f096261e740 1 --2- 172.21.15.28:0/1511330816 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5620c0edd190 0x5620c0ee0d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:15.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.058+0000 7f096261e740 1 -- 172.21.15.28:0/1511330816 >> 172.21.15.28:0/1511330816 conn(0x5620c0ea48a0 msgr2=0x5620c0ede620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:08:15.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.058+0000 7f093ea00640 0 client.14877 ms_handle_reset on v2:172.21.15.28:6838/123892158 2024-05-12T22:08:15.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.058+0000 7f096261e740 1 -- 172.21.15.28:0/1511330816 shutdown_connections 2024-05-12T22:08:15.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.058+0000 7f096261e740 1 -- 172.21.15.28:0/1511330816 wait complete. 2024-05-12T22:08:15.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.058+0000 7f096261e740 20 client.14877 trim_cache size 0 max 16384 2024-05-12T22:08:15.066 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-05-12T22:08:15.066 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "6b92e7b4-10aa-11ef-bc98-c7b262605968", 2024-05-12T22:08:15.066 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 2, 2024-05-12T22:08:15.066 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 24532, 2024-05-12T22:08:15.066 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-05-12T22:08:15.066 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-05-12T22:08:15.066 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-05-12T22:08:15.066 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 96.745356115999996, 2024-05-12T22:08:15.066 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 23, 2024-05-12T22:08:15.066 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 88, 2024-05-12T22:08:15.066 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 86, 2024-05-12T22:08:15.066 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 114.311362815 2024-05-12T22:08:15.067 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-05-12T22:08:15.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.059+0000 7f095e200640 1 -- 172.21.15.28:0/2867452509 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f09400a2b90 msgr2=0x7f09400a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:15.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.059+0000 7f095e200640 1 --2- 172.21.15.28:0/2867452509 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f09400a2b90 0x7f09400a5050 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f0944000c00 tx=0x7f09440315b0 comp rx=0 tx=0).stop 2024-05-12T22:08:15.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.059+0000 7f095e200640 1 -- 172.21.15.28:0/2867452509 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0958109920 msgr2=0x7f09581a55e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:15.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.059+0000 7f095e200640 1 --2- 172.21.15.28:0/2867452509 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0958109920 0x7f09581a55e0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f095000eb10 tx=0x7f095000c2d0 comp rx=0 tx=0).stop 2024-05-12T22:08:15.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.059+0000 7f095e200640 1 -- 172.21.15.28:0/2867452509 shutdown_connections 2024-05-12T22:08:15.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.059+0000 7f095e200640 1 --2- 172.21.15.28:0/2867452509 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f09400a2b90 0x7f09400a5050 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:15.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.059+0000 7f095e200640 1 --2- 172.21.15.28:0/2867452509 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0958109920 0x7f09581a55e0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:15.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.059+0000 7f095e200640 1 --2- 172.21.15.28:0/2867452509 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0958108f60 0x7f095819e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:15.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.059+0000 7f095e200640 1 --2- 172.21.15.28:0/2867452509 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0958107d70 0x7f095819e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:15.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.059+0000 7f095e200640 1 -- 172.21.15.28:0/2867452509 >> 172.21.15.28:0/2867452509 conn(0x7f0958076d00 msgr2=0x7f095810c200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:08:15.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.059+0000 7f095e200640 1 -- 172.21.15.28:0/2867452509 shutdown_connections 2024-05-12T22:08:15.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.059+0000 7f095e200640 1 -- 172.21.15.28:0/2867452509 wait complete. 2024-05-12T22:08:15.075 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:14 smithi028 ceph-mon[29847]: pgmap v362: 97 pgs: 97 active+clean; 641 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-12T22:08:15.076 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:14 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/1590771468' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-12T22:08:15.076 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:14 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/1696090119' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:08:15.079 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-05-12T22:08:15.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:14 smithi086 ceph-mon[34297]: pgmap v362: 97 pgs: 97 active+clean; 641 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-12T22:08:15.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:14 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/1590771468' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-12T22:08:15.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:14 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/1696090119' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:08:15.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.288+0000 7f7a53200640 1 -- 172.21.15.28:0/559288319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a54107d70 msgr2=0x7f7a54108190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:15.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.288+0000 7f7a53200640 1 --2- 172.21.15.28:0/559288319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a54107d70 0x7f7a54108190 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f7a44009900 tx=0x7f7a4402a3f0 comp rx=0 tx=0).stop 2024-05-12T22:08:15.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.288+0000 7f7a53200640 1 -- 172.21.15.28:0/559288319 shutdown_connections 2024-05-12T22:08:15.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.288+0000 7f7a53200640 1 --2- 172.21.15.28:0/559288319 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7a54109920 0x7f7a541101b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:15.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.288+0000 7f7a53200640 1 --2- 172.21.15.28:0/559288319 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7a54108f60 0x7f7a541093e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:15.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.288+0000 7f7a53200640 1 --2- 172.21.15.28:0/559288319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a54107d70 0x7f7a54108190 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:15.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.288+0000 7f7a53200640 1 -- 172.21.15.28:0/559288319 >> 172.21.15.28:0/559288319 conn(0x7f7a54076d00 msgr2=0x7f7a54079140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:08:15.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.288+0000 7f7a53200640 1 -- 172.21.15.28:0/559288319 shutdown_connections 2024-05-12T22:08:15.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.288+0000 7f7a53200640 1 -- 172.21.15.28:0/559288319 wait complete. 2024-05-12T22:08:15.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.290+0000 7f7a53200640 1 Processor -- start 2024-05-12T22:08:15.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.290+0000 7f7a53200640 1 -- start start 2024-05-12T22:08:15.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.290+0000 7f7a53200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7a54107d70 0x7f7a5419e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:15.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.290+0000 7f7a53200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a54108f60 0x7f7a5419e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:15.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.290+0000 7f7a53200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7a54109920 0x7f7a541a5600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:15.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.290+0000 7f7a53200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7a54113a40 con 0x7f7a54108f60 2024-05-12T22:08:15.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.290+0000 7f7a53200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f7a541138c0 con 0x7f7a54109920 2024-05-12T22:08:15.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.290+0000 7f7a53200640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f7a54113bc0 con 0x7f7a54107d70 2024-05-12T22:08:15.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.290+0000 7f7a51e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7a54107d70 0x7f7a5419e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:15.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.291+0000 7f7a51e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7a54107d70 0x7f7a5419e050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:34528/0 (socket says 172.21.15.28:34528) 2024-05-12T22:08:15.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.291+0000 7f7a51e00640 1 -- 172.21.15.28:0/3650479349 learned_addr learned my addr 172.21.15.28:0/3650479349 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:08:15.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.291+0000 7f7a51400640 1 --2- 172.21.15.28:0/3650479349 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a54108f60 0x7f7a5419e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:15.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.291+0000 7f7a52800640 1 --2- 172.21.15.28:0/3650479349 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7a54109920 0x7f7a541a5600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:15.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.291+0000 7f7a51e00640 1 -- 172.21.15.28:0/3650479349 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7a54109920 msgr2=0x7f7a541a5600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:15.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.291+0000 7f7a51e00640 1 --2- 172.21.15.28:0/3650479349 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7a54109920 0x7f7a541a5600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:15.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.291+0000 7f7a51e00640 1 -- 172.21.15.28:0/3650479349 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a54108f60 msgr2=0x7f7a5419e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:15.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.291+0000 7f7a51e00640 1 --2- 172.21.15.28:0/3650479349 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a54108f60 0x7f7a5419e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:15.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.291+0000 7f7a51e00640 1 -- 172.21.15.28:0/3650479349 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a541a5c70 con 0x7f7a54107d70 2024-05-12T22:08:15.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.291+0000 7f7a51e00640 1 --2- 172.21.15.28:0/3650479349 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7a54107d70 0x7f7a5419e050 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f7a44031440 tx=0x7f7a44032e20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:08:15.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.292+0000 7f7a4aa00640 1 -- 172.21.15.28:0/3650479349 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a4403f030 con 0x7f7a54107d70 2024-05-12T22:08:15.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.292+0000 7f7a4aa00640 1 -- 172.21.15.28:0/3650479349 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7a44003ce0 con 0x7f7a54107d70 2024-05-12T22:08:15.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.292+0000 7f7a53200640 1 -- 172.21.15.28:0/3650479349 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a541a5f00 con 0x7f7a54107d70 2024-05-12T22:08:15.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.292+0000 7f7a4aa00640 1 -- 172.21.15.28:0/3650479349 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a44035070 con 0x7f7a54107d70 2024-05-12T22:08:15.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.292+0000 7f7a53200640 1 -- 172.21.15.28:0/3650479349 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a541a6360 con 0x7f7a54107d70 2024-05-12T22:08:15.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.295+0000 7f7a5b9fd740 1 Processor -- start 2024-05-12T22:08:15.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.295+0000 7f7a5b9fd740 1 -- start start 2024-05-12T22:08:15.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.295+0000 7f7a4aa00640 1 -- 172.21.15.28:0/3650479349 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f7a44042030 con 0x7f7a54107d70 2024-05-12T22:08:15.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.295+0000 7f7a5b9fd740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56327c424190 0x56327c4245b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:15.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.295+0000 7f7a5b9fd740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56327c424b80 0x56327c429680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:15.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.296+0000 7f7a5b9fd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56327c429cf0 0x56327c42c180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:15.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.296+0000 7f7a5b9fd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56327c387480 con 0x56327c429cf0 2024-05-12T22:08:15.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.296+0000 7f7a5b9fd740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x56327c32f5c0 con 0x56327c424190 2024-05-12T22:08:15.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.296+0000 7f7a5b9fd740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x56327c383440 con 0x56327c424b80 2024-05-12T22:08:15.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.296+0000 7f7a52800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56327c424190 0x56327c4245b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:15.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.296+0000 7f7a51e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56327c424b80 0x56327c429680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:15.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.296+0000 7f7a51400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56327c429cf0 0x56327c42c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:15.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.296+0000 7f7a52800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56327c424190 0x56327c4245b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:51276/0 (socket says 172.21.15.28:51276) 2024-05-12T22:08:15.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.296+0000 7f7a52800640 1 -- 172.21.15.28:0/1442670866 learned_addr learned my addr 172.21.15.28:0/1442670866 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:08:15.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.296+0000 7f7a52800640 1 -- 172.21.15.28:0/1442670866 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56327c424b80 msgr2=0x56327c429680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:08:15.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.296+0000 7f7a52800640 1 --2- 172.21.15.28:0/1442670866 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56327c424b80 0x56327c429680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:15.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.296+0000 7f7a52800640 1 -- 172.21.15.28:0/1442670866 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56327c429cf0 msgr2=0x56327c42c180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:08:15.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.296+0000 7f7a52800640 1 --2- 172.21.15.28:0/1442670866 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56327c429cf0 0x56327c42c180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:15.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.296+0000 7f7a52800640 1 -- 172.21.15.28:0/1442670866 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56327c09b2d0 con 0x56327c424190 2024-05-12T22:08:15.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.297+0000 7f7a52800640 1 --2- 172.21.15.28:0/1442670866 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56327c424190 0x56327c4245b0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f7a4c010370 tx=0x7f7a4c010830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4515385cd3faedb2 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:08:15.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.297+0000 7f7a3f000640 1 -- 172.21.15.28:0/1442670866 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a4c016070 con 0x56327c424190 2024-05-12T22:08:15.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.297+0000 7f7a3f000640 1 -- 172.21.15.28:0/1442670866 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7a4c004d20 con 0x56327c424190 2024-05-12T22:08:15.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.297+0000 7f7a3f000640 1 -- 172.21.15.28:0/1442670866 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a4c005480 con 0x56327c424190 2024-05-12T22:08:15.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.297+0000 7f7a4aa00640 1 --2- 172.21.15.28:0/3650479349 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f7a380a2c10 0x7f7a380a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:15.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.297+0000 7f7a4aa00640 1 -- 172.21.15.28:0/3650479349 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f7a440ef5c0 con 0x7f7a54107d70 2024-05-12T22:08:15.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.297+0000 7f7a51e00640 1 --2- 172.21.15.28:0/3650479349 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f7a380a2c10 0x7f7a380a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:15.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.297+0000 7f7a5b9fd740 1 -- 172.21.15.28:0/1442670866 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56327c424190 msgr2=0x56327c4245b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:15.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.297+0000 7f7a5b9fd740 1 --2- 172.21.15.28:0/1442670866 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56327c424190 0x56327c4245b0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f7a4c010370 tx=0x7f7a4c010830 comp rx=0 tx=0).stop 2024-05-12T22:08:15.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.298+0000 7f7a5b9fd740 1 -- 172.21.15.28:0/1442670866 shutdown_connections 2024-05-12T22:08:15.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.298+0000 7f7a5b9fd740 1 --2- 172.21.15.28:0/1442670866 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56327c429cf0 0x56327c42c180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:15.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.298+0000 7f7a5b9fd740 1 --2- 172.21.15.28:0/1442670866 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56327c424b80 0x56327c429680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:15.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.298+0000 7f7a5b9fd740 1 --2- 172.21.15.28:0/1442670866 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56327c424190 0x56327c4245b0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:15.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.298+0000 7f7a5b9fd740 1 -- 172.21.15.28:0/1442670866 >> 172.21.15.28:0/1442670866 conn(0x56327c3eb8a0 msgr2=0x56327c41fc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:08:15.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.298+0000 7f7a5b9fd740 1 -- 172.21.15.28:0/1442670866 shutdown_connections 2024-05-12T22:08:15.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.298+0000 7f7a5b9fd740 1 -- 172.21.15.28:0/1442670866 wait complete. 2024-05-12T22:08:15.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.298+0000 7f7a51e00640 1 --2- 172.21.15.28:0/3650479349 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f7a380a2c10 0x7f7a380a50d0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f7a440bbe10 tx=0x7f7a440f5060 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:08:15.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.298+0000 7f7a5b9fd740 1 -- start start 2024-05-12T22:08:15.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.299+0000 7f7a5b9fd740 1 Processor -- start 2024-05-12T22:08:15.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.299+0000 7f7a5b9fd740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56327c424190 0x56327c42e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:15.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.299+0000 7f7a5b9fd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56327c424b80 0x56327c42e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:15.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.299+0000 7f7a5b9fd740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56327c429cf0 0x56327c435800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:15.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.299+0000 7f7a5b9fd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56327c42cca0 con 0x56327c424b80 2024-05-12T22:08:15.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.299+0000 7f7a5b9fd740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x56327c342980 con 0x56327c429cf0 2024-05-12T22:08:15.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.299+0000 7f7a5b9fd740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x56327c42ce20 con 0x56327c424190 2024-05-12T22:08:15.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.299+0000 7f7a5b9fd740 10 client.0 _ll_register_callbacks cb 0x56327c37dfc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:08:15.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.299+0000 7f7a51400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56327c424b80 0x56327c42e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:15.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.299+0000 7f7a51400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56327c424b80 0x56327c42e7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38462/0 (socket says 172.21.15.28:38462) 2024-05-12T22:08:15.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.299+0000 7f7a51400640 1 -- 172.21.15.28:0/4228963079 learned_addr learned my addr 172.21.15.28:0/4228963079 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:08:15.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.299+0000 7f7a52800640 1 --2- 172.21.15.28:0/4228963079 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56327c424190 0x56327c42e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:15.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.299+0000 7f7a52800640 1 --2- 172.21.15.28:0/4228963079 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56327c429cf0 0x56327c435800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:15.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.300+0000 7f7a51400640 1 -- 172.21.15.28:0/4228963079 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56327c424190 msgr2=0x56327c42e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:08:15.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.300+0000 7f7a51400640 1 --2- 172.21.15.28:0/4228963079 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56327c424190 0x56327c42e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:15.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.300+0000 7f7a51400640 1 -- 172.21.15.28:0/4228963079 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56327c429cf0 msgr2=0x56327c435800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:08:15.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.300+0000 7f7a51400640 1 --2- 172.21.15.28:0/4228963079 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56327c429cf0 0x56327c435800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:15.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.300+0000 7f7a51400640 1 -- 172.21.15.28:0/4228963079 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56327c1656b0 con 0x56327c424b80 2024-05-12T22:08:15.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.300+0000 7f7a52800640 1 --2- 172.21.15.28:0/4228963079 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56327c429cf0 0x56327c435800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:08:15.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.300+0000 7f7a51400640 1 --2- 172.21.15.28:0/4228963079 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56327c424b80 0x56327c42e7c0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f7a4000b8c0 tx=0x7f7a4000bd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84dd7d0e3f06a3e0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:08:15.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.300+0000 7f7a33400640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-12T22:08:15.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.300+0000 7f7a33400640 1 -- 172.21.15.28:0/4228963079 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a40003d80 con 0x56327c424b80 2024-05-12T22:08:15.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.300+0000 7f7a33400640 1 -- 172.21.15.28:0/4228963079 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f7a40011440 con 0x56327c424b80 2024-05-12T22:08:15.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.300+0000 7f7a5b9fd740 1 -- 172.21.15.28:0/4228963079 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7a4c004d20 con 0x56327c424b80 2024-05-12T22:08:15.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.301+0000 7f7a33400640 1 -- 172.21.15.28:0/4228963079 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f7a40005020 con 0x56327c424b80 2024-05-12T22:08:15.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.301+0000 7f7a5b9fd740 10 client.14889 fetch_fsmap learned FSMap version 23 2024-05-12T22:08:15.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.301+0000 7f7a5b9fd740 1 -- 172.21.15.28:0/4228963079 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x56327c438900 con 0x56327c424b80 2024-05-12T22:08:15.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.301+0000 7f7a33400640 1 -- 172.21.15.28:0/4228963079 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7a4c004d20 con 0x56327c424b80 2024-05-12T22:08:15.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.301+0000 7f7a33400640 1 -- 172.21.15.28:0/4228963079 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f7a40024020 con 0x56327c424b80 2024-05-12T22:08:15.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.301+0000 7f7a5b9fd740 10 client.14889 fetch_fsmap finished waiting for FSMap version 23 2024-05-12T22:08:15.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.301+0000 7f7a5b9fd740 10 client.14889 resolve_mds: resolved name 'a' to daemon mds.a 2024-05-12T22:08:15.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.301+0000 7f7a5b9fd740 20 client.14889 populate_metadata read hostname 'smithi028' 2024-05-12T22:08:15.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.301+0000 7f7a5b9fd740 1 --2- 172.21.15.28:0/4228963079 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x56327c43afa0 0x56327c43c410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:15.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.302+0000 7f7a5b9fd740 4 client.14889 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-12T22:08:15.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.302+0000 7f7a5b9fd740 1 -- 172.21.15.28:0/4228963079 --> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56327c39b060 con 0x56327c43afa0 2024-05-12T22:08:15.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.302+0000 7f7a52800640 1 --2- 172.21.15.28:0/4228963079 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x56327c43afa0 0x56327c43c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:15.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.302+0000 7f7a33400640 10 client.14889 ms_handle_connect on v2:172.21.15.28:6840/761183870 2024-05-12T22:08:15.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.302+0000 7f7a52800640 1 --2- 172.21.15.28:0/4228963079 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x56327c43afa0 0x56327c43c410 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=1ebda3b3ed989ad server_cookie=b969395ef0c8fdd6 in_seq=0 out_seq=0 2024-05-12T22:08:15.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.303+0000 7f7a33400640 1 -- 172.21.15.28:0/4228963079 <== mds.1 v2:172.21.15.28:6840/761183870 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56327c39b060 con 0x56327c43afa0 2024-05-12T22:08:15.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.303+0000 7f7a33400640 10 client.14889 handle_command_reply: tid=0 2024-05-12T22:08:15.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.303+0000 7f7a5b9fd740 1 client.14889 shutdown 2024-05-12T22:08:15.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.304+0000 7f7a5b9fd740 1 -- 172.21.15.28:0/4228963079 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56327c424b80 msgr2=0x56327c42e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:15.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.304+0000 7f7a5b9fd740 1 --2- 172.21.15.28:0/4228963079 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56327c424b80 0x56327c42e7c0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f7a4000b8c0 tx=0x7f7a4000bd80 comp rx=0 tx=0).stop 2024-05-12T22:08:15.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.304+0000 7f7a5b9fd740 1 -- 172.21.15.28:0/4228963079 shutdown_connections 2024-05-12T22:08:15.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.304+0000 7f7a5b9fd740 1 --2- 172.21.15.28:0/4228963079 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x56327c43afa0 0x56327c43c410 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:15.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.304+0000 7f7a5b9fd740 1 --2- 172.21.15.28:0/4228963079 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56327c429cf0 0x56327c435800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:15.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.304+0000 7f7a5b9fd740 1 --2- 172.21.15.28:0/4228963079 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56327c424b80 0x56327c42e7c0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:15.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.304+0000 7f7a33400640 0 client.14889 ms_handle_reset on v2:172.21.15.28:6840/761183870 2024-05-12T22:08:15.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.304+0000 7f7a5b9fd740 1 --2- 172.21.15.28:0/4228963079 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56327c424190 0x56327c42e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:15.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.304+0000 7f7a5b9fd740 1 -- 172.21.15.28:0/4228963079 >> 172.21.15.28:0/4228963079 conn(0x56327c3eb8a0 msgr2=0x56327c41ec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:08:15.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.304+0000 7f7a5b9fd740 1 -- 172.21.15.28:0/4228963079 shutdown_connections 2024-05-12T22:08:15.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.304+0000 7f7a5b9fd740 1 -- 172.21.15.28:0/4228963079 wait complete. 2024-05-12T22:08:15.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.304+0000 7f7a5b9fd740 20 client.14889 trim_cache size 0 max 16384 2024-05-12T22:08:15.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.324+0000 7f7a5b9fd740 1 Processor -- start 2024-05-12T22:08:15.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.324+0000 7f7a5b9fd740 1 -- start start 2024-05-12T22:08:15.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.324+0000 7f7a5b9fd740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56327c424190 0x56327c41e390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:15.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.325+0000 7f7a5b9fd740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56327c424b80 0x56327c4357c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:15.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.325+0000 7f7a5b9fd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56327c429cf0 0x56327c41e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:15.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.325+0000 7f7a5b9fd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56327c43a750 con 0x56327c429cf0 2024-05-12T22:08:15.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.325+0000 7f7a5b9fd740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x56327c43ae20 con 0x56327c424190 2024-05-12T22:08:15.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.325+0000 7f7a5b9fd740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x56327c43b4f0 con 0x56327c424b80 2024-05-12T22:08:15.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.325+0000 7f7a52800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56327c429cf0 0x56327c41e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:15.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.325+0000 7f7a52800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56327c424190 0x56327c41e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:15.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.325+0000 7f7a51400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56327c424b80 0x56327c4357c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:15.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.325+0000 7f7a52800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56327c424190 0x56327c41e390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:51296/0 (socket says 172.21.15.28:51296) 2024-05-12T22:08:15.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.325+0000 7f7a52800640 1 -- 172.21.15.28:0/2561121948 learned_addr learned my addr 172.21.15.28:0/2561121948 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:08:15.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.326+0000 7f7a51400640 1 -- 172.21.15.28:0/2561121948 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56327c424190 msgr2=0x56327c41e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:08:15.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.326+0000 7f7a51400640 1 --2- 172.21.15.28:0/2561121948 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56327c424190 0x56327c41e390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:15.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.326+0000 7f7a51400640 1 -- 172.21.15.28:0/2561121948 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56327c429cf0 msgr2=0x56327c41e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:08:15.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.326+0000 7f7a51400640 1 --2- 172.21.15.28:0/2561121948 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56327c429cf0 0x56327c41e8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:15.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.326+0000 7f7a51400640 1 -- 172.21.15.28:0/2561121948 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56327c425410 con 0x56327c424b80 2024-05-12T22:08:15.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.326+0000 7f7a51400640 1 --2- 172.21.15.28:0/2561121948 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56327c424b80 0x56327c4357c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f7a40000c00 tx=0x7f7a400039c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7901c382e190513c server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:08:15.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.326+0000 7f7a3e600640 1 -- 172.21.15.28:0/2561121948 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a400114c0 con 0x56327c424b80 2024-05-12T22:08:15.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.326+0000 7f7a3e600640 1 -- 172.21.15.28:0/2561121948 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7a40021070 con 0x56327c424b80 2024-05-12T22:08:15.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.326+0000 7f7a3e600640 1 -- 172.21.15.28:0/2561121948 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a40004a80 con 0x56327c424b80 2024-05-12T22:08:15.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.327+0000 7f7a5b9fd740 1 -- 172.21.15.28:0/2561121948 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56327c424b80 msgr2=0x56327c4357c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:15.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.327+0000 7f7a5b9fd740 1 --2- 172.21.15.28:0/2561121948 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56327c424b80 0x56327c4357c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f7a40000c00 tx=0x7f7a400039c0 comp rx=0 tx=0).stop 2024-05-12T22:08:15.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.327+0000 7f7a5b9fd740 1 -- 172.21.15.28:0/2561121948 shutdown_connections 2024-05-12T22:08:15.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.327+0000 7f7a5b9fd740 1 --2- 172.21.15.28:0/2561121948 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56327c429cf0 0x56327c41e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:15.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.327+0000 7f7a5b9fd740 1 --2- 172.21.15.28:0/2561121948 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56327c424b80 0x56327c4357c0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:15.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.327+0000 7f7a5b9fd740 1 --2- 172.21.15.28:0/2561121948 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56327c424190 0x56327c41e390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:15.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.327+0000 7f7a5b9fd740 1 -- 172.21.15.28:0/2561121948 >> 172.21.15.28:0/2561121948 conn(0x56327c3eb8a0 msgr2=0x56327c41f6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:08:15.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.327+0000 7f7a5b9fd740 1 -- 172.21.15.28:0/2561121948 shutdown_connections 2024-05-12T22:08:15.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.327+0000 7f7a5b9fd740 1 -- 172.21.15.28:0/2561121948 wait complete. 2024-05-12T22:08:15.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.328+0000 7f7a5b9fd740 1 -- start start 2024-05-12T22:08:15.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.328+0000 7f7a5b9fd740 1 Processor -- start 2024-05-12T22:08:15.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.328+0000 7f7a5b9fd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56327c424190 0x56327c4337b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:15.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.328+0000 7f7a5b9fd740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56327c424b80 0x56327c49cf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:15.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.328+0000 7f7a5b9fd740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56327c429cf0 0x56327c49f480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:15.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.328+0000 7f7a5b9fd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56327c43c290 con 0x56327c424190 2024-05-12T22:08:15.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.328+0000 7f7a5b9fd740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x56327c43bbc0 con 0x56327c429cf0 2024-05-12T22:08:15.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.328+0000 7f7a5b9fd740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x56327c43c960 con 0x56327c424b80 2024-05-12T22:08:15.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.328+0000 7f7a5b9fd740 10 client.0 _ll_register_callbacks cb 0x7f7a40004040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:08:15.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.329+0000 7f7a52800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56327c424190 0x56327c4337b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:15.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.329+0000 7f7a51400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56327c424b80 0x56327c49cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:15.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.329+0000 7f7a52800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56327c424190 0x56327c4337b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38476/0 (socket says 172.21.15.28:38476) 2024-05-12T22:08:15.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.329+0000 7f7a52800640 1 -- 172.21.15.28:0/3028110608 learned_addr learned my addr 172.21.15.28:0/3028110608 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:08:15.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.329+0000 7f7a51400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56327c424b80 0x56327c49cf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:34576/0 (socket says 172.21.15.28:34576) 2024-05-12T22:08:15.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.329+0000 7f7a52800640 1 --2- 172.21.15.28:0/3028110608 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56327c429cf0 0x56327c49f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:15.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.329+0000 7f7a52800640 1 -- 172.21.15.28:0/3028110608 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56327c424b80 msgr2=0x56327c49cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:08:15.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.329+0000 7f7a52800640 1 --2- 172.21.15.28:0/3028110608 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56327c424b80 0x56327c49cf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:15.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.329+0000 7f7a52800640 1 -- 172.21.15.28:0/3028110608 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56327c429cf0 msgr2=0x56327c49f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:08:15.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.329+0000 7f7a52800640 1 --2- 172.21.15.28:0/3028110608 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56327c429cf0 0x56327c49f480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:15.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.329+0000 7f7a52800640 1 -- 172.21.15.28:0/3028110608 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56327c49f9c0 con 0x56327c424190 2024-05-12T22:08:15.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.329+0000 7f7a52800640 1 --2- 172.21.15.28:0/3028110608 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56327c424190 0x56327c4337b0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f7a4c0098b0 tx=0x7f7a4c009600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7746e1a7bff7f528 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:08:15.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.329+0000 7f7a33400640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-12T22:08:15.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.330+0000 7f7a33400640 1 -- 172.21.15.28:0/3028110608 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a4c016040 con 0x56327c424190 2024-05-12T22:08:15.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.330+0000 7f7a33400640 1 -- 172.21.15.28:0/3028110608 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f7a4c02c310 con 0x56327c424190 2024-05-12T22:08:15.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.330+0000 7f7a5b9fd740 1 -- 172.21.15.28:0/3028110608 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7a40021070 con 0x56327c424190 2024-05-12T22:08:15.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.330+0000 7f7a33400640 1 -- 172.21.15.28:0/3028110608 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f7a4c02c510 con 0x56327c424190 2024-05-12T22:08:15.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.330+0000 7f7a5b9fd740 10 client.14895 fetch_fsmap learned FSMap version 23 2024-05-12T22:08:15.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.330+0000 7f7a5b9fd740 1 -- 172.21.15.28:0/3028110608 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x56327c4a0ba0 con 0x56327c424190 2024-05-12T22:08:15.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.330+0000 7f7a33400640 1 -- 172.21.15.28:0/3028110608 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7a40021070 con 0x56327c424190 2024-05-12T22:08:15.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.330+0000 7f7a33400640 1 -- 172.21.15.28:0/3028110608 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f7a4c028040 con 0x56327c424190 2024-05-12T22:08:15.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.331+0000 7f7a5b9fd740 10 client.14895 fetch_fsmap finished waiting for FSMap version 23 2024-05-12T22:08:15.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.331+0000 7f7a5b9fd740 10 client.14895 resolve_mds: resolved name 'a' to daemon mds.a 2024-05-12T22:08:15.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.331+0000 7f7a5b9fd740 20 client.14895 populate_metadata read hostname 'smithi028' 2024-05-12T22:08:15.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.331+0000 7f7a5b9fd740 1 --2- 172.21.15.28:0/3028110608 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x56327c4a54d0 0x56327c4a41e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:15.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.331+0000 7f7a5b9fd740 4 client.14895 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-12T22:08:15.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.331+0000 7f7a5b9fd740 1 -- 172.21.15.28:0/3028110608 --> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] -- command(tid 0: {"prefix": "status"}) -- 0x56327c41fec0 con 0x56327c4a54d0 2024-05-12T22:08:15.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.331+0000 7f7a51400640 1 --2- 172.21.15.28:0/3028110608 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x56327c4a54d0 0x56327c4a41e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:15.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.332+0000 7f7a51400640 1 --2- 172.21.15.28:0/3028110608 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x56327c4a54d0 0x56327c4a41e0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=53804cf7ca7a83f0 server_cookie=8e9e22c1a5816e44 in_seq=0 out_seq=0 2024-05-12T22:08:15.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.332+0000 7f7a33400640 10 client.14895 ms_handle_connect on v2:172.21.15.28:6840/761183870 2024-05-12T22:08:15.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.332+0000 7f7a33400640 1 -- 172.21.15.28:0/3028110608 <== mds.1 v2:172.21.15.28:6840/761183870 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x56327c41fec0 con 0x56327c4a54d0 2024-05-12T22:08:15.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.332+0000 7f7a33400640 10 client.14895 handle_command_reply: tid=0 2024-05-12T22:08:15.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.332+0000 7f7a5b9fd740 1 client.14895 shutdown 2024-05-12T22:08:15.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.332+0000 7f7a5b9fd740 1 -- 172.21.15.28:0/3028110608 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56327c424190 msgr2=0x56327c4337b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:15.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.332+0000 7f7a5b9fd740 1 --2- 172.21.15.28:0/3028110608 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56327c424190 0x56327c4337b0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f7a4c0098b0 tx=0x7f7a4c009600 comp rx=0 tx=0).stop 2024-05-12T22:08:15.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.332+0000 7f7a5b9fd740 1 -- 172.21.15.28:0/3028110608 shutdown_connections 2024-05-12T22:08:15.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.332+0000 7f7a5b9fd740 1 --2- 172.21.15.28:0/3028110608 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x56327c4a54d0 0x56327c4a41e0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:15.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.332+0000 7f7a33400640 0 client.14895 ms_handle_reset on v2:172.21.15.28:6840/761183870 2024-05-12T22:08:15.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.332+0000 7f7a5b9fd740 1 --2- 172.21.15.28:0/3028110608 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56327c429cf0 0x56327c49f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:15.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.332+0000 7f7a5b9fd740 1 --2- 172.21.15.28:0/3028110608 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56327c424b80 0x56327c49cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:15.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.332+0000 7f7a5b9fd740 1 --2- 172.21.15.28:0/3028110608 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56327c424190 0x56327c4337b0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:15.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.332+0000 7f7a5b9fd740 1 -- 172.21.15.28:0/3028110608 >> 172.21.15.28:0/3028110608 conn(0x56327c3eb8a0 msgr2=0x56327c425620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:08:15.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.332+0000 7f7a5b9fd740 1 -- 172.21.15.28:0/3028110608 shutdown_connections 2024-05-12T22:08:15.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.332+0000 7f7a5b9fd740 1 -- 172.21.15.28:0/3028110608 wait complete. 2024-05-12T22:08:15.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.333+0000 7f7a5b9fd740 20 client.14895 trim_cache size 0 max 16384 2024-05-12T22:08:15.336 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-05-12T22:08:15.336 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "6b92e7b4-10aa-11ef-bc98-c7b262605968", 2024-05-12T22:08:15.336 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 1, 2024-05-12T22:08:15.337 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 24550, 2024-05-12T22:08:15.337 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-05-12T22:08:15.337 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-05-12T22:08:15.337 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-05-12T22:08:15.337 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 98.030118329999993, 2024-05-12T22:08:15.337 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 23, 2024-05-12T22:08:15.337 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 88, 2024-05-12T22:08:15.337 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 86, 2024-05-12T22:08:15.337 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 110.51972041099999 2024-05-12T22:08:15.337 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-05-12T22:08:15.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.333+0000 7f7a53200640 1 -- 172.21.15.28:0/3650479349 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f7a380a2c10 msgr2=0x7f7a380a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:15.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.333+0000 7f7a53200640 1 --2- 172.21.15.28:0/3650479349 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f7a380a2c10 0x7f7a380a50d0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f7a440bbe10 tx=0x7f7a440f5060 comp rx=0 tx=0).stop 2024-05-12T22:08:15.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.333+0000 7f7a53200640 1 -- 172.21.15.28:0/3650479349 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7a54107d70 msgr2=0x7f7a5419e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:15.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.333+0000 7f7a53200640 1 --2- 172.21.15.28:0/3650479349 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7a54107d70 0x7f7a5419e050 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f7a44031440 tx=0x7f7a44032e20 comp rx=0 tx=0).stop 2024-05-12T22:08:15.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.333+0000 7f7a53200640 1 -- 172.21.15.28:0/3650479349 shutdown_connections 2024-05-12T22:08:15.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.333+0000 7f7a53200640 1 --2- 172.21.15.28:0/3650479349 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f7a380a2c10 0x7f7a380a50d0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:15.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.333+0000 7f7a53200640 1 --2- 172.21.15.28:0/3650479349 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7a54109920 0x7f7a541a5600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:15.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.333+0000 7f7a53200640 1 --2- 172.21.15.28:0/3650479349 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a54108f60 0x7f7a5419e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:15.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.333+0000 7f7a53200640 1 --2- 172.21.15.28:0/3650479349 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7a54107d70 0x7f7a5419e050 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:15.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.333+0000 7f7a53200640 1 -- 172.21.15.28:0/3650479349 >> 172.21.15.28:0/3650479349 conn(0x7f7a54076d00 msgr2=0x7f7a5410c200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:08:15.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.333+0000 7f7a53200640 1 -- 172.21.15.28:0/3650479349 shutdown_connections 2024-05-12T22:08:15.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.333+0000 7f7a53200640 1 -- 172.21.15.28:0/3650479349 wait complete. 2024-05-12T22:08:15.353 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-05-12T22:08:15.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:14 smithi186 ceph-mon[34785]: pgmap v362: 97 pgs: 97 active+clean; 641 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-12T22:08:15.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:14 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/1590771468' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-12T22:08:15.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:14 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/1696090119' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:08:15.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.571+0000 7fbec5200640 1 -- 172.21.15.28:0/3024626035 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbec0102930 msgr2=0x7fbec0102db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:15.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.571+0000 7fbec5200640 1 --2- 172.21.15.28:0/3024626035 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbec0102930 0x7fbec0102db0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fbeac009900 tx=0x7fbeac02a3f0 comp rx=0 tx=0).stop 2024-05-12T22:08:15.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.571+0000 7fbec5200640 1 -- 172.21.15.28:0/3024626035 shutdown_connections 2024-05-12T22:08:15.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.571+0000 7fbec5200640 1 --2- 172.21.15.28:0/3024626035 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbec01032f0 0x7fbec0109b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:15.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.571+0000 7fbec5200640 1 --2- 172.21.15.28:0/3024626035 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbec0102930 0x7fbec0102db0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:15.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.571+0000 7fbec5200640 1 --2- 172.21.15.28:0/3024626035 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbec0101740 0x7fbec0101b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:15.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.571+0000 7fbec5200640 1 -- 172.21.15.28:0/3024626035 >> 172.21.15.28:0/3024626035 conn(0x7fbec00fccd0 msgr2=0x7fbec00ff0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:08:15.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.571+0000 7fbec5200640 1 -- 172.21.15.28:0/3024626035 shutdown_connections 2024-05-12T22:08:15.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.571+0000 7fbec5200640 1 -- 172.21.15.28:0/3024626035 wait complete. 2024-05-12T22:08:15.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.572+0000 7fbec5200640 1 Processor -- start 2024-05-12T22:08:15.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.573+0000 7fbec5200640 1 -- start start 2024-05-12T22:08:15.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.573+0000 7fbec5200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbec0101740 0x7fbec0199c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:15.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.573+0000 7fbec5200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbec0102930 0x7fbec019a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:15.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.573+0000 7fbec5200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbec01032f0 0x7fbec01a1230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:15.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.573+0000 7fbec5200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbec010d3f0 con 0x7fbec01032f0 2024-05-12T22:08:15.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.573+0000 7fbec5200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fbec010d270 con 0x7fbec0102930 2024-05-12T22:08:15.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.573+0000 7fbec5200640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fbec010d570 con 0x7fbec0101740 2024-05-12T22:08:15.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.573+0000 7fbebfe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbec01032f0 0x7fbec01a1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:15.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.573+0000 7fbebfe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbec01032f0 0x7fbec01a1230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38494/0 (socket says 172.21.15.28:38494) 2024-05-12T22:08:15.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.573+0000 7fbebfe00640 1 -- 172.21.15.28:0/2172036780 learned_addr learned my addr 172.21.15.28:0/2172036780 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:08:15.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.573+0000 7fbebf400640 1 --2- 172.21.15.28:0/2172036780 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbec0101740 0x7fbec0199c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:15.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.573+0000 7fbebea00640 1 --2- 172.21.15.28:0/2172036780 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbec0102930 0x7fbec019a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:15.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.574+0000 7fbebfe00640 1 -- 172.21.15.28:0/2172036780 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbec0101740 msgr2=0x7fbec0199c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:15.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.574+0000 7fbebfe00640 1 --2- 172.21.15.28:0/2172036780 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbec0101740 0x7fbec0199c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:15.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.574+0000 7fbebfe00640 1 -- 172.21.15.28:0/2172036780 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbec0102930 msgr2=0x7fbec019a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:15.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.574+0000 7fbebfe00640 1 --2- 172.21.15.28:0/2172036780 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbec0102930 0x7fbec019a1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:15.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.574+0000 7fbebfe00640 1 -- 172.21.15.28:0/2172036780 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbec01a18d0 con 0x7fbec01032f0 2024-05-12T22:08:15.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.574+0000 7fbebf400640 1 --2- 172.21.15.28:0/2172036780 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbec0101740 0x7fbec0199c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:08:15.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.574+0000 7fbebea00640 1 --2- 172.21.15.28:0/2172036780 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbec0102930 0x7fbec019a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:08:15.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.574+0000 7fbebfe00640 1 --2- 172.21.15.28:0/2172036780 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbec01032f0 0x7fbec01a1230 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fbeb8009fd0 tx=0x7fbeb800b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:08:15.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.574+0000 7fbeb7e00640 1 -- 172.21.15.28:0/2172036780 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbeb8017070 con 0x7fbec01032f0 2024-05-12T22:08:15.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.574+0000 7fbeb7e00640 1 -- 172.21.15.28:0/2172036780 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbeb8003e50 con 0x7fbec01032f0 2024-05-12T22:08:15.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.574+0000 7fbec5200640 1 -- 172.21.15.28:0/2172036780 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbec01a1bc0 con 0x7fbec01032f0 2024-05-12T22:08:15.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.574+0000 7fbeb7e00640 1 -- 172.21.15.28:0/2172036780 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbeb80106c0 con 0x7fbec01032f0 2024-05-12T22:08:15.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.575+0000 7fbec5200640 1 -- 172.21.15.28:0/2172036780 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbec01a2090 con 0x7fbec01032f0 2024-05-12T22:08:15.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.578+0000 7fbec95da740 1 Processor -- start 2024-05-12T22:08:15.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.578+0000 7fbeb7e00640 1 -- 172.21.15.28:0/2172036780 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fbeb8003970 con 0x7fbec01032f0 2024-05-12T22:08:15.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.578+0000 7fbec95da740 1 -- start start 2024-05-12T22:08:15.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.578+0000 7fbec95da740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a0cad27190 0x55a0cad275b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:15.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.578+0000 7fbec95da740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a0cad27b80 0x55a0cad2c680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:15.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.578+0000 7fbec95da740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a0cad2ccf0 0x55a0cad2f180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:15.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.578+0000 7fbec95da740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a0cac8a480 con 0x55a0cad2ccf0 2024-05-12T22:08:15.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.578+0000 7fbec95da740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x55a0cac325c0 con 0x55a0cad27b80 2024-05-12T22:08:15.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.578+0000 7fbec95da740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55a0cac86440 con 0x55a0cad27190 2024-05-12T22:08:15.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.579+0000 7fbebea00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a0cad27190 0x55a0cad275b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:15.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.579+0000 7fbebf400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a0cad27b80 0x55a0cad2c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:15.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.579+0000 7fbebea00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a0cad27190 0x55a0cad275b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:34600/0 (socket says 172.21.15.28:34600) 2024-05-12T22:08:15.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.579+0000 7fbebea00640 1 -- 172.21.15.28:0/2102745503 learned_addr learned my addr 172.21.15.28:0/2102745503 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:08:15.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.579+0000 7fbebea00640 1 --2- 172.21.15.28:0/2102745503 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a0cad2ccf0 0x55a0cad2f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:15.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.579+0000 7fbebf400640 1 -- 172.21.15.28:0/2102745503 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a0cad27190 msgr2=0x55a0cad275b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:08:15.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.579+0000 7fbebf400640 1 --2- 172.21.15.28:0/2102745503 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a0cad27190 0x55a0cad275b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:15.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.579+0000 7fbebf400640 1 -- 172.21.15.28:0/2102745503 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a0cad2ccf0 msgr2=0x55a0cad2f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:08:15.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.579+0000 7fbebf400640 1 --2- 172.21.15.28:0/2102745503 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a0cad2ccf0 0x55a0cad2f180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:15.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.579+0000 7fbebf400640 1 -- 172.21.15.28:0/2102745503 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a0ca99e2d0 con 0x55a0cad27b80 2024-05-12T22:08:15.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.580+0000 7fbebf400640 1 --2- 172.21.15.28:0/2102745503 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a0cad27b80 0x55a0cad2c680 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fbec0102790 tx=0x7fbeb00118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a666a773215ef81e server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:08:15.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.580+0000 7fbeb7e00640 1 --2- 172.21.15.28:0/2172036780 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fbea40a2b90 0x7fbea40a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:15.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.580+0000 7fbeb4c00640 1 -- 172.21.15.28:0/2102745503 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbeb0004310 con 0x55a0cad27b80 2024-05-12T22:08:15.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.580+0000 7fbeb4c00640 1 -- 172.21.15.28:0/2102745503 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbeb00044b0 con 0x55a0cad27b80 2024-05-12T22:08:15.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.580+0000 7fbeb4c00640 1 -- 172.21.15.28:0/2102745503 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbeb00035f0 con 0x55a0cad27b80 2024-05-12T22:08:15.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.580+0000 7fbeb7e00640 1 -- 172.21.15.28:0/2172036780 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7fbeb80ced80 con 0x7fbec01032f0 2024-05-12T22:08:15.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.580+0000 7fbebea00640 1 --2- 172.21.15.28:0/2172036780 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fbea40a2b90 0x7fbea40a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:15.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.581+0000 7fbec95da740 1 -- 172.21.15.28:0/2102745503 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a0cad27b80 msgr2=0x55a0cad2c680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:15.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.581+0000 7fbebea00640 1 --2- 172.21.15.28:0/2172036780 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fbea40a2b90 0x7fbea40a5050 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fbeac006010 tx=0x7fbeac0315b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:08:15.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.581+0000 7fbec95da740 1 --2- 172.21.15.28:0/2102745503 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a0cad27b80 0x55a0cad2c680 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fbec0102790 tx=0x7fbeb00118d0 comp rx=0 tx=0).stop 2024-05-12T22:08:15.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.581+0000 7fbec95da740 1 -- 172.21.15.28:0/2102745503 shutdown_connections 2024-05-12T22:08:15.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.581+0000 7fbec95da740 1 --2- 172.21.15.28:0/2102745503 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a0cad2ccf0 0x55a0cad2f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:15.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.581+0000 7fbec95da740 1 --2- 172.21.15.28:0/2102745503 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a0cad27b80 0x55a0cad2c680 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:15.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.581+0000 7fbec95da740 1 --2- 172.21.15.28:0/2102745503 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a0cad27190 0x55a0cad275b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:15.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.581+0000 7fbec95da740 1 -- 172.21.15.28:0/2102745503 >> 172.21.15.28:0/2102745503 conn(0x55a0cacee8a0 msgr2=0x55a0cad22c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:08:15.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.581+0000 7fbec95da740 1 -- 172.21.15.28:0/2102745503 shutdown_connections 2024-05-12T22:08:15.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.581+0000 7fbec95da740 1 -- 172.21.15.28:0/2102745503 wait complete. 2024-05-12T22:08:15.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.582+0000 7fbec95da740 1 -- start start 2024-05-12T22:08:15.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.582+0000 7fbec95da740 1 Processor -- start 2024-05-12T22:08:15.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.582+0000 7fbec95da740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a0cad27190 0x55a0cad31220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:15.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.582+0000 7fbec95da740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a0cad27b80 0x55a0cad31760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:15.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.582+0000 7fbec95da740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a0cad2ccf0 0x55a0cad387a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:15.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.582+0000 7fbec95da740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a0cad2fca0 con 0x55a0cad2ccf0 2024-05-12T22:08:15.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.582+0000 7fbec95da740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x55a0cac45980 con 0x55a0cad27190 2024-05-12T22:08:15.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.582+0000 7fbec95da740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55a0cad2fe20 con 0x55a0cad27b80 2024-05-12T22:08:15.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.582+0000 7fbec95da740 10 client.0 _ll_register_callbacks cb 0x55a0cac80fc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:08:15.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.583+0000 7fbebf400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a0cad27190 0x55a0cad31220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:15.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.583+0000 7fbebf400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a0cad27190 0x55a0cad31220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:51354/0 (socket says 172.21.15.28:51354) 2024-05-12T22:08:15.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.583+0000 7fbebf400640 1 -- 172.21.15.28:0/2097691378 learned_addr learned my addr 172.21.15.28:0/2097691378 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:08:15.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.583+0000 7fbebea00640 1 --2- 172.21.15.28:0/2097691378 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a0cad27b80 0x55a0cad31760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:15.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.583+0000 7fbebfe00640 1 --2- 172.21.15.28:0/2097691378 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a0cad2ccf0 0x55a0cad387a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:15.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.583+0000 7fbebf400640 1 -- 172.21.15.28:0/2097691378 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a0cad27b80 msgr2=0x55a0cad31760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:08:15.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.583+0000 7fbebf400640 1 --2- 172.21.15.28:0/2097691378 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a0cad27b80 0x55a0cad31760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:15.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.583+0000 7fbebf400640 1 -- 172.21.15.28:0/2097691378 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a0cad2ccf0 msgr2=0x55a0cad387a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:08:15.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.583+0000 7fbebf400640 1 --2- 172.21.15.28:0/2097691378 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a0cad2ccf0 0x55a0cad387a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:15.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.583+0000 7fbebf400640 1 -- 172.21.15.28:0/2097691378 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a0caa686b0 con 0x55a0cad27190 2024-05-12T22:08:15.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.583+0000 7fbebf400640 1 --2- 172.21.15.28:0/2097691378 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a0cad27190 0x55a0cad31220 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fbeb0011dd0 tx=0x7fbeb00038a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=800788d0dd328ca3 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:08:15.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.583+0000 7fbea9600640 10 client.0 ms_handle_connect on v2:172.21.15.86:3300/0 2024-05-12T22:08:15.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.584+0000 7fbea9600640 1 -- 172.21.15.28:0/2097691378 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbeb0029070 con 0x55a0cad27190 2024-05-12T22:08:15.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.584+0000 7fbea9600640 1 -- 172.21.15.28:0/2097691378 <== mon.2 v2:172.21.15.86:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7fbeb0004d20 con 0x55a0cad27190 2024-05-12T22:08:15.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.584+0000 7fbec95da740 1 -- 172.21.15.28:0/2097691378 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbeb00044b0 con 0x55a0cad27190 2024-05-12T22:08:15.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.584+0000 7fbea9600640 1 -- 172.21.15.28:0/2097691378 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fbeb0035050 con 0x55a0cad27190 2024-05-12T22:08:15.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.584+0000 7fbec95da740 10 client.25022 fetch_fsmap learned FSMap version 23 2024-05-12T22:08:15.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.584+0000 7fbec95da740 1 -- 172.21.15.28:0/2097691378 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55a0cad3b7d0 con 0x55a0cad27190 2024-05-12T22:08:15.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.584+0000 7fbea9600640 1 -- 172.21.15.28:0/2097691378 <== mon.2 v2:172.21.15.86:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbeb00044b0 con 0x55a0cad27190 2024-05-12T22:08:15.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.584+0000 7fbea9600640 1 -- 172.21.15.28:0/2097691378 <== mon.2 v2:172.21.15.86:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fbeb000f480 con 0x55a0cad27190 2024-05-12T22:08:15.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.584+0000 7fbec95da740 10 client.25022 fetch_fsmap finished waiting for FSMap version 23 2024-05-12T22:08:15.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.584+0000 7fbec95da740 10 client.25022 resolve_mds: resolved name 'b' to daemon mds.b 2024-05-12T22:08:15.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.584+0000 7fbec95da740 20 client.25022 populate_metadata read hostname 'smithi028' 2024-05-12T22:08:15.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.584+0000 7fbec95da740 1 --2- 172.21.15.28:0/2097691378 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55a0cad3dc00 0x55a0cad3f070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:15.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.585+0000 7fbec95da740 4 client.25022 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-12T22:08:15.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.585+0000 7fbec95da740 1 -- 172.21.15.28:0/2097691378 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a0cac9e060 con 0x55a0cad3dc00 2024-05-12T22:08:15.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.585+0000 7fbebea00640 1 --2- 172.21.15.28:0/2097691378 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55a0cad3dc00 0x55a0cad3f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:15.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.585+0000 7fbebea00640 1 --2- 172.21.15.28:0/2097691378 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55a0cad3dc00 0x55a0cad3f070 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a334d780378f7f99 server_cookie=1768bb295c0cb029 in_seq=0 out_seq=0 2024-05-12T22:08:15.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.585+0000 7fbea9600640 10 client.25022 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:08:15.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.586+0000 7fbea9600640 1 -- 172.21.15.28:0/2097691378 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a0cac9e060 con 0x55a0cad3dc00 2024-05-12T22:08:15.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.586+0000 7fbea9600640 10 client.25022 handle_command_reply: tid=0 2024-05-12T22:08:15.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.586+0000 7fbec95da740 1 client.25022 shutdown 2024-05-12T22:08:15.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.586+0000 7fbec95da740 1 -- 172.21.15.28:0/2097691378 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a0cad27190 msgr2=0x55a0cad31220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:15.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.586+0000 7fbec95da740 1 --2- 172.21.15.28:0/2097691378 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a0cad27190 0x55a0cad31220 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fbeb0011dd0 tx=0x7fbeb00038a0 comp rx=0 tx=0).stop 2024-05-12T22:08:15.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.586+0000 7fbec95da740 1 -- 172.21.15.28:0/2097691378 shutdown_connections 2024-05-12T22:08:15.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.586+0000 7fbec95da740 1 --2- 172.21.15.28:0/2097691378 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55a0cad3dc00 0x55a0cad3f070 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:15.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.586+0000 7fbec95da740 1 --2- 172.21.15.28:0/2097691378 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a0cad2ccf0 0x55a0cad387a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:15.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.586+0000 7fbec95da740 1 --2- 172.21.15.28:0/2097691378 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a0cad27b80 0x55a0cad31760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:15.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.586+0000 7fbec95da740 1 --2- 172.21.15.28:0/2097691378 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a0cad27190 0x55a0cad31220 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:15.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.586+0000 7fbec95da740 1 -- 172.21.15.28:0/2097691378 >> 172.21.15.28:0/2097691378 conn(0x55a0cacee8a0 msgr2=0x55a0cad21c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:08:15.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.586+0000 7fbea9600640 0 client.25022 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:08:15.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.587+0000 7fbec95da740 1 -- 172.21.15.28:0/2097691378 shutdown_connections 2024-05-12T22:08:15.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.587+0000 7fbec95da740 1 -- 172.21.15.28:0/2097691378 wait complete. 2024-05-12T22:08:15.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.587+0000 7fbec95da740 20 client.25022 trim_cache size 0 max 16384 2024-05-12T22:08:15.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.608+0000 7fbec95da740 1 Processor -- start 2024-05-12T22:08:15.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.608+0000 7fbec95da740 1 -- start start 2024-05-12T22:08:15.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.608+0000 7fbec95da740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a0cad27190 0x55a0cad2f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:15.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.608+0000 7fbec95da740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a0cad27b80 0x55a0cad38760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:15.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.608+0000 7fbec95da740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a0cad2ccf0 0x55a0cad257d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:15.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.608+0000 7fbec95da740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a0cad3d3b0 con 0x55a0cad2ccf0 2024-05-12T22:08:15.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.608+0000 7fbec95da740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x55a0cad3da80 con 0x55a0cad27190 2024-05-12T22:08:15.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.608+0000 7fbec95da740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55a0cad3e150 con 0x55a0cad27b80 2024-05-12T22:08:15.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.608+0000 7fbebfe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a0cad2ccf0 0x55a0cad257d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:15.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.608+0000 7fbebfe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a0cad2ccf0 0x55a0cad257d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38532/0 (socket says 172.21.15.28:38532) 2024-05-12T22:08:15.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.608+0000 7fbebfe00640 1 -- 172.21.15.28:0/2111792508 learned_addr learned my addr 172.21.15.28:0/2111792508 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:08:15.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.608+0000 7fbebea00640 1 --2- 172.21.15.28:0/2111792508 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a0cad27b80 0x55a0cad38760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:15.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.609+0000 7fbebf400640 1 --2- 172.21.15.28:0/2111792508 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a0cad27190 0x55a0cad2f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:15.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.609+0000 7fbebea00640 1 -- 172.21.15.28:0/2111792508 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a0cad27190 msgr2=0x55a0cad2f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:08:15.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.609+0000 7fbebea00640 1 --2- 172.21.15.28:0/2111792508 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a0cad27190 0x55a0cad2f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:15.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.609+0000 7fbebea00640 1 -- 172.21.15.28:0/2111792508 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a0cad2ccf0 msgr2=0x55a0cad257d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:08:15.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.609+0000 7fbebea00640 1 --2- 172.21.15.28:0/2111792508 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a0cad2ccf0 0x55a0cad257d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:15.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.609+0000 7fbebea00640 1 -- 172.21.15.28:0/2111792508 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a0cad32350 con 0x55a0cad27b80 2024-05-12T22:08:15.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.609+0000 7fbebea00640 1 --2- 172.21.15.28:0/2111792508 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a0cad27b80 0x55a0cad38760 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fbeac005ec0 tx=0x7fbeac009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=61a313b12c38f08f server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:08:15.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.609+0000 7fbeabe00640 1 -- 172.21.15.28:0/2111792508 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbeac05a070 con 0x55a0cad27b80 2024-05-12T22:08:15.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.609+0000 7fbeabe00640 1 -- 172.21.15.28:0/2111792508 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbeac0548b0 con 0x55a0cad27b80 2024-05-12T22:08:15.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.609+0000 7fbeabe00640 1 -- 172.21.15.28:0/2111792508 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbeac004bb0 con 0x55a0cad27b80 2024-05-12T22:08:15.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.610+0000 7fbec95da740 1 -- 172.21.15.28:0/2111792508 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a0cad27b80 msgr2=0x55a0cad38760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:15.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.610+0000 7fbec95da740 1 --2- 172.21.15.28:0/2111792508 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a0cad27b80 0x55a0cad38760 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fbeac005ec0 tx=0x7fbeac009260 comp rx=0 tx=0).stop 2024-05-12T22:08:15.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.610+0000 7fbec95da740 1 -- 172.21.15.28:0/2111792508 shutdown_connections 2024-05-12T22:08:15.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.610+0000 7fbec95da740 1 --2- 172.21.15.28:0/2111792508 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a0cad2ccf0 0x55a0cad257d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:15.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.610+0000 7fbec95da740 1 --2- 172.21.15.28:0/2111792508 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a0cad27b80 0x55a0cad38760 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:15.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.610+0000 7fbec95da740 1 --2- 172.21.15.28:0/2111792508 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a0cad27190 0x55a0cad2f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:15.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.610+0000 7fbec95da740 1 -- 172.21.15.28:0/2111792508 >> 172.21.15.28:0/2111792508 conn(0x55a0cacee8a0 msgr2=0x55a0cad3cb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:08:15.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.610+0000 7fbec95da740 1 -- 172.21.15.28:0/2111792508 shutdown_connections 2024-05-12T22:08:15.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.610+0000 7fbec95da740 1 -- 172.21.15.28:0/2111792508 wait complete. 2024-05-12T22:08:15.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.611+0000 7fbec95da740 1 -- start start 2024-05-12T22:08:15.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.611+0000 7fbec95da740 1 Processor -- start 2024-05-12T22:08:15.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.611+0000 7fbec95da740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a0cad27190 0x55a0cad33450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:15.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.611+0000 7fbec95da740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a0cad27b80 0x55a0cad33990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:15.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.611+0000 7fbec95da740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a0cad2ccf0 0x55a0cadaa3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:15.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.611+0000 7fbec95da740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a0cad3eef0 con 0x55a0cad2ccf0 2024-05-12T22:08:15.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.611+0000 7fbec95da740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x55a0cad3e820 con 0x55a0cad27190 2024-05-12T22:08:15.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.611+0000 7fbec95da740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55a0cad3f5c0 con 0x55a0cad27b80 2024-05-12T22:08:15.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.611+0000 7fbec95da740 10 client.0 _ll_register_callbacks cb 0x7fbeac0326c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:08:15.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.612+0000 7fbebfe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a0cad2ccf0 0x55a0cadaa3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:15.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.612+0000 7fbebea00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a0cad27b80 0x55a0cad33990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:15.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.612+0000 7fbebf400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a0cad27190 0x55a0cad33450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:15.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.612+0000 7fbebea00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a0cad27b80 0x55a0cad33990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:34636/0 (socket says 172.21.15.28:34636) 2024-05-12T22:08:15.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.612+0000 7fbebea00640 1 -- 172.21.15.28:0/224924745 learned_addr learned my addr 172.21.15.28:0/224924745 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:08:15.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.612+0000 7fbebea00640 1 -- 172.21.15.28:0/224924745 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a0cad27190 msgr2=0x55a0cad33450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:08:15.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.612+0000 7fbebea00640 1 --2- 172.21.15.28:0/224924745 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a0cad27190 0x55a0cad33450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:15.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.612+0000 7fbebea00640 1 -- 172.21.15.28:0/224924745 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a0cad2ccf0 msgr2=0x55a0cadaa3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:08:15.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.612+0000 7fbebea00640 1 --2- 172.21.15.28:0/224924745 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a0cad2ccf0 0x55a0cadaa3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:15.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.612+0000 7fbebea00640 1 -- 172.21.15.28:0/224924745 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a0cad33f80 con 0x55a0cad27b80 2024-05-12T22:08:15.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.612+0000 7fbebea00640 1 --2- 172.21.15.28:0/224924745 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a0cad27b80 0x55a0cad33990 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fbeac000c00 tx=0x7fbeac02a3f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=53c42610162e8d25 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:08:15.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.612+0000 7fbea9600640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-12T22:08:15.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.613+0000 7fbea9600640 1 -- 172.21.15.28:0/224924745 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbeac05a070 con 0x55a0cad27b80 2024-05-12T22:08:15.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.613+0000 7fbea9600640 1 -- 172.21.15.28:0/224924745 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7fbeac005490 con 0x55a0cad27b80 2024-05-12T22:08:15.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.613+0000 7fbec95da740 1 -- 172.21.15.28:0/224924745 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbeac0548b0 con 0x55a0cad27b80 2024-05-12T22:08:15.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.613+0000 7fbea9600640 1 -- 172.21.15.28:0/224924745 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fbeac005690 con 0x55a0cad27b80 2024-05-12T22:08:15.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.613+0000 7fbec95da740 10 client.25006 fetch_fsmap learned FSMap version 23 2024-05-12T22:08:15.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.613+0000 7fbec95da740 1 -- 172.21.15.28:0/224924745 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55a0cadab960 con 0x55a0cad27b80 2024-05-12T22:08:15.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.613+0000 7fbea9600640 1 -- 172.21.15.28:0/224924745 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbeac0548b0 con 0x55a0cad27b80 2024-05-12T22:08:15.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.614+0000 7fbea9600640 1 -- 172.21.15.28:0/224924745 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fbeac003710 con 0x55a0cad27b80 2024-05-12T22:08:15.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.614+0000 7fbec95da740 10 client.25006 fetch_fsmap finished waiting for FSMap version 23 2024-05-12T22:08:15.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.614+0000 7fbec95da740 10 client.25006 resolve_mds: resolved name 'b' to daemon mds.b 2024-05-12T22:08:15.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.614+0000 7fbec95da740 20 client.25006 populate_metadata read hostname 'smithi028' 2024-05-12T22:08:15.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.614+0000 7fbec95da740 1 --2- 172.21.15.28:0/224924745 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55a0cadb0040 0x55a0cadaed50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:15.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.614+0000 7fbec95da740 4 client.25006 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-12T22:08:15.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.614+0000 7fbec95da740 1 -- 172.21.15.28:0/224924745 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "status"}) -- 0x55a0cac97bb0 con 0x55a0cadb0040 2024-05-12T22:08:15.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.614+0000 7fbebf400640 1 --2- 172.21.15.28:0/224924745 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55a0cadb0040 0x55a0cadaed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:15.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.615+0000 7fbebf400640 1 --2- 172.21.15.28:0/224924745 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55a0cadb0040 0x55a0cadaed50 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5089a2ddb622077 server_cookie=923d024cfdd272ce in_seq=0 out_seq=0 2024-05-12T22:08:15.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.615+0000 7fbea9600640 10 client.25006 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:08:15.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.615+0000 7fbea9600640 1 -- 172.21.15.28:0/224924745 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x55a0cac97bb0 con 0x55a0cadb0040 2024-05-12T22:08:15.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.615+0000 7fbea9600640 10 client.25006 handle_command_reply: tid=0 2024-05-12T22:08:15.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.615+0000 7fbec95da740 1 client.25006 shutdown 2024-05-12T22:08:15.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.615+0000 7fbec95da740 1 -- 172.21.15.28:0/224924745 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a0cad27b80 msgr2=0x55a0cad33990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:15.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.615+0000 7fbec95da740 1 --2- 172.21.15.28:0/224924745 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a0cad27b80 0x55a0cad33990 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fbeac000c00 tx=0x7fbeac02a3f0 comp rx=0 tx=0).stop 2024-05-12T22:08:15.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.615+0000 7fbec95da740 1 -- 172.21.15.28:0/224924745 shutdown_connections 2024-05-12T22:08:15.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.615+0000 7fbec95da740 1 --2- 172.21.15.28:0/224924745 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55a0cadb0040 0x55a0cadaed50 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:15.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.615+0000 7fbec95da740 1 --2- 172.21.15.28:0/224924745 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a0cad2ccf0 0x55a0cadaa3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:15.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.615+0000 7fbec95da740 1 --2- 172.21.15.28:0/224924745 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a0cad27b80 0x55a0cad33990 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:15.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.615+0000 7fbec95da740 1 --2- 172.21.15.28:0/224924745 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a0cad27190 0x55a0cad33450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:15.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.615+0000 7fbec95da740 1 -- 172.21.15.28:0/224924745 >> 172.21.15.28:0/224924745 conn(0x55a0cacee8a0 msgr2=0x55a0cad28530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:08:15.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.615+0000 7fbea9600640 0 client.25006 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:08:15.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.615+0000 7fbec95da740 1 -- 172.21.15.28:0/224924745 shutdown_connections 2024-05-12T22:08:15.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.615+0000 7fbec95da740 1 -- 172.21.15.28:0/224924745 wait complete. 2024-05-12T22:08:15.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.615+0000 7fbec95da740 20 client.25006 trim_cache size 0 max 16384 2024-05-12T22:08:15.620 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-05-12T22:08:15.620 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "6b92e7b4-10aa-11ef-bc98-c7b262605968", 2024-05-12T22:08:15.620 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 0, 2024-05-12T22:08:15.620 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 24581, 2024-05-12T22:08:15.620 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-05-12T22:08:15.620 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-05-12T22:08:15.621 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-05-12T22:08:15.621 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 101.354050207, 2024-05-12T22:08:15.621 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 23, 2024-05-12T22:08:15.621 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 88, 2024-05-12T22:08:15.621 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 86, 2024-05-12T22:08:15.621 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 108.130376893 2024-05-12T22:08:15.621 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-05-12T22:08:15.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.616+0000 7fbec5200640 1 -- 172.21.15.28:0/2172036780 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fbea40a2b90 msgr2=0x7fbea40a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:15.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.616+0000 7fbec5200640 1 --2- 172.21.15.28:0/2172036780 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fbea40a2b90 0x7fbea40a5050 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fbeac006010 tx=0x7fbeac0315b0 comp rx=0 tx=0).stop 2024-05-12T22:08:15.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.616+0000 7fbec5200640 1 -- 172.21.15.28:0/2172036780 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbec01032f0 msgr2=0x7fbec01a1230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:15.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.616+0000 7fbec5200640 1 --2- 172.21.15.28:0/2172036780 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbec01032f0 0x7fbec01a1230 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fbeb8009fd0 tx=0x7fbeb800b260 comp rx=0 tx=0).stop 2024-05-12T22:08:15.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.616+0000 7fbec5200640 1 -- 172.21.15.28:0/2172036780 shutdown_connections 2024-05-12T22:08:15.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.616+0000 7fbec5200640 1 --2- 172.21.15.28:0/2172036780 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fbea40a2b90 0x7fbea40a5050 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:15.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.616+0000 7fbec5200640 1 --2- 172.21.15.28:0/2172036780 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbec01032f0 0x7fbec01a1230 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:15.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.616+0000 7fbec5200640 1 --2- 172.21.15.28:0/2172036780 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbec0102930 0x7fbec019a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:15.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.616+0000 7fbec5200640 1 --2- 172.21.15.28:0/2172036780 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbec0101740 0x7fbec0199c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:15.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.616+0000 7fbec5200640 1 -- 172.21.15.28:0/2172036780 >> 172.21.15.28:0/2172036780 conn(0x7fbec00fccd0 msgr2=0x7fbec0073a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:08:15.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.616+0000 7fbec5200640 1 -- 172.21.15.28:0/2172036780 shutdown_connections 2024-05-12T22:08:15.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:15.616+0000 7fbec5200640 1 -- 172.21.15.28:0/2172036780 wait complete. 2024-05-12T22:08:15.635 INFO:tasks.cephadm:Setup complete, yielding 2024-05-12T22:08:15.635 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-05-12T22:08:15.645 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi028.front.sepia.ceph.com 2024-05-12T22:08:15.646 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f72fecff68e1d400c4568684327c900485c20d6a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6b92e7b4-10aa-11ef-bc98-c7b262605968 -- bash -c 'ceph orch status' 2024-05-12T22:08:16.109 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/mon.a/config 2024-05-12T22:08:17.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:16 smithi086 ceph-mon[34297]: pgmap v363: 97 pgs: 97 active+clean; 641 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-12T22:08:17.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:16 smithi028 ceph-mon[29847]: pgmap v363: 97 pgs: 97 active+clean; 641 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-12T22:08:17.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:16 smithi186 ceph-mon[34785]: pgmap v363: 97 pgs: 97 active+clean; 641 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-12T22:08:18.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:18.557+0000 7feacac00700 1 -- 172.21.15.28:0/154210653 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feac4102e10 msgr2=0x7feac4105200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:18.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:18.557+0000 7feacac00700 1 --2- 172.21.15.28:0/154210653 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feac4102e10 0x7feac4105200 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7feab4009990 tx=0x7feab4009ca0 comp rx=0 tx=0).stop 2024-05-12T22:08:18.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:18.557+0000 7feacac00700 1 -- 172.21.15.28:0/154210653 shutdown_connections 2024-05-12T22:08:18.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:18.557+0000 7feacac00700 1 --2- 172.21.15.28:0/154210653 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feac4108070 0x7feac410a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:18.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:18.557+0000 7feacac00700 1 --2- 172.21.15.28:0/154210653 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feac4102e10 0x7feac4105200 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:18.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:18.557+0000 7feacac00700 1 --2- 172.21.15.28:0/154210653 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feac4105740 0x7feac4107b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:18.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:18.557+0000 7feacac00700 1 -- 172.21.15.28:0/154210653 >> 172.21.15.28:0/154210653 conn(0x7feac40fc820 msgr2=0x7feac40fec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:08:18.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:18.557+0000 7feacac00700 1 -- 172.21.15.28:0/154210653 shutdown_connections 2024-05-12T22:08:18.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:18.557+0000 7feacac00700 1 -- 172.21.15.28:0/154210653 wait complete. 2024-05-12T22:08:18.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:18.558+0000 7feacac00700 1 Processor -- start 2024-05-12T22:08:18.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:18.558+0000 7feacac00700 1 -- start start 2024-05-12T22:08:18.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:18.559+0000 7feacac00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feac4102e10 0x7feac41985e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:18.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:18.559+0000 7feacac00700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feac4105740 0x7feac4198b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:18.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:18.559+0000 7feacac00700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feac4108070 0x7feac419fba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:18.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:18.559+0000 7feacac00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7feac410df90 con 0x7feac4102e10 2024-05-12T22:08:18.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:18.559+0000 7feacac00700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7feac410de10 con 0x7feac4105740 2024-05-12T22:08:18.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:18.559+0000 7feacac00700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7feac410e110 con 0x7feac4108070 2024-05-12T22:08:18.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:18.559+0000 7feac3e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feac4102e10 0x7feac41985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:18.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:18.559+0000 7feac3e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feac4102e10 0x7feac41985e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48230/0 (socket says 172.21.15.28:48230) 2024-05-12T22:08:18.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:18.559+0000 7feac3e00700 1 -- 172.21.15.28:0/693856862 learned_addr learned my addr 172.21.15.28:0/693856862 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:08:18.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:18.559+0000 7feac8c00700 1 --2- 172.21.15.28:0/693856862 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feac4108070 0x7feac419fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:18.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:18.559+0000 7feac3400700 1 --2- 172.21.15.28:0/693856862 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feac4105740 0x7feac4198b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:18.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:18.560+0000 7feac8c00700 1 -- 172.21.15.28:0/693856862 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feac4105740 msgr2=0x7feac4198b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:18.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:18.560+0000 7feac8c00700 1 --2- 172.21.15.28:0/693856862 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feac4105740 0x7feac4198b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:18.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:18.560+0000 7feac8c00700 1 -- 172.21.15.28:0/693856862 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feac4102e10 msgr2=0x7feac41985e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:18.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:18.560+0000 7feac8c00700 1 --2- 172.21.15.28:0/693856862 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feac4102e10 0x7feac41985e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:18.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:18.560+0000 7feac8c00700 1 -- 172.21.15.28:0/693856862 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feac41a0240 con 0x7feac4108070 2024-05-12T22:08:18.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:18.560+0000 7feac3e00700 1 --2- 172.21.15.28:0/693856862 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feac4102e10 0x7feac41985e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:08:18.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:18.560+0000 7feac8c00700 1 --2- 172.21.15.28:0/693856862 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feac4108070 0x7feac419fba0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7feabc00c3c0 tx=0x7feabc00c6d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:08:18.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:18.560+0000 7feac0c00700 1 -- 172.21.15.28:0/693856862 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feabc004bb0 con 0x7feac4108070 2024-05-12T22:08:18.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:18.561+0000 7feacac00700 1 -- 172.21.15.28:0/693856862 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feac4100c00 con 0x7feac4108070 2024-05-12T22:08:18.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:18.561+0000 7feac0c00700 1 -- 172.21.15.28:0/693856862 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feabc004d50 con 0x7feac4108070 2024-05-12T22:08:18.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:18.561+0000 7feac0c00700 1 -- 172.21.15.28:0/693856862 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feabc0044e0 con 0x7feac4108070 2024-05-12T22:08:18.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:18.561+0000 7feacac00700 1 -- 172.21.15.28:0/693856862 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feac41010f0 con 0x7feac4108070 2024-05-12T22:08:18.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:18.562+0000 7feacac00700 1 -- 172.21.15.28:0/693856862 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feac4069010 con 0x7feac4108070 2024-05-12T22:08:18.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:18.564+0000 7feac0c00700 1 -- 172.21.15.28:0/693856862 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7feabc021020 con 0x7feac4108070 2024-05-12T22:08:18.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:18.565+0000 7feac0c00700 1 --2- 172.21.15.28:0/693856862 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7feaa80a2c20 0x7feaa80a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:18.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:18.566+0000 7feac0c00700 1 -- 172.21.15.28:0/693856862 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7feabc0d2be0 con 0x7feac4108070 2024-05-12T22:08:18.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:18.566+0000 7feac3e00700 1 --2- 172.21.15.28:0/693856862 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7feaa80a2c20 0x7feaa80a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:18.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:18.567+0000 7feac3e00700 1 --2- 172.21.15.28:0/693856862 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7feaa80a2c20 0x7feaa80a50e0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7feab400b7b0 tx=0x7feab400b8a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:08:18.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:18.568+0000 7feac0c00700 1 -- 172.21.15.28:0/693856862 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7feabc09faf0 con 0x7feac4108070 2024-05-12T22:08:18.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:18.818+0000 7feacac00700 1 -- 172.21.15.28:0/693856862 --> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) -- 0x7feac4062760 con 0x7feaa80a2c20 2024-05-12T22:08:18.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:18.820+0000 7feac0c00700 1 -- 172.21.15.28:0/693856862 <== mgr.14152 v2:172.21.15.28:6800/3123328651 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7feac4062760 con 0x7feaa80a2c20 2024-05-12T22:08:18.821 INFO:teuthology.orchestra.run.smithi028.stdout:Backend: cephadm 2024-05-12T22:08:18.821 INFO:teuthology.orchestra.run.smithi028.stdout:Available: Yes 2024-05-12T22:08:18.821 INFO:teuthology.orchestra.run.smithi028.stdout:Paused: No 2024-05-12T22:08:18.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:18.823+0000 7feacac00700 1 -- 172.21.15.28:0/693856862 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7feaa80a2c20 msgr2=0x7feaa80a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:18.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:18.823+0000 7feacac00700 1 --2- 172.21.15.28:0/693856862 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7feaa80a2c20 0x7feaa80a50e0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7feab400b7b0 tx=0x7feab400b8a0 comp rx=0 tx=0).stop 2024-05-12T22:08:18.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:18.823+0000 7feacac00700 1 -- 172.21.15.28:0/693856862 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feac4108070 msgr2=0x7feac419fba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:18.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:18.823+0000 7feacac00700 1 --2- 172.21.15.28:0/693856862 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feac4108070 0x7feac419fba0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7feabc00c3c0 tx=0x7feabc00c6d0 comp rx=0 tx=0).stop 2024-05-12T22:08:18.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:18.823+0000 7feacac00700 1 -- 172.21.15.28:0/693856862 shutdown_connections 2024-05-12T22:08:18.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:18.823+0000 7feacac00700 1 --2- 172.21.15.28:0/693856862 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7feaa80a2c20 0x7feaa80a50e0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:18.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:18.823+0000 7feacac00700 1 --2- 172.21.15.28:0/693856862 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feac4108070 0x7feac419fba0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:18.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:18.823+0000 7feacac00700 1 --2- 172.21.15.28:0/693856862 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feac4102e10 0x7feac41985e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:18.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:18.823+0000 7feacac00700 1 --2- 172.21.15.28:0/693856862 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feac4105740 0x7feac4198b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:18.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:18.823+0000 7feacac00700 1 -- 172.21.15.28:0/693856862 >> 172.21.15.28:0/693856862 conn(0x7feac40fc820 msgr2=0x7feac4103890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:08:18.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:18.823+0000 7feacac00700 1 -- 172.21.15.28:0/693856862 shutdown_connections 2024-05-12T22:08:18.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:18.823+0000 7feacac00700 1 -- 172.21.15.28:0/693856862 wait complete. 2024-05-12T22:08:19.177 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:18 smithi028 ceph-mon[29847]: pgmap v364: 97 pgs: 97 active+clean; 641 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-12T22:08:19.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:18 smithi086 ceph-mon[34297]: pgmap v364: 97 pgs: 97 active+clean; 641 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-12T22:08:19.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:18 smithi186 ceph-mon[34785]: pgmap v364: 97 pgs: 97 active+clean; 641 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-12T22:08:19.551 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f72fecff68e1d400c4568684327c900485c20d6a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6b92e7b4-10aa-11ef-bc98-c7b262605968 -- bash -c 'ceph orch ps' 2024-05-12T22:08:19.999 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/mon.a/config 2024-05-12T22:08:20.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:19 smithi086 ceph-mon[34297]: from='client.25012 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-12T22:08:20.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:19 smithi028 ceph-mon[29847]: from='client.25012 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-12T22:08:20.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:19 smithi186 ceph-mon[34785]: from='client.25012 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-12T22:08:21.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:20 smithi086 ceph-mon[34297]: pgmap v365: 97 pgs: 97 active+clean; 641 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-12T22:08:21.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:20 smithi028 ceph-mon[29847]: pgmap v365: 97 pgs: 97 active+clean; 641 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-12T22:08:21.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:20 smithi186 ceph-mon[34785]: pgmap v365: 97 pgs: 97 active+clean; 641 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-12T22:08:22.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:22.307+0000 7f2997200700 1 -- 172.21.15.28:0/2405291542 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2998101e40 msgr2=0x7f29981022c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:22.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:22.307+0000 7f2997200700 1 --2- 172.21.15.28:0/2405291542 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2998101e40 0x7f29981022c0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f299000b0a0 tx=0x7f299000b3b0 comp rx=0 tx=0).stop 2024-05-12T22:08:22.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:22.307+0000 7f2997200700 1 -- 172.21.15.28:0/2405291542 shutdown_connections 2024-05-12T22:08:22.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:22.307+0000 7f2997200700 1 --2- 172.21.15.28:0/2405291542 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2998100b20 0x7f2998100f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:22.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:22.307+0000 7f2997200700 1 --2- 172.21.15.28:0/2405291542 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2998101e40 0x7f29981022c0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:22.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:22.307+0000 7f2997200700 1 --2- 172.21.15.28:0/2405291542 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2998101480 0x7f2998101900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:22.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:22.307+0000 7f2997200700 1 -- 172.21.15.28:0/2405291542 >> 172.21.15.28:0/2405291542 conn(0x7f29980fc740 msgr2=0x7f29980feb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:08:22.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:22.307+0000 7f2997200700 1 -- 172.21.15.28:0/2405291542 shutdown_connections 2024-05-12T22:08:22.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:22.307+0000 7f2997200700 1 -- 172.21.15.28:0/2405291542 wait complete. 2024-05-12T22:08:22.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:22.309+0000 7f2997200700 1 Processor -- start 2024-05-12T22:08:22.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:22.309+0000 7f2997200700 1 -- start start 2024-05-12T22:08:22.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:22.310+0000 7f2997200700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2998100b20 0x7f299819a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:22.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:22.310+0000 7f2997200700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2998101480 0x7f299819ab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:22.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:22.310+0000 7f2997200700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2998101e40 0x7f29981a1bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:22.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:22.310+0000 7f2997200700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f29981100b0 con 0x7f2998100b20 2024-05-12T22:08:22.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:22.310+0000 7f2997200700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f299810ff30 con 0x7f2998101480 2024-05-12T22:08:22.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:22.310+0000 7f2997200700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f2998110230 con 0x7f2998101e40 2024-05-12T22:08:22.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:22.310+0000 7f2995e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2998100b20 0x7f299819a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:22.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:22.310+0000 7f2996800700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2998101e40 0x7f29981a1bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:22.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:22.310+0000 7f2995e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2998100b20 0x7f299819a600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48258/0 (socket says 172.21.15.28:48258) 2024-05-12T22:08:22.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:22.310+0000 7f2995e00700 1 -- 172.21.15.28:0/2741608710 learned_addr learned my addr 172.21.15.28:0/2741608710 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:08:22.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:22.310+0000 7f2995400700 1 --2- 172.21.15.28:0/2741608710 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2998101480 0x7f299819ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:22.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:22.310+0000 7f2996800700 1 -- 172.21.15.28:0/2741608710 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2998101480 msgr2=0x7f299819ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:22.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:22.310+0000 7f2996800700 1 --2- 172.21.15.28:0/2741608710 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2998101480 0x7f299819ab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:22.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:22.310+0000 7f2996800700 1 -- 172.21.15.28:0/2741608710 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2998100b20 msgr2=0x7f299819a600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:22.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:22.310+0000 7f2996800700 1 --2- 172.21.15.28:0/2741608710 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2998100b20 0x7f299819a600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:22.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:22.310+0000 7f2996800700 1 -- 172.21.15.28:0/2741608710 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29981a2260 con 0x7f2998101e40 2024-05-12T22:08:22.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:22.311+0000 7f2996800700 1 --2- 172.21.15.28:0/2741608710 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2998101e40 0x7f29981a1bc0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f299000b070 tx=0x7f2990004c60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:08:22.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:22.311+0000 7f298ea00700 1 -- 172.21.15.28:0/2741608710 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2990021070 con 0x7f2998101e40 2024-05-12T22:08:22.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:22.311+0000 7f2997200700 1 -- 172.21.15.28:0/2741608710 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29981a24f0 con 0x7f2998101e40 2024-05-12T22:08:22.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:22.311+0000 7f298ea00700 1 -- 172.21.15.28:0/2741608710 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2990004e50 con 0x7f2998101e40 2024-05-12T22:08:22.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:22.311+0000 7f298ea00700 1 -- 172.21.15.28:0/2741608710 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2990004780 con 0x7f2998101e40 2024-05-12T22:08:22.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:22.311+0000 7f2997200700 1 -- 172.21.15.28:0/2741608710 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29981a29f0 con 0x7f2998101e40 2024-05-12T22:08:22.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:22.312+0000 7f2997200700 1 -- 172.21.15.28:0/2741608710 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2998069010 con 0x7f2998101e40 2024-05-12T22:08:22.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:22.315+0000 7f298ea00700 1 -- 172.21.15.28:0/2741608710 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f2990003a00 con 0x7f2998101e40 2024-05-12T22:08:22.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:22.316+0000 7f298ea00700 1 --2- 172.21.15.28:0/2741608710 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f297c0a2c20 0x7f297c0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:22.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:22.316+0000 7f298ea00700 1 -- 172.21.15.28:0/2741608710 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f299000f020 con 0x7f2998101e40 2024-05-12T22:08:22.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:22.316+0000 7f2995e00700 1 --2- 172.21.15.28:0/2741608710 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f297c0a2c20 0x7f297c0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:22.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:22.317+0000 7f2995e00700 1 --2- 172.21.15.28:0/2741608710 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f297c0a2c20 0x7f297c0a50e0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f2988009d20 tx=0x7f2988009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:08:22.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:22.318+0000 7f298ea00700 1 -- 172.21.15.28:0/2741608710 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f299002a020 con 0x7f2998101e40 2024-05-12T22:08:22.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:22.556+0000 7f2997200700 1 -- 172.21.15.28:0/2741608710 --> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7f299802df30 con 0x7f297c0a2c20 2024-05-12T22:08:22.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:22.599+0000 7f298ea00700 1 -- 172.21.15.28:0/2741608710 <== mgr.14152 v2:172.21.15.28:6800/3123328651 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+4185 (secure 0 0 0) 0x7f299802df30 con 0x7f297c0a2c20 2024-05-12T22:08:22.601 INFO:teuthology.orchestra.run.smithi028.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-05-12T22:08:22.601 INFO:teuthology.orchestra.run.smithi028.stdout:mds.a smithi028 running (118s) 21s ago 117s 17.3M - 19.0.0-2555-gf72fecff 8252b7944d3b 8d25c5440080 2024-05-12T22:08:22.601 INFO:teuthology.orchestra.run.smithi028.stdout:mds.b smithi086 running (115s) 46s ago 115s 28.2M - 19.0.0-2555-gf72fecff 8252b7944d3b c6618c6944eb 2024-05-12T22:08:22.601 INFO:teuthology.orchestra.run.smithi028.stdout:mds.c smithi186 running (2m) 46s ago 2m 19.2M - 19.0.0-2555-gf72fecff 8252b7944d3b bfbd5c1aad99 2024-05-12T22:08:22.601 INFO:teuthology.orchestra.run.smithi028.stdout:mds.d smithi028 running (2m) 21s ago 2m 12.7M - 19.0.0-2555-gf72fecff 8252b7944d3b a971d49e34e4 2024-05-12T22:08:22.601 INFO:teuthology.orchestra.run.smithi028.stdout:mds.e smithi086 running (2m) 46s ago 2m 13.9M - 19.0.0-2555-gf72fecff 8252b7944d3b d1d854c7ed0b 2024-05-12T22:08:22.601 INFO:teuthology.orchestra.run.smithi028.stdout:mds.f smithi186 running (2m) 46s ago 2m 15.3M - 19.0.0-2555-gf72fecff 8252b7944d3b 24c6a9b76bab 2024-05-12T22:08:22.601 INFO:teuthology.orchestra.run.smithi028.stdout:mds.g smithi028 running (2m) 21s ago 2m 17.4M - 19.0.0-2555-gf72fecff 8252b7944d3b 82c3e6745e1f 2024-05-12T22:08:22.601 INFO:teuthology.orchestra.run.smithi028.stdout:mds.h smithi086 running (2m) 46s ago 2m 17.9M - 19.0.0-2555-gf72fecff 8252b7944d3b 8e70c957f69d 2024-05-12T22:08:22.601 INFO:teuthology.orchestra.run.smithi028.stdout:mds.i smithi186 running (2m) 46s ago 2m 17.0M - 19.0.0-2555-gf72fecff 8252b7944d3b ce0d3c9e0bae 2024-05-12T22:08:22.601 INFO:teuthology.orchestra.run.smithi028.stdout:mds.j smithi028 running (2m) 21s ago 2m 12.6M - 19.0.0-2555-gf72fecff 8252b7944d3b 78eeedeae0c5 2024-05-12T22:08:22.601 INFO:teuthology.orchestra.run.smithi028.stdout:mds.k smithi086 running (2m) 46s ago 2m 14.0M - 19.0.0-2555-gf72fecff 8252b7944d3b 5c29b9521792 2024-05-12T22:08:22.602 INFO:teuthology.orchestra.run.smithi028.stdout:mds.l smithi186 running (2m) 46s ago 2m 14.1M - 19.0.0-2555-gf72fecff 8252b7944d3b c639841ebac2 2024-05-12T22:08:22.602 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.x smithi028 *:9283,8765 running (10m) 21s ago 10m 516M - 19.0.0-2555-gf72fecff 8252b7944d3b 1cf9bf897773 2024-05-12T22:08:22.602 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.y smithi086 *:8443,8765 running (8m) 46s ago 8m 454M - 19.0.0-2555-gf72fecff 8252b7944d3b 529d155079c6 2024-05-12T22:08:22.602 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.z smithi186 *:8443,8765 running (8m) 46s ago 8m 456M - 19.0.0-2555-gf72fecff 8252b7944d3b 893dc82d49b9 2024-05-12T22:08:22.602 INFO:teuthology.orchestra.run.smithi028.stdout:mon.a smithi028 running (10m) 21s ago 10m 46.4M 2048M 19.0.0-2555-gf72fecff 8252b7944d3b d414f473c699 2024-05-12T22:08:22.602 INFO:teuthology.orchestra.run.smithi028.stdout:mon.b smithi086 running (8m) 46s ago 8m 41.0M 2048M 19.0.0-2555-gf72fecff 8252b7944d3b d12e520b6c5c 2024-05-12T22:08:22.602 INFO:teuthology.orchestra.run.smithi028.stdout:mon.c smithi186 running (8m) 46s ago 8m 41.4M 2048M 19.0.0-2555-gf72fecff 8252b7944d3b 5a2c2778badf 2024-05-12T22:08:22.602 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0 smithi028 running (7m) 21s ago 7m 51.1M 4273M 19.0.0-2555-gf72fecff 8252b7944d3b f675be533a36 2024-05-12T22:08:22.602 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1 smithi086 running (7m) 46s ago 7m 56.1M 4272M 19.0.0-2555-gf72fecff 8252b7944d3b 801b9a5181ff 2024-05-12T22:08:22.602 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2 smithi186 running (6m) 46s ago 6m 52.7M 4272M 19.0.0-2555-gf72fecff 8252b7944d3b 20fe3c8a9449 2024-05-12T22:08:22.602 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3 smithi028 running (6m) 21s ago 6m 46.3M 4273M 19.0.0-2555-gf72fecff 8252b7944d3b 613a90bed9f6 2024-05-12T22:08:22.602 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4 smithi086 running (6m) 46s ago 6m 44.1M 4272M 19.0.0-2555-gf72fecff 8252b7944d3b db131f4bd6a1 2024-05-12T22:08:22.602 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5 smithi186 running (5m) 46s ago 5m 45.8M 4272M 19.0.0-2555-gf72fecff 8252b7944d3b e3ead61e84df 2024-05-12T22:08:22.602 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6 smithi028 running (5m) 21s ago 5m 45.1M 4273M 19.0.0-2555-gf72fecff 8252b7944d3b be82f07b03d7 2024-05-12T22:08:22.603 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7 smithi086 running (4m) 46s ago 4m 46.5M 4272M 19.0.0-2555-gf72fecff 8252b7944d3b 455895dd3a6c 2024-05-12T22:08:22.603 INFO:teuthology.orchestra.run.smithi028.stdout:osd.8 smithi186 running (4m) 46s ago 4m 52.0M 4272M 19.0.0-2555-gf72fecff 8252b7944d3b fa294356d5fb 2024-05-12T22:08:22.603 INFO:teuthology.orchestra.run.smithi028.stdout:osd.9 smithi028 running (3m) 21s ago 3m 45.0M 4273M 19.0.0-2555-gf72fecff 8252b7944d3b d4a26aca41c7 2024-05-12T22:08:22.603 INFO:teuthology.orchestra.run.smithi028.stdout:osd.10 smithi086 running (3m) 46s ago 3m 47.4M 4272M 19.0.0-2555-gf72fecff 8252b7944d3b 989788f62f1c 2024-05-12T22:08:22.603 INFO:teuthology.orchestra.run.smithi028.stdout:osd.11 smithi186 running (2m) 46s ago 2m 46.7M 4272M 19.0.0-2555-gf72fecff 8252b7944d3b 8f5f3e5c471e 2024-05-12T22:08:22.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:22.603+0000 7f2997200700 1 -- 172.21.15.28:0/2741608710 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f297c0a2c20 msgr2=0x7f297c0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:22.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:22.603+0000 7f2997200700 1 --2- 172.21.15.28:0/2741608710 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f297c0a2c20 0x7f297c0a50e0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f2988009d20 tx=0x7f2988009260 comp rx=0 tx=0).stop 2024-05-12T22:08:22.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:22.603+0000 7f2997200700 1 -- 172.21.15.28:0/2741608710 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2998101e40 msgr2=0x7f29981a1bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:22.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:22.603+0000 7f2997200700 1 --2- 172.21.15.28:0/2741608710 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2998101e40 0x7f29981a1bc0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f299000b070 tx=0x7f2990004c60 comp rx=0 tx=0).stop 2024-05-12T22:08:22.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:22.603+0000 7f2997200700 1 -- 172.21.15.28:0/2741608710 shutdown_connections 2024-05-12T22:08:22.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:22.603+0000 7f2997200700 1 --2- 172.21.15.28:0/2741608710 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f297c0a2c20 0x7f297c0a50e0 secure :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f2988009d20 tx=0x7f2988009260 comp rx=0 tx=0).stop 2024-05-12T22:08:22.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:22.603+0000 7f2997200700 1 --2- 172.21.15.28:0/2741608710 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2998101e40 0x7f29981a1bc0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:22.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:22.603+0000 7f2997200700 1 --2- 172.21.15.28:0/2741608710 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2998100b20 0x7f299819a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:22.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:22.603+0000 7f2997200700 1 --2- 172.21.15.28:0/2741608710 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2998101480 0x7f299819ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:22.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:22.603+0000 7f2997200700 1 -- 172.21.15.28:0/2741608710 >> 172.21.15.28:0/2741608710 conn(0x7f29980fc740 msgr2=0x7f29980fe240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:08:22.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:22.603+0000 7f2997200700 1 -- 172.21.15.28:0/2741608710 shutdown_connections 2024-05-12T22:08:22.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:22.603+0000 7f2997200700 1 -- 172.21.15.28:0/2741608710 wait complete. 2024-05-12T22:08:23.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:22 smithi086 ceph-mon[34297]: pgmap v366: 97 pgs: 97 active+clean; 641 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:08:23.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:22 smithi028 ceph-mon[29847]: pgmap v366: 97 pgs: 97 active+clean; 641 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:08:23.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:22 smithi186 ceph-mon[34785]: pgmap v366: 97 pgs: 97 active+clean; 641 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:08:23.566 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f72fecff68e1d400c4568684327c900485c20d6a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6b92e7b4-10aa-11ef-bc98-c7b262605968 -- bash -c 'ceph orch ls' 2024-05-12T22:08:24.035 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/mon.a/config 2024-05-12T22:08:24.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:23 smithi086 ceph-mon[34297]: from='client.25018 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-12T22:08:24.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:23 smithi028 ceph-mon[29847]: from='client.25018 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-12T22:08:24.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:23 smithi186 ceph-mon[34785]: from='client.25018 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-12T22:08:25.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:24 smithi086 ceph-mon[34297]: pgmap v367: 97 pgs: 97 active+clean; 641 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-12T22:08:25.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:24 smithi028 ceph-mon[29847]: pgmap v367: 97 pgs: 97 active+clean; 641 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-12T22:08:25.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:24 smithi186 ceph-mon[34785]: pgmap v367: 97 pgs: 97 active+clean; 641 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-12T22:08:26.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:26.254+0000 7fcec3e00700 1 -- 172.21.15.28:0/2973342747 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcec4102d40 msgr2=0x7fcec4105130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:26.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:26.254+0000 7fcec3e00700 1 --2- 172.21.15.28:0/2973342747 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcec4102d40 0x7fcec4105130 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fceb4009990 tx=0x7fceb4009ca0 comp rx=0 tx=0).stop 2024-05-12T22:08:26.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:26.254+0000 7fcec3e00700 1 -- 172.21.15.28:0/2973342747 shutdown_connections 2024-05-12T22:08:26.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:26.254+0000 7fcec3e00700 1 --2- 172.21.15.28:0/2973342747 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcec4102d40 0x7fcec4105130 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:26.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:26.254+0000 7fcec3e00700 1 --2- 172.21.15.28:0/2973342747 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcec4107fa0 0x7fcec410a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:26.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:26.254+0000 7fcec3e00700 1 --2- 172.21.15.28:0/2973342747 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcec4105670 0x7fcec4107a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:26.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:26.254+0000 7fcec3e00700 1 -- 172.21.15.28:0/2973342747 >> 172.21.15.28:0/2973342747 conn(0x7fcec40fc750 msgr2=0x7fcec40feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:08:26.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:26.255+0000 7fcec3e00700 1 -- 172.21.15.28:0/2973342747 shutdown_connections 2024-05-12T22:08:26.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:26.255+0000 7fcec3e00700 1 -- 172.21.15.28:0/2973342747 wait complete. 2024-05-12T22:08:26.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:26.256+0000 7fcec3e00700 1 Processor -- start 2024-05-12T22:08:26.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:26.256+0000 7fcec3e00700 1 -- start start 2024-05-12T22:08:26.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:26.256+0000 7fcec3e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcec4102d40 0x7fcec41026a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:26.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:26.256+0000 7fcec3e00700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcec4105670 0x7fcec4100cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:26.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:26.256+0000 7fcec3e00700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcec4107fa0 0x7fcec4101230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:26.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:26.256+0000 7fcec3e00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcec410ded0 con 0x7fcec4102d40 2024-05-12T22:08:26.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:26.256+0000 7fcec3e00700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fcec410dd50 con 0x7fcec4107fa0 2024-05-12T22:08:26.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:26.256+0000 7fcec3e00700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fcec410e050 con 0x7fcec4105670 2024-05-12T22:08:26.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:26.257+0000 7fcec2a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcec4102d40 0x7fcec41026a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:26.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:26.257+0000 7fcec2a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcec4102d40 0x7fcec41026a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40594/0 (socket says 172.21.15.28:40594) 2024-05-12T22:08:26.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:26.257+0000 7fcec3400700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcec4107fa0 0x7fcec4101230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:26.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:26.257+0000 7fcebbe00700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcec4105670 0x7fcec4100cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:26.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:26.257+0000 7fcec2a00700 1 -- 172.21.15.28:0/164033824 learned_addr learned my addr 172.21.15.28:0/164033824 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:08:26.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:26.257+0000 7fcec3400700 1 -- 172.21.15.28:0/164033824 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcec4105670 msgr2=0x7fcec4100cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:26.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:26.257+0000 7fcec3400700 1 --2- 172.21.15.28:0/164033824 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcec4105670 0x7fcec4100cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:26.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:26.257+0000 7fcec3400700 1 -- 172.21.15.28:0/164033824 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcec4102d40 msgr2=0x7fcec41026a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:26.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:26.257+0000 7fcec3400700 1 --2- 172.21.15.28:0/164033824 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcec4102d40 0x7fcec41026a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:26.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:26.257+0000 7fcec3400700 1 -- 172.21.15.28:0/164033824 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcec4101ae0 con 0x7fcec4107fa0 2024-05-12T22:08:26.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:26.257+0000 7fcec3400700 1 --2- 172.21.15.28:0/164033824 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcec4107fa0 0x7fcec4101230 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fcebc00ef90 tx=0x7fcebc00c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:08:26.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:26.258+0000 7fcebb400700 1 -- 172.21.15.28:0/164033824 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcebc0092e0 con 0x7fcec4107fa0 2024-05-12T22:08:26.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:26.258+0000 7fcebb400700 1 -- 172.21.15.28:0/164033824 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcebc004bb0 con 0x7fcec4107fa0 2024-05-12T22:08:26.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:26.258+0000 7fcec3e00700 1 -- 172.21.15.28:0/164033824 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcec41a25e0 con 0x7fcec4107fa0 2024-05-12T22:08:26.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:26.258+0000 7fcebb400700 1 -- 172.21.15.28:0/164033824 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcebc004280 con 0x7fcec4107fa0 2024-05-12T22:08:26.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:26.258+0000 7fcec3e00700 1 -- 172.21.15.28:0/164033824 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcec41a2aa0 con 0x7fcec4107fa0 2024-05-12T22:08:26.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:26.259+0000 7fcec3e00700 1 -- 172.21.15.28:0/164033824 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcec4069010 con 0x7fcec4107fa0 2024-05-12T22:08:26.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:26.261+0000 7fcebb400700 1 -- 172.21.15.28:0/164033824 <== mon.2 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fcebc01b030 con 0x7fcec4107fa0 2024-05-12T22:08:26.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:26.262+0000 7fcebb400700 1 --2- 172.21.15.28:0/164033824 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fcea80a2c20 0x7fcea80a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:26.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:26.262+0000 7fcebb400700 1 -- 172.21.15.28:0/164033824 <== mon.2 v2:172.21.15.86:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7fcebc0cecc0 con 0x7fcec4107fa0 2024-05-12T22:08:26.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:26.263+0000 7fcec2a00700 1 --2- 172.21.15.28:0/164033824 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fcea80a2c20 0x7fcea80a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:26.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:26.263+0000 7fcec2a00700 1 --2- 172.21.15.28:0/164033824 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fcea80a2c20 0x7fcea80a50e0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fceb400b2e0 tx=0x7fceb400b880 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:08:26.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:26.267+0000 7fcebb400700 1 -- 172.21.15.28:0/164033824 <== mon.2 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fcebc0d5080 con 0x7fcec4107fa0 2024-05-12T22:08:26.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:26.506+0000 7fcec3e00700 1 -- 172.21.15.28:0/164033824 --> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) -- 0x7fcec4062760 con 0x7fcea80a2c20 2024-05-12T22:08:26.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:26.516+0000 7fcebb400700 1 -- 172.21.15.28:0/164033824 <== mgr.14152 v2:172.21.15.28:6800/3123328651 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+980 (secure 0 0 0) 0x7fcec4062760 con 0x7fcea80a2c20 2024-05-12T22:08:26.517 INFO:teuthology.orchestra.run.smithi028.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-05-12T22:08:26.517 INFO:teuthology.orchestra.run.smithi028.stdout:mds.all 12/12 50s ago 2m smithi028=a;smithi028=d;smithi028=g;smithi028=j;smithi086=b;smithi086=e;smithi086=h;smithi086=k;smithi186=c;smithi186=f;smithi186=i;smithi186=l;count:12 2024-05-12T22:08:26.517 INFO:teuthology.orchestra.run.smithi028.stdout:mgr 3/3 50s ago 8m smithi028=x;smithi086=y;smithi186=z;count:3 2024-05-12T22:08:26.517 INFO:teuthology.orchestra.run.smithi028.stdout:mon 3/3 50s ago 8m smithi028:172.21.15.28=a;smithi086:172.21.15.86=b;smithi186:172.21.15.186=c;count:3 2024-05-12T22:08:26.517 INFO:teuthology.orchestra.run.smithi028.stdout:osd 12 50s ago - 2024-05-12T22:08:26.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:26.519+0000 7fcec3e00700 1 -- 172.21.15.28:0/164033824 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fcea80a2c20 msgr2=0x7fcea80a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:26.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:26.519+0000 7fcec3e00700 1 --2- 172.21.15.28:0/164033824 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fcea80a2c20 0x7fcea80a50e0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fceb400b2e0 tx=0x7fceb400b880 comp rx=0 tx=0).stop 2024-05-12T22:08:26.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:26.519+0000 7fcec3e00700 1 -- 172.21.15.28:0/164033824 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcec4107fa0 msgr2=0x7fcec4101230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:26.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:26.519+0000 7fcec3e00700 1 --2- 172.21.15.28:0/164033824 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcec4107fa0 0x7fcec4101230 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fcebc00ef90 tx=0x7fcebc00c3a0 comp rx=0 tx=0).stop 2024-05-12T22:08:26.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:26.519+0000 7fcec3e00700 1 -- 172.21.15.28:0/164033824 shutdown_connections 2024-05-12T22:08:26.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:26.519+0000 7fcec3e00700 1 --2- 172.21.15.28:0/164033824 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fcea80a2c20 0x7fcea80a50e0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:26.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:26.519+0000 7fcec3e00700 1 --2- 172.21.15.28:0/164033824 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcec4107fa0 0x7fcec4101230 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:26.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:26.519+0000 7fcec3e00700 1 --2- 172.21.15.28:0/164033824 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcec4102d40 0x7fcec41026a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:26.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:26.519+0000 7fcec3e00700 1 --2- 172.21.15.28:0/164033824 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcec4105670 0x7fcec4100cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:26.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:26.519+0000 7fcec3e00700 1 -- 172.21.15.28:0/164033824 >> 172.21.15.28:0/164033824 conn(0x7fcec40fc750 msgr2=0x7fcec4103600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:08:26.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:26.520+0000 7fcec3e00700 1 -- 172.21.15.28:0/164033824 shutdown_connections 2024-05-12T22:08:26.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:26.520+0000 7fcec3e00700 1 -- 172.21.15.28:0/164033824 wait complete. 2024-05-12T22:08:27.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:26 smithi086 ceph-mon[34297]: pgmap v368: 97 pgs: 97 active+clean; 641 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-12T22:08:27.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:26 smithi028 ceph-mon[29847]: pgmap v368: 97 pgs: 97 active+clean; 641 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-12T22:08:27.412 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f72fecff68e1d400c4568684327c900485c20d6a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6b92e7b4-10aa-11ef-bc98-c7b262605968 -- bash -c 'ceph orch host ls' 2024-05-12T22:08:27.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:26 smithi186 ceph-mon[34785]: pgmap v368: 97 pgs: 97 active+clean; 641 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-12T22:08:27.896 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/mon.a/config 2024-05-12T22:08:28.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:27 smithi086 ceph-mon[34297]: from='client.25046 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-12T22:08:28.266 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:27 smithi028 ceph-mon[29847]: from='client.25046 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-12T22:08:28.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:27 smithi186 ceph-mon[34785]: from='client.25046 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-12T22:08:29.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:28 smithi086 ceph-mon[34297]: pgmap v369: 97 pgs: 97 active+clean; 641 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-12T22:08:29.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:28 smithi028 ceph-mon[29847]: pgmap v369: 97 pgs: 97 active+clean; 641 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-12T22:08:29.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:28 smithi186 ceph-mon[34785]: pgmap v369: 97 pgs: 97 active+clean; 641 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-12T22:08:29.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:29.942+0000 7f8b09e00700 1 -- 172.21.15.28:0/811179468 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8b041089c0 msgr2=0x7f8b04108de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:29.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:29.942+0000 7f8b09e00700 1 --2- 172.21.15.28:0/811179468 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8b041089c0 0x7f8b04108de0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f8af40099c0 tx=0x7f8af4009cd0 comp rx=0 tx=0).stop 2024-05-12T22:08:29.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:29.942+0000 7f8b09e00700 1 -- 172.21.15.28:0/811179468 shutdown_connections 2024-05-12T22:08:29.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:29.942+0000 7f8b09e00700 1 --2- 172.21.15.28:0/811179468 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8b0410a560 0x7f8b04110e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:29.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:29.942+0000 7f8b09e00700 1 --2- 172.21.15.28:0/811179468 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8b041089c0 0x7f8b04108de0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:29.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:29.942+0000 7f8b09e00700 1 --2- 172.21.15.28:0/811179468 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b04109bc0 0x7f8b0410a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:29.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:29.942+0000 7f8b09e00700 1 -- 172.21.15.28:0/811179468 >> 172.21.15.28:0/811179468 conn(0x7f8b04077990 msgr2=0x7f8b04079dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:08:29.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:29.943+0000 7f8b09e00700 1 -- 172.21.15.28:0/811179468 shutdown_connections 2024-05-12T22:08:29.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:29.943+0000 7f8b09e00700 1 -- 172.21.15.28:0/811179468 wait complete. 2024-05-12T22:08:29.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:29.944+0000 7f8b09e00700 1 Processor -- start 2024-05-12T22:08:29.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:29.944+0000 7f8b09e00700 1 -- start start 2024-05-12T22:08:29.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:29.944+0000 7f8b09e00700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8b041089c0 0x7f8b0419eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:29.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:29.944+0000 7f8b09e00700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8b04109bc0 0x7f8b0419f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:29.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:29.945+0000 7f8b09e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b0410a560 0x7f8b041a60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:29.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:29.945+0000 7f8b09e00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8b041145a0 con 0x7f8b0410a560 2024-05-12T22:08:29.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:29.945+0000 7f8b09e00700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f8b04114420 con 0x7f8b04109bc0 2024-05-12T22:08:29.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:29.945+0000 7f8b09e00700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f8b04114720 con 0x7f8b041089c0 2024-05-12T22:08:29.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:29.945+0000 7f8b02a00700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8b041089c0 0x7f8b0419eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:29.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:29.945+0000 7f8b02000700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8b04109bc0 0x7f8b0419f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:29.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:29.945+0000 7f8b03400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b0410a560 0x7f8b041a60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:29.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:29.945+0000 7f8b02a00700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8b041089c0 0x7f8b0419eb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:41772/0 (socket says 172.21.15.28:41772) 2024-05-12T22:08:29.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:29.945+0000 7f8b02a00700 1 -- 172.21.15.28:0/4218682454 learned_addr learned my addr 172.21.15.28:0/4218682454 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:08:29.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:29.945+0000 7f8b02000700 1 -- 172.21.15.28:0/4218682454 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8b041089c0 msgr2=0x7f8b0419eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:29.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:29.945+0000 7f8b02000700 1 --2- 172.21.15.28:0/4218682454 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8b041089c0 0x7f8b0419eb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:29.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:29.945+0000 7f8b02000700 1 -- 172.21.15.28:0/4218682454 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b0410a560 msgr2=0x7f8b041a60e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:29.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:29.945+0000 7f8b02000700 1 --2- 172.21.15.28:0/4218682454 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b0410a560 0x7f8b041a60e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:29.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:29.945+0000 7f8b02000700 1 -- 172.21.15.28:0/4218682454 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b041a6780 con 0x7f8b04109bc0 2024-05-12T22:08:29.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:29.945+0000 7f8b03400700 1 --2- 172.21.15.28:0/4218682454 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b0410a560 0x7f8b041a60e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:08:29.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:29.946+0000 7f8b02000700 1 --2- 172.21.15.28:0/4218682454 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8b04109bc0 0x7f8b0419f060 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f8af000e9f0 tx=0x7f8af000ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:08:29.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:29.946+0000 7f8b08a00700 1 -- 172.21.15.28:0/4218682454 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8af000c9c0 con 0x7f8b04109bc0 2024-05-12T22:08:29.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:29.946+0000 7f8b08a00700 1 -- 172.21.15.28:0/4218682454 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8af000cb60 con 0x7f8b04109bc0 2024-05-12T22:08:29.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:29.946+0000 7f8b08a00700 1 -- 172.21.15.28:0/4218682454 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8af0018680 con 0x7f8b04109bc0 2024-05-12T22:08:29.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:29.946+0000 7f8b09e00700 1 -- 172.21.15.28:0/4218682454 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b041a6a70 con 0x7f8b04109bc0 2024-05-12T22:08:29.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:29.946+0000 7f8b09e00700 1 -- 172.21.15.28:0/4218682454 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b041a7000 con 0x7f8b04109bc0 2024-05-12T22:08:29.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:29.947+0000 7f8b09e00700 1 -- 172.21.15.28:0/4218682454 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b04069010 con 0x7f8b04109bc0 2024-05-12T22:08:29.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:29.950+0000 7f8b08a00700 1 -- 172.21.15.28:0/4218682454 <== mon.2 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f8af0010430 con 0x7f8b04109bc0 2024-05-12T22:08:29.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:29.951+0000 7f8b08a00700 1 --2- 172.21.15.28:0/4218682454 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f8ae80a2ca0 0x7f8ae80a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:29.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:29.951+0000 7f8b08a00700 1 -- 172.21.15.28:0/4218682454 <== mon.2 v2:172.21.15.86:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f8af00cf070 con 0x7f8b04109bc0 2024-05-12T22:08:29.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:29.951+0000 7f8b02a00700 1 --2- 172.21.15.28:0/4218682454 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f8ae80a2ca0 0x7f8ae80a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:29.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:29.952+0000 7f8b02a00700 1 --2- 172.21.15.28:0/4218682454 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f8ae80a2ca0 0x7f8ae80a5160 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f8af4000c00 tx=0x7f8af4005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:08:29.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:29.956+0000 7f8b08a00700 1 -- 172.21.15.28:0/4218682454 <== mon.2 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f8af009bfe0 con 0x7f8b04109bc0 2024-05-12T22:08:30.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:30.193+0000 7f8b09e00700 1 -- 172.21.15.28:0/4218682454 --> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) -- 0x7f8b04062760 con 0x7f8ae80a2ca0 2024-05-12T22:08:30.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:30.197+0000 7f8b08a00700 1 -- 172.21.15.28:0/4218682454 <== mgr.14152 v2:172.21.15.28:6800/3123328651 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+191 (secure 0 0 0) 0x7f8b04062760 con 0x7f8ae80a2ca0 2024-05-12T22:08:30.198 INFO:teuthology.orchestra.run.smithi028.stdout:HOST ADDR LABELS STATUS 2024-05-12T22:08:30.198 INFO:teuthology.orchestra.run.smithi028.stdout:smithi028 172.21.15.28 2024-05-12T22:08:30.199 INFO:teuthology.orchestra.run.smithi028.stdout:smithi086 172.21.15.86 2024-05-12T22:08:30.199 INFO:teuthology.orchestra.run.smithi028.stdout:smithi186 172.21.15.186 2024-05-12T22:08:30.199 INFO:teuthology.orchestra.run.smithi028.stdout:3 hosts in cluster 2024-05-12T22:08:30.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:30.200+0000 7f8b09e00700 1 -- 172.21.15.28:0/4218682454 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f8ae80a2ca0 msgr2=0x7f8ae80a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:30.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:30.200+0000 7f8b09e00700 1 --2- 172.21.15.28:0/4218682454 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f8ae80a2ca0 0x7f8ae80a5160 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f8af4000c00 tx=0x7f8af4005fb0 comp rx=0 tx=0).stop 2024-05-12T22:08:30.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:30.200+0000 7f8b09e00700 1 -- 172.21.15.28:0/4218682454 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8b04109bc0 msgr2=0x7f8b0419f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:30.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:30.200+0000 7f8b09e00700 1 --2- 172.21.15.28:0/4218682454 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8b04109bc0 0x7f8b0419f060 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f8af000e9f0 tx=0x7f8af000ed00 comp rx=0 tx=0).stop 2024-05-12T22:08:30.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:30.200+0000 7f8b09e00700 1 -- 172.21.15.28:0/4218682454 shutdown_connections 2024-05-12T22:08:30.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:30.200+0000 7f8b09e00700 1 --2- 172.21.15.28:0/4218682454 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f8ae80a2ca0 0x7f8ae80a5160 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:30.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:30.200+0000 7f8b09e00700 1 --2- 172.21.15.28:0/4218682454 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8b041089c0 0x7f8b0419eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:30.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:30.200+0000 7f8b09e00700 1 --2- 172.21.15.28:0/4218682454 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b0410a560 0x7f8b041a60e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:30.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:30.200+0000 7f8b09e00700 1 --2- 172.21.15.28:0/4218682454 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8b04109bc0 0x7f8b0419f060 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:30.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:30.200+0000 7f8b09e00700 1 -- 172.21.15.28:0/4218682454 >> 172.21.15.28:0/4218682454 conn(0x7f8b04077990 msgr2=0x7f8b0410ce60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:08:30.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:30.201+0000 7f8b09e00700 1 -- 172.21.15.28:0/4218682454 shutdown_connections 2024-05-12T22:08:30.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:30.201+0000 7f8b09e00700 1 -- 172.21.15.28:0/4218682454 wait complete. 2024-05-12T22:08:31.087 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f72fecff68e1d400c4568684327c900485c20d6a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6b92e7b4-10aa-11ef-bc98-c7b262605968 -- bash -c 'ceph orch device ls' 2024-05-12T22:08:31.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:30 smithi028 ceph-mon[29847]: pgmap v370: 97 pgs: 97 active+clean; 641 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-12T22:08:31.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:30 smithi028 ceph-mon[29847]: from='client.25052 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-12T22:08:31.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:30 smithi186 ceph-mon[34785]: pgmap v370: 97 pgs: 97 active+clean; 641 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-12T22:08:31.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:30 smithi186 ceph-mon[34785]: from='client.25052 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-12T22:08:31.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:30 smithi086 ceph-mon[34297]: pgmap v370: 97 pgs: 97 active+clean; 641 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-12T22:08:31.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:30 smithi086 ceph-mon[34297]: from='client.25052 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-12T22:08:31.554 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/mon.a/config 2024-05-12T22:08:33.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:32 smithi028 ceph-mon[29847]: pgmap v371: 97 pgs: 97 active+clean; 641 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:08:33.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:32 smithi186 ceph-mon[34785]: pgmap v371: 97 pgs: 97 active+clean; 641 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:08:33.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:32 smithi086 ceph-mon[34297]: pgmap v371: 97 pgs: 97 active+clean; 641 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:08:33.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:33.680+0000 7fb38d200700 1 -- 172.21.15.28:0/1939745353 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb388106130 msgr2=0x7fb38810c9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:33.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:33.680+0000 7fb38d200700 1 --2- 172.21.15.28:0/1939745353 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb388106130 0x7fb38810c9e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fb380009fc0 tx=0x7fb38000b330 comp rx=0 tx=0).stop 2024-05-12T22:08:33.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:33.680+0000 7fb38d200700 1 -- 172.21.15.28:0/1939745353 shutdown_connections 2024-05-12T22:08:33.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:33.680+0000 7fb38d200700 1 --2- 172.21.15.28:0/1939745353 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb388106130 0x7fb38810c9e0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:33.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:33.680+0000 7fb38d200700 1 --2- 172.21.15.28:0/1939745353 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb388104590 0x7fb3881049b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:33.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:33.680+0000 7fb38d200700 1 --2- 172.21.15.28:0/1939745353 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb388105790 0x7fb388105bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:33.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:33.680+0000 7fb38d200700 1 -- 172.21.15.28:0/1939745353 >> 172.21.15.28:0/1939745353 conn(0x7fb3880ffb30 msgr2=0x7fb388101f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:08:33.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:33.680+0000 7fb38d200700 1 -- 172.21.15.28:0/1939745353 shutdown_connections 2024-05-12T22:08:33.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:33.680+0000 7fb38d200700 1 -- 172.21.15.28:0/1939745353 wait complete. 2024-05-12T22:08:33.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:33.681+0000 7fb38d200700 1 Processor -- start 2024-05-12T22:08:33.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:33.682+0000 7fb38d200700 1 -- start start 2024-05-12T22:08:33.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:33.682+0000 7fb38d200700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb388104590 0x7fb38819a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:33.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:33.682+0000 7fb38d200700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb388105790 0x7fb38819aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:33.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:33.682+0000 7fb38d200700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb388106130 0x7fb3881a1d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:33.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:33.682+0000 7fb38d200700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb3881101b0 con 0x7fb388105790 2024-05-12T22:08:33.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:33.682+0000 7fb38d200700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fb388110030 con 0x7fb388106130 2024-05-12T22:08:33.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:33.682+0000 7fb38d200700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb388110330 con 0x7fb388104590 2024-05-12T22:08:33.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:33.682+0000 7fb385e00700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb388104590 0x7fb38819a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:33.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:33.682+0000 7fb385e00700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb388104590 0x7fb38819a760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:41790/0 (socket says 172.21.15.28:41790) 2024-05-12T22:08:33.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:33.682+0000 7fb385e00700 1 -- 172.21.15.28:0/583111242 learned_addr learned my addr 172.21.15.28:0/583111242 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:08:33.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:33.682+0000 7fb385400700 1 --2- 172.21.15.28:0/583111242 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb388105790 0x7fb38819aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:33.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:33.683+0000 7fb386800700 1 --2- 172.21.15.28:0/583111242 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb388106130 0x7fb3881a1d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:33.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:33.683+0000 7fb385e00700 1 -- 172.21.15.28:0/583111242 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb388106130 msgr2=0x7fb3881a1d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:33.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:33.683+0000 7fb385e00700 1 --2- 172.21.15.28:0/583111242 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb388106130 0x7fb3881a1d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:33.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:33.683+0000 7fb385e00700 1 -- 172.21.15.28:0/583111242 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb388105790 msgr2=0x7fb38819aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:33.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:33.683+0000 7fb385e00700 1 --2- 172.21.15.28:0/583111242 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb388105790 0x7fb38819aca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:33.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:33.683+0000 7fb385e00700 1 -- 172.21.15.28:0/583111242 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3881a23c0 con 0x7fb388104590 2024-05-12T22:08:33.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:33.683+0000 7fb386800700 1 --2- 172.21.15.28:0/583111242 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb388106130 0x7fb3881a1d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-12T22:08:33.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:33.683+0000 7fb385e00700 1 --2- 172.21.15.28:0/583111242 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb388104590 0x7fb38819a760 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fb37800ea20 tx=0x7fb37800ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:08:33.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:33.683+0000 7fb37ea00700 1 -- 172.21.15.28:0/583111242 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb37800cbd0 con 0x7fb388104590 2024-05-12T22:08:33.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:33.683+0000 7fb37ea00700 1 -- 172.21.15.28:0/583111242 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb378004500 con 0x7fb388104590 2024-05-12T22:08:33.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:33.683+0000 7fb38d200700 1 -- 172.21.15.28:0/583111242 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3881a26b0 con 0x7fb388104590 2024-05-12T22:08:33.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:33.683+0000 7fb37ea00700 1 -- 172.21.15.28:0/583111242 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb378003d80 con 0x7fb388104590 2024-05-12T22:08:33.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:33.684+0000 7fb38d200700 1 -- 172.21.15.28:0/583111242 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3881a2bf0 con 0x7fb388104590 2024-05-12T22:08:33.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:33.685+0000 7fb38d200700 1 -- 172.21.15.28:0/583111242 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb388069010 con 0x7fb388104590 2024-05-12T22:08:33.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:33.687+0000 7fb37ea00700 1 -- 172.21.15.28:0/583111242 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fb378043ae0 con 0x7fb388104590 2024-05-12T22:08:33.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:33.688+0000 7fb37ea00700 1 --2- 172.21.15.28:0/583111242 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fb36c0a2c20 0x7fb36c0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:33.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:33.688+0000 7fb37ea00700 1 -- 172.21.15.28:0/583111242 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7fb3780cf2b0 con 0x7fb388104590 2024-05-12T22:08:33.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:33.688+0000 7fb385400700 1 --2- 172.21.15.28:0/583111242 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fb36c0a2c20 0x7fb36c0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:33.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:33.689+0000 7fb385400700 1 --2- 172.21.15.28:0/583111242 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fb36c0a2c20 0x7fb36c0a50e0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fb374009c60 tx=0x7fb374009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:08:33.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:33.690+0000 7fb37ea00700 1 -- 172.21.15.28:0/583111242 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fb37809c1c0 con 0x7fb388104590 2024-05-12T22:08:33.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:33.927+0000 7fb38d200700 1 -- 172.21.15.28:0/583111242 --> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) -- 0x7fb388062760 con 0x7fb36c0a2c20 2024-05-12T22:08:33.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:33.931+0000 7fb37ea00700 1 -- 172.21.15.28:0/583111242 <== mgr.14152 v2:172.21.15.28:6800/3123328651 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+532 (secure 0 0 0) 0x7fb388062760 con 0x7fb36c0a2c20 2024-05-12T22:08:33.931 INFO:teuthology.orchestra.run.smithi028.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-05-12T22:08:33.931 INFO:teuthology.orchestra.run.smithi028.stdout:smithi028 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB747401F4480DGN 447G No 3m ago Has a FileSystem, LVM detected 2024-05-12T22:08:33.931 INFO:teuthology.orchestra.run.smithi028.stdout:smithi086 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400UL400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-05-12T22:08:33.932 INFO:teuthology.orchestra.run.smithi028.stdout:smithi186 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT623300CB400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-05-12T22:08:33.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:33.934+0000 7fb38d200700 1 -- 172.21.15.28:0/583111242 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fb36c0a2c20 msgr2=0x7fb36c0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:33.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:33.934+0000 7fb38d200700 1 --2- 172.21.15.28:0/583111242 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fb36c0a2c20 0x7fb36c0a50e0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fb374009c60 tx=0x7fb374009260 comp rx=0 tx=0).stop 2024-05-12T22:08:33.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:33.934+0000 7fb38d200700 1 -- 172.21.15.28:0/583111242 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb388104590 msgr2=0x7fb38819a760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:33.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:33.934+0000 7fb38d200700 1 --2- 172.21.15.28:0/583111242 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb388104590 0x7fb38819a760 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fb37800ea20 tx=0x7fb37800ed30 comp rx=0 tx=0).stop 2024-05-12T22:08:33.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:33.935+0000 7fb38d200700 1 -- 172.21.15.28:0/583111242 shutdown_connections 2024-05-12T22:08:33.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:33.935+0000 7fb38d200700 1 --2- 172.21.15.28:0/583111242 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fb36c0a2c20 0x7fb36c0a50e0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:33.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:33.935+0000 7fb38d200700 1 --2- 172.21.15.28:0/583111242 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb388104590 0x7fb38819a760 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:33.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:33.935+0000 7fb38d200700 1 --2- 172.21.15.28:0/583111242 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb388106130 0x7fb3881a1d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:33.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:33.935+0000 7fb38d200700 1 --2- 172.21.15.28:0/583111242 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb388105790 0x7fb38819aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:33.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:33.935+0000 7fb38d200700 1 -- 172.21.15.28:0/583111242 >> 172.21.15.28:0/583111242 conn(0x7fb3880ffb30 msgr2=0x7fb388108a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:08:33.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:33.935+0000 7fb38d200700 1 -- 172.21.15.28:0/583111242 shutdown_connections 2024-05-12T22:08:33.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:33.935+0000 7fb38d200700 1 -- 172.21.15.28:0/583111242 wait complete. 2024-05-12T22:08:34.703 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-05-12T22:08:34.713 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi028.front.sepia.ceph.com 2024-05-12T22:08:34.713 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f72fecff68e1d400c4568684327c900485c20d6a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6b92e7b4-10aa-11ef-bc98-c7b262605968 -- bash -c 'ceph fs dump' 2024-05-12T22:08:35.185 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/mon.a/config 2024-05-12T22:08:35.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:35 smithi028 ceph-mon[29847]: pgmap v372: 97 pgs: 97 active+clean; 641 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-12T22:08:35.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:35 smithi028 ceph-mon[29847]: from='client.25036 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-12T22:08:35.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:35 smithi186 ceph-mon[34785]: pgmap v372: 97 pgs: 97 active+clean; 641 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-12T22:08:35.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:35 smithi186 ceph-mon[34785]: from='client.25036 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-12T22:08:35.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:35 smithi086 ceph-mon[34297]: pgmap v372: 97 pgs: 97 active+clean; 641 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-12T22:08:35.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:35 smithi086 ceph-mon[34297]: from='client.25036 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-12T22:08:37.153 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:37 smithi028 ceph-mon[29847]: pgmap v373: 97 pgs: 97 active+clean; 641 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-12T22:08:37.294 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:37 smithi086 ceph-mon[34297]: pgmap v373: 97 pgs: 97 active+clean; 641 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-12T22:08:37.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:37.331+0000 7fbfa3e00700 1 -- 172.21.15.28:0/2441621631 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbfa41034a0 msgr2=0x7fbfa4103900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:37.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:37.331+0000 7fbfa3e00700 1 --2- 172.21.15.28:0/2441621631 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbfa41034a0 0x7fbfa4103900 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fbf8c009990 tx=0x7fbf8c009ca0 comp rx=0 tx=0).stop 2024-05-12T22:08:37.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:37.331+0000 7fbfa3e00700 1 -- 172.21.15.28:0/2441621631 shutdown_connections 2024-05-12T22:08:37.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:37.331+0000 7fbfa3e00700 1 --2- 172.21.15.28:0/2441621631 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbfa4103e40 0x7fbfa410a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:37.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:37.331+0000 7fbfa3e00700 1 --2- 172.21.15.28:0/2441621631 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbfa41022a0 0x7fbfa41026c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:37.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:37.331+0000 7fbfa3e00700 1 --2- 172.21.15.28:0/2441621631 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbfa41034a0 0x7fbfa4103900 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:37.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:37.331+0000 7fbfa3e00700 1 -- 172.21.15.28:0/2441621631 >> 172.21.15.28:0/2441621631 conn(0x7fbfa40fd860 msgr2=0x7fbfa40ffc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:08:37.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:37.331+0000 7fbfa3e00700 1 -- 172.21.15.28:0/2441621631 shutdown_connections 2024-05-12T22:08:37.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:37.331+0000 7fbfa3e00700 1 -- 172.21.15.28:0/2441621631 wait complete. 2024-05-12T22:08:37.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:37.333+0000 7fbfa3e00700 1 Processor -- start 2024-05-12T22:08:37.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:37.333+0000 7fbfa3e00700 1 -- start start 2024-05-12T22:08:37.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:37.333+0000 7fbfa3e00700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbfa41022a0 0x7fbfa4198440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:37.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:37.333+0000 7fbfa3e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbfa41034a0 0x7fbfa4198980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:37.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:37.333+0000 7fbfa3e00700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbfa4103e40 0x7fbfa419fa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:37.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:37.333+0000 7fbfa3e00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbfa410de60 con 0x7fbfa41034a0 2024-05-12T22:08:37.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:37.333+0000 7fbfa3e00700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fbfa410dce0 con 0x7fbfa4103e40 2024-05-12T22:08:37.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:37.333+0000 7fbfa3e00700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fbfa410dfe0 con 0x7fbfa41022a0 2024-05-12T22:08:37.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:37.334+0000 7fbf9a000700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbfa41034a0 0x7fbfa4198980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:37.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:37.334+0000 7fbfa2a00700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbfa41022a0 0x7fbfa4198440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:37.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:37.334+0000 7fbf9a000700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbfa41034a0 0x7fbfa4198980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53482/0 (socket says 172.21.15.28:53482) 2024-05-12T22:08:37.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:37.334+0000 7fbfa3400700 1 --2- 172.21.15.28:0/220876563 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbfa4103e40 0x7fbfa419fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:37.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:37.334+0000 7fbf9a000700 1 -- 172.21.15.28:0/220876563 learned_addr learned my addr 172.21.15.28:0/220876563 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:08:37.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:37.334+0000 7fbfa2a00700 1 -- 172.21.15.28:0/220876563 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbfa4103e40 msgr2=0x7fbfa419fa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:37.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:37.334+0000 7fbfa2a00700 1 --2- 172.21.15.28:0/220876563 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbfa4103e40 0x7fbfa419fa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:37.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:37.334+0000 7fbfa2a00700 1 -- 172.21.15.28:0/220876563 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbfa41034a0 msgr2=0x7fbfa4198980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:37.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:37.334+0000 7fbfa2a00700 1 --2- 172.21.15.28:0/220876563 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbfa41034a0 0x7fbfa4198980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:37.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:37.334+0000 7fbfa2a00700 1 -- 172.21.15.28:0/220876563 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfa41a00a0 con 0x7fbfa41022a0 2024-05-12T22:08:37.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:37.334+0000 7fbfa3400700 1 --2- 172.21.15.28:0/220876563 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbfa4103e40 0x7fbfa419fa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:08:37.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:37.335+0000 7fbfa2a00700 1 --2- 172.21.15.28:0/220876563 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbfa41022a0 0x7fbfa4198440 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fbf9400e9f0 tx=0x7fbf9400ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:08:37.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:37.335+0000 7fbfa8a00700 1 -- 172.21.15.28:0/220876563 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf9400cb70 con 0x7fbfa41022a0 2024-05-12T22:08:37.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:37.335+0000 7fbfa8a00700 1 -- 172.21.15.28:0/220876563 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbf9400ee90 con 0x7fbfa41022a0 2024-05-12T22:08:37.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:37.335+0000 7fbfa3e00700 1 -- 172.21.15.28:0/220876563 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbfa41a0390 con 0x7fbfa41022a0 2024-05-12T22:08:37.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:37.335+0000 7fbfa8a00700 1 -- 172.21.15.28:0/220876563 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf94018680 con 0x7fbfa41022a0 2024-05-12T22:08:37.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:37.335+0000 7fbfa3e00700 1 -- 172.21.15.28:0/220876563 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbfa41a08d0 con 0x7fbfa41022a0 2024-05-12T22:08:37.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:37.336+0000 7fbfa3e00700 1 -- 172.21.15.28:0/220876563 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbfa4069010 con 0x7fbfa41022a0 2024-05-12T22:08:37.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:37.338+0000 7fbfa8a00700 1 -- 172.21.15.28:0/220876563 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fbf9401f020 con 0x7fbfa41022a0 2024-05-12T22:08:37.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:37.339+0000 7fbfa8a00700 1 --2- 172.21.15.28:0/220876563 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fbf880a2c20 0x7fbf880a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:37.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:37.340+0000 7fbfa8a00700 1 -- 172.21.15.28:0/220876563 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7fbf940cef50 con 0x7fbfa41022a0 2024-05-12T22:08:37.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:37.340+0000 7fbf9a000700 1 --2- 172.21.15.28:0/220876563 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fbf880a2c20 0x7fbf880a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:37.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:37.341+0000 7fbf9a000700 1 --2- 172.21.15.28:0/220876563 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fbf880a2c20 0x7fbf880a50e0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fbf8c0098e0 tx=0x7fbf8c005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:08:37.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:37.344+0000 7fbfa8a00700 1 -- 172.21.15.28:0/220876563 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fbf94053850 con 0x7fbfa41022a0 2024-05-12T22:08:37.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:37 smithi186 ceph-mon[34785]: pgmap v373: 97 pgs: 97 active+clean; 641 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-12T22:08:37.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:37.626+0000 7fbfa3e00700 1 -- 172.21.15.28:0/220876563 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump"} v 0) -- 0x7fbfa4062f30 con 0x7fbfa41022a0 2024-05-12T22:08:37.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:37.627+0000 7fbfa8a00700 1 -- 172.21.15.28:0/220876563 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 23 v23) ==== 76+0+3046 (secure 0 0 0) 0x7fbf9409be60 con 0x7fbfa41022a0 2024-05-12T22:08:37.627 INFO:teuthology.orchestra.run.smithi028.stdout:e23 2024-05-12T22:08:37.627 INFO:teuthology.orchestra.run.smithi028.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-05-12T22:08:37.627 INFO:teuthology.orchestra.run.smithi028.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-05-12T22:08:37.627 INFO:teuthology.orchestra.run.smithi028.stdout:legacy client fscid: 1 2024-05-12T22:08:37.627 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-12T22:08:37.627 INFO:teuthology.orchestra.run.smithi028.stdout:Filesystem 'cephfs' (1) 2024-05-12T22:08:37.627 INFO:teuthology.orchestra.run.smithi028.stdout:fs_name cephfs 2024-05-12T22:08:37.627 INFO:teuthology.orchestra.run.smithi028.stdout:epoch 23 2024-05-12T22:08:37.627 INFO:teuthology.orchestra.run.smithi028.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-05-12T22:08:37.627 INFO:teuthology.orchestra.run.smithi028.stdout:created 2024-05-12T22:06:33.426986+0000 2024-05-12T22:08:37.627 INFO:teuthology.orchestra.run.smithi028.stdout:modified 2024-05-12T22:06:38.789246+0000 2024-05-12T22:08:37.627 INFO:teuthology.orchestra.run.smithi028.stdout:tableserver 0 2024-05-12T22:08:37.628 INFO:teuthology.orchestra.run.smithi028.stdout:root 0 2024-05-12T22:08:37.628 INFO:teuthology.orchestra.run.smithi028.stdout:session_timeout 300 2024-05-12T22:08:37.628 INFO:teuthology.orchestra.run.smithi028.stdout:session_autoclose 300 2024-05-12T22:08:37.628 INFO:teuthology.orchestra.run.smithi028.stdout:max_file_size 1099511627776 2024-05-12T22:08:37.628 INFO:teuthology.orchestra.run.smithi028.stdout:max_xattr_size 65536 2024-05-12T22:08:37.628 INFO:teuthology.orchestra.run.smithi028.stdout:required_client_features {} 2024-05-12T22:08:37.628 INFO:teuthology.orchestra.run.smithi028.stdout:last_failure 0 2024-05-12T22:08:37.628 INFO:teuthology.orchestra.run.smithi028.stdout:last_failure_osd_epoch 0 2024-05-12T22:08:37.628 INFO:teuthology.orchestra.run.smithi028.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-05-12T22:08:37.628 INFO:teuthology.orchestra.run.smithi028.stdout:max_mds 3 2024-05-12T22:08:37.628 INFO:teuthology.orchestra.run.smithi028.stdout:in 0,1,2 2024-05-12T22:08:37.628 INFO:teuthology.orchestra.run.smithi028.stdout:up {0=24581,1=24550,2=24532} 2024-05-12T22:08:37.628 INFO:teuthology.orchestra.run.smithi028.stdout:failed 2024-05-12T22:08:37.628 INFO:teuthology.orchestra.run.smithi028.stdout:damaged 2024-05-12T22:08:37.628 INFO:teuthology.orchestra.run.smithi028.stdout:stopped 2024-05-12T22:08:37.628 INFO:teuthology.orchestra.run.smithi028.stdout:data_pools [3] 2024-05-12T22:08:37.629 INFO:teuthology.orchestra.run.smithi028.stdout:metadata_pool 2 2024-05-12T22:08:37.629 INFO:teuthology.orchestra.run.smithi028.stdout:inline_data disabled 2024-05-12T22:08:37.629 INFO:teuthology.orchestra.run.smithi028.stdout:balancer 2024-05-12T22:08:37.629 INFO:teuthology.orchestra.run.smithi028.stdout:bal_rank_mask -1 2024-05-12T22:08:37.629 INFO:teuthology.orchestra.run.smithi028.stdout:standby_count_wanted 0 2024-05-12T22:08:37.629 INFO:teuthology.orchestra.run.smithi028.stdout:qdb_cluster leader: 24581 members: 24550,24532,24581 2024-05-12T22:08:37.629 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.b{0:24581} state up:active seq 3 addr [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] compat {c=[1],r=[1],i=[1fff]}] 2024-05-12T22:08:37.629 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.h{0:24530} state up:standby-replay seq 1 addr [v2:172.21.15.86:6836/3782448884,v1:172.21.15.86:6837/3782448884] compat {c=[1],r=[1],i=[1fff]}] 2024-05-12T22:08:37.629 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.a{1:24550} state up:active seq 5 addr [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] compat {c=[1],r=[1],i=[1fff]}] 2024-05-12T22:08:37.629 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.i{1:24520} state up:standby-replay seq 1 addr [v2:172.21.15.186:6838/2399610807,v1:172.21.15.186:6839/2399610807] compat {c=[1],r=[1],i=[1fff]}] 2024-05-12T22:08:37.629 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.g{2:24532} state up:active seq 6 addr [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] compat {c=[1],r=[1],i=[1fff]}] 2024-05-12T22:08:37.629 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.c{2:24499} state up:standby-replay seq 1 addr [v2:172.21.15.186:6836/3600809235,v1:172.21.15.186:6837/3600809235] compat {c=[1],r=[1],i=[1fff]}] 2024-05-12T22:08:37.629 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-12T22:08:37.629 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-12T22:08:37.629 INFO:teuthology.orchestra.run.smithi028.stdout:Standby daemons: 2024-05-12T22:08:37.629 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-12T22:08:37.630 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.l{-1:24427} state up:standby seq 1 addr [v2:172.21.15.186:6832/1844141745,v1:172.21.15.186:6833/1844141745] compat {c=[1],r=[1],i=[1fff]}] 2024-05-12T22:08:37.630 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.k{-1:24452} state up:standby seq 1 addr [v2:172.21.15.86:6832/2960763758,v1:172.21.15.86:6833/2960763758] compat {c=[1],r=[1],i=[1fff]}] 2024-05-12T22:08:37.630 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.j{-1:24470} state up:standby seq 1 addr [v2:172.21.15.28:6834/4131134219,v1:172.21.15.28:6835/4131134219] compat {c=[1],r=[1],i=[1fff]}] 2024-05-12T22:08:37.630 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.f{-1:24481} state up:standby seq 1 addr [v2:172.21.15.186:6834/1727869064,v1:172.21.15.186:6835/1727869064] compat {c=[1],r=[1],i=[1fff]}] 2024-05-12T22:08:37.630 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.d{-1:24482} state up:standby seq 1 addr [v2:172.21.15.28:6836/2938260704,v1:172.21.15.28:6837/2938260704] compat {c=[1],r=[1],i=[1fff]}] 2024-05-12T22:08:37.630 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.e{-1:24494} state up:standby seq 1 addr [v2:172.21.15.86:6834/3473700910,v1:172.21.15.86:6835/3473700910] compat {c=[1],r=[1],i=[1fff]}] 2024-05-12T22:08:37.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:37.631+0000 7fbfa3e00700 1 -- 172.21.15.28:0/220876563 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fbf880a2c20 msgr2=0x7fbf880a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:37.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:37.631+0000 7fbfa3e00700 1 --2- 172.21.15.28:0/220876563 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fbf880a2c20 0x7fbf880a50e0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fbf8c0098e0 tx=0x7fbf8c005960 comp rx=0 tx=0).stop 2024-05-12T22:08:37.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:37.631+0000 7fbfa3e00700 1 -- 172.21.15.28:0/220876563 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbfa41022a0 msgr2=0x7fbfa4198440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:37.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:37.631+0000 7fbfa3e00700 1 --2- 172.21.15.28:0/220876563 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbfa41022a0 0x7fbfa4198440 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fbf9400e9f0 tx=0x7fbf9400ed00 comp rx=0 tx=0).stop 2024-05-12T22:08:37.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:37.632+0000 7fbfa3e00700 1 -- 172.21.15.28:0/220876563 shutdown_connections 2024-05-12T22:08:37.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:37.632+0000 7fbfa3e00700 1 --2- 172.21.15.28:0/220876563 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fbf880a2c20 0x7fbf880a50e0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:37.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:37.632+0000 7fbfa3e00700 1 --2- 172.21.15.28:0/220876563 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbfa41022a0 0x7fbfa4198440 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:37.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:37.632+0000 7fbfa3e00700 1 --2- 172.21.15.28:0/220876563 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbfa4103e40 0x7fbfa419fa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:37.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:37.632+0000 7fbfa3e00700 1 --2- 172.21.15.28:0/220876563 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbfa41034a0 0x7fbfa4198980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:37.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:37.632+0000 7fbfa3e00700 1 -- 172.21.15.28:0/220876563 >> 172.21.15.28:0/220876563 conn(0x7fbfa40fd860 msgr2=0x7fbfa41067a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:08:37.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:37.632+0000 7fbfa3e00700 1 -- 172.21.15.28:0/220876563 shutdown_connections 2024-05-12T22:08:37.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:37.632+0000 7fbfa3e00700 1 -- 172.21.15.28:0/220876563 wait complete. 2024-05-12T22:08:37.633 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 23 2024-05-12T22:08:38.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:38 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/220876563' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-05-12T22:08:38.427 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f72fecff68e1d400c4568684327c900485c20d6a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6b92e7b4-10aa-11ef-bc98-c7b262605968 -- bash -c 'ceph osd dump' 2024-05-12T22:08:38.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:38 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/220876563' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-05-12T22:08:38.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:38 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/220876563' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-05-12T22:08:38.900 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/mon.a/config 2024-05-12T22:08:39.275 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:39 smithi028 ceph-mon[29847]: pgmap v374: 97 pgs: 97 active+clean; 641 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-12T22:08:39.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:39 smithi186 ceph-mon[34785]: pgmap v374: 97 pgs: 97 active+clean; 641 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-12T22:08:39.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:39 smithi086 ceph-mon[34297]: pgmap v374: 97 pgs: 97 active+clean; 641 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-12T22:08:41.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:41.131+0000 7f4395400700 1 -- 172.21.15.28:0/2176506982 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f439010a560 msgr2=0x7f4390110e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:41.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:41.131+0000 7f4395400700 1 --2- 172.21.15.28:0/2176506982 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f439010a560 0x7f4390110e10 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f438800b0d0 tx=0x7f438800b3e0 comp rx=0 tx=0).stop 2024-05-12T22:08:41.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:41.131+0000 7f4395400700 1 -- 172.21.15.28:0/2176506982 shutdown_connections 2024-05-12T22:08:41.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:41.131+0000 7f4395400700 1 --2- 172.21.15.28:0/2176506982 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f439010a560 0x7f4390110e10 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:41.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:41.131+0000 7f4395400700 1 --2- 172.21.15.28:0/2176506982 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f43901089c0 0x7f4390108de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:41.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:41.131+0000 7f4395400700 1 --2- 172.21.15.28:0/2176506982 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4390109bc0 0x7f439010a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:41.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:41.131+0000 7f4395400700 1 -- 172.21.15.28:0/2176506982 >> 172.21.15.28:0/2176506982 conn(0x7f4390077990 msgr2=0x7f4390079dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:08:41.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:41.131+0000 7f4395400700 1 -- 172.21.15.28:0/2176506982 shutdown_connections 2024-05-12T22:08:41.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:41.131+0000 7f4395400700 1 -- 172.21.15.28:0/2176506982 wait complete. 2024-05-12T22:08:41.132 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:41 smithi028 ceph-mon[29847]: pgmap v375: 97 pgs: 97 active+clean; 641 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-12T22:08:41.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:41.132+0000 7f4395400700 1 Processor -- start 2024-05-12T22:08:41.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:41.133+0000 7f4395400700 1 -- start start 2024-05-12T22:08:41.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:41.133+0000 7f4395400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f43901089c0 0x7f439019eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:41.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:41.133+0000 7f4395400700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4390109bc0 0x7f439019f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:41.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:41.133+0000 7f4395400700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f439010a560 0x7f43901a60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:41.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:41.133+0000 7f4395400700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4390114600 con 0x7f43901089c0 2024-05-12T22:08:41.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:41.133+0000 7f4395400700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f4390114480 con 0x7f439010a560 2024-05-12T22:08:41.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:41.133+0000 7f4395400700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f4390114780 con 0x7f4390109bc0 2024-05-12T22:08:41.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:41.133+0000 7f438e800700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f43901089c0 0x7f439019eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:41.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:41.133+0000 7f438e800700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f43901089c0 0x7f439019eb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53500/0 (socket says 172.21.15.28:53500) 2024-05-12T22:08:41.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:41.133+0000 7f438f200700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f439010a560 0x7f43901a60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:41.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:41.133+0000 7f438e800700 1 -- 172.21.15.28:0/1751004210 learned_addr learned my addr 172.21.15.28:0/1751004210 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:08:41.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:41.133+0000 7f438de00700 1 --2- 172.21.15.28:0/1751004210 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4390109bc0 0x7f439019f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:41.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:41.134+0000 7f438f200700 1 -- 172.21.15.28:0/1751004210 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4390109bc0 msgr2=0x7f439019f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:41.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:41.134+0000 7f438f200700 1 --2- 172.21.15.28:0/1751004210 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4390109bc0 0x7f439019f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:41.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:41.134+0000 7f438f200700 1 -- 172.21.15.28:0/1751004210 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f43901089c0 msgr2=0x7f439019eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:41.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:41.134+0000 7f438f200700 1 --2- 172.21.15.28:0/1751004210 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f43901089c0 0x7f439019eb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:41.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:41.134+0000 7f438f200700 1 -- 172.21.15.28:0/1751004210 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43901a6780 con 0x7f439010a560 2024-05-12T22:08:41.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:41.134+0000 7f438f200700 1 --2- 172.21.15.28:0/1751004210 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f439010a560 0x7f43901a60e0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f4388009fd0 tx=0x7f4388009810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:08:41.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:41.134+0000 7f4387400700 1 -- 172.21.15.28:0/1751004210 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4388021020 con 0x7f439010a560 2024-05-12T22:08:41.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:41.134+0000 7f4395400700 1 -- 172.21.15.28:0/1751004210 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43901a6a10 con 0x7f439010a560 2024-05-12T22:08:41.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:41.135+0000 7f4387400700 1 -- 172.21.15.28:0/1751004210 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4388004c50 con 0x7f439010a560 2024-05-12T22:08:41.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:41.135+0000 7f4387400700 1 -- 172.21.15.28:0/1751004210 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43880052b0 con 0x7f439010a560 2024-05-12T22:08:41.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:41.135+0000 7f4395400700 1 -- 172.21.15.28:0/1751004210 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43901a6f10 con 0x7f439010a560 2024-05-12T22:08:41.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:41.136+0000 7f4395400700 1 -- 172.21.15.28:0/1751004210 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4390069010 con 0x7f439010a560 2024-05-12T22:08:41.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:41.138+0000 7f4387400700 1 -- 172.21.15.28:0/1751004210 <== mon.2 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f4388024020 con 0x7f439010a560 2024-05-12T22:08:41.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:41.139+0000 7f4387400700 1 --2- 172.21.15.28:0/1751004210 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f43740a2c20 0x7f43740a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:41.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:41.139+0000 7f4387400700 1 -- 172.21.15.28:0/1751004210 <== mon.2 v2:172.21.15.86:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f438800f020 con 0x7f439010a560 2024-05-12T22:08:41.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:41.139+0000 7f438e800700 1 --2- 172.21.15.28:0/1751004210 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f43740a2c20 0x7f43740a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:41.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:41.140+0000 7f438e800700 1 --2- 172.21.15.28:0/1751004210 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f43740a2c20 0x7f43740a50e0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f4380005950 tx=0x7f4380009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:08:41.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:41.143+0000 7f4387400700 1 -- 172.21.15.28:0/1751004210 <== mon.2 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f438809d030 con 0x7f439010a560 2024-05-12T22:08:41.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:41.372+0000 7f4395400700 1 -- 172.21.15.28:0/1751004210 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "osd dump"} v 0) -- 0x7f4390062f30 con 0x7f439010a560 2024-05-12T22:08:41.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:41.373+0000 7f4387400700 1 -- 172.21.15.28:0/1751004210 <== mon.2 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v88) ==== 56+0+5266 (secure 0 0 0) 0x7f438809c930 con 0x7f439010a560 2024-05-12T22:08:41.374 INFO:teuthology.orchestra.run.smithi028.stdout:epoch 88 2024-05-12T22:08:41.374 INFO:teuthology.orchestra.run.smithi028.stdout:fsid 6b92e7b4-10aa-11ef-bc98-c7b262605968 2024-05-12T22:08:41.375 INFO:teuthology.orchestra.run.smithi028.stdout:created 2024-05-12T21:57:37.276782+0000 2024-05-12T22:08:41.375 INFO:teuthology.orchestra.run.smithi028.stdout:modified 2024-05-12T22:06:46.768798+0000 2024-05-12T22:08:41.375 INFO:teuthology.orchestra.run.smithi028.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-05-12T22:08:41.375 INFO:teuthology.orchestra.run.smithi028.stdout:crush_version 26 2024-05-12T22:08:41.375 INFO:teuthology.orchestra.run.smithi028.stdout:full_ratio 0.95 2024-05-12T22:08:41.375 INFO:teuthology.orchestra.run.smithi028.stdout:backfillfull_ratio 0.9 2024-05-12T22:08:41.376 INFO:teuthology.orchestra.run.smithi028.stdout:nearfull_ratio 0.85 2024-05-12T22:08:41.376 INFO:teuthology.orchestra.run.smithi028.stdout:require_min_compat_client luminous 2024-05-12T22:08:41.376 INFO:teuthology.orchestra.run.smithi028.stdout:min_compat_client luminous 2024-05-12T22:08:41.376 INFO:teuthology.orchestra.run.smithi028.stdout:require_osd_release squid 2024-05-12T22:08:41.376 INFO:teuthology.orchestra.run.smithi028.stdout:stretch_mode_enabled false 2024-05-12T22:08:41.376 INFO:teuthology.orchestra.run.smithi028.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 22 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-05-12T22:08:41.377 INFO:teuthology.orchestra.run.smithi028.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 85 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2024-05-12T22:08:41.377 INFO:teuthology.orchestra.run.smithi028.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 86 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.9 application cephfs read_balance_score 1.50 2024-05-12T22:08:41.377 INFO:teuthology.orchestra.run.smithi028.stdout:max_osd 12 2024-05-12T22:08:41.377 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0 up in weight 1 up_from 8 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.28:6802/906003882,v1:172.21.15.28:6803/906003882] [v2:172.21.15.28:6804/906003882,v1:172.21.15.28:6805/906003882] exists,up 6992ce0a-844b-40e3-8ed1-181c8474dccf 2024-05-12T22:08:41.377 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1 up in weight 1 up_from 13 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.86:6800/3354141819,v1:172.21.15.86:6801/3354141819] [v2:172.21.15.86:6802/3354141819,v1:172.21.15.86:6803/3354141819] exists,up ced20c96-c481-4d48-90d1-6b43228959ce 2024-05-12T22:08:41.377 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2 up in weight 1 up_from 18 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.186:6800/906459365,v1:172.21.15.186:6801/906459365] [v2:172.21.15.186:6802/906459365,v1:172.21.15.186:6803/906459365] exists,up 54a2f221-fa65-4795-9f5e-e660cc7c0098 2024-05-12T22:08:41.378 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3 up in weight 1 up_from 26 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.28:6810/4182884436,v1:172.21.15.28:6811/4182884436] [v2:172.21.15.28:6812/4182884436,v1:172.21.15.28:6813/4182884436] exists,up 86836737-ac9a-487b-93f2-7fcd3c406960 2024-05-12T22:08:41.378 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4 up in weight 1 up_from 34 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.86:6808/2596998708,v1:172.21.15.86:6809/2596998708] [v2:172.21.15.86:6810/2596998708,v1:172.21.15.86:6811/2596998708] exists,up d162ebcd-83c0-4fd5-9925-28b2cecea935 2024-05-12T22:08:41.378 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5 up in weight 1 up_from 39 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.186:6808/1944205756,v1:172.21.15.186:6809/1944205756] [v2:172.21.15.186:6810/1944205756,v1:172.21.15.186:6811/1944205756] exists,up c794336a-a105-425c-b6c7-a32b90c05bb8 2024-05-12T22:08:41.378 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6 up in weight 1 up_from 47 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.28:6818/1642308925,v1:172.21.15.28:6819/1642308925] [v2:172.21.15.28:6820/1642308925,v1:172.21.15.28:6821/1642308925] exists,up 516932ac-af00-45b7-ad3c-d6dec7db442f 2024-05-12T22:08:41.378 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7 up in weight 1 up_from 52 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.86:6816/726062711,v1:172.21.15.86:6817/726062711] [v2:172.21.15.86:6818/726062711,v1:172.21.15.86:6819/726062711] exists,up ca7a059f-0414-438e-b2d1-a10f348036be 2024-05-12T22:08:41.378 INFO:teuthology.orchestra.run.smithi028.stdout:osd.8 up in weight 1 up_from 60 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.186:6816/3007687277,v1:172.21.15.186:6817/3007687277] [v2:172.21.15.186:6818/3007687277,v1:172.21.15.186:6819/3007687277] exists,up 92e3f41f-aebd-422e-a99e-555f7ba4aa53 2024-05-12T22:08:41.379 INFO:teuthology.orchestra.run.smithi028.stdout:osd.9 up in weight 1 up_from 68 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.28:6826/136220472,v1:172.21.15.28:6827/136220472] [v2:172.21.15.28:6828/136220472,v1:172.21.15.28:6829/136220472] exists,up 38ff7d9d-ee85-404c-8c59-adc327e1ccfa 2024-05-12T22:08:41.379 INFO:teuthology.orchestra.run.smithi028.stdout:osd.10 up in weight 1 up_from 73 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.86:6824/1189968310,v1:172.21.15.86:6825/1189968310] [v2:172.21.15.86:6826/1189968310,v1:172.21.15.86:6827/1189968310] exists,up 5e9ff657-afa8-4bab-8491-f209038392b4 2024-05-12T22:08:41.379 INFO:teuthology.orchestra.run.smithi028.stdout:osd.11 up in weight 1 up_from 81 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.186:6824/3801668110,v1:172.21.15.186:6825/3801668110] [v2:172.21.15.186:6826/3801668110,v1:172.21.15.186:6827/3801668110] exists,up f112e3c6-65b6-4253-a966-62001db7d069 2024-05-12T22:08:41.379 INFO:teuthology.orchestra.run.smithi028.stdout:pg_upmap_items 2.b [2,1] 2024-05-12T22:08:41.379 INFO:teuthology.orchestra.run.smithi028.stdout:pg_upmap_items 2.16 [2,4] 2024-05-12T22:08:41.379 INFO:teuthology.orchestra.run.smithi028.stdout:pg_upmap_items 2.1c [8,11] 2024-05-12T22:08:41.379 INFO:teuthology.orchestra.run.smithi028.stdout:pg_upmap_items 2.1f [7,11] 2024-05-12T22:08:41.380 INFO:teuthology.orchestra.run.smithi028.stdout:pg_upmap_items 3.13 [4,9] 2024-05-12T22:08:41.380 INFO:teuthology.orchestra.run.smithi028.stdout:pg_upmap_items 3.25 [5,9] 2024-05-12T22:08:41.380 INFO:teuthology.orchestra.run.smithi028.stdout:pg_upmap_items 3.27 [4,8] 2024-05-12T22:08:41.380 INFO:teuthology.orchestra.run.smithi028.stdout:pg_upmap_items 3.3c [4,9] 2024-05-12T22:08:41.380 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:0/1177009533 expires 2024-05-13T21:58:45.550459+0000 2024-05-12T22:08:41.380 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:6800/3760499540 expires 2024-05-13T21:58:45.550459+0000 2024-05-12T22:08:41.381 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:6801/3783742728 expires 2024-05-13T21:58:10.421564+0000 2024-05-12T22:08:41.381 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:6800/3783742728 expires 2024-05-13T21:58:10.421564+0000 2024-05-12T22:08:41.381 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:0/2229156260 expires 2024-05-13T21:58:45.550459+0000 2024-05-12T22:08:41.381 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:0/1202140963 expires 2024-05-13T21:58:10.421564+0000 2024-05-12T22:08:41.381 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:0/2924223784 expires 2024-05-13T21:58:45.550459+0000 2024-05-12T22:08:41.381 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:6801/3760499540 expires 2024-05-13T21:58:45.550459+0000 2024-05-12T22:08:41.382 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:0/1240027206 expires 2024-05-13T21:58:10.421564+0000 2024-05-12T22:08:41.382 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:0/171417728 expires 2024-05-13T21:58:10.421564+0000 2024-05-12T22:08:41.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:41.377+0000 7f4395400700 1 -- 172.21.15.28:0/1751004210 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f43740a2c20 msgr2=0x7f43740a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:41.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:41.377+0000 7f4395400700 1 --2- 172.21.15.28:0/1751004210 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f43740a2c20 0x7f43740a50e0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f4380005950 tx=0x7f4380009260 comp rx=0 tx=0).stop 2024-05-12T22:08:41.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:41.377+0000 7f4395400700 1 -- 172.21.15.28:0/1751004210 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f439010a560 msgr2=0x7f43901a60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:41.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:41.377+0000 7f4395400700 1 --2- 172.21.15.28:0/1751004210 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f439010a560 0x7f43901a60e0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f4388009fd0 tx=0x7f4388009810 comp rx=0 tx=0).stop 2024-05-12T22:08:41.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:41.377+0000 7f4395400700 1 -- 172.21.15.28:0/1751004210 shutdown_connections 2024-05-12T22:08:41.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:41.377+0000 7f4395400700 1 --2- 172.21.15.28:0/1751004210 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f43740a2c20 0x7f43740a50e0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:41.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:41.377+0000 7f4395400700 1 --2- 172.21.15.28:0/1751004210 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f439010a560 0x7f43901a60e0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:41.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:41.377+0000 7f4395400700 1 --2- 172.21.15.28:0/1751004210 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f43901089c0 0x7f439019eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:41.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:41.377+0000 7f4395400700 1 --2- 172.21.15.28:0/1751004210 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4390109bc0 0x7f439019f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:41.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:41.377+0000 7f4395400700 1 -- 172.21.15.28:0/1751004210 >> 172.21.15.28:0/1751004210 conn(0x7f4390077990 msgr2=0x7f439010cdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:08:41.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:41.377+0000 7f4395400700 1 -- 172.21.15.28:0/1751004210 shutdown_connections 2024-05-12T22:08:41.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:41.378+0000 7f4395400700 1 -- 172.21.15.28:0/1751004210 wait complete. 2024-05-12T22:08:41.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:41 smithi186 ceph-mon[34785]: pgmap v375: 97 pgs: 97 active+clean; 641 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-12T22:08:41.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:41 smithi086 ceph-mon[34297]: pgmap v375: 97 pgs: 97 active+clean; 641 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-12T22:08:42.096 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:42 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/1751004210' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-05-12T22:08:42.178 INFO:teuthology.run_tasks:Running task fs.ready... 2024-05-12T22:08:42.190 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-12T22:08:42.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:42.414+0000 7f0f8ea00640 1 -- 172.21.15.28:0/675023603 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0f88108ea0 msgr2=0x7f0f88109320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:42.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:42.414+0000 7f0f8ea00640 1 --2- 172.21.15.28:0/675023603 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0f88108ea0 0x7f0f88109320 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f0f74009860 tx=0x7f0f7402a340 comp rx=0 tx=0).stop 2024-05-12T22:08:42.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:42.414+0000 7f0f8ea00640 1 -- 172.21.15.28:0/675023603 shutdown_connections 2024-05-12T22:08:42.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:42.414+0000 7f0f8ea00640 1 --2- 172.21.15.28:0/675023603 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f88109860 0x7f0f881100f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:42.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:42.414+0000 7f0f8ea00640 1 --2- 172.21.15.28:0/675023603 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0f88108ea0 0x7f0f88109320 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:42.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:42.414+0000 7f0f8ea00640 1 --2- 172.21.15.28:0/675023603 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0f88107cb0 0x7f0f881080d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:42.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:42.414+0000 7f0f8ea00640 1 -- 172.21.15.28:0/675023603 >> 172.21.15.28:0/675023603 conn(0x7f0f88076c70 msgr2=0x7f0f88079090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:08:42.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:42.414+0000 7f0f8ea00640 1 -- 172.21.15.28:0/675023603 shutdown_connections 2024-05-12T22:08:42.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:42.414+0000 7f0f8ea00640 1 -- 172.21.15.28:0/675023603 wait complete. 2024-05-12T22:08:42.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:42.415+0000 7f0f8ea00640 1 Processor -- start 2024-05-12T22:08:42.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:42.415+0000 7f0f8ea00640 1 -- start start 2024-05-12T22:08:42.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:42.416+0000 7f0f8ea00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0f88107cb0 0x7f0f8819e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:42.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:42.416+0000 7f0f8ea00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0f88108ea0 0x7f0f8819e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:42.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:42.416+0000 7f0f8ea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f88109860 0x7f0f881a5580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:42.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:42.416+0000 7f0f8ea00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0f88113930 con 0x7f0f88109860 2024-05-12T22:08:42.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:42.416+0000 7f0f8ea00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f0f881137b0 con 0x7f0f88107cb0 2024-05-12T22:08:42.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:42.416+0000 7f0f8ea00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f0f88113ab0 con 0x7f0f88108ea0 2024-05-12T22:08:42.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:42.416+0000 7f0f8e000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f88109860 0x7f0f881a5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:42.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:42.416+0000 7f0f8e000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f88109860 0x7f0f881a5580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53512/0 (socket says 172.21.15.28:53512) 2024-05-12T22:08:42.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:42.416+0000 7f0f8e000640 1 -- 172.21.15.28:0/2774140924 learned_addr learned my addr 172.21.15.28:0/2774140924 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:08:42.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:42.416+0000 7f0f8d600640 1 --2- 172.21.15.28:0/2774140924 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0f88107cb0 0x7f0f8819e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:42.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:42.416+0000 7f0f8cc00640 1 --2- 172.21.15.28:0/2774140924 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0f88108ea0 0x7f0f8819e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:42.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:42.417+0000 7f0f8d600640 1 -- 172.21.15.28:0/2774140924 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0f88108ea0 msgr2=0x7f0f8819e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:42.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:42.417+0000 7f0f8d600640 1 --2- 172.21.15.28:0/2774140924 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0f88108ea0 0x7f0f8819e560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:42.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:42.417+0000 7f0f8d600640 1 -- 172.21.15.28:0/2774140924 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f88109860 msgr2=0x7f0f881a5580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:42.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:42.417+0000 7f0f8d600640 1 --2- 172.21.15.28:0/2774140924 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f88109860 0x7f0f881a5580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:42.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:42.417+0000 7f0f8d600640 1 -- 172.21.15.28:0/2774140924 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f881a5bf0 con 0x7f0f88107cb0 2024-05-12T22:08:42.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:42.417+0000 7f0f8cc00640 1 --2- 172.21.15.28:0/2774140924 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0f88108ea0 0x7f0f8819e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:08:42.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:42.417+0000 7f0f8e000640 1 --2- 172.21.15.28:0/2774140924 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f88109860 0x7f0f881a5580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:08:42.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:42.417+0000 7f0f8d600640 1 --2- 172.21.15.28:0/2774140924 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0f88107cb0 0x7f0f8819e020 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f0f7c00e9c0 tx=0x7f0f7c00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:08:42.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:42.417+0000 7f0f86000640 1 -- 172.21.15.28:0/2774140924 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f7c00ccc0 con 0x7f0f88107cb0 2024-05-12T22:08:42.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:42.417+0000 7f0f86000640 1 -- 172.21.15.28:0/2774140924 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0f7c0035f0 con 0x7f0f88107cb0 2024-05-12T22:08:42.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:42.417+0000 7f0f86000640 1 -- 172.21.15.28:0/2774140924 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f7c00ce60 con 0x7f0f88107cb0 2024-05-12T22:08:42.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:42.417+0000 7f0f8ea00640 1 -- 172.21.15.28:0/2774140924 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f881a5ee0 con 0x7f0f88107cb0 2024-05-12T22:08:42.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:42.417+0000 7f0f8ea00640 1 -- 172.21.15.28:0/2774140924 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f881a62f0 con 0x7f0f88107cb0 2024-05-12T22:08:42.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:42.418+0000 7f0f8ea00640 1 -- 172.21.15.28:0/2774140924 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f8810efd0 con 0x7f0f88107cb0 2024-05-12T22:08:42.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:42.421+0000 7f0f86000640 1 -- 172.21.15.28:0/2774140924 <== mon.2 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f0f7c0037a0 con 0x7f0f88107cb0 2024-05-12T22:08:42.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:42.422+0000 7f0f86000640 1 --2- 172.21.15.28:0/2774140924 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f0f700a2b90 0x7f0f700a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:42.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:42.422+0000 7f0f86000640 1 -- 172.21.15.28:0/2774140924 <== mon.2 v2:172.21.15.86:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f0f7c0cf6c0 con 0x7f0f88107cb0 2024-05-12T22:08:42.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:42.423+0000 7f0f8cc00640 1 --2- 172.21.15.28:0/2774140924 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f0f700a2b90 0x7f0f700a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:42.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:42.423+0000 7f0f8cc00640 1 --2- 172.21.15.28:0/2774140924 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f0f700a2b90 0x7f0f700a5050 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f0f7402a930 tx=0x7f0f74030570 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:08:42.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:42.424+0000 7f0f86000640 1 -- 172.21.15.28:0/2774140924 <== mon.2 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f0f7c09c5d0 con 0x7f0f88107cb0 2024-05-12T22:08:42.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:42 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/1751004210' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-05-12T22:08:42.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:42 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/1751004210' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-05-12T22:08:42.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:42.629+0000 7f0f8ea00640 1 -- 172.21.15.28:0/2774140924 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0f8802d450 con 0x7f0f88107cb0 2024-05-12T22:08:42.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:42.632+0000 7f0f86000640 1 -- 172.21.15.28:0/2774140924 <== mon.2 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22595 (secure 0 0 0) 0x7f0f7c09bed0 con 0x7f0f88107cb0 2024-05-12T22:08:42.633 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-12T22:08:42.633 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":88,"fsid":"6b92e7b4-10aa-11ef-bc98-c7b262605968","created":"2024-05-12T21:57:37.276782+0000","modified":"2024-05-12T22:06:46.768798+0000","last_up_change":"2024-05-12T22:05:37.929562+0000","last_in_change":"2024-05-12T22:05:18.750768+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-12T22:01:32.292826+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-12T22:06:31.540263+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-12T22:06:32.906118+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6992ce0a-844b-40e3-8ed1-181c8474dccf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":906003882},{"type":"v1","addr":"172.21.15.28:6803","nonce":906003882}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":906003882},{"type":"v1","addr":"172.21.15.28:6805","nonce":906003882}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":906003882},{"type":"v1","addr":"172.21.15.28:6809","nonce":906003882}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":906003882},{"type":"v1","addr":"172.21.15.28:6807","nonce":906003882}]},"public_addr":"172.21.15.28:6803/906003882","cluster_addr":"172.21.15.28:6805/906003882","heartbeat_back_addr":"172.21.15.28:6809/906003882","heartbeat_front_addr":"172.21.15.28:6807/906003882","state":["exists","up"]},{"osd":1,"uuid":"ced20c96-c481-4d48-90d1-6b43228959ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":3354141819},{"type":"v1","addr":"172.21.15.86:6801","nonce":3354141819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":3354141819},{"type":"v1","addr":"172.21.15.86:6803","nonce":3354141819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":3354141819},{"type":"v1","addr":"172.21.15.86:6807","nonce":3354141819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":3354141819},{"type":"v1","addr":"172.21.15.86:6805","nonce":3354141819}]},"public_addr":"172.21.15.86:6801/3354141819","cluster_addr":"172.21.15.86:6803/3354141819","heartbeat_back_addr":"172.21.15.86:6807/3354141819","heartbeat_front_addr":"172.21.15.86:6805/3354141819","state":["exists","up"]},{"osd":2,"uuid":"54a2f221-fa65-4795-9f5e-e660cc7c0098","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":906459365},{"type":"v1","addr":"172.21.15.186:6801","nonce":906459365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":906459365},{"type":"v1","addr":"172.21.15.186:6803","nonce":906459365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":906459365},{"type":"v1","addr":"172.21.15.186:6807","nonce":906459365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":906459365},{"type":"v1","addr":"172.21.15.186:6805","nonce":906459365}]},"public_addr":"172.21.15.186:6801/906459365","cluster_addr":"172.21.15.186:6803/906459365","heartbeat_back_addr":"172.21.15.186:6807/906459365","heartbeat_front_addr":"172.21.15.186:6805/906459365","state":["exists","up"]},{"osd":3,"uuid":"86836737-ac9a-487b-93f2-7fcd3c406960","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":4182884436},{"type":"v1","addr":"172.21.15.28:6811","nonce":4182884436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":4182884436},{"type":"v1","addr":"172.21.15.28:6813","nonce":4182884436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":4182884436},{"type":"v1","addr":"172.21.15.28:6817","nonce":4182884436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":4182884436},{"type":"v1","addr":"172.21.15.28:6815","nonce":4182884436}]},"public_addr":"172.21.15.28:6811/4182884436","cluster_addr":"172.21.15.28:6813/4182884436","heartbeat_back_addr":"172.21.15.28:6817/4182884436","heartbeat_front_addr":"172.21.15.28:6815/4182884436","state":["exists","up"]},{"osd":4,"uuid":"d162ebcd-83c0-4fd5-9925-28b2cecea935","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2596998708},{"type":"v1","addr":"172.21.15.86:6809","nonce":2596998708}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2596998708},{"type":"v1","addr":"172.21.15.86:6811","nonce":2596998708}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2596998708},{"type":"v1","addr":"172.21.15.86:6815","nonce":2596998708}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2596998708},{"type":"v1","addr":"172.21.15.86:6813","nonce":2596998708}]},"public_addr":"172.21.15.86:6809/2596998708","cluster_addr":"172.21.15.86:6811/2596998708","heartbeat_back_addr":"172.21.15.86:6815/2596998708","heartbeat_front_addr":"172.21.15.86:6813/2596998708","state":["exists","up"]},{"osd":5,"uuid":"c794336a-a105-425c-b6c7-a32b90c05bb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":1944205756},{"type":"v1","addr":"172.21.15.186:6809","nonce":1944205756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":1944205756},{"type":"v1","addr":"172.21.15.186:6811","nonce":1944205756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":1944205756},{"type":"v1","addr":"172.21.15.186:6815","nonce":1944205756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":1944205756},{"type":"v1","addr":"172.21.15.186:6813","nonce":1944205756}]},"public_addr":"172.21.15.186:6809/1944205756","cluster_addr":"172.21.15.186:6811/1944205756","heartbeat_back_addr":"172.21.15.186:6815/1944205756","heartbeat_front_addr":"172.21.15.186:6813/1944205756","state":["exists","up"]},{"osd":6,"uuid":"516932ac-af00-45b7-ad3c-d6dec7db442f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":1642308925},{"type":"v1","addr":"172.21.15.28:6819","nonce":1642308925}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":1642308925},{"type":"v1","addr":"172.21.15.28:6821","nonce":1642308925}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1642308925},{"type":"v1","addr":"172.21.15.28:6825","nonce":1642308925}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":1642308925},{"type":"v1","addr":"172.21.15.28:6823","nonce":1642308925}]},"public_addr":"172.21.15.28:6819/1642308925","cluster_addr":"172.21.15.28:6821/1642308925","heartbeat_back_addr":"172.21.15.28:6825/1642308925","heartbeat_front_addr":"172.21.15.28:6823/1642308925","state":["exists","up"]},{"osd":7,"uuid":"ca7a059f-0414-438e-b2d1-a10f348036be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":726062711},{"type":"v1","addr":"172.21.15.86:6817","nonce":726062711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":726062711},{"type":"v1","addr":"172.21.15.86:6819","nonce":726062711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":726062711},{"type":"v1","addr":"172.21.15.86:6823","nonce":726062711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":726062711},{"type":"v1","addr":"172.21.15.86:6821","nonce":726062711}]},"public_addr":"172.21.15.86:6817/726062711","cluster_addr":"172.21.15.86:6819/726062711","heartbeat_back_addr":"172.21.15.86:6823/726062711","heartbeat_front_addr":"172.21.15.86:6821/726062711","state":["exists","up"]},{"osd":8,"uuid":"92e3f41f-aebd-422e-a99e-555f7ba4aa53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":3007687277},{"type":"v1","addr":"172.21.15.186:6817","nonce":3007687277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":3007687277},{"type":"v1","addr":"172.21.15.186:6819","nonce":3007687277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":3007687277},{"type":"v1","addr":"172.21.15.186:6823","nonce":3007687277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":3007687277},{"type":"v1","addr":"172.21.15.186:6821","nonce":3007687277}]},"public_addr":"172.21.15.186:6817/3007687277","cluster_addr":"172.21.15.186:6819/3007687277","heartbeat_back_addr":"172.21.15.186:6823/3007687277","heartbeat_front_addr":"172.21.15.186:6821/3007687277","state":["exists","up"]},{"osd":9,"uuid":"38ff7d9d-ee85-404c-8c59-adc327e1ccfa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":136220472},{"type":"v1","addr":"172.21.15.28:6827","nonce":136220472}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":136220472},{"type":"v1","addr":"172.21.15.28:6829","nonce":136220472}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":136220472},{"type":"v1","addr":"172.21.15.28:6833","nonce":136220472}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":136220472},{"type":"v1","addr":"172.21.15.28:6831","nonce":136220472}]},"public_addr":"172.21.15.28:6827/136220472","cluster_addr":"172.21.15.28:6829/136220472","heartbeat_back_addr":"172.21.15.28:6833/136220472","heartbeat_front_addr":"172.21.15.28:6831/136220472","state":["exists","up"]},{"osd":10,"uuid":"5e9ff657-afa8-4bab-8491-f209038392b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1189968310},{"type":"v1","addr":"172.21.15.86:6825","nonce":1189968310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":1189968310},{"type":"v1","addr":"172.21.15.86:6827","nonce":1189968310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":1189968310},{"type":"v1","addr":"172.21.15.86:6831","nonce":1189968310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":1189968310},{"type":"v1","addr":"172.21.15.86:6829","nonce":1189968310}]},"public_addr":"172.21.15.86:6825/1189968310","cluster_addr":"172.21.15.86:6827/1189968310","heartbeat_back_addr":"172.21.15.86:6831/1189968310","heartbeat_front_addr":"172.21.15.86:6829/1189968310","state":["exists","up"]},{"osd":11,"uuid":"f112e3c6-65b6-4253-a966-62001db7d069","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3801668110},{"type":"v1","addr":"172.21.15.186:6825","nonce":3801668110}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3801668110},{"type":"v1","addr":"172.21.15.186:6827","nonce":3801668110}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3801668110},{"type":"v1","addr":"172.21.15.186:6831","nonce":3801668110}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3801668110},{"type":"v1","addr":"172.21.15.186:6829","nonce":3801668110}]},"public_addr":"172.21.15.186:6825/3801668110","cluster_addr":"172.21.15.186:6827/3801668110","heartbeat_back_addr":"172.21.15.186:6831/3801668110","heartbeat_front_addr":"172.21.15.186:6829/3801668110","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:00:40.483623+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:01:05.284385+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:01:28.645716+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:02:01.295092+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:02:26.486601+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:02:50.807478+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:03:24.756169+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:03:49.975173+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:04:13.772233+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:04:46.758555+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:05:12.331685+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:05:36.462548+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":2,"to":1}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.25","mappings":[{"from":5,"to":9}]},{"pgid":"3.27","mappings":[{"from":4,"to":8}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/1177009533":"2024-05-13T21:58:45.550459+0000","172.21.15.28:6800/3760499540":"2024-05-13T21:58:45.550459+0000","172.21.15.28:6801/3783742728":"2024-05-13T21:58:10.421564+0000","172.21.15.28:6800/3783742728":"2024-05-13T21:58:10.421564+0000","172.21.15.28:0/2229156260":"2024-05-13T21:58:45.550459+0000","172.21.15.28:0/1202140963":"2024-05-13T21:58:10.421564+0000","172.21.15.28:0/2924223784":"2024-05-13T21:58:45.550459+0000","172.21.15.28:6801/3760499540":"2024-05-13T21:58:45.550459+0000","172.21.15.28:0/1240027206":"2024-05-13T21:58:10.421564+0000","172.21.15.28:0/171417728":"2024-05-13T21:58:10.421564+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-12T22:08:42.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:42.634+0000 7f0f8ea00640 1 -- 172.21.15.28:0/2774140924 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f0f700a2b90 msgr2=0x7f0f700a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:42.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:42.634+0000 7f0f8ea00640 1 --2- 172.21.15.28:0/2774140924 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f0f700a2b90 0x7f0f700a5050 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f0f7402a930 tx=0x7f0f74030570 comp rx=0 tx=0).stop 2024-05-12T22:08:42.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:42.634+0000 7f0f8ea00640 1 -- 172.21.15.28:0/2774140924 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0f88107cb0 msgr2=0x7f0f8819e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:42.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:42.635+0000 7f0f8ea00640 1 --2- 172.21.15.28:0/2774140924 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0f88107cb0 0x7f0f8819e020 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f0f7c00e9c0 tx=0x7f0f7c00ee80 comp rx=0 tx=0).stop 2024-05-12T22:08:42.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:42.635+0000 7f0f8ea00640 1 -- 172.21.15.28:0/2774140924 shutdown_connections 2024-05-12T22:08:42.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:42.635+0000 7f0f8ea00640 1 --2- 172.21.15.28:0/2774140924 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f0f700a2b90 0x7f0f700a5050 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:42.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:42.635+0000 7f0f8ea00640 1 --2- 172.21.15.28:0/2774140924 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f88109860 0x7f0f881a5580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:42.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:42.635+0000 7f0f8ea00640 1 --2- 172.21.15.28:0/2774140924 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0f88108ea0 0x7f0f8819e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:42.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:42.635+0000 7f0f8ea00640 1 --2- 172.21.15.28:0/2774140924 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0f88107cb0 0x7f0f8819e020 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:42.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:42.635+0000 7f0f8ea00640 1 -- 172.21.15.28:0/2774140924 >> 172.21.15.28:0/2774140924 conn(0x7f0f88076c70 msgr2=0x7f0f8810c1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:08:42.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:42.635+0000 7f0f8ea00640 1 -- 172.21.15.28:0/2774140924 shutdown_connections 2024-05-12T22:08:42.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:42.635+0000 7f0f8ea00640 1 -- 172.21.15.28:0/2774140924 wait complete. 2024-05-12T22:08:42.657 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-12T22:01:32.292826+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-12T22:06:31.540263+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-12T22:06:32.906118+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-12T22:08:42.657 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-12T22:08:42.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:42.874+0000 7f77a7000640 1 -- 172.21.15.28:0/3531891736 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f77a0103970 msgr2=0x7f77a0103d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:42.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:42.874+0000 7f77a7000640 1 --2- 172.21.15.28:0/3531891736 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f77a0103970 0x7f77a0103d90 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f7794009900 tx=0x7f779402a410 comp rx=0 tx=0).stop 2024-05-12T22:08:42.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:42.874+0000 7f77a7000640 1 -- 172.21.15.28:0/3531891736 shutdown_connections 2024-05-12T22:08:42.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:42.874+0000 7f77a7000640 1 --2- 172.21.15.28:0/3531891736 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77a0105520 0x7f77a010bdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:42.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:42.874+0000 7f77a7000640 1 --2- 172.21.15.28:0/3531891736 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f77a0104b60 0x7f77a0104fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:42.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:42.874+0000 7f77a7000640 1 --2- 172.21.15.28:0/3531891736 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f77a0103970 0x7f77a0103d90 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:42.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:42.874+0000 7f77a7000640 1 -- 172.21.15.28:0/3531891736 >> 172.21.15.28:0/3531891736 conn(0x7f77a00feee0 msgr2=0x7f77a0101320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:08:42.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:42.874+0000 7f77a7000640 1 -- 172.21.15.28:0/3531891736 shutdown_connections 2024-05-12T22:08:42.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:42.874+0000 7f77a7000640 1 -- 172.21.15.28:0/3531891736 wait complete. 2024-05-12T22:08:42.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:42.876+0000 7f77a7000640 1 Processor -- start 2024-05-12T22:08:42.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:42.876+0000 7f77a7000640 1 -- start start 2024-05-12T22:08:42.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:42.876+0000 7f77a7000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f77a0103970 0x7f77a0199ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:42.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:42.876+0000 7f77a7000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f77a0104b60 0x7f77a019a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:42.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:42.876+0000 7f77a7000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77a0105520 0x7f77a01a1240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:42.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:42.876+0000 7f77a7000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f77a010f630 con 0x7f77a0105520 2024-05-12T22:08:42.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:42.876+0000 7f77a7000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f77a010f4b0 con 0x7f77a0103970 2024-05-12T22:08:42.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:42.876+0000 7f77a7000640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f77a010f7b0 con 0x7f77a0104b60 2024-05-12T22:08:42.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:42.876+0000 7f77a5c00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f77a0103970 0x7f77a0199ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:42.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:42.876+0000 7f77a5c00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f77a0103970 0x7f77a0199ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:44614/0 (socket says 172.21.15.28:44614) 2024-05-12T22:08:42.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:42.876+0000 7f77a5c00640 1 -- 172.21.15.28:0/152452818 learned_addr learned my addr 172.21.15.28:0/152452818 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:08:42.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:42.876+0000 7f77a6600640 1 --2- 172.21.15.28:0/152452818 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77a0105520 0x7f77a01a1240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:42.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:42.876+0000 7f77a5200640 1 --2- 172.21.15.28:0/152452818 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f77a0104b60 0x7f77a019a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:42.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:42.877+0000 7f77a5c00640 1 -- 172.21.15.28:0/152452818 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f77a0104b60 msgr2=0x7f77a019a1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:42.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:42.877+0000 7f77a5c00640 1 --2- 172.21.15.28:0/152452818 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f77a0104b60 0x7f77a019a1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:42.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:42.877+0000 7f77a5c00640 1 -- 172.21.15.28:0/152452818 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77a0105520 msgr2=0x7f77a01a1240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:42.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:42.877+0000 7f77a5c00640 1 --2- 172.21.15.28:0/152452818 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77a0105520 0x7f77a01a1240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:42.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:42.877+0000 7f77a5c00640 1 -- 172.21.15.28:0/152452818 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77a01a18e0 con 0x7f77a0103970 2024-05-12T22:08:42.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:42.877+0000 7f77a5c00640 1 --2- 172.21.15.28:0/152452818 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f77a0103970 0x7f77a0199ca0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f7794031440 tx=0x7f7794032ce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:08:42.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:42.877+0000 7f779e000640 1 -- 172.21.15.28:0/152452818 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f779403f030 con 0x7f77a0103970 2024-05-12T22:08:42.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:42.878+0000 7f77a7000640 1 -- 172.21.15.28:0/152452818 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77a01a1b70 con 0x7f77a0103970 2024-05-12T22:08:42.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:42.878+0000 7f779e000640 1 -- 172.21.15.28:0/152452818 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7794003ce0 con 0x7f77a0103970 2024-05-12T22:08:42.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:42.878+0000 7f779e000640 1 -- 172.21.15.28:0/152452818 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7794035070 con 0x7f77a0103970 2024-05-12T22:08:42.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:42.878+0000 7f77a7000640 1 -- 172.21.15.28:0/152452818 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77a01a2060 con 0x7f77a0103970 2024-05-12T22:08:42.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:42.879+0000 7f77a7000640 1 -- 172.21.15.28:0/152452818 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f77a010ac90 con 0x7f77a0103970 2024-05-12T22:08:42.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:42.882+0000 7f779e000640 1 -- 172.21.15.28:0/152452818 <== mon.2 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f7794042030 con 0x7f77a0103970 2024-05-12T22:08:42.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:42.883+0000 7f779e000640 1 --2- 172.21.15.28:0/152452818 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f77880a2b90 0x7f77880a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:42.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:42.883+0000 7f779e000640 1 -- 172.21.15.28:0/152452818 <== mon.2 v2:172.21.15.86:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f77940ef8c0 con 0x7f77a0103970 2024-05-12T22:08:42.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:42.884+0000 7f77a5200640 1 --2- 172.21.15.28:0/152452818 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f77880a2b90 0x7f77880a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:42.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:42.884+0000 7f77a5200640 1 --2- 172.21.15.28:0/152452818 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f77880a2b90 0x7f77880a5050 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f77a019b1a0 tx=0x7f7790005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:08:42.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:42.887+0000 7f779e000640 1 -- 172.21.15.28:0/152452818 <== mon.2 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f77940bc180 con 0x7f77a0103970 2024-05-12T22:08:43.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:43.092+0000 7f77a7000640 1 -- 172.21.15.28:0/152452818 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f77a0062430 con 0x7f77a0103970 2024-05-12T22:08:43.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:43.093+0000 7f779e000640 1 -- 172.21.15.28:0/152452818 <== mon.2 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) ==== 93+0+10 (secure 0 0 0) 0x7f77940c0480 con 0x7f77a0103970 2024-05-12T22:08:43.093 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-05-12T22:08:43.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:43.095+0000 7f77a7000640 1 -- 172.21.15.28:0/152452818 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f77880a2b90 msgr2=0x7f77880a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:43.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:43.095+0000 7f77a7000640 1 --2- 172.21.15.28:0/152452818 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f77880a2b90 0x7f77880a5050 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f77a019b1a0 tx=0x7f7790005f90 comp rx=0 tx=0).stop 2024-05-12T22:08:43.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:43.095+0000 7f77a7000640 1 -- 172.21.15.28:0/152452818 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f77a0103970 msgr2=0x7f77a0199ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:43.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:43.095+0000 7f77a7000640 1 --2- 172.21.15.28:0/152452818 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f77a0103970 0x7f77a0199ca0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f7794031440 tx=0x7f7794032ce0 comp rx=0 tx=0).stop 2024-05-12T22:08:43.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:43.096+0000 7f77a7000640 1 -- 172.21.15.28:0/152452818 shutdown_connections 2024-05-12T22:08:43.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:43.096+0000 7f77a7000640 1 --2- 172.21.15.28:0/152452818 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f77880a2b90 0x7f77880a5050 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:43.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:43.096+0000 7f77a7000640 1 --2- 172.21.15.28:0/152452818 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77a0105520 0x7f77a01a1240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:43.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:43.096+0000 7f77a7000640 1 --2- 172.21.15.28:0/152452818 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f77a0104b60 0x7f77a019a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:43.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:43.096+0000 7f77a7000640 1 --2- 172.21.15.28:0/152452818 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f77a0103970 0x7f77a0199ca0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:43.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:43.096+0000 7f77a7000640 1 -- 172.21.15.28:0/152452818 >> 172.21.15.28:0/152452818 conn(0x7f77a00feee0 msgr2=0x7f77a0107e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:08:43.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:43.096+0000 7f77a7000640 1 -- 172.21.15.28:0/152452818 shutdown_connections 2024-05-12T22:08:43.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:43.096+0000 7f77a7000640 1 -- 172.21.15.28:0/152452818 wait complete. 2024-05-12T22:08:43.117 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-12T22:08:43.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:43.339+0000 7f945ca00640 1 -- 172.21.15.28:0/3904832335 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9458108f60 msgr2=0x7f94581093e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:43.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:43.339+0000 7f945ca00640 1 --2- 172.21.15.28:0/3904832335 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9458108f60 0x7f94581093e0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f9440009860 tx=0x7f944002a340 comp rx=0 tx=0).stop 2024-05-12T22:08:43.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:43.340+0000 7f945ca00640 1 -- 172.21.15.28:0/3904832335 shutdown_connections 2024-05-12T22:08:43.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:43.340+0000 7f945ca00640 1 --2- 172.21.15.28:0/3904832335 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9458109920 0x7f94581101b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:43.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:43.340+0000 7f945ca00640 1 --2- 172.21.15.28:0/3904832335 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9458108f60 0x7f94581093e0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:43.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:43.340+0000 7f945ca00640 1 --2- 172.21.15.28:0/3904832335 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9458107d70 0x7f9458108190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:43.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:43.340+0000 7f945ca00640 1 -- 172.21.15.28:0/3904832335 >> 172.21.15.28:0/3904832335 conn(0x7f9458076d00 msgr2=0x7f9458079140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:08:43.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:43.340+0000 7f945ca00640 1 -- 172.21.15.28:0/3904832335 shutdown_connections 2024-05-12T22:08:43.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:43.340+0000 7f945ca00640 1 -- 172.21.15.28:0/3904832335 wait complete. 2024-05-12T22:08:43.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:43.341+0000 7f945ca00640 1 Processor -- start 2024-05-12T22:08:43.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:43.341+0000 7f945ca00640 1 -- start start 2024-05-12T22:08:43.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:43.342+0000 7f945ca00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9458107d70 0x7f945819e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:43.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:43.342+0000 7f945ca00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9458108f60 0x7f945819e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:43.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:43.342+0000 7f945ca00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9458109920 0x7f94581a5610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:43.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:43.342+0000 7f945ca00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f94581139f0 con 0x7f9458108f60 2024-05-12T22:08:43.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:43.342+0000 7f945ca00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f9458113870 con 0x7f9458109920 2024-05-12T22:08:43.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:43.342+0000 7f945ca00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f9458113b70 con 0x7f9458107d70 2024-05-12T22:08:43.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:43.342+0000 7f9457400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9458107d70 0x7f945819e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:43.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:43.342+0000 7f9457400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9458107d70 0x7f945819e060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:40024/0 (socket says 172.21.15.28:40024) 2024-05-12T22:08:43.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:43.342+0000 7f9457400640 1 -- 172.21.15.28:0/3350968597 learned_addr learned my addr 172.21.15.28:0/3350968597 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:08:43.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:43.342+0000 7f9456a00640 1 --2- 172.21.15.28:0/3350968597 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9458108f60 0x7f945819e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:43.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:43.342+0000 7f9457e00640 1 --2- 172.21.15.28:0/3350968597 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9458109920 0x7f94581a5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:43.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:43.343+0000 7f9457400640 1 -- 172.21.15.28:0/3350968597 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9458109920 msgr2=0x7f94581a5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:43.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:43.343+0000 7f9457400640 1 --2- 172.21.15.28:0/3350968597 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9458109920 0x7f94581a5610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:43.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:43.343+0000 7f9457400640 1 -- 172.21.15.28:0/3350968597 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9458108f60 msgr2=0x7f945819e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:43.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:43.343+0000 7f9457400640 1 --2- 172.21.15.28:0/3350968597 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9458108f60 0x7f945819e5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:43.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:43.343+0000 7f9457400640 1 -- 172.21.15.28:0/3350968597 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94581a5c80 con 0x7f9458107d70 2024-05-12T22:08:43.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:43.343+0000 7f9457400640 1 --2- 172.21.15.28:0/3350968597 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9458107d70 0x7f945819e060 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f944800e9f0 tx=0x7f944800edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:08:43.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:43.343+0000 7f944fe00640 1 -- 172.21.15.28:0/3350968597 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f944800ccc0 con 0x7f9458107d70 2024-05-12T22:08:43.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:43.343+0000 7f944fe00640 1 -- 172.21.15.28:0/3350968597 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f94480035f0 con 0x7f9458107d70 2024-05-12T22:08:43.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:43.343+0000 7f945ca00640 1 -- 172.21.15.28:0/3350968597 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94581a5f70 con 0x7f9458107d70 2024-05-12T22:08:43.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:43.343+0000 7f944fe00640 1 -- 172.21.15.28:0/3350968597 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f944800ce60 con 0x7f9458107d70 2024-05-12T22:08:43.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:43.344+0000 7f945ca00640 1 -- 172.21.15.28:0/3350968597 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94581a63d0 con 0x7f9458107d70 2024-05-12T22:08:43.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:43.345+0000 7f945ca00640 1 -- 172.21.15.28:0/3350968597 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f945810f090 con 0x7f9458107d70 2024-05-12T22:08:43.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:43.347+0000 7f944fe00640 1 -- 172.21.15.28:0/3350968597 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f94480037a0 con 0x7f9458107d70 2024-05-12T22:08:43.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:43.348+0000 7f944fe00640 1 --2- 172.21.15.28:0/3350968597 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f943c0a2c10 0x7f943c0a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:43.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:43.348+0000 7f944fe00640 1 -- 172.21.15.28:0/3350968597 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f94480cf270 con 0x7f9458107d70 2024-05-12T22:08:43.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:43.348+0000 7f9456a00640 1 --2- 172.21.15.28:0/3350968597 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f943c0a2c10 0x7f943c0a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:43.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:43.349+0000 7f9456a00640 1 --2- 172.21.15.28:0/3350968597 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f943c0a2c10 0x7f943c0a50d0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f9440009830 tx=0x7f9440030550 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:08:43.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:43.350+0000 7f944fe00640 1 -- 172.21.15.28:0/3350968597 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f944809c180 con 0x7f9458107d70 2024-05-12T22:08:43.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:43 smithi186 ceph-mon[34785]: pgmap v376: 97 pgs: 97 active+clean; 641 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:08:43.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:43 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/2774140924' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-12T22:08:43.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:43 smithi086 ceph-mon[34297]: pgmap v376: 97 pgs: 97 active+clean; 641 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:08:43.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:43 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/2774140924' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-12T22:08:43.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:43 smithi028 ceph-mon[29847]: pgmap v376: 97 pgs: 97 active+clean; 641 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:08:43.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:43 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/2774140924' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-12T22:08:43.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:43.553+0000 7f945ca00640 1 -- 172.21.15.28:0/3350968597 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f945802d350 con 0x7f9458107d70 2024-05-12T22:08:43.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:43.554+0000 7f944fe00640 1 -- 172.21.15.28:0/3350968597 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) ==== 104+0+11 (secure 0 0 0) 0x7f944809ba80 con 0x7f9458107d70 2024-05-12T22:08:43.555 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-05-12T22:08:43.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:43.557+0000 7f945ca00640 1 -- 172.21.15.28:0/3350968597 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f943c0a2c10 msgr2=0x7f943c0a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:43.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:43.557+0000 7f945ca00640 1 --2- 172.21.15.28:0/3350968597 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f943c0a2c10 0x7f943c0a50d0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f9440009830 tx=0x7f9440030550 comp rx=0 tx=0).stop 2024-05-12T22:08:43.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:43.557+0000 7f945ca00640 1 -- 172.21.15.28:0/3350968597 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9458107d70 msgr2=0x7f945819e060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:43.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:43.557+0000 7f945ca00640 1 --2- 172.21.15.28:0/3350968597 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9458107d70 0x7f945819e060 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f944800e9f0 tx=0x7f944800edf0 comp rx=0 tx=0).stop 2024-05-12T22:08:43.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:43.557+0000 7f945ca00640 1 -- 172.21.15.28:0/3350968597 shutdown_connections 2024-05-12T22:08:43.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:43.557+0000 7f945ca00640 1 --2- 172.21.15.28:0/3350968597 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f943c0a2c10 0x7f943c0a50d0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:43.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:43.557+0000 7f945ca00640 1 --2- 172.21.15.28:0/3350968597 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9458109920 0x7f94581a5610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:43.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:43.557+0000 7f945ca00640 1 --2- 172.21.15.28:0/3350968597 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9458108f60 0x7f945819e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:43.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:43.557+0000 7f945ca00640 1 --2- 172.21.15.28:0/3350968597 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9458107d70 0x7f945819e060 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:43.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:43.557+0000 7f945ca00640 1 -- 172.21.15.28:0/3350968597 >> 172.21.15.28:0/3350968597 conn(0x7f9458076d00 msgr2=0x7f945810c260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:08:43.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:43.557+0000 7f945ca00640 1 -- 172.21.15.28:0/3350968597 shutdown_connections 2024-05-12T22:08:43.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:43.557+0000 7f945ca00640 1 -- 172.21.15.28:0/3350968597 wait complete. 2024-05-12T22:08:43.578 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-12T22:08:43.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:43.796+0000 7f9d80a00640 1 -- 172.21.15.28:0/1214693210 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9d7c102950 msgr2=0x7f9d7c102dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:43.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:43.796+0000 7f9d80a00640 1 --2- 172.21.15.28:0/1214693210 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9d7c102950 0x7f9d7c102dd0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f9d64009900 tx=0x7f9d6402a3f0 comp rx=0 tx=0).stop 2024-05-12T22:08:43.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:43.796+0000 7f9d80a00640 1 -- 172.21.15.28:0/1214693210 shutdown_connections 2024-05-12T22:08:43.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:43.796+0000 7f9d80a00640 1 --2- 172.21.15.28:0/1214693210 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d7c103310 0x7f9d7c109ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:43.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:43.796+0000 7f9d80a00640 1 --2- 172.21.15.28:0/1214693210 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9d7c102950 0x7f9d7c102dd0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:43.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:43.796+0000 7f9d80a00640 1 --2- 172.21.15.28:0/1214693210 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9d7c101760 0x7f9d7c101b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:43.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:43.796+0000 7f9d80a00640 1 -- 172.21.15.28:0/1214693210 >> 172.21.15.28:0/1214693210 conn(0x7f9d7c0fccd0 msgr2=0x7f9d7c0ff110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:08:43.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:43.796+0000 7f9d80a00640 1 -- 172.21.15.28:0/1214693210 shutdown_connections 2024-05-12T22:08:43.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:43.796+0000 7f9d80a00640 1 -- 172.21.15.28:0/1214693210 wait complete. 2024-05-12T22:08:43.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:43.797+0000 7f9d80a00640 1 Processor -- start 2024-05-12T22:08:43.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:43.797+0000 7f9d80a00640 1 -- start start 2024-05-12T22:08:43.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:43.798+0000 7f9d80a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9d7c101760 0x7f9d7c199d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:43.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:43.798+0000 7f9d80a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d7c102950 0x7f9d7c19a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:43.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:43.798+0000 7f9d80a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9d7c103310 0x7f9d7c1a12b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:43.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:43.798+0000 7f9d80a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9d7c10d3e0 con 0x7f9d7c102950 2024-05-12T22:08:43.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:43.798+0000 7f9d80a00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f9d7c10d260 con 0x7f9d7c103310 2024-05-12T22:08:43.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:43.798+0000 7f9d80a00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f9d7c10d560 con 0x7f9d7c101760 2024-05-12T22:08:43.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:43.798+0000 7f9d73e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d7c102950 0x7f9d7c19a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:43.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:43.798+0000 7f9d73e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d7c102950 0x7f9d7c19a280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53570/0 (socket says 172.21.15.28:53570) 2024-05-12T22:08:43.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:43.798+0000 7f9d7a800640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9d7c101760 0x7f9d7c199d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:43.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:43.798+0000 7f9d73e00640 1 -- 172.21.15.28:0/1046468720 learned_addr learned my addr 172.21.15.28:0/1046468720 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:08:43.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:43.798+0000 7f9d7b200640 1 --2- 172.21.15.28:0/1046468720 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9d7c103310 0x7f9d7c1a12b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:43.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:43.798+0000 7f9d73e00640 1 -- 172.21.15.28:0/1046468720 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9d7c101760 msgr2=0x7f9d7c199d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:43.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:43.799+0000 7f9d73e00640 1 --2- 172.21.15.28:0/1046468720 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9d7c101760 0x7f9d7c199d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:43.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:43.799+0000 7f9d73e00640 1 -- 172.21.15.28:0/1046468720 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9d7c103310 msgr2=0x7f9d7c1a12b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:43.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:43.799+0000 7f9d73e00640 1 --2- 172.21.15.28:0/1046468720 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9d7c103310 0x7f9d7c1a12b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:43.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:43.799+0000 7f9d73e00640 1 -- 172.21.15.28:0/1046468720 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d7c1a1950 con 0x7f9d7c102950 2024-05-12T22:08:43.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:43.799+0000 7f9d73e00640 1 --2- 172.21.15.28:0/1046468720 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d7c102950 0x7f9d7c19a280 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f9d64009230 tx=0x7f9d640042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:08:43.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:43.799+0000 7f9d73400640 1 -- 172.21.15.28:0/1046468720 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d6403f030 con 0x7f9d7c102950 2024-05-12T22:08:43.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:43.799+0000 7f9d80a00640 1 -- 172.21.15.28:0/1046468720 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d7c1a1be0 con 0x7f9d7c102950 2024-05-12T22:08:43.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:43.799+0000 7f9d73400640 1 -- 172.21.15.28:0/1046468720 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9d64003ce0 con 0x7f9d7c102950 2024-05-12T22:08:43.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:43.799+0000 7f9d73400640 1 -- 172.21.15.28:0/1046468720 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d64035070 con 0x7f9d7c102950 2024-05-12T22:08:43.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:43.799+0000 7f9d80a00640 1 -- 172.21.15.28:0/1046468720 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d7c1a20d0 con 0x7f9d7c102950 2024-05-12T22:08:43.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:43.801+0000 7f9d80a00640 1 -- 172.21.15.28:0/1046468720 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d7c1a2300 con 0x7f9d7c102950 2024-05-12T22:08:43.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:43.802+0000 7f9d73400640 1 -- 172.21.15.28:0/1046468720 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f9d64042030 con 0x7f9d7c102950 2024-05-12T22:08:43.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:43.804+0000 7f9d73400640 1 --2- 172.21.15.28:0/1046468720 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f9d600a2b90 0x7f9d600a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:43.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:43.804+0000 7f9d73400640 1 -- 172.21.15.28:0/1046468720 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f9d640ef570 con 0x7f9d7c102950 2024-05-12T22:08:43.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:43.804+0000 7f9d7a800640 1 --2- 172.21.15.28:0/1046468720 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f9d600a2b90 0x7f9d600a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:43.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:43.805+0000 7f9d7a800640 1 --2- 172.21.15.28:0/1046468720 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f9d600a2b90 0x7f9d600a5050 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f9d7c19b210 tx=0x7f9d6c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:08:43.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:43.808+0000 7f9d73400640 1 -- 172.21.15.28:0/1046468720 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f9d640bc480 con 0x7f9d7c102950 2024-05-12T22:08:44.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:44.015+0000 7f9d80a00640 1 -- 172.21.15.28:0/1046468720 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f9d7c1a2690 con 0x7f9d7c102950 2024-05-12T22:08:44.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:44.016+0000 7f9d73400640 1 -- 172.21.15.28:0/1046468720 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) ==== 100+0+11 (secure 0 0 0) 0x7f9d640bbd80 con 0x7f9d7c102950 2024-05-12T22:08:44.016 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-05-12T22:08:44.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:44.019+0000 7f9d80a00640 1 -- 172.21.15.28:0/1046468720 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f9d600a2b90 msgr2=0x7f9d600a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:44.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:44.019+0000 7f9d80a00640 1 --2- 172.21.15.28:0/1046468720 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f9d600a2b90 0x7f9d600a5050 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f9d7c19b210 tx=0x7f9d6c005f90 comp rx=0 tx=0).stop 2024-05-12T22:08:44.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:44.019+0000 7f9d80a00640 1 -- 172.21.15.28:0/1046468720 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d7c102950 msgr2=0x7f9d7c19a280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:44.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:44.019+0000 7f9d80a00640 1 --2- 172.21.15.28:0/1046468720 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d7c102950 0x7f9d7c19a280 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f9d64009230 tx=0x7f9d640042f0 comp rx=0 tx=0).stop 2024-05-12T22:08:44.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:44.019+0000 7f9d80a00640 1 -- 172.21.15.28:0/1046468720 shutdown_connections 2024-05-12T22:08:44.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:44.019+0000 7f9d80a00640 1 --2- 172.21.15.28:0/1046468720 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f9d600a2b90 0x7f9d600a5050 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:44.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:44.019+0000 7f9d80a00640 1 --2- 172.21.15.28:0/1046468720 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9d7c103310 0x7f9d7c1a12b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:44.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:44.019+0000 7f9d80a00640 1 --2- 172.21.15.28:0/1046468720 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d7c102950 0x7f9d7c19a280 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:44.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:44.019+0000 7f9d80a00640 1 --2- 172.21.15.28:0/1046468720 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9d7c101760 0x7f9d7c199d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:44.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:44.019+0000 7f9d80a00640 1 -- 172.21.15.28:0/1046468720 >> 172.21.15.28:0/1046468720 conn(0x7f9d7c0fccd0 msgr2=0x7f9d7c073a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:08:44.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:44.019+0000 7f9d80a00640 1 -- 172.21.15.28:0/1046468720 shutdown_connections 2024-05-12T22:08:44.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:44.019+0000 7f9d80a00640 1 -- 172.21.15.28:0/1046468720 wait complete. 2024-05-12T22:08:44.040 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-12T22:08:44.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:44.256+0000 7f643fe00640 1 -- 172.21.15.28:0/4143905705 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6438105640 msgr2=0x7f643810bed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:44.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:44.256+0000 7f643fe00640 1 --2- 172.21.15.28:0/4143905705 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6438105640 0x7f643810bed0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f6430009be0 tx=0x7f643002a6d0 comp rx=0 tx=0).stop 2024-05-12T22:08:44.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:44.256+0000 7f643fe00640 1 -- 172.21.15.28:0/4143905705 shutdown_connections 2024-05-12T22:08:44.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:44.256+0000 7f643fe00640 1 --2- 172.21.15.28:0/4143905705 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6438105640 0x7f643810bed0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:44.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:44.256+0000 7f643fe00640 1 --2- 172.21.15.28:0/4143905705 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6438104c80 0x7f6438105100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:44.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:44.256+0000 7f643fe00640 1 --2- 172.21.15.28:0/4143905705 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6438103a90 0x7f6438103eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:44.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:44.256+0000 7f643fe00640 1 -- 172.21.15.28:0/4143905705 >> 172.21.15.28:0/4143905705 conn(0x7f64380ff020 msgr2=0x7f6438101440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:08:44.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:44.256+0000 7f643fe00640 1 -- 172.21.15.28:0/4143905705 shutdown_connections 2024-05-12T22:08:44.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:44.256+0000 7f643fe00640 1 -- 172.21.15.28:0/4143905705 wait complete. 2024-05-12T22:08:44.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:44.257+0000 7f643fe00640 1 Processor -- start 2024-05-12T22:08:44.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:44.258+0000 7f643fe00640 1 -- start start 2024-05-12T22:08:44.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:44.258+0000 7f643fe00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6438103a90 0x7f6438199d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:44.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:44.258+0000 7f643fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6438104c80 0x7f643819a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:44.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:44.258+0000 7f643fe00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6438105640 0x7f64381a1290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:44.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:44.258+0000 7f643fe00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f643810f6f0 con 0x7f6438104c80 2024-05-12T22:08:44.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:44.258+0000 7f643fe00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f643810f570 con 0x7f6438103a90 2024-05-12T22:08:44.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:44.258+0000 7f643fe00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f643810f870 con 0x7f6438105640 2024-05-12T22:08:44.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:44.258+0000 7f643ca00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6438104c80 0x7f643819a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:44.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:44.258+0000 7f643ca00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6438104c80 0x7f643819a260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53584/0 (socket says 172.21.15.28:53584) 2024-05-12T22:08:44.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:44.258+0000 7f643ca00640 1 -- 172.21.15.28:0/2898974136 learned_addr learned my addr 172.21.15.28:0/2898974136 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:08:44.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:44.258+0000 7f643d400640 1 --2- 172.21.15.28:0/2898974136 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6438103a90 0x7f6438199d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:44.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:44.258+0000 7f643de00640 1 --2- 172.21.15.28:0/2898974136 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6438105640 0x7f64381a1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:44.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:44.259+0000 7f643ca00640 1 -- 172.21.15.28:0/2898974136 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6438105640 msgr2=0x7f64381a1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:44.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:44.259+0000 7f643ca00640 1 --2- 172.21.15.28:0/2898974136 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6438105640 0x7f64381a1290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:44.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:44.259+0000 7f643ca00640 1 -- 172.21.15.28:0/2898974136 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6438103a90 msgr2=0x7f6438199d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:44.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:44.259+0000 7f643ca00640 1 --2- 172.21.15.28:0/2898974136 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6438103a90 0x7f6438199d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:44.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:44.259+0000 7f643ca00640 1 -- 172.21.15.28:0/2898974136 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64381a1930 con 0x7f6438104c80 2024-05-12T22:08:44.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:44.259+0000 7f643ca00640 1 --2- 172.21.15.28:0/2898974136 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6438104c80 0x7f643819a260 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f642400e9f0 tx=0x7f642400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:08:44.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:44.259+0000 7f6436000640 1 -- 172.21.15.28:0/2898974136 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f642400ccc0 con 0x7f6438104c80 2024-05-12T22:08:44.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:44.259+0000 7f6436000640 1 -- 172.21.15.28:0/2898974136 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6424010910 con 0x7f6438104c80 2024-05-12T22:08:44.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:44.259+0000 7f6436000640 1 -- 172.21.15.28:0/2898974136 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64240188a0 con 0x7f6438104c80 2024-05-12T22:08:44.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:44.259+0000 7f643fe00640 1 -- 172.21.15.28:0/2898974136 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64381a1c20 con 0x7f6438104c80 2024-05-12T22:08:44.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:44.260+0000 7f643fe00640 1 -- 172.21.15.28:0/2898974136 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64381a2120 con 0x7f6438104c80 2024-05-12T22:08:44.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:44.261+0000 7f643fe00640 1 -- 172.21.15.28:0/2898974136 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f643810ad50 con 0x7f6438104c80 2024-05-12T22:08:44.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:44.263+0000 7f6436000640 1 -- 172.21.15.28:0/2898974136 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f6424010ac0 con 0x7f6438104c80 2024-05-12T22:08:44.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:44.265+0000 7f6436000640 1 --2- 172.21.15.28:0/2898974136 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f64200a2c10 0x7f64200a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:44.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:44.265+0000 7f6436000640 1 -- 172.21.15.28:0/2898974136 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f64240cf4d0 con 0x7f6438104c80 2024-05-12T22:08:44.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:44.265+0000 7f643d400640 1 --2- 172.21.15.28:0/2898974136 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f64200a2c10 0x7f64200a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:44.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:44.266+0000 7f643d400640 1 --2- 172.21.15.28:0/2898974136 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f64200a2c10 0x7f64200a50d0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f643819b1f0 tx=0x7f642c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:08:44.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:44.269+0000 7f6436000640 1 -- 172.21.15.28:0/2898974136 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f642409c3e0 con 0x7f6438104c80 2024-05-12T22:08:44.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:44 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/152452818' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-12T22:08:44.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:44 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/3350968597' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-12T22:08:44.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:44 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/1046468720' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-12T22:08:44.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:44 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/152452818' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-12T22:08:44.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:44 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/3350968597' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-12T22:08:44.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:44 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/1046468720' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-12T22:08:44.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:44.517+0000 7f643fe00640 1 -- 172.21.15.28:0/2898974136 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f643802d1c0 con 0x7f6438104c80 2024-05-12T22:08:44.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:44.519+0000 7f6436000640 1 -- 172.21.15.28:0/2898974136 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11864 (secure 0 0 0) 0x7f642409bce0 con 0x7f6438104c80 2024-05-12T22:08:44.520 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-12T22:08:44.520 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1844141745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1844141745},{"type":"v1","addr":"172.21.15.186:6833","nonce":1844141745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6833/2960763758","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":2960763758},{"type":"v1","addr":"172.21.15.86:6833","nonce":2960763758}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24470,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/4131134219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":4131134219},{"type":"v1","addr":"172.21.15.28:6835","nonce":4131134219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24481,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/1727869064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1727869064},{"type":"v1","addr":"172.21.15.186:6835","nonce":1727869064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24482,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2938260704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2938260704},{"type":"v1","addr":"172.21.15.28:6837","nonce":2938260704}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24494,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3473700910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3473700910},{"type":"v1","addr":"172.21.15.86:6835","nonce":3473700910}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-12T22:06:33.426986+0000","modified":"2024-05-12T22:06:38.789246+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24581,"mds_1":24550,"mds_2":24532},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24499":{"gid":24499,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6837/3600809235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":3600809235},{"type":"v1","addr":"172.21.15.186:6837","nonce":3600809235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"i","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6839/2399610807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":2399610807},{"type":"v1","addr":"172.21.15.186:6839","nonce":2399610807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24530":{"gid":24530,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.86:6837/3782448884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":3782448884},{"type":"v1","addr":"172.21.15.86:6837","nonce":3782448884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24532":{"gid":24532,"name":"g","rank":2,"incarnation":21,"state":"up:active","state_seq":6,"addr":"172.21.15.28:6839/123892158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":123892158},{"type":"v1","addr":"172.21.15.28:6839","nonce":123892158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24550":{"gid":24550,"name":"a","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.28:6841/761183870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":761183870},{"type":"v1","addr":"172.21.15.28:6841","nonce":761183870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24581":{"gid":24581,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6839/1251999130","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6838","nonce":1251999130},{"type":"v1","addr":"172.21.15.86:6839","nonce":1251999130}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24581,"qdb_cluster":[24550,24532,24581]},"id":1}]} 2024-05-12T22:08:44.520 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 23 2024-05-12T22:08:44.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:44.521+0000 7f643fe00640 1 -- 172.21.15.28:0/2898974136 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f64200a2c10 msgr2=0x7f64200a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:44.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:44.521+0000 7f643fe00640 1 --2- 172.21.15.28:0/2898974136 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f64200a2c10 0x7f64200a50d0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f643819b1f0 tx=0x7f642c009260 comp rx=0 tx=0).stop 2024-05-12T22:08:44.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:44.521+0000 7f643fe00640 1 -- 172.21.15.28:0/2898974136 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6438104c80 msgr2=0x7f643819a260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:44.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:44.521+0000 7f643fe00640 1 --2- 172.21.15.28:0/2898974136 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6438104c80 0x7f643819a260 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f642400e9f0 tx=0x7f642400edf0 comp rx=0 tx=0).stop 2024-05-12T22:08:44.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:44.522+0000 7f643fe00640 1 -- 172.21.15.28:0/2898974136 shutdown_connections 2024-05-12T22:08:44.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:44.522+0000 7f643fe00640 1 --2- 172.21.15.28:0/2898974136 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f64200a2c10 0x7f64200a50d0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:44.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:44.522+0000 7f643fe00640 1 --2- 172.21.15.28:0/2898974136 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6438105640 0x7f64381a1290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:44.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:44.522+0000 7f643fe00640 1 --2- 172.21.15.28:0/2898974136 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6438104c80 0x7f643819a260 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:44.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:44.522+0000 7f643fe00640 1 --2- 172.21.15.28:0/2898974136 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6438103a90 0x7f6438199d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:44.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:44.522+0000 7f643fe00640 1 -- 172.21.15.28:0/2898974136 >> 172.21.15.28:0/2898974136 conn(0x7f64380ff020 msgr2=0x7f6438107f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:08:44.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:44.522+0000 7f643fe00640 1 -- 172.21.15.28:0/2898974136 shutdown_connections 2024-05-12T22:08:44.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:44.522+0000 7f643fe00640 1 -- 172.21.15.28:0/2898974136 wait complete. 2024-05-12T22:08:44.534 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:44 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/152452818' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-12T22:08:44.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:44 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/3350968597' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-12T22:08:44.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:44 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/1046468720' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-12T22:08:44.544 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-12T22:08:44.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:44.759+0000 7fbfeca00640 1 -- 172.21.15.28:0/2239924670 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbfe8074250 msgr2=0x7fbfe80746d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:44.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:44.759+0000 7fbfeca00640 1 --2- 172.21.15.28:0/2239924670 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbfe8074250 0x7fbfe80746d0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fbfd8009ca0 tx=0x7fbfd802a8c0 comp rx=0 tx=0).stop 2024-05-12T22:08:44.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:44.760+0000 7fbfeca00640 1 -- 172.21.15.28:0/2239924670 shutdown_connections 2024-05-12T22:08:44.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:44.760+0000 7fbfeca00640 1 --2- 172.21.15.28:0/2239924670 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbfe8109550 0x7fbfe810b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:44.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:44.760+0000 7fbfeca00640 1 --2- 172.21.15.28:0/2239924670 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbfe8074250 0x7fbfe80746d0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:44.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:44.760+0000 7fbfeca00640 1 --2- 172.21.15.28:0/2239924670 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbfe8075830 0x7fbfe8073c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:44.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:44.760+0000 7fbfeca00640 1 -- 172.21.15.28:0/2239924670 >> 172.21.15.28:0/2239924670 conn(0x7fbfe80fcc10 msgr2=0x7fbfe80ff050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:08:44.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:44.760+0000 7fbfeca00640 1 -- 172.21.15.28:0/2239924670 shutdown_connections 2024-05-12T22:08:44.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:44.760+0000 7fbfeca00640 1 -- 172.21.15.28:0/2239924670 wait complete. 2024-05-12T22:08:44.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:44.761+0000 7fbfeca00640 1 Processor -- start 2024-05-12T22:08:44.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:44.761+0000 7fbfeca00640 1 -- start start 2024-05-12T22:08:44.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:44.761+0000 7fbfeca00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbfe8074250 0x7fbfe8199b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:44.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:44.761+0000 7fbfeca00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbfe8075830 0x7fbfe819a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:44.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:44.761+0000 7fbfeca00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbfe8109550 0x7fbfe81a1130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:44.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:44.761+0000 7fbfeca00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbfe810f550 con 0x7fbfe8074250 2024-05-12T22:08:44.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:44.761+0000 7fbfeca00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fbfe810f3d0 con 0x7fbfe8075830 2024-05-12T22:08:44.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:44.761+0000 7fbfeca00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fbfe810f6d0 con 0x7fbfe8109550 2024-05-12T22:08:44.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:44.761+0000 7fbfe7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbfe8074250 0x7fbfe8199b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:44.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:44.761+0000 7fbfe7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbfe8074250 0x7fbfe8199b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53600/0 (socket says 172.21.15.28:53600) 2024-05-12T22:08:44.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:44.762+0000 7fbfe7400640 1 -- 172.21.15.28:0/3556866294 learned_addr learned my addr 172.21.15.28:0/3556866294 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:08:44.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:44.762+0000 7fbfe7e00640 1 --2- 172.21.15.28:0/3556866294 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbfe8109550 0x7fbfe81a1130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:44.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:44.762+0000 7fbfe6a00640 1 --2- 172.21.15.28:0/3556866294 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbfe8075830 0x7fbfe819a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:44.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:44.762+0000 7fbfe7400640 1 -- 172.21.15.28:0/3556866294 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbfe8109550 msgr2=0x7fbfe81a1130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:44.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:44.762+0000 7fbfe7400640 1 --2- 172.21.15.28:0/3556866294 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbfe8109550 0x7fbfe81a1130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:44.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:44.762+0000 7fbfe7400640 1 -- 172.21.15.28:0/3556866294 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbfe8075830 msgr2=0x7fbfe819a0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:44.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:44.762+0000 7fbfe7400640 1 --2- 172.21.15.28:0/3556866294 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbfe8075830 0x7fbfe819a0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:44.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:44.762+0000 7fbfe7400640 1 -- 172.21.15.28:0/3556866294 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfe81a17a0 con 0x7fbfe8074250 2024-05-12T22:08:44.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:44.762+0000 7fbfe7400640 1 --2- 172.21.15.28:0/3556866294 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbfe8074250 0x7fbfe8199b80 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fbfd000e9f0 tx=0x7fbfd000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:08:44.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:44.762+0000 7fbfdfe00640 1 -- 172.21.15.28:0/3556866294 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfd000cd80 con 0x7fbfe8074250 2024-05-12T22:08:44.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:44.763+0000 7fbfeca00640 1 -- 172.21.15.28:0/3556866294 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbfe81a1a90 con 0x7fbfe8074250 2024-05-12T22:08:44.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:44.763+0000 7fbfdfe00640 1 -- 172.21.15.28:0/3556866294 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbfd0010910 con 0x7fbfe8074250 2024-05-12T22:08:44.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:44.763+0000 7fbfdfe00640 1 -- 172.21.15.28:0/3556866294 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfd00188f0 con 0x7fbfe8074250 2024-05-12T22:08:44.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:44.763+0000 7fbfeca00640 1 -- 172.21.15.28:0/3556866294 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbfe81a2000 con 0x7fbfe8074250 2024-05-12T22:08:44.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:44.764+0000 7fbfeca00640 1 -- 172.21.15.28:0/3556866294 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbfe81082d0 con 0x7fbfe8074250 2024-05-12T22:08:44.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:44.766+0000 7fbfdfe00640 1 -- 172.21.15.28:0/3556866294 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fbfd0010ac0 con 0x7fbfe8074250 2024-05-12T22:08:44.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:44.767+0000 7fbfdfe00640 1 --2- 172.21.15.28:0/3556866294 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fbfcc0a2b90 0x7fbfcc0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:44.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:44.767+0000 7fbfdfe00640 1 -- 172.21.15.28:0/3556866294 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7fbfd00cf440 con 0x7fbfe8074250 2024-05-12T22:08:44.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:44.767+0000 7fbfe6a00640 1 --2- 172.21.15.28:0/3556866294 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fbfcc0a2b90 0x7fbfcc0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:44.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:44.768+0000 7fbfe6a00640 1 --2- 172.21.15.28:0/3556866294 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fbfcc0a2b90 0x7fbfcc0a5050 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fbfd8005e30 tx=0x7fbfd8005f50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:08:44.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:44.771+0000 7fbfdfe00640 1 -- 172.21.15.28:0/3556866294 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fbfd009bd00 con 0x7fbfe8074250 2024-05-12T22:08:44.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:44.977+0000 7fbfeca00640 1 -- 172.21.15.28:0/3556866294 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbfe8062460 con 0x7fbfe8074250 2024-05-12T22:08:44.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:44.981+0000 7fbfdfe00640 1 -- 172.21.15.28:0/3556866294 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22595 (secure 0 0 0) 0x7fbfd00a0000 con 0x7fbfe8074250 2024-05-12T22:08:44.982 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-12T22:08:44.983 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":88,"fsid":"6b92e7b4-10aa-11ef-bc98-c7b262605968","created":"2024-05-12T21:57:37.276782+0000","modified":"2024-05-12T22:06:46.768798+0000","last_up_change":"2024-05-12T22:05:37.929562+0000","last_in_change":"2024-05-12T22:05:18.750768+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-12T22:01:32.292826+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-12T22:06:31.540263+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-12T22:06:32.906118+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6992ce0a-844b-40e3-8ed1-181c8474dccf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":906003882},{"type":"v1","addr":"172.21.15.28:6803","nonce":906003882}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":906003882},{"type":"v1","addr":"172.21.15.28:6805","nonce":906003882}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":906003882},{"type":"v1","addr":"172.21.15.28:6809","nonce":906003882}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":906003882},{"type":"v1","addr":"172.21.15.28:6807","nonce":906003882}]},"public_addr":"172.21.15.28:6803/906003882","cluster_addr":"172.21.15.28:6805/906003882","heartbeat_back_addr":"172.21.15.28:6809/906003882","heartbeat_front_addr":"172.21.15.28:6807/906003882","state":["exists","up"]},{"osd":1,"uuid":"ced20c96-c481-4d48-90d1-6b43228959ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":3354141819},{"type":"v1","addr":"172.21.15.86:6801","nonce":3354141819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":3354141819},{"type":"v1","addr":"172.21.15.86:6803","nonce":3354141819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":3354141819},{"type":"v1","addr":"172.21.15.86:6807","nonce":3354141819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":3354141819},{"type":"v1","addr":"172.21.15.86:6805","nonce":3354141819}]},"public_addr":"172.21.15.86:6801/3354141819","cluster_addr":"172.21.15.86:6803/3354141819","heartbeat_back_addr":"172.21.15.86:6807/3354141819","heartbeat_front_addr":"172.21.15.86:6805/3354141819","state":["exists","up"]},{"osd":2,"uuid":"54a2f221-fa65-4795-9f5e-e660cc7c0098","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":906459365},{"type":"v1","addr":"172.21.15.186:6801","nonce":906459365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":906459365},{"type":"v1","addr":"172.21.15.186:6803","nonce":906459365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":906459365},{"type":"v1","addr":"172.21.15.186:6807","nonce":906459365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":906459365},{"type":"v1","addr":"172.21.15.186:6805","nonce":906459365}]},"public_addr":"172.21.15.186:6801/906459365","cluster_addr":"172.21.15.186:6803/906459365","heartbeat_back_addr":"172.21.15.186:6807/906459365","heartbeat_front_addr":"172.21.15.186:6805/906459365","state":["exists","up"]},{"osd":3,"uuid":"86836737-ac9a-487b-93f2-7fcd3c406960","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":4182884436},{"type":"v1","addr":"172.21.15.28:6811","nonce":4182884436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":4182884436},{"type":"v1","addr":"172.21.15.28:6813","nonce":4182884436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":4182884436},{"type":"v1","addr":"172.21.15.28:6817","nonce":4182884436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":4182884436},{"type":"v1","addr":"172.21.15.28:6815","nonce":4182884436}]},"public_addr":"172.21.15.28:6811/4182884436","cluster_addr":"172.21.15.28:6813/4182884436","heartbeat_back_addr":"172.21.15.28:6817/4182884436","heartbeat_front_addr":"172.21.15.28:6815/4182884436","state":["exists","up"]},{"osd":4,"uuid":"d162ebcd-83c0-4fd5-9925-28b2cecea935","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2596998708},{"type":"v1","addr":"172.21.15.86:6809","nonce":2596998708}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2596998708},{"type":"v1","addr":"172.21.15.86:6811","nonce":2596998708}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2596998708},{"type":"v1","addr":"172.21.15.86:6815","nonce":2596998708}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2596998708},{"type":"v1","addr":"172.21.15.86:6813","nonce":2596998708}]},"public_addr":"172.21.15.86:6809/2596998708","cluster_addr":"172.21.15.86:6811/2596998708","heartbeat_back_addr":"172.21.15.86:6815/2596998708","heartbeat_front_addr":"172.21.15.86:6813/2596998708","state":["exists","up"]},{"osd":5,"uuid":"c794336a-a105-425c-b6c7-a32b90c05bb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":1944205756},{"type":"v1","addr":"172.21.15.186:6809","nonce":1944205756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":1944205756},{"type":"v1","addr":"172.21.15.186:6811","nonce":1944205756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":1944205756},{"type":"v1","addr":"172.21.15.186:6815","nonce":1944205756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":1944205756},{"type":"v1","addr":"172.21.15.186:6813","nonce":1944205756}]},"public_addr":"172.21.15.186:6809/1944205756","cluster_addr":"172.21.15.186:6811/1944205756","heartbeat_back_addr":"172.21.15.186:6815/1944205756","heartbeat_front_addr":"172.21.15.186:6813/1944205756","state":["exists","up"]},{"osd":6,"uuid":"516932ac-af00-45b7-ad3c-d6dec7db442f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":1642308925},{"type":"v1","addr":"172.21.15.28:6819","nonce":1642308925}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":1642308925},{"type":"v1","addr":"172.21.15.28:6821","nonce":1642308925}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1642308925},{"type":"v1","addr":"172.21.15.28:6825","nonce":1642308925}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":1642308925},{"type":"v1","addr":"172.21.15.28:6823","nonce":1642308925}]},"public_addr":"172.21.15.28:6819/1642308925","cluster_addr":"172.21.15.28:6821/1642308925","heartbeat_back_addr":"172.21.15.28:6825/1642308925","heartbeat_front_addr":"172.21.15.28:6823/1642308925","state":["exists","up"]},{"osd":7,"uuid":"ca7a059f-0414-438e-b2d1-a10f348036be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":726062711},{"type":"v1","addr":"172.21.15.86:6817","nonce":726062711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":726062711},{"type":"v1","addr":"172.21.15.86:6819","nonce":726062711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":726062711},{"type":"v1","addr":"172.21.15.86:6823","nonce":726062711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":726062711},{"type":"v1","addr":"172.21.15.86:6821","nonce":726062711}]},"public_addr":"172.21.15.86:6817/726062711","cluster_addr":"172.21.15.86:6819/726062711","heartbeat_back_addr":"172.21.15.86:6823/726062711","heartbeat_front_addr":"172.21.15.86:6821/726062711","state":["exists","up"]},{"osd":8,"uuid":"92e3f41f-aebd-422e-a99e-555f7ba4aa53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":3007687277},{"type":"v1","addr":"172.21.15.186:6817","nonce":3007687277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":3007687277},{"type":"v1","addr":"172.21.15.186:6819","nonce":3007687277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":3007687277},{"type":"v1","addr":"172.21.15.186:6823","nonce":3007687277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":3007687277},{"type":"v1","addr":"172.21.15.186:6821","nonce":3007687277}]},"public_addr":"172.21.15.186:6817/3007687277","cluster_addr":"172.21.15.186:6819/3007687277","heartbeat_back_addr":"172.21.15.186:6823/3007687277","heartbeat_front_addr":"172.21.15.186:6821/3007687277","state":["exists","up"]},{"osd":9,"uuid":"38ff7d9d-ee85-404c-8c59-adc327e1ccfa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":136220472},{"type":"v1","addr":"172.21.15.28:6827","nonce":136220472}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":136220472},{"type":"v1","addr":"172.21.15.28:6829","nonce":136220472}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":136220472},{"type":"v1","addr":"172.21.15.28:6833","nonce":136220472}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":136220472},{"type":"v1","addr":"172.21.15.28:6831","nonce":136220472}]},"public_addr":"172.21.15.28:6827/136220472","cluster_addr":"172.21.15.28:6829/136220472","heartbeat_back_addr":"172.21.15.28:6833/136220472","heartbeat_front_addr":"172.21.15.28:6831/136220472","state":["exists","up"]},{"osd":10,"uuid":"5e9ff657-afa8-4bab-8491-f209038392b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1189968310},{"type":"v1","addr":"172.21.15.86:6825","nonce":1189968310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":1189968310},{"type":"v1","addr":"172.21.15.86:6827","nonce":1189968310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":1189968310},{"type":"v1","addr":"172.21.15.86:6831","nonce":1189968310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":1189968310},{"type":"v1","addr":"172.21.15.86:6829","nonce":1189968310}]},"public_addr":"172.21.15.86:6825/1189968310","cluster_addr":"172.21.15.86:6827/1189968310","heartbeat_back_addr":"172.21.15.86:6831/1189968310","heartbeat_front_addr":"172.21.15.86:6829/1189968310","state":["exists","up"]},{"osd":11,"uuid":"f112e3c6-65b6-4253-a966-62001db7d069","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3801668110},{"type":"v1","addr":"172.21.15.186:6825","nonce":3801668110}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3801668110},{"type":"v1","addr":"172.21.15.186:6827","nonce":3801668110}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3801668110},{"type":"v1","addr":"172.21.15.186:6831","nonce":3801668110}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3801668110},{"type":"v1","addr":"172.21.15.186:6829","nonce":3801668110}]},"public_addr":"172.21.15.186:6825/3801668110","cluster_addr":"172.21.15.186:6827/3801668110","heartbeat_back_addr":"172.21.15.186:6831/3801668110","heartbeat_front_addr":"172.21.15.186:6829/3801668110","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:00:40.483623+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:01:05.284385+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:01:28.645716+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:02:01.295092+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:02:26.486601+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:02:50.807478+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:03:24.756169+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:03:49.975173+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:04:13.772233+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:04:46.758555+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:05:12.331685+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:05:36.462548+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":2,"to":1}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.25","mappings":[{"from":5,"to":9}]},{"pgid":"3.27","mappings":[{"from":4,"to":8}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/1177009533":"2024-05-13T21:58:45.550459+0000","172.21.15.28:6800/3760499540":"2024-05-13T21:58:45.550459+0000","172.21.15.28:6801/3783742728":"2024-05-13T21:58:10.421564+0000","172.21.15.28:6800/3783742728":"2024-05-13T21:58:10.421564+0000","172.21.15.28:0/2229156260":"2024-05-13T21:58:45.550459+0000","172.21.15.28:0/1202140963":"2024-05-13T21:58:10.421564+0000","172.21.15.28:0/2924223784":"2024-05-13T21:58:45.550459+0000","172.21.15.28:6801/3760499540":"2024-05-13T21:58:45.550459+0000","172.21.15.28:0/1240027206":"2024-05-13T21:58:10.421564+0000","172.21.15.28:0/171417728":"2024-05-13T21:58:10.421564+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-12T22:08:44.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:44.984+0000 7fbfeca00640 1 -- 172.21.15.28:0/3556866294 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fbfcc0a2b90 msgr2=0x7fbfcc0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:44.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:44.984+0000 7fbfeca00640 1 --2- 172.21.15.28:0/3556866294 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fbfcc0a2b90 0x7fbfcc0a5050 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fbfd8005e30 tx=0x7fbfd8005f50 comp rx=0 tx=0).stop 2024-05-12T22:08:44.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:44.984+0000 7fbfeca00640 1 -- 172.21.15.28:0/3556866294 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbfe8074250 msgr2=0x7fbfe8199b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:44.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:44.984+0000 7fbfeca00640 1 --2- 172.21.15.28:0/3556866294 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbfe8074250 0x7fbfe8199b80 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fbfd000e9f0 tx=0x7fbfd000eeb0 comp rx=0 tx=0).stop 2024-05-12T22:08:44.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:44.984+0000 7fbfeca00640 1 -- 172.21.15.28:0/3556866294 shutdown_connections 2024-05-12T22:08:44.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:44.984+0000 7fbfeca00640 1 --2- 172.21.15.28:0/3556866294 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7fbfcc0a2b90 0x7fbfcc0a5050 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:44.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:44.984+0000 7fbfeca00640 1 --2- 172.21.15.28:0/3556866294 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbfe8109550 0x7fbfe81a1130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:44.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:44.984+0000 7fbfeca00640 1 --2- 172.21.15.28:0/3556866294 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbfe8075830 0x7fbfe819a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:44.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:44.984+0000 7fbfeca00640 1 --2- 172.21.15.28:0/3556866294 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbfe8074250 0x7fbfe8199b80 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:44.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:44.984+0000 7fbfeca00640 1 -- 172.21.15.28:0/3556866294 >> 172.21.15.28:0/3556866294 conn(0x7fbfe80fcc10 msgr2=0x7fbfe81054a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:08:44.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:44.984+0000 7fbfeca00640 1 -- 172.21.15.28:0/3556866294 shutdown_connections 2024-05-12T22:08:44.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:44.984+0000 7fbfeca00640 1 -- 172.21.15.28:0/3556866294 wait complete. 2024-05-12T22:08:45.005 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-12T22:01:32.292826+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-12T22:06:31.540263+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-12T22:06:32.906118+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-12T22:08:45.005 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-12T22:08:45.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:45.217+0000 7f6e87200640 1 -- 172.21.15.28:0/4082151183 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e88103970 msgr2=0x7f6e88103d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:45.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:45.217+0000 7f6e87200640 1 --2- 172.21.15.28:0/4082151183 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e88103970 0x7f6e88103d90 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f6e78009900 tx=0x7f6e7802a410 comp rx=0 tx=0).stop 2024-05-12T22:08:45.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:45.217+0000 7f6e87200640 1 -- 172.21.15.28:0/4082151183 shutdown_connections 2024-05-12T22:08:45.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:45.217+0000 7f6e87200640 1 --2- 172.21.15.28:0/4082151183 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6e88105520 0x7f6e8810bdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:45.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:45.217+0000 7f6e87200640 1 --2- 172.21.15.28:0/4082151183 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6e88104b60 0x7f6e88104fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:45.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:45.217+0000 7f6e87200640 1 --2- 172.21.15.28:0/4082151183 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e88103970 0x7f6e88103d90 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:45.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:45.217+0000 7f6e87200640 1 -- 172.21.15.28:0/4082151183 >> 172.21.15.28:0/4082151183 conn(0x7f6e880feee0 msgr2=0x7f6e88101320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:08:45.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:45.217+0000 7f6e87200640 1 -- 172.21.15.28:0/4082151183 shutdown_connections 2024-05-12T22:08:45.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:45.217+0000 7f6e87200640 1 -- 172.21.15.28:0/4082151183 wait complete. 2024-05-12T22:08:45.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:45.218+0000 7f6e87200640 1 Processor -- start 2024-05-12T22:08:45.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:45.218+0000 7f6e87200640 1 -- start start 2024-05-12T22:08:45.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:45.218+0000 7f6e87200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e88103970 0x7f6e88199c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:45.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:45.218+0000 7f6e87200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6e88104b60 0x7f6e8819a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:45.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:45.218+0000 7f6e87200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6e88105520 0x7f6e881a1220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:45.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:45.218+0000 7f6e87200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6e8810f610 con 0x7f6e88103970 2024-05-12T22:08:45.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:45.218+0000 7f6e87200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f6e8810f490 con 0x7f6e88104b60 2024-05-12T22:08:45.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:45.218+0000 7f6e87200640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f6e8810f790 con 0x7f6e88105520 2024-05-12T22:08:45.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:45.219+0000 7f6e85e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e88103970 0x7f6e88199c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:45.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:45.219+0000 7f6e85e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e88103970 0x7f6e88199c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53626/0 (socket says 172.21.15.28:53626) 2024-05-12T22:08:45.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:45.219+0000 7f6e85e00640 1 -- 172.21.15.28:0/1581316302 learned_addr learned my addr 172.21.15.28:0/1581316302 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:08:45.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:45.219+0000 7f6e86800640 1 --2- 172.21.15.28:0/1581316302 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6e88105520 0x7f6e881a1220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:45.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:45.219+0000 7f6e85400640 1 --2- 172.21.15.28:0/1581316302 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6e88104b60 0x7f6e8819a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:45.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:45.219+0000 7f6e86800640 1 -- 172.21.15.28:0/1581316302 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6e88104b60 msgr2=0x7f6e8819a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:45.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:45.219+0000 7f6e86800640 1 --2- 172.21.15.28:0/1581316302 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6e88104b60 0x7f6e8819a1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:45.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:45.219+0000 7f6e86800640 1 -- 172.21.15.28:0/1581316302 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e88103970 msgr2=0x7f6e88199c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:45.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:45.219+0000 7f6e86800640 1 --2- 172.21.15.28:0/1581316302 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e88103970 0x7f6e88199c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:45.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:45.219+0000 7f6e86800640 1 -- 172.21.15.28:0/1581316302 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e881a18c0 con 0x7f6e88105520 2024-05-12T22:08:45.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:45.219+0000 7f6e85400640 1 --2- 172.21.15.28:0/1581316302 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6e88104b60 0x7f6e8819a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:08:45.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:45.219+0000 7f6e85e00640 1 --2- 172.21.15.28:0/1581316302 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e88103970 0x7f6e88199c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:08:45.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:45.220+0000 7f6e86800640 1 --2- 172.21.15.28:0/1581316302 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6e88105520 0x7f6e881a1220 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f6e8000ec50 tx=0x7f6e8000c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:08:45.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:45.220+0000 7f6e7ea00640 1 -- 172.21.15.28:0/1581316302 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e80003b70 con 0x7f6e88105520 2024-05-12T22:08:45.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:45.220+0000 7f6e87200640 1 -- 172.21.15.28:0/1581316302 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e881a1bb0 con 0x7f6e88105520 2024-05-12T22:08:45.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:45.220+0000 7f6e7ea00640 1 -- 172.21.15.28:0/1581316302 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6e80003d10 con 0x7f6e88105520 2024-05-12T22:08:45.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:45.220+0000 7f6e7ea00640 1 -- 172.21.15.28:0/1581316302 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e80010890 con 0x7f6e88105520 2024-05-12T22:08:45.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:45.220+0000 7f6e87200640 1 -- 172.21.15.28:0/1581316302 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e881a20b0 con 0x7f6e88105520 2024-05-12T22:08:45.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:45.221+0000 7f6e87200640 1 -- 172.21.15.28:0/1581316302 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e8810ac90 con 0x7f6e88105520 2024-05-12T22:08:45.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:45.223+0000 7f6e7ea00640 1 -- 172.21.15.28:0/1581316302 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f6e80003fc0 con 0x7f6e88105520 2024-05-12T22:08:45.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:45.224+0000 7f6e7ea00640 1 --2- 172.21.15.28:0/1581316302 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f6e6c0a2b90 0x7f6e6c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:45.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:45.225+0000 7f6e7ea00640 1 -- 172.21.15.28:0/1581316302 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f6e800cf640 con 0x7f6e88105520 2024-05-12T22:08:45.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:45.225+0000 7f6e85e00640 1 --2- 172.21.15.28:0/1581316302 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f6e6c0a2b90 0x7f6e6c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:45.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:45.226+0000 7f6e85e00640 1 --2- 172.21.15.28:0/1581316302 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f6e6c0a2b90 0x7f6e6c0a5050 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f6e78006010 tx=0x7f6e780315b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:08:45.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:45.229+0000 7f6e7ea00640 1 -- 172.21.15.28:0/1581316302 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f6e8009c550 con 0x7f6e88105520 2024-05-12T22:08:45.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:45 smithi028 ceph-mon[29847]: pgmap v377: 97 pgs: 97 active+clean; 641 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-12T22:08:45.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:45 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/2898974136' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:08:45.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:45 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/3556866294' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-12T22:08:45.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:45.423+0000 7f6e87200640 1 -- 172.21.15.28:0/1581316302 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f6e88062430 con 0x7f6e88105520 2024-05-12T22:08:45.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:45.424+0000 7f6e7ea00640 1 -- 172.21.15.28:0/1581316302 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) ==== 93+0+10 (secure 0 0 0) 0x7f6e8009be50 con 0x7f6e88105520 2024-05-12T22:08:45.424 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-05-12T22:08:45.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:45.426+0000 7f6e87200640 1 -- 172.21.15.28:0/1581316302 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f6e6c0a2b90 msgr2=0x7f6e6c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:45.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:45.426+0000 7f6e87200640 1 --2- 172.21.15.28:0/1581316302 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f6e6c0a2b90 0x7f6e6c0a5050 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f6e78006010 tx=0x7f6e780315b0 comp rx=0 tx=0).stop 2024-05-12T22:08:45.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:45.426+0000 7f6e87200640 1 -- 172.21.15.28:0/1581316302 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6e88105520 msgr2=0x7f6e881a1220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:45.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:45.426+0000 7f6e87200640 1 --2- 172.21.15.28:0/1581316302 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6e88105520 0x7f6e881a1220 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f6e8000ec50 tx=0x7f6e8000c6a0 comp rx=0 tx=0).stop 2024-05-12T22:08:45.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:45.426+0000 7f6e87200640 1 -- 172.21.15.28:0/1581316302 shutdown_connections 2024-05-12T22:08:45.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:45.426+0000 7f6e87200640 1 --2- 172.21.15.28:0/1581316302 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f6e6c0a2b90 0x7f6e6c0a5050 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:45.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:45.426+0000 7f6e87200640 1 --2- 172.21.15.28:0/1581316302 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6e88105520 0x7f6e881a1220 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:45.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:45.426+0000 7f6e87200640 1 --2- 172.21.15.28:0/1581316302 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6e88104b60 0x7f6e8819a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:45.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:45.426+0000 7f6e87200640 1 --2- 172.21.15.28:0/1581316302 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e88103970 0x7f6e88199c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:45.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:45.426+0000 7f6e87200640 1 -- 172.21.15.28:0/1581316302 >> 172.21.15.28:0/1581316302 conn(0x7f6e880feee0 msgr2=0x7f6e88107e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:08:45.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:45.426+0000 7f6e87200640 1 -- 172.21.15.28:0/1581316302 shutdown_connections 2024-05-12T22:08:45.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:45.426+0000 7f6e87200640 1 -- 172.21.15.28:0/1581316302 wait complete. 2024-05-12T22:08:45.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:45 smithi186 ceph-mon[34785]: pgmap v377: 97 pgs: 97 active+clean; 641 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-12T22:08:45.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:45 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/2898974136' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:08:45.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:45 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/3556866294' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-12T22:08:45.445 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-12T22:08:45.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:45 smithi086 ceph-mon[34297]: pgmap v377: 97 pgs: 97 active+clean; 641 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-12T22:08:45.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:45 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/2898974136' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:08:45.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:45 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/3556866294' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-12T22:08:45.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:45.655+0000 7efd23a00640 1 -- 172.21.15.28:0/4118183883 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efd1c074290 msgr2=0x7efd1c074710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:45.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:45.655+0000 7efd23a00640 1 --2- 172.21.15.28:0/4118183883 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efd1c074290 0x7efd1c074710 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7efd08009860 tx=0x7efd0802a340 comp rx=0 tx=0).stop 2024-05-12T22:08:45.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:45.656+0000 7efd23a00640 1 -- 172.21.15.28:0/4118183883 shutdown_connections 2024-05-12T22:08:45.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:45.656+0000 7efd23a00640 1 --2- 172.21.15.28:0/4118183883 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efd1c109740 0x7efd1c10bbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:45.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:45.656+0000 7efd23a00640 1 --2- 172.21.15.28:0/4118183883 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efd1c074290 0x7efd1c074710 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:45.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:45.656+0000 7efd23a00640 1 --2- 172.21.15.28:0/4118183883 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd1c075870 0x7efd1c073cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:45.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:45.656+0000 7efd23a00640 1 -- 172.21.15.28:0/4118183883 >> 172.21.15.28:0/4118183883 conn(0x7efd1c0fce10 msgr2=0x7efd1c0ff230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:08:45.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:45.656+0000 7efd23a00640 1 -- 172.21.15.28:0/4118183883 shutdown_connections 2024-05-12T22:08:45.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:45.656+0000 7efd23a00640 1 -- 172.21.15.28:0/4118183883 wait complete. 2024-05-12T22:08:45.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:45.657+0000 7efd23a00640 1 Processor -- start 2024-05-12T22:08:45.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:45.657+0000 7efd23a00640 1 -- start start 2024-05-12T22:08:45.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:45.658+0000 7efd23a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efd1c074290 0x7efd1c199dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:45.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:45.658+0000 7efd23a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd1c075870 0x7efd1c19a300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:45.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:45.658+0000 7efd23a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efd1c109740 0x7efd1c1a1330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:45.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:45.658+0000 7efd23a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7efd1c10f6e0 con 0x7efd1c075870 2024-05-12T22:08:45.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:45.658+0000 7efd23a00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7efd1c10f560 con 0x7efd1c109740 2024-05-12T22:08:45.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:45.658+0000 7efd23a00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7efd1c10f860 con 0x7efd1c074290 2024-05-12T22:08:45.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:45.658+0000 7efd1be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd1c075870 0x7efd1c19a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:45.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:45.658+0000 7efd21000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efd1c074290 0x7efd1c199dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:45.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:45.658+0000 7efd1be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd1c075870 0x7efd1c19a300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53634/0 (socket says 172.21.15.28:53634) 2024-05-12T22:08:45.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:45.658+0000 7efd1be00640 1 -- 172.21.15.28:0/3758497398 learned_addr learned my addr 172.21.15.28:0/3758497398 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:08:45.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:45.658+0000 7efd21a00640 1 --2- 172.21.15.28:0/3758497398 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efd1c109740 0x7efd1c1a1330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:45.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:45.658+0000 7efd1be00640 1 -- 172.21.15.28:0/3758497398 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efd1c074290 msgr2=0x7efd1c199dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:45.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:45.658+0000 7efd1be00640 1 --2- 172.21.15.28:0/3758497398 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efd1c074290 0x7efd1c199dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:45.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:45.658+0000 7efd1be00640 1 -- 172.21.15.28:0/3758497398 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efd1c109740 msgr2=0x7efd1c1a1330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:45.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:45.658+0000 7efd1be00640 1 --2- 172.21.15.28:0/3758497398 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efd1c109740 0x7efd1c1a1330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:45.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:45.658+0000 7efd1be00640 1 -- 172.21.15.28:0/3758497398 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd1c1a19d0 con 0x7efd1c075870 2024-05-12T22:08:45.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:45.658+0000 7efd21a00640 1 --2- 172.21.15.28:0/3758497398 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efd1c109740 0x7efd1c1a1330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:08:45.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:45.659+0000 7efd1be00640 1 --2- 172.21.15.28:0/3758497398 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd1c075870 0x7efd1c19a300 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7efd08035040 tx=0x7efd08033d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:08:45.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:45.659+0000 7efd19600640 1 -- 172.21.15.28:0/3758497398 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd08037040 con 0x7efd1c075870 2024-05-12T22:08:45.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:45.659+0000 7efd23a00640 1 -- 172.21.15.28:0/3758497398 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd1c1a1c60 con 0x7efd1c075870 2024-05-12T22:08:45.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:45.659+0000 7efd19600640 1 -- 172.21.15.28:0/3758497398 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efd08032770 con 0x7efd1c075870 2024-05-12T22:08:45.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:45.659+0000 7efd19600640 1 -- 172.21.15.28:0/3758497398 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd080046c0 con 0x7efd1c075870 2024-05-12T22:08:45.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:45.659+0000 7efd23a00640 1 -- 172.21.15.28:0/3758497398 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd1c1a21a0 con 0x7efd1c075870 2024-05-12T22:08:45.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:45.660+0000 7efd23a00640 1 -- 172.21.15.28:0/3758497398 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd1c108470 con 0x7efd1c075870 2024-05-12T22:08:45.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:45.663+0000 7efd19600640 1 -- 172.21.15.28:0/3758497398 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7efd08004050 con 0x7efd1c075870 2024-05-12T22:08:45.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:45.664+0000 7efd19600640 1 --2- 172.21.15.28:0/3758497398 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7efd040a2c10 0x7efd040a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:45.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:45.664+0000 7efd19600640 1 -- 172.21.15.28:0/3758497398 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7efd080ef360 con 0x7efd1c075870 2024-05-12T22:08:45.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:45.664+0000 7efd21000640 1 --2- 172.21.15.28:0/3758497398 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7efd040a2c10 0x7efd040a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:45.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:45.665+0000 7efd21000640 1 --2- 172.21.15.28:0/3758497398 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7efd040a2c10 0x7efd040a50d0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7efd1c19b290 tx=0x7efd10009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:08:45.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:45.667+0000 7efd19600640 1 -- 172.21.15.28:0/3758497398 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7efd080bc270 con 0x7efd1c075870 2024-05-12T22:08:45.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:45.874+0000 7efd23a00640 1 -- 172.21.15.28:0/3758497398 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7efd1c02d3e0 con 0x7efd1c075870 2024-05-12T22:08:45.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:45.875+0000 7efd19600640 1 -- 172.21.15.28:0/3758497398 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) ==== 104+0+11 (secure 0 0 0) 0x7efd080bbb70 con 0x7efd1c075870 2024-05-12T22:08:45.875 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-05-12T22:08:45.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:45.877+0000 7efd23a00640 1 -- 172.21.15.28:0/3758497398 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7efd040a2c10 msgr2=0x7efd040a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:45.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:45.877+0000 7efd23a00640 1 --2- 172.21.15.28:0/3758497398 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7efd040a2c10 0x7efd040a50d0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7efd1c19b290 tx=0x7efd10009260 comp rx=0 tx=0).stop 2024-05-12T22:08:45.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:45.877+0000 7efd23a00640 1 -- 172.21.15.28:0/3758497398 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd1c075870 msgr2=0x7efd1c19a300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:45.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:45.877+0000 7efd23a00640 1 --2- 172.21.15.28:0/3758497398 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd1c075870 0x7efd1c19a300 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7efd08035040 tx=0x7efd08033d90 comp rx=0 tx=0).stop 2024-05-12T22:08:45.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:45.878+0000 7efd23a00640 1 -- 172.21.15.28:0/3758497398 shutdown_connections 2024-05-12T22:08:45.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:45.878+0000 7efd23a00640 1 --2- 172.21.15.28:0/3758497398 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7efd040a2c10 0x7efd040a50d0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:45.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:45.878+0000 7efd23a00640 1 --2- 172.21.15.28:0/3758497398 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efd1c109740 0x7efd1c1a1330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:45.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:45.878+0000 7efd23a00640 1 --2- 172.21.15.28:0/3758497398 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd1c075870 0x7efd1c19a300 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:45.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:45.878+0000 7efd23a00640 1 --2- 172.21.15.28:0/3758497398 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efd1c074290 0x7efd1c199dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:45.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:45.878+0000 7efd23a00640 1 -- 172.21.15.28:0/3758497398 >> 172.21.15.28:0/3758497398 conn(0x7efd1c0fce10 msgr2=0x7efd1c105640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:08:45.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:45.878+0000 7efd23a00640 1 -- 172.21.15.28:0/3758497398 shutdown_connections 2024-05-12T22:08:45.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:45.878+0000 7efd23a00640 1 -- 172.21.15.28:0/3758497398 wait complete. 2024-05-12T22:08:45.900 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-12T22:08:46.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:46.115+0000 7ff1bfe00640 1 -- 172.21.15.28:0/2241614236 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff1c0105520 msgr2=0x7ff1c010bdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:46.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:46.115+0000 7ff1bfe00640 1 --2- 172.21.15.28:0/2241614236 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff1c0105520 0x7ff1c010bdb0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7ff1b8009ca0 tx=0x7ff1b802a8c0 comp rx=0 tx=0).stop 2024-05-12T22:08:46.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:46.115+0000 7ff1bfe00640 1 -- 172.21.15.28:0/2241614236 shutdown_connections 2024-05-12T22:08:46.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:46.115+0000 7ff1bfe00640 1 --2- 172.21.15.28:0/2241614236 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff1c0105520 0x7ff1c010bdb0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:46.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:46.115+0000 7ff1bfe00640 1 --2- 172.21.15.28:0/2241614236 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1c0104b60 0x7ff1c0104fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:46.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:46.115+0000 7ff1bfe00640 1 --2- 172.21.15.28:0/2241614236 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1c0103970 0x7ff1c0103d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:46.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:46.115+0000 7ff1bfe00640 1 -- 172.21.15.28:0/2241614236 >> 172.21.15.28:0/2241614236 conn(0x7ff1c00feee0 msgr2=0x7ff1c0101320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:08:46.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:46.115+0000 7ff1bfe00640 1 -- 172.21.15.28:0/2241614236 shutdown_connections 2024-05-12T22:08:46.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:46.115+0000 7ff1bfe00640 1 -- 172.21.15.28:0/2241614236 wait complete. 2024-05-12T22:08:46.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:46.116+0000 7ff1bfe00640 1 Processor -- start 2024-05-12T22:08:46.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:46.116+0000 7ff1bfe00640 1 -- start start 2024-05-12T22:08:46.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:46.117+0000 7ff1bfe00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff1c0103970 0x7ff1c0199bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:46.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:46.117+0000 7ff1bfe00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1c0104b60 0x7ff1c019a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:46.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:46.117+0000 7ff1bfe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1c0105520 0x7ff1c01a1180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:46.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:46.117+0000 7ff1bfe00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff1c010f640 con 0x7ff1c0105520 2024-05-12T22:08:46.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:46.117+0000 7ff1bfe00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7ff1c010f4c0 con 0x7ff1c0103970 2024-05-12T22:08:46.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:46.117+0000 7ff1bfe00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff1c010f7c0 con 0x7ff1c0104b60 2024-05-12T22:08:46.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:46.117+0000 7ff1b7e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1c0104b60 0x7ff1c019a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:46.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:46.117+0000 7ff1bea00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff1c0103970 0x7ff1c0199bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:46.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:46.117+0000 7ff1bf400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1c0105520 0x7ff1c01a1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:46.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:46.117+0000 7ff1bea00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff1c0103970 0x7ff1c0199bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:37742/0 (socket says 172.21.15.28:37742) 2024-05-12T22:08:46.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:46.117+0000 7ff1b7e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1c0104b60 0x7ff1c019a110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:37752/0 (socket says 172.21.15.28:37752) 2024-05-12T22:08:46.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:46.117+0000 7ff1bea00640 1 -- 172.21.15.28:0/3212896078 learned_addr learned my addr 172.21.15.28:0/3212896078 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:08:46.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:46.118+0000 7ff1bea00640 1 -- 172.21.15.28:0/3212896078 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1c0104b60 msgr2=0x7ff1c019a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:46.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:46.118+0000 7ff1bea00640 1 --2- 172.21.15.28:0/3212896078 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1c0104b60 0x7ff1c019a110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:46.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:46.118+0000 7ff1bea00640 1 -- 172.21.15.28:0/3212896078 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1c0105520 msgr2=0x7ff1c01a1180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:46.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:46.118+0000 7ff1bea00640 1 --2- 172.21.15.28:0/3212896078 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1c0105520 0x7ff1c01a1180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:46.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:46.118+0000 7ff1bea00640 1 -- 172.21.15.28:0/3212896078 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1c01a17f0 con 0x7ff1c0103970 2024-05-12T22:08:46.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:46.118+0000 7ff1b7e00640 1 --2- 172.21.15.28:0/3212896078 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1c0104b60 0x7ff1c019a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:08:46.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:46.118+0000 7ff1bea00640 1 --2- 172.21.15.28:0/3212896078 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff1c0103970 0x7ff1c0199bd0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7ff1b000e9c0 tx=0x7ff1b000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:08:46.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:46.118+0000 7ff1c4a00640 1 -- 172.21.15.28:0/3212896078 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1b000cd80 con 0x7ff1c0103970 2024-05-12T22:08:46.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:46.118+0000 7ff1c4a00640 1 -- 172.21.15.28:0/3212896078 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff1b00035f0 con 0x7ff1c0103970 2024-05-12T22:08:46.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:46.118+0000 7ff1bfe00640 1 -- 172.21.15.28:0/3212896078 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1c01a1ae0 con 0x7ff1c0103970 2024-05-12T22:08:46.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:46.118+0000 7ff1c4a00640 1 -- 172.21.15.28:0/3212896078 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1b0010690 con 0x7ff1c0103970 2024-05-12T22:08:46.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:46.118+0000 7ff1bfe00640 1 -- 172.21.15.28:0/3212896078 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1c01a2030 con 0x7ff1c0103970 2024-05-12T22:08:46.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:46.120+0000 7ff1bfe00640 1 -- 172.21.15.28:0/3212896078 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1c010abc0 con 0x7ff1c0103970 2024-05-12T22:08:46.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:46.121+0000 7ff1c4a00640 1 -- 172.21.15.28:0/3212896078 <== mon.2 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7ff1b00037a0 con 0x7ff1c0103970 2024-05-12T22:08:46.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:46.123+0000 7ff1c4a00640 1 --2- 172.21.15.28:0/3212896078 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7ff1a40a2c10 0x7ff1a40a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:46.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:46.123+0000 7ff1c4a00640 1 -- 172.21.15.28:0/3212896078 <== mon.2 v2:172.21.15.86:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7ff1b00cf390 con 0x7ff1c0103970 2024-05-12T22:08:46.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:46.123+0000 7ff1b7e00640 1 --2- 172.21.15.28:0/3212896078 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7ff1a40a2c10 0x7ff1a40a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:46.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:46.124+0000 7ff1b7e00640 1 --2- 172.21.15.28:0/3212896078 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7ff1a40a2c10 0x7ff1a40a50d0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7ff1c019b0e0 tx=0x7ff1a8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:08:46.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:46.127+0000 7ff1c4a00640 1 -- 172.21.15.28:0/3212896078 <== mon.2 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7ff1b009c2a0 con 0x7ff1c0103970 2024-05-12T22:08:46.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:46 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/1581316302' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-12T22:08:46.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:46 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/3758497398' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-12T22:08:46.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:46.331+0000 7ff1bfe00640 1 -- 172.21.15.28:0/3212896078 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7ff1c0062430 con 0x7ff1c0103970 2024-05-12T22:08:46.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:46.332+0000 7ff1c4a00640 1 -- 172.21.15.28:0/3212896078 <== mon.2 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) ==== 100+0+11 (secure 0 0 0) 0x7ff1b009bba0 con 0x7ff1c0103970 2024-05-12T22:08:46.334 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-05-12T22:08:46.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:46.336+0000 7ff1bfe00640 1 -- 172.21.15.28:0/3212896078 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7ff1a40a2c10 msgr2=0x7ff1a40a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:46.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:46.336+0000 7ff1bfe00640 1 --2- 172.21.15.28:0/3212896078 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7ff1a40a2c10 0x7ff1a40a50d0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7ff1c019b0e0 tx=0x7ff1a8005f90 comp rx=0 tx=0).stop 2024-05-12T22:08:46.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:46.336+0000 7ff1bfe00640 1 -- 172.21.15.28:0/3212896078 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff1c0103970 msgr2=0x7ff1c0199bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:46.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:46.336+0000 7ff1bfe00640 1 --2- 172.21.15.28:0/3212896078 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff1c0103970 0x7ff1c0199bd0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7ff1b000e9c0 tx=0x7ff1b000ee80 comp rx=0 tx=0).stop 2024-05-12T22:08:46.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:46.336+0000 7ff1bfe00640 1 -- 172.21.15.28:0/3212896078 shutdown_connections 2024-05-12T22:08:46.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:46.336+0000 7ff1bfe00640 1 --2- 172.21.15.28:0/3212896078 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7ff1a40a2c10 0x7ff1a40a50d0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:46.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:46.336+0000 7ff1bfe00640 1 --2- 172.21.15.28:0/3212896078 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1c0105520 0x7ff1c01a1180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:46.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:46.336+0000 7ff1bfe00640 1 --2- 172.21.15.28:0/3212896078 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1c0104b60 0x7ff1c019a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:46.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:46.336+0000 7ff1bfe00640 1 --2- 172.21.15.28:0/3212896078 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff1c0103970 0x7ff1c0199bd0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:46.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:46.336+0000 7ff1bfe00640 1 -- 172.21.15.28:0/3212896078 >> 172.21.15.28:0/3212896078 conn(0x7ff1c00feee0 msgr2=0x7ff1c0107d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:08:46.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:46.336+0000 7ff1bfe00640 1 -- 172.21.15.28:0/3212896078 shutdown_connections 2024-05-12T22:08:46.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:46.336+0000 7ff1bfe00640 1 -- 172.21.15.28:0/3212896078 wait complete. 2024-05-12T22:08:46.356 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-12T22:08:46.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:46 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/1581316302' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-12T22:08:46.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:46 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/3758497398' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-12T22:08:46.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:46 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/1581316302' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-12T22:08:46.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:46 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/3758497398' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-12T22:08:46.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:46.566+0000 7f1272e00640 1 -- 172.21.15.28:0/1323584422 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f126c1028c0 msgr2=0x7f126c102d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:46.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:46.566+0000 7f1272e00640 1 --2- 172.21.15.28:0/1323584422 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f126c1028c0 0x7f126c102d40 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f1260009900 tx=0x7f126002a3f0 comp rx=0 tx=0).stop 2024-05-12T22:08:46.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:46.566+0000 7f1272e00640 1 -- 172.21.15.28:0/1323584422 shutdown_connections 2024-05-12T22:08:46.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:46.566+0000 7f1272e00640 1 --2- 172.21.15.28:0/1323584422 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f126c103280 0x7f126c109b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:46.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:46.566+0000 7f1272e00640 1 --2- 172.21.15.28:0/1323584422 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f126c1028c0 0x7f126c102d40 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:46.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:46.566+0000 7f1272e00640 1 --2- 172.21.15.28:0/1323584422 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f126c1016d0 0x7f126c101af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:46.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:46.566+0000 7f1272e00640 1 -- 172.21.15.28:0/1323584422 >> 172.21.15.28:0/1323584422 conn(0x7f126c0fcc40 msgr2=0x7f126c0ff080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:08:46.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:46.566+0000 7f1272e00640 1 -- 172.21.15.28:0/1323584422 shutdown_connections 2024-05-12T22:08:46.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:46.566+0000 7f1272e00640 1 -- 172.21.15.28:0/1323584422 wait complete. 2024-05-12T22:08:46.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:46.567+0000 7f1272e00640 1 Processor -- start 2024-05-12T22:08:46.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:46.567+0000 7f1272e00640 1 -- start start 2024-05-12T22:08:46.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:46.568+0000 7f1272e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f126c1016d0 0x7f126c199c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:46.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:46.568+0000 7f1272e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f126c1028c0 0x7f126c19a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:46.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:46.568+0000 7f1272e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f126c103280 0x7f126c1a11a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:46.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:46.568+0000 7f1272e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f126c10d380 con 0x7f126c103280 2024-05-12T22:08:46.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:46.568+0000 7f1272e00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f126c10d200 con 0x7f126c1016d0 2024-05-12T22:08:46.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:46.568+0000 7f1272e00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f126c10d500 con 0x7f126c1028c0 2024-05-12T22:08:46.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:46.568+0000 7f1272400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f126c103280 0x7f126c1a11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:46.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:46.568+0000 7f1271a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f126c1016d0 0x7f126c199c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:46.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:46.568+0000 7f1272400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f126c103280 0x7f126c1a11a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42504/0 (socket says 172.21.15.28:42504) 2024-05-12T22:08:46.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:46.568+0000 7f1272400640 1 -- 172.21.15.28:0/2427264621 learned_addr learned my addr 172.21.15.28:0/2427264621 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:08:46.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:46.568+0000 7f1271000640 1 --2- 172.21.15.28:0/2427264621 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f126c1028c0 0x7f126c19a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:46.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:46.568+0000 7f1272400640 1 -- 172.21.15.28:0/2427264621 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f126c1028c0 msgr2=0x7f126c19a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:46.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:46.568+0000 7f1272400640 1 --2- 172.21.15.28:0/2427264621 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f126c1028c0 0x7f126c19a1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:46.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:46.568+0000 7f1272400640 1 -- 172.21.15.28:0/2427264621 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f126c1016d0 msgr2=0x7f126c199c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:46.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:46.568+0000 7f1272400640 1 --2- 172.21.15.28:0/2427264621 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f126c1016d0 0x7f126c199c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:46.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:46.568+0000 7f1272400640 1 -- 172.21.15.28:0/2427264621 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f126c1a1840 con 0x7f126c103280 2024-05-12T22:08:46.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:46.569+0000 7f1272400640 1 --2- 172.21.15.28:0/2427264621 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f126c103280 0x7f126c1a11a0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f126400eb10 tx=0x7f126400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:08:46.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:46.569+0000 7f126a000640 1 -- 172.21.15.28:0/2427264621 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1264004b20 con 0x7f126c103280 2024-05-12T22:08:46.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:46.569+0000 7f126a000640 1 -- 172.21.15.28:0/2427264621 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1264004cc0 con 0x7f126c103280 2024-05-12T22:08:46.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:46.569+0000 7f126a000640 1 -- 172.21.15.28:0/2427264621 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1264010760 con 0x7f126c103280 2024-05-12T22:08:46.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:46.569+0000 7f1272e00640 1 -- 172.21.15.28:0/2427264621 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f126c1a1b30 con 0x7f126c103280 2024-05-12T22:08:46.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:46.569+0000 7f1272e00640 1 -- 172.21.15.28:0/2427264621 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f126c1a2030 con 0x7f126c103280 2024-05-12T22:08:46.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:46.570+0000 7f1272e00640 1 -- 172.21.15.28:0/2427264621 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f126c1a2200 con 0x7f126c103280 2024-05-12T22:08:46.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:46.572+0000 7f126a000640 1 -- 172.21.15.28:0/2427264621 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f1264005020 con 0x7f126c103280 2024-05-12T22:08:46.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:46.574+0000 7f126a000640 1 --2- 172.21.15.28:0/2427264621 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f12540a2b90 0x7f12540a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:46.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:46.574+0000 7f126a000640 1 -- 172.21.15.28:0/2427264621 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f12640cf6d0 con 0x7f126c103280 2024-05-12T22:08:46.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:46.574+0000 7f1271a00640 1 --2- 172.21.15.28:0/2427264621 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f12540a2b90 0x7f12540a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:46.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:46.575+0000 7f1271a00640 1 --2- 172.21.15.28:0/2427264621 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f12540a2b90 0x7f12540a5050 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f126c19b100 tx=0x7f125c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:08:46.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:46.578+0000 7f126a000640 1 -- 172.21.15.28:0/2427264621 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f126409c5e0 con 0x7f126c103280 2024-05-12T22:08:46.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:46.830+0000 7f1272e00640 1 -- 172.21.15.28:0/2427264621 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f126c1a23d0 con 0x7f126c103280 2024-05-12T22:08:46.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:46.831+0000 7f126a000640 1 -- 172.21.15.28:0/2427264621 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11864 (secure 0 0 0) 0x7f126409bee0 con 0x7f126c103280 2024-05-12T22:08:46.832 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-12T22:08:46.832 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1844141745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1844141745},{"type":"v1","addr":"172.21.15.186:6833","nonce":1844141745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6833/2960763758","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":2960763758},{"type":"v1","addr":"172.21.15.86:6833","nonce":2960763758}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24470,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/4131134219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":4131134219},{"type":"v1","addr":"172.21.15.28:6835","nonce":4131134219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24481,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/1727869064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1727869064},{"type":"v1","addr":"172.21.15.186:6835","nonce":1727869064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24482,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2938260704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2938260704},{"type":"v1","addr":"172.21.15.28:6837","nonce":2938260704}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24494,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3473700910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3473700910},{"type":"v1","addr":"172.21.15.86:6835","nonce":3473700910}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-12T22:06:33.426986+0000","modified":"2024-05-12T22:06:38.789246+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24581,"mds_1":24550,"mds_2":24532},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24499":{"gid":24499,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6837/3600809235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":3600809235},{"type":"v1","addr":"172.21.15.186:6837","nonce":3600809235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"i","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6839/2399610807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":2399610807},{"type":"v1","addr":"172.21.15.186:6839","nonce":2399610807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24530":{"gid":24530,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.86:6837/3782448884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":3782448884},{"type":"v1","addr":"172.21.15.86:6837","nonce":3782448884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24532":{"gid":24532,"name":"g","rank":2,"incarnation":21,"state":"up:active","state_seq":6,"addr":"172.21.15.28:6839/123892158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":123892158},{"type":"v1","addr":"172.21.15.28:6839","nonce":123892158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24550":{"gid":24550,"name":"a","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.28:6841/761183870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":761183870},{"type":"v1","addr":"172.21.15.28:6841","nonce":761183870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24581":{"gid":24581,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6839/1251999130","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6838","nonce":1251999130},{"type":"v1","addr":"172.21.15.86:6839","nonce":1251999130}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24581,"qdb_cluster":[24550,24532,24581]},"id":1}]} 2024-05-12T22:08:46.832 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 23 2024-05-12T22:08:46.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:46.834+0000 7f1272e00640 1 -- 172.21.15.28:0/2427264621 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f12540a2b90 msgr2=0x7f12540a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:46.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:46.834+0000 7f1272e00640 1 --2- 172.21.15.28:0/2427264621 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f12540a2b90 0x7f12540a5050 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f126c19b100 tx=0x7f125c005f90 comp rx=0 tx=0).stop 2024-05-12T22:08:46.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:46.834+0000 7f1272e00640 1 -- 172.21.15.28:0/2427264621 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f126c103280 msgr2=0x7f126c1a11a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:46.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:46.834+0000 7f1272e00640 1 --2- 172.21.15.28:0/2427264621 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f126c103280 0x7f126c1a11a0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f126400eb10 tx=0x7f126400c2d0 comp rx=0 tx=0).stop 2024-05-12T22:08:46.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:46.834+0000 7f1272e00640 1 -- 172.21.15.28:0/2427264621 shutdown_connections 2024-05-12T22:08:46.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:46.834+0000 7f1272e00640 1 --2- 172.21.15.28:0/2427264621 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f12540a2b90 0x7f12540a5050 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:46.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:46.834+0000 7f1272e00640 1 --2- 172.21.15.28:0/2427264621 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f126c103280 0x7f126c1a11a0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:46.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:46.834+0000 7f1272e00640 1 --2- 172.21.15.28:0/2427264621 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f126c1028c0 0x7f126c19a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:46.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:46.834+0000 7f1272e00640 1 --2- 172.21.15.28:0/2427264621 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f126c1016d0 0x7f126c199c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:46.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:46.834+0000 7f1272e00640 1 -- 172.21.15.28:0/2427264621 >> 172.21.15.28:0/2427264621 conn(0x7f126c0fcc40 msgr2=0x7f126c073a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:08:46.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:46.834+0000 7f1272e00640 1 -- 172.21.15.28:0/2427264621 shutdown_connections 2024-05-12T22:08:46.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:46.834+0000 7f1272e00640 1 -- 172.21.15.28:0/2427264621 wait complete. 2024-05-12T22:08:46.855 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-12T22:08:47.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.064+0000 7f950ec00640 1 -- 172.21.15.28:0/1834293355 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9508109860 msgr2=0x7f95081100f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:47.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.064+0000 7f950ec00640 1 --2- 172.21.15.28:0/1834293355 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9508109860 0x7f95081100f0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f9500009be0 tx=0x7f950002a6c0 comp rx=0 tx=0).stop 2024-05-12T22:08:47.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.064+0000 7f950ec00640 1 -- 172.21.15.28:0/1834293355 shutdown_connections 2024-05-12T22:08:47.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.064+0000 7f950ec00640 1 --2- 172.21.15.28:0/1834293355 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9508109860 0x7f95081100f0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:47.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.064+0000 7f950ec00640 1 --2- 172.21.15.28:0/1834293355 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9508108ea0 0x7f9508109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:47.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.064+0000 7f950ec00640 1 --2- 172.21.15.28:0/1834293355 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9508107cb0 0x7f95081080d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:47.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.064+0000 7f950ec00640 1 -- 172.21.15.28:0/1834293355 >> 172.21.15.28:0/1834293355 conn(0x7f9508076c70 msgr2=0x7f9508079090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:08:47.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.064+0000 7f950ec00640 1 -- 172.21.15.28:0/1834293355 shutdown_connections 2024-05-12T22:08:47.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.064+0000 7f950ec00640 1 -- 172.21.15.28:0/1834293355 wait complete. 2024-05-12T22:08:47.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.065+0000 7f950ec00640 1 Processor -- start 2024-05-12T22:08:47.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.065+0000 7f950ec00640 1 -- start start 2024-05-12T22:08:47.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.066+0000 7f950ec00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9508107cb0 0x7f950819df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:47.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.066+0000 7f950ec00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9508108ea0 0x7f950819e490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:47.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.066+0000 7f950ec00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9508109860 0x7f95081a5500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:47.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.066+0000 7f950ec00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9508113910 con 0x7f9508109860 2024-05-12T22:08:47.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.066+0000 7f950ec00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f9508113790 con 0x7f9508108ea0 2024-05-12T22:08:47.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.066+0000 7f950ec00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f9508113a90 con 0x7f9508107cb0 2024-05-12T22:08:47.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.066+0000 7f950d800640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9508107cb0 0x7f950819df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:47.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.066+0000 7f950e200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9508109860 0x7f95081a5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:47.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.066+0000 7f950d800640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9508107cb0 0x7f950819df50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:37782/0 (socket says 172.21.15.28:37782) 2024-05-12T22:08:47.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.066+0000 7f950d800640 1 -- 172.21.15.28:0/511460812 learned_addr learned my addr 172.21.15.28:0/511460812 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:08:47.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.066+0000 7f950ce00640 1 --2- 172.21.15.28:0/511460812 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9508108ea0 0x7f950819e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:47.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.066+0000 7f950d800640 1 -- 172.21.15.28:0/511460812 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9508108ea0 msgr2=0x7f950819e490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:47.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.066+0000 7f950d800640 1 --2- 172.21.15.28:0/511460812 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9508108ea0 0x7f950819e490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:47.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.066+0000 7f950d800640 1 -- 172.21.15.28:0/511460812 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9508109860 msgr2=0x7f95081a5500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:47.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.066+0000 7f950d800640 1 --2- 172.21.15.28:0/511460812 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9508109860 0x7f95081a5500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:47.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.066+0000 7f950d800640 1 -- 172.21.15.28:0/511460812 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95081a5b70 con 0x7f9508107cb0 2024-05-12T22:08:47.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.067+0000 7f950d800640 1 --2- 172.21.15.28:0/511460812 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9508107cb0 0x7f950819df50 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f94fc00e9c0 tx=0x7f94fc00edc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:08:47.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.067+0000 7f9506000640 1 -- 172.21.15.28:0/511460812 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94fc00cd80 con 0x7f9508107cb0 2024-05-12T22:08:47.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.067+0000 7f9506000640 1 -- 172.21.15.28:0/511460812 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f94fc010910 con 0x7f9508107cb0 2024-05-12T22:08:47.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.067+0000 7f9506000640 1 -- 172.21.15.28:0/511460812 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94fc018950 con 0x7f9508107cb0 2024-05-12T22:08:47.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.067+0000 7f950ec00640 1 -- 172.21.15.28:0/511460812 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95081a5e60 con 0x7f9508107cb0 2024-05-12T22:08:47.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.067+0000 7f950ec00640 1 -- 172.21.15.28:0/511460812 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95081a6210 con 0x7f9508107cb0 2024-05-12T22:08:47.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.068+0000 7f950ec00640 1 -- 172.21.15.28:0/511460812 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f950810ef70 con 0x7f9508107cb0 2024-05-12T22:08:47.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.070+0000 7f9506000640 1 -- 172.21.15.28:0/511460812 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f94fc01f020 con 0x7f9508107cb0 2024-05-12T22:08:47.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.072+0000 7f9506000640 1 --2- 172.21.15.28:0/511460812 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f94f00a2c10 0x7f94f00a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:47.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.072+0000 7f9506000640 1 -- 172.21.15.28:0/511460812 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f94fc0cf0b0 con 0x7f9508107cb0 2024-05-12T22:08:47.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.072+0000 7f950ce00640 1 --2- 172.21.15.28:0/511460812 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f94f00a2c10 0x7f94f00a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:47.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.073+0000 7f950ce00640 1 --2- 172.21.15.28:0/511460812 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f94f00a2c10 0x7f94f00a50d0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f950819f460 tx=0x7f94f4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:08:47.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.076+0000 7f9506000640 1 -- 172.21.15.28:0/511460812 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f94fc09bfc0 con 0x7f9508107cb0 2024-05-12T22:08:47.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.283+0000 7f950ec00640 1 -- 172.21.15.28:0/511460812 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f95080625b0 con 0x7f9508107cb0 2024-05-12T22:08:47.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:47 smithi028 ceph-mon[29847]: pgmap v378: 97 pgs: 97 active+clean; 641 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-12T22:08:47.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:47 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/3212896078' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-12T22:08:47.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:47 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/2427264621' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:08:47.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.285+0000 7f9506000640 1 -- 172.21.15.28:0/511460812 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22595 (secure 0 0 0) 0x7f94fc09b8c0 con 0x7f9508107cb0 2024-05-12T22:08:47.287 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-12T22:08:47.287 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":88,"fsid":"6b92e7b4-10aa-11ef-bc98-c7b262605968","created":"2024-05-12T21:57:37.276782+0000","modified":"2024-05-12T22:06:46.768798+0000","last_up_change":"2024-05-12T22:05:37.929562+0000","last_in_change":"2024-05-12T22:05:18.750768+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-12T22:01:32.292826+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-12T22:06:31.540263+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-12T22:06:32.906118+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6992ce0a-844b-40e3-8ed1-181c8474dccf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":906003882},{"type":"v1","addr":"172.21.15.28:6803","nonce":906003882}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":906003882},{"type":"v1","addr":"172.21.15.28:6805","nonce":906003882}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":906003882},{"type":"v1","addr":"172.21.15.28:6809","nonce":906003882}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":906003882},{"type":"v1","addr":"172.21.15.28:6807","nonce":906003882}]},"public_addr":"172.21.15.28:6803/906003882","cluster_addr":"172.21.15.28:6805/906003882","heartbeat_back_addr":"172.21.15.28:6809/906003882","heartbeat_front_addr":"172.21.15.28:6807/906003882","state":["exists","up"]},{"osd":1,"uuid":"ced20c96-c481-4d48-90d1-6b43228959ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":3354141819},{"type":"v1","addr":"172.21.15.86:6801","nonce":3354141819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":3354141819},{"type":"v1","addr":"172.21.15.86:6803","nonce":3354141819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":3354141819},{"type":"v1","addr":"172.21.15.86:6807","nonce":3354141819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":3354141819},{"type":"v1","addr":"172.21.15.86:6805","nonce":3354141819}]},"public_addr":"172.21.15.86:6801/3354141819","cluster_addr":"172.21.15.86:6803/3354141819","heartbeat_back_addr":"172.21.15.86:6807/3354141819","heartbeat_front_addr":"172.21.15.86:6805/3354141819","state":["exists","up"]},{"osd":2,"uuid":"54a2f221-fa65-4795-9f5e-e660cc7c0098","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":906459365},{"type":"v1","addr":"172.21.15.186:6801","nonce":906459365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":906459365},{"type":"v1","addr":"172.21.15.186:6803","nonce":906459365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":906459365},{"type":"v1","addr":"172.21.15.186:6807","nonce":906459365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":906459365},{"type":"v1","addr":"172.21.15.186:6805","nonce":906459365}]},"public_addr":"172.21.15.186:6801/906459365","cluster_addr":"172.21.15.186:6803/906459365","heartbeat_back_addr":"172.21.15.186:6807/906459365","heartbeat_front_addr":"172.21.15.186:6805/906459365","state":["exists","up"]},{"osd":3,"uuid":"86836737-ac9a-487b-93f2-7fcd3c406960","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":4182884436},{"type":"v1","addr":"172.21.15.28:6811","nonce":4182884436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":4182884436},{"type":"v1","addr":"172.21.15.28:6813","nonce":4182884436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":4182884436},{"type":"v1","addr":"172.21.15.28:6817","nonce":4182884436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":4182884436},{"type":"v1","addr":"172.21.15.28:6815","nonce":4182884436}]},"public_addr":"172.21.15.28:6811/4182884436","cluster_addr":"172.21.15.28:6813/4182884436","heartbeat_back_addr":"172.21.15.28:6817/4182884436","heartbeat_front_addr":"172.21.15.28:6815/4182884436","state":["exists","up"]},{"osd":4,"uuid":"d162ebcd-83c0-4fd5-9925-28b2cecea935","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2596998708},{"type":"v1","addr":"172.21.15.86:6809","nonce":2596998708}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2596998708},{"type":"v1","addr":"172.21.15.86:6811","nonce":2596998708}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2596998708},{"type":"v1","addr":"172.21.15.86:6815","nonce":2596998708}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2596998708},{"type":"v1","addr":"172.21.15.86:6813","nonce":2596998708}]},"public_addr":"172.21.15.86:6809/2596998708","cluster_addr":"172.21.15.86:6811/2596998708","heartbeat_back_addr":"172.21.15.86:6815/2596998708","heartbeat_front_addr":"172.21.15.86:6813/2596998708","state":["exists","up"]},{"osd":5,"uuid":"c794336a-a105-425c-b6c7-a32b90c05bb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":1944205756},{"type":"v1","addr":"172.21.15.186:6809","nonce":1944205756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":1944205756},{"type":"v1","addr":"172.21.15.186:6811","nonce":1944205756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":1944205756},{"type":"v1","addr":"172.21.15.186:6815","nonce":1944205756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":1944205756},{"type":"v1","addr":"172.21.15.186:6813","nonce":1944205756}]},"public_addr":"172.21.15.186:6809/1944205756","cluster_addr":"172.21.15.186:6811/1944205756","heartbeat_back_addr":"172.21.15.186:6815/1944205756","heartbeat_front_addr":"172.21.15.186:6813/1944205756","state":["exists","up"]},{"osd":6,"uuid":"516932ac-af00-45b7-ad3c-d6dec7db442f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":1642308925},{"type":"v1","addr":"172.21.15.28:6819","nonce":1642308925}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":1642308925},{"type":"v1","addr":"172.21.15.28:6821","nonce":1642308925}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1642308925},{"type":"v1","addr":"172.21.15.28:6825","nonce":1642308925}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":1642308925},{"type":"v1","addr":"172.21.15.28:6823","nonce":1642308925}]},"public_addr":"172.21.15.28:6819/1642308925","cluster_addr":"172.21.15.28:6821/1642308925","heartbeat_back_addr":"172.21.15.28:6825/1642308925","heartbeat_front_addr":"172.21.15.28:6823/1642308925","state":["exists","up"]},{"osd":7,"uuid":"ca7a059f-0414-438e-b2d1-a10f348036be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":726062711},{"type":"v1","addr":"172.21.15.86:6817","nonce":726062711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":726062711},{"type":"v1","addr":"172.21.15.86:6819","nonce":726062711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":726062711},{"type":"v1","addr":"172.21.15.86:6823","nonce":726062711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":726062711},{"type":"v1","addr":"172.21.15.86:6821","nonce":726062711}]},"public_addr":"172.21.15.86:6817/726062711","cluster_addr":"172.21.15.86:6819/726062711","heartbeat_back_addr":"172.21.15.86:6823/726062711","heartbeat_front_addr":"172.21.15.86:6821/726062711","state":["exists","up"]},{"osd":8,"uuid":"92e3f41f-aebd-422e-a99e-555f7ba4aa53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":3007687277},{"type":"v1","addr":"172.21.15.186:6817","nonce":3007687277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":3007687277},{"type":"v1","addr":"172.21.15.186:6819","nonce":3007687277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":3007687277},{"type":"v1","addr":"172.21.15.186:6823","nonce":3007687277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":3007687277},{"type":"v1","addr":"172.21.15.186:6821","nonce":3007687277}]},"public_addr":"172.21.15.186:6817/3007687277","cluster_addr":"172.21.15.186:6819/3007687277","heartbeat_back_addr":"172.21.15.186:6823/3007687277","heartbeat_front_addr":"172.21.15.186:6821/3007687277","state":["exists","up"]},{"osd":9,"uuid":"38ff7d9d-ee85-404c-8c59-adc327e1ccfa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":136220472},{"type":"v1","addr":"172.21.15.28:6827","nonce":136220472}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":136220472},{"type":"v1","addr":"172.21.15.28:6829","nonce":136220472}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":136220472},{"type":"v1","addr":"172.21.15.28:6833","nonce":136220472}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":136220472},{"type":"v1","addr":"172.21.15.28:6831","nonce":136220472}]},"public_addr":"172.21.15.28:6827/136220472","cluster_addr":"172.21.15.28:6829/136220472","heartbeat_back_addr":"172.21.15.28:6833/136220472","heartbeat_front_addr":"172.21.15.28:6831/136220472","state":["exists","up"]},{"osd":10,"uuid":"5e9ff657-afa8-4bab-8491-f209038392b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1189968310},{"type":"v1","addr":"172.21.15.86:6825","nonce":1189968310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":1189968310},{"type":"v1","addr":"172.21.15.86:6827","nonce":1189968310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":1189968310},{"type":"v1","addr":"172.21.15.86:6831","nonce":1189968310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":1189968310},{"type":"v1","addr":"172.21.15.86:6829","nonce":1189968310}]},"public_addr":"172.21.15.86:6825/1189968310","cluster_addr":"172.21.15.86:6827/1189968310","heartbeat_back_addr":"172.21.15.86:6831/1189968310","heartbeat_front_addr":"172.21.15.86:6829/1189968310","state":["exists","up"]},{"osd":11,"uuid":"f112e3c6-65b6-4253-a966-62001db7d069","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3801668110},{"type":"v1","addr":"172.21.15.186:6825","nonce":3801668110}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3801668110},{"type":"v1","addr":"172.21.15.186:6827","nonce":3801668110}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3801668110},{"type":"v1","addr":"172.21.15.186:6831","nonce":3801668110}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3801668110},{"type":"v1","addr":"172.21.15.186:6829","nonce":3801668110}]},"public_addr":"172.21.15.186:6825/3801668110","cluster_addr":"172.21.15.186:6827/3801668110","heartbeat_back_addr":"172.21.15.186:6831/3801668110","heartbeat_front_addr":"172.21.15.186:6829/3801668110","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:00:40.483623+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:01:05.284385+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:01:28.645716+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:02:01.295092+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:02:26.486601+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:02:50.807478+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:03:24.756169+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:03:49.975173+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:04:13.772233+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:04:46.758555+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:05:12.331685+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:05:36.462548+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":2,"to":1}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.25","mappings":[{"from":5,"to":9}]},{"pgid":"3.27","mappings":[{"from":4,"to":8}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/1177009533":"2024-05-13T21:58:45.550459+0000","172.21.15.28:6800/3760499540":"2024-05-13T21:58:45.550459+0000","172.21.15.28:6801/3783742728":"2024-05-13T21:58:10.421564+0000","172.21.15.28:6800/3783742728":"2024-05-13T21:58:10.421564+0000","172.21.15.28:0/2229156260":"2024-05-13T21:58:45.550459+0000","172.21.15.28:0/1202140963":"2024-05-13T21:58:10.421564+0000","172.21.15.28:0/2924223784":"2024-05-13T21:58:45.550459+0000","172.21.15.28:6801/3760499540":"2024-05-13T21:58:45.550459+0000","172.21.15.28:0/1240027206":"2024-05-13T21:58:10.421564+0000","172.21.15.28:0/171417728":"2024-05-13T21:58:10.421564+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-12T22:08:47.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.289+0000 7f950ec00640 1 -- 172.21.15.28:0/511460812 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f94f00a2c10 msgr2=0x7f94f00a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:47.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.289+0000 7f950ec00640 1 --2- 172.21.15.28:0/511460812 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f94f00a2c10 0x7f94f00a50d0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f950819f460 tx=0x7f94f4009260 comp rx=0 tx=0).stop 2024-05-12T22:08:47.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.289+0000 7f950ec00640 1 -- 172.21.15.28:0/511460812 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9508107cb0 msgr2=0x7f950819df50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:47.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.289+0000 7f950ec00640 1 --2- 172.21.15.28:0/511460812 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9508107cb0 0x7f950819df50 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f94fc00e9c0 tx=0x7f94fc00edc0 comp rx=0 tx=0).stop 2024-05-12T22:08:47.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.289+0000 7f950ec00640 1 -- 172.21.15.28:0/511460812 shutdown_connections 2024-05-12T22:08:47.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.289+0000 7f950ec00640 1 --2- 172.21.15.28:0/511460812 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f94f00a2c10 0x7f94f00a50d0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:47.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.289+0000 7f950ec00640 1 --2- 172.21.15.28:0/511460812 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9508109860 0x7f95081a5500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:47.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.289+0000 7f950ec00640 1 --2- 172.21.15.28:0/511460812 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9508108ea0 0x7f950819e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:47.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.289+0000 7f950ec00640 1 --2- 172.21.15.28:0/511460812 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9508107cb0 0x7f950819df50 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:47.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.289+0000 7f950ec00640 1 -- 172.21.15.28:0/511460812 >> 172.21.15.28:0/511460812 conn(0x7f9508076c70 msgr2=0x7f950810c140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:08:47.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.289+0000 7f950ec00640 1 -- 172.21.15.28:0/511460812 shutdown_connections 2024-05-12T22:08:47.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.289+0000 7f950ec00640 1 -- 172.21.15.28:0/511460812 wait complete. 2024-05-12T22:08:47.309 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-05-12T22:06:33.426986+0000', 'modified': '2024-05-12T22:06:38.789246+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24581, 'mds_1': 24550, 'mds_2': 24532}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24499': {'gid': 24499, 'name': 'c', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.186:6837/3600809235', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.186:6836', 'nonce': 3600809235}, {'type': 'v1', 'addr': '172.21.15.186:6837', 'nonce': 3600809235}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24520': {'gid': 24520, 'name': 'i', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.186:6839/2399610807', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.186:6838', 'nonce': 2399610807}, {'type': 'v1', 'addr': '172.21.15.186:6839', 'nonce': 2399610807}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24530': {'gid': 24530, 'name': 'h', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.86:6837/3782448884', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6836', 'nonce': 3782448884}, {'type': 'v1', 'addr': '172.21.15.86:6837', 'nonce': 3782448884}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24532': {'gid': 24532, 'name': 'g', 'rank': 2, 'incarnation': 21, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.28:6839/123892158', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6838', 'nonce': 123892158}, {'type': 'v1', 'addr': '172.21.15.28:6839', 'nonce': 123892158}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24550': {'gid': 24550, 'name': 'a', 'rank': 1, 'incarnation': 19, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.28:6841/761183870', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6840', 'nonce': 761183870}, {'type': 'v1', 'addr': '172.21.15.28:6841', 'nonce': 761183870}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24581': {'gid': 24581, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.86:6839/1251999130', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6838', 'nonce': 1251999130}, {'type': 'v1', 'addr': '172.21.15.86:6839', 'nonce': 1251999130}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24581, 'qdb_cluster': [24550, 24532, 24581]} 2024-05-12T22:08:47.309 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-05-12T22:08:47.309 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.g status 2024-05-12T22:08:47.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:47 smithi186 ceph-mon[34785]: pgmap v378: 97 pgs: 97 active+clean; 641 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-12T22:08:47.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:47 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/3212896078' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-12T22:08:47.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:47 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/2427264621' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:08:47.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:47 smithi086 ceph-mon[34297]: pgmap v378: 97 pgs: 97 active+clean; 641 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-12T22:08:47.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:47 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/3212896078' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-12T22:08:47.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:47 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/2427264621' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:08:47.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.511+0000 7ff4ad000640 1 -- 172.21.15.28:0/1289607407 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff4a8101760 msgr2=0x7ff4a8101b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:47.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.511+0000 7ff4ad000640 1 --2- 172.21.15.28:0/1289607407 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff4a8101760 0x7ff4a8101b80 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7ff498009900 tx=0x7ff49802a3f0 comp rx=0 tx=0).stop 2024-05-12T22:08:47.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.511+0000 7ff4ad000640 1 -- 172.21.15.28:0/1289607407 shutdown_connections 2024-05-12T22:08:47.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.511+0000 7ff4ad000640 1 --2- 172.21.15.28:0/1289607407 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4a8103310 0x7ff4a8109ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:47.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.511+0000 7ff4ad000640 1 --2- 172.21.15.28:0/1289607407 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff4a8102950 0x7ff4a8102dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:47.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.511+0000 7ff4ad000640 1 --2- 172.21.15.28:0/1289607407 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff4a8101760 0x7ff4a8101b80 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:47.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.511+0000 7ff4ad000640 1 -- 172.21.15.28:0/1289607407 >> 172.21.15.28:0/1289607407 conn(0x7ff4a80fccd0 msgr2=0x7ff4a80ff110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:08:47.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.511+0000 7ff4ad000640 1 -- 172.21.15.28:0/1289607407 shutdown_connections 2024-05-12T22:08:47.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.511+0000 7ff4ad000640 1 -- 172.21.15.28:0/1289607407 wait complete. 2024-05-12T22:08:47.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.513+0000 7ff4ad000640 1 Processor -- start 2024-05-12T22:08:47.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.513+0000 7ff4ad000640 1 -- start start 2024-05-12T22:08:47.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.513+0000 7ff4ad000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4a8101760 0x7ff4a8199c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:47.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.513+0000 7ff4ad000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff4a8102950 0x7ff4a819a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:47.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.513+0000 7ff4ad000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff4a8103310 0x7ff4a81a11e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:47.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.513+0000 7ff4ad000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff4a810d400 con 0x7ff4a8101760 2024-05-12T22:08:47.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.513+0000 7ff4ad000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7ff4a810d280 con 0x7ff4a8103310 2024-05-12T22:08:47.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.513+0000 7ff4ad000640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff4a810d580 con 0x7ff4a8102950 2024-05-12T22:08:47.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.514+0000 7ff4a7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4a8101760 0x7ff4a8199c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:47.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.514+0000 7ff4a7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4a8101760 0x7ff4a8199c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42542/0 (socket says 172.21.15.28:42542) 2024-05-12T22:08:47.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.514+0000 7ff4a7400640 1 -- 172.21.15.28:0/1059374608 learned_addr learned my addr 172.21.15.28:0/1059374608 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:08:47.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.514+0000 7ff4a7e00640 1 --2- 172.21.15.28:0/1059374608 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff4a8103310 0x7ff4a81a11e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:47.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.514+0000 7ff49fe00640 1 --2- 172.21.15.28:0/1059374608 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff4a8102950 0x7ff4a819a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:47.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.514+0000 7ff4a7400640 1 -- 172.21.15.28:0/1059374608 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff4a8102950 msgr2=0x7ff4a819a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:47.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.514+0000 7ff4a7400640 1 --2- 172.21.15.28:0/1059374608 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff4a8102950 0x7ff4a819a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:47.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.514+0000 7ff4a7400640 1 -- 172.21.15.28:0/1059374608 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff4a8103310 msgr2=0x7ff4a81a11e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:47.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.514+0000 7ff4a7400640 1 --2- 172.21.15.28:0/1059374608 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff4a8103310 0x7ff4a81a11e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:47.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.514+0000 7ff4a7400640 1 -- 172.21.15.28:0/1059374608 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4a81a1880 con 0x7ff4a8101760 2024-05-12T22:08:47.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.514+0000 7ff4a7e00640 1 --2- 172.21.15.28:0/1059374608 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff4a8103310 0x7ff4a81a11e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:08:47.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.515+0000 7ff4a7400640 1 --2- 172.21.15.28:0/1059374608 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4a8101760 0x7ff4a8199c70 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7ff498031440 tx=0x7ff498032ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:08:47.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.515+0000 7ff4a4c00640 1 -- 172.21.15.28:0/1059374608 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff49803f030 con 0x7ff4a8101760 2024-05-12T22:08:47.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.515+0000 7ff4a4c00640 1 -- 172.21.15.28:0/1059374608 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff498003ce0 con 0x7ff4a8101760 2024-05-12T22:08:47.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.515+0000 7ff4ad000640 1 -- 172.21.15.28:0/1059374608 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4a81a1b10 con 0x7ff4a8101760 2024-05-12T22:08:47.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.515+0000 7ff4a4c00640 1 -- 172.21.15.28:0/1059374608 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff498035070 con 0x7ff4a8101760 2024-05-12T22:08:47.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.515+0000 7ff4ad000640 1 -- 172.21.15.28:0/1059374608 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4a81a2000 con 0x7ff4a8101760 2024-05-12T22:08:47.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.518+0000 7ff4b1439740 1 Processor -- start 2024-05-12T22:08:47.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.518+0000 7ff4b1439740 1 -- start start 2024-05-12T22:08:47.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.519+0000 7ff4a4c00640 1 -- 172.21.15.28:0/1059374608 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7ff498042030 con 0x7ff4a8101760 2024-05-12T22:08:47.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.519+0000 7ff4b1439740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d48a5f190 0x564d48a5f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:47.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.519+0000 7ff4b1439740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564d48a5fb80 0x564d48a64680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:47.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.519+0000 7ff4b1439740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564d48a64cf0 0x564d48a67180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:47.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.519+0000 7ff4b1439740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x564d489c2480 con 0x564d48a5f190 2024-05-12T22:08:47.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.519+0000 7ff4b1439740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x564d4896a5c0 con 0x564d48a64cf0 2024-05-12T22:08:47.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.519+0000 7ff4b1439740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x564d489be440 con 0x564d48a5fb80 2024-05-12T22:08:47.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.519+0000 7ff4a7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d48a5f190 0x564d48a5f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:47.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.519+0000 7ff4a7400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564d48a5fb80 0x564d48a64680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:47.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.519+0000 7ff4a7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d48a5f190 0x564d48a5f5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42554/0 (socket says 172.21.15.28:42554) 2024-05-12T22:08:47.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.519+0000 7ff4a7e00640 1 -- 172.21.15.28:0/4162945955 learned_addr learned my addr 172.21.15.28:0/4162945955 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:08:47.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.519+0000 7ff49fe00640 1 --2- 172.21.15.28:0/4162945955 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564d48a64cf0 0x564d48a67180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:47.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.520+0000 7ff4a7400640 1 -- 172.21.15.28:0/4162945955 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564d48a64cf0 msgr2=0x564d48a67180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:08:47.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.520+0000 7ff4a7400640 1 --2- 172.21.15.28:0/4162945955 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564d48a64cf0 0x564d48a67180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:47.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.520+0000 7ff4a7400640 1 -- 172.21.15.28:0/4162945955 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d48a5f190 msgr2=0x564d48a5f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:08:47.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.520+0000 7ff4a7400640 1 --2- 172.21.15.28:0/4162945955 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d48a5f190 0x564d48a5f5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:47.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.520+0000 7ff4a7400640 1 -- 172.21.15.28:0/4162945955 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564d486d62d0 con 0x564d48a5fb80 2024-05-12T22:08:47.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.520+0000 7ff4a7e00640 1 --2- 172.21.15.28:0/4162945955 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d48a5f190 0x564d48a5f5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:08:47.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.520+0000 7ff4a7400640 1 --2- 172.21.15.28:0/4162945955 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564d48a5fb80 0x564d48a64680 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7ff4980ed000 tx=0x7ff4980f5740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=10db688425e81519 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:08:47.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.520+0000 7ff49cc00640 1 -- 172.21.15.28:0/4162945955 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4980efdb0 con 0x564d48a5fb80 2024-05-12T22:08:47.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.520+0000 7ff49cc00640 1 -- 172.21.15.28:0/4162945955 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff4980fc030 con 0x564d48a5fb80 2024-05-12T22:08:47.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.520+0000 7ff49cc00640 1 -- 172.21.15.28:0/4162945955 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff498101ac0 con 0x564d48a5fb80 2024-05-12T22:08:47.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.521+0000 7ff4b1439740 1 -- 172.21.15.28:0/4162945955 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564d48a5fb80 msgr2=0x564d48a64680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:47.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.521+0000 7ff4b1439740 1 --2- 172.21.15.28:0/4162945955 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564d48a5fb80 0x564d48a64680 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7ff4980ed000 tx=0x7ff4980f5740 comp rx=0 tx=0).stop 2024-05-12T22:08:47.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.521+0000 7ff4b1439740 1 -- 172.21.15.28:0/4162945955 shutdown_connections 2024-05-12T22:08:47.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.521+0000 7ff4b1439740 1 --2- 172.21.15.28:0/4162945955 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564d48a64cf0 0x564d48a67180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:47.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.521+0000 7ff4b1439740 1 --2- 172.21.15.28:0/4162945955 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564d48a5fb80 0x564d48a64680 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:47.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.521+0000 7ff4b1439740 1 --2- 172.21.15.28:0/4162945955 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d48a5f190 0x564d48a5f5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:47.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.521+0000 7ff4b1439740 1 -- 172.21.15.28:0/4162945955 >> 172.21.15.28:0/4162945955 conn(0x564d48a268a0 msgr2=0x564d48a5ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:08:47.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.521+0000 7ff4b1439740 1 -- 172.21.15.28:0/4162945955 shutdown_connections 2024-05-12T22:08:47.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.521+0000 7ff4b1439740 1 -- 172.21.15.28:0/4162945955 wait complete. 2024-05-12T22:08:47.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.521+0000 7ff4a4c00640 1 --2- 172.21.15.28:0/1059374608 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7ff4880a2aa0 0x7ff4880a4f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:47.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.522+0000 7ff4b1439740 1 -- start start 2024-05-12T22:08:47.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.522+0000 7ff4a4c00640 1 -- 172.21.15.28:0/1059374608 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7ff4980ef5b0 con 0x7ff4a8101760 2024-05-12T22:08:47.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.522+0000 7ff49fe00640 1 --2- 172.21.15.28:0/1059374608 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7ff4880a2aa0 0x7ff4880a4f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:47.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.522+0000 7ff4b1439740 1 Processor -- start 2024-05-12T22:08:47.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.522+0000 7ff4b1439740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d48a5f190 0x564d48a68f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:47.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.522+0000 7ff4b1439740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564d48a5fb80 0x564d48a69460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:47.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.523+0000 7ff4b1439740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564d48a64cf0 0x564d48a704a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:47.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.523+0000 7ff49fe00640 1 --2- 172.21.15.28:0/1059374608 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7ff4880a2aa0 0x7ff4880a4f60 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x564d48a5bac0 tx=0x7ff490005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:08:47.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.523+0000 7ff4b1439740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x564d48a67a60 con 0x564d48a5f190 2024-05-12T22:08:47.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.523+0000 7ff4b1439740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x564d4897d980 con 0x564d48a64cf0 2024-05-12T22:08:47.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.523+0000 7ff4b1439740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x564d48a67be0 con 0x564d48a5fb80 2024-05-12T22:08:47.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.523+0000 7ff4b1439740 10 client.0 _ll_register_callbacks cb 0x564d489b8fc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:08:47.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.523+0000 7ff4a7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d48a5f190 0x564d48a68f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:47.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.523+0000 7ff4a7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d48a5f190 0x564d48a68f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42560/0 (socket says 172.21.15.28:42560) 2024-05-12T22:08:47.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.523+0000 7ff4a7400640 1 -- 172.21.15.28:0/4187605925 learned_addr learned my addr 172.21.15.28:0/4187605925 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:08:47.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.523+0000 7ff4a7e00640 1 --2- 172.21.15.28:0/4187605925 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564d48a64cf0 0x564d48a704a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:47.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.523+0000 7ff49fe00640 1 --2- 172.21.15.28:0/4187605925 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564d48a5fb80 0x564d48a69460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:47.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.523+0000 7ff4a7400640 1 -- 172.21.15.28:0/4187605925 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564d48a5fb80 msgr2=0x564d48a69460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:08:47.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.523+0000 7ff4a7400640 1 --2- 172.21.15.28:0/4187605925 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564d48a5fb80 0x564d48a69460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:47.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.523+0000 7ff4a7400640 1 -- 172.21.15.28:0/4187605925 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564d48a64cf0 msgr2=0x564d48a704a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:08:47.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.523+0000 7ff4a7400640 1 --2- 172.21.15.28:0/4187605925 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564d48a64cf0 0x564d48a704a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:47.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.523+0000 7ff4a7400640 1 -- 172.21.15.28:0/4187605925 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564d487a06b0 con 0x564d48a5f190 2024-05-12T22:08:47.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.523+0000 7ff49fe00640 1 --2- 172.21.15.28:0/4187605925 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564d48a5fb80 0x564d48a69460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:08:47.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.524+0000 7ff4a7400640 1 --2- 172.21.15.28:0/4187605925 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d48a5f190 0x564d48a68f20 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7ff4980ecfd0 tx=0x7ff498005c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e96cfd0c3f0c8ea server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:08:47.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.524+0000 7ff495600640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-12T22:08:47.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.524+0000 7ff495600640 1 -- 172.21.15.28:0/4187605925 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff498049070 con 0x564d48a5f190 2024-05-12T22:08:47.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.524+0000 7ff495600640 1 -- 172.21.15.28:0/4187605925 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7ff498003800 con 0x564d48a5f190 2024-05-12T22:08:47.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.524+0000 7ff4b1439740 1 -- 172.21.15.28:0/4187605925 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff4980fc030 con 0x564d48a5f190 2024-05-12T22:08:47.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.524+0000 7ff495600640 1 -- 172.21.15.28:0/4187605925 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7ff4980b8080 con 0x564d48a5f190 2024-05-12T22:08:47.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.524+0000 7ff4b1439740 10 client.15033 fetch_fsmap learned FSMap version 23 2024-05-12T22:08:47.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.524+0000 7ff4b1439740 1 -- 172.21.15.28:0/4187605925 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x564d48a67a60 con 0x564d48a5f190 2024-05-12T22:08:47.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.524+0000 7ff495600640 1 -- 172.21.15.28:0/4187605925 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff4980fc030 con 0x564d48a5f190 2024-05-12T22:08:47.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.525+0000 7ff495600640 1 -- 172.21.15.28:0/4187605925 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7ff4980b8450 con 0x564d48a5f190 2024-05-12T22:08:47.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.525+0000 7ff4b1439740 10 client.15033 fetch_fsmap finished waiting for FSMap version 23 2024-05-12T22:08:47.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.525+0000 7ff4b1439740 10 client.15033 resolve_mds: resolved name 'g' to daemon mds.g 2024-05-12T22:08:47.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.525+0000 7ff4b1439740 20 client.15033 populate_metadata read hostname 'smithi028' 2024-05-12T22:08:47.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.525+0000 7ff4b1439740 1 --2- 172.21.15.28:0/4187605925 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x564d48a758e0 0x564d48a76d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:47.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.525+0000 7ff4b1439740 4 client.15033 mds_command: new command op to 24532 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-12T22:08:47.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.525+0000 7ff4b1439740 1 -- 172.21.15.28:0/4187605925 --> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564d489d6060 con 0x564d48a758e0 2024-05-12T22:08:47.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.525+0000 7ff49fe00640 1 --2- 172.21.15.28:0/4187605925 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x564d48a758e0 0x564d48a76d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:47.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.526+0000 7ff49fe00640 1 --2- 172.21.15.28:0/4187605925 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x564d48a758e0 0x564d48a76d50 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=23447f783f07e829 server_cookie=b3bb089ca97b5849 in_seq=0 out_seq=0 2024-05-12T22:08:47.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.526+0000 7ff495600640 10 client.15033 ms_handle_connect on v2:172.21.15.28:6838/123892158 2024-05-12T22:08:47.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.526+0000 7ff495600640 1 -- 172.21.15.28:0/4187605925 <== mds.2 v2:172.21.15.28:6838/123892158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564d489d6060 con 0x564d48a758e0 2024-05-12T22:08:47.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.526+0000 7ff495600640 10 client.15033 handle_command_reply: tid=0 2024-05-12T22:08:47.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.527+0000 7ff4b1439740 1 client.15033 shutdown 2024-05-12T22:08:47.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.527+0000 7ff4b1439740 1 -- 172.21.15.28:0/4187605925 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d48a5f190 msgr2=0x564d48a68f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:47.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.527+0000 7ff4b1439740 1 --2- 172.21.15.28:0/4187605925 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d48a5f190 0x564d48a68f20 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7ff4980ecfd0 tx=0x7ff498005c90 comp rx=0 tx=0).stop 2024-05-12T22:08:47.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.527+0000 7ff4b1439740 1 -- 172.21.15.28:0/4187605925 shutdown_connections 2024-05-12T22:08:47.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.527+0000 7ff4b1439740 1 --2- 172.21.15.28:0/4187605925 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x564d48a758e0 0x564d48a76d50 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:47.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.527+0000 7ff4b1439740 1 --2- 172.21.15.28:0/4187605925 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564d48a64cf0 0x564d48a704a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:47.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.527+0000 7ff4b1439740 1 --2- 172.21.15.28:0/4187605925 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564d48a5fb80 0x564d48a69460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:47.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.527+0000 7ff4b1439740 1 --2- 172.21.15.28:0/4187605925 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d48a5f190 0x564d48a68f20 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:47.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.527+0000 7ff4b1439740 1 -- 172.21.15.28:0/4187605925 >> 172.21.15.28:0/4187605925 conn(0x564d48a268a0 msgr2=0x564d48a59c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:08:47.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.527+0000 7ff495600640 0 client.15033 ms_handle_reset on v2:172.21.15.28:6838/123892158 2024-05-12T22:08:47.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.527+0000 7ff4b1439740 1 -- 172.21.15.28:0/4187605925 shutdown_connections 2024-05-12T22:08:47.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.527+0000 7ff4b1439740 1 -- 172.21.15.28:0/4187605925 wait complete. 2024-05-12T22:08:47.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.527+0000 7ff4b1439740 20 client.15033 trim_cache size 0 max 16384 2024-05-12T22:08:47.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.547+0000 7ff4b1439740 1 Processor -- start 2024-05-12T22:08:47.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.547+0000 7ff4b1439740 1 -- start start 2024-05-12T22:08:47.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.548+0000 7ff4b1439740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564d48a5f190 0x564d48a68b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:47.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.548+0000 7ff4b1439740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d48a5fb80 0x564d48a70430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:47.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.548+0000 7ff4b1439740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564d48a64cf0 0x564d48a69070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:47.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.548+0000 7ff4b1439740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x564d48a75090 con 0x564d48a5fb80 2024-05-12T22:08:47.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.548+0000 7ff4b1439740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x564d48a75760 con 0x564d48a5f190 2024-05-12T22:08:47.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.548+0000 7ff4b1439740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x564d48a75e30 con 0x564d48a64cf0 2024-05-12T22:08:47.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.548+0000 7ff49fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d48a5fb80 0x564d48a70430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:47.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.548+0000 7ff4a7400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564d48a5f190 0x564d48a68b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:47.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.548+0000 7ff49fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d48a5fb80 0x564d48a70430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42568/0 (socket says 172.21.15.28:42568) 2024-05-12T22:08:47.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.548+0000 7ff49fe00640 1 -- 172.21.15.28:0/4072096927 learned_addr learned my addr 172.21.15.28:0/4072096927 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:08:47.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.548+0000 7ff4a7400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564d48a5f190 0x564d48a68b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:37854/0 (socket says 172.21.15.28:37854) 2024-05-12T22:08:47.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.548+0000 7ff4a7e00640 1 --2- 172.21.15.28:0/4072096927 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564d48a64cf0 0x564d48a69070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:47.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.548+0000 7ff4a7400640 1 -- 172.21.15.28:0/4072096927 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564d48a64cf0 msgr2=0x564d48a69070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:08:47.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.548+0000 7ff4a7400640 1 --2- 172.21.15.28:0/4072096927 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564d48a64cf0 0x564d48a69070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:47.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.548+0000 7ff4a7400640 1 -- 172.21.15.28:0/4072096927 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d48a5fb80 msgr2=0x564d48a70430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:08:47.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.548+0000 7ff4a7400640 1 --2- 172.21.15.28:0/4072096927 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d48a5fb80 0x564d48a70430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:47.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.548+0000 7ff4a7400640 1 -- 172.21.15.28:0/4072096927 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564d48a5ff60 con 0x564d48a5f190 2024-05-12T22:08:47.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.549+0000 7ff4a7400640 1 --2- 172.21.15.28:0/4072096927 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564d48a5f190 0x564d48a68b30 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7ff498032ef0 tx=0x7ff4980ba380 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8f02bb77bcbab547 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:08:47.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.549+0000 7ff497e00640 1 -- 172.21.15.28:0/4072096927 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff498049070 con 0x564d48a5f190 2024-05-12T22:08:47.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.549+0000 7ff497e00640 1 -- 172.21.15.28:0/4072096927 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff4980fc030 con 0x564d48a5f190 2024-05-12T22:08:47.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.549+0000 7ff497e00640 1 -- 172.21.15.28:0/4072096927 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff498005d50 con 0x564d48a5f190 2024-05-12T22:08:47.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.550+0000 7ff4b1439740 1 -- 172.21.15.28:0/4072096927 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564d48a5f190 msgr2=0x564d48a68b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:47.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.550+0000 7ff4b1439740 1 --2- 172.21.15.28:0/4072096927 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564d48a5f190 0x564d48a68b30 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7ff498032ef0 tx=0x7ff4980ba380 comp rx=0 tx=0).stop 2024-05-12T22:08:47.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.550+0000 7ff4b1439740 1 -- 172.21.15.28:0/4072096927 shutdown_connections 2024-05-12T22:08:47.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.550+0000 7ff4b1439740 1 --2- 172.21.15.28:0/4072096927 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564d48a64cf0 0x564d48a69070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:47.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.550+0000 7ff4b1439740 1 --2- 172.21.15.28:0/4072096927 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d48a5fb80 0x564d48a70430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:47.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.550+0000 7ff4b1439740 1 --2- 172.21.15.28:0/4072096927 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564d48a5f190 0x564d48a68b30 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:47.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.550+0000 7ff4b1439740 1 -- 172.21.15.28:0/4072096927 >> 172.21.15.28:0/4072096927 conn(0x564d48a268a0 msgr2=0x564d48a641d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:08:47.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.550+0000 7ff4b1439740 1 -- 172.21.15.28:0/4072096927 shutdown_connections 2024-05-12T22:08:47.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.550+0000 7ff4b1439740 1 -- 172.21.15.28:0/4072096927 wait complete. 2024-05-12T22:08:47.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.550+0000 7ff4b1439740 1 -- start start 2024-05-12T22:08:47.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.550+0000 7ff4b1439740 1 Processor -- start 2024-05-12T22:08:47.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.551+0000 7ff4b1439740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564d48a5f190 0x564d48a62a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:47.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.551+0000 7ff4b1439740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564d48a5fb80 0x564d48a62f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:47.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.551+0000 7ff4b1439740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d48a64cf0 0x564d48ad9880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:47.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.551+0000 7ff4b1439740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x564d48a76bd0 con 0x564d48a64cf0 2024-05-12T22:08:47.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.551+0000 7ff4b1439740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x564d48a76500 con 0x564d48a5f190 2024-05-12T22:08:47.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.551+0000 7ff4b1439740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x564d48a772a0 con 0x564d48a5fb80 2024-05-12T22:08:47.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.551+0000 7ff4b1439740 10 client.0 _ll_register_callbacks cb 0x7ff4a0009ee0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:08:47.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.551+0000 7ff4a7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d48a64cf0 0x564d48ad9880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:47.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.551+0000 7ff4a7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d48a64cf0 0x564d48ad9880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42574/0 (socket says 172.21.15.28:42574) 2024-05-12T22:08:47.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.551+0000 7ff4a7400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564d48a5f190 0x564d48a62a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:47.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.551+0000 7ff4a7e00640 1 -- 172.21.15.28:0/1004970457 learned_addr learned my addr 172.21.15.28:0/1004970457 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:08:47.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.551+0000 7ff49fe00640 1 --2- 172.21.15.28:0/1004970457 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564d48a5fb80 0x564d48a62f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:47.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.552+0000 7ff4a7400640 1 -- 172.21.15.28:0/1004970457 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564d48a5fb80 msgr2=0x564d48a62f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:08:47.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.552+0000 7ff4a7400640 1 --2- 172.21.15.28:0/1004970457 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564d48a5fb80 0x564d48a62f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:47.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.552+0000 7ff4a7400640 1 -- 172.21.15.28:0/1004970457 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d48a64cf0 msgr2=0x564d48ad9880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:08:47.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.552+0000 7ff4a7400640 1 --2- 172.21.15.28:0/1004970457 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d48a64cf0 0x564d48ad9880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:47.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.552+0000 7ff4a7400640 1 -- 172.21.15.28:0/1004970457 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564d48a63580 con 0x564d48a5f190 2024-05-12T22:08:47.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.552+0000 7ff4a7e00640 1 --2- 172.21.15.28:0/1004970457 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d48a64cf0 0x564d48ad9880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:08:47.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.552+0000 7ff4a7400640 1 --2- 172.21.15.28:0/1004970457 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564d48a5f190 0x564d48a62a00 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x564d48a64cc0 tx=0x7ff4980ed370 comp rx=0 tx=0).ready entity=mon.2 client_cookie=256a130d986a1414 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:08:47.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.552+0000 7ff495600640 10 client.0 ms_handle_connect on v2:172.21.15.86:3300/0 2024-05-12T22:08:47.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.552+0000 7ff495600640 1 -- 172.21.15.28:0/1004970457 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff498049070 con 0x564d48a5f190 2024-05-12T22:08:47.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.552+0000 7ff495600640 1 -- 172.21.15.28:0/1004970457 <== mon.2 v2:172.21.15.86:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7ff49803b460 con 0x564d48a5f190 2024-05-12T22:08:47.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.552+0000 7ff4b1439740 1 -- 172.21.15.28:0/1004970457 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff4980fc030 con 0x564d48a5f190 2024-05-12T22:08:47.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.552+0000 7ff495600640 1 -- 172.21.15.28:0/1004970457 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7ff4980b9fb0 con 0x564d48a5f190 2024-05-12T22:08:47.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.552+0000 7ff4b1439740 10 client.25151 fetch_fsmap learned FSMap version 23 2024-05-12T22:08:47.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.552+0000 7ff4b1439740 1 -- 172.21.15.28:0/1004970457 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x564d48a63580 con 0x564d48a5f190 2024-05-12T22:08:47.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.553+0000 7ff495600640 1 -- 172.21.15.28:0/1004970457 <== mon.2 v2:172.21.15.86:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff4980fc030 con 0x564d48a5f190 2024-05-12T22:08:47.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.553+0000 7ff495600640 1 -- 172.21.15.28:0/1004970457 <== mon.2 v2:172.21.15.86:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7ff4980f5170 con 0x564d48a5f190 2024-05-12T22:08:47.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.553+0000 7ff4b1439740 10 client.25151 fetch_fsmap finished waiting for FSMap version 23 2024-05-12T22:08:47.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.553+0000 7ff4b1439740 10 client.25151 resolve_mds: resolved name 'g' to daemon mds.g 2024-05-12T22:08:47.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.553+0000 7ff4b1439740 20 client.25151 populate_metadata read hostname 'smithi028' 2024-05-12T22:08:47.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.553+0000 7ff4b1439740 1 --2- 172.21.15.28:0/1004970457 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x564d48adf410 0x564d48ade120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:47.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.553+0000 7ff4b1439740 4 client.25151 mds_command: new command op to 24532 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-12T22:08:47.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.553+0000 7ff4b1439740 1 -- 172.21.15.28:0/1004970457 --> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] -- command(tid 0: {"prefix": "status"}) -- 0x564d489cfbb0 con 0x564d48adf410 2024-05-12T22:08:47.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.553+0000 7ff49fe00640 1 --2- 172.21.15.28:0/1004970457 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x564d48adf410 0x564d48ade120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:47.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.554+0000 7ff495600640 10 client.25151 ms_handle_connect on v2:172.21.15.28:6838/123892158 2024-05-12T22:08:47.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.554+0000 7ff49fe00640 1 --2- 172.21.15.28:0/1004970457 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x564d48adf410 0x564d48ade120 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=20af969c4cd401eb server_cookie=e5dfd498fc817d28 in_seq=0 out_seq=0 2024-05-12T22:08:47.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.554+0000 7ff495600640 1 -- 172.21.15.28:0/1004970457 <== mds.2 v2:172.21.15.28:6838/123892158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x564d489cfbb0 con 0x564d48adf410 2024-05-12T22:08:47.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.554+0000 7ff495600640 10 client.25151 handle_command_reply: tid=0 2024-05-12T22:08:47.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.554+0000 7ff4b1439740 1 client.25151 shutdown 2024-05-12T22:08:47.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.555+0000 7ff4b1439740 1 -- 172.21.15.28:0/1004970457 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564d48a5f190 msgr2=0x564d48a62a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:47.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.555+0000 7ff4b1439740 1 --2- 172.21.15.28:0/1004970457 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564d48a5f190 0x564d48a62a00 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x564d48a64cc0 tx=0x7ff4980ed370 comp rx=0 tx=0).stop 2024-05-12T22:08:47.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.555+0000 7ff4b1439740 1 -- 172.21.15.28:0/1004970457 shutdown_connections 2024-05-12T22:08:47.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.555+0000 7ff4b1439740 1 --2- 172.21.15.28:0/1004970457 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d48a64cf0 0x564d48ad9880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:47.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.555+0000 7ff4b1439740 1 --2- 172.21.15.28:0/1004970457 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564d48a5fb80 0x564d48a62f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:47.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.555+0000 7ff4b1439740 1 --2- 172.21.15.28:0/1004970457 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x564d48adf410 0x564d48ade120 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:47.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.555+0000 7ff4b1439740 1 --2- 172.21.15.28:0/1004970457 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564d48a5f190 0x564d48a62a00 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:47.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.555+0000 7ff4b1439740 1 -- 172.21.15.28:0/1004970457 >> 172.21.15.28:0/1004970457 conn(0x564d48a268a0 msgr2=0x564d48a68ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:08:47.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.555+0000 7ff495600640 0 client.25151 ms_handle_reset on v2:172.21.15.28:6838/123892158 2024-05-12T22:08:47.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.555+0000 7ff4b1439740 1 -- 172.21.15.28:0/1004970457 shutdown_connections 2024-05-12T22:08:47.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.555+0000 7ff4b1439740 1 -- 172.21.15.28:0/1004970457 wait complete. 2024-05-12T22:08:47.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.555+0000 7ff4b1439740 20 client.25151 trim_cache size 0 max 16384 2024-05-12T22:08:47.561 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-05-12T22:08:47.561 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "6b92e7b4-10aa-11ef-bc98-c7b262605968", 2024-05-12T22:08:47.561 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 2, 2024-05-12T22:08:47.561 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 24532, 2024-05-12T22:08:47.561 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-05-12T22:08:47.561 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-05-12T22:08:47.561 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-05-12T22:08:47.561 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 129.241917812, 2024-05-12T22:08:47.561 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 23, 2024-05-12T22:08:47.561 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 88, 2024-05-12T22:08:47.562 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 86, 2024-05-12T22:08:47.562 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 146.80792529499999 2024-05-12T22:08:47.562 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-05-12T22:08:47.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.556+0000 7ff4ad000640 1 -- 172.21.15.28:0/1059374608 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7ff4880a2aa0 msgr2=0x7ff4880a4f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:47.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.556+0000 7ff4ad000640 1 --2- 172.21.15.28:0/1059374608 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7ff4880a2aa0 0x7ff4880a4f60 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x564d48a5bac0 tx=0x7ff490005f90 comp rx=0 tx=0).stop 2024-05-12T22:08:47.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.556+0000 7ff4ad000640 1 -- 172.21.15.28:0/1059374608 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4a8101760 msgr2=0x7ff4a8199c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:47.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.556+0000 7ff4ad000640 1 --2- 172.21.15.28:0/1059374608 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4a8101760 0x7ff4a8199c70 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7ff498031440 tx=0x7ff498032ce0 comp rx=0 tx=0).stop 2024-05-12T22:08:47.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.556+0000 7ff4ad000640 1 -- 172.21.15.28:0/1059374608 shutdown_connections 2024-05-12T22:08:47.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.556+0000 7ff4ad000640 1 --2- 172.21.15.28:0/1059374608 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7ff4880a2aa0 0x7ff4880a4f60 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:47.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.556+0000 7ff4ad000640 1 --2- 172.21.15.28:0/1059374608 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff4a8103310 0x7ff4a81a11e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:47.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.556+0000 7ff4ad000640 1 --2- 172.21.15.28:0/1059374608 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff4a8102950 0x7ff4a819a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:47.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.556+0000 7ff4ad000640 1 --2- 172.21.15.28:0/1059374608 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4a8101760 0x7ff4a8199c70 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:47.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.556+0000 7ff4ad000640 1 -- 172.21.15.28:0/1059374608 >> 172.21.15.28:0/1059374608 conn(0x7ff4a80fccd0 msgr2=0x7ff4a8073a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:08:47.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.556+0000 7ff4ad000640 1 -- 172.21.15.28:0/1059374608 shutdown_connections 2024-05-12T22:08:47.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.556+0000 7ff4ad000640 1 -- 172.21.15.28:0/1059374608 wait complete. 2024-05-12T22:08:47.575 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-05-12T22:08:47.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.779+0000 7f9657800640 1 -- 172.21.15.28:0/1068371748 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f96501022f0 msgr2=0x7f96501046e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:47.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.779+0000 7f9657800640 1 --2- 172.21.15.28:0/1068371748 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f96501022f0 0x7f96501046e0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f9644009860 tx=0x7f964402a340 comp rx=0 tx=0).stop 2024-05-12T22:08:47.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.779+0000 7f9657800640 1 -- 172.21.15.28:0/1068371748 shutdown_connections 2024-05-12T22:08:47.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.779+0000 7f9657800640 1 --2- 172.21.15.28:0/1068371748 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9650107550 0x7f96501099a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:47.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.779+0000 7f9657800640 1 --2- 172.21.15.28:0/1068371748 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9650104c20 0x7f9650107010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:47.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.779+0000 7f9657800640 1 --2- 172.21.15.28:0/1068371748 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f96501022f0 0x7f96501046e0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:47.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.779+0000 7f9657800640 1 -- 172.21.15.28:0/1068371748 >> 172.21.15.28:0/1068371748 conn(0x7f96500fbce0 msgr2=0x7f96500fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:08:47.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.779+0000 7f9657800640 1 -- 172.21.15.28:0/1068371748 shutdown_connections 2024-05-12T22:08:47.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.779+0000 7f9657800640 1 -- 172.21.15.28:0/1068371748 wait complete. 2024-05-12T22:08:47.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.780+0000 7f9657800640 1 Processor -- start 2024-05-12T22:08:47.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.780+0000 7f9657800640 1 -- start start 2024-05-12T22:08:47.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.781+0000 7f9657800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f96501022f0 0x7f9650197bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:47.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.781+0000 7f9657800640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9650104c20 0x7f96501980f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:47.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.781+0000 7f9657800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9650107550 0x7f965019f160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:47.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.781+0000 7f9657800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f965010d510 con 0x7f96501022f0 2024-05-12T22:08:47.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.781+0000 7f9657800640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f965010d390 con 0x7f9650107550 2024-05-12T22:08:47.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.781+0000 7f9657800640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f965010d690 con 0x7f9650104c20 2024-05-12T22:08:47.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.781+0000 7f9654e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f96501022f0 0x7f9650197bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:47.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.781+0000 7f9654e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f96501022f0 0x7f9650197bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42592/0 (socket says 172.21.15.28:42592) 2024-05-12T22:08:47.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.781+0000 7f9654e00640 1 -- 172.21.15.28:0/823555382 learned_addr learned my addr 172.21.15.28:0/823555382 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:08:47.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.781+0000 7f9655800640 1 --2- 172.21.15.28:0/823555382 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9650107550 0x7f965019f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:47.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.781+0000 7f964fe00640 1 --2- 172.21.15.28:0/823555382 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9650104c20 0x7f96501980f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:47.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.781+0000 7f9654e00640 1 -- 172.21.15.28:0/823555382 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9650104c20 msgr2=0x7f96501980f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:47.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.781+0000 7f9654e00640 1 --2- 172.21.15.28:0/823555382 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9650104c20 0x7f96501980f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:47.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.781+0000 7f9654e00640 1 -- 172.21.15.28:0/823555382 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9650107550 msgr2=0x7f965019f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:47.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.781+0000 7f9654e00640 1 --2- 172.21.15.28:0/823555382 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9650107550 0x7f965019f160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:47.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.781+0000 7f9654e00640 1 -- 172.21.15.28:0/823555382 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f965019f7d0 con 0x7f96501022f0 2024-05-12T22:08:47.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.782+0000 7f9654e00640 1 --2- 172.21.15.28:0/823555382 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f96501022f0 0x7f9650197bb0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f9644005940 tx=0x7f9644033d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:08:47.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.782+0000 7f964d600640 1 -- 172.21.15.28:0/823555382 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f964403f020 con 0x7f96501022f0 2024-05-12T22:08:47.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.782+0000 7f964d600640 1 -- 172.21.15.28:0/823555382 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9644030850 con 0x7f96501022f0 2024-05-12T22:08:47.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.782+0000 7f9657800640 1 -- 172.21.15.28:0/823555382 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f965019fa60 con 0x7f96501022f0 2024-05-12T22:08:47.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.782+0000 7f964d600640 1 -- 172.21.15.28:0/823555382 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9644037040 con 0x7f96501022f0 2024-05-12T22:08:47.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.782+0000 7f9657800640 1 -- 172.21.15.28:0/823555382 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f965019ffa0 con 0x7f96501022f0 2024-05-12T22:08:47.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.786+0000 7f965a5f2740 1 Processor -- start 2024-05-12T22:08:47.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.786+0000 7f965a5f2740 1 -- start start 2024-05-12T22:08:47.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.786+0000 7f965a5f2740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555c9dc8bfb0 0x555c9dc8c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:47.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.786+0000 7f965a5f2740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x555c9dc8c9a0 0x555c9dc914a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:47.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.786+0000 7f965a5f2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c9dc91b10 0x555c9dc93fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:47.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.786+0000 7f965a5f2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x555c9dbef480 con 0x555c9dc91b10 2024-05-12T22:08:47.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.786+0000 7f965a5f2740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x555c9db975c0 con 0x555c9dc8c9a0 2024-05-12T22:08:47.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.786+0000 7f965a5f2740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x555c9dbeb440 con 0x555c9dc8bfb0 2024-05-12T22:08:47.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.786+0000 7f964fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c9dc91b10 0x555c9dc93fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:47.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.787+0000 7f9655800640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555c9dc8bfb0 0x555c9dc8c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:47.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.787+0000 7f964fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c9dc91b10 0x555c9dc93fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42598/0 (socket says 172.21.15.28:42598) 2024-05-12T22:08:47.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.787+0000 7f964fe00640 1 -- 172.21.15.28:0/378509548 learned_addr learned my addr 172.21.15.28:0/378509548 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:08:47.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.787+0000 7f9655800640 1 -- 172.21.15.28:0/378509548 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x555c9dc8c9a0 msgr2=0x555c9dc914a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-12T22:08:47.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.787+0000 7f9655800640 1 --2- 172.21.15.28:0/378509548 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x555c9dc8c9a0 0x555c9dc914a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:47.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.787+0000 7f9655800640 1 -- 172.21.15.28:0/378509548 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c9dc91b10 msgr2=0x555c9dc93fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:08:47.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.787+0000 7f9655800640 1 --2- 172.21.15.28:0/378509548 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c9dc91b10 0x555c9dc93fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:47.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.787+0000 7f9655800640 1 -- 172.21.15.28:0/378509548 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555c9d9032d0 con 0x555c9dc8bfb0 2024-05-12T22:08:47.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.787+0000 7f964fe00640 1 --2- 172.21.15.28:0/378509548 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c9dc91b10 0x555c9dc93fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:08:47.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.787+0000 7f964d600640 1 -- 172.21.15.28:0/823555382 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f9644042020 con 0x7f96501022f0 2024-05-12T22:08:47.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.787+0000 7f9655800640 1 --2- 172.21.15.28:0/378509548 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555c9dc8bfb0 0x555c9dc8c3d0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f9648009f00 tx=0x7f9648011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dc0608d83048ca43 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:08:47.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.788+0000 7f9642000640 1 -- 172.21.15.28:0/378509548 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9648004d20 con 0x555c9dc8bfb0 2024-05-12T22:08:47.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.788+0000 7f9642000640 1 -- 172.21.15.28:0/378509548 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9648004030 con 0x555c9dc8bfb0 2024-05-12T22:08:47.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.788+0000 7f9642000640 1 -- 172.21.15.28:0/378509548 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9648005260 con 0x555c9dc8bfb0 2024-05-12T22:08:47.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.789+0000 7f965a5f2740 1 -- 172.21.15.28:0/378509548 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555c9dc8bfb0 msgr2=0x555c9dc8c3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:47.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.789+0000 7f965a5f2740 1 --2- 172.21.15.28:0/378509548 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555c9dc8bfb0 0x555c9dc8c3d0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f9648009f00 tx=0x7f9648011810 comp rx=0 tx=0).stop 2024-05-12T22:08:47.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.789+0000 7f964d600640 1 --2- 172.21.15.28:0/823555382 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f96380a2c10 0x7f96380a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:47.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.789+0000 7f965a5f2740 1 -- 172.21.15.28:0/378509548 shutdown_connections 2024-05-12T22:08:47.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.789+0000 7f964d600640 1 -- 172.21.15.28:0/823555382 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f96440ef290 con 0x7f96501022f0 2024-05-12T22:08:47.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.789+0000 7f965a5f2740 1 --2- 172.21.15.28:0/378509548 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c9dc91b10 0x555c9dc93fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:47.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.789+0000 7f965a5f2740 1 --2- 172.21.15.28:0/378509548 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x555c9dc8c9a0 0x555c9dc914a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:47.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.789+0000 7f965a5f2740 1 --2- 172.21.15.28:0/378509548 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555c9dc8bfb0 0x555c9dc8c3d0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:47.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.789+0000 7f965a5f2740 1 -- 172.21.15.28:0/378509548 >> 172.21.15.28:0/378509548 conn(0x555c9dc538a0 msgr2=0x555c9dc87a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:08:47.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.789+0000 7f9655800640 1 --2- 172.21.15.28:0/823555382 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f96380a2c10 0x7f96380a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:47.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.789+0000 7f965a5f2740 1 -- 172.21.15.28:0/378509548 shutdown_connections 2024-05-12T22:08:47.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.789+0000 7f965a5f2740 1 -- 172.21.15.28:0/378509548 wait complete. 2024-05-12T22:08:47.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.790+0000 7f965a5f2740 1 -- start start 2024-05-12T22:08:47.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.790+0000 7f9655800640 1 --2- 172.21.15.28:0/823555382 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f96380a2c10 0x7f96380a50d0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f9648009d40 tx=0x7f9648004870 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:08:47.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.790+0000 7f965a5f2740 1 Processor -- start 2024-05-12T22:08:47.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.790+0000 7f965a5f2740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555c9dc8bfb0 0x555c9dc96130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:47.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.790+0000 7f965a5f2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c9dc8c9a0 0x555c9dc96670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:47.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.790+0000 7f965a5f2740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x555c9dc91b10 0x555c9dc9d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:47.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.790+0000 7f965a5f2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x555c9dc94ac0 con 0x555c9dc8c9a0 2024-05-12T22:08:47.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.790+0000 7f965a5f2740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x555c9dbaa980 con 0x555c9dc91b10 2024-05-12T22:08:47.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.790+0000 7f965a5f2740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x555c9dc94c40 con 0x555c9dc8bfb0 2024-05-12T22:08:47.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.791+0000 7f965a5f2740 10 client.0 _ll_register_callbacks cb 0x555c9dbe5fc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:08:47.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.791+0000 7f964fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c9dc8c9a0 0x555c9dc96670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:47.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.791+0000 7f9654e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555c9dc8bfb0 0x555c9dc96130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:47.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.791+0000 7f964fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c9dc8c9a0 0x555c9dc96670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42608/0 (socket says 172.21.15.28:42608) 2024-05-12T22:08:47.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.791+0000 7f964fe00640 1 -- 172.21.15.28:0/1511013373 learned_addr learned my addr 172.21.15.28:0/1511013373 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:08:47.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.791+0000 7f9654e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555c9dc8bfb0 0x555c9dc96130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:37880/0 (socket says 172.21.15.28:37880) 2024-05-12T22:08:47.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.791+0000 7f9655800640 1 --2- 172.21.15.28:0/1511013373 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x555c9dc91b10 0x555c9dc9d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:47.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.791+0000 7f9654e00640 1 -- 172.21.15.28:0/1511013373 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x555c9dc91b10 msgr2=0x555c9dc9d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:08:47.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.791+0000 7f9654e00640 1 --2- 172.21.15.28:0/1511013373 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x555c9dc91b10 0x555c9dc9d6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:47.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.791+0000 7f9654e00640 1 -- 172.21.15.28:0/1511013373 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c9dc8c9a0 msgr2=0x555c9dc96670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:08:47.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.791+0000 7f9654e00640 1 --2- 172.21.15.28:0/1511013373 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c9dc8c9a0 0x555c9dc96670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:47.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.791+0000 7f9654e00640 1 -- 172.21.15.28:0/1511013373 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555c9d9cd6b0 con 0x555c9dc8bfb0 2024-05-12T22:08:47.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.791+0000 7f964fe00640 1 --2- 172.21.15.28:0/1511013373 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c9dc8c9a0 0x555c9dc96670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:08:47.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.791+0000 7f9654e00640 1 --2- 172.21.15.28:0/1511013373 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555c9dc8bfb0 0x555c9dc96130 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x555c9dc8d410 tx=0x7f9644009560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cc79d0ed7126219 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:08:47.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.792+0000 7f9636a00640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-12T22:08:47.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.792+0000 7f9636a00640 1 -- 172.21.15.28:0/1511013373 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96440d4060 con 0x555c9dc8bfb0 2024-05-12T22:08:47.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.792+0000 7f9636a00640 1 -- 172.21.15.28:0/1511013373 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f96440c42e0 con 0x555c9dc8bfb0 2024-05-12T22:08:47.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.792+0000 7f965a5f2740 1 -- 172.21.15.28:0/1511013373 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9648004030 con 0x555c9dc8bfb0 2024-05-12T22:08:47.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.792+0000 7f9636a00640 1 -- 172.21.15.28:0/1511013373 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f96440bb460 con 0x555c9dc8bfb0 2024-05-12T22:08:47.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.792+0000 7f965a5f2740 10 client.25144 fetch_fsmap learned FSMap version 23 2024-05-12T22:08:47.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.792+0000 7f965a5f2740 1 -- 172.21.15.28:0/1511013373 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x555c9dc94b30 con 0x555c9dc8bfb0 2024-05-12T22:08:47.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.792+0000 7f9636a00640 1 -- 172.21.15.28:0/1511013373 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9648004030 con 0x555c9dc8bfb0 2024-05-12T22:08:47.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.792+0000 7f9636a00640 1 -- 172.21.15.28:0/1511013373 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f964403a030 con 0x555c9dc8bfb0 2024-05-12T22:08:47.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.793+0000 7f965a5f2740 10 client.25144 fetch_fsmap finished waiting for FSMap version 23 2024-05-12T22:08:47.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.793+0000 7f965a5f2740 10 client.25144 resolve_mds: resolved name 'a' to daemon mds.a 2024-05-12T22:08:47.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.793+0000 7f965a5f2740 20 client.25144 populate_metadata read hostname 'smithi028' 2024-05-12T22:08:47.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.793+0000 7f965a5f2740 1 --2- 172.21.15.28:0/1511013373 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x555c9dca2c30 0x555c9dca40a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:47.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.793+0000 7f965a5f2740 4 client.25144 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-12T22:08:47.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.793+0000 7f965a5f2740 1 -- 172.21.15.28:0/1511013373 --> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555c9dc03060 con 0x555c9dca2c30 2024-05-12T22:08:47.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.793+0000 7f964fe00640 1 --2- 172.21.15.28:0/1511013373 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x555c9dca2c30 0x555c9dca40a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:47.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.793+0000 7f9636a00640 10 client.25144 ms_handle_connect on v2:172.21.15.28:6840/761183870 2024-05-12T22:08:47.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.793+0000 7f964fe00640 1 --2- 172.21.15.28:0/1511013373 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x555c9dca2c30 0x555c9dca40a0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=4fbec62aac8e602e server_cookie=9fe15bce8924b89 in_seq=0 out_seq=0 2024-05-12T22:08:47.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.794+0000 7f9636a00640 1 -- 172.21.15.28:0/1511013373 <== mds.1 v2:172.21.15.28:6840/761183870 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555c9dc03060 con 0x555c9dca2c30 2024-05-12T22:08:47.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.794+0000 7f9636a00640 10 client.25144 handle_command_reply: tid=0 2024-05-12T22:08:47.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.794+0000 7f965a5f2740 1 client.25144 shutdown 2024-05-12T22:08:47.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.795+0000 7f965a5f2740 1 -- 172.21.15.28:0/1511013373 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555c9dc8bfb0 msgr2=0x555c9dc96130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:47.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.795+0000 7f965a5f2740 1 --2- 172.21.15.28:0/1511013373 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555c9dc8bfb0 0x555c9dc96130 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x555c9dc8d410 tx=0x7f9644009560 comp rx=0 tx=0).stop 2024-05-12T22:08:47.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.795+0000 7f965a5f2740 1 -- 172.21.15.28:0/1511013373 shutdown_connections 2024-05-12T22:08:47.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.795+0000 7f965a5f2740 1 --2- 172.21.15.28:0/1511013373 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x555c9dca2c30 0x555c9dca40a0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:47.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.795+0000 7f965a5f2740 1 --2- 172.21.15.28:0/1511013373 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x555c9dc91b10 0x555c9dc9d6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:47.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.795+0000 7f965a5f2740 1 --2- 172.21.15.28:0/1511013373 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c9dc8c9a0 0x555c9dc96670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:47.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.795+0000 7f965a5f2740 1 --2- 172.21.15.28:0/1511013373 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555c9dc8bfb0 0x555c9dc96130 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:47.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.795+0000 7f9636a00640 0 client.25144 ms_handle_reset on v2:172.21.15.28:6840/761183870 2024-05-12T22:08:47.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.795+0000 7f965a5f2740 1 -- 172.21.15.28:0/1511013373 >> 172.21.15.28:0/1511013373 conn(0x555c9dc538a0 msgr2=0x555c9dc86b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:08:47.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.795+0000 7f965a5f2740 1 -- 172.21.15.28:0/1511013373 shutdown_connections 2024-05-12T22:08:47.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.795+0000 7f965a5f2740 1 -- 172.21.15.28:0/1511013373 wait complete. 2024-05-12T22:08:47.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.795+0000 7f965a5f2740 20 client.25144 trim_cache size 0 max 16384 2024-05-12T22:08:47.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.816+0000 7f965a5f2740 1 Processor -- start 2024-05-12T22:08:47.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.816+0000 7f965a5f2740 1 -- start start 2024-05-12T22:08:47.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.816+0000 7f965a5f2740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x555c9dc8bfb0 0x555c9dc95010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:47.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.816+0000 7f965a5f2740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555c9dc8c9a0 0x555c9dc9d670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:47.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.816+0000 7f965a5f2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c9dc91b10 0x555c9dc95550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:47.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.816+0000 7f965a5f2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x555c9dca23e0 con 0x555c9dc91b10 2024-05-12T22:08:47.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.816+0000 7f965a5f2740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x555c9dca2ab0 con 0x555c9dc8bfb0 2024-05-12T22:08:47.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.816+0000 7f965a5f2740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x555c9dca3180 con 0x555c9dc8c9a0 2024-05-12T22:08:47.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.816+0000 7f9655800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c9dc91b10 0x555c9dc95550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:47.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.816+0000 7f9654e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x555c9dc8bfb0 0x555c9dc95010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:47.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.816+0000 7f964fe00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555c9dc8c9a0 0x555c9dc9d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:47.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.816+0000 7f9655800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c9dc91b10 0x555c9dc95550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42610/0 (socket says 172.21.15.28:42610) 2024-05-12T22:08:47.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.816+0000 7f9655800640 1 -- 172.21.15.28:0/2127331028 learned_addr learned my addr 172.21.15.28:0/2127331028 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:08:47.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.816+0000 7f9654e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x555c9dc8bfb0 0x555c9dc95010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:37892/0 (socket says 172.21.15.28:37892) 2024-05-12T22:08:47.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.817+0000 7f9654e00640 1 -- 172.21.15.28:0/2127331028 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555c9dc8c9a0 msgr2=0x555c9dc9d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:08:47.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.817+0000 7f9654e00640 1 --2- 172.21.15.28:0/2127331028 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555c9dc8c9a0 0x555c9dc9d670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:47.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.817+0000 7f9654e00640 1 -- 172.21.15.28:0/2127331028 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c9dc91b10 msgr2=0x555c9dc95550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:08:47.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.817+0000 7f9654e00640 1 --2- 172.21.15.28:0/2127331028 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c9dc91b10 0x555c9dc95550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:47.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.817+0000 7f9654e00640 1 -- 172.21.15.28:0/2127331028 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555c9dc8cd80 con 0x555c9dc8bfb0 2024-05-12T22:08:47.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.817+0000 7f964fe00640 1 --2- 172.21.15.28:0/2127331028 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555c9dc8c9a0 0x555c9dc9d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:08:47.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.817+0000 7f9655800640 1 --2- 172.21.15.28:0/2127331028 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c9dc91b10 0x555c9dc95550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:08:47.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.817+0000 7f9654e00640 1 --2- 172.21.15.28:0/2127331028 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x555c9dc8bfb0 0x555c9dc95010 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f96440b8480 tx=0x7f96440ba010 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9adb40a5f5780c5f server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:08:47.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.817+0000 7f9641600640 1 -- 172.21.15.28:0/2127331028 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96440d4050 con 0x555c9dc8bfb0 2024-05-12T22:08:47.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.818+0000 7f9641600640 1 -- 172.21.15.28:0/2127331028 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9644030a00 con 0x555c9dc8bfb0 2024-05-12T22:08:47.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.818+0000 7f9641600640 1 -- 172.21.15.28:0/2127331028 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96440bb1d0 con 0x555c9dc8bfb0 2024-05-12T22:08:47.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.818+0000 7f965a5f2740 1 -- 172.21.15.28:0/2127331028 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x555c9dc8bfb0 msgr2=0x555c9dc95010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:47.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.818+0000 7f965a5f2740 1 --2- 172.21.15.28:0/2127331028 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x555c9dc8bfb0 0x555c9dc95010 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f96440b8480 tx=0x7f96440ba010 comp rx=0 tx=0).stop 2024-05-12T22:08:47.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.818+0000 7f965a5f2740 1 -- 172.21.15.28:0/2127331028 shutdown_connections 2024-05-12T22:08:47.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.818+0000 7f965a5f2740 1 --2- 172.21.15.28:0/2127331028 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c9dc91b10 0x555c9dc95550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:47.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.818+0000 7f965a5f2740 1 --2- 172.21.15.28:0/2127331028 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555c9dc8c9a0 0x555c9dc9d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:47.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.818+0000 7f965a5f2740 1 --2- 172.21.15.28:0/2127331028 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x555c9dc8bfb0 0x555c9dc95010 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:47.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.818+0000 7f965a5f2740 1 -- 172.21.15.28:0/2127331028 >> 172.21.15.28:0/2127331028 conn(0x555c9dc538a0 msgr2=0x555c9dc91000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:08:47.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.818+0000 7f965a5f2740 1 -- 172.21.15.28:0/2127331028 shutdown_connections 2024-05-12T22:08:47.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.819+0000 7f965a5f2740 1 -- 172.21.15.28:0/2127331028 wait complete. 2024-05-12T22:08:47.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.819+0000 7f965a5f2740 1 -- start start 2024-05-12T22:08:47.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.819+0000 7f965a5f2740 1 Processor -- start 2024-05-12T22:08:47.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.819+0000 7f965a5f2740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555c9dc8bfb0 0x555c9dc8f890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:47.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.819+0000 7f965a5f2740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x555c9dc8c9a0 0x555c9dc8fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:47.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.819+0000 7f965a5f2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c9dc91b10 0x555c9dd0aed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:47.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.820+0000 7f965a5f2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x555c9dca3f20 con 0x555c9dc91b10 2024-05-12T22:08:47.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.820+0000 7f965a5f2740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x555c9dca3850 con 0x555c9dc8c9a0 2024-05-12T22:08:47.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.820+0000 7f965a5f2740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x555c9dca45f0 con 0x555c9dc8bfb0 2024-05-12T22:08:47.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.820+0000 7f965a5f2740 10 client.0 _ll_register_callbacks cb 0x7f96440097b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:08:47.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.820+0000 7f9655800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c9dc91b10 0x555c9dd0aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:47.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.820+0000 7f9655800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c9dc91b10 0x555c9dd0aed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42620/0 (socket says 172.21.15.28:42620) 2024-05-12T22:08:47.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.820+0000 7f9655800640 1 -- 172.21.15.28:0/1059464118 learned_addr learned my addr 172.21.15.28:0/1059464118 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:08:47.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.820+0000 7f964fe00640 1 --2- 172.21.15.28:0/1059464118 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x555c9dc8c9a0 0x555c9dc8fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:47.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.820+0000 7f9654e00640 1 --2- 172.21.15.28:0/1059464118 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555c9dc8bfb0 0x555c9dc8f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:47.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.820+0000 7f964fe00640 1 -- 172.21.15.28:0/1059464118 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555c9dc8bfb0 msgr2=0x555c9dc8f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:08:47.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.820+0000 7f964fe00640 1 --2- 172.21.15.28:0/1059464118 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555c9dc8bfb0 0x555c9dc8f890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:47.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.820+0000 7f964fe00640 1 -- 172.21.15.28:0/1059464118 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c9dc91b10 msgr2=0x555c9dd0aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:08:47.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.820+0000 7f964fe00640 1 --2- 172.21.15.28:0/1059464118 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c9dc91b10 0x555c9dd0aed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:47.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.820+0000 7f964fe00640 1 -- 172.21.15.28:0/1059464118 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555c9dd0b4e0 con 0x555c9dc8c9a0 2024-05-12T22:08:47.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.821+0000 7f964fe00640 1 --2- 172.21.15.28:0/1059464118 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x555c9dc8c9a0 0x555c9dc8fdd0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x555c9dca4ee0 tx=0x7f963c0039c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2ab937e8a126d1d4 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:08:47.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.821+0000 7f9636a00640 10 client.0 ms_handle_connect on v2:172.21.15.86:3300/0 2024-05-12T22:08:47.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.821+0000 7f9636a00640 1 -- 172.21.15.28:0/1059464118 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f963c0205b0 con 0x555c9dc8c9a0 2024-05-12T22:08:47.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.821+0000 7f9636a00640 1 -- 172.21.15.28:0/1059464118 <== mon.2 v2:172.21.15.86:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f963c020750 con 0x555c9dc8c9a0 2024-05-12T22:08:47.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.821+0000 7f965a5f2740 1 -- 172.21.15.28:0/1059464118 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9644030a00 con 0x555c9dc8c9a0 2024-05-12T22:08:47.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.821+0000 7f9636a00640 1 -- 172.21.15.28:0/1059464118 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f963c010430 con 0x555c9dc8c9a0 2024-05-12T22:08:47.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.821+0000 7f965a5f2740 10 client.25166 fetch_fsmap learned FSMap version 23 2024-05-12T22:08:47.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.821+0000 7f965a5f2740 1 -- 172.21.15.28:0/1059464118 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x555c9dd0c8b0 con 0x555c9dc8c9a0 2024-05-12T22:08:47.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.822+0000 7f9636a00640 1 -- 172.21.15.28:0/1059464118 <== mon.2 v2:172.21.15.86:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9644030a00 con 0x555c9dc8c9a0 2024-05-12T22:08:47.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.822+0000 7f9636a00640 1 -- 172.21.15.28:0/1059464118 <== mon.2 v2:172.21.15.86:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f963c029750 con 0x555c9dc8c9a0 2024-05-12T22:08:47.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.822+0000 7f965a5f2740 10 client.25166 fetch_fsmap finished waiting for FSMap version 23 2024-05-12T22:08:47.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.822+0000 7f965a5f2740 10 client.25166 resolve_mds: resolved name 'a' to daemon mds.a 2024-05-12T22:08:47.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.822+0000 7f965a5f2740 20 client.25166 populate_metadata read hostname 'smithi028' 2024-05-12T22:08:47.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.822+0000 7f965a5f2740 1 --2- 172.21.15.28:0/1059464118 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x555c9dd11440 0x555c9dd10150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:47.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.822+0000 7f965a5f2740 4 client.25166 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-12T22:08:47.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.822+0000 7f965a5f2740 1 -- 172.21.15.28:0/1059464118 --> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] -- command(tid 0: {"prefix": "status"}) -- 0x555c9dbfcbb0 con 0x555c9dd11440 2024-05-12T22:08:47.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.822+0000 7f9654e00640 1 --2- 172.21.15.28:0/1059464118 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x555c9dd11440 0x555c9dd10150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:47.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.823+0000 7f9654e00640 1 --2- 172.21.15.28:0/1059464118 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x555c9dd11440 0x555c9dd10150 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=603c1c8493f9b6c2 server_cookie=b929ddd9f2a37a23 in_seq=0 out_seq=0 2024-05-12T22:08:47.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.823+0000 7f9636a00640 10 client.25166 ms_handle_connect on v2:172.21.15.28:6840/761183870 2024-05-12T22:08:47.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.823+0000 7f9636a00640 1 -- 172.21.15.28:0/1059464118 <== mds.1 v2:172.21.15.28:6840/761183870 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x555c9dbfcbb0 con 0x555c9dd11440 2024-05-12T22:08:47.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.823+0000 7f9636a00640 10 client.25166 handle_command_reply: tid=0 2024-05-12T22:08:47.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.823+0000 7f965a5f2740 1 client.25166 shutdown 2024-05-12T22:08:47.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.823+0000 7f965a5f2740 1 -- 172.21.15.28:0/1059464118 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x555c9dc8c9a0 msgr2=0x555c9dc8fdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:47.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.823+0000 7f965a5f2740 1 --2- 172.21.15.28:0/1059464118 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x555c9dc8c9a0 0x555c9dc8fdd0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x555c9dca4ee0 tx=0x7f963c0039c0 comp rx=0 tx=0).stop 2024-05-12T22:08:47.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.823+0000 7f965a5f2740 1 -- 172.21.15.28:0/1059464118 shutdown_connections 2024-05-12T22:08:47.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.823+0000 7f965a5f2740 1 --2- 172.21.15.28:0/1059464118 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x555c9dd11440 0x555c9dd10150 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:47.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.823+0000 7f965a5f2740 1 --2- 172.21.15.28:0/1059464118 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c9dc91b10 0x555c9dd0aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:47.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.823+0000 7f965a5f2740 1 --2- 172.21.15.28:0/1059464118 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x555c9dc8c9a0 0x555c9dc8fdd0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:47.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.823+0000 7f965a5f2740 1 --2- 172.21.15.28:0/1059464118 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555c9dc8bfb0 0x555c9dc8f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:47.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.823+0000 7f9636a00640 0 client.25166 ms_handle_reset on v2:172.21.15.28:6840/761183870 2024-05-12T22:08:47.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.823+0000 7f965a5f2740 1 -- 172.21.15.28:0/1059464118 >> 172.21.15.28:0/1059464118 conn(0x555c9dc538a0 msgr2=0x555c9dc8a6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:08:47.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.824+0000 7f965a5f2740 1 -- 172.21.15.28:0/1059464118 shutdown_connections 2024-05-12T22:08:47.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.824+0000 7f965a5f2740 1 -- 172.21.15.28:0/1059464118 wait complete. 2024-05-12T22:08:47.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.824+0000 7f965a5f2740 20 client.25166 trim_cache size 0 max 16384 2024-05-12T22:08:47.830 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-05-12T22:08:47.831 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "6b92e7b4-10aa-11ef-bc98-c7b262605968", 2024-05-12T22:08:47.831 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 1, 2024-05-12T22:08:47.831 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 24550, 2024-05-12T22:08:47.831 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-05-12T22:08:47.831 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-05-12T22:08:47.831 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-05-12T22:08:47.831 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 130.52138176299999, 2024-05-12T22:08:47.831 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 23, 2024-05-12T22:08:47.831 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 88, 2024-05-12T22:08:47.831 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 86, 2024-05-12T22:08:47.831 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 143.010984235 2024-05-12T22:08:47.831 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-05-12T22:08:47.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.824+0000 7f9657800640 1 -- 172.21.15.28:0/823555382 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f96380a2c10 msgr2=0x7f96380a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:47.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.824+0000 7f9657800640 1 --2- 172.21.15.28:0/823555382 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f96380a2c10 0x7f96380a50d0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f9648009d40 tx=0x7f9648004870 comp rx=0 tx=0).stop 2024-05-12T22:08:47.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.824+0000 7f9657800640 1 -- 172.21.15.28:0/823555382 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f96501022f0 msgr2=0x7f9650197bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:47.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.824+0000 7f9657800640 1 --2- 172.21.15.28:0/823555382 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f96501022f0 0x7f9650197bb0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f9644005940 tx=0x7f9644033d90 comp rx=0 tx=0).stop 2024-05-12T22:08:47.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.825+0000 7f9657800640 1 -- 172.21.15.28:0/823555382 shutdown_connections 2024-05-12T22:08:47.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.825+0000 7f9657800640 1 --2- 172.21.15.28:0/823555382 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f96380a2c10 0x7f96380a50d0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:47.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.825+0000 7f9657800640 1 --2- 172.21.15.28:0/823555382 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9650107550 0x7f965019f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:47.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.825+0000 7f9657800640 1 --2- 172.21.15.28:0/823555382 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9650104c20 0x7f96501980f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:47.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.825+0000 7f9657800640 1 --2- 172.21.15.28:0/823555382 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f96501022f0 0x7f9650197bb0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:47.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.825+0000 7f9657800640 1 -- 172.21.15.28:0/823555382 >> 172.21.15.28:0/823555382 conn(0x7f96500fbce0 msgr2=0x7f9650102d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:08:47.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.825+0000 7f9657800640 1 -- 172.21.15.28:0/823555382 shutdown_connections 2024-05-12T22:08:47.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:47.825+0000 7f9657800640 1 -- 172.21.15.28:0/823555382 wait complete. 2024-05-12T22:08:47.843 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-05-12T22:08:48.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.033+0000 7f7f22400640 1 -- 172.21.15.28:0/3899830656 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7f1c101880 msgr2=0x7f7f1c101ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:48.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.033+0000 7f7f22400640 1 --2- 172.21.15.28:0/3899830656 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7f1c101880 0x7f7f1c101ca0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f7f0c009900 tx=0x7f7f0c02a3f0 comp rx=0 tx=0).stop 2024-05-12T22:08:48.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.033+0000 7f7f22400640 1 -- 172.21.15.28:0/3899830656 shutdown_connections 2024-05-12T22:08:48.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.033+0000 7f7f22400640 1 --2- 172.21.15.28:0/3899830656 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f1c103430 0x7f7f1c109cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:48.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.033+0000 7f7f22400640 1 --2- 172.21.15.28:0/3899830656 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7f1c102a70 0x7f7f1c102ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:48.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.033+0000 7f7f22400640 1 --2- 172.21.15.28:0/3899830656 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7f1c101880 0x7f7f1c101ca0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:48.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.033+0000 7f7f22400640 1 -- 172.21.15.28:0/3899830656 >> 172.21.15.28:0/3899830656 conn(0x7f7f1c0fce10 msgr2=0x7f7f1c0ff230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:08:48.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.033+0000 7f7f22400640 1 -- 172.21.15.28:0/3899830656 shutdown_connections 2024-05-12T22:08:48.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.033+0000 7f7f22400640 1 -- 172.21.15.28:0/3899830656 wait complete. 2024-05-12T22:08:48.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.034+0000 7f7f22400640 1 Processor -- start 2024-05-12T22:08:48.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.034+0000 7f7f22400640 1 -- start start 2024-05-12T22:08:48.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.035+0000 7f7f22400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f1c101880 0x7f7f1c0755e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:48.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.035+0000 7f7f22400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7f1c102a70 0x7f7f1c073c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:48.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.035+0000 7f7f22400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7f1c103430 0x7f7f1c074160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:48.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.035+0000 7f7f22400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7f1c10d520 con 0x7f7f1c101880 2024-05-12T22:08:48.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.035+0000 7f7f22400640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f7f1c10d3a0 con 0x7f7f1c103430 2024-05-12T22:08:48.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.035+0000 7f7f22400640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f7f1c10d6a0 con 0x7f7f1c102a70 2024-05-12T22:08:48.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.035+0000 7f7f1b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f1c101880 0x7f7f1c0755e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:48.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.035+0000 7f7f1b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f1c101880 0x7f7f1c0755e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42640/0 (socket says 172.21.15.28:42640) 2024-05-12T22:08:48.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.035+0000 7f7f1b400640 1 -- 172.21.15.28:0/45718314 learned_addr learned my addr 172.21.15.28:0/45718314 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:08:48.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.035+0000 7f7f13e00640 1 --2- 172.21.15.28:0/45718314 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7f1c102a70 0x7f7f1c073c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:48.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.035+0000 7f7f1be00640 1 --2- 172.21.15.28:0/45718314 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7f1c103430 0x7f7f1c074160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:48.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.035+0000 7f7f1b400640 1 -- 172.21.15.28:0/45718314 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7f1c102a70 msgr2=0x7f7f1c073c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:48.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.035+0000 7f7f1b400640 1 --2- 172.21.15.28:0/45718314 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7f1c102a70 0x7f7f1c073c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:48.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.035+0000 7f7f1b400640 1 -- 172.21.15.28:0/45718314 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7f1c103430 msgr2=0x7f7f1c074160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:48.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.035+0000 7f7f1b400640 1 --2- 172.21.15.28:0/45718314 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7f1c103430 0x7f7f1c074160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:48.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.035+0000 7f7f1b400640 1 -- 172.21.15.28:0/45718314 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f1c0749f0 con 0x7f7f1c101880 2024-05-12T22:08:48.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.036+0000 7f7f1b400640 1 --2- 172.21.15.28:0/45718314 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f1c101880 0x7f7f1c0755e0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f7f0c031440 tx=0x7f7f0c032ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:08:48.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.036+0000 7f7f18c00640 1 -- 172.21.15.28:0/45718314 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f0c03f030 con 0x7f7f1c101880 2024-05-12T22:08:48.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.036+0000 7f7f18c00640 1 -- 172.21.15.28:0/45718314 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f0c003ce0 con 0x7f7f1c101880 2024-05-12T22:08:48.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.036+0000 7f7f18c00640 1 -- 172.21.15.28:0/45718314 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f0c035070 con 0x7f7f1c101880 2024-05-12T22:08:48.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.036+0000 7f7f22400640 1 -- 172.21.15.28:0/45718314 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f1c1a1c80 con 0x7f7f1c101880 2024-05-12T22:08:48.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.037+0000 7f7f22400640 1 -- 172.21.15.28:0/45718314 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f1c1a21d0 con 0x7f7f1c101880 2024-05-12T22:08:48.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.040+0000 7f7f18c00640 1 -- 172.21.15.28:0/45718314 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f7f0c042030 con 0x7f7f1c101880 2024-05-12T22:08:48.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.040+0000 7f7f2519b740 1 Processor -- start 2024-05-12T22:08:48.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.040+0000 7f7f2519b740 1 -- start start 2024-05-12T22:08:48.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.041+0000 7f7f2519b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c224b7fb0 0x563c224b83d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:48.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.041+0000 7f7f2519b740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563c224b89a0 0x563c224bd4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:48.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.041+0000 7f7f2519b740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563c224bdb10 0x563c224bffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:48.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.041+0000 7f7f2519b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x563c2241b480 con 0x563c224b7fb0 2024-05-12T22:08:48.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.041+0000 7f7f2519b740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x563c223c35c0 con 0x563c224bdb10 2024-05-12T22:08:48.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.041+0000 7f7f2519b740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x563c22417440 con 0x563c224b89a0 2024-05-12T22:08:48.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.041+0000 7f7f1be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c224b7fb0 0x563c224b83d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:48.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.041+0000 7f7f1b400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563c224b89a0 0x563c224bd4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:48.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.041+0000 7f7f1be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c224b7fb0 0x563c224b83d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42646/0 (socket says 172.21.15.28:42646) 2024-05-12T22:08:48.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.041+0000 7f7f1be00640 1 -- 172.21.15.28:0/3594324177 learned_addr learned my addr 172.21.15.28:0/3594324177 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:08:48.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.041+0000 7f7f13e00640 1 --2- 172.21.15.28:0/3594324177 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563c224bdb10 0x563c224bffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:48.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.042+0000 7f7f1b400640 1 -- 172.21.15.28:0/3594324177 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563c224bdb10 msgr2=0x563c224bffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:08:48.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.042+0000 7f7f1b400640 1 --2- 172.21.15.28:0/3594324177 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563c224bdb10 0x563c224bffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:48.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.042+0000 7f7f1b400640 1 -- 172.21.15.28:0/3594324177 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c224b7fb0 msgr2=0x563c224b83d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:08:48.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.042+0000 7f7f1b400640 1 --2- 172.21.15.28:0/3594324177 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c224b7fb0 0x563c224b83d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:48.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.042+0000 7f7f1b400640 1 -- 172.21.15.28:0/3594324177 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563c2212f2d0 con 0x563c224b89a0 2024-05-12T22:08:48.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.042+0000 7f7f1be00640 1 --2- 172.21.15.28:0/3594324177 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c224b7fb0 0x563c224b83d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:08:48.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.042+0000 7f7f18c00640 1 --2- 172.21.15.28:0/45718314 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f7f000a2b90 0x7f7f000a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:48.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.042+0000 7f7f1b400640 1 --2- 172.21.15.28:0/3594324177 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563c224b89a0 0x563c224bd4a0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f7f0c0ecc40 tx=0x7f7f0c0f7540 comp rx=0 tx=0).ready entity=mon.1 client_cookie=240272372927d122 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:08:48.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.042+0000 7f7f1be00640 1 --2- 172.21.15.28:0/45718314 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f7f000a2b90 0x7f7f000a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:48.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.042+0000 7f7f18c00640 1 -- 172.21.15.28:0/45718314 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f7f0c0ef1f0 con 0x7f7f1c101880 2024-05-12T22:08:48.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.042+0000 7f7f10c00640 1 -- 172.21.15.28:0/3594324177 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f0c0b7d50 con 0x563c224b89a0 2024-05-12T22:08:48.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.042+0000 7f7f10c00640 1 -- 172.21.15.28:0/3594324177 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f0c0bbab0 con 0x563c224b89a0 2024-05-12T22:08:48.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.042+0000 7f7f10c00640 1 -- 172.21.15.28:0/3594324177 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f0c0bb410 con 0x563c224b89a0 2024-05-12T22:08:48.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.043+0000 7f7f2519b740 1 -- 172.21.15.28:0/3594324177 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563c224b89a0 msgr2=0x563c224bd4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:48.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.043+0000 7f7f1be00640 1 --2- 172.21.15.28:0/45718314 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f7f000a2b90 0x7f7f000a5050 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x563c224b9200 tx=0x7f7f14009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:08:48.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.043+0000 7f7f2519b740 1 --2- 172.21.15.28:0/3594324177 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563c224b89a0 0x563c224bd4a0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f7f0c0ecc40 tx=0x7f7f0c0f7540 comp rx=0 tx=0).stop 2024-05-12T22:08:48.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.043+0000 7f7f2519b740 1 -- 172.21.15.28:0/3594324177 shutdown_connections 2024-05-12T22:08:48.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.043+0000 7f7f2519b740 1 --2- 172.21.15.28:0/3594324177 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563c224bdb10 0x563c224bffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:48.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.043+0000 7f7f2519b740 1 --2- 172.21.15.28:0/3594324177 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563c224b89a0 0x563c224bd4a0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:48.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.043+0000 7f7f2519b740 1 --2- 172.21.15.28:0/3594324177 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c224b7fb0 0x563c224b83d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:48.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.043+0000 7f7f2519b740 1 -- 172.21.15.28:0/3594324177 >> 172.21.15.28:0/3594324177 conn(0x563c2247f8a0 msgr2=0x563c224b3a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:08:48.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.044+0000 7f7f2519b740 1 -- 172.21.15.28:0/3594324177 shutdown_connections 2024-05-12T22:08:48.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.044+0000 7f7f2519b740 1 -- 172.21.15.28:0/3594324177 wait complete. 2024-05-12T22:08:48.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.044+0000 7f7f2519b740 1 -- start start 2024-05-12T22:08:48.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.044+0000 7f7f2519b740 1 Processor -- start 2024-05-12T22:08:48.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.045+0000 7f7f2519b740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563c224b7fb0 0x563c224c1d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:48.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.045+0000 7f7f2519b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c224b89a0 0x563c224c2270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:48.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.045+0000 7f7f2519b740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563c224bdb10 0x563c224c92b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:48.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.045+0000 7f7f2519b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x563c224c0880 con 0x563c224b89a0 2024-05-12T22:08:48.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.045+0000 7f7f2519b740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x563c223d6980 con 0x563c224b7fb0 2024-05-12T22:08:48.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.045+0000 7f7f2519b740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x563c224c0a00 con 0x563c224bdb10 2024-05-12T22:08:48.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.045+0000 7f7f2519b740 10 client.0 _ll_register_callbacks cb 0x563c22411fc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:08:48.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.045+0000 7f7f13e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c224b89a0 0x563c224c2270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:48.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.045+0000 7f7f1be00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563c224bdb10 0x563c224c92b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:48.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.045+0000 7f7f1b400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563c224b7fb0 0x563c224c1d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:48.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.045+0000 7f7f13e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c224b89a0 0x563c224c2270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42648/0 (socket says 172.21.15.28:42648) 2024-05-12T22:08:48.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.045+0000 7f7f13e00640 1 -- 172.21.15.28:0/3845370255 learned_addr learned my addr 172.21.15.28:0/3845370255 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:08:48.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.045+0000 7f7f1b400640 1 -- 172.21.15.28:0/3845370255 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563c224bdb10 msgr2=0x563c224c92b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:08:48.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.045+0000 7f7f1b400640 1 --2- 172.21.15.28:0/3845370255 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563c224bdb10 0x563c224c92b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:48.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.045+0000 7f7f1b400640 1 -- 172.21.15.28:0/3845370255 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c224b89a0 msgr2=0x563c224c2270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:08:48.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.045+0000 7f7f1b400640 1 --2- 172.21.15.28:0/3845370255 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c224b89a0 0x563c224c2270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:48.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.045+0000 7f7f1b400640 1 -- 172.21.15.28:0/3845370255 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563c221f96b0 con 0x563c224b7fb0 2024-05-12T22:08:48.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.046+0000 7f7f1b400640 1 --2- 172.21.15.28:0/3845370255 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563c224b7fb0 0x563c224c1d30 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f7f0c005ec0 tx=0x7f7f0c0f2760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=24fb5f7cd286f8b8 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:08:48.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.046+0000 7f7f09600640 10 client.0 ms_handle_connect on v2:172.21.15.86:3300/0 2024-05-12T22:08:48.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.046+0000 7f7f09600640 1 -- 172.21.15.28:0/3845370255 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f0c049030 con 0x563c224b7fb0 2024-05-12T22:08:48.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.046+0000 7f7f09600640 1 -- 172.21.15.28:0/3845370255 <== mon.2 v2:172.21.15.86:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f7f0c003800 con 0x563c224b7fb0 2024-05-12T22:08:48.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.046+0000 7f7f2519b740 1 -- 172.21.15.28:0/3845370255 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7f0c0bbab0 con 0x563c224b7fb0 2024-05-12T22:08:48.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.046+0000 7f7f09600640 1 -- 172.21.15.28:0/3845370255 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f7f0c0bb5f0 con 0x563c224b7fb0 2024-05-12T22:08:48.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.046+0000 7f7f2519b740 10 client.25178 fetch_fsmap learned FSMap version 23 2024-05-12T22:08:48.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.046+0000 7f7f2519b740 1 -- 172.21.15.28:0/3845370255 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x563c224c0880 con 0x563c224b7fb0 2024-05-12T22:08:48.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.046+0000 7f7f09600640 1 -- 172.21.15.28:0/3845370255 <== mon.2 v2:172.21.15.86:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f0c0bbab0 con 0x563c224b7fb0 2024-05-12T22:08:48.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.047+0000 7f7f09600640 1 -- 172.21.15.28:0/3845370255 <== mon.2 v2:172.21.15.86:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f7f0c04c060 con 0x563c224b7fb0 2024-05-12T22:08:48.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.047+0000 7f7f2519b740 10 client.25178 fetch_fsmap finished waiting for FSMap version 23 2024-05-12T22:08:48.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.047+0000 7f7f2519b740 10 client.25178 resolve_mds: resolved name 'b' to daemon mds.b 2024-05-12T22:08:48.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.047+0000 7f7f2519b740 20 client.25178 populate_metadata read hostname 'smithi028' 2024-05-12T22:08:48.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.047+0000 7f7f2519b740 1 --2- 172.21.15.28:0/3845370255 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x563c224ce620 0x563c224cfa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:48.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.047+0000 7f7f2519b740 4 client.25178 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-12T22:08:48.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.047+0000 7f7f2519b740 1 -- 172.21.15.28:0/3845370255 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563c2242f060 con 0x563c224ce620 2024-05-12T22:08:48.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.047+0000 7f7f13e00640 1 --2- 172.21.15.28:0/3845370255 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x563c224ce620 0x563c224cfa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:48.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.048+0000 7f7f13e00640 1 --2- 172.21.15.28:0/3845370255 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x563c224ce620 0x563c224cfa90 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=de43d5ac75f67dd7 server_cookie=395026ab94f5ae29 in_seq=0 out_seq=0 2024-05-12T22:08:48.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.048+0000 7f7f09600640 10 client.25178 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:08:48.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.048+0000 7f7f09600640 1 -- 172.21.15.28:0/3845370255 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563c2242f060 con 0x563c224ce620 2024-05-12T22:08:48.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.048+0000 7f7f09600640 10 client.25178 handle_command_reply: tid=0 2024-05-12T22:08:48.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.048+0000 7f7f2519b740 1 client.25178 shutdown 2024-05-12T22:08:48.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.049+0000 7f7f2519b740 1 -- 172.21.15.28:0/3845370255 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563c224b7fb0 msgr2=0x563c224c1d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:48.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.049+0000 7f7f2519b740 1 --2- 172.21.15.28:0/3845370255 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563c224b7fb0 0x563c224c1d30 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f7f0c005ec0 tx=0x7f7f0c0f2760 comp rx=0 tx=0).stop 2024-05-12T22:08:48.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.049+0000 7f7f2519b740 1 -- 172.21.15.28:0/3845370255 shutdown_connections 2024-05-12T22:08:48.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.049+0000 7f7f2519b740 1 --2- 172.21.15.28:0/3845370255 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x563c224ce620 0x563c224cfa90 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:48.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.049+0000 7f7f2519b740 1 --2- 172.21.15.28:0/3845370255 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563c224bdb10 0x563c224c92b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:48.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.049+0000 7f7f2519b740 1 --2- 172.21.15.28:0/3845370255 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c224b89a0 0x563c224c2270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:48.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.049+0000 7f7f2519b740 1 --2- 172.21.15.28:0/3845370255 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563c224b7fb0 0x563c224c1d30 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:48.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.049+0000 7f7f2519b740 1 -- 172.21.15.28:0/3845370255 >> 172.21.15.28:0/3845370255 conn(0x563c2247f8a0 msgr2=0x563c224b2b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:08:48.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.049+0000 7f7f09600640 0 client.25178 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:08:48.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.049+0000 7f7f2519b740 1 -- 172.21.15.28:0/3845370255 shutdown_connections 2024-05-12T22:08:48.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.049+0000 7f7f2519b740 1 -- 172.21.15.28:0/3845370255 wait complete. 2024-05-12T22:08:48.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.049+0000 7f7f2519b740 20 client.25178 trim_cache size 0 max 16384 2024-05-12T22:08:48.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.071+0000 7f7f2519b740 1 Processor -- start 2024-05-12T22:08:48.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.071+0000 7f7f2519b740 1 -- start start 2024-05-12T22:08:48.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.071+0000 7f7f2519b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c224b7fb0 0x563c224b68f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:48.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.071+0000 7f7f2519b740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563c224b89a0 0x563c224c9270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:48.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.071+0000 7f7f2519b740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563c224bdb10 0x563c224b6e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:48.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.071+0000 7f7f2519b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x563c224cddd0 con 0x563c224b7fb0 2024-05-12T22:08:48.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.071+0000 7f7f2519b740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x563c224ce4a0 con 0x563c224bdb10 2024-05-12T22:08:48.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.071+0000 7f7f2519b740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x563c224ceb70 con 0x563c224b89a0 2024-05-12T22:08:48.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.072+0000 7f7f1b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c224b7fb0 0x563c224b68f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:48.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.072+0000 7f7f1b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c224b7fb0 0x563c224b68f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42664/0 (socket says 172.21.15.28:42664) 2024-05-12T22:08:48.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.072+0000 7f7f1b400640 1 -- 172.21.15.28:0/3100170762 learned_addr learned my addr 172.21.15.28:0/3100170762 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:08:48.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.072+0000 7f7f13e00640 1 --2- 172.21.15.28:0/3100170762 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563c224b89a0 0x563c224c9270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:48.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.072+0000 7f7f1be00640 1 --2- 172.21.15.28:0/3100170762 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563c224bdb10 0x563c224b6e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:48.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.072+0000 7f7f1b400640 1 -- 172.21.15.28:0/3100170762 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563c224b89a0 msgr2=0x563c224c9270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:08:48.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.072+0000 7f7f1b400640 1 --2- 172.21.15.28:0/3100170762 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563c224b89a0 0x563c224c9270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:48.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.072+0000 7f7f1b400640 1 -- 172.21.15.28:0/3100170762 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563c224bdb10 msgr2=0x563c224b6e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:08:48.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.072+0000 7f7f1b400640 1 --2- 172.21.15.28:0/3100170762 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563c224bdb10 0x563c224b6e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:48.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.072+0000 7f7f1b400640 1 -- 172.21.15.28:0/3100170762 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563c224b46e0 con 0x563c224b7fb0 2024-05-12T22:08:48.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.072+0000 7f7f1b400640 1 --2- 172.21.15.28:0/3100170762 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c224b7fb0 0x563c224b68f0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f7f0c005d00 tx=0x7f7f0c0f7540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0d3f7df9520ce45 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:08:48.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.072+0000 7f7f0be00640 1 -- 172.21.15.28:0/3100170762 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f0c049070 con 0x563c224b7fb0 2024-05-12T22:08:48.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.073+0000 7f7f0be00640 1 -- 172.21.15.28:0/3100170762 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f0c0f2ac0 con 0x563c224b7fb0 2024-05-12T22:08:48.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.073+0000 7f7f0be00640 1 -- 172.21.15.28:0/3100170762 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f0c0bb880 con 0x563c224b7fb0 2024-05-12T22:08:48.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.073+0000 7f7f2519b740 1 -- 172.21.15.28:0/3100170762 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c224b7fb0 msgr2=0x563c224b68f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:48.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.073+0000 7f7f2519b740 1 --2- 172.21.15.28:0/3100170762 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c224b7fb0 0x563c224b68f0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f7f0c005d00 tx=0x7f7f0c0f7540 comp rx=0 tx=0).stop 2024-05-12T22:08:48.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.073+0000 7f7f2519b740 1 -- 172.21.15.28:0/3100170762 shutdown_connections 2024-05-12T22:08:48.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.074+0000 7f7f2519b740 1 --2- 172.21.15.28:0/3100170762 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563c224bdb10 0x563c224b6e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:48.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.074+0000 7f7f2519b740 1 --2- 172.21.15.28:0/3100170762 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563c224b89a0 0x563c224c9270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:48.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.074+0000 7f7f2519b740 1 --2- 172.21.15.28:0/3100170762 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c224b7fb0 0x563c224b68f0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:48.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.074+0000 7f7f2519b740 1 -- 172.21.15.28:0/3100170762 >> 172.21.15.28:0/3100170762 conn(0x563c2247f8a0 msgr2=0x563c224bd000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:08:48.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.074+0000 7f7f2519b740 1 -- 172.21.15.28:0/3100170762 shutdown_connections 2024-05-12T22:08:48.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.074+0000 7f7f2519b740 1 -- 172.21.15.28:0/3100170762 wait complete. 2024-05-12T22:08:48.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.074+0000 7f7f2519b740 1 -- start start 2024-05-12T22:08:48.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.074+0000 7f7f2519b740 1 Processor -- start 2024-05-12T22:08:48.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.075+0000 7f7f2519b740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563c224b7fb0 0x563c224c4e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:48.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.075+0000 7f7f2519b740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563c224b89a0 0x563c224c5350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:48.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.075+0000 7f7f2519b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c224bdb10 0x563c224c8370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:48.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.075+0000 7f7f2519b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x563c224cf910 con 0x563c224bdb10 2024-05-12T22:08:48.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.075+0000 7f7f2519b740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x563c224cf240 con 0x563c224b7fb0 2024-05-12T22:08:48.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.075+0000 7f7f2519b740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x563c224cffe0 con 0x563c224b89a0 2024-05-12T22:08:48.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.075+0000 7f7f2519b740 10 client.0 _ll_register_callbacks cb 0x7f7f14009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:08:48.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.075+0000 7f7f1be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c224bdb10 0x563c224c8370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:48.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.075+0000 7f7f1be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c224bdb10 0x563c224c8370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42668/0 (socket says 172.21.15.28:42668) 2024-05-12T22:08:48.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.075+0000 7f7f1be00640 1 -- 172.21.15.28:0/887700618 learned_addr learned my addr 172.21.15.28:0/887700618 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:08:48.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.075+0000 7f7f1b400640 1 --2- 172.21.15.28:0/887700618 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563c224b7fb0 0x563c224c4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:48.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.075+0000 7f7f13e00640 1 --2- 172.21.15.28:0/887700618 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563c224b89a0 0x563c224c5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:48.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.075+0000 7f7f1b400640 1 -- 172.21.15.28:0/887700618 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563c224b89a0 msgr2=0x563c224c5350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:08:48.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.075+0000 7f7f1b400640 1 --2- 172.21.15.28:0/887700618 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563c224b89a0 0x563c224c5350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:48.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.075+0000 7f7f1b400640 1 -- 172.21.15.28:0/887700618 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c224bdb10 msgr2=0x563c224c8370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:08:48.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.075+0000 7f7f1b400640 1 --2- 172.21.15.28:0/887700618 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c224bdb10 0x563c224c8370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:48.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.075+0000 7f7f1b400640 1 -- 172.21.15.28:0/887700618 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563c224c8960 con 0x563c224b7fb0 2024-05-12T22:08:48.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.075+0000 7f7f13e00640 1 --2- 172.21.15.28:0/887700618 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563c224b89a0 0x563c224c5350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:08:48.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.075+0000 7f7f1be00640 1 --2- 172.21.15.28:0/887700618 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c224bdb10 0x563c224c8370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:08:48.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.076+0000 7f7f1b400640 1 --2- 172.21.15.28:0/887700618 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563c224b7fb0 0x563c224c4e10 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f7f0c0b9d60 tx=0x7f7f0c0c47f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=76cbf0afb31f64fa server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:08:48.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.076+0000 7f7f09600640 10 client.0 ms_handle_connect on v2:172.21.15.86:3300/0 2024-05-12T22:08:48.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.076+0000 7f7f09600640 1 -- 172.21.15.28:0/887700618 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f0c049030 con 0x563c224b7fb0 2024-05-12T22:08:48.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.076+0000 7f7f2519b740 1 -- 172.21.15.28:0/887700618 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7f0c0f2ac0 con 0x563c224b7fb0 2024-05-12T22:08:48.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.076+0000 7f7f09600640 1 -- 172.21.15.28:0/887700618 <== mon.2 v2:172.21.15.86:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f7f0c03b460 con 0x563c224b7fb0 2024-05-12T22:08:48.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.076+0000 7f7f09600640 1 -- 172.21.15.28:0/887700618 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f7f0c0bb880 con 0x563c224b7fb0 2024-05-12T22:08:48.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.076+0000 7f7f2519b740 10 client.25184 fetch_fsmap learned FSMap version 23 2024-05-12T22:08:48.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.076+0000 7f7f2519b740 1 -- 172.21.15.28:0/887700618 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x563c224c8960 con 0x563c224b7fb0 2024-05-12T22:08:48.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.076+0000 7f7f09600640 1 -- 172.21.15.28:0/887700618 <== mon.2 v2:172.21.15.86:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f0c0f2ac0 con 0x563c224b7fb0 2024-05-12T22:08:48.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.077+0000 7f7f09600640 1 -- 172.21.15.28:0/887700618 <== mon.2 v2:172.21.15.86:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f7f0c0f7c20 con 0x563c224b7fb0 2024-05-12T22:08:48.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.077+0000 7f7f2519b740 10 client.25184 fetch_fsmap finished waiting for FSMap version 23 2024-05-12T22:08:48.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.077+0000 7f7f2519b740 10 client.25184 resolve_mds: resolved name 'b' to daemon mds.b 2024-05-12T22:08:48.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.077+0000 7f7f2519b740 20 client.25184 populate_metadata read hostname 'smithi028' 2024-05-12T22:08:48.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.077+0000 7f7f2519b740 1 --2- 172.21.15.28:0/887700618 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x563c22539120 0x563c22537e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:48.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.077+0000 7f7f2519b740 4 client.25184 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-12T22:08:48.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.077+0000 7f7f2519b740 1 -- 172.21.15.28:0/887700618 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "status"}) -- 0x563c22428bb0 con 0x563c22539120 2024-05-12T22:08:48.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.077+0000 7f7f13e00640 1 --2- 172.21.15.28:0/887700618 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x563c22539120 0x563c22537e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:48.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.077+0000 7f7f13e00640 1 --2- 172.21.15.28:0/887700618 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x563c22539120 0x563c22537e30 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ab2679cea9769df0 server_cookie=2f8a893522427c62 in_seq=0 out_seq=0 2024-05-12T22:08:48.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.078+0000 7f7f09600640 10 client.25184 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:08:48.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.078+0000 7f7f09600640 1 -- 172.21.15.28:0/887700618 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x563c22428bb0 con 0x563c22539120 2024-05-12T22:08:48.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.078+0000 7f7f09600640 10 client.25184 handle_command_reply: tid=0 2024-05-12T22:08:48.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.078+0000 7f7f2519b740 1 client.25184 shutdown 2024-05-12T22:08:48.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.078+0000 7f7f2519b740 1 -- 172.21.15.28:0/887700618 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563c224b7fb0 msgr2=0x563c224c4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:48.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.078+0000 7f7f2519b740 1 --2- 172.21.15.28:0/887700618 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563c224b7fb0 0x563c224c4e10 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f7f0c0b9d60 tx=0x7f7f0c0c47f0 comp rx=0 tx=0).stop 2024-05-12T22:08:48.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.078+0000 7f7f2519b740 1 -- 172.21.15.28:0/887700618 shutdown_connections 2024-05-12T22:08:48.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.078+0000 7f7f2519b740 1 --2- 172.21.15.28:0/887700618 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x563c22539120 0x563c22537e30 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:48.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.078+0000 7f7f2519b740 1 --2- 172.21.15.28:0/887700618 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c224bdb10 0x563c224c8370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:48.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.078+0000 7f7f09600640 0 client.25184 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:08:48.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.078+0000 7f7f2519b740 1 --2- 172.21.15.28:0/887700618 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563c224b89a0 0x563c224c5350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:48.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.078+0000 7f7f2519b740 1 --2- 172.21.15.28:0/887700618 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563c224b7fb0 0x563c224c4e10 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:48.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.078+0000 7f7f2519b740 1 -- 172.21.15.28:0/887700618 >> 172.21.15.28:0/887700618 conn(0x563c2247f8a0 msgr2=0x563c224b9410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:08:48.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.078+0000 7f7f2519b740 1 -- 172.21.15.28:0/887700618 shutdown_connections 2024-05-12T22:08:48.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.078+0000 7f7f2519b740 1 -- 172.21.15.28:0/887700618 wait complete. 2024-05-12T22:08:48.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.078+0000 7f7f2519b740 20 client.25184 trim_cache size 0 max 16384 2024-05-12T22:08:48.083 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-05-12T22:08:48.083 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "6b92e7b4-10aa-11ef-bc98-c7b262605968", 2024-05-12T22:08:48.083 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 0, 2024-05-12T22:08:48.083 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 24581, 2024-05-12T22:08:48.083 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-05-12T22:08:48.083 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-05-12T22:08:48.083 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-05-12T22:08:48.083 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 133.81695505600001, 2024-05-12T22:08:48.084 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 23, 2024-05-12T22:08:48.084 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 88, 2024-05-12T22:08:48.084 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 86, 2024-05-12T22:08:48.084 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 140.59328310800001 2024-05-12T22:08:48.084 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-05-12T22:08:48.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.079+0000 7f7f22400640 1 -- 172.21.15.28:0/45718314 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f7f000a2b90 msgr2=0x7f7f000a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:48.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.079+0000 7f7f22400640 1 --2- 172.21.15.28:0/45718314 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f7f000a2b90 0x7f7f000a5050 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x563c224b9200 tx=0x7f7f14009500 comp rx=0 tx=0).stop 2024-05-12T22:08:48.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.079+0000 7f7f22400640 1 -- 172.21.15.28:0/45718314 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f1c101880 msgr2=0x7f7f1c0755e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:48.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.079+0000 7f7f22400640 1 --2- 172.21.15.28:0/45718314 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f1c101880 0x7f7f1c0755e0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f7f0c031440 tx=0x7f7f0c032ce0 comp rx=0 tx=0).stop 2024-05-12T22:08:48.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.079+0000 7f7f22400640 1 -- 172.21.15.28:0/45718314 shutdown_connections 2024-05-12T22:08:48.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.079+0000 7f7f22400640 1 --2- 172.21.15.28:0/45718314 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f7f000a2b90 0x7f7f000a5050 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:48.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.079+0000 7f7f22400640 1 --2- 172.21.15.28:0/45718314 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7f1c103430 0x7f7f1c074160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:48.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.079+0000 7f7f22400640 1 --2- 172.21.15.28:0/45718314 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7f1c102a70 0x7f7f1c073c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:48.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.079+0000 7f7f22400640 1 --2- 172.21.15.28:0/45718314 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f1c101880 0x7f7f1c0755e0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:48.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.079+0000 7f7f22400640 1 -- 172.21.15.28:0/45718314 >> 172.21.15.28:0/45718314 conn(0x7f7f1c0fce10 msgr2=0x7f7f1c105df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:08:48.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.079+0000 7f7f22400640 1 -- 172.21.15.28:0/45718314 shutdown_connections 2024-05-12T22:08:48.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.079+0000 7f7f22400640 1 -- 172.21.15.28:0/45718314 wait complete. 2024-05-12T22:08:48.095 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:48 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/511460812' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-12T22:08:48.097 INFO:teuthology.run_tasks:Running task sequential... 2024-05-12T22:08:48.107 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-05-12T22:08:48.107 INFO:teuthology.task.sequential:In sequential, running task print... 2024-05-12T22:08:48.108 INFO:teuthology.task.print:Enabling mgr modules 2024-05-12T22:08:48.108 INFO:teuthology.task.sequential:In sequential, running task exec... 2024-05-12T22:08:48.108 INFO:teuthology.task.exec:Executing custom commands... 2024-05-12T22:08:48.108 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi028.front.sepia.ceph.com 2024-05-12T22:08:48.109 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph mgr module enable snap_schedule' 2024-05-12T22:08:48.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.298+0000 7f8f29600640 1 -- 172.21.15.28:0/209418552 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f241028b0 msgr2=0x7f8f24102d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:48.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.298+0000 7f8f29600640 1 --2- 172.21.15.28:0/209418552 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f241028b0 0x7f8f24102d30 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f8f0c009900 tx=0x7f8f0c02a410 comp rx=0 tx=0).stop 2024-05-12T22:08:48.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.298+0000 7f8f29600640 1 -- 172.21.15.28:0/209418552 shutdown_connections 2024-05-12T22:08:48.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.298+0000 7f8f29600640 1 --2- 172.21.15.28:0/209418552 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8f24103270 0x7f8f24109b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:48.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.298+0000 7f8f29600640 1 --2- 172.21.15.28:0/209418552 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f241028b0 0x7f8f24102d30 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:48.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.298+0000 7f8f29600640 1 --2- 172.21.15.28:0/209418552 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8f241016c0 0x7f8f24101ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:48.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.298+0000 7f8f29600640 1 -- 172.21.15.28:0/209418552 >> 172.21.15.28:0/209418552 conn(0x7f8f240fcc30 msgr2=0x7f8f240ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:08:48.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.299+0000 7f8f29600640 1 -- 172.21.15.28:0/209418552 shutdown_connections 2024-05-12T22:08:48.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.299+0000 7f8f29600640 1 -- 172.21.15.28:0/209418552 wait complete. 2024-05-12T22:08:48.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.300+0000 7f8f29600640 1 Processor -- start 2024-05-12T22:08:48.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.300+0000 7f8f29600640 1 -- start start 2024-05-12T22:08:48.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.300+0000 7f8f29600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8f241016c0 0x7f8f24199bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:48.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.301+0000 7f8f29600640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8f241028b0 0x7f8f2419a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:48.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.301+0000 7f8f29600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f24103270 0x7f8f241a1160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:48.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.301+0000 7f8f29600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8f2410d370 con 0x7f8f24103270 2024-05-12T22:08:48.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.301+0000 7f8f29600640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f8f2410d1f0 con 0x7f8f241016c0 2024-05-12T22:08:48.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.301+0000 7f8f29600640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f8f2410d4f0 con 0x7f8f241028b0 2024-05-12T22:08:48.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.301+0000 7f8f28c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f24103270 0x7f8f241a1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:48.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.301+0000 7f8f28c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f24103270 0x7f8f241a1160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42696/0 (socket says 172.21.15.28:42696) 2024-05-12T22:08:48.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.301+0000 7f8f28c00640 1 -- 172.21.15.28:0/1936008260 learned_addr learned my addr 172.21.15.28:0/1936008260 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:08:48.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.301+0000 7f8f23e00640 1 --2- 172.21.15.28:0/1936008260 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8f241016c0 0x7f8f24199bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:48.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.301+0000 7f8f1b400640 1 --2- 172.21.15.28:0/1936008260 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8f241028b0 0x7f8f2419a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:48.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.301+0000 7f8f28c00640 1 -- 172.21.15.28:0/1936008260 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8f241028b0 msgr2=0x7f8f2419a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:48.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.301+0000 7f8f28c00640 1 --2- 172.21.15.28:0/1936008260 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8f241028b0 0x7f8f2419a0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:48.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.301+0000 7f8f28c00640 1 -- 172.21.15.28:0/1936008260 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8f241016c0 msgr2=0x7f8f24199bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:48.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.301+0000 7f8f28c00640 1 --2- 172.21.15.28:0/1936008260 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8f241016c0 0x7f8f24199bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:48.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.301+0000 7f8f28c00640 1 -- 172.21.15.28:0/1936008260 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f241a17d0 con 0x7f8f24103270 2024-05-12T22:08:48.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.302+0000 7f8f28c00640 1 --2- 172.21.15.28:0/1936008260 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f24103270 0x7f8f241a1160 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f8f1c00d950 tx=0x7f8f1c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:08:48.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.302+0000 7f8f21600640 1 -- 172.21.15.28:0/1936008260 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f1c017030 con 0x7f8f24103270 2024-05-12T22:08:48.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.302+0000 7f8f21600640 1 -- 172.21.15.28:0/1936008260 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8f1c004020 con 0x7f8f24103270 2024-05-12T22:08:48.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.302+0000 7f8f21600640 1 -- 172.21.15.28:0/1936008260 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f1c003c00 con 0x7f8f24103270 2024-05-12T22:08:48.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.302+0000 7f8f29600640 1 -- 172.21.15.28:0/1936008260 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f241a1ac0 con 0x7f8f24103270 2024-05-12T22:08:48.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.302+0000 7f8f29600640 1 -- 172.21.15.28:0/1936008260 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f241a1f90 con 0x7f8f24103270 2024-05-12T22:08:48.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.304+0000 7f8f29600640 1 -- 172.21.15.28:0/1936008260 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f241a2280 con 0x7f8f24103270 2024-05-12T22:08:48.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.305+0000 7f8f21600640 1 -- 172.21.15.28:0/1936008260 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f8f1c01a070 con 0x7f8f24103270 2024-05-12T22:08:48.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.308+0000 7f8f21600640 1 --2- 172.21.15.28:0/1936008260 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f8f080a2c00 0x7f8f080a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:48.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.308+0000 7f8f21600640 1 -- 172.21.15.28:0/1936008260 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f8f1c0ce830 con 0x7f8f24103270 2024-05-12T22:08:48.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.308+0000 7f8f23e00640 1 --2- 172.21.15.28:0/1936008260 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f8f080a2c00 0x7f8f080a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:48.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.309+0000 7f8f23e00640 1 --2- 172.21.15.28:0/1936008260 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f8f080a2c00 0x7f8f080a50c0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f8f140099a0 tx=0x7f8f14009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:08:48.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.311+0000 7f8f21600640 1 -- 172.21.15.28:0/1936008260 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f8f1c021070 con 0x7f8f24103270 2024-05-12T22:08:48.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:48 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/511460812' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-12T22:08:48.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:48 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/511460812' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-12T22:08:48.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:48.567+0000 7f8f29600640 1 -- 172.21.15.28:0/1936008260 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) -- 0x7f8f241a2450 con 0x7f8f24103270 2024-05-12T22:08:49.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.082+0000 7f8f21600640 1 -- 172.21.15.28:0/1936008260 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v19) ==== 92+0+0 (secure 0 0 0) 0x7f8f1c09b740 con 0x7f8f24103270 2024-05-12T22:08:49.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.090+0000 7f8f29600640 1 -- 172.21.15.28:0/1936008260 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f8f080a2c00 msgr2=0x7f8f080a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:49.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.090+0000 7f8f29600640 1 --2- 172.21.15.28:0/1936008260 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f8f080a2c00 0x7f8f080a50c0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f8f140099a0 tx=0x7f8f14009500 comp rx=0 tx=0).stop 2024-05-12T22:08:49.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.090+0000 7f8f29600640 1 -- 172.21.15.28:0/1936008260 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f24103270 msgr2=0x7f8f241a1160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:49.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.090+0000 7f8f29600640 1 --2- 172.21.15.28:0/1936008260 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f24103270 0x7f8f241a1160 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f8f1c00d950 tx=0x7f8f1c00de10 comp rx=0 tx=0).stop 2024-05-12T22:08:49.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.091+0000 7f8f29600640 1 -- 172.21.15.28:0/1936008260 shutdown_connections 2024-05-12T22:08:49.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.091+0000 7f8f29600640 1 --2- 172.21.15.28:0/1936008260 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f8f080a2c00 0x7f8f080a50c0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:49.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.091+0000 7f8f29600640 1 --2- 172.21.15.28:0/1936008260 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f24103270 0x7f8f241a1160 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:49.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.091+0000 7f8f29600640 1 --2- 172.21.15.28:0/1936008260 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8f241028b0 0x7f8f2419a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:49.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.091+0000 7f8f29600640 1 --2- 172.21.15.28:0/1936008260 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8f241016c0 0x7f8f24199bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:49.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.091+0000 7f8f29600640 1 -- 172.21.15.28:0/1936008260 >> 172.21.15.28:0/1936008260 conn(0x7f8f240fcc30 msgr2=0x7f8f24073a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:08:49.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.091+0000 7f8f29600640 1 -- 172.21.15.28:0/1936008260 shutdown_connections 2024-05-12T22:08:49.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.091+0000 7f8f29600640 1 -- 172.21.15.28:0/1936008260 wait complete. 2024-05-12T22:08:49.114 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/allow_m_granularity true' 2024-05-12T22:08:49.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:49 smithi028 ceph-mon[29847]: pgmap v379: 97 pgs: 97 active+clean; 641 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-12T22:08:49.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:49 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/1936008260' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-05-12T22:08:49.285 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 22:08:49 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: ignoring --setuser ceph since I am not root 2024-05-12T22:08:49.285 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 22:08:49 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: ignoring --setgroup ceph since I am not root 2024-05-12T22:08:49.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.296+0000 7faed5a00640 1 -- 172.21.15.28:0/358528477 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faed0107e70 msgr2=0x7faed0108290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:49.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.296+0000 7faed5a00640 1 --2- 172.21.15.28:0/358528477 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faed0107e70 0x7faed0108290 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7faeb8009860 tx=0x7faeb802a340 comp rx=0 tx=0).stop 2024-05-12T22:08:49.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.296+0000 7faed5a00640 1 -- 172.21.15.28:0/358528477 shutdown_connections 2024-05-12T22:08:49.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.296+0000 7faed5a00640 1 --2- 172.21.15.28:0/358528477 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faed0109a10 0x7faed0110330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:49.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.296+0000 7faed5a00640 1 --2- 172.21.15.28:0/358528477 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faed0109050 0x7faed01094d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:49.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.296+0000 7faed5a00640 1 --2- 172.21.15.28:0/358528477 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faed0107e70 0x7faed0108290 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:49.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.296+0000 7faed5a00640 1 -- 172.21.15.28:0/358528477 >> 172.21.15.28:0/358528477 conn(0x7faed0076e80 msgr2=0x7faed00792a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:08:49.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.296+0000 7faed5a00640 1 -- 172.21.15.28:0/358528477 shutdown_connections 2024-05-12T22:08:49.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.297+0000 7faed5a00640 1 -- 172.21.15.28:0/358528477 wait complete. 2024-05-12T22:08:49.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.297+0000 7faed5a00640 1 Processor -- start 2024-05-12T22:08:49.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.298+0000 7faed5a00640 1 -- start start 2024-05-12T22:08:49.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.298+0000 7faed5a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faed0107e70 0x7faed019e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:49.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.298+0000 7faed5a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faed0109050 0x7faed019e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:49.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.298+0000 7faed5a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faed0109a10 0x7faed01a56a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:49.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.298+0000 7faed5a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7faed0113b30 con 0x7faed0107e70 2024-05-12T22:08:49.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.298+0000 7faed5a00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7faed01139b0 con 0x7faed0109050 2024-05-12T22:08:49.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.298+0000 7faed5a00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7faed0113cb0 con 0x7faed0109a10 2024-05-12T22:08:49.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.298+0000 7faec7e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faed0109050 0x7faed019e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:49.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.298+0000 7faecf400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faed0109a10 0x7faed01a56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:49.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.298+0000 7faecea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faed0107e70 0x7faed019e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:49.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.298+0000 7faec7e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faed0109050 0x7faed019e630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:38012/0 (socket says 172.21.15.28:38012) 2024-05-12T22:08:49.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.298+0000 7faec7e00640 1 -- 172.21.15.28:0/1816873454 learned_addr learned my addr 172.21.15.28:0/1816873454 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:08:49.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.299+0000 7faecf400640 1 -- 172.21.15.28:0/1816873454 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faed0109050 msgr2=0x7faed019e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:49.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.299+0000 7faecf400640 1 --2- 172.21.15.28:0/1816873454 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faed0109050 0x7faed019e630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:49.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.299+0000 7faecf400640 1 -- 172.21.15.28:0/1816873454 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faed0107e70 msgr2=0x7faed019e0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:49.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.299+0000 7faecf400640 1 --2- 172.21.15.28:0/1816873454 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faed0107e70 0x7faed019e0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:49.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.299+0000 7faecf400640 1 -- 172.21.15.28:0/1816873454 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faed01a5d10 con 0x7faed0109a10 2024-05-12T22:08:49.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.299+0000 7faecea00640 1 --2- 172.21.15.28:0/1816873454 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faed0107e70 0x7faed019e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:08:49.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.299+0000 7faec7e00640 1 --2- 172.21.15.28:0/1816873454 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faed0109050 0x7faed019e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:08:49.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.299+0000 7faecf400640 1 --2- 172.21.15.28:0/1816873454 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faed0109a10 0x7faed01a56a0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7faec800efc0 tx=0x7faec800c800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:08:49.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.299+0000 7faec7400640 1 -- 172.21.15.28:0/1816873454 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faec8003970 con 0x7faed0109a10 2024-05-12T22:08:49.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.299+0000 7faec7400640 1 -- 172.21.15.28:0/1816873454 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faec8004020 con 0x7faed0109a10 2024-05-12T22:08:49.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.299+0000 7faed5a00640 1 -- 172.21.15.28:0/1816873454 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faed01a5fa0 con 0x7faed0109a10 2024-05-12T22:08:49.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.299+0000 7faec7400640 1 -- 172.21.15.28:0/1816873454 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faec800b9f0 con 0x7faed0109a10 2024-05-12T22:08:49.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.299+0000 7faed5a00640 1 -- 172.21.15.28:0/1816873454 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faed01a6470 con 0x7faed0109a10 2024-05-12T22:08:49.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.301+0000 7faed5a00640 1 -- 172.21.15.28:0/1816873454 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faed010f1b0 con 0x7faed0109a10 2024-05-12T22:08:49.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.302+0000 7faec7400640 1 -- 172.21.15.28:0/1816873454 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 19) ==== 137850+0+0 (secure 0 0 0) 0x7faec801d020 con 0x7faed0109a10 2024-05-12T22:08:49.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.304+0000 7faec7400640 1 --2- 172.21.15.28:0/1816873454 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7faeb00a2c80 0x7faeb00a5140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:49.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.304+0000 7faecea00640 1 -- 172.21.15.28:0/1816873454 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7faeb00a2c80 msgr2=0x7faeb00a5140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/3123328651 2024-05-12T22:08:49.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.304+0000 7faec7400640 1 -- 172.21.15.28:0/1816873454 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7faec80cef90 con 0x7faed0109a10 2024-05-12T22:08:49.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.304+0000 7faecea00640 1 --2- 172.21.15.28:0/1816873454 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7faeb00a2c80 0x7faeb00a5140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-12T22:08:49.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.307+0000 7faec7400640 1 -- 172.21.15.28:0/1816873454 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7faec809bea0 con 0x7faed0109a10 2024-05-12T22:08:49.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:49 smithi186 ceph-mon[34785]: pgmap v379: 97 pgs: 97 active+clean; 641 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-12T22:08:49.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:49 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/1936008260' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-05-12T22:08:49.429 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 12 22:08:49 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-z[36460]: ignoring --setuser ceph since I am not root 2024-05-12T22:08:49.429 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 12 22:08:49 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-z[36460]: ignoring --setgroup ceph since I am not root 2024-05-12T22:08:49.429 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 12 22:08:49 smithi186 ceph-mgr[36480]: -- 172.21.15.186:0/3589446980 <== mon.1 v2:172.21.15.186:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55920b5a05a0 con 0x55920b574c00 2024-05-12T22:08:49.430 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 12 22:08:49 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-z[36460]: 2024-05-12T22:08:49.261+0000 7fb47cc53200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-12T22:08:49.452 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:49 smithi086 ceph-mon[34297]: pgmap v379: 97 pgs: 97 active+clean; 641 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-12T22:08:49.452 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:49 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/1936008260' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-05-12T22:08:49.453 INFO:journalctl@ceph.mgr.y.smithi086.stdout:May 12 22:08:49 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-y[35723]: ignoring --setuser ceph since I am not root 2024-05-12T22:08:49.453 INFO:journalctl@ceph.mgr.y.smithi086.stdout:May 12 22:08:49 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-y[35723]: ignoring --setgroup ceph since I am not root 2024-05-12T22:08:49.453 INFO:journalctl@ceph.mgr.y.smithi086.stdout:May 12 22:08:49 smithi086 ceph-mgr[35727]: -- 172.21.15.86:0/2408111527 <== mon.2 v2:172.21.15.86:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d0e24fa5a0 con 0x55d0e24ce800 2024-05-12T22:08:49.453 INFO:journalctl@ceph.mgr.y.smithi086.stdout:May 12 22:08:49 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-y[35723]: 2024-05-12T22:08:49.268+0000 7f173194f200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-12T22:08:49.453 INFO:journalctl@ceph.mgr.y.smithi086.stdout:May 12 22:08:49 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-y[35723]: 2024-05-12T22:08:49.452+0000 7f173194f200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-12T22:08:49.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.505+0000 7faecea00640 1 -- 172.21.15.28:0/1816873454 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7faeb00a2c80 msgr2=0x7faeb00a5140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/3123328651 2024-05-12T22:08:49.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.505+0000 7faecea00640 1 --2- 172.21.15.28:0/1816873454 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7faeb00a2c80 0x7faeb00a5140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-12T22:08:49.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.515+0000 7faed5a00640 1 -- 172.21.15.28:0/1816873454 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) -- 0x7faed0062430 con 0x7faed0109a10 2024-05-12T22:08:49.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.522+0000 7faec7400640 1 -- 172.21.15.28:0/1816873454 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v25)=0 v25) ==== 138+0+0 (secure 0 0 0) 0x7faec809b7a0 con 0x7faed0109a10 2024-05-12T22:08:49.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.524+0000 7faed5a00640 1 -- 172.21.15.28:0/1816873454 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7faeb00a2c80 msgr2=0x7faeb00a5140 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-12T22:08:49.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.524+0000 7faed5a00640 1 --2- 172.21.15.28:0/1816873454 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7faeb00a2c80 0x7faeb00a5140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:49.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.524+0000 7faed5a00640 1 -- 172.21.15.28:0/1816873454 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faed0109a10 msgr2=0x7faed01a56a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:49.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.524+0000 7faed5a00640 1 --2- 172.21.15.28:0/1816873454 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faed0109a10 0x7faed01a56a0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7faec800efc0 tx=0x7faec800c800 comp rx=0 tx=0).stop 2024-05-12T22:08:49.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.525+0000 7faed5a00640 1 -- 172.21.15.28:0/1816873454 shutdown_connections 2024-05-12T22:08:49.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.525+0000 7faed5a00640 1 --2- 172.21.15.28:0/1816873454 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7faeb00a2c80 0x7faeb00a5140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:49.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.525+0000 7faed5a00640 1 --2- 172.21.15.28:0/1816873454 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faed0109a10 0x7faed01a56a0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:49.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.525+0000 7faed5a00640 1 --2- 172.21.15.28:0/1816873454 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faed0109050 0x7faed019e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:49.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.525+0000 7faed5a00640 1 --2- 172.21.15.28:0/1816873454 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faed0107e70 0x7faed019e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:49.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.525+0000 7faed5a00640 1 -- 172.21.15.28:0/1816873454 >> 172.21.15.28:0/1816873454 conn(0x7faed0076e80 msgr2=0x7faed010c380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:08:49.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.525+0000 7faed5a00640 1 -- 172.21.15.28:0/1816873454 shutdown_connections 2024-05-12T22:08:49.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.525+0000 7faed5a00640 1 -- 172.21.15.28:0/1816873454 wait complete. 2024-05-12T22:08:49.544 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/dump_on_update true' 2024-05-12T22:08:49.663 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 22:08:49 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T22:08:49.402+0000 7f3d35da7200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-12T22:08:49.663 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 22:08:49 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T22:08:49.662+0000 7f3d35da7200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-12T22:08:49.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.733+0000 7f0b45a00640 1 -- 172.21.15.28:0/389658150 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0b40074290 msgr2=0x7f0b40074710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:49.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.733+0000 7f0b45a00640 1 --2- 172.21.15.28:0/389658150 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0b40074290 0x7f0b40074710 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f0b2c009900 tx=0x7f0b2c02a410 comp rx=0 tx=0).stop 2024-05-12T22:08:49.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.733+0000 7f0b45a00640 1 -- 172.21.15.28:0/389658150 shutdown_connections 2024-05-12T22:08:49.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.733+0000 7f0b45a00640 1 --2- 172.21.15.28:0/389658150 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0b4010dc90 0x7f0b40110080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:49.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.733+0000 7f0b45a00640 1 --2- 172.21.15.28:0/389658150 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0b40074290 0x7f0b40074710 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:49.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.733+0000 7f0b45a00640 1 --2- 172.21.15.28:0/389658150 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0b40075870 0x7f0b40073cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:49.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.733+0000 7f0b45a00640 1 -- 172.21.15.28:0/389658150 >> 172.21.15.28:0/389658150 conn(0x7f0b400fcc30 msgr2=0x7f0b400ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:08:49.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.733+0000 7f0b45a00640 1 -- 172.21.15.28:0/389658150 shutdown_connections 2024-05-12T22:08:49.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.733+0000 7f0b45a00640 1 -- 172.21.15.28:0/389658150 wait complete. 2024-05-12T22:08:49.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.734+0000 7f0b45a00640 1 Processor -- start 2024-05-12T22:08:49.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.735+0000 7f0b45a00640 1 -- start start 2024-05-12T22:08:49.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.735+0000 7f0b45a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0b40074290 0x7f0b4019e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:49.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.735+0000 7f0b45a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0b40075870 0x7f0b4019e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:49.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.735+0000 7f0b45a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0b4010dc90 0x7f0b401a55c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:49.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.735+0000 7f0b45a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0b40113960 con 0x7f0b4010dc90 2024-05-12T22:08:49.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.735+0000 7f0b45a00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f0b401137e0 con 0x7f0b40075870 2024-05-12T22:08:49.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.735+0000 7f0b45a00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f0b40113ae0 con 0x7f0b40074290 2024-05-12T22:08:49.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.735+0000 7f0b3fe00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0b40074290 0x7f0b4019e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:49.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.735+0000 7f0b3f400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0b40075870 0x7f0b4019e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:49.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.735+0000 7f0b45000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0b4010dc90 0x7f0b401a55c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:49.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.735+0000 7f0b3fe00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0b40074290 0x7f0b4019e010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:37982/0 (socket says 172.21.15.28:37982) 2024-05-12T22:08:49.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.735+0000 7f0b3fe00640 1 -- 172.21.15.28:0/999567625 learned_addr learned my addr 172.21.15.28:0/999567625 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:08:49.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.736+0000 7f0b3f400640 1 -- 172.21.15.28:0/999567625 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0b40074290 msgr2=0x7f0b4019e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:49.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.736+0000 7f0b3f400640 1 --2- 172.21.15.28:0/999567625 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0b40074290 0x7f0b4019e010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:49.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.736+0000 7f0b3f400640 1 -- 172.21.15.28:0/999567625 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0b4010dc90 msgr2=0x7f0b401a55c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:49.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.736+0000 7f0b3f400640 1 --2- 172.21.15.28:0/999567625 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0b4010dc90 0x7f0b401a55c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:49.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.736+0000 7f0b3f400640 1 -- 172.21.15.28:0/999567625 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b401a5c30 con 0x7f0b40075870 2024-05-12T22:08:49.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.736+0000 7f0b3fe00640 1 --2- 172.21.15.28:0/999567625 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0b40074290 0x7f0b4019e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:08:49.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.736+0000 7f0b3f400640 1 --2- 172.21.15.28:0/999567625 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0b40075870 0x7f0b4019e550 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f0b2c02a930 tx=0x7f0b2c032c70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:08:49.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.736+0000 7f0b3cc00640 1 -- 172.21.15.28:0/999567625 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b2c03f030 con 0x7f0b40075870 2024-05-12T22:08:49.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.736+0000 7f0b3cc00640 1 -- 172.21.15.28:0/999567625 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0b2c003ce0 con 0x7f0b40075870 2024-05-12T22:08:49.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.736+0000 7f0b3cc00640 1 -- 172.21.15.28:0/999567625 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b2c035070 con 0x7f0b40075870 2024-05-12T22:08:49.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.736+0000 7f0b45a00640 1 -- 172.21.15.28:0/999567625 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b401a5ec0 con 0x7f0b40075870 2024-05-12T22:08:49.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.737+0000 7f0b45a00640 1 -- 172.21.15.28:0/999567625 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b401a62d0 con 0x7f0b40075870 2024-05-12T22:08:49.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.738+0000 7f0b45a00640 1 -- 172.21.15.28:0/999567625 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b40103e60 con 0x7f0b40075870 2024-05-12T22:08:49.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.740+0000 7f0b3cc00640 1 -- 172.21.15.28:0/999567625 <== mon.2 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 19) ==== 137850+0+0 (secure 0 0 0) 0x7f0b2c042030 con 0x7f0b40075870 2024-05-12T22:08:49.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.741+0000 7f0b3cc00640 1 --2- 172.21.15.28:0/999567625 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f0b240a2c50 0x7f0b240a5110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:49.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.741+0000 7f0b3fe00640 1 -- 172.21.15.28:0/999567625 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f0b240a2c50 msgr2=0x7f0b240a5110 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/3123328651 2024-05-12T22:08:49.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.741+0000 7f0b3cc00640 1 -- 172.21.15.28:0/999567625 <== mon.2 v2:172.21.15.86:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f0b2c0ef3c0 con 0x7f0b40075870 2024-05-12T22:08:49.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.741+0000 7f0b3fe00640 1 --2- 172.21.15.28:0/999567625 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f0b240a2c50 0x7f0b240a5110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-12T22:08:49.742 INFO:journalctl@ceph.mgr.y.smithi086.stdout:May 12 22:08:49 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-y[35723]: 2024-05-12T22:08:49.556+0000 7f173194f200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-12T22:08:49.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.745+0000 7f0b3cc00640 1 -- 172.21.15.28:0/999567625 <== mon.2 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f0b2c0bc2d0 con 0x7f0b40075870 2024-05-12T22:08:49.859 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 12 22:08:49 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-z[36460]: 2024-05-12T22:08:49.447+0000 7fb47cc53200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-12T22:08:49.859 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 12 22:08:49 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-z[36460]: 2024-05-12T22:08:49.550+0000 7fb47cc53200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-12T22:08:49.859 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 12 22:08:49 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-z[36460]: 2024-05-12T22:08:49.858+0000 7fb47cc53200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-12T22:08:49.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.942+0000 7f0b3fe00640 1 -- 172.21.15.28:0/999567625 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f0b240a2c50 msgr2=0x7f0b240a5110 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/3123328651 2024-05-12T22:08:49.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.942+0000 7f0b3fe00640 1 --2- 172.21.15.28:0/999567625 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f0b240a2c50 0x7f0b240a5110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-12T22:08:49.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.953+0000 7f0b45a00640 1 -- 172.21.15.28:0/999567625 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) -- 0x7f0b400625a0 con 0x7f0b40075870 2024-05-12T22:08:49.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.960+0000 7f0b3cc00640 1 -- 172.21.15.28:0/999567625 <== mon.2 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v26)=0 v26) ==== 133+0+0 (secure 0 0 0) 0x7f0b2c0bbbd0 con 0x7f0b40075870 2024-05-12T22:08:49.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.962+0000 7f0b45a00640 1 -- 172.21.15.28:0/999567625 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f0b240a2c50 msgr2=0x7f0b240a5110 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-12T22:08:49.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.962+0000 7f0b45a00640 1 --2- 172.21.15.28:0/999567625 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f0b240a2c50 0x7f0b240a5110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:49.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.963+0000 7f0b45a00640 1 -- 172.21.15.28:0/999567625 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0b40075870 msgr2=0x7f0b4019e550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:49.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.963+0000 7f0b45a00640 1 --2- 172.21.15.28:0/999567625 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0b40075870 0x7f0b4019e550 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f0b2c02a930 tx=0x7f0b2c032c70 comp rx=0 tx=0).stop 2024-05-12T22:08:49.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.963+0000 7f0b45a00640 1 -- 172.21.15.28:0/999567625 shutdown_connections 2024-05-12T22:08:49.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.963+0000 7f0b45a00640 1 --2- 172.21.15.28:0/999567625 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f0b240a2c50 0x7f0b240a5110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:49.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.963+0000 7f0b45a00640 1 --2- 172.21.15.28:0/999567625 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0b4010dc90 0x7f0b401a55c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:49.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.963+0000 7f0b45a00640 1 --2- 172.21.15.28:0/999567625 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0b40075870 0x7f0b4019e550 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:49.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.963+0000 7f0b45a00640 1 --2- 172.21.15.28:0/999567625 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0b40074290 0x7f0b4019e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:49.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.963+0000 7f0b45a00640 1 -- 172.21.15.28:0/999567625 >> 172.21.15.28:0/999567625 conn(0x7f0b400fcc30 msgr2=0x7f0b40101030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:08:49.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.963+0000 7f0b45a00640 1 -- 172.21.15.28:0/999567625 shutdown_connections 2024-05-12T22:08:49.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:49.963+0000 7f0b45a00640 1 -- 172.21.15.28:0/999567625 wait complete. 2024-05-12T22:08:49.980 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 22:08:49 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T22:08:49.802+0000 7f3d35da7200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-12T22:08:49.983 INFO:teuthology.run_tasks:Running task kclient... 2024-05-12T22:08:50.011 INFO:tasks.kclient:Mounting kernel clients... 2024-05-12T22:08:50.011 INFO:tasks.kclient:config is {'client.0': {}} 2024-05-12T22:08:50.011 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-12T22:08:50.011 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list 2024-05-12T22:08:50.063 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-12T22:08:50.064 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link delete ceph-brx 2024-05-12T22:08:50.134 INFO:teuthology.orchestra.run.smithi028.stderr:Cannot find device "ceph-brx" 2024-05-12T22:08:50.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-12T22:08:50.138 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=legacy', 'wsync'], 'syntax': 'v2', 'mount_subvol_num': 1} 2024-05-12T22:08:50.138 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-05-12T22:08:50.138 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-05-12T22:08:50.138 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f72fecff68e1d400c4568684327c900485c20d6a shell --fsid 6b92e7b4-10aa-11ef-bc98-c7b262605968 -- ceph fs ls 2024-05-12T22:08:50.179 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 12 22:08:49 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-z[36460]: 2024-05-12T22:08:49.944+0000 7fb47cc53200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-12T22:08:50.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:50 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/1936008260' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-05-12T22:08:50.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:50 smithi186 ceph-mon[34785]: mgrmap e19: x(active, since 10m), standbys: z, y 2024-05-12T22:08:50.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:50 smithi186 ceph-mon[34785]: from='client.? ' entity='client.admin' 2024-05-12T22:08:50.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:50 smithi186 ceph-mon[34785]: from='client.? ' entity='client.admin' 2024-05-12T22:08:50.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:50 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/1936008260' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-05-12T22:08:50.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:50 smithi086 ceph-mon[34297]: mgrmap e19: x(active, since 10m), standbys: z, y 2024-05-12T22:08:50.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:50 smithi086 ceph-mon[34297]: from='client.? ' entity='client.admin' 2024-05-12T22:08:50.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:50 smithi086 ceph-mon[34297]: from='client.? ' entity='client.admin' 2024-05-12T22:08:50.243 INFO:journalctl@ceph.mgr.y.smithi086.stdout:May 12 22:08:49 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-y[35723]: 2024-05-12T22:08:49.866+0000 7f173194f200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-12T22:08:50.243 INFO:journalctl@ceph.mgr.y.smithi086.stdout:May 12 22:08:49 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-y[35723]: 2024-05-12T22:08:49.951+0000 7f173194f200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-12T22:08:50.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:50 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/1936008260' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-05-12T22:08:50.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:50 smithi028 ceph-mon[29847]: mgrmap e19: x(active, since 10m), standbys: z, y 2024-05-12T22:08:50.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:50 smithi028 ceph-mon[29847]: from='client.? ' entity='client.admin' 2024-05-12T22:08:50.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:50 smithi028 ceph-mon[29847]: from='client.? ' entity='client.admin' 2024-05-12T22:08:50.286 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 22:08:50 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T22:08:50.225+0000 7f3d35da7200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-12T22:08:50.551 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/mon.a/config 2024-05-12T22:08:50.691 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 22:08:50 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T22:08:50.347+0000 7f3d35da7200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-12T22:08:51.429 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 12 22:08:51 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-z[36460]: 2024-05-12T22:08:51.140+0000 7fb47cc53200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-12T22:08:51.429 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 12 22:08:51 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-z[36460]: 2024-05-12T22:08:51.311+0000 7fb47cc53200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-12T22:08:51.468 INFO:journalctl@ceph.mgr.y.smithi086.stdout:May 12 22:08:51 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-y[35723]: 2024-05-12T22:08:51.133+0000 7f173194f200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-12T22:08:51.468 INFO:journalctl@ceph.mgr.y.smithi086.stdout:May 12 22:08:51 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-y[35723]: 2024-05-12T22:08:51.299+0000 7f173194f200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-12T22:08:51.468 INFO:journalctl@ceph.mgr.y.smithi086.stdout:May 12 22:08:51 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-y[35723]: 2024-05-12T22:08:51.468+0000 7f173194f200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-12T22:08:51.776 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 12 22:08:51 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-z[36460]: 2024-05-12T22:08:51.484+0000 7fb47cc53200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-12T22:08:52.050 INFO:journalctl@ceph.mgr.y.smithi086.stdout:May 12 22:08:51 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-y[35723]: 2024-05-12T22:08:51.756+0000 7f173194f200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-12T22:08:52.051 INFO:journalctl@ceph.mgr.y.smithi086.stdout:May 12 22:08:51 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-y[35723]: 2024-05-12T22:08:51.848+0000 7f173194f200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-12T22:08:52.051 INFO:journalctl@ceph.mgr.y.smithi086.stdout:May 12 22:08:52 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-y[35723]: 2024-05-12T22:08:52.050+0000 7f173194f200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-12T22:08:52.076 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 12 22:08:51 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-z[36460]: 2024-05-12T22:08:51.775+0000 7fb47cc53200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-12T22:08:52.076 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 12 22:08:51 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-z[36460]: 2024-05-12T22:08:51.869+0000 7fb47cc53200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-12T22:08:52.076 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 12 22:08:52 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-z[36460]: 2024-05-12T22:08:52.075+0000 7fb47cc53200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-12T22:08:52.120 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 22:08:51 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T22:08:51.977+0000 7f3d35da7200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-12T22:08:52.372 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 22:08:52 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T22:08:52.220+0000 7f3d35da7200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-12T22:08:52.429 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 12 22:08:52 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-z[36460]: 2024-05-12T22:08:52.269+0000 7fb47cc53200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-12T22:08:52.492 INFO:journalctl@ceph.mgr.y.smithi086.stdout:May 12 22:08:52 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-y[35723]: 2024-05-12T22:08:52.242+0000 7f173194f200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-12T22:08:52.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:52.630+0000 7f199a400700 1 -- 172.21.15.28:0/3123564391 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1994103520 msgr2=0x7f1994103900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:52.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:52.630+0000 7f199a400700 1 --2- 172.21.15.28:0/3123564391 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1994103520 0x7f1994103900 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f1988009990 tx=0x7f1988009ca0 comp rx=0 tx=0).stop 2024-05-12T22:08:52.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:52.630+0000 7f199a400700 1 -- 172.21.15.28:0/3123564391 shutdown_connections 2024-05-12T22:08:52.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:52.630+0000 7f199a400700 1 --2- 172.21.15.28:0/3123564391 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1994103520 0x7f1994103900 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:52.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:52.630+0000 7f199a400700 1 --2- 172.21.15.28:0/3123564391 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f199410c900 0x7f199410ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:52.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:52.630+0000 7f199a400700 1 --2- 172.21.15.28:0/3123564391 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1994103e40 0x7f199410c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:52.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:52.630+0000 7f199a400700 1 -- 172.21.15.28:0/3123564391 >> 172.21.15.28:0/3123564391 conn(0x7f19940fccb0 msgr2=0x7f19940ff0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:08:52.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:52.630+0000 7f199a400700 1 -- 172.21.15.28:0/3123564391 shutdown_connections 2024-05-12T22:08:52.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:52.630+0000 7f199a400700 1 -- 172.21.15.28:0/3123564391 wait complete. 2024-05-12T22:08:52.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:52.631+0000 7f199a400700 1 Processor -- start 2024-05-12T22:08:52.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:52.632+0000 7f199a400700 1 -- start start 2024-05-12T22:08:52.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:52.632+0000 7f199a400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1994103520 0x7f1994101180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:52.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:52.632+0000 7f199a400700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1994103e40 0x7f19941016c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:52.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:52.632+0000 7f199a400700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f199410c900 0x7f1994101c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:52.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:52.632+0000 7f199a400700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1994112600 con 0x7f1994103520 2024-05-12T22:08:52.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:52.632+0000 7f199a400700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f1994112480 con 0x7f1994103e40 2024-05-12T22:08:52.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:52.632+0000 7f199a400700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f1994112780 con 0x7f199410c900 2024-05-12T22:08:52.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:52.632+0000 7f1993e00700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1994103e40 0x7f19941016c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:52.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:52.632+0000 7f1999000700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1994103520 0x7f1994101180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:52.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:52.632+0000 7f1999000700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1994103520 0x7f1994101180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42764/0 (socket says 172.21.15.28:42764) 2024-05-12T22:08:52.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:52.632+0000 7f1993e00700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1994103e40 0x7f19941016c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:38054/0 (socket says 172.21.15.28:38054) 2024-05-12T22:08:52.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:52.632+0000 7f1999000700 1 -- 172.21.15.28:0/1700102214 learned_addr learned my addr 172.21.15.28:0/1700102214 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:08:52.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:52.632+0000 7f1999a00700 1 --2- 172.21.15.28:0/1700102214 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f199410c900 0x7f1994101c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:52.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:52.633+0000 7f1993e00700 1 -- 172.21.15.28:0/1700102214 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f199410c900 msgr2=0x7f1994101c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:52.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:52.633+0000 7f1993e00700 1 --2- 172.21.15.28:0/1700102214 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f199410c900 0x7f1994101c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:52.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:52.633+0000 7f1993e00700 1 -- 172.21.15.28:0/1700102214 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1994103520 msgr2=0x7f1994101180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:52.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:52.633+0000 7f1993e00700 1 --2- 172.21.15.28:0/1700102214 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1994103520 0x7f1994101180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:52.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:52.633+0000 7f1993e00700 1 -- 172.21.15.28:0/1700102214 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19941a6ff0 con 0x7f1994103e40 2024-05-12T22:08:52.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:52.633+0000 7f1993e00700 1 --2- 172.21.15.28:0/1700102214 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1994103e40 0x7f19941016c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f198000e9c0 tx=0x7f198000ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:08:52.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:52.633+0000 7f1991600700 1 -- 172.21.15.28:0/1700102214 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1980004220 con 0x7f1994103e40 2024-05-12T22:08:52.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:52.633+0000 7f199a400700 1 -- 172.21.15.28:0/1700102214 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19941a72e0 con 0x7f1994103e40 2024-05-12T22:08:52.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:52.633+0000 7f1991600700 1 -- 172.21.15.28:0/1700102214 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1980004d10 con 0x7f1994103e40 2024-05-12T22:08:52.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:52.633+0000 7f1991600700 1 -- 172.21.15.28:0/1700102214 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1980005020 con 0x7f1994103e40 2024-05-12T22:08:52.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:52.634+0000 7f199a400700 1 -- 172.21.15.28:0/1700102214 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19941a7820 con 0x7f1994103e40 2024-05-12T22:08:52.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:52.635+0000 7f199a400700 1 -- 172.21.15.28:0/1700102214 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1994069010 con 0x7f1994103e40 2024-05-12T22:08:52.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:52.637+0000 7f1991600700 1 -- 172.21.15.28:0/1700102214 <== mon.2 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 19) ==== 137850+0+0 (secure 0 0 0) 0x7f1980004750 con 0x7f1994103e40 2024-05-12T22:08:52.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:52.638+0000 7f1991600700 1 --2- 172.21.15.28:0/1700102214 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f197c0a2cf0 0x7f197c0a51b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:52.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:52.638+0000 7f1999000700 1 -- 172.21.15.28:0/1700102214 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f197c0a2cf0 msgr2=0x7f197c0a51b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/3123328651 2024-05-12T22:08:52.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:52.638+0000 7f1991600700 1 -- 172.21.15.28:0/1700102214 <== mon.2 v2:172.21.15.86:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f19800cef50 con 0x7f1994103e40 2024-05-12T22:08:52.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:52.638+0000 7f1999000700 1 --2- 172.21.15.28:0/1700102214 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f197c0a2cf0 0x7f197c0a51b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-12T22:08:52.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:52.646+0000 7f1991600700 1 -- 172.21.15.28:0/1700102214 <== mon.2 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f198009be60 con 0x7f1994103e40 2024-05-12T22:08:52.785 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 22:08:52 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T22:08:52.445+0000 7f3d35da7200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-12T22:08:52.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:52.839+0000 7f1999000700 1 -- 172.21.15.28:0/1700102214 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f197c0a2cf0 msgr2=0x7f197c0a51b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/3123328651 2024-05-12T22:08:52.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:52.839+0000 7f1999000700 1 --2- 172.21.15.28:0/1700102214 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f197c0a2cf0 0x7f197c0a51b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-12T22:08:52.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:52.924+0000 7f199a400700 1 -- 172.21.15.28:0/1700102214 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f1994102660 con 0x7f1994103e40 2024-05-12T22:08:52.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:52.924+0000 7f1991600700 1 -- 172.21.15.28:0/1700102214 <== mon.2 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v23) ==== 53+0+73 (secure 0 0 0) 0x7f198009b760 con 0x7f1994103e40 2024-05-12T22:08:52.925 INFO:teuthology.orchestra.run.smithi028.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-05-12T22:08:52.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:52.927+0000 7f199a400700 1 -- 172.21.15.28:0/1700102214 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f197c0a2cf0 msgr2=0x7f197c0a51b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-12T22:08:52.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:52.927+0000 7f199a400700 1 --2- 172.21.15.28:0/1700102214 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f197c0a2cf0 0x7f197c0a51b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:52.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:52.927+0000 7f199a400700 1 -- 172.21.15.28:0/1700102214 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1994103e40 msgr2=0x7f19941016c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:52.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:52.927+0000 7f199a400700 1 --2- 172.21.15.28:0/1700102214 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1994103e40 0x7f19941016c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f198000e9c0 tx=0x7f198000ecd0 comp rx=0 tx=0).stop 2024-05-12T22:08:52.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:52.928+0000 7f199a400700 1 -- 172.21.15.28:0/1700102214 shutdown_connections 2024-05-12T22:08:52.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:52.928+0000 7f199a400700 1 --2- 172.21.15.28:0/1700102214 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f197c0a2cf0 0x7f197c0a51b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:52.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:52.928+0000 7f199a400700 1 --2- 172.21.15.28:0/1700102214 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f199410c900 0x7f1994101c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:52.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:52.928+0000 7f199a400700 1 --2- 172.21.15.28:0/1700102214 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1994103520 0x7f1994101180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:52.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:52.928+0000 7f199a400700 1 --2- 172.21.15.28:0/1700102214 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1994103e40 0x7f19941016c0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:52.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:52.928+0000 7f199a400700 1 -- 172.21.15.28:0/1700102214 >> 172.21.15.28:0/1700102214 conn(0x7f19940fccb0 msgr2=0x7f19940fd9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:08:52.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:52.928+0000 7f199a400700 1 -- 172.21.15.28:0/1700102214 shutdown_connections 2024-05-12T22:08:52.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:52.928+0000 7f199a400700 1 -- 172.21.15.28:0/1700102214 wait complete. 2024-05-12T22:08:53.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:52 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/1700102214' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-05-12T22:08:53.179 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 12 22:08:52 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-z[36460]: 2024-05-12T22:08:52.882+0000 7fb47cc53200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-12T22:08:53.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:52 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/1700102214' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-05-12T22:08:53.242 INFO:journalctl@ceph.mgr.y.smithi086.stdout:May 12 22:08:52 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-y[35723]: 2024-05-12T22:08:52.847+0000 7f173194f200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-12T22:08:53.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:52 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/1700102214' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-05-12T22:08:53.280 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 22:08:52 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T22:08:52.862+0000 7f3d35da7200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-12T22:08:53.281 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 22:08:53 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T22:08:53.006+0000 7f3d35da7200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-12T22:08:53.535 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 22:08:53 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T22:08:53.279+0000 7f3d35da7200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-12T22:08:53.567 INFO:journalctl@ceph.mgr.y.smithi086.stdout:May 12 22:08:53 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-y[35723]: 2024-05-12T22:08:53.270+0000 7f173194f200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-12T22:08:53.568 INFO:journalctl@ceph.mgr.y.smithi086.stdout:May 12 22:08:53 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-y[35723]: 2024-05-12T22:08:53.367+0000 7f173194f200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-12T22:08:53.568 INFO:journalctl@ceph.mgr.y.smithi086.stdout:May 12 22:08:53 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-y[35723]: 2024-05-12T22:08:53.567+0000 7f173194f200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-12T22:08:53.611 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 12 22:08:53 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-z[36460]: 2024-05-12T22:08:53.309+0000 7fb47cc53200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-12T22:08:53.611 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 12 22:08:53 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-z[36460]: 2024-05-12T22:08:53.408+0000 7fb47cc53200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-12T22:08:53.611 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 12 22:08:53 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-z[36460]: 2024-05-12T22:08:53.611+0000 7fb47cc53200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-12T22:08:53.743 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-05-12T22:08:53.744 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-05-12T22:08:53.744 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi028.front.sepia.ceph.com 2024-05-12T22:08:53.744 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-05-12T22:08:53.744 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-05-12T22:08:53.744 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-05-12T22:08:53.744 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-05-12T22:08:53.744 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-05-12T22:08:53.744 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-05-12T22:08:53.744 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-12T22:08:53.744 DEBUG:teuthology.orchestra.run.smithi028:> ip addr 2024-05-12T22:08:53.792 INFO:teuthology.orchestra.run.smithi028.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-05-12T22:08:53.793 INFO:teuthology.orchestra.run.smithi028.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-05-12T22:08:53.793 INFO:teuthology.orchestra.run.smithi028.stdout: inet 127.0.0.1/8 scope host lo 2024-05-12T22:08:53.793 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-05-12T22:08:53.793 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 ::1/128 scope host 2024-05-12T22:08:53.793 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-05-12T22:08:53.793 INFO:teuthology.orchestra.run.smithi028.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-12T22:08:53.793 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:16 brd ff:ff:ff:ff:ff:ff 2024-05-12T22:08:53.793 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f0 2024-05-12T22:08:53.794 INFO:teuthology.orchestra.run.smithi028.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-12T22:08:53.794 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:17 brd ff:ff:ff:ff:ff:ff 2024-05-12T22:08:53.794 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f1 2024-05-12T22:08:53.794 INFO:teuthology.orchestra.run.smithi028.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-05-12T22:08:53.794 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3c brd ff:ff:ff:ff:ff:ff 2024-05-12T22:08:53.794 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f0 2024-05-12T22:08:53.794 INFO:teuthology.orchestra.run.smithi028.stdout: inet 172.21.15.28/20 brd 172.21.15.255 scope global dynamic ens1f0 2024-05-12T22:08:53.794 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft 41947sec preferred_lft 41947sec 2024-05-12T22:08:53.795 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::ae1f:6bff:fea5:ca3c/64 scope link noprefixroute 2024-05-12T22:08:53.795 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-05-12T22:08:53.795 INFO:teuthology.orchestra.run.smithi028.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-12T22:08:53.795 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3d brd ff:ff:ff:ff:ff:ff 2024-05-12T22:08:53.795 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f1 2024-05-12T22:08:53.796 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-05-12T22:08:53.796 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-12T22:08:53.796 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-05-12T22:08:53.796 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link add name ceph-brx type bridge 2024-05-12T22:08:53.796 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip addr flush dev ceph-brx 2024-05-12T22:08:53.796 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link set ceph-brx up 2024-05-12T22:08:53.796 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-05-12T22:08:53.796 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-05-12T22:08:53.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:53 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-12T22:08:53.916 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 22:08:53 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T22:08:53.548+0000 7f3d35da7200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-12T22:08:54.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:54 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-12T22:08:54.269 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-12T22:08:54.269 DEBUG:teuthology.orchestra.run.smithi028:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-05-12T22:08:54.333 INFO:teuthology.orchestra.run.smithi028.stdout:1 2024-05-12T22:08:54.337 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-12T22:08:54.337 DEBUG:teuthology.orchestra.run.smithi028:> route 2024-05-12T22:08:54.390 INFO:teuthology.orchestra.run.smithi028.stdout:Kernel IP routing table 2024-05-12T22:08:54.390 INFO:teuthology.orchestra.run.smithi028.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-05-12T22:08:54.390 INFO:teuthology.orchestra.run.smithi028.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-05-12T22:08:54.390 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-05-12T22:08:54.391 INFO:teuthology.orchestra.run.smithi028.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-05-12T22:08:54.391 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-12T22:08:54.391 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-05-12T22:08:54.391 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-05-12T22:08:54.391 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-05-12T22:08:54.391 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-05-12T22:08:54.391 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-05-12T22:08:54.429 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 12 22:08:54 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-z[36460]: 2024-05-12T22:08:54.136+0000 7fb47cc53200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-12T22:08:54.429 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 12 22:08:54 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-z[36460]: 2024-05-12T22:08:54.228+0000 7fb47cc53200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-12T22:08:54.434 INFO:journalctl@ceph.mgr.y.smithi086.stdout:May 12 22:08:54 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-y[35723]: 2024-05-12T22:08:54.097+0000 7f173194f200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-12T22:08:54.434 INFO:journalctl@ceph.mgr.y.smithi086.stdout:May 12 22:08:54 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-y[35723]: 2024-05-12T22:08:54.191+0000 7f173194f200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-12T22:08:54.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:54 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-12T22:08:54.507 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 22:08:54 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T22:08:54.407+0000 7f3d35da7200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-12T22:08:54.742 INFO:journalctl@ceph.mgr.y.smithi086.stdout:May 12 22:08:54 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-y[35723]: 2024-05-12T22:08:54.434+0000 7f173194f200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-12T22:08:54.742 INFO:journalctl@ceph.mgr.y.smithi086.stdout:May 12 22:08:54 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-y[35723]: 2024-05-12T22:08:54.567+0000 7f173194f200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-12T22:08:54.742 INFO:journalctl@ceph.mgr.y.smithi086.stdout:May 12 22:08:54 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-y[35723]: 2024-05-12T22:08:54.661+0000 7f173194f200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-12T22:08:54.821 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 12 22:08:54 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-z[36460]: 2024-05-12T22:08:54.473+0000 7fb47cc53200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-12T22:08:54.821 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 12 22:08:54 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-z[36460]: 2024-05-12T22:08:54.608+0000 7fb47cc53200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-12T22:08:54.821 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 12 22:08:54 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-z[36460]: 2024-05-12T22:08:54.703+0000 7fb47cc53200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-12T22:08:55.035 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 22:08:54 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T22:08:54.996+0000 7f3d35da7200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-12T22:08:55.079 INFO:journalctl@ceph.mgr.y.smithi086.stdout:May 12 22:08:54 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-y[35723]: 2024-05-12T22:08:54.779+0000 7f173194f200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-12T22:08:55.126 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 12 22:08:54 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-z[36460]: 2024-05-12T22:08:54.821+0000 7fb47cc53200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-12T22:08:55.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:55 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-12T22:08:55.148 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-12T22:08:55.149 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list 2024-05-12T22:08:55.194 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-12T22:08:55.194 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list-id 2024-05-12T22:08:55.284 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-12T22:08:55.284 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-05-12T22:08:55.284 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-12T22:08:55.284 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-05-12T22:08:55.284 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-05-12T22:08:55.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:55 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-12T22:08:55.405 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 22:08:55 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T22:08:55.134+0000 7f3d35da7200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-12T22:08:55.430 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 12 22:08:55 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-z[36460]: 2024-05-12T22:08:55.126+0000 7fb47cc53200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-12T22:08:55.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:55 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-12T22:08:55.461 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-05-12T22:08:55.461 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-12T22:08:55.461 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-05-12T22:08:55.461 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-05-12T22:08:55.461 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-05-12T22:08:55.462 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-05-12T22:08:55.462 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-05-12T22:08:55.462 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-05-12T22:08:55.462 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-05-12T22:08:55.492 INFO:journalctl@ceph.mgr.y.smithi086.stdout:May 12 22:08:55 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-y[35723]: 2024-05-12T22:08:55.079+0000 7f173194f200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-12T22:08:55.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:55 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-12T22:08:55.672 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 22:08:55 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T22:08:55.423+0000 7f3d35da7200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-12T22:08:55.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:55 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-12T22:08:55.759 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-12T22:08:55.759 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-05-12T22:08:55.759 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link set brx.0 up 2024-05-12T22:08:55.759 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link set dev brx.0 master ceph-brx 2024-05-12T22:08:55.760 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-05-12T22:08:55.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:55 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-12T22:08:55.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:55 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-12T22:08:55.961 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-12T22:08:55.962 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-05-12T22:08:55.962 INFO:journalctl@ceph.mgr.y.smithi086.stdout:May 12 22:08:55 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-y[35723]: 2024-05-12T22:08:55.679+0000 7f173194f200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-12T22:08:55.962 INFO:journalctl@ceph.mgr.y.smithi086.stdout:May 12 22:08:55 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-y[35723]: 2024-05-12T22:08:55.892+0000 7f173194f200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-12T22:08:56.004 INFO:teuthology.orchestra.run.smithi028.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-05-12T22:08:56.005 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-12T22:08:56.005 DEBUG:teuthology.orchestra.run.smithi028:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-05-12T22:08:56.095 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-05-12T22:08:56.095 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-05-12T22:08:56.096 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/volumes/qa/sv_1/6f9c1691-e25c-43e3-9e4e-205c2965b112 2024-05-12T22:08:56.096 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-12T22:08:56.096 DEBUG:teuthology.orchestra.run.smithi028:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/volumes/qa/sv_1/6f9c1691-e25c-43e3-9e4e-205c2965b112 /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=legacy,wsync,nofallback 2024-05-12T22:08:56.179 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 12 22:08:55 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-z[36460]: 2024-05-12T22:08:55.731+0000 7fb47cc53200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-12T22:08:56.180 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 12 22:08:55 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-z[36460]: 2024-05-12T22:08:55.949+0000 7fb47cc53200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-12T22:08:56.180 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:55 smithi186 ceph-mon[34785]: Standby manager daemon y restarted 2024-05-12T22:08:56.180 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:55 smithi186 ceph-mon[34785]: Standby manager daemon y started 2024-05-12T22:08:56.180 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:55 smithi186 ceph-mon[34785]: from='mgr.? 172.21.15.86:0/4141958903' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-05-12T22:08:56.180 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:55 smithi186 ceph-mon[34785]: from='mgr.? 172.21.15.86:0/4141958903' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-12T22:08:56.181 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:55 smithi186 ceph-mon[34785]: from='mgr.? 172.21.15.86:0/4141958903' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-05-12T22:08:56.181 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:55 smithi186 ceph-mon[34785]: from='mgr.? 172.21.15.86:0/4141958903' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-12T22:08:56.199 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:55 smithi028 ceph-mon[29847]: Standby manager daemon y restarted 2024-05-12T22:08:56.199 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:55 smithi028 ceph-mon[29847]: Standby manager daemon y started 2024-05-12T22:08:56.199 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:55 smithi028 ceph-mon[29847]: from='mgr.? 172.21.15.86:0/4141958903' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-05-12T22:08:56.199 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:55 smithi028 ceph-mon[29847]: from='mgr.? 172.21.15.86:0/4141958903' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-12T22:08:56.199 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:55 smithi028 ceph-mon[29847]: from='mgr.? 172.21.15.86:0/4141958903' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-05-12T22:08:56.199 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:55 smithi028 ceph-mon[29847]: from='mgr.? 172.21.15.86:0/4141958903' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-12T22:08:56.200 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 22:08:56 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T22:08:56.123+0000 7f3d35da7200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-12T22:08:56.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:55 smithi086 ceph-mon[34297]: Standby manager daemon y restarted 2024-05-12T22:08:56.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:55 smithi086 ceph-mon[34297]: Standby manager daemon y started 2024-05-12T22:08:56.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:55 smithi086 ceph-mon[34297]: from='mgr.? 172.21.15.86:0/4141958903' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-05-12T22:08:56.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:55 smithi086 ceph-mon[34297]: from='mgr.? 172.21.15.86:0/4141958903' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-12T22:08:56.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:55 smithi086 ceph-mon[34297]: from='mgr.? 172.21.15.86:0/4141958903' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-05-12T22:08:56.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:55 smithi086 ceph-mon[34297]: from='mgr.? 172.21.15.86:0/4141958903' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-12T22:08:56.268 INFO:teuthology.orchestra.run.smithi028.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=legacy,wsync,nofallback 2024-05-12T22:08:56.268 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=legacy,wsync". 2024-05-12T22:08:56.535 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 22:08:56 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T22:08:56.260+0000 7f3d35da7200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-12T22:08:56.813 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 22:08:56 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T22:08:56.622+0000 7f3d35da7200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-12T22:08:57.028 INFO:teuthology.orchestra.run.smithi028.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=legacy,wsync,nofallback 2024-05-12T22:08:57.028 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=legacy,wsync". 2024-05-12T22:08:57.028 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: resolved to: "172.21.15.28:6789,172.21.15.86:6789,172.21.15.186:6789" 2024-05-12T22:08:57.028 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: trying mount with new device syntax: 0@6b92e7b4-10aa-11ef-bc98-c7b262605968.cephfs=/volumes/qa/sv_1/6f9c1691-e25c-43e3-9e4e-205c2965b112 2024-05-12T22:08:57.028 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=legacy,wsync,name=0,key=0,mon_addr=172.21.15.28:6789/172.21.15.86:6789/172.21.15.186:6789" will pass to kernel 2024-05-12T22:08:57.037 INFO:teuthology.orchestra.run.smithi028.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-05-12T22:08:57.037 INFO:teuthology.orchestra.run.smithi028.stdout: You just mounted a file system that supports labels which does not 2024-05-12T22:08:57.037 INFO:teuthology.orchestra.run.smithi028.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-05-12T22:08:57.037 INFO:teuthology.orchestra.run.smithi028.stdout: applications will generate AVC messages and not be allowed access to 2024-05-12T22:08:57.037 INFO:teuthology.orchestra.run.smithi028.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-05-12T22:08:57.041 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-05-12T22:08:57.041 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-12T22:08:57.041 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-05-12T22:08:57.099 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 22:08:56 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T22:08:56.812+0000 7f3d35da7200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-12T22:08:57.100 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 22:08:56 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T22:08:56.946+0000 7f3d35da7200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-12T22:08:57.100 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:56 smithi028 ceph-mon[29847]: mgrmap e20: x(active, since 10m), standbys: z, y 2024-05-12T22:08:57.100 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:56 smithi028 ceph-mon[29847]: from='mgr.? 172.21.15.186:0/1626987571' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-05-12T22:08:57.100 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:56 smithi028 ceph-mon[29847]: Standby manager daemon z restarted 2024-05-12T22:08:57.100 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:56 smithi028 ceph-mon[29847]: Standby manager daemon z started 2024-05-12T22:08:57.100 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:56 smithi028 ceph-mon[29847]: from='mgr.? 172.21.15.186:0/1626987571' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-12T22:08:57.100 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:56 smithi028 ceph-mon[29847]: from='mgr.? 172.21.15.186:0/1626987571' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-05-12T22:08:57.100 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:56 smithi028 ceph-mon[29847]: from='mgr.? 172.21.15.186:0/1626987571' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-12T22:08:57.113 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-05-12T22:08:57.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:57 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-05-12T22:08:57.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:57 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-12T22:08:57.228 INFO:teuthology.orchestra.run.smithi028.stdout:client25226 2024-05-12T22:08:57.228 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-05-12T22:08:57.228 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-12T22:08:57.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:56 smithi086 ceph-mon[34297]: mgrmap e20: x(active, since 10m), standbys: z, y 2024-05-12T22:08:57.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:56 smithi086 ceph-mon[34297]: from='mgr.? 172.21.15.186:0/1626987571' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-05-12T22:08:57.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:56 smithi086 ceph-mon[34297]: Standby manager daemon z restarted 2024-05-12T22:08:57.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:56 smithi086 ceph-mon[34297]: Standby manager daemon z started 2024-05-12T22:08:57.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:56 smithi086 ceph-mon[34297]: from='mgr.? 172.21.15.186:0/1626987571' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-12T22:08:57.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:56 smithi086 ceph-mon[34297]: from='mgr.? 172.21.15.186:0/1626987571' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-05-12T22:08:57.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:56 smithi086 ceph-mon[34297]: from='mgr.? 172.21.15.186:0/1626987571' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-12T22:08:57.276 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-05-12T22:08:57.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:57 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-05-12T22:08:57.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:57 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-12T22:08:57.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:56 smithi186 ceph-mon[34785]: mgrmap e20: x(active, since 10m), standbys: z, y 2024-05-12T22:08:57.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:56 smithi186 ceph-mon[34785]: from='mgr.? 172.21.15.186:0/1626987571' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-05-12T22:08:57.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:56 smithi186 ceph-mon[34785]: Standby manager daemon z restarted 2024-05-12T22:08:57.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:56 smithi186 ceph-mon[34785]: Standby manager daemon z started 2024-05-12T22:08:57.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:56 smithi186 ceph-mon[34785]: from='mgr.? 172.21.15.186:0/1626987571' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-12T22:08:57.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:56 smithi186 ceph-mon[34785]: from='mgr.? 172.21.15.186:0/1626987571' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-05-12T22:08:57.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:56 smithi186 ceph-mon[34785]: from='mgr.? 172.21.15.186:0/1626987571' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-12T22:08:57.433 INFO:teuthology.orchestra.run.smithi028.stdout:client25226 2024-05-12T22:08:57.434 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-05-12T22:08:57.513 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 22:08:57 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T22:08:57.128+0000 7f3d35da7200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-12T22:08:57.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:57 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-05-12T22:08:57.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:57 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-12T22:08:57.554 INFO:teuthology.orchestra.run.smithi028.stdout:client25226 2024-05-12T22:08:57.554 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968.client25226/status') 2024-05-12T22:08:57.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:57 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-12T22:08:57.685 INFO:teuthology.orchestra.run.smithi028.stdout:instance: client.25226 (3)172.21.15.28:0/2860389051 2024-05-12T22:08:57.685 INFO:teuthology.orchestra.run.smithi028.stdout:blocklisted: false 2024-05-12T22:08:57.686 INFO:teuthology.orchestra.run.smithi028.stderr:0+1 records in 2024-05-12T22:08:57.686 INFO:teuthology.orchestra.run.smithi028.stderr:0+1 records out 2024-05-12T22:08:57.686 INFO:teuthology.orchestra.run.smithi028.stderr:71 bytes copied, 0.000102327 s, 694 kB/s 2024-05-12T22:08:57.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:57 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-12T22:08:57.698 INFO:teuthology.run_tasks:Running task exec... 2024-05-12T22:08:57.709 INFO:teuthology.task.exec:Executing custom commands... 2024-05-12T22:08:57.709 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi028.front.sepia.ceph.com 2024-05-12T22:08:57.709 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate false' 2024-05-12T22:08:57.784 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 22:08:57 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T22:08:57.512+0000 7f3d35da7200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-12T22:08:57.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:57.914+0000 7f9801a00640 1 -- 172.21.15.28:0/1520171133 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97fc101750 msgr2=0x7f97fc101b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:57.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:57.914+0000 7f9801a00640 1 --2- 172.21.15.28:0/1520171133 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97fc101750 0x7f97fc101b70 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f97ec009ca0 tx=0x7f97ec02a8b0 comp rx=0 tx=0).stop 2024-05-12T22:08:57.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:57.915+0000 7f9801a00640 1 -- 172.21.15.28:0/1520171133 shutdown_connections 2024-05-12T22:08:57.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:57.915+0000 7f9801a00640 1 --2- 172.21.15.28:0/1520171133 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f97fc103300 0x7f97fc109b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:57.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:57.915+0000 7f9801a00640 1 --2- 172.21.15.28:0/1520171133 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f97fc102940 0x7f97fc102dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:57.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:57.915+0000 7f9801a00640 1 --2- 172.21.15.28:0/1520171133 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97fc101750 0x7f97fc101b70 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:57.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:57.915+0000 7f9801a00640 1 -- 172.21.15.28:0/1520171133 >> 172.21.15.28:0/1520171133 conn(0x7f97fc0fccc0 msgr2=0x7f97fc0ff100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:08:57.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:57.915+0000 7f9801a00640 1 -- 172.21.15.28:0/1520171133 shutdown_connections 2024-05-12T22:08:57.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:57.915+0000 7f9801a00640 1 -- 172.21.15.28:0/1520171133 wait complete. 2024-05-12T22:08:57.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:57.916+0000 7f9801a00640 1 Processor -- start 2024-05-12T22:08:57.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:57.916+0000 7f9801a00640 1 -- start start 2024-05-12T22:08:57.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:57.917+0000 7f9801a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97fc101750 0x7f97fc0755e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:57.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:57.917+0000 7f9801a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f97fc102940 0x7f97fc073c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:57.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:57.917+0000 7f9801a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f97fc103300 0x7f97fc074160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:57.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:57.917+0000 7f9801a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f97fc10d360 con 0x7f97fc101750 2024-05-12T22:08:57.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:57.917+0000 7f9801a00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f97fc10d1e0 con 0x7f97fc103300 2024-05-12T22:08:57.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:57.917+0000 7f9801a00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f97fc10d4e0 con 0x7f97fc102940 2024-05-12T22:08:57.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:57.917+0000 7f97fbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97fc101750 0x7f97fc0755e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:57.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:57.917+0000 7f97fb400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f97fc102940 0x7f97fc073c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:57.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:57.917+0000 7f97fbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97fc101750 0x7f97fc0755e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50986/0 (socket says 172.21.15.28:50986) 2024-05-12T22:08:57.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:57.917+0000 7f97fbe00640 1 -- 172.21.15.28:0/3119899590 learned_addr learned my addr 172.21.15.28:0/3119899590 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:08:57.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:57.917+0000 7f9801000640 1 --2- 172.21.15.28:0/3119899590 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f97fc103300 0x7f97fc074160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:57.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:57.918+0000 7f97fbe00640 1 -- 172.21.15.28:0/3119899590 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f97fc102940 msgr2=0x7f97fc073c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:57.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:57.918+0000 7f97fbe00640 1 --2- 172.21.15.28:0/3119899590 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f97fc102940 0x7f97fc073c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:57.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:57.918+0000 7f97fbe00640 1 -- 172.21.15.28:0/3119899590 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f97fc103300 msgr2=0x7f97fc074160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:57.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:57.918+0000 7f97fbe00640 1 --2- 172.21.15.28:0/3119899590 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f97fc103300 0x7f97fc074160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:57.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:57.918+0000 7f97fbe00640 1 -- 172.21.15.28:0/3119899590 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97fc0749f0 con 0x7f97fc101750 2024-05-12T22:08:57.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:57.918+0000 7f97fb400640 1 --2- 172.21.15.28:0/3119899590 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f97fc102940 0x7f97fc073c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:08:57.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:57.918+0000 7f97fbe00640 1 --2- 172.21.15.28:0/3119899590 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97fc101750 0x7f97fc0755e0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f97ec009880 tx=0x7f97ec009360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:08:57.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:57.918+0000 7f97f8c00640 1 -- 172.21.15.28:0/3119899590 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97ec004040 con 0x7f97fc101750 2024-05-12T22:08:57.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:57.918+0000 7f97f8c00640 1 -- 172.21.15.28:0/3119899590 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f97ec040070 con 0x7f97fc101750 2024-05-12T22:08:57.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:57.918+0000 7f9801a00640 1 -- 172.21.15.28:0/3119899590 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97fc1a1b00 con 0x7f97fc101750 2024-05-12T22:08:57.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:57.918+0000 7f97f8c00640 1 -- 172.21.15.28:0/3119899590 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97ec045ce0 con 0x7f97fc101750 2024-05-12T22:08:57.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:57.919+0000 7f9801a00640 1 -- 172.21.15.28:0/3119899590 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97fc1a20a0 con 0x7f97fc101750 2024-05-12T22:08:57.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:57.920+0000 7f9801a00640 1 -- 172.21.15.28:0/3119899590 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97fc108a70 con 0x7f97fc101750 2024-05-12T22:08:57.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:57.921+0000 7f97f8c00640 1 -- 172.21.15.28:0/3119899590 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 21) ==== 137850+0+0 (secure 0 0 0) 0x7f97ec045470 con 0x7f97fc101750 2024-05-12T22:08:57.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:57.924+0000 7f97f8c00640 1 --2- 172.21.15.28:0/3119899590 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f97e00a2c50 0x7f97e00a5110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:57.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:57.924+0000 7f97fb400640 1 -- 172.21.15.28:0/3119899590 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f97e00a2c50 msgr2=0x7f97e00a5110 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/3123328651 2024-05-12T22:08:57.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:57.924+0000 7f97f8c00640 1 -- 172.21.15.28:0/3119899590 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8211+0+0 (secure 0 0 0) 0x7f97ec0f01d0 con 0x7f97fc101750 2024-05-12T22:08:57.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:57.924+0000 7f97fb400640 1 --2- 172.21.15.28:0/3119899590 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f97e00a2c50 0x7f97e00a5110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-12T22:08:57.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:57.929+0000 7f97f8c00640 1 -- 172.21.15.28:0/3119899590 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f97ec0bd0e0 con 0x7f97fc101750 2024-05-12T22:08:58.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:58.125+0000 7f97fb400640 1 -- 172.21.15.28:0/3119899590 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f97e00a2c50 msgr2=0x7f97e00a5110 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/3123328651 2024-05-12T22:08:58.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:58.125+0000 7f97fb400640 1 --2- 172.21.15.28:0/3119899590 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f97e00a2c50 0x7f97e00a5110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-12T22:08:58.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:58.177+0000 7f9801a00640 1 -- 172.21.15.28:0/3119899590 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"} v 0) -- 0x7f97fc062430 con 0x7f97fc101750 2024-05-12T22:08:58.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:57 smithi086 ceph-mon[34297]: mgrmap e21: x(active, since 10m), standbys: z, y 2024-05-12T22:08:58.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:57 smithi028 ceph-mon[29847]: mgrmap e21: x(active, since 10m), standbys: z, y 2024-05-12T22:08:58.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:57 smithi186 ceph-mon[34785]: mgrmap e21: x(active, since 10m), standbys: z, y 2024-05-12T22:08:58.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:58.526+0000 7f97fb400640 1 -- 172.21.15.28:0/3119899590 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f97e00a2c50 msgr2=0x7f97e00a5110 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/3123328651 2024-05-12T22:08:58.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:58.526+0000 7f97fb400640 1 --2- 172.21.15.28:0/3119899590 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f97e00a2c50 0x7f97e00a5110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-12T22:08:58.634 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 22:08:58 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T22:08:58.330+0000 7f3d35da7200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-12T22:08:58.634 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 22:08:58 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T22:08:58.633+0000 7f3d35da7200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-12T22:08:58.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:58.657+0000 7f97f8c00640 1 -- 172.21.15.28:0/3119899590 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mgrmap(e 22) ==== 137518+0+0 (secure 0 0 0) 0x7f97ec0b9050 con 0x7f97fc101750 2024-05-12T22:08:58.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:58.658+0000 7f97f8c00640 1 -- 172.21.15.28:0/3119899590 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f97e00a2c50 msgr2=0x7f97e00a5110 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-12T22:08:58.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:58.658+0000 7f97f8c00640 1 --2- 172.21.15.28:0/3119899590 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f97e00a2c50 0x7f97e00a5110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:58.958 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 22:08:58 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T22:08:58.956+0000 7f3c88600700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-12T22:08:58.958 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 22:08:58 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T22:08:58.956+0000 7f3c88600700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-12T22:08:58.958 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 22:08:58 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T22:08:58.956+0000 7f3c88600700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-12T22:08:58.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:58.983+0000 7f97f8c00640 1 -- 172.21.15.28:0/3119899590 <== mon.0 v2:172.21.15.28:3300/0 8 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]=0 v24) ==== 118+0+0 (secure 0 0 0) 0x7f97ec0098b0 con 0x7f97fc101750 2024-05-12T22:08:58.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:58.986+0000 7f9801a00640 1 -- 172.21.15.28:0/3119899590 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97fc101750 msgr2=0x7f97fc0755e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:58.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:58.986+0000 7f9801a00640 1 --2- 172.21.15.28:0/3119899590 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97fc101750 0x7f97fc0755e0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f97ec009880 tx=0x7f97ec009360 comp rx=0 tx=0).stop 2024-05-12T22:08:58.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:58.987+0000 7f9801a00640 1 -- 172.21.15.28:0/3119899590 shutdown_connections 2024-05-12T22:08:58.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:58.987+0000 7f9801a00640 1 --2- 172.21.15.28:0/3119899590 >> [v2:172.21.15.28:6800/3123328651,v1:172.21.15.28:6801/3123328651] conn(0x7f97e00a2c50 0x7f97e00a5110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:58.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:58.987+0000 7f9801a00640 1 --2- 172.21.15.28:0/3119899590 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f97fc103300 0x7f97fc074160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:58.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:58.987+0000 7f9801a00640 1 --2- 172.21.15.28:0/3119899590 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f97fc102940 0x7f97fc073c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:58.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:58.987+0000 7f9801a00640 1 --2- 172.21.15.28:0/3119899590 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97fc101750 0x7f97fc0755e0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:58.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:58.987+0000 7f9801a00640 1 -- 172.21.15.28:0/3119899590 >> 172.21.15.28:0/3119899590 conn(0x7f97fc0fccc0 msgr2=0x7f97fc105c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:08:58.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:58.988+0000 7f9801a00640 1 -- 172.21.15.28:0/3119899590 shutdown_connections 2024-05-12T22:08:58.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:58.988+0000 7f9801a00640 1 -- 172.21.15.28:0/3119899590 wait complete. 2024-05-12T22:08:59.013 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs subvolumegroup pin cephfs qa random 0.10' 2024-05-12T22:08:59.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:58 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/3119899590' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-05-12T22:08:59.180 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:58 smithi186 ceph-mon[34785]: Active manager daemon x restarted 2024-05-12T22:08:59.180 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:58 smithi186 ceph-mon[34785]: Activating manager daemon x 2024-05-12T22:08:59.180 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:58 smithi186 ceph-mon[34785]: osdmap e89: 12 total, 12 up, 12 in 2024-05-12T22:08:59.180 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:58 smithi186 ceph-mon[34785]: mgrmap e22: x(active, starting, since 0.0204304s), standbys: z, y 2024-05-12T22:08:59.180 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:58 smithi186 ceph-mon[34785]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-12T22:08:59.180 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:58 smithi186 ceph-mon[34785]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-12T22:08:59.180 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:58 smithi186 ceph-mon[34785]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-12T22:08:59.180 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:58 smithi186 ceph-mon[34785]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-12T22:08:59.180 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:58 smithi186 ceph-mon[34785]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-12T22:08:59.180 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:58 smithi186 ceph-mon[34785]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-12T22:08:59.180 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:58 smithi186 ceph-mon[34785]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-12T22:08:59.180 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:58 smithi186 ceph-mon[34785]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-12T22:08:59.181 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:58 smithi186 ceph-mon[34785]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-12T22:08:59.181 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:58 smithi186 ceph-mon[34785]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-12T22:08:59.181 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:58 smithi186 ceph-mon[34785]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-12T22:08:59.181 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:58 smithi186 ceph-mon[34785]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-12T22:08:59.181 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:58 smithi186 ceph-mon[34785]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-12T22:08:59.181 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:58 smithi186 ceph-mon[34785]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-12T22:08:59.181 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:58 smithi186 ceph-mon[34785]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-12T22:08:59.181 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:58 smithi186 ceph-mon[34785]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-12T22:08:59.181 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:58 smithi186 ceph-mon[34785]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-05-12T22:08:59.181 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:58 smithi186 ceph-mon[34785]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-12T22:08:59.181 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:58 smithi186 ceph-mon[34785]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-12T22:08:59.181 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:58 smithi186 ceph-mon[34785]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-12T22:08:59.181 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:58 smithi186 ceph-mon[34785]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-12T22:08:59.181 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:58 smithi186 ceph-mon[34785]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-12T22:08:59.181 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:58 smithi186 ceph-mon[34785]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-12T22:08:59.182 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:58 smithi186 ceph-mon[34785]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-12T22:08:59.182 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:58 smithi186 ceph-mon[34785]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-12T22:08:59.182 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:58 smithi186 ceph-mon[34785]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-12T22:08:59.182 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:58 smithi186 ceph-mon[34785]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-12T22:08:59.182 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:58 smithi186 ceph-mon[34785]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-12T22:08:59.182 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:58 smithi186 ceph-mon[34785]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-12T22:08:59.182 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:58 smithi186 ceph-mon[34785]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-12T22:08:59.182 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:58 smithi186 ceph-mon[34785]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-12T22:08:59.182 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:58 smithi186 ceph-mon[34785]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-12T22:08:59.183 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:58 smithi186 ceph-mon[34785]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-12T22:08:59.183 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:58 smithi186 ceph-mon[34785]: Manager daemon x is now available 2024-05-12T22:08:59.183 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:58 smithi186 ceph-mon[34785]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:08:59.183 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:58 smithi186 ceph-mon[34785]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-05-12T22:08:59.183 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:58 smithi186 ceph-mon[34785]: from='mgr.25189 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-05-12T22:08:59.183 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:58 smithi186 ceph-mon[34785]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-05-12T22:08:59.183 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:58 smithi186 ceph-mon[34785]: from='mgr.25189 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-05-12T22:08:59.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:59.184+0000 7fdea3000640 1 -- 172.21.15.28:0/2313230033 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fde9c1016f0 msgr2=0x7fde9c101b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:59.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:59.184+0000 7fdea3000640 1 --2- 172.21.15.28:0/2313230033 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fde9c1016f0 0x7fde9c101b10 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fde90009900 tx=0x7fde9002a410 comp rx=0 tx=0).stop 2024-05-12T22:08:59.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:59.184+0000 7fdea3000640 1 -- 172.21.15.28:0/2313230033 shutdown_connections 2024-05-12T22:08:59.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:59.184+0000 7fdea3000640 1 --2- 172.21.15.28:0/2313230033 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fde9c1032a0 0x7fde9c109b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:59.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:59.184+0000 7fdea3000640 1 --2- 172.21.15.28:0/2313230033 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde9c1028e0 0x7fde9c102d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:59.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:59.184+0000 7fdea3000640 1 --2- 172.21.15.28:0/2313230033 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fde9c1016f0 0x7fde9c101b10 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:59.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:59.184+0000 7fdea3000640 1 -- 172.21.15.28:0/2313230033 >> 172.21.15.28:0/2313230033 conn(0x7fde9c0fcc60 msgr2=0x7fde9c0ff0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:08:59.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:59.184+0000 7fdea3000640 1 -- 172.21.15.28:0/2313230033 shutdown_connections 2024-05-12T22:08:59.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:59.184+0000 7fdea3000640 1 -- 172.21.15.28:0/2313230033 wait complete. 2024-05-12T22:08:59.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:59.185+0000 7fdea3000640 1 Processor -- start 2024-05-12T22:08:59.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:59.185+0000 7fdea3000640 1 -- start start 2024-05-12T22:08:59.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:59.186+0000 7fdea3000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde9c1016f0 0x7fde9c1979a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:59.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:59.186+0000 7fdea3000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fde9c1028e0 0x7fde9c197ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:59.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:59.186+0000 7fdea3000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fde9c1032a0 0x7fde9c19ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:59.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:59.186+0000 7fdea3000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fde9c10d390 con 0x7fde9c1016f0 2024-05-12T22:08:59.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:59.186+0000 7fdea3000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fde9c10d210 con 0x7fde9c1028e0 2024-05-12T22:08:59.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:59.186+0000 7fdea3000640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fde9c10d510 con 0x7fde9c1032a0 2024-05-12T22:08:59.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:59.186+0000 7fdea1c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde9c1016f0 0x7fde9c1979a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:59.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:59.186+0000 7fdea1200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fde9c1028e0 0x7fde9c197ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:59.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:59.186+0000 7fdea1200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fde9c1028e0 0x7fde9c197ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:49192/0 (socket says 172.21.15.28:49192) 2024-05-12T22:08:59.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:59.186+0000 7fdea1200640 1 -- 172.21.15.28:0/277833666 learned_addr learned my addr 172.21.15.28:0/277833666 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:08:59.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:59.186+0000 7fdea1c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde9c1016f0 0x7fde9c1979a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51126/0 (socket says 172.21.15.28:51126) 2024-05-12T22:08:59.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:59.186+0000 7fdea2600640 1 --2- 172.21.15.28:0/277833666 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fde9c1032a0 0x7fde9c19ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:59.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:59.187+0000 7fdea1c00640 1 -- 172.21.15.28:0/277833666 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fde9c1032a0 msgr2=0x7fde9c19ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:59.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:59.187+0000 7fdea1c00640 1 --2- 172.21.15.28:0/277833666 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fde9c1032a0 0x7fde9c19ef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:59.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:59.187+0000 7fdea1c00640 1 -- 172.21.15.28:0/277833666 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fde9c1028e0 msgr2=0x7fde9c197ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:59.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:59.187+0000 7fdea1c00640 1 --2- 172.21.15.28:0/277833666 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fde9c1028e0 0x7fde9c197ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:59.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:59.187+0000 7fdea1c00640 1 -- 172.21.15.28:0/277833666 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde9c19f5c0 con 0x7fde9c1016f0 2024-05-12T22:08:59.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:59.187+0000 7fdea1200640 1 --2- 172.21.15.28:0/277833666 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fde9c1028e0 0x7fde9c197ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:08:59.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:59.187+0000 7fdea1c00640 1 --2- 172.21.15.28:0/277833666 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde9c1016f0 0x7fde9c1979a0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fde90009230 tx=0x7fde90032ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:08:59.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:59.187+0000 7fde9a000640 1 -- 172.21.15.28:0/277833666 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde9003f030 con 0x7fde9c1016f0 2024-05-12T22:08:59.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:59.187+0000 7fde9a000640 1 -- 172.21.15.28:0/277833666 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fde90003ce0 con 0x7fde9c1016f0 2024-05-12T22:08:59.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:59.187+0000 7fdea3000640 1 -- 172.21.15.28:0/277833666 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde9c19f850 con 0x7fde9c1016f0 2024-05-12T22:08:59.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:59.187+0000 7fde9a000640 1 -- 172.21.15.28:0/277833666 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde90035070 con 0x7fde9c1016f0 2024-05-12T22:08:59.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:59.187+0000 7fdea3000640 1 -- 172.21.15.28:0/277833666 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde9c19fd10 con 0x7fde9c1016f0 2024-05-12T22:08:59.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:59.189+0000 7fdea3000640 1 -- 172.21.15.28:0/277833666 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde9c108a10 con 0x7fde9c1016f0 2024-05-12T22:08:59.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:59.191+0000 7fde9a000640 1 -- 172.21.15.28:0/277833666 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 137518+0+0 (secure 0 0 0) 0x7fde90042030 con 0x7fde9c1016f0 2024-05-12T22:08:59.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:59.192+0000 7fde9a000640 1 -- 172.21.15.28:0/277833666 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7fde900eefe0 con 0x7fde9c1016f0 2024-05-12T22:08:59.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:59.196+0000 7fde9a000640 1 -- 172.21.15.28:0/277833666 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fde900bbc00 con 0x7fde9c1016f0 2024-05-12T22:08:59.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:58 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/3119899590' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-05-12T22:08:59.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:58 smithi086 ceph-mon[34297]: Active manager daemon x restarted 2024-05-12T22:08:59.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:58 smithi086 ceph-mon[34297]: Activating manager daemon x 2024-05-12T22:08:59.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:58 smithi086 ceph-mon[34297]: osdmap e89: 12 total, 12 up, 12 in 2024-05-12T22:08:59.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:58 smithi086 ceph-mon[34297]: mgrmap e22: x(active, starting, since 0.0204304s), standbys: z, y 2024-05-12T22:08:59.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:58 smithi086 ceph-mon[34297]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-12T22:08:59.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:58 smithi086 ceph-mon[34297]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-12T22:08:59.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:58 smithi086 ceph-mon[34297]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-12T22:08:59.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:58 smithi086 ceph-mon[34297]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-12T22:08:59.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:58 smithi086 ceph-mon[34297]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-12T22:08:59.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:58 smithi086 ceph-mon[34297]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-12T22:08:59.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:58 smithi086 ceph-mon[34297]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-12T22:08:59.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:58 smithi086 ceph-mon[34297]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-12T22:08:59.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:58 smithi086 ceph-mon[34297]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-12T22:08:59.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:58 smithi086 ceph-mon[34297]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-12T22:08:59.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:58 smithi086 ceph-mon[34297]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-12T22:08:59.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:58 smithi086 ceph-mon[34297]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-12T22:08:59.244 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:58 smithi086 ceph-mon[34297]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-12T22:08:59.244 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:58 smithi086 ceph-mon[34297]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-12T22:08:59.244 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:58 smithi086 ceph-mon[34297]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-12T22:08:59.244 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:58 smithi086 ceph-mon[34297]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-12T22:08:59.244 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:58 smithi086 ceph-mon[34297]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-05-12T22:08:59.244 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:58 smithi086 ceph-mon[34297]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-12T22:08:59.244 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:58 smithi086 ceph-mon[34297]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-12T22:08:59.244 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:58 smithi086 ceph-mon[34297]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-12T22:08:59.244 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:58 smithi086 ceph-mon[34297]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-12T22:08:59.244 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:58 smithi086 ceph-mon[34297]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-12T22:08:59.244 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:58 smithi086 ceph-mon[34297]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-12T22:08:59.244 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:58 smithi086 ceph-mon[34297]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-12T22:08:59.244 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:58 smithi086 ceph-mon[34297]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-12T22:08:59.245 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:58 smithi086 ceph-mon[34297]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-12T22:08:59.245 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:58 smithi086 ceph-mon[34297]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-12T22:08:59.245 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:58 smithi086 ceph-mon[34297]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-12T22:08:59.245 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:58 smithi086 ceph-mon[34297]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-12T22:08:59.245 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:58 smithi086 ceph-mon[34297]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-12T22:08:59.245 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:58 smithi086 ceph-mon[34297]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-12T22:08:59.245 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:58 smithi086 ceph-mon[34297]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-12T22:08:59.245 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:58 smithi086 ceph-mon[34297]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-12T22:08:59.245 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:58 smithi086 ceph-mon[34297]: Manager daemon x is now available 2024-05-12T22:08:59.245 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:58 smithi086 ceph-mon[34297]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:08:59.245 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:58 smithi086 ceph-mon[34297]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-05-12T22:08:59.245 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:58 smithi086 ceph-mon[34297]: from='mgr.25189 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-05-12T22:08:59.245 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:58 smithi086 ceph-mon[34297]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-05-12T22:08:59.245 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:58 smithi086 ceph-mon[34297]: from='mgr.25189 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-05-12T22:08:59.285 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 22:08:58 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T22:08:58.956+0000 7f3c88600700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-12T22:08:59.285 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 22:08:58 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T22:08:58.956+0000 7f3c88600700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-12T22:08:59.285 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 22:08:58 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T22:08:58.957+0000 7f3c8b800700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-12T22:08:59.285 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 22:08:58 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T22:08:58.957+0000 7f3c8b800700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-12T22:08:59.285 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 22:08:58 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T22:08:58.957+0000 7f3c8b800700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-12T22:08:59.285 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 22:08:58 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T22:08:58.957+0000 7f3c8b800700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-12T22:08:59.285 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 22:08:58 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T22:08:58.957+0000 7f3c8b800700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-12T22:08:59.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:58 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/3119899590' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-05-12T22:08:59.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:58 smithi028 ceph-mon[29847]: Active manager daemon x restarted 2024-05-12T22:08:59.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:58 smithi028 ceph-mon[29847]: Activating manager daemon x 2024-05-12T22:08:59.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:58 smithi028 ceph-mon[29847]: osdmap e89: 12 total, 12 up, 12 in 2024-05-12T22:08:59.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:58 smithi028 ceph-mon[29847]: mgrmap e22: x(active, starting, since 0.0204304s), standbys: z, y 2024-05-12T22:08:59.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:58 smithi028 ceph-mon[29847]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-12T22:08:59.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:58 smithi028 ceph-mon[29847]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-12T22:08:59.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:58 smithi028 ceph-mon[29847]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-12T22:08:59.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:58 smithi028 ceph-mon[29847]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-12T22:08:59.287 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:58 smithi028 ceph-mon[29847]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-12T22:08:59.287 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:58 smithi028 ceph-mon[29847]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-12T22:08:59.287 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:58 smithi028 ceph-mon[29847]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-12T22:08:59.287 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:58 smithi028 ceph-mon[29847]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-12T22:08:59.287 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:58 smithi028 ceph-mon[29847]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-12T22:08:59.287 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:58 smithi028 ceph-mon[29847]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-12T22:08:59.287 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:58 smithi028 ceph-mon[29847]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-12T22:08:59.287 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:58 smithi028 ceph-mon[29847]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-12T22:08:59.287 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:58 smithi028 ceph-mon[29847]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-12T22:08:59.287 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:58 smithi028 ceph-mon[29847]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-12T22:08:59.287 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:58 smithi028 ceph-mon[29847]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-12T22:08:59.287 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:58 smithi028 ceph-mon[29847]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-12T22:08:59.288 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:58 smithi028 ceph-mon[29847]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-05-12T22:08:59.288 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:58 smithi028 ceph-mon[29847]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-12T22:08:59.288 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:58 smithi028 ceph-mon[29847]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-12T22:08:59.288 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:58 smithi028 ceph-mon[29847]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-12T22:08:59.288 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:58 smithi028 ceph-mon[29847]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-12T22:08:59.288 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:58 smithi028 ceph-mon[29847]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-12T22:08:59.288 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:58 smithi028 ceph-mon[29847]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-12T22:08:59.288 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:58 smithi028 ceph-mon[29847]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-12T22:08:59.288 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:58 smithi028 ceph-mon[29847]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-12T22:08:59.288 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:58 smithi028 ceph-mon[29847]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-12T22:08:59.288 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:58 smithi028 ceph-mon[29847]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-12T22:08:59.288 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:58 smithi028 ceph-mon[29847]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-12T22:08:59.288 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:58 smithi028 ceph-mon[29847]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-12T22:08:59.288 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:58 smithi028 ceph-mon[29847]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-12T22:08:59.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:58 smithi028 ceph-mon[29847]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-12T22:08:59.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:58 smithi028 ceph-mon[29847]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-12T22:08:59.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:58 smithi028 ceph-mon[29847]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-12T22:08:59.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:58 smithi028 ceph-mon[29847]: Manager daemon x is now available 2024-05-12T22:08:59.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:58 smithi028 ceph-mon[29847]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:08:59.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:58 smithi028 ceph-mon[29847]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-05-12T22:08:59.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:58 smithi028 ceph-mon[29847]: from='mgr.25189 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-05-12T22:08:59.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:58 smithi028 ceph-mon[29847]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-05-12T22:08:59.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:58 smithi028 ceph-mon[29847]: from='mgr.25189 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-05-12T22:08:59.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:59.675+0000 7fde9a000640 1 -- 172.21.15.28:0/277833666 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mgrmap(e 23) ==== 137645+0+0 (secure 0 0 0) 0x7fde9003b460 con 0x7fde9c1016f0 2024-05-12T22:08:59.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:59.678+0000 7fde9a000640 1 --2- 172.21.15.28:0/277833666 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fde840a9880 0x7fde840abc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:59.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:59.678+0000 7fde9a000640 1 -- 172.21.15.28:0/277833666 --> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}) -- 0x7fde90003b30 con 0x7fde840a9880 2024-05-12T22:08:59.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:59.681+0000 7fdea1200640 1 --2- 172.21.15.28:0/277833666 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fde840a9880 0x7fde840abc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:59.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:59.684+0000 7fdea1200640 1 --2- 172.21.15.28:0/277833666 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fde840a9880 0x7fde840abc70 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fde9c198eb0 tx=0x7fde88005f90 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:08:59.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:59.709+0000 7fde9a000640 1 -- 172.21.15.28:0/277833666 <== mgr.25189 v2:172.21.15.28:6800/3177640985 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+3 (secure 0 0 0) 0x7fde90003b30 con 0x7fde840a9880 2024-05-12T22:08:59.710 INFO:teuthology.orchestra.run.smithi028.stdout:{} 2024-05-12T22:08:59.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:59.712+0000 7fdea3000640 1 -- 172.21.15.28:0/277833666 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fde840a9880 msgr2=0x7fde840abc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:59.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:59.712+0000 7fdea3000640 1 --2- 172.21.15.28:0/277833666 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fde840a9880 0x7fde840abc70 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fde9c198eb0 tx=0x7fde88005f90 comp rx=0 tx=0).stop 2024-05-12T22:08:59.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:59.712+0000 7fdea3000640 1 -- 172.21.15.28:0/277833666 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde9c1016f0 msgr2=0x7fde9c1979a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:59.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:59.712+0000 7fdea3000640 1 --2- 172.21.15.28:0/277833666 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde9c1016f0 0x7fde9c1979a0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fde90009230 tx=0x7fde90032ce0 comp rx=0 tx=0).stop 2024-05-12T22:08:59.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:59.712+0000 7fdea3000640 1 -- 172.21.15.28:0/277833666 shutdown_connections 2024-05-12T22:08:59.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:59.712+0000 7fdea3000640 1 --2- 172.21.15.28:0/277833666 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fde840a9880 0x7fde840abc70 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:59.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:59.712+0000 7fdea3000640 1 --2- 172.21.15.28:0/277833666 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fde9c1032a0 0x7fde9c19ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:59.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:59.712+0000 7fdea3000640 1 --2- 172.21.15.28:0/277833666 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fde9c1028e0 0x7fde9c197ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:59.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:59.712+0000 7fdea3000640 1 --2- 172.21.15.28:0/277833666 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde9c1016f0 0x7fde9c1979a0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:59.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:59.712+0000 7fdea3000640 1 -- 172.21.15.28:0/277833666 >> 172.21.15.28:0/277833666 conn(0x7fde9c0fcc60 msgr2=0x7fde9c105be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:08:59.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:59.712+0000 7fdea3000640 1 -- 172.21.15.28:0/277833666 shutdown_connections 2024-05-12T22:08:59.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:59.712+0000 7fdea3000640 1 -- 172.21.15.28:0/277833666 wait complete. 2024-05-12T22:08:59.730 INFO:teuthology.run_tasks:Running task check-counter... 2024-05-12T22:08:59.743 INFO:teuthology.run_tasks:Running task fwd_scrub... 2024-05-12T22:08:59.753 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-12T22:08:59.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:59.930+0000 7f0498400640 1 -- 172.21.15.28:0/1143425260 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0490102310 msgr2=0x7f0490104700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:59.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:59.930+0000 7f0498400640 1 --2- 172.21.15.28:0/1143425260 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0490102310 0x7f0490104700 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f0484009900 tx=0x7f048402a3f0 comp rx=0 tx=0).stop 2024-05-12T22:08:59.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:59.931+0000 7f0498400640 1 -- 172.21.15.28:0/1143425260 shutdown_connections 2024-05-12T22:08:59.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:59.931+0000 7f0498400640 1 --2- 172.21.15.28:0/1143425260 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0490107570 0x7f04901099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:59.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:59.931+0000 7f0498400640 1 --2- 172.21.15.28:0/1143425260 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0490104c40 0x7f0490107030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:59.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:59.931+0000 7f0498400640 1 --2- 172.21.15.28:0/1143425260 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0490102310 0x7f0490104700 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:59.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:59.931+0000 7f0498400640 1 -- 172.21.15.28:0/1143425260 >> 172.21.15.28:0/1143425260 conn(0x7f04900fbd20 msgr2=0x7f04900fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:08:59.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:59.931+0000 7f0498400640 1 -- 172.21.15.28:0/1143425260 shutdown_connections 2024-05-12T22:08:59.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:59.931+0000 7f0498400640 1 -- 172.21.15.28:0/1143425260 wait complete. 2024-05-12T22:08:59.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:59.932+0000 7f0498400640 1 Processor -- start 2024-05-12T22:08:59.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:59.932+0000 7f0498400640 1 -- start start 2024-05-12T22:08:59.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:59.932+0000 7f0498400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0490102310 0x7f0490199e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:59.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:59.932+0000 7f0498400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0490104c40 0x7f049019a350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:59.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:59.932+0000 7f0498400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0490107570 0x7f04901a13c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:59.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:59.932+0000 7f0498400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f049010d560 con 0x7f0490104c40 2024-05-12T22:08:59.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:59.932+0000 7f0498400640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f049010d3e0 con 0x7f0490102310 2024-05-12T22:08:59.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:59.932+0000 7f0498400640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f049010d6e0 con 0x7f0490107570 2024-05-12T22:08:59.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:59.933+0000 7f0495a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0490102310 0x7f0490199e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:59.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:59.933+0000 7f0495a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0490102310 0x7f0490199e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:49238/0 (socket says 172.21.15.28:49238) 2024-05-12T22:08:59.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:59.933+0000 7f0495a00640 1 -- 172.21.15.28:0/3552281136 learned_addr learned my addr 172.21.15.28:0/3552281136 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:08:59.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:59.933+0000 7f0495000640 1 --2- 172.21.15.28:0/3552281136 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0490104c40 0x7f049019a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:59.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:59.933+0000 7f0496400640 1 --2- 172.21.15.28:0/3552281136 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0490107570 0x7f04901a13c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:59.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:59.933+0000 7f0495a00640 1 -- 172.21.15.28:0/3552281136 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0490107570 msgr2=0x7f04901a13c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:59.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:59.933+0000 7f0495a00640 1 --2- 172.21.15.28:0/3552281136 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0490107570 0x7f04901a13c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:59.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:59.933+0000 7f0495a00640 1 -- 172.21.15.28:0/3552281136 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0490104c40 msgr2=0x7f049019a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:08:59.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:59.933+0000 7f0495a00640 1 --2- 172.21.15.28:0/3552281136 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0490104c40 0x7f049019a350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:08:59.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:59.933+0000 7f0495a00640 1 -- 172.21.15.28:0/3552281136 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04901a1a30 con 0x7f0490102310 2024-05-12T22:08:59.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:59.933+0000 7f0496400640 1 --2- 172.21.15.28:0/3552281136 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0490107570 0x7f04901a13c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:08:59.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:59.933+0000 7f0495000640 1 --2- 172.21.15.28:0/3552281136 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0490104c40 0x7f049019a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:08:59.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:59.933+0000 7f0495a00640 1 --2- 172.21.15.28:0/3552281136 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0490102310 0x7f0490199e10 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f0484031440 tx=0x7f0484032e20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:08:59.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:59.934+0000 7f048e000640 1 -- 172.21.15.28:0/3552281136 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f048403f030 con 0x7f0490102310 2024-05-12T22:08:59.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:59.934+0000 7f048e000640 1 -- 172.21.15.28:0/3552281136 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0484003ce0 con 0x7f0490102310 2024-05-12T22:08:59.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:59.934+0000 7f048e000640 1 -- 172.21.15.28:0/3552281136 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0484035070 con 0x7f0490102310 2024-05-12T22:08:59.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:59.934+0000 7f0498400640 1 -- 172.21.15.28:0/3552281136 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04901a1cc0 con 0x7f0490102310 2024-05-12T22:08:59.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:59.934+0000 7f0498400640 1 -- 172.21.15.28:0/3552281136 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04901a2160 con 0x7f0490102310 2024-05-12T22:08:59.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:59.935+0000 7f0498400640 1 -- 172.21.15.28:0/3552281136 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04901090c0 con 0x7f0490102310 2024-05-12T22:08:59.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:59.937+0000 7f048e000640 1 -- 172.21.15.28:0/3552281136 <== mon.2 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 23) ==== 137645+0+0 (secure 0 0 0) 0x7f0484042030 con 0x7f0490102310 2024-05-12T22:08:59.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:59.938+0000 7f048e000640 1 --2- 172.21.15.28:0/3552281136 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f04780a2a10 0x7f04780a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:08:59.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:59.938+0000 7f048e000640 1 -- 172.21.15.28:0/3552281136 <== mon.2 v2:172.21.15.86:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f04840ef3d0 con 0x7f0490102310 2024-05-12T22:08:59.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:59.938+0000 7f0495000640 1 --2- 172.21.15.28:0/3552281136 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f04780a2a10 0x7f04780a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:08:59.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:59.939+0000 7f0495000640 1 --2- 172.21.15.28:0/3552281136 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f04780a2a10 0x7f04780a4ed0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f047c0058c0 tx=0x7f047c005850 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:08:59.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:08:59.942+0000 7f048e000640 1 -- 172.21.15.28:0/3552281136 <== mon.2 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f04840bc4f0 con 0x7f0490102310 2024-05-12T22:09:00.116 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 22:08:59 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T22:08:59.694+0000 7f3c84a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-12T22:09:00.116 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 22:08:59 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T22:08:59.694+0000 7f3c84a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-12T22:09:00.116 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 22:08:59 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T22:08:59.694+0000 7f3c84a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-12T22:09:00.116 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 22:08:59 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T22:08:59.694+0000 7f3c84a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-12T22:09:00.116 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 22:08:59 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T22:08:59.694+0000 7f3c84a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-12T22:09:00.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:00.152+0000 7f0498400640 1 -- 172.21.15.28:0/3552281136 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f04900624d0 con 0x7f0490102310 2024-05-12T22:09:00.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:00.154+0000 7f048e000640 1 -- 172.21.15.28:0/3552281136 <== mon.2 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+22972 (secure 0 0 0) 0x7f04840bbdf0 con 0x7f0490102310 2024-05-12T22:09:00.155 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-12T22:09:00.155 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":89,"fsid":"6b92e7b4-10aa-11ef-bc98-c7b262605968","created":"2024-05-12T21:57:37.276782+0000","modified":"2024-05-12T22:08:58.638578+0000","last_up_change":"2024-05-12T22:05:37.929562+0000","last_in_change":"2024-05-12T22:05:18.750768+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-12T22:01:32.292826+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-12T22:06:31.540263+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-12T22:06:32.906118+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6992ce0a-844b-40e3-8ed1-181c8474dccf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":906003882},{"type":"v1","addr":"172.21.15.28:6803","nonce":906003882}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":906003882},{"type":"v1","addr":"172.21.15.28:6805","nonce":906003882}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":906003882},{"type":"v1","addr":"172.21.15.28:6809","nonce":906003882}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":906003882},{"type":"v1","addr":"172.21.15.28:6807","nonce":906003882}]},"public_addr":"172.21.15.28:6803/906003882","cluster_addr":"172.21.15.28:6805/906003882","heartbeat_back_addr":"172.21.15.28:6809/906003882","heartbeat_front_addr":"172.21.15.28:6807/906003882","state":["exists","up"]},{"osd":1,"uuid":"ced20c96-c481-4d48-90d1-6b43228959ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":3354141819},{"type":"v1","addr":"172.21.15.86:6801","nonce":3354141819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":3354141819},{"type":"v1","addr":"172.21.15.86:6803","nonce":3354141819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":3354141819},{"type":"v1","addr":"172.21.15.86:6807","nonce":3354141819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":3354141819},{"type":"v1","addr":"172.21.15.86:6805","nonce":3354141819}]},"public_addr":"172.21.15.86:6801/3354141819","cluster_addr":"172.21.15.86:6803/3354141819","heartbeat_back_addr":"172.21.15.86:6807/3354141819","heartbeat_front_addr":"172.21.15.86:6805/3354141819","state":["exists","up"]},{"osd":2,"uuid":"54a2f221-fa65-4795-9f5e-e660cc7c0098","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":906459365},{"type":"v1","addr":"172.21.15.186:6801","nonce":906459365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":906459365},{"type":"v1","addr":"172.21.15.186:6803","nonce":906459365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":906459365},{"type":"v1","addr":"172.21.15.186:6807","nonce":906459365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":906459365},{"type":"v1","addr":"172.21.15.186:6805","nonce":906459365}]},"public_addr":"172.21.15.186:6801/906459365","cluster_addr":"172.21.15.186:6803/906459365","heartbeat_back_addr":"172.21.15.186:6807/906459365","heartbeat_front_addr":"172.21.15.186:6805/906459365","state":["exists","up"]},{"osd":3,"uuid":"86836737-ac9a-487b-93f2-7fcd3c406960","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":4182884436},{"type":"v1","addr":"172.21.15.28:6811","nonce":4182884436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":4182884436},{"type":"v1","addr":"172.21.15.28:6813","nonce":4182884436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":4182884436},{"type":"v1","addr":"172.21.15.28:6817","nonce":4182884436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":4182884436},{"type":"v1","addr":"172.21.15.28:6815","nonce":4182884436}]},"public_addr":"172.21.15.28:6811/4182884436","cluster_addr":"172.21.15.28:6813/4182884436","heartbeat_back_addr":"172.21.15.28:6817/4182884436","heartbeat_front_addr":"172.21.15.28:6815/4182884436","state":["exists","up"]},{"osd":4,"uuid":"d162ebcd-83c0-4fd5-9925-28b2cecea935","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2596998708},{"type":"v1","addr":"172.21.15.86:6809","nonce":2596998708}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2596998708},{"type":"v1","addr":"172.21.15.86:6811","nonce":2596998708}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2596998708},{"type":"v1","addr":"172.21.15.86:6815","nonce":2596998708}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2596998708},{"type":"v1","addr":"172.21.15.86:6813","nonce":2596998708}]},"public_addr":"172.21.15.86:6809/2596998708","cluster_addr":"172.21.15.86:6811/2596998708","heartbeat_back_addr":"172.21.15.86:6815/2596998708","heartbeat_front_addr":"172.21.15.86:6813/2596998708","state":["exists","up"]},{"osd":5,"uuid":"c794336a-a105-425c-b6c7-a32b90c05bb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":1944205756},{"type":"v1","addr":"172.21.15.186:6809","nonce":1944205756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":1944205756},{"type":"v1","addr":"172.21.15.186:6811","nonce":1944205756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":1944205756},{"type":"v1","addr":"172.21.15.186:6815","nonce":1944205756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":1944205756},{"type":"v1","addr":"172.21.15.186:6813","nonce":1944205756}]},"public_addr":"172.21.15.186:6809/1944205756","cluster_addr":"172.21.15.186:6811/1944205756","heartbeat_back_addr":"172.21.15.186:6815/1944205756","heartbeat_front_addr":"172.21.15.186:6813/1944205756","state":["exists","up"]},{"osd":6,"uuid":"516932ac-af00-45b7-ad3c-d6dec7db442f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":1642308925},{"type":"v1","addr":"172.21.15.28:6819","nonce":1642308925}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":1642308925},{"type":"v1","addr":"172.21.15.28:6821","nonce":1642308925}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1642308925},{"type":"v1","addr":"172.21.15.28:6825","nonce":1642308925}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":1642308925},{"type":"v1","addr":"172.21.15.28:6823","nonce":1642308925}]},"public_addr":"172.21.15.28:6819/1642308925","cluster_addr":"172.21.15.28:6821/1642308925","heartbeat_back_addr":"172.21.15.28:6825/1642308925","heartbeat_front_addr":"172.21.15.28:6823/1642308925","state":["exists","up"]},{"osd":7,"uuid":"ca7a059f-0414-438e-b2d1-a10f348036be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":726062711},{"type":"v1","addr":"172.21.15.86:6817","nonce":726062711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":726062711},{"type":"v1","addr":"172.21.15.86:6819","nonce":726062711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":726062711},{"type":"v1","addr":"172.21.15.86:6823","nonce":726062711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":726062711},{"type":"v1","addr":"172.21.15.86:6821","nonce":726062711}]},"public_addr":"172.21.15.86:6817/726062711","cluster_addr":"172.21.15.86:6819/726062711","heartbeat_back_addr":"172.21.15.86:6823/726062711","heartbeat_front_addr":"172.21.15.86:6821/726062711","state":["exists","up"]},{"osd":8,"uuid":"92e3f41f-aebd-422e-a99e-555f7ba4aa53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":3007687277},{"type":"v1","addr":"172.21.15.186:6817","nonce":3007687277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":3007687277},{"type":"v1","addr":"172.21.15.186:6819","nonce":3007687277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":3007687277},{"type":"v1","addr":"172.21.15.186:6823","nonce":3007687277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":3007687277},{"type":"v1","addr":"172.21.15.186:6821","nonce":3007687277}]},"public_addr":"172.21.15.186:6817/3007687277","cluster_addr":"172.21.15.186:6819/3007687277","heartbeat_back_addr":"172.21.15.186:6823/3007687277","heartbeat_front_addr":"172.21.15.186:6821/3007687277","state":["exists","up"]},{"osd":9,"uuid":"38ff7d9d-ee85-404c-8c59-adc327e1ccfa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":136220472},{"type":"v1","addr":"172.21.15.28:6827","nonce":136220472}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":136220472},{"type":"v1","addr":"172.21.15.28:6829","nonce":136220472}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":136220472},{"type":"v1","addr":"172.21.15.28:6833","nonce":136220472}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":136220472},{"type":"v1","addr":"172.21.15.28:6831","nonce":136220472}]},"public_addr":"172.21.15.28:6827/136220472","cluster_addr":"172.21.15.28:6829/136220472","heartbeat_back_addr":"172.21.15.28:6833/136220472","heartbeat_front_addr":"172.21.15.28:6831/136220472","state":["exists","up"]},{"osd":10,"uuid":"5e9ff657-afa8-4bab-8491-f209038392b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1189968310},{"type":"v1","addr":"172.21.15.86:6825","nonce":1189968310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":1189968310},{"type":"v1","addr":"172.21.15.86:6827","nonce":1189968310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":1189968310},{"type":"v1","addr":"172.21.15.86:6831","nonce":1189968310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":1189968310},{"type":"v1","addr":"172.21.15.86:6829","nonce":1189968310}]},"public_addr":"172.21.15.86:6825/1189968310","cluster_addr":"172.21.15.86:6827/1189968310","heartbeat_back_addr":"172.21.15.86:6831/1189968310","heartbeat_front_addr":"172.21.15.86:6829/1189968310","state":["exists","up"]},{"osd":11,"uuid":"f112e3c6-65b6-4253-a966-62001db7d069","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3801668110},{"type":"v1","addr":"172.21.15.186:6825","nonce":3801668110}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3801668110},{"type":"v1","addr":"172.21.15.186:6827","nonce":3801668110}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3801668110},{"type":"v1","addr":"172.21.15.186:6831","nonce":3801668110}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3801668110},{"type":"v1","addr":"172.21.15.186:6829","nonce":3801668110}]},"public_addr":"172.21.15.186:6825/3801668110","cluster_addr":"172.21.15.186:6827/3801668110","heartbeat_back_addr":"172.21.15.186:6831/3801668110","heartbeat_front_addr":"172.21.15.186:6829/3801668110","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:00:40.483623+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:01:05.284385+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:01:28.645716+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:02:01.295092+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:02:26.486601+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:02:50.807478+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:03:24.756169+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:03:49.975173+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:04:13.772233+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:04:46.758555+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:05:12.331685+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:05:36.462548+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":2,"to":1}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.25","mappings":[{"from":5,"to":9}]},{"pgid":"3.27","mappings":[{"from":4,"to":8}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/2580087728":"2024-05-13T22:08:58.638536+0000","172.21.15.28:6801/3123328651":"2024-05-13T22:08:58.638536+0000","172.21.15.28:0/820158394":"2024-05-13T22:08:58.638536+0000","172.21.15.28:0/1978033727":"2024-05-13T22:08:58.638536+0000","172.21.15.28:0/1177009533":"2024-05-13T21:58:45.550459+0000","172.21.15.28:6800/3760499540":"2024-05-13T21:58:45.550459+0000","172.21.15.28:6801/3783742728":"2024-05-13T21:58:10.421564+0000","172.21.15.28:6800/3783742728":"2024-05-13T21:58:10.421564+0000","172.21.15.28:0/2229156260":"2024-05-13T21:58:45.550459+0000","172.21.15.28:0/1202140963":"2024-05-13T21:58:10.421564+0000","172.21.15.28:6800/3123328651":"2024-05-13T22:08:58.638536+0000","172.21.15.28:0/2924223784":"2024-05-13T21:58:45.550459+0000","172.21.15.28:6801/3760499540":"2024-05-13T21:58:45.550459+0000","172.21.15.28:0/1240027206":"2024-05-13T21:58:10.421564+0000","172.21.15.28:0/1581755795":"2024-05-13T22:08:58.638536+0000","172.21.15.28:0/171417728":"2024-05-13T21:58:10.421564+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-12T22:09:00.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:00.157+0000 7f0498400640 1 -- 172.21.15.28:0/3552281136 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f04780a2a10 msgr2=0x7f04780a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:00.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:00.157+0000 7f0498400640 1 --2- 172.21.15.28:0/3552281136 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f04780a2a10 0x7f04780a4ed0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f047c0058c0 tx=0x7f047c005850 comp rx=0 tx=0).stop 2024-05-12T22:09:00.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:00.157+0000 7f0498400640 1 -- 172.21.15.28:0/3552281136 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0490102310 msgr2=0x7f0490199e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:00.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:00.157+0000 7f0498400640 1 --2- 172.21.15.28:0/3552281136 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0490102310 0x7f0490199e10 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f0484031440 tx=0x7f0484032e20 comp rx=0 tx=0).stop 2024-05-12T22:09:00.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:00.157+0000 7f0498400640 1 -- 172.21.15.28:0/3552281136 shutdown_connections 2024-05-12T22:09:00.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:00.157+0000 7f0498400640 1 --2- 172.21.15.28:0/3552281136 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f04780a2a10 0x7f04780a4ed0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:00.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:00.157+0000 7f0498400640 1 --2- 172.21.15.28:0/3552281136 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0490107570 0x7f04901a13c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:00.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:00.157+0000 7f0498400640 1 --2- 172.21.15.28:0/3552281136 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0490104c40 0x7f049019a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:00.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:00.157+0000 7f0498400640 1 --2- 172.21.15.28:0/3552281136 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0490102310 0x7f0490199e10 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:00.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:00.157+0000 7f0498400640 1 -- 172.21.15.28:0/3552281136 >> 172.21.15.28:0/3552281136 conn(0x7f04900fbd20 msgr2=0x7f04901019f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:09:00.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:00.157+0000 7f0498400640 1 -- 172.21.15.28:0/3552281136 shutdown_connections 2024-05-12T22:09:00.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:00.157+0000 7f0498400640 1 -- 172.21.15.28:0/3552281136 wait complete. 2024-05-12T22:09:00.176 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-12T22:01:32.292826+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-12T22:06:31.540263+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-12T22:06:32.906118+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-12T22:09:00.177 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-12T22:09:00.241 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:59 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/3119899590' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-05-12T22:09:00.241 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:59 smithi028 ceph-mon[29847]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=g=up:active} 3 up:standby-replay 6 up:standby 2024-05-12T22:09:00.242 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:59 smithi028 ceph-mon[29847]: [12/May/2024:22:08:59] ENGINE Bus STARTING 2024-05-12T22:09:00.242 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:59 smithi028 ceph-mon[29847]: [12/May/2024:22:08:59] ENGINE Serving on http://172.21.15.28:8765 2024-05-12T22:09:00.242 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:59 smithi028 ceph-mon[29847]: mgrmap e23: x(active, since 1.03391s), standbys: z, y 2024-05-12T22:09:00.242 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:59 smithi028 ceph-mon[29847]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:09:00.242 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:08:59 smithi028 ceph-mon[29847]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:09:00.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:59 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/3119899590' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-05-12T22:09:00.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:59 smithi086 ceph-mon[34297]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=g=up:active} 3 up:standby-replay 6 up:standby 2024-05-12T22:09:00.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:59 smithi086 ceph-mon[34297]: [12/May/2024:22:08:59] ENGINE Bus STARTING 2024-05-12T22:09:00.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:59 smithi086 ceph-mon[34297]: [12/May/2024:22:08:59] ENGINE Serving on http://172.21.15.28:8765 2024-05-12T22:09:00.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:59 smithi086 ceph-mon[34297]: mgrmap e23: x(active, since 1.03391s), standbys: z, y 2024-05-12T22:09:00.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:59 smithi086 ceph-mon[34297]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:09:00.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:08:59 smithi086 ceph-mon[34297]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:09:00.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:00.380+0000 7f38d5800640 1 -- 172.21.15.28:0/1235401044 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f38d0102300 msgr2=0x7f38d01046f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:00.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:00.380+0000 7f38d5800640 1 --2- 172.21.15.28:0/1235401044 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f38d0102300 0x7f38d01046f0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f38c0009900 tx=0x7f38c002a410 comp rx=0 tx=0).stop 2024-05-12T22:09:00.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:00.380+0000 7f38d5800640 1 -- 172.21.15.28:0/1235401044 shutdown_connections 2024-05-12T22:09:00.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:00.380+0000 7f38d5800640 1 --2- 172.21.15.28:0/1235401044 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f38d0107560 0x7f38d01099b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:00.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:00.380+0000 7f38d5800640 1 --2- 172.21.15.28:0/1235401044 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38d0104c30 0x7f38d0107020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:00.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:00.380+0000 7f38d5800640 1 --2- 172.21.15.28:0/1235401044 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f38d0102300 0x7f38d01046f0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:00.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:00.380+0000 7f38d5800640 1 -- 172.21.15.28:0/1235401044 >> 172.21.15.28:0/1235401044 conn(0x7f38d00fbcf0 msgr2=0x7f38d00fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:09:00.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:00.380+0000 7f38d5800640 1 -- 172.21.15.28:0/1235401044 shutdown_connections 2024-05-12T22:09:00.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:00.380+0000 7f38d5800640 1 -- 172.21.15.28:0/1235401044 wait complete. 2024-05-12T22:09:00.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:00.381+0000 7f38d5800640 1 Processor -- start 2024-05-12T22:09:00.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:00.381+0000 7f38d5800640 1 -- start start 2024-05-12T22:09:00.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:00.381+0000 7f38d5800640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f38d0102300 0x7f38d0197bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:00.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:00.381+0000 7f38d5800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f38d0104c30 0x7f38d0198100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:00.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:00.381+0000 7f38d5800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38d0107560 0x7f38d019f170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:00.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:00.381+0000 7f38d5800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f38d010d520 con 0x7f38d0107560 2024-05-12T22:09:00.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:00.381+0000 7f38d5800640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f38d010d3a0 con 0x7f38d0104c30 2024-05-12T22:09:00.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:00.381+0000 7f38d5800640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f38d010d6a0 con 0x7f38d0102300 2024-05-12T22:09:00.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:00.381+0000 7f38ce000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f38d0104c30 0x7f38d0198100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:00.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:00.381+0000 7f38cf400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38d0107560 0x7f38d019f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:00.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:00.381+0000 7f38ce000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f38d0104c30 0x7f38d0198100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:49260/0 (socket says 172.21.15.28:49260) 2024-05-12T22:09:00.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:00.381+0000 7f38ce000640 1 -- 172.21.15.28:0/2964579230 learned_addr learned my addr 172.21.15.28:0/2964579230 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:09:00.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:00.382+0000 7f38cea00640 1 --2- 172.21.15.28:0/2964579230 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f38d0102300 0x7f38d0197bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:00.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:00.382+0000 7f38ce000640 1 -- 172.21.15.28:0/2964579230 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f38d0102300 msgr2=0x7f38d0197bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:00.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:00.382+0000 7f38ce000640 1 --2- 172.21.15.28:0/2964579230 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f38d0102300 0x7f38d0197bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:00.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:00.382+0000 7f38ce000640 1 -- 172.21.15.28:0/2964579230 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38d0107560 msgr2=0x7f38d019f170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:00.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:00.382+0000 7f38ce000640 1 --2- 172.21.15.28:0/2964579230 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38d0107560 0x7f38d019f170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:00.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:00.382+0000 7f38ce000640 1 -- 172.21.15.28:0/2964579230 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38d019f7e0 con 0x7f38d0104c30 2024-05-12T22:09:00.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:00.382+0000 7f38cf400640 1 --2- 172.21.15.28:0/2964579230 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38d0107560 0x7f38d019f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:09:00.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:00.382+0000 7f38ce000640 1 --2- 172.21.15.28:0/2964579230 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f38d0104c30 0x7f38d0198100 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f38bc00e9c0 tx=0x7f38bc00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:09:00.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:00.382+0000 7f38c7400640 1 -- 172.21.15.28:0/2964579230 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38bc00cd80 con 0x7f38d0104c30 2024-05-12T22:09:00.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:00.382+0000 7f38c7400640 1 -- 172.21.15.28:0/2964579230 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f38bc005500 con 0x7f38d0104c30 2024-05-12T22:09:00.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:00.382+0000 7f38c7400640 1 -- 172.21.15.28:0/2964579230 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38bc010690 con 0x7f38d0104c30 2024-05-12T22:09:00.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:00.382+0000 7f38d5800640 1 -- 172.21.15.28:0/2964579230 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38d019fad0 con 0x7f38d0104c30 2024-05-12T22:09:00.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:00.383+0000 7f38d5800640 1 -- 172.21.15.28:0/2964579230 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38d01a0070 con 0x7f38d0104c30 2024-05-12T22:09:00.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:00.384+0000 7f38d5800640 1 -- 172.21.15.28:0/2964579230 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f38d00fd7e0 con 0x7f38d0104c30 2024-05-12T22:09:00.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:00.386+0000 7f38c7400640 1 -- 172.21.15.28:0/2964579230 <== mon.2 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 23) ==== 137645+0+0 (secure 0 0 0) 0x7f38bc0046c0 con 0x7f38d0104c30 2024-05-12T22:09:00.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:00.388+0000 7f38c7400640 1 --2- 172.21.15.28:0/2964579230 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f38b40a2a10 0x7f38b40a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:00.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:00.388+0000 7f38c7400640 1 -- 172.21.15.28:0/2964579230 <== mon.2 v2:172.21.15.86:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f38bc0cf070 con 0x7f38d0104c30 2024-05-12T22:09:00.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:00.388+0000 7f38cea00640 1 --2- 172.21.15.28:0/2964579230 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f38b40a2a10 0x7f38b40a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:00.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:00.389+0000 7f38cea00640 1 --2- 172.21.15.28:0/2964579230 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f38b40a2a10 0x7f38b40a4ed0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f38c00314e0 tx=0x7f38c0000bc0 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:09:00.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:00.392+0000 7f38c7400640 1 -- 172.21.15.28:0/2964579230 <== mon.2 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f38bc09c190 con 0x7f38d0104c30 2024-05-12T22:09:00.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:59 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/3119899590' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-05-12T22:09:00.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:59 smithi186 ceph-mon[34785]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=g=up:active} 3 up:standby-replay 6 up:standby 2024-05-12T22:09:00.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:59 smithi186 ceph-mon[34785]: [12/May/2024:22:08:59] ENGINE Bus STARTING 2024-05-12T22:09:00.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:59 smithi186 ceph-mon[34785]: [12/May/2024:22:08:59] ENGINE Serving on http://172.21.15.28:8765 2024-05-12T22:09:00.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:59 smithi186 ceph-mon[34785]: mgrmap e23: x(active, since 1.03391s), standbys: z, y 2024-05-12T22:09:00.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:59 smithi186 ceph-mon[34785]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:09:00.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:08:59 smithi186 ceph-mon[34785]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:09:00.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:00.595+0000 7f38d5800640 1 -- 172.21.15.28:0/2964579230 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f38d0062430 con 0x7f38d0104c30 2024-05-12T22:09:00.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:00.596+0000 7f38c7400640 1 -- 172.21.15.28:0/2964579230 <== mon.2 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) ==== 93+0+10 (secure 0 0 0) 0x7f38bc09ba90 con 0x7f38d0104c30 2024-05-12T22:09:00.596 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-05-12T22:09:00.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:00.598+0000 7f38d5800640 1 -- 172.21.15.28:0/2964579230 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f38b40a2a10 msgr2=0x7f38b40a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:00.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:00.598+0000 7f38d5800640 1 --2- 172.21.15.28:0/2964579230 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f38b40a2a10 0x7f38b40a4ed0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f38c00314e0 tx=0x7f38c0000bc0 comp rx=0 tx=0).stop 2024-05-12T22:09:00.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:00.598+0000 7f38d5800640 1 -- 172.21.15.28:0/2964579230 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f38d0104c30 msgr2=0x7f38d0198100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:00.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:00.598+0000 7f38d5800640 1 --2- 172.21.15.28:0/2964579230 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f38d0104c30 0x7f38d0198100 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f38bc00e9c0 tx=0x7f38bc00ee80 comp rx=0 tx=0).stop 2024-05-12T22:09:00.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:00.598+0000 7f38d5800640 1 -- 172.21.15.28:0/2964579230 shutdown_connections 2024-05-12T22:09:00.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:00.599+0000 7f38d5800640 1 --2- 172.21.15.28:0/2964579230 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f38b40a2a10 0x7f38b40a4ed0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:00.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:00.599+0000 7f38d5800640 1 --2- 172.21.15.28:0/2964579230 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38d0107560 0x7f38d019f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:00.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:00.599+0000 7f38d5800640 1 --2- 172.21.15.28:0/2964579230 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f38d0104c30 0x7f38d0198100 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:00.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:00.599+0000 7f38d5800640 1 --2- 172.21.15.28:0/2964579230 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f38d0102300 0x7f38d0197bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:00.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:00.599+0000 7f38d5800640 1 -- 172.21.15.28:0/2964579230 >> 172.21.15.28:0/2964579230 conn(0x7f38d00fbcf0 msgr2=0x7f38d0102da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:09:00.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:00.599+0000 7f38d5800640 1 -- 172.21.15.28:0/2964579230 shutdown_connections 2024-05-12T22:09:00.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:00.599+0000 7f38d5800640 1 -- 172.21.15.28:0/2964579230 wait complete. 2024-05-12T22:09:00.623 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-12T22:09:00.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:00.840+0000 7f54f9600640 1 -- 172.21.15.28:0/92310152 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54f4104ce0 msgr2=0x7f54f41070d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:00.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:00.840+0000 7f54f9600640 1 --2- 172.21.15.28:0/92310152 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54f4104ce0 0x7f54f41070d0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f54e0009900 tx=0x7f54e002a410 comp rx=0 tx=0).stop 2024-05-12T22:09:00.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:00.841+0000 7f54f9600640 1 -- 172.21.15.28:0/92310152 shutdown_connections 2024-05-12T22:09:00.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:00.841+0000 7f54f9600640 1 --2- 172.21.15.28:0/92310152 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f54f4107610 0x7f54f4109a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:00.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:00.841+0000 7f54f9600640 1 --2- 172.21.15.28:0/92310152 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54f4104ce0 0x7f54f41070d0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:00.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:00.841+0000 7f54f9600640 1 --2- 172.21.15.28:0/92310152 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f54f41023b0 0x7f54f41047a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:00.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:00.841+0000 7f54f9600640 1 -- 172.21.15.28:0/92310152 >> 172.21.15.28:0/92310152 conn(0x7f54f40fbd20 msgr2=0x7f54f40fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:09:00.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:00.841+0000 7f54f9600640 1 -- 172.21.15.28:0/92310152 shutdown_connections 2024-05-12T22:09:00.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:00.841+0000 7f54f9600640 1 -- 172.21.15.28:0/92310152 wait complete. 2024-05-12T22:09:00.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:00.842+0000 7f54f9600640 1 Processor -- start 2024-05-12T22:09:00.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:00.842+0000 7f54f9600640 1 -- start start 2024-05-12T22:09:00.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:00.842+0000 7f54f9600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f54f41023b0 0x7f54f4101d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:00.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:00.842+0000 7f54f9600640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f54f4104ce0 0x7f54f4100360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:00.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:00.842+0000 7f54f9600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54f4107610 0x7f54f41008a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:00.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:00.842+0000 7f54f9600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f54f410d510 con 0x7f54f4107610 2024-05-12T22:09:00.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:00.842+0000 7f54f9600640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f54f410d390 con 0x7f54f41023b0 2024-05-12T22:09:00.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:00.842+0000 7f54f9600640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f54f410d690 con 0x7f54f4104ce0 2024-05-12T22:09:00.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:00.842+0000 7f54f3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54f4107610 0x7f54f41008a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:00.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:00.842+0000 7f54f2a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f54f41023b0 0x7f54f4101d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:00.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:00.843+0000 7f54f2a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f54f41023b0 0x7f54f4101d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:49292/0 (socket says 172.21.15.28:49292) 2024-05-12T22:09:00.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:00.843+0000 7f54f2a00640 1 -- 172.21.15.28:0/833711502 learned_addr learned my addr 172.21.15.28:0/833711502 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:09:00.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:00.843+0000 7f54f3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54f4107610 0x7f54f41008a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51216/0 (socket says 172.21.15.28:51216) 2024-05-12T22:09:00.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:00.843+0000 7f54f2000640 1 --2- 172.21.15.28:0/833711502 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f54f4104ce0 0x7f54f4100360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:00.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:00.843+0000 7f54f2a00640 1 -- 172.21.15.28:0/833711502 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f54f4104ce0 msgr2=0x7f54f4100360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:00.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:00.843+0000 7f54f2a00640 1 --2- 172.21.15.28:0/833711502 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f54f4104ce0 0x7f54f4100360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:00.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:00.843+0000 7f54f2a00640 1 -- 172.21.15.28:0/833711502 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54f4107610 msgr2=0x7f54f41008a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:00.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:00.843+0000 7f54f2a00640 1 --2- 172.21.15.28:0/833711502 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54f4107610 0x7f54f41008a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:00.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:00.843+0000 7f54f2a00640 1 -- 172.21.15.28:0/833711502 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54f4101130 con 0x7f54f41023b0 2024-05-12T22:09:00.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:00.843+0000 7f54f2a00640 1 --2- 172.21.15.28:0/833711502 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f54f41023b0 0x7f54f4101d20 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f54e400ea60 tx=0x7f54e400ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:09:00.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:00.844+0000 7f54eb400640 1 -- 172.21.15.28:0/833711502 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54e400cde0 con 0x7f54f41023b0 2024-05-12T22:09:00.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:00.844+0000 7f54eb400640 1 -- 172.21.15.28:0/833711502 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f54e40035f0 con 0x7f54f41023b0 2024-05-12T22:09:00.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:00.844+0000 7f54eb400640 1 -- 172.21.15.28:0/833711502 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54e4010670 con 0x7f54f41023b0 2024-05-12T22:09:00.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:00.844+0000 7f54f9600640 1 -- 172.21.15.28:0/833711502 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54f41a1d30 con 0x7f54f41023b0 2024-05-12T22:09:00.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:00.844+0000 7f54f9600640 1 -- 172.21.15.28:0/833711502 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54f41a21a0 con 0x7f54f41023b0 2024-05-12T22:09:00.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:00.845+0000 7f54f9600640 1 -- 172.21.15.28:0/833711502 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f54f40fd810 con 0x7f54f41023b0 2024-05-12T22:09:00.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:00.847+0000 7f54eb400640 1 -- 172.21.15.28:0/833711502 <== mon.2 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 23) ==== 137645+0+0 (secure 0 0 0) 0x7f54e40037a0 con 0x7f54f41023b0 2024-05-12T22:09:00.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:00.848+0000 7f54eb400640 1 --2- 172.21.15.28:0/833711502 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f54d80a2990 0x7f54d80a4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:00.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:00.849+0000 7f54eb400640 1 -- 172.21.15.28:0/833711502 <== mon.2 v2:172.21.15.86:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f54e40cf0c0 con 0x7f54f41023b0 2024-05-12T22:09:00.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:00.849+0000 7f54f2000640 1 --2- 172.21.15.28:0/833711502 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f54d80a2990 0x7f54d80a4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:00.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:00.850+0000 7f54f2000640 1 --2- 172.21.15.28:0/833711502 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f54d80a2990 0x7f54d80a4e50 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f54e0005ab0 tx=0x7f54e0000bc0 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:09:00.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:00.852+0000 7f54eb400640 1 -- 172.21.15.28:0/833711502 <== mon.2 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f54e409c1e0 con 0x7f54f41023b0 2024-05-12T22:09:01.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:01.002+0000 7f54eb400640 1 -- 172.21.15.28:0/833711502 <== mon.2 v2:172.21.15.86:3300/0 7 ==== mgrmap(e 24) ==== 138004+0+0 (secure 0 0 0) 0x7f54e4098150 con 0x7f54f41023b0 2024-05-12T22:09:01.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:01.090+0000 7f54f9600640 1 -- 172.21.15.28:0/833711502 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f54f4062430 con 0x7f54f41023b0 2024-05-12T22:09:01.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:01.091+0000 7f54eb400640 1 -- 172.21.15.28:0/833711502 <== mon.2 v2:172.21.15.86:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) ==== 104+0+11 (secure 0 0 0) 0x7f54e4064750 con 0x7f54f41023b0 2024-05-12T22:09:01.092 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-05-12T22:09:01.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:01.094+0000 7f54f9600640 1 -- 172.21.15.28:0/833711502 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f54d80a2990 msgr2=0x7f54d80a4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:01.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:01.094+0000 7f54f9600640 1 --2- 172.21.15.28:0/833711502 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f54d80a2990 0x7f54d80a4e50 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f54e0005ab0 tx=0x7f54e0000bc0 comp rx=0 tx=0).stop 2024-05-12T22:09:01.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:01.094+0000 7f54f9600640 1 -- 172.21.15.28:0/833711502 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f54f41023b0 msgr2=0x7f54f4101d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:01.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:01.094+0000 7f54f9600640 1 --2- 172.21.15.28:0/833711502 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f54f41023b0 0x7f54f4101d20 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f54e400ea60 tx=0x7f54e400ef20 comp rx=0 tx=0).stop 2024-05-12T22:09:01.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:01.094+0000 7f54f9600640 1 -- 172.21.15.28:0/833711502 shutdown_connections 2024-05-12T22:09:01.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:01.094+0000 7f54f9600640 1 --2- 172.21.15.28:0/833711502 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f54d80a2990 0x7f54d80a4e50 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:01.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:01.094+0000 7f54f9600640 1 --2- 172.21.15.28:0/833711502 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54f4107610 0x7f54f41008a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:01.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:01.094+0000 7f54f9600640 1 --2- 172.21.15.28:0/833711502 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f54f4104ce0 0x7f54f4100360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:01.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:01.094+0000 7f54f9600640 1 --2- 172.21.15.28:0/833711502 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f54f41023b0 0x7f54f4101d20 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:01.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:01.094+0000 7f54f9600640 1 -- 172.21.15.28:0/833711502 >> 172.21.15.28:0/833711502 conn(0x7f54f40fbd20 msgr2=0x7f54f4102e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:09:01.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:01.094+0000 7f54f9600640 1 -- 172.21.15.28:0/833711502 shutdown_connections 2024-05-12T22:09:01.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:01.094+0000 7f54f9600640 1 -- 172.21.15.28:0/833711502 wait complete. 2024-05-12T22:09:01.113 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-12T22:09:01.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:01 smithi028 ceph-mon[29847]: [12/May/2024:22:08:59] ENGINE Serving on https://172.21.15.28:7150 2024-05-12T22:09:01.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:01 smithi028 ceph-mon[29847]: [12/May/2024:22:08:59] ENGINE Bus STARTED 2024-05-12T22:09:01.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:01 smithi028 ceph-mon[29847]: from='client.15159 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-05-12T22:09:01.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:01 smithi028 ceph-mon[29847]: pgmap v3: 97 pgs: 97 active+clean; 1.1 MiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-12T22:09:01.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:01 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/3552281136' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-12T22:09:01.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:01 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/2964579230' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-12T22:09:01.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:01 smithi028 ceph-mon[29847]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:09:01.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:01 smithi028 ceph-mon[29847]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:09:01.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:01 smithi028 ceph-mon[29847]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:09:01.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:01 smithi028 ceph-mon[29847]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:09:01.330 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:01 smithi186 ceph-mon[34785]: [12/May/2024:22:08:59] ENGINE Serving on https://172.21.15.28:7150 2024-05-12T22:09:01.330 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:01 smithi186 ceph-mon[34785]: [12/May/2024:22:08:59] ENGINE Bus STARTED 2024-05-12T22:09:01.330 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:01 smithi186 ceph-mon[34785]: from='client.15159 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-05-12T22:09:01.330 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:01 smithi186 ceph-mon[34785]: pgmap v3: 97 pgs: 97 active+clean; 1.1 MiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-12T22:09:01.330 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:01 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/3552281136' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-12T22:09:01.330 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:01 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/2964579230' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-12T22:09:01.330 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:01 smithi186 ceph-mon[34785]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:09:01.331 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:01 smithi186 ceph-mon[34785]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:09:01.331 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:01 smithi186 ceph-mon[34785]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:09:01.331 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:01 smithi186 ceph-mon[34785]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:09:01.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:01.337+0000 7fa896600640 1 -- 172.21.15.28:0/1446831264 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa890101880 msgr2=0x7fa890101ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:01.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:01.337+0000 7fa896600640 1 --2- 172.21.15.28:0/1446831264 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa890101880 0x7fa890101ca0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fa880009900 tx=0x7fa88002a410 comp rx=0 tx=0).stop 2024-05-12T22:09:01.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:01.337+0000 7fa896600640 1 -- 172.21.15.28:0/1446831264 shutdown_connections 2024-05-12T22:09:01.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:01.338+0000 7fa896600640 1 --2- 172.21.15.28:0/1446831264 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa890103430 0x7fa890109cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:01.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:01.338+0000 7fa896600640 1 --2- 172.21.15.28:0/1446831264 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa890102a70 0x7fa890102ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:01.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:01.338+0000 7fa896600640 1 --2- 172.21.15.28:0/1446831264 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa890101880 0x7fa890101ca0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:01.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:01.338+0000 7fa896600640 1 -- 172.21.15.28:0/1446831264 >> 172.21.15.28:0/1446831264 conn(0x7fa8900fce10 msgr2=0x7fa8900ff230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:09:01.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:01.338+0000 7fa896600640 1 -- 172.21.15.28:0/1446831264 shutdown_connections 2024-05-12T22:09:01.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:01.338+0000 7fa896600640 1 -- 172.21.15.28:0/1446831264 wait complete. 2024-05-12T22:09:01.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:01.339+0000 7fa896600640 1 Processor -- start 2024-05-12T22:09:01.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:01.340+0000 7fa896600640 1 -- start start 2024-05-12T22:09:01.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:01.340+0000 7fa896600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa890101880 0x7fa890075590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:01.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:01.340+0000 7fa896600640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa890102a70 0x7fa890073c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:01.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:01.340+0000 7fa896600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa890103430 0x7fa890074160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:01.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:01.340+0000 7fa896600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa89010d540 con 0x7fa890103430 2024-05-12T22:09:01.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:01.340+0000 7fa896600640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fa89010d3c0 con 0x7fa890101880 2024-05-12T22:09:01.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:01.340+0000 7fa896600640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa89010d6c0 con 0x7fa890102a70 2024-05-12T22:09:01.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:01.340+0000 7fa88f400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa890101880 0x7fa890075590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:01.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:01.341+0000 7fa88f400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa890101880 0x7fa890075590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:49314/0 (socket says 172.21.15.28:49314) 2024-05-12T22:09:01.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:01.341+0000 7fa88ea00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa890102a70 0x7fa890073c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:01.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:01.341+0000 7fa88f400640 1 -- 172.21.15.28:0/1985424170 learned_addr learned my addr 172.21.15.28:0/1985424170 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:09:01.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:01.341+0000 7fa88fe00640 1 --2- 172.21.15.28:0/1985424170 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa890103430 0x7fa890074160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:01.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:01.341+0000 7fa88f400640 1 -- 172.21.15.28:0/1985424170 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa890102a70 msgr2=0x7fa890073c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:01.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:01.341+0000 7fa88f400640 1 --2- 172.21.15.28:0/1985424170 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa890102a70 0x7fa890073c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:01.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:01.341+0000 7fa88f400640 1 -- 172.21.15.28:0/1985424170 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa890103430 msgr2=0x7fa890074160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:01.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:01.341+0000 7fa88f400640 1 --2- 172.21.15.28:0/1985424170 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa890103430 0x7fa890074160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:01.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:01.341+0000 7fa88f400640 1 -- 172.21.15.28:0/1985424170 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa890074a20 con 0x7fa890101880 2024-05-12T22:09:01.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:01.341+0000 7fa88fe00640 1 --2- 172.21.15.28:0/1985424170 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa890103430 0x7fa890074160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-12T22:09:01.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:01.342+0000 7fa88f400640 1 --2- 172.21.15.28:0/1985424170 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa890101880 0x7fa890075590 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fa880031440 tx=0x7fa880032ce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:09:01.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:01.342+0000 7fa887e00640 1 -- 172.21.15.28:0/1985424170 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa88003f030 con 0x7fa890101880 2024-05-12T22:09:01.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:01.342+0000 7fa887e00640 1 -- 172.21.15.28:0/1985424170 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa880003ce0 con 0x7fa890101880 2024-05-12T22:09:01.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:01.342+0000 7fa896600640 1 -- 172.21.15.28:0/1985424170 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8901a1cb0 con 0x7fa890101880 2024-05-12T22:09:01.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:01.342+0000 7fa887e00640 1 -- 172.21.15.28:0/1985424170 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa880035070 con 0x7fa890101880 2024-05-12T22:09:01.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:01.342+0000 7fa896600640 1 -- 172.21.15.28:0/1985424170 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8901a2170 con 0x7fa890101880 2024-05-12T22:09:01.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:01.344+0000 7fa896600640 1 -- 172.21.15.28:0/1985424170 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa890108c20 con 0x7fa890101880 2024-05-12T22:09:01.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:01.345+0000 7fa887e00640 1 -- 172.21.15.28:0/1985424170 <== mon.2 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 24) ==== 138004+0+0 (secure 0 0 0) 0x7fa880042030 con 0x7fa890101880 2024-05-12T22:09:01.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:01.347+0000 7fa887e00640 1 --2- 172.21.15.28:0/1985424170 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fa8740a2d90 0x7fa8740a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:01.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:01.347+0000 7fa887e00640 1 -- 172.21.15.28:0/1985424170 <== mon.2 v2:172.21.15.86:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7fa8800efc30 con 0x7fa890101880 2024-05-12T22:09:01.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:01.347+0000 7fa88ea00640 1 --2- 172.21.15.28:0/1985424170 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fa8740a2d90 0x7fa8740a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:01.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:01.348+0000 7fa88ea00640 1 --2- 172.21.15.28:0/1985424170 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fa8740a2d90 0x7fa8740a5250 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fa890075300 tx=0x7fa87c005f90 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:09:01.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:01.351+0000 7fa887e00640 1 -- 172.21.15.28:0/1985424170 <== mon.2 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fa8800bc6b0 con 0x7fa890101880 2024-05-12T22:09:01.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:01 smithi086 ceph-mon[34297]: [12/May/2024:22:08:59] ENGINE Serving on https://172.21.15.28:7150 2024-05-12T22:09:01.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:01 smithi086 ceph-mon[34297]: [12/May/2024:22:08:59] ENGINE Bus STARTED 2024-05-12T22:09:01.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:01 smithi086 ceph-mon[34297]: from='client.15159 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-05-12T22:09:01.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:01 smithi086 ceph-mon[34297]: pgmap v3: 97 pgs: 97 active+clean; 1.1 MiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-12T22:09:01.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:01 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/3552281136' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-12T22:09:01.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:01 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/2964579230' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-12T22:09:01.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:01 smithi086 ceph-mon[34297]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:09:01.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:01 smithi086 ceph-mon[34297]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:09:01.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:01 smithi086 ceph-mon[34297]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:09:01.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:01 smithi086 ceph-mon[34297]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:09:01.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:01.561+0000 7fa896600640 1 -- 172.21.15.28:0/1985424170 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fa89002d410 con 0x7fa890101880 2024-05-12T22:09:01.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:01.562+0000 7fa887e00640 1 -- 172.21.15.28:0/1985424170 <== mon.2 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) ==== 100+0+11 (secure 0 0 0) 0x7fa8800bbfb0 con 0x7fa890101880 2024-05-12T22:09:01.563 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-05-12T22:09:01.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:01.567+0000 7fa896600640 1 -- 172.21.15.28:0/1985424170 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fa8740a2d90 msgr2=0x7fa8740a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:01.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:01.567+0000 7fa896600640 1 --2- 172.21.15.28:0/1985424170 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fa8740a2d90 0x7fa8740a5250 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fa890075300 tx=0x7fa87c005f90 comp rx=0 tx=0).stop 2024-05-12T22:09:01.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:01.567+0000 7fa896600640 1 -- 172.21.15.28:0/1985424170 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa890101880 msgr2=0x7fa890075590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:01.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:01.567+0000 7fa896600640 1 --2- 172.21.15.28:0/1985424170 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa890101880 0x7fa890075590 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fa880031440 tx=0x7fa880032ce0 comp rx=0 tx=0).stop 2024-05-12T22:09:01.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:01.567+0000 7fa896600640 1 -- 172.21.15.28:0/1985424170 shutdown_connections 2024-05-12T22:09:01.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:01.567+0000 7fa896600640 1 --2- 172.21.15.28:0/1985424170 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fa8740a2d90 0x7fa8740a5250 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:01.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:01.567+0000 7fa896600640 1 --2- 172.21.15.28:0/1985424170 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa890103430 0x7fa890074160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:01.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:01.567+0000 7fa896600640 1 --2- 172.21.15.28:0/1985424170 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa890102a70 0x7fa890073c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:01.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:01.567+0000 7fa896600640 1 --2- 172.21.15.28:0/1985424170 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa890101880 0x7fa890075590 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:01.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:01.567+0000 7fa896600640 1 -- 172.21.15.28:0/1985424170 >> 172.21.15.28:0/1985424170 conn(0x7fa8900fce10 msgr2=0x7fa890105df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:09:01.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:01.567+0000 7fa896600640 1 -- 172.21.15.28:0/1985424170 shutdown_connections 2024-05-12T22:09:01.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:01.567+0000 7fa896600640 1 -- 172.21.15.28:0/1985424170 wait complete. 2024-05-12T22:09:01.586 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-12T22:09:01.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:01.782+0000 7fc864c00640 1 -- 172.21.15.28:0/4288353820 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc860107cb0 msgr2=0x7fc8601080d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:01.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:01.782+0000 7fc864c00640 1 --2- 172.21.15.28:0/4288353820 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc860107cb0 0x7fc8601080d0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fc850009900 tx=0x7fc85002a3f0 comp rx=0 tx=0).stop 2024-05-12T22:09:01.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:01.782+0000 7fc864c00640 1 -- 172.21.15.28:0/4288353820 shutdown_connections 2024-05-12T22:09:01.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:01.782+0000 7fc864c00640 1 --2- 172.21.15.28:0/4288353820 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc860109860 0x7fc8601100f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:01.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:01.782+0000 7fc864c00640 1 --2- 172.21.15.28:0/4288353820 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc860108ea0 0x7fc860109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:01.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:01.782+0000 7fc864c00640 1 --2- 172.21.15.28:0/4288353820 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc860107cb0 0x7fc8601080d0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:01.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:01.782+0000 7fc864c00640 1 -- 172.21.15.28:0/4288353820 >> 172.21.15.28:0/4288353820 conn(0x7fc860076c70 msgr2=0x7fc860079090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:09:01.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:01.782+0000 7fc864c00640 1 -- 172.21.15.28:0/4288353820 shutdown_connections 2024-05-12T22:09:01.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:01.782+0000 7fc864c00640 1 -- 172.21.15.28:0/4288353820 wait complete. 2024-05-12T22:09:01.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:01.783+0000 7fc864c00640 1 Processor -- start 2024-05-12T22:09:01.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:01.783+0000 7fc864c00640 1 -- start start 2024-05-12T22:09:01.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:01.784+0000 7fc864c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc860107cb0 0x7fc86019e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:01.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:01.784+0000 7fc864c00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc860108ea0 0x7fc86019e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:01.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:01.784+0000 7fc864c00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc860109860 0x7fc8601a55c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:01.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:01.784+0000 7fc864c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc860113900 con 0x7fc860107cb0 2024-05-12T22:09:01.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:01.784+0000 7fc864c00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fc860113780 con 0x7fc860108ea0 2024-05-12T22:09:01.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:01.784+0000 7fc864c00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc860113a80 con 0x7fc860109860 2024-05-12T22:09:01.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:01.784+0000 7fc85ea00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc860108ea0 0x7fc86019e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:01.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:01.784+0000 7fc85ea00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc860108ea0 0x7fc86019e550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:49326/0 (socket says 172.21.15.28:49326) 2024-05-12T22:09:01.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:01.784+0000 7fc85fe00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc860109860 0x7fc8601a55c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:01.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:01.784+0000 7fc85ea00640 1 -- 172.21.15.28:0/1280445378 learned_addr learned my addr 172.21.15.28:0/1280445378 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:09:01.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:01.784+0000 7fc85f400640 1 --2- 172.21.15.28:0/1280445378 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc860107cb0 0x7fc86019e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:01.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:01.784+0000 7fc85ea00640 1 -- 172.21.15.28:0/1280445378 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc860109860 msgr2=0x7fc8601a55c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:01.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:01.784+0000 7fc85ea00640 1 --2- 172.21.15.28:0/1280445378 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc860109860 0x7fc8601a55c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:01.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:01.784+0000 7fc85ea00640 1 -- 172.21.15.28:0/1280445378 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc860107cb0 msgr2=0x7fc86019e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:01.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:01.784+0000 7fc85ea00640 1 --2- 172.21.15.28:0/1280445378 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc860107cb0 0x7fc86019e010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:01.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:01.784+0000 7fc85ea00640 1 -- 172.21.15.28:0/1280445378 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8601a5c30 con 0x7fc860108ea0 2024-05-12T22:09:01.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:01.785+0000 7fc85f400640 1 --2- 172.21.15.28:0/1280445378 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc860107cb0 0x7fc86019e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:09:01.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:01.785+0000 7fc85ea00640 1 --2- 172.21.15.28:0/1280445378 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc860108ea0 0x7fc86019e550 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fc84c00e9f0 tx=0x7fc84c00edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:09:01.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:01.785+0000 7fc857e00640 1 -- 172.21.15.28:0/1280445378 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc84c00ccc0 con 0x7fc860108ea0 2024-05-12T22:09:01.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:01.785+0000 7fc857e00640 1 -- 172.21.15.28:0/1280445378 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc84c010910 con 0x7fc860108ea0 2024-05-12T22:09:01.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:01.785+0000 7fc857e00640 1 -- 172.21.15.28:0/1280445378 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc84c0188a0 con 0x7fc860108ea0 2024-05-12T22:09:01.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:01.785+0000 7fc864c00640 1 -- 172.21.15.28:0/1280445378 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8601a5f20 con 0x7fc860108ea0 2024-05-12T22:09:01.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:01.785+0000 7fc864c00640 1 -- 172.21.15.28:0/1280445378 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8601a6390 con 0x7fc860108ea0 2024-05-12T22:09:01.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:01.786+0000 7fc864c00640 1 -- 172.21.15.28:0/1280445378 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc86010f0c0 con 0x7fc860108ea0 2024-05-12T22:09:01.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:01.789+0000 7fc857e00640 1 -- 172.21.15.28:0/1280445378 <== mon.2 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 24) ==== 138004+0+0 (secure 0 0 0) 0x7fc84c010ac0 con 0x7fc860108ea0 2024-05-12T22:09:01.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:01.790+0000 7fc857e00640 1 --2- 172.21.15.28:0/1280445378 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fc8440a2e10 0x7fc8440a52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:01.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:01.791+0000 7fc857e00640 1 -- 172.21.15.28:0/1280445378 <== mon.2 v2:172.21.15.86:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7fc84c0cfa60 con 0x7fc860108ea0 2024-05-12T22:09:01.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:01.791+0000 7fc85f400640 1 --2- 172.21.15.28:0/1280445378 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fc8440a2e10 0x7fc8440a52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:01.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:01.791+0000 7fc85f400640 1 --2- 172.21.15.28:0/1280445378 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fc8440a2e10 0x7fc8440a52d0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fc850006010 tx=0x7fc850032cf0 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:09:01.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:01.794+0000 7fc857e00640 1 -- 172.21.15.28:0/1280445378 <== mon.2 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fc84c09c4e0 con 0x7fc860108ea0 2024-05-12T22:09:02.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:02.023+0000 7fc864c00640 1 -- 172.21.15.28:0/1280445378 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc86002d410 con 0x7fc860108ea0 2024-05-12T22:09:02.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:02.025+0000 7fc857e00640 1 -- 172.21.15.28:0/1280445378 <== mon.2 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+22972 (secure 0 0 0) 0x7fc84c09bde0 con 0x7fc860108ea0 2024-05-12T22:09:02.026 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-12T22:09:02.026 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":89,"fsid":"6b92e7b4-10aa-11ef-bc98-c7b262605968","created":"2024-05-12T21:57:37.276782+0000","modified":"2024-05-12T22:08:58.638578+0000","last_up_change":"2024-05-12T22:05:37.929562+0000","last_in_change":"2024-05-12T22:05:18.750768+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-12T22:01:32.292826+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-12T22:06:31.540263+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-12T22:06:32.906118+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6992ce0a-844b-40e3-8ed1-181c8474dccf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":906003882},{"type":"v1","addr":"172.21.15.28:6803","nonce":906003882}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":906003882},{"type":"v1","addr":"172.21.15.28:6805","nonce":906003882}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":906003882},{"type":"v1","addr":"172.21.15.28:6809","nonce":906003882}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":906003882},{"type":"v1","addr":"172.21.15.28:6807","nonce":906003882}]},"public_addr":"172.21.15.28:6803/906003882","cluster_addr":"172.21.15.28:6805/906003882","heartbeat_back_addr":"172.21.15.28:6809/906003882","heartbeat_front_addr":"172.21.15.28:6807/906003882","state":["exists","up"]},{"osd":1,"uuid":"ced20c96-c481-4d48-90d1-6b43228959ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":3354141819},{"type":"v1","addr":"172.21.15.86:6801","nonce":3354141819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":3354141819},{"type":"v1","addr":"172.21.15.86:6803","nonce":3354141819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":3354141819},{"type":"v1","addr":"172.21.15.86:6807","nonce":3354141819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":3354141819},{"type":"v1","addr":"172.21.15.86:6805","nonce":3354141819}]},"public_addr":"172.21.15.86:6801/3354141819","cluster_addr":"172.21.15.86:6803/3354141819","heartbeat_back_addr":"172.21.15.86:6807/3354141819","heartbeat_front_addr":"172.21.15.86:6805/3354141819","state":["exists","up"]},{"osd":2,"uuid":"54a2f221-fa65-4795-9f5e-e660cc7c0098","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":906459365},{"type":"v1","addr":"172.21.15.186:6801","nonce":906459365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":906459365},{"type":"v1","addr":"172.21.15.186:6803","nonce":906459365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":906459365},{"type":"v1","addr":"172.21.15.186:6807","nonce":906459365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":906459365},{"type":"v1","addr":"172.21.15.186:6805","nonce":906459365}]},"public_addr":"172.21.15.186:6801/906459365","cluster_addr":"172.21.15.186:6803/906459365","heartbeat_back_addr":"172.21.15.186:6807/906459365","heartbeat_front_addr":"172.21.15.186:6805/906459365","state":["exists","up"]},{"osd":3,"uuid":"86836737-ac9a-487b-93f2-7fcd3c406960","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":4182884436},{"type":"v1","addr":"172.21.15.28:6811","nonce":4182884436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":4182884436},{"type":"v1","addr":"172.21.15.28:6813","nonce":4182884436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":4182884436},{"type":"v1","addr":"172.21.15.28:6817","nonce":4182884436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":4182884436},{"type":"v1","addr":"172.21.15.28:6815","nonce":4182884436}]},"public_addr":"172.21.15.28:6811/4182884436","cluster_addr":"172.21.15.28:6813/4182884436","heartbeat_back_addr":"172.21.15.28:6817/4182884436","heartbeat_front_addr":"172.21.15.28:6815/4182884436","state":["exists","up"]},{"osd":4,"uuid":"d162ebcd-83c0-4fd5-9925-28b2cecea935","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2596998708},{"type":"v1","addr":"172.21.15.86:6809","nonce":2596998708}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2596998708},{"type":"v1","addr":"172.21.15.86:6811","nonce":2596998708}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2596998708},{"type":"v1","addr":"172.21.15.86:6815","nonce":2596998708}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2596998708},{"type":"v1","addr":"172.21.15.86:6813","nonce":2596998708}]},"public_addr":"172.21.15.86:6809/2596998708","cluster_addr":"172.21.15.86:6811/2596998708","heartbeat_back_addr":"172.21.15.86:6815/2596998708","heartbeat_front_addr":"172.21.15.86:6813/2596998708","state":["exists","up"]},{"osd":5,"uuid":"c794336a-a105-425c-b6c7-a32b90c05bb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":1944205756},{"type":"v1","addr":"172.21.15.186:6809","nonce":1944205756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":1944205756},{"type":"v1","addr":"172.21.15.186:6811","nonce":1944205756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":1944205756},{"type":"v1","addr":"172.21.15.186:6815","nonce":1944205756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":1944205756},{"type":"v1","addr":"172.21.15.186:6813","nonce":1944205756}]},"public_addr":"172.21.15.186:6809/1944205756","cluster_addr":"172.21.15.186:6811/1944205756","heartbeat_back_addr":"172.21.15.186:6815/1944205756","heartbeat_front_addr":"172.21.15.186:6813/1944205756","state":["exists","up"]},{"osd":6,"uuid":"516932ac-af00-45b7-ad3c-d6dec7db442f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":1642308925},{"type":"v1","addr":"172.21.15.28:6819","nonce":1642308925}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":1642308925},{"type":"v1","addr":"172.21.15.28:6821","nonce":1642308925}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1642308925},{"type":"v1","addr":"172.21.15.28:6825","nonce":1642308925}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":1642308925},{"type":"v1","addr":"172.21.15.28:6823","nonce":1642308925}]},"public_addr":"172.21.15.28:6819/1642308925","cluster_addr":"172.21.15.28:6821/1642308925","heartbeat_back_addr":"172.21.15.28:6825/1642308925","heartbeat_front_addr":"172.21.15.28:6823/1642308925","state":["exists","up"]},{"osd":7,"uuid":"ca7a059f-0414-438e-b2d1-a10f348036be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":726062711},{"type":"v1","addr":"172.21.15.86:6817","nonce":726062711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":726062711},{"type":"v1","addr":"172.21.15.86:6819","nonce":726062711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":726062711},{"type":"v1","addr":"172.21.15.86:6823","nonce":726062711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":726062711},{"type":"v1","addr":"172.21.15.86:6821","nonce":726062711}]},"public_addr":"172.21.15.86:6817/726062711","cluster_addr":"172.21.15.86:6819/726062711","heartbeat_back_addr":"172.21.15.86:6823/726062711","heartbeat_front_addr":"172.21.15.86:6821/726062711","state":["exists","up"]},{"osd":8,"uuid":"92e3f41f-aebd-422e-a99e-555f7ba4aa53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":3007687277},{"type":"v1","addr":"172.21.15.186:6817","nonce":3007687277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":3007687277},{"type":"v1","addr":"172.21.15.186:6819","nonce":3007687277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":3007687277},{"type":"v1","addr":"172.21.15.186:6823","nonce":3007687277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":3007687277},{"type":"v1","addr":"172.21.15.186:6821","nonce":3007687277}]},"public_addr":"172.21.15.186:6817/3007687277","cluster_addr":"172.21.15.186:6819/3007687277","heartbeat_back_addr":"172.21.15.186:6823/3007687277","heartbeat_front_addr":"172.21.15.186:6821/3007687277","state":["exists","up"]},{"osd":9,"uuid":"38ff7d9d-ee85-404c-8c59-adc327e1ccfa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":136220472},{"type":"v1","addr":"172.21.15.28:6827","nonce":136220472}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":136220472},{"type":"v1","addr":"172.21.15.28:6829","nonce":136220472}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":136220472},{"type":"v1","addr":"172.21.15.28:6833","nonce":136220472}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":136220472},{"type":"v1","addr":"172.21.15.28:6831","nonce":136220472}]},"public_addr":"172.21.15.28:6827/136220472","cluster_addr":"172.21.15.28:6829/136220472","heartbeat_back_addr":"172.21.15.28:6833/136220472","heartbeat_front_addr":"172.21.15.28:6831/136220472","state":["exists","up"]},{"osd":10,"uuid":"5e9ff657-afa8-4bab-8491-f209038392b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1189968310},{"type":"v1","addr":"172.21.15.86:6825","nonce":1189968310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":1189968310},{"type":"v1","addr":"172.21.15.86:6827","nonce":1189968310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":1189968310},{"type":"v1","addr":"172.21.15.86:6831","nonce":1189968310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":1189968310},{"type":"v1","addr":"172.21.15.86:6829","nonce":1189968310}]},"public_addr":"172.21.15.86:6825/1189968310","cluster_addr":"172.21.15.86:6827/1189968310","heartbeat_back_addr":"172.21.15.86:6831/1189968310","heartbeat_front_addr":"172.21.15.86:6829/1189968310","state":["exists","up"]},{"osd":11,"uuid":"f112e3c6-65b6-4253-a966-62001db7d069","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3801668110},{"type":"v1","addr":"172.21.15.186:6825","nonce":3801668110}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3801668110},{"type":"v1","addr":"172.21.15.186:6827","nonce":3801668110}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3801668110},{"type":"v1","addr":"172.21.15.186:6831","nonce":3801668110}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3801668110},{"type":"v1","addr":"172.21.15.186:6829","nonce":3801668110}]},"public_addr":"172.21.15.186:6825/3801668110","cluster_addr":"172.21.15.186:6827/3801668110","heartbeat_back_addr":"172.21.15.186:6831/3801668110","heartbeat_front_addr":"172.21.15.186:6829/3801668110","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:00:40.483623+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:01:05.284385+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:01:28.645716+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:02:01.295092+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:02:26.486601+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:02:50.807478+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:03:24.756169+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:03:49.975173+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:04:13.772233+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:04:46.758555+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:05:12.331685+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:05:36.462548+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":2,"to":1}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.25","mappings":[{"from":5,"to":9}]},{"pgid":"3.27","mappings":[{"from":4,"to":8}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/2580087728":"2024-05-13T22:08:58.638536+0000","172.21.15.28:6801/3123328651":"2024-05-13T22:08:58.638536+0000","172.21.15.28:0/820158394":"2024-05-13T22:08:58.638536+0000","172.21.15.28:0/1978033727":"2024-05-13T22:08:58.638536+0000","172.21.15.28:0/1177009533":"2024-05-13T21:58:45.550459+0000","172.21.15.28:6800/3760499540":"2024-05-13T21:58:45.550459+0000","172.21.15.28:6801/3783742728":"2024-05-13T21:58:10.421564+0000","172.21.15.28:6800/3783742728":"2024-05-13T21:58:10.421564+0000","172.21.15.28:0/2229156260":"2024-05-13T21:58:45.550459+0000","172.21.15.28:0/1202140963":"2024-05-13T21:58:10.421564+0000","172.21.15.28:6800/3123328651":"2024-05-13T22:08:58.638536+0000","172.21.15.28:0/2924223784":"2024-05-13T21:58:45.550459+0000","172.21.15.28:6801/3760499540":"2024-05-13T21:58:45.550459+0000","172.21.15.28:0/1240027206":"2024-05-13T21:58:10.421564+0000","172.21.15.28:0/1581755795":"2024-05-13T22:08:58.638536+0000","172.21.15.28:0/171417728":"2024-05-13T21:58:10.421564+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-12T22:09:02.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:02.027+0000 7fc864c00640 1 -- 172.21.15.28:0/1280445378 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fc8440a2e10 msgr2=0x7fc8440a52d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:02.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:02.027+0000 7fc864c00640 1 --2- 172.21.15.28:0/1280445378 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fc8440a2e10 0x7fc8440a52d0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fc850006010 tx=0x7fc850032cf0 comp rx=0 tx=0).stop 2024-05-12T22:09:02.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:02.027+0000 7fc864c00640 1 -- 172.21.15.28:0/1280445378 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc860108ea0 msgr2=0x7fc86019e550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:02.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:02.027+0000 7fc864c00640 1 --2- 172.21.15.28:0/1280445378 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc860108ea0 0x7fc86019e550 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fc84c00e9f0 tx=0x7fc84c00edf0 comp rx=0 tx=0).stop 2024-05-12T22:09:02.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:02.028+0000 7fc864c00640 1 -- 172.21.15.28:0/1280445378 shutdown_connections 2024-05-12T22:09:02.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:02.028+0000 7fc864c00640 1 --2- 172.21.15.28:0/1280445378 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fc8440a2e10 0x7fc8440a52d0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:02.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:02.028+0000 7fc864c00640 1 --2- 172.21.15.28:0/1280445378 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc860109860 0x7fc8601a55c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:02.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:02.028+0000 7fc864c00640 1 --2- 172.21.15.28:0/1280445378 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc860108ea0 0x7fc86019e550 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:02.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:02.028+0000 7fc864c00640 1 --2- 172.21.15.28:0/1280445378 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc860107cb0 0x7fc86019e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:02.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:02.028+0000 7fc864c00640 1 -- 172.21.15.28:0/1280445378 >> 172.21.15.28:0/1280445378 conn(0x7fc860076c70 msgr2=0x7fc86010c290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:09:02.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:02.028+0000 7fc864c00640 1 -- 172.21.15.28:0/1280445378 shutdown_connections 2024-05-12T22:09:02.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:02.028+0000 7fc864c00640 1 -- 172.21.15.28:0/1280445378 wait complete. 2024-05-12T22:09:02.049 INFO:tasks.fwd_scrub.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-12T22:01:32.292826+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-12T22:06:31.540263+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-12T22:06:32.906118+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-12T22:09:02.049 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-12T22:09:02.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:02 smithi186 ceph-mon[34785]: pgmap v4: 97 pgs: 97 active+clean; 1.1 MiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-12T22:09:02.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:02 smithi186 ceph-mon[34785]: mgrmap e24: x(active, since 2s), standbys: z, y 2024-05-12T22:09:02.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:02 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/833711502' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-12T22:09:02.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:02 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/1985424170' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-12T22:09:02.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:02.248+0000 7f612a400640 1 -- 172.21.15.28:0/100364570 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6124107420 msgr2=0x7f6124109870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:02.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:02.248+0000 7f612a400640 1 --2- 172.21.15.28:0/100364570 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6124107420 0x7f6124109870 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f611c009be0 tx=0x7f611c02a6c0 comp rx=0 tx=0).stop 2024-05-12T22:09:02.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:02.248+0000 7f612a400640 1 -- 172.21.15.28:0/100364570 shutdown_connections 2024-05-12T22:09:02.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:02.248+0000 7f612a400640 1 --2- 172.21.15.28:0/100364570 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6124107420 0x7f6124109870 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:02.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:02.248+0000 7f612a400640 1 --2- 172.21.15.28:0/100364570 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6124104af0 0x7f6124106ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:02.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:02.248+0000 7f612a400640 1 --2- 172.21.15.28:0/100364570 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f61241021c0 0x7f61241045b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:02.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:02.248+0000 7f612a400640 1 -- 172.21.15.28:0/100364570 >> 172.21.15.28:0/100364570 conn(0x7f61240fbbd0 msgr2=0x7f61240fdff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:09:02.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:02.248+0000 7f612a400640 1 -- 172.21.15.28:0/100364570 shutdown_connections 2024-05-12T22:09:02.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:02.248+0000 7f612a400640 1 -- 172.21.15.28:0/100364570 wait complete. 2024-05-12T22:09:02.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:02.249+0000 7f612a400640 1 Processor -- start 2024-05-12T22:09:02.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:02.250+0000 7f612a400640 1 -- start start 2024-05-12T22:09:02.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:02.250+0000 7f612a400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f61241021c0 0x7f6124199c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:02.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:02.250+0000 7f612a400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6124104af0 0x7f612419a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:02.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:02.250+0000 7f612a400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6124107420 0x7f61241a11c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:02.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:02.250+0000 7f612a400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f612410d3e0 con 0x7f6124104af0 2024-05-12T22:09:02.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:02.250+0000 7f612a400640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f612410d260 con 0x7f6124107420 2024-05-12T22:09:02.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:02.250+0000 7f612a400640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f612410d560 con 0x7f61241021c0 2024-05-12T22:09:02.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:02.250+0000 7f6123e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6124104af0 0x7f612419a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:02.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:02.250+0000 7f6129000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f61241021c0 0x7f6124199c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:02.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:02.250+0000 7f6129000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f61241021c0 0x7f6124199c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:42418/0 (socket says 172.21.15.28:42418) 2024-05-12T22:09:02.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:02.250+0000 7f6129000640 1 -- 172.21.15.28:0/3641622770 learned_addr learned my addr 172.21.15.28:0/3641622770 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:09:02.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:02.250+0000 7f6129a00640 1 --2- 172.21.15.28:0/3641622770 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6124107420 0x7f61241a11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:02.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:02.251+0000 7f6129000640 1 -- 172.21.15.28:0/3641622770 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6124107420 msgr2=0x7f61241a11c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:02.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:02.251+0000 7f6129000640 1 --2- 172.21.15.28:0/3641622770 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6124107420 0x7f61241a11c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:02.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:02.251+0000 7f6129000640 1 -- 172.21.15.28:0/3641622770 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6124104af0 msgr2=0x7f612419a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:02.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:02.251+0000 7f6129000640 1 --2- 172.21.15.28:0/3641622770 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6124104af0 0x7f612419a190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:02.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:02.251+0000 7f6129000640 1 -- 172.21.15.28:0/3641622770 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61241a1860 con 0x7f61241021c0 2024-05-12T22:09:02.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:02.251+0000 7f6129000640 1 --2- 172.21.15.28:0/3641622770 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f61241021c0 0x7f6124199c50 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f611800e9f0 tx=0x7f611800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:09:02.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:02.251+0000 7f6121600640 1 -- 172.21.15.28:0/3641622770 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f611800ccc0 con 0x7f61241021c0 2024-05-12T22:09:02.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:02.251+0000 7f6121600640 1 -- 172.21.15.28:0/3641622770 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6118005500 con 0x7f61241021c0 2024-05-12T22:09:02.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:02.251+0000 7f612a400640 1 -- 172.21.15.28:0/3641622770 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61241a1b50 con 0x7f61241021c0 2024-05-12T22:09:02.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:02.251+0000 7f6121600640 1 -- 172.21.15.28:0/3641622770 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6118010690 con 0x7f61241021c0 2024-05-12T22:09:02.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:02.252+0000 7f612a400640 1 -- 172.21.15.28:0/3641622770 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61241a2050 con 0x7f61241021c0 2024-05-12T22:09:02.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:02.253+0000 7f612a400640 1 -- 172.21.15.28:0/3641622770 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6124108f10 con 0x7f61241021c0 2024-05-12T22:09:02.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:02.255+0000 7f6121600640 1 -- 172.21.15.28:0/3641622770 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 24) ==== 138004+0+0 (secure 0 0 0) 0x7f61180046c0 con 0x7f61241021c0 2024-05-12T22:09:02.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:02.256+0000 7f6121600640 1 --2- 172.21.15.28:0/3641622770 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f610c0a2d90 0x7f610c0a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:02.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:02.256+0000 7f6121600640 1 -- 172.21.15.28:0/3641622770 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f61180cf940 con 0x7f61241021c0 2024-05-12T22:09:02.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:02.256+0000 7f6123e00640 1 --2- 172.21.15.28:0/3641622770 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f610c0a2d90 0x7f610c0a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:02.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:02.257+0000 7f6123e00640 1 --2- 172.21.15.28:0/3641622770 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f610c0a2d90 0x7f610c0a5250 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f6124069520 tx=0x7f6110005f90 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:09:02.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:02.258+0000 7f6121600640 1 -- 172.21.15.28:0/3641622770 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f611809c3c0 con 0x7f61241021c0 2024-05-12T22:09:02.284 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:02 smithi028 ceph-mon[29847]: pgmap v4: 97 pgs: 97 active+clean; 1.1 MiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-12T22:09:02.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:02 smithi028 ceph-mon[29847]: mgrmap e24: x(active, since 2s), standbys: z, y 2024-05-12T22:09:02.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:02 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/833711502' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-12T22:09:02.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:02 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/1985424170' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-12T22:09:02.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:02.467+0000 7f612a400640 1 -- 172.21.15.28:0/3641622770 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f612402d350 con 0x7f61241021c0 2024-05-12T22:09:02.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:02.467+0000 7f6121600640 1 -- 172.21.15.28:0/3641622770 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) ==== 93+0+10 (secure 0 0 0) 0x7f611809bcc0 con 0x7f61241021c0 2024-05-12T22:09:02.468 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-05-12T22:09:02.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:02.469+0000 7f612a400640 1 -- 172.21.15.28:0/3641622770 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f610c0a2d90 msgr2=0x7f610c0a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:02.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:02.469+0000 7f612a400640 1 --2- 172.21.15.28:0/3641622770 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f610c0a2d90 0x7f610c0a5250 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f6124069520 tx=0x7f6110005f90 comp rx=0 tx=0).stop 2024-05-12T22:09:02.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:02.469+0000 7f612a400640 1 -- 172.21.15.28:0/3641622770 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f61241021c0 msgr2=0x7f6124199c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:02.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:02.469+0000 7f612a400640 1 --2- 172.21.15.28:0/3641622770 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f61241021c0 0x7f6124199c50 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f611800e9f0 tx=0x7f611800eeb0 comp rx=0 tx=0).stop 2024-05-12T22:09:02.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:02.470+0000 7f612a400640 1 -- 172.21.15.28:0/3641622770 shutdown_connections 2024-05-12T22:09:02.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:02.470+0000 7f612a400640 1 --2- 172.21.15.28:0/3641622770 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f610c0a2d90 0x7f610c0a5250 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:02.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:02.470+0000 7f612a400640 1 --2- 172.21.15.28:0/3641622770 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6124107420 0x7f61241a11c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:02.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:02.470+0000 7f612a400640 1 --2- 172.21.15.28:0/3641622770 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6124104af0 0x7f612419a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:02.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:02.470+0000 7f612a400640 1 --2- 172.21.15.28:0/3641622770 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f61241021c0 0x7f6124199c50 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:02.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:02.470+0000 7f612a400640 1 -- 172.21.15.28:0/3641622770 >> 172.21.15.28:0/3641622770 conn(0x7f61240fbbd0 msgr2=0x7f6124101890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:09:02.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:02.470+0000 7f612a400640 1 -- 172.21.15.28:0/3641622770 shutdown_connections 2024-05-12T22:09:02.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:02.470+0000 7f612a400640 1 -- 172.21.15.28:0/3641622770 wait complete. 2024-05-12T22:09:02.489 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-12T22:09:02.491 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:02 smithi086 ceph-mon[34297]: pgmap v4: 97 pgs: 97 active+clean; 1.1 MiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-12T22:09:02.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:02 smithi086 ceph-mon[34297]: mgrmap e24: x(active, since 2s), standbys: z, y 2024-05-12T22:09:02.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:02 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/833711502' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-12T22:09:02.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:02 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/1985424170' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-12T22:09:02.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:02.673+0000 7f4c23200640 1 -- 172.21.15.28:0/888363683 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c24101760 msgr2=0x7f4c24101b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:02.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:02.673+0000 7f4c23200640 1 --2- 172.21.15.28:0/888363683 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c24101760 0x7f4c24101b80 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f4c14009900 tx=0x7f4c1402a410 comp rx=0 tx=0).stop 2024-05-12T22:09:02.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:02.673+0000 7f4c23200640 1 -- 172.21.15.28:0/888363683 shutdown_connections 2024-05-12T22:09:02.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:02.673+0000 7f4c23200640 1 --2- 172.21.15.28:0/888363683 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4c24103310 0x7f4c24109ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:02.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:02.673+0000 7f4c23200640 1 --2- 172.21.15.28:0/888363683 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4c24102950 0x7f4c24102dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:02.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:02.673+0000 7f4c23200640 1 --2- 172.21.15.28:0/888363683 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c24101760 0x7f4c24101b80 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:02.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:02.673+0000 7f4c23200640 1 -- 172.21.15.28:0/888363683 >> 172.21.15.28:0/888363683 conn(0x7f4c240fccd0 msgr2=0x7f4c240ff110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:09:02.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:02.673+0000 7f4c23200640 1 -- 172.21.15.28:0/888363683 shutdown_connections 2024-05-12T22:09:02.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:02.673+0000 7f4c23200640 1 -- 172.21.15.28:0/888363683 wait complete. 2024-05-12T22:09:02.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:02.674+0000 7f4c23200640 1 Processor -- start 2024-05-12T22:09:02.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:02.674+0000 7f4c23200640 1 -- start start 2024-05-12T22:09:02.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:02.674+0000 7f4c23200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4c24101760 0x7f4c240755e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:02.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:02.674+0000 7f4c23200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c24102950 0x7f4c24073c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:02.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:02.674+0000 7f4c23200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4c24103310 0x7f4c24074160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:02.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:02.674+0000 7f4c23200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4c2410d400 con 0x7f4c24102950 2024-05-12T22:09:02.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:02.674+0000 7f4c23200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f4c2410d280 con 0x7f4c24101760 2024-05-12T22:09:02.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:02.674+0000 7f4c23200640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f4c2410d580 con 0x7f4c24103310 2024-05-12T22:09:02.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:02.674+0000 7f4c21e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4c24101760 0x7f4c240755e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:02.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:02.674+0000 7f4c21e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4c24101760 0x7f4c240755e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:49366/0 (socket says 172.21.15.28:49366) 2024-05-12T22:09:02.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:02.674+0000 7f4c21e00640 1 -- 172.21.15.28:0/2976303736 learned_addr learned my addr 172.21.15.28:0/2976303736 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:09:02.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:02.675+0000 7f4c22800640 1 --2- 172.21.15.28:0/2976303736 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4c24103310 0x7f4c24074160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:02.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:02.675+0000 7f4c21400640 1 --2- 172.21.15.28:0/2976303736 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c24102950 0x7f4c24073c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:02.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:02.675+0000 7f4c22800640 1 -- 172.21.15.28:0/2976303736 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4c24101760 msgr2=0x7f4c240755e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:02.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:02.675+0000 7f4c22800640 1 --2- 172.21.15.28:0/2976303736 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4c24101760 0x7f4c240755e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:02.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:02.675+0000 7f4c22800640 1 -- 172.21.15.28:0/2976303736 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c24102950 msgr2=0x7f4c24073c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:02.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:02.675+0000 7f4c22800640 1 --2- 172.21.15.28:0/2976303736 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c24102950 0x7f4c24073c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:02.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:02.675+0000 7f4c22800640 1 -- 172.21.15.28:0/2976303736 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c240749f0 con 0x7f4c24103310 2024-05-12T22:09:02.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:02.675+0000 7f4c22800640 1 --2- 172.21.15.28:0/2976303736 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4c24103310 0x7f4c24074160 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f4c1c009fd0 tx=0x7f4c1c00ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:09:02.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:02.676+0000 7f4c1aa00640 1 -- 172.21.15.28:0/2976303736 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c1c00ce20 con 0x7f4c24103310 2024-05-12T22:09:02.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:02.676+0000 7f4c23200640 1 -- 172.21.15.28:0/2976303736 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c241a1b70 con 0x7f4c24103310 2024-05-12T22:09:02.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:02.676+0000 7f4c1aa00640 1 -- 172.21.15.28:0/2976303736 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4c1c005020 con 0x7f4c24103310 2024-05-12T22:09:02.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:02.676+0000 7f4c1aa00640 1 -- 172.21.15.28:0/2976303736 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c1c010720 con 0x7f4c24103310 2024-05-12T22:09:02.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:02.676+0000 7f4c23200640 1 -- 172.21.15.28:0/2976303736 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c241a2070 con 0x7f4c24103310 2024-05-12T22:09:02.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:02.677+0000 7f4c23200640 1 -- 172.21.15.28:0/2976303736 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c24108a80 con 0x7f4c24103310 2024-05-12T22:09:02.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:02.679+0000 7f4c1aa00640 1 -- 172.21.15.28:0/2976303736 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 24) ==== 138004+0+0 (secure 0 0 0) 0x7f4c1c005540 con 0x7f4c24103310 2024-05-12T22:09:02.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:02.680+0000 7f4c1aa00640 1 --2- 172.21.15.28:0/2976303736 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f4c080a2d90 0x7f4c080a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:02.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:02.680+0000 7f4c1aa00640 1 -- 172.21.15.28:0/2976303736 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f4c1c0cfd50 con 0x7f4c24103310 2024-05-12T22:09:02.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:02.681+0000 7f4c21e00640 1 --2- 172.21.15.28:0/2976303736 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f4c080a2d90 0x7f4c080a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:02.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:02.681+0000 7f4c21e00640 1 --2- 172.21.15.28:0/2976303736 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f4c080a2d90 0x7f4c080a5250 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f4c14000c00 tx=0x7f4c140315b0 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:09:02.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:02.684+0000 7f4c1aa00640 1 -- 172.21.15.28:0/2976303736 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f4c1c09c7d0 con 0x7f4c24103310 2024-05-12T22:09:02.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:02.892+0000 7f4c23200640 1 -- 172.21.15.28:0/2976303736 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f4c2402d370 con 0x7f4c24103310 2024-05-12T22:09:02.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:02.893+0000 7f4c1aa00640 1 -- 172.21.15.28:0/2976303736 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) ==== 104+0+11 (secure 0 0 0) 0x7f4c1c09c0d0 con 0x7f4c24103310 2024-05-12T22:09:02.894 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-05-12T22:09:02.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:02.896+0000 7f4c23200640 1 -- 172.21.15.28:0/2976303736 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f4c080a2d90 msgr2=0x7f4c080a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:02.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:02.896+0000 7f4c23200640 1 --2- 172.21.15.28:0/2976303736 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f4c080a2d90 0x7f4c080a5250 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f4c14000c00 tx=0x7f4c140315b0 comp rx=0 tx=0).stop 2024-05-12T22:09:02.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:02.896+0000 7f4c23200640 1 -- 172.21.15.28:0/2976303736 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4c24103310 msgr2=0x7f4c24074160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:02.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:02.896+0000 7f4c23200640 1 --2- 172.21.15.28:0/2976303736 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4c24103310 0x7f4c24074160 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f4c1c009fd0 tx=0x7f4c1c00ef10 comp rx=0 tx=0).stop 2024-05-12T22:09:02.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:02.896+0000 7f4c23200640 1 -- 172.21.15.28:0/2976303736 shutdown_connections 2024-05-12T22:09:02.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:02.896+0000 7f4c23200640 1 --2- 172.21.15.28:0/2976303736 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f4c080a2d90 0x7f4c080a5250 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:02.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:02.896+0000 7f4c23200640 1 --2- 172.21.15.28:0/2976303736 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4c24103310 0x7f4c24074160 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:02.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:02.896+0000 7f4c23200640 1 --2- 172.21.15.28:0/2976303736 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c24102950 0x7f4c24073c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:02.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:02.896+0000 7f4c23200640 1 --2- 172.21.15.28:0/2976303736 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4c24101760 0x7f4c240755e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:02.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:02.896+0000 7f4c23200640 1 -- 172.21.15.28:0/2976303736 >> 172.21.15.28:0/2976303736 conn(0x7f4c240fccd0 msgr2=0x7f4c24105c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:09:02.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:02.896+0000 7f4c23200640 1 -- 172.21.15.28:0/2976303736 shutdown_connections 2024-05-12T22:09:02.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:02.896+0000 7f4c23200640 1 -- 172.21.15.28:0/2976303736 wait complete. 2024-05-12T22:09:02.914 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-12T22:09:03.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:03.116+0000 7f6d6e000640 1 -- 172.21.15.28:0/1259790773 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d68107420 msgr2=0x7f6d68109870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:03.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:03.116+0000 7f6d6e000640 1 --2- 172.21.15.28:0/1259790773 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d68107420 0x7f6d68109870 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f6d60009ca0 tx=0x7f6d6002a8c0 comp rx=0 tx=0).stop 2024-05-12T22:09:03.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:03.116+0000 7f6d6e000640 1 -- 172.21.15.28:0/1259790773 shutdown_connections 2024-05-12T22:09:03.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:03.116+0000 7f6d6e000640 1 --2- 172.21.15.28:0/1259790773 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d68107420 0x7f6d68109870 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:03.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:03.116+0000 7f6d6e000640 1 --2- 172.21.15.28:0/1259790773 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d68104af0 0x7f6d68106ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:03.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:03.116+0000 7f6d6e000640 1 --2- 172.21.15.28:0/1259790773 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6d681021c0 0x7f6d681045b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:03.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:03.116+0000 7f6d6e000640 1 -- 172.21.15.28:0/1259790773 >> 172.21.15.28:0/1259790773 conn(0x7f6d680fbbd0 msgr2=0x7f6d680fdff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:09:03.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:03.116+0000 7f6d6e000640 1 -- 172.21.15.28:0/1259790773 shutdown_connections 2024-05-12T22:09:03.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:03.116+0000 7f6d6e000640 1 -- 172.21.15.28:0/1259790773 wait complete. 2024-05-12T22:09:03.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:03.117+0000 7f6d6e000640 1 Processor -- start 2024-05-12T22:09:03.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:03.117+0000 7f6d6e000640 1 -- start start 2024-05-12T22:09:03.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:03.118+0000 7f6d6e000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d681021c0 0x7f6d68101b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:03.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:03.118+0000 7f6d6e000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d68104af0 0x7f6d68100170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:03.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:03.118+0000 7f6d6e000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6d68107420 0x7f6d681006b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:03.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:03.118+0000 7f6d6e000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6d6810d410 con 0x7f6d681021c0 2024-05-12T22:09:03.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:03.118+0000 7f6d6e000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f6d6810d290 con 0x7f6d68104af0 2024-05-12T22:09:03.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:03.118+0000 7f6d6e000640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f6d6810d590 con 0x7f6d68107420 2024-05-12T22:09:03.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:03.118+0000 7f6d6cc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d681021c0 0x7f6d68101b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:03.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:03.118+0000 7f6d67e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d68104af0 0x7f6d68100170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:03.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:03.118+0000 7f6d67e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d68104af0 0x7f6d68100170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:49376/0 (socket says 172.21.15.28:49376) 2024-05-12T22:09:03.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:03.118+0000 7f6d6cc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d681021c0 0x7f6d68101b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51294/0 (socket says 172.21.15.28:51294) 2024-05-12T22:09:03.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:03.118+0000 7f6d67e00640 1 -- 172.21.15.28:0/2524130045 learned_addr learned my addr 172.21.15.28:0/2524130045 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:09:03.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:03.118+0000 7f6d6d600640 1 --2- 172.21.15.28:0/2524130045 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6d68107420 0x7f6d681006b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:03.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:03.118+0000 7f6d67e00640 1 -- 172.21.15.28:0/2524130045 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6d68107420 msgr2=0x7f6d681006b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:03.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:03.118+0000 7f6d67e00640 1 --2- 172.21.15.28:0/2524130045 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6d68107420 0x7f6d681006b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:03.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:03.118+0000 7f6d67e00640 1 -- 172.21.15.28:0/2524130045 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d681021c0 msgr2=0x7f6d68101b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:03.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:03.118+0000 7f6d67e00640 1 --2- 172.21.15.28:0/2524130045 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d681021c0 0x7f6d68101b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:03.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:03.119+0000 7f6d67e00640 1 -- 172.21.15.28:0/2524130045 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d68100f40 con 0x7f6d68104af0 2024-05-12T22:09:03.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:03.119+0000 7f6d6cc00640 1 --2- 172.21.15.28:0/2524130045 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d681021c0 0x7f6d68101b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:09:03.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:03.119+0000 7f6d67e00640 1 --2- 172.21.15.28:0/2524130045 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d68104af0 0x7f6d68100170 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f6d5400e9a0 tx=0x7f6d5400ee60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:09:03.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:03.119+0000 7f6d65600640 1 -- 172.21.15.28:0/2524130045 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d5400cd80 con 0x7f6d68104af0 2024-05-12T22:09:03.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:03.119+0000 7f6d65600640 1 -- 172.21.15.28:0/2524130045 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6d540035f0 con 0x7f6d68104af0 2024-05-12T22:09:03.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:03.119+0000 7f6d65600640 1 -- 172.21.15.28:0/2524130045 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d54010670 con 0x7f6d68104af0 2024-05-12T22:09:03.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:03.119+0000 7f6d6e000640 1 -- 172.21.15.28:0/2524130045 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d681a1b00 con 0x7f6d68104af0 2024-05-12T22:09:03.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:03.119+0000 7f6d6e000640 1 -- 172.21.15.28:0/2524130045 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d681a20a0 con 0x7f6d68104af0 2024-05-12T22:09:03.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:03.120+0000 7f6d6e000640 1 -- 172.21.15.28:0/2524130045 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d680fd6c0 con 0x7f6d68104af0 2024-05-12T22:09:03.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:03.125+0000 7f6d65600640 1 -- 172.21.15.28:0/2524130045 <== mon.2 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 25) ==== 138060+0+0 (secure 0 0 0) 0x7f6d540037a0 con 0x7f6d68104af0 2024-05-12T22:09:03.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:03.127+0000 7f6d65600640 1 --2- 172.21.15.28:0/2524130045 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f6d500a2ea0 0x7f6d500a5360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:03.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:03.127+0000 7f6d65600640 1 -- 172.21.15.28:0/2524130045 <== mon.2 v2:172.21.15.86:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f6d540cf980 con 0x7f6d68104af0 2024-05-12T22:09:03.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:03.127+0000 7f6d6cc00640 1 --2- 172.21.15.28:0/2524130045 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f6d500a2ea0 0x7f6d500a5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:03.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:03.128+0000 7f6d6cc00640 1 --2- 172.21.15.28:0/2524130045 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f6d500a2ea0 0x7f6d500a5360 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f6d5c009fc0 tx=0x7f6d5c005f90 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:09:03.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:03.131+0000 7f6d65600640 1 -- 172.21.15.28:0/2524130045 <== mon.2 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f6d5409c320 con 0x7f6d68104af0 2024-05-12T22:09:03.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:03 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/1280445378' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-12T22:09:03.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:03 smithi028 ceph-mon[29847]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:09:03.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:03 smithi028 ceph-mon[29847]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:09:03.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:03 smithi028 ceph-mon[29847]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:09:03.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:03 smithi028 ceph-mon[29847]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:09:03.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:03 smithi028 ceph-mon[29847]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:09:03.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:03 smithi028 ceph-mon[29847]: from='mgr.25189 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:09:03.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:03 smithi028 ceph-mon[29847]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:09:03.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:03 smithi028 ceph-mon[29847]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:09:03.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:03 smithi028 ceph-mon[29847]: from='mgr.25189 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:09:03.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:03 smithi028 ceph-mon[29847]: from='mgr.25189 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:09:03.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:03 smithi028 ceph-mon[29847]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:09:03.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:03 smithi028 ceph-mon[29847]: from='mgr.25189 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:09:03.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:03 smithi028 ceph-mon[29847]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:09:03.287 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:03 smithi028 ceph-mon[29847]: from='mgr.25189 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:09:03.287 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:03 smithi028 ceph-mon[29847]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:09:03.287 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:03 smithi028 ceph-mon[29847]: from='mgr.25189 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:09:03.287 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:03 smithi028 ceph-mon[29847]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:09:03.287 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:03 smithi028 ceph-mon[29847]: from='mgr.25189 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:09:03.287 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:03 smithi028 ceph-mon[29847]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:09:03.287 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:03 smithi028 ceph-mon[29847]: from='mgr.25189 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:09:03.287 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:03 smithi028 ceph-mon[29847]: Adjusting osd_memory_target on smithi086 to 176.2M 2024-05-12T22:09:03.287 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:03 smithi028 ceph-mon[29847]: Adjusting osd_memory_target on smithi186 to 176.2M 2024-05-12T22:09:03.287 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:03 smithi028 ceph-mon[29847]: Unable to set osd_memory_target on smithi086 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-05-12T22:09:03.287 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:03 smithi028 ceph-mon[29847]: Unable to set osd_memory_target on smithi186 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-05-12T22:09:03.287 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:03 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/3641622770' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-12T22:09:03.287 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:03 smithi028 ceph-mon[29847]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:09:03.287 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:03 smithi028 ceph-mon[29847]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:09:03.287 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:03 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/2976303736' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-12T22:09:03.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:03.340+0000 7f6d6e000640 1 -- 172.21.15.28:0/2524130045 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f6d6802d3e0 con 0x7f6d68104af0 2024-05-12T22:09:03.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:03.341+0000 7f6d65600640 1 -- 172.21.15.28:0/2524130045 <== mon.2 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) ==== 100+0+11 (secure 0 0 0) 0x7f6d5409bc20 con 0x7f6d68104af0 2024-05-12T22:09:03.341 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-05-12T22:09:03.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:03.343+0000 7f6d6e000640 1 -- 172.21.15.28:0/2524130045 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f6d500a2ea0 msgr2=0x7f6d500a5360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:03.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:03.343+0000 7f6d6e000640 1 --2- 172.21.15.28:0/2524130045 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f6d500a2ea0 0x7f6d500a5360 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f6d5c009fc0 tx=0x7f6d5c005f90 comp rx=0 tx=0).stop 2024-05-12T22:09:03.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:03.343+0000 7f6d6e000640 1 -- 172.21.15.28:0/2524130045 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d68104af0 msgr2=0x7f6d68100170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:03.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:03.343+0000 7f6d6e000640 1 --2- 172.21.15.28:0/2524130045 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d68104af0 0x7f6d68100170 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f6d5400e9a0 tx=0x7f6d5400ee60 comp rx=0 tx=0).stop 2024-05-12T22:09:03.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:03.343+0000 7f6d6e000640 1 -- 172.21.15.28:0/2524130045 shutdown_connections 2024-05-12T22:09:03.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:03.343+0000 7f6d6e000640 1 --2- 172.21.15.28:0/2524130045 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f6d500a2ea0 0x7f6d500a5360 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:03.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:03.343+0000 7f6d6e000640 1 --2- 172.21.15.28:0/2524130045 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6d68107420 0x7f6d681006b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:03.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:03.344+0000 7f6d6e000640 1 --2- 172.21.15.28:0/2524130045 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d68104af0 0x7f6d68100170 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:03.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:03.344+0000 7f6d6e000640 1 --2- 172.21.15.28:0/2524130045 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d681021c0 0x7f6d68101b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:03.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:03.344+0000 7f6d6e000640 1 -- 172.21.15.28:0/2524130045 >> 172.21.15.28:0/2524130045 conn(0x7f6d680fbbd0 msgr2=0x7f6d68102ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:09:03.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:03.344+0000 7f6d6e000640 1 -- 172.21.15.28:0/2524130045 shutdown_connections 2024-05-12T22:09:03.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:03.344+0000 7f6d6e000640 1 -- 172.21.15.28:0/2524130045 wait complete. 2024-05-12T22:09:03.365 INFO:tasks.fwd_scrub:Wait for all MDSs to reach steady state... 2024-05-12T22:09:03.365 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-12T22:09:03.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:03 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/1280445378' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-12T22:09:03.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:03 smithi186 ceph-mon[34785]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:09:03.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:03 smithi186 ceph-mon[34785]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:09:03.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:03 smithi186 ceph-mon[34785]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:09:03.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:03 smithi186 ceph-mon[34785]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:09:03.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:03 smithi186 ceph-mon[34785]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:09:03.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:03 smithi186 ceph-mon[34785]: from='mgr.25189 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:09:03.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:03 smithi186 ceph-mon[34785]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:09:03.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:03 smithi186 ceph-mon[34785]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:09:03.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:03 smithi186 ceph-mon[34785]: from='mgr.25189 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:09:03.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:03 smithi186 ceph-mon[34785]: from='mgr.25189 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:09:03.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:03 smithi186 ceph-mon[34785]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:09:03.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:03 smithi186 ceph-mon[34785]: from='mgr.25189 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:09:03.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:03 smithi186 ceph-mon[34785]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:09:03.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:03 smithi186 ceph-mon[34785]: from='mgr.25189 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:09:03.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:03 smithi186 ceph-mon[34785]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:09:03.431 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:03 smithi186 ceph-mon[34785]: from='mgr.25189 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:09:03.431 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:03 smithi186 ceph-mon[34785]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:09:03.431 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:03 smithi186 ceph-mon[34785]: from='mgr.25189 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:09:03.431 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:03 smithi186 ceph-mon[34785]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:09:03.431 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:03 smithi186 ceph-mon[34785]: from='mgr.25189 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:09:03.431 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:03 smithi186 ceph-mon[34785]: Adjusting osd_memory_target on smithi086 to 176.2M 2024-05-12T22:09:03.431 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:03 smithi186 ceph-mon[34785]: Adjusting osd_memory_target on smithi186 to 176.2M 2024-05-12T22:09:03.431 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:03 smithi186 ceph-mon[34785]: Unable to set osd_memory_target on smithi086 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-05-12T22:09:03.431 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:03 smithi186 ceph-mon[34785]: Unable to set osd_memory_target on smithi186 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-05-12T22:09:03.431 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:03 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/3641622770' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-12T22:09:03.431 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:03 smithi186 ceph-mon[34785]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:09:03.431 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:03 smithi186 ceph-mon[34785]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:09:03.431 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:03 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/2976303736' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-12T22:09:03.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:03 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/1280445378' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-12T22:09:03.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:03 smithi086 ceph-mon[34297]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:09:03.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:03 smithi086 ceph-mon[34297]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:09:03.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:03 smithi086 ceph-mon[34297]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:09:03.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:03 smithi086 ceph-mon[34297]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:09:03.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:03 smithi086 ceph-mon[34297]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:09:03.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:03 smithi086 ceph-mon[34297]: from='mgr.25189 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:09:03.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:03 smithi086 ceph-mon[34297]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:09:03.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:03 smithi086 ceph-mon[34297]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:09:03.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:03 smithi086 ceph-mon[34297]: from='mgr.25189 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:09:03.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:03 smithi086 ceph-mon[34297]: from='mgr.25189 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:09:03.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:03 smithi086 ceph-mon[34297]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:09:03.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:03 smithi086 ceph-mon[34297]: from='mgr.25189 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:09:03.494 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:03 smithi086 ceph-mon[34297]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:09:03.494 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:03 smithi086 ceph-mon[34297]: from='mgr.25189 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:09:03.494 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:03 smithi086 ceph-mon[34297]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:09:03.494 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:03 smithi086 ceph-mon[34297]: from='mgr.25189 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:09:03.494 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:03 smithi086 ceph-mon[34297]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:09:03.494 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:03 smithi086 ceph-mon[34297]: from='mgr.25189 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:09:03.494 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:03 smithi086 ceph-mon[34297]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:09:03.494 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:03 smithi086 ceph-mon[34297]: from='mgr.25189 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:09:03.494 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:03 smithi086 ceph-mon[34297]: Adjusting osd_memory_target on smithi086 to 176.2M 2024-05-12T22:09:03.494 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:03 smithi086 ceph-mon[34297]: Adjusting osd_memory_target on smithi186 to 176.2M 2024-05-12T22:09:03.494 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:03 smithi086 ceph-mon[34297]: Unable to set osd_memory_target on smithi086 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-05-12T22:09:03.495 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:03 smithi086 ceph-mon[34297]: Unable to set osd_memory_target on smithi186 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-05-12T22:09:03.495 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:03 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/3641622770' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-12T22:09:03.495 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:03 smithi086 ceph-mon[34297]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:09:03.495 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:03 smithi086 ceph-mon[34297]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:09:03.495 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:03 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/2976303736' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-12T22:09:03.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:03.566+0000 7f6552c00640 1 -- 172.21.15.28:0/2720090862 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f654c103280 msgr2=0x7f654c109b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:03.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:03.566+0000 7f6552c00640 1 --2- 172.21.15.28:0/2720090862 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f654c103280 0x7f654c109b10 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f6544009ca0 tx=0x7f654402a8c0 comp rx=0 tx=0).stop 2024-05-12T22:09:03.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:03.566+0000 7f6552c00640 1 -- 172.21.15.28:0/2720090862 shutdown_connections 2024-05-12T22:09:03.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:03.566+0000 7f6552c00640 1 --2- 172.21.15.28:0/2720090862 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f654c103280 0x7f654c109b10 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:03.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:03.566+0000 7f6552c00640 1 --2- 172.21.15.28:0/2720090862 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f654c1028c0 0x7f654c102d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:03.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:03.566+0000 7f6552c00640 1 --2- 172.21.15.28:0/2720090862 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f654c1016d0 0x7f654c101af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:03.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:03.566+0000 7f6552c00640 1 -- 172.21.15.28:0/2720090862 >> 172.21.15.28:0/2720090862 conn(0x7f654c0fcc40 msgr2=0x7f654c0ff080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:09:03.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:03.566+0000 7f6552c00640 1 -- 172.21.15.28:0/2720090862 shutdown_connections 2024-05-12T22:09:03.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:03.566+0000 7f6552c00640 1 -- 172.21.15.28:0/2720090862 wait complete. 2024-05-12T22:09:03.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:03.567+0000 7f6552c00640 1 Processor -- start 2024-05-12T22:09:03.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:03.567+0000 7f6552c00640 1 -- start start 2024-05-12T22:09:03.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:03.568+0000 7f6552c00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f654c1016d0 0x7f654c0755e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:03.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:03.568+0000 7f6552c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f654c1028c0 0x7f654c073c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:03.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:03.568+0000 7f6552c00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f654c103280 0x7f654c074160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:03.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:03.568+0000 7f6552c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f654c10d380 con 0x7f654c1028c0 2024-05-12T22:09:03.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:03.568+0000 7f6552c00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f654c10d200 con 0x7f654c103280 2024-05-12T22:09:03.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:03.568+0000 7f6552c00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f654c10d500 con 0x7f654c1016d0 2024-05-12T22:09:03.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:03.568+0000 7f6550e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f654c1028c0 0x7f654c073c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:03.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:03.568+0000 7f6551800640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f654c1016d0 0x7f654c0755e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:03.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:03.568+0000 7f6550e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f654c1028c0 0x7f654c073c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51312/0 (socket says 172.21.15.28:51312) 2024-05-12T22:09:03.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:03.568+0000 7f6550e00640 1 -- 172.21.15.28:0/4111270902 learned_addr learned my addr 172.21.15.28:0/4111270902 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:09:03.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:03.568+0000 7f6552200640 1 --2- 172.21.15.28:0/4111270902 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f654c103280 0x7f654c074160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:03.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:03.569+0000 7f6550e00640 1 -- 172.21.15.28:0/4111270902 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f654c1016d0 msgr2=0x7f654c0755e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:03.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:03.569+0000 7f6550e00640 1 --2- 172.21.15.28:0/4111270902 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f654c1016d0 0x7f654c0755e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:03.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:03.569+0000 7f6550e00640 1 -- 172.21.15.28:0/4111270902 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f654c103280 msgr2=0x7f654c074160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:03.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:03.569+0000 7f6550e00640 1 --2- 172.21.15.28:0/4111270902 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f654c103280 0x7f654c074160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:03.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:03.569+0000 7f6550e00640 1 -- 172.21.15.28:0/4111270902 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f654c0749f0 con 0x7f654c1028c0 2024-05-12T22:09:03.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:03.569+0000 7f6551800640 1 --2- 172.21.15.28:0/4111270902 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f654c1016d0 0x7f654c0755e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:09:03.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:03.569+0000 7f6550e00640 1 --2- 172.21.15.28:0/4111270902 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f654c1028c0 0x7f654c073c20 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f653800e9f0 tx=0x7f653800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:09:03.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:03.570+0000 7f654a000640 1 -- 172.21.15.28:0/4111270902 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f653800cd80 con 0x7f654c1028c0 2024-05-12T22:09:03.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:03.570+0000 7f654a000640 1 -- 172.21.15.28:0/4111270902 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6538010910 con 0x7f654c1028c0 2024-05-12T22:09:03.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:03.570+0000 7f6552c00640 1 -- 172.21.15.28:0/4111270902 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f654c1a1a50 con 0x7f654c1028c0 2024-05-12T22:09:03.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:03.570+0000 7f654a000640 1 -- 172.21.15.28:0/4111270902 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6538018990 con 0x7f654c1028c0 2024-05-12T22:09:03.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:03.570+0000 7f6552c00640 1 -- 172.21.15.28:0/4111270902 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f654c1a1ff0 con 0x7f654c1028c0 2024-05-12T22:09:03.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:03.571+0000 7f6552c00640 1 -- 172.21.15.28:0/4111270902 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f654c108990 con 0x7f654c1028c0 2024-05-12T22:09:03.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:03.577+0000 7f654a000640 1 -- 172.21.15.28:0/4111270902 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138060+0+0 (secure 0 0 0) 0x7f6538010ac0 con 0x7f654c1028c0 2024-05-12T22:09:03.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:03.579+0000 7f654a000640 1 --2- 172.21.15.28:0/4111270902 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f65340a2ea0 0x7f65340a5360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:03.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:03.579+0000 7f654a000640 1 -- 172.21.15.28:0/4111270902 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f65380cf9a0 con 0x7f654c1028c0 2024-05-12T22:09:03.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:03.579+0000 7f6551800640 1 --2- 172.21.15.28:0/4111270902 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f65340a2ea0 0x7f65340a5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:03.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:03.580+0000 7f6551800640 1 --2- 172.21.15.28:0/4111270902 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f65340a2ea0 0x7f65340a5360 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f654c102720 tx=0x7f6540005f90 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:09:03.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:03.584+0000 7f654a000640 1 -- 172.21.15.28:0/4111270902 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f653809bcf0 con 0x7f654c1028c0 2024-05-12T22:09:03.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:03.857+0000 7f6552c00640 1 -- 172.21.15.28:0/4111270902 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f654c02d3c0 con 0x7f654c1028c0 2024-05-12T22:09:03.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:03.859+0000 7f654a000640 1 -- 172.21.15.28:0/4111270902 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11864 (secure 0 0 0) 0x7f653809fff0 con 0x7f654c1028c0 2024-05-12T22:09:03.859 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-12T22:09:03.859 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1844141745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1844141745},{"type":"v1","addr":"172.21.15.186:6833","nonce":1844141745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6833/2960763758","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":2960763758},{"type":"v1","addr":"172.21.15.86:6833","nonce":2960763758}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24470,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/4131134219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":4131134219},{"type":"v1","addr":"172.21.15.28:6835","nonce":4131134219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24481,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/1727869064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1727869064},{"type":"v1","addr":"172.21.15.186:6835","nonce":1727869064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24482,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2938260704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2938260704},{"type":"v1","addr":"172.21.15.28:6837","nonce":2938260704}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24494,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3473700910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3473700910},{"type":"v1","addr":"172.21.15.86:6835","nonce":3473700910}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-12T22:06:33.426986+0000","modified":"2024-05-12T22:08:58.179491+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24581,"mds_1":24550,"mds_2":24532},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24499":{"gid":24499,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6837/3600809235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":3600809235},{"type":"v1","addr":"172.21.15.186:6837","nonce":3600809235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"i","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6839/2399610807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":2399610807},{"type":"v1","addr":"172.21.15.186:6839","nonce":2399610807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24530":{"gid":24530,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.86:6837/3782448884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":3782448884},{"type":"v1","addr":"172.21.15.86:6837","nonce":3782448884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24532":{"gid":24532,"name":"g","rank":2,"incarnation":21,"state":"up:active","state_seq":6,"addr":"172.21.15.28:6839/123892158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":123892158},{"type":"v1","addr":"172.21.15.28:6839","nonce":123892158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24550":{"gid":24550,"name":"a","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.28:6841/761183870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":761183870},{"type":"v1","addr":"172.21.15.28:6841","nonce":761183870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24581":{"gid":24581,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6839/1251999130","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6838","nonce":1251999130},{"type":"v1","addr":"172.21.15.86:6839","nonce":1251999130}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24581,"qdb_cluster":[24550,24581,24532]},"id":1}]} 2024-05-12T22:09:03.860 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 24 2024-05-12T22:09:03.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:03.861+0000 7f6552c00640 1 -- 172.21.15.28:0/4111270902 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f65340a2ea0 msgr2=0x7f65340a5360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:03.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:03.861+0000 7f6552c00640 1 --2- 172.21.15.28:0/4111270902 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f65340a2ea0 0x7f65340a5360 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f654c102720 tx=0x7f6540005f90 comp rx=0 tx=0).stop 2024-05-12T22:09:03.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:03.861+0000 7f6552c00640 1 -- 172.21.15.28:0/4111270902 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f654c1028c0 msgr2=0x7f654c073c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:03.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:03.861+0000 7f6552c00640 1 --2- 172.21.15.28:0/4111270902 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f654c1028c0 0x7f654c073c20 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f653800e9f0 tx=0x7f653800eeb0 comp rx=0 tx=0).stop 2024-05-12T22:09:03.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:03.861+0000 7f6552c00640 1 -- 172.21.15.28:0/4111270902 shutdown_connections 2024-05-12T22:09:03.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:03.861+0000 7f6552c00640 1 --2- 172.21.15.28:0/4111270902 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f65340a2ea0 0x7f65340a5360 secure :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f654c102720 tx=0x7f6540005f90 comp rx=0 tx=0).stop 2024-05-12T22:09:03.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:03.861+0000 7f6552c00640 1 --2- 172.21.15.28:0/4111270902 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f654c103280 0x7f654c074160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:03.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:03.861+0000 7f6552c00640 1 --2- 172.21.15.28:0/4111270902 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f654c1028c0 0x7f654c073c20 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:03.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:03.861+0000 7f6552c00640 1 --2- 172.21.15.28:0/4111270902 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f654c1016d0 0x7f654c0755e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:03.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:03.861+0000 7f6552c00640 1 -- 172.21.15.28:0/4111270902 >> 172.21.15.28:0/4111270902 conn(0x7f654c0fcc40 msgr2=0x7f654c105b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:09:03.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:03.861+0000 7f6552c00640 1 -- 172.21.15.28:0/4111270902 shutdown_connections 2024-05-12T22:09:03.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:03.862+0000 7f6552c00640 1 -- 172.21.15.28:0/4111270902 wait complete. 2024-05-12T22:09:03.882 INFO:tasks.fwd_scrub:Ready to start scrub thrashing 2024-05-12T22:09:03.882 INFO:tasks.fwd_scrub.ceph_manager:waiting for clean 2024-05-12T22:09:03.882 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-12T22:09:04.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:04.081+0000 7f58ab600640 1 -- 172.21.15.28:0/3009204658 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f58a410de50 msgr2=0x7f58a4110240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:04.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:04.081+0000 7f58ab600640 1 --2- 172.21.15.28:0/3009204658 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f58a410de50 0x7f58a4110240 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f589c009ca0 tx=0x7f589c02a8c0 comp rx=0 tx=0).stop 2024-05-12T22:09:04.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:04.081+0000 7f58ab600640 1 -- 172.21.15.28:0/3009204658 shutdown_connections 2024-05-12T22:09:04.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:04.081+0000 7f58ab600640 1 --2- 172.21.15.28:0/3009204658 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f58a410de50 0x7f58a4110240 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:04.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:04.081+0000 7f58ab600640 1 --2- 172.21.15.28:0/3009204658 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58a4074290 0x7f58a4074710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:04.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:04.081+0000 7f58ab600640 1 --2- 172.21.15.28:0/3009204658 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f58a4075870 0x7f58a4073cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:04.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:04.081+0000 7f58ab600640 1 -- 172.21.15.28:0/3009204658 >> 172.21.15.28:0/3009204658 conn(0x7f58a40fce10 msgr2=0x7f58a40ff230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:09:04.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:04.081+0000 7f58ab600640 1 -- 172.21.15.28:0/3009204658 shutdown_connections 2024-05-12T22:09:04.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:04.081+0000 7f58ab600640 1 -- 172.21.15.28:0/3009204658 wait complete. 2024-05-12T22:09:04.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:04.082+0000 7f58ab600640 1 Processor -- start 2024-05-12T22:09:04.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:04.082+0000 7f58ab600640 1 -- start start 2024-05-12T22:09:04.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:04.083+0000 7f58ab600640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f58a4074290 0x7f58a419e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:04.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:04.083+0000 7f58ab600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f58a4075870 0x7f58a419e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:04.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:04.083+0000 7f58ab600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58a410de50 0x7f58a41a56b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:04.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:04.083+0000 7f58ab600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f58a4113b40 con 0x7f58a410de50 2024-05-12T22:09:04.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:04.083+0000 7f58ab600640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f58a41139c0 con 0x7f58a4075870 2024-05-12T22:09:04.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:04.083+0000 7f58ab600640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f58a4113cc0 con 0x7f58a4074290 2024-05-12T22:09:04.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:04.083+0000 7f58a8c00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f58a4074290 0x7f58a419e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:04.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:04.083+0000 7f58a8c00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f58a4074290 0x7f58a419e100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:42506/0 (socket says 172.21.15.28:42506) 2024-05-12T22:09:04.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:04.083+0000 7f58a8c00640 1 -- 172.21.15.28:0/1771255121 learned_addr learned my addr 172.21.15.28:0/1771255121 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:09:04.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:04.083+0000 7f58a3e00640 1 --2- 172.21.15.28:0/1771255121 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f58a4075870 0x7f58a419e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:04.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:04.083+0000 7f58a9600640 1 --2- 172.21.15.28:0/1771255121 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58a410de50 0x7f58a41a56b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:04.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:04.083+0000 7f58a8c00640 1 -- 172.21.15.28:0/1771255121 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f58a4075870 msgr2=0x7f58a419e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:04.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:04.084+0000 7f58a8c00640 1 --2- 172.21.15.28:0/1771255121 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f58a4075870 0x7f58a419e640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:04.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:04.084+0000 7f58a8c00640 1 -- 172.21.15.28:0/1771255121 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58a410de50 msgr2=0x7f58a41a56b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:04.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:04.084+0000 7f58a8c00640 1 --2- 172.21.15.28:0/1771255121 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58a410de50 0x7f58a41a56b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:04.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:04.084+0000 7f58a8c00640 1 -- 172.21.15.28:0/1771255121 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58a41a5d20 con 0x7f58a4074290 2024-05-12T22:09:04.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:04.084+0000 7f58a3e00640 1 --2- 172.21.15.28:0/1771255121 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f58a4075870 0x7f58a419e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:09:04.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:04.084+0000 7f58a8c00640 1 --2- 172.21.15.28:0/1771255121 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f58a4074290 0x7f58a419e100 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f589800e9f0 tx=0x7f589800edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:09:04.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:04.084+0000 7f58a1600640 1 -- 172.21.15.28:0/1771255121 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f589800cd80 con 0x7f58a4074290 2024-05-12T22:09:04.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:04.084+0000 7f58ab600640 1 -- 172.21.15.28:0/1771255121 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58a41a5fb0 con 0x7f58a4074290 2024-05-12T22:09:04.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:04.084+0000 7f58a1600640 1 -- 172.21.15.28:0/1771255121 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5898005500 con 0x7f58a4074290 2024-05-12T22:09:04.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:04.084+0000 7f58a1600640 1 -- 172.21.15.28:0/1771255121 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5898010690 con 0x7f58a4074290 2024-05-12T22:09:04.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:04.084+0000 7f58ab600640 1 -- 172.21.15.28:0/1771255121 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58a41a6470 con 0x7f58a4074290 2024-05-12T22:09:04.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:04.086+0000 7f58ab600640 1 -- 172.21.15.28:0/1771255121 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f58a4103f10 con 0x7f58a4074290 2024-05-12T22:09:04.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:04.088+0000 7f58a1600640 1 -- 172.21.15.28:0/1771255121 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 25) ==== 138060+0+0 (secure 0 0 0) 0x7f58980046c0 con 0x7f58a4074290 2024-05-12T22:09:04.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:04.090+0000 7f58a1600640 1 --2- 172.21.15.28:0/1771255121 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f588c0a2ea0 0x7f588c0a5360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:04.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:04.090+0000 7f58a1600640 1 -- 172.21.15.28:0/1771255121 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f58980cfc70 con 0x7f58a4074290 2024-05-12T22:09:04.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:04.090+0000 7f58a3e00640 1 --2- 172.21.15.28:0/1771255121 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f588c0a2ea0 0x7f588c0a5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:04.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:04.091+0000 7f58a3e00640 1 --2- 172.21.15.28:0/1771255121 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f588c0a2ea0 0x7f588c0a5360 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f58a419f610 tx=0x7f5890005f90 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:09:04.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:04.092+0000 7f58a1600640 1 -- 172.21.15.28:0/1771255121 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f5898053850 con 0x7f58a4074290 2024-05-12T22:09:04.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:04.297+0000 7f58ab600640 1 -- 172.21.15.28:0/1771255121 --> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f58a4061c70 con 0x7f588c0a2ea0 2024-05-12T22:09:04.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:04.311+0000 7f58a1600640 1 -- 172.21.15.28:0/1771255121 <== mgr.25189 v2:172.21.15.28:6800/3177640985 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+257774 (secure 0 0 0) 0x7f58a4061c70 con 0x7f588c0a2ea0 2024-05-12T22:09:04.316 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-12T22:09:04.317 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-05-12T22:09:04.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:04.316+0000 7f58ab600640 1 -- 172.21.15.28:0/1771255121 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f588c0a2ea0 msgr2=0x7f588c0a5360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:04.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:04.316+0000 7f58ab600640 1 --2- 172.21.15.28:0/1771255121 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f588c0a2ea0 0x7f588c0a5360 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f58a419f610 tx=0x7f5890005f90 comp rx=0 tx=0).stop 2024-05-12T22:09:04.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:04.316+0000 7f58ab600640 1 -- 172.21.15.28:0/1771255121 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f58a4074290 msgr2=0x7f58a419e100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:04.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:04.316+0000 7f58ab600640 1 --2- 172.21.15.28:0/1771255121 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f58a4074290 0x7f58a419e100 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f589800e9f0 tx=0x7f589800edf0 comp rx=0 tx=0).stop 2024-05-12T22:09:04.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:04.316+0000 7f58ab600640 1 -- 172.21.15.28:0/1771255121 shutdown_connections 2024-05-12T22:09:04.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:04.316+0000 7f58ab600640 1 --2- 172.21.15.28:0/1771255121 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f588c0a2ea0 0x7f588c0a5360 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:04.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:04.316+0000 7f58ab600640 1 --2- 172.21.15.28:0/1771255121 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58a410de50 0x7f58a41a56b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:04.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:04.316+0000 7f58ab600640 1 --2- 172.21.15.28:0/1771255121 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f58a4075870 0x7f58a419e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:04.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:04.316+0000 7f58ab600640 1 --2- 172.21.15.28:0/1771255121 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f58a4074290 0x7f58a419e100 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:04.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:04.316+0000 7f58ab600640 1 -- 172.21.15.28:0/1771255121 >> 172.21.15.28:0/1771255121 conn(0x7f58a40fce10 msgr2=0x7f58a41010e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:09:04.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:04.316+0000 7f58ab600640 1 -- 172.21.15.28:0/1771255121 shutdown_connections 2024-05-12T22:09:04.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:04.317+0000 7f58ab600640 1 -- 172.21.15.28:0/1771255121 wait complete. 2024-05-12T22:09:04.336 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":5,"stamp":"2024-05-12T22:09:02.669434+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1188852,"num_objects":66,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":1108,"num_read_kb":715,"num_write":477,"num_write_kb":4383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":1181318,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":871,"ondisk_log_size":871,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":351624,"kb_used_data":14860,"kb_used_omap":18,"kb_used_meta":336557,"kb_avail":1124590200,"statfs":{"total":1151940427776,"available":1151580364800,"internally_reserved":0,"allocated":15216640,"data_stored":9026654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":344634681},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.975799"},"pg_stats":[{"pgid":"3.3f","version":"88'8","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:59.370219+0000","last_change":"2024-05-12T22:06:33.982079+0000","last_active":"2024-05-12T22:08:59.370219+0000","last_peered":"2024-05-12T22:08:59.370219+0000","last_clean":"2024-05-12T22:08:59.370219+0000","last_became_active":"2024-05-12T22:06:33.981703+0000","last_became_peered":"2024-05-12T22:06:33.981703+0000","last_unstale":"2024-05-12T22:08:59.370219+0000","last_undegraded":"2024-05-12T22:08:59.370219+0000","last_fullsized":"2024-05-12T22:08:59.370219+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T08:52:47.157067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"88'4","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.974850+0000","last_change":"2024-05-12T22:06:33.984194+0000","last_active":"2024-05-12T22:08:58.974850+0000","last_peered":"2024-05-12T22:08:58.974850+0000","last_clean":"2024-05-12T22:08:58.974850+0000","last_became_active":"2024-05-12T22:06:33.979787+0000","last_became_peered":"2024-05-12T22:06:33.979787+0000","last_unstale":"2024-05-12T22:08:58.974850+0000","last_undegraded":"2024-05-12T22:08:58.974850+0000","last_fullsized":"2024-05-12T22:08:58.974850+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T07:36:46.588641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"88'7","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.974018+0000","last_change":"2024-05-12T22:06:33.982428+0000","last_active":"2024-05-12T22:08:58.974018+0000","last_peered":"2024-05-12T22:08:58.974018+0000","last_clean":"2024-05-12T22:08:58.974018+0000","last_became_active":"2024-05-12T22:06:33.979563+0000","last_became_peered":"2024-05-12T22:06:33.979563+0000","last_unstale":"2024-05-12T22:08:58.974018+0000","last_undegraded":"2024-05-12T22:08:58.974018+0000","last_fullsized":"2024-05-12T22:08:58.974018+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T06:26:12.586519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"88'8","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.776473+0000","last_change":"2024-05-12T22:06:46.796152+0000","last_active":"2024-05-12T22:08:58.776473+0000","last_peered":"2024-05-12T22:08:58.776473+0000","last_clean":"2024-05-12T22:08:58.776473+0000","last_became_active":"2024-05-12T22:06:46.795484+0000","last_became_peered":"2024-05-12T22:06:46.795484+0000","last_unstale":"2024-05-12T22:08:58.776473+0000","last_undegraded":"2024-05-12T22:08:58.776473+0000","last_fullsized":"2024-05-12T22:08:58.776473+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T03:13:45.901507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,2],"acting":[9,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.3b","version":"88'4","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.778898+0000","last_change":"2024-05-12T22:06:33.967492+0000","last_active":"2024-05-12T22:08:58.778898+0000","last_peered":"2024-05-12T22:08:58.778898+0000","last_clean":"2024-05-12T22:08:58.778898+0000","last_became_active":"2024-05-12T22:06:33.967146+0000","last_became_peered":"2024-05-12T22:06:33.967146+0000","last_unstale":"2024-05-12T22:08:58.778898+0000","last_undegraded":"2024-05-12T22:08:58.778898+0000","last_fullsized":"2024-05-12T22:08:58.778898+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T07:31:05.946473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"88'6","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.649499+0000","last_change":"2024-05-12T22:06:33.973653+0000","last_active":"2024-05-12T22:08:58.649499+0000","last_peered":"2024-05-12T22:08:58.649499+0000","last_clean":"2024-05-12T22:08:58.649499+0000","last_became_active":"2024-05-12T22:06:33.973290+0000","last_became_peered":"2024-05-12T22:06:33.973290+0000","last_unstale":"2024-05-12T22:08:58.649499+0000","last_undegraded":"2024-05-12T22:08:58.649499+0000","last_fullsized":"2024-05-12T22:08:58.649499+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T03:07:21.933141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"88'5","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.967170+0000","last_change":"2024-05-12T22:06:33.973807+0000","last_active":"2024-05-12T22:08:58.967170+0000","last_peered":"2024-05-12T22:08:58.967170+0000","last_clean":"2024-05-12T22:08:58.967170+0000","last_became_active":"2024-05-12T22:06:33.973466+0000","last_became_peered":"2024-05-12T22:06:33.973466+0000","last_unstale":"2024-05-12T22:08:58.967170+0000","last_undegraded":"2024-05-12T22:08:58.967170+0000","last_fullsized":"2024-05-12T22:08:58.967170+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T02:31:13.608309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"88'9","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.648390+0000","last_change":"2024-05-12T22:06:33.967356+0000","last_active":"2024-05-12T22:08:58.648390+0000","last_peered":"2024-05-12T22:08:58.648390+0000","last_clean":"2024-05-12T22:08:58.648390+0000","last_became_active":"2024-05-12T22:06:33.966054+0000","last_became_peered":"2024-05-12T22:06:33.966054+0000","last_unstale":"2024-05-12T22:08:58.648390+0000","last_undegraded":"2024-05-12T22:08:58.648390+0000","last_fullsized":"2024-05-12T22:08:58.648390+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T23:54:08.285720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"88'11","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.981661+0000","last_change":"2024-05-12T22:06:33.968782+0000","last_active":"2024-05-12T22:08:58.981661+0000","last_peered":"2024-05-12T22:08:58.981661+0000","last_clean":"2024-05-12T22:08:58.981661+0000","last_became_active":"2024-05-12T22:06:33.968435+0000","last_became_peered":"2024-05-12T22:06:33.968435+0000","last_unstale":"2024-05-12T22:08:58.981661+0000","last_undegraded":"2024-05-12T22:08:58.981661+0000","last_fullsized":"2024-05-12T22:08:58.981661+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T07:01:21.056777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"88'16","reported_seq":36,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.649922+0000","last_change":"2024-05-12T22:06:33.962018+0000","last_active":"2024-05-12T22:08:58.649922+0000","last_peered":"2024-05-12T22:08:58.649922+0000","last_clean":"2024-05-12T22:08:58.649922+0000","last_became_active":"2024-05-12T22:06:33.961583+0000","last_became_peered":"2024-05-12T22:06:33.961583+0000","last_unstale":"2024-05-12T22:08:58.649922+0000","last_undegraded":"2024-05-12T22:08:58.649922+0000","last_fullsized":"2024-05-12T22:08:58.649922+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T00:16:40.483194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"88'6","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.967780+0000","last_change":"2024-05-12T22:06:33.965776+0000","last_active":"2024-05-12T22:08:58.967780+0000","last_peered":"2024-05-12T22:08:58.967780+0000","last_clean":"2024-05-12T22:08:58.967780+0000","last_became_active":"2024-05-12T22:06:33.965439+0000","last_became_peered":"2024-05-12T22:06:33.965439+0000","last_unstale":"2024-05-12T22:08:58.967780+0000","last_undegraded":"2024-05-12T22:08:58.967780+0000","last_fullsized":"2024-05-12T22:08:58.967780+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T23:59:56.925209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"88'7","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.966781+0000","last_change":"2024-05-12T22:06:33.967906+0000","last_active":"2024-05-12T22:08:58.966781+0000","last_peered":"2024-05-12T22:08:58.966781+0000","last_clean":"2024-05-12T22:08:58.966781+0000","last_became_active":"2024-05-12T22:06:33.967516+0000","last_became_peered":"2024-05-12T22:06:33.967516+0000","last_unstale":"2024-05-12T22:08:58.966781+0000","last_undegraded":"2024-05-12T22:08:58.966781+0000","last_fullsized":"2024-05-12T22:08:58.966781+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T08:16:56.892024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"88'10","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.968023+0000","last_change":"2024-05-12T22:06:33.971456+0000","last_active":"2024-05-12T22:08:58.968023+0000","last_peered":"2024-05-12T22:08:58.968023+0000","last_clean":"2024-05-12T22:08:58.968023+0000","last_became_active":"2024-05-12T22:06:33.971110+0000","last_became_peered":"2024-05-12T22:06:33.971110+0000","last_unstale":"2024-05-12T22:08:58.968023+0000","last_undegraded":"2024-05-12T22:08:58.968023+0000","last_fullsized":"2024-05-12T22:08:58.968023+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T09:48:56.275346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"88'7","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.973912+0000","last_change":"2024-05-12T22:06:33.983421+0000","last_active":"2024-05-12T22:08:58.973912+0000","last_peered":"2024-05-12T22:08:58.973912+0000","last_clean":"2024-05-12T22:08:58.973912+0000","last_became_active":"2024-05-12T22:06:33.978515+0000","last_became_peered":"2024-05-12T22:06:33.978515+0000","last_unstale":"2024-05-12T22:08:58.973912+0000","last_undegraded":"2024-05-12T22:08:58.973912+0000","last_fullsized":"2024-05-12T22:08:58.973912+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T23:09:48.264541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"88'7","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.974576+0000","last_change":"2024-05-12T22:06:33.980956+0000","last_active":"2024-05-12T22:08:58.974576+0000","last_peered":"2024-05-12T22:08:58.974576+0000","last_clean":"2024-05-12T22:08:58.974576+0000","last_became_active":"2024-05-12T22:06:33.979860+0000","last_became_peered":"2024-05-12T22:06:33.979860+0000","last_unstale":"2024-05-12T22:08:58.974576+0000","last_undegraded":"2024-05-12T22:08:58.974576+0000","last_fullsized":"2024-05-12T22:08:58.974576+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T02:58:29.255704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"88'9","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.981122+0000","last_change":"2024-05-12T22:06:33.978682+0000","last_active":"2024-05-12T22:08:58.981122+0000","last_peered":"2024-05-12T22:08:58.981122+0000","last_clean":"2024-05-12T22:08:58.981122+0000","last_became_active":"2024-05-12T22:06:33.978234+0000","last_became_peered":"2024-05-12T22:06:33.978234+0000","last_unstale":"2024-05-12T22:08:58.981122+0000","last_undegraded":"2024-05-12T22:08:58.981122+0000","last_fullsized":"2024-05-12T22:08:58.981122+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T07:40:48.540839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"88'8","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:59.370117+0000","last_change":"2024-05-12T22:06:33.986598+0000","last_active":"2024-05-12T22:08:59.370117+0000","last_peered":"2024-05-12T22:08:59.370117+0000","last_clean":"2024-05-12T22:08:59.370117+0000","last_became_active":"2024-05-12T22:06:33.986346+0000","last_became_peered":"2024-05-12T22:06:33.986346+0000","last_unstale":"2024-05-12T22:08:59.370117+0000","last_undegraded":"2024-05-12T22:08:59.370117+0000","last_fullsized":"2024-05-12T22:08:59.370117+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T05:52:05.473099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"88'10","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.776735+0000","last_change":"2024-05-12T22:06:33.978900+0000","last_active":"2024-05-12T22:08:58.776735+0000","last_peered":"2024-05-12T22:08:58.776735+0000","last_clean":"2024-05-12T22:08:58.776735+0000","last_became_active":"2024-05-12T22:06:33.978474+0000","last_became_peered":"2024-05-12T22:06:33.978474+0000","last_unstale":"2024-05-12T22:08:58.776735+0000","last_undegraded":"2024-05-12T22:08:58.776735+0000","last_fullsized":"2024-05-12T22:08:58.776735+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T03:47:53.014263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"88'7","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.651554+0000","last_change":"2024-05-12T22:06:33.966752+0000","last_active":"2024-05-12T22:08:58.651554+0000","last_peered":"2024-05-12T22:08:58.651554+0000","last_clean":"2024-05-12T22:08:58.651554+0000","last_became_active":"2024-05-12T22:06:33.966320+0000","last_became_peered":"2024-05-12T22:06:33.966320+0000","last_unstale":"2024-05-12T22:08:58.651554+0000","last_undegraded":"2024-05-12T22:08:58.651554+0000","last_fullsized":"2024-05-12T22:08:58.651554+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T22:11:48.356156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'9","reported_seq":48,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.776810+0000","last_change":"2024-05-12T22:06:46.808696+0000","last_active":"2024-05-12T22:08:58.776810+0000","last_peered":"2024-05-12T22:08:58.776810+0000","last_clean":"2024-05-12T22:08:58.776810+0000","last_became_active":"2024-05-12T22:06:46.796654+0000","last_became_peered":"2024-05-12T22:06:46.796654+0000","last_unstale":"2024-05-12T22:08:58.776810+0000","last_undegraded":"2024-05-12T22:08:58.776810+0000","last_fullsized":"2024-05-12T22:08:58.776810+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T09:32:33.209844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,8],"acting":[9,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"88'5","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.648988+0000","last_change":"2024-05-12T22:06:33.963875+0000","last_active":"2024-05-12T22:08:58.648988+0000","last_peered":"2024-05-12T22:08:58.648988+0000","last_clean":"2024-05-12T22:08:58.648988+0000","last_became_active":"2024-05-12T22:06:33.963239+0000","last_became_peered":"2024-05-12T22:06:33.963239+0000","last_unstale":"2024-05-12T22:08:58.648988+0000","last_undegraded":"2024-05-12T22:08:58.648988+0000","last_fullsized":"2024-05-12T22:08:58.648988+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T23:15:01.832082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.649401+0000","last_change":"2024-05-12T22:06:32.950352+0000","last_active":"2024-05-12T22:08:58.649401+0000","last_peered":"2024-05-12T22:08:58.649401+0000","last_clean":"2024-05-12T22:08:58.649401+0000","last_became_active":"2024-05-12T22:06:32.949759+0000","last_became_peered":"2024-05-12T22:06:32.949759+0000","last_unstale":"2024-05-12T22:08:58.649401+0000","last_undegraded":"2024-05-12T22:08:58.649401+0000","last_fullsized":"2024-05-12T22:08:58.649401+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T06:28:07.613351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"88'6","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.975223+0000","last_change":"2024-05-12T22:06:33.990783+0000","last_active":"2024-05-12T22:08:58.975223+0000","last_peered":"2024-05-12T22:08:58.975223+0000","last_clean":"2024-05-12T22:08:58.975223+0000","last_became_active":"2024-05-12T22:06:33.990387+0000","last_became_peered":"2024-05-12T22:06:33.990387+0000","last_unstale":"2024-05-12T22:08:58.975223+0000","last_undegraded":"2024-05-12T22:08:58.975223+0000","last_fullsized":"2024-05-12T22:08:58.975223+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T05:52:52.247458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"86'4","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.779322+0000","last_change":"2024-05-12T22:06:32.931718+0000","last_active":"2024-05-12T22:08:58.779322+0000","last_peered":"2024-05-12T22:08:58.779322+0000","last_clean":"2024-05-12T22:08:58.779322+0000","last_became_active":"2024-05-12T22:06:32.931404+0000","last_became_peered":"2024-05-12T22:06:32.931404+0000","last_unstale":"2024-05-12T22:08:58.779322+0000","last_undegraded":"2024-05-12T22:08:58.779322+0000","last_fullsized":"2024-05-12T22:08:58.779322+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T02:55:57.770504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"88'9","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.981153+0000","last_change":"2024-05-12T22:06:33.963299+0000","last_active":"2024-05-12T22:08:58.981153+0000","last_peered":"2024-05-12T22:08:58.981153+0000","last_clean":"2024-05-12T22:08:58.981153+0000","last_became_active":"2024-05-12T22:06:33.962916+0000","last_became_peered":"2024-05-12T22:06:33.962916+0000","last_unstale":"2024-05-12T22:08:58.981153+0000","last_undegraded":"2024-05-12T22:08:58.981153+0000","last_fullsized":"2024-05-12T22:08:58.981153+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T05:46:32.131918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:59.369771+0000","last_change":"2024-05-12T22:06:46.800870+0000","last_active":"2024-05-12T22:08:59.369771+0000","last_peered":"2024-05-12T22:08:59.369771+0000","last_clean":"2024-05-12T22:08:59.369771+0000","last_became_active":"2024-05-12T22:06:46.797716+0000","last_became_peered":"2024-05-12T22:06:46.797716+0000","last_unstale":"2024-05-12T22:08:59.369771+0000","last_undegraded":"2024-05-12T22:08:59.369771+0000","last_fullsized":"2024-05-12T22:08:59.369771+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T08:31:57.557564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1d","version":"88'15","reported_seq":34,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.786793+0000","last_change":"2024-05-12T22:06:33.967926+0000","last_active":"2024-05-12T22:08:58.786793+0000","last_peered":"2024-05-12T22:08:58.786793+0000","last_clean":"2024-05-12T22:08:58.786793+0000","last_became_active":"2024-05-12T22:06:33.967588+0000","last_became_peered":"2024-05-12T22:06:33.967588+0000","last_unstale":"2024-05-12T22:08:58.786793+0000","last_undegraded":"2024-05-12T22:08:58.786793+0000","last_fullsized":"2024-05-12T22:08:58.786793+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T22:39:54.434117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"89'8","reported_seq":175,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.966357+0000","last_change":"2024-05-12T22:06:32.930347+0000","last_active":"2024-05-12T22:08:58.966357+0000","last_peered":"2024-05-12T22:08:58.966357+0000","last_clean":"2024-05-12T22:08:58.966357+0000","last_became_active":"2024-05-12T22:06:32.930083+0000","last_became_peered":"2024-05-12T22:06:32.930083+0000","last_unstale":"2024-05-12T22:08:58.966357+0000","last_undegraded":"2024-05-12T22:08:58.966357+0000","last_fullsized":"2024-05-12T22:08:58.966357+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T07:10:08.898490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1545,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":145,"num_read_kb":4,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"88'7","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.778962+0000","last_change":"2024-05-12T22:06:33.973793+0000","last_active":"2024-05-12T22:08:58.778962+0000","last_peered":"2024-05-12T22:08:58.778962+0000","last_clean":"2024-05-12T22:08:58.778962+0000","last_became_active":"2024-05-12T22:06:33.973380+0000","last_became_peered":"2024-05-12T22:06:33.973380+0000","last_unstale":"2024-05-12T22:08:58.778962+0000","last_undegraded":"2024-05-12T22:08:58.778962+0000","last_fullsized":"2024-05-12T22:08:58.778962+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T04:32:01.817780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"86'3","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.649371+0000","last_change":"2024-05-12T22:06:32.938215+0000","last_active":"2024-05-12T22:08:58.649371+0000","last_peered":"2024-05-12T22:08:58.649371+0000","last_clean":"2024-05-12T22:08:58.649371+0000","last_became_active":"2024-05-12T22:06:32.937939+0000","last_became_peered":"2024-05-12T22:06:32.937939+0000","last_unstale":"2024-05-12T22:08:58.649371+0000","last_undegraded":"2024-05-12T22:08:58.649371+0000","last_fullsized":"2024-05-12T22:08:58.649371+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T04:15:28.821835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"88'8","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.786653+0000","last_change":"2024-05-12T22:06:33.977362+0000","last_active":"2024-05-12T22:08:58.786653+0000","last_peered":"2024-05-12T22:08:58.786653+0000","last_clean":"2024-05-12T22:08:58.786653+0000","last_became_active":"2024-05-12T22:06:33.977074+0000","last_became_peered":"2024-05-12T22:06:33.977074+0000","last_unstale":"2024-05-12T22:08:58.786653+0000","last_undegraded":"2024-05-12T22:08:58.786653+0000","last_fullsized":"2024-05-12T22:08:58.786653+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T22:21:00.062780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"86'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.966862+0000","last_change":"2024-05-12T22:06:32.935508+0000","last_active":"2024-05-12T22:08:58.966862+0000","last_peered":"2024-05-12T22:08:58.966862+0000","last_clean":"2024-05-12T22:08:58.966862+0000","last_became_active":"2024-05-12T22:06:32.934760+0000","last_became_peered":"2024-05-12T22:06:32.934760+0000","last_unstale":"2024-05-12T22:08:58.966862+0000","last_undegraded":"2024-05-12T22:08:58.966862+0000","last_fullsized":"2024-05-12T22:08:58.966862+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T01:58:09.450187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"88'8","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.648596+0000","last_change":"2024-05-12T22:06:33.966421+0000","last_active":"2024-05-12T22:08:58.648596+0000","last_peered":"2024-05-12T22:08:58.648596+0000","last_clean":"2024-05-12T22:08:58.648596+0000","last_became_active":"2024-05-12T22:06:33.965627+0000","last_became_peered":"2024-05-12T22:06:33.965627+0000","last_unstale":"2024-05-12T22:08:58.648596+0000","last_undegraded":"2024-05-12T22:08:58.648596+0000","last_fullsized":"2024-05-12T22:08:58.648596+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T09:25:03.836082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"86'1","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.786416+0000","last_change":"2024-05-12T22:06:32.928907+0000","last_active":"2024-05-12T22:08:58.786416+0000","last_peered":"2024-05-12T22:08:58.786416+0000","last_clean":"2024-05-12T22:08:58.786416+0000","last_became_active":"2024-05-12T22:06:32.928586+0000","last_became_peered":"2024-05-12T22:06:32.928586+0000","last_unstale":"2024-05-12T22:08:58.786416+0000","last_undegraded":"2024-05-12T22:08:58.786416+0000","last_fullsized":"2024-05-12T22:08:58.786416+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T04:24:21.658957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"88'11","reported_seq":31,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.776199+0000","last_change":"2024-05-12T22:06:33.982873+0000","last_active":"2024-05-12T22:08:58.776199+0000","last_peered":"2024-05-12T22:08:58.776199+0000","last_clean":"2024-05-12T22:08:58.776199+0000","last_became_active":"2024-05-12T22:06:33.982495+0000","last_became_peered":"2024-05-12T22:06:33.982495+0000","last_unstale":"2024-05-12T22:08:58.776199+0000","last_undegraded":"2024-05-12T22:08:58.776199+0000","last_fullsized":"2024-05-12T22:08:58.776199+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T05:45:39.404282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"89'5","reported_seq":168,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.966975+0000","last_change":"2024-05-12T22:06:32.937716+0000","last_active":"2024-05-12T22:08:58.966975+0000","last_peered":"2024-05-12T22:08:58.966975+0000","last_clean":"2024-05-12T22:08:58.966975+0000","last_became_active":"2024-05-12T22:06:32.937418+0000","last_became_peered":"2024-05-12T22:06:32.937418+0000","last_unstale":"2024-05-12T22:08:58.966975+0000","last_undegraded":"2024-05-12T22:08:58.966975+0000","last_fullsized":"2024-05-12T22:08:58.966975+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T08:20:13.639958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":141,"num_read_kb":141,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"88'10","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.981357+0000","last_change":"2024-05-12T22:06:33.974702+0000","last_active":"2024-05-12T22:08:58.981357+0000","last_peered":"2024-05-12T22:08:58.981357+0000","last_clean":"2024-05-12T22:08:58.981357+0000","last_became_active":"2024-05-12T22:06:33.974363+0000","last_became_peered":"2024-05-12T22:06:33.974363+0000","last_unstale":"2024-05-12T22:08:58.981357+0000","last_undegraded":"2024-05-12T22:08:58.981357+0000","last_fullsized":"2024-05-12T22:08:58.981357+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T09:16:33.620945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"86'2","reported_seq":34,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.967542+0000","last_change":"2024-05-12T22:06:46.793331+0000","last_active":"2024-05-12T22:08:58.967542+0000","last_peered":"2024-05-12T22:08:58.967542+0000","last_clean":"2024-05-12T22:08:58.967542+0000","last_became_active":"2024-05-12T22:06:46.789984+0000","last_became_peered":"2024-05-12T22:06:46.789984+0000","last_unstale":"2024-05-12T22:08:58.967542+0000","last_undegraded":"2024-05-12T22:08:58.967542+0000","last_fullsized":"2024-05-12T22:08:58.967542+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T08:16:29.505145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"88'9","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.975167+0000","last_change":"2024-05-12T22:06:33.980496+0000","last_active":"2024-05-12T22:08:58.975167+0000","last_peered":"2024-05-12T22:08:58.975167+0000","last_clean":"2024-05-12T22:08:58.975167+0000","last_became_active":"2024-05-12T22:06:33.980155+0000","last_became_peered":"2024-05-12T22:06:33.980155+0000","last_unstale":"2024-05-12T22:08:58.975167+0000","last_undegraded":"2024-05-12T22:08:58.975167+0000","last_fullsized":"2024-05-12T22:08:58.975167+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T01:54:03.831828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"86'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.786948+0000","last_change":"2024-05-12T22:06:32.951633+0000","last_active":"2024-05-12T22:08:58.786948+0000","last_peered":"2024-05-12T22:08:58.786948+0000","last_clean":"2024-05-12T22:08:58.786948+0000","last_became_active":"2024-05-12T22:06:32.951382+0000","last_became_peered":"2024-05-12T22:06:32.951382+0000","last_unstale":"2024-05-12T22:08:58.786948+0000","last_undegraded":"2024-05-12T22:08:58.786948+0000","last_fullsized":"2024-05-12T22:08:58.786948+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T02:06:21.408593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"88'11","reported_seq":31,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.981428+0000","last_change":"2024-05-12T22:06:33.963493+0000","last_active":"2024-05-12T22:08:58.981428+0000","last_peered":"2024-05-12T22:08:58.981428+0000","last_clean":"2024-05-12T22:08:58.981428+0000","last_became_active":"2024-05-12T22:06:33.963127+0000","last_became_peered":"2024-05-12T22:06:33.963127+0000","last_unstale":"2024-05-12T22:08:58.981428+0000","last_undegraded":"2024-05-12T22:08:58.981428+0000","last_fullsized":"2024-05-12T22:08:58.981428+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T05:39:31.038446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"89'38","reported_seq":353,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.982668+0000","last_change":"2024-05-12T22:06:32.932026+0000","last_active":"2024-05-12T22:08:58.982668+0000","last_peered":"2024-05-12T22:08:58.982668+0000","last_clean":"2024-05-12T22:08:58.982668+0000","last_became_active":"2024-05-12T22:06:32.931667+0000","last_became_peered":"2024-05-12T22:06:32.931667+0000","last_unstale":"2024-05-12T22:08:58.982668+0000","last_undegraded":"2024-05-12T22:08:58.982668+0000","last_fullsized":"2024-05-12T22:08:58.982668+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T06:16:28.259047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67587,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":293,"num_read_kb":212,"num_write":39,"num_write_kb":85,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"88'6","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.779148+0000","last_change":"2024-05-12T22:06:33.962677+0000","last_active":"2024-05-12T22:08:58.779148+0000","last_peered":"2024-05-12T22:08:58.779148+0000","last_clean":"2024-05-12T22:08:58.779148+0000","last_became_active":"2024-05-12T22:06:33.962295+0000","last_became_peered":"2024-05-12T22:06:33.962295+0000","last_unstale":"2024-05-12T22:08:58.779148+0000","last_undegraded":"2024-05-12T22:08:58.779148+0000","last_fullsized":"2024-05-12T22:08:58.779148+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T01:31:04.587046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"86'6","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.651259+0000","last_change":"2024-05-12T22:06:32.936104+0000","last_active":"2024-05-12T22:08:58.651259+0000","last_peered":"2024-05-12T22:08:58.651259+0000","last_clean":"2024-05-12T22:08:58.651259+0000","last_became_active":"2024-05-12T22:06:32.935701+0000","last_became_peered":"2024-05-12T22:06:32.935701+0000","last_unstale":"2024-05-12T22:08:58.651259+0000","last_undegraded":"2024-05-12T22:08:58.651259+0000","last_fullsized":"2024-05-12T22:08:58.651259+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T07:04:10.115302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"88'9","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.967648+0000","last_change":"2024-05-12T22:06:33.977850+0000","last_active":"2024-05-12T22:08:58.967648+0000","last_peered":"2024-05-12T22:08:58.967648+0000","last_clean":"2024-05-12T22:08:58.967648+0000","last_became_active":"2024-05-12T22:06:33.977511+0000","last_became_peered":"2024-05-12T22:06:33.977511+0000","last_unstale":"2024-05-12T22:08:58.967648+0000","last_undegraded":"2024-05-12T22:08:58.967648+0000","last_fullsized":"2024-05-12T22:08:58.967648+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T23:10:17.267823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"86'3","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.786663+0000","last_change":"2024-05-12T22:06:32.929955+0000","last_active":"2024-05-12T22:08:58.786663+0000","last_peered":"2024-05-12T22:08:58.786663+0000","last_clean":"2024-05-12T22:08:58.786663+0000","last_became_active":"2024-05-12T22:06:32.929623+0000","last_became_peered":"2024-05-12T22:06:32.929623+0000","last_unstale":"2024-05-12T22:08:58.786663+0000","last_undegraded":"2024-05-12T22:08:58.786663+0000","last_fullsized":"2024-05-12T22:08:58.786663+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T08:15:22.622908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"88'15","reported_seq":41,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.967533+0000","last_change":"2024-05-12T22:06:46.931736+0000","last_active":"2024-05-12T22:08:58.967533+0000","last_peered":"2024-05-12T22:08:58.967533+0000","last_clean":"2024-05-12T22:08:58.967533+0000","last_became_active":"2024-05-12T22:06:46.931265+0000","last_became_peered":"2024-05-12T22:06:46.931265+0000","last_unstale":"2024-05-12T22:08:58.967533+0000","last_undegraded":"2024-05-12T22:08:58.967533+0000","last_fullsized":"2024-05-12T22:08:58.967533+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T02:25:34.002024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"89'7","reported_seq":175,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:59.010336+0000","last_change":"2024-05-12T22:06:32.927707+0000","last_active":"2024-05-12T22:08:59.010336+0000","last_peered":"2024-05-12T22:08:59.010336+0000","last_clean":"2024-05-12T22:08:59.010336+0000","last_became_active":"2024-05-12T22:06:32.927406+0000","last_became_peered":"2024-05-12T22:06:32.927406+0000","last_unstale":"2024-05-12T22:08:59.010336+0000","last_undegraded":"2024-05-12T22:08:59.010336+0000","last_fullsized":"2024-05-12T22:08:59.010336+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T05:14:34.268589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1902,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":147,"num_read_kb":6,"num_write":6,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"88'6","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.966849+0000","last_change":"2024-05-12T22:06:33.964996+0000","last_active":"2024-05-12T22:08:58.966849+0000","last_peered":"2024-05-12T22:08:58.966849+0000","last_clean":"2024-05-12T22:08:58.966849+0000","last_became_active":"2024-05-12T22:06:33.964587+0000","last_became_peered":"2024-05-12T22:06:33.964587+0000","last_unstale":"2024-05-12T22:08:58.966849+0000","last_undegraded":"2024-05-12T22:08:58.966849+0000","last_fullsized":"2024-05-12T22:08:58.966849+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T22:27:28.316377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"86'15","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.778916+0000","last_change":"2024-05-12T22:06:32.935543+0000","last_active":"2024-05-12T22:08:58.778916+0000","last_peered":"2024-05-12T22:08:58.778916+0000","last_clean":"2024-05-12T22:08:58.778916+0000","last_became_active":"2024-05-12T22:06:32.935145+0000","last_became_peered":"2024-05-12T22:06:32.935145+0000","last_unstale":"2024-05-12T22:08:58.778916+0000","last_undegraded":"2024-05-12T22:08:58.778916+0000","last_fullsized":"2024-05-12T22:08:58.778916+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T01:13:23.289057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"88'6","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.786436+0000","last_change":"2024-05-12T22:06:33.964367+0000","last_active":"2024-05-12T22:08:58.786436+0000","last_peered":"2024-05-12T22:08:58.786436+0000","last_clean":"2024-05-12T22:08:58.786436+0000","last_became_active":"2024-05-12T22:06:33.963876+0000","last_became_peered":"2024-05-12T22:06:33.963876+0000","last_unstale":"2024-05-12T22:08:58.786436+0000","last_undegraded":"2024-05-12T22:08:58.786436+0000","last_fullsized":"2024-05-12T22:08:58.786436+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T07:09:20.962965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:59.356563+0000","last_change":"2024-05-12T22:06:32.927270+0000","last_active":"2024-05-12T22:08:59.356563+0000","last_peered":"2024-05-12T22:08:59.356563+0000","last_clean":"2024-05-12T22:08:59.356563+0000","last_became_active":"2024-05-12T22:06:32.926930+0000","last_became_peered":"2024-05-12T22:06:32.926930+0000","last_unstale":"2024-05-12T22:08:59.356563+0000","last_undegraded":"2024-05-12T22:08:59.356563+0000","last_fullsized":"2024-05-12T22:08:59.356563+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T07:04:48.336599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"88'5","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:59.369986+0000","last_change":"2024-05-12T22:06:33.966350+0000","last_active":"2024-05-12T22:08:59.369986+0000","last_peered":"2024-05-12T22:08:59.369986+0000","last_clean":"2024-05-12T22:08:59.369986+0000","last_became_active":"2024-05-12T22:06:33.965974+0000","last_became_peered":"2024-05-12T22:06:33.965974+0000","last_unstale":"2024-05-12T22:08:59.369986+0000","last_undegraded":"2024-05-12T22:08:59.369986+0000","last_fullsized":"2024-05-12T22:08:59.369986+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T02:08:16.889684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.776349+0000","last_change":"2024-05-12T22:06:32.944945+0000","last_active":"2024-05-12T22:08:58.776349+0000","last_peered":"2024-05-12T22:08:58.776349+0000","last_clean":"2024-05-12T22:08:58.776349+0000","last_became_active":"2024-05-12T22:06:32.944576+0000","last_became_peered":"2024-05-12T22:06:32.944576+0000","last_unstale":"2024-05-12T22:08:58.776349+0000","last_undegraded":"2024-05-12T22:08:58.776349+0000","last_fullsized":"2024-05-12T22:08:58.776349+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T07:29:14.544821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"88'5","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.649553+0000","last_change":"2024-05-12T22:06:33.966299+0000","last_active":"2024-05-12T22:08:58.649553+0000","last_peered":"2024-05-12T22:08:58.649553+0000","last_clean":"2024-05-12T22:08:58.649553+0000","last_became_active":"2024-05-12T22:06:33.965902+0000","last_became_peered":"2024-05-12T22:06:33.965902+0000","last_unstale":"2024-05-12T22:08:58.649553+0000","last_undegraded":"2024-05-12T22:08:58.649553+0000","last_fullsized":"2024-05-12T22:08:58.649553+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T22:37:51.476546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"86'1","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.967232+0000","last_change":"2024-05-12T22:06:32.937210+0000","last_active":"2024-05-12T22:08:58.967232+0000","last_peered":"2024-05-12T22:08:58.967232+0000","last_clean":"2024-05-12T22:08:58.967232+0000","last_became_active":"2024-05-12T22:06:32.936805+0000","last_became_peered":"2024-05-12T22:06:32.936805+0000","last_unstale":"2024-05-12T22:08:58.967232+0000","last_undegraded":"2024-05-12T22:08:58.967232+0000","last_fullsized":"2024-05-12T22:08:58.967232+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T06:14:20.825258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"88'7","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.649737+0000","last_change":"2024-05-12T22:06:33.983604+0000","last_active":"2024-05-12T22:08:58.649737+0000","last_peered":"2024-05-12T22:08:58.649737+0000","last_clean":"2024-05-12T22:08:58.649737+0000","last_became_active":"2024-05-12T22:06:33.983168+0000","last_became_peered":"2024-05-12T22:06:33.983168+0000","last_unstale":"2024-05-12T22:08:58.649737+0000","last_undegraded":"2024-05-12T22:08:58.649737+0000","last_fullsized":"2024-05-12T22:08:58.649737+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T05:54:55.854593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"88'8","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.786490+0000","last_change":"2024-05-12T22:06:33.968848+0000","last_active":"2024-05-12T22:08:58.786490+0000","last_peered":"2024-05-12T22:08:58.786490+0000","last_clean":"2024-05-12T22:08:58.786490+0000","last_became_active":"2024-05-12T22:06:33.968529+0000","last_became_peered":"2024-05-12T22:06:33.968529+0000","last_unstale":"2024-05-12T22:08:58.786490+0000","last_undegraded":"2024-05-12T22:08:58.786490+0000","last_fullsized":"2024-05-12T22:08:58.786490+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T03:00:41.129662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'6","reported_seq":169,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.974762+0000","last_change":"2024-05-12T22:06:32.946670+0000","last_active":"2024-05-12T22:08:58.974762+0000","last_peered":"2024-05-12T22:08:58.974762+0000","last_clean":"2024-05-12T22:08:58.974762+0000","last_became_active":"2024-05-12T22:06:32.946059+0000","last_became_peered":"2024-05-12T22:06:32.946059+0000","last_unstale":"2024-05-12T22:08:58.974762+0000","last_undegraded":"2024-05-12T22:08:58.974762+0000","last_fullsized":"2024-05-12T22:08:58.974762+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T22:35:56.116979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":141,"num_read_kb":141,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"88'8","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.648638+0000","last_change":"2024-05-12T22:06:33.962177+0000","last_active":"2024-05-12T22:08:58.648638+0000","last_peered":"2024-05-12T22:08:58.648638+0000","last_clean":"2024-05-12T22:08:58.648638+0000","last_became_active":"2024-05-12T22:06:33.960919+0000","last_became_peered":"2024-05-12T22:06:33.960919+0000","last_unstale":"2024-05-12T22:08:58.648638+0000","last_undegraded":"2024-05-12T22:08:58.648638+0000","last_fullsized":"2024-05-12T22:08:58.648638+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T00:39:54.566460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'3","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.966691+0000","last_change":"2024-05-12T22:06:32.927885+0000","last_active":"2024-05-12T22:08:58.966691+0000","last_peered":"2024-05-12T22:08:58.966691+0000","last_clean":"2024-05-12T22:08:58.966691+0000","last_became_active":"2024-05-12T22:06:32.927567+0000","last_became_peered":"2024-05-12T22:06:32.927567+0000","last_unstale":"2024-05-12T22:08:58.966691+0000","last_undegraded":"2024-05-12T22:08:58.966691+0000","last_fullsized":"2024-05-12T22:08:58.966691+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T07:13:50.811992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":355,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.966894+0000","last_change":"2024-05-12T22:05:17.683363+0000","last_active":"2024-05-12T22:08:58.966894+0000","last_peered":"2024-05-12T22:08:58.966894+0000","last_clean":"2024-05-12T22:08:58.966894+0000","last_became_active":"2024-05-12T22:05:17.682824+0000","last_became_peered":"2024-05-12T22:05:17.682824+0000","last_unstale":"2024-05-12T22:08:58.966894+0000","last_undegraded":"2024-05-12T22:08:58.966894+0000","last_fullsized":"2024-05-12T22:08:58.966894+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:01:32.304714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:01:32.304714+0000","last_clean_scrub_stamp":"2024-05-12T22:01:32.304714+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T06:35:22.474820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"88'5","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.966617+0000","last_change":"2024-05-12T22:06:33.965332+0000","last_active":"2024-05-12T22:08:58.966617+0000","last_peered":"2024-05-12T22:08:58.966617+0000","last_clean":"2024-05-12T22:08:58.966617+0000","last_became_active":"2024-05-12T22:06:33.964858+0000","last_became_peered":"2024-05-12T22:06:33.964858+0000","last_unstale":"2024-05-12T22:08:58.966617+0000","last_undegraded":"2024-05-12T22:08:58.966617+0000","last_fullsized":"2024-05-12T22:08:58.966617+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T05:14:09.825614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'3","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.967071+0000","last_change":"2024-05-12T22:06:32.929773+0000","last_active":"2024-05-12T22:08:58.967071+0000","last_peered":"2024-05-12T22:08:58.967071+0000","last_clean":"2024-05-12T22:08:58.967071+0000","last_became_active":"2024-05-12T22:06:32.929458+0000","last_became_peered":"2024-05-12T22:06:32.929458+0000","last_unstale":"2024-05-12T22:08:58.967071+0000","last_undegraded":"2024-05-12T22:08:58.967071+0000","last_fullsized":"2024-05-12T22:08:58.967071+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T01:30:42.853183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"86'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.779092+0000","last_change":"2024-05-12T22:06:32.947014+0000","last_active":"2024-05-12T22:08:58.779092+0000","last_peered":"2024-05-12T22:08:58.779092+0000","last_clean":"2024-05-12T22:08:58.779092+0000","last_became_active":"2024-05-12T22:06:32.946661+0000","last_became_peered":"2024-05-12T22:06:32.946661+0000","last_unstale":"2024-05-12T22:08:58.779092+0000","last_undegraded":"2024-05-12T22:08:58.779092+0000","last_fullsized":"2024-05-12T22:08:58.779092+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T23:09:19.378353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"88'9","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.967616+0000","last_change":"2024-05-12T22:06:33.980490+0000","last_active":"2024-05-12T22:08:58.967616+0000","last_peered":"2024-05-12T22:08:58.967616+0000","last_clean":"2024-05-12T22:08:58.967616+0000","last_became_active":"2024-05-12T22:06:33.980133+0000","last_became_peered":"2024-05-12T22:06:33.980133+0000","last_unstale":"2024-05-12T22:08:58.967616+0000","last_undegraded":"2024-05-12T22:08:58.967616+0000","last_fullsized":"2024-05-12T22:08:58.967616+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T00:28:10.461428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"89'46","reported_seq":91,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.918040+0000","last_change":"2024-05-12T22:06:32.943152+0000","last_active":"2024-05-12T22:08:58.918040+0000","last_peered":"2024-05-12T22:08:58.918040+0000","last_clean":"2024-05-12T22:08:58.918040+0000","last_became_active":"2024-05-12T22:06:32.942769+0000","last_became_peered":"2024-05-12T22:06:32.942769+0000","last_unstale":"2024-05-12T22:08:58.918040+0000","last_undegraded":"2024-05-12T22:08:58.918040+0000","last_fullsized":"2024-05-12T22:08:58.918040+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T01:23:24.955739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":82,"num_write_kb":837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"88'5","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.649177+0000","last_change":"2024-05-12T22:06:33.964081+0000","last_active":"2024-05-12T22:08:58.649177+0000","last_peered":"2024-05-12T22:08:58.649177+0000","last_clean":"2024-05-12T22:08:58.649177+0000","last_became_active":"2024-05-12T22:06:33.963054+0000","last_became_peered":"2024-05-12T22:06:33.963054+0000","last_unstale":"2024-05-12T22:08:58.649177+0000","last_undegraded":"2024-05-12T22:08:58.649177+0000","last_fullsized":"2024-05-12T22:08:58.649177+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T09:40:03.396200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"86'4","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.966937+0000","last_change":"2024-05-12T22:06:32.936432+0000","last_active":"2024-05-12T22:08:58.966937+0000","last_peered":"2024-05-12T22:08:58.966937+0000","last_clean":"2024-05-12T22:08:58.966937+0000","last_became_active":"2024-05-12T22:06:32.936094+0000","last_became_peered":"2024-05-12T22:06:32.936094+0000","last_unstale":"2024-05-12T22:08:58.966937+0000","last_undegraded":"2024-05-12T22:08:58.966937+0000","last_fullsized":"2024-05-12T22:08:58.966937+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T02:40:31.346118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"88'7","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.974922+0000","last_change":"2024-05-12T22:06:33.985953+0000","last_active":"2024-05-12T22:08:58.974922+0000","last_peered":"2024-05-12T22:08:58.974922+0000","last_clean":"2024-05-12T22:08:58.974922+0000","last_became_active":"2024-05-12T22:06:33.984864+0000","last_became_peered":"2024-05-12T22:06:33.984864+0000","last_unstale":"2024-05-12T22:08:58.974922+0000","last_undegraded":"2024-05-12T22:08:58.974922+0000","last_fullsized":"2024-05-12T22:08:58.974922+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T07:28:22.357841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.981163+0000","last_change":"2024-05-12T22:06:32.934510+0000","last_active":"2024-05-12T22:08:58.981163+0000","last_peered":"2024-05-12T22:08:58.981163+0000","last_clean":"2024-05-12T22:08:58.981163+0000","last_became_active":"2024-05-12T22:06:32.934222+0000","last_became_peered":"2024-05-12T22:06:32.934222+0000","last_unstale":"2024-05-12T22:08:58.981163+0000","last_undegraded":"2024-05-12T22:08:58.981163+0000","last_fullsized":"2024-05-12T22:08:58.981163+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T00:57:44.321629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"88'2","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.778837+0000","last_change":"2024-05-12T22:06:33.963870+0000","last_active":"2024-05-12T22:08:58.778837+0000","last_peered":"2024-05-12T22:08:58.778837+0000","last_clean":"2024-05-12T22:08:58.778837+0000","last_became_active":"2024-05-12T22:06:33.963533+0000","last_became_peered":"2024-05-12T22:06:33.963533+0000","last_unstale":"2024-05-12T22:08:58.778837+0000","last_undegraded":"2024-05-12T22:08:58.778837+0000","last_fullsized":"2024-05-12T22:08:58.778837+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T08:56:59.495469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"86'3","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.648478+0000","last_change":"2024-05-12T22:06:32.939738+0000","last_active":"2024-05-12T22:08:58.648478+0000","last_peered":"2024-05-12T22:08:58.648478+0000","last_clean":"2024-05-12T22:08:58.648478+0000","last_became_active":"2024-05-12T22:06:32.939365+0000","last_became_peered":"2024-05-12T22:06:32.939365+0000","last_unstale":"2024-05-12T22:08:58.648478+0000","last_undegraded":"2024-05-12T22:08:58.648478+0000","last_fullsized":"2024-05-12T22:08:58.648478+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T02:19:31.180834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"88'9","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:59.369732+0000","last_change":"2024-05-12T22:06:33.975064+0000","last_active":"2024-05-12T22:08:59.369732+0000","last_peered":"2024-05-12T22:08:59.369732+0000","last_clean":"2024-05-12T22:08:59.369732+0000","last_became_active":"2024-05-12T22:06:33.974613+0000","last_became_peered":"2024-05-12T22:06:33.974613+0000","last_unstale":"2024-05-12T22:08:59.369732+0000","last_undegraded":"2024-05-12T22:08:59.369732+0000","last_fullsized":"2024-05-12T22:08:59.369732+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T08:46:27.343777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"86'1","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.651327+0000","last_change":"2024-05-12T22:06:32.931148+0000","last_active":"2024-05-12T22:08:58.651327+0000","last_peered":"2024-05-12T22:08:58.651327+0000","last_clean":"2024-05-12T22:08:58.651327+0000","last_became_active":"2024-05-12T22:06:32.930717+0000","last_became_peered":"2024-05-12T22:06:32.930717+0000","last_unstale":"2024-05-12T22:08:58.651327+0000","last_undegraded":"2024-05-12T22:08:58.651327+0000","last_fullsized":"2024-05-12T22:08:58.651327+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T07:26:20.915467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"88'8","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.648890+0000","last_change":"2024-05-12T22:06:33.988390+0000","last_active":"2024-05-12T22:08:58.648890+0000","last_peered":"2024-05-12T22:08:58.648890+0000","last_clean":"2024-05-12T22:08:58.648890+0000","last_became_active":"2024-05-12T22:06:33.986970+0000","last_became_peered":"2024-05-12T22:06:33.986970+0000","last_unstale":"2024-05-12T22:08:58.648890+0000","last_undegraded":"2024-05-12T22:08:58.648890+0000","last_fullsized":"2024-05-12T22:08:58.648890+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T05:03:31.930261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.779155+0000","last_change":"2024-05-12T22:06:46.798123+0000","last_active":"2024-05-12T22:08:58.779155+0000","last_peered":"2024-05-12T22:08:58.779155+0000","last_clean":"2024-05-12T22:08:58.779155+0000","last_became_active":"2024-05-12T22:06:46.791029+0000","last_became_peered":"2024-05-12T22:06:46.791029+0000","last_unstale":"2024-05-12T22:08:58.779155+0000","last_undegraded":"2024-05-12T22:08:58.779155+0000","last_fullsized":"2024-05-12T22:08:58.779155+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T06:31:25.108659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,7],"acting":[8,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"88'5","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.967282+0000","last_change":"2024-05-12T22:06:33.968256+0000","last_active":"2024-05-12T22:08:58.967282+0000","last_peered":"2024-05-12T22:08:58.967282+0000","last_clean":"2024-05-12T22:08:58.967282+0000","last_became_active":"2024-05-12T22:06:33.967885+0000","last_became_peered":"2024-05-12T22:06:33.967885+0000","last_unstale":"2024-05-12T22:08:58.967282+0000","last_undegraded":"2024-05-12T22:08:58.967282+0000","last_fullsized":"2024-05-12T22:08:58.967282+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T09:12:40.687367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"86'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.651789+0000","last_change":"2024-05-12T22:06:32.944736+0000","last_active":"2024-05-12T22:08:58.651789+0000","last_peered":"2024-05-12T22:08:58.651789+0000","last_clean":"2024-05-12T22:08:58.651789+0000","last_became_active":"2024-05-12T22:06:32.944269+0000","last_became_peered":"2024-05-12T22:06:32.944269+0000","last_unstale":"2024-05-12T22:08:58.651789+0000","last_undegraded":"2024-05-12T22:08:58.651789+0000","last_fullsized":"2024-05-12T22:08:58.651789+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T22:56:24.391267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"88'7","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.967821+0000","last_change":"2024-05-12T22:06:33.968360+0000","last_active":"2024-05-12T22:08:58.967821+0000","last_peered":"2024-05-12T22:08:58.967821+0000","last_clean":"2024-05-12T22:08:58.967821+0000","last_became_active":"2024-05-12T22:06:33.967967+0000","last_became_peered":"2024-05-12T22:06:33.967967+0000","last_unstale":"2024-05-12T22:08:58.967821+0000","last_undegraded":"2024-05-12T22:08:58.967821+0000","last_fullsized":"2024-05-12T22:08:58.967821+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T00:41:12.693858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"89'39","reported_seq":69,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.914994+0000","last_change":"2024-05-12T22:06:32.935993+0000","last_active":"2024-05-12T22:08:58.914994+0000","last_peered":"2024-05-12T22:08:58.914994+0000","last_clean":"2024-05-12T22:08:58.914994+0000","last_became_active":"2024-05-12T22:06:32.935615+0000","last_became_peered":"2024-05-12T22:06:32.935615+0000","last_unstale":"2024-05-12T22:08:58.914994+0000","last_undegraded":"2024-05-12T22:08:58.914994+0000","last_fullsized":"2024-05-12T22:08:58.914994+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T00:43:39.254637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"88'7","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.967474+0000","last_change":"2024-05-12T22:06:33.966110+0000","last_active":"2024-05-12T22:08:58.967474+0000","last_peered":"2024-05-12T22:08:58.967474+0000","last_clean":"2024-05-12T22:08:58.967474+0000","last_became_active":"2024-05-12T22:06:33.965755+0000","last_became_peered":"2024-05-12T22:06:33.965755+0000","last_unstale":"2024-05-12T22:08:58.967474+0000","last_undegraded":"2024-05-12T22:08:58.967474+0000","last_fullsized":"2024-05-12T22:08:58.967474+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T22:18:53.941335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.968104+0000","last_change":"2024-05-12T22:06:32.942229+0000","last_active":"2024-05-12T22:08:58.968104+0000","last_peered":"2024-05-12T22:08:58.968104+0000","last_clean":"2024-05-12T22:08:58.968104+0000","last_became_active":"2024-05-12T22:06:32.941826+0000","last_became_peered":"2024-05-12T22:06:32.941826+0000","last_unstale":"2024-05-12T22:08:58.968104+0000","last_undegraded":"2024-05-12T22:08:58.968104+0000","last_fullsized":"2024-05-12T22:08:58.968104+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T03:52:34.886318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"88'9","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.967759+0000","last_change":"2024-05-12T22:06:33.963081+0000","last_active":"2024-05-12T22:08:58.967759+0000","last_peered":"2024-05-12T22:08:58.967759+0000","last_clean":"2024-05-12T22:08:58.967759+0000","last_became_active":"2024-05-12T22:06:33.962696+0000","last_became_peered":"2024-05-12T22:06:33.962696+0000","last_unstale":"2024-05-12T22:08:58.967759+0000","last_undegraded":"2024-05-12T22:08:58.967759+0000","last_fullsized":"2024-05-12T22:08:58.967759+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T05:55:45.581823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"88'9","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:59.369801+0000","last_change":"2024-05-12T22:06:33.968865+0000","last_active":"2024-05-12T22:08:59.369801+0000","last_peered":"2024-05-12T22:08:59.369801+0000","last_clean":"2024-05-12T22:08:59.369801+0000","last_became_active":"2024-05-12T22:06:33.968185+0000","last_became_peered":"2024-05-12T22:06:33.968185+0000","last_unstale":"2024-05-12T22:08:59.369801+0000","last_undegraded":"2024-05-12T22:08:59.369801+0000","last_fullsized":"2024-05-12T22:08:59.369801+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T01:23:41.083778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"88'12","reported_seq":32,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.651171+0000","last_change":"2024-05-12T22:06:33.964786+0000","last_active":"2024-05-12T22:08:58.651171+0000","last_peered":"2024-05-12T22:08:58.651171+0000","last_clean":"2024-05-12T22:08:58.651171+0000","last_became_active":"2024-05-12T22:06:33.964227+0000","last_became_peered":"2024-05-12T22:06:33.964227+0000","last_unstale":"2024-05-12T22:08:58.651171+0000","last_undegraded":"2024-05-12T22:08:58.651171+0000","last_fullsized":"2024-05-12T22:08:58.651171+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T02:53:42.739455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"88'8","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.651132+0000","last_change":"2024-05-12T22:06:33.978857+0000","last_active":"2024-05-12T22:08:58.651132+0000","last_peered":"2024-05-12T22:08:58.651132+0000","last_clean":"2024-05-12T22:08:58.651132+0000","last_became_active":"2024-05-12T22:06:33.978531+0000","last_became_peered":"2024-05-12T22:06:33.978531+0000","last_unstale":"2024-05-12T22:08:58.651132+0000","last_undegraded":"2024-05-12T22:08:58.651132+0000","last_fullsized":"2024-05-12T22:08:58.651132+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T02:51:05.609267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"88'7","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.974185+0000","last_change":"2024-05-12T22:06:33.980662+0000","last_active":"2024-05-12T22:08:58.974185+0000","last_peered":"2024-05-12T22:08:58.974185+0000","last_clean":"2024-05-12T22:08:58.974185+0000","last_became_active":"2024-05-12T22:06:33.980332+0000","last_became_peered":"2024-05-12T22:06:33.980332+0000","last_unstale":"2024-05-12T22:08:58.974185+0000","last_undegraded":"2024-05-12T22:08:58.974185+0000","last_fullsized":"2024-05-12T22:08:58.974185+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T06:21:59.715792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"88'4","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.967687+0000","last_change":"2024-05-12T22:06:33.987338+0000","last_active":"2024-05-12T22:08:58.967687+0000","last_peered":"2024-05-12T22:08:58.967687+0000","last_clean":"2024-05-12T22:08:58.967687+0000","last_became_active":"2024-05-12T22:06:33.986942+0000","last_became_peered":"2024-05-12T22:06:33.986942+0000","last_unstale":"2024-05-12T22:08:58.967687+0000","last_undegraded":"2024-05-12T22:08:58.967687+0000","last_fullsized":"2024-05-12T22:08:58.967687+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T01:02:05.137558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"88'6","reported_seq":32,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:59.370006+0000","last_change":"2024-05-12T22:06:46.796752+0000","last_active":"2024-05-12T22:08:59.370006+0000","last_peered":"2024-05-12T22:08:59.370006+0000","last_clean":"2024-05-12T22:08:59.370006+0000","last_became_active":"2024-05-12T22:06:46.796369+0000","last_became_peered":"2024-05-12T22:06:46.796369+0000","last_unstale":"2024-05-12T22:08:59.370006+0000","last_undegraded":"2024-05-12T22:08:59.370006+0000","last_fullsized":"2024-05-12T22:08:59.370006+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T08:23:24.685495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,9],"acting":[11,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"88'15","reported_seq":35,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.651395+0000","last_change":"2024-05-12T22:06:33.963902+0000","last_active":"2024-05-12T22:08:58.651395+0000","last_peered":"2024-05-12T22:08:58.651395+0000","last_clean":"2024-05-12T22:08:58.651395+0000","last_became_active":"2024-05-12T22:06:33.963436+0000","last_became_peered":"2024-05-12T22:06:33.963436+0000","last_unstale":"2024-05-12T22:08:58.651395+0000","last_undegraded":"2024-05-12T22:08:58.651395+0000","last_fullsized":"2024-05-12T22:08:58.651395+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T02:51:56.415167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"88'2","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.649008+0000","last_change":"2024-05-12T22:06:46.794434+0000","last_active":"2024-05-12T22:08:58.649008+0000","last_peered":"2024-05-12T22:08:58.649008+0000","last_clean":"2024-05-12T22:08:58.649008+0000","last_became_active":"2024-05-12T22:06:46.793147+0000","last_became_peered":"2024-05-12T22:06:46.793147+0000","last_unstale":"2024-05-12T22:08:58.649008+0000","last_undegraded":"2024-05-12T22:08:58.649008+0000","last_fullsized":"2024-05-12T22:08:58.649008+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T23:10:39.529863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,8,2],"acting":[3,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"88'6","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.786430+0000","last_change":"2024-05-12T22:06:33.973490+0000","last_active":"2024-05-12T22:08:58.786430+0000","last_peered":"2024-05-12T22:08:58.786430+0000","last_clean":"2024-05-12T22:08:58.786430+0000","last_became_active":"2024-05-12T22:06:33.973161+0000","last_became_peered":"2024-05-12T22:06:33.973161+0000","last_unstale":"2024-05-12T22:08:58.786430+0000","last_undegraded":"2024-05-12T22:08:58.786430+0000","last_fullsized":"2024-05-12T22:08:58.786430+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T03:29:36.972631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"88'8","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.648590+0000","last_change":"2024-05-12T22:06:33.957320+0000","last_active":"2024-05-12T22:08:58.648590+0000","last_peered":"2024-05-12T22:08:58.648590+0000","last_clean":"2024-05-12T22:08:58.648590+0000","last_became_active":"2024-05-12T22:06:33.956837+0000","last_became_peered":"2024-05-12T22:06:33.956837+0000","last_unstale":"2024-05-12T22:08:58.648590+0000","last_undegraded":"2024-05-12T22:08:58.648590+0000","last_fullsized":"2024-05-12T22:08:58.648590+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T01:20:51.018175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"88'10","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.981497+0000","last_change":"2024-05-12T22:06:33.968363+0000","last_active":"2024-05-12T22:08:58.981497+0000","last_peered":"2024-05-12T22:08:58.981497+0000","last_clean":"2024-05-12T22:08:58.981497+0000","last_became_active":"2024-05-12T22:06:33.968031+0000","last_became_peered":"2024-05-12T22:06:33.968031+0000","last_unstale":"2024-05-12T22:08:58.981497+0000","last_undegraded":"2024-05-12T22:08:58.981497+0000","last_fullsized":"2024-05-12T22:08:58.981497+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T00:14:07.124058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"88'11","reported_seq":31,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.967933+0000","last_change":"2024-05-12T22:06:33.984823+0000","last_active":"2024-05-12T22:08:58.967933+0000","last_peered":"2024-05-12T22:08:58.967933+0000","last_clean":"2024-05-12T22:08:58.967933+0000","last_became_active":"2024-05-12T22:06:33.984430+0000","last_became_peered":"2024-05-12T22:06:33.984430+0000","last_unstale":"2024-05-12T22:08:58.967933+0000","last_undegraded":"2024-05-12T22:08:58.967933+0000","last_fullsized":"2024-05-12T22:08:58.967933+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T01:30:18.109265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"88'8","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:59.369891+0000","last_change":"2024-05-12T22:06:33.962660+0000","last_active":"2024-05-12T22:08:59.369891+0000","last_peered":"2024-05-12T22:08:59.369891+0000","last_clean":"2024-05-12T22:08:59.369891+0000","last_became_active":"2024-05-12T22:06:33.962284+0000","last_became_peered":"2024-05-12T22:06:33.962284+0000","last_unstale":"2024-05-12T22:08:59.369891+0000","last_undegraded":"2024-05-12T22:08:59.369891+0000","last_fullsized":"2024-05-12T22:08:59.369891+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T01:08:53.888992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":598250,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":972,"num_read_kb":596,"num_write":294,"num_write_kb":1443,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2027520,"data_stored":1794750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":232,"ondisk_log_size":232,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351018,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27948,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717204,"statfs":{"total":95995035648,"available":95966416896,"internally_reserved":0,"allocated":954368,"data_stored":459830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612654,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28840,"kb_used_data":1764,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716312,"statfs":{"total":95995035648,"available":95965503488,"internally_reserved":0,"allocated":1806336,"data_stored":1260157,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776179,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28364,"kb_used_data":1344,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716788,"statfs":{"total":95995035648,"available":95965990912,"internally_reserved":0,"allocated":1376256,"data_stored":861627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037818,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32524,"kb_used_data":1344,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712628,"statfs":{"total":95995035648,"available":95961731072,"internally_reserved":0,"allocated":1376256,"data_stored":861547,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299454,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28556,"kb_used_data":1536,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716596,"statfs":{"total":95995035648,"available":95965794304,"internally_reserved":0,"allocated":1572864,"data_stored":1061525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462981,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":970752,"data_stored":467800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574944,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36356,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708796,"statfs":{"total":95995035648,"available":95957807104,"internally_reserved":0,"allocated":1040384,"data_stored":535180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738469,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":991232,"data_stored":469071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411419,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28060,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717092,"statfs":{"total":95995035648,"available":95966302208,"internally_reserved":0,"allocated":1064960,"data_stored":536280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149781,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28496,"kb_used_data":1480,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716656,"statfs":{"total":95995035648,"available":95965855744,"internally_reserved":0,"allocated":1515520,"data_stored":992927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888142,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27972,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717180,"statfs":{"total":95995035648,"available":95966392320,"internally_reserved":0,"allocated":974848,"data_stored":469551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724618,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28552,"kb_used_data":1536,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716600,"statfs":{"total":95995035648,"available":95965798400,"internally_reserved":0,"allocated":1572864,"data_stored":1051159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":67587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":69860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":544768,"data_stored":525100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":133999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3447,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":413696,"data_stored":393954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":413696,"data_stored":394034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":202079,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-12T22:09:04.340 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-12T22:09:04.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:04 smithi186 ceph-mon[34785]: pgmap v5: 97 pgs: 97 active+clean; 1.1 MiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-12T22:09:04.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:04 smithi186 ceph-mon[34785]: mgrmap e25: x(active, since 4s), standbys: z, y 2024-05-12T22:09:04.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:04 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/2524130045' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-12T22:09:04.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:04 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/4111270902' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:09:04.449 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:04 smithi028 ceph-mon[29847]: pgmap v5: 97 pgs: 97 active+clean; 1.1 MiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-12T22:09:04.449 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:04 smithi028 ceph-mon[29847]: mgrmap e25: x(active, since 4s), standbys: z, y 2024-05-12T22:09:04.449 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:04 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/2524130045' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-12T22:09:04.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:04 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/4111270902' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:09:04.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:04 smithi086 ceph-mon[34297]: pgmap v5: 97 pgs: 97 active+clean; 1.1 MiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-12T22:09:04.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:04 smithi086 ceph-mon[34297]: mgrmap e25: x(active, since 4s), standbys: z, y 2024-05-12T22:09:04.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:04 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/2524130045' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-12T22:09:04.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:04 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/4111270902' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:09:04.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:04.583+0000 7fcbf2600640 1 -- 172.21.15.28:0/2195408872 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcbec102330 msgr2=0x7fcbec108bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:04.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:04.583+0000 7fcbf2600640 1 --2- 172.21.15.28:0/2195408872 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcbec102330 0x7fcbec108bc0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fcbe4009be0 tx=0x7fcbe402a6c0 comp rx=0 tx=0).stop 2024-05-12T22:09:04.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:04.583+0000 7fcbf2600640 1 -- 172.21.15.28:0/2195408872 shutdown_connections 2024-05-12T22:09:04.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:04.583+0000 7fcbf2600640 1 --2- 172.21.15.28:0/2195408872 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcbec102330 0x7fcbec108bc0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:04.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:04.583+0000 7fcbf2600640 1 --2- 172.21.15.28:0/2195408872 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcbec101970 0x7fcbec101df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:04.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:04.583+0000 7fcbf2600640 1 --2- 172.21.15.28:0/2195408872 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcbec100780 0x7fcbec100ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:04.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:04.583+0000 7fcbf2600640 1 -- 172.21.15.28:0/2195408872 >> 172.21.15.28:0/2195408872 conn(0x7fcbec0fbd10 msgr2=0x7fcbec0fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:09:04.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:04.583+0000 7fcbf2600640 1 -- 172.21.15.28:0/2195408872 shutdown_connections 2024-05-12T22:09:04.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:04.583+0000 7fcbf2600640 1 -- 172.21.15.28:0/2195408872 wait complete. 2024-05-12T22:09:04.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:04.584+0000 7fcbf2600640 1 Processor -- start 2024-05-12T22:09:04.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:04.584+0000 7fcbf2600640 1 -- start start 2024-05-12T22:09:04.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:04.585+0000 7fcbf2600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcbec100780 0x7fcbec199de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:04.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:04.585+0000 7fcbf2600640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcbec101970 0x7fcbec19a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:04.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:04.585+0000 7fcbf2600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcbec102330 0x7fcbec1a1390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:04.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:04.585+0000 7fcbf2600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcbec10c3b0 con 0x7fcbec102330 2024-05-12T22:09:04.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:04.585+0000 7fcbf2600640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fcbec10c230 con 0x7fcbec100780 2024-05-12T22:09:04.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:04.585+0000 7fcbf2600640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fcbec10c530 con 0x7fcbec101970 2024-05-12T22:09:04.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:04.585+0000 7fcbe3e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcbec101970 0x7fcbec19a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:04.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:04.585+0000 7fcbebe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcbec102330 0x7fcbec1a1390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:04.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:04.585+0000 7fcbe3e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcbec101970 0x7fcbec19a320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:42530/0 (socket says 172.21.15.28:42530) 2024-05-12T22:09:04.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:04.585+0000 7fcbe3e00640 1 -- 172.21.15.28:0/4005415678 learned_addr learned my addr 172.21.15.28:0/4005415678 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:09:04.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:04.585+0000 7fcbeb400640 1 --2- 172.21.15.28:0/4005415678 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcbec100780 0x7fcbec199de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:04.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:04.585+0000 7fcbe3e00640 1 -- 172.21.15.28:0/4005415678 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcbec100780 msgr2=0x7fcbec199de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:04.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:04.586+0000 7fcbe3e00640 1 --2- 172.21.15.28:0/4005415678 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcbec100780 0x7fcbec199de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:04.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:04.586+0000 7fcbe3e00640 1 -- 172.21.15.28:0/4005415678 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcbec102330 msgr2=0x7fcbec1a1390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:04.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:04.586+0000 7fcbe3e00640 1 --2- 172.21.15.28:0/4005415678 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcbec102330 0x7fcbec1a1390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:04.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:04.586+0000 7fcbe3e00640 1 -- 172.21.15.28:0/4005415678 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbec1a1a00 con 0x7fcbec101970 2024-05-12T22:09:04.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:04.586+0000 7fcbeb400640 1 --2- 172.21.15.28:0/4005415678 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcbec100780 0x7fcbec199de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:09:04.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:04.586+0000 7fcbe3e00640 1 --2- 172.21.15.28:0/4005415678 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcbec101970 0x7fcbec19a320 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fcbd400e9f0 tx=0x7fcbd400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:09:04.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:04.586+0000 7fcbe8c00640 1 -- 172.21.15.28:0/4005415678 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbd400ccc0 con 0x7fcbec101970 2024-05-12T22:09:04.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:04.586+0000 7fcbe8c00640 1 -- 172.21.15.28:0/4005415678 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcbd4005500 con 0x7fcbec101970 2024-05-12T22:09:04.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:04.586+0000 7fcbe8c00640 1 -- 172.21.15.28:0/4005415678 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbd4010690 con 0x7fcbec101970 2024-05-12T22:09:04.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:04.586+0000 7fcbf2600640 1 -- 172.21.15.28:0/4005415678 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcbec1a1cf0 con 0x7fcbec101970 2024-05-12T22:09:04.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:04.586+0000 7fcbf2600640 1 -- 172.21.15.28:0/4005415678 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcbec1a2190 con 0x7fcbec101970 2024-05-12T22:09:04.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:04.588+0000 7fcbf2600640 1 -- 172.21.15.28:0/4005415678 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcbec1a2420 con 0x7fcbec101970 2024-05-12T22:09:04.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:04.590+0000 7fcbe8c00640 1 -- 172.21.15.28:0/4005415678 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 25) ==== 138060+0+0 (secure 0 0 0) 0x7fcbd40046c0 con 0x7fcbec101970 2024-05-12T22:09:04.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:04.592+0000 7fcbe8c00640 1 --2- 172.21.15.28:0/4005415678 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fcbd00a2ea0 0x7fcbd00a5360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:04.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:04.592+0000 7fcbe8c00640 1 -- 172.21.15.28:0/4005415678 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7fcbd40cf910 con 0x7fcbec101970 2024-05-12T22:09:04.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:04.592+0000 7fcbeb400640 1 --2- 172.21.15.28:0/4005415678 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fcbd00a2ea0 0x7fcbd00a5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:04.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:04.593+0000 7fcbeb400640 1 --2- 172.21.15.28:0/4005415678 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fcbd00a2ea0 0x7fcbd00a5360 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fcbec1017d0 tx=0x7fcbdc005f90 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:09:04.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:04.595+0000 7fcbe8c00640 1 -- 172.21.15.28:0/4005415678 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fcbd409c310 con 0x7fcbec101970 2024-05-12T22:09:04.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:04.812+0000 7fcbf2600640 1 -- 172.21.15.28:0/4005415678 --> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fcbec0fdec0 con 0x7fcbd00a2ea0 2024-05-12T22:09:04.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:04.820+0000 7fcbe8c00640 1 -- 172.21.15.28:0/4005415678 <== mgr.25189 v2:172.21.15.28:6800/3177640985 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+257774 (secure 0 0 0) 0x7fcbec0fdec0 con 0x7fcbd00a2ea0 2024-05-12T22:09:04.823 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-12T22:09:04.826 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-05-12T22:09:04.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:04.827+0000 7fcbf2600640 1 -- 172.21.15.28:0/4005415678 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fcbd00a2ea0 msgr2=0x7fcbd00a5360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:04.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:04.827+0000 7fcbf2600640 1 --2- 172.21.15.28:0/4005415678 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fcbd00a2ea0 0x7fcbd00a5360 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fcbec1017d0 tx=0x7fcbdc005f90 comp rx=0 tx=0).stop 2024-05-12T22:09:04.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:04.827+0000 7fcbf2600640 1 -- 172.21.15.28:0/4005415678 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcbec101970 msgr2=0x7fcbec19a320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:04.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:04.827+0000 7fcbf2600640 1 --2- 172.21.15.28:0/4005415678 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcbec101970 0x7fcbec19a320 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fcbd400e9f0 tx=0x7fcbd400eeb0 comp rx=0 tx=0).stop 2024-05-12T22:09:04.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:04.827+0000 7fcbf2600640 1 -- 172.21.15.28:0/4005415678 shutdown_connections 2024-05-12T22:09:04.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:04.827+0000 7fcbf2600640 1 --2- 172.21.15.28:0/4005415678 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fcbd00a2ea0 0x7fcbd00a5360 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:04.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:04.827+0000 7fcbf2600640 1 --2- 172.21.15.28:0/4005415678 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcbec102330 0x7fcbec1a1390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:04.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:04.827+0000 7fcbf2600640 1 --2- 172.21.15.28:0/4005415678 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcbec101970 0x7fcbec19a320 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:04.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:04.827+0000 7fcbf2600640 1 --2- 172.21.15.28:0/4005415678 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcbec100780 0x7fcbec199de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:04.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:04.827+0000 7fcbf2600640 1 -- 172.21.15.28:0/4005415678 >> 172.21.15.28:0/4005415678 conn(0x7fcbec0fbd10 msgr2=0x7fcbec0fd7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:09:04.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:04.828+0000 7fcbf2600640 1 -- 172.21.15.28:0/4005415678 shutdown_connections 2024-05-12T22:09:04.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:04.828+0000 7fcbf2600640 1 -- 172.21.15.28:0/4005415678 wait complete. 2024-05-12T22:09:04.851 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":6,"stamp":"2024-05-12T22:09:04.669911+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1188852,"num_objects":66,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":1108,"num_read_kb":715,"num_write":477,"num_write_kb":4383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":1181318,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":871,"ondisk_log_size":871,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":351624,"kb_used_data":14860,"kb_used_omap":18,"kb_used_meta":336557,"kb_avail":1124590200,"statfs":{"total":1151940427776,"available":1151580364800,"internally_reserved":0,"allocated":15216640,"data_stored":9026654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":344634681},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.976276"},"pg_stats":[{"pgid":"3.3f","version":"88'8","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:59.370219+0000","last_change":"2024-05-12T22:06:33.982079+0000","last_active":"2024-05-12T22:08:59.370219+0000","last_peered":"2024-05-12T22:08:59.370219+0000","last_clean":"2024-05-12T22:08:59.370219+0000","last_became_active":"2024-05-12T22:06:33.981703+0000","last_became_peered":"2024-05-12T22:06:33.981703+0000","last_unstale":"2024-05-12T22:08:59.370219+0000","last_undegraded":"2024-05-12T22:08:59.370219+0000","last_fullsized":"2024-05-12T22:08:59.370219+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T08:52:47.157067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"88'4","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.974850+0000","last_change":"2024-05-12T22:06:33.984194+0000","last_active":"2024-05-12T22:08:58.974850+0000","last_peered":"2024-05-12T22:08:58.974850+0000","last_clean":"2024-05-12T22:08:58.974850+0000","last_became_active":"2024-05-12T22:06:33.979787+0000","last_became_peered":"2024-05-12T22:06:33.979787+0000","last_unstale":"2024-05-12T22:08:58.974850+0000","last_undegraded":"2024-05-12T22:08:58.974850+0000","last_fullsized":"2024-05-12T22:08:58.974850+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T07:36:46.588641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"88'7","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.974018+0000","last_change":"2024-05-12T22:06:33.982428+0000","last_active":"2024-05-12T22:08:58.974018+0000","last_peered":"2024-05-12T22:08:58.974018+0000","last_clean":"2024-05-12T22:08:58.974018+0000","last_became_active":"2024-05-12T22:06:33.979563+0000","last_became_peered":"2024-05-12T22:06:33.979563+0000","last_unstale":"2024-05-12T22:08:58.974018+0000","last_undegraded":"2024-05-12T22:08:58.974018+0000","last_fullsized":"2024-05-12T22:08:58.974018+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T06:26:12.586519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"88'8","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.776473+0000","last_change":"2024-05-12T22:06:46.796152+0000","last_active":"2024-05-12T22:08:58.776473+0000","last_peered":"2024-05-12T22:08:58.776473+0000","last_clean":"2024-05-12T22:08:58.776473+0000","last_became_active":"2024-05-12T22:06:46.795484+0000","last_became_peered":"2024-05-12T22:06:46.795484+0000","last_unstale":"2024-05-12T22:08:58.776473+0000","last_undegraded":"2024-05-12T22:08:58.776473+0000","last_fullsized":"2024-05-12T22:08:58.776473+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T03:13:45.901507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,2],"acting":[9,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.3b","version":"88'4","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.778898+0000","last_change":"2024-05-12T22:06:33.967492+0000","last_active":"2024-05-12T22:08:58.778898+0000","last_peered":"2024-05-12T22:08:58.778898+0000","last_clean":"2024-05-12T22:08:58.778898+0000","last_became_active":"2024-05-12T22:06:33.967146+0000","last_became_peered":"2024-05-12T22:06:33.967146+0000","last_unstale":"2024-05-12T22:08:58.778898+0000","last_undegraded":"2024-05-12T22:08:58.778898+0000","last_fullsized":"2024-05-12T22:08:58.778898+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T07:31:05.946473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"88'6","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.649499+0000","last_change":"2024-05-12T22:06:33.973653+0000","last_active":"2024-05-12T22:08:58.649499+0000","last_peered":"2024-05-12T22:08:58.649499+0000","last_clean":"2024-05-12T22:08:58.649499+0000","last_became_active":"2024-05-12T22:06:33.973290+0000","last_became_peered":"2024-05-12T22:06:33.973290+0000","last_unstale":"2024-05-12T22:08:58.649499+0000","last_undegraded":"2024-05-12T22:08:58.649499+0000","last_fullsized":"2024-05-12T22:08:58.649499+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T03:07:21.933141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"88'5","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.967170+0000","last_change":"2024-05-12T22:06:33.973807+0000","last_active":"2024-05-12T22:08:58.967170+0000","last_peered":"2024-05-12T22:08:58.967170+0000","last_clean":"2024-05-12T22:08:58.967170+0000","last_became_active":"2024-05-12T22:06:33.973466+0000","last_became_peered":"2024-05-12T22:06:33.973466+0000","last_unstale":"2024-05-12T22:08:58.967170+0000","last_undegraded":"2024-05-12T22:08:58.967170+0000","last_fullsized":"2024-05-12T22:08:58.967170+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T02:31:13.608309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"88'9","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.648390+0000","last_change":"2024-05-12T22:06:33.967356+0000","last_active":"2024-05-12T22:08:58.648390+0000","last_peered":"2024-05-12T22:08:58.648390+0000","last_clean":"2024-05-12T22:08:58.648390+0000","last_became_active":"2024-05-12T22:06:33.966054+0000","last_became_peered":"2024-05-12T22:06:33.966054+0000","last_unstale":"2024-05-12T22:08:58.648390+0000","last_undegraded":"2024-05-12T22:08:58.648390+0000","last_fullsized":"2024-05-12T22:08:58.648390+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T23:54:08.285720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"88'11","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.981661+0000","last_change":"2024-05-12T22:06:33.968782+0000","last_active":"2024-05-12T22:08:58.981661+0000","last_peered":"2024-05-12T22:08:58.981661+0000","last_clean":"2024-05-12T22:08:58.981661+0000","last_became_active":"2024-05-12T22:06:33.968435+0000","last_became_peered":"2024-05-12T22:06:33.968435+0000","last_unstale":"2024-05-12T22:08:58.981661+0000","last_undegraded":"2024-05-12T22:08:58.981661+0000","last_fullsized":"2024-05-12T22:08:58.981661+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T07:01:21.056777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"88'16","reported_seq":36,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.649922+0000","last_change":"2024-05-12T22:06:33.962018+0000","last_active":"2024-05-12T22:08:58.649922+0000","last_peered":"2024-05-12T22:08:58.649922+0000","last_clean":"2024-05-12T22:08:58.649922+0000","last_became_active":"2024-05-12T22:06:33.961583+0000","last_became_peered":"2024-05-12T22:06:33.961583+0000","last_unstale":"2024-05-12T22:08:58.649922+0000","last_undegraded":"2024-05-12T22:08:58.649922+0000","last_fullsized":"2024-05-12T22:08:58.649922+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T00:16:40.483194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"88'6","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.967780+0000","last_change":"2024-05-12T22:06:33.965776+0000","last_active":"2024-05-12T22:08:58.967780+0000","last_peered":"2024-05-12T22:08:58.967780+0000","last_clean":"2024-05-12T22:08:58.967780+0000","last_became_active":"2024-05-12T22:06:33.965439+0000","last_became_peered":"2024-05-12T22:06:33.965439+0000","last_unstale":"2024-05-12T22:08:58.967780+0000","last_undegraded":"2024-05-12T22:08:58.967780+0000","last_fullsized":"2024-05-12T22:08:58.967780+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T23:59:56.925209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"88'7","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.966781+0000","last_change":"2024-05-12T22:06:33.967906+0000","last_active":"2024-05-12T22:08:58.966781+0000","last_peered":"2024-05-12T22:08:58.966781+0000","last_clean":"2024-05-12T22:08:58.966781+0000","last_became_active":"2024-05-12T22:06:33.967516+0000","last_became_peered":"2024-05-12T22:06:33.967516+0000","last_unstale":"2024-05-12T22:08:58.966781+0000","last_undegraded":"2024-05-12T22:08:58.966781+0000","last_fullsized":"2024-05-12T22:08:58.966781+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T08:16:56.892024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"88'10","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.968023+0000","last_change":"2024-05-12T22:06:33.971456+0000","last_active":"2024-05-12T22:08:58.968023+0000","last_peered":"2024-05-12T22:08:58.968023+0000","last_clean":"2024-05-12T22:08:58.968023+0000","last_became_active":"2024-05-12T22:06:33.971110+0000","last_became_peered":"2024-05-12T22:06:33.971110+0000","last_unstale":"2024-05-12T22:08:58.968023+0000","last_undegraded":"2024-05-12T22:08:58.968023+0000","last_fullsized":"2024-05-12T22:08:58.968023+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T09:48:56.275346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"88'7","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.973912+0000","last_change":"2024-05-12T22:06:33.983421+0000","last_active":"2024-05-12T22:08:58.973912+0000","last_peered":"2024-05-12T22:08:58.973912+0000","last_clean":"2024-05-12T22:08:58.973912+0000","last_became_active":"2024-05-12T22:06:33.978515+0000","last_became_peered":"2024-05-12T22:06:33.978515+0000","last_unstale":"2024-05-12T22:08:58.973912+0000","last_undegraded":"2024-05-12T22:08:58.973912+0000","last_fullsized":"2024-05-12T22:08:58.973912+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T23:09:48.264541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"88'7","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.974576+0000","last_change":"2024-05-12T22:06:33.980956+0000","last_active":"2024-05-12T22:08:58.974576+0000","last_peered":"2024-05-12T22:08:58.974576+0000","last_clean":"2024-05-12T22:08:58.974576+0000","last_became_active":"2024-05-12T22:06:33.979860+0000","last_became_peered":"2024-05-12T22:06:33.979860+0000","last_unstale":"2024-05-12T22:08:58.974576+0000","last_undegraded":"2024-05-12T22:08:58.974576+0000","last_fullsized":"2024-05-12T22:08:58.974576+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T02:58:29.255704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"88'9","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.981122+0000","last_change":"2024-05-12T22:06:33.978682+0000","last_active":"2024-05-12T22:08:58.981122+0000","last_peered":"2024-05-12T22:08:58.981122+0000","last_clean":"2024-05-12T22:08:58.981122+0000","last_became_active":"2024-05-12T22:06:33.978234+0000","last_became_peered":"2024-05-12T22:06:33.978234+0000","last_unstale":"2024-05-12T22:08:58.981122+0000","last_undegraded":"2024-05-12T22:08:58.981122+0000","last_fullsized":"2024-05-12T22:08:58.981122+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T07:40:48.540839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"88'8","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:59.370117+0000","last_change":"2024-05-12T22:06:33.986598+0000","last_active":"2024-05-12T22:08:59.370117+0000","last_peered":"2024-05-12T22:08:59.370117+0000","last_clean":"2024-05-12T22:08:59.370117+0000","last_became_active":"2024-05-12T22:06:33.986346+0000","last_became_peered":"2024-05-12T22:06:33.986346+0000","last_unstale":"2024-05-12T22:08:59.370117+0000","last_undegraded":"2024-05-12T22:08:59.370117+0000","last_fullsized":"2024-05-12T22:08:59.370117+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T05:52:05.473099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"88'10","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.776735+0000","last_change":"2024-05-12T22:06:33.978900+0000","last_active":"2024-05-12T22:08:58.776735+0000","last_peered":"2024-05-12T22:08:58.776735+0000","last_clean":"2024-05-12T22:08:58.776735+0000","last_became_active":"2024-05-12T22:06:33.978474+0000","last_became_peered":"2024-05-12T22:06:33.978474+0000","last_unstale":"2024-05-12T22:08:58.776735+0000","last_undegraded":"2024-05-12T22:08:58.776735+0000","last_fullsized":"2024-05-12T22:08:58.776735+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T03:47:53.014263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"88'7","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.651554+0000","last_change":"2024-05-12T22:06:33.966752+0000","last_active":"2024-05-12T22:08:58.651554+0000","last_peered":"2024-05-12T22:08:58.651554+0000","last_clean":"2024-05-12T22:08:58.651554+0000","last_became_active":"2024-05-12T22:06:33.966320+0000","last_became_peered":"2024-05-12T22:06:33.966320+0000","last_unstale":"2024-05-12T22:08:58.651554+0000","last_undegraded":"2024-05-12T22:08:58.651554+0000","last_fullsized":"2024-05-12T22:08:58.651554+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T22:11:48.356156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'9","reported_seq":48,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.776810+0000","last_change":"2024-05-12T22:06:46.808696+0000","last_active":"2024-05-12T22:08:58.776810+0000","last_peered":"2024-05-12T22:08:58.776810+0000","last_clean":"2024-05-12T22:08:58.776810+0000","last_became_active":"2024-05-12T22:06:46.796654+0000","last_became_peered":"2024-05-12T22:06:46.796654+0000","last_unstale":"2024-05-12T22:08:58.776810+0000","last_undegraded":"2024-05-12T22:08:58.776810+0000","last_fullsized":"2024-05-12T22:08:58.776810+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T09:32:33.209844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,8],"acting":[9,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"88'5","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.648988+0000","last_change":"2024-05-12T22:06:33.963875+0000","last_active":"2024-05-12T22:08:58.648988+0000","last_peered":"2024-05-12T22:08:58.648988+0000","last_clean":"2024-05-12T22:08:58.648988+0000","last_became_active":"2024-05-12T22:06:33.963239+0000","last_became_peered":"2024-05-12T22:06:33.963239+0000","last_unstale":"2024-05-12T22:08:58.648988+0000","last_undegraded":"2024-05-12T22:08:58.648988+0000","last_fullsized":"2024-05-12T22:08:58.648988+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T23:15:01.832082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.649401+0000","last_change":"2024-05-12T22:06:32.950352+0000","last_active":"2024-05-12T22:08:58.649401+0000","last_peered":"2024-05-12T22:08:58.649401+0000","last_clean":"2024-05-12T22:08:58.649401+0000","last_became_active":"2024-05-12T22:06:32.949759+0000","last_became_peered":"2024-05-12T22:06:32.949759+0000","last_unstale":"2024-05-12T22:08:58.649401+0000","last_undegraded":"2024-05-12T22:08:58.649401+0000","last_fullsized":"2024-05-12T22:08:58.649401+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T06:28:07.613351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"88'6","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.975223+0000","last_change":"2024-05-12T22:06:33.990783+0000","last_active":"2024-05-12T22:08:58.975223+0000","last_peered":"2024-05-12T22:08:58.975223+0000","last_clean":"2024-05-12T22:08:58.975223+0000","last_became_active":"2024-05-12T22:06:33.990387+0000","last_became_peered":"2024-05-12T22:06:33.990387+0000","last_unstale":"2024-05-12T22:08:58.975223+0000","last_undegraded":"2024-05-12T22:08:58.975223+0000","last_fullsized":"2024-05-12T22:08:58.975223+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T05:52:52.247458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"86'4","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.779322+0000","last_change":"2024-05-12T22:06:32.931718+0000","last_active":"2024-05-12T22:08:58.779322+0000","last_peered":"2024-05-12T22:08:58.779322+0000","last_clean":"2024-05-12T22:08:58.779322+0000","last_became_active":"2024-05-12T22:06:32.931404+0000","last_became_peered":"2024-05-12T22:06:32.931404+0000","last_unstale":"2024-05-12T22:08:58.779322+0000","last_undegraded":"2024-05-12T22:08:58.779322+0000","last_fullsized":"2024-05-12T22:08:58.779322+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T02:55:57.770504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"88'9","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.981153+0000","last_change":"2024-05-12T22:06:33.963299+0000","last_active":"2024-05-12T22:08:58.981153+0000","last_peered":"2024-05-12T22:08:58.981153+0000","last_clean":"2024-05-12T22:08:58.981153+0000","last_became_active":"2024-05-12T22:06:33.962916+0000","last_became_peered":"2024-05-12T22:06:33.962916+0000","last_unstale":"2024-05-12T22:08:58.981153+0000","last_undegraded":"2024-05-12T22:08:58.981153+0000","last_fullsized":"2024-05-12T22:08:58.981153+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T05:46:32.131918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:59.369771+0000","last_change":"2024-05-12T22:06:46.800870+0000","last_active":"2024-05-12T22:08:59.369771+0000","last_peered":"2024-05-12T22:08:59.369771+0000","last_clean":"2024-05-12T22:08:59.369771+0000","last_became_active":"2024-05-12T22:06:46.797716+0000","last_became_peered":"2024-05-12T22:06:46.797716+0000","last_unstale":"2024-05-12T22:08:59.369771+0000","last_undegraded":"2024-05-12T22:08:59.369771+0000","last_fullsized":"2024-05-12T22:08:59.369771+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T08:31:57.557564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1d","version":"88'15","reported_seq":34,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.786793+0000","last_change":"2024-05-12T22:06:33.967926+0000","last_active":"2024-05-12T22:08:58.786793+0000","last_peered":"2024-05-12T22:08:58.786793+0000","last_clean":"2024-05-12T22:08:58.786793+0000","last_became_active":"2024-05-12T22:06:33.967588+0000","last_became_peered":"2024-05-12T22:06:33.967588+0000","last_unstale":"2024-05-12T22:08:58.786793+0000","last_undegraded":"2024-05-12T22:08:58.786793+0000","last_fullsized":"2024-05-12T22:08:58.786793+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T22:39:54.434117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"89'8","reported_seq":175,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.966357+0000","last_change":"2024-05-12T22:06:32.930347+0000","last_active":"2024-05-12T22:08:58.966357+0000","last_peered":"2024-05-12T22:08:58.966357+0000","last_clean":"2024-05-12T22:08:58.966357+0000","last_became_active":"2024-05-12T22:06:32.930083+0000","last_became_peered":"2024-05-12T22:06:32.930083+0000","last_unstale":"2024-05-12T22:08:58.966357+0000","last_undegraded":"2024-05-12T22:08:58.966357+0000","last_fullsized":"2024-05-12T22:08:58.966357+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T07:10:08.898490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1545,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":145,"num_read_kb":4,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"88'7","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.778962+0000","last_change":"2024-05-12T22:06:33.973793+0000","last_active":"2024-05-12T22:08:58.778962+0000","last_peered":"2024-05-12T22:08:58.778962+0000","last_clean":"2024-05-12T22:08:58.778962+0000","last_became_active":"2024-05-12T22:06:33.973380+0000","last_became_peered":"2024-05-12T22:06:33.973380+0000","last_unstale":"2024-05-12T22:08:58.778962+0000","last_undegraded":"2024-05-12T22:08:58.778962+0000","last_fullsized":"2024-05-12T22:08:58.778962+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T04:32:01.817780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"86'3","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.649371+0000","last_change":"2024-05-12T22:06:32.938215+0000","last_active":"2024-05-12T22:08:58.649371+0000","last_peered":"2024-05-12T22:08:58.649371+0000","last_clean":"2024-05-12T22:08:58.649371+0000","last_became_active":"2024-05-12T22:06:32.937939+0000","last_became_peered":"2024-05-12T22:06:32.937939+0000","last_unstale":"2024-05-12T22:08:58.649371+0000","last_undegraded":"2024-05-12T22:08:58.649371+0000","last_fullsized":"2024-05-12T22:08:58.649371+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T04:15:28.821835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"88'8","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.786653+0000","last_change":"2024-05-12T22:06:33.977362+0000","last_active":"2024-05-12T22:08:58.786653+0000","last_peered":"2024-05-12T22:08:58.786653+0000","last_clean":"2024-05-12T22:08:58.786653+0000","last_became_active":"2024-05-12T22:06:33.977074+0000","last_became_peered":"2024-05-12T22:06:33.977074+0000","last_unstale":"2024-05-12T22:08:58.786653+0000","last_undegraded":"2024-05-12T22:08:58.786653+0000","last_fullsized":"2024-05-12T22:08:58.786653+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T22:21:00.062780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"86'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.966862+0000","last_change":"2024-05-12T22:06:32.935508+0000","last_active":"2024-05-12T22:08:58.966862+0000","last_peered":"2024-05-12T22:08:58.966862+0000","last_clean":"2024-05-12T22:08:58.966862+0000","last_became_active":"2024-05-12T22:06:32.934760+0000","last_became_peered":"2024-05-12T22:06:32.934760+0000","last_unstale":"2024-05-12T22:08:58.966862+0000","last_undegraded":"2024-05-12T22:08:58.966862+0000","last_fullsized":"2024-05-12T22:08:58.966862+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T01:58:09.450187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"88'8","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.648596+0000","last_change":"2024-05-12T22:06:33.966421+0000","last_active":"2024-05-12T22:08:58.648596+0000","last_peered":"2024-05-12T22:08:58.648596+0000","last_clean":"2024-05-12T22:08:58.648596+0000","last_became_active":"2024-05-12T22:06:33.965627+0000","last_became_peered":"2024-05-12T22:06:33.965627+0000","last_unstale":"2024-05-12T22:08:58.648596+0000","last_undegraded":"2024-05-12T22:08:58.648596+0000","last_fullsized":"2024-05-12T22:08:58.648596+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T09:25:03.836082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"86'1","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.786416+0000","last_change":"2024-05-12T22:06:32.928907+0000","last_active":"2024-05-12T22:08:58.786416+0000","last_peered":"2024-05-12T22:08:58.786416+0000","last_clean":"2024-05-12T22:08:58.786416+0000","last_became_active":"2024-05-12T22:06:32.928586+0000","last_became_peered":"2024-05-12T22:06:32.928586+0000","last_unstale":"2024-05-12T22:08:58.786416+0000","last_undegraded":"2024-05-12T22:08:58.786416+0000","last_fullsized":"2024-05-12T22:08:58.786416+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T04:24:21.658957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"88'11","reported_seq":31,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.776199+0000","last_change":"2024-05-12T22:06:33.982873+0000","last_active":"2024-05-12T22:08:58.776199+0000","last_peered":"2024-05-12T22:08:58.776199+0000","last_clean":"2024-05-12T22:08:58.776199+0000","last_became_active":"2024-05-12T22:06:33.982495+0000","last_became_peered":"2024-05-12T22:06:33.982495+0000","last_unstale":"2024-05-12T22:08:58.776199+0000","last_undegraded":"2024-05-12T22:08:58.776199+0000","last_fullsized":"2024-05-12T22:08:58.776199+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T05:45:39.404282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"89'5","reported_seq":168,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.966975+0000","last_change":"2024-05-12T22:06:32.937716+0000","last_active":"2024-05-12T22:08:58.966975+0000","last_peered":"2024-05-12T22:08:58.966975+0000","last_clean":"2024-05-12T22:08:58.966975+0000","last_became_active":"2024-05-12T22:06:32.937418+0000","last_became_peered":"2024-05-12T22:06:32.937418+0000","last_unstale":"2024-05-12T22:08:58.966975+0000","last_undegraded":"2024-05-12T22:08:58.966975+0000","last_fullsized":"2024-05-12T22:08:58.966975+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T08:20:13.639958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":141,"num_read_kb":141,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"88'10","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.981357+0000","last_change":"2024-05-12T22:06:33.974702+0000","last_active":"2024-05-12T22:08:58.981357+0000","last_peered":"2024-05-12T22:08:58.981357+0000","last_clean":"2024-05-12T22:08:58.981357+0000","last_became_active":"2024-05-12T22:06:33.974363+0000","last_became_peered":"2024-05-12T22:06:33.974363+0000","last_unstale":"2024-05-12T22:08:58.981357+0000","last_undegraded":"2024-05-12T22:08:58.981357+0000","last_fullsized":"2024-05-12T22:08:58.981357+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T09:16:33.620945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"86'2","reported_seq":34,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.967542+0000","last_change":"2024-05-12T22:06:46.793331+0000","last_active":"2024-05-12T22:08:58.967542+0000","last_peered":"2024-05-12T22:08:58.967542+0000","last_clean":"2024-05-12T22:08:58.967542+0000","last_became_active":"2024-05-12T22:06:46.789984+0000","last_became_peered":"2024-05-12T22:06:46.789984+0000","last_unstale":"2024-05-12T22:08:58.967542+0000","last_undegraded":"2024-05-12T22:08:58.967542+0000","last_fullsized":"2024-05-12T22:08:58.967542+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T08:16:29.505145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"88'9","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.975167+0000","last_change":"2024-05-12T22:06:33.980496+0000","last_active":"2024-05-12T22:08:58.975167+0000","last_peered":"2024-05-12T22:08:58.975167+0000","last_clean":"2024-05-12T22:08:58.975167+0000","last_became_active":"2024-05-12T22:06:33.980155+0000","last_became_peered":"2024-05-12T22:06:33.980155+0000","last_unstale":"2024-05-12T22:08:58.975167+0000","last_undegraded":"2024-05-12T22:08:58.975167+0000","last_fullsized":"2024-05-12T22:08:58.975167+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T01:54:03.831828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"86'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.786948+0000","last_change":"2024-05-12T22:06:32.951633+0000","last_active":"2024-05-12T22:08:58.786948+0000","last_peered":"2024-05-12T22:08:58.786948+0000","last_clean":"2024-05-12T22:08:58.786948+0000","last_became_active":"2024-05-12T22:06:32.951382+0000","last_became_peered":"2024-05-12T22:06:32.951382+0000","last_unstale":"2024-05-12T22:08:58.786948+0000","last_undegraded":"2024-05-12T22:08:58.786948+0000","last_fullsized":"2024-05-12T22:08:58.786948+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T02:06:21.408593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"88'11","reported_seq":31,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.981428+0000","last_change":"2024-05-12T22:06:33.963493+0000","last_active":"2024-05-12T22:08:58.981428+0000","last_peered":"2024-05-12T22:08:58.981428+0000","last_clean":"2024-05-12T22:08:58.981428+0000","last_became_active":"2024-05-12T22:06:33.963127+0000","last_became_peered":"2024-05-12T22:06:33.963127+0000","last_unstale":"2024-05-12T22:08:58.981428+0000","last_undegraded":"2024-05-12T22:08:58.981428+0000","last_fullsized":"2024-05-12T22:08:58.981428+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T05:39:31.038446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"89'38","reported_seq":353,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.982668+0000","last_change":"2024-05-12T22:06:32.932026+0000","last_active":"2024-05-12T22:08:58.982668+0000","last_peered":"2024-05-12T22:08:58.982668+0000","last_clean":"2024-05-12T22:08:58.982668+0000","last_became_active":"2024-05-12T22:06:32.931667+0000","last_became_peered":"2024-05-12T22:06:32.931667+0000","last_unstale":"2024-05-12T22:08:58.982668+0000","last_undegraded":"2024-05-12T22:08:58.982668+0000","last_fullsized":"2024-05-12T22:08:58.982668+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T06:16:28.259047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67587,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":293,"num_read_kb":212,"num_write":39,"num_write_kb":85,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"88'6","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.779148+0000","last_change":"2024-05-12T22:06:33.962677+0000","last_active":"2024-05-12T22:08:58.779148+0000","last_peered":"2024-05-12T22:08:58.779148+0000","last_clean":"2024-05-12T22:08:58.779148+0000","last_became_active":"2024-05-12T22:06:33.962295+0000","last_became_peered":"2024-05-12T22:06:33.962295+0000","last_unstale":"2024-05-12T22:08:58.779148+0000","last_undegraded":"2024-05-12T22:08:58.779148+0000","last_fullsized":"2024-05-12T22:08:58.779148+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T01:31:04.587046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"86'6","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.651259+0000","last_change":"2024-05-12T22:06:32.936104+0000","last_active":"2024-05-12T22:08:58.651259+0000","last_peered":"2024-05-12T22:08:58.651259+0000","last_clean":"2024-05-12T22:08:58.651259+0000","last_became_active":"2024-05-12T22:06:32.935701+0000","last_became_peered":"2024-05-12T22:06:32.935701+0000","last_unstale":"2024-05-12T22:08:58.651259+0000","last_undegraded":"2024-05-12T22:08:58.651259+0000","last_fullsized":"2024-05-12T22:08:58.651259+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T07:04:10.115302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"88'9","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.967648+0000","last_change":"2024-05-12T22:06:33.977850+0000","last_active":"2024-05-12T22:08:58.967648+0000","last_peered":"2024-05-12T22:08:58.967648+0000","last_clean":"2024-05-12T22:08:58.967648+0000","last_became_active":"2024-05-12T22:06:33.977511+0000","last_became_peered":"2024-05-12T22:06:33.977511+0000","last_unstale":"2024-05-12T22:08:58.967648+0000","last_undegraded":"2024-05-12T22:08:58.967648+0000","last_fullsized":"2024-05-12T22:08:58.967648+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T23:10:17.267823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"86'3","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.786663+0000","last_change":"2024-05-12T22:06:32.929955+0000","last_active":"2024-05-12T22:08:58.786663+0000","last_peered":"2024-05-12T22:08:58.786663+0000","last_clean":"2024-05-12T22:08:58.786663+0000","last_became_active":"2024-05-12T22:06:32.929623+0000","last_became_peered":"2024-05-12T22:06:32.929623+0000","last_unstale":"2024-05-12T22:08:58.786663+0000","last_undegraded":"2024-05-12T22:08:58.786663+0000","last_fullsized":"2024-05-12T22:08:58.786663+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T08:15:22.622908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"88'15","reported_seq":41,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.967533+0000","last_change":"2024-05-12T22:06:46.931736+0000","last_active":"2024-05-12T22:08:58.967533+0000","last_peered":"2024-05-12T22:08:58.967533+0000","last_clean":"2024-05-12T22:08:58.967533+0000","last_became_active":"2024-05-12T22:06:46.931265+0000","last_became_peered":"2024-05-12T22:06:46.931265+0000","last_unstale":"2024-05-12T22:08:58.967533+0000","last_undegraded":"2024-05-12T22:08:58.967533+0000","last_fullsized":"2024-05-12T22:08:58.967533+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T02:25:34.002024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"89'7","reported_seq":175,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:59.010336+0000","last_change":"2024-05-12T22:06:32.927707+0000","last_active":"2024-05-12T22:08:59.010336+0000","last_peered":"2024-05-12T22:08:59.010336+0000","last_clean":"2024-05-12T22:08:59.010336+0000","last_became_active":"2024-05-12T22:06:32.927406+0000","last_became_peered":"2024-05-12T22:06:32.927406+0000","last_unstale":"2024-05-12T22:08:59.010336+0000","last_undegraded":"2024-05-12T22:08:59.010336+0000","last_fullsized":"2024-05-12T22:08:59.010336+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T05:14:34.268589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1902,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":147,"num_read_kb":6,"num_write":6,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"88'6","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.966849+0000","last_change":"2024-05-12T22:06:33.964996+0000","last_active":"2024-05-12T22:08:58.966849+0000","last_peered":"2024-05-12T22:08:58.966849+0000","last_clean":"2024-05-12T22:08:58.966849+0000","last_became_active":"2024-05-12T22:06:33.964587+0000","last_became_peered":"2024-05-12T22:06:33.964587+0000","last_unstale":"2024-05-12T22:08:58.966849+0000","last_undegraded":"2024-05-12T22:08:58.966849+0000","last_fullsized":"2024-05-12T22:08:58.966849+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T22:27:28.316377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"86'15","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.778916+0000","last_change":"2024-05-12T22:06:32.935543+0000","last_active":"2024-05-12T22:08:58.778916+0000","last_peered":"2024-05-12T22:08:58.778916+0000","last_clean":"2024-05-12T22:08:58.778916+0000","last_became_active":"2024-05-12T22:06:32.935145+0000","last_became_peered":"2024-05-12T22:06:32.935145+0000","last_unstale":"2024-05-12T22:08:58.778916+0000","last_undegraded":"2024-05-12T22:08:58.778916+0000","last_fullsized":"2024-05-12T22:08:58.778916+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T01:13:23.289057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"88'6","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.786436+0000","last_change":"2024-05-12T22:06:33.964367+0000","last_active":"2024-05-12T22:08:58.786436+0000","last_peered":"2024-05-12T22:08:58.786436+0000","last_clean":"2024-05-12T22:08:58.786436+0000","last_became_active":"2024-05-12T22:06:33.963876+0000","last_became_peered":"2024-05-12T22:06:33.963876+0000","last_unstale":"2024-05-12T22:08:58.786436+0000","last_undegraded":"2024-05-12T22:08:58.786436+0000","last_fullsized":"2024-05-12T22:08:58.786436+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T07:09:20.962965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:59.356563+0000","last_change":"2024-05-12T22:06:32.927270+0000","last_active":"2024-05-12T22:08:59.356563+0000","last_peered":"2024-05-12T22:08:59.356563+0000","last_clean":"2024-05-12T22:08:59.356563+0000","last_became_active":"2024-05-12T22:06:32.926930+0000","last_became_peered":"2024-05-12T22:06:32.926930+0000","last_unstale":"2024-05-12T22:08:59.356563+0000","last_undegraded":"2024-05-12T22:08:59.356563+0000","last_fullsized":"2024-05-12T22:08:59.356563+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T07:04:48.336599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"88'5","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:59.369986+0000","last_change":"2024-05-12T22:06:33.966350+0000","last_active":"2024-05-12T22:08:59.369986+0000","last_peered":"2024-05-12T22:08:59.369986+0000","last_clean":"2024-05-12T22:08:59.369986+0000","last_became_active":"2024-05-12T22:06:33.965974+0000","last_became_peered":"2024-05-12T22:06:33.965974+0000","last_unstale":"2024-05-12T22:08:59.369986+0000","last_undegraded":"2024-05-12T22:08:59.369986+0000","last_fullsized":"2024-05-12T22:08:59.369986+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T02:08:16.889684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.776349+0000","last_change":"2024-05-12T22:06:32.944945+0000","last_active":"2024-05-12T22:08:58.776349+0000","last_peered":"2024-05-12T22:08:58.776349+0000","last_clean":"2024-05-12T22:08:58.776349+0000","last_became_active":"2024-05-12T22:06:32.944576+0000","last_became_peered":"2024-05-12T22:06:32.944576+0000","last_unstale":"2024-05-12T22:08:58.776349+0000","last_undegraded":"2024-05-12T22:08:58.776349+0000","last_fullsized":"2024-05-12T22:08:58.776349+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T07:29:14.544821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"88'5","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.649553+0000","last_change":"2024-05-12T22:06:33.966299+0000","last_active":"2024-05-12T22:08:58.649553+0000","last_peered":"2024-05-12T22:08:58.649553+0000","last_clean":"2024-05-12T22:08:58.649553+0000","last_became_active":"2024-05-12T22:06:33.965902+0000","last_became_peered":"2024-05-12T22:06:33.965902+0000","last_unstale":"2024-05-12T22:08:58.649553+0000","last_undegraded":"2024-05-12T22:08:58.649553+0000","last_fullsized":"2024-05-12T22:08:58.649553+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T22:37:51.476546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"86'1","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.967232+0000","last_change":"2024-05-12T22:06:32.937210+0000","last_active":"2024-05-12T22:08:58.967232+0000","last_peered":"2024-05-12T22:08:58.967232+0000","last_clean":"2024-05-12T22:08:58.967232+0000","last_became_active":"2024-05-12T22:06:32.936805+0000","last_became_peered":"2024-05-12T22:06:32.936805+0000","last_unstale":"2024-05-12T22:08:58.967232+0000","last_undegraded":"2024-05-12T22:08:58.967232+0000","last_fullsized":"2024-05-12T22:08:58.967232+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T06:14:20.825258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"88'7","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.649737+0000","last_change":"2024-05-12T22:06:33.983604+0000","last_active":"2024-05-12T22:08:58.649737+0000","last_peered":"2024-05-12T22:08:58.649737+0000","last_clean":"2024-05-12T22:08:58.649737+0000","last_became_active":"2024-05-12T22:06:33.983168+0000","last_became_peered":"2024-05-12T22:06:33.983168+0000","last_unstale":"2024-05-12T22:08:58.649737+0000","last_undegraded":"2024-05-12T22:08:58.649737+0000","last_fullsized":"2024-05-12T22:08:58.649737+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T05:54:55.854593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"88'8","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.786490+0000","last_change":"2024-05-12T22:06:33.968848+0000","last_active":"2024-05-12T22:08:58.786490+0000","last_peered":"2024-05-12T22:08:58.786490+0000","last_clean":"2024-05-12T22:08:58.786490+0000","last_became_active":"2024-05-12T22:06:33.968529+0000","last_became_peered":"2024-05-12T22:06:33.968529+0000","last_unstale":"2024-05-12T22:08:58.786490+0000","last_undegraded":"2024-05-12T22:08:58.786490+0000","last_fullsized":"2024-05-12T22:08:58.786490+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T03:00:41.129662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'6","reported_seq":169,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.974762+0000","last_change":"2024-05-12T22:06:32.946670+0000","last_active":"2024-05-12T22:08:58.974762+0000","last_peered":"2024-05-12T22:08:58.974762+0000","last_clean":"2024-05-12T22:08:58.974762+0000","last_became_active":"2024-05-12T22:06:32.946059+0000","last_became_peered":"2024-05-12T22:06:32.946059+0000","last_unstale":"2024-05-12T22:08:58.974762+0000","last_undegraded":"2024-05-12T22:08:58.974762+0000","last_fullsized":"2024-05-12T22:08:58.974762+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T22:35:56.116979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":141,"num_read_kb":141,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"88'8","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.648638+0000","last_change":"2024-05-12T22:06:33.962177+0000","last_active":"2024-05-12T22:08:58.648638+0000","last_peered":"2024-05-12T22:08:58.648638+0000","last_clean":"2024-05-12T22:08:58.648638+0000","last_became_active":"2024-05-12T22:06:33.960919+0000","last_became_peered":"2024-05-12T22:06:33.960919+0000","last_unstale":"2024-05-12T22:08:58.648638+0000","last_undegraded":"2024-05-12T22:08:58.648638+0000","last_fullsized":"2024-05-12T22:08:58.648638+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T00:39:54.566460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'3","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.966691+0000","last_change":"2024-05-12T22:06:32.927885+0000","last_active":"2024-05-12T22:08:58.966691+0000","last_peered":"2024-05-12T22:08:58.966691+0000","last_clean":"2024-05-12T22:08:58.966691+0000","last_became_active":"2024-05-12T22:06:32.927567+0000","last_became_peered":"2024-05-12T22:06:32.927567+0000","last_unstale":"2024-05-12T22:08:58.966691+0000","last_undegraded":"2024-05-12T22:08:58.966691+0000","last_fullsized":"2024-05-12T22:08:58.966691+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T07:13:50.811992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":355,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.966894+0000","last_change":"2024-05-12T22:05:17.683363+0000","last_active":"2024-05-12T22:08:58.966894+0000","last_peered":"2024-05-12T22:08:58.966894+0000","last_clean":"2024-05-12T22:08:58.966894+0000","last_became_active":"2024-05-12T22:05:17.682824+0000","last_became_peered":"2024-05-12T22:05:17.682824+0000","last_unstale":"2024-05-12T22:08:58.966894+0000","last_undegraded":"2024-05-12T22:08:58.966894+0000","last_fullsized":"2024-05-12T22:08:58.966894+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:01:32.304714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:01:32.304714+0000","last_clean_scrub_stamp":"2024-05-12T22:01:32.304714+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T06:35:22.474820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"88'5","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.966617+0000","last_change":"2024-05-12T22:06:33.965332+0000","last_active":"2024-05-12T22:08:58.966617+0000","last_peered":"2024-05-12T22:08:58.966617+0000","last_clean":"2024-05-12T22:08:58.966617+0000","last_became_active":"2024-05-12T22:06:33.964858+0000","last_became_peered":"2024-05-12T22:06:33.964858+0000","last_unstale":"2024-05-12T22:08:58.966617+0000","last_undegraded":"2024-05-12T22:08:58.966617+0000","last_fullsized":"2024-05-12T22:08:58.966617+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T05:14:09.825614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'3","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.967071+0000","last_change":"2024-05-12T22:06:32.929773+0000","last_active":"2024-05-12T22:08:58.967071+0000","last_peered":"2024-05-12T22:08:58.967071+0000","last_clean":"2024-05-12T22:08:58.967071+0000","last_became_active":"2024-05-12T22:06:32.929458+0000","last_became_peered":"2024-05-12T22:06:32.929458+0000","last_unstale":"2024-05-12T22:08:58.967071+0000","last_undegraded":"2024-05-12T22:08:58.967071+0000","last_fullsized":"2024-05-12T22:08:58.967071+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T01:30:42.853183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"86'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.779092+0000","last_change":"2024-05-12T22:06:32.947014+0000","last_active":"2024-05-12T22:08:58.779092+0000","last_peered":"2024-05-12T22:08:58.779092+0000","last_clean":"2024-05-12T22:08:58.779092+0000","last_became_active":"2024-05-12T22:06:32.946661+0000","last_became_peered":"2024-05-12T22:06:32.946661+0000","last_unstale":"2024-05-12T22:08:58.779092+0000","last_undegraded":"2024-05-12T22:08:58.779092+0000","last_fullsized":"2024-05-12T22:08:58.779092+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T23:09:19.378353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"88'9","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.967616+0000","last_change":"2024-05-12T22:06:33.980490+0000","last_active":"2024-05-12T22:08:58.967616+0000","last_peered":"2024-05-12T22:08:58.967616+0000","last_clean":"2024-05-12T22:08:58.967616+0000","last_became_active":"2024-05-12T22:06:33.980133+0000","last_became_peered":"2024-05-12T22:06:33.980133+0000","last_unstale":"2024-05-12T22:08:58.967616+0000","last_undegraded":"2024-05-12T22:08:58.967616+0000","last_fullsized":"2024-05-12T22:08:58.967616+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T00:28:10.461428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"89'46","reported_seq":91,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.918040+0000","last_change":"2024-05-12T22:06:32.943152+0000","last_active":"2024-05-12T22:08:58.918040+0000","last_peered":"2024-05-12T22:08:58.918040+0000","last_clean":"2024-05-12T22:08:58.918040+0000","last_became_active":"2024-05-12T22:06:32.942769+0000","last_became_peered":"2024-05-12T22:06:32.942769+0000","last_unstale":"2024-05-12T22:08:58.918040+0000","last_undegraded":"2024-05-12T22:08:58.918040+0000","last_fullsized":"2024-05-12T22:08:58.918040+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T01:23:24.955739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":82,"num_write_kb":837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"88'5","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.649177+0000","last_change":"2024-05-12T22:06:33.964081+0000","last_active":"2024-05-12T22:08:58.649177+0000","last_peered":"2024-05-12T22:08:58.649177+0000","last_clean":"2024-05-12T22:08:58.649177+0000","last_became_active":"2024-05-12T22:06:33.963054+0000","last_became_peered":"2024-05-12T22:06:33.963054+0000","last_unstale":"2024-05-12T22:08:58.649177+0000","last_undegraded":"2024-05-12T22:08:58.649177+0000","last_fullsized":"2024-05-12T22:08:58.649177+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T09:40:03.396200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"86'4","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.966937+0000","last_change":"2024-05-12T22:06:32.936432+0000","last_active":"2024-05-12T22:08:58.966937+0000","last_peered":"2024-05-12T22:08:58.966937+0000","last_clean":"2024-05-12T22:08:58.966937+0000","last_became_active":"2024-05-12T22:06:32.936094+0000","last_became_peered":"2024-05-12T22:06:32.936094+0000","last_unstale":"2024-05-12T22:08:58.966937+0000","last_undegraded":"2024-05-12T22:08:58.966937+0000","last_fullsized":"2024-05-12T22:08:58.966937+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T02:40:31.346118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"88'7","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.974922+0000","last_change":"2024-05-12T22:06:33.985953+0000","last_active":"2024-05-12T22:08:58.974922+0000","last_peered":"2024-05-12T22:08:58.974922+0000","last_clean":"2024-05-12T22:08:58.974922+0000","last_became_active":"2024-05-12T22:06:33.984864+0000","last_became_peered":"2024-05-12T22:06:33.984864+0000","last_unstale":"2024-05-12T22:08:58.974922+0000","last_undegraded":"2024-05-12T22:08:58.974922+0000","last_fullsized":"2024-05-12T22:08:58.974922+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T07:28:22.357841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.981163+0000","last_change":"2024-05-12T22:06:32.934510+0000","last_active":"2024-05-12T22:08:58.981163+0000","last_peered":"2024-05-12T22:08:58.981163+0000","last_clean":"2024-05-12T22:08:58.981163+0000","last_became_active":"2024-05-12T22:06:32.934222+0000","last_became_peered":"2024-05-12T22:06:32.934222+0000","last_unstale":"2024-05-12T22:08:58.981163+0000","last_undegraded":"2024-05-12T22:08:58.981163+0000","last_fullsized":"2024-05-12T22:08:58.981163+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T00:57:44.321629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"88'2","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.778837+0000","last_change":"2024-05-12T22:06:33.963870+0000","last_active":"2024-05-12T22:08:58.778837+0000","last_peered":"2024-05-12T22:08:58.778837+0000","last_clean":"2024-05-12T22:08:58.778837+0000","last_became_active":"2024-05-12T22:06:33.963533+0000","last_became_peered":"2024-05-12T22:06:33.963533+0000","last_unstale":"2024-05-12T22:08:58.778837+0000","last_undegraded":"2024-05-12T22:08:58.778837+0000","last_fullsized":"2024-05-12T22:08:58.778837+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T08:56:59.495469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"86'3","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.648478+0000","last_change":"2024-05-12T22:06:32.939738+0000","last_active":"2024-05-12T22:08:58.648478+0000","last_peered":"2024-05-12T22:08:58.648478+0000","last_clean":"2024-05-12T22:08:58.648478+0000","last_became_active":"2024-05-12T22:06:32.939365+0000","last_became_peered":"2024-05-12T22:06:32.939365+0000","last_unstale":"2024-05-12T22:08:58.648478+0000","last_undegraded":"2024-05-12T22:08:58.648478+0000","last_fullsized":"2024-05-12T22:08:58.648478+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T02:19:31.180834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"88'9","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:59.369732+0000","last_change":"2024-05-12T22:06:33.975064+0000","last_active":"2024-05-12T22:08:59.369732+0000","last_peered":"2024-05-12T22:08:59.369732+0000","last_clean":"2024-05-12T22:08:59.369732+0000","last_became_active":"2024-05-12T22:06:33.974613+0000","last_became_peered":"2024-05-12T22:06:33.974613+0000","last_unstale":"2024-05-12T22:08:59.369732+0000","last_undegraded":"2024-05-12T22:08:59.369732+0000","last_fullsized":"2024-05-12T22:08:59.369732+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T08:46:27.343777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"86'1","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.651327+0000","last_change":"2024-05-12T22:06:32.931148+0000","last_active":"2024-05-12T22:08:58.651327+0000","last_peered":"2024-05-12T22:08:58.651327+0000","last_clean":"2024-05-12T22:08:58.651327+0000","last_became_active":"2024-05-12T22:06:32.930717+0000","last_became_peered":"2024-05-12T22:06:32.930717+0000","last_unstale":"2024-05-12T22:08:58.651327+0000","last_undegraded":"2024-05-12T22:08:58.651327+0000","last_fullsized":"2024-05-12T22:08:58.651327+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T07:26:20.915467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"88'8","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.648890+0000","last_change":"2024-05-12T22:06:33.988390+0000","last_active":"2024-05-12T22:08:58.648890+0000","last_peered":"2024-05-12T22:08:58.648890+0000","last_clean":"2024-05-12T22:08:58.648890+0000","last_became_active":"2024-05-12T22:06:33.986970+0000","last_became_peered":"2024-05-12T22:06:33.986970+0000","last_unstale":"2024-05-12T22:08:58.648890+0000","last_undegraded":"2024-05-12T22:08:58.648890+0000","last_fullsized":"2024-05-12T22:08:58.648890+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T05:03:31.930261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.779155+0000","last_change":"2024-05-12T22:06:46.798123+0000","last_active":"2024-05-12T22:08:58.779155+0000","last_peered":"2024-05-12T22:08:58.779155+0000","last_clean":"2024-05-12T22:08:58.779155+0000","last_became_active":"2024-05-12T22:06:46.791029+0000","last_became_peered":"2024-05-12T22:06:46.791029+0000","last_unstale":"2024-05-12T22:08:58.779155+0000","last_undegraded":"2024-05-12T22:08:58.779155+0000","last_fullsized":"2024-05-12T22:08:58.779155+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T06:31:25.108659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,7],"acting":[8,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"88'5","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.967282+0000","last_change":"2024-05-12T22:06:33.968256+0000","last_active":"2024-05-12T22:08:58.967282+0000","last_peered":"2024-05-12T22:08:58.967282+0000","last_clean":"2024-05-12T22:08:58.967282+0000","last_became_active":"2024-05-12T22:06:33.967885+0000","last_became_peered":"2024-05-12T22:06:33.967885+0000","last_unstale":"2024-05-12T22:08:58.967282+0000","last_undegraded":"2024-05-12T22:08:58.967282+0000","last_fullsized":"2024-05-12T22:08:58.967282+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T09:12:40.687367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"86'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.651789+0000","last_change":"2024-05-12T22:06:32.944736+0000","last_active":"2024-05-12T22:08:58.651789+0000","last_peered":"2024-05-12T22:08:58.651789+0000","last_clean":"2024-05-12T22:08:58.651789+0000","last_became_active":"2024-05-12T22:06:32.944269+0000","last_became_peered":"2024-05-12T22:06:32.944269+0000","last_unstale":"2024-05-12T22:08:58.651789+0000","last_undegraded":"2024-05-12T22:08:58.651789+0000","last_fullsized":"2024-05-12T22:08:58.651789+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T22:56:24.391267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"88'7","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.967821+0000","last_change":"2024-05-12T22:06:33.968360+0000","last_active":"2024-05-12T22:08:58.967821+0000","last_peered":"2024-05-12T22:08:58.967821+0000","last_clean":"2024-05-12T22:08:58.967821+0000","last_became_active":"2024-05-12T22:06:33.967967+0000","last_became_peered":"2024-05-12T22:06:33.967967+0000","last_unstale":"2024-05-12T22:08:58.967821+0000","last_undegraded":"2024-05-12T22:08:58.967821+0000","last_fullsized":"2024-05-12T22:08:58.967821+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T00:41:12.693858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"89'39","reported_seq":69,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.914994+0000","last_change":"2024-05-12T22:06:32.935993+0000","last_active":"2024-05-12T22:08:58.914994+0000","last_peered":"2024-05-12T22:08:58.914994+0000","last_clean":"2024-05-12T22:08:58.914994+0000","last_became_active":"2024-05-12T22:06:32.935615+0000","last_became_peered":"2024-05-12T22:06:32.935615+0000","last_unstale":"2024-05-12T22:08:58.914994+0000","last_undegraded":"2024-05-12T22:08:58.914994+0000","last_fullsized":"2024-05-12T22:08:58.914994+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T00:43:39.254637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"88'7","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.967474+0000","last_change":"2024-05-12T22:06:33.966110+0000","last_active":"2024-05-12T22:08:58.967474+0000","last_peered":"2024-05-12T22:08:58.967474+0000","last_clean":"2024-05-12T22:08:58.967474+0000","last_became_active":"2024-05-12T22:06:33.965755+0000","last_became_peered":"2024-05-12T22:06:33.965755+0000","last_unstale":"2024-05-12T22:08:58.967474+0000","last_undegraded":"2024-05-12T22:08:58.967474+0000","last_fullsized":"2024-05-12T22:08:58.967474+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T22:18:53.941335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.968104+0000","last_change":"2024-05-12T22:06:32.942229+0000","last_active":"2024-05-12T22:08:58.968104+0000","last_peered":"2024-05-12T22:08:58.968104+0000","last_clean":"2024-05-12T22:08:58.968104+0000","last_became_active":"2024-05-12T22:06:32.941826+0000","last_became_peered":"2024-05-12T22:06:32.941826+0000","last_unstale":"2024-05-12T22:08:58.968104+0000","last_undegraded":"2024-05-12T22:08:58.968104+0000","last_fullsized":"2024-05-12T22:08:58.968104+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T03:52:34.886318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"88'9","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.967759+0000","last_change":"2024-05-12T22:06:33.963081+0000","last_active":"2024-05-12T22:08:58.967759+0000","last_peered":"2024-05-12T22:08:58.967759+0000","last_clean":"2024-05-12T22:08:58.967759+0000","last_became_active":"2024-05-12T22:06:33.962696+0000","last_became_peered":"2024-05-12T22:06:33.962696+0000","last_unstale":"2024-05-12T22:08:58.967759+0000","last_undegraded":"2024-05-12T22:08:58.967759+0000","last_fullsized":"2024-05-12T22:08:58.967759+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T05:55:45.581823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"88'9","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:59.369801+0000","last_change":"2024-05-12T22:06:33.968865+0000","last_active":"2024-05-12T22:08:59.369801+0000","last_peered":"2024-05-12T22:08:59.369801+0000","last_clean":"2024-05-12T22:08:59.369801+0000","last_became_active":"2024-05-12T22:06:33.968185+0000","last_became_peered":"2024-05-12T22:06:33.968185+0000","last_unstale":"2024-05-12T22:08:59.369801+0000","last_undegraded":"2024-05-12T22:08:59.369801+0000","last_fullsized":"2024-05-12T22:08:59.369801+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T01:23:41.083778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"88'12","reported_seq":32,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.651171+0000","last_change":"2024-05-12T22:06:33.964786+0000","last_active":"2024-05-12T22:08:58.651171+0000","last_peered":"2024-05-12T22:08:58.651171+0000","last_clean":"2024-05-12T22:08:58.651171+0000","last_became_active":"2024-05-12T22:06:33.964227+0000","last_became_peered":"2024-05-12T22:06:33.964227+0000","last_unstale":"2024-05-12T22:08:58.651171+0000","last_undegraded":"2024-05-12T22:08:58.651171+0000","last_fullsized":"2024-05-12T22:08:58.651171+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T02:53:42.739455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"88'8","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.651132+0000","last_change":"2024-05-12T22:06:33.978857+0000","last_active":"2024-05-12T22:08:58.651132+0000","last_peered":"2024-05-12T22:08:58.651132+0000","last_clean":"2024-05-12T22:08:58.651132+0000","last_became_active":"2024-05-12T22:06:33.978531+0000","last_became_peered":"2024-05-12T22:06:33.978531+0000","last_unstale":"2024-05-12T22:08:58.651132+0000","last_undegraded":"2024-05-12T22:08:58.651132+0000","last_fullsized":"2024-05-12T22:08:58.651132+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T02:51:05.609267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"88'7","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.974185+0000","last_change":"2024-05-12T22:06:33.980662+0000","last_active":"2024-05-12T22:08:58.974185+0000","last_peered":"2024-05-12T22:08:58.974185+0000","last_clean":"2024-05-12T22:08:58.974185+0000","last_became_active":"2024-05-12T22:06:33.980332+0000","last_became_peered":"2024-05-12T22:06:33.980332+0000","last_unstale":"2024-05-12T22:08:58.974185+0000","last_undegraded":"2024-05-12T22:08:58.974185+0000","last_fullsized":"2024-05-12T22:08:58.974185+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T06:21:59.715792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"88'4","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.967687+0000","last_change":"2024-05-12T22:06:33.987338+0000","last_active":"2024-05-12T22:08:58.967687+0000","last_peered":"2024-05-12T22:08:58.967687+0000","last_clean":"2024-05-12T22:08:58.967687+0000","last_became_active":"2024-05-12T22:06:33.986942+0000","last_became_peered":"2024-05-12T22:06:33.986942+0000","last_unstale":"2024-05-12T22:08:58.967687+0000","last_undegraded":"2024-05-12T22:08:58.967687+0000","last_fullsized":"2024-05-12T22:08:58.967687+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T01:02:05.137558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"88'6","reported_seq":32,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:59.370006+0000","last_change":"2024-05-12T22:06:46.796752+0000","last_active":"2024-05-12T22:08:59.370006+0000","last_peered":"2024-05-12T22:08:59.370006+0000","last_clean":"2024-05-12T22:08:59.370006+0000","last_became_active":"2024-05-12T22:06:46.796369+0000","last_became_peered":"2024-05-12T22:06:46.796369+0000","last_unstale":"2024-05-12T22:08:59.370006+0000","last_undegraded":"2024-05-12T22:08:59.370006+0000","last_fullsized":"2024-05-12T22:08:59.370006+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T08:23:24.685495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,9],"acting":[11,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"88'15","reported_seq":35,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.651395+0000","last_change":"2024-05-12T22:06:33.963902+0000","last_active":"2024-05-12T22:08:58.651395+0000","last_peered":"2024-05-12T22:08:58.651395+0000","last_clean":"2024-05-12T22:08:58.651395+0000","last_became_active":"2024-05-12T22:06:33.963436+0000","last_became_peered":"2024-05-12T22:06:33.963436+0000","last_unstale":"2024-05-12T22:08:58.651395+0000","last_undegraded":"2024-05-12T22:08:58.651395+0000","last_fullsized":"2024-05-12T22:08:58.651395+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T02:51:56.415167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"88'2","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.649008+0000","last_change":"2024-05-12T22:06:46.794434+0000","last_active":"2024-05-12T22:08:58.649008+0000","last_peered":"2024-05-12T22:08:58.649008+0000","last_clean":"2024-05-12T22:08:58.649008+0000","last_became_active":"2024-05-12T22:06:46.793147+0000","last_became_peered":"2024-05-12T22:06:46.793147+0000","last_unstale":"2024-05-12T22:08:58.649008+0000","last_undegraded":"2024-05-12T22:08:58.649008+0000","last_fullsized":"2024-05-12T22:08:58.649008+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T23:10:39.529863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,8,2],"acting":[3,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"88'6","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.786430+0000","last_change":"2024-05-12T22:06:33.973490+0000","last_active":"2024-05-12T22:08:58.786430+0000","last_peered":"2024-05-12T22:08:58.786430+0000","last_clean":"2024-05-12T22:08:58.786430+0000","last_became_active":"2024-05-12T22:06:33.973161+0000","last_became_peered":"2024-05-12T22:06:33.973161+0000","last_unstale":"2024-05-12T22:08:58.786430+0000","last_undegraded":"2024-05-12T22:08:58.786430+0000","last_fullsized":"2024-05-12T22:08:58.786430+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T03:29:36.972631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"88'8","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.648590+0000","last_change":"2024-05-12T22:06:33.957320+0000","last_active":"2024-05-12T22:08:58.648590+0000","last_peered":"2024-05-12T22:08:58.648590+0000","last_clean":"2024-05-12T22:08:58.648590+0000","last_became_active":"2024-05-12T22:06:33.956837+0000","last_became_peered":"2024-05-12T22:06:33.956837+0000","last_unstale":"2024-05-12T22:08:58.648590+0000","last_undegraded":"2024-05-12T22:08:58.648590+0000","last_fullsized":"2024-05-12T22:08:58.648590+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T01:20:51.018175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"88'10","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.981497+0000","last_change":"2024-05-12T22:06:33.968363+0000","last_active":"2024-05-12T22:08:58.981497+0000","last_peered":"2024-05-12T22:08:58.981497+0000","last_clean":"2024-05-12T22:08:58.981497+0000","last_became_active":"2024-05-12T22:06:33.968031+0000","last_became_peered":"2024-05-12T22:06:33.968031+0000","last_unstale":"2024-05-12T22:08:58.981497+0000","last_undegraded":"2024-05-12T22:08:58.981497+0000","last_fullsized":"2024-05-12T22:08:58.981497+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T00:14:07.124058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"88'11","reported_seq":31,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.967933+0000","last_change":"2024-05-12T22:06:33.984823+0000","last_active":"2024-05-12T22:08:58.967933+0000","last_peered":"2024-05-12T22:08:58.967933+0000","last_clean":"2024-05-12T22:08:58.967933+0000","last_became_active":"2024-05-12T22:06:33.984430+0000","last_became_peered":"2024-05-12T22:06:33.984430+0000","last_unstale":"2024-05-12T22:08:58.967933+0000","last_undegraded":"2024-05-12T22:08:58.967933+0000","last_fullsized":"2024-05-12T22:08:58.967933+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T01:30:18.109265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"88'8","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:59.369891+0000","last_change":"2024-05-12T22:06:33.962660+0000","last_active":"2024-05-12T22:08:59.369891+0000","last_peered":"2024-05-12T22:08:59.369891+0000","last_clean":"2024-05-12T22:08:59.369891+0000","last_became_active":"2024-05-12T22:06:33.962284+0000","last_became_peered":"2024-05-12T22:06:33.962284+0000","last_unstale":"2024-05-12T22:08:59.369891+0000","last_undegraded":"2024-05-12T22:08:59.369891+0000","last_fullsized":"2024-05-12T22:08:59.369891+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T01:08:53.888992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":598250,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":972,"num_read_kb":596,"num_write":294,"num_write_kb":1443,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2027520,"data_stored":1794750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":232,"ondisk_log_size":232,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351018,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27948,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717204,"statfs":{"total":95995035648,"available":95966416896,"internally_reserved":0,"allocated":954368,"data_stored":459830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612654,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28840,"kb_used_data":1764,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716312,"statfs":{"total":95995035648,"available":95965503488,"internally_reserved":0,"allocated":1806336,"data_stored":1260157,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776179,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28364,"kb_used_data":1344,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716788,"statfs":{"total":95995035648,"available":95965990912,"internally_reserved":0,"allocated":1376256,"data_stored":861627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037818,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32524,"kb_used_data":1344,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712628,"statfs":{"total":95995035648,"available":95961731072,"internally_reserved":0,"allocated":1376256,"data_stored":861547,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299454,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28556,"kb_used_data":1536,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716596,"statfs":{"total":95995035648,"available":95965794304,"internally_reserved":0,"allocated":1572864,"data_stored":1061525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462981,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":970752,"data_stored":467800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574944,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36356,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708796,"statfs":{"total":95995035648,"available":95957807104,"internally_reserved":0,"allocated":1040384,"data_stored":535180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738469,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":991232,"data_stored":469071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411419,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28060,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717092,"statfs":{"total":95995035648,"available":95966302208,"internally_reserved":0,"allocated":1064960,"data_stored":536280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149781,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28496,"kb_used_data":1480,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716656,"statfs":{"total":95995035648,"available":95965855744,"internally_reserved":0,"allocated":1515520,"data_stored":992927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888142,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27972,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717180,"statfs":{"total":95995035648,"available":95966392320,"internally_reserved":0,"allocated":974848,"data_stored":469551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724618,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28552,"kb_used_data":1536,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716600,"statfs":{"total":95995035648,"available":95965798400,"internally_reserved":0,"allocated":1572864,"data_stored":1051159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":67587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":69860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":544768,"data_stored":525100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":133999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3447,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":413696,"data_stored":393954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":413696,"data_stored":394034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":202079,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-12T22:09:04.856 INFO:tasks.fwd_scrub.ceph_manager:clean! 2024-05-12T22:09:04.856 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-12T22:09:05.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:05.075+0000 7f6507e00640 1 -- 172.21.15.28:0/20485316 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6508077430 msgr2=0x7f6508077890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:05.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:05.075+0000 7f6507e00640 1 --2- 172.21.15.28:0/20485316 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6508077430 0x7f6508077890 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f64f0009900 tx=0x7f64f002a410 comp rx=0 tx=0).stop 2024-05-12T22:09:05.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:05.075+0000 7f6507e00640 1 -- 172.21.15.28:0/20485316 shutdown_connections 2024-05-12T22:09:05.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:05.075+0000 7f6507e00640 1 --2- 172.21.15.28:0/20485316 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6508077dd0 0x7f6508078670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:05.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:05.075+0000 7f6507e00640 1 --2- 172.21.15.28:0/20485316 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6508077430 0x7f6508077890 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:05.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:05.075+0000 7f6507e00640 1 --2- 172.21.15.28:0/20485316 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65080761f0 0x7f6508076610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:05.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:05.075+0000 7f6507e00640 1 -- 172.21.15.28:0/20485316 >> 172.21.15.28:0/20485316 conn(0x7f65080fee40 msgr2=0x7f6508101260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:09:05.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:05.075+0000 7f6507e00640 1 -- 172.21.15.28:0/20485316 shutdown_connections 2024-05-12T22:09:05.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:05.075+0000 7f6507e00640 1 -- 172.21.15.28:0/20485316 wait complete. 2024-05-12T22:09:05.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:05.077+0000 7f6507e00640 1 Processor -- start 2024-05-12T22:09:05.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:05.078+0000 7f6507e00640 1 -- start start 2024-05-12T22:09:05.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:05.078+0000 7f6507e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f65080761f0 0x7f650819df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:05.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:05.078+0000 7f6507e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6508077430 0x7f650819e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:05.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:05.078+0000 7f6507e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6508077dd0 0x7f65081a5520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:05.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:05.078+0000 7f6507e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6508113980 con 0x7f6508077dd0 2024-05-12T22:09:05.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:05.078+0000 7f6507e00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f6508113800 con 0x7f6508077430 2024-05-12T22:09:05.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:05.078+0000 7f6507e00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f6508113b00 con 0x7f65080761f0 2024-05-12T22:09:05.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:05.079+0000 7f64ffe00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6508077430 0x7f650819e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:05.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:05.079+0000 7f6506a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f65080761f0 0x7f650819df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:05.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:05.079+0000 7f64ffe00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6508077430 0x7f650819e4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:49442/0 (socket says 172.21.15.28:49442) 2024-05-12T22:09:05.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:05.079+0000 7f64ffe00640 1 -- 172.21.15.28:0/4238226232 learned_addr learned my addr 172.21.15.28:0/4238226232 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:09:05.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:05.079+0000 7f6507400640 1 --2- 172.21.15.28:0/4238226232 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6508077dd0 0x7f65081a5520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:05.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:05.079+0000 7f64ffe00640 1 -- 172.21.15.28:0/4238226232 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f65080761f0 msgr2=0x7f650819df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:05.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:05.079+0000 7f64ffe00640 1 --2- 172.21.15.28:0/4238226232 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f65080761f0 0x7f650819df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:05.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:05.079+0000 7f64ffe00640 1 -- 172.21.15.28:0/4238226232 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6508077dd0 msgr2=0x7f65081a5520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:05.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:05.079+0000 7f64ffe00640 1 --2- 172.21.15.28:0/4238226232 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6508077dd0 0x7f65081a5520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:05.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:05.079+0000 7f64ffe00640 1 -- 172.21.15.28:0/4238226232 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65081a5b90 con 0x7f6508077430 2024-05-12T22:09:05.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:05.079+0000 7f6506a00640 1 --2- 172.21.15.28:0/4238226232 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f65080761f0 0x7f650819df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:09:05.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:05.079+0000 7f6507400640 1 --2- 172.21.15.28:0/4238226232 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6508077dd0 0x7f65081a5520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:09:05.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:05.079+0000 7f64ffe00640 1 --2- 172.21.15.28:0/4238226232 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6508077430 0x7f650819e4b0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f64f0031440 tx=0x7f64f0032ce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:09:05.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:05.080+0000 7f64ff400640 1 -- 172.21.15.28:0/4238226232 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64f003f030 con 0x7f6508077430 2024-05-12T22:09:05.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:05.080+0000 7f64ff400640 1 -- 172.21.15.28:0/4238226232 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f64f0003ce0 con 0x7f6508077430 2024-05-12T22:09:05.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:05.080+0000 7f64ff400640 1 -- 172.21.15.28:0/4238226232 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64f0035070 con 0x7f6508077430 2024-05-12T22:09:05.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:05.080+0000 7f6507e00640 1 -- 172.21.15.28:0/4238226232 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65081a5e20 con 0x7f6508077430 2024-05-12T22:09:05.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:05.080+0000 7f6507e00640 1 -- 172.21.15.28:0/4238226232 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65081a6290 con 0x7f6508077430 2024-05-12T22:09:05.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:05.081+0000 7f6507e00640 1 -- 172.21.15.28:0/4238226232 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6508110360 con 0x7f6508077430 2024-05-12T22:09:05.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:05.083+0000 7f64ff400640 1 -- 172.21.15.28:0/4238226232 <== mon.2 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 25) ==== 138060+0+0 (secure 0 0 0) 0x7f64f0042030 con 0x7f6508077430 2024-05-12T22:09:05.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:05.085+0000 7f64ff400640 1 --2- 172.21.15.28:0/4238226232 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f64ec0a2ea0 0x7f64ec0a5360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:05.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:05.086+0000 7f64ff400640 1 -- 172.21.15.28:0/4238226232 <== mon.2 v2:172.21.15.86:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f64f00efad0 con 0x7f6508077430 2024-05-12T22:09:05.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:05.086+0000 7f6506a00640 1 --2- 172.21.15.28:0/4238226232 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f64ec0a2ea0 0x7f64ec0a5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:05.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:05.087+0000 7f6506a00640 1 --2- 172.21.15.28:0/4238226232 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f64ec0a2ea0 0x7f64ec0a5360 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f64f8009fc0 tx=0x7f64f8005f90 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:09:05.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:05.088+0000 7f64ff400640 1 -- 172.21.15.28:0/4238226232 <== mon.2 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f64f00bc540 con 0x7f6508077430 2024-05-12T22:09:05.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:05.335+0000 7f6507e00640 1 -- 172.21.15.28:0/4238226232 --> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f6508061c70 con 0x7f64ec0a2ea0 2024-05-12T22:09:05.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:05.344+0000 7f64ff400640 1 -- 172.21.15.28:0/4238226232 <== mgr.25189 v2:172.21.15.28:6800/3177640985 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+257774 (secure 0 0 0) 0x7f6508061c70 con 0x7f64ec0a2ea0 2024-05-12T22:09:05.350 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-12T22:09:05.352 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-05-12T22:09:05.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:05.348+0000 7f6507e00640 1 -- 172.21.15.28:0/4238226232 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f64ec0a2ea0 msgr2=0x7f64ec0a5360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:05.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:05.348+0000 7f6507e00640 1 --2- 172.21.15.28:0/4238226232 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f64ec0a2ea0 0x7f64ec0a5360 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f64f8009fc0 tx=0x7f64f8005f90 comp rx=0 tx=0).stop 2024-05-12T22:09:05.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:05.348+0000 7f6507e00640 1 -- 172.21.15.28:0/4238226232 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6508077430 msgr2=0x7f650819e4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:05.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:05.348+0000 7f6507e00640 1 --2- 172.21.15.28:0/4238226232 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6508077430 0x7f650819e4b0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f64f0031440 tx=0x7f64f0032ce0 comp rx=0 tx=0).stop 2024-05-12T22:09:05.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:05.348+0000 7f6507e00640 1 -- 172.21.15.28:0/4238226232 shutdown_connections 2024-05-12T22:09:05.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:05.348+0000 7f6507e00640 1 --2- 172.21.15.28:0/4238226232 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f64ec0a2ea0 0x7f64ec0a5360 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:05.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:05.348+0000 7f6507e00640 1 --2- 172.21.15.28:0/4238226232 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6508077dd0 0x7f65081a5520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:05.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:05.348+0000 7f6507e00640 1 --2- 172.21.15.28:0/4238226232 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6508077430 0x7f650819e4b0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:05.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:05.348+0000 7f6507e00640 1 --2- 172.21.15.28:0/4238226232 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f65080761f0 0x7f650819df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:05.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:05.348+0000 7f6507e00640 1 -- 172.21.15.28:0/4238226232 >> 172.21.15.28:0/4238226232 conn(0x7f65080fee40 msgr2=0x7f6508100470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:09:05.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:05.348+0000 7f6507e00640 1 -- 172.21.15.28:0/4238226232 shutdown_connections 2024-05-12T22:09:05.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:05.348+0000 7f6507e00640 1 -- 172.21.15.28:0/4238226232 wait complete. 2024-05-12T22:09:05.368 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":6,"stamp":"2024-05-12T22:09:04.669911+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1188852,"num_objects":66,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":1108,"num_read_kb":715,"num_write":477,"num_write_kb":4383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":1181318,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":871,"ondisk_log_size":871,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":351624,"kb_used_data":14860,"kb_used_omap":18,"kb_used_meta":336557,"kb_avail":1124590200,"statfs":{"total":1151940427776,"available":1151580364800,"internally_reserved":0,"allocated":15216640,"data_stored":9026654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":344634681},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.976276"},"pg_stats":[{"pgid":"3.3f","version":"88'8","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:59.370219+0000","last_change":"2024-05-12T22:06:33.982079+0000","last_active":"2024-05-12T22:08:59.370219+0000","last_peered":"2024-05-12T22:08:59.370219+0000","last_clean":"2024-05-12T22:08:59.370219+0000","last_became_active":"2024-05-12T22:06:33.981703+0000","last_became_peered":"2024-05-12T22:06:33.981703+0000","last_unstale":"2024-05-12T22:08:59.370219+0000","last_undegraded":"2024-05-12T22:08:59.370219+0000","last_fullsized":"2024-05-12T22:08:59.370219+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T08:52:47.157067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"88'4","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.974850+0000","last_change":"2024-05-12T22:06:33.984194+0000","last_active":"2024-05-12T22:08:58.974850+0000","last_peered":"2024-05-12T22:08:58.974850+0000","last_clean":"2024-05-12T22:08:58.974850+0000","last_became_active":"2024-05-12T22:06:33.979787+0000","last_became_peered":"2024-05-12T22:06:33.979787+0000","last_unstale":"2024-05-12T22:08:58.974850+0000","last_undegraded":"2024-05-12T22:08:58.974850+0000","last_fullsized":"2024-05-12T22:08:58.974850+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T07:36:46.588641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"88'7","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.974018+0000","last_change":"2024-05-12T22:06:33.982428+0000","last_active":"2024-05-12T22:08:58.974018+0000","last_peered":"2024-05-12T22:08:58.974018+0000","last_clean":"2024-05-12T22:08:58.974018+0000","last_became_active":"2024-05-12T22:06:33.979563+0000","last_became_peered":"2024-05-12T22:06:33.979563+0000","last_unstale":"2024-05-12T22:08:58.974018+0000","last_undegraded":"2024-05-12T22:08:58.974018+0000","last_fullsized":"2024-05-12T22:08:58.974018+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T06:26:12.586519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"88'8","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.776473+0000","last_change":"2024-05-12T22:06:46.796152+0000","last_active":"2024-05-12T22:08:58.776473+0000","last_peered":"2024-05-12T22:08:58.776473+0000","last_clean":"2024-05-12T22:08:58.776473+0000","last_became_active":"2024-05-12T22:06:46.795484+0000","last_became_peered":"2024-05-12T22:06:46.795484+0000","last_unstale":"2024-05-12T22:08:58.776473+0000","last_undegraded":"2024-05-12T22:08:58.776473+0000","last_fullsized":"2024-05-12T22:08:58.776473+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T03:13:45.901507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,2],"acting":[9,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.3b","version":"88'4","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.778898+0000","last_change":"2024-05-12T22:06:33.967492+0000","last_active":"2024-05-12T22:08:58.778898+0000","last_peered":"2024-05-12T22:08:58.778898+0000","last_clean":"2024-05-12T22:08:58.778898+0000","last_became_active":"2024-05-12T22:06:33.967146+0000","last_became_peered":"2024-05-12T22:06:33.967146+0000","last_unstale":"2024-05-12T22:08:58.778898+0000","last_undegraded":"2024-05-12T22:08:58.778898+0000","last_fullsized":"2024-05-12T22:08:58.778898+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T07:31:05.946473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"88'6","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.649499+0000","last_change":"2024-05-12T22:06:33.973653+0000","last_active":"2024-05-12T22:08:58.649499+0000","last_peered":"2024-05-12T22:08:58.649499+0000","last_clean":"2024-05-12T22:08:58.649499+0000","last_became_active":"2024-05-12T22:06:33.973290+0000","last_became_peered":"2024-05-12T22:06:33.973290+0000","last_unstale":"2024-05-12T22:08:58.649499+0000","last_undegraded":"2024-05-12T22:08:58.649499+0000","last_fullsized":"2024-05-12T22:08:58.649499+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T03:07:21.933141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"88'5","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.967170+0000","last_change":"2024-05-12T22:06:33.973807+0000","last_active":"2024-05-12T22:08:58.967170+0000","last_peered":"2024-05-12T22:08:58.967170+0000","last_clean":"2024-05-12T22:08:58.967170+0000","last_became_active":"2024-05-12T22:06:33.973466+0000","last_became_peered":"2024-05-12T22:06:33.973466+0000","last_unstale":"2024-05-12T22:08:58.967170+0000","last_undegraded":"2024-05-12T22:08:58.967170+0000","last_fullsized":"2024-05-12T22:08:58.967170+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T02:31:13.608309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"88'9","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.648390+0000","last_change":"2024-05-12T22:06:33.967356+0000","last_active":"2024-05-12T22:08:58.648390+0000","last_peered":"2024-05-12T22:08:58.648390+0000","last_clean":"2024-05-12T22:08:58.648390+0000","last_became_active":"2024-05-12T22:06:33.966054+0000","last_became_peered":"2024-05-12T22:06:33.966054+0000","last_unstale":"2024-05-12T22:08:58.648390+0000","last_undegraded":"2024-05-12T22:08:58.648390+0000","last_fullsized":"2024-05-12T22:08:58.648390+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T23:54:08.285720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"88'11","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.981661+0000","last_change":"2024-05-12T22:06:33.968782+0000","last_active":"2024-05-12T22:08:58.981661+0000","last_peered":"2024-05-12T22:08:58.981661+0000","last_clean":"2024-05-12T22:08:58.981661+0000","last_became_active":"2024-05-12T22:06:33.968435+0000","last_became_peered":"2024-05-12T22:06:33.968435+0000","last_unstale":"2024-05-12T22:08:58.981661+0000","last_undegraded":"2024-05-12T22:08:58.981661+0000","last_fullsized":"2024-05-12T22:08:58.981661+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T07:01:21.056777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"88'16","reported_seq":36,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.649922+0000","last_change":"2024-05-12T22:06:33.962018+0000","last_active":"2024-05-12T22:08:58.649922+0000","last_peered":"2024-05-12T22:08:58.649922+0000","last_clean":"2024-05-12T22:08:58.649922+0000","last_became_active":"2024-05-12T22:06:33.961583+0000","last_became_peered":"2024-05-12T22:06:33.961583+0000","last_unstale":"2024-05-12T22:08:58.649922+0000","last_undegraded":"2024-05-12T22:08:58.649922+0000","last_fullsized":"2024-05-12T22:08:58.649922+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T00:16:40.483194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"88'6","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.967780+0000","last_change":"2024-05-12T22:06:33.965776+0000","last_active":"2024-05-12T22:08:58.967780+0000","last_peered":"2024-05-12T22:08:58.967780+0000","last_clean":"2024-05-12T22:08:58.967780+0000","last_became_active":"2024-05-12T22:06:33.965439+0000","last_became_peered":"2024-05-12T22:06:33.965439+0000","last_unstale":"2024-05-12T22:08:58.967780+0000","last_undegraded":"2024-05-12T22:08:58.967780+0000","last_fullsized":"2024-05-12T22:08:58.967780+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T23:59:56.925209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"88'7","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.966781+0000","last_change":"2024-05-12T22:06:33.967906+0000","last_active":"2024-05-12T22:08:58.966781+0000","last_peered":"2024-05-12T22:08:58.966781+0000","last_clean":"2024-05-12T22:08:58.966781+0000","last_became_active":"2024-05-12T22:06:33.967516+0000","last_became_peered":"2024-05-12T22:06:33.967516+0000","last_unstale":"2024-05-12T22:08:58.966781+0000","last_undegraded":"2024-05-12T22:08:58.966781+0000","last_fullsized":"2024-05-12T22:08:58.966781+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T08:16:56.892024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"88'10","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.968023+0000","last_change":"2024-05-12T22:06:33.971456+0000","last_active":"2024-05-12T22:08:58.968023+0000","last_peered":"2024-05-12T22:08:58.968023+0000","last_clean":"2024-05-12T22:08:58.968023+0000","last_became_active":"2024-05-12T22:06:33.971110+0000","last_became_peered":"2024-05-12T22:06:33.971110+0000","last_unstale":"2024-05-12T22:08:58.968023+0000","last_undegraded":"2024-05-12T22:08:58.968023+0000","last_fullsized":"2024-05-12T22:08:58.968023+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T09:48:56.275346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"88'7","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.973912+0000","last_change":"2024-05-12T22:06:33.983421+0000","last_active":"2024-05-12T22:08:58.973912+0000","last_peered":"2024-05-12T22:08:58.973912+0000","last_clean":"2024-05-12T22:08:58.973912+0000","last_became_active":"2024-05-12T22:06:33.978515+0000","last_became_peered":"2024-05-12T22:06:33.978515+0000","last_unstale":"2024-05-12T22:08:58.973912+0000","last_undegraded":"2024-05-12T22:08:58.973912+0000","last_fullsized":"2024-05-12T22:08:58.973912+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T23:09:48.264541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"88'7","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.974576+0000","last_change":"2024-05-12T22:06:33.980956+0000","last_active":"2024-05-12T22:08:58.974576+0000","last_peered":"2024-05-12T22:08:58.974576+0000","last_clean":"2024-05-12T22:08:58.974576+0000","last_became_active":"2024-05-12T22:06:33.979860+0000","last_became_peered":"2024-05-12T22:06:33.979860+0000","last_unstale":"2024-05-12T22:08:58.974576+0000","last_undegraded":"2024-05-12T22:08:58.974576+0000","last_fullsized":"2024-05-12T22:08:58.974576+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T02:58:29.255704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"88'9","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.981122+0000","last_change":"2024-05-12T22:06:33.978682+0000","last_active":"2024-05-12T22:08:58.981122+0000","last_peered":"2024-05-12T22:08:58.981122+0000","last_clean":"2024-05-12T22:08:58.981122+0000","last_became_active":"2024-05-12T22:06:33.978234+0000","last_became_peered":"2024-05-12T22:06:33.978234+0000","last_unstale":"2024-05-12T22:08:58.981122+0000","last_undegraded":"2024-05-12T22:08:58.981122+0000","last_fullsized":"2024-05-12T22:08:58.981122+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T07:40:48.540839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"88'8","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:59.370117+0000","last_change":"2024-05-12T22:06:33.986598+0000","last_active":"2024-05-12T22:08:59.370117+0000","last_peered":"2024-05-12T22:08:59.370117+0000","last_clean":"2024-05-12T22:08:59.370117+0000","last_became_active":"2024-05-12T22:06:33.986346+0000","last_became_peered":"2024-05-12T22:06:33.986346+0000","last_unstale":"2024-05-12T22:08:59.370117+0000","last_undegraded":"2024-05-12T22:08:59.370117+0000","last_fullsized":"2024-05-12T22:08:59.370117+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T05:52:05.473099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"88'10","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.776735+0000","last_change":"2024-05-12T22:06:33.978900+0000","last_active":"2024-05-12T22:08:58.776735+0000","last_peered":"2024-05-12T22:08:58.776735+0000","last_clean":"2024-05-12T22:08:58.776735+0000","last_became_active":"2024-05-12T22:06:33.978474+0000","last_became_peered":"2024-05-12T22:06:33.978474+0000","last_unstale":"2024-05-12T22:08:58.776735+0000","last_undegraded":"2024-05-12T22:08:58.776735+0000","last_fullsized":"2024-05-12T22:08:58.776735+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T03:47:53.014263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"88'7","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.651554+0000","last_change":"2024-05-12T22:06:33.966752+0000","last_active":"2024-05-12T22:08:58.651554+0000","last_peered":"2024-05-12T22:08:58.651554+0000","last_clean":"2024-05-12T22:08:58.651554+0000","last_became_active":"2024-05-12T22:06:33.966320+0000","last_became_peered":"2024-05-12T22:06:33.966320+0000","last_unstale":"2024-05-12T22:08:58.651554+0000","last_undegraded":"2024-05-12T22:08:58.651554+0000","last_fullsized":"2024-05-12T22:08:58.651554+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T22:11:48.356156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'9","reported_seq":48,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.776810+0000","last_change":"2024-05-12T22:06:46.808696+0000","last_active":"2024-05-12T22:08:58.776810+0000","last_peered":"2024-05-12T22:08:58.776810+0000","last_clean":"2024-05-12T22:08:58.776810+0000","last_became_active":"2024-05-12T22:06:46.796654+0000","last_became_peered":"2024-05-12T22:06:46.796654+0000","last_unstale":"2024-05-12T22:08:58.776810+0000","last_undegraded":"2024-05-12T22:08:58.776810+0000","last_fullsized":"2024-05-12T22:08:58.776810+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T09:32:33.209844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,8],"acting":[9,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"88'5","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.648988+0000","last_change":"2024-05-12T22:06:33.963875+0000","last_active":"2024-05-12T22:08:58.648988+0000","last_peered":"2024-05-12T22:08:58.648988+0000","last_clean":"2024-05-12T22:08:58.648988+0000","last_became_active":"2024-05-12T22:06:33.963239+0000","last_became_peered":"2024-05-12T22:06:33.963239+0000","last_unstale":"2024-05-12T22:08:58.648988+0000","last_undegraded":"2024-05-12T22:08:58.648988+0000","last_fullsized":"2024-05-12T22:08:58.648988+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T23:15:01.832082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.649401+0000","last_change":"2024-05-12T22:06:32.950352+0000","last_active":"2024-05-12T22:08:58.649401+0000","last_peered":"2024-05-12T22:08:58.649401+0000","last_clean":"2024-05-12T22:08:58.649401+0000","last_became_active":"2024-05-12T22:06:32.949759+0000","last_became_peered":"2024-05-12T22:06:32.949759+0000","last_unstale":"2024-05-12T22:08:58.649401+0000","last_undegraded":"2024-05-12T22:08:58.649401+0000","last_fullsized":"2024-05-12T22:08:58.649401+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T06:28:07.613351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"88'6","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.975223+0000","last_change":"2024-05-12T22:06:33.990783+0000","last_active":"2024-05-12T22:08:58.975223+0000","last_peered":"2024-05-12T22:08:58.975223+0000","last_clean":"2024-05-12T22:08:58.975223+0000","last_became_active":"2024-05-12T22:06:33.990387+0000","last_became_peered":"2024-05-12T22:06:33.990387+0000","last_unstale":"2024-05-12T22:08:58.975223+0000","last_undegraded":"2024-05-12T22:08:58.975223+0000","last_fullsized":"2024-05-12T22:08:58.975223+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T05:52:52.247458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"86'4","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.779322+0000","last_change":"2024-05-12T22:06:32.931718+0000","last_active":"2024-05-12T22:08:58.779322+0000","last_peered":"2024-05-12T22:08:58.779322+0000","last_clean":"2024-05-12T22:08:58.779322+0000","last_became_active":"2024-05-12T22:06:32.931404+0000","last_became_peered":"2024-05-12T22:06:32.931404+0000","last_unstale":"2024-05-12T22:08:58.779322+0000","last_undegraded":"2024-05-12T22:08:58.779322+0000","last_fullsized":"2024-05-12T22:08:58.779322+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T02:55:57.770504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"88'9","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.981153+0000","last_change":"2024-05-12T22:06:33.963299+0000","last_active":"2024-05-12T22:08:58.981153+0000","last_peered":"2024-05-12T22:08:58.981153+0000","last_clean":"2024-05-12T22:08:58.981153+0000","last_became_active":"2024-05-12T22:06:33.962916+0000","last_became_peered":"2024-05-12T22:06:33.962916+0000","last_unstale":"2024-05-12T22:08:58.981153+0000","last_undegraded":"2024-05-12T22:08:58.981153+0000","last_fullsized":"2024-05-12T22:08:58.981153+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T05:46:32.131918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:59.369771+0000","last_change":"2024-05-12T22:06:46.800870+0000","last_active":"2024-05-12T22:08:59.369771+0000","last_peered":"2024-05-12T22:08:59.369771+0000","last_clean":"2024-05-12T22:08:59.369771+0000","last_became_active":"2024-05-12T22:06:46.797716+0000","last_became_peered":"2024-05-12T22:06:46.797716+0000","last_unstale":"2024-05-12T22:08:59.369771+0000","last_undegraded":"2024-05-12T22:08:59.369771+0000","last_fullsized":"2024-05-12T22:08:59.369771+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T08:31:57.557564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1d","version":"88'15","reported_seq":34,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.786793+0000","last_change":"2024-05-12T22:06:33.967926+0000","last_active":"2024-05-12T22:08:58.786793+0000","last_peered":"2024-05-12T22:08:58.786793+0000","last_clean":"2024-05-12T22:08:58.786793+0000","last_became_active":"2024-05-12T22:06:33.967588+0000","last_became_peered":"2024-05-12T22:06:33.967588+0000","last_unstale":"2024-05-12T22:08:58.786793+0000","last_undegraded":"2024-05-12T22:08:58.786793+0000","last_fullsized":"2024-05-12T22:08:58.786793+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T22:39:54.434117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"89'8","reported_seq":175,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.966357+0000","last_change":"2024-05-12T22:06:32.930347+0000","last_active":"2024-05-12T22:08:58.966357+0000","last_peered":"2024-05-12T22:08:58.966357+0000","last_clean":"2024-05-12T22:08:58.966357+0000","last_became_active":"2024-05-12T22:06:32.930083+0000","last_became_peered":"2024-05-12T22:06:32.930083+0000","last_unstale":"2024-05-12T22:08:58.966357+0000","last_undegraded":"2024-05-12T22:08:58.966357+0000","last_fullsized":"2024-05-12T22:08:58.966357+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T07:10:08.898490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1545,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":145,"num_read_kb":4,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"88'7","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.778962+0000","last_change":"2024-05-12T22:06:33.973793+0000","last_active":"2024-05-12T22:08:58.778962+0000","last_peered":"2024-05-12T22:08:58.778962+0000","last_clean":"2024-05-12T22:08:58.778962+0000","last_became_active":"2024-05-12T22:06:33.973380+0000","last_became_peered":"2024-05-12T22:06:33.973380+0000","last_unstale":"2024-05-12T22:08:58.778962+0000","last_undegraded":"2024-05-12T22:08:58.778962+0000","last_fullsized":"2024-05-12T22:08:58.778962+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T04:32:01.817780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"86'3","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.649371+0000","last_change":"2024-05-12T22:06:32.938215+0000","last_active":"2024-05-12T22:08:58.649371+0000","last_peered":"2024-05-12T22:08:58.649371+0000","last_clean":"2024-05-12T22:08:58.649371+0000","last_became_active":"2024-05-12T22:06:32.937939+0000","last_became_peered":"2024-05-12T22:06:32.937939+0000","last_unstale":"2024-05-12T22:08:58.649371+0000","last_undegraded":"2024-05-12T22:08:58.649371+0000","last_fullsized":"2024-05-12T22:08:58.649371+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T04:15:28.821835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"88'8","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.786653+0000","last_change":"2024-05-12T22:06:33.977362+0000","last_active":"2024-05-12T22:08:58.786653+0000","last_peered":"2024-05-12T22:08:58.786653+0000","last_clean":"2024-05-12T22:08:58.786653+0000","last_became_active":"2024-05-12T22:06:33.977074+0000","last_became_peered":"2024-05-12T22:06:33.977074+0000","last_unstale":"2024-05-12T22:08:58.786653+0000","last_undegraded":"2024-05-12T22:08:58.786653+0000","last_fullsized":"2024-05-12T22:08:58.786653+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T22:21:00.062780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"86'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.966862+0000","last_change":"2024-05-12T22:06:32.935508+0000","last_active":"2024-05-12T22:08:58.966862+0000","last_peered":"2024-05-12T22:08:58.966862+0000","last_clean":"2024-05-12T22:08:58.966862+0000","last_became_active":"2024-05-12T22:06:32.934760+0000","last_became_peered":"2024-05-12T22:06:32.934760+0000","last_unstale":"2024-05-12T22:08:58.966862+0000","last_undegraded":"2024-05-12T22:08:58.966862+0000","last_fullsized":"2024-05-12T22:08:58.966862+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T01:58:09.450187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"88'8","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.648596+0000","last_change":"2024-05-12T22:06:33.966421+0000","last_active":"2024-05-12T22:08:58.648596+0000","last_peered":"2024-05-12T22:08:58.648596+0000","last_clean":"2024-05-12T22:08:58.648596+0000","last_became_active":"2024-05-12T22:06:33.965627+0000","last_became_peered":"2024-05-12T22:06:33.965627+0000","last_unstale":"2024-05-12T22:08:58.648596+0000","last_undegraded":"2024-05-12T22:08:58.648596+0000","last_fullsized":"2024-05-12T22:08:58.648596+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T09:25:03.836082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"86'1","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.786416+0000","last_change":"2024-05-12T22:06:32.928907+0000","last_active":"2024-05-12T22:08:58.786416+0000","last_peered":"2024-05-12T22:08:58.786416+0000","last_clean":"2024-05-12T22:08:58.786416+0000","last_became_active":"2024-05-12T22:06:32.928586+0000","last_became_peered":"2024-05-12T22:06:32.928586+0000","last_unstale":"2024-05-12T22:08:58.786416+0000","last_undegraded":"2024-05-12T22:08:58.786416+0000","last_fullsized":"2024-05-12T22:08:58.786416+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T04:24:21.658957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"88'11","reported_seq":31,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.776199+0000","last_change":"2024-05-12T22:06:33.982873+0000","last_active":"2024-05-12T22:08:58.776199+0000","last_peered":"2024-05-12T22:08:58.776199+0000","last_clean":"2024-05-12T22:08:58.776199+0000","last_became_active":"2024-05-12T22:06:33.982495+0000","last_became_peered":"2024-05-12T22:06:33.982495+0000","last_unstale":"2024-05-12T22:08:58.776199+0000","last_undegraded":"2024-05-12T22:08:58.776199+0000","last_fullsized":"2024-05-12T22:08:58.776199+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T05:45:39.404282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"89'5","reported_seq":168,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.966975+0000","last_change":"2024-05-12T22:06:32.937716+0000","last_active":"2024-05-12T22:08:58.966975+0000","last_peered":"2024-05-12T22:08:58.966975+0000","last_clean":"2024-05-12T22:08:58.966975+0000","last_became_active":"2024-05-12T22:06:32.937418+0000","last_became_peered":"2024-05-12T22:06:32.937418+0000","last_unstale":"2024-05-12T22:08:58.966975+0000","last_undegraded":"2024-05-12T22:08:58.966975+0000","last_fullsized":"2024-05-12T22:08:58.966975+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T08:20:13.639958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":141,"num_read_kb":141,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"88'10","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.981357+0000","last_change":"2024-05-12T22:06:33.974702+0000","last_active":"2024-05-12T22:08:58.981357+0000","last_peered":"2024-05-12T22:08:58.981357+0000","last_clean":"2024-05-12T22:08:58.981357+0000","last_became_active":"2024-05-12T22:06:33.974363+0000","last_became_peered":"2024-05-12T22:06:33.974363+0000","last_unstale":"2024-05-12T22:08:58.981357+0000","last_undegraded":"2024-05-12T22:08:58.981357+0000","last_fullsized":"2024-05-12T22:08:58.981357+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T09:16:33.620945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"86'2","reported_seq":34,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.967542+0000","last_change":"2024-05-12T22:06:46.793331+0000","last_active":"2024-05-12T22:08:58.967542+0000","last_peered":"2024-05-12T22:08:58.967542+0000","last_clean":"2024-05-12T22:08:58.967542+0000","last_became_active":"2024-05-12T22:06:46.789984+0000","last_became_peered":"2024-05-12T22:06:46.789984+0000","last_unstale":"2024-05-12T22:08:58.967542+0000","last_undegraded":"2024-05-12T22:08:58.967542+0000","last_fullsized":"2024-05-12T22:08:58.967542+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T08:16:29.505145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"88'9","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.975167+0000","last_change":"2024-05-12T22:06:33.980496+0000","last_active":"2024-05-12T22:08:58.975167+0000","last_peered":"2024-05-12T22:08:58.975167+0000","last_clean":"2024-05-12T22:08:58.975167+0000","last_became_active":"2024-05-12T22:06:33.980155+0000","last_became_peered":"2024-05-12T22:06:33.980155+0000","last_unstale":"2024-05-12T22:08:58.975167+0000","last_undegraded":"2024-05-12T22:08:58.975167+0000","last_fullsized":"2024-05-12T22:08:58.975167+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T01:54:03.831828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"86'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.786948+0000","last_change":"2024-05-12T22:06:32.951633+0000","last_active":"2024-05-12T22:08:58.786948+0000","last_peered":"2024-05-12T22:08:58.786948+0000","last_clean":"2024-05-12T22:08:58.786948+0000","last_became_active":"2024-05-12T22:06:32.951382+0000","last_became_peered":"2024-05-12T22:06:32.951382+0000","last_unstale":"2024-05-12T22:08:58.786948+0000","last_undegraded":"2024-05-12T22:08:58.786948+0000","last_fullsized":"2024-05-12T22:08:58.786948+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T02:06:21.408593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"88'11","reported_seq":31,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.981428+0000","last_change":"2024-05-12T22:06:33.963493+0000","last_active":"2024-05-12T22:08:58.981428+0000","last_peered":"2024-05-12T22:08:58.981428+0000","last_clean":"2024-05-12T22:08:58.981428+0000","last_became_active":"2024-05-12T22:06:33.963127+0000","last_became_peered":"2024-05-12T22:06:33.963127+0000","last_unstale":"2024-05-12T22:08:58.981428+0000","last_undegraded":"2024-05-12T22:08:58.981428+0000","last_fullsized":"2024-05-12T22:08:58.981428+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T05:39:31.038446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"89'38","reported_seq":353,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.982668+0000","last_change":"2024-05-12T22:06:32.932026+0000","last_active":"2024-05-12T22:08:58.982668+0000","last_peered":"2024-05-12T22:08:58.982668+0000","last_clean":"2024-05-12T22:08:58.982668+0000","last_became_active":"2024-05-12T22:06:32.931667+0000","last_became_peered":"2024-05-12T22:06:32.931667+0000","last_unstale":"2024-05-12T22:08:58.982668+0000","last_undegraded":"2024-05-12T22:08:58.982668+0000","last_fullsized":"2024-05-12T22:08:58.982668+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T06:16:28.259047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67587,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":293,"num_read_kb":212,"num_write":39,"num_write_kb":85,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"88'6","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.779148+0000","last_change":"2024-05-12T22:06:33.962677+0000","last_active":"2024-05-12T22:08:58.779148+0000","last_peered":"2024-05-12T22:08:58.779148+0000","last_clean":"2024-05-12T22:08:58.779148+0000","last_became_active":"2024-05-12T22:06:33.962295+0000","last_became_peered":"2024-05-12T22:06:33.962295+0000","last_unstale":"2024-05-12T22:08:58.779148+0000","last_undegraded":"2024-05-12T22:08:58.779148+0000","last_fullsized":"2024-05-12T22:08:58.779148+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T01:31:04.587046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"86'6","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.651259+0000","last_change":"2024-05-12T22:06:32.936104+0000","last_active":"2024-05-12T22:08:58.651259+0000","last_peered":"2024-05-12T22:08:58.651259+0000","last_clean":"2024-05-12T22:08:58.651259+0000","last_became_active":"2024-05-12T22:06:32.935701+0000","last_became_peered":"2024-05-12T22:06:32.935701+0000","last_unstale":"2024-05-12T22:08:58.651259+0000","last_undegraded":"2024-05-12T22:08:58.651259+0000","last_fullsized":"2024-05-12T22:08:58.651259+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T07:04:10.115302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"88'9","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.967648+0000","last_change":"2024-05-12T22:06:33.977850+0000","last_active":"2024-05-12T22:08:58.967648+0000","last_peered":"2024-05-12T22:08:58.967648+0000","last_clean":"2024-05-12T22:08:58.967648+0000","last_became_active":"2024-05-12T22:06:33.977511+0000","last_became_peered":"2024-05-12T22:06:33.977511+0000","last_unstale":"2024-05-12T22:08:58.967648+0000","last_undegraded":"2024-05-12T22:08:58.967648+0000","last_fullsized":"2024-05-12T22:08:58.967648+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T23:10:17.267823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"86'3","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.786663+0000","last_change":"2024-05-12T22:06:32.929955+0000","last_active":"2024-05-12T22:08:58.786663+0000","last_peered":"2024-05-12T22:08:58.786663+0000","last_clean":"2024-05-12T22:08:58.786663+0000","last_became_active":"2024-05-12T22:06:32.929623+0000","last_became_peered":"2024-05-12T22:06:32.929623+0000","last_unstale":"2024-05-12T22:08:58.786663+0000","last_undegraded":"2024-05-12T22:08:58.786663+0000","last_fullsized":"2024-05-12T22:08:58.786663+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T08:15:22.622908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"88'15","reported_seq":41,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.967533+0000","last_change":"2024-05-12T22:06:46.931736+0000","last_active":"2024-05-12T22:08:58.967533+0000","last_peered":"2024-05-12T22:08:58.967533+0000","last_clean":"2024-05-12T22:08:58.967533+0000","last_became_active":"2024-05-12T22:06:46.931265+0000","last_became_peered":"2024-05-12T22:06:46.931265+0000","last_unstale":"2024-05-12T22:08:58.967533+0000","last_undegraded":"2024-05-12T22:08:58.967533+0000","last_fullsized":"2024-05-12T22:08:58.967533+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T02:25:34.002024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"89'7","reported_seq":175,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:59.010336+0000","last_change":"2024-05-12T22:06:32.927707+0000","last_active":"2024-05-12T22:08:59.010336+0000","last_peered":"2024-05-12T22:08:59.010336+0000","last_clean":"2024-05-12T22:08:59.010336+0000","last_became_active":"2024-05-12T22:06:32.927406+0000","last_became_peered":"2024-05-12T22:06:32.927406+0000","last_unstale":"2024-05-12T22:08:59.010336+0000","last_undegraded":"2024-05-12T22:08:59.010336+0000","last_fullsized":"2024-05-12T22:08:59.010336+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T05:14:34.268589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1902,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":147,"num_read_kb":6,"num_write":6,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"88'6","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.966849+0000","last_change":"2024-05-12T22:06:33.964996+0000","last_active":"2024-05-12T22:08:58.966849+0000","last_peered":"2024-05-12T22:08:58.966849+0000","last_clean":"2024-05-12T22:08:58.966849+0000","last_became_active":"2024-05-12T22:06:33.964587+0000","last_became_peered":"2024-05-12T22:06:33.964587+0000","last_unstale":"2024-05-12T22:08:58.966849+0000","last_undegraded":"2024-05-12T22:08:58.966849+0000","last_fullsized":"2024-05-12T22:08:58.966849+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T22:27:28.316377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"86'15","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.778916+0000","last_change":"2024-05-12T22:06:32.935543+0000","last_active":"2024-05-12T22:08:58.778916+0000","last_peered":"2024-05-12T22:08:58.778916+0000","last_clean":"2024-05-12T22:08:58.778916+0000","last_became_active":"2024-05-12T22:06:32.935145+0000","last_became_peered":"2024-05-12T22:06:32.935145+0000","last_unstale":"2024-05-12T22:08:58.778916+0000","last_undegraded":"2024-05-12T22:08:58.778916+0000","last_fullsized":"2024-05-12T22:08:58.778916+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T01:13:23.289057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"88'6","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.786436+0000","last_change":"2024-05-12T22:06:33.964367+0000","last_active":"2024-05-12T22:08:58.786436+0000","last_peered":"2024-05-12T22:08:58.786436+0000","last_clean":"2024-05-12T22:08:58.786436+0000","last_became_active":"2024-05-12T22:06:33.963876+0000","last_became_peered":"2024-05-12T22:06:33.963876+0000","last_unstale":"2024-05-12T22:08:58.786436+0000","last_undegraded":"2024-05-12T22:08:58.786436+0000","last_fullsized":"2024-05-12T22:08:58.786436+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T07:09:20.962965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:59.356563+0000","last_change":"2024-05-12T22:06:32.927270+0000","last_active":"2024-05-12T22:08:59.356563+0000","last_peered":"2024-05-12T22:08:59.356563+0000","last_clean":"2024-05-12T22:08:59.356563+0000","last_became_active":"2024-05-12T22:06:32.926930+0000","last_became_peered":"2024-05-12T22:06:32.926930+0000","last_unstale":"2024-05-12T22:08:59.356563+0000","last_undegraded":"2024-05-12T22:08:59.356563+0000","last_fullsized":"2024-05-12T22:08:59.356563+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T07:04:48.336599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"88'5","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:59.369986+0000","last_change":"2024-05-12T22:06:33.966350+0000","last_active":"2024-05-12T22:08:59.369986+0000","last_peered":"2024-05-12T22:08:59.369986+0000","last_clean":"2024-05-12T22:08:59.369986+0000","last_became_active":"2024-05-12T22:06:33.965974+0000","last_became_peered":"2024-05-12T22:06:33.965974+0000","last_unstale":"2024-05-12T22:08:59.369986+0000","last_undegraded":"2024-05-12T22:08:59.369986+0000","last_fullsized":"2024-05-12T22:08:59.369986+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T02:08:16.889684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.776349+0000","last_change":"2024-05-12T22:06:32.944945+0000","last_active":"2024-05-12T22:08:58.776349+0000","last_peered":"2024-05-12T22:08:58.776349+0000","last_clean":"2024-05-12T22:08:58.776349+0000","last_became_active":"2024-05-12T22:06:32.944576+0000","last_became_peered":"2024-05-12T22:06:32.944576+0000","last_unstale":"2024-05-12T22:08:58.776349+0000","last_undegraded":"2024-05-12T22:08:58.776349+0000","last_fullsized":"2024-05-12T22:08:58.776349+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T07:29:14.544821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"88'5","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.649553+0000","last_change":"2024-05-12T22:06:33.966299+0000","last_active":"2024-05-12T22:08:58.649553+0000","last_peered":"2024-05-12T22:08:58.649553+0000","last_clean":"2024-05-12T22:08:58.649553+0000","last_became_active":"2024-05-12T22:06:33.965902+0000","last_became_peered":"2024-05-12T22:06:33.965902+0000","last_unstale":"2024-05-12T22:08:58.649553+0000","last_undegraded":"2024-05-12T22:08:58.649553+0000","last_fullsized":"2024-05-12T22:08:58.649553+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T22:37:51.476546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"86'1","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.967232+0000","last_change":"2024-05-12T22:06:32.937210+0000","last_active":"2024-05-12T22:08:58.967232+0000","last_peered":"2024-05-12T22:08:58.967232+0000","last_clean":"2024-05-12T22:08:58.967232+0000","last_became_active":"2024-05-12T22:06:32.936805+0000","last_became_peered":"2024-05-12T22:06:32.936805+0000","last_unstale":"2024-05-12T22:08:58.967232+0000","last_undegraded":"2024-05-12T22:08:58.967232+0000","last_fullsized":"2024-05-12T22:08:58.967232+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T06:14:20.825258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"88'7","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.649737+0000","last_change":"2024-05-12T22:06:33.983604+0000","last_active":"2024-05-12T22:08:58.649737+0000","last_peered":"2024-05-12T22:08:58.649737+0000","last_clean":"2024-05-12T22:08:58.649737+0000","last_became_active":"2024-05-12T22:06:33.983168+0000","last_became_peered":"2024-05-12T22:06:33.983168+0000","last_unstale":"2024-05-12T22:08:58.649737+0000","last_undegraded":"2024-05-12T22:08:58.649737+0000","last_fullsized":"2024-05-12T22:08:58.649737+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T05:54:55.854593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"88'8","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.786490+0000","last_change":"2024-05-12T22:06:33.968848+0000","last_active":"2024-05-12T22:08:58.786490+0000","last_peered":"2024-05-12T22:08:58.786490+0000","last_clean":"2024-05-12T22:08:58.786490+0000","last_became_active":"2024-05-12T22:06:33.968529+0000","last_became_peered":"2024-05-12T22:06:33.968529+0000","last_unstale":"2024-05-12T22:08:58.786490+0000","last_undegraded":"2024-05-12T22:08:58.786490+0000","last_fullsized":"2024-05-12T22:08:58.786490+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T03:00:41.129662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'6","reported_seq":169,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.974762+0000","last_change":"2024-05-12T22:06:32.946670+0000","last_active":"2024-05-12T22:08:58.974762+0000","last_peered":"2024-05-12T22:08:58.974762+0000","last_clean":"2024-05-12T22:08:58.974762+0000","last_became_active":"2024-05-12T22:06:32.946059+0000","last_became_peered":"2024-05-12T22:06:32.946059+0000","last_unstale":"2024-05-12T22:08:58.974762+0000","last_undegraded":"2024-05-12T22:08:58.974762+0000","last_fullsized":"2024-05-12T22:08:58.974762+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T22:35:56.116979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":141,"num_read_kb":141,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"88'8","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.648638+0000","last_change":"2024-05-12T22:06:33.962177+0000","last_active":"2024-05-12T22:08:58.648638+0000","last_peered":"2024-05-12T22:08:58.648638+0000","last_clean":"2024-05-12T22:08:58.648638+0000","last_became_active":"2024-05-12T22:06:33.960919+0000","last_became_peered":"2024-05-12T22:06:33.960919+0000","last_unstale":"2024-05-12T22:08:58.648638+0000","last_undegraded":"2024-05-12T22:08:58.648638+0000","last_fullsized":"2024-05-12T22:08:58.648638+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T00:39:54.566460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'3","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.966691+0000","last_change":"2024-05-12T22:06:32.927885+0000","last_active":"2024-05-12T22:08:58.966691+0000","last_peered":"2024-05-12T22:08:58.966691+0000","last_clean":"2024-05-12T22:08:58.966691+0000","last_became_active":"2024-05-12T22:06:32.927567+0000","last_became_peered":"2024-05-12T22:06:32.927567+0000","last_unstale":"2024-05-12T22:08:58.966691+0000","last_undegraded":"2024-05-12T22:08:58.966691+0000","last_fullsized":"2024-05-12T22:08:58.966691+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T07:13:50.811992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":355,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.966894+0000","last_change":"2024-05-12T22:05:17.683363+0000","last_active":"2024-05-12T22:08:58.966894+0000","last_peered":"2024-05-12T22:08:58.966894+0000","last_clean":"2024-05-12T22:08:58.966894+0000","last_became_active":"2024-05-12T22:05:17.682824+0000","last_became_peered":"2024-05-12T22:05:17.682824+0000","last_unstale":"2024-05-12T22:08:58.966894+0000","last_undegraded":"2024-05-12T22:08:58.966894+0000","last_fullsized":"2024-05-12T22:08:58.966894+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:01:32.304714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:01:32.304714+0000","last_clean_scrub_stamp":"2024-05-12T22:01:32.304714+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T06:35:22.474820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"88'5","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.966617+0000","last_change":"2024-05-12T22:06:33.965332+0000","last_active":"2024-05-12T22:08:58.966617+0000","last_peered":"2024-05-12T22:08:58.966617+0000","last_clean":"2024-05-12T22:08:58.966617+0000","last_became_active":"2024-05-12T22:06:33.964858+0000","last_became_peered":"2024-05-12T22:06:33.964858+0000","last_unstale":"2024-05-12T22:08:58.966617+0000","last_undegraded":"2024-05-12T22:08:58.966617+0000","last_fullsized":"2024-05-12T22:08:58.966617+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T05:14:09.825614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'3","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.967071+0000","last_change":"2024-05-12T22:06:32.929773+0000","last_active":"2024-05-12T22:08:58.967071+0000","last_peered":"2024-05-12T22:08:58.967071+0000","last_clean":"2024-05-12T22:08:58.967071+0000","last_became_active":"2024-05-12T22:06:32.929458+0000","last_became_peered":"2024-05-12T22:06:32.929458+0000","last_unstale":"2024-05-12T22:08:58.967071+0000","last_undegraded":"2024-05-12T22:08:58.967071+0000","last_fullsized":"2024-05-12T22:08:58.967071+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T01:30:42.853183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"86'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.779092+0000","last_change":"2024-05-12T22:06:32.947014+0000","last_active":"2024-05-12T22:08:58.779092+0000","last_peered":"2024-05-12T22:08:58.779092+0000","last_clean":"2024-05-12T22:08:58.779092+0000","last_became_active":"2024-05-12T22:06:32.946661+0000","last_became_peered":"2024-05-12T22:06:32.946661+0000","last_unstale":"2024-05-12T22:08:58.779092+0000","last_undegraded":"2024-05-12T22:08:58.779092+0000","last_fullsized":"2024-05-12T22:08:58.779092+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T23:09:19.378353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"88'9","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.967616+0000","last_change":"2024-05-12T22:06:33.980490+0000","last_active":"2024-05-12T22:08:58.967616+0000","last_peered":"2024-05-12T22:08:58.967616+0000","last_clean":"2024-05-12T22:08:58.967616+0000","last_became_active":"2024-05-12T22:06:33.980133+0000","last_became_peered":"2024-05-12T22:06:33.980133+0000","last_unstale":"2024-05-12T22:08:58.967616+0000","last_undegraded":"2024-05-12T22:08:58.967616+0000","last_fullsized":"2024-05-12T22:08:58.967616+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T00:28:10.461428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"89'46","reported_seq":91,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.918040+0000","last_change":"2024-05-12T22:06:32.943152+0000","last_active":"2024-05-12T22:08:58.918040+0000","last_peered":"2024-05-12T22:08:58.918040+0000","last_clean":"2024-05-12T22:08:58.918040+0000","last_became_active":"2024-05-12T22:06:32.942769+0000","last_became_peered":"2024-05-12T22:06:32.942769+0000","last_unstale":"2024-05-12T22:08:58.918040+0000","last_undegraded":"2024-05-12T22:08:58.918040+0000","last_fullsized":"2024-05-12T22:08:58.918040+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T01:23:24.955739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":82,"num_write_kb":837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"88'5","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.649177+0000","last_change":"2024-05-12T22:06:33.964081+0000","last_active":"2024-05-12T22:08:58.649177+0000","last_peered":"2024-05-12T22:08:58.649177+0000","last_clean":"2024-05-12T22:08:58.649177+0000","last_became_active":"2024-05-12T22:06:33.963054+0000","last_became_peered":"2024-05-12T22:06:33.963054+0000","last_unstale":"2024-05-12T22:08:58.649177+0000","last_undegraded":"2024-05-12T22:08:58.649177+0000","last_fullsized":"2024-05-12T22:08:58.649177+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T09:40:03.396200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"86'4","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.966937+0000","last_change":"2024-05-12T22:06:32.936432+0000","last_active":"2024-05-12T22:08:58.966937+0000","last_peered":"2024-05-12T22:08:58.966937+0000","last_clean":"2024-05-12T22:08:58.966937+0000","last_became_active":"2024-05-12T22:06:32.936094+0000","last_became_peered":"2024-05-12T22:06:32.936094+0000","last_unstale":"2024-05-12T22:08:58.966937+0000","last_undegraded":"2024-05-12T22:08:58.966937+0000","last_fullsized":"2024-05-12T22:08:58.966937+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T02:40:31.346118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"88'7","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.974922+0000","last_change":"2024-05-12T22:06:33.985953+0000","last_active":"2024-05-12T22:08:58.974922+0000","last_peered":"2024-05-12T22:08:58.974922+0000","last_clean":"2024-05-12T22:08:58.974922+0000","last_became_active":"2024-05-12T22:06:33.984864+0000","last_became_peered":"2024-05-12T22:06:33.984864+0000","last_unstale":"2024-05-12T22:08:58.974922+0000","last_undegraded":"2024-05-12T22:08:58.974922+0000","last_fullsized":"2024-05-12T22:08:58.974922+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T07:28:22.357841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.981163+0000","last_change":"2024-05-12T22:06:32.934510+0000","last_active":"2024-05-12T22:08:58.981163+0000","last_peered":"2024-05-12T22:08:58.981163+0000","last_clean":"2024-05-12T22:08:58.981163+0000","last_became_active":"2024-05-12T22:06:32.934222+0000","last_became_peered":"2024-05-12T22:06:32.934222+0000","last_unstale":"2024-05-12T22:08:58.981163+0000","last_undegraded":"2024-05-12T22:08:58.981163+0000","last_fullsized":"2024-05-12T22:08:58.981163+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T00:57:44.321629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"88'2","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.778837+0000","last_change":"2024-05-12T22:06:33.963870+0000","last_active":"2024-05-12T22:08:58.778837+0000","last_peered":"2024-05-12T22:08:58.778837+0000","last_clean":"2024-05-12T22:08:58.778837+0000","last_became_active":"2024-05-12T22:06:33.963533+0000","last_became_peered":"2024-05-12T22:06:33.963533+0000","last_unstale":"2024-05-12T22:08:58.778837+0000","last_undegraded":"2024-05-12T22:08:58.778837+0000","last_fullsized":"2024-05-12T22:08:58.778837+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T08:56:59.495469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"86'3","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.648478+0000","last_change":"2024-05-12T22:06:32.939738+0000","last_active":"2024-05-12T22:08:58.648478+0000","last_peered":"2024-05-12T22:08:58.648478+0000","last_clean":"2024-05-12T22:08:58.648478+0000","last_became_active":"2024-05-12T22:06:32.939365+0000","last_became_peered":"2024-05-12T22:06:32.939365+0000","last_unstale":"2024-05-12T22:08:58.648478+0000","last_undegraded":"2024-05-12T22:08:58.648478+0000","last_fullsized":"2024-05-12T22:08:58.648478+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T02:19:31.180834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"88'9","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:59.369732+0000","last_change":"2024-05-12T22:06:33.975064+0000","last_active":"2024-05-12T22:08:59.369732+0000","last_peered":"2024-05-12T22:08:59.369732+0000","last_clean":"2024-05-12T22:08:59.369732+0000","last_became_active":"2024-05-12T22:06:33.974613+0000","last_became_peered":"2024-05-12T22:06:33.974613+0000","last_unstale":"2024-05-12T22:08:59.369732+0000","last_undegraded":"2024-05-12T22:08:59.369732+0000","last_fullsized":"2024-05-12T22:08:59.369732+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T08:46:27.343777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"86'1","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.651327+0000","last_change":"2024-05-12T22:06:32.931148+0000","last_active":"2024-05-12T22:08:58.651327+0000","last_peered":"2024-05-12T22:08:58.651327+0000","last_clean":"2024-05-12T22:08:58.651327+0000","last_became_active":"2024-05-12T22:06:32.930717+0000","last_became_peered":"2024-05-12T22:06:32.930717+0000","last_unstale":"2024-05-12T22:08:58.651327+0000","last_undegraded":"2024-05-12T22:08:58.651327+0000","last_fullsized":"2024-05-12T22:08:58.651327+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T07:26:20.915467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"88'8","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.648890+0000","last_change":"2024-05-12T22:06:33.988390+0000","last_active":"2024-05-12T22:08:58.648890+0000","last_peered":"2024-05-12T22:08:58.648890+0000","last_clean":"2024-05-12T22:08:58.648890+0000","last_became_active":"2024-05-12T22:06:33.986970+0000","last_became_peered":"2024-05-12T22:06:33.986970+0000","last_unstale":"2024-05-12T22:08:58.648890+0000","last_undegraded":"2024-05-12T22:08:58.648890+0000","last_fullsized":"2024-05-12T22:08:58.648890+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T05:03:31.930261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.779155+0000","last_change":"2024-05-12T22:06:46.798123+0000","last_active":"2024-05-12T22:08:58.779155+0000","last_peered":"2024-05-12T22:08:58.779155+0000","last_clean":"2024-05-12T22:08:58.779155+0000","last_became_active":"2024-05-12T22:06:46.791029+0000","last_became_peered":"2024-05-12T22:06:46.791029+0000","last_unstale":"2024-05-12T22:08:58.779155+0000","last_undegraded":"2024-05-12T22:08:58.779155+0000","last_fullsized":"2024-05-12T22:08:58.779155+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T06:31:25.108659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,7],"acting":[8,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"88'5","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.967282+0000","last_change":"2024-05-12T22:06:33.968256+0000","last_active":"2024-05-12T22:08:58.967282+0000","last_peered":"2024-05-12T22:08:58.967282+0000","last_clean":"2024-05-12T22:08:58.967282+0000","last_became_active":"2024-05-12T22:06:33.967885+0000","last_became_peered":"2024-05-12T22:06:33.967885+0000","last_unstale":"2024-05-12T22:08:58.967282+0000","last_undegraded":"2024-05-12T22:08:58.967282+0000","last_fullsized":"2024-05-12T22:08:58.967282+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T09:12:40.687367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"86'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.651789+0000","last_change":"2024-05-12T22:06:32.944736+0000","last_active":"2024-05-12T22:08:58.651789+0000","last_peered":"2024-05-12T22:08:58.651789+0000","last_clean":"2024-05-12T22:08:58.651789+0000","last_became_active":"2024-05-12T22:06:32.944269+0000","last_became_peered":"2024-05-12T22:06:32.944269+0000","last_unstale":"2024-05-12T22:08:58.651789+0000","last_undegraded":"2024-05-12T22:08:58.651789+0000","last_fullsized":"2024-05-12T22:08:58.651789+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T22:56:24.391267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"88'7","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.967821+0000","last_change":"2024-05-12T22:06:33.968360+0000","last_active":"2024-05-12T22:08:58.967821+0000","last_peered":"2024-05-12T22:08:58.967821+0000","last_clean":"2024-05-12T22:08:58.967821+0000","last_became_active":"2024-05-12T22:06:33.967967+0000","last_became_peered":"2024-05-12T22:06:33.967967+0000","last_unstale":"2024-05-12T22:08:58.967821+0000","last_undegraded":"2024-05-12T22:08:58.967821+0000","last_fullsized":"2024-05-12T22:08:58.967821+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T00:41:12.693858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"89'39","reported_seq":69,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.914994+0000","last_change":"2024-05-12T22:06:32.935993+0000","last_active":"2024-05-12T22:08:58.914994+0000","last_peered":"2024-05-12T22:08:58.914994+0000","last_clean":"2024-05-12T22:08:58.914994+0000","last_became_active":"2024-05-12T22:06:32.935615+0000","last_became_peered":"2024-05-12T22:06:32.935615+0000","last_unstale":"2024-05-12T22:08:58.914994+0000","last_undegraded":"2024-05-12T22:08:58.914994+0000","last_fullsized":"2024-05-12T22:08:58.914994+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T00:43:39.254637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"88'7","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.967474+0000","last_change":"2024-05-12T22:06:33.966110+0000","last_active":"2024-05-12T22:08:58.967474+0000","last_peered":"2024-05-12T22:08:58.967474+0000","last_clean":"2024-05-12T22:08:58.967474+0000","last_became_active":"2024-05-12T22:06:33.965755+0000","last_became_peered":"2024-05-12T22:06:33.965755+0000","last_unstale":"2024-05-12T22:08:58.967474+0000","last_undegraded":"2024-05-12T22:08:58.967474+0000","last_fullsized":"2024-05-12T22:08:58.967474+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T22:18:53.941335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.968104+0000","last_change":"2024-05-12T22:06:32.942229+0000","last_active":"2024-05-12T22:08:58.968104+0000","last_peered":"2024-05-12T22:08:58.968104+0000","last_clean":"2024-05-12T22:08:58.968104+0000","last_became_active":"2024-05-12T22:06:32.941826+0000","last_became_peered":"2024-05-12T22:06:32.941826+0000","last_unstale":"2024-05-12T22:08:58.968104+0000","last_undegraded":"2024-05-12T22:08:58.968104+0000","last_fullsized":"2024-05-12T22:08:58.968104+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:31.872756+0000","last_clean_scrub_stamp":"2024-05-12T22:06:31.872756+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T03:52:34.886318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"88'9","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.967759+0000","last_change":"2024-05-12T22:06:33.963081+0000","last_active":"2024-05-12T22:08:58.967759+0000","last_peered":"2024-05-12T22:08:58.967759+0000","last_clean":"2024-05-12T22:08:58.967759+0000","last_became_active":"2024-05-12T22:06:33.962696+0000","last_became_peered":"2024-05-12T22:06:33.962696+0000","last_unstale":"2024-05-12T22:08:58.967759+0000","last_undegraded":"2024-05-12T22:08:58.967759+0000","last_fullsized":"2024-05-12T22:08:58.967759+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T05:55:45.581823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"88'9","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:59.369801+0000","last_change":"2024-05-12T22:06:33.968865+0000","last_active":"2024-05-12T22:08:59.369801+0000","last_peered":"2024-05-12T22:08:59.369801+0000","last_clean":"2024-05-12T22:08:59.369801+0000","last_became_active":"2024-05-12T22:06:33.968185+0000","last_became_peered":"2024-05-12T22:06:33.968185+0000","last_unstale":"2024-05-12T22:08:59.369801+0000","last_undegraded":"2024-05-12T22:08:59.369801+0000","last_fullsized":"2024-05-12T22:08:59.369801+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T01:23:41.083778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"88'12","reported_seq":32,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.651171+0000","last_change":"2024-05-12T22:06:33.964786+0000","last_active":"2024-05-12T22:08:58.651171+0000","last_peered":"2024-05-12T22:08:58.651171+0000","last_clean":"2024-05-12T22:08:58.651171+0000","last_became_active":"2024-05-12T22:06:33.964227+0000","last_became_peered":"2024-05-12T22:06:33.964227+0000","last_unstale":"2024-05-12T22:08:58.651171+0000","last_undegraded":"2024-05-12T22:08:58.651171+0000","last_fullsized":"2024-05-12T22:08:58.651171+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T02:53:42.739455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"88'8","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.651132+0000","last_change":"2024-05-12T22:06:33.978857+0000","last_active":"2024-05-12T22:08:58.651132+0000","last_peered":"2024-05-12T22:08:58.651132+0000","last_clean":"2024-05-12T22:08:58.651132+0000","last_became_active":"2024-05-12T22:06:33.978531+0000","last_became_peered":"2024-05-12T22:06:33.978531+0000","last_unstale":"2024-05-12T22:08:58.651132+0000","last_undegraded":"2024-05-12T22:08:58.651132+0000","last_fullsized":"2024-05-12T22:08:58.651132+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T02:51:05.609267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"88'7","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.974185+0000","last_change":"2024-05-12T22:06:33.980662+0000","last_active":"2024-05-12T22:08:58.974185+0000","last_peered":"2024-05-12T22:08:58.974185+0000","last_clean":"2024-05-12T22:08:58.974185+0000","last_became_active":"2024-05-12T22:06:33.980332+0000","last_became_peered":"2024-05-12T22:06:33.980332+0000","last_unstale":"2024-05-12T22:08:58.974185+0000","last_undegraded":"2024-05-12T22:08:58.974185+0000","last_fullsized":"2024-05-12T22:08:58.974185+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T06:21:59.715792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"88'4","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.967687+0000","last_change":"2024-05-12T22:06:33.987338+0000","last_active":"2024-05-12T22:08:58.967687+0000","last_peered":"2024-05-12T22:08:58.967687+0000","last_clean":"2024-05-12T22:08:58.967687+0000","last_became_active":"2024-05-12T22:06:33.986942+0000","last_became_peered":"2024-05-12T22:06:33.986942+0000","last_unstale":"2024-05-12T22:08:58.967687+0000","last_undegraded":"2024-05-12T22:08:58.967687+0000","last_fullsized":"2024-05-12T22:08:58.967687+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T01:02:05.137558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"88'6","reported_seq":32,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:59.370006+0000","last_change":"2024-05-12T22:06:46.796752+0000","last_active":"2024-05-12T22:08:59.370006+0000","last_peered":"2024-05-12T22:08:59.370006+0000","last_clean":"2024-05-12T22:08:59.370006+0000","last_became_active":"2024-05-12T22:06:46.796369+0000","last_became_peered":"2024-05-12T22:06:46.796369+0000","last_unstale":"2024-05-12T22:08:59.370006+0000","last_undegraded":"2024-05-12T22:08:59.370006+0000","last_fullsized":"2024-05-12T22:08:59.370006+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T08:23:24.685495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,9],"acting":[11,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"88'15","reported_seq":35,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.651395+0000","last_change":"2024-05-12T22:06:33.963902+0000","last_active":"2024-05-12T22:08:58.651395+0000","last_peered":"2024-05-12T22:08:58.651395+0000","last_clean":"2024-05-12T22:08:58.651395+0000","last_became_active":"2024-05-12T22:06:33.963436+0000","last_became_peered":"2024-05-12T22:06:33.963436+0000","last_unstale":"2024-05-12T22:08:58.651395+0000","last_undegraded":"2024-05-12T22:08:58.651395+0000","last_fullsized":"2024-05-12T22:08:58.651395+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T02:51:56.415167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"88'2","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.649008+0000","last_change":"2024-05-12T22:06:46.794434+0000","last_active":"2024-05-12T22:08:58.649008+0000","last_peered":"2024-05-12T22:08:58.649008+0000","last_clean":"2024-05-12T22:08:58.649008+0000","last_became_active":"2024-05-12T22:06:46.793147+0000","last_became_peered":"2024-05-12T22:06:46.793147+0000","last_unstale":"2024-05-12T22:08:58.649008+0000","last_undegraded":"2024-05-12T22:08:58.649008+0000","last_fullsized":"2024-05-12T22:08:58.649008+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T23:10:39.529863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,8,2],"acting":[3,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"88'6","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.786430+0000","last_change":"2024-05-12T22:06:33.973490+0000","last_active":"2024-05-12T22:08:58.786430+0000","last_peered":"2024-05-12T22:08:58.786430+0000","last_clean":"2024-05-12T22:08:58.786430+0000","last_became_active":"2024-05-12T22:06:33.973161+0000","last_became_peered":"2024-05-12T22:06:33.973161+0000","last_unstale":"2024-05-12T22:08:58.786430+0000","last_undegraded":"2024-05-12T22:08:58.786430+0000","last_fullsized":"2024-05-12T22:08:58.786430+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T03:29:36.972631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"88'8","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.648590+0000","last_change":"2024-05-12T22:06:33.957320+0000","last_active":"2024-05-12T22:08:58.648590+0000","last_peered":"2024-05-12T22:08:58.648590+0000","last_clean":"2024-05-12T22:08:58.648590+0000","last_became_active":"2024-05-12T22:06:33.956837+0000","last_became_peered":"2024-05-12T22:06:33.956837+0000","last_unstale":"2024-05-12T22:08:58.648590+0000","last_undegraded":"2024-05-12T22:08:58.648590+0000","last_fullsized":"2024-05-12T22:08:58.648590+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T01:20:51.018175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"88'10","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.981497+0000","last_change":"2024-05-12T22:06:33.968363+0000","last_active":"2024-05-12T22:08:58.981497+0000","last_peered":"2024-05-12T22:08:58.981497+0000","last_clean":"2024-05-12T22:08:58.981497+0000","last_became_active":"2024-05-12T22:06:33.968031+0000","last_became_peered":"2024-05-12T22:06:33.968031+0000","last_unstale":"2024-05-12T22:08:58.981497+0000","last_undegraded":"2024-05-12T22:08:58.981497+0000","last_fullsized":"2024-05-12T22:08:58.981497+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T00:14:07.124058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"88'11","reported_seq":31,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:58.967933+0000","last_change":"2024-05-12T22:06:33.984823+0000","last_active":"2024-05-12T22:08:58.967933+0000","last_peered":"2024-05-12T22:08:58.967933+0000","last_clean":"2024-05-12T22:08:58.967933+0000","last_became_active":"2024-05-12T22:06:33.984430+0000","last_became_peered":"2024-05-12T22:06:33.984430+0000","last_unstale":"2024-05-12T22:08:58.967933+0000","last_undegraded":"2024-05-12T22:08:58.967933+0000","last_fullsized":"2024-05-12T22:08:58.967933+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T01:30:18.109265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"88'8","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-12T22:08:59.369891+0000","last_change":"2024-05-12T22:06:33.962660+0000","last_active":"2024-05-12T22:08:59.369891+0000","last_peered":"2024-05-12T22:08:59.369891+0000","last_clean":"2024-05-12T22:08:59.369891+0000","last_became_active":"2024-05-12T22:06:33.962284+0000","last_became_peered":"2024-05-12T22:06:33.962284+0000","last_unstale":"2024-05-12T22:08:59.369891+0000","last_undegraded":"2024-05-12T22:08:59.369891+0000","last_fullsized":"2024-05-12T22:08:59.369891+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T22:06:32.906529+0000","last_clean_scrub_stamp":"2024-05-12T22:06:32.906529+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-14T01:08:53.888992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":598250,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":972,"num_read_kb":596,"num_write":294,"num_write_kb":1443,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2027520,"data_stored":1794750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":232,"ondisk_log_size":232,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351018,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27948,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717204,"statfs":{"total":95995035648,"available":95966416896,"internally_reserved":0,"allocated":954368,"data_stored":459830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612654,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28840,"kb_used_data":1764,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716312,"statfs":{"total":95995035648,"available":95965503488,"internally_reserved":0,"allocated":1806336,"data_stored":1260157,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776179,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28364,"kb_used_data":1344,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716788,"statfs":{"total":95995035648,"available":95965990912,"internally_reserved":0,"allocated":1376256,"data_stored":861627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037818,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32524,"kb_used_data":1344,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712628,"statfs":{"total":95995035648,"available":95961731072,"internally_reserved":0,"allocated":1376256,"data_stored":861547,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299454,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28556,"kb_used_data":1536,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716596,"statfs":{"total":95995035648,"available":95965794304,"internally_reserved":0,"allocated":1572864,"data_stored":1061525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462981,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":970752,"data_stored":467800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574944,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36356,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708796,"statfs":{"total":95995035648,"available":95957807104,"internally_reserved":0,"allocated":1040384,"data_stored":535180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738469,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":991232,"data_stored":469071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411419,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28060,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717092,"statfs":{"total":95995035648,"available":95966302208,"internally_reserved":0,"allocated":1064960,"data_stored":536280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149781,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28496,"kb_used_data":1480,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716656,"statfs":{"total":95995035648,"available":95965855744,"internally_reserved":0,"allocated":1515520,"data_stored":992927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888142,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27972,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717180,"statfs":{"total":95995035648,"available":95966392320,"internally_reserved":0,"allocated":974848,"data_stored":469551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724618,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28552,"kb_used_data":1536,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716600,"statfs":{"total":95995035648,"available":95965798400,"internally_reserved":0,"allocated":1572864,"data_stored":1051159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":67587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":69860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":544768,"data_stored":525100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":133999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3447,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":413696,"data_stored":393954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":413696,"data_stored":394034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":202079,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-12T22:09:05.373 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-12T22:09:05.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:05.608+0000 7fee3bc00640 1 -- 172.21.15.28:0/2585221588 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fee34104c80 msgr2=0x7fee34105100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:05.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:05.608+0000 7fee3bc00640 1 --2- 172.21.15.28:0/2585221588 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fee34104c80 0x7fee34105100 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fee20009900 tx=0x7fee2002a400 comp rx=0 tx=0).stop 2024-05-12T22:09:05.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:05.608+0000 7fee3bc00640 1 -- 172.21.15.28:0/2585221588 shutdown_connections 2024-05-12T22:09:05.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:05.608+0000 7fee3bc00640 1 --2- 172.21.15.28:0/2585221588 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee34105640 0x7fee3410bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:05.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:05.608+0000 7fee3bc00640 1 --2- 172.21.15.28:0/2585221588 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fee34104c80 0x7fee34105100 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:05.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:05.608+0000 7fee3bc00640 1 --2- 172.21.15.28:0/2585221588 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fee34103a90 0x7fee34103eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:05.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:05.608+0000 7fee3bc00640 1 -- 172.21.15.28:0/2585221588 >> 172.21.15.28:0/2585221588 conn(0x7fee340ff020 msgr2=0x7fee34101440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:09:05.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:05.608+0000 7fee3bc00640 1 -- 172.21.15.28:0/2585221588 shutdown_connections 2024-05-12T22:09:05.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:05.608+0000 7fee3bc00640 1 -- 172.21.15.28:0/2585221588 wait complete. 2024-05-12T22:09:05.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:05.609+0000 7fee3bc00640 1 Processor -- start 2024-05-12T22:09:05.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:05.610+0000 7fee3bc00640 1 -- start start 2024-05-12T22:09:05.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:05.610+0000 7fee3bc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee34103a90 0x7fee34199db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:05.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:05.610+0000 7fee3bc00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fee34104c80 0x7fee3419a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:05.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:05.610+0000 7fee3bc00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fee34105640 0x7fee341a1360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:05.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:05.610+0000 7fee3bc00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fee3410f740 con 0x7fee34103a90 2024-05-12T22:09:05.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:05.610+0000 7fee3bc00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fee3410f5c0 con 0x7fee34105640 2024-05-12T22:09:05.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:05.610+0000 7fee3bc00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fee3410f8c0 con 0x7fee34104c80 2024-05-12T22:09:05.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:05.610+0000 7fee39200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee34103a90 0x7fee34199db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:05.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:05.610+0000 7fee33e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fee34104c80 0x7fee3419a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:05.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:05.610+0000 7fee33e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fee34104c80 0x7fee3419a2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:42560/0 (socket says 172.21.15.28:42560) 2024-05-12T22:09:05.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:05.610+0000 7fee33e00640 1 -- 172.21.15.28:0/1171565211 learned_addr learned my addr 172.21.15.28:0/1171565211 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:09:05.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:05.610+0000 7fee39c00640 1 --2- 172.21.15.28:0/1171565211 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fee34105640 0x7fee341a1360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:05.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:05.611+0000 7fee33e00640 1 -- 172.21.15.28:0/1171565211 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fee34105640 msgr2=0x7fee341a1360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:05.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:05.611+0000 7fee33e00640 1 --2- 172.21.15.28:0/1171565211 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fee34105640 0x7fee341a1360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:05.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:05.611+0000 7fee33e00640 1 -- 172.21.15.28:0/1171565211 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee34103a90 msgr2=0x7fee34199db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:05.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:05.611+0000 7fee33e00640 1 --2- 172.21.15.28:0/1171565211 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee34103a90 0x7fee34199db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:05.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:05.611+0000 7fee33e00640 1 -- 172.21.15.28:0/1171565211 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee341a19d0 con 0x7fee34104c80 2024-05-12T22:09:05.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:05.611+0000 7fee33e00640 1 --2- 172.21.15.28:0/1171565211 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fee34104c80 0x7fee3419a2f0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fee20031440 tx=0x7fee20032ce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:09:05.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:05.612+0000 7fee31600640 1 -- 172.21.15.28:0/1171565211 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee2003f030 con 0x7fee34104c80 2024-05-12T22:09:05.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:05.612+0000 7fee31600640 1 -- 172.21.15.28:0/1171565211 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fee20003ce0 con 0x7fee34104c80 2024-05-12T22:09:05.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:05.612+0000 7fee31600640 1 -- 172.21.15.28:0/1171565211 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee20035070 con 0x7fee34104c80 2024-05-12T22:09:05.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:05.612+0000 7fee3bc00640 1 -- 172.21.15.28:0/1171565211 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee341a1c60 con 0x7fee34104c80 2024-05-12T22:09:05.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:05.612+0000 7fee3bc00640 1 -- 172.21.15.28:0/1171565211 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee341a21a0 con 0x7fee34104c80 2024-05-12T22:09:05.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:05.613+0000 7fee3bc00640 1 -- 172.21.15.28:0/1171565211 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee3410adb0 con 0x7fee34104c80 2024-05-12T22:09:05.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:05.617+0000 7fee31600640 1 -- 172.21.15.28:0/1171565211 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 25) ==== 138060+0+0 (secure 0 0 0) 0x7fee20042030 con 0x7fee34104c80 2024-05-12T22:09:05.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:05.618+0000 7fee31600640 1 --2- 172.21.15.28:0/1171565211 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fee1c0a2ea0 0x7fee1c0a5360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:05.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:05.618+0000 7fee31600640 1 -- 172.21.15.28:0/1171565211 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7fee200efa00 con 0x7fee34104c80 2024-05-12T22:09:05.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:05.618+0000 7fee39200640 1 --2- 172.21.15.28:0/1171565211 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fee1c0a2ea0 0x7fee1c0a5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:05.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:05.619+0000 7fee39200640 1 --2- 172.21.15.28:0/1171565211 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fee1c0a2ea0 0x7fee1c0a5360 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fee34104ae0 tx=0x7fee28005f90 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:09:05.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:05.622+0000 7fee31600640 1 -- 172.21.15.28:0/1171565211 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fee200bc4d0 con 0x7fee34104c80 2024-05-12T22:09:05.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:05.845+0000 7fee3bc00640 1 -- 172.21.15.28:0/1171565211 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fee3402d3a0 con 0x7fee34104c80 2024-05-12T22:09:05.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:05.846+0000 7fee31600640 1 -- 172.21.15.28:0/1171565211 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+22972 (secure 0 0 0) 0x7fee200bbdd0 con 0x7fee34104c80 2024-05-12T22:09:05.847 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-12T22:09:05.848 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":89,"fsid":"6b92e7b4-10aa-11ef-bc98-c7b262605968","created":"2024-05-12T21:57:37.276782+0000","modified":"2024-05-12T22:08:58.638578+0000","last_up_change":"2024-05-12T22:05:37.929562+0000","last_in_change":"2024-05-12T22:05:18.750768+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-12T22:01:32.292826+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-12T22:06:31.540263+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-12T22:06:32.906118+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6992ce0a-844b-40e3-8ed1-181c8474dccf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":906003882},{"type":"v1","addr":"172.21.15.28:6803","nonce":906003882}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":906003882},{"type":"v1","addr":"172.21.15.28:6805","nonce":906003882}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":906003882},{"type":"v1","addr":"172.21.15.28:6809","nonce":906003882}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":906003882},{"type":"v1","addr":"172.21.15.28:6807","nonce":906003882}]},"public_addr":"172.21.15.28:6803/906003882","cluster_addr":"172.21.15.28:6805/906003882","heartbeat_back_addr":"172.21.15.28:6809/906003882","heartbeat_front_addr":"172.21.15.28:6807/906003882","state":["exists","up"]},{"osd":1,"uuid":"ced20c96-c481-4d48-90d1-6b43228959ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":3354141819},{"type":"v1","addr":"172.21.15.86:6801","nonce":3354141819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":3354141819},{"type":"v1","addr":"172.21.15.86:6803","nonce":3354141819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":3354141819},{"type":"v1","addr":"172.21.15.86:6807","nonce":3354141819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":3354141819},{"type":"v1","addr":"172.21.15.86:6805","nonce":3354141819}]},"public_addr":"172.21.15.86:6801/3354141819","cluster_addr":"172.21.15.86:6803/3354141819","heartbeat_back_addr":"172.21.15.86:6807/3354141819","heartbeat_front_addr":"172.21.15.86:6805/3354141819","state":["exists","up"]},{"osd":2,"uuid":"54a2f221-fa65-4795-9f5e-e660cc7c0098","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":906459365},{"type":"v1","addr":"172.21.15.186:6801","nonce":906459365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":906459365},{"type":"v1","addr":"172.21.15.186:6803","nonce":906459365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":906459365},{"type":"v1","addr":"172.21.15.186:6807","nonce":906459365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":906459365},{"type":"v1","addr":"172.21.15.186:6805","nonce":906459365}]},"public_addr":"172.21.15.186:6801/906459365","cluster_addr":"172.21.15.186:6803/906459365","heartbeat_back_addr":"172.21.15.186:6807/906459365","heartbeat_front_addr":"172.21.15.186:6805/906459365","state":["exists","up"]},{"osd":3,"uuid":"86836737-ac9a-487b-93f2-7fcd3c406960","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":4182884436},{"type":"v1","addr":"172.21.15.28:6811","nonce":4182884436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":4182884436},{"type":"v1","addr":"172.21.15.28:6813","nonce":4182884436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":4182884436},{"type":"v1","addr":"172.21.15.28:6817","nonce":4182884436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":4182884436},{"type":"v1","addr":"172.21.15.28:6815","nonce":4182884436}]},"public_addr":"172.21.15.28:6811/4182884436","cluster_addr":"172.21.15.28:6813/4182884436","heartbeat_back_addr":"172.21.15.28:6817/4182884436","heartbeat_front_addr":"172.21.15.28:6815/4182884436","state":["exists","up"]},{"osd":4,"uuid":"d162ebcd-83c0-4fd5-9925-28b2cecea935","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2596998708},{"type":"v1","addr":"172.21.15.86:6809","nonce":2596998708}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2596998708},{"type":"v1","addr":"172.21.15.86:6811","nonce":2596998708}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2596998708},{"type":"v1","addr":"172.21.15.86:6815","nonce":2596998708}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2596998708},{"type":"v1","addr":"172.21.15.86:6813","nonce":2596998708}]},"public_addr":"172.21.15.86:6809/2596998708","cluster_addr":"172.21.15.86:6811/2596998708","heartbeat_back_addr":"172.21.15.86:6815/2596998708","heartbeat_front_addr":"172.21.15.86:6813/2596998708","state":["exists","up"]},{"osd":5,"uuid":"c794336a-a105-425c-b6c7-a32b90c05bb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":1944205756},{"type":"v1","addr":"172.21.15.186:6809","nonce":1944205756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":1944205756},{"type":"v1","addr":"172.21.15.186:6811","nonce":1944205756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":1944205756},{"type":"v1","addr":"172.21.15.186:6815","nonce":1944205756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":1944205756},{"type":"v1","addr":"172.21.15.186:6813","nonce":1944205756}]},"public_addr":"172.21.15.186:6809/1944205756","cluster_addr":"172.21.15.186:6811/1944205756","heartbeat_back_addr":"172.21.15.186:6815/1944205756","heartbeat_front_addr":"172.21.15.186:6813/1944205756","state":["exists","up"]},{"osd":6,"uuid":"516932ac-af00-45b7-ad3c-d6dec7db442f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":1642308925},{"type":"v1","addr":"172.21.15.28:6819","nonce":1642308925}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":1642308925},{"type":"v1","addr":"172.21.15.28:6821","nonce":1642308925}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1642308925},{"type":"v1","addr":"172.21.15.28:6825","nonce":1642308925}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":1642308925},{"type":"v1","addr":"172.21.15.28:6823","nonce":1642308925}]},"public_addr":"172.21.15.28:6819/1642308925","cluster_addr":"172.21.15.28:6821/1642308925","heartbeat_back_addr":"172.21.15.28:6825/1642308925","heartbeat_front_addr":"172.21.15.28:6823/1642308925","state":["exists","up"]},{"osd":7,"uuid":"ca7a059f-0414-438e-b2d1-a10f348036be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":726062711},{"type":"v1","addr":"172.21.15.86:6817","nonce":726062711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":726062711},{"type":"v1","addr":"172.21.15.86:6819","nonce":726062711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":726062711},{"type":"v1","addr":"172.21.15.86:6823","nonce":726062711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":726062711},{"type":"v1","addr":"172.21.15.86:6821","nonce":726062711}]},"public_addr":"172.21.15.86:6817/726062711","cluster_addr":"172.21.15.86:6819/726062711","heartbeat_back_addr":"172.21.15.86:6823/726062711","heartbeat_front_addr":"172.21.15.86:6821/726062711","state":["exists","up"]},{"osd":8,"uuid":"92e3f41f-aebd-422e-a99e-555f7ba4aa53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":3007687277},{"type":"v1","addr":"172.21.15.186:6817","nonce":3007687277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":3007687277},{"type":"v1","addr":"172.21.15.186:6819","nonce":3007687277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":3007687277},{"type":"v1","addr":"172.21.15.186:6823","nonce":3007687277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":3007687277},{"type":"v1","addr":"172.21.15.186:6821","nonce":3007687277}]},"public_addr":"172.21.15.186:6817/3007687277","cluster_addr":"172.21.15.186:6819/3007687277","heartbeat_back_addr":"172.21.15.186:6823/3007687277","heartbeat_front_addr":"172.21.15.186:6821/3007687277","state":["exists","up"]},{"osd":9,"uuid":"38ff7d9d-ee85-404c-8c59-adc327e1ccfa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":136220472},{"type":"v1","addr":"172.21.15.28:6827","nonce":136220472}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":136220472},{"type":"v1","addr":"172.21.15.28:6829","nonce":136220472}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":136220472},{"type":"v1","addr":"172.21.15.28:6833","nonce":136220472}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":136220472},{"type":"v1","addr":"172.21.15.28:6831","nonce":136220472}]},"public_addr":"172.21.15.28:6827/136220472","cluster_addr":"172.21.15.28:6829/136220472","heartbeat_back_addr":"172.21.15.28:6833/136220472","heartbeat_front_addr":"172.21.15.28:6831/136220472","state":["exists","up"]},{"osd":10,"uuid":"5e9ff657-afa8-4bab-8491-f209038392b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1189968310},{"type":"v1","addr":"172.21.15.86:6825","nonce":1189968310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":1189968310},{"type":"v1","addr":"172.21.15.86:6827","nonce":1189968310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":1189968310},{"type":"v1","addr":"172.21.15.86:6831","nonce":1189968310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":1189968310},{"type":"v1","addr":"172.21.15.86:6829","nonce":1189968310}]},"public_addr":"172.21.15.86:6825/1189968310","cluster_addr":"172.21.15.86:6827/1189968310","heartbeat_back_addr":"172.21.15.86:6831/1189968310","heartbeat_front_addr":"172.21.15.86:6829/1189968310","state":["exists","up"]},{"osd":11,"uuid":"f112e3c6-65b6-4253-a966-62001db7d069","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3801668110},{"type":"v1","addr":"172.21.15.186:6825","nonce":3801668110}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3801668110},{"type":"v1","addr":"172.21.15.186:6827","nonce":3801668110}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3801668110},{"type":"v1","addr":"172.21.15.186:6831","nonce":3801668110}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3801668110},{"type":"v1","addr":"172.21.15.186:6829","nonce":3801668110}]},"public_addr":"172.21.15.186:6825/3801668110","cluster_addr":"172.21.15.186:6827/3801668110","heartbeat_back_addr":"172.21.15.186:6831/3801668110","heartbeat_front_addr":"172.21.15.186:6829/3801668110","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:00:40.483623+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:01:05.284385+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:01:28.645716+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:02:01.295092+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:02:26.486601+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:02:50.807478+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:03:24.756169+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:03:49.975173+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:04:13.772233+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:04:46.758555+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:05:12.331685+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:05:36.462548+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":2,"to":1}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.25","mappings":[{"from":5,"to":9}]},{"pgid":"3.27","mappings":[{"from":4,"to":8}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/2580087728":"2024-05-13T22:08:58.638536+0000","172.21.15.28:6801/3123328651":"2024-05-13T22:08:58.638536+0000","172.21.15.28:0/820158394":"2024-05-13T22:08:58.638536+0000","172.21.15.28:0/1978033727":"2024-05-13T22:08:58.638536+0000","172.21.15.28:0/1177009533":"2024-05-13T21:58:45.550459+0000","172.21.15.28:6800/3760499540":"2024-05-13T21:58:45.550459+0000","172.21.15.28:6801/3783742728":"2024-05-13T21:58:10.421564+0000","172.21.15.28:6800/3783742728":"2024-05-13T21:58:10.421564+0000","172.21.15.28:0/2229156260":"2024-05-13T21:58:45.550459+0000","172.21.15.28:0/1202140963":"2024-05-13T21:58:10.421564+0000","172.21.15.28:6800/3123328651":"2024-05-13T22:08:58.638536+0000","172.21.15.28:0/2924223784":"2024-05-13T21:58:45.550459+0000","172.21.15.28:6801/3760499540":"2024-05-13T21:58:45.550459+0000","172.21.15.28:0/1240027206":"2024-05-13T21:58:10.421564+0000","172.21.15.28:0/1581755795":"2024-05-13T22:08:58.638536+0000","172.21.15.28:0/171417728":"2024-05-13T21:58:10.421564+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-12T22:09:05.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:05.849+0000 7fee3bc00640 1 -- 172.21.15.28:0/1171565211 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fee1c0a2ea0 msgr2=0x7fee1c0a5360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:05.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:05.849+0000 7fee3bc00640 1 --2- 172.21.15.28:0/1171565211 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fee1c0a2ea0 0x7fee1c0a5360 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fee34104ae0 tx=0x7fee28005f90 comp rx=0 tx=0).stop 2024-05-12T22:09:05.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:05.850+0000 7fee3bc00640 1 -- 172.21.15.28:0/1171565211 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fee34104c80 msgr2=0x7fee3419a2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:05.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:05.850+0000 7fee3bc00640 1 --2- 172.21.15.28:0/1171565211 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fee34104c80 0x7fee3419a2f0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fee20031440 tx=0x7fee20032ce0 comp rx=0 tx=0).stop 2024-05-12T22:09:05.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:05.850+0000 7fee3bc00640 1 -- 172.21.15.28:0/1171565211 shutdown_connections 2024-05-12T22:09:05.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:05.850+0000 7fee3bc00640 1 --2- 172.21.15.28:0/1171565211 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fee1c0a2ea0 0x7fee1c0a5360 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:05.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:05.850+0000 7fee3bc00640 1 --2- 172.21.15.28:0/1171565211 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fee34105640 0x7fee341a1360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:05.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:05.850+0000 7fee3bc00640 1 --2- 172.21.15.28:0/1171565211 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fee34104c80 0x7fee3419a2f0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:05.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:05.850+0000 7fee3bc00640 1 --2- 172.21.15.28:0/1171565211 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee34103a90 0x7fee34199db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:05.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:05.850+0000 7fee3bc00640 1 -- 172.21.15.28:0/1171565211 >> 172.21.15.28:0/1171565211 conn(0x7fee340ff020 msgr2=0x7fee34107f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:09:05.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:05.850+0000 7fee3bc00640 1 -- 172.21.15.28:0/1171565211 shutdown_connections 2024-05-12T22:09:05.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:05.850+0000 7fee3bc00640 1 -- 172.21.15.28:0/1171565211 wait complete. 2024-05-12T22:09:05.876 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-12T22:01:32.292826+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-12T22:06:31.540263+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-12T22:06:32.906118+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-12T22:09:05.876 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-12T22:09:06.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:05 smithi028 ceph-mon[29847]: from='client.25324 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-12T22:09:06.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:05 smithi028 ceph-mon[29847]: pgmap v6: 97 pgs: 97 active+clean; 1.1 MiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-12T22:09:06.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:05 smithi028 ceph-mon[29847]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:09:06.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:05 smithi028 ceph-mon[29847]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:09:06.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:05 smithi028 ceph-mon[29847]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:09:06.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:05 smithi028 ceph-mon[29847]: from='mgr.25189 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:09:06.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:05 smithi028 ceph-mon[29847]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:09:06.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:05 smithi028 ceph-mon[29847]: from='mgr.25189 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:09:06.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:05 smithi028 ceph-mon[29847]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:09:06.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:05 smithi028 ceph-mon[29847]: from='mgr.25189 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:09:06.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:05 smithi028 ceph-mon[29847]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:09:06.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:05 smithi028 ceph-mon[29847]: from='mgr.25189 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:09:06.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:05 smithi028 ceph-mon[29847]: Adjusting osd_memory_target on smithi028 to 177.5M 2024-05-12T22:09:06.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:05 smithi028 ceph-mon[29847]: Unable to set osd_memory_target on smithi028 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-05-12T22:09:06.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:05 smithi028 ceph-mon[29847]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:09:06.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:05 smithi028 ceph-mon[29847]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:09:06.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:05 smithi028 ceph-mon[29847]: Updating smithi028:/etc/ceph/ceph.conf 2024-05-12T22:09:06.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:05 smithi028 ceph-mon[29847]: Updating smithi086:/etc/ceph/ceph.conf 2024-05-12T22:09:06.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:05 smithi028 ceph-mon[29847]: Updating smithi186:/etc/ceph/ceph.conf 2024-05-12T22:09:06.037 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:05 smithi028 ceph-mon[29847]: from='client.25330 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-12T22:09:06.037 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:05 smithi028 ceph-mon[29847]: Updating smithi086:/var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-12T22:09:06.037 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:05 smithi028 ceph-mon[29847]: Updating smithi186:/var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-12T22:09:06.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.069+0000 7fb6f3000640 1 -- 172.21.15.28:0/2832189102 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb6ec103a90 msgr2=0x7fb6ec103eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:06.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.069+0000 7fb6f3000640 1 --2- 172.21.15.28:0/2832189102 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb6ec103a90 0x7fb6ec103eb0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fb6dc009900 tx=0x7fb6dc02a410 comp rx=0 tx=0).stop 2024-05-12T22:09:06.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.069+0000 7fb6f3000640 1 -- 172.21.15.28:0/2832189102 shutdown_connections 2024-05-12T22:09:06.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.069+0000 7fb6f3000640 1 --2- 172.21.15.28:0/2832189102 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6ec105640 0x7fb6ec10bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:06.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.069+0000 7fb6f3000640 1 --2- 172.21.15.28:0/2832189102 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb6ec104c80 0x7fb6ec105100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:06.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.069+0000 7fb6f3000640 1 --2- 172.21.15.28:0/2832189102 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb6ec103a90 0x7fb6ec103eb0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:06.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.069+0000 7fb6f3000640 1 -- 172.21.15.28:0/2832189102 >> 172.21.15.28:0/2832189102 conn(0x7fb6ec0ff020 msgr2=0x7fb6ec101440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:09:06.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.069+0000 7fb6f3000640 1 -- 172.21.15.28:0/2832189102 shutdown_connections 2024-05-12T22:09:06.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.069+0000 7fb6f3000640 1 -- 172.21.15.28:0/2832189102 wait complete. 2024-05-12T22:09:06.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.070+0000 7fb6f3000640 1 Processor -- start 2024-05-12T22:09:06.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.070+0000 7fb6f3000640 1 -- start start 2024-05-12T22:09:06.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.071+0000 7fb6f3000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb6ec103a90 0x7fb6ec199e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:06.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.071+0000 7fb6f3000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6ec104c80 0x7fb6ec19a360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:06.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.071+0000 7fb6f3000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb6ec105640 0x7fb6ec1a1380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:06.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.071+0000 7fb6f3000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb6ec10f730 con 0x7fb6ec104c80 2024-05-12T22:09:06.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.071+0000 7fb6f3000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fb6ec10f5b0 con 0x7fb6ec105640 2024-05-12T22:09:06.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.071+0000 7fb6f3000640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb6ec10f8b0 con 0x7fb6ec103a90 2024-05-12T22:09:06.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.071+0000 7fb6e3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6ec104c80 0x7fb6ec19a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:06.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.071+0000 7fb6ebe00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb6ec103a90 0x7fb6ec199e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:06.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.071+0000 7fb6e3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6ec104c80 0x7fb6ec19a360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49416/0 (socket says 172.21.15.28:49416) 2024-05-12T22:09:06.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.071+0000 7fb6e3400640 1 -- 172.21.15.28:0/3671947483 learned_addr learned my addr 172.21.15.28:0/3671947483 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:09:06.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.071+0000 7fb6ebe00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb6ec103a90 0x7fb6ec199e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:53282/0 (socket says 172.21.15.28:53282) 2024-05-12T22:09:06.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.071+0000 7fb6f1000640 1 --2- 172.21.15.28:0/3671947483 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb6ec105640 0x7fb6ec1a1380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:06.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.071+0000 7fb6e3400640 1 -- 172.21.15.28:0/3671947483 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb6ec103a90 msgr2=0x7fb6ec199e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:06.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.071+0000 7fb6e3400640 1 --2- 172.21.15.28:0/3671947483 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb6ec103a90 0x7fb6ec199e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:06.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.071+0000 7fb6e3400640 1 -- 172.21.15.28:0/3671947483 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb6ec105640 msgr2=0x7fb6ec1a1380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:06.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.071+0000 7fb6e3400640 1 --2- 172.21.15.28:0/3671947483 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb6ec105640 0x7fb6ec1a1380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:06.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.072+0000 7fb6e3400640 1 -- 172.21.15.28:0/3671947483 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6ec1a19f0 con 0x7fb6ec104c80 2024-05-12T22:09:06.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.072+0000 7fb6f1000640 1 --2- 172.21.15.28:0/3671947483 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb6ec105640 0x7fb6ec1a1380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:09:06.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.072+0000 7fb6e3400640 1 --2- 172.21.15.28:0/3671947483 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6ec104c80 0x7fb6ec19a360 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fb6d400e9f0 tx=0x7fb6d400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:09:06.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.072+0000 7fb6e9600640 1 -- 172.21.15.28:0/3671947483 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6d400cd80 con 0x7fb6ec104c80 2024-05-12T22:09:06.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.072+0000 7fb6e9600640 1 -- 172.21.15.28:0/3671947483 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb6d4010910 con 0x7fb6ec104c80 2024-05-12T22:09:06.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.072+0000 7fb6f3000640 1 -- 172.21.15.28:0/3671947483 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6ec1a1ce0 con 0x7fb6ec104c80 2024-05-12T22:09:06.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.072+0000 7fb6e9600640 1 -- 172.21.15.28:0/3671947483 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6d4018990 con 0x7fb6ec104c80 2024-05-12T22:09:06.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.072+0000 7fb6f3000640 1 -- 172.21.15.28:0/3671947483 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6ec1a2220 con 0x7fb6ec104c80 2024-05-12T22:09:06.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.074+0000 7fb6f3000640 1 -- 172.21.15.28:0/3671947483 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6ec10ae80 con 0x7fb6ec104c80 2024-05-12T22:09:06.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.076+0000 7fb6e9600640 1 -- 172.21.15.28:0/3671947483 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138060+0+0 (secure 0 0 0) 0x7fb6d4010ac0 con 0x7fb6ec104c80 2024-05-12T22:09:06.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.077+0000 7fb6e9600640 1 --2- 172.21.15.28:0/3671947483 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fb6d00a2ea0 0x7fb6d00a5360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:06.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.077+0000 7fb6e9600640 1 -- 172.21.15.28:0/3671947483 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7fb6d40cf9a0 con 0x7fb6ec104c80 2024-05-12T22:09:06.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.078+0000 7fb6ebe00640 1 --2- 172.21.15.28:0/3671947483 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fb6d00a2ea0 0x7fb6d00a5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:06.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.079+0000 7fb6ebe00640 1 --2- 172.21.15.28:0/3671947483 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fb6d00a2ea0 0x7fb6d00a5360 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fb6dc02aa00 tx=0x7fb6dc0315b0 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:09:06.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.084+0000 7fb6e9600640 1 -- 172.21.15.28:0/3671947483 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fb6d409bcf0 con 0x7fb6ec104c80 2024-05-12T22:09:06.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:05 smithi186 ceph-mon[34785]: from='client.25324 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-12T22:09:06.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:05 smithi186 ceph-mon[34785]: pgmap v6: 97 pgs: 97 active+clean; 1.1 MiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-12T22:09:06.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:05 smithi186 ceph-mon[34785]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:09:06.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:05 smithi186 ceph-mon[34785]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:09:06.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:05 smithi186 ceph-mon[34785]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:09:06.180 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:05 smithi186 ceph-mon[34785]: from='mgr.25189 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:09:06.180 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:05 smithi186 ceph-mon[34785]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:09:06.180 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:05 smithi186 ceph-mon[34785]: from='mgr.25189 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:09:06.180 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:05 smithi186 ceph-mon[34785]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:09:06.180 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:05 smithi186 ceph-mon[34785]: from='mgr.25189 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:09:06.180 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:05 smithi186 ceph-mon[34785]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:09:06.180 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:05 smithi186 ceph-mon[34785]: from='mgr.25189 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:09:06.180 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:05 smithi186 ceph-mon[34785]: Adjusting osd_memory_target on smithi028 to 177.5M 2024-05-12T22:09:06.180 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:05 smithi186 ceph-mon[34785]: Unable to set osd_memory_target on smithi028 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-05-12T22:09:06.180 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:05 smithi186 ceph-mon[34785]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:09:06.180 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:05 smithi186 ceph-mon[34785]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:09:06.180 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:05 smithi186 ceph-mon[34785]: Updating smithi028:/etc/ceph/ceph.conf 2024-05-12T22:09:06.180 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:05 smithi186 ceph-mon[34785]: Updating smithi086:/etc/ceph/ceph.conf 2024-05-12T22:09:06.180 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:05 smithi186 ceph-mon[34785]: Updating smithi186:/etc/ceph/ceph.conf 2024-05-12T22:09:06.181 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:05 smithi186 ceph-mon[34785]: from='client.25330 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-12T22:09:06.181 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:05 smithi186 ceph-mon[34785]: Updating smithi086:/var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-12T22:09:06.181 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:05 smithi186 ceph-mon[34785]: Updating smithi186:/var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-12T22:09:06.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:05 smithi086 ceph-mon[34297]: from='client.25324 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-12T22:09:06.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:05 smithi086 ceph-mon[34297]: pgmap v6: 97 pgs: 97 active+clean; 1.1 MiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-12T22:09:06.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:05 smithi086 ceph-mon[34297]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:09:06.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:05 smithi086 ceph-mon[34297]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:09:06.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:05 smithi086 ceph-mon[34297]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:09:06.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:05 smithi086 ceph-mon[34297]: from='mgr.25189 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:09:06.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:05 smithi086 ceph-mon[34297]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:09:06.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:05 smithi086 ceph-mon[34297]: from='mgr.25189 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:09:06.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:05 smithi086 ceph-mon[34297]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:09:06.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:05 smithi086 ceph-mon[34297]: from='mgr.25189 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:09:06.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:05 smithi086 ceph-mon[34297]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:09:06.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:05 smithi086 ceph-mon[34297]: from='mgr.25189 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:09:06.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:05 smithi086 ceph-mon[34297]: Adjusting osd_memory_target on smithi028 to 177.5M 2024-05-12T22:09:06.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:05 smithi086 ceph-mon[34297]: Unable to set osd_memory_target on smithi028 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-05-12T22:09:06.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:05 smithi086 ceph-mon[34297]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:09:06.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:05 smithi086 ceph-mon[34297]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:09:06.244 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:05 smithi086 ceph-mon[34297]: Updating smithi028:/etc/ceph/ceph.conf 2024-05-12T22:09:06.244 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:05 smithi086 ceph-mon[34297]: Updating smithi086:/etc/ceph/ceph.conf 2024-05-12T22:09:06.244 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:05 smithi086 ceph-mon[34297]: Updating smithi186:/etc/ceph/ceph.conf 2024-05-12T22:09:06.244 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:05 smithi086 ceph-mon[34297]: from='client.25330 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-12T22:09:06.244 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:05 smithi086 ceph-mon[34297]: Updating smithi086:/var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-12T22:09:06.244 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:05 smithi086 ceph-mon[34297]: Updating smithi186:/var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-12T22:09:06.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.298+0000 7fb6f3000640 1 -- 172.21.15.28:0/3671947483 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fb6ec02d410 con 0x7fb6ec104c80 2024-05-12T22:09:06.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.299+0000 7fb6e9600640 1 -- 172.21.15.28:0/3671947483 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) ==== 93+0+10 (secure 0 0 0) 0x7fb6d409fff0 con 0x7fb6ec104c80 2024-05-12T22:09:06.299 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-05-12T22:09:06.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.301+0000 7fb6f3000640 1 -- 172.21.15.28:0/3671947483 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fb6d00a2ea0 msgr2=0x7fb6d00a5360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:06.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.301+0000 7fb6f3000640 1 --2- 172.21.15.28:0/3671947483 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fb6d00a2ea0 0x7fb6d00a5360 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fb6dc02aa00 tx=0x7fb6dc0315b0 comp rx=0 tx=0).stop 2024-05-12T22:09:06.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.301+0000 7fb6f3000640 1 -- 172.21.15.28:0/3671947483 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6ec104c80 msgr2=0x7fb6ec19a360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:06.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.301+0000 7fb6f3000640 1 --2- 172.21.15.28:0/3671947483 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6ec104c80 0x7fb6ec19a360 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fb6d400e9f0 tx=0x7fb6d400eeb0 comp rx=0 tx=0).stop 2024-05-12T22:09:06.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.301+0000 7fb6f3000640 1 -- 172.21.15.28:0/3671947483 shutdown_connections 2024-05-12T22:09:06.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.301+0000 7fb6f3000640 1 --2- 172.21.15.28:0/3671947483 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fb6d00a2ea0 0x7fb6d00a5360 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:06.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.301+0000 7fb6f3000640 1 --2- 172.21.15.28:0/3671947483 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb6ec105640 0x7fb6ec1a1380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:06.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.301+0000 7fb6f3000640 1 --2- 172.21.15.28:0/3671947483 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6ec104c80 0x7fb6ec19a360 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:06.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.301+0000 7fb6f3000640 1 --2- 172.21.15.28:0/3671947483 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb6ec103a90 0x7fb6ec199e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:06.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.301+0000 7fb6f3000640 1 -- 172.21.15.28:0/3671947483 >> 172.21.15.28:0/3671947483 conn(0x7fb6ec0ff020 msgr2=0x7fb6ec108050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:09:06.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.301+0000 7fb6f3000640 1 -- 172.21.15.28:0/3671947483 shutdown_connections 2024-05-12T22:09:06.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.302+0000 7fb6f3000640 1 -- 172.21.15.28:0/3671947483 wait complete. 2024-05-12T22:09:06.322 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-12T22:09:06.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.513+0000 7f4cc8600640 1 -- 172.21.15.28:0/3894986106 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4cc0107570 msgr2=0x7f4cc01099c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:06.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.513+0000 7f4cc8600640 1 --2- 172.21.15.28:0/3894986106 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4cc0107570 0x7f4cc01099c0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f4cb8009be0 tx=0x7f4cb802a6c0 comp rx=0 tx=0).stop 2024-05-12T22:09:06.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.513+0000 7f4cc8600640 1 -- 172.21.15.28:0/3894986106 shutdown_connections 2024-05-12T22:09:06.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.513+0000 7f4cc8600640 1 --2- 172.21.15.28:0/3894986106 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4cc0107570 0x7f4cc01099c0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:06.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.513+0000 7f4cc8600640 1 --2- 172.21.15.28:0/3894986106 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4cc0104c40 0x7f4cc0107030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:06.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.513+0000 7f4cc8600640 1 --2- 172.21.15.28:0/3894986106 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4cc0102310 0x7f4cc0104700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:06.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.513+0000 7f4cc8600640 1 -- 172.21.15.28:0/3894986106 >> 172.21.15.28:0/3894986106 conn(0x7f4cc00fbd20 msgr2=0x7f4cc00fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:09:06.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.513+0000 7f4cc8600640 1 -- 172.21.15.28:0/3894986106 shutdown_connections 2024-05-12T22:09:06.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.513+0000 7f4cc8600640 1 -- 172.21.15.28:0/3894986106 wait complete. 2024-05-12T22:09:06.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.514+0000 7f4cc8600640 1 Processor -- start 2024-05-12T22:09:06.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.515+0000 7f4cc8600640 1 -- start start 2024-05-12T22:09:06.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.515+0000 7f4cc8600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4cc0102310 0x7f4cc0101c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:06.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.515+0000 7f4cc8600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4cc0104c40 0x7f4cc01002c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:06.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.515+0000 7f4cc8600640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4cc0107570 0x7f4cc0100800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:06.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.515+0000 7f4cc8600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4cc010d510 con 0x7f4cc0104c40 2024-05-12T22:09:06.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.515+0000 7f4cc8600640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f4cc010d390 con 0x7f4cc0102310 2024-05-12T22:09:06.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.515+0000 7f4cc8600640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f4cc010d690 con 0x7f4cc0107570 2024-05-12T22:09:06.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.515+0000 7f4cc5c00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4cc0102310 0x7f4cc0101c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:06.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.515+0000 7f4cc5200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4cc0104c40 0x7f4cc01002c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:06.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.515+0000 7f4cc5c00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4cc0102310 0x7f4cc0101c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:51702/0 (socket says 172.21.15.28:51702) 2024-05-12T22:09:06.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.515+0000 7f4cc5c00640 1 -- 172.21.15.28:0/3225614891 learned_addr learned my addr 172.21.15.28:0/3225614891 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:09:06.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.515+0000 7f4cc6600640 1 --2- 172.21.15.28:0/3225614891 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4cc0107570 0x7f4cc0100800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:06.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.516+0000 7f4cc5c00640 1 -- 172.21.15.28:0/3225614891 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4cc0107570 msgr2=0x7f4cc0100800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:06.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.516+0000 7f4cc5c00640 1 --2- 172.21.15.28:0/3225614891 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4cc0107570 0x7f4cc0100800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:06.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.516+0000 7f4cc5c00640 1 -- 172.21.15.28:0/3225614891 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4cc0104c40 msgr2=0x7f4cc01002c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:06.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.516+0000 7f4cc5c00640 1 --2- 172.21.15.28:0/3225614891 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4cc0104c40 0x7f4cc01002c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:06.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.516+0000 7f4cc5c00640 1 -- 172.21.15.28:0/3225614891 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cc0101090 con 0x7f4cc0102310 2024-05-12T22:09:06.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.516+0000 7f4cc5c00640 1 --2- 172.21.15.28:0/3225614891 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4cc0102310 0x7f4cc0101c80 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f4cb400e9c0 tx=0x7f4cb400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:09:06.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.516+0000 7f4cbe000640 1 -- 172.21.15.28:0/3225614891 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cb400cde0 con 0x7f4cc0102310 2024-05-12T22:09:06.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.516+0000 7f4cbe000640 1 -- 172.21.15.28:0/3225614891 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4cb40035f0 con 0x7f4cc0102310 2024-05-12T22:09:06.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.516+0000 7f4cc8600640 1 -- 172.21.15.28:0/3225614891 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4cc01a1c90 con 0x7f4cc0102310 2024-05-12T22:09:06.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.516+0000 7f4cbe000640 1 -- 172.21.15.28:0/3225614891 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cb4010690 con 0x7f4cc0102310 2024-05-12T22:09:06.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.517+0000 7f4cc8600640 1 -- 172.21.15.28:0/3225614891 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4cc01a2180 con 0x7f4cc0102310 2024-05-12T22:09:06.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.518+0000 7f4cc8600640 1 -- 172.21.15.28:0/3225614891 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4cc00fd810 con 0x7f4cc0102310 2024-05-12T22:09:06.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.520+0000 7f4cbe000640 1 -- 172.21.15.28:0/3225614891 <== mon.2 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 25) ==== 138060+0+0 (secure 0 0 0) 0x7f4cb40037a0 con 0x7f4cc0102310 2024-05-12T22:09:06.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.521+0000 7f4cbe000640 1 --2- 172.21.15.28:0/3225614891 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f4ca80a2ea0 0x7f4ca80a5360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:06.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.521+0000 7f4cbe000640 1 -- 172.21.15.28:0/3225614891 <== mon.2 v2:172.21.15.86:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f4cb40cfa20 con 0x7f4cc0102310 2024-05-12T22:09:06.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.521+0000 7f4cc5200640 1 --2- 172.21.15.28:0/3225614891 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f4ca80a2ea0 0x7f4ca80a5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:06.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.522+0000 7f4cc5200640 1 --2- 172.21.15.28:0/3225614891 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f4ca80a2ea0 0x7f4ca80a5360 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f4cc01019f0 tx=0x7f4cac005f90 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:09:06.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.525+0000 7f4cbe000640 1 -- 172.21.15.28:0/3225614891 <== mon.2 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f4cb409c3c0 con 0x7f4cc0102310 2024-05-12T22:09:06.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.738+0000 7f4cc8600640 1 -- 172.21.15.28:0/3225614891 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f4cc002d440 con 0x7f4cc0102310 2024-05-12T22:09:06.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.738+0000 7f4cbe000640 1 -- 172.21.15.28:0/3225614891 <== mon.2 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) ==== 104+0+11 (secure 0 0 0) 0x7f4cb409bcc0 con 0x7f4cc0102310 2024-05-12T22:09:06.739 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-05-12T22:09:06.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.741+0000 7f4cc8600640 1 -- 172.21.15.28:0/3225614891 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f4ca80a2ea0 msgr2=0x7f4ca80a5360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:06.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.741+0000 7f4cc8600640 1 --2- 172.21.15.28:0/3225614891 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f4ca80a2ea0 0x7f4ca80a5360 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f4cc01019f0 tx=0x7f4cac005f90 comp rx=0 tx=0).stop 2024-05-12T22:09:06.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.741+0000 7f4cc8600640 1 -- 172.21.15.28:0/3225614891 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4cc0102310 msgr2=0x7f4cc0101c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:06.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.741+0000 7f4cc8600640 1 --2- 172.21.15.28:0/3225614891 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4cc0102310 0x7f4cc0101c80 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f4cb400e9c0 tx=0x7f4cb400ee80 comp rx=0 tx=0).stop 2024-05-12T22:09:06.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.742+0000 7f4cc8600640 1 -- 172.21.15.28:0/3225614891 shutdown_connections 2024-05-12T22:09:06.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.742+0000 7f4cc8600640 1 --2- 172.21.15.28:0/3225614891 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f4ca80a2ea0 0x7f4ca80a5360 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:06.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.742+0000 7f4cc8600640 1 --2- 172.21.15.28:0/3225614891 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4cc0107570 0x7f4cc0100800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:06.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.742+0000 7f4cc8600640 1 --2- 172.21.15.28:0/3225614891 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4cc0104c40 0x7f4cc01002c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:06.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.742+0000 7f4cc8600640 1 --2- 172.21.15.28:0/3225614891 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4cc0102310 0x7f4cc0101c80 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:06.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.742+0000 7f4cc8600640 1 -- 172.21.15.28:0/3225614891 >> 172.21.15.28:0/3225614891 conn(0x7f4cc00fbd20 msgr2=0x7f4cc0102c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:09:06.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.742+0000 7f4cc8600640 1 -- 172.21.15.28:0/3225614891 shutdown_connections 2024-05-12T22:09:06.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.742+0000 7f4cc8600640 1 -- 172.21.15.28:0/3225614891 wait complete. 2024-05-12T22:09:06.765 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-12T22:09:06.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.960+0000 7f55a4a00640 1 -- 172.21.15.28:0/2074892781 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55a010e240 msgr2=0x7f55a0110630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:06.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.960+0000 7f55a4a00640 1 --2- 172.21.15.28:0/2074892781 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55a010e240 0x7f55a0110630 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f5598009ca0 tx=0x7f559802a8b0 comp rx=0 tx=0).stop 2024-05-12T22:09:06.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.960+0000 7f55a4a00640 1 -- 172.21.15.28:0/2074892781 shutdown_connections 2024-05-12T22:09:06.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.960+0000 7f55a4a00640 1 --2- 172.21.15.28:0/2074892781 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55a010e240 0x7f55a0110630 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:06.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.960+0000 7f55a4a00640 1 --2- 172.21.15.28:0/2074892781 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f55a0100fb0 0x7f55a010dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:06.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.960+0000 7f55a4a00640 1 --2- 172.21.15.28:0/2074892781 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f55a0100690 0x7f55a0100a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:06.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.960+0000 7f55a4a00640 1 -- 172.21.15.28:0/2074892781 >> 172.21.15.28:0/2074892781 conn(0x7f55a0077890 msgr2=0x7f55a0077ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:09:06.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.960+0000 7f55a4a00640 1 -- 172.21.15.28:0/2074892781 shutdown_connections 2024-05-12T22:09:06.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.960+0000 7f55a4a00640 1 -- 172.21.15.28:0/2074892781 wait complete. 2024-05-12T22:09:06.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.961+0000 7f55a4a00640 1 Processor -- start 2024-05-12T22:09:06.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.961+0000 7f55a4a00640 1 -- start start 2024-05-12T22:09:06.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.961+0000 7f55a4a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f55a0100690 0x7f55a019a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:06.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.961+0000 7f55a4a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55a0100fb0 0x7f55a019a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:06.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.961+0000 7f55a4a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f55a010e240 0x7f55a019ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:06.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.961+0000 7f55a4a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f55a0113e70 con 0x7f55a0100fb0 2024-05-12T22:09:06.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.961+0000 7f55a4a00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f55a0113cf0 con 0x7f55a0100690 2024-05-12T22:09:06.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.961+0000 7f55a4a00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f55a0113ff0 con 0x7f55a010e240 2024-05-12T22:09:06.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.962+0000 7f559d400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55a0100fb0 0x7f55a019a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:06.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.962+0000 7f559de00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f55a0100690 0x7f55a019a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:06.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.962+0000 7f559de00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f55a0100690 0x7f55a019a2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:51708/0 (socket says 172.21.15.28:51708) 2024-05-12T22:09:06.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.962+0000 7f559de00640 1 -- 172.21.15.28:0/87697977 learned_addr learned my addr 172.21.15.28:0/87697977 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:09:06.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.962+0000 7f559d400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55a0100fb0 0x7f55a019a830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49438/0 (socket says 172.21.15.28:49438) 2024-05-12T22:09:06.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.962+0000 7f559e800640 1 --2- 172.21.15.28:0/87697977 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f55a010e240 0x7f55a019ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:06.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.962+0000 7f559d400640 1 -- 172.21.15.28:0/87697977 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f55a010e240 msgr2=0x7f55a019ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:06.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.962+0000 7f559d400640 1 --2- 172.21.15.28:0/87697977 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f55a010e240 0x7f55a019ebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:06.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.962+0000 7f559d400640 1 -- 172.21.15.28:0/87697977 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f55a0100690 msgr2=0x7f55a019a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:06.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.962+0000 7f559d400640 1 --2- 172.21.15.28:0/87697977 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f55a0100690 0x7f55a019a2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:06.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.962+0000 7f559d400640 1 -- 172.21.15.28:0/87697977 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55a019f2e0 con 0x7f55a0100fb0 2024-05-12T22:09:06.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.962+0000 7f559de00640 1 --2- 172.21.15.28:0/87697977 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f55a0100690 0x7f55a019a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:09:06.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.963+0000 7f559d400640 1 --2- 172.21.15.28:0/87697977 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55a0100fb0 0x7f55a019a830 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f558c00e9f0 tx=0x7f558c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:09:06.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.963+0000 7f5596a00640 1 -- 172.21.15.28:0/87697977 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f558c00ccc0 con 0x7f55a0100fb0 2024-05-12T22:09:06.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.963+0000 7f5596a00640 1 -- 172.21.15.28:0/87697977 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f558c005020 con 0x7f55a0100fb0 2024-05-12T22:09:06.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.963+0000 7f5596a00640 1 -- 172.21.15.28:0/87697977 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f558c0049e0 con 0x7f55a0100fb0 2024-05-12T22:09:06.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.963+0000 7f55a4a00640 1 -- 172.21.15.28:0/87697977 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55a019f5d0 con 0x7f55a0100fb0 2024-05-12T22:09:06.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.963+0000 7f55a4a00640 1 -- 172.21.15.28:0/87697977 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55a01a6e60 con 0x7f55a0100fb0 2024-05-12T22:09:06.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.966+0000 7f55a4a00640 1 -- 172.21.15.28:0/87697977 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f55a0067dc0 con 0x7f55a0100fb0 2024-05-12T22:09:06.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.967+0000 7f5596a00640 1 -- 172.21.15.28:0/87697977 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138060+0+0 (secure 0 0 0) 0x7f558c005560 con 0x7f55a0100fb0 2024-05-12T22:09:06.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.969+0000 7f5596a00640 1 --2- 172.21.15.28:0/87697977 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f55840a2ea0 0x7f55840a5360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:06.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.969+0000 7f5596a00640 1 -- 172.21.15.28:0/87697977 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f558c0cf510 con 0x7f55a0100fb0 2024-05-12T22:09:06.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.969+0000 7f559de00640 1 --2- 172.21.15.28:0/87697977 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f55840a2ea0 0x7f55840a5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:06.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.970+0000 7f559de00640 1 --2- 172.21.15.28:0/87697977 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f55840a2ea0 0x7f55840a5360 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f55900095f0 tx=0x7f5590005f90 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:09:06.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:06.973+0000 7f5596a00640 1 -- 172.21.15.28:0/87697977 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f558c0051d0 con 0x7f55a0100fb0 2024-05-12T22:09:07.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:06 smithi028 ceph-mon[29847]: Updating smithi086:/etc/ceph/ceph.client.admin.keyring 2024-05-12T22:09:07.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:06 smithi028 ceph-mon[29847]: from='client.25355 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-12T22:09:07.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:06 smithi028 ceph-mon[29847]: Updating smithi186:/etc/ceph/ceph.client.admin.keyring 2024-05-12T22:09:07.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:06 smithi028 ceph-mon[29847]: Updating smithi028:/var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-12T22:09:07.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:06 smithi028 ceph-mon[29847]: Updating smithi086:/var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/config/ceph.client.admin.keyring 2024-05-12T22:09:07.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:06 smithi028 ceph-mon[29847]: Updating smithi186:/var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/config/ceph.client.admin.keyring 2024-05-12T22:09:07.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:06 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/1171565211' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-12T22:09:07.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:06 smithi028 ceph-mon[29847]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:09:07.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:06 smithi028 ceph-mon[29847]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:09:07.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:06 smithi028 ceph-mon[29847]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:09:07.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:06 smithi028 ceph-mon[29847]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:09:07.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:06 smithi028 ceph-mon[29847]: Updating smithi028:/etc/ceph/ceph.client.admin.keyring 2024-05-12T22:09:07.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:06 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/3671947483' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-12T22:09:07.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:06 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/3225614891' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-12T22:09:07.145 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:06 smithi086 ceph-mon[34297]: Updating smithi086:/etc/ceph/ceph.client.admin.keyring 2024-05-12T22:09:07.145 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:06 smithi086 ceph-mon[34297]: from='client.25355 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-12T22:09:07.145 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:06 smithi086 ceph-mon[34297]: Updating smithi186:/etc/ceph/ceph.client.admin.keyring 2024-05-12T22:09:07.145 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:06 smithi086 ceph-mon[34297]: Updating smithi028:/var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-12T22:09:07.145 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:06 smithi086 ceph-mon[34297]: Updating smithi086:/var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/config/ceph.client.admin.keyring 2024-05-12T22:09:07.145 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:06 smithi086 ceph-mon[34297]: Updating smithi186:/var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/config/ceph.client.admin.keyring 2024-05-12T22:09:07.145 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:06 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/1171565211' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-12T22:09:07.145 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:06 smithi086 ceph-mon[34297]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:09:07.145 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:06 smithi086 ceph-mon[34297]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:09:07.145 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:06 smithi086 ceph-mon[34297]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:09:07.145 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:06 smithi086 ceph-mon[34297]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:09:07.145 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:06 smithi086 ceph-mon[34297]: Updating smithi028:/etc/ceph/ceph.client.admin.keyring 2024-05-12T22:09:07.145 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:06 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/3671947483' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-12T22:09:07.146 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:06 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/3225614891' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-12T22:09:07.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:06 smithi186 ceph-mon[34785]: Updating smithi086:/etc/ceph/ceph.client.admin.keyring 2024-05-12T22:09:07.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:06 smithi186 ceph-mon[34785]: from='client.25355 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-12T22:09:07.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:06 smithi186 ceph-mon[34785]: Updating smithi186:/etc/ceph/ceph.client.admin.keyring 2024-05-12T22:09:07.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:06 smithi186 ceph-mon[34785]: Updating smithi028:/var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-12T22:09:07.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:06 smithi186 ceph-mon[34785]: Updating smithi086:/var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/config/ceph.client.admin.keyring 2024-05-12T22:09:07.180 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:06 smithi186 ceph-mon[34785]: Updating smithi186:/var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/config/ceph.client.admin.keyring 2024-05-12T22:09:07.180 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:06 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/1171565211' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-12T22:09:07.180 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:06 smithi186 ceph-mon[34785]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:09:07.180 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:06 smithi186 ceph-mon[34785]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:09:07.180 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:06 smithi186 ceph-mon[34785]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:09:07.180 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:06 smithi186 ceph-mon[34785]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:09:07.180 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:06 smithi186 ceph-mon[34785]: Updating smithi028:/etc/ceph/ceph.client.admin.keyring 2024-05-12T22:09:07.180 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:06 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/3671947483' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-12T22:09:07.180 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:06 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/3225614891' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-12T22:09:07.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:07.182+0000 7f55a4a00640 1 -- 172.21.15.28:0/87697977 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f55a0109430 con 0x7f55a0100fb0 2024-05-12T22:09:07.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:07.182+0000 7f5596a00640 1 -- 172.21.15.28:0/87697977 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) ==== 100+0+11 (secure 0 0 0) 0x7f558c09bf10 con 0x7f55a0100fb0 2024-05-12T22:09:07.183 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-05-12T22:09:07.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:07.185+0000 7f55a4a00640 1 -- 172.21.15.28:0/87697977 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f55840a2ea0 msgr2=0x7f55840a5360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:07.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:07.185+0000 7f55a4a00640 1 --2- 172.21.15.28:0/87697977 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f55840a2ea0 0x7f55840a5360 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f55900095f0 tx=0x7f5590005f90 comp rx=0 tx=0).stop 2024-05-12T22:09:07.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:07.185+0000 7f55a4a00640 1 -- 172.21.15.28:0/87697977 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55a0100fb0 msgr2=0x7f55a019a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:07.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:07.185+0000 7f55a4a00640 1 --2- 172.21.15.28:0/87697977 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55a0100fb0 0x7f55a019a830 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f558c00e9f0 tx=0x7f558c00eeb0 comp rx=0 tx=0).stop 2024-05-12T22:09:07.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:07.185+0000 7f55a4a00640 1 -- 172.21.15.28:0/87697977 shutdown_connections 2024-05-12T22:09:07.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:07.185+0000 7f55a4a00640 1 --2- 172.21.15.28:0/87697977 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f55840a2ea0 0x7f55840a5360 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:07.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:07.185+0000 7f55a4a00640 1 --2- 172.21.15.28:0/87697977 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f55a010e240 0x7f55a019ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:07.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:07.185+0000 7f55a4a00640 1 --2- 172.21.15.28:0/87697977 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55a0100fb0 0x7f55a019a830 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:07.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:07.185+0000 7f55a4a00640 1 --2- 172.21.15.28:0/87697977 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f55a0100690 0x7f55a019a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:07.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:07.185+0000 7f55a4a00640 1 -- 172.21.15.28:0/87697977 >> 172.21.15.28:0/87697977 conn(0x7f55a0077890 msgr2=0x7f55a010f650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:09:07.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:07.185+0000 7f55a4a00640 1 -- 172.21.15.28:0/87697977 shutdown_connections 2024-05-12T22:09:07.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:07.185+0000 7f55a4a00640 1 -- 172.21.15.28:0/87697977 wait complete. 2024-05-12T22:09:07.204 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-12T22:09:07.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:07.396+0000 7ff9ee600640 1 -- 172.21.15.28:0/958269981 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff9e80693b0 msgr2=0x7ff9e810ce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:07.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:07.396+0000 7ff9ee600640 1 --2- 172.21.15.28:0/958269981 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff9e80693b0 0x7ff9e810ce80 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7ff9d8009ca0 tx=0x7ff9d802a8c0 comp rx=0 tx=0).stop 2024-05-12T22:09:07.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:07.396+0000 7ff9ee600640 1 -- 172.21.15.28:0/958269981 shutdown_connections 2024-05-12T22:09:07.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:07.396+0000 7ff9ee600640 1 --2- 172.21.15.28:0/958269981 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff9e8113120 0x7ff9e8115510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:07.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:07.396+0000 7ff9ee600640 1 --2- 172.21.15.28:0/958269981 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff9e81093c0 0x7ff9e810d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:07.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:07.396+0000 7ff9ee600640 1 --2- 172.21.15.28:0/958269981 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff9e80693b0 0x7ff9e810ce80 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:07.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:07.396+0000 7ff9ee600640 1 -- 172.21.15.28:0/958269981 >> 172.21.15.28:0/958269981 conn(0x7ff9e806cdb0 msgr2=0x7ff9e806d1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:09:07.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:07.396+0000 7ff9ee600640 1 -- 172.21.15.28:0/958269981 shutdown_connections 2024-05-12T22:09:07.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:07.396+0000 7ff9ee600640 1 -- 172.21.15.28:0/958269981 wait complete. 2024-05-12T22:09:07.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:07.396+0000 7ff9ee600640 1 Processor -- start 2024-05-12T22:09:07.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:07.396+0000 7ff9ee600640 1 -- start start 2024-05-12T22:09:07.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:07.397+0000 7ff9ee600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff9e80693b0 0x7ff9e81a8900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:07.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:07.397+0000 7ff9ee600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff9e81093c0 0x7ff9e81a8e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:07.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:07.397+0000 7ff9ee600640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff9e8113120 0x7ff9e8111e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:07.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:07.397+0000 7ff9ee600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff9e811a420 con 0x7ff9e81093c0 2024-05-12T22:09:07.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:07.397+0000 7ff9ee600640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7ff9e811a2a0 con 0x7ff9e80693b0 2024-05-12T22:09:07.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:07.397+0000 7ff9ee600640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff9e811a5a0 con 0x7ff9e8113120 2024-05-12T22:09:07.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:07.397+0000 7ff9dea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff9e81093c0 0x7ff9e81a8e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:07.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:07.397+0000 7ff9e7e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff9e8113120 0x7ff9e8111e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:07.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:07.397+0000 7ff9dea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff9e81093c0 0x7ff9e81a8e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49454/0 (socket says 172.21.15.28:49454) 2024-05-12T22:09:07.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:07.397+0000 7ff9dea00640 1 -- 172.21.15.28:0/354031703 learned_addr learned my addr 172.21.15.28:0/354031703 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:09:07.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:07.397+0000 7ff9e7400640 1 --2- 172.21.15.28:0/354031703 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff9e80693b0 0x7ff9e81a8900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:07.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:07.398+0000 7ff9dea00640 1 -- 172.21.15.28:0/354031703 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff9e8113120 msgr2=0x7ff9e8111e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:07.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:07.398+0000 7ff9dea00640 1 --2- 172.21.15.28:0/354031703 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff9e8113120 0x7ff9e8111e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:07.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:07.398+0000 7ff9dea00640 1 -- 172.21.15.28:0/354031703 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff9e80693b0 msgr2=0x7ff9e81a8900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:07.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:07.398+0000 7ff9dea00640 1 --2- 172.21.15.28:0/354031703 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff9e80693b0 0x7ff9e81a8900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:07.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:07.398+0000 7ff9dea00640 1 -- 172.21.15.28:0/354031703 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9e81126c0 con 0x7ff9e81093c0 2024-05-12T22:09:07.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:07.398+0000 7ff9e7e00640 1 --2- 172.21.15.28:0/354031703 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff9e8113120 0x7ff9e8111e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:09:07.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:07.398+0000 7ff9dea00640 1 --2- 172.21.15.28:0/354031703 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff9e81093c0 0x7ff9e81a8e40 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7ff9d000e9f0 tx=0x7ff9d000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:09:07.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:07.398+0000 7ff9e4c00640 1 -- 172.21.15.28:0/354031703 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9d000ccc0 con 0x7ff9e81093c0 2024-05-12T22:09:07.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:07.398+0000 7ff9e4c00640 1 -- 172.21.15.28:0/354031703 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff9d000ce60 con 0x7ff9e81093c0 2024-05-12T22:09:07.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:07.398+0000 7ff9e4c00640 1 -- 172.21.15.28:0/354031703 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9d0010690 con 0x7ff9e81093c0 2024-05-12T22:09:07.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:07.398+0000 7ff9ee600640 1 -- 172.21.15.28:0/354031703 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9e81129b0 con 0x7ff9e81093c0 2024-05-12T22:09:07.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:07.399+0000 7ff9ee600640 1 -- 172.21.15.28:0/354031703 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9e81af850 con 0x7ff9e81093c0 2024-05-12T22:09:07.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:07.400+0000 7ff9ee600640 1 -- 172.21.15.28:0/354031703 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff9e8109a50 con 0x7ff9e81093c0 2024-05-12T22:09:07.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:07.402+0000 7ff9e4c00640 1 -- 172.21.15.28:0/354031703 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138060+0+0 (secure 0 0 0) 0x7ff9d00046c0 con 0x7ff9e81093c0 2024-05-12T22:09:07.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:07.403+0000 7ff9e4c00640 1 --2- 172.21.15.28:0/354031703 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7ff9cc0a2e20 0x7ff9cc0a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:07.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:07.404+0000 7ff9e4c00640 1 -- 172.21.15.28:0/354031703 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7ff9d00cfb50 con 0x7ff9e81093c0 2024-05-12T22:09:07.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:07.404+0000 7ff9e7400640 1 --2- 172.21.15.28:0/354031703 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7ff9cc0a2e20 0x7ff9cc0a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:07.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:07.405+0000 7ff9e7400640 1 --2- 172.21.15.28:0/354031703 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7ff9cc0a2e20 0x7ff9cc0a52e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ff9d8006010 tx=0x7ff9d8009820 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:09:07.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:07.409+0000 7ff9e4c00640 1 -- 172.21.15.28:0/354031703 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7ff9d009c4f0 con 0x7ff9e81093c0 2024-05-12T22:09:07.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:07.666+0000 7ff9ee600640 1 -- 172.21.15.28:0/354031703 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff9e81afa20 con 0x7ff9e81093c0 2024-05-12T22:09:07.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:07.667+0000 7ff9e4c00640 1 -- 172.21.15.28:0/354031703 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11864 (secure 0 0 0) 0x7ff9d009bdf0 con 0x7ff9e81093c0 2024-05-12T22:09:07.667 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-12T22:09:07.667 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1844141745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1844141745},{"type":"v1","addr":"172.21.15.186:6833","nonce":1844141745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6833/2960763758","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":2960763758},{"type":"v1","addr":"172.21.15.86:6833","nonce":2960763758}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24470,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/4131134219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":4131134219},{"type":"v1","addr":"172.21.15.28:6835","nonce":4131134219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24481,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/1727869064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1727869064},{"type":"v1","addr":"172.21.15.186:6835","nonce":1727869064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24482,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2938260704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2938260704},{"type":"v1","addr":"172.21.15.28:6837","nonce":2938260704}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24494,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3473700910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3473700910},{"type":"v1","addr":"172.21.15.86:6835","nonce":3473700910}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-12T22:06:33.426986+0000","modified":"2024-05-12T22:08:58.179491+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24581,"mds_1":24550,"mds_2":24532},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24499":{"gid":24499,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6837/3600809235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":3600809235},{"type":"v1","addr":"172.21.15.186:6837","nonce":3600809235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"i","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6839/2399610807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":2399610807},{"type":"v1","addr":"172.21.15.186:6839","nonce":2399610807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24530":{"gid":24530,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.86:6837/3782448884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":3782448884},{"type":"v1","addr":"172.21.15.86:6837","nonce":3782448884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24532":{"gid":24532,"name":"g","rank":2,"incarnation":21,"state":"up:active","state_seq":6,"addr":"172.21.15.28:6839/123892158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":123892158},{"type":"v1","addr":"172.21.15.28:6839","nonce":123892158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24550":{"gid":24550,"name":"a","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.28:6841/761183870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":761183870},{"type":"v1","addr":"172.21.15.28:6841","nonce":761183870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24581":{"gid":24581,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6839/1251999130","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6838","nonce":1251999130},{"type":"v1","addr":"172.21.15.86:6839","nonce":1251999130}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24581,"qdb_cluster":[24550,24581,24532]},"id":1}]} 2024-05-12T22:09:07.668 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 24 2024-05-12T22:09:07.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:07.670+0000 7ff9ee600640 1 -- 172.21.15.28:0/354031703 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7ff9cc0a2e20 msgr2=0x7ff9cc0a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:07.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:07.670+0000 7ff9ee600640 1 --2- 172.21.15.28:0/354031703 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7ff9cc0a2e20 0x7ff9cc0a52e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ff9d8006010 tx=0x7ff9d8009820 comp rx=0 tx=0).stop 2024-05-12T22:09:07.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:07.670+0000 7ff9ee600640 1 -- 172.21.15.28:0/354031703 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff9e81093c0 msgr2=0x7ff9e81a8e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:07.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:07.670+0000 7ff9ee600640 1 --2- 172.21.15.28:0/354031703 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff9e81093c0 0x7ff9e81a8e40 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7ff9d000e9f0 tx=0x7ff9d000edf0 comp rx=0 tx=0).stop 2024-05-12T22:09:07.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:07.670+0000 7ff9ee600640 1 -- 172.21.15.28:0/354031703 shutdown_connections 2024-05-12T22:09:07.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:07.670+0000 7ff9ee600640 1 --2- 172.21.15.28:0/354031703 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7ff9cc0a2e20 0x7ff9cc0a52e0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:07.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:07.670+0000 7ff9ee600640 1 --2- 172.21.15.28:0/354031703 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff9e8113120 0x7ff9e8111e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:07.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:07.670+0000 7ff9ee600640 1 --2- 172.21.15.28:0/354031703 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff9e81093c0 0x7ff9e81a8e40 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:07.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:07.670+0000 7ff9ee600640 1 --2- 172.21.15.28:0/354031703 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff9e80693b0 0x7ff9e81a8900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:07.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:07.670+0000 7ff9ee600640 1 -- 172.21.15.28:0/354031703 >> 172.21.15.28:0/354031703 conn(0x7ff9e806cdb0 msgr2=0x7ff9e8113f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:09:07.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:07.671+0000 7ff9ee600640 1 -- 172.21.15.28:0/354031703 shutdown_connections 2024-05-12T22:09:07.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:07.671+0000 7ff9ee600640 1 -- 172.21.15.28:0/354031703 wait complete. 2024-05-12T22:09:07.691 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-12T22:09:07.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:07.886+0000 7f3ba7200640 1 -- 172.21.15.28:0/1441986689 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3ba8109840 msgr2=0x7f3ba8109c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:07.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:07.886+0000 7f3ba7200640 1 --2- 172.21.15.28:0/1441986689 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3ba8109840 0x7f3ba8109c20 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f3b98009900 tx=0x7f3b9802a410 comp rx=0 tx=0).stop 2024-05-12T22:09:07.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:07.886+0000 7f3ba7200640 1 -- 172.21.15.28:0/1441986689 shutdown_connections 2024-05-12T22:09:07.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:07.886+0000 7f3ba7200640 1 --2- 172.21.15.28:0/1441986689 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ba81041f0 0x7f3ba8110700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:07.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:07.886+0000 7f3ba7200640 1 --2- 172.21.15.28:0/1441986689 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3ba8103830 0x7f3ba8103cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:07.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:07.886+0000 7f3ba7200640 1 --2- 172.21.15.28:0/1441986689 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3ba8109840 0x7f3ba8109c20 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:07.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:07.886+0000 7f3ba7200640 1 -- 172.21.15.28:0/1441986689 >> 172.21.15.28:0/1441986689 conn(0x7f3ba80ff2c0 msgr2=0x7f3ba81016e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:09:07.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:07.886+0000 7f3ba7200640 1 -- 172.21.15.28:0/1441986689 shutdown_connections 2024-05-12T22:09:07.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:07.886+0000 7f3ba7200640 1 -- 172.21.15.28:0/1441986689 wait complete. 2024-05-12T22:09:07.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:07.887+0000 7f3ba7200640 1 Processor -- start 2024-05-12T22:09:07.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:07.887+0000 7f3ba7200640 1 -- start start 2024-05-12T22:09:07.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:07.888+0000 7f3ba7200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3ba8103830 0x7f3ba819a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:07.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:07.888+0000 7f3ba7200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ba81041f0 0x7f3ba819a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:07.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:07.888+0000 7f3ba7200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3ba8109840 0x7f3ba819e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:07.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:07.888+0000 7f3ba7200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3ba8113ca0 con 0x7f3ba81041f0 2024-05-12T22:09:07.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:07.888+0000 7f3ba7200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f3ba8113b20 con 0x7f3ba8109840 2024-05-12T22:09:07.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:07.888+0000 7f3ba7200640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f3ba8113e20 con 0x7f3ba8103830 2024-05-12T22:09:07.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:07.888+0000 7f3ba6800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3ba8109840 0x7f3ba819e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:07.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:07.888+0000 7f3ba5e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3ba8103830 0x7f3ba819a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:07.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:07.888+0000 7f3ba5400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ba81041f0 0x7f3ba819a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:07.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:07.888+0000 7f3ba5e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3ba8103830 0x7f3ba819a120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:53344/0 (socket says 172.21.15.28:53344) 2024-05-12T22:09:07.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:07.888+0000 7f3ba5e00640 1 -- 172.21.15.28:0/2644805504 learned_addr learned my addr 172.21.15.28:0/2644805504 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:09:07.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:07.888+0000 7f3ba6800640 1 -- 172.21.15.28:0/2644805504 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3ba8103830 msgr2=0x7f3ba819a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:07.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:07.888+0000 7f3ba6800640 1 --2- 172.21.15.28:0/2644805504 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3ba8103830 0x7f3ba819a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:07.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:07.888+0000 7f3ba6800640 1 -- 172.21.15.28:0/2644805504 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ba81041f0 msgr2=0x7f3ba819a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:07.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:07.888+0000 7f3ba6800640 1 --2- 172.21.15.28:0/2644805504 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ba81041f0 0x7f3ba819a660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:07.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:07.888+0000 7f3ba6800640 1 -- 172.21.15.28:0/2644805504 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ba819f110 con 0x7f3ba8109840 2024-05-12T22:09:07.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:07.889+0000 7f3ba6800640 1 --2- 172.21.15.28:0/2644805504 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3ba8109840 0x7f3ba819e9e0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f3ba000da10 tx=0x7f3ba000ded0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:09:07.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:07.889+0000 7f3b9ea00640 1 -- 172.21.15.28:0/2644805504 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ba000bd70 con 0x7f3ba8109840 2024-05-12T22:09:07.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:07.889+0000 7f3b9ea00640 1 -- 172.21.15.28:0/2644805504 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3ba0003e50 con 0x7f3ba8109840 2024-05-12T22:09:07.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:07.889+0000 7f3ba7200640 1 -- 172.21.15.28:0/2644805504 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ba819f400 con 0x7f3ba8109840 2024-05-12T22:09:07.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:07.889+0000 7f3b9ea00640 1 -- 172.21.15.28:0/2644805504 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ba00106a0 con 0x7f3ba8109840 2024-05-12T22:09:07.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:07.889+0000 7f3ba7200640 1 -- 172.21.15.28:0/2644805504 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ba81a6c90 con 0x7f3ba8109840 2024-05-12T22:09:07.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:07.890+0000 7f3ba7200640 1 -- 172.21.15.28:0/2644805504 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ba80689c0 con 0x7f3ba8109840 2024-05-12T22:09:07.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:07.892+0000 7f3b9ea00640 1 -- 172.21.15.28:0/2644805504 <== mon.2 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 25) ==== 138060+0+0 (secure 0 0 0) 0x7f3ba0003970 con 0x7f3ba8109840 2024-05-12T22:09:07.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:07.893+0000 7f3b9ea00640 1 --2- 172.21.15.28:0/2644805504 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f3b8c0a2e20 0x7f3b8c0a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:07.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:07.894+0000 7f3b9ea00640 1 -- 172.21.15.28:0/2644805504 <== mon.2 v2:172.21.15.86:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f3ba00cfa40 con 0x7f3ba8109840 2024-05-12T22:09:07.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:07.894+0000 7f3ba5e00640 1 --2- 172.21.15.28:0/2644805504 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f3b8c0a2e20 0x7f3b8c0a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:07.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:07.895+0000 7f3ba5e00640 1 --2- 172.21.15.28:0/2644805504 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f3b8c0a2e20 0x7f3b8c0a52e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3b98000c00 tx=0x7f3b980046c0 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:09:07.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:07.898+0000 7f3b9ea00640 1 -- 172.21.15.28:0/2644805504 <== mon.2 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f3ba009c3e0 con 0x7f3ba8109840 2024-05-12T22:09:07.932 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:07 smithi086 ceph-mon[34297]: pgmap v7: 97 pgs: 97 active+clean; 1.1 MiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 1.1 KiB/s wr, 23 op/s 2024-05-12T22:09:07.932 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:07 smithi086 ceph-mon[34297]: Updating smithi028:/var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/config/ceph.client.admin.keyring 2024-05-12T22:09:07.932 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:07 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/87697977' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-12T22:09:07.932 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:07 smithi086 ceph-mon[34297]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:09:07.932 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:07 smithi086 ceph-mon[34297]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:09:07.932 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:07 smithi086 ceph-mon[34297]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:09:07.932 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:07 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/354031703' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:09:07.932 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:07 smithi086 ceph-mon[34297]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:09:07.932 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:07 smithi086 ceph-mon[34297]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:09:07.933 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:07 smithi086 ceph-mon[34297]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:09:07.933 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:07 smithi086 ceph-mon[34297]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:09:08.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:07 smithi028 ceph-mon[29847]: pgmap v7: 97 pgs: 97 active+clean; 1.1 MiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 1.1 KiB/s wr, 23 op/s 2024-05-12T22:09:08.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:07 smithi028 ceph-mon[29847]: Updating smithi028:/var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/config/ceph.client.admin.keyring 2024-05-12T22:09:08.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:07 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/87697977' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-12T22:09:08.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:07 smithi028 ceph-mon[29847]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:09:08.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:07 smithi028 ceph-mon[29847]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:09:08.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:07 smithi028 ceph-mon[29847]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:09:08.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:07 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/354031703' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:09:08.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:07 smithi028 ceph-mon[29847]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:09:08.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:07 smithi028 ceph-mon[29847]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:09:08.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:07 smithi028 ceph-mon[29847]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:09:08.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:07 smithi028 ceph-mon[29847]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:09:08.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.129+0000 7f3ba7200640 1 -- 172.21.15.28:0/2644805504 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3ba802d1c0 con 0x7f3ba8109840 2024-05-12T22:09:08.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.131+0000 7f3b9ea00640 1 -- 172.21.15.28:0/2644805504 <== mon.2 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+22972 (secure 0 0 0) 0x7f3ba009bce0 con 0x7f3ba8109840 2024-05-12T22:09:08.131 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-12T22:09:08.131 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":89,"fsid":"6b92e7b4-10aa-11ef-bc98-c7b262605968","created":"2024-05-12T21:57:37.276782+0000","modified":"2024-05-12T22:08:58.638578+0000","last_up_change":"2024-05-12T22:05:37.929562+0000","last_in_change":"2024-05-12T22:05:18.750768+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-12T22:01:32.292826+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-12T22:06:31.540263+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-12T22:06:32.906118+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6992ce0a-844b-40e3-8ed1-181c8474dccf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":906003882},{"type":"v1","addr":"172.21.15.28:6803","nonce":906003882}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":906003882},{"type":"v1","addr":"172.21.15.28:6805","nonce":906003882}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":906003882},{"type":"v1","addr":"172.21.15.28:6809","nonce":906003882}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":906003882},{"type":"v1","addr":"172.21.15.28:6807","nonce":906003882}]},"public_addr":"172.21.15.28:6803/906003882","cluster_addr":"172.21.15.28:6805/906003882","heartbeat_back_addr":"172.21.15.28:6809/906003882","heartbeat_front_addr":"172.21.15.28:6807/906003882","state":["exists","up"]},{"osd":1,"uuid":"ced20c96-c481-4d48-90d1-6b43228959ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":3354141819},{"type":"v1","addr":"172.21.15.86:6801","nonce":3354141819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":3354141819},{"type":"v1","addr":"172.21.15.86:6803","nonce":3354141819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":3354141819},{"type":"v1","addr":"172.21.15.86:6807","nonce":3354141819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":3354141819},{"type":"v1","addr":"172.21.15.86:6805","nonce":3354141819}]},"public_addr":"172.21.15.86:6801/3354141819","cluster_addr":"172.21.15.86:6803/3354141819","heartbeat_back_addr":"172.21.15.86:6807/3354141819","heartbeat_front_addr":"172.21.15.86:6805/3354141819","state":["exists","up"]},{"osd":2,"uuid":"54a2f221-fa65-4795-9f5e-e660cc7c0098","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":906459365},{"type":"v1","addr":"172.21.15.186:6801","nonce":906459365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":906459365},{"type":"v1","addr":"172.21.15.186:6803","nonce":906459365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":906459365},{"type":"v1","addr":"172.21.15.186:6807","nonce":906459365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":906459365},{"type":"v1","addr":"172.21.15.186:6805","nonce":906459365}]},"public_addr":"172.21.15.186:6801/906459365","cluster_addr":"172.21.15.186:6803/906459365","heartbeat_back_addr":"172.21.15.186:6807/906459365","heartbeat_front_addr":"172.21.15.186:6805/906459365","state":["exists","up"]},{"osd":3,"uuid":"86836737-ac9a-487b-93f2-7fcd3c406960","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":4182884436},{"type":"v1","addr":"172.21.15.28:6811","nonce":4182884436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":4182884436},{"type":"v1","addr":"172.21.15.28:6813","nonce":4182884436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":4182884436},{"type":"v1","addr":"172.21.15.28:6817","nonce":4182884436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":4182884436},{"type":"v1","addr":"172.21.15.28:6815","nonce":4182884436}]},"public_addr":"172.21.15.28:6811/4182884436","cluster_addr":"172.21.15.28:6813/4182884436","heartbeat_back_addr":"172.21.15.28:6817/4182884436","heartbeat_front_addr":"172.21.15.28:6815/4182884436","state":["exists","up"]},{"osd":4,"uuid":"d162ebcd-83c0-4fd5-9925-28b2cecea935","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2596998708},{"type":"v1","addr":"172.21.15.86:6809","nonce":2596998708}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2596998708},{"type":"v1","addr":"172.21.15.86:6811","nonce":2596998708}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2596998708},{"type":"v1","addr":"172.21.15.86:6815","nonce":2596998708}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2596998708},{"type":"v1","addr":"172.21.15.86:6813","nonce":2596998708}]},"public_addr":"172.21.15.86:6809/2596998708","cluster_addr":"172.21.15.86:6811/2596998708","heartbeat_back_addr":"172.21.15.86:6815/2596998708","heartbeat_front_addr":"172.21.15.86:6813/2596998708","state":["exists","up"]},{"osd":5,"uuid":"c794336a-a105-425c-b6c7-a32b90c05bb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":1944205756},{"type":"v1","addr":"172.21.15.186:6809","nonce":1944205756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":1944205756},{"type":"v1","addr":"172.21.15.186:6811","nonce":1944205756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":1944205756},{"type":"v1","addr":"172.21.15.186:6815","nonce":1944205756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":1944205756},{"type":"v1","addr":"172.21.15.186:6813","nonce":1944205756}]},"public_addr":"172.21.15.186:6809/1944205756","cluster_addr":"172.21.15.186:6811/1944205756","heartbeat_back_addr":"172.21.15.186:6815/1944205756","heartbeat_front_addr":"172.21.15.186:6813/1944205756","state":["exists","up"]},{"osd":6,"uuid":"516932ac-af00-45b7-ad3c-d6dec7db442f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":1642308925},{"type":"v1","addr":"172.21.15.28:6819","nonce":1642308925}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":1642308925},{"type":"v1","addr":"172.21.15.28:6821","nonce":1642308925}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1642308925},{"type":"v1","addr":"172.21.15.28:6825","nonce":1642308925}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":1642308925},{"type":"v1","addr":"172.21.15.28:6823","nonce":1642308925}]},"public_addr":"172.21.15.28:6819/1642308925","cluster_addr":"172.21.15.28:6821/1642308925","heartbeat_back_addr":"172.21.15.28:6825/1642308925","heartbeat_front_addr":"172.21.15.28:6823/1642308925","state":["exists","up"]},{"osd":7,"uuid":"ca7a059f-0414-438e-b2d1-a10f348036be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":726062711},{"type":"v1","addr":"172.21.15.86:6817","nonce":726062711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":726062711},{"type":"v1","addr":"172.21.15.86:6819","nonce":726062711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":726062711},{"type":"v1","addr":"172.21.15.86:6823","nonce":726062711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":726062711},{"type":"v1","addr":"172.21.15.86:6821","nonce":726062711}]},"public_addr":"172.21.15.86:6817/726062711","cluster_addr":"172.21.15.86:6819/726062711","heartbeat_back_addr":"172.21.15.86:6823/726062711","heartbeat_front_addr":"172.21.15.86:6821/726062711","state":["exists","up"]},{"osd":8,"uuid":"92e3f41f-aebd-422e-a99e-555f7ba4aa53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":3007687277},{"type":"v1","addr":"172.21.15.186:6817","nonce":3007687277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":3007687277},{"type":"v1","addr":"172.21.15.186:6819","nonce":3007687277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":3007687277},{"type":"v1","addr":"172.21.15.186:6823","nonce":3007687277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":3007687277},{"type":"v1","addr":"172.21.15.186:6821","nonce":3007687277}]},"public_addr":"172.21.15.186:6817/3007687277","cluster_addr":"172.21.15.186:6819/3007687277","heartbeat_back_addr":"172.21.15.186:6823/3007687277","heartbeat_front_addr":"172.21.15.186:6821/3007687277","state":["exists","up"]},{"osd":9,"uuid":"38ff7d9d-ee85-404c-8c59-adc327e1ccfa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":136220472},{"type":"v1","addr":"172.21.15.28:6827","nonce":136220472}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":136220472},{"type":"v1","addr":"172.21.15.28:6829","nonce":136220472}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":136220472},{"type":"v1","addr":"172.21.15.28:6833","nonce":136220472}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":136220472},{"type":"v1","addr":"172.21.15.28:6831","nonce":136220472}]},"public_addr":"172.21.15.28:6827/136220472","cluster_addr":"172.21.15.28:6829/136220472","heartbeat_back_addr":"172.21.15.28:6833/136220472","heartbeat_front_addr":"172.21.15.28:6831/136220472","state":["exists","up"]},{"osd":10,"uuid":"5e9ff657-afa8-4bab-8491-f209038392b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1189968310},{"type":"v1","addr":"172.21.15.86:6825","nonce":1189968310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":1189968310},{"type":"v1","addr":"172.21.15.86:6827","nonce":1189968310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":1189968310},{"type":"v1","addr":"172.21.15.86:6831","nonce":1189968310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":1189968310},{"type":"v1","addr":"172.21.15.86:6829","nonce":1189968310}]},"public_addr":"172.21.15.86:6825/1189968310","cluster_addr":"172.21.15.86:6827/1189968310","heartbeat_back_addr":"172.21.15.86:6831/1189968310","heartbeat_front_addr":"172.21.15.86:6829/1189968310","state":["exists","up"]},{"osd":11,"uuid":"f112e3c6-65b6-4253-a966-62001db7d069","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3801668110},{"type":"v1","addr":"172.21.15.186:6825","nonce":3801668110}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3801668110},{"type":"v1","addr":"172.21.15.186:6827","nonce":3801668110}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3801668110},{"type":"v1","addr":"172.21.15.186:6831","nonce":3801668110}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3801668110},{"type":"v1","addr":"172.21.15.186:6829","nonce":3801668110}]},"public_addr":"172.21.15.186:6825/3801668110","cluster_addr":"172.21.15.186:6827/3801668110","heartbeat_back_addr":"172.21.15.186:6831/3801668110","heartbeat_front_addr":"172.21.15.186:6829/3801668110","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:00:40.483623+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:01:05.284385+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:01:28.645716+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:02:01.295092+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:02:26.486601+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:02:50.807478+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:03:24.756169+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:03:49.975173+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:04:13.772233+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:04:46.758555+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:05:12.331685+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:05:36.462548+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":2,"to":1}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.25","mappings":[{"from":5,"to":9}]},{"pgid":"3.27","mappings":[{"from":4,"to":8}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/2580087728":"2024-05-13T22:08:58.638536+0000","172.21.15.28:6801/3123328651":"2024-05-13T22:08:58.638536+0000","172.21.15.28:0/820158394":"2024-05-13T22:08:58.638536+0000","172.21.15.28:0/1978033727":"2024-05-13T22:08:58.638536+0000","172.21.15.28:0/1177009533":"2024-05-13T21:58:45.550459+0000","172.21.15.28:6800/3760499540":"2024-05-13T21:58:45.550459+0000","172.21.15.28:6801/3783742728":"2024-05-13T21:58:10.421564+0000","172.21.15.28:6800/3783742728":"2024-05-13T21:58:10.421564+0000","172.21.15.28:0/2229156260":"2024-05-13T21:58:45.550459+0000","172.21.15.28:0/1202140963":"2024-05-13T21:58:10.421564+0000","172.21.15.28:6800/3123328651":"2024-05-13T22:08:58.638536+0000","172.21.15.28:0/2924223784":"2024-05-13T21:58:45.550459+0000","172.21.15.28:6801/3760499540":"2024-05-13T21:58:45.550459+0000","172.21.15.28:0/1240027206":"2024-05-13T21:58:10.421564+0000","172.21.15.28:0/1581755795":"2024-05-13T22:08:58.638536+0000","172.21.15.28:0/171417728":"2024-05-13T21:58:10.421564+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-12T22:09:08.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.134+0000 7f3ba7200640 1 -- 172.21.15.28:0/2644805504 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f3b8c0a2e20 msgr2=0x7f3b8c0a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:08.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.134+0000 7f3ba7200640 1 --2- 172.21.15.28:0/2644805504 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f3b8c0a2e20 0x7f3b8c0a52e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3b98000c00 tx=0x7f3b980046c0 comp rx=0 tx=0).stop 2024-05-12T22:09:08.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.134+0000 7f3ba7200640 1 -- 172.21.15.28:0/2644805504 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3ba8109840 msgr2=0x7f3ba819e9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:08.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.134+0000 7f3ba7200640 1 --2- 172.21.15.28:0/2644805504 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3ba8109840 0x7f3ba819e9e0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f3ba000da10 tx=0x7f3ba000ded0 comp rx=0 tx=0).stop 2024-05-12T22:09:08.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.134+0000 7f3ba7200640 1 -- 172.21.15.28:0/2644805504 shutdown_connections 2024-05-12T22:09:08.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.134+0000 7f3ba7200640 1 --2- 172.21.15.28:0/2644805504 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f3b8c0a2e20 0x7f3b8c0a52e0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:08.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.134+0000 7f3ba7200640 1 --2- 172.21.15.28:0/2644805504 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3ba8109840 0x7f3ba819e9e0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:08.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.134+0000 7f3ba7200640 1 --2- 172.21.15.28:0/2644805504 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ba81041f0 0x7f3ba819a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:08.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.134+0000 7f3ba7200640 1 --2- 172.21.15.28:0/2644805504 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3ba8103830 0x7f3ba819a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:08.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.134+0000 7f3ba7200640 1 -- 172.21.15.28:0/2644805504 >> 172.21.15.28:0/2644805504 conn(0x7f3ba80ff2c0 msgr2=0x7f3ba80ff8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:09:08.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.134+0000 7f3ba7200640 1 -- 172.21.15.28:0/2644805504 shutdown_connections 2024-05-12T22:09:08.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.134+0000 7f3ba7200640 1 -- 172.21.15.28:0/2644805504 wait complete. 2024-05-12T22:09:08.153 DEBUG:tasks.fwd_scrub:Yielding 2024-05-12T22:09:08.153 INFO:teuthology.run_tasks:Running task exec... 2024-05-12T22:09:08.164 INFO:teuthology.task.exec:Executing custom commands... 2024-05-12T22:09:08.164 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi028.front.sepia.ceph.com 2024-05-12T22:09:08.164 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m' 2024-05-12T22:09:08.165 INFO:tasks.fwd_scrub.fs.[cephfs]:start scrubbing fs: cephfs 2024-05-12T22:09:08.165 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-12T22:09:08.165 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-12T22:09:08.165 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-05-12T22:09:08.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:07 smithi186 ceph-mon[34785]: pgmap v7: 97 pgs: 97 active+clean; 1.1 MiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 1.1 KiB/s wr, 23 op/s 2024-05-12T22:09:08.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:07 smithi186 ceph-mon[34785]: Updating smithi028:/var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/config/ceph.client.admin.keyring 2024-05-12T22:09:08.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:07 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/87697977' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-12T22:09:08.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:07 smithi186 ceph-mon[34785]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:09:08.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:07 smithi186 ceph-mon[34785]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:09:08.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:07 smithi186 ceph-mon[34785]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:09:08.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:07 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/354031703' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:09:08.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:07 smithi186 ceph-mon[34785]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:09:08.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:07 smithi186 ceph-mon[34785]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:09:08.180 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:07 smithi186 ceph-mon[34785]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:09:08.180 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:07 smithi186 ceph-mon[34785]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:09:08.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.357+0000 7fcaa6400640 1 -- 172.21.15.28:0/507058796 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcaa0074580 msgr2=0x7fcaa010d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:08.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.357+0000 7fcaa6400640 1 --2- 172.21.15.28:0/507058796 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcaa0074580 0x7fcaa010d930 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fca8c009900 tx=0x7fca8c02a480 comp rx=0 tx=0).stop 2024-05-12T22:09:08.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.358+0000 7fcaa6400640 1 -- 172.21.15.28:0/507058796 shutdown_connections 2024-05-12T22:09:08.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.358+0000 7fcaa6400640 1 --2- 172.21.15.28:0/507058796 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcaa010de70 0x7fcaa01102c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:08.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.358+0000 7fcaa6400640 1 --2- 172.21.15.28:0/507058796 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcaa0074580 0x7fcaa010d930 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:08.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.358+0000 7fcaa6400640 1 --2- 172.21.15.28:0/507058796 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcaa0073c60 0x7fcaa0074040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:08.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.358+0000 7fcaa6400640 1 -- 172.21.15.28:0/507058796 >> 172.21.15.28:0/507058796 conn(0x7fcaa00fd180 msgr2=0x7fcaa00ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:09:08.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.358+0000 7fcaa6400640 1 -- 172.21.15.28:0/507058796 shutdown_connections 2024-05-12T22:09:08.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.358+0000 7fcaa6400640 1 -- 172.21.15.28:0/507058796 wait complete. 2024-05-12T22:09:08.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.359+0000 7fcaa6400640 1 Processor -- start 2024-05-12T22:09:08.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.359+0000 7fcaa6400640 1 -- start start 2024-05-12T22:09:08.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.359+0000 7fcaa6400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcaa0073c60 0x7fcaa019a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:08.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.359+0000 7fcaa6400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcaa0074580 0x7fcaa019a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:08.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.359+0000 7fcaa6400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcaa010de70 0x7fcaa019e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:08.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.359+0000 7fcaa6400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcaa0113c10 con 0x7fcaa0074580 2024-05-12T22:09:08.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.359+0000 7fcaa6400640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fcaa0113a90 con 0x7fcaa0073c60 2024-05-12T22:09:08.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.359+0000 7fcaa6400640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fcaa0113d90 con 0x7fcaa010de70 2024-05-12T22:09:08.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.359+0000 7fcaa5000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcaa0073c60 0x7fcaa019a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:08.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.360+0000 7fcaa5000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcaa0073c60 0x7fcaa019a130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:51772/0 (socket says 172.21.15.28:51772) 2024-05-12T22:09:08.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.360+0000 7fcaa5000640 1 -- 172.21.15.28:0/3032665643 learned_addr learned my addr 172.21.15.28:0/3032665643 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:09:08.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.360+0000 7fca9fe00640 1 --2- 172.21.15.28:0/3032665643 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcaa0074580 0x7fcaa019a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:08.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.360+0000 7f74b5800640 1 -- 172.21.15.28:0/780187326 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f74b010e0f0 msgr2=0x7f74b01104e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:08.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.360+0000 7f74b5800640 1 --2- 172.21.15.28:0/780187326 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f74b010e0f0 0x7f74b01104e0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f74a8009ca0 tx=0x7f74a802a950 comp rx=0 tx=0).stop 2024-05-12T22:09:08.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.360+0000 7fcaa5a00640 1 --2- 172.21.15.28:0/3032665643 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcaa010de70 0x7fcaa019e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:08.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.360+0000 7f74b5800640 1 -- 172.21.15.28:0/780187326 shutdown_connections 2024-05-12T22:09:08.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.360+0000 7f74b5800640 1 --2- 172.21.15.28:0/780187326 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f74b010e0f0 0x7f74b01104e0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:08.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.360+0000 7f74b5800640 1 --2- 172.21.15.28:0/780187326 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f74b006a370 0x7f74b006a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:08.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.360+0000 7f74b5800640 1 --2- 172.21.15.28:0/780187326 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f74b01027e0 0x7f74b0069e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:08.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.360+0000 7fcaa5000640 1 -- 172.21.15.28:0/3032665643 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcaa010de70 msgr2=0x7fcaa019e9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:08.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.360+0000 7fcaa5000640 1 --2- 172.21.15.28:0/3032665643 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcaa010de70 0x7fcaa019e9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:08.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.360+0000 7fcaa5000640 1 -- 172.21.15.28:0/3032665643 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcaa0074580 msgr2=0x7fcaa019a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:08.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.360+0000 7fcaa5000640 1 --2- 172.21.15.28:0/3032665643 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcaa0074580 0x7fcaa019a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:08.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.360+0000 7f74b5800640 1 -- 172.21.15.28:0/780187326 >> 172.21.15.28:0/780187326 conn(0x7f74b0076530 msgr2=0x7f74b0076940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:09:08.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.360+0000 7fcaa5000640 1 -- 172.21.15.28:0/3032665643 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcaa019f120 con 0x7fcaa0073c60 2024-05-12T22:09:08.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.360+0000 7fcaa5a00640 1 --2- 172.21.15.28:0/3032665643 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcaa010de70 0x7fcaa019e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-12T22:09:08.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.360+0000 7f74b5800640 1 -- 172.21.15.28:0/780187326 shutdown_connections 2024-05-12T22:09:08.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.360+0000 7f74b5800640 1 -- 172.21.15.28:0/780187326 wait complete. 2024-05-12T22:09:08.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.360+0000 7fcaa5000640 1 --2- 172.21.15.28:0/3032665643 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcaa0073c60 0x7fcaa019a130 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fca9400e9f0 tx=0x7fca9400edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:09:08.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.361+0000 7fca9d600640 1 -- 172.21.15.28:0/3032665643 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca9400cd80 con 0x7fcaa0073c60 2024-05-12T22:09:08.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.361+0000 7fca9d600640 1 -- 172.21.15.28:0/3032665643 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fca94005500 con 0x7fcaa0073c60 2024-05-12T22:09:08.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.361+0000 7fcaa6400640 1 -- 172.21.15.28:0/3032665643 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcaa019f410 con 0x7fcaa0073c60 2024-05-12T22:09:08.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.361+0000 7fca9d600640 1 -- 172.21.15.28:0/3032665643 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca94010690 con 0x7fcaa0073c60 2024-05-12T22:09:08.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.361+0000 7fcaa6400640 1 -- 172.21.15.28:0/3032665643 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcaa01a6ca0 con 0x7fcaa0073c60 2024-05-12T22:09:08.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.361+0000 7f74b5800640 1 Processor -- start 2024-05-12T22:09:08.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.362+0000 7f74b5800640 1 -- start start 2024-05-12T22:09:08.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.362+0000 7f74b5800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f74b006a370 0x7f74b019a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:08.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.362+0000 7f74b5800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f74b01027e0 0x7f74b019a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:08.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.362+0000 7f74b5800640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f74b010e0f0 0x7f74b019eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:08.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.362+0000 7f74b5800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f74b0113e00 con 0x7f74b006a370 2024-05-12T22:09:08.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.362+0000 7f74b5800640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f74b0113c80 con 0x7f74b01027e0 2024-05-12T22:09:08.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.362+0000 7f74b5800640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f74b0113f80 con 0x7f74b010e0f0 2024-05-12T22:09:08.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.362+0000 7fcaa6400640 1 -- 172.21.15.28:0/3032665643 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcaa0075310 con 0x7fcaa0073c60 2024-05-12T22:09:08.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.362+0000 7f74ae000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f74b01027e0 0x7f74b019a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:08.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.362+0000 7f74ae000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f74b01027e0 0x7f74b019a800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:51774/0 (socket says 172.21.15.28:51774) 2024-05-12T22:09:08.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.362+0000 7f74ae000640 1 -- 172.21.15.28:0/2221064016 learned_addr learned my addr 172.21.15.28:0/2221064016 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:09:08.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.362+0000 7f74aea00640 1 --2- 172.21.15.28:0/2221064016 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f74b006a370 0x7f74b019a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:08.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.363+0000 7f74af400640 1 --2- 172.21.15.28:0/2221064016 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f74b010e0f0 0x7f74b019eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:08.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.363+0000 7f74ae000640 1 -- 172.21.15.28:0/2221064016 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f74b010e0f0 msgr2=0x7f74b019eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:08.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.363+0000 7f74ae000640 1 --2- 172.21.15.28:0/2221064016 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f74b010e0f0 0x7f74b019eb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:08.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.363+0000 7f74ae000640 1 -- 172.21.15.28:0/2221064016 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f74b006a370 msgr2=0x7f74b019a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:08.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.363+0000 7f74ae000640 1 --2- 172.21.15.28:0/2221064016 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f74b006a370 0x7f74b019a2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:08.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.363+0000 7f74ae000640 1 -- 172.21.15.28:0/2221064016 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74b019f2b0 con 0x7f74b01027e0 2024-05-12T22:09:08.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.363+0000 7f74af400640 1 --2- 172.21.15.28:0/2221064016 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f74b010e0f0 0x7f74b019eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:09:08.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.363+0000 7f74ae000640 1 --2- 172.21.15.28:0/2221064016 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f74b01027e0 0x7f74b019a800 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f749c00e9c0 tx=0x7f749c00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:09:08.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.364+0000 7fca9d600640 1 -- 172.21.15.28:0/3032665643 <== mon.2 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 25) ==== 138060+0+0 (secure 0 0 0) 0x7fca940046c0 con 0x7fcaa0073c60 2024-05-12T22:09:08.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.366+0000 7fca9d600640 1 --2- 172.21.15.28:0/3032665643 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fca880a2e90 0x7fca880a5350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:08.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.366+0000 7fca9d600640 1 -- 172.21.15.28:0/3032665643 <== mon.2 v2:172.21.15.86:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7fca940cfed0 con 0x7fcaa0073c60 2024-05-12T22:09:08.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.366+0000 7fca9fe00640 1 --2- 172.21.15.28:0/3032665643 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fca880a2e90 0x7fca880a5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:08.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.367+0000 7fca9fe00640 1 --2- 172.21.15.28:0/3032665643 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fca880a2e90 0x7fca880a5350 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fca8c009230 tx=0x7fca8c0058d0 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:09:08.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.367+0000 7f74a7400640 1 -- 172.21.15.28:0/2221064016 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f749c00cd80 con 0x7f74b01027e0 2024-05-12T22:09:08.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.367+0000 7f74a7400640 1 -- 172.21.15.28:0/2221064016 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f749c0035f0 con 0x7f74b01027e0 2024-05-12T22:09:08.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.367+0000 7f74b5800640 1 -- 172.21.15.28:0/2221064016 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74b019f5a0 con 0x7f74b01027e0 2024-05-12T22:09:08.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.367+0000 7f74a7400640 1 -- 172.21.15.28:0/2221064016 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f749c010690 con 0x7f74b01027e0 2024-05-12T22:09:08.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.368+0000 7f74b5800640 1 -- 172.21.15.28:0/2221064016 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74b01a6e30 con 0x7f74b01027e0 2024-05-12T22:09:08.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.368+0000 7fca9d600640 1 -- 172.21.15.28:0/3032665643 <== mon.2 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fca9409c870 con 0x7fcaa0073c60 2024-05-12T22:09:08.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.371+0000 7f74b862f740 1 Processor -- start 2024-05-12T22:09:08.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.371+0000 7f74b862f740 1 -- start start 2024-05-12T22:09:08.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.371+0000 7f74b862f740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ada72d82c0 0x55ada72d86a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:08.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.371+0000 7f74b862f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ada72d8c70 0x55ada72e2d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:08.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.371+0000 7f74b862f740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ada72e32e0 0x55ada72e5730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:08.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.371+0000 7f74b862f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ada723c7c0 con 0x55ada72d8c70 2024-05-12T22:09:08.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.371+0000 7f74b862f740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x55ada71e4790 con 0x55ada72d82c0 2024-05-12T22:09:08.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.371+0000 7f74b862f740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55ada7238780 con 0x55ada72e32e0 2024-05-12T22:09:08.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.372+0000 7f74af400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ada72d82c0 0x55ada72d86a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:08.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.372+0000 7f74aea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ada72d8c70 0x55ada72e2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:08.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.372+0000 7f74af400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ada72d82c0 0x55ada72d86a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:51790/0 (socket says 172.21.15.28:51790) 2024-05-12T22:09:08.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.372+0000 7f74af400640 1 -- 172.21.15.28:0/3384343917 learned_addr learned my addr 172.21.15.28:0/3384343917 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:09:08.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.372+0000 7f74af400640 1 -- 172.21.15.28:0/3384343917 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ada72e32e0 msgr2=0x55ada72e5730 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-12T22:09:08.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.372+0000 7f74af400640 1 --2- 172.21.15.28:0/3384343917 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ada72e32e0 0x55ada72e5730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:08.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.372+0000 7f74af400640 1 -- 172.21.15.28:0/3384343917 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ada72d8c70 msgr2=0x55ada72e2d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:09:08.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.372+0000 7f74af400640 1 --2- 172.21.15.28:0/3384343917 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ada72d8c70 0x55ada72e2d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:08.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.372+0000 7f74af400640 1 -- 172.21.15.28:0/3384343917 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ada72046a0 con 0x55ada72d82c0 2024-05-12T22:09:08.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.372+0000 7f74af400640 1 --2- 172.21.15.28:0/3384343917 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ada72d82c0 0x55ada72d86a0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f74b0069c40 tx=0x7f74a8009a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7de3a2e0e028016a server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:09:08.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.373+0000 7f749be00640 1 -- 172.21.15.28:0/3384343917 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74a8034040 con 0x55ada72d82c0 2024-05-12T22:09:08.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.373+0000 7f749be00640 1 -- 172.21.15.28:0/3384343917 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f74a8003e50 con 0x55ada72d82c0 2024-05-12T22:09:08.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.373+0000 7f749be00640 1 -- 172.21.15.28:0/3384343917 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74a8004330 con 0x55ada72d82c0 2024-05-12T22:09:08.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.373+0000 7f74a7400640 1 -- 172.21.15.28:0/2221064016 <== mon.2 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 25) ==== 138060+0+0 (secure 0 0 0) 0x7f749c0037a0 con 0x7f74b01027e0 2024-05-12T22:09:08.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.374+0000 7f74b862f740 1 -- 172.21.15.28:0/3384343917 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ada72d82c0 msgr2=0x55ada72d86a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:08.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.374+0000 7f74b862f740 1 --2- 172.21.15.28:0/3384343917 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ada72d82c0 0x55ada72d86a0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f74b0069c40 tx=0x7f74a8009a20 comp rx=0 tx=0).stop 2024-05-12T22:09:08.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.374+0000 7f74b862f740 1 -- 172.21.15.28:0/3384343917 shutdown_connections 2024-05-12T22:09:08.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.374+0000 7f74b862f740 1 --2- 172.21.15.28:0/3384343917 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ada72e32e0 0x55ada72e5730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:08.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.374+0000 7f74b862f740 1 --2- 172.21.15.28:0/3384343917 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ada72d8c70 0x55ada72e2d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:08.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.374+0000 7f74b862f740 1 --2- 172.21.15.28:0/3384343917 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ada72d82c0 0x55ada72d86a0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:08.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.374+0000 7f74b862f740 1 -- 172.21.15.28:0/3384343917 >> 172.21.15.28:0/3384343917 conn(0x55ada72d4370 msgr2=0x55ada72d4750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:09:08.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.374+0000 7f74b862f740 1 -- 172.21.15.28:0/3384343917 shutdown_connections 2024-05-12T22:09:08.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.374+0000 7f74b862f740 1 -- 172.21.15.28:0/3384343917 wait complete. 2024-05-12T22:09:08.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.374+0000 7f74b862f740 1 -- start start 2024-05-12T22:09:08.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.375+0000 7f74b862f740 1 Processor -- start 2024-05-12T22:09:08.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.375+0000 7f74a7400640 1 --2- 172.21.15.28:0/2221064016 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f74940a2db0 0x7f74940a5270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:08.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.375+0000 7f74a7400640 1 -- 172.21.15.28:0/2221064016 <== mon.2 v2:172.21.15.86:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f749c0cfaf0 con 0x7f74b01027e0 2024-05-12T22:09:08.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.375+0000 7f74af400640 1 --2- 172.21.15.28:0/2221064016 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f74940a2db0 0x7f74940a5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:08.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.375+0000 7f74b862f740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ada72d82c0 0x55ada72e0170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:08.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.375+0000 7f74b862f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ada72d8c70 0x55ada72e06b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:08.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.375+0000 7f74b862f740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ada72e32e0 0x55ada72e0bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:08.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.375+0000 7f74b862f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ada72e6290 con 0x55ada72d8c70 2024-05-12T22:09:08.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.375+0000 7f74b862f740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x55ada71f7a90 con 0x55ada72d82c0 2024-05-12T22:09:08.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.375+0000 7f74b862f740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55ada72e6410 con 0x55ada72e32e0 2024-05-12T22:09:08.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.376+0000 7f74b862f740 10 client.0 _ll_register_callbacks cb 0x55ada7232af0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:09:08.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.376+0000 7f74aea00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ada72d82c0 0x55ada72e0170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:08.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.376+0000 7f74aea00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ada72d82c0 0x55ada72e0170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:51800/0 (socket says 172.21.15.28:51800) 2024-05-12T22:09:08.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.376+0000 7f74aea00640 1 -- 172.21.15.28:0/3343137752 learned_addr learned my addr 172.21.15.28:0/3343137752 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:09:08.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.376+0000 7f74af400640 1 --2- 172.21.15.28:0/3343137752 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ada72e32e0 0x55ada72e0bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:08.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.376+0000 7f74ae000640 1 --2- 172.21.15.28:0/3343137752 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ada72d8c70 0x55ada72e06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:08.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.376+0000 7f74aea00640 1 -- 172.21.15.28:0/3343137752 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ada72e32e0 msgr2=0x55ada72e0bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:09:08.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.376+0000 7f74aea00640 1 --2- 172.21.15.28:0/3343137752 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ada72e32e0 0x55ada72e0bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:08.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.376+0000 7f74aea00640 1 -- 172.21.15.28:0/3343137752 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ada72d8c70 msgr2=0x55ada72e06b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:09:08.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.376+0000 7f74aea00640 1 --2- 172.21.15.28:0/3343137752 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ada72d8c70 0x55ada72e06b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:08.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.376+0000 7f74aea00640 1 -- 172.21.15.28:0/3343137752 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ada70187e0 con 0x55ada72d82c0 2024-05-12T22:09:08.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.376+0000 7f74aea00640 1 --2- 172.21.15.28:0/3343137752 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ada72d82c0 0x55ada72e0170 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f74a0012730 tx=0x7f74a0012bf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2a4ef0f72020bb18 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:09:08.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.376+0000 7f7498c00640 10 client.0 ms_handle_connect on v2:172.21.15.86:3300/0 2024-05-12T22:09:08.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.377+0000 7f74af400640 1 --2- 172.21.15.28:0/2221064016 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f74940a2db0 0x7f74940a5270 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f74a8006010 tx=0x7f74a802fe70 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:09:08.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.377+0000 7f7498c00640 1 -- 172.21.15.28:0/3343137752 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74a0004030 con 0x55ada72d82c0 2024-05-12T22:09:08.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.377+0000 7f7498c00640 1 -- 172.21.15.28:0/3343137752 <== mon.2 v2:172.21.15.86:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f74a00041d0 con 0x55ada72d82c0 2024-05-12T22:09:08.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.377+0000 7f74b862f740 1 -- 172.21.15.28:0/3343137752 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f74a8003e50 con 0x55ada72d82c0 2024-05-12T22:09:08.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.377+0000 7f7498c00640 1 -- 172.21.15.28:0/3343137752 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f74a00043d0 con 0x55ada72d82c0 2024-05-12T22:09:08.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.377+0000 7f74b862f740 10 client.25409 fetch_fsmap learned FSMap version 24 2024-05-12T22:09:08.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.377+0000 7f74b862f740 1 -- 172.21.15.28:0/3343137752 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55ada7078f70 con 0x55ada72d82c0 2024-05-12T22:09:08.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.377+0000 7f7498c00640 1 -- 172.21.15.28:0/3343137752 <== mon.2 v2:172.21.15.86:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f74a8003e50 con 0x55ada72d82c0 2024-05-12T22:09:08.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.377+0000 7f7498c00640 1 -- 172.21.15.28:0/3343137752 <== mon.2 v2:172.21.15.86:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f74a0003b80 con 0x55ada72d82c0 2024-05-12T22:09:08.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.378+0000 7f74b862f740 10 client.25409 fetch_fsmap finished waiting for FSMap version 24 2024-05-12T22:09:08.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.378+0000 7f74b862f740 10 client.25409 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:09:08.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.378+0000 7f74b862f740 20 client.25409 populate_metadata read hostname 'smithi028' 2024-05-12T22:09:08.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.378+0000 7f74b862f740 1 --2- 172.21.15.28:0/3343137752 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55ada72f3400 0x55ada72f5850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:08.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.378+0000 7f74b862f740 4 client.25409 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-12T22:09:08.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.378+0000 7f74b862f740 1 -- 172.21.15.28:0/3343137752 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ada72503a0 con 0x55ada72f3400 2024-05-12T22:09:08.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.378+0000 7f74ae000640 1 --2- 172.21.15.28:0/3343137752 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55ada72f3400 0x55ada72f5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:08.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.378+0000 7f74ae000640 1 --2- 172.21.15.28:0/3343137752 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55ada72f3400 0x55ada72f5850 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8c343d5c42822bf3 server_cookie=6c22cc6d10ac95ac in_seq=0 out_seq=0 2024-05-12T22:09:08.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.378+0000 7f7498c00640 10 client.25409 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:09:08.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.379+0000 7f7498c00640 1 -- 172.21.15.28:0/3343137752 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ada72503a0 con 0x55ada72f3400 2024-05-12T22:09:08.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.379+0000 7f7498c00640 10 client.25409 handle_command_reply: tid=0 2024-05-12T22:09:08.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.379+0000 7f74b862f740 1 client.25409 shutdown 2024-05-12T22:09:08.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.379+0000 7f74b862f740 1 -- 172.21.15.28:0/3343137752 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ada72d82c0 msgr2=0x55ada72e0170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:08.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.379+0000 7f74b862f740 1 --2- 172.21.15.28:0/3343137752 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ada72d82c0 0x55ada72e0170 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f74a0012730 tx=0x7f74a0012bf0 comp rx=0 tx=0).stop 2024-05-12T22:09:08.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.379+0000 7f74b862f740 1 -- 172.21.15.28:0/3343137752 shutdown_connections 2024-05-12T22:09:08.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.379+0000 7f74b862f740 1 --2- 172.21.15.28:0/3343137752 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55ada72f3400 0x55ada72f5850 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:08.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.379+0000 7f74b862f740 1 --2- 172.21.15.28:0/3343137752 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ada72e32e0 0x55ada72e0bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:08.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.379+0000 7f7498c00640 0 client.25409 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:09:08.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.379+0000 7f74b862f740 1 --2- 172.21.15.28:0/3343137752 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ada72d8c70 0x55ada72e06b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:08.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.379+0000 7f74b862f740 1 --2- 172.21.15.28:0/3343137752 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ada72d82c0 0x55ada72e0170 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:08.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.379+0000 7f74b862f740 1 -- 172.21.15.28:0/3343137752 >> 172.21.15.28:0/3343137752 conn(0x55ada72d4370 msgr2=0x55ada72da3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:09:08.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.380+0000 7f74b862f740 1 -- 172.21.15.28:0/3343137752 shutdown_connections 2024-05-12T22:09:08.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.380+0000 7f74b862f740 1 -- 172.21.15.28:0/3343137752 wait complete. 2024-05-12T22:09:08.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.380+0000 7f74b862f740 20 client.25409 trim_cache size 0 max 16384 2024-05-12T22:09:08.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.399+0000 7f74b862f740 1 Processor -- start 2024-05-12T22:09:08.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.399+0000 7f74b862f740 1 -- start start 2024-05-12T22:09:08.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.400+0000 7f74b862f740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ada72d82c0 0x55ada72ef180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:08.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.400+0000 7f74b862f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ada72d8c70 0x55ada72df0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:08.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.400+0000 7f74b862f740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ada72e32e0 0x55ada72fb120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:08.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.400+0000 7f74b862f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f74b01b2050 con 0x55ada72d8c70 2024-05-12T22:09:08.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.400+0000 7f74b862f740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f74b01af920 con 0x55ada72d82c0 2024-05-12T22:09:08.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.400+0000 7f74b862f740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f74b01ad770 con 0x55ada72e32e0 2024-05-12T22:09:08.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.400+0000 7f74aea00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ada72d82c0 0x55ada72ef180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:08.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.400+0000 7f74aea00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ada72d82c0 0x55ada72ef180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:51810/0 (socket says 172.21.15.28:51810) 2024-05-12T22:09:08.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.400+0000 7f74aea00640 1 -- 172.21.15.28:0/3573121975 learned_addr learned my addr 172.21.15.28:0/3573121975 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:09:08.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.400+0000 7f74ae000640 1 --2- 172.21.15.28:0/3573121975 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ada72d8c70 0x55ada72df0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:08.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.400+0000 7f74af400640 1 --2- 172.21.15.28:0/3573121975 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ada72e32e0 0x55ada72fb120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:08.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.400+0000 7f74aea00640 1 -- 172.21.15.28:0/3573121975 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ada72e32e0 msgr2=0x55ada72fb120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:09:08.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.400+0000 7f74aea00640 1 --2- 172.21.15.28:0/3573121975 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ada72e32e0 0x55ada72fb120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:08.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.400+0000 7f74aea00640 1 -- 172.21.15.28:0/3573121975 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ada72d8c70 msgr2=0x55ada72df0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:09:08.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.400+0000 7f74aea00640 1 --2- 172.21.15.28:0/3573121975 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ada72d8c70 0x55ada72df0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:08.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.400+0000 7f74aea00640 1 -- 172.21.15.28:0/3573121975 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ada6f10170 con 0x55ada72d82c0 2024-05-12T22:09:08.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.401+0000 7f74aea00640 1 --2- 172.21.15.28:0/3573121975 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ada72d82c0 0x55ada72ef180 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f74a000f7b0 tx=0x7f74a000f7e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=15f2fa792e68ebca server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:09:08.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.401+0000 7f749b400640 1 -- 172.21.15.28:0/3573121975 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74a0029030 con 0x55ada72d82c0 2024-05-12T22:09:08.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.401+0000 7f749b400640 1 -- 172.21.15.28:0/3573121975 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f74a000fe00 con 0x55ada72d82c0 2024-05-12T22:09:08.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.401+0000 7f749b400640 1 -- 172.21.15.28:0/3573121975 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74a001cbc0 con 0x55ada72d82c0 2024-05-12T22:09:08.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.402+0000 7f74b862f740 1 -- 172.21.15.28:0/3573121975 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ada72d82c0 msgr2=0x55ada72ef180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:08.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.402+0000 7f74b862f740 1 --2- 172.21.15.28:0/3573121975 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ada72d82c0 0x55ada72ef180 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f74a000f7b0 tx=0x7f74a000f7e0 comp rx=0 tx=0).stop 2024-05-12T22:09:08.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.402+0000 7f74b862f740 1 -- 172.21.15.28:0/3573121975 shutdown_connections 2024-05-12T22:09:08.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.402+0000 7f74b862f740 1 --2- 172.21.15.28:0/3573121975 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ada72e32e0 0x55ada72fb120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:08.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.402+0000 7f74b862f740 1 --2- 172.21.15.28:0/3573121975 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ada72d8c70 0x55ada72df0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:08.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.402+0000 7f74b862f740 1 --2- 172.21.15.28:0/3573121975 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ada72d82c0 0x55ada72ef180 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:08.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.402+0000 7f74b862f740 1 -- 172.21.15.28:0/3573121975 >> 172.21.15.28:0/3573121975 conn(0x55ada72d4370 msgr2=0x55ada72d34f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:09:08.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.402+0000 7f74b862f740 1 -- 172.21.15.28:0/3573121975 shutdown_connections 2024-05-12T22:09:08.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.402+0000 7f74b862f740 1 -- 172.21.15.28:0/3573121975 wait complete. 2024-05-12T22:09:08.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.403+0000 7f74b862f740 1 -- start start 2024-05-12T22:09:08.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.403+0000 7f74b862f740 1 Processor -- start 2024-05-12T22:09:08.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.404+0000 7f74b862f740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ada72d82c0 0x55ada72e7350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:08.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.404+0000 7f74b862f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ada72d8c70 0x55ada72e7890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:08.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.404+0000 7f74b862f740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ada72e32e0 0x55ada72e7dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:08.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.404+0000 7f74b862f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f74b01adde0 con 0x55ada72d8c70 2024-05-12T22:09:08.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.404+0000 7f74b862f740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f74b01af250 con 0x55ada72e32e0 2024-05-12T22:09:08.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.404+0000 7f74b862f740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f74b01ae4b0 con 0x55ada72d82c0 2024-05-12T22:09:08.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.404+0000 7f74b862f740 10 client.0 _ll_register_callbacks cb 0x7f74a0009850 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:09:08.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.404+0000 7f74ae000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ada72d8c70 0x55ada72e7890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:08.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.404+0000 7f74af400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ada72e32e0 0x55ada72e7dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:08.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.404+0000 7f74ae000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ada72d8c70 0x55ada72e7890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49546/0 (socket says 172.21.15.28:49546) 2024-05-12T22:09:08.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.404+0000 7f74ae000640 1 -- 172.21.15.28:0/1922454201 learned_addr learned my addr 172.21.15.28:0/1922454201 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:09:08.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.404+0000 7f74aea00640 1 --2- 172.21.15.28:0/1922454201 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ada72d82c0 0x55ada72e7350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:08.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.405+0000 7f74af400640 1 -- 172.21.15.28:0/1922454201 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ada72d82c0 msgr2=0x55ada72e7350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:09:08.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.405+0000 7f74af400640 1 --2- 172.21.15.28:0/1922454201 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ada72d82c0 0x55ada72e7350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:08.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.405+0000 7f74af400640 1 -- 172.21.15.28:0/1922454201 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ada72d8c70 msgr2=0x55ada72e7890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:09:08.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.405+0000 7f74af400640 1 --2- 172.21.15.28:0/1922454201 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ada72d8c70 0x55ada72e7890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:08.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.405+0000 7f74af400640 1 -- 172.21.15.28:0/1922454201 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ada7358c30 con 0x55ada72e32e0 2024-05-12T22:09:08.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.405+0000 7f74af400640 1 --2- 172.21.15.28:0/1922454201 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ada72e32e0 0x55ada72e7dd0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f74a8005fa0 tx=0x7f74a8009a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9df2041a0010eb12 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:09:08.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.405+0000 7f7498c00640 10 client.0 ms_handle_connect on v2:172.21.15.86:3300/0 2024-05-12T22:09:08.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.405+0000 7f7498c00640 1 -- 172.21.15.28:0/1922454201 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74a805e060 con 0x55ada72e32e0 2024-05-12T22:09:08.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.405+0000 7f74b862f740 1 -- 172.21.15.28:0/1922454201 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f74a000fe00 con 0x55ada72e32e0 2024-05-12T22:09:08.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.405+0000 7f7498c00640 1 -- 172.21.15.28:0/1922454201 <== mon.2 v2:172.21.15.86:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f74a8004040 con 0x55ada72e32e0 2024-05-12T22:09:08.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.405+0000 7f7498c00640 1 -- 172.21.15.28:0/1922454201 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f74a8004240 con 0x55ada72e32e0 2024-05-12T22:09:08.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.405+0000 7f74b862f740 10 client.25415 fetch_fsmap learned FSMap version 24 2024-05-12T22:09:08.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.405+0000 7f74b862f740 1 -- 172.21.15.28:0/1922454201 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55ada7359f70 con 0x55ada72e32e0 2024-05-12T22:09:08.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.406+0000 7f7498c00640 1 -- 172.21.15.28:0/1922454201 <== mon.2 v2:172.21.15.86:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f74a000fe00 con 0x55ada72e32e0 2024-05-12T22:09:08.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.406+0000 7f7498c00640 1 -- 172.21.15.28:0/1922454201 <== mon.2 v2:172.21.15.86:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f74a8058030 con 0x55ada72e32e0 2024-05-12T22:09:08.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.406+0000 7f74b862f740 10 client.25415 fetch_fsmap finished waiting for FSMap version 24 2024-05-12T22:09:08.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.406+0000 7f74b862f740 10 client.25415 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:09:08.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.406+0000 7f74b862f740 20 client.25415 populate_metadata read hostname 'smithi028' 2024-05-12T22:09:08.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.406+0000 7f74b862f740 1 --2- 172.21.15.28:0/1922454201 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55ada72f3400 0x55ada735c150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:08.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.406+0000 7f74b862f740 4 client.25415 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-12T22:09:08.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.406+0000 7f74b862f740 1 -- 172.21.15.28:0/1922454201 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55ada7249ef0 con 0x55ada72f3400 2024-05-12T22:09:08.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.406+0000 7f74aea00640 1 --2- 172.21.15.28:0/1922454201 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55ada72f3400 0x55ada735c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:08.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.407+0000 7f74aea00640 1 --2- 172.21.15.28:0/1922454201 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55ada72f3400 0x55ada735c150 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6beb55bd9ee87838 server_cookie=9043f9997c621265 in_seq=0 out_seq=0 2024-05-12T22:09:08.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.407+0000 7f7498c00640 10 client.25415 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:09:08.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.408+0000 7f7498c00640 1 -- 172.21.15.28:0/1922454201 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55ada7249ef0 con 0x55ada72f3400 2024-05-12T22:09:08.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.408+0000 7f7498c00640 10 client.25415 handle_command_reply: tid=0 2024-05-12T22:09:08.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.408+0000 7f74b862f740 1 client.25415 shutdown 2024-05-12T22:09:08.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.408+0000 7f74b862f740 1 -- 172.21.15.28:0/1922454201 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ada72e32e0 msgr2=0x55ada72e7dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:08.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.408+0000 7f74b862f740 1 --2- 172.21.15.28:0/1922454201 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ada72e32e0 0x55ada72e7dd0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f74a8005fa0 tx=0x7f74a8009a20 comp rx=0 tx=0).stop 2024-05-12T22:09:08.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.408+0000 7f74b862f740 1 -- 172.21.15.28:0/1922454201 shutdown_connections 2024-05-12T22:09:08.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.408+0000 7f74b862f740 1 --2- 172.21.15.28:0/1922454201 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55ada72f3400 0x55ada735c150 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:08.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.408+0000 7f74b862f740 1 --2- 172.21.15.28:0/1922454201 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ada72e32e0 0x55ada72e7dd0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:08.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.408+0000 7f74b862f740 1 --2- 172.21.15.28:0/1922454201 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ada72d8c70 0x55ada72e7890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:08.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.408+0000 7f74b862f740 1 --2- 172.21.15.28:0/1922454201 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ada72d82c0 0x55ada72e7350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:08.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.408+0000 7f74b862f740 1 -- 172.21.15.28:0/1922454201 >> 172.21.15.28:0/1922454201 conn(0x55ada72d4370 msgr2=0x55ada72d99d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:09:08.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.408+0000 7f7498c00640 0 client.25415 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:09:08.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.408+0000 7f74b862f740 1 -- 172.21.15.28:0/1922454201 shutdown_connections 2024-05-12T22:09:08.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.408+0000 7f74b862f740 1 -- 172.21.15.28:0/1922454201 wait complete. 2024-05-12T22:09:08.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.408+0000 7f74b862f740 20 client.25415 trim_cache size 0 max 16384 2024-05-12T22:09:08.411 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-05-12T22:09:08.411 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-05-12T22:09:08.411 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "86318bd8-7621-4ea4-be1f-32b76b44b012", 2024-05-12T22:09:08.411 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-05-12T22:09:08.411 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-05-12T22:09:08.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.409+0000 7f74b5800640 1 -- 172.21.15.28:0/2221064016 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f74940a2db0 msgr2=0x7f74940a5270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:08.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.409+0000 7f74b5800640 1 --2- 172.21.15.28:0/2221064016 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f74940a2db0 0x7f74940a5270 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f74a8006010 tx=0x7f74a802fe70 comp rx=0 tx=0).stop 2024-05-12T22:09:08.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.409+0000 7f74b5800640 1 -- 172.21.15.28:0/2221064016 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f74b01027e0 msgr2=0x7f74b019a800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:08.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.409+0000 7f74b5800640 1 --2- 172.21.15.28:0/2221064016 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f74b01027e0 0x7f74b019a800 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f749c00e9c0 tx=0x7f749c00ee80 comp rx=0 tx=0).stop 2024-05-12T22:09:08.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.409+0000 7f74b5800640 1 -- 172.21.15.28:0/2221064016 shutdown_connections 2024-05-12T22:09:08.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.409+0000 7f74b5800640 1 --2- 172.21.15.28:0/2221064016 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f74940a2db0 0x7f74940a5270 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:08.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.409+0000 7f74b5800640 1 --2- 172.21.15.28:0/2221064016 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f74b010e0f0 0x7f74b019eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:08.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.409+0000 7f74b5800640 1 --2- 172.21.15.28:0/2221064016 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f74b01027e0 0x7f74b019a800 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:08.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.409+0000 7f74b5800640 1 --2- 172.21.15.28:0/2221064016 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f74b006a370 0x7f74b019a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:08.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.409+0000 7f74b5800640 1 -- 172.21.15.28:0/2221064016 >> 172.21.15.28:0/2221064016 conn(0x7f74b0076530 msgr2=0x7f74b010ef00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:09:08.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.410+0000 7f74b5800640 1 -- 172.21.15.28:0/2221064016 shutdown_connections 2024-05-12T22:09:08.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.410+0000 7f74b5800640 1 -- 172.21.15.28:0/2221064016 wait complete. 2024-05-12T22:09:08.428 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-12T22:09:08.428 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-12T22:09:08.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.606+0000 7fcaa6400640 1 -- 172.21.15.28:0/3032665643 --> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7fcaa0061c70 con 0x7fca880a2e90 2024-05-12T22:09:08.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.641+0000 7f6756a00640 1 -- 172.21.15.28:0/3242819321 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6750104290 msgr2=0x7f67501107a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:08.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.641+0000 7f6756a00640 1 --2- 172.21.15.28:0/3242819321 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6750104290 0x7f67501107a0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f6748009ca0 tx=0x7f674802a8c0 comp rx=0 tx=0).stop 2024-05-12T22:09:08.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.641+0000 7f6756a00640 1 -- 172.21.15.28:0/3242819321 shutdown_connections 2024-05-12T22:09:08.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.641+0000 7f6756a00640 1 --2- 172.21.15.28:0/3242819321 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6750104290 0x7f67501107a0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:08.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.641+0000 7f6756a00640 1 --2- 172.21.15.28:0/3242819321 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67501038d0 0x7f6750103d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:08.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.641+0000 7f6756a00640 1 --2- 172.21.15.28:0/3242819321 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f67501098e0 0x7f6750109cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:08.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.641+0000 7f6756a00640 1 -- 172.21.15.28:0/3242819321 >> 172.21.15.28:0/3242819321 conn(0x7f67500ff340 msgr2=0x7f6750101760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:09:08.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.641+0000 7f6756a00640 1 -- 172.21.15.28:0/3242819321 shutdown_connections 2024-05-12T22:09:08.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.641+0000 7f6756a00640 1 -- 172.21.15.28:0/3242819321 wait complete. 2024-05-12T22:09:08.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.642+0000 7f6756a00640 1 Processor -- start 2024-05-12T22:09:08.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.642+0000 7f6756a00640 1 -- start start 2024-05-12T22:09:08.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.643+0000 7f6756a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f67501038d0 0x7f675019a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:08.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.643+0000 7f6756a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6750104290 0x7f675019a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:08.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.643+0000 7f6756a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67501098e0 0x7f675019ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:08.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.643+0000 7f6756a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6750113d40 con 0x7f67501098e0 2024-05-12T22:09:08.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.643+0000 7f6756a00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f6750113bc0 con 0x7f6750104290 2024-05-12T22:09:08.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.643+0000 7f6756a00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f6750113ec0 con 0x7f67501038d0 2024-05-12T22:09:08.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.643+0000 7f6755600640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f67501038d0 0x7f675019a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:08.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.643+0000 7f6755600640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f67501038d0 0x7f675019a180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:53424/0 (socket says 172.21.15.28:53424) 2024-05-12T22:09:08.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.643+0000 7f6755600640 1 -- 172.21.15.28:0/3811351249 learned_addr learned my addr 172.21.15.28:0/3811351249 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:09:08.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.643+0000 7f6756000640 1 --2- 172.21.15.28:0/3811351249 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67501098e0 0x7f675019ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:08.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.643+0000 7f6754c00640 1 --2- 172.21.15.28:0/3811351249 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6750104290 0x7f675019a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:08.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.644+0000 7f6755600640 1 -- 172.21.15.28:0/3811351249 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6750104290 msgr2=0x7f675019a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:08.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.644+0000 7f6755600640 1 --2- 172.21.15.28:0/3811351249 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6750104290 0x7f675019a6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:08.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.644+0000 7f6755600640 1 -- 172.21.15.28:0/3811351249 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67501098e0 msgr2=0x7f675019ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:08.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.644+0000 7f6755600640 1 --2- 172.21.15.28:0/3811351249 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67501098e0 0x7f675019ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:08.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.644+0000 7f6755600640 1 -- 172.21.15.28:0/3811351249 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f675019f170 con 0x7f67501038d0 2024-05-12T22:09:08.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.644+0000 7f6754c00640 1 --2- 172.21.15.28:0/3811351249 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6750104290 0x7f675019a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:09:08.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.644+0000 7f6755600640 1 --2- 172.21.15.28:0/3811351249 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f67501038d0 0x7f675019a180 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f674400e9f0 tx=0x7f674400edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:09:08.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.644+0000 7f674e000640 1 -- 172.21.15.28:0/3811351249 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f674400cd80 con 0x7f67501038d0 2024-05-12T22:09:08.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.644+0000 7f674e000640 1 -- 172.21.15.28:0/3811351249 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6744005500 con 0x7f67501038d0 2024-05-12T22:09:08.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.644+0000 7f674e000640 1 -- 172.21.15.28:0/3811351249 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6744010690 con 0x7f67501038d0 2024-05-12T22:09:08.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.644+0000 7f6756a00640 1 -- 172.21.15.28:0/3811351249 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f675019f460 con 0x7f67501038d0 2024-05-12T22:09:08.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.644+0000 7f6756a00640 1 -- 172.21.15.28:0/3811351249 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67501a6cf0 con 0x7f67501038d0 2024-05-12T22:09:08.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.648+0000 7f674e000640 1 -- 172.21.15.28:0/3811351249 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 25) ==== 138060+0+0 (secure 0 0 0) 0x7f67440046c0 con 0x7f67501038d0 2024-05-12T22:09:08.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.649+0000 7f675afac740 1 Processor -- start 2024-05-12T22:09:08.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.649+0000 7f675afac740 1 -- start start 2024-05-12T22:09:08.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.649+0000 7f675afac740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55837526bdd0 0x55837526c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:08.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.649+0000 7f675afac740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55837526c6f0 0x558375276880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:08.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.649+0000 7f675afac740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558375276dc0 0x5583752791b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:08.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.649+0000 7f675afac740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5583752342a0 con 0x55837526c6f0 2024-05-12T22:09:08.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.649+0000 7f675afac740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x5583751d0b90 con 0x55837526bdd0 2024-05-12T22:09:08.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.649+0000 7f675afac740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5583751785b0 con 0x558375276dc0 2024-05-12T22:09:08.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.650+0000 7f6756000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55837526bdd0 0x55837526c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:08.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.650+0000 7f6756000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55837526bdd0 0x55837526c1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:51848/0 (socket says 172.21.15.28:51848) 2024-05-12T22:09:08.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.650+0000 7f6754c00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558375276dc0 0x5583752791b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:08.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.650+0000 7f6756000640 1 -- 172.21.15.28:0/2040384759 learned_addr learned my addr 172.21.15.28:0/2040384759 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:09:08.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.650+0000 7f6755600640 1 --2- 172.21.15.28:0/2040384759 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55837526c6f0 0x558375276880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:08.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.650+0000 7f6754c00640 1 -- 172.21.15.28:0/2040384759 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55837526bdd0 msgr2=0x55837526c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:09:08.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.650+0000 7f6754c00640 1 --2- 172.21.15.28:0/2040384759 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55837526bdd0 0x55837526c1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:08.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.650+0000 7f6754c00640 1 -- 172.21.15.28:0/2040384759 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55837526c6f0 msgr2=0x558375276880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:09:08.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.650+0000 7f6754c00640 1 --2- 172.21.15.28:0/2040384759 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55837526c6f0 0x558375276880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:08.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.650+0000 7f6754c00640 1 -- 172.21.15.28:0/2040384759 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558374fae7d0 con 0x558375276dc0 2024-05-12T22:09:08.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.650+0000 7f6756000640 1 --2- 172.21.15.28:0/2040384759 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55837526bdd0 0x55837526c1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:09:08.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.650+0000 7f674e000640 1 --2- 172.21.15.28:0/3811351249 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f67380a2ea0 0x7f67380a5360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:08.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.651+0000 7f6754c00640 1 --2- 172.21.15.28:0/2040384759 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558375276dc0 0x5583752791b0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f673c009fa0 tx=0x7f673c0118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e42baf874f768e34 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:09:08.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.651+0000 7f674e000640 1 -- 172.21.15.28:0/3811351249 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f67440cfa20 con 0x7f67501038d0 2024-05-12T22:09:08.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.651+0000 7f6756000640 1 --2- 172.21.15.28:0/3811351249 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f67380a2ea0 0x7f67380a5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:08.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.651+0000 7f6742600640 1 -- 172.21.15.28:0/2040384759 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f673c004030 con 0x558375276dc0 2024-05-12T22:09:08.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.651+0000 7f6742600640 1 -- 172.21.15.28:0/2040384759 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f673c005500 con 0x558375276dc0 2024-05-12T22:09:08.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.651+0000 7f6742600640 1 -- 172.21.15.28:0/2040384759 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f673c004db0 con 0x558375276dc0 2024-05-12T22:09:08.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.652+0000 7f675afac740 1 -- 172.21.15.28:0/2040384759 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558375276dc0 msgr2=0x5583752791b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:08.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.652+0000 7f675afac740 1 --2- 172.21.15.28:0/2040384759 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558375276dc0 0x5583752791b0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f673c009fa0 tx=0x7f673c0118d0 comp rx=0 tx=0).stop 2024-05-12T22:09:08.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.652+0000 7f6756000640 1 --2- 172.21.15.28:0/3811351249 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f67380a2ea0 0x7f67380a5360 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f674802aeb0 tx=0x7f6748005d80 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:09:08.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.652+0000 7f675afac740 1 -- 172.21.15.28:0/2040384759 shutdown_connections 2024-05-12T22:09:08.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.652+0000 7f675afac740 1 --2- 172.21.15.28:0/2040384759 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558375276dc0 0x5583752791b0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:08.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.652+0000 7f675afac740 1 --2- 172.21.15.28:0/2040384759 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55837526c6f0 0x558375276880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:08.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.652+0000 7f675afac740 1 --2- 172.21.15.28:0/2040384759 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55837526bdd0 0x55837526c1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:08.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.652+0000 7f675afac740 1 -- 172.21.15.28:0/2040384759 >> 172.21.15.28:0/2040384759 conn(0x5583752680f0 msgr2=0x5583752684d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:09:08.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.652+0000 7f675afac740 1 -- 172.21.15.28:0/2040384759 shutdown_connections 2024-05-12T22:09:08.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.652+0000 7f675afac740 1 -- 172.21.15.28:0/2040384759 wait complete. 2024-05-12T22:09:08.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.653+0000 7f675afac740 1 -- start start 2024-05-12T22:09:08.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.653+0000 7f675afac740 1 Processor -- start 2024-05-12T22:09:08.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.654+0000 7f675afac740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55837526bdd0 0x558375276700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:08.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.654+0000 7f675afac740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55837526c6f0 0x5583752739f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:08.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.654+0000 7f675afac740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558375276dc0 0x558375273f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:08.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.654+0000 7f675afac740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55837518b9f0 con 0x55837526c6f0 2024-05-12T22:09:08.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.654+0000 7f675afac740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x5583751ccb50 con 0x558375276dc0 2024-05-12T22:09:08.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.654+0000 7f675afac740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x558375279bb0 con 0x55837526bdd0 2024-05-12T22:09:08.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.654+0000 7f675afac740 10 client.0 _ll_register_callbacks cb 0x558374ee3be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:09:08.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.654+0000 7f6754c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55837526c6f0 0x5583752739f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:08.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.654+0000 7f6754c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55837526c6f0 0x5583752739f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49586/0 (socket says 172.21.15.28:49586) 2024-05-12T22:09:08.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.654+0000 7f6754c00640 1 -- 172.21.15.28:0/3682547008 learned_addr learned my addr 172.21.15.28:0/3682547008 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:09:08.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.654+0000 7f6756000640 1 --2- 172.21.15.28:0/3682547008 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558375276dc0 0x558375273f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:08.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.655+0000 7f6754c00640 1 -- 172.21.15.28:0/3682547008 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55837526bdd0 msgr2=0x558375276700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:09:08.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.655+0000 7f6754c00640 1 --2- 172.21.15.28:0/3682547008 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55837526bdd0 0x558375276700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:08.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.655+0000 7f6754c00640 1 -- 172.21.15.28:0/3682547008 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558375276dc0 msgr2=0x558375273f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:09:08.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.655+0000 7f6754c00640 1 --2- 172.21.15.28:0/3682547008 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558375276dc0 0x558375273f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:08.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.655+0000 7f6754c00640 1 -- 172.21.15.28:0/3682547008 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558374ee43f0 con 0x55837526c6f0 2024-05-12T22:09:08.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.655+0000 7f6756000640 1 --2- 172.21.15.28:0/3682547008 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558375276dc0 0x558375273f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:09:08.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.655+0000 7f6754c00640 1 --2- 172.21.15.28:0/3682547008 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55837526c6f0 0x5583752739f0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f673c009560 tx=0x7f673c0056b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3183fd24dab2d87 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:09:08.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.655+0000 7f6736a00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-12T22:09:08.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.655+0000 7f6736a00640 1 -- 172.21.15.28:0/3682547008 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f673c01d050 con 0x55837526c6f0 2024-05-12T22:09:08.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.656+0000 7f6736a00640 1 -- 172.21.15.28:0/3682547008 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f673c003800 con 0x55837526c6f0 2024-05-12T22:09:08.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.656+0000 7f675afac740 1 -- 172.21.15.28:0/3682547008 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f673c005500 con 0x55837526c6f0 2024-05-12T22:09:08.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.656+0000 7f6736a00640 1 -- 172.21.15.28:0/3682547008 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f673c032050 con 0x55837526c6f0 2024-05-12T22:09:08.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.656+0000 7f675afac740 10 client.15309 fetch_fsmap learned FSMap version 24 2024-05-12T22:09:08.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.656+0000 7f675afac740 1 -- 172.21.15.28:0/3682547008 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x558374ea40f0 con 0x55837526c6f0 2024-05-12T22:09:08.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.656+0000 7f6736a00640 1 -- 172.21.15.28:0/3682547008 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f673c005500 con 0x55837526c6f0 2024-05-12T22:09:08.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.657+0000 7f6736a00640 1 -- 172.21.15.28:0/3682547008 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f673c0051a0 con 0x55837526c6f0 2024-05-12T22:09:08.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.657+0000 7f675afac740 10 client.15309 fetch_fsmap finished waiting for FSMap version 24 2024-05-12T22:09:08.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.657+0000 7f675afac740 10 client.15309 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:09:08.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.657+0000 7f675afac740 20 client.15309 populate_metadata read hostname 'smithi028' 2024-05-12T22:09:08.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.657+0000 7f675afac740 1 --2- 172.21.15.28:0/3682547008 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5583752867d0 0x558375288bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:08.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.657+0000 7f675afac740 4 client.15309 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-12T22:09:08.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.657+0000 7f675afac740 1 -- 172.21.15.28:0/3682547008 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5583751de2c0 con 0x5583752867d0 2024-05-12T22:09:08.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.657+0000 7f6755600640 1 --2- 172.21.15.28:0/3682547008 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5583752867d0 0x558375288bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:08.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.658+0000 7f6755600640 1 --2- 172.21.15.28:0/3682547008 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5583752867d0 0x558375288bc0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9113cddb1212f2a9 server_cookie=319111490fc8103e in_seq=0 out_seq=0 2024-05-12T22:09:08.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.658+0000 7f6736a00640 10 client.15309 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:09:08.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.658+0000 7f6736a00640 1 -- 172.21.15.28:0/3682547008 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5583751de2c0 con 0x5583752867d0 2024-05-12T22:09:08.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.658+0000 7f6736a00640 10 client.15309 handle_command_reply: tid=0 2024-05-12T22:09:08.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.658+0000 7f675afac740 1 client.15309 shutdown 2024-05-12T22:09:08.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.659+0000 7f675afac740 1 -- 172.21.15.28:0/3682547008 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55837526c6f0 msgr2=0x5583752739f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:08.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.659+0000 7f675afac740 1 --2- 172.21.15.28:0/3682547008 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55837526c6f0 0x5583752739f0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f673c009560 tx=0x7f673c0056b0 comp rx=0 tx=0).stop 2024-05-12T22:09:08.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.659+0000 7f675afac740 1 -- 172.21.15.28:0/3682547008 shutdown_connections 2024-05-12T22:09:08.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.659+0000 7f675afac740 1 --2- 172.21.15.28:0/3682547008 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5583752867d0 0x558375288bc0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:08.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.659+0000 7f675afac740 1 --2- 172.21.15.28:0/3682547008 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558375276dc0 0x558375273f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:08.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.659+0000 7f675afac740 1 --2- 172.21.15.28:0/3682547008 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55837526c6f0 0x5583752739f0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:08.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.659+0000 7f675afac740 1 --2- 172.21.15.28:0/3682547008 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55837526bdd0 0x558375276700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:08.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.659+0000 7f675afac740 1 -- 172.21.15.28:0/3682547008 >> 172.21.15.28:0/3682547008 conn(0x5583752680f0 msgr2=0x55837526def0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:09:08.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.659+0000 7f6736a00640 0 client.15309 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:09:08.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.659+0000 7f675afac740 1 -- 172.21.15.28:0/3682547008 shutdown_connections 2024-05-12T22:09:08.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.659+0000 7f675afac740 1 -- 172.21.15.28:0/3682547008 wait complete. 2024-05-12T22:09:08.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.659+0000 7f675afac740 20 client.15309 trim_cache size 0 max 16384 2024-05-12T22:09:08.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.685+0000 7f675afac740 1 Processor -- start 2024-05-12T22:09:08.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.685+0000 7f675afac740 1 -- start start 2024-05-12T22:09:08.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.686+0000 7f675afac740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55837526bdd0 0x5583752631c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:08.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.686+0000 7f675afac740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55837526c6f0 0x558375272b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:08.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.686+0000 7f675afac740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558375276dc0 0x5583752828f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:08.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.686+0000 7f675afac740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f67501b1d70 con 0x55837526c6f0 2024-05-12T22:09:08.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.686+0000 7f675afac740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f67501af640 con 0x558375276dc0 2024-05-12T22:09:08.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.686+0000 7f675afac740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f67501ad490 con 0x55837526bdd0 2024-05-12T22:09:08.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.686+0000 7f6756000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558375276dc0 0x5583752828f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:08.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.686+0000 7f6755600640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55837526bdd0 0x5583752631c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:08.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.686+0000 7f6756000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558375276dc0 0x5583752828f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:51862/0 (socket says 172.21.15.28:51862) 2024-05-12T22:09:08.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.686+0000 7f6756000640 1 -- 172.21.15.28:0/2553268617 learned_addr learned my addr 172.21.15.28:0/2553268617 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:09:08.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.686+0000 7f6754c00640 1 --2- 172.21.15.28:0/2553268617 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55837526c6f0 0x558375272b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:08.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.686+0000 7f6756000640 1 -- 172.21.15.28:0/2553268617 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55837526bdd0 msgr2=0x5583752631c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:09:08.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.686+0000 7f6756000640 1 --2- 172.21.15.28:0/2553268617 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55837526bdd0 0x5583752631c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:08.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.687+0000 7f6756000640 1 -- 172.21.15.28:0/2553268617 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55837526c6f0 msgr2=0x558375272b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:09:08.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.687+0000 7f6756000640 1 --2- 172.21.15.28:0/2553268617 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55837526c6f0 0x558375272b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:08.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.687+0000 7f6756000640 1 -- 172.21.15.28:0/2553268617 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558375263820 con 0x558375276dc0 2024-05-12T22:09:08.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.687+0000 7f6756000640 1 --2- 172.21.15.28:0/2553268617 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558375276dc0 0x5583752828f0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f6748009ca0 tx=0x7f6748044000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b3582bea75f87670 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:09:08.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.687+0000 7f6741c00640 1 -- 172.21.15.28:0/2553268617 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f674802f610 con 0x558375276dc0 2024-05-12T22:09:08.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.687+0000 7f6741c00640 1 -- 172.21.15.28:0/2553268617 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6748036050 con 0x558375276dc0 2024-05-12T22:09:08.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.687+0000 7f6741c00640 1 -- 172.21.15.28:0/2553268617 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6748030d90 con 0x558375276dc0 2024-05-12T22:09:08.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.688+0000 7f675afac740 1 -- 172.21.15.28:0/2553268617 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558375276dc0 msgr2=0x5583752828f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:08.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.688+0000 7f675afac740 1 --2- 172.21.15.28:0/2553268617 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558375276dc0 0x5583752828f0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f6748009ca0 tx=0x7f6748044000 comp rx=0 tx=0).stop 2024-05-12T22:09:08.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.688+0000 7f675afac740 1 -- 172.21.15.28:0/2553268617 shutdown_connections 2024-05-12T22:09:08.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.688+0000 7f675afac740 1 --2- 172.21.15.28:0/2553268617 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558375276dc0 0x5583752828f0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:08.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.688+0000 7f675afac740 1 --2- 172.21.15.28:0/2553268617 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55837526c6f0 0x558375272b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:08.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.688+0000 7f675afac740 1 --2- 172.21.15.28:0/2553268617 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55837526bdd0 0x5583752631c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:08.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.688+0000 7f675afac740 1 -- 172.21.15.28:0/2553268617 >> 172.21.15.28:0/2553268617 conn(0x5583752680f0 msgr2=0x558375266aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:09:08.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.688+0000 7f675afac740 1 -- 172.21.15.28:0/2553268617 shutdown_connections 2024-05-12T22:09:08.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.689+0000 7f675afac740 1 -- 172.21.15.28:0/2553268617 wait complete. 2024-05-12T22:09:08.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.689+0000 7f675afac740 1 -- start start 2024-05-12T22:09:08.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.689+0000 7f675afac740 1 Processor -- start 2024-05-12T22:09:08.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.690+0000 7f675afac740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55837526bdd0 0x55837527f370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:08.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.690+0000 7f675afac740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55837526c6f0 0x55837527bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:08.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.690+0000 7f675afac740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558375276dc0 0x55837527c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:08.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.690+0000 7f675afac740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f67501adb00 con 0x55837526bdd0 2024-05-12T22:09:08.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.690+0000 7f675afac740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f67501aef70 con 0x558375276dc0 2024-05-12T22:09:08.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.690+0000 7f675afac740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f67501ae1d0 con 0x55837526c6f0 2024-05-12T22:09:08.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.690+0000 7f675afac740 10 client.0 _ll_register_callbacks cb 0x7f6744005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:09:08.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.690+0000 7f6756000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558375276dc0 0x55837527c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:08.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.690+0000 7f6755600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55837526bdd0 0x55837527f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:08.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.690+0000 7f6755600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55837526bdd0 0x55837527f370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49614/0 (socket says 172.21.15.28:49614) 2024-05-12T22:09:08.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.690+0000 7f6755600640 1 -- 172.21.15.28:0/2551515667 learned_addr learned my addr 172.21.15.28:0/2551515667 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:09:08.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.690+0000 7f6754c00640 1 --2- 172.21.15.28:0/2551515667 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55837526c6f0 0x55837527bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:08.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.691+0000 7f6755600640 1 -- 172.21.15.28:0/2551515667 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55837526c6f0 msgr2=0x55837527bdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:09:08.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.691+0000 7f6755600640 1 --2- 172.21.15.28:0/2551515667 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55837526c6f0 0x55837527bdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:08.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.691+0000 7f6755600640 1 -- 172.21.15.28:0/2551515667 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558375276dc0 msgr2=0x55837527c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:09:08.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.691+0000 7f6755600640 1 --2- 172.21.15.28:0/2551515667 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558375276dc0 0x55837527c4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:08.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.691+0000 7f6755600640 1 -- 172.21.15.28:0/2551515667 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5583752eac60 con 0x55837526bdd0 2024-05-12T22:09:08.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.691+0000 7f6755600640 1 --2- 172.21.15.28:0/2551515667 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55837526bdd0 0x55837527f370 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f674400e9c0 tx=0x7f674409bb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6bb2a3078b4c254 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:09:08.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.691+0000 7f6736a00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-12T22:09:08.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.691+0000 7f6736a00640 1 -- 172.21.15.28:0/2551515667 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f674409c3c0 con 0x55837526bdd0 2024-05-12T22:09:08.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.691+0000 7f6736a00640 1 -- 172.21.15.28:0/2551515667 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f6744098330 con 0x55837526bdd0 2024-05-12T22:09:08.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.691+0000 7f675afac740 1 -- 172.21.15.28:0/2551515667 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6748036050 con 0x55837526bdd0 2024-05-12T22:09:08.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.692+0000 7f6736a00640 1 -- 172.21.15.28:0/2551515667 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f674409bcc0 con 0x55837526bdd0 2024-05-12T22:09:08.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.692+0000 7f675afac740 10 client.15315 fetch_fsmap learned FSMap version 24 2024-05-12T22:09:08.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.692+0000 7f675afac740 1 -- 172.21.15.28:0/2551515667 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5583752ebda0 con 0x55837526bdd0 2024-05-12T22:09:08.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.692+0000 7f6736a00640 1 -- 172.21.15.28:0/2551515667 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6748036050 con 0x55837526bdd0 2024-05-12T22:09:08.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.692+0000 7f6736a00640 1 -- 172.21.15.28:0/2551515667 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f67440a4b80 con 0x55837526bdd0 2024-05-12T22:09:08.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.692+0000 7f675afac740 10 client.15315 fetch_fsmap finished waiting for FSMap version 24 2024-05-12T22:09:08.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.692+0000 7f675afac740 10 client.15315 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:09:08.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.692+0000 7f675afac740 20 client.15315 populate_metadata read hostname 'smithi028' 2024-05-12T22:09:08.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.692+0000 7f675afac740 1 --2- 172.21.15.28:0/2551515667 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5583752ec420 0x5583752ee810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:08.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.692+0000 7f675afac740 4 client.15315 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-12T22:09:08.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.692+0000 7f675afac740 1 -- 172.21.15.28:0/2551515667 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5583751d8610 con 0x5583752ec420 2024-05-12T22:09:08.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.693+0000 7f6754c00640 1 --2- 172.21.15.28:0/2551515667 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5583752ec420 0x5583752ee810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:08.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.693+0000 7f6754c00640 1 --2- 172.21.15.28:0/2551515667 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5583752ec420 0x5583752ee810 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=16e795d6a09e6ca6 server_cookie=bcef4d0191971792 in_seq=0 out_seq=0 2024-05-12T22:09:08.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.693+0000 7f6736a00640 10 client.15315 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:09:08.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.694+0000 7f6736a00640 1 -- 172.21.15.28:0/2551515667 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5583751d8610 con 0x5583752ec420 2024-05-12T22:09:08.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.694+0000 7f6736a00640 10 client.15315 handle_command_reply: tid=0 2024-05-12T22:09:08.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.694+0000 7f675afac740 1 client.15315 shutdown 2024-05-12T22:09:08.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.694+0000 7f675afac740 1 -- 172.21.15.28:0/2551515667 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55837526bdd0 msgr2=0x55837527f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:08.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.694+0000 7f675afac740 1 --2- 172.21.15.28:0/2551515667 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55837526bdd0 0x55837527f370 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f674400e9c0 tx=0x7f674409bb50 comp rx=0 tx=0).stop 2024-05-12T22:09:08.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.694+0000 7f675afac740 1 -- 172.21.15.28:0/2551515667 shutdown_connections 2024-05-12T22:09:08.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.694+0000 7f675afac740 1 --2- 172.21.15.28:0/2551515667 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5583752ec420 0x5583752ee810 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:08.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.694+0000 7f675afac740 1 --2- 172.21.15.28:0/2551515667 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558375276dc0 0x55837527c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:08.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.694+0000 7f675afac740 1 --2- 172.21.15.28:0/2551515667 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55837526c6f0 0x55837527bdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:08.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.694+0000 7f675afac740 1 --2- 172.21.15.28:0/2551515667 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55837526bdd0 0x55837527f370 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:08.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.694+0000 7f6736a00640 0 client.15315 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:09:08.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.694+0000 7f675afac740 1 -- 172.21.15.28:0/2551515667 >> 172.21.15.28:0/2551515667 conn(0x5583752680f0 msgr2=0x5583752729d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:09:08.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.695+0000 7f675afac740 1 -- 172.21.15.28:0/2551515667 shutdown_connections 2024-05-12T22:09:08.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.695+0000 7f675afac740 1 -- 172.21.15.28:0/2551515667 wait complete. 2024-05-12T22:09:08.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.695+0000 7f675afac740 20 client.15315 trim_cache size 0 max 16384 2024-05-12T22:09:08.700 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-05-12T22:09:08.700 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-12T22:09:08.700 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-05-12T22:09:08.700 INFO:teuthology.orchestra.run.smithi028.stdout: "86318bd8-7621-4ea4-be1f-32b76b44b012": { 2024-05-12T22:09:08.700 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-05-12T22:09:08.700 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "86318bd8-7621-4ea4-be1f-32b76b44b012", 2024-05-12T22:09:08.700 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-05-12T22:09:08.700 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:09:08.700 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:09:08.700 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-05-12T22:09:08.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.696+0000 7f6756a00640 1 -- 172.21.15.28:0/3811351249 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f67380a2ea0 msgr2=0x7f67380a5360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:08.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.696+0000 7f6756a00640 1 --2- 172.21.15.28:0/3811351249 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f67380a2ea0 0x7f67380a5360 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f674802aeb0 tx=0x7f6748005d80 comp rx=0 tx=0).stop 2024-05-12T22:09:08.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.696+0000 7f6756a00640 1 -- 172.21.15.28:0/3811351249 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f67501038d0 msgr2=0x7f675019a180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:08.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.696+0000 7f6756a00640 1 --2- 172.21.15.28:0/3811351249 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f67501038d0 0x7f675019a180 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f674400e9f0 tx=0x7f674400edf0 comp rx=0 tx=0).stop 2024-05-12T22:09:08.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.696+0000 7f6756a00640 1 -- 172.21.15.28:0/3811351249 shutdown_connections 2024-05-12T22:09:08.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.696+0000 7f6756a00640 1 --2- 172.21.15.28:0/3811351249 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f67380a2ea0 0x7f67380a5360 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:08.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.696+0000 7f6756a00640 1 --2- 172.21.15.28:0/3811351249 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67501098e0 0x7f675019ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:08.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.696+0000 7f6756a00640 1 --2- 172.21.15.28:0/3811351249 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6750104290 0x7f675019a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:08.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.696+0000 7f6756a00640 1 --2- 172.21.15.28:0/3811351249 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f67501038d0 0x7f675019a180 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:08.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.696+0000 7f6756a00640 1 -- 172.21.15.28:0/3811351249 >> 172.21.15.28:0/3811351249 conn(0x7f67500ff340 msgr2=0x7f67500ff860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:09:08.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.696+0000 7f6756a00640 1 -- 172.21.15.28:0/3811351249 shutdown_connections 2024-05-12T22:09:08.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.696+0000 7f6756a00640 1 -- 172.21.15.28:0/3811351249 wait complete. 2024-05-12T22:09:08.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.696+0000 7fca9d600640 1 -- 172.21.15.28:0/3032665643 <== mgr.25189 v2:172.21.15.28:6800/3177640985 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fcaa0061c70 con 0x7fca880a2e90 2024-05-12T22:09:08.702 INFO:teuthology.orchestra.run.smithi028.stdout:Schedule set for path / 2024-05-12T22:09:08.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.699+0000 7fcaa6400640 1 -- 172.21.15.28:0/3032665643 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fca880a2e90 msgr2=0x7fca880a5350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:08.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.699+0000 7fcaa6400640 1 --2- 172.21.15.28:0/3032665643 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fca880a2e90 0x7fca880a5350 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fca8c009230 tx=0x7fca8c0058d0 comp rx=0 tx=0).stop 2024-05-12T22:09:08.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.699+0000 7fcaa6400640 1 -- 172.21.15.28:0/3032665643 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcaa0073c60 msgr2=0x7fcaa019a130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:08.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.699+0000 7fcaa6400640 1 --2- 172.21.15.28:0/3032665643 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcaa0073c60 0x7fcaa019a130 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fca9400e9f0 tx=0x7fca9400edf0 comp rx=0 tx=0).stop 2024-05-12T22:09:08.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.699+0000 7fcaa6400640 1 -- 172.21.15.28:0/3032665643 shutdown_connections 2024-05-12T22:09:08.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.699+0000 7fcaa6400640 1 --2- 172.21.15.28:0/3032665643 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fca880a2e90 0x7fca880a5350 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:08.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.699+0000 7fcaa6400640 1 --2- 172.21.15.28:0/3032665643 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcaa010de70 0x7fcaa019e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:08.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.699+0000 7fcaa6400640 1 --2- 172.21.15.28:0/3032665643 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcaa0074580 0x7fcaa019a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:08.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.699+0000 7fcaa6400640 1 --2- 172.21.15.28:0/3032665643 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcaa0073c60 0x7fcaa019a130 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:08.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.699+0000 7fcaa6400640 1 -- 172.21.15.28:0/3032665643 >> 172.21.15.28:0/3032665643 conn(0x7fcaa00fd180 msgr2=0x7fcaa00fdeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:09:08.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.700+0000 7fcaa6400640 1 -- 172.21.15.28:0/3032665643 shutdown_connections 2024-05-12T22:09:08.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.700+0000 7fcaa6400640 1 -- 172.21.15.28:0/3032665643 wait complete. 2024-05-12T22:09:08.714 INFO:tasks.cephfs.filesystem:scrub status for tag:86318bd8-7621-4ea4-be1f-32b76b44b012 - {'path': '/', 'tag': '86318bd8-7621-4ea4-be1f-32b76b44b012', 'options': 'recursive,force'} 2024-05-12T22:09:08.717 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h' 2024-05-12T22:09:08.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.965+0000 7f3853e00640 1 -- 172.21.15.28:0/795734687 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3854103870 msgr2=0x7f3854103cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:08.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.965+0000 7f3853e00640 1 --2- 172.21.15.28:0/795734687 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3854103870 0x7f3854103cf0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f383c009900 tx=0x7f383c02a3f0 comp rx=0 tx=0).stop 2024-05-12T22:09:08.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.966+0000 7f3853e00640 1 -- 172.21.15.28:0/795734687 shutdown_connections 2024-05-12T22:09:08.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.966+0000 7f3853e00640 1 --2- 172.21.15.28:0/795734687 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3854104230 0x7f3854110740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:08.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.966+0000 7f3853e00640 1 --2- 172.21.15.28:0/795734687 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3854103870 0x7f3854103cf0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:08.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.966+0000 7f3853e00640 1 --2- 172.21.15.28:0/795734687 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3854109880 0x7f3854109c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:08.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.966+0000 7f3853e00640 1 -- 172.21.15.28:0/795734687 >> 172.21.15.28:0/795734687 conn(0x7f38540ff340 msgr2=0x7f3854101760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:09:08.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.966+0000 7f3853e00640 1 -- 172.21.15.28:0/795734687 shutdown_connections 2024-05-12T22:09:08.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.966+0000 7f3853e00640 1 -- 172.21.15.28:0/795734687 wait complete. 2024-05-12T22:09:08.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.967+0000 7f3853e00640 1 Processor -- start 2024-05-12T22:09:08.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.967+0000 7f3853e00640 1 -- start start 2024-05-12T22:09:08.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.967+0000 7f3853e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3854103870 0x7f385419a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:08.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.967+0000 7f3853e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3854104230 0x7f385419a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:08.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.967+0000 7f3853e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3854109880 0x7f385419e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:08.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.967+0000 7f3853e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3854113ce0 con 0x7f3854109880 2024-05-12T22:09:08.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.967+0000 7f3853e00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f3854113b60 con 0x7f3854104230 2024-05-12T22:09:08.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.967+0000 7f3853e00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f3854113e60 con 0x7f3854103870 2024-05-12T22:09:08.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.968+0000 7f3852000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3854104230 0x7f385419a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:08.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.968+0000 7f3852000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3854104230 0x7f385419a6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:51884/0 (socket says 172.21.15.28:51884) 2024-05-12T22:09:08.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.968+0000 7f3852000640 1 -- 172.21.15.28:0/40301795 learned_addr learned my addr 172.21.15.28:0/40301795 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:09:08.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.968+0000 7f3853400640 1 --2- 172.21.15.28:0/40301795 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3854109880 0x7f385419e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:08.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.968+0000 7f3852a00640 1 --2- 172.21.15.28:0/40301795 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3854103870 0x7f385419a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:08.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.968+0000 7f3852000640 1 -- 172.21.15.28:0/40301795 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3854103870 msgr2=0x7f385419a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:08.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.968+0000 7f3852000640 1 --2- 172.21.15.28:0/40301795 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3854103870 0x7f385419a160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:08.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.968+0000 7f3852000640 1 -- 172.21.15.28:0/40301795 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3854109880 msgr2=0x7f385419e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:08.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.968+0000 7f3852000640 1 --2- 172.21.15.28:0/40301795 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3854109880 0x7f385419e9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:08.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.968+0000 7f3852000640 1 -- 172.21.15.28:0/40301795 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f385419f100 con 0x7f3854104230 2024-05-12T22:09:08.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.968+0000 7f3852000640 1 --2- 172.21.15.28:0/40301795 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3854104230 0x7f385419a6a0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f383c02a9e0 tx=0x7f383c003c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:09:08.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.969+0000 7f384b400640 1 -- 172.21.15.28:0/40301795 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f383c03f070 con 0x7f3854104230 2024-05-12T22:09:08.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.969+0000 7f384b400640 1 -- 172.21.15.28:0/40301795 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f383c031c30 con 0x7f3854104230 2024-05-12T22:09:08.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.969+0000 7f3853e00640 1 -- 172.21.15.28:0/40301795 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f385419f390 con 0x7f3854104230 2024-05-12T22:09:08.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.969+0000 7f384b400640 1 -- 172.21.15.28:0/40301795 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f383c035030 con 0x7f3854104230 2024-05-12T22:09:08.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.969+0000 7f3853e00640 1 -- 172.21.15.28:0/40301795 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38541a6c80 con 0x7f3854104230 2024-05-12T22:09:08.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.970+0000 7f3853e00640 1 -- 172.21.15.28:0/40301795 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f385404fe40 con 0x7f3854104230 2024-05-12T22:09:08.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.971+0000 7f384b400640 1 -- 172.21.15.28:0/40301795 <== mon.2 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 25) ==== 138060+0+0 (secure 0 0 0) 0x7f383c042070 con 0x7f3854104230 2024-05-12T22:09:08.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.973+0000 7f384b400640 1 --2- 172.21.15.28:0/40301795 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f38380a2e90 0x7f38380a5350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:08.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.973+0000 7f384b400640 1 -- 172.21.15.28:0/40301795 <== mon.2 v2:172.21.15.86:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f383c0efc20 con 0x7f3854104230 2024-05-12T22:09:08.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.973+0000 7f3852a00640 1 --2- 172.21.15.28:0/40301795 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f38380a2e90 0x7f38380a5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:08.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.974+0000 7f3852a00640 1 --2- 172.21.15.28:0/40301795 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f38380a2e90 0x7f38380a5350 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f3844009ae0 tx=0x7f3844005f90 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:09:08.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:08.977+0000 7f384b400640 1 -- 172.21.15.28:0/40301795 <== mon.2 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f383c0bc690 con 0x7f3854104230 2024-05-12T22:09:09.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:08 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/2644805504' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-12T22:09:09.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:08 smithi028 ceph-mon[29847]: scrub queued for path: / 2024-05-12T22:09:09.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:08 smithi028 ceph-mon[29847]: scrub summary: idle+waiting paths [/] 2024-05-12T22:09:09.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:08 smithi028 ceph-mon[29847]: scrub summary: active paths [/] 2024-05-12T22:09:09.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:08 smithi028 ceph-mon[29847]: scrub summary: idle+waiting paths [/] 2024-05-12T22:09:09.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:08 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/2644805504' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-12T22:09:09.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:08 smithi186 ceph-mon[34785]: scrub queued for path: / 2024-05-12T22:09:09.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:08 smithi186 ceph-mon[34785]: scrub summary: idle+waiting paths [/] 2024-05-12T22:09:09.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:08 smithi186 ceph-mon[34785]: scrub summary: active paths [/] 2024-05-12T22:09:09.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:08 smithi186 ceph-mon[34785]: scrub summary: idle+waiting paths [/] 2024-05-12T22:09:09.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:09.205+0000 7f3853e00640 1 -- 172.21.15.28:0/40301795 --> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}) -- 0x7f3854061c70 con 0x7f38380a2e90 2024-05-12T22:09:09.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:08 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/2644805504' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-12T22:09:09.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:08 smithi086 ceph-mon[34297]: scrub queued for path: / 2024-05-12T22:09:09.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:08 smithi086 ceph-mon[34297]: scrub summary: idle+waiting paths [/] 2024-05-12T22:09:09.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:08 smithi086 ceph-mon[34297]: scrub summary: active paths [/] 2024-05-12T22:09:09.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:08 smithi086 ceph-mon[34297]: scrub summary: idle+waiting paths [/] 2024-05-12T22:09:09.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:09.357+0000 7f384b400640 1 -- 172.21.15.28:0/40301795 <== mgr.25189 v2:172.21.15.28:6800/3177640985 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+26 (secure 0 0 0) 0x7f3854061c70 con 0x7f38380a2e90 2024-05-12T22:09:09.357 INFO:teuthology.orchestra.run.smithi028.stdout:Retention added to path / 2024-05-12T22:09:09.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:09.362+0000 7f3853e00640 1 -- 172.21.15.28:0/40301795 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f38380a2e90 msgr2=0x7f38380a5350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:09.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:09.362+0000 7f3853e00640 1 --2- 172.21.15.28:0/40301795 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f38380a2e90 0x7f38380a5350 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f3844009ae0 tx=0x7f3844005f90 comp rx=0 tx=0).stop 2024-05-12T22:09:09.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:09.362+0000 7f3853e00640 1 -- 172.21.15.28:0/40301795 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3854104230 msgr2=0x7f385419a6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:09.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:09.362+0000 7f3853e00640 1 --2- 172.21.15.28:0/40301795 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3854104230 0x7f385419a6a0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f383c02a9e0 tx=0x7f383c003c10 comp rx=0 tx=0).stop 2024-05-12T22:09:09.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:09.362+0000 7f3853e00640 1 -- 172.21.15.28:0/40301795 shutdown_connections 2024-05-12T22:09:09.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:09.362+0000 7f3853e00640 1 --2- 172.21.15.28:0/40301795 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f38380a2e90 0x7f38380a5350 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:09.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:09.362+0000 7f3853e00640 1 --2- 172.21.15.28:0/40301795 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3854109880 0x7f385419e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:09.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:09.362+0000 7f3853e00640 1 --2- 172.21.15.28:0/40301795 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3854104230 0x7f385419a6a0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:09.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:09.362+0000 7f3853e00640 1 --2- 172.21.15.28:0/40301795 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3854103870 0x7f385419a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:09.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:09.362+0000 7f3853e00640 1 -- 172.21.15.28:0/40301795 >> 172.21.15.28:0/40301795 conn(0x7f38540ff340 msgr2=0x7f38540ff940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:09:09.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:09.362+0000 7f3853e00640 1 -- 172.21.15.28:0/40301795 shutdown_connections 2024-05-12T22:09:09.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:09.362+0000 7f3853e00640 1 -- 172.21.15.28:0/40301795 wait complete. 2024-05-12T22:09:09.384 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-05-12T22:09:09.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:09.593+0000 7f19a9200640 1 -- 172.21.15.28:0/2697952554 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f19a41126d0 msgr2=0x7f19a4114ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:09.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:09.593+0000 7f19a9200640 1 --2- 172.21.15.28:0/2697952554 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f19a41126d0 0x7f19a4114ac0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f199c009c10 tx=0x7f199c02a6f0 comp rx=0 tx=0).stop 2024-05-12T22:09:09.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:09.593+0000 7f19a9200640 1 -- 172.21.15.28:0/2697952554 shutdown_connections 2024-05-12T22:09:09.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:09.593+0000 7f19a9200640 1 --2- 172.21.15.28:0/2697952554 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f19a41126d0 0x7f19a4114ac0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:09.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:09.593+0000 7f19a9200640 1 --2- 172.21.15.28:0/2697952554 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f19a4076680 0x7f19a4112190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:09.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:09.593+0000 7f19a9200640 1 --2- 172.21.15.28:0/2697952554 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19a4075d60 0x7f19a4076140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:09.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:09.593+0000 7f19a9200640 1 -- 172.21.15.28:0/2697952554 >> 172.21.15.28:0/2697952554 conn(0x7f19a40ff560 msgr2=0x7f19a4101980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:09:09.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:09.593+0000 7f19a9200640 1 -- 172.21.15.28:0/2697952554 shutdown_connections 2024-05-12T22:09:09.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:09.593+0000 7f19a9200640 1 -- 172.21.15.28:0/2697952554 wait complete. 2024-05-12T22:09:09.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:09.594+0000 7f19a9200640 1 Processor -- start 2024-05-12T22:09:09.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:09.594+0000 7f19a9200640 1 -- start start 2024-05-12T22:09:09.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:09.595+0000 7f19a9200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19a4075d60 0x7f19a419e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:09.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:09.595+0000 7f19a9200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f19a4076680 0x7f19a419ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:09.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:09.595+0000 7f19a9200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f19a41126d0 0x7f19a41a2fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:09.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:09.595+0000 7f19a9200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f19a4118210 con 0x7f19a4075d60 2024-05-12T22:09:09.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:09.595+0000 7f19a9200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f19a4118090 con 0x7f19a41126d0 2024-05-12T22:09:09.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:09.595+0000 7f19a9200640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f19a4118390 con 0x7f19a4076680 2024-05-12T22:09:09.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:09.595+0000 7f19a1e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19a4075d60 0x7f19a419e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:09.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:09.595+0000 7f19a1e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19a4075d60 0x7f19a419e6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49658/0 (socket says 172.21.15.28:49658) 2024-05-12T22:09:09.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:09.595+0000 7f19a1e00640 1 -- 172.21.15.28:0/689067674 learned_addr learned my addr 172.21.15.28:0/689067674 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:09:09.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:09.596+0000 7f19a2800640 1 --2- 172.21.15.28:0/689067674 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f19a41126d0 0x7f19a41a2fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:09.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:09.596+0000 7f199be00640 1 --2- 172.21.15.28:0/689067674 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f19a4076680 0x7f19a419ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:09.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:09.596+0000 7f19a1e00640 1 -- 172.21.15.28:0/689067674 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f19a4076680 msgr2=0x7f19a419ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:09.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:09.596+0000 7f19a1e00640 1 --2- 172.21.15.28:0/689067674 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f19a4076680 0x7f19a419ec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:09.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:09.596+0000 7f19a1e00640 1 -- 172.21.15.28:0/689067674 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f19a41126d0 msgr2=0x7f19a41a2fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:09.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:09.596+0000 7f19a1e00640 1 --2- 172.21.15.28:0/689067674 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f19a41126d0 0x7f19a41a2fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:09.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:09.596+0000 7f19a1e00640 1 -- 172.21.15.28:0/689067674 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19a41a36e0 con 0x7f19a4075d60 2024-05-12T22:09:09.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:09.596+0000 7f19a1e00640 1 --2- 172.21.15.28:0/689067674 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19a4075d60 0x7f19a419e6f0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f199400e9c0 tx=0x7f199400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:09:09.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:09.597+0000 7f199aa00640 1 -- 172.21.15.28:0/689067674 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f199400ccc0 con 0x7f19a4075d60 2024-05-12T22:09:09.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:09.597+0000 7f199aa00640 1 -- 172.21.15.28:0/689067674 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1994005500 con 0x7f19a4075d60 2024-05-12T22:09:09.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:09.597+0000 7f19a9200640 1 -- 172.21.15.28:0/689067674 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19a41a39d0 con 0x7f19a4075d60 2024-05-12T22:09:09.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:09.597+0000 7f199aa00640 1 -- 172.21.15.28:0/689067674 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1994010430 con 0x7f19a4075d60 2024-05-12T22:09:09.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:09.597+0000 7f19a9200640 1 -- 172.21.15.28:0/689067674 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19a41ab260 con 0x7f19a4075d60 2024-05-12T22:09:09.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:09.598+0000 7f19a9200640 1 -- 172.21.15.28:0/689067674 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f19a404f5f0 con 0x7f19a4075d60 2024-05-12T22:09:09.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:09.600+0000 7f199aa00640 1 -- 172.21.15.28:0/689067674 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138060+0+0 (secure 0 0 0) 0x7f19940046c0 con 0x7f19a4075d60 2024-05-12T22:09:09.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:09.602+0000 7f199aa00640 1 --2- 172.21.15.28:0/689067674 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f19880a2ec0 0x7f19880a5380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:09.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:09.602+0000 7f199aa00640 1 -- 172.21.15.28:0/689067674 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f19940cfa20 con 0x7f19a4075d60 2024-05-12T22:09:09.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:09.602+0000 7f199be00640 1 --2- 172.21.15.28:0/689067674 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f19880a2ec0 0x7f19880a5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:09.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:09.603+0000 7f199be00640 1 --2- 172.21.15.28:0/689067674 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f19880a2ec0 0x7f19880a5380 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f19a4078ce0 tx=0x7f198c005f90 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:09:09.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:09.606+0000 7f199aa00640 1 -- 172.21.15.28:0/689067674 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f199409c3c0 con 0x7f19a4075d60 2024-05-12T22:09:09.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:09.834+0000 7f19a9200640 1 -- 172.21.15.28:0/689067674 --> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7f19a410d8d0 con 0x7f19880a2ec0 2024-05-12T22:09:09.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:09.846+0000 7f199aa00640 1 -- 172.21.15.28:0/689067674 <== mgr.25189 v2:172.21.15.28:6800/3177640985 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+298 (secure 0 0 0) 0x7f198c011be0 con 0x7f19880a2ec0 2024-05-12T22:09:09.845 INFO:teuthology.orchestra.run.smithi028.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-05-12T00:00:00", "created": "2024-05-12T22:09:08", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true} 2024-05-12T22:09:09.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:09.848+0000 7f19a9200640 1 -- 172.21.15.28:0/689067674 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f19880a2ec0 msgr2=0x7f19880a5380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:09.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:09.848+0000 7f19a9200640 1 --2- 172.21.15.28:0/689067674 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f19880a2ec0 0x7f19880a5380 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f19a4078ce0 tx=0x7f198c005f90 comp rx=0 tx=0).stop 2024-05-12T22:09:09.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:09.848+0000 7f19a9200640 1 -- 172.21.15.28:0/689067674 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19a4075d60 msgr2=0x7f19a419e6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:09.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:09.848+0000 7f19a9200640 1 --2- 172.21.15.28:0/689067674 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19a4075d60 0x7f19a419e6f0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f199400e9c0 tx=0x7f199400ee80 comp rx=0 tx=0).stop 2024-05-12T22:09:09.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:09.849+0000 7f19a9200640 1 -- 172.21.15.28:0/689067674 shutdown_connections 2024-05-12T22:09:09.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:09.849+0000 7f19a9200640 1 --2- 172.21.15.28:0/689067674 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f19880a2ec0 0x7f19880a5380 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:09.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:09.849+0000 7f19a9200640 1 --2- 172.21.15.28:0/689067674 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f19a41126d0 0x7f19a41a2fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:09.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:09.849+0000 7f19a9200640 1 --2- 172.21.15.28:0/689067674 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f19a4076680 0x7f19a419ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:09.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:09.849+0000 7f19a9200640 1 --2- 172.21.15.28:0/689067674 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19a4075d60 0x7f19a419e6f0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:09.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:09.849+0000 7f19a9200640 1 -- 172.21.15.28:0/689067674 >> 172.21.15.28:0/689067674 conn(0x7f19a40ff560 msgr2=0x7f19a4113340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:09:09.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:09.849+0000 7f19a9200640 1 -- 172.21.15.28:0/689067674 shutdown_connections 2024-05-12T22:09:09.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:09.849+0000 7f19a9200640 1 -- 172.21.15.28:0/689067674 wait complete. 2024-05-12T22:09:09.868 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-05-12T22:09:10.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:10.081+0000 7fc8e8a00640 1 -- 172.21.15.28:0/1614461024 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc8e410e120 msgr2=0x7fc8e4110510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:10.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:10.081+0000 7fc8e8a00640 1 --2- 172.21.15.28:0/1614461024 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc8e410e120 0x7fc8e4110510 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fc8dc009ca0 tx=0x7fc8dc02a950 comp rx=0 tx=0).stop 2024-05-12T22:09:10.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:10.081+0000 7fc8e8a00640 1 -- 172.21.15.28:0/1614461024 shutdown_connections 2024-05-12T22:09:10.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:10.081+0000 7fc8e8a00640 1 --2- 172.21.15.28:0/1614461024 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc8e410e120 0x7fc8e4110510 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:10.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:10.081+0000 7fc8e8a00640 1 --2- 172.21.15.28:0/1614461024 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc8e4074450 0x7fc8e410dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:10.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:10.081+0000 7fc8e8a00640 1 --2- 172.21.15.28:0/1614461024 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc8e4073b30 0x7fc8e4073f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:10.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:10.081+0000 7fc8e8a00640 1 -- 172.21.15.28:0/1614461024 >> 172.21.15.28:0/1614461024 conn(0x7fc8e4078cc0 msgr2=0x7fc8e40790d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:09:10.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:10.081+0000 7fc8e8a00640 1 -- 172.21.15.28:0/1614461024 shutdown_connections 2024-05-12T22:09:10.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:10.081+0000 7fc8e8a00640 1 -- 172.21.15.28:0/1614461024 wait complete. 2024-05-12T22:09:10.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:10.082+0000 7fc8e8a00640 1 Processor -- start 2024-05-12T22:09:10.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:10.082+0000 7fc8e8a00640 1 -- start start 2024-05-12T22:09:10.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:10.082+0000 7fc8e8a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc8e4073b30 0x7fc8e419a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:10.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:10.082+0000 7fc8e8a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc8e4074450 0x7fc8e419a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:10.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:10.082+0000 7fc8e8a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc8e410e120 0x7fc8e419ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:10.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:10.082+0000 7fc8e8a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc8e4113ea0 con 0x7fc8e4073b30 2024-05-12T22:09:10.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:10.082+0000 7fc8e8a00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fc8e4113d20 con 0x7fc8e410e120 2024-05-12T22:09:10.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:10.082+0000 7fc8e8a00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc8e4114020 con 0x7fc8e4074450 2024-05-12T22:09:10.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:10.083+0000 7fc8e1e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc8e4073b30 0x7fc8e419a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:10.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:10.083+0000 7fc8e1e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc8e4073b30 0x7fc8e419a1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49690/0 (socket says 172.21.15.28:49690) 2024-05-12T22:09:10.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:10.083+0000 7fc8e1e00640 1 -- 172.21.15.28:0/3877630090 learned_addr learned my addr 172.21.15.28:0/3877630090 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:09:10.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:10.083+0000 7fc8d9400640 1 --2- 172.21.15.28:0/3877630090 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc8e4074450 0x7fc8e419a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:10.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:10.083+0000 7fc8e2800640 1 --2- 172.21.15.28:0/3877630090 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc8e410e120 0x7fc8e419ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:10.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:10.083+0000 7fc8e1e00640 1 -- 172.21.15.28:0/3877630090 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc8e4074450 msgr2=0x7fc8e419a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:10.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:10.083+0000 7fc8e1e00640 1 --2- 172.21.15.28:0/3877630090 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc8e4074450 0x7fc8e419a6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:10.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:10.083+0000 7fc8e1e00640 1 -- 172.21.15.28:0/3877630090 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc8e410e120 msgr2=0x7fc8e419ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:10.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:10.083+0000 7fc8e1e00640 1 --2- 172.21.15.28:0/3877630090 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc8e410e120 0x7fc8e419ea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:10.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:10.083+0000 7fc8e1e00640 1 -- 172.21.15.28:0/3877630090 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8e419f190 con 0x7fc8e4073b30 2024-05-12T22:09:10.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:10.084+0000 7fc8e1e00640 1 --2- 172.21.15.28:0/3877630090 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc8e4073b30 0x7fc8e419a1a0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fc8d400ea60 tx=0x7fc8d400ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:09:10.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:10.084+0000 7fc8db400640 1 -- 172.21.15.28:0/3877630090 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8d400cde0 con 0x7fc8e4073b30 2024-05-12T22:09:10.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:10.084+0000 7fc8db400640 1 -- 172.21.15.28:0/3877630090 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc8d40035f0 con 0x7fc8e4073b30 2024-05-12T22:09:10.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:10.084+0000 7fc8db400640 1 -- 172.21.15.28:0/3877630090 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8d4010670 con 0x7fc8e4073b30 2024-05-12T22:09:10.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:10.084+0000 7fc8e8a00640 1 -- 172.21.15.28:0/3877630090 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8e419f480 con 0x7fc8e4073b30 2024-05-12T22:09:10.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:10.084+0000 7fc8e8a00640 1 -- 172.21.15.28:0/3877630090 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8e41a6cc0 con 0x7fc8e4073b30 2024-05-12T22:09:10.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:10.085+0000 7fc8e8a00640 1 -- 172.21.15.28:0/3877630090 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8e404f5f0 con 0x7fc8e4073b30 2024-05-12T22:09:10.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:10.087+0000 7fc8db400640 1 -- 172.21.15.28:0/3877630090 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138060+0+0 (secure 0 0 0) 0x7fc8d40037a0 con 0x7fc8e4073b30 2024-05-12T22:09:10.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:10.090+0000 7fc8db400640 1 --2- 172.21.15.28:0/3877630090 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fc8c80a2f40 0x7fc8c80a5400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:10.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:10.090+0000 7fc8db400640 1 -- 172.21.15.28:0/3877630090 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7fc8d40cf9a0 con 0x7fc8e4073b30 2024-05-12T22:09:10.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:10.090+0000 7fc8d9400640 1 --2- 172.21.15.28:0/3877630090 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fc8c80a2f40 0x7fc8c80a5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:10.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:10.091+0000 7fc8d9400640 1 --2- 172.21.15.28:0/3877630090 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fc8c80a2f40 0x7fc8c80a5400 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fc8e4100f30 tx=0x7fc8cc005f90 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:09:10.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:10.094+0000 7fc8db400640 1 -- 172.21.15.28:0/3877630090 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fc8d409c340 con 0x7fc8e4073b30 2024-05-12T22:09:10.182 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:09 smithi186 ceph-mon[34785]: from='client.25400 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-05-12T22:09:10.182 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:09 smithi186 ceph-mon[34785]: pgmap v8: 97 pgs: 97 active+clean; 1.1 MiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 912 B/s wr, 18 op/s 2024-05-12T22:09:10.182 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:09 smithi186 ceph-mon[34785]: from='client.25439 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-05-12T22:09:10.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:09 smithi086 ceph-mon[34297]: from='client.25400 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-05-12T22:09:10.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:09 smithi086 ceph-mon[34297]: pgmap v8: 97 pgs: 97 active+clean; 1.1 MiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 912 B/s wr, 18 op/s 2024-05-12T22:09:10.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:09 smithi086 ceph-mon[34297]: from='client.25439 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-05-12T22:09:10.284 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:09 smithi028 ceph-mon[29847]: from='client.25400 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-05-12T22:09:10.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:09 smithi028 ceph-mon[29847]: pgmap v8: 97 pgs: 97 active+clean; 1.1 MiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 912 B/s wr, 18 op/s 2024-05-12T22:09:10.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:09 smithi028 ceph-mon[29847]: from='client.25439 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-05-12T22:09:10.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:10.323+0000 7fc8e8a00640 1 -- 172.21.15.28:0/3877630090 --> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7fc8e40fed20 con 0x7fc8c80a2f40 2024-05-12T22:09:10.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:10.339+0000 7fc8db400640 1 -- 172.21.15.28:0/3877630090 <== mgr.25189 v2:172.21.15.28:6800/3177640985 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7fc8e40fed20 con 0x7fc8c80a2f40 2024-05-12T22:09:10.338 INFO:teuthology.orchestra.run.smithi028.stdout:/ 1m 6m3h 2024-05-12T22:09:10.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:10.341+0000 7fc8e8a00640 1 -- 172.21.15.28:0/3877630090 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fc8c80a2f40 msgr2=0x7fc8c80a5400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:10.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:10.341+0000 7fc8e8a00640 1 --2- 172.21.15.28:0/3877630090 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fc8c80a2f40 0x7fc8c80a5400 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fc8e4100f30 tx=0x7fc8cc005f90 comp rx=0 tx=0).stop 2024-05-12T22:09:10.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:10.342+0000 7fc8e8a00640 1 -- 172.21.15.28:0/3877630090 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc8e4073b30 msgr2=0x7fc8e419a1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:10.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:10.342+0000 7fc8e8a00640 1 --2- 172.21.15.28:0/3877630090 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc8e4073b30 0x7fc8e419a1a0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fc8d400ea60 tx=0x7fc8d400ef20 comp rx=0 tx=0).stop 2024-05-12T22:09:10.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:10.342+0000 7fc8e8a00640 1 -- 172.21.15.28:0/3877630090 shutdown_connections 2024-05-12T22:09:10.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:10.342+0000 7fc8e8a00640 1 --2- 172.21.15.28:0/3877630090 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fc8c80a2f40 0x7fc8c80a5400 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:10.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:10.342+0000 7fc8e8a00640 1 --2- 172.21.15.28:0/3877630090 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc8e410e120 0x7fc8e419ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:10.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:10.342+0000 7fc8e8a00640 1 --2- 172.21.15.28:0/3877630090 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc8e4074450 0x7fc8e419a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:10.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:10.342+0000 7fc8e8a00640 1 --2- 172.21.15.28:0/3877630090 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc8e4073b30 0x7fc8e419a1a0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:10.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:10.342+0000 7fc8e8a00640 1 -- 172.21.15.28:0/3877630090 >> 172.21.15.28:0/3877630090 conn(0x7fc8e4078cc0 msgr2=0x7fc8e40fe610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:09:10.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:10.342+0000 7fc8e8a00640 1 -- 172.21.15.28:0/3877630090 shutdown_connections 2024-05-12T22:09:10.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:10.342+0000 7fc8e8a00640 1 -- 172.21.15.28:0/3877630090 wait complete. 2024-05-12T22:09:10.361 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > START_TIME' 2024-05-12T22:09:10.453 INFO:teuthology.run_tasks:Running task full_sequential_finally... 2024-05-12T22:09:10.465 INFO:teuthology.run_tasks:Running task workunit... 2024-05-12T22:09:10.478 INFO:tasks.workunit:Pulling workunits from ref c50db29d375ad9ca1a200449f9347b89e25fb278 2024-05-12T22:09:10.479 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-05-12T22:09:10.479 INFO:tasks.workunit:timeout=3h 2024-05-12T22:09:10.479 INFO:tasks.workunit:cleanup=True 2024-05-12T22:09:10.479 DEBUG:teuthology.orchestra.run.smithi028:> stat -- /home/ubuntu/cephtest/mnt.0 2024-05-12T22:09:10.539 INFO:teuthology.orchestra.run.smithi028.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-05-12T22:09:10.539 INFO:teuthology.orchestra.run.smithi028.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-05-12T22:09:10.539 INFO:teuthology.orchestra.run.smithi028.stdout:Device: 72h/114d Inode: 1099511628283 Links: 2 2024-05-12T22:09:10.539 INFO:teuthology.orchestra.run.smithi028.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-12T22:09:10.539 INFO:teuthology.orchestra.run.smithi028.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-05-12T22:09:10.539 INFO:teuthology.orchestra.run.smithi028.stdout:Access: 2024-05-12 22:06:41.758957534 +0000 2024-05-12T22:09:10.539 INFO:teuthology.orchestra.run.smithi028.stdout:Modify: 2024-05-12 22:06:41.758957534 +0000 2024-05-12T22:09:10.539 INFO:teuthology.orchestra.run.smithi028.stdout:Change: 2024-05-12 22:08:57.107566454 +0000 2024-05-12T22:09:10.539 INFO:teuthology.orchestra.run.smithi028.stdout: Birth: 2024-05-12 22:06:41.758957534 +0000 2024-05-12T22:09:10.539 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-05-12T22:09:10.540 DEBUG:teuthology.orchestra.run.smithi028:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-05-12T22:09:10.644 DEBUG:teuthology.orchestra.run.smithi028:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout c50db29d375ad9ca1a200449f9347b89e25fb278 2024-05-12T22:09:10.706 INFO:tasks.workunit.client.0.smithi028.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-05-12T22:09:11.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:10 smithi086 ceph-mon[34297]: from='client.15327 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-05-12T22:09:11.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:10 smithi028 ceph-mon[29847]: from='client.15327 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-05-12T22:09:11.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:10 smithi186 ceph-mon[34785]: from='client.15327 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-05-12T22:09:12.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:11 smithi186 ceph-mon[34785]: from='client.15333 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-12T22:09:12.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:11 smithi186 ceph-mon[34785]: pgmap v9: 97 pgs: 97 active+clean; 1.3 MiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 84 KiB/s wr, 49 op/s 2024-05-12T22:09:12.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:11 smithi086 ceph-mon[34297]: from='client.15333 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-12T22:09:12.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:11 smithi086 ceph-mon[34297]: pgmap v9: 97 pgs: 97 active+clean; 1.3 MiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 84 KiB/s wr, 49 op/s 2024-05-12T22:09:12.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:11 smithi028 ceph-mon[29847]: from='client.15333 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-12T22:09:12.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:11 smithi028 ceph-mon[29847]: pgmap v9: 97 pgs: 97 active+clean; 1.3 MiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 84 KiB/s wr, 49 op/s 2024-05-12T22:09:13.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:12 smithi186 ceph-mon[34785]: scrub summary: idle 2024-05-12T22:09:13.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:12 smithi086 ceph-mon[34297]: scrub summary: idle 2024-05-12T22:09:13.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:12 smithi028 ceph-mon[29847]: scrub summary: idle 2024-05-12T22:09:14.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:13 smithi086 ceph-mon[34297]: pgmap v10: 97 pgs: 97 active+clean; 1.3 MiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 77 KiB/s wr, 45 op/s 2024-05-12T22:09:14.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:13 smithi028 ceph-mon[29847]: pgmap v10: 97 pgs: 97 active+clean; 1.3 MiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 77 KiB/s wr, 45 op/s 2024-05-12T22:09:14.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:13 smithi186 ceph-mon[34785]: pgmap v10: 97 pgs: 97 active+clean; 1.3 MiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 77 KiB/s wr, 45 op/s 2024-05-12T22:09:16.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:15 smithi086 ceph-mon[34297]: pgmap v11: 97 pgs: 97 active+clean; 1.3 MiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 77 KiB/s wr, 45 op/s 2024-05-12T22:09:16.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:15 smithi028 ceph-mon[29847]: pgmap v11: 97 pgs: 97 active+clean; 1.3 MiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 77 KiB/s wr, 45 op/s 2024-05-12T22:09:16.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:15 smithi186 ceph-mon[34785]: pgmap v11: 97 pgs: 97 active+clean; 1.3 MiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 77 KiB/s wr, 45 op/s 2024-05-12T22:09:18.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:17 smithi086 ceph-mon[34297]: pgmap v12: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 77 KiB/s wr, 50 op/s 2024-05-12T22:09:18.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:17 smithi028 ceph-mon[29847]: pgmap v12: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 77 KiB/s wr, 50 op/s 2024-05-12T22:09:18.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:17 smithi186 ceph-mon[34785]: pgmap v12: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 77 KiB/s wr, 50 op/s 2024-05-12T22:09:20.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:19 smithi086 ceph-mon[34297]: pgmap v13: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 77 KiB/s wr, 37 op/s 2024-05-12T22:09:20.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:19 smithi028 ceph-mon[29847]: pgmap v13: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 77 KiB/s wr, 37 op/s 2024-05-12T22:09:20.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:19 smithi186 ceph-mon[34785]: pgmap v13: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 77 KiB/s wr, 37 op/s 2024-05-12T22:09:22.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:21 smithi086 ceph-mon[34297]: pgmap v14: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 77 KiB/s wr, 39 op/s 2024-05-12T22:09:22.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:21 smithi028 ceph-mon[29847]: pgmap v14: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 77 KiB/s wr, 39 op/s 2024-05-12T22:09:22.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:21 smithi186 ceph-mon[34785]: pgmap v14: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 77 KiB/s wr, 39 op/s 2024-05-12T22:09:24.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:23 smithi086 ceph-mon[34297]: pgmap v15: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 682 B/s wr, 7 op/s 2024-05-12T22:09:24.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:23 smithi028 ceph-mon[29847]: pgmap v15: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 682 B/s wr, 7 op/s 2024-05-12T22:09:24.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:23 smithi186 ceph-mon[34785]: pgmap v15: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 682 B/s wr, 7 op/s 2024-05-12T22:09:26.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:25 smithi028 ceph-mon[29847]: pgmap v16: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 682 B/s wr, 7 op/s 2024-05-12T22:09:26.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:25 smithi186 ceph-mon[34785]: pgmap v16: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 682 B/s wr, 7 op/s 2024-05-12T22:09:26.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:25 smithi086 ceph-mon[34297]: pgmap v16: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 682 B/s wr, 7 op/s 2024-05-12T22:09:28.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:28 smithi028 ceph-mon[29847]: pgmap v17: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 682 B/s wr, 10 op/s 2024-05-12T22:09:28.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:28 smithi186 ceph-mon[34785]: pgmap v17: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 682 B/s wr, 10 op/s 2024-05-12T22:09:28.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:28 smithi086 ceph-mon[34297]: pgmap v17: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 682 B/s wr, 10 op/s 2024-05-12T22:09:30.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:30 smithi028 ceph-mon[29847]: pgmap v18: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:09:30.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:30 smithi186 ceph-mon[34785]: pgmap v18: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:09:30.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:30 smithi086 ceph-mon[34297]: pgmap v18: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:09:32.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:32 smithi028 ceph-mon[29847]: pgmap v19: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-12T22:09:32.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:32 smithi186 ceph-mon[34785]: pgmap v19: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-12T22:09:32.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:32 smithi086 ceph-mon[34297]: pgmap v19: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-12T22:09:34.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:34 smithi028 ceph-mon[29847]: pgmap v20: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:09:34.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:34 smithi186 ceph-mon[34785]: pgmap v20: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:09:34.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:34 smithi086 ceph-mon[34297]: pgmap v20: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:09:36.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:36 smithi028 ceph-mon[29847]: pgmap v21: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-12T22:09:36.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:36 smithi186 ceph-mon[34785]: pgmap v21: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-12T22:09:36.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:36 smithi086 ceph-mon[34297]: pgmap v21: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-12T22:09:38.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:38 smithi028 ceph-mon[29847]: pgmap v22: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-12T22:09:38.294 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:38 smithi086 ceph-mon[34297]: pgmap v22: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-12T22:09:38.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:38 smithi186 ceph-mon[34785]: pgmap v22: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-12T22:09:38.715 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-12T22:09:38.715 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-12T22:09:38.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.949+0000 7f12b1c00640 1 -- 172.21.15.28:0/1375557981 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f12ac107dd0 msgr2=0x7f12ac108230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:38.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.949+0000 7f12b1c00640 1 --2- 172.21.15.28:0/1375557981 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f12ac107dd0 0x7f12ac108230 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f129c009be0 tx=0x7f129c02a750 comp rx=0 tx=0).stop 2024-05-12T22:09:38.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.949+0000 7f12b1c00640 1 -- 172.21.15.28:0/1375557981 shutdown_connections 2024-05-12T22:09:38.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.949+0000 7f12b1c00640 1 --2- 172.21.15.28:0/1375557981 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f12ac108770 0x7f12ac114b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:38.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.949+0000 7f12b1c00640 1 --2- 172.21.15.28:0/1375557981 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f12ac107dd0 0x7f12ac108230 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:38.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.949+0000 7f12b1c00640 1 --2- 172.21.15.28:0/1375557981 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f12ac10ddf0 0x7f12ac10e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:38.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.949+0000 7f12b1c00640 1 -- 172.21.15.28:0/1375557981 >> 172.21.15.28:0/1375557981 conn(0x7f12ac077650 msgr2=0x7f12ac106650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:09:38.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.949+0000 7f12b1c00640 1 -- 172.21.15.28:0/1375557981 shutdown_connections 2024-05-12T22:09:38.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.950+0000 7f12b1c00640 1 -- 172.21.15.28:0/1375557981 wait complete. 2024-05-12T22:09:38.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.951+0000 7f12b1c00640 1 Processor -- start 2024-05-12T22:09:38.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.951+0000 7f12b1c00640 1 -- start start 2024-05-12T22:09:38.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.951+0000 7f12b1c00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f12ac107dd0 0x7f12ac19e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:38.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.951+0000 7f12b1c00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f12ac108770 0x7f12ac19ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:38.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.951+0000 7f12b1c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f12ac10ddf0 0x7f12ac1a2e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:38.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.951+0000 7f12b1c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f12ac1180b0 con 0x7f12ac10ddf0 2024-05-12T22:09:38.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.951+0000 7f12b1c00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f12ac117f30 con 0x7f12ac107dd0 2024-05-12T22:09:38.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.951+0000 7f12b1c00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f12ac118230 con 0x7f12ac108770 2024-05-12T22:09:38.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.952+0000 7f12abe00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f12ac107dd0 0x7f12ac19e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:38.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.952+0000 7f12abe00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f12ac107dd0 0x7f12ac19e590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:34122/0 (socket says 172.21.15.28:34122) 2024-05-12T22:09:38.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.952+0000 7f12ab400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f12ac108770 0x7f12ac19ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:38.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.952+0000 7f12abe00640 1 -- 172.21.15.28:0/305408994 learned_addr learned my addr 172.21.15.28:0/305408994 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:09:38.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.952+0000 7f12b1200640 1 --2- 172.21.15.28:0/305408994 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f12ac10ddf0 0x7f12ac1a2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:38.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.952+0000 7f12ab400640 1 -- 172.21.15.28:0/305408994 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f12ac107dd0 msgr2=0x7f12ac19e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:38.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.952+0000 7f12ab400640 1 --2- 172.21.15.28:0/305408994 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f12ac107dd0 0x7f12ac19e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:38.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.952+0000 7f12ab400640 1 -- 172.21.15.28:0/305408994 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f12ac10ddf0 msgr2=0x7f12ac1a2e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:38.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.952+0000 7f12ab400640 1 --2- 172.21.15.28:0/305408994 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f12ac10ddf0 0x7f12ac1a2e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:38.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.952+0000 7f12ab400640 1 -- 172.21.15.28:0/305408994 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12ac1a3530 con 0x7f12ac108770 2024-05-12T22:09:38.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.952+0000 7f12b1200640 1 --2- 172.21.15.28:0/305408994 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f12ac10ddf0 0x7f12ac1a2e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:09:38.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.952+0000 7f12abe00640 1 --2- 172.21.15.28:0/305408994 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f12ac107dd0 0x7f12ac19e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:09:38.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.953+0000 7f12ab400640 1 --2- 172.21.15.28:0/305408994 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f12ac108770 0x7f12ac19ead0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f129c02ac70 tx=0x7f129c0305d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:09:38.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.953+0000 7f12a8c00640 1 -- 172.21.15.28:0/305408994 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f129c041020 con 0x7f12ac108770 2024-05-12T22:09:38.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.953+0000 7f12b1c00640 1 -- 172.21.15.28:0/305408994 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12ac1a37c0 con 0x7f12ac108770 2024-05-12T22:09:38.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.953+0000 7f12a8c00640 1 -- 172.21.15.28:0/305408994 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f129c003e50 con 0x7f12ac108770 2024-05-12T22:09:38.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.953+0000 7f12a8c00640 1 -- 172.21.15.28:0/305408994 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f129c036040 con 0x7f12ac108770 2024-05-12T22:09:38.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.953+0000 7f12b1c00640 1 -- 172.21.15.28:0/305408994 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12ac1ab0b0 con 0x7f12ac108770 2024-05-12T22:09:38.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.956+0000 7f12b600d740 1 Processor -- start 2024-05-12T22:09:38.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.957+0000 7f12b600d740 1 -- start start 2024-05-12T22:09:38.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.957+0000 7f12a8c00640 1 -- 172.21.15.28:0/305408994 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 25) ==== 138060+0+0 (secure 0 0 0) 0x7f129c044020 con 0x7f12ac108770 2024-05-12T22:09:38.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.957+0000 7f12b600d740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558c7c58ddd0 0x558c7c58e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:38.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.957+0000 7f12b600d740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558c7c58e6f0 0x558c7c598880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:38.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.957+0000 7f12b600d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c7c598dc0 0x558c7c59b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:38.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.957+0000 7f12b600d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558c7c5562a0 con 0x558c7c598dc0 2024-05-12T22:09:38.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.957+0000 7f12b600d740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x558c7c4f2b90 con 0x558c7c58e6f0 2024-05-12T22:09:38.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.957+0000 7f12b600d740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x558c7c49a5b0 con 0x558c7c58ddd0 2024-05-12T22:09:38.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.957+0000 7f12b1200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558c7c58ddd0 0x558c7c58e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:38.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.957+0000 7f12ab400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c7c598dc0 0x558c7c59b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:38.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.957+0000 7f12b1200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558c7c58ddd0 0x558c7c58e1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:34620/0 (socket says 172.21.15.28:34620) 2024-05-12T22:09:38.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.957+0000 7f12abe00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558c7c58e6f0 0x558c7c598880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:38.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.957+0000 7f12b1200640 1 -- 172.21.15.28:0/395460109 learned_addr learned my addr 172.21.15.28:0/395460109 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:09:38.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.958+0000 7f12b1200640 1 -- 172.21.15.28:0/395460109 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558c7c58e6f0 msgr2=0x558c7c598880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:09:38.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.958+0000 7f12b1200640 1 --2- 172.21.15.28:0/395460109 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558c7c58e6f0 0x558c7c598880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:38.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.958+0000 7f12b1200640 1 -- 172.21.15.28:0/395460109 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c7c598dc0 msgr2=0x558c7c59b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:09:38.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.958+0000 7f12b1200640 1 --2- 172.21.15.28:0/395460109 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c7c598dc0 0x558c7c59b1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:38.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.958+0000 7f12b1200640 1 -- 172.21.15.28:0/395460109 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558c7c2d07d0 con 0x558c7c58ddd0 2024-05-12T22:09:38.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.958+0000 7f12b1200640 1 --2- 172.21.15.28:0/395460109 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558c7c58ddd0 0x558c7c58e1b0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f12a4005ab0 tx=0x7f12a40117b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e03a10ee8776a894 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:09:38.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.958+0000 7f12a1600640 1 -- 172.21.15.28:0/395460109 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12a4004030 con 0x558c7c58ddd0 2024-05-12T22:09:38.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.958+0000 7f12a8c00640 1 --2- 172.21.15.28:0/305408994 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f12900a2e20 0x7f12900a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:38.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.959+0000 7f12a1600640 1 -- 172.21.15.28:0/395460109 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f12a40041d0 con 0x558c7c58ddd0 2024-05-12T22:09:38.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.959+0000 7f12a1600640 1 -- 172.21.15.28:0/395460109 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12a40127d0 con 0x558c7c58ddd0 2024-05-12T22:09:38.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.959+0000 7f12a8c00640 1 -- 172.21.15.28:0/305408994 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f129c0f0120 con 0x7f12ac108770 2024-05-12T22:09:38.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.959+0000 7f12abe00640 1 --2- 172.21.15.28:0/305408994 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f12900a2e20 0x7f12900a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:38.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.959+0000 7f12b600d740 1 -- 172.21.15.28:0/395460109 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558c7c58ddd0 msgr2=0x558c7c58e1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:38.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.959+0000 7f12b600d740 1 --2- 172.21.15.28:0/395460109 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558c7c58ddd0 0x558c7c58e1b0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f12a4005ab0 tx=0x7f12a40117b0 comp rx=0 tx=0).stop 2024-05-12T22:09:38.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.959+0000 7f12b600d740 1 -- 172.21.15.28:0/395460109 shutdown_connections 2024-05-12T22:09:38.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.959+0000 7f12b600d740 1 --2- 172.21.15.28:0/395460109 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c7c598dc0 0x558c7c59b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:38.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.959+0000 7f12b600d740 1 --2- 172.21.15.28:0/395460109 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558c7c58e6f0 0x558c7c598880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:38.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.959+0000 7f12b600d740 1 --2- 172.21.15.28:0/395460109 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558c7c58ddd0 0x558c7c58e1b0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:38.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.959+0000 7f12b600d740 1 -- 172.21.15.28:0/395460109 >> 172.21.15.28:0/395460109 conn(0x558c7c58a0f0 msgr2=0x558c7c58a4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:09:38.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.960+0000 7f12b600d740 1 -- 172.21.15.28:0/395460109 shutdown_connections 2024-05-12T22:09:38.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.960+0000 7f12b600d740 1 -- 172.21.15.28:0/395460109 wait complete. 2024-05-12T22:09:38.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.960+0000 7f12abe00640 1 --2- 172.21.15.28:0/305408994 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f12900a2e20 0x7f12900a52e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f1294009620 tx=0x7f1294005f90 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:09:38.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.960+0000 7f12b600d740 1 -- start start 2024-05-12T22:09:38.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.960+0000 7f12b600d740 1 Processor -- start 2024-05-12T22:09:38.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.961+0000 7f12b600d740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558c7c58ddd0 0x558c7c5986c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:38.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.961+0000 7f12b600d740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558c7c58e6f0 0x558c7c5959b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:38.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.961+0000 7f12b600d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c7c598dc0 0x558c7c595ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:38.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.961+0000 7f12b600d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558c7c4ad9f0 con 0x558c7c598dc0 2024-05-12T22:09:38.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.961+0000 7f12b600d740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x558c7c4eeb50 con 0x558c7c58e6f0 2024-05-12T22:09:38.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.961+0000 7f12b600d740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x558c7c59bbb0 con 0x558c7c58ddd0 2024-05-12T22:09:38.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.961+0000 7f12b600d740 10 client.0 _ll_register_callbacks cb 0x558c7c205be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:09:38.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.961+0000 7f12b1200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c7c598dc0 0x558c7c595ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:38.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.961+0000 7f12b1200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c7c598dc0 0x558c7c595ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50764/0 (socket says 172.21.15.28:50764) 2024-05-12T22:09:38.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.961+0000 7f12b1200640 1 -- 172.21.15.28:0/640382533 learned_addr learned my addr 172.21.15.28:0/640382533 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:09:38.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.961+0000 7f12abe00640 1 --2- 172.21.15.28:0/640382533 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558c7c58ddd0 0x558c7c5986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:38.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.961+0000 7f12ab400640 1 --2- 172.21.15.28:0/640382533 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558c7c58e6f0 0x558c7c5959b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:38.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.962+0000 7f12ab400640 1 -- 172.21.15.28:0/640382533 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558c7c58ddd0 msgr2=0x558c7c5986c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:09:38.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.962+0000 7f12ab400640 1 --2- 172.21.15.28:0/640382533 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558c7c58ddd0 0x558c7c5986c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:38.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.962+0000 7f12ab400640 1 -- 172.21.15.28:0/640382533 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c7c598dc0 msgr2=0x558c7c595ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:09:38.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.962+0000 7f12ab400640 1 --2- 172.21.15.28:0/640382533 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c7c598dc0 0x558c7c595ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:38.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.962+0000 7f12ab400640 1 -- 172.21.15.28:0/640382533 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558c7c2063f0 con 0x558c7c58e6f0 2024-05-12T22:09:38.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.962+0000 7f12ab400640 1 --2- 172.21.15.28:0/640382533 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558c7c58e6f0 0x558c7c5959b0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f129c0ed700 tx=0x7f129c02a750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c14ca03a9627972d server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:09:38.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.962+0000 7f129a000640 10 client.0 ms_handle_connect on v2:172.21.15.86:3300/0 2024-05-12T22:09:38.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.962+0000 7f129a000640 1 -- 172.21.15.28:0/640382533 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f129c0b8770 con 0x558c7c58e6f0 2024-05-12T22:09:38.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.962+0000 7f12b600d740 1 -- 172.21.15.28:0/640382533 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f12a40041d0 con 0x558c7c58e6f0 2024-05-12T22:09:38.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.962+0000 7f129a000640 1 -- 172.21.15.28:0/640382533 <== mon.2 v2:172.21.15.86:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f129c0c4cb0 con 0x558c7c58e6f0 2024-05-12T22:09:38.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.962+0000 7f129a000640 1 -- 172.21.15.28:0/640382533 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f129c0bbe30 con 0x558c7c58e6f0 2024-05-12T22:09:38.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.963+0000 7f12b600d740 10 client.25463 fetch_fsmap learned FSMap version 24 2024-05-12T22:09:38.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.963+0000 7f12b600d740 1 -- 172.21.15.28:0/640382533 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x558c7c1c60f0 con 0x558c7c58e6f0 2024-05-12T22:09:38.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.963+0000 7f129a000640 1 -- 172.21.15.28:0/640382533 <== mon.2 v2:172.21.15.86:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f12a40041d0 con 0x558c7c58e6f0 2024-05-12T22:09:38.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.963+0000 7f129a000640 1 -- 172.21.15.28:0/640382533 <== mon.2 v2:172.21.15.86:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f129c0d51f0 con 0x558c7c58e6f0 2024-05-12T22:09:38.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.963+0000 7f12b600d740 10 client.25463 fetch_fsmap finished waiting for FSMap version 24 2024-05-12T22:09:38.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.963+0000 7f12b600d740 10 client.25463 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:09:38.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.963+0000 7f12b600d740 20 client.25463 populate_metadata read hostname 'smithi028' 2024-05-12T22:09:38.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.963+0000 7f12b600d740 1 --2- 172.21.15.28:0/640382533 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x558c7c5a87d0 0x558c7c5aac20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:38.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.963+0000 7f12b600d740 4 client.25463 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-12T22:09:38.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.963+0000 7f12b600d740 1 -- 172.21.15.28:0/640382533 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558c7c5002c0 con 0x558c7c5a87d0 2024-05-12T22:09:38.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.963+0000 7f12abe00640 1 --2- 172.21.15.28:0/640382533 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x558c7c5a87d0 0x558c7c5aac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:38.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.964+0000 7f12abe00640 1 --2- 172.21.15.28:0/640382533 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x558c7c5a87d0 0x558c7c5aac20 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c1e84301e7713f5 server_cookie=cca4e9170af5880d in_seq=0 out_seq=0 2024-05-12T22:09:38.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.964+0000 7f129a000640 10 client.25463 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:09:38.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.965+0000 7f129a000640 1 -- 172.21.15.28:0/640382533 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558c7c5002c0 con 0x558c7c5a87d0 2024-05-12T22:09:38.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.965+0000 7f129a000640 10 client.25463 handle_command_reply: tid=0 2024-05-12T22:09:38.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.965+0000 7f12b600d740 1 client.25463 shutdown 2024-05-12T22:09:38.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.965+0000 7f12b600d740 1 -- 172.21.15.28:0/640382533 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558c7c58e6f0 msgr2=0x558c7c5959b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:38.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.965+0000 7f12b600d740 1 --2- 172.21.15.28:0/640382533 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558c7c58e6f0 0x558c7c5959b0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f129c0ed700 tx=0x7f129c02a750 comp rx=0 tx=0).stop 2024-05-12T22:09:38.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.965+0000 7f12b600d740 1 -- 172.21.15.28:0/640382533 shutdown_connections 2024-05-12T22:09:38.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.965+0000 7f12b600d740 1 --2- 172.21.15.28:0/640382533 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x558c7c5a87d0 0x558c7c5aac20 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:38.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.965+0000 7f12b600d740 1 --2- 172.21.15.28:0/640382533 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c7c598dc0 0x558c7c595ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:38.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.965+0000 7f12b600d740 1 --2- 172.21.15.28:0/640382533 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558c7c58e6f0 0x558c7c5959b0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:38.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.965+0000 7f129a000640 0 client.25463 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:09:38.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.965+0000 7f12b600d740 1 --2- 172.21.15.28:0/640382533 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558c7c58ddd0 0x558c7c5986c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:38.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.965+0000 7f12b600d740 1 -- 172.21.15.28:0/640382533 >> 172.21.15.28:0/640382533 conn(0x558c7c58a0f0 msgr2=0x558c7c58fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:09:38.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.965+0000 7f12b600d740 1 -- 172.21.15.28:0/640382533 shutdown_connections 2024-05-12T22:09:38.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.965+0000 7f12b600d740 1 -- 172.21.15.28:0/640382533 wait complete. 2024-05-12T22:09:38.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.966+0000 7f12b600d740 20 client.25463 trim_cache size 0 max 16384 2024-05-12T22:09:38.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.985+0000 7f12b600d740 1 Processor -- start 2024-05-12T22:09:38.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.985+0000 7f12b600d740 1 -- start start 2024-05-12T22:09:38.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.985+0000 7f12b600d740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558c7c58ddd0 0x558c7c5b04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:38.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.985+0000 7f12b600d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c7c58e6f0 0x558c7c5b0a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:38.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.985+0000 7f12b600d740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558c7c598dc0 0x558c7c592c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:38.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.985+0000 7f12b600d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f12ac1b60d0 con 0x558c7c58e6f0 2024-05-12T22:09:38.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.985+0000 7f12b600d740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f12ac1b39a0 con 0x558c7c598dc0 2024-05-12T22:09:38.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.985+0000 7f12b600d740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f12ac1b17f0 con 0x558c7c58ddd0 2024-05-12T22:09:38.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.985+0000 7f12ab400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c7c58e6f0 0x558c7c5b0a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:38.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.986+0000 7f12ab400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c7c58e6f0 0x558c7c5b0a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50768/0 (socket says 172.21.15.28:50768) 2024-05-12T22:09:38.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.986+0000 7f12abe00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558c7c58ddd0 0x558c7c5b04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:38.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.986+0000 7f12ab400640 1 -- 172.21.15.28:0/1148207909 learned_addr learned my addr 172.21.15.28:0/1148207909 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:09:38.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.986+0000 7f12b1200640 1 --2- 172.21.15.28:0/1148207909 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558c7c598dc0 0x558c7c592c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:38.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.986+0000 7f12ab400640 1 -- 172.21.15.28:0/1148207909 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558c7c58ddd0 msgr2=0x558c7c5b04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:09:38.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.986+0000 7f12ab400640 1 --2- 172.21.15.28:0/1148207909 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558c7c58ddd0 0x558c7c5b04f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:38.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.986+0000 7f12ab400640 1 -- 172.21.15.28:0/1148207909 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558c7c598dc0 msgr2=0x558c7c592c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:09:38.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.986+0000 7f12ab400640 1 --2- 172.21.15.28:0/1148207909 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558c7c598dc0 0x558c7c592c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:38.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.986+0000 7f12ab400640 1 -- 172.21.15.28:0/1148207909 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558c7c585ab0 con 0x558c7c58e6f0 2024-05-12T22:09:38.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.986+0000 7f12abe00640 1 --2- 172.21.15.28:0/1148207909 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558c7c58ddd0 0x558c7c5b04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:09:38.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.986+0000 7f12b1200640 1 --2- 172.21.15.28:0/1148207909 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558c7c598dc0 0x558c7c592c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:09:38.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.986+0000 7f12ab400640 1 --2- 172.21.15.28:0/1148207909 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c7c58e6f0 0x558c7c5b0a30 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f129c0ba780 tx=0x7f129c0ba9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a22361a0f9324f0b server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:09:38.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.987+0000 7f12a0c00640 1 -- 172.21.15.28:0/1148207909 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f129c0bc010 con 0x558c7c58e6f0 2024-05-12T22:09:38.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.987+0000 7f12a0c00640 1 -- 172.21.15.28:0/1148207909 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f129c0f3bf0 con 0x558c7c58e6f0 2024-05-12T22:09:38.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.987+0000 7f12a0c00640 1 -- 172.21.15.28:0/1148207909 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f129c0bc2a0 con 0x558c7c58e6f0 2024-05-12T22:09:38.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.987+0000 7f12b600d740 1 -- 172.21.15.28:0/1148207909 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c7c58e6f0 msgr2=0x558c7c5b0a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:38.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.987+0000 7f12b600d740 1 --2- 172.21.15.28:0/1148207909 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c7c58e6f0 0x558c7c5b0a30 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f129c0ba780 tx=0x7f129c0ba9e0 comp rx=0 tx=0).stop 2024-05-12T22:09:38.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.987+0000 7f12b600d740 1 -- 172.21.15.28:0/1148207909 shutdown_connections 2024-05-12T22:09:38.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.987+0000 7f12b600d740 1 --2- 172.21.15.28:0/1148207909 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558c7c598dc0 0x558c7c592c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:38.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.987+0000 7f12b600d740 1 --2- 172.21.15.28:0/1148207909 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c7c58e6f0 0x558c7c5b0a30 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:38.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.987+0000 7f12b600d740 1 --2- 172.21.15.28:0/1148207909 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558c7c58ddd0 0x558c7c5b04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:38.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.987+0000 7f12b600d740 1 -- 172.21.15.28:0/1148207909 >> 172.21.15.28:0/1148207909 conn(0x558c7c58a0f0 msgr2=0x558c7c5a38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:09:38.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.988+0000 7f12b600d740 1 -- 172.21.15.28:0/1148207909 shutdown_connections 2024-05-12T22:09:38.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.988+0000 7f12b600d740 1 -- 172.21.15.28:0/1148207909 wait complete. 2024-05-12T22:09:38.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.988+0000 7f12b600d740 1 -- start start 2024-05-12T22:09:38.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.988+0000 7f12b600d740 1 Processor -- start 2024-05-12T22:09:38.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.989+0000 7f12b600d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c7c58ddd0 0x558c7c5a12a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:38.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.989+0000 7f12b600d740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558c7c58e6f0 0x558c7c59e630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:38.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.989+0000 7f12b600d740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558c7c598dc0 0x558c7c59ec40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:38.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.989+0000 7f12b600d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f12ac1b1e60 con 0x558c7c58ddd0 2024-05-12T22:09:38.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.989+0000 7f12b600d740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f12ac1b32d0 con 0x558c7c598dc0 2024-05-12T22:09:38.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.989+0000 7f12b600d740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f12ac1b2530 con 0x558c7c58e6f0 2024-05-12T22:09:38.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.989+0000 7f12b600d740 10 client.0 _ll_register_callbacks cb 0x7f129c009280 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:09:38.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.989+0000 7f12ab400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558c7c58e6f0 0x558c7c59e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:38.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.989+0000 7f12ab400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558c7c58e6f0 0x558c7c59e630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:34646/0 (socket says 172.21.15.28:34646) 2024-05-12T22:09:38.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.989+0000 7f12ab400640 1 -- 172.21.15.28:0/962836929 learned_addr learned my addr 172.21.15.28:0/962836929 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:09:38.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.989+0000 7f12abe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c7c58ddd0 0x558c7c5a12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:38.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.989+0000 7f12b1200640 1 --2- 172.21.15.28:0/962836929 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558c7c598dc0 0x558c7c59ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:38.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.989+0000 7f12ab400640 1 -- 172.21.15.28:0/962836929 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558c7c598dc0 msgr2=0x558c7c59ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:09:38.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.989+0000 7f12ab400640 1 --2- 172.21.15.28:0/962836929 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558c7c598dc0 0x558c7c59ec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:38.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.989+0000 7f12ab400640 1 -- 172.21.15.28:0/962836929 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c7c58ddd0 msgr2=0x558c7c5a12a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:09:38.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.989+0000 7f12ab400640 1 --2- 172.21.15.28:0/962836929 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c7c58ddd0 0x558c7c5a12a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:38.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.989+0000 7f12ab400640 1 -- 172.21.15.28:0/962836929 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558c7c59f410 con 0x558c7c58e6f0 2024-05-12T22:09:38.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.990+0000 7f12ab400640 1 --2- 172.21.15.28:0/962836929 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558c7c58e6f0 0x558c7c59e630 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f129c009b30 tx=0x7f129c02a750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=47cdff8d2b525602 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:09:38.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.990+0000 7f129a000640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-12T22:09:38.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.990+0000 7f129a000640 1 -- 172.21.15.28:0/962836929 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f129c0bc2a0 con 0x558c7c58e6f0 2024-05-12T22:09:38.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.990+0000 7f12b600d740 1 -- 172.21.15.28:0/962836929 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f129c0f3bf0 con 0x558c7c58e6f0 2024-05-12T22:09:38.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.990+0000 7f129a000640 1 -- 172.21.15.28:0/962836929 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f129c004d40 con 0x558c7c58e6f0 2024-05-12T22:09:38.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.990+0000 7f129a000640 1 -- 172.21.15.28:0/962836929 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f129c0b8770 con 0x558c7c58e6f0 2024-05-12T22:09:38.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.990+0000 7f12b600d740 10 client.25441 fetch_fsmap learned FSMap version 24 2024-05-12T22:09:38.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.990+0000 7f12b600d740 1 -- 172.21.15.28:0/962836929 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x558c7c59f410 con 0x558c7c58e6f0 2024-05-12T22:09:38.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.990+0000 7f129a000640 1 -- 172.21.15.28:0/962836929 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f129c0f3bf0 con 0x558c7c58e6f0 2024-05-12T22:09:38.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.990+0000 7f129a000640 1 -- 172.21.15.28:0/962836929 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f129c0872b0 con 0x558c7c58e6f0 2024-05-12T22:09:38.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.991+0000 7f12b600d740 10 client.25441 fetch_fsmap finished waiting for FSMap version 24 2024-05-12T22:09:38.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.991+0000 7f12b600d740 10 client.25441 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:09:38.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.991+0000 7f12b600d740 20 client.25441 populate_metadata read hostname 'smithi028' 2024-05-12T22:09:38.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.991+0000 7f12b600d740 1 --2- 172.21.15.28:0/962836929 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x558c7c60f100 0x558c7c6114f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:38.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.991+0000 7f12b600d740 4 client.25441 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-12T22:09:38.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.991+0000 7f12b600d740 1 -- 172.21.15.28:0/962836929 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "scrub status"}) -- 0x558c7c4fa610 con 0x558c7c60f100 2024-05-12T22:09:38.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.991+0000 7f12abe00640 1 --2- 172.21.15.28:0/962836929 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x558c7c60f100 0x558c7c6114f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:38.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.991+0000 7f12abe00640 1 --2- 172.21.15.28:0/962836929 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x558c7c60f100 0x558c7c6114f0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d3868763fc2eb85e server_cookie=49c47c91de8c0a78 in_seq=0 out_seq=0 2024-05-12T22:09:38.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.991+0000 7f129a000640 10 client.25441 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:09:38.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.992+0000 7f129a000640 1 -- 172.21.15.28:0/962836929 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x558c7c4fa610 con 0x558c7c60f100 2024-05-12T22:09:38.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.992+0000 7f129a000640 10 client.25441 handle_command_reply: tid=0 2024-05-12T22:09:38.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.992+0000 7f12b600d740 1 client.25441 shutdown 2024-05-12T22:09:38.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.992+0000 7f12b600d740 1 -- 172.21.15.28:0/962836929 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558c7c58e6f0 msgr2=0x558c7c59e630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:38.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.992+0000 7f12b600d740 1 --2- 172.21.15.28:0/962836929 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558c7c58e6f0 0x558c7c59e630 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f129c009b30 tx=0x7f129c02a750 comp rx=0 tx=0).stop 2024-05-12T22:09:38.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.992+0000 7f12b600d740 1 -- 172.21.15.28:0/962836929 shutdown_connections 2024-05-12T22:09:38.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.992+0000 7f12b600d740 1 --2- 172.21.15.28:0/962836929 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x558c7c60f100 0x558c7c6114f0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:38.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.992+0000 7f12b600d740 1 --2- 172.21.15.28:0/962836929 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558c7c598dc0 0x558c7c59ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:38.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.992+0000 7f12b600d740 1 --2- 172.21.15.28:0/962836929 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558c7c58e6f0 0x558c7c59e630 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:38.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.992+0000 7f12b600d740 1 --2- 172.21.15.28:0/962836929 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c7c58ddd0 0x558c7c5a12a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:38.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.992+0000 7f129a000640 0 client.25441 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:09:38.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.992+0000 7f12b600d740 1 -- 172.21.15.28:0/962836929 >> 172.21.15.28:0/962836929 conn(0x558c7c58a0f0 msgr2=0x558c7c597a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:09:38.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.992+0000 7f12b600d740 1 -- 172.21.15.28:0/962836929 shutdown_connections 2024-05-12T22:09:38.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.992+0000 7f12b600d740 1 -- 172.21.15.28:0/962836929 wait complete. 2024-05-12T22:09:38.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.992+0000 7f12b600d740 20 client.25441 trim_cache size 0 max 16384 2024-05-12T22:09:38.997 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-05-12T22:09:38.997 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-05-12T22:09:38.997 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-05-12T22:09:38.997 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-05-12T22:09:38.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.993+0000 7f12b1c00640 1 -- 172.21.15.28:0/305408994 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f12900a2e20 msgr2=0x7f12900a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:38.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.993+0000 7f12b1c00640 1 --2- 172.21.15.28:0/305408994 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f12900a2e20 0x7f12900a52e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f1294009620 tx=0x7f1294005f90 comp rx=0 tx=0).stop 2024-05-12T22:09:38.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.993+0000 7f12b1c00640 1 -- 172.21.15.28:0/305408994 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f12ac108770 msgr2=0x7f12ac19ead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:38.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.993+0000 7f12b1c00640 1 --2- 172.21.15.28:0/305408994 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f12ac108770 0x7f12ac19ead0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f129c02ac70 tx=0x7f129c0305d0 comp rx=0 tx=0).stop 2024-05-12T22:09:38.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.993+0000 7f12b1c00640 1 -- 172.21.15.28:0/305408994 shutdown_connections 2024-05-12T22:09:38.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.993+0000 7f12b1c00640 1 --2- 172.21.15.28:0/305408994 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f12900a2e20 0x7f12900a52e0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:38.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.993+0000 7f12b1c00640 1 --2- 172.21.15.28:0/305408994 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f12ac10ddf0 0x7f12ac1a2e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:38.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.993+0000 7f12b1c00640 1 --2- 172.21.15.28:0/305408994 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f12ac108770 0x7f12ac19ead0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:38.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.993+0000 7f12b1c00640 1 --2- 172.21.15.28:0/305408994 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f12ac107dd0 0x7f12ac19e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:38.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.993+0000 7f12b1c00640 1 -- 172.21.15.28:0/305408994 >> 172.21.15.28:0/305408994 conn(0x7f12ac077650 msgr2=0x7f12ac078760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:09:38.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.993+0000 7f12b1c00640 1 -- 172.21.15.28:0/305408994 shutdown_connections 2024-05-12T22:09:38.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:38.993+0000 7f12b1c00640 1 -- 172.21.15.28:0/305408994 wait complete. 2024-05-12T22:09:39.012 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-12T22:09:39.013 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-12T22:09:39.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.225+0000 7f166a200640 1 -- 172.21.15.28:0/3370751984 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f166410d180 msgr2=0x7f1664114d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:39.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.225+0000 7f166a200640 1 --2- 172.21.15.28:0/3370751984 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f166410d180 0x7f1664114d50 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f165c009be0 tx=0x7f165c02a6c0 comp rx=0 tx=0).stop 2024-05-12T22:09:39.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.225+0000 7f166a200640 1 -- 172.21.15.28:0/3370751984 shutdown_connections 2024-05-12T22:09:39.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.225+0000 7f166a200640 1 --2- 172.21.15.28:0/3370751984 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f166410d180 0x7f1664114d50 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:39.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.225+0000 7f166a200640 1 --2- 172.21.15.28:0/3370751984 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1664108a60 0x7f166410caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:39.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.225+0000 7f166a200640 1 --2- 172.21.15.28:0/3370751984 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16641080b0 0x7f1664108490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:39.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.225+0000 7f166a200640 1 -- 172.21.15.28:0/3370751984 >> 172.21.15.28:0/3370751984 conn(0x7f1664075030 msgr2=0x7f1664106650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:09:39.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.225+0000 7f166a200640 1 -- 172.21.15.28:0/3370751984 shutdown_connections 2024-05-12T22:09:39.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.226+0000 7f166a200640 1 -- 172.21.15.28:0/3370751984 wait complete. 2024-05-12T22:09:39.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.227+0000 7f166a200640 1 Processor -- start 2024-05-12T22:09:39.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.227+0000 7f166a200640 1 -- start start 2024-05-12T22:09:39.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.228+0000 7f166a200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f16641080b0 0x7f166419e730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:39.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.228+0000 7f166a200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f166410d180 0x7f166419ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:39.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.228+0000 7f166a200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16641a2ff0 0x7f16641a34a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:39.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.228+0000 7f166a200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f16641181f0 con 0x7f16641a2ff0 2024-05-12T22:09:39.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.228+0000 7f166a200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f1664118070 con 0x7f166410d180 2024-05-12T22:09:39.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.228+0000 7f166a200640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f1664118370 con 0x7f16641080b0 2024-05-12T22:09:39.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.228+0000 7f1663400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f16641080b0 0x7f166419e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:39.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.228+0000 7f1663e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16641a2ff0 0x7f16641a34a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:39.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.228+0000 7f1663400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f16641080b0 0x7f166419e730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:34660/0 (socket says 172.21.15.28:34660) 2024-05-12T22:09:39.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.228+0000 7f1663400640 1 -- 172.21.15.28:0/1761328157 learned_addr learned my addr 172.21.15.28:0/1761328157 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:09:39.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.228+0000 7f1662a00640 1 --2- 172.21.15.28:0/1761328157 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f166410d180 0x7f166419ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:39.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.228+0000 7f1663400640 1 -- 172.21.15.28:0/1761328157 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f166410d180 msgr2=0x7f166419ec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:39.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.228+0000 7f1663400640 1 --2- 172.21.15.28:0/1761328157 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f166410d180 0x7f166419ec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:39.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.228+0000 7f1663400640 1 -- 172.21.15.28:0/1761328157 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16641a2ff0 msgr2=0x7f16641a34a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:39.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.228+0000 7f1663400640 1 --2- 172.21.15.28:0/1761328157 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16641a2ff0 0x7f16641a34a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:39.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.228+0000 7f1663400640 1 -- 172.21.15.28:0/1761328157 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16641a3ae0 con 0x7f16641080b0 2024-05-12T22:09:39.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.229+0000 7f1663400640 1 --2- 172.21.15.28:0/1761328157 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f16641080b0 0x7f166419e730 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f165400e9c0 tx=0x7f165400ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:09:39.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.229+0000 7f165be00640 1 -- 172.21.15.28:0/1761328157 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f165400cde0 con 0x7f16641080b0 2024-05-12T22:09:39.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.229+0000 7f165be00640 1 -- 172.21.15.28:0/1761328157 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f16540035f0 con 0x7f16641080b0 2024-05-12T22:09:39.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.229+0000 7f166a200640 1 -- 172.21.15.28:0/1761328157 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16641a3dd0 con 0x7f16641080b0 2024-05-12T22:09:39.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.229+0000 7f165be00640 1 -- 172.21.15.28:0/1761328157 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1654010690 con 0x7f16641080b0 2024-05-12T22:09:39.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.229+0000 7f166a200640 1 -- 172.21.15.28:0/1761328157 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16641ab5c0 con 0x7f16641080b0 2024-05-12T22:09:39.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.231+0000 7f166a200640 1 -- 172.21.15.28:0/1761328157 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1664068aa0 con 0x7f16641080b0 2024-05-12T22:09:39.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.233+0000 7f165be00640 1 -- 172.21.15.28:0/1761328157 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 25) ==== 138060+0+0 (secure 0 0 0) 0x7f16540037a0 con 0x7f16641080b0 2024-05-12T22:09:39.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.234+0000 7f165be00640 1 --2- 172.21.15.28:0/1761328157 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f16480a2e20 0x7f16480a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:39.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.235+0000 7f165be00640 1 -- 172.21.15.28:0/1761328157 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f16540cfa20 con 0x7f16641080b0 2024-05-12T22:09:39.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.235+0000 7f1662a00640 1 --2- 172.21.15.28:0/1761328157 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f16480a2e20 0x7f16480a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:39.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.235+0000 7f1662a00640 1 --2- 172.21.15.28:0/1761328157 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f16480a2e20 0x7f16480a52e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f1664107770 tx=0x7f1650005f90 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:09:39.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.239+0000 7f165be00640 1 -- 172.21.15.28:0/1761328157 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f165409c3c0 con 0x7f16641080b0 2024-05-12T22:09:39.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.484+0000 7f166a200640 1 -- 172.21.15.28:0/1761328157 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1664110410 con 0x7f16641080b0 2024-05-12T22:09:39.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.486+0000 7f165be00640 1 -- 172.21.15.28:0/1761328157 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11864 (secure 0 0 0) 0x7f165409bcc0 con 0x7f16641080b0 2024-05-12T22:09:39.487 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-12T22:09:39.487 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1844141745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1844141745},{"type":"v1","addr":"172.21.15.186:6833","nonce":1844141745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6833/2960763758","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":2960763758},{"type":"v1","addr":"172.21.15.86:6833","nonce":2960763758}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24470,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/4131134219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":4131134219},{"type":"v1","addr":"172.21.15.28:6835","nonce":4131134219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24481,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/1727869064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1727869064},{"type":"v1","addr":"172.21.15.186:6835","nonce":1727869064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24482,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2938260704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2938260704},{"type":"v1","addr":"172.21.15.28:6837","nonce":2938260704}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24494,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3473700910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3473700910},{"type":"v1","addr":"172.21.15.86:6835","nonce":3473700910}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-12T22:06:33.426986+0000","modified":"2024-05-12T22:08:58.179491+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24581,"mds_1":24550,"mds_2":24532},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24499":{"gid":24499,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6837/3600809235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":3600809235},{"type":"v1","addr":"172.21.15.186:6837","nonce":3600809235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"i","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6839/2399610807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":2399610807},{"type":"v1","addr":"172.21.15.186:6839","nonce":2399610807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24530":{"gid":24530,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.86:6837/3782448884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":3782448884},{"type":"v1","addr":"172.21.15.86:6837","nonce":3782448884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24532":{"gid":24532,"name":"g","rank":2,"incarnation":21,"state":"up:active","state_seq":6,"addr":"172.21.15.28:6839/123892158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":123892158},{"type":"v1","addr":"172.21.15.28:6839","nonce":123892158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24550":{"gid":24550,"name":"a","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.28:6841/761183870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":761183870},{"type":"v1","addr":"172.21.15.28:6841","nonce":761183870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24581":{"gid":24581,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6839/1251999130","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6838","nonce":1251999130},{"type":"v1","addr":"172.21.15.86:6839","nonce":1251999130}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24581,"qdb_cluster":[24550,24581,24532]},"id":1}]} 2024-05-12T22:09:39.487 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 24 2024-05-12T22:09:39.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.488+0000 7f166a200640 1 -- 172.21.15.28:0/1761328157 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f16480a2e20 msgr2=0x7f16480a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:39.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.488+0000 7f166a200640 1 --2- 172.21.15.28:0/1761328157 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f16480a2e20 0x7f16480a52e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f1664107770 tx=0x7f1650005f90 comp rx=0 tx=0).stop 2024-05-12T22:09:39.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.488+0000 7f166a200640 1 -- 172.21.15.28:0/1761328157 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f16641080b0 msgr2=0x7f166419e730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:39.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.489+0000 7f166a200640 1 --2- 172.21.15.28:0/1761328157 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f16641080b0 0x7f166419e730 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f165400e9c0 tx=0x7f165400ee80 comp rx=0 tx=0).stop 2024-05-12T22:09:39.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.489+0000 7f166a200640 1 -- 172.21.15.28:0/1761328157 shutdown_connections 2024-05-12T22:09:39.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.489+0000 7f166a200640 1 --2- 172.21.15.28:0/1761328157 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f16480a2e20 0x7f16480a52e0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:39.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.489+0000 7f166a200640 1 --2- 172.21.15.28:0/1761328157 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16641a2ff0 0x7f16641a34a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:39.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.489+0000 7f166a200640 1 --2- 172.21.15.28:0/1761328157 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f166410d180 0x7f166419ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:39.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.489+0000 7f166a200640 1 --2- 172.21.15.28:0/1761328157 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f16641080b0 0x7f166419e730 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:39.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.489+0000 7f166a200640 1 -- 172.21.15.28:0/1761328157 >> 172.21.15.28:0/1761328157 conn(0x7f1664075030 msgr2=0x7f1664075650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:09:39.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.489+0000 7f166a200640 1 -- 172.21.15.28:0/1761328157 shutdown_connections 2024-05-12T22:09:39.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.489+0000 7f166a200640 1 -- 172.21.15.28:0/1761328157 wait complete. 2024-05-12T22:09:39.510 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-05-12T22:09:39.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.717+0000 7f090ea00640 1 -- 172.21.15.28:0/1774117034 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09081038d0 msgr2=0x7f0908103d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:39.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.717+0000 7f090ea00640 1 --2- 172.21.15.28:0/1774117034 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09081038d0 0x7f0908103d50 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f08f8009930 tx=0x7f08f802a420 comp rx=0 tx=0).stop 2024-05-12T22:09:39.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.717+0000 7f090ea00640 1 -- 172.21.15.28:0/1774117034 shutdown_connections 2024-05-12T22:09:39.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.717+0000 7f090ea00640 1 --2- 172.21.15.28:0/1774117034 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0908104290 0x7f09081107a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:39.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.717+0000 7f090ea00640 1 --2- 172.21.15.28:0/1774117034 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09081038d0 0x7f0908103d50 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:39.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.717+0000 7f090ea00640 1 --2- 172.21.15.28:0/1774117034 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f09081098e0 0x7f0908109cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:39.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.717+0000 7f090ea00640 1 -- 172.21.15.28:0/1774117034 >> 172.21.15.28:0/1774117034 conn(0x7f09080ff360 msgr2=0x7f0908101780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:09:39.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.718+0000 7f090ea00640 1 -- 172.21.15.28:0/1774117034 shutdown_connections 2024-05-12T22:09:39.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.718+0000 7f090ea00640 1 -- 172.21.15.28:0/1774117034 wait complete. 2024-05-12T22:09:39.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.719+0000 7f090ea00640 1 Processor -- start 2024-05-12T22:09:39.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.719+0000 7f090ea00640 1 -- start start 2024-05-12T22:09:39.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.719+0000 7f090ea00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09081038d0 0x7f090819a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:39.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.719+0000 7f090ea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0908104290 0x7f090819a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:39.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.719+0000 7f090ea00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f09081098e0 0x7f090819ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:39.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.719+0000 7f090ea00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0908113d40 con 0x7f0908104290 2024-05-12T22:09:39.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.719+0000 7f090ea00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f0908113bc0 con 0x7f09081098e0 2024-05-12T22:09:39.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.719+0000 7f090ea00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f0908113ec0 con 0x7f09081038d0 2024-05-12T22:09:39.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.720+0000 7f090cc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0908104290 0x7f090819a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:39.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.720+0000 7f090d600640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09081038d0 0x7f090819a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:39.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.720+0000 7f090d600640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09081038d0 0x7f090819a220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:34682/0 (socket says 172.21.15.28:34682) 2024-05-12T22:09:39.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.720+0000 7f090cc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0908104290 0x7f090819a760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50814/0 (socket says 172.21.15.28:50814) 2024-05-12T22:09:39.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.720+0000 7f090d600640 1 -- 172.21.15.28:0/1188921651 learned_addr learned my addr 172.21.15.28:0/1188921651 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:09:39.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.720+0000 7f090e000640 1 --2- 172.21.15.28:0/1188921651 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f09081098e0 0x7f090819ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:39.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.720+0000 7f090e000640 1 -- 172.21.15.28:0/1188921651 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09081038d0 msgr2=0x7f090819a220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:39.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.720+0000 7f090e000640 1 --2- 172.21.15.28:0/1188921651 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09081038d0 0x7f090819a220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:39.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.720+0000 7f090e000640 1 -- 172.21.15.28:0/1188921651 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0908104290 msgr2=0x7f090819a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:39.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.720+0000 7f090e000640 1 --2- 172.21.15.28:0/1188921651 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0908104290 0x7f090819a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:39.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.720+0000 7f090e000640 1 -- 172.21.15.28:0/1188921651 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f090819f1c0 con 0x7f09081098e0 2024-05-12T22:09:39.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.720+0000 7f090cc00640 1 --2- 172.21.15.28:0/1188921651 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0908104290 0x7f090819a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:09:39.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.721+0000 7f090e000640 1 --2- 172.21.15.28:0/1188921651 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f09081098e0 0x7f090819ea90 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f0900009fd0 tx=0x7f090000ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:09:39.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.721+0000 7f0906000640 1 -- 172.21.15.28:0/1188921651 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f090000ce20 con 0x7f09081098e0 2024-05-12T22:09:39.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.721+0000 7f0906000640 1 -- 172.21.15.28:0/1188921651 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0900005020 con 0x7f09081098e0 2024-05-12T22:09:39.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.721+0000 7f0906000640 1 -- 172.21.15.28:0/1188921651 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0900010720 con 0x7f09081098e0 2024-05-12T22:09:39.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.721+0000 7f090ea00640 1 -- 172.21.15.28:0/1188921651 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f090819f4b0 con 0x7f09081098e0 2024-05-12T22:09:39.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.721+0000 7f090ea00640 1 -- 172.21.15.28:0/1188921651 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09081a6d40 con 0x7f09081098e0 2024-05-12T22:09:39.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.725+0000 7f0912d6a740 1 Processor -- start 2024-05-12T22:09:39.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.725+0000 7f0906000640 1 -- 172.21.15.28:0/1188921651 <== mon.2 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 25) ==== 138060+0+0 (secure 0 0 0) 0x7f0900005540 con 0x7f09081098e0 2024-05-12T22:09:39.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.725+0000 7f0912d6a740 1 -- start start 2024-05-12T22:09:39.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.725+0000 7f0912d6a740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563091c1fb60 0x563091c1ff40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:39.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.725+0000 7f0912d6a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563091c20480 0x563091c2a610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:39.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.725+0000 7f0912d6a740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563091c2ab50 0x563091c2cf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:39.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.725+0000 7f0912d6a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x563091b84a30 con 0x563091c20480 2024-05-12T22:09:39.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.725+0000 7f0912d6a740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x563091b2c700 con 0x563091c1fb60 2024-05-12T22:09:39.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.725+0000 7f0912d6a740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x563091b809f0 con 0x563091c2ab50 2024-05-12T22:09:39.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.726+0000 7f090d600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563091c20480 0x563091c2a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:39.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.726+0000 7f090d600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563091c20480 0x563091c2a610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50824/0 (socket says 172.21.15.28:50824) 2024-05-12T22:09:39.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.726+0000 7f090cc00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563091c1fb60 0x563091c1ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:39.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.726+0000 7f090d600640 1 -- 172.21.15.28:0/3012811027 learned_addr learned my addr 172.21.15.28:0/3012811027 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:09:39.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.726+0000 7f090cc00640 1 --2- 172.21.15.28:0/3012811027 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563091c2ab50 0x563091c2cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:39.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.726+0000 7f090d600640 1 -- 172.21.15.28:0/3012811027 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563091c2ab50 msgr2=0x563091c2cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:09:39.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.726+0000 7f090d600640 1 --2- 172.21.15.28:0/3012811027 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563091c2ab50 0x563091c2cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:39.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.726+0000 7f090d600640 1 -- 172.21.15.28:0/3012811027 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563091c1fb60 msgr2=0x563091c1ff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:09:39.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.726+0000 7f090d600640 1 --2- 172.21.15.28:0/3012811027 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563091c1fb60 0x563091c1ff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:39.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.726+0000 7f090d600640 1 -- 172.21.15.28:0/3012811027 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5630918983f0 con 0x563091c20480 2024-05-12T22:09:39.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.727+0000 7f0906000640 1 --2- 172.21.15.28:0/1188921651 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f08f00a2ea0 0x7f08f00a5360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:39.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.727+0000 7f090d600640 1 --2- 172.21.15.28:0/3012811027 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563091c20480 0x563091c2a610 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f08fc009d90 tx=0x7f08fc011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32fbd9629d0af591 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:09:39.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.727+0000 7f0906000640 1 -- 172.21.15.28:0/1188921651 <== mon.2 v2:172.21.15.86:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f09000cfcc0 con 0x7f09081098e0 2024-05-12T22:09:39.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.727+0000 7f090cc00640 1 --2- 172.21.15.28:0/1188921651 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f08f00a2ea0 0x7f08f00a5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:39.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.727+0000 7f08f6600640 1 -- 172.21.15.28:0/3012811027 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08fc01e070 con 0x563091c20480 2024-05-12T22:09:39.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.727+0000 7f08f6600640 1 -- 172.21.15.28:0/3012811027 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f08fc019b10 con 0x563091c20480 2024-05-12T22:09:39.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.727+0000 7f08f6600640 1 -- 172.21.15.28:0/3012811027 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08fc00fb30 con 0x563091c20480 2024-05-12T22:09:39.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.728+0000 7f0912d6a740 1 -- 172.21.15.28:0/3012811027 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563091c20480 msgr2=0x563091c2a610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:39.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.728+0000 7f090cc00640 1 --2- 172.21.15.28:0/1188921651 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f08f00a2ea0 0x7f08f00a5360 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f08f8006010 tx=0x7f08f8041040 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:09:39.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.728+0000 7f0912d6a740 1 --2- 172.21.15.28:0/3012811027 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563091c20480 0x563091c2a610 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f08fc009d90 tx=0x7f08fc011810 comp rx=0 tx=0).stop 2024-05-12T22:09:39.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.728+0000 7f0912d6a740 1 -- 172.21.15.28:0/3012811027 shutdown_connections 2024-05-12T22:09:39.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.728+0000 7f0912d6a740 1 --2- 172.21.15.28:0/3012811027 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563091c2ab50 0x563091c2cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:39.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.728+0000 7f0912d6a740 1 --2- 172.21.15.28:0/3012811027 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563091c20480 0x563091c2a610 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:39.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.728+0000 7f0912d6a740 1 --2- 172.21.15.28:0/3012811027 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563091c1fb60 0x563091c1ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:39.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.728+0000 7f0912d6a740 1 -- 172.21.15.28:0/3012811027 >> 172.21.15.28:0/3012811027 conn(0x563091c1be80 msgr2=0x563091c1c260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:09:39.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.728+0000 7f0912d6a740 1 -- 172.21.15.28:0/3012811027 shutdown_connections 2024-05-12T22:09:39.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.728+0000 7f0912d6a740 1 -- 172.21.15.28:0/3012811027 wait complete. 2024-05-12T22:09:39.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.728+0000 7f0912d6a740 1 -- start start 2024-05-12T22:09:39.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.729+0000 7f0912d6a740 1 Processor -- start 2024-05-12T22:09:39.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.729+0000 7f0912d6a740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563091c1fb60 0x563091c27a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:39.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.729+0000 7f0912d6a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563091c20480 0x563091c27fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:39.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.729+0000 7f0912d6a740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563091c2ab50 0x563091c28510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:39.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.729+0000 7f0912d6a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x563091c2d940 con 0x563091c20480 2024-05-12T22:09:39.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.729+0000 7f0912d6a740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x563091b3fa70 con 0x563091c1fb60 2024-05-12T22:09:39.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.729+0000 7f0912d6a740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x563091c2dac0 con 0x563091c2ab50 2024-05-12T22:09:39.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.729+0000 7f0912d6a740 10 client.0 _ll_register_callbacks cb 0x563091b22ba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:09:39.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.730+0000 7f090d600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563091c1fb60 0x563091c27a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:39.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.730+0000 7f090d600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563091c1fb60 0x563091c27a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:34200/0 (socket says 172.21.15.28:34200) 2024-05-12T22:09:39.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.730+0000 7f090cc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563091c20480 0x563091c27fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:39.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.730+0000 7f090d600640 1 -- 172.21.15.28:0/1555296522 learned_addr learned my addr 172.21.15.28:0/1555296522 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:09:39.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.730+0000 7f090e000640 1 --2- 172.21.15.28:0/1555296522 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563091c2ab50 0x563091c28510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:39.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.730+0000 7f090d600640 1 -- 172.21.15.28:0/1555296522 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563091c2ab50 msgr2=0x563091c28510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:09:39.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.730+0000 7f090d600640 1 --2- 172.21.15.28:0/1555296522 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563091c2ab50 0x563091c28510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:39.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.730+0000 7f090d600640 1 -- 172.21.15.28:0/1555296522 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563091c20480 msgr2=0x563091c27fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:09:39.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.730+0000 7f090d600640 1 --2- 172.21.15.28:0/1555296522 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563091c20480 0x563091c27fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:39.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.730+0000 7f090d600640 1 -- 172.21.15.28:0/1555296522 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5630919627d0 con 0x563091c1fb60 2024-05-12T22:09:39.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.730+0000 7f090d600640 1 --2- 172.21.15.28:0/1555296522 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563091c1fb60 0x563091c27a90 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f08fc01c040 tx=0x7f08fc004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=515f6339d55ab6d5 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:09:39.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.730+0000 7f08eaa00640 10 client.0 ms_handle_connect on v2:172.21.15.86:3300/0 2024-05-12T22:09:39.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.731+0000 7f08eaa00640 1 -- 172.21.15.28:0/1555296522 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08fc027020 con 0x563091c1fb60 2024-05-12T22:09:39.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.731+0000 7f08eaa00640 1 -- 172.21.15.28:0/1555296522 <== mon.2 v2:172.21.15.86:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f08fc019630 con 0x563091c1fb60 2024-05-12T22:09:39.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.731+0000 7f0912d6a740 1 -- 172.21.15.28:0/1555296522 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f08fc019b10 con 0x563091c1fb60 2024-05-12T22:09:39.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.731+0000 7f08eaa00640 1 -- 172.21.15.28:0/1555296522 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f08fc032050 con 0x563091c1fb60 2024-05-12T22:09:39.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.731+0000 7f0912d6a740 10 client.25487 fetch_fsmap learned FSMap version 24 2024-05-12T22:09:39.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.731+0000 7f0912d6a740 1 -- 172.21.15.28:0/1555296522 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x563091c39b90 con 0x563091c1fb60 2024-05-12T22:09:39.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.732+0000 7f08eaa00640 1 -- 172.21.15.28:0/1555296522 <== mon.2 v2:172.21.15.86:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f08fc019b10 con 0x563091c1fb60 2024-05-12T22:09:39.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.732+0000 7f08eaa00640 1 -- 172.21.15.28:0/1555296522 <== mon.2 v2:172.21.15.86:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f08fc005320 con 0x563091c1fb60 2024-05-12T22:09:39.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.732+0000 7f0912d6a740 10 client.25487 fetch_fsmap finished waiting for FSMap version 24 2024-05-12T22:09:39.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.732+0000 7f0912d6a740 10 client.25487 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.g 2024-05-12T22:09:39.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.732+0000 7f0912d6a740 20 client.25487 populate_metadata read hostname 'smithi028' 2024-05-12T22:09:39.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.732+0000 7f0912d6a740 1 --2- 172.21.15.28:0/1555296522 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x563091c3aff0 0x563091c3d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:39.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.732+0000 7f0912d6a740 4 client.25487 mds_command: new command op to 24532 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-12T22:09:39.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.732+0000 7f0912d6a740 1 -- 172.21.15.28:0/1555296522 --> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563091b92160 con 0x563091c3aff0 2024-05-12T22:09:39.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.732+0000 7f090cc00640 1 --2- 172.21.15.28:0/1555296522 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x563091c3aff0 0x563091c3d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:39.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.733+0000 7f090cc00640 1 --2- 172.21.15.28:0/1555296522 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x563091c3aff0 0x563091c3d3e0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=44819ff52253e8b0 server_cookie=1a73cbe0e6d0c45 in_seq=0 out_seq=0 2024-05-12T22:09:39.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.733+0000 7f08eaa00640 10 client.25487 ms_handle_connect on v2:172.21.15.28:6838/123892158 2024-05-12T22:09:39.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.734+0000 7f08eaa00640 1 -- 172.21.15.28:0/1555296522 <== mds.2 v2:172.21.15.28:6838/123892158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563091b92160 con 0x563091c3aff0 2024-05-12T22:09:39.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.734+0000 7f08eaa00640 10 client.25487 handle_command_reply: tid=0 2024-05-12T22:09:39.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.734+0000 7f0912d6a740 1 client.25487 shutdown 2024-05-12T22:09:39.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.734+0000 7f0912d6a740 1 -- 172.21.15.28:0/1555296522 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563091c1fb60 msgr2=0x563091c27a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:39.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.734+0000 7f0912d6a740 1 --2- 172.21.15.28:0/1555296522 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563091c1fb60 0x563091c27a90 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f08fc01c040 tx=0x7f08fc004270 comp rx=0 tx=0).stop 2024-05-12T22:09:39.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.734+0000 7f0912d6a740 1 -- 172.21.15.28:0/1555296522 shutdown_connections 2024-05-12T22:09:39.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.734+0000 7f0912d6a740 1 --2- 172.21.15.28:0/1555296522 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563091c2ab50 0x563091c28510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:39.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.734+0000 7f0912d6a740 1 --2- 172.21.15.28:0/1555296522 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563091c20480 0x563091c27fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:39.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.734+0000 7f0912d6a740 1 --2- 172.21.15.28:0/1555296522 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x563091c3aff0 0x563091c3d3e0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:39.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.734+0000 7f0912d6a740 1 --2- 172.21.15.28:0/1555296522 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563091c1fb60 0x563091c27a90 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:39.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.734+0000 7f08eaa00640 0 client.25487 ms_handle_reset on v2:172.21.15.28:6838/123892158 2024-05-12T22:09:39.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.734+0000 7f0912d6a740 1 -- 172.21.15.28:0/1555296522 >> 172.21.15.28:0/1555296522 conn(0x563091c1be80 msgr2=0x563091c1b000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:09:39.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.734+0000 7f0912d6a740 1 -- 172.21.15.28:0/1555296522 shutdown_connections 2024-05-12T22:09:39.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.734+0000 7f0912d6a740 1 -- 172.21.15.28:0/1555296522 wait complete. 2024-05-12T22:09:39.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.734+0000 7f0912d6a740 20 client.25487 trim_cache size 0 max 16384 2024-05-12T22:09:39.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.749+0000 7f0912d6a740 1 Processor -- start 2024-05-12T22:09:39.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.749+0000 7f0912d6a740 1 -- start start 2024-05-12T22:09:39.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.750+0000 7f0912d6a740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563091c1fb60 0x563091c36800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:39.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.750+0000 7f0912d6a740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563091c20480 0x563091c42bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:39.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.750+0000 7f0912d6a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563091c2ab50 0x563091c43130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:39.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.750+0000 7f0912d6a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f09081b1da0 con 0x563091c2ab50 2024-05-12T22:09:39.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.750+0000 7f0912d6a740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f09081af670 con 0x563091c20480 2024-05-12T22:09:39.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.750+0000 7f0912d6a740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f09081ad4c0 con 0x563091c1fb60 2024-05-12T22:09:39.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.750+0000 7f090e000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563091c2ab50 0x563091c43130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:39.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.750+0000 7f090e000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563091c2ab50 0x563091c43130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50848/0 (socket says 172.21.15.28:50848) 2024-05-12T22:09:39.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.750+0000 7f090cc00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563091c20480 0x563091c42bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:39.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.750+0000 7f090e000640 1 -- 172.21.15.28:0/2749948918 learned_addr learned my addr 172.21.15.28:0/2749948918 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:09:39.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.750+0000 7f090d600640 1 --2- 172.21.15.28:0/2749948918 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563091c1fb60 0x563091c36800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:39.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.751+0000 7f090e000640 1 -- 172.21.15.28:0/2749948918 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563091c1fb60 msgr2=0x563091c36800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:09:39.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.751+0000 7f090e000640 1 --2- 172.21.15.28:0/2749948918 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563091c1fb60 0x563091c36800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:39.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.751+0000 7f090e000640 1 -- 172.21.15.28:0/2749948918 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563091c20480 msgr2=0x563091c42bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:09:39.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.751+0000 7f090e000640 1 --2- 172.21.15.28:0/2749948918 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563091c20480 0x563091c42bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:39.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.751+0000 7f090e000640 1 -- 172.21.15.28:0/2749948918 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563091c202b0 con 0x563091c2ab50 2024-05-12T22:09:39.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.751+0000 7f090d600640 1 --2- 172.21.15.28:0/2749948918 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563091c1fb60 0x563091c36800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:09:39.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.751+0000 7f090cc00640 1 --2- 172.21.15.28:0/2749948918 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563091c20480 0x563091c42bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:09:39.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.751+0000 7f090e000640 1 --2- 172.21.15.28:0/2749948918 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563091c2ab50 0x563091c43130 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f09000983c0 tx=0x7f0900005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56047cd8a930808f server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:09:39.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.751+0000 7f08f5c00640 1 -- 172.21.15.28:0/2749948918 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09000b4070 con 0x563091c2ab50 2024-05-12T22:09:39.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.751+0000 7f08f5c00640 1 -- 172.21.15.28:0/2749948918 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f09000a4bd0 con 0x563091c2ab50 2024-05-12T22:09:39.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.751+0000 7f08f5c00640 1 -- 172.21.15.28:0/2749948918 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09000cd540 con 0x563091c2ab50 2024-05-12T22:09:39.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.752+0000 7f0912d6a740 1 -- 172.21.15.28:0/2749948918 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563091c2ab50 msgr2=0x563091c43130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:39.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.752+0000 7f0912d6a740 1 --2- 172.21.15.28:0/2749948918 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563091c2ab50 0x563091c43130 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f09000983c0 tx=0x7f0900005f90 comp rx=0 tx=0).stop 2024-05-12T22:09:39.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.752+0000 7f0912d6a740 1 -- 172.21.15.28:0/2749948918 shutdown_connections 2024-05-12T22:09:39.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.752+0000 7f0912d6a740 1 --2- 172.21.15.28:0/2749948918 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563091c2ab50 0x563091c43130 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:39.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.752+0000 7f0912d6a740 1 --2- 172.21.15.28:0/2749948918 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563091c20480 0x563091c42bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:39.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.752+0000 7f0912d6a740 1 --2- 172.21.15.28:0/2749948918 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563091c1fb60 0x563091c36800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:39.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.752+0000 7f0912d6a740 1 -- 172.21.15.28:0/2749948918 >> 172.21.15.28:0/2749948918 conn(0x563091c1be80 msgr2=0x563091c22cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:09:39.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.752+0000 7f0912d6a740 1 -- 172.21.15.28:0/2749948918 shutdown_connections 2024-05-12T22:09:39.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.752+0000 7f0912d6a740 1 -- 172.21.15.28:0/2749948918 wait complete. 2024-05-12T22:09:39.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.752+0000 7f0912d6a740 1 -- start start 2024-05-12T22:09:39.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.753+0000 7f0912d6a740 1 Processor -- start 2024-05-12T22:09:39.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.753+0000 7f0912d6a740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563091c1fb60 0x563091c2e670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:39.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.753+0000 7f0912d6a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563091c20480 0x563091c2ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:39.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.753+0000 7f0912d6a740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563091c2ab50 0x563091c2f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:39.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.753+0000 7f0912d6a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f09081adb30 con 0x563091c20480 2024-05-12T22:09:39.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.753+0000 7f0912d6a740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f09081aefa0 con 0x563091c1fb60 2024-05-12T22:09:39.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.753+0000 7f0912d6a740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f09081ae200 con 0x563091c2ab50 2024-05-12T22:09:39.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.753+0000 7f0912d6a740 10 client.0 _ll_register_callbacks cb 0x7f08fc009c10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:09:39.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.753+0000 7f090cc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563091c20480 0x563091c2ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:39.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.753+0000 7f090cc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563091c20480 0x563091c2ebb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50864/0 (socket says 172.21.15.28:50864) 2024-05-12T22:09:39.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.753+0000 7f090cc00640 1 -- 172.21.15.28:0/1442338235 learned_addr learned my addr 172.21.15.28:0/1442338235 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:09:39.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.753+0000 7f090d600640 1 --2- 172.21.15.28:0/1442338235 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563091c1fb60 0x563091c2e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:39.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.754+0000 7f090e000640 1 --2- 172.21.15.28:0/1442338235 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563091c2ab50 0x563091c2f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:39.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.754+0000 7f090cc00640 1 -- 172.21.15.28:0/1442338235 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563091c2ab50 msgr2=0x563091c2f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:09:39.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.754+0000 7f090cc00640 1 --2- 172.21.15.28:0/1442338235 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563091c2ab50 0x563091c2f0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:39.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.754+0000 7f090cc00640 1 -- 172.21.15.28:0/1442338235 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563091c1fb60 msgr2=0x563091c2e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:09:39.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.754+0000 7f090cc00640 1 --2- 172.21.15.28:0/1442338235 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563091c1fb60 0x563091c2e670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:39.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.754+0000 7f090cc00640 1 -- 172.21.15.28:0/1442338235 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563091ca0460 con 0x563091c20480 2024-05-12T22:09:39.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.754+0000 7f090d600640 1 --2- 172.21.15.28:0/1442338235 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563091c1fb60 0x563091c2e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:09:39.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.754+0000 7f090cc00640 1 --2- 172.21.15.28:0/1442338235 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563091c20480 0x563091c2ebb0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f08f802aa10 tx=0x7f08f8031510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a49757ef05cb2cc server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:09:39.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.754+0000 7f08eaa00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-12T22:09:39.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.754+0000 7f08eaa00640 1 -- 172.21.15.28:0/1442338235 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08f8035040 con 0x563091c20480 2024-05-12T22:09:39.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.754+0000 7f08eaa00640 1 -- 172.21.15.28:0/1442338235 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f08f80546b0 con 0x563091c20480 2024-05-12T22:09:39.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.754+0000 7f0912d6a740 1 -- 172.21.15.28:0/1442338235 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f09000a4bd0 con 0x563091c20480 2024-05-12T22:09:39.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.755+0000 7f08eaa00640 1 -- 172.21.15.28:0/1442338235 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f08f8063050 con 0x563091c20480 2024-05-12T22:09:39.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.755+0000 7f0912d6a740 10 client.15375 fetch_fsmap learned FSMap version 24 2024-05-12T22:09:39.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.755+0000 7f0912d6a740 1 -- 172.21.15.28:0/1442338235 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x563091ca16d0 con 0x563091c20480 2024-05-12T22:09:39.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.755+0000 7f08eaa00640 1 -- 172.21.15.28:0/1442338235 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f09000a4bd0 con 0x563091c20480 2024-05-12T22:09:39.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.755+0000 7f08eaa00640 1 -- 172.21.15.28:0/1442338235 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f08f8032970 con 0x563091c20480 2024-05-12T22:09:39.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.755+0000 7f0912d6a740 10 client.15375 fetch_fsmap finished waiting for FSMap version 24 2024-05-12T22:09:39.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.755+0000 7f0912d6a740 10 client.15375 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.g 2024-05-12T22:09:39.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.755+0000 7f0912d6a740 20 client.15375 populate_metadata read hostname 'smithi028' 2024-05-12T22:09:39.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.755+0000 7f0912d6a740 1 --2- 172.21.15.28:0/1442338235 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x563091ca1e80 0x563091ca42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:39.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.755+0000 7f0912d6a740 4 client.15375 mds_command: new command op to 24532 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-12T22:09:39.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.755+0000 7f0912d6a740 1 -- 172.21.15.28:0/1442338235 --> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563091b8c4b0 con 0x563091ca1e80 2024-05-12T22:09:39.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.756+0000 7f090d600640 1 --2- 172.21.15.28:0/1442338235 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x563091ca1e80 0x563091ca42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:39.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.756+0000 7f090d600640 1 --2- 172.21.15.28:0/1442338235 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x563091ca1e80 0x563091ca42a0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=5d8ea6138a39a459 server_cookie=a0ffc858ed77e56a in_seq=0 out_seq=0 2024-05-12T22:09:39.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.756+0000 7f08eaa00640 10 client.15375 ms_handle_connect on v2:172.21.15.28:6838/123892158 2024-05-12T22:09:39.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.756+0000 7f08eaa00640 1 -- 172.21.15.28:0/1442338235 <== mds.2 v2:172.21.15.28:6838/123892158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563091b8c4b0 con 0x563091ca1e80 2024-05-12T22:09:39.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.756+0000 7f08eaa00640 10 client.15375 handle_command_reply: tid=0 2024-05-12T22:09:39.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.756+0000 7f0912d6a740 1 client.15375 shutdown 2024-05-12T22:09:39.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.757+0000 7f0912d6a740 1 -- 172.21.15.28:0/1442338235 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563091c20480 msgr2=0x563091c2ebb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:39.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.757+0000 7f0912d6a740 1 --2- 172.21.15.28:0/1442338235 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563091c20480 0x563091c2ebb0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f08f802aa10 tx=0x7f08f8031510 comp rx=0 tx=0).stop 2024-05-12T22:09:39.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.757+0000 7f0912d6a740 1 -- 172.21.15.28:0/1442338235 shutdown_connections 2024-05-12T22:09:39.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.757+0000 7f0912d6a740 1 --2- 172.21.15.28:0/1442338235 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563091c2ab50 0x563091c2f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:39.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.757+0000 7f0912d6a740 1 --2- 172.21.15.28:0/1442338235 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563091c20480 0x563091c2ebb0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:39.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.757+0000 7f0912d6a740 1 --2- 172.21.15.28:0/1442338235 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x563091ca1e80 0x563091ca42a0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:39.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.757+0000 7f0912d6a740 1 --2- 172.21.15.28:0/1442338235 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563091c1fb60 0x563091c2e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:39.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.757+0000 7f0912d6a740 1 -- 172.21.15.28:0/1442338235 >> 172.21.15.28:0/1442338235 conn(0x563091c1be80 msgr2=0x563091c1a770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:09:39.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.757+0000 7f08eaa00640 0 client.15375 ms_handle_reset on v2:172.21.15.28:6838/123892158 2024-05-12T22:09:39.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.757+0000 7f0912d6a740 1 -- 172.21.15.28:0/1442338235 shutdown_connections 2024-05-12T22:09:39.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.757+0000 7f0912d6a740 1 -- 172.21.15.28:0/1442338235 wait complete. 2024-05-12T22:09:39.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.757+0000 7f0912d6a740 20 client.15375 trim_cache size 0 max 16384 2024-05-12T22:09:39.762 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-05-12T22:09:39.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.758+0000 7f090ea00640 1 -- 172.21.15.28:0/1188921651 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f08f00a2ea0 msgr2=0x7f08f00a5360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:39.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.758+0000 7f090ea00640 1 --2- 172.21.15.28:0/1188921651 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f08f00a2ea0 0x7f08f00a5360 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f08f8006010 tx=0x7f08f8041040 comp rx=0 tx=0).stop 2024-05-12T22:09:39.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.758+0000 7f090ea00640 1 -- 172.21.15.28:0/1188921651 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f09081098e0 msgr2=0x7f090819ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:39.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.758+0000 7f090ea00640 1 --2- 172.21.15.28:0/1188921651 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f09081098e0 0x7f090819ea90 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f0900009fd0 tx=0x7f090000ef10 comp rx=0 tx=0).stop 2024-05-12T22:09:39.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.758+0000 7f090ea00640 1 -- 172.21.15.28:0/1188921651 shutdown_connections 2024-05-12T22:09:39.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.758+0000 7f090ea00640 1 --2- 172.21.15.28:0/1188921651 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f08f00a2ea0 0x7f08f00a5360 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:39.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.758+0000 7f090ea00640 1 --2- 172.21.15.28:0/1188921651 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f09081098e0 0x7f090819ea90 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:39.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.758+0000 7f090ea00640 1 --2- 172.21.15.28:0/1188921651 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0908104290 0x7f090819a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:39.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.758+0000 7f090ea00640 1 --2- 172.21.15.28:0/1188921651 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09081038d0 0x7f090819a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:39.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.758+0000 7f090ea00640 1 -- 172.21.15.28:0/1188921651 >> 172.21.15.28:0/1188921651 conn(0x7f09080ff360 msgr2=0x7f09080ffa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:09:39.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.758+0000 7f090ea00640 1 -- 172.21.15.28:0/1188921651 shutdown_connections 2024-05-12T22:09:39.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.758+0000 7f090ea00640 1 -- 172.21.15.28:0/1188921651 wait complete. 2024-05-12T22:09:39.778 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-05-12T22:09:39.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.991+0000 7f53c5c00640 1 -- 172.21.15.28:0/1281331533 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f53c0074500 msgr2=0x7f53c010da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:39.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.991+0000 7f53c5c00640 1 --2- 172.21.15.28:0/1281331533 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f53c0074500 0x7f53c010da60 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f53a8009900 tx=0x7f53a802a490 comp rx=0 tx=0).stop 2024-05-12T22:09:39.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.991+0000 7f53c5c00640 1 -- 172.21.15.28:0/1281331533 shutdown_connections 2024-05-12T22:09:39.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.991+0000 7f53c5c00640 1 --2- 172.21.15.28:0/1281331533 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f53c010dfa0 0x7f53c0110390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:39.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.991+0000 7f53c5c00640 1 --2- 172.21.15.28:0/1281331533 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f53c0074500 0x7f53c010da60 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:39.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.991+0000 7f53c5c00640 1 --2- 172.21.15.28:0/1281331533 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53c0073be0 0x7f53c0073fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:39.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.991+0000 7f53c5c00640 1 -- 172.21.15.28:0/1281331533 >> 172.21.15.28:0/1281331533 conn(0x7f53c00fd1d0 msgr2=0x7f53c00ff5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:09:39.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.991+0000 7f53c5c00640 1 -- 172.21.15.28:0/1281331533 shutdown_connections 2024-05-12T22:09:39.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.991+0000 7f53c5c00640 1 -- 172.21.15.28:0/1281331533 wait complete. 2024-05-12T22:09:39.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.992+0000 7f53c5c00640 1 Processor -- start 2024-05-12T22:09:39.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.992+0000 7f53c5c00640 1 -- start start 2024-05-12T22:09:39.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.993+0000 7f53c5c00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f53c0073be0 0x7f53c019a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:39.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.993+0000 7f53c5c00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f53c0074500 0x7f53c019a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:39.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.993+0000 7f53c5c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53c010dfa0 0x7f53c019ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:39.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.993+0000 7f53c5c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f53c0113cd0 con 0x7f53c010dfa0 2024-05-12T22:09:39.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.993+0000 7f53c5c00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f53c0113b50 con 0x7f53c0073be0 2024-05-12T22:09:39.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.993+0000 7f53c5c00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f53c0113e50 con 0x7f53c0074500 2024-05-12T22:09:39.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.993+0000 7f53b7e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f53c0074500 0x7f53c019a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:39.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.993+0000 7f53c5200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53c010dfa0 0x7f53c019ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:39.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.993+0000 7f53bfe00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f53c0073be0 0x7f53c019a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:39.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.993+0000 7f53bfe00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f53c0073be0 0x7f53c019a210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:34238/0 (socket says 172.21.15.28:34238) 2024-05-12T22:09:39.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.993+0000 7f53bfe00640 1 -- 172.21.15.28:0/542553883 learned_addr learned my addr 172.21.15.28:0/542553883 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:09:39.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.993+0000 7f53b7e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f53c0074500 0x7f53c019a750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:34732/0 (socket says 172.21.15.28:34732) 2024-05-12T22:09:39.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.994+0000 7f53bfe00640 1 -- 172.21.15.28:0/542553883 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f53c0074500 msgr2=0x7f53c019a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:39.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.994+0000 7f53bfe00640 1 --2- 172.21.15.28:0/542553883 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f53c0074500 0x7f53c019a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:39.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.994+0000 7f53bfe00640 1 -- 172.21.15.28:0/542553883 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53c010dfa0 msgr2=0x7f53c019ead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:39.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.994+0000 7f53bfe00640 1 --2- 172.21.15.28:0/542553883 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53c010dfa0 0x7f53c019ead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:39.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.994+0000 7f53bfe00640 1 -- 172.21.15.28:0/542553883 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53c019f200 con 0x7f53c0073be0 2024-05-12T22:09:39.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.994+0000 7f53bfe00640 1 --2- 172.21.15.28:0/542553883 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f53c0073be0 0x7f53c019a210 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f53b000d950 tx=0x7f53b000de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:09:39.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.994+0000 7f53bd600640 1 -- 172.21.15.28:0/542553883 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53b0017030 con 0x7f53c0073be0 2024-05-12T22:09:39.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.994+0000 7f53bd600640 1 -- 172.21.15.28:0/542553883 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f53b0004020 con 0x7f53c0073be0 2024-05-12T22:09:39.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.995+0000 7f53c5c00640 1 -- 172.21.15.28:0/542553883 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53c019f4f0 con 0x7f53c0073be0 2024-05-12T22:09:39.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.995+0000 7f53bd600640 1 -- 172.21.15.28:0/542553883 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53b00047a0 con 0x7f53c0073be0 2024-05-12T22:09:39.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.995+0000 7f53c5c00640 1 -- 172.21.15.28:0/542553883 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53c01a6d80 con 0x7f53c0073be0 2024-05-12T22:09:39.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.998+0000 7f53bd600640 1 -- 172.21.15.28:0/542553883 <== mon.2 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 25) ==== 138060+0+0 (secure 0 0 0) 0x7f53b00041d0 con 0x7f53c0073be0 2024-05-12T22:09:39.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.998+0000 7f53ca1a4740 1 Processor -- start 2024-05-12T22:09:39.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.999+0000 7f53ca1a4740 1 -- start start 2024-05-12T22:09:40.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.999+0000 7f53ca1a4740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c132627b60 0x55c132627f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:40.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.999+0000 7f53ca1a4740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c132628480 0x55c132632610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:40.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.999+0000 7f53ca1a4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c132632b50 0x55c132634f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:40.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.999+0000 7f53ca1a4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c13258ca30 con 0x55c132632b50 2024-05-12T22:09:40.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.999+0000 7f53ca1a4740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x55c132534700 con 0x55c132628480 2024-05-12T22:09:40.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.999+0000 7f53ca1a4740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55c1325889f0 con 0x55c132627b60 2024-05-12T22:09:40.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.999+0000 7f53c5200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c132627b60 0x55c132627f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:40.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.999+0000 7f53c5200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c132627b60 0x55c132627f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:34742/0 (socket says 172.21.15.28:34742) 2024-05-12T22:09:40.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.999+0000 7f53c5200640 1 -- 172.21.15.28:0/3388915919 learned_addr learned my addr 172.21.15.28:0/3388915919 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:09:40.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.999+0000 7f53b7e00640 1 --2- 172.21.15.28:0/3388915919 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c132632b50 0x55c132634f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:40.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:39.999+0000 7f53bfe00640 1 --2- 172.21.15.28:0/3388915919 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c132628480 0x55c132632610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:40.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.000+0000 7f53c5200640 1 -- 172.21.15.28:0/3388915919 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c132628480 msgr2=0x55c132632610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:09:40.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.000+0000 7f53c5200640 1 --2- 172.21.15.28:0/3388915919 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c132628480 0x55c132632610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:40.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.000+0000 7f53c5200640 1 -- 172.21.15.28:0/3388915919 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c132632b50 msgr2=0x55c132634f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:09:40.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.000+0000 7f53c5200640 1 --2- 172.21.15.28:0/3388915919 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c132632b50 0x55c132634f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:40.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.000+0000 7f53c5200640 1 -- 172.21.15.28:0/3388915919 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c1322a03f0 con 0x55c132627b60 2024-05-12T22:09:40.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.000+0000 7f53b7e00640 1 --2- 172.21.15.28:0/3388915919 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c132632b50 0x55c132634f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:09:40.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.000+0000 7f53c5200640 1 --2- 172.21.15.28:0/3388915919 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c132627b60 0x55c132627f40 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f53b8009f70 tx=0x7f53b8010800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d3f9305949c9304b server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:09:40.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.000+0000 7f53b5600640 1 -- 172.21.15.28:0/3388915919 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53b8016070 con 0x55c132627b60 2024-05-12T22:09:40.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.000+0000 7f53b5600640 1 -- 172.21.15.28:0/3388915919 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f53b8004d20 con 0x55c132627b60 2024-05-12T22:09:40.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.000+0000 7f53b5600640 1 -- 172.21.15.28:0/3388915919 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53b8012910 con 0x55c132627b60 2024-05-12T22:09:40.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.000+0000 7f53bd600640 1 --2- 172.21.15.28:0/542553883 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f53a40a2ea0 0x7f53a40a5360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:40.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.001+0000 7f53bfe00640 1 --2- 172.21.15.28:0/542553883 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f53a40a2ea0 0x7f53a40a5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:40.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.001+0000 7f53bd600640 1 -- 172.21.15.28:0/542553883 <== mon.2 v2:172.21.15.86:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f53b00ceff0 con 0x7f53c0073be0 2024-05-12T22:09:40.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.001+0000 7f53ca1a4740 1 -- 172.21.15.28:0/3388915919 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c132627b60 msgr2=0x55c132627f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:40.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.001+0000 7f53ca1a4740 1 --2- 172.21.15.28:0/3388915919 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c132627b60 0x55c132627f40 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f53b8009f70 tx=0x7f53b8010800 comp rx=0 tx=0).stop 2024-05-12T22:09:40.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.001+0000 7f53bfe00640 1 --2- 172.21.15.28:0/542553883 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f53a40a2ea0 0x7f53a40a5360 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f53b00cc610 tx=0x7f53b00d2aa0 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:09:40.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.001+0000 7f53ca1a4740 1 -- 172.21.15.28:0/3388915919 shutdown_connections 2024-05-12T22:09:40.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.001+0000 7f53ca1a4740 1 --2- 172.21.15.28:0/3388915919 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c132632b50 0x55c132634f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:40.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.001+0000 7f53ca1a4740 1 --2- 172.21.15.28:0/3388915919 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c132628480 0x55c132632610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:40.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.001+0000 7f53ca1a4740 1 --2- 172.21.15.28:0/3388915919 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c132627b60 0x55c132627f40 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:40.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.001+0000 7f53ca1a4740 1 -- 172.21.15.28:0/3388915919 >> 172.21.15.28:0/3388915919 conn(0x55c132623e80 msgr2=0x55c132624260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:09:40.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.002+0000 7f53ca1a4740 1 -- 172.21.15.28:0/3388915919 shutdown_connections 2024-05-12T22:09:40.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.002+0000 7f53ca1a4740 1 -- 172.21.15.28:0/3388915919 wait complete. 2024-05-12T22:09:40.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.002+0000 7f53ca1a4740 1 -- start start 2024-05-12T22:09:40.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.002+0000 7f53ca1a4740 1 Processor -- start 2024-05-12T22:09:40.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.003+0000 7f53ca1a4740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c132627b60 0x55c13262fb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:40.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.003+0000 7f53ca1a4740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c132628480 0x55c132630080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:40.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.003+0000 7f53ca1a4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c132632b50 0x55c1326305c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:40.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.003+0000 7f53ca1a4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c132635940 con 0x55c132632b50 2024-05-12T22:09:40.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.003+0000 7f53ca1a4740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x55c132547a70 con 0x55c132627b60 2024-05-12T22:09:40.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.003+0000 7f53ca1a4740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55c132635ac0 con 0x55c132628480 2024-05-12T22:09:40.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.003+0000 7f53ca1a4740 10 client.0 _ll_register_callbacks cb 0x55c13252aba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:09:40.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.003+0000 7f53c5200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c132627b60 0x55c13262fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:40.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.003+0000 7f53b7e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c132628480 0x55c132630080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:40.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.003+0000 7f53c5200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c132627b60 0x55c13262fb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:34258/0 (socket says 172.21.15.28:34258) 2024-05-12T22:09:40.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.003+0000 7f53c5200640 1 -- 172.21.15.28:0/1194459583 learned_addr learned my addr 172.21.15.28:0/1194459583 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:09:40.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.003+0000 7f53c5200640 1 --2- 172.21.15.28:0/1194459583 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c132632b50 0x55c1326305c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:40.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.004+0000 7f53b7e00640 1 -- 172.21.15.28:0/1194459583 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c132627b60 msgr2=0x55c13262fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:09:40.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.004+0000 7f53b7e00640 1 --2- 172.21.15.28:0/1194459583 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c132627b60 0x55c13262fb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:40.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.004+0000 7f53b7e00640 1 -- 172.21.15.28:0/1194459583 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c132632b50 msgr2=0x55c1326305c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:09:40.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.004+0000 7f53b7e00640 1 --2- 172.21.15.28:0/1194459583 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c132632b50 0x55c1326305c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:40.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.004+0000 7f53b7e00640 1 -- 172.21.15.28:0/1194459583 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c13236a7d0 con 0x55c132628480 2024-05-12T22:09:40.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.004+0000 7f53b7e00640 1 --2- 172.21.15.28:0/1194459583 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c132628480 0x55c132630080 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f53a802aa80 tx=0x7f53a80043b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3ea40b2b8d238246 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:09:40.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.004+0000 7f53ae000640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-12T22:09:40.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.004+0000 7f53ae000640 1 -- 172.21.15.28:0/1194459583 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53a8035040 con 0x55c132628480 2024-05-12T22:09:40.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.004+0000 7f53ae000640 1 -- 172.21.15.28:0/1194459583 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f53a8005450 con 0x55c132628480 2024-05-12T22:09:40.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.004+0000 7f53ca1a4740 1 -- 172.21.15.28:0/1194459583 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f53b8004d20 con 0x55c132628480 2024-05-12T22:09:40.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.005+0000 7f53ae000640 1 -- 172.21.15.28:0/1194459583 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f53a8005650 con 0x55c132628480 2024-05-12T22:09:40.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.005+0000 7f53ca1a4740 10 client.25477 fetch_fsmap learned FSMap version 24 2024-05-12T22:09:40.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.005+0000 7f53ca1a4740 1 -- 172.21.15.28:0/1194459583 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55c132641c40 con 0x55c132628480 2024-05-12T22:09:40.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.005+0000 7f53ae000640 1 -- 172.21.15.28:0/1194459583 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f53b8004d20 con 0x55c132628480 2024-05-12T22:09:40.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.005+0000 7f53ae000640 1 -- 172.21.15.28:0/1194459583 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f53a8003a30 con 0x55c132628480 2024-05-12T22:09:40.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.005+0000 7f53ca1a4740 10 client.25477 fetch_fsmap finished waiting for FSMap version 24 2024-05-12T22:09:40.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.005+0000 7f53ca1a4740 10 client.25477 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-05-12T22:09:40.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.005+0000 7f53ca1a4740 20 client.25477 populate_metadata read hostname 'smithi028' 2024-05-12T22:09:40.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.005+0000 7f53ca1a4740 1 --2- 172.21.15.28:0/1194459583 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x55c132643100 0x55c132645520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:40.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.005+0000 7f53ca1a4740 4 client.25477 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-12T22:09:40.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.005+0000 7f53ca1a4740 1 -- 172.21.15.28:0/1194459583 --> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c13259a160 con 0x55c132643100 2024-05-12T22:09:40.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.006+0000 7f53c5200640 1 --2- 172.21.15.28:0/1194459583 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x55c132643100 0x55c132645520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:40.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.006+0000 7f53c5200640 1 --2- 172.21.15.28:0/1194459583 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x55c132643100 0x55c132645520 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=6096a425818c4ea9 server_cookie=39a064afbbe3e949 in_seq=0 out_seq=0 2024-05-12T22:09:40.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.006+0000 7f53ae000640 10 client.25477 ms_handle_connect on v2:172.21.15.28:6840/761183870 2024-05-12T22:09:40.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.007+0000 7f53ae000640 1 -- 172.21.15.28:0/1194459583 <== mds.1 v2:172.21.15.28:6840/761183870 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c13259a160 con 0x55c132643100 2024-05-12T22:09:40.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.007+0000 7f53ae000640 10 client.25477 handle_command_reply: tid=0 2024-05-12T22:09:40.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.007+0000 7f53ca1a4740 1 client.25477 shutdown 2024-05-12T22:09:40.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.007+0000 7f53ca1a4740 1 -- 172.21.15.28:0/1194459583 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c132628480 msgr2=0x55c132630080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:40.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.007+0000 7f53ca1a4740 1 --2- 172.21.15.28:0/1194459583 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c132628480 0x55c132630080 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f53a802aa80 tx=0x7f53a80043b0 comp rx=0 tx=0).stop 2024-05-12T22:09:40.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.007+0000 7f53ca1a4740 1 -- 172.21.15.28:0/1194459583 shutdown_connections 2024-05-12T22:09:40.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.007+0000 7f53ca1a4740 1 --2- 172.21.15.28:0/1194459583 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x55c132643100 0x55c132645520 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:40.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.007+0000 7f53ca1a4740 1 --2- 172.21.15.28:0/1194459583 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c132632b50 0x55c1326305c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:40.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.007+0000 7f53ca1a4740 1 --2- 172.21.15.28:0/1194459583 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c132628480 0x55c132630080 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:40.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.007+0000 7f53ca1a4740 1 --2- 172.21.15.28:0/1194459583 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c132627b60 0x55c13262fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:40.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.007+0000 7f53ae000640 0 client.25477 ms_handle_reset on v2:172.21.15.28:6840/761183870 2024-05-12T22:09:40.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.007+0000 7f53ca1a4740 1 -- 172.21.15.28:0/1194459583 >> 172.21.15.28:0/1194459583 conn(0x55c132623e80 msgr2=0x55c132629c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:09:40.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.008+0000 7f53ca1a4740 1 -- 172.21.15.28:0/1194459583 shutdown_connections 2024-05-12T22:09:40.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.008+0000 7f53ca1a4740 1 -- 172.21.15.28:0/1194459583 wait complete. 2024-05-12T22:09:40.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.008+0000 7f53ca1a4740 20 client.25477 trim_cache size 0 max 16384 2024-05-12T22:09:40.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.023+0000 7f53ca1a4740 1 Processor -- start 2024-05-12T22:09:40.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.023+0000 7f53ca1a4740 1 -- start start 2024-05-12T22:09:40.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.023+0000 7f53ca1a4740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c132627b60 0x55c13263e800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:40.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.023+0000 7f53ca1a4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c132628480 0x55c1326303e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:40.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.023+0000 7f53ca1a4740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c132632b50 0x55c132629a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:40.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.023+0000 7f53ca1a4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f53c01b1eb0 con 0x55c132628480 2024-05-12T22:09:40.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.023+0000 7f53ca1a4740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f53c01af780 con 0x55c132627b60 2024-05-12T22:09:40.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.023+0000 7f53ca1a4740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f53c01ad5d0 con 0x55c132632b50 2024-05-12T22:09:40.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.024+0000 7f53b7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c132628480 0x55c1326303e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:40.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.024+0000 7f53b7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c132628480 0x55c1326303e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50912/0 (socket says 172.21.15.28:50912) 2024-05-12T22:09:40.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.024+0000 7f53b7e00640 1 -- 172.21.15.28:0/1610615935 learned_addr learned my addr 172.21.15.28:0/1610615935 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:09:40.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.024+0000 7f53c5200640 1 --2- 172.21.15.28:0/1610615935 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c132627b60 0x55c13263e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:40.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.024+0000 7f53c5200640 1 --2- 172.21.15.28:0/1610615935 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c132632b50 0x55c132629a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:40.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.024+0000 7f53b7e00640 1 -- 172.21.15.28:0/1610615935 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c132632b50 msgr2=0x55c132629a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:09:40.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.024+0000 7f53b7e00640 1 --2- 172.21.15.28:0/1610615935 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c132632b50 0x55c132629a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:40.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.024+0000 7f53b7e00640 1 -- 172.21.15.28:0/1610615935 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c132627b60 msgr2=0x55c13263e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:09:40.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.024+0000 7f53b7e00640 1 --2- 172.21.15.28:0/1610615935 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c132627b60 0x55c13263e800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:40.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.024+0000 7f53b7e00640 1 -- 172.21.15.28:0/1610615935 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c13262e0b0 con 0x55c132628480 2024-05-12T22:09:40.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.025+0000 7f53b7e00640 1 --2- 172.21.15.28:0/1610615935 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c132628480 0x55c1326303e0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f53a8006010 tx=0x7f53a8003fa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e724a5ac9167030 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:09:40.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.025+0000 7f53b4c00640 1 -- 172.21.15.28:0/1610615935 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53a8035030 con 0x55c132628480 2024-05-12T22:09:40.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.025+0000 7f53b4c00640 1 -- 172.21.15.28:0/1610615935 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f53a8040070 con 0x55c132628480 2024-05-12T22:09:40.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.025+0000 7f53b4c00640 1 -- 172.21.15.28:0/1610615935 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53a8052070 con 0x55c132628480 2024-05-12T22:09:40.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.026+0000 7f53ca1a4740 1 -- 172.21.15.28:0/1610615935 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c132628480 msgr2=0x55c1326303e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:40.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.026+0000 7f53ca1a4740 1 --2- 172.21.15.28:0/1610615935 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c132628480 0x55c1326303e0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f53a8006010 tx=0x7f53a8003fa0 comp rx=0 tx=0).stop 2024-05-12T22:09:40.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.026+0000 7f53ca1a4740 1 -- 172.21.15.28:0/1610615935 shutdown_connections 2024-05-12T22:09:40.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.026+0000 7f53ca1a4740 1 --2- 172.21.15.28:0/1610615935 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c132632b50 0x55c132629a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:40.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.026+0000 7f53ca1a4740 1 --2- 172.21.15.28:0/1610615935 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c132628480 0x55c1326303e0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:40.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.026+0000 7f53ca1a4740 1 --2- 172.21.15.28:0/1610615935 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c132627b60 0x55c13263e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:40.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.026+0000 7f53ca1a4740 1 -- 172.21.15.28:0/1610615935 >> 172.21.15.28:0/1610615935 conn(0x55c132623e80 msgr2=0x55c1326292f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:09:40.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.026+0000 7f53ca1a4740 1 -- 172.21.15.28:0/1610615935 shutdown_connections 2024-05-12T22:09:40.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.026+0000 7f53ca1a4740 1 -- 172.21.15.28:0/1610615935 wait complete. 2024-05-12T22:09:40.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.026+0000 7f53ca1a4740 1 -- start start 2024-05-12T22:09:40.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.026+0000 7f53ca1a4740 1 Processor -- start 2024-05-12T22:09:40.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.027+0000 7f53ca1a4740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c132627b60 0x55c13262c900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:40.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.027+0000 7f53ca1a4740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c132628480 0x55c13262ce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:40.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.027+0000 7f53ca1a4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c132632b50 0x55c13262d380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:40.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.027+0000 7f53ca1a4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f53c01adc40 con 0x55c132632b50 2024-05-12T22:09:40.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.027+0000 7f53ca1a4740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f53c01af0b0 con 0x55c132627b60 2024-05-12T22:09:40.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.027+0000 7f53ca1a4740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f53c01ae310 con 0x55c132628480 2024-05-12T22:09:40.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.027+0000 7f53ca1a4740 10 client.0 _ll_register_callbacks cb 0x7f53a8004040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:09:40.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.027+0000 7f53c5200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c132627b60 0x55c13262c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:40.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.027+0000 7f53b7e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c132628480 0x55c13262ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:40.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.027+0000 7f53c5200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c132627b60 0x55c13262c900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:34266/0 (socket says 172.21.15.28:34266) 2024-05-12T22:09:40.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.027+0000 7f53c5200640 1 -- 172.21.15.28:0/2119392521 learned_addr learned my addr 172.21.15.28:0/2119392521 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:09:40.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.027+0000 7f53c5200640 1 --2- 172.21.15.28:0/2119392521 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c132632b50 0x55c13262d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:40.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.028+0000 7f53b7e00640 1 -- 172.21.15.28:0/2119392521 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c132627b60 msgr2=0x55c13262c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:09:40.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.028+0000 7f53b7e00640 1 --2- 172.21.15.28:0/2119392521 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c132627b60 0x55c13262c900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:40.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.028+0000 7f53b7e00640 1 -- 172.21.15.28:0/2119392521 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c132632b50 msgr2=0x55c13262d380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:09:40.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.028+0000 7f53b7e00640 1 --2- 172.21.15.28:0/2119392521 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c132632b50 0x55c13262d380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:40.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.028+0000 7f53b7e00640 1 -- 172.21.15.28:0/2119392521 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c1326a8760 con 0x55c132628480 2024-05-12T22:09:40.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.028+0000 7f53b7e00640 1 --2- 172.21.15.28:0/2119392521 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c132628480 0x55c13262ce40 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x55c132630c80 tx=0x7f53a8031640 comp rx=0 tx=0).ready entity=mon.1 client_cookie=236ba424d21cc7ae server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:09:40.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.028+0000 7f53ae000640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-12T22:09:40.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.028+0000 7f53ae000640 1 -- 172.21.15.28:0/2119392521 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53a8052070 con 0x55c132628480 2024-05-12T22:09:40.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.029+0000 7f53ae000640 1 -- 172.21.15.28:0/2119392521 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f53a8003a00 con 0x55c132628480 2024-05-12T22:09:40.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.029+0000 7f53ca1a4740 1 -- 172.21.15.28:0/2119392521 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f53a8040070 con 0x55c132628480 2024-05-12T22:09:40.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.033+0000 7f53ae000640 1 -- 172.21.15.28:0/2119392521 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f53a8035070 con 0x55c132628480 2024-05-12T22:09:40.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.033+0000 7f53ca1a4740 10 client.25483 fetch_fsmap learned FSMap version 24 2024-05-12T22:09:40.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.033+0000 7f53ca1a4740 1 -- 172.21.15.28:0/2119392521 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55c1326a9a00 con 0x55c132628480 2024-05-12T22:09:40.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.033+0000 7f53ae000640 1 -- 172.21.15.28:0/2119392521 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f53a8040070 con 0x55c132628480 2024-05-12T22:09:40.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.033+0000 7f53ae000640 1 -- 172.21.15.28:0/2119392521 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f53a80047c0 con 0x55c132628480 2024-05-12T22:09:40.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.033+0000 7f53ca1a4740 10 client.25483 fetch_fsmap finished waiting for FSMap version 24 2024-05-12T22:09:40.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.033+0000 7f53ca1a4740 10 client.25483 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-05-12T22:09:40.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.033+0000 7f53ca1a4740 20 client.25483 populate_metadata read hostname 'smithi028' 2024-05-12T22:09:40.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.033+0000 7f53ca1a4740 1 --2- 172.21.15.28:0/2119392521 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x55c1326aa000 0x55c1326ac420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:40.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.033+0000 7f53ca1a4740 4 client.25483 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-12T22:09:40.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.033+0000 7f53ca1a4740 1 -- 172.21.15.28:0/2119392521 --> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c1325944b0 con 0x55c1326aa000 2024-05-12T22:09:40.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.034+0000 7f53c5200640 1 --2- 172.21.15.28:0/2119392521 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x55c1326aa000 0x55c1326ac420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:40.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.034+0000 7f53c5200640 1 --2- 172.21.15.28:0/2119392521 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x55c1326aa000 0x55c1326ac420 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=863dd3f9f15a8982 server_cookie=a2ae42aa327ef733 in_seq=0 out_seq=0 2024-05-12T22:09:40.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.034+0000 7f53ae000640 10 client.25483 ms_handle_connect on v2:172.21.15.28:6840/761183870 2024-05-12T22:09:40.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.034+0000 7f53ae000640 1 -- 172.21.15.28:0/2119392521 <== mds.1 v2:172.21.15.28:6840/761183870 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c1325944b0 con 0x55c1326aa000 2024-05-12T22:09:40.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.034+0000 7f53ae000640 10 client.25483 handle_command_reply: tid=0 2024-05-12T22:09:40.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.034+0000 7f53ca1a4740 1 client.25483 shutdown 2024-05-12T22:09:40.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.035+0000 7f53ca1a4740 1 -- 172.21.15.28:0/2119392521 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c132628480 msgr2=0x55c13262ce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:40.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.035+0000 7f53ca1a4740 1 --2- 172.21.15.28:0/2119392521 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c132628480 0x55c13262ce40 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x55c132630c80 tx=0x7f53a8031640 comp rx=0 tx=0).stop 2024-05-12T22:09:40.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.035+0000 7f53ca1a4740 1 -- 172.21.15.28:0/2119392521 shutdown_connections 2024-05-12T22:09:40.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.035+0000 7f53ca1a4740 1 --2- 172.21.15.28:0/2119392521 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x55c1326aa000 0x55c1326ac420 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:40.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.035+0000 7f53ca1a4740 1 --2- 172.21.15.28:0/2119392521 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c132632b50 0x55c13262d380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:40.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.035+0000 7f53ca1a4740 1 --2- 172.21.15.28:0/2119392521 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c132628480 0x55c13262ce40 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:40.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.035+0000 7f53ca1a4740 1 --2- 172.21.15.28:0/2119392521 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c132627b60 0x55c13262c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:40.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.035+0000 7f53ae000640 0 client.25483 ms_handle_reset on v2:172.21.15.28:6840/761183870 2024-05-12T22:09:40.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.035+0000 7f53ca1a4740 1 -- 172.21.15.28:0/2119392521 >> 172.21.15.28:0/2119392521 conn(0x55c132623e80 msgr2=0x55c132629b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:09:40.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.035+0000 7f53ca1a4740 1 -- 172.21.15.28:0/2119392521 shutdown_connections 2024-05-12T22:09:40.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.035+0000 7f53ca1a4740 1 -- 172.21.15.28:0/2119392521 wait complete. 2024-05-12T22:09:40.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.035+0000 7f53ca1a4740 20 client.25483 trim_cache size 0 max 16384 2024-05-12T22:09:40.037 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-05-12T22:09:40.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.036+0000 7f53c5c00640 1 -- 172.21.15.28:0/542553883 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f53a40a2ea0 msgr2=0x7f53a40a5360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:40.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.036+0000 7f53c5c00640 1 --2- 172.21.15.28:0/542553883 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f53a40a2ea0 0x7f53a40a5360 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f53b00cc610 tx=0x7f53b00d2aa0 comp rx=0 tx=0).stop 2024-05-12T22:09:40.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.036+0000 7f53c5c00640 1 -- 172.21.15.28:0/542553883 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f53c0073be0 msgr2=0x7f53c019a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:40.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.036+0000 7f53c5c00640 1 --2- 172.21.15.28:0/542553883 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f53c0073be0 0x7f53c019a210 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f53b000d950 tx=0x7f53b000de10 comp rx=0 tx=0).stop 2024-05-12T22:09:40.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.036+0000 7f53c5c00640 1 -- 172.21.15.28:0/542553883 shutdown_connections 2024-05-12T22:09:40.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.036+0000 7f53c5c00640 1 --2- 172.21.15.28:0/542553883 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f53a40a2ea0 0x7f53a40a5360 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:40.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.036+0000 7f53c5c00640 1 --2- 172.21.15.28:0/542553883 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53c010dfa0 0x7f53c019ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:40.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.036+0000 7f53c5c00640 1 --2- 172.21.15.28:0/542553883 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f53c0074500 0x7f53c019a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:40.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.036+0000 7f53c5c00640 1 --2- 172.21.15.28:0/542553883 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f53c0073be0 0x7f53c019a210 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:40.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.036+0000 7f53c5c00640 1 -- 172.21.15.28:0/542553883 >> 172.21.15.28:0/542553883 conn(0x7f53c00fd1d0 msgr2=0x7f53c00fe960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:09:40.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.036+0000 7f53c5c00640 1 -- 172.21.15.28:0/542553883 shutdown_connections 2024-05-12T22:09:40.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.036+0000 7f53c5c00640 1 -- 172.21.15.28:0/542553883 wait complete. 2024-05-12T22:09:40.056 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-05-12T22:09:40.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.271+0000 7f6fbfe00640 1 -- 172.21.15.28:0/4068501983 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6fc010e050 msgr2=0x7f6fc0110440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:40.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.271+0000 7f6fbfe00640 1 --2- 172.21.15.28:0/4068501983 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6fc010e050 0x7f6fc0110440 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f6fb8009860 tx=0x7f6fb802a340 comp rx=0 tx=0).stop 2024-05-12T22:09:40.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.271+0000 7f6fbfe00640 1 -- 172.21.15.28:0/4068501983 shutdown_connections 2024-05-12T22:09:40.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.271+0000 7f6fbfe00640 1 --2- 172.21.15.28:0/4068501983 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6fc010e050 0x7f6fc0110440 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:40.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.271+0000 7f6fbfe00640 1 --2- 172.21.15.28:0/4068501983 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6fc0100e50 0x7f6fc010da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:40.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.271+0000 7f6fbfe00640 1 --2- 172.21.15.28:0/4068501983 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6fc01004a0 0x7f6fc0100880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:40.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.271+0000 7f6fbfe00640 1 -- 172.21.15.28:0/4068501983 >> 172.21.15.28:0/4068501983 conn(0x7f6fc00fc0b0 msgr2=0x7f6fc00fe4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:09:40.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.272+0000 7f6fbfe00640 1 -- 172.21.15.28:0/4068501983 shutdown_connections 2024-05-12T22:09:40.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.272+0000 7f6fbfe00640 1 -- 172.21.15.28:0/4068501983 wait complete. 2024-05-12T22:09:40.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.273+0000 7f6fbfe00640 1 Processor -- start 2024-05-12T22:09:40.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.273+0000 7f6fbfe00640 1 -- start start 2024-05-12T22:09:40.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.273+0000 7f6fbfe00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6fc01004a0 0x7f6fc019a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:40.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.273+0000 7f6fbfe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6fc0100e50 0x7f6fc019a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:40.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.273+0000 7f6fbfe00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6fc010e050 0x7f6fc019ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:40.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.273+0000 7f6fbfe00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6fc0113cc0 con 0x7f6fc0100e50 2024-05-12T22:09:40.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.273+0000 7f6fbfe00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f6fc0113b40 con 0x7f6fc010e050 2024-05-12T22:09:40.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.273+0000 7f6fbfe00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f6fc0113e40 con 0x7f6fc01004a0 2024-05-12T22:09:40.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.273+0000 7f6fbe000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6fc0100e50 0x7f6fc019a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:40.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.274+0000 7f6fbe000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6fc0100e50 0x7f6fc019a6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50950/0 (socket says 172.21.15.28:50950) 2024-05-12T22:09:40.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.274+0000 7f6fbe000640 1 -- 172.21.15.28:0/1805217660 learned_addr learned my addr 172.21.15.28:0/1805217660 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:09:40.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.274+0000 7f6fbf400640 1 --2- 172.21.15.28:0/1805217660 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6fc010e050 0x7f6fc019ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:40.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.274+0000 7f6fbea00640 1 --2- 172.21.15.28:0/1805217660 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6fc01004a0 0x7f6fc019a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:40.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.274+0000 7f6fbe000640 1 -- 172.21.15.28:0/1805217660 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6fc01004a0 msgr2=0x7f6fc019a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:40.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.274+0000 7f6fbe000640 1 --2- 172.21.15.28:0/1805217660 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6fc01004a0 0x7f6fc019a1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:40.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.274+0000 7f6fbe000640 1 -- 172.21.15.28:0/1805217660 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6fc010e050 msgr2=0x7f6fc019ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:40.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.274+0000 7f6fbe000640 1 --2- 172.21.15.28:0/1805217660 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6fc010e050 0x7f6fc019ea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:40.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.274+0000 7f6fbe000640 1 -- 172.21.15.28:0/1805217660 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fc019f190 con 0x7f6fc0100e50 2024-05-12T22:09:40.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.274+0000 7f6fbe000640 1 --2- 172.21.15.28:0/1805217660 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6fc0100e50 0x7f6fc019a6e0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f6fa800e9c0 tx=0x7f6fa800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:09:40.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.275+0000 7f6fb7400640 1 -- 172.21.15.28:0/1805217660 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fa800ccc0 con 0x7f6fc0100e50 2024-05-12T22:09:40.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.275+0000 7f6fb7400640 1 -- 172.21.15.28:0/1805217660 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6fa8005500 con 0x7f6fc0100e50 2024-05-12T22:09:40.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.275+0000 7f6fb7400640 1 -- 172.21.15.28:0/1805217660 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fa8010430 con 0x7f6fc0100e50 2024-05-12T22:09:40.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.275+0000 7f6fbfe00640 1 -- 172.21.15.28:0/1805217660 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6fc019f480 con 0x7f6fc0100e50 2024-05-12T22:09:40.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.275+0000 7f6fbfe00640 1 -- 172.21.15.28:0/1805217660 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6fc01a6d10 con 0x7f6fc0100e50 2024-05-12T22:09:40.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.278+0000 7f6fc8bc2740 1 Processor -- start 2024-05-12T22:09:40.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.278+0000 7f6fc8bc2740 1 -- start start 2024-05-12T22:09:40.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.278+0000 7f6fc8bc2740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562afa015b60 0x562afa015f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:40.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.278+0000 7f6fb7400640 1 -- 172.21.15.28:0/1805217660 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138060+0+0 (secure 0 0 0) 0x7f6fa80046c0 con 0x7f6fc0100e50 2024-05-12T22:09:40.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.278+0000 7f6fc8bc2740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562afa016480 0x562afa020610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:40.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.279+0000 7f6fc8bc2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562afa020b50 0x562afa022f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:40.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.279+0000 7f6fc8bc2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562af9f7aa30 con 0x562afa020b50 2024-05-12T22:09:40.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.279+0000 7f6fc8bc2740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x562af9f22700 con 0x562afa016480 2024-05-12T22:09:40.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.279+0000 7f6fc8bc2740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x562af9f769f0 con 0x562afa015b60 2024-05-12T22:09:40.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.279+0000 7f6fbe000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562afa020b50 0x562afa022f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:40.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.279+0000 7f6fbea00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562afa016480 0x562afa020610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:40.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.279+0000 7f6fbf400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562afa015b60 0x562afa015f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:40.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.279+0000 7f6fbea00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562afa016480 0x562afa020610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:34298/0 (socket says 172.21.15.28:34298) 2024-05-12T22:09:40.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.279+0000 7f6fbea00640 1 -- 172.21.15.28:0/3016918467 learned_addr learned my addr 172.21.15.28:0/3016918467 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:09:40.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.279+0000 7f6fbea00640 1 -- 172.21.15.28:0/3016918467 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562afa015b60 msgr2=0x562afa015f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:09:40.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.279+0000 7f6fbea00640 1 --2- 172.21.15.28:0/3016918467 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562afa015b60 0x562afa015f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:40.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.279+0000 7f6fbea00640 1 -- 172.21.15.28:0/3016918467 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562afa020b50 msgr2=0x562afa022f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:09:40.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.279+0000 7f6fbea00640 1 --2- 172.21.15.28:0/3016918467 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562afa020b50 0x562afa022f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:40.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.279+0000 7f6fbea00640 1 -- 172.21.15.28:0/3016918467 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562af9c8e3f0 con 0x562afa016480 2024-05-12T22:09:40.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.280+0000 7f6fbea00640 1 --2- 172.21.15.28:0/3016918467 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562afa016480 0x562afa020610 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f6fb0010400 tx=0x7f6fb00108c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4ecdc316124ffdca server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:09:40.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.280+0000 7f6fafe00640 1 -- 172.21.15.28:0/3016918467 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fb0003b70 con 0x562afa016480 2024-05-12T22:09:40.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.280+0000 7f6fafe00640 1 -- 172.21.15.28:0/3016918467 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6fb0003d10 con 0x562afa016480 2024-05-12T22:09:40.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.280+0000 7f6fafe00640 1 -- 172.21.15.28:0/3016918467 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fb0011b40 con 0x562afa016480 2024-05-12T22:09:40.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.280+0000 7f6fb7400640 1 --2- 172.21.15.28:0/1805217660 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f6fa40a2ea0 0x7f6fa40a5360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:40.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.281+0000 7f6fbf400640 1 --2- 172.21.15.28:0/1805217660 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f6fa40a2ea0 0x7f6fa40a5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:40.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.281+0000 7f6fb7400640 1 -- 172.21.15.28:0/1805217660 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f6fa80cf9e0 con 0x7f6fc0100e50 2024-05-12T22:09:40.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.281+0000 7f6fc8bc2740 1 -- 172.21.15.28:0/3016918467 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562afa016480 msgr2=0x562afa020610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:40.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.281+0000 7f6fc8bc2740 1 --2- 172.21.15.28:0/3016918467 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562afa016480 0x562afa020610 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f6fb0010400 tx=0x7f6fb00108c0 comp rx=0 tx=0).stop 2024-05-12T22:09:40.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.281+0000 7f6fc8bc2740 1 -- 172.21.15.28:0/3016918467 shutdown_connections 2024-05-12T22:09:40.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.281+0000 7f6fc8bc2740 1 --2- 172.21.15.28:0/3016918467 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562afa020b50 0x562afa022f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:40.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.281+0000 7f6fc8bc2740 1 --2- 172.21.15.28:0/3016918467 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562afa016480 0x562afa020610 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:40.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.281+0000 7f6fc8bc2740 1 --2- 172.21.15.28:0/3016918467 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562afa015b60 0x562afa015f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:40.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.281+0000 7f6fc8bc2740 1 -- 172.21.15.28:0/3016918467 >> 172.21.15.28:0/3016918467 conn(0x562afa011e80 msgr2=0x562afa012260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:09:40.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.281+0000 7f6fc8bc2740 1 -- 172.21.15.28:0/3016918467 shutdown_connections 2024-05-12T22:09:40.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.281+0000 7f6fbf400640 1 --2- 172.21.15.28:0/1805217660 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f6fa40a2ea0 0x7f6fa40a5360 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f6fb802a860 tx=0x7f6fb80056f0 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:09:40.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.281+0000 7f6fc8bc2740 1 -- 172.21.15.28:0/3016918467 wait complete. 2024-05-12T22:09:40.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.282+0000 7f6fc8bc2740 1 -- start start 2024-05-12T22:09:40.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.282+0000 7f6fc8bc2740 1 Processor -- start 2024-05-12T22:09:40.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.282+0000 7f6fc8bc2740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562afa015b60 0x562afa01dae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:40.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.282+0000 7f6fc8bc2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562afa016480 0x562afa01e020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:40.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.282+0000 7f6fc8bc2740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562afa020b50 0x562afa01e560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:40.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.282+0000 7f6fc8bc2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562afa023940 con 0x562afa016480 2024-05-12T22:09:40.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.282+0000 7f6fc8bc2740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x562af9f35a70 con 0x562afa015b60 2024-05-12T22:09:40.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.282+0000 7f6fc8bc2740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x562afa023ac0 con 0x562afa020b50 2024-05-12T22:09:40.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.283+0000 7f6fc8bc2740 10 client.0 _ll_register_callbacks cb 0x562af9f18ba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:09:40.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.283+0000 7f6fbe000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562afa016480 0x562afa01e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:40.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.283+0000 7f6fbe000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562afa016480 0x562afa01e020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50960/0 (socket says 172.21.15.28:50960) 2024-05-12T22:09:40.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.283+0000 7f6fbea00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562afa015b60 0x562afa01dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:40.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.283+0000 7f6fbe000640 1 -- 172.21.15.28:0/3827825339 learned_addr learned my addr 172.21.15.28:0/3827825339 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:09:40.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.283+0000 7f6fbf400640 1 --2- 172.21.15.28:0/3827825339 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562afa020b50 0x562afa01e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:40.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.283+0000 7f6fbea00640 1 -- 172.21.15.28:0/3827825339 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562afa020b50 msgr2=0x562afa01e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:09:40.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.283+0000 7f6fbea00640 1 --2- 172.21.15.28:0/3827825339 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562afa020b50 0x562afa01e560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:40.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.283+0000 7f6fbea00640 1 -- 172.21.15.28:0/3827825339 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562afa016480 msgr2=0x562afa01e020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:09:40.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.283+0000 7f6fbea00640 1 --2- 172.21.15.28:0/3827825339 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562afa016480 0x562afa01e020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:40.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.283+0000 7f6fbea00640 1 -- 172.21.15.28:0/3827825339 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562af9d587d0 con 0x562afa015b60 2024-05-12T22:09:40.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.283+0000 7f6fbe000640 1 --2- 172.21.15.28:0/3827825339 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562afa016480 0x562afa01e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:09:40.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.284+0000 7f6fbea00640 1 --2- 172.21.15.28:0/3827825339 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562afa015b60 0x562afa01dae0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f6fb0000f80 tx=0x7f6fb0004c70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=35faab28e00c2f85 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:09:40.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.284+0000 7f6facc00640 10 client.0 ms_handle_connect on v2:172.21.15.86:3300/0 2024-05-12T22:09:40.288 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:40 smithi028 ceph-mon[29847]: pgmap v23: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:09:40.288 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:40 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/1761328157' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:09:40.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.284+0000 7f6facc00640 1 -- 172.21.15.28:0/3827825339 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fb0004e60 con 0x562afa015b60 2024-05-12T22:09:40.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.284+0000 7f6facc00640 1 -- 172.21.15.28:0/3827825339 <== mon.2 v2:172.21.15.86:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f6fb0011c50 con 0x562afa015b60 2024-05-12T22:09:40.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.284+0000 7f6fc8bc2740 1 -- 172.21.15.28:0/3827825339 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6fb0003d10 con 0x562afa015b60 2024-05-12T22:09:40.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.284+0000 7f6facc00640 1 -- 172.21.15.28:0/3827825339 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f6fb002b050 con 0x562afa015b60 2024-05-12T22:09:40.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.285+0000 7f6fc8bc2740 10 client.25523 fetch_fsmap learned FSMap version 24 2024-05-12T22:09:40.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.285+0000 7f6fc8bc2740 1 -- 172.21.15.28:0/3827825339 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x562afa02fbe0 con 0x562afa015b60 2024-05-12T22:09:40.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.285+0000 7f6facc00640 1 -- 172.21.15.28:0/3827825339 <== mon.2 v2:172.21.15.86:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6fb0003d10 con 0x562afa015b60 2024-05-12T22:09:40.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.285+0000 7f6facc00640 1 -- 172.21.15.28:0/3827825339 <== mon.2 v2:172.21.15.86:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f6fb0005040 con 0x562afa015b60 2024-05-12T22:09:40.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.285+0000 7f6fc8bc2740 10 client.25523 fetch_fsmap finished waiting for FSMap version 24 2024-05-12T22:09:40.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.285+0000 7f6fc8bc2740 10 client.25523 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:09:40.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.285+0000 7f6fc8bc2740 20 client.25523 populate_metadata read hostname 'smithi028' 2024-05-12T22:09:40.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.285+0000 7f6fc8bc2740 1 --2- 172.21.15.28:0/3827825339 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x562afa031040 0x562afa033430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:40.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.285+0000 7f6fc8bc2740 4 client.25523 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-12T22:09:40.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.285+0000 7f6fc8bc2740 1 -- 172.21.15.28:0/3827825339 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562af9f88160 con 0x562afa031040 2024-05-12T22:09:40.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.286+0000 7f6fbe000640 1 --2- 172.21.15.28:0/3827825339 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x562afa031040 0x562afa033430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:40.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.286+0000 7f6fbe000640 1 --2- 172.21.15.28:0/3827825339 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x562afa031040 0x562afa033430 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9e50c796997e6496 server_cookie=90ca4165d9f0798c in_seq=0 out_seq=0 2024-05-12T22:09:40.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.286+0000 7f6facc00640 10 client.25523 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:09:40.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.287+0000 7f6facc00640 1 -- 172.21.15.28:0/3827825339 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562af9f88160 con 0x562afa031040 2024-05-12T22:09:40.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.287+0000 7f6facc00640 10 client.25523 handle_command_reply: tid=0 2024-05-12T22:09:40.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.287+0000 7f6fc8bc2740 1 client.25523 shutdown 2024-05-12T22:09:40.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.287+0000 7f6fc8bc2740 1 -- 172.21.15.28:0/3827825339 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562afa015b60 msgr2=0x562afa01dae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:40.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.287+0000 7f6fc8bc2740 1 --2- 172.21.15.28:0/3827825339 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562afa015b60 0x562afa01dae0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f6fb0000f80 tx=0x7f6fb0004c70 comp rx=0 tx=0).stop 2024-05-12T22:09:40.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.287+0000 7f6fc8bc2740 1 -- 172.21.15.28:0/3827825339 shutdown_connections 2024-05-12T22:09:40.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.287+0000 7f6fc8bc2740 1 --2- 172.21.15.28:0/3827825339 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x562afa031040 0x562afa033430 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:40.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.287+0000 7f6fc8bc2740 1 --2- 172.21.15.28:0/3827825339 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562afa020b50 0x562afa01e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:40.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.287+0000 7f6fc8bc2740 1 --2- 172.21.15.28:0/3827825339 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562afa016480 0x562afa01e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:40.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.287+0000 7f6facc00640 0 client.25523 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:09:40.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.287+0000 7f6fc8bc2740 1 --2- 172.21.15.28:0/3827825339 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562afa015b60 0x562afa01dae0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:40.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.287+0000 7f6fc8bc2740 1 -- 172.21.15.28:0/3827825339 >> 172.21.15.28:0/3827825339 conn(0x562afa011e80 msgr2=0x562afa017c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:09:40.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.287+0000 7f6fc8bc2740 1 -- 172.21.15.28:0/3827825339 shutdown_connections 2024-05-12T22:09:40.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.287+0000 7f6fc8bc2740 1 -- 172.21.15.28:0/3827825339 wait complete. 2024-05-12T22:09:40.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.288+0000 7f6fc8bc2740 20 client.25523 trim_cache size 0 max 16384 2024-05-12T22:09:40.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.303+0000 7f6fc8bc2740 1 Processor -- start 2024-05-12T22:09:40.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.303+0000 7f6fc8bc2740 1 -- start start 2024-05-12T22:09:40.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.303+0000 7f6fc8bc2740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562afa015b60 0x562afa01ad40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:40.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.303+0000 7f6fc8bc2740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562afa016480 0x562afa01b280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:40.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.303+0000 7f6fc8bc2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562afa020b50 0x562afa010770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:40.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.303+0000 7f6fc8bc2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6fc01b1e60 con 0x562afa020b50 2024-05-12T22:09:40.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.303+0000 7f6fc8bc2740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f6fc01af730 con 0x562afa015b60 2024-05-12T22:09:40.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.303+0000 7f6fc8bc2740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f6fc01ad580 con 0x562afa016480 2024-05-12T22:09:40.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.303+0000 7f6fbea00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562afa015b60 0x562afa01ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:40.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.303+0000 7f6fbf400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562afa020b50 0x562afa010770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:40.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.304+0000 7f6fbe000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562afa016480 0x562afa01b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:40.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.304+0000 7f6fbe000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562afa016480 0x562afa01b280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:34812/0 (socket says 172.21.15.28:34812) 2024-05-12T22:09:40.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.304+0000 7f6fbe000640 1 -- 172.21.15.28:0/675579725 learned_addr learned my addr 172.21.15.28:0/675579725 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:09:40.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.304+0000 7f6fbf400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562afa020b50 0x562afa010770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50976/0 (socket says 172.21.15.28:50976) 2024-05-12T22:09:40.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.304+0000 7f6fbe000640 1 -- 172.21.15.28:0/675579725 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562afa015b60 msgr2=0x562afa01ad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:09:40.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.304+0000 7f6fbe000640 1 --2- 172.21.15.28:0/675579725 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562afa015b60 0x562afa01ad40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:40.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.304+0000 7f6fbe000640 1 -- 172.21.15.28:0/675579725 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562afa020b50 msgr2=0x562afa010770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:09:40.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.304+0000 7f6fbe000640 1 --2- 172.21.15.28:0/675579725 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562afa020b50 0x562afa010770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:40.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.304+0000 7f6fbe000640 1 -- 172.21.15.28:0/675579725 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562afa0162b0 con 0x562afa016480 2024-05-12T22:09:40.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.304+0000 7f6fbf400640 1 --2- 172.21.15.28:0/675579725 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562afa020b50 0x562afa010770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:09:40.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.304+0000 7f6fbea00640 1 --2- 172.21.15.28:0/675579725 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562afa015b60 0x562afa01ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:09:40.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.304+0000 7f6fbe000640 1 --2- 172.21.15.28:0/675579725 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562afa016480 0x562afa01b280 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f6fa809bc10 tx=0x7f6fa8009f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cad888c1f0962ea3 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:09:40.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.305+0000 7f6faf400640 1 -- 172.21.15.28:0/675579725 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fa809b7c0 con 0x562afa016480 2024-05-12T22:09:40.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.305+0000 7f6faf400640 1 -- 172.21.15.28:0/675579725 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6fa809c380 con 0x562afa016480 2024-05-12T22:09:40.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.305+0000 7f6faf400640 1 -- 172.21.15.28:0/675579725 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fa809b5e0 con 0x562afa016480 2024-05-12T22:09:40.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.305+0000 7f6fc8bc2740 1 -- 172.21.15.28:0/675579725 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562afa016480 msgr2=0x562afa01b280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:40.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.305+0000 7f6fc8bc2740 1 --2- 172.21.15.28:0/675579725 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562afa016480 0x562afa01b280 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f6fa809bc10 tx=0x7f6fa8009f90 comp rx=0 tx=0).stop 2024-05-12T22:09:40.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.305+0000 7f6fc8bc2740 1 -- 172.21.15.28:0/675579725 shutdown_connections 2024-05-12T22:09:40.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.305+0000 7f6fc8bc2740 1 --2- 172.21.15.28:0/675579725 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562afa020b50 0x562afa010770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:40.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.305+0000 7f6fc8bc2740 1 --2- 172.21.15.28:0/675579725 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562afa016480 0x562afa01b280 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:40.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.305+0000 7f6fc8bc2740 1 --2- 172.21.15.28:0/675579725 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562afa015b60 0x562afa01ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:40.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.305+0000 7f6fc8bc2740 1 -- 172.21.15.28:0/675579725 >> 172.21.15.28:0/675579725 conn(0x562afa011e80 msgr2=0x562afa018dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:09:40.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.305+0000 7f6fc8bc2740 1 -- 172.21.15.28:0/675579725 shutdown_connections 2024-05-12T22:09:40.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.306+0000 7f6fc8bc2740 1 -- 172.21.15.28:0/675579725 wait complete. 2024-05-12T22:09:40.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.306+0000 7f6fc8bc2740 1 -- start start 2024-05-12T22:09:40.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.306+0000 7f6fc8bc2740 1 Processor -- start 2024-05-12T22:09:40.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.307+0000 7f6fc8bc2740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562afa015b60 0x562afa036490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:40.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.307+0000 7f6fc8bc2740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562afa016480 0x562afa0369d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:40.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.307+0000 7f6fc8bc2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562afa020b50 0x562afa033a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:40.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.307+0000 7f6fc8bc2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6fc01adbf0 con 0x562afa020b50 2024-05-12T22:09:40.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.307+0000 7f6fc8bc2740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f6fc01af060 con 0x562afa016480 2024-05-12T22:09:40.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.307+0000 7f6fc8bc2740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f6fc01ae2c0 con 0x562afa015b60 2024-05-12T22:09:40.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.307+0000 7f6fc8bc2740 10 client.0 _ll_register_callbacks cb 0x7f6fb0009e70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:09:40.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.307+0000 7f6fbf400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562afa020b50 0x562afa033a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:40.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.307+0000 7f6fbf400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562afa020b50 0x562afa033a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50978/0 (socket says 172.21.15.28:50978) 2024-05-12T22:09:40.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.307+0000 7f6fbf400640 1 -- 172.21.15.28:0/3138572129 learned_addr learned my addr 172.21.15.28:0/3138572129 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:09:40.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.307+0000 7f6fbea00640 1 --2- 172.21.15.28:0/3138572129 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562afa015b60 0x562afa036490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:40.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.307+0000 7f6fbe000640 1 --2- 172.21.15.28:0/3138572129 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562afa016480 0x562afa0369d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:40.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.307+0000 7f6fbf400640 1 -- 172.21.15.28:0/3138572129 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562afa015b60 msgr2=0x562afa036490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:09:40.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.307+0000 7f6fbf400640 1 --2- 172.21.15.28:0/3138572129 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562afa015b60 0x562afa036490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:40.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.307+0000 7f6fbf400640 1 -- 172.21.15.28:0/3138572129 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562afa016480 msgr2=0x562afa0369d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:09:40.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.307+0000 7f6fbf400640 1 --2- 172.21.15.28:0/3138572129 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562afa016480 0x562afa0369d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:40.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.307+0000 7f6fbf400640 1 -- 172.21.15.28:0/3138572129 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562afa034200 con 0x562afa020b50 2024-05-12T22:09:40.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.308+0000 7f6fbf400640 1 --2- 172.21.15.28:0/3138572129 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562afa020b50 0x562afa033a00 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f6fb8009860 tx=0x7f6fb802a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f069af119332242 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:09:40.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.308+0000 7f6facc00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-12T22:09:40.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.308+0000 7f6facc00640 1 -- 172.21.15.28:0/3138572129 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fb804e070 con 0x562afa020b50 2024-05-12T22:09:40.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.308+0000 7f6facc00640 1 -- 172.21.15.28:0/3138572129 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f6fb8003a00 con 0x562afa020b50 2024-05-12T22:09:40.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.308+0000 7f6fc8bc2740 1 -- 172.21.15.28:0/3138572129 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6fa809c380 con 0x562afa020b50 2024-05-12T22:09:40.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.308+0000 7f6facc00640 1 -- 172.21.15.28:0/3138572129 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f6fb805a050 con 0x562afa020b50 2024-05-12T22:09:40.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.308+0000 7f6fc8bc2740 10 client.15411 fetch_fsmap learned FSMap version 24 2024-05-12T22:09:40.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.308+0000 7f6fc8bc2740 1 -- 172.21.15.28:0/3138572129 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x562afa096e80 con 0x562afa020b50 2024-05-12T22:09:40.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.309+0000 7f6facc00640 1 -- 172.21.15.28:0/3138572129 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6fa809c380 con 0x562afa020b50 2024-05-12T22:09:40.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.309+0000 7f6facc00640 1 -- 172.21.15.28:0/3138572129 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f6fb8032440 con 0x562afa020b50 2024-05-12T22:09:40.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.309+0000 7f6fc8bc2740 10 client.15411 fetch_fsmap finished waiting for FSMap version 24 2024-05-12T22:09:40.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.309+0000 7f6fc8bc2740 10 client.15411 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:09:40.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.309+0000 7f6fc8bc2740 20 client.15411 populate_metadata read hostname 'smithi028' 2024-05-12T22:09:40.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.309+0000 7f6fc8bc2740 1 --2- 172.21.15.28:0/3138572129 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x562afa097600 0x562afa099a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:40.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.309+0000 7f6fc8bc2740 4 client.15411 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-12T22:09:40.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.309+0000 7f6fc8bc2740 1 -- 172.21.15.28:0/3138572129 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562af9f824b0 con 0x562afa097600 2024-05-12T22:09:40.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.309+0000 7f6fbea00640 1 --2- 172.21.15.28:0/3138572129 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x562afa097600 0x562afa099a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:40.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.310+0000 7f6fbea00640 1 --2- 172.21.15.28:0/3138572129 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x562afa097600 0x562afa099a20 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=36c60de3ae5c6b64 server_cookie=b3ccd1cf48949a1a in_seq=0 out_seq=0 2024-05-12T22:09:40.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.310+0000 7f6facc00640 10 client.15411 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:09:40.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.310+0000 7f6facc00640 1 -- 172.21.15.28:0/3138572129 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562af9f824b0 con 0x562afa097600 2024-05-12T22:09:40.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.310+0000 7f6facc00640 10 client.15411 handle_command_reply: tid=0 2024-05-12T22:09:40.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.310+0000 7f6fc8bc2740 1 client.15411 shutdown 2024-05-12T22:09:40.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.310+0000 7f6fc8bc2740 1 -- 172.21.15.28:0/3138572129 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562afa020b50 msgr2=0x562afa033a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:40.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.310+0000 7f6fc8bc2740 1 --2- 172.21.15.28:0/3138572129 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562afa020b50 0x562afa033a00 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f6fb8009860 tx=0x7f6fb802a340 comp rx=0 tx=0).stop 2024-05-12T22:09:40.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.310+0000 7f6fc8bc2740 1 -- 172.21.15.28:0/3138572129 shutdown_connections 2024-05-12T22:09:40.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.310+0000 7f6fc8bc2740 1 --2- 172.21.15.28:0/3138572129 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x562afa097600 0x562afa099a20 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:40.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.310+0000 7f6fc8bc2740 1 --2- 172.21.15.28:0/3138572129 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562afa020b50 0x562afa033a00 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:40.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.310+0000 7f6fc8bc2740 1 --2- 172.21.15.28:0/3138572129 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562afa016480 0x562afa0369d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:40.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.310+0000 7f6fc8bc2740 1 --2- 172.21.15.28:0/3138572129 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562afa015b60 0x562afa036490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:40.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.310+0000 7f6facc00640 0 client.15411 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:09:40.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.310+0000 7f6fc8bc2740 1 -- 172.21.15.28:0/3138572129 >> 172.21.15.28:0/3138572129 conn(0x562afa011e80 msgr2=0x562afa010770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:09:40.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.311+0000 7f6fc8bc2740 1 -- 172.21.15.28:0/3138572129 shutdown_connections 2024-05-12T22:09:40.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.311+0000 7f6fc8bc2740 1 -- 172.21.15.28:0/3138572129 wait complete. 2024-05-12T22:09:40.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.311+0000 7f6fc8bc2740 20 client.15411 trim_cache size 0 max 16384 2024-05-12T22:09:40.320 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-05-12T22:09:40.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.311+0000 7f6fbfe00640 1 -- 172.21.15.28:0/1805217660 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f6fa40a2ea0 msgr2=0x7f6fa40a5360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:40.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.311+0000 7f6fbfe00640 1 --2- 172.21.15.28:0/1805217660 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f6fa40a2ea0 0x7f6fa40a5360 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f6fb802a860 tx=0x7f6fb80056f0 comp rx=0 tx=0).stop 2024-05-12T22:09:40.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.311+0000 7f6fbfe00640 1 -- 172.21.15.28:0/1805217660 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6fc0100e50 msgr2=0x7f6fc019a6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:40.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.311+0000 7f6fbfe00640 1 --2- 172.21.15.28:0/1805217660 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6fc0100e50 0x7f6fc019a6e0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f6fa800e9c0 tx=0x7f6fa800ee80 comp rx=0 tx=0).stop 2024-05-12T22:09:40.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.312+0000 7f6fbfe00640 1 -- 172.21.15.28:0/1805217660 shutdown_connections 2024-05-12T22:09:40.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.312+0000 7f6fbfe00640 1 --2- 172.21.15.28:0/1805217660 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f6fa40a2ea0 0x7f6fa40a5360 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:40.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.312+0000 7f6fbfe00640 1 --2- 172.21.15.28:0/1805217660 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6fc010e050 0x7f6fc019ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:40.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.312+0000 7f6fbfe00640 1 --2- 172.21.15.28:0/1805217660 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6fc0100e50 0x7f6fc019a6e0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:40.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.312+0000 7f6fbfe00640 1 --2- 172.21.15.28:0/1805217660 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6fc01004a0 0x7f6fc019a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:40.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.312+0000 7f6fbfe00640 1 -- 172.21.15.28:0/1805217660 >> 172.21.15.28:0/1805217660 conn(0x7f6fc00fc0b0 msgr2=0x7f6fc010f360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:09:40.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.312+0000 7f6fbfe00640 1 -- 172.21.15.28:0/1805217660 shutdown_connections 2024-05-12T22:09:40.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:40.312+0000 7f6fbfe00640 1 -- 172.21.15.28:0/1805217660 wait complete. 2024-05-12T22:09:40.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:40 smithi186 ceph-mon[34785]: pgmap v23: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:09:40.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:40 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/1761328157' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:09:40.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:40 smithi086 ceph-mon[34297]: pgmap v23: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:09:40.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:40 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/1761328157' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:09:41.332 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-12T22:09:41.333 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-12T22:09:41.333 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-05-12T22:09:41.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.566+0000 7f09cbe00640 1 -- 172.21.15.28:0/758093683 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f09cc073bd0 msgr2=0x7f09cc073fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:41.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.566+0000 7f09cbe00640 1 --2- 172.21.15.28:0/758093683 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f09cc073bd0 0x7f09cc073fb0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f09bc009ca0 tx=0x7f09bc02a950 comp rx=0 tx=0).stop 2024-05-12T22:09:41.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.566+0000 7f09cbe00640 1 -- 172.21.15.28:0/758093683 shutdown_connections 2024-05-12T22:09:41.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.566+0000 7f09cbe00640 1 --2- 172.21.15.28:0/758093683 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09cc1126e0 0x7f09cc114ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:41.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.566+0000 7f09cbe00640 1 --2- 172.21.15.28:0/758093683 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09cc0744f0 0x7f09cc1121a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:41.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.566+0000 7f09cbe00640 1 --2- 172.21.15.28:0/758093683 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f09cc073bd0 0x7f09cc073fb0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:41.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.566+0000 7f09cbe00640 1 -- 172.21.15.28:0/758093683 >> 172.21.15.28:0/758093683 conn(0x7f09cc0fd200 msgr2=0x7f09cc0ff620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:09:41.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.566+0000 7f09cbe00640 1 -- 172.21.15.28:0/758093683 shutdown_connections 2024-05-12T22:09:41.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.566+0000 7f09cbe00640 1 -- 172.21.15.28:0/758093683 wait complete. 2024-05-12T22:09:41.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.567+0000 7f09cbe00640 1 Processor -- start 2024-05-12T22:09:41.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.567+0000 7f09cbe00640 1 -- start start 2024-05-12T22:09:41.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.567+0000 7f09cbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09cc073bd0 0x7f09cc19e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:41.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.567+0000 7f09cbe00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09cc0744f0 0x7f09cc19eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:41.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.567+0000 7f09cbe00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f09cc1126e0 0x7f09cc1a2ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:41.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.567+0000 7f09cbe00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f09cc1180d0 con 0x7f09cc073bd0 2024-05-12T22:09:41.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.567+0000 7f09cbe00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f09cc117f50 con 0x7f09cc1126e0 2024-05-12T22:09:41.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.567+0000 7f09cbe00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f09cc118250 con 0x7f09cc0744f0 2024-05-12T22:09:41.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.568+0000 7f09caa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09cc073bd0 0x7f09cc19e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:41.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.568+0000 7f09caa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09cc073bd0 0x7f09cc19e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50996/0 (socket says 172.21.15.28:50996) 2024-05-12T22:09:41.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.568+0000 7f09caa00640 1 -- 172.21.15.28:0/661418810 learned_addr learned my addr 172.21.15.28:0/661418810 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:09:41.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.568+0000 7f09ca000640 1 --2- 172.21.15.28:0/661418810 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09cc0744f0 0x7f09cc19eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:41.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.568+0000 7f09cb400640 1 --2- 172.21.15.28:0/661418810 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f09cc1126e0 0x7f09cc1a2ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:41.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.568+0000 7f09caa00640 1 -- 172.21.15.28:0/661418810 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09cc0744f0 msgr2=0x7f09cc19eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:41.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.568+0000 7f09caa00640 1 --2- 172.21.15.28:0/661418810 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09cc0744f0 0x7f09cc19eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:41.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.568+0000 7f09caa00640 1 -- 172.21.15.28:0/661418810 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f09cc1126e0 msgr2=0x7f09cc1a2ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:41.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.568+0000 7f09caa00640 1 --2- 172.21.15.28:0/661418810 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f09cc1126e0 0x7f09cc1a2ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:41.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.568+0000 7f09caa00640 1 -- 172.21.15.28:0/661418810 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09cc1a35f0 con 0x7f09cc073bd0 2024-05-12T22:09:41.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.568+0000 7f09cb400640 1 --2- 172.21.15.28:0/661418810 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f09cc1126e0 0x7f09cc1a2ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:09:41.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.569+0000 7f09caa00640 1 --2- 172.21.15.28:0/661418810 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09cc073bd0 0x7f09cc19e650 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f09bc005cc0 tx=0x7f09bc02f420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:09:41.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.569+0000 7f09c3400640 1 -- 172.21.15.28:0/661418810 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09bc034040 con 0x7f09cc073bd0 2024-05-12T22:09:41.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.569+0000 7f09c3400640 1 -- 172.21.15.28:0/661418810 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f09bc030920 con 0x7f09cc073bd0 2024-05-12T22:09:41.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.569+0000 7f09cbe00640 1 -- 172.21.15.28:0/661418810 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09cc1a3880 con 0x7f09cc073bd0 2024-05-12T22:09:41.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.569+0000 7f09c3400640 1 -- 172.21.15.28:0/661418810 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09bc0046e0 con 0x7f09cc073bd0 2024-05-12T22:09:41.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.569+0000 7f09cbe00640 1 -- 172.21.15.28:0/661418810 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09cc1ab170 con 0x7f09cc073bd0 2024-05-12T22:09:41.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.572+0000 7f09d496b740 1 Processor -- start 2024-05-12T22:09:41.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.573+0000 7f09c3400640 1 -- 172.21.15.28:0/661418810 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138060+0+0 (secure 0 0 0) 0x7f09bc037020 con 0x7f09cc073bd0 2024-05-12T22:09:41.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.573+0000 7f09d496b740 1 -- start start 2024-05-12T22:09:41.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.573+0000 7f09d496b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b3c34bf500 0x55b3c34bf8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:41.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.573+0000 7f09d496b740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b3c34bfeb0 0x55b3c34c9f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:41.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.573+0000 7f09d496b740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b3c34ca520 0x55b3c34cc970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:41.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.573+0000 7f09d496b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b3c34237c0 con 0x55b3c34bf500 2024-05-12T22:09:41.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.573+0000 7f09d496b740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x55b3c33cb790 con 0x55b3c34ca520 2024-05-12T22:09:41.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.573+0000 7f09d496b740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55b3c341f780 con 0x55b3c34bfeb0 2024-05-12T22:09:41.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.573+0000 7f09cb400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b3c34bf500 0x55b3c34bf8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:41.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.573+0000 7f09ca000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b3c34ca520 0x55b3c34cc970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:41.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.574+0000 7f09cb400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b3c34bf500 0x55b3c34bf8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51012/0 (socket says 172.21.15.28:51012) 2024-05-12T22:09:41.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.574+0000 7f09cb400640 1 -- 172.21.15.28:0/1844580622 learned_addr learned my addr 172.21.15.28:0/1844580622 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:09:41.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.574+0000 7f09caa00640 1 --2- 172.21.15.28:0/1844580622 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b3c34bfeb0 0x55b3c34c9f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:41.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.574+0000 7f09ca000640 1 -- 172.21.15.28:0/1844580622 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b3c34bfeb0 msgr2=0x55b3c34c9f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:09:41.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.574+0000 7f09ca000640 1 --2- 172.21.15.28:0/1844580622 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b3c34bfeb0 0x55b3c34c9f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:41.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.574+0000 7f09ca000640 1 -- 172.21.15.28:0/1844580622 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b3c34bf500 msgr2=0x55b3c34bf8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:09:41.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.574+0000 7f09ca000640 1 --2- 172.21.15.28:0/1844580622 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b3c34bf500 0x55b3c34bf8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:41.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.574+0000 7f09ca000640 1 -- 172.21.15.28:0/1844580622 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b3c33eb6a0 con 0x55b3c34ca520 2024-05-12T22:09:41.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.574+0000 7f09caa00640 1 --2- 172.21.15.28:0/1844580622 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b3c34bfeb0 0x55b3c34c9f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:09:41.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.574+0000 7f09ca000640 1 --2- 172.21.15.28:0/1844580622 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b3c34ca520 0x55b3c34cc970 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f09b4009bd0 tx=0x7f09b4011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4cf33bee35fb7253 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:09:41.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.574+0000 7f09c3400640 1 --2- 172.21.15.28:0/661418810 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f09b00a2e20 0x7f09b00a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:41.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.575+0000 7f09bbe00640 1 -- 172.21.15.28:0/1844580622 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09b4004230 con 0x55b3c34ca520 2024-05-12T22:09:41.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.575+0000 7f09bbe00640 1 -- 172.21.15.28:0/1844580622 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f09b4009e40 con 0x55b3c34ca520 2024-05-12T22:09:41.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.575+0000 7f09bbe00640 1 -- 172.21.15.28:0/1844580622 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09b4004db0 con 0x55b3c34ca520 2024-05-12T22:09:41.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.575+0000 7f09cb400640 1 --2- 172.21.15.28:0/661418810 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f09b00a2e20 0x7f09b00a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:41.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.575+0000 7f09c3400640 1 -- 172.21.15.28:0/661418810 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f09bc0f04c0 con 0x7f09cc073bd0 2024-05-12T22:09:41.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.575+0000 7f09d496b740 1 -- 172.21.15.28:0/1844580622 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b3c34ca520 msgr2=0x55b3c34cc970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:41.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.575+0000 7f09d496b740 1 --2- 172.21.15.28:0/1844580622 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b3c34ca520 0x55b3c34cc970 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f09b4009bd0 tx=0x7f09b4011810 comp rx=0 tx=0).stop 2024-05-12T22:09:41.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.576+0000 7f09cb400640 1 --2- 172.21.15.28:0/661418810 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f09b00a2e20 0x7f09b00a52e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f09c40058c0 tx=0x7f09c4005f90 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:09:41.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.576+0000 7f09d496b740 1 -- 172.21.15.28:0/1844580622 shutdown_connections 2024-05-12T22:09:41.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.576+0000 7f09d496b740 1 --2- 172.21.15.28:0/1844580622 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b3c34ca520 0x55b3c34cc970 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:41.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.576+0000 7f09d496b740 1 --2- 172.21.15.28:0/1844580622 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b3c34bfeb0 0x55b3c34c9f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:41.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.576+0000 7f09d496b740 1 --2- 172.21.15.28:0/1844580622 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b3c34bf500 0x55b3c34bf8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:41.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.576+0000 7f09d496b740 1 -- 172.21.15.28:0/1844580622 >> 172.21.15.28:0/1844580622 conn(0x55b3c34bb5b0 msgr2=0x55b3c34bb990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:09:41.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.576+0000 7f09d496b740 1 -- 172.21.15.28:0/1844580622 shutdown_connections 2024-05-12T22:09:41.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.576+0000 7f09d496b740 1 -- 172.21.15.28:0/1844580622 wait complete. 2024-05-12T22:09:41.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.576+0000 7f09d496b740 1 -- start start 2024-05-12T22:09:41.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.576+0000 7f09d496b740 1 Processor -- start 2024-05-12T22:09:41.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.577+0000 7f09d496b740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b3c34bf500 0x55b3c34c73e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:41.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.577+0000 7f09d496b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b3c34bfeb0 0x55b3c34c7920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:41.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.577+0000 7f09d496b740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b3c34ca520 0x55b3c34c7e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:41.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.577+0000 7f09d496b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b3c34cd530 con 0x55b3c34bfeb0 2024-05-12T22:09:41.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.577+0000 7f09d496b740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x55b3c33dea90 con 0x55b3c34bf500 2024-05-12T22:09:41.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.577+0000 7f09d496b740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55b3c34cd6b0 con 0x55b3c34ca520 2024-05-12T22:09:41.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.577+0000 7f09d496b740 10 client.0 _ll_register_callbacks cb 0x55b3c3419af0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:09:41.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.577+0000 7f09ca000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b3c34bfeb0 0x55b3c34c7920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:41.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.577+0000 7f09cb400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b3c34ca520 0x55b3c34c7e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:41.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.577+0000 7f09caa00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b3c34bf500 0x55b3c34c73e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:41.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.577+0000 7f09ca000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b3c34bfeb0 0x55b3c34c7920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51026/0 (socket says 172.21.15.28:51026) 2024-05-12T22:09:41.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.577+0000 7f09ca000640 1 -- 172.21.15.28:0/1385065910 learned_addr learned my addr 172.21.15.28:0/1385065910 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:09:41.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.578+0000 7f09caa00640 1 -- 172.21.15.28:0/1385065910 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b3c34ca520 msgr2=0x55b3c34c7e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:09:41.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.578+0000 7f09caa00640 1 --2- 172.21.15.28:0/1385065910 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b3c34ca520 0x55b3c34c7e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:41.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.578+0000 7f09caa00640 1 -- 172.21.15.28:0/1385065910 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b3c34bfeb0 msgr2=0x55b3c34c7920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:09:41.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.578+0000 7f09caa00640 1 --2- 172.21.15.28:0/1385065910 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b3c34bfeb0 0x55b3c34c7920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:41.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.578+0000 7f09caa00640 1 -- 172.21.15.28:0/1385065910 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b3c31ff7e0 con 0x55b3c34bf500 2024-05-12T22:09:41.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.578+0000 7f09caa00640 1 --2- 172.21.15.28:0/1385065910 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b3c34bf500 0x55b3c34c73e0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f09bc0edb70 tx=0x7f09bc0095c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5a46d6ecd876cab5 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:09:41.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.578+0000 7f09b8c00640 10 client.0 ms_handle_connect on v2:172.21.15.86:3300/0 2024-05-12T22:09:41.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.578+0000 7f09b8c00640 1 -- 172.21.15.28:0/1385065910 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09bc0bc3b0 con 0x55b3c34bf500 2024-05-12T22:09:41.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.578+0000 7f09b8c00640 1 -- 172.21.15.28:0/1385065910 <== mon.2 v2:172.21.15.86:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f09bc0c5050 con 0x55b3c34bf500 2024-05-12T22:09:41.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.578+0000 7f09d496b740 1 -- 172.21.15.28:0/1385065910 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55b3c34373a0 con 0x55b3c34bf500 2024-05-12T22:09:41.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.578+0000 7f09b8c00640 1 -- 172.21.15.28:0/1385065910 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f09bc0bc1d0 con 0x55b3c34bf500 2024-05-12T22:09:41.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.578+0000 7f09d496b740 10 client.25541 fetch_fsmap learned FSMap version 24 2024-05-12T22:09:41.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.578+0000 7f09d496b740 1 -- 172.21.15.28:0/1385065910 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55b3c325ff70 con 0x55b3c34bf500 2024-05-12T22:09:41.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.579+0000 7f09b8c00640 1 -- 172.21.15.28:0/1385065910 <== mon.2 v2:172.21.15.86:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55b3c34373a0 con 0x55b3c34bf500 2024-05-12T22:09:41.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.579+0000 7f09b8c00640 1 -- 172.21.15.28:0/1385065910 <== mon.2 v2:172.21.15.86:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f09bc04d030 con 0x55b3c34bf500 2024-05-12T22:09:41.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.579+0000 7f09d496b740 10 client.25541 fetch_fsmap finished waiting for FSMap version 24 2024-05-12T22:09:41.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.579+0000 7f09d496b740 10 client.25541 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:09:41.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.579+0000 7f09d496b740 20 client.25541 populate_metadata read hostname 'smithi028' 2024-05-12T22:09:41.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.579+0000 7f09d496b740 1 --2- 172.21.15.28:0/1385065910 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55b3c34da570 0x55b3c34dc9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:41.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.579+0000 7f09d496b740 4 client.25541 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-12T22:09:41.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.579+0000 7f09d496b740 1 -- 172.21.15.28:0/1385065910 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b3c3430ef0 con 0x55b3c34da570 2024-05-12T22:09:41.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.579+0000 7f09ca000640 1 --2- 172.21.15.28:0/1385065910 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55b3c34da570 0x55b3c34dc9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:41.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.580+0000 7f09ca000640 1 --2- 172.21.15.28:0/1385065910 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55b3c34da570 0x55b3c34dc9e0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=778a87867166c33e server_cookie=9ba41542745197ea in_seq=0 out_seq=0 2024-05-12T22:09:41.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.580+0000 7f09b8c00640 10 client.25541 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:09:41.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.581+0000 7f09b8c00640 1 -- 172.21.15.28:0/1385065910 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b3c3430ef0 con 0x55b3c34da570 2024-05-12T22:09:41.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.581+0000 7f09b8c00640 10 client.25541 handle_command_reply: tid=0 2024-05-12T22:09:41.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.581+0000 7f09d496b740 1 client.25541 shutdown 2024-05-12T22:09:41.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.581+0000 7f09d496b740 1 -- 172.21.15.28:0/1385065910 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b3c34bf500 msgr2=0x55b3c34c73e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:41.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.581+0000 7f09d496b740 1 --2- 172.21.15.28:0/1385065910 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b3c34bf500 0x55b3c34c73e0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f09bc0edb70 tx=0x7f09bc0095c0 comp rx=0 tx=0).stop 2024-05-12T22:09:41.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.581+0000 7f09d496b740 1 -- 172.21.15.28:0/1385065910 shutdown_connections 2024-05-12T22:09:41.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.581+0000 7f09d496b740 1 --2- 172.21.15.28:0/1385065910 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55b3c34da570 0x55b3c34dc9e0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:41.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.581+0000 7f09d496b740 1 --2- 172.21.15.28:0/1385065910 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b3c34ca520 0x55b3c34c7e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:41.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.581+0000 7f09d496b740 1 --2- 172.21.15.28:0/1385065910 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b3c34bfeb0 0x55b3c34c7920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:41.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.581+0000 7f09d496b740 1 --2- 172.21.15.28:0/1385065910 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b3c34bf500 0x55b3c34c73e0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:41.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.581+0000 7f09d496b740 1 -- 172.21.15.28:0/1385065910 >> 172.21.15.28:0/1385065910 conn(0x55b3c34bb5b0 msgr2=0x55b3c34c1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:09:41.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.581+0000 7f09b8c00640 0 client.25541 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:09:41.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.581+0000 7f09d496b740 1 -- 172.21.15.28:0/1385065910 shutdown_connections 2024-05-12T22:09:41.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.581+0000 7f09d496b740 1 -- 172.21.15.28:0/1385065910 wait complete. 2024-05-12T22:09:41.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.581+0000 7f09d496b740 20 client.25541 trim_cache size 0 max 16384 2024-05-12T22:09:41.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.601+0000 7f09d496b740 1 Processor -- start 2024-05-12T22:09:41.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.601+0000 7f09d496b740 1 -- start start 2024-05-12T22:09:41.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.601+0000 7f09d496b740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b3c34bf500 0x55b3c34c1950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:41.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.601+0000 7f09d496b740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b3c34bfeb0 0x55b3c34d63f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:41.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.601+0000 7f09d496b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b3c34ca520 0x55b3c34ba5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:41.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.601+0000 7f09d496b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f09cc1b6020 con 0x55b3c34ca520 2024-05-12T22:09:41.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.601+0000 7f09d496b740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f09cc1b38f0 con 0x55b3c34bfeb0 2024-05-12T22:09:41.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.601+0000 7f09d496b740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f09cc1b1740 con 0x55b3c34bf500 2024-05-12T22:09:41.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.602+0000 7f09cb400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b3c34ca520 0x55b3c34ba5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:41.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.602+0000 7f09cb400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b3c34ca520 0x55b3c34ba5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51038/0 (socket says 172.21.15.28:51038) 2024-05-12T22:09:41.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.602+0000 7f09cb400640 1 -- 172.21.15.28:0/1975880232 learned_addr learned my addr 172.21.15.28:0/1975880232 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:09:41.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.602+0000 7f09caa00640 1 --2- 172.21.15.28:0/1975880232 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b3c34bf500 0x55b3c34c1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:41.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.602+0000 7f09ca000640 1 --2- 172.21.15.28:0/1975880232 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b3c34bfeb0 0x55b3c34d63f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:41.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.602+0000 7f09cb400640 1 -- 172.21.15.28:0/1975880232 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b3c34bf500 msgr2=0x55b3c34c1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:09:41.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.602+0000 7f09cb400640 1 --2- 172.21.15.28:0/1975880232 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b3c34bf500 0x55b3c34c1950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:41.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.602+0000 7f09cb400640 1 -- 172.21.15.28:0/1975880232 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b3c34bfeb0 msgr2=0x55b3c34d63f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:09:41.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.602+0000 7f09cb400640 1 --2- 172.21.15.28:0/1975880232 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b3c34bfeb0 0x55b3c34d63f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:41.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.602+0000 7f09cb400640 1 -- 172.21.15.28:0/1975880232 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b3c30f7170 con 0x55b3c34ca520 2024-05-12T22:09:41.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.602+0000 7f09caa00640 1 --2- 172.21.15.28:0/1975880232 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b3c34bf500 0x55b3c34c1950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:09:41.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.603+0000 7f09cb400640 1 --2- 172.21.15.28:0/1975880232 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b3c34ca520 0x55b3c34ba5b0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f09c401f770 tx=0x7f09c401a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ecd931479b25f8 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:09:41.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.603+0000 7f09bb400640 1 -- 172.21.15.28:0/1975880232 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09c4005020 con 0x55b3c34ca520 2024-05-12T22:09:41.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.603+0000 7f09bb400640 1 -- 172.21.15.28:0/1975880232 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f09c40047a0 con 0x55b3c34ca520 2024-05-12T22:09:41.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.603+0000 7f09bb400640 1 -- 172.21.15.28:0/1975880232 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09c40119e0 con 0x55b3c34ca520 2024-05-12T22:09:41.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.604+0000 7f09d496b740 1 -- 172.21.15.28:0/1975880232 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b3c34ca520 msgr2=0x55b3c34ba5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:41.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.604+0000 7f09d496b740 1 --2- 172.21.15.28:0/1975880232 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b3c34ca520 0x55b3c34ba5b0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f09c401f770 tx=0x7f09c401a000 comp rx=0 tx=0).stop 2024-05-12T22:09:41.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.604+0000 7f09d496b740 1 -- 172.21.15.28:0/1975880232 shutdown_connections 2024-05-12T22:09:41.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.604+0000 7f09d496b740 1 --2- 172.21.15.28:0/1975880232 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b3c34ca520 0x55b3c34ba5b0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:41.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.604+0000 7f09d496b740 1 --2- 172.21.15.28:0/1975880232 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b3c34bfeb0 0x55b3c34d63f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:41.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.604+0000 7f09d496b740 1 --2- 172.21.15.28:0/1975880232 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b3c34bf500 0x55b3c34c1950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:41.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.604+0000 7f09d496b740 1 -- 172.21.15.28:0/1975880232 >> 172.21.15.28:0/1975880232 conn(0x55b3c34bb5b0 msgr2=0x55b3c34b9ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:09:41.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.604+0000 7f09d496b740 1 -- 172.21.15.28:0/1975880232 shutdown_connections 2024-05-12T22:09:41.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.604+0000 7f09d496b740 1 -- 172.21.15.28:0/1975880232 wait complete. 2024-05-12T22:09:41.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.604+0000 7f09d496b740 1 -- start start 2024-05-12T22:09:41.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.605+0000 7f09d496b740 1 Processor -- start 2024-05-12T22:09:41.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.605+0000 7f09d496b740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b3c34bf500 0x55b3c34c4190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:41.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.605+0000 7f09d496b740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b3c34bfeb0 0x55b3c34c46d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:41.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.605+0000 7f09d496b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b3c34ca520 0x55b3c34c4c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:41.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.605+0000 7f09d496b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f09cc1b1db0 con 0x55b3c34ca520 2024-05-12T22:09:41.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.605+0000 7f09d496b740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f09cc1b3220 con 0x55b3c34bfeb0 2024-05-12T22:09:41.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.605+0000 7f09d496b740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f09cc1b2480 con 0x55b3c34bf500 2024-05-12T22:09:41.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.605+0000 7f09d496b740 10 client.0 _ll_register_callbacks cb 0x7f09bc005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:09:41.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.605+0000 7f09caa00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b3c34bf500 0x55b3c34c4190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:41.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.605+0000 7f09cb400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b3c34ca520 0x55b3c34c4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:41.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.605+0000 7f09caa00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b3c34bf500 0x55b3c34c4190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:34874/0 (socket says 172.21.15.28:34874) 2024-05-12T22:09:41.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.605+0000 7f09caa00640 1 -- 172.21.15.28:0/2163586473 learned_addr learned my addr 172.21.15.28:0/2163586473 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:09:41.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.605+0000 7f09ca000640 1 --2- 172.21.15.28:0/2163586473 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b3c34bfeb0 0x55b3c34c46d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:41.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.606+0000 7f09caa00640 1 -- 172.21.15.28:0/2163586473 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b3c34bfeb0 msgr2=0x55b3c34c46d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:09:41.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.606+0000 7f09caa00640 1 --2- 172.21.15.28:0/2163586473 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b3c34bfeb0 0x55b3c34c46d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:41.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.606+0000 7f09caa00640 1 -- 172.21.15.28:0/2163586473 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b3c34ca520 msgr2=0x55b3c34c4c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:09:41.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.606+0000 7f09caa00640 1 --2- 172.21.15.28:0/2163586473 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b3c34ca520 0x55b3c34c4c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:41.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.606+0000 7f09caa00640 1 -- 172.21.15.28:0/2163586473 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b3c353fb70 con 0x55b3c34bf500 2024-05-12T22:09:41.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.606+0000 7f09caa00640 1 --2- 172.21.15.28:0/2163586473 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b3c34bf500 0x55b3c34c4190 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f09bc0bab20 tx=0x7f09bc0c5260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ae303dc2098edff1 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:09:41.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.606+0000 7f09b8c00640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-12T22:09:41.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.606+0000 7f09b8c00640 1 -- 172.21.15.28:0/2163586473 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09bc0bc3b0 con 0x55b3c34bf500 2024-05-12T22:09:41.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.606+0000 7f09b8c00640 1 -- 172.21.15.28:0/2163586473 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f09bc030440 con 0x55b3c34bf500 2024-05-12T22:09:41.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.606+0000 7f09d496b740 1 -- 172.21.15.28:0/2163586473 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f09c40047a0 con 0x55b3c34bf500 2024-05-12T22:09:41.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.606+0000 7f09b8c00640 1 -- 172.21.15.28:0/2163586473 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f09bc0c0c20 con 0x55b3c34bf500 2024-05-12T22:09:41.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.607+0000 7f09d496b740 10 client.25519 fetch_fsmap learned FSMap version 24 2024-05-12T22:09:41.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.607+0000 7f09d496b740 1 -- 172.21.15.28:0/2163586473 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55b3c3540e90 con 0x55b3c34bf500 2024-05-12T22:09:41.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.607+0000 7f09b8c00640 1 -- 172.21.15.28:0/2163586473 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f09c40047a0 con 0x55b3c34bf500 2024-05-12T22:09:41.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.607+0000 7f09b8c00640 1 -- 172.21.15.28:0/2163586473 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f09bc04d040 con 0x55b3c34bf500 2024-05-12T22:09:41.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.607+0000 7f09d496b740 10 client.25519 fetch_fsmap finished waiting for FSMap version 24 2024-05-12T22:09:41.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.607+0000 7f09d496b740 10 client.25519 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:09:41.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.607+0000 7f09d496b740 20 client.25519 populate_metadata read hostname 'smithi028' 2024-05-12T22:09:41.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.607+0000 7f09d496b740 1 --2- 172.21.15.28:0/2163586473 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55b3c3541480 0x55b3c35438a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:41.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.607+0000 7f09d496b740 4 client.25519 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-12T22:09:41.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.607+0000 7f09d496b740 1 -- 172.21.15.28:0/2163586473 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55b3c342b240 con 0x55b3c3541480 2024-05-12T22:09:41.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.607+0000 7f09ca000640 1 --2- 172.21.15.28:0/2163586473 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55b3c3541480 0x55b3c35438a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:41.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.608+0000 7f09ca000640 1 --2- 172.21.15.28:0/2163586473 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55b3c3541480 0x55b3c35438a0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7662d008c9067f48 server_cookie=d70577077380abb7 in_seq=0 out_seq=0 2024-05-12T22:09:41.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.608+0000 7f09b8c00640 10 client.25519 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:09:41.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.609+0000 7f09b8c00640 1 -- 172.21.15.28:0/2163586473 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55b3c342b240 con 0x55b3c3541480 2024-05-12T22:09:41.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.609+0000 7f09b8c00640 10 client.25519 handle_command_reply: tid=0 2024-05-12T22:09:41.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.609+0000 7f09d496b740 1 client.25519 shutdown 2024-05-12T22:09:41.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.609+0000 7f09d496b740 1 -- 172.21.15.28:0/2163586473 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b3c34bf500 msgr2=0x55b3c34c4190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:41.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.609+0000 7f09d496b740 1 --2- 172.21.15.28:0/2163586473 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b3c34bf500 0x55b3c34c4190 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f09bc0bab20 tx=0x7f09bc0c5260 comp rx=0 tx=0).stop 2024-05-12T22:09:41.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.609+0000 7f09d496b740 1 -- 172.21.15.28:0/2163586473 shutdown_connections 2024-05-12T22:09:41.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.609+0000 7f09d496b740 1 --2- 172.21.15.28:0/2163586473 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55b3c3541480 0x55b3c35438a0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:41.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.609+0000 7f09d496b740 1 --2- 172.21.15.28:0/2163586473 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b3c34ca520 0x55b3c34c4c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:41.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.609+0000 7f09d496b740 1 --2- 172.21.15.28:0/2163586473 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b3c34bfeb0 0x55b3c34c46d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:41.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.609+0000 7f09d496b740 1 --2- 172.21.15.28:0/2163586473 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b3c34bf500 0x55b3c34c4190 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:41.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.609+0000 7f09b8c00640 0 client.25519 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:09:41.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.609+0000 7f09d496b740 1 -- 172.21.15.28:0/2163586473 >> 172.21.15.28:0/2163586473 conn(0x55b3c34bb5b0 msgr2=0x55b3c34ba730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:09:41.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.609+0000 7f09d496b740 1 -- 172.21.15.28:0/2163586473 shutdown_connections 2024-05-12T22:09:41.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.609+0000 7f09d496b740 1 -- 172.21.15.28:0/2163586473 wait complete. 2024-05-12T22:09:41.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.610+0000 7f09d496b740 20 client.25519 trim_cache size 0 max 16384 2024-05-12T22:09:41.615 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-05-12T22:09:41.615 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-05-12T22:09:41.615 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "5bb0ab83-29d9-4339-85ed-4ccb15c90b95", 2024-05-12T22:09:41.615 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-05-12T22:09:41.615 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-05-12T22:09:41.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.610+0000 7f09cbe00640 1 -- 172.21.15.28:0/661418810 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f09b00a2e20 msgr2=0x7f09b00a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:41.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.610+0000 7f09cbe00640 1 --2- 172.21.15.28:0/661418810 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f09b00a2e20 0x7f09b00a52e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f09c40058c0 tx=0x7f09c4005f90 comp rx=0 tx=0).stop 2024-05-12T22:09:41.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.610+0000 7f09cbe00640 1 -- 172.21.15.28:0/661418810 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09cc073bd0 msgr2=0x7f09cc19e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:41.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.610+0000 7f09cbe00640 1 --2- 172.21.15.28:0/661418810 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09cc073bd0 0x7f09cc19e650 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f09bc005cc0 tx=0x7f09bc02f420 comp rx=0 tx=0).stop 2024-05-12T22:09:41.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.610+0000 7f09cbe00640 1 -- 172.21.15.28:0/661418810 shutdown_connections 2024-05-12T22:09:41.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.611+0000 7f09cbe00640 1 --2- 172.21.15.28:0/661418810 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f09b00a2e20 0x7f09b00a52e0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:41.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.611+0000 7f09cbe00640 1 --2- 172.21.15.28:0/661418810 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f09cc1126e0 0x7f09cc1a2ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:41.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.611+0000 7f09cbe00640 1 --2- 172.21.15.28:0/661418810 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09cc0744f0 0x7f09cc19eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:41.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.611+0000 7f09cbe00640 1 --2- 172.21.15.28:0/661418810 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09cc073bd0 0x7f09cc19e650 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:41.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.611+0000 7f09cbe00640 1 -- 172.21.15.28:0/661418810 >> 172.21.15.28:0/661418810 conn(0x7f09cc0fd200 msgr2=0x7f09cc0feb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:09:41.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.611+0000 7f09cbe00640 1 -- 172.21.15.28:0/661418810 shutdown_connections 2024-05-12T22:09:41.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.611+0000 7f09cbe00640 1 -- 172.21.15.28:0/661418810 wait complete. 2024-05-12T22:09:41.630 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-12T22:09:41.630 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-12T22:09:41.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.844+0000 7fe80d600640 1 -- 172.21.15.28:0/1366418332 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe8081016e0 msgr2=0x7fe808101b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:41.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.844+0000 7fe80d600640 1 --2- 172.21.15.28:0/1366418332 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe8081016e0 0x7fe808101b60 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fe7f0009900 tx=0x7fe7f002a400 comp rx=0 tx=0).stop 2024-05-12T22:09:41.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.844+0000 7fe80d600640 1 -- 172.21.15.28:0/1366418332 shutdown_connections 2024-05-12T22:09:41.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.844+0000 7fe80d600640 1 --2- 172.21.15.28:0/1366418332 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8081020a0 0x7fe80810e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:41.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.844+0000 7fe80d600640 1 --2- 172.21.15.28:0/1366418332 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe8081016e0 0x7fe808101b60 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:41.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.844+0000 7fe80d600640 1 --2- 172.21.15.28:0/1366418332 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe8081076f0 0x7fe808107ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:41.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.844+0000 7fe80d600640 1 -- 172.21.15.28:0/1366418332 >> 172.21.15.28:0/1366418332 conn(0x7fe8080fd1d0 msgr2=0x7fe8080ff5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:09:41.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.845+0000 7fe80d600640 1 -- 172.21.15.28:0/1366418332 shutdown_connections 2024-05-12T22:09:41.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.845+0000 7fe80d600640 1 -- 172.21.15.28:0/1366418332 wait complete. 2024-05-12T22:09:41.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.846+0000 7fe80d600640 1 Processor -- start 2024-05-12T22:09:41.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.846+0000 7fe80d600640 1 -- start start 2024-05-12T22:09:41.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.846+0000 7fe80d600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8081016e0 0x7fe808197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:41.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.846+0000 7fe80d600640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe8081020a0 0x7fe808198510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:41.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.846+0000 7fe80d600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe8081076f0 0x7fe80819c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:41.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.846+0000 7fe80d600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe808111b50 con 0x7fe8081016e0 2024-05-12T22:09:41.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.846+0000 7fe80d600640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fe8081119d0 con 0x7fe8081076f0 2024-05-12T22:09:41.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.846+0000 7fe80d600640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe808111cd0 con 0x7fe8081020a0 2024-05-12T22:09:41.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.846+0000 7fe807e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8081016e0 0x7fe808197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:41.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.847+0000 7fe7ff400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe8081020a0 0x7fe808198510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:41.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.847+0000 7fe7ff400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe8081020a0 0x7fe808198510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:34892/0 (socket says 172.21.15.28:34892) 2024-05-12T22:09:41.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.847+0000 7fe807e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8081016e0 0x7fe808197fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51056/0 (socket says 172.21.15.28:51056) 2024-05-12T22:09:41.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.847+0000 7fe7ff400640 1 -- 172.21.15.28:0/2771719685 learned_addr learned my addr 172.21.15.28:0/2771719685 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:09:41.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.847+0000 7fe80cc00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe8081076f0 0x7fe80819c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:41.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.847+0000 7fe7ff400640 1 -- 172.21.15.28:0/2771719685 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe8081076f0 msgr2=0x7fe80819c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:41.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.847+0000 7fe7ff400640 1 --2- 172.21.15.28:0/2771719685 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe8081076f0 0x7fe80819c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:41.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.847+0000 7fe7ff400640 1 -- 172.21.15.28:0/2771719685 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8081016e0 msgr2=0x7fe808197fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:41.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.847+0000 7fe7ff400640 1 --2- 172.21.15.28:0/2771719685 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8081016e0 0x7fe808197fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:41.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.847+0000 7fe7ff400640 1 -- 172.21.15.28:0/2771719685 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe80819cfc0 con 0x7fe8081020a0 2024-05-12T22:09:41.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.847+0000 7fe80cc00640 1 --2- 172.21.15.28:0/2771719685 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe8081076f0 0x7fe80819c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:09:41.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.847+0000 7fe7ff400640 1 --2- 172.21.15.28:0/2771719685 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe8081020a0 0x7fe808198510 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fe7f002a9f0 tx=0x7fe7f0003c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:09:41.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.848+0000 7fe805600640 1 -- 172.21.15.28:0/2771719685 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7f003f070 con 0x7fe8081020a0 2024-05-12T22:09:41.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.848+0000 7fe80d600640 1 -- 172.21.15.28:0/2771719685 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe80819d250 con 0x7fe8081020a0 2024-05-12T22:09:41.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.848+0000 7fe805600640 1 -- 172.21.15.28:0/2771719685 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe7f0031c30 con 0x7fe8081020a0 2024-05-12T22:09:41.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.848+0000 7fe805600640 1 -- 172.21.15.28:0/2771719685 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7f0035030 con 0x7fe8081020a0 2024-05-12T22:09:41.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.848+0000 7fe80d600640 1 -- 172.21.15.28:0/2771719685 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe808073a60 con 0x7fe8081020a0 2024-05-12T22:09:41.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.852+0000 7fe81196a740 1 Processor -- start 2024-05-12T22:09:41.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.852+0000 7fe81196a740 1 -- start start 2024-05-12T22:09:41.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.853+0000 7fe81196a740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5629b266add0 0x5629b266b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:41.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.853+0000 7fe81196a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5629b266b6f0 0x5629b2675880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:41.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.853+0000 7fe81196a740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5629b2675dc0 0x5629b26781b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:41.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.853+0000 7fe81196a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5629b26332a0 con 0x5629b266b6f0 2024-05-12T22:09:41.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.853+0000 7fe81196a740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x5629b25cfb90 con 0x5629b266add0 2024-05-12T22:09:41.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.853+0000 7fe81196a740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5629b25775b0 con 0x5629b2675dc0 2024-05-12T22:09:41.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.853+0000 7fe805600640 1 -- 172.21.15.28:0/2771719685 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 25) ==== 138060+0+0 (secure 0 0 0) 0x7fe7f0042070 con 0x7fe8081020a0 2024-05-12T22:09:41.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.853+0000 7fe807e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5629b266b6f0 0x5629b2675880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:41.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.853+0000 7fe807e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5629b266b6f0 0x5629b2675880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51062/0 (socket says 172.21.15.28:51062) 2024-05-12T22:09:41.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.853+0000 7fe807e00640 1 -- 172.21.15.28:0/15376378 learned_addr learned my addr 172.21.15.28:0/15376378 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:09:41.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.853+0000 7fe80cc00640 1 --2- 172.21.15.28:0/15376378 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5629b266add0 0x5629b266b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:41.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.854+0000 7fe807e00640 1 -- 172.21.15.28:0/15376378 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5629b2675dc0 msgr2=0x5629b26781b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:09:41.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.854+0000 7fe807e00640 1 --2- 172.21.15.28:0/15376378 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5629b2675dc0 0x5629b26781b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:41.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.854+0000 7fe807e00640 1 -- 172.21.15.28:0/15376378 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5629b266add0 msgr2=0x5629b266b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:09:41.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.854+0000 7fe807e00640 1 --2- 172.21.15.28:0/15376378 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5629b266add0 0x5629b266b1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:41.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.854+0000 7fe807e00640 1 -- 172.21.15.28:0/15376378 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5629b23ad7d0 con 0x5629b266b6f0 2024-05-12T22:09:41.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.854+0000 7fe807e00640 1 --2- 172.21.15.28:0/15376378 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5629b266b6f0 0x5629b2675880 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fe7f8009ca0 tx=0x7fe7f8011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9bf45eef21745fd server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:09:41.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.854+0000 7fe7fd200640 1 -- 172.21.15.28:0/15376378 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7f801e070 con 0x5629b266b6f0 2024-05-12T22:09:41.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.854+0000 7fe7fd200640 1 -- 172.21.15.28:0/15376378 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe7f8019b10 con 0x5629b266b6f0 2024-05-12T22:09:41.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.854+0000 7fe7fd200640 1 -- 172.21.15.28:0/15376378 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7f800ea80 con 0x5629b266b6f0 2024-05-12T22:09:41.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.855+0000 7fe805600640 1 --2- 172.21.15.28:0/2771719685 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fe7ec0a2e20 0x7fe7ec0a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:41.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.855+0000 7fe805600640 1 -- 172.21.15.28:0/2771719685 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7fe7f00ef860 con 0x7fe8081020a0 2024-05-12T22:09:41.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.855+0000 7fe80cc00640 1 --2- 172.21.15.28:0/2771719685 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fe7ec0a2e20 0x7fe7ec0a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:41.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.855+0000 7fe81196a740 1 -- 172.21.15.28:0/15376378 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5629b266b6f0 msgr2=0x5629b2675880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:41.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.855+0000 7fe81196a740 1 --2- 172.21.15.28:0/15376378 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5629b266b6f0 0x5629b2675880 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fe7f8009ca0 tx=0x7fe7f8011810 comp rx=0 tx=0).stop 2024-05-12T22:09:41.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.855+0000 7fe81196a740 1 -- 172.21.15.28:0/15376378 shutdown_connections 2024-05-12T22:09:41.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.855+0000 7fe81196a740 1 --2- 172.21.15.28:0/15376378 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5629b2675dc0 0x5629b26781b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:41.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.855+0000 7fe81196a740 1 --2- 172.21.15.28:0/15376378 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5629b266b6f0 0x5629b2675880 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:41.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.855+0000 7fe81196a740 1 --2- 172.21.15.28:0/15376378 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5629b266add0 0x5629b266b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:41.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.855+0000 7fe81196a740 1 -- 172.21.15.28:0/15376378 >> 172.21.15.28:0/15376378 conn(0x5629b26670f0 msgr2=0x5629b26674d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:09:41.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.855+0000 7fe81196a740 1 -- 172.21.15.28:0/15376378 shutdown_connections 2024-05-12T22:09:41.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.856+0000 7fe81196a740 1 -- 172.21.15.28:0/15376378 wait complete. 2024-05-12T22:09:41.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.856+0000 7fe80cc00640 1 --2- 172.21.15.28:0/2771719685 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fe7ec0a2e20 0x7fe7ec0a52e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fe800009fd0 tx=0x7fe800012040 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:09:41.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.856+0000 7fe81196a740 1 -- start start 2024-05-12T22:09:41.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.856+0000 7fe81196a740 1 Processor -- start 2024-05-12T22:09:41.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.857+0000 7fe81196a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5629b266add0 0x5629b26757a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:41.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.857+0000 7fe81196a740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5629b266b6f0 0x5629b2672a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:41.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.857+0000 7fe81196a740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5629b2675dc0 0x5629b26730a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:41.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.857+0000 7fe81196a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5629b258a9f0 con 0x5629b266add0 2024-05-12T22:09:41.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.857+0000 7fe81196a740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x5629b25cbb50 con 0x5629b266b6f0 2024-05-12T22:09:41.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.857+0000 7fe81196a740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5629b2678bb0 con 0x5629b2675dc0 2024-05-12T22:09:41.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.857+0000 7fe81196a740 10 client.0 _ll_register_callbacks cb 0x5629b22e2be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:09:41.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.857+0000 7fe80cc00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5629b2675dc0 0x5629b26730a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:41.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.857+0000 7fe80cc00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5629b2675dc0 0x5629b26730a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:34910/0 (socket says 172.21.15.28:34910) 2024-05-12T22:09:41.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.857+0000 7fe80cc00640 1 -- 172.21.15.28:0/2641433733 learned_addr learned my addr 172.21.15.28:0/2641433733 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:09:41.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.857+0000 7fe807e00640 1 --2- 172.21.15.28:0/2641433733 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5629b266add0 0x5629b26757a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:41.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.857+0000 7fe7ff400640 1 --2- 172.21.15.28:0/2641433733 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5629b266b6f0 0x5629b2672a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:41.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.858+0000 7fe80cc00640 1 -- 172.21.15.28:0/2641433733 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5629b266b6f0 msgr2=0x5629b2672a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:09:41.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.858+0000 7fe80cc00640 1 --2- 172.21.15.28:0/2641433733 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5629b266b6f0 0x5629b2672a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:41.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.858+0000 7fe80cc00640 1 -- 172.21.15.28:0/2641433733 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5629b266add0 msgr2=0x5629b26757a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:09:41.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.858+0000 7fe80cc00640 1 --2- 172.21.15.28:0/2641433733 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5629b266add0 0x5629b26757a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:41.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.858+0000 7fe80cc00640 1 -- 172.21.15.28:0/2641433733 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5629b22e33f0 con 0x5629b2675dc0 2024-05-12T22:09:41.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.858+0000 7fe7ff400640 1 --2- 172.21.15.28:0/2641433733 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5629b266b6f0 0x5629b2672a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:09:41.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.858+0000 7fe807e00640 1 --2- 172.21.15.28:0/2641433733 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5629b266add0 0x5629b26757a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:09:41.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.858+0000 7fe80cc00640 1 --2- 172.21.15.28:0/2641433733 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5629b2675dc0 0x5629b26730a0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fe800005c00 tx=0x7fe800004ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=26820e602ef3f28c server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:09:41.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.858+0000 7fe7f5600640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-12T22:09:41.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.858+0000 7fe7f5600640 1 -- 172.21.15.28:0/2641433733 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe80002d060 con 0x5629b2675dc0 2024-05-12T22:09:41.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.858+0000 7fe7f5600640 1 -- 172.21.15.28:0/2641433733 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7fe80001d450 con 0x5629b2675dc0 2024-05-12T22:09:41.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.858+0000 7fe81196a740 1 -- 172.21.15.28:0/2641433733 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe7f8019b10 con 0x5629b2675dc0 2024-05-12T22:09:41.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.859+0000 7fe7f5600640 1 -- 172.21.15.28:0/2641433733 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fe80001d650 con 0x5629b2675dc0 2024-05-12T22:09:41.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.859+0000 7fe81196a740 10 client.25528 fetch_fsmap learned FSMap version 24 2024-05-12T22:09:41.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.859+0000 7fe81196a740 1 -- 172.21.15.28:0/2641433733 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5629b22a30f0 con 0x5629b2675dc0 2024-05-12T22:09:41.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.859+0000 7fe7f5600640 1 -- 172.21.15.28:0/2641433733 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe7f8019b10 con 0x5629b2675dc0 2024-05-12T22:09:41.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.859+0000 7fe7f5600640 1 -- 172.21.15.28:0/2641433733 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7fe800028a30 con 0x5629b2675dc0 2024-05-12T22:09:41.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.859+0000 7fe81196a740 10 client.25528 fetch_fsmap finished waiting for FSMap version 24 2024-05-12T22:09:41.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.859+0000 7fe81196a740 10 client.25528 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:09:41.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.859+0000 7fe81196a740 20 client.25528 populate_metadata read hostname 'smithi028' 2024-05-12T22:09:41.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.860+0000 7fe81196a740 1 --2- 172.21.15.28:0/2641433733 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5629b2685a30 0x5629b2687e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:41.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.860+0000 7fe81196a740 4 client.25528 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-12T22:09:41.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.860+0000 7fe81196a740 1 -- 172.21.15.28:0/2641433733 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5629b25dd2c0 con 0x5629b2685a30 2024-05-12T22:09:41.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.860+0000 7fe807e00640 1 --2- 172.21.15.28:0/2641433733 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5629b2685a30 0x5629b2687e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:41.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.861+0000 7fe807e00640 1 --2- 172.21.15.28:0/2641433733 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5629b2685a30 0x5629b2687e20 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=13ea37cf9c43a68f server_cookie=5c06750c67c4a01b in_seq=0 out_seq=0 2024-05-12T22:09:41.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.861+0000 7fe7f5600640 10 client.25528 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:09:41.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.861+0000 7fe7f5600640 1 -- 172.21.15.28:0/2641433733 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5629b25dd2c0 con 0x5629b2685a30 2024-05-12T22:09:41.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.861+0000 7fe7f5600640 10 client.25528 handle_command_reply: tid=0 2024-05-12T22:09:41.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.861+0000 7fe81196a740 1 client.25528 shutdown 2024-05-12T22:09:41.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.862+0000 7fe81196a740 1 -- 172.21.15.28:0/2641433733 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5629b2675dc0 msgr2=0x5629b26730a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:41.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.862+0000 7fe81196a740 1 --2- 172.21.15.28:0/2641433733 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5629b2675dc0 0x5629b26730a0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fe800005c00 tx=0x7fe800004ab0 comp rx=0 tx=0).stop 2024-05-12T22:09:41.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.862+0000 7fe81196a740 1 -- 172.21.15.28:0/2641433733 shutdown_connections 2024-05-12T22:09:41.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.862+0000 7fe81196a740 1 --2- 172.21.15.28:0/2641433733 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5629b2685a30 0x5629b2687e20 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:41.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.862+0000 7fe81196a740 1 --2- 172.21.15.28:0/2641433733 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5629b2675dc0 0x5629b26730a0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:41.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.862+0000 7fe81196a740 1 --2- 172.21.15.28:0/2641433733 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5629b266b6f0 0x5629b2672a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:41.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.862+0000 7fe7f5600640 0 client.25528 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:09:41.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.862+0000 7fe81196a740 1 --2- 172.21.15.28:0/2641433733 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5629b266add0 0x5629b26757a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:41.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.862+0000 7fe81196a740 1 -- 172.21.15.28:0/2641433733 >> 172.21.15.28:0/2641433733 conn(0x5629b26670f0 msgr2=0x5629b2666330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:09:41.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.862+0000 7fe81196a740 1 -- 172.21.15.28:0/2641433733 shutdown_connections 2024-05-12T22:09:41.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.862+0000 7fe81196a740 1 -- 172.21.15.28:0/2641433733 wait complete. 2024-05-12T22:09:41.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.862+0000 7fe81196a740 20 client.25528 trim_cache size 0 max 16384 2024-05-12T22:09:41.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.881+0000 7fe81196a740 1 Processor -- start 2024-05-12T22:09:41.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.881+0000 7fe81196a740 1 -- start start 2024-05-12T22:09:41.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.882+0000 7fe81196a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5629b26670f0 0x5629b26752d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:41.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.882+0000 7fe81196a740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5629b266add0 0x5629b268d740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:41.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.882+0000 7fe81196a740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5629b266b6f0 0x5629b268dc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:41.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.882+0000 7fe81196a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe8081ad910 con 0x5629b26670f0 2024-05-12T22:09:41.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.882+0000 7fe81196a740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fe8081ab1e0 con 0x5629b266add0 2024-05-12T22:09:41.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.882+0000 7fe81196a740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe8081a9030 con 0x5629b266b6f0 2024-05-12T22:09:41.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.882+0000 7fe80cc00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5629b266b6f0 0x5629b268dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:41.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.882+0000 7fe807e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5629b26670f0 0x5629b26752d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:41.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.882+0000 7fe7ff400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5629b266add0 0x5629b268d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:41.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.882+0000 7fe80cc00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5629b266b6f0 0x5629b268dc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:34914/0 (socket says 172.21.15.28:34914) 2024-05-12T22:09:41.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.882+0000 7fe80cc00640 1 -- 172.21.15.28:0/792601282 learned_addr learned my addr 172.21.15.28:0/792601282 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:09:41.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.883+0000 7fe7ff400640 1 -- 172.21.15.28:0/792601282 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5629b266b6f0 msgr2=0x5629b268dc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:09:41.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.883+0000 7fe7ff400640 1 --2- 172.21.15.28:0/792601282 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5629b266b6f0 0x5629b268dc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:41.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.883+0000 7fe7ff400640 1 -- 172.21.15.28:0/792601282 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5629b26670f0 msgr2=0x5629b26752d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:09:41.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.883+0000 7fe7ff400640 1 --2- 172.21.15.28:0/792601282 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5629b26670f0 0x5629b26752d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:41.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.883+0000 7fe7ff400640 1 -- 172.21.15.28:0/792601282 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5629b2678520 con 0x5629b266add0 2024-05-12T22:09:41.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.883+0000 7fe7ff400640 1 --2- 172.21.15.28:0/792601282 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5629b266add0 0x5629b268d740 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fe7f00ece00 tx=0x7fe7f003b650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e5c2e45da5359957 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:09:41.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.883+0000 7fe7f7e00640 1 -- 172.21.15.28:0/792601282 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7f0049040 con 0x5629b266add0 2024-05-12T22:09:41.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.883+0000 7fe7f7e00640 1 -- 172.21.15.28:0/792601282 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe7f0031de0 con 0x5629b266add0 2024-05-12T22:09:41.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.883+0000 7fe7f7e00640 1 -- 172.21.15.28:0/792601282 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7f00c4610 con 0x5629b266add0 2024-05-12T22:09:41.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.884+0000 7fe81196a740 1 -- 172.21.15.28:0/792601282 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5629b266add0 msgr2=0x5629b268d740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:41.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.884+0000 7fe81196a740 1 --2- 172.21.15.28:0/792601282 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5629b266add0 0x5629b268d740 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fe7f00ece00 tx=0x7fe7f003b650 comp rx=0 tx=0).stop 2024-05-12T22:09:41.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.884+0000 7fe81196a740 1 -- 172.21.15.28:0/792601282 shutdown_connections 2024-05-12T22:09:41.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.884+0000 7fe81196a740 1 --2- 172.21.15.28:0/792601282 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5629b266b6f0 0x5629b268dc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:41.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.884+0000 7fe81196a740 1 --2- 172.21.15.28:0/792601282 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5629b266add0 0x5629b268d740 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:41.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.884+0000 7fe81196a740 1 --2- 172.21.15.28:0/792601282 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5629b26670f0 0x5629b26752d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:41.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.884+0000 7fe81196a740 1 -- 172.21.15.28:0/792601282 >> 172.21.15.28:0/792601282 conn(0x5629b26716d0 msgr2=0x5629b266cdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:09:41.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.884+0000 7fe81196a740 1 -- 172.21.15.28:0/792601282 shutdown_connections 2024-05-12T22:09:41.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.884+0000 7fe81196a740 1 -- 172.21.15.28:0/792601282 wait complete. 2024-05-12T22:09:41.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.885+0000 7fe81196a740 1 -- start start 2024-05-12T22:09:41.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.885+0000 7fe81196a740 1 Processor -- start 2024-05-12T22:09:41.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.885+0000 7fe81196a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5629b26670f0 0x5629b267de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:41.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.885+0000 7fe81196a740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5629b266add0 0x5629b267e340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:41.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.885+0000 7fe81196a740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5629b266b6f0 0x5629b266f170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:41.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.885+0000 7fe81196a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe8081a96a0 con 0x5629b26670f0 2024-05-12T22:09:41.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.885+0000 7fe81196a740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fe8081aab10 con 0x5629b266b6f0 2024-05-12T22:09:41.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.885+0000 7fe81196a740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe8081a9d70 con 0x5629b266add0 2024-05-12T22:09:41.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.885+0000 7fe81196a740 10 client.0 _ll_register_callbacks cb 0x7fe800010980 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:09:41.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.886+0000 7fe807e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5629b26670f0 0x5629b267de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:41.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.886+0000 7fe807e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5629b26670f0 0x5629b267de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51090/0 (socket says 172.21.15.28:51090) 2024-05-12T22:09:41.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.886+0000 7fe807e00640 1 -- 172.21.15.28:0/4126910313 learned_addr learned my addr 172.21.15.28:0/4126910313 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:09:41.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.886+0000 7fe7ff400640 1 --2- 172.21.15.28:0/4126910313 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5629b266add0 0x5629b267e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:41.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.886+0000 7fe80cc00640 1 --2- 172.21.15.28:0/4126910313 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5629b266b6f0 0x5629b266f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:41.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.886+0000 7fe807e00640 1 -- 172.21.15.28:0/4126910313 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5629b266add0 msgr2=0x5629b267e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:09:41.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.886+0000 7fe807e00640 1 --2- 172.21.15.28:0/4126910313 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5629b266add0 0x5629b267e340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:41.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.886+0000 7fe807e00640 1 -- 172.21.15.28:0/4126910313 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5629b266b6f0 msgr2=0x5629b266f170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:09:41.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.886+0000 7fe807e00640 1 --2- 172.21.15.28:0/4126910313 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5629b266b6f0 0x5629b266f170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:41.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.886+0000 7fe807e00640 1 -- 172.21.15.28:0/4126910313 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5629b266f9f0 con 0x5629b26670f0 2024-05-12T22:09:41.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.886+0000 7fe807e00640 1 --2- 172.21.15.28:0/4126910313 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5629b26670f0 0x5629b267de00 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fe7f800e610 tx=0x7fe7f8009ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7b7d0d6cb037eda server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:09:41.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.886+0000 7fe7f5600640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-12T22:09:41.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.887+0000 7fe7f5600640 1 -- 172.21.15.28:0/4126910313 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7f801e040 con 0x5629b26670f0 2024-05-12T22:09:41.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.887+0000 7fe7f5600640 1 -- 172.21.15.28:0/4126910313 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7fe7f8005450 con 0x5629b26670f0 2024-05-12T22:09:41.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.887+0000 7fe81196a740 1 -- 172.21.15.28:0/4126910313 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe7f0031de0 con 0x5629b26670f0 2024-05-12T22:09:41.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.887+0000 7fe7f5600640 1 -- 172.21.15.28:0/4126910313 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fe7f803c050 con 0x5629b26670f0 2024-05-12T22:09:41.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.887+0000 7fe81196a740 10 client.15447 fetch_fsmap learned FSMap version 24 2024-05-12T22:09:41.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.887+0000 7fe81196a740 1 -- 172.21.15.28:0/4126910313 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5629b26ebb60 con 0x5629b26670f0 2024-05-12T22:09:41.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.887+0000 7fe7f5600640 1 -- 172.21.15.28:0/4126910313 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe7f0031de0 con 0x5629b26670f0 2024-05-12T22:09:41.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.887+0000 7fe7f5600640 1 -- 172.21.15.28:0/4126910313 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7fe7f80048c0 con 0x5629b26670f0 2024-05-12T22:09:41.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.888+0000 7fe81196a740 10 client.15447 fetch_fsmap finished waiting for FSMap version 24 2024-05-12T22:09:41.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.888+0000 7fe81196a740 10 client.15447 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:09:41.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.888+0000 7fe81196a740 20 client.15447 populate_metadata read hostname 'smithi028' 2024-05-12T22:09:41.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.888+0000 7fe81196a740 1 --2- 172.21.15.28:0/4126910313 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5629b2675dc0 0x5629b26ee0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:09:41.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.888+0000 7fe81196a740 4 client.15447 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-12T22:09:41.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.888+0000 7fe81196a740 1 -- 172.21.15.28:0/4126910313 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5629b25d7610 con 0x5629b2675dc0 2024-05-12T22:09:41.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.888+0000 7fe7ff400640 1 --2- 172.21.15.28:0/4126910313 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5629b2675dc0 0x5629b26ee0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:09:41.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.888+0000 7fe7ff400640 1 --2- 172.21.15.28:0/4126910313 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5629b2675dc0 0x5629b26ee0a0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3bef2748343c353c server_cookie=fb055c8e4ded8050 in_seq=0 out_seq=0 2024-05-12T22:09:41.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.888+0000 7fe7f5600640 10 client.15447 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:09:41.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.889+0000 7fe7f5600640 1 -- 172.21.15.28:0/4126910313 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5629b25d7610 con 0x5629b2675dc0 2024-05-12T22:09:41.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.889+0000 7fe7f5600640 10 client.15447 handle_command_reply: tid=0 2024-05-12T22:09:41.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.889+0000 7fe81196a740 1 client.15447 shutdown 2024-05-12T22:09:41.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.889+0000 7fe81196a740 1 -- 172.21.15.28:0/4126910313 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5629b26670f0 msgr2=0x5629b267de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:41.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.889+0000 7fe81196a740 1 --2- 172.21.15.28:0/4126910313 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5629b26670f0 0x5629b267de00 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fe7f800e610 tx=0x7fe7f8009ba0 comp rx=0 tx=0).stop 2024-05-12T22:09:41.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.889+0000 7fe81196a740 1 -- 172.21.15.28:0/4126910313 shutdown_connections 2024-05-12T22:09:41.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.889+0000 7fe81196a740 1 --2- 172.21.15.28:0/4126910313 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5629b2675dc0 0x5629b26ee0a0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:41.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.889+0000 7fe81196a740 1 --2- 172.21.15.28:0/4126910313 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5629b266b6f0 0x5629b266f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:41.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.889+0000 7fe81196a740 1 --2- 172.21.15.28:0/4126910313 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5629b266add0 0x5629b267e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:41.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.889+0000 7fe81196a740 1 --2- 172.21.15.28:0/4126910313 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5629b26670f0 0x5629b267de00 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:41.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.889+0000 7fe7f5600640 0 client.15447 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:09:41.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.889+0000 7fe81196a740 1 -- 172.21.15.28:0/4126910313 >> 172.21.15.28:0/4126910313 conn(0x5629b26716d0 msgr2=0x5629b266d5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:09:41.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.889+0000 7fe81196a740 1 -- 172.21.15.28:0/4126910313 shutdown_connections 2024-05-12T22:09:41.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.889+0000 7fe81196a740 1 -- 172.21.15.28:0/4126910313 wait complete. 2024-05-12T22:09:41.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.889+0000 7fe81196a740 20 client.15447 trim_cache size 0 max 16384 2024-05-12T22:09:41.894 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-05-12T22:09:41.894 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-12T22:09:41.894 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-05-12T22:09:41.894 INFO:teuthology.orchestra.run.smithi028.stdout: "5bb0ab83-29d9-4339-85ed-4ccb15c90b95": { 2024-05-12T22:09:41.894 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-05-12T22:09:41.894 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "5bb0ab83-29d9-4339-85ed-4ccb15c90b95", 2024-05-12T22:09:41.894 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-05-12T22:09:41.894 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:09:41.894 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:09:41.894 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-05-12T22:09:41.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.890+0000 7fe80d600640 1 -- 172.21.15.28:0/2771719685 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fe7ec0a2e20 msgr2=0x7fe7ec0a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:41.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.890+0000 7fe80d600640 1 --2- 172.21.15.28:0/2771719685 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fe7ec0a2e20 0x7fe7ec0a52e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fe800009fd0 tx=0x7fe800012040 comp rx=0 tx=0).stop 2024-05-12T22:09:41.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.890+0000 7fe80d600640 1 -- 172.21.15.28:0/2771719685 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe8081020a0 msgr2=0x7fe808198510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:09:41.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.890+0000 7fe80d600640 1 --2- 172.21.15.28:0/2771719685 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe8081020a0 0x7fe808198510 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fe7f002a9f0 tx=0x7fe7f0003c10 comp rx=0 tx=0).stop 2024-05-12T22:09:41.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.890+0000 7fe80d600640 1 -- 172.21.15.28:0/2771719685 shutdown_connections 2024-05-12T22:09:41.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.890+0000 7fe80d600640 1 --2- 172.21.15.28:0/2771719685 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fe7ec0a2e20 0x7fe7ec0a52e0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:41.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.890+0000 7fe80d600640 1 --2- 172.21.15.28:0/2771719685 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe8081076f0 0x7fe80819c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:41.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.890+0000 7fe80d600640 1 --2- 172.21.15.28:0/2771719685 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe8081020a0 0x7fe808198510 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:41.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.890+0000 7fe80d600640 1 --2- 172.21.15.28:0/2771719685 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8081016e0 0x7fe808197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:09:41.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.890+0000 7fe80d600640 1 -- 172.21.15.28:0/2771719685 >> 172.21.15.28:0/2771719685 conn(0x7fe8080fd1d0 msgr2=0x7fe8080fd950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:09:41.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.890+0000 7fe80d600640 1 -- 172.21.15.28:0/2771719685 shutdown_connections 2024-05-12T22:09:41.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:09:41.890+0000 7fe80d600640 1 -- 172.21.15.28:0/2771719685 wait complete. 2024-05-12T22:09:41.909 INFO:tasks.cephfs.filesystem:scrub status for tag:5bb0ab83-29d9-4339-85ed-4ccb15c90b95 - {'path': '/', 'tag': '5bb0ab83-29d9-4339-85ed-4ccb15c90b95', 'options': 'recursive,force'} 2024-05-12T22:09:42.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:42 smithi186 ceph-mon[34785]: pgmap v24: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-12T22:09:42.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:42 smithi086 ceph-mon[34297]: pgmap v24: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-12T22:09:42.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:42 smithi028 ceph-mon[29847]: pgmap v24: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-12T22:09:43.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:43 smithi186 ceph-mon[34785]: scrub queued for path: / 2024-05-12T22:09:43.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:43 smithi186 ceph-mon[34785]: scrub summary: idle+waiting paths [/] 2024-05-12T22:09:43.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:43 smithi186 ceph-mon[34785]: scrub summary: active paths [/] 2024-05-12T22:09:43.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:43 smithi186 ceph-mon[34785]: scrub summary: idle+waiting paths [/] 2024-05-12T22:09:43.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:43 smithi186 ceph-mon[34785]: scrub summary: idle 2024-05-12T22:09:43.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:43 smithi086 ceph-mon[34297]: scrub queued for path: / 2024-05-12T22:09:43.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:43 smithi086 ceph-mon[34297]: scrub summary: idle+waiting paths [/] 2024-05-12T22:09:43.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:43 smithi086 ceph-mon[34297]: scrub summary: active paths [/] 2024-05-12T22:09:43.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:43 smithi086 ceph-mon[34297]: scrub summary: idle+waiting paths [/] 2024-05-12T22:09:43.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:43 smithi086 ceph-mon[34297]: scrub summary: idle 2024-05-12T22:09:43.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:43 smithi028 ceph-mon[29847]: scrub queued for path: / 2024-05-12T22:09:43.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:43 smithi028 ceph-mon[29847]: scrub summary: idle+waiting paths [/] 2024-05-12T22:09:43.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:43 smithi028 ceph-mon[29847]: scrub summary: active paths [/] 2024-05-12T22:09:43.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:43 smithi028 ceph-mon[29847]: scrub summary: idle+waiting paths [/] 2024-05-12T22:09:43.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:43 smithi028 ceph-mon[29847]: scrub summary: idle 2024-05-12T22:09:44.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:44 smithi186 ceph-mon[34785]: pgmap v25: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:09:44.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:44 smithi086 ceph-mon[34297]: pgmap v25: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:09:44.534 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:44 smithi028 ceph-mon[29847]: pgmap v25: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:09:46.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:46 smithi186 ceph-mon[34785]: pgmap v26: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-12T22:09:46.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:46 smithi086 ceph-mon[34297]: pgmap v26: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-12T22:09:46.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:46 smithi028 ceph-mon[29847]: pgmap v26: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-12T22:09:48.337 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:48 smithi028 ceph-mon[29847]: pgmap v27: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-05-12T22:09:48.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:48 smithi186 ceph-mon[34785]: pgmap v27: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-05-12T22:09:48.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:48 smithi086 ceph-mon[34297]: pgmap v27: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-05-12T22:09:50.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:50 smithi186 ceph-mon[34785]: pgmap v28: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-05-12T22:09:50.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:50 smithi086 ceph-mon[34297]: pgmap v28: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-05-12T22:09:50.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:50 smithi028 ceph-mon[29847]: pgmap v28: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-05-12T22:09:52.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:52 smithi186 ceph-mon[34785]: pgmap v29: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 682 B/s wr, 8 op/s 2024-05-12T22:09:52.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:52 smithi086 ceph-mon[34297]: pgmap v29: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 682 B/s wr, 8 op/s 2024-05-12T22:09:52.534 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:52 smithi028 ceph-mon[29847]: pgmap v29: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 682 B/s wr, 8 op/s 2024-05-12T22:09:54.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:54 smithi186 ceph-mon[34785]: pgmap v30: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 682 B/s wr, 5 op/s 2024-05-12T22:09:54.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:54 smithi086 ceph-mon[34297]: pgmap v30: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 682 B/s wr, 5 op/s 2024-05-12T22:09:54.534 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:54 smithi028 ceph-mon[29847]: pgmap v30: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 682 B/s wr, 5 op/s 2024-05-12T22:09:56.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:56 smithi186 ceph-mon[34785]: pgmap v31: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 682 B/s wr, 7 op/s 2024-05-12T22:09:56.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:56 smithi086 ceph-mon[34297]: pgmap v31: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 682 B/s wr, 7 op/s 2024-05-12T22:09:56.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:56 smithi028 ceph-mon[29847]: pgmap v31: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 682 B/s wr, 7 op/s 2024-05-12T22:09:58.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:09:58 smithi186 ceph-mon[34785]: pgmap v32: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 682 B/s wr, 7 op/s 2024-05-12T22:09:58.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:09:58 smithi086 ceph-mon[34297]: pgmap v32: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 682 B/s wr, 7 op/s 2024-05-12T22:09:58.534 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:09:58 smithi028 ceph-mon[29847]: pgmap v32: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 682 B/s wr, 7 op/s 2024-05-12T22:10:00.372 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:10:00 smithi028 ceph-mon[29847]: pgmap v33: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 682 B/s wr, 5 op/s 2024-05-12T22:10:00.372 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:10:00 smithi028 ceph-mon[29847]: overall HEALTH_OK 2024-05-12T22:10:00.372 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 22:10:00 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T22:10:00.369+0000 7f3c57000700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-12T22:10:00.373 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 22:10:00 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T22:10:00.369+0000 7f3c57000700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-12T22:10:00.373 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 22:10:00 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T22:10:00.369+0000 7f3c57000700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-12T22:10:00.373 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 22:10:00 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T22:10:00.369+0000 7f3c57000700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-12T22:10:00.373 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 22:10:00 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T22:10:00.369+0000 7f3c57000700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-12T22:10:00.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:10:00 smithi186 ceph-mon[34785]: pgmap v33: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 682 B/s wr, 5 op/s 2024-05-12T22:10:00.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:10:00 smithi186 ceph-mon[34785]: overall HEALTH_OK 2024-05-12T22:10:00.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:10:00 smithi086 ceph-mon[34297]: pgmap v33: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 682 B/s wr, 5 op/s 2024-05-12T22:10:00.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:10:00 smithi086 ceph-mon[34297]: overall HEALTH_OK 2024-05-12T22:10:02.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:10:02 smithi186 ceph-mon[34785]: pgmap v34: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 682 B/s wr, 7 op/s 2024-05-12T22:10:02.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:10:02 smithi186 ceph-mon[34785]: mgrmap e26: x(active, since 62s), standbys: z, y 2024-05-12T22:10:02.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:10:02 smithi086 ceph-mon[34297]: pgmap v34: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 682 B/s wr, 7 op/s 2024-05-12T22:10:02.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:10:02 smithi086 ceph-mon[34297]: mgrmap e26: x(active, since 62s), standbys: z, y 2024-05-12T22:10:02.534 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:10:02 smithi028 ceph-mon[29847]: pgmap v34: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 682 B/s wr, 7 op/s 2024-05-12T22:10:02.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:10:02 smithi028 ceph-mon[29847]: mgrmap e26: x(active, since 62s), standbys: z, y 2024-05-12T22:10:04.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:10:04 smithi186 ceph-mon[34785]: pgmap v35: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:10:04.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:10:04 smithi086 ceph-mon[34297]: pgmap v35: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:10:04.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:10:04 smithi028 ceph-mon[29847]: pgmap v35: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:10:06.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:10:06 smithi186 ceph-mon[34785]: pgmap v36: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 12 KiB/s wr, 9 op/s 2024-05-12T22:10:06.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:10:06 smithi086 ceph-mon[34297]: pgmap v36: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 12 KiB/s wr, 9 op/s 2024-05-12T22:10:06.534 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:10:06 smithi028 ceph-mon[29847]: pgmap v36: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 12 KiB/s wr, 9 op/s 2024-05-12T22:10:08.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:10:08 smithi186 ceph-mon[34785]: pgmap v37: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 21 op/s 2024-05-12T22:10:08.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:10:08 smithi186 ceph-mon[34785]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:10:08.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:10:08 smithi086 ceph-mon[34297]: pgmap v37: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 21 op/s 2024-05-12T22:10:08.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:10:08 smithi086 ceph-mon[34297]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:10:08.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:10:08 smithi028 ceph-mon[29847]: pgmap v37: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 21 op/s 2024-05-12T22:10:08.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:10:08 smithi028 ceph-mon[29847]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:10:09.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:10:09 smithi186 ceph-mon[34785]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:10:09.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:10:09 smithi186 ceph-mon[34785]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:10:09.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:10:09 smithi186 ceph-mon[34785]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:10:09.930 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:10:09 smithi186 ceph-mon[34785]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:10:09.930 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:10:09 smithi186 ceph-mon[34785]: pgmap v38: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-05-12T22:10:09.930 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:10:09 smithi186 ceph-mon[34785]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:10:09.930 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:10:09 smithi186 ceph-mon[34785]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:10:09.930 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:10:09 smithi186 ceph-mon[34785]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:10:09.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:10:09 smithi086 ceph-mon[34297]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:10:09.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:10:09 smithi086 ceph-mon[34297]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:10:09.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:10:09 smithi086 ceph-mon[34297]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:10:09.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:10:09 smithi086 ceph-mon[34297]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:10:09.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:10:09 smithi086 ceph-mon[34297]: pgmap v38: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-05-12T22:10:09.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:10:09 smithi086 ceph-mon[34297]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:10:09.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:10:09 smithi086 ceph-mon[34297]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:10:09.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:10:09 smithi086 ceph-mon[34297]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:10:10.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:10:09 smithi028 ceph-mon[29847]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:10:10.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:10:09 smithi028 ceph-mon[29847]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:10:10.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:10:09 smithi028 ceph-mon[29847]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:10:10.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:10:09 smithi028 ceph-mon[29847]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:10:10.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:10:09 smithi028 ceph-mon[29847]: pgmap v38: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-05-12T22:10:10.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:10:09 smithi028 ceph-mon[29847]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:10:10.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:10:09 smithi028 ceph-mon[29847]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:10:10.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:10:09 smithi028 ceph-mon[29847]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:10:11.909 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-12T22:10:11.910 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-12T22:10:12.002 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:10:11 smithi028 ceph-mon[29847]: pgmap v39: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-05-12T22:10:12.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.164+0000 7fd0d6c00640 1 -- 172.21.15.28:0/2727053934 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd0d0074500 msgr2=0x7fd0d010da50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:10:12.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.164+0000 7fd0d6c00640 1 --2- 172.21.15.28:0/2727053934 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd0d0074500 0x7fd0d010da50 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fd0bc009860 tx=0x7fd0bc02a3d0 comp rx=0 tx=0).stop 2024-05-12T22:10:12.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.164+0000 7fd0d6c00640 1 -- 172.21.15.28:0/2727053934 shutdown_connections 2024-05-12T22:10:12.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.164+0000 7fd0d6c00640 1 --2- 172.21.15.28:0/2727053934 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd0d010df90 0x7fd0d0110380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:12.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.164+0000 7fd0d6c00640 1 --2- 172.21.15.28:0/2727053934 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd0d0074500 0x7fd0d010da50 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:12.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.164+0000 7fd0d6c00640 1 --2- 172.21.15.28:0/2727053934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0d0073be0 0x7fd0d0073fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:12.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.164+0000 7fd0d6c00640 1 -- 172.21.15.28:0/2727053934 >> 172.21.15.28:0/2727053934 conn(0x7fd0d00fd200 msgr2=0x7fd0d00ff620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:10:12.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.165+0000 7fd0d6c00640 1 -- 172.21.15.28:0/2727053934 shutdown_connections 2024-05-12T22:10:12.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.165+0000 7fd0d6c00640 1 -- 172.21.15.28:0/2727053934 wait complete. 2024-05-12T22:10:12.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.166+0000 7fd0d6c00640 1 Processor -- start 2024-05-12T22:10:12.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.166+0000 7fd0d6c00640 1 -- start start 2024-05-12T22:10:12.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.166+0000 7fd0d6c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0d0073be0 0x7fd0d019a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:12.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.166+0000 7fd0d6c00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd0d0074500 0x7fd0d019a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:12.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.166+0000 7fd0d6c00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd0d010df90 0x7fd0d019ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:12.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.166+0000 7fd0d6c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd0d0113cb0 con 0x7fd0d0073be0 2024-05-12T22:10:12.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.166+0000 7fd0d6c00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fd0d0113b30 con 0x7fd0d0074500 2024-05-12T22:10:12.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.166+0000 7fd0d6c00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd0d0113e30 con 0x7fd0d010df90 2024-05-12T22:10:12.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.166+0000 7fd0d4e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd0d0074500 0x7fd0d019a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:12.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.167+0000 7fd0d4e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd0d0074500 0x7fd0d019a7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:54442/0 (socket says 172.21.15.28:54442) 2024-05-12T22:10:12.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.167+0000 7fd0d4e00640 1 -- 172.21.15.28:0/1905354129 learned_addr learned my addr 172.21.15.28:0/1905354129 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:10:12.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.167+0000 7fd0d5800640 1 --2- 172.21.15.28:0/1905354129 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0d0073be0 0x7fd0d019a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:12.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.167+0000 7fd0d6200640 1 --2- 172.21.15.28:0/1905354129 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd0d010df90 0x7fd0d019ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:12.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.167+0000 7fd0d4e00640 1 -- 172.21.15.28:0/1905354129 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd0d010df90 msgr2=0x7fd0d019ead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:10:12.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.167+0000 7fd0d4e00640 1 --2- 172.21.15.28:0/1905354129 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd0d010df90 0x7fd0d019ead0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:12.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.167+0000 7fd0d4e00640 1 -- 172.21.15.28:0/1905354129 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0d0073be0 msgr2=0x7fd0d019a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:10:12.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.167+0000 7fd0d4e00640 1 --2- 172.21.15.28:0/1905354129 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0d0073be0 0x7fd0d019a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:12.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.167+0000 7fd0d4e00640 1 -- 172.21.15.28:0/1905354129 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0d019f200 con 0x7fd0d0074500 2024-05-12T22:10:12.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.167+0000 7fd0d4e00640 1 --2- 172.21.15.28:0/1905354129 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd0d0074500 0x7fd0d019a7a0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fd0bc02a8f0 tx=0x7fd0bc030650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:10:12.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.168+0000 7fd0ce000640 1 -- 172.21.15.28:0/1905354129 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0bc040020 con 0x7fd0d0074500 2024-05-12T22:10:12.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.168+0000 7fd0d6c00640 1 -- 172.21.15.28:0/1905354129 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0d019f490 con 0x7fd0d0074500 2024-05-12T22:10:12.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.168+0000 7fd0d6c00640 1 -- 172.21.15.28:0/1905354129 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0d01a6d80 con 0x7fd0d0074500 2024-05-12T22:10:12.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.168+0000 7fd0ce000640 1 -- 172.21.15.28:0/1905354129 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd0bc004460 con 0x7fd0d0074500 2024-05-12T22:10:12.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.168+0000 7fd0ce000640 1 -- 172.21.15.28:0/1905354129 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0bc0046c0 con 0x7fd0d0074500 2024-05-12T22:10:12.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.172+0000 7fd0db1c8740 1 Processor -- start 2024-05-12T22:10:12.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.172+0000 7fd0db1c8740 1 -- start start 2024-05-12T22:10:12.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.172+0000 7fd0db1c8740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55704e7925e0 0x55704e7929c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:12.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.173+0000 7fd0db1c8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55704e792f00 0x55704e79d090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:12.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.173+0000 7fd0db1c8740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55704e79d5d0 0x55704e79f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:12.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.173+0000 7fd0db1c8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55704e7522a0 con 0x55704e792f00 2024-05-12T22:10:12.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.173+0000 7fd0db1c8740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x55704e6eeb90 con 0x55704e79d5d0 2024-05-12T22:10:12.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.173+0000 7fd0db1c8740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55704e6965b0 con 0x55704e7925e0 2024-05-12T22:10:12.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.173+0000 7fd0ce000640 1 -- 172.21.15.28:0/1905354129 <== mon.2 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 26) ==== 138117+0+0 (secure 0 0 0) 0x7fd0bc043020 con 0x7fd0d0074500 2024-05-12T22:10:12.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.173+0000 7fd0d6200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55704e7925e0 0x55704e7929c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:12.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.173+0000 7fd0d6200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55704e7925e0 0x55704e7929c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:42906/0 (socket says 172.21.15.28:42906) 2024-05-12T22:10:12.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.173+0000 7fd0d6200640 1 -- 172.21.15.28:0/414260813 learned_addr learned my addr 172.21.15.28:0/414260813 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:10:12.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.173+0000 7fd0d4e00640 1 --2- 172.21.15.28:0/414260813 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55704e79d5d0 0x55704e79f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:12.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.173+0000 7fd0d5800640 1 --2- 172.21.15.28:0/414260813 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55704e792f00 0x55704e79d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:12.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.174+0000 7fd0d6200640 1 -- 172.21.15.28:0/414260813 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55704e79d5d0 msgr2=0x55704e79f9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:10:12.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.174+0000 7fd0d6200640 1 --2- 172.21.15.28:0/414260813 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55704e79d5d0 0x55704e79f9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:12.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.174+0000 7fd0d6200640 1 -- 172.21.15.28:0/414260813 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55704e792f00 msgr2=0x55704e79d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:10:12.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.174+0000 7fd0d6200640 1 --2- 172.21.15.28:0/414260813 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55704e792f00 0x55704e79d090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:12.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.174+0000 7fd0d6200640 1 -- 172.21.15.28:0/414260813 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55704e4cc7d0 con 0x55704e7925e0 2024-05-12T22:10:12.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.174+0000 7fd0d5800640 1 --2- 172.21.15.28:0/414260813 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55704e792f00 0x55704e79d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:10:12.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.174+0000 7fd0d6200640 1 --2- 172.21.15.28:0/414260813 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55704e7925e0 0x55704e7929c0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fd0c8009e30 tx=0x7fd0c800f730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ea82d171bb0161fe server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:10:12.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.175+0000 7fd0c2a00640 1 -- 172.21.15.28:0/414260813 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0c8011640 con 0x55704e7925e0 2024-05-12T22:10:12.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.175+0000 7fd0c2a00640 1 -- 172.21.15.28:0/414260813 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd0c8011cc0 con 0x55704e7925e0 2024-05-12T22:10:12.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.175+0000 7fd0c2a00640 1 -- 172.21.15.28:0/414260813 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0c801bd00 con 0x55704e7925e0 2024-05-12T22:10:12.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.175+0000 7fd0ce000640 1 --2- 172.21.15.28:0/1905354129 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fd0b80a2f30 0x7fd0b80a53f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:12.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.175+0000 7fd0db1c8740 1 -- 172.21.15.28:0/414260813 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55704e7925e0 msgr2=0x55704e7929c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:10:12.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.175+0000 7fd0db1c8740 1 --2- 172.21.15.28:0/414260813 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55704e7925e0 0x55704e7929c0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fd0c8009e30 tx=0x7fd0c800f730 comp rx=0 tx=0).stop 2024-05-12T22:10:12.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.176+0000 7fd0ce000640 1 -- 172.21.15.28:0/1905354129 <== mon.2 v2:172.21.15.86:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7fd0bc0f04b0 con 0x7fd0d0074500 2024-05-12T22:10:12.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.176+0000 7fd0db1c8740 1 -- 172.21.15.28:0/414260813 shutdown_connections 2024-05-12T22:10:12.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.176+0000 7fd0db1c8740 1 --2- 172.21.15.28:0/414260813 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55704e79d5d0 0x55704e79f9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:12.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.176+0000 7fd0db1c8740 1 --2- 172.21.15.28:0/414260813 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55704e792f00 0x55704e79d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:12.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.176+0000 7fd0db1c8740 1 --2- 172.21.15.28:0/414260813 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55704e7925e0 0x55704e7929c0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:12.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.176+0000 7fd0db1c8740 1 -- 172.21.15.28:0/414260813 >> 172.21.15.28:0/414260813 conn(0x55704e78e900 msgr2=0x55704e78ece0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:10:12.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.176+0000 7fd0d5800640 1 --2- 172.21.15.28:0/1905354129 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fd0b80a2f30 0x7fd0b80a53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:12.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.176+0000 7fd0db1c8740 1 -- 172.21.15.28:0/414260813 shutdown_connections 2024-05-12T22:10:12.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.176+0000 7fd0db1c8740 1 -- 172.21.15.28:0/414260813 wait complete. 2024-05-12T22:10:12.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.177+0000 7fd0db1c8740 1 -- start start 2024-05-12T22:10:12.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.177+0000 7fd0d5800640 1 --2- 172.21.15.28:0/1905354129 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fd0b80a2f30 0x7fd0b80a53f0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fd0c40096c0 tx=0x7fd0c4009260 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:10:12.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.177+0000 7fd0db1c8740 1 Processor -- start 2024-05-12T22:10:12.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.177+0000 7fd0db1c8740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55704e7925e0 0x55704e79cee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:12.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.177+0000 7fd0db1c8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55704e792f00 0x55704e79a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:12.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.177+0000 7fd0db1c8740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55704e79d5d0 0x55704e79a710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:12.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.177+0000 7fd0db1c8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55704e6a99f0 con 0x55704e792f00 2024-05-12T22:10:12.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.177+0000 7fd0db1c8740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x55704e6eab50 con 0x55704e79d5d0 2024-05-12T22:10:12.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.178+0000 7fd0db1c8740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55704e7a03c0 con 0x55704e7925e0 2024-05-12T22:10:12.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.178+0000 7fd0db1c8740 10 client.0 _ll_register_callbacks cb 0x55704e401be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:10:12.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.178+0000 7fd0d5800640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55704e7925e0 0x55704e79cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:12.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.178+0000 7fd0d6200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55704e79d5d0 0x55704e79a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:12.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.178+0000 7fd0d5800640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55704e7925e0 0x55704e79cee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:42914/0 (socket says 172.21.15.28:42914) 2024-05-12T22:10:12.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.178+0000 7fd0d5800640 1 -- 172.21.15.28:0/3679768677 learned_addr learned my addr 172.21.15.28:0/3679768677 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:10:12.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.178+0000 7fd0d4e00640 1 --2- 172.21.15.28:0/3679768677 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55704e792f00 0x55704e79a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:12.180 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:10:11 smithi186 ceph-mon[34785]: pgmap v39: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-05-12T22:10:12.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.178+0000 7fd0d6200640 1 -- 172.21.15.28:0/3679768677 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55704e7925e0 msgr2=0x55704e79cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:10:12.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.178+0000 7fd0d6200640 1 --2- 172.21.15.28:0/3679768677 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55704e7925e0 0x55704e79cee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:12.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.178+0000 7fd0d6200640 1 -- 172.21.15.28:0/3679768677 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55704e792f00 msgr2=0x55704e79a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:10:12.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.178+0000 7fd0d6200640 1 --2- 172.21.15.28:0/3679768677 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55704e792f00 0x55704e79a1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:12.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.178+0000 7fd0d6200640 1 -- 172.21.15.28:0/3679768677 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55704e4023f0 con 0x55704e79d5d0 2024-05-12T22:10:12.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.178+0000 7fd0d4e00640 1 --2- 172.21.15.28:0/3679768677 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55704e792f00 0x55704e79a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:10:12.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.179+0000 7fd0d6200640 1 --2- 172.21.15.28:0/3679768677 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55704e79d5d0 0x55704e79a710 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fd0c800fc30 tx=0x7fd0c8005ac0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3cb627bfc2ff7f13 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:10:12.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.179+0000 7fd0b7400640 10 client.0 ms_handle_connect on v2:172.21.15.86:3300/0 2024-05-12T22:10:12.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.179+0000 7fd0b7400640 1 -- 172.21.15.28:0/3679768677 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0c8003970 con 0x55704e79d5d0 2024-05-12T22:10:12.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.179+0000 7fd0b7400640 1 -- 172.21.15.28:0/3679768677 <== mon.2 v2:172.21.15.86:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7fd0c8003b10 con 0x55704e79d5d0 2024-05-12T22:10:12.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.179+0000 7fd0db1c8740 1 -- 172.21.15.28:0/3679768677 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd0c8011cc0 con 0x55704e79d5d0 2024-05-12T22:10:12.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.180+0000 7fd0b7400640 1 -- 172.21.15.28:0/3679768677 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fd0c8003d10 con 0x55704e79d5d0 2024-05-12T22:10:12.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.180+0000 7fd0db1c8740 10 client.25583 fetch_fsmap learned FSMap version 24 2024-05-12T22:10:12.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.180+0000 7fd0db1c8740 1 -- 172.21.15.28:0/3679768677 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55704e3c20f0 con 0x55704e79d5d0 2024-05-12T22:10:12.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.180+0000 7fd0b7400640 1 -- 172.21.15.28:0/3679768677 <== mon.2 v2:172.21.15.86:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd0c8011cc0 con 0x55704e79d5d0 2024-05-12T22:10:12.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.180+0000 7fd0b7400640 1 -- 172.21.15.28:0/3679768677 <== mon.2 v2:172.21.15.86:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7fd0c8024070 con 0x55704e79d5d0 2024-05-12T22:10:12.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.180+0000 7fd0db1c8740 10 client.25583 fetch_fsmap finished waiting for FSMap version 24 2024-05-12T22:10:12.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.180+0000 7fd0db1c8740 10 client.25583 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:10:12.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.180+0000 7fd0db1c8740 20 client.25583 populate_metadata read hostname 'smithi028' 2024-05-12T22:10:12.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.180+0000 7fd0db1c8740 1 --2- 172.21.15.28:0/3679768677 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55704e7ad010 0x55704e7af400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:12.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.180+0000 7fd0db1c8740 4 client.25583 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-12T22:10:12.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.180+0000 7fd0db1c8740 1 -- 172.21.15.28:0/3679768677 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55704e6fc2c0 con 0x55704e7ad010 2024-05-12T22:10:12.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.181+0000 7fd0d5800640 1 --2- 172.21.15.28:0/3679768677 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55704e7ad010 0x55704e7af400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:12.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.181+0000 7fd0d5800640 1 --2- 172.21.15.28:0/3679768677 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55704e7ad010 0x55704e7af400 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eee1d6f31e4a4269 server_cookie=cbdb2f4d9693b67f in_seq=0 out_seq=0 2024-05-12T22:10:12.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.181+0000 7fd0b7400640 10 client.25583 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:10:12.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.182+0000 7fd0b7400640 1 -- 172.21.15.28:0/3679768677 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55704e6fc2c0 con 0x55704e7ad010 2024-05-12T22:10:12.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.182+0000 7fd0b7400640 10 client.25583 handle_command_reply: tid=0 2024-05-12T22:10:12.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.182+0000 7fd0db1c8740 1 client.25583 shutdown 2024-05-12T22:10:12.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.183+0000 7fd0db1c8740 1 -- 172.21.15.28:0/3679768677 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55704e79d5d0 msgr2=0x55704e79a710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:10:12.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.183+0000 7fd0db1c8740 1 --2- 172.21.15.28:0/3679768677 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55704e79d5d0 0x55704e79a710 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fd0c800fc30 tx=0x7fd0c8005ac0 comp rx=0 tx=0).stop 2024-05-12T22:10:12.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.183+0000 7fd0db1c8740 1 -- 172.21.15.28:0/3679768677 shutdown_connections 2024-05-12T22:10:12.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.183+0000 7fd0db1c8740 1 --2- 172.21.15.28:0/3679768677 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55704e7ad010 0x55704e7af400 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:12.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.183+0000 7fd0db1c8740 1 --2- 172.21.15.28:0/3679768677 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55704e79d5d0 0x55704e79a710 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:12.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.183+0000 7fd0db1c8740 1 --2- 172.21.15.28:0/3679768677 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55704e792f00 0x55704e79a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:12.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.183+0000 7fd0b7400640 0 client.25583 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:10:12.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.183+0000 7fd0db1c8740 1 --2- 172.21.15.28:0/3679768677 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55704e7925e0 0x55704e79cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:12.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.183+0000 7fd0db1c8740 1 -- 172.21.15.28:0/3679768677 >> 172.21.15.28:0/3679768677 conn(0x55704e78e900 msgr2=0x55704e793d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:10:12.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.183+0000 7fd0db1c8740 1 -- 172.21.15.28:0/3679768677 shutdown_connections 2024-05-12T22:10:12.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.183+0000 7fd0db1c8740 1 -- 172.21.15.28:0/3679768677 wait complete. 2024-05-12T22:10:12.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.183+0000 7fd0db1c8740 20 client.25583 trim_cache size 0 max 16384 2024-05-12T22:10:12.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.202+0000 7fd0db1c8740 1 Processor -- start 2024-05-12T22:10:12.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.202+0000 7fd0db1c8740 1 -- start start 2024-05-12T22:10:12.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.202+0000 7fd0db1c8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55704e7925e0 0x55704e794440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:12.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.202+0000 7fd0db1c8740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55704e792f00 0x55704e79ca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:12.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.202+0000 7fd0db1c8740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55704e79d5d0 0x55704e7b4a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:12.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.202+0000 7fd0db1c8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd0b0010f80 con 0x55704e7925e0 2024-05-12T22:10:12.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.202+0000 7fd0db1c8740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fd0b000e850 con 0x55704e79d5d0 2024-05-12T22:10:12.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.202+0000 7fd0db1c8740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd0b000c6a0 con 0x55704e792f00 2024-05-12T22:10:12.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.203+0000 7fd0d5800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55704e7925e0 0x55704e794440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:12.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.203+0000 7fd0d4e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55704e792f00 0x55704e79ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:12.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.203+0000 7fd0d4e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55704e792f00 0x55704e79ca10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:42918/0 (socket says 172.21.15.28:42918) 2024-05-12T22:10:12.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.203+0000 7fd0d5800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55704e7925e0 0x55704e794440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42162/0 (socket says 172.21.15.28:42162) 2024-05-12T22:10:12.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.203+0000 7fd0d4e00640 1 -- 172.21.15.28:0/2941224274 learned_addr learned my addr 172.21.15.28:0/2941224274 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:10:12.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.203+0000 7fd0d6200640 1 --2- 172.21.15.28:0/2941224274 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55704e79d5d0 0x55704e7b4a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:12.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.203+0000 7fd0d4e00640 1 -- 172.21.15.28:0/2941224274 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55704e79d5d0 msgr2=0x55704e7b4a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:10:12.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.203+0000 7fd0d4e00640 1 --2- 172.21.15.28:0/2941224274 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55704e79d5d0 0x55704e7b4a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:12.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.203+0000 7fd0d4e00640 1 -- 172.21.15.28:0/2941224274 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55704e7925e0 msgr2=0x55704e794440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:10:12.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.203+0000 7fd0d4e00640 1 --2- 172.21.15.28:0/2941224274 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55704e7925e0 0x55704e794440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:12.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.203+0000 7fd0d4e00640 1 -- 172.21.15.28:0/2941224274 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55704e792d30 con 0x55704e792f00 2024-05-12T22:10:12.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.204+0000 7fd0d4e00640 1 --2- 172.21.15.28:0/2941224274 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55704e792f00 0x55704e79ca10 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fd0bc009830 tx=0x7fd0bc0a7040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=59f1cca8bba106b6 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:10:12.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.204+0000 7fd0c2000640 1 -- 172.21.15.28:0/2941224274 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0bc0bc300 con 0x55704e792f00 2024-05-12T22:10:12.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.204+0000 7fd0c2000640 1 -- 172.21.15.28:0/2941224274 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd0bc0bcec0 con 0x55704e792f00 2024-05-12T22:10:12.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.204+0000 7fd0c2000640 1 -- 172.21.15.28:0/2941224274 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0bc0bc120 con 0x55704e792f00 2024-05-12T22:10:12.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.205+0000 7fd0db1c8740 1 -- 172.21.15.28:0/2941224274 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55704e792f00 msgr2=0x55704e79ca10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:10:12.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.205+0000 7fd0db1c8740 1 --2- 172.21.15.28:0/2941224274 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55704e792f00 0x55704e79ca10 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fd0bc009830 tx=0x7fd0bc0a7040 comp rx=0 tx=0).stop 2024-05-12T22:10:12.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.206+0000 7fd0db1c8740 1 -- 172.21.15.28:0/2941224274 shutdown_connections 2024-05-12T22:10:12.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.206+0000 7fd0db1c8740 1 --2- 172.21.15.28:0/2941224274 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55704e79d5d0 0x55704e7b4a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:12.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.206+0000 7fd0db1c8740 1 --2- 172.21.15.28:0/2941224274 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55704e792f00 0x55704e79ca10 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:12.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.206+0000 7fd0db1c8740 1 --2- 172.21.15.28:0/2941224274 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55704e7925e0 0x55704e794440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:12.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.206+0000 7fd0db1c8740 1 -- 172.21.15.28:0/2941224274 >> 172.21.15.28:0/2941224274 conn(0x55704e78e900 msgr2=0x55704e793d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:10:12.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.206+0000 7fd0db1c8740 1 -- 172.21.15.28:0/2941224274 shutdown_connections 2024-05-12T22:10:12.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.206+0000 7fd0db1c8740 1 -- 172.21.15.28:0/2941224274 wait complete. 2024-05-12T22:10:12.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.207+0000 7fd0db1c8740 1 -- start start 2024-05-12T22:10:12.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.207+0000 7fd0db1c8740 1 Processor -- start 2024-05-12T22:10:12.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.208+0000 7fd0db1c8740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55704e7925e0 0x55704e7a5740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:12.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.208+0000 7fd0db1c8740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55704e792f00 0x55704e7a1ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:12.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.208+0000 7fd0db1c8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55704e79d5d0 0x55704e7a2000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:12.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.208+0000 7fd0db1c8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd0b000cd10 con 0x55704e79d5d0 2024-05-12T22:10:12.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.208+0000 7fd0db1c8740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fd0b000e180 con 0x55704e7925e0 2024-05-12T22:10:12.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.208+0000 7fd0db1c8740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd0b000d3e0 con 0x55704e792f00 2024-05-12T22:10:12.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.208+0000 7fd0db1c8740 10 client.0 _ll_register_callbacks cb 0x7fd0c8015070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:10:12.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.208+0000 7fd0d5800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55704e7925e0 0x55704e7a5740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:12.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.208+0000 7fd0d5800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55704e7925e0 0x55704e7a5740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:54496/0 (socket says 172.21.15.28:54496) 2024-05-12T22:10:12.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.208+0000 7fd0d5800640 1 -- 172.21.15.28:0/1867329081 learned_addr learned my addr 172.21.15.28:0/1867329081 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:10:12.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.209+0000 7fd0d6200640 1 --2- 172.21.15.28:0/1867329081 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55704e79d5d0 0x55704e7a2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:12.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.209+0000 7fd0d4e00640 1 --2- 172.21.15.28:0/1867329081 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55704e792f00 0x55704e7a1ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:12.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.209+0000 7fd0d4e00640 1 -- 172.21.15.28:0/1867329081 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55704e7925e0 msgr2=0x55704e7a5740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:10:12.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.209+0000 7fd0d4e00640 1 --2- 172.21.15.28:0/1867329081 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55704e7925e0 0x55704e7a5740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:12.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.209+0000 7fd0d4e00640 1 -- 172.21.15.28:0/1867329081 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55704e79d5d0 msgr2=0x55704e7a2000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:10:12.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.209+0000 7fd0d4e00640 1 --2- 172.21.15.28:0/1867329081 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55704e79d5d0 0x55704e7a2000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:12.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.209+0000 7fd0d4e00640 1 -- 172.21.15.28:0/1867329081 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55704e7a2540 con 0x55704e792f00 2024-05-12T22:10:12.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.209+0000 7fd0d5800640 1 --2- 172.21.15.28:0/1867329081 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55704e7925e0 0x55704e7a5740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:10:12.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.209+0000 7fd0d4e00640 1 --2- 172.21.15.28:0/1867329081 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55704e792f00 0x55704e7a1ac0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fd0bc0c0d30 tx=0x7fd0bc0bacd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=40e5396d0ea6a9ac server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:10:12.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.209+0000 7fd0b7400640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-12T22:10:12.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.210+0000 7fd0b7400640 1 -- 172.21.15.28:0/1867329081 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0bc0bc120 con 0x55704e792f00 2024-05-12T22:10:12.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.210+0000 7fd0b7400640 1 -- 172.21.15.28:0/1867329081 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7fd0bc0c4fa0 con 0x55704e792f00 2024-05-12T22:10:12.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.210+0000 7fd0db1c8740 1 -- 172.21.15.28:0/1867329081 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd0bc0bcec0 con 0x55704e792f00 2024-05-12T22:10:12.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.210+0000 7fd0b7400640 1 -- 172.21.15.28:0/1867329081 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fd0bc0bc300 con 0x55704e792f00 2024-05-12T22:10:12.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.211+0000 7fd0db1c8740 10 client.25555 fetch_fsmap learned FSMap version 24 2024-05-12T22:10:12.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.211+0000 7fd0db1c8740 1 -- 172.21.15.28:0/1867329081 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55704e814000 con 0x55704e792f00 2024-05-12T22:10:12.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.211+0000 7fd0b7400640 1 -- 172.21.15.28:0/1867329081 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd0bc0bcec0 con 0x55704e792f00 2024-05-12T22:10:12.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.211+0000 7fd0b7400640 1 -- 172.21.15.28:0/1867329081 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7fd0bc0d0ce0 con 0x55704e792f00 2024-05-12T22:10:12.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.211+0000 7fd0db1c8740 10 client.25555 fetch_fsmap finished waiting for FSMap version 24 2024-05-12T22:10:12.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.211+0000 7fd0db1c8740 10 client.25555 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:10:12.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.211+0000 7fd0db1c8740 20 client.25555 populate_metadata read hostname 'smithi028' 2024-05-12T22:10:12.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.211+0000 7fd0db1c8740 1 --2- 172.21.15.28:0/1867329081 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55704e814540 0x55704e816960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:12.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.212+0000 7fd0db1c8740 4 client.25555 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-12T22:10:12.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.212+0000 7fd0db1c8740 1 -- 172.21.15.28:0/1867329081 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55704e6f6610 con 0x55704e814540 2024-05-12T22:10:12.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.212+0000 7fd0d5800640 1 --2- 172.21.15.28:0/1867329081 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55704e814540 0x55704e816960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:12.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.213+0000 7fd0d5800640 1 --2- 172.21.15.28:0/1867329081 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55704e814540 0x55704e816960 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a6b9f7e6db9f5456 server_cookie=ae98f50e3d9fc054 in_seq=0 out_seq=0 2024-05-12T22:10:12.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.213+0000 7fd0b7400640 10 client.25555 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:10:12.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.213+0000 7fd0b7400640 1 -- 172.21.15.28:0/1867329081 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55704e6f6610 con 0x55704e814540 2024-05-12T22:10:12.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.213+0000 7fd0b7400640 10 client.25555 handle_command_reply: tid=0 2024-05-12T22:10:12.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.213+0000 7fd0db1c8740 1 client.25555 shutdown 2024-05-12T22:10:12.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.213+0000 7fd0db1c8740 1 -- 172.21.15.28:0/1867329081 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55704e792f00 msgr2=0x55704e7a1ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:10:12.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.213+0000 7fd0db1c8740 1 --2- 172.21.15.28:0/1867329081 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55704e792f00 0x55704e7a1ac0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fd0bc0c0d30 tx=0x7fd0bc0bacd0 comp rx=0 tx=0).stop 2024-05-12T22:10:12.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.213+0000 7fd0db1c8740 1 -- 172.21.15.28:0/1867329081 shutdown_connections 2024-05-12T22:10:12.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.213+0000 7fd0db1c8740 1 --2- 172.21.15.28:0/1867329081 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55704e814540 0x55704e816960 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:12.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.213+0000 7fd0db1c8740 1 --2- 172.21.15.28:0/1867329081 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55704e79d5d0 0x55704e7a2000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:12.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.213+0000 7fd0db1c8740 1 --2- 172.21.15.28:0/1867329081 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55704e792f00 0x55704e7a1ac0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:12.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.213+0000 7fd0b7400640 0 client.25555 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:10:12.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.213+0000 7fd0db1c8740 1 --2- 172.21.15.28:0/1867329081 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55704e7925e0 0x55704e7a5740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:12.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.213+0000 7fd0db1c8740 1 -- 172.21.15.28:0/1867329081 >> 172.21.15.28:0/1867329081 conn(0x55704e78e900 msgr2=0x55704e79c290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:10:12.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.214+0000 7fd0db1c8740 1 -- 172.21.15.28:0/1867329081 shutdown_connections 2024-05-12T22:10:12.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.214+0000 7fd0db1c8740 1 -- 172.21.15.28:0/1867329081 wait complete. 2024-05-12T22:10:12.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.214+0000 7fd0db1c8740 20 client.25555 trim_cache size 0 max 16384 2024-05-12T22:10:12.216 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-05-12T22:10:12.216 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-05-12T22:10:12.216 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-05-12T22:10:12.216 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-05-12T22:10:12.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.215+0000 7fd0d6c00640 1 -- 172.21.15.28:0/1905354129 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fd0b80a2f30 msgr2=0x7fd0b80a53f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:10:12.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.215+0000 7fd0d6c00640 1 --2- 172.21.15.28:0/1905354129 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fd0b80a2f30 0x7fd0b80a53f0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fd0c40096c0 tx=0x7fd0c4009260 comp rx=0 tx=0).stop 2024-05-12T22:10:12.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.215+0000 7fd0d6c00640 1 -- 172.21.15.28:0/1905354129 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd0d0074500 msgr2=0x7fd0d019a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:10:12.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.215+0000 7fd0d6c00640 1 --2- 172.21.15.28:0/1905354129 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd0d0074500 0x7fd0d019a7a0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fd0bc02a8f0 tx=0x7fd0bc030650 comp rx=0 tx=0).stop 2024-05-12T22:10:12.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.215+0000 7fd0d6c00640 1 -- 172.21.15.28:0/1905354129 shutdown_connections 2024-05-12T22:10:12.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.215+0000 7fd0d6c00640 1 --2- 172.21.15.28:0/1905354129 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fd0b80a2f30 0x7fd0b80a53f0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:12.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.215+0000 7fd0d6c00640 1 --2- 172.21.15.28:0/1905354129 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd0d010df90 0x7fd0d019ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:12.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.215+0000 7fd0d6c00640 1 --2- 172.21.15.28:0/1905354129 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd0d0074500 0x7fd0d019a7a0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:12.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.215+0000 7fd0d6c00640 1 --2- 172.21.15.28:0/1905354129 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0d0073be0 0x7fd0d019a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:12.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.215+0000 7fd0d6c00640 1 -- 172.21.15.28:0/1905354129 >> 172.21.15.28:0/1905354129 conn(0x7fd0d00fd200 msgr2=0x7fd0d00fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:10:12.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.215+0000 7fd0d6c00640 1 -- 172.21.15.28:0/1905354129 shutdown_connections 2024-05-12T22:10:12.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.215+0000 7fd0d6c00640 1 -- 172.21.15.28:0/1905354129 wait complete. 2024-05-12T22:10:12.241 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-12T22:10:12.241 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-12T22:10:12.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:10:11 smithi086 ceph-mon[34297]: pgmap v39: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-05-12T22:10:12.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.454+0000 7fc34b000640 1 -- 172.21.15.28:0/1664998296 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc344102010 msgr2=0x7fc34410e520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:10:12.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.454+0000 7fc34b000640 1 --2- 172.21.15.28:0/1664998296 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc344102010 0x7fc34410e520 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fc33c009cd0 tx=0x7fc33c02a8f0 comp rx=0 tx=0).stop 2024-05-12T22:10:12.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.454+0000 7fc34b000640 1 -- 172.21.15.28:0/1664998296 shutdown_connections 2024-05-12T22:10:12.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.454+0000 7fc34b000640 1 --2- 172.21.15.28:0/1664998296 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc344102010 0x7fc34410e520 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:12.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.454+0000 7fc34b000640 1 --2- 172.21.15.28:0/1664998296 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc344101650 0x7fc344101ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:12.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.454+0000 7fc34b000640 1 --2- 172.21.15.28:0/1664998296 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc344107660 0x7fc344107a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:12.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.454+0000 7fc34b000640 1 -- 172.21.15.28:0/1664998296 >> 172.21.15.28:0/1664998296 conn(0x7fc3440fd160 msgr2=0x7fc3440ff580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:10:12.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.454+0000 7fc34b000640 1 -- 172.21.15.28:0/1664998296 shutdown_connections 2024-05-12T22:10:12.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.454+0000 7fc34b000640 1 -- 172.21.15.28:0/1664998296 wait complete. 2024-05-12T22:10:12.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.455+0000 7fc34b000640 1 Processor -- start 2024-05-12T22:10:12.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.455+0000 7fc34b000640 1 -- start start 2024-05-12T22:10:12.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.455+0000 7fc34b000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc344101650 0x7fc34419a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:12.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.455+0000 7fc34b000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc344102010 0x7fc34419a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:12.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.455+0000 7fc34b000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc344107660 0x7fc34419ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:12.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.455+0000 7fc34b000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc344074800 con 0x7fc344101650 2024-05-12T22:10:12.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.455+0000 7fc34b000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fc344074680 con 0x7fc344107660 2024-05-12T22:10:12.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.455+0000 7fc34b000640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc344074980 con 0x7fc344102010 2024-05-12T22:10:12.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.456+0000 7fc349c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc344101650 0x7fc34419a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:12.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.456+0000 7fc34a600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc344107660 0x7fc34419ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:12.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.456+0000 7fc349c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc344101650 0x7fc34419a200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42184/0 (socket says 172.21.15.28:42184) 2024-05-12T22:10:12.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.456+0000 7fc349c00640 1 -- 172.21.15.28:0/3369791554 learned_addr learned my addr 172.21.15.28:0/3369791554 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:10:12.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.456+0000 7fc349200640 1 --2- 172.21.15.28:0/3369791554 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc344102010 0x7fc34419a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:12.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.456+0000 7fc34a600640 1 -- 172.21.15.28:0/3369791554 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc344102010 msgr2=0x7fc34419a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:10:12.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.456+0000 7fc34a600640 1 --2- 172.21.15.28:0/3369791554 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc344102010 0x7fc34419a740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:12.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.456+0000 7fc34a600640 1 -- 172.21.15.28:0/3369791554 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc344101650 msgr2=0x7fc34419a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:10:12.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.456+0000 7fc34a600640 1 --2- 172.21.15.28:0/3369791554 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc344101650 0x7fc34419a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:12.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.456+0000 7fc34a600640 1 -- 172.21.15.28:0/3369791554 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc34419f170 con 0x7fc344107660 2024-05-12T22:10:12.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.456+0000 7fc349c00640 1 --2- 172.21.15.28:0/3369791554 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc344101650 0x7fc34419a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:10:12.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.457+0000 7fc34a600640 1 --2- 172.21.15.28:0/3369791554 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc344107660 0x7fc34419ea40 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fc33c02ae10 tx=0x7fc33c003f10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:10:12.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.457+0000 7fc342000640 1 -- 172.21.15.28:0/3369791554 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc33c030dc0 con 0x7fc344107660 2024-05-12T22:10:12.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.457+0000 7fc34b000640 1 -- 172.21.15.28:0/3369791554 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc34419f400 con 0x7fc344107660 2024-05-12T22:10:12.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.457+0000 7fc342000640 1 -- 172.21.15.28:0/3369791554 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc33c040070 con 0x7fc344107660 2024-05-12T22:10:12.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.457+0000 7fc342000640 1 -- 172.21.15.28:0/3369791554 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc33c045a30 con 0x7fc344107660 2024-05-12T22:10:12.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.457+0000 7fc34b000640 1 -- 172.21.15.28:0/3369791554 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3441a6cf0 con 0x7fc344107660 2024-05-12T22:10:12.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.459+0000 7fc34b000640 1 -- 172.21.15.28:0/3369791554 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc344068570 con 0x7fc344107660 2024-05-12T22:10:12.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.462+0000 7fc342000640 1 -- 172.21.15.28:0/3369791554 <== mon.2 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 26) ==== 138117+0+0 (secure 0 0 0) 0x7fc33c004690 con 0x7fc344107660 2024-05-12T22:10:12.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.464+0000 7fc342000640 1 --2- 172.21.15.28:0/3369791554 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fc32c0a2eb0 0x7fc32c0a5370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:12.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.464+0000 7fc342000640 1 -- 172.21.15.28:0/3369791554 <== mon.2 v2:172.21.15.86:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7fc33c0f04b0 con 0x7fc344107660 2024-05-12T22:10:12.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.465+0000 7fc349c00640 1 --2- 172.21.15.28:0/3369791554 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fc32c0a2eb0 0x7fc32c0a5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:12.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.466+0000 7fc349c00640 1 --2- 172.21.15.28:0/3369791554 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fc32c0a2eb0 0x7fc32c0a5370 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fc338011440 tx=0x7fc338005f90 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:10:12.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.468+0000 7fc342000640 1 -- 172.21.15.28:0/3369791554 <== mon.2 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fc33c0bd060 con 0x7fc344107660 2024-05-12T22:10:12.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.789+0000 7fc34b000640 1 -- 172.21.15.28:0/3369791554 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc3441a6f20 con 0x7fc344107660 2024-05-12T22:10:12.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.790+0000 7fc342000640 1 -- 172.21.15.28:0/3369791554 <== mon.2 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11864 (secure 0 0 0) 0x7fc33c0bc960 con 0x7fc344107660 2024-05-12T22:10:12.791 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-12T22:10:12.791 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1844141745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1844141745},{"type":"v1","addr":"172.21.15.186:6833","nonce":1844141745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6833/2960763758","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":2960763758},{"type":"v1","addr":"172.21.15.86:6833","nonce":2960763758}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24470,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/4131134219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":4131134219},{"type":"v1","addr":"172.21.15.28:6835","nonce":4131134219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24481,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/1727869064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1727869064},{"type":"v1","addr":"172.21.15.186:6835","nonce":1727869064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24482,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2938260704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2938260704},{"type":"v1","addr":"172.21.15.28:6837","nonce":2938260704}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24494,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3473700910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3473700910},{"type":"v1","addr":"172.21.15.86:6835","nonce":3473700910}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-12T22:06:33.426986+0000","modified":"2024-05-12T22:08:58.179491+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24581,"mds_1":24550,"mds_2":24532},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24499":{"gid":24499,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6837/3600809235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":3600809235},{"type":"v1","addr":"172.21.15.186:6837","nonce":3600809235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"i","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6839/2399610807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":2399610807},{"type":"v1","addr":"172.21.15.186:6839","nonce":2399610807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24530":{"gid":24530,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.86:6837/3782448884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":3782448884},{"type":"v1","addr":"172.21.15.86:6837","nonce":3782448884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24532":{"gid":24532,"name":"g","rank":2,"incarnation":21,"state":"up:active","state_seq":6,"addr":"172.21.15.28:6839/123892158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":123892158},{"type":"v1","addr":"172.21.15.28:6839","nonce":123892158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24550":{"gid":24550,"name":"a","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.28:6841/761183870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":761183870},{"type":"v1","addr":"172.21.15.28:6841","nonce":761183870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24581":{"gid":24581,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6839/1251999130","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6838","nonce":1251999130},{"type":"v1","addr":"172.21.15.86:6839","nonce":1251999130}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24581,"qdb_cluster":[24550,24581,24532]},"id":1}]} 2024-05-12T22:10:12.791 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 24 2024-05-12T22:10:12.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.792+0000 7fc34b000640 1 -- 172.21.15.28:0/3369791554 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fc32c0a2eb0 msgr2=0x7fc32c0a5370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:10:12.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.792+0000 7fc34b000640 1 --2- 172.21.15.28:0/3369791554 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fc32c0a2eb0 0x7fc32c0a5370 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fc338011440 tx=0x7fc338005f90 comp rx=0 tx=0).stop 2024-05-12T22:10:12.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.792+0000 7fc34b000640 1 -- 172.21.15.28:0/3369791554 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc344107660 msgr2=0x7fc34419ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:10:12.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.792+0000 7fc34b000640 1 --2- 172.21.15.28:0/3369791554 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc344107660 0x7fc34419ea40 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fc33c02ae10 tx=0x7fc33c003f10 comp rx=0 tx=0).stop 2024-05-12T22:10:12.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.793+0000 7fc34b000640 1 -- 172.21.15.28:0/3369791554 shutdown_connections 2024-05-12T22:10:12.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.793+0000 7fc34b000640 1 --2- 172.21.15.28:0/3369791554 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fc32c0a2eb0 0x7fc32c0a5370 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:12.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.793+0000 7fc34b000640 1 --2- 172.21.15.28:0/3369791554 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc344107660 0x7fc34419ea40 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:12.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.793+0000 7fc34b000640 1 --2- 172.21.15.28:0/3369791554 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc344102010 0x7fc34419a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:12.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.793+0000 7fc34b000640 1 --2- 172.21.15.28:0/3369791554 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc344101650 0x7fc34419a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:12.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.793+0000 7fc34b000640 1 -- 172.21.15.28:0/3369791554 >> 172.21.15.28:0/3369791554 conn(0x7fc3440fd160 msgr2=0x7fc344105160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:10:12.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.793+0000 7fc34b000640 1 -- 172.21.15.28:0/3369791554 shutdown_connections 2024-05-12T22:10:12.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.793+0000 7fc34b000640 1 -- 172.21.15.28:0/3369791554 wait complete. 2024-05-12T22:10:12.810 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-05-12T22:10:12.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.981+0000 7f648d800640 1 -- 172.21.15.28:0/2699202674 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6488073be0 msgr2=0x7f6488073fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:10:12.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.981+0000 7f648d800640 1 --2- 172.21.15.28:0/2699202674 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6488073be0 0x7f6488073fc0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f6478009900 tx=0x7f647802a480 comp rx=0 tx=0).stop 2024-05-12T22:10:12.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.981+0000 7f648d800640 1 -- 172.21.15.28:0/2699202674 shutdown_connections 2024-05-12T22:10:12.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.981+0000 7f648d800640 1 --2- 172.21.15.28:0/2699202674 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f648810df90 0x7f6488110380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:12.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.981+0000 7f648d800640 1 --2- 172.21.15.28:0/2699202674 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6488074500 0x7f648810da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:12.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.981+0000 7f648d800640 1 --2- 172.21.15.28:0/2699202674 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6488073be0 0x7f6488073fc0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:12.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.981+0000 7f648d800640 1 -- 172.21.15.28:0/2699202674 >> 172.21.15.28:0/2699202674 conn(0x7f64880fd200 msgr2=0x7f64880ff620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:10:12.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.981+0000 7f648d800640 1 -- 172.21.15.28:0/2699202674 shutdown_connections 2024-05-12T22:10:12.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.981+0000 7f648d800640 1 -- 172.21.15.28:0/2699202674 wait complete. 2024-05-12T22:10:12.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.982+0000 7f648d800640 1 Processor -- start 2024-05-12T22:10:12.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.982+0000 7f648d800640 1 -- start start 2024-05-12T22:10:12.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.983+0000 7f648d800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6488073be0 0x7f648819a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:12.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.983+0000 7f648d800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6488074500 0x7f648819a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:12.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.983+0000 7f648d800640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f648810df90 0x7f648819eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:12.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.983+0000 7f648d800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6488113cb0 con 0x7f6488073be0 2024-05-12T22:10:12.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.983+0000 7f648d800640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f6488113b30 con 0x7f6488074500 2024-05-12T22:10:12.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.983+0000 7f648d800640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f6488113e30 con 0x7f648810df90 2024-05-12T22:10:12.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.983+0000 7f6487e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6488073be0 0x7f648819a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:12.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.983+0000 7f6487e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6488073be0 0x7f648819a1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42206/0 (socket says 172.21.15.28:42206) 2024-05-12T22:10:12.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.983+0000 7f6487e00640 1 -- 172.21.15.28:0/1864505394 learned_addr learned my addr 172.21.15.28:0/1864505394 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:10:12.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.983+0000 7f648ce00640 1 --2- 172.21.15.28:0/1864505394 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f648810df90 0x7f648819eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:12.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.983+0000 7f6487400640 1 --2- 172.21.15.28:0/1864505394 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6488074500 0x7f648819a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:12.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.984+0000 7f6487e00640 1 -- 172.21.15.28:0/1864505394 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f648810df90 msgr2=0x7f648819eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:10:12.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.984+0000 7f6487e00640 1 --2- 172.21.15.28:0/1864505394 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f648810df90 0x7f648819eab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:12.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.984+0000 7f6487e00640 1 -- 172.21.15.28:0/1864505394 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6488074500 msgr2=0x7f648819a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:10:12.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.984+0000 7f6487e00640 1 --2- 172.21.15.28:0/1864505394 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6488074500 0x7f648819a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:12.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.984+0000 7f6487e00640 1 -- 172.21.15.28:0/1864505394 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f648819f1e0 con 0x7f6488073be0 2024-05-12T22:10:12.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.984+0000 7f648ce00640 1 --2- 172.21.15.28:0/1864505394 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f648810df90 0x7f648819eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:10:12.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.984+0000 7f6487e00640 1 --2- 172.21.15.28:0/1864505394 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6488073be0 0x7f648819a1f0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f6478006010 tx=0x7f64780042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:10:12.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.985+0000 7f6484c00640 1 -- 172.21.15.28:0/1864505394 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f647803f030 con 0x7f6488073be0 2024-05-12T22:10:12.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.985+0000 7f6484c00640 1 -- 172.21.15.28:0/1864505394 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6478003ce0 con 0x7f6488073be0 2024-05-12T22:10:12.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.985+0000 7f648d800640 1 -- 172.21.15.28:0/1864505394 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f648819f470 con 0x7f6488073be0 2024-05-12T22:10:12.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.985+0000 7f648d800640 1 -- 172.21.15.28:0/1864505394 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64881a6d60 con 0x7f6488073be0 2024-05-12T22:10:12.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.985+0000 7f6484c00640 1 -- 172.21.15.28:0/1864505394 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6478035070 con 0x7f6488073be0 2024-05-12T22:10:12.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.989+0000 7f6491c03740 1 Processor -- start 2024-05-12T22:10:12.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.989+0000 7f6491c03740 1 -- start start 2024-05-12T22:10:12.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.990+0000 7f6491c03740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x559cb986bb60 0x559cb986bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:12.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.990+0000 7f6491c03740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559cb986c480 0x559cb9876610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:12.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.990+0000 7f6491c03740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559cb9876b50 0x559cb9878f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:12.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.990+0000 7f6491c03740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x559cb97d0a30 con 0x559cb9876b50 2024-05-12T22:10:12.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.990+0000 7f6491c03740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x559cb9778700 con 0x559cb986bb60 2024-05-12T22:10:12.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.990+0000 7f6491c03740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x559cb97cc9f0 con 0x559cb986c480 2024-05-12T22:10:12.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.990+0000 7f6487400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559cb9876b50 0x559cb9878f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:12.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.990+0000 7f6487400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559cb9876b50 0x559cb9878f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42208/0 (socket says 172.21.15.28:42208) 2024-05-12T22:10:12.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.990+0000 7f6487400640 1 -- 172.21.15.28:0/1606613743 learned_addr learned my addr 172.21.15.28:0/1606613743 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:10:12.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.990+0000 7f648ce00640 1 --2- 172.21.15.28:0/1606613743 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x559cb986bb60 0x559cb986bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:12.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.991+0000 7f6484c00640 1 -- 172.21.15.28:0/1864505394 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138117+0+0 (secure 0 0 0) 0x7f6478042030 con 0x7f6488073be0 2024-05-12T22:10:12.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.991+0000 7f648ce00640 1 -- 172.21.15.28:0/1606613743 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559cb986c480 msgr2=0x559cb9876610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:10:12.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.991+0000 7f648ce00640 1 --2- 172.21.15.28:0/1606613743 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559cb986c480 0x559cb9876610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:12.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.991+0000 7f648ce00640 1 -- 172.21.15.28:0/1606613743 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559cb9876b50 msgr2=0x559cb9878f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:10:12.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.991+0000 7f648ce00640 1 --2- 172.21.15.28:0/1606613743 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559cb9876b50 0x559cb9878f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:12.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.991+0000 7f648ce00640 1 -- 172.21.15.28:0/1606613743 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559cb94e43f0 con 0x559cb986bb60 2024-05-12T22:10:12.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.991+0000 7f6487400640 1 --2- 172.21.15.28:0/1606613743 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559cb9876b50 0x559cb9878f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:10:12.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.991+0000 7f648ce00640 1 --2- 172.21.15.28:0/1606613743 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x559cb986bb60 0x559cb986bf40 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f6480010770 tx=0x7f6480010c30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d0375e40053f6491 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:10:12.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.992+0000 7f647d600640 1 -- 172.21.15.28:0/1606613743 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6480003970 con 0x559cb986bb60 2024-05-12T22:10:12.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.992+0000 7f647d600640 1 -- 172.21.15.28:0/1606613743 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f648000e930 con 0x559cb986bb60 2024-05-12T22:10:12.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.992+0000 7f647d600640 1 -- 172.21.15.28:0/1606613743 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6480004610 con 0x559cb986bb60 2024-05-12T22:10:12.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.993+0000 7f6491c03740 1 -- 172.21.15.28:0/1606613743 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x559cb986bb60 msgr2=0x559cb986bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:10:12.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.993+0000 7f6491c03740 1 --2- 172.21.15.28:0/1606613743 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x559cb986bb60 0x559cb986bf40 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f6480010770 tx=0x7f6480010c30 comp rx=0 tx=0).stop 2024-05-12T22:10:12.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.993+0000 7f6484c00640 1 --2- 172.21.15.28:0/1864505394 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f646c0a2eb0 0x7f646c0a5370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:12.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.993+0000 7f6484c00640 1 -- 172.21.15.28:0/1864505394 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f64780efb30 con 0x7f6488073be0 2024-05-12T22:10:12.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.993+0000 7f648ce00640 1 --2- 172.21.15.28:0/1864505394 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f646c0a2eb0 0x7f646c0a5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:12.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.994+0000 7f6491c03740 1 -- 172.21.15.28:0/1606613743 shutdown_connections 2024-05-12T22:10:12.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.994+0000 7f6491c03740 1 --2- 172.21.15.28:0/1606613743 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559cb9876b50 0x559cb9878f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:12.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.994+0000 7f6491c03740 1 --2- 172.21.15.28:0/1606613743 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559cb986c480 0x559cb9876610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:12.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.994+0000 7f6491c03740 1 --2- 172.21.15.28:0/1606613743 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x559cb986bb60 0x559cb986bf40 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:12.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.994+0000 7f6491c03740 1 -- 172.21.15.28:0/1606613743 >> 172.21.15.28:0/1606613743 conn(0x559cb9867e80 msgr2=0x559cb9868260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:10:12.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.994+0000 7f6491c03740 1 -- 172.21.15.28:0/1606613743 shutdown_connections 2024-05-12T22:10:12.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.994+0000 7f6491c03740 1 -- 172.21.15.28:0/1606613743 wait complete. 2024-05-12T22:10:12.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.994+0000 7f648ce00640 1 --2- 172.21.15.28:0/1864505394 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f646c0a2eb0 0x7f646c0a5370 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f64800042b0 tx=0x7f6480009920 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:10:12.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.995+0000 7f6491c03740 1 -- start start 2024-05-12T22:10:12.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.995+0000 7f6491c03740 1 Processor -- start 2024-05-12T22:10:12.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.996+0000 7f6491c03740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x559cb986bb60 0x559cb9873b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:12.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.996+0000 7f6491c03740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559cb986c480 0x559cb9874080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:12.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.996+0000 7f6491c03740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559cb9876b50 0x559cb98745c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:12.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.996+0000 7f6491c03740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x559cb9879940 con 0x559cb9876b50 2024-05-12T22:10:12.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.996+0000 7f6491c03740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x559cb978ba70 con 0x559cb986bb60 2024-05-12T22:10:12.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.996+0000 7f6491c03740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x559cb9879ac0 con 0x559cb986c480 2024-05-12T22:10:12.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.996+0000 7f6491c03740 10 client.0 _ll_register_callbacks cb 0x559cb976eba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:10:12.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.996+0000 7f6487e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x559cb986bb60 0x559cb9873b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:12.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.996+0000 7f6487400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559cb986c480 0x559cb9874080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:12.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.996+0000 7f6487e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x559cb986bb60 0x559cb9873b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:54566/0 (socket says 172.21.15.28:54566) 2024-05-12T22:10:12.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.996+0000 7f6487e00640 1 -- 172.21.15.28:0/3459219845 learned_addr learned my addr 172.21.15.28:0/3459219845 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:10:12.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.996+0000 7f648ce00640 1 --2- 172.21.15.28:0/3459219845 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559cb9876b50 0x559cb98745c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:12.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.997+0000 7f6487400640 1 -- 172.21.15.28:0/3459219845 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x559cb986bb60 msgr2=0x559cb9873b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:10:12.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.997+0000 7f6487400640 1 --2- 172.21.15.28:0/3459219845 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x559cb986bb60 0x559cb9873b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:12.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.997+0000 7f6487400640 1 -- 172.21.15.28:0/3459219845 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559cb9876b50 msgr2=0x559cb98745c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:10:12.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.997+0000 7f6487400640 1 --2- 172.21.15.28:0/3459219845 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559cb9876b50 0x559cb98745c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:12.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.997+0000 7f6487400640 1 -- 172.21.15.28:0/3459219845 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559cb95ae7d0 con 0x559cb986c480 2024-05-12T22:10:12.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.997+0000 7f6487e00640 1 --2- 172.21.15.28:0/3459219845 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x559cb986bb60 0x559cb9873b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:10:12.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.997+0000 7f6487400640 1 --2- 172.21.15.28:0/3459219845 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559cb986c480 0x559cb9874080 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f64740126d0 tx=0x7f6474012b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e5957d83dc745024 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:10:12.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.997+0000 7f6472000640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-12T22:10:12.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.997+0000 7f6472000640 1 -- 172.21.15.28:0/3459219845 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6474004030 con 0x559cb986c480 2024-05-12T22:10:12.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.998+0000 7f6472000640 1 -- 172.21.15.28:0/3459219845 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f64740041d0 con 0x559cb986c480 2024-05-12T22:10:12.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.998+0000 7f6491c03740 1 -- 172.21.15.28:0/3459219845 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f648000e930 con 0x559cb986c480 2024-05-12T22:10:12.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.998+0000 7f6472000640 1 -- 172.21.15.28:0/3459219845 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f64740043d0 con 0x559cb986c480 2024-05-12T22:10:13.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.998+0000 7f6491c03740 10 client.25570 fetch_fsmap learned FSMap version 24 2024-05-12T22:10:13.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.998+0000 7f6491c03740 1 -- 172.21.15.28:0/3459219845 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x559cb9885ca0 con 0x559cb986c480 2024-05-12T22:10:13.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.998+0000 7f6472000640 1 -- 172.21.15.28:0/3459219845 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f648000e930 con 0x559cb986c480 2024-05-12T22:10:13.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.999+0000 7f6472000640 1 -- 172.21.15.28:0/3459219845 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f6474003c20 con 0x559cb986c480 2024-05-12T22:10:13.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.999+0000 7f6491c03740 10 client.25570 fetch_fsmap finished waiting for FSMap version 24 2024-05-12T22:10:13.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.999+0000 7f6491c03740 10 client.25570 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.g 2024-05-12T22:10:13.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.999+0000 7f6491c03740 20 client.25570 populate_metadata read hostname 'smithi028' 2024-05-12T22:10:13.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.999+0000 7f6491c03740 1 --2- 172.21.15.28:0/3459219845 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x559cb9887150 0x559cb9889570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:13.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.999+0000 7f6491c03740 4 client.25570 mds_command: new command op to 24532 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-12T22:10:13.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.999+0000 7f6491c03740 1 -- 172.21.15.28:0/3459219845 --> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559cb97de160 con 0x559cb9887150 2024-05-12T22:10:13.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:12.999+0000 7f6487e00640 1 --2- 172.21.15.28:0/3459219845 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x559cb9887150 0x559cb9889570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:13.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.000+0000 7f6487e00640 1 --2- 172.21.15.28:0/3459219845 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x559cb9887150 0x559cb9889570 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=699063306680320c server_cookie=b73559c2425d42d7 in_seq=0 out_seq=0 2024-05-12T22:10:13.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.000+0000 7f6472000640 10 client.25570 ms_handle_connect on v2:172.21.15.28:6838/123892158 2024-05-12T22:10:13.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.000+0000 7f6472000640 1 -- 172.21.15.28:0/3459219845 <== mds.2 v2:172.21.15.28:6838/123892158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559cb97de160 con 0x559cb9887150 2024-05-12T22:10:13.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.000+0000 7f6472000640 10 client.25570 handle_command_reply: tid=0 2024-05-12T22:10:13.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.001+0000 7f6491c03740 1 client.25570 shutdown 2024-05-12T22:10:13.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.001+0000 7f6491c03740 1 -- 172.21.15.28:0/3459219845 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559cb986c480 msgr2=0x559cb9874080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:10:13.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.001+0000 7f6491c03740 1 --2- 172.21.15.28:0/3459219845 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559cb986c480 0x559cb9874080 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f64740126d0 tx=0x7f6474012b90 comp rx=0 tx=0).stop 2024-05-12T22:10:13.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.001+0000 7f6491c03740 1 -- 172.21.15.28:0/3459219845 shutdown_connections 2024-05-12T22:10:13.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.001+0000 7f6491c03740 1 --2- 172.21.15.28:0/3459219845 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559cb9876b50 0x559cb98745c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:13.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.001+0000 7f6491c03740 1 --2- 172.21.15.28:0/3459219845 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559cb986c480 0x559cb9874080 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:13.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.001+0000 7f6491c03740 1 --2- 172.21.15.28:0/3459219845 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x559cb9887150 0x559cb9889570 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:13.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.001+0000 7f6491c03740 1 --2- 172.21.15.28:0/3459219845 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x559cb986bb60 0x559cb9873b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:13.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.002+0000 7f6491c03740 1 -- 172.21.15.28:0/3459219845 >> 172.21.15.28:0/3459219845 conn(0x559cb9867e80 msgr2=0x559cb986dc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:10:13.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.002+0000 7f6472000640 0 client.25570 ms_handle_reset on v2:172.21.15.28:6838/123892158 2024-05-12T22:10:13.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.002+0000 7f6491c03740 1 -- 172.21.15.28:0/3459219845 shutdown_connections 2024-05-12T22:10:13.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.002+0000 7f6491c03740 1 -- 172.21.15.28:0/3459219845 wait complete. 2024-05-12T22:10:13.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.002+0000 7f6491c03740 20 client.25570 trim_cache size 0 max 16384 2024-05-12T22:10:13.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.030+0000 7f6491c03740 1 Processor -- start 2024-05-12T22:10:13.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.030+0000 7f6491c03740 1 -- start start 2024-05-12T22:10:13.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.030+0000 7f6491c03740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559cb986bb60 0x559cb9882800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:13.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.030+0000 7f6491c03740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x559cb986c480 0x559cb9867780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:13.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.030+0000 7f6491c03740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559cb9876b50 0x559cb988ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:13.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.030+0000 7f6491c03740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f64881b1e20 con 0x559cb9876b50 2024-05-12T22:10:13.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.030+0000 7f6491c03740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f64881af6f0 con 0x559cb986c480 2024-05-12T22:10:13.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.030+0000 7f6491c03740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f64881ad540 con 0x559cb986bb60 2024-05-12T22:10:13.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.031+0000 7f6487400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x559cb986c480 0x559cb9867780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:13.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.031+0000 7f6487400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x559cb986c480 0x559cb9867780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:54578/0 (socket says 172.21.15.28:54578) 2024-05-12T22:10:13.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.031+0000 7f6487400640 1 -- 172.21.15.28:0/1078587869 learned_addr learned my addr 172.21.15.28:0/1078587869 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:10:13.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.031+0000 7f6487e00640 1 --2- 172.21.15.28:0/1078587869 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559cb986bb60 0x559cb9882800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:13.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.031+0000 7f648ce00640 1 --2- 172.21.15.28:0/1078587869 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559cb9876b50 0x559cb988ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:13.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.031+0000 7f6487400640 1 -- 172.21.15.28:0/1078587869 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559cb986bb60 msgr2=0x559cb9882800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:10:13.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.031+0000 7f6487400640 1 --2- 172.21.15.28:0/1078587869 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559cb986bb60 0x559cb9882800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:13.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.031+0000 7f6487400640 1 -- 172.21.15.28:0/1078587869 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559cb9876b50 msgr2=0x559cb988ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:10:13.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.031+0000 7f6487400640 1 --2- 172.21.15.28:0/1078587869 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559cb9876b50 0x559cb988ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:13.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.031+0000 7f6487400640 1 -- 172.21.15.28:0/1078587869 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559cb98623e0 con 0x559cb986c480 2024-05-12T22:10:13.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.031+0000 7f648ce00640 1 --2- 172.21.15.28:0/1078587869 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559cb9876b50 0x559cb988ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-12T22:10:13.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.031+0000 7f6487400640 1 --2- 172.21.15.28:0/1078587869 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x559cb986c480 0x559cb9867780 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f6474000c00 tx=0x7f6474005d90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=492bbbe707ccd76d server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:10:13.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.032+0000 7f647cc00640 1 -- 172.21.15.28:0/1078587869 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6474025660 con 0x559cb986c480 2024-05-12T22:10:13.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.032+0000 7f647cc00640 1 -- 172.21.15.28:0/1078587869 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6474025800 con 0x559cb986c480 2024-05-12T22:10:13.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.032+0000 7f647cc00640 1 -- 172.21.15.28:0/1078587869 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f647400fce0 con 0x559cb986c480 2024-05-12T22:10:13.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.033+0000 7f6491c03740 1 -- 172.21.15.28:0/1078587869 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x559cb986c480 msgr2=0x559cb9867780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:10:13.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.033+0000 7f6491c03740 1 --2- 172.21.15.28:0/1078587869 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x559cb986c480 0x559cb9867780 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f6474000c00 tx=0x7f6474005d90 comp rx=0 tx=0).stop 2024-05-12T22:10:13.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.033+0000 7f6491c03740 1 -- 172.21.15.28:0/1078587869 shutdown_connections 2024-05-12T22:10:13.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.033+0000 7f6491c03740 1 --2- 172.21.15.28:0/1078587869 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559cb9876b50 0x559cb988ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:13.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.033+0000 7f6491c03740 1 --2- 172.21.15.28:0/1078587869 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x559cb986c480 0x559cb9867780 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:13.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.033+0000 7f6491c03740 1 --2- 172.21.15.28:0/1078587869 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559cb986bb60 0x559cb9882800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:13.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.033+0000 7f6491c03740 1 -- 172.21.15.28:0/1078587869 >> 172.21.15.28:0/1078587869 conn(0x559cb9867e80 msgr2=0x559cb9874390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:10:13.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.033+0000 7f6491c03740 1 -- 172.21.15.28:0/1078587869 shutdown_connections 2024-05-12T22:10:13.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.033+0000 7f6491c03740 1 -- 172.21.15.28:0/1078587869 wait complete. 2024-05-12T22:10:13.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.033+0000 7f6491c03740 1 -- start start 2024-05-12T22:10:13.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.034+0000 7f6491c03740 1 Processor -- start 2024-05-12T22:10:13.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.034+0000 7f6491c03740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559cb986bb60 0x559cb9870a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:13.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.034+0000 7f6491c03740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x559cb986c480 0x559cb9870f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:13.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.034+0000 7f6491c03740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559cb9876b50 0x559cb9871490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:13.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.034+0000 7f6491c03740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f64881adbb0 con 0x559cb9876b50 2024-05-12T22:10:13.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.034+0000 7f6491c03740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f64881af020 con 0x559cb986c480 2024-05-12T22:10:13.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.034+0000 7f6491c03740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f64881ae280 con 0x559cb986bb60 2024-05-12T22:10:13.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.035+0000 7f6491c03740 10 client.0 _ll_register_callbacks cb 0x7f6474009e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:10:13.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.035+0000 7f648ce00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559cb9876b50 0x559cb9871490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:13.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.035+0000 7f648ce00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559cb9876b50 0x559cb9871490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42230/0 (socket says 172.21.15.28:42230) 2024-05-12T22:10:13.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.035+0000 7f648ce00640 1 -- 172.21.15.28:0/3683677640 learned_addr learned my addr 172.21.15.28:0/3683677640 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:10:13.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.035+0000 7f6487e00640 1 --2- 172.21.15.28:0/3683677640 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559cb986bb60 0x559cb9870a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:13.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.035+0000 7f6487400640 1 --2- 172.21.15.28:0/3683677640 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x559cb986c480 0x559cb9870f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:13.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.035+0000 7f6487400640 1 -- 172.21.15.28:0/3683677640 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559cb986bb60 msgr2=0x559cb9870a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:10:13.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.035+0000 7f6487400640 1 --2- 172.21.15.28:0/3683677640 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559cb986bb60 0x559cb9870a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:13.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.035+0000 7f6487400640 1 -- 172.21.15.28:0/3683677640 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559cb9876b50 msgr2=0x559cb9871490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:10:13.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.036+0000 7f6487400640 1 --2- 172.21.15.28:0/3683677640 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559cb9876b50 0x559cb9871490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:13.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.036+0000 7f6487400640 1 -- 172.21.15.28:0/3683677640 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559cb98ec890 con 0x559cb986c480 2024-05-12T22:10:13.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.036+0000 7f6487e00640 1 --2- 172.21.15.28:0/3683677640 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559cb986bb60 0x559cb9870a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:10:13.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.036+0000 7f648ce00640 1 --2- 172.21.15.28:0/3683677640 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559cb9876b50 0x559cb9871490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:10:13.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.036+0000 7f6487400640 1 --2- 172.21.15.28:0/3683677640 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x559cb986c480 0x559cb9870f50 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x559cb986e8f0 tx=0x7f647400f810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b218f25dbf3eed0c server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:10:13.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.036+0000 7f6472000640 10 client.0 ms_handle_connect on v2:172.21.15.86:3300/0 2024-05-12T22:10:13.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.036+0000 7f6472000640 1 -- 172.21.15.28:0/3683677640 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6474025cf0 con 0x559cb986c480 2024-05-12T22:10:13.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.036+0000 7f6472000640 1 -- 172.21.15.28:0/3683677640 <== mon.2 v2:172.21.15.86:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f6474018070 con 0x559cb986c480 2024-05-12T22:10:13.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.037+0000 7f6491c03740 1 -- 172.21.15.28:0/3683677640 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6474025800 con 0x559cb986c480 2024-05-12T22:10:13.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.037+0000 7f6472000640 1 -- 172.21.15.28:0/3683677640 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f647400fa20 con 0x559cb986c480 2024-05-12T22:10:13.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.037+0000 7f6491c03740 10 client.25613 fetch_fsmap learned FSMap version 24 2024-05-12T22:10:13.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.037+0000 7f6491c03740 1 -- 172.21.15.28:0/3683677640 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x559cb98edae0 con 0x559cb986c480 2024-05-12T22:10:13.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.038+0000 7f6472000640 1 -- 172.21.15.28:0/3683677640 <== mon.2 v2:172.21.15.86:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6474025800 con 0x559cb986c480 2024-05-12T22:10:13.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.038+0000 7f6472000640 1 -- 172.21.15.28:0/3683677640 <== mon.2 v2:172.21.15.86:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f6474010c40 con 0x559cb986c480 2024-05-12T22:10:13.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.038+0000 7f6491c03740 10 client.25613 fetch_fsmap finished waiting for FSMap version 24 2024-05-12T22:10:13.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.038+0000 7f6491c03740 10 client.25613 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.g 2024-05-12T22:10:13.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.038+0000 7f6491c03740 20 client.25613 populate_metadata read hostname 'smithi028' 2024-05-12T22:10:13.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.038+0000 7f6491c03740 1 --2- 172.21.15.28:0/3683677640 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x559cb98ee090 0x559cb98f0480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:13.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.038+0000 7f6491c03740 4 client.25613 mds_command: new command op to 24532 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-12T22:10:13.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.038+0000 7f6491c03740 1 -- 172.21.15.28:0/3683677640 --> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559cb97d84b0 con 0x559cb98ee090 2024-05-12T22:10:13.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.038+0000 7f6487e00640 1 --2- 172.21.15.28:0/3683677640 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x559cb98ee090 0x559cb98f0480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:13.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.039+0000 7f6487e00640 1 --2- 172.21.15.28:0/3683677640 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x559cb98ee090 0x559cb98f0480 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=17431f9c1294509b server_cookie=8bdee0407b302f29 in_seq=0 out_seq=0 2024-05-12T22:10:13.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.039+0000 7f6472000640 10 client.25613 ms_handle_connect on v2:172.21.15.28:6838/123892158 2024-05-12T22:10:13.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.039+0000 7f6472000640 1 -- 172.21.15.28:0/3683677640 <== mds.2 v2:172.21.15.28:6838/123892158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x559cb97d84b0 con 0x559cb98ee090 2024-05-12T22:10:13.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.040+0000 7f6472000640 10 client.25613 handle_command_reply: tid=0 2024-05-12T22:10:13.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.040+0000 7f6491c03740 1 client.25613 shutdown 2024-05-12T22:10:13.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.040+0000 7f6491c03740 1 -- 172.21.15.28:0/3683677640 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x559cb986c480 msgr2=0x559cb9870f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:10:13.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.040+0000 7f6491c03740 1 --2- 172.21.15.28:0/3683677640 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x559cb986c480 0x559cb9870f50 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x559cb986e8f0 tx=0x7f647400f810 comp rx=0 tx=0).stop 2024-05-12T22:10:13.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.040+0000 7f6491c03740 1 -- 172.21.15.28:0/3683677640 shutdown_connections 2024-05-12T22:10:13.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.040+0000 7f6491c03740 1 --2- 172.21.15.28:0/3683677640 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559cb9876b50 0x559cb9871490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:13.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.040+0000 7f6491c03740 1 --2- 172.21.15.28:0/3683677640 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x559cb98ee090 0x559cb98f0480 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:13.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.040+0000 7f6491c03740 1 --2- 172.21.15.28:0/3683677640 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x559cb986c480 0x559cb9870f50 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:13.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.040+0000 7f6491c03740 1 --2- 172.21.15.28:0/3683677640 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559cb986bb60 0x559cb9870a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:13.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.040+0000 7f6472000640 0 client.25613 ms_handle_reset on v2:172.21.15.28:6838/123892158 2024-05-12T22:10:13.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.040+0000 7f6491c03740 1 -- 172.21.15.28:0/3683677640 >> 172.21.15.28:0/3683677640 conn(0x559cb9867e80 msgr2=0x559cb9873b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:10:13.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.040+0000 7f6491c03740 1 -- 172.21.15.28:0/3683677640 shutdown_connections 2024-05-12T22:10:13.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.040+0000 7f6491c03740 1 -- 172.21.15.28:0/3683677640 wait complete. 2024-05-12T22:10:13.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.040+0000 7f6491c03740 20 client.25613 trim_cache size 0 max 16384 2024-05-12T22:10:13.045 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-05-12T22:10:13.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.041+0000 7f648d800640 1 -- 172.21.15.28:0/1864505394 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f646c0a2eb0 msgr2=0x7f646c0a5370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:10:13.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.041+0000 7f648d800640 1 --2- 172.21.15.28:0/1864505394 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f646c0a2eb0 0x7f646c0a5370 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f64800042b0 tx=0x7f6480009920 comp rx=0 tx=0).stop 2024-05-12T22:10:13.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.041+0000 7f648d800640 1 -- 172.21.15.28:0/1864505394 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6488073be0 msgr2=0x7f648819a1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:10:13.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.041+0000 7f648d800640 1 --2- 172.21.15.28:0/1864505394 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6488073be0 0x7f648819a1f0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f6478006010 tx=0x7f64780042f0 comp rx=0 tx=0).stop 2024-05-12T22:10:13.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.042+0000 7f648d800640 1 -- 172.21.15.28:0/1864505394 shutdown_connections 2024-05-12T22:10:13.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.042+0000 7f648d800640 1 --2- 172.21.15.28:0/1864505394 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f646c0a2eb0 0x7f646c0a5370 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:13.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.042+0000 7f648d800640 1 --2- 172.21.15.28:0/1864505394 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f648810df90 0x7f648819eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:13.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.042+0000 7f648d800640 1 --2- 172.21.15.28:0/1864505394 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6488074500 0x7f648819a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:13.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.042+0000 7f648d800640 1 --2- 172.21.15.28:0/1864505394 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6488073be0 0x7f648819a1f0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:13.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.042+0000 7f648d800640 1 -- 172.21.15.28:0/1864505394 >> 172.21.15.28:0/1864505394 conn(0x7f64880fd200 msgr2=0x7f64880fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:10:13.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.042+0000 7f648d800640 1 -- 172.21.15.28:0/1864505394 shutdown_connections 2024-05-12T22:10:13.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.042+0000 7f648d800640 1 -- 172.21.15.28:0/1864505394 wait complete. 2024-05-12T22:10:13.066 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-05-12T22:10:13.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.248+0000 7f89ae200640 1 -- 172.21.15.28:0/4085647685 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f89a8107810 msgr2=0x7f89a8107bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:10:13.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.248+0000 7f89ae200640 1 --2- 172.21.15.28:0/4085647685 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f89a8107810 0x7f89a8107bf0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f8998009860 tx=0x7f899802a340 comp rx=0 tx=0).stop 2024-05-12T22:10:13.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.248+0000 7f89ae200640 1 -- 172.21.15.28:0/4085647685 shutdown_connections 2024-05-12T22:10:13.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.248+0000 7f89ae200640 1 --2- 172.21.15.28:0/4085647685 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89a806a7f0 0x7f89a810e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:13.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.248+0000 7f89ae200640 1 --2- 172.21.15.28:0/4085647685 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f89a8069e30 0x7f89a806a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:13.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.248+0000 7f89ae200640 1 --2- 172.21.15.28:0/4085647685 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f89a8107810 0x7f89a8107bf0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:13.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.248+0000 7f89ae200640 1 -- 172.21.15.28:0/4085647685 >> 172.21.15.28:0/4085647685 conn(0x7f89a8076530 msgr2=0x7f89a8076940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:10:13.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.248+0000 7f89ae200640 1 -- 172.21.15.28:0/4085647685 shutdown_connections 2024-05-12T22:10:13.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.248+0000 7f89ae200640 1 -- 172.21.15.28:0/4085647685 wait complete. 2024-05-12T22:10:13.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.249+0000 7f89ae200640 1 Processor -- start 2024-05-12T22:10:13.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.249+0000 7f89ae200640 1 -- start start 2024-05-12T22:10:13.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.249+0000 7f89ae200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f89a8069e30 0x7f89a8100b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:13.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.249+0000 7f89ae200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89a806a7f0 0x7f89a8101090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:13.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.249+0000 7f89ae200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f89a8107810 0x7f89a81015d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:13.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.249+0000 7f89ae200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f89a8111bc0 con 0x7f89a806a7f0 2024-05-12T22:10:13.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.249+0000 7f89ae200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f89a8111a40 con 0x7f89a8069e30 2024-05-12T22:10:13.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.249+0000 7f89ae200640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f89a8111d40 con 0x7f89a8107810 2024-05-12T22:10:13.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.250+0000 7f89a6a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89a806a7f0 0x7f89a8101090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:13.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.250+0000 7f89a7400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f89a8069e30 0x7f89a8100b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:13.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.250+0000 7f89a7400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f89a8069e30 0x7f89a8100b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:54614/0 (socket says 172.21.15.28:54614) 2024-05-12T22:10:13.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.250+0000 7f89a7400640 1 -- 172.21.15.28:0/237467389 learned_addr learned my addr 172.21.15.28:0/237467389 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:10:13.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.250+0000 7f89a7e00640 1 --2- 172.21.15.28:0/237467389 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f89a8107810 0x7f89a81015d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:13.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.250+0000 7f89a6a00640 1 -- 172.21.15.28:0/237467389 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f89a8107810 msgr2=0x7f89a81015d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:10:13.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.250+0000 7f89a6a00640 1 --2- 172.21.15.28:0/237467389 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f89a8107810 0x7f89a81015d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:13.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.250+0000 7f89a6a00640 1 -- 172.21.15.28:0/237467389 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f89a8069e30 msgr2=0x7f89a8100b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:10:13.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.250+0000 7f89a6a00640 1 --2- 172.21.15.28:0/237467389 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f89a8069e30 0x7f89a8100b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:13.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.250+0000 7f89a6a00640 1 -- 172.21.15.28:0/237467389 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89a81a65c0 con 0x7f89a806a7f0 2024-05-12T22:10:13.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.250+0000 7f89a6a00640 1 --2- 172.21.15.28:0/237467389 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89a806a7f0 0x7f89a8101090 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f8994005fd0 tx=0x7f899400ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:10:13.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.251+0000 7f899fe00640 1 -- 172.21.15.28:0/237467389 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f899400cd00 con 0x7f89a806a7f0 2024-05-12T22:10:13.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.251+0000 7f899fe00640 1 -- 172.21.15.28:0/237467389 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89940035f0 con 0x7f89a806a7f0 2024-05-12T22:10:13.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.251+0000 7f899fe00640 1 -- 172.21.15.28:0/237467389 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8994010670 con 0x7f89a806a7f0 2024-05-12T22:10:13.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.251+0000 7f89ae200640 1 -- 172.21.15.28:0/237467389 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89a81a68b0 con 0x7f89a806a7f0 2024-05-12T22:10:13.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.251+0000 7f89ae200640 1 -- 172.21.15.28:0/237467389 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89a81a6db0 con 0x7f89a806a7f0 2024-05-12T22:10:13.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.255+0000 7f89b0fd9740 1 Processor -- start 2024-05-12T22:10:13.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.255+0000 7f89b0fd9740 1 -- start start 2024-05-12T22:10:13.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.256+0000 7f89b0fd9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5569bdc43980 0x5569bdc43d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:13.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.256+0000 7f89b0fd9740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5569bdc442a0 0x5569bdc4e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:13.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.256+0000 7f89b0fd9740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5569bdc4e970 0x5569bdc50d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:13.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.256+0000 7f89b0fd9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5569bdba8a30 con 0x5569bdc43980 2024-05-12T22:10:13.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.256+0000 7f89b0fd9740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x5569bdb50700 con 0x5569bdc4e970 2024-05-12T22:10:13.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.256+0000 7f89b0fd9740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5569bdba49f0 con 0x5569bdc442a0 2024-05-12T22:10:13.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.256+0000 7f89a7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5569bdc43980 0x5569bdc43d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:13.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.256+0000 7f89a7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5569bdc43980 0x5569bdc43d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42266/0 (socket says 172.21.15.28:42266) 2024-05-12T22:10:13.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.256+0000 7f89a7e00640 1 -- 172.21.15.28:0/3029664691 learned_addr learned my addr 172.21.15.28:0/3029664691 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:10:13.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.256+0000 7f89a7400640 1 --2- 172.21.15.28:0/3029664691 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5569bdc442a0 0x5569bdc4e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:13.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.257+0000 7f89a7e00640 1 -- 172.21.15.28:0/3029664691 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5569bdc442a0 msgr2=0x5569bdc4e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:10:13.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.257+0000 7f89a7e00640 1 --2- 172.21.15.28:0/3029664691 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5569bdc442a0 0x5569bdc4e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:13.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.257+0000 7f89a7e00640 1 -- 172.21.15.28:0/3029664691 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5569bdc4e970 msgr2=0x5569bdc50d60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-12T22:10:13.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.257+0000 7f89a7e00640 1 --2- 172.21.15.28:0/3029664691 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5569bdc4e970 0x5569bdc50d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:13.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.257+0000 7f89a7e00640 1 -- 172.21.15.28:0/3029664691 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5569bd8bc3f0 con 0x5569bdc43980 2024-05-12T22:10:13.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.257+0000 7f89a7e00640 1 --2- 172.21.15.28:0/3029664691 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5569bdc43980 0x5569bdc43d60 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f89a0009f50 tx=0x7f89a00118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7e70a1162f2a763 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:10:13.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.257+0000 7f8993e00640 1 -- 172.21.15.28:0/3029664691 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89a0004d20 con 0x5569bdc43980 2024-05-12T22:10:13.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.257+0000 7f8993e00640 1 -- 172.21.15.28:0/3029664691 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89a0004050 con 0x5569bdc43980 2024-05-12T22:10:13.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.257+0000 7f8993e00640 1 -- 172.21.15.28:0/3029664691 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89a0005280 con 0x5569bdc43980 2024-05-12T22:10:13.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.258+0000 7f89b0fd9740 1 -- 172.21.15.28:0/3029664691 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5569bdc43980 msgr2=0x5569bdc43d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:10:13.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.258+0000 7f89b0fd9740 1 --2- 172.21.15.28:0/3029664691 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5569bdc43980 0x5569bdc43d60 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f89a0009f50 tx=0x7f89a00118d0 comp rx=0 tx=0).stop 2024-05-12T22:10:13.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.258+0000 7f89b0fd9740 1 -- 172.21.15.28:0/3029664691 shutdown_connections 2024-05-12T22:10:13.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.258+0000 7f89b0fd9740 1 --2- 172.21.15.28:0/3029664691 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5569bdc4e970 0x5569bdc50d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:13.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.258+0000 7f89b0fd9740 1 --2- 172.21.15.28:0/3029664691 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5569bdc442a0 0x5569bdc4e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:13.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.258+0000 7f89b0fd9740 1 --2- 172.21.15.28:0/3029664691 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5569bdc43980 0x5569bdc43d60 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:13.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.259+0000 7f89b0fd9740 1 -- 172.21.15.28:0/3029664691 >> 172.21.15.28:0/3029664691 conn(0x5569bdc3fca0 msgr2=0x5569bdc40080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:10:13.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.259+0000 7f899fe00640 1 -- 172.21.15.28:0/237467389 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138117+0+0 (secure 0 0 0) 0x7f89940037a0 con 0x7f89a806a7f0 2024-05-12T22:10:13.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.259+0000 7f89b0fd9740 1 -- 172.21.15.28:0/3029664691 shutdown_connections 2024-05-12T22:10:13.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.259+0000 7f89b0fd9740 1 -- 172.21.15.28:0/3029664691 wait complete. 2024-05-12T22:10:13.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.260+0000 7f89b0fd9740 1 -- start start 2024-05-12T22:10:13.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.260+0000 7f89b0fd9740 1 Processor -- start 2024-05-12T22:10:13.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.261+0000 7f899fe00640 1 --2- 172.21.15.28:0/237467389 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f898c0a2f40 0x7f898c0a5400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:13.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.261+0000 7f899fe00640 1 -- 172.21.15.28:0/237467389 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f89940cf790 con 0x7f89a806a7f0 2024-05-12T22:10:13.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.261+0000 7f89b0fd9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5569bdc43980 0x5569bdc4ba90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:13.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.261+0000 7f89b0fd9740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5569bdc442a0 0x5569bdc4bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:13.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.261+0000 7f89b0fd9740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5569bdc4c510 0x5569bdc633e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:13.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.261+0000 7f89b0fd9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5569bdc51760 con 0x5569bdc43980 2024-05-12T22:10:13.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.261+0000 7f89b0fd9740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x5569bdb63a70 con 0x5569bdc442a0 2024-05-12T22:10:13.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.261+0000 7f89b0fd9740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5569bdc518e0 con 0x5569bdc4c510 2024-05-12T22:10:13.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.261+0000 7f89b0fd9740 10 client.0 _ll_register_callbacks cb 0x5569bdb46ba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:10:13.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.261+0000 7f89a7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5569bdc43980 0x5569bdc4ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:13.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.261+0000 7f89a6a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5569bdc442a0 0x5569bdc4bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:13.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.261+0000 7f89a7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5569bdc43980 0x5569bdc4ba90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42282/0 (socket says 172.21.15.28:42282) 2024-05-12T22:10:13.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.261+0000 7f89a7400640 1 -- 172.21.15.28:0/737965494 learned_addr learned my addr 172.21.15.28:0/737965494 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:10:13.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.262+0000 7f89a7e00640 1 --2- 172.21.15.28:0/237467389 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f898c0a2f40 0x7f898c0a5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:13.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.262+0000 7f89a7400640 1 -- 172.21.15.28:0/737965494 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5569bdc4c510 msgr2=0x5569bdc633e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-12T22:10:13.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.262+0000 7f89a7400640 1 --2- 172.21.15.28:0/737965494 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5569bdc4c510 0x5569bdc633e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:13.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.262+0000 7f89a7400640 1 -- 172.21.15.28:0/737965494 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5569bdc442a0 msgr2=0x5569bdc4bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:10:13.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.262+0000 7f89a7400640 1 --2- 172.21.15.28:0/737965494 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5569bdc442a0 0x5569bdc4bfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:13.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.262+0000 7f89a7400640 1 -- 172.21.15.28:0/737965494 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5569bd9867d0 con 0x5569bdc43980 2024-05-12T22:10:13.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.262+0000 7f89a6a00640 1 --2- 172.21.15.28:0/737965494 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5569bdc442a0 0x5569bdc4bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:10:13.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.262+0000 7f89a7400640 1 --2- 172.21.15.28:0/737965494 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5569bdc43980 0x5569bdc4ba90 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f89980057c0 tx=0x7f89980303d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b7bd6705089f90d server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:10:13.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.262+0000 7f8990c00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-12T22:10:13.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.263+0000 7f8990c00640 1 -- 172.21.15.28:0/737965494 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8998037040 con 0x5569bdc43980 2024-05-12T22:10:13.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.263+0000 7f8990c00640 1 -- 172.21.15.28:0/737965494 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f8998004040 con 0x5569bdc43980 2024-05-12T22:10:13.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.263+0000 7f89b0fd9740 1 -- 172.21.15.28:0/737965494 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f89a0004050 con 0x5569bdc43980 2024-05-12T22:10:13.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.263+0000 7f89a7e00640 1 --2- 172.21.15.28:0/237467389 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f898c0a2f40 0x7f898c0a5400 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f89a0004950 tx=0x7f89a0005fd0 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:10:13.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.263+0000 7f8990c00640 1 -- 172.21.15.28:0/737965494 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f899804c050 con 0x5569bdc43980 2024-05-12T22:10:13.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.263+0000 7f89b0fd9740 10 client.15504 fetch_fsmap learned FSMap version 24 2024-05-12T22:10:13.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.263+0000 7f89b0fd9740 1 -- 172.21.15.28:0/737965494 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5569bdc66780 con 0x5569bdc43980 2024-05-12T22:10:13.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.264+0000 7f8990c00640 1 -- 172.21.15.28:0/737965494 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89a0004050 con 0x5569bdc43980 2024-05-12T22:10:13.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.264+0000 7f8990c00640 1 -- 172.21.15.28:0/737965494 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f899803d450 con 0x5569bdc43980 2024-05-12T22:10:13.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.264+0000 7f89b0fd9740 10 client.15504 fetch_fsmap finished waiting for FSMap version 24 2024-05-12T22:10:13.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.264+0000 7f89b0fd9740 10 client.15504 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-05-12T22:10:13.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.264+0000 7f89b0fd9740 20 client.15504 populate_metadata read hostname 'smithi028' 2024-05-12T22:10:13.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.265+0000 7f89b0fd9740 1 --2- 172.21.15.28:0/737965494 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x5569bdc67c40 0x5569bdc6a070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:13.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.265+0000 7f89b0fd9740 4 client.15504 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-12T22:10:13.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.265+0000 7f89b0fd9740 1 -- 172.21.15.28:0/737965494 --> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5569bdbb6160 con 0x5569bdc67c40 2024-05-12T22:10:13.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.265+0000 7f89a6a00640 1 --2- 172.21.15.28:0/737965494 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x5569bdc67c40 0x5569bdc6a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:13.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.265+0000 7f89a6a00640 1 --2- 172.21.15.28:0/737965494 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x5569bdc67c40 0x5569bdc6a070 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=af71a7e85fc943dc server_cookie=2423135665094139 in_seq=0 out_seq=0 2024-05-12T22:10:13.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.265+0000 7f8990c00640 10 client.15504 ms_handle_connect on v2:172.21.15.28:6840/761183870 2024-05-12T22:10:13.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.266+0000 7f8990c00640 1 -- 172.21.15.28:0/737965494 <== mds.1 v2:172.21.15.28:6840/761183870 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5569bdbb6160 con 0x5569bdc67c40 2024-05-12T22:10:13.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.266+0000 7f8990c00640 10 client.15504 handle_command_reply: tid=0 2024-05-12T22:10:13.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.266+0000 7f89b0fd9740 1 client.15504 shutdown 2024-05-12T22:10:13.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.266+0000 7f89b0fd9740 1 -- 172.21.15.28:0/737965494 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5569bdc43980 msgr2=0x5569bdc4ba90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:10:13.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.266+0000 7f89b0fd9740 1 --2- 172.21.15.28:0/737965494 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5569bdc43980 0x5569bdc4ba90 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f89980057c0 tx=0x7f89980303d0 comp rx=0 tx=0).stop 2024-05-12T22:10:13.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.266+0000 7f89b0fd9740 1 -- 172.21.15.28:0/737965494 shutdown_connections 2024-05-12T22:10:13.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.266+0000 7f89b0fd9740 1 --2- 172.21.15.28:0/737965494 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x5569bdc67c40 0x5569bdc6a070 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:13.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.266+0000 7f89b0fd9740 1 --2- 172.21.15.28:0/737965494 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5569bdc4c510 0x5569bdc633e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:13.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.267+0000 7f89b0fd9740 1 --2- 172.21.15.28:0/737965494 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5569bdc442a0 0x5569bdc4bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:13.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.267+0000 7f89b0fd9740 1 --2- 172.21.15.28:0/737965494 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5569bdc43980 0x5569bdc4ba90 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:13.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.267+0000 7f89b0fd9740 1 -- 172.21.15.28:0/737965494 >> 172.21.15.28:0/737965494 conn(0x5569bdc3fca0 msgr2=0x5569bdc45aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:10:13.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.266+0000 7f8990c00640 0 client.15504 ms_handle_reset on v2:172.21.15.28:6840/761183870 2024-05-12T22:10:13.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.267+0000 7f89b0fd9740 1 -- 172.21.15.28:0/737965494 shutdown_connections 2024-05-12T22:10:13.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.267+0000 7f89b0fd9740 1 -- 172.21.15.28:0/737965494 wait complete. 2024-05-12T22:10:13.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.267+0000 7f89b0fd9740 20 client.15504 trim_cache size 0 max 16384 2024-05-12T22:10:13.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.281+0000 7f89b0fd9740 1 Processor -- start 2024-05-12T22:10:13.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.282+0000 7f89b0fd9740 1 -- start start 2024-05-12T22:10:13.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.282+0000 7f89b0fd9740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5569bdc43980 0x5569bdc6f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:13.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.282+0000 7f89b0fd9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5569bdc442a0 0x5569bdc6fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:13.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.282+0000 7f89b0fd9740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5569bdc4c510 0x5569bdc62eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:13.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.282+0000 7f89b0fd9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f89a81adb40 con 0x5569bdc442a0 2024-05-12T22:10:13.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.282+0000 7f89b0fd9740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f89a81ab410 con 0x5569bdc4c510 2024-05-12T22:10:13.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.282+0000 7f89b0fd9740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f89a81a9260 con 0x5569bdc43980 2024-05-12T22:10:13.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.282+0000 7f89a6a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5569bdc442a0 0x5569bdc6fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:13.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.283+0000 7f89a7e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5569bdc4c510 0x5569bdc62eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:13.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.283+0000 7f89a7400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5569bdc43980 0x5569bdc6f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:13.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.283+0000 7f89a6a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5569bdc442a0 0x5569bdc6fe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42298/0 (socket says 172.21.15.28:42298) 2024-05-12T22:10:13.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.283+0000 7f89a6a00640 1 -- 172.21.15.28:0/2232991447 learned_addr learned my addr 172.21.15.28:0/2232991447 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:10:13.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.283+0000 7f89a7e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5569bdc4c510 0x5569bdc62eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:54632/0 (socket says 172.21.15.28:54632) 2024-05-12T22:10:13.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.283+0000 7f89a6a00640 1 -- 172.21.15.28:0/2232991447 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5569bdc43980 msgr2=0x5569bdc6f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:10:13.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.283+0000 7f89a6a00640 1 --2- 172.21.15.28:0/2232991447 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5569bdc43980 0x5569bdc6f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:13.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.283+0000 7f89a6a00640 1 -- 172.21.15.28:0/2232991447 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5569bdc4c510 msgr2=0x5569bdc62eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:10:13.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.283+0000 7f89a6a00640 1 --2- 172.21.15.28:0/2232991447 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5569bdc4c510 0x5569bdc62eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:13.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.283+0000 7f89a6a00640 1 -- 172.21.15.28:0/2232991447 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5569bdc50d60 con 0x5569bdc442a0 2024-05-12T22:10:13.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.283+0000 7f89a6a00640 1 --2- 172.21.15.28:0/2232991447 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5569bdc442a0 0x5569bdc6fe60 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f8994097f00 tx=0x7f8994009f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdbfd7ad24d5428b server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:10:13.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.284+0000 7f8993400640 1 -- 172.21.15.28:0/2232991447 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89940a4710 con 0x5569bdc442a0 2024-05-12T22:10:13.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.284+0000 7f8993400640 1 -- 172.21.15.28:0/2232991447 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8994099da0 con 0x5569bdc442a0 2024-05-12T22:10:13.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.284+0000 7f8993400640 1 -- 172.21.15.28:0/2232991447 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89940ccfe0 con 0x5569bdc442a0 2024-05-12T22:10:13.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.284+0000 7f89b0fd9740 1 -- 172.21.15.28:0/2232991447 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5569bdc442a0 msgr2=0x5569bdc6fe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:10:13.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.284+0000 7f89b0fd9740 1 --2- 172.21.15.28:0/2232991447 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5569bdc442a0 0x5569bdc6fe60 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f8994097f00 tx=0x7f8994009f70 comp rx=0 tx=0).stop 2024-05-12T22:10:13.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.285+0000 7f89b0fd9740 1 -- 172.21.15.28:0/2232991447 shutdown_connections 2024-05-12T22:10:13.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.285+0000 7f89b0fd9740 1 --2- 172.21.15.28:0/2232991447 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5569bdc4c510 0x5569bdc62eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:13.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.285+0000 7f89b0fd9740 1 --2- 172.21.15.28:0/2232991447 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5569bdc442a0 0x5569bdc6fe60 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:13.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.285+0000 7f89b0fd9740 1 --2- 172.21.15.28:0/2232991447 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5569bdc43980 0x5569bdc6f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:13.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.285+0000 7f89b0fd9740 1 -- 172.21.15.28:0/2232991447 >> 172.21.15.28:0/2232991447 conn(0x5569bdc3fca0 msgr2=0x5569bdc46c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:10:13.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.285+0000 7f89b0fd9740 1 -- 172.21.15.28:0/2232991447 shutdown_connections 2024-05-12T22:10:13.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.285+0000 7f89b0fd9740 1 -- 172.21.15.28:0/2232991447 wait complete. 2024-05-12T22:10:13.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.285+0000 7f89b0fd9740 1 -- start start 2024-05-12T22:10:13.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.285+0000 7f89b0fd9740 1 Processor -- start 2024-05-12T22:10:13.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.286+0000 7f89b0fd9740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5569bdc43980 0x5569bdc48b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:13.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.286+0000 7f89b0fd9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5569bdc442a0 0x5569bdc49080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:13.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.286+0000 7f89b0fd9740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5569bdc4c510 0x5569bdccd190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:13.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.286+0000 7f89b0fd9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f89a81a98d0 con 0x5569bdc442a0 2024-05-12T22:10:13.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.286+0000 7f89b0fd9740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f89a81aad40 con 0x5569bdc43980 2024-05-12T22:10:13.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.286+0000 7f89b0fd9740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f89a81a9fa0 con 0x5569bdc4c510 2024-05-12T22:10:13.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.286+0000 7f89b0fd9740 10 client.0 _ll_register_callbacks cb 0x7f8998009520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:10:13.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.286+0000 7f89a7400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5569bdc43980 0x5569bdc48b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:13.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.286+0000 7f89a6a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5569bdc442a0 0x5569bdc49080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:13.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.286+0000 7f89a6a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5569bdc442a0 0x5569bdc49080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42314/0 (socket says 172.21.15.28:42314) 2024-05-12T22:10:13.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.286+0000 7f89a7400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5569bdc43980 0x5569bdc48b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:54640/0 (socket says 172.21.15.28:54640) 2024-05-12T22:10:13.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.286+0000 7f89a6a00640 1 -- 172.21.15.28:0/2701486803 learned_addr learned my addr 172.21.15.28:0/2701486803 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:10:13.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.287+0000 7f89a7e00640 1 --2- 172.21.15.28:0/2701486803 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5569bdc4c510 0x5569bdccd190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:13.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.287+0000 7f89a6a00640 1 -- 172.21.15.28:0/2701486803 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5569bdc4c510 msgr2=0x5569bdccd190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:10:13.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.287+0000 7f89a6a00640 1 --2- 172.21.15.28:0/2701486803 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5569bdc4c510 0x5569bdccd190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:13.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.287+0000 7f89a6a00640 1 -- 172.21.15.28:0/2701486803 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5569bdc43980 msgr2=0x5569bdc48b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:10:13.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.287+0000 7f89a6a00640 1 --2- 172.21.15.28:0/2701486803 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5569bdc43980 0x5569bdc48b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:13.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.287+0000 7f89a6a00640 1 -- 172.21.15.28:0/2701486803 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5569bdc51b70 con 0x5569bdc442a0 2024-05-12T22:10:13.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.287+0000 7f89a7400640 1 --2- 172.21.15.28:0/2701486803 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5569bdc43980 0x5569bdc48b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:10:13.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.287+0000 7f89a6a00640 1 --2- 172.21.15.28:0/2701486803 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5569bdc442a0 0x5569bdc49080 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f8994009230 tx=0x7f89940cccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ec712e524390331 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:10:13.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.287+0000 7f8990c00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-12T22:10:13.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.287+0000 7f8990c00640 1 -- 172.21.15.28:0/2701486803 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89940a9060 con 0x5569bdc442a0 2024-05-12T22:10:13.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.287+0000 7f8990c00640 1 -- 172.21.15.28:0/2701486803 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f89940ba300 con 0x5569bdc442a0 2024-05-12T22:10:13.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.287+0000 7f89b0fd9740 1 -- 172.21.15.28:0/2701486803 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8994099da0 con 0x5569bdc442a0 2024-05-12T22:10:13.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.288+0000 7f8990c00640 1 -- 172.21.15.28:0/2701486803 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f89940a4710 con 0x5569bdc442a0 2024-05-12T22:10:13.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.288+0000 7f89b0fd9740 10 client.15510 fetch_fsmap learned FSMap version 24 2024-05-12T22:10:13.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.288+0000 7f89b0fd9740 1 -- 172.21.15.28:0/2701486803 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5569bdc52e50 con 0x5569bdc442a0 2024-05-12T22:10:13.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.288+0000 7f8990c00640 1 -- 172.21.15.28:0/2701486803 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8994099da0 con 0x5569bdc442a0 2024-05-12T22:10:13.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.288+0000 7f8990c00640 1 -- 172.21.15.28:0/2701486803 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f89940ccfe0 con 0x5569bdc442a0 2024-05-12T22:10:13.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.288+0000 7f89b0fd9740 10 client.15510 fetch_fsmap finished waiting for FSMap version 24 2024-05-12T22:10:13.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.288+0000 7f89b0fd9740 10 client.15510 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-05-12T22:10:13.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.288+0000 7f89b0fd9740 20 client.15510 populate_metadata read hostname 'smithi028' 2024-05-12T22:10:13.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.288+0000 7f89b0fd9740 1 --2- 172.21.15.28:0/2701486803 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x5569bdc53560 0x5569bdc55980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:13.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.288+0000 7f89b0fd9740 4 client.15510 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-12T22:10:13.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.289+0000 7f89b0fd9740 1 -- 172.21.15.28:0/2701486803 --> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5569bdbb04b0 con 0x5569bdc53560 2024-05-12T22:10:13.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.289+0000 7f89a7400640 1 --2- 172.21.15.28:0/2701486803 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x5569bdc53560 0x5569bdc55980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:13.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.289+0000 7f89a7400640 1 --2- 172.21.15.28:0/2701486803 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x5569bdc53560 0x5569bdc55980 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c621bd130984bc37 server_cookie=41390230069a7b60 in_seq=0 out_seq=0 2024-05-12T22:10:13.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.289+0000 7f8990c00640 10 client.15510 ms_handle_connect on v2:172.21.15.28:6840/761183870 2024-05-12T22:10:13.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.290+0000 7f8990c00640 1 -- 172.21.15.28:0/2701486803 <== mds.1 v2:172.21.15.28:6840/761183870 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5569bdbb04b0 con 0x5569bdc53560 2024-05-12T22:10:13.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.290+0000 7f8990c00640 10 client.15510 handle_command_reply: tid=0 2024-05-12T22:10:13.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.290+0000 7f89b0fd9740 1 client.15510 shutdown 2024-05-12T22:10:13.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.290+0000 7f89b0fd9740 1 -- 172.21.15.28:0/2701486803 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5569bdc442a0 msgr2=0x5569bdc49080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:10:13.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.290+0000 7f89b0fd9740 1 --2- 172.21.15.28:0/2701486803 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5569bdc442a0 0x5569bdc49080 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f8994009230 tx=0x7f89940cccf0 comp rx=0 tx=0).stop 2024-05-12T22:10:13.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.290+0000 7f89b0fd9740 1 -- 172.21.15.28:0/2701486803 shutdown_connections 2024-05-12T22:10:13.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.290+0000 7f89b0fd9740 1 --2- 172.21.15.28:0/2701486803 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x5569bdc53560 0x5569bdc55980 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:13.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.290+0000 7f89b0fd9740 1 --2- 172.21.15.28:0/2701486803 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5569bdc4c510 0x5569bdccd190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:13.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.290+0000 7f89b0fd9740 1 --2- 172.21.15.28:0/2701486803 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5569bdc442a0 0x5569bdc49080 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:13.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.290+0000 7f89b0fd9740 1 --2- 172.21.15.28:0/2701486803 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5569bdc43980 0x5569bdc48b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:13.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.290+0000 7f8990c00640 0 client.15510 ms_handle_reset on v2:172.21.15.28:6840/761183870 2024-05-12T22:10:13.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.290+0000 7f89b0fd9740 1 -- 172.21.15.28:0/2701486803 >> 172.21.15.28:0/2701486803 conn(0x5569bdc3fca0 msgr2=0x5569bdc45190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:10:13.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.291+0000 7f89b0fd9740 1 -- 172.21.15.28:0/2701486803 shutdown_connections 2024-05-12T22:10:13.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.291+0000 7f89b0fd9740 1 -- 172.21.15.28:0/2701486803 wait complete. 2024-05-12T22:10:13.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.291+0000 7f89b0fd9740 20 client.15510 trim_cache size 0 max 16384 2024-05-12T22:10:13.294 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-05-12T22:10:13.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.291+0000 7f89ae200640 1 -- 172.21.15.28:0/237467389 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f898c0a2f40 msgr2=0x7f898c0a5400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:10:13.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.291+0000 7f89ae200640 1 --2- 172.21.15.28:0/237467389 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f898c0a2f40 0x7f898c0a5400 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f89a0004950 tx=0x7f89a0005fd0 comp rx=0 tx=0).stop 2024-05-12T22:10:13.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.291+0000 7f89ae200640 1 -- 172.21.15.28:0/237467389 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89a806a7f0 msgr2=0x7f89a8101090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:10:13.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.291+0000 7f89ae200640 1 --2- 172.21.15.28:0/237467389 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89a806a7f0 0x7f89a8101090 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f8994005fd0 tx=0x7f899400ee50 comp rx=0 tx=0).stop 2024-05-12T22:10:13.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.292+0000 7f89ae200640 1 -- 172.21.15.28:0/237467389 shutdown_connections 2024-05-12T22:10:13.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.292+0000 7f89ae200640 1 --2- 172.21.15.28:0/237467389 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f898c0a2f40 0x7f898c0a5400 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:13.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.292+0000 7f89ae200640 1 --2- 172.21.15.28:0/237467389 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f89a8107810 0x7f89a81015d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:13.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.292+0000 7f89ae200640 1 --2- 172.21.15.28:0/237467389 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89a806a7f0 0x7f89a8101090 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:13.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.292+0000 7f89ae200640 1 --2- 172.21.15.28:0/237467389 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f89a8069e30 0x7f89a8100b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:13.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.292+0000 7f89ae200640 1 -- 172.21.15.28:0/237467389 >> 172.21.15.28:0/237467389 conn(0x7f89a8076530 msgr2=0x7f89a80fe750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:10:13.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.292+0000 7f89ae200640 1 -- 172.21.15.28:0/237467389 shutdown_connections 2024-05-12T22:10:13.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.292+0000 7f89ae200640 1 -- 172.21.15.28:0/237467389 wait complete. 2024-05-12T22:10:13.308 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-05-12T22:10:13.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.479+0000 7fab50a00640 1 -- 172.21.15.28:0/3868395793 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fab4c073be0 msgr2=0x7fab4c073fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:10:13.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.479+0000 7fab50a00640 1 --2- 172.21.15.28:0/3868395793 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fab4c073be0 0x7fab4c073fc0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fab3c009900 tx=0x7fab3c02a490 comp rx=0 tx=0).stop 2024-05-12T22:10:13.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.479+0000 7fab50a00640 1 -- 172.21.15.28:0/3868395793 shutdown_connections 2024-05-12T22:10:13.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.479+0000 7fab50a00640 1 --2- 172.21.15.28:0/3868395793 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab4c10df90 0x7fab4c110380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:13.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.479+0000 7fab50a00640 1 --2- 172.21.15.28:0/3868395793 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fab4c074500 0x7fab4c10da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:13.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.479+0000 7fab50a00640 1 --2- 172.21.15.28:0/3868395793 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fab4c073be0 0x7fab4c073fc0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:13.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.479+0000 7fab50a00640 1 -- 172.21.15.28:0/3868395793 >> 172.21.15.28:0/3868395793 conn(0x7fab4c0fd200 msgr2=0x7fab4c0ff620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:10:13.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.479+0000 7fab50a00640 1 -- 172.21.15.28:0/3868395793 shutdown_connections 2024-05-12T22:10:13.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.479+0000 7fab50a00640 1 -- 172.21.15.28:0/3868395793 wait complete. 2024-05-12T22:10:13.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.480+0000 7fab50a00640 1 Processor -- start 2024-05-12T22:10:13.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.481+0000 7fab50a00640 1 -- start start 2024-05-12T22:10:13.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.481+0000 7fab50a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab4c073be0 0x7fab4c19a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:13.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.481+0000 7fab50a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fab4c074500 0x7fab4c19a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:13.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.481+0000 7fab50a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fab4c10df90 0x7fab4c19eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:13.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.481+0000 7fab50a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fab4c113cb0 con 0x7fab4c073be0 2024-05-12T22:10:13.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.481+0000 7fab50a00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fab4c113b30 con 0x7fab4c074500 2024-05-12T22:10:13.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.481+0000 7fab50a00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fab4c113e30 con 0x7fab4c10df90 2024-05-12T22:10:13.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.482+0000 7fab4aa00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fab4c074500 0x7fab4c19a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:13.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.482+0000 7fab4aa00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fab4c074500 0x7fab4c19a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:54666/0 (socket says 172.21.15.28:54666) 2024-05-12T22:10:13.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.482+0000 7fab4aa00640 1 -- 172.21.15.28:0/1127976040 learned_addr learned my addr 172.21.15.28:0/1127976040 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:10:13.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.482+0000 7fab4b400640 1 --2- 172.21.15.28:0/1127976040 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab4c073be0 0x7fab4c19a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:13.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.482+0000 7fab4be00640 1 --2- 172.21.15.28:0/1127976040 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fab4c10df90 0x7fab4c19eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:13.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.482+0000 7fab4aa00640 1 -- 172.21.15.28:0/1127976040 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fab4c10df90 msgr2=0x7fab4c19eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:10:13.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.482+0000 7fab4aa00640 1 --2- 172.21.15.28:0/1127976040 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fab4c10df90 0x7fab4c19eab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:13.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.482+0000 7fab4aa00640 1 -- 172.21.15.28:0/1127976040 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab4c073be0 msgr2=0x7fab4c19a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:10:13.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.482+0000 7fab4aa00640 1 --2- 172.21.15.28:0/1127976040 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab4c073be0 0x7fab4c19a1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:13.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.482+0000 7fab4aa00640 1 -- 172.21.15.28:0/1127976040 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab4c19f1e0 con 0x7fab4c074500 2024-05-12T22:10:13.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.482+0000 7fab4b400640 1 --2- 172.21.15.28:0/1127976040 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab4c073be0 0x7fab4c19a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:10:13.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.483+0000 7fab4aa00640 1 --2- 172.21.15.28:0/1127976040 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fab4c074500 0x7fab4c19a730 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fab3800e9f0 tx=0x7fab3800edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:10:13.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.483+0000 7fab43e00640 1 -- 172.21.15.28:0/1127976040 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab3800ccc0 con 0x7fab4c074500 2024-05-12T22:10:13.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.483+0000 7fab50a00640 1 -- 172.21.15.28:0/1127976040 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab4c19f4d0 con 0x7fab4c074500 2024-05-12T22:10:13.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.483+0000 7fab43e00640 1 -- 172.21.15.28:0/1127976040 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fab3800ce60 con 0x7fab4c074500 2024-05-12T22:10:13.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.483+0000 7fab43e00640 1 -- 172.21.15.28:0/1127976040 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab38010690 con 0x7fab4c074500 2024-05-12T22:10:13.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.483+0000 7fab50a00640 1 -- 172.21.15.28:0/1127976040 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab4c1a6d60 con 0x7fab4c074500 2024-05-12T22:10:13.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.488+0000 7fab54f6d740 1 Processor -- start 2024-05-12T22:10:13.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.488+0000 7fab54f6d740 1 -- start start 2024-05-12T22:10:13.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.488+0000 7fab54f6d740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56488b0b0b60 0x56488b0b0f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:13.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.488+0000 7fab54f6d740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56488b0b1480 0x56488b0bb610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:13.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.488+0000 7fab54f6d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56488b0bbb50 0x56488b0bdf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:13.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.488+0000 7fab54f6d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56488b015a30 con 0x56488b0bbb50 2024-05-12T22:10:13.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.488+0000 7fab54f6d740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x56488afbd700 con 0x56488b0b1480 2024-05-12T22:10:13.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.488+0000 7fab54f6d740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x56488b0119f0 con 0x56488b0b0b60 2024-05-12T22:10:13.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.489+0000 7fab4be00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56488b0b0b60 0x56488b0b0f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:13.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.489+0000 7fab4be00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56488b0b0b60 0x56488b0b0f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:43076/0 (socket says 172.21.15.28:43076) 2024-05-12T22:10:13.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.489+0000 7fab4be00640 1 -- 172.21.15.28:0/3215215813 learned_addr learned my addr 172.21.15.28:0/3215215813 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:10:13.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.489+0000 7fab4b400640 1 --2- 172.21.15.28:0/3215215813 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56488b0b1480 0x56488b0bb610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:13.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.489+0000 7fab4be00640 1 -- 172.21.15.28:0/3215215813 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56488b0b1480 msgr2=0x56488b0bb610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:10:13.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.489+0000 7fab4be00640 1 --2- 172.21.15.28:0/3215215813 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56488b0b1480 0x56488b0bb610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:13.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.489+0000 7fab4be00640 1 -- 172.21.15.28:0/3215215813 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56488b0bbb50 msgr2=0x56488b0bdf40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-12T22:10:13.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.489+0000 7fab4be00640 1 --2- 172.21.15.28:0/3215215813 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56488b0bbb50 0x56488b0bdf40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:13.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.489+0000 7fab4be00640 1 -- 172.21.15.28:0/3215215813 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56488ad293f0 con 0x56488b0b0b60 2024-05-12T22:10:13.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.490+0000 7fab43e00640 1 -- 172.21.15.28:0/1127976040 <== mon.2 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 26) ==== 138117+0+0 (secure 0 0 0) 0x7fab380046c0 con 0x7fab4c074500 2024-05-12T22:10:13.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.490+0000 7fab4be00640 1 --2- 172.21.15.28:0/3215215813 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56488b0b0b60 0x56488b0b0f40 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fab44009f60 tx=0x7fab44009f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fbeaf0660af18406 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:10:13.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.490+0000 7fab37e00640 1 -- 172.21.15.28:0/3215215813 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab4401e070 con 0x56488b0b0b60 2024-05-12T22:10:13.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.490+0000 7fab37e00640 1 -- 172.21.15.28:0/3215215813 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fab44003c30 con 0x56488b0b0b60 2024-05-12T22:10:13.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.490+0000 7fab37e00640 1 -- 172.21.15.28:0/3215215813 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab4400f920 con 0x56488b0b0b60 2024-05-12T22:10:13.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.491+0000 7fab54f6d740 1 -- 172.21.15.28:0/3215215813 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56488b0b0b60 msgr2=0x56488b0b0f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:10:13.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.491+0000 7fab54f6d740 1 --2- 172.21.15.28:0/3215215813 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56488b0b0b60 0x56488b0b0f40 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fab44009f60 tx=0x7fab44009f90 comp rx=0 tx=0).stop 2024-05-12T22:10:13.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.491+0000 7fab54f6d740 1 -- 172.21.15.28:0/3215215813 shutdown_connections 2024-05-12T22:10:13.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.491+0000 7fab54f6d740 1 --2- 172.21.15.28:0/3215215813 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56488b0bbb50 0x56488b0bdf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:13.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.491+0000 7fab54f6d740 1 --2- 172.21.15.28:0/3215215813 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56488b0b1480 0x56488b0bb610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:13.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.491+0000 7fab54f6d740 1 --2- 172.21.15.28:0/3215215813 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56488b0b0b60 0x56488b0b0f40 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:13.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.491+0000 7fab54f6d740 1 -- 172.21.15.28:0/3215215813 >> 172.21.15.28:0/3215215813 conn(0x56488b0ace80 msgr2=0x56488b0ad260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:10:13.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.491+0000 7fab43e00640 1 --2- 172.21.15.28:0/1127976040 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fab300a2f30 0x7fab300a53f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:13.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.492+0000 7fab54f6d740 1 -- 172.21.15.28:0/3215215813 shutdown_connections 2024-05-12T22:10:13.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.492+0000 7fab54f6d740 1 -- 172.21.15.28:0/3215215813 wait complete. 2024-05-12T22:10:13.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.492+0000 7fab43e00640 1 -- 172.21.15.28:0/1127976040 <== mon.2 v2:172.21.15.86:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7fab380cfb50 con 0x7fab4c074500 2024-05-12T22:10:13.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.492+0000 7fab54f6d740 1 -- start start 2024-05-12T22:10:13.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.492+0000 7fab4b400640 1 --2- 172.21.15.28:0/1127976040 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fab300a2f30 0x7fab300a53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:13.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.492+0000 7fab54f6d740 1 Processor -- start 2024-05-12T22:10:13.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.493+0000 7fab54f6d740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56488b0b0b60 0x56488b0b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:13.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.493+0000 7fab54f6d740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56488b0b1480 0x56488b0b90c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:13.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.493+0000 7fab54f6d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56488b0bbb50 0x56488b0b9600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:13.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.493+0000 7fab54f6d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56488b0be940 con 0x56488b0bbb50 2024-05-12T22:10:13.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.493+0000 7fab54f6d740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x56488afd0a70 con 0x56488b0b1480 2024-05-12T22:10:13.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.493+0000 7fab54f6d740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x56488b0beac0 con 0x56488b0b0b60 2024-05-12T22:10:13.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.493+0000 7fab54f6d740 10 client.0 _ll_register_callbacks cb 0x56488afb3ba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:10:13.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.493+0000 7fab4aa00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56488b0b1480 0x56488b0b90c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:13.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.494+0000 7fab4aa00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56488b0b1480 0x56488b0b90c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:54692/0 (socket says 172.21.15.28:54692) 2024-05-12T22:10:13.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.494+0000 7fab4aa00640 1 -- 172.21.15.28:0/41217626 learned_addr learned my addr 172.21.15.28:0/41217626 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:10:13.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.494+0000 7fab4be00640 1 --2- 172.21.15.28:0/41217626 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56488b0bbb50 0x56488b0b9600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:13.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.494+0000 7fab4aa00640 1 -- 172.21.15.28:0/41217626 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56488b0b0b60 msgr2=0x56488b0b8b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-12T22:10:13.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.494+0000 7fab4aa00640 1 --2- 172.21.15.28:0/41217626 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56488b0b0b60 0x56488b0b8b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:13.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.494+0000 7fab4aa00640 1 -- 172.21.15.28:0/41217626 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56488b0bbb50 msgr2=0x56488b0b9600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:10:13.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.494+0000 7fab4aa00640 1 --2- 172.21.15.28:0/41217626 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56488b0bbb50 0x56488b0b9600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:13.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.494+0000 7fab4aa00640 1 -- 172.21.15.28:0/41217626 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56488adf37d0 con 0x56488b0b1480 2024-05-12T22:10:13.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.494+0000 7fab4b400640 1 --2- 172.21.15.28:0/1127976040 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fab300a2f30 0x7fab300a53f0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fab3c009230 tx=0x7fab3c0058d0 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:10:13.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.494+0000 7fab4aa00640 1 --2- 172.21.15.28:0/41217626 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56488b0b1480 0x56488b0b90c0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fab3800e9c0 tx=0x7fab3809c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=593738390bbf5e2 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:10:13.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.494+0000 7fab34c00640 10 client.0 ms_handle_connect on v2:172.21.15.86:3300/0 2024-05-12T22:10:13.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.495+0000 7fab34c00640 1 -- 172.21.15.28:0/41217626 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab380a4ad0 con 0x56488b0b1480 2024-05-12T22:10:13.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.495+0000 7fab34c00640 1 -- 172.21.15.28:0/41217626 <== mon.2 v2:172.21.15.86:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7fab3809b2b0 con 0x56488b0b1480 2024-05-12T22:10:13.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.495+0000 7fab54f6d740 1 -- 172.21.15.28:0/41217626 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fab44003c30 con 0x56488b0b1480 2024-05-12T22:10:13.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.495+0000 7fab34c00640 1 -- 172.21.15.28:0/41217626 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fab38066050 con 0x56488b0b1480 2024-05-12T22:10:13.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.495+0000 7fab54f6d740 10 client.25640 fetch_fsmap learned FSMap version 24 2024-05-12T22:10:13.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.495+0000 7fab54f6d740 1 -- 172.21.15.28:0/41217626 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x56488b0d3480 con 0x56488b0b1480 2024-05-12T22:10:13.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.495+0000 7fab34c00640 1 -- 172.21.15.28:0/41217626 <== mon.2 v2:172.21.15.86:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fab44003c30 con 0x56488b0b1480 2024-05-12T22:10:13.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.495+0000 7fab34c00640 1 -- 172.21.15.28:0/41217626 <== mon.2 v2:172.21.15.86:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7fab380babc0 con 0x56488b0b1480 2024-05-12T22:10:13.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.495+0000 7fab54f6d740 10 client.25640 fetch_fsmap finished waiting for FSMap version 24 2024-05-12T22:10:13.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.495+0000 7fab54f6d740 10 client.25640 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:10:13.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.496+0000 7fab54f6d740 20 client.25640 populate_metadata read hostname 'smithi028' 2024-05-12T22:10:13.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.496+0000 7fab54f6d740 1 --2- 172.21.15.28:0/41217626 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x56488b0d4940 0x56488b0d6d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:13.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.496+0000 7fab54f6d740 4 client.25640 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-12T22:10:13.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.496+0000 7fab54f6d740 1 -- 172.21.15.28:0/41217626 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56488b023160 con 0x56488b0d4940 2024-05-12T22:10:13.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.496+0000 7fab4b400640 1 --2- 172.21.15.28:0/41217626 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x56488b0d4940 0x56488b0d6d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:13.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.496+0000 7fab4b400640 1 --2- 172.21.15.28:0/41217626 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x56488b0d4940 0x56488b0d6d30 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=302fe589704e6af9 server_cookie=781b1681145a47bb in_seq=0 out_seq=0 2024-05-12T22:10:13.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.496+0000 7fab34c00640 10 client.25640 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:10:13.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.497+0000 7fab34c00640 1 -- 172.21.15.28:0/41217626 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56488b023160 con 0x56488b0d4940 2024-05-12T22:10:13.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.497+0000 7fab34c00640 10 client.25640 handle_command_reply: tid=0 2024-05-12T22:10:13.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.497+0000 7fab54f6d740 1 client.25640 shutdown 2024-05-12T22:10:13.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.497+0000 7fab54f6d740 1 -- 172.21.15.28:0/41217626 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56488b0b1480 msgr2=0x56488b0b90c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:10:13.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.497+0000 7fab54f6d740 1 --2- 172.21.15.28:0/41217626 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56488b0b1480 0x56488b0b90c0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fab3800e9c0 tx=0x7fab3809c6a0 comp rx=0 tx=0).stop 2024-05-12T22:10:13.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.498+0000 7fab54f6d740 1 -- 172.21.15.28:0/41217626 shutdown_connections 2024-05-12T22:10:13.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.498+0000 7fab54f6d740 1 --2- 172.21.15.28:0/41217626 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x56488b0d4940 0x56488b0d6d30 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:13.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.498+0000 7fab54f6d740 1 --2- 172.21.15.28:0/41217626 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56488b0bbb50 0x56488b0b9600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:13.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.498+0000 7fab54f6d740 1 --2- 172.21.15.28:0/41217626 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56488b0b1480 0x56488b0b90c0 secure :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fab3800e9c0 tx=0x7fab3809c6a0 comp rx=0 tx=0).stop 2024-05-12T22:10:13.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.498+0000 7fab54f6d740 1 --2- 172.21.15.28:0/41217626 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56488b0b0b60 0x56488b0b8b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:13.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.498+0000 7fab54f6d740 1 -- 172.21.15.28:0/41217626 >> 172.21.15.28:0/41217626 conn(0x56488b0ace80 msgr2=0x56488b0b2c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:10:13.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.498+0000 7fab34c00640 0 client.25640 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:10:13.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.499+0000 7fab54f6d740 1 -- 172.21.15.28:0/41217626 shutdown_connections 2024-05-12T22:10:13.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.499+0000 7fab54f6d740 1 -- 172.21.15.28:0/41217626 wait complete. 2024-05-12T22:10:13.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.499+0000 7fab54f6d740 20 client.25640 trim_cache size 0 max 16384 2024-05-12T22:10:13.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.514+0000 7fab54f6d740 1 Processor -- start 2024-05-12T22:10:13.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.514+0000 7fab54f6d740 1 -- start start 2024-05-12T22:10:13.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.514+0000 7fab54f6d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56488b0b0b60 0x56488b0ac780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:13.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.514+0000 7fab54f6d740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56488b0bbb50 0x56488b0b33d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:13.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.514+0000 7fab54f6d740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56488b0bb3c0 0x56488b0dc600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:13.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.514+0000 7fab54f6d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fab4c1ad9b0 con 0x56488b0b0b60 2024-05-12T22:10:13.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.514+0000 7fab54f6d740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fab4c1ab280 con 0x56488b0bb3c0 2024-05-12T22:10:13.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.514+0000 7fab54f6d740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fab4c1a90d0 con 0x56488b0bbb50 2024-05-12T22:10:13.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.514+0000 7fab4be00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56488b0bb3c0 0x56488b0dc600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:13.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.514+0000 7fab4b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56488b0b0b60 0x56488b0ac780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:13.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.514+0000 7fab4be00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56488b0bb3c0 0x56488b0dc600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:54702/0 (socket says 172.21.15.28:54702) 2024-05-12T22:10:13.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.514+0000 7fab4aa00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56488b0bbb50 0x56488b0b33d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:13.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.514+0000 7fab4be00640 1 -- 172.21.15.28:0/1612706955 learned_addr learned my addr 172.21.15.28:0/1612706955 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:10:13.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.515+0000 7fab4be00640 1 -- 172.21.15.28:0/1612706955 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56488b0bbb50 msgr2=0x56488b0b33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:10:13.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.515+0000 7fab4be00640 1 --2- 172.21.15.28:0/1612706955 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56488b0bbb50 0x56488b0b33d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:13.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.515+0000 7fab4be00640 1 -- 172.21.15.28:0/1612706955 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56488b0b0b60 msgr2=0x56488b0ac780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:10:13.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.515+0000 7fab4be00640 1 --2- 172.21.15.28:0/1612706955 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56488b0b0b60 0x56488b0ac780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:13.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.515+0000 7fab4be00640 1 -- 172.21.15.28:0/1612706955 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56488b0ae0b0 con 0x56488b0bb3c0 2024-05-12T22:10:13.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.515+0000 7fab4be00640 1 --2- 172.21.15.28:0/1612706955 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56488b0bb3c0 0x56488b0dc600 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fab44006010 tx=0x7fab44005eb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8dbc71aa66a577b7 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:10:13.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.515+0000 7fab37400640 1 -- 172.21.15.28:0/1612706955 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab44029070 con 0x56488b0bb3c0 2024-05-12T22:10:13.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.516+0000 7fab37400640 1 -- 172.21.15.28:0/1612706955 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fab44003db0 con 0x56488b0bb3c0 2024-05-12T22:10:13.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.516+0000 7fab37400640 1 -- 172.21.15.28:0/1612706955 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab4401e040 con 0x56488b0bb3c0 2024-05-12T22:10:13.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.516+0000 7fab54f6d740 1 -- 172.21.15.28:0/1612706955 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56488b0bb3c0 msgr2=0x56488b0dc600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:10:13.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.516+0000 7fab54f6d740 1 --2- 172.21.15.28:0/1612706955 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56488b0bb3c0 0x56488b0dc600 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fab44006010 tx=0x7fab44005eb0 comp rx=0 tx=0).stop 2024-05-12T22:10:13.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.516+0000 7fab54f6d740 1 -- 172.21.15.28:0/1612706955 shutdown_connections 2024-05-12T22:10:13.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.516+0000 7fab54f6d740 1 --2- 172.21.15.28:0/1612706955 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56488b0bb3c0 0x56488b0dc600 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:13.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.516+0000 7fab54f6d740 1 --2- 172.21.15.28:0/1612706955 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56488b0bbb50 0x56488b0b33d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:13.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.516+0000 7fab54f6d740 1 --2- 172.21.15.28:0/1612706955 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56488b0b0b60 0x56488b0ac780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:13.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.516+0000 7fab54f6d740 1 -- 172.21.15.28:0/1612706955 >> 172.21.15.28:0/1612706955 conn(0x56488b0ace80 msgr2=0x56488b0b93d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:10:13.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.516+0000 7fab54f6d740 1 -- 172.21.15.28:0/1612706955 shutdown_connections 2024-05-12T22:10:13.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.516+0000 7fab54f6d740 1 -- 172.21.15.28:0/1612706955 wait complete. 2024-05-12T22:10:13.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.517+0000 7fab54f6d740 1 -- start start 2024-05-12T22:10:13.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.517+0000 7fab54f6d740 1 Processor -- start 2024-05-12T22:10:13.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.517+0000 7fab54f6d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56488b0b0b60 0x56488b0cc5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:13.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.518+0000 7fab54f6d740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56488b0bb3c0 0x56488b0ccae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:13.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.518+0000 7fab54f6d740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56488b0bbb50 0x56488b0c9570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:13.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.518+0000 7fab54f6d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fab4c1a9740 con 0x56488b0b0b60 2024-05-12T22:10:13.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.518+0000 7fab54f6d740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fab4c1aabb0 con 0x56488b0bbb50 2024-05-12T22:10:13.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.518+0000 7fab54f6d740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fab4c1a9e10 con 0x56488b0bb3c0 2024-05-12T22:10:13.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.518+0000 7fab54f6d740 10 client.0 _ll_register_callbacks cb 0x7fab3809b7c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:10:13.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.518+0000 7fab4aa00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56488b0bb3c0 0x56488b0ccae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:13.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.518+0000 7fab4aa00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56488b0bb3c0 0x56488b0ccae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:43104/0 (socket says 172.21.15.28:43104) 2024-05-12T22:10:13.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.518+0000 7fab4aa00640 1 -- 172.21.15.28:0/2157933561 learned_addr learned my addr 172.21.15.28:0/2157933561 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:10:13.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.518+0000 7fab4be00640 1 --2- 172.21.15.28:0/2157933561 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56488b0bbb50 0x56488b0c9570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:13.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.518+0000 7fab4b400640 1 --2- 172.21.15.28:0/2157933561 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56488b0b0b60 0x56488b0cc5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:13.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.518+0000 7fab4aa00640 1 -- 172.21.15.28:0/2157933561 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56488b0bbb50 msgr2=0x56488b0c9570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:10:13.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.518+0000 7fab4aa00640 1 --2- 172.21.15.28:0/2157933561 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56488b0bbb50 0x56488b0c9570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:13.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.518+0000 7fab4aa00640 1 -- 172.21.15.28:0/2157933561 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56488b0b0b60 msgr2=0x56488b0cc5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:10:13.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.519+0000 7fab4aa00640 1 --2- 172.21.15.28:0/2157933561 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56488b0b0b60 0x56488b0cc5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:13.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.519+0000 7fab4aa00640 1 -- 172.21.15.28:0/2157933561 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56488b0c9c60 con 0x56488b0bb3c0 2024-05-12T22:10:13.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.519+0000 7fab4be00640 1 --2- 172.21.15.28:0/2157933561 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56488b0bbb50 0x56488b0c9570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:10:13.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.519+0000 7fab4b400640 1 --2- 172.21.15.28:0/2157933561 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56488b0b0b60 0x56488b0cc5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:10:13.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.519+0000 7fab4aa00640 1 --2- 172.21.15.28:0/2157933561 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56488b0bb3c0 0x56488b0ccae0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fab3800c6a0 tx=0x7fab38009310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8f9e64919141e56b server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:10:13.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.519+0000 7fab34c00640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-12T22:10:13.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.519+0000 7fab34c00640 1 -- 172.21.15.28:0/2157933561 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab380a8070 con 0x56488b0bb3c0 2024-05-12T22:10:13.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.519+0000 7fab34c00640 1 -- 172.21.15.28:0/2157933561 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7fab3809b4b0 con 0x56488b0bb3c0 2024-05-12T22:10:13.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.519+0000 7fab54f6d740 1 -- 172.21.15.28:0/2157933561 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fab44003db0 con 0x56488b0bb3c0 2024-05-12T22:10:13.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.520+0000 7fab34c00640 1 -- 172.21.15.28:0/2157933561 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fab380a4ad0 con 0x56488b0bb3c0 2024-05-12T22:10:13.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.520+0000 7fab54f6d740 10 client.25606 fetch_fsmap learned FSMap version 24 2024-05-12T22:10:13.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.520+0000 7fab54f6d740 1 -- 172.21.15.28:0/2157933561 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x56488b13b860 con 0x56488b0bb3c0 2024-05-12T22:10:13.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.520+0000 7fab34c00640 1 -- 172.21.15.28:0/2157933561 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fab44003db0 con 0x56488b0bb3c0 2024-05-12T22:10:13.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.520+0000 7fab34c00640 1 -- 172.21.15.28:0/2157933561 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7fab380ab030 con 0x56488b0bb3c0 2024-05-12T22:10:13.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.520+0000 7fab54f6d740 10 client.25606 fetch_fsmap finished waiting for FSMap version 24 2024-05-12T22:10:13.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.520+0000 7fab54f6d740 10 client.25606 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:10:13.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.520+0000 7fab54f6d740 20 client.25606 populate_metadata read hostname 'smithi028' 2024-05-12T22:10:13.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.520+0000 7fab54f6d740 1 --2- 172.21.15.28:0/2157933561 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x56488b13bea0 0x56488b13e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:13.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.520+0000 7fab54f6d740 4 client.25606 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-12T22:10:13.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.520+0000 7fab54f6d740 1 -- 172.21.15.28:0/2157933561 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56488b01d4b0 con 0x56488b13bea0 2024-05-12T22:10:13.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.521+0000 7fab4b400640 1 --2- 172.21.15.28:0/2157933561 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x56488b13bea0 0x56488b13e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:13.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.521+0000 7fab34c00640 10 client.25606 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:10:13.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.521+0000 7fab4b400640 1 --2- 172.21.15.28:0/2157933561 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x56488b13bea0 0x56488b13e2c0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5b9c5dc5ff30d1cd server_cookie=44d79942b9c0a1ef in_seq=0 out_seq=0 2024-05-12T22:10:13.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.522+0000 7fab34c00640 1 -- 172.21.15.28:0/2157933561 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56488b01d4b0 con 0x56488b13bea0 2024-05-12T22:10:13.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.522+0000 7fab34c00640 10 client.25606 handle_command_reply: tid=0 2024-05-12T22:10:13.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.522+0000 7fab54f6d740 1 client.25606 shutdown 2024-05-12T22:10:13.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.522+0000 7fab54f6d740 1 -- 172.21.15.28:0/2157933561 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56488b0bb3c0 msgr2=0x56488b0ccae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:10:13.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.522+0000 7fab54f6d740 1 --2- 172.21.15.28:0/2157933561 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56488b0bb3c0 0x56488b0ccae0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fab3800c6a0 tx=0x7fab38009310 comp rx=0 tx=0).stop 2024-05-12T22:10:13.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.522+0000 7fab54f6d740 1 -- 172.21.15.28:0/2157933561 shutdown_connections 2024-05-12T22:10:13.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.522+0000 7fab54f6d740 1 --2- 172.21.15.28:0/2157933561 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x56488b13bea0 0x56488b13e2c0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:13.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.522+0000 7fab54f6d740 1 --2- 172.21.15.28:0/2157933561 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56488b0bbb50 0x56488b0c9570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:13.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.522+0000 7fab54f6d740 1 --2- 172.21.15.28:0/2157933561 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56488b0bb3c0 0x56488b0ccae0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:13.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.522+0000 7fab54f6d740 1 --2- 172.21.15.28:0/2157933561 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56488b0b0b60 0x56488b0cc5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:13.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.522+0000 7fab34c00640 0 client.25606 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:10:13.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.522+0000 7fab54f6d740 1 -- 172.21.15.28:0/2157933561 >> 172.21.15.28:0/2157933561 conn(0x56488b0ace80 msgr2=0x56488b0b8ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:10:13.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.522+0000 7fab54f6d740 1 -- 172.21.15.28:0/2157933561 shutdown_connections 2024-05-12T22:10:13.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.522+0000 7fab54f6d740 1 -- 172.21.15.28:0/2157933561 wait complete. 2024-05-12T22:10:13.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.522+0000 7fab54f6d740 20 client.25606 trim_cache size 0 max 16384 2024-05-12T22:10:13.526 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-05-12T22:10:13.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.523+0000 7fab50a00640 1 -- 172.21.15.28:0/1127976040 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fab300a2f30 msgr2=0x7fab300a53f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:10:13.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.523+0000 7fab50a00640 1 --2- 172.21.15.28:0/1127976040 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fab300a2f30 0x7fab300a53f0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fab3c009230 tx=0x7fab3c0058d0 comp rx=0 tx=0).stop 2024-05-12T22:10:13.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.523+0000 7fab50a00640 1 -- 172.21.15.28:0/1127976040 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fab4c074500 msgr2=0x7fab4c19a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:10:13.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.523+0000 7fab50a00640 1 --2- 172.21.15.28:0/1127976040 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fab4c074500 0x7fab4c19a730 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fab3800e9f0 tx=0x7fab3800edf0 comp rx=0 tx=0).stop 2024-05-12T22:10:13.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.523+0000 7fab50a00640 1 -- 172.21.15.28:0/1127976040 shutdown_connections 2024-05-12T22:10:13.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.523+0000 7fab50a00640 1 --2- 172.21.15.28:0/1127976040 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fab300a2f30 0x7fab300a53f0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:13.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.523+0000 7fab50a00640 1 --2- 172.21.15.28:0/1127976040 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fab4c10df90 0x7fab4c19eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:13.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.523+0000 7fab50a00640 1 --2- 172.21.15.28:0/1127976040 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fab4c074500 0x7fab4c19a730 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:13.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.523+0000 7fab50a00640 1 --2- 172.21.15.28:0/1127976040 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab4c073be0 0x7fab4c19a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:13.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.523+0000 7fab50a00640 1 -- 172.21.15.28:0/1127976040 >> 172.21.15.28:0/1127976040 conn(0x7fab4c0fd200 msgr2=0x7fab4c0fdf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:10:13.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.524+0000 7fab50a00640 1 -- 172.21.15.28:0/1127976040 shutdown_connections 2024-05-12T22:10:13.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:13.524+0000 7fab50a00640 1 -- 172.21.15.28:0/1127976040 wait complete. 2024-05-12T22:10:14.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:10:13 smithi028 ceph-mon[29847]: pgmap v40: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-05-12T22:10:14.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:10:13 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/3369791554' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:10:14.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:10:13 smithi186 ceph-mon[34785]: pgmap v40: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-05-12T22:10:14.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:10:13 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/3369791554' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:10:14.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:10:13 smithi086 ceph-mon[34297]: pgmap v40: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-05-12T22:10:14.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:10:13 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/3369791554' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:10:14.541 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-12T22:10:14.542 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-12T22:10:14.542 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-05-12T22:10:14.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.824+0000 7f0728c00640 1 -- 172.21.15.28:0/2187734963 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0724102890 msgr2=0x7f0724102c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:10:14.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.824+0000 7f0728c00640 1 --2- 172.21.15.28:0/2187734963 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0724102890 0x7f0724102c70 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f0714009860 tx=0x7f071402a3d0 comp rx=0 tx=0).stop 2024-05-12T22:10:14.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.824+0000 7f0728c00640 1 -- 172.21.15.28:0/2187734963 shutdown_connections 2024-05-12T22:10:14.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.824+0000 7f0728c00640 1 --2- 172.21.15.28:0/2187734963 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f072410bcb0 0x7f072410e150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:14.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.824+0000 7f0728c00640 1 --2- 172.21.15.28:0/2187734963 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07241031b0 0x7f072410b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:14.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.824+0000 7f0728c00640 1 --2- 172.21.15.28:0/2187734963 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0724102890 0x7f0724102c70 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:14.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.824+0000 7f0728c00640 1 -- 172.21.15.28:0/2187734963 >> 172.21.15.28:0/2187734963 conn(0x7f07240fc0b0 msgr2=0x7f07240fe4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:10:14.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.825+0000 7f0728c00640 1 -- 172.21.15.28:0/2187734963 shutdown_connections 2024-05-12T22:10:14.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.825+0000 7f0728c00640 1 -- 172.21.15.28:0/2187734963 wait complete. 2024-05-12T22:10:14.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.826+0000 7f0728c00640 1 Processor -- start 2024-05-12T22:10:14.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.826+0000 7f0728c00640 1 -- start start 2024-05-12T22:10:14.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.827+0000 7f0728c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0724102890 0x7f07241004e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:14.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.827+0000 7f0728c00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f07241031b0 0x7f0724100a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:14.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.827+0000 7f0728c00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f072410bcb0 0x7f0724100f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:14.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.827+0000 7f0728c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0724111b00 con 0x7f0724102890 2024-05-12T22:10:14.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.827+0000 7f0728c00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f0724111980 con 0x7f072410bcb0 2024-05-12T22:10:14.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.827+0000 7f0728c00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f0724111c80 con 0x7f07241031b0 2024-05-12T22:10:14.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.827+0000 7f0722a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f07241031b0 0x7f0724100a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:14.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.827+0000 7f0723400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0724102890 0x7f07241004e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:14.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.827+0000 7f0722a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f07241031b0 0x7f0724100a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:43126/0 (socket says 172.21.15.28:43126) 2024-05-12T22:10:14.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.827+0000 7f0722a00640 1 -- 172.21.15.28:0/2852154917 learned_addr learned my addr 172.21.15.28:0/2852154917 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:10:14.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.827+0000 7f0723e00640 1 --2- 172.21.15.28:0/2852154917 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f072410bcb0 0x7f0724100f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:14.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.828+0000 7f0722a00640 1 -- 172.21.15.28:0/2852154917 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f072410bcb0 msgr2=0x7f0724100f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:10:14.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.828+0000 7f0722a00640 1 --2- 172.21.15.28:0/2852154917 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f072410bcb0 0x7f0724100f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:14.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.828+0000 7f0722a00640 1 -- 172.21.15.28:0/2852154917 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0724102890 msgr2=0x7f07241004e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:10:14.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.828+0000 7f0722a00640 1 --2- 172.21.15.28:0/2852154917 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0724102890 0x7f07241004e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:14.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.828+0000 7f0722a00640 1 -- 172.21.15.28:0/2852154917 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07241a64c0 con 0x7f07241031b0 2024-05-12T22:10:14.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.828+0000 7f0723e00640 1 --2- 172.21.15.28:0/2852154917 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f072410bcb0 0x7f0724100f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:10:14.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.828+0000 7f0723400640 1 --2- 172.21.15.28:0/2852154917 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0724102890 0x7f07241004e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:10:14.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.829+0000 7f0722a00640 1 --2- 172.21.15.28:0/2852154917 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f07241031b0 0x7f0724100a20 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f071000e910 tx=0x7f071000edd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:10:14.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.829+0000 7f071be00640 1 -- 172.21.15.28:0/2852154917 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f071000cd80 con 0x7f07241031b0 2024-05-12T22:10:14.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.829+0000 7f071be00640 1 -- 172.21.15.28:0/2852154917 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f07100035f0 con 0x7f07241031b0 2024-05-12T22:10:14.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.829+0000 7f0728c00640 1 -- 172.21.15.28:0/2852154917 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07241a67b0 con 0x7f07241031b0 2024-05-12T22:10:14.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.829+0000 7f071be00640 1 -- 172.21.15.28:0/2852154917 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0710010670 con 0x7f07241031b0 2024-05-12T22:10:14.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.829+0000 7f0728c00640 1 -- 172.21.15.28:0/2852154917 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07241a6d00 con 0x7f07241031b0 2024-05-12T22:10:14.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.834+0000 7f072cf7e740 1 Processor -- start 2024-05-12T22:10:14.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.835+0000 7f072cf7e740 1 -- start start 2024-05-12T22:10:14.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.835+0000 7f072cf7e740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5564c914c3e0 0x5564c914c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:14.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.835+0000 7f072cf7e740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5564c914cd90 0x5564c915f760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:14.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.835+0000 7f072cf7e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564c915fd30 0x5564c9162140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:14.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.835+0000 7f072cf7e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5564c90b07c0 con 0x5564c915fd30 2024-05-12T22:10:14.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.835+0000 7f072cf7e740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x5564c9058790 con 0x5564c914c3e0 2024-05-12T22:10:14.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.835+0000 7f072cf7e740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5564c90ac780 con 0x5564c914cd90 2024-05-12T22:10:14.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.835+0000 7f071be00640 1 -- 172.21.15.28:0/2852154917 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 26) ==== 138117+0+0 (secure 0 0 0) 0x7f07100037a0 con 0x7f07241031b0 2024-05-12T22:10:14.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.836+0000 7f0723400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5564c914cd90 0x5564c915f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:14.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.836+0000 7f0723400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5564c914cd90 0x5564c915f760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:43138/0 (socket says 172.21.15.28:43138) 2024-05-12T22:10:14.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.836+0000 7f0723400640 1 -- 172.21.15.28:0/654258691 learned_addr learned my addr 172.21.15.28:0/654258691 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:10:14.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.836+0000 7f0723e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5564c914c3e0 0x5564c914c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:14.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.836+0000 7f0722a00640 1 --2- 172.21.15.28:0/654258691 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564c915fd30 0x5564c9162140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:14.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.836+0000 7f0723400640 1 -- 172.21.15.28:0/654258691 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5564c914c3e0 msgr2=0x5564c914c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:10:14.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.836+0000 7f0723400640 1 --2- 172.21.15.28:0/654258691 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5564c914c3e0 0x5564c914c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:14.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.836+0000 7f0723400640 1 -- 172.21.15.28:0/654258691 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564c915fd30 msgr2=0x5564c9162140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:10:14.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.836+0000 7f0723400640 1 --2- 172.21.15.28:0/654258691 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564c915fd30 0x5564c9162140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:14.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.836+0000 7f0723400640 1 -- 172.21.15.28:0/654258691 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5564c90786a0 con 0x5564c914cd90 2024-05-12T22:10:14.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.837+0000 7f0723400640 1 --2- 172.21.15.28:0/654258691 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5564c914cd90 0x5564c915f760 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f0714034000 tx=0x7f0714030e60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=722bf5c6ff7b9de1 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:10:14.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.837+0000 7f070fe00640 1 -- 172.21.15.28:0/654258691 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0714004230 con 0x5564c914cd90 2024-05-12T22:10:14.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.837+0000 7f070fe00640 1 -- 172.21.15.28:0/654258691 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0714052030 con 0x5564c914cd90 2024-05-12T22:10:14.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.837+0000 7f070fe00640 1 -- 172.21.15.28:0/654258691 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0714004920 con 0x5564c914cd90 2024-05-12T22:10:14.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.838+0000 7f071be00640 1 --2- 172.21.15.28:0/2852154917 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f07080a2f60 0x7f07080a5420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:14.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.838+0000 7f071be00640 1 -- 172.21.15.28:0/2852154917 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f07100cf870 con 0x7f07241031b0 2024-05-12T22:10:14.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.838+0000 7f072cf7e740 1 -- 172.21.15.28:0/654258691 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5564c914cd90 msgr2=0x5564c915f760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:10:14.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.838+0000 7f072cf7e740 1 --2- 172.21.15.28:0/654258691 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5564c914cd90 0x5564c915f760 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f0714034000 tx=0x7f0714030e60 comp rx=0 tx=0).stop 2024-05-12T22:10:14.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.838+0000 7f0723e00640 1 --2- 172.21.15.28:0/2852154917 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f07080a2f60 0x7f07080a5420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:14.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.838+0000 7f072cf7e740 1 -- 172.21.15.28:0/654258691 shutdown_connections 2024-05-12T22:10:14.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.838+0000 7f072cf7e740 1 --2- 172.21.15.28:0/654258691 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564c915fd30 0x5564c9162140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:14.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.838+0000 7f072cf7e740 1 --2- 172.21.15.28:0/654258691 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5564c914cd90 0x5564c915f760 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:14.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.838+0000 7f072cf7e740 1 --2- 172.21.15.28:0/654258691 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5564c914c3e0 0x5564c914c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:14.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.839+0000 7f072cf7e740 1 -- 172.21.15.28:0/654258691 >> 172.21.15.28:0/654258691 conn(0x5564c91485b0 msgr2=0x5564c9148990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:10:14.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.839+0000 7f072cf7e740 1 -- 172.21.15.28:0/654258691 shutdown_connections 2024-05-12T22:10:14.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.839+0000 7f072cf7e740 1 -- 172.21.15.28:0/654258691 wait complete. 2024-05-12T22:10:14.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.840+0000 7f072cf7e740 1 -- start start 2024-05-12T22:10:14.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.840+0000 7f0723e00640 1 --2- 172.21.15.28:0/2852154917 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f07080a2f60 0x7f07080a5420 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f071c009fd0 tx=0x7f071c009500 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:10:14.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.840+0000 7f072cf7e740 1 Processor -- start 2024-05-12T22:10:14.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.841+0000 7f072cf7e740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5564c914c3e0 0x5564c91597e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:14.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.841+0000 7f072cf7e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564c914cd90 0x5564c9159d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:14.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.841+0000 7f072cf7e740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5564c915fd30 0x5564c915f500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:14.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.841+0000 7f072cf7e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5564c914f170 con 0x5564c914cd90 2024-05-12T22:10:14.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.841+0000 7f072cf7e740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x5564c906ba90 con 0x5564c915fd30 2024-05-12T22:10:14.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.841+0000 7f072cf7e740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5564c914f2f0 con 0x5564c914c3e0 2024-05-12T22:10:14.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.841+0000 7f072cf7e740 10 client.0 _ll_register_callbacks cb 0x5564c90a6af0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:10:14.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.841+0000 7f0722a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564c914cd90 0x5564c9159d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:14.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.841+0000 7f0723e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5564c915fd30 0x5564c915f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:14.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.841+0000 7f0722a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564c914cd90 0x5564c9159d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42384/0 (socket says 172.21.15.28:42384) 2024-05-12T22:10:14.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.841+0000 7f0722a00640 1 -- 172.21.15.28:0/1600175373 learned_addr learned my addr 172.21.15.28:0/1600175373 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:10:14.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.841+0000 7f0723400640 1 --2- 172.21.15.28:0/1600175373 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5564c914c3e0 0x5564c91597e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:14.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.842+0000 7f0723e00640 1 -- 172.21.15.28:0/1600175373 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5564c914c3e0 msgr2=0x5564c91597e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:10:14.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.842+0000 7f0723e00640 1 --2- 172.21.15.28:0/1600175373 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5564c914c3e0 0x5564c91597e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:14.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.842+0000 7f0723e00640 1 -- 172.21.15.28:0/1600175373 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564c914cd90 msgr2=0x5564c9159d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:10:14.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.842+0000 7f0723e00640 1 --2- 172.21.15.28:0/1600175373 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564c914cd90 0x5564c9159d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:14.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.842+0000 7f0723e00640 1 -- 172.21.15.28:0/1600175373 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5564c8e8c7e0 con 0x5564c915fd30 2024-05-12T22:10:14.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.842+0000 7f0723400640 1 --2- 172.21.15.28:0/1600175373 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5564c914c3e0 0x5564c91597e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-12T22:10:14.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.842+0000 7f0722a00640 1 --2- 172.21.15.28:0/1600175373 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564c914cd90 0x5564c9159d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:10:14.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.842+0000 7f0723e00640 1 --2- 172.21.15.28:0/1600175373 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5564c915fd30 0x5564c915f500 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f071c00fec0 tx=0x7f071c004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f41edf51dd37503f server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:10:14.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.842+0000 7f070cc00640 10 client.0 ms_handle_connect on v2:172.21.15.86:3300/0 2024-05-12T22:10:14.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.843+0000 7f070cc00640 1 -- 172.21.15.28:0/1600175373 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f071c02e060 con 0x5564c915fd30 2024-05-12T22:10:14.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.843+0000 7f072cf7e740 1 -- 172.21.15.28:0/1600175373 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0714052030 con 0x5564c915fd30 2024-05-12T22:10:14.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.843+0000 7f070cc00640 1 -- 172.21.15.28:0/1600175373 <== mon.2 v2:172.21.15.86:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f071c018030 con 0x5564c915fd30 2024-05-12T22:10:14.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.843+0000 7f070cc00640 1 -- 172.21.15.28:0/1600175373 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f071c012020 con 0x5564c915fd30 2024-05-12T22:10:14.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.844+0000 7f072cf7e740 10 client.25658 fetch_fsmap learned FSMap version 24 2024-05-12T22:10:14.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.844+0000 7f072cf7e740 1 -- 172.21.15.28:0/1600175373 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5564c8eecf70 con 0x5564c915fd30 2024-05-12T22:10:14.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.844+0000 7f070cc00640 1 -- 172.21.15.28:0/1600175373 <== mon.2 v2:172.21.15.86:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0714052030 con 0x5564c915fd30 2024-05-12T22:10:14.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.844+0000 7f070cc00640 1 -- 172.21.15.28:0/1600175373 <== mon.2 v2:172.21.15.86:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f071c0118b0 con 0x5564c915fd30 2024-05-12T22:10:14.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.844+0000 7f072cf7e740 10 client.25658 fetch_fsmap finished waiting for FSMap version 24 2024-05-12T22:10:14.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.844+0000 7f072cf7e740 10 client.25658 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:10:14.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.844+0000 7f072cf7e740 20 client.25658 populate_metadata read hostname 'smithi028' 2024-05-12T22:10:14.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.845+0000 7f072cf7e740 1 --2- 172.21.15.28:0/1600175373 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5564c9167710 0x5564c9169b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:14.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.845+0000 7f072cf7e740 4 client.25658 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-12T22:10:14.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.845+0000 7f072cf7e740 1 -- 172.21.15.28:0/1600175373 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5564c90c43a0 con 0x5564c9167710 2024-05-12T22:10:14.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.845+0000 7f0723400640 1 --2- 172.21.15.28:0/1600175373 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5564c9167710 0x5564c9169b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:14.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.846+0000 7f0723400640 1 --2- 172.21.15.28:0/1600175373 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5564c9167710 0x5564c9169b80 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=69b6072834493e3d server_cookie=69b6ed34aa44235b in_seq=0 out_seq=0 2024-05-12T22:10:14.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.846+0000 7f070cc00640 10 client.25658 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:10:14.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.847+0000 7f070cc00640 1 -- 172.21.15.28:0/1600175373 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5564c90c43a0 con 0x5564c9167710 2024-05-12T22:10:14.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.847+0000 7f070cc00640 10 client.25658 handle_command_reply: tid=0 2024-05-12T22:10:14.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.847+0000 7f072cf7e740 1 client.25658 shutdown 2024-05-12T22:10:14.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.847+0000 7f072cf7e740 1 -- 172.21.15.28:0/1600175373 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5564c915fd30 msgr2=0x5564c915f500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:10:14.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.847+0000 7f072cf7e740 1 --2- 172.21.15.28:0/1600175373 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5564c915fd30 0x5564c915f500 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f071c00fec0 tx=0x7f071c004ab0 comp rx=0 tx=0).stop 2024-05-12T22:10:14.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.847+0000 7f072cf7e740 1 -- 172.21.15.28:0/1600175373 shutdown_connections 2024-05-12T22:10:14.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.847+0000 7f072cf7e740 1 --2- 172.21.15.28:0/1600175373 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5564c9167710 0x5564c9169b80 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:14.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.847+0000 7f072cf7e740 1 --2- 172.21.15.28:0/1600175373 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5564c915fd30 0x5564c915f500 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:14.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.847+0000 7f072cf7e740 1 --2- 172.21.15.28:0/1600175373 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564c914cd90 0x5564c9159d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:14.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.847+0000 7f070cc00640 0 client.25658 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:10:14.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.847+0000 7f072cf7e740 1 --2- 172.21.15.28:0/1600175373 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5564c914c3e0 0x5564c91597e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:14.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.847+0000 7f072cf7e740 1 -- 172.21.15.28:0/1600175373 >> 172.21.15.28:0/1600175373 conn(0x5564c91485b0 msgr2=0x5564c914e630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:10:14.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.847+0000 7f072cf7e740 1 -- 172.21.15.28:0/1600175373 shutdown_connections 2024-05-12T22:10:14.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.847+0000 7f072cf7e740 1 -- 172.21.15.28:0/1600175373 wait complete. 2024-05-12T22:10:14.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.848+0000 7f072cf7e740 20 client.25658 trim_cache size 0 max 16384 2024-05-12T22:10:14.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.884+0000 7f072cf7e740 1 Processor -- start 2024-05-12T22:10:14.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.885+0000 7f072cf7e740 1 -- start start 2024-05-12T22:10:14.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.885+0000 7f072cf7e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564c914c3e0 0x5564c91475b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:14.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.885+0000 7f072cf7e740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5564c914cd90 0x5564c916f450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:14.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.885+0000 7f072cf7e740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5564c915fd30 0x5564c916f990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:14.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.885+0000 7f072cf7e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f07241ad9d0 con 0x5564c914c3e0 2024-05-12T22:10:14.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.885+0000 7f072cf7e740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f07241ab2a0 con 0x5564c914cd90 2024-05-12T22:10:14.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.885+0000 7f072cf7e740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f07241a90f0 con 0x5564c915fd30 2024-05-12T22:10:14.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.885+0000 7f0723400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564c914c3e0 0x5564c91475b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:14.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.886+0000 7f0723e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5564c915fd30 0x5564c916f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:14.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.886+0000 7f0723400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564c914c3e0 0x5564c91475b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42390/0 (socket says 172.21.15.28:42390) 2024-05-12T22:10:14.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.886+0000 7f0723400640 1 -- 172.21.15.28:0/4236048792 learned_addr learned my addr 172.21.15.28:0/4236048792 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:10:14.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.886+0000 7f0722a00640 1 --2- 172.21.15.28:0/4236048792 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5564c914cd90 0x5564c916f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:14.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.886+0000 7f0723e00640 1 -- 172.21.15.28:0/4236048792 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5564c914cd90 msgr2=0x5564c916f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:10:14.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.886+0000 7f0723e00640 1 --2- 172.21.15.28:0/4236048792 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5564c914cd90 0x5564c916f450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:14.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.886+0000 7f0723e00640 1 -- 172.21.15.28:0/4236048792 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564c914c3e0 msgr2=0x5564c91475b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:10:14.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.886+0000 7f0723e00640 1 --2- 172.21.15.28:0/4236048792 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564c914c3e0 0x5564c91475b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:14.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.886+0000 7f0723e00640 1 -- 172.21.15.28:0/4236048792 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5564c8d84170 con 0x5564c915fd30 2024-05-12T22:10:14.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.886+0000 7f0723400640 1 --2- 172.21.15.28:0/4236048792 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564c914c3e0 0x5564c91475b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:10:14.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.887+0000 7f0723e00640 1 --2- 172.21.15.28:0/4236048792 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5564c915fd30 0x5564c916f990 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f071c011f70 tx=0x7f071c005c50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5cfd2eb8701c230 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:10:14.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.887+0000 7f070f400640 1 -- 172.21.15.28:0/4236048792 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f071c0121c0 con 0x5564c915fd30 2024-05-12T22:10:14.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.887+0000 7f070f400640 1 -- 172.21.15.28:0/4236048792 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f071c011920 con 0x5564c915fd30 2024-05-12T22:10:14.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.887+0000 7f070f400640 1 -- 172.21.15.28:0/4236048792 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f071c01ee60 con 0x5564c915fd30 2024-05-12T22:10:14.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.888+0000 7f072cf7e740 1 -- 172.21.15.28:0/4236048792 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5564c915fd30 msgr2=0x5564c916f990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:10:14.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.888+0000 7f072cf7e740 1 --2- 172.21.15.28:0/4236048792 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5564c915fd30 0x5564c916f990 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f071c011f70 tx=0x7f071c005c50 comp rx=0 tx=0).stop 2024-05-12T22:10:14.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.888+0000 7f072cf7e740 1 -- 172.21.15.28:0/4236048792 shutdown_connections 2024-05-12T22:10:14.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.888+0000 7f072cf7e740 1 --2- 172.21.15.28:0/4236048792 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5564c915fd30 0x5564c916f990 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:14.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.888+0000 7f072cf7e740 1 --2- 172.21.15.28:0/4236048792 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5564c914cd90 0x5564c916f450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:14.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.888+0000 7f072cf7e740 1 --2- 172.21.15.28:0/4236048792 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564c914c3e0 0x5564c91475b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:14.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.888+0000 7f072cf7e740 1 -- 172.21.15.28:0/4236048792 >> 172.21.15.28:0/4236048792 conn(0x5564c91485b0 msgr2=0x5564c9146ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:10:14.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.888+0000 7f072cf7e740 1 -- 172.21.15.28:0/4236048792 shutdown_connections 2024-05-12T22:10:14.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.888+0000 7f072cf7e740 1 -- 172.21.15.28:0/4236048792 wait complete. 2024-05-12T22:10:14.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.889+0000 7f072cf7e740 1 -- start start 2024-05-12T22:10:14.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.889+0000 7f072cf7e740 1 Processor -- start 2024-05-12T22:10:14.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.890+0000 7f072cf7e740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5564c914c3e0 0x5564c9151810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:14.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.890+0000 7f072cf7e740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5564c914cd90 0x5564c9151d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:14.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.890+0000 7f072cf7e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564c915fd30 0x5564c9152290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:14.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.890+0000 7f072cf7e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f07241a9760 con 0x5564c915fd30 2024-05-12T22:10:14.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.890+0000 7f072cf7e740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f07241aabd0 con 0x5564c914cd90 2024-05-12T22:10:14.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.890+0000 7f072cf7e740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f07241a9e30 con 0x5564c914c3e0 2024-05-12T22:10:14.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.890+0000 7f072cf7e740 10 client.0 _ll_register_callbacks cb 0x7f071c012f90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:10:14.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.890+0000 7f0723400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5564c914c3e0 0x5564c9151810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:14.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.890+0000 7f0723400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5564c914c3e0 0x5564c9151810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:43164/0 (socket says 172.21.15.28:43164) 2024-05-12T22:10:14.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.890+0000 7f0723400640 1 -- 172.21.15.28:0/2284471168 learned_addr learned my addr 172.21.15.28:0/2284471168 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:10:14.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.890+0000 7f0723e00640 1 --2- 172.21.15.28:0/2284471168 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564c915fd30 0x5564c9152290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:14.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.890+0000 7f0722a00640 1 --2- 172.21.15.28:0/2284471168 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5564c914cd90 0x5564c9151d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:14.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.891+0000 7f0723400640 1 -- 172.21.15.28:0/2284471168 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5564c914cd90 msgr2=0x5564c9151d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:10:14.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.891+0000 7f0723400640 1 --2- 172.21.15.28:0/2284471168 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5564c914cd90 0x5564c9151d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:14.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.891+0000 7f0723400640 1 -- 172.21.15.28:0/2284471168 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564c915fd30 msgr2=0x5564c9152290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:10:14.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.891+0000 7f0723400640 1 --2- 172.21.15.28:0/2284471168 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564c915fd30 0x5564c9152290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:14.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.891+0000 7f0723400640 1 -- 172.21.15.28:0/2284471168 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5564c91cc850 con 0x5564c914c3e0 2024-05-12T22:10:14.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.891+0000 7f0722a00640 1 --2- 172.21.15.28:0/2284471168 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5564c914cd90 0x5564c9151d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:10:14.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.891+0000 7f0723e00640 1 --2- 172.21.15.28:0/2284471168 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564c915fd30 0x5564c9152290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:10:14.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.891+0000 7f0723400640 1 --2- 172.21.15.28:0/2284471168 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5564c914c3e0 0x5564c9151810 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x5564c914dd70 tx=0x7f071402ab50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8106d5454b1e44d6 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:10:14.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.891+0000 7f070cc00640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-12T22:10:14.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.892+0000 7f070cc00640 1 -- 172.21.15.28:0/2284471168 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0714004220 con 0x5564c914c3e0 2024-05-12T22:10:14.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.892+0000 7f070cc00640 1 -- 172.21.15.28:0/2284471168 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f07140043c0 con 0x5564c914c3e0 2024-05-12T22:10:14.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.892+0000 7f072cf7e740 1 -- 172.21.15.28:0/2284471168 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f071c011920 con 0x5564c914c3e0 2024-05-12T22:10:14.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.892+0000 7f070cc00640 1 -- 172.21.15.28:0/2284471168 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f07140303a0 con 0x5564c914c3e0 2024-05-12T22:10:14.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.892+0000 7f072cf7e740 10 client.25621 fetch_fsmap learned FSMap version 24 2024-05-12T22:10:14.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.892+0000 7f072cf7e740 1 -- 172.21.15.28:0/2284471168 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5564c91cdb70 con 0x5564c914c3e0 2024-05-12T22:10:14.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.892+0000 7f070cc00640 1 -- 172.21.15.28:0/2284471168 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f071c011920 con 0x5564c914c3e0 2024-05-12T22:10:14.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.893+0000 7f070cc00640 1 -- 172.21.15.28:0/2284471168 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f07140035f0 con 0x5564c914c3e0 2024-05-12T22:10:14.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.893+0000 7f072cf7e740 10 client.25621 fetch_fsmap finished waiting for FSMap version 24 2024-05-12T22:10:14.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.893+0000 7f072cf7e740 10 client.25621 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:10:14.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.893+0000 7f072cf7e740 20 client.25621 populate_metadata read hostname 'smithi028' 2024-05-12T22:10:14.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.893+0000 7f072cf7e740 1 --2- 172.21.15.28:0/2284471168 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5564c91cdf70 0x5564c91d0360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:14.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.893+0000 7f072cf7e740 4 client.25621 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-12T22:10:14.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.893+0000 7f072cf7e740 1 -- 172.21.15.28:0/2284471168 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5564c90bdef0 con 0x5564c91cdf70 2024-05-12T22:10:14.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.894+0000 7f0722a00640 1 --2- 172.21.15.28:0/2284471168 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5564c91cdf70 0x5564c91d0360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:14.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.894+0000 7f0722a00640 1 --2- 172.21.15.28:0/2284471168 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5564c91cdf70 0x5564c91d0360 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=63751a3f2c498179 server_cookie=5a1ebd07c7a2333 in_seq=0 out_seq=0 2024-05-12T22:10:14.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.894+0000 7f070cc00640 10 client.25621 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:10:14.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.895+0000 7f070cc00640 1 -- 172.21.15.28:0/2284471168 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5564c90bdef0 con 0x5564c91cdf70 2024-05-12T22:10:14.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.895+0000 7f070cc00640 10 client.25621 handle_command_reply: tid=0 2024-05-12T22:10:14.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.896+0000 7f072cf7e740 1 client.25621 shutdown 2024-05-12T22:10:14.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.896+0000 7f072cf7e740 1 -- 172.21.15.28:0/2284471168 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5564c914c3e0 msgr2=0x5564c9151810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:10:14.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.896+0000 7f072cf7e740 1 --2- 172.21.15.28:0/2284471168 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5564c914c3e0 0x5564c9151810 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x5564c914dd70 tx=0x7f071402ab50 comp rx=0 tx=0).stop 2024-05-12T22:10:14.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.896+0000 7f072cf7e740 1 -- 172.21.15.28:0/2284471168 shutdown_connections 2024-05-12T22:10:14.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.896+0000 7f072cf7e740 1 --2- 172.21.15.28:0/2284471168 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5564c91cdf70 0x5564c91d0360 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:14.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.896+0000 7f072cf7e740 1 --2- 172.21.15.28:0/2284471168 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564c915fd30 0x5564c9152290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:14.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.896+0000 7f072cf7e740 1 --2- 172.21.15.28:0/2284471168 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5564c914cd90 0x5564c9151d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:14.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.896+0000 7f070cc00640 0 client.25621 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:10:14.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.896+0000 7f072cf7e740 1 --2- 172.21.15.28:0/2284471168 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5564c914c3e0 0x5564c9151810 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:14.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.896+0000 7f072cf7e740 1 -- 172.21.15.28:0/2284471168 >> 172.21.15.28:0/2284471168 conn(0x5564c91485b0 msgr2=0x5564c9147730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:10:14.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.896+0000 7f072cf7e740 1 -- 172.21.15.28:0/2284471168 shutdown_connections 2024-05-12T22:10:14.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.896+0000 7f072cf7e740 1 -- 172.21.15.28:0/2284471168 wait complete. 2024-05-12T22:10:14.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.897+0000 7f072cf7e740 20 client.25621 trim_cache size 0 max 16384 2024-05-12T22:10:14.898 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-05-12T22:10:14.898 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-05-12T22:10:14.899 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "62279362-d1ad-4983-8131-1dd4c341cf05", 2024-05-12T22:10:14.899 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-05-12T22:10:14.899 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-05-12T22:10:14.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.898+0000 7f0728c00640 1 -- 172.21.15.28:0/2852154917 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f07080a2f60 msgr2=0x7f07080a5420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:10:14.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.898+0000 7f0728c00640 1 --2- 172.21.15.28:0/2852154917 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f07080a2f60 0x7f07080a5420 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f071c009fd0 tx=0x7f071c009500 comp rx=0 tx=0).stop 2024-05-12T22:10:14.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.898+0000 7f0728c00640 1 -- 172.21.15.28:0/2852154917 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f07241031b0 msgr2=0x7f0724100a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:10:14.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.898+0000 7f0728c00640 1 --2- 172.21.15.28:0/2852154917 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f07241031b0 0x7f0724100a20 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f071000e910 tx=0x7f071000edd0 comp rx=0 tx=0).stop 2024-05-12T22:10:14.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.898+0000 7f0728c00640 1 -- 172.21.15.28:0/2852154917 shutdown_connections 2024-05-12T22:10:14.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.898+0000 7f0728c00640 1 --2- 172.21.15.28:0/2852154917 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f07080a2f60 0x7f07080a5420 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:14.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.898+0000 7f0728c00640 1 --2- 172.21.15.28:0/2852154917 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f072410bcb0 0x7f0724100f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:14.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.898+0000 7f0728c00640 1 --2- 172.21.15.28:0/2852154917 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f07241031b0 0x7f0724100a20 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:14.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.898+0000 7f0728c00640 1 --2- 172.21.15.28:0/2852154917 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0724102890 0x7f07241004e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:14.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.898+0000 7f0728c00640 1 -- 172.21.15.28:0/2852154917 >> 172.21.15.28:0/2852154917 conn(0x7f07240fc0b0 msgr2=0x7f07240fd8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:10:14.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.899+0000 7f0728c00640 1 -- 172.21.15.28:0/2852154917 shutdown_connections 2024-05-12T22:10:14.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:14.899+0000 7f0728c00640 1 -- 172.21.15.28:0/2852154917 wait complete. 2024-05-12T22:10:14.924 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-12T22:10:14.924 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-12T22:10:15.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.130+0000 7f947fe00640 1 -- 172.21.15.28:0/277239338 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94801016d0 msgr2=0x7f9480101b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:10:15.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.130+0000 7f947fe00640 1 --2- 172.21.15.28:0/277239338 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94801016d0 0x7f9480101b50 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f946c009930 tx=0x7f946c02a420 comp rx=0 tx=0).stop 2024-05-12T22:10:15.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.130+0000 7f947fe00640 1 -- 172.21.15.28:0/277239338 shutdown_connections 2024-05-12T22:10:15.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.130+0000 7f947fe00640 1 --2- 172.21.15.28:0/277239338 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9480102090 0x7f948010e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:15.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.130+0000 7f947fe00640 1 --2- 172.21.15.28:0/277239338 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94801016d0 0x7f9480101b50 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:15.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.130+0000 7f947fe00640 1 --2- 172.21.15.28:0/277239338 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f94801076e0 0x7f9480107ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:15.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.130+0000 7f947fe00640 1 -- 172.21.15.28:0/277239338 >> 172.21.15.28:0/277239338 conn(0x7f94800fd200 msgr2=0x7f94800ff620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:10:15.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.130+0000 7f947fe00640 1 -- 172.21.15.28:0/277239338 shutdown_connections 2024-05-12T22:10:15.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.130+0000 7f947fe00640 1 -- 172.21.15.28:0/277239338 wait complete. 2024-05-12T22:10:15.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.131+0000 7f947fe00640 1 Processor -- start 2024-05-12T22:10:15.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.131+0000 7f947fe00640 1 -- start start 2024-05-12T22:10:15.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.132+0000 7f947fe00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f94801016d0 0x7f9480073b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:15.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.132+0000 7f947fe00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9480102090 0x7f9480074070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:15.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.132+0000 7f947fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94801076e0 0x7f94800745b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:15.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.132+0000 7f947fe00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9480111b40 con 0x7f94801076e0 2024-05-12T22:10:15.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.132+0000 7f947fe00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f94801119c0 con 0x7f94801016d0 2024-05-12T22:10:15.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.132+0000 7f947fe00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f9480111cc0 con 0x7f9480102090 2024-05-12T22:10:15.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.132+0000 7f947f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94801076e0 0x7f94800745b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:15.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.132+0000 7f947f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94801076e0 0x7f94800745b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42432/0 (socket says 172.21.15.28:42432) 2024-05-12T22:10:15.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.132+0000 7f947f400640 1 -- 172.21.15.28:0/3872808683 learned_addr learned my addr 172.21.15.28:0/3872808683 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:10:15.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.132+0000 7f947ea00640 1 --2- 172.21.15.28:0/3872808683 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f94801016d0 0x7f9480073b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:15.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.132+0000 7f947e000640 1 --2- 172.21.15.28:0/3872808683 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9480102090 0x7f9480074070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:15.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.133+0000 7f947ea00640 1 -- 172.21.15.28:0/3872808683 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9480102090 msgr2=0x7f9480074070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:10:15.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.133+0000 7f947ea00640 1 --2- 172.21.15.28:0/3872808683 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9480102090 0x7f9480074070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:15.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.133+0000 7f947ea00640 1 -- 172.21.15.28:0/3872808683 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94801076e0 msgr2=0x7f94800745b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:10:15.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.133+0000 7f947ea00640 1 --2- 172.21.15.28:0/3872808683 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94801076e0 0x7f94800745b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:15.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.133+0000 7f947ea00640 1 -- 172.21.15.28:0/3872808683 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94801a6580 con 0x7f94801016d0 2024-05-12T22:10:15.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.133+0000 7f947e000640 1 --2- 172.21.15.28:0/3872808683 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9480102090 0x7f9480074070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-12T22:10:15.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.133+0000 7f947f400640 1 --2- 172.21.15.28:0/3872808683 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94801076e0 0x7f94800745b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:10:15.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.133+0000 7f947ea00640 1 --2- 172.21.15.28:0/3872808683 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f94801016d0 0x7f9480073b30 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f947000e9f0 tx=0x7f947000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:10:15.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.133+0000 7f9477400640 1 -- 172.21.15.28:0/3872808683 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f947000ccc0 con 0x7f94801016d0 2024-05-12T22:10:15.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.133+0000 7f9477400640 1 -- 172.21.15.28:0/3872808683 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9470010910 con 0x7f94801016d0 2024-05-12T22:10:15.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.133+0000 7f9477400640 1 -- 172.21.15.28:0/3872808683 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9470018960 con 0x7f94801016d0 2024-05-12T22:10:15.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.134+0000 7f947fe00640 1 -- 172.21.15.28:0/3872808683 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94801a6870 con 0x7f94801016d0 2024-05-12T22:10:15.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.134+0000 7f947fe00640 1 -- 172.21.15.28:0/3872808683 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94801a6d70 con 0x7f94801016d0 2024-05-12T22:10:15.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.137+0000 7f94883d5740 1 Processor -- start 2024-05-12T22:10:15.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.137+0000 7f94883d5740 1 -- start start 2024-05-12T22:10:15.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.138+0000 7f94883d5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5561e8fec5e0 0x5561e8fec9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:15.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.138+0000 7f94883d5740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5561e8fecf00 0x5561e8ff7090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:15.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.138+0000 7f94883d5740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5561e8ff75d0 0x5561e8ff99c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:15.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.138+0000 7f94883d5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5561e8fac2a0 con 0x5561e8fec5e0 2024-05-12T22:10:15.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.138+0000 7f94883d5740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x5561e8f48b90 con 0x5561e8fecf00 2024-05-12T22:10:15.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.138+0000 7f94883d5740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5561e8ef05b0 con 0x5561e8ff75d0 2024-05-12T22:10:15.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.139+0000 7f947f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5561e8fec5e0 0x5561e8fec9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:15.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.139+0000 7f947f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5561e8fec5e0 0x5561e8fec9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42446/0 (socket says 172.21.15.28:42446) 2024-05-12T22:10:15.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.139+0000 7f947f400640 1 -- 172.21.15.28:0/649622944 learned_addr learned my addr 172.21.15.28:0/649622944 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:10:15.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.139+0000 7f947f400640 1 -- 172.21.15.28:0/649622944 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5561e8ff75d0 msgr2=0x5561e8ff99c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-12T22:10:15.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.139+0000 7f947e000640 1 --2- 172.21.15.28:0/649622944 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5561e8ff75d0 0x5561e8ff99c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:15.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.139+0000 7f947f400640 1 --2- 172.21.15.28:0/649622944 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5561e8ff75d0 0x5561e8ff99c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:15.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.139+0000 7f947f400640 1 -- 172.21.15.28:0/649622944 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5561e8fecf00 msgr2=0x5561e8ff7090 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-12T22:10:15.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.139+0000 7f947f400640 1 --2- 172.21.15.28:0/649622944 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5561e8fecf00 0x5561e8ff7090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:15.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.139+0000 7f947f400640 1 -- 172.21.15.28:0/649622944 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5561e8d267d0 con 0x5561e8fec5e0 2024-05-12T22:10:15.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.139+0000 7f947e000640 1 --2- 172.21.15.28:0/649622944 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5561e8ff75d0 0x5561e8ff99c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-12T22:10:15.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.140+0000 7f947f400640 1 --2- 172.21.15.28:0/649622944 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5561e8fec5e0 0x5561e8fec9c0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f94780118b0 tx=0x7f9478011d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8498d87d8af92055 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:10:15.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.140+0000 7f946b400640 1 -- 172.21.15.28:0/649622944 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f947801e070 con 0x5561e8fec5e0 2024-05-12T22:10:15.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.140+0000 7f946b400640 1 -- 172.21.15.28:0/649622944 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9478011310 con 0x5561e8fec5e0 2024-05-12T22:10:15.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.140+0000 7f946b400640 1 -- 172.21.15.28:0/649622944 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9478019e10 con 0x5561e8fec5e0 2024-05-12T22:10:15.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.140+0000 7f9477400640 1 -- 172.21.15.28:0/3872808683 <== mon.2 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 26) ==== 138117+0+0 (secure 0 0 0) 0x7f9470010ac0 con 0x7f94801016d0 2024-05-12T22:10:15.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.141+0000 7f94883d5740 1 -- 172.21.15.28:0/649622944 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5561e8fec5e0 msgr2=0x5561e8fec9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:10:15.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.141+0000 7f94883d5740 1 --2- 172.21.15.28:0/649622944 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5561e8fec5e0 0x5561e8fec9c0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f94780118b0 tx=0x7f9478011d70 comp rx=0 tx=0).stop 2024-05-12T22:10:15.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.142+0000 7f94883d5740 1 -- 172.21.15.28:0/649622944 shutdown_connections 2024-05-12T22:10:15.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.142+0000 7f94883d5740 1 --2- 172.21.15.28:0/649622944 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5561e8ff75d0 0x5561e8ff99c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:15.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.142+0000 7f94883d5740 1 --2- 172.21.15.28:0/649622944 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5561e8fecf00 0x5561e8ff7090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:15.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.142+0000 7f94883d5740 1 --2- 172.21.15.28:0/649622944 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5561e8fec5e0 0x5561e8fec9c0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:15.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.142+0000 7f94883d5740 1 -- 172.21.15.28:0/649622944 >> 172.21.15.28:0/649622944 conn(0x5561e8fe8900 msgr2=0x5561e8fe8ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:10:15.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.142+0000 7f94883d5740 1 -- 172.21.15.28:0/649622944 shutdown_connections 2024-05-12T22:10:15.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.142+0000 7f94883d5740 1 -- 172.21.15.28:0/649622944 wait complete. 2024-05-12T22:10:15.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.142+0000 7f94883d5740 1 -- start start 2024-05-12T22:10:15.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.143+0000 7f94883d5740 1 Processor -- start 2024-05-12T22:10:15.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.143+0000 7f9477400640 1 --2- 172.21.15.28:0/3872808683 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f94640a2e40 0x7f94640a5300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:15.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.143+0000 7f9477400640 1 -- 172.21.15.28:0/3872808683 <== mon.2 v2:172.21.15.86:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f94700cfa60 con 0x7f94801016d0 2024-05-12T22:10:15.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.143+0000 7f94883d5740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5561e8fec5e0 0x5561e8ff6f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:15.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.144+0000 7f947f400640 1 --2- 172.21.15.28:0/3872808683 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f94640a2e40 0x7f94640a5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:15.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.144+0000 7f94883d5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5561e8fecf00 0x5561e8ff4220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:15.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.144+0000 7f94883d5740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5561e8ff75d0 0x5561e8ff4780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:15.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.144+0000 7f94883d5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5561e8f039f0 con 0x5561e8fecf00 2024-05-12T22:10:15.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.144+0000 7f94883d5740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x5561e8f44b50 con 0x5561e8ff75d0 2024-05-12T22:10:15.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.144+0000 7f94883d5740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5561e8ffa3c0 con 0x5561e8fec5e0 2024-05-12T22:10:15.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.144+0000 7f94883d5740 10 client.0 _ll_register_callbacks cb 0x5561e8c5bbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:10:15.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.144+0000 7f947f400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5561e8ff75d0 0x5561e8ff4780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:15.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.144+0000 7f947f400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5561e8ff75d0 0x5561e8ff4780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:54776/0 (socket says 172.21.15.28:54776) 2024-05-12T22:10:15.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.144+0000 7f947f400640 1 -- 172.21.15.28:0/1665800904 learned_addr learned my addr 172.21.15.28:0/1665800904 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:10:15.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.144+0000 7f947e000640 1 --2- 172.21.15.28:0/1665800904 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5561e8fecf00 0x5561e8ff4220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:15.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.144+0000 7f947ea00640 1 --2- 172.21.15.28:0/1665800904 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5561e8fec5e0 0x5561e8ff6f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:15.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.145+0000 7f947f400640 1 --2- 172.21.15.28:0/3872808683 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f94640a2e40 0x7f94640a5300 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f9478011880 tx=0x7f9478019970 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:10:15.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.145+0000 7f947f400640 1 -- 172.21.15.28:0/1665800904 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5561e8fec5e0 msgr2=0x5561e8ff6f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:10:15.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.145+0000 7f947f400640 1 --2- 172.21.15.28:0/1665800904 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5561e8fec5e0 0x5561e8ff6f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:15.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.145+0000 7f947f400640 1 -- 172.21.15.28:0/1665800904 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5561e8fecf00 msgr2=0x5561e8ff4220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:10:15.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.145+0000 7f947f400640 1 --2- 172.21.15.28:0/1665800904 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5561e8fecf00 0x5561e8ff4220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:15.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.145+0000 7f947f400640 1 -- 172.21.15.28:0/1665800904 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5561e8c5c3f0 con 0x5561e8ff75d0 2024-05-12T22:10:15.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.145+0000 7f947e000640 1 --2- 172.21.15.28:0/1665800904 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5561e8fecf00 0x5561e8ff4220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:10:15.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.145+0000 7f947f400640 1 --2- 172.21.15.28:0/1665800904 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5561e8ff75d0 0x5561e8ff4780 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f94780118b0 tx=0x7f9478009500 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4bd587efb04a2e1d server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:10:15.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.145+0000 7f945fe00640 10 client.0 ms_handle_connect on v2:172.21.15.86:3300/0 2024-05-12T22:10:15.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.146+0000 7f945fe00640 1 -- 172.21.15.28:0/1665800904 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f947802b070 con 0x5561e8ff75d0 2024-05-12T22:10:15.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.146+0000 7f945fe00640 1 -- 172.21.15.28:0/1665800904 <== mon.2 v2:172.21.15.86:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f9478005040 con 0x5561e8ff75d0 2024-05-12T22:10:15.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.146+0000 7f94883d5740 1 -- 172.21.15.28:0/1665800904 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9478011310 con 0x5561e8ff75d0 2024-05-12T22:10:15.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.146+0000 7f945fe00640 1 -- 172.21.15.28:0/1665800904 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f947801e040 con 0x5561e8ff75d0 2024-05-12T22:10:15.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.146+0000 7f94883d5740 10 client.25673 fetch_fsmap learned FSMap version 24 2024-05-12T22:10:15.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.146+0000 7f94883d5740 1 -- 172.21.15.28:0/1665800904 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5561e8c1c0f0 con 0x5561e8ff75d0 2024-05-12T22:10:15.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.146+0000 7f945fe00640 1 -- 172.21.15.28:0/1665800904 <== mon.2 v2:172.21.15.86:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9478011310 con 0x5561e8ff75d0 2024-05-12T22:10:15.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.147+0000 7f945fe00640 1 -- 172.21.15.28:0/1665800904 <== mon.2 v2:172.21.15.86:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f947802e070 con 0x5561e8ff75d0 2024-05-12T22:10:15.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.147+0000 7f94883d5740 10 client.25673 fetch_fsmap finished waiting for FSMap version 24 2024-05-12T22:10:15.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.147+0000 7f94883d5740 10 client.25673 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:10:15.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.147+0000 7f94883d5740 20 client.25673 populate_metadata read hostname 'smithi028' 2024-05-12T22:10:15.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.147+0000 7f94883d5740 1 --2- 172.21.15.28:0/1665800904 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5561e9007000 0x5561e90093f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:15.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.147+0000 7f94883d5740 4 client.25673 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-12T22:10:15.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.147+0000 7f94883d5740 1 -- 172.21.15.28:0/1665800904 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5561e8f562c0 con 0x5561e9007000 2024-05-12T22:10:15.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.147+0000 7f947ea00640 1 --2- 172.21.15.28:0/1665800904 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5561e9007000 0x5561e90093f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:15.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.148+0000 7f947ea00640 1 --2- 172.21.15.28:0/1665800904 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5561e9007000 0x5561e90093f0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f5224b44e871de87 server_cookie=c759d619988a4cd8 in_seq=0 out_seq=0 2024-05-12T22:10:15.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.148+0000 7f945fe00640 10 client.25673 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:10:15.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.149+0000 7f945fe00640 1 -- 172.21.15.28:0/1665800904 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5561e8f562c0 con 0x5561e9007000 2024-05-12T22:10:15.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.149+0000 7f945fe00640 10 client.25673 handle_command_reply: tid=0 2024-05-12T22:10:15.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.149+0000 7f94883d5740 1 client.25673 shutdown 2024-05-12T22:10:15.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.149+0000 7f94883d5740 1 -- 172.21.15.28:0/1665800904 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5561e8ff75d0 msgr2=0x5561e8ff4780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:10:15.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.149+0000 7f94883d5740 1 --2- 172.21.15.28:0/1665800904 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5561e8ff75d0 0x5561e8ff4780 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f94780118b0 tx=0x7f9478009500 comp rx=0 tx=0).stop 2024-05-12T22:10:15.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.149+0000 7f94883d5740 1 -- 172.21.15.28:0/1665800904 shutdown_connections 2024-05-12T22:10:15.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.149+0000 7f94883d5740 1 --2- 172.21.15.28:0/1665800904 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5561e9007000 0x5561e90093f0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:15.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.149+0000 7f94883d5740 1 --2- 172.21.15.28:0/1665800904 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5561e8ff75d0 0x5561e8ff4780 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:15.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.149+0000 7f94883d5740 1 --2- 172.21.15.28:0/1665800904 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5561e8fecf00 0x5561e8ff4220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:15.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.149+0000 7f94883d5740 1 --2- 172.21.15.28:0/1665800904 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5561e8fec5e0 0x5561e8ff6f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:15.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.149+0000 7f94883d5740 1 -- 172.21.15.28:0/1665800904 >> 172.21.15.28:0/1665800904 conn(0x5561e8fe8900 msgr2=0x5561e8fdf330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:10:15.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.149+0000 7f945fe00640 0 client.25673 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:10:15.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.150+0000 7f94883d5740 1 -- 172.21.15.28:0/1665800904 shutdown_connections 2024-05-12T22:10:15.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.150+0000 7f94883d5740 1 -- 172.21.15.28:0/1665800904 wait complete. 2024-05-12T22:10:15.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.150+0000 7f94883d5740 20 client.25673 trim_cache size 0 max 16384 2024-05-12T22:10:15.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.187+0000 7f94883d5740 1 Processor -- start 2024-05-12T22:10:15.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.187+0000 7f94883d5740 1 -- start start 2024-05-12T22:10:15.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.187+0000 7f94883d5740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5561e8fec5e0 0x5561e8fdf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:15.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.187+0000 7f94883d5740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5561e8fecf00 0x5561e8ff6a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:15.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.187+0000 7f94883d5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5561e8ff75d0 0x5561e900eb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:15.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.187+0000 7f94883d5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f94801ada60 con 0x5561e8ff75d0 2024-05-12T22:10:15.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.187+0000 7f94883d5740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f94801ab330 con 0x5561e8fec5e0 2024-05-12T22:10:15.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.187+0000 7f94883d5740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f94801a9180 con 0x5561e8fecf00 2024-05-12T22:10:15.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.188+0000 7f947ea00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5561e8fec5e0 0x5561e8fdf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:15.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.188+0000 7f947ea00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5561e8fec5e0 0x5561e8fdf1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:54778/0 (socket says 172.21.15.28:54778) 2024-05-12T22:10:15.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.188+0000 7f947ea00640 1 -- 172.21.15.28:0/3104440551 learned_addr learned my addr 172.21.15.28:0/3104440551 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:10:15.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.188+0000 7f947e000640 1 --2- 172.21.15.28:0/3104440551 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5561e8fecf00 0x5561e8ff6a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:15.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.188+0000 7f947f400640 1 --2- 172.21.15.28:0/3104440551 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5561e8ff75d0 0x5561e900eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:15.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.188+0000 7f947ea00640 1 -- 172.21.15.28:0/3104440551 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5561e8fecf00 msgr2=0x5561e8ff6a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:10:15.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.188+0000 7f947ea00640 1 --2- 172.21.15.28:0/3104440551 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5561e8fecf00 0x5561e8ff6a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:15.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.188+0000 7f947ea00640 1 -- 172.21.15.28:0/3104440551 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5561e8ff75d0 msgr2=0x5561e900eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:10:15.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.188+0000 7f947ea00640 1 --2- 172.21.15.28:0/3104440551 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5561e8ff75d0 0x5561e900eb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:15.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.188+0000 7f947ea00640 1 -- 172.21.15.28:0/3104440551 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5561e8fe8730 con 0x5561e8fec5e0 2024-05-12T22:10:15.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.188+0000 7f947e000640 1 --2- 172.21.15.28:0/3104440551 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5561e8fecf00 0x5561e8ff6a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:10:15.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.188+0000 7f947f400640 1 --2- 172.21.15.28:0/3104440551 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5561e8ff75d0 0x5561e900eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:10:15.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.189+0000 7f947ea00640 1 --2- 172.21.15.28:0/3104440551 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5561e8fec5e0 0x5561e8fdf1b0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f94700cd0e0 tx=0x7f947000cf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b3ba7f7a63274e75 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:10:15.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.189+0000 7f946aa00640 1 -- 172.21.15.28:0/3104440551 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f947009a070 con 0x5561e8fec5e0 2024-05-12T22:10:15.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.189+0000 7f946aa00640 1 -- 172.21.15.28:0/3104440551 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f94700983e0 con 0x5561e8fec5e0 2024-05-12T22:10:15.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.189+0000 7f946aa00640 1 -- 172.21.15.28:0/3104440551 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9470010690 con 0x5561e8fec5e0 2024-05-12T22:10:15.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.190+0000 7f94883d5740 1 -- 172.21.15.28:0/3104440551 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5561e8fec5e0 msgr2=0x5561e8fdf1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:10:15.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.190+0000 7f94883d5740 1 --2- 172.21.15.28:0/3104440551 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5561e8fec5e0 0x5561e8fdf1b0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f94700cd0e0 tx=0x7f947000cf60 comp rx=0 tx=0).stop 2024-05-12T22:10:15.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.190+0000 7f94883d5740 1 -- 172.21.15.28:0/3104440551 shutdown_connections 2024-05-12T22:10:15.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.190+0000 7f94883d5740 1 --2- 172.21.15.28:0/3104440551 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5561e8ff75d0 0x5561e900eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:15.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.190+0000 7f94883d5740 1 --2- 172.21.15.28:0/3104440551 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5561e8fecf00 0x5561e8ff6a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:15.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.190+0000 7f94883d5740 1 --2- 172.21.15.28:0/3104440551 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5561e8fec5e0 0x5561e8fdf1b0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:15.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.190+0000 7f94883d5740 1 -- 172.21.15.28:0/3104440551 >> 172.21.15.28:0/3104440551 conn(0x5561e8fe8900 msgr2=0x5561e8fdeaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:10:15.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.190+0000 7f94883d5740 1 -- 172.21.15.28:0/3104440551 shutdown_connections 2024-05-12T22:10:15.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.190+0000 7f94883d5740 1 -- 172.21.15.28:0/3104440551 wait complete. 2024-05-12T22:10:15.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.191+0000 7f94883d5740 1 -- start start 2024-05-12T22:10:15.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.191+0000 7f94883d5740 1 Processor -- start 2024-05-12T22:10:15.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.192+0000 7f94883d5740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5561e8fec5e0 0x5561e8ffb390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:15.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.192+0000 7f94883d5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5561e8fecf00 0x5561e8ffb8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:15.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.192+0000 7f94883d5740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5561e8ff75d0 0x5561e906c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:15.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.192+0000 7f94883d5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f94801a97f0 con 0x5561e8fecf00 2024-05-12T22:10:15.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.192+0000 7f94883d5740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f94801aac60 con 0x5561e8ff75d0 2024-05-12T22:10:15.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.192+0000 7f94883d5740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f94801a9ec0 con 0x5561e8fec5e0 2024-05-12T22:10:15.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.192+0000 7f94883d5740 10 client.0 _ll_register_callbacks cb 0x7f9478019760 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:10:15.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.192+0000 7f947e000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5561e8fecf00 0x5561e8ffb8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:15.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.192+0000 7f947ea00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5561e8fec5e0 0x5561e8ffb390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:15.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.192+0000 7f947ea00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5561e8fec5e0 0x5561e8ffb390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:43204/0 (socket says 172.21.15.28:43204) 2024-05-12T22:10:15.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.192+0000 7f947ea00640 1 -- 172.21.15.28:0/2662986128 learned_addr learned my addr 172.21.15.28:0/2662986128 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:10:15.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.192+0000 7f947f400640 1 --2- 172.21.15.28:0/2662986128 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5561e8ff75d0 0x5561e906c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:15.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.192+0000 7f947ea00640 1 -- 172.21.15.28:0/2662986128 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5561e8ff75d0 msgr2=0x5561e906c3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:10:15.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.192+0000 7f947ea00640 1 --2- 172.21.15.28:0/2662986128 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5561e8ff75d0 0x5561e906c3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:15.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.192+0000 7f947ea00640 1 -- 172.21.15.28:0/2662986128 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5561e8fecf00 msgr2=0x5561e8ffb8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:10:15.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.193+0000 7f947ea00640 1 --2- 172.21.15.28:0/2662986128 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5561e8fecf00 0x5561e8ffb8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:15.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.193+0000 7f947ea00640 1 -- 172.21.15.28:0/2662986128 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5561e8ff99c0 con 0x5561e8fec5e0 2024-05-12T22:10:15.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.193+0000 7f947e000640 1 --2- 172.21.15.28:0/2662986128 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5561e8fecf00 0x5561e8ffb8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:10:15.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.193+0000 7f947ea00640 1 --2- 172.21.15.28:0/2662986128 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5561e8fec5e0 0x5561e8ffb390 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f94700a44b0 tx=0x7f947009b5f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=66b159596ffe51f4 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:10:15.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.193+0000 7f945fe00640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-12T22:10:15.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.193+0000 7f945fe00640 1 -- 172.21.15.28:0/2662986128 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f947007e020 con 0x5561e8fec5e0 2024-05-12T22:10:15.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.193+0000 7f945fe00640 1 -- 172.21.15.28:0/2662986128 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f9470010430 con 0x5561e8fec5e0 2024-05-12T22:10:15.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.193+0000 7f94883d5740 1 -- 172.21.15.28:0/2662986128 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f94700983e0 con 0x5561e8fec5e0 2024-05-12T22:10:15.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.194+0000 7f945fe00640 1 -- 172.21.15.28:0/2662986128 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f947009b3c0 con 0x5561e8fec5e0 2024-05-12T22:10:15.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.194+0000 7f94883d5740 10 client.25633 fetch_fsmap learned FSMap version 24 2024-05-12T22:10:15.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.194+0000 7f94883d5740 1 -- 172.21.15.28:0/2662986128 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5561e906d830 con 0x5561e8fec5e0 2024-05-12T22:10:15.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.194+0000 7f945fe00640 1 -- 172.21.15.28:0/2662986128 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f94700983e0 con 0x5561e8fec5e0 2024-05-12T22:10:15.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.194+0000 7f945fe00640 1 -- 172.21.15.28:0/2662986128 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f9470097ac0 con 0x5561e8fec5e0 2024-05-12T22:10:15.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.194+0000 7f94883d5740 10 client.25633 fetch_fsmap finished waiting for FSMap version 24 2024-05-12T22:10:15.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.194+0000 7f94883d5740 10 client.25633 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:10:15.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.194+0000 7f94883d5740 20 client.25633 populate_metadata read hostname 'smithi028' 2024-05-12T22:10:15.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.194+0000 7f94883d5740 1 --2- 172.21.15.28:0/2662986128 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5561e906def0 0x5561e90702e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:15.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.194+0000 7f94883d5740 4 client.25633 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-12T22:10:15.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.194+0000 7f94883d5740 1 -- 172.21.15.28:0/2662986128 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5561e8f50610 con 0x5561e906def0 2024-05-12T22:10:15.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.195+0000 7f947e000640 1 --2- 172.21.15.28:0/2662986128 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5561e906def0 0x5561e90702e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:15.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.195+0000 7f947e000640 1 --2- 172.21.15.28:0/2662986128 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5561e906def0 0x5561e90702e0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=59ee46cd6d1d02bf server_cookie=2d71ee13efc3bfb2 in_seq=0 out_seq=0 2024-05-12T22:10:15.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.196+0000 7f945fe00640 10 client.25633 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:10:15.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.196+0000 7f945fe00640 1 -- 172.21.15.28:0/2662986128 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5561e8f50610 con 0x5561e906def0 2024-05-12T22:10:15.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.196+0000 7f945fe00640 10 client.25633 handle_command_reply: tid=0 2024-05-12T22:10:15.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.196+0000 7f94883d5740 1 client.25633 shutdown 2024-05-12T22:10:15.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.196+0000 7f94883d5740 1 -- 172.21.15.28:0/2662986128 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5561e8fec5e0 msgr2=0x5561e8ffb390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:10:15.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.196+0000 7f94883d5740 1 --2- 172.21.15.28:0/2662986128 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5561e8fec5e0 0x5561e8ffb390 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f94700a44b0 tx=0x7f947009b5f0 comp rx=0 tx=0).stop 2024-05-12T22:10:15.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.196+0000 7f94883d5740 1 -- 172.21.15.28:0/2662986128 shutdown_connections 2024-05-12T22:10:15.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.196+0000 7f94883d5740 1 --2- 172.21.15.28:0/2662986128 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5561e906def0 0x5561e90702e0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:15.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.196+0000 7f94883d5740 1 --2- 172.21.15.28:0/2662986128 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5561e8ff75d0 0x5561e906c3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:15.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.196+0000 7f94883d5740 1 --2- 172.21.15.28:0/2662986128 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5561e8fecf00 0x5561e8ffb8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:15.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.196+0000 7f94883d5740 1 --2- 172.21.15.28:0/2662986128 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5561e8fec5e0 0x5561e8ffb390 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:15.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.196+0000 7f94883d5740 1 -- 172.21.15.28:0/2662986128 >> 172.21.15.28:0/2662986128 conn(0x5561e8fe8900 msgr2=0x5561e8fdf330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:10:15.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.196+0000 7f945fe00640 0 client.25633 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:10:15.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.196+0000 7f94883d5740 1 -- 172.21.15.28:0/2662986128 shutdown_connections 2024-05-12T22:10:15.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.197+0000 7f94883d5740 1 -- 172.21.15.28:0/2662986128 wait complete. 2024-05-12T22:10:15.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.197+0000 7f94883d5740 20 client.25633 trim_cache size 0 max 16384 2024-05-12T22:10:15.200 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-05-12T22:10:15.200 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-12T22:10:15.200 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-05-12T22:10:15.200 INFO:teuthology.orchestra.run.smithi028.stdout: "62279362-d1ad-4983-8131-1dd4c341cf05": { 2024-05-12T22:10:15.200 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-05-12T22:10:15.200 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "62279362-d1ad-4983-8131-1dd4c341cf05", 2024-05-12T22:10:15.200 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-05-12T22:10:15.201 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:10:15.201 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:10:15.201 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-05-12T22:10:15.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.197+0000 7f947fe00640 1 -- 172.21.15.28:0/3872808683 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f94640a2e40 msgr2=0x7f94640a5300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:10:15.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.197+0000 7f947fe00640 1 --2- 172.21.15.28:0/3872808683 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f94640a2e40 0x7f94640a5300 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f9478011880 tx=0x7f9478019970 comp rx=0 tx=0).stop 2024-05-12T22:10:15.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.197+0000 7f947fe00640 1 -- 172.21.15.28:0/3872808683 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f94801016d0 msgr2=0x7f9480073b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:10:15.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.197+0000 7f947fe00640 1 --2- 172.21.15.28:0/3872808683 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f94801016d0 0x7f9480073b30 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f947000e9f0 tx=0x7f947000eeb0 comp rx=0 tx=0).stop 2024-05-12T22:10:15.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.198+0000 7f947fe00640 1 -- 172.21.15.28:0/3872808683 shutdown_connections 2024-05-12T22:10:15.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.198+0000 7f947fe00640 1 --2- 172.21.15.28:0/3872808683 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f94640a2e40 0x7f94640a5300 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:15.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.198+0000 7f947fe00640 1 --2- 172.21.15.28:0/3872808683 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94801076e0 0x7f94800745b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:15.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.198+0000 7f947fe00640 1 --2- 172.21.15.28:0/3872808683 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9480102090 0x7f9480074070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:15.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.198+0000 7f947fe00640 1 --2- 172.21.15.28:0/3872808683 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f94801016d0 0x7f9480073b30 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:15.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.198+0000 7f947fe00640 1 -- 172.21.15.28:0/3872808683 >> 172.21.15.28:0/3872808683 conn(0x7f94800fd200 msgr2=0x7f94800fd9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:10:15.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.198+0000 7f947fe00640 1 -- 172.21.15.28:0/3872808683 shutdown_connections 2024-05-12T22:10:15.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:15.198+0000 7f947fe00640 1 -- 172.21.15.28:0/3872808683 wait complete. 2024-05-12T22:10:15.222 INFO:tasks.cephfs.filesystem:scrub status for tag:62279362-d1ad-4983-8131-1dd4c341cf05 - {'path': '/', 'tag': '62279362-d1ad-4983-8131-1dd4c341cf05', 'options': 'recursive,force'} 2024-05-12T22:10:16.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:10:15 smithi028 ceph-mon[29847]: pgmap v41: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-05-12T22:10:16.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:10:15 smithi028 ceph-mon[29847]: scrub queued for path: / 2024-05-12T22:10:16.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:10:15 smithi028 ceph-mon[29847]: scrub summary: idle+waiting paths [/] 2024-05-12T22:10:16.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:10:15 smithi028 ceph-mon[29847]: scrub summary: active paths [/] 2024-05-12T22:10:16.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:10:15 smithi028 ceph-mon[29847]: scrub summary: idle+waiting paths [/] 2024-05-12T22:10:16.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:10:15 smithi186 ceph-mon[34785]: pgmap v41: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-05-12T22:10:16.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:10:15 smithi186 ceph-mon[34785]: scrub queued for path: / 2024-05-12T22:10:16.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:10:15 smithi186 ceph-mon[34785]: scrub summary: idle+waiting paths [/] 2024-05-12T22:10:16.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:10:15 smithi186 ceph-mon[34785]: scrub summary: active paths [/] 2024-05-12T22:10:16.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:10:15 smithi186 ceph-mon[34785]: scrub summary: idle+waiting paths [/] 2024-05-12T22:10:16.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:10:15 smithi086 ceph-mon[34297]: pgmap v41: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-05-12T22:10:16.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:10:15 smithi086 ceph-mon[34297]: scrub queued for path: / 2024-05-12T22:10:16.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:10:15 smithi086 ceph-mon[34297]: scrub summary: idle+waiting paths [/] 2024-05-12T22:10:16.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:10:15 smithi086 ceph-mon[34297]: scrub summary: active paths [/] 2024-05-12T22:10:16.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:10:15 smithi086 ceph-mon[34297]: scrub summary: idle+waiting paths [/] 2024-05-12T22:10:18.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:10:17 smithi028 ceph-mon[29847]: pgmap v42: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 11 KiB/s wr, 18 op/s 2024-05-12T22:10:18.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:10:17 smithi028 ceph-mon[29847]: scrub summary: idle 2024-05-12T22:10:18.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:10:17 smithi186 ceph-mon[34785]: pgmap v42: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 11 KiB/s wr, 18 op/s 2024-05-12T22:10:18.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:10:17 smithi186 ceph-mon[34785]: scrub summary: idle 2024-05-12T22:10:18.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:10:17 smithi086 ceph-mon[34297]: pgmap v42: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 11 KiB/s wr, 18 op/s 2024-05-12T22:10:18.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:10:17 smithi086 ceph-mon[34297]: scrub summary: idle 2024-05-12T22:10:19.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:10:19 smithi086 ceph-mon[34297]: pgmap v43: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-05-12T22:10:20.034 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:10:19 smithi028 ceph-mon[29847]: pgmap v43: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-05-12T22:10:20.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:10:19 smithi186 ceph-mon[34785]: pgmap v43: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-05-12T22:10:22.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:10:21 smithi028 ceph-mon[29847]: pgmap v44: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 85 B/s wr, 7 op/s 2024-05-12T22:10:22.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:10:21 smithi186 ceph-mon[34785]: pgmap v44: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 85 B/s wr, 7 op/s 2024-05-12T22:10:22.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:10:21 smithi086 ceph-mon[34297]: pgmap v44: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 85 B/s wr, 7 op/s 2024-05-12T22:10:24.034 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:10:23 smithi028 ceph-mon[29847]: pgmap v45: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-05-12T22:10:24.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:10:23 smithi186 ceph-mon[34785]: pgmap v45: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-05-12T22:10:24.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:10:23 smithi086 ceph-mon[34297]: pgmap v45: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-05-12T22:10:26.037 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:10:25 smithi028 ceph-mon[29847]: pgmap v46: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-05-12T22:10:26.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:10:25 smithi186 ceph-mon[34785]: pgmap v46: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-05-12T22:10:26.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:10:25 smithi086 ceph-mon[34297]: pgmap v46: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-05-12T22:10:28.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:10:27 smithi028 ceph-mon[29847]: pgmap v47: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-05-12T22:10:28.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:10:27 smithi186 ceph-mon[34785]: pgmap v47: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-05-12T22:10:28.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:10:27 smithi086 ceph-mon[34297]: pgmap v47: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-05-12T22:10:30.034 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:10:29 smithi028 ceph-mon[29847]: pgmap v48: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-05-12T22:10:30.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:10:29 smithi186 ceph-mon[34785]: pgmap v48: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-05-12T22:10:30.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:10:29 smithi086 ceph-mon[34297]: pgmap v48: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-05-12T22:10:31.154 INFO:tasks.workunit.client.0.smithi028.stderr:Updating files: 61% (7682/12534) Updating files: 62% (7772/12534) Updating files: 63% (7897/12534) Updating files: 64% (8022/12534) Updating files: 65% (8148/12534) Updating files: 66% (8273/12534) Updating files: 67% (8398/12534) Updating files: 68% (8524/12534) Updating files: 69% (8649/12534) Updating files: 70% (8774/12534) Updating files: 71% (8900/12534) Updating files: 72% (9025/12534) Updating files: 73% (9150/12534) Updating files: 74% (9276/12534) Updating files: 75% (9401/12534) Updating files: 76% (9526/12534) Updating files: 77% (9652/12534) Updating files: 78% (9777/12534) Updating files: 79% (9902/12534) Updating files: 80% (10028/12534) Updating files: 81% (10153/12534) Updating files: 82% (10278/12534) Updating files: 83% (10404/12534) Updating files: 84% (10529/12534) Updating files: 85% (10654/12534) Updating files: 86% (10780/12534) Updating files: 87% (10905/12534) Updating files: 88% (11030/12534) Updating files: 89% (11156/12534) Updating files: 90% (11281/12534) Updating files: 91% (11406/12534) Updating files: 92% (11532/12534) Updating files: 93% (11657/12534) Updating files: 94% (11782/12534) Updating files: 95% (11908/12534) Updating files: 96% (12033/12534) Updating files: 96% (12107/12534) Updating files: 97% (12158/12534) Updating files: 98% (12284/12534) Updating files: 99% (12409/12534) Updating files: 100% (12534/12534) Updating files: 100% (12534/12534), done. 2024-05-12T22:10:31.628 INFO:tasks.workunit.client.0.smithi028.stderr:Note: switching to 'c50db29d375ad9ca1a200449f9347b89e25fb278'. 2024-05-12T22:10:31.628 INFO:tasks.workunit.client.0.smithi028.stderr: 2024-05-12T22:10:31.628 INFO:tasks.workunit.client.0.smithi028.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-05-12T22:10:31.628 INFO:tasks.workunit.client.0.smithi028.stderr:changes and commit them, and you can discard any commits you make in this 2024-05-12T22:10:31.628 INFO:tasks.workunit.client.0.smithi028.stderr:state without impacting any branches by switching back to a branch. 2024-05-12T22:10:31.628 INFO:tasks.workunit.client.0.smithi028.stderr: 2024-05-12T22:10:31.629 INFO:tasks.workunit.client.0.smithi028.stderr:If you want to create a new branch to retain commits you create, you may 2024-05-12T22:10:31.629 INFO:tasks.workunit.client.0.smithi028.stderr:do so (now or later) by using -c with the switch command. Example: 2024-05-12T22:10:31.629 INFO:tasks.workunit.client.0.smithi028.stderr: 2024-05-12T22:10:31.629 INFO:tasks.workunit.client.0.smithi028.stderr: git switch -c 2024-05-12T22:10:31.629 INFO:tasks.workunit.client.0.smithi028.stderr: 2024-05-12T22:10:31.629 INFO:tasks.workunit.client.0.smithi028.stderr:Or undo this operation with: 2024-05-12T22:10:31.629 INFO:tasks.workunit.client.0.smithi028.stderr: 2024-05-12T22:10:31.629 INFO:tasks.workunit.client.0.smithi028.stderr: git switch - 2024-05-12T22:10:31.629 INFO:tasks.workunit.client.0.smithi028.stderr: 2024-05-12T22:10:31.629 INFO:tasks.workunit.client.0.smithi028.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-05-12T22:10:31.629 INFO:tasks.workunit.client.0.smithi028.stderr: 2024-05-12T22:10:31.629 INFO:tasks.workunit.client.0.smithi028.stderr:HEAD is now at c50db29d375 Merge pull request #57417 from zdover23/wip-doc-2024-05-12-backport-57415-to-squid 2024-05-12T22:10:31.637 DEBUG:teuthology.orchestra.run.smithi028:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-05-12T22:10:31.695 INFO:tasks.workunit.client.0.smithi028.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-05-12T22:10:31.702 INFO:tasks.workunit.client.0.smithi028.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-05-12T22:10:31.702 INFO:tasks.workunit.client.0.smithi028.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-05-12T22:10:32.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:10:31 smithi186 ceph-mon[34785]: pgmap v49: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1023 B/s wr, 9 op/s 2024-05-12T22:10:32.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:10:31 smithi186 ceph-mon[34785]: mgrmap e27: x(active, since 92s), standbys: z, y 2024-05-12T22:10:32.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:10:31 smithi086 ceph-mon[34297]: pgmap v49: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1023 B/s wr, 9 op/s 2024-05-12T22:10:32.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:10:31 smithi086 ceph-mon[34297]: mgrmap e27: x(active, since 92s), standbys: z, y 2024-05-12T22:10:32.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:10:31 smithi028 ceph-mon[29847]: pgmap v49: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1023 B/s wr, 9 op/s 2024-05-12T22:10:32.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:10:31 smithi028 ceph-mon[29847]: mgrmap e27: x(active, since 92s), standbys: z, y 2024-05-12T22:10:32.526 INFO:tasks.workunit.client.0.smithi028.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-05-12T22:10:32.642 INFO:tasks.workunit.client.0.smithi028.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-05-12T22:10:32.732 INFO:tasks.workunit.client.0.smithi028.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-05-12T22:10:32.737 INFO:tasks.workunit.client.0.smithi028.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-05-12T22:10:32.737 INFO:tasks.workunit.client.0.smithi028.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-05-12T22:10:32.821 INFO:tasks.workunit.client.0.smithi028.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-05-12T22:10:32.832 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-05-12T22:10:32.832 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-05-12T22:10:32.886 INFO:tasks.workunit:Running workunits matching suites/fsx.sh on client.0... 2024-05-12T22:10:32.887 INFO:tasks.workunit:Running workunit suites/fsx.sh... 2024-05-12T22:10:32.887 DEBUG:teuthology.orchestra.run.smithi028:workunit test suites/fsx.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c50db29d375ad9ca1a200449f9347b89e25fb278 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsx.sh 2024-05-12T22:10:32.961 INFO:tasks.workunit.client.0.smithi028.stderr:+ set -e 2024-05-12T22:10:32.961 INFO:tasks.workunit.client.0.smithi028.stderr:+ git clone https://git.ceph.com/xfstests-dev.git 2024-05-12T22:10:32.968 INFO:tasks.workunit.client.0.smithi028.stderr:Cloning into 'xfstests-dev'... 2024-05-12T22:10:34.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:10:33 smithi186 ceph-mon[34785]: pgmap v50: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1023 B/s wr, 6 op/s 2024-05-12T22:10:34.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:10:33 smithi086 ceph-mon[34297]: pgmap v50: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1023 B/s wr, 6 op/s 2024-05-12T22:10:34.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:10:33 smithi028 ceph-mon[29847]: pgmap v50: 97 pgs: 97 active+clean; 1.3 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1023 B/s wr, 6 op/s 2024-05-12T22:10:36.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:10:35 smithi186 ceph-mon[34785]: pgmap v51: 97 pgs: 97 active+clean; 1.5 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 16 KiB/s wr, 9 op/s 2024-05-12T22:10:36.241 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:10:35 smithi086 ceph-mon[34297]: pgmap v51: 97 pgs: 97 active+clean; 1.5 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 16 KiB/s wr, 9 op/s 2024-05-12T22:10:36.284 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:10:35 smithi028 ceph-mon[29847]: pgmap v51: 97 pgs: 97 active+clean; 1.5 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 16 KiB/s wr, 9 op/s 2024-05-12T22:10:38.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:10:37 smithi186 ceph-mon[34785]: pgmap v52: 97 pgs: 97 active+clean; 1.5 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 16 KiB/s wr, 7 op/s 2024-05-12T22:10:38.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:10:37 smithi086 ceph-mon[34297]: pgmap v52: 97 pgs: 97 active+clean; 1.5 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 16 KiB/s wr, 7 op/s 2024-05-12T22:10:38.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:10:37 smithi028 ceph-mon[29847]: pgmap v52: 97 pgs: 97 active+clean; 1.5 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 16 KiB/s wr, 7 op/s 2024-05-12T22:10:40.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:10:39 smithi186 ceph-mon[34785]: pgmap v53: 97 pgs: 97 active+clean; 1.5 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 16 KiB/s wr, 7 op/s 2024-05-12T22:10:40.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:10:39 smithi086 ceph-mon[34297]: pgmap v53: 97 pgs: 97 active+clean; 1.5 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 16 KiB/s wr, 7 op/s 2024-05-12T22:10:40.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:10:39 smithi028 ceph-mon[29847]: pgmap v53: 97 pgs: 97 active+clean; 1.5 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 16 KiB/s wr, 7 op/s 2024-05-12T22:10:42.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:10:41 smithi186 ceph-mon[34785]: pgmap v54: 97 pgs: 97 active+clean; 15 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.2 MiB/s wr, 14 op/s 2024-05-12T22:10:42.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:10:41 smithi086 ceph-mon[34297]: pgmap v54: 97 pgs: 97 active+clean; 15 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.2 MiB/s wr, 14 op/s 2024-05-12T22:10:42.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:10:41 smithi028 ceph-mon[29847]: pgmap v54: 97 pgs: 97 active+clean; 15 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.2 MiB/s wr, 14 op/s 2024-05-12T22:10:44.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:10:43 smithi186 ceph-mon[34785]: pgmap v55: 97 pgs: 97 active+clean; 15 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.2 MiB/s wr, 10 op/s 2024-05-12T22:10:44.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:10:43 smithi186 ceph-mon[34785]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=g=up:active} 3 up:standby-replay 6 up:standby 2024-05-12T22:10:44.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:10:43 smithi086 ceph-mon[34297]: pgmap v55: 97 pgs: 97 active+clean; 15 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.2 MiB/s wr, 10 op/s 2024-05-12T22:10:44.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:10:43 smithi086 ceph-mon[34297]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=g=up:active} 3 up:standby-replay 6 up:standby 2024-05-12T22:10:44.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:10:43 smithi028 ceph-mon[29847]: pgmap v55: 97 pgs: 97 active+clean; 15 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.2 MiB/s wr, 10 op/s 2024-05-12T22:10:44.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:10:43 smithi028 ceph-mon[29847]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=g=up:active} 3 up:standby-replay 6 up:standby 2024-05-12T22:10:45.223 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-12T22:10:45.223 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-12T22:10:45.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.459+0000 7f3043200640 1 -- 172.21.15.28:0/2352363247 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f303c074420 msgr2=0x7f303c10dbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:10:45.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.459+0000 7f3043200640 1 --2- 172.21.15.28:0/2352363247 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f303c074420 0x7f303c10dbd0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f3028009900 tx=0x7f302802a490 comp rx=0 tx=0).stop 2024-05-12T22:10:45.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.459+0000 7f3043200640 1 -- 172.21.15.28:0/2352363247 shutdown_connections 2024-05-12T22:10:45.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.459+0000 7f3043200640 1 --2- 172.21.15.28:0/2352363247 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f303c10e110 0x7f303c110500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:45.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.459+0000 7f3043200640 1 --2- 172.21.15.28:0/2352363247 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f303c074420 0x7f303c10dbd0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:45.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.459+0000 7f3043200640 1 --2- 172.21.15.28:0/2352363247 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f303c073b00 0x7f303c073ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:45.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.459+0000 7f3043200640 1 -- 172.21.15.28:0/2352363247 >> 172.21.15.28:0/2352363247 conn(0x7f303c078c80 msgr2=0x7f303c079090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:10:45.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.459+0000 7f3043200640 1 -- 172.21.15.28:0/2352363247 shutdown_connections 2024-05-12T22:10:45.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.459+0000 7f3043200640 1 -- 172.21.15.28:0/2352363247 wait complete. 2024-05-12T22:10:45.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.460+0000 7f3043200640 1 Processor -- start 2024-05-12T22:10:45.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.460+0000 7f3043200640 1 -- start start 2024-05-12T22:10:45.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.460+0000 7f3043200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f303c073b00 0x7f303c19a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:45.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.460+0000 7f3043200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f303c074420 0x7f303c19a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:45.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.460+0000 7f3043200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f303c10e110 0x7f303c19ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:45.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.460+0000 7f3043200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f303c113e10 con 0x7f303c10e110 2024-05-12T22:10:45.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.460+0000 7f3043200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f303c113c90 con 0x7f303c074420 2024-05-12T22:10:45.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.460+0000 7f3043200640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f303c113f90 con 0x7f303c073b00 2024-05-12T22:10:45.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.461+0000 7f3041200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f303c10e110 0x7f303c19ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:45.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.461+0000 7f303be00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f303c073b00 0x7f303c19a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:45.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.461+0000 7f303b400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f303c074420 0x7f303c19a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:45.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.461+0000 7f3041200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f303c10e110 0x7f303c19ec70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58938/0 (socket says 172.21.15.28:58938) 2024-05-12T22:10:45.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.461+0000 7f3041200640 1 -- 172.21.15.28:0/218883705 learned_addr learned my addr 172.21.15.28:0/218883705 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:10:45.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.461+0000 7f303be00640 1 --2- 172.21.15.28:0/218883705 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f303c073b00 0x7f303c19a3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:42018/0 (socket says 172.21.15.28:42018) 2024-05-12T22:10:45.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.461+0000 7f303b400640 1 -- 172.21.15.28:0/218883705 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f303c073b00 msgr2=0x7f303c19a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:10:45.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.461+0000 7f303b400640 1 --2- 172.21.15.28:0/218883705 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f303c073b00 0x7f303c19a3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:45.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.461+0000 7f303b400640 1 -- 172.21.15.28:0/218883705 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f303c10e110 msgr2=0x7f303c19ec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:10:45.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.461+0000 7f303b400640 1 --2- 172.21.15.28:0/218883705 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f303c10e110 0x7f303c19ec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:45.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.461+0000 7f303b400640 1 -- 172.21.15.28:0/218883705 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f303c19f3a0 con 0x7f303c074420 2024-05-12T22:10:45.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.462+0000 7f303b400640 1 --2- 172.21.15.28:0/218883705 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f303c074420 0x7f303c19a8f0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f3028005940 tx=0x7f3028004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:10:45.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.462+0000 7f3038c00640 1 -- 172.21.15.28:0/218883705 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f302803f030 con 0x7f303c074420 2024-05-12T22:10:45.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.462+0000 7f3038c00640 1 -- 172.21.15.28:0/218883705 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3028003ad0 con 0x7f303c074420 2024-05-12T22:10:45.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.462+0000 7f3038c00640 1 -- 172.21.15.28:0/218883705 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3028035070 con 0x7f303c074420 2024-05-12T22:10:45.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.462+0000 7f3043200640 1 -- 172.21.15.28:0/218883705 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f303c19f630 con 0x7f303c074420 2024-05-12T22:10:45.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.462+0000 7f3043200640 1 -- 172.21.15.28:0/218883705 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f303c1a6f20 con 0x7f303c074420 2024-05-12T22:10:45.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.465+0000 7f3046000740 1 Processor -- start 2024-05-12T22:10:45.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.465+0000 7f3046000740 1 -- start start 2024-05-12T22:10:45.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.466+0000 7f3046000740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5651c55f3bc0 0x5651c55f3fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:45.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.466+0000 7f3046000740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5651c55f44e0 0x5651c55fe670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:45.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.466+0000 7f3046000740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5651c55febb0 0x5651c5600fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:45.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.466+0000 7f3046000740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5651c55bc2a0 con 0x5651c55f3bc0 2024-05-12T22:10:45.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.466+0000 7f3046000740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x5651c5558b90 con 0x5651c55f44e0 2024-05-12T22:10:45.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.466+0000 7f3046000740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5651c55005b0 con 0x5651c55febb0 2024-05-12T22:10:45.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.466+0000 7f3038c00640 1 -- 172.21.15.28:0/218883705 <== mon.2 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7f3028042030 con 0x7f303c074420 2024-05-12T22:10:45.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.466+0000 7f303be00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5651c55f44e0 0x5651c55fe670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:45.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.466+0000 7f3041200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5651c55f3bc0 0x5651c55f3fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:45.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.466+0000 7f303be00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5651c55f44e0 0x5651c55fe670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:45958/0 (socket says 172.21.15.28:45958) 2024-05-12T22:10:45.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.466+0000 7f303be00640 1 -- 172.21.15.28:0/108989351 learned_addr learned my addr 172.21.15.28:0/108989351 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:10:45.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.466+0000 7f303b400640 1 --2- 172.21.15.28:0/108989351 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5651c55febb0 0x5651c5600fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:45.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.466+0000 7f303be00640 1 -- 172.21.15.28:0/108989351 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5651c55febb0 msgr2=0x5651c5600fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:10:45.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.467+0000 7f303be00640 1 --2- 172.21.15.28:0/108989351 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5651c55febb0 0x5651c5600fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:45.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.467+0000 7f303be00640 1 -- 172.21.15.28:0/108989351 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5651c55f3bc0 msgr2=0x5651c55f3fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:10:45.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.467+0000 7f303be00640 1 --2- 172.21.15.28:0/108989351 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5651c55f3bc0 0x5651c55f3fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:45.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.467+0000 7f303be00640 1 -- 172.21.15.28:0/108989351 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5651c53367d0 con 0x5651c55f44e0 2024-05-12T22:10:45.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.467+0000 7f303b400640 1 --2- 172.21.15.28:0/108989351 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5651c55febb0 0x5651c5600fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:10:45.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.467+0000 7f303be00640 1 --2- 172.21.15.28:0/108989351 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5651c55f44e0 0x5651c55fe670 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f302c009cf0 tx=0x7f302c011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=54f4770727945ac3 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:10:45.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.467+0000 7f3031600640 1 -- 172.21.15.28:0/108989351 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f302c004230 con 0x5651c55f44e0 2024-05-12T22:10:45.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.467+0000 7f3031600640 1 -- 172.21.15.28:0/108989351 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f302c0043d0 con 0x5651c55f44e0 2024-05-12T22:10:45.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.467+0000 7f3031600640 1 -- 172.21.15.28:0/108989351 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f302c019920 con 0x5651c55f44e0 2024-05-12T22:10:45.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.468+0000 7f3038c00640 1 --2- 172.21.15.28:0/218883705 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f30200a2d80 0x7f30200a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:45.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.468+0000 7f3046000740 1 -- 172.21.15.28:0/108989351 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5651c55f44e0 msgr2=0x5651c55fe670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:10:45.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.468+0000 7f3046000740 1 --2- 172.21.15.28:0/108989351 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5651c55f44e0 0x5651c55fe670 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f302c009cf0 tx=0x7f302c011810 comp rx=0 tx=0).stop 2024-05-12T22:10:45.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.468+0000 7f3046000740 1 -- 172.21.15.28:0/108989351 shutdown_connections 2024-05-12T22:10:45.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.468+0000 7f3046000740 1 --2- 172.21.15.28:0/108989351 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5651c55febb0 0x5651c5600fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:45.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.468+0000 7f3046000740 1 --2- 172.21.15.28:0/108989351 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5651c55f44e0 0x5651c55fe670 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:45.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.468+0000 7f3046000740 1 --2- 172.21.15.28:0/108989351 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5651c55f3bc0 0x5651c55f3fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:45.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.468+0000 7f3046000740 1 -- 172.21.15.28:0/108989351 >> 172.21.15.28:0/108989351 conn(0x5651c55efee0 msgr2=0x5651c55f02c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:10:45.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.468+0000 7f3038c00640 1 -- 172.21.15.28:0/218883705 <== mon.2 v2:172.21.15.86:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f30280ef6b0 con 0x7f303c074420 2024-05-12T22:10:45.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.468+0000 7f3041200640 1 --2- 172.21.15.28:0/218883705 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f30200a2d80 0x7f30200a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:45.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.468+0000 7f3046000740 1 -- 172.21.15.28:0/108989351 shutdown_connections 2024-05-12T22:10:45.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.468+0000 7f3046000740 1 -- 172.21.15.28:0/108989351 wait complete. 2024-05-12T22:10:45.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.469+0000 7f3046000740 1 -- start start 2024-05-12T22:10:45.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.469+0000 7f3041200640 1 --2- 172.21.15.28:0/218883705 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f30200a2d80 0x7f30200a5240 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f3034013570 tx=0x7f3034009640 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:10:45.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.469+0000 7f3046000740 1 Processor -- start 2024-05-12T22:10:45.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.470+0000 7f3046000740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5651c55f3bc0 0x5651c55fe480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:45.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.470+0000 7f3046000740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5651c55f44e0 0x5651c55fb770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:45.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.470+0000 7f3046000740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5651c55febb0 0x5651c55fbcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:45.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.470+0000 7f3046000740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5651c55139f0 con 0x5651c55febb0 2024-05-12T22:10:45.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.470+0000 7f3046000740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x5651c5554b50 con 0x5651c55f44e0 2024-05-12T22:10:45.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.470+0000 7f3046000740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5651c56019a0 con 0x5651c55f3bc0 2024-05-12T22:10:45.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.470+0000 7f3046000740 10 client.0 _ll_register_callbacks cb 0x5651c526bbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:10:45.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.470+0000 7f303be00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5651c55f3bc0 0x5651c55fe480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:45.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.470+0000 7f303be00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5651c55f3bc0 0x5651c55fe480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:42038/0 (socket says 172.21.15.28:42038) 2024-05-12T22:10:45.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.470+0000 7f303be00640 1 -- 172.21.15.28:0/2934587996 learned_addr learned my addr 172.21.15.28:0/2934587996 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:10:45.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.470+0000 7f3041200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5651c55febb0 0x5651c55fbcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:45.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.470+0000 7f303b400640 1 --2- 172.21.15.28:0/2934587996 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5651c55f44e0 0x5651c55fb770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:45.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.470+0000 7f303be00640 1 -- 172.21.15.28:0/2934587996 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5651c55f44e0 msgr2=0x5651c55fb770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:10:45.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.470+0000 7f303be00640 1 --2- 172.21.15.28:0/2934587996 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5651c55f44e0 0x5651c55fb770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:45.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.470+0000 7f303be00640 1 -- 172.21.15.28:0/2934587996 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5651c55febb0 msgr2=0x5651c55fbcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:10:45.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.470+0000 7f303be00640 1 --2- 172.21.15.28:0/2934587996 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5651c55febb0 0x5651c55fbcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:45.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.470+0000 7f303be00640 1 -- 172.21.15.28:0/2934587996 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5651c526c3f0 con 0x5651c55f3bc0 2024-05-12T22:10:45.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.470+0000 7f303b400640 1 --2- 172.21.15.28:0/2934587996 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5651c55f44e0 0x5651c55fb770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:10:45.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.471+0000 7f303be00640 1 --2- 172.21.15.28:0/2934587996 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5651c55f3bc0 0x5651c55fe480 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f302c0043a0 tx=0x7f302c0096f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5d76afe440522c05 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:10:45.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.471+0000 7f3026000640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-12T22:10:45.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.471+0000 7f3026000640 1 -- 172.21.15.28:0/2934587996 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f302c027070 con 0x5651c55f3bc0 2024-05-12T22:10:45.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.471+0000 7f3026000640 1 -- 172.21.15.28:0/2934587996 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f302c005240 con 0x5651c55f3bc0 2024-05-12T22:10:45.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.471+0000 7f3046000740 1 -- 172.21.15.28:0/2934587996 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f302c0043d0 con 0x5651c55f3bc0 2024-05-12T22:10:45.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.472+0000 7f3026000640 1 -- 172.21.15.28:0/2934587996 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f302c033050 con 0x5651c55f3bc0 2024-05-12T22:10:45.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.472+0000 7f3046000740 10 client.25645 fetch_fsmap learned FSMap version 25 2024-05-12T22:10:45.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.472+0000 7f3046000740 1 -- 172.21.15.28:0/2934587996 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5651c522c0f0 con 0x5651c55f3bc0 2024-05-12T22:10:45.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.472+0000 7f3026000640 1 -- 172.21.15.28:0/2934587996 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f302c0043d0 con 0x5651c55f3bc0 2024-05-12T22:10:45.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.472+0000 7f3026000640 1 -- 172.21.15.28:0/2934587996 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7f302c004ba0 con 0x5651c55f3bc0 2024-05-12T22:10:45.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.472+0000 7f3046000740 10 client.25645 fetch_fsmap finished waiting for FSMap version 25 2024-05-12T22:10:45.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.472+0000 7f3046000740 10 client.25645 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:10:45.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.472+0000 7f3046000740 20 client.25645 populate_metadata read hostname 'smithi028' 2024-05-12T22:10:45.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.472+0000 7f3046000740 1 --2- 172.21.15.28:0/2934587996 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5651c560e5f0 0x5651c56109e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:45.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.472+0000 7f3046000740 4 client.25645 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-12T22:10:45.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.472+0000 7f3046000740 1 -- 172.21.15.28:0/2934587996 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5651c55662c0 con 0x5651c560e5f0 2024-05-12T22:10:45.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.473+0000 7f303b400640 1 --2- 172.21.15.28:0/2934587996 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5651c560e5f0 0x5651c56109e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:45.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.473+0000 7f303b400640 1 --2- 172.21.15.28:0/2934587996 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5651c560e5f0 0x5651c56109e0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b8d092f0a77e976e server_cookie=f0bcd8043924463a in_seq=0 out_seq=0 2024-05-12T22:10:45.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.473+0000 7f3026000640 10 client.25645 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:10:45.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.474+0000 7f3026000640 1 -- 172.21.15.28:0/2934587996 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5651c55662c0 con 0x5651c560e5f0 2024-05-12T22:10:45.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.474+0000 7f3026000640 10 client.25645 handle_command_reply: tid=0 2024-05-12T22:10:45.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.474+0000 7f3046000740 1 client.25645 shutdown 2024-05-12T22:10:45.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.474+0000 7f3046000740 1 -- 172.21.15.28:0/2934587996 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5651c55f3bc0 msgr2=0x5651c55fe480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:10:45.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.474+0000 7f3046000740 1 --2- 172.21.15.28:0/2934587996 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5651c55f3bc0 0x5651c55fe480 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f302c0043a0 tx=0x7f302c0096f0 comp rx=0 tx=0).stop 2024-05-12T22:10:45.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.475+0000 7f3046000740 1 -- 172.21.15.28:0/2934587996 shutdown_connections 2024-05-12T22:10:45.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.475+0000 7f3046000740 1 --2- 172.21.15.28:0/2934587996 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5651c560e5f0 0x5651c56109e0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:45.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.475+0000 7f3046000740 1 --2- 172.21.15.28:0/2934587996 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5651c55febb0 0x5651c55fbcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:45.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.475+0000 7f3046000740 1 --2- 172.21.15.28:0/2934587996 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5651c55f44e0 0x5651c55fb770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:45.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.475+0000 7f3046000740 1 --2- 172.21.15.28:0/2934587996 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5651c55f3bc0 0x5651c55fe480 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:45.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.475+0000 7f3026000640 0 client.25645 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:10:45.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.475+0000 7f3046000740 1 -- 172.21.15.28:0/2934587996 >> 172.21.15.28:0/2934587996 conn(0x5651c55efee0 msgr2=0x5651c55ef0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:10:45.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.475+0000 7f3046000740 1 -- 172.21.15.28:0/2934587996 shutdown_connections 2024-05-12T22:10:45.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.475+0000 7f3046000740 1 -- 172.21.15.28:0/2934587996 wait complete. 2024-05-12T22:10:45.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.475+0000 7f3046000740 20 client.25645 trim_cache size 0 max 16384 2024-05-12T22:10:45.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.494+0000 7f3046000740 1 Processor -- start 2024-05-12T22:10:45.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.494+0000 7f3046000740 1 -- start start 2024-05-12T22:10:45.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.495+0000 7f3046000740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5651c55f3bc0 0x5651c55ef860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:45.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.495+0000 7f3046000740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5651c55f44e0 0x5651c55f6ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:45.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.495+0000 7f3046000740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5651c55febb0 0x5651c55fdfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:45.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.495+0000 7f3046000740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f303c1b2030 con 0x5651c55f3bc0 2024-05-12T22:10:45.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.495+0000 7f3046000740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f303c1af900 con 0x5651c55f44e0 2024-05-12T22:10:45.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.495+0000 7f3046000740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f303c1ad750 con 0x5651c55febb0 2024-05-12T22:10:45.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.495+0000 7f303be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5651c55f3bc0 0x5651c55ef860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:45.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.495+0000 7f303be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5651c55f3bc0 0x5651c55ef860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58958/0 (socket says 172.21.15.28:58958) 2024-05-12T22:10:45.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.495+0000 7f303be00640 1 -- 172.21.15.28:0/2587364734 learned_addr learned my addr 172.21.15.28:0/2587364734 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:10:45.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.495+0000 7f303b400640 1 --2- 172.21.15.28:0/2587364734 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5651c55f44e0 0x5651c55f6ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:45.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.495+0000 7f3041200640 1 --2- 172.21.15.28:0/2587364734 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5651c55febb0 0x5651c55fdfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:45.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.495+0000 7f303be00640 1 -- 172.21.15.28:0/2587364734 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5651c55febb0 msgr2=0x5651c55fdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:10:45.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.495+0000 7f303be00640 1 --2- 172.21.15.28:0/2587364734 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5651c55febb0 0x5651c55fdfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:45.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.495+0000 7f303be00640 1 -- 172.21.15.28:0/2587364734 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5651c55f44e0 msgr2=0x5651c55f6ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:10:45.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.495+0000 7f303be00640 1 --2- 172.21.15.28:0/2587364734 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5651c55f44e0 0x5651c55f6ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:45.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.495+0000 7f303be00640 1 -- 172.21.15.28:0/2587364734 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5651c5611040 con 0x5651c55f3bc0 2024-05-12T22:10:45.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.496+0000 7f303be00640 1 --2- 172.21.15.28:0/2587364734 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5651c55f3bc0 0x5651c55ef860 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f302c019490 tx=0x7f302c009bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8061acda97b215e4 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:10:45.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.496+0000 7f3030c00640 1 -- 172.21.15.28:0/2587364734 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f302c030070 con 0x5651c55f3bc0 2024-05-12T22:10:45.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.496+0000 7f3030c00640 1 -- 172.21.15.28:0/2587364734 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f302c011d10 con 0x5651c55f3bc0 2024-05-12T22:10:45.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.496+0000 7f3030c00640 1 -- 172.21.15.28:0/2587364734 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f302c027070 con 0x5651c55f3bc0 2024-05-12T22:10:45.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.497+0000 7f3046000740 1 -- 172.21.15.28:0/2587364734 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5651c55f3bc0 msgr2=0x5651c55ef860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:10:45.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.497+0000 7f3046000740 1 --2- 172.21.15.28:0/2587364734 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5651c55f3bc0 0x5651c55ef860 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f302c019490 tx=0x7f302c009bf0 comp rx=0 tx=0).stop 2024-05-12T22:10:45.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.497+0000 7f3046000740 1 -- 172.21.15.28:0/2587364734 shutdown_connections 2024-05-12T22:10:45.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.497+0000 7f3046000740 1 --2- 172.21.15.28:0/2587364734 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5651c55febb0 0x5651c55fdfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:45.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.497+0000 7f3046000740 1 --2- 172.21.15.28:0/2587364734 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5651c55f44e0 0x5651c55f6ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:45.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.497+0000 7f3046000740 1 --2- 172.21.15.28:0/2587364734 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5651c55f3bc0 0x5651c55ef860 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:45.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.497+0000 7f3046000740 1 -- 172.21.15.28:0/2587364734 >> 172.21.15.28:0/2587364734 conn(0x5651c55efee0 msgr2=0x5651c55f50f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:10:45.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.497+0000 7f3046000740 1 -- 172.21.15.28:0/2587364734 shutdown_connections 2024-05-12T22:10:45.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.497+0000 7f3046000740 1 -- 172.21.15.28:0/2587364734 wait complete. 2024-05-12T22:10:45.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.497+0000 7f3046000740 1 -- start start 2024-05-12T22:10:45.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.497+0000 7f3046000740 1 Processor -- start 2024-05-12T22:10:45.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.498+0000 7f3046000740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5651c55f3bc0 0x5651c5601be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:45.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.498+0000 7f3046000740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5651c55f44e0 0x5651c5602120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:45.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.498+0000 7f3046000740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5651c55febb0 0x5651c5672ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:45.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.498+0000 7f3046000740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f303c1addc0 con 0x5651c55f3bc0 2024-05-12T22:10:45.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.498+0000 7f3046000740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f303c1af230 con 0x5651c55febb0 2024-05-12T22:10:45.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.498+0000 7f3046000740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f303c1ae490 con 0x5651c55f44e0 2024-05-12T22:10:45.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.498+0000 7f3046000740 10 client.0 _ll_register_callbacks cb 0x7f3034009a60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:10:45.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.498+0000 7f303be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5651c55f3bc0 0x5651c5601be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:45.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.498+0000 7f303be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5651c55f3bc0 0x5651c5601be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58974/0 (socket says 172.21.15.28:58974) 2024-05-12T22:10:45.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.498+0000 7f303be00640 1 -- 172.21.15.28:0/674843820 learned_addr learned my addr 172.21.15.28:0/674843820 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:10:45.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.498+0000 7f303b400640 1 --2- 172.21.15.28:0/674843820 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5651c55f44e0 0x5651c5602120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:45.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.498+0000 7f3041200640 1 --2- 172.21.15.28:0/674843820 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5651c55febb0 0x5651c5672ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:45.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.499+0000 7f303be00640 1 -- 172.21.15.28:0/674843820 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5651c55f44e0 msgr2=0x5651c5602120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:10:45.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.499+0000 7f303be00640 1 --2- 172.21.15.28:0/674843820 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5651c55f44e0 0x5651c5602120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:45.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.499+0000 7f303be00640 1 -- 172.21.15.28:0/674843820 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5651c55febb0 msgr2=0x5651c5672ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:10:45.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.499+0000 7f303be00640 1 --2- 172.21.15.28:0/674843820 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5651c55febb0 0x5651c5672ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:45.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.499+0000 7f303be00640 1 -- 172.21.15.28:0/674843820 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5651c56029b0 con 0x5651c55f3bc0 2024-05-12T22:10:45.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.499+0000 7f303be00640 1 --2- 172.21.15.28:0/674843820 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5651c55f3bc0 0x5651c5601be0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f302c009560 tx=0x7f302c019a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ad457408a08eff8 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:10:45.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.499+0000 7f3026000640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-12T22:10:45.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.499+0000 7f3026000640 1 -- 172.21.15.28:0/674843820 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f302c030040 con 0x5651c55f3bc0 2024-05-12T22:10:45.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.499+0000 7f3026000640 1 -- 172.21.15.28:0/674843820 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f302c004fc0 con 0x5651c55f3bc0 2024-05-12T22:10:45.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.499+0000 7f3046000740 1 -- 172.21.15.28:0/674843820 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f302c011d10 con 0x5651c55f3bc0 2024-05-12T22:10:45.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.499+0000 7f3026000640 1 -- 172.21.15.28:0/674843820 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f302c027040 con 0x5651c55f3bc0 2024-05-12T22:10:45.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.499+0000 7f3046000740 10 client.15579 fetch_fsmap learned FSMap version 25 2024-05-12T22:10:45.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.499+0000 7f3046000740 1 -- 172.21.15.28:0/674843820 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5651c56740f0 con 0x5651c55f3bc0 2024-05-12T22:10:45.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.500+0000 7f3026000640 1 -- 172.21.15.28:0/674843820 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f302c011d10 con 0x5651c55f3bc0 2024-05-12T22:10:45.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.500+0000 7f3026000640 1 -- 172.21.15.28:0/674843820 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7f302c0046f0 con 0x5651c55f3bc0 2024-05-12T22:10:45.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.500+0000 7f3046000740 10 client.15579 fetch_fsmap finished waiting for FSMap version 25 2024-05-12T22:10:45.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.500+0000 7f3046000740 10 client.15579 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:10:45.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.500+0000 7f3046000740 20 client.15579 populate_metadata read hostname 'smithi028' 2024-05-12T22:10:45.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.500+0000 7f3046000740 1 --2- 172.21.15.28:0/674843820 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5651c5674760 0x5651c5676b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:45.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.500+0000 7f3046000740 4 client.15579 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-12T22:10:45.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.500+0000 7f3046000740 1 -- 172.21.15.28:0/674843820 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5651c5560610 con 0x5651c5674760 2024-05-12T22:10:45.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.500+0000 7f303b400640 1 --2- 172.21.15.28:0/674843820 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5651c5674760 0x5651c5676b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:45.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.501+0000 7f303b400640 1 --2- 172.21.15.28:0/674843820 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5651c5674760 0x5651c5676b80 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=600731e3f09d8101 server_cookie=de67d37f6b40d5fa in_seq=0 out_seq=0 2024-05-12T22:10:45.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.501+0000 7f3026000640 10 client.15579 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:10:45.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.501+0000 7f3026000640 1 -- 172.21.15.28:0/674843820 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5651c5560610 con 0x5651c5674760 2024-05-12T22:10:45.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.501+0000 7f3026000640 10 client.15579 handle_command_reply: tid=0 2024-05-12T22:10:45.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.501+0000 7f3046000740 1 client.15579 shutdown 2024-05-12T22:10:45.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.501+0000 7f3046000740 1 -- 172.21.15.28:0/674843820 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5651c55f3bc0 msgr2=0x5651c5601be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:10:45.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.501+0000 7f3046000740 1 --2- 172.21.15.28:0/674843820 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5651c55f3bc0 0x5651c5601be0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f302c009560 tx=0x7f302c019a90 comp rx=0 tx=0).stop 2024-05-12T22:10:45.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.501+0000 7f3046000740 1 -- 172.21.15.28:0/674843820 shutdown_connections 2024-05-12T22:10:45.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.501+0000 7f3046000740 1 --2- 172.21.15.28:0/674843820 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5651c5674760 0x5651c5676b80 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:45.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.501+0000 7f3046000740 1 --2- 172.21.15.28:0/674843820 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5651c55febb0 0x5651c5672ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:45.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.501+0000 7f3046000740 1 --2- 172.21.15.28:0/674843820 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5651c55f44e0 0x5651c5602120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:45.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.501+0000 7f3046000740 1 --2- 172.21.15.28:0/674843820 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5651c55f3bc0 0x5651c5601be0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:45.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.501+0000 7f3026000640 0 client.15579 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:10:45.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.501+0000 7f3046000740 1 -- 172.21.15.28:0/674843820 >> 172.21.15.28:0/674843820 conn(0x5651c55efee0 msgr2=0x5651c55fcfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:10:45.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.502+0000 7f3046000740 1 -- 172.21.15.28:0/674843820 shutdown_connections 2024-05-12T22:10:45.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.502+0000 7f3046000740 1 -- 172.21.15.28:0/674843820 wait complete. 2024-05-12T22:10:45.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.502+0000 7f3046000740 20 client.15579 trim_cache size 0 max 16384 2024-05-12T22:10:45.506 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-05-12T22:10:45.507 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-05-12T22:10:45.507 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-05-12T22:10:45.507 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-05-12T22:10:45.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.502+0000 7f3043200640 1 -- 172.21.15.28:0/218883705 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f30200a2d80 msgr2=0x7f30200a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:10:45.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.502+0000 7f3043200640 1 --2- 172.21.15.28:0/218883705 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f30200a2d80 0x7f30200a5240 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f3034013570 tx=0x7f3034009640 comp rx=0 tx=0).stop 2024-05-12T22:10:45.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.502+0000 7f3043200640 1 -- 172.21.15.28:0/218883705 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f303c074420 msgr2=0x7f303c19a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:10:45.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.502+0000 7f3043200640 1 --2- 172.21.15.28:0/218883705 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f303c074420 0x7f303c19a8f0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f3028005940 tx=0x7f3028004270 comp rx=0 tx=0).stop 2024-05-12T22:10:45.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.502+0000 7f3043200640 1 -- 172.21.15.28:0/218883705 shutdown_connections 2024-05-12T22:10:45.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.502+0000 7f3043200640 1 --2- 172.21.15.28:0/218883705 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f30200a2d80 0x7f30200a5240 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:45.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.502+0000 7f3043200640 1 --2- 172.21.15.28:0/218883705 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f303c10e110 0x7f303c19ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:45.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.502+0000 7f3043200640 1 --2- 172.21.15.28:0/218883705 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f303c074420 0x7f303c19a8f0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:45.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.502+0000 7f3043200640 1 --2- 172.21.15.28:0/218883705 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f303c073b00 0x7f303c19a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:45.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.502+0000 7f3043200640 1 -- 172.21.15.28:0/218883705 >> 172.21.15.28:0/218883705 conn(0x7f303c078c80 msgr2=0x7f303c0ffaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:10:45.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.503+0000 7f3043200640 1 -- 172.21.15.28:0/218883705 shutdown_connections 2024-05-12T22:10:45.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.503+0000 7f3043200640 1 -- 172.21.15.28:0/218883705 wait complete. 2024-05-12T22:10:45.521 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-12T22:10:45.522 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-12T22:10:45.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.735+0000 7fd6ed600640 1 -- 172.21.15.28:0/303195418 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd6e81076b0 msgr2=0x7fd6e8107a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:10:45.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.735+0000 7fd6ed600640 1 --2- 172.21.15.28:0/303195418 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd6e81076b0 0x7fd6e8107a90 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fd6d8009cd0 tx=0x7fd6d802a8e0 comp rx=0 tx=0).stop 2024-05-12T22:10:45.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.735+0000 7fd6ed600640 1 -- 172.21.15.28:0/303195418 shutdown_connections 2024-05-12T22:10:45.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.735+0000 7fd6ed600640 1 --2- 172.21.15.28:0/303195418 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6e8101fd0 0x7fd6e810e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:45.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.735+0000 7fd6ed600640 1 --2- 172.21.15.28:0/303195418 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd6e8101610 0x7fd6e8101a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:45.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.735+0000 7fd6ed600640 1 --2- 172.21.15.28:0/303195418 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd6e81076b0 0x7fd6e8107a90 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:45.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.735+0000 7fd6ed600640 1 -- 172.21.15.28:0/303195418 >> 172.21.15.28:0/303195418 conn(0x7fd6e80fd160 msgr2=0x7fd6e80ff580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:10:45.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.735+0000 7fd6ed600640 1 -- 172.21.15.28:0/303195418 shutdown_connections 2024-05-12T22:10:45.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.735+0000 7fd6ed600640 1 -- 172.21.15.28:0/303195418 wait complete. 2024-05-12T22:10:45.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.736+0000 7fd6ed600640 1 Processor -- start 2024-05-12T22:10:45.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.736+0000 7fd6ed600640 1 -- start start 2024-05-12T22:10:45.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.737+0000 7fd6ed600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6e8101610 0x7fd6e819a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:45.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.737+0000 7fd6ed600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd6e8101fd0 0x7fd6e819a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:45.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.737+0000 7fd6ed600640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd6e81076b0 0x7fd6e819e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:45.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.737+0000 7fd6ed600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd6e8111a80 con 0x7fd6e8101610 2024-05-12T22:10:45.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.737+0000 7fd6ed600640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fd6e8111900 con 0x7fd6e8101fd0 2024-05-12T22:10:45.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.737+0000 7fd6ed600640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd6e8111c00 con 0x7fd6e81076b0 2024-05-12T22:10:45.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.737+0000 7fd6e7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6e8101610 0x7fd6e819a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:45.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.737+0000 7fd6e7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6e8101610 0x7fd6e819a0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58990/0 (socket says 172.21.15.28:58990) 2024-05-12T22:10:45.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.737+0000 7fd6e7e00640 1 -- 172.21.15.28:0/2970932550 learned_addr learned my addr 172.21.15.28:0/2970932550 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:10:45.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.737+0000 7fd6ecc00640 1 --2- 172.21.15.28:0/2970932550 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd6e81076b0 0x7fd6e819e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:45.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.737+0000 7fd6df400640 1 --2- 172.21.15.28:0/2970932550 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd6e8101fd0 0x7fd6e819a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:45.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.738+0000 7fd6e7e00640 1 -- 172.21.15.28:0/2970932550 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd6e81076b0 msgr2=0x7fd6e819e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:10:45.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.738+0000 7fd6e7e00640 1 --2- 172.21.15.28:0/2970932550 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd6e81076b0 0x7fd6e819e9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:45.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.738+0000 7fd6e7e00640 1 -- 172.21.15.28:0/2970932550 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd6e8101fd0 msgr2=0x7fd6e819a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:10:45.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.738+0000 7fd6e7e00640 1 --2- 172.21.15.28:0/2970932550 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd6e8101fd0 0x7fd6e819a620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:45.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.738+0000 7fd6e7e00640 1 -- 172.21.15.28:0/2970932550 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6e819f0d0 con 0x7fd6e8101610 2024-05-12T22:10:45.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.738+0000 7fd6e7e00640 1 --2- 172.21.15.28:0/2970932550 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6e8101610 0x7fd6e819a0e0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fd6d8000f80 tx=0x7fd6d8031e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:10:45.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.738+0000 7fd6e5600640 1 -- 172.21.15.28:0/2970932550 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6d8003d30 con 0x7fd6e8101610 2024-05-12T22:10:45.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.738+0000 7fd6ed600640 1 -- 172.21.15.28:0/2970932550 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6e819f360 con 0x7fd6e8101610 2024-05-12T22:10:45.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.738+0000 7fd6e5600640 1 -- 172.21.15.28:0/2970932550 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd6d8004520 con 0x7fd6e8101610 2024-05-12T22:10:45.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.738+0000 7fd6e5600640 1 -- 172.21.15.28:0/2970932550 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6d8030d10 con 0x7fd6e8101610 2024-05-12T22:10:45.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.739+0000 7fd6ed600640 1 -- 172.21.15.28:0/2970932550 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6e81a6c50 con 0x7fd6e8101610 2024-05-12T22:10:45.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.740+0000 7fd6ed600640 1 -- 172.21.15.28:0/2970932550 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6e80689c0 con 0x7fd6e8101610 2024-05-12T22:10:45.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.742+0000 7fd6e5600640 1 -- 172.21.15.28:0/2970932550 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7fd6d80046d0 con 0x7fd6e8101610 2024-05-12T22:10:45.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.743+0000 7fd6e5600640 1 --2- 172.21.15.28:0/2970932550 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fd6cc0a2d00 0x7fd6cc0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:45.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.743+0000 7fd6e5600640 1 -- 172.21.15.28:0/2970932550 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7fd6d802e030 con 0x7fd6e8101610 2024-05-12T22:10:45.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.743+0000 7fd6df400640 1 --2- 172.21.15.28:0/2970932550 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fd6cc0a2d00 0x7fd6cc0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:45.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.744+0000 7fd6df400640 1 --2- 172.21.15.28:0/2970932550 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fd6cc0a2d00 0x7fd6cc0a51c0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd6e810e970 tx=0x7fd6d0005f90 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:10:45.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.748+0000 7fd6e5600640 1 -- 172.21.15.28:0/2970932550 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fd6d80bd170 con 0x7fd6e8101610 2024-05-12T22:10:45.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.996+0000 7fd6ed600640 1 -- 172.21.15.28:0/2970932550 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd6e81a6e80 con 0x7fd6e8101610 2024-05-12T22:10:45.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:45.998+0000 7fd6e5600640 1 -- 172.21.15.28:0/2970932550 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11865 (secure 0 0 0) 0x7fd6d80bca70 con 0x7fd6e8101610 2024-05-12T22:10:45.999 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-12T22:10:45.999 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1844141745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1844141745},{"type":"v1","addr":"172.21.15.186:6833","nonce":1844141745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6833/2960763758","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":2960763758},{"type":"v1","addr":"172.21.15.86:6833","nonce":2960763758}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24470,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/4131134219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":4131134219},{"type":"v1","addr":"172.21.15.28:6835","nonce":4131134219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24481,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/1727869064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1727869064},{"type":"v1","addr":"172.21.15.186:6835","nonce":1727869064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24482,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2938260704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2938260704},{"type":"v1","addr":"172.21.15.28:6837","nonce":2938260704}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24494,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3473700910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3473700910},{"type":"v1","addr":"172.21.15.86:6835","nonce":3473700910}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-12T22:06:33.426986+0000","modified":"2024-05-12T22:10:42.498031+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24581,"mds_1":24550,"mds_2":24532},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24499":{"gid":24499,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6837/3600809235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":3600809235},{"type":"v1","addr":"172.21.15.186:6837","nonce":3600809235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"i","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6839/2399610807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":2399610807},{"type":"v1","addr":"172.21.15.186:6839","nonce":2399610807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24530":{"gid":24530,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.86:6837/3782448884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":3782448884},{"type":"v1","addr":"172.21.15.86:6837","nonce":3782448884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24532":{"gid":24532,"name":"g","rank":2,"incarnation":21,"state":"up:active","state_seq":6,"addr":"172.21.15.28:6839/123892158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":123892158},{"type":"v1","addr":"172.21.15.28:6839","nonce":123892158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24550":{"gid":24550,"name":"a","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.28:6841/761183870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":761183870},{"type":"v1","addr":"172.21.15.28:6841","nonce":761183870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24581":{"gid":24581,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6839/1251999130","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6838","nonce":1251999130},{"type":"v1","addr":"172.21.15.86:6839","nonce":1251999130}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24581,"qdb_cluster":[24550,24532,24581]},"id":1}]} 2024-05-12T22:10:45.999 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 25 2024-05-12T22:10:46.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.000+0000 7fd6ed600640 1 -- 172.21.15.28:0/2970932550 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fd6cc0a2d00 msgr2=0x7fd6cc0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:10:46.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.000+0000 7fd6ed600640 1 --2- 172.21.15.28:0/2970932550 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fd6cc0a2d00 0x7fd6cc0a51c0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd6e810e970 tx=0x7fd6d0005f90 comp rx=0 tx=0).stop 2024-05-12T22:10:46.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.000+0000 7fd6ed600640 1 -- 172.21.15.28:0/2970932550 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6e8101610 msgr2=0x7fd6e819a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:10:46.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.000+0000 7fd6ed600640 1 --2- 172.21.15.28:0/2970932550 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6e8101610 0x7fd6e819a0e0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fd6d8000f80 tx=0x7fd6d8031e60 comp rx=0 tx=0).stop 2024-05-12T22:10:46.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.001+0000 7fd6ed600640 1 -- 172.21.15.28:0/2970932550 shutdown_connections 2024-05-12T22:10:46.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.001+0000 7fd6ed600640 1 --2- 172.21.15.28:0/2970932550 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fd6cc0a2d00 0x7fd6cc0a51c0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:46.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.001+0000 7fd6ed600640 1 --2- 172.21.15.28:0/2970932550 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd6e81076b0 0x7fd6e819e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:46.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.001+0000 7fd6ed600640 1 --2- 172.21.15.28:0/2970932550 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd6e8101fd0 0x7fd6e819a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:46.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.001+0000 7fd6ed600640 1 --2- 172.21.15.28:0/2970932550 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6e8101610 0x7fd6e819a0e0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:46.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.001+0000 7fd6ed600640 1 -- 172.21.15.28:0/2970932550 >> 172.21.15.28:0/2970932550 conn(0x7fd6e80fd160 msgr2=0x7fd6e80fe640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:10:46.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.001+0000 7fd6ed600640 1 -- 172.21.15.28:0/2970932550 shutdown_connections 2024-05-12T22:10:46.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.001+0000 7fd6ed600640 1 -- 172.21.15.28:0/2970932550 wait complete. 2024-05-12T22:10:46.018 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:10:45 smithi028 ceph-mon[29847]: pgmap v56: 97 pgs: 97 active+clean; 16 MiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 17 op/s 2024-05-12T22:10:46.022 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-05-12T22:10:46.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:10:45 smithi186 ceph-mon[34785]: pgmap v56: 97 pgs: 97 active+clean; 16 MiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 17 op/s 2024-05-12T22:10:46.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:10:45 smithi086 ceph-mon[34297]: pgmap v56: 97 pgs: 97 active+clean; 16 MiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 17 op/s 2024-05-12T22:10:46.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.243+0000 7f9973200640 1 -- 172.21.15.28:0/852388800 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9974100fb0 msgr2=0x7f9974105470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:10:46.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.243+0000 7f9973200640 1 --2- 172.21.15.28:0/852388800 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9974100fb0 0x7f9974105470 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f9960009860 tx=0x7f996002a3c0 comp rx=0 tx=0).stop 2024-05-12T22:10:46.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.243+0000 7f9973200640 1 -- 172.21.15.28:0/852388800 shutdown_connections 2024-05-12T22:10:46.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.243+0000 7f9973200640 1 --2- 172.21.15.28:0/852388800 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9974105a90 0x7f9974105f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:46.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.243+0000 7f9973200640 1 --2- 172.21.15.28:0/852388800 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9974100fb0 0x7f9974105470 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:46.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.243+0000 7f9973200640 1 --2- 172.21.15.28:0/852388800 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9974100690 0x7f9974100a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:46.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.243+0000 7f9973200640 1 -- 172.21.15.28:0/852388800 >> 172.21.15.28:0/852388800 conn(0x7f99740fc080 msgr2=0x7f99740fe4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:10:46.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.243+0000 7f9973200640 1 -- 172.21.15.28:0/852388800 shutdown_connections 2024-05-12T22:10:46.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.243+0000 7f9973200640 1 -- 172.21.15.28:0/852388800 wait complete. 2024-05-12T22:10:46.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.244+0000 7f9973200640 1 Processor -- start 2024-05-12T22:10:46.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.244+0000 7f9973200640 1 -- start start 2024-05-12T22:10:46.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.245+0000 7f9973200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9974100690 0x7f997419e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:46.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.245+0000 7f9973200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9974100fb0 0x7f997419eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:46.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.245+0000 7f9973200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9974105a90 0x7f99741a2e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:46.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.245+0000 7f9973200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9974118090 con 0x7f9974105a90 2024-05-12T22:10:46.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.245+0000 7f9973200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f9974117f10 con 0x7f9974100fb0 2024-05-12T22:10:46.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.245+0000 7f9973200640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f9974118210 con 0x7f9974100690 2024-05-12T22:10:46.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.245+0000 7f9972800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9974105a90 0x7f99741a2e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:46.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.245+0000 7f9971e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9974100690 0x7f997419e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:46.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.245+0000 7f9971400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9974100fb0 0x7f997419eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:46.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.245+0000 7f9971e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9974100690 0x7f997419e5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:48322/0 (socket says 172.21.15.28:48322) 2024-05-12T22:10:46.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.245+0000 7f9972800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9974105a90 0x7f99741a2e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49054/0 (socket says 172.21.15.28:49054) 2024-05-12T22:10:46.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.245+0000 7f9971e00640 1 -- 172.21.15.28:0/4111318270 learned_addr learned my addr 172.21.15.28:0/4111318270 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:10:46.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.246+0000 7f9971400640 1 -- 172.21.15.28:0/4111318270 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9974100690 msgr2=0x7f997419e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:10:46.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.246+0000 7f9971400640 1 --2- 172.21.15.28:0/4111318270 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9974100690 0x7f997419e5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:46.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.246+0000 7f9971400640 1 -- 172.21.15.28:0/4111318270 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9974105a90 msgr2=0x7f99741a2e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:10:46.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.246+0000 7f9971400640 1 --2- 172.21.15.28:0/4111318270 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9974105a90 0x7f99741a2e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:46.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.246+0000 7f9971400640 1 -- 172.21.15.28:0/4111318270 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99741a35a0 con 0x7f9974100fb0 2024-05-12T22:10:46.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.246+0000 7f9971400640 1 --2- 172.21.15.28:0/4111318270 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9974100fb0 0x7f997419eaf0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f9960035040 tx=0x7f9960030430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:10:46.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.246+0000 7f996aa00640 1 -- 172.21.15.28:0/4111318270 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f996003f020 con 0x7f9974100fb0 2024-05-12T22:10:46.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.247+0000 7f9973200640 1 -- 172.21.15.28:0/4111318270 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99741a3830 con 0x7f9974100fb0 2024-05-12T22:10:46.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.247+0000 7f996aa00640 1 -- 172.21.15.28:0/4111318270 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f99600042c0 con 0x7f9974100fb0 2024-05-12T22:10:46.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.247+0000 7f996aa00640 1 -- 172.21.15.28:0/4111318270 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9960037040 con 0x7f9974100fb0 2024-05-12T22:10:46.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.247+0000 7f9973200640 1 -- 172.21.15.28:0/4111318270 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99741ab120 con 0x7f9974100fb0 2024-05-12T22:10:46.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.250+0000 7f997be50740 1 Processor -- start 2024-05-12T22:10:46.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.250+0000 7f997be50740 1 -- start start 2024-05-12T22:10:46.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.250+0000 7f996aa00640 1 -- 172.21.15.28:0/4111318270 <== mon.2 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7f9960042020 con 0x7f9974100fb0 2024-05-12T22:10:46.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.250+0000 7f997be50740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bb73771b60 0x55bb73771f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:46.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.250+0000 7f997be50740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55bb73772480 0x55bb7377c610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:46.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.250+0000 7f997be50740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb7377cb50 0x55bb7377ef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:46.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.250+0000 7f997be50740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55bb736d6a30 con 0x55bb7377cb50 2024-05-12T22:10:46.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.250+0000 7f997be50740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x55bb7367e700 con 0x55bb73772480 2024-05-12T22:10:46.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.250+0000 7f997be50740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55bb736d29f0 con 0x55bb73771b60 2024-05-12T22:10:46.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.251+0000 7f9971400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb7377cb50 0x55bb7377ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:46.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.251+0000 7f9971e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55bb73772480 0x55bb7377c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:46.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.251+0000 7f9971400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb7377cb50 0x55bb7377ef40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49058/0 (socket says 172.21.15.28:49058) 2024-05-12T22:10:46.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.251+0000 7f9971400640 1 -- 172.21.15.28:0/1110648079 learned_addr learned my addr 172.21.15.28:0/1110648079 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:10:46.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.251+0000 7f9972800640 1 --2- 172.21.15.28:0/1110648079 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bb73771b60 0x55bb73771f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:46.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.251+0000 7f9972800640 1 -- 172.21.15.28:0/1110648079 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55bb73772480 msgr2=0x55bb7377c610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:10:46.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.251+0000 7f9972800640 1 --2- 172.21.15.28:0/1110648079 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55bb73772480 0x55bb7377c610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:46.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.251+0000 7f9972800640 1 -- 172.21.15.28:0/1110648079 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb7377cb50 msgr2=0x55bb7377ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:10:46.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.251+0000 7f9972800640 1 --2- 172.21.15.28:0/1110648079 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb7377cb50 0x55bb7377ef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:46.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.251+0000 7f9972800640 1 -- 172.21.15.28:0/1110648079 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bb733ea3f0 con 0x55bb73771b60 2024-05-12T22:10:46.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.251+0000 7f9971e00640 1 --2- 172.21.15.28:0/1110648079 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55bb73772480 0x55bb7377c610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:10:46.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.251+0000 7f9971400640 1 --2- 172.21.15.28:0/1110648079 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb7377cb50 0x55bb7377ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:10:46.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.252+0000 7f9972800640 1 --2- 172.21.15.28:0/1110648079 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bb73771b60 0x55bb73771f40 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f9964009b80 tx=0x7f9964011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=91a0a9cbcf56d9e4 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:10:46.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.252+0000 7f996aa00640 1 --2- 172.21.15.28:0/4111318270 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f99580a2d00 0x7f99580a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:46.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.252+0000 7f995f400640 1 -- 172.21.15.28:0/1110648079 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9964004050 con 0x55bb73771b60 2024-05-12T22:10:46.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.252+0000 7f995f400640 1 -- 172.21.15.28:0/1110648079 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9964005500 con 0x55bb73771b60 2024-05-12T22:10:46.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.252+0000 7f995f400640 1 -- 172.21.15.28:0/1110648079 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9964004d30 con 0x55bb73771b60 2024-05-12T22:10:46.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.252+0000 7f996aa00640 1 -- 172.21.15.28:0/4111318270 <== mon.2 v2:172.21.15.86:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f99600eeda0 con 0x7f9974100fb0 2024-05-12T22:10:46.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.252+0000 7f9971e00640 1 --2- 172.21.15.28:0/4111318270 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f99580a2d00 0x7f99580a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:46.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.253+0000 7f997be50740 1 -- 172.21.15.28:0/1110648079 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bb73771b60 msgr2=0x55bb73771f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:10:46.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.253+0000 7f997be50740 1 --2- 172.21.15.28:0/1110648079 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bb73771b60 0x55bb73771f40 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f9964009b80 tx=0x7f9964011810 comp rx=0 tx=0).stop 2024-05-12T22:10:46.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.253+0000 7f997be50740 1 -- 172.21.15.28:0/1110648079 shutdown_connections 2024-05-12T22:10:46.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.253+0000 7f997be50740 1 --2- 172.21.15.28:0/1110648079 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb7377cb50 0x55bb7377ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:46.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.253+0000 7f997be50740 1 --2- 172.21.15.28:0/1110648079 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55bb73772480 0x55bb7377c610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:46.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.253+0000 7f997be50740 1 --2- 172.21.15.28:0/1110648079 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bb73771b60 0x55bb73771f40 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:46.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.253+0000 7f9971e00640 1 --2- 172.21.15.28:0/4111318270 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f99580a2d00 0x7f99580a51c0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x55bb73773120 tx=0x7f996c010040 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:10:46.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.253+0000 7f997be50740 1 -- 172.21.15.28:0/1110648079 >> 172.21.15.28:0/1110648079 conn(0x55bb7376de80 msgr2=0x55bb7376e260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:10:46.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.253+0000 7f997be50740 1 -- 172.21.15.28:0/1110648079 shutdown_connections 2024-05-12T22:10:46.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.253+0000 7f997be50740 1 -- 172.21.15.28:0/1110648079 wait complete. 2024-05-12T22:10:46.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.254+0000 7f997be50740 1 -- start start 2024-05-12T22:10:46.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.254+0000 7f997be50740 1 Processor -- start 2024-05-12T22:10:46.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.254+0000 7f997be50740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55bb73771b60 0x55bb73779b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:46.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.254+0000 7f997be50740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb73772480 0x55bb7377a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:46.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.254+0000 7f997be50740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bb7377cb50 0x55bb7377a610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:46.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.254+0000 7f997be50740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55bb7377f940 con 0x55bb73772480 2024-05-12T22:10:46.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.254+0000 7f997be50740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x55bb73691a70 con 0x55bb73771b60 2024-05-12T22:10:46.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.254+0000 7f997be50740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55bb7377fac0 con 0x55bb7377cb50 2024-05-12T22:10:46.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.254+0000 7f997be50740 10 client.0 _ll_register_callbacks cb 0x55bb73674ba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:10:46.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.255+0000 7f9971400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb73772480 0x55bb7377a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:46.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.255+0000 7f9971e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55bb73771b60 0x55bb73779b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:46.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.255+0000 7f9971e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55bb73771b60 0x55bb73779b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:41442/0 (socket says 172.21.15.28:41442) 2024-05-12T22:10:46.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.255+0000 7f9971400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb73772480 0x55bb7377a0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49064/0 (socket says 172.21.15.28:49064) 2024-05-12T22:10:46.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.255+0000 7f9971e00640 1 -- 172.21.15.28:0/95472373 learned_addr learned my addr 172.21.15.28:0/95472373 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:10:46.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.255+0000 7f9972800640 1 --2- 172.21.15.28:0/95472373 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bb7377cb50 0x55bb7377a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:46.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.255+0000 7f9971400640 1 -- 172.21.15.28:0/95472373 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bb7377cb50 msgr2=0x55bb7377a610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:10:46.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.255+0000 7f9971400640 1 --2- 172.21.15.28:0/95472373 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bb7377cb50 0x55bb7377a610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:46.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.255+0000 7f9971400640 1 -- 172.21.15.28:0/95472373 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55bb73771b60 msgr2=0x55bb73779b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:10:46.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.255+0000 7f9971400640 1 --2- 172.21.15.28:0/95472373 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55bb73771b60 0x55bb73779b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:46.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.255+0000 7f9971400640 1 -- 172.21.15.28:0/95472373 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bb734b47d0 con 0x55bb73772480 2024-05-12T22:10:46.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.255+0000 7f9971e00640 1 --2- 172.21.15.28:0/95472373 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55bb73771b60 0x55bb73779b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:10:46.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.255+0000 7f9971400640 1 --2- 172.21.15.28:0/95472373 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb73772480 0x55bb7377a0d0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f996002a9b0 tx=0x7f9960005bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19849efc3db18651 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:10:46.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.255+0000 7f9953e00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-12T22:10:46.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.256+0000 7f9953e00640 1 -- 172.21.15.28:0/95472373 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99600baff0 con 0x55bb73772480 2024-05-12T22:10:46.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.256+0000 7f9953e00640 1 -- 172.21.15.28:0/95472373 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f99600c3be0 con 0x55bb73772480 2024-05-12T22:10:46.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.256+0000 7f997be50740 1 -- 172.21.15.28:0/95472373 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9964005500 con 0x55bb73772480 2024-05-12T22:10:46.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.256+0000 7f9953e00640 1 -- 172.21.15.28:0/95472373 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f99600bae10 con 0x55bb73772480 2024-05-12T22:10:46.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.256+0000 7f997be50740 10 client.15597 fetch_fsmap learned FSMap version 25 2024-05-12T22:10:46.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.256+0000 7f997be50740 1 -- 172.21.15.28:0/95472373 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55bb7378bca0 con 0x55bb73772480 2024-05-12T22:10:46.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.256+0000 7f9953e00640 1 -- 172.21.15.28:0/95472373 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9964005500 con 0x55bb73772480 2024-05-12T22:10:46.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.256+0000 7f9953e00640 1 -- 172.21.15.28:0/95472373 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7f996003a030 con 0x55bb73772480 2024-05-12T22:10:46.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.257+0000 7f997be50740 10 client.15597 fetch_fsmap finished waiting for FSMap version 25 2024-05-12T22:10:46.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.257+0000 7f997be50740 10 client.15597 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.g 2024-05-12T22:10:46.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.257+0000 7f997be50740 20 client.15597 populate_metadata read hostname 'smithi028' 2024-05-12T22:10:46.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.257+0000 7f997be50740 1 --2- 172.21.15.28:0/95472373 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x55bb7378d040 0x55bb7378f470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:46.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.257+0000 7f997be50740 4 client.15597 mds_command: new command op to 24532 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-12T22:10:46.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.257+0000 7f997be50740 1 -- 172.21.15.28:0/95472373 --> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bb736e4160 con 0x55bb7378d040 2024-05-12T22:10:46.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.257+0000 7f9971e00640 1 --2- 172.21.15.28:0/95472373 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x55bb7378d040 0x55bb7378f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:46.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.258+0000 7f9971e00640 1 --2- 172.21.15.28:0/95472373 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x55bb7378d040 0x55bb7378f470 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a840d5774ffc532c server_cookie=a10739803e85d15d in_seq=0 out_seq=0 2024-05-12T22:10:46.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.258+0000 7f9953e00640 10 client.15597 ms_handle_connect on v2:172.21.15.28:6838/123892158 2024-05-12T22:10:46.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.258+0000 7f9953e00640 1 -- 172.21.15.28:0/95472373 <== mds.2 v2:172.21.15.28:6838/123892158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55bb736e4160 con 0x55bb7378d040 2024-05-12T22:10:46.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.258+0000 7f9953e00640 10 client.15597 handle_command_reply: tid=0 2024-05-12T22:10:46.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.258+0000 7f997be50740 1 client.15597 shutdown 2024-05-12T22:10:46.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.259+0000 7f997be50740 1 -- 172.21.15.28:0/95472373 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb73772480 msgr2=0x55bb7377a0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:10:46.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.259+0000 7f997be50740 1 --2- 172.21.15.28:0/95472373 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb73772480 0x55bb7377a0d0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f996002a9b0 tx=0x7f9960005bd0 comp rx=0 tx=0).stop 2024-05-12T22:10:46.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.259+0000 7f997be50740 1 -- 172.21.15.28:0/95472373 shutdown_connections 2024-05-12T22:10:46.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.259+0000 7f997be50740 1 --2- 172.21.15.28:0/95472373 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bb7377cb50 0x55bb7377a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:46.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.259+0000 7f997be50740 1 --2- 172.21.15.28:0/95472373 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb73772480 0x55bb7377a0d0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:46.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.259+0000 7f997be50740 1 --2- 172.21.15.28:0/95472373 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x55bb7378d040 0x55bb7378f470 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:46.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.259+0000 7f997be50740 1 --2- 172.21.15.28:0/95472373 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55bb73771b60 0x55bb73779b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:46.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.259+0000 7f9953e00640 0 client.15597 ms_handle_reset on v2:172.21.15.28:6838/123892158 2024-05-12T22:10:46.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.259+0000 7f997be50740 1 -- 172.21.15.28:0/95472373 >> 172.21.15.28:0/95472373 conn(0x55bb7376de80 msgr2=0x55bb73773c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:10:46.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.259+0000 7f997be50740 1 -- 172.21.15.28:0/95472373 shutdown_connections 2024-05-12T22:10:46.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.259+0000 7f997be50740 1 -- 172.21.15.28:0/95472373 wait complete. 2024-05-12T22:10:46.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.259+0000 7f997be50740 20 client.15597 trim_cache size 0 max 16384 2024-05-12T22:10:46.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.274+0000 7f997be50740 1 Processor -- start 2024-05-12T22:10:46.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.274+0000 7f997be50740 1 -- start start 2024-05-12T22:10:46.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.274+0000 7f997be50740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bb73771b60 0x55bb73788800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:46.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.274+0000 7f997be50740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb73772480 0x55bb73774300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:46.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.274+0000 7f997be50740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55bb7377cb50 0x55bb73794d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:46.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.274+0000 7f997be50740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f99741b6090 con 0x55bb73772480 2024-05-12T22:10:46.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.274+0000 7f997be50740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f99741b3960 con 0x55bb7377cb50 2024-05-12T22:10:46.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.274+0000 7f997be50740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f99741b17b0 con 0x55bb73771b60 2024-05-12T22:10:46.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.275+0000 7f9971400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb73772480 0x55bb73774300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:46.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.275+0000 7f9971e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bb73771b60 0x55bb73788800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:46.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.275+0000 7f9971e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bb73771b60 0x55bb73788800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:48330/0 (socket says 172.21.15.28:48330) 2024-05-12T22:10:46.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.275+0000 7f9971e00640 1 -- 172.21.15.28:0/2922599613 learned_addr learned my addr 172.21.15.28:0/2922599613 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:10:46.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.275+0000 7f9972800640 1 --2- 172.21.15.28:0/2922599613 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55bb7377cb50 0x55bb73794d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:46.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.275+0000 7f9971e00640 1 -- 172.21.15.28:0/2922599613 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55bb7377cb50 msgr2=0x55bb73794d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:10:46.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.275+0000 7f9971e00640 1 --2- 172.21.15.28:0/2922599613 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55bb7377cb50 0x55bb73794d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:46.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.275+0000 7f9971e00640 1 -- 172.21.15.28:0/2922599613 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb73772480 msgr2=0x55bb73774300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:10:46.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.275+0000 7f9971e00640 1 --2- 172.21.15.28:0/2922599613 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb73772480 0x55bb73774300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:46.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.275+0000 7f9971e00640 1 -- 172.21.15.28:0/2922599613 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bb7378faf0 con 0x55bb73771b60 2024-05-12T22:10:46.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.275+0000 7f9972800640 1 --2- 172.21.15.28:0/2922599613 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55bb7377cb50 0x55bb73794d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:10:46.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.275+0000 7f9971e00640 1 --2- 172.21.15.28:0/2922599613 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bb73771b60 0x55bb73788800 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f996c0098d0 tx=0x7f996c016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c22ab763fc348690 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:10:46.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.276+0000 7f995ea00640 1 -- 172.21.15.28:0/2922599613 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f996c020040 con 0x55bb73771b60 2024-05-12T22:10:46.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.276+0000 7f995ea00640 1 -- 172.21.15.28:0/2922599613 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f996c031070 con 0x55bb73771b60 2024-05-12T22:10:46.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.276+0000 7f995ea00640 1 -- 172.21.15.28:0/2922599613 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f996c027bb0 con 0x55bb73771b60 2024-05-12T22:10:46.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.276+0000 7f997be50740 1 -- 172.21.15.28:0/2922599613 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bb73771b60 msgr2=0x55bb73788800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:10:46.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.276+0000 7f997be50740 1 --2- 172.21.15.28:0/2922599613 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bb73771b60 0x55bb73788800 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f996c0098d0 tx=0x7f996c016000 comp rx=0 tx=0).stop 2024-05-12T22:10:46.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.277+0000 7f997be50740 1 -- 172.21.15.28:0/2922599613 shutdown_connections 2024-05-12T22:10:46.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.277+0000 7f997be50740 1 --2- 172.21.15.28:0/2922599613 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55bb7377cb50 0x55bb73794d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:46.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.277+0000 7f997be50740 1 --2- 172.21.15.28:0/2922599613 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb73772480 0x55bb73774300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:46.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.277+0000 7f997be50740 1 --2- 172.21.15.28:0/2922599613 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bb73771b60 0x55bb73788800 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:46.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.277+0000 7f997be50740 1 -- 172.21.15.28:0/2922599613 >> 172.21.15.28:0/2922599613 conn(0x55bb7376de80 msgr2=0x55bb7376d000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:10:46.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.277+0000 7f997be50740 1 -- 172.21.15.28:0/2922599613 shutdown_connections 2024-05-12T22:10:46.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.277+0000 7f997be50740 1 -- 172.21.15.28:0/2922599613 wait complete. 2024-05-12T22:10:46.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.277+0000 7f997be50740 1 -- start start 2024-05-12T22:10:46.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.277+0000 7f997be50740 1 Processor -- start 2024-05-12T22:10:46.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.278+0000 7f997be50740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb73771b60 0x55bb737851d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:46.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.278+0000 7f997be50740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bb73772480 0x55bb73780540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:46.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.278+0000 7f997be50740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55bb7377cb50 0x55bb73780b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:46.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.278+0000 7f997be50740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f99741b1e20 con 0x55bb73771b60 2024-05-12T22:10:46.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.278+0000 7f997be50740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f99741b3290 con 0x55bb7377cb50 2024-05-12T22:10:46.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.278+0000 7f997be50740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f99741b24f0 con 0x55bb73772480 2024-05-12T22:10:46.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.278+0000 7f997be50740 10 client.0 _ll_register_callbacks cb 0x7f99600f25d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:10:46.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.278+0000 7f9971e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb73771b60 0x55bb737851d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:46.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.278+0000 7f9971e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb73771b60 0x55bb737851d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49076/0 (socket says 172.21.15.28:49076) 2024-05-12T22:10:46.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.278+0000 7f9971400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bb73772480 0x55bb73780540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:46.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.278+0000 7f9971e00640 1 -- 172.21.15.28:0/296909521 learned_addr learned my addr 172.21.15.28:0/296909521 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:10:46.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.278+0000 7f9972800640 1 --2- 172.21.15.28:0/296909521 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55bb7377cb50 0x55bb73780b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:46.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.279+0000 7f9971400640 1 -- 172.21.15.28:0/296909521 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55bb7377cb50 msgr2=0x55bb73780b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:10:46.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.279+0000 7f9971400640 1 --2- 172.21.15.28:0/296909521 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55bb7377cb50 0x55bb73780b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:46.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.279+0000 7f9971400640 1 -- 172.21.15.28:0/296909521 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb73771b60 msgr2=0x55bb737851d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:10:46.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.279+0000 7f9971400640 1 --2- 172.21.15.28:0/296909521 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb73771b60 0x55bb737851d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:46.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.279+0000 7f9971400640 1 -- 172.21.15.28:0/296909521 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bb73781350 con 0x55bb73772480 2024-05-12T22:10:46.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.279+0000 7f9971400640 1 --2- 172.21.15.28:0/296909521 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bb73772480 0x55bb73780540 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f99600bb1b0 tx=0x7f99600f2990 comp rx=0 tx=0).ready entity=mon.1 client_cookie=62a53a9710f475 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:10:46.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.279+0000 7f9953e00640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-12T22:10:46.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.279+0000 7f9953e00640 1 -- 172.21.15.28:0/296909521 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99600baff0 con 0x55bb73772480 2024-05-12T22:10:46.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.279+0000 7f9953e00640 1 -- 172.21.15.28:0/296909521 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f99600c3be0 con 0x55bb73772480 2024-05-12T22:10:46.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.279+0000 7f997be50740 1 -- 172.21.15.28:0/296909521 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f996c031070 con 0x55bb73772480 2024-05-12T22:10:46.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.279+0000 7f9953e00640 1 -- 172.21.15.28:0/296909521 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f99600bf7b0 con 0x55bb73772480 2024-05-12T22:10:46.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.280+0000 7f997be50740 10 client.25675 fetch_fsmap learned FSMap version 25 2024-05-12T22:10:46.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.280+0000 7f997be50740 1 -- 172.21.15.28:0/296909521 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55bb737f36f0 con 0x55bb73772480 2024-05-12T22:10:46.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.280+0000 7f9953e00640 1 -- 172.21.15.28:0/296909521 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f996c031070 con 0x55bb73772480 2024-05-12T22:10:46.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.280+0000 7f9953e00640 1 -- 172.21.15.28:0/296909521 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7f996003a040 con 0x55bb73772480 2024-05-12T22:10:46.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.280+0000 7f997be50740 10 client.25675 fetch_fsmap finished waiting for FSMap version 25 2024-05-12T22:10:46.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.280+0000 7f997be50740 10 client.25675 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.g 2024-05-12T22:10:46.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.280+0000 7f997be50740 20 client.25675 populate_metadata read hostname 'smithi028' 2024-05-12T22:10:46.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.280+0000 7f997be50740 1 --2- 172.21.15.28:0/296909521 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x55bb737f3ce0 0x55bb737f6100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:46.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.280+0000 7f997be50740 4 client.25675 mds_command: new command op to 24532 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-12T22:10:46.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.280+0000 7f997be50740 1 -- 172.21.15.28:0/296909521 --> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55bb736de4b0 con 0x55bb737f3ce0 2024-05-12T22:10:46.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.280+0000 7f9971e00640 1 --2- 172.21.15.28:0/296909521 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x55bb737f3ce0 0x55bb737f6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:46.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.281+0000 7f9971e00640 1 --2- 172.21.15.28:0/296909521 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x55bb737f3ce0 0x55bb737f6100 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=4887207a1cb12f8 server_cookie=b0ccedc978ed9b4a in_seq=0 out_seq=0 2024-05-12T22:10:46.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.281+0000 7f9953e00640 10 client.25675 ms_handle_connect on v2:172.21.15.28:6838/123892158 2024-05-12T22:10:46.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.281+0000 7f9953e00640 1 -- 172.21.15.28:0/296909521 <== mds.2 v2:172.21.15.28:6838/123892158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55bb736de4b0 con 0x55bb737f3ce0 2024-05-12T22:10:46.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.281+0000 7f9953e00640 10 client.25675 handle_command_reply: tid=0 2024-05-12T22:10:46.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.281+0000 7f997be50740 1 client.25675 shutdown 2024-05-12T22:10:46.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.281+0000 7f997be50740 1 -- 172.21.15.28:0/296909521 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bb73772480 msgr2=0x55bb73780540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:10:46.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.281+0000 7f997be50740 1 --2- 172.21.15.28:0/296909521 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bb73772480 0x55bb73780540 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f99600bb1b0 tx=0x7f99600f2990 comp rx=0 tx=0).stop 2024-05-12T22:10:46.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.281+0000 7f997be50740 1 -- 172.21.15.28:0/296909521 shutdown_connections 2024-05-12T22:10:46.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.281+0000 7f997be50740 1 --2- 172.21.15.28:0/296909521 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x55bb737f3ce0 0x55bb737f6100 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:46.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.281+0000 7f997be50740 1 --2- 172.21.15.28:0/296909521 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55bb7377cb50 0x55bb73780b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:46.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.281+0000 7f997be50740 1 --2- 172.21.15.28:0/296909521 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bb73772480 0x55bb73780540 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:46.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.281+0000 7f997be50740 1 --2- 172.21.15.28:0/296909521 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb73771b60 0x55bb737851d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:46.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.281+0000 7f9953e00640 0 client.25675 ms_handle_reset on v2:172.21.15.28:6838/123892158 2024-05-12T22:10:46.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.282+0000 7f997be50740 1 -- 172.21.15.28:0/296909521 >> 172.21.15.28:0/296909521 conn(0x55bb7376de80 msgr2=0x55bb73779b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:10:46.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.282+0000 7f997be50740 1 -- 172.21.15.28:0/296909521 shutdown_connections 2024-05-12T22:10:46.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.282+0000 7f997be50740 1 -- 172.21.15.28:0/296909521 wait complete. 2024-05-12T22:10:46.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.282+0000 7f997be50740 20 client.25675 trim_cache size 0 max 16384 2024-05-12T22:10:46.285 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-05-12T22:10:46.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.282+0000 7f9973200640 1 -- 172.21.15.28:0/4111318270 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f99580a2d00 msgr2=0x7f99580a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:10:46.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.282+0000 7f9973200640 1 --2- 172.21.15.28:0/4111318270 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f99580a2d00 0x7f99580a51c0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x55bb73773120 tx=0x7f996c010040 comp rx=0 tx=0).stop 2024-05-12T22:10:46.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.282+0000 7f9973200640 1 -- 172.21.15.28:0/4111318270 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9974100fb0 msgr2=0x7f997419eaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:10:46.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.282+0000 7f9973200640 1 --2- 172.21.15.28:0/4111318270 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9974100fb0 0x7f997419eaf0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f9960035040 tx=0x7f9960030430 comp rx=0 tx=0).stop 2024-05-12T22:10:46.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.283+0000 7f9973200640 1 -- 172.21.15.28:0/4111318270 shutdown_connections 2024-05-12T22:10:46.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.283+0000 7f9973200640 1 --2- 172.21.15.28:0/4111318270 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f99580a2d00 0x7f99580a51c0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:46.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.283+0000 7f9973200640 1 --2- 172.21.15.28:0/4111318270 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9974105a90 0x7f99741a2e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:46.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.283+0000 7f9973200640 1 --2- 172.21.15.28:0/4111318270 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9974100fb0 0x7f997419eaf0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:46.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.283+0000 7f9973200640 1 --2- 172.21.15.28:0/4111318270 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9974100690 0x7f997419e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:46.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.283+0000 7f9973200640 1 -- 172.21.15.28:0/4111318270 >> 172.21.15.28:0/4111318270 conn(0x7f99740fc080 msgr2=0x7f99740fd0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:10:46.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.283+0000 7f9973200640 1 -- 172.21.15.28:0/4111318270 shutdown_connections 2024-05-12T22:10:46.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.283+0000 7f9973200640 1 -- 172.21.15.28:0/4111318270 wait complete. 2024-05-12T22:10:46.303 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-05-12T22:10:46.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.513+0000 7fa8c3200640 1 -- 172.21.15.28:0/705333040 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa8bc1098f0 msgr2=0x7fa8bc109cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:10:46.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.513+0000 7fa8c3200640 1 --2- 172.21.15.28:0/705333040 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa8bc1098f0 0x7fa8bc109cd0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fa8b0009930 tx=0x7fa8b002a440 comp rx=0 tx=0).stop 2024-05-12T22:10:46.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.513+0000 7fa8c3200640 1 -- 172.21.15.28:0/705333040 shutdown_connections 2024-05-12T22:10:46.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.513+0000 7fa8c3200640 1 --2- 172.21.15.28:0/705333040 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa8bc1042a0 0x7fa8bc1107b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:46.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.513+0000 7fa8c3200640 1 --2- 172.21.15.28:0/705333040 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa8bc1038e0 0x7fa8bc103d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:46.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.513+0000 7fa8c3200640 1 --2- 172.21.15.28:0/705333040 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa8bc1098f0 0x7fa8bc109cd0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:46.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.513+0000 7fa8c3200640 1 -- 172.21.15.28:0/705333040 >> 172.21.15.28:0/705333040 conn(0x7fa8bc0ff370 msgr2=0x7fa8bc101790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:10:46.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.513+0000 7fa8c3200640 1 -- 172.21.15.28:0/705333040 shutdown_connections 2024-05-12T22:10:46.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.513+0000 7fa8c3200640 1 -- 172.21.15.28:0/705333040 wait complete. 2024-05-12T22:10:46.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.514+0000 7fa8c3200640 1 Processor -- start 2024-05-12T22:10:46.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.514+0000 7fa8c3200640 1 -- start start 2024-05-12T22:10:46.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.515+0000 7fa8c3200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa8bc1038e0 0x7fa8bc19a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:46.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.515+0000 7fa8c3200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa8bc1042a0 0x7fa8bc19a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:46.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.515+0000 7fa8c3200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa8bc1098f0 0x7fa8bc19eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:46.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.515+0000 7fa8c3200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa8bc113d20 con 0x7fa8bc1098f0 2024-05-12T22:10:46.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.515+0000 7fa8c3200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fa8bc113ba0 con 0x7fa8bc1042a0 2024-05-12T22:10:46.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.515+0000 7fa8c3200640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa8bc113ea0 con 0x7fa8bc1038e0 2024-05-12T22:10:46.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.515+0000 7fa8c2800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa8bc1098f0 0x7fa8bc19eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:46.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.515+0000 7fa8c2800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa8bc1098f0 0x7fa8bc19eac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49082/0 (socket says 172.21.15.28:49082) 2024-05-12T22:10:46.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.515+0000 7fa8c2800640 1 -- 172.21.15.28:0/1541003614 learned_addr learned my addr 172.21.15.28:0/1541003614 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:10:46.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.515+0000 7fa8c1400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa8bc1042a0 0x7fa8bc19a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:46.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.515+0000 7fa8c1e00640 1 --2- 172.21.15.28:0/1541003614 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa8bc1038e0 0x7fa8bc19a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:46.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.516+0000 7fa8c1400640 1 -- 172.21.15.28:0/1541003614 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa8bc1038e0 msgr2=0x7fa8bc19a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:10:46.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.516+0000 7fa8c1400640 1 --2- 172.21.15.28:0/1541003614 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa8bc1038e0 0x7fa8bc19a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:46.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.516+0000 7fa8c1400640 1 -- 172.21.15.28:0/1541003614 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa8bc1098f0 msgr2=0x7fa8bc19eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:10:46.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.516+0000 7fa8c1400640 1 --2- 172.21.15.28:0/1541003614 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa8bc1098f0 0x7fa8bc19eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:46.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.516+0000 7fa8c1400640 1 -- 172.21.15.28:0/1541003614 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8bc19f1f0 con 0x7fa8bc1042a0 2024-05-12T22:10:46.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.516+0000 7fa8c2800640 1 --2- 172.21.15.28:0/1541003614 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa8bc1098f0 0x7fa8bc19eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:10:46.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.516+0000 7fa8c1400640 1 --2- 172.21.15.28:0/1541003614 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa8bc1042a0 0x7fa8bc19a740 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fa8ac00e9f0 tx=0x7fa8ac00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:10:46.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.516+0000 7fa8b6a00640 1 -- 172.21.15.28:0/1541003614 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8ac00ccc0 con 0x7fa8bc1042a0 2024-05-12T22:10:46.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.516+0000 7fa8c3200640 1 -- 172.21.15.28:0/1541003614 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8bc19f4e0 con 0x7fa8bc1042a0 2024-05-12T22:10:46.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.516+0000 7fa8b6a00640 1 -- 172.21.15.28:0/1541003614 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa8ac010910 con 0x7fa8bc1042a0 2024-05-12T22:10:46.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.516+0000 7fa8b6a00640 1 -- 172.21.15.28:0/1541003614 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8ac018960 con 0x7fa8bc1042a0 2024-05-12T22:10:46.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.516+0000 7fa8c3200640 1 -- 172.21.15.28:0/1541003614 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8bc1a6d70 con 0x7fa8bc1042a0 2024-05-12T22:10:46.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.520+0000 7fa8c7636740 1 Processor -- start 2024-05-12T22:10:46.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.520+0000 7fa8c7636740 1 -- start start 2024-05-12T22:10:46.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.520+0000 7fa8c7636740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563a88cd4b60 0x563a88cd4f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:46.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.520+0000 7fa8c7636740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563a88cd5480 0x563a88cdf610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:46.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.520+0000 7fa8c7636740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563a88cdfb50 0x563a88ce1f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:46.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.520+0000 7fa8b6a00640 1 -- 172.21.15.28:0/1541003614 <== mon.2 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7fa8ac010ac0 con 0x7fa8bc1042a0 2024-05-12T22:10:46.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.520+0000 7fa8c7636740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x563a88c39a30 con 0x563a88cd4b60 2024-05-12T22:10:46.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.520+0000 7fa8c7636740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x563a88be1700 con 0x563a88cd5480 2024-05-12T22:10:46.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.520+0000 7fa8c7636740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x563a88c359f0 con 0x563a88cdfb50 2024-05-12T22:10:46.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.520+0000 7fa8c2800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563a88cd4b60 0x563a88cd4f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:46.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.520+0000 7fa8c2800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563a88cd4b60 0x563a88cd4f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49088/0 (socket says 172.21.15.28:49088) 2024-05-12T22:10:46.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.520+0000 7fa8c2800640 1 -- 172.21.15.28:0/3780924818 learned_addr learned my addr 172.21.15.28:0/3780924818 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:10:46.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.521+0000 7fa8c1e00640 1 --2- 172.21.15.28:0/3780924818 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563a88cd5480 0x563a88cdf610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:46.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.521+0000 7fa8c1400640 1 --2- 172.21.15.28:0/3780924818 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563a88cdfb50 0x563a88ce1f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:46.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.521+0000 7fa8c2800640 1 -- 172.21.15.28:0/3780924818 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563a88cdfb50 msgr2=0x563a88ce1f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:10:46.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.521+0000 7fa8c2800640 1 --2- 172.21.15.28:0/3780924818 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563a88cdfb50 0x563a88ce1f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:46.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.521+0000 7fa8c2800640 1 -- 172.21.15.28:0/3780924818 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563a88cd5480 msgr2=0x563a88cdf610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:10:46.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.521+0000 7fa8c2800640 1 --2- 172.21.15.28:0/3780924818 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563a88cd5480 0x563a88cdf610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:46.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.521+0000 7fa8c2800640 1 -- 172.21.15.28:0/3780924818 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563a8894d3f0 con 0x563a88cd4b60 2024-05-12T22:10:46.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.521+0000 7fa8c1e00640 1 --2- 172.21.15.28:0/3780924818 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563a88cd5480 0x563a88cdf610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:10:46.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.521+0000 7fa8c2800640 1 --2- 172.21.15.28:0/3780924818 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563a88cd4b60 0x563a88cd4f40 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fa8b8011850 tx=0x7fa8b8011d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c995c115e72429b server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:10:46.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.522+0000 7fa8ab000640 1 -- 172.21.15.28:0/3780924818 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8b8011330 con 0x563a88cd4b60 2024-05-12T22:10:46.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.522+0000 7fa8ab000640 1 -- 172.21.15.28:0/3780924818 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa8b800fe80 con 0x563a88cd4b60 2024-05-12T22:10:46.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.522+0000 7fa8ab000640 1 -- 172.21.15.28:0/3780924818 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8b80193e0 con 0x563a88cd4b60 2024-05-12T22:10:46.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.522+0000 7fa8b6a00640 1 --2- 172.21.15.28:0/1541003614 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fa8a40a2d00 0x7fa8a40a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:46.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.522+0000 7fa8b6a00640 1 -- 172.21.15.28:0/1541003614 <== mon.2 v2:172.21.15.86:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7fa8ac0cf7f0 con 0x7fa8bc1042a0 2024-05-12T22:10:46.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.522+0000 7fa8c1e00640 1 --2- 172.21.15.28:0/1541003614 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fa8a40a2d00 0x7fa8a40a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:46.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.522+0000 7fa8c7636740 1 -- 172.21.15.28:0/3780924818 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563a88cd4b60 msgr2=0x563a88cd4f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:10:46.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.522+0000 7fa8c7636740 1 --2- 172.21.15.28:0/3780924818 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563a88cd4b60 0x563a88cd4f40 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fa8b8011850 tx=0x7fa8b8011d10 comp rx=0 tx=0).stop 2024-05-12T22:10:46.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.523+0000 7fa8c7636740 1 -- 172.21.15.28:0/3780924818 shutdown_connections 2024-05-12T22:10:46.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.523+0000 7fa8c7636740 1 --2- 172.21.15.28:0/3780924818 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563a88cdfb50 0x563a88ce1f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:46.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.523+0000 7fa8c7636740 1 --2- 172.21.15.28:0/3780924818 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563a88cd5480 0x563a88cdf610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:46.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.523+0000 7fa8c7636740 1 --2- 172.21.15.28:0/3780924818 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563a88cd4b60 0x563a88cd4f40 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:46.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.523+0000 7fa8c7636740 1 -- 172.21.15.28:0/3780924818 >> 172.21.15.28:0/3780924818 conn(0x563a88cd0e80 msgr2=0x563a88cd1260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:10:46.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.523+0000 7fa8c7636740 1 -- 172.21.15.28:0/3780924818 shutdown_connections 2024-05-12T22:10:46.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.523+0000 7fa8c7636740 1 -- 172.21.15.28:0/3780924818 wait complete. 2024-05-12T22:10:46.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.523+0000 7fa8c1e00640 1 --2- 172.21.15.28:0/1541003614 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fa8a40a2d00 0x7fa8a40a51c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fa8b0006010 tx=0x7fa8b0000bc0 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:10:46.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.523+0000 7fa8c7636740 1 -- start start 2024-05-12T22:10:46.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.524+0000 7fa8c7636740 1 Processor -- start 2024-05-12T22:10:46.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.524+0000 7fa8c7636740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563a88cd4b60 0x563a88cdc9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:46.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.524+0000 7fa8c7636740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563a88cd5480 0x563a88cdcf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:46.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.524+0000 7fa8c7636740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563a88cdfb50 0x563a88cdd440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:46.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.524+0000 7fa8c7636740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x563a88ce2940 con 0x563a88cd4b60 2024-05-12T22:10:46.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.524+0000 7fa8c7636740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x563a88bf4a70 con 0x563a88cd5480 2024-05-12T22:10:46.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.524+0000 7fa8c7636740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x563a88ce2ac0 con 0x563a88cdfb50 2024-05-12T22:10:46.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.524+0000 7fa8c7636740 10 client.0 _ll_register_callbacks cb 0x563a88bd7ba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:10:46.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.524+0000 7fa8c1e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563a88cd4b60 0x563a88cdc9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:46.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.524+0000 7fa8c1e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563a88cd4b60 0x563a88cdc9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49090/0 (socket says 172.21.15.28:49090) 2024-05-12T22:10:46.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.524+0000 7fa8c1e00640 1 -- 172.21.15.28:0/2170784082 learned_addr learned my addr 172.21.15.28:0/2170784082 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:10:46.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.524+0000 7fa8c1400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563a88cd5480 0x563a88cdcf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:46.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.525+0000 7fa8c2800640 1 --2- 172.21.15.28:0/2170784082 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563a88cdfb50 0x563a88cdd440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:46.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.525+0000 7fa8c1e00640 1 -- 172.21.15.28:0/2170784082 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563a88cdfb50 msgr2=0x563a88cdd440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:10:46.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.525+0000 7fa8c1e00640 1 --2- 172.21.15.28:0/2170784082 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563a88cdfb50 0x563a88cdd440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:46.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.525+0000 7fa8c1e00640 1 -- 172.21.15.28:0/2170784082 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563a88cd5480 msgr2=0x563a88cdcf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:10:46.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.525+0000 7fa8c1e00640 1 --2- 172.21.15.28:0/2170784082 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563a88cd5480 0x563a88cdcf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:46.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.525+0000 7fa8c1e00640 1 -- 172.21.15.28:0/2170784082 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563a88a177d0 con 0x563a88cd4b60 2024-05-12T22:10:46.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.525+0000 7fa8c1400640 1 --2- 172.21.15.28:0/2170784082 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563a88cd5480 0x563a88cdcf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:10:46.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.525+0000 7fa8c1e00640 1 --2- 172.21.15.28:0/2170784082 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563a88cd4b60 0x563a88cdc9c0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fa8b00052c0 tx=0x7fa8b0009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c2a0a02c747e36a server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:10:46.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.525+0000 7fa89f400640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-12T22:10:46.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.526+0000 7fa89f400640 1 -- 172.21.15.28:0/2170784082 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8b004e070 con 0x563a88cd4b60 2024-05-12T22:10:46.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.526+0000 7fa89f400640 1 -- 172.21.15.28:0/2170784082 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7fa8b0004bf0 con 0x563a88cd4b60 2024-05-12T22:10:46.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.526+0000 7fa8c7636740 1 -- 172.21.15.28:0/2170784082 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x563a88c47160 con 0x563a88cd4b60 2024-05-12T22:10:46.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.526+0000 7fa89f400640 1 -- 172.21.15.28:0/2170784082 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fa8b005a050 con 0x563a88cd4b60 2024-05-12T22:10:46.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.526+0000 7fa8c7636740 10 client.15615 fetch_fsmap learned FSMap version 25 2024-05-12T22:10:46.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.526+0000 7fa8c7636740 1 -- 172.21.15.28:0/2170784082 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x563a88ceead0 con 0x563a88cd4b60 2024-05-12T22:10:46.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.526+0000 7fa89f400640 1 -- 172.21.15.28:0/2170784082 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x563a88c47160 con 0x563a88cd4b60 2024-05-12T22:10:46.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.527+0000 7fa89f400640 1 -- 172.21.15.28:0/2170784082 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7fa8b0003760 con 0x563a88cd4b60 2024-05-12T22:10:46.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.527+0000 7fa8c7636740 10 client.15615 fetch_fsmap finished waiting for FSMap version 25 2024-05-12T22:10:46.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.527+0000 7fa8c7636740 10 client.15615 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-05-12T22:10:46.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.527+0000 7fa8c7636740 20 client.15615 populate_metadata read hostname 'smithi028' 2024-05-12T22:10:46.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.527+0000 7fa8c7636740 1 --2- 172.21.15.28:0/2170784082 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x563a88ceff30 0x563a88cf2360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:46.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.527+0000 7fa8c7636740 4 client.15615 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-12T22:10:46.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.527+0000 7fa8c7636740 1 -- 172.21.15.28:0/2170784082 --> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563a88c414b0 con 0x563a88ceff30 2024-05-12T22:10:46.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.527+0000 7fa8c1400640 1 --2- 172.21.15.28:0/2170784082 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x563a88ceff30 0x563a88cf2360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:46.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.528+0000 7fa8c1400640 1 --2- 172.21.15.28:0/2170784082 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x563a88ceff30 0x563a88cf2360 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=78ccb2ae2a9cb619 server_cookie=389f18bec7a7a197 in_seq=0 out_seq=0 2024-05-12T22:10:46.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.528+0000 7fa89f400640 10 client.15615 ms_handle_connect on v2:172.21.15.28:6840/761183870 2024-05-12T22:10:46.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.528+0000 7fa89f400640 1 -- 172.21.15.28:0/2170784082 <== mds.1 v2:172.21.15.28:6840/761183870 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563a88c414b0 con 0x563a88ceff30 2024-05-12T22:10:46.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.528+0000 7fa89f400640 10 client.15615 handle_command_reply: tid=0 2024-05-12T22:10:46.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.528+0000 7fa8c7636740 1 client.15615 shutdown 2024-05-12T22:10:46.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.529+0000 7fa8c7636740 1 -- 172.21.15.28:0/2170784082 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563a88cd4b60 msgr2=0x563a88cdc9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:10:46.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.529+0000 7fa8c7636740 1 --2- 172.21.15.28:0/2170784082 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563a88cd4b60 0x563a88cdc9c0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fa8b00052c0 tx=0x7fa8b0009260 comp rx=0 tx=0).stop 2024-05-12T22:10:46.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.529+0000 7fa8c7636740 1 -- 172.21.15.28:0/2170784082 shutdown_connections 2024-05-12T22:10:46.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.529+0000 7fa8c7636740 1 --2- 172.21.15.28:0/2170784082 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x563a88ceff30 0x563a88cf2360 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:46.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.529+0000 7fa8c7636740 1 --2- 172.21.15.28:0/2170784082 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563a88cdfb50 0x563a88cdd440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:46.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.529+0000 7fa8c7636740 1 --2- 172.21.15.28:0/2170784082 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563a88cd5480 0x563a88cdcf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:46.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.529+0000 7fa89f400640 0 client.15615 ms_handle_reset on v2:172.21.15.28:6840/761183870 2024-05-12T22:10:46.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.529+0000 7fa8c7636740 1 --2- 172.21.15.28:0/2170784082 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563a88cd4b60 0x563a88cdc9c0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:46.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.529+0000 7fa8c7636740 1 -- 172.21.15.28:0/2170784082 >> 172.21.15.28:0/2170784082 conn(0x563a88cd0e80 msgr2=0x563a88cd6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:10:46.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.529+0000 7fa8c7636740 1 -- 172.21.15.28:0/2170784082 shutdown_connections 2024-05-12T22:10:46.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.529+0000 7fa8c7636740 1 -- 172.21.15.28:0/2170784082 wait complete. 2024-05-12T22:10:46.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.529+0000 7fa8c7636740 20 client.15615 trim_cache size 0 max 16384 2024-05-12T22:10:46.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.544+0000 7fa8c7636740 1 Processor -- start 2024-05-12T22:10:46.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.544+0000 7fa8c7636740 1 -- start start 2024-05-12T22:10:46.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.544+0000 7fa8c7636740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563a88cd4b60 0x563a88cd72c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:46.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.544+0000 7fa8c7636740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563a88cd5480 0x563a88cdc980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:46.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.544+0000 7fa8c7636740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563a88cdfb50 0x563a88cdc170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:46.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.544+0000 7fa8c7636740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa8bc1b1e50 con 0x563a88cdfb50 2024-05-12T22:10:46.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.544+0000 7fa8c7636740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fa8bc1af720 con 0x563a88cd5480 2024-05-12T22:10:46.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.544+0000 7fa8c7636740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa8bc1ad570 con 0x563a88cd4b60 2024-05-12T22:10:46.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.545+0000 7fa8c2800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563a88cdfb50 0x563a88cdc170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:46.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.545+0000 7fa8c2800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563a88cdfb50 0x563a88cdc170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49094/0 (socket says 172.21.15.28:49094) 2024-05-12T22:10:46.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.545+0000 7fa8c2800640 1 -- 172.21.15.28:0/2543154914 learned_addr learned my addr 172.21.15.28:0/2543154914 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:10:46.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.545+0000 7fa8c1400640 1 --2- 172.21.15.28:0/2543154914 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563a88cd5480 0x563a88cdc980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:46.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.545+0000 7fa8c1e00640 1 --2- 172.21.15.28:0/2543154914 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563a88cd4b60 0x563a88cd72c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:46.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.545+0000 7fa8c1400640 1 -- 172.21.15.28:0/2543154914 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563a88cd4b60 msgr2=0x563a88cd72c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:10:46.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.545+0000 7fa8c1400640 1 --2- 172.21.15.28:0/2543154914 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563a88cd4b60 0x563a88cd72c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:46.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.545+0000 7fa8c1400640 1 -- 172.21.15.28:0/2543154914 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563a88cdfb50 msgr2=0x563a88cdc170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:10:46.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.545+0000 7fa8c1400640 1 --2- 172.21.15.28:0/2543154914 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563a88cdfb50 0x563a88cdc170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:46.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.545+0000 7fa8c1400640 1 -- 172.21.15.28:0/2543154914 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563a88cdd060 con 0x563a88cd5480 2024-05-12T22:10:46.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.545+0000 7fa8c1400640 1 --2- 172.21.15.28:0/2543154914 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563a88cd5480 0x563a88cdc980 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fa8ac0ccf10 tx=0x7fa8ac0d1040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=96af936be7334754 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:10:46.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.546+0000 7fa8aa600640 1 -- 172.21.15.28:0/2543154914 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8ac0a4950 con 0x563a88cd5480 2024-05-12T22:10:46.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.546+0000 7fa8aa600640 1 -- 172.21.15.28:0/2543154914 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa8ac09a090 con 0x563a88cd5480 2024-05-12T22:10:46.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.546+0000 7fa8aa600640 1 -- 172.21.15.28:0/2543154914 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8ac09b440 con 0x563a88cd5480 2024-05-12T22:10:46.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.546+0000 7fa8c7636740 1 -- 172.21.15.28:0/2543154914 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563a88cd5480 msgr2=0x563a88cdc980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:10:46.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.546+0000 7fa8c7636740 1 --2- 172.21.15.28:0/2543154914 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563a88cd5480 0x563a88cdc980 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fa8ac0ccf10 tx=0x7fa8ac0d1040 comp rx=0 tx=0).stop 2024-05-12T22:10:46.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.546+0000 7fa8c7636740 1 -- 172.21.15.28:0/2543154914 shutdown_connections 2024-05-12T22:10:46.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.546+0000 7fa8c7636740 1 --2- 172.21.15.28:0/2543154914 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563a88cdfb50 0x563a88cdc170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:46.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.546+0000 7fa8c7636740 1 --2- 172.21.15.28:0/2543154914 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563a88cd5480 0x563a88cdc980 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:46.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.546+0000 7fa8c7636740 1 --2- 172.21.15.28:0/2543154914 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563a88cd4b60 0x563a88cd72c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:46.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.546+0000 7fa8c7636740 1 -- 172.21.15.28:0/2543154914 >> 172.21.15.28:0/2543154914 conn(0x563a88cd0e80 msgr2=0x563a88cd0000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:10:46.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.546+0000 7fa8c7636740 1 -- 172.21.15.28:0/2543154914 shutdown_connections 2024-05-12T22:10:46.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.546+0000 7fa8c7636740 1 -- 172.21.15.28:0/2543154914 wait complete. 2024-05-12T22:10:46.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.547+0000 7fa8c7636740 1 -- start start 2024-05-12T22:10:46.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.547+0000 7fa8c7636740 1 Processor -- start 2024-05-12T22:10:46.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.547+0000 7fa8c7636740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563a88cd4b60 0x563a88ce7e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:46.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.547+0000 7fa8c7636740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563a88cd5480 0x563a88ce32d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:46.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.547+0000 7fa8c7636740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563a88cdfb50 0x563a88ce3810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:46.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.547+0000 7fa8c7636740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa8bc1adbe0 con 0x563a88cdfb50 2024-05-12T22:10:46.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.548+0000 7fa8c7636740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fa8bc1af050 con 0x563a88cd4b60 2024-05-12T22:10:46.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.548+0000 7fa8c7636740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa8bc1ae2b0 con 0x563a88cd5480 2024-05-12T22:10:46.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.548+0000 7fa8c7636740 10 client.0 _ll_register_callbacks cb 0x7fa8b0004320 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:10:46.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.548+0000 7fa8c1e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563a88cd4b60 0x563a88ce7e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:46.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.548+0000 7fa8c1e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563a88cd4b60 0x563a88ce7e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:41498/0 (socket says 172.21.15.28:41498) 2024-05-12T22:10:46.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.548+0000 7fa8c1e00640 1 -- 172.21.15.28:0/3975967706 learned_addr learned my addr 172.21.15.28:0/3975967706 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:10:46.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.548+0000 7fa8c2800640 1 --2- 172.21.15.28:0/3975967706 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563a88cdfb50 0x563a88ce3810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:46.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.548+0000 7fa8c1400640 1 --2- 172.21.15.28:0/3975967706 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563a88cd5480 0x563a88ce32d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:46.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.548+0000 7fa8c1e00640 1 -- 172.21.15.28:0/3975967706 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563a88cd5480 msgr2=0x563a88ce32d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:10:46.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.548+0000 7fa8c1e00640 1 --2- 172.21.15.28:0/3975967706 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563a88cd5480 0x563a88ce32d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:46.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.548+0000 7fa8c1e00640 1 -- 172.21.15.28:0/3975967706 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563a88cdfb50 msgr2=0x563a88ce3810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:10:46.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.548+0000 7fa8c1e00640 1 --2- 172.21.15.28:0/3975967706 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563a88cdfb50 0x563a88ce3810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:46.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.548+0000 7fa8c1e00640 1 -- 172.21.15.28:0/3975967706 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563a88cd52b0 con 0x563a88cd4b60 2024-05-12T22:10:46.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.549+0000 7fa8c1e00640 1 --2- 172.21.15.28:0/3975967706 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563a88cd4b60 0x563a88ce7e20 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fa8b0003bb0 tx=0x7fa8b0005990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=69d21af71a89e530 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:10:46.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.549+0000 7fa89f400640 10 client.0 ms_handle_connect on v2:172.21.15.86:3300/0 2024-05-12T22:10:46.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.549+0000 7fa89f400640 1 -- 172.21.15.28:0/3975967706 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8b0063060 con 0x563a88cd4b60 2024-05-12T22:10:46.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.549+0000 7fa89f400640 1 -- 172.21.15.28:0/3975967706 <== mon.2 v2:172.21.15.86:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7fa8b00049f0 con 0x563a88cd4b60 2024-05-12T22:10:46.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.549+0000 7fa8c7636740 1 -- 172.21.15.28:0/3975967706 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa8ac09a090 con 0x563a88cd4b60 2024-05-12T22:10:46.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.549+0000 7fa89f400640 1 -- 172.21.15.28:0/3975967706 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fa8b004e070 con 0x563a88cd4b60 2024-05-12T22:10:46.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.549+0000 7fa8c7636740 10 client.25739 fetch_fsmap learned FSMap version 25 2024-05-12T22:10:46.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.549+0000 7fa8c7636740 1 -- 172.21.15.28:0/3975967706 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x563a88d56020 con 0x563a88cd4b60 2024-05-12T22:10:46.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.549+0000 7fa89f400640 1 -- 172.21.15.28:0/3975967706 <== mon.2 v2:172.21.15.86:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa8ac09a090 con 0x563a88cd4b60 2024-05-12T22:10:46.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.550+0000 7fa89f400640 1 -- 172.21.15.28:0/3975967706 <== mon.2 v2:172.21.15.86:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7fa8b0068480 con 0x563a88cd4b60 2024-05-12T22:10:46.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.550+0000 7fa8c7636740 10 client.25739 fetch_fsmap finished waiting for FSMap version 25 2024-05-12T22:10:46.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.550+0000 7fa8c7636740 10 client.25739 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-05-12T22:10:46.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.550+0000 7fa8c7636740 20 client.25739 populate_metadata read hostname 'smithi028' 2024-05-12T22:10:46.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.550+0000 7fa8c7636740 1 --2- 172.21.15.28:0/3975967706 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x563a88d567a0 0x563a88d58bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:46.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.550+0000 7fa8c7636740 4 client.25739 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-12T22:10:46.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.550+0000 7fa8c7636740 1 -- 172.21.15.28:0/3975967706 --> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563a88bbe6e0 con 0x563a88d567a0 2024-05-12T22:10:46.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.550+0000 7fa8c1400640 1 --2- 172.21.15.28:0/3975967706 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x563a88d567a0 0x563a88d58bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:46.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.551+0000 7fa8c1400640 1 --2- 172.21.15.28:0/3975967706 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x563a88d567a0 0x563a88d58bc0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c4539c1f8912997a server_cookie=7f858d37b3e0453d in_seq=0 out_seq=0 2024-05-12T22:10:46.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.551+0000 7fa89f400640 10 client.25739 ms_handle_connect on v2:172.21.15.28:6840/761183870 2024-05-12T22:10:46.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.551+0000 7fa89f400640 1 -- 172.21.15.28:0/3975967706 <== mds.1 v2:172.21.15.28:6840/761183870 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563a88bbe6e0 con 0x563a88d567a0 2024-05-12T22:10:46.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.551+0000 7fa89f400640 10 client.25739 handle_command_reply: tid=0 2024-05-12T22:10:46.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.551+0000 7fa8c7636740 1 client.25739 shutdown 2024-05-12T22:10:46.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.551+0000 7fa8c7636740 1 -- 172.21.15.28:0/3975967706 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563a88cd4b60 msgr2=0x563a88ce7e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:10:46.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.551+0000 7fa8c7636740 1 --2- 172.21.15.28:0/3975967706 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563a88cd4b60 0x563a88ce7e20 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fa8b0003bb0 tx=0x7fa8b0005990 comp rx=0 tx=0).stop 2024-05-12T22:10:46.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.551+0000 7fa8c7636740 1 -- 172.21.15.28:0/3975967706 shutdown_connections 2024-05-12T22:10:46.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.551+0000 7fa8c7636740 1 --2- 172.21.15.28:0/3975967706 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x563a88d567a0 0x563a88d58bc0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:46.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.551+0000 7fa8c7636740 1 --2- 172.21.15.28:0/3975967706 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563a88cdfb50 0x563a88ce3810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:46.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.551+0000 7fa8c7636740 1 --2- 172.21.15.28:0/3975967706 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563a88cd5480 0x563a88ce32d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:46.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.551+0000 7fa8c7636740 1 --2- 172.21.15.28:0/3975967706 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563a88cd4b60 0x563a88ce7e20 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:46.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.551+0000 7fa8c7636740 1 -- 172.21.15.28:0/3975967706 >> 172.21.15.28:0/3975967706 conn(0x563a88cd0e80 msgr2=0x563a88cde1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:10:46.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.551+0000 7fa89f400640 0 client.25739 ms_handle_reset on v2:172.21.15.28:6840/761183870 2024-05-12T22:10:46.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.551+0000 7fa8c7636740 1 -- 172.21.15.28:0/3975967706 shutdown_connections 2024-05-12T22:10:46.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.551+0000 7fa8c7636740 1 -- 172.21.15.28:0/3975967706 wait complete. 2024-05-12T22:10:46.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.552+0000 7fa8c7636740 20 client.25739 trim_cache size 0 max 16384 2024-05-12T22:10:46.556 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-05-12T22:10:46.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.552+0000 7fa8c3200640 1 -- 172.21.15.28:0/1541003614 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fa8a40a2d00 msgr2=0x7fa8a40a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:10:46.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.552+0000 7fa8c3200640 1 --2- 172.21.15.28:0/1541003614 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fa8a40a2d00 0x7fa8a40a51c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fa8b0006010 tx=0x7fa8b0000bc0 comp rx=0 tx=0).stop 2024-05-12T22:10:46.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.552+0000 7fa8c3200640 1 -- 172.21.15.28:0/1541003614 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa8bc1042a0 msgr2=0x7fa8bc19a740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:10:46.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.552+0000 7fa8c3200640 1 --2- 172.21.15.28:0/1541003614 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa8bc1042a0 0x7fa8bc19a740 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fa8ac00e9f0 tx=0x7fa8ac00eeb0 comp rx=0 tx=0).stop 2024-05-12T22:10:46.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.552+0000 7fa8c3200640 1 -- 172.21.15.28:0/1541003614 shutdown_connections 2024-05-12T22:10:46.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.552+0000 7fa8c3200640 1 --2- 172.21.15.28:0/1541003614 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fa8a40a2d00 0x7fa8a40a51c0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:46.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.552+0000 7fa8c3200640 1 --2- 172.21.15.28:0/1541003614 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa8bc1098f0 0x7fa8bc19eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:46.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.552+0000 7fa8c3200640 1 --2- 172.21.15.28:0/1541003614 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa8bc1042a0 0x7fa8bc19a740 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:46.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.552+0000 7fa8c3200640 1 --2- 172.21.15.28:0/1541003614 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa8bc1038e0 0x7fa8bc19a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:46.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.552+0000 7fa8c3200640 1 -- 172.21.15.28:0/1541003614 >> 172.21.15.28:0/1541003614 conn(0x7fa8bc0ff370 msgr2=0x7fa8bc0ffa80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:10:46.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.552+0000 7fa8c3200640 1 -- 172.21.15.28:0/1541003614 shutdown_connections 2024-05-12T22:10:46.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.552+0000 7fa8c3200640 1 -- 172.21.15.28:0/1541003614 wait complete. 2024-05-12T22:10:46.571 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-05-12T22:10:46.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.775+0000 7f24b7200640 1 -- 172.21.15.28:0/898542525 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f24b81098e0 msgr2=0x7f24b8109cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:10:46.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.775+0000 7f24b7200640 1 --2- 172.21.15.28:0/898542525 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f24b81098e0 0x7f24b8109cc0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f24a8009860 tx=0x7f24a802a340 comp rx=0 tx=0).stop 2024-05-12T22:10:46.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.775+0000 7f24b7200640 1 -- 172.21.15.28:0/898542525 shutdown_connections 2024-05-12T22:10:46.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.775+0000 7f24b7200640 1 --2- 172.21.15.28:0/898542525 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f24b8104290 0x7f24b81107a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:46.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.775+0000 7f24b7200640 1 --2- 172.21.15.28:0/898542525 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24b81038d0 0x7f24b8103d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:46.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.775+0000 7f24b7200640 1 --2- 172.21.15.28:0/898542525 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f24b81098e0 0x7f24b8109cc0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:46.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.775+0000 7f24b7200640 1 -- 172.21.15.28:0/898542525 >> 172.21.15.28:0/898542525 conn(0x7f24b80ff3a0 msgr2=0x7f24b81017c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:10:46.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.775+0000 7f24b7200640 1 -- 172.21.15.28:0/898542525 shutdown_connections 2024-05-12T22:10:46.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.776+0000 7f24b7200640 1 -- 172.21.15.28:0/898542525 wait complete. 2024-05-12T22:10:46.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.777+0000 7f24b7200640 1 Processor -- start 2024-05-12T22:10:46.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.777+0000 7f24b7200640 1 -- start start 2024-05-12T22:10:46.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.777+0000 7f24b7200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f24b81038d0 0x7f24b819a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:46.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.778+0000 7f24b7200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24b8104290 0x7f24b819a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:46.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.778+0000 7f24b7200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f24b81098e0 0x7f24b819eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:46.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.778+0000 7f24b7200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f24b8113cb0 con 0x7f24b8104290 2024-05-12T22:10:46.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.778+0000 7f24b7200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f24b8113b30 con 0x7f24b81038d0 2024-05-12T22:10:46.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.778+0000 7f24b7200640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f24b8113e30 con 0x7f24b81098e0 2024-05-12T22:10:46.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.778+0000 7f24b5400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24b8104290 0x7f24b819a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:46.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.778+0000 7f24b5400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24b8104290 0x7f24b819a770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49122/0 (socket says 172.21.15.28:49122) 2024-05-12T22:10:46.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.778+0000 7f24b5400640 1 -- 172.21.15.28:0/3382140218 learned_addr learned my addr 172.21.15.28:0/3382140218 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:10:46.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.778+0000 7f24b5e00640 1 --2- 172.21.15.28:0/3382140218 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f24b81038d0 0x7f24b819a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:46.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.778+0000 7f24b6800640 1 --2- 172.21.15.28:0/3382140218 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f24b81098e0 0x7f24b819eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:46.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.778+0000 7f24b5400640 1 -- 172.21.15.28:0/3382140218 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f24b81098e0 msgr2=0x7f24b819eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:10:46.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.778+0000 7f24b5400640 1 --2- 172.21.15.28:0/3382140218 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f24b81098e0 0x7f24b819eaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:46.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.778+0000 7f24b5400640 1 -- 172.21.15.28:0/3382140218 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f24b81038d0 msgr2=0x7f24b819a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:10:46.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.778+0000 7f24b5400640 1 --2- 172.21.15.28:0/3382140218 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f24b81038d0 0x7f24b819a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:46.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.778+0000 7f24b5400640 1 -- 172.21.15.28:0/3382140218 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24b819f1d0 con 0x7f24b8104290 2024-05-12T22:10:46.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.779+0000 7f24b5400640 1 --2- 172.21.15.28:0/3382140218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24b8104290 0x7f24b819a770 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f24a400e9c0 tx=0x7f24a400edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:10:46.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.779+0000 7f24aea00640 1 -- 172.21.15.28:0/3382140218 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24a400cd80 con 0x7f24b8104290 2024-05-12T22:10:46.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.779+0000 7f24aea00640 1 -- 172.21.15.28:0/3382140218 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f24a40035f0 con 0x7f24b8104290 2024-05-12T22:10:46.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.779+0000 7f24aea00640 1 -- 172.21.15.28:0/3382140218 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24a4010670 con 0x7f24b8104290 2024-05-12T22:10:46.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.779+0000 7f24b7200640 1 -- 172.21.15.28:0/3382140218 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24b819f4c0 con 0x7f24b8104290 2024-05-12T22:10:46.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.779+0000 7f24b7200640 1 -- 172.21.15.28:0/3382140218 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24b81a6d50 con 0x7f24b8104290 2024-05-12T22:10:46.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.783+0000 7f24bfe2d740 1 Processor -- start 2024-05-12T22:10:46.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.783+0000 7f24aea00640 1 -- 172.21.15.28:0/3382140218 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7f24a40037a0 con 0x7f24b8104290 2024-05-12T22:10:46.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.783+0000 7f24bfe2d740 1 -- start start 2024-05-12T22:10:46.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.783+0000 7f24bfe2d740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5627d77a2b60 0x5627d77a2f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:46.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.783+0000 7f24bfe2d740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5627d77a3480 0x5627d77ad610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:46.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.783+0000 7f24bfe2d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5627d77adb50 0x5627d77aff40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:46.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.784+0000 7f24bfe2d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5627d7707a30 con 0x5627d77adb50 2024-05-12T22:10:46.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.784+0000 7f24bfe2d740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x5627d76af700 con 0x5627d77a2b60 2024-05-12T22:10:46.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.784+0000 7f24bfe2d740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5627d77039f0 con 0x5627d77a3480 2024-05-12T22:10:46.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.784+0000 7f24b6800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5627d77a2b60 0x5627d77a2f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:46.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.784+0000 7f24b6800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5627d77a2b60 0x5627d77a2f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:41518/0 (socket says 172.21.15.28:41518) 2024-05-12T22:10:46.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.784+0000 7f24b6800640 1 -- 172.21.15.28:0/3364208618 learned_addr learned my addr 172.21.15.28:0/3364208618 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:10:46.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.784+0000 7f24b5400640 1 --2- 172.21.15.28:0/3364208618 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5627d77adb50 0x5627d77aff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:46.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.784+0000 7f24b5e00640 1 --2- 172.21.15.28:0/3364208618 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5627d77a3480 0x5627d77ad610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:46.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.784+0000 7f24b6800640 1 -- 172.21.15.28:0/3364208618 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5627d77a3480 msgr2=0x5627d77ad610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:10:46.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.784+0000 7f24b6800640 1 --2- 172.21.15.28:0/3364208618 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5627d77a3480 0x5627d77ad610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:46.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.784+0000 7f24b6800640 1 -- 172.21.15.28:0/3364208618 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5627d77adb50 msgr2=0x5627d77aff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:10:46.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.784+0000 7f24b6800640 1 --2- 172.21.15.28:0/3364208618 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5627d77adb50 0x5627d77aff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:46.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.784+0000 7f24b6800640 1 -- 172.21.15.28:0/3364208618 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5627d741b3f0 con 0x5627d77a2b60 2024-05-12T22:10:46.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.785+0000 7f24aea00640 1 --2- 172.21.15.28:0/3382140218 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f249c0a2d80 0x7f249c0a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:46.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.785+0000 7f24aea00640 1 -- 172.21.15.28:0/3382140218 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f24a40cf620 con 0x7f24b8104290 2024-05-12T22:10:46.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.785+0000 7f24b6800640 1 --2- 172.21.15.28:0/3364208618 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5627d77a2b60 0x5627d77a2f40 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f24b0009f60 tx=0x7f24b0009f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8189ce30765a7a4a server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:10:46.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.785+0000 7f24b5e00640 1 --2- 172.21.15.28:0/3382140218 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f249c0a2d80 0x7f249c0a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:46.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.785+0000 7f24a3400640 1 -- 172.21.15.28:0/3364208618 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24b001e070 con 0x5627d77a2b60 2024-05-12T22:10:46.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.785+0000 7f24a3400640 1 -- 172.21.15.28:0/3364208618 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f24b0003c30 con 0x5627d77a2b60 2024-05-12T22:10:46.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.785+0000 7f24a3400640 1 -- 172.21.15.28:0/3364208618 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24b000fa20 con 0x5627d77a2b60 2024-05-12T22:10:46.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.786+0000 7f24bfe2d740 1 -- 172.21.15.28:0/3364208618 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5627d77a2b60 msgr2=0x5627d77a2f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:10:46.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.786+0000 7f24bfe2d740 1 --2- 172.21.15.28:0/3364208618 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5627d77a2b60 0x5627d77a2f40 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f24b0009f60 tx=0x7f24b0009f90 comp rx=0 tx=0).stop 2024-05-12T22:10:46.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.786+0000 7f24b5e00640 1 --2- 172.21.15.28:0/3382140218 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f249c0a2d80 0x7f249c0a5240 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f24a802a930 tx=0x7f24a80056d0 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:10:46.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.786+0000 7f24bfe2d740 1 -- 172.21.15.28:0/3364208618 shutdown_connections 2024-05-12T22:10:46.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.786+0000 7f24bfe2d740 1 --2- 172.21.15.28:0/3364208618 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5627d77adb50 0x5627d77aff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:46.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.786+0000 7f24bfe2d740 1 --2- 172.21.15.28:0/3364208618 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5627d77a3480 0x5627d77ad610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:46.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.786+0000 7f24bfe2d740 1 --2- 172.21.15.28:0/3364208618 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5627d77a2b60 0x5627d77a2f40 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:46.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.786+0000 7f24bfe2d740 1 -- 172.21.15.28:0/3364208618 >> 172.21.15.28:0/3364208618 conn(0x5627d779ee80 msgr2=0x5627d779f260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:10:46.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.786+0000 7f24bfe2d740 1 -- 172.21.15.28:0/3364208618 shutdown_connections 2024-05-12T22:10:46.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.786+0000 7f24bfe2d740 1 -- 172.21.15.28:0/3364208618 wait complete. 2024-05-12T22:10:46.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.787+0000 7f24bfe2d740 1 -- start start 2024-05-12T22:10:46.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.787+0000 7f24bfe2d740 1 Processor -- start 2024-05-12T22:10:46.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.787+0000 7f24bfe2d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5627d77a2b60 0x5627d77aab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:46.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.788+0000 7f24bfe2d740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5627d77a3480 0x5627d77ab0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:46.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.788+0000 7f24bfe2d740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5627d77adb50 0x5627d77ab5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:46.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.788+0000 7f24bfe2d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5627d77b0940 con 0x5627d77a2b60 2024-05-12T22:10:46.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.788+0000 7f24bfe2d740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x5627d76c2a70 con 0x5627d77a3480 2024-05-12T22:10:46.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.788+0000 7f24bfe2d740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5627d77b0ac0 con 0x5627d77adb50 2024-05-12T22:10:46.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.788+0000 7f24bfe2d740 10 client.0 _ll_register_callbacks cb 0x5627d76a5ba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:10:46.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.788+0000 7f24b5e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5627d77a2b60 0x5627d77aab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:46.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.788+0000 7f24b5e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5627d77a2b60 0x5627d77aab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49134/0 (socket says 172.21.15.28:49134) 2024-05-12T22:10:46.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.788+0000 7f24b5e00640 1 -- 172.21.15.28:0/3059170423 learned_addr learned my addr 172.21.15.28:0/3059170423 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:10:46.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.788+0000 7f24b5400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5627d77a3480 0x5627d77ab0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:46.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.788+0000 7f24b6800640 1 --2- 172.21.15.28:0/3059170423 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5627d77adb50 0x5627d77ab5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:46.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.788+0000 7f24b5e00640 1 -- 172.21.15.28:0/3059170423 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5627d77adb50 msgr2=0x5627d77ab5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:10:46.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.788+0000 7f24b5e00640 1 --2- 172.21.15.28:0/3059170423 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5627d77adb50 0x5627d77ab5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:46.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.788+0000 7f24b5e00640 1 -- 172.21.15.28:0/3059170423 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5627d77a3480 msgr2=0x5627d77ab0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:10:46.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.788+0000 7f24b5e00640 1 --2- 172.21.15.28:0/3059170423 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5627d77a3480 0x5627d77ab0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:46.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.788+0000 7f24b5e00640 1 -- 172.21.15.28:0/3059170423 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5627d74e57d0 con 0x5627d77a2b60 2024-05-12T22:10:46.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.789+0000 7f24b5e00640 1 --2- 172.21.15.28:0/3059170423 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5627d77a2b60 0x5627d77aab70 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f24a8009230 tx=0x7f24a802a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7f7bd075359f281 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:10:46.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.789+0000 7f2497e00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-12T22:10:46.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.789+0000 7f2497e00640 1 -- 172.21.15.28:0/3059170423 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24a8037040 con 0x5627d77a2b60 2024-05-12T22:10:46.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.789+0000 7f2497e00640 1 -- 172.21.15.28:0/3059170423 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f24a8003a00 con 0x5627d77a2b60 2024-05-12T22:10:46.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.789+0000 7f24bfe2d740 1 -- 172.21.15.28:0/3059170423 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f24b0003c30 con 0x5627d77a2b60 2024-05-12T22:10:46.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.789+0000 7f2497e00640 1 -- 172.21.15.28:0/3059170423 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f24a8056050 con 0x5627d77a2b60 2024-05-12T22:10:46.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.789+0000 7f24bfe2d740 10 client.15633 fetch_fsmap learned FSMap version 25 2024-05-12T22:10:46.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.789+0000 7f24bfe2d740 1 -- 172.21.15.28:0/3059170423 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5627d77bcc70 con 0x5627d77a2b60 2024-05-12T22:10:46.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.790+0000 7f2497e00640 1 -- 172.21.15.28:0/3059170423 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f24b0003c30 con 0x5627d77a2b60 2024-05-12T22:10:46.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.790+0000 7f2497e00640 1 -- 172.21.15.28:0/3059170423 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7f24a8048440 con 0x5627d77a2b60 2024-05-12T22:10:46.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.790+0000 7f24bfe2d740 10 client.15633 fetch_fsmap finished waiting for FSMap version 25 2024-05-12T22:10:46.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.790+0000 7f24bfe2d740 10 client.15633 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:10:46.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.790+0000 7f24bfe2d740 20 client.15633 populate_metadata read hostname 'smithi028' 2024-05-12T22:10:46.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.790+0000 7f24bfe2d740 1 --2- 172.21.15.28:0/3059170423 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5627d77be180 0x5627d77c05f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:46.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.790+0000 7f24bfe2d740 4 client.15633 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-12T22:10:46.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.790+0000 7f24bfe2d740 1 -- 172.21.15.28:0/3059170423 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5627d7715160 con 0x5627d77be180 2024-05-12T22:10:46.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.790+0000 7f24b5400640 1 --2- 172.21.15.28:0/3059170423 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5627d77be180 0x5627d77c05f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:46.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.791+0000 7f24b5400640 1 --2- 172.21.15.28:0/3059170423 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5627d77be180 0x5627d77c05f0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=411e2e1bc452afd7 server_cookie=93e3d4b555b6c035 in_seq=0 out_seq=0 2024-05-12T22:10:46.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.791+0000 7f2497e00640 10 client.15633 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:10:46.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.792+0000 7f2497e00640 1 -- 172.21.15.28:0/3059170423 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5627d7715160 con 0x5627d77be180 2024-05-12T22:10:46.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.792+0000 7f2497e00640 10 client.15633 handle_command_reply: tid=0 2024-05-12T22:10:46.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.792+0000 7f24bfe2d740 1 client.15633 shutdown 2024-05-12T22:10:46.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.792+0000 7f24bfe2d740 1 -- 172.21.15.28:0/3059170423 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5627d77a2b60 msgr2=0x5627d77aab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:10:46.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.792+0000 7f24bfe2d740 1 --2- 172.21.15.28:0/3059170423 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5627d77a2b60 0x5627d77aab70 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f24a8009230 tx=0x7f24a802a340 comp rx=0 tx=0).stop 2024-05-12T22:10:46.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.792+0000 7f24bfe2d740 1 -- 172.21.15.28:0/3059170423 shutdown_connections 2024-05-12T22:10:46.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.792+0000 7f24bfe2d740 1 --2- 172.21.15.28:0/3059170423 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5627d77be180 0x5627d77c05f0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:46.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.792+0000 7f24bfe2d740 1 --2- 172.21.15.28:0/3059170423 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5627d77adb50 0x5627d77ab5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:46.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.792+0000 7f24bfe2d740 1 --2- 172.21.15.28:0/3059170423 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5627d77a3480 0x5627d77ab0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:46.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.792+0000 7f24bfe2d740 1 --2- 172.21.15.28:0/3059170423 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5627d77a2b60 0x5627d77aab70 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:46.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.792+0000 7f2497e00640 0 client.15633 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:10:46.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.792+0000 7f24bfe2d740 1 -- 172.21.15.28:0/3059170423 >> 172.21.15.28:0/3059170423 conn(0x5627d779ee80 msgr2=0x5627d77a4c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:10:46.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.792+0000 7f24bfe2d740 1 -- 172.21.15.28:0/3059170423 shutdown_connections 2024-05-12T22:10:46.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.792+0000 7f24bfe2d740 1 -- 172.21.15.28:0/3059170423 wait complete. 2024-05-12T22:10:46.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.792+0000 7f24bfe2d740 20 client.15633 trim_cache size 0 max 16384 2024-05-12T22:10:46.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.808+0000 7f24bfe2d740 1 Processor -- start 2024-05-12T22:10:46.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.808+0000 7f24bfe2d740 1 -- start start 2024-05-12T22:10:46.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.808+0000 7f24bfe2d740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5627d77a2b60 0x5627d77b9800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:46.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.808+0000 7f24bfe2d740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5627d77a3480 0x5627d77a52c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:46.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.808+0000 7f24bfe2d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5627d77adb50 0x5627d77c5ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:46.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.808+0000 7f24bfe2d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f24b81b1d90 con 0x5627d77adb50 2024-05-12T22:10:46.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.808+0000 7f24bfe2d740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f24b81af660 con 0x5627d77a2b60 2024-05-12T22:10:46.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.808+0000 7f24bfe2d740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f24b81ad4b0 con 0x5627d77a3480 2024-05-12T22:10:46.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.808+0000 7f24b6800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5627d77adb50 0x5627d77c5ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:46.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.808+0000 7f24b5e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5627d77a2b60 0x5627d77b9800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:46.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.808+0000 7f24b5400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5627d77a3480 0x5627d77a52c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:46.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.808+0000 7f24b6800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5627d77adb50 0x5627d77c5ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49148/0 (socket says 172.21.15.28:49148) 2024-05-12T22:10:46.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.808+0000 7f24b6800640 1 -- 172.21.15.28:0/1260141599 learned_addr learned my addr 172.21.15.28:0/1260141599 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:10:46.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.809+0000 7f24b5e00640 1 -- 172.21.15.28:0/1260141599 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5627d77a3480 msgr2=0x5627d77a52c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:10:46.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.809+0000 7f24b5e00640 1 --2- 172.21.15.28:0/1260141599 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5627d77a3480 0x5627d77a52c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:46.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.809+0000 7f24b5e00640 1 -- 172.21.15.28:0/1260141599 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5627d77adb50 msgr2=0x5627d77c5ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:10:46.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.809+0000 7f24b5e00640 1 --2- 172.21.15.28:0/1260141599 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5627d77adb50 0x5627d77c5ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:46.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.809+0000 7f24b5e00640 1 -- 172.21.15.28:0/1260141599 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5627d77aff40 con 0x5627d77a2b60 2024-05-12T22:10:46.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.809+0000 7f24b5e00640 1 --2- 172.21.15.28:0/1260141599 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5627d77a2b60 0x5627d77b9800 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f24a802a860 tx=0x7f24a8005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=77903ea45fc20bf0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:10:46.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.809+0000 7f24a2a00640 1 -- 172.21.15.28:0/1260141599 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24a8037050 con 0x5627d77a2b60 2024-05-12T22:10:46.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.810+0000 7f24a2a00640 1 -- 172.21.15.28:0/1260141599 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f24a805f070 con 0x5627d77a2b60 2024-05-12T22:10:46.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.810+0000 7f24a2a00640 1 -- 172.21.15.28:0/1260141599 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24a8003ea0 con 0x5627d77a2b60 2024-05-12T22:10:46.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.810+0000 7f24bfe2d740 1 -- 172.21.15.28:0/1260141599 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5627d77a2b60 msgr2=0x5627d77b9800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:10:46.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.810+0000 7f24bfe2d740 1 --2- 172.21.15.28:0/1260141599 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5627d77a2b60 0x5627d77b9800 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f24a802a860 tx=0x7f24a8005fd0 comp rx=0 tx=0).stop 2024-05-12T22:10:46.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.810+0000 7f24bfe2d740 1 -- 172.21.15.28:0/1260141599 shutdown_connections 2024-05-12T22:10:46.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.810+0000 7f24bfe2d740 1 --2- 172.21.15.28:0/1260141599 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5627d77adb50 0x5627d77c5ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:46.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.810+0000 7f24bfe2d740 1 --2- 172.21.15.28:0/1260141599 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5627d77a3480 0x5627d77a52c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:46.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.810+0000 7f24bfe2d740 1 --2- 172.21.15.28:0/1260141599 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5627d77a2b60 0x5627d77b9800 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:46.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.810+0000 7f24bfe2d740 1 -- 172.21.15.28:0/1260141599 >> 172.21.15.28:0/1260141599 conn(0x5627d779ee80 msgr2=0x5627d779e000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:10:46.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.810+0000 7f24bfe2d740 1 -- 172.21.15.28:0/1260141599 shutdown_connections 2024-05-12T22:10:46.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.810+0000 7f24bfe2d740 1 -- 172.21.15.28:0/1260141599 wait complete. 2024-05-12T22:10:46.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.811+0000 7f24bfe2d740 1 -- start start 2024-05-12T22:10:46.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.811+0000 7f24bfe2d740 1 Processor -- start 2024-05-12T22:10:46.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.811+0000 7f24bfe2d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5627d77a2b60 0x5627d77b1720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:46.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.811+0000 7f24bfe2d740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5627d77a3480 0x5627d77b1c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:46.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.811+0000 7f24bfe2d740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5627d77adb50 0x5627d77b21a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:46.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.811+0000 7f24bfe2d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f24b81adb20 con 0x5627d77a2b60 2024-05-12T22:10:46.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.811+0000 7f24bfe2d740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f24b81aef90 con 0x5627d77adb50 2024-05-12T22:10:46.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.811+0000 7f24bfe2d740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f24b81ae1f0 con 0x5627d77a3480 2024-05-12T22:10:46.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.811+0000 7f24bfe2d740 10 client.0 _ll_register_callbacks cb 0x7f24a8030960 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:10:46.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.812+0000 7f24b5e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5627d77a2b60 0x5627d77b1720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:46.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.812+0000 7f24b5400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5627d77a3480 0x5627d77b1c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:46.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.812+0000 7f24b6800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5627d77adb50 0x5627d77b21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:46.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.812+0000 7f24b5400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5627d77a3480 0x5627d77b1c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:48434/0 (socket says 172.21.15.28:48434) 2024-05-12T22:10:46.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.812+0000 7f24b5400640 1 -- 172.21.15.28:0/4270576658 learned_addr learned my addr 172.21.15.28:0/4270576658 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:10:46.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.812+0000 7f24b5e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5627d77a2b60 0x5627d77b1720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49154/0 (socket says 172.21.15.28:49154) 2024-05-12T22:10:46.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.812+0000 7f24b5e00640 1 -- 172.21.15.28:0/4270576658 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5627d77a3480 msgr2=0x5627d77b1c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:10:46.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.812+0000 7f24b5e00640 1 --2- 172.21.15.28:0/4270576658 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5627d77a3480 0x5627d77b1c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:46.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.812+0000 7f24b5e00640 1 -- 172.21.15.28:0/4270576658 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5627d77adb50 msgr2=0x5627d77b21a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:10:46.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.812+0000 7f24b5e00640 1 --2- 172.21.15.28:0/4270576658 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5627d77adb50 0x5627d77b21a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:46.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.812+0000 7f24b5e00640 1 -- 172.21.15.28:0/4270576658 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5627d7823730 con 0x5627d77a2b60 2024-05-12T22:10:46.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.812+0000 7f24b6800640 1 --2- 172.21.15.28:0/4270576658 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5627d77adb50 0x5627d77b21a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:10:46.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.813+0000 7f24b5e00640 1 --2- 172.21.15.28:0/4270576658 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5627d77a2b60 0x5627d77b1720 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f24a8031000 tx=0x7f24a80486f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4231d454a2470ae server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:10:46.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.813+0000 7f2497e00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-12T22:10:46.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.813+0000 7f2497e00640 1 -- 172.21.15.28:0/4270576658 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24a8037040 con 0x5627d77a2b60 2024-05-12T22:10:46.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.813+0000 7f2497e00640 1 -- 172.21.15.28:0/4270576658 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f24a8003800 con 0x5627d77a2b60 2024-05-12T22:10:46.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.813+0000 7f24bfe2d740 1 -- 172.21.15.28:0/4270576658 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f24a805f070 con 0x5627d77a2b60 2024-05-12T22:10:46.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.813+0000 7f2497e00640 1 -- 172.21.15.28:0/4270576658 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f24a8003e80 con 0x5627d77a2b60 2024-05-12T22:10:46.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.813+0000 7f24bfe2d740 10 client.15639 fetch_fsmap learned FSMap version 25 2024-05-12T22:10:46.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.813+0000 7f24bfe2d740 1 -- 172.21.15.28:0/4270576658 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5627d7824a10 con 0x5627d77a2b60 2024-05-12T22:10:46.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.813+0000 7f2497e00640 1 -- 172.21.15.28:0/4270576658 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f24a805f070 con 0x5627d77a2b60 2024-05-12T22:10:46.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.814+0000 7f2497e00640 1 -- 172.21.15.28:0/4270576658 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7f24a803db90 con 0x5627d77a2b60 2024-05-12T22:10:46.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.814+0000 7f24bfe2d740 10 client.15639 fetch_fsmap finished waiting for FSMap version 25 2024-05-12T22:10:46.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.814+0000 7f24bfe2d740 10 client.15639 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:10:46.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.814+0000 7f24bfe2d740 20 client.15639 populate_metadata read hostname 'smithi028' 2024-05-12T22:10:46.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.814+0000 7f24bfe2d740 1 --2- 172.21.15.28:0/4270576658 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5627d78250a0 0x5627d78274c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:46.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.814+0000 7f24bfe2d740 4 client.15639 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-12T22:10:46.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.814+0000 7f24bfe2d740 1 -- 172.21.15.28:0/4270576658 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5627d770f4b0 con 0x5627d78250a0 2024-05-12T22:10:46.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.814+0000 7f24b5400640 1 --2- 172.21.15.28:0/4270576658 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5627d78250a0 0x5627d78274c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:46.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.815+0000 7f24b5400640 1 --2- 172.21.15.28:0/4270576658 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5627d78250a0 0x5627d78274c0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3099fa89d7b7bf0b server_cookie=4f09b3c8cfcf2ac6 in_seq=0 out_seq=0 2024-05-12T22:10:46.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.815+0000 7f2497e00640 10 client.15639 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:10:46.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.815+0000 7f2497e00640 1 -- 172.21.15.28:0/4270576658 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5627d770f4b0 con 0x5627d78250a0 2024-05-12T22:10:46.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.815+0000 7f2497e00640 10 client.15639 handle_command_reply: tid=0 2024-05-12T22:10:46.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.815+0000 7f24bfe2d740 1 client.15639 shutdown 2024-05-12T22:10:46.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.815+0000 7f24bfe2d740 1 -- 172.21.15.28:0/4270576658 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5627d77a2b60 msgr2=0x5627d77b1720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:10:46.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.815+0000 7f24bfe2d740 1 --2- 172.21.15.28:0/4270576658 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5627d77a2b60 0x5627d77b1720 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f24a8031000 tx=0x7f24a80486f0 comp rx=0 tx=0).stop 2024-05-12T22:10:46.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.815+0000 7f24bfe2d740 1 -- 172.21.15.28:0/4270576658 shutdown_connections 2024-05-12T22:10:46.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.815+0000 7f24bfe2d740 1 --2- 172.21.15.28:0/4270576658 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5627d78250a0 0x5627d78274c0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:46.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.815+0000 7f24bfe2d740 1 --2- 172.21.15.28:0/4270576658 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5627d77adb50 0x5627d77b21a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:46.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.815+0000 7f24bfe2d740 1 --2- 172.21.15.28:0/4270576658 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5627d77a3480 0x5627d77b1c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:46.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.815+0000 7f24bfe2d740 1 --2- 172.21.15.28:0/4270576658 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5627d77a2b60 0x5627d77b1720 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:46.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.815+0000 7f24bfe2d740 1 -- 172.21.15.28:0/4270576658 >> 172.21.15.28:0/4270576658 conn(0x5627d779ee80 msgr2=0x5627d77aa960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:10:46.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.815+0000 7f2497e00640 0 client.15639 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:10:46.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.815+0000 7f24bfe2d740 1 -- 172.21.15.28:0/4270576658 shutdown_connections 2024-05-12T22:10:46.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.815+0000 7f24bfe2d740 1 -- 172.21.15.28:0/4270576658 wait complete. 2024-05-12T22:10:46.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.815+0000 7f24bfe2d740 20 client.15639 trim_cache size 0 max 16384 2024-05-12T22:10:46.821 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-05-12T22:10:46.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.816+0000 7f24b7200640 1 -- 172.21.15.28:0/3382140218 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f249c0a2d80 msgr2=0x7f249c0a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:10:46.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.816+0000 7f24b7200640 1 --2- 172.21.15.28:0/3382140218 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f249c0a2d80 0x7f249c0a5240 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f24a802a930 tx=0x7f24a80056d0 comp rx=0 tx=0).stop 2024-05-12T22:10:46.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.816+0000 7f24b7200640 1 -- 172.21.15.28:0/3382140218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24b8104290 msgr2=0x7f24b819a770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:10:46.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.816+0000 7f24b7200640 1 --2- 172.21.15.28:0/3382140218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24b8104290 0x7f24b819a770 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f24a400e9c0 tx=0x7f24a400edc0 comp rx=0 tx=0).stop 2024-05-12T22:10:46.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.816+0000 7f24b7200640 1 -- 172.21.15.28:0/3382140218 shutdown_connections 2024-05-12T22:10:46.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.816+0000 7f24b7200640 1 --2- 172.21.15.28:0/3382140218 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f249c0a2d80 0x7f249c0a5240 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:46.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.816+0000 7f24b7200640 1 --2- 172.21.15.28:0/3382140218 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f24b81098e0 0x7f24b819eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:46.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.816+0000 7f24b7200640 1 --2- 172.21.15.28:0/3382140218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24b8104290 0x7f24b819a770 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:46.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.816+0000 7f24b7200640 1 --2- 172.21.15.28:0/3382140218 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f24b81038d0 0x7f24b819a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:46.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.816+0000 7f24b7200640 1 -- 172.21.15.28:0/3382140218 >> 172.21.15.28:0/3382140218 conn(0x7f24b80ff3a0 msgr2=0x7f24b80ffc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:10:46.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.816+0000 7f24b7200640 1 -- 172.21.15.28:0/3382140218 shutdown_connections 2024-05-12T22:10:46.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:46.816+0000 7f24b7200640 1 -- 172.21.15.28:0/3382140218 wait complete. 2024-05-12T22:10:47.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:10:46 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/2970932550' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:10:47.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:10:46 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/2970932550' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:10:47.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:10:46 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/2970932550' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:10:47.836 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-12T22:10:47.836 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-12T22:10:47.836 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-05-12T22:10:48.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.055+0000 7f347be00640 1 -- 172.21.15.28:0/3467741149 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f347c107dd0 msgr2=0x7f347c108230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:10:48.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.055+0000 7f347be00640 1 --2- 172.21.15.28:0/3467741149 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f347c107dd0 0x7f347c108230 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f3464009900 tx=0x7f346402a480 comp rx=0 tx=0).stop 2024-05-12T22:10:48.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.056+0000 7f347be00640 1 -- 172.21.15.28:0/3467741149 shutdown_connections 2024-05-12T22:10:48.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.056+0000 7f347be00640 1 --2- 172.21.15.28:0/3467741149 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f347c108770 0x7f347c114b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:48.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.056+0000 7f347be00640 1 --2- 172.21.15.28:0/3467741149 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f347c107dd0 0x7f347c108230 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:48.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.056+0000 7f347be00640 1 --2- 172.21.15.28:0/3467741149 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f347c10ddf0 0x7f347c10e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:48.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.056+0000 7f347be00640 1 -- 172.21.15.28:0/3467741149 >> 172.21.15.28:0/3467741149 conn(0x7f347c077650 msgr2=0x7f347c106650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:10:48.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.056+0000 7f347be00640 1 -- 172.21.15.28:0/3467741149 shutdown_connections 2024-05-12T22:10:48.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.056+0000 7f347be00640 1 -- 172.21.15.28:0/3467741149 wait complete. 2024-05-12T22:10:48.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.057+0000 7f347be00640 1 Processor -- start 2024-05-12T22:10:48.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.057+0000 7f347be00640 1 -- start start 2024-05-12T22:10:48.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.057+0000 7f347be00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f347c107dd0 0x7f347c19e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:48.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.057+0000 7f347be00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f347c108770 0x7f347c19ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:48.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.057+0000 7f347be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f347c10ddf0 0x7f347c1a2e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:48.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.057+0000 7f347be00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f347c118120 con 0x7f347c10ddf0 2024-05-12T22:10:48.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.057+0000 7f347be00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f347c117fa0 con 0x7f347c108770 2024-05-12T22:10:48.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.057+0000 7f347be00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f347c1182a0 con 0x7f347c107dd0 2024-05-12T22:10:48.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.057+0000 7f3472000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f347c108770 0x7f347c19ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:48.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.058+0000 7f3472000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f347c108770 0x7f347c19ea80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:41556/0 (socket says 172.21.15.28:41556) 2024-05-12T22:10:48.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.058+0000 7f3472000640 1 -- 172.21.15.28:0/1814709361 learned_addr learned my addr 172.21.15.28:0/1814709361 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:10:48.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.058+0000 7f347b400640 1 --2- 172.21.15.28:0/1814709361 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f347c10ddf0 0x7f347c1a2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:48.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.058+0000 7f347aa00640 1 --2- 172.21.15.28:0/1814709361 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f347c107dd0 0x7f347c19e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:48.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.058+0000 7f3472000640 1 -- 172.21.15.28:0/1814709361 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f347c107dd0 msgr2=0x7f347c19e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:10:48.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.058+0000 7f3472000640 1 --2- 172.21.15.28:0/1814709361 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f347c107dd0 0x7f347c19e540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:48.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.058+0000 7f3472000640 1 -- 172.21.15.28:0/1814709361 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f347c10ddf0 msgr2=0x7f347c1a2e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:10:48.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.058+0000 7f3472000640 1 --2- 172.21.15.28:0/1814709361 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f347c10ddf0 0x7f347c1a2e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:48.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.058+0000 7f3472000640 1 -- 172.21.15.28:0/1814709361 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f347c1a3530 con 0x7f347c108770 2024-05-12T22:10:48.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.058+0000 7f3472000640 1 --2- 172.21.15.28:0/1814709361 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f347c108770 0x7f347c19ea80 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f3464004770 tx=0x7f3464003c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:10:48.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.058+0000 7f3473e00640 1 -- 172.21.15.28:0/1814709361 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f346403f070 con 0x7f347c108770 2024-05-12T22:10:48.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.059+0000 7f3473e00640 1 -- 172.21.15.28:0/1814709361 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3464031ca0 con 0x7f347c108770 2024-05-12T22:10:48.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.059+0000 7f3473e00640 1 -- 172.21.15.28:0/1814709361 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3464035030 con 0x7f347c108770 2024-05-12T22:10:48.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.059+0000 7f347be00640 1 -- 172.21.15.28:0/1814709361 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f347c1a37c0 con 0x7f347c108770 2024-05-12T22:10:48.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.059+0000 7f347be00640 1 -- 172.21.15.28:0/1814709361 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f347c1ab0b0 con 0x7f347c108770 2024-05-12T22:10:48.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.062+0000 7f3473e00640 1 -- 172.21.15.28:0/1814709361 <== mon.2 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7f3464042070 con 0x7f347c108770 2024-05-12T22:10:48.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.062+0000 7f3484401740 1 Processor -- start 2024-05-12T22:10:48.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.062+0000 7f3484401740 1 -- start start 2024-05-12T22:10:48.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.063+0000 7f3484401740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c84de5b500 0x55c84de5b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:48.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.063+0000 7f3484401740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c84de5beb0 0x55c84de65f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:48.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.063+0000 7f3484401740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c84de66520 0x55c84de68970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:48.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.063+0000 7f3484401740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c84ddbf7c0 con 0x55c84de5beb0 2024-05-12T22:10:48.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.063+0000 7f3484401740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x55c84dd67790 con 0x55c84de5b500 2024-05-12T22:10:48.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.063+0000 7f3484401740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55c84ddbb780 con 0x55c84de66520 2024-05-12T22:10:48.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.063+0000 7f347b400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c84de5b500 0x55c84de5b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:48.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.063+0000 7f347aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c84de5beb0 0x55c84de65f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:48.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.063+0000 7f347b400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c84de5b500 0x55c84de5b8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:41562/0 (socket says 172.21.15.28:41562) 2024-05-12T22:10:48.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.063+0000 7f347b400640 1 -- 172.21.15.28:0/142832044 learned_addr learned my addr 172.21.15.28:0/142832044 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:10:48.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.063+0000 7f3472000640 1 --2- 172.21.15.28:0/142832044 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c84de66520 0x55c84de68970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:48.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.063+0000 7f347b400640 1 -- 172.21.15.28:0/142832044 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c84de66520 msgr2=0x55c84de68970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:10:48.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.063+0000 7f347b400640 1 --2- 172.21.15.28:0/142832044 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c84de66520 0x55c84de68970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:48.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.063+0000 7f347b400640 1 -- 172.21.15.28:0/142832044 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c84de5beb0 msgr2=0x55c84de65f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:10:48.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.063+0000 7f347b400640 1 --2- 172.21.15.28:0/142832044 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c84de5beb0 0x55c84de65f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:48.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.063+0000 7f347b400640 1 -- 172.21.15.28:0/142832044 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c84dd876a0 con 0x55c84de5b500 2024-05-12T22:10:48.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.064+0000 7f3473e00640 1 --2- 172.21.15.28:0/1814709361 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f34600a2d80 0x7f34600a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:48.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.064+0000 7f347b400640 1 --2- 172.21.15.28:0/142832044 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c84de5b500 0x55c84de5b8e0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f3474009fc0 tx=0x7f34740118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b4d5305cdd625890 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:10:48.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.064+0000 7f3473e00640 1 -- 172.21.15.28:0/1814709361 <== mon.2 v2:172.21.15.86:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f34640ef990 con 0x7f347c108770 2024-05-12T22:10:48.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.064+0000 7f346b400640 1 -- 172.21.15.28:0/142832044 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3474004d20 con 0x55c84de5b500 2024-05-12T22:10:48.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.064+0000 7f346b400640 1 -- 172.21.15.28:0/142832044 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3474004050 con 0x55c84de5b500 2024-05-12T22:10:48.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.064+0000 7f346b400640 1 -- 172.21.15.28:0/142832044 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3474005260 con 0x55c84de5b500 2024-05-12T22:10:48.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.064+0000 7f347aa00640 1 --2- 172.21.15.28:0/1814709361 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f34600a2d80 0x7f34600a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:48.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.065+0000 7f3484401740 1 -- 172.21.15.28:0/142832044 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c84de5b500 msgr2=0x55c84de5b8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:10:48.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.065+0000 7f3484401740 1 --2- 172.21.15.28:0/142832044 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c84de5b500 0x55c84de5b8e0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f3474009fc0 tx=0x7f34740118d0 comp rx=0 tx=0).stop 2024-05-12T22:10:48.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.065+0000 7f347aa00640 1 --2- 172.21.15.28:0/1814709361 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f34600a2d80 0x7f34600a5240 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x55c84de58830 tx=0x7f346c005f90 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:10:48.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.065+0000 7f3484401740 1 -- 172.21.15.28:0/142832044 shutdown_connections 2024-05-12T22:10:48.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.065+0000 7f3484401740 1 --2- 172.21.15.28:0/142832044 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c84de66520 0x55c84de68970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:48.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.065+0000 7f3484401740 1 --2- 172.21.15.28:0/142832044 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c84de5beb0 0x55c84de65f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:48.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.065+0000 7f3484401740 1 --2- 172.21.15.28:0/142832044 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c84de5b500 0x55c84de5b8e0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:48.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.065+0000 7f3484401740 1 -- 172.21.15.28:0/142832044 >> 172.21.15.28:0/142832044 conn(0x55c84de575b0 msgr2=0x55c84de57990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:10:48.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.065+0000 7f3484401740 1 -- 172.21.15.28:0/142832044 shutdown_connections 2024-05-12T22:10:48.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.065+0000 7f3484401740 1 -- 172.21.15.28:0/142832044 wait complete. 2024-05-12T22:10:48.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.066+0000 7f3484401740 1 -- start start 2024-05-12T22:10:48.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.066+0000 7f3484401740 1 Processor -- start 2024-05-12T22:10:48.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.066+0000 7f3484401740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c84de5b500 0x55c84de633f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:48.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.066+0000 7f3484401740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c84de5beb0 0x55c84de63930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:48.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.066+0000 7f3484401740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c84de66520 0x55c84de63e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:48.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.066+0000 7f3484401740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c84de69530 con 0x55c84de5beb0 2024-05-12T22:10:48.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.066+0000 7f3484401740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x55c84dd7aa90 con 0x55c84de5b500 2024-05-12T22:10:48.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.066+0000 7f3484401740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55c84de696b0 con 0x55c84de66520 2024-05-12T22:10:48.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.066+0000 7f3484401740 10 client.0 _ll_register_callbacks cb 0x55c84ddb5af0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:10:48.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.067+0000 7f347aa00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c84de5b500 0x55c84de633f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:48.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.067+0000 7f3472000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c84de5beb0 0x55c84de63930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:48.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.067+0000 7f347aa00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c84de5b500 0x55c84de633f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:41574/0 (socket says 172.21.15.28:41574) 2024-05-12T22:10:48.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.067+0000 7f347aa00640 1 -- 172.21.15.28:0/1725558891 learned_addr learned my addr 172.21.15.28:0/1725558891 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:10:48.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.067+0000 7f347b400640 1 --2- 172.21.15.28:0/1725558891 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c84de66520 0x55c84de63e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:48.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.067+0000 7f347aa00640 1 -- 172.21.15.28:0/1725558891 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c84de66520 msgr2=0x55c84de63e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:10:48.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.067+0000 7f347aa00640 1 --2- 172.21.15.28:0/1725558891 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c84de66520 0x55c84de63e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:48.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.067+0000 7f347aa00640 1 -- 172.21.15.28:0/1725558891 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c84de5beb0 msgr2=0x55c84de63930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:10:48.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.067+0000 7f347aa00640 1 --2- 172.21.15.28:0/1725558891 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c84de5beb0 0x55c84de63930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:48.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.067+0000 7f347aa00640 1 -- 172.21.15.28:0/1725558891 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c84db9b7e0 con 0x55c84de5b500 2024-05-12T22:10:48.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.067+0000 7f347aa00640 1 --2- 172.21.15.28:0/1725558891 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c84de5b500 0x55c84de633f0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f346c004600 tx=0x7f346c004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ea98dd02894becff server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:10:48.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.067+0000 7f345fe00640 10 client.0 ms_handle_connect on v2:172.21.15.86:3300/0 2024-05-12T22:10:48.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.068+0000 7f345fe00640 1 -- 172.21.15.28:0/1725558891 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f346c005020 con 0x55c84de5b500 2024-05-12T22:10:48.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.068+0000 7f345fe00640 1 -- 172.21.15.28:0/1725558891 <== mon.2 v2:172.21.15.86:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f346c0051c0 con 0x55c84de5b500 2024-05-12T22:10:48.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.068+0000 7f3484401740 1 -- 172.21.15.28:0/1725558891 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3474004050 con 0x55c84de5b500 2024-05-12T22:10:48.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.068+0000 7f345fe00640 1 -- 172.21.15.28:0/1725558891 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f346c0053c0 con 0x55c84de5b500 2024-05-12T22:10:48.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.068+0000 7f3484401740 10 client.25769 fetch_fsmap learned FSMap version 25 2024-05-12T22:10:48.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.068+0000 7f3484401740 1 -- 172.21.15.28:0/1725558891 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55c84dbfbf70 con 0x55c84de5b500 2024-05-12T22:10:48.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.068+0000 7f345fe00640 1 -- 172.21.15.28:0/1725558891 <== mon.2 v2:172.21.15.86:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3474004050 con 0x55c84de5b500 2024-05-12T22:10:48.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.069+0000 7f345fe00640 1 -- 172.21.15.28:0/1725558891 <== mon.2 v2:172.21.15.86:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7f346c027070 con 0x55c84de5b500 2024-05-12T22:10:48.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.069+0000 7f3484401740 10 client.25769 fetch_fsmap finished waiting for FSMap version 25 2024-05-12T22:10:48.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.069+0000 7f3484401740 10 client.25769 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:10:48.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.069+0000 7f3484401740 20 client.25769 populate_metadata read hostname 'smithi028' 2024-05-12T22:10:48.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.069+0000 7f3484401740 1 --2- 172.21.15.28:0/1725558891 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55c84de76800 0x55c84de78ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:48.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.069+0000 7f3484401740 4 client.25769 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-12T22:10:48.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.069+0000 7f3484401740 1 -- 172.21.15.28:0/1725558891 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c84ddd33a0 con 0x55c84de76800 2024-05-12T22:10:48.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.069+0000 7f3472000640 1 --2- 172.21.15.28:0/1725558891 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55c84de76800 0x55c84de78ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:48.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.070+0000 7f3472000640 1 --2- 172.21.15.28:0/1725558891 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55c84de76800 0x55c84de78ca0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c9c4491d442a58d5 server_cookie=a5db7c80c22a75f6 in_seq=0 out_seq=0 2024-05-12T22:10:48.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.070+0000 7f345fe00640 10 client.25769 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:10:48.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.071+0000 7f345fe00640 1 -- 172.21.15.28:0/1725558891 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c84ddd33a0 con 0x55c84de76800 2024-05-12T22:10:48.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.071+0000 7f345fe00640 10 client.25769 handle_command_reply: tid=0 2024-05-12T22:10:48.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.071+0000 7f3484401740 1 client.25769 shutdown 2024-05-12T22:10:48.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.072+0000 7f3484401740 1 -- 172.21.15.28:0/1725558891 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c84de5b500 msgr2=0x55c84de633f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:10:48.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.072+0000 7f3484401740 1 --2- 172.21.15.28:0/1725558891 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c84de5b500 0x55c84de633f0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f346c004600 tx=0x7f346c004730 comp rx=0 tx=0).stop 2024-05-12T22:10:48.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.072+0000 7f3484401740 1 -- 172.21.15.28:0/1725558891 shutdown_connections 2024-05-12T22:10:48.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.072+0000 7f3484401740 1 --2- 172.21.15.28:0/1725558891 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55c84de76800 0x55c84de78ca0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:48.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.072+0000 7f3484401740 1 --2- 172.21.15.28:0/1725558891 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c84de66520 0x55c84de63e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:48.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.072+0000 7f3484401740 1 --2- 172.21.15.28:0/1725558891 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c84de5beb0 0x55c84de63930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:48.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.072+0000 7f3484401740 1 --2- 172.21.15.28:0/1725558891 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c84de5b500 0x55c84de633f0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:48.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.072+0000 7f3484401740 1 -- 172.21.15.28:0/1725558891 >> 172.21.15.28:0/1725558891 conn(0x55c84de575b0 msgr2=0x55c84de5d630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:10:48.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.072+0000 7f345fe00640 0 client.25769 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:10:48.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.072+0000 7f3484401740 1 -- 172.21.15.28:0/1725558891 shutdown_connections 2024-05-12T22:10:48.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.072+0000 7f3484401740 1 -- 172.21.15.28:0/1725558891 wait complete. 2024-05-12T22:10:48.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.072+0000 7f3484401740 20 client.25769 trim_cache size 0 max 16384 2024-05-12T22:10:48.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.094+0000 7f3484401740 1 Processor -- start 2024-05-12T22:10:48.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.094+0000 7f3484401740 1 -- start start 2024-05-12T22:10:48.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.094+0000 7f3484401740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c84de5b500 0x55c84de72450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:48.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.094+0000 7f3484401740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c84de5beb0 0x55c84de5dc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:48.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.094+0000 7f3484401740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c84de66520 0x55c84de7e570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:48.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.094+0000 7f3484401740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f347c1b60d0 con 0x55c84de5beb0 2024-05-12T22:10:48.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.094+0000 7f3484401740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f347c1b39a0 con 0x55c84de5b500 2024-05-12T22:10:48.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.094+0000 7f3484401740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f347c1b17f0 con 0x55c84de66520 2024-05-12T22:10:48.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.095+0000 7f3472000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c84de5beb0 0x55c84de5dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:48.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.095+0000 7f347aa00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c84de5b500 0x55c84de72450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:48.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.095+0000 7f3472000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c84de5beb0 0x55c84de5dc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49198/0 (socket says 172.21.15.28:49198) 2024-05-12T22:10:48.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.095+0000 7f347aa00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c84de5b500 0x55c84de72450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:41584/0 (socket says 172.21.15.28:41584) 2024-05-12T22:10:48.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.095+0000 7f3472000640 1 -- 172.21.15.28:0/621371352 learned_addr learned my addr 172.21.15.28:0/621371352 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:10:48.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.095+0000 7f347b400640 1 --2- 172.21.15.28:0/621371352 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c84de66520 0x55c84de7e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:48.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.095+0000 7f347aa00640 1 -- 172.21.15.28:0/621371352 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c84de66520 msgr2=0x55c84de7e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:10:48.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.095+0000 7f347aa00640 1 --2- 172.21.15.28:0/621371352 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c84de66520 0x55c84de7e570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:48.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.095+0000 7f347aa00640 1 -- 172.21.15.28:0/621371352 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c84de5beb0 msgr2=0x55c84de5dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:10:48.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.095+0000 7f347aa00640 1 --2- 172.21.15.28:0/621371352 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c84de5beb0 0x55c84de5dc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:48.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.095+0000 7f347aa00640 1 -- 172.21.15.28:0/621371352 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c84da93170 con 0x55c84de5b500 2024-05-12T22:10:48.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.095+0000 7f347b400640 1 --2- 172.21.15.28:0/621371352 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c84de66520 0x55c84de7e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:10:48.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.095+0000 7f347aa00640 1 --2- 172.21.15.28:0/621371352 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c84de5b500 0x55c84de72450 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f346c01ea70 tx=0x7f346c009da0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=75337ebbbabe96cf server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:10:48.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.096+0000 7f346aa00640 1 -- 172.21.15.28:0/621371352 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f346c035020 con 0x55c84de5b500 2024-05-12T22:10:48.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.096+0000 7f346aa00640 1 -- 172.21.15.28:0/621371352 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f346c005560 con 0x55c84de5b500 2024-05-12T22:10:48.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.096+0000 7f346aa00640 1 -- 172.21.15.28:0/621371352 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f346c01fc10 con 0x55c84de5b500 2024-05-12T22:10:48.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.096+0000 7f3484401740 1 -- 172.21.15.28:0/621371352 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c84de5b500 msgr2=0x55c84de72450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:10:48.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.096+0000 7f3484401740 1 --2- 172.21.15.28:0/621371352 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c84de5b500 0x55c84de72450 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f346c01ea70 tx=0x7f346c009da0 comp rx=0 tx=0).stop 2024-05-12T22:10:48.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.096+0000 7f3484401740 1 -- 172.21.15.28:0/621371352 shutdown_connections 2024-05-12T22:10:48.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.096+0000 7f3484401740 1 --2- 172.21.15.28:0/621371352 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c84de66520 0x55c84de7e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:48.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.096+0000 7f3484401740 1 --2- 172.21.15.28:0/621371352 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c84de5beb0 0x55c84de5dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:48.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.096+0000 7f3484401740 1 --2- 172.21.15.28:0/621371352 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c84de5b500 0x55c84de72450 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:48.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.097+0000 7f3484401740 1 -- 172.21.15.28:0/621371352 >> 172.21.15.28:0/621371352 conn(0x55c84de575b0 msgr2=0x55c84de56730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:10:48.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.097+0000 7f3484401740 1 -- 172.21.15.28:0/621371352 shutdown_connections 2024-05-12T22:10:48.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.097+0000 7f3484401740 1 -- 172.21.15.28:0/621371352 wait complete. 2024-05-12T22:10:48.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.097+0000 7f3484401740 1 -- start start 2024-05-12T22:10:48.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.097+0000 7f3484401740 1 Processor -- start 2024-05-12T22:10:48.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.098+0000 7f3484401740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c84de5b500 0x55c84de6e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:48.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.098+0000 7f3484401740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c84de5beb0 0x55c84de6eee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:48.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.098+0000 7f3484401740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c84de66520 0x55c84de6a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:48.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.098+0000 7f3484401740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f347c1b1e60 con 0x55c84de5beb0 2024-05-12T22:10:48.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.098+0000 7f3484401740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f347c1b32d0 con 0x55c84de5b500 2024-05-12T22:10:48.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.098+0000 7f3484401740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f347c1b2530 con 0x55c84de66520 2024-05-12T22:10:48.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.098+0000 7f3484401740 10 client.0 _ll_register_callbacks cb 0x7f346c009ab0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:10:48.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.098+0000 7f3472000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c84de5beb0 0x55c84de6eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:48.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.098+0000 7f347aa00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c84de5b500 0x55c84de6e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:48.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.098+0000 7f3472000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c84de5beb0 0x55c84de6eee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49208/0 (socket says 172.21.15.28:49208) 2024-05-12T22:10:48.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.098+0000 7f3472000640 1 -- 172.21.15.28:0/637037227 learned_addr learned my addr 172.21.15.28:0/637037227 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:10:48.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.098+0000 7f347b400640 1 --2- 172.21.15.28:0/637037227 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c84de66520 0x55c84de6a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:48.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.098+0000 7f3472000640 1 -- 172.21.15.28:0/637037227 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c84de66520 msgr2=0x55c84de6a0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:10:48.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.098+0000 7f3472000640 1 --2- 172.21.15.28:0/637037227 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c84de66520 0x55c84de6a0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:48.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.099+0000 7f3472000640 1 -- 172.21.15.28:0/637037227 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c84de5b500 msgr2=0x55c84de6e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:10:48.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.099+0000 7f3472000640 1 --2- 172.21.15.28:0/637037227 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c84de5b500 0x55c84de6e9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:48.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.099+0000 7f3472000640 1 -- 172.21.15.28:0/637037227 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c84de6a8e0 con 0x55c84de5beb0 2024-05-12T22:10:48.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.099+0000 7f347aa00640 1 --2- 172.21.15.28:0/637037227 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c84de5b500 0x55c84de6e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:10:48.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.099+0000 7f3472000640 1 --2- 172.21.15.28:0/637037227 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c84de5beb0 0x55c84de6eee0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f34640b89c0 tx=0x7f346403b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0445e6f70191cba server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:10:48.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.099+0000 7f345fe00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-12T22:10:48.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.099+0000 7f345fe00640 1 -- 172.21.15.28:0/637037227 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3464049040 con 0x55c84de5beb0 2024-05-12T22:10:48.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.099+0000 7f345fe00640 1 -- 172.21.15.28:0/637037227 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f34640b8660 con 0x55c84de5beb0 2024-05-12T22:10:48.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.099+0000 7f3484401740 1 -- 172.21.15.28:0/637037227 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f346c005560 con 0x55c84de5beb0 2024-05-12T22:10:48.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.099+0000 7f345fe00640 1 -- 172.21.15.28:0/637037227 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f34640c02f0 con 0x55c84de5beb0 2024-05-12T22:10:48.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.099+0000 7f3484401740 10 client.15657 fetch_fsmap learned FSMap version 25 2024-05-12T22:10:48.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.099+0000 7f3484401740 1 -- 172.21.15.28:0/637037227 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55c84dedca80 con 0x55c84de5beb0 2024-05-12T22:10:48.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.100+0000 7f345fe00640 1 -- 172.21.15.28:0/637037227 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f346c005560 con 0x55c84de5beb0 2024-05-12T22:10:48.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.100+0000 7f345fe00640 1 -- 172.21.15.28:0/637037227 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7f34640c4720 con 0x55c84de5beb0 2024-05-12T22:10:48.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.100+0000 7f3484401740 10 client.15657 fetch_fsmap finished waiting for FSMap version 25 2024-05-12T22:10:48.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.100+0000 7f3484401740 10 client.15657 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:10:48.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.100+0000 7f3484401740 20 client.15657 populate_metadata read hostname 'smithi028' 2024-05-12T22:10:48.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.100+0000 7f3484401740 1 --2- 172.21.15.28:0/637037227 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55c84dedd230 0x55c84dedf650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:48.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.100+0000 7f3484401740 4 client.15657 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-12T22:10:48.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.100+0000 7f3484401740 1 -- 172.21.15.28:0/637037227 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55c84ddccef0 con 0x55c84dedd230 2024-05-12T22:10:48.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.100+0000 7f347aa00640 1 --2- 172.21.15.28:0/637037227 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55c84dedd230 0x55c84dedf650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:48.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.101+0000 7f347aa00640 1 --2- 172.21.15.28:0/637037227 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55c84dedd230 0x55c84dedf650 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d249c8440ad217d9 server_cookie=7ebfaf753a33a2b9 in_seq=0 out_seq=0 2024-05-12T22:10:48.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.101+0000 7f345fe00640 10 client.15657 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:10:48.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.102+0000 7f345fe00640 1 -- 172.21.15.28:0/637037227 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55c84ddccef0 con 0x55c84dedd230 2024-05-12T22:10:48.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.102+0000 7f345fe00640 10 client.15657 handle_command_reply: tid=0 2024-05-12T22:10:48.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.102+0000 7f3484401740 1 client.15657 shutdown 2024-05-12T22:10:48.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.102+0000 7f3484401740 1 -- 172.21.15.28:0/637037227 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c84de5beb0 msgr2=0x55c84de6eee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:10:48.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.102+0000 7f3484401740 1 --2- 172.21.15.28:0/637037227 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c84de5beb0 0x55c84de6eee0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f34640b89c0 tx=0x7f346403b890 comp rx=0 tx=0).stop 2024-05-12T22:10:48.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.103+0000 7f3484401740 1 -- 172.21.15.28:0/637037227 shutdown_connections 2024-05-12T22:10:48.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.103+0000 7f3484401740 1 --2- 172.21.15.28:0/637037227 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55c84dedd230 0x55c84dedf650 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:48.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.103+0000 7f3484401740 1 --2- 172.21.15.28:0/637037227 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c84de66520 0x55c84de6a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:48.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.103+0000 7f3484401740 1 --2- 172.21.15.28:0/637037227 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c84de5beb0 0x55c84de6eee0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:48.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.103+0000 7f3484401740 1 --2- 172.21.15.28:0/637037227 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c84de5b500 0x55c84de6e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:48.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.103+0000 7f345fe00640 0 client.15657 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:10:48.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.103+0000 7f3484401740 1 -- 172.21.15.28:0/637037227 >> 172.21.15.28:0/637037227 conn(0x55c84de575b0 msgr2=0x55c84de63360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:10:48.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.103+0000 7f3484401740 1 -- 172.21.15.28:0/637037227 shutdown_connections 2024-05-12T22:10:48.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.103+0000 7f3484401740 1 -- 172.21.15.28:0/637037227 wait complete. 2024-05-12T22:10:48.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.103+0000 7f3484401740 20 client.15657 trim_cache size 0 max 16384 2024-05-12T22:10:48.109 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-05-12T22:10:48.109 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-05-12T22:10:48.109 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "c6795731-9ad4-4f65-8aac-3ba1abff4a6d", 2024-05-12T22:10:48.109 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-05-12T22:10:48.109 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-05-12T22:10:48.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.104+0000 7f347be00640 1 -- 172.21.15.28:0/1814709361 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f34600a2d80 msgr2=0x7f34600a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:10:48.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.104+0000 7f347be00640 1 --2- 172.21.15.28:0/1814709361 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f34600a2d80 0x7f34600a5240 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x55c84de58830 tx=0x7f346c005f90 comp rx=0 tx=0).stop 2024-05-12T22:10:48.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.104+0000 7f347be00640 1 -- 172.21.15.28:0/1814709361 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f347c108770 msgr2=0x7f347c19ea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:10:48.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.104+0000 7f347be00640 1 --2- 172.21.15.28:0/1814709361 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f347c108770 0x7f347c19ea80 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f3464004770 tx=0x7f3464003c10 comp rx=0 tx=0).stop 2024-05-12T22:10:48.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.104+0000 7f347be00640 1 -- 172.21.15.28:0/1814709361 shutdown_connections 2024-05-12T22:10:48.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.104+0000 7f347be00640 1 --2- 172.21.15.28:0/1814709361 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f34600a2d80 0x7f34600a5240 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:48.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.104+0000 7f347be00640 1 --2- 172.21.15.28:0/1814709361 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f347c10ddf0 0x7f347c1a2e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:48.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.104+0000 7f347be00640 1 --2- 172.21.15.28:0/1814709361 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f347c108770 0x7f347c19ea80 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:48.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.104+0000 7f347be00640 1 --2- 172.21.15.28:0/1814709361 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f347c107dd0 0x7f347c19e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:48.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.104+0000 7f347be00640 1 -- 172.21.15.28:0/1814709361 >> 172.21.15.28:0/1814709361 conn(0x7f347c077650 msgr2=0x7f347c0785e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:10:48.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.104+0000 7f347be00640 1 -- 172.21.15.28:0/1814709361 shutdown_connections 2024-05-12T22:10:48.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.104+0000 7f347be00640 1 -- 172.21.15.28:0/1814709361 wait complete. 2024-05-12T22:10:48.122 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:10:47 smithi028 ceph-mon[29847]: pgmap v57: 97 pgs: 97 active+clean; 16 MiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 14 op/s 2024-05-12T22:10:48.125 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-12T22:10:48.125 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-12T22:10:48.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:10:47 smithi186 ceph-mon[34785]: pgmap v57: 97 pgs: 97 active+clean; 16 MiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 14 op/s 2024-05-12T22:10:48.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:10:47 smithi086 ceph-mon[34297]: pgmap v57: 97 pgs: 97 active+clean; 16 MiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 14 op/s 2024-05-12T22:10:48.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.406+0000 7fa400c00640 1 -- 172.21.15.28:0/3698764121 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa3fc1076e0 msgr2=0x7fa3fc107ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:10:48.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.406+0000 7fa400c00640 1 --2- 172.21.15.28:0/3698764121 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa3fc1076e0 0x7fa3fc107ac0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fa3ec009860 tx=0x7fa3ec02a340 comp rx=0 tx=0).stop 2024-05-12T22:10:48.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.406+0000 7fa400c00640 1 -- 172.21.15.28:0/3698764121 shutdown_connections 2024-05-12T22:10:48.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.406+0000 7fa400c00640 1 --2- 172.21.15.28:0/3698764121 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa3fc102090 0x7fa3fc10e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:48.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.406+0000 7fa400c00640 1 --2- 172.21.15.28:0/3698764121 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3fc1016d0 0x7fa3fc101b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:48.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.406+0000 7fa400c00640 1 --2- 172.21.15.28:0/3698764121 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa3fc1076e0 0x7fa3fc107ac0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:48.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.406+0000 7fa400c00640 1 -- 172.21.15.28:0/3698764121 >> 172.21.15.28:0/3698764121 conn(0x7fa3fc0fd200 msgr2=0x7fa3fc0ff620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:10:48.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.406+0000 7fa400c00640 1 -- 172.21.15.28:0/3698764121 shutdown_connections 2024-05-12T22:10:48.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.406+0000 7fa400c00640 1 -- 172.21.15.28:0/3698764121 wait complete. 2024-05-12T22:10:48.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.407+0000 7fa400c00640 1 Processor -- start 2024-05-12T22:10:48.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.408+0000 7fa400c00640 1 -- start start 2024-05-12T22:10:48.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.408+0000 7fa400c00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa3fc1016d0 0x7fa3fc19a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:48.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.408+0000 7fa400c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3fc102090 0x7fa3fc19a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:48.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.408+0000 7fa400c00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa3fc1076e0 0x7fa3fc19ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:48.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.408+0000 7fa400c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa3fc111af0 con 0x7fa3fc102090 2024-05-12T22:10:48.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.408+0000 7fa400c00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fa3fc111970 con 0x7fa3fc1016d0 2024-05-12T22:10:48.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.408+0000 7fa400c00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa3fc111c70 con 0x7fa3fc1076e0 2024-05-12T22:10:48.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.408+0000 7fa3fb400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa3fc1016d0 0x7fa3fc19a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:48.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.408+0000 7fa3fb400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa3fc1016d0 0x7fa3fc19a190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:41610/0 (socket says 172.21.15.28:41610) 2024-05-12T22:10:48.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.408+0000 7fa3fb400640 1 -- 172.21.15.28:0/2752963899 learned_addr learned my addr 172.21.15.28:0/2752963899 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:10:48.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.408+0000 7fa3faa00640 1 --2- 172.21.15.28:0/2752963899 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3fc102090 0x7fa3fc19a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:48.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.408+0000 7fa3fbe00640 1 --2- 172.21.15.28:0/2752963899 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa3fc1076e0 0x7fa3fc19ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:48.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.409+0000 7fa3fb400640 1 -- 172.21.15.28:0/2752963899 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa3fc1076e0 msgr2=0x7fa3fc19ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:10:48.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.409+0000 7fa3fb400640 1 --2- 172.21.15.28:0/2752963899 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa3fc1076e0 0x7fa3fc19ea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:48.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.409+0000 7fa3fb400640 1 -- 172.21.15.28:0/2752963899 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3fc102090 msgr2=0x7fa3fc19a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:10:48.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.409+0000 7fa3fb400640 1 --2- 172.21.15.28:0/2752963899 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3fc102090 0x7fa3fc19a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:48.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.409+0000 7fa3fb400640 1 -- 172.21.15.28:0/2752963899 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3fc19f180 con 0x7fa3fc1016d0 2024-05-12T22:10:48.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.409+0000 7fa3fb400640 1 --2- 172.21.15.28:0/2752963899 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa3fc1016d0 0x7fa3fc19a190 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fa3ec035040 tx=0x7fa3ec033b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:10:48.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.409+0000 7fa3f3e00640 1 -- 172.21.15.28:0/2752963899 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3ec037040 con 0x7fa3fc1016d0 2024-05-12T22:10:48.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.409+0000 7fa3f3e00640 1 -- 172.21.15.28:0/2752963899 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa3ec040070 con 0x7fa3fc1016d0 2024-05-12T22:10:48.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.409+0000 7fa400c00640 1 -- 172.21.15.28:0/2752963899 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3fc19f410 con 0x7fa3fc1016d0 2024-05-12T22:10:48.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.410+0000 7fa3f3e00640 1 -- 172.21.15.28:0/2752963899 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3ec003a10 con 0x7fa3fc1016d0 2024-05-12T22:10:48.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.410+0000 7fa400c00640 1 -- 172.21.15.28:0/2752963899 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3fc1a6d00 con 0x7fa3fc1016d0 2024-05-12T22:10:48.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.414+0000 7fa4051c9740 1 Processor -- start 2024-05-12T22:10:48.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.415+0000 7fa3f3e00640 1 -- 172.21.15.28:0/2752963899 <== mon.2 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7fa3ec004150 con 0x7fa3fc1016d0 2024-05-12T22:10:48.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.415+0000 7fa4051c9740 1 -- start start 2024-05-12T22:10:48.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.415+0000 7fa4051c9740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5609bc9cedd0 0x5609bc9cf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:48.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.415+0000 7fa4051c9740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5609bc9cf6f0 0x5609bc9d9880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:48.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.415+0000 7fa4051c9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5609bc9d9dc0 0x5609bc9dc1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:48.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.415+0000 7fa4051c9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5609bc9972a0 con 0x5609bc9d9dc0 2024-05-12T22:10:48.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.415+0000 7fa4051c9740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x5609bc933b90 con 0x5609bc9cedd0 2024-05-12T22:10:48.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.415+0000 7fa4051c9740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5609bc8db5b0 con 0x5609bc9cf6f0 2024-05-12T22:10:48.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.415+0000 7fa3fbe00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5609bc9cedd0 0x5609bc9cf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:48.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.415+0000 7fa3fb400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5609bc9cf6f0 0x5609bc9d9880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:48.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.415+0000 7fa3fbe00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5609bc9cedd0 0x5609bc9cf1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:41622/0 (socket says 172.21.15.28:41622) 2024-05-12T22:10:48.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.415+0000 7fa3fbe00640 1 -- 172.21.15.28:0/2306231969 learned_addr learned my addr 172.21.15.28:0/2306231969 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:10:48.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.416+0000 7fa3faa00640 1 --2- 172.21.15.28:0/2306231969 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5609bc9d9dc0 0x5609bc9dc1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:48.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.416+0000 7fa3fb400640 1 -- 172.21.15.28:0/2306231969 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5609bc9cedd0 msgr2=0x5609bc9cf1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:10:48.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.416+0000 7fa3fb400640 1 --2- 172.21.15.28:0/2306231969 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5609bc9cedd0 0x5609bc9cf1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:48.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.416+0000 7fa3fb400640 1 -- 172.21.15.28:0/2306231969 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5609bc9d9dc0 msgr2=0x5609bc9dc1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:10:48.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.416+0000 7fa3fb400640 1 --2- 172.21.15.28:0/2306231969 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5609bc9d9dc0 0x5609bc9dc1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:48.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.416+0000 7fa3fb400640 1 -- 172.21.15.28:0/2306231969 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5609bc7117d0 con 0x5609bc9cf6f0 2024-05-12T22:10:48.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.416+0000 7fa3fb400640 1 --2- 172.21.15.28:0/2306231969 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5609bc9cf6f0 0x5609bc9d9880 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fa3ec0ecb30 tx=0x7fa3ec0f73f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=da98e715ee767f13 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:10:48.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.416+0000 7fa3e7e00640 1 -- 172.21.15.28:0/2306231969 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3ec0f10d0 con 0x5609bc9cf6f0 2024-05-12T22:10:48.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.416+0000 7fa3e7e00640 1 -- 172.21.15.28:0/2306231969 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa3ec0fc030 con 0x5609bc9cf6f0 2024-05-12T22:10:48.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.416+0000 7fa3f3e00640 1 --2- 172.21.15.28:0/2752963899 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fa3e00a2d00 0x7fa3e00a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:48.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.417+0000 7fa3e7e00640 1 -- 172.21.15.28:0/2306231969 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3ec100610 con 0x5609bc9cf6f0 2024-05-12T22:10:48.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.417+0000 7fa3f3e00640 1 -- 172.21.15.28:0/2752963899 <== mon.2 v2:172.21.15.86:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7fa3ec0ef300 con 0x7fa3fc1016d0 2024-05-12T22:10:48.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.417+0000 7fa3fbe00640 1 --2- 172.21.15.28:0/2752963899 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fa3e00a2d00 0x7fa3e00a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:48.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.417+0000 7fa4051c9740 1 -- 172.21.15.28:0/2306231969 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5609bc9cf6f0 msgr2=0x5609bc9d9880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:10:48.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.417+0000 7fa4051c9740 1 --2- 172.21.15.28:0/2306231969 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5609bc9cf6f0 0x5609bc9d9880 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fa3ec0ecb30 tx=0x7fa3ec0f73f0 comp rx=0 tx=0).stop 2024-05-12T22:10:48.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.418+0000 7fa4051c9740 1 -- 172.21.15.28:0/2306231969 shutdown_connections 2024-05-12T22:10:48.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.418+0000 7fa4051c9740 1 --2- 172.21.15.28:0/2306231969 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5609bc9d9dc0 0x5609bc9dc1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:48.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.418+0000 7fa4051c9740 1 --2- 172.21.15.28:0/2306231969 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5609bc9cf6f0 0x5609bc9d9880 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:48.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.418+0000 7fa4051c9740 1 --2- 172.21.15.28:0/2306231969 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5609bc9cedd0 0x5609bc9cf1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:48.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.418+0000 7fa4051c9740 1 -- 172.21.15.28:0/2306231969 >> 172.21.15.28:0/2306231969 conn(0x5609bc9cb0f0 msgr2=0x5609bc9cb4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:10:48.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.418+0000 7fa3fbe00640 1 --2- 172.21.15.28:0/2752963899 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fa3e00a2d00 0x7fa3e00a51c0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fa3f4011630 tx=0x7fa3f4009500 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:10:48.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.418+0000 7fa4051c9740 1 -- 172.21.15.28:0/2306231969 shutdown_connections 2024-05-12T22:10:48.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.418+0000 7fa4051c9740 1 -- 172.21.15.28:0/2306231969 wait complete. 2024-05-12T22:10:48.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.418+0000 7fa4051c9740 1 -- start start 2024-05-12T22:10:48.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.419+0000 7fa4051c9740 1 Processor -- start 2024-05-12T22:10:48.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.419+0000 7fa4051c9740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5609bc9cedd0 0x5609bc9d9780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:48.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.419+0000 7fa4051c9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5609bc9cf6f0 0x5609bc9d6a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:48.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.419+0000 7fa4051c9740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5609bc9d9dc0 0x5609bc9d7080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:48.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.419+0000 7fa4051c9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5609bc8ee9f0 con 0x5609bc9cf6f0 2024-05-12T22:10:48.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.419+0000 7fa4051c9740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x5609bc92fb50 con 0x5609bc9d9dc0 2024-05-12T22:10:48.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.419+0000 7fa4051c9740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5609bc9dc970 con 0x5609bc9cedd0 2024-05-12T22:10:48.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.419+0000 7fa4051c9740 10 client.0 _ll_register_callbacks cb 0x5609bc646be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:10:48.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.420+0000 7fa3faa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5609bc9cf6f0 0x5609bc9d6a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:48.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.420+0000 7fa3fb400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5609bc9cedd0 0x5609bc9d9780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:48.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.420+0000 7fa3fbe00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5609bc9d9dc0 0x5609bc9d7080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:48.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.420+0000 7fa3faa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5609bc9cf6f0 0x5609bc9d6a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49246/0 (socket says 172.21.15.28:49246) 2024-05-12T22:10:48.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.420+0000 7fa3faa00640 1 -- 172.21.15.28:0/3336913331 learned_addr learned my addr 172.21.15.28:0/3336913331 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:10:48.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.420+0000 7fa3fb400640 1 -- 172.21.15.28:0/3336913331 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5609bc9d9dc0 msgr2=0x5609bc9d7080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:10:48.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.420+0000 7fa3fb400640 1 --2- 172.21.15.28:0/3336913331 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5609bc9d9dc0 0x5609bc9d7080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:48.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.420+0000 7fa3fb400640 1 -- 172.21.15.28:0/3336913331 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5609bc9cf6f0 msgr2=0x5609bc9d6a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:10:48.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.420+0000 7fa3fb400640 1 --2- 172.21.15.28:0/3336913331 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5609bc9cf6f0 0x5609bc9d6a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:48.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.420+0000 7fa3fb400640 1 -- 172.21.15.28:0/3336913331 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5609bc6473f0 con 0x5609bc9cedd0 2024-05-12T22:10:48.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.420+0000 7fa3faa00640 1 --2- 172.21.15.28:0/3336913331 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5609bc9cf6f0 0x5609bc9d6a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:10:48.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.420+0000 7fa3fb400640 1 --2- 172.21.15.28:0/3336913331 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5609bc9cedd0 0x5609bc9d9780 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fa3ec0ecb90 tx=0x7fa3ec0044f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cac0b26cb7047953 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:10:48.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.421+0000 7fa3e4c00640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-12T22:10:48.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.421+0000 7fa3e4c00640 1 -- 172.21.15.28:0/3336913331 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3ec0ee030 con 0x5609bc9cedd0 2024-05-12T22:10:48.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.421+0000 7fa4051c9740 1 -- 172.21.15.28:0/3336913331 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa3ec0fc030 con 0x5609bc9cedd0 2024-05-12T22:10:48.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.421+0000 7fa3e4c00640 1 -- 172.21.15.28:0/3336913331 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7fa3ec0f7e00 con 0x5609bc9cedd0 2024-05-12T22:10:48.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.421+0000 7fa3e4c00640 1 -- 172.21.15.28:0/3336913331 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fa3ec0efe60 con 0x5609bc9cedd0 2024-05-12T22:10:48.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.421+0000 7fa4051c9740 10 client.25741 fetch_fsmap learned FSMap version 25 2024-05-12T22:10:48.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.421+0000 7fa4051c9740 1 -- 172.21.15.28:0/3336913331 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5609bc6070f0 con 0x5609bc9cedd0 2024-05-12T22:10:48.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.421+0000 7fa3e4c00640 1 -- 172.21.15.28:0/3336913331 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa3ec0fc030 con 0x5609bc9cedd0 2024-05-12T22:10:48.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.422+0000 7fa3e4c00640 1 -- 172.21.15.28:0/3336913331 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7fa3ec03a070 con 0x5609bc9cedd0 2024-05-12T22:10:48.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.422+0000 7fa4051c9740 10 client.25741 fetch_fsmap finished waiting for FSMap version 25 2024-05-12T22:10:48.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.422+0000 7fa4051c9740 10 client.25741 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:10:48.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.422+0000 7fa4051c9740 20 client.25741 populate_metadata read hostname 'smithi028' 2024-05-12T22:10:48.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.422+0000 7fa4051c9740 1 --2- 172.21.15.28:0/3336913331 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5609bc9e9500 0x5609bc9eb8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:48.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.422+0000 7fa4051c9740 4 client.25741 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-12T22:10:48.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.422+0000 7fa4051c9740 1 -- 172.21.15.28:0/3336913331 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5609bc9412c0 con 0x5609bc9e9500 2024-05-12T22:10:48.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.422+0000 7fa3faa00640 1 --2- 172.21.15.28:0/3336913331 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5609bc9e9500 0x5609bc9eb8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:48.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.423+0000 7fa3faa00640 1 --2- 172.21.15.28:0/3336913331 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5609bc9e9500 0x5609bc9eb8f0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=651d8c7cd9700173 server_cookie=1d44f33f3e682e88 in_seq=0 out_seq=0 2024-05-12T22:10:48.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.423+0000 7fa3e4c00640 10 client.25741 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:10:48.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.424+0000 7fa3e4c00640 1 -- 172.21.15.28:0/3336913331 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5609bc9412c0 con 0x5609bc9e9500 2024-05-12T22:10:48.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.424+0000 7fa3e4c00640 10 client.25741 handle_command_reply: tid=0 2024-05-12T22:10:48.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.424+0000 7fa4051c9740 1 client.25741 shutdown 2024-05-12T22:10:48.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.424+0000 7fa4051c9740 1 -- 172.21.15.28:0/3336913331 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5609bc9cedd0 msgr2=0x5609bc9d9780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:10:48.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.424+0000 7fa4051c9740 1 --2- 172.21.15.28:0/3336913331 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5609bc9cedd0 0x5609bc9d9780 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fa3ec0ecb90 tx=0x7fa3ec0044f0 comp rx=0 tx=0).stop 2024-05-12T22:10:48.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.425+0000 7fa4051c9740 1 -- 172.21.15.28:0/3336913331 shutdown_connections 2024-05-12T22:10:48.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.425+0000 7fa4051c9740 1 --2- 172.21.15.28:0/3336913331 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5609bc9e9500 0x5609bc9eb8f0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:48.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.425+0000 7fa3e4c00640 0 client.25741 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:10:48.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.425+0000 7fa4051c9740 1 --2- 172.21.15.28:0/3336913331 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5609bc9d9dc0 0x5609bc9d7080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:48.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.425+0000 7fa4051c9740 1 --2- 172.21.15.28:0/3336913331 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5609bc9cf6f0 0x5609bc9d6a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:48.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.425+0000 7fa4051c9740 1 --2- 172.21.15.28:0/3336913331 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5609bc9cedd0 0x5609bc9d9780 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:48.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.425+0000 7fa4051c9740 1 -- 172.21.15.28:0/3336913331 >> 172.21.15.28:0/3336913331 conn(0x5609bc9cb0f0 msgr2=0x5609bc9d0ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:10:48.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.425+0000 7fa4051c9740 1 -- 172.21.15.28:0/3336913331 shutdown_connections 2024-05-12T22:10:48.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.425+0000 7fa4051c9740 1 -- 172.21.15.28:0/3336913331 wait complete. 2024-05-12T22:10:48.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.425+0000 7fa4051c9740 20 client.25741 trim_cache size 0 max 16384 2024-05-12T22:10:48.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.446+0000 7fa4051c9740 1 Processor -- start 2024-05-12T22:10:48.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.447+0000 7fa4051c9740 1 -- start start 2024-05-12T22:10:48.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.447+0000 7fa4051c9740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5609bc9cb0f0 0x5609bc9d92b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:48.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.447+0000 7fa4051c9740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5609bc9cedd0 0x5609bc9e54a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:48.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.447+0000 7fa4051c9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5609bc9cf6f0 0x5609bc9e59e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:48.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.447+0000 7fa4051c9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa3fc1b1d90 con 0x5609bc9cf6f0 2024-05-12T22:10:48.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.447+0000 7fa4051c9740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fa3fc1af660 con 0x5609bc9cedd0 2024-05-12T22:10:48.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.447+0000 7fa4051c9740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa3fc1ad4b0 con 0x5609bc9cb0f0 2024-05-12T22:10:48.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.447+0000 7fa3faa00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5609bc9cedd0 0x5609bc9e54a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:48.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.447+0000 7fa3faa00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5609bc9cedd0 0x5609bc9e54a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:41640/0 (socket says 172.21.15.28:41640) 2024-05-12T22:10:48.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.447+0000 7fa3faa00640 1 -- 172.21.15.28:0/2733911462 learned_addr learned my addr 172.21.15.28:0/2733911462 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:10:48.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.447+0000 7fa3fbe00640 1 --2- 172.21.15.28:0/2733911462 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5609bc9cf6f0 0x5609bc9e59e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:48.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.447+0000 7fa3fb400640 1 --2- 172.21.15.28:0/2733911462 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5609bc9cb0f0 0x5609bc9d92b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:48.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.448+0000 7fa3faa00640 1 -- 172.21.15.28:0/2733911462 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5609bc9cb0f0 msgr2=0x5609bc9d92b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:10:48.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.448+0000 7fa3faa00640 1 --2- 172.21.15.28:0/2733911462 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5609bc9cb0f0 0x5609bc9d92b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:48.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.448+0000 7fa3faa00640 1 -- 172.21.15.28:0/2733911462 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5609bc9cf6f0 msgr2=0x5609bc9e59e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:10:48.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.448+0000 7fa3faa00640 1 --2- 172.21.15.28:0/2733911462 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5609bc9cf6f0 0x5609bc9e59e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:48.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.448+0000 7fa3faa00640 1 -- 172.21.15.28:0/2733911462 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5609bc9cf520 con 0x5609bc9cedd0 2024-05-12T22:10:48.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.448+0000 7fa3faa00640 1 --2- 172.21.15.28:0/2733911462 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5609bc9cedd0 0x5609bc9e54a0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fa3e8004700 tx=0x7fa3e8009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6dd7ca27acd45ed5 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:10:48.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.448+0000 7fa3e7400640 1 -- 172.21.15.28:0/2733911462 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3e8004020 con 0x5609bc9cedd0 2024-05-12T22:10:48.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.448+0000 7fa3e7400640 1 -- 172.21.15.28:0/2733911462 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa3e8024030 con 0x5609bc9cedd0 2024-05-12T22:10:48.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.448+0000 7fa3e7400640 1 -- 172.21.15.28:0/2733911462 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3e8004bc0 con 0x5609bc9cedd0 2024-05-12T22:10:48.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.449+0000 7fa4051c9740 1 -- 172.21.15.28:0/2733911462 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5609bc9cedd0 msgr2=0x5609bc9e54a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:10:48.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.449+0000 7fa4051c9740 1 --2- 172.21.15.28:0/2733911462 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5609bc9cedd0 0x5609bc9e54a0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fa3e8004700 tx=0x7fa3e8009260 comp rx=0 tx=0).stop 2024-05-12T22:10:48.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.449+0000 7fa4051c9740 1 -- 172.21.15.28:0/2733911462 shutdown_connections 2024-05-12T22:10:48.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.449+0000 7fa4051c9740 1 --2- 172.21.15.28:0/2733911462 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5609bc9cf6f0 0x5609bc9e59e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:48.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.449+0000 7fa4051c9740 1 --2- 172.21.15.28:0/2733911462 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5609bc9cedd0 0x5609bc9e54a0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:48.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.449+0000 7fa4051c9740 1 --2- 172.21.15.28:0/2733911462 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5609bc9cb0f0 0x5609bc9d92b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:48.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.449+0000 7fa4051c9740 1 -- 172.21.15.28:0/2733911462 >> 172.21.15.28:0/2733911462 conn(0x5609bc9c6530 msgr2=0x5609bc9ca330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:10:48.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.450+0000 7fa4051c9740 1 -- 172.21.15.28:0/2733911462 shutdown_connections 2024-05-12T22:10:48.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.450+0000 7fa4051c9740 1 -- 172.21.15.28:0/2733911462 wait complete. 2024-05-12T22:10:48.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.450+0000 7fa4051c9740 1 -- start start 2024-05-12T22:10:48.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.450+0000 7fa4051c9740 1 Processor -- start 2024-05-12T22:10:48.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.451+0000 7fa4051c9740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5609bc9cb0f0 0x5609bc9df6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:48.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.451+0000 7fa4051c9740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5609bc9cedd0 0x5609bc9dfbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:48.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.451+0000 7fa4051c9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5609bc9cf6f0 0x5609bc9e0120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:48.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.451+0000 7fa4051c9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa3fc1adb20 con 0x5609bc9cf6f0 2024-05-12T22:10:48.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.451+0000 7fa4051c9740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fa3fc1aef90 con 0x5609bc9cedd0 2024-05-12T22:10:48.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.451+0000 7fa4051c9740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa3fc1ae1f0 con 0x5609bc9cb0f0 2024-05-12T22:10:48.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.451+0000 7fa4051c9740 10 client.0 _ll_register_callbacks cb 0x7fa3f4009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:10:48.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.451+0000 7fa3fb400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5609bc9cb0f0 0x5609bc9df6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:48.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.451+0000 7fa3faa00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5609bc9cedd0 0x5609bc9dfbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:48.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.451+0000 7fa3fb400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5609bc9cb0f0 0x5609bc9df6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:48522/0 (socket says 172.21.15.28:48522) 2024-05-12T22:10:48.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.451+0000 7fa3fb400640 1 -- 172.21.15.28:0/1113963825 learned_addr learned my addr 172.21.15.28:0/1113963825 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:10:48.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.451+0000 7fa3fbe00640 1 --2- 172.21.15.28:0/1113963825 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5609bc9cf6f0 0x5609bc9e0120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:48.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.451+0000 7fa3fb400640 1 -- 172.21.15.28:0/1113963825 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5609bc9cedd0 msgr2=0x5609bc9dfbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:10:48.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.451+0000 7fa3fb400640 1 --2- 172.21.15.28:0/1113963825 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5609bc9cedd0 0x5609bc9dfbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:48.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.451+0000 7fa3fb400640 1 -- 172.21.15.28:0/1113963825 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5609bc9cf6f0 msgr2=0x5609bc9e0120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:10:48.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.451+0000 7fa3fb400640 1 --2- 172.21.15.28:0/1113963825 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5609bc9cf6f0 0x5609bc9e0120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:48.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.451+0000 7fa3fb400640 1 -- 172.21.15.28:0/1113963825 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5609bca4ebf0 con 0x5609bc9cb0f0 2024-05-12T22:10:48.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.452+0000 7fa3fbe00640 1 --2- 172.21.15.28:0/1113963825 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5609bc9cf6f0 0x5609bc9e0120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:10:48.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.452+0000 7fa3fb400640 1 --2- 172.21.15.28:0/1113963825 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5609bc9cb0f0 0x5609bc9df6a0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fa3ec0b8280 tx=0x7fa3ec0bc0d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=695fe68508b065c7 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:10:48.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.452+0000 7fa3e4c00640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-12T22:10:48.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.452+0000 7fa3e4c00640 1 -- 172.21.15.28:0/1113963825 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3ec0ee070 con 0x5609bc9cb0f0 2024-05-12T22:10:48.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.452+0000 7fa3e4c00640 1 -- 172.21.15.28:0/1113963825 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7fa3ec0c42e0 con 0x5609bc9cb0f0 2024-05-12T22:10:48.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.452+0000 7fa4051c9740 1 -- 172.21.15.28:0/1113963825 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa3e8024030 con 0x5609bc9cb0f0 2024-05-12T22:10:48.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.452+0000 7fa3e4c00640 1 -- 172.21.15.28:0/1113963825 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fa3ec0bb440 con 0x5609bc9cb0f0 2024-05-12T22:10:48.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.452+0000 7fa4051c9740 10 client.25747 fetch_fsmap learned FSMap version 25 2024-05-12T22:10:48.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.452+0000 7fa4051c9740 1 -- 172.21.15.28:0/1113963825 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5609bca4ebf0 con 0x5609bc9cb0f0 2024-05-12T22:10:48.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.453+0000 7fa3e4c00640 1 -- 172.21.15.28:0/1113963825 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa3e8024030 con 0x5609bc9cb0f0 2024-05-12T22:10:48.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.453+0000 7fa3e4c00640 1 -- 172.21.15.28:0/1113963825 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7fa3ec03a070 con 0x5609bc9cb0f0 2024-05-12T22:10:48.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.453+0000 7fa4051c9740 10 client.25747 fetch_fsmap finished waiting for FSMap version 25 2024-05-12T22:10:48.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.453+0000 7fa4051c9740 10 client.25747 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:10:48.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.453+0000 7fa4051c9740 20 client.25747 populate_metadata read hostname 'smithi028' 2024-05-12T22:10:48.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.453+0000 7fa4051c9740 1 --2- 172.21.15.28:0/1113963825 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5609bc9d9dc0 0x5609bca52260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:10:48.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.453+0000 7fa4051c9740 4 client.25747 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-12T22:10:48.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.453+0000 7fa4051c9740 1 -- 172.21.15.28:0/1113963825 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5609bc93b610 con 0x5609bc9d9dc0 2024-05-12T22:10:48.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.453+0000 7fa3faa00640 1 --2- 172.21.15.28:0/1113963825 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5609bc9d9dc0 0x5609bca52260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:10:48.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.454+0000 7fa3faa00640 1 --2- 172.21.15.28:0/1113963825 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5609bc9d9dc0 0x5609bca52260 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5059982112c0358e server_cookie=1bbee7c935cb7adc in_seq=0 out_seq=0 2024-05-12T22:10:48.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.454+0000 7fa3e4c00640 10 client.25747 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:10:48.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.455+0000 7fa3e4c00640 1 -- 172.21.15.28:0/1113963825 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+266 (crc 0 0 0) 0x5609bc93b610 con 0x5609bc9d9dc0 2024-05-12T22:10:48.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.456+0000 7fa3e4c00640 10 client.25747 handle_command_reply: tid=0 2024-05-12T22:10:48.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.456+0000 7fa4051c9740 1 client.25747 shutdown 2024-05-12T22:10:48.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.456+0000 7fa4051c9740 1 -- 172.21.15.28:0/1113963825 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5609bc9cb0f0 msgr2=0x5609bc9df6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:10:48.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.456+0000 7fa4051c9740 1 --2- 172.21.15.28:0/1113963825 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5609bc9cb0f0 0x5609bc9df6a0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fa3ec0b8280 tx=0x7fa3ec0bc0d0 comp rx=0 tx=0).stop 2024-05-12T22:10:48.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.456+0000 7fa4051c9740 1 -- 172.21.15.28:0/1113963825 shutdown_connections 2024-05-12T22:10:48.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.456+0000 7fa4051c9740 1 --2- 172.21.15.28:0/1113963825 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5609bc9d9dc0 0x5609bca52260 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:48.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.456+0000 7fa4051c9740 1 --2- 172.21.15.28:0/1113963825 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5609bc9cf6f0 0x5609bc9e0120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:48.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.456+0000 7fa3e4c00640 0 client.25747 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:10:48.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.456+0000 7fa4051c9740 1 --2- 172.21.15.28:0/1113963825 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5609bc9cedd0 0x5609bc9dfbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:48.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.456+0000 7fa4051c9740 1 --2- 172.21.15.28:0/1113963825 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5609bc9cb0f0 0x5609bc9df6a0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:48.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.456+0000 7fa4051c9740 1 -- 172.21.15.28:0/1113963825 >> 172.21.15.28:0/1113963825 conn(0x5609bc9c6530 msgr2=0x5609bc9d82a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:10:48.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.456+0000 7fa4051c9740 1 -- 172.21.15.28:0/1113963825 shutdown_connections 2024-05-12T22:10:48.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.456+0000 7fa4051c9740 1 -- 172.21.15.28:0/1113963825 wait complete. 2024-05-12T22:10:48.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.456+0000 7fa4051c9740 20 client.25747 trim_cache size 0 max 16384 2024-05-12T22:10:48.459 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-05-12T22:10:48.460 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "scrub active (25 inodes in the stack)", 2024-05-12T22:10:48.460 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-05-12T22:10:48.460 INFO:teuthology.orchestra.run.smithi028.stdout: "c6795731-9ad4-4f65-8aac-3ba1abff4a6d": { 2024-05-12T22:10:48.460 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-05-12T22:10:48.460 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "c6795731-9ad4-4f65-8aac-3ba1abff4a6d", 2024-05-12T22:10:48.460 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-05-12T22:10:48.460 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:10:48.460 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:10:48.460 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-05-12T22:10:48.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.457+0000 7fa400c00640 1 -- 172.21.15.28:0/2752963899 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fa3e00a2d00 msgr2=0x7fa3e00a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:10:48.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.457+0000 7fa400c00640 1 --2- 172.21.15.28:0/2752963899 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fa3e00a2d00 0x7fa3e00a51c0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fa3f4011630 tx=0x7fa3f4009500 comp rx=0 tx=0).stop 2024-05-12T22:10:48.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.457+0000 7fa400c00640 1 -- 172.21.15.28:0/2752963899 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa3fc1016d0 msgr2=0x7fa3fc19a190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:10:48.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.457+0000 7fa400c00640 1 --2- 172.21.15.28:0/2752963899 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa3fc1016d0 0x7fa3fc19a190 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fa3ec035040 tx=0x7fa3ec033b60 comp rx=0 tx=0).stop 2024-05-12T22:10:48.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.457+0000 7fa400c00640 1 -- 172.21.15.28:0/2752963899 shutdown_connections 2024-05-12T22:10:48.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.457+0000 7fa400c00640 1 --2- 172.21.15.28:0/2752963899 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fa3e00a2d00 0x7fa3e00a51c0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:48.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.457+0000 7fa400c00640 1 --2- 172.21.15.28:0/2752963899 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa3fc1076e0 0x7fa3fc19ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:48.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.457+0000 7fa400c00640 1 --2- 172.21.15.28:0/2752963899 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3fc102090 0x7fa3fc19a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:48.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.457+0000 7fa400c00640 1 --2- 172.21.15.28:0/2752963899 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa3fc1016d0 0x7fa3fc19a190 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:10:48.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.457+0000 7fa400c00640 1 -- 172.21.15.28:0/2752963899 >> 172.21.15.28:0/2752963899 conn(0x7fa3fc0fd200 msgr2=0x7fa3fc10ce20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:10:48.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.457+0000 7fa400c00640 1 -- 172.21.15.28:0/2752963899 shutdown_connections 2024-05-12T22:10:48.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:10:48.457+0000 7fa400c00640 1 -- 172.21.15.28:0/2752963899 wait complete. 2024-05-12T22:10:48.476 INFO:tasks.cephfs.filesystem:scrub status for tag:c6795731-9ad4-4f65-8aac-3ba1abff4a6d - {'path': '/', 'tag': 'c6795731-9ad4-4f65-8aac-3ba1abff4a6d', 'options': 'recursive,force'} 2024-05-12T22:10:49.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:10:48 smithi186 ceph-mon[34785]: scrub queued for path: / 2024-05-12T22:10:49.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:10:48 smithi186 ceph-mon[34785]: scrub summary: idle+waiting paths [/] 2024-05-12T22:10:49.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:10:48 smithi186 ceph-mon[34785]: scrub summary: active paths [/] 2024-05-12T22:10:49.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:10:48 smithi186 ceph-mon[34785]: scrub summary: idle+waiting paths [/] 2024-05-12T22:10:49.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:10:48 smithi086 ceph-mon[34297]: scrub queued for path: / 2024-05-12T22:10:49.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:10:48 smithi086 ceph-mon[34297]: scrub summary: idle+waiting paths [/] 2024-05-12T22:10:49.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:10:48 smithi086 ceph-mon[34297]: scrub summary: active paths [/] 2024-05-12T22:10:49.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:10:48 smithi086 ceph-mon[34297]: scrub summary: idle+waiting paths [/] 2024-05-12T22:10:49.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:10:48 smithi028 ceph-mon[29847]: scrub queued for path: / 2024-05-12T22:10:49.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:10:48 smithi028 ceph-mon[29847]: scrub summary: idle+waiting paths [/] 2024-05-12T22:10:49.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:10:48 smithi028 ceph-mon[29847]: scrub summary: active paths [/] 2024-05-12T22:10:49.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:10:48 smithi028 ceph-mon[29847]: scrub summary: idle+waiting paths [/] 2024-05-12T22:10:50.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:10:49 smithi186 ceph-mon[34785]: scrub summary: active paths [/] 2024-05-12T22:10:50.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:10:49 smithi186 ceph-mon[34785]: scrub summary: idle+waiting paths [/] 2024-05-12T22:10:50.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:10:49 smithi186 ceph-mon[34785]: pgmap v58: 97 pgs: 97 active+clean; 16 MiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 14 op/s 2024-05-12T22:10:50.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:10:49 smithi086 ceph-mon[34297]: scrub summary: active paths [/] 2024-05-12T22:10:50.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:10:49 smithi086 ceph-mon[34297]: scrub summary: idle+waiting paths [/] 2024-05-12T22:10:50.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:10:49 smithi086 ceph-mon[34297]: pgmap v58: 97 pgs: 97 active+clean; 16 MiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 14 op/s 2024-05-12T22:10:50.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:10:49 smithi028 ceph-mon[29847]: scrub summary: active paths [/] 2024-05-12T22:10:50.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:10:49 smithi028 ceph-mon[29847]: scrub summary: idle+waiting paths [/] 2024-05-12T22:10:50.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:10:49 smithi028 ceph-mon[29847]: pgmap v58: 97 pgs: 97 active+clean; 16 MiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 14 op/s 2024-05-12T22:10:52.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:10:51 smithi186 ceph-mon[34785]: pgmap v59: 97 pgs: 97 active+clean; 19 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.5 MiB/s wr, 21 op/s 2024-05-12T22:10:52.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:10:51 smithi086 ceph-mon[34297]: pgmap v59: 97 pgs: 97 active+clean; 19 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.5 MiB/s wr, 21 op/s 2024-05-12T22:10:52.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:10:51 smithi028 ceph-mon[29847]: pgmap v59: 97 pgs: 97 active+clean; 19 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.5 MiB/s wr, 21 op/s 2024-05-12T22:10:54.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:10:53 smithi186 ceph-mon[34785]: pgmap v60: 97 pgs: 97 active+clean; 19 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 277 KiB/s rd, 384 KiB/s wr, 14 op/s 2024-05-12T22:10:54.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:10:53 smithi086 ceph-mon[34297]: pgmap v60: 97 pgs: 97 active+clean; 19 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 277 KiB/s rd, 384 KiB/s wr, 14 op/s 2024-05-12T22:10:54.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:10:53 smithi028 ceph-mon[29847]: pgmap v60: 97 pgs: 97 active+clean; 19 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 277 KiB/s rd, 384 KiB/s wr, 14 op/s 2024-05-12T22:10:56.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:10:55 smithi186 ceph-mon[34785]: pgmap v61: 97 pgs: 97 active+clean; 20 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 444 KiB/s rd, 564 KiB/s wr, 33 op/s 2024-05-12T22:10:56.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:10:55 smithi086 ceph-mon[34297]: pgmap v61: 97 pgs: 97 active+clean; 20 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 444 KiB/s rd, 564 KiB/s wr, 33 op/s 2024-05-12T22:10:56.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:10:55 smithi028 ceph-mon[29847]: pgmap v61: 97 pgs: 97 active+clean; 20 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 444 KiB/s rd, 564 KiB/s wr, 33 op/s 2024-05-12T22:10:58.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:10:57 smithi186 ceph-mon[34785]: pgmap v62: 97 pgs: 97 active+clean; 20 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 382 KiB/s rd, 489 KiB/s wr, 26 op/s 2024-05-12T22:10:58.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:10:57 smithi186 ceph-mon[34785]: scrub summary: idle 2024-05-12T22:10:58.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:10:57 smithi186 ceph-mon[34785]: scrub summary: idle 2024-05-12T22:10:58.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:10:57 smithi086 ceph-mon[34297]: pgmap v62: 97 pgs: 97 active+clean; 20 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 382 KiB/s rd, 489 KiB/s wr, 26 op/s 2024-05-12T22:10:58.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:10:57 smithi086 ceph-mon[34297]: scrub summary: idle 2024-05-12T22:10:58.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:10:57 smithi086 ceph-mon[34297]: scrub summary: idle 2024-05-12T22:10:58.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:10:57 smithi028 ceph-mon[29847]: pgmap v62: 97 pgs: 97 active+clean; 20 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 382 KiB/s rd, 489 KiB/s wr, 26 op/s 2024-05-12T22:10:58.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:10:57 smithi028 ceph-mon[29847]: scrub summary: idle 2024-05-12T22:10:58.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:10:57 smithi028 ceph-mon[29847]: scrub summary: idle 2024-05-12T22:11:00.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:10:59 smithi186 ceph-mon[34785]: pgmap v63: 97 pgs: 97 active+clean; 20 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 382 KiB/s rd, 489 KiB/s wr, 26 op/s 2024-05-12T22:11:00.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:10:59 smithi086 ceph-mon[34297]: pgmap v63: 97 pgs: 97 active+clean; 20 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 382 KiB/s rd, 489 KiB/s wr, 26 op/s 2024-05-12T22:11:00.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:10:59 smithi028 ceph-mon[29847]: pgmap v63: 97 pgs: 97 active+clean; 20 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 382 KiB/s rd, 489 KiB/s wr, 26 op/s 2024-05-12T22:11:01.008 INFO:tasks.workunit.client.0.smithi028.stderr:Updating files: 7% (321/4295) Updating files: 8% (344/4295) Updating files: 9% (387/4295) Updating files: 10% (430/4295) Updating files: 11% (473/4295) Updating files: 11% (490/4295) Updating files: 12% (516/4295) Updating files: 13% (559/4295) Updating files: 14% (602/4295) Updating files: 15% (645/4295) Updating files: 16% (688/4295) Updating files: 17% (731/4295) Updating files: 18% (774/4295) Updating files: 18% (815/4295) Updating files: 19% (817/4295) Updating files: 20% (859/4295) Updating files: 21% (902/4295) Updating files: 22% (945/4295) Updating files: 23% (988/4295) Updating files: 24% (1031/4295) Updating files: 25% (1074/4295) Updating files: 26% (1117/4295) Updating files: 27% (1160/4295) Updating files: 28% (1203/4295) Updating files: 29% (1246/4295) Updating files: 29% (1288/4295) Updating files: 30% (1289/4295) Updating files: 31% (1332/4295) Updating files: 32% (1375/4295) Updating files: 33% (1418/4295) Updating files: 34% (1461/4295) Updating files: 35% (1504/4295) Updating files: 35% (1537/4295) Updating files: 36% (1547/4295) Updating files: 37% (1590/4295) Updating files: 38% (1633/4295) Updating files: 39% (1676/4295) Updating files: 40% (1718/4295) Updating files: 41% (1761/4295) Updating files: 42% (1804/4295) Updating files: 43% (1847/4295) Updating files: 44% (1890/4295) Updating files: 45% (1933/4295) Updating files: 46% (1976/4295) Updating files: 47% (2019/4295) Updating files: 48% (2062/4295) Updating files: 48% (2097/4295) Updating files: 49% (2105/4295) Updating files: 50% (2148/4295) Updating files: 51% (2191/4295) Updating files: 52% (2234/4295) Updating files: 53% (2277/4295) Updating files: 54% (2320/4295) Updating files: 55% (2363/4295) Updating files: 56% (2406/4295) Updating files: 57% (2449/4295) Updating files: 58% (2492/4295) Updating files: 58% (2494/4295) Updating files: 59% (2535/4295) Updating files: 60% (2577/4295) Updating files: 61% (2620/4295) Updating files: 62% (2663/4295) Updating files: 63% (2706/4295) Updating files: 64% (2749/4295) Updating files: 64% (2786/4295) Updating files: 65% (2792/4295) Updating files: 66% (2835/4295) Updating files: 67% (2878/4295) Updating files: 68% (2921/4295) Updating files: 69% (2964/4295) Updating files: 70% (3007/4295) Updating files: 70% (3024/4295) Updating files: 71% (3050/4295) Updating files: 72% (3093/4295) Updating files: 73% (3136/4295) Updating files: 74% (3179/4295) Updating files: 75% (3222/4295) Updating files: 76% (3265/4295) Updating files: 77% (3308/4295) Updating files: 78% (3351/4295) Updating files: 78% (3393/4295) Updating files: 79% (3394/4295) Updating files: 80% (3436/4295) Updating files: 81% (3479/4295) Updating files: 82% (3522/4295) Updating files: 83% (3565/4295) Updating files: 84% (3608/4295) Updating files: 85% (3651/4295) Updating files: 85% (3691/4295) Updating files: 86% (3694/4295) Updating files: 87% (3737/4295) Updating files: 88% (3780/4295) Updating files: 89% (3823/4295) Updating files: 90% (3866/4295) Updating files: 91% (3909/4295) Updating files: 92% (3952/4295) Updating files: 93% (3995/4295) Updating files: 94% (4038/4295) Updating files: 95% (4081/4295) Updating files: 96% (4124/4295) Updating files: 97% (4167/4295) Updating files: 98% (4210/4295) Updating files: 99% (4253/4295) Updating files: 100% (4295/4295) Updating files: 100% (4295/4295), done. 2024-05-12T22:11:01.027 INFO:tasks.workunit.client.0.smithi028.stderr:+ cd xfstests-dev 2024-05-12T22:11:01.027 INFO:tasks.workunit.client.0.smithi028.stderr:+ git checkout 12973fc04fd10d4af086901e10ffa8e48866b735 2024-05-12T22:11:02.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:11:01 smithi186 ceph-mon[34785]: pgmap v64: 97 pgs: 97 active+clean; 26 MiB data, 425 MiB used, 1.0 TiB / 1.0 TiB avail; 534 KiB/s rd, 1000 KiB/s wr, 102 op/s 2024-05-12T22:11:02.241 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:11:01 smithi086 ceph-mon[34297]: pgmap v64: 97 pgs: 97 active+clean; 26 MiB data, 425 MiB used, 1.0 TiB / 1.0 TiB avail; 534 KiB/s rd, 1000 KiB/s wr, 102 op/s 2024-05-12T22:11:02.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:11:01 smithi028 ceph-mon[29847]: pgmap v64: 97 pgs: 97 active+clean; 26 MiB data, 425 MiB used, 1.0 TiB / 1.0 TiB avail; 534 KiB/s rd, 1000 KiB/s wr, 102 op/s 2024-05-12T22:11:04.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:11:03 smithi186 ceph-mon[34785]: pgmap v65: 97 pgs: 97 active+clean; 26 MiB data, 425 MiB used, 1.0 TiB / 1.0 TiB avail; 319 KiB/s rd, 690 KiB/s wr, 95 op/s 2024-05-12T22:11:04.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:11:03 smithi086 ceph-mon[34297]: pgmap v65: 97 pgs: 97 active+clean; 26 MiB data, 425 MiB used, 1.0 TiB / 1.0 TiB avail; 319 KiB/s rd, 690 KiB/s wr, 95 op/s 2024-05-12T22:11:04.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:11:03 smithi028 ceph-mon[29847]: pgmap v65: 97 pgs: 97 active+clean; 26 MiB data, 425 MiB used, 1.0 TiB / 1.0 TiB avail; 319 KiB/s rd, 690 KiB/s wr, 95 op/s 2024-05-12T22:11:06.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:11:05 smithi186 ceph-mon[34785]: pgmap v66: 97 pgs: 97 active+clean; 42 MiB data, 490 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.3 MiB/s wr, 381 op/s 2024-05-12T22:11:06.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:11:05 smithi086 ceph-mon[34297]: pgmap v66: 97 pgs: 97 active+clean; 42 MiB data, 490 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.3 MiB/s wr, 381 op/s 2024-05-12T22:11:06.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:11:05 smithi028 ceph-mon[29847]: pgmap v66: 97 pgs: 97 active+clean; 42 MiB data, 490 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.3 MiB/s wr, 381 op/s 2024-05-12T22:11:08.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:11:07 smithi186 ceph-mon[34785]: pgmap v67: 97 pgs: 97 active+clean; 42 MiB data, 490 MiB used, 1.0 TiB / 1.0 TiB avail; 890 KiB/s rd, 2.1 MiB/s wr, 362 op/s 2024-05-12T22:11:08.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:11:07 smithi086 ceph-mon[34297]: pgmap v67: 97 pgs: 97 active+clean; 42 MiB data, 490 MiB used, 1.0 TiB / 1.0 TiB avail; 890 KiB/s rd, 2.1 MiB/s wr, 362 op/s 2024-05-12T22:11:08.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:11:07 smithi028 ceph-mon[29847]: pgmap v67: 97 pgs: 97 active+clean; 42 MiB data, 490 MiB used, 1.0 TiB / 1.0 TiB avail; 890 KiB/s rd, 2.1 MiB/s wr, 362 op/s 2024-05-12T22:11:10.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:11:09 smithi186 ceph-mon[34785]: pgmap v68: 97 pgs: 97 active+clean; 42 MiB data, 490 MiB used, 1.0 TiB / 1.0 TiB avail; 890 KiB/s rd, 2.1 MiB/s wr, 362 op/s 2024-05-12T22:11:10.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:11:09 smithi186 ceph-mon[34785]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:11:10.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:11:09 smithi086 ceph-mon[34297]: pgmap v68: 97 pgs: 97 active+clean; 42 MiB data, 490 MiB used, 1.0 TiB / 1.0 TiB avail; 890 KiB/s rd, 2.1 MiB/s wr, 362 op/s 2024-05-12T22:11:10.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:11:09 smithi086 ceph-mon[34297]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:11:10.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:11:09 smithi028 ceph-mon[29847]: pgmap v68: 97 pgs: 97 active+clean; 42 MiB data, 490 MiB used, 1.0 TiB / 1.0 TiB avail; 890 KiB/s rd, 2.1 MiB/s wr, 362 op/s 2024-05-12T22:11:10.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:11:09 smithi028 ceph-mon[29847]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:11:12.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:11:11 smithi186 ceph-mon[34785]: pgmap v69: 97 pgs: 97 active+clean; 58 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.5 MiB/s wr, 425 op/s 2024-05-12T22:11:12.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:11:11 smithi186 ceph-mon[34785]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:11:12.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:11:11 smithi186 ceph-mon[34785]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:11:12.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:11:11 smithi186 ceph-mon[34785]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:11:12.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:11:11 smithi086 ceph-mon[34297]: pgmap v69: 97 pgs: 97 active+clean; 58 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.5 MiB/s wr, 425 op/s 2024-05-12T22:11:12.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:11:11 smithi086 ceph-mon[34297]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:11:12.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:11:11 smithi086 ceph-mon[34297]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:11:12.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:11:11 smithi086 ceph-mon[34297]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:11:12.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:11:11 smithi028 ceph-mon[29847]: pgmap v69: 97 pgs: 97 active+clean; 58 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.5 MiB/s wr, 425 op/s 2024-05-12T22:11:12.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:11:11 smithi028 ceph-mon[29847]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:11:12.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:11:11 smithi028 ceph-mon[29847]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:11:12.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:11:11 smithi028 ceph-mon[29847]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:11:14.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:11:13 smithi186 ceph-mon[34785]: pgmap v70: 97 pgs: 97 active+clean; 58 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.0 MiB/s wr, 349 op/s 2024-05-12T22:11:14.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:11:13 smithi086 ceph-mon[34297]: pgmap v70: 97 pgs: 97 active+clean; 58 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.0 MiB/s wr, 349 op/s 2024-05-12T22:11:14.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:11:13 smithi028 ceph-mon[29847]: pgmap v70: 97 pgs: 97 active+clean; 58 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.0 MiB/s wr, 349 op/s 2024-05-12T22:11:16.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:11:15 smithi186 ceph-mon[34785]: pgmap v71: 97 pgs: 97 active+clean; 69 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.9 MiB/s wr, 440 op/s 2024-05-12T22:11:16.185 INFO:tasks.workunit.client.0.smithi028.stderr:Note: switching to '12973fc04fd10d4af086901e10ffa8e48866b735'. 2024-05-12T22:11:16.185 INFO:tasks.workunit.client.0.smithi028.stderr: 2024-05-12T22:11:16.185 INFO:tasks.workunit.client.0.smithi028.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-05-12T22:11:16.185 INFO:tasks.workunit.client.0.smithi028.stderr:changes and commit them, and you can discard any commits you make in this 2024-05-12T22:11:16.185 INFO:tasks.workunit.client.0.smithi028.stderr:state without impacting any branches by switching back to a branch. 2024-05-12T22:11:16.185 INFO:tasks.workunit.client.0.smithi028.stderr: 2024-05-12T22:11:16.186 INFO:tasks.workunit.client.0.smithi028.stderr:If you want to create a new branch to retain commits you create, you may 2024-05-12T22:11:16.186 INFO:tasks.workunit.client.0.smithi028.stderr:do so (now or later) by using -c with the switch command. Example: 2024-05-12T22:11:16.186 INFO:tasks.workunit.client.0.smithi028.stderr: 2024-05-12T22:11:16.186 INFO:tasks.workunit.client.0.smithi028.stderr: git switch -c 2024-05-12T22:11:16.186 INFO:tasks.workunit.client.0.smithi028.stderr: 2024-05-12T22:11:16.186 INFO:tasks.workunit.client.0.smithi028.stderr:Or undo this operation with: 2024-05-12T22:11:16.186 INFO:tasks.workunit.client.0.smithi028.stderr: 2024-05-12T22:11:16.186 INFO:tasks.workunit.client.0.smithi028.stderr: git switch - 2024-05-12T22:11:16.186 INFO:tasks.workunit.client.0.smithi028.stderr: 2024-05-12T22:11:16.186 INFO:tasks.workunit.client.0.smithi028.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-05-12T22:11:16.186 INFO:tasks.workunit.client.0.smithi028.stderr: 2024-05-12T22:11:16.186 INFO:tasks.workunit.client.0.smithi028.stderr:HEAD is now at 12973fc0 btrfs/022: Correct seq number in golden output 2024-05-12T22:11:16.199 INFO:tasks.workunit.client.0.smithi028.stderr:+ make -j4 2024-05-12T22:11:16.226 INFO:tasks.workunit.client.0.smithi028.stdout:libtoolize -cfi 2024-05-12T22:11:16.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:11:15 smithi086 ceph-mon[34297]: pgmap v71: 97 pgs: 97 active+clean; 69 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.9 MiB/s wr, 440 op/s 2024-05-12T22:11:16.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:11:15 smithi028 ceph-mon[29847]: pgmap v71: 97 pgs: 97 active+clean; 69 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.9 MiB/s wr, 440 op/s 2024-05-12T22:11:16.989 INFO:tasks.workunit.client.0.smithi028.stdout:libtoolize: putting auxiliary files in '.'. 2024-05-12T22:11:16.989 INFO:tasks.workunit.client.0.smithi028.stdout:libtoolize: copying file './config.guess' 2024-05-12T22:11:17.006 INFO:tasks.workunit.client.0.smithi028.stdout:libtoolize: copying file './config.sub' 2024-05-12T22:11:17.022 INFO:tasks.workunit.client.0.smithi028.stdout:libtoolize: copying file './install-sh' 2024-05-12T22:11:17.040 INFO:tasks.workunit.client.0.smithi028.stdout:libtoolize: copying file './ltmain.sh' 2024-05-12T22:11:17.131 INFO:tasks.workunit.client.0.smithi028.stdout:libtoolize: putting macros in AC_CONFIG_MACRO_DIRS, 'm4'. 2024-05-12T22:11:17.131 INFO:tasks.workunit.client.0.smithi028.stdout:libtoolize: copying file 'm4/libtool.m4' 2024-05-12T22:11:17.150 INFO:tasks.workunit.client.0.smithi028.stdout:libtoolize: copying file 'm4/ltoptions.m4' 2024-05-12T22:11:17.170 INFO:tasks.workunit.client.0.smithi028.stdout:libtoolize: copying file 'm4/ltsugar.m4' 2024-05-12T22:11:17.190 INFO:tasks.workunit.client.0.smithi028.stdout:libtoolize: copying file 'm4/ltversion.m4' 2024-05-12T22:11:17.210 INFO:tasks.workunit.client.0.smithi028.stdout:libtoolize: copying file 'm4/lt~obsolete.m4' 2024-05-12T22:11:17.234 INFO:tasks.workunit.client.0.smithi028.stdout:libtoolize: Consider adding '-I m4' to ACLOCAL_AMFLAGS in Makefile.am. 2024-05-12T22:11:17.238 INFO:tasks.workunit.client.0.smithi028.stdout:cp include/install-sh . 2024-05-12T22:11:17.268 INFO:tasks.workunit.client.0.smithi028.stdout:aclocal -I m4 2024-05-12T22:11:18.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:11:17 smithi086 ceph-mon[34297]: pgmap v72: 97 pgs: 97 active+clean; 69 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.3 MiB/s wr, 154 op/s 2024-05-12T22:11:18.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:11:17 smithi028 ceph-mon[29847]: pgmap v72: 97 pgs: 97 active+clean; 69 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.3 MiB/s wr, 154 op/s 2024-05-12T22:11:18.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:11:17 smithi186 ceph-mon[34785]: pgmap v72: 97 pgs: 97 active+clean; 69 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.3 MiB/s wr, 154 op/s 2024-05-12T22:11:18.477 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-12T22:11:18.477 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-12T22:11:18.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.683+0000 7f5929600640 1 -- 172.21.15.28:0/3166107385 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f592410bcb0 msgr2=0x7f592410e150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:11:18.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.683+0000 7f5929600640 1 --2- 172.21.15.28:0/3166107385 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f592410bcb0 0x7f592410e150 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f591c009900 tx=0x7f591c02a480 comp rx=0 tx=0).stop 2024-05-12T22:11:18.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.683+0000 7f5929600640 1 -- 172.21.15.28:0/3166107385 shutdown_connections 2024-05-12T22:11:18.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.683+0000 7f5929600640 1 --2- 172.21.15.28:0/3166107385 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f592410bcb0 0x7f592410e150 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:18.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.683+0000 7f5929600640 1 --2- 172.21.15.28:0/3166107385 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f59241031b0 0x7f592410b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:18.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.683+0000 7f5929600640 1 --2- 172.21.15.28:0/3166107385 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5924102890 0x7f5924102c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:18.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.683+0000 7f5929600640 1 -- 172.21.15.28:0/3166107385 >> 172.21.15.28:0/3166107385 conn(0x7f59240fc0b0 msgr2=0x7f59240fe4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:11:18.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.683+0000 7f5929600640 1 -- 172.21.15.28:0/3166107385 shutdown_connections 2024-05-12T22:11:18.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.683+0000 7f5929600640 1 -- 172.21.15.28:0/3166107385 wait complete. 2024-05-12T22:11:18.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.684+0000 7f5929600640 1 Processor -- start 2024-05-12T22:11:18.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.684+0000 7f5929600640 1 -- start start 2024-05-12T22:11:18.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.685+0000 7f5929600640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5924102890 0x7f59241004e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:18.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.685+0000 7f5929600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59241031b0 0x7f5924100a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:18.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.685+0000 7f5929600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f592410bcb0 0x7f5924100f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:18.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.685+0000 7f5929600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5924111b00 con 0x7f59241031b0 2024-05-12T22:11:18.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.685+0000 7f5929600640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f5924111980 con 0x7f592410bcb0 2024-05-12T22:11:18.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.685+0000 7f5929600640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f5924111c80 con 0x7f5924102890 2024-05-12T22:11:18.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.685+0000 7f5923e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5924102890 0x7f59241004e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:18.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.685+0000 7f5923e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5924102890 0x7f59241004e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:58984/0 (socket says 172.21.15.28:58984) 2024-05-12T22:11:18.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.685+0000 7f5923e00640 1 -- 172.21.15.28:0/3178669112 learned_addr learned my addr 172.21.15.28:0/3178669112 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:11:18.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.685+0000 7f5923400640 1 --2- 172.21.15.28:0/3178669112 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59241031b0 0x7f5924100a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:18.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.685+0000 7f5928c00640 1 --2- 172.21.15.28:0/3178669112 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f592410bcb0 0x7f5924100f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:18.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.686+0000 7f5923e00640 1 -- 172.21.15.28:0/3178669112 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f592410bcb0 msgr2=0x7f5924100f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:11:18.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.686+0000 7f5923e00640 1 --2- 172.21.15.28:0/3178669112 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f592410bcb0 0x7f5924100f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:18.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.686+0000 7f5923e00640 1 -- 172.21.15.28:0/3178669112 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59241031b0 msgr2=0x7f5924100a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:11:18.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.686+0000 7f5923e00640 1 --2- 172.21.15.28:0/3178669112 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59241031b0 0x7f5924100a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:18.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.686+0000 7f5923e00640 1 -- 172.21.15.28:0/3178669112 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59241a6440 con 0x7f5924102890 2024-05-12T22:11:18.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.686+0000 7f5923e00640 1 --2- 172.21.15.28:0/3178669112 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5924102890 0x7f59241004e0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f5914009d50 tx=0x7f591400c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:11:18.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.686+0000 7f5920c00640 1 -- 172.21.15.28:0/3178669112 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5914003b70 con 0x7f5924102890 2024-05-12T22:11:18.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.686+0000 7f5929600640 1 -- 172.21.15.28:0/3178669112 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59241a6730 con 0x7f5924102890 2024-05-12T22:11:18.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.686+0000 7f5920c00640 1 -- 172.21.15.28:0/3178669112 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5914003d10 con 0x7f5924102890 2024-05-12T22:11:18.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.686+0000 7f5920c00640 1 -- 172.21.15.28:0/3178669112 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5914010a20 con 0x7f5924102890 2024-05-12T22:11:18.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.686+0000 7f5929600640 1 -- 172.21.15.28:0/3178669112 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59241a6cd0 con 0x7f5924102890 2024-05-12T22:11:18.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.689+0000 7f592dbd1740 1 Processor -- start 2024-05-12T22:11:18.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.690+0000 7f592dbd1740 1 -- start start 2024-05-12T22:11:18.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.690+0000 7f5920c00640 1 -- 172.21.15.28:0/3178669112 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7f5914003ec0 con 0x7f5924102890 2024-05-12T22:11:18.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.690+0000 7f592dbd1740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561fe8933dd0 0x561fe89341b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:18.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.690+0000 7f592dbd1740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561fe89346f0 0x561fe893e880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:18.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.690+0000 7f592dbd1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561fe893edc0 0x561fe89411b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:18.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.690+0000 7f592dbd1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x561fe88fc2a0 con 0x561fe893edc0 2024-05-12T22:11:18.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.690+0000 7f592dbd1740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x561fe8898b90 con 0x561fe8933dd0 2024-05-12T22:11:18.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.690+0000 7f592dbd1740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x561fe88405b0 con 0x561fe89346f0 2024-05-12T22:11:18.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.690+0000 7f5928c00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561fe8933dd0 0x561fe89341b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:18.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.690+0000 7f5928c00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561fe8933dd0 0x561fe89341b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:41576/0 (socket says 172.21.15.28:41576) 2024-05-12T22:11:18.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.690+0000 7f5928c00640 1 -- 172.21.15.28:0/4021404137 learned_addr learned my addr 172.21.15.28:0/4021404137 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:11:18.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.690+0000 7f5923e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561fe89346f0 0x561fe893e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:18.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.691+0000 7f5923400640 1 --2- 172.21.15.28:0/4021404137 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561fe893edc0 0x561fe89411b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:18.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.691+0000 7f5928c00640 1 -- 172.21.15.28:0/4021404137 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561fe89346f0 msgr2=0x561fe893e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:11:18.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.691+0000 7f5928c00640 1 --2- 172.21.15.28:0/4021404137 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561fe89346f0 0x561fe893e880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:18.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.691+0000 7f5928c00640 1 -- 172.21.15.28:0/4021404137 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561fe893edc0 msgr2=0x561fe89411b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:11:18.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.691+0000 7f5928c00640 1 --2- 172.21.15.28:0/4021404137 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561fe893edc0 0x561fe89411b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:18.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.691+0000 7f5928c00640 1 -- 172.21.15.28:0/4021404137 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561fe86767d0 con 0x561fe8933dd0 2024-05-12T22:11:18.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.691+0000 7f5928c00640 1 --2- 172.21.15.28:0/4021404137 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561fe8933dd0 0x561fe89341b0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f591c02aa70 tx=0x7f591c005c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fd9cd65711089213 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:11:18.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.692+0000 7f5920c00640 1 --2- 172.21.15.28:0/3178669112 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f59080a2db0 0x7f59080a5270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:18.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.692+0000 7f5919200640 1 -- 172.21.15.28:0/4021404137 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f591c040070 con 0x561fe8933dd0 2024-05-12T22:11:18.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.692+0000 7f5919200640 1 -- 172.21.15.28:0/4021404137 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f591c004300 con 0x561fe8933dd0 2024-05-12T22:11:18.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.692+0000 7f5919200640 1 -- 172.21.15.28:0/4021404137 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f591c035040 con 0x561fe8933dd0 2024-05-12T22:11:18.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.692+0000 7f5920c00640 1 -- 172.21.15.28:0/3178669112 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f59140cfa70 con 0x7f5924102890 2024-05-12T22:11:18.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.692+0000 7f5923e00640 1 --2- 172.21.15.28:0/3178669112 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f59080a2db0 0x7f59080a5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:18.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.692+0000 7f592dbd1740 1 -- 172.21.15.28:0/4021404137 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561fe8933dd0 msgr2=0x561fe89341b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:11:18.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.692+0000 7f592dbd1740 1 --2- 172.21.15.28:0/4021404137 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561fe8933dd0 0x561fe89341b0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f591c02aa70 tx=0x7f591c005c00 comp rx=0 tx=0).stop 2024-05-12T22:11:18.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.693+0000 7f592dbd1740 1 -- 172.21.15.28:0/4021404137 shutdown_connections 2024-05-12T22:11:18.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.693+0000 7f592dbd1740 1 --2- 172.21.15.28:0/4021404137 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561fe893edc0 0x561fe89411b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:18.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.693+0000 7f592dbd1740 1 --2- 172.21.15.28:0/4021404137 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561fe89346f0 0x561fe893e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:18.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.693+0000 7f592dbd1740 1 --2- 172.21.15.28:0/4021404137 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561fe8933dd0 0x561fe89341b0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:18.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.693+0000 7f592dbd1740 1 -- 172.21.15.28:0/4021404137 >> 172.21.15.28:0/4021404137 conn(0x561fe89300f0 msgr2=0x561fe89304d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:11:18.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.693+0000 7f592dbd1740 1 -- 172.21.15.28:0/4021404137 shutdown_connections 2024-05-12T22:11:18.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.693+0000 7f592dbd1740 1 -- 172.21.15.28:0/4021404137 wait complete. 2024-05-12T22:11:18.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.693+0000 7f5923e00640 1 --2- 172.21.15.28:0/3178669112 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f59080a2db0 0x7f59080a5270 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x561fe8935390 tx=0x7f59140d34a0 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:11:18.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.693+0000 7f592dbd1740 1 -- start start 2024-05-12T22:11:18.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.693+0000 7f592dbd1740 1 Processor -- start 2024-05-12T22:11:18.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.694+0000 7f592dbd1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561fe8933dd0 0x561fe893e710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:18.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.694+0000 7f592dbd1740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561fe89346f0 0x561fe893ba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:18.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.694+0000 7f592dbd1740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561fe893edc0 0x561fe893bf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:18.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.694+0000 7f592dbd1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x561fe88539f0 con 0x561fe8933dd0 2024-05-12T22:11:18.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.694+0000 7f592dbd1740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x561fe8894b50 con 0x561fe893edc0 2024-05-12T22:11:18.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.694+0000 7f592dbd1740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x561fe8941b50 con 0x561fe89346f0 2024-05-12T22:11:18.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.694+0000 7f592dbd1740 10 client.0 _ll_register_callbacks cb 0x561fe85abbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:11:18.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.694+0000 7f5928c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561fe8933dd0 0x561fe893e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:18.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.694+0000 7f5928c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561fe8933dd0 0x561fe893e710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60784/0 (socket says 172.21.15.28:60784) 2024-05-12T22:11:18.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.694+0000 7f5928c00640 1 -- 172.21.15.28:0/2749304577 learned_addr learned my addr 172.21.15.28:0/2749304577 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:11:18.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.694+0000 7f5923400640 1 --2- 172.21.15.28:0/2749304577 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561fe89346f0 0x561fe893ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:18.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.695+0000 7f5928c00640 1 --2- 172.21.15.28:0/2749304577 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561fe893edc0 0x561fe893bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:18.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.695+0000 7f5923400640 1 -- 172.21.15.28:0/2749304577 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561fe893edc0 msgr2=0x561fe893bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:11:18.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.695+0000 7f5923400640 1 --2- 172.21.15.28:0/2749304577 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561fe893edc0 0x561fe893bf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:18.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.695+0000 7f5923400640 1 -- 172.21.15.28:0/2749304577 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561fe8933dd0 msgr2=0x561fe893e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:11:18.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.695+0000 7f5923400640 1 --2- 172.21.15.28:0/2749304577 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561fe8933dd0 0x561fe893e710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:18.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.695+0000 7f5923400640 1 -- 172.21.15.28:0/2749304577 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561fe85ac3f0 con 0x561fe89346f0 2024-05-12T22:11:18.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.695+0000 7f5928c00640 1 --2- 172.21.15.28:0/2749304577 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561fe893edc0 0x561fe893bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:11:18.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.695+0000 7f5923400640 1 --2- 172.21.15.28:0/2749304577 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561fe89346f0 0x561fe893ba00 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f5924101f20 tx=0x7f590c012b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=be061cc88e81fd36 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:11:18.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.695+0000 7f5911600640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-12T22:11:18.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.695+0000 7f5911600640 1 -- 172.21.15.28:0/2749304577 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f590c004050 con 0x561fe89346f0 2024-05-12T22:11:18.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.696+0000 7f5911600640 1 -- 172.21.15.28:0/2749304577 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f590c0041f0 con 0x561fe89346f0 2024-05-12T22:11:18.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.696+0000 7f592dbd1740 1 -- 172.21.15.28:0/2749304577 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f591c004300 con 0x561fe89346f0 2024-05-12T22:11:18.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.696+0000 7f5911600640 1 -- 172.21.15.28:0/2749304577 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f590c004430 con 0x561fe89346f0 2024-05-12T22:11:18.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.696+0000 7f592dbd1740 10 client.25759 fetch_fsmap learned FSMap version 25 2024-05-12T22:11:18.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.696+0000 7f592dbd1740 1 -- 172.21.15.28:0/2749304577 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x561fe856c0f0 con 0x561fe89346f0 2024-05-12T22:11:18.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.696+0000 7f5911600640 1 -- 172.21.15.28:0/2749304577 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f591c004300 con 0x561fe89346f0 2024-05-12T22:11:18.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.696+0000 7f5911600640 1 -- 172.21.15.28:0/2749304577 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7f590c003bf0 con 0x561fe89346f0 2024-05-12T22:11:18.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.696+0000 7f592dbd1740 10 client.25759 fetch_fsmap finished waiting for FSMap version 25 2024-05-12T22:11:18.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.696+0000 7f592dbd1740 10 client.25759 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:11:18.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.697+0000 7f592dbd1740 20 client.25759 populate_metadata read hostname 'smithi028' 2024-05-12T22:11:18.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.697+0000 7f592dbd1740 1 --2- 172.21.15.28:0/2749304577 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x561fe894e9d0 0x561fe8950e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:18.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.697+0000 7f592dbd1740 4 client.25759 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-12T22:11:18.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.697+0000 7f592dbd1740 1 -- 172.21.15.28:0/2749304577 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561fe88a62c0 con 0x561fe894e9d0 2024-05-12T22:11:18.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.697+0000 7f5928c00640 1 --2- 172.21.15.28:0/2749304577 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x561fe894e9d0 0x561fe8950e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:18.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.697+0000 7f5928c00640 1 --2- 172.21.15.28:0/2749304577 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x561fe894e9d0 0x561fe8950e10 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dac88e5b5ed569db server_cookie=eeb307829c9bba4 in_seq=0 out_seq=0 2024-05-12T22:11:18.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.697+0000 7f5911600640 10 client.25759 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:11:18.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.698+0000 7f5911600640 1 -- 172.21.15.28:0/2749304577 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561fe88a62c0 con 0x561fe894e9d0 2024-05-12T22:11:18.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.698+0000 7f5911600640 10 client.25759 handle_command_reply: tid=0 2024-05-12T22:11:18.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.698+0000 7f592dbd1740 1 client.25759 shutdown 2024-05-12T22:11:18.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.698+0000 7f592dbd1740 1 -- 172.21.15.28:0/2749304577 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561fe89346f0 msgr2=0x561fe893ba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:11:18.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.698+0000 7f592dbd1740 1 --2- 172.21.15.28:0/2749304577 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561fe89346f0 0x561fe893ba00 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f5924101f20 tx=0x7f590c012b60 comp rx=0 tx=0).stop 2024-05-12T22:11:18.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.699+0000 7f592dbd1740 1 -- 172.21.15.28:0/2749304577 shutdown_connections 2024-05-12T22:11:18.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.699+0000 7f592dbd1740 1 --2- 172.21.15.28:0/2749304577 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x561fe894e9d0 0x561fe8950e10 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:18.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.699+0000 7f592dbd1740 1 --2- 172.21.15.28:0/2749304577 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561fe893edc0 0x561fe893bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:18.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.699+0000 7f592dbd1740 1 --2- 172.21.15.28:0/2749304577 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561fe89346f0 0x561fe893ba00 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:18.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.699+0000 7f592dbd1740 1 --2- 172.21.15.28:0/2749304577 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561fe8933dd0 0x561fe893e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:18.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.699+0000 7f5911600640 0 client.25759 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:11:18.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.699+0000 7f592dbd1740 1 -- 172.21.15.28:0/2749304577 >> 172.21.15.28:0/2749304577 conn(0x561fe89300f0 msgr2=0x561fe8935ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:11:18.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.699+0000 7f592dbd1740 1 -- 172.21.15.28:0/2749304577 shutdown_connections 2024-05-12T22:11:18.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.699+0000 7f592dbd1740 1 -- 172.21.15.28:0/2749304577 wait complete. 2024-05-12T22:11:18.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.699+0000 7f592dbd1740 20 client.25759 trim_cache size 0 max 16384 2024-05-12T22:11:18.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.734+0000 7f592dbd1740 1 Processor -- start 2024-05-12T22:11:18.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.734+0000 7f592dbd1740 1 -- start start 2024-05-12T22:11:18.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.734+0000 7f592dbd1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561fe8933dd0 0x561fe893e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:18.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.734+0000 7f592dbd1740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561fe89346f0 0x561fe894a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:18.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.734+0000 7f592dbd1740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561fe893edc0 0x561fe89566e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:18.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.734+0000 7f592dbd1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f59241b1dc0 con 0x561fe8933dd0 2024-05-12T22:11:18.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.734+0000 7f592dbd1740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f59241af690 con 0x561fe893edc0 2024-05-12T22:11:18.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.734+0000 7f592dbd1740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f59241ad4e0 con 0x561fe89346f0 2024-05-12T22:11:18.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.735+0000 7f5928c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561fe8933dd0 0x561fe893e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:18.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.735+0000 7f5928c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561fe8933dd0 0x561fe893e240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60790/0 (socket says 172.21.15.28:60790) 2024-05-12T22:11:18.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.735+0000 7f5928c00640 1 -- 172.21.15.28:0/3112881867 learned_addr learned my addr 172.21.15.28:0/3112881867 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:11:18.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.735+0000 7f5923400640 1 --2- 172.21.15.28:0/3112881867 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561fe89346f0 0x561fe894a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:18.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.735+0000 7f5928c00640 1 --2- 172.21.15.28:0/3112881867 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561fe893edc0 0x561fe89566e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:18.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.735+0000 7f5928c00640 1 -- 172.21.15.28:0/3112881867 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561fe89346f0 msgr2=0x561fe894a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:11:18.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.735+0000 7f5928c00640 1 --2- 172.21.15.28:0/3112881867 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561fe89346f0 0x561fe894a8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:18.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.735+0000 7f5928c00640 1 -- 172.21.15.28:0/3112881867 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561fe893edc0 msgr2=0x561fe89566e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:11:18.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.735+0000 7f5928c00640 1 --2- 172.21.15.28:0/3112881867 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561fe893edc0 0x561fe89566e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:18.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.735+0000 7f5928c00640 1 -- 172.21.15.28:0/3112881867 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561fe8934520 con 0x561fe8933dd0 2024-05-12T22:11:18.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.735+0000 7f5923400640 1 --2- 172.21.15.28:0/3112881867 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561fe89346f0 0x561fe894a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:11:18.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.735+0000 7f5928c00640 1 --2- 172.21.15.28:0/3112881867 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561fe8933dd0 0x561fe893e240 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f591c0096c0 tx=0x7f591c0314b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44503b15cd6e6162 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:11:18.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.736+0000 7f5913e00640 1 -- 172.21.15.28:0/3112881867 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f591c035040 con 0x561fe8933dd0 2024-05-12T22:11:18.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.736+0000 7f5913e00640 1 -- 172.21.15.28:0/3112881867 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f591c04a070 con 0x561fe8933dd0 2024-05-12T22:11:18.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.736+0000 7f5913e00640 1 -- 172.21.15.28:0/3112881867 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f591c040040 con 0x561fe8933dd0 2024-05-12T22:11:18.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.736+0000 7f592dbd1740 1 -- 172.21.15.28:0/3112881867 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561fe8933dd0 msgr2=0x561fe893e240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:11:18.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.736+0000 7f592dbd1740 1 --2- 172.21.15.28:0/3112881867 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561fe8933dd0 0x561fe893e240 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f591c0096c0 tx=0x7f591c0314b0 comp rx=0 tx=0).stop 2024-05-12T22:11:18.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.737+0000 7f592dbd1740 1 -- 172.21.15.28:0/3112881867 shutdown_connections 2024-05-12T22:11:18.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.737+0000 7f592dbd1740 1 --2- 172.21.15.28:0/3112881867 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561fe893edc0 0x561fe89566e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:18.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.737+0000 7f592dbd1740 1 --2- 172.21.15.28:0/3112881867 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561fe89346f0 0x561fe894a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:18.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.737+0000 7f592dbd1740 1 --2- 172.21.15.28:0/3112881867 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561fe8933dd0 0x561fe893e240 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:18.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.737+0000 7f592dbd1740 1 -- 172.21.15.28:0/3112881867 >> 172.21.15.28:0/3112881867 conn(0x561fe89300f0 msgr2=0x561fe892f330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:11:18.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.737+0000 7f592dbd1740 1 -- 172.21.15.28:0/3112881867 shutdown_connections 2024-05-12T22:11:18.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.737+0000 7f592dbd1740 1 -- 172.21.15.28:0/3112881867 wait complete. 2024-05-12T22:11:18.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.737+0000 7f592dbd1740 1 -- start start 2024-05-12T22:11:18.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.738+0000 7f592dbd1740 1 Processor -- start 2024-05-12T22:11:18.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.738+0000 7f592dbd1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561fe8933dd0 0x561fe89427d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:18.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.738+0000 7f592dbd1740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561fe89346f0 0x561fe8942d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:18.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.738+0000 7f592dbd1740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561fe893edc0 0x561fe8943250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:18.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.738+0000 7f592dbd1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f59241adb50 con 0x561fe8933dd0 2024-05-12T22:11:18.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.738+0000 7f592dbd1740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f59241aefc0 con 0x561fe893edc0 2024-05-12T22:11:18.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.738+0000 7f592dbd1740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f59241ae220 con 0x561fe89346f0 2024-05-12T22:11:18.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.738+0000 7f592dbd1740 10 client.0 _ll_register_callbacks cb 0x7f590c009570 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:11:18.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.739+0000 7f5928c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561fe8933dd0 0x561fe89427d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:18.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.739+0000 7f5928c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561fe8933dd0 0x561fe89427d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60800/0 (socket says 172.21.15.28:60800) 2024-05-12T22:11:18.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.739+0000 7f5928c00640 1 -- 172.21.15.28:0/1054708818 learned_addr learned my addr 172.21.15.28:0/1054708818 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:11:18.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.739+0000 7f5923400640 1 --2- 172.21.15.28:0/1054708818 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561fe89346f0 0x561fe8942d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:18.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.739+0000 7f5928c00640 1 --2- 172.21.15.28:0/1054708818 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561fe893edc0 0x561fe8943250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:18.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.739+0000 7f5923400640 1 -- 172.21.15.28:0/1054708818 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561fe893edc0 msgr2=0x561fe8943250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:11:18.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.739+0000 7f5923400640 1 --2- 172.21.15.28:0/1054708818 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561fe893edc0 0x561fe8943250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:18.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.739+0000 7f5923400640 1 -- 172.21.15.28:0/1054708818 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561fe8933dd0 msgr2=0x561fe89427d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:11:18.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.739+0000 7f5923400640 1 --2- 172.21.15.28:0/1054708818 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561fe8933dd0 0x561fe89427d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:18.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.739+0000 7f5923400640 1 -- 172.21.15.28:0/1054708818 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561fe89b3f50 con 0x561fe89346f0 2024-05-12T22:11:18.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.739+0000 7f5928c00640 1 --2- 172.21.15.28:0/1054708818 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561fe8933dd0 0x561fe89427d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:11:18.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.740+0000 7f5923400640 1 --2- 172.21.15.28:0/1054708818 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561fe89346f0 0x561fe8942d10 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f590c010670 tx=0x7f590c0093e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f438b44d9cab938c server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:11:18.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.740+0000 7f5911600640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-12T22:11:18.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.740+0000 7f5911600640 1 -- 172.21.15.28:0/1054708818 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f590c029070 con 0x561fe89346f0 2024-05-12T22:11:18.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.740+0000 7f5911600640 1 -- 172.21.15.28:0/1054708818 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f590c010a80 con 0x561fe89346f0 2024-05-12T22:11:18.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.740+0000 7f592dbd1740 1 -- 172.21.15.28:0/1054708818 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f591c04a070 con 0x561fe89346f0 2024-05-12T22:11:18.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.740+0000 7f5911600640 1 -- 172.21.15.28:0/1054708818 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f590c010ca0 con 0x561fe89346f0 2024-05-12T22:11:18.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.740+0000 7f592dbd1740 10 client.25765 fetch_fsmap learned FSMap version 25 2024-05-12T22:11:18.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.740+0000 7f592dbd1740 1 -- 172.21.15.28:0/1054708818 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x561fe89b5230 con 0x561fe89346f0 2024-05-12T22:11:18.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.741+0000 7f5911600640 1 -- 172.21.15.28:0/1054708818 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f591c04a070 con 0x561fe89346f0 2024-05-12T22:11:18.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.741+0000 7f5911600640 1 -- 172.21.15.28:0/1054708818 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7f590c02e820 con 0x561fe89346f0 2024-05-12T22:11:18.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.741+0000 7f592dbd1740 10 client.25765 fetch_fsmap finished waiting for FSMap version 25 2024-05-12T22:11:18.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.741+0000 7f592dbd1740 10 client.25765 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:11:18.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.741+0000 7f592dbd1740 20 client.25765 populate_metadata read hostname 'smithi028' 2024-05-12T22:11:18.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.741+0000 7f592dbd1740 1 --2- 172.21.15.28:0/1054708818 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x561fe89b5a00 0x561fe89b7e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:18.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.741+0000 7f592dbd1740 4 client.25765 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-12T22:11:18.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.741+0000 7f592dbd1740 1 -- 172.21.15.28:0/1054708818 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "scrub status"}) -- 0x561fe88a0610 con 0x561fe89b5a00 2024-05-12T22:11:18.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.742+0000 7f5928c00640 1 --2- 172.21.15.28:0/1054708818 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x561fe89b5a00 0x561fe89b7e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:18.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.742+0000 7f5928c00640 1 --2- 172.21.15.28:0/1054708818 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x561fe89b5a00 0x561fe89b7e20 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8914195f68cd8c42 server_cookie=8b5658def67227ee in_seq=0 out_seq=0 2024-05-12T22:11:18.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.742+0000 7f5911600640 10 client.25765 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:11:18.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.742+0000 7f5911600640 1 -- 172.21.15.28:0/1054708818 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x561fe88a0610 con 0x561fe89b5a00 2024-05-12T22:11:18.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.742+0000 7f5911600640 10 client.25765 handle_command_reply: tid=0 2024-05-12T22:11:18.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.742+0000 7f592dbd1740 1 client.25765 shutdown 2024-05-12T22:11:18.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.743+0000 7f592dbd1740 1 -- 172.21.15.28:0/1054708818 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561fe89346f0 msgr2=0x561fe8942d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:11:18.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.743+0000 7f592dbd1740 1 --2- 172.21.15.28:0/1054708818 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561fe89346f0 0x561fe8942d10 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f590c010670 tx=0x7f590c0093e0 comp rx=0 tx=0).stop 2024-05-12T22:11:18.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.743+0000 7f592dbd1740 1 -- 172.21.15.28:0/1054708818 shutdown_connections 2024-05-12T22:11:18.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.743+0000 7f592dbd1740 1 --2- 172.21.15.28:0/1054708818 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x561fe89b5a00 0x561fe89b7e20 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:18.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.743+0000 7f592dbd1740 1 --2- 172.21.15.28:0/1054708818 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561fe893edc0 0x561fe8943250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:18.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.743+0000 7f592dbd1740 1 --2- 172.21.15.28:0/1054708818 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561fe89346f0 0x561fe8942d10 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:18.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.743+0000 7f592dbd1740 1 --2- 172.21.15.28:0/1054708818 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561fe8933dd0 0x561fe89427d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:18.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.743+0000 7f5911600640 0 client.25765 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:11:18.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.743+0000 7f592dbd1740 1 -- 172.21.15.28:0/1054708818 >> 172.21.15.28:0/1054708818 conn(0x561fe89300f0 msgr2=0x561fe893d230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:11:18.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.743+0000 7f592dbd1740 1 -- 172.21.15.28:0/1054708818 shutdown_connections 2024-05-12T22:11:18.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.743+0000 7f592dbd1740 1 -- 172.21.15.28:0/1054708818 wait complete. 2024-05-12T22:11:18.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.743+0000 7f592dbd1740 20 client.25765 trim_cache size 0 max 16384 2024-05-12T22:11:18.747 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-05-12T22:11:18.747 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-05-12T22:11:18.747 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-05-12T22:11:18.747 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-05-12T22:11:18.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.744+0000 7f5929600640 1 -- 172.21.15.28:0/3178669112 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f59080a2db0 msgr2=0x7f59080a5270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:11:18.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.744+0000 7f5929600640 1 --2- 172.21.15.28:0/3178669112 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f59080a2db0 0x7f59080a5270 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x561fe8935390 tx=0x7f59140d34a0 comp rx=0 tx=0).stop 2024-05-12T22:11:18.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.744+0000 7f5929600640 1 -- 172.21.15.28:0/3178669112 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5924102890 msgr2=0x7f59241004e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:11:18.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.744+0000 7f5929600640 1 --2- 172.21.15.28:0/3178669112 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5924102890 0x7f59241004e0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f5914009d50 tx=0x7f591400c6a0 comp rx=0 tx=0).stop 2024-05-12T22:11:18.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.744+0000 7f5929600640 1 -- 172.21.15.28:0/3178669112 shutdown_connections 2024-05-12T22:11:18.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.744+0000 7f5929600640 1 --2- 172.21.15.28:0/3178669112 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f59080a2db0 0x7f59080a5270 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:18.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.744+0000 7f5929600640 1 --2- 172.21.15.28:0/3178669112 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f592410bcb0 0x7f5924100f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:18.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.744+0000 7f5929600640 1 --2- 172.21.15.28:0/3178669112 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59241031b0 0x7f5924100a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:18.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.744+0000 7f5929600640 1 --2- 172.21.15.28:0/3178669112 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5924102890 0x7f59241004e0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:18.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.744+0000 7f5929600640 1 -- 172.21.15.28:0/3178669112 >> 172.21.15.28:0/3178669112 conn(0x7f59240fc0b0 msgr2=0x7f59240fd7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:11:18.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.744+0000 7f5929600640 1 -- 172.21.15.28:0/3178669112 shutdown_connections 2024-05-12T22:11:18.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.744+0000 7f5929600640 1 -- 172.21.15.28:0/3178669112 wait complete. 2024-05-12T22:11:18.763 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-12T22:11:18.763 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-12T22:11:18.863 INFO:tasks.workunit.client.0.smithi028.stdout:autoheader 2024-05-12T22:11:18.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.963+0000 7f3678a00640 1 -- 172.21.15.28:0/3342106070 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f367410e0c0 msgr2=0x7f36741104b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:11:18.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.963+0000 7f3678a00640 1 --2- 172.21.15.28:0/3342106070 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f367410e0c0 0x7f36741104b0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f366c009ca0 tx=0x7f366c02a950 comp rx=0 tx=0).stop 2024-05-12T22:11:18.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.963+0000 7f3678a00640 1 -- 172.21.15.28:0/3342106070 shutdown_connections 2024-05-12T22:11:18.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.963+0000 7f3678a00640 1 --2- 172.21.15.28:0/3342106070 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f367410e0c0 0x7f36741104b0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:18.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.963+0000 7f3678a00640 1 --2- 172.21.15.28:0/3342106070 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3674074500 0x7f367410db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:18.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.963+0000 7f3678a00640 1 --2- 172.21.15.28:0/3342106070 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3674073be0 0x7f3674073fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:18.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.963+0000 7f3678a00640 1 -- 172.21.15.28:0/3342106070 >> 172.21.15.28:0/3342106070 conn(0x7f3674078bf0 msgr2=0x7f3674079000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:11:18.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.963+0000 7f3678a00640 1 -- 172.21.15.28:0/3342106070 shutdown_connections 2024-05-12T22:11:18.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.963+0000 7f3678a00640 1 -- 172.21.15.28:0/3342106070 wait complete. 2024-05-12T22:11:18.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.964+0000 7f3678a00640 1 Processor -- start 2024-05-12T22:11:18.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.964+0000 7f3678a00640 1 -- start start 2024-05-12T22:11:18.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.964+0000 7f3678a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3674073be0 0x7f367419a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:18.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.964+0000 7f3678a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3674074500 0x7f367419a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:18.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.964+0000 7f3678a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f367410e0c0 0x7f367419eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:18.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.964+0000 7f3678a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3674113e30 con 0x7f367410e0c0 2024-05-12T22:11:18.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.964+0000 7f3678a00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f3674113cb0 con 0x7f3674073be0 2024-05-12T22:11:18.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.964+0000 7f3678a00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f3674113fb0 con 0x7f3674074500 2024-05-12T22:11:18.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.965+0000 7f3671e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3674073be0 0x7f367419a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:18.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.965+0000 7f3672800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f367410e0c0 0x7f367419eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:18.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.965+0000 7f3671e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3674073be0 0x7f367419a2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:41606/0 (socket says 172.21.15.28:41606) 2024-05-12T22:11:18.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.965+0000 7f3671e00640 1 -- 172.21.15.28:0/1691108371 learned_addr learned my addr 172.21.15.28:0/1691108371 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:11:18.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.965+0000 7f3671400640 1 --2- 172.21.15.28:0/1691108371 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3674074500 0x7f367419a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:18.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.965+0000 7f3671e00640 1 -- 172.21.15.28:0/1691108371 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3674074500 msgr2=0x7f367419a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:11:18.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.965+0000 7f3671e00640 1 --2- 172.21.15.28:0/1691108371 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3674074500 0x7f367419a830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:18.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.965+0000 7f3671e00640 1 -- 172.21.15.28:0/1691108371 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f367410e0c0 msgr2=0x7f367419eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:11:18.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.965+0000 7f3671e00640 1 --2- 172.21.15.28:0/1691108371 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f367410e0c0 0x7f367419eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:18.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.965+0000 7f3671e00640 1 -- 172.21.15.28:0/1691108371 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f367419f290 con 0x7f3674073be0 2024-05-12T22:11:18.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.965+0000 7f3672800640 1 --2- 172.21.15.28:0/1691108371 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f367410e0c0 0x7f367419eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:11:18.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.966+0000 7f3671e00640 1 --2- 172.21.15.28:0/1691108371 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3674073be0 0x7f367419a2f0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f366400e9f0 tx=0x7f366400edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:11:18.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.966+0000 7f366aa00640 1 -- 172.21.15.28:0/1691108371 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f366400cd80 con 0x7f3674073be0 2024-05-12T22:11:18.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.966+0000 7f366aa00640 1 -- 172.21.15.28:0/1691108371 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3664005500 con 0x7f3674073be0 2024-05-12T22:11:18.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.966+0000 7f3678a00640 1 -- 172.21.15.28:0/1691108371 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f367419f580 con 0x7f3674073be0 2024-05-12T22:11:18.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.966+0000 7f366aa00640 1 -- 172.21.15.28:0/1691108371 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3664010690 con 0x7f3674073be0 2024-05-12T22:11:18.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.966+0000 7f3678a00640 1 -- 172.21.15.28:0/1691108371 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36741a6e10 con 0x7f3674073be0 2024-05-12T22:11:18.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.967+0000 7f3678a00640 1 -- 172.21.15.28:0/1691108371 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3674068aa0 con 0x7f3674073be0 2024-05-12T22:11:18.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.969+0000 7f366aa00640 1 -- 172.21.15.28:0/1691108371 <== mon.2 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7f36640046c0 con 0x7f3674073be0 2024-05-12T22:11:18.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.970+0000 7f366aa00640 1 --2- 172.21.15.28:0/1691108371 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f36580a2d00 0x7f36580a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:18.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.970+0000 7f366aa00640 1 -- 172.21.15.28:0/1691108371 <== mon.2 v2:172.21.15.86:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f36640cf8a0 con 0x7f3674073be0 2024-05-12T22:11:18.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.971+0000 7f3671400640 1 --2- 172.21.15.28:0/1691108371 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f36580a2d00 0x7f36580a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:18.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.971+0000 7f3671400640 1 --2- 172.21.15.28:0/1691108371 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f36580a2d00 0x7f36580a51c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f36600095f0 tx=0x7f3660005f90 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:11:18.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:18.974+0000 7f366aa00640 1 -- 172.21.15.28:0/1691108371 <== mon.2 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f366409c490 con 0x7f3674073be0 2024-05-12T22:11:19.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.227+0000 7f3678a00640 1 -- 172.21.15.28:0/1691108371 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f36741092a0 con 0x7f3674073be0 2024-05-12T22:11:19.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.228+0000 7f366aa00640 1 -- 172.21.15.28:0/1691108371 <== mon.2 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11865 (secure 0 0 0) 0x7f366409bd90 con 0x7f3674073be0 2024-05-12T22:11:19.229 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-12T22:11:19.229 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1844141745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1844141745},{"type":"v1","addr":"172.21.15.186:6833","nonce":1844141745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6833/2960763758","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":2960763758},{"type":"v1","addr":"172.21.15.86:6833","nonce":2960763758}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24470,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/4131134219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":4131134219},{"type":"v1","addr":"172.21.15.28:6835","nonce":4131134219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24481,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/1727869064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1727869064},{"type":"v1","addr":"172.21.15.186:6835","nonce":1727869064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24482,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2938260704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2938260704},{"type":"v1","addr":"172.21.15.28:6837","nonce":2938260704}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24494,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3473700910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3473700910},{"type":"v1","addr":"172.21.15.86:6835","nonce":3473700910}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-12T22:06:33.426986+0000","modified":"2024-05-12T22:10:42.498031+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24581,"mds_1":24550,"mds_2":24532},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24499":{"gid":24499,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6837/3600809235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":3600809235},{"type":"v1","addr":"172.21.15.186:6837","nonce":3600809235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"i","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6839/2399610807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":2399610807},{"type":"v1","addr":"172.21.15.186:6839","nonce":2399610807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24530":{"gid":24530,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.86:6837/3782448884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":3782448884},{"type":"v1","addr":"172.21.15.86:6837","nonce":3782448884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24532":{"gid":24532,"name":"g","rank":2,"incarnation":21,"state":"up:active","state_seq":6,"addr":"172.21.15.28:6839/123892158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":123892158},{"type":"v1","addr":"172.21.15.28:6839","nonce":123892158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24550":{"gid":24550,"name":"a","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.28:6841/761183870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":761183870},{"type":"v1","addr":"172.21.15.28:6841","nonce":761183870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24581":{"gid":24581,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6839/1251999130","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6838","nonce":1251999130},{"type":"v1","addr":"172.21.15.86:6839","nonce":1251999130}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24581,"qdb_cluster":[24550,24532,24581]},"id":1}]} 2024-05-12T22:11:19.230 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 25 2024-05-12T22:11:19.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.231+0000 7f3678a00640 1 -- 172.21.15.28:0/1691108371 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f36580a2d00 msgr2=0x7f36580a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:11:19.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.231+0000 7f3678a00640 1 --2- 172.21.15.28:0/1691108371 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f36580a2d00 0x7f36580a51c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f36600095f0 tx=0x7f3660005f90 comp rx=0 tx=0).stop 2024-05-12T22:11:19.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.231+0000 7f3678a00640 1 -- 172.21.15.28:0/1691108371 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3674073be0 msgr2=0x7f367419a2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:11:19.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.231+0000 7f3678a00640 1 --2- 172.21.15.28:0/1691108371 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3674073be0 0x7f367419a2f0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f366400e9f0 tx=0x7f366400edf0 comp rx=0 tx=0).stop 2024-05-12T22:11:19.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.231+0000 7f3678a00640 1 -- 172.21.15.28:0/1691108371 shutdown_connections 2024-05-12T22:11:19.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.231+0000 7f3678a00640 1 --2- 172.21.15.28:0/1691108371 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f36580a2d00 0x7f36580a51c0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:19.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.231+0000 7f3678a00640 1 --2- 172.21.15.28:0/1691108371 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f367410e0c0 0x7f367419eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:19.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.231+0000 7f3678a00640 1 --2- 172.21.15.28:0/1691108371 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3674074500 0x7f367419a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:19.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.231+0000 7f3678a00640 1 --2- 172.21.15.28:0/1691108371 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3674073be0 0x7f367419a2f0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:19.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.231+0000 7f3678a00640 1 -- 172.21.15.28:0/1691108371 >> 172.21.15.28:0/1691108371 conn(0x7f3674078bf0 msgr2=0x7f36740fe710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:11:19.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.232+0000 7f3678a00640 1 -- 172.21.15.28:0/1691108371 shutdown_connections 2024-05-12T22:11:19.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.232+0000 7f3678a00640 1 -- 172.21.15.28:0/1691108371 wait complete. 2024-05-12T22:11:19.252 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-05-12T22:11:19.313 INFO:tasks.workunit.client.0.smithi028.stdout:autoconf 2024-05-12T22:11:19.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.448+0000 7fef2b600640 1 -- 172.21.15.28:0/3045576778 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fef24103a70 msgr2=0x7fef24103ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:11:19.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.448+0000 7fef2b600640 1 --2- 172.21.15.28:0/3045576778 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fef24103a70 0x7fef24103ef0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fef10009900 tx=0x7fef1002a3f0 comp rx=0 tx=0).stop 2024-05-12T22:11:19.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.448+0000 7fef2b600640 1 -- 172.21.15.28:0/3045576778 shutdown_connections 2024-05-12T22:11:19.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.448+0000 7fef2b600640 1 --2- 172.21.15.28:0/3045576778 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fef24104430 0x7fef24110940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:19.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.448+0000 7fef2b600640 1 --2- 172.21.15.28:0/3045576778 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fef24103a70 0x7fef24103ef0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:19.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.448+0000 7fef2b600640 1 --2- 172.21.15.28:0/3045576778 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fef24109a80 0x7fef24109e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:19.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.448+0000 7fef2b600640 1 -- 172.21.15.28:0/3045576778 >> 172.21.15.28:0/3045576778 conn(0x7fef240ff5c0 msgr2=0x7fef241019e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:11:19.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.448+0000 7fef2b600640 1 -- 172.21.15.28:0/3045576778 shutdown_connections 2024-05-12T22:11:19.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.448+0000 7fef2b600640 1 -- 172.21.15.28:0/3045576778 wait complete. 2024-05-12T22:11:19.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.449+0000 7fef2b600640 1 Processor -- start 2024-05-12T22:11:19.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.450+0000 7fef2b600640 1 -- start start 2024-05-12T22:11:19.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.450+0000 7fef2b600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fef24103a70 0x7fef2419a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:19.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.450+0000 7fef2b600640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fef24104430 0x7fef2419a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:19.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.450+0000 7fef2b600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fef24109a80 0x7fef2419ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:19.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.450+0000 7fef2b600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fef24113ee0 con 0x7fef24103a70 2024-05-12T22:11:19.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.450+0000 7fef2b600640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fef24113d60 con 0x7fef24109a80 2024-05-12T22:11:19.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.450+0000 7fef2b600640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fef24114060 con 0x7fef24104430 2024-05-12T22:11:19.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.450+0000 7fef28c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fef24103a70 0x7fef2419a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:19.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.451+0000 7fef23e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fef24104430 0x7fef2419a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:19.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.451+0000 7fef29600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fef24109a80 0x7fef2419ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:19.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.451+0000 7fef28c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fef24103a70 0x7fef2419a400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60832/0 (socket says 172.21.15.28:60832) 2024-05-12T22:11:19.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.451+0000 7fef28c00640 1 -- 172.21.15.28:0/2216045665 learned_addr learned my addr 172.21.15.28:0/2216045665 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:11:19.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.451+0000 7fef28c00640 1 -- 172.21.15.28:0/2216045665 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fef24104430 msgr2=0x7fef2419a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:11:19.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.451+0000 7fef28c00640 1 --2- 172.21.15.28:0/2216045665 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fef24104430 0x7fef2419a940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:19.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.451+0000 7fef28c00640 1 -- 172.21.15.28:0/2216045665 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fef24109a80 msgr2=0x7fef2419ec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:11:19.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.451+0000 7fef28c00640 1 --2- 172.21.15.28:0/2216045665 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fef24109a80 0x7fef2419ec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:19.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.451+0000 7fef28c00640 1 -- 172.21.15.28:0/2216045665 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef2419f3a0 con 0x7fef24103a70 2024-05-12T22:11:19.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.451+0000 7fef29600640 1 --2- 172.21.15.28:0/2216045665 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fef24109a80 0x7fef2419ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:11:19.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.451+0000 7fef28c00640 1 --2- 172.21.15.28:0/2216045665 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fef24103a70 0x7fef2419a400 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fef1800e9f0 tx=0x7fef1800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:11:19.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.452+0000 7fef21600640 1 -- 172.21.15.28:0/2216045665 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef1800ccc0 con 0x7fef24103a70 2024-05-12T22:11:19.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.452+0000 7fef21600640 1 -- 172.21.15.28:0/2216045665 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fef1800ce60 con 0x7fef24103a70 2024-05-12T22:11:19.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.452+0000 7fef2b600640 1 -- 172.21.15.28:0/2216045665 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef2419f690 con 0x7fef24103a70 2024-05-12T22:11:19.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.452+0000 7fef21600640 1 -- 172.21.15.28:0/2216045665 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef18018630 con 0x7fef24103a70 2024-05-12T22:11:19.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.452+0000 7fef2b600640 1 -- 172.21.15.28:0/2216045665 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef241a6f20 con 0x7fef24103a70 2024-05-12T22:11:19.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.455+0000 7fef2e430740 1 Processor -- start 2024-05-12T22:11:19.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.455+0000 7fef21600640 1 -- 172.21.15.28:0/2216045665 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7fef180046c0 con 0x7fef24103a70 2024-05-12T22:11:19.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.455+0000 7fef2e430740 1 -- start start 2024-05-12T22:11:19.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.456+0000 7fef2e430740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55d0d3ea3980 0x55d0d3ea3d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:19.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.456+0000 7fef2e430740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0d3ea42a0 0x55d0d3eae430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:19.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.456+0000 7fef2e430740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d0d3eae970 0x55d0d3eb0d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:19.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.456+0000 7fef2e430740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d0d3e08a30 con 0x55d0d3ea42a0 2024-05-12T22:11:19.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.456+0000 7fef2e430740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x55d0d3db0700 con 0x55d0d3ea3980 2024-05-12T22:11:19.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.456+0000 7fef2e430740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55d0d3e049f0 con 0x55d0d3eae970 2024-05-12T22:11:19.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.456+0000 7fef28c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0d3ea42a0 0x55d0d3eae430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:19.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.456+0000 7fef29600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55d0d3ea3980 0x55d0d3ea3d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:19.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.456+0000 7fef29600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55d0d3ea3980 0x55d0d3ea3d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:41638/0 (socket says 172.21.15.28:41638) 2024-05-12T22:11:19.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.456+0000 7fef29600640 1 -- 172.21.15.28:0/2442688434 learned_addr learned my addr 172.21.15.28:0/2442688434 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:11:19.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.456+0000 7fef23e00640 1 --2- 172.21.15.28:0/2442688434 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d0d3eae970 0x55d0d3eb0d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:19.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.457+0000 7fef23e00640 1 -- 172.21.15.28:0/2442688434 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55d0d3ea3980 msgr2=0x55d0d3ea3d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:11:19.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.457+0000 7fef23e00640 1 --2- 172.21.15.28:0/2442688434 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55d0d3ea3980 0x55d0d3ea3d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:19.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.457+0000 7fef23e00640 1 -- 172.21.15.28:0/2442688434 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0d3ea42a0 msgr2=0x55d0d3eae430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:11:19.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.457+0000 7fef23e00640 1 --2- 172.21.15.28:0/2442688434 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0d3ea42a0 0x55d0d3eae430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:19.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.457+0000 7fef23e00640 1 -- 172.21.15.28:0/2442688434 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d0d3b1c3f0 con 0x55d0d3eae970 2024-05-12T22:11:19.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.457+0000 7fef23e00640 1 --2- 172.21.15.28:0/2442688434 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d0d3eae970 0x55d0d3eb0d60 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fef1002a9e0 tx=0x7fef10005e00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=730b6792174d6474 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:11:19.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.457+0000 7fef15c00640 1 -- 172.21.15.28:0/2442688434 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef10035040 con 0x55d0d3eae970 2024-05-12T22:11:19.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.458+0000 7fef15c00640 1 -- 172.21.15.28:0/2442688434 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fef10004400 con 0x55d0d3eae970 2024-05-12T22:11:19.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.458+0000 7fef15c00640 1 -- 172.21.15.28:0/2442688434 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef100035f0 con 0x55d0d3eae970 2024-05-12T22:11:19.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.458+0000 7fef2e430740 1 -- 172.21.15.28:0/2442688434 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d0d3eae970 msgr2=0x55d0d3eb0d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:11:19.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.458+0000 7fef2e430740 1 --2- 172.21.15.28:0/2442688434 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d0d3eae970 0x55d0d3eb0d60 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fef1002a9e0 tx=0x7fef10005e00 comp rx=0 tx=0).stop 2024-05-12T22:11:19.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.458+0000 7fef2e430740 1 -- 172.21.15.28:0/2442688434 shutdown_connections 2024-05-12T22:11:19.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.458+0000 7fef2e430740 1 --2- 172.21.15.28:0/2442688434 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d0d3eae970 0x55d0d3eb0d60 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:19.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.459+0000 7fef2e430740 1 --2- 172.21.15.28:0/2442688434 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0d3ea42a0 0x55d0d3eae430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:19.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.459+0000 7fef2e430740 1 --2- 172.21.15.28:0/2442688434 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55d0d3ea3980 0x55d0d3ea3d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:19.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.459+0000 7fef2e430740 1 -- 172.21.15.28:0/2442688434 >> 172.21.15.28:0/2442688434 conn(0x55d0d3e9fca0 msgr2=0x55d0d3ea0080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:11:19.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.459+0000 7fef2e430740 1 -- 172.21.15.28:0/2442688434 shutdown_connections 2024-05-12T22:11:19.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.459+0000 7fef2e430740 1 -- 172.21.15.28:0/2442688434 wait complete. 2024-05-12T22:11:19.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.459+0000 7fef21600640 1 --2- 172.21.15.28:0/2216045665 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fef0c0a2c10 0x7fef0c0a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:19.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.459+0000 7fef2e430740 1 -- start start 2024-05-12T22:11:19.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.459+0000 7fef21600640 1 -- 172.21.15.28:0/2216045665 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7fef180cf8a0 con 0x7fef24103a70 2024-05-12T22:11:19.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.459+0000 7fef29600640 1 --2- 172.21.15.28:0/2216045665 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fef0c0a2c10 0x7fef0c0a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:19.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.459+0000 7fef2e430740 1 Processor -- start 2024-05-12T22:11:19.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.460+0000 7fef2e430740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0d3ea3980 0x55d0d3eab900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:19.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.460+0000 7fef2e430740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55d0d3ea42a0 0x55d0d3eabe40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:19.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.460+0000 7fef2e430740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d0d3eae970 0x55d0d3eac380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:19.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.460+0000 7fef2e430740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d0d3eb1700 con 0x55d0d3ea3980 2024-05-12T22:11:19.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.460+0000 7fef2e430740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x55d0d3dc3a70 con 0x55d0d3ea42a0 2024-05-12T22:11:19.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.460+0000 7fef2e430740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55d0d3eb1880 con 0x55d0d3eae970 2024-05-12T22:11:19.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.460+0000 7fef29600640 1 --2- 172.21.15.28:0/2216045665 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fef0c0a2c10 0x7fef0c0a50d0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fef1c012940 tx=0x7fef1c009500 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:11:19.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.460+0000 7fef2e430740 10 client.0 _ll_register_callbacks cb 0x55d0d3da6ba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:11:19.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.461+0000 7fef23e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55d0d3ea42a0 0x55d0d3eabe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:19.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.461+0000 7fef28c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0d3ea3980 0x55d0d3eab900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:19.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.461+0000 7fef23e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55d0d3ea42a0 0x55d0d3eabe40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:41642/0 (socket says 172.21.15.28:41642) 2024-05-12T22:11:19.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.461+0000 7fef23e00640 1 -- 172.21.15.28:0/3567011626 learned_addr learned my addr 172.21.15.28:0/3567011626 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:11:19.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.461+0000 7fef29600640 1 --2- 172.21.15.28:0/3567011626 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d0d3eae970 0x55d0d3eac380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:19.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.461+0000 7fef23e00640 1 -- 172.21.15.28:0/3567011626 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d0d3eae970 msgr2=0x55d0d3eac380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:11:19.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.461+0000 7fef23e00640 1 --2- 172.21.15.28:0/3567011626 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d0d3eae970 0x55d0d3eac380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:19.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.461+0000 7fef23e00640 1 -- 172.21.15.28:0/3567011626 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0d3ea3980 msgr2=0x55d0d3eab900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:11:19.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.461+0000 7fef23e00640 1 --2- 172.21.15.28:0/3567011626 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0d3ea3980 0x55d0d3eab900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:19.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.461+0000 7fef23e00640 1 -- 172.21.15.28:0/3567011626 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d0d3be67d0 con 0x55d0d3ea42a0 2024-05-12T22:11:19.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.461+0000 7fef29600640 1 --2- 172.21.15.28:0/3567011626 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d0d3eae970 0x55d0d3eac380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:11:19.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.461+0000 7fef23e00640 1 --2- 172.21.15.28:0/3567011626 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55d0d3ea42a0 0x55d0d3eabe40 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fef1002a430 tx=0x7fef10004950 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5bc908d5e7a3749b server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:11:19.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.461+0000 7fef0a000640 10 client.0 ms_handle_connect on v2:172.21.15.86:3300/0 2024-05-12T22:11:19.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.462+0000 7fef0a000640 1 -- 172.21.15.28:0/3567011626 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef10035070 con 0x55d0d3ea42a0 2024-05-12T22:11:19.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.462+0000 7fef2e430740 1 -- 172.21.15.28:0/3567011626 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fef10004400 con 0x55d0d3ea42a0 2024-05-12T22:11:19.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.462+0000 7fef0a000640 1 -- 172.21.15.28:0/3567011626 <== mon.2 v2:172.21.15.86:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7fef10003800 con 0x55d0d3ea42a0 2024-05-12T22:11:19.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.462+0000 7fef0a000640 1 -- 172.21.15.28:0/3567011626 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fef10049070 con 0x55d0d3ea42a0 2024-05-12T22:11:19.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.462+0000 7fef2e430740 10 client.25829 fetch_fsmap learned FSMap version 25 2024-05-12T22:11:19.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.462+0000 7fef2e430740 1 -- 172.21.15.28:0/3567011626 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55d0d3ebd9a0 con 0x55d0d3ea42a0 2024-05-12T22:11:19.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.462+0000 7fef0a000640 1 -- 172.21.15.28:0/3567011626 <== mon.2 v2:172.21.15.86:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fef10004400 con 0x55d0d3ea42a0 2024-05-12T22:11:19.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.462+0000 7fef0a000640 1 -- 172.21.15.28:0/3567011626 <== mon.2 v2:172.21.15.86:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7fef10031bf0 con 0x55d0d3ea42a0 2024-05-12T22:11:19.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.463+0000 7fef2e430740 10 client.25829 fetch_fsmap finished waiting for FSMap version 25 2024-05-12T22:11:19.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.463+0000 7fef2e430740 10 client.25829 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.g 2024-05-12T22:11:19.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.463+0000 7fef2e430740 20 client.25829 populate_metadata read hostname 'smithi028' 2024-05-12T22:11:19.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.463+0000 7fef2e430740 1 --2- 172.21.15.28:0/3567011626 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x55d0d3ebecf0 0x55d0d3ec10e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:19.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.463+0000 7fef2e430740 4 client.25829 mds_command: new command op to 24532 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-12T22:11:19.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.463+0000 7fef2e430740 1 -- 172.21.15.28:0/3567011626 --> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d0d3e16160 con 0x55d0d3ebecf0 2024-05-12T22:11:19.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.463+0000 7fef28c00640 1 --2- 172.21.15.28:0/3567011626 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x55d0d3ebecf0 0x55d0d3ec10e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:19.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.463+0000 7fef28c00640 1 --2- 172.21.15.28:0/3567011626 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x55d0d3ebecf0 0x55d0d3ec10e0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=5db5c6981b2cd84e server_cookie=14614ff8300d9de4 in_seq=0 out_seq=0 2024-05-12T22:11:19.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.463+0000 7fef0a000640 10 client.25829 ms_handle_connect on v2:172.21.15.28:6838/123892158 2024-05-12T22:11:19.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.465+0000 7fef0a000640 1 -- 172.21.15.28:0/3567011626 <== mds.2 v2:172.21.15.28:6838/123892158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d0d3e16160 con 0x55d0d3ebecf0 2024-05-12T22:11:19.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.465+0000 7fef0a000640 10 client.25829 handle_command_reply: tid=0 2024-05-12T22:11:19.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.465+0000 7fef2e430740 1 client.25829 shutdown 2024-05-12T22:11:19.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.465+0000 7fef2e430740 1 -- 172.21.15.28:0/3567011626 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55d0d3ea42a0 msgr2=0x55d0d3eabe40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:11:19.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.465+0000 7fef2e430740 1 --2- 172.21.15.28:0/3567011626 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55d0d3ea42a0 0x55d0d3eabe40 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fef1002a430 tx=0x7fef10004950 comp rx=0 tx=0).stop 2024-05-12T22:11:19.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.465+0000 7fef2e430740 1 -- 172.21.15.28:0/3567011626 shutdown_connections 2024-05-12T22:11:19.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.465+0000 7fef2e430740 1 --2- 172.21.15.28:0/3567011626 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d0d3eae970 0x55d0d3eac380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:19.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.465+0000 7fef2e430740 1 --2- 172.21.15.28:0/3567011626 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x55d0d3ebecf0 0x55d0d3ec10e0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:19.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.465+0000 7fef2e430740 1 --2- 172.21.15.28:0/3567011626 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55d0d3ea42a0 0x55d0d3eabe40 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:19.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.465+0000 7fef2e430740 1 --2- 172.21.15.28:0/3567011626 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0d3ea3980 0x55d0d3eab900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:19.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.465+0000 7fef0a000640 0 client.25829 ms_handle_reset on v2:172.21.15.28:6838/123892158 2024-05-12T22:11:19.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.465+0000 7fef2e430740 1 -- 172.21.15.28:0/3567011626 >> 172.21.15.28:0/3567011626 conn(0x55d0d3e9fca0 msgr2=0x55d0d3ea5aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:11:19.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.465+0000 7fef2e430740 1 -- 172.21.15.28:0/3567011626 shutdown_connections 2024-05-12T22:11:19.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.465+0000 7fef2e430740 1 -- 172.21.15.28:0/3567011626 wait complete. 2024-05-12T22:11:19.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.465+0000 7fef2e430740 20 client.25829 trim_cache size 0 max 16384 2024-05-12T22:11:19.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.488+0000 7fef2e430740 1 Processor -- start 2024-05-12T22:11:19.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.488+0000 7fef2e430740 1 -- start start 2024-05-12T22:11:19.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.489+0000 7fef2e430740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55d0d3ea3980 0x55d0d3eba5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:19.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.489+0000 7fef2e430740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d0d3ea42a0 0x55d0d3ea6120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:19.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.489+0000 7fef2e430740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0d3eae970 0x55d0d3ec66b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:19.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.489+0000 7fef2e430740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fef241b1fa0 con 0x55d0d3eae970 2024-05-12T22:11:19.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.489+0000 7fef2e430740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fef241af870 con 0x55d0d3ea3980 2024-05-12T22:11:19.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.489+0000 7fef2e430740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fef241ad6c0 con 0x55d0d3ea42a0 2024-05-12T22:11:19.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.489+0000 7fef29600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0d3eae970 0x55d0d3ec66b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:19.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.489+0000 7fef29600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0d3eae970 0x55d0d3ec66b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60868/0 (socket says 172.21.15.28:60868) 2024-05-12T22:11:19.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.490+0000 7fef23e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d0d3ea42a0 0x55d0d3ea6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:19.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.490+0000 7fef29600640 1 -- 172.21.15.28:0/1391402136 learned_addr learned my addr 172.21.15.28:0/1391402136 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:11:19.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.490+0000 7fef28c00640 1 --2- 172.21.15.28:0/1391402136 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55d0d3ea3980 0x55d0d3eba5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:19.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.490+0000 7fef23e00640 1 -- 172.21.15.28:0/1391402136 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55d0d3ea3980 msgr2=0x55d0d3eba5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:11:19.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.490+0000 7fef23e00640 1 --2- 172.21.15.28:0/1391402136 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55d0d3ea3980 0x55d0d3eba5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:19.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.490+0000 7fef23e00640 1 -- 172.21.15.28:0/1391402136 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0d3eae970 msgr2=0x55d0d3ec66b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:11:19.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.490+0000 7fef23e00640 1 --2- 172.21.15.28:0/1391402136 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0d3eae970 0x55d0d3ec66b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:19.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.490+0000 7fef23e00640 1 -- 172.21.15.28:0/1391402136 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d0d3e9a210 con 0x55d0d3ea42a0 2024-05-12T22:11:19.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.490+0000 7fef29600640 1 --2- 172.21.15.28:0/1391402136 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0d3eae970 0x55d0d3ec66b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:11:19.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.490+0000 7fef28c00640 1 --2- 172.21.15.28:0/1391402136 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55d0d3ea3980 0x55d0d3eba5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:11:19.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.491+0000 7fef23e00640 1 --2- 172.21.15.28:0/1391402136 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d0d3ea42a0 0x55d0d3ea6120 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fef1002a910 tx=0x7fef10005e00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4fbfbcfb2b2269f4 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:11:19.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.492+0000 7fef15200640 1 -- 172.21.15.28:0/1391402136 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef10035070 con 0x55d0d3ea42a0 2024-05-12T22:11:19.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.492+0000 7fef15200640 1 -- 172.21.15.28:0/1391402136 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fef10004200 con 0x55d0d3ea42a0 2024-05-12T22:11:19.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.492+0000 7fef15200640 1 -- 172.21.15.28:0/1391402136 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef10003d40 con 0x55d0d3ea42a0 2024-05-12T22:11:19.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.492+0000 7fef2e430740 1 -- 172.21.15.28:0/1391402136 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d0d3ea42a0 msgr2=0x55d0d3ea6120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:11:19.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.492+0000 7fef2e430740 1 --2- 172.21.15.28:0/1391402136 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d0d3ea42a0 0x55d0d3ea6120 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fef1002a910 tx=0x7fef10005e00 comp rx=0 tx=0).stop 2024-05-12T22:11:19.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.493+0000 7fef2e430740 1 -- 172.21.15.28:0/1391402136 shutdown_connections 2024-05-12T22:11:19.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.493+0000 7fef2e430740 1 --2- 172.21.15.28:0/1391402136 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0d3eae970 0x55d0d3ec66b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:19.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.493+0000 7fef2e430740 1 --2- 172.21.15.28:0/1391402136 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d0d3ea42a0 0x55d0d3ea6120 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:19.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.493+0000 7fef2e430740 1 --2- 172.21.15.28:0/1391402136 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55d0d3ea3980 0x55d0d3eba5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:19.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.493+0000 7fef2e430740 1 -- 172.21.15.28:0/1391402136 >> 172.21.15.28:0/1391402136 conn(0x55d0d3e9fca0 msgr2=0x55d0d3e9edc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:11:19.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.493+0000 7fef2e430740 1 -- 172.21.15.28:0/1391402136 shutdown_connections 2024-05-12T22:11:19.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.493+0000 7fef2e430740 1 -- 172.21.15.28:0/1391402136 wait complete. 2024-05-12T22:11:19.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.493+0000 7fef2e430740 1 -- start start 2024-05-12T22:11:19.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.494+0000 7fef2e430740 1 Processor -- start 2024-05-12T22:11:19.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.494+0000 7fef2e430740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d0d3ea3980 0x55d0d3ea8790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:19.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.494+0000 7fef2e430740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55d0d3ea42a0 0x55d0d3ea8cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:19.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.494+0000 7fef2e430740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0d3eae970 0x55d0d3ea9210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:19.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.494+0000 7fef2e430740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fef241add30 con 0x55d0d3eae970 2024-05-12T22:11:19.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.494+0000 7fef2e430740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fef241af1a0 con 0x55d0d3ea42a0 2024-05-12T22:11:19.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.494+0000 7fef2e430740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fef241ae400 con 0x55d0d3ea3980 2024-05-12T22:11:19.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.494+0000 7fef2e430740 10 client.0 _ll_register_callbacks cb 0x7fef10009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:11:19.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.494+0000 7fef28c00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d0d3ea3980 0x55d0d3ea8790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:19.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.495+0000 7fef23e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55d0d3ea42a0 0x55d0d3ea8cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:19.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.495+0000 7fef29600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0d3eae970 0x55d0d3ea9210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:19.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.495+0000 7fef28c00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d0d3ea3980 0x55d0d3ea8790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:59100/0 (socket says 172.21.15.28:59100) 2024-05-12T22:11:19.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.495+0000 7fef28c00640 1 -- 172.21.15.28:0/1544221603 learned_addr learned my addr 172.21.15.28:0/1544221603 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:11:19.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.495+0000 7fef28c00640 1 -- 172.21.15.28:0/1544221603 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55d0d3ea42a0 msgr2=0x55d0d3ea8cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:11:19.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.495+0000 7fef28c00640 1 --2- 172.21.15.28:0/1544221603 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55d0d3ea42a0 0x55d0d3ea8cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:19.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.495+0000 7fef28c00640 1 -- 172.21.15.28:0/1544221603 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0d3eae970 msgr2=0x55d0d3ea9210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:11:19.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.495+0000 7fef28c00640 1 --2- 172.21.15.28:0/1544221603 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0d3eae970 0x55d0d3ea9210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:19.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.495+0000 7fef28c00640 1 -- 172.21.15.28:0/1544221603 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d0d3f240c0 con 0x55d0d3ea3980 2024-05-12T22:11:19.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.495+0000 7fef23e00640 1 --2- 172.21.15.28:0/1544221603 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55d0d3ea42a0 0x55d0d3ea8cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:11:19.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.495+0000 7fef29600640 1 --2- 172.21.15.28:0/1544221603 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0d3eae970 0x55d0d3ea9210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:11:19.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.496+0000 7fef28c00640 1 --2- 172.21.15.28:0/1544221603 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d0d3ea3980 0x55d0d3ea8790 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fef180987c0 tx=0x7fef180cfc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d159030a104c661e server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:11:19.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.496+0000 7fef0a000640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-12T22:11:19.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.496+0000 7fef0a000640 1 -- 172.21.15.28:0/1544221603 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef1809bdf0 con 0x55d0d3ea3980 2024-05-12T22:11:19.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.496+0000 7fef0a000640 1 -- 172.21.15.28:0/1544221603 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7fef18098460 con 0x55d0d3ea3980 2024-05-12T22:11:19.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.496+0000 7fef2e430740 1 -- 172.21.15.28:0/1544221603 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fef10004200 con 0x55d0d3ea3980 2024-05-12T22:11:19.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.497+0000 7fef0a000640 1 -- 172.21.15.28:0/1544221603 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fef180a01a0 con 0x55d0d3ea3980 2024-05-12T22:11:19.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.497+0000 7fef2e430740 10 client.25789 fetch_fsmap learned FSMap version 25 2024-05-12T22:11:19.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.497+0000 7fef2e430740 1 -- 172.21.15.28:0/1544221603 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55d0d3f253e0 con 0x55d0d3ea3980 2024-05-12T22:11:19.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.497+0000 7fef0a000640 1 -- 172.21.15.28:0/1544221603 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fef10004200 con 0x55d0d3ea3980 2024-05-12T22:11:19.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.497+0000 7fef0a000640 1 -- 172.21.15.28:0/1544221603 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7fef1809acb0 con 0x55d0d3ea3980 2024-05-12T22:11:19.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.497+0000 7fef2e430740 10 client.25789 fetch_fsmap finished waiting for FSMap version 25 2024-05-12T22:11:19.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.497+0000 7fef2e430740 10 client.25789 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.g 2024-05-12T22:11:19.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.497+0000 7fef2e430740 20 client.25789 populate_metadata read hostname 'smithi028' 2024-05-12T22:11:19.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.498+0000 7fef2e430740 1 --2- 172.21.15.28:0/1544221603 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x55d0d3f25b80 0x55d0d3f27fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:19.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.498+0000 7fef2e430740 4 client.25789 mds_command: new command op to 24532 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-12T22:11:19.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.498+0000 7fef2e430740 1 -- 172.21.15.28:0/1544221603 --> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d0d3e104b0 con 0x55d0d3f25b80 2024-05-12T22:11:19.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.498+0000 7fef23e00640 1 --2- 172.21.15.28:0/1544221603 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x55d0d3f25b80 0x55d0d3f27fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:19.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.498+0000 7fef23e00640 1 --2- 172.21.15.28:0/1544221603 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x55d0d3f25b80 0x55d0d3f27fa0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=801f4e470e1d447d server_cookie=c29d50ab41c42a8b in_seq=0 out_seq=0 2024-05-12T22:11:19.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.498+0000 7fef0a000640 10 client.25789 ms_handle_connect on v2:172.21.15.28:6838/123892158 2024-05-12T22:11:19.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.499+0000 7fef0a000640 1 -- 172.21.15.28:0/1544221603 <== mds.2 v2:172.21.15.28:6838/123892158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d0d3e104b0 con 0x55d0d3f25b80 2024-05-12T22:11:19.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.499+0000 7fef0a000640 10 client.25789 handle_command_reply: tid=0 2024-05-12T22:11:19.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.499+0000 7fef2e430740 1 client.25789 shutdown 2024-05-12T22:11:19.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.499+0000 7fef2e430740 1 -- 172.21.15.28:0/1544221603 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d0d3ea3980 msgr2=0x55d0d3ea8790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:11:19.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.499+0000 7fef2e430740 1 --2- 172.21.15.28:0/1544221603 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d0d3ea3980 0x55d0d3ea8790 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fef180987c0 tx=0x7fef180cfc30 comp rx=0 tx=0).stop 2024-05-12T22:11:19.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.499+0000 7fef2e430740 1 -- 172.21.15.28:0/1544221603 shutdown_connections 2024-05-12T22:11:19.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.499+0000 7fef2e430740 1 --2- 172.21.15.28:0/1544221603 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0d3eae970 0x55d0d3ea9210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:19.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.499+0000 7fef2e430740 1 --2- 172.21.15.28:0/1544221603 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x55d0d3f25b80 0x55d0d3f27fa0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:19.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.499+0000 7fef2e430740 1 --2- 172.21.15.28:0/1544221603 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55d0d3ea42a0 0x55d0d3ea8cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:19.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.499+0000 7fef2e430740 1 --2- 172.21.15.28:0/1544221603 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d0d3ea3980 0x55d0d3ea8790 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:19.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.499+0000 7fef0a000640 0 client.25789 ms_handle_reset on v2:172.21.15.28:6838/123892158 2024-05-12T22:11:19.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.499+0000 7fef2e430740 1 -- 172.21.15.28:0/1544221603 >> 172.21.15.28:0/1544221603 conn(0x55d0d3e9fca0 msgr2=0x55d0d3eab8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:11:19.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.499+0000 7fef2e430740 1 -- 172.21.15.28:0/1544221603 shutdown_connections 2024-05-12T22:11:19.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.499+0000 7fef2e430740 1 -- 172.21.15.28:0/1544221603 wait complete. 2024-05-12T22:11:19.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.499+0000 7fef2e430740 20 client.25789 trim_cache size 0 max 16384 2024-05-12T22:11:19.502 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-05-12T22:11:19.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.500+0000 7fef2b600640 1 -- 172.21.15.28:0/2216045665 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fef0c0a2c10 msgr2=0x7fef0c0a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:11:19.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.500+0000 7fef2b600640 1 --2- 172.21.15.28:0/2216045665 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fef0c0a2c10 0x7fef0c0a50d0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fef1c012940 tx=0x7fef1c009500 comp rx=0 tx=0).stop 2024-05-12T22:11:19.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.500+0000 7fef2b600640 1 -- 172.21.15.28:0/2216045665 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fef24103a70 msgr2=0x7fef2419a400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:11:19.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.500+0000 7fef2b600640 1 --2- 172.21.15.28:0/2216045665 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fef24103a70 0x7fef2419a400 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fef1800e9f0 tx=0x7fef1800edf0 comp rx=0 tx=0).stop 2024-05-12T22:11:19.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.502+0000 7fef2b600640 1 -- 172.21.15.28:0/2216045665 shutdown_connections 2024-05-12T22:11:19.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.502+0000 7fef2b600640 1 --2- 172.21.15.28:0/2216045665 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fef0c0a2c10 0x7fef0c0a50d0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:19.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.502+0000 7fef2b600640 1 --2- 172.21.15.28:0/2216045665 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fef24109a80 0x7fef2419ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:19.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.502+0000 7fef2b600640 1 --2- 172.21.15.28:0/2216045665 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fef24104430 0x7fef2419a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:19.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.502+0000 7fef2b600640 1 --2- 172.21.15.28:0/2216045665 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fef24103a70 0x7fef2419a400 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:19.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.502+0000 7fef2b600640 1 -- 172.21.15.28:0/2216045665 >> 172.21.15.28:0/2216045665 conn(0x7fef240ff5c0 msgr2=0x7fef240ffd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:11:19.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.502+0000 7fef2b600640 1 -- 172.21.15.28:0/2216045665 shutdown_connections 2024-05-12T22:11:19.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.502+0000 7fef2b600640 1 -- 172.21.15.28:0/2216045665 wait complete. 2024-05-12T22:11:19.532 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-05-12T22:11:19.635 INFO:tasks.workunit.client.0.smithi028.stdout:./configure \ 2024-05-12T22:11:19.635 INFO:tasks.workunit.client.0.smithi028.stdout: --libexecdir=/usr/lib \ 2024-05-12T22:11:19.635 INFO:tasks.workunit.client.0.smithi028.stdout: --exec_prefix=/var/lib 2024-05-12T22:11:19.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.726+0000 7f2013c00640 1 -- 172.21.15.28:0/1015295443 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f200c1043b0 msgr2=0x7f200c1108c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:11:19.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.726+0000 7f2013c00640 1 --2- 172.21.15.28:0/1015295443 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f200c1043b0 0x7f200c1108c0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f2004009ca0 tx=0x7f200402a8c0 comp rx=0 tx=0).stop 2024-05-12T22:11:19.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.726+0000 7f2013c00640 1 -- 172.21.15.28:0/1015295443 shutdown_connections 2024-05-12T22:11:19.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.726+0000 7f2013c00640 1 --2- 172.21.15.28:0/1015295443 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f200c1043b0 0x7f200c1108c0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:19.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.726+0000 7f2013c00640 1 --2- 172.21.15.28:0/1015295443 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f200c1039f0 0x7f200c103e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:19.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.726+0000 7f2013c00640 1 --2- 172.21.15.28:0/1015295443 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f200c109a00 0x7f200c109de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:19.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.726+0000 7f2013c00640 1 -- 172.21.15.28:0/1015295443 >> 172.21.15.28:0/1015295443 conn(0x7f200c0ff4e0 msgr2=0x7f200c101900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:11:19.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.726+0000 7f2013c00640 1 -- 172.21.15.28:0/1015295443 shutdown_connections 2024-05-12T22:11:19.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.726+0000 7f2013c00640 1 -- 172.21.15.28:0/1015295443 wait complete. 2024-05-12T22:11:19.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.727+0000 7f2013c00640 1 Processor -- start 2024-05-12T22:11:19.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.728+0000 7f2013c00640 1 -- start start 2024-05-12T22:11:19.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.728+0000 7f2013c00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f200c1039f0 0x7f200c19a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:19.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.728+0000 7f2013c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f200c1043b0 0x7f200c19a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:19.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.728+0000 7f2013c00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f200c109a00 0x7f200c19eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:19.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.728+0000 7f2013c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f200c113e10 con 0x7f200c1043b0 2024-05-12T22:11:19.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.728+0000 7f2013c00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f200c113c90 con 0x7f200c109a00 2024-05-12T22:11:19.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.728+0000 7f2013c00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f200c113f90 con 0x7f200c1039f0 2024-05-12T22:11:19.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.728+0000 7f2011c00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f200c109a00 0x7f200c19eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:19.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.728+0000 7f200be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f200c1043b0 0x7f200c19a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:19.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.728+0000 7f2011200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f200c1039f0 0x7f200c19a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:19.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.728+0000 7f2011c00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f200c109a00 0x7f200c19eb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:41674/0 (socket says 172.21.15.28:41674) 2024-05-12T22:11:19.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.728+0000 7f2011c00640 1 -- 172.21.15.28:0/1236002844 learned_addr learned my addr 172.21.15.28:0/1236002844 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:11:19.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.729+0000 7f2011c00640 1 -- 172.21.15.28:0/1236002844 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f200c1039f0 msgr2=0x7f200c19a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:11:19.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.729+0000 7f2011c00640 1 --2- 172.21.15.28:0/1236002844 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f200c1039f0 0x7f200c19a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:19.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.729+0000 7f2011c00640 1 -- 172.21.15.28:0/1236002844 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f200c1043b0 msgr2=0x7f200c19a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:11:19.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.729+0000 7f2011c00640 1 --2- 172.21.15.28:0/1236002844 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f200c1043b0 0x7f200c19a7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:19.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.729+0000 7f2011c00640 1 -- 172.21.15.28:0/1236002844 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f200c19f250 con 0x7f200c109a00 2024-05-12T22:11:19.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.729+0000 7f2011200640 1 --2- 172.21.15.28:0/1236002844 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f200c1039f0 0x7f200c19a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:11:19.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.729+0000 7f2011c00640 1 --2- 172.21.15.28:0/1236002844 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f200c109a00 0x7f200c19eb20 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f2004006010 tx=0x7f2004003ce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:11:19.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.730+0000 7f2009600640 1 -- 172.21.15.28:0/1236002844 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2004031650 con 0x7f200c109a00 2024-05-12T22:11:19.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.730+0000 7f2009600640 1 -- 172.21.15.28:0/1236002844 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f20040317f0 con 0x7f200c109a00 2024-05-12T22:11:19.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.730+0000 7f2013c00640 1 -- 172.21.15.28:0/1236002844 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f200c19f4e0 con 0x7f200c109a00 2024-05-12T22:11:19.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.730+0000 7f2009600640 1 -- 172.21.15.28:0/1236002844 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20040309b0 con 0x7f200c109a00 2024-05-12T22:11:19.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.730+0000 7f2013c00640 1 -- 172.21.15.28:0/1236002844 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f200c1a6dd0 con 0x7f200c109a00 2024-05-12T22:11:19.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.733+0000 7f20169c8740 1 Processor -- start 2024-05-12T22:11:19.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.733+0000 7f2009600640 1 -- 172.21.15.28:0/1236002844 <== mon.2 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7f2004030b50 con 0x7f200c109a00 2024-05-12T22:11:19.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.734+0000 7f20169c8740 1 -- start start 2024-05-12T22:11:19.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.734+0000 7f20169c8740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55635341b980 0x55635341bd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:19.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.734+0000 7f20169c8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55635341c2a0 0x556353426430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:19.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.734+0000 7f20169c8740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556353426970 0x556353428d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:19.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.734+0000 7f20169c8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x556353380a30 con 0x55635341c2a0 2024-05-12T22:11:19.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.735+0000 7f20169c8740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x556353328700 con 0x55635341b980 2024-05-12T22:11:19.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.735+0000 7f20169c8740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55635337c9f0 con 0x556353426970 2024-05-12T22:11:19.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.735+0000 7f2011200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55635341c2a0 0x556353426430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:19.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.735+0000 7f200be00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55635341b980 0x55635341bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:19.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.735+0000 7f2011200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55635341c2a0 0x556353426430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60910/0 (socket says 172.21.15.28:60910) 2024-05-12T22:11:19.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.735+0000 7f2011200640 1 -- 172.21.15.28:0/2180089276 learned_addr learned my addr 172.21.15.28:0/2180089276 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:11:19.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.735+0000 7f200be00640 1 --2- 172.21.15.28:0/2180089276 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556353426970 0x556353428d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:19.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.735+0000 7f2011200640 1 -- 172.21.15.28:0/2180089276 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556353426970 msgr2=0x556353428d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:11:19.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.735+0000 7f2011200640 1 --2- 172.21.15.28:0/2180089276 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556353426970 0x556353428d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:19.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.735+0000 7f2009600640 1 --2- 172.21.15.28:0/1236002844 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f1ff40a2d80 0x7f1ff40a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:19.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.735+0000 7f2011200640 1 -- 172.21.15.28:0/2180089276 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55635341b980 msgr2=0x55635341bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:11:19.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.735+0000 7f2011200640 1 --2- 172.21.15.28:0/2180089276 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55635341b980 0x55635341bd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:19.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.735+0000 7f2011200640 1 -- 172.21.15.28:0/2180089276 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5563530943f0 con 0x55635341c2a0 2024-05-12T22:11:19.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.735+0000 7f200be00640 1 --2- 172.21.15.28:0/2180089276 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556353426970 0x556353428d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:11:19.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.736+0000 7f2009600640 1 -- 172.21.15.28:0/1236002844 <== mon.2 v2:172.21.15.86:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f20040f06c0 con 0x7f200c109a00 2024-05-12T22:11:19.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.736+0000 7f2011c00640 1 --2- 172.21.15.28:0/1236002844 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f1ff40a2d80 0x7f1ff40a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:19.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.736+0000 7f2011200640 1 --2- 172.21.15.28:0/2180089276 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55635341c2a0 0x556353426430 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f20000114f0 tx=0x7f20000119b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4054b5c97183afaf server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:11:19.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.736+0000 7f1ffe000640 1 -- 172.21.15.28:0/2180089276 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f200001e070 con 0x55635341c2a0 2024-05-12T22:11:19.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.736+0000 7f1ffe000640 1 -- 172.21.15.28:0/2180089276 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f200000faa0 con 0x55635341c2a0 2024-05-12T22:11:19.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.736+0000 7f1ffe000640 1 -- 172.21.15.28:0/2180089276 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2000019b30 con 0x55635341c2a0 2024-05-12T22:11:19.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.737+0000 7f20169c8740 1 -- 172.21.15.28:0/2180089276 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55635341c2a0 msgr2=0x556353426430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:11:19.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.737+0000 7f20169c8740 1 --2- 172.21.15.28:0/2180089276 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55635341c2a0 0x556353426430 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f20000114f0 tx=0x7f20000119b0 comp rx=0 tx=0).stop 2024-05-12T22:11:19.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.737+0000 7f2011c00640 1 --2- 172.21.15.28:0/1236002844 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f1ff40a2d80 0x7f1ff40a5240 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f20040c5400 tx=0x7f20040048c0 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:11:19.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.737+0000 7f20169c8740 1 -- 172.21.15.28:0/2180089276 shutdown_connections 2024-05-12T22:11:19.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.737+0000 7f20169c8740 1 --2- 172.21.15.28:0/2180089276 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556353426970 0x556353428d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:19.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.737+0000 7f20169c8740 1 --2- 172.21.15.28:0/2180089276 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55635341c2a0 0x556353426430 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:19.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.737+0000 7f20169c8740 1 --2- 172.21.15.28:0/2180089276 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55635341b980 0x55635341bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:19.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.737+0000 7f20169c8740 1 -- 172.21.15.28:0/2180089276 >> 172.21.15.28:0/2180089276 conn(0x556353417ca0 msgr2=0x556353418080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:11:19.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.737+0000 7f20169c8740 1 -- 172.21.15.28:0/2180089276 shutdown_connections 2024-05-12T22:11:19.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.737+0000 7f20169c8740 1 -- 172.21.15.28:0/2180089276 wait complete. 2024-05-12T22:11:19.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.738+0000 7f20169c8740 1 -- start start 2024-05-12T22:11:19.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.738+0000 7f20169c8740 1 Processor -- start 2024-05-12T22:11:19.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.738+0000 7f20169c8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55635341b980 0x5563534238b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:19.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.739+0000 7f20169c8740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55635341c2a0 0x556353423df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:19.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.739+0000 7f20169c8740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556353426970 0x556353424330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:19.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.739+0000 7f20169c8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x556353429760 con 0x55635341b980 2024-05-12T22:11:19.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.739+0000 7f20169c8740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x55635333ba70 con 0x55635341c2a0 2024-05-12T22:11:19.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.739+0000 7f20169c8740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5563534298e0 con 0x556353426970 2024-05-12T22:11:19.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.739+0000 7f20169c8740 10 client.0 _ll_register_callbacks cb 0x55635331eba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:11:19.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.739+0000 7f200be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55635341b980 0x5563534238b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:19.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.739+0000 7f200be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55635341b980 0x5563534238b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60920/0 (socket says 172.21.15.28:60920) 2024-05-12T22:11:19.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.739+0000 7f200be00640 1 -- 172.21.15.28:0/2318406879 learned_addr learned my addr 172.21.15.28:0/2318406879 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:11:19.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.739+0000 7f2011200640 1 --2- 172.21.15.28:0/2318406879 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55635341c2a0 0x556353423df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:19.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.739+0000 7f200be00640 1 --2- 172.21.15.28:0/2318406879 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556353426970 0x556353424330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:19.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.739+0000 7f200be00640 1 -- 172.21.15.28:0/2318406879 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556353426970 msgr2=0x556353424330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:11:19.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.739+0000 7f200be00640 1 --2- 172.21.15.28:0/2318406879 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556353426970 0x556353424330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:19.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.739+0000 7f200be00640 1 -- 172.21.15.28:0/2318406879 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55635341c2a0 msgr2=0x556353423df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:11:19.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.739+0000 7f200be00640 1 --2- 172.21.15.28:0/2318406879 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55635341c2a0 0x556353423df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:19.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.739+0000 7f200be00640 1 -- 172.21.15.28:0/2318406879 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55635315e7d0 con 0x55635341b980 2024-05-12T22:11:19.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.740+0000 7f200be00640 1 --2- 172.21.15.28:0/2318406879 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55635341b980 0x5563534238b0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f1ff8012ab0 tx=0x7f1ff800bbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a85d71887c20307 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:11:19.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.740+0000 7f1ff2a00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-12T22:11:19.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.740+0000 7f1ff2a00640 1 -- 172.21.15.28:0/2318406879 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ff8011bf0 con 0x55635341b980 2024-05-12T22:11:19.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.740+0000 7f1ff2a00640 1 -- 172.21.15.28:0/2318406879 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f1ff8011d90 con 0x55635341b980 2024-05-12T22:11:19.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.740+0000 7f20169c8740 1 -- 172.21.15.28:0/2318406879 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f200000faa0 con 0x55635341b980 2024-05-12T22:11:19.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.740+0000 7f1ff2a00640 1 -- 172.21.15.28:0/2318406879 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f1ff8004020 con 0x55635341b980 2024-05-12T22:11:19.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.740+0000 7f20169c8740 10 client.15729 fetch_fsmap learned FSMap version 25 2024-05-12T22:11:19.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.740+0000 7f20169c8740 1 -- 172.21.15.28:0/2318406879 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x556353435a40 con 0x55635341b980 2024-05-12T22:11:19.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.741+0000 7f1ff2a00640 1 -- 172.21.15.28:0/2318406879 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f200000faa0 con 0x55635341b980 2024-05-12T22:11:19.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.741+0000 7f1ff2a00640 1 -- 172.21.15.28:0/2318406879 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7f1ff8028400 con 0x55635341b980 2024-05-12T22:11:19.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.741+0000 7f20169c8740 10 client.15729 fetch_fsmap finished waiting for FSMap version 25 2024-05-12T22:11:19.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.741+0000 7f20169c8740 10 client.15729 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-05-12T22:11:19.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.741+0000 7f20169c8740 20 client.15729 populate_metadata read hostname 'smithi028' 2024-05-12T22:11:19.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.741+0000 7f20169c8740 1 --2- 172.21.15.28:0/2318406879 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x556353436e50 0x556353439280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:19.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.741+0000 7f20169c8740 4 client.15729 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-12T22:11:19.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.741+0000 7f20169c8740 1 -- 172.21.15.28:0/2318406879 --> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55635338e160 con 0x556353436e50 2024-05-12T22:11:19.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.742+0000 7f2011200640 1 --2- 172.21.15.28:0/2318406879 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x556353436e50 0x556353439280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:19.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.742+0000 7f2011200640 1 --2- 172.21.15.28:0/2318406879 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x556353436e50 0x556353439280 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=69d4de46854808ae server_cookie=8893872778d09f8e in_seq=0 out_seq=0 2024-05-12T22:11:19.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.742+0000 7f1ff2a00640 10 client.15729 ms_handle_connect on v2:172.21.15.28:6840/761183870 2024-05-12T22:11:19.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.743+0000 7f1ff2a00640 1 -- 172.21.15.28:0/2318406879 <== mds.1 v2:172.21.15.28:6840/761183870 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55635338e160 con 0x556353436e50 2024-05-12T22:11:19.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.743+0000 7f1ff2a00640 10 client.15729 handle_command_reply: tid=0 2024-05-12T22:11:19.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.743+0000 7f20169c8740 1 client.15729 shutdown 2024-05-12T22:11:19.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.743+0000 7f20169c8740 1 -- 172.21.15.28:0/2318406879 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55635341b980 msgr2=0x5563534238b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:11:19.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.743+0000 7f20169c8740 1 --2- 172.21.15.28:0/2318406879 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55635341b980 0x5563534238b0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f1ff8012ab0 tx=0x7f1ff800bbb0 comp rx=0 tx=0).stop 2024-05-12T22:11:19.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.743+0000 7f20169c8740 1 -- 172.21.15.28:0/2318406879 shutdown_connections 2024-05-12T22:11:19.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.743+0000 7f20169c8740 1 --2- 172.21.15.28:0/2318406879 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x556353436e50 0x556353439280 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:19.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.744+0000 7f20169c8740 1 --2- 172.21.15.28:0/2318406879 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556353426970 0x556353424330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:19.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.744+0000 7f20169c8740 1 --2- 172.21.15.28:0/2318406879 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55635341c2a0 0x556353423df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:19.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.744+0000 7f1ff2a00640 0 client.15729 ms_handle_reset on v2:172.21.15.28:6840/761183870 2024-05-12T22:11:19.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.744+0000 7f20169c8740 1 --2- 172.21.15.28:0/2318406879 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55635341b980 0x5563534238b0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:19.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.744+0000 7f20169c8740 1 -- 172.21.15.28:0/2318406879 >> 172.21.15.28:0/2318406879 conn(0x556353417ca0 msgr2=0x55635341d9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:11:19.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.744+0000 7f20169c8740 1 -- 172.21.15.28:0/2318406879 shutdown_connections 2024-05-12T22:11:19.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.744+0000 7f20169c8740 1 -- 172.21.15.28:0/2318406879 wait complete. 2024-05-12T22:11:19.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.744+0000 7f20169c8740 20 client.15729 trim_cache size 0 max 16384 2024-05-12T22:11:19.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.760+0000 7f20169c8740 1 Processor -- start 2024-05-12T22:11:19.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.760+0000 7f20169c8740 1 -- start start 2024-05-12T22:11:19.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.760+0000 7f20169c8740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55635341b980 0x556353432650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:19.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.760+0000 7f20169c8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55635341c2a0 0x556353422a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:19.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.760+0000 7f20169c8740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556353426970 0x556353416c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:19.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.760+0000 7f20169c8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f200c1b1fe0 con 0x55635341c2a0 2024-05-12T22:11:19.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.760+0000 7f20169c8740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f200c1af8b0 con 0x55635341b980 2024-05-12T22:11:19.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.760+0000 7f20169c8740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f200c1ad700 con 0x556353426970 2024-05-12T22:11:19.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.761+0000 7f200be00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55635341b980 0x556353432650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:19.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.761+0000 7f200be00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55635341b980 0x556353432650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:41704/0 (socket says 172.21.15.28:41704) 2024-05-12T22:11:19.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.761+0000 7f200be00640 1 -- 172.21.15.28:0/2689389299 learned_addr learned my addr 172.21.15.28:0/2689389299 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:11:19.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.761+0000 7f2011200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55635341c2a0 0x556353422a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:19.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.761+0000 7f200be00640 1 --2- 172.21.15.28:0/2689389299 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556353426970 0x556353416c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:19.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.761+0000 7f2011200640 1 -- 172.21.15.28:0/2689389299 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556353426970 msgr2=0x556353416c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:11:19.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.761+0000 7f2011200640 1 --2- 172.21.15.28:0/2689389299 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556353426970 0x556353416c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:19.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.761+0000 7f2011200640 1 -- 172.21.15.28:0/2689389299 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55635341b980 msgr2=0x556353432650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:11:19.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.761+0000 7f2011200640 1 --2- 172.21.15.28:0/2689389299 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55635341b980 0x556353432650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:19.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.761+0000 7f2011200640 1 -- 172.21.15.28:0/2689389299 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556353413690 con 0x55635341c2a0 2024-05-12T22:11:19.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.762+0000 7f2011200640 1 --2- 172.21.15.28:0/2689389299 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55635341c2a0 0x556353422a10 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f2000000c00 tx=0x7f2000005200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c7a0e82f0b702cb server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:11:19.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.762+0000 7f1ffd600640 1 -- 172.21.15.28:0/2689389299 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2000029070 con 0x55635341c2a0 2024-05-12T22:11:19.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.762+0000 7f1ffd600640 1 -- 172.21.15.28:0/2689389299 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f200002a740 con 0x55635341c2a0 2024-05-12T22:11:19.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.762+0000 7f1ffd600640 1 -- 172.21.15.28:0/2689389299 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f200001e040 con 0x55635341c2a0 2024-05-12T22:11:19.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.763+0000 7f20169c8740 1 -- 172.21.15.28:0/2689389299 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55635341c2a0 msgr2=0x556353422a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:11:19.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.763+0000 7f20169c8740 1 --2- 172.21.15.28:0/2689389299 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55635341c2a0 0x556353422a10 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f2000000c00 tx=0x7f2000005200 comp rx=0 tx=0).stop 2024-05-12T22:11:19.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.763+0000 7f20169c8740 1 -- 172.21.15.28:0/2689389299 shutdown_connections 2024-05-12T22:11:19.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.763+0000 7f20169c8740 1 --2- 172.21.15.28:0/2689389299 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556353426970 0x556353416c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:19.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.763+0000 7f20169c8740 1 --2- 172.21.15.28:0/2689389299 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55635341c2a0 0x556353422a10 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:19.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.763+0000 7f20169c8740 1 --2- 172.21.15.28:0/2689389299 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55635341b980 0x556353432650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:19.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.763+0000 7f20169c8740 1 -- 172.21.15.28:0/2689389299 >> 172.21.15.28:0/2689389299 conn(0x556353417ca0 msgr2=0x556353416530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:11:19.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.763+0000 7f20169c8740 1 -- 172.21.15.28:0/2689389299 shutdown_connections 2024-05-12T22:11:19.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.763+0000 7f20169c8740 1 -- 172.21.15.28:0/2689389299 wait complete. 2024-05-12T22:11:19.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.764+0000 7f20169c8740 1 -- start start 2024-05-12T22:11:19.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.764+0000 7f20169c8740 1 Processor -- start 2024-05-12T22:11:19.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.764+0000 7f20169c8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55635341b980 0x55635342eb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:19.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.764+0000 7f20169c8740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55635341c2a0 0x55635342f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:19.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.764+0000 7f20169c8740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556353426970 0x556353420060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:19.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.764+0000 7f20169c8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f200c1add70 con 0x55635341b980 2024-05-12T22:11:19.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.764+0000 7f20169c8740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f200c1af1e0 con 0x556353426970 2024-05-12T22:11:19.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.764+0000 7f20169c8740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f200c1ae440 con 0x55635341c2a0 2024-05-12T22:11:19.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.764+0000 7f20169c8740 10 client.0 _ll_register_callbacks cb 0x7f1ff8012590 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:11:19.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.765+0000 7f200be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55635341b980 0x55635342eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:19.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.765+0000 7f200be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55635341b980 0x55635342eb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60934/0 (socket says 172.21.15.28:60934) 2024-05-12T22:11:19.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.765+0000 7f200be00640 1 -- 172.21.15.28:0/1051262603 learned_addr learned my addr 172.21.15.28:0/1051262603 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:11:19.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.765+0000 7f2011200640 1 --2- 172.21.15.28:0/1051262603 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55635341c2a0 0x55635342f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:19.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.765+0000 7f200be00640 1 --2- 172.21.15.28:0/1051262603 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556353426970 0x556353420060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:19.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.765+0000 7f200be00640 1 -- 172.21.15.28:0/1051262603 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55635341c2a0 msgr2=0x55635342f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:11:19.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.765+0000 7f200be00640 1 --2- 172.21.15.28:0/1051262603 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55635341c2a0 0x55635342f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:19.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.765+0000 7f200be00640 1 -- 172.21.15.28:0/1051262603 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556353426970 msgr2=0x556353420060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:11:19.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.765+0000 7f200be00640 1 --2- 172.21.15.28:0/1051262603 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556353426970 0x556353420060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:19.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.765+0000 7f200be00640 1 -- 172.21.15.28:0/1051262603 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556353420860 con 0x55635341b980 2024-05-12T22:11:19.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.765+0000 7f2011200640 1 --2- 172.21.15.28:0/1051262603 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55635341c2a0 0x55635342f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:11:19.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.765+0000 7f200be00640 1 --2- 172.21.15.28:0/1051262603 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55635341b980 0x55635342eb60 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f1ff8012ab0 tx=0x7f1ff800b5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6339e1a94d711d6 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:11:19.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.765+0000 7f1ff2a00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-12T22:11:19.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.766+0000 7f1ff2a00640 1 -- 172.21.15.28:0/1051262603 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ff8028c00 con 0x55635341b980 2024-05-12T22:11:19.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.766+0000 7f1ff2a00640 1 -- 172.21.15.28:0/1051262603 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f1ff8011bf0 con 0x55635341b980 2024-05-12T22:11:19.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.766+0000 7f20169c8740 1 -- 172.21.15.28:0/1051262603 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f200002a740 con 0x55635341b980 2024-05-12T22:11:19.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.766+0000 7f1ff2a00640 1 -- 172.21.15.28:0/1051262603 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f1ff8011df0 con 0x55635341b980 2024-05-12T22:11:19.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.766+0000 7f20169c8740 10 client.15735 fetch_fsmap learned FSMap version 25 2024-05-12T22:11:19.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.766+0000 7f20169c8740 1 -- 172.21.15.28:0/1051262603 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55635349cf80 con 0x55635341b980 2024-05-12T22:11:19.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.766+0000 7f1ff2a00640 1 -- 172.21.15.28:0/1051262603 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f200002a740 con 0x55635341b980 2024-05-12T22:11:19.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.767+0000 7f1ff2a00640 1 -- 172.21.15.28:0/1051262603 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7f1ff8034040 con 0x55635341b980 2024-05-12T22:11:19.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.767+0000 7f20169c8740 10 client.15735 fetch_fsmap finished waiting for FSMap version 25 2024-05-12T22:11:19.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.767+0000 7f20169c8740 10 client.15735 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-05-12T22:11:19.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.767+0000 7f20169c8740 20 client.15735 populate_metadata read hostname 'smithi028' 2024-05-12T22:11:19.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.767+0000 7f20169c8740 1 --2- 172.21.15.28:0/1051262603 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x55635349d680 0x55635349faa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:19.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.767+0000 7f20169c8740 4 client.15735 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-12T22:11:19.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.767+0000 7f20169c8740 1 -- 172.21.15.28:0/1051262603 --> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5563533884b0 con 0x55635349d680 2024-05-12T22:11:19.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.767+0000 7f2011200640 1 --2- 172.21.15.28:0/1051262603 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x55635349d680 0x55635349faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:19.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.768+0000 7f2011200640 1 --2- 172.21.15.28:0/1051262603 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x55635349d680 0x55635349faa0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f68957c58ba13e9c server_cookie=a488cce790686761 in_seq=0 out_seq=0 2024-05-12T22:11:19.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.768+0000 7f1ff2a00640 10 client.15735 ms_handle_connect on v2:172.21.15.28:6840/761183870 2024-05-12T22:11:19.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.768+0000 7f1ff2a00640 1 -- 172.21.15.28:0/1051262603 <== mds.1 v2:172.21.15.28:6840/761183870 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5563533884b0 con 0x55635349d680 2024-05-12T22:11:19.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.768+0000 7f1ff2a00640 10 client.15735 handle_command_reply: tid=0 2024-05-12T22:11:19.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.768+0000 7f20169c8740 1 client.15735 shutdown 2024-05-12T22:11:19.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.768+0000 7f20169c8740 1 -- 172.21.15.28:0/1051262603 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55635341b980 msgr2=0x55635342eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:11:19.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.768+0000 7f20169c8740 1 --2- 172.21.15.28:0/1051262603 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55635341b980 0x55635342eb60 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f1ff8012ab0 tx=0x7f1ff800b5c0 comp rx=0 tx=0).stop 2024-05-12T22:11:19.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.768+0000 7f20169c8740 1 -- 172.21.15.28:0/1051262603 shutdown_connections 2024-05-12T22:11:19.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.768+0000 7f20169c8740 1 --2- 172.21.15.28:0/1051262603 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x55635349d680 0x55635349faa0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:19.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.768+0000 7f20169c8740 1 --2- 172.21.15.28:0/1051262603 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556353426970 0x556353420060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:19.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.768+0000 7f20169c8740 1 --2- 172.21.15.28:0/1051262603 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55635341c2a0 0x55635342f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:19.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.768+0000 7f20169c8740 1 --2- 172.21.15.28:0/1051262603 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55635341b980 0x55635342eb60 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:19.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.768+0000 7f1ff2a00640 0 client.15735 ms_handle_reset on v2:172.21.15.28:6840/761183870 2024-05-12T22:11:19.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.768+0000 7f20169c8740 1 -- 172.21.15.28:0/1051262603 >> 172.21.15.28:0/1051262603 conn(0x556353417ca0 msgr2=0x556353423870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:11:19.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.769+0000 7f20169c8740 1 -- 172.21.15.28:0/1051262603 shutdown_connections 2024-05-12T22:11:19.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.769+0000 7f20169c8740 1 -- 172.21.15.28:0/1051262603 wait complete. 2024-05-12T22:11:19.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.769+0000 7f20169c8740 20 client.15735 trim_cache size 0 max 16384 2024-05-12T22:11:19.772 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-05-12T22:11:19.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.769+0000 7f2013c00640 1 -- 172.21.15.28:0/1236002844 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f1ff40a2d80 msgr2=0x7f1ff40a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:11:19.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.769+0000 7f2013c00640 1 --2- 172.21.15.28:0/1236002844 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f1ff40a2d80 0x7f1ff40a5240 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f20040c5400 tx=0x7f20040048c0 comp rx=0 tx=0).stop 2024-05-12T22:11:19.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.769+0000 7f2013c00640 1 -- 172.21.15.28:0/1236002844 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f200c109a00 msgr2=0x7f200c19eb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:11:19.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.769+0000 7f2013c00640 1 --2- 172.21.15.28:0/1236002844 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f200c109a00 0x7f200c19eb20 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f2004006010 tx=0x7f2004003ce0 comp rx=0 tx=0).stop 2024-05-12T22:11:19.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.770+0000 7f2013c00640 1 -- 172.21.15.28:0/1236002844 shutdown_connections 2024-05-12T22:11:19.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.770+0000 7f2013c00640 1 --2- 172.21.15.28:0/1236002844 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f1ff40a2d80 0x7f1ff40a5240 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:19.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.770+0000 7f2013c00640 1 --2- 172.21.15.28:0/1236002844 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f200c109a00 0x7f200c19eb20 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:19.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.770+0000 7f2013c00640 1 --2- 172.21.15.28:0/1236002844 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f200c1043b0 0x7f200c19a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:19.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.770+0000 7f2013c00640 1 --2- 172.21.15.28:0/1236002844 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f200c1039f0 0x7f200c19a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:19.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.770+0000 7f2013c00640 1 -- 172.21.15.28:0/1236002844 >> 172.21.15.28:0/1236002844 conn(0x7f200c0ff4e0 msgr2=0x7f200c0ffc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:11:19.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.770+0000 7f2013c00640 1 -- 172.21.15.28:0/1236002844 shutdown_connections 2024-05-12T22:11:19.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.770+0000 7f2013c00640 1 -- 172.21.15.28:0/1236002844 wait complete. 2024-05-12T22:11:19.788 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-05-12T22:11:19.972 INFO:tasks.workunit.client.0.smithi028.stdout:checking build system type... x86_64-pc-linux-gnu 2024-05-12T22:11:19.972 INFO:tasks.workunit.client.0.smithi028.stdout:checking host system type... x86_64-pc-linux-gnu 2024-05-12T22:11:19.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.973+0000 7facdf800640 1 -- 172.21.15.28:0/571700011 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7facd8107810 msgr2=0x7facd8107bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:11:19.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.973+0000 7facdf800640 1 --2- 172.21.15.28:0/571700011 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7facd8107810 0x7facd8107bf0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7faccc009930 tx=0x7faccc02a420 comp rx=0 tx=0).stop 2024-05-12T22:11:19.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.973+0000 7facdf800640 1 -- 172.21.15.28:0/571700011 shutdown_connections 2024-05-12T22:11:19.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.973+0000 7facdf800640 1 --2- 172.21.15.28:0/571700011 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7facd81021c0 0x7facd810e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:19.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.973+0000 7facdf800640 1 --2- 172.21.15.28:0/571700011 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7facd8101800 0x7facd8101c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:19.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.973+0000 7facdf800640 1 --2- 172.21.15.28:0/571700011 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7facd8107810 0x7facd8107bf0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:19.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.973+0000 7facdf800640 1 -- 172.21.15.28:0/571700011 >> 172.21.15.28:0/571700011 conn(0x7facd8078c80 msgr2=0x7facd8079090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:11:19.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.973+0000 7facdf800640 1 -- 172.21.15.28:0/571700011 shutdown_connections 2024-05-12T22:11:19.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.973+0000 7facdf800640 1 -- 172.21.15.28:0/571700011 wait complete. 2024-05-12T22:11:19.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.974+0000 7facdf800640 1 Processor -- start 2024-05-12T22:11:19.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.974+0000 7facdf800640 1 -- start start 2024-05-12T22:11:19.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.975+0000 7facdf800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7facd8101800 0x7facd819a330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:19.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.975+0000 7facdf800640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7facd81021c0 0x7facd819a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:19.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.975+0000 7facdf800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7facd8107810 0x7facd819ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:19.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.975+0000 7facdf800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7facd8111c30 con 0x7facd8107810 2024-05-12T22:11:19.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.975+0000 7facdf800640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7facd8111ab0 con 0x7facd8101800 2024-05-12T22:11:19.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.975+0000 7facdf800640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7facd8111db0 con 0x7facd81021c0 2024-05-12T22:11:19.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.975+0000 7facd7e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7facd81021c0 0x7facd819a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:19.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.975+0000 7facdd800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7facd8107810 0x7facd819ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:19.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.975+0000 7facd7e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7facd81021c0 0x7facd819a870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:59170/0 (socket says 172.21.15.28:59170) 2024-05-12T22:11:19.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.975+0000 7facd7e00640 1 -- 172.21.15.28:0/3075378123 learned_addr learned my addr 172.21.15.28:0/3075378123 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:11:19.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.975+0000 7facdce00640 1 --2- 172.21.15.28:0/3075378123 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7facd8101800 0x7facd819a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:19.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.975+0000 7facd7e00640 1 -- 172.21.15.28:0/3075378123 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7facd8101800 msgr2=0x7facd819a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:11:19.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.976+0000 7facd7e00640 1 --2- 172.21.15.28:0/3075378123 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7facd8101800 0x7facd819a330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:19.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.976+0000 7facd7e00640 1 -- 172.21.15.28:0/3075378123 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7facd8107810 msgr2=0x7facd819ebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:11:19.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.976+0000 7facd7e00640 1 --2- 172.21.15.28:0/3075378123 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7facd8107810 0x7facd819ebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:19.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.976+0000 7facd7e00640 1 -- 172.21.15.28:0/3075378123 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7facd819f320 con 0x7facd81021c0 2024-05-12T22:11:19.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.976+0000 7facdce00640 1 --2- 172.21.15.28:0/3075378123 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7facd8101800 0x7facd819a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:11:19.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.976+0000 7facdd800640 1 --2- 172.21.15.28:0/3075378123 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7facd8107810 0x7facd819ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:11:19.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.976+0000 7facd7e00640 1 --2- 172.21.15.28:0/3075378123 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7facd81021c0 0x7facd819a870 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7facc400e9f0 tx=0x7facc400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:11:19.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.976+0000 7facd5600640 1 -- 172.21.15.28:0/3075378123 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facc400cd80 con 0x7facd81021c0 2024-05-12T22:11:19.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.976+0000 7facd5600640 1 -- 172.21.15.28:0/3075378123 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7facc4010910 con 0x7facd81021c0 2024-05-12T22:11:19.978 INFO:tasks.workunit.client.0.smithi028.stdout:checking how to print strings... printf 2024-05-12T22:11:19.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.976+0000 7facd5600640 1 -- 172.21.15.28:0/3075378123 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facc4018990 con 0x7facd81021c0 2024-05-12T22:11:19.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.976+0000 7facdf800640 1 -- 172.21.15.28:0/3075378123 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7facd819f610 con 0x7facd81021c0 2024-05-12T22:11:19.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.977+0000 7facdf800640 1 -- 172.21.15.28:0/3075378123 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7facd81a6ea0 con 0x7facd81021c0 2024-05-12T22:11:19.979 INFO:tasks.workunit.client.0.smithi028.stdout:checking for gcc... gcc 2024-05-12T22:11:19.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.980+0000 7face27a0740 1 Processor -- start 2024-05-12T22:11:19.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.980+0000 7face27a0740 1 -- start start 2024-05-12T22:11:19.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.980+0000 7face27a0740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564ce197d980 0x564ce197dd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:19.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.980+0000 7facd5600640 1 -- 172.21.15.28:0/3075378123 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7facc4010ac0 con 0x7facd81021c0 2024-05-12T22:11:19.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.980+0000 7face27a0740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564ce197e2a0 0x564ce1988430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:19.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.981+0000 7face27a0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ce1988970 0x564ce198ad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:19.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.981+0000 7face27a0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x564ce18e2a30 con 0x564ce1988970 2024-05-12T22:11:19.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.981+0000 7face27a0740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x564ce188a700 con 0x564ce197d980 2024-05-12T22:11:19.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.981+0000 7face27a0740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x564ce18de9f0 con 0x564ce197e2a0 2024-05-12T22:11:19.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.981+0000 7facdd800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564ce197d980 0x564ce197dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:19.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.981+0000 7facdd800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564ce197d980 0x564ce197dd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:41744/0 (socket says 172.21.15.28:41744) 2024-05-12T22:11:19.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.981+0000 7facdd800640 1 -- 172.21.15.28:0/1151471066 learned_addr learned my addr 172.21.15.28:0/1151471066 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:11:19.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.981+0000 7facdce00640 1 --2- 172.21.15.28:0/1151471066 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564ce197e2a0 0x564ce1988430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:19.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.981+0000 7facd7e00640 1 --2- 172.21.15.28:0/1151471066 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ce1988970 0x564ce198ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:19.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.981+0000 7facdd800640 1 -- 172.21.15.28:0/1151471066 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564ce197e2a0 msgr2=0x564ce1988430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:11:19.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.981+0000 7facdd800640 1 --2- 172.21.15.28:0/1151471066 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564ce197e2a0 0x564ce1988430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:19.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.981+0000 7facdd800640 1 -- 172.21.15.28:0/1151471066 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ce1988970 msgr2=0x564ce198ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:11:19.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.981+0000 7facdd800640 1 --2- 172.21.15.28:0/1151471066 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ce1988970 0x564ce198ad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:19.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.981+0000 7facdd800640 1 -- 172.21.15.28:0/1151471066 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564ce15f63f0 con 0x564ce197d980 2024-05-12T22:11:19.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.981+0000 7facd7e00640 1 --2- 172.21.15.28:0/1151471066 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ce1988970 0x564ce198ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:11:19.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.982+0000 7facdd800640 1 --2- 172.21.15.28:0/1151471066 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564ce197d980 0x564ce197dd60 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7facd0011790 tx=0x7facd0011c50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=409b55aa29be0a02 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:11:19.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.982+0000 7facca000640 1 -- 172.21.15.28:0/1151471066 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facd0003c30 con 0x564ce197d980 2024-05-12T22:11:19.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.982+0000 7facca000640 1 -- 172.21.15.28:0/1151471066 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7facd000fe80 con 0x564ce197d980 2024-05-12T22:11:19.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.982+0000 7facca000640 1 -- 172.21.15.28:0/1151471066 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facd00193e0 con 0x564ce197d980 2024-05-12T22:11:19.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.982+0000 7facd5600640 1 --2- 172.21.15.28:0/3075378123 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7facc00a2d80 0x7facc00a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:19.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.982+0000 7facd5600640 1 -- 172.21.15.28:0/3075378123 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7facc40cf6f0 con 0x7facd81021c0 2024-05-12T22:11:19.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.983+0000 7facdce00640 1 --2- 172.21.15.28:0/3075378123 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7facc00a2d80 0x7facc00a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:19.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.983+0000 7face27a0740 1 -- 172.21.15.28:0/1151471066 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564ce197d980 msgr2=0x564ce197dd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:11:19.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.983+0000 7face27a0740 1 --2- 172.21.15.28:0/1151471066 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564ce197d980 0x564ce197dd60 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7facd0011790 tx=0x7facd0011c50 comp rx=0 tx=0).stop 2024-05-12T22:11:19.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.983+0000 7face27a0740 1 -- 172.21.15.28:0/1151471066 shutdown_connections 2024-05-12T22:11:19.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.983+0000 7face27a0740 1 --2- 172.21.15.28:0/1151471066 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ce1988970 0x564ce198ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:19.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.983+0000 7face27a0740 1 --2- 172.21.15.28:0/1151471066 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564ce197e2a0 0x564ce1988430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:19.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.983+0000 7face27a0740 1 --2- 172.21.15.28:0/1151471066 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564ce197d980 0x564ce197dd60 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:19.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.983+0000 7face27a0740 1 -- 172.21.15.28:0/1151471066 >> 172.21.15.28:0/1151471066 conn(0x564ce1979ca0 msgr2=0x564ce197a080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:11:19.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.983+0000 7face27a0740 1 -- 172.21.15.28:0/1151471066 shutdown_connections 2024-05-12T22:11:19.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.983+0000 7face27a0740 1 -- 172.21.15.28:0/1151471066 wait complete. 2024-05-12T22:11:19.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.984+0000 7facdce00640 1 --2- 172.21.15.28:0/3075378123 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7facc00a2d80 0x7facc00a5240 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7faccc006010 tx=0x7faccc031510 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:11:19.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.984+0000 7face27a0740 1 -- start start 2024-05-12T22:11:19.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.984+0000 7face27a0740 1 Processor -- start 2024-05-12T22:11:19.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.984+0000 7face27a0740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564ce197d980 0x564ce19858e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:19.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.984+0000 7face27a0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ce197e2a0 0x564ce1985e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:19.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.985+0000 7face27a0740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564ce1988970 0x564ce1986360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:19.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.985+0000 7face27a0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x564ce198b760 con 0x564ce197e2a0 2024-05-12T22:11:19.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.985+0000 7face27a0740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x564ce189da70 con 0x564ce197d980 2024-05-12T22:11:19.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.985+0000 7face27a0740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x564ce198b8e0 con 0x564ce1988970 2024-05-12T22:11:19.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.985+0000 7face27a0740 10 client.0 _ll_register_callbacks cb 0x564ce1880ba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:11:19.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.985+0000 7facd7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ce197e2a0 0x564ce1985e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:19.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.985+0000 7facdce00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564ce197d980 0x564ce19858e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:19.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.985+0000 7facdd800640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564ce1988970 0x564ce1986360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:19.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.985+0000 7facd7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ce197e2a0 0x564ce1985e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60954/0 (socket says 172.21.15.28:60954) 2024-05-12T22:11:19.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.985+0000 7facd7e00640 1 -- 172.21.15.28:0/621817719 learned_addr learned my addr 172.21.15.28:0/621817719 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:11:19.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.985+0000 7facd7e00640 1 -- 172.21.15.28:0/621817719 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564ce1988970 msgr2=0x564ce1986360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:11:19.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.985+0000 7facd7e00640 1 --2- 172.21.15.28:0/621817719 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564ce1988970 0x564ce1986360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:19.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.985+0000 7facd7e00640 1 -- 172.21.15.28:0/621817719 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564ce197d980 msgr2=0x564ce19858e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:11:19.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.985+0000 7facd7e00640 1 --2- 172.21.15.28:0/621817719 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564ce197d980 0x564ce19858e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:19.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.985+0000 7facd7e00640 1 -- 172.21.15.28:0/621817719 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564ce16c07d0 con 0x564ce197e2a0 2024-05-12T22:11:19.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.985+0000 7facdd800640 1 --2- 172.21.15.28:0/621817719 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564ce1988970 0x564ce1986360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:11:19.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.985+0000 7facdce00640 1 --2- 172.21.15.28:0/621817719 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564ce197d980 0x564ce19858e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:11:19.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.986+0000 7facd7e00640 1 --2- 172.21.15.28:0/621817719 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ce197e2a0 0x564ce1985e20 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7facc400e9c0 tx=0x7facc409bb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56dc487cd4c64ad4 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:11:19.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.986+0000 7facbea00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-12T22:11:19.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.986+0000 7facbea00640 1 -- 172.21.15.28:0/621817719 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facc40cd2a0 con 0x564ce197e2a0 2024-05-12T22:11:19.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.986+0000 7facbea00640 1 -- 172.21.15.28:0/621817719 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7facc4018b30 con 0x564ce197e2a0 2024-05-12T22:11:19.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.986+0000 7face27a0740 1 -- 172.21.15.28:0/621817719 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x564ce18f0160 con 0x564ce197e2a0 2024-05-12T22:11:19.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.986+0000 7facbea00640 1 -- 172.21.15.28:0/621817719 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7facc409bc90 con 0x564ce197e2a0 2024-05-12T22:11:19.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.986+0000 7face27a0740 10 client.15747 fetch_fsmap learned FSMap version 25 2024-05-12T22:11:19.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.986+0000 7face27a0740 1 -- 172.21.15.28:0/621817719 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x564ce1997a40 con 0x564ce197e2a0 2024-05-12T22:11:19.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.987+0000 7facbea00640 1 -- 172.21.15.28:0/621817719 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x564ce18f0160 con 0x564ce197e2a0 2024-05-12T22:11:19.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.987+0000 7facbea00640 1 -- 172.21.15.28:0/621817719 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7facc40cfae0 con 0x564ce197e2a0 2024-05-12T22:11:19.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.987+0000 7face27a0740 10 client.15747 fetch_fsmap finished waiting for FSMap version 25 2024-05-12T22:11:19.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.987+0000 7face27a0740 10 client.15747 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:11:19.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.987+0000 7face27a0740 20 client.15747 populate_metadata read hostname 'smithi028' 2024-05-12T22:11:19.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.987+0000 7face27a0740 1 --2- 172.21.15.28:0/621817719 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x564ce1998e50 0x564ce199b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:19.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.987+0000 7face27a0740 4 client.15747 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-12T22:11:19.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.987+0000 7face27a0740 1 -- 172.21.15.28:0/621817719 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564ce18ea4b0 con 0x564ce1998e50 2024-05-12T22:11:19.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.987+0000 7facdce00640 1 --2- 172.21.15.28:0/621817719 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x564ce1998e50 0x564ce199b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:19.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.988+0000 7facdce00640 1 --2- 172.21.15.28:0/621817719 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x564ce1998e50 0x564ce199b2a0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d166a85e0fd82c74 server_cookie=31bba441654e87f in_seq=0 out_seq=0 2024-05-12T22:11:19.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.988+0000 7facbea00640 10 client.15747 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:11:19.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.989+0000 7facbea00640 1 -- 172.21.15.28:0/621817719 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564ce18ea4b0 con 0x564ce1998e50 2024-05-12T22:11:19.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.989+0000 7facbea00640 10 client.15747 handle_command_reply: tid=0 2024-05-12T22:11:19.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.989+0000 7face27a0740 1 client.15747 shutdown 2024-05-12T22:11:19.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.989+0000 7face27a0740 1 -- 172.21.15.28:0/621817719 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ce197e2a0 msgr2=0x564ce1985e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:11:19.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.989+0000 7face27a0740 1 --2- 172.21.15.28:0/621817719 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ce197e2a0 0x564ce1985e20 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7facc400e9c0 tx=0x7facc409bb50 comp rx=0 tx=0).stop 2024-05-12T22:11:19.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.989+0000 7face27a0740 1 -- 172.21.15.28:0/621817719 shutdown_connections 2024-05-12T22:11:19.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.989+0000 7face27a0740 1 --2- 172.21.15.28:0/621817719 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x564ce1998e50 0x564ce199b2a0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:19.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.989+0000 7face27a0740 1 --2- 172.21.15.28:0/621817719 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564ce1988970 0x564ce1986360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:19.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.989+0000 7face27a0740 1 --2- 172.21.15.28:0/621817719 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ce197e2a0 0x564ce1985e20 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:19.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.989+0000 7face27a0740 1 --2- 172.21.15.28:0/621817719 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564ce197d980 0x564ce19858e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:19.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.989+0000 7facbea00640 0 client.15747 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:11:19.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.989+0000 7face27a0740 1 -- 172.21.15.28:0/621817719 >> 172.21.15.28:0/621817719 conn(0x564ce1979ca0 msgr2=0x564ce197faa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:11:19.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.990+0000 7face27a0740 1 -- 172.21.15.28:0/621817719 shutdown_connections 2024-05-12T22:11:19.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.990+0000 7face27a0740 1 -- 172.21.15.28:0/621817719 wait complete. 2024-05-12T22:11:19.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:19.990+0000 7face27a0740 20 client.15747 trim_cache size 0 max 16384 2024-05-12T22:11:20.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:20.005+0000 7face27a0740 1 Processor -- start 2024-05-12T22:11:20.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:20.005+0000 7face27a0740 1 -- start start 2024-05-12T22:11:20.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:20.006+0000 7face27a0740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564ce197d980 0x564ce1994620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:20.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:20.006+0000 7face27a0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ce197e2a0 0x564ce1978c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:20.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:20.006+0000 7face27a0740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564ce1988970 0x564ce19a0b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:20.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:20.006+0000 7face27a0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7facd81b1ff0 con 0x564ce197e2a0 2024-05-12T22:11:20.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:20.006+0000 7face27a0740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7facd81af8c0 con 0x564ce197d980 2024-05-12T22:11:20.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:20.006+0000 7face27a0740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7facd81ad710 con 0x564ce1988970 2024-05-12T22:11:20.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:20.006+0000 7facdce00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564ce197d980 0x564ce1994620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:20.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:20.006+0000 7facdce00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564ce197d980 0x564ce1994620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:41766/0 (socket says 172.21.15.28:41766) 2024-05-12T22:11:20.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:20.006+0000 7facd7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ce197e2a0 0x564ce1978c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:20.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:20.006+0000 7facdce00640 1 -- 172.21.15.28:0/811627928 learned_addr learned my addr 172.21.15.28:0/811627928 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:11:20.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:20.006+0000 7facdd800640 1 --2- 172.21.15.28:0/811627928 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564ce1988970 0x564ce19a0b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:20.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:20.006+0000 7facdce00640 1 -- 172.21.15.28:0/811627928 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564ce1988970 msgr2=0x564ce19a0b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:11:20.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:20.007+0000 7facdce00640 1 --2- 172.21.15.28:0/811627928 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564ce1988970 0x564ce19a0b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:20.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:20.007+0000 7facdce00640 1 -- 172.21.15.28:0/811627928 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ce197e2a0 msgr2=0x564ce1978c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:11:20.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:20.007+0000 7facdce00640 1 --2- 172.21.15.28:0/811627928 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ce197e2a0 0x564ce1978c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:20.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:20.007+0000 7facdce00640 1 -- 172.21.15.28:0/811627928 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564ce1983e50 con 0x564ce197d980 2024-05-12T22:11:20.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:20.007+0000 7facdce00640 1 --2- 172.21.15.28:0/811627928 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564ce197d980 0x564ce1994620 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7faccc02aa10 tx=0x7faccc03f040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f7c386e02c8daead server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:11:20.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:20.007+0000 7facc9600640 1 -- 172.21.15.28:0/811627928 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faccc05a070 con 0x564ce197d980 2024-05-12T22:11:20.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:20.007+0000 7facc9600640 1 -- 172.21.15.28:0/811627928 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faccc004c30 con 0x564ce197d980 2024-05-12T22:11:20.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:20.007+0000 7facc9600640 1 -- 172.21.15.28:0/811627928 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faccc035040 con 0x564ce197d980 2024-05-12T22:11:20.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:20.008+0000 7face27a0740 1 -- 172.21.15.28:0/811627928 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564ce197d980 msgr2=0x564ce1994620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:11:20.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:20.008+0000 7face27a0740 1 --2- 172.21.15.28:0/811627928 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564ce197d980 0x564ce1994620 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7faccc02aa10 tx=0x7faccc03f040 comp rx=0 tx=0).stop 2024-05-12T22:11:20.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:20.008+0000 7face27a0740 1 -- 172.21.15.28:0/811627928 shutdown_connections 2024-05-12T22:11:20.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:20.008+0000 7face27a0740 1 --2- 172.21.15.28:0/811627928 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564ce1988970 0x564ce19a0b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:20.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:20.008+0000 7face27a0740 1 --2- 172.21.15.28:0/811627928 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ce197e2a0 0x564ce1978c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:20.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:20.008+0000 7face27a0740 1 --2- 172.21.15.28:0/811627928 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564ce197d980 0x564ce1994620 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:20.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:20.008+0000 7face27a0740 1 -- 172.21.15.28:0/811627928 >> 172.21.15.28:0/811627928 conn(0x564ce1979ca0 msgr2=0x564ce1978530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:11:20.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:20.008+0000 7face27a0740 1 -- 172.21.15.28:0/811627928 shutdown_connections 2024-05-12T22:11:20.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:20.008+0000 7face27a0740 1 -- 172.21.15.28:0/811627928 wait complete. 2024-05-12T22:11:20.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:20.009+0000 7face27a0740 1 -- start start 2024-05-12T22:11:20.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:20.009+0000 7face27a0740 1 Processor -- start 2024-05-12T22:11:20.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:20.009+0000 7face27a0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ce197d980 0x564ce1982480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:20.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:20.009+0000 7face27a0740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564ce197e2a0 0x564ce19829c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:20.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:20.009+0000 7face27a0740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564ce1988970 0x564ce1982f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:20.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:20.009+0000 7face27a0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7facd81add80 con 0x564ce197d980 2024-05-12T22:11:20.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:20.009+0000 7face27a0740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7facd81af1f0 con 0x564ce197e2a0 2024-05-12T22:11:20.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:20.009+0000 7face27a0740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7facd81ae450 con 0x564ce1988970 2024-05-12T22:11:20.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:20.009+0000 7face27a0740 10 client.0 _ll_register_callbacks cb 0x7facc4098020 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:11:20.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:20.010+0000 7facd7e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564ce197e2a0 0x564ce19829c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:20.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:20.010+0000 7facdce00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ce197d980 0x564ce1982480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:20.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:20.010+0000 7facd7e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564ce197e2a0 0x564ce19829c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:41770/0 (socket says 172.21.15.28:41770) 2024-05-12T22:11:20.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:20.010+0000 7facdce00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ce197d980 0x564ce1982480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60972/0 (socket says 172.21.15.28:60972) 2024-05-12T22:11:20.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:20.010+0000 7facd7e00640 1 -- 172.21.15.28:0/3184447693 learned_addr learned my addr 172.21.15.28:0/3184447693 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:11:20.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:20.010+0000 7facdd800640 1 --2- 172.21.15.28:0/3184447693 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564ce1988970 0x564ce1982f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:20.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:20.010+0000 7facd7e00640 1 -- 172.21.15.28:0/3184447693 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564ce1988970 msgr2=0x564ce1982f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:11:20.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:20.010+0000 7facd7e00640 1 --2- 172.21.15.28:0/3184447693 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564ce1988970 0x564ce1982f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:20.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:20.010+0000 7facd7e00640 1 -- 172.21.15.28:0/3184447693 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ce197d980 msgr2=0x564ce1982480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:11:20.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:20.010+0000 7facd7e00640 1 --2- 172.21.15.28:0/3184447693 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ce197d980 0x564ce1982480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:20.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:20.010+0000 7facd7e00640 1 -- 172.21.15.28:0/3184447693 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564ce19fe3e0 con 0x564ce197e2a0 2024-05-12T22:11:20.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:20.010+0000 7facdce00640 1 --2- 172.21.15.28:0/3184447693 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ce197d980 0x564ce1982480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:11:20.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:20.010+0000 7facdd800640 1 --2- 172.21.15.28:0/3184447693 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564ce1988970 0x564ce1982f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:11:20.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:20.010+0000 7facd7e00640 1 --2- 172.21.15.28:0/3184447693 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564ce197e2a0 0x564ce19829c0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7facc409bc20 tx=0x7facc4018dc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e284512fafce80b7 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:11:20.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:20.010+0000 7facbea00640 10 client.0 ms_handle_connect on v2:172.21.15.86:3300/0 2024-05-12T22:11:20.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:20.011+0000 7facbea00640 1 -- 172.21.15.28:0/3184447693 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facc40cd2a0 con 0x564ce197e2a0 2024-05-12T22:11:20.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:20.011+0000 7facbea00640 1 -- 172.21.15.28:0/3184447693 <== mon.2 v2:172.21.15.86:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7facc409b2f0 con 0x564ce197e2a0 2024-05-12T22:11:20.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:20.011+0000 7face27a0740 1 -- 172.21.15.28:0/3184447693 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faccc004c30 con 0x564ce197e2a0 2024-05-12T22:11:20.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:20.011+0000 7facbea00640 1 -- 172.21.15.28:0/3184447693 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7facc409b5f0 con 0x564ce197e2a0 2024-05-12T22:11:20.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:20.011+0000 7face27a0740 10 client.25871 fetch_fsmap learned FSMap version 25 2024-05-12T22:11:20.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:20.011+0000 7face27a0740 1 -- 172.21.15.28:0/3184447693 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x564ce19ff650 con 0x564ce197e2a0 2024-05-12T22:11:20.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:20.011+0000 7facbea00640 1 -- 172.21.15.28:0/3184447693 <== mon.2 v2:172.21.15.86:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faccc004c30 con 0x564ce197e2a0 2024-05-12T22:11:20.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:20.011+0000 7facbea00640 1 -- 172.21.15.28:0/3184447693 <== mon.2 v2:172.21.15.86:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7facc409ace0 con 0x564ce197e2a0 2024-05-12T22:11:20.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:20.011+0000 7face27a0740 10 client.25871 fetch_fsmap finished waiting for FSMap version 25 2024-05-12T22:11:20.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:20.011+0000 7face27a0740 10 client.25871 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:11:20.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:20.011+0000 7face27a0740 20 client.25871 populate_metadata read hostname 'smithi028' 2024-05-12T22:11:20.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:20.012+0000 7face27a0740 1 --2- 172.21.15.28:0/3184447693 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x564ce19ffc20 0x564ce1a02040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:20.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:20.012+0000 7face27a0740 4 client.25871 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-12T22:11:20.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:20.012+0000 7face27a0740 1 -- 172.21.15.28:0/3184447693 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564ce18676e0 con 0x564ce19ffc20 2024-05-12T22:11:20.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:20.012+0000 7facdce00640 1 --2- 172.21.15.28:0/3184447693 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x564ce19ffc20 0x564ce1a02040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:20.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:20.012+0000 7facdce00640 1 --2- 172.21.15.28:0/3184447693 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x564ce19ffc20 0x564ce1a02040 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=194549d842d17da8 server_cookie=dcf7ab939c67c7ee in_seq=0 out_seq=0 2024-05-12T22:11:20.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:20.012+0000 7facbea00640 10 client.25871 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:11:20.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:20.013+0000 7facbea00640 1 -- 172.21.15.28:0/3184447693 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564ce18676e0 con 0x564ce19ffc20 2024-05-12T22:11:20.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:20.013+0000 7facbea00640 10 client.25871 handle_command_reply: tid=0 2024-05-12T22:11:20.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:20.013+0000 7face27a0740 1 client.25871 shutdown 2024-05-12T22:11:20.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:20.013+0000 7face27a0740 1 -- 172.21.15.28:0/3184447693 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564ce197e2a0 msgr2=0x564ce19829c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:11:20.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:20.013+0000 7face27a0740 1 --2- 172.21.15.28:0/3184447693 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564ce197e2a0 0x564ce19829c0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7facc409bc20 tx=0x7facc4018dc0 comp rx=0 tx=0).stop 2024-05-12T22:11:20.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:20.013+0000 7face27a0740 1 -- 172.21.15.28:0/3184447693 shutdown_connections 2024-05-12T22:11:20.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:20.013+0000 7face27a0740 1 --2- 172.21.15.28:0/3184447693 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x564ce19ffc20 0x564ce1a02040 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:20.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:20.013+0000 7face27a0740 1 --2- 172.21.15.28:0/3184447693 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564ce1988970 0x564ce1982f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:20.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:20.013+0000 7face27a0740 1 --2- 172.21.15.28:0/3184447693 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564ce197e2a0 0x564ce19829c0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:20.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:20.013+0000 7face27a0740 1 --2- 172.21.15.28:0/3184447693 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ce197d980 0x564ce1982480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:20.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:20.013+0000 7facbea00640 0 client.25871 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:11:20.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:20.013+0000 7face27a0740 1 -- 172.21.15.28:0/3184447693 >> 172.21.15.28:0/3184447693 conn(0x564ce1979ca0 msgr2=0x564ce1986130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:11:20.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:20.014+0000 7face27a0740 1 -- 172.21.15.28:0/3184447693 shutdown_connections 2024-05-12T22:11:20.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:20.014+0000 7face27a0740 1 -- 172.21.15.28:0/3184447693 wait complete. 2024-05-12T22:11:20.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:20.014+0000 7face27a0740 20 client.25871 trim_cache size 0 max 16384 2024-05-12T22:11:20.018 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-05-12T22:11:20.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:20.014+0000 7facdf800640 1 -- 172.21.15.28:0/3075378123 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7facc00a2d80 msgr2=0x7facc00a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:11:20.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:20.014+0000 7facdf800640 1 --2- 172.21.15.28:0/3075378123 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7facc00a2d80 0x7facc00a5240 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7faccc006010 tx=0x7faccc031510 comp rx=0 tx=0).stop 2024-05-12T22:11:20.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:20.014+0000 7facdf800640 1 -- 172.21.15.28:0/3075378123 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7facd81021c0 msgr2=0x7facd819a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:11:20.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:20.014+0000 7facdf800640 1 --2- 172.21.15.28:0/3075378123 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7facd81021c0 0x7facd819a870 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7facc400e9f0 tx=0x7facc400eeb0 comp rx=0 tx=0).stop 2024-05-12T22:11:20.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:20.015+0000 7facdf800640 1 -- 172.21.15.28:0/3075378123 shutdown_connections 2024-05-12T22:11:20.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:20.015+0000 7facdf800640 1 --2- 172.21.15.28:0/3075378123 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7facc00a2d80 0x7facc00a5240 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:20.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:20.015+0000 7facdf800640 1 --2- 172.21.15.28:0/3075378123 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7facd8107810 0x7facd819ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:20.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:20.015+0000 7facdf800640 1 --2- 172.21.15.28:0/3075378123 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7facd81021c0 0x7facd819a870 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:20.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:20.015+0000 7facdf800640 1 --2- 172.21.15.28:0/3075378123 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7facd8101800 0x7facd819a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:20.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:20.015+0000 7facdf800640 1 -- 172.21.15.28:0/3075378123 >> 172.21.15.28:0/3075378123 conn(0x7facd8078c80 msgr2=0x7facd810cf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:11:20.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:20.015+0000 7facdf800640 1 -- 172.21.15.28:0/3075378123 shutdown_connections 2024-05-12T22:11:20.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:20.015+0000 7facdf800640 1 -- 172.21.15.28:0/3075378123 wait complete. 2024-05-12T22:11:20.126 INFO:tasks.workunit.client.0.smithi028.stdout:checking whether the C compiler works... yes 2024-05-12T22:11:20.126 INFO:tasks.workunit.client.0.smithi028.stdout:checking for C compiler default output file name... a.out 2024-05-12T22:11:20.213 INFO:tasks.workunit.client.0.smithi028.stdout:checking for suffix of executables... 2024-05-12T22:11:20.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:11:19 smithi086 ceph-mon[34297]: pgmap v73: 97 pgs: 97 active+clean; 69 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.3 MiB/s wr, 154 op/s 2024-05-12T22:11:20.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:11:19 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/1691108371' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:11:20.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:11:19 smithi028 ceph-mon[29847]: pgmap v73: 97 pgs: 97 active+clean; 69 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.3 MiB/s wr, 154 op/s 2024-05-12T22:11:20.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:11:19 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/1691108371' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:11:20.306 INFO:tasks.workunit.client.0.smithi028.stdout:checking whether we are cross compiling... no 2024-05-12T22:11:20.383 INFO:tasks.workunit.client.0.smithi028.stdout:checking for suffix of object files... o 2024-05-12T22:11:20.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:11:19 smithi186 ceph-mon[34785]: pgmap v73: 97 pgs: 97 active+clean; 69 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.3 MiB/s wr, 154 op/s 2024-05-12T22:11:20.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:11:19 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/1691108371' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:11:20.444 INFO:tasks.workunit.client.0.smithi028.stdout:checking whether we are using the GNU C compiler... yes 2024-05-12T22:11:20.506 INFO:tasks.workunit.client.0.smithi028.stdout:checking whether gcc accepts -g... yes 2024-05-12T22:11:20.643 INFO:tasks.workunit.client.0.smithi028.stdout:checking for gcc option to accept ISO C89... none needed 2024-05-12T22:11:20.656 INFO:tasks.workunit.client.0.smithi028.stdout:checking for a sed that does not truncate output... /usr/bin/sed 2024-05-12T22:11:20.666 INFO:tasks.workunit.client.0.smithi028.stdout:checking for grep that handles long lines and -e... /usr/bin/grep 2024-05-12T22:11:20.670 INFO:tasks.workunit.client.0.smithi028.stdout:checking for egrep... /usr/bin/grep -E 2024-05-12T22:11:20.674 INFO:tasks.workunit.client.0.smithi028.stdout:checking for fgrep... /usr/bin/grep -F 2024-05-12T22:11:20.686 INFO:tasks.workunit.client.0.smithi028.stdout:checking for ld used by gcc... /usr/bin/ld 2024-05-12T22:11:20.693 INFO:tasks.workunit.client.0.smithi028.stdout:checking if the linker (/usr/bin/ld) is GNU ld... yes 2024-05-12T22:11:20.702 INFO:tasks.workunit.client.0.smithi028.stdout:checking for BSD- or MS-compatible name lister (nm)... /usr/bin/nm -B 2024-05-12T22:11:21.034 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-12T22:11:21.034 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-12T22:11:21.034 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-05-12T22:11:21.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.268+0000 7fdc72e00640 1 -- 172.21.15.28:0/3306658138 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc6c10e1c0 msgr2=0x7fdc6c1105b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:11:21.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.268+0000 7fdc72e00640 1 --2- 172.21.15.28:0/3306658138 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc6c10e1c0 0x7fdc6c1105b0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fdc64009ca0 tx=0x7fdc6402a8b0 comp rx=0 tx=0).stop 2024-05-12T22:11:21.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.268+0000 7fdc72e00640 1 -- 172.21.15.28:0/3306658138 shutdown_connections 2024-05-12T22:11:21.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.268+0000 7fdc72e00640 1 --2- 172.21.15.28:0/3306658138 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc6c10e1c0 0x7fdc6c1105b0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:21.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.268+0000 7fdc72e00640 1 --2- 172.21.15.28:0/3306658138 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdc6c100f30 0x7fdc6c10dc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:21.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.268+0000 7fdc72e00640 1 --2- 172.21.15.28:0/3306658138 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdc6c100610 0x7fdc6c1009f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:21.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.268+0000 7fdc72e00640 1 -- 172.21.15.28:0/3306658138 >> 172.21.15.28:0/3306658138 conn(0x7fdc6c076ab0 msgr2=0x7fdc6c076ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:11:21.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.268+0000 7fdc72e00640 1 -- 172.21.15.28:0/3306658138 shutdown_connections 2024-05-12T22:11:21.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.268+0000 7fdc72e00640 1 -- 172.21.15.28:0/3306658138 wait complete. 2024-05-12T22:11:21.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.269+0000 7fdc72e00640 1 Processor -- start 2024-05-12T22:11:21.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.269+0000 7fdc72e00640 1 -- start start 2024-05-12T22:11:21.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.270+0000 7fdc72e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc6c100610 0x7fdc6c19a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:21.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.270+0000 7fdc72e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdc6c100f30 0x7fdc6c19a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:21.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.270+0000 7fdc72e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdc6c10e1c0 0x7fdc6c19eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:21.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.270+0000 7fdc72e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdc6c113e70 con 0x7fdc6c100610 2024-05-12T22:11:21.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.270+0000 7fdc72e00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fdc6c113cf0 con 0x7fdc6c10e1c0 2024-05-12T22:11:21.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.270+0000 7fdc72e00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fdc6c113ff0 con 0x7fdc6c100f30 2024-05-12T22:11:21.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.270+0000 7fdc6be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc6c100610 0x7fdc6c19a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:21.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.270+0000 7fdc70e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdc6c10e1c0 0x7fdc6c19eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:21.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.270+0000 7fdc6be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc6c100610 0x7fdc6c19a2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:32772/0 (socket says 172.21.15.28:32772) 2024-05-12T22:11:21.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.270+0000 7fdc6be00640 1 -- 172.21.15.28:0/3045644668 learned_addr learned my addr 172.21.15.28:0/3045644668 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:11:21.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.270+0000 7fdc6b400640 1 --2- 172.21.15.28:0/3045644668 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdc6c100f30 0x7fdc6c19a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:21.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.271+0000 7fdc70e00640 1 -- 172.21.15.28:0/3045644668 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdc6c100f30 msgr2=0x7fdc6c19a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:11:21.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.271+0000 7fdc70e00640 1 --2- 172.21.15.28:0/3045644668 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdc6c100f30 0x7fdc6c19a830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:21.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.271+0000 7fdc70e00640 1 -- 172.21.15.28:0/3045644668 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc6c100610 msgr2=0x7fdc6c19a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:11:21.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.271+0000 7fdc70e00640 1 --2- 172.21.15.28:0/3045644668 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc6c100610 0x7fdc6c19a2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:21.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.271+0000 7fdc70e00640 1 -- 172.21.15.28:0/3045644668 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc6c19f290 con 0x7fdc6c10e1c0 2024-05-12T22:11:21.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.271+0000 7fdc70e00640 1 --2- 172.21.15.28:0/3045644668 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdc6c10e1c0 0x7fdc6c19eb60 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fdc6402aea0 tx=0x7fdc64003f10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:11:21.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.271+0000 7fdc68c00640 1 -- 172.21.15.28:0/3045644668 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc64030dc0 con 0x7fdc6c10e1c0 2024-05-12T22:11:21.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.271+0000 7fdc68c00640 1 -- 172.21.15.28:0/3045644668 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdc64040070 con 0x7fdc6c10e1c0 2024-05-12T22:11:21.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.272+0000 7fdc68c00640 1 -- 172.21.15.28:0/3045644668 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc64031a80 con 0x7fdc6c10e1c0 2024-05-12T22:11:21.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.272+0000 7fdc72e00640 1 -- 172.21.15.28:0/3045644668 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc6c19f520 con 0x7fdc6c10e1c0 2024-05-12T22:11:21.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.272+0000 7fdc72e00640 1 -- 172.21.15.28:0/3045644668 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc6c1a6e10 con 0x7fdc6c10e1c0 2024-05-12T22:11:21.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.275+0000 7fdc75c0e740 1 Processor -- start 2024-05-12T22:11:21.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.275+0000 7fdc68c00640 1 -- 172.21.15.28:0/3045644668 <== mon.2 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7fdc64031c20 con 0x7fdc6c10e1c0 2024-05-12T22:11:21.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.276+0000 7fdc75c0e740 1 -- start start 2024-05-12T22:11:21.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.276+0000 7fdc75c0e740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cd4e3be2c0 0x55cd4e3be6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:21.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.276+0000 7fdc75c0e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd4e3bec70 0x55cd4e3c8d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:21.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.276+0000 7fdc75c0e740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cd4e3c92e0 0x55cd4e3cb730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:21.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.276+0000 7fdc75c0e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55cd4e3227c0 con 0x55cd4e3bec70 2024-05-12T22:11:21.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.276+0000 7fdc75c0e740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x55cd4e2ca790 con 0x55cd4e3c92e0 2024-05-12T22:11:21.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.276+0000 7fdc75c0e740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55cd4e31e780 con 0x55cd4e3be2c0 2024-05-12T22:11:21.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.276+0000 7fdc6be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd4e3bec70 0x55cd4e3c8d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:21.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.277+0000 7fdc6b400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cd4e3be2c0 0x55cd4e3be6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:21.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.277+0000 7fdc6be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd4e3bec70 0x55cd4e3c8d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:32780/0 (socket says 172.21.15.28:32780) 2024-05-12T22:11:21.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.277+0000 7fdc6be00640 1 -- 172.21.15.28:0/4153428719 learned_addr learned my addr 172.21.15.28:0/4153428719 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:11:21.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.277+0000 7fdc6b400640 1 --2- 172.21.15.28:0/4153428719 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cd4e3c92e0 0x55cd4e3cb730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:21.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.277+0000 7fdc6be00640 1 -- 172.21.15.28:0/4153428719 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cd4e3be2c0 msgr2=0x55cd4e3be6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:11:21.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.277+0000 7fdc6be00640 1 --2- 172.21.15.28:0/4153428719 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cd4e3be2c0 0x55cd4e3be6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:21.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.277+0000 7fdc6be00640 1 -- 172.21.15.28:0/4153428719 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cd4e3c92e0 msgr2=0x55cd4e3cb730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:11:21.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.277+0000 7fdc6be00640 1 --2- 172.21.15.28:0/4153428719 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cd4e3c92e0 0x55cd4e3cb730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:21.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.277+0000 7fdc6be00640 1 -- 172.21.15.28:0/4153428719 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cd4e2ea6a0 con 0x55cd4e3bec70 2024-05-12T22:11:21.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.277+0000 7fdc6b400640 1 --2- 172.21.15.28:0/4153428719 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cd4e3be2c0 0x55cd4e3be6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:11:21.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.277+0000 7fdc6be00640 1 --2- 172.21.15.28:0/4153428719 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd4e3bec70 0x55cd4e3c8d10 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fdc5c009dc0 tx=0x7fdc5c011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e8edd50f0a52b server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:11:21.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.278+0000 7fdc61600640 1 -- 172.21.15.28:0/4153428719 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc5c005220 con 0x55cd4e3bec70 2024-05-12T22:11:21.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.278+0000 7fdc61600640 1 -- 172.21.15.28:0/4153428719 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdc5c004c80 con 0x55cd4e3bec70 2024-05-12T22:11:21.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.278+0000 7fdc61600640 1 -- 172.21.15.28:0/4153428719 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc5c0198b0 con 0x55cd4e3bec70 2024-05-12T22:11:21.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.278+0000 7fdc68c00640 1 --2- 172.21.15.28:0/3045644668 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fdc500a2d00 0x7fdc500a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:21.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.278+0000 7fdc68c00640 1 -- 172.21.15.28:0/3045644668 <== mon.2 v2:172.21.15.86:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7fdc640f0080 con 0x7fdc6c10e1c0 2024-05-12T22:11:21.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.278+0000 7fdc6b400640 1 --2- 172.21.15.28:0/3045644668 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fdc500a2d00 0x7fdc500a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:21.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.279+0000 7fdc75c0e740 1 -- 172.21.15.28:0/4153428719 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd4e3bec70 msgr2=0x55cd4e3c8d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:11:21.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.279+0000 7fdc75c0e740 1 --2- 172.21.15.28:0/4153428719 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd4e3bec70 0x55cd4e3c8d10 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fdc5c009dc0 tx=0x7fdc5c011810 comp rx=0 tx=0).stop 2024-05-12T22:11:21.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.279+0000 7fdc75c0e740 1 -- 172.21.15.28:0/4153428719 shutdown_connections 2024-05-12T22:11:21.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.279+0000 7fdc75c0e740 1 --2- 172.21.15.28:0/4153428719 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cd4e3c92e0 0x55cd4e3cb730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:21.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.279+0000 7fdc75c0e740 1 --2- 172.21.15.28:0/4153428719 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd4e3bec70 0x55cd4e3c8d10 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:21.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.279+0000 7fdc75c0e740 1 --2- 172.21.15.28:0/4153428719 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cd4e3be2c0 0x55cd4e3be6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:21.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.279+0000 7fdc75c0e740 1 -- 172.21.15.28:0/4153428719 >> 172.21.15.28:0/4153428719 conn(0x55cd4e3ba370 msgr2=0x55cd4e3ba750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:11:21.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.279+0000 7fdc6b400640 1 --2- 172.21.15.28:0/3045644668 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fdc500a2d00 0x7fdc500a51c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fdc58009710 tx=0x7fdc58005f90 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:11:21.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.279+0000 7fdc75c0e740 1 -- 172.21.15.28:0/4153428719 shutdown_connections 2024-05-12T22:11:21.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.279+0000 7fdc75c0e740 1 -- 172.21.15.28:0/4153428719 wait complete. 2024-05-12T22:11:21.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.279+0000 7fdc75c0e740 1 -- start start 2024-05-12T22:11:21.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.280+0000 7fdc75c0e740 1 Processor -- start 2024-05-12T22:11:21.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.280+0000 7fdc75c0e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd4e3be2c0 0x55cd4e3c61a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:21.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.280+0000 7fdc75c0e740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cd4e3bec70 0x55cd4e3c66e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:21.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.280+0000 7fdc75c0e740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cd4e3c92e0 0x55cd4e3c6c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:21.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.280+0000 7fdc75c0e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55cd4e3cc2f0 con 0x55cd4e3be2c0 2024-05-12T22:11:21.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.280+0000 7fdc75c0e740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x55cd4e2dda90 con 0x55cd4e3bec70 2024-05-12T22:11:21.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.280+0000 7fdc75c0e740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55cd4e3cc470 con 0x55cd4e3c92e0 2024-05-12T22:11:21.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.280+0000 7fdc75c0e740 10 client.0 _ll_register_callbacks cb 0x55cd4e318af0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:11:21.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.280+0000 7fdc6be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd4e3be2c0 0x55cd4e3c61a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:21.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.280+0000 7fdc6be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd4e3be2c0 0x55cd4e3c61a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:32794/0 (socket says 172.21.15.28:32794) 2024-05-12T22:11:21.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.280+0000 7fdc6be00640 1 -- 172.21.15.28:0/311088502 learned_addr learned my addr 172.21.15.28:0/311088502 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:11:21.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.281+0000 7fdc70e00640 1 --2- 172.21.15.28:0/311088502 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cd4e3c92e0 0x55cd4e3c6c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:21.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.281+0000 7fdc6b400640 1 --2- 172.21.15.28:0/311088502 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cd4e3bec70 0x55cd4e3c66e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:21.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.281+0000 7fdc6be00640 1 -- 172.21.15.28:0/311088502 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cd4e3c92e0 msgr2=0x55cd4e3c6c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:11:21.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.281+0000 7fdc6be00640 1 --2- 172.21.15.28:0/311088502 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cd4e3c92e0 0x55cd4e3c6c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:21.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.281+0000 7fdc6be00640 1 -- 172.21.15.28:0/311088502 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cd4e3bec70 msgr2=0x55cd4e3c66e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:11:21.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.281+0000 7fdc6be00640 1 --2- 172.21.15.28:0/311088502 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cd4e3bec70 0x55cd4e3c66e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:21.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.281+0000 7fdc6be00640 1 -- 172.21.15.28:0/311088502 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cd4e0fe7e0 con 0x55cd4e3be2c0 2024-05-12T22:11:21.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.281+0000 7fdc6be00640 1 --2- 172.21.15.28:0/311088502 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd4e3be2c0 0x55cd4e3c61a0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fdc5c000c00 tx=0x7fdc5c004970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54dbf3c79554ebcf server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:11:21.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.281+0000 7fdc56000640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-12T22:11:21.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.282+0000 7fdc56000640 1 -- 172.21.15.28:0/311088502 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc5c027070 con 0x55cd4e3be2c0 2024-05-12T22:11:21.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.282+0000 7fdc56000640 1 -- 172.21.15.28:0/311088502 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7fdc5c004450 con 0x55cd4e3be2c0 2024-05-12T22:11:21.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.282+0000 7fdc75c0e740 1 -- 172.21.15.28:0/311088502 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdc5c004c80 con 0x55cd4e3be2c0 2024-05-12T22:11:21.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.282+0000 7fdc56000640 1 -- 172.21.15.28:0/311088502 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fdc5c0099d0 con 0x55cd4e3be2c0 2024-05-12T22:11:21.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.282+0000 7fdc75c0e740 10 client.15765 fetch_fsmap learned FSMap version 25 2024-05-12T22:11:21.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.282+0000 7fdc75c0e740 1 -- 172.21.15.28:0/311088502 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55cd4e15ef70 con 0x55cd4e3be2c0 2024-05-12T22:11:21.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.282+0000 7fdc56000640 1 -- 172.21.15.28:0/311088502 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdc5c004c80 con 0x55cd4e3be2c0 2024-05-12T22:11:21.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.282+0000 7fdc56000640 1 -- 172.21.15.28:0/311088502 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7fdc5c0195b0 con 0x55cd4e3be2c0 2024-05-12T22:11:21.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.283+0000 7fdc75c0e740 10 client.15765 fetch_fsmap finished waiting for FSMap version 25 2024-05-12T22:11:21.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.283+0000 7fdc75c0e740 10 client.15765 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:11:21.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.283+0000 7fdc75c0e740 20 client.15765 populate_metadata read hostname 'smithi028' 2024-05-12T22:11:21.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.283+0000 7fdc75c0e740 1 --2- 172.21.15.28:0/311088502 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55cd4e3d9390 0x55cd4e3db780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:21.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.283+0000 7fdc75c0e740 4 client.15765 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-12T22:11:21.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.283+0000 7fdc75c0e740 1 -- 172.21.15.28:0/311088502 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cd4e3363a0 con 0x55cd4e3d9390 2024-05-12T22:11:21.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.283+0000 7fdc6b400640 1 --2- 172.21.15.28:0/311088502 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55cd4e3d9390 0x55cd4e3db780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:21.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.284+0000 7fdc6b400640 1 --2- 172.21.15.28:0/311088502 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55cd4e3d9390 0x55cd4e3db780 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=20ad724fb3a25e4c server_cookie=269115dee421d8ac in_seq=0 out_seq=0 2024-05-12T22:11:21.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.284+0000 7fdc56000640 10 client.15765 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:11:21.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.285+0000 7fdc56000640 1 -- 172.21.15.28:0/311088502 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55cd4e3363a0 con 0x55cd4e3d9390 2024-05-12T22:11:21.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.285+0000 7fdc56000640 10 client.15765 handle_command_reply: tid=0 2024-05-12T22:11:21.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.285+0000 7fdc75c0e740 1 client.15765 shutdown 2024-05-12T22:11:21.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.285+0000 7fdc75c0e740 1 -- 172.21.15.28:0/311088502 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd4e3be2c0 msgr2=0x55cd4e3c61a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:11:21.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.285+0000 7fdc75c0e740 1 --2- 172.21.15.28:0/311088502 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd4e3be2c0 0x55cd4e3c61a0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fdc5c000c00 tx=0x7fdc5c004970 comp rx=0 tx=0).stop 2024-05-12T22:11:21.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.285+0000 7fdc75c0e740 1 -- 172.21.15.28:0/311088502 shutdown_connections 2024-05-12T22:11:21.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.285+0000 7fdc75c0e740 1 --2- 172.21.15.28:0/311088502 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55cd4e3d9390 0x55cd4e3db780 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:21.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.285+0000 7fdc75c0e740 1 --2- 172.21.15.28:0/311088502 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cd4e3c92e0 0x55cd4e3c6c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:21.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.285+0000 7fdc75c0e740 1 --2- 172.21.15.28:0/311088502 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cd4e3bec70 0x55cd4e3c66e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:21.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.285+0000 7fdc56000640 0 client.15765 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:11:21.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.285+0000 7fdc75c0e740 1 --2- 172.21.15.28:0/311088502 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd4e3be2c0 0x55cd4e3c61a0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:21.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.285+0000 7fdc75c0e740 1 -- 172.21.15.28:0/311088502 >> 172.21.15.28:0/311088502 conn(0x55cd4e3ba370 msgr2=0x55cd4e3c03f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:11:21.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.285+0000 7fdc75c0e740 1 -- 172.21.15.28:0/311088502 shutdown_connections 2024-05-12T22:11:21.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.285+0000 7fdc75c0e740 1 -- 172.21.15.28:0/311088502 wait complete. 2024-05-12T22:11:21.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.285+0000 7fdc75c0e740 20 client.15765 trim_cache size 0 max 16384 2024-05-12T22:11:21.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.305+0000 7fdc75c0e740 1 Processor -- start 2024-05-12T22:11:21.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.305+0000 7fdc75c0e740 1 -- start start 2024-05-12T22:11:21.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.305+0000 7fdc75c0e740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cd4e3be2c0 0x55cd4e3d51b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:21.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.305+0000 7fdc75c0e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd4e3bec70 0x55cd4e3c6870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:21.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.305+0000 7fdc75c0e740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cd4e3c92e0 0x55cd4e3e0f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:21.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.305+0000 7fdc75c0e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdc6c1b1ec0 con 0x55cd4e3bec70 2024-05-12T22:11:21.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.305+0000 7fdc75c0e740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fdc6c1af790 con 0x55cd4e3c92e0 2024-05-12T22:11:21.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.305+0000 7fdc75c0e740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fdc6c1ad5e0 con 0x55cd4e3be2c0 2024-05-12T22:11:21.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.306+0000 7fdc6b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd4e3bec70 0x55cd4e3c6870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:21.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.306+0000 7fdc6b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd4e3bec70 0x55cd4e3c6870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:32806/0 (socket says 172.21.15.28:32806) 2024-05-12T22:11:21.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.306+0000 7fdc6b400640 1 -- 172.21.15.28:0/2687035288 learned_addr learned my addr 172.21.15.28:0/2687035288 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:11:21.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.306+0000 7fdc6be00640 1 --2- 172.21.15.28:0/2687035288 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cd4e3be2c0 0x55cd4e3d51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:21.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.306+0000 7fdc70e00640 1 --2- 172.21.15.28:0/2687035288 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cd4e3c92e0 0x55cd4e3e0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:21.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.306+0000 7fdc6b400640 1 -- 172.21.15.28:0/2687035288 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cd4e3be2c0 msgr2=0x55cd4e3d51b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:11:21.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.306+0000 7fdc6b400640 1 --2- 172.21.15.28:0/2687035288 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cd4e3be2c0 0x55cd4e3d51b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:21.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.306+0000 7fdc6b400640 1 -- 172.21.15.28:0/2687035288 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cd4e3c92e0 msgr2=0x55cd4e3e0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:11:21.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.306+0000 7fdc6b400640 1 --2- 172.21.15.28:0/2687035288 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cd4e3c92e0 0x55cd4e3e0f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:21.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.306+0000 7fdc6b400640 1 -- 172.21.15.28:0/2687035288 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cd4dff6170 con 0x55cd4e3bec70 2024-05-12T22:11:21.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.307+0000 7fdc6b400640 1 --2- 172.21.15.28:0/2687035288 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd4e3bec70 0x55cd4e3c6870 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x55cd4e3dbf30 tx=0x7fdc58004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fde48cd9d8281b95 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:11:21.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.307+0000 7fdc60c00640 1 -- 172.21.15.28:0/2687035288 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc5800fda0 con 0x55cd4e3bec70 2024-05-12T22:11:21.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.307+0000 7fdc60c00640 1 -- 172.21.15.28:0/2687035288 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdc5802d900 con 0x55cd4e3bec70 2024-05-12T22:11:21.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.307+0000 7fdc60c00640 1 -- 172.21.15.28:0/2687035288 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc58021a50 con 0x55cd4e3bec70 2024-05-12T22:11:21.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.308+0000 7fdc75c0e740 1 -- 172.21.15.28:0/2687035288 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd4e3bec70 msgr2=0x55cd4e3c6870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:11:21.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.308+0000 7fdc75c0e740 1 --2- 172.21.15.28:0/2687035288 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd4e3bec70 0x55cd4e3c6870 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x55cd4e3dbf30 tx=0x7fdc58004730 comp rx=0 tx=0).stop 2024-05-12T22:11:21.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.308+0000 7fdc75c0e740 1 -- 172.21.15.28:0/2687035288 shutdown_connections 2024-05-12T22:11:21.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.308+0000 7fdc75c0e740 1 --2- 172.21.15.28:0/2687035288 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cd4e3c92e0 0x55cd4e3e0f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:21.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.308+0000 7fdc75c0e740 1 --2- 172.21.15.28:0/2687035288 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd4e3bec70 0x55cd4e3c6870 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:21.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.308+0000 7fdc75c0e740 1 --2- 172.21.15.28:0/2687035288 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cd4e3be2c0 0x55cd4e3d51b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:21.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.308+0000 7fdc75c0e740 1 -- 172.21.15.28:0/2687035288 >> 172.21.15.28:0/2687035288 conn(0x55cd4e3ba370 msgr2=0x55cd4e3c6160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:11:21.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.308+0000 7fdc75c0e740 1 -- 172.21.15.28:0/2687035288 shutdown_connections 2024-05-12T22:11:21.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.308+0000 7fdc75c0e740 1 -- 172.21.15.28:0/2687035288 wait complete. 2024-05-12T22:11:21.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.308+0000 7fdc75c0e740 1 -- start start 2024-05-12T22:11:21.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.308+0000 7fdc75c0e740 1 Processor -- start 2024-05-12T22:11:21.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.309+0000 7fdc75c0e740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cd4e3be2c0 0x55cd4e3d1b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:21.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.309+0000 7fdc75c0e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd4e3bec70 0x55cd4e3cceb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:21.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.309+0000 7fdc75c0e740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cd4e3c92e0 0x55cd4e3cd450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:21.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.309+0000 7fdc75c0e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdc6c1adc50 con 0x55cd4e3bec70 2024-05-12T22:11:21.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.309+0000 7fdc75c0e740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fdc6c1af0c0 con 0x55cd4e3be2c0 2024-05-12T22:11:21.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.309+0000 7fdc75c0e740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fdc6c1ae320 con 0x55cd4e3c92e0 2024-05-12T22:11:21.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.309+0000 7fdc75c0e740 10 client.0 _ll_register_callbacks cb 0x7fdc58009d80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:11:21.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.309+0000 7fdc6b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd4e3bec70 0x55cd4e3cceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:21.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.309+0000 7fdc6b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd4e3bec70 0x55cd4e3cceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:32810/0 (socket says 172.21.15.28:32810) 2024-05-12T22:11:21.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.309+0000 7fdc6b400640 1 -- 172.21.15.28:0/4209474665 learned_addr learned my addr 172.21.15.28:0/4209474665 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:11:21.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.309+0000 7fdc70e00640 1 --2- 172.21.15.28:0/4209474665 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cd4e3c92e0 0x55cd4e3cd450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:21.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.309+0000 7fdc6be00640 1 --2- 172.21.15.28:0/4209474665 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cd4e3be2c0 0x55cd4e3d1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:21.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.310+0000 7fdc6b400640 1 -- 172.21.15.28:0/4209474665 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cd4e3c92e0 msgr2=0x55cd4e3cd450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:11:21.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.310+0000 7fdc6b400640 1 --2- 172.21.15.28:0/4209474665 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cd4e3c92e0 0x55cd4e3cd450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:21.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.310+0000 7fdc6b400640 1 -- 172.21.15.28:0/4209474665 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cd4e3be2c0 msgr2=0x55cd4e3d1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:11:21.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.310+0000 7fdc6b400640 1 --2- 172.21.15.28:0/4209474665 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cd4e3be2c0 0x55cd4e3d1b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:21.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.310+0000 7fdc6b400640 1 -- 172.21.15.28:0/4209474665 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cd4e3cdc80 con 0x55cd4e3bec70 2024-05-12T22:11:21.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.310+0000 7fdc70e00640 1 --2- 172.21.15.28:0/4209474665 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cd4e3c92e0 0x55cd4e3cd450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:11:21.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.310+0000 7fdc6b400640 1 --2- 172.21.15.28:0/4209474665 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd4e3bec70 0x55cd4e3cceb0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fdc58031000 tx=0x7fdc580096a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f158f9124b30408c server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:11:21.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.310+0000 7fdc56000640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-12T22:11:21.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.310+0000 7fdc56000640 1 -- 172.21.15.28:0/4209474665 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc58011050 con 0x55cd4e3bec70 2024-05-12T22:11:21.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.310+0000 7fdc56000640 1 -- 172.21.15.28:0/4209474665 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7fdc580111f0 con 0x55cd4e3bec70 2024-05-12T22:11:21.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.310+0000 7fdc75c0e740 1 -- 172.21.15.28:0/4209474665 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdc5802d900 con 0x55cd4e3bec70 2024-05-12T22:11:21.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.311+0000 7fdc56000640 1 -- 172.21.15.28:0/4209474665 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fdc58044050 con 0x55cd4e3bec70 2024-05-12T22:11:21.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.311+0000 7fdc75c0e740 10 client.15771 fetch_fsmap learned FSMap version 25 2024-05-12T22:11:21.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.311+0000 7fdc75c0e740 1 -- 172.21.15.28:0/4209474665 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55cd4e448120 con 0x55cd4e3bec70 2024-05-12T22:11:21.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.311+0000 7fdc56000640 1 -- 172.21.15.28:0/4209474665 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdc5802d900 con 0x55cd4e3bec70 2024-05-12T22:11:21.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.311+0000 7fdc56000640 1 -- 172.21.15.28:0/4209474665 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7fdc58010430 con 0x55cd4e3bec70 2024-05-12T22:11:21.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.311+0000 7fdc75c0e740 10 client.15771 fetch_fsmap finished waiting for FSMap version 25 2024-05-12T22:11:21.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.311+0000 7fdc75c0e740 10 client.15771 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:11:21.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.311+0000 7fdc75c0e740 20 client.15771 populate_metadata read hostname 'smithi028' 2024-05-12T22:11:21.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.311+0000 7fdc75c0e740 1 --2- 172.21.15.28:0/4209474665 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55cd4e448860 0x55cd4e44ac80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:21.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.311+0000 7fdc75c0e740 4 client.15771 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-12T22:11:21.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.311+0000 7fdc75c0e740 1 -- 172.21.15.28:0/4209474665 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55cd4e32fef0 con 0x55cd4e448860 2024-05-12T22:11:21.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.311+0000 7fdc6be00640 1 --2- 172.21.15.28:0/4209474665 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55cd4e448860 0x55cd4e44ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:21.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.312+0000 7fdc6be00640 1 --2- 172.21.15.28:0/4209474665 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55cd4e448860 0x55cd4e44ac80 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ea3c4ce710cc322d server_cookie=2ea36e719cc2b472 in_seq=0 out_seq=0 2024-05-12T22:11:21.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.312+0000 7fdc56000640 10 client.15771 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:11:21.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.313+0000 7fdc56000640 1 -- 172.21.15.28:0/4209474665 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55cd4e32fef0 con 0x55cd4e448860 2024-05-12T22:11:21.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.313+0000 7fdc56000640 10 client.15771 handle_command_reply: tid=0 2024-05-12T22:11:21.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.313+0000 7fdc75c0e740 1 client.15771 shutdown 2024-05-12T22:11:21.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.313+0000 7fdc75c0e740 1 -- 172.21.15.28:0/4209474665 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd4e3bec70 msgr2=0x55cd4e3cceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:11:21.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.313+0000 7fdc75c0e740 1 --2- 172.21.15.28:0/4209474665 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd4e3bec70 0x55cd4e3cceb0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fdc58031000 tx=0x7fdc580096a0 comp rx=0 tx=0).stop 2024-05-12T22:11:21.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.313+0000 7fdc75c0e740 1 -- 172.21.15.28:0/4209474665 shutdown_connections 2024-05-12T22:11:21.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.313+0000 7fdc75c0e740 1 --2- 172.21.15.28:0/4209474665 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55cd4e448860 0x55cd4e44ac80 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:21.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.313+0000 7fdc75c0e740 1 --2- 172.21.15.28:0/4209474665 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cd4e3c92e0 0x55cd4e3cd450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:21.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.313+0000 7fdc75c0e740 1 --2- 172.21.15.28:0/4209474665 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd4e3bec70 0x55cd4e3cceb0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:21.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.313+0000 7fdc75c0e740 1 --2- 172.21.15.28:0/4209474665 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cd4e3be2c0 0x55cd4e3d1b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:21.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.313+0000 7fdc75c0e740 1 -- 172.21.15.28:0/4209474665 >> 172.21.15.28:0/4209474665 conn(0x55cd4e3ba370 msgr2=0x55cd4e3b94f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:11:21.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.314+0000 7fdc56000640 0 client.15771 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:11:21.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.314+0000 7fdc75c0e740 1 -- 172.21.15.28:0/4209474665 shutdown_connections 2024-05-12T22:11:21.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.314+0000 7fdc75c0e740 1 -- 172.21.15.28:0/4209474665 wait complete. 2024-05-12T22:11:21.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.314+0000 7fdc75c0e740 20 client.15771 trim_cache size 0 max 16384 2024-05-12T22:11:21.321 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-05-12T22:11:21.321 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-05-12T22:11:21.321 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "5e11bd9a-b731-4929-b736-1ad850b8755d", 2024-05-12T22:11:21.321 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-05-12T22:11:21.321 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-05-12T22:11:21.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.315+0000 7fdc72e00640 1 -- 172.21.15.28:0/3045644668 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fdc500a2d00 msgr2=0x7fdc500a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:11:21.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.315+0000 7fdc72e00640 1 --2- 172.21.15.28:0/3045644668 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fdc500a2d00 0x7fdc500a51c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fdc58009710 tx=0x7fdc58005f90 comp rx=0 tx=0).stop 2024-05-12T22:11:21.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.315+0000 7fdc72e00640 1 -- 172.21.15.28:0/3045644668 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdc6c10e1c0 msgr2=0x7fdc6c19eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:11:21.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.315+0000 7fdc72e00640 1 --2- 172.21.15.28:0/3045644668 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdc6c10e1c0 0x7fdc6c19eb60 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fdc6402aea0 tx=0x7fdc64003f10 comp rx=0 tx=0).stop 2024-05-12T22:11:21.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.315+0000 7fdc72e00640 1 -- 172.21.15.28:0/3045644668 shutdown_connections 2024-05-12T22:11:21.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.315+0000 7fdc72e00640 1 --2- 172.21.15.28:0/3045644668 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fdc500a2d00 0x7fdc500a51c0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:21.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.315+0000 7fdc72e00640 1 --2- 172.21.15.28:0/3045644668 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdc6c10e1c0 0x7fdc6c19eb60 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:21.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.315+0000 7fdc72e00640 1 --2- 172.21.15.28:0/3045644668 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdc6c100f30 0x7fdc6c19a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:21.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.315+0000 7fdc72e00640 1 --2- 172.21.15.28:0/3045644668 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc6c100610 0x7fdc6c19a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:21.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.315+0000 7fdc72e00640 1 -- 172.21.15.28:0/3045644668 >> 172.21.15.28:0/3045644668 conn(0x7fdc6c076ab0 msgr2=0x7fdc6c10f660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:11:21.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.315+0000 7fdc72e00640 1 -- 172.21.15.28:0/3045644668 shutdown_connections 2024-05-12T22:11:21.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.315+0000 7fdc72e00640 1 -- 172.21.15.28:0/3045644668 wait complete. 2024-05-12T22:11:21.335 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-12T22:11:21.336 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-12T22:11:21.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.556+0000 7f3485000640 1 -- 172.21.15.28:0/1170801319 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3480109150 msgr2=0x7f3480114440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:11:21.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.556+0000 7f3485000640 1 --2- 172.21.15.28:0/1170801319 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3480109150 0x7f3480114440 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f346c009900 tx=0x7f346c02a400 comp rx=0 tx=0).stop 2024-05-12T22:11:21.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.556+0000 7f3485000640 1 -- 172.21.15.28:0/1170801319 shutdown_connections 2024-05-12T22:11:21.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.556+0000 7f3485000640 1 --2- 172.21.15.28:0/1170801319 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3480114980 0x7f3480116d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:21.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.556+0000 7f3485000640 1 --2- 172.21.15.28:0/1170801319 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3480109150 0x7f3480114440 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:21.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.556+0000 7f3485000640 1 --2- 172.21.15.28:0/1170801319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3480108830 0x7f3480108c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:21.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.556+0000 7f3485000640 1 -- 172.21.15.28:0/1170801319 >> 172.21.15.28:0/1170801319 conn(0x7f348006c7b0 msgr2=0x7f348006cbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:11:21.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.556+0000 7f3485000640 1 -- 172.21.15.28:0/1170801319 shutdown_connections 2024-05-12T22:11:21.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.556+0000 7f3485000640 1 -- 172.21.15.28:0/1170801319 wait complete. 2024-05-12T22:11:21.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.557+0000 7f3485000640 1 Processor -- start 2024-05-12T22:11:21.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.557+0000 7f3485000640 1 -- start start 2024-05-12T22:11:21.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.558+0000 7f3485000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3480108830 0x7f34801a86d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:21.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.558+0000 7f3485000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3480109150 0x7f34801a8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:21.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.558+0000 7f3485000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3480114980 0x7f34801a2840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:21.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.558+0000 7f3485000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f348011a2a0 con 0x7f3480109150 2024-05-12T22:11:21.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.558+0000 7f3485000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f348011a120 con 0x7f3480114980 2024-05-12T22:11:21.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.558+0000 7f3485000640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f348011a420 con 0x7f3480108830 2024-05-12T22:11:21.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.558+0000 7f347f400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3480108830 0x7f34801a86d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:21.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.558+0000 7f347f400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3480108830 0x7f34801a86d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:59248/0 (socket says 172.21.15.28:59248) 2024-05-12T22:11:21.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.558+0000 7f347f400640 1 -- 172.21.15.28:0/3583558641 learned_addr learned my addr 172.21.15.28:0/3583558641 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:11:21.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.558+0000 7f347ea00640 1 --2- 172.21.15.28:0/3583558641 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3480109150 0x7f34801a8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:21.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.558+0000 7f347fe00640 1 --2- 172.21.15.28:0/3583558641 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3480114980 0x7f34801a2840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:21.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.559+0000 7f347f400640 1 -- 172.21.15.28:0/3583558641 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3480114980 msgr2=0x7f34801a2840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:11:21.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.559+0000 7f347f400640 1 --2- 172.21.15.28:0/3583558641 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3480114980 0x7f34801a2840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:21.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.559+0000 7f347f400640 1 -- 172.21.15.28:0/3583558641 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3480109150 msgr2=0x7f34801a8c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:11:21.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.559+0000 7f347f400640 1 --2- 172.21.15.28:0/3583558641 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3480109150 0x7f34801a8c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:21.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.559+0000 7f347f400640 1 -- 172.21.15.28:0/3583558641 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34801a3010 con 0x7f3480108830 2024-05-12T22:11:21.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.559+0000 7f347fe00640 1 --2- 172.21.15.28:0/3583558641 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3480114980 0x7f34801a2840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:11:21.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.559+0000 7f347f400640 1 --2- 172.21.15.28:0/3583558641 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3480108830 0x7f34801a86d0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f347000ea60 tx=0x7f347000ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:11:21.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.559+0000 7f3477e00640 1 -- 172.21.15.28:0/3583558641 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f347000cde0 con 0x7f3480108830 2024-05-12T22:11:21.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.559+0000 7f3477e00640 1 -- 172.21.15.28:0/3583558641 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f34700035f0 con 0x7f3480108830 2024-05-12T22:11:21.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.559+0000 7f3477e00640 1 -- 172.21.15.28:0/3583558641 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3470010670 con 0x7f3480108830 2024-05-12T22:11:21.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.559+0000 7f3485000640 1 -- 172.21.15.28:0/3583558641 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34801a3300 con 0x7f3480108830 2024-05-12T22:11:21.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.560+0000 7f3485000640 1 -- 172.21.15.28:0/3583558641 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34801af520 con 0x7f3480108830 2024-05-12T22:11:21.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.563+0000 7f3477e00640 1 -- 172.21.15.28:0/3583558641 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7f34700037a0 con 0x7f3480108830 2024-05-12T22:11:21.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.564+0000 7f34895de740 1 Processor -- start 2024-05-12T22:11:21.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.564+0000 7f34895de740 1 -- start start 2024-05-12T22:11:21.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.564+0000 7f34895de740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ccc21eddd0 0x55ccc21ee1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:21.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.564+0000 7f34895de740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ccc21ee6f0 0x55ccc21f8880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:21.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.564+0000 7f34895de740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ccc21f8dc0 0x55ccc21fb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:21.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.564+0000 7f34895de740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ccc21b62a0 con 0x55ccc21f8dc0 2024-05-12T22:11:21.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.564+0000 7f34895de740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x55ccc2152b90 con 0x55ccc21ee6f0 2024-05-12T22:11:21.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.564+0000 7f34895de740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55ccc20fa5b0 con 0x55ccc21eddd0 2024-05-12T22:11:21.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.564+0000 7f347fe00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ccc21eddd0 0x55ccc21ee1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:21.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.565+0000 7f347fe00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ccc21eddd0 0x55ccc21ee1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:59256/0 (socket says 172.21.15.28:59256) 2024-05-12T22:11:21.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.565+0000 7f347fe00640 1 -- 172.21.15.28:0/967965330 learned_addr learned my addr 172.21.15.28:0/967965330 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:11:21.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.565+0000 7f347f400640 1 --2- 172.21.15.28:0/967965330 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ccc21ee6f0 0x55ccc21f8880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:21.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.565+0000 7f347fe00640 1 -- 172.21.15.28:0/967965330 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ccc21ee6f0 msgr2=0x55ccc21f8880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:11:21.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.565+0000 7f347fe00640 1 --2- 172.21.15.28:0/967965330 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ccc21ee6f0 0x55ccc21f8880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:21.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.565+0000 7f347ea00640 1 --2- 172.21.15.28:0/967965330 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ccc21f8dc0 0x55ccc21fb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:21.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.565+0000 7f347fe00640 1 -- 172.21.15.28:0/967965330 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ccc21f8dc0 msgr2=0x55ccc21fb1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:11:21.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.565+0000 7f347fe00640 1 --2- 172.21.15.28:0/967965330 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ccc21f8dc0 0x55ccc21fb1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:21.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.565+0000 7f347fe00640 1 -- 172.21.15.28:0/967965330 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ccc1f307d0 con 0x55ccc21eddd0 2024-05-12T22:11:21.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.565+0000 7f347ea00640 1 --2- 172.21.15.28:0/967965330 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ccc21f8dc0 0x55ccc21fb1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-12T22:11:21.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.565+0000 7f3477e00640 1 --2- 172.21.15.28:0/3583558641 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f34640a2d00 0x7f34640a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:21.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.565+0000 7f3477e00640 1 -- 172.21.15.28:0/3583558641 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f34700cf730 con 0x7f3480108830 2024-05-12T22:11:21.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.566+0000 7f347fe00640 1 --2- 172.21.15.28:0/967965330 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ccc21eddd0 0x55ccc21ee1b0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f3478011790 tx=0x7f3478011c50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=557e1f3840704e20 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:11:21.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.566+0000 7f347f400640 1 --2- 172.21.15.28:0/3583558641 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f34640a2d00 0x7f34640a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:21.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.566+0000 7f3474c00640 1 -- 172.21.15.28:0/967965330 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3478003c30 con 0x55ccc21eddd0 2024-05-12T22:11:21.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.566+0000 7f3474c00640 1 -- 172.21.15.28:0/967965330 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3478003dd0 con 0x55ccc21eddd0 2024-05-12T22:11:21.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.566+0000 7f3474c00640 1 -- 172.21.15.28:0/967965330 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34780193e0 con 0x55ccc21eddd0 2024-05-12T22:11:21.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.567+0000 7f34895de740 1 -- 172.21.15.28:0/967965330 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ccc21eddd0 msgr2=0x55ccc21ee1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:11:21.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.567+0000 7f34895de740 1 --2- 172.21.15.28:0/967965330 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ccc21eddd0 0x55ccc21ee1b0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f3478011790 tx=0x7f3478011c50 comp rx=0 tx=0).stop 2024-05-12T22:11:21.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.567+0000 7f347f400640 1 --2- 172.21.15.28:0/3583558641 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f34640a2d00 0x7f34640a51c0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x55ccc21ef390 tx=0x7f347000ef90 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:11:21.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.567+0000 7f34895de740 1 -- 172.21.15.28:0/967965330 shutdown_connections 2024-05-12T22:11:21.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.567+0000 7f34895de740 1 --2- 172.21.15.28:0/967965330 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ccc21f8dc0 0x55ccc21fb1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:21.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.567+0000 7f34895de740 1 --2- 172.21.15.28:0/967965330 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ccc21ee6f0 0x55ccc21f8880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:21.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.567+0000 7f34895de740 1 --2- 172.21.15.28:0/967965330 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ccc21eddd0 0x55ccc21ee1b0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:21.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.567+0000 7f34895de740 1 -- 172.21.15.28:0/967965330 >> 172.21.15.28:0/967965330 conn(0x55ccc21ea0f0 msgr2=0x55ccc21ea4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:11:21.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.567+0000 7f34895de740 1 -- 172.21.15.28:0/967965330 shutdown_connections 2024-05-12T22:11:21.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.567+0000 7f34895de740 1 -- 172.21.15.28:0/967965330 wait complete. 2024-05-12T22:11:21.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.568+0000 7f34895de740 1 -- start start 2024-05-12T22:11:21.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.568+0000 7f34895de740 1 Processor -- start 2024-05-12T22:11:21.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.569+0000 7f34895de740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ccc21eddd0 0x55ccc21f8790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:21.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.569+0000 7f34895de740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ccc21ee6f0 0x55ccc21f5a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:21.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.569+0000 7f34895de740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ccc21f8dc0 0x55ccc21f6090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:21.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.569+0000 7f34895de740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ccc210d9f0 con 0x55ccc21f8dc0 2024-05-12T22:11:21.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.569+0000 7f34895de740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x55ccc214eb50 con 0x55ccc21eddd0 2024-05-12T22:11:21.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.569+0000 7f34895de740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55ccc21fbbb0 con 0x55ccc21ee6f0 2024-05-12T22:11:21.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.569+0000 7f34895de740 10 client.0 _ll_register_callbacks cb 0x55ccc1e65be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:11:21.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.569+0000 7f347ea00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ccc21ee6f0 0x55ccc21f5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:21.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.569+0000 7f347ea00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ccc21ee6f0 0x55ccc21f5a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:59266/0 (socket says 172.21.15.28:59266) 2024-05-12T22:11:21.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.569+0000 7f347ea00640 1 -- 172.21.15.28:0/1008881702 learned_addr learned my addr 172.21.15.28:0/1008881702 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:11:21.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.569+0000 7f347fe00640 1 --2- 172.21.15.28:0/1008881702 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ccc21eddd0 0x55ccc21f8790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:21.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.570+0000 7f347ea00640 1 -- 172.21.15.28:0/1008881702 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ccc21eddd0 msgr2=0x55ccc21f8790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:11:21.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.570+0000 7f347ea00640 1 --2- 172.21.15.28:0/1008881702 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ccc21eddd0 0x55ccc21f8790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:21.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.570+0000 7f347fe00640 1 --2- 172.21.15.28:0/1008881702 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ccc21f8dc0 0x55ccc21f6090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:21.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.570+0000 7f347ea00640 1 -- 172.21.15.28:0/1008881702 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ccc21f8dc0 msgr2=0x55ccc21f6090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:11:21.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.570+0000 7f347ea00640 1 --2- 172.21.15.28:0/1008881702 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ccc21f8dc0 0x55ccc21f6090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:21.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.570+0000 7f347ea00640 1 -- 172.21.15.28:0/1008881702 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ccc1e663f0 con 0x55ccc21ee6f0 2024-05-12T22:11:21.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.570+0000 7f347fe00640 1 --2- 172.21.15.28:0/1008881702 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ccc21f8dc0 0x55ccc21f6090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-12T22:11:21.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.571+0000 7f347ea00640 1 --2- 172.21.15.28:0/1008881702 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ccc21ee6f0 0x55ccc21f5a80 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f346c02a920 tx=0x7f346c0317e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6ad56ea919542dae server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:11:21.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.571+0000 7f3469600640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-12T22:11:21.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.571+0000 7f3469600640 1 -- 172.21.15.28:0/1008881702 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f346c040070 con 0x55ccc21ee6f0 2024-05-12T22:11:21.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.572+0000 7f34895de740 1 -- 172.21.15.28:0/1008881702 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3478003dd0 con 0x55ccc21ee6f0 2024-05-12T22:11:21.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.572+0000 7f3469600640 1 -- 172.21.15.28:0/1008881702 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f346c004250 con 0x55ccc21ee6f0 2024-05-12T22:11:21.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.572+0000 7f3469600640 1 -- 172.21.15.28:0/1008881702 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f346c035040 con 0x55ccc21ee6f0 2024-05-12T22:11:21.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.572+0000 7f34895de740 10 client.25855 fetch_fsmap learned FSMap version 25 2024-05-12T22:11:21.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.572+0000 7f34895de740 1 -- 172.21.15.28:0/1008881702 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55ccc1e260f0 con 0x55ccc21ee6f0 2024-05-12T22:11:21.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.573+0000 7f3469600640 1 -- 172.21.15.28:0/1008881702 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3478003dd0 con 0x55ccc21ee6f0 2024-05-12T22:11:21.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.573+0000 7f3469600640 1 -- 172.21.15.28:0/1008881702 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7f346c0455e0 con 0x55ccc21ee6f0 2024-05-12T22:11:21.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.573+0000 7f34895de740 10 client.25855 fetch_fsmap finished waiting for FSMap version 25 2024-05-12T22:11:21.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.573+0000 7f34895de740 10 client.25855 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:11:21.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.573+0000 7f34895de740 20 client.25855 populate_metadata read hostname 'smithi028' 2024-05-12T22:11:21.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.573+0000 7f34895de740 1 --2- 172.21.15.28:0/1008881702 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55ccc2208920 0x55ccc220ad40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:21.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.573+0000 7f34895de740 4 client.25855 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-12T22:11:21.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.573+0000 7f34895de740 1 -- 172.21.15.28:0/1008881702 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ccc21602c0 con 0x55ccc2208920 2024-05-12T22:11:21.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.573+0000 7f347fe00640 1 --2- 172.21.15.28:0/1008881702 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55ccc2208920 0x55ccc220ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:21.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.574+0000 7f347fe00640 1 --2- 172.21.15.28:0/1008881702 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55ccc2208920 0x55ccc220ad40 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b84e8bc0783ab163 server_cookie=ddd48c806359d2ad in_seq=0 out_seq=0 2024-05-12T22:11:21.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.574+0000 7f3469600640 10 client.25855 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:11:21.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.575+0000 7f3469600640 1 -- 172.21.15.28:0/1008881702 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ccc21602c0 con 0x55ccc2208920 2024-05-12T22:11:21.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.575+0000 7f3469600640 10 client.25855 handle_command_reply: tid=0 2024-05-12T22:11:21.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.575+0000 7f34895de740 1 client.25855 shutdown 2024-05-12T22:11:21.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.575+0000 7f34895de740 1 -- 172.21.15.28:0/1008881702 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ccc21ee6f0 msgr2=0x55ccc21f5a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:11:21.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.575+0000 7f34895de740 1 --2- 172.21.15.28:0/1008881702 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ccc21ee6f0 0x55ccc21f5a80 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f346c02a920 tx=0x7f346c0317e0 comp rx=0 tx=0).stop 2024-05-12T22:11:21.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.575+0000 7f34895de740 1 -- 172.21.15.28:0/1008881702 shutdown_connections 2024-05-12T22:11:21.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.575+0000 7f34895de740 1 --2- 172.21.15.28:0/1008881702 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55ccc2208920 0x55ccc220ad40 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:21.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.575+0000 7f34895de740 1 --2- 172.21.15.28:0/1008881702 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ccc21f8dc0 0x55ccc21f6090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:21.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.575+0000 7f34895de740 1 --2- 172.21.15.28:0/1008881702 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ccc21ee6f0 0x55ccc21f5a80 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:21.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.575+0000 7f34895de740 1 --2- 172.21.15.28:0/1008881702 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ccc21eddd0 0x55ccc21f8790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:21.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.575+0000 7f3469600640 0 client.25855 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:11:21.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.575+0000 7f34895de740 1 -- 172.21.15.28:0/1008881702 >> 172.21.15.28:0/1008881702 conn(0x55ccc21ea0f0 msgr2=0x55ccc21efef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:11:21.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.576+0000 7f34895de740 1 -- 172.21.15.28:0/1008881702 shutdown_connections 2024-05-12T22:11:21.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.576+0000 7f34895de740 1 -- 172.21.15.28:0/1008881702 wait complete. 2024-05-12T22:11:21.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.576+0000 7f34895de740 20 client.25855 trim_cache size 0 max 16384 2024-05-12T22:11:21.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.601+0000 7f34895de740 1 Processor -- start 2024-05-12T22:11:21.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.601+0000 7f34895de740 1 -- start start 2024-05-12T22:11:21.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.602+0000 7f34895de740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ccc21eddd0 0x55ccc21f10b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:21.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.602+0000 7f34895de740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ccc21ee6f0 0x55ccc21f82c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:21.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.602+0000 7f34895de740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ccc21f8dc0 0x55ccc22105f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:21.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.602+0000 7f34895de740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f34801ba630 con 0x55ccc21ee6f0 2024-05-12T22:11:21.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.602+0000 7f34895de740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f34801b7f00 con 0x55ccc21eddd0 2024-05-12T22:11:21.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.602+0000 7f34895de740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f34801b5d50 con 0x55ccc21f8dc0 2024-05-12T22:11:21.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.602+0000 7f347fe00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ccc21eddd0 0x55ccc21f10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:21.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.602+0000 7f347fe00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ccc21eddd0 0x55ccc21f10b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:41880/0 (socket says 172.21.15.28:41880) 2024-05-12T22:11:21.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.602+0000 7f347fe00640 1 -- 172.21.15.28:0/396624950 learned_addr learned my addr 172.21.15.28:0/396624950 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:11:21.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.602+0000 7f347ea00640 1 --2- 172.21.15.28:0/396624950 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ccc21ee6f0 0x55ccc21f82c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:21.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.602+0000 7f347fe00640 1 --2- 172.21.15.28:0/396624950 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ccc21f8dc0 0x55ccc22105f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:21.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.602+0000 7f347fe00640 1 -- 172.21.15.28:0/396624950 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ccc21f8dc0 msgr2=0x55ccc22105f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:11:21.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.602+0000 7f347fe00640 1 --2- 172.21.15.28:0/396624950 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ccc21f8dc0 0x55ccc22105f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:21.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.602+0000 7f347fe00640 1 -- 172.21.15.28:0/396624950 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ccc21ee6f0 msgr2=0x55ccc21f82c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:11:21.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.602+0000 7f347fe00640 1 --2- 172.21.15.28:0/396624950 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ccc21ee6f0 0x55ccc21f82c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:21.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.602+0000 7f347fe00640 1 -- 172.21.15.28:0/396624950 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ccc21fb1b0 con 0x55ccc21eddd0 2024-05-12T22:11:21.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.603+0000 7f347fe00640 1 --2- 172.21.15.28:0/396624950 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ccc21eddd0 0x55ccc21f10b0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f3478011730 tx=0x7f3478011c50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4d5893c92f3be909 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:11:21.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.603+0000 7f346be00640 1 -- 172.21.15.28:0/396624950 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3478004190 con 0x55ccc21eddd0 2024-05-12T22:11:21.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.603+0000 7f346be00640 1 -- 172.21.15.28:0/396624950 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f347800f8c0 con 0x55ccc21eddd0 2024-05-12T22:11:21.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.603+0000 7f346be00640 1 -- 172.21.15.28:0/396624950 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3478004aa0 con 0x55ccc21eddd0 2024-05-12T22:11:21.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.604+0000 7f34895de740 1 -- 172.21.15.28:0/396624950 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ccc21eddd0 msgr2=0x55ccc21f10b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:11:21.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.604+0000 7f34895de740 1 --2- 172.21.15.28:0/396624950 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ccc21eddd0 0x55ccc21f10b0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f3478011730 tx=0x7f3478011c50 comp rx=0 tx=0).stop 2024-05-12T22:11:21.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.604+0000 7f34895de740 1 -- 172.21.15.28:0/396624950 shutdown_connections 2024-05-12T22:11:21.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.604+0000 7f34895de740 1 --2- 172.21.15.28:0/396624950 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ccc21f8dc0 0x55ccc22105f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:21.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.604+0000 7f34895de740 1 --2- 172.21.15.28:0/396624950 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ccc21ee6f0 0x55ccc21f82c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:21.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.604+0000 7f34895de740 1 --2- 172.21.15.28:0/396624950 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ccc21eddd0 0x55ccc21f10b0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:21.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.604+0000 7f34895de740 1 -- 172.21.15.28:0/396624950 >> 172.21.15.28:0/396624950 conn(0x55ccc21ea0f0 msgr2=0x55ccc21ef520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:11:21.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.604+0000 7f34895de740 1 -- 172.21.15.28:0/396624950 shutdown_connections 2024-05-12T22:11:21.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.604+0000 7f34895de740 1 -- 172.21.15.28:0/396624950 wait complete. 2024-05-12T22:11:21.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.605+0000 7f34895de740 1 -- start start 2024-05-12T22:11:21.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.605+0000 7f34895de740 1 Processor -- start 2024-05-12T22:11:21.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.606+0000 7f34895de740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ccc21eddd0 0x55ccc21fd000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:21.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.606+0000 7f34895de740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ccc21ee6f0 0x55ccc21fd540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:21.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.606+0000 7f34895de740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ccc21f8dc0 0x55ccc226de60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:21.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.606+0000 7f34895de740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f34801b63c0 con 0x55ccc21ee6f0 2024-05-12T22:11:21.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.606+0000 7f34895de740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f34801b7830 con 0x55ccc21eddd0 2024-05-12T22:11:21.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.606+0000 7f34895de740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f34801b6a90 con 0x55ccc21f8dc0 2024-05-12T22:11:21.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.606+0000 7f34895de740 10 client.0 _ll_register_callbacks cb 0x7f34780093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:11:21.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.606+0000 7f347fe00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ccc21eddd0 0x55ccc21fd000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:21.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.606+0000 7f347fe00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ccc21eddd0 0x55ccc21fd000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:41894/0 (socket says 172.21.15.28:41894) 2024-05-12T22:11:21.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.606+0000 7f347fe00640 1 -- 172.21.15.28:0/3939999572 learned_addr learned my addr 172.21.15.28:0/3939999572 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:11:21.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.606+0000 7f347fe00640 1 --2- 172.21.15.28:0/3939999572 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ccc21f8dc0 0x55ccc226de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:21.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.607+0000 7f347fe00640 1 -- 172.21.15.28:0/3939999572 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ccc21f8dc0 msgr2=0x55ccc226de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:11:21.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.607+0000 7f347fe00640 1 --2- 172.21.15.28:0/3939999572 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ccc21f8dc0 0x55ccc226de60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:21.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.607+0000 7f347fe00640 1 -- 172.21.15.28:0/3939999572 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ccc21ee6f0 msgr2=0x55ccc21fd540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:11:21.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.607+0000 7f347fe00640 1 --2- 172.21.15.28:0/3939999572 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ccc21ee6f0 0x55ccc21fd540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:21.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.607+0000 7f347fe00640 1 -- 172.21.15.28:0/3939999572 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ccc226e450 con 0x55ccc21eddd0 2024-05-12T22:11:21.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.607+0000 7f347fe00640 1 --2- 172.21.15.28:0/3939999572 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ccc21eddd0 0x55ccc21fd000 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f3478005d50 tx=0x7f347802a040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5d9b0f8967c98fc server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:11:21.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.607+0000 7f3469600640 10 client.0 ms_handle_connect on v2:172.21.15.86:3300/0 2024-05-12T22:11:21.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.607+0000 7f3469600640 1 -- 172.21.15.28:0/3939999572 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3478004530 con 0x55ccc21eddd0 2024-05-12T22:11:21.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.607+0000 7f34895de740 1 -- 172.21.15.28:0/3939999572 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f347800f8c0 con 0x55ccc21eddd0 2024-05-12T22:11:21.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.608+0000 7f3469600640 1 -- 172.21.15.28:0/3939999572 <== mon.2 v2:172.21.15.86:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f3478003a30 con 0x55ccc21eddd0 2024-05-12T22:11:21.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.608+0000 7f3469600640 1 -- 172.21.15.28:0/3939999572 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f3478003c30 con 0x55ccc21eddd0 2024-05-12T22:11:21.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.608+0000 7f34895de740 10 client.25907 fetch_fsmap learned FSMap version 25 2024-05-12T22:11:21.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.608+0000 7f34895de740 1 -- 172.21.15.28:0/3939999572 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55ccc226f740 con 0x55ccc21eddd0 2024-05-12T22:11:21.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.608+0000 7f3469600640 1 -- 172.21.15.28:0/3939999572 <== mon.2 v2:172.21.15.86:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f347800f8c0 con 0x55ccc21eddd0 2024-05-12T22:11:21.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.608+0000 7f3469600640 1 -- 172.21.15.28:0/3939999572 <== mon.2 v2:172.21.15.86:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7f3478033040 con 0x55ccc21eddd0 2024-05-12T22:11:21.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.608+0000 7f34895de740 10 client.25907 fetch_fsmap finished waiting for FSMap version 25 2024-05-12T22:11:21.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.608+0000 7f34895de740 10 client.25907 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:11:21.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.608+0000 7f34895de740 20 client.25907 populate_metadata read hostname 'smithi028' 2024-05-12T22:11:21.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.608+0000 7f34895de740 1 --2- 172.21.15.28:0/3939999572 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55ccc226fc20 0x55ccc2272040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:21.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.609+0000 7f34895de740 4 client.25907 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-12T22:11:21.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.609+0000 7f34895de740 1 -- 172.21.15.28:0/3939999572 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55ccc215a610 con 0x55ccc226fc20 2024-05-12T22:11:21.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.609+0000 7f347ea00640 1 --2- 172.21.15.28:0/3939999572 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55ccc226fc20 0x55ccc2272040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:21.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.609+0000 7f347ea00640 1 --2- 172.21.15.28:0/3939999572 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55ccc226fc20 0x55ccc2272040 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7a264982e6b10e68 server_cookie=c2a109f72aa99e87 in_seq=0 out_seq=0 2024-05-12T22:11:21.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.609+0000 7f3469600640 10 client.25907 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:11:21.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.610+0000 7f3469600640 1 -- 172.21.15.28:0/3939999572 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+268 (crc 0 0 0) 0x55ccc215a610 con 0x55ccc226fc20 2024-05-12T22:11:21.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.610+0000 7f3469600640 10 client.25907 handle_command_reply: tid=0 2024-05-12T22:11:21.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.610+0000 7f34895de740 1 client.25907 shutdown 2024-05-12T22:11:21.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.610+0000 7f34895de740 1 -- 172.21.15.28:0/3939999572 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ccc21eddd0 msgr2=0x55ccc21fd000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:11:21.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.610+0000 7f34895de740 1 --2- 172.21.15.28:0/3939999572 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ccc21eddd0 0x55ccc21fd000 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f3478005d50 tx=0x7f347802a040 comp rx=0 tx=0).stop 2024-05-12T22:11:21.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.610+0000 7f34895de740 1 -- 172.21.15.28:0/3939999572 shutdown_connections 2024-05-12T22:11:21.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.610+0000 7f34895de740 1 --2- 172.21.15.28:0/3939999572 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55ccc226fc20 0x55ccc2272040 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:21.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.610+0000 7f34895de740 1 --2- 172.21.15.28:0/3939999572 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ccc21f8dc0 0x55ccc226de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:21.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.610+0000 7f3469600640 0 client.25907 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:11:21.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.610+0000 7f34895de740 1 --2- 172.21.15.28:0/3939999572 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ccc21ee6f0 0x55ccc21fd540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:21.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.610+0000 7f34895de740 1 --2- 172.21.15.28:0/3939999572 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ccc21eddd0 0x55ccc21fd000 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:21.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.610+0000 7f34895de740 1 -- 172.21.15.28:0/3939999572 >> 172.21.15.28:0/3939999572 conn(0x55ccc21ea0f0 msgr2=0x55ccc21f72b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:11:21.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.610+0000 7f34895de740 1 -- 172.21.15.28:0/3939999572 shutdown_connections 2024-05-12T22:11:21.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.611+0000 7f34895de740 1 -- 172.21.15.28:0/3939999572 wait complete. 2024-05-12T22:11:21.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.611+0000 7f34895de740 20 client.25907 trim_cache size 0 max 16384 2024-05-12T22:11:21.614 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-05-12T22:11:21.614 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "scrub active (1462 inodes in the stack)", 2024-05-12T22:11:21.614 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-05-12T22:11:21.614 INFO:teuthology.orchestra.run.smithi028.stdout: "5e11bd9a-b731-4929-b736-1ad850b8755d": { 2024-05-12T22:11:21.614 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-05-12T22:11:21.615 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "5e11bd9a-b731-4929-b736-1ad850b8755d", 2024-05-12T22:11:21.615 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-05-12T22:11:21.615 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:11:21.615 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:11:21.615 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-05-12T22:11:21.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.612+0000 7f3485000640 1 -- 172.21.15.28:0/3583558641 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f34640a2d00 msgr2=0x7f34640a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:11:21.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.612+0000 7f3485000640 1 --2- 172.21.15.28:0/3583558641 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f34640a2d00 0x7f34640a51c0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x55ccc21ef390 tx=0x7f347000ef90 comp rx=0 tx=0).stop 2024-05-12T22:11:21.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.612+0000 7f3485000640 1 -- 172.21.15.28:0/3583558641 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3480108830 msgr2=0x7f34801a86d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:11:21.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.612+0000 7f3485000640 1 --2- 172.21.15.28:0/3583558641 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3480108830 0x7f34801a86d0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f347000ea60 tx=0x7f347000ef20 comp rx=0 tx=0).stop 2024-05-12T22:11:21.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.612+0000 7f3485000640 1 -- 172.21.15.28:0/3583558641 shutdown_connections 2024-05-12T22:11:21.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.612+0000 7f3485000640 1 --2- 172.21.15.28:0/3583558641 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f34640a2d00 0x7f34640a51c0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:21.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.612+0000 7f3485000640 1 --2- 172.21.15.28:0/3583558641 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3480114980 0x7f34801a2840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:21.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.612+0000 7f3485000640 1 --2- 172.21.15.28:0/3583558641 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3480109150 0x7f34801a8c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:21.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.612+0000 7f3485000640 1 --2- 172.21.15.28:0/3583558641 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3480108830 0x7f34801a86d0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:21.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.612+0000 7f3485000640 1 -- 172.21.15.28:0/3583558641 >> 172.21.15.28:0/3583558641 conn(0x7f348006c7b0 msgr2=0x7f348010a270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:11:21.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.612+0000 7f3485000640 1 -- 172.21.15.28:0/3583558641 shutdown_connections 2024-05-12T22:11:21.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:21.613+0000 7f3485000640 1 -- 172.21.15.28:0/3583558641 wait complete. 2024-05-12T22:11:21.633 INFO:tasks.cephfs.filesystem:scrub status for tag:5e11bd9a-b731-4929-b736-1ad850b8755d - {'path': '/', 'tag': '5e11bd9a-b731-4929-b736-1ad850b8755d', 'options': 'recursive,force'} 2024-05-12T22:11:21.649 INFO:tasks.workunit.client.0.smithi028.stdout:checking the name lister (/usr/bin/nm -B) interface... BSD nm 2024-05-12T22:11:21.649 INFO:tasks.workunit.client.0.smithi028.stdout:checking whether ln -s works... yes 2024-05-12T22:11:21.663 INFO:tasks.workunit.client.0.smithi028.stdout:checking the maximum length of command line arguments... 1572864 2024-05-12T22:11:21.671 INFO:tasks.workunit.client.0.smithi028.stdout:checking how to convert x86_64-pc-linux-gnu file names to x86_64-pc-linux-gnu format... func_convert_file_noop 2024-05-12T22:11:21.671 INFO:tasks.workunit.client.0.smithi028.stdout:checking how to convert x86_64-pc-linux-gnu file names to toolchain format... func_convert_file_noop 2024-05-12T22:11:21.671 INFO:tasks.workunit.client.0.smithi028.stdout:checking for /usr/bin/ld option to reload object files... -r 2024-05-12T22:11:21.672 INFO:tasks.workunit.client.0.smithi028.stdout:checking for objdump... objdump 2024-05-12T22:11:21.672 INFO:tasks.workunit.client.0.smithi028.stdout:checking how to recognize dependent libraries... pass_all 2024-05-12T22:11:21.673 INFO:tasks.workunit.client.0.smithi028.stdout:checking for dlltool... no 2024-05-12T22:11:21.673 INFO:tasks.workunit.client.0.smithi028.stdout:checking how to associate runtime and link libraries... printf %s\n 2024-05-12T22:11:21.673 INFO:tasks.workunit.client.0.smithi028.stdout:checking for ar... ar 2024-05-12T22:11:21.821 INFO:tasks.workunit.client.0.smithi028.stdout:checking for archiver @FILE support... @ 2024-05-12T22:11:21.821 INFO:tasks.workunit.client.0.smithi028.stdout:checking for strip... strip 2024-05-12T22:11:21.822 INFO:tasks.workunit.client.0.smithi028.stdout:checking for ranlib... ranlib 2024-05-12T22:11:21.822 INFO:tasks.workunit.client.0.smithi028.stdout:checking for gawk... gawk 2024-05-12T22:11:22.069 INFO:tasks.workunit.client.0.smithi028.stdout:checking command to parse /usr/bin/nm -B output from gcc object... ok 2024-05-12T22:11:22.074 INFO:tasks.workunit.client.0.smithi028.stdout:checking for sysroot... no 2024-05-12T22:11:22.097 INFO:tasks.workunit.client.0.smithi028.stdout:checking for a working dd... /usr/bin/dd 2024-05-12T22:11:22.120 INFO:tasks.workunit.client.0.smithi028.stdout:checking how to truncate binary pipes... /usr/bin/dd bs=4096 count=1 2024-05-12T22:11:22.191 INFO:tasks.workunit.client.0.smithi028.stdout:checking for mt... no 2024-05-12T22:11:22.216 INFO:tasks.workunit.client.0.smithi028.stdout:checking if : is a manifest tool... no 2024-05-12T22:11:22.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:11:21 smithi086 ceph-mon[34297]: pgmap v74: 97 pgs: 97 active+clean; 74 MiB data, 648 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.8 MiB/s wr, 245 op/s 2024-05-12T22:11:22.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:11:21 smithi086 ceph-mon[34297]: scrub queued for path: / 2024-05-12T22:11:22.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:11:21 smithi086 ceph-mon[34297]: scrub summary: idle+waiting paths [/] 2024-05-12T22:11:22.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:11:21 smithi086 ceph-mon[34297]: scrub summary: active paths [/] 2024-05-12T22:11:22.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:11:21 smithi086 ceph-mon[34297]: scrub summary: active paths [/] 2024-05-12T22:11:22.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:11:21 smithi086 ceph-mon[34297]: scrub summary: idle+waiting paths [/] 2024-05-12T22:11:22.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:11:21 smithi086 ceph-mon[34297]: scrub summary: active paths [/] 2024-05-12T22:11:22.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:11:21 smithi086 ceph-mon[34297]: scrub summary: idle+waiting paths [/] 2024-05-12T22:11:22.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:11:21 smithi086 ceph-mon[34297]: scrub summary: active paths [/] 2024-05-12T22:11:22.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:11:21 smithi086 ceph-mon[34297]: scrub summary: idle+waiting paths [/] 2024-05-12T22:11:22.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:11:21 smithi028 ceph-mon[29847]: pgmap v74: 97 pgs: 97 active+clean; 74 MiB data, 648 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.8 MiB/s wr, 245 op/s 2024-05-12T22:11:22.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:11:21 smithi028 ceph-mon[29847]: scrub queued for path: / 2024-05-12T22:11:22.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:11:21 smithi028 ceph-mon[29847]: scrub summary: idle+waiting paths [/] 2024-05-12T22:11:22.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:11:21 smithi028 ceph-mon[29847]: scrub summary: active paths [/] 2024-05-12T22:11:22.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:11:21 smithi028 ceph-mon[29847]: scrub summary: active paths [/] 2024-05-12T22:11:22.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:11:21 smithi028 ceph-mon[29847]: scrub summary: idle+waiting paths [/] 2024-05-12T22:11:22.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:11:21 smithi028 ceph-mon[29847]: scrub summary: active paths [/] 2024-05-12T22:11:22.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:11:21 smithi028 ceph-mon[29847]: scrub summary: idle+waiting paths [/] 2024-05-12T22:11:22.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:11:21 smithi028 ceph-mon[29847]: scrub summary: active paths [/] 2024-05-12T22:11:22.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:11:21 smithi028 ceph-mon[29847]: scrub summary: idle+waiting paths [/] 2024-05-12T22:11:22.371 INFO:tasks.workunit.client.0.smithi028.stdout:checking how to run the C preprocessor... gcc -E 2024-05-12T22:11:22.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:11:21 smithi186 ceph-mon[34785]: pgmap v74: 97 pgs: 97 active+clean; 74 MiB data, 648 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.8 MiB/s wr, 245 op/s 2024-05-12T22:11:22.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:11:21 smithi186 ceph-mon[34785]: scrub queued for path: / 2024-05-12T22:11:22.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:11:21 smithi186 ceph-mon[34785]: scrub summary: idle+waiting paths [/] 2024-05-12T22:11:22.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:11:21 smithi186 ceph-mon[34785]: scrub summary: active paths [/] 2024-05-12T22:11:22.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:11:21 smithi186 ceph-mon[34785]: scrub summary: active paths [/] 2024-05-12T22:11:22.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:11:21 smithi186 ceph-mon[34785]: scrub summary: idle+waiting paths [/] 2024-05-12T22:11:22.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:11:21 smithi186 ceph-mon[34785]: scrub summary: active paths [/] 2024-05-12T22:11:22.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:11:21 smithi186 ceph-mon[34785]: scrub summary: idle+waiting paths [/] 2024-05-12T22:11:22.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:11:21 smithi186 ceph-mon[34785]: scrub summary: active paths [/] 2024-05-12T22:11:22.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:11:21 smithi186 ceph-mon[34785]: scrub summary: idle+waiting paths [/] 2024-05-12T22:11:22.809 INFO:tasks.workunit.client.0.smithi028.stdout:checking for ANSI C header files... yes 2024-05-12T22:11:22.890 INFO:tasks.workunit.client.0.smithi028.stdout:checking for sys/types.h... yes 2024-05-12T22:11:22.985 INFO:tasks.workunit.client.0.smithi028.stdout:checking for sys/stat.h... yes 2024-05-12T22:11:23.080 INFO:tasks.workunit.client.0.smithi028.stdout:checking for stdlib.h... yes 2024-05-12T22:11:23.178 INFO:tasks.workunit.client.0.smithi028.stdout:checking for string.h... yes 2024-05-12T22:11:23.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:11:22 smithi086 ceph-mon[34297]: scrub summary: idle+waiting paths [/] 2024-05-12T22:11:23.269 INFO:tasks.workunit.client.0.smithi028.stdout:checking for memory.h... yes 2024-05-12T22:11:23.284 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:11:22 smithi028 ceph-mon[29847]: scrub summary: idle+waiting paths [/] 2024-05-12T22:11:23.359 INFO:tasks.workunit.client.0.smithi028.stdout:checking for strings.h... yes 2024-05-12T22:11:23.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:11:22 smithi186 ceph-mon[34785]: scrub summary: idle+waiting paths [/] 2024-05-12T22:11:23.454 INFO:tasks.workunit.client.0.smithi028.stdout:checking for inttypes.h... yes 2024-05-12T22:11:23.555 INFO:tasks.workunit.client.0.smithi028.stdout:checking for stdint.h... yes 2024-05-12T22:11:23.655 INFO:tasks.workunit.client.0.smithi028.stdout:checking for unistd.h... yes 2024-05-12T22:11:23.750 INFO:tasks.workunit.client.0.smithi028.stdout:checking for dlfcn.h... yes 2024-05-12T22:11:23.768 INFO:tasks.workunit.client.0.smithi028.stdout:checking for objdir... .libs 2024-05-12T22:11:24.033 INFO:tasks.workunit.client.0.smithi028.stdout:checking if gcc supports -fno-rtti -fno-exceptions... no 2024-05-12T22:11:24.033 INFO:tasks.workunit.client.0.smithi028.stdout:checking for gcc option to produce PIC... -fPIC -DPIC 2024-05-12T22:11:24.115 INFO:tasks.workunit.client.0.smithi028.stdout:checking if gcc PIC flag -fPIC -DPIC works... yes 2024-05-12T22:11:24.208 INFO:tasks.workunit.client.0.smithi028.stdout:checking if gcc static flag -static works... no 2024-05-12T22:11:24.241 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:11:23 smithi086 ceph-mon[34297]: pgmap v75: 97 pgs: 97 active+clean; 74 MiB data, 648 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 181 op/s 2024-05-12T22:11:24.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:11:23 smithi028 ceph-mon[29847]: pgmap v75: 97 pgs: 97 active+clean; 74 MiB data, 648 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 181 op/s 2024-05-12T22:11:24.350 INFO:tasks.workunit.client.0.smithi028.stdout:checking if gcc supports -c -o file.o... yes 2024-05-12T22:11:24.351 INFO:tasks.workunit.client.0.smithi028.stdout:checking if gcc supports -c -o file.o... (cached) yes 2024-05-12T22:11:24.376 INFO:tasks.workunit.client.0.smithi028.stdout:checking whether the gcc linker (/usr/bin/ld -m elf_x86_64) supports shared libraries... yes 2024-05-12T22:11:24.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:11:23 smithi186 ceph-mon[34785]: pgmap v75: 97 pgs: 97 active+clean; 74 MiB data, 648 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 181 op/s 2024-05-12T22:11:24.491 INFO:tasks.workunit.client.0.smithi028.stdout:checking whether -lc should be explicitly linked in... no 2024-05-12T22:11:24.665 INFO:tasks.workunit.client.0.smithi028.stdout:checking dynamic linker characteristics... GNU/Linux ld.so 2024-05-12T22:11:24.665 INFO:tasks.workunit.client.0.smithi028.stdout:checking how to hardcode library paths into programs... immediate 2024-05-12T22:11:24.669 INFO:tasks.workunit.client.0.smithi028.stdout:checking whether stripping libraries is possible... yes 2024-05-12T22:11:24.669 INFO:tasks.workunit.client.0.smithi028.stdout:checking if libtool supports shared libraries... yes 2024-05-12T22:11:24.669 INFO:tasks.workunit.client.0.smithi028.stdout:checking whether to build shared libraries... yes 2024-05-12T22:11:24.669 INFO:tasks.workunit.client.0.smithi028.stdout:checking whether to build static libraries... yes 2024-05-12T22:11:24.690 INFO:tasks.workunit.client.0.smithi028.stdout:checking for gcc... (cached) gcc 2024-05-12T22:11:24.775 INFO:tasks.workunit.client.0.smithi028.stdout:checking whether we are using the GNU C compiler... (cached) yes 2024-05-12T22:11:24.775 INFO:tasks.workunit.client.0.smithi028.stdout:checking whether gcc accepts -g... (cached) yes 2024-05-12T22:11:24.776 INFO:tasks.workunit.client.0.smithi028.stdout:checking for gcc option to accept ISO C89... (cached) none needed 2024-05-12T22:11:24.778 INFO:tasks.workunit.client.0.smithi028.stdout:checking for gmake... /usr/bin/gmake 2024-05-12T22:11:24.778 INFO:tasks.workunit.client.0.smithi028.stderr:./configure: line 11968: test: $(SHELL): binary operator expected 2024-05-12T22:11:24.779 INFO:tasks.workunit.client.0.smithi028.stdout:checking for tar... /bin/tar 2024-05-12T22:11:24.779 INFO:tasks.workunit.client.0.smithi028.stdout:checking for gzip... /bin/gzip 2024-05-12T22:11:24.832 INFO:tasks.workunit.client.0.smithi028.stdout:checking whether gcc -MM is supported... yes 2024-05-12T22:11:24.833 INFO:tasks.workunit.client.0.smithi028.stdout:checking for sort... /bin/sort 2024-05-12T22:11:24.834 INFO:tasks.workunit.client.0.smithi028.stdout:checking whether ln -s works... yes 2024-05-12T22:11:24.835 INFO:tasks.workunit.client.0.smithi028.stdout:checking for rpm... /bin/rpm 2024-05-12T22:11:24.849 INFO:tasks.workunit.client.0.smithi028.stdout:checking for rpmbuild... no 2024-05-12T22:11:24.850 INFO:tasks.workunit.client.0.smithi028.stdout:checking for ANSI C header files... (cached) yes 2024-05-12T22:11:24.969 INFO:tasks.workunit.client.0.smithi028.stdout:checking assert.h usability... yes 2024-05-12T22:11:25.009 INFO:tasks.workunit.client.0.smithi028.stdout:checking assert.h presence... yes 2024-05-12T22:11:25.009 INFO:tasks.workunit.client.0.smithi028.stdout:checking for assert.h... yes 2024-05-12T22:11:25.125 INFO:tasks.workunit.client.0.smithi028.stdout:checking bstring.h usability... no 2024-05-12T22:11:25.188 INFO:tasks.workunit.client.0.smithi028.stdout:checking bstring.h presence... no 2024-05-12T22:11:25.188 INFO:tasks.workunit.client.0.smithi028.stdout:checking for bstring.h... no 2024-05-12T22:11:25.284 INFO:tasks.workunit.client.0.smithi028.stdout:checking libgen.h usability... yes 2024-05-12T22:11:25.322 INFO:tasks.workunit.client.0.smithi028.stdout:checking libgen.h presence... yes 2024-05-12T22:11:25.322 INFO:tasks.workunit.client.0.smithi028.stdout:checking for libgen.h... yes 2024-05-12T22:11:25.445 INFO:tasks.workunit.client.0.smithi028.stdout:checking dirent.h usability... yes 2024-05-12T22:11:25.486 INFO:tasks.workunit.client.0.smithi028.stdout:checking dirent.h presence... yes 2024-05-12T22:11:25.486 INFO:tasks.workunit.client.0.smithi028.stdout:checking for dirent.h... yes 2024-05-12T22:11:25.593 INFO:tasks.workunit.client.0.smithi028.stdout:checking errno.h usability... yes 2024-05-12T22:11:25.634 INFO:tasks.workunit.client.0.smithi028.stdout:checking errno.h presence... yes 2024-05-12T22:11:25.634 INFO:tasks.workunit.client.0.smithi028.stdout:checking for errno.h... yes 2024-05-12T22:11:25.740 INFO:tasks.workunit.client.0.smithi028.stdout:checking malloc.h usability... yes 2024-05-12T22:11:25.782 INFO:tasks.workunit.client.0.smithi028.stdout:checking malloc.h presence... yes 2024-05-12T22:11:25.782 INFO:tasks.workunit.client.0.smithi028.stdout:checking for malloc.h... yes 2024-05-12T22:11:25.903 INFO:tasks.workunit.client.0.smithi028.stdout:checking uuid.h usability... no 2024-05-12T22:11:25.971 INFO:tasks.workunit.client.0.smithi028.stdout:checking uuid.h presence... no 2024-05-12T22:11:25.971 INFO:tasks.workunit.client.0.smithi028.stdout:checking for uuid.h... no 2024-05-12T22:11:26.074 INFO:tasks.workunit.client.0.smithi028.stdout:checking uuid/uuid.h usability... yes 2024-05-12T22:11:26.119 INFO:tasks.workunit.client.0.smithi028.stdout:checking uuid/uuid.h presence... yes 2024-05-12T22:11:26.119 INFO:tasks.workunit.client.0.smithi028.stdout:checking for uuid/uuid.h... yes 2024-05-12T22:11:26.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:11:25 smithi086 ceph-mon[34297]: pgmap v76: 97 pgs: 97 active+clean; 75 MiB data, 652 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.5 MiB/s wr, 191 op/s 2024-05-12T22:11:26.245 INFO:tasks.workunit.client.0.smithi028.stdout:checking sys/uuid.h usability... no 2024-05-12T22:11:26.284 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:11:25 smithi028 ceph-mon[29847]: pgmap v76: 97 pgs: 97 active+clean; 75 MiB data, 652 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.5 MiB/s wr, 191 op/s 2024-05-12T22:11:26.317 INFO:tasks.workunit.client.0.smithi028.stdout:checking sys/uuid.h presence... no 2024-05-12T22:11:26.317 INFO:tasks.workunit.client.0.smithi028.stdout:checking for sys/uuid.h... no 2024-05-12T22:11:26.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:11:25 smithi186 ceph-mon[34785]: pgmap v76: 97 pgs: 97 active+clean; 75 MiB data, 652 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.5 MiB/s wr, 191 op/s 2024-05-12T22:11:26.432 INFO:tasks.workunit.client.0.smithi028.stdout:checking sys/file.h usability... yes 2024-05-12T22:11:26.476 INFO:tasks.workunit.client.0.smithi028.stdout:checking sys/file.h presence... yes 2024-05-12T22:11:26.476 INFO:tasks.workunit.client.0.smithi028.stdout:checking for sys/file.h... yes 2024-05-12T22:11:26.593 INFO:tasks.workunit.client.0.smithi028.stdout:checking sys/fcntl.h usability... yes 2024-05-12T22:11:26.635 INFO:tasks.workunit.client.0.smithi028.stdout:checking sys/fcntl.h presence... yes 2024-05-12T22:11:26.635 INFO:tasks.workunit.client.0.smithi028.stdout:checking for sys/fcntl.h... yes 2024-05-12T22:11:26.758 INFO:tasks.workunit.client.0.smithi028.stdout:checking sys/syssgi.h usability... no 2024-05-12T22:11:26.821 INFO:tasks.workunit.client.0.smithi028.stdout:checking sys/syssgi.h presence... no 2024-05-12T22:11:26.821 INFO:tasks.workunit.client.0.smithi028.stdout:checking for sys/syssgi.h... no 2024-05-12T22:11:26.951 INFO:tasks.workunit.client.0.smithi028.stdout:checking sys/param.h usability... yes 2024-05-12T22:11:27.000 INFO:tasks.workunit.client.0.smithi028.stdout:checking sys/param.h presence... yes 2024-05-12T22:11:27.000 INFO:tasks.workunit.client.0.smithi028.stdout:checking for sys/param.h... yes 2024-05-12T22:11:27.033 INFO:tasks.workunit.client.0.smithi028.stdout:checking for sys/stat.h... (cached) yes 2024-05-12T22:11:27.130 INFO:tasks.workunit.client.0.smithi028.stdout:checking sys/statvfs.h usability... yes 2024-05-12T22:11:27.172 INFO:tasks.workunit.client.0.smithi028.stdout:checking sys/statvfs.h presence... yes 2024-05-12T22:11:27.172 INFO:tasks.workunit.client.0.smithi028.stdout:checking for sys/statvfs.h... yes 2024-05-12T22:11:27.282 INFO:tasks.workunit.client.0.smithi028.stdout:checking sys/time.h usability... yes 2024-05-12T22:11:27.324 INFO:tasks.workunit.client.0.smithi028.stdout:checking sys/time.h presence... yes 2024-05-12T22:11:27.324 INFO:tasks.workunit.client.0.smithi028.stdout:checking for sys/time.h... yes 2024-05-12T22:11:27.424 INFO:tasks.workunit.client.0.smithi028.stdout:checking sys/ioctl.h usability... yes 2024-05-12T22:11:27.467 INFO:tasks.workunit.client.0.smithi028.stdout:checking sys/ioctl.h presence... yes 2024-05-12T22:11:27.468 INFO:tasks.workunit.client.0.smithi028.stdout:checking for sys/ioctl.h... yes 2024-05-12T22:11:27.678 INFO:tasks.workunit.client.0.smithi028.stdout:checking sys/wait.h usability... yes 2024-05-12T22:11:27.721 INFO:tasks.workunit.client.0.smithi028.stdout:checking sys/wait.h presence... yes 2024-05-12T22:11:27.721 INFO:tasks.workunit.client.0.smithi028.stdout:checking for sys/wait.h... yes 2024-05-12T22:11:27.808 INFO:tasks.workunit.client.0.smithi028.stdout:checking for sys/types.h... (cached) yes 2024-05-12T22:11:27.827 INFO:tasks.workunit.client.0.smithi028.stdout:checking for strings.h... (cached) yes 2024-05-12T22:11:27.924 INFO:tasks.workunit.client.0.smithi028.stdout:checking err.h usability... yes 2024-05-12T22:11:27.964 INFO:tasks.workunit.client.0.smithi028.stdout:checking err.h presence... yes 2024-05-12T22:11:27.965 INFO:tasks.workunit.client.0.smithi028.stdout:checking for err.h... yes 2024-05-12T22:11:28.070 INFO:tasks.workunit.client.0.smithi028.stdout:checking linux/falloc.h usability... yes 2024-05-12T22:11:28.108 INFO:tasks.workunit.client.0.smithi028.stdout:checking linux/falloc.h presence... yes 2024-05-12T22:11:28.108 INFO:tasks.workunit.client.0.smithi028.stdout:checking for linux/falloc.h... yes 2024-05-12T22:11:28.216 INFO:tasks.workunit.client.0.smithi028.stdout:checking sys/fs/xfs_fsops.h usability... no 2024-05-12T22:11:28.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:11:27 smithi086 ceph-mon[34297]: pgmap v77: 97 pgs: 97 active+clean; 75 MiB data, 652 MiB used, 1.0 TiB / 1.0 TiB avail; 532 KiB/s rd, 590 KiB/s wr, 99 op/s 2024-05-12T22:11:28.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:11:27 smithi086 ceph-mon[34297]: scrub summary: idle 2024-05-12T22:11:28.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:11:27 smithi086 ceph-mon[34297]: scrub summary: idle 2024-05-12T22:11:28.278 INFO:tasks.workunit.client.0.smithi028.stdout:checking sys/fs/xfs_fsops.h presence... no 2024-05-12T22:11:28.279 INFO:tasks.workunit.client.0.smithi028.stdout:checking for sys/fs/xfs_fsops.h... no 2024-05-12T22:11:28.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:11:27 smithi028 ceph-mon[29847]: pgmap v77: 97 pgs: 97 active+clean; 75 MiB data, 652 MiB used, 1.0 TiB / 1.0 TiB avail; 532 KiB/s rd, 590 KiB/s wr, 99 op/s 2024-05-12T22:11:28.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:11:27 smithi028 ceph-mon[29847]: scrub summary: idle 2024-05-12T22:11:28.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:11:27 smithi028 ceph-mon[29847]: scrub summary: idle 2024-05-12T22:11:28.378 INFO:tasks.workunit.client.0.smithi028.stdout:checking sys/fs/xfs_itable.h usability... no 2024-05-12T22:11:28.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:11:27 smithi186 ceph-mon[34785]: pgmap v77: 97 pgs: 97 active+clean; 75 MiB data, 652 MiB used, 1.0 TiB / 1.0 TiB avail; 532 KiB/s rd, 590 KiB/s wr, 99 op/s 2024-05-12T22:11:28.485 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:11:27 smithi186 ceph-mon[34785]: scrub summary: idle 2024-05-12T22:11:28.486 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:11:27 smithi186 ceph-mon[34785]: scrub summary: idle 2024-05-12T22:11:28.487 INFO:tasks.workunit.client.0.smithi028.stdout:checking sys/fs/xfs_itable.h presence... no 2024-05-12T22:11:28.487 INFO:tasks.workunit.client.0.smithi028.stdout:checking for sys/fs/xfs_itable.h... no 2024-05-12T22:11:28.574 INFO:tasks.workunit.client.0.smithi028.stdout:checking xfs/platform_defs.h usability... no 2024-05-12T22:11:28.626 INFO:tasks.workunit.client.0.smithi028.stdout:checking xfs/platform_defs.h presence... no 2024-05-12T22:11:28.626 INFO:tasks.workunit.client.0.smithi028.stdout:checking for xfs/platform_defs.h... no 2024-05-12T22:11:28.720 INFO:tasks.workunit.client.0.smithi028.stdout:checking btrfs/ioctl.h usability... no 2024-05-12T22:11:28.781 INFO:tasks.workunit.client.0.smithi028.stdout:checking btrfs/ioctl.h presence... no 2024-05-12T22:11:28.781 INFO:tasks.workunit.client.0.smithi028.stdout:checking for btrfs/ioctl.h... no 2024-05-12T22:11:28.876 INFO:tasks.workunit.client.0.smithi028.stdout:checking cifs/ioctl.h usability... no 2024-05-12T22:11:28.941 INFO:tasks.workunit.client.0.smithi028.stdout:checking cifs/ioctl.h presence... no 2024-05-12T22:11:28.943 INFO:tasks.workunit.client.0.smithi028.stdout:checking for cifs/ioctl.h... no 2024-05-12T22:11:29.051 INFO:tasks.workunit.client.0.smithi028.stdout:checking sys/mman.h usability... yes 2024-05-12T22:11:29.097 INFO:tasks.workunit.client.0.smithi028.stdout:checking sys/mman.h presence... yes 2024-05-12T22:11:29.098 INFO:tasks.workunit.client.0.smithi028.stdout:checking for sys/mman.h... yes 2024-05-12T22:11:29.190 INFO:tasks.workunit.client.0.smithi028.stdout:checking for xfs/xfs_log_format.h... no 2024-05-12T22:11:29.320 INFO:tasks.workunit.client.0.smithi028.stdout:checking for uuid_compare... no 2024-05-12T22:11:29.424 INFO:tasks.workunit.client.0.smithi028.stdout:checking for uuid_compare in -luuid... yes 2024-05-12T22:11:29.556 INFO:tasks.workunit.client.0.smithi028.stdout:checking for xfs/xfs.h... yes 2024-05-12T22:11:29.632 INFO:tasks.workunit.client.0.smithi028.stdout:checking for xfs/libxfs.h... no 2024-05-12T22:11:29.731 INFO:tasks.workunit.client.0.smithi028.stdout:checking whether xlog_assign_lsn is declared... no 2024-05-12T22:11:29.826 INFO:tasks.workunit.client.0.smithi028.stdout:checking for xfs/xqm.h... yes 2024-05-12T22:11:29.954 INFO:tasks.workunit.client.0.smithi028.stdout:checking xfsctl from xfs/xfs.h... ok 2024-05-12T22:11:30.048 INFO:tasks.workunit.client.0.smithi028.stdout:checking xfs/handle.h usability... yes 2024-05-12T22:11:30.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:11:29 smithi086 ceph-mon[34297]: pgmap v78: 97 pgs: 97 active+clean; 75 MiB data, 652 MiB used, 1.0 TiB / 1.0 TiB avail; 532 KiB/s rd, 590 KiB/s wr, 99 op/s 2024-05-12T22:11:30.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:11:29 smithi028 ceph-mon[29847]: pgmap v78: 97 pgs: 97 active+clean; 75 MiB data, 652 MiB used, 1.0 TiB / 1.0 TiB avail; 532 KiB/s rd, 590 KiB/s wr, 99 op/s 2024-05-12T22:11:30.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:11:29 smithi186 ceph-mon[34785]: pgmap v78: 97 pgs: 97 active+clean; 75 MiB data, 652 MiB used, 1.0 TiB / 1.0 TiB avail; 532 KiB/s rd, 590 KiB/s wr, 99 op/s 2024-05-12T22:11:31.382 INFO:tasks.workunit.client.0.smithi028.stdout:checking xfs/handle.h presence... yes 2024-05-12T22:11:31.382 INFO:tasks.workunit.client.0.smithi028.stdout:checking for xfs/handle.h... yes 2024-05-12T22:11:31.496 INFO:tasks.workunit.client.0.smithi028.stdout:checking for attr_list_by_handle in -lhandle... yes 2024-05-12T22:11:31.687 INFO:tasks.workunit.client.0.smithi028.stdout:checking attr/attributes.h usability... yes 2024-05-12T22:11:31.728 INFO:tasks.workunit.client.0.smithi028.stdout:checking attr/attributes.h presence... yes 2024-05-12T22:11:31.728 INFO:tasks.workunit.client.0.smithi028.stdout:checking for attr/attributes.h... yes 2024-05-12T22:11:31.865 INFO:tasks.workunit.client.0.smithi028.stdout:checking sys/attributes.h usability... no 2024-05-12T22:11:31.938 INFO:tasks.workunit.client.0.smithi028.stdout:checking sys/attributes.h presence... no 2024-05-12T22:11:31.938 INFO:tasks.workunit.client.0.smithi028.stdout:checking for sys/attributes.h... no 2024-05-12T22:11:32.043 INFO:tasks.workunit.client.0.smithi028.stdout:checking for attr_list in -lattr... yes 2024-05-12T22:11:32.146 INFO:tasks.workunit.client.0.smithi028.stdout:checking for attr_set in -lattr... yes 2024-05-12T22:11:32.235 INFO:tasks.workunit.client.0.smithi028.stdout:checking sys/acl.h usability... yes 2024-05-12T22:11:32.241 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:11:31 smithi086 ceph-mon[34297]: pgmap v79: 97 pgs: 97 active+clean; 78 MiB data, 664 MiB used, 1.0 TiB / 1.0 TiB avail; 747 KiB/s rd, 823 KiB/s wr, 128 op/s 2024-05-12T22:11:32.281 INFO:tasks.workunit.client.0.smithi028.stdout:checking sys/acl.h presence... yes 2024-05-12T22:11:32.282 INFO:tasks.workunit.client.0.smithi028.stdout:checking for sys/acl.h... yes 2024-05-12T22:11:32.284 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:11:31 smithi028 ceph-mon[29847]: pgmap v79: 97 pgs: 97 active+clean; 78 MiB data, 664 MiB used, 1.0 TiB / 1.0 TiB avail; 747 KiB/s rd, 823 KiB/s wr, 128 op/s 2024-05-12T22:11:32.386 INFO:tasks.workunit.client.0.smithi028.stdout:checking acl/libacl.h usability... yes 2024-05-12T22:11:32.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:11:31 smithi186 ceph-mon[34785]: pgmap v79: 97 pgs: 97 active+clean; 78 MiB data, 664 MiB used, 1.0 TiB / 1.0 TiB avail; 747 KiB/s rd, 823 KiB/s wr, 128 op/s 2024-05-12T22:11:32.431 INFO:tasks.workunit.client.0.smithi028.stdout:checking acl/libacl.h presence... yes 2024-05-12T22:11:32.431 INFO:tasks.workunit.client.0.smithi028.stdout:checking for acl/libacl.h... yes 2024-05-12T22:11:32.540 INFO:tasks.workunit.client.0.smithi028.stdout:checking for acl_init in -lacl... yes 2024-05-12T22:11:32.635 INFO:tasks.workunit.client.0.smithi028.stdout:checking gdbm-ndbm.h usability... no 2024-05-12T22:11:32.684 INFO:tasks.workunit.client.0.smithi028.stdout:checking gdbm-ndbm.h presence... no 2024-05-12T22:11:32.684 INFO:tasks.workunit.client.0.smithi028.stdout:checking for gdbm-ndbm.h... no 2024-05-12T22:11:32.759 INFO:tasks.workunit.client.0.smithi028.stdout:checking gdbm/ndbm.h usability... no 2024-05-12T22:11:32.822 INFO:tasks.workunit.client.0.smithi028.stdout:checking gdbm/ndbm.h presence... no 2024-05-12T22:11:32.822 INFO:tasks.workunit.client.0.smithi028.stdout:checking for gdbm/ndbm.h... no 2024-05-12T22:11:32.909 INFO:tasks.workunit.client.0.smithi028.stdout:checking ndbm.h usability... no 2024-05-12T22:11:32.974 INFO:tasks.workunit.client.0.smithi028.stdout:checking ndbm.h presence... no 2024-05-12T22:11:32.974 INFO:tasks.workunit.client.0.smithi028.stdout:checking for ndbm.h... no 2024-05-12T22:11:33.054 INFO:tasks.workunit.client.0.smithi028.stdout:checking libaio.h usability... yes 2024-05-12T22:11:33.100 INFO:tasks.workunit.client.0.smithi028.stdout:checking libaio.h presence... yes 2024-05-12T22:11:33.100 INFO:tasks.workunit.client.0.smithi028.stdout:checking for libaio.h... yes 2024-05-12T22:11:33.218 INFO:tasks.workunit.client.0.smithi028.stdout:checking for dm_make_handle in -ldm... no 2024-05-12T22:11:33.218 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-05-12T22:11:33.218 INFO:tasks.workunit.client.0.smithi028.stdout:WARNING: could not find a valid DMAPI base library. 2024-05-12T22:11:33.218 INFO:tasks.workunit.client.0.smithi028.stdout:If you want DMAPI support please install the data migration 2024-05-12T22:11:33.218 INFO:tasks.workunit.client.0.smithi028.stdout:API (dmapi) library package. Alternatively, run "make install" 2024-05-12T22:11:33.218 INFO:tasks.workunit.client.0.smithi028.stdout:from the dmapi source. 2024-05-12T22:11:33.218 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-05-12T22:11:33.294 INFO:tasks.workunit.client.0.smithi028.stdout:checking linux/fiemap.h usability... yes 2024-05-12T22:11:33.333 INFO:tasks.workunit.client.0.smithi028.stdout:checking linux/fiemap.h presence... yes 2024-05-12T22:11:33.333 INFO:tasks.workunit.client.0.smithi028.stdout:checking for linux/fiemap.h... yes 2024-05-12T22:11:33.435 INFO:tasks.workunit.client.0.smithi028.stdout:checking for fallocate... yes 2024-05-12T22:11:33.540 INFO:tasks.workunit.client.0.smithi028.stdout:checking for open_by_handle_at... yes 2024-05-12T22:11:33.633 INFO:tasks.workunit.client.0.smithi028.stdout:checking sys/prctl.h usability... yes 2024-05-12T22:11:33.675 INFO:tasks.workunit.client.0.smithi028.stdout:checking sys/prctl.h presence... yes 2024-05-12T22:11:33.676 INFO:tasks.workunit.client.0.smithi028.stdout:checking for sys/prctl.h... yes 2024-05-12T22:11:33.761 INFO:tasks.workunit.client.0.smithi028.stdout:checking linux/fs.h usability... yes 2024-05-12T22:11:33.802 INFO:tasks.workunit.client.0.smithi028.stdout:checking linux/fs.h presence... yes 2024-05-12T22:11:33.802 INFO:tasks.workunit.client.0.smithi028.stdout:checking for linux/fs.h... yes 2024-05-12T22:11:33.917 INFO:tasks.workunit.client.0.smithi028.stdout:checking btrfsutil.h usability... no 2024-05-12T22:11:33.985 INFO:tasks.workunit.client.0.smithi028.stdout:checking btrfsutil.h presence... no 2024-05-12T22:11:33.985 INFO:tasks.workunit.client.0.smithi028.stdout:checking for btrfsutil.h... no 2024-05-12T22:11:34.086 INFO:tasks.workunit.client.0.smithi028.stdout:checking for copy_file_range... yes 2024-05-12T22:11:34.194 INFO:tasks.workunit.client.0.smithi028.stdout:checking for renameat2... yes 2024-05-12T22:11:34.231 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:11:33 smithi186 ceph-mon[34785]: pgmap v80: 97 pgs: 97 active+clean; 78 MiB data, 664 MiB used, 1.0 TiB / 1.0 TiB avail; 331 KiB/s rd, 343 KiB/s wr, 37 op/s 2024-05-12T22:11:34.233 INFO:tasks.workunit.client.0.smithi028.stdout:configure: creating ./config.status 2024-05-12T22:11:34.241 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:11:33 smithi086 ceph-mon[34297]: pgmap v80: 97 pgs: 97 active+clean; 78 MiB data, 664 MiB used, 1.0 TiB / 1.0 TiB avail; 331 KiB/s rd, 343 KiB/s wr, 37 op/s 2024-05-12T22:11:34.284 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:11:33 smithi028 ceph-mon[29847]: pgmap v80: 97 pgs: 97 active+clean; 78 MiB data, 664 MiB used, 1.0 TiB / 1.0 TiB avail; 331 KiB/s rd, 343 KiB/s wr, 37 op/s 2024-05-12T22:11:35.993 INFO:tasks.workunit.client.0.smithi028.stdout:config.status: creating include/builddefs 2024-05-12T22:11:36.047 INFO:tasks.workunit.client.0.smithi028.stdout:config.status: creating include/config.h 2024-05-12T22:11:36.084 INFO:tasks.workunit.client.0.smithi028.stdout:config.status: executing libtool commands 2024-05-12T22:11:36.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:11:35 smithi086 ceph-mon[34297]: pgmap v81: 97 pgs: 97 active+clean; 89 MiB data, 717 MiB used, 1.0 TiB / 1.0 TiB avail; 820 KiB/s rd, 1.4 MiB/s wr, 215 op/s 2024-05-12T22:11:36.267 INFO:tasks.workunit.client.0.smithi028.stdout:Building include 2024-05-12T22:11:36.267 INFO:tasks.workunit.client.0.smithi028.stdout:Building lib 2024-05-12T22:11:36.268 INFO:tasks.workunit.client.0.smithi028.stdout:Building tests 2024-05-12T22:11:36.279 INFO:tasks.workunit.client.0.smithi028.stdout: [LTDEP] 2024-05-12T22:11:36.284 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:11:35 smithi028 ceph-mon[29847]: pgmap v81: 97 pgs: 97 active+clean; 89 MiB data, 717 MiB used, 1.0 TiB / 1.0 TiB avail; 820 KiB/s rd, 1.4 MiB/s wr, 215 op/s 2024-05-12T22:11:36.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:11:35 smithi186 ceph-mon[34785]: pgmap v81: 97 pgs: 97 active+clean; 89 MiB data, 717 MiB used, 1.0 TiB / 1.0 TiB avail; 820 KiB/s rd, 1.4 MiB/s wr, 215 op/s 2024-05-12T22:11:36.701 INFO:tasks.workunit.client.0.smithi028.stdout: [CC] dataascii.lo 2024-05-12T22:11:36.702 INFO:tasks.workunit.client.0.smithi028.stdout: [CC] datapid.lo 2024-05-12T22:11:36.703 INFO:tasks.workunit.client.0.smithi028.stdout: [CC] databin.lo 2024-05-12T22:11:36.703 INFO:tasks.workunit.client.0.smithi028.stdout: [CC] file_lock.lo 2024-05-12T22:11:36.944 INFO:tasks.workunit.client.0.smithi028.stdout: [CC] forker.lo 2024-05-12T22:11:36.953 INFO:tasks.workunit.client.0.smithi028.stdout: [CC] pattern.lo 2024-05-12T22:11:37.000 INFO:tasks.workunit.client.0.smithi028.stdout: [CC] open_flags.lo 2024-05-12T22:11:37.017 INFO:tasks.workunit.client.0.smithi028.stdout: [CC] random_range.lo 2024-05-12T22:11:37.203 INFO:tasks.workunit.client.0.smithi028.stdout: [CC] string_to_tokens.lo 2024-05-12T22:11:37.236 INFO:tasks.workunit.client.0.smithi028.stdout: [CC] str_to_bytes.lo 2024-05-12T22:11:37.280 INFO:tasks.workunit.client.0.smithi028.stdout: [CC] tlibio.lo 2024-05-12T22:11:37.428 INFO:tasks.workunit.client.0.smithi028.stdout: [CC] write_log.lo 2024-05-12T22:11:37.438 INFO:tasks.workunit.client.0.smithi028.stdout: [CC] random.lo 2024-05-12T22:11:37.831 INFO:tasks.workunit.client.0.smithi028.stdout: [LD] 2024-05-12T22:11:38.100 INFO:tasks.workunit.client.0.smithi028.stdout:Building ltp 2024-05-12T22:11:38.101 INFO:tasks.workunit.client.0.smithi028.stdout:Building src 2024-05-12T22:11:38.101 INFO:tasks.workunit.client.0.smithi028.stdout:Building m4 2024-05-12T22:11:38.102 INFO:tasks.workunit.client.0.smithi028.stdout:Building common 2024-05-12T22:11:38.114 INFO:tasks.workunit.client.0.smithi028.stdout: [DEP] 2024-05-12T22:11:38.134 INFO:tasks.workunit.client.0.smithi028.stdout: [DEP] 2024-05-12T22:11:38.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:11:37 smithi086 ceph-mon[34297]: pgmap v82: 97 pgs: 97 active+clean; 89 MiB data, 717 MiB used, 1.0 TiB / 1.0 TiB avail; 704 KiB/s rd, 1.3 MiB/s wr, 206 op/s 2024-05-12T22:11:38.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:11:37 smithi028 ceph-mon[29847]: pgmap v82: 97 pgs: 97 active+clean; 89 MiB data, 717 MiB used, 1.0 TiB / 1.0 TiB avail; 704 KiB/s rd, 1.3 MiB/s wr, 206 op/s 2024-05-12T22:11:38.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:11:37 smithi186 ceph-mon[34785]: pgmap v82: 97 pgs: 97 active+clean; 89 MiB data, 717 MiB used, 1.0 TiB / 1.0 TiB avail; 704 KiB/s rd, 1.3 MiB/s wr, 206 op/s 2024-05-12T22:11:38.522 INFO:tasks.workunit.client.0.smithi028.stdout: [CC] doio 2024-05-12T22:11:38.522 INFO:tasks.workunit.client.0.smithi028.stdout: [CC] fsstress 2024-05-12T22:11:38.523 INFO:tasks.workunit.client.0.smithi028.stdout: [CC] fsx 2024-05-12T22:11:38.815 INFO:tasks.workunit.client.0.smithi028.stderr:fsstress.c: In function 'attr_list_path': 2024-05-12T22:11:38.815 INFO:tasks.workunit.client.0.smithi028.stderr:fsstress.c:859:9: warning: 'attr_list' is deprecated: Use listxattr or llistxattr instead [-Wdeprecated-declarations] 2024-05-12T22:11:38.815 INFO:tasks.workunit.client.0.smithi028.stderr: 859 | rval = attr_list(name->path, buffer, buffersize, flags, cursor); 2024-05-12T22:11:38.815 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~ 2024-05-12T22:11:38.815 INFO:tasks.workunit.client.0.smithi028.stderr:In file included from ../src/global.h:25, 2024-05-12T22:11:38.815 INFO:tasks.workunit.client.0.smithi028.stderr: from fsstress.c:12: 2024-05-12T22:11:38.815 INFO:tasks.workunit.client.0.smithi028.stderr:/usr/include/attr/attributes.h:165:12: note: declared here 2024-05-12T22:11:38.815 INFO:tasks.workunit.client.0.smithi028.stderr: 165 | extern int attr_list(const char *__path, char *__buffer, const int __buffersize, 2024-05-12T22:11:38.815 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~~ 2024-05-12T22:11:38.815 INFO:tasks.workunit.client.0.smithi028.stderr:fsstress.c: In function 'attr_remove_path': 2024-05-12T22:11:38.815 INFO:tasks.workunit.client.0.smithi028.stderr:fsstress.c:878:9: warning: 'attr_remove' is deprecated: Use removexattr or lremovexattr instead [-Wdeprecated-declarations] 2024-05-12T22:11:38.815 INFO:tasks.workunit.client.0.smithi028.stderr: 878 | rval = attr_remove(name->path, attrname, flags); 2024-05-12T22:11:38.815 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~ 2024-05-12T22:11:38.816 INFO:tasks.workunit.client.0.smithi028.stderr:In file included from ../src/global.h:25, 2024-05-12T22:11:38.816 INFO:tasks.workunit.client.0.smithi028.stderr: from fsstress.c:12: 2024-05-12T22:11:38.816 INFO:tasks.workunit.client.0.smithi028.stderr:/usr/include/attr/attributes.h:152:12: note: declared here 2024-05-12T22:11:38.816 INFO:tasks.workunit.client.0.smithi028.stderr: 152 | extern int attr_remove (const char *__path, const char *__attrname, 2024-05-12T22:11:38.816 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~~~~ 2024-05-12T22:11:38.816 INFO:tasks.workunit.client.0.smithi028.stderr:fsstress.c: In function 'attr_set_path': 2024-05-12T22:11:38.816 INFO:tasks.workunit.client.0.smithi028.stderr:fsstress.c:898:9: warning: 'attr_set' is deprecated: Use setxattr or lsetxattr instead [-Wdeprecated-declarations] 2024-05-12T22:11:38.816 INFO:tasks.workunit.client.0.smithi028.stderr: 898 | rval = attr_set(name->path, attrname, attrvalue, valuelength, flags); 2024-05-12T22:11:38.816 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~ 2024-05-12T22:11:38.816 INFO:tasks.workunit.client.0.smithi028.stderr:In file included from ../src/global.h:25, 2024-05-12T22:11:38.816 INFO:tasks.workunit.client.0.smithi028.stderr: from fsstress.c:12: 2024-05-12T22:11:38.816 INFO:tasks.workunit.client.0.smithi028.stderr:/usr/include/attr/attributes.h:139:12: note: declared here 2024-05-12T22:11:38.816 INFO:tasks.workunit.client.0.smithi028.stderr: 139 | extern int attr_set (const char *__path, const char *__attrname, 2024-05-12T22:11:38.816 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~ 2024-05-12T22:11:38.827 INFO:tasks.workunit.client.0.smithi028.stderr:fsstress.c: In function 'allocsp_f': 2024-05-12T22:11:38.828 INFO:tasks.workunit.client.0.smithi028.stderr:fsstress.c:2047:32: error: 'XFS_IOC_ALLOCSP64' undeclared (first use in this function); did you mean 'XFS_IOC_RESVSP64'? 2024-05-12T22:11:38.828 INFO:tasks.workunit.client.0.smithi028.stderr: 2047 | e = xfsctl(f.path, fd, XFS_IOC_ALLOCSP64, &fl) < 0 ? errno : 0; 2024-05-12T22:11:38.828 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~~~~~~~~~~ 2024-05-12T22:11:38.828 INFO:tasks.workunit.client.0.smithi028.stderr: | XFS_IOC_RESVSP64 2024-05-12T22:11:38.828 INFO:tasks.workunit.client.0.smithi028.stderr:fsstress.c:2047:32: note: each undeclared identifier is reported only once for each function it appears in 2024-05-12T22:11:38.848 INFO:tasks.workunit.client.0.smithi028.stderr:fsstress.c: In function 'freesp_f': 2024-05-12T22:11:38.848 INFO:tasks.workunit.client.0.smithi028.stderr:fsstress.c:3640:32: error: 'XFS_IOC_FREESP64' undeclared (first use in this function); did you mean 'XFS_IOC_RESVSP64'? 2024-05-12T22:11:38.848 INFO:tasks.workunit.client.0.smithi028.stderr: 3640 | e = xfsctl(f.path, fd, XFS_IOC_FREESP64, &fl) < 0 ? errno : 0; 2024-05-12T22:11:38.848 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~~~~~~~~~ 2024-05-12T22:11:38.849 INFO:tasks.workunit.client.0.smithi028.stderr: | XFS_IOC_RESVSP64 2024-05-12T22:11:38.894 INFO:tasks.workunit.client.0.smithi028.stderr:gmake[3]: *** [Makefile:47: fsstress] Error 1 2024-05-12T22:11:38.894 INFO:tasks.workunit.client.0.smithi028.stderr:gmake[3]: *** Waiting for unfinished jobs.... 2024-05-12T22:11:39.297 INFO:tasks.workunit.client.0.smithi028.stderr:doio.c: In function 'parse_cmdline': 2024-05-12T22:11:39.297 INFO:tasks.workunit.client.0.smithi028.stderr:doio.c:3113:33: warning: '%s' directive argument is null [-Wformat-overflow=] 2024-05-12T22:11:39.297 INFO:tasks.workunit.client.0.smithi028.stderr: 3113 | fprintf(stderr, 2024-05-12T22:11:39.297 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~~~~~~~~ 2024-05-12T22:11:39.297 INFO:tasks.workunit.client.0.smithi028.stderr: 3114 | "%s%s: Illegal -C arg (%s). Must be one of: ", 2024-05-12T22:11:39.297 INFO:tasks.workunit.client.0.smithi028.stderr: | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-05-12T22:11:39.297 INFO:tasks.workunit.client.0.smithi028.stderr: 3115 | Prog, TagName, tok); 2024-05-12T22:11:39.297 INFO:tasks.workunit.client.0.smithi028.stderr: | ~~~~~~~~~~~~~~~~~~~ 2024-05-12T22:11:39.327 INFO:tasks.workunit.client.0.smithi028.stderr:fsx.c: In function 'main': 2024-05-12T22:11:39.327 INFO:tasks.workunit.client.0.smithi028.stderr:fsx.c:2662:33: warning: 'strncpy' specified bound 4096 equals destination size [-Wstringop-truncation] 2024-05-12T22:11:39.327 INFO:tasks.workunit.client.0.smithi028.stderr: 2662 | strncpy(opsfile, optarg, sizeof(opsfile)); 2024-05-12T22:11:39.327 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-05-12T22:11:39.327 INFO:tasks.workunit.client.0.smithi028.stderr:fsx.c:2632:25: warning: 'strncpy' specified bound 1024 equals destination size [-Wstringop-truncation] 2024-05-12T22:11:39.327 INFO:tasks.workunit.client.0.smithi028.stderr: 2632 | strncpy(dname, optarg, sizeof(dname)); 2024-05-12T22:11:39.327 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-05-12T22:11:39.479 INFO:tasks.workunit.client.0.smithi028.stderr:gmake[2]: *** [include/buildrules:30: ltp] Error 2 2024-05-12T22:11:39.480 INFO:tasks.workunit.client.0.smithi028.stderr:gmake[2]: *** Waiting for unfinished jobs.... 2024-05-12T22:11:40.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:11:39 smithi086 ceph-mon[34297]: pgmap v83: 97 pgs: 97 active+clean; 89 MiB data, 717 MiB used, 1.0 TiB / 1.0 TiB avail; 704 KiB/s rd, 1.3 MiB/s wr, 206 op/s 2024-05-12T22:11:40.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:11:39 smithi028 ceph-mon[29847]: pgmap v83: 97 pgs: 97 active+clean; 89 MiB data, 717 MiB used, 1.0 TiB / 1.0 TiB avail; 704 KiB/s rd, 1.3 MiB/s wr, 206 op/s 2024-05-12T22:11:40.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:11:39 smithi186 ceph-mon[34785]: pgmap v83: 97 pgs: 97 active+clean; 89 MiB data, 717 MiB used, 1.0 TiB / 1.0 TiB avail; 704 KiB/s rd, 1.3 MiB/s wr, 206 op/s 2024-05-12T22:11:41.813 INFO:tasks.workunit.client.0.smithi028.stdout: [DEP] 2024-05-12T22:11:41.817 INFO:tasks.workunit.client.0.smithi028.stdout: [DEP] 2024-05-12T22:11:41.817 INFO:tasks.workunit.client.0.smithi028.stdout: [DEP] 2024-05-12T22:11:42.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:11:41 smithi086 ceph-mon[34297]: pgmap v84: 97 pgs: 97 active+clean; 95 MiB data, 736 MiB used, 1.0 TiB / 1.0 TiB avail; 915 KiB/s rd, 1.8 MiB/s wr, 287 op/s 2024-05-12T22:11:42.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:11:41 smithi028 ceph-mon[29847]: pgmap v84: 97 pgs: 97 active+clean; 95 MiB data, 736 MiB used, 1.0 TiB / 1.0 TiB avail; 915 KiB/s rd, 1.8 MiB/s wr, 287 op/s 2024-05-12T22:11:42.366 INFO:tasks.workunit.client.0.smithi028.stdout: [CC] dirstress 2024-05-12T22:11:42.370 INFO:tasks.workunit.client.0.smithi028.stdout: [CC] fill 2024-05-12T22:11:42.370 INFO:tasks.workunit.client.0.smithi028.stdout: [CC] fill2 2024-05-12T22:11:42.374 INFO:tasks.workunit.client.0.smithi028.stdout: [CC] getpagesize 2024-05-12T22:11:42.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:11:41 smithi186 ceph-mon[34785]: pgmap v84: 97 pgs: 97 active+clean; 95 MiB data, 736 MiB used, 1.0 TiB / 1.0 TiB avail; 915 KiB/s rd, 1.8 MiB/s wr, 287 op/s 2024-05-12T22:11:42.661 INFO:tasks.workunit.client.0.smithi028.stdout: [CC] holes 2024-05-12T22:11:42.676 INFO:tasks.workunit.client.0.smithi028.stdout: [CC] lstat64 2024-05-12T22:11:42.776 INFO:tasks.workunit.client.0.smithi028.stdout: [CC] nametest 2024-05-12T22:11:42.786 INFO:tasks.workunit.client.0.smithi028.stdout: [CC] permname 2024-05-12T22:11:43.010 INFO:tasks.workunit.client.0.smithi028.stdout: [CC] randholes 2024-05-12T22:11:43.042 INFO:tasks.workunit.client.0.smithi028.stdout: [CC] runas 2024-05-12T22:11:43.160 INFO:tasks.workunit.client.0.smithi028.stdout: [CC] truncfile 2024-05-12T22:11:43.171 INFO:tasks.workunit.client.0.smithi028.stdout: [CC] usemem 2024-05-12T22:11:43.372 INFO:tasks.workunit.client.0.smithi028.stdout: [CC] mmapcat 2024-05-12T22:11:43.420 INFO:tasks.workunit.client.0.smithi028.stdout: [CC] append_reader 2024-05-12T22:11:43.466 INFO:tasks.workunit.client.0.smithi028.stdout: [CC] append_writer 2024-05-12T22:11:43.486 INFO:tasks.workunit.client.0.smithi028.stdout: [CC] dirperf 2024-05-12T22:11:43.666 INFO:tasks.workunit.client.0.smithi028.stdout: [CC] metaperf 2024-05-12T22:11:43.717 INFO:tasks.workunit.client.0.smithi028.stdout: [CC] devzero 2024-05-12T22:11:43.746 INFO:tasks.workunit.client.0.smithi028.stdout: [CC] feature 2024-05-12T22:11:43.887 INFO:tasks.workunit.client.0.smithi028.stdout: [CC] alloc 2024-05-12T22:11:44.036 INFO:tasks.workunit.client.0.smithi028.stdout: [CC] fault 2024-05-12T22:11:44.103 INFO:tasks.workunit.client.0.smithi028.stdout: [CC] fstest 2024-05-12T22:11:44.110 INFO:tasks.workunit.client.0.smithi028.stdout: [CC] t_access_root 2024-05-12T22:11:44.132 INFO:tasks.workunit.client.0.smithi028.stderr:alloc.c: In function 'main': 2024-05-12T22:11:44.132 INFO:tasks.workunit.client.0.smithi028.stderr:alloc.c:154:1: error: 'bozo' undeclared (first use in this function) 2024-05-12T22:11:44.132 INFO:tasks.workunit.client.0.smithi028.stderr: 154 | bozo! 2024-05-12T22:11:44.132 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~ 2024-05-12T22:11:44.133 INFO:tasks.workunit.client.0.smithi028.stderr:alloc.c:154:1: note: each undeclared identifier is reported only once for each function it appears in 2024-05-12T22:11:44.133 INFO:tasks.workunit.client.0.smithi028.stderr:alloc.c:154:5: error: expected ';' before '!' token 2024-05-12T22:11:44.133 INFO:tasks.workunit.client.0.smithi028.stderr: 154 | bozo! 2024-05-12T22:11:44.133 INFO:tasks.workunit.client.0.smithi028.stderr: | ^ 2024-05-12T22:11:44.133 INFO:tasks.workunit.client.0.smithi028.stderr: | ; 2024-05-12T22:11:44.133 INFO:tasks.workunit.client.0.smithi028.stderr:alloc.c:184:25: error: 'rflag' undeclared (first use in this function); did you mean 'nflag'? 2024-05-12T22:11:44.133 INFO:tasks.workunit.client.0.smithi028.stderr: 184 | rflag = 1; 2024-05-12T22:11:44.133 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~ 2024-05-12T22:11:44.133 INFO:tasks.workunit.client.0.smithi028.stderr: | nflag 2024-05-12T22:11:44.134 INFO:tasks.workunit.client.0.smithi028.stderr:alloc.c:320:5: error: expected ';' before '!' token 2024-05-12T22:11:44.134 INFO:tasks.workunit.client.0.smithi028.stderr: 320 | bozo! 2024-05-12T22:11:44.134 INFO:tasks.workunit.client.0.smithi028.stderr: | ^ 2024-05-12T22:11:44.134 INFO:tasks.workunit.client.0.smithi028.stderr: | ; 2024-05-12T22:11:44.134 INFO:tasks.workunit.client.0.smithi028.stderr:alloc.c:128:25: warning: variable 'f' set but not used [-Wunused-but-set-variable] 2024-05-12T22:11:44.134 INFO:tasks.workunit.client.0.smithi028.stderr: 128 | struct flock64 f; 2024-05-12T22:11:44.134 INFO:tasks.workunit.client.0.smithi028.stderr: | ^ 2024-05-12T22:11:44.141 INFO:tasks.workunit.client.0.smithi028.stderr:gmake[3]: *** [Makefile:83: alloc] Error 1 2024-05-12T22:11:44.141 INFO:tasks.workunit.client.0.smithi028.stderr:gmake[3]: *** Waiting for unfinished jobs.... 2024-05-12T22:11:44.284 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:11:43 smithi028 ceph-mon[29847]: pgmap v85: 97 pgs: 97 active+clean; 95 MiB data, 736 MiB used, 1.0 TiB / 1.0 TiB avail; 700 KiB/s rd, 1.6 MiB/s wr, 259 op/s 2024-05-12T22:11:44.355 INFO:tasks.workunit.client.0.smithi028.stderr:fstest.c: In function 'run_child': 2024-05-12T22:11:44.355 INFO:tasks.workunit.client.0.smithi028.stderr:fstest.c:95:27: warning: '/file' directive writing 5 bytes into a region of size between 1 and 1024 [-Wformat-overflow=] 2024-05-12T22:11:44.355 INFO:tasks.workunit.client.0.smithi028.stderr: 95 | sprintf(fname, "%s/file%d", dir, fnum); 2024-05-12T22:11:44.356 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~ 2024-05-12T22:11:44.356 INFO:tasks.workunit.client.0.smithi028.stderr:fstest.c:95:24: note: directive argument in the range [0, 2147483647] 2024-05-12T22:11:44.356 INFO:tasks.workunit.client.0.smithi028.stderr: 95 | sprintf(fname, "%s/file%d", dir, fnum); 2024-05-12T22:11:44.356 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~~~~ 2024-05-12T22:11:44.356 INFO:tasks.workunit.client.0.smithi028.stderr:fstest.c:95:9: note: 'sprintf' output between 7 and 1039 bytes into a destination of size 1024 2024-05-12T22:11:44.356 INFO:tasks.workunit.client.0.smithi028.stderr: 95 | sprintf(fname, "%s/file%d", dir, fnum); 2024-05-12T22:11:44.356 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-05-12T22:11:44.356 INFO:tasks.workunit.client.0.smithi028.stderr:fstest.c:166:27: warning: '/file' directive writing 5 bytes into a region of size between 1 and 1024 [-Wformat-overflow=] 2024-05-12T22:11:44.356 INFO:tasks.workunit.client.0.smithi028.stderr: 166 | sprintf(fname, "%s/file%d", dir, fnum); 2024-05-12T22:11:44.356 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~ 2024-05-12T22:11:44.356 INFO:tasks.workunit.client.0.smithi028.stderr:fstest.c:166:24: note: directive argument in the range [0, 2147483647] 2024-05-12T22:11:44.356 INFO:tasks.workunit.client.0.smithi028.stderr: 166 | sprintf(fname, "%s/file%d", dir, fnum); 2024-05-12T22:11:44.357 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~~~~ 2024-05-12T22:11:44.357 INFO:tasks.workunit.client.0.smithi028.stderr:fstest.c:166:9: note: 'sprintf' output between 7 and 1039 bytes into a destination of size 1024 2024-05-12T22:11:44.357 INFO:tasks.workunit.client.0.smithi028.stderr: 166 | sprintf(fname, "%s/file%d", dir, fnum); 2024-05-12T22:11:44.357 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-05-12T22:11:44.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:11:43 smithi186 ceph-mon[34785]: pgmap v85: 97 pgs: 97 active+clean; 95 MiB data, 736 MiB used, 1.0 TiB / 1.0 TiB avail; 700 KiB/s rd, 1.6 MiB/s wr, 259 op/s 2024-05-12T22:11:44.465 INFO:tasks.workunit.client.0.smithi028.stderr:gmake[2]: *** [include/buildrules:30: src] Error 2 2024-05-12T22:11:44.465 INFO:tasks.workunit.client.0.smithi028.stderr:make[1]: *** [Makefile:53: default] Error 2 2024-05-12T22:11:44.466 INFO:tasks.workunit.client.0.smithi028.stderr:make: *** [Makefile:51: default] Error 2 2024-05-12T22:11:44.468 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-05-12T22:11:44.469 INFO:tasks.workunit:Stopping ['suites/fsx.sh'] on client.0... 2024-05-12T22:11:44.469 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-05-12T22:11:44.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:11:43 smithi086 ceph-mon[34297]: pgmap v85: 97 pgs: 97 active+clean; 95 MiB data, 736 MiB used, 1.0 TiB / 1.0 TiB avail; 700 KiB/s rd, 1.6 MiB/s wr, 259 op/s 2024-05-12T22:11:45.647 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_c50db29d375ad9ca1a200449f9347b89e25fb278/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph_c50db29d375ad9ca1a200449f9347b89e25fb278/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_c50db29d375ad9ca1a200449f9347b89e25fb278/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test suites/fsx.sh) on smithi028 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c50db29d375ad9ca1a200449f9347b89e25fb278 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsx.sh' 2024-05-12T22:11:45.904 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=e81098b5d28a49e2bee829231cd48db9 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_c50db29d375ad9ca1a200449f9347b89e25fb278/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph_c50db29d375ad9ca1a200449f9347b89e25fb278/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_c50db29d375ad9ca1a200449f9347b89e25fb278/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test suites/fsx.sh) on smithi028 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c50db29d375ad9ca1a200449f9347b89e25fb278 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsx.sh' 2024-05-12T22:11:45.907 DEBUG:teuthology.run_tasks:Unwinding manager full_sequential_finally 2024-05-12T22:11:45.918 INFO:teuthology.task.full_sequential_finally:In full_sequential_finally, running task exec... 2024-05-12T22:11:45.918 INFO:teuthology.task.exec:Executing custom commands... 2024-05-12T22:11:45.919 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi028.front.sepia.ceph.com 2024-05-12T22:11:45.919 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > END_TIME' 2024-05-12T22:11:46.030 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi' 2024-05-12T22:11:46.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:11:45 smithi028 ceph-mon[29847]: pgmap v86: 97 pgs: 97 active+clean; 95 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 709 KiB/s rd, 1.6 MiB/s wr, 264 op/s 2024-05-12T22:11:46.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:11:45 smithi186 ceph-mon[34785]: pgmap v86: 97 pgs: 97 active+clean; 95 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 709 KiB/s rd, 1.6 MiB/s wr, 264 op/s 2024-05-12T22:11:46.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:11:45 smithi086 ceph-mon[34297]: pgmap v86: 97 pgs: 97 active+clean; 95 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 709 KiB/s rd, 1.6 MiB/s wr, 264 op/s 2024-05-12T22:11:48.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:11:48 smithi028 ceph-mon[29847]: pgmap v87: 97 pgs: 97 active+clean; 95 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 594 KiB/s wr, 86 op/s 2024-05-12T22:11:48.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:11:48 smithi186 ceph-mon[34785]: pgmap v87: 97 pgs: 97 active+clean; 95 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 594 KiB/s wr, 86 op/s 2024-05-12T22:11:48.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:11:48 smithi086 ceph-mon[34297]: pgmap v87: 97 pgs: 97 active+clean; 95 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 594 KiB/s wr, 86 op/s 2024-05-12T22:11:50.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:11:50 smithi028 ceph-mon[29847]: pgmap v88: 97 pgs: 97 active+clean; 95 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 594 KiB/s wr, 86 op/s 2024-05-12T22:11:50.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:11:50 smithi186 ceph-mon[34785]: pgmap v88: 97 pgs: 97 active+clean; 95 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 594 KiB/s wr, 86 op/s 2024-05-12T22:11:50.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:11:50 smithi086 ceph-mon[34297]: pgmap v88: 97 pgs: 97 active+clean; 95 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 594 KiB/s wr, 86 op/s 2024-05-12T22:11:51.634 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-12T22:11:51.635 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-12T22:11:51.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.860+0000 7f0838600640 1 -- 172.21.15.28:0/2875622526 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0830109a20 msgr2=0x7f0830109e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:11:51.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.860+0000 7f0838600640 1 --2- 172.21.15.28:0/2875622526 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0830109a20 0x7f0830109e00 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f0824009900 tx=0x7f082402a3f0 comp rx=0 tx=0).stop 2024-05-12T22:11:51.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.860+0000 7f0838600640 1 -- 172.21.15.28:0/2875622526 shutdown_connections 2024-05-12T22:11:51.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.860+0000 7f0838600640 1 --2- 172.21.15.28:0/2875622526 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f08301043d0 0x7f08301108e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:51.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.860+0000 7f0838600640 1 --2- 172.21.15.28:0/2875622526 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0830103a10 0x7f0830103e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:51.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.860+0000 7f0838600640 1 --2- 172.21.15.28:0/2875622526 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0830109a20 0x7f0830109e00 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:51.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.860+0000 7f0838600640 1 -- 172.21.15.28:0/2875622526 >> 172.21.15.28:0/2875622526 conn(0x7f08300ff4a0 msgr2=0x7f08301018c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:11:51.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.860+0000 7f0838600640 1 -- 172.21.15.28:0/2875622526 shutdown_connections 2024-05-12T22:11:51.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.861+0000 7f0838600640 1 -- 172.21.15.28:0/2875622526 wait complete. 2024-05-12T22:11:51.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.861+0000 7f0838600640 1 Processor -- start 2024-05-12T22:11:51.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.862+0000 7f0838600640 1 -- start start 2024-05-12T22:11:51.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.862+0000 7f0838600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0830103a10 0x7f083019a390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:51.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.862+0000 7f0838600640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f08301043d0 0x7f083019a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:51.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.862+0000 7f0838600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0830109a20 0x7f083019ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:51.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.862+0000 7f0838600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0830113e30 con 0x7f0830103a10 2024-05-12T22:11:51.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.862+0000 7f0838600640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f0830113cb0 con 0x7f0830109a20 2024-05-12T22:11:51.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.862+0000 7f0838600640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f0830113fb0 con 0x7f08301043d0 2024-05-12T22:11:51.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.862+0000 7f0835c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0830103a10 0x7f083019a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:51.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.862+0000 7f0835c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0830103a10 0x7f083019a390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48520/0 (socket says 172.21.15.28:48520) 2024-05-12T22:11:51.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.862+0000 7f0836600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0830109a20 0x7f083019ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:51.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.862+0000 7f0835c00640 1 -- 172.21.15.28:0/2153301939 learned_addr learned my addr 172.21.15.28:0/2153301939 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:11:51.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.863+0000 7f0835200640 1 --2- 172.21.15.28:0/2153301939 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f08301043d0 0x7f083019a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:51.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.863+0000 7f0836600640 1 -- 172.21.15.28:0/2153301939 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f08301043d0 msgr2=0x7f083019a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:11:51.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.863+0000 7f0836600640 1 --2- 172.21.15.28:0/2153301939 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f08301043d0 0x7f083019a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:51.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.863+0000 7f0836600640 1 -- 172.21.15.28:0/2153301939 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0830103a10 msgr2=0x7f083019a390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:11:51.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.863+0000 7f0836600640 1 --2- 172.21.15.28:0/2153301939 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0830103a10 0x7f083019a390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:51.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.863+0000 7f0836600640 1 -- 172.21.15.28:0/2153301939 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f083019f380 con 0x7f0830109a20 2024-05-12T22:11:51.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.863+0000 7f0835200640 1 --2- 172.21.15.28:0/2153301939 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f08301043d0 0x7f083019a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:11:51.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.863+0000 7f0836600640 1 --2- 172.21.15.28:0/2153301939 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0830109a20 0x7f083019ec50 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f0828009fd0 tx=0x7f082800ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:11:51.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.863+0000 7f082e000640 1 -- 172.21.15.28:0/2153301939 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f082800ce20 con 0x7f0830109a20 2024-05-12T22:11:51.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.863+0000 7f082e000640 1 -- 172.21.15.28:0/2153301939 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0828005020 con 0x7f0830109a20 2024-05-12T22:11:51.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.863+0000 7f0838600640 1 -- 172.21.15.28:0/2153301939 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f083019f670 con 0x7f0830109a20 2024-05-12T22:11:51.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.864+0000 7f082e000640 1 -- 172.21.15.28:0/2153301939 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0828010720 con 0x7f0830109a20 2024-05-12T22:11:51.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.864+0000 7f0838600640 1 -- 172.21.15.28:0/2153301939 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08301a6f00 con 0x7f0830109a20 2024-05-12T22:11:51.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.867+0000 7f083b5a2740 1 Processor -- start 2024-05-12T22:11:51.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.867+0000 7f082e000640 1 -- 172.21.15.28:0/2153301939 <== mon.2 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7f0828005540 con 0x7f0830109a20 2024-05-12T22:11:51.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.867+0000 7f083b5a2740 1 -- start start 2024-05-12T22:11:51.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.867+0000 7f083b5a2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b16efcdbc0 0x55b16efcdfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:51.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.867+0000 7f083b5a2740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b16efce4e0 0x55b16efd8670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:51.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.867+0000 7f083b5a2740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b16efd8bb0 0x55b16efdafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:51.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.867+0000 7f083b5a2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b16ef962a0 con 0x55b16efcdbc0 2024-05-12T22:11:51.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.867+0000 7f083b5a2740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x55b16ef32b90 con 0x55b16efce4e0 2024-05-12T22:11:51.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.867+0000 7f083b5a2740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55b16eeda5b0 con 0x55b16efd8bb0 2024-05-12T22:11:51.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.868+0000 7f0835200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b16efcdbc0 0x55b16efcdfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:51.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.868+0000 7f0835200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b16efcdbc0 0x55b16efcdfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48536/0 (socket says 172.21.15.28:48536) 2024-05-12T22:11:51.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.868+0000 7f0835200640 1 -- 172.21.15.28:0/540693697 learned_addr learned my addr 172.21.15.28:0/540693697 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:11:51.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.868+0000 7f0835c00640 1 --2- 172.21.15.28:0/540693697 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b16efce4e0 0x55b16efd8670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:51.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.868+0000 7f0835200640 1 --2- 172.21.15.28:0/540693697 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b16efd8bb0 0x55b16efdafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:51.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.868+0000 7f0835200640 1 -- 172.21.15.28:0/540693697 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b16efd8bb0 msgr2=0x55b16efdafa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:11:51.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.868+0000 7f0835200640 1 --2- 172.21.15.28:0/540693697 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b16efd8bb0 0x55b16efdafa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:51.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.868+0000 7f0835200640 1 -- 172.21.15.28:0/540693697 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b16efce4e0 msgr2=0x55b16efd8670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:11:51.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.868+0000 7f0835200640 1 --2- 172.21.15.28:0/540693697 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b16efce4e0 0x55b16efd8670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:51.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.868+0000 7f0835200640 1 -- 172.21.15.28:0/540693697 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b16ed107d0 con 0x55b16efcdbc0 2024-05-12T22:11:51.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.869+0000 7f0835200640 1 --2- 172.21.15.28:0/540693697 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b16efcdbc0 0x55b16efcdfa0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f081c013590 tx=0x7f081c0097d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0ff8d0a6f0eede0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:11:51.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.869+0000 7f082e000640 1 --2- 172.21.15.28:0/2153301939 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f08180a2d00 0x7f08180a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:51.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.869+0000 7f082e000640 1 -- 172.21.15.28:0/2153301939 <== mon.2 v2:172.21.15.86:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f08280cfb20 con 0x7f0830109a20 2024-05-12T22:11:51.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.869+0000 7f0822a00640 1 -- 172.21.15.28:0/540693697 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f081c00f520 con 0x55b16efcdbc0 2024-05-12T22:11:51.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.869+0000 7f0835c00640 1 --2- 172.21.15.28:0/2153301939 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f08180a2d00 0x7f08180a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:51.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.869+0000 7f0822a00640 1 -- 172.21.15.28:0/540693697 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f081c00fba0 con 0x55b16efcdbc0 2024-05-12T22:11:51.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.869+0000 7f0822a00640 1 -- 172.21.15.28:0/540693697 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f081c011ad0 con 0x55b16efcdbc0 2024-05-12T22:11:51.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.870+0000 7f083b5a2740 1 -- 172.21.15.28:0/540693697 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b16efcdbc0 msgr2=0x55b16efcdfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:11:51.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.870+0000 7f083b5a2740 1 --2- 172.21.15.28:0/540693697 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b16efcdbc0 0x55b16efcdfa0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f081c013590 tx=0x7f081c0097d0 comp rx=0 tx=0).stop 2024-05-12T22:11:51.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.870+0000 7f0835c00640 1 --2- 172.21.15.28:0/2153301939 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f08180a2d00 0x7f08180a51c0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x55b16efcf180 tx=0x7f0824005a20 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:11:51.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.870+0000 7f083b5a2740 1 -- 172.21.15.28:0/540693697 shutdown_connections 2024-05-12T22:11:51.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.870+0000 7f083b5a2740 1 --2- 172.21.15.28:0/540693697 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b16efd8bb0 0x55b16efdafa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:51.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.870+0000 7f083b5a2740 1 --2- 172.21.15.28:0/540693697 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b16efce4e0 0x55b16efd8670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:51.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.870+0000 7f083b5a2740 1 --2- 172.21.15.28:0/540693697 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b16efcdbc0 0x55b16efcdfa0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:51.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.870+0000 7f083b5a2740 1 -- 172.21.15.28:0/540693697 >> 172.21.15.28:0/540693697 conn(0x55b16efc9ee0 msgr2=0x55b16efca2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:11:51.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.870+0000 7f083b5a2740 1 -- 172.21.15.28:0/540693697 shutdown_connections 2024-05-12T22:11:51.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.870+0000 7f083b5a2740 1 -- 172.21.15.28:0/540693697 wait complete. 2024-05-12T22:11:51.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.871+0000 7f083b5a2740 1 -- start start 2024-05-12T22:11:51.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.871+0000 7f083b5a2740 1 Processor -- start 2024-05-12T22:11:51.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.871+0000 7f083b5a2740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b16efcdbc0 0x55b16efd8510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:51.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.871+0000 7f083b5a2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b16efce4e0 0x55b16efd5800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:51.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.871+0000 7f083b5a2740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b16efd8bb0 0x55b16efd5d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:51.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.871+0000 7f083b5a2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b16eeed9f0 con 0x55b16efce4e0 2024-05-12T22:11:51.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.871+0000 7f083b5a2740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x55b16ef2eb50 con 0x55b16efcdbc0 2024-05-12T22:11:51.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.871+0000 7f083b5a2740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55b16efdb9a0 con 0x55b16efd8bb0 2024-05-12T22:11:51.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.871+0000 7f083b5a2740 10 client.0 _ll_register_callbacks cb 0x55b16ec45be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:11:51.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.872+0000 7f0835c00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b16efcdbc0 0x55b16efd8510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:51.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.872+0000 7f0835200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b16efce4e0 0x55b16efd5800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:51.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.872+0000 7f0835c00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b16efcdbc0 0x55b16efd8510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:36864/0 (socket says 172.21.15.28:36864) 2024-05-12T22:11:51.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.872+0000 7f0835c00640 1 -- 172.21.15.28:0/660650771 learned_addr learned my addr 172.21.15.28:0/660650771 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:11:51.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.872+0000 7f0836600640 1 --2- 172.21.15.28:0/660650771 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b16efd8bb0 0x55b16efd5d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:51.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.872+0000 7f0835c00640 1 -- 172.21.15.28:0/660650771 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b16efd8bb0 msgr2=0x55b16efd5d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:11:51.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.872+0000 7f0835c00640 1 --2- 172.21.15.28:0/660650771 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b16efd8bb0 0x55b16efd5d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:51.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.872+0000 7f0835c00640 1 -- 172.21.15.28:0/660650771 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b16efce4e0 msgr2=0x55b16efd5800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:11:51.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.872+0000 7f0835c00640 1 --2- 172.21.15.28:0/660650771 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b16efce4e0 0x55b16efd5800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:51.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.872+0000 7f0835c00640 1 -- 172.21.15.28:0/660650771 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b16ec463f0 con 0x55b16efcdbc0 2024-05-12T22:11:51.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.872+0000 7f0835c00640 1 --2- 172.21.15.28:0/660650771 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b16efcdbc0 0x55b16efd8510 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f0824009a20 tx=0x7f0824009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=37234a027f136814 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:11:51.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.872+0000 7f0817400640 10 client.0 ms_handle_connect on v2:172.21.15.86:3300/0 2024-05-12T22:11:51.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.873+0000 7f0817400640 1 -- 172.21.15.28:0/660650771 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f082404e070 con 0x55b16efcdbc0 2024-05-12T22:11:51.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.873+0000 7f0817400640 1 -- 172.21.15.28:0/660650771 <== mon.2 v2:172.21.15.86:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f0824031950 con 0x55b16efcdbc0 2024-05-12T22:11:51.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.873+0000 7f083b5a2740 1 -- 172.21.15.28:0/660650771 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f081c00fba0 con 0x55b16efcdbc0 2024-05-12T22:11:51.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.873+0000 7f0817400640 1 -- 172.21.15.28:0/660650771 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f082405a050 con 0x55b16efcdbc0 2024-05-12T22:11:51.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.873+0000 7f083b5a2740 10 client.25919 fetch_fsmap learned FSMap version 25 2024-05-12T22:11:51.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.873+0000 7f083b5a2740 1 -- 172.21.15.28:0/660650771 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55b16ec060f0 con 0x55b16efcdbc0 2024-05-12T22:11:51.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.873+0000 7f0817400640 1 -- 172.21.15.28:0/660650771 <== mon.2 v2:172.21.15.86:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f081c00fba0 con 0x55b16efcdbc0 2024-05-12T22:11:51.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.874+0000 7f0817400640 1 -- 172.21.15.28:0/660650771 <== mon.2 v2:172.21.15.86:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7f082404ac00 con 0x55b16efcdbc0 2024-05-12T22:11:51.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.874+0000 7f083b5a2740 10 client.25919 fetch_fsmap finished waiting for FSMap version 25 2024-05-12T22:11:51.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.874+0000 7f083b5a2740 10 client.25919 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:11:51.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.874+0000 7f083b5a2740 20 client.25919 populate_metadata read hostname 'smithi028' 2024-05-12T22:11:51.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.874+0000 7f083b5a2740 1 --2- 172.21.15.28:0/660650771 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55b16efe8710 0x55b16efeab60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:51.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.874+0000 7f083b5a2740 4 client.25919 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-12T22:11:51.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.874+0000 7f083b5a2740 1 -- 172.21.15.28:0/660650771 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b16ef402c0 con 0x55b16efe8710 2024-05-12T22:11:51.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.874+0000 7f0835200640 1 --2- 172.21.15.28:0/660650771 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55b16efe8710 0x55b16efeab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:51.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.875+0000 7f0835200640 1 --2- 172.21.15.28:0/660650771 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55b16efe8710 0x55b16efeab60 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=68eaa33ee55e2113 server_cookie=e36ad721e8fc0f97 in_seq=0 out_seq=0 2024-05-12T22:11:51.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.875+0000 7f0817400640 10 client.25919 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:11:51.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.875+0000 7f0817400640 1 -- 172.21.15.28:0/660650771 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b16ef402c0 con 0x55b16efe8710 2024-05-12T22:11:51.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.875+0000 7f0817400640 10 client.25919 handle_command_reply: tid=0 2024-05-12T22:11:51.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.876+0000 7f083b5a2740 1 client.25919 shutdown 2024-05-12T22:11:51.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.876+0000 7f083b5a2740 1 -- 172.21.15.28:0/660650771 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b16efcdbc0 msgr2=0x55b16efd8510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:11:51.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.876+0000 7f083b5a2740 1 --2- 172.21.15.28:0/660650771 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b16efcdbc0 0x55b16efd8510 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f0824009a20 tx=0x7f0824009260 comp rx=0 tx=0).stop 2024-05-12T22:11:51.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.876+0000 7f083b5a2740 1 -- 172.21.15.28:0/660650771 shutdown_connections 2024-05-12T22:11:51.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.876+0000 7f083b5a2740 1 --2- 172.21.15.28:0/660650771 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55b16efe8710 0x55b16efeab60 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:51.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.876+0000 7f083b5a2740 1 --2- 172.21.15.28:0/660650771 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b16efd8bb0 0x55b16efd5d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:51.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.876+0000 7f083b5a2740 1 --2- 172.21.15.28:0/660650771 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b16efce4e0 0x55b16efd5800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:51.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.876+0000 7f083b5a2740 1 --2- 172.21.15.28:0/660650771 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b16efcdbc0 0x55b16efd8510 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:51.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.876+0000 7f083b5a2740 1 -- 172.21.15.28:0/660650771 >> 172.21.15.28:0/660650771 conn(0x55b16efc9ee0 msgr2=0x55b16efcfbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:11:51.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.876+0000 7f0817400640 0 client.25919 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:11:51.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.876+0000 7f083b5a2740 1 -- 172.21.15.28:0/660650771 shutdown_connections 2024-05-12T22:11:51.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.876+0000 7f083b5a2740 1 -- 172.21.15.28:0/660650771 wait complete. 2024-05-12T22:11:51.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.876+0000 7f083b5a2740 20 client.25919 trim_cache size 0 max 16384 2024-05-12T22:11:51.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.895+0000 7f083b5a2740 1 Processor -- start 2024-05-12T22:11:51.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.895+0000 7f083b5a2740 1 -- start start 2024-05-12T22:11:51.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.896+0000 7f083b5a2740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b16efc9ee0 0x55b16efe4740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:51.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.896+0000 7f083b5a2740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b16efcdbc0 0x55b16eff0480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:51.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.896+0000 7f083b5a2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b16efce4e0 0x55b16eff09c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:51.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.896+0000 7f083b5a2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f08301b1fa0 con 0x55b16efce4e0 2024-05-12T22:11:51.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.896+0000 7f083b5a2740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f08301af870 con 0x55b16efcdbc0 2024-05-12T22:11:51.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.896+0000 7f083b5a2740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f08301ad6c0 con 0x55b16efc9ee0 2024-05-12T22:11:51.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.896+0000 7f0836600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b16efce4e0 0x55b16eff09c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:51.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.896+0000 7f0835200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b16efcdbc0 0x55b16eff0480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:51.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.896+0000 7f0835c00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b16efc9ee0 0x55b16efe4740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:51.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.896+0000 7f0835200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b16efcdbc0 0x55b16eff0480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:36870/0 (socket says 172.21.15.28:36870) 2024-05-12T22:11:51.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.896+0000 7f0835200640 1 -- 172.21.15.28:0/3653188201 learned_addr learned my addr 172.21.15.28:0/3653188201 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:11:51.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.896+0000 7f0836600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b16efce4e0 0x55b16eff09c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48552/0 (socket says 172.21.15.28:48552) 2024-05-12T22:11:51.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.896+0000 7f0835200640 1 -- 172.21.15.28:0/3653188201 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b16efc9ee0 msgr2=0x55b16efe4740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:11:51.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.896+0000 7f0835200640 1 --2- 172.21.15.28:0/3653188201 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b16efc9ee0 0x55b16efe4740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:51.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.897+0000 7f0835200640 1 -- 172.21.15.28:0/3653188201 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b16efce4e0 msgr2=0x55b16eff09c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:11:51.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.897+0000 7f0835200640 1 --2- 172.21.15.28:0/3653188201 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b16efce4e0 0x55b16eff09c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:51.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.897+0000 7f0835200640 1 -- 172.21.15.28:0/3653188201 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b16efdb1d0 con 0x55b16efcdbc0 2024-05-12T22:11:51.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.897+0000 7f0835200640 1 --2- 172.21.15.28:0/3653188201 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b16efcdbc0 0x55b16eff0480 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f081c013ee0 tx=0x7f081c011750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9ad74430ea213389 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:11:51.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.897+0000 7f0822000640 1 -- 172.21.15.28:0/3653188201 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f081c0132d0 con 0x55b16efcdbc0 2024-05-12T22:11:51.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.897+0000 7f0822000640 1 -- 172.21.15.28:0/3653188201 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f081c011470 con 0x55b16efcdbc0 2024-05-12T22:11:51.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.897+0000 7f0822000640 1 -- 172.21.15.28:0/3653188201 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f081c029940 con 0x55b16efcdbc0 2024-05-12T22:11:51.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.898+0000 7f083b5a2740 1 -- 172.21.15.28:0/3653188201 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b16efcdbc0 msgr2=0x55b16eff0480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:11:51.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.898+0000 7f083b5a2740 1 --2- 172.21.15.28:0/3653188201 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b16efcdbc0 0x55b16eff0480 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f081c013ee0 tx=0x7f081c011750 comp rx=0 tx=0).stop 2024-05-12T22:11:51.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.898+0000 7f083b5a2740 1 -- 172.21.15.28:0/3653188201 shutdown_connections 2024-05-12T22:11:51.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.898+0000 7f083b5a2740 1 --2- 172.21.15.28:0/3653188201 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b16efce4e0 0x55b16eff09c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:51.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.898+0000 7f083b5a2740 1 --2- 172.21.15.28:0/3653188201 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b16efcdbc0 0x55b16eff0480 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:51.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.898+0000 7f083b5a2740 1 --2- 172.21.15.28:0/3653188201 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b16efc9ee0 0x55b16efe4740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:51.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.898+0000 7f083b5a2740 1 -- 172.21.15.28:0/3653188201 >> 172.21.15.28:0/3653188201 conn(0x55b16efd4460 msgr2=0x55b16efc90e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:11:51.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.898+0000 7f083b5a2740 1 -- 172.21.15.28:0/3653188201 shutdown_connections 2024-05-12T22:11:51.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.898+0000 7f083b5a2740 1 -- 172.21.15.28:0/3653188201 wait complete. 2024-05-12T22:11:51.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.898+0000 7f083b5a2740 1 -- start start 2024-05-12T22:11:51.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.899+0000 7f083b5a2740 1 Processor -- start 2024-05-12T22:11:51.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.899+0000 7f083b5a2740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b16efc9ee0 0x55b16efde9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:51.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.899+0000 7f083b5a2740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b16efcdbc0 0x55b16efdeee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:51.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.899+0000 7f083b5a2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b16efce4e0 0x55b16efdf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:51.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.899+0000 7f083b5a2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f08301add30 con 0x55b16efce4e0 2024-05-12T22:11:51.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.899+0000 7f083b5a2740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f08301af1a0 con 0x55b16efc9ee0 2024-05-12T22:11:51.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.899+0000 7f083b5a2740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f08301ae400 con 0x55b16efcdbc0 2024-05-12T22:11:51.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.899+0000 7f083b5a2740 10 client.0 _ll_register_callbacks cb 0x7f0824009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:11:51.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.900+0000 7f0835c00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b16efc9ee0 0x55b16efde9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:51.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.900+0000 7f0835c00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b16efc9ee0 0x55b16efde9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:36874/0 (socket says 172.21.15.28:36874) 2024-05-12T22:11:51.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.900+0000 7f0835c00640 1 -- 172.21.15.28:0/213987277 learned_addr learned my addr 172.21.15.28:0/213987277 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:11:51.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.900+0000 7f0836600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b16efce4e0 0x55b16efdf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:51.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.900+0000 7f0835200640 1 --2- 172.21.15.28:0/213987277 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b16efcdbc0 0x55b16efdeee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:51.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.900+0000 7f0835c00640 1 -- 172.21.15.28:0/213987277 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b16efcdbc0 msgr2=0x55b16efdeee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:11:51.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.900+0000 7f0835c00640 1 --2- 172.21.15.28:0/213987277 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b16efcdbc0 0x55b16efdeee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:51.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.900+0000 7f0835c00640 1 -- 172.21.15.28:0/213987277 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b16efce4e0 msgr2=0x55b16efdf440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:11:51.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.900+0000 7f0835c00640 1 --2- 172.21.15.28:0/213987277 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b16efce4e0 0x55b16efdf440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:51.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.900+0000 7f0835c00640 1 -- 172.21.15.28:0/213987277 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b16f04dfd0 con 0x55b16efc9ee0 2024-05-12T22:11:51.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.900+0000 7f0836600640 1 --2- 172.21.15.28:0/213987277 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b16efce4e0 0x55b16efdf440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:11:51.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.900+0000 7f0835c00640 1 --2- 172.21.15.28:0/213987277 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b16efc9ee0 0x55b16efde9a0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f082402a430 tx=0x7f082403d870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8dd844312b6b2634 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:11:51.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.900+0000 7f0817400640 10 client.0 ms_handle_connect on v2:172.21.15.86:3300/0 2024-05-12T22:11:51.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.901+0000 7f0817400640 1 -- 172.21.15.28:0/213987277 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f082404e040 con 0x55b16efc9ee0 2024-05-12T22:11:51.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.901+0000 7f083b5a2740 1 -- 172.21.15.28:0/213987277 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f081c011470 con 0x55b16efc9ee0 2024-05-12T22:11:51.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.901+0000 7f0817400640 1 -- 172.21.15.28:0/213987277 <== mon.2 v2:172.21.15.86:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f082403dac0 con 0x55b16efc9ee0 2024-05-12T22:11:51.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.901+0000 7f0817400640 1 -- 172.21.15.28:0/213987277 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f082406c050 con 0x55b16efc9ee0 2024-05-12T22:11:51.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.901+0000 7f083b5a2740 10 client.25925 fetch_fsmap learned FSMap version 25 2024-05-12T22:11:51.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.901+0000 7f083b5a2740 1 -- 172.21.15.28:0/213987277 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55b16f04f2e0 con 0x55b16efc9ee0 2024-05-12T22:11:51.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.901+0000 7f0817400640 1 -- 172.21.15.28:0/213987277 <== mon.2 v2:172.21.15.86:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f081c011470 con 0x55b16efc9ee0 2024-05-12T22:11:51.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.902+0000 7f0817400640 1 -- 172.21.15.28:0/213987277 <== mon.2 v2:172.21.15.86:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7f082404a570 con 0x55b16efc9ee0 2024-05-12T22:11:51.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.902+0000 7f083b5a2740 10 client.25925 fetch_fsmap finished waiting for FSMap version 25 2024-05-12T22:11:51.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.902+0000 7f083b5a2740 10 client.25925 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:11:51.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.902+0000 7f083b5a2740 20 client.25925 populate_metadata read hostname 'smithi028' 2024-05-12T22:11:51.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.902+0000 7f083b5a2740 1 --2- 172.21.15.28:0/213987277 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55b16efd8bb0 0x55b16f051a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:51.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.902+0000 7f083b5a2740 4 client.25925 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-12T22:11:51.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.902+0000 7f083b5a2740 1 -- 172.21.15.28:0/213987277 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55b16ef3a610 con 0x55b16efd8bb0 2024-05-12T22:11:51.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.902+0000 7f0835200640 1 --2- 172.21.15.28:0/213987277 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55b16efd8bb0 0x55b16f051a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:51.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.903+0000 7f0835200640 1 --2- 172.21.15.28:0/213987277 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55b16efd8bb0 0x55b16f051a80 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=41645fad43fa78ae server_cookie=dfae3a04d010ca60 in_seq=0 out_seq=0 2024-05-12T22:11:51.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.903+0000 7f0817400640 10 client.25925 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:11:51.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.903+0000 7f0817400640 1 -- 172.21.15.28:0/213987277 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55b16ef3a610 con 0x55b16efd8bb0 2024-05-12T22:11:51.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.903+0000 7f0817400640 10 client.25925 handle_command_reply: tid=0 2024-05-12T22:11:51.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.903+0000 7f083b5a2740 1 client.25925 shutdown 2024-05-12T22:11:51.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.903+0000 7f083b5a2740 1 -- 172.21.15.28:0/213987277 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b16efc9ee0 msgr2=0x55b16efde9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:11:51.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.903+0000 7f083b5a2740 1 --2- 172.21.15.28:0/213987277 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b16efc9ee0 0x55b16efde9a0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f082402a430 tx=0x7f082403d870 comp rx=0 tx=0).stop 2024-05-12T22:11:51.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.903+0000 7f083b5a2740 1 -- 172.21.15.28:0/213987277 shutdown_connections 2024-05-12T22:11:51.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.903+0000 7f083b5a2740 1 --2- 172.21.15.28:0/213987277 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55b16efd8bb0 0x55b16f051a80 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:51.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.903+0000 7f083b5a2740 1 --2- 172.21.15.28:0/213987277 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b16efce4e0 0x55b16efdf440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:51.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.903+0000 7f083b5a2740 1 --2- 172.21.15.28:0/213987277 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b16efcdbc0 0x55b16efdeee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:51.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.903+0000 7f083b5a2740 1 --2- 172.21.15.28:0/213987277 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b16efc9ee0 0x55b16efde9a0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:51.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.903+0000 7f0817400640 0 client.25925 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:11:51.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.903+0000 7f083b5a2740 1 -- 172.21.15.28:0/213987277 >> 172.21.15.28:0/213987277 conn(0x55b16efd4460 msgr2=0x55b16efd7030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:11:51.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.903+0000 7f083b5a2740 1 -- 172.21.15.28:0/213987277 shutdown_connections 2024-05-12T22:11:51.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.903+0000 7f083b5a2740 1 -- 172.21.15.28:0/213987277 wait complete. 2024-05-12T22:11:51.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.903+0000 7f083b5a2740 20 client.25925 trim_cache size 0 max 16384 2024-05-12T22:11:51.911 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-05-12T22:11:51.912 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-05-12T22:11:51.912 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-05-12T22:11:51.912 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-05-12T22:11:51.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.904+0000 7f0838600640 1 -- 172.21.15.28:0/2153301939 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f08180a2d00 msgr2=0x7f08180a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:11:51.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.904+0000 7f0838600640 1 --2- 172.21.15.28:0/2153301939 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f08180a2d00 0x7f08180a51c0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x55b16efcf180 tx=0x7f0824005a20 comp rx=0 tx=0).stop 2024-05-12T22:11:51.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.904+0000 7f0838600640 1 -- 172.21.15.28:0/2153301939 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0830109a20 msgr2=0x7f083019ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:11:51.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.904+0000 7f0838600640 1 --2- 172.21.15.28:0/2153301939 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0830109a20 0x7f083019ec50 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f0828009fd0 tx=0x7f082800ef10 comp rx=0 tx=0).stop 2024-05-12T22:11:51.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.904+0000 7f0838600640 1 -- 172.21.15.28:0/2153301939 shutdown_connections 2024-05-12T22:11:51.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.904+0000 7f0838600640 1 --2- 172.21.15.28:0/2153301939 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f08180a2d00 0x7f08180a51c0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:51.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.904+0000 7f0838600640 1 --2- 172.21.15.28:0/2153301939 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0830109a20 0x7f083019ec50 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:51.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.904+0000 7f0838600640 1 --2- 172.21.15.28:0/2153301939 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f08301043d0 0x7f083019a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:51.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.904+0000 7f0838600640 1 --2- 172.21.15.28:0/2153301939 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0830103a10 0x7f083019a390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:51.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.904+0000 7f0838600640 1 -- 172.21.15.28:0/2153301939 >> 172.21.15.28:0/2153301939 conn(0x7f08300ff4a0 msgr2=0x7f08300ffbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:11:51.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.904+0000 7f0838600640 1 -- 172.21.15.28:0/2153301939 shutdown_connections 2024-05-12T22:11:51.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:51.904+0000 7f0838600640 1 -- 172.21.15.28:0/2153301939 wait complete. 2024-05-12T22:11:51.923 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-12T22:11:51.923 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-12T22:11:52.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.134+0000 7f8965800640 1 -- 172.21.15.28:0/3079750371 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8960109830 msgr2=0x7f8960109c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:11:52.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.134+0000 7f8965800640 1 --2- 172.21.15.28:0/3079750371 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8960109830 0x7f8960109c10 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f8950009900 tx=0x7f895002a400 comp rx=0 tx=0).stop 2024-05-12T22:11:52.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.134+0000 7f8965800640 1 -- 172.21.15.28:0/3079750371 shutdown_connections 2024-05-12T22:11:52.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.134+0000 7f8965800640 1 --2- 172.21.15.28:0/3079750371 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f89601041e0 0x7f89601106f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:52.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.134+0000 7f8965800640 1 --2- 172.21.15.28:0/3079750371 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8960103820 0x7f8960103ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:52.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.134+0000 7f8965800640 1 --2- 172.21.15.28:0/3079750371 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8960109830 0x7f8960109c10 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:52.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.134+0000 7f8965800640 1 -- 172.21.15.28:0/3079750371 >> 172.21.15.28:0/3079750371 conn(0x7f89600ff2d0 msgr2=0x7f89601016f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:11:52.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.134+0000 7f8965800640 1 -- 172.21.15.28:0/3079750371 shutdown_connections 2024-05-12T22:11:52.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.134+0000 7f8965800640 1 -- 172.21.15.28:0/3079750371 wait complete. 2024-05-12T22:11:52.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.136+0000 7f8965800640 1 Processor -- start 2024-05-12T22:11:52.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.136+0000 7f8965800640 1 -- start start 2024-05-12T22:11:52.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.136+0000 7f8965800640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8960103820 0x7f896019a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:52.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.136+0000 7f8965800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89601041e0 0x7f896019a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:52.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.136+0000 7f8965800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8960109830 0x7f896019e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:52.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.136+0000 7f8965800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8960113c50 con 0x7f89601041e0 2024-05-12T22:11:52.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.136+0000 7f8965800640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f8960113ad0 con 0x7f8960109830 2024-05-12T22:11:52.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.136+0000 7f8965800640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f8960113dd0 con 0x7f8960103820 2024-05-12T22:11:52.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.137+0000 7f8964e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8960109830 0x7f896019e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:52.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.137+0000 7f895fe00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8960103820 0x7f896019a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:52.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.137+0000 7f8964e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8960109830 0x7f896019e9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:36880/0 (socket says 172.21.15.28:36880) 2024-05-12T22:11:52.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.137+0000 7f8964e00640 1 -- 172.21.15.28:0/2347010975 learned_addr learned my addr 172.21.15.28:0/2347010975 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:11:52.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.137+0000 7f895f400640 1 --2- 172.21.15.28:0/2347010975 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89601041e0 0x7f896019a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:52.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.137+0000 7f8964e00640 1 -- 172.21.15.28:0/2347010975 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8960103820 msgr2=0x7f896019a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:11:52.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.137+0000 7f8964e00640 1 --2- 172.21.15.28:0/2347010975 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8960103820 0x7f896019a110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:52.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.137+0000 7f8964e00640 1 -- 172.21.15.28:0/2347010975 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89601041e0 msgr2=0x7f896019a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:11:52.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.137+0000 7f8964e00640 1 --2- 172.21.15.28:0/2347010975 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89601041e0 0x7f896019a650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:52.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.137+0000 7f8964e00640 1 -- 172.21.15.28:0/2347010975 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f896019f100 con 0x7f8960109830 2024-05-12T22:11:52.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.137+0000 7f895fe00640 1 --2- 172.21.15.28:0/2347010975 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8960103820 0x7f896019a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:11:52.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.138+0000 7f8964e00640 1 --2- 172.21.15.28:0/2347010975 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8960109830 0x7f896019e9d0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f895800eaa0 tx=0x7f895800ef60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:11:52.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.138+0000 7f895cc00640 1 -- 172.21.15.28:0/2347010975 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f895800cdb0 con 0x7f8960109830 2024-05-12T22:11:52.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.138+0000 7f895cc00640 1 -- 172.21.15.28:0/2347010975 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8958005020 con 0x7f8960109830 2024-05-12T22:11:52.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.138+0000 7f895cc00640 1 -- 172.21.15.28:0/2347010975 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89580116b0 con 0x7f8960109830 2024-05-12T22:11:52.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.138+0000 7f8965800640 1 -- 172.21.15.28:0/2347010975 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f896019f3f0 con 0x7f8960109830 2024-05-12T22:11:52.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.138+0000 7f8965800640 1 -- 172.21.15.28:0/2347010975 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89601a6c80 con 0x7f8960109830 2024-05-12T22:11:52.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.139+0000 7f8965800640 1 -- 172.21.15.28:0/2347010975 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89600689c0 con 0x7f8960109830 2024-05-12T22:11:52.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.141+0000 7f895cc00640 1 -- 172.21.15.28:0/2347010975 <== mon.2 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7f8958005560 con 0x7f8960109830 2024-05-12T22:11:52.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.143+0000 7f895cc00640 1 --2- 172.21.15.28:0/2347010975 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f89440a2d80 0x7f89440a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:52.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.143+0000 7f895cc00640 1 -- 172.21.15.28:0/2347010975 <== mon.2 v2:172.21.15.86:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f89580cf560 con 0x7f8960109830 2024-05-12T22:11:52.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.143+0000 7f895fe00640 1 --2- 172.21.15.28:0/2347010975 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f89440a2d80 0x7f89440a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:52.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.144+0000 7f895fe00640 1 --2- 172.21.15.28:0/2347010975 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f89440a2d80 0x7f89440a5240 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f8950000c00 tx=0x7f8950005b50 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:11:52.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.147+0000 7f895cc00640 1 -- 172.21.15.28:0/2347010975 <== mon.2 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f8958016030 con 0x7f8960109830 2024-05-12T22:11:52.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:11:52 smithi028 ceph-mon[29847]: pgmap v89: 97 pgs: 97 active+clean; 95 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 230 KiB/s rd, 603 KiB/s wr, 89 op/s 2024-05-12T22:11:52.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.392+0000 7f8965800640 1 -- 172.21.15.28:0/2347010975 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f896002d1c0 con 0x7f8960109830 2024-05-12T22:11:52.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.393+0000 7f895cc00640 1 -- 172.21.15.28:0/2347010975 <== mon.2 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11865 (secure 0 0 0) 0x7f895809bb00 con 0x7f8960109830 2024-05-12T22:11:52.394 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-12T22:11:52.394 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1844141745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1844141745},{"type":"v1","addr":"172.21.15.186:6833","nonce":1844141745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6833/2960763758","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":2960763758},{"type":"v1","addr":"172.21.15.86:6833","nonce":2960763758}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24470,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/4131134219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":4131134219},{"type":"v1","addr":"172.21.15.28:6835","nonce":4131134219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24481,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/1727869064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1727869064},{"type":"v1","addr":"172.21.15.186:6835","nonce":1727869064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24482,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2938260704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2938260704},{"type":"v1","addr":"172.21.15.28:6837","nonce":2938260704}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24494,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3473700910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3473700910},{"type":"v1","addr":"172.21.15.86:6835","nonce":3473700910}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-12T22:06:33.426986+0000","modified":"2024-05-12T22:10:42.498031+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24581,"mds_1":24550,"mds_2":24532},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24499":{"gid":24499,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6837/3600809235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":3600809235},{"type":"v1","addr":"172.21.15.186:6837","nonce":3600809235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"i","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6839/2399610807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":2399610807},{"type":"v1","addr":"172.21.15.186:6839","nonce":2399610807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24530":{"gid":24530,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.86:6837/3782448884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":3782448884},{"type":"v1","addr":"172.21.15.86:6837","nonce":3782448884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24532":{"gid":24532,"name":"g","rank":2,"incarnation":21,"state":"up:active","state_seq":6,"addr":"172.21.15.28:6839/123892158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":123892158},{"type":"v1","addr":"172.21.15.28:6839","nonce":123892158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24550":{"gid":24550,"name":"a","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.28:6841/761183870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":761183870},{"type":"v1","addr":"172.21.15.28:6841","nonce":761183870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24581":{"gid":24581,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6839/1251999130","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6838","nonce":1251999130},{"type":"v1","addr":"172.21.15.86:6839","nonce":1251999130}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24581,"qdb_cluster":[24550,24532,24581]},"id":1}]} 2024-05-12T22:11:52.395 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 25 2024-05-12T22:11:52.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.396+0000 7f8965800640 1 -- 172.21.15.28:0/2347010975 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f89440a2d80 msgr2=0x7f89440a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:11:52.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.396+0000 7f8965800640 1 --2- 172.21.15.28:0/2347010975 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f89440a2d80 0x7f89440a5240 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f8950000c00 tx=0x7f8950005b50 comp rx=0 tx=0).stop 2024-05-12T22:11:52.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.396+0000 7f8965800640 1 -- 172.21.15.28:0/2347010975 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8960109830 msgr2=0x7f896019e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:11:52.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.396+0000 7f8965800640 1 --2- 172.21.15.28:0/2347010975 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8960109830 0x7f896019e9d0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f895800eaa0 tx=0x7f895800ef60 comp rx=0 tx=0).stop 2024-05-12T22:11:52.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.396+0000 7f8965800640 1 -- 172.21.15.28:0/2347010975 shutdown_connections 2024-05-12T22:11:52.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.396+0000 7f8965800640 1 --2- 172.21.15.28:0/2347010975 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f89440a2d80 0x7f89440a5240 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:52.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.396+0000 7f8965800640 1 --2- 172.21.15.28:0/2347010975 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8960109830 0x7f896019e9d0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:52.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.396+0000 7f8965800640 1 --2- 172.21.15.28:0/2347010975 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89601041e0 0x7f896019a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:52.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.396+0000 7f8965800640 1 --2- 172.21.15.28:0/2347010975 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8960103820 0x7f896019a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:52.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.396+0000 7f8965800640 1 -- 172.21.15.28:0/2347010975 >> 172.21.15.28:0/2347010975 conn(0x7f89600ff2d0 msgr2=0x7f89600ff8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:11:52.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.396+0000 7f8965800640 1 -- 172.21.15.28:0/2347010975 shutdown_connections 2024-05-12T22:11:52.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.396+0000 7f8965800640 1 -- 172.21.15.28:0/2347010975 wait complete. 2024-05-12T22:11:52.417 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-05-12T22:11:52.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:11:52 smithi186 ceph-mon[34785]: pgmap v89: 97 pgs: 97 active+clean; 95 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 230 KiB/s rd, 603 KiB/s wr, 89 op/s 2024-05-12T22:11:52.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:11:52 smithi086 ceph-mon[34297]: pgmap v89: 97 pgs: 97 active+clean; 95 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 230 KiB/s rd, 603 KiB/s wr, 89 op/s 2024-05-12T22:11:52.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.635+0000 7f6f0a600640 1 -- 172.21.15.28:0/1646212466 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6f041126b0 msgr2=0x7f6f04114aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:11:52.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.635+0000 7f6f0a600640 1 --2- 172.21.15.28:0/1646212466 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6f041126b0 0x7f6f04114aa0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f6efc009c10 tx=0x7f6efc02a6e0 comp rx=0 tx=0).stop 2024-05-12T22:11:52.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.635+0000 7f6f0a600640 1 -- 172.21.15.28:0/1646212466 shutdown_connections 2024-05-12T22:11:52.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.635+0000 7f6f0a600640 1 --2- 172.21.15.28:0/1646212466 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6f041126b0 0x7f6f04114aa0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:52.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.635+0000 7f6f0a600640 1 --2- 172.21.15.28:0/1646212466 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6f04076680 0x7f6f04112170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:52.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.635+0000 7f6f0a600640 1 --2- 172.21.15.28:0/1646212466 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f04075d60 0x7f6f04076140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:52.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.635+0000 7f6f0a600640 1 -- 172.21.15.28:0/1646212466 >> 172.21.15.28:0/1646212466 conn(0x7f6f040ff4a0 msgr2=0x7f6f041018c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:11:52.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.635+0000 7f6f0a600640 1 -- 172.21.15.28:0/1646212466 shutdown_connections 2024-05-12T22:11:52.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.635+0000 7f6f0a600640 1 -- 172.21.15.28:0/1646212466 wait complete. 2024-05-12T22:11:52.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.636+0000 7f6f0a600640 1 Processor -- start 2024-05-12T22:11:52.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.637+0000 7f6f0a600640 1 -- start start 2024-05-12T22:11:52.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.637+0000 7f6f0a600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f04075d60 0x7f6f0419e730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:52.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.637+0000 7f6f0a600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6f04076680 0x7f6f0419ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:52.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.637+0000 7f6f0a600640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6f041126b0 0x7f6f041a2ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:52.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.637+0000 7f6f0a600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6f041181d0 con 0x7f6f04075d60 2024-05-12T22:11:52.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.637+0000 7f6f0a600640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f6f04118050 con 0x7f6f04076680 2024-05-12T22:11:52.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.637+0000 7f6f0a600640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f6f04118350 con 0x7f6f041126b0 2024-05-12T22:11:52.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.637+0000 7f6f03400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f04075d60 0x7f6f0419e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:52.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.637+0000 7f6f03e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6f041126b0 0x7f6f041a2ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:52.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.637+0000 7f6f03e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6f041126b0 0x7f6f041a2ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:59168/0 (socket says 172.21.15.28:59168) 2024-05-12T22:11:52.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.637+0000 7f6f03e00640 1 -- 172.21.15.28:0/2137569952 learned_addr learned my addr 172.21.15.28:0/2137569952 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:11:52.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.637+0000 7f6f02a00640 1 --2- 172.21.15.28:0/2137569952 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6f04076680 0x7f6f0419ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:52.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.638+0000 7f6f03e00640 1 -- 172.21.15.28:0/2137569952 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6f04076680 msgr2=0x7f6f0419ec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:11:52.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.638+0000 7f6f03e00640 1 --2- 172.21.15.28:0/2137569952 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6f04076680 0x7f6f0419ec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:52.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.638+0000 7f6f03e00640 1 -- 172.21.15.28:0/2137569952 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f04075d60 msgr2=0x7f6f0419e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:11:52.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.638+0000 7f6f03e00640 1 --2- 172.21.15.28:0/2137569952 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f04075d60 0x7f6f0419e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:52.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.638+0000 7f6f03e00640 1 -- 172.21.15.28:0/2137569952 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f041a3720 con 0x7f6f041126b0 2024-05-12T22:11:52.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.638+0000 7f6f03e00640 1 --2- 172.21.15.28:0/2137569952 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6f041126b0 0x7f6f041a2ff0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f6efc02ac00 tx=0x7f6efc0305d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:11:52.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.638+0000 7f6efbe00640 1 -- 172.21.15.28:0/2137569952 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6efc041020 con 0x7f6f041126b0 2024-05-12T22:11:52.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.638+0000 7f6efbe00640 1 -- 172.21.15.28:0/2137569952 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6efc003e50 con 0x7f6f041126b0 2024-05-12T22:11:52.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.638+0000 7f6efbe00640 1 -- 172.21.15.28:0/2137569952 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6efc036040 con 0x7f6f041126b0 2024-05-12T22:11:52.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.639+0000 7f6f0a600640 1 -- 172.21.15.28:0/2137569952 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f041a39b0 con 0x7f6f041126b0 2024-05-12T22:11:52.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.639+0000 7f6f0a600640 1 -- 172.21.15.28:0/2137569952 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f041ab2a0 con 0x7f6f041126b0 2024-05-12T22:11:52.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.642+0000 7f6f0d428740 1 Processor -- start 2024-05-12T22:11:52.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.642+0000 7f6efbe00640 1 -- 172.21.15.28:0/2137569952 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7f6efc044020 con 0x7f6f041126b0 2024-05-12T22:11:52.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.642+0000 7f6f0d428740 1 -- start start 2024-05-12T22:11:52.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.642+0000 7f6f0d428740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5604daf10980 0x5604daf10d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:52.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.643+0000 7f6f0d428740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604daf112a0 0x5604daf1b430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:52.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.643+0000 7f6f0d428740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5604daf1b970 0x5604daf1dd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:52.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.643+0000 7f6f0d428740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5604dae75a30 con 0x5604daf112a0 2024-05-12T22:11:52.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.643+0000 7f6f0d428740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x5604dae1d700 con 0x5604daf10980 2024-05-12T22:11:52.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.643+0000 7f6f0d428740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5604dae719f0 con 0x5604daf1b970 2024-05-12T22:11:52.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.643+0000 7f6f02a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5604daf10980 0x5604daf10d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:52.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.643+0000 7f6f03400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604daf112a0 0x5604daf1b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:52.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.643+0000 7f6f02a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5604daf10980 0x5604daf10d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:36894/0 (socket says 172.21.15.28:36894) 2024-05-12T22:11:52.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.643+0000 7f6f02a00640 1 -- 172.21.15.28:0/743286987 learned_addr learned my addr 172.21.15.28:0/743286987 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:11:52.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.643+0000 7f6f02a00640 1 --2- 172.21.15.28:0/743286987 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5604daf1b970 0x5604daf1dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:52.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.643+0000 7f6f03400640 1 -- 172.21.15.28:0/743286987 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5604daf1b970 msgr2=0x5604daf1dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:11:52.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.643+0000 7f6f03400640 1 --2- 172.21.15.28:0/743286987 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5604daf1b970 0x5604daf1dd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:52.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.643+0000 7f6f03400640 1 -- 172.21.15.28:0/743286987 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5604daf10980 msgr2=0x5604daf10d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:11:52.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.643+0000 7f6f03400640 1 --2- 172.21.15.28:0/743286987 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5604daf10980 0x5604daf10d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:52.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.643+0000 7f6f03400640 1 -- 172.21.15.28:0/743286987 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5604dab893f0 con 0x5604daf112a0 2024-05-12T22:11:52.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.643+0000 7f6f02a00640 1 --2- 172.21.15.28:0/743286987 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5604daf10980 0x5604daf10d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:11:52.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.644+0000 7f6efbe00640 1 --2- 172.21.15.28:0/2137569952 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f6ee80a2d00 0x7f6ee80a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:52.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.644+0000 7f6f03400640 1 --2- 172.21.15.28:0/743286987 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604daf112a0 0x5604daf1b430 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f6ef4009c30 tx=0x7f6ef4011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8442174d0d097677 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:11:52.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.644+0000 7f6efbe00640 1 -- 172.21.15.28:0/2137569952 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f6efc0f00e0 con 0x7f6f041126b0 2024-05-12T22:11:52.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.644+0000 7f6f02a00640 1 --2- 172.21.15.28:0/2137569952 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f6ee80a2d00 0x7f6ee80a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:52.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.644+0000 7f6ef8c00640 1 -- 172.21.15.28:0/743286987 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ef401e070 con 0x5604daf112a0 2024-05-12T22:11:52.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.644+0000 7f6ef8c00640 1 -- 172.21.15.28:0/743286987 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6ef40035f0 con 0x5604daf112a0 2024-05-12T22:11:52.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.644+0000 7f6ef8c00640 1 -- 172.21.15.28:0/743286987 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ef4019690 con 0x5604daf112a0 2024-05-12T22:11:52.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.645+0000 7f6f0d428740 1 -- 172.21.15.28:0/743286987 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604daf112a0 msgr2=0x5604daf1b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:11:52.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.645+0000 7f6f02a00640 1 --2- 172.21.15.28:0/2137569952 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f6ee80a2d00 0x7f6ee80a51c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x5604daf0de30 tx=0x7f6eec009770 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:11:52.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.645+0000 7f6f0d428740 1 --2- 172.21.15.28:0/743286987 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604daf112a0 0x5604daf1b430 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f6ef4009c30 tx=0x7f6ef4011810 comp rx=0 tx=0).stop 2024-05-12T22:11:52.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.645+0000 7f6f0d428740 1 -- 172.21.15.28:0/743286987 shutdown_connections 2024-05-12T22:11:52.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.645+0000 7f6f0d428740 1 --2- 172.21.15.28:0/743286987 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5604daf1b970 0x5604daf1dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:52.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.645+0000 7f6f0d428740 1 --2- 172.21.15.28:0/743286987 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604daf112a0 0x5604daf1b430 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:52.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.645+0000 7f6f0d428740 1 --2- 172.21.15.28:0/743286987 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5604daf10980 0x5604daf10d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:52.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.645+0000 7f6f0d428740 1 -- 172.21.15.28:0/743286987 >> 172.21.15.28:0/743286987 conn(0x5604daf0cca0 msgr2=0x5604daf0d080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:11:52.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.645+0000 7f6f0d428740 1 -- 172.21.15.28:0/743286987 shutdown_connections 2024-05-12T22:11:52.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.645+0000 7f6f0d428740 1 -- 172.21.15.28:0/743286987 wait complete. 2024-05-12T22:11:52.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.646+0000 7f6f0d428740 1 -- start start 2024-05-12T22:11:52.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.646+0000 7f6f0d428740 1 Processor -- start 2024-05-12T22:11:52.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.646+0000 7f6f0d428740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5604daf10980 0x5604daf18930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:52.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.646+0000 7f6f0d428740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5604daf112a0 0x5604daf18e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:52.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.646+0000 7f6f0d428740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604daf1b970 0x5604daf193b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:52.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.646+0000 7f6f0d428740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5604daf1e760 con 0x5604daf1b970 2024-05-12T22:11:52.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.646+0000 7f6f0d428740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x5604dae30a70 con 0x5604daf112a0 2024-05-12T22:11:52.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.646+0000 7f6f0d428740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5604daf1e8e0 con 0x5604daf10980 2024-05-12T22:11:52.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.646+0000 7f6f0d428740 10 client.0 _ll_register_callbacks cb 0x5604dae13ba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:11:52.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.647+0000 7f6f02a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5604daf112a0 0x5604daf18e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:52.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.647+0000 7f6f03e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604daf1b970 0x5604daf193b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:52.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.647+0000 7f6f03400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5604daf10980 0x5604daf18930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:52.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.647+0000 7f6f02a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5604daf112a0 0x5604daf18e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:36904/0 (socket says 172.21.15.28:36904) 2024-05-12T22:11:52.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.647+0000 7f6f02a00640 1 -- 172.21.15.28:0/1539823015 learned_addr learned my addr 172.21.15.28:0/1539823015 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:11:52.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.647+0000 7f6f03400640 1 -- 172.21.15.28:0/1539823015 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5604daf112a0 msgr2=0x5604daf18e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:11:52.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.647+0000 7f6f03400640 1 --2- 172.21.15.28:0/1539823015 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5604daf112a0 0x5604daf18e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:52.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.647+0000 7f6f03400640 1 -- 172.21.15.28:0/1539823015 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604daf1b970 msgr2=0x5604daf193b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:11:52.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.647+0000 7f6f03400640 1 --2- 172.21.15.28:0/1539823015 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604daf1b970 0x5604daf193b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:52.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.647+0000 7f6f03400640 1 -- 172.21.15.28:0/1539823015 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5604dac537d0 con 0x5604daf10980 2024-05-12T22:11:52.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.647+0000 7f6f03400640 1 --2- 172.21.15.28:0/1539823015 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5604daf10980 0x5604daf18930 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f6ef4011d10 tx=0x7f6ef4004480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=12554e2573a61bcc server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:11:52.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.647+0000 7f6ef1600640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-12T22:11:52.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.648+0000 7f6ef1600640 1 -- 172.21.15.28:0/1539823015 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ef4027020 con 0x5604daf10980 2024-05-12T22:11:52.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.648+0000 7f6ef1600640 1 -- 172.21.15.28:0/1539823015 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f6ef40039a0 con 0x5604daf10980 2024-05-12T22:11:52.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.648+0000 7f6f0d428740 1 -- 172.21.15.28:0/1539823015 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6ef40035f0 con 0x5604daf10980 2024-05-12T22:11:52.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.648+0000 7f6ef1600640 1 -- 172.21.15.28:0/1539823015 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f6ef4032050 con 0x5604daf10980 2024-05-12T22:11:52.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.648+0000 7f6f0d428740 10 client.25897 fetch_fsmap learned FSMap version 25 2024-05-12T22:11:52.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.648+0000 7f6f0d428740 1 -- 172.21.15.28:0/1539823015 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5604daf2aa60 con 0x5604daf10980 2024-05-12T22:11:52.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.648+0000 7f6ef1600640 1 -- 172.21.15.28:0/1539823015 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6ef40035f0 con 0x5604daf10980 2024-05-12T22:11:52.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.649+0000 7f6ef1600640 1 -- 172.21.15.28:0/1539823015 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7f6ef4004980 con 0x5604daf10980 2024-05-12T22:11:52.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.649+0000 7f6f0d428740 10 client.25897 fetch_fsmap finished waiting for FSMap version 25 2024-05-12T22:11:52.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.649+0000 7f6f0d428740 10 client.25897 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.g 2024-05-12T22:11:52.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.649+0000 7f6f0d428740 20 client.25897 populate_metadata read hostname 'smithi028' 2024-05-12T22:11:52.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.649+0000 7f6f0d428740 1 --2- 172.21.15.28:0/1539823015 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x5604daf2bec0 0x5604daf2e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:52.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.649+0000 7f6f0d428740 4 client.25897 mds_command: new command op to 24532 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-12T22:11:52.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.649+0000 7f6f0d428740 1 -- 172.21.15.28:0/1539823015 --> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5604dae83160 con 0x5604daf2bec0 2024-05-12T22:11:52.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.649+0000 7f6f02a00640 1 --2- 172.21.15.28:0/1539823015 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x5604daf2bec0 0x5604daf2e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:52.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.650+0000 7f6f02a00640 1 --2- 172.21.15.28:0/1539823015 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x5604daf2bec0 0x5604daf2e2b0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=397038f607971753 server_cookie=ea6ce5647ce07f26 in_seq=0 out_seq=0 2024-05-12T22:11:52.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.650+0000 7f6ef1600640 10 client.25897 ms_handle_connect on v2:172.21.15.28:6838/123892158 2024-05-12T22:11:52.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.651+0000 7f6ef1600640 1 -- 172.21.15.28:0/1539823015 <== mds.2 v2:172.21.15.28:6838/123892158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5604dae83160 con 0x5604daf2bec0 2024-05-12T22:11:52.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.651+0000 7f6ef1600640 10 client.25897 handle_command_reply: tid=0 2024-05-12T22:11:52.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.651+0000 7f6f0d428740 1 client.25897 shutdown 2024-05-12T22:11:52.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.651+0000 7f6f0d428740 1 -- 172.21.15.28:0/1539823015 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5604daf10980 msgr2=0x5604daf18930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:11:52.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.651+0000 7f6f0d428740 1 --2- 172.21.15.28:0/1539823015 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5604daf10980 0x5604daf18930 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f6ef4011d10 tx=0x7f6ef4004480 comp rx=0 tx=0).stop 2024-05-12T22:11:52.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.651+0000 7f6f0d428740 1 -- 172.21.15.28:0/1539823015 shutdown_connections 2024-05-12T22:11:52.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.651+0000 7f6f0d428740 1 --2- 172.21.15.28:0/1539823015 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604daf1b970 0x5604daf193b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:52.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.651+0000 7f6f0d428740 1 --2- 172.21.15.28:0/1539823015 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x5604daf2bec0 0x5604daf2e2b0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:52.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.651+0000 7f6f0d428740 1 --2- 172.21.15.28:0/1539823015 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5604daf112a0 0x5604daf18e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:52.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.651+0000 7f6f0d428740 1 --2- 172.21.15.28:0/1539823015 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5604daf10980 0x5604daf18930 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:52.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.651+0000 7f6f0d428740 1 -- 172.21.15.28:0/1539823015 >> 172.21.15.28:0/1539823015 conn(0x5604daf0cca0 msgr2=0x5604daf12960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:11:52.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.651+0000 7f6ef1600640 0 client.25897 ms_handle_reset on v2:172.21.15.28:6838/123892158 2024-05-12T22:11:52.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.651+0000 7f6f0d428740 1 -- 172.21.15.28:0/1539823015 shutdown_connections 2024-05-12T22:11:52.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.651+0000 7f6f0d428740 1 -- 172.21.15.28:0/1539823015 wait complete. 2024-05-12T22:11:52.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.651+0000 7f6f0d428740 20 client.25897 trim_cache size 0 max 16384 2024-05-12T22:11:52.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.666+0000 7f6f0d428740 1 Processor -- start 2024-05-12T22:11:52.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.666+0000 7f6f0d428740 1 -- start start 2024-05-12T22:11:52.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.667+0000 7f6f0d428740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5604daf10980 0x5604daf27650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:52.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.667+0000 7f6f0d428740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5604daf112a0 0x5604daf33aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:52.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.667+0000 7f6f0d428740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604daf1b970 0x5604daf33fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:52.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.667+0000 7f6f0d428740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6f041b6490 con 0x5604daf1b970 2024-05-12T22:11:52.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.667+0000 7f6f0d428740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f6f041b3d60 con 0x5604daf112a0 2024-05-12T22:11:52.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.667+0000 7f6f0d428740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f6f041b1bb0 con 0x5604daf10980 2024-05-12T22:11:52.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.667+0000 7f6f03e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604daf1b970 0x5604daf33fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:52.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.667+0000 7f6f03400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5604daf10980 0x5604daf27650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:52.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.667+0000 7f6f03e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604daf1b970 0x5604daf33fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48642/0 (socket says 172.21.15.28:48642) 2024-05-12T22:11:52.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.667+0000 7f6f03400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5604daf10980 0x5604daf27650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:59204/0 (socket says 172.21.15.28:59204) 2024-05-12T22:11:52.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.667+0000 7f6f03e00640 1 -- 172.21.15.28:0/3900580336 learned_addr learned my addr 172.21.15.28:0/3900580336 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:11:52.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.667+0000 7f6f02a00640 1 --2- 172.21.15.28:0/3900580336 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5604daf112a0 0x5604daf33aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:52.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.668+0000 7f6f03400640 1 -- 172.21.15.28:0/3900580336 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5604daf112a0 msgr2=0x5604daf33aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:11:52.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.668+0000 7f6f03400640 1 --2- 172.21.15.28:0/3900580336 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5604daf112a0 0x5604daf33aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:52.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.668+0000 7f6f03400640 1 -- 172.21.15.28:0/3900580336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604daf1b970 msgr2=0x5604daf33fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:11:52.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.668+0000 7f6f03400640 1 --2- 172.21.15.28:0/3900580336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604daf1b970 0x5604daf33fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:52.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.668+0000 7f6f03400640 1 -- 172.21.15.28:0/3900580336 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5604daf083f0 con 0x5604daf10980 2024-05-12T22:11:52.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.668+0000 7f6f03e00640 1 --2- 172.21.15.28:0/3900580336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604daf1b970 0x5604daf33fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:11:52.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.668+0000 7f6f03400640 1 --2- 172.21.15.28:0/3900580336 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5604daf10980 0x5604daf27650 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f6ef400a010 tx=0x7f6ef4011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b8190e188977669c server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:11:52.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.668+0000 7f6ef3e00640 1 -- 172.21.15.28:0/3900580336 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ef4027040 con 0x5604daf10980 2024-05-12T22:11:52.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.668+0000 7f6ef3e00640 1 -- 172.21.15.28:0/3900580336 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6ef4003ad0 con 0x5604daf10980 2024-05-12T22:11:52.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.668+0000 7f6ef3e00640 1 -- 172.21.15.28:0/3900580336 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ef401e040 con 0x5604daf10980 2024-05-12T22:11:52.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.669+0000 7f6f0d428740 1 -- 172.21.15.28:0/3900580336 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5604daf10980 msgr2=0x5604daf27650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:11:52.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.669+0000 7f6f0d428740 1 --2- 172.21.15.28:0/3900580336 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5604daf10980 0x5604daf27650 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f6ef400a010 tx=0x7f6ef4011810 comp rx=0 tx=0).stop 2024-05-12T22:11:52.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.669+0000 7f6f0d428740 1 -- 172.21.15.28:0/3900580336 shutdown_connections 2024-05-12T22:11:52.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.669+0000 7f6f0d428740 1 --2- 172.21.15.28:0/3900580336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604daf1b970 0x5604daf33fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:52.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.669+0000 7f6f0d428740 1 --2- 172.21.15.28:0/3900580336 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5604daf112a0 0x5604daf33aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:52.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.669+0000 7f6f0d428740 1 --2- 172.21.15.28:0/3900580336 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5604daf10980 0x5604daf27650 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:52.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.669+0000 7f6f0d428740 1 -- 172.21.15.28:0/3900580336 >> 172.21.15.28:0/3900580336 conn(0x5604daf0cca0 msgr2=0x5604daf17980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:11:52.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.669+0000 7f6f0d428740 1 -- 172.21.15.28:0/3900580336 shutdown_connections 2024-05-12T22:11:52.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.669+0000 7f6f0d428740 1 -- 172.21.15.28:0/3900580336 wait complete. 2024-05-12T22:11:52.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.670+0000 7f6f0d428740 1 -- start start 2024-05-12T22:11:52.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.670+0000 7f6f0d428740 1 Processor -- start 2024-05-12T22:11:52.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.670+0000 7f6f0d428740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604daf10980 0x5604daf23d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:52.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.670+0000 7f6f0d428740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5604daf112a0 0x5604daf151c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:52.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.670+0000 7f6f0d428740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5604daf1b970 0x5604daf15700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:52.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.670+0000 7f6f0d428740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6f041b2220 con 0x5604daf10980 2024-05-12T22:11:52.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.670+0000 7f6f0d428740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f6f041b3690 con 0x5604daf1b970 2024-05-12T22:11:52.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.670+0000 7f6f0d428740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f6f041b28f0 con 0x5604daf112a0 2024-05-12T22:11:52.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.671+0000 7f6f0d428740 10 client.0 _ll_register_callbacks cb 0x7f6eec009e20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:11:52.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.671+0000 7f6f03400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604daf10980 0x5604daf23d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:52.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.671+0000 7f6f03400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604daf10980 0x5604daf23d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48648/0 (socket says 172.21.15.28:48648) 2024-05-12T22:11:52.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.671+0000 7f6f03400640 1 -- 172.21.15.28:0/567096738 learned_addr learned my addr 172.21.15.28:0/567096738 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:11:52.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.671+0000 7f6f03e00640 1 --2- 172.21.15.28:0/567096738 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5604daf1b970 0x5604daf15700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:52.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.671+0000 7f6f02a00640 1 --2- 172.21.15.28:0/567096738 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5604daf112a0 0x5604daf151c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:52.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.671+0000 7f6f03400640 1 -- 172.21.15.28:0/567096738 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5604daf112a0 msgr2=0x5604daf151c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:11:52.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.671+0000 7f6f03400640 1 --2- 172.21.15.28:0/567096738 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5604daf112a0 0x5604daf151c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:52.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.671+0000 7f6f03400640 1 -- 172.21.15.28:0/567096738 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5604daf1b970 msgr2=0x5604daf15700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:11:52.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.671+0000 7f6f03400640 1 --2- 172.21.15.28:0/567096738 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5604daf1b970 0x5604daf15700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:52.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.671+0000 7f6f03400640 1 -- 172.21.15.28:0/567096738 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5604daf15cc0 con 0x5604daf10980 2024-05-12T22:11:52.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.672+0000 7f6f03400640 1 --2- 172.21.15.28:0/567096738 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604daf10980 0x5604daf23d50 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f6ef4004ec0 tx=0x7f6ef4003fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4491a22f58183b84 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:11:52.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.672+0000 7f6ef1600640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-12T22:11:52.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.672+0000 7f6ef1600640 1 -- 172.21.15.28:0/567096738 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ef4045060 con 0x5604daf10980 2024-05-12T22:11:52.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.672+0000 7f6ef1600640 1 -- 172.21.15.28:0/567096738 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f6ef40045a0 con 0x5604daf10980 2024-05-12T22:11:52.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.672+0000 7f6f0d428740 1 -- 172.21.15.28:0/567096738 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6ef4003ad0 con 0x5604daf10980 2024-05-12T22:11:52.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.672+0000 7f6ef1600640 1 -- 172.21.15.28:0/567096738 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f6ef4027070 con 0x5604daf10980 2024-05-12T22:11:52.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.672+0000 7f6f0d428740 10 client.15822 fetch_fsmap learned FSMap version 25 2024-05-12T22:11:52.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.672+0000 7f6f0d428740 1 -- 172.21.15.28:0/567096738 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5604daf920a0 con 0x5604daf10980 2024-05-12T22:11:52.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.672+0000 7f6ef1600640 1 -- 172.21.15.28:0/567096738 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6ef4003ad0 con 0x5604daf10980 2024-05-12T22:11:52.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.673+0000 7f6ef1600640 1 -- 172.21.15.28:0/567096738 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7f6ef40053a0 con 0x5604daf10980 2024-05-12T22:11:52.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.673+0000 7f6f0d428740 10 client.15822 fetch_fsmap finished waiting for FSMap version 25 2024-05-12T22:11:52.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.673+0000 7f6f0d428740 10 client.15822 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.g 2024-05-12T22:11:52.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.673+0000 7f6f0d428740 20 client.15822 populate_metadata read hostname 'smithi028' 2024-05-12T22:11:52.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.673+0000 7f6f0d428740 1 --2- 172.21.15.28:0/567096738 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x5604daf92700 0x5604daf94af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:52.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.673+0000 7f6f0d428740 4 client.15822 mds_command: new command op to 24532 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-12T22:11:52.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.673+0000 7f6f0d428740 1 -- 172.21.15.28:0/567096738 --> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5604dae7d4b0 con 0x5604daf92700 2024-05-12T22:11:52.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.673+0000 7f6f02a00640 1 --2- 172.21.15.28:0/567096738 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x5604daf92700 0x5604daf94af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:52.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.674+0000 7f6f02a00640 1 --2- 172.21.15.28:0/567096738 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x5604daf92700 0x5604daf94af0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=168c76dce56dcbaa server_cookie=9059bf2626c41298 in_seq=0 out_seq=0 2024-05-12T22:11:52.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.674+0000 7f6ef1600640 10 client.15822 ms_handle_connect on v2:172.21.15.28:6838/123892158 2024-05-12T22:11:52.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.674+0000 7f6ef1600640 1 -- 172.21.15.28:0/567096738 <== mds.2 v2:172.21.15.28:6838/123892158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5604dae7d4b0 con 0x5604daf92700 2024-05-12T22:11:52.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.674+0000 7f6ef1600640 10 client.15822 handle_command_reply: tid=0 2024-05-12T22:11:52.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.674+0000 7f6f0d428740 1 client.15822 shutdown 2024-05-12T22:11:52.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.674+0000 7f6f0d428740 1 -- 172.21.15.28:0/567096738 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604daf10980 msgr2=0x5604daf23d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:11:52.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.674+0000 7f6f0d428740 1 --2- 172.21.15.28:0/567096738 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604daf10980 0x5604daf23d50 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f6ef4004ec0 tx=0x7f6ef4003fd0 comp rx=0 tx=0).stop 2024-05-12T22:11:52.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.674+0000 7f6f0d428740 1 -- 172.21.15.28:0/567096738 shutdown_connections 2024-05-12T22:11:52.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.674+0000 7f6f0d428740 1 --2- 172.21.15.28:0/567096738 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x5604daf92700 0x5604daf94af0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:52.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.674+0000 7f6f0d428740 1 --2- 172.21.15.28:0/567096738 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5604daf1b970 0x5604daf15700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:52.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.674+0000 7f6f0d428740 1 --2- 172.21.15.28:0/567096738 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5604daf112a0 0x5604daf151c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:52.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.674+0000 7f6f0d428740 1 --2- 172.21.15.28:0/567096738 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604daf10980 0x5604daf23d50 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:52.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.674+0000 7f6ef1600640 0 client.15822 ms_handle_reset on v2:172.21.15.28:6838/123892158 2024-05-12T22:11:52.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.674+0000 7f6f0d428740 1 -- 172.21.15.28:0/567096738 >> 172.21.15.28:0/567096738 conn(0x5604daf0cca0 msgr2=0x5604daf0b530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:11:52.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.675+0000 7f6f0d428740 1 -- 172.21.15.28:0/567096738 shutdown_connections 2024-05-12T22:11:52.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.675+0000 7f6f0d428740 1 -- 172.21.15.28:0/567096738 wait complete. 2024-05-12T22:11:52.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.675+0000 7f6f0d428740 20 client.15822 trim_cache size 0 max 16384 2024-05-12T22:11:52.684 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-05-12T22:11:52.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.675+0000 7f6f0a600640 1 -- 172.21.15.28:0/2137569952 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f6ee80a2d00 msgr2=0x7f6ee80a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:11:52.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.675+0000 7f6f0a600640 1 --2- 172.21.15.28:0/2137569952 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f6ee80a2d00 0x7f6ee80a51c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x5604daf0de30 tx=0x7f6eec009770 comp rx=0 tx=0).stop 2024-05-12T22:11:52.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.675+0000 7f6f0a600640 1 -- 172.21.15.28:0/2137569952 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6f041126b0 msgr2=0x7f6f041a2ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:11:52.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.675+0000 7f6f0a600640 1 --2- 172.21.15.28:0/2137569952 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6f041126b0 0x7f6f041a2ff0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f6efc02ac00 tx=0x7f6efc0305d0 comp rx=0 tx=0).stop 2024-05-12T22:11:52.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.675+0000 7f6f0a600640 1 -- 172.21.15.28:0/2137569952 shutdown_connections 2024-05-12T22:11:52.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.675+0000 7f6f0a600640 1 --2- 172.21.15.28:0/2137569952 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f6ee80a2d00 0x7f6ee80a51c0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:52.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.675+0000 7f6f0a600640 1 --2- 172.21.15.28:0/2137569952 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6f041126b0 0x7f6f041a2ff0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:52.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.675+0000 7f6f0a600640 1 --2- 172.21.15.28:0/2137569952 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6f04076680 0x7f6f0419ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:52.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.675+0000 7f6f0a600640 1 --2- 172.21.15.28:0/2137569952 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f04075d60 0x7f6f0419e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:52.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.676+0000 7f6f0a600640 1 -- 172.21.15.28:0/2137569952 >> 172.21.15.28:0/2137569952 conn(0x7f6f040ff4a0 msgr2=0x7f6f04101890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:11:52.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.676+0000 7f6f0a600640 1 -- 172.21.15.28:0/2137569952 shutdown_connections 2024-05-12T22:11:52.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.676+0000 7f6f0a600640 1 -- 172.21.15.28:0/2137569952 wait complete. 2024-05-12T22:11:52.694 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-05-12T22:11:52.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.891+0000 7f7bfe600640 1 -- 172.21.15.28:0/245747414 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7bf810df70 msgr2=0x7f7bf8110380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:11:52.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.891+0000 7f7bfe600640 1 --2- 172.21.15.28:0/245747414 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7bf810df70 0x7f7bf8110380 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f7bf0009ca0 tx=0x7f7bf002a950 comp rx=0 tx=0).stop 2024-05-12T22:11:52.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.892+0000 7f7bfe600640 1 -- 172.21.15.28:0/245747414 shutdown_connections 2024-05-12T22:11:52.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.892+0000 7f7bfe600640 1 --2- 172.21.15.28:0/245747414 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7bf810df70 0x7f7bf8110380 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:52.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.892+0000 7f7bfe600640 1 --2- 172.21.15.28:0/245747414 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7bf8074520 0x7f7bf810da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:52.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.892+0000 7f7bfe600640 1 --2- 172.21.15.28:0/245747414 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7bf8073c00 0x7f7bf8073fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:52.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.892+0000 7f7bfe600640 1 -- 172.21.15.28:0/245747414 >> 172.21.15.28:0/245747414 conn(0x7f7bf80fd200 msgr2=0x7f7bf80ff620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:11:52.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.892+0000 7f7bfe600640 1 -- 172.21.15.28:0/245747414 shutdown_connections 2024-05-12T22:11:52.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.892+0000 7f7bfe600640 1 -- 172.21.15.28:0/245747414 wait complete. 2024-05-12T22:11:52.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.893+0000 7f7bfe600640 1 Processor -- start 2024-05-12T22:11:52.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.893+0000 7f7bfe600640 1 -- start start 2024-05-12T22:11:52.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.893+0000 7f7bfe600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7bf8073c00 0x7f7bf819a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:52.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.893+0000 7f7bfe600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7bf8074520 0x7f7bf819a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:52.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.893+0000 7f7bfe600640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7bf810df70 0x7f7bf819eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:52.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.893+0000 7f7bfe600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7bf8113d20 con 0x7f7bf8073c00 2024-05-12T22:11:52.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.893+0000 7f7bfe600640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f7bf8113ba0 con 0x7f7bf8074520 2024-05-12T22:11:52.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.893+0000 7f7bfe600640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f7bf8113ea0 con 0x7f7bf810df70 2024-05-12T22:11:52.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.894+0000 7f7bfd200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7bf8073c00 0x7f7bf819a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:52.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.894+0000 7f7befe00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7bf8074520 0x7f7bf819a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:52.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.894+0000 7f7befe00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7bf8074520 0x7f7bf819a740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:36938/0 (socket says 172.21.15.28:36938) 2024-05-12T22:11:52.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.894+0000 7f7befe00640 1 -- 172.21.15.28:0/335092517 learned_addr learned my addr 172.21.15.28:0/335092517 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:11:52.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.894+0000 7f7bfdc00640 1 --2- 172.21.15.28:0/335092517 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7bf810df70 0x7f7bf819eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:52.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.894+0000 7f7bfd200640 1 -- 172.21.15.28:0/335092517 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7bf810df70 msgr2=0x7f7bf819eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:11:52.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.894+0000 7f7bfd200640 1 --2- 172.21.15.28:0/335092517 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7bf810df70 0x7f7bf819eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:52.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.894+0000 7f7bfd200640 1 -- 172.21.15.28:0/335092517 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7bf8074520 msgr2=0x7f7bf819a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:11:52.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.894+0000 7f7bfd200640 1 --2- 172.21.15.28:0/335092517 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7bf8074520 0x7f7bf819a740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:52.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.894+0000 7f7bfd200640 1 -- 172.21.15.28:0/335092517 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7bf819f1f0 con 0x7f7bf8073c00 2024-05-12T22:11:52.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.894+0000 7f7befe00640 1 --2- 172.21.15.28:0/335092517 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7bf8074520 0x7f7bf819a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:11:52.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.894+0000 7f7bfd200640 1 --2- 172.21.15.28:0/335092517 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7bf8073c00 0x7f7bf819a200 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f7be800e9f0 tx=0x7f7be800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:11:52.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.895+0000 7f7bf6000640 1 -- 172.21.15.28:0/335092517 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7be800ccc0 con 0x7f7bf8073c00 2024-05-12T22:11:52.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.895+0000 7f7bf6000640 1 -- 172.21.15.28:0/335092517 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7be8010910 con 0x7f7bf8073c00 2024-05-12T22:11:52.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.895+0000 7f7bf6000640 1 -- 172.21.15.28:0/335092517 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7be8018960 con 0x7f7bf8073c00 2024-05-12T22:11:52.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.895+0000 7f7bfe600640 1 -- 172.21.15.28:0/335092517 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7bf819f4e0 con 0x7f7bf8073c00 2024-05-12T22:11:52.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.895+0000 7f7bfe600640 1 -- 172.21.15.28:0/335092517 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7bf81a6d70 con 0x7f7bf8073c00 2024-05-12T22:11:52.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.898+0000 7f7c02991740 1 Processor -- start 2024-05-12T22:11:52.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.898+0000 7f7c02991740 1 -- start start 2024-05-12T22:11:52.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.898+0000 7f7bf6000640 1 -- 172.21.15.28:0/335092517 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7f7be8010ac0 con 0x7f7bf8073c00 2024-05-12T22:11:52.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.899+0000 7f7c02991740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bbdbc2fb60 0x55bbdbc2ff40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:52.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.899+0000 7f7c02991740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bbdbc30480 0x55bbdbc3a610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:52.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.899+0000 7f7c02991740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55bbdbc3ab50 0x55bbdbc3cf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:52.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.899+0000 7f7c02991740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55bbdbb94a30 con 0x55bbdbc30480 2024-05-12T22:11:52.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.899+0000 7f7c02991740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x55bbdbb3c700 con 0x55bbdbc3ab50 2024-05-12T22:11:52.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.899+0000 7f7c02991740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55bbdbb909f0 con 0x55bbdbc2fb60 2024-05-12T22:11:52.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.899+0000 7f7bfdc00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bbdbc2fb60 0x55bbdbc2ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:52.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.899+0000 7f7befe00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55bbdbc3ab50 0x55bbdbc3cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:52.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.899+0000 7f7bfdc00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bbdbc2fb60 0x55bbdbc2ff40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:59250/0 (socket says 172.21.15.28:59250) 2024-05-12T22:11:52.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.899+0000 7f7bfdc00640 1 -- 172.21.15.28:0/1693183670 learned_addr learned my addr 172.21.15.28:0/1693183670 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:11:52.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.899+0000 7f7bfd200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bbdbc30480 0x55bbdbc3a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:52.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.899+0000 7f7bfdc00640 1 -- 172.21.15.28:0/1693183670 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55bbdbc3ab50 msgr2=0x55bbdbc3cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:11:52.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.899+0000 7f7bfdc00640 1 --2- 172.21.15.28:0/1693183670 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55bbdbc3ab50 0x55bbdbc3cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:52.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.899+0000 7f7bfdc00640 1 -- 172.21.15.28:0/1693183670 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bbdbc30480 msgr2=0x55bbdbc3a610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:11:52.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.899+0000 7f7bfdc00640 1 --2- 172.21.15.28:0/1693183670 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bbdbc30480 0x55bbdbc3a610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:52.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.899+0000 7f7bfdc00640 1 -- 172.21.15.28:0/1693183670 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bbdb8a83f0 con 0x55bbdbc2fb60 2024-05-12T22:11:52.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.899+0000 7f7bfd200640 1 --2- 172.21.15.28:0/1693183670 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bbdbc30480 0x55bbdbc3a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:11:52.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.900+0000 7f7bfdc00640 1 --2- 172.21.15.28:0/1693183670 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bbdbc2fb60 0x55bbdbc2ff40 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f7bf8074b50 tx=0x7f7bf0005f80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bd5d82e5970657eb server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:11:52.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.900+0000 7f7bf6000640 1 --2- 172.21.15.28:0/335092517 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f7bdc0a2d00 0x7f7bdc0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:52.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.900+0000 7f7bee000640 1 -- 172.21.15.28:0/1693183670 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bf0048070 con 0x55bbdbc2fb60 2024-05-12T22:11:52.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.900+0000 7f7bee000640 1 -- 172.21.15.28:0/1693183670 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7bf0003e50 con 0x55bbdbc2fb60 2024-05-12T22:11:52.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.900+0000 7f7bee000640 1 -- 172.21.15.28:0/1693183670 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bf0034040 con 0x55bbdbc2fb60 2024-05-12T22:11:52.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.900+0000 7f7bf6000640 1 -- 172.21.15.28:0/335092517 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f7be80cf7f0 con 0x7f7bf8073c00 2024-05-12T22:11:52.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.901+0000 7f7bfd200640 1 --2- 172.21.15.28:0/335092517 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f7bdc0a2d00 0x7f7bdc0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:52.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.901+0000 7f7c02991740 1 -- 172.21.15.28:0/1693183670 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bbdbc2fb60 msgr2=0x55bbdbc2ff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:11:52.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.901+0000 7f7c02991740 1 --2- 172.21.15.28:0/1693183670 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bbdbc2fb60 0x55bbdbc2ff40 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f7bf8074b50 tx=0x7f7bf0005f80 comp rx=0 tx=0).stop 2024-05-12T22:11:52.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.901+0000 7f7c02991740 1 -- 172.21.15.28:0/1693183670 shutdown_connections 2024-05-12T22:11:52.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.901+0000 7f7c02991740 1 --2- 172.21.15.28:0/1693183670 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55bbdbc3ab50 0x55bbdbc3cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:52.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.901+0000 7f7c02991740 1 --2- 172.21.15.28:0/1693183670 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bbdbc30480 0x55bbdbc3a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:52.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.901+0000 7f7c02991740 1 --2- 172.21.15.28:0/1693183670 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bbdbc2fb60 0x55bbdbc2ff40 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:52.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.901+0000 7f7c02991740 1 -- 172.21.15.28:0/1693183670 >> 172.21.15.28:0/1693183670 conn(0x55bbdbc2be80 msgr2=0x55bbdbc2c260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:11:52.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.901+0000 7f7bfd200640 1 --2- 172.21.15.28:0/335092517 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f7bdc0a2d00 0x7f7bdc0a51c0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x55bbdbc31120 tx=0x7f7be8009700 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:11:52.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.901+0000 7f7c02991740 1 -- 172.21.15.28:0/1693183670 shutdown_connections 2024-05-12T22:11:52.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.901+0000 7f7c02991740 1 -- 172.21.15.28:0/1693183670 wait complete. 2024-05-12T22:11:52.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.902+0000 7f7c02991740 1 -- start start 2024-05-12T22:11:52.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.902+0000 7f7c02991740 1 Processor -- start 2024-05-12T22:11:52.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.902+0000 7f7c02991740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55bbdbc2fb60 0x55bbdbc37ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:52.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.902+0000 7f7c02991740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bbdbc30480 0x55bbdbc37ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:52.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.902+0000 7f7c02991740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bbdbc3ab50 0x55bbdbc38530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:52.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.902+0000 7f7c02991740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55bbdbc3d8e0 con 0x55bbdbc30480 2024-05-12T22:11:52.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.902+0000 7f7c02991740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x55bbdbb4fa70 con 0x55bbdbc2fb60 2024-05-12T22:11:52.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.902+0000 7f7c02991740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55bbdbc3da60 con 0x55bbdbc3ab50 2024-05-12T22:11:52.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.903+0000 7f7c02991740 10 client.0 _ll_register_callbacks cb 0x55bbdbb32ba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:11:52.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.903+0000 7f7befe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bbdbc30480 0x55bbdbc37ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:52.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.903+0000 7f7befe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bbdbc30480 0x55bbdbc37ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48692/0 (socket says 172.21.15.28:48692) 2024-05-12T22:11:52.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.903+0000 7f7befe00640 1 -- 172.21.15.28:0/1991241995 learned_addr learned my addr 172.21.15.28:0/1991241995 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:11:52.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.903+0000 7f7bfdc00640 1 --2- 172.21.15.28:0/1991241995 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55bbdbc2fb60 0x55bbdbc37ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:52.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.903+0000 7f7bfdc00640 1 --2- 172.21.15.28:0/1991241995 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bbdbc3ab50 0x55bbdbc38530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:52.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.903+0000 7f7befe00640 1 -- 172.21.15.28:0/1991241995 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bbdbc3ab50 msgr2=0x55bbdbc38530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:11:52.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.903+0000 7f7befe00640 1 --2- 172.21.15.28:0/1991241995 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bbdbc3ab50 0x55bbdbc38530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:52.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.903+0000 7f7befe00640 1 -- 172.21.15.28:0/1991241995 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55bbdbc2fb60 msgr2=0x55bbdbc37ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:11:52.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.903+0000 7f7befe00640 1 --2- 172.21.15.28:0/1991241995 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55bbdbc2fb60 0x55bbdbc37ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:52.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.903+0000 7f7befe00640 1 -- 172.21.15.28:0/1991241995 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bbdb9727d0 con 0x55bbdbc30480 2024-05-12T22:11:52.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.903+0000 7f7bfdc00640 1 --2- 172.21.15.28:0/1991241995 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55bbdbc2fb60 0x55bbdbc37ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:11:52.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.904+0000 7f7befe00640 1 --2- 172.21.15.28:0/1991241995 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bbdbc30480 0x55bbdbc37ff0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f7be0012d20 tx=0x7f7be000f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89f3996e07effcd8 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:11:52.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.904+0000 7f7be6a00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-12T22:11:52.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.904+0000 7f7be6a00640 1 -- 172.21.15.28:0/1991241995 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7be0005020 con 0x55bbdbc30480 2024-05-12T22:11:52.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.904+0000 7f7be6a00640 1 -- 172.21.15.28:0/1991241995 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f7be00051c0 con 0x55bbdbc30480 2024-05-12T22:11:52.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.904+0000 7f7c02991740 1 -- 172.21.15.28:0/1991241995 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7bf0003e50 con 0x55bbdbc30480 2024-05-12T22:11:52.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.904+0000 7f7be6a00640 1 -- 172.21.15.28:0/1991241995 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f7be00053c0 con 0x55bbdbc30480 2024-05-12T22:11:52.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.904+0000 7f7c02991740 10 client.15834 fetch_fsmap learned FSMap version 25 2024-05-12T22:11:52.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.904+0000 7f7c02991740 1 -- 172.21.15.28:0/1991241995 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55bbdbc49bd0 con 0x55bbdbc30480 2024-05-12T22:11:52.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.904+0000 7f7be6a00640 1 -- 172.21.15.28:0/1991241995 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7bf0003e50 con 0x55bbdbc30480 2024-05-12T22:11:52.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.905+0000 7f7be6a00640 1 -- 172.21.15.28:0/1991241995 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7f7be001c470 con 0x55bbdbc30480 2024-05-12T22:11:52.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.905+0000 7f7c02991740 10 client.15834 fetch_fsmap finished waiting for FSMap version 25 2024-05-12T22:11:52.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.905+0000 7f7c02991740 10 client.15834 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-05-12T22:11:52.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.905+0000 7f7c02991740 20 client.15834 populate_metadata read hostname 'smithi028' 2024-05-12T22:11:52.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.905+0000 7f7c02991740 1 --2- 172.21.15.28:0/1991241995 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x55bbdbc4afe0 0x55bbdbc4d410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:52.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.905+0000 7f7c02991740 4 client.15834 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-12T22:11:52.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.905+0000 7f7c02991740 1 -- 172.21.15.28:0/1991241995 --> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bbdbba2160 con 0x55bbdbc4afe0 2024-05-12T22:11:52.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.905+0000 7f7bfdc00640 1 --2- 172.21.15.28:0/1991241995 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x55bbdbc4afe0 0x55bbdbc4d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:52.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.906+0000 7f7bfdc00640 1 --2- 172.21.15.28:0/1991241995 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x55bbdbc4afe0 0x55bbdbc4d410 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a57c237bdc0d2b22 server_cookie=6be531cef8bf09c1 in_seq=0 out_seq=0 2024-05-12T22:11:52.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.906+0000 7f7be6a00640 10 client.15834 ms_handle_connect on v2:172.21.15.28:6840/761183870 2024-05-12T22:11:52.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.907+0000 7f7be6a00640 1 -- 172.21.15.28:0/1991241995 <== mds.1 v2:172.21.15.28:6840/761183870 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55bbdbba2160 con 0x55bbdbc4afe0 2024-05-12T22:11:52.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.907+0000 7f7be6a00640 10 client.15834 handle_command_reply: tid=0 2024-05-12T22:11:52.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.907+0000 7f7c02991740 1 client.15834 shutdown 2024-05-12T22:11:52.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.907+0000 7f7c02991740 1 -- 172.21.15.28:0/1991241995 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bbdbc30480 msgr2=0x55bbdbc37ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:11:52.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.907+0000 7f7c02991740 1 --2- 172.21.15.28:0/1991241995 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bbdbc30480 0x55bbdbc37ff0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f7be0012d20 tx=0x7f7be000f790 comp rx=0 tx=0).stop 2024-05-12T22:11:52.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.907+0000 7f7c02991740 1 -- 172.21.15.28:0/1991241995 shutdown_connections 2024-05-12T22:11:52.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.907+0000 7f7c02991740 1 --2- 172.21.15.28:0/1991241995 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x55bbdbc4afe0 0x55bbdbc4d410 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:52.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.907+0000 7f7c02991740 1 --2- 172.21.15.28:0/1991241995 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bbdbc3ab50 0x55bbdbc38530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:52.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.907+0000 7f7c02991740 1 --2- 172.21.15.28:0/1991241995 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bbdbc30480 0x55bbdbc37ff0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:52.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.907+0000 7f7c02991740 1 --2- 172.21.15.28:0/1991241995 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55bbdbc2fb60 0x55bbdbc37ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:52.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.907+0000 7f7c02991740 1 -- 172.21.15.28:0/1991241995 >> 172.21.15.28:0/1991241995 conn(0x55bbdbc2be80 msgr2=0x55bbdbc31b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:11:52.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.907+0000 7f7be6a00640 0 client.15834 ms_handle_reset on v2:172.21.15.28:6840/761183870 2024-05-12T22:11:52.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.907+0000 7f7c02991740 1 -- 172.21.15.28:0/1991241995 shutdown_connections 2024-05-12T22:11:52.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.907+0000 7f7c02991740 1 -- 172.21.15.28:0/1991241995 wait complete. 2024-05-12T22:11:52.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.907+0000 7f7c02991740 20 client.15834 trim_cache size 0 max 16384 2024-05-12T22:11:52.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.922+0000 7f7c02991740 1 Processor -- start 2024-05-12T22:11:52.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.922+0000 7f7c02991740 1 -- start start 2024-05-12T22:11:52.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.922+0000 7f7c02991740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bbdbc2fb60 0x55bbdbc467d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:52.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.923+0000 7f7c02991740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bbdbc30480 0x55bbdbc322c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:52.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.923+0000 7f7c02991740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55bbdbc3ab50 0x55bbdbc52ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:52.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.923+0000 7f7c02991740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7bf81b1dd0 con 0x55bbdbc30480 2024-05-12T22:11:52.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.923+0000 7f7c02991740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f7bf81af6a0 con 0x55bbdbc3ab50 2024-05-12T22:11:52.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.923+0000 7f7c02991740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f7bf81ad4f0 con 0x55bbdbc2fb60 2024-05-12T22:11:52.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.923+0000 7f7befe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bbdbc30480 0x55bbdbc322c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:52.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.923+0000 7f7befe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bbdbc30480 0x55bbdbc322c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48708/0 (socket says 172.21.15.28:48708) 2024-05-12T22:11:52.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.923+0000 7f7befe00640 1 -- 172.21.15.28:0/1335265028 learned_addr learned my addr 172.21.15.28:0/1335265028 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:11:52.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.923+0000 7f7bfdc00640 1 --2- 172.21.15.28:0/1335265028 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bbdbc2fb60 0x55bbdbc467d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:52.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.923+0000 7f7bfdc00640 1 --2- 172.21.15.28:0/1335265028 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55bbdbc3ab50 0x55bbdbc52ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:52.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.923+0000 7f7befe00640 1 -- 172.21.15.28:0/1335265028 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bbdbc2fb60 msgr2=0x55bbdbc467d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:11:52.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.923+0000 7f7befe00640 1 --2- 172.21.15.28:0/1335265028 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bbdbc2fb60 0x55bbdbc467d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:52.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.923+0000 7f7befe00640 1 -- 172.21.15.28:0/1335265028 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55bbdbc3ab50 msgr2=0x55bbdbc52ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:11:52.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.923+0000 7f7befe00640 1 --2- 172.21.15.28:0/1335265028 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55bbdbc3ab50 0x55bbdbc52ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:52.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.923+0000 7f7befe00640 1 -- 172.21.15.28:0/1335265028 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bbdbc275f0 con 0x55bbdbc30480 2024-05-12T22:11:52.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.924+0000 7f7befe00640 1 --2- 172.21.15.28:0/1335265028 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bbdbc30480 0x55bbdbc322c0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f7be00098e0 tx=0x7f7be00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c37a0cca75d6ab09 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:11:52.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.924+0000 7f7bed600640 1 -- 172.21.15.28:0/1335265028 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7be00050c0 con 0x55bbdbc30480 2024-05-12T22:11:52.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.924+0000 7f7bed600640 1 -- 172.21.15.28:0/1335265028 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7be0005bf0 con 0x55bbdbc30480 2024-05-12T22:11:52.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.924+0000 7f7bed600640 1 -- 172.21.15.28:0/1335265028 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7be0003ad0 con 0x55bbdbc30480 2024-05-12T22:11:52.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.925+0000 7f7c02991740 1 -- 172.21.15.28:0/1335265028 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bbdbc30480 msgr2=0x55bbdbc322c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:11:52.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.925+0000 7f7c02991740 1 --2- 172.21.15.28:0/1335265028 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bbdbc30480 0x55bbdbc322c0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f7be00098e0 tx=0x7f7be00057f0 comp rx=0 tx=0).stop 2024-05-12T22:11:52.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.925+0000 7f7c02991740 1 -- 172.21.15.28:0/1335265028 shutdown_connections 2024-05-12T22:11:52.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.925+0000 7f7c02991740 1 --2- 172.21.15.28:0/1335265028 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55bbdbc3ab50 0x55bbdbc52ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:52.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.925+0000 7f7c02991740 1 --2- 172.21.15.28:0/1335265028 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bbdbc30480 0x55bbdbc322c0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:52.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.925+0000 7f7c02991740 1 --2- 172.21.15.28:0/1335265028 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bbdbc2fb60 0x55bbdbc467d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:52.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.925+0000 7f7c02991740 1 -- 172.21.15.28:0/1335265028 >> 172.21.15.28:0/1335265028 conn(0x55bbdbc2be80 msgr2=0x55bbdbc38300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:11:52.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.925+0000 7f7c02991740 1 -- 172.21.15.28:0/1335265028 shutdown_connections 2024-05-12T22:11:52.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.925+0000 7f7c02991740 1 -- 172.21.15.28:0/1335265028 wait complete. 2024-05-12T22:11:52.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.925+0000 7f7c02991740 1 -- start start 2024-05-12T22:11:52.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.926+0000 7f7c02991740 1 Processor -- start 2024-05-12T22:11:52.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.926+0000 7f7c02991740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55bbdbc2fb60 0x55bbdbc347a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:52.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.926+0000 7f7c02991740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bbdbc30480 0x55bbdbc34ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:52.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.926+0000 7f7c02991740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bbdbc3ab50 0x55bbdbc35220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:52.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.926+0000 7f7c02991740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7bf81adb60 con 0x55bbdbc3ab50 2024-05-12T22:11:52.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.926+0000 7f7c02991740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f7bf81aefd0 con 0x55bbdbc2fb60 2024-05-12T22:11:52.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.926+0000 7f7c02991740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f7bf81ae230 con 0x55bbdbc30480 2024-05-12T22:11:52.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.926+0000 7f7c02991740 10 client.0 _ll_register_callbacks cb 0x7f7be0012930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:11:52.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.926+0000 7f7bfdc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bbdbc3ab50 0x55bbdbc35220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:52.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.927+0000 7f7bfdc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bbdbc3ab50 0x55bbdbc35220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48724/0 (socket says 172.21.15.28:48724) 2024-05-12T22:11:52.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.927+0000 7f7bfdc00640 1 -- 172.21.15.28:0/2023133061 learned_addr learned my addr 172.21.15.28:0/2023133061 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:11:52.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.927+0000 7f7befe00640 1 --2- 172.21.15.28:0/2023133061 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bbdbc30480 0x55bbdbc34ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:52.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.927+0000 7f7bfdc00640 1 --2- 172.21.15.28:0/2023133061 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55bbdbc2fb60 0x55bbdbc347a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:52.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.927+0000 7f7befe00640 1 -- 172.21.15.28:0/2023133061 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55bbdbc2fb60 msgr2=0x55bbdbc347a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:11:52.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.927+0000 7f7befe00640 1 --2- 172.21.15.28:0/2023133061 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55bbdbc2fb60 0x55bbdbc347a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:52.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.927+0000 7f7befe00640 1 -- 172.21.15.28:0/2023133061 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bbdbc3ab50 msgr2=0x55bbdbc35220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:11:52.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.927+0000 7f7befe00640 1 --2- 172.21.15.28:0/2023133061 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bbdbc3ab50 0x55bbdbc35220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:52.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.927+0000 7f7befe00640 1 -- 172.21.15.28:0/2023133061 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bbdbcb0600 con 0x55bbdbc30480 2024-05-12T22:11:52.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.927+0000 7f7bfdc00640 1 --2- 172.21.15.28:0/2023133061 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bbdbc3ab50 0x55bbdbc35220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:11:52.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.927+0000 7f7befe00640 1 --2- 172.21.15.28:0/2023133061 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bbdbc30480 0x55bbdbc34ce0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x55bbdbc278b0 tx=0x7f7be001c470 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9b918bdbf2a66cb6 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:11:52.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.927+0000 7f7be6a00640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-12T22:11:52.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.928+0000 7f7be6a00640 1 -- 172.21.15.28:0/2023133061 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7be0032070 con 0x55bbdbc30480 2024-05-12T22:11:52.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.928+0000 7f7c02991740 1 -- 172.21.15.28:0/2023133061 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55bbdbb9c4b0 con 0x55bbdbc30480 2024-05-12T22:11:52.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.928+0000 7f7be6a00640 1 -- 172.21.15.28:0/2023133061 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f7be001ca30 con 0x55bbdbc30480 2024-05-12T22:11:52.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.928+0000 7f7be6a00640 1 -- 172.21.15.28:0/2023133061 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f7be003e050 con 0x55bbdbc30480 2024-05-12T22:11:52.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.928+0000 7f7c02991740 10 client.25921 fetch_fsmap learned FSMap version 25 2024-05-12T22:11:52.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.928+0000 7f7c02991740 1 -- 172.21.15.28:0/2023133061 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55bbdbcb18e0 con 0x55bbdbc30480 2024-05-12T22:11:52.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.928+0000 7f7be6a00640 1 -- 172.21.15.28:0/2023133061 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55bbdbb9c4b0 con 0x55bbdbc30480 2024-05-12T22:11:52.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.928+0000 7f7be6a00640 1 -- 172.21.15.28:0/2023133061 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7f7be0003ad0 con 0x55bbdbc30480 2024-05-12T22:11:52.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.929+0000 7f7c02991740 10 client.25921 fetch_fsmap finished waiting for FSMap version 25 2024-05-12T22:11:52.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.929+0000 7f7c02991740 10 client.25921 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-05-12T22:11:52.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.929+0000 7f7c02991740 20 client.25921 populate_metadata read hostname 'smithi028' 2024-05-12T22:11:52.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.929+0000 7f7c02991740 1 --2- 172.21.15.28:0/2023133061 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x55bbdbcb1e20 0x55bbdbcb4240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:52.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.929+0000 7f7c02991740 4 client.25921 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-12T22:11:52.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.929+0000 7f7c02991740 1 -- 172.21.15.28:0/2023133061 --> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55bbdbb196e0 con 0x55bbdbcb1e20 2024-05-12T22:11:52.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.929+0000 7f7bfdc00640 1 --2- 172.21.15.28:0/2023133061 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x55bbdbcb1e20 0x55bbdbcb4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:52.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.929+0000 7f7bfdc00640 1 --2- 172.21.15.28:0/2023133061 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x55bbdbcb1e20 0x55bbdbcb4240 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=191fc35084210231 server_cookie=31093551e2f3154d in_seq=0 out_seq=0 2024-05-12T22:11:52.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.929+0000 7f7be6a00640 10 client.25921 ms_handle_connect on v2:172.21.15.28:6840/761183870 2024-05-12T22:11:52.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.930+0000 7f7be6a00640 1 -- 172.21.15.28:0/2023133061 <== mds.1 v2:172.21.15.28:6840/761183870 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55bbdbb196e0 con 0x55bbdbcb1e20 2024-05-12T22:11:52.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.930+0000 7f7be6a00640 10 client.25921 handle_command_reply: tid=0 2024-05-12T22:11:52.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.930+0000 7f7c02991740 1 client.25921 shutdown 2024-05-12T22:11:52.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.930+0000 7f7c02991740 1 -- 172.21.15.28:0/2023133061 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bbdbc30480 msgr2=0x55bbdbc34ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:11:52.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.930+0000 7f7c02991740 1 --2- 172.21.15.28:0/2023133061 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bbdbc30480 0x55bbdbc34ce0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x55bbdbc278b0 tx=0x7f7be001c470 comp rx=0 tx=0).stop 2024-05-12T22:11:52.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.930+0000 7f7c02991740 1 -- 172.21.15.28:0/2023133061 shutdown_connections 2024-05-12T22:11:52.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.930+0000 7f7c02991740 1 --2- 172.21.15.28:0/2023133061 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x55bbdbcb1e20 0x55bbdbcb4240 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:52.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.930+0000 7f7c02991740 1 --2- 172.21.15.28:0/2023133061 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bbdbc3ab50 0x55bbdbc35220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:52.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.930+0000 7f7c02991740 1 --2- 172.21.15.28:0/2023133061 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bbdbc30480 0x55bbdbc34ce0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:52.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.930+0000 7f7c02991740 1 --2- 172.21.15.28:0/2023133061 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55bbdbc2fb60 0x55bbdbc347a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:52.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.930+0000 7f7be6a00640 0 client.25921 ms_handle_reset on v2:172.21.15.28:6840/761183870 2024-05-12T22:11:52.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.930+0000 7f7c02991740 1 -- 172.21.15.28:0/2023133061 >> 172.21.15.28:0/2023133061 conn(0x55bbdbc2be80 msgr2=0x55bbdbc392e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:11:52.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.930+0000 7f7c02991740 1 -- 172.21.15.28:0/2023133061 shutdown_connections 2024-05-12T22:11:52.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.930+0000 7f7c02991740 1 -- 172.21.15.28:0/2023133061 wait complete. 2024-05-12T22:11:52.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.930+0000 7f7c02991740 20 client.25921 trim_cache size 0 max 16384 2024-05-12T22:11:52.939 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-05-12T22:11:52.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.931+0000 7f7bfe600640 1 -- 172.21.15.28:0/335092517 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f7bdc0a2d00 msgr2=0x7f7bdc0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:11:52.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.931+0000 7f7bfe600640 1 --2- 172.21.15.28:0/335092517 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f7bdc0a2d00 0x7f7bdc0a51c0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x55bbdbc31120 tx=0x7f7be8009700 comp rx=0 tx=0).stop 2024-05-12T22:11:52.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.931+0000 7f7bfe600640 1 -- 172.21.15.28:0/335092517 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7bf8073c00 msgr2=0x7f7bf819a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:11:52.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.931+0000 7f7bfe600640 1 --2- 172.21.15.28:0/335092517 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7bf8073c00 0x7f7bf819a200 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f7be800e9f0 tx=0x7f7be800eeb0 comp rx=0 tx=0).stop 2024-05-12T22:11:52.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.931+0000 7f7bfe600640 1 -- 172.21.15.28:0/335092517 shutdown_connections 2024-05-12T22:11:52.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.931+0000 7f7bfe600640 1 --2- 172.21.15.28:0/335092517 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f7bdc0a2d00 0x7f7bdc0a51c0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:52.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.931+0000 7f7bfe600640 1 --2- 172.21.15.28:0/335092517 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7bf810df70 0x7f7bf819eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:52.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.931+0000 7f7bfe600640 1 --2- 172.21.15.28:0/335092517 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7bf8074520 0x7f7bf819a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:52.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.931+0000 7f7bfe600640 1 --2- 172.21.15.28:0/335092517 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7bf8073c00 0x7f7bf819a200 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:52.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.931+0000 7f7bfe600640 1 -- 172.21.15.28:0/335092517 >> 172.21.15.28:0/335092517 conn(0x7f7bf80fd200 msgr2=0x7f7bf80fded0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:11:52.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.931+0000 7f7bfe600640 1 -- 172.21.15.28:0/335092517 shutdown_connections 2024-05-12T22:11:52.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:52.931+0000 7f7bfe600640 1 -- 172.21.15.28:0/335092517 wait complete. 2024-05-12T22:11:52.952 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-05-12T22:11:53.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.165+0000 7fa6dcc00640 1 -- 172.21.15.28:0/1933617084 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa6d8100700 msgr2=0x7fa6d8100b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:11:53.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.165+0000 7fa6dcc00640 1 --2- 172.21.15.28:0/1933617084 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa6d8100700 0x7fa6d8100b80 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fa6c4009930 tx=0x7fa6c402a440 comp rx=0 tx=0).stop 2024-05-12T22:11:53.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.165+0000 7fa6dcc00640 1 -- 172.21.15.28:0/1933617084 shutdown_connections 2024-05-12T22:11:53.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.165+0000 7fa6dcc00640 1 --2- 172.21.15.28:0/1933617084 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa6d81010c0 0x7fa6d810d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:53.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.165+0000 7fa6dcc00640 1 --2- 172.21.15.28:0/1933617084 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa6d8100700 0x7fa6d8100b80 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:53.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.165+0000 7fa6dcc00640 1 --2- 172.21.15.28:0/1933617084 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6d8106710 0x7fa6d8106af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:53.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.165+0000 7fa6dcc00640 1 -- 172.21.15.28:0/1933617084 >> 172.21.15.28:0/1933617084 conn(0x7fa6d8077830 msgr2=0x7fa6d8077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:11:53.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.165+0000 7fa6dcc00640 1 -- 172.21.15.28:0/1933617084 shutdown_connections 2024-05-12T22:11:53.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.165+0000 7fa6dcc00640 1 -- 172.21.15.28:0/1933617084 wait complete. 2024-05-12T22:11:53.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.166+0000 7fa6dcc00640 1 Processor -- start 2024-05-12T22:11:53.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.166+0000 7fa6dcc00640 1 -- start start 2024-05-12T22:11:53.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.167+0000 7fa6dcc00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa6d8100700 0x7fa6d8198190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:53.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.167+0000 7fa6dcc00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa6d81010c0 0x7fa6d81986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:53.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.167+0000 7fa6dcc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6d8106710 0x7fa6d819ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:53.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.167+0000 7fa6dcc00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa6d8110b70 con 0x7fa6d8106710 2024-05-12T22:11:53.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.167+0000 7fa6dcc00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fa6d81109f0 con 0x7fa6d8100700 2024-05-12T22:11:53.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.167+0000 7fa6dcc00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa6d8110cf0 con 0x7fa6d81010c0 2024-05-12T22:11:53.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.167+0000 7fa6d5e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa6d8100700 0x7fa6d8198190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:53.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.167+0000 7fa6d6800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6d8106710 0x7fa6d819ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:53.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.167+0000 7fa6d5e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa6d8100700 0x7fa6d8198190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:37002/0 (socket says 172.21.15.28:37002) 2024-05-12T22:11:53.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.167+0000 7fa6d5e00640 1 -- 172.21.15.28:0/1054037948 learned_addr learned my addr 172.21.15.28:0/1054037948 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:11:53.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.167+0000 7fa6d5400640 1 --2- 172.21.15.28:0/1054037948 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa6d81010c0 0x7fa6d81986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:53.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.168+0000 7fa6d6800640 1 -- 172.21.15.28:0/1054037948 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa6d81010c0 msgr2=0x7fa6d81986d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:11:53.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.168+0000 7fa6d6800640 1 --2- 172.21.15.28:0/1054037948 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa6d81010c0 0x7fa6d81986d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:53.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.168+0000 7fa6d6800640 1 -- 172.21.15.28:0/1054037948 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa6d8100700 msgr2=0x7fa6d8198190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:11:53.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.168+0000 7fa6d6800640 1 --2- 172.21.15.28:0/1054037948 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa6d8100700 0x7fa6d8198190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:53.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.168+0000 7fa6d6800640 1 -- 172.21.15.28:0/1054037948 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6d819d180 con 0x7fa6d8106710 2024-05-12T22:11:53.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.168+0000 7fa6d5e00640 1 --2- 172.21.15.28:0/1054037948 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa6d8100700 0x7fa6d8198190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:11:53.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.168+0000 7fa6d6800640 1 --2- 172.21.15.28:0/1054037948 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6d8106710 0x7fa6d819ca50 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fa6d0009fd0 tx=0x7fa6d000ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:11:53.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.168+0000 7fa6cea00640 1 -- 172.21.15.28:0/1054037948 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6d000ce20 con 0x7fa6d8106710 2024-05-12T22:11:53.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.168+0000 7fa6cea00640 1 -- 172.21.15.28:0/1054037948 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa6d0005020 con 0x7fa6d8106710 2024-05-12T22:11:53.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.168+0000 7fa6cea00640 1 -- 172.21.15.28:0/1054037948 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6d0010720 con 0x7fa6d8106710 2024-05-12T22:11:53.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.168+0000 7fa6dcc00640 1 -- 172.21.15.28:0/1054037948 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6d819d470 con 0x7fa6d8106710 2024-05-12T22:11:53.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.168+0000 7fa6dcc00640 1 -- 172.21.15.28:0/1054037948 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6d81a4d00 con 0x7fa6d8106710 2024-05-12T22:11:53.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.173+0000 7fa6dfb77740 1 Processor -- start 2024-05-12T22:11:53.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.173+0000 7fa6cea00640 1 -- 172.21.15.28:0/1054037948 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7fa6d0005540 con 0x7fa6d8106710 2024-05-12T22:11:53.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.173+0000 7fa6dfb77740 1 -- start start 2024-05-12T22:11:53.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.174+0000 7fa6dfb77740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563b5c491650 0x563b5c491a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:53.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.174+0000 7fa6dfb77740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563b5c491f70 0x563b5c4a0840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:53.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.174+0000 7fa6dfb77740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563b5c4a0d80 0x563b5c4a3170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:53.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.174+0000 7fa6dfb77740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x563b5c3f6a30 con 0x563b5c491650 2024-05-12T22:11:53.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.174+0000 7fa6dfb77740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x563b5c39e700 con 0x563b5c4a0d80 2024-05-12T22:11:53.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.174+0000 7fa6dfb77740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x563b5c3f29f0 con 0x563b5c491f70 2024-05-12T22:11:53.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.174+0000 7fa6d5400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563b5c491650 0x563b5c491a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:53.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.174+0000 7fa6d5e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563b5c491f70 0x563b5c4a0840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:53.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.174+0000 7fa6d5400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563b5c491650 0x563b5c491a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48754/0 (socket says 172.21.15.28:48754) 2024-05-12T22:11:53.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.174+0000 7fa6d5e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563b5c491f70 0x563b5c4a0840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:59296/0 (socket says 172.21.15.28:59296) 2024-05-12T22:11:53.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.174+0000 7fa6d5400640 1 -- 172.21.15.28:0/4187856940 learned_addr learned my addr 172.21.15.28:0/4187856940 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:11:53.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.174+0000 7fa6d5400640 1 --2- 172.21.15.28:0/4187856940 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563b5c4a0d80 0x563b5c4a3170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:53.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.175+0000 7fa6d5e00640 1 -- 172.21.15.28:0/4187856940 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563b5c4a0d80 msgr2=0x563b5c4a3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:11:53.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.175+0000 7fa6d5e00640 1 --2- 172.21.15.28:0/4187856940 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563b5c4a0d80 0x563b5c4a3170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:53.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.175+0000 7fa6d5e00640 1 -- 172.21.15.28:0/4187856940 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563b5c491650 msgr2=0x563b5c491a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:11:53.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.175+0000 7fa6d5e00640 1 --2- 172.21.15.28:0/4187856940 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563b5c491650 0x563b5c491a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:53.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.175+0000 7fa6d5e00640 1 -- 172.21.15.28:0/4187856940 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563b5c10a3f0 con 0x563b5c491f70 2024-05-12T22:11:53.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.175+0000 7fa6d5400640 1 --2- 172.21.15.28:0/4187856940 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563b5c491650 0x563b5c491a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:11:53.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.175+0000 7fa6d5e00640 1 --2- 172.21.15.28:0/4187856940 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563b5c491f70 0x563b5c4a0840 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fa6c8011440 tx=0x7fa6c8011900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=233ae9d154b144b server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:11:53.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.175+0000 7fa6cea00640 1 --2- 172.21.15.28:0/1054037948 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fa6bc0a2d00 0x7fa6bc0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:53.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.175+0000 7fa6cea00640 1 -- 172.21.15.28:0/1054037948 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7fa6d00cfa90 con 0x7fa6d8106710 2024-05-12T22:11:53.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.175+0000 7fa6c3000640 1 -- 172.21.15.28:0/4187856940 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6c8004030 con 0x563b5c491f70 2024-05-12T22:11:53.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.175+0000 7fa6c3000640 1 -- 172.21.15.28:0/4187856940 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa6c80041d0 con 0x563b5c491f70 2024-05-12T22:11:53.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.175+0000 7fa6c3000640 1 -- 172.21.15.28:0/4187856940 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6c800f8f0 con 0x563b5c491f70 2024-05-12T22:11:53.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.175+0000 7fa6d5400640 1 --2- 172.21.15.28:0/1054037948 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fa6bc0a2d00 0x7fa6bc0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:53.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.176+0000 7fa6dfb77740 1 -- 172.21.15.28:0/4187856940 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563b5c491f70 msgr2=0x563b5c4a0840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:11:53.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.176+0000 7fa6dfb77740 1 --2- 172.21.15.28:0/4187856940 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563b5c491f70 0x563b5c4a0840 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fa6c8011440 tx=0x7fa6c8011900 comp rx=0 tx=0).stop 2024-05-12T22:11:53.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.176+0000 7fa6d5400640 1 --2- 172.21.15.28:0/1054037948 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fa6bc0a2d00 0x7fa6bc0a51c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fa6c4005c40 tx=0x7fa6c4005b50 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:11:53.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.176+0000 7fa6dfb77740 1 -- 172.21.15.28:0/4187856940 shutdown_connections 2024-05-12T22:11:53.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.176+0000 7fa6dfb77740 1 --2- 172.21.15.28:0/4187856940 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563b5c4a0d80 0x563b5c4a3170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:53.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.176+0000 7fa6dfb77740 1 --2- 172.21.15.28:0/4187856940 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563b5c491f70 0x563b5c4a0840 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:53.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.176+0000 7fa6dfb77740 1 --2- 172.21.15.28:0/4187856940 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563b5c491650 0x563b5c491a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:53.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.176+0000 7fa6dfb77740 1 -- 172.21.15.28:0/4187856940 >> 172.21.15.28:0/4187856940 conn(0x563b5c48dca0 msgr2=0x563b5c48e080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:11:53.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.177+0000 7fa6dfb77740 1 -- 172.21.15.28:0/4187856940 shutdown_connections 2024-05-12T22:11:53.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.177+0000 7fa6dfb77740 1 -- 172.21.15.28:0/4187856940 wait complete. 2024-05-12T22:11:53.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.177+0000 7fa6dfb77740 1 -- start start 2024-05-12T22:11:53.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.177+0000 7fa6dfb77740 1 Processor -- start 2024-05-12T22:11:53.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.178+0000 7fa6dfb77740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563b5c491650 0x563b5c49dd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:53.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.178+0000 7fa6dfb77740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563b5c491f70 0x563b5c49e260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:53.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.178+0000 7fa6dfb77740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563b5c4a0d80 0x563b5c49e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:53.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.178+0000 7fa6dfb77740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x563b5c4a3b40 con 0x563b5c491f70 2024-05-12T22:11:53.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.178+0000 7fa6dfb77740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x563b5c3b1a70 con 0x563b5c4a0d80 2024-05-12T22:11:53.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.178+0000 7fa6dfb77740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x563b5c4a3cc0 con 0x563b5c491650 2024-05-12T22:11:53.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.178+0000 7fa6dfb77740 10 client.0 _ll_register_callbacks cb 0x563b5c394ba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:11:53.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.178+0000 7fa6d5e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563b5c491650 0x563b5c49dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:53.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.178+0000 7fa6d6800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563b5c4a0d80 0x563b5c49e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:53.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.178+0000 7fa6d5e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563b5c491650 0x563b5c49dd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:59300/0 (socket says 172.21.15.28:59300) 2024-05-12T22:11:53.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.178+0000 7fa6d5e00640 1 -- 172.21.15.28:0/4264862748 learned_addr learned my addr 172.21.15.28:0/4264862748 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:11:53.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.178+0000 7fa6d5400640 1 --2- 172.21.15.28:0/4264862748 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563b5c491f70 0x563b5c49e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:53.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.179+0000 7fa6d6800640 1 -- 172.21.15.28:0/4264862748 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563b5c491650 msgr2=0x563b5c49dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:11:53.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.179+0000 7fa6d6800640 1 --2- 172.21.15.28:0/4264862748 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563b5c491650 0x563b5c49dd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:53.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.179+0000 7fa6d6800640 1 -- 172.21.15.28:0/4264862748 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563b5c491f70 msgr2=0x563b5c49e260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:11:53.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.179+0000 7fa6d6800640 1 --2- 172.21.15.28:0/4264862748 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563b5c491f70 0x563b5c49e260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:53.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.179+0000 7fa6d6800640 1 -- 172.21.15.28:0/4264862748 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563b5c1d47d0 con 0x563b5c4a0d80 2024-05-12T22:11:53.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.179+0000 7fa6d5e00640 1 --2- 172.21.15.28:0/4264862748 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563b5c491650 0x563b5c49dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:11:53.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.179+0000 7fa6d6800640 1 --2- 172.21.15.28:0/4264862748 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563b5c4a0d80 0x563b5c49e7a0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fa6d00cd1b0 tx=0x7fa6d009c040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=acf37f36187e2634 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:11:53.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.179+0000 7fa6b7400640 10 client.0 ms_handle_connect on v2:172.21.15.86:3300/0 2024-05-12T22:11:53.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.179+0000 7fa6b7400640 1 -- 172.21.15.28:0/4264862748 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6d00bc070 con 0x563b5c4a0d80 2024-05-12T22:11:53.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.179+0000 7fa6b7400640 1 -- 172.21.15.28:0/4264862748 <== mon.2 v2:172.21.15.86:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7fa6d00a4760 con 0x563b5c4a0d80 2024-05-12T22:11:53.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.179+0000 7fa6dfb77740 1 -- 172.21.15.28:0/4264862748 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa6c80041d0 con 0x563b5c4a0d80 2024-05-12T22:11:53.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.179+0000 7fa6b7400640 1 -- 172.21.15.28:0/4264862748 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fa6d00a4960 con 0x563b5c4a0d80 2024-05-12T22:11:53.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.179+0000 7fa6dfb77740 10 client.25979 fetch_fsmap learned FSMap version 25 2024-05-12T22:11:53.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.179+0000 7fa6dfb77740 1 -- 172.21.15.28:0/4264862748 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x563b5c4afed0 con 0x563b5c4a0d80 2024-05-12T22:11:53.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.180+0000 7fa6b7400640 1 -- 172.21.15.28:0/4264862748 <== mon.2 v2:172.21.15.86:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa6c80041d0 con 0x563b5c4a0d80 2024-05-12T22:11:53.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.180+0000 7fa6b7400640 1 -- 172.21.15.28:0/4264862748 <== mon.2 v2:172.21.15.86:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7fa6d0097cd0 con 0x563b5c4a0d80 2024-05-12T22:11:53.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.180+0000 7fa6dfb77740 10 client.25979 fetch_fsmap finished waiting for FSMap version 25 2024-05-12T22:11:53.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.180+0000 7fa6dfb77740 10 client.25979 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:11:53.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.180+0000 7fa6dfb77740 20 client.25979 populate_metadata read hostname 'smithi028' 2024-05-12T22:11:53.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.180+0000 7fa6dfb77740 1 --2- 172.21.15.28:0/4264862748 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x563b5c4b1390 0x563b5c4b37c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:53.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.180+0000 7fa6dfb77740 4 client.25979 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-12T22:11:53.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.180+0000 7fa6dfb77740 1 -- 172.21.15.28:0/4264862748 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563b5c404160 con 0x563b5c4b1390 2024-05-12T22:11:53.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.180+0000 7fa6d5e00640 1 --2- 172.21.15.28:0/4264862748 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x563b5c4b1390 0x563b5c4b37c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:53.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.181+0000 7fa6d5e00640 1 --2- 172.21.15.28:0/4264862748 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x563b5c4b1390 0x563b5c4b37c0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4d3ef66b8c4d9368 server_cookie=4fd78f246290668c in_seq=0 out_seq=0 2024-05-12T22:11:53.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.181+0000 7fa6b7400640 10 client.25979 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:11:53.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.182+0000 7fa6b7400640 1 -- 172.21.15.28:0/4264862748 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563b5c404160 con 0x563b5c4b1390 2024-05-12T22:11:53.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.182+0000 7fa6b7400640 10 client.25979 handle_command_reply: tid=0 2024-05-12T22:11:53.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.182+0000 7fa6dfb77740 1 client.25979 shutdown 2024-05-12T22:11:53.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.182+0000 7fa6dfb77740 1 -- 172.21.15.28:0/4264862748 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563b5c4a0d80 msgr2=0x563b5c49e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:11:53.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.182+0000 7fa6dfb77740 1 --2- 172.21.15.28:0/4264862748 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563b5c4a0d80 0x563b5c49e7a0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fa6d00cd1b0 tx=0x7fa6d009c040 comp rx=0 tx=0).stop 2024-05-12T22:11:53.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.182+0000 7fa6dfb77740 1 -- 172.21.15.28:0/4264862748 shutdown_connections 2024-05-12T22:11:53.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.182+0000 7fa6dfb77740 1 --2- 172.21.15.28:0/4264862748 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x563b5c4b1390 0x563b5c4b37c0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:53.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.182+0000 7fa6dfb77740 1 --2- 172.21.15.28:0/4264862748 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563b5c4a0d80 0x563b5c49e7a0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:53.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.182+0000 7fa6dfb77740 1 --2- 172.21.15.28:0/4264862748 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563b5c491f70 0x563b5c49e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:53.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.182+0000 7fa6b7400640 0 client.25979 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:11:53.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.182+0000 7fa6dfb77740 1 --2- 172.21.15.28:0/4264862748 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563b5c491650 0x563b5c49dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:53.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.182+0000 7fa6dfb77740 1 -- 172.21.15.28:0/4264862748 >> 172.21.15.28:0/4264862748 conn(0x563b5c48dca0 msgr2=0x563b5c493b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:11:53.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.182+0000 7fa6dfb77740 1 -- 172.21.15.28:0/4264862748 shutdown_connections 2024-05-12T22:11:53.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.182+0000 7fa6dfb77740 1 -- 172.21.15.28:0/4264862748 wait complete. 2024-05-12T22:11:53.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.182+0000 7fa6dfb77740 20 client.25979 trim_cache size 0 max 16384 2024-05-12T22:11:53.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.197+0000 7fa6dfb77740 1 Processor -- start 2024-05-12T22:11:53.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.197+0000 7fa6dfb77740 1 -- start start 2024-05-12T22:11:53.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.197+0000 7fa6dfb77740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563b5c491650 0x563b5c4aca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:53.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.197+0000 7fa6dfb77740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563b5c491f70 0x563b5c48cc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:53.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.197+0000 7fa6dfb77740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563b5c4a0d80 0x563b5c4b9090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:53.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.197+0000 7fa6dfb77740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa6d81adbd0 con 0x563b5c4a0d80 2024-05-12T22:11:53.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.197+0000 7fa6dfb77740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fa6d81ab4a0 con 0x563b5c491f70 2024-05-12T22:11:53.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.197+0000 7fa6dfb77740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa6d81a92f0 con 0x563b5c491650 2024-05-12T22:11:53.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.198+0000 7fa6d5e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563b5c491650 0x563b5c4aca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:53.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.198+0000 7fa6d5e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563b5c491650 0x563b5c4aca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:59310/0 (socket says 172.21.15.28:59310) 2024-05-12T22:11:53.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.198+0000 7fa6d5e00640 1 -- 172.21.15.28:0/1616778032 learned_addr learned my addr 172.21.15.28:0/1616778032 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:11:53.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.198+0000 7fa6d6800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563b5c4a0d80 0x563b5c4b9090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:53.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.198+0000 7fa6d5400640 1 --2- 172.21.15.28:0/1616778032 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563b5c491f70 0x563b5c48cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:53.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.198+0000 7fa6d5e00640 1 -- 172.21.15.28:0/1616778032 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563b5c491f70 msgr2=0x563b5c48cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:11:53.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.198+0000 7fa6d5e00640 1 --2- 172.21.15.28:0/1616778032 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563b5c491f70 0x563b5c48cc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:53.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.198+0000 7fa6d5e00640 1 -- 172.21.15.28:0/1616778032 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563b5c4a0d80 msgr2=0x563b5c4b9090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:11:53.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.198+0000 7fa6d5e00640 1 --2- 172.21.15.28:0/1616778032 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563b5c4a0d80 0x563b5c4b9090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:53.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.198+0000 7fa6d5e00640 1 -- 172.21.15.28:0/1616778032 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563b5c49c270 con 0x563b5c491650 2024-05-12T22:11:53.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.198+0000 7fa6d6800640 1 --2- 172.21.15.28:0/1616778032 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563b5c4a0d80 0x563b5c4b9090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:11:53.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.199+0000 7fa6d5e00640 1 --2- 172.21.15.28:0/1616778032 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563b5c491650 0x563b5c4aca70 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fa6c8003ea0 tx=0x7fa6c8011900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8884ad6b5b1a3981 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:11:53.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.199+0000 7fa6c2600640 1 -- 172.21.15.28:0/1616778032 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6c80040d0 con 0x563b5c491650 2024-05-12T22:11:53.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.199+0000 7fa6c2600640 1 -- 172.21.15.28:0/1616778032 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa6c8004270 con 0x563b5c491650 2024-05-12T22:11:53.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.199+0000 7fa6c2600640 1 -- 172.21.15.28:0/1616778032 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6c802ad40 con 0x563b5c491650 2024-05-12T22:11:53.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.200+0000 7fa6dfb77740 1 -- 172.21.15.28:0/1616778032 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563b5c491650 msgr2=0x563b5c4aca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:11:53.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.200+0000 7fa6dfb77740 1 --2- 172.21.15.28:0/1616778032 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563b5c491650 0x563b5c4aca70 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fa6c8003ea0 tx=0x7fa6c8011900 comp rx=0 tx=0).stop 2024-05-12T22:11:53.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.200+0000 7fa6dfb77740 1 -- 172.21.15.28:0/1616778032 shutdown_connections 2024-05-12T22:11:53.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.200+0000 7fa6dfb77740 1 --2- 172.21.15.28:0/1616778032 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563b5c4a0d80 0x563b5c4b9090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:53.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.200+0000 7fa6dfb77740 1 --2- 172.21.15.28:0/1616778032 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563b5c491f70 0x563b5c48cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:53.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.200+0000 7fa6dfb77740 1 --2- 172.21.15.28:0/1616778032 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563b5c491650 0x563b5c4aca70 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:53.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.200+0000 7fa6dfb77740 1 -- 172.21.15.28:0/1616778032 >> 172.21.15.28:0/1616778032 conn(0x563b5c48dca0 msgr2=0x563b5c48c530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:11:53.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.200+0000 7fa6dfb77740 1 -- 172.21.15.28:0/1616778032 shutdown_connections 2024-05-12T22:11:53.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.200+0000 7fa6dfb77740 1 -- 172.21.15.28:0/1616778032 wait complete. 2024-05-12T22:11:53.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.200+0000 7fa6dfb77740 1 -- start start 2024-05-12T22:11:53.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.201+0000 7fa6dfb77740 1 Processor -- start 2024-05-12T22:11:53.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.201+0000 7fa6dfb77740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563b5c491650 0x563b5c4a9400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:53.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.201+0000 7fa6dfb77740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563b5c491f70 0x563b5c4a4770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:53.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.201+0000 7fa6dfb77740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563b5c4a0d80 0x563b5c4a4d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:53.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.201+0000 7fa6dfb77740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa6d81a9960 con 0x563b5c491650 2024-05-12T22:11:53.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.201+0000 7fa6dfb77740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fa6d81aadd0 con 0x563b5c4a0d80 2024-05-12T22:11:53.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.201+0000 7fa6dfb77740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa6d81aa030 con 0x563b5c491f70 2024-05-12T22:11:53.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.201+0000 7fa6dfb77740 10 client.0 _ll_register_callbacks cb 0x7fa6d009c680 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:11:53.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.201+0000 7fa6d5e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563b5c491650 0x563b5c4a9400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:53.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.201+0000 7fa6d5e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563b5c491650 0x563b5c4a9400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48786/0 (socket says 172.21.15.28:48786) 2024-05-12T22:11:53.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.201+0000 7fa6d5e00640 1 -- 172.21.15.28:0/1021242724 learned_addr learned my addr 172.21.15.28:0/1021242724 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:11:53.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.202+0000 7fa6d5400640 1 --2- 172.21.15.28:0/1021242724 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563b5c491f70 0x563b5c4a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:53.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.202+0000 7fa6d6800640 1 --2- 172.21.15.28:0/1021242724 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563b5c4a0d80 0x563b5c4a4d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:53.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.202+0000 7fa6d5e00640 1 -- 172.21.15.28:0/1021242724 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563b5c491f70 msgr2=0x563b5c4a4770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:11:53.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.202+0000 7fa6d5e00640 1 --2- 172.21.15.28:0/1021242724 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563b5c491f70 0x563b5c4a4770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:53.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.202+0000 7fa6d5e00640 1 -- 172.21.15.28:0/1021242724 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563b5c4a0d80 msgr2=0x563b5c4a4d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:11:53.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.202+0000 7fa6d5e00640 1 --2- 172.21.15.28:0/1021242724 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563b5c4a0d80 0x563b5c4a4d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:53.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.202+0000 7fa6d5e00640 1 -- 172.21.15.28:0/1021242724 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563b5c4a5540 con 0x563b5c491650 2024-05-12T22:11:53.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.202+0000 7fa6d6800640 1 --2- 172.21.15.28:0/1021242724 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563b5c4a0d80 0x563b5c4a4d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:11:53.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.202+0000 7fa6d5e00640 1 --2- 172.21.15.28:0/1021242724 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563b5c491650 0x563b5c4a9400 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fa6c8005fc0 tx=0x7fa6c802aae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27d2ecd27228d87f server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:11:53.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.202+0000 7fa6b7400640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-12T22:11:53.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.203+0000 7fa6b7400640 1 -- 172.21.15.28:0/1021242724 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6c80040d0 con 0x563b5c491650 2024-05-12T22:11:53.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.203+0000 7fa6b7400640 1 -- 172.21.15.28:0/1021242724 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7fa6c8005350 con 0x563b5c491650 2024-05-12T22:11:53.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.203+0000 7fa6dfb77740 1 -- 172.21.15.28:0/1021242724 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa6c8004270 con 0x563b5c491650 2024-05-12T22:11:53.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.203+0000 7fa6b7400640 1 -- 172.21.15.28:0/1021242724 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fa6c8005550 con 0x563b5c491650 2024-05-12T22:11:53.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.203+0000 7fa6dfb77740 10 client.15858 fetch_fsmap learned FSMap version 25 2024-05-12T22:11:53.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.203+0000 7fa6dfb77740 1 -- 172.21.15.28:0/1021242724 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x563b5c5179d0 con 0x563b5c491650 2024-05-12T22:11:53.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.203+0000 7fa6b7400640 1 -- 172.21.15.28:0/1021242724 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa6c8004270 con 0x563b5c491650 2024-05-12T22:11:53.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.203+0000 7fa6b7400640 1 -- 172.21.15.28:0/1021242724 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7fa6c803c030 con 0x563b5c491650 2024-05-12T22:11:53.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.204+0000 7fa6dfb77740 10 client.15858 fetch_fsmap finished waiting for FSMap version 25 2024-05-12T22:11:53.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.204+0000 7fa6dfb77740 10 client.15858 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:11:53.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.204+0000 7fa6dfb77740 20 client.15858 populate_metadata read hostname 'smithi028' 2024-05-12T22:11:53.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.204+0000 7fa6dfb77740 1 --2- 172.21.15.28:0/1021242724 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x563b5c517f60 0x563b5c51a380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:53.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.204+0000 7fa6dfb77740 4 client.15858 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-12T22:11:53.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.204+0000 7fa6dfb77740 1 -- 172.21.15.28:0/1021242724 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563b5c3fe4b0 con 0x563b5c517f60 2024-05-12T22:11:53.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.204+0000 7fa6d5400640 1 --2- 172.21.15.28:0/1021242724 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x563b5c517f60 0x563b5c51a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:53.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.205+0000 7fa6d5400640 1 --2- 172.21.15.28:0/1021242724 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x563b5c517f60 0x563b5c51a380 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9e4e5c04dbbf965f server_cookie=82bff2fb04d4ff37 in_seq=0 out_seq=0 2024-05-12T22:11:53.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.205+0000 7fa6b7400640 10 client.15858 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:11:53.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.205+0000 7fa6b7400640 1 -- 172.21.15.28:0/1021242724 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563b5c3fe4b0 con 0x563b5c517f60 2024-05-12T22:11:53.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.205+0000 7fa6b7400640 10 client.15858 handle_command_reply: tid=0 2024-05-12T22:11:53.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.205+0000 7fa6dfb77740 1 client.15858 shutdown 2024-05-12T22:11:53.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.205+0000 7fa6dfb77740 1 -- 172.21.15.28:0/1021242724 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563b5c491650 msgr2=0x563b5c4a9400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:11:53.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.205+0000 7fa6dfb77740 1 --2- 172.21.15.28:0/1021242724 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563b5c491650 0x563b5c4a9400 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fa6c8005fc0 tx=0x7fa6c802aae0 comp rx=0 tx=0).stop 2024-05-12T22:11:53.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.205+0000 7fa6dfb77740 1 -- 172.21.15.28:0/1021242724 shutdown_connections 2024-05-12T22:11:53.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.205+0000 7fa6dfb77740 1 --2- 172.21.15.28:0/1021242724 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x563b5c517f60 0x563b5c51a380 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:53.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.205+0000 7fa6dfb77740 1 --2- 172.21.15.28:0/1021242724 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563b5c4a0d80 0x563b5c4a4d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:53.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.205+0000 7fa6dfb77740 1 --2- 172.21.15.28:0/1021242724 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563b5c491f70 0x563b5c4a4770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:53.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.205+0000 7fa6dfb77740 1 --2- 172.21.15.28:0/1021242724 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563b5c491650 0x563b5c4a9400 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:53.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.205+0000 7fa6dfb77740 1 -- 172.21.15.28:0/1021242724 >> 172.21.15.28:0/1021242724 conn(0x563b5c48dca0 msgr2=0x563b5c48cdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:11:53.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.205+0000 7fa6b7400640 0 client.15858 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:11:53.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.205+0000 7fa6dfb77740 1 -- 172.21.15.28:0/1021242724 shutdown_connections 2024-05-12T22:11:53.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.205+0000 7fa6dfb77740 1 -- 172.21.15.28:0/1021242724 wait complete. 2024-05-12T22:11:53.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.205+0000 7fa6dfb77740 20 client.15858 trim_cache size 0 max 16384 2024-05-12T22:11:53.213 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-05-12T22:11:53.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.206+0000 7fa6dcc00640 1 -- 172.21.15.28:0/1054037948 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fa6bc0a2d00 msgr2=0x7fa6bc0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:11:53.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.206+0000 7fa6dcc00640 1 --2- 172.21.15.28:0/1054037948 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fa6bc0a2d00 0x7fa6bc0a51c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fa6c4005c40 tx=0x7fa6c4005b50 comp rx=0 tx=0).stop 2024-05-12T22:11:53.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.206+0000 7fa6dcc00640 1 -- 172.21.15.28:0/1054037948 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6d8106710 msgr2=0x7fa6d819ca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:11:53.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.206+0000 7fa6dcc00640 1 --2- 172.21.15.28:0/1054037948 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6d8106710 0x7fa6d819ca50 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fa6d0009fd0 tx=0x7fa6d000ef10 comp rx=0 tx=0).stop 2024-05-12T22:11:53.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.206+0000 7fa6dcc00640 1 -- 172.21.15.28:0/1054037948 shutdown_connections 2024-05-12T22:11:53.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.206+0000 7fa6dcc00640 1 --2- 172.21.15.28:0/1054037948 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fa6bc0a2d00 0x7fa6bc0a51c0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:53.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.206+0000 7fa6dcc00640 1 --2- 172.21.15.28:0/1054037948 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6d8106710 0x7fa6d819ca50 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:53.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.206+0000 7fa6dcc00640 1 --2- 172.21.15.28:0/1054037948 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa6d81010c0 0x7fa6d81986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:53.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.206+0000 7fa6dcc00640 1 --2- 172.21.15.28:0/1054037948 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa6d8100700 0x7fa6d8198190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:53.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.206+0000 7fa6dcc00640 1 -- 172.21.15.28:0/1054037948 >> 172.21.15.28:0/1054037948 conn(0x7fa6d8077830 msgr2=0x7fa6d8068d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:11:53.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.206+0000 7fa6dcc00640 1 -- 172.21.15.28:0/1054037948 shutdown_connections 2024-05-12T22:11:53.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:53.207+0000 7fa6dcc00640 1 -- 172.21.15.28:0/1054037948 wait complete. 2024-05-12T22:11:53.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:11:53 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/2347010975' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:11:53.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:11:53 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/2347010975' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:11:53.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:11:53 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/2347010975' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:11:54.226 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-12T22:11:54.227 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-12T22:11:54.227 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-05-12T22:11:54.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:11:54 smithi028 ceph-mon[29847]: pgmap v90: 97 pgs: 97 active+clean; 95 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 KiB/s wr, 8 op/s 2024-05-12T22:11:54.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:11:54 smithi186 ceph-mon[34785]: pgmap v90: 97 pgs: 97 active+clean; 95 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 KiB/s wr, 8 op/s 2024-05-12T22:11:54.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.457+0000 7fce43e00640 1 -- 172.21.15.28:0/2155792419 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fce44107dd0 msgr2=0x7fce44108230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:11:54.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.457+0000 7fce43e00640 1 --2- 172.21.15.28:0/2155792419 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fce44107dd0 0x7fce44108230 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fce30009900 tx=0x7fce3002a480 comp rx=0 tx=0).stop 2024-05-12T22:11:54.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.457+0000 7fce43e00640 1 -- 172.21.15.28:0/2155792419 shutdown_connections 2024-05-12T22:11:54.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.457+0000 7fce43e00640 1 --2- 172.21.15.28:0/2155792419 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fce44108770 0x7fce44114b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:54.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.457+0000 7fce43e00640 1 --2- 172.21.15.28:0/2155792419 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fce44107dd0 0x7fce44108230 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:54.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.457+0000 7fce43e00640 1 --2- 172.21.15.28:0/2155792419 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce4410ddf0 0x7fce4410e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:54.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.457+0000 7fce43e00640 1 -- 172.21.15.28:0/2155792419 >> 172.21.15.28:0/2155792419 conn(0x7fce44077650 msgr2=0x7fce44106650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:11:54.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.457+0000 7fce43e00640 1 -- 172.21.15.28:0/2155792419 shutdown_connections 2024-05-12T22:11:54.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.457+0000 7fce43e00640 1 -- 172.21.15.28:0/2155792419 wait complete. 2024-05-12T22:11:54.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.458+0000 7fce43e00640 1 Processor -- start 2024-05-12T22:11:54.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.459+0000 7fce43e00640 1 -- start start 2024-05-12T22:11:54.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.459+0000 7fce43e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fce44107dd0 0x7fce4419e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:54.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.459+0000 7fce43e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fce44108770 0x7fce4419ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:54.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.459+0000 7fce43e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce4410ddf0 0x7fce441a2de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:54.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.459+0000 7fce43e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fce44118120 con 0x7fce4410ddf0 2024-05-12T22:11:54.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.459+0000 7fce43e00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fce44117fa0 con 0x7fce44107dd0 2024-05-12T22:11:54.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.459+0000 7fce43e00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fce441182a0 con 0x7fce44108770 2024-05-12T22:11:54.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.459+0000 7fce43400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce4410ddf0 0x7fce441a2de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:54.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.459+0000 7fce42a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fce44107dd0 0x7fce4419e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:54.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.459+0000 7fce42a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fce44107dd0 0x7fce4419e520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:37064/0 (socket says 172.21.15.28:37064) 2024-05-12T22:11:54.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.459+0000 7fce42a00640 1 -- 172.21.15.28:0/3791662766 learned_addr learned my addr 172.21.15.28:0/3791662766 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:11:54.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.459+0000 7fce42000640 1 --2- 172.21.15.28:0/3791662766 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fce44108770 0x7fce4419ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:54.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.460+0000 7fce43400640 1 -- 172.21.15.28:0/3791662766 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fce44108770 msgr2=0x7fce4419ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:11:54.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.460+0000 7fce43400640 1 --2- 172.21.15.28:0/3791662766 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fce44108770 0x7fce4419ea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:54.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.460+0000 7fce43400640 1 -- 172.21.15.28:0/3791662766 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fce44107dd0 msgr2=0x7fce4419e520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:11:54.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.460+0000 7fce43400640 1 --2- 172.21.15.28:0/3791662766 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fce44107dd0 0x7fce4419e520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:54.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.460+0000 7fce43400640 1 -- 172.21.15.28:0/3791662766 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce441a3510 con 0x7fce4410ddf0 2024-05-12T22:11:54.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.460+0000 7fce42a00640 1 --2- 172.21.15.28:0/3791662766 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fce44107dd0 0x7fce4419e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:11:54.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.460+0000 7fce43400640 1 --2- 172.21.15.28:0/3791662766 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce4410ddf0 0x7fce441a2de0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fce3c005fd0 tx=0x7fce3c00ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:11:54.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.460+0000 7fce3b400640 1 -- 172.21.15.28:0/3791662766 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce3c00cd80 con 0x7fce4410ddf0 2024-05-12T22:11:54.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.461+0000 7fce3b400640 1 -- 172.21.15.28:0/3791662766 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fce3c005020 con 0x7fce4410ddf0 2024-05-12T22:11:54.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.461+0000 7fce43e00640 1 -- 172.21.15.28:0/3791662766 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce441a3800 con 0x7fce4410ddf0 2024-05-12T22:11:54.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.461+0000 7fce3b400640 1 -- 172.21.15.28:0/3791662766 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce3c010720 con 0x7fce4410ddf0 2024-05-12T22:11:54.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.461+0000 7fce43e00640 1 -- 172.21.15.28:0/3791662766 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce441ab090 con 0x7fce4410ddf0 2024-05-12T22:11:54.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.464+0000 7fce4c82c740 1 Processor -- start 2024-05-12T22:11:54.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.464+0000 7fce3b400640 1 -- 172.21.15.28:0/3791662766 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7fce3c0051d0 con 0x7fce4410ddf0 2024-05-12T22:11:54.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.464+0000 7fce4c82c740 1 -- start start 2024-05-12T22:11:54.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.465+0000 7fce4c82c740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562a30452500 0x562a304528e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:54.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.465+0000 7fce4c82c740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562a30452eb0 0x562a3045cf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:54.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.465+0000 7fce4c82c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a3045d520 0x562a3045f970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:54.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.465+0000 7fce4c82c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562a303b67c0 con 0x562a3045d520 2024-05-12T22:11:54.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.465+0000 7fce4c82c740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x562a3035e790 con 0x562a30452eb0 2024-05-12T22:11:54.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.465+0000 7fce4c82c740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x562a303b2780 con 0x562a30452500 2024-05-12T22:11:54.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.465+0000 7fce42000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a3045d520 0x562a3045f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:54.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.465+0000 7fce42000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562a30452500 0x562a304528e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:54.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.465+0000 7fce42a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562a30452eb0 0x562a3045cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:54.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.465+0000 7fce42000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562a30452500 0x562a304528e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:59344/0 (socket says 172.21.15.28:59344) 2024-05-12T22:11:54.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.465+0000 7fce42000640 1 -- 172.21.15.28:0/3407300168 learned_addr learned my addr 172.21.15.28:0/3407300168 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:11:54.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.465+0000 7fce3b400640 1 --2- 172.21.15.28:0/3791662766 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fce280a2d00 0x7fce280a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:54.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.466+0000 7fce42a00640 1 -- 172.21.15.28:0/3407300168 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562a30452500 msgr2=0x562a304528e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:11:54.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.466+0000 7fce42a00640 1 --2- 172.21.15.28:0/3407300168 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562a30452500 0x562a304528e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:54.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.466+0000 7fce42a00640 1 -- 172.21.15.28:0/3407300168 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a3045d520 msgr2=0x562a3045f970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:11:54.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.466+0000 7fce42a00640 1 --2- 172.21.15.28:0/3407300168 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a3045d520 0x562a3045f970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:54.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.466+0000 7fce3b400640 1 -- 172.21.15.28:0/3791662766 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7fce3c0cf9b0 con 0x7fce4410ddf0 2024-05-12T22:11:54.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.466+0000 7fce42a00640 1 -- 172.21.15.28:0/3407300168 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562a3037e6a0 con 0x562a30452eb0 2024-05-12T22:11:54.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.466+0000 7fce43400640 1 --2- 172.21.15.28:0/3791662766 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fce280a2d00 0x7fce280a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:54.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.466+0000 7fce42a00640 1 --2- 172.21.15.28:0/3407300168 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562a30452eb0 0x562a3045cf50 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fce34011510 tx=0x7fce340119d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=221d52c75dd668d0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:11:54.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.466+0000 7fce2fe00640 1 -- 172.21.15.28:0/3407300168 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce34005020 con 0x562a30452eb0 2024-05-12T22:11:54.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.467+0000 7fce43400640 1 --2- 172.21.15.28:0/3791662766 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fce280a2d00 0x7fce280a51c0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fce3c0cd1e0 tx=0x7fce3c005840 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:11:54.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.467+0000 7fce2fe00640 1 -- 172.21.15.28:0/3407300168 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fce340047a0 con 0x562a30452eb0 2024-05-12T22:11:54.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.467+0000 7fce2fe00640 1 -- 172.21.15.28:0/3407300168 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce3400f8e0 con 0x562a30452eb0 2024-05-12T22:11:54.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.467+0000 7fce4c82c740 1 -- 172.21.15.28:0/3407300168 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562a30452eb0 msgr2=0x562a3045cf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:11:54.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.467+0000 7fce4c82c740 1 --2- 172.21.15.28:0/3407300168 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562a30452eb0 0x562a3045cf50 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fce34011510 tx=0x7fce340119d0 comp rx=0 tx=0).stop 2024-05-12T22:11:54.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.467+0000 7fce4c82c740 1 -- 172.21.15.28:0/3407300168 shutdown_connections 2024-05-12T22:11:54.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.467+0000 7fce4c82c740 1 --2- 172.21.15.28:0/3407300168 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a3045d520 0x562a3045f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:54.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.467+0000 7fce4c82c740 1 --2- 172.21.15.28:0/3407300168 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562a30452eb0 0x562a3045cf50 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:54.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.467+0000 7fce4c82c740 1 --2- 172.21.15.28:0/3407300168 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562a30452500 0x562a304528e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:54.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.467+0000 7fce4c82c740 1 -- 172.21.15.28:0/3407300168 >> 172.21.15.28:0/3407300168 conn(0x562a3044e5b0 msgr2=0x562a3044e990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:11:54.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.468+0000 7fce4c82c740 1 -- 172.21.15.28:0/3407300168 shutdown_connections 2024-05-12T22:11:54.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.468+0000 7fce4c82c740 1 -- 172.21.15.28:0/3407300168 wait complete. 2024-05-12T22:11:54.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.468+0000 7fce4c82c740 1 -- start start 2024-05-12T22:11:54.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.468+0000 7fce4c82c740 1 Processor -- start 2024-05-12T22:11:54.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.469+0000 7fce4c82c740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562a30452500 0x562a3045a410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:54.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.469+0000 7fce4c82c740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562a30452eb0 0x562a3045a950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:54.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.469+0000 7fce4c82c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a3045d520 0x562a3045ae90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:54.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.469+0000 7fce4c82c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562a30460530 con 0x562a3045d520 2024-05-12T22:11:54.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.469+0000 7fce4c82c740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x562a30371a90 con 0x562a30452eb0 2024-05-12T22:11:54.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.469+0000 7fce4c82c740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x562a304606b0 con 0x562a30452500 2024-05-12T22:11:54.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.469+0000 7fce4c82c740 10 client.0 _ll_register_callbacks cb 0x562a303acaf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:11:54.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.469+0000 7fce42000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562a30452500 0x562a3045a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:54.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.469+0000 7fce42000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562a30452500 0x562a3045a410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:59350/0 (socket says 172.21.15.28:59350) 2024-05-12T22:11:54.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.469+0000 7fce42000640 1 -- 172.21.15.28:0/3886965621 learned_addr learned my addr 172.21.15.28:0/3886965621 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:11:54.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.469+0000 7fce42000640 1 --2- 172.21.15.28:0/3886965621 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a3045d520 0x562a3045ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:54.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.469+0000 7fce42a00640 1 --2- 172.21.15.28:0/3886965621 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562a30452eb0 0x562a3045a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:54.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.469+0000 7fce42000640 1 -- 172.21.15.28:0/3886965621 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562a30452eb0 msgr2=0x562a3045a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:11:54.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.469+0000 7fce42000640 1 --2- 172.21.15.28:0/3886965621 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562a30452eb0 0x562a3045a950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:54.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.469+0000 7fce42000640 1 -- 172.21.15.28:0/3886965621 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a3045d520 msgr2=0x562a3045ae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:11:54.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.469+0000 7fce42000640 1 --2- 172.21.15.28:0/3886965621 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a3045d520 0x562a3045ae90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:54.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.469+0000 7fce42000640 1 -- 172.21.15.28:0/3886965621 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562a301927e0 con 0x562a30452500 2024-05-12T22:11:54.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.470+0000 7fce42000640 1 --2- 172.21.15.28:0/3886965621 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562a30452500 0x562a3045a410 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fce30000c00 tx=0x7fce30005d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=97c49e657be6210f server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:11:54.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.470+0000 7fce2cc00640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-12T22:11:54.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.470+0000 7fce2cc00640 1 -- 172.21.15.28:0/3886965621 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce30035040 con 0x562a30452500 2024-05-12T22:11:54.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.470+0000 7fce2cc00640 1 -- 172.21.15.28:0/3886965621 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7fce30005250 con 0x562a30452500 2024-05-12T22:11:54.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.470+0000 7fce4c82c740 1 -- 172.21.15.28:0/3886965621 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fce340047a0 con 0x562a30452500 2024-05-12T22:11:54.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.470+0000 7fce2cc00640 1 -- 172.21.15.28:0/3886965621 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fce30032e30 con 0x562a30452500 2024-05-12T22:11:54.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.471+0000 7fce4c82c740 10 client.25951 fetch_fsmap learned FSMap version 25 2024-05-12T22:11:54.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.471+0000 7fce4c82c740 1 -- 172.21.15.28:0/3886965621 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x562a301f2f70 con 0x562a30452500 2024-05-12T22:11:54.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.471+0000 7fce2cc00640 1 -- 172.21.15.28:0/3886965621 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fce340047a0 con 0x562a30452500 2024-05-12T22:11:54.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.471+0000 7fce2cc00640 1 -- 172.21.15.28:0/3886965621 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7fce30003680 con 0x562a30452500 2024-05-12T22:11:54.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.471+0000 7fce4c82c740 10 client.25951 fetch_fsmap finished waiting for FSMap version 25 2024-05-12T22:11:54.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.471+0000 7fce4c82c740 10 client.25951 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:11:54.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.471+0000 7fce4c82c740 20 client.25951 populate_metadata read hostname 'smithi028' 2024-05-12T22:11:54.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.471+0000 7fce4c82c740 1 --2- 172.21.15.28:0/3886965621 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x562a3046d6c0 0x562a3046fad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:54.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.471+0000 7fce4c82c740 4 client.25951 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-12T22:11:54.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.471+0000 7fce4c82c740 1 -- 172.21.15.28:0/3886965621 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562a303ca3a0 con 0x562a3046d6c0 2024-05-12T22:11:54.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.472+0000 7fce42a00640 1 --2- 172.21.15.28:0/3886965621 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x562a3046d6c0 0x562a3046fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:54.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.472+0000 7fce42a00640 1 --2- 172.21.15.28:0/3886965621 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x562a3046d6c0 0x562a3046fad0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9385d5a821071d6d server_cookie=d2b10cf080d037a0 in_seq=0 out_seq=0 2024-05-12T22:11:54.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.472+0000 7fce2cc00640 10 client.25951 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:11:54.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.473+0000 7fce2cc00640 1 -- 172.21.15.28:0/3886965621 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562a303ca3a0 con 0x562a3046d6c0 2024-05-12T22:11:54.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.473+0000 7fce2cc00640 10 client.25951 handle_command_reply: tid=0 2024-05-12T22:11:54.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.473+0000 7fce4c82c740 1 client.25951 shutdown 2024-05-12T22:11:54.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.473+0000 7fce4c82c740 1 -- 172.21.15.28:0/3886965621 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562a30452500 msgr2=0x562a3045a410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:11:54.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.473+0000 7fce4c82c740 1 --2- 172.21.15.28:0/3886965621 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562a30452500 0x562a3045a410 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fce30000c00 tx=0x7fce30005d50 comp rx=0 tx=0).stop 2024-05-12T22:11:54.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.473+0000 7fce4c82c740 1 -- 172.21.15.28:0/3886965621 shutdown_connections 2024-05-12T22:11:54.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.473+0000 7fce4c82c740 1 --2- 172.21.15.28:0/3886965621 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x562a3046d6c0 0x562a3046fad0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:54.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.473+0000 7fce4c82c740 1 --2- 172.21.15.28:0/3886965621 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a3045d520 0x562a3045ae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:54.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.473+0000 7fce4c82c740 1 --2- 172.21.15.28:0/3886965621 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562a30452eb0 0x562a3045a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:54.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.473+0000 7fce4c82c740 1 --2- 172.21.15.28:0/3886965621 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562a30452500 0x562a3045a410 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:54.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.473+0000 7fce2cc00640 0 client.25951 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:11:54.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.473+0000 7fce4c82c740 1 -- 172.21.15.28:0/3886965621 >> 172.21.15.28:0/3886965621 conn(0x562a3044e5b0 msgr2=0x562a30454630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:11:54.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.474+0000 7fce4c82c740 1 -- 172.21.15.28:0/3886965621 shutdown_connections 2024-05-12T22:11:54.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.474+0000 7fce4c82c740 1 -- 172.21.15.28:0/3886965621 wait complete. 2024-05-12T22:11:54.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.474+0000 7fce4c82c740 20 client.25951 trim_cache size 0 max 16384 2024-05-12T22:11:54.491 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:11:54 smithi086 ceph-mon[34297]: pgmap v90: 97 pgs: 97 active+clean; 95 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 KiB/s wr, 8 op/s 2024-05-12T22:11:54.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.493+0000 7fce4c82c740 1 Processor -- start 2024-05-12T22:11:54.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.493+0000 7fce4c82c740 1 -- start start 2024-05-12T22:11:54.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.493+0000 7fce4c82c740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562a30452500 0x562a30469450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:54.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.493+0000 7fce4c82c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a30452eb0 0x562a3045acb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:54.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.493+0000 7fce4c82c740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562a3045d520 0x562a30475350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:54.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.493+0000 7fce4c82c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fce441b6110 con 0x562a30452eb0 2024-05-12T22:11:54.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.493+0000 7fce4c82c740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fce441b39e0 con 0x562a30452500 2024-05-12T22:11:54.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.493+0000 7fce4c82c740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fce441b1830 con 0x562a3045d520 2024-05-12T22:11:54.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.494+0000 7fce42a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a30452eb0 0x562a3045acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:54.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.494+0000 7fce42000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562a30452500 0x562a30469450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:54.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.494+0000 7fce42000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562a3045d520 0x562a30475350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:54.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.494+0000 7fce42a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a30452eb0 0x562a3045acb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48850/0 (socket says 172.21.15.28:48850) 2024-05-12T22:11:54.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.494+0000 7fce42a00640 1 -- 172.21.15.28:0/2324957247 learned_addr learned my addr 172.21.15.28:0/2324957247 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:11:54.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.494+0000 7fce42000640 1 -- 172.21.15.28:0/2324957247 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562a30452500 msgr2=0x562a30469450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:11:54.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.494+0000 7fce42000640 1 --2- 172.21.15.28:0/2324957247 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562a30452500 0x562a30469450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:54.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.494+0000 7fce42000640 1 -- 172.21.15.28:0/2324957247 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a30452eb0 msgr2=0x562a3045acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:11:54.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.494+0000 7fce42000640 1 --2- 172.21.15.28:0/2324957247 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a30452eb0 0x562a3045acb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:54.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.494+0000 7fce42000640 1 -- 172.21.15.28:0/2324957247 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562a30452c50 con 0x562a3045d520 2024-05-12T22:11:54.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.494+0000 7fce42000640 1 --2- 172.21.15.28:0/2324957247 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562a3045d520 0x562a30475350 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fce30032900 tx=0x7fce30005fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4b0675b2fe19f24c server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:11:54.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.495+0000 7fce2f400640 1 -- 172.21.15.28:0/2324957247 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce30035030 con 0x562a3045d520 2024-05-12T22:11:54.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.495+0000 7fce2f400640 1 -- 172.21.15.28:0/2324957247 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fce3005c040 con 0x562a3045d520 2024-05-12T22:11:54.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.495+0000 7fce2f400640 1 -- 172.21.15.28:0/2324957247 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce3005b070 con 0x562a3045d520 2024-05-12T22:11:54.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.495+0000 7fce4c82c740 1 -- 172.21.15.28:0/2324957247 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562a3045d520 msgr2=0x562a30475350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:11:54.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.495+0000 7fce4c82c740 1 --2- 172.21.15.28:0/2324957247 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562a3045d520 0x562a30475350 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fce30032900 tx=0x7fce30005fd0 comp rx=0 tx=0).stop 2024-05-12T22:11:54.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.495+0000 7fce4c82c740 1 -- 172.21.15.28:0/2324957247 shutdown_connections 2024-05-12T22:11:54.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.495+0000 7fce4c82c740 1 --2- 172.21.15.28:0/2324957247 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562a3045d520 0x562a30475350 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:54.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.495+0000 7fce4c82c740 1 --2- 172.21.15.28:0/2324957247 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a30452eb0 0x562a3045acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:54.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.495+0000 7fce4c82c740 1 --2- 172.21.15.28:0/2324957247 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562a30452500 0x562a30469450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:54.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.495+0000 7fce4c82c740 1 -- 172.21.15.28:0/2324957247 >> 172.21.15.28:0/2324957247 conn(0x562a3044e5b0 msgr2=0x562a3045bc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:11:54.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.496+0000 7fce4c82c740 1 -- 172.21.15.28:0/2324957247 shutdown_connections 2024-05-12T22:11:54.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.496+0000 7fce4c82c740 1 -- 172.21.15.28:0/2324957247 wait complete. 2024-05-12T22:11:54.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.496+0000 7fce4c82c740 1 -- start start 2024-05-12T22:11:54.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.496+0000 7fce4c82c740 1 Processor -- start 2024-05-12T22:11:54.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.497+0000 7fce4c82c740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562a30452500 0x562a30461320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:54.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.497+0000 7fce4c82c740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562a30452eb0 0x562a30461860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:54.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.497+0000 7fce4c82c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a3045d520 0x562a30461da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:54.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.497+0000 7fce4c82c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fce441b1ea0 con 0x562a3045d520 2024-05-12T22:11:54.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.497+0000 7fce4c82c740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fce441b3310 con 0x562a30452eb0 2024-05-12T22:11:54.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.497+0000 7fce4c82c740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fce441b2570 con 0x562a30452500 2024-05-12T22:11:54.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.497+0000 7fce4c82c740 10 client.0 _ll_register_callbacks cb 0x7fce300327f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:11:54.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.497+0000 7fce42000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562a30452500 0x562a30461320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:54.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.497+0000 7fce42000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562a30452500 0x562a30461320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:59366/0 (socket says 172.21.15.28:59366) 2024-05-12T22:11:54.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.497+0000 7fce42000640 1 -- 172.21.15.28:0/1275785240 learned_addr learned my addr 172.21.15.28:0/1275785240 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:11:54.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.497+0000 7fce42000640 1 --2- 172.21.15.28:0/1275785240 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a3045d520 0x562a30461da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:54.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.497+0000 7fce42a00640 1 --2- 172.21.15.28:0/1275785240 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562a30452eb0 0x562a30461860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:54.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.498+0000 7fce42000640 1 -- 172.21.15.28:0/1275785240 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562a30452500 msgr2=0x562a30461320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:11:54.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.498+0000 7fce42000640 1 --2- 172.21.15.28:0/1275785240 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562a30452500 0x562a30461320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:54.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.498+0000 7fce42000640 1 -- 172.21.15.28:0/1275785240 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562a30452eb0 msgr2=0x562a30461860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:11:54.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.498+0000 7fce42000640 1 --2- 172.21.15.28:0/1275785240 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562a30452eb0 0x562a30461860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:54.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.498+0000 7fce42000640 1 -- 172.21.15.28:0/1275785240 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562a3008a170 con 0x562a3045d520 2024-05-12T22:11:54.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.498+0000 7fce42a00640 1 --2- 172.21.15.28:0/1275785240 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562a30452eb0 0x562a30461860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:11:54.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.498+0000 7fce42000640 1 --2- 172.21.15.28:0/1275785240 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a3045d520 0x562a30461da0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fce3002a9a0 tx=0x7fce30009680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ee8000547138581 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:11:54.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.498+0000 7fce2cc00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-12T22:11:54.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.498+0000 7fce2cc00640 1 -- 172.21.15.28:0/1275785240 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce30068060 con 0x562a3045d520 2024-05-12T22:11:54.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.498+0000 7fce4c82c740 1 -- 172.21.15.28:0/1275785240 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fce3005c040 con 0x562a3045d520 2024-05-12T22:11:54.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.498+0000 7fce2cc00640 1 -- 172.21.15.28:0/1275785240 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7fce300035f0 con 0x562a3045d520 2024-05-12T22:11:54.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.498+0000 7fce2cc00640 1 -- 172.21.15.28:0/1275785240 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fce3005b050 con 0x562a3045d520 2024-05-12T22:11:54.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.498+0000 7fce4c82c740 10 client.15876 fetch_fsmap learned FSMap version 25 2024-05-12T22:11:54.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.498+0000 7fce4c82c740 1 -- 172.21.15.28:0/1275785240 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x562a304d3cd0 con 0x562a3045d520 2024-05-12T22:11:54.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.499+0000 7fce2cc00640 1 -- 172.21.15.28:0/1275785240 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fce3005c040 con 0x562a3045d520 2024-05-12T22:11:54.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.499+0000 7fce2cc00640 1 -- 172.21.15.28:0/1275785240 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7fce30004960 con 0x562a3045d520 2024-05-12T22:11:54.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.499+0000 7fce4c82c740 10 client.15876 fetch_fsmap finished waiting for FSMap version 25 2024-05-12T22:11:54.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.499+0000 7fce4c82c740 10 client.15876 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:11:54.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.499+0000 7fce4c82c740 20 client.15876 populate_metadata read hostname 'smithi028' 2024-05-12T22:11:54.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.499+0000 7fce4c82c740 1 --2- 172.21.15.28:0/1275785240 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x562a304d4350 0x562a304d6770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:54.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.499+0000 7fce4c82c740 4 client.15876 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-12T22:11:54.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.499+0000 7fce4c82c740 1 -- 172.21.15.28:0/1275785240 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x562a303c3ef0 con 0x562a304d4350 2024-05-12T22:11:54.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.499+0000 7fce42a00640 1 --2- 172.21.15.28:0/1275785240 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x562a304d4350 0x562a304d6770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:54.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.500+0000 7fce42a00640 1 --2- 172.21.15.28:0/1275785240 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x562a304d4350 0x562a304d6770 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b724f8795bd8c8ac server_cookie=2f6977e32cd6848c in_seq=0 out_seq=0 2024-05-12T22:11:54.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.500+0000 7fce2cc00640 10 client.15876 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:11:54.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.501+0000 7fce2cc00640 1 -- 172.21.15.28:0/1275785240 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x562a303c3ef0 con 0x562a304d4350 2024-05-12T22:11:54.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.501+0000 7fce2cc00640 10 client.15876 handle_command_reply: tid=0 2024-05-12T22:11:54.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.501+0000 7fce4c82c740 1 client.15876 shutdown 2024-05-12T22:11:54.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.501+0000 7fce4c82c740 1 -- 172.21.15.28:0/1275785240 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a3045d520 msgr2=0x562a30461da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:11:54.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.501+0000 7fce4c82c740 1 --2- 172.21.15.28:0/1275785240 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a3045d520 0x562a30461da0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fce3002a9a0 tx=0x7fce30009680 comp rx=0 tx=0).stop 2024-05-12T22:11:54.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.501+0000 7fce4c82c740 1 -- 172.21.15.28:0/1275785240 shutdown_connections 2024-05-12T22:11:54.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.501+0000 7fce4c82c740 1 --2- 172.21.15.28:0/1275785240 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x562a304d4350 0x562a304d6770 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:54.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.501+0000 7fce4c82c740 1 --2- 172.21.15.28:0/1275785240 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a3045d520 0x562a30461da0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:54.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.501+0000 7fce4c82c740 1 --2- 172.21.15.28:0/1275785240 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562a30452eb0 0x562a30461860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:54.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.501+0000 7fce2cc00640 0 client.15876 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:11:54.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.501+0000 7fce4c82c740 1 --2- 172.21.15.28:0/1275785240 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562a30452500 0x562a30461320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:54.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.501+0000 7fce4c82c740 1 -- 172.21.15.28:0/1275785240 >> 172.21.15.28:0/1275785240 conn(0x562a3044e5b0 msgr2=0x562a3045c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:11:54.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.502+0000 7fce4c82c740 1 -- 172.21.15.28:0/1275785240 shutdown_connections 2024-05-12T22:11:54.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.502+0000 7fce4c82c740 1 -- 172.21.15.28:0/1275785240 wait complete. 2024-05-12T22:11:54.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.502+0000 7fce4c82c740 20 client.15876 trim_cache size 0 max 16384 2024-05-12T22:11:54.510 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-05-12T22:11:54.510 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-05-12T22:11:54.510 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "cf340000-4a8a-4b66-b60e-039395406168", 2024-05-12T22:11:54.510 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-05-12T22:11:54.510 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-05-12T22:11:54.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.502+0000 7fce43e00640 1 -- 172.21.15.28:0/3791662766 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fce280a2d00 msgr2=0x7fce280a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:11:54.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.502+0000 7fce43e00640 1 --2- 172.21.15.28:0/3791662766 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fce280a2d00 0x7fce280a51c0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fce3c0cd1e0 tx=0x7fce3c005840 comp rx=0 tx=0).stop 2024-05-12T22:11:54.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.502+0000 7fce43e00640 1 -- 172.21.15.28:0/3791662766 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce4410ddf0 msgr2=0x7fce441a2de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:11:54.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.502+0000 7fce43e00640 1 --2- 172.21.15.28:0/3791662766 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce4410ddf0 0x7fce441a2de0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fce3c005fd0 tx=0x7fce3c00ee60 comp rx=0 tx=0).stop 2024-05-12T22:11:54.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.503+0000 7fce43e00640 1 -- 172.21.15.28:0/3791662766 shutdown_connections 2024-05-12T22:11:54.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.503+0000 7fce43e00640 1 --2- 172.21.15.28:0/3791662766 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fce280a2d00 0x7fce280a51c0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:54.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.503+0000 7fce43e00640 1 --2- 172.21.15.28:0/3791662766 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce4410ddf0 0x7fce441a2de0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:54.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.503+0000 7fce43e00640 1 --2- 172.21.15.28:0/3791662766 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fce44108770 0x7fce4419ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:54.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.503+0000 7fce43e00640 1 --2- 172.21.15.28:0/3791662766 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fce44107dd0 0x7fce4419e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:54.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.503+0000 7fce43e00640 1 -- 172.21.15.28:0/3791662766 >> 172.21.15.28:0/3791662766 conn(0x7fce44077650 msgr2=0x7fce44078590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:11:54.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.503+0000 7fce43e00640 1 -- 172.21.15.28:0/3791662766 shutdown_connections 2024-05-12T22:11:54.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.503+0000 7fce43e00640 1 -- 172.21.15.28:0/3791662766 wait complete. 2024-05-12T22:11:54.525 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-12T22:11:54.525 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-12T22:11:54.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.760+0000 7fcdeda00640 1 -- 172.21.15.28:0/1189330316 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcde8103270 msgr2=0x7fcde810b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:11:54.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.760+0000 7fcdeda00640 1 --2- 172.21.15.28:0/1189330316 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcde8103270 0x7fcde810b750 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fcdd0009900 tx=0x7fcdd002a480 comp rx=0 tx=0).stop 2024-05-12T22:11:54.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.760+0000 7fcdeda00640 1 -- 172.21.15.28:0/1189330316 shutdown_connections 2024-05-12T22:11:54.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.760+0000 7fcdeda00640 1 --2- 172.21.15.28:0/1189330316 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcde810bd20 0x7fcde810e150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:54.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.760+0000 7fcdeda00640 1 --2- 172.21.15.28:0/1189330316 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcde8103270 0x7fcde810b750 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:54.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.760+0000 7fcdeda00640 1 --2- 172.21.15.28:0/1189330316 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcde8102950 0x7fcde8102d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:54.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.760+0000 7fcdeda00640 1 -- 172.21.15.28:0/1189330316 >> 172.21.15.28:0/1189330316 conn(0x7fcde80fc0b0 msgr2=0x7fcde80fe4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:11:54.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.760+0000 7fcdeda00640 1 -- 172.21.15.28:0/1189330316 shutdown_connections 2024-05-12T22:11:54.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.760+0000 7fcdeda00640 1 -- 172.21.15.28:0/1189330316 wait complete. 2024-05-12T22:11:54.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.761+0000 7fcdeda00640 1 Processor -- start 2024-05-12T22:11:54.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.761+0000 7fcdeda00640 1 -- start start 2024-05-12T22:11:54.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.762+0000 7fcdeda00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcde8102950 0x7fcde819a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:54.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.762+0000 7fcdeda00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcde8103270 0x7fcde819a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:54.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.762+0000 7fcdeda00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcde810bd20 0x7fcde819ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:54.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.762+0000 7fcdeda00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcde8111b00 con 0x7fcde8102950 2024-05-12T22:11:54.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.762+0000 7fcdeda00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fcde8111980 con 0x7fcde810bd20 2024-05-12T22:11:54.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.762+0000 7fcdeda00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fcde8111c80 con 0x7fcde8103270 2024-05-12T22:11:54.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.762+0000 7fcddfe00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcde8103270 0x7fcde819a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:54.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.762+0000 7fcded000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcde810bd20 0x7fcde819ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:54.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.762+0000 7fcde7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcde8102950 0x7fcde819a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:54.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.762+0000 7fcddfe00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcde8103270 0x7fcde819a750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:59386/0 (socket says 172.21.15.28:59386) 2024-05-12T22:11:54.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.762+0000 7fcded000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcde810bd20 0x7fcde819ead0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:37110/0 (socket says 172.21.15.28:37110) 2024-05-12T22:11:54.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.762+0000 7fcddfe00640 1 -- 172.21.15.28:0/347948312 learned_addr learned my addr 172.21.15.28:0/347948312 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:11:54.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.763+0000 7fcded000640 1 -- 172.21.15.28:0/347948312 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcde8103270 msgr2=0x7fcde819a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:11:54.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.763+0000 7fcded000640 1 --2- 172.21.15.28:0/347948312 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcde8103270 0x7fcde819a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:54.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.763+0000 7fcded000640 1 -- 172.21.15.28:0/347948312 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcde8102950 msgr2=0x7fcde819a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:11:54.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.763+0000 7fcded000640 1 --2- 172.21.15.28:0/347948312 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcde8102950 0x7fcde819a210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:54.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.763+0000 7fcded000640 1 -- 172.21.15.28:0/347948312 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcde819f200 con 0x7fcde810bd20 2024-05-12T22:11:54.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.763+0000 7fcded000640 1 --2- 172.21.15.28:0/347948312 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcde810bd20 0x7fcde819ead0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fcde0005fd0 tx=0x7fcde000ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:11:54.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.763+0000 7fcde5600640 1 -- 172.21.15.28:0/347948312 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcde000cdf0 con 0x7fcde810bd20 2024-05-12T22:11:54.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.763+0000 7fcde5600640 1 -- 172.21.15.28:0/347948312 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcde0005020 con 0x7fcde810bd20 2024-05-12T22:11:54.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.763+0000 7fcde5600640 1 -- 172.21.15.28:0/347948312 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcde00186d0 con 0x7fcde810bd20 2024-05-12T22:11:54.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.764+0000 7fcdeda00640 1 -- 172.21.15.28:0/347948312 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcde819f4f0 con 0x7fcde810bd20 2024-05-12T22:11:54.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.764+0000 7fcdeda00640 1 -- 172.21.15.28:0/347948312 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcde81a6d80 con 0x7fcde810bd20 2024-05-12T22:11:54.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.768+0000 7fcde5600640 1 -- 172.21.15.28:0/347948312 <== mon.2 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7fcde00054d0 con 0x7fcde810bd20 2024-05-12T22:11:54.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.768+0000 7fcdf1f8f740 1 Processor -- start 2024-05-12T22:11:54.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.768+0000 7fcdf1f8f740 1 -- start start 2024-05-12T22:11:54.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.769+0000 7fcdf1f8f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5638c2233dd0 0x5638c22341b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:54.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.769+0000 7fcdf1f8f740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5638c22346f0 0x5638c223e880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:54.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.769+0000 7fcdf1f8f740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5638c223edc0 0x5638c22411b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:54.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.769+0000 7fcdf1f8f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5638c21fc2a0 con 0x5638c2233dd0 2024-05-12T22:11:54.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.769+0000 7fcdf1f8f740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x5638c2198b90 con 0x5638c223edc0 2024-05-12T22:11:54.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.769+0000 7fcdf1f8f740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5638c21405b0 con 0x5638c22346f0 2024-05-12T22:11:54.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.769+0000 7fcde7e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5638c22346f0 0x5638c223e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:54.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.769+0000 7fcde7e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5638c22346f0 0x5638c223e880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:59390/0 (socket says 172.21.15.28:59390) 2024-05-12T22:11:54.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.769+0000 7fcde7e00640 1 -- 172.21.15.28:0/4193068757 learned_addr learned my addr 172.21.15.28:0/4193068757 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:11:54.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.769+0000 7fcddfe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5638c2233dd0 0x5638c22341b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:54.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.769+0000 7fcddfe00640 1 --2- 172.21.15.28:0/4193068757 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5638c223edc0 0x5638c22411b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:54.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.769+0000 7fcde7e00640 1 -- 172.21.15.28:0/4193068757 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5638c223edc0 msgr2=0x5638c22411b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:11:54.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.769+0000 7fcde7e00640 1 --2- 172.21.15.28:0/4193068757 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5638c223edc0 0x5638c22411b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:54.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.769+0000 7fcde7e00640 1 -- 172.21.15.28:0/4193068757 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5638c2233dd0 msgr2=0x5638c22341b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:11:54.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.769+0000 7fcde7e00640 1 --2- 172.21.15.28:0/4193068757 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5638c2233dd0 0x5638c22341b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:54.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.769+0000 7fcde7e00640 1 -- 172.21.15.28:0/4193068757 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5638c1f767d0 con 0x5638c22346f0 2024-05-12T22:11:54.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.770+0000 7fcddfe00640 1 --2- 172.21.15.28:0/4193068757 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5638c223edc0 0x5638c22411b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-12T22:11:54.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.770+0000 7fcde5600640 1 --2- 172.21.15.28:0/347948312 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fcdcc0a2d80 0x7fcdcc0a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:54.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.770+0000 7fcde7e00640 1 --2- 172.21.15.28:0/4193068757 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5638c22346f0 0x5638c223e880 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fcdd8009cf0 tx=0x7fcdd8011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d1198f8f85dd54f4 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:11:54.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.770+0000 7fcde5600640 1 -- 172.21.15.28:0/347948312 <== mon.2 v2:172.21.15.86:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7fcde00cfa10 con 0x7fcde810bd20 2024-05-12T22:11:54.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.770+0000 7fcddfe00640 1 --2- 172.21.15.28:0/347948312 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fcdcc0a2d80 0x7fcdcc0a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:54.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.771+0000 7fcddd600640 1 -- 172.21.15.28:0/4193068757 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdd8004230 con 0x5638c22346f0 2024-05-12T22:11:54.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.771+0000 7fcddd600640 1 -- 172.21.15.28:0/4193068757 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcdd80043d0 con 0x5638c22346f0 2024-05-12T22:11:54.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.771+0000 7fcddd600640 1 -- 172.21.15.28:0/4193068757 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdd8019960 con 0x5638c22346f0 2024-05-12T22:11:54.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.772+0000 7fcdf1f8f740 1 -- 172.21.15.28:0/4193068757 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5638c22346f0 msgr2=0x5638c223e880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:11:54.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.772+0000 7fcdf1f8f740 1 --2- 172.21.15.28:0/4193068757 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5638c22346f0 0x5638c223e880 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fcdd8009cf0 tx=0x7fcdd8011810 comp rx=0 tx=0).stop 2024-05-12T22:11:54.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.772+0000 7fcddfe00640 1 --2- 172.21.15.28:0/347948312 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fcdcc0a2d80 0x7fcdcc0a5240 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x5638c2231280 tx=0x7fcdd00046c0 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:11:54.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.772+0000 7fcdf1f8f740 1 -- 172.21.15.28:0/4193068757 shutdown_connections 2024-05-12T22:11:54.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.772+0000 7fcdf1f8f740 1 --2- 172.21.15.28:0/4193068757 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5638c223edc0 0x5638c22411b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:54.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.772+0000 7fcdf1f8f740 1 --2- 172.21.15.28:0/4193068757 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5638c22346f0 0x5638c223e880 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:54.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.772+0000 7fcdf1f8f740 1 --2- 172.21.15.28:0/4193068757 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5638c2233dd0 0x5638c22341b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:54.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.772+0000 7fcdf1f8f740 1 -- 172.21.15.28:0/4193068757 >> 172.21.15.28:0/4193068757 conn(0x5638c22300f0 msgr2=0x5638c22304d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:11:54.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.772+0000 7fcdf1f8f740 1 -- 172.21.15.28:0/4193068757 shutdown_connections 2024-05-12T22:11:54.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.772+0000 7fcdf1f8f740 1 -- 172.21.15.28:0/4193068757 wait complete. 2024-05-12T22:11:54.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.772+0000 7fcdf1f8f740 1 -- start start 2024-05-12T22:11:54.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.773+0000 7fcdf1f8f740 1 Processor -- start 2024-05-12T22:11:54.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.773+0000 7fcdf1f8f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5638c2233dd0 0x5638c223e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:54.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.773+0000 7fcdf1f8f740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5638c22346f0 0x5638c223b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:54.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.773+0000 7fcdf1f8f740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5638c223edc0 0x5638c223bf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:54.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.773+0000 7fcdf1f8f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5638c21539f0 con 0x5638c2233dd0 2024-05-12T22:11:54.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.773+0000 7fcdf1f8f740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x5638c2194b50 con 0x5638c223edc0 2024-05-12T22:11:54.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.773+0000 7fcdf1f8f740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5638c2241bb0 con 0x5638c22346f0 2024-05-12T22:11:54.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.773+0000 7fcdf1f8f740 10 client.0 _ll_register_callbacks cb 0x5638c1eabbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:11:54.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.774+0000 7fcddfe00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5638c22346f0 0x5638c223b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:54.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.774+0000 7fcddfe00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5638c22346f0 0x5638c223b9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:59392/0 (socket says 172.21.15.28:59392) 2024-05-12T22:11:54.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.774+0000 7fcddfe00640 1 -- 172.21.15.28:0/207213273 learned_addr learned my addr 172.21.15.28:0/207213273 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:11:54.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.774+0000 7fcde7e00640 1 --2- 172.21.15.28:0/207213273 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5638c2233dd0 0x5638c223e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:54.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.774+0000 7fcded000640 1 --2- 172.21.15.28:0/207213273 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5638c223edc0 0x5638c223bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:54.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.774+0000 7fcddfe00640 1 -- 172.21.15.28:0/207213273 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5638c223edc0 msgr2=0x5638c223bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:11:54.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.774+0000 7fcddfe00640 1 --2- 172.21.15.28:0/207213273 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5638c223edc0 0x5638c223bf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:54.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.774+0000 7fcddfe00640 1 -- 172.21.15.28:0/207213273 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5638c2233dd0 msgr2=0x5638c223e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:11:54.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.774+0000 7fcddfe00640 1 --2- 172.21.15.28:0/207213273 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5638c2233dd0 0x5638c223e6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:54.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.774+0000 7fcddfe00640 1 -- 172.21.15.28:0/207213273 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5638c1eac3f0 con 0x5638c22346f0 2024-05-12T22:11:54.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.774+0000 7fcde7e00640 1 --2- 172.21.15.28:0/207213273 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5638c2233dd0 0x5638c223e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:11:54.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.775+0000 7fcddfe00640 1 --2- 172.21.15.28:0/207213273 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5638c22346f0 0x5638c223b9e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fcdd0006010 tx=0x7fcdd0009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=424e01f2958cf235 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:11:54.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.775+0000 7fcdd6000640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-12T22:11:54.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.775+0000 7fcdd6000640 1 -- 172.21.15.28:0/207213273 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdd0035040 con 0x5638c22346f0 2024-05-12T22:11:54.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.775+0000 7fcdd6000640 1 -- 172.21.15.28:0/207213273 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7fcdd00037f0 con 0x5638c22346f0 2024-05-12T22:11:54.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.775+0000 7fcdf1f8f740 1 -- 172.21.15.28:0/207213273 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcdd80043d0 con 0x5638c22346f0 2024-05-12T22:11:54.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.776+0000 7fcdd6000640 1 -- 172.21.15.28:0/207213273 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fcdd00039f0 con 0x5638c22346f0 2024-05-12T22:11:54.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.776+0000 7fcdf1f8f740 10 client.25969 fetch_fsmap learned FSMap version 25 2024-05-12T22:11:54.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.776+0000 7fcdf1f8f740 1 -- 172.21.15.28:0/207213273 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5638c1e6c0f0 con 0x5638c22346f0 2024-05-12T22:11:54.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.776+0000 7fcdd6000640 1 -- 172.21.15.28:0/207213273 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcdd80043d0 con 0x5638c22346f0 2024-05-12T22:11:54.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.777+0000 7fcdd6000640 1 -- 172.21.15.28:0/207213273 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7fcdd004ab90 con 0x5638c22346f0 2024-05-12T22:11:54.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.777+0000 7fcdf1f8f740 10 client.25969 fetch_fsmap finished waiting for FSMap version 25 2024-05-12T22:11:54.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.777+0000 7fcdf1f8f740 10 client.25969 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:11:54.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.777+0000 7fcdf1f8f740 20 client.25969 populate_metadata read hostname 'smithi028' 2024-05-12T22:11:54.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.777+0000 7fcdf1f8f740 1 --2- 172.21.15.28:0/207213273 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5638c224e7f0 0x5638c2250c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:54.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.777+0000 7fcdf1f8f740 4 client.25969 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-12T22:11:54.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.777+0000 7fcdf1f8f740 1 -- 172.21.15.28:0/207213273 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5638c21a62c0 con 0x5638c224e7f0 2024-05-12T22:11:54.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.777+0000 7fcde7e00640 1 --2- 172.21.15.28:0/207213273 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5638c224e7f0 0x5638c2250c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:54.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.778+0000 7fcde7e00640 1 --2- 172.21.15.28:0/207213273 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5638c224e7f0 0x5638c2250c80 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=69806b02b5e2eaaf server_cookie=24c2f2fde2b29ec8 in_seq=0 out_seq=0 2024-05-12T22:11:54.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.778+0000 7fcdd6000640 10 client.25969 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:11:54.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.779+0000 7fcdd6000640 1 -- 172.21.15.28:0/207213273 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5638c21a62c0 con 0x5638c224e7f0 2024-05-12T22:11:54.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.779+0000 7fcdd6000640 10 client.25969 handle_command_reply: tid=0 2024-05-12T22:11:54.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.779+0000 7fcdf1f8f740 1 client.25969 shutdown 2024-05-12T22:11:54.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.779+0000 7fcdf1f8f740 1 -- 172.21.15.28:0/207213273 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5638c22346f0 msgr2=0x5638c223b9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:11:54.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.779+0000 7fcdf1f8f740 1 --2- 172.21.15.28:0/207213273 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5638c22346f0 0x5638c223b9e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fcdd0006010 tx=0x7fcdd0009260 comp rx=0 tx=0).stop 2024-05-12T22:11:54.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.779+0000 7fcdf1f8f740 1 -- 172.21.15.28:0/207213273 shutdown_connections 2024-05-12T22:11:54.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.779+0000 7fcdf1f8f740 1 --2- 172.21.15.28:0/207213273 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5638c224e7f0 0x5638c2250c80 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:54.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.779+0000 7fcdf1f8f740 1 --2- 172.21.15.28:0/207213273 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5638c223edc0 0x5638c223bf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:54.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.779+0000 7fcdf1f8f740 1 --2- 172.21.15.28:0/207213273 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5638c22346f0 0x5638c223b9e0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:54.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.779+0000 7fcdf1f8f740 1 --2- 172.21.15.28:0/207213273 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5638c2233dd0 0x5638c223e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:54.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.779+0000 7fcdd6000640 0 client.25969 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:11:54.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.779+0000 7fcdf1f8f740 1 -- 172.21.15.28:0/207213273 >> 172.21.15.28:0/207213273 conn(0x5638c22300f0 msgr2=0x5638c2235ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:11:54.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.780+0000 7fcdf1f8f740 1 -- 172.21.15.28:0/207213273 shutdown_connections 2024-05-12T22:11:54.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.780+0000 7fcdf1f8f740 1 -- 172.21.15.28:0/207213273 wait complete. 2024-05-12T22:11:54.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.780+0000 7fcdf1f8f740 20 client.25969 trim_cache size 0 max 16384 2024-05-12T22:11:54.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.804+0000 7fcdf1f8f740 1 Processor -- start 2024-05-12T22:11:54.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.804+0000 7fcdf1f8f740 1 -- start start 2024-05-12T22:11:54.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.804+0000 7fcdf1f8f740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5638c2233dd0 0x5638c22370b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:54.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.804+0000 7fcdf1f8f740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5638c22346f0 0x5638c223a890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:54.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.804+0000 7fcdf1f8f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5638c223edc0 0x5638c224a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:54.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.804+0000 7fcdf1f8f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcde81b1e20 con 0x5638c223edc0 2024-05-12T22:11:54.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.804+0000 7fcdf1f8f740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fcde81af6f0 con 0x5638c22346f0 2024-05-12T22:11:54.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.804+0000 7fcdf1f8f740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fcde81ad540 con 0x5638c2233dd0 2024-05-12T22:11:54.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.805+0000 7fcde7e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5638c2233dd0 0x5638c22370b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:54.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.805+0000 7fcddfe00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5638c22346f0 0x5638c223a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:54.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.805+0000 7fcde7e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5638c2233dd0 0x5638c22370b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:59402/0 (socket says 172.21.15.28:59402) 2024-05-12T22:11:54.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.805+0000 7fcddfe00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5638c22346f0 0x5638c223a890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:37130/0 (socket says 172.21.15.28:37130) 2024-05-12T22:11:54.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.805+0000 7fcde7e00640 1 -- 172.21.15.28:0/804491288 learned_addr learned my addr 172.21.15.28:0/804491288 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:11:54.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.805+0000 7fcded000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5638c223edc0 0x5638c224a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:54.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.805+0000 7fcde7e00640 1 -- 172.21.15.28:0/804491288 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5638c22346f0 msgr2=0x5638c223a890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:11:54.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.805+0000 7fcde7e00640 1 --2- 172.21.15.28:0/804491288 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5638c22346f0 0x5638c223a890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:54.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.805+0000 7fcde7e00640 1 -- 172.21.15.28:0/804491288 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5638c223edc0 msgr2=0x5638c224a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:11:54.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.805+0000 7fcde7e00640 1 --2- 172.21.15.28:0/804491288 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5638c223edc0 0x5638c224a8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:54.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.805+0000 7fcde7e00640 1 -- 172.21.15.28:0/804491288 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5638c222ff20 con 0x5638c2233dd0 2024-05-12T22:11:54.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.805+0000 7fcddfe00640 1 --2- 172.21.15.28:0/804491288 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5638c22346f0 0x5638c223a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:11:54.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.805+0000 7fcde7e00640 1 --2- 172.21.15.28:0/804491288 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5638c2233dd0 0x5638c22370b0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fcdd8009250 tx=0x7fcdd8011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3cad7e56d1cdcbb0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:11:54.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.806+0000 7fcddcc00640 1 -- 172.21.15.28:0/804491288 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdd8031070 con 0x5638c2233dd0 2024-05-12T22:11:54.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.806+0000 7fcddcc00640 1 -- 172.21.15.28:0/804491288 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcdd8009910 con 0x5638c2233dd0 2024-05-12T22:11:54.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.806+0000 7fcddcc00640 1 -- 172.21.15.28:0/804491288 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdd80052a0 con 0x5638c2233dd0 2024-05-12T22:11:54.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.807+0000 7fcdf1f8f740 1 -- 172.21.15.28:0/804491288 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5638c2233dd0 msgr2=0x5638c22370b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:11:54.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.807+0000 7fcdf1f8f740 1 --2- 172.21.15.28:0/804491288 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5638c2233dd0 0x5638c22370b0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fcdd8009250 tx=0x7fcdd8011810 comp rx=0 tx=0).stop 2024-05-12T22:11:54.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.807+0000 7fcdf1f8f740 1 -- 172.21.15.28:0/804491288 shutdown_connections 2024-05-12T22:11:54.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.807+0000 7fcdf1f8f740 1 --2- 172.21.15.28:0/804491288 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5638c223edc0 0x5638c224a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:54.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.807+0000 7fcdf1f8f740 1 --2- 172.21.15.28:0/804491288 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5638c22346f0 0x5638c223a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:54.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.807+0000 7fcdf1f8f740 1 --2- 172.21.15.28:0/804491288 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5638c2233dd0 0x5638c22370b0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:54.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.807+0000 7fcdf1f8f740 1 -- 172.21.15.28:0/804491288 >> 172.21.15.28:0/804491288 conn(0x5638c22300f0 msgr2=0x5638c223a180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:11:54.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.807+0000 7fcdf1f8f740 1 -- 172.21.15.28:0/804491288 shutdown_connections 2024-05-12T22:11:54.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.807+0000 7fcdf1f8f740 1 -- 172.21.15.28:0/804491288 wait complete. 2024-05-12T22:11:54.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.807+0000 7fcdf1f8f740 1 -- start start 2024-05-12T22:11:54.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.808+0000 7fcdf1f8f740 1 Processor -- start 2024-05-12T22:11:54.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.808+0000 7fcdf1f8f740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5638c2233dd0 0x5638c22420d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:54.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.808+0000 7fcdf1f8f740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5638c22346f0 0x5638c2242610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:54.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.808+0000 7fcdf1f8f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5638c223edc0 0x5638c22acdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:54.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.808+0000 7fcdf1f8f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcde81adbb0 con 0x5638c223edc0 2024-05-12T22:11:54.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.808+0000 7fcdf1f8f740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fcde81af020 con 0x5638c2233dd0 2024-05-12T22:11:54.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.808+0000 7fcdf1f8f740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fcde81ae280 con 0x5638c22346f0 2024-05-12T22:11:54.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.808+0000 7fcdf1f8f740 10 client.0 _ll_register_callbacks cb 0x7fcde00093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:11:54.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.809+0000 7fcde7e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5638c2233dd0 0x5638c22420d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:54.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.809+0000 7fcde7e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5638c2233dd0 0x5638c22420d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:37140/0 (socket says 172.21.15.28:37140) 2024-05-12T22:11:54.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.809+0000 7fcde7e00640 1 -- 172.21.15.28:0/2874847821 learned_addr learned my addr 172.21.15.28:0/2874847821 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:11:54.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.809+0000 7fcddfe00640 1 --2- 172.21.15.28:0/2874847821 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5638c22346f0 0x5638c2242610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:54.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.809+0000 7fcded000640 1 --2- 172.21.15.28:0/2874847821 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5638c223edc0 0x5638c22acdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:54.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.809+0000 7fcde7e00640 1 -- 172.21.15.28:0/2874847821 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5638c22346f0 msgr2=0x5638c2242610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:11:54.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.809+0000 7fcde7e00640 1 --2- 172.21.15.28:0/2874847821 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5638c22346f0 0x5638c2242610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:54.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.809+0000 7fcde7e00640 1 -- 172.21.15.28:0/2874847821 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5638c223edc0 msgr2=0x5638c22acdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:11:54.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.809+0000 7fcde7e00640 1 --2- 172.21.15.28:0/2874847821 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5638c223edc0 0x5638c22acdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:54.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.809+0000 7fcde7e00640 1 -- 172.21.15.28:0/2874847821 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5638c22ad430 con 0x5638c2233dd0 2024-05-12T22:11:54.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.810+0000 7fcde7e00640 1 --2- 172.21.15.28:0/2874847821 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5638c2233dd0 0x5638c22420d0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fcdd8005890 tx=0x7fcdd80044c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=87ad83b5028fc67 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:11:54.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.810+0000 7fcdd6000640 10 client.0 ms_handle_connect on v2:172.21.15.86:3300/0 2024-05-12T22:11:54.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.810+0000 7fcdd6000640 1 -- 172.21.15.28:0/2874847821 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdd8039070 con 0x5638c2233dd0 2024-05-12T22:11:54.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.810+0000 7fcdd6000640 1 -- 172.21.15.28:0/2874847821 <== mon.2 v2:172.21.15.86:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7fcdd802b360 con 0x5638c2233dd0 2024-05-12T22:11:54.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.810+0000 7fcdf1f8f740 1 -- 172.21.15.28:0/2874847821 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcdd8009910 con 0x5638c2233dd0 2024-05-12T22:11:54.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.810+0000 7fcdd6000640 1 -- 172.21.15.28:0/2874847821 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fcdd8031040 con 0x5638c2233dd0 2024-05-12T22:11:54.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.810+0000 7fcdf1f8f740 10 client.26018 fetch_fsmap learned FSMap version 25 2024-05-12T22:11:54.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.810+0000 7fcdf1f8f740 1 -- 172.21.15.28:0/2874847821 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5638c22b47f0 con 0x5638c2233dd0 2024-05-12T22:11:54.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.811+0000 7fcdd6000640 1 -- 172.21.15.28:0/2874847821 <== mon.2 v2:172.21.15.86:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcdd8009910 con 0x5638c2233dd0 2024-05-12T22:11:54.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.811+0000 7fcdd6000640 1 -- 172.21.15.28:0/2874847821 <== mon.2 v2:172.21.15.86:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7fcdd8034050 con 0x5638c2233dd0 2024-05-12T22:11:54.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.811+0000 7fcdf1f8f740 10 client.26018 fetch_fsmap finished waiting for FSMap version 25 2024-05-12T22:11:54.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.811+0000 7fcdf1f8f740 10 client.26018 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:11:54.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.811+0000 7fcdf1f8f740 20 client.26018 populate_metadata read hostname 'smithi028' 2024-05-12T22:11:54.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.811+0000 7fcdf1f8f740 1 --2- 172.21.15.28:0/2874847821 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5638c22b4dd0 0x5638c22b71f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:11:54.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.811+0000 7fcdf1f8f740 4 client.26018 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-12T22:11:54.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.811+0000 7fcdf1f8f740 1 -- 172.21.15.28:0/2874847821 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5638c21a0610 con 0x5638c22b4dd0 2024-05-12T22:11:54.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.812+0000 7fcddfe00640 1 --2- 172.21.15.28:0/2874847821 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5638c22b4dd0 0x5638c22b71f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:11:54.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.812+0000 7fcddfe00640 1 --2- 172.21.15.28:0/2874847821 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5638c22b4dd0 0x5638c22b71f0 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=57e6485e3f9aa794 server_cookie=64376e52557b88be in_seq=0 out_seq=0 2024-05-12T22:11:54.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.812+0000 7fcdd6000640 10 client.26018 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:11:54.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.813+0000 7fcdd6000640 1 -- 172.21.15.28:0/2874847821 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+268 (crc 0 0 0) 0x5638c21a0610 con 0x5638c22b4dd0 2024-05-12T22:11:54.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.813+0000 7fcdd6000640 10 client.26018 handle_command_reply: tid=0 2024-05-12T22:11:54.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.813+0000 7fcdf1f8f740 1 client.26018 shutdown 2024-05-12T22:11:54.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.813+0000 7fcdf1f8f740 1 -- 172.21.15.28:0/2874847821 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5638c2233dd0 msgr2=0x5638c22420d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:11:54.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.813+0000 7fcdf1f8f740 1 --2- 172.21.15.28:0/2874847821 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5638c2233dd0 0x5638c22420d0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fcdd8005890 tx=0x7fcdd80044c0 comp rx=0 tx=0).stop 2024-05-12T22:11:54.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.813+0000 7fcdf1f8f740 1 -- 172.21.15.28:0/2874847821 shutdown_connections 2024-05-12T22:11:54.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.813+0000 7fcdf1f8f740 1 --2- 172.21.15.28:0/2874847821 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5638c22b4dd0 0x5638c22b71f0 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:54.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.813+0000 7fcdf1f8f740 1 --2- 172.21.15.28:0/2874847821 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5638c223edc0 0x5638c22acdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:54.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.813+0000 7fcdf1f8f740 1 --2- 172.21.15.28:0/2874847821 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5638c22346f0 0x5638c2242610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:54.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.813+0000 7fcdf1f8f740 1 --2- 172.21.15.28:0/2874847821 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5638c2233dd0 0x5638c22420d0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:54.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.813+0000 7fcdd6000640 0 client.26018 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:11:54.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.813+0000 7fcdf1f8f740 1 -- 172.21.15.28:0/2874847821 >> 172.21.15.28:0/2874847821 conn(0x5638c22300f0 msgr2=0x5638c2236e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:11:54.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.813+0000 7fcdf1f8f740 1 -- 172.21.15.28:0/2874847821 shutdown_connections 2024-05-12T22:11:54.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.813+0000 7fcdf1f8f740 1 -- 172.21.15.28:0/2874847821 wait complete. 2024-05-12T22:11:54.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.814+0000 7fcdf1f8f740 20 client.26018 trim_cache size 0 max 16384 2024-05-12T22:11:54.820 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-05-12T22:11:54.821 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "scrub active (1284 inodes in the stack)", 2024-05-12T22:11:54.821 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-05-12T22:11:54.821 INFO:teuthology.orchestra.run.smithi028.stdout: "cf340000-4a8a-4b66-b60e-039395406168": { 2024-05-12T22:11:54.821 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-05-12T22:11:54.821 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "cf340000-4a8a-4b66-b60e-039395406168", 2024-05-12T22:11:54.821 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-05-12T22:11:54.821 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:11:54.821 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:11:54.821 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-05-12T22:11:54.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.814+0000 7fcdeda00640 1 -- 172.21.15.28:0/347948312 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fcdcc0a2d80 msgr2=0x7fcdcc0a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:11:54.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.814+0000 7fcdeda00640 1 --2- 172.21.15.28:0/347948312 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fcdcc0a2d80 0x7fcdcc0a5240 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x5638c2231280 tx=0x7fcdd00046c0 comp rx=0 tx=0).stop 2024-05-12T22:11:54.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.814+0000 7fcdeda00640 1 -- 172.21.15.28:0/347948312 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcde810bd20 msgr2=0x7fcde819ead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:11:54.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.814+0000 7fcdeda00640 1 --2- 172.21.15.28:0/347948312 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcde810bd20 0x7fcde819ead0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fcde0005fd0 tx=0x7fcde000ef10 comp rx=0 tx=0).stop 2024-05-12T22:11:54.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.814+0000 7fcdeda00640 1 -- 172.21.15.28:0/347948312 shutdown_connections 2024-05-12T22:11:54.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.814+0000 7fcdeda00640 1 --2- 172.21.15.28:0/347948312 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fcdcc0a2d80 0x7fcdcc0a5240 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:54.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.814+0000 7fcdeda00640 1 --2- 172.21.15.28:0/347948312 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcde810bd20 0x7fcde819ead0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:54.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.815+0000 7fcdeda00640 1 --2- 172.21.15.28:0/347948312 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcde8103270 0x7fcde819a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:54.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.815+0000 7fcdeda00640 1 --2- 172.21.15.28:0/347948312 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcde8102950 0x7fcde819a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:11:54.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.815+0000 7fcdeda00640 1 -- 172.21.15.28:0/347948312 >> 172.21.15.28:0/347948312 conn(0x7fcde80fc0b0 msgr2=0x7fcde81004d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:11:54.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.815+0000 7fcdeda00640 1 -- 172.21.15.28:0/347948312 shutdown_connections 2024-05-12T22:11:54.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:11:54.815+0000 7fcdeda00640 1 -- 172.21.15.28:0/347948312 wait complete. 2024-05-12T22:11:54.837 INFO:tasks.cephfs.filesystem:scrub status for tag:cf340000-4a8a-4b66-b60e-039395406168 - {'path': '/', 'tag': 'cf340000-4a8a-4b66-b60e-039395406168', 'options': 'recursive,force'} 2024-05-12T22:11:55.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:11:55 smithi028 ceph-mon[29847]: scrub queued for path: / 2024-05-12T22:11:55.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:11:55 smithi028 ceph-mon[29847]: scrub summary: idle+waiting paths [/] 2024-05-12T22:11:55.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:11:55 smithi028 ceph-mon[29847]: scrub summary: active paths [/] 2024-05-12T22:11:55.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:11:55 smithi028 ceph-mon[29847]: scrub summary: active paths [/] 2024-05-12T22:11:55.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:11:55 smithi028 ceph-mon[29847]: scrub summary: idle+waiting paths [/] 2024-05-12T22:11:55.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:11:55 smithi028 ceph-mon[29847]: scrub summary: active paths [/] 2024-05-12T22:11:55.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:11:55 smithi028 ceph-mon[29847]: scrub summary: idle+waiting paths [/] 2024-05-12T22:11:55.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:11:55 smithi186 ceph-mon[34785]: scrub queued for path: / 2024-05-12T22:11:55.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:11:55 smithi186 ceph-mon[34785]: scrub summary: idle+waiting paths [/] 2024-05-12T22:11:55.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:11:55 smithi186 ceph-mon[34785]: scrub summary: active paths [/] 2024-05-12T22:11:55.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:11:55 smithi186 ceph-mon[34785]: scrub summary: active paths [/] 2024-05-12T22:11:55.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:11:55 smithi186 ceph-mon[34785]: scrub summary: idle+waiting paths [/] 2024-05-12T22:11:55.431 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:11:55 smithi186 ceph-mon[34785]: scrub summary: active paths [/] 2024-05-12T22:11:55.431 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:11:55 smithi186 ceph-mon[34785]: scrub summary: idle+waiting paths [/] 2024-05-12T22:11:55.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:11:55 smithi086 ceph-mon[34297]: scrub queued for path: / 2024-05-12T22:11:55.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:11:55 smithi086 ceph-mon[34297]: scrub summary: idle+waiting paths [/] 2024-05-12T22:11:55.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:11:55 smithi086 ceph-mon[34297]: scrub summary: active paths [/] 2024-05-12T22:11:55.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:11:55 smithi086 ceph-mon[34297]: scrub summary: active paths [/] 2024-05-12T22:11:55.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:11:55 smithi086 ceph-mon[34297]: scrub summary: idle+waiting paths [/] 2024-05-12T22:11:55.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:11:55 smithi086 ceph-mon[34297]: scrub summary: active paths [/] 2024-05-12T22:11:55.494 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:11:55 smithi086 ceph-mon[34297]: scrub summary: idle+waiting paths [/] 2024-05-12T22:11:56.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:11:56 smithi186 ceph-mon[34785]: pgmap v91: 97 pgs: 97 active+clean; 95 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 11 op/s 2024-05-12T22:11:56.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:11:56 smithi186 ceph-mon[34785]: scrub summary: active paths [/] 2024-05-12T22:11:56.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:11:56 smithi186 ceph-mon[34785]: scrub summary: idle+waiting paths [/] 2024-05-12T22:11:56.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:11:56 smithi186 ceph-mon[34785]: scrub summary: idle+waiting paths [/] 2024-05-12T22:11:56.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:11:56 smithi086 ceph-mon[34297]: pgmap v91: 97 pgs: 97 active+clean; 95 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 11 op/s 2024-05-12T22:11:56.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:11:56 smithi086 ceph-mon[34297]: scrub summary: active paths [/] 2024-05-12T22:11:56.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:11:56 smithi086 ceph-mon[34297]: scrub summary: idle+waiting paths [/] 2024-05-12T22:11:56.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:11:56 smithi086 ceph-mon[34297]: scrub summary: idle+waiting paths [/] 2024-05-12T22:11:56.537 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:11:56 smithi028 ceph-mon[29847]: pgmap v91: 97 pgs: 97 active+clean; 95 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 11 op/s 2024-05-12T22:11:56.537 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:11:56 smithi028 ceph-mon[29847]: scrub summary: active paths [/] 2024-05-12T22:11:56.538 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:11:56 smithi028 ceph-mon[29847]: scrub summary: idle+waiting paths [/] 2024-05-12T22:11:56.538 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:11:56 smithi028 ceph-mon[29847]: scrub summary: idle+waiting paths [/] 2024-05-12T22:11:58.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:11:58 smithi186 ceph-mon[34785]: pgmap v92: 97 pgs: 97 active+clean; 95 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 6 op/s 2024-05-12T22:11:58.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:11:58 smithi086 ceph-mon[34297]: pgmap v92: 97 pgs: 97 active+clean; 95 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 6 op/s 2024-05-12T22:11:58.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:11:58 smithi028 ceph-mon[29847]: pgmap v92: 97 pgs: 97 active+clean; 95 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 6 op/s 2024-05-12T22:12:00.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:12:00 smithi186 ceph-mon[34785]: pgmap v93: 97 pgs: 97 active+clean; 95 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 6 op/s 2024-05-12T22:12:00.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:12:00 smithi086 ceph-mon[34297]: pgmap v93: 97 pgs: 97 active+clean; 95 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 6 op/s 2024-05-12T22:12:00.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:12:00 smithi028 ceph-mon[29847]: pgmap v93: 97 pgs: 97 active+clean; 95 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 6 op/s 2024-05-12T22:12:02.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:12:02 smithi186 ceph-mon[34785]: pgmap v94: 97 pgs: 97 active+clean; 97 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 266 KiB/s wr, 32 op/s 2024-05-12T22:12:02.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:12:02 smithi086 ceph-mon[34297]: pgmap v94: 97 pgs: 97 active+clean; 97 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 266 KiB/s wr, 32 op/s 2024-05-12T22:12:02.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:12:02 smithi028 ceph-mon[29847]: pgmap v94: 97 pgs: 97 active+clean; 97 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 266 KiB/s wr, 32 op/s 2024-05-12T22:12:03.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:12:03 smithi186 ceph-mon[34785]: scrub summary: idle 2024-05-12T22:12:03.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:12:03 smithi186 ceph-mon[34785]: scrub summary: idle 2024-05-12T22:12:03.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:12:03 smithi086 ceph-mon[34297]: scrub summary: idle 2024-05-12T22:12:03.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:12:03 smithi086 ceph-mon[34297]: scrub summary: idle 2024-05-12T22:12:03.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:12:03 smithi028 ceph-mon[29847]: scrub summary: idle 2024-05-12T22:12:03.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:12:03 smithi028 ceph-mon[29847]: scrub summary: idle 2024-05-12T22:12:04.330 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:12:04 smithi186 ceph-mon[34785]: pgmap v95: 97 pgs: 97 active+clean; 97 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 256 KiB/s wr, 28 op/s 2024-05-12T22:12:04.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:12:04 smithi086 ceph-mon[34297]: pgmap v95: 97 pgs: 97 active+clean; 97 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 256 KiB/s wr, 28 op/s 2024-05-12T22:12:04.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:12:04 smithi028 ceph-mon[29847]: pgmap v95: 97 pgs: 97 active+clean; 97 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 256 KiB/s wr, 28 op/s 2024-05-12T22:12:06.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:12:06 smithi186 ceph-mon[34785]: pgmap v96: 97 pgs: 97 active+clean; 97 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 281 KiB/s wr, 46 op/s 2024-05-12T22:12:06.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:12:06 smithi086 ceph-mon[34297]: pgmap v96: 97 pgs: 97 active+clean; 97 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 281 KiB/s wr, 46 op/s 2024-05-12T22:12:06.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:12:06 smithi028 ceph-mon[29847]: pgmap v96: 97 pgs: 97 active+clean; 97 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 281 KiB/s wr, 46 op/s 2024-05-12T22:12:08.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:12:08 smithi186 ceph-mon[34785]: pgmap v97: 97 pgs: 97 active+clean; 97 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 278 KiB/s wr, 43 op/s 2024-05-12T22:12:08.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:12:08 smithi086 ceph-mon[34297]: pgmap v97: 97 pgs: 97 active+clean; 97 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 278 KiB/s wr, 43 op/s 2024-05-12T22:12:08.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:12:08 smithi028 ceph-mon[29847]: pgmap v97: 97 pgs: 97 active+clean; 97 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 278 KiB/s wr, 43 op/s 2024-05-12T22:12:10.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:12:10 smithi186 ceph-mon[34785]: pgmap v98: 97 pgs: 97 active+clean; 97 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 278 KiB/s wr, 43 op/s 2024-05-12T22:12:10.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:12:10 smithi086 ceph-mon[34297]: pgmap v98: 97 pgs: 97 active+clean; 97 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 278 KiB/s wr, 43 op/s 2024-05-12T22:12:10.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:12:10 smithi028 ceph-mon[29847]: pgmap v98: 97 pgs: 97 active+clean; 97 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 278 KiB/s wr, 43 op/s 2024-05-12T22:12:12.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:12:12 smithi186 ceph-mon[34785]: pgmap v99: 97 pgs: 97 active+clean; 97 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 278 KiB/s wr, 46 op/s 2024-05-12T22:12:12.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:12:12 smithi186 ceph-mon[34785]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:12:12.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:12:12 smithi086 ceph-mon[34297]: pgmap v99: 97 pgs: 97 active+clean; 97 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 278 KiB/s wr, 46 op/s 2024-05-12T22:12:12.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:12:12 smithi086 ceph-mon[34297]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:12:12.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:12:12 smithi028 ceph-mon[29847]: pgmap v99: 97 pgs: 97 active+clean; 97 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 278 KiB/s wr, 46 op/s 2024-05-12T22:12:12.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:12:12 smithi028 ceph-mon[29847]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:12:13.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:12:13 smithi186 ceph-mon[34785]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:12:13.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:12:13 smithi186 ceph-mon[34785]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:12:13.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:12:13 smithi186 ceph-mon[34785]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:12:13.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:12:13 smithi086 ceph-mon[34297]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:12:13.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:12:13 smithi086 ceph-mon[34297]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:12:13.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:12:13 smithi086 ceph-mon[34297]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:12:13.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:12:13 smithi028 ceph-mon[29847]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:12:13.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:12:13 smithi028 ceph-mon[29847]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:12:13.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:12:13 smithi028 ceph-mon[29847]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:12:14.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:12:14 smithi186 ceph-mon[34785]: pgmap v100: 97 pgs: 97 active+clean; 97 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 25 KiB/s wr, 20 op/s 2024-05-12T22:12:14.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:12:14 smithi086 ceph-mon[34297]: pgmap v100: 97 pgs: 97 active+clean; 97 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 25 KiB/s wr, 20 op/s 2024-05-12T22:12:14.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:12:14 smithi028 ceph-mon[29847]: pgmap v100: 97 pgs: 97 active+clean; 97 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 25 KiB/s wr, 20 op/s 2024-05-12T22:12:16.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:12:16 smithi186 ceph-mon[34785]: pgmap v101: 97 pgs: 97 active+clean; 97 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 27 KiB/s wr, 23 op/s 2024-05-12T22:12:16.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:12:16 smithi086 ceph-mon[34297]: pgmap v101: 97 pgs: 97 active+clean; 97 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 27 KiB/s wr, 23 op/s 2024-05-12T22:12:16.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:12:16 smithi028 ceph-mon[29847]: pgmap v101: 97 pgs: 97 active+clean; 97 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 27 KiB/s wr, 23 op/s 2024-05-12T22:12:18.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:12:18 smithi186 ceph-mon[34785]: pgmap v102: 97 pgs: 97 active+clean; 97 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 2.3 KiB/s wr, 5 op/s 2024-05-12T22:12:18.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:12:18 smithi086 ceph-mon[34297]: pgmap v102: 97 pgs: 97 active+clean; 97 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 2.3 KiB/s wr, 5 op/s 2024-05-12T22:12:18.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:12:18 smithi028 ceph-mon[29847]: pgmap v102: 97 pgs: 97 active+clean; 97 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 2.3 KiB/s wr, 5 op/s 2024-05-12T22:12:20.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:12:20 smithi186 ceph-mon[34785]: pgmap v103: 97 pgs: 97 active+clean; 97 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 2.3 KiB/s wr, 5 op/s 2024-05-12T22:12:20.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:12:20 smithi086 ceph-mon[34297]: pgmap v103: 97 pgs: 97 active+clean; 97 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 2.3 KiB/s wr, 5 op/s 2024-05-12T22:12:20.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:12:20 smithi028 ceph-mon[29847]: pgmap v103: 97 pgs: 97 active+clean; 97 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 2.3 KiB/s wr, 5 op/s 2024-05-12T22:12:22.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:12:22 smithi186 ceph-mon[34785]: pgmap v104: 97 pgs: 97 active+clean; 97 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 2.3 KiB/s wr, 8 op/s 2024-05-12T22:12:22.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:12:22 smithi086 ceph-mon[34297]: pgmap v104: 97 pgs: 97 active+clean; 97 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 2.3 KiB/s wr, 8 op/s 2024-05-12T22:12:22.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:12:22 smithi028 ceph-mon[29847]: pgmap v104: 97 pgs: 97 active+clean; 97 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 2.3 KiB/s wr, 8 op/s 2024-05-12T22:12:24.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:12:24 smithi186 ceph-mon[34785]: pgmap v105: 97 pgs: 97 active+clean; 97 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 2.2 KiB/s wr, 5 op/s 2024-05-12T22:12:24.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:12:24 smithi086 ceph-mon[34297]: pgmap v105: 97 pgs: 97 active+clean; 97 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 2.2 KiB/s wr, 5 op/s 2024-05-12T22:12:24.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:12:24 smithi028 ceph-mon[29847]: pgmap v105: 97 pgs: 97 active+clean; 97 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 2.2 KiB/s wr, 5 op/s 2024-05-12T22:12:24.838 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-12T22:12:24.838 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-12T22:12:25.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.066+0000 7f022ac00640 1 -- 172.21.15.28:0/765425416 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f02241126b0 msgr2=0x7f0224114aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:12:25.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.066+0000 7f022ac00640 1 --2- 172.21.15.28:0/765425416 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f02241126b0 0x7f0224114aa0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f021c009ca0 tx=0x7f021c02a8b0 comp rx=0 tx=0).stop 2024-05-12T22:12:25.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.066+0000 7f022ac00640 1 -- 172.21.15.28:0/765425416 shutdown_connections 2024-05-12T22:12:25.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.066+0000 7f022ac00640 1 --2- 172.21.15.28:0/765425416 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f02241126b0 0x7f0224114aa0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:25.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.066+0000 7f022ac00640 1 --2- 172.21.15.28:0/765425416 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0224076680 0x7f0224112170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:25.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.066+0000 7f022ac00640 1 --2- 172.21.15.28:0/765425416 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0224075d60 0x7f0224076140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:25.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.066+0000 7f022ac00640 1 -- 172.21.15.28:0/765425416 >> 172.21.15.28:0/765425416 conn(0x7f02240ff4a0 msgr2=0x7f02241018c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:12:25.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.067+0000 7f022ac00640 1 -- 172.21.15.28:0/765425416 shutdown_connections 2024-05-12T22:12:25.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.067+0000 7f022ac00640 1 -- 172.21.15.28:0/765425416 wait complete. 2024-05-12T22:12:25.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.068+0000 7f022ac00640 1 Processor -- start 2024-05-12T22:12:25.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.068+0000 7f022ac00640 1 -- start start 2024-05-12T22:12:25.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.068+0000 7f022ac00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0224075d60 0x7f022419e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:25.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.068+0000 7f022ac00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0224076680 0x7f022419ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:25.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.068+0000 7f022ac00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f02241126b0 0x7f02241a2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:25.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.068+0000 7f022ac00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f02241181f0 con 0x7f0224075d60 2024-05-12T22:12:25.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.068+0000 7f022ac00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f0224118070 con 0x7f02241126b0 2024-05-12T22:12:25.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.068+0000 7f022ac00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f0224118370 con 0x7f0224076680 2024-05-12T22:12:25.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.068+0000 7f0223e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0224075d60 0x7f022419e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:25.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.068+0000 7f0228c00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f02241126b0 0x7f02241a2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:25.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.068+0000 7f0228c00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f02241126b0 0x7f02241a2fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:32878/0 (socket says 172.21.15.28:32878) 2024-05-12T22:12:25.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.069+0000 7f0223400640 1 --2- 172.21.15.28:0/585188107 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0224076680 0x7f022419ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:25.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.069+0000 7f0228c00640 1 -- 172.21.15.28:0/585188107 learned_addr learned my addr 172.21.15.28:0/585188107 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:12:25.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.069+0000 7f0228c00640 1 -- 172.21.15.28:0/585188107 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0224076680 msgr2=0x7f022419ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:12:25.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.069+0000 7f0228c00640 1 --2- 172.21.15.28:0/585188107 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0224076680 0x7f022419ecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:25.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.069+0000 7f0228c00640 1 -- 172.21.15.28:0/585188107 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0224075d60 msgr2=0x7f022419e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:12:25.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.069+0000 7f0228c00640 1 --2- 172.21.15.28:0/585188107 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0224075d60 0x7f022419e770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:25.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.069+0000 7f0228c00640 1 -- 172.21.15.28:0/585188107 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02241a3710 con 0x7f02241126b0 2024-05-12T22:12:25.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.069+0000 7f0228c00640 1 --2- 172.21.15.28:0/585188107 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f02241126b0 0x7f02241a2fe0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f021c030e50 tx=0x7f021c030e80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:12:25.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.070+0000 7f0220c00640 1 -- 172.21.15.28:0/585188107 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f021c003e40 con 0x7f02241126b0 2024-05-12T22:12:25.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.070+0000 7f0220c00640 1 -- 172.21.15.28:0/585188107 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f021c040070 con 0x7f02241126b0 2024-05-12T22:12:25.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.070+0000 7f022ac00640 1 -- 172.21.15.28:0/585188107 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02241a39a0 con 0x7f02241126b0 2024-05-12T22:12:25.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.070+0000 7f0220c00640 1 -- 172.21.15.28:0/585188107 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f021c031aa0 con 0x7f02241126b0 2024-05-12T22:12:25.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.070+0000 7f022ac00640 1 -- 172.21.15.28:0/585188107 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02241ab290 con 0x7f02241126b0 2024-05-12T22:12:25.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.073+0000 7f022d9c7740 1 Processor -- start 2024-05-12T22:12:25.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.073+0000 7f0220c00640 1 -- 172.21.15.28:0/585188107 <== mon.2 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7f021c031c40 con 0x7f02241126b0 2024-05-12T22:12:25.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.073+0000 7f022d9c7740 1 -- start start 2024-05-12T22:12:25.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.074+0000 7f022d9c7740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f659c2bbc0 0x55f659c2bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:25.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.074+0000 7f022d9c7740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f659c2c4e0 0x55f659c36670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:25.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.074+0000 7f022d9c7740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f659c36bb0 0x55f659c38fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:25.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.074+0000 7f022d9c7740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f659bf42a0 con 0x55f659c2bbc0 2024-05-12T22:12:25.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.074+0000 7f022d9c7740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x55f659b90b90 con 0x55f659c2c4e0 2024-05-12T22:12:25.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.074+0000 7f022d9c7740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55f659b385b0 con 0x55f659c36bb0 2024-05-12T22:12:25.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.074+0000 7f0223400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f659c2bbc0 0x55f659c2bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:25.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.074+0000 7f0223400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f659c2bbc0 0x55f659c2bfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52584/0 (socket says 172.21.15.28:52584) 2024-05-12T22:12:25.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.074+0000 7f0223400640 1 -- 172.21.15.28:0/1194731204 learned_addr learned my addr 172.21.15.28:0/1194731204 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:12:25.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.074+0000 7f0223e00640 1 --2- 172.21.15.28:0/1194731204 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f659c2c4e0 0x55f659c36670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:25.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.074+0000 7f0223400640 1 --2- 172.21.15.28:0/1194731204 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f659c36bb0 0x55f659c38fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:25.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.075+0000 7f0223400640 1 -- 172.21.15.28:0/1194731204 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f659c36bb0 msgr2=0x55f659c38fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:12:25.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.075+0000 7f0223400640 1 --2- 172.21.15.28:0/1194731204 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f659c36bb0 0x55f659c38fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:25.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.075+0000 7f0223400640 1 -- 172.21.15.28:0/1194731204 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f659c2c4e0 msgr2=0x55f659c36670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:12:25.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.075+0000 7f0223400640 1 --2- 172.21.15.28:0/1194731204 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f659c2c4e0 0x55f659c36670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:25.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.075+0000 7f0223400640 1 -- 172.21.15.28:0/1194731204 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f65996e7d0 con 0x55f659c2bbc0 2024-05-12T22:12:25.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.075+0000 7f0223400640 1 --2- 172.21.15.28:0/1194731204 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f659c2bbc0 0x55f659c2bfa0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f0210013530 tx=0x7f0210005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e7b22cc3267ca11 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:12:25.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.075+0000 7f0220c00640 1 --2- 172.21.15.28:0/585188107 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f02080a2d80 0x7f02080a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:25.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.075+0000 7f0219200640 1 -- 172.21.15.28:0/1194731204 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0210004050 con 0x55f659c2bbc0 2024-05-12T22:12:25.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.075+0000 7f0219200640 1 -- 172.21.15.28:0/1194731204 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0210005500 con 0x55f659c2bbc0 2024-05-12T22:12:25.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.075+0000 7f0219200640 1 -- 172.21.15.28:0/1194731204 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0210004be0 con 0x55f659c2bbc0 2024-05-12T22:12:25.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.075+0000 7f0220c00640 1 -- 172.21.15.28:0/585188107 <== mon.2 v2:172.21.15.86:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f021c0f0400 con 0x7f02241126b0 2024-05-12T22:12:25.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.075+0000 7f0223e00640 1 --2- 172.21.15.28:0/585188107 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f02080a2d80 0x7f02080a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:25.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.076+0000 7f022d9c7740 1 -- 172.21.15.28:0/1194731204 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f659c2bbc0 msgr2=0x55f659c2bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:12:25.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.076+0000 7f022d9c7740 1 --2- 172.21.15.28:0/1194731204 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f659c2bbc0 0x55f659c2bfa0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f0210013530 tx=0x7f0210005850 comp rx=0 tx=0).stop 2024-05-12T22:12:25.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.076+0000 7f022d9c7740 1 -- 172.21.15.28:0/1194731204 shutdown_connections 2024-05-12T22:12:25.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.076+0000 7f022d9c7740 1 --2- 172.21.15.28:0/1194731204 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f659c36bb0 0x55f659c38fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:25.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.076+0000 7f022d9c7740 1 --2- 172.21.15.28:0/1194731204 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f659c2c4e0 0x55f659c36670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:25.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.076+0000 7f022d9c7740 1 --2- 172.21.15.28:0/1194731204 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f659c2bbc0 0x55f659c2bfa0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:25.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.076+0000 7f022d9c7740 1 -- 172.21.15.28:0/1194731204 >> 172.21.15.28:0/1194731204 conn(0x55f659c27ee0 msgr2=0x55f659c282c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:12:25.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.076+0000 7f0223e00640 1 --2- 172.21.15.28:0/585188107 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f02080a2d80 0x7f02080a5240 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x55f659c2d180 tx=0x7f0214005f90 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:12:25.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.076+0000 7f022d9c7740 1 -- 172.21.15.28:0/1194731204 shutdown_connections 2024-05-12T22:12:25.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.076+0000 7f022d9c7740 1 -- 172.21.15.28:0/1194731204 wait complete. 2024-05-12T22:12:25.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.077+0000 7f022d9c7740 1 -- start start 2024-05-12T22:12:25.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.077+0000 7f022d9c7740 1 Processor -- start 2024-05-12T22:12:25.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.077+0000 7f022d9c7740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f659c2bbc0 0x55f659c36530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:25.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.078+0000 7f022d9c7740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f659c2c4e0 0x55f659c33820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:25.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.078+0000 7f022d9c7740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f659c36bb0 0x55f659c33da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:25.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.078+0000 7f022d9c7740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f659b4b9f0 con 0x55f659c2bbc0 2024-05-12T22:12:25.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.078+0000 7f022d9c7740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x55f659b8cb50 con 0x55f659c36bb0 2024-05-12T22:12:25.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.078+0000 7f022d9c7740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55f659c399a0 con 0x55f659c2c4e0 2024-05-12T22:12:25.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.078+0000 7f022d9c7740 10 client.0 _ll_register_callbacks cb 0x55f6598a3be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:12:25.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.078+0000 7f0223e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f659c2bbc0 0x55f659c36530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:25.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.078+0000 7f0223e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f659c2bbc0 0x55f659c36530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52592/0 (socket says 172.21.15.28:52592) 2024-05-12T22:12:25.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.078+0000 7f0223e00640 1 -- 172.21.15.28:0/4165388796 learned_addr learned my addr 172.21.15.28:0/4165388796 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:12:25.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.078+0000 7f0223400640 1 --2- 172.21.15.28:0/4165388796 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f659c2c4e0 0x55f659c33820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:25.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.078+0000 7f0228c00640 1 --2- 172.21.15.28:0/4165388796 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f659c36bb0 0x55f659c33da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:25.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.078+0000 7f0223e00640 1 -- 172.21.15.28:0/4165388796 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f659c2c4e0 msgr2=0x55f659c33820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:12:25.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.078+0000 7f0223e00640 1 --2- 172.21.15.28:0/4165388796 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f659c2c4e0 0x55f659c33820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:25.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.078+0000 7f0223e00640 1 -- 172.21.15.28:0/4165388796 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f659c36bb0 msgr2=0x55f659c33da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:12:25.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.078+0000 7f0223e00640 1 --2- 172.21.15.28:0/4165388796 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f659c36bb0 0x55f659c33da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:25.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.078+0000 7f0223e00640 1 -- 172.21.15.28:0/4165388796 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f6598a43f0 con 0x55f659c2bbc0 2024-05-12T22:12:25.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.079+0000 7f0223e00640 1 --2- 172.21.15.28:0/4165388796 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f659c2bbc0 0x55f659c36530 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f0214009730 tx=0x7f0214004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=513e8eb8ff1f4669 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:12:25.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.079+0000 7f020d600640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-12T22:12:25.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.079+0000 7f020d600640 1 -- 172.21.15.28:0/4165388796 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f021401f640 con 0x55f659c2bbc0 2024-05-12T22:12:25.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.079+0000 7f020d600640 1 -- 172.21.15.28:0/4165388796 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f021401f7e0 con 0x55f659c2bbc0 2024-05-12T22:12:25.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.079+0000 7f022d9c7740 1 -- 172.21.15.28:0/4165388796 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0210005500 con 0x55f659c2bbc0 2024-05-12T22:12:25.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.079+0000 7f020d600640 1 -- 172.21.15.28:0/4165388796 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f021401f9e0 con 0x55f659c2bbc0 2024-05-12T22:12:25.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.079+0000 7f022d9c7740 10 client.15906 fetch_fsmap learned FSMap version 25 2024-05-12T22:12:25.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.079+0000 7f022d9c7740 1 -- 172.21.15.28:0/4165388796 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55f6598640f0 con 0x55f659c2bbc0 2024-05-12T22:12:25.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.080+0000 7f020d600640 1 -- 172.21.15.28:0/4165388796 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0210005500 con 0x55f659c2bbc0 2024-05-12T22:12:25.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.080+0000 7f020d600640 1 -- 172.21.15.28:0/4165388796 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7f0214027070 con 0x55f659c2bbc0 2024-05-12T22:12:25.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.080+0000 7f022d9c7740 10 client.15906 fetch_fsmap finished waiting for FSMap version 25 2024-05-12T22:12:25.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.080+0000 7f022d9c7740 10 client.15906 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:12:25.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.080+0000 7f022d9c7740 20 client.15906 populate_metadata read hostname 'smithi028' 2024-05-12T22:12:25.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.080+0000 7f022d9c7740 1 --2- 172.21.15.28:0/4165388796 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55f659c46810 0x55f659c48c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:25.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.080+0000 7f022d9c7740 4 client.15906 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-12T22:12:25.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.080+0000 7f022d9c7740 1 -- 172.21.15.28:0/4165388796 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f659b9e2c0 con 0x55f659c46810 2024-05-12T22:12:25.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.080+0000 7f0223400640 1 --2- 172.21.15.28:0/4165388796 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55f659c46810 0x55f659c48c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:25.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.081+0000 7f0223400640 1 --2- 172.21.15.28:0/4165388796 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55f659c46810 0x55f659c48c60 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d5d9c0703e6e9dde server_cookie=89954de72c081271 in_seq=0 out_seq=0 2024-05-12T22:12:25.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.081+0000 7f020d600640 10 client.15906 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:12:25.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.082+0000 7f020d600640 1 -- 172.21.15.28:0/4165388796 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f659b9e2c0 con 0x55f659c46810 2024-05-12T22:12:25.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.082+0000 7f020d600640 10 client.15906 handle_command_reply: tid=0 2024-05-12T22:12:25.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.082+0000 7f022d9c7740 1 client.15906 shutdown 2024-05-12T22:12:25.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.082+0000 7f022d9c7740 1 -- 172.21.15.28:0/4165388796 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f659c2bbc0 msgr2=0x55f659c36530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:12:25.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.082+0000 7f022d9c7740 1 --2- 172.21.15.28:0/4165388796 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f659c2bbc0 0x55f659c36530 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f0214009730 tx=0x7f0214004730 comp rx=0 tx=0).stop 2024-05-12T22:12:25.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.082+0000 7f022d9c7740 1 -- 172.21.15.28:0/4165388796 shutdown_connections 2024-05-12T22:12:25.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.082+0000 7f022d9c7740 1 --2- 172.21.15.28:0/4165388796 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55f659c46810 0x55f659c48c60 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:25.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.082+0000 7f022d9c7740 1 --2- 172.21.15.28:0/4165388796 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f659c36bb0 0x55f659c33da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:25.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.082+0000 7f022d9c7740 1 --2- 172.21.15.28:0/4165388796 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f659c2c4e0 0x55f659c33820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:25.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.082+0000 7f020d600640 0 client.15906 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:12:25.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.082+0000 7f022d9c7740 1 --2- 172.21.15.28:0/4165388796 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f659c2bbc0 0x55f659c36530 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:25.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.082+0000 7f022d9c7740 1 -- 172.21.15.28:0/4165388796 >> 172.21.15.28:0/4165388796 conn(0x55f659c27ee0 msgr2=0x55f659c270e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:12:25.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.082+0000 7f022d9c7740 1 -- 172.21.15.28:0/4165388796 shutdown_connections 2024-05-12T22:12:25.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.082+0000 7f022d9c7740 1 -- 172.21.15.28:0/4165388796 wait complete. 2024-05-12T22:12:25.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.083+0000 7f022d9c7740 20 client.15906 trim_cache size 0 max 16384 2024-05-12T22:12:25.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.102+0000 7f022d9c7740 1 Processor -- start 2024-05-12T22:12:25.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.102+0000 7f022d9c7740 1 -- start start 2024-05-12T22:12:25.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.102+0000 7f022d9c7740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f659c2bbc0 0x55f659c426e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:25.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.102+0000 7f022d9c7740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f659c2c4e0 0x55f659c36060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:25.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.102+0000 7f022d9c7740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f659c36bb0 0x55f659c4e530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:25.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.102+0000 7f022d9c7740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f02241b6350 con 0x55f659c2bbc0 2024-05-12T22:12:25.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.102+0000 7f022d9c7740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f02241b3c20 con 0x55f659c36bb0 2024-05-12T22:12:25.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.102+0000 7f022d9c7740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f02241b1a70 con 0x55f659c2c4e0 2024-05-12T22:12:25.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.102+0000 7f0223e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f659c2bbc0 0x55f659c426e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:25.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.103+0000 7f0223e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f659c2bbc0 0x55f659c426e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52602/0 (socket says 172.21.15.28:52602) 2024-05-12T22:12:25.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.103+0000 7f0223e00640 1 -- 172.21.15.28:0/385422588 learned_addr learned my addr 172.21.15.28:0/385422588 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:12:25.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.103+0000 7f0228c00640 1 --2- 172.21.15.28:0/385422588 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f659c36bb0 0x55f659c4e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:25.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.103+0000 7f0223400640 1 --2- 172.21.15.28:0/385422588 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f659c2c4e0 0x55f659c36060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:25.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.103+0000 7f0223e00640 1 -- 172.21.15.28:0/385422588 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f659c2c4e0 msgr2=0x55f659c36060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:12:25.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.103+0000 7f0223e00640 1 --2- 172.21.15.28:0/385422588 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f659c2c4e0 0x55f659c36060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:25.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.103+0000 7f0223e00640 1 -- 172.21.15.28:0/385422588 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f659c36bb0 msgr2=0x55f659c4e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:12:25.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.103+0000 7f0223e00640 1 --2- 172.21.15.28:0/385422588 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f659c36bb0 0x55f659c4e530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:25.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.103+0000 7f0223e00640 1 -- 172.21.15.28:0/385422588 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f659c31d70 con 0x55f659c2bbc0 2024-05-12T22:12:25.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.103+0000 7f0223e00640 1 --2- 172.21.15.28:0/385422588 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f659c2bbc0 0x55f659c426e0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f0214012cb0 tx=0x7f0214009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c58498ac7277c275 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:12:25.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.104+0000 7f020fe00640 1 -- 172.21.15.28:0/385422588 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0214036070 con 0x55f659c2bbc0 2024-05-12T22:12:25.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.104+0000 7f020fe00640 1 -- 172.21.15.28:0/385422588 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f021401de00 con 0x55f659c2bbc0 2024-05-12T22:12:25.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.104+0000 7f020fe00640 1 -- 172.21.15.28:0/385422588 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0214005020 con 0x55f659c2bbc0 2024-05-12T22:12:25.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.104+0000 7f022d9c7740 1 -- 172.21.15.28:0/385422588 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f659c2bbc0 msgr2=0x55f659c426e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:12:25.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.104+0000 7f022d9c7740 1 --2- 172.21.15.28:0/385422588 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f659c2bbc0 0x55f659c426e0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f0214012cb0 tx=0x7f0214009c00 comp rx=0 tx=0).stop 2024-05-12T22:12:25.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.104+0000 7f022d9c7740 1 -- 172.21.15.28:0/385422588 shutdown_connections 2024-05-12T22:12:25.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.104+0000 7f022d9c7740 1 --2- 172.21.15.28:0/385422588 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f659c36bb0 0x55f659c4e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:25.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.104+0000 7f022d9c7740 1 --2- 172.21.15.28:0/385422588 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f659c2c4e0 0x55f659c36060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:25.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.104+0000 7f022d9c7740 1 --2- 172.21.15.28:0/385422588 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f659c2bbc0 0x55f659c426e0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:25.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.104+0000 7f022d9c7740 1 -- 172.21.15.28:0/385422588 >> 172.21.15.28:0/385422588 conn(0x55f659c27ee0 msgr2=0x55f659c2dba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:12:25.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.104+0000 7f022d9c7740 1 -- 172.21.15.28:0/385422588 shutdown_connections 2024-05-12T22:12:25.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.104+0000 7f022d9c7740 1 -- 172.21.15.28:0/385422588 wait complete. 2024-05-12T22:12:25.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.105+0000 7f022d9c7740 1 -- start start 2024-05-12T22:12:25.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.105+0000 7f022d9c7740 1 Processor -- start 2024-05-12T22:12:25.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.105+0000 7f022d9c7740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f659c2bbc0 0x55f659c3ee10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:25.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.105+0000 7f022d9c7740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f659c2c4e0 0x55f659c3a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:25.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.105+0000 7f022d9c7740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f659c36bb0 0x55f659c3a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:25.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.105+0000 7f022d9c7740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f02241b20e0 con 0x55f659c36bb0 2024-05-12T22:12:25.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.105+0000 7f022d9c7740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f02241b3550 con 0x55f659c2c4e0 2024-05-12T22:12:25.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.105+0000 7f022d9c7740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f02241b27b0 con 0x55f659c2bbc0 2024-05-12T22:12:25.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.106+0000 7f022d9c7740 10 client.0 _ll_register_callbacks cb 0x7f0214009ad0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:12:25.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.106+0000 7f0228c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f659c36bb0 0x55f659c3a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:25.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.106+0000 7f0223400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f659c2c4e0 0x55f659c3a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:25.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.106+0000 7f0223400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f659c2c4e0 0x55f659c3a1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:32922/0 (socket says 172.21.15.28:32922) 2024-05-12T22:12:25.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.106+0000 7f0223400640 1 -- 172.21.15.28:0/2166718528 learned_addr learned my addr 172.21.15.28:0/2166718528 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:12:25.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.106+0000 7f0223e00640 1 --2- 172.21.15.28:0/2166718528 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f659c2bbc0 0x55f659c3ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:25.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.106+0000 7f0223400640 1 -- 172.21.15.28:0/2166718528 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f659c2bbc0 msgr2=0x55f659c3ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:12:25.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.106+0000 7f0223400640 1 --2- 172.21.15.28:0/2166718528 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f659c2bbc0 0x55f659c3ee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:25.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.106+0000 7f0223400640 1 -- 172.21.15.28:0/2166718528 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f659c36bb0 msgr2=0x55f659c3a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:12:25.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.106+0000 7f0223400640 1 --2- 172.21.15.28:0/2166718528 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f659c36bb0 0x55f659c3a6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:25.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.106+0000 7f0223400640 1 -- 172.21.15.28:0/2166718528 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f659c3ad30 con 0x55f659c2c4e0 2024-05-12T22:12:25.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.106+0000 7f0223400640 1 --2- 172.21.15.28:0/2166718528 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f659c2c4e0 0x55f659c3a1b0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f0210005f60 tx=0x7f0210004840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dd2c25c8861bf8d0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:12:25.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.106+0000 7f020d600640 10 client.0 ms_handle_connect on v2:172.21.15.86:3300/0 2024-05-12T22:12:25.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.107+0000 7f020d600640 1 -- 172.21.15.28:0/2166718528 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f021002d280 con 0x55f659c2c4e0 2024-05-12T22:12:25.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.107+0000 7f020d600640 1 -- 172.21.15.28:0/2166718528 <== mon.2 v2:172.21.15.86:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f021002d420 con 0x55f659c2c4e0 2024-05-12T22:12:25.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.107+0000 7f022d9c7740 1 -- 172.21.15.28:0/2166718528 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f021401de00 con 0x55f659c2c4e0 2024-05-12T22:12:25.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.107+0000 7f020d600640 1 -- 172.21.15.28:0/2166718528 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f021002d620 con 0x55f659c2c4e0 2024-05-12T22:12:25.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.107+0000 7f022d9c7740 10 client.26036 fetch_fsmap learned FSMap version 25 2024-05-12T22:12:25.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.107+0000 7f022d9c7740 1 -- 172.21.15.28:0/2166718528 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55f659cacc20 con 0x55f659c2c4e0 2024-05-12T22:12:25.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.107+0000 7f020d600640 1 -- 172.21.15.28:0/2166718528 <== mon.2 v2:172.21.15.86:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f021401de00 con 0x55f659c2c4e0 2024-05-12T22:12:25.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.107+0000 7f020d600640 1 -- 172.21.15.28:0/2166718528 <== mon.2 v2:172.21.15.86:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7f021002a030 con 0x55f659c2c4e0 2024-05-12T22:12:25.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.108+0000 7f022d9c7740 10 client.26036 fetch_fsmap finished waiting for FSMap version 25 2024-05-12T22:12:25.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.108+0000 7f022d9c7740 10 client.26036 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:12:25.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.108+0000 7f022d9c7740 20 client.26036 populate_metadata read hostname 'smithi028' 2024-05-12T22:12:25.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.108+0000 7f022d9c7740 1 --2- 172.21.15.28:0/2166718528 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55f659cad230 0x55f659caf650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:25.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.108+0000 7f022d9c7740 4 client.26036 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-12T22:12:25.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.108+0000 7f022d9c7740 1 -- 172.21.15.28:0/2166718528 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55f659b98610 con 0x55f659cad230 2024-05-12T22:12:25.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.108+0000 7f0223e00640 1 --2- 172.21.15.28:0/2166718528 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55f659cad230 0x55f659caf650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:25.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.108+0000 7f0223e00640 1 --2- 172.21.15.28:0/2166718528 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55f659cad230 0x55f659caf650 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a30706dfb88f0f85 server_cookie=15b374609a6d5d4e in_seq=0 out_seq=0 2024-05-12T22:12:25.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.108+0000 7f020d600640 10 client.26036 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:12:25.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.109+0000 7f020d600640 1 -- 172.21.15.28:0/2166718528 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55f659b98610 con 0x55f659cad230 2024-05-12T22:12:25.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.109+0000 7f020d600640 10 client.26036 handle_command_reply: tid=0 2024-05-12T22:12:25.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.109+0000 7f022d9c7740 1 client.26036 shutdown 2024-05-12T22:12:25.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.109+0000 7f022d9c7740 1 -- 172.21.15.28:0/2166718528 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f659c2c4e0 msgr2=0x55f659c3a1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:12:25.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.109+0000 7f022d9c7740 1 --2- 172.21.15.28:0/2166718528 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f659c2c4e0 0x55f659c3a1b0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f0210005f60 tx=0x7f0210004840 comp rx=0 tx=0).stop 2024-05-12T22:12:25.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.109+0000 7f022d9c7740 1 -- 172.21.15.28:0/2166718528 shutdown_connections 2024-05-12T22:12:25.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.109+0000 7f022d9c7740 1 --2- 172.21.15.28:0/2166718528 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55f659cad230 0x55f659caf650 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:25.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.109+0000 7f022d9c7740 1 --2- 172.21.15.28:0/2166718528 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f659c36bb0 0x55f659c3a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:25.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.109+0000 7f022d9c7740 1 --2- 172.21.15.28:0/2166718528 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f659c2c4e0 0x55f659c3a1b0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:25.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.109+0000 7f022d9c7740 1 --2- 172.21.15.28:0/2166718528 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f659c2bbc0 0x55f659c3ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:25.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.109+0000 7f020d600640 0 client.26036 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:12:25.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.109+0000 7f022d9c7740 1 -- 172.21.15.28:0/2166718528 >> 172.21.15.28:0/2166718528 conn(0x55f659c27ee0 msgr2=0x55f659c26850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:12:25.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.109+0000 7f022d9c7740 1 -- 172.21.15.28:0/2166718528 shutdown_connections 2024-05-12T22:12:25.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.109+0000 7f022d9c7740 1 -- 172.21.15.28:0/2166718528 wait complete. 2024-05-12T22:12:25.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.109+0000 7f022d9c7740 20 client.26036 trim_cache size 0 max 16384 2024-05-12T22:12:25.118 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-05-12T22:12:25.118 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-05-12T22:12:25.119 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-05-12T22:12:25.119 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-05-12T22:12:25.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.110+0000 7f022ac00640 1 -- 172.21.15.28:0/585188107 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f02080a2d80 msgr2=0x7f02080a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:12:25.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.110+0000 7f022ac00640 1 --2- 172.21.15.28:0/585188107 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f02080a2d80 0x7f02080a5240 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x55f659c2d180 tx=0x7f0214005f90 comp rx=0 tx=0).stop 2024-05-12T22:12:25.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.110+0000 7f022ac00640 1 -- 172.21.15.28:0/585188107 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f02241126b0 msgr2=0x7f02241a2fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:12:25.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.110+0000 7f022ac00640 1 --2- 172.21.15.28:0/585188107 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f02241126b0 0x7f02241a2fe0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f021c030e50 tx=0x7f021c030e80 comp rx=0 tx=0).stop 2024-05-12T22:12:25.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.110+0000 7f022ac00640 1 -- 172.21.15.28:0/585188107 shutdown_connections 2024-05-12T22:12:25.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.110+0000 7f022ac00640 1 --2- 172.21.15.28:0/585188107 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f02080a2d80 0x7f02080a5240 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:25.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.110+0000 7f022ac00640 1 --2- 172.21.15.28:0/585188107 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f02241126b0 0x7f02241a2fe0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:25.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.110+0000 7f022ac00640 1 --2- 172.21.15.28:0/585188107 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0224076680 0x7f022419ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:25.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.110+0000 7f022ac00640 1 --2- 172.21.15.28:0/585188107 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0224075d60 0x7f022419e770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:25.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.110+0000 7f022ac00640 1 -- 172.21.15.28:0/585188107 >> 172.21.15.28:0/585188107 conn(0x7f02240ff4a0 msgr2=0x7f0224101890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:12:25.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.110+0000 7f022ac00640 1 -- 172.21.15.28:0/585188107 shutdown_connections 2024-05-12T22:12:25.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.110+0000 7f022ac00640 1 -- 172.21.15.28:0/585188107 wait complete. 2024-05-12T22:12:25.129 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-12T22:12:25.130 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-12T22:12:25.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.329+0000 7fb328a00640 1 -- 172.21.15.28:0/3704298269 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb3241076b0 msgr2=0x7fb324107a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:12:25.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.329+0000 7fb328a00640 1 --2- 172.21.15.28:0/3704298269 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb3241076b0 0x7fb324107a90 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fb314009900 tx=0x7fb31402a410 comp rx=0 tx=0).stop 2024-05-12T22:12:25.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.329+0000 7fb328a00640 1 -- 172.21.15.28:0/3704298269 shutdown_connections 2024-05-12T22:12:25.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.329+0000 7fb328a00640 1 --2- 172.21.15.28:0/3704298269 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb324101fd0 0x7fb32410e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:25.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.329+0000 7fb328a00640 1 --2- 172.21.15.28:0/3704298269 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb324101610 0x7fb324101a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:25.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.329+0000 7fb328a00640 1 --2- 172.21.15.28:0/3704298269 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb3241076b0 0x7fb324107a90 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:25.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.329+0000 7fb328a00640 1 -- 172.21.15.28:0/3704298269 >> 172.21.15.28:0/3704298269 conn(0x7fb3240fd160 msgr2=0x7fb3240ff580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:12:25.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.330+0000 7fb328a00640 1 -- 172.21.15.28:0/3704298269 shutdown_connections 2024-05-12T22:12:25.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.330+0000 7fb328a00640 1 -- 172.21.15.28:0/3704298269 wait complete. 2024-05-12T22:12:25.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.331+0000 7fb328a00640 1 Processor -- start 2024-05-12T22:12:25.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.331+0000 7fb328a00640 1 -- start start 2024-05-12T22:12:25.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.331+0000 7fb328a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb324101610 0x7fb32419a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:25.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.331+0000 7fb328a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb324101fd0 0x7fb32419a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:25.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.332+0000 7fb328a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb3241076b0 0x7fb32419ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:25.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.332+0000 7fb328a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb324111a80 con 0x7fb3241076b0 2024-05-12T22:12:25.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.332+0000 7fb328a00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fb324111900 con 0x7fb324101610 2024-05-12T22:12:25.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.332+0000 7fb328a00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb324111c00 con 0x7fb324101fd0 2024-05-12T22:12:25.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.332+0000 7fb323200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb3241076b0 0x7fb32419ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:25.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.332+0000 7fb323200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb3241076b0 0x7fb32419ea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52628/0 (socket says 172.21.15.28:52628) 2024-05-12T22:12:25.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.332+0000 7fb322800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb324101610 0x7fb32419a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:25.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.332+0000 7fb321e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb324101fd0 0x7fb32419a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:25.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.332+0000 7fb323200640 1 -- 172.21.15.28:0/3027396790 learned_addr learned my addr 172.21.15.28:0/3027396790 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:12:25.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.332+0000 7fb321e00640 1 -- 172.21.15.28:0/3027396790 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb324101610 msgr2=0x7fb32419a180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:12:25.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.332+0000 7fb321e00640 1 --2- 172.21.15.28:0/3027396790 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb324101610 0x7fb32419a180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:25.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.332+0000 7fb321e00640 1 -- 172.21.15.28:0/3027396790 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb3241076b0 msgr2=0x7fb32419ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:12:25.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.332+0000 7fb321e00640 1 --2- 172.21.15.28:0/3027396790 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb3241076b0 0x7fb32419ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:25.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.332+0000 7fb321e00640 1 -- 172.21.15.28:0/3027396790 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb32419f170 con 0x7fb324101fd0 2024-05-12T22:12:25.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.333+0000 7fb321e00640 1 --2- 172.21.15.28:0/3027396790 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb324101fd0 0x7fb32419a6c0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fb31000e9f0 tx=0x7fb31000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:12:25.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.333+0000 7fb31b400640 1 -- 172.21.15.28:0/3027396790 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb31000ccc0 con 0x7fb324101fd0 2024-05-12T22:12:25.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.333+0000 7fb31b400640 1 -- 172.21.15.28:0/3027396790 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb310010910 con 0x7fb324101fd0 2024-05-12T22:12:25.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.333+0000 7fb31b400640 1 -- 172.21.15.28:0/3027396790 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb310018960 con 0x7fb324101fd0 2024-05-12T22:12:25.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.333+0000 7fb328a00640 1 -- 172.21.15.28:0/3027396790 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb32419f460 con 0x7fb324101fd0 2024-05-12T22:12:25.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.333+0000 7fb328a00640 1 -- 172.21.15.28:0/3027396790 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3241a6cf0 con 0x7fb324101fd0 2024-05-12T22:12:25.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.334+0000 7fb328a00640 1 -- 172.21.15.28:0/3027396790 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb3240689c0 con 0x7fb324101fd0 2024-05-12T22:12:25.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.337+0000 7fb31b400640 1 -- 172.21.15.28:0/3027396790 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7fb310010ac0 con 0x7fb324101fd0 2024-05-12T22:12:25.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.339+0000 7fb31b400640 1 --2- 172.21.15.28:0/3027396790 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fb3080a2d80 0x7fb3080a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:25.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.339+0000 7fb31b400640 1 -- 172.21.15.28:0/3027396790 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7fb3100cf790 con 0x7fb324101fd0 2024-05-12T22:12:25.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.339+0000 7fb322800640 1 --2- 172.21.15.28:0/3027396790 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fb3080a2d80 0x7fb3080a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:25.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.340+0000 7fb322800640 1 --2- 172.21.15.28:0/3027396790 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fb3080a2d80 0x7fb3080a5240 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fb314006010 tx=0x7fb314000bc0 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:12:25.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.343+0000 7fb31b400640 1 -- 172.21.15.28:0/3027396790 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fb31009bd30 con 0x7fb324101fd0 2024-05-12T22:12:25.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.588+0000 7fb328a00640 1 -- 172.21.15.28:0/3027396790 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb3241a6f80 con 0x7fb324101fd0 2024-05-12T22:12:25.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.589+0000 7fb31b400640 1 -- 172.21.15.28:0/3027396790 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11865 (secure 0 0 0) 0x7fb3100a0030 con 0x7fb324101fd0 2024-05-12T22:12:25.590 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-12T22:12:25.591 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1844141745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1844141745},{"type":"v1","addr":"172.21.15.186:6833","nonce":1844141745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6833/2960763758","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":2960763758},{"type":"v1","addr":"172.21.15.86:6833","nonce":2960763758}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24470,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/4131134219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":4131134219},{"type":"v1","addr":"172.21.15.28:6835","nonce":4131134219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24481,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/1727869064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1727869064},{"type":"v1","addr":"172.21.15.186:6835","nonce":1727869064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24482,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2938260704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2938260704},{"type":"v1","addr":"172.21.15.28:6837","nonce":2938260704}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24494,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3473700910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3473700910},{"type":"v1","addr":"172.21.15.86:6835","nonce":3473700910}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-12T22:06:33.426986+0000","modified":"2024-05-12T22:10:42.498031+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24581,"mds_1":24550,"mds_2":24532},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24499":{"gid":24499,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6837/3600809235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":3600809235},{"type":"v1","addr":"172.21.15.186:6837","nonce":3600809235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"i","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6839/2399610807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":2399610807},{"type":"v1","addr":"172.21.15.186:6839","nonce":2399610807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24530":{"gid":24530,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.86:6837/3782448884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":3782448884},{"type":"v1","addr":"172.21.15.86:6837","nonce":3782448884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24532":{"gid":24532,"name":"g","rank":2,"incarnation":21,"state":"up:active","state_seq":6,"addr":"172.21.15.28:6839/123892158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":123892158},{"type":"v1","addr":"172.21.15.28:6839","nonce":123892158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24550":{"gid":24550,"name":"a","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.28:6841/761183870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":761183870},{"type":"v1","addr":"172.21.15.28:6841","nonce":761183870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24581":{"gid":24581,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6839/1251999130","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6838","nonce":1251999130},{"type":"v1","addr":"172.21.15.86:6839","nonce":1251999130}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24581,"qdb_cluster":[24550,24532,24581]},"id":1}]} 2024-05-12T22:12:25.591 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 25 2024-05-12T22:12:25.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.593+0000 7fb328a00640 1 -- 172.21.15.28:0/3027396790 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fb3080a2d80 msgr2=0x7fb3080a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:12:25.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.593+0000 7fb328a00640 1 --2- 172.21.15.28:0/3027396790 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fb3080a2d80 0x7fb3080a5240 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fb314006010 tx=0x7fb314000bc0 comp rx=0 tx=0).stop 2024-05-12T22:12:25.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.593+0000 7fb328a00640 1 -- 172.21.15.28:0/3027396790 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb324101fd0 msgr2=0x7fb32419a6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:12:25.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.593+0000 7fb328a00640 1 --2- 172.21.15.28:0/3027396790 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb324101fd0 0x7fb32419a6c0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fb31000e9f0 tx=0x7fb31000eeb0 comp rx=0 tx=0).stop 2024-05-12T22:12:25.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.593+0000 7fb328a00640 1 -- 172.21.15.28:0/3027396790 shutdown_connections 2024-05-12T22:12:25.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.593+0000 7fb328a00640 1 --2- 172.21.15.28:0/3027396790 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fb3080a2d80 0x7fb3080a5240 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:25.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.593+0000 7fb328a00640 1 --2- 172.21.15.28:0/3027396790 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb3241076b0 0x7fb32419ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:25.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.593+0000 7fb328a00640 1 --2- 172.21.15.28:0/3027396790 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb324101fd0 0x7fb32419a6c0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:25.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.593+0000 7fb328a00640 1 --2- 172.21.15.28:0/3027396790 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb324101610 0x7fb32419a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:25.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.593+0000 7fb328a00640 1 -- 172.21.15.28:0/3027396790 >> 172.21.15.28:0/3027396790 conn(0x7fb3240fd160 msgr2=0x7fb32410ca40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:12:25.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.593+0000 7fb328a00640 1 -- 172.21.15.28:0/3027396790 shutdown_connections 2024-05-12T22:12:25.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.593+0000 7fb328a00640 1 -- 172.21.15.28:0/3027396790 wait complete. 2024-05-12T22:12:25.613 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-05-12T22:12:25.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.831+0000 7f3e62e00640 1 -- 172.21.15.28:0/1844413977 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3e5c10ddf0 msgr2=0x7f3e5c10e1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:12:25.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.831+0000 7f3e62e00640 1 --2- 172.21.15.28:0/1844413977 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3e5c10ddf0 0x7f3e5c10e1d0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f3e4c009860 tx=0x7f3e4c02a3d0 comp rx=0 tx=0).stop 2024-05-12T22:12:25.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.831+0000 7f3e62e00640 1 -- 172.21.15.28:0/1844413977 shutdown_connections 2024-05-12T22:12:25.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.831+0000 7f3e62e00640 1 --2- 172.21.15.28:0/1844413977 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e5c108770 0x7f3e5c114b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:25.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.831+0000 7f3e62e00640 1 --2- 172.21.15.28:0/1844413977 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3e5c107dd0 0x7f3e5c108230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:25.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.831+0000 7f3e62e00640 1 --2- 172.21.15.28:0/1844413977 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3e5c10ddf0 0x7f3e5c10e1d0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:25.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.831+0000 7f3e62e00640 1 -- 172.21.15.28:0/1844413977 >> 172.21.15.28:0/1844413977 conn(0x7f3e5c077650 msgr2=0x7f3e5c106650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:12:25.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.831+0000 7f3e62e00640 1 -- 172.21.15.28:0/1844413977 shutdown_connections 2024-05-12T22:12:25.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.831+0000 7f3e62e00640 1 -- 172.21.15.28:0/1844413977 wait complete. 2024-05-12T22:12:25.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.832+0000 7f3e62e00640 1 Processor -- start 2024-05-12T22:12:25.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.833+0000 7f3e62e00640 1 -- start start 2024-05-12T22:12:25.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.833+0000 7f3e62e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3e5c107dd0 0x7f3e5c19e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:25.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.833+0000 7f3e62e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3e5c108770 0x7f3e5c19eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:25.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.833+0000 7f3e62e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e5c10ddf0 0x7f3e5c1a2e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:25.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.833+0000 7f3e62e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3e5c118100 con 0x7f3e5c10ddf0 2024-05-12T22:12:25.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.833+0000 7f3e62e00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f3e5c117f80 con 0x7f3e5c107dd0 2024-05-12T22:12:25.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.833+0000 7f3e62e00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f3e5c118280 con 0x7f3e5c108770 2024-05-12T22:12:25.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.833+0000 7f3e62400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e5c10ddf0 0x7f3e5c1a2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:25.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.833+0000 7f3e62400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e5c10ddf0 0x7f3e5c1a2e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39482/0 (socket says 172.21.15.28:39482) 2024-05-12T22:12:25.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.833+0000 7f3e62400640 1 -- 172.21.15.28:0/3808848072 learned_addr learned my addr 172.21.15.28:0/3808848072 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:12:25.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.833+0000 7f3e61a00640 1 --2- 172.21.15.28:0/3808848072 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3e5c107dd0 0x7f3e5c19e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:25.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.833+0000 7f3e61000640 1 --2- 172.21.15.28:0/3808848072 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3e5c108770 0x7f3e5c19eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:25.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.834+0000 7f3e62400640 1 -- 172.21.15.28:0/3808848072 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3e5c108770 msgr2=0x7f3e5c19eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:12:25.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.834+0000 7f3e62400640 1 --2- 172.21.15.28:0/3808848072 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3e5c108770 0x7f3e5c19eaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:25.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.834+0000 7f3e62400640 1 -- 172.21.15.28:0/3808848072 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3e5c107dd0 msgr2=0x7f3e5c19e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:12:25.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.834+0000 7f3e62400640 1 --2- 172.21.15.28:0/3808848072 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3e5c107dd0 0x7f3e5c19e560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:25.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.834+0000 7f3e62400640 1 -- 172.21.15.28:0/3808848072 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e5c1a3550 con 0x7f3e5c10ddf0 2024-05-12T22:12:25.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.834+0000 7f3e61000640 1 --2- 172.21.15.28:0/3808848072 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3e5c108770 0x7f3e5c19eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:12:25.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.834+0000 7f3e62400640 1 --2- 172.21.15.28:0/3808848072 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e5c10ddf0 0x7f3e5c1a2e20 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f3e5400ec70 tx=0x7f3e5400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:12:25.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.834+0000 7f3e5a000640 1 -- 172.21.15.28:0/3808848072 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e54003b70 con 0x7f3e5c10ddf0 2024-05-12T22:12:25.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.834+0000 7f3e5a000640 1 -- 172.21.15.28:0/3808848072 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e5400ee80 con 0x7f3e5c10ddf0 2024-05-12T22:12:25.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.834+0000 7f3e5a000640 1 -- 172.21.15.28:0/3808848072 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e540108a0 con 0x7f3e5c10ddf0 2024-05-12T22:12:25.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.834+0000 7f3e62e00640 1 -- 172.21.15.28:0/3808848072 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e5c1a3840 con 0x7f3e5c10ddf0 2024-05-12T22:12:25.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.835+0000 7f3e62e00640 1 -- 172.21.15.28:0/3808848072 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e5c1ab0d0 con 0x7f3e5c10ddf0 2024-05-12T22:12:25.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.838+0000 7f3e6720c740 1 Processor -- start 2024-05-12T22:12:25.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.838+0000 7f3e5a000640 1 -- 172.21.15.28:0/3808848072 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7f3e54004020 con 0x7f3e5c10ddf0 2024-05-12T22:12:25.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.838+0000 7f3e6720c740 1 -- start start 2024-05-12T22:12:25.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.839+0000 7f3e6720c740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bfe943bb60 0x55bfe943bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:25.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.839+0000 7f3e6720c740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55bfe943c480 0x55bfe9446610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:25.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.839+0000 7f3e6720c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bfe9446b50 0x55bfe9448f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:25.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.839+0000 7f3e6720c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55bfe93a0a30 con 0x55bfe9446b50 2024-05-12T22:12:25.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.839+0000 7f3e6720c740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x55bfe9348700 con 0x55bfe943c480 2024-05-12T22:12:25.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.839+0000 7f3e6720c740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55bfe939c9f0 con 0x55bfe943bb60 2024-05-12T22:12:25.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.839+0000 7f3e61a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55bfe943c480 0x55bfe9446610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:25.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.839+0000 7f3e61000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bfe9446b50 0x55bfe9448f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:25.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.839+0000 7f3e61a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55bfe943c480 0x55bfe9446610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:49392/0 (socket says 172.21.15.28:49392) 2024-05-12T22:12:25.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.839+0000 7f3e61a00640 1 -- 172.21.15.28:0/2197021411 learned_addr learned my addr 172.21.15.28:0/2197021411 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:12:25.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.839+0000 7f3e61000640 1 --2- 172.21.15.28:0/2197021411 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bfe943bb60 0x55bfe943bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:25.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.839+0000 7f3e61a00640 1 -- 172.21.15.28:0/2197021411 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bfe943bb60 msgr2=0x55bfe943bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:12:25.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.840+0000 7f3e61a00640 1 --2- 172.21.15.28:0/2197021411 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bfe943bb60 0x55bfe943bf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:25.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.840+0000 7f3e61a00640 1 -- 172.21.15.28:0/2197021411 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bfe9446b50 msgr2=0x55bfe9448f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:12:25.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.840+0000 7f3e61a00640 1 --2- 172.21.15.28:0/2197021411 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bfe9446b50 0x55bfe9448f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:25.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.840+0000 7f3e61a00640 1 -- 172.21.15.28:0/2197021411 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bfe90b43f0 con 0x55bfe943c480 2024-05-12T22:12:25.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.840+0000 7f3e61000640 1 --2- 172.21.15.28:0/2197021411 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bfe943bb60 0x55bfe943bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:12:25.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.840+0000 7f3e5a000640 1 --2- 172.21.15.28:0/3808848072 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f3e400a2d80 0x7f3e400a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:25.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.840+0000 7f3e5a000640 1 -- 172.21.15.28:0/3808848072 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f3e540cf7b0 con 0x7f3e5c10ddf0 2024-05-12T22:12:25.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.840+0000 7f3e61a00640 1 --2- 172.21.15.28:0/2197021411 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55bfe943c480 0x55bfe9446610 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f3e4c034000 tx=0x7f3e4c030ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=397930bba0f90899 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:12:25.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.840+0000 7f3e61000640 1 --2- 172.21.15.28:0/3808848072 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f3e400a2d80 0x7f3e400a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:25.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.840+0000 7f3e52a00640 1 -- 172.21.15.28:0/2197021411 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e4c004030 con 0x55bfe943c480 2024-05-12T22:12:25.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.840+0000 7f3e52a00640 1 -- 172.21.15.28:0/2197021411 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e4c041030 con 0x55bfe943c480 2024-05-12T22:12:25.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.840+0000 7f3e52a00640 1 -- 172.21.15.28:0/2197021411 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e4c005430 con 0x55bfe943c480 2024-05-12T22:12:25.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.841+0000 7f3e6720c740 1 -- 172.21.15.28:0/2197021411 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55bfe943c480 msgr2=0x55bfe9446610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:12:25.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.841+0000 7f3e6720c740 1 --2- 172.21.15.28:0/2197021411 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55bfe943c480 0x55bfe9446610 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f3e4c034000 tx=0x7f3e4c030ab0 comp rx=0 tx=0).stop 2024-05-12T22:12:25.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.841+0000 7f3e61000640 1 --2- 172.21.15.28:0/3808848072 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f3e400a2d80 0x7f3e400a5240 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x55bfe9439010 tx=0x7f3e480095b0 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:12:25.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.841+0000 7f3e6720c740 1 -- 172.21.15.28:0/2197021411 shutdown_connections 2024-05-12T22:12:25.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.841+0000 7f3e6720c740 1 --2- 172.21.15.28:0/2197021411 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bfe9446b50 0x55bfe9448f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:25.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.841+0000 7f3e6720c740 1 --2- 172.21.15.28:0/2197021411 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55bfe943c480 0x55bfe9446610 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:25.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.841+0000 7f3e6720c740 1 --2- 172.21.15.28:0/2197021411 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bfe943bb60 0x55bfe943bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:25.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.841+0000 7f3e6720c740 1 -- 172.21.15.28:0/2197021411 >> 172.21.15.28:0/2197021411 conn(0x55bfe9437e80 msgr2=0x55bfe9438260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:12:25.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.841+0000 7f3e6720c740 1 -- 172.21.15.28:0/2197021411 shutdown_connections 2024-05-12T22:12:25.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.841+0000 7f3e6720c740 1 -- 172.21.15.28:0/2197021411 wait complete. 2024-05-12T22:12:25.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.842+0000 7f3e6720c740 1 -- start start 2024-05-12T22:12:25.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.842+0000 7f3e6720c740 1 Processor -- start 2024-05-12T22:12:25.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.842+0000 7f3e6720c740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55bfe943bb60 0x55bfe9443ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:25.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.842+0000 7f3e6720c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bfe943c480 0x55bfe9444010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:25.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.843+0000 7f3e6720c740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bfe9446b50 0x55bfe9444550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:25.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.843+0000 7f3e6720c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55bfe94498e0 con 0x55bfe943c480 2024-05-12T22:12:25.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.843+0000 7f3e6720c740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x55bfe935ba70 con 0x55bfe943bb60 2024-05-12T22:12:25.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.843+0000 7f3e6720c740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55bfe9449a60 con 0x55bfe9446b50 2024-05-12T22:12:25.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.843+0000 7f3e6720c740 10 client.0 _ll_register_callbacks cb 0x55bfe933eba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:12:25.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.843+0000 7f3e61000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bfe943c480 0x55bfe9444010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:25.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.843+0000 7f3e61000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bfe943c480 0x55bfe9444010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39504/0 (socket says 172.21.15.28:39504) 2024-05-12T22:12:25.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.843+0000 7f3e61000640 1 -- 172.21.15.28:0/3327669810 learned_addr learned my addr 172.21.15.28:0/3327669810 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:12:25.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.843+0000 7f3e62400640 1 --2- 172.21.15.28:0/3327669810 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bfe9446b50 0x55bfe9444550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:25.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.843+0000 7f3e61a00640 1 --2- 172.21.15.28:0/3327669810 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55bfe943bb60 0x55bfe9443ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:25.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.843+0000 7f3e61000640 1 -- 172.21.15.28:0/3327669810 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bfe9446b50 msgr2=0x55bfe9444550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:12:25.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.843+0000 7f3e61000640 1 --2- 172.21.15.28:0/3327669810 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bfe9446b50 0x55bfe9444550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:25.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.843+0000 7f3e61000640 1 -- 172.21.15.28:0/3327669810 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55bfe943bb60 msgr2=0x55bfe9443ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:12:25.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.843+0000 7f3e61000640 1 --2- 172.21.15.28:0/3327669810 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55bfe943bb60 0x55bfe9443ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:25.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.843+0000 7f3e61000640 1 -- 172.21.15.28:0/3327669810 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bfe917e7d0 con 0x55bfe943c480 2024-05-12T22:12:25.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.844+0000 7f3e61000640 1 --2- 172.21.15.28:0/3327669810 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bfe943c480 0x55bfe9444010 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f3e4801f6b0 tx=0x7f3e48004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7b54ee6e1ccd3ff server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:12:25.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.844+0000 7f3e47400640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-12T22:12:25.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.844+0000 7f3e47400640 1 -- 172.21.15.28:0/3327669810 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e4802d060 con 0x55bfe943c480 2024-05-12T22:12:25.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.844+0000 7f3e47400640 1 -- 172.21.15.28:0/3327669810 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f3e4801a030 con 0x55bfe943c480 2024-05-12T22:12:25.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.844+0000 7f3e6720c740 1 -- 172.21.15.28:0/3327669810 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3e4c041030 con 0x55bfe943c480 2024-05-12T22:12:25.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.844+0000 7f3e47400640 1 -- 172.21.15.28:0/3327669810 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f3e4801d6b0 con 0x55bfe943c480 2024-05-12T22:12:25.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.844+0000 7f3e6720c740 10 client.15930 fetch_fsmap learned FSMap version 25 2024-05-12T22:12:25.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.844+0000 7f3e6720c740 1 -- 172.21.15.28:0/3327669810 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55bfe9455bf0 con 0x55bfe943c480 2024-05-12T22:12:25.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.845+0000 7f3e47400640 1 -- 172.21.15.28:0/3327669810 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e4c041030 con 0x55bfe943c480 2024-05-12T22:12:25.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.845+0000 7f3e47400640 1 -- 172.21.15.28:0/3327669810 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7f3e480278c0 con 0x55bfe943c480 2024-05-12T22:12:25.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.845+0000 7f3e6720c740 10 client.15930 fetch_fsmap finished waiting for FSMap version 25 2024-05-12T22:12:25.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.845+0000 7f3e6720c740 10 client.15930 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.g 2024-05-12T22:12:25.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.845+0000 7f3e6720c740 20 client.15930 populate_metadata read hostname 'smithi028' 2024-05-12T22:12:25.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.845+0000 7f3e6720c740 1 --2- 172.21.15.28:0/3327669810 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x55bfe9457100 0x55bfe9459530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:25.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.845+0000 7f3e6720c740 4 client.15930 mds_command: new command op to 24532 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-12T22:12:25.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.845+0000 7f3e6720c740 1 -- 172.21.15.28:0/3327669810 --> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bfe93ae160 con 0x55bfe9457100 2024-05-12T22:12:25.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.845+0000 7f3e61a00640 1 --2- 172.21.15.28:0/3327669810 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x55bfe9457100 0x55bfe9459530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:25.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.846+0000 7f3e61a00640 1 --2- 172.21.15.28:0/3327669810 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x55bfe9457100 0x55bfe9459530 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=870a448eca0f12c0 server_cookie=cc3fdec3b977d102 in_seq=0 out_seq=0 2024-05-12T22:12:25.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.846+0000 7f3e47400640 10 client.15930 ms_handle_connect on v2:172.21.15.28:6838/123892158 2024-05-12T22:12:25.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.847+0000 7f3e47400640 1 -- 172.21.15.28:0/3327669810 <== mds.2 v2:172.21.15.28:6838/123892158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55bfe93ae160 con 0x55bfe9457100 2024-05-12T22:12:25.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.847+0000 7f3e47400640 10 client.15930 handle_command_reply: tid=0 2024-05-12T22:12:25.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.847+0000 7f3e6720c740 1 client.15930 shutdown 2024-05-12T22:12:25.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.847+0000 7f3e6720c740 1 -- 172.21.15.28:0/3327669810 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bfe943c480 msgr2=0x55bfe9444010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:12:25.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.847+0000 7f3e6720c740 1 --2- 172.21.15.28:0/3327669810 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bfe943c480 0x55bfe9444010 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f3e4801f6b0 tx=0x7f3e48004730 comp rx=0 tx=0).stop 2024-05-12T22:12:25.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.847+0000 7f3e6720c740 1 -- 172.21.15.28:0/3327669810 shutdown_connections 2024-05-12T22:12:25.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.847+0000 7f3e6720c740 1 --2- 172.21.15.28:0/3327669810 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bfe9446b50 0x55bfe9444550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:25.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.847+0000 7f3e6720c740 1 --2- 172.21.15.28:0/3327669810 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bfe943c480 0x55bfe9444010 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:25.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.847+0000 7f3e6720c740 1 --2- 172.21.15.28:0/3327669810 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x55bfe9457100 0x55bfe9459530 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:25.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.847+0000 7f3e6720c740 1 --2- 172.21.15.28:0/3327669810 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55bfe943bb60 0x55bfe9443ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:25.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.847+0000 7f3e6720c740 1 -- 172.21.15.28:0/3327669810 >> 172.21.15.28:0/3327669810 conn(0x55bfe9437e80 msgr2=0x55bfe943dc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:12:25.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.847+0000 7f3e47400640 0 client.15930 ms_handle_reset on v2:172.21.15.28:6838/123892158 2024-05-12T22:12:25.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.847+0000 7f3e6720c740 1 -- 172.21.15.28:0/3327669810 shutdown_connections 2024-05-12T22:12:25.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.848+0000 7f3e6720c740 1 -- 172.21.15.28:0/3327669810 wait complete. 2024-05-12T22:12:25.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.848+0000 7f3e6720c740 20 client.15930 trim_cache size 0 max 16384 2024-05-12T22:12:25.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.863+0000 7f3e6720c740 1 Processor -- start 2024-05-12T22:12:25.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.863+0000 7f3e6720c740 1 -- start start 2024-05-12T22:12:25.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.863+0000 7f3e6720c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bfe943bb60 0x55bfe94527d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:25.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.863+0000 7f3e6720c740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55bfe9446b50 0x55bfe943e300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:25.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.863+0000 7f3e6720c740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bfe945ee40 0x55bfe945f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:25.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.863+0000 7f3e6720c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3e5c1b60e0 con 0x55bfe943bb60 2024-05-12T22:12:25.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.863+0000 7f3e6720c740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f3e5c1b39b0 con 0x55bfe9446b50 2024-05-12T22:12:25.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.863+0000 7f3e6720c740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f3e5c1b1800 con 0x55bfe945ee40 2024-05-12T22:12:25.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.863+0000 7f3e61000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55bfe9446b50 0x55bfe943e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:25.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.864+0000 7f3e61000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55bfe9446b50 0x55bfe943e300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:49410/0 (socket says 172.21.15.28:49410) 2024-05-12T22:12:25.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.864+0000 7f3e61000640 1 -- 172.21.15.28:0/928838994 learned_addr learned my addr 172.21.15.28:0/928838994 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:12:25.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.864+0000 7f3e61a00640 1 --2- 172.21.15.28:0/928838994 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bfe943bb60 0x55bfe94527d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:25.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.864+0000 7f3e61000640 1 -- 172.21.15.28:0/928838994 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bfe945ee40 msgr2=0x55bfe945f2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:12:25.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.864+0000 7f3e61000640 1 --2- 172.21.15.28:0/928838994 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bfe945ee40 0x55bfe945f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:25.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.864+0000 7f3e61000640 1 -- 172.21.15.28:0/928838994 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bfe943bb60 msgr2=0x55bfe94527d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:12:25.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.864+0000 7f3e61000640 1 --2- 172.21.15.28:0/928838994 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bfe943bb60 0x55bfe94527d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:25.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.864+0000 7f3e61000640 1 -- 172.21.15.28:0/928838994 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bfe943c270 con 0x55bfe9446b50 2024-05-12T22:12:25.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.864+0000 7f3e61a00640 1 --2- 172.21.15.28:0/928838994 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bfe943bb60 0x55bfe94527d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-12T22:12:25.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.864+0000 7f3e61000640 1 --2- 172.21.15.28:0/928838994 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55bfe9446b50 0x55bfe943e300 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f3e48027d10 tx=0x7f3e48027f80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=58a2fae9058323de server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:12:25.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.865+0000 7f3e52000640 1 -- 172.21.15.28:0/928838994 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e48005df0 con 0x55bfe9446b50 2024-05-12T22:12:25.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.865+0000 7f3e52000640 1 -- 172.21.15.28:0/928838994 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e48030380 con 0x55bfe9446b50 2024-05-12T22:12:25.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.865+0000 7f3e52000640 1 -- 172.21.15.28:0/928838994 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e48027840 con 0x55bfe9446b50 2024-05-12T22:12:25.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.865+0000 7f3e6720c740 1 -- 172.21.15.28:0/928838994 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55bfe9446b50 msgr2=0x55bfe943e300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:12:25.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.865+0000 7f3e6720c740 1 --2- 172.21.15.28:0/928838994 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55bfe9446b50 0x55bfe943e300 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f3e48027d10 tx=0x7f3e48027f80 comp rx=0 tx=0).stop 2024-05-12T22:12:25.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.865+0000 7f3e6720c740 1 -- 172.21.15.28:0/928838994 shutdown_connections 2024-05-12T22:12:25.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.865+0000 7f3e6720c740 1 --2- 172.21.15.28:0/928838994 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bfe945ee40 0x55bfe945f2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:25.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.865+0000 7f3e6720c740 1 --2- 172.21.15.28:0/928838994 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55bfe9446b50 0x55bfe943e300 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:25.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.865+0000 7f3e6720c740 1 --2- 172.21.15.28:0/928838994 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bfe943bb60 0x55bfe94527d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:25.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.865+0000 7f3e6720c740 1 -- 172.21.15.28:0/928838994 >> 172.21.15.28:0/928838994 conn(0x55bfe9437e80 msgr2=0x55bfe9444320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:12:25.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.865+0000 7f3e6720c740 1 -- 172.21.15.28:0/928838994 shutdown_connections 2024-05-12T22:12:25.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.865+0000 7f3e6720c740 1 -- 172.21.15.28:0/928838994 wait complete. 2024-05-12T22:12:25.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.866+0000 7f3e6720c740 1 -- start start 2024-05-12T22:12:25.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.866+0000 7f3e6720c740 1 Processor -- start 2024-05-12T22:12:25.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.866+0000 7f3e6720c740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55bfe943bb60 0x55bfe9440d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:25.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.866+0000 7f3e6720c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bfe9446b50 0x55bfe94412a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:25.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.866+0000 7f3e6720c740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bfe945ee40 0x55bfe94bc6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:25.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.867+0000 7f3e6720c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3e5c1b1e70 con 0x55bfe9446b50 2024-05-12T22:12:25.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.867+0000 7f3e6720c740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f3e5c1b32e0 con 0x55bfe943bb60 2024-05-12T22:12:25.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.867+0000 7f3e6720c740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f3e5c1b2540 con 0x55bfe945ee40 2024-05-12T22:12:25.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.867+0000 7f3e6720c740 10 client.0 _ll_register_callbacks cb 0x7f3e4800ef70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:12:25.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.867+0000 7f3e61000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bfe9446b50 0x55bfe94412a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:25.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.867+0000 7f3e61000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bfe9446b50 0x55bfe94412a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39522/0 (socket says 172.21.15.28:39522) 2024-05-12T22:12:25.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.867+0000 7f3e61000640 1 -- 172.21.15.28:0/3281854603 learned_addr learned my addr 172.21.15.28:0/3281854603 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:12:25.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.867+0000 7f3e61a00640 1 --2- 172.21.15.28:0/3281854603 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55bfe943bb60 0x55bfe9440d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:25.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.867+0000 7f3e62400640 1 --2- 172.21.15.28:0/3281854603 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bfe945ee40 0x55bfe94bc6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:25.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.867+0000 7f3e61a00640 1 -- 172.21.15.28:0/3281854603 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bfe945ee40 msgr2=0x55bfe94bc6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:12:25.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.867+0000 7f3e61a00640 1 --2- 172.21.15.28:0/3281854603 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bfe945ee40 0x55bfe94bc6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:25.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.867+0000 7f3e61a00640 1 -- 172.21.15.28:0/3281854603 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bfe9446b50 msgr2=0x55bfe94412a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:12:25.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.867+0000 7f3e61a00640 1 --2- 172.21.15.28:0/3281854603 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bfe9446b50 0x55bfe94412a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:25.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.867+0000 7f3e61a00640 1 -- 172.21.15.28:0/3281854603 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bfe94bcbf0 con 0x55bfe943bb60 2024-05-12T22:12:25.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.867+0000 7f3e61000640 1 --2- 172.21.15.28:0/3281854603 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bfe9446b50 0x55bfe94412a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:12:25.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.867+0000 7f3e61a00640 1 --2- 172.21.15.28:0/3281854603 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55bfe943bb60 0x55bfe9440d60 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f3e4c032ff0 tx=0x7f3e4c009450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d6dfb3d195b19bf4 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:12:25.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.867+0000 7f3e47400640 10 client.0 ms_handle_connect on v2:172.21.15.86:3300/0 2024-05-12T22:12:25.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.868+0000 7f3e47400640 1 -- 172.21.15.28:0/3281854603 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e4c0303a0 con 0x55bfe943bb60 2024-05-12T22:12:25.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.868+0000 7f3e47400640 1 -- 172.21.15.28:0/3281854603 <== mon.2 v2:172.21.15.86:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f3e4c030540 con 0x55bfe943bb60 2024-05-12T22:12:25.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.868+0000 7f3e6720c740 1 -- 172.21.15.28:0/3281854603 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3e48030380 con 0x55bfe943bb60 2024-05-12T22:12:25.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.868+0000 7f3e47400640 1 -- 172.21.15.28:0/3281854603 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f3e4c030740 con 0x55bfe943bb60 2024-05-12T22:12:25.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.868+0000 7f3e6720c740 10 client.26060 fetch_fsmap learned FSMap version 25 2024-05-12T22:12:25.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.868+0000 7f3e6720c740 1 -- 172.21.15.28:0/3281854603 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55bfe94bdd00 con 0x55bfe943bb60 2024-05-12T22:12:25.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.868+0000 7f3e47400640 1 -- 172.21.15.28:0/3281854603 <== mon.2 v2:172.21.15.86:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e48030380 con 0x55bfe943bb60 2024-05-12T22:12:25.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.868+0000 7f3e47400640 1 -- 172.21.15.28:0/3281854603 <== mon.2 v2:172.21.15.86:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7f3e4c004aa0 con 0x55bfe943bb60 2024-05-12T22:12:25.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.868+0000 7f3e6720c740 10 client.26060 fetch_fsmap finished waiting for FSMap version 25 2024-05-12T22:12:25.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.868+0000 7f3e6720c740 10 client.26060 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.g 2024-05-12T22:12:25.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.869+0000 7f3e6720c740 20 client.26060 populate_metadata read hostname 'smithi028' 2024-05-12T22:12:25.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.869+0000 7f3e6720c740 1 --2- 172.21.15.28:0/3281854603 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x55bfe94bdff0 0x55bfe94c03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:25.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.869+0000 7f3e6720c740 4 client.26060 mds_command: new command op to 24532 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-12T22:12:25.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.869+0000 7f3e6720c740 1 -- 172.21.15.28:0/3281854603 --> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55bfe93a84b0 con 0x55bfe94bdff0 2024-05-12T22:12:25.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.869+0000 7f3e61000640 1 --2- 172.21.15.28:0/3281854603 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x55bfe94bdff0 0x55bfe94c03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:25.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.869+0000 7f3e61000640 1 --2- 172.21.15.28:0/3281854603 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x55bfe94bdff0 0x55bfe94c03e0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b1ea8b99f33eec34 server_cookie=977b0c27c0cc627d in_seq=0 out_seq=0 2024-05-12T22:12:25.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.869+0000 7f3e47400640 10 client.26060 ms_handle_connect on v2:172.21.15.28:6838/123892158 2024-05-12T22:12:25.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.870+0000 7f3e47400640 1 -- 172.21.15.28:0/3281854603 <== mds.2 v2:172.21.15.28:6838/123892158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55bfe93a84b0 con 0x55bfe94bdff0 2024-05-12T22:12:25.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.870+0000 7f3e47400640 10 client.26060 handle_command_reply: tid=0 2024-05-12T22:12:25.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.870+0000 7f3e6720c740 1 client.26060 shutdown 2024-05-12T22:12:25.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.870+0000 7f3e6720c740 1 -- 172.21.15.28:0/3281854603 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55bfe943bb60 msgr2=0x55bfe9440d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:12:25.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.870+0000 7f3e6720c740 1 --2- 172.21.15.28:0/3281854603 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55bfe943bb60 0x55bfe9440d60 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f3e4c032ff0 tx=0x7f3e4c009450 comp rx=0 tx=0).stop 2024-05-12T22:12:25.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.870+0000 7f3e6720c740 1 -- 172.21.15.28:0/3281854603 shutdown_connections 2024-05-12T22:12:25.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.870+0000 7f3e6720c740 1 --2- 172.21.15.28:0/3281854603 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bfe945ee40 0x55bfe94bc6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:25.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.870+0000 7f3e6720c740 1 --2- 172.21.15.28:0/3281854603 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bfe9446b50 0x55bfe94412a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:25.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.870+0000 7f3e6720c740 1 --2- 172.21.15.28:0/3281854603 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x55bfe94bdff0 0x55bfe94c03e0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:25.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.870+0000 7f3e6720c740 1 --2- 172.21.15.28:0/3281854603 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55bfe943bb60 0x55bfe9440d60 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:25.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.870+0000 7f3e6720c740 1 -- 172.21.15.28:0/3281854603 >> 172.21.15.28:0/3281854603 conn(0x55bfe9437e80 msgr2=0x55bfe945de30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:12:25.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.870+0000 7f3e47400640 0 client.26060 ms_handle_reset on v2:172.21.15.28:6838/123892158 2024-05-12T22:12:25.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.870+0000 7f3e6720c740 1 -- 172.21.15.28:0/3281854603 shutdown_connections 2024-05-12T22:12:25.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.870+0000 7f3e6720c740 1 -- 172.21.15.28:0/3281854603 wait complete. 2024-05-12T22:12:25.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.870+0000 7f3e6720c740 20 client.26060 trim_cache size 0 max 16384 2024-05-12T22:12:25.886 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-05-12T22:12:25.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.871+0000 7f3e62e00640 1 -- 172.21.15.28:0/3808848072 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f3e400a2d80 msgr2=0x7f3e400a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:12:25.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.871+0000 7f3e62e00640 1 --2- 172.21.15.28:0/3808848072 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f3e400a2d80 0x7f3e400a5240 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x55bfe9439010 tx=0x7f3e480095b0 comp rx=0 tx=0).stop 2024-05-12T22:12:25.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.871+0000 7f3e62e00640 1 -- 172.21.15.28:0/3808848072 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e5c10ddf0 msgr2=0x7f3e5c1a2e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:12:25.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.871+0000 7f3e62e00640 1 --2- 172.21.15.28:0/3808848072 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e5c10ddf0 0x7f3e5c1a2e20 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f3e5400ec70 tx=0x7f3e5400c6a0 comp rx=0 tx=0).stop 2024-05-12T22:12:25.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.871+0000 7f3e62e00640 1 -- 172.21.15.28:0/3808848072 shutdown_connections 2024-05-12T22:12:25.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.871+0000 7f3e62e00640 1 --2- 172.21.15.28:0/3808848072 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f3e400a2d80 0x7f3e400a5240 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:25.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.871+0000 7f3e62e00640 1 --2- 172.21.15.28:0/3808848072 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e5c10ddf0 0x7f3e5c1a2e20 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:25.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.871+0000 7f3e62e00640 1 --2- 172.21.15.28:0/3808848072 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3e5c108770 0x7f3e5c19eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:25.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.871+0000 7f3e62e00640 1 --2- 172.21.15.28:0/3808848072 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3e5c107dd0 0x7f3e5c19e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:25.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.871+0000 7f3e62e00640 1 -- 172.21.15.28:0/3808848072 >> 172.21.15.28:0/3808848072 conn(0x7f3e5c077650 msgr2=0x7f3e5c078660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:12:25.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.871+0000 7f3e62e00640 1 -- 172.21.15.28:0/3808848072 shutdown_connections 2024-05-12T22:12:25.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:25.871+0000 7f3e62e00640 1 -- 172.21.15.28:0/3808848072 wait complete. 2024-05-12T22:12:25.890 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-05-12T22:12:26.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.107+0000 7f6d71200640 1 -- 172.21.15.28:0/651744260 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6d6c10ddf0 msgr2=0x7f6d6c10e1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:12:26.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.107+0000 7f6d71200640 1 --2- 172.21.15.28:0/651744260 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6d6c10ddf0 0x7f6d6c10e1d0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f6d5c009900 tx=0x7f6d5c02a480 comp rx=0 tx=0).stop 2024-05-12T22:12:26.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.107+0000 7f6d71200640 1 -- 172.21.15.28:0/651744260 shutdown_connections 2024-05-12T22:12:26.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.107+0000 7f6d71200640 1 --2- 172.21.15.28:0/651744260 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d6c108770 0x7f6d6c114b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:26.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.107+0000 7f6d71200640 1 --2- 172.21.15.28:0/651744260 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d6c107dd0 0x7f6d6c108230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:26.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.107+0000 7f6d71200640 1 --2- 172.21.15.28:0/651744260 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6d6c10ddf0 0x7f6d6c10e1d0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:26.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.107+0000 7f6d71200640 1 -- 172.21.15.28:0/651744260 >> 172.21.15.28:0/651744260 conn(0x7f6d6c077650 msgr2=0x7f6d6c106650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:12:26.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.107+0000 7f6d71200640 1 -- 172.21.15.28:0/651744260 shutdown_connections 2024-05-12T22:12:26.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.107+0000 7f6d71200640 1 -- 172.21.15.28:0/651744260 wait complete. 2024-05-12T22:12:26.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.108+0000 7f6d71200640 1 Processor -- start 2024-05-12T22:12:26.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.108+0000 7f6d71200640 1 -- start start 2024-05-12T22:12:26.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.109+0000 7f6d71200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6d6c107dd0 0x7f6d6c19e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:26.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.109+0000 7f6d71200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d6c108770 0x7f6d6c19eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:26.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.109+0000 7f6d71200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d6c10ddf0 0x7f6d6c1a2e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:26.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.109+0000 7f6d71200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6d6c118120 con 0x7f6d6c108770 2024-05-12T22:12:26.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.109+0000 7f6d71200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f6d6c117fa0 con 0x7f6d6c10ddf0 2024-05-12T22:12:26.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.109+0000 7f6d71200640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f6d6c1182a0 con 0x7f6d6c107dd0 2024-05-12T22:12:26.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.109+0000 7f6d6aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d6c108770 0x7f6d6c19eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:26.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.109+0000 7f6d6b400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6d6c107dd0 0x7f6d6c19e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:26.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.109+0000 7f6d6aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d6c108770 0x7f6d6c19eac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39548/0 (socket says 172.21.15.28:39548) 2024-05-12T22:12:26.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.109+0000 7f6d6aa00640 1 -- 172.21.15.28:0/4097248452 learned_addr learned my addr 172.21.15.28:0/4097248452 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:12:26.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.109+0000 7f6d6be00640 1 --2- 172.21.15.28:0/4097248452 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d6c10ddf0 0x7f6d6c1a2e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:26.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.110+0000 7f6d6be00640 1 -- 172.21.15.28:0/4097248452 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6d6c107dd0 msgr2=0x7f6d6c19e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:12:26.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.110+0000 7f6d6be00640 1 --2- 172.21.15.28:0/4097248452 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6d6c107dd0 0x7f6d6c19e580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:26.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.110+0000 7f6d6be00640 1 -- 172.21.15.28:0/4097248452 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d6c108770 msgr2=0x7f6d6c19eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:12:26.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.110+0000 7f6d6be00640 1 --2- 172.21.15.28:0/4097248452 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d6c108770 0x7f6d6c19eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:26.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.110+0000 7f6d6be00640 1 -- 172.21.15.28:0/4097248452 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d6c1a3570 con 0x7f6d6c10ddf0 2024-05-12T22:12:26.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.110+0000 7f6d6b400640 1 --2- 172.21.15.28:0/4097248452 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6d6c107dd0 0x7f6d6c19e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:12:26.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.110+0000 7f6d6be00640 1 --2- 172.21.15.28:0/4097248452 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d6c10ddf0 0x7f6d6c1a2e40 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f6d6400d9c0 tx=0x7f6d6400de80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:12:26.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.110+0000 7f6d63e00640 1 -- 172.21.15.28:0/4097248452 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d6400bd10 con 0x7f6d6c10ddf0 2024-05-12T22:12:26.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.110+0000 7f6d71200640 1 -- 172.21.15.28:0/4097248452 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d6c1a3860 con 0x7f6d6c10ddf0 2024-05-12T22:12:26.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.110+0000 7f6d63e00640 1 -- 172.21.15.28:0/4097248452 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6d64003e50 con 0x7f6d6c10ddf0 2024-05-12T22:12:26.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.110+0000 7f6d71200640 1 -- 172.21.15.28:0/4097248452 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d6c1ab0f0 con 0x7f6d6c10ddf0 2024-05-12T22:12:26.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.110+0000 7f6d63e00640 1 -- 172.21.15.28:0/4097248452 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d640106c0 con 0x7f6d6c10ddf0 2024-05-12T22:12:26.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.114+0000 7f6d75782740 1 Processor -- start 2024-05-12T22:12:26.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.114+0000 7f6d63e00640 1 -- 172.21.15.28:0/4097248452 <== mon.2 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7f6d64004230 con 0x7f6d6c10ddf0 2024-05-12T22:12:26.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.114+0000 7f6d75782740 1 -- start start 2024-05-12T22:12:26.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.114+0000 7f6d75782740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c789244b60 0x55c789244f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:26.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.114+0000 7f6d75782740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c789245480 0x55c78924f610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:26.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.114+0000 7f6d75782740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c78924fb50 0x55c789251f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:26.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.114+0000 7f6d75782740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c7891a9a30 con 0x55c789244b60 2024-05-12T22:12:26.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.114+0000 7f6d75782740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x55c789151700 con 0x55c78924fb50 2024-05-12T22:12:26.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.114+0000 7f6d75782740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55c7891a59f0 con 0x55c789245480 2024-05-12T22:12:26.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.115+0000 7f6d6aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c789244b60 0x55c789244f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:26.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.115+0000 7f6d6aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c789244b60 0x55c789244f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39560/0 (socket says 172.21.15.28:39560) 2024-05-12T22:12:26.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.115+0000 7f6d6aa00640 1 -- 172.21.15.28:0/3174165656 learned_addr learned my addr 172.21.15.28:0/3174165656 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:12:26.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.115+0000 7f6d6b400640 1 --2- 172.21.15.28:0/3174165656 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c789245480 0x55c78924f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:26.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.115+0000 7f6d6aa00640 1 --2- 172.21.15.28:0/3174165656 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c78924fb50 0x55c789251f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:26.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.115+0000 7f6d6aa00640 1 -- 172.21.15.28:0/3174165656 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c789245480 msgr2=0x55c78924f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:12:26.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.115+0000 7f6d6aa00640 1 --2- 172.21.15.28:0/3174165656 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c789245480 0x55c78924f610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:26.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.115+0000 7f6d6aa00640 1 -- 172.21.15.28:0/3174165656 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c78924fb50 msgr2=0x55c789251f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:12:26.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.115+0000 7f6d6aa00640 1 --2- 172.21.15.28:0/3174165656 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c78924fb50 0x55c789251f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:26.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.115+0000 7f6d6aa00640 1 -- 172.21.15.28:0/3174165656 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c788ebd3f0 con 0x55c789244b60 2024-05-12T22:12:26.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.115+0000 7f6d6aa00640 1 --2- 172.21.15.28:0/3174165656 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c789244b60 0x55c789244f40 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f6d58013720 tx=0x7f6d58009930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b673bbddf699b083 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:12:26.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.116+0000 7f6d60c00640 1 -- 172.21.15.28:0/3174165656 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d5800f8c0 con 0x55c789244b60 2024-05-12T22:12:26.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.116+0000 7f6d63e00640 1 --2- 172.21.15.28:0/4097248452 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f6d500a2d00 0x7f6d500a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:26.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.116+0000 7f6d60c00640 1 -- 172.21.15.28:0/3174165656 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6d58005020 con 0x55c789244b60 2024-05-12T22:12:26.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.116+0000 7f6d60c00640 1 -- 172.21.15.28:0/3174165656 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d58004b10 con 0x55c789244b60 2024-05-12T22:12:26.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.116+0000 7f6d63e00640 1 -- 172.21.15.28:0/4097248452 <== mon.2 v2:172.21.15.86:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f6d640cf420 con 0x7f6d6c10ddf0 2024-05-12T22:12:26.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.116+0000 7f6d6b400640 1 --2- 172.21.15.28:0/4097248452 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f6d500a2d00 0x7f6d500a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:26.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.116+0000 7f6d75782740 1 -- 172.21.15.28:0/3174165656 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c789244b60 msgr2=0x55c789244f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:12:26.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.116+0000 7f6d75782740 1 --2- 172.21.15.28:0/3174165656 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c789244b60 0x55c789244f40 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f6d58013720 tx=0x7f6d58009930 comp rx=0 tx=0).stop 2024-05-12T22:12:26.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.117+0000 7f6d75782740 1 -- 172.21.15.28:0/3174165656 shutdown_connections 2024-05-12T22:12:26.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.117+0000 7f6d75782740 1 --2- 172.21.15.28:0/3174165656 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c78924fb50 0x55c789251f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:26.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.117+0000 7f6d75782740 1 --2- 172.21.15.28:0/3174165656 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c789245480 0x55c78924f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:26.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.117+0000 7f6d75782740 1 --2- 172.21.15.28:0/3174165656 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c789244b60 0x55c789244f40 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:26.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.117+0000 7f6d75782740 1 -- 172.21.15.28:0/3174165656 >> 172.21.15.28:0/3174165656 conn(0x55c789240e80 msgr2=0x55c789241260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:12:26.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.117+0000 7f6d75782740 1 -- 172.21.15.28:0/3174165656 shutdown_connections 2024-05-12T22:12:26.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.117+0000 7f6d6b400640 1 --2- 172.21.15.28:0/4097248452 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f6d500a2d00 0x7f6d500a51c0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f6d5c000c00 tx=0x7f6d5c031510 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:12:26.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.117+0000 7f6d75782740 1 -- 172.21.15.28:0/3174165656 wait complete. 2024-05-12T22:12:26.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.117+0000 7f6d75782740 1 -- start start 2024-05-12T22:12:26.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.117+0000 7f6d75782740 1 Processor -- start 2024-05-12T22:12:26.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.118+0000 7f6d75782740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c789244b60 0x55c78924ca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:26.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.118+0000 7f6d75782740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c789245480 0x55c78924cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:26.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.118+0000 7f6d75782740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c78924fb50 0x55c78924d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:26.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.118+0000 7f6d75782740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c789252940 con 0x55c789245480 2024-05-12T22:12:26.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.118+0000 7f6d75782740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x55c789164a70 con 0x55c78924fb50 2024-05-12T22:12:26.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.118+0000 7f6d75782740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55c789252ac0 con 0x55c789244b60 2024-05-12T22:12:26.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.118+0000 7f6d75782740 10 client.0 _ll_register_callbacks cb 0x55c789147ba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:12:26.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.118+0000 7f6d6aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c789245480 0x55c78924cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:26.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.118+0000 7f6d6aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c789245480 0x55c78924cfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39564/0 (socket says 172.21.15.28:39564) 2024-05-12T22:12:26.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.118+0000 7f6d6aa00640 1 -- 172.21.15.28:0/338113630 learned_addr learned my addr 172.21.15.28:0/338113630 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:12:26.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.118+0000 7f6d6be00640 1 --2- 172.21.15.28:0/338113630 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c78924fb50 0x55c78924d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:26.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.118+0000 7f6d6b400640 1 --2- 172.21.15.28:0/338113630 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c789244b60 0x55c78924ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:26.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.119+0000 7f6d6be00640 1 -- 172.21.15.28:0/338113630 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c789244b60 msgr2=0x55c78924ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:12:26.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.119+0000 7f6d6be00640 1 --2- 172.21.15.28:0/338113630 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c789244b60 0x55c78924ca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:26.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.119+0000 7f6d6be00640 1 -- 172.21.15.28:0/338113630 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c789245480 msgr2=0x55c78924cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:12:26.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.119+0000 7f6d6be00640 1 --2- 172.21.15.28:0/338113630 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c789245480 0x55c78924cfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:26.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.119+0000 7f6d6be00640 1 -- 172.21.15.28:0/338113630 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c788f877d0 con 0x55c78924fb50 2024-05-12T22:12:26.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.119+0000 7f6d6be00640 1 --2- 172.21.15.28:0/338113630 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c78924fb50 0x55c78924d4f0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f6d640ccb40 tx=0x7f6d6409b270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=96e3df82b9410b30 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:12:26.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.119+0000 7f6d55600640 10 client.0 ms_handle_connect on v2:172.21.15.86:3300/0 2024-05-12T22:12:26.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.119+0000 7f6d55600640 1 -- 172.21.15.28:0/338113630 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d640b4060 con 0x55c78924fb50 2024-05-12T22:12:26.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.119+0000 7f6d55600640 1 -- 172.21.15.28:0/338113630 <== mon.2 v2:172.21.15.86:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f6d64097f80 con 0x55c78924fb50 2024-05-12T22:12:26.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.119+0000 7f6d75782740 1 -- 172.21.15.28:0/338113630 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6d58005020 con 0x55c78924fb50 2024-05-12T22:12:26.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.120+0000 7f6d55600640 1 -- 172.21.15.28:0/338113630 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f6d6409fcc0 con 0x55c78924fb50 2024-05-12T22:12:26.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.120+0000 7f6d75782740 10 client.26072 fetch_fsmap learned FSMap version 25 2024-05-12T22:12:26.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.120+0000 7f6d75782740 1 -- 172.21.15.28:0/338113630 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55c78925ebb0 con 0x55c78924fb50 2024-05-12T22:12:26.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.120+0000 7f6d55600640 1 -- 172.21.15.28:0/338113630 <== mon.2 v2:172.21.15.86:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6d58005020 con 0x55c78924fb50 2024-05-12T22:12:26.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.120+0000 7f6d55600640 1 -- 172.21.15.28:0/338113630 <== mon.2 v2:172.21.15.86:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7f6d640cf070 con 0x55c78924fb50 2024-05-12T22:12:26.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.120+0000 7f6d75782740 10 client.26072 fetch_fsmap finished waiting for FSMap version 25 2024-05-12T22:12:26.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.120+0000 7f6d75782740 10 client.26072 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-05-12T22:12:26.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.120+0000 7f6d75782740 20 client.26072 populate_metadata read hostname 'smithi028' 2024-05-12T22:12:26.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.120+0000 7f6d75782740 1 --2- 172.21.15.28:0/338113630 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x55c78925ffb0 0x55c7892623f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:26.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.120+0000 7f6d75782740 4 client.26072 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-12T22:12:26.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.120+0000 7f6d75782740 1 -- 172.21.15.28:0/338113630 --> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c7891b7160 con 0x55c78925ffb0 2024-05-12T22:12:26.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.121+0000 7f6d6b400640 1 --2- 172.21.15.28:0/338113630 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x55c78925ffb0 0x55c7892623f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:26.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.121+0000 7f6d6b400640 1 --2- 172.21.15.28:0/338113630 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x55c78925ffb0 0x55c7892623f0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=657b8d260ef2ef53 server_cookie=24043cba144d6402 in_seq=0 out_seq=0 2024-05-12T22:12:26.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.121+0000 7f6d55600640 10 client.26072 ms_handle_connect on v2:172.21.15.28:6840/761183870 2024-05-12T22:12:26.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.122+0000 7f6d55600640 1 -- 172.21.15.28:0/338113630 <== mds.1 v2:172.21.15.28:6840/761183870 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c7891b7160 con 0x55c78925ffb0 2024-05-12T22:12:26.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.122+0000 7f6d55600640 10 client.26072 handle_command_reply: tid=0 2024-05-12T22:12:26.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.122+0000 7f6d75782740 1 client.26072 shutdown 2024-05-12T22:12:26.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.122+0000 7f6d75782740 1 -- 172.21.15.28:0/338113630 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c78924fb50 msgr2=0x55c78924d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:12:26.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.122+0000 7f6d75782740 1 --2- 172.21.15.28:0/338113630 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c78924fb50 0x55c78924d4f0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f6d640ccb40 tx=0x7f6d6409b270 comp rx=0 tx=0).stop 2024-05-12T22:12:26.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.122+0000 7f6d75782740 1 -- 172.21.15.28:0/338113630 shutdown_connections 2024-05-12T22:12:26.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.122+0000 7f6d75782740 1 --2- 172.21.15.28:0/338113630 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x55c78925ffb0 0x55c7892623f0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:26.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.122+0000 7f6d75782740 1 --2- 172.21.15.28:0/338113630 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c78924fb50 0x55c78924d4f0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:26.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.122+0000 7f6d75782740 1 --2- 172.21.15.28:0/338113630 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c789245480 0x55c78924cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:26.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.122+0000 7f6d55600640 0 client.26072 ms_handle_reset on v2:172.21.15.28:6840/761183870 2024-05-12T22:12:26.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.122+0000 7f6d75782740 1 --2- 172.21.15.28:0/338113630 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c789244b60 0x55c78924ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:26.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.122+0000 7f6d75782740 1 -- 172.21.15.28:0/338113630 >> 172.21.15.28:0/338113630 conn(0x55c789240e80 msgr2=0x55c789246b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:12:26.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.122+0000 7f6d75782740 1 -- 172.21.15.28:0/338113630 shutdown_connections 2024-05-12T22:12:26.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.122+0000 7f6d75782740 1 -- 172.21.15.28:0/338113630 wait complete. 2024-05-12T22:12:26.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.123+0000 7f6d75782740 20 client.26072 trim_cache size 0 max 16384 2024-05-12T22:12:26.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.138+0000 7f6d75782740 1 Processor -- start 2024-05-12T22:12:26.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.139+0000 7f6d75782740 1 -- start start 2024-05-12T22:12:26.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.139+0000 7f6d75782740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c789244b60 0x55c7892472c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:26.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.139+0000 7f6d75782740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c789245480 0x55c78924d310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:26.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.139+0000 7f6d75782740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c78924fb50 0x55c789267cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:26.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.139+0000 7f6d75782740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6d6c1b6080 con 0x55c789244b60 2024-05-12T22:12:26.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.139+0000 7f6d75782740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f6d6c1b3950 con 0x55c78924fb50 2024-05-12T22:12:26.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.139+0000 7f6d75782740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f6d6c1b17a0 con 0x55c789245480 2024-05-12T22:12:26.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.139+0000 7f6d6b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c789244b60 0x55c7892472c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:26.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.139+0000 7f6d6aa00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c789245480 0x55c78924d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:26.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.139+0000 7f6d6b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c789244b60 0x55c7892472c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39572/0 (socket says 172.21.15.28:39572) 2024-05-12T22:12:26.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.139+0000 7f6d6b400640 1 -- 172.21.15.28:0/3342892537 learned_addr learned my addr 172.21.15.28:0/3342892537 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:12:26.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.139+0000 7f6d6be00640 1 --2- 172.21.15.28:0/3342892537 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c78924fb50 0x55c789267cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:26.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.140+0000 7f6d6aa00640 1 -- 172.21.15.28:0/3342892537 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c78924fb50 msgr2=0x55c789267cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:12:26.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.140+0000 7f6d6aa00640 1 --2- 172.21.15.28:0/3342892537 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c78924fb50 0x55c789267cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:26.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.140+0000 7f6d6aa00640 1 -- 172.21.15.28:0/3342892537 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c789244b60 msgr2=0x55c7892472c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:12:26.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.140+0000 7f6d6aa00640 1 --2- 172.21.15.28:0/3342892537 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c789244b60 0x55c7892472c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:26.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.140+0000 7f6d6aa00640 1 -- 172.21.15.28:0/3342892537 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c7892459d0 con 0x55c789245480 2024-05-12T22:12:26.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.140+0000 7f6d6b400640 1 --2- 172.21.15.28:0/3342892537 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c789244b60 0x55c7892472c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:12:26.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.140+0000 7f6d6aa00640 1 --2- 172.21.15.28:0/3342892537 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c789245480 0x55c78924d310 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f6d5800f790 tx=0x7f6d580095d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7f3f0f738d249e4d server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:12:26.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.140+0000 7f6d57e00640 1 -- 172.21.15.28:0/3342892537 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d58013170 con 0x55c789245480 2024-05-12T22:12:26.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.141+0000 7f6d57e00640 1 -- 172.21.15.28:0/3342892537 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6d5800fda0 con 0x55c789245480 2024-05-12T22:12:26.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.141+0000 7f6d57e00640 1 -- 172.21.15.28:0/3342892537 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d580048e0 con 0x55c789245480 2024-05-12T22:12:26.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.141+0000 7f6d75782740 1 -- 172.21.15.28:0/3342892537 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c789245480 msgr2=0x55c78924d310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:12:26.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.141+0000 7f6d75782740 1 --2- 172.21.15.28:0/3342892537 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c789245480 0x55c78924d310 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f6d5800f790 tx=0x7f6d580095d0 comp rx=0 tx=0).stop 2024-05-12T22:12:26.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.141+0000 7f6d75782740 1 -- 172.21.15.28:0/3342892537 shutdown_connections 2024-05-12T22:12:26.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.141+0000 7f6d75782740 1 --2- 172.21.15.28:0/3342892537 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c78924fb50 0x55c789267cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:26.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.141+0000 7f6d75782740 1 --2- 172.21.15.28:0/3342892537 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c789245480 0x55c78924d310 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:26.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.141+0000 7f6d75782740 1 --2- 172.21.15.28:0/3342892537 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c789244b60 0x55c7892472c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:26.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.141+0000 7f6d75782740 1 -- 172.21.15.28:0/3342892537 >> 172.21.15.28:0/3342892537 conn(0x55c789240e80 msgr2=0x55c789240000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:12:26.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.141+0000 7f6d75782740 1 -- 172.21.15.28:0/3342892537 shutdown_connections 2024-05-12T22:12:26.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.142+0000 7f6d75782740 1 -- 172.21.15.28:0/3342892537 wait complete. 2024-05-12T22:12:26.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.142+0000 7f6d75782740 1 -- start start 2024-05-12T22:12:26.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.142+0000 7f6d75782740 1 Processor -- start 2024-05-12T22:12:26.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.143+0000 7f6d75782740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c789244b60 0x55c789258130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:26.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.143+0000 7f6d75782740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c789245480 0x55c789249490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:26.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.143+0000 7f6d75782740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c78924fb50 0x55c7892499d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:26.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.143+0000 7f6d75782740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6d6c1b1e10 con 0x55c78924fb50 2024-05-12T22:12:26.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.143+0000 7f6d75782740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f6d6c1b3280 con 0x55c789244b60 2024-05-12T22:12:26.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.143+0000 7f6d75782740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f6d6c1b24e0 con 0x55c789245480 2024-05-12T22:12:26.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.143+0000 7f6d75782740 10 client.0 _ll_register_callbacks cb 0x7f6d6409b7d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:12:26.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.143+0000 7f6d6be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c78924fb50 0x55c7892499d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:26.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.143+0000 7f6d6aa00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c789245480 0x55c789249490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:26.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.143+0000 7f6d6be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c78924fb50 0x55c7892499d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39582/0 (socket says 172.21.15.28:39582) 2024-05-12T22:12:26.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.143+0000 7f6d6be00640 1 -- 172.21.15.28:0/464803665 learned_addr learned my addr 172.21.15.28:0/464803665 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:12:26.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.143+0000 7f6d6b400640 1 --2- 172.21.15.28:0/464803665 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c789244b60 0x55c789258130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:26.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.143+0000 7f6d6aa00640 1 -- 172.21.15.28:0/464803665 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c789244b60 msgr2=0x55c789258130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:12:26.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.143+0000 7f6d6aa00640 1 --2- 172.21.15.28:0/464803665 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c789244b60 0x55c789258130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:26.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.143+0000 7f6d6aa00640 1 -- 172.21.15.28:0/464803665 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c78924fb50 msgr2=0x55c7892499d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:12:26.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.143+0000 7f6d6aa00640 1 --2- 172.21.15.28:0/464803665 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c78924fb50 0x55c7892499d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:26.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.143+0000 7f6d6aa00640 1 -- 172.21.15.28:0/464803665 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c78924a200 con 0x55c789245480 2024-05-12T22:12:26.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.143+0000 7f6d6be00640 1 --2- 172.21.15.28:0/464803665 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c78924fb50 0x55c7892499d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:12:26.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.143+0000 7f6d6b400640 1 --2- 172.21.15.28:0/464803665 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c789244b60 0x55c789258130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:12:26.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.144+0000 7f6d6aa00640 1 --2- 172.21.15.28:0/464803665 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c789245480 0x55c789249490 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x55c78924dcd0 tx=0x7f6d58005b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f27f11309e8a9465 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:12:26.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.144+0000 7f6d55600640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-12T22:12:26.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.144+0000 7f6d55600640 1 -- 172.21.15.28:0/464803665 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d5800f9c0 con 0x55c789245480 2024-05-12T22:12:26.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.144+0000 7f6d75782740 1 -- 172.21.15.28:0/464803665 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6d5800fda0 con 0x55c789245480 2024-05-12T22:12:26.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.144+0000 7f6d55600640 1 -- 172.21.15.28:0/464803665 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f6d5800fb60 con 0x55c789245480 2024-05-12T22:12:26.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.144+0000 7f6d55600640 1 -- 172.21.15.28:0/464803665 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f6d5803a050 con 0x55c789245480 2024-05-12T22:12:26.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.144+0000 7f6d75782740 10 client.26032 fetch_fsmap learned FSMap version 25 2024-05-12T22:12:26.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.144+0000 7f6d75782740 1 -- 172.21.15.28:0/464803665 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55c7892c6560 con 0x55c789245480 2024-05-12T22:12:26.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.145+0000 7f6d55600640 1 -- 172.21.15.28:0/464803665 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6d5800fda0 con 0x55c789245480 2024-05-12T22:12:26.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.145+0000 7f6d55600640 1 -- 172.21.15.28:0/464803665 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7f6d5801d020 con 0x55c789245480 2024-05-12T22:12:26.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.145+0000 7f6d75782740 10 client.26032 fetch_fsmap finished waiting for FSMap version 25 2024-05-12T22:12:26.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.145+0000 7f6d75782740 10 client.26032 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-05-12T22:12:26.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.145+0000 7f6d75782740 20 client.26032 populate_metadata read hostname 'smithi028' 2024-05-12T22:12:26.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.145+0000 7f6d75782740 1 --2- 172.21.15.28:0/464803665 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x55c7892c6a30 0x55c7892c8e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:26.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.145+0000 7f6d75782740 4 client.26032 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-12T22:12:26.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.145+0000 7f6d75782740 1 -- 172.21.15.28:0/464803665 --> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c7891b14b0 con 0x55c7892c6a30 2024-05-12T22:12:26.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.145+0000 7f6d6b400640 1 --2- 172.21.15.28:0/464803665 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x55c7892c6a30 0x55c7892c8e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:26.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.146+0000 7f6d6b400640 1 --2- 172.21.15.28:0/464803665 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x55c7892c6a30 0x55c7892c8e20 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=dba1b1f0bacf3471 server_cookie=4548fabedf69443b in_seq=0 out_seq=0 2024-05-12T22:12:26.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.146+0000 7f6d55600640 10 client.26032 ms_handle_connect on v2:172.21.15.28:6840/761183870 2024-05-12T22:12:26.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.146+0000 7f6d55600640 1 -- 172.21.15.28:0/464803665 <== mds.1 v2:172.21.15.28:6840/761183870 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c7891b14b0 con 0x55c7892c6a30 2024-05-12T22:12:26.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.146+0000 7f6d55600640 10 client.26032 handle_command_reply: tid=0 2024-05-12T22:12:26.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.146+0000 7f6d75782740 1 client.26032 shutdown 2024-05-12T22:12:26.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.146+0000 7f6d75782740 1 -- 172.21.15.28:0/464803665 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c789245480 msgr2=0x55c789249490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:12:26.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.146+0000 7f6d75782740 1 --2- 172.21.15.28:0/464803665 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c789245480 0x55c789249490 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x55c78924dcd0 tx=0x7f6d58005b20 comp rx=0 tx=0).stop 2024-05-12T22:12:26.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.147+0000 7f6d75782740 1 -- 172.21.15.28:0/464803665 shutdown_connections 2024-05-12T22:12:26.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.147+0000 7f6d75782740 1 --2- 172.21.15.28:0/464803665 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x55c7892c6a30 0x55c7892c8e20 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:26.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.147+0000 7f6d75782740 1 --2- 172.21.15.28:0/464803665 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c78924fb50 0x55c7892499d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:26.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.147+0000 7f6d75782740 1 --2- 172.21.15.28:0/464803665 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c789245480 0x55c789249490 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:26.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.147+0000 7f6d75782740 1 --2- 172.21.15.28:0/464803665 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c789244b60 0x55c789258130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:26.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.147+0000 7f6d55600640 0 client.26032 ms_handle_reset on v2:172.21.15.28:6840/761183870 2024-05-12T22:12:26.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.147+0000 7f6d75782740 1 -- 172.21.15.28:0/464803665 >> 172.21.15.28:0/464803665 conn(0x55c789240e80 msgr2=0x55c78924e2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:12:26.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.147+0000 7f6d75782740 1 -- 172.21.15.28:0/464803665 shutdown_connections 2024-05-12T22:12:26.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.147+0000 7f6d75782740 1 -- 172.21.15.28:0/464803665 wait complete. 2024-05-12T22:12:26.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.147+0000 7f6d75782740 20 client.26032 trim_cache size 0 max 16384 2024-05-12T22:12:26.157 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-05-12T22:12:26.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.147+0000 7f6d71200640 1 -- 172.21.15.28:0/4097248452 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f6d500a2d00 msgr2=0x7f6d500a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:12:26.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.147+0000 7f6d71200640 1 --2- 172.21.15.28:0/4097248452 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f6d500a2d00 0x7f6d500a51c0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f6d5c000c00 tx=0x7f6d5c031510 comp rx=0 tx=0).stop 2024-05-12T22:12:26.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.147+0000 7f6d71200640 1 -- 172.21.15.28:0/4097248452 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d6c10ddf0 msgr2=0x7f6d6c1a2e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:12:26.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.147+0000 7f6d71200640 1 --2- 172.21.15.28:0/4097248452 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d6c10ddf0 0x7f6d6c1a2e40 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f6d6400d9c0 tx=0x7f6d6400de80 comp rx=0 tx=0).stop 2024-05-12T22:12:26.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.148+0000 7f6d71200640 1 -- 172.21.15.28:0/4097248452 shutdown_connections 2024-05-12T22:12:26.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.148+0000 7f6d71200640 1 --2- 172.21.15.28:0/4097248452 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f6d500a2d00 0x7f6d500a51c0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:26.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.148+0000 7f6d71200640 1 --2- 172.21.15.28:0/4097248452 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d6c10ddf0 0x7f6d6c1a2e40 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:26.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.148+0000 7f6d71200640 1 --2- 172.21.15.28:0/4097248452 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d6c108770 0x7f6d6c19eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:26.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.148+0000 7f6d71200640 1 --2- 172.21.15.28:0/4097248452 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6d6c107dd0 0x7f6d6c19e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:26.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.148+0000 7f6d71200640 1 -- 172.21.15.28:0/4097248452 >> 172.21.15.28:0/4097248452 conn(0x7f6d6c077650 msgr2=0x7f6d6c0786a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:12:26.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.148+0000 7f6d71200640 1 -- 172.21.15.28:0/4097248452 shutdown_connections 2024-05-12T22:12:26.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.148+0000 7f6d71200640 1 -- 172.21.15.28:0/4097248452 wait complete. 2024-05-12T22:12:26.164 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:12:26 smithi028 ceph-mon[29847]: pgmap v106: 97 pgs: 97 active+clean; 97 MiB data, 752 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 4.1 KiB/s wr, 8 op/s 2024-05-12T22:12:26.164 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:12:26 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/3027396790' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:12:26.168 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-05-12T22:12:26.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.373+0000 7f85bbe00640 1 -- 172.21.15.28:0/3810732146 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f85bc1126f0 msgr2=0x7f85bc114ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:12:26.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.373+0000 7f85bbe00640 1 --2- 172.21.15.28:0/3810732146 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f85bc1126f0 0x7f85bc114ae0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f85b4009ca0 tx=0x7f85b402a950 comp rx=0 tx=0).stop 2024-05-12T22:12:26.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.373+0000 7f85bbe00640 1 -- 172.21.15.28:0/3810732146 shutdown_connections 2024-05-12T22:12:26.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.373+0000 7f85bbe00640 1 --2- 172.21.15.28:0/3810732146 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f85bc1126f0 0x7f85bc114ae0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:26.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.373+0000 7f85bbe00640 1 --2- 172.21.15.28:0/3810732146 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85bc0744f0 0x7f85bc1121b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:26.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.373+0000 7f85bbe00640 1 --2- 172.21.15.28:0/3810732146 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f85bc073bd0 0x7f85bc073fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:26.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.373+0000 7f85bbe00640 1 -- 172.21.15.28:0/3810732146 >> 172.21.15.28:0/3810732146 conn(0x7f85bc0fd1d0 msgr2=0x7f85bc0ff5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:12:26.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.373+0000 7f85bbe00640 1 -- 172.21.15.28:0/3810732146 shutdown_connections 2024-05-12T22:12:26.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.373+0000 7f85bbe00640 1 -- 172.21.15.28:0/3810732146 wait complete. 2024-05-12T22:12:26.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.374+0000 7f85bbe00640 1 Processor -- start 2024-05-12T22:12:26.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.375+0000 7f85bbe00640 1 -- start start 2024-05-12T22:12:26.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.375+0000 7f85bbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85bc073bd0 0x7f85bc19e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:26.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.375+0000 7f85bbe00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f85bc0744f0 0x7f85bc19eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:26.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.375+0000 7f85bbe00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f85bc1126f0 0x7f85bc1a2f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:26.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.375+0000 7f85bbe00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f85bc118090 con 0x7f85bc073bd0 2024-05-12T22:12:26.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.375+0000 7f85bbe00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f85bc117f10 con 0x7f85bc0744f0 2024-05-12T22:12:26.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.375+0000 7f85bbe00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f85bc118210 con 0x7f85bc1126f0 2024-05-12T22:12:26.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.375+0000 7f85baa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85bc073bd0 0x7f85bc19e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:26.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.375+0000 7f85ba000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f85bc0744f0 0x7f85bc19eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:26.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.375+0000 7f85baa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85bc073bd0 0x7f85bc19e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39604/0 (socket says 172.21.15.28:39604) 2024-05-12T22:12:26.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.375+0000 7f85baa00640 1 -- 172.21.15.28:0/1525753705 learned_addr learned my addr 172.21.15.28:0/1525753705 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:12:26.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.375+0000 7f85bb400640 1 --2- 172.21.15.28:0/1525753705 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f85bc1126f0 0x7f85bc1a2f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:26.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.376+0000 7f85ba000640 1 -- 172.21.15.28:0/1525753705 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f85bc1126f0 msgr2=0x7f85bc1a2f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:12:26.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.376+0000 7f85ba000640 1 --2- 172.21.15.28:0/1525753705 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f85bc1126f0 0x7f85bc1a2f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:26.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.376+0000 7f85ba000640 1 -- 172.21.15.28:0/1525753705 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85bc073bd0 msgr2=0x7f85bc19e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:12:26.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.376+0000 7f85ba000640 1 --2- 172.21.15.28:0/1525753705 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85bc073bd0 0x7f85bc19e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:26.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.376+0000 7f85ba000640 1 -- 172.21.15.28:0/1525753705 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85bc1a3640 con 0x7f85bc0744f0 2024-05-12T22:12:26.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.376+0000 7f85ba000640 1 --2- 172.21.15.28:0/1525753705 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f85bc0744f0 0x7f85bc19eb90 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f85a800e9f0 tx=0x7f85a800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:12:26.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.376+0000 7f85b3400640 1 -- 172.21.15.28:0/1525753705 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85a800ccc0 con 0x7f85bc0744f0 2024-05-12T22:12:26.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.377+0000 7f85b3400640 1 -- 172.21.15.28:0/1525753705 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f85a8010910 con 0x7f85bc0744f0 2024-05-12T22:12:26.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.377+0000 7f85b3400640 1 -- 172.21.15.28:0/1525753705 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85a8018960 con 0x7f85bc0744f0 2024-05-12T22:12:26.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.377+0000 7f85bbe00640 1 -- 172.21.15.28:0/1525753705 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85bc1a3930 con 0x7f85bc0744f0 2024-05-12T22:12:26.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.377+0000 7f85bbe00640 1 -- 172.21.15.28:0/1525753705 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85bc1ab1c0 con 0x7f85bc0744f0 2024-05-12T22:12:26.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.380+0000 7f85c4683740 1 Processor -- start 2024-05-12T22:12:26.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.380+0000 7f85c4683740 1 -- start start 2024-05-12T22:12:26.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.380+0000 7f85c4683740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5573df880b60 0x5573df880f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:26.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.380+0000 7f85b3400640 1 -- 172.21.15.28:0/1525753705 <== mon.2 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7f85a8010ac0 con 0x7f85bc0744f0 2024-05-12T22:12:26.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.380+0000 7f85c4683740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5573df881480 0x5573df88b610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:26.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.380+0000 7f85c4683740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5573df88bb50 0x5573df88df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:26.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.381+0000 7f85c4683740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5573df7e5a30 con 0x5573df881480 2024-05-12T22:12:26.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.381+0000 7f85c4683740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x5573df78d700 con 0x5573df88bb50 2024-05-12T22:12:26.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.381+0000 7f85c4683740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5573df7e19f0 con 0x5573df880b60 2024-05-12T22:12:26.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.381+0000 7f85baa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5573df881480 0x5573df88b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:26.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.381+0000 7f85baa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5573df881480 0x5573df88b610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39620/0 (socket says 172.21.15.28:39620) 2024-05-12T22:12:26.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.381+0000 7f85baa00640 1 -- 172.21.15.28:0/524158105 learned_addr learned my addr 172.21.15.28:0/524158105 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:12:26.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.381+0000 7f85bb400640 1 --2- 172.21.15.28:0/524158105 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5573df880b60 0x5573df880f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:26.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.381+0000 7f85ba000640 1 --2- 172.21.15.28:0/524158105 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5573df88bb50 0x5573df88df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:26.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.381+0000 7f85baa00640 1 -- 172.21.15.28:0/524158105 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5573df880b60 msgr2=0x5573df880f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:12:26.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.381+0000 7f85baa00640 1 --2- 172.21.15.28:0/524158105 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5573df880b60 0x5573df880f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:26.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.381+0000 7f85baa00640 1 -- 172.21.15.28:0/524158105 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5573df88bb50 msgr2=0x5573df88df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:12:26.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.381+0000 7f85baa00640 1 --2- 172.21.15.28:0/524158105 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5573df88bb50 0x5573df88df40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:26.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.381+0000 7f85baa00640 1 -- 172.21.15.28:0/524158105 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5573df4f93f0 con 0x5573df881480 2024-05-12T22:12:26.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.382+0000 7f85baa00640 1 --2- 172.21.15.28:0/524158105 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5573df881480 0x5573df88b610 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f85ac00bda0 tx=0x7f85ac0107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=307cb8805677cbae server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:12:26.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.382+0000 7f85a7e00640 1 -- 172.21.15.28:0/524158105 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85ac01c680 con 0x5573df881480 2024-05-12T22:12:26.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.382+0000 7f85a7e00640 1 -- 172.21.15.28:0/524158105 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f85ac011920 con 0x5573df881480 2024-05-12T22:12:26.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.382+0000 7f85b3400640 1 --2- 172.21.15.28:0/1525753705 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f85a00a2d80 0x7f85a00a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:26.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.382+0000 7f85a7e00640 1 -- 172.21.15.28:0/524158105 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85ac01a8d0 con 0x5573df881480 2024-05-12T22:12:26.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.382+0000 7f85b3400640 1 -- 172.21.15.28:0/1525753705 <== mon.2 v2:172.21.15.86:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f85a80cf6b0 con 0x7f85bc0744f0 2024-05-12T22:12:26.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.382+0000 7f85bb400640 1 --2- 172.21.15.28:0/1525753705 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f85a00a2d80 0x7f85a00a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:26.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.383+0000 7f85c4683740 1 -- 172.21.15.28:0/524158105 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5573df881480 msgr2=0x5573df88b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:12:26.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.383+0000 7f85c4683740 1 --2- 172.21.15.28:0/524158105 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5573df881480 0x5573df88b610 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f85ac00bda0 tx=0x7f85ac0107a0 comp rx=0 tx=0).stop 2024-05-12T22:12:26.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.383+0000 7f85c4683740 1 -- 172.21.15.28:0/524158105 shutdown_connections 2024-05-12T22:12:26.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.383+0000 7f85c4683740 1 --2- 172.21.15.28:0/524158105 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5573df88bb50 0x5573df88df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:26.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.383+0000 7f85c4683740 1 --2- 172.21.15.28:0/524158105 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5573df881480 0x5573df88b610 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:26.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.383+0000 7f85c4683740 1 --2- 172.21.15.28:0/524158105 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5573df880b60 0x5573df880f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:26.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.383+0000 7f85c4683740 1 -- 172.21.15.28:0/524158105 >> 172.21.15.28:0/524158105 conn(0x5573df87ce80 msgr2=0x5573df87d260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:12:26.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.383+0000 7f85c4683740 1 -- 172.21.15.28:0/524158105 shutdown_connections 2024-05-12T22:12:26.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.383+0000 7f85c4683740 1 -- 172.21.15.28:0/524158105 wait complete. 2024-05-12T22:12:26.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.383+0000 7f85bb400640 1 --2- 172.21.15.28:0/1525753705 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f85a00a2d80 0x7f85a00a5240 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f85b4005e30 tx=0x7f85b4005c50 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:12:26.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.384+0000 7f85c4683740 1 -- start start 2024-05-12T22:12:26.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.384+0000 7f85c4683740 1 Processor -- start 2024-05-12T22:12:26.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.384+0000 7f85c4683740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5573df880b60 0x5573df888a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:26.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.384+0000 7f85c4683740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5573df881480 0x5573df888f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:26.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.384+0000 7f85c4683740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5573df88bb50 0x5573df889490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:26.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.384+0000 7f85c4683740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5573df88e940 con 0x5573df881480 2024-05-12T22:12:26.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.384+0000 7f85c4683740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x5573df7a0a70 con 0x5573df88bb50 2024-05-12T22:12:26.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.384+0000 7f85c4683740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5573df88eac0 con 0x5573df880b60 2024-05-12T22:12:26.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.384+0000 7f85c4683740 10 client.0 _ll_register_callbacks cb 0x5573df783ba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:12:26.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.385+0000 7f85ba000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5573df881480 0x5573df888f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:26.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.385+0000 7f85ba000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5573df881480 0x5573df888f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39624/0 (socket says 172.21.15.28:39624) 2024-05-12T22:12:26.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.385+0000 7f85ba000640 1 -- 172.21.15.28:0/2735587746 learned_addr learned my addr 172.21.15.28:0/2735587746 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:12:26.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.385+0000 7f85bb400640 1 --2- 172.21.15.28:0/2735587746 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5573df88bb50 0x5573df889490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:26.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.385+0000 7f85baa00640 1 --2- 172.21.15.28:0/2735587746 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5573df880b60 0x5573df888a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:26.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.385+0000 7f85ba000640 1 -- 172.21.15.28:0/2735587746 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5573df880b60 msgr2=0x5573df888a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:12:26.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.385+0000 7f85ba000640 1 --2- 172.21.15.28:0/2735587746 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5573df880b60 0x5573df888a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:26.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.385+0000 7f85ba000640 1 -- 172.21.15.28:0/2735587746 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5573df88bb50 msgr2=0x5573df889490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:12:26.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.385+0000 7f85ba000640 1 --2- 172.21.15.28:0/2735587746 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5573df88bb50 0x5573df889490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:26.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.385+0000 7f85ba000640 1 -- 172.21.15.28:0/2735587746 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5573df5c37d0 con 0x5573df881480 2024-05-12T22:12:26.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.386+0000 7f85ba000640 1 --2- 172.21.15.28:0/2735587746 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5573df881480 0x5573df888f50 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f85a800c6a0 tx=0x7f85a8010e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a45887b6cf16545 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:12:26.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.386+0000 7f85a4c00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-12T22:12:26.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.386+0000 7f85a4c00640 1 -- 172.21.15.28:0/2735587746 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85a809ba80 con 0x5573df881480 2024-05-12T22:12:26.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.386+0000 7f85a4c00640 1 -- 172.21.15.28:0/2735587746 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f85a80cfd60 con 0x5573df881480 2024-05-12T22:12:26.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.386+0000 7f85c4683740 1 -- 172.21.15.28:0/2735587746 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f85ac011920 con 0x5573df881480 2024-05-12T22:12:26.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.386+0000 7f85a4c00640 1 -- 172.21.15.28:0/2735587746 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f85a809b610 con 0x5573df881480 2024-05-12T22:12:26.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.386+0000 7f85c4683740 10 client.15963 fetch_fsmap learned FSMap version 25 2024-05-12T22:12:26.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.386+0000 7f85c4683740 1 -- 172.21.15.28:0/2735587746 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5573df89ab40 con 0x5573df881480 2024-05-12T22:12:26.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.386+0000 7f85a4c00640 1 -- 172.21.15.28:0/2735587746 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f85a80b2070 con 0x5573df881480 2024-05-12T22:12:26.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.387+0000 7f85a4c00640 1 -- 172.21.15.28:0/2735587746 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7f85a8018b00 con 0x5573df881480 2024-05-12T22:12:26.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.387+0000 7f85c4683740 10 client.15963 fetch_fsmap finished waiting for FSMap version 25 2024-05-12T22:12:26.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.387+0000 7f85c4683740 10 client.15963 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:12:26.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.387+0000 7f85c4683740 20 client.15963 populate_metadata read hostname 'smithi028' 2024-05-12T22:12:26.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.387+0000 7f85c4683740 1 --2- 172.21.15.28:0/2735587746 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5573df89bf50 0x5573df89e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:26.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.387+0000 7f85c4683740 4 client.15963 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-12T22:12:26.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.387+0000 7f85c4683740 1 -- 172.21.15.28:0/2735587746 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5573df7f3160 con 0x5573df89bf50 2024-05-12T22:12:26.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.387+0000 7f85baa00640 1 --2- 172.21.15.28:0/2735587746 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5573df89bf50 0x5573df89e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:26.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.388+0000 7f85baa00640 1 --2- 172.21.15.28:0/2735587746 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5573df89bf50 0x5573df89e3a0 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5e354d3e81f0b6cc server_cookie=d90b8d5926af177f in_seq=0 out_seq=0 2024-05-12T22:12:26.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.388+0000 7f85a4c00640 10 client.15963 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:12:26.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.389+0000 7f85a4c00640 1 -- 172.21.15.28:0/2735587746 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5573df7f3160 con 0x5573df89bf50 2024-05-12T22:12:26.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.389+0000 7f85a4c00640 10 client.15963 handle_command_reply: tid=0 2024-05-12T22:12:26.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.389+0000 7f85c4683740 1 client.15963 shutdown 2024-05-12T22:12:26.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.389+0000 7f85c4683740 1 -- 172.21.15.28:0/2735587746 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5573df881480 msgr2=0x5573df888f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:12:26.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.389+0000 7f85c4683740 1 --2- 172.21.15.28:0/2735587746 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5573df881480 0x5573df888f50 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f85a800c6a0 tx=0x7f85a8010e30 comp rx=0 tx=0).stop 2024-05-12T22:12:26.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.389+0000 7f85c4683740 1 -- 172.21.15.28:0/2735587746 shutdown_connections 2024-05-12T22:12:26.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.389+0000 7f85c4683740 1 --2- 172.21.15.28:0/2735587746 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5573df89bf50 0x5573df89e3a0 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:26.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.389+0000 7f85c4683740 1 --2- 172.21.15.28:0/2735587746 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5573df88bb50 0x5573df889490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:26.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.389+0000 7f85c4683740 1 --2- 172.21.15.28:0/2735587746 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5573df881480 0x5573df888f50 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:26.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.389+0000 7f85c4683740 1 --2- 172.21.15.28:0/2735587746 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5573df880b60 0x5573df888a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:26.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.389+0000 7f85a4c00640 0 client.15963 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:12:26.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.389+0000 7f85c4683740 1 -- 172.21.15.28:0/2735587746 >> 172.21.15.28:0/2735587746 conn(0x5573df87ce80 msgr2=0x5573df87c000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:12:26.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.389+0000 7f85c4683740 1 -- 172.21.15.28:0/2735587746 shutdown_connections 2024-05-12T22:12:26.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.389+0000 7f85c4683740 1 -- 172.21.15.28:0/2735587746 wait complete. 2024-05-12T22:12:26.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.390+0000 7f85c4683740 20 client.15963 trim_cache size 0 max 16384 2024-05-12T22:12:26.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.405+0000 7f85c4683740 1 Processor -- start 2024-05-12T22:12:26.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.405+0000 7f85c4683740 1 -- start start 2024-05-12T22:12:26.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.405+0000 7f85c4683740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5573df880b60 0x5573df897830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:26.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.405+0000 7f85c4683740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5573df881480 0x5573df87be80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:26.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.405+0000 7f85c4683740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5573df88bb50 0x5573df8a3c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:26.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.405+0000 7f85c4683740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f85bc1b62d0 con 0x5573df880b60 2024-05-12T22:12:26.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.405+0000 7f85c4683740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f85bc1b3ba0 con 0x5573df881480 2024-05-12T22:12:26.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.405+0000 7f85c4683740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f85bc1b19f0 con 0x5573df88bb50 2024-05-12T22:12:26.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.405+0000 7f85baa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5573df880b60 0x5573df897830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:26.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.405+0000 7f85baa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5573df880b60 0x5573df897830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39628/0 (socket says 172.21.15.28:39628) 2024-05-12T22:12:26.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.405+0000 7f85baa00640 1 -- 172.21.15.28:0/1152556972 learned_addr learned my addr 172.21.15.28:0/1152556972 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:12:26.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.405+0000 7f85bb400640 1 --2- 172.21.15.28:0/1152556972 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5573df88bb50 0x5573df8a3c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:26.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.406+0000 7f85ba000640 1 --2- 172.21.15.28:0/1152556972 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5573df881480 0x5573df87be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:26.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.406+0000 7f85baa00640 1 -- 172.21.15.28:0/1152556972 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5573df88bb50 msgr2=0x5573df8a3c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:12:26.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.406+0000 7f85baa00640 1 --2- 172.21.15.28:0/1152556972 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5573df88bb50 0x5573df8a3c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:26.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.406+0000 7f85baa00640 1 -- 172.21.15.28:0/1152556972 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5573df881480 msgr2=0x5573df87be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:12:26.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.406+0000 7f85baa00640 1 --2- 172.21.15.28:0/1152556972 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5573df881480 0x5573df87be80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:26.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.406+0000 7f85baa00640 1 -- 172.21.15.28:0/1152556972 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5573df8845e0 con 0x5573df880b60 2024-05-12T22:12:26.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.406+0000 7f85ba000640 1 --2- 172.21.15.28:0/1152556972 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5573df881480 0x5573df87be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:12:26.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.406+0000 7f85baa00640 1 --2- 172.21.15.28:0/1152556972 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5573df880b60 0x5573df897830 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f85ac00bda0 tx=0x7f85ac010fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f90fbcc70c4294c server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:12:26.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.407+0000 7f85a7400640 1 -- 172.21.15.28:0/1152556972 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85ac011dd0 con 0x5573df880b60 2024-05-12T22:12:26.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.407+0000 7f85a7400640 1 -- 172.21.15.28:0/1152556972 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f85ac02b460 con 0x5573df880b60 2024-05-12T22:12:26.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.407+0000 7f85a7400640 1 -- 172.21.15.28:0/1152556972 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85ac005480 con 0x5573df880b60 2024-05-12T22:12:26.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.407+0000 7f85c4683740 1 -- 172.21.15.28:0/1152556972 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5573df880b60 msgr2=0x5573df897830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:12:26.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.407+0000 7f85c4683740 1 --2- 172.21.15.28:0/1152556972 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5573df880b60 0x5573df897830 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f85ac00bda0 tx=0x7f85ac010fd0 comp rx=0 tx=0).stop 2024-05-12T22:12:26.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.407+0000 7f85c4683740 1 -- 172.21.15.28:0/1152556972 shutdown_connections 2024-05-12T22:12:26.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.407+0000 7f85c4683740 1 --2- 172.21.15.28:0/1152556972 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5573df88bb50 0x5573df8a3c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:26.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.407+0000 7f85c4683740 1 --2- 172.21.15.28:0/1152556972 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5573df881480 0x5573df87be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:26.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.407+0000 7f85c4683740 1 --2- 172.21.15.28:0/1152556972 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5573df880b60 0x5573df897830 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:26.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.407+0000 7f85c4683740 1 -- 172.21.15.28:0/1152556972 >> 172.21.15.28:0/1152556972 conn(0x5573df87ce80 msgr2=0x5573df87b770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:12:26.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.408+0000 7f85c4683740 1 -- 172.21.15.28:0/1152556972 shutdown_connections 2024-05-12T22:12:26.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.408+0000 7f85c4683740 1 -- 172.21.15.28:0/1152556972 wait complete. 2024-05-12T22:12:26.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.408+0000 7f85c4683740 1 -- start start 2024-05-12T22:12:26.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.408+0000 7f85c4683740 1 Processor -- start 2024-05-12T22:12:26.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.409+0000 7f85c4683740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5573df880b60 0x5573df88f980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:26.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.409+0000 7f85c4683740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5573df881480 0x5573df88fec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:26.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.409+0000 7f85c4683740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5573df88bb50 0x5573df890400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:26.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.409+0000 7f85c4683740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f85bc1b2060 con 0x5573df880b60 2024-05-12T22:12:26.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.409+0000 7f85c4683740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f85bc1b34d0 con 0x5573df881480 2024-05-12T22:12:26.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.409+0000 7f85c4683740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f85bc1b2730 con 0x5573df88bb50 2024-05-12T22:12:26.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.409+0000 7f85c4683740 10 client.0 _ll_register_callbacks cb 0x7f85a800ce60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:12:26.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.409+0000 7f85baa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5573df880b60 0x5573df88f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:26.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.409+0000 7f85baa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5573df880b60 0x5573df88f980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39638/0 (socket says 172.21.15.28:39638) 2024-05-12T22:12:26.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.409+0000 7f85baa00640 1 -- 172.21.15.28:0/1331737658 learned_addr learned my addr 172.21.15.28:0/1331737658 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:12:26.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.409+0000 7f85ba000640 1 --2- 172.21.15.28:0/1331737658 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5573df881480 0x5573df88fec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:26.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.409+0000 7f85bb400640 1 --2- 172.21.15.28:0/1331737658 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5573df88bb50 0x5573df890400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:26.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.409+0000 7f85baa00640 1 -- 172.21.15.28:0/1331737658 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5573df88bb50 msgr2=0x5573df890400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:12:26.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.409+0000 7f85baa00640 1 --2- 172.21.15.28:0/1331737658 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5573df88bb50 0x5573df890400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:26.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.409+0000 7f85baa00640 1 -- 172.21.15.28:0/1331737658 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5573df881480 msgr2=0x5573df88fec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:12:26.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.409+0000 7f85baa00640 1 --2- 172.21.15.28:0/1331737658 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5573df881480 0x5573df88fec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:26.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.409+0000 7f85baa00640 1 -- 172.21.15.28:0/1331737658 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5573df9016e0 con 0x5573df880b60 2024-05-12T22:12:26.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.409+0000 7f85ba000640 1 --2- 172.21.15.28:0/1331737658 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5573df881480 0x5573df88fec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:12:26.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.410+0000 7f85baa00640 1 --2- 172.21.15.28:0/1331737658 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5573df880b60 0x5573df88f980 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f85ac005080 tx=0x7f85ac005f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=609c29a6d675e8f8 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:12:26.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.410+0000 7f85a4c00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-12T22:12:26.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.410+0000 7f85a4c00640 1 -- 172.21.15.28:0/1331737658 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85ac011de0 con 0x5573df880b60 2024-05-12T22:12:26.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.410+0000 7f85a4c00640 1 -- 172.21.15.28:0/1331737658 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f85ac02b610 con 0x5573df880b60 2024-05-12T22:12:26.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.410+0000 7f85c4683740 1 -- 172.21.15.28:0/1331737658 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f85ac02b460 con 0x5573df880b60 2024-05-12T22:12:26.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.410+0000 7f85a4c00640 1 -- 172.21.15.28:0/1331737658 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f85ac02b810 con 0x5573df880b60 2024-05-12T22:12:26.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.410+0000 7f85c4683740 10 client.15969 fetch_fsmap learned FSMap version 25 2024-05-12T22:12:26.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.410+0000 7f85c4683740 1 -- 172.21.15.28:0/1331737658 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5573df9029a0 con 0x5573df880b60 2024-05-12T22:12:26.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.411+0000 7f85a4c00640 1 -- 172.21.15.28:0/1331737658 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f85ac02b460 con 0x5573df880b60 2024-05-12T22:12:26.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.411+0000 7f85a4c00640 1 -- 172.21.15.28:0/1331737658 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7f85ac0036f0 con 0x5573df880b60 2024-05-12T22:12:26.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.411+0000 7f85c4683740 10 client.15969 fetch_fsmap finished waiting for FSMap version 25 2024-05-12T22:12:26.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.411+0000 7f85c4683740 10 client.15969 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:12:26.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.411+0000 7f85c4683740 20 client.15969 populate_metadata read hostname 'smithi028' 2024-05-12T22:12:26.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.411+0000 7f85c4683740 1 --2- 172.21.15.28:0/1331737658 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5573df902dc0 0x5573df9051b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:26.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.411+0000 7f85c4683740 4 client.15969 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-12T22:12:26.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.411+0000 7f85c4683740 1 -- 172.21.15.28:0/1331737658 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5573df7ed4b0 con 0x5573df902dc0 2024-05-12T22:12:26.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.411+0000 7f85ba000640 1 --2- 172.21.15.28:0/1331737658 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5573df902dc0 0x5573df9051b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:26.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.412+0000 7f85ba000640 1 --2- 172.21.15.28:0/1331737658 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5573df902dc0 0x5573df9051b0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9a39faf64031187f server_cookie=a053f0b94c74f21f in_seq=0 out_seq=0 2024-05-12T22:12:26.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.412+0000 7f85a4c00640 10 client.15969 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:12:26.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.412+0000 7f85a4c00640 1 -- 172.21.15.28:0/1331737658 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5573df7ed4b0 con 0x5573df902dc0 2024-05-12T22:12:26.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.412+0000 7f85a4c00640 10 client.15969 handle_command_reply: tid=0 2024-05-12T22:12:26.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.412+0000 7f85c4683740 1 client.15969 shutdown 2024-05-12T22:12:26.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.412+0000 7f85c4683740 1 -- 172.21.15.28:0/1331737658 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5573df880b60 msgr2=0x5573df88f980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:12:26.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.412+0000 7f85c4683740 1 --2- 172.21.15.28:0/1331737658 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5573df880b60 0x5573df88f980 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f85ac005080 tx=0x7f85ac005f30 comp rx=0 tx=0).stop 2024-05-12T22:12:26.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.412+0000 7f85c4683740 1 -- 172.21.15.28:0/1331737658 shutdown_connections 2024-05-12T22:12:26.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.412+0000 7f85c4683740 1 --2- 172.21.15.28:0/1331737658 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5573df902dc0 0x5573df9051b0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:26.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.412+0000 7f85c4683740 1 --2- 172.21.15.28:0/1331737658 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5573df88bb50 0x5573df890400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:26.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.412+0000 7f85c4683740 1 --2- 172.21.15.28:0/1331737658 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5573df881480 0x5573df88fec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:26.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.412+0000 7f85c4683740 1 --2- 172.21.15.28:0/1331737658 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5573df880b60 0x5573df88f980 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:26.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.412+0000 7f85a4c00640 0 client.15969 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:12:26.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.412+0000 7f85c4683740 1 -- 172.21.15.28:0/1331737658 >> 172.21.15.28:0/1331737658 conn(0x5573df87ce80 msgr2=0x5573df87c000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:12:26.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.413+0000 7f85c4683740 1 -- 172.21.15.28:0/1331737658 shutdown_connections 2024-05-12T22:12:26.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.413+0000 7f85c4683740 1 -- 172.21.15.28:0/1331737658 wait complete. 2024-05-12T22:12:26.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.413+0000 7f85c4683740 20 client.15969 trim_cache size 0 max 16384 2024-05-12T22:12:26.422 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-05-12T22:12:26.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.413+0000 7f85bbe00640 1 -- 172.21.15.28:0/1525753705 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f85a00a2d80 msgr2=0x7f85a00a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:12:26.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.413+0000 7f85bbe00640 1 --2- 172.21.15.28:0/1525753705 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f85a00a2d80 0x7f85a00a5240 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f85b4005e30 tx=0x7f85b4005c50 comp rx=0 tx=0).stop 2024-05-12T22:12:26.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.413+0000 7f85bbe00640 1 -- 172.21.15.28:0/1525753705 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f85bc0744f0 msgr2=0x7f85bc19eb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:12:26.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.413+0000 7f85bbe00640 1 --2- 172.21.15.28:0/1525753705 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f85bc0744f0 0x7f85bc19eb90 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f85a800e9f0 tx=0x7f85a800eeb0 comp rx=0 tx=0).stop 2024-05-12T22:12:26.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.413+0000 7f85bbe00640 1 -- 172.21.15.28:0/1525753705 shutdown_connections 2024-05-12T22:12:26.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.413+0000 7f85bbe00640 1 --2- 172.21.15.28:0/1525753705 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f85a00a2d80 0x7f85a00a5240 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:26.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.413+0000 7f85bbe00640 1 --2- 172.21.15.28:0/1525753705 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f85bc1126f0 0x7f85bc1a2f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:26.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.413+0000 7f85bbe00640 1 --2- 172.21.15.28:0/1525753705 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f85bc0744f0 0x7f85bc19eb90 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:26.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.413+0000 7f85bbe00640 1 --2- 172.21.15.28:0/1525753705 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85bc073bd0 0x7f85bc19e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:26.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.414+0000 7f85bbe00640 1 -- 172.21.15.28:0/1525753705 >> 172.21.15.28:0/1525753705 conn(0x7f85bc0fd1d0 msgr2=0x7f85bc0fea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:12:26.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.414+0000 7f85bbe00640 1 -- 172.21.15.28:0/1525753705 shutdown_connections 2024-05-12T22:12:26.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:26.414+0000 7f85bbe00640 1 -- 172.21.15.28:0/1525753705 wait complete. 2024-05-12T22:12:26.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:12:26 smithi186 ceph-mon[34785]: pgmap v106: 97 pgs: 97 active+clean; 97 MiB data, 752 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 4.1 KiB/s wr, 8 op/s 2024-05-12T22:12:26.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:12:26 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/3027396790' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:12:26.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:12:26 smithi086 ceph-mon[34297]: pgmap v106: 97 pgs: 97 active+clean; 97 MiB data, 752 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 4.1 KiB/s wr, 8 op/s 2024-05-12T22:12:26.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:12:26 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/3027396790' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:12:27.433 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-12T22:12:27.434 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-12T22:12:27.434 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-05-12T22:12:27.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.663+0000 7fdc48e00640 1 -- 172.21.15.28:0/2059805343 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdc441020a0 msgr2=0x7fdc4410e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:12:27.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.663+0000 7fdc48e00640 1 --2- 172.21.15.28:0/2059805343 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdc441020a0 0x7fdc4410e5b0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fdc3c009c10 tx=0x7fdc3c02a6e0 comp rx=0 tx=0).stop 2024-05-12T22:12:27.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.663+0000 7fdc48e00640 1 -- 172.21.15.28:0/2059805343 shutdown_connections 2024-05-12T22:12:27.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.663+0000 7fdc48e00640 1 --2- 172.21.15.28:0/2059805343 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdc441020a0 0x7fdc4410e5b0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:27.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.663+0000 7fdc48e00640 1 --2- 172.21.15.28:0/2059805343 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdc441016e0 0x7fdc44101b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:27.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.663+0000 7fdc48e00640 1 --2- 172.21.15.28:0/2059805343 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc441076f0 0x7fdc44107ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:27.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.663+0000 7fdc48e00640 1 -- 172.21.15.28:0/2059805343 >> 172.21.15.28:0/2059805343 conn(0x7fdc440fd1d0 msgr2=0x7fdc440ff5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:12:27.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.664+0000 7fdc48e00640 1 -- 172.21.15.28:0/2059805343 shutdown_connections 2024-05-12T22:12:27.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.664+0000 7fdc48e00640 1 -- 172.21.15.28:0/2059805343 wait complete. 2024-05-12T22:12:27.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.665+0000 7fdc48e00640 1 Processor -- start 2024-05-12T22:12:27.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.665+0000 7fdc48e00640 1 -- start start 2024-05-12T22:12:27.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.666+0000 7fdc48e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdc441016e0 0x7fdc44197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:27.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.666+0000 7fdc48e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdc441020a0 0x7fdc441984e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:27.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.666+0000 7fdc48e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc441076f0 0x7fdc4419c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:27.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.666+0000 7fdc48e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdc44111b00 con 0x7fdc441076f0 2024-05-12T22:12:27.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.666+0000 7fdc48e00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fdc44111980 con 0x7fdc441016e0 2024-05-12T22:12:27.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.666+0000 7fdc48e00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fdc44111c80 con 0x7fdc441020a0 2024-05-12T22:12:27.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.666+0000 7fdc43400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdc441016e0 0x7fdc44197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:27.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.666+0000 7fdc3be00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdc441020a0 0x7fdc441984e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:27.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.666+0000 7fdc43400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdc441016e0 0x7fdc44197fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:49528/0 (socket says 172.21.15.28:49528) 2024-05-12T22:12:27.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.666+0000 7fdc43400640 1 -- 172.21.15.28:0/2792829776 learned_addr learned my addr 172.21.15.28:0/2792829776 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:12:27.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.666+0000 7fdc43e00640 1 --2- 172.21.15.28:0/2792829776 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc441076f0 0x7fdc4419c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:27.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.667+0000 7fdc3be00640 1 -- 172.21.15.28:0/2792829776 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdc441016e0 msgr2=0x7fdc44197fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:12:27.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.667+0000 7fdc3be00640 1 --2- 172.21.15.28:0/2792829776 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdc441016e0 0x7fdc44197fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:27.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.667+0000 7fdc3be00640 1 -- 172.21.15.28:0/2792829776 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc441076f0 msgr2=0x7fdc4419c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:12:27.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.667+0000 7fdc3be00640 1 --2- 172.21.15.28:0/2792829776 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc441076f0 0x7fdc4419c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:27.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.667+0000 7fdc3be00640 1 -- 172.21.15.28:0/2792829776 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc4419cf90 con 0x7fdc441020a0 2024-05-12T22:12:27.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.667+0000 7fdc3be00640 1 --2- 172.21.15.28:0/2792829776 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdc441020a0 0x7fdc441984e0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fdc2c00e9f0 tx=0x7fdc2c00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:12:27.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.667+0000 7fdc40c00640 1 -- 172.21.15.28:0/2792829776 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc2c00ccc0 con 0x7fdc441020a0 2024-05-12T22:12:27.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.667+0000 7fdc40c00640 1 -- 172.21.15.28:0/2792829776 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdc2c005020 con 0x7fdc441020a0 2024-05-12T22:12:27.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.667+0000 7fdc48e00640 1 -- 172.21.15.28:0/2792829776 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc4419d280 con 0x7fdc441020a0 2024-05-12T22:12:27.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.667+0000 7fdc40c00640 1 -- 172.21.15.28:0/2792829776 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc2c0049e0 con 0x7fdc441020a0 2024-05-12T22:12:27.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.667+0000 7fdc48e00640 1 -- 172.21.15.28:0/2792829776 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc441a4b10 con 0x7fdc441020a0 2024-05-12T22:12:27.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.671+0000 7fdc4d21d740 1 Processor -- start 2024-05-12T22:12:27.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.671+0000 7fdc4d21d740 1 -- start start 2024-05-12T22:12:27.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.671+0000 7fdc4d21d740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564e2f589910 0x564e2f589cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:27.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.671+0000 7fdc4d21d740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564e2f58a2c0 0x564e2f594360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:27.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.671+0000 7fdc4d21d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e2f594930 0x564e2f596d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:27.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.671+0000 7fdc4d21d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x564e2f4e97c0 con 0x564e2f594930 2024-05-12T22:12:27.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.671+0000 7fdc4d21d740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x564e2f491790 con 0x564e2f589910 2024-05-12T22:12:27.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.671+0000 7fdc4d21d740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x564e2f4e5780 con 0x564e2f58a2c0 2024-05-12T22:12:27.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.671+0000 7fdc43e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564e2f589910 0x564e2f589cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:27.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.671+0000 7fdc43e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564e2f589910 0x564e2f589cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:49542/0 (socket says 172.21.15.28:49542) 2024-05-12T22:12:27.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.671+0000 7fdc43e00640 1 -- 172.21.15.28:0/3772282139 learned_addr learned my addr 172.21.15.28:0/3772282139 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:12:27.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.672+0000 7fdc43400640 1 --2- 172.21.15.28:0/3772282139 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564e2f58a2c0 0x564e2f594360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:27.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.672+0000 7fdc43e00640 1 -- 172.21.15.28:0/3772282139 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564e2f58a2c0 msgr2=0x564e2f594360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:12:27.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.672+0000 7fdc43e00640 1 --2- 172.21.15.28:0/3772282139 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564e2f58a2c0 0x564e2f594360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:27.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.672+0000 7fdc43e00640 1 -- 172.21.15.28:0/3772282139 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e2f594930 msgr2=0x564e2f596d80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-12T22:12:27.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.672+0000 7fdc43e00640 1 --2- 172.21.15.28:0/3772282139 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e2f594930 0x564e2f596d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:27.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.672+0000 7fdc43e00640 1 -- 172.21.15.28:0/3772282139 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564e2f4b16a0 con 0x564e2f589910 2024-05-12T22:12:27.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.672+0000 7fdc43e00640 1 --2- 172.21.15.28:0/3772282139 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564e2f589910 0x564e2f589cf0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fdc3c009be0 tx=0x7fdc3c030a30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e80b22dac611a2d0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:12:27.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.672+0000 7fdc40c00640 1 -- 172.21.15.28:0/2792829776 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7fdc2c005560 con 0x7fdc441020a0 2024-05-12T22:12:27.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.673+0000 7fdc38c00640 1 -- 172.21.15.28:0/3772282139 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc3c041030 con 0x564e2f589910 2024-05-12T22:12:27.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.673+0000 7fdc38c00640 1 -- 172.21.15.28:0/3772282139 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdc3c030c60 con 0x564e2f589910 2024-05-12T22:12:27.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.673+0000 7fdc38c00640 1 -- 172.21.15.28:0/3772282139 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc3c036040 con 0x564e2f589910 2024-05-12T22:12:27.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.673+0000 7fdc4d21d740 1 -- 172.21.15.28:0/3772282139 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564e2f589910 msgr2=0x564e2f589cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:12:27.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.673+0000 7fdc4d21d740 1 --2- 172.21.15.28:0/3772282139 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564e2f589910 0x564e2f589cf0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fdc3c009be0 tx=0x7fdc3c030a30 comp rx=0 tx=0).stop 2024-05-12T22:12:27.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.674+0000 7fdc4d21d740 1 -- 172.21.15.28:0/3772282139 shutdown_connections 2024-05-12T22:12:27.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.674+0000 7fdc4d21d740 1 --2- 172.21.15.28:0/3772282139 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e2f594930 0x564e2f596d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:27.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.674+0000 7fdc4d21d740 1 --2- 172.21.15.28:0/3772282139 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564e2f58a2c0 0x564e2f594360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:27.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.674+0000 7fdc4d21d740 1 --2- 172.21.15.28:0/3772282139 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564e2f589910 0x564e2f589cf0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:27.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.674+0000 7fdc4d21d740 1 -- 172.21.15.28:0/3772282139 >> 172.21.15.28:0/3772282139 conn(0x564e2f5859c0 msgr2=0x564e2f585da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:12:27.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.674+0000 7fdc4d21d740 1 -- 172.21.15.28:0/3772282139 shutdown_connections 2024-05-12T22:12:27.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.674+0000 7fdc4d21d740 1 -- 172.21.15.28:0/3772282139 wait complete. 2024-05-12T22:12:27.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.674+0000 7fdc40c00640 1 --2- 172.21.15.28:0/2792829776 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fdc280a2c90 0x7fdc280a5150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:27.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.674+0000 7fdc4d21d740 1 -- start start 2024-05-12T22:12:27.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.674+0000 7fdc40c00640 1 -- 172.21.15.28:0/2792829776 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7fdc2c0cf310 con 0x7fdc441020a0 2024-05-12T22:12:27.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.674+0000 7fdc43e00640 1 --2- 172.21.15.28:0/2792829776 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fdc280a2c90 0x7fdc280a5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:27.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.674+0000 7fdc4d21d740 1 Processor -- start 2024-05-12T22:12:27.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.675+0000 7fdc4d21d740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564e2f589910 0x564e2f591830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:27.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.675+0000 7fdc4d21d740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564e2f58a2c0 0x564e2f591d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:27.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.675+0000 7fdc4d21d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e2f594930 0x564e2f5922b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:27.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.675+0000 7fdc4d21d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x564e2f5978e0 con 0x564e2f594930 2024-05-12T22:12:27.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.675+0000 7fdc4d21d740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x564e2f4a4a90 con 0x564e2f58a2c0 2024-05-12T22:12:27.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.675+0000 7fdc4d21d740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x564e2f597a60 con 0x564e2f589910 2024-05-12T22:12:27.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.675+0000 7fdc4d21d740 10 client.0 _ll_register_callbacks cb 0x564e2f4dfaf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:12:27.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.675+0000 7fdc43e00640 1 --2- 172.21.15.28:0/2792829776 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fdc280a2c90 0x7fdc280a5150 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fdc3c009b30 tx=0x7fdc3c02a6e0 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:12:27.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.675+0000 7fdc3be00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564e2f58a2c0 0x564e2f591d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:27.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.675+0000 7fdc43400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564e2f589910 0x564e2f591830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:27.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.675+0000 7fdc3be00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564e2f58a2c0 0x564e2f591d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:49548/0 (socket says 172.21.15.28:49548) 2024-05-12T22:12:27.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.675+0000 7fdc3be00640 1 -- 172.21.15.28:0/3780487910 learned_addr learned my addr 172.21.15.28:0/3780487910 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:12:27.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.675+0000 7fdc43e00640 1 --2- 172.21.15.28:0/3780487910 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e2f594930 0x564e2f5922b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:27.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.676+0000 7fdc3be00640 1 -- 172.21.15.28:0/3780487910 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564e2f589910 msgr2=0x564e2f591830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:12:27.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.676+0000 7fdc3be00640 1 --2- 172.21.15.28:0/3780487910 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564e2f589910 0x564e2f591830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:27.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.676+0000 7fdc3be00640 1 -- 172.21.15.28:0/3780487910 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e2f594930 msgr2=0x564e2f5922b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:12:27.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.676+0000 7fdc3be00640 1 --2- 172.21.15.28:0/3780487910 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e2f594930 0x564e2f5922b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:27.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.676+0000 7fdc3be00640 1 -- 172.21.15.28:0/3780487910 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564e2f2c57e0 con 0x564e2f58a2c0 2024-05-12T22:12:27.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.676+0000 7fdc3be00640 1 --2- 172.21.15.28:0/3780487910 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564e2f58a2c0 0x564e2f591d70 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fdc2c0cd090 tx=0x7fdc2c09b280 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1af4197fb39c3c4d server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:12:27.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.676+0000 7fdc31600640 10 client.0 ms_handle_connect on v2:172.21.15.86:3300/0 2024-05-12T22:12:27.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.676+0000 7fdc31600640 1 -- 172.21.15.28:0/3780487910 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc2c0b4060 con 0x564e2f58a2c0 2024-05-12T22:12:27.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.676+0000 7fdc4d21d740 1 -- 172.21.15.28:0/3780487910 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdc3c030c60 con 0x564e2f58a2c0 2024-05-12T22:12:27.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.676+0000 7fdc31600640 1 -- 172.21.15.28:0/3780487910 <== mon.2 v2:172.21.15.86:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7fdc2c086070 con 0x564e2f58a2c0 2024-05-12T22:12:27.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.677+0000 7fdc31600640 1 -- 172.21.15.28:0/3780487910 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fdc2c01c040 con 0x564e2f58a2c0 2024-05-12T22:12:27.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.677+0000 7fdc4d21d740 10 client.26108 fetch_fsmap learned FSMap version 25 2024-05-12T22:12:27.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.677+0000 7fdc4d21d740 1 -- 172.21.15.28:0/3780487910 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x564e2f325f70 con 0x564e2f58a2c0 2024-05-12T22:12:27.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.677+0000 7fdc31600640 1 -- 172.21.15.28:0/3780487910 <== mon.2 v2:172.21.15.86:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdc3c030c60 con 0x564e2f58a2c0 2024-05-12T22:12:27.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.677+0000 7fdc31600640 1 -- 172.21.15.28:0/3780487910 <== mon.2 v2:172.21.15.86:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7fdc2c0a4040 con 0x564e2f58a2c0 2024-05-12T22:12:27.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.677+0000 7fdc4d21d740 10 client.26108 fetch_fsmap finished waiting for FSMap version 25 2024-05-12T22:12:27.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.677+0000 7fdc4d21d740 10 client.26108 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:12:27.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.677+0000 7fdc4d21d740 20 client.26108 populate_metadata read hostname 'smithi028' 2024-05-12T22:12:27.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.677+0000 7fdc4d21d740 1 --2- 172.21.15.28:0/3780487910 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x564e2f5a4aa0 0x564e2f5a6e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:27.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.677+0000 7fdc4d21d740 4 client.26108 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-12T22:12:27.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.677+0000 7fdc4d21d740 1 -- 172.21.15.28:0/3780487910 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564e2f4fd3a0 con 0x564e2f5a4aa0 2024-05-12T22:12:27.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.678+0000 7fdc43400640 1 --2- 172.21.15.28:0/3780487910 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x564e2f5a4aa0 0x564e2f5a6e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:27.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.678+0000 7fdc43400640 1 --2- 172.21.15.28:0/3780487910 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x564e2f5a4aa0 0x564e2f5a6e90 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8fe181667c693a42 server_cookie=7d261142a7d80efc in_seq=0 out_seq=0 2024-05-12T22:12:27.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.678+0000 7fdc31600640 10 client.26108 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:12:27.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.679+0000 7fdc31600640 1 -- 172.21.15.28:0/3780487910 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564e2f4fd3a0 con 0x564e2f5a4aa0 2024-05-12T22:12:27.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.679+0000 7fdc31600640 10 client.26108 handle_command_reply: tid=0 2024-05-12T22:12:27.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.679+0000 7fdc4d21d740 1 client.26108 shutdown 2024-05-12T22:12:27.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.679+0000 7fdc4d21d740 1 -- 172.21.15.28:0/3780487910 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564e2f58a2c0 msgr2=0x564e2f591d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:12:27.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.679+0000 7fdc4d21d740 1 --2- 172.21.15.28:0/3780487910 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564e2f58a2c0 0x564e2f591d70 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fdc2c0cd090 tx=0x7fdc2c09b280 comp rx=0 tx=0).stop 2024-05-12T22:12:27.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.679+0000 7fdc4d21d740 1 -- 172.21.15.28:0/3780487910 shutdown_connections 2024-05-12T22:12:27.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.679+0000 7fdc4d21d740 1 --2- 172.21.15.28:0/3780487910 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x564e2f5a4aa0 0x564e2f5a6e90 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:27.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.679+0000 7fdc4d21d740 1 --2- 172.21.15.28:0/3780487910 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e2f594930 0x564e2f5922b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:27.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.679+0000 7fdc4d21d740 1 --2- 172.21.15.28:0/3780487910 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564e2f58a2c0 0x564e2f591d70 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:27.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.679+0000 7fdc4d21d740 1 --2- 172.21.15.28:0/3780487910 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564e2f589910 0x564e2f591830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:27.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.679+0000 7fdc31600640 0 client.26108 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:12:27.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.679+0000 7fdc4d21d740 1 -- 172.21.15.28:0/3780487910 >> 172.21.15.28:0/3780487910 conn(0x564e2f5859c0 msgr2=0x564e2f58ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:12:27.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.680+0000 7fdc4d21d740 1 -- 172.21.15.28:0/3780487910 shutdown_connections 2024-05-12T22:12:27.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.680+0000 7fdc4d21d740 1 -- 172.21.15.28:0/3780487910 wait complete. 2024-05-12T22:12:27.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.680+0000 7fdc4d21d740 20 client.26108 trim_cache size 0 max 16384 2024-05-12T22:12:27.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.699+0000 7fdc4d21d740 1 Processor -- start 2024-05-12T22:12:27.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.699+0000 7fdc4d21d740 1 -- start start 2024-05-12T22:12:27.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.699+0000 7fdc4d21d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e2f5859c0 0x564e2f5a07d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:27.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.699+0000 7fdc4d21d740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564e2f589910 0x564e2f5805b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:27.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.700+0000 7fdc4d21d740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564e2f58a2c0 0x564e2f5ac710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:27.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.700+0000 7fdc4d21d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdc441ad8e0 con 0x564e2f5859c0 2024-05-12T22:12:27.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.700+0000 7fdc4d21d740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fdc441ab1b0 con 0x564e2f589910 2024-05-12T22:12:27.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.700+0000 7fdc4d21d740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fdc441a9020 con 0x564e2f58a2c0 2024-05-12T22:12:27.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.700+0000 7fdc43400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e2f5859c0 0x564e2f5a07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:27.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.700+0000 7fdc43400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e2f5859c0 0x564e2f5a07d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39678/0 (socket says 172.21.15.28:39678) 2024-05-12T22:12:27.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.700+0000 7fdc43400640 1 -- 172.21.15.28:0/3194593619 learned_addr learned my addr 172.21.15.28:0/3194593619 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:12:27.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.700+0000 7fdc3be00640 1 --2- 172.21.15.28:0/3194593619 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564e2f589910 0x564e2f5805b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:27.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.700+0000 7fdc43e00640 1 --2- 172.21.15.28:0/3194593619 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564e2f58a2c0 0x564e2f5ac710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:27.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.701+0000 7fdc3be00640 1 -- 172.21.15.28:0/3194593619 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564e2f58a2c0 msgr2=0x564e2f5ac710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:12:27.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.701+0000 7fdc3be00640 1 --2- 172.21.15.28:0/3194593619 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564e2f58a2c0 0x564e2f5ac710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:27.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.701+0000 7fdc3be00640 1 -- 172.21.15.28:0/3194593619 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e2f5859c0 msgr2=0x564e2f5a07d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:12:27.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.701+0000 7fdc3be00640 1 --2- 172.21.15.28:0/3194593619 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e2f5859c0 0x564e2f5a07d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:27.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.701+0000 7fdc3be00640 1 -- 172.21.15.28:0/3194593619 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564e2f58a060 con 0x564e2f589910 2024-05-12T22:12:27.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.701+0000 7fdc3be00640 1 --2- 172.21.15.28:0/3194593619 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564e2f589910 0x564e2f5805b0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fdc2c09b560 tx=0x7fdc2c09b2f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1955307a6a6f0e8c server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:12:27.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.701+0000 7fdc33e00640 1 -- 172.21.15.28:0/3194593619 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc2c0a44d0 con 0x564e2f589910 2024-05-12T22:12:27.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.701+0000 7fdc33e00640 1 -- 172.21.15.28:0/3194593619 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdc2c0860c0 con 0x564e2f589910 2024-05-12T22:12:27.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.701+0000 7fdc33e00640 1 -- 172.21.15.28:0/3194593619 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc2c09cb90 con 0x564e2f589910 2024-05-12T22:12:27.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.702+0000 7fdc4d21d740 1 -- 172.21.15.28:0/3194593619 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564e2f589910 msgr2=0x564e2f5805b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:12:27.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.702+0000 7fdc4d21d740 1 --2- 172.21.15.28:0/3194593619 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564e2f589910 0x564e2f5805b0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fdc2c09b560 tx=0x7fdc2c09b2f0 comp rx=0 tx=0).stop 2024-05-12T22:12:27.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.702+0000 7fdc4d21d740 1 -- 172.21.15.28:0/3194593619 shutdown_connections 2024-05-12T22:12:27.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.702+0000 7fdc4d21d740 1 --2- 172.21.15.28:0/3194593619 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564e2f58a2c0 0x564e2f5ac710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:27.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.702+0000 7fdc4d21d740 1 --2- 172.21.15.28:0/3194593619 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564e2f589910 0x564e2f5805b0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:27.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.702+0000 7fdc4d21d740 1 --2- 172.21.15.28:0/3194593619 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e2f5859c0 0x564e2f5a07d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:27.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.702+0000 7fdc4d21d740 1 -- 172.21.15.28:0/3194593619 >> 172.21.15.28:0/3194593619 conn(0x564e2f590490 msgr2=0x564e2f57fea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:12:27.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.702+0000 7fdc4d21d740 1 -- 172.21.15.28:0/3194593619 shutdown_connections 2024-05-12T22:12:27.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.702+0000 7fdc4d21d740 1 -- 172.21.15.28:0/3194593619 wait complete. 2024-05-12T22:12:27.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.702+0000 7fdc4d21d740 1 -- start start 2024-05-12T22:12:27.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.702+0000 7fdc4d21d740 1 Processor -- start 2024-05-12T22:12:27.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.703+0000 7fdc4d21d740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564e2f5859c0 0x564e2f599cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:27.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.703+0000 7fdc4d21d740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564e2f589910 0x564e2f59a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:27.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.703+0000 7fdc4d21d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e2f58a2c0 0x564e2f59a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:27.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.703+0000 7fdc4d21d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdc441a9670 con 0x564e2f58a2c0 2024-05-12T22:12:27.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.703+0000 7fdc4d21d740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fdc441aaae0 con 0x564e2f5859c0 2024-05-12T22:12:27.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.703+0000 7fdc4d21d740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fdc441a9d40 con 0x564e2f589910 2024-05-12T22:12:27.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.703+0000 7fdc4d21d740 10 client.0 _ll_register_callbacks cb 0x7fdc2c00e940 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:12:27.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.703+0000 7fdc43e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e2f58a2c0 0x564e2f59a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:27.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.703+0000 7fdc43e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e2f58a2c0 0x564e2f59a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39680/0 (socket says 172.21.15.28:39680) 2024-05-12T22:12:27.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.703+0000 7fdc3be00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564e2f589910 0x564e2f59a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:27.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.703+0000 7fdc43e00640 1 -- 172.21.15.28:0/3220959744 learned_addr learned my addr 172.21.15.28:0/3220959744 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:12:27.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.703+0000 7fdc43400640 1 --2- 172.21.15.28:0/3220959744 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564e2f5859c0 0x564e2f599cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:27.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.704+0000 7fdc43400640 1 -- 172.21.15.28:0/3220959744 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564e2f589910 msgr2=0x564e2f59a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:12:27.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.704+0000 7fdc43400640 1 --2- 172.21.15.28:0/3220959744 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564e2f589910 0x564e2f59a1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:27.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.704+0000 7fdc43400640 1 -- 172.21.15.28:0/3220959744 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e2f58a2c0 msgr2=0x564e2f59a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:12:27.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.704+0000 7fdc43400640 1 --2- 172.21.15.28:0/3220959744 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e2f58a2c0 0x564e2f59a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:27.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.704+0000 7fdc43400640 1 -- 172.21.15.28:0/3220959744 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564e2f1bd170 con 0x564e2f5859c0 2024-05-12T22:12:27.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.704+0000 7fdc3be00640 1 --2- 172.21.15.28:0/3220959744 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564e2f589910 0x564e2f59a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:12:27.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.704+0000 7fdc43e00640 1 --2- 172.21.15.28:0/3220959744 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e2f58a2c0 0x564e2f59a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:12:27.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.704+0000 7fdc43400640 1 --2- 172.21.15.28:0/3220959744 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564e2f5859c0 0x564e2f599cb0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fdc34004590 tx=0x7fdc34009650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=330580c2907b775b server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:12:27.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.704+0000 7fdc31600640 10 client.0 ms_handle_connect on v2:172.21.15.86:3300/0 2024-05-12T22:12:27.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.704+0000 7fdc31600640 1 -- 172.21.15.28:0/3220959744 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc340202e0 con 0x564e2f5859c0 2024-05-12T22:12:27.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.704+0000 7fdc31600640 1 -- 172.21.15.28:0/3220959744 <== mon.2 v2:172.21.15.86:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7fdc34020480 con 0x564e2f5859c0 2024-05-12T22:12:27.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.704+0000 7fdc4d21d740 1 -- 172.21.15.28:0/3220959744 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdc2c0860c0 con 0x564e2f5859c0 2024-05-12T22:12:27.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.705+0000 7fdc31600640 1 -- 172.21.15.28:0/3220959744 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fdc34020680 con 0x564e2f5859c0 2024-05-12T22:12:27.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.705+0000 7fdc4d21d740 10 client.26114 fetch_fsmap learned FSMap version 25 2024-05-12T22:12:27.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.705+0000 7fdc4d21d740 1 -- 172.21.15.28:0/3220959744 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x564e2f60c270 con 0x564e2f5859c0 2024-05-12T22:12:27.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.705+0000 7fdc31600640 1 -- 172.21.15.28:0/3220959744 <== mon.2 v2:172.21.15.86:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdc2c0860c0 con 0x564e2f5859c0 2024-05-12T22:12:27.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.705+0000 7fdc31600640 1 -- 172.21.15.28:0/3220959744 <== mon.2 v2:172.21.15.86:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7fdc34010430 con 0x564e2f5859c0 2024-05-12T22:12:27.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.705+0000 7fdc4d21d740 10 client.26114 fetch_fsmap finished waiting for FSMap version 25 2024-05-12T22:12:27.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.705+0000 7fdc4d21d740 10 client.26114 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:12:27.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.705+0000 7fdc4d21d740 20 client.26114 populate_metadata read hostname 'smithi028' 2024-05-12T22:12:27.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.705+0000 7fdc4d21d740 1 --2- 172.21.15.28:0/3220959744 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x564e2f594930 0x564e2f60e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:27.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.705+0000 7fdc4d21d740 4 client.26114 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-12T22:12:27.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.705+0000 7fdc4d21d740 1 -- 172.21.15.28:0/3220959744 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x564e2f4f6ef0 con 0x564e2f594930 2024-05-12T22:12:27.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.706+0000 7fdc3be00640 1 --2- 172.21.15.28:0/3220959744 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x564e2f594930 0x564e2f60e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:27.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.706+0000 7fdc3be00640 1 --2- 172.21.15.28:0/3220959744 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x564e2f594930 0x564e2f60e9e0 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=32449bfca3a80359 server_cookie=301df8493eb7d4c1 in_seq=0 out_seq=0 2024-05-12T22:12:27.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.706+0000 7fdc31600640 10 client.26114 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:12:27.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.707+0000 7fdc31600640 1 -- 172.21.15.28:0/3220959744 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x564e2f4f6ef0 con 0x564e2f594930 2024-05-12T22:12:27.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.707+0000 7fdc31600640 10 client.26114 handle_command_reply: tid=0 2024-05-12T22:12:27.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.708+0000 7fdc4d21d740 1 client.26114 shutdown 2024-05-12T22:12:27.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.708+0000 7fdc4d21d740 1 -- 172.21.15.28:0/3220959744 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564e2f5859c0 msgr2=0x564e2f599cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:12:27.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.708+0000 7fdc4d21d740 1 --2- 172.21.15.28:0/3220959744 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564e2f5859c0 0x564e2f599cb0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fdc34004590 tx=0x7fdc34009650 comp rx=0 tx=0).stop 2024-05-12T22:12:27.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.708+0000 7fdc4d21d740 1 -- 172.21.15.28:0/3220959744 shutdown_connections 2024-05-12T22:12:27.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.708+0000 7fdc4d21d740 1 --2- 172.21.15.28:0/3220959744 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x564e2f594930 0x564e2f60e9e0 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:27.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.708+0000 7fdc4d21d740 1 --2- 172.21.15.28:0/3220959744 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e2f58a2c0 0x564e2f59a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:27.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.708+0000 7fdc4d21d740 1 --2- 172.21.15.28:0/3220959744 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564e2f589910 0x564e2f59a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:27.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.708+0000 7fdc4d21d740 1 --2- 172.21.15.28:0/3220959744 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564e2f5859c0 0x564e2f599cb0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:27.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.708+0000 7fdc31600640 0 client.26114 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:12:27.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.708+0000 7fdc4d21d740 1 -- 172.21.15.28:0/3220959744 >> 172.21.15.28:0/3220959744 conn(0x564e2f590490 msgr2=0x564e2f580730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:12:27.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.708+0000 7fdc4d21d740 1 -- 172.21.15.28:0/3220959744 shutdown_connections 2024-05-12T22:12:27.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.708+0000 7fdc4d21d740 1 -- 172.21.15.28:0/3220959744 wait complete. 2024-05-12T22:12:27.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.708+0000 7fdc4d21d740 20 client.26114 trim_cache size 0 max 16384 2024-05-12T22:12:27.720 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-05-12T22:12:27.720 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-05-12T22:12:27.720 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "2d602076-1a6b-40b2-9519-9ad565de1683", 2024-05-12T22:12:27.721 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-05-12T22:12:27.721 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-05-12T22:12:27.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.709+0000 7fdc48e00640 1 -- 172.21.15.28:0/2792829776 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fdc280a2c90 msgr2=0x7fdc280a5150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:12:27.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.709+0000 7fdc48e00640 1 --2- 172.21.15.28:0/2792829776 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fdc280a2c90 0x7fdc280a5150 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fdc3c009b30 tx=0x7fdc3c02a6e0 comp rx=0 tx=0).stop 2024-05-12T22:12:27.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.709+0000 7fdc48e00640 1 -- 172.21.15.28:0/2792829776 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdc441020a0 msgr2=0x7fdc441984e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:12:27.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.709+0000 7fdc48e00640 1 --2- 172.21.15.28:0/2792829776 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdc441020a0 0x7fdc441984e0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fdc2c00e9f0 tx=0x7fdc2c00eeb0 comp rx=0 tx=0).stop 2024-05-12T22:12:27.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.709+0000 7fdc48e00640 1 -- 172.21.15.28:0/2792829776 shutdown_connections 2024-05-12T22:12:27.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.709+0000 7fdc48e00640 1 --2- 172.21.15.28:0/2792829776 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fdc280a2c90 0x7fdc280a5150 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:27.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.709+0000 7fdc48e00640 1 --2- 172.21.15.28:0/2792829776 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc441076f0 0x7fdc4419c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:27.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.709+0000 7fdc48e00640 1 --2- 172.21.15.28:0/2792829776 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdc441020a0 0x7fdc441984e0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:27.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.709+0000 7fdc48e00640 1 --2- 172.21.15.28:0/2792829776 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdc441016e0 0x7fdc44197fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:27.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.709+0000 7fdc48e00640 1 -- 172.21.15.28:0/2792829776 >> 172.21.15.28:0/2792829776 conn(0x7fdc440fd1d0 msgr2=0x7fdc440fd8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:12:27.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.709+0000 7fdc48e00640 1 -- 172.21.15.28:0/2792829776 shutdown_connections 2024-05-12T22:12:27.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.709+0000 7fdc48e00640 1 -- 172.21.15.28:0/2792829776 wait complete. 2024-05-12T22:12:27.734 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-12T22:12:27.734 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-12T22:12:27.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.974+0000 7f4b86200640 1 -- 172.21.15.28:0/2929685579 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4b801038e0 msgr2=0x7f4b80103d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:12:27.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.974+0000 7f4b86200640 1 --2- 172.21.15.28:0/2929685579 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4b801038e0 0x7f4b80103d60 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f4b6c009900 tx=0x7f4b6c02a3f0 comp rx=0 tx=0).stop 2024-05-12T22:12:27.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.974+0000 7f4b86200640 1 -- 172.21.15.28:0/2929685579 shutdown_connections 2024-05-12T22:12:27.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.974+0000 7f4b86200640 1 --2- 172.21.15.28:0/2929685579 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4b801042a0 0x7f4b801107b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:27.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.974+0000 7f4b86200640 1 --2- 172.21.15.28:0/2929685579 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4b801038e0 0x7f4b80103d60 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:27.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.974+0000 7f4b86200640 1 --2- 172.21.15.28:0/2929685579 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b801098f0 0x7f4b80109cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:27.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.974+0000 7f4b86200640 1 -- 172.21.15.28:0/2929685579 >> 172.21.15.28:0/2929685579 conn(0x7f4b800ff370 msgr2=0x7f4b80101790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:12:27.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.974+0000 7f4b86200640 1 -- 172.21.15.28:0/2929685579 shutdown_connections 2024-05-12T22:12:27.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.974+0000 7f4b86200640 1 -- 172.21.15.28:0/2929685579 wait complete. 2024-05-12T22:12:27.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.975+0000 7f4b86200640 1 Processor -- start 2024-05-12T22:12:27.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.976+0000 7f4b86200640 1 -- start start 2024-05-12T22:12:27.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.976+0000 7f4b86200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4b801038e0 0x7f4b8019a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:27.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.976+0000 7f4b86200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b801042a0 0x7f4b8019a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:27.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.976+0000 7f4b86200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4b801098f0 0x7f4b8019eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:27.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.976+0000 7f4b86200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4b80113d50 con 0x7f4b801042a0 2024-05-12T22:12:27.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.976+0000 7f4b86200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f4b80113bd0 con 0x7f4b801098f0 2024-05-12T22:12:27.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.976+0000 7f4b86200640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f4b80113ed0 con 0x7f4b801038e0 2024-05-12T22:12:27.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.976+0000 7f4b84e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4b801038e0 0x7f4b8019a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:27.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.977+0000 7f4b84e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4b801038e0 0x7f4b8019a1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:56058/0 (socket says 172.21.15.28:56058) 2024-05-12T22:12:27.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.977+0000 7f4b85800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4b801098f0 0x7f4b8019eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:27.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.977+0000 7f4b84e00640 1 -- 172.21.15.28:0/680334726 learned_addr learned my addr 172.21.15.28:0/680334726 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:12:27.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.977+0000 7f4b7fe00640 1 --2- 172.21.15.28:0/680334726 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b801042a0 0x7f4b8019a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:27.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.977+0000 7f4b85800640 1 -- 172.21.15.28:0/680334726 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4b801038e0 msgr2=0x7f4b8019a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:12:27.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.977+0000 7f4b85800640 1 --2- 172.21.15.28:0/680334726 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4b801038e0 0x7f4b8019a1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:27.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.977+0000 7f4b85800640 1 -- 172.21.15.28:0/680334726 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b801042a0 msgr2=0x7f4b8019a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:12:27.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.977+0000 7f4b85800640 1 --2- 172.21.15.28:0/680334726 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b801042a0 0x7f4b8019a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:27.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.977+0000 7f4b85800640 1 -- 172.21.15.28:0/680334726 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b8019f1e0 con 0x7f4b801098f0 2024-05-12T22:12:27.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.977+0000 7f4b85800640 1 --2- 172.21.15.28:0/680334726 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4b801098f0 0x7f4b8019eab0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f4b7800da10 tx=0x7f4b7800ded0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:12:27.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.978+0000 7f4b7d600640 1 -- 172.21.15.28:0/680334726 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b7800bd70 con 0x7f4b801098f0 2024-05-12T22:12:27.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.978+0000 7f4b7d600640 1 -- 172.21.15.28:0/680334726 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4b78003e50 con 0x7f4b801098f0 2024-05-12T22:12:27.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.978+0000 7f4b7d600640 1 -- 172.21.15.28:0/680334726 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b780106a0 con 0x7f4b801098f0 2024-05-12T22:12:27.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.978+0000 7f4b86200640 1 -- 172.21.15.28:0/680334726 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b8019f4d0 con 0x7f4b801098f0 2024-05-12T22:12:27.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.978+0000 7f4b86200640 1 -- 172.21.15.28:0/680334726 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b801a6d60 con 0x7f4b801098f0 2024-05-12T22:12:27.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.983+0000 7f4b7d600640 1 -- 172.21.15.28:0/680334726 <== mon.2 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7f4b78003970 con 0x7f4b801098f0 2024-05-12T22:12:27.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.983+0000 7f4b8a7e9740 1 Processor -- start 2024-05-12T22:12:27.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.983+0000 7f4b8a7e9740 1 -- start start 2024-05-12T22:12:27.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.984+0000 7f4b8a7e9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb51948b20 0x55cb51948f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:27.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.984+0000 7f4b8a7e9740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cb519494d0 0x55cb5195c090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:27.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.984+0000 7f4b8a7e9740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cb5195c5d0 0x55cb5195e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:27.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.984+0000 7f4b8a7e9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55cb519112a0 con 0x55cb51948b20 2024-05-12T22:12:27.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.984+0000 7f4b8a7e9740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x55cb518adb90 con 0x55cb519494d0 2024-05-12T22:12:27.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.984+0000 7f4b8a7e9740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55cb518555b0 con 0x55cb5195c5d0 2024-05-12T22:12:27.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.984+0000 7f4b84e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cb519494d0 0x55cb5195c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:27.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.984+0000 7f4b84e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cb519494d0 0x55cb5195c090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:49570/0 (socket says 172.21.15.28:49570) 2024-05-12T22:12:27.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.984+0000 7f4b84e00640 1 -- 172.21.15.28:0/181255942 learned_addr learned my addr 172.21.15.28:0/181255942 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:12:27.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.985+0000 7f4b7fe00640 1 --2- 172.21.15.28:0/181255942 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cb5195c5d0 0x55cb5195e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:27.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.985+0000 7f4b7fe00640 1 --2- 172.21.15.28:0/181255942 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb51948b20 0x55cb51948f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:27.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.985+0000 7f4b84e00640 1 -- 172.21.15.28:0/181255942 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cb5195c5d0 msgr2=0x55cb5195e9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:12:27.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.985+0000 7f4b84e00640 1 --2- 172.21.15.28:0/181255942 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cb5195c5d0 0x55cb5195e9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:27.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.985+0000 7f4b84e00640 1 -- 172.21.15.28:0/181255942 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb51948b20 msgr2=0x55cb51948f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:12:27.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.985+0000 7f4b84e00640 1 --2- 172.21.15.28:0/181255942 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb51948b20 0x55cb51948f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:27.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.985+0000 7f4b84e00640 1 -- 172.21.15.28:0/181255942 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cb5168b7d0 con 0x55cb519494d0 2024-05-12T22:12:27.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.985+0000 7f4b7fe00640 1 --2- 172.21.15.28:0/181255942 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb51948b20 0x55cb51948f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-12T22:12:27.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.985+0000 7f4b7d600640 1 --2- 172.21.15.28:0/680334726 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f4b680a2d80 0x7f4b680a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:27.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.985+0000 7f4b7d600640 1 -- 172.21.15.28:0/680334726 <== mon.2 v2:172.21.15.86:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f4b780cf7c0 con 0x7f4b801098f0 2024-05-12T22:12:27.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.985+0000 7f4b84e00640 1 --2- 172.21.15.28:0/181255942 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cb519494d0 0x55cb5195c090 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f4b74009d40 tx=0x7f4b74011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f4b803cde72750b4 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:12:27.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.986+0000 7f4b7fe00640 1 --2- 172.21.15.28:0/680334726 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f4b680a2d80 0x7f4b680a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:27.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.986+0000 7f4b72000640 1 -- 172.21.15.28:0/181255942 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b74004220 con 0x55cb519494d0 2024-05-12T22:12:27.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.986+0000 7f4b72000640 1 -- 172.21.15.28:0/181255942 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4b740043c0 con 0x55cb519494d0 2024-05-12T22:12:27.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.986+0000 7f4b72000640 1 -- 172.21.15.28:0/181255942 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b74004e20 con 0x55cb519494d0 2024-05-12T22:12:27.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.987+0000 7f4b8a7e9740 1 -- 172.21.15.28:0/181255942 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cb519494d0 msgr2=0x55cb5195c090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:12:27.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.987+0000 7f4b8a7e9740 1 --2- 172.21.15.28:0/181255942 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cb519494d0 0x55cb5195c090 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f4b74009d40 tx=0x7f4b74011810 comp rx=0 tx=0).stop 2024-05-12T22:12:27.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.987+0000 7f4b7fe00640 1 --2- 172.21.15.28:0/680334726 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f4b680a2d80 0x7f4b680a5240 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f4b6c000c00 tx=0x7f4b6c02a3f0 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:12:27.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.987+0000 7f4b8a7e9740 1 -- 172.21.15.28:0/181255942 shutdown_connections 2024-05-12T22:12:27.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.987+0000 7f4b8a7e9740 1 --2- 172.21.15.28:0/181255942 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cb5195c5d0 0x55cb5195e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:27.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.987+0000 7f4b8a7e9740 1 --2- 172.21.15.28:0/181255942 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cb519494d0 0x55cb5195c090 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:27.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.987+0000 7f4b8a7e9740 1 --2- 172.21.15.28:0/181255942 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb51948b20 0x55cb51948f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:27.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.987+0000 7f4b8a7e9740 1 -- 172.21.15.28:0/181255942 >> 172.21.15.28:0/181255942 conn(0x55cb519450f0 msgr2=0x55cb519454d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:12:27.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.987+0000 7f4b8a7e9740 1 -- 172.21.15.28:0/181255942 shutdown_connections 2024-05-12T22:12:27.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.987+0000 7f4b8a7e9740 1 -- 172.21.15.28:0/181255942 wait complete. 2024-05-12T22:12:27.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.988+0000 7f4b8a7e9740 1 -- start start 2024-05-12T22:12:27.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.988+0000 7f4b8a7e9740 1 Processor -- start 2024-05-12T22:12:27.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.988+0000 7f4b8a7e9740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cb51948b20 0x55cb5195bed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:27.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.988+0000 7f4b8a7e9740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cb519494d0 0x55cb519591c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:27.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.988+0000 7f4b8a7e9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb5195c5d0 0x55cb51959700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:27.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.988+0000 7f4b8a7e9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55cb518689f0 con 0x55cb5195c5d0 2024-05-12T22:12:27.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.988+0000 7f4b8a7e9740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x55cb518a9b50 con 0x55cb519494d0 2024-05-12T22:12:27.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.988+0000 7f4b8a7e9740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55cb5195f390 con 0x55cb51948b20 2024-05-12T22:12:27.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.989+0000 7f4b8a7e9740 10 client.0 _ll_register_callbacks cb 0x55cb515c0be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:12:27.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.989+0000 7f4b85800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb5195c5d0 0x55cb51959700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:27.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.989+0000 7f4b84e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cb51948b20 0x55cb5195bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:27.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.989+0000 7f4b85800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb5195c5d0 0x55cb51959700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39718/0 (socket says 172.21.15.28:39718) 2024-05-12T22:12:27.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.989+0000 7f4b85800640 1 -- 172.21.15.28:0/2026072682 learned_addr learned my addr 172.21.15.28:0/2026072682 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:12:27.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.989+0000 7f4b7fe00640 1 --2- 172.21.15.28:0/2026072682 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cb519494d0 0x55cb519591c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:27.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.989+0000 7f4b84e00640 1 -- 172.21.15.28:0/2026072682 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cb519494d0 msgr2=0x55cb519591c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:12:27.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.989+0000 7f4b84e00640 1 --2- 172.21.15.28:0/2026072682 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cb519494d0 0x55cb519591c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:27.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.989+0000 7f4b84e00640 1 -- 172.21.15.28:0/2026072682 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb5195c5d0 msgr2=0x55cb51959700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:12:27.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.989+0000 7f4b84e00640 1 --2- 172.21.15.28:0/2026072682 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb5195c5d0 0x55cb51959700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:27.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.989+0000 7f4b84e00640 1 -- 172.21.15.28:0/2026072682 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cb515c13f0 con 0x55cb51948b20 2024-05-12T22:12:27.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.989+0000 7f4b7fe00640 1 --2- 172.21.15.28:0/2026072682 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cb519494d0 0x55cb519591c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:12:27.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.990+0000 7f4b84e00640 1 --2- 172.21.15.28:0/2026072682 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cb51948b20 0x55cb5195bed0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f4b74009d40 tx=0x7f4b74019b10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a6094238bb92c555 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:12:27.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.990+0000 7f4b66a00640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-12T22:12:27.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.990+0000 7f4b66a00640 1 -- 172.21.15.28:0/2026072682 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b74028020 con 0x55cb51948b20 2024-05-12T22:12:27.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.990+0000 7f4b66a00640 1 -- 172.21.15.28:0/2026072682 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f4b74009860 con 0x55cb51948b20 2024-05-12T22:12:27.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.990+0000 7f4b8a7e9740 1 -- 172.21.15.28:0/2026072682 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4b740043c0 con 0x55cb51948b20 2024-05-12T22:12:27.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.991+0000 7f4b66a00640 1 -- 172.21.15.28:0/2026072682 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f4b7401d030 con 0x55cb51948b20 2024-05-12T22:12:27.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.991+0000 7f4b8a7e9740 10 client.26080 fetch_fsmap learned FSMap version 25 2024-05-12T22:12:27.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.991+0000 7f4b8a7e9740 1 -- 172.21.15.28:0/2026072682 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55cb515810f0 con 0x55cb51948b20 2024-05-12T22:12:27.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.991+0000 7f4b66a00640 1 -- 172.21.15.28:0/2026072682 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4b740043c0 con 0x55cb51948b20 2024-05-12T22:12:27.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.992+0000 7f4b66a00640 1 -- 172.21.15.28:0/2026072682 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7f4b74023070 con 0x55cb51948b20 2024-05-12T22:12:27.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.992+0000 7f4b8a7e9740 10 client.26080 fetch_fsmap finished waiting for FSMap version 25 2024-05-12T22:12:27.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.992+0000 7f4b8a7e9740 10 client.26080 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:12:27.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.992+0000 7f4b8a7e9740 20 client.26080 populate_metadata read hostname 'smithi028' 2024-05-12T22:12:27.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.992+0000 7f4b8a7e9740 1 --2- 172.21.15.28:0/2026072682 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55cb5196bfd0 0x55cb5196e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:27.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.992+0000 7f4b8a7e9740 4 client.26080 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-12T22:12:27.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.992+0000 7f4b8a7e9740 1 -- 172.21.15.28:0/2026072682 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cb518bb2c0 con 0x55cb5196bfd0 2024-05-12T22:12:27.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.992+0000 7f4b7fe00640 1 --2- 172.21.15.28:0/2026072682 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55cb5196bfd0 0x55cb5196e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:27.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.993+0000 7f4b7fe00640 1 --2- 172.21.15.28:0/2026072682 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55cb5196bfd0 0x55cb5196e3c0 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1ae60799fc625dc2 server_cookie=d7ab4c42ff8dc22 in_seq=0 out_seq=0 2024-05-12T22:12:27.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.993+0000 7f4b66a00640 10 client.26080 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:12:27.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.994+0000 7f4b66a00640 1 -- 172.21.15.28:0/2026072682 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55cb518bb2c0 con 0x55cb5196bfd0 2024-05-12T22:12:27.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.994+0000 7f4b66a00640 10 client.26080 handle_command_reply: tid=0 2024-05-12T22:12:27.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.994+0000 7f4b8a7e9740 1 client.26080 shutdown 2024-05-12T22:12:27.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.995+0000 7f4b8a7e9740 1 -- 172.21.15.28:0/2026072682 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cb51948b20 msgr2=0x55cb5195bed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:12:28.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.995+0000 7f4b8a7e9740 1 --2- 172.21.15.28:0/2026072682 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cb51948b20 0x55cb5195bed0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f4b74009d40 tx=0x7f4b74019b10 comp rx=0 tx=0).stop 2024-05-12T22:12:28.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.995+0000 7f4b8a7e9740 1 -- 172.21.15.28:0/2026072682 shutdown_connections 2024-05-12T22:12:28.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.995+0000 7f4b8a7e9740 1 --2- 172.21.15.28:0/2026072682 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55cb5196bfd0 0x55cb5196e3c0 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:28.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.995+0000 7f4b8a7e9740 1 --2- 172.21.15.28:0/2026072682 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb5195c5d0 0x55cb51959700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:28.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.995+0000 7f4b8a7e9740 1 --2- 172.21.15.28:0/2026072682 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cb519494d0 0x55cb519591c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:28.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.995+0000 7f4b66a00640 0 client.26080 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:12:28.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.995+0000 7f4b8a7e9740 1 --2- 172.21.15.28:0/2026072682 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cb51948b20 0x55cb5195bed0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:28.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.995+0000 7f4b8a7e9740 1 -- 172.21.15.28:0/2026072682 >> 172.21.15.28:0/2026072682 conn(0x55cb519450f0 msgr2=0x55cb5194aef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:12:28.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.995+0000 7f4b8a7e9740 1 -- 172.21.15.28:0/2026072682 shutdown_connections 2024-05-12T22:12:28.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.995+0000 7f4b8a7e9740 1 -- 172.21.15.28:0/2026072682 wait complete. 2024-05-12T22:12:28.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:27.995+0000 7f4b8a7e9740 20 client.26080 trim_cache size 0 max 16384 2024-05-12T22:12:28.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:28.022+0000 7f4b8a7e9740 1 Processor -- start 2024-05-12T22:12:28.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:28.022+0000 7f4b8a7e9740 1 -- start start 2024-05-12T22:12:28.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:28.022+0000 7f4b8a7e9740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cb51948b20 0x55cb51944330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:28.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:28.022+0000 7f4b8a7e9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb519494d0 0x55cb51958970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:28.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:28.022+0000 7f4b8a7e9740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cb5195c5d0 0x55cb5194ad70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:28.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:28.022+0000 7f4b8a7e9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4b801ada90 con 0x55cb519494d0 2024-05-12T22:12:28.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:28.022+0000 7f4b8a7e9740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f4b801ab360 con 0x55cb5195c5d0 2024-05-12T22:12:28.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:28.022+0000 7f4b8a7e9740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f4b801a91b0 con 0x55cb51948b20 2024-05-12T22:12:28.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:28.023+0000 7f4b84e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cb51948b20 0x55cb51944330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:28.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:28.023+0000 7f4b84e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cb51948b20 0x55cb51944330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:56074/0 (socket says 172.21.15.28:56074) 2024-05-12T22:12:28.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:28.023+0000 7f4b84e00640 1 -- 172.21.15.28:0/2447128847 learned_addr learned my addr 172.21.15.28:0/2447128847 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:12:28.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:28.023+0000 7f4b85800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cb5195c5d0 0x55cb5194ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:28.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:28.023+0000 7f4b7fe00640 1 --2- 172.21.15.28:0/2447128847 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb519494d0 0x55cb51958970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:28.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:28.023+0000 7f4b85800640 1 -- 172.21.15.28:0/2447128847 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cb51948b20 msgr2=0x55cb51944330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:12:28.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:28.023+0000 7f4b85800640 1 --2- 172.21.15.28:0/2447128847 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cb51948b20 0x55cb51944330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:28.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:28.023+0000 7f4b85800640 1 -- 172.21.15.28:0/2447128847 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb519494d0 msgr2=0x55cb51958970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:12:28.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:28.023+0000 7f4b85800640 1 --2- 172.21.15.28:0/2447128847 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb519494d0 0x55cb51958970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:28.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:28.023+0000 7f4b85800640 1 -- 172.21.15.28:0/2447128847 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cb51944f20 con 0x55cb5195c5d0 2024-05-12T22:12:28.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:28.024+0000 7f4b85800640 1 --2- 172.21.15.28:0/2447128847 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cb5195c5d0 0x55cb5194ad70 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f4b78098680 tx=0x7f4b7809b6d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5258aeeb2f27b5f server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:12:28.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:28.024+0000 7f4b71600640 1 -- 172.21.15.28:0/2447128847 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b780a0060 con 0x55cb5195c5d0 2024-05-12T22:12:28.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:28.024+0000 7f4b71600640 1 -- 172.21.15.28:0/2447128847 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4b780a4920 con 0x55cb5195c5d0 2024-05-12T22:12:28.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:28.024+0000 7f4b71600640 1 -- 172.21.15.28:0/2447128847 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b780cd160 con 0x55cb5195c5d0 2024-05-12T22:12:28.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:28.025+0000 7f4b8a7e9740 1 -- 172.21.15.28:0/2447128847 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cb5195c5d0 msgr2=0x55cb5194ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:12:28.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:28.025+0000 7f4b8a7e9740 1 --2- 172.21.15.28:0/2447128847 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cb5195c5d0 0x55cb5194ad70 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f4b78098680 tx=0x7f4b7809b6d0 comp rx=0 tx=0).stop 2024-05-12T22:12:28.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:28.025+0000 7f4b8a7e9740 1 -- 172.21.15.28:0/2447128847 shutdown_connections 2024-05-12T22:12:28.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:28.025+0000 7f4b8a7e9740 1 --2- 172.21.15.28:0/2447128847 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cb5195c5d0 0x55cb5194ad70 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:28.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:28.025+0000 7f4b8a7e9740 1 --2- 172.21.15.28:0/2447128847 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb519494d0 0x55cb51958970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:28.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:28.025+0000 7f4b8a7e9740 1 --2- 172.21.15.28:0/2447128847 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cb51948b20 0x55cb51944330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:28.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:28.025+0000 7f4b8a7e9740 1 -- 172.21.15.28:0/2447128847 >> 172.21.15.28:0/2447128847 conn(0x55cb519450f0 msgr2=0x55cb5194a660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:12:28.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:28.025+0000 7f4b8a7e9740 1 -- 172.21.15.28:0/2447128847 shutdown_connections 2024-05-12T22:12:28.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:28.025+0000 7f4b8a7e9740 1 -- 172.21.15.28:0/2447128847 wait complete. 2024-05-12T22:12:28.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:28.026+0000 7f4b8a7e9740 1 -- start start 2024-05-12T22:12:28.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:28.026+0000 7f4b8a7e9740 1 Processor -- start 2024-05-12T22:12:28.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:28.027+0000 7f4b8a7e9740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cb51948b20 0x55cb5195f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:28.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:28.027+0000 7f4b8a7e9740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cb519494d0 0x55cb5195fd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:28.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:28.027+0000 7f4b8a7e9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb5195c5d0 0x55cb519d03b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:28.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:28.027+0000 7f4b8a7e9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4b801a9820 con 0x55cb5195c5d0 2024-05-12T22:12:28.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:28.027+0000 7f4b8a7e9740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f4b801aac90 con 0x55cb51948b20 2024-05-12T22:12:28.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:28.027+0000 7f4b8a7e9740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f4b801a9ef0 con 0x55cb519494d0 2024-05-12T22:12:28.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:28.027+0000 7f4b8a7e9740 10 client.0 _ll_register_callbacks cb 0x7f4b74009bc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:12:28.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:28.027+0000 7f4b84e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cb51948b20 0x55cb5195f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:28.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:28.027+0000 7f4b7fe00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cb519494d0 0x55cb5195fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:28.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:28.027+0000 7f4b84e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cb51948b20 0x55cb5195f7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:49588/0 (socket says 172.21.15.28:49588) 2024-05-12T22:12:28.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:28.027+0000 7f4b84e00640 1 -- 172.21.15.28:0/3389881595 learned_addr learned my addr 172.21.15.28:0/3389881595 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:12:28.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:28.027+0000 7f4b85800640 1 --2- 172.21.15.28:0/3389881595 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb5195c5d0 0x55cb519d03b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:28.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:28.028+0000 7f4b84e00640 1 -- 172.21.15.28:0/3389881595 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cb519494d0 msgr2=0x55cb5195fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:12:28.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:28.028+0000 7f4b84e00640 1 --2- 172.21.15.28:0/3389881595 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cb519494d0 0x55cb5195fd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:28.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:28.028+0000 7f4b84e00640 1 -- 172.21.15.28:0/3389881595 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb5195c5d0 msgr2=0x55cb519d03b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:12:28.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:28.028+0000 7f4b84e00640 1 --2- 172.21.15.28:0/3389881595 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb5195c5d0 0x55cb519d03b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:28.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:28.028+0000 7f4b84e00640 1 -- 172.21.15.28:0/3389881595 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cb519d08f0 con 0x55cb51948b20 2024-05-12T22:12:28.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:28.028+0000 7f4b84e00640 1 --2- 172.21.15.28:0/3389881595 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cb51948b20 0x55cb5195f7d0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f4b740048f0 tx=0x7f4b74009d00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4acc9cb6d3580393 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:12:28.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:28.028+0000 7f4b66a00640 10 client.0 ms_handle_connect on v2:172.21.15.86:3300/0 2024-05-12T22:12:28.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:28.029+0000 7f4b66a00640 1 -- 172.21.15.28:0/3389881595 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b74028040 con 0x55cb51948b20 2024-05-12T22:12:28.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:28.029+0000 7f4b66a00640 1 -- 172.21.15.28:0/3389881595 <== mon.2 v2:172.21.15.86:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f4b7401e070 con 0x55cb51948b20 2024-05-12T22:12:28.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:28.029+0000 7f4b8a7e9740 1 -- 172.21.15.28:0/3389881595 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4b780a4920 con 0x55cb51948b20 2024-05-12T22:12:28.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:28.029+0000 7f4b66a00640 1 -- 172.21.15.28:0/3389881595 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f4b74019740 con 0x55cb51948b20 2024-05-12T22:12:28.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:28.029+0000 7f4b8a7e9740 10 client.26132 fetch_fsmap learned FSMap version 25 2024-05-12T22:12:28.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:28.029+0000 7f4b8a7e9740 1 -- 172.21.15.28:0/3389881595 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55cb519d1bb0 con 0x55cb51948b20 2024-05-12T22:12:28.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:28.029+0000 7f4b66a00640 1 -- 172.21.15.28:0/3389881595 <== mon.2 v2:172.21.15.86:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4b780a4920 con 0x55cb51948b20 2024-05-12T22:12:28.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:28.030+0000 7f4b66a00640 1 -- 172.21.15.28:0/3389881595 <== mon.2 v2:172.21.15.86:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7f4b74023040 con 0x55cb51948b20 2024-05-12T22:12:28.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:28.030+0000 7f4b8a7e9740 10 client.26132 fetch_fsmap finished waiting for FSMap version 25 2024-05-12T22:12:28.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:28.030+0000 7f4b8a7e9740 10 client.26132 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:12:28.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:28.030+0000 7f4b8a7e9740 20 client.26132 populate_metadata read hostname 'smithi028' 2024-05-12T22:12:28.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:28.030+0000 7f4b8a7e9740 1 --2- 172.21.15.28:0/3389881595 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55cb519d2270 0x55cb519d4690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:28.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:28.030+0000 7f4b8a7e9740 4 client.26132 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-12T22:12:28.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:28.030+0000 7f4b8a7e9740 1 -- 172.21.15.28:0/3389881595 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55cb518b5610 con 0x55cb519d2270 2024-05-12T22:12:28.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:28.030+0000 7f4b7fe00640 1 --2- 172.21.15.28:0/3389881595 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55cb519d2270 0x55cb519d4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:28.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:28.031+0000 7f4b7fe00640 1 --2- 172.21.15.28:0/3389881595 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55cb519d2270 0x55cb519d4690 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f4cf92c9fc2199a4 server_cookie=911e1fc15cf792d5 in_seq=0 out_seq=0 2024-05-12T22:12:28.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:28.031+0000 7f4b66a00640 10 client.26132 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:12:28.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:28.031+0000 7f4b66a00640 1 -- 172.21.15.28:0/3389881595 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+268 (crc 0 0 0) 0x55cb518b5610 con 0x55cb519d2270 2024-05-12T22:12:28.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:28.031+0000 7f4b66a00640 10 client.26132 handle_command_reply: tid=0 2024-05-12T22:12:28.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:28.031+0000 7f4b8a7e9740 1 client.26132 shutdown 2024-05-12T22:12:28.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:28.031+0000 7f4b8a7e9740 1 -- 172.21.15.28:0/3389881595 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cb51948b20 msgr2=0x55cb5195f7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:12:28.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:28.031+0000 7f4b8a7e9740 1 --2- 172.21.15.28:0/3389881595 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cb51948b20 0x55cb5195f7d0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f4b740048f0 tx=0x7f4b74009d00 comp rx=0 tx=0).stop 2024-05-12T22:12:28.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:28.031+0000 7f4b8a7e9740 1 -- 172.21.15.28:0/3389881595 shutdown_connections 2024-05-12T22:12:28.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:28.031+0000 7f4b8a7e9740 1 --2- 172.21.15.28:0/3389881595 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55cb519d2270 0x55cb519d4690 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:28.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:28.032+0000 7f4b8a7e9740 1 --2- 172.21.15.28:0/3389881595 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb5195c5d0 0x55cb519d03b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:28.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:28.032+0000 7f4b8a7e9740 1 --2- 172.21.15.28:0/3389881595 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cb519494d0 0x55cb5195fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:28.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:28.032+0000 7f4b8a7e9740 1 --2- 172.21.15.28:0/3389881595 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cb51948b20 0x55cb5195f7d0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:28.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:28.032+0000 7f4b8a7e9740 1 -- 172.21.15.28:0/3389881595 >> 172.21.15.28:0/3389881595 conn(0x55cb519450f0 msgr2=0x55cb5194aef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:12:28.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:28.032+0000 7f4b66a00640 0 client.26132 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:12:28.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:28.032+0000 7f4b8a7e9740 1 -- 172.21.15.28:0/3389881595 shutdown_connections 2024-05-12T22:12:28.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:28.032+0000 7f4b8a7e9740 1 -- 172.21.15.28:0/3389881595 wait complete. 2024-05-12T22:12:28.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:28.032+0000 7f4b8a7e9740 20 client.26132 trim_cache size 0 max 16384 2024-05-12T22:12:28.038 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-05-12T22:12:28.038 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "scrub active (1237 inodes in the stack)", 2024-05-12T22:12:28.038 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-05-12T22:12:28.038 INFO:teuthology.orchestra.run.smithi028.stdout: "2d602076-1a6b-40b2-9519-9ad565de1683": { 2024-05-12T22:12:28.038 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-05-12T22:12:28.038 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "2d602076-1a6b-40b2-9519-9ad565de1683", 2024-05-12T22:12:28.038 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-05-12T22:12:28.039 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:12:28.039 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:12:28.039 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-05-12T22:12:28.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:28.033+0000 7f4b86200640 1 -- 172.21.15.28:0/680334726 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f4b680a2d80 msgr2=0x7f4b680a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:12:28.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:28.033+0000 7f4b86200640 1 --2- 172.21.15.28:0/680334726 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f4b680a2d80 0x7f4b680a5240 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f4b6c000c00 tx=0x7f4b6c02a3f0 comp rx=0 tx=0).stop 2024-05-12T22:12:28.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:28.033+0000 7f4b86200640 1 -- 172.21.15.28:0/680334726 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4b801098f0 msgr2=0x7f4b8019eab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:12:28.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:28.033+0000 7f4b86200640 1 --2- 172.21.15.28:0/680334726 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4b801098f0 0x7f4b8019eab0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f4b7800da10 tx=0x7f4b7800ded0 comp rx=0 tx=0).stop 2024-05-12T22:12:28.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:28.033+0000 7f4b86200640 1 -- 172.21.15.28:0/680334726 shutdown_connections 2024-05-12T22:12:28.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:28.033+0000 7f4b86200640 1 --2- 172.21.15.28:0/680334726 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f4b680a2d80 0x7f4b680a5240 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:28.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:28.033+0000 7f4b86200640 1 --2- 172.21.15.28:0/680334726 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4b801098f0 0x7f4b8019eab0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:28.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:28.033+0000 7f4b86200640 1 --2- 172.21.15.28:0/680334726 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b801042a0 0x7f4b8019a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:28.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:28.033+0000 7f4b86200640 1 --2- 172.21.15.28:0/680334726 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4b801038e0 0x7f4b8019a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:28.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:28.033+0000 7f4b86200640 1 -- 172.21.15.28:0/680334726 >> 172.21.15.28:0/680334726 conn(0x7f4b800ff370 msgr2=0x7f4b800ffb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:12:28.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:28.033+0000 7f4b86200640 1 -- 172.21.15.28:0/680334726 shutdown_connections 2024-05-12T22:12:28.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:28.034+0000 7f4b86200640 1 -- 172.21.15.28:0/680334726 wait complete. 2024-05-12T22:12:28.055 INFO:tasks.cephfs.filesystem:scrub status for tag:2d602076-1a6b-40b2-9519-9ad565de1683 - {'path': '/', 'tag': '2d602076-1a6b-40b2-9519-9ad565de1683', 'options': 'recursive,force'} 2024-05-12T22:12:28.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:12:28 smithi186 ceph-mon[34785]: pgmap v107: 97 pgs: 97 active+clean; 97 MiB data, 752 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.8 KiB/s wr, 5 op/s 2024-05-12T22:12:28.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:12:28 smithi186 ceph-mon[34785]: scrub summary: active paths [/] 2024-05-12T22:12:28.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:12:28 smithi186 ceph-mon[34785]: scrub summary: idle+waiting paths [/] 2024-05-12T22:12:28.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:12:28 smithi186 ceph-mon[34785]: scrub summary: active paths [/] 2024-05-12T22:12:28.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:12:28 smithi186 ceph-mon[34785]: scrub summary: idle+waiting paths [/] 2024-05-12T22:12:28.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:12:28 smithi086 ceph-mon[34297]: pgmap v107: 97 pgs: 97 active+clean; 97 MiB data, 752 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.8 KiB/s wr, 5 op/s 2024-05-12T22:12:28.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:12:28 smithi086 ceph-mon[34297]: scrub summary: active paths [/] 2024-05-12T22:12:28.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:12:28 smithi086 ceph-mon[34297]: scrub summary: idle+waiting paths [/] 2024-05-12T22:12:28.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:12:28 smithi086 ceph-mon[34297]: scrub summary: active paths [/] 2024-05-12T22:12:28.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:12:28 smithi086 ceph-mon[34297]: scrub summary: idle+waiting paths [/] 2024-05-12T22:12:28.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:12:28 smithi028 ceph-mon[29847]: pgmap v107: 97 pgs: 97 active+clean; 97 MiB data, 752 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.8 KiB/s wr, 5 op/s 2024-05-12T22:12:28.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:12:28 smithi028 ceph-mon[29847]: scrub summary: active paths [/] 2024-05-12T22:12:28.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:12:28 smithi028 ceph-mon[29847]: scrub summary: idle+waiting paths [/] 2024-05-12T22:12:28.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:12:28 smithi028 ceph-mon[29847]: scrub summary: active paths [/] 2024-05-12T22:12:28.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:12:28 smithi028 ceph-mon[29847]: scrub summary: idle+waiting paths [/] 2024-05-12T22:12:29.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:12:29 smithi186 ceph-mon[34785]: scrub queued for path: / 2024-05-12T22:12:29.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:12:29 smithi186 ceph-mon[34785]: scrub summary: idle+waiting paths [/] 2024-05-12T22:12:29.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:12:29 smithi186 ceph-mon[34785]: scrub summary: active paths [/] 2024-05-12T22:12:29.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:12:29 smithi186 ceph-mon[34785]: scrub summary: active paths [/] 2024-05-12T22:12:29.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:12:29 smithi186 ceph-mon[34785]: scrub summary: idle+waiting paths [/] 2024-05-12T22:12:29.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:12:29 smithi186 ceph-mon[34785]: scrub summary: idle+waiting paths [/] 2024-05-12T22:12:29.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:12:29 smithi086 ceph-mon[34297]: scrub queued for path: / 2024-05-12T22:12:29.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:12:29 smithi086 ceph-mon[34297]: scrub summary: idle+waiting paths [/] 2024-05-12T22:12:29.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:12:29 smithi086 ceph-mon[34297]: scrub summary: active paths [/] 2024-05-12T22:12:29.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:12:29 smithi086 ceph-mon[34297]: scrub summary: active paths [/] 2024-05-12T22:12:29.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:12:29 smithi086 ceph-mon[34297]: scrub summary: idle+waiting paths [/] 2024-05-12T22:12:29.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:12:29 smithi086 ceph-mon[34297]: scrub summary: idle+waiting paths [/] 2024-05-12T22:12:29.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:12:29 smithi028 ceph-mon[29847]: scrub queued for path: / 2024-05-12T22:12:29.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:12:29 smithi028 ceph-mon[29847]: scrub summary: idle+waiting paths [/] 2024-05-12T22:12:29.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:12:29 smithi028 ceph-mon[29847]: scrub summary: active paths [/] 2024-05-12T22:12:29.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:12:29 smithi028 ceph-mon[29847]: scrub summary: active paths [/] 2024-05-12T22:12:29.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:12:29 smithi028 ceph-mon[29847]: scrub summary: idle+waiting paths [/] 2024-05-12T22:12:29.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:12:29 smithi028 ceph-mon[29847]: scrub summary: idle+waiting paths [/] 2024-05-12T22:12:30.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:12:30 smithi186 ceph-mon[34785]: pgmap v108: 97 pgs: 97 active+clean; 97 MiB data, 752 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.8 KiB/s wr, 5 op/s 2024-05-12T22:12:30.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:12:30 smithi086 ceph-mon[34297]: pgmap v108: 97 pgs: 97 active+clean; 97 MiB data, 752 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.8 KiB/s wr, 5 op/s 2024-05-12T22:12:30.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:12:30 smithi028 ceph-mon[29847]: pgmap v108: 97 pgs: 97 active+clean; 97 MiB data, 752 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.8 KiB/s wr, 5 op/s 2024-05-12T22:12:32.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:12:32 smithi186 ceph-mon[34785]: pgmap v109: 97 pgs: 97 active+clean; 97 MiB data, 752 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 1.8 KiB/s wr, 8 op/s 2024-05-12T22:12:32.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:12:32 smithi086 ceph-mon[34297]: pgmap v109: 97 pgs: 97 active+clean; 97 MiB data, 752 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 1.8 KiB/s wr, 8 op/s 2024-05-12T22:12:32.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:12:32 smithi028 ceph-mon[29847]: pgmap v109: 97 pgs: 97 active+clean; 97 MiB data, 752 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 1.8 KiB/s wr, 8 op/s 2024-05-12T22:12:34.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:12:34 smithi186 ceph-mon[34785]: pgmap v110: 97 pgs: 97 active+clean; 97 MiB data, 752 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 1.8 KiB/s wr, 6 op/s 2024-05-12T22:12:34.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:12:34 smithi086 ceph-mon[34297]: pgmap v110: 97 pgs: 97 active+clean; 97 MiB data, 752 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 1.8 KiB/s wr, 6 op/s 2024-05-12T22:12:34.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:12:34 smithi028 ceph-mon[29847]: pgmap v110: 97 pgs: 97 active+clean; 97 MiB data, 752 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 1.8 KiB/s wr, 6 op/s 2024-05-12T22:12:36.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:12:36 smithi186 ceph-mon[34785]: pgmap v111: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 3.2 KiB/s wr, 9 op/s 2024-05-12T22:12:36.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:12:36 smithi086 ceph-mon[34297]: pgmap v111: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 3.2 KiB/s wr, 9 op/s 2024-05-12T22:12:36.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:12:36 smithi028 ceph-mon[29847]: pgmap v111: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 3.2 KiB/s wr, 9 op/s 2024-05-12T22:12:38.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:12:38 smithi186 ceph-mon[34785]: pgmap v112: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.3 KiB/s wr, 6 op/s 2024-05-12T22:12:38.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:12:38 smithi186 ceph-mon[34785]: scrub summary: idle 2024-05-12T22:12:38.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:12:38 smithi186 ceph-mon[34785]: scrub summary: idle 2024-05-12T22:12:38.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:12:38 smithi086 ceph-mon[34297]: pgmap v112: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.3 KiB/s wr, 6 op/s 2024-05-12T22:12:38.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:12:38 smithi086 ceph-mon[34297]: scrub summary: idle 2024-05-12T22:12:38.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:12:38 smithi086 ceph-mon[34297]: scrub summary: idle 2024-05-12T22:12:38.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:12:38 smithi028 ceph-mon[29847]: pgmap v112: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.3 KiB/s wr, 6 op/s 2024-05-12T22:12:38.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:12:38 smithi028 ceph-mon[29847]: scrub summary: idle 2024-05-12T22:12:38.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:12:38 smithi028 ceph-mon[29847]: scrub summary: idle 2024-05-12T22:12:40.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:12:40 smithi186 ceph-mon[34785]: pgmap v113: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.3 KiB/s wr, 6 op/s 2024-05-12T22:12:40.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:12:40 smithi086 ceph-mon[34297]: pgmap v113: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.3 KiB/s wr, 6 op/s 2024-05-12T22:12:40.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:12:40 smithi028 ceph-mon[29847]: pgmap v113: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.3 KiB/s wr, 6 op/s 2024-05-12T22:12:42.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:12:42 smithi186 ceph-mon[34785]: pgmap v114: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.3 KiB/s wr, 9 op/s 2024-05-12T22:12:42.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:12:42 smithi086 ceph-mon[34297]: pgmap v114: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.3 KiB/s wr, 9 op/s 2024-05-12T22:12:42.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:12:42 smithi028 ceph-mon[29847]: pgmap v114: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.3 KiB/s wr, 9 op/s 2024-05-12T22:12:44.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:12:44 smithi086 ceph-mon[34297]: pgmap v115: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.3 KiB/s wr, 6 op/s 2024-05-12T22:12:44.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:12:44 smithi028 ceph-mon[29847]: pgmap v115: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.3 KiB/s wr, 6 op/s 2024-05-12T22:12:44.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:12:44 smithi186 ceph-mon[34785]: pgmap v115: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.3 KiB/s wr, 6 op/s 2024-05-12T22:12:46.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:12:46 smithi086 ceph-mon[34297]: pgmap v116: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 1.3 KiB/s wr, 8 op/s 2024-05-12T22:12:46.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:12:46 smithi028 ceph-mon[29847]: pgmap v116: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 1.3 KiB/s wr, 8 op/s 2024-05-12T22:12:46.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:12:46 smithi186 ceph-mon[34785]: pgmap v116: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 1.3 KiB/s wr, 8 op/s 2024-05-12T22:12:48.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:12:48 smithi086 ceph-mon[34297]: pgmap v117: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-12T22:12:48.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:12:48 smithi028 ceph-mon[29847]: pgmap v117: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-12T22:12:48.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:12:48 smithi186 ceph-mon[34785]: pgmap v117: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-12T22:12:50.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:12:50 smithi086 ceph-mon[34297]: pgmap v118: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-12T22:12:50.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:12:50 smithi028 ceph-mon[29847]: pgmap v118: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-12T22:12:50.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:12:50 smithi186 ceph-mon[34785]: pgmap v118: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-12T22:12:52.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:12:52 smithi086 ceph-mon[34297]: pgmap v119: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 8 op/s 2024-05-12T22:12:52.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:12:52 smithi028 ceph-mon[29847]: pgmap v119: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 8 op/s 2024-05-12T22:12:52.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:12:52 smithi186 ceph-mon[34785]: pgmap v119: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 8 op/s 2024-05-12T22:12:54.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:12:54 smithi086 ceph-mon[34297]: pgmap v120: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-05-12T22:12:54.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:12:54 smithi028 ceph-mon[29847]: pgmap v120: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-05-12T22:12:54.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:12:54 smithi186 ceph-mon[34785]: pgmap v120: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-05-12T22:12:56.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:12:56 smithi086 ceph-mon[34297]: pgmap v121: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 3.6 KiB/s wr, 8 op/s 2024-05-12T22:12:56.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:12:56 smithi028 ceph-mon[29847]: pgmap v121: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 3.6 KiB/s wr, 8 op/s 2024-05-12T22:12:56.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:12:56 smithi186 ceph-mon[34785]: pgmap v121: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 3.6 KiB/s wr, 8 op/s 2024-05-12T22:12:58.057 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-12T22:12:58.057 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-12T22:12:58.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.283+0000 7fb0a6600640 1 -- 172.21.15.28:0/1005596522 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0a0100680 msgr2=0x7fb0a0100a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:12:58.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.283+0000 7fb0a6600640 1 --2- 172.21.15.28:0/1005596522 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0a0100680 0x7fb0a0100a60 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fb090009ca0 tx=0x7fb09002a940 comp rx=0 tx=0).stop 2024-05-12T22:12:58.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.283+0000 7fb0a6600640 1 -- 172.21.15.28:0/1005596522 shutdown_connections 2024-05-12T22:12:58.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.283+0000 7fb0a6600640 1 --2- 172.21.15.28:0/1005596522 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb0a0105a80 0x7fb0a0105f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:58.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.283+0000 7fb0a6600640 1 --2- 172.21.15.28:0/1005596522 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb0a0100fa0 0x7fb0a0105460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:58.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.283+0000 7fb0a6600640 1 --2- 172.21.15.28:0/1005596522 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0a0100680 0x7fb0a0100a60 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:58.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.283+0000 7fb0a6600640 1 -- 172.21.15.28:0/1005596522 >> 172.21.15.28:0/1005596522 conn(0x7fb0a00fc0b0 msgr2=0x7fb0a00fe4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:12:58.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.283+0000 7fb0a6600640 1 -- 172.21.15.28:0/1005596522 shutdown_connections 2024-05-12T22:12:58.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.283+0000 7fb0a6600640 1 -- 172.21.15.28:0/1005596522 wait complete. 2024-05-12T22:12:58.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.285+0000 7fb0a6600640 1 Processor -- start 2024-05-12T22:12:58.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.285+0000 7fb0a6600640 1 -- start start 2024-05-12T22:12:58.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.285+0000 7fb0a6600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb0a0100680 0x7fb0a019e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:58.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.285+0000 7fb0a6600640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb0a0100fa0 0x7fb0a019eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:58.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.285+0000 7fb0a6600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0a0105a80 0x7fb0a01a2e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:58.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.285+0000 7fb0a6600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb0a01180f0 con 0x7fb0a0105a80 2024-05-12T22:12:58.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.285+0000 7fb0a6600640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fb0a0117f70 con 0x7fb0a0100680 2024-05-12T22:12:58.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.285+0000 7fb0a6600640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb0a0118270 con 0x7fb0a0100fa0 2024-05-12T22:12:58.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.286+0000 7fb0a5200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb0a0100680 0x7fb0a019e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:58.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.286+0000 7fb0a5c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0a0105a80 0x7fb0a01a2e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:58.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.286+0000 7fb0a5200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb0a0100680 0x7fb0a019e5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:46080/0 (socket says 172.21.15.28:46080) 2024-05-12T22:12:58.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.286+0000 7fb0a5200640 1 -- 172.21.15.28:0/3634307427 learned_addr learned my addr 172.21.15.28:0/3634307427 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:12:58.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.286+0000 7fb09fe00640 1 --2- 172.21.15.28:0/3634307427 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb0a0100fa0 0x7fb0a019eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:58.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.286+0000 7fb0a5200640 1 -- 172.21.15.28:0/3634307427 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb0a0100fa0 msgr2=0x7fb0a019eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:12:58.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.286+0000 7fb0a5200640 1 --2- 172.21.15.28:0/3634307427 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb0a0100fa0 0x7fb0a019eaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:58.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.286+0000 7fb0a5200640 1 -- 172.21.15.28:0/3634307427 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0a0105a80 msgr2=0x7fb0a01a2e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:12:58.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.286+0000 7fb0a5200640 1 --2- 172.21.15.28:0/3634307427 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0a0105a80 0x7fb0a01a2e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:58.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.286+0000 7fb0a5200640 1 -- 172.21.15.28:0/3634307427 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0a01a35a0 con 0x7fb0a0100680 2024-05-12T22:12:58.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.287+0000 7fb0a5200640 1 --2- 172.21.15.28:0/3634307427 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb0a0100680 0x7fb0a019e5b0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fb090004af0 tx=0x7fb09002f6b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:12:58.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.287+0000 7fb09d600640 1 -- 172.21.15.28:0/3634307427 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb090034040 con 0x7fb0a0100680 2024-05-12T22:12:58.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.287+0000 7fb09d600640 1 -- 172.21.15.28:0/3634307427 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb09002fe10 con 0x7fb0a0100680 2024-05-12T22:12:58.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.287+0000 7fb0a6600640 1 -- 172.21.15.28:0/3634307427 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0a01a3830 con 0x7fb0a0100680 2024-05-12T22:12:58.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.287+0000 7fb09d600640 1 -- 172.21.15.28:0/3634307427 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0900045d0 con 0x7fb0a0100680 2024-05-12T22:12:58.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.287+0000 7fb0a6600640 1 -- 172.21.15.28:0/3634307427 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0a01ab120 con 0x7fb0a0100680 2024-05-12T22:12:58.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.290+0000 7fb09d600640 1 -- 172.21.15.28:0/3634307427 <== mon.2 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7fb090037020 con 0x7fb0a0100680 2024-05-12T22:12:58.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.290+0000 7fb0aaa22740 1 Processor -- start 2024-05-12T22:12:58.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.291+0000 7fb0aaa22740 1 -- start start 2024-05-12T22:12:58.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.291+0000 7fb0aaa22740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56176096edd0 0x56176096f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:58.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.291+0000 7fb0aaa22740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56176096f6f0 0x561760979880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:58.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.291+0000 7fb0aaa22740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561760979dc0 0x56176097c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:58.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.291+0000 7fb0aaa22740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5617609372a0 con 0x56176096f6f0 2024-05-12T22:12:58.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.291+0000 7fb0aaa22740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x5617608d3b90 con 0x561760979dc0 2024-05-12T22:12:58.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.291+0000 7fb0aaa22740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x56176087b5b0 con 0x56176096edd0 2024-05-12T22:12:58.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.291+0000 7fb0a5c00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56176096edd0 0x56176096f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:58.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.291+0000 7fb0a5c00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56176096edd0 0x56176096f1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:35824/0 (socket says 172.21.15.28:35824) 2024-05-12T22:12:58.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.291+0000 7fb0a5200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56176096f6f0 0x561760979880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:58.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.291+0000 7fb0a5c00640 1 -- 172.21.15.28:0/4063337256 learned_addr learned my addr 172.21.15.28:0/4063337256 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:12:58.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.292+0000 7fb09fe00640 1 --2- 172.21.15.28:0/4063337256 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561760979dc0 0x56176097c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:58.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.292+0000 7fb0a5c00640 1 -- 172.21.15.28:0/4063337256 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561760979dc0 msgr2=0x56176097c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:12:58.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.292+0000 7fb0a5c00640 1 --2- 172.21.15.28:0/4063337256 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561760979dc0 0x56176097c1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:58.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.292+0000 7fb0a5c00640 1 -- 172.21.15.28:0/4063337256 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56176096f6f0 msgr2=0x561760979880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:12:58.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.292+0000 7fb0a5c00640 1 --2- 172.21.15.28:0/4063337256 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56176096f6f0 0x561760979880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:58.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.292+0000 7fb0a5c00640 1 -- 172.21.15.28:0/4063337256 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5617606b17d0 con 0x56176096edd0 2024-05-12T22:12:58.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.292+0000 7fb0a5c00640 1 --2- 172.21.15.28:0/4063337256 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56176096edd0 0x56176096f1b0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fb098009c10 tx=0x7fb098011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7f23a34b7cdea0fb server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:12:58.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.292+0000 7fb096000640 1 -- 172.21.15.28:0/4063337256 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb098004230 con 0x56176096edd0 2024-05-12T22:12:58.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.293+0000 7fb09d600640 1 --2- 172.21.15.28:0/3634307427 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fb0840a2d00 0x7fb0840a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:58.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.293+0000 7fb096000640 1 -- 172.21.15.28:0/4063337256 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb0980043d0 con 0x56176096edd0 2024-05-12T22:12:58.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.293+0000 7fb096000640 1 -- 172.21.15.28:0/4063337256 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb098019730 con 0x56176096edd0 2024-05-12T22:12:58.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.293+0000 7fb09d600640 1 -- 172.21.15.28:0/3634307427 <== mon.2 v2:172.21.15.86:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7fb0900f0100 con 0x7fb0a0100680 2024-05-12T22:12:58.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.293+0000 7fb0a5200640 1 --2- 172.21.15.28:0/3634307427 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fb0840a2d00 0x7fb0840a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:58.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.293+0000 7fb0aaa22740 1 -- 172.21.15.28:0/4063337256 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56176096edd0 msgr2=0x56176096f1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:12:58.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.293+0000 7fb0aaa22740 1 --2- 172.21.15.28:0/4063337256 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56176096edd0 0x56176096f1b0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fb098009c10 tx=0x7fb098011810 comp rx=0 tx=0).stop 2024-05-12T22:12:58.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.293+0000 7fb0aaa22740 1 -- 172.21.15.28:0/4063337256 shutdown_connections 2024-05-12T22:12:58.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.293+0000 7fb0aaa22740 1 --2- 172.21.15.28:0/4063337256 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561760979dc0 0x56176097c1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:58.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.293+0000 7fb0aaa22740 1 --2- 172.21.15.28:0/4063337256 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56176096f6f0 0x561760979880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:58.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.293+0000 7fb0aaa22740 1 --2- 172.21.15.28:0/4063337256 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56176096edd0 0x56176096f1b0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:58.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.293+0000 7fb0aaa22740 1 -- 172.21.15.28:0/4063337256 >> 172.21.15.28:0/4063337256 conn(0x56176096b0f0 msgr2=0x56176096b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:12:58.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.294+0000 7fb0aaa22740 1 -- 172.21.15.28:0/4063337256 shutdown_connections 2024-05-12T22:12:58.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.294+0000 7fb0aaa22740 1 -- 172.21.15.28:0/4063337256 wait complete. 2024-05-12T22:12:58.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.294+0000 7fb0a5200640 1 --2- 172.21.15.28:0/3634307427 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fb0840a2d00 0x7fb0840a51c0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x561760970390 tx=0x7fb0900040d0 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:12:58.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.294+0000 7fb0aaa22740 1 -- start start 2024-05-12T22:12:58.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.294+0000 7fb0aaa22740 1 Processor -- start 2024-05-12T22:12:58.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.295+0000 7fb0aaa22740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56176096edd0 0x561760979710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:58.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.295+0000 7fb0aaa22740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56176096f6f0 0x561760976a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:58.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.295+0000 7fb0aaa22740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561760979dc0 0x561760976f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:58.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.295+0000 7fb0aaa22740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56176088e9f0 con 0x56176096f6f0 2024-05-12T22:12:58.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.295+0000 7fb0aaa22740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x5617608cfb50 con 0x56176096edd0 2024-05-12T22:12:58.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.295+0000 7fb0aaa22740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x56176097cbb0 con 0x561760979dc0 2024-05-12T22:12:58.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.295+0000 7fb0aaa22740 10 client.0 _ll_register_callbacks cb 0x5617605e6be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:12:58.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.295+0000 7fb09fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56176096f6f0 0x561760976a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:58.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.295+0000 7fb09fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56176096f6f0 0x561760976a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36722/0 (socket says 172.21.15.28:36722) 2024-05-12T22:12:58.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.295+0000 7fb09fe00640 1 -- 172.21.15.28:0/113545886 learned_addr learned my addr 172.21.15.28:0/113545886 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:12:58.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.295+0000 7fb0a5c00640 1 --2- 172.21.15.28:0/113545886 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56176096edd0 0x561760979710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:58.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.295+0000 7fb0a5c00640 1 --2- 172.21.15.28:0/113545886 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561760979dc0 0x561760976f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:58.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.296+0000 7fb09fe00640 1 -- 172.21.15.28:0/113545886 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561760979dc0 msgr2=0x561760976f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:12:58.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.296+0000 7fb09fe00640 1 --2- 172.21.15.28:0/113545886 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561760979dc0 0x561760976f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:58.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.296+0000 7fb09fe00640 1 -- 172.21.15.28:0/113545886 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56176096edd0 msgr2=0x561760979710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:12:58.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.296+0000 7fb09fe00640 1 --2- 172.21.15.28:0/113545886 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56176096edd0 0x561760979710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:58.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.296+0000 7fb09fe00640 1 -- 172.21.15.28:0/113545886 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5617605e73f0 con 0x56176096f6f0 2024-05-12T22:12:58.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.296+0000 7fb0a5c00640 1 --2- 172.21.15.28:0/113545886 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56176096edd0 0x561760979710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:12:58.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.296+0000 7fb09fe00640 1 --2- 172.21.15.28:0/113545886 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56176096f6f0 0x561760976a00 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fb08c0126d0 tx=0x7fb08c012b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edc99e33e8401d14 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:12:58.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.296+0000 7fb08aa00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-12T22:12:58.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.296+0000 7fb08aa00640 1 -- 172.21.15.28:0/113545886 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb08c00f920 con 0x56176096f6f0 2024-05-12T22:12:58.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.296+0000 7fb08aa00640 1 -- 172.21.15.28:0/113545886 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7fb08c00fac0 con 0x56176096f6f0 2024-05-12T22:12:58.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.296+0000 7fb0aaa22740 1 -- 172.21.15.28:0/113545886 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb0980043d0 con 0x56176096f6f0 2024-05-12T22:12:58.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.297+0000 7fb08aa00640 1 -- 172.21.15.28:0/113545886 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fb08c00fd00 con 0x56176096f6f0 2024-05-12T22:12:58.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.297+0000 7fb0aaa22740 10 client.16011 fetch_fsmap learned FSMap version 25 2024-05-12T22:12:58.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.297+0000 7fb0aaa22740 1 -- 172.21.15.28:0/113545886 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5617605a70f0 con 0x56176096f6f0 2024-05-12T22:12:58.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.297+0000 7fb08aa00640 1 -- 172.21.15.28:0/113545886 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb0980043d0 con 0x56176096f6f0 2024-05-12T22:12:58.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.297+0000 7fb08aa00640 1 -- 172.21.15.28:0/113545886 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7fb08c025570 con 0x56176096f6f0 2024-05-12T22:12:58.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.297+0000 7fb0aaa22740 10 client.16011 fetch_fsmap finished waiting for FSMap version 25 2024-05-12T22:12:58.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.297+0000 7fb0aaa22740 10 client.16011 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:12:58.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.297+0000 7fb0aaa22740 20 client.16011 populate_metadata read hostname 'smithi028' 2024-05-12T22:12:58.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.297+0000 7fb0aaa22740 1 --2- 172.21.15.28:0/113545886 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x561760989950 0x56176098bd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:58.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.297+0000 7fb0aaa22740 4 client.16011 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-12T22:12:58.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.297+0000 7fb0aaa22740 1 -- 172.21.15.28:0/113545886 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5617608e12c0 con 0x561760989950 2024-05-12T22:12:58.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.298+0000 7fb0a5c00640 1 --2- 172.21.15.28:0/113545886 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x561760989950 0x56176098bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:58.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.298+0000 7fb0a5c00640 1 --2- 172.21.15.28:0/113545886 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x561760989950 0x56176098bd60 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8d955c80db851d10 server_cookie=835b1ed519203ada in_seq=0 out_seq=0 2024-05-12T22:12:58.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.298+0000 7fb08aa00640 10 client.16011 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:12:58.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.299+0000 7fb08aa00640 1 -- 172.21.15.28:0/113545886 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5617608e12c0 con 0x561760989950 2024-05-12T22:12:58.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.299+0000 7fb08aa00640 10 client.16011 handle_command_reply: tid=0 2024-05-12T22:12:58.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.299+0000 7fb0aaa22740 1 client.16011 shutdown 2024-05-12T22:12:58.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.300+0000 7fb0aaa22740 1 -- 172.21.15.28:0/113545886 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56176096f6f0 msgr2=0x561760976a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:12:58.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.300+0000 7fb0aaa22740 1 --2- 172.21.15.28:0/113545886 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56176096f6f0 0x561760976a00 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fb08c0126d0 tx=0x7fb08c012b90 comp rx=0 tx=0).stop 2024-05-12T22:12:58.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.300+0000 7fb0aaa22740 1 -- 172.21.15.28:0/113545886 shutdown_connections 2024-05-12T22:12:58.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.300+0000 7fb0aaa22740 1 --2- 172.21.15.28:0/113545886 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x561760989950 0x56176098bd60 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:58.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.300+0000 7fb0aaa22740 1 --2- 172.21.15.28:0/113545886 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561760979dc0 0x561760976f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:58.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.300+0000 7fb0aaa22740 1 --2- 172.21.15.28:0/113545886 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56176096f6f0 0x561760976a00 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:58.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.300+0000 7fb0aaa22740 1 --2- 172.21.15.28:0/113545886 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56176096edd0 0x561760979710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:58.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.300+0000 7fb0aaa22740 1 -- 172.21.15.28:0/113545886 >> 172.21.15.28:0/113545886 conn(0x56176096b0f0 msgr2=0x561760970ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:12:58.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.300+0000 7fb08aa00640 0 client.16011 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:12:58.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.300+0000 7fb0aaa22740 1 -- 172.21.15.28:0/113545886 shutdown_connections 2024-05-12T22:12:58.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.300+0000 7fb0aaa22740 1 -- 172.21.15.28:0/113545886 wait complete. 2024-05-12T22:12:58.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.300+0000 7fb0aaa22740 20 client.16011 trim_cache size 0 max 16384 2024-05-12T22:12:58.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.319+0000 7fb0aaa22740 1 Processor -- start 2024-05-12T22:12:58.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.320+0000 7fb0aaa22740 1 -- start start 2024-05-12T22:12:58.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.320+0000 7fb0aaa22740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56176096edd0 0x561760969aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:58.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.320+0000 7fb0aaa22740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56176096f6f0 0x5617609858f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:58.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.320+0000 7fb0aaa22740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561760979dc0 0x561760991630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:58.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.320+0000 7fb0aaa22740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb0a01b6140 con 0x56176096f6f0 2024-05-12T22:12:58.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.320+0000 7fb0aaa22740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fb0a01b3a10 con 0x56176096edd0 2024-05-12T22:12:58.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.320+0000 7fb0aaa22740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb0a01b1860 con 0x561760979dc0 2024-05-12T22:12:58.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.320+0000 7fb0a5c00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56176096edd0 0x561760969aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:58.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.320+0000 7fb0a5c00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56176096edd0 0x561760969aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:46106/0 (socket says 172.21.15.28:46106) 2024-05-12T22:12:58.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.320+0000 7fb0a5c00640 1 -- 172.21.15.28:0/2092728436 learned_addr learned my addr 172.21.15.28:0/2092728436 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:12:58.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.320+0000 7fb0a5c00640 1 --2- 172.21.15.28:0/2092728436 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561760979dc0 0x561760991630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:58.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.320+0000 7fb09fe00640 1 --2- 172.21.15.28:0/2092728436 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56176096f6f0 0x5617609858f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:58.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.321+0000 7fb0a5c00640 1 -- 172.21.15.28:0/2092728436 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561760979dc0 msgr2=0x561760991630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:12:58.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.321+0000 7fb0a5c00640 1 --2- 172.21.15.28:0/2092728436 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561760979dc0 0x561760991630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:58.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.321+0000 7fb0a5c00640 1 -- 172.21.15.28:0/2092728436 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56176096f6f0 msgr2=0x5617609858f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:12:58.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.321+0000 7fb0a5c00640 1 --2- 172.21.15.28:0/2092728436 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56176096f6f0 0x5617609858f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:58.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.321+0000 7fb0a5c00640 1 -- 172.21.15.28:0/2092728436 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56176096e9a0 con 0x56176096edd0 2024-05-12T22:12:58.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.321+0000 7fb09fe00640 1 --2- 172.21.15.28:0/2092728436 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56176096f6f0 0x5617609858f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-12T22:12:58.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.321+0000 7fb0a5c00640 1 --2- 172.21.15.28:0/2092728436 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56176096edd0 0x561760969aa0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fb098003e80 tx=0x7fb098003eb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=af0662265c14d35a server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:12:58.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.321+0000 7fb095600640 1 -- 172.21.15.28:0/2092728436 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb098005240 con 0x56176096edd0 2024-05-12T22:12:58.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.321+0000 7fb095600640 1 -- 172.21.15.28:0/2092728436 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb098003ad0 con 0x56176096edd0 2024-05-12T22:12:58.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.321+0000 7fb095600640 1 -- 172.21.15.28:0/2092728436 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb09802f680 con 0x56176096edd0 2024-05-12T22:12:58.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.322+0000 7fb0aaa22740 1 -- 172.21.15.28:0/2092728436 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56176096edd0 msgr2=0x561760969aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:12:58.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.322+0000 7fb0aaa22740 1 --2- 172.21.15.28:0/2092728436 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56176096edd0 0x561760969aa0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fb098003e80 tx=0x7fb098003eb0 comp rx=0 tx=0).stop 2024-05-12T22:12:58.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.322+0000 7fb0aaa22740 1 -- 172.21.15.28:0/2092728436 shutdown_connections 2024-05-12T22:12:58.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.322+0000 7fb0aaa22740 1 --2- 172.21.15.28:0/2092728436 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561760979dc0 0x561760991630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:58.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.322+0000 7fb0aaa22740 1 --2- 172.21.15.28:0/2092728436 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56176096f6f0 0x5617609858f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:58.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.322+0000 7fb0aaa22740 1 --2- 172.21.15.28:0/2092728436 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56176096edd0 0x561760969aa0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:58.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.322+0000 7fb0aaa22740 1 -- 172.21.15.28:0/2092728436 >> 172.21.15.28:0/2092728436 conn(0x56176096b0f0 msgr2=0x561760976b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:12:58.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.322+0000 7fb0aaa22740 1 -- 172.21.15.28:0/2092728436 shutdown_connections 2024-05-12T22:12:58.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.322+0000 7fb0aaa22740 1 -- 172.21.15.28:0/2092728436 wait complete. 2024-05-12T22:12:58.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.322+0000 7fb0aaa22740 1 -- start start 2024-05-12T22:12:58.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.323+0000 7fb0aaa22740 1 Processor -- start 2024-05-12T22:12:58.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.323+0000 7fb0aaa22740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56176096edd0 0x56176097efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:58.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.323+0000 7fb0aaa22740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56176096f6f0 0x56176097f510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:58.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.323+0000 7fb0aaa22740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561760979dc0 0x5617609edea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:58.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.323+0000 7fb0aaa22740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb0a01b1ed0 con 0x561760979dc0 2024-05-12T22:12:58.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.323+0000 7fb0aaa22740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fb0a01b3340 con 0x56176096edd0 2024-05-12T22:12:58.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.323+0000 7fb0aaa22740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb0a01b25a0 con 0x56176096f6f0 2024-05-12T22:12:58.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.323+0000 7fb0aaa22740 10 client.0 _ll_register_callbacks cb 0x7fb08c009900 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:12:58.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.324+0000 7fb0a5c00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56176096edd0 0x56176097efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:58.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.324+0000 7fb09fe00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56176096f6f0 0x56176097f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:58.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.324+0000 7fb0a5c00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56176096edd0 0x56176097efd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:46120/0 (socket says 172.21.15.28:46120) 2024-05-12T22:12:58.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.324+0000 7fb09fe00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56176096f6f0 0x56176097f510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:35856/0 (socket says 172.21.15.28:35856) 2024-05-12T22:12:58.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.324+0000 7fb0a5c00640 1 -- 172.21.15.28:0/1801372487 learned_addr learned my addr 172.21.15.28:0/1801372487 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:12:58.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.324+0000 7fb0a5c00640 1 --2- 172.21.15.28:0/1801372487 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561760979dc0 0x5617609edea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:58.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.324+0000 7fb09fe00640 1 -- 172.21.15.28:0/1801372487 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56176096edd0 msgr2=0x56176097efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:12:58.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.324+0000 7fb09fe00640 1 --2- 172.21.15.28:0/1801372487 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56176096edd0 0x56176097efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:58.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.324+0000 7fb09fe00640 1 -- 172.21.15.28:0/1801372487 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561760979dc0 msgr2=0x5617609edea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:12:58.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.324+0000 7fb09fe00640 1 --2- 172.21.15.28:0/1801372487 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561760979dc0 0x5617609edea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:58.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.324+0000 7fb09fe00640 1 -- 172.21.15.28:0/1801372487 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5617609ee460 con 0x56176096f6f0 2024-05-12T22:12:58.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.324+0000 7fb09fe00640 1 --2- 172.21.15.28:0/1801372487 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56176096f6f0 0x56176097f510 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fb08c025990 tx=0x7fb08c0095b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e9aa6a643fb7aeb9 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:12:58.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.324+0000 7fb08aa00640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-12T22:12:58.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.325+0000 7fb08aa00640 1 -- 172.21.15.28:0/1801372487 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb08c025b80 con 0x56176096f6f0 2024-05-12T22:12:58.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.325+0000 7fb08aa00640 1 -- 172.21.15.28:0/1801372487 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7fb08c025d20 con 0x56176096f6f0 2024-05-12T22:12:58.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.325+0000 7fb0aaa22740 1 -- 172.21.15.28:0/1801372487 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb098003ad0 con 0x56176096f6f0 2024-05-12T22:12:58.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.325+0000 7fb08aa00640 1 -- 172.21.15.28:0/1801372487 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fb08c034050 con 0x56176096f6f0 2024-05-12T22:12:58.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.325+0000 7fb0aaa22740 10 client.26104 fetch_fsmap learned FSMap version 25 2024-05-12T22:12:58.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.325+0000 7fb0aaa22740 1 -- 172.21.15.28:0/1801372487 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5617609ef750 con 0x56176096f6f0 2024-05-12T22:12:58.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.326+0000 7fb08aa00640 1 -- 172.21.15.28:0/1801372487 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb098003ad0 con 0x56176096f6f0 2024-05-12T22:12:58.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.326+0000 7fb08aa00640 1 -- 172.21.15.28:0/1801372487 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7fb08c0254e0 con 0x56176096f6f0 2024-05-12T22:12:58.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.326+0000 7fb0aaa22740 10 client.26104 fetch_fsmap finished waiting for FSMap version 25 2024-05-12T22:12:58.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.326+0000 7fb0aaa22740 10 client.26104 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:12:58.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.326+0000 7fb0aaa22740 20 client.26104 populate_metadata read hostname 'smithi028' 2024-05-12T22:12:58.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.326+0000 7fb0aaa22740 1 --2- 172.21.15.28:0/1801372487 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5617609eff80 0x5617609f23a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:58.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.326+0000 7fb0aaa22740 4 client.26104 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-12T22:12:58.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.326+0000 7fb0aaa22740 1 -- 172.21.15.28:0/1801372487 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5617608db610 con 0x5617609eff80 2024-05-12T22:12:58.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.327+0000 7fb0a5c00640 1 --2- 172.21.15.28:0/1801372487 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5617609eff80 0x5617609f23a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:58.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.327+0000 7fb0a5c00640 1 --2- 172.21.15.28:0/1801372487 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5617609eff80 0x5617609f23a0 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e6edcf4bf1d58efb server_cookie=f0faf0479181d47d in_seq=0 out_seq=0 2024-05-12T22:12:58.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.327+0000 7fb08aa00640 10 client.26104 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:12:58.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.327+0000 7fb08aa00640 1 -- 172.21.15.28:0/1801372487 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5617608db610 con 0x5617609eff80 2024-05-12T22:12:58.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.327+0000 7fb08aa00640 10 client.26104 handle_command_reply: tid=0 2024-05-12T22:12:58.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.327+0000 7fb0aaa22740 1 client.26104 shutdown 2024-05-12T22:12:58.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.328+0000 7fb0aaa22740 1 -- 172.21.15.28:0/1801372487 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56176096f6f0 msgr2=0x56176097f510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:12:58.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.328+0000 7fb0aaa22740 1 --2- 172.21.15.28:0/1801372487 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56176096f6f0 0x56176097f510 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fb08c025990 tx=0x7fb08c0095b0 comp rx=0 tx=0).stop 2024-05-12T22:12:58.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.328+0000 7fb0aaa22740 1 -- 172.21.15.28:0/1801372487 shutdown_connections 2024-05-12T22:12:58.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.328+0000 7fb0aaa22740 1 --2- 172.21.15.28:0/1801372487 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5617609eff80 0x5617609f23a0 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:58.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.328+0000 7fb0aaa22740 1 --2- 172.21.15.28:0/1801372487 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561760979dc0 0x5617609edea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:58.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.328+0000 7fb0aaa22740 1 --2- 172.21.15.28:0/1801372487 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56176096f6f0 0x56176097f510 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:58.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.328+0000 7fb08aa00640 0 client.26104 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:12:58.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.328+0000 7fb0aaa22740 1 --2- 172.21.15.28:0/1801372487 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56176096edd0 0x56176097efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:58.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.328+0000 7fb0aaa22740 1 -- 172.21.15.28:0/1801372487 >> 172.21.15.28:0/1801372487 conn(0x56176096b0f0 msgr2=0x561760978230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:12:58.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.328+0000 7fb0aaa22740 1 -- 172.21.15.28:0/1801372487 shutdown_connections 2024-05-12T22:12:58.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.328+0000 7fb0aaa22740 1 -- 172.21.15.28:0/1801372487 wait complete. 2024-05-12T22:12:58.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.328+0000 7fb0aaa22740 20 client.26104 trim_cache size 0 max 16384 2024-05-12T22:12:58.336 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-05-12T22:12:58.336 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-05-12T22:12:58.336 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-05-12T22:12:58.337 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-05-12T22:12:58.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.329+0000 7fb0a6600640 1 -- 172.21.15.28:0/3634307427 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fb0840a2d00 msgr2=0x7fb0840a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:12:58.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.329+0000 7fb0a6600640 1 --2- 172.21.15.28:0/3634307427 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fb0840a2d00 0x7fb0840a51c0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x561760970390 tx=0x7fb0900040d0 comp rx=0 tx=0).stop 2024-05-12T22:12:58.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.329+0000 7fb0a6600640 1 -- 172.21.15.28:0/3634307427 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb0a0100680 msgr2=0x7fb0a019e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:12:58.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.329+0000 7fb0a6600640 1 --2- 172.21.15.28:0/3634307427 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb0a0100680 0x7fb0a019e5b0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fb090004af0 tx=0x7fb09002f6b0 comp rx=0 tx=0).stop 2024-05-12T22:12:58.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.329+0000 7fb0a6600640 1 -- 172.21.15.28:0/3634307427 shutdown_connections 2024-05-12T22:12:58.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.329+0000 7fb0a6600640 1 --2- 172.21.15.28:0/3634307427 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fb0840a2d00 0x7fb0840a51c0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:58.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.329+0000 7fb0a6600640 1 --2- 172.21.15.28:0/3634307427 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0a0105a80 0x7fb0a01a2e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:58.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.329+0000 7fb0a6600640 1 --2- 172.21.15.28:0/3634307427 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb0a0100fa0 0x7fb0a019eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:58.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.329+0000 7fb0a6600640 1 --2- 172.21.15.28:0/3634307427 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb0a0100680 0x7fb0a019e5b0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:58.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.329+0000 7fb0a6600640 1 -- 172.21.15.28:0/3634307427 >> 172.21.15.28:0/3634307427 conn(0x7fb0a00fc0b0 msgr2=0x7fb0a00fd060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:12:58.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.329+0000 7fb0a6600640 1 -- 172.21.15.28:0/3634307427 shutdown_connections 2024-05-12T22:12:58.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.329+0000 7fb0a6600640 1 -- 172.21.15.28:0/3634307427 wait complete. 2024-05-12T22:12:58.350 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-12T22:12:58.350 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-12T22:12:58.423 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:12:58 smithi028 ceph-mon[29847]: pgmap v122: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 3.6 KiB/s wr, 5 op/s 2024-05-12T22:12:58.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:12:58 smithi086 ceph-mon[34297]: pgmap v122: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 3.6 KiB/s wr, 5 op/s 2024-05-12T22:12:58.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.564+0000 7fc993200640 1 -- 172.21.15.28:0/2656452198 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc9941041f0 msgr2=0x7fc994110700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:12:58.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.564+0000 7fc993200640 1 --2- 172.21.15.28:0/2656452198 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc9941041f0 0x7fc994110700 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fc98c009c10 tx=0x7fc98c02a6e0 comp rx=0 tx=0).stop 2024-05-12T22:12:58.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.564+0000 7fc993200640 1 -- 172.21.15.28:0/2656452198 shutdown_connections 2024-05-12T22:12:58.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.564+0000 7fc993200640 1 --2- 172.21.15.28:0/2656452198 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc9941041f0 0x7fc994110700 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:58.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.564+0000 7fc993200640 1 --2- 172.21.15.28:0/2656452198 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc994103830 0x7fc994103cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:58.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.564+0000 7fc993200640 1 --2- 172.21.15.28:0/2656452198 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc994109840 0x7fc994109c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:58.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.564+0000 7fc993200640 1 -- 172.21.15.28:0/2656452198 >> 172.21.15.28:0/2656452198 conn(0x7fc9940ff300 msgr2=0x7fc994101720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:12:58.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.565+0000 7fc993200640 1 -- 172.21.15.28:0/2656452198 shutdown_connections 2024-05-12T22:12:58.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.565+0000 7fc993200640 1 -- 172.21.15.28:0/2656452198 wait complete. 2024-05-12T22:12:58.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.566+0000 7fc993200640 1 Processor -- start 2024-05-12T22:12:58.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.566+0000 7fc993200640 1 -- start start 2024-05-12T22:12:58.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.567+0000 7fc993200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc994103830 0x7fc99419a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:58.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.567+0000 7fc993200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc9941041f0 0x7fc99419a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:58.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.567+0000 7fc993200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc994109840 0x7fc99419e990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:58.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.567+0000 7fc993200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc994113c10 con 0x7fc994109840 2024-05-12T22:12:58.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.567+0000 7fc993200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fc994113a90 con 0x7fc9941041f0 2024-05-12T22:12:58.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.567+0000 7fc993200640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc994113d90 con 0x7fc994103830 2024-05-12T22:12:58.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.567+0000 7fc991e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc994103830 0x7fc99419a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:58.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.567+0000 7fc991e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc994103830 0x7fc99419a0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:35868/0 (socket says 172.21.15.28:35868) 2024-05-12T22:12:58.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.567+0000 7fc991e00640 1 -- 172.21.15.28:0/3260465127 learned_addr learned my addr 172.21.15.28:0/3260465127 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:12:58.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.567+0000 7fc992800640 1 --2- 172.21.15.28:0/3260465127 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc994109840 0x7fc99419e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:58.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.567+0000 7fc991400640 1 --2- 172.21.15.28:0/3260465127 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc9941041f0 0x7fc99419a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:58.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.568+0000 7fc992800640 1 -- 172.21.15.28:0/3260465127 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc994103830 msgr2=0x7fc99419a0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:12:58.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.568+0000 7fc992800640 1 --2- 172.21.15.28:0/3260465127 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc994103830 0x7fc99419a0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:58.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.568+0000 7fc992800640 1 -- 172.21.15.28:0/3260465127 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc9941041f0 msgr2=0x7fc99419a610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:12:58.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.568+0000 7fc992800640 1 --2- 172.21.15.28:0/3260465127 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc9941041f0 0x7fc99419a610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:58.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.568+0000 7fc992800640 1 -- 172.21.15.28:0/3260465127 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc99419f0c0 con 0x7fc994109840 2024-05-12T22:12:58.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.568+0000 7fc991e00640 1 --2- 172.21.15.28:0/3260465127 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc994103830 0x7fc99419a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:12:58.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.568+0000 7fc991400640 1 --2- 172.21.15.28:0/3260465127 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc9941041f0 0x7fc99419a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:12:58.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.568+0000 7fc992800640 1 --2- 172.21.15.28:0/3260465127 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc994109840 0x7fc99419e990 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fc98c034040 tx=0x7fc98c0305d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:12:58.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.568+0000 7fc98aa00640 1 -- 172.21.15.28:0/3260465127 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc98c041020 con 0x7fc994109840 2024-05-12T22:12:58.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.568+0000 7fc98aa00640 1 -- 172.21.15.28:0/3260465127 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc98c003e50 con 0x7fc994109840 2024-05-12T22:12:58.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.568+0000 7fc993200640 1 -- 172.21.15.28:0/3260465127 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc99419f350 con 0x7fc994109840 2024-05-12T22:12:58.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.568+0000 7fc98aa00640 1 -- 172.21.15.28:0/3260465127 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc98c036040 con 0x7fc994109840 2024-05-12T22:12:58.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.569+0000 7fc993200640 1 -- 172.21.15.28:0/3260465127 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9941a6c40 con 0x7fc994109840 2024-05-12T22:12:58.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.570+0000 7fc993200640 1 -- 172.21.15.28:0/3260465127 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc994104f70 con 0x7fc994109840 2024-05-12T22:12:58.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.572+0000 7fc98aa00640 1 -- 172.21.15.28:0/3260465127 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7fc98c044020 con 0x7fc994109840 2024-05-12T22:12:58.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.574+0000 7fc98aa00640 1 --2- 172.21.15.28:0/3260465127 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fc9780a2d00 0x7fc9780a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:58.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.574+0000 7fc98aa00640 1 -- 172.21.15.28:0/3260465127 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7fc98c0f0050 con 0x7fc994109840 2024-05-12T22:12:58.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.574+0000 7fc991e00640 1 --2- 172.21.15.28:0/3260465127 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fc9780a2d00 0x7fc9780a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:58.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.575+0000 7fc991e00640 1 --2- 172.21.15.28:0/3260465127 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fc9780a2d00 0x7fc9780a51c0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fc9840096c0 tx=0x7fc984009260 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:12:58.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.579+0000 7fc98aa00640 1 -- 172.21.15.28:0/3260465127 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fc98c04b020 con 0x7fc994109840 2024-05-12T22:12:58.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:12:58 smithi186 ceph-mon[34785]: pgmap v122: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 3.6 KiB/s wr, 5 op/s 2024-05-12T22:12:58.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.825+0000 7fc993200640 1 -- 172.21.15.28:0/3260465127 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc99410bdc0 con 0x7fc994109840 2024-05-12T22:12:58.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.827+0000 7fc98aa00640 1 -- 172.21.15.28:0/3260465127 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11865 (secure 0 0 0) 0x7fc98c0bcca0 con 0x7fc994109840 2024-05-12T22:12:58.828 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-12T22:12:58.828 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1844141745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1844141745},{"type":"v1","addr":"172.21.15.186:6833","nonce":1844141745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6833/2960763758","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":2960763758},{"type":"v1","addr":"172.21.15.86:6833","nonce":2960763758}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24470,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/4131134219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":4131134219},{"type":"v1","addr":"172.21.15.28:6835","nonce":4131134219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24481,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/1727869064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1727869064},{"type":"v1","addr":"172.21.15.186:6835","nonce":1727869064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24482,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2938260704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2938260704},{"type":"v1","addr":"172.21.15.28:6837","nonce":2938260704}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24494,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3473700910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3473700910},{"type":"v1","addr":"172.21.15.86:6835","nonce":3473700910}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-12T22:06:33.426986+0000","modified":"2024-05-12T22:10:42.498031+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24581,"mds_1":24550,"mds_2":24532},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24499":{"gid":24499,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6837/3600809235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":3600809235},{"type":"v1","addr":"172.21.15.186:6837","nonce":3600809235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"i","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6839/2399610807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":2399610807},{"type":"v1","addr":"172.21.15.186:6839","nonce":2399610807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24530":{"gid":24530,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.86:6837/3782448884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":3782448884},{"type":"v1","addr":"172.21.15.86:6837","nonce":3782448884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24532":{"gid":24532,"name":"g","rank":2,"incarnation":21,"state":"up:active","state_seq":6,"addr":"172.21.15.28:6839/123892158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":123892158},{"type":"v1","addr":"172.21.15.28:6839","nonce":123892158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24550":{"gid":24550,"name":"a","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.28:6841/761183870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":761183870},{"type":"v1","addr":"172.21.15.28:6841","nonce":761183870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24581":{"gid":24581,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6839/1251999130","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6838","nonce":1251999130},{"type":"v1","addr":"172.21.15.86:6839","nonce":1251999130}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24581,"qdb_cluster":[24550,24532,24581]},"id":1}]} 2024-05-12T22:12:58.829 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 25 2024-05-12T22:12:58.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.830+0000 7fc993200640 1 -- 172.21.15.28:0/3260465127 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fc9780a2d00 msgr2=0x7fc9780a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:12:58.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.830+0000 7fc993200640 1 --2- 172.21.15.28:0/3260465127 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fc9780a2d00 0x7fc9780a51c0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fc9840096c0 tx=0x7fc984009260 comp rx=0 tx=0).stop 2024-05-12T22:12:58.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.830+0000 7fc993200640 1 -- 172.21.15.28:0/3260465127 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc994109840 msgr2=0x7fc99419e990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:12:58.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.830+0000 7fc993200640 1 --2- 172.21.15.28:0/3260465127 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc994109840 0x7fc99419e990 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fc98c034040 tx=0x7fc98c0305d0 comp rx=0 tx=0).stop 2024-05-12T22:12:58.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.830+0000 7fc993200640 1 -- 172.21.15.28:0/3260465127 shutdown_connections 2024-05-12T22:12:58.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.830+0000 7fc993200640 1 --2- 172.21.15.28:0/3260465127 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fc9780a2d00 0x7fc9780a51c0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:58.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.830+0000 7fc993200640 1 --2- 172.21.15.28:0/3260465127 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc994109840 0x7fc99419e990 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:58.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.830+0000 7fc993200640 1 --2- 172.21.15.28:0/3260465127 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc9941041f0 0x7fc99419a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:58.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.830+0000 7fc993200640 1 --2- 172.21.15.28:0/3260465127 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc994103830 0x7fc99419a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:58.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.830+0000 7fc993200640 1 -- 172.21.15.28:0/3260465127 >> 172.21.15.28:0/3260465127 conn(0x7fc9940ff300 msgr2=0x7fc9940ffa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:12:58.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.831+0000 7fc993200640 1 -- 172.21.15.28:0/3260465127 shutdown_connections 2024-05-12T22:12:58.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:58.831+0000 7fc993200640 1 -- 172.21.15.28:0/3260465127 wait complete. 2024-05-12T22:12:58.852 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-05-12T22:12:59.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.071+0000 7f5de7000640 1 -- 172.21.15.28:0/2711734316 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5de0074580 msgr2=0x7f5de010dae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:12:59.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.071+0000 7f5de7000640 1 --2- 172.21.15.28:0/2711734316 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5de0074580 0x7f5de010dae0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f5dc8009900 tx=0x7f5dc802a480 comp rx=0 tx=0).stop 2024-05-12T22:12:59.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.071+0000 7f5de7000640 1 -- 172.21.15.28:0/2711734316 shutdown_connections 2024-05-12T22:12:59.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.071+0000 7f5de7000640 1 --2- 172.21.15.28:0/2711734316 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5de010e020 0x7f5de0110470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:59.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.071+0000 7f5de7000640 1 --2- 172.21.15.28:0/2711734316 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5de0074580 0x7f5de010dae0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:59.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.071+0000 7f5de7000640 1 --2- 172.21.15.28:0/2711734316 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5de0073c60 0x7f5de0074040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:59.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.071+0000 7f5de7000640 1 -- 172.21.15.28:0/2711734316 >> 172.21.15.28:0/2711734316 conn(0x7f5de00fd2b0 msgr2=0x7f5de00ff6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:12:59.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.071+0000 7f5de7000640 1 -- 172.21.15.28:0/2711734316 shutdown_connections 2024-05-12T22:12:59.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.071+0000 7f5de7000640 1 -- 172.21.15.28:0/2711734316 wait complete. 2024-05-12T22:12:59.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.072+0000 7f5de7000640 1 Processor -- start 2024-05-12T22:12:59.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.072+0000 7f5de7000640 1 -- start start 2024-05-12T22:12:59.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.073+0000 7f5de7000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5de0073c60 0x7f5de019a330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:59.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.073+0000 7f5de7000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5de0074580 0x7f5de019a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:59.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.073+0000 7f5de7000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5de010e020 0x7f5de019ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:59.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.073+0000 7f5de7000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5de0113dd0 con 0x7f5de0073c60 2024-05-12T22:12:59.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.073+0000 7f5de7000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f5de0113c50 con 0x7f5de0074580 2024-05-12T22:12:59.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.073+0000 7f5de7000640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f5de0113f50 con 0x7f5de010e020 2024-05-12T22:12:59.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.073+0000 7f5ddfe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5de0073c60 0x7f5de019a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:59.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.073+0000 7f5de5000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5de010e020 0x7f5de019ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:59.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.073+0000 7f5ddfe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5de0073c60 0x7f5de019a330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36776/0 (socket says 172.21.15.28:36776) 2024-05-12T22:12:59.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.073+0000 7f5ddfe00640 1 -- 172.21.15.28:0/2179429868 learned_addr learned my addr 172.21.15.28:0/2179429868 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:12:59.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.073+0000 7f5ddf400640 1 --2- 172.21.15.28:0/2179429868 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5de0074580 0x7f5de019a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:59.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.074+0000 7f5de5000640 1 -- 172.21.15.28:0/2179429868 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5de0074580 msgr2=0x7f5de019a870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:12:59.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.074+0000 7f5de5000640 1 --2- 172.21.15.28:0/2179429868 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5de0074580 0x7f5de019a870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:59.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.074+0000 7f5de5000640 1 -- 172.21.15.28:0/2179429868 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5de0073c60 msgr2=0x7f5de019a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:12:59.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.074+0000 7f5de5000640 1 --2- 172.21.15.28:0/2179429868 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5de0073c60 0x7f5de019a330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:59.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.074+0000 7f5de5000640 1 -- 172.21.15.28:0/2179429868 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5de019f320 con 0x7f5de010e020 2024-05-12T22:12:59.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.074+0000 7f5ddfe00640 1 --2- 172.21.15.28:0/2179429868 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5de0073c60 0x7f5de019a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:12:59.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.074+0000 7f5de5000640 1 --2- 172.21.15.28:0/2179429868 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5de010e020 0x7f5de019ebf0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f5dd800d6b0 tx=0x7f5dd800db70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:12:59.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.074+0000 7f5ddcc00640 1 -- 172.21.15.28:0/2179429868 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dd8017070 con 0x7f5de010e020 2024-05-12T22:12:59.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.074+0000 7f5de7000640 1 -- 172.21.15.28:0/2179429868 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5de019f610 con 0x7f5de010e020 2024-05-12T22:12:59.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.074+0000 7f5ddcc00640 1 -- 172.21.15.28:0/2179429868 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5dd8009d00 con 0x7f5de010e020 2024-05-12T22:12:59.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.074+0000 7f5ddcc00640 1 -- 172.21.15.28:0/2179429868 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dd8003d60 con 0x7f5de010e020 2024-05-12T22:12:59.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.075+0000 7f5de7000640 1 -- 172.21.15.28:0/2179429868 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5de01a6ea0 con 0x7f5de010e020 2024-05-12T22:12:59.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.078+0000 7f5de9da8740 1 Processor -- start 2024-05-12T22:12:59.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.078+0000 7f5ddcc00640 1 -- 172.21.15.28:0/2179429868 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7f5dd8009820 con 0x7f5de010e020 2024-05-12T22:12:59.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.079+0000 7f5de9da8740 1 -- start start 2024-05-12T22:12:59.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.079+0000 7f5de9da8740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556579bb4980 0x556579bb4d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:59.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.079+0000 7f5de9da8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556579bb52a0 0x556579bbf430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:59.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.079+0000 7f5de9da8740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556579bbf970 0x556579bc1d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:59.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.079+0000 7f5de9da8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x556579b19a30 con 0x556579bb52a0 2024-05-12T22:12:59.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.079+0000 7f5de9da8740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x556579ac1700 con 0x556579bbf970 2024-05-12T22:12:59.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.079+0000 7f5de9da8740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x556579b159f0 con 0x556579bb4980 2024-05-12T22:12:59.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.080+0000 7f5ddfe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556579bb52a0 0x556579bbf430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:59.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.080+0000 7f5ddf400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556579bbf970 0x556579bc1d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:59.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.080+0000 7f5ddfe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556579bb52a0 0x556579bbf430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36792/0 (socket says 172.21.15.28:36792) 2024-05-12T22:12:59.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.080+0000 7f5ddf400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556579bbf970 0x556579bc1d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:46184/0 (socket says 172.21.15.28:46184) 2024-05-12T22:12:59.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.080+0000 7f5ddfe00640 1 -- 172.21.15.28:0/3301358460 learned_addr learned my addr 172.21.15.28:0/3301358460 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:12:59.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.080+0000 7f5ddf400640 1 --2- 172.21.15.28:0/3301358460 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556579bb4980 0x556579bb4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:59.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.080+0000 7f5ddfe00640 1 -- 172.21.15.28:0/3301358460 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556579bb4980 msgr2=0x556579bb4d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:12:59.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.080+0000 7f5ddfe00640 1 --2- 172.21.15.28:0/3301358460 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556579bb4980 0x556579bb4d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:59.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.080+0000 7f5ddfe00640 1 -- 172.21.15.28:0/3301358460 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556579bbf970 msgr2=0x556579bc1d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:12:59.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.080+0000 7f5ddfe00640 1 --2- 172.21.15.28:0/3301358460 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556579bbf970 0x556579bc1d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:59.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.080+0000 7f5ddfe00640 1 -- 172.21.15.28:0/3301358460 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55657982d3f0 con 0x556579bb52a0 2024-05-12T22:12:59.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.080+0000 7f5ddcc00640 1 --2- 172.21.15.28:0/2179429868 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f5dc40a2d00 0x7f5dc40a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:59.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.080+0000 7f5ddfe00640 1 --2- 172.21.15.28:0/3301358460 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556579bb52a0 0x556579bbf430 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f5dd000f8d0 tx=0x7f5dd000fd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=310d120a5e034c26 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:12:59.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.081+0000 7f5ddcc00640 1 -- 172.21.15.28:0/2179429868 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f5dd80cec80 con 0x7f5de010e020 2024-05-12T22:12:59.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.081+0000 7f5ddf400640 1 --2- 172.21.15.28:0/2179429868 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f5dc40a2d00 0x7f5dc40a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:59.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.081+0000 7f5dd5600640 1 -- 172.21.15.28:0/3301358460 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dd0011ba0 con 0x556579bb52a0 2024-05-12T22:12:59.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.081+0000 7f5dd5600640 1 -- 172.21.15.28:0/3301358460 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5dd0011d40 con 0x556579bb52a0 2024-05-12T22:12:59.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.081+0000 7f5dd5600640 1 -- 172.21.15.28:0/3301358460 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dd00187b0 con 0x556579bb52a0 2024-05-12T22:12:59.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.082+0000 7f5de9da8740 1 -- 172.21.15.28:0/3301358460 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556579bb52a0 msgr2=0x556579bbf430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:12:59.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.082+0000 7f5ddf400640 1 --2- 172.21.15.28:0/2179429868 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f5dc40a2d00 0x7f5dc40a51c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f5dc8000c00 tx=0x7f5dc802a480 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:12:59.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.082+0000 7f5de9da8740 1 --2- 172.21.15.28:0/3301358460 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556579bb52a0 0x556579bbf430 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f5dd000f8d0 tx=0x7f5dd000fd90 comp rx=0 tx=0).stop 2024-05-12T22:12:59.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.082+0000 7f5de9da8740 1 -- 172.21.15.28:0/3301358460 shutdown_connections 2024-05-12T22:12:59.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.082+0000 7f5de9da8740 1 --2- 172.21.15.28:0/3301358460 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556579bbf970 0x556579bc1d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:59.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.082+0000 7f5de9da8740 1 --2- 172.21.15.28:0/3301358460 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556579bb52a0 0x556579bbf430 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:59.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.082+0000 7f5de9da8740 1 --2- 172.21.15.28:0/3301358460 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556579bb4980 0x556579bb4d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:59.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.082+0000 7f5de9da8740 1 -- 172.21.15.28:0/3301358460 >> 172.21.15.28:0/3301358460 conn(0x556579bb0ca0 msgr2=0x556579bb1080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:12:59.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.082+0000 7f5de9da8740 1 -- 172.21.15.28:0/3301358460 shutdown_connections 2024-05-12T22:12:59.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.082+0000 7f5de9da8740 1 -- 172.21.15.28:0/3301358460 wait complete. 2024-05-12T22:12:59.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.082+0000 7f5de9da8740 1 -- start start 2024-05-12T22:12:59.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.083+0000 7f5de9da8740 1 Processor -- start 2024-05-12T22:12:59.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.083+0000 7f5de9da8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556579bb4980 0x556579bbc890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:59.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.083+0000 7f5de9da8740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556579bb52a0 0x556579bbcdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:59.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.083+0000 7f5de9da8740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556579bbf970 0x556579bbd310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:59.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.083+0000 7f5de9da8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x556579bc2760 con 0x556579bb4980 2024-05-12T22:12:59.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.083+0000 7f5de9da8740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x556579ad4a70 con 0x556579bbf970 2024-05-12T22:12:59.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.083+0000 7f5de9da8740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x556579bc28e0 con 0x556579bb52a0 2024-05-12T22:12:59.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.083+0000 7f5de9da8740 10 client.0 _ll_register_callbacks cb 0x556579ab7ba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:12:59.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.083+0000 7f5ddfe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556579bb4980 0x556579bbc890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:59.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.084+0000 7f5ddfe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556579bb4980 0x556579bbc890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36808/0 (socket says 172.21.15.28:36808) 2024-05-12T22:12:59.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.084+0000 7f5ddf400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556579bb52a0 0x556579bbcdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:59.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.084+0000 7f5ddfe00640 1 -- 172.21.15.28:0/3875684305 learned_addr learned my addr 172.21.15.28:0/3875684305 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:12:59.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.084+0000 7f5de5000640 1 --2- 172.21.15.28:0/3875684305 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556579bbf970 0x556579bbd310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:59.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.084+0000 7f5de5000640 1 -- 172.21.15.28:0/3875684305 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556579bb52a0 msgr2=0x556579bbcdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:12:59.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.084+0000 7f5de5000640 1 --2- 172.21.15.28:0/3875684305 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556579bb52a0 0x556579bbcdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:59.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.084+0000 7f5de5000640 1 -- 172.21.15.28:0/3875684305 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556579bb4980 msgr2=0x556579bbc890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:12:59.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.084+0000 7f5de5000640 1 --2- 172.21.15.28:0/3875684305 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556579bb4980 0x556579bbc890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:59.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.084+0000 7f5de5000640 1 -- 172.21.15.28:0/3875684305 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5565798f77d0 con 0x556579bbf970 2024-05-12T22:12:59.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.084+0000 7f5ddfe00640 1 --2- 172.21.15.28:0/3875684305 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556579bb4980 0x556579bbc890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:12:59.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.084+0000 7f5de5000640 1 --2- 172.21.15.28:0/3875684305 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556579bbf970 0x556579bbd310 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f5dd80cc3a0 tx=0x7f5dd809b110 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c79d6ab6411f57da server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:12:59.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.084+0000 7f5dce000640 10 client.0 ms_handle_connect on v2:172.21.15.86:3300/0 2024-05-12T22:12:59.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.084+0000 7f5dce000640 1 -- 172.21.15.28:0/3875684305 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dd8020040 con 0x556579bbf970 2024-05-12T22:12:59.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.085+0000 7f5dce000640 1 -- 172.21.15.28:0/3875684305 <== mon.2 v2:172.21.15.86:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f5dd80bb070 con 0x556579bbf970 2024-05-12T22:12:59.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.085+0000 7f5de9da8740 1 -- 172.21.15.28:0/3875684305 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5dd0011d40 con 0x556579bbf970 2024-05-12T22:12:59.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.085+0000 7f5dce000640 1 -- 172.21.15.28:0/3875684305 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f5dd80bb270 con 0x556579bbf970 2024-05-12T22:12:59.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.085+0000 7f5de9da8740 10 client.26168 fetch_fsmap learned FSMap version 25 2024-05-12T22:12:59.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.085+0000 7f5de9da8740 1 -- 172.21.15.28:0/3875684305 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x556579bce9a0 con 0x556579bbf970 2024-05-12T22:12:59.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.085+0000 7f5dce000640 1 -- 172.21.15.28:0/3875684305 <== mon.2 v2:172.21.15.86:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5dd0011d40 con 0x556579bbf970 2024-05-12T22:12:59.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.085+0000 7f5dce000640 1 -- 172.21.15.28:0/3875684305 <== mon.2 v2:172.21.15.86:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7f5dd809a670 con 0x556579bbf970 2024-05-12T22:12:59.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.085+0000 7f5de9da8740 10 client.26168 fetch_fsmap finished waiting for FSMap version 25 2024-05-12T22:12:59.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.085+0000 7f5de9da8740 10 client.26168 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.g 2024-05-12T22:12:59.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.085+0000 7f5de9da8740 20 client.26168 populate_metadata read hostname 'smithi028' 2024-05-12T22:12:59.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.085+0000 7f5de9da8740 1 --2- 172.21.15.28:0/3875684305 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x556579bcfe50 0x556579bd22c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:59.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.085+0000 7f5de9da8740 4 client.26168 mds_command: new command op to 24532 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-12T22:12:59.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.085+0000 7f5de9da8740 1 -- 172.21.15.28:0/3875684305 --> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556579b27160 con 0x556579bcfe50 2024-05-12T22:12:59.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.086+0000 7f5ddfe00640 1 --2- 172.21.15.28:0/3875684305 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x556579bcfe50 0x556579bd22c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:59.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.086+0000 7f5ddfe00640 1 --2- 172.21.15.28:0/3875684305 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x556579bcfe50 0x556579bd22c0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=3305c3f2b2f2c22 server_cookie=9536fb5044813b5f in_seq=0 out_seq=0 2024-05-12T22:12:59.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.086+0000 7f5dce000640 10 client.26168 ms_handle_connect on v2:172.21.15.28:6838/123892158 2024-05-12T22:12:59.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.087+0000 7f5dce000640 1 -- 172.21.15.28:0/3875684305 <== mds.2 v2:172.21.15.28:6838/123892158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556579b27160 con 0x556579bcfe50 2024-05-12T22:12:59.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.087+0000 7f5dce000640 10 client.26168 handle_command_reply: tid=0 2024-05-12T22:12:59.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.087+0000 7f5de9da8740 1 client.26168 shutdown 2024-05-12T22:12:59.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.087+0000 7f5de9da8740 1 -- 172.21.15.28:0/3875684305 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556579bbf970 msgr2=0x556579bbd310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:12:59.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.087+0000 7f5de9da8740 1 --2- 172.21.15.28:0/3875684305 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556579bbf970 0x556579bbd310 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f5dd80cc3a0 tx=0x7f5dd809b110 comp rx=0 tx=0).stop 2024-05-12T22:12:59.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.087+0000 7f5de9da8740 1 -- 172.21.15.28:0/3875684305 shutdown_connections 2024-05-12T22:12:59.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.087+0000 7f5de9da8740 1 --2- 172.21.15.28:0/3875684305 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x556579bcfe50 0x556579bd22c0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:59.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.087+0000 7f5de9da8740 1 --2- 172.21.15.28:0/3875684305 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556579bbf970 0x556579bbd310 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:59.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.087+0000 7f5de9da8740 1 --2- 172.21.15.28:0/3875684305 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556579bb52a0 0x556579bbcdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:59.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.087+0000 7f5de9da8740 1 --2- 172.21.15.28:0/3875684305 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556579bb4980 0x556579bbc890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:59.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.087+0000 7f5de9da8740 1 -- 172.21.15.28:0/3875684305 >> 172.21.15.28:0/3875684305 conn(0x556579bb0ca0 msgr2=0x556579bb6aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:12:59.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.087+0000 7f5dce000640 0 client.26168 ms_handle_reset on v2:172.21.15.28:6838/123892158 2024-05-12T22:12:59.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.088+0000 7f5de9da8740 1 -- 172.21.15.28:0/3875684305 shutdown_connections 2024-05-12T22:12:59.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.088+0000 7f5de9da8740 1 -- 172.21.15.28:0/3875684305 wait complete. 2024-05-12T22:12:59.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.088+0000 7f5de9da8740 20 client.26168 trim_cache size 0 max 16384 2024-05-12T22:12:59.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.103+0000 7f5de9da8740 1 Processor -- start 2024-05-12T22:12:59.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.103+0000 7f5de9da8740 1 -- start start 2024-05-12T22:12:59.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.103+0000 7f5de9da8740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556579bb4980 0x556579bcb620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:59.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.103+0000 7f5de9da8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556579bb52a0 0x556579bb71d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:59.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.103+0000 7f5de9da8740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556579bbf970 0x556579bd7b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:59.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.103+0000 7f5de9da8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5de01b2000 con 0x556579bb52a0 2024-05-12T22:12:59.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.103+0000 7f5de9da8740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f5de01af8d0 con 0x556579bb4980 2024-05-12T22:12:59.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.103+0000 7f5de9da8740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f5de01ad720 con 0x556579bbf970 2024-05-12T22:12:59.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.103+0000 7f5ddf400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556579bb52a0 0x556579bb71d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:59.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.103+0000 7f5ddfe00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556579bb4980 0x556579bcb620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:59.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.103+0000 7f5ddfe00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556579bb4980 0x556579bcb620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:46204/0 (socket says 172.21.15.28:46204) 2024-05-12T22:12:59.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.103+0000 7f5ddf400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556579bb52a0 0x556579bb71d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36810/0 (socket says 172.21.15.28:36810) 2024-05-12T22:12:59.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.103+0000 7f5ddfe00640 1 -- 172.21.15.28:0/917585226 learned_addr learned my addr 172.21.15.28:0/917585226 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:12:59.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.104+0000 7f5de5000640 1 --2- 172.21.15.28:0/917585226 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556579bbf970 0x556579bd7b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:59.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.104+0000 7f5ddfe00640 1 -- 172.21.15.28:0/917585226 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556579bbf970 msgr2=0x556579bd7b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:12:59.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.104+0000 7f5ddfe00640 1 --2- 172.21.15.28:0/917585226 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556579bbf970 0x556579bd7b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:59.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.104+0000 7f5ddfe00640 1 -- 172.21.15.28:0/917585226 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556579bb52a0 msgr2=0x556579bb71d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:12:59.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.104+0000 7f5ddfe00640 1 --2- 172.21.15.28:0/917585226 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556579bb52a0 0x556579bb71d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:59.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.104+0000 7f5ddfe00640 1 -- 172.21.15.28:0/917585226 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556579bbade0 con 0x556579bb4980 2024-05-12T22:12:59.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.104+0000 7f5de5000640 1 --2- 172.21.15.28:0/917585226 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556579bbf970 0x556579bd7b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:12:59.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.104+0000 7f5ddfe00640 1 --2- 172.21.15.28:0/917585226 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556579bb4980 0x556579bcb620 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f5dd0011810 tx=0x7f5dd0019f40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7931753cd74bdf31 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:12:59.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.104+0000 7f5dd4c00640 1 -- 172.21.15.28:0/917585226 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dd002bb50 con 0x556579bb4980 2024-05-12T22:12:59.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.104+0000 7f5dd4c00640 1 -- 172.21.15.28:0/917585226 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5dd002bcf0 con 0x556579bb4980 2024-05-12T22:12:59.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.104+0000 7f5dd4c00640 1 -- 172.21.15.28:0/917585226 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dd0005340 con 0x556579bb4980 2024-05-12T22:12:59.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.105+0000 7f5de9da8740 1 -- 172.21.15.28:0/917585226 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556579bb4980 msgr2=0x556579bcb620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:12:59.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.105+0000 7f5de9da8740 1 --2- 172.21.15.28:0/917585226 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556579bb4980 0x556579bcb620 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f5dd0011810 tx=0x7f5dd0019f40 comp rx=0 tx=0).stop 2024-05-12T22:12:59.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.105+0000 7f5de9da8740 1 -- 172.21.15.28:0/917585226 shutdown_connections 2024-05-12T22:12:59.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.105+0000 7f5de9da8740 1 --2- 172.21.15.28:0/917585226 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556579bbf970 0x556579bd7b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:59.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.105+0000 7f5de9da8740 1 --2- 172.21.15.28:0/917585226 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556579bb52a0 0x556579bb71d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:59.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.105+0000 7f5de9da8740 1 --2- 172.21.15.28:0/917585226 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556579bb4980 0x556579bcb620 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:59.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.105+0000 7f5de9da8740 1 -- 172.21.15.28:0/917585226 >> 172.21.15.28:0/917585226 conn(0x556579bb0ca0 msgr2=0x556579bb7bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:12:59.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.105+0000 7f5de9da8740 1 -- 172.21.15.28:0/917585226 shutdown_connections 2024-05-12T22:12:59.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.105+0000 7f5de9da8740 1 -- 172.21.15.28:0/917585226 wait complete. 2024-05-12T22:12:59.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.106+0000 7f5de9da8740 1 -- start start 2024-05-12T22:12:59.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.106+0000 7f5de9da8740 1 Processor -- start 2024-05-12T22:12:59.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.106+0000 7f5de9da8740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556579bb4980 0x556579bc7b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:59.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.106+0000 7f5de9da8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556579bb52a0 0x556579bc8070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:59.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.106+0000 7f5de9da8740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556579bbf970 0x556579bb8b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:59.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.106+0000 7f5de9da8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5de01add90 con 0x556579bb52a0 2024-05-12T22:12:59.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.106+0000 7f5de9da8740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f5de01af200 con 0x556579bbf970 2024-05-12T22:12:59.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.106+0000 7f5de9da8740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f5de01ae460 con 0x556579bb4980 2024-05-12T22:12:59.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.106+0000 7f5de9da8740 10 client.0 _ll_register_callbacks cb 0x7f5dd80975b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:12:59.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.107+0000 7f5ddf400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556579bb52a0 0x556579bc8070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:59.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.107+0000 7f5de5000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556579bbf970 0x556579bb8b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:59.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.107+0000 7f5de5000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556579bbf970 0x556579bb8b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:46218/0 (socket says 172.21.15.28:46218) 2024-05-12T22:12:59.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.107+0000 7f5de5000640 1 -- 172.21.15.28:0/3912547140 learned_addr learned my addr 172.21.15.28:0/3912547140 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:12:59.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.107+0000 7f5ddfe00640 1 --2- 172.21.15.28:0/3912547140 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556579bb4980 0x556579bc7b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:59.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.107+0000 7f5de5000640 1 -- 172.21.15.28:0/3912547140 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556579bb4980 msgr2=0x556579bc7b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:12:59.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.107+0000 7f5de5000640 1 --2- 172.21.15.28:0/3912547140 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556579bb4980 0x556579bc7b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:59.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.107+0000 7f5de5000640 1 -- 172.21.15.28:0/3912547140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556579bb52a0 msgr2=0x556579bc8070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:12:59.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.107+0000 7f5de5000640 1 --2- 172.21.15.28:0/3912547140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556579bb52a0 0x556579bc8070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:59.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.107+0000 7f5de5000640 1 -- 172.21.15.28:0/3912547140 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556579bb9100 con 0x556579bbf970 2024-05-12T22:12:59.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.107+0000 7f5ddfe00640 1 --2- 172.21.15.28:0/3912547140 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556579bb4980 0x556579bc7b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:12:59.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.107+0000 7f5ddf400640 1 --2- 172.21.15.28:0/3912547140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556579bb52a0 0x556579bc8070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:12:59.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.107+0000 7f5de5000640 1 --2- 172.21.15.28:0/3912547140 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556579bbf970 0x556579bb8b40 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f5dd8097bf0 tx=0x7f5dd8097b40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dff5bffb7e4aa76a server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:12:59.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.107+0000 7f5dce000640 10 client.0 ms_handle_connect on v2:172.21.15.86:3300/0 2024-05-12T22:12:59.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.108+0000 7f5dce000640 1 -- 172.21.15.28:0/3912547140 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dd8020070 con 0x556579bbf970 2024-05-12T22:12:59.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.108+0000 7f5dce000640 1 -- 172.21.15.28:0/3912547140 <== mon.2 v2:172.21.15.86:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f5dd80cccf0 con 0x556579bbf970 2024-05-12T22:12:59.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.108+0000 7f5de9da8740 1 -- 172.21.15.28:0/3912547140 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5dd002bcf0 con 0x556579bbf970 2024-05-12T22:12:59.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.108+0000 7f5dce000640 1 -- 172.21.15.28:0/3912547140 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f5dd807f050 con 0x556579bbf970 2024-05-12T22:12:59.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.108+0000 7f5de9da8740 10 client.26174 fetch_fsmap learned FSMap version 25 2024-05-12T22:12:59.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.108+0000 7f5de9da8740 1 -- 172.21.15.28:0/3912547140 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x556579c35cc0 con 0x556579bbf970 2024-05-12T22:12:59.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.108+0000 7f5dce000640 1 -- 172.21.15.28:0/3912547140 <== mon.2 v2:172.21.15.86:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5dd002bcf0 con 0x556579bbf970 2024-05-12T22:12:59.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.108+0000 7f5dce000640 1 -- 172.21.15.28:0/3912547140 <== mon.2 v2:172.21.15.86:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7f5dd80a06b0 con 0x556579bbf970 2024-05-12T22:12:59.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.108+0000 7f5de9da8740 10 client.26174 fetch_fsmap finished waiting for FSMap version 25 2024-05-12T22:12:59.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.108+0000 7f5de9da8740 10 client.26174 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.g 2024-05-12T22:12:59.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.108+0000 7f5de9da8740 20 client.26174 populate_metadata read hostname 'smithi028' 2024-05-12T22:12:59.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.108+0000 7f5de9da8740 1 --2- 172.21.15.28:0/3912547140 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x556579c363e0 0x556579c38800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:59.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.108+0000 7f5de9da8740 4 client.26174 mds_command: new command op to 24532 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-12T22:12:59.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.108+0000 7f5de9da8740 1 -- 172.21.15.28:0/3912547140 --> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556579b214b0 con 0x556579c363e0 2024-05-12T22:12:59.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.109+0000 7f5ddfe00640 1 --2- 172.21.15.28:0/3912547140 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x556579c363e0 0x556579c38800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:59.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.109+0000 7f5ddfe00640 1 --2- 172.21.15.28:0/3912547140 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x556579c363e0 0x556579c38800 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=5f172244533b227 server_cookie=4642d4cb8236f3fe in_seq=0 out_seq=0 2024-05-12T22:12:59.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.109+0000 7f5dce000640 10 client.26174 ms_handle_connect on v2:172.21.15.28:6838/123892158 2024-05-12T22:12:59.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.109+0000 7f5dce000640 1 -- 172.21.15.28:0/3912547140 <== mds.2 v2:172.21.15.28:6838/123892158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556579b214b0 con 0x556579c363e0 2024-05-12T22:12:59.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.109+0000 7f5dce000640 10 client.26174 handle_command_reply: tid=0 2024-05-12T22:12:59.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.109+0000 7f5de9da8740 1 client.26174 shutdown 2024-05-12T22:12:59.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.110+0000 7f5de9da8740 1 -- 172.21.15.28:0/3912547140 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556579bbf970 msgr2=0x556579bb8b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:12:59.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.110+0000 7f5de9da8740 1 --2- 172.21.15.28:0/3912547140 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556579bbf970 0x556579bb8b40 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f5dd8097bf0 tx=0x7f5dd8097b40 comp rx=0 tx=0).stop 2024-05-12T22:12:59.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.110+0000 7f5de9da8740 1 -- 172.21.15.28:0/3912547140 shutdown_connections 2024-05-12T22:12:59.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.110+0000 7f5de9da8740 1 --2- 172.21.15.28:0/3912547140 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x556579c363e0 0x556579c38800 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:59.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.110+0000 7f5de9da8740 1 --2- 172.21.15.28:0/3912547140 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556579bbf970 0x556579bb8b40 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:59.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.110+0000 7f5de9da8740 1 --2- 172.21.15.28:0/3912547140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556579bb52a0 0x556579bc8070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:59.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.110+0000 7f5de9da8740 1 --2- 172.21.15.28:0/3912547140 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556579bb4980 0x556579bc7b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:59.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.110+0000 7f5dce000640 0 client.26174 ms_handle_reset on v2:172.21.15.28:6838/123892158 2024-05-12T22:12:59.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.110+0000 7f5de9da8740 1 -- 172.21.15.28:0/3912547140 >> 172.21.15.28:0/3912547140 conn(0x556579bb0ca0 msgr2=0x556579baf530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:12:59.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.110+0000 7f5de9da8740 1 -- 172.21.15.28:0/3912547140 shutdown_connections 2024-05-12T22:12:59.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.110+0000 7f5de9da8740 1 -- 172.21.15.28:0/3912547140 wait complete. 2024-05-12T22:12:59.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.110+0000 7f5de9da8740 20 client.26174 trim_cache size 0 max 16384 2024-05-12T22:12:59.121 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-05-12T22:12:59.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.111+0000 7f5de7000640 1 -- 172.21.15.28:0/2179429868 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f5dc40a2d00 msgr2=0x7f5dc40a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:12:59.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.111+0000 7f5de7000640 1 --2- 172.21.15.28:0/2179429868 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f5dc40a2d00 0x7f5dc40a51c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f5dc8000c00 tx=0x7f5dc802a480 comp rx=0 tx=0).stop 2024-05-12T22:12:59.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.111+0000 7f5de7000640 1 -- 172.21.15.28:0/2179429868 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5de010e020 msgr2=0x7f5de019ebf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:12:59.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.111+0000 7f5de7000640 1 --2- 172.21.15.28:0/2179429868 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5de010e020 0x7f5de019ebf0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f5dd800d6b0 tx=0x7f5dd800db70 comp rx=0 tx=0).stop 2024-05-12T22:12:59.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.111+0000 7f5de7000640 1 -- 172.21.15.28:0/2179429868 shutdown_connections 2024-05-12T22:12:59.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.111+0000 7f5de7000640 1 --2- 172.21.15.28:0/2179429868 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f5dc40a2d00 0x7f5dc40a51c0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:59.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.111+0000 7f5de7000640 1 --2- 172.21.15.28:0/2179429868 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5de010e020 0x7f5de019ebf0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:59.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.111+0000 7f5de7000640 1 --2- 172.21.15.28:0/2179429868 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5de0074580 0x7f5de019a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:59.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.111+0000 7f5de7000640 1 --2- 172.21.15.28:0/2179429868 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5de0073c60 0x7f5de019a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:59.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.111+0000 7f5de7000640 1 -- 172.21.15.28:0/2179429868 >> 172.21.15.28:0/2179429868 conn(0x7f5de00fd2b0 msgr2=0x7f5de00fe0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:12:59.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.111+0000 7f5de7000640 1 -- 172.21.15.28:0/2179429868 shutdown_connections 2024-05-12T22:12:59.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.111+0000 7f5de7000640 1 -- 172.21.15.28:0/2179429868 wait complete. 2024-05-12T22:12:59.129 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-05-12T22:12:59.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.345+0000 7f974e800640 1 -- 172.21.15.28:0/2935882922 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97481080b0 msgr2=0x7f9748108490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:12:59.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.345+0000 7f974e800640 1 --2- 172.21.15.28:0/2935882922 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97481080b0 0x7f9748108490 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f9738009ca0 tx=0x7f973802a8c0 comp rx=0 tx=0).stop 2024-05-12T22:12:59.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.345+0000 7f974e800640 1 -- 172.21.15.28:0/2935882922 shutdown_connections 2024-05-12T22:12:59.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.345+0000 7f974e800640 1 --2- 172.21.15.28:0/2935882922 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f974810d130 0x7f9748114d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:59.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.345+0000 7f974e800640 1 --2- 172.21.15.28:0/2935882922 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9748108a60 0x7f974810ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:59.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.345+0000 7f974e800640 1 --2- 172.21.15.28:0/2935882922 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97481080b0 0x7f9748108490 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:59.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.345+0000 7f974e800640 1 -- 172.21.15.28:0/2935882922 >> 172.21.15.28:0/2935882922 conn(0x7f9748075d70 msgr2=0x7f9748078190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:12:59.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.345+0000 7f974e800640 1 -- 172.21.15.28:0/2935882922 shutdown_connections 2024-05-12T22:12:59.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.345+0000 7f974e800640 1 -- 172.21.15.28:0/2935882922 wait complete. 2024-05-12T22:12:59.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.346+0000 7f974e800640 1 Processor -- start 2024-05-12T22:12:59.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.346+0000 7f974e800640 1 -- start start 2024-05-12T22:12:59.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.347+0000 7f974e800640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f97481080b0 0x7f974819e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:59.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.347+0000 7f974e800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f974810d130 0x7f974819ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:59.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.347+0000 7f974e800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97481a2fc0 0x7f97481a3470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:59.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.347+0000 7f974e800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f97481181c0 con 0x7f97481a2fc0 2024-05-12T22:12:59.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.347+0000 7f974e800640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f9748118040 con 0x7f974810d130 2024-05-12T22:12:59.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.347+0000 7f974e800640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f9748118340 con 0x7f97481080b0 2024-05-12T22:12:59.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.347+0000 7f9747400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f97481080b0 0x7f974819e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:59.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.347+0000 7f9747e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97481a2fc0 0x7f97481a3470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:59.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.347+0000 7f9747400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f97481080b0 0x7f974819e700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:35950/0 (socket says 172.21.15.28:35950) 2024-05-12T22:12:59.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.347+0000 7f9747400640 1 -- 172.21.15.28:0/419256014 learned_addr learned my addr 172.21.15.28:0/419256014 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:12:59.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.347+0000 7f9746a00640 1 --2- 172.21.15.28:0/419256014 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f974810d130 0x7f974819ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:59.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.348+0000 7f9746a00640 1 -- 172.21.15.28:0/419256014 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f97481080b0 msgr2=0x7f974819e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:12:59.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.348+0000 7f9746a00640 1 --2- 172.21.15.28:0/419256014 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f97481080b0 0x7f974819e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:59.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.348+0000 7f9746a00640 1 -- 172.21.15.28:0/419256014 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97481a2fc0 msgr2=0x7f97481a3470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:12:59.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.348+0000 7f9746a00640 1 --2- 172.21.15.28:0/419256014 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97481a2fc0 0x7f97481a3470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:59.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.348+0000 7f9746a00640 1 -- 172.21.15.28:0/419256014 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97481a3ab0 con 0x7f974810d130 2024-05-12T22:12:59.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.348+0000 7f9747400640 1 --2- 172.21.15.28:0/419256014 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f97481080b0 0x7f974819e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:12:59.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.348+0000 7f9746a00640 1 --2- 172.21.15.28:0/419256014 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f974810d130 0x7f974819ec40 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f973000e9c0 tx=0x7f973000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:12:59.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.348+0000 7f973fe00640 1 -- 172.21.15.28:0/419256014 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f973000ccc0 con 0x7f974810d130 2024-05-12T22:12:59.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.348+0000 7f973fe00640 1 -- 172.21.15.28:0/419256014 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9730010910 con 0x7f974810d130 2024-05-12T22:12:59.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.348+0000 7f973fe00640 1 -- 172.21.15.28:0/419256014 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97300188c0 con 0x7f974810d130 2024-05-12T22:12:59.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.348+0000 7f974e800640 1 -- 172.21.15.28:0/419256014 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97481a3da0 con 0x7f974810d130 2024-05-12T22:12:59.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.348+0000 7f974e800640 1 -- 172.21.15.28:0/419256014 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97481ab540 con 0x7f974810d130 2024-05-12T22:12:59.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.352+0000 7f97517ba740 1 Processor -- start 2024-05-12T22:12:59.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.352+0000 7f973fe00640 1 -- 172.21.15.28:0/419256014 <== mon.2 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7f9730010ac0 con 0x7f974810d130 2024-05-12T22:12:59.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.352+0000 7f97517ba740 1 -- start start 2024-05-12T22:12:59.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.352+0000 7f97517ba740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aa499eb980 0x55aa499ebd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:59.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.352+0000 7f97517ba740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aa499ec2a0 0x55aa499f6430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:59.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.352+0000 7f97517ba740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55aa499f6970 0x55aa499f8d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:59.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.352+0000 7f97517ba740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55aa49950a30 con 0x55aa499ec2a0 2024-05-12T22:12:59.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.352+0000 7f97517ba740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x55aa498f8700 con 0x55aa499f6970 2024-05-12T22:12:59.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.352+0000 7f97517ba740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55aa4994c9f0 con 0x55aa499eb980 2024-05-12T22:12:59.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.352+0000 7f9747400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aa499ec2a0 0x55aa499f6430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:59.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.352+0000 7f9747e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aa499eb980 0x55aa499ebd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:59.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.353+0000 7f9747400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aa499ec2a0 0x55aa499f6430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36838/0 (socket says 172.21.15.28:36838) 2024-05-12T22:12:59.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.353+0000 7f9747400640 1 -- 172.21.15.28:0/564513129 learned_addr learned my addr 172.21.15.28:0/564513129 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:12:59.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.353+0000 7f9746a00640 1 --2- 172.21.15.28:0/564513129 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55aa499f6970 0x55aa499f8d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:59.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.353+0000 7f9747400640 1 -- 172.21.15.28:0/564513129 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aa499eb980 msgr2=0x55aa499ebd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:12:59.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.353+0000 7f9747400640 1 --2- 172.21.15.28:0/564513129 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aa499eb980 0x55aa499ebd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:59.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.353+0000 7f9747400640 1 -- 172.21.15.28:0/564513129 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55aa499f6970 msgr2=0x55aa499f8d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:12:59.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.353+0000 7f9747400640 1 --2- 172.21.15.28:0/564513129 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55aa499f6970 0x55aa499f8d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:59.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.353+0000 7f9747400640 1 -- 172.21.15.28:0/564513129 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aa496643f0 con 0x55aa499ec2a0 2024-05-12T22:12:59.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.353+0000 7f9746a00640 1 --2- 172.21.15.28:0/564513129 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55aa499f6970 0x55aa499f8d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:12:59.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.353+0000 7f973fe00640 1 --2- 172.21.15.28:0/419256014 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f972c0a2d80 0x7f972c0a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:59.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.353+0000 7f9747400640 1 --2- 172.21.15.28:0/564513129 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aa499ec2a0 0x55aa499f6430 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f97380097b0 tx=0x7f9738009800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=799b415e995605cc server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:12:59.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.354+0000 7f973fe00640 1 -- 172.21.15.28:0/419256014 <== mon.2 v2:172.21.15.86:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f97300cf760 con 0x7f974810d130 2024-05-12T22:12:59.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.354+0000 7f9747e00640 1 --2- 172.21.15.28:0/419256014 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f972c0a2d80 0x7f972c0a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:59.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.354+0000 7f973cc00640 1 -- 172.21.15.28:0/564513129 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9738049070 con 0x55aa499ec2a0 2024-05-12T22:12:59.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.354+0000 7f973cc00640 1 -- 172.21.15.28:0/564513129 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9738036040 con 0x55aa499ec2a0 2024-05-12T22:12:59.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.354+0000 7f973cc00640 1 -- 172.21.15.28:0/564513129 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97380305c0 con 0x55aa499ec2a0 2024-05-12T22:12:59.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.355+0000 7f97517ba740 1 -- 172.21.15.28:0/564513129 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aa499ec2a0 msgr2=0x55aa499f6430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:12:59.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.355+0000 7f9747e00640 1 --2- 172.21.15.28:0/419256014 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f972c0a2d80 0x7f972c0a5240 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f974000b6c0 tx=0x7f974000b2e0 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:12:59.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.355+0000 7f97517ba740 1 --2- 172.21.15.28:0/564513129 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aa499ec2a0 0x55aa499f6430 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f97380097b0 tx=0x7f9738009800 comp rx=0 tx=0).stop 2024-05-12T22:12:59.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.355+0000 7f97517ba740 1 -- 172.21.15.28:0/564513129 shutdown_connections 2024-05-12T22:12:59.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.355+0000 7f97517ba740 1 --2- 172.21.15.28:0/564513129 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55aa499f6970 0x55aa499f8d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:59.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.355+0000 7f97517ba740 1 --2- 172.21.15.28:0/564513129 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aa499ec2a0 0x55aa499f6430 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:59.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.355+0000 7f97517ba740 1 --2- 172.21.15.28:0/564513129 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aa499eb980 0x55aa499ebd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:59.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.355+0000 7f97517ba740 1 -- 172.21.15.28:0/564513129 >> 172.21.15.28:0/564513129 conn(0x55aa499e7ca0 msgr2=0x55aa499e8080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:12:59.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.355+0000 7f97517ba740 1 -- 172.21.15.28:0/564513129 shutdown_connections 2024-05-12T22:12:59.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.355+0000 7f97517ba740 1 -- 172.21.15.28:0/564513129 wait complete. 2024-05-12T22:12:59.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.355+0000 7f97517ba740 1 -- start start 2024-05-12T22:12:59.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.356+0000 7f97517ba740 1 Processor -- start 2024-05-12T22:12:59.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.356+0000 7f97517ba740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aa499eb980 0x55aa499f3930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:59.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.356+0000 7f97517ba740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55aa499ec2a0 0x55aa499f3e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:59.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.356+0000 7f97517ba740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aa499f6970 0x55aa499f43b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:59.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.356+0000 7f97517ba740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55aa499f9580 con 0x55aa499eb980 2024-05-12T22:12:59.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.356+0000 7f97517ba740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x55aa4990ba70 con 0x55aa499ec2a0 2024-05-12T22:12:59.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.356+0000 7f97517ba740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55aa499f9700 con 0x55aa499f6970 2024-05-12T22:12:59.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.356+0000 7f97517ba740 10 client.0 _ll_register_callbacks cb 0x55aa498eeba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:12:59.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.357+0000 7f9747400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aa499eb980 0x55aa499f3930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:59.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.357+0000 7f9747400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aa499eb980 0x55aa499f3930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36846/0 (socket says 172.21.15.28:36846) 2024-05-12T22:12:59.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.357+0000 7f9747400640 1 -- 172.21.15.28:0/2454352287 learned_addr learned my addr 172.21.15.28:0/2454352287 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:12:59.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.357+0000 7f9746a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55aa499ec2a0 0x55aa499f3e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:59.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.357+0000 7f9747e00640 1 --2- 172.21.15.28:0/2454352287 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aa499f6970 0x55aa499f43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:59.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.357+0000 7f9747400640 1 -- 172.21.15.28:0/2454352287 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aa499f6970 msgr2=0x55aa499f43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:12:59.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.357+0000 7f9747400640 1 --2- 172.21.15.28:0/2454352287 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aa499f6970 0x55aa499f43b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:59.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.357+0000 7f9747400640 1 -- 172.21.15.28:0/2454352287 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55aa499ec2a0 msgr2=0x55aa499f3e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:12:59.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.357+0000 7f9747400640 1 --2- 172.21.15.28:0/2454352287 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55aa499ec2a0 0x55aa499f3e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:59.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.357+0000 7f9747400640 1 -- 172.21.15.28:0/2454352287 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aa4972e7d0 con 0x55aa499eb980 2024-05-12T22:12:59.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.357+0000 7f9746a00640 1 --2- 172.21.15.28:0/2454352287 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55aa499ec2a0 0x55aa499f3e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:12:59.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.357+0000 7f9747400640 1 --2- 172.21.15.28:0/2454352287 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aa499eb980 0x55aa499f3930 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f9738004ad0 tx=0x7f973802fa60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35de273540487085 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:12:59.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.357+0000 7f9735600640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-12T22:12:59.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.358+0000 7f9735600640 1 -- 172.21.15.28:0/2454352287 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9738054060 con 0x55aa499eb980 2024-05-12T22:12:59.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.358+0000 7f9735600640 1 -- 172.21.15.28:0/2454352287 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f9738041040 con 0x55aa499eb980 2024-05-12T22:12:59.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.358+0000 7f97517ba740 1 -- 172.21.15.28:0/2454352287 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9738036040 con 0x55aa499eb980 2024-05-12T22:12:59.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.358+0000 7f9735600640 1 -- 172.21.15.28:0/2454352287 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f9738049030 con 0x55aa499eb980 2024-05-12T22:12:59.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.358+0000 7f97517ba740 10 client.16050 fetch_fsmap learned FSMap version 25 2024-05-12T22:12:59.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.358+0000 7f97517ba740 1 -- 172.21.15.28:0/2454352287 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55aa499f9580 con 0x55aa499eb980 2024-05-12T22:12:59.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.358+0000 7f9735600640 1 -- 172.21.15.28:0/2454352287 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9738036040 con 0x55aa499eb980 2024-05-12T22:12:59.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.359+0000 7f9735600640 1 -- 172.21.15.28:0/2454352287 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7f9738044410 con 0x55aa499eb980 2024-05-12T22:12:59.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.359+0000 7f97517ba740 10 client.16050 fetch_fsmap finished waiting for FSMap version 25 2024-05-12T22:12:59.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.359+0000 7f97517ba740 10 client.16050 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-05-12T22:12:59.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.359+0000 7f97517ba740 20 client.16050 populate_metadata read hostname 'smithi028' 2024-05-12T22:12:59.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.359+0000 7f97517ba740 1 --2- 172.21.15.28:0/2454352287 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x55aa49a069c0 0x55aa49a08dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:59.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.359+0000 7f97517ba740 4 client.16050 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-12T22:12:59.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.359+0000 7f97517ba740 1 -- 172.21.15.28:0/2454352287 --> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55aa4995e160 con 0x55aa49a069c0 2024-05-12T22:12:59.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.359+0000 7f9746a00640 1 --2- 172.21.15.28:0/2454352287 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x55aa49a069c0 0x55aa49a08dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:59.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.360+0000 7f9746a00640 1 --2- 172.21.15.28:0/2454352287 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x55aa49a069c0 0x55aa49a08dd0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=2a8ee0d5ea5b2ed0 server_cookie=e74b8b1c0409497b in_seq=0 out_seq=0 2024-05-12T22:12:59.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.360+0000 7f9735600640 10 client.16050 ms_handle_connect on v2:172.21.15.28:6840/761183870 2024-05-12T22:12:59.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.360+0000 7f9735600640 1 -- 172.21.15.28:0/2454352287 <== mds.1 v2:172.21.15.28:6840/761183870 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55aa4995e160 con 0x55aa49a069c0 2024-05-12T22:12:59.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.360+0000 7f9735600640 10 client.16050 handle_command_reply: tid=0 2024-05-12T22:12:59.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.360+0000 7f97517ba740 1 client.16050 shutdown 2024-05-12T22:12:59.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.361+0000 7f97517ba740 1 -- 172.21.15.28:0/2454352287 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aa499eb980 msgr2=0x55aa499f3930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:12:59.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.361+0000 7f97517ba740 1 --2- 172.21.15.28:0/2454352287 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aa499eb980 0x55aa499f3930 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f9738004ad0 tx=0x7f973802fa60 comp rx=0 tx=0).stop 2024-05-12T22:12:59.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.361+0000 7f97517ba740 1 -- 172.21.15.28:0/2454352287 shutdown_connections 2024-05-12T22:12:59.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.361+0000 7f97517ba740 1 --2- 172.21.15.28:0/2454352287 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x55aa49a069c0 0x55aa49a08dd0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:59.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.361+0000 7f97517ba740 1 --2- 172.21.15.28:0/2454352287 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aa499f6970 0x55aa499f43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:59.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.361+0000 7f97517ba740 1 --2- 172.21.15.28:0/2454352287 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55aa499ec2a0 0x55aa499f3e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:59.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.361+0000 7f97517ba740 1 --2- 172.21.15.28:0/2454352287 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aa499eb980 0x55aa499f3930 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:59.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.361+0000 7f9735600640 0 client.16050 ms_handle_reset on v2:172.21.15.28:6840/761183870 2024-05-12T22:12:59.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.361+0000 7f97517ba740 1 -- 172.21.15.28:0/2454352287 >> 172.21.15.28:0/2454352287 conn(0x55aa499e7ca0 msgr2=0x55aa499edaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:12:59.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.361+0000 7f97517ba740 1 -- 172.21.15.28:0/2454352287 shutdown_connections 2024-05-12T22:12:59.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.361+0000 7f97517ba740 1 -- 172.21.15.28:0/2454352287 wait complete. 2024-05-12T22:12:59.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.361+0000 7f97517ba740 20 client.16050 trim_cache size 0 max 16384 2024-05-12T22:12:59.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.376+0000 7f97517ba740 1 Processor -- start 2024-05-12T22:12:59.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.376+0000 7f97517ba740 1 -- start start 2024-05-12T22:12:59.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.376+0000 7f97517ba740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aa499eb980 0x55aa499f6170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:59.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.376+0000 7f97517ba740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aa499ec2a0 0x55aa49a02440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:59.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.376+0000 7f97517ba740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55aa499f6970 0x55aa499e6dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:59.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.376+0000 7f97517ba740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f97481b6340 con 0x55aa499ec2a0 2024-05-12T22:12:59.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.376+0000 7f97517ba740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f97481b3c10 con 0x55aa499f6970 2024-05-12T22:12:59.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.376+0000 7f97517ba740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f97481b1a60 con 0x55aa499eb980 2024-05-12T22:12:59.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.376+0000 7f9746a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aa499ec2a0 0x55aa49a02440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:59.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.377+0000 7f9746a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aa499ec2a0 0x55aa49a02440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36860/0 (socket says 172.21.15.28:36860) 2024-05-12T22:12:59.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.377+0000 7f9746a00640 1 -- 172.21.15.28:0/4258304979 learned_addr learned my addr 172.21.15.28:0/4258304979 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:12:59.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.377+0000 7f9747400640 1 --2- 172.21.15.28:0/4258304979 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aa499eb980 0x55aa499f6170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:59.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.377+0000 7f9747e00640 1 --2- 172.21.15.28:0/4258304979 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55aa499f6970 0x55aa499e6dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:59.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.377+0000 7f9746a00640 1 -- 172.21.15.28:0/4258304979 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aa499eb980 msgr2=0x55aa499f6170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:12:59.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.377+0000 7f9746a00640 1 --2- 172.21.15.28:0/4258304979 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aa499eb980 0x55aa499f6170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:59.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.377+0000 7f9746a00640 1 -- 172.21.15.28:0/4258304979 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55aa499f6970 msgr2=0x55aa499e6dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:12:59.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.377+0000 7f9746a00640 1 --2- 172.21.15.28:0/4258304979 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55aa499f6970 0x55aa499e6dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:59.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.377+0000 7f9746a00640 1 -- 172.21.15.28:0/4258304979 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aa499f28e0 con 0x55aa499ec2a0 2024-05-12T22:12:59.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.377+0000 7f9746a00640 1 --2- 172.21.15.28:0/4258304979 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aa499ec2a0 0x55aa49a02440 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f9730010f50 tx=0x7f97300d5530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76e1ca4fe7cd2d50 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:12:59.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.378+0000 7f9737e00640 1 -- 172.21.15.28:0/4258304979 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97300b1070 con 0x55aa499ec2a0 2024-05-12T22:12:59.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.378+0000 7f9737e00640 1 -- 172.21.15.28:0/4258304979 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f973009bd00 con 0x55aa499ec2a0 2024-05-12T22:12:59.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.378+0000 7f9737e00640 1 -- 172.21.15.28:0/4258304979 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97300cfb50 con 0x55aa499ec2a0 2024-05-12T22:12:59.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.378+0000 7f97517ba740 1 -- 172.21.15.28:0/4258304979 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aa499ec2a0 msgr2=0x55aa49a02440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:12:59.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.378+0000 7f97517ba740 1 --2- 172.21.15.28:0/4258304979 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aa499ec2a0 0x55aa49a02440 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f9730010f50 tx=0x7f97300d5530 comp rx=0 tx=0).stop 2024-05-12T22:12:59.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.378+0000 7f97517ba740 1 -- 172.21.15.28:0/4258304979 shutdown_connections 2024-05-12T22:12:59.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.378+0000 7f97517ba740 1 --2- 172.21.15.28:0/4258304979 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55aa499f6970 0x55aa499e6dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:59.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.378+0000 7f97517ba740 1 --2- 172.21.15.28:0/4258304979 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aa499ec2a0 0x55aa49a02440 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:59.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.378+0000 7f97517ba740 1 --2- 172.21.15.28:0/4258304979 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aa499eb980 0x55aa499f6170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:59.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.378+0000 7f97517ba740 1 -- 172.21.15.28:0/4258304979 >> 172.21.15.28:0/4258304979 conn(0x55aa499e7ca0 msgr2=0x55aa499e8080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:12:59.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.378+0000 7f97517ba740 1 -- 172.21.15.28:0/4258304979 shutdown_connections 2024-05-12T22:12:59.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.378+0000 7f97517ba740 1 -- 172.21.15.28:0/4258304979 wait complete. 2024-05-12T22:12:59.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.379+0000 7f97517ba740 1 -- start start 2024-05-12T22:12:59.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.379+0000 7f97517ba740 1 Processor -- start 2024-05-12T22:12:59.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.380+0000 7f97517ba740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aa499eb980 0x55aa499fa510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:59.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.380+0000 7f97517ba740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55aa499ec2a0 0x55aa499faa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:59.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.380+0000 7f97517ba740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aa499f6970 0x55aa499faf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:59.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.380+0000 7f97517ba740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f97481b20d0 con 0x55aa499f6970 2024-05-12T22:12:59.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.380+0000 7f97517ba740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f97481b3540 con 0x55aa499ec2a0 2024-05-12T22:12:59.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.380+0000 7f97517ba740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f97481b27a0 con 0x55aa499eb980 2024-05-12T22:12:59.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.380+0000 7f97517ba740 10 client.0 _ll_register_callbacks cb 0x7f9740004040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:12:59.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.380+0000 7f9747e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aa499f6970 0x55aa499faf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:59.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.380+0000 7f9747400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aa499eb980 0x55aa499fa510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:59.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.380+0000 7f9747e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aa499f6970 0x55aa499faf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36864/0 (socket says 172.21.15.28:36864) 2024-05-12T22:12:59.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.380+0000 7f9747e00640 1 -- 172.21.15.28:0/800521657 learned_addr learned my addr 172.21.15.28:0/800521657 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:12:59.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.380+0000 7f9747400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aa499eb980 0x55aa499fa510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:35988/0 (socket says 172.21.15.28:35988) 2024-05-12T22:12:59.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.380+0000 7f9746a00640 1 --2- 172.21.15.28:0/800521657 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55aa499ec2a0 0x55aa499faa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:59.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.380+0000 7f9747e00640 1 -- 172.21.15.28:0/800521657 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aa499eb980 msgr2=0x55aa499fa510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:12:59.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.380+0000 7f9747e00640 1 --2- 172.21.15.28:0/800521657 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aa499eb980 0x55aa499fa510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:59.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.380+0000 7f9747e00640 1 -- 172.21.15.28:0/800521657 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55aa499ec2a0 msgr2=0x55aa499faa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:12:59.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.380+0000 7f9747e00640 1 --2- 172.21.15.28:0/800521657 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55aa499ec2a0 0x55aa499faa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:59.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.380+0000 7f9747e00640 1 -- 172.21.15.28:0/800521657 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aa499ec0d0 con 0x55aa499f6970 2024-05-12T22:12:59.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.381+0000 7f9747e00640 1 --2- 172.21.15.28:0/800521657 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aa499f6970 0x55aa499faf90 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f9740003ec0 tx=0x7f9740004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21a5c6948e8a0a3d server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:12:59.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.381+0000 7f9735600640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-12T22:12:59.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.381+0000 7f9735600640 1 -- 172.21.15.28:0/800521657 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9740005020 con 0x55aa499f6970 2024-05-12T22:12:59.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.381+0000 7f9735600640 1 -- 172.21.15.28:0/800521657 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f97400051c0 con 0x55aa499f6970 2024-05-12T22:12:59.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.381+0000 7f97517ba740 1 -- 172.21.15.28:0/800521657 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f973009bd00 con 0x55aa499f6970 2024-05-12T22:12:59.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.381+0000 7f9735600640 1 -- 172.21.15.28:0/800521657 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f97400053c0 con 0x55aa499f6970 2024-05-12T22:12:59.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.381+0000 7f97517ba740 10 client.16056 fetch_fsmap learned FSMap version 25 2024-05-12T22:12:59.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.381+0000 7f97517ba740 1 -- 172.21.15.28:0/800521657 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55aa49a6d230 con 0x55aa499f6970 2024-05-12T22:12:59.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.382+0000 7f9735600640 1 -- 172.21.15.28:0/800521657 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f973009bd00 con 0x55aa499f6970 2024-05-12T22:12:59.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.382+0000 7f9735600640 1 -- 172.21.15.28:0/800521657 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7f9740028070 con 0x55aa499f6970 2024-05-12T22:12:59.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.382+0000 7f97517ba740 10 client.16056 fetch_fsmap finished waiting for FSMap version 25 2024-05-12T22:12:59.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.382+0000 7f97517ba740 10 client.16056 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-05-12T22:12:59.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.382+0000 7f97517ba740 20 client.16056 populate_metadata read hostname 'smithi028' 2024-05-12T22:12:59.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.382+0000 7f97517ba740 1 --2- 172.21.15.28:0/800521657 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x55aa49a6da30 0x55aa49a6fe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:59.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.382+0000 7f97517ba740 4 client.16056 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-12T22:12:59.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.382+0000 7f97517ba740 1 -- 172.21.15.28:0/800521657 --> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55aa499584b0 con 0x55aa49a6da30 2024-05-12T22:12:59.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.382+0000 7f9747400640 1 --2- 172.21.15.28:0/800521657 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x55aa49a6da30 0x55aa49a6fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:59.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.383+0000 7f9747400640 1 --2- 172.21.15.28:0/800521657 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x55aa49a6da30 0x55aa49a6fe50 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=eaa5ebdebbaed9e server_cookie=843a801228748e61 in_seq=0 out_seq=0 2024-05-12T22:12:59.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.383+0000 7f9735600640 10 client.16056 ms_handle_connect on v2:172.21.15.28:6840/761183870 2024-05-12T22:12:59.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.383+0000 7f9735600640 1 -- 172.21.15.28:0/800521657 <== mds.1 v2:172.21.15.28:6840/761183870 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55aa499584b0 con 0x55aa49a6da30 2024-05-12T22:12:59.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.383+0000 7f9735600640 10 client.16056 handle_command_reply: tid=0 2024-05-12T22:12:59.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.383+0000 7f97517ba740 1 client.16056 shutdown 2024-05-12T22:12:59.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.383+0000 7f97517ba740 1 -- 172.21.15.28:0/800521657 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aa499f6970 msgr2=0x55aa499faf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:12:59.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.383+0000 7f97517ba740 1 --2- 172.21.15.28:0/800521657 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aa499f6970 0x55aa499faf90 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f9740003ec0 tx=0x7f9740004730 comp rx=0 tx=0).stop 2024-05-12T22:12:59.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.384+0000 7f97517ba740 1 -- 172.21.15.28:0/800521657 shutdown_connections 2024-05-12T22:12:59.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.384+0000 7f97517ba740 1 --2- 172.21.15.28:0/800521657 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x55aa49a6da30 0x55aa49a6fe50 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:59.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.384+0000 7f97517ba740 1 --2- 172.21.15.28:0/800521657 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aa499f6970 0x55aa499faf90 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:59.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.384+0000 7f97517ba740 1 --2- 172.21.15.28:0/800521657 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55aa499ec2a0 0x55aa499faa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:59.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.384+0000 7f97517ba740 1 --2- 172.21.15.28:0/800521657 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aa499eb980 0x55aa499fa510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:59.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.384+0000 7f97517ba740 1 -- 172.21.15.28:0/800521657 >> 172.21.15.28:0/800521657 conn(0x55aa499e7ca0 msgr2=0x55aa499f4160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:12:59.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.384+0000 7f9735600640 0 client.16056 ms_handle_reset on v2:172.21.15.28:6840/761183870 2024-05-12T22:12:59.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.384+0000 7f97517ba740 1 -- 172.21.15.28:0/800521657 shutdown_connections 2024-05-12T22:12:59.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.384+0000 7f97517ba740 1 -- 172.21.15.28:0/800521657 wait complete. 2024-05-12T22:12:59.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.384+0000 7f97517ba740 20 client.16056 trim_cache size 0 max 16384 2024-05-12T22:12:59.393 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-05-12T22:12:59.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.384+0000 7f974e800640 1 -- 172.21.15.28:0/419256014 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f972c0a2d80 msgr2=0x7f972c0a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:12:59.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.384+0000 7f974e800640 1 --2- 172.21.15.28:0/419256014 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f972c0a2d80 0x7f972c0a5240 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f974000b6c0 tx=0x7f974000b2e0 comp rx=0 tx=0).stop 2024-05-12T22:12:59.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.385+0000 7f974e800640 1 -- 172.21.15.28:0/419256014 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f974810d130 msgr2=0x7f974819ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:12:59.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.385+0000 7f974e800640 1 --2- 172.21.15.28:0/419256014 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f974810d130 0x7f974819ec40 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f973000e9c0 tx=0x7f973000ee80 comp rx=0 tx=0).stop 2024-05-12T22:12:59.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.385+0000 7f974e800640 1 -- 172.21.15.28:0/419256014 shutdown_connections 2024-05-12T22:12:59.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.385+0000 7f974e800640 1 --2- 172.21.15.28:0/419256014 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f972c0a2d80 0x7f972c0a5240 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:59.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.385+0000 7f974e800640 1 --2- 172.21.15.28:0/419256014 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97481a2fc0 0x7f97481a3470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:59.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.385+0000 7f974e800640 1 --2- 172.21.15.28:0/419256014 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f974810d130 0x7f974819ec40 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:59.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.385+0000 7f974e800640 1 --2- 172.21.15.28:0/419256014 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f97481080b0 0x7f974819e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:59.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.385+0000 7f974e800640 1 -- 172.21.15.28:0/419256014 >> 172.21.15.28:0/419256014 conn(0x7f9748075d70 msgr2=0x7f9748077140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:12:59.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.385+0000 7f974e800640 1 -- 172.21.15.28:0/419256014 shutdown_connections 2024-05-12T22:12:59.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.385+0000 7f974e800640 1 -- 172.21.15.28:0/419256014 wait complete. 2024-05-12T22:12:59.401 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:12:59 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/3260465127' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:12:59.403 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-05-12T22:12:59.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:12:59 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/3260465127' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:12:59.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.599+0000 7f17b3000640 1 -- 172.21.15.28:0/1943059576 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17ac076680 msgr2=0x7f17ac112040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:12:59.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.599+0000 7f17b3000640 1 --2- 172.21.15.28:0/1943059576 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17ac076680 0x7f17ac112040 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f1798009900 tx=0x7f179802a410 comp rx=0 tx=0).stop 2024-05-12T22:12:59.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.599+0000 7f17b3000640 1 -- 172.21.15.28:0/1943059576 shutdown_connections 2024-05-12T22:12:59.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.599+0000 7f17b3000640 1 --2- 172.21.15.28:0/1943059576 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f17ac112580 0x7f17ac114970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:59.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.599+0000 7f17b3000640 1 --2- 172.21.15.28:0/1943059576 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17ac076680 0x7f17ac112040 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:59.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.599+0000 7f17b3000640 1 --2- 172.21.15.28:0/1943059576 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f17ac075d60 0x7f17ac076140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:59.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.599+0000 7f17b3000640 1 -- 172.21.15.28:0/1943059576 >> 172.21.15.28:0/1943059576 conn(0x7f17ac0ff370 msgr2=0x7f17ac101790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:12:59.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.599+0000 7f17b3000640 1 -- 172.21.15.28:0/1943059576 shutdown_connections 2024-05-12T22:12:59.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.599+0000 7f17b3000640 1 -- 172.21.15.28:0/1943059576 wait complete. 2024-05-12T22:12:59.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.600+0000 7f17b3000640 1 Processor -- start 2024-05-12T22:12:59.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.600+0000 7f17b3000640 1 -- start start 2024-05-12T22:12:59.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.601+0000 7f17b3000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f17ac075d60 0x7f17ac19e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:59.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.601+0000 7f17b3000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17ac076680 0x7f17ac19eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:59.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.601+0000 7f17b3000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f17ac112580 0x7f17ac1a2ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:59.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.601+0000 7f17b3000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f17ac1180d0 con 0x7f17ac076680 2024-05-12T22:12:59.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.601+0000 7f17b3000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f17ac117f50 con 0x7f17ac075d60 2024-05-12T22:12:59.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.601+0000 7f17b3000640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f17ac118250 con 0x7f17ac112580 2024-05-12T22:12:59.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.601+0000 7f17b2600640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f17ac112580 0x7f17ac1a2ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:59.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.601+0000 7f17b1c00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f17ac075d60 0x7f17ac19e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:59.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.601+0000 7f17b1200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17ac076680 0x7f17ac19eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:59.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.601+0000 7f17b2600640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f17ac112580 0x7f17ac1a2ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:36012/0 (socket says 172.21.15.28:36012) 2024-05-12T22:12:59.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.601+0000 7f17b2600640 1 -- 172.21.15.28:0/1792030337 learned_addr learned my addr 172.21.15.28:0/1792030337 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:12:59.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.601+0000 7f17b1c00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f17ac075d60 0x7f17ac19e630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:46304/0 (socket says 172.21.15.28:46304) 2024-05-12T22:12:59.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.602+0000 7f17b2600640 1 -- 172.21.15.28:0/1792030337 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f17ac075d60 msgr2=0x7f17ac19e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:12:59.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.602+0000 7f17b2600640 1 --2- 172.21.15.28:0/1792030337 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f17ac075d60 0x7f17ac19e630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:59.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.602+0000 7f17b2600640 1 -- 172.21.15.28:0/1792030337 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17ac076680 msgr2=0x7f17ac19eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:12:59.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.602+0000 7f17b2600640 1 --2- 172.21.15.28:0/1792030337 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17ac076680 0x7f17ac19eb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:59.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.602+0000 7f17b2600640 1 -- 172.21.15.28:0/1792030337 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17ac1a3620 con 0x7f17ac112580 2024-05-12T22:12:59.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.602+0000 7f17b1c00640 1 --2- 172.21.15.28:0/1792030337 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f17ac075d60 0x7f17ac19e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:12:59.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.602+0000 7f17b2600640 1 --2- 172.21.15.28:0/1792030337 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f17ac112580 0x7f17ac1a2ef0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f17a4009c90 tx=0x7f17a400c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:12:59.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.602+0000 7f17aa000640 1 -- 172.21.15.28:0/1792030337 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17a4003b70 con 0x7f17ac112580 2024-05-12T22:12:59.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.602+0000 7f17aa000640 1 -- 172.21.15.28:0/1792030337 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f17a4003d10 con 0x7f17ac112580 2024-05-12T22:12:59.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.602+0000 7f17aa000640 1 -- 172.21.15.28:0/1792030337 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17a4010960 con 0x7f17ac112580 2024-05-12T22:12:59.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.602+0000 7f17b3000640 1 -- 172.21.15.28:0/1792030337 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17ac1a3910 con 0x7f17ac112580 2024-05-12T22:12:59.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.602+0000 7f17b3000640 1 -- 172.21.15.28:0/1792030337 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17ac1ab1a0 con 0x7f17ac112580 2024-05-12T22:12:59.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.605+0000 7f17b7432740 1 Processor -- start 2024-05-12T22:12:59.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.606+0000 7f17aa000640 1 -- 172.21.15.28:0/1792030337 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7f17a4004020 con 0x7f17ac112580 2024-05-12T22:12:59.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.606+0000 7f17b7432740 1 -- start start 2024-05-12T22:12:59.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.606+0000 7f17b7432740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c52091cb60 0x55c52091cf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:59.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.606+0000 7f17b7432740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c52091d480 0x55c520927610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:59.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.606+0000 7f17b7432740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c520927b50 0x55c520929f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:59.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.606+0000 7f17b7432740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c520881a30 con 0x55c52091cb60 2024-05-12T22:12:59.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.606+0000 7f17b7432740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x55c520829700 con 0x55c520927b50 2024-05-12T22:12:59.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.606+0000 7f17b7432740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55c52087d9f0 con 0x55c52091d480 2024-05-12T22:12:59.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.607+0000 7f17b1200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c52091cb60 0x55c52091cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:59.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.607+0000 7f17b1c00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c52091d480 0x55c520927610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:59.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.607+0000 7f17b1c00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c52091d480 0x55c520927610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:36014/0 (socket says 172.21.15.28:36014) 2024-05-12T22:12:59.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.607+0000 7f17b1c00640 1 -- 172.21.15.28:0/3324086375 learned_addr learned my addr 172.21.15.28:0/3324086375 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:12:59.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.607+0000 7f17b1200640 1 --2- 172.21.15.28:0/3324086375 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c520927b50 0x55c520929f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:59.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.607+0000 7f17b1c00640 1 -- 172.21.15.28:0/3324086375 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c520927b50 msgr2=0x55c520929f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:12:59.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.607+0000 7f17b1c00640 1 --2- 172.21.15.28:0/3324086375 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c520927b50 0x55c520929f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:59.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.607+0000 7f17b1c00640 1 -- 172.21.15.28:0/3324086375 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c52091cb60 msgr2=0x55c52091cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:12:59.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.607+0000 7f17b1c00640 1 --2- 172.21.15.28:0/3324086375 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c52091cb60 0x55c52091cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:59.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.607+0000 7f17b1c00640 1 -- 172.21.15.28:0/3324086375 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c5205953f0 con 0x55c52091d480 2024-05-12T22:12:59.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.607+0000 7f17b1200640 1 --2- 172.21.15.28:0/3324086375 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c520927b50 0x55c520929f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-12T22:12:59.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.607+0000 7f17b1c00640 1 --2- 172.21.15.28:0/3324086375 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c52091d480 0x55c520927610 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f17a00115f0 tx=0x7f17a0011ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=17528d59fc324397 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:12:59.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.608+0000 7f17aa000640 1 --2- 172.21.15.28:0/1792030337 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f17940a2d80 0x7f17940a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:59.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.608+0000 7f179ea00640 1 -- 172.21.15.28:0/3324086375 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17a001e070 con 0x55c52091d480 2024-05-12T22:12:59.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.608+0000 7f179ea00640 1 -- 172.21.15.28:0/3324086375 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f17a000fb20 con 0x55c52091d480 2024-05-12T22:12:59.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.608+0000 7f179ea00640 1 -- 172.21.15.28:0/3324086375 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17a0019c40 con 0x55c52091d480 2024-05-12T22:12:59.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.608+0000 7f17aa000640 1 -- 172.21.15.28:0/1792030337 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f17a40cf930 con 0x7f17ac112580 2024-05-12T22:12:59.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.608+0000 7f17b1200640 1 --2- 172.21.15.28:0/1792030337 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f17940a2d80 0x7f17940a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:59.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.608+0000 7f17b7432740 1 -- 172.21.15.28:0/3324086375 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c52091d480 msgr2=0x55c520927610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:12:59.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.608+0000 7f17b7432740 1 --2- 172.21.15.28:0/3324086375 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c52091d480 0x55c520927610 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f17a00115f0 tx=0x7f17a0011ab0 comp rx=0 tx=0).stop 2024-05-12T22:12:59.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.609+0000 7f17b7432740 1 -- 172.21.15.28:0/3324086375 shutdown_connections 2024-05-12T22:12:59.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.609+0000 7f17b7432740 1 --2- 172.21.15.28:0/3324086375 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c520927b50 0x55c520929f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:59.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.609+0000 7f17b7432740 1 --2- 172.21.15.28:0/3324086375 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c52091d480 0x55c520927610 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:59.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.609+0000 7f17b7432740 1 --2- 172.21.15.28:0/3324086375 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c52091cb60 0x55c52091cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:59.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.609+0000 7f17b7432740 1 -- 172.21.15.28:0/3324086375 >> 172.21.15.28:0/3324086375 conn(0x55c520918e80 msgr2=0x55c520919260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:12:59.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.609+0000 7f17b1200640 1 --2- 172.21.15.28:0/1792030337 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f17940a2d80 0x7f17940a5240 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f1798009230 tx=0x7f1798031400 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:12:59.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.609+0000 7f17b7432740 1 -- 172.21.15.28:0/3324086375 shutdown_connections 2024-05-12T22:12:59.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.609+0000 7f17b7432740 1 -- 172.21.15.28:0/3324086375 wait complete. 2024-05-12T22:12:59.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.609+0000 7f17b7432740 1 -- start start 2024-05-12T22:12:59.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.609+0000 7f17b7432740 1 Processor -- start 2024-05-12T22:12:59.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.610+0000 7f17b7432740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c52091cb60 0x55c520924aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:59.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.610+0000 7f17b7432740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c52091d480 0x55c520924fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:59.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.610+0000 7f17b7432740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c520927b50 0x55c520925520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:59.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.610+0000 7f17b7432740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c52092a940 con 0x55c52091d480 2024-05-12T22:12:59.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.610+0000 7f17b7432740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x55c52083ca70 con 0x55c520927b50 2024-05-12T22:12:59.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.610+0000 7f17b7432740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55c52092aac0 con 0x55c52091cb60 2024-05-12T22:12:59.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.610+0000 7f17b7432740 10 client.0 _ll_register_callbacks cb 0x55c52081fba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:12:59.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.610+0000 7f17b1200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c52091d480 0x55c520924fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:59.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.610+0000 7f17b1c00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c52091cb60 0x55c520924aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:59.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.610+0000 7f17b1200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c52091d480 0x55c520924fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36914/0 (socket says 172.21.15.28:36914) 2024-05-12T22:12:59.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.610+0000 7f17b1200640 1 -- 172.21.15.28:0/255935119 learned_addr learned my addr 172.21.15.28:0/255935119 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:12:59.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.610+0000 7f17b2600640 1 --2- 172.21.15.28:0/255935119 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c520927b50 0x55c520925520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:59.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.611+0000 7f17b1c00640 1 -- 172.21.15.28:0/255935119 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c520927b50 msgr2=0x55c520925520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:12:59.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.611+0000 7f17b1c00640 1 --2- 172.21.15.28:0/255935119 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c520927b50 0x55c520925520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:59.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.611+0000 7f17b1c00640 1 -- 172.21.15.28:0/255935119 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c52091d480 msgr2=0x55c520924fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:12:59.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.611+0000 7f17b1c00640 1 --2- 172.21.15.28:0/255935119 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c52091d480 0x55c520924fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:59.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.611+0000 7f17b1c00640 1 -- 172.21.15.28:0/255935119 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c52065f7d0 con 0x55c52091cb60 2024-05-12T22:12:59.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.611+0000 7f17b2600640 1 --2- 172.21.15.28:0/255935119 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c520927b50 0x55c520925520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:12:59.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.611+0000 7f17b1c00640 1 --2- 172.21.15.28:0/255935119 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c52091cb60 0x55c520924aa0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f17a00115f0 tx=0x7f17a000f870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b3aa20c17df341ea server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:12:59.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.611+0000 7f1793400640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-12T22:12:59.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.611+0000 7f1793400640 1 -- 172.21.15.28:0/255935119 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17a0027060 con 0x55c52091cb60 2024-05-12T22:12:59.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.611+0000 7f1793400640 1 -- 172.21.15.28:0/255935119 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f17a00199c0 con 0x55c52091cb60 2024-05-12T22:12:59.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.611+0000 7f17b7432740 1 -- 172.21.15.28:0/255935119 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f17a000fb20 con 0x55c52091cb60 2024-05-12T22:12:59.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.612+0000 7f1793400640 1 -- 172.21.15.28:0/255935119 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f17a001e040 con 0x55c52091cb60 2024-05-12T22:12:59.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.612+0000 7f17b7432740 10 client.26158 fetch_fsmap learned FSMap version 25 2024-05-12T22:12:59.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.612+0000 7f17b7432740 1 -- 172.21.15.28:0/255935119 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55c520936b00 con 0x55c52091cb60 2024-05-12T22:12:59.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.612+0000 7f1793400640 1 -- 172.21.15.28:0/255935119 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f17a000fb20 con 0x55c52091cb60 2024-05-12T22:12:59.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.612+0000 7f1793400640 1 -- 172.21.15.28:0/255935119 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7f17a0021030 con 0x55c52091cb60 2024-05-12T22:12:59.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.612+0000 7f17b7432740 10 client.26158 fetch_fsmap finished waiting for FSMap version 25 2024-05-12T22:12:59.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.612+0000 7f17b7432740 10 client.26158 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:12:59.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.612+0000 7f17b7432740 20 client.26158 populate_metadata read hostname 'smithi028' 2024-05-12T22:12:59.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.612+0000 7f17b7432740 1 --2- 172.21.15.28:0/255935119 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55c520937eb0 0x55c52093a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:59.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.612+0000 7f17b7432740 4 client.26158 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-12T22:12:59.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.612+0000 7f17b7432740 1 -- 172.21.15.28:0/255935119 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c52088f160 con 0x55c520937eb0 2024-05-12T22:12:59.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.613+0000 7f17b1200640 1 --2- 172.21.15.28:0/255935119 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55c520937eb0 0x55c52093a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:59.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.613+0000 7f17b1200640 1 --2- 172.21.15.28:0/255935119 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55c520937eb0 0x55c52093a2a0 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4dff8532cd06f059 server_cookie=3b4a6dabb9a377d6 in_seq=0 out_seq=0 2024-05-12T22:12:59.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.613+0000 7f1793400640 10 client.26158 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:12:59.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.614+0000 7f1793400640 1 -- 172.21.15.28:0/255935119 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c52088f160 con 0x55c520937eb0 2024-05-12T22:12:59.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.614+0000 7f1793400640 10 client.26158 handle_command_reply: tid=0 2024-05-12T22:12:59.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.614+0000 7f17b7432740 1 client.26158 shutdown 2024-05-12T22:12:59.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.614+0000 7f17b7432740 1 -- 172.21.15.28:0/255935119 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c52091cb60 msgr2=0x55c520924aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:12:59.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.614+0000 7f17b7432740 1 --2- 172.21.15.28:0/255935119 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c52091cb60 0x55c520924aa0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f17a00115f0 tx=0x7f17a000f870 comp rx=0 tx=0).stop 2024-05-12T22:12:59.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.614+0000 7f17b7432740 1 -- 172.21.15.28:0/255935119 shutdown_connections 2024-05-12T22:12:59.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.614+0000 7f17b7432740 1 --2- 172.21.15.28:0/255935119 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55c520937eb0 0x55c52093a2a0 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:59.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.614+0000 7f17b7432740 1 --2- 172.21.15.28:0/255935119 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c520927b50 0x55c520925520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:59.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.614+0000 7f17b7432740 1 --2- 172.21.15.28:0/255935119 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c52091d480 0x55c520924fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:59.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.614+0000 7f1793400640 0 client.26158 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:12:59.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.614+0000 7f17b7432740 1 --2- 172.21.15.28:0/255935119 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c52091cb60 0x55c520924aa0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:59.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.614+0000 7f17b7432740 1 -- 172.21.15.28:0/255935119 >> 172.21.15.28:0/255935119 conn(0x55c520918e80 msgr2=0x55c52091ec80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:12:59.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.614+0000 7f17b7432740 1 -- 172.21.15.28:0/255935119 shutdown_connections 2024-05-12T22:12:59.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.614+0000 7f17b7432740 1 -- 172.21.15.28:0/255935119 wait complete. 2024-05-12T22:12:59.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.615+0000 7f17b7432740 20 client.26158 trim_cache size 0 max 16384 2024-05-12T22:12:59.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.630+0000 7f17b7432740 1 Processor -- start 2024-05-12T22:12:59.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.630+0000 7f17b7432740 1 -- start start 2024-05-12T22:12:59.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.630+0000 7f17b7432740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c52091cb60 0x55c520933800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:59.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.630+0000 7f17b7432740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c52091d480 0x55c52091ee00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:59.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.630+0000 7f17b7432740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c520927b50 0x55c52093faa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:59.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.630+0000 7f17b7432740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f17ac1b61b0 con 0x55c52091cb60 2024-05-12T22:12:59.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.630+0000 7f17b7432740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f17ac1b3a80 con 0x55c52091d480 2024-05-12T22:12:59.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.630+0000 7f17b7432740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f17ac1b18d0 con 0x55c520927b50 2024-05-12T22:12:59.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.630+0000 7f17b1c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c52091cb60 0x55c520933800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:59.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.630+0000 7f17b1200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c52091d480 0x55c52091ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:59.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.630+0000 7f17b1c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c52091cb60 0x55c520933800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36930/0 (socket says 172.21.15.28:36930) 2024-05-12T22:12:59.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.630+0000 7f17b1c00640 1 -- 172.21.15.28:0/2786175025 learned_addr learned my addr 172.21.15.28:0/2786175025 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:12:59.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.630+0000 7f17b2600640 1 --2- 172.21.15.28:0/2786175025 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c520927b50 0x55c52093faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:59.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.631+0000 7f17b1200640 1 -- 172.21.15.28:0/2786175025 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c520927b50 msgr2=0x55c52093faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:12:59.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.631+0000 7f17b1200640 1 --2- 172.21.15.28:0/2786175025 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c520927b50 0x55c52093faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:59.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.631+0000 7f17b1200640 1 -- 172.21.15.28:0/2786175025 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c52091cb60 msgr2=0x55c520933800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:12:59.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.631+0000 7f17b1200640 1 --2- 172.21.15.28:0/2786175025 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c52091cb60 0x55c520933800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:59.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.631+0000 7f17b1200640 1 -- 172.21.15.28:0/2786175025 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c52091d2b0 con 0x55c52091d480 2024-05-12T22:12:59.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.631+0000 7f17b1200640 1 --2- 172.21.15.28:0/2786175025 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c52091d480 0x55c52091ee00 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x55c52093aa40 tx=0x7f179803e040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=36145d6c0db389f server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:12:59.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.631+0000 7f179e000640 1 -- 172.21.15.28:0/2786175025 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1798035070 con 0x55c52091d480 2024-05-12T22:12:59.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.631+0000 7f179e000640 1 -- 172.21.15.28:0/2786175025 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1798031c60 con 0x55c52091d480 2024-05-12T22:12:59.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.631+0000 7f179e000640 1 -- 172.21.15.28:0/2786175025 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1798003930 con 0x55c52091d480 2024-05-12T22:12:59.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.632+0000 7f17b7432740 1 -- 172.21.15.28:0/2786175025 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c52091d480 msgr2=0x55c52091ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:12:59.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.632+0000 7f17b7432740 1 --2- 172.21.15.28:0/2786175025 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c52091d480 0x55c52091ee00 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x55c52093aa40 tx=0x7f179803e040 comp rx=0 tx=0).stop 2024-05-12T22:12:59.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.632+0000 7f17b7432740 1 -- 172.21.15.28:0/2786175025 shutdown_connections 2024-05-12T22:12:59.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.632+0000 7f17b7432740 1 --2- 172.21.15.28:0/2786175025 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c520927b50 0x55c52093faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:59.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.632+0000 7f17b7432740 1 --2- 172.21.15.28:0/2786175025 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c52091d480 0x55c52091ee00 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:59.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.632+0000 7f17b7432740 1 --2- 172.21.15.28:0/2786175025 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c52091cb60 0x55c520933800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:59.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.632+0000 7f17b7432740 1 -- 172.21.15.28:0/2786175025 >> 172.21.15.28:0/2786175025 conn(0x55c520918e80 msgr2=0x55c520918000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:12:59.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.632+0000 7f17b7432740 1 -- 172.21.15.28:0/2786175025 shutdown_connections 2024-05-12T22:12:59.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.632+0000 7f17b7432740 1 -- 172.21.15.28:0/2786175025 wait complete. 2024-05-12T22:12:59.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.632+0000 7f17b7432740 1 -- start start 2024-05-12T22:12:59.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.633+0000 7f17b7432740 1 Processor -- start 2024-05-12T22:12:59.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.633+0000 7f17b7432740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c52091cb60 0x55c52092ff00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:59.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.633+0000 7f17b7432740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c52091d480 0x55c520921310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:59.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.633+0000 7f17b7432740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c520927b50 0x55c520921850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:59.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.633+0000 7f17b7432740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f17ac1b1f40 con 0x55c52091d480 2024-05-12T22:12:59.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.633+0000 7f17b7432740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f17ac1b33b0 con 0x55c520927b50 2024-05-12T22:12:59.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.633+0000 7f17b7432740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f17ac1b2610 con 0x55c52091cb60 2024-05-12T22:12:59.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.633+0000 7f17b7432740 10 client.0 _ll_register_callbacks cb 0x7f17a0011440 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:12:59.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.633+0000 7f17b1200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c52091d480 0x55c520921310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:59.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.634+0000 7f17b1200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c52091d480 0x55c520921310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36934/0 (socket says 172.21.15.28:36934) 2024-05-12T22:12:59.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.634+0000 7f17b1200640 1 -- 172.21.15.28:0/2555642530 learned_addr learned my addr 172.21.15.28:0/2555642530 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:12:59.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.634+0000 7f17b2600640 1 --2- 172.21.15.28:0/2555642530 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c520927b50 0x55c520921850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:59.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.634+0000 7f17b1c00640 1 --2- 172.21.15.28:0/2555642530 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c52091cb60 0x55c52092ff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:59.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.634+0000 7f17b1200640 1 -- 172.21.15.28:0/2555642530 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c52091cb60 msgr2=0x55c52092ff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:12:59.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.634+0000 7f17b1200640 1 --2- 172.21.15.28:0/2555642530 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c52091cb60 0x55c52092ff00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:59.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.634+0000 7f17b1200640 1 -- 172.21.15.28:0/2555642530 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c520927b50 msgr2=0x55c520921850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:12:59.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.634+0000 7f17b1200640 1 --2- 172.21.15.28:0/2555642530 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c520927b50 0x55c520921850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:59.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.634+0000 7f17b1200640 1 -- 172.21.15.28:0/2555642530 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c520921e40 con 0x55c52091d480 2024-05-12T22:12:59.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.634+0000 7f17b1200640 1 --2- 172.21.15.28:0/2555642530 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c52091d480 0x55c520921310 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f1798000c00 tx=0x7f1798005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b3e2155eb0564ed server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:12:59.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.634+0000 7f1793400640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-12T22:12:59.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.635+0000 7f1793400640 1 -- 172.21.15.28:0/2555642530 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1798035050 con 0x55c52091d480 2024-05-12T22:12:59.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.635+0000 7f1793400640 1 -- 172.21.15.28:0/2555642530 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f1798053430 con 0x55c52091d480 2024-05-12T22:12:59.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.635+0000 7f17b7432740 1 -- 172.21.15.28:0/2555642530 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1798031c60 con 0x55c52091d480 2024-05-12T22:12:59.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.635+0000 7f1793400640 1 -- 172.21.15.28:0/2555642530 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f1798031e10 con 0x55c52091d480 2024-05-12T22:12:59.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.635+0000 7f17b7432740 10 client.16074 fetch_fsmap learned FSMap version 25 2024-05-12T22:12:59.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.635+0000 7f17b7432740 1 -- 172.21.15.28:0/2555642530 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55c52099e0f0 con 0x55c52091d480 2024-05-12T22:12:59.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.635+0000 7f1793400640 1 -- 172.21.15.28:0/2555642530 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1798031c60 con 0x55c52091d480 2024-05-12T22:12:59.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.635+0000 7f1793400640 1 -- 172.21.15.28:0/2555642530 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7f1798047040 con 0x55c52091d480 2024-05-12T22:12:59.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.636+0000 7f17b7432740 10 client.16074 fetch_fsmap finished waiting for FSMap version 25 2024-05-12T22:12:59.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.636+0000 7f17b7432740 10 client.16074 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:12:59.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.636+0000 7f17b7432740 20 client.16074 populate_metadata read hostname 'smithi028' 2024-05-12T22:12:59.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.636+0000 7f17b7432740 1 --2- 172.21.15.28:0/2555642530 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55c52099e890 0x55c5209a0cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:12:59.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.636+0000 7f17b7432740 4 client.16074 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-12T22:12:59.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.636+0000 7f17b7432740 1 -- 172.21.15.28:0/2555642530 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c5208894b0 con 0x55c52099e890 2024-05-12T22:12:59.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.636+0000 7f17b1c00640 1 --2- 172.21.15.28:0/2555642530 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55c52099e890 0x55c5209a0cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:12:59.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.636+0000 7f17b1c00640 1 --2- 172.21.15.28:0/2555642530 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55c52099e890 0x55c5209a0cb0 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d3d891368eec470b server_cookie=61e5ee99f5913188 in_seq=0 out_seq=0 2024-05-12T22:12:59.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.636+0000 7f1793400640 10 client.16074 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:12:59.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.637+0000 7f1793400640 1 -- 172.21.15.28:0/2555642530 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c5208894b0 con 0x55c52099e890 2024-05-12T22:12:59.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.637+0000 7f1793400640 10 client.16074 handle_command_reply: tid=0 2024-05-12T22:12:59.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.637+0000 7f17b7432740 1 client.16074 shutdown 2024-05-12T22:12:59.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.637+0000 7f17b7432740 1 -- 172.21.15.28:0/2555642530 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c52091d480 msgr2=0x55c520921310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:12:59.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.637+0000 7f17b7432740 1 --2- 172.21.15.28:0/2555642530 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c52091d480 0x55c520921310 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f1798000c00 tx=0x7f1798005850 comp rx=0 tx=0).stop 2024-05-12T22:12:59.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.637+0000 7f17b7432740 1 -- 172.21.15.28:0/2555642530 shutdown_connections 2024-05-12T22:12:59.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.637+0000 7f17b7432740 1 --2- 172.21.15.28:0/2555642530 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55c52099e890 0x55c5209a0cb0 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:59.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.637+0000 7f17b7432740 1 --2- 172.21.15.28:0/2555642530 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c520927b50 0x55c520921850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:59.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.637+0000 7f17b7432740 1 --2- 172.21.15.28:0/2555642530 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c52091d480 0x55c520921310 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:59.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.637+0000 7f17b7432740 1 --2- 172.21.15.28:0/2555642530 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c52091cb60 0x55c52092ff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:59.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.637+0000 7f1793400640 0 client.16074 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:12:59.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.637+0000 7f17b7432740 1 -- 172.21.15.28:0/2555642530 >> 172.21.15.28:0/2555642530 conn(0x55c520918e80 msgr2=0x55c520924a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:12:59.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.637+0000 7f17b7432740 1 -- 172.21.15.28:0/2555642530 shutdown_connections 2024-05-12T22:12:59.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.637+0000 7f17b7432740 1 -- 172.21.15.28:0/2555642530 wait complete. 2024-05-12T22:12:59.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.637+0000 7f17b7432740 20 client.16074 trim_cache size 0 max 16384 2024-05-12T22:12:59.645 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-05-12T22:12:59.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.638+0000 7f17b3000640 1 -- 172.21.15.28:0/1792030337 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f17940a2d80 msgr2=0x7f17940a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:12:59.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.638+0000 7f17b3000640 1 --2- 172.21.15.28:0/1792030337 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f17940a2d80 0x7f17940a5240 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f1798009230 tx=0x7f1798031400 comp rx=0 tx=0).stop 2024-05-12T22:12:59.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.638+0000 7f17b3000640 1 -- 172.21.15.28:0/1792030337 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f17ac112580 msgr2=0x7f17ac1a2ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:12:59.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.638+0000 7f17b3000640 1 --2- 172.21.15.28:0/1792030337 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f17ac112580 0x7f17ac1a2ef0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f17a4009c90 tx=0x7f17a400c6a0 comp rx=0 tx=0).stop 2024-05-12T22:12:59.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.638+0000 7f17b3000640 1 -- 172.21.15.28:0/1792030337 shutdown_connections 2024-05-12T22:12:59.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.638+0000 7f17b3000640 1 --2- 172.21.15.28:0/1792030337 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f17940a2d80 0x7f17940a5240 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:59.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.638+0000 7f17b3000640 1 --2- 172.21.15.28:0/1792030337 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f17ac112580 0x7f17ac1a2ef0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:59.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.638+0000 7f17b3000640 1 --2- 172.21.15.28:0/1792030337 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17ac076680 0x7f17ac19eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:59.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.638+0000 7f17b3000640 1 --2- 172.21.15.28:0/1792030337 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f17ac075d60 0x7f17ac19e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:12:59.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.638+0000 7f17b3000640 1 -- 172.21.15.28:0/1792030337 >> 172.21.15.28:0/1792030337 conn(0x7f17ac0ff370 msgr2=0x7f17ac101760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:12:59.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.638+0000 7f17b3000640 1 -- 172.21.15.28:0/1792030337 shutdown_connections 2024-05-12T22:12:59.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:12:59.638+0000 7f17b3000640 1 -- 172.21.15.28:0/1792030337 wait complete. 2024-05-12T22:12:59.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:12:59 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/3260465127' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:13:00.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:13:00 smithi086 ceph-mon[34297]: pgmap v123: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 3.6 KiB/s wr, 5 op/s 2024-05-12T22:13:00.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:13:00 smithi028 ceph-mon[29847]: pgmap v123: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 3.6 KiB/s wr, 5 op/s 2024-05-12T22:13:00.658 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-12T22:13:00.689 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-12T22:13:00.689 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-05-12T22:13:00.690 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:13:00 smithi186 ceph-mon[34785]: pgmap v123: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 3.6 KiB/s wr, 5 op/s 2024-05-12T22:13:00.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.937+0000 7f016e200640 1 -- 172.21.15.28:0/2099554677 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f01681020a0 msgr2=0x7f016810e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:13:00.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.937+0000 7f016e200640 1 --2- 172.21.15.28:0/2099554677 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f01681020a0 0x7f016810e5b0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f0160009ca0 tx=0x7f016002a8b0 comp rx=0 tx=0).stop 2024-05-12T22:13:00.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.938+0000 7f016e200640 1 -- 172.21.15.28:0/2099554677 shutdown_connections 2024-05-12T22:13:00.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.938+0000 7f016e200640 1 --2- 172.21.15.28:0/2099554677 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f01681020a0 0x7f016810e5b0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:00.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.938+0000 7f016e200640 1 --2- 172.21.15.28:0/2099554677 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01681016e0 0x7f0168101b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:00.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.938+0000 7f016e200640 1 --2- 172.21.15.28:0/2099554677 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f01681076f0 0x7f0168107ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:00.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.938+0000 7f016e200640 1 -- 172.21.15.28:0/2099554677 >> 172.21.15.28:0/2099554677 conn(0x7f01680fd1d0 msgr2=0x7f01680ff5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:13:00.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.938+0000 7f016e200640 1 -- 172.21.15.28:0/2099554677 shutdown_connections 2024-05-12T22:13:00.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.938+0000 7f016e200640 1 -- 172.21.15.28:0/2099554677 wait complete. 2024-05-12T22:13:00.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.939+0000 7f016e200640 1 Processor -- start 2024-05-12T22:13:00.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.939+0000 7f016e200640 1 -- start start 2024-05-12T22:13:00.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.940+0000 7f016e200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01681016e0 0x7f016819a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:00.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.940+0000 7f016e200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f01681020a0 0x7f016819a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:00.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.940+0000 7f016e200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f01681076f0 0x7f016819ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:00.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.940+0000 7f016e200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0168111b00 con 0x7f01681016e0 2024-05-12T22:13:00.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.940+0000 7f016e200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f0168111980 con 0x7f01681020a0 2024-05-12T22:13:00.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.940+0000 7f016e200640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f0168111c80 con 0x7f01681076f0 2024-05-12T22:13:00.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.940+0000 7f016ce00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01681016e0 0x7f016819a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:00.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.940+0000 7f0167e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f01681020a0 0x7f016819a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:00.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.940+0000 7f016ce00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01681016e0 0x7f016819a1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36942/0 (socket says 172.21.15.28:36942) 2024-05-12T22:13:00.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.940+0000 7f016ce00640 1 -- 172.21.15.28:0/1437661447 learned_addr learned my addr 172.21.15.28:0/1437661447 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:13:00.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.940+0000 7f016d800640 1 --2- 172.21.15.28:0/1437661447 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f01681076f0 0x7f016819ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:00.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.941+0000 7f0167e00640 1 -- 172.21.15.28:0/1437661447 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f01681076f0 msgr2=0x7f016819ea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:13:00.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.941+0000 7f0167e00640 1 --2- 172.21.15.28:0/1437661447 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f01681076f0 0x7f016819ea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:00.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.941+0000 7f0167e00640 1 -- 172.21.15.28:0/1437661447 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01681016e0 msgr2=0x7f016819a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:13:00.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.941+0000 7f0167e00640 1 --2- 172.21.15.28:0/1437661447 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01681016e0 0x7f016819a1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:00.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.941+0000 7f0167e00640 1 -- 172.21.15.28:0/1437661447 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f016819f1b0 con 0x7f01681020a0 2024-05-12T22:13:00.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.941+0000 7f0167e00640 1 --2- 172.21.15.28:0/1437661447 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f01681020a0 0x7f016819a700 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f015400d6d0 tx=0x7f015400db90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:13:00.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.941+0000 7f0165600640 1 -- 172.21.15.28:0/1437661447 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0154017070 con 0x7f01681020a0 2024-05-12T22:13:00.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.942+0000 7f0165600640 1 -- 172.21.15.28:0/1437661447 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0154009820 con 0x7f01681020a0 2024-05-12T22:13:00.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.942+0000 7f016e200640 1 -- 172.21.15.28:0/1437661447 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f016819f4a0 con 0x7f01681020a0 2024-05-12T22:13:00.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.942+0000 7f0165600640 1 -- 172.21.15.28:0/1437661447 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0154004d00 con 0x7f01681020a0 2024-05-12T22:13:00.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.942+0000 7f016e200640 1 -- 172.21.15.28:0/1437661447 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01681a6d30 con 0x7f01681020a0 2024-05-12T22:13:00.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.946+0000 7f01725a1740 1 Processor -- start 2024-05-12T22:13:00.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.946+0000 7f01725a1740 1 -- start start 2024-05-12T22:13:00.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.946+0000 7f01725a1740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564f855d3500 0x564f855d38e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:00.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.946+0000 7f0165600640 1 -- 172.21.15.28:0/1437661447 <== mon.2 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7f0154010460 con 0x7f01681020a0 2024-05-12T22:13:00.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.946+0000 7f01725a1740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564f855d3eb0 0x564f855ddf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:00.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.946+0000 7f01725a1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564f855de520 0x564f855e0970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:00.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.946+0000 7f01725a1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x564f855377c0 con 0x564f855de520 2024-05-12T22:13:00.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.946+0000 7f01725a1740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x564f854df790 con 0x564f855d3500 2024-05-12T22:13:00.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.946+0000 7f01725a1740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x564f85533780 con 0x564f855d3eb0 2024-05-12T22:13:00.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.947+0000 7f016d800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564f855d3500 0x564f855d38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:00.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.947+0000 7f016d800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564f855d3500 0x564f855d38e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:46356/0 (socket says 172.21.15.28:46356) 2024-05-12T22:13:00.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.947+0000 7f016d800640 1 -- 172.21.15.28:0/3841046177 learned_addr learned my addr 172.21.15.28:0/3841046177 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:13:00.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.947+0000 7f016ce00640 1 --2- 172.21.15.28:0/3841046177 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564f855d3eb0 0x564f855ddf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:00.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.947+0000 7f0167e00640 1 --2- 172.21.15.28:0/3841046177 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564f855de520 0x564f855e0970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:00.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.947+0000 7f016d800640 1 -- 172.21.15.28:0/3841046177 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564f855d3eb0 msgr2=0x564f855ddf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:13:00.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.947+0000 7f016d800640 1 --2- 172.21.15.28:0/3841046177 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564f855d3eb0 0x564f855ddf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:00.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.947+0000 7f016d800640 1 -- 172.21.15.28:0/3841046177 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564f855de520 msgr2=0x564f855e0970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:13:00.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.947+0000 7f016d800640 1 --2- 172.21.15.28:0/3841046177 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564f855de520 0x564f855e0970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:00.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.947+0000 7f016d800640 1 -- 172.21.15.28:0/3841046177 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564f854ff6a0 con 0x564f855d3500 2024-05-12T22:13:00.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.947+0000 7f0167e00640 1 --2- 172.21.15.28:0/3841046177 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564f855de520 0x564f855e0970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:13:00.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.948+0000 7f016d800640 1 --2- 172.21.15.28:0/3841046177 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564f855d3500 0x564f855d38e0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f0160009410 tx=0x7f0160005ef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3d5994d89310d3f5 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:13:00.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.948+0000 7f015a000640 1 -- 172.21.15.28:0/3841046177 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0160030620 con 0x564f855d3500 2024-05-12T22:13:00.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.948+0000 7f015a000640 1 -- 172.21.15.28:0/3841046177 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0160036030 con 0x564f855d3500 2024-05-12T22:13:00.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.948+0000 7f015a000640 1 -- 172.21.15.28:0/3841046177 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0160004440 con 0x564f855d3500 2024-05-12T22:13:00.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.948+0000 7f0165600640 1 --2- 172.21.15.28:0/1437661447 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f01500a2d80 0x7f01500a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:00.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.949+0000 7f0165600640 1 -- 172.21.15.28:0/1437661447 <== mon.2 v2:172.21.15.86:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f01540cec70 con 0x7f01681020a0 2024-05-12T22:13:00.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.949+0000 7f016ce00640 1 --2- 172.21.15.28:0/1437661447 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f01500a2d80 0x7f01500a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:00.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.949+0000 7f01725a1740 1 -- 172.21.15.28:0/3841046177 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564f855d3500 msgr2=0x564f855d38e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:13:00.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.949+0000 7f01725a1740 1 --2- 172.21.15.28:0/3841046177 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564f855d3500 0x564f855d38e0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f0160009410 tx=0x7f0160005ef0 comp rx=0 tx=0).stop 2024-05-12T22:13:00.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.950+0000 7f01725a1740 1 -- 172.21.15.28:0/3841046177 shutdown_connections 2024-05-12T22:13:00.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.950+0000 7f01725a1740 1 --2- 172.21.15.28:0/3841046177 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564f855de520 0x564f855e0970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:00.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.950+0000 7f01725a1740 1 --2- 172.21.15.28:0/3841046177 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564f855d3eb0 0x564f855ddf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:00.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.950+0000 7f01725a1740 1 --2- 172.21.15.28:0/3841046177 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564f855d3500 0x564f855d38e0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:00.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.950+0000 7f01725a1740 1 -- 172.21.15.28:0/3841046177 >> 172.21.15.28:0/3841046177 conn(0x564f855cf5b0 msgr2=0x564f855cf990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:13:00.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.950+0000 7f016ce00640 1 --2- 172.21.15.28:0/1437661447 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f01500a2d80 0x7f01500a5240 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x564f855d0830 tx=0x7f015c005f90 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:13:00.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.950+0000 7f01725a1740 1 -- 172.21.15.28:0/3841046177 shutdown_connections 2024-05-12T22:13:00.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.950+0000 7f01725a1740 1 -- 172.21.15.28:0/3841046177 wait complete. 2024-05-12T22:13:00.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.951+0000 7f01725a1740 1 -- start start 2024-05-12T22:13:00.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.951+0000 7f01725a1740 1 Processor -- start 2024-05-12T22:13:00.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.951+0000 7f01725a1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564f855d3500 0x564f855db450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:00.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.951+0000 7f01725a1740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564f855d3eb0 0x564f855db990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:00.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.951+0000 7f01725a1740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564f855de520 0x564f855dbed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:00.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.951+0000 7f01725a1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x564f855e14d0 con 0x564f855d3500 2024-05-12T22:13:00.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.951+0000 7f01725a1740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x564f854f2a90 con 0x564f855de520 2024-05-12T22:13:00.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.951+0000 7f01725a1740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x564f855e1650 con 0x564f855d3eb0 2024-05-12T22:13:00.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.952+0000 7f01725a1740 10 client.0 _ll_register_callbacks cb 0x564f8552daf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:13:00.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.952+0000 7f016ce00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564f855d3500 0x564f855db450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:00.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.952+0000 7f0167e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564f855d3eb0 0x564f855db990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:00.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.952+0000 7f016ce00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564f855d3500 0x564f855db450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36954/0 (socket says 172.21.15.28:36954) 2024-05-12T22:13:00.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.952+0000 7f0167e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564f855d3eb0 0x564f855db990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:36096/0 (socket says 172.21.15.28:36096) 2024-05-12T22:13:00.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.952+0000 7f016ce00640 1 -- 172.21.15.28:0/248387277 learned_addr learned my addr 172.21.15.28:0/248387277 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:13:00.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.952+0000 7f016d800640 1 --2- 172.21.15.28:0/248387277 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564f855de520 0x564f855dbed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:00.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.952+0000 7f0167e00640 1 -- 172.21.15.28:0/248387277 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564f855de520 msgr2=0x564f855dbed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:13:00.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.952+0000 7f0167e00640 1 --2- 172.21.15.28:0/248387277 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564f855de520 0x564f855dbed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:00.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.952+0000 7f0167e00640 1 -- 172.21.15.28:0/248387277 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564f855d3500 msgr2=0x564f855db450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:13:00.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.952+0000 7f0167e00640 1 --2- 172.21.15.28:0/248387277 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564f855d3500 0x564f855db450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:00.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.952+0000 7f0167e00640 1 -- 172.21.15.28:0/248387277 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564f853137e0 con 0x564f855d3eb0 2024-05-12T22:13:00.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.952+0000 7f016d800640 1 --2- 172.21.15.28:0/248387277 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564f855de520 0x564f855dbed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-12T22:13:00.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.953+0000 7f0167e00640 1 --2- 172.21.15.28:0/248387277 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564f855d3eb0 0x564f855db990 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f01540cc3d0 tx=0x7f015409b020 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f49eef100a4a17ae server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:13:00.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.953+0000 7f014ea00640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-12T22:13:00.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.953+0000 7f014ea00640 1 -- 172.21.15.28:0/248387277 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0154020070 con 0x564f855d3eb0 2024-05-12T22:13:00.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.953+0000 7f01725a1740 1 -- 172.21.15.28:0/248387277 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0160036030 con 0x564f855d3eb0 2024-05-12T22:13:00.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.953+0000 7f014ea00640 1 -- 172.21.15.28:0/248387277 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f01540099d0 con 0x564f855d3eb0 2024-05-12T22:13:00.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.953+0000 7f014ea00640 1 -- 172.21.15.28:0/248387277 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f015409b250 con 0x564f855d3eb0 2024-05-12T22:13:00.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.953+0000 7f01725a1740 10 client.26176 fetch_fsmap learned FSMap version 25 2024-05-12T22:13:00.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.953+0000 7f01725a1740 1 -- 172.21.15.28:0/248387277 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x564f85373f70 con 0x564f855d3eb0 2024-05-12T22:13:00.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.954+0000 7f014ea00640 1 -- 172.21.15.28:0/248387277 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0160036030 con 0x564f855d3eb0 2024-05-12T22:13:00.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.954+0000 7f014ea00640 1 -- 172.21.15.28:0/248387277 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7f01540bb120 con 0x564f855d3eb0 2024-05-12T22:13:00.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.954+0000 7f01725a1740 10 client.26176 fetch_fsmap finished waiting for FSMap version 25 2024-05-12T22:13:00.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.954+0000 7f01725a1740 10 client.26176 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:13:00.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.954+0000 7f01725a1740 20 client.26176 populate_metadata read hostname 'smithi028' 2024-05-12T22:13:00.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.954+0000 7f01725a1740 1 --2- 172.21.15.28:0/248387277 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x564f855ee6c0 0x564f855f0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:00.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.954+0000 7f01725a1740 4 client.26176 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-12T22:13:00.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.954+0000 7f01725a1740 1 -- 172.21.15.28:0/248387277 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564f8554b3a0 con 0x564f855ee6c0 2024-05-12T22:13:00.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.954+0000 7f016ce00640 1 --2- 172.21.15.28:0/248387277 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x564f855ee6c0 0x564f855f0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:00.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.955+0000 7f016ce00640 1 --2- 172.21.15.28:0/248387277 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x564f855ee6c0 0x564f855f0ab0 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=29b25dc153898adb server_cookie=13b5001ed1623bd4 in_seq=0 out_seq=0 2024-05-12T22:13:00.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.955+0000 7f014ea00640 10 client.26176 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:13:00.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.956+0000 7f014ea00640 1 -- 172.21.15.28:0/248387277 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564f8554b3a0 con 0x564f855ee6c0 2024-05-12T22:13:00.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.956+0000 7f014ea00640 10 client.26176 handle_command_reply: tid=0 2024-05-12T22:13:00.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.956+0000 7f01725a1740 1 client.26176 shutdown 2024-05-12T22:13:00.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.956+0000 7f01725a1740 1 -- 172.21.15.28:0/248387277 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564f855d3eb0 msgr2=0x564f855db990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:13:00.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.956+0000 7f01725a1740 1 --2- 172.21.15.28:0/248387277 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564f855d3eb0 0x564f855db990 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f01540cc3d0 tx=0x7f015409b020 comp rx=0 tx=0).stop 2024-05-12T22:13:00.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.956+0000 7f01725a1740 1 -- 172.21.15.28:0/248387277 shutdown_connections 2024-05-12T22:13:00.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.956+0000 7f01725a1740 1 --2- 172.21.15.28:0/248387277 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x564f855ee6c0 0x564f855f0ab0 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:00.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.956+0000 7f01725a1740 1 --2- 172.21.15.28:0/248387277 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564f855de520 0x564f855dbed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:00.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.956+0000 7f01725a1740 1 --2- 172.21.15.28:0/248387277 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564f855d3eb0 0x564f855db990 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:00.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.956+0000 7f014ea00640 0 client.26176 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:13:00.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.956+0000 7f01725a1740 1 --2- 172.21.15.28:0/248387277 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564f855d3500 0x564f855db450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:00.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.956+0000 7f01725a1740 1 -- 172.21.15.28:0/248387277 >> 172.21.15.28:0/248387277 conn(0x564f855cf5b0 msgr2=0x564f855d5630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:13:00.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.957+0000 7f01725a1740 1 -- 172.21.15.28:0/248387277 shutdown_connections 2024-05-12T22:13:00.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.957+0000 7f01725a1740 1 -- 172.21.15.28:0/248387277 wait complete. 2024-05-12T22:13:00.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.957+0000 7f01725a1740 20 client.26176 trim_cache size 0 max 16384 2024-05-12T22:13:00.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.981+0000 7f01725a1740 1 Processor -- start 2024-05-12T22:13:00.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.981+0000 7f01725a1740 1 -- start start 2024-05-12T22:13:00.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.981+0000 7f01725a1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564f855d3500 0x564f855ea3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:00.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.981+0000 7f01725a1740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564f855d3eb0 0x564f855f6310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:00.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.981+0000 7f01725a1740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564f855de520 0x564f855f6850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:00.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.982+0000 7f01725a1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f01681b1df0 con 0x564f855d3500 2024-05-12T22:13:00.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.982+0000 7f01725a1740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f01681af6c0 con 0x564f855de520 2024-05-12T22:13:00.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.982+0000 7f01725a1740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f01681ad510 con 0x564f855d3eb0 2024-05-12T22:13:00.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.982+0000 7f0167e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564f855d3eb0 0x564f855f6310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:00.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.982+0000 7f016ce00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564f855d3500 0x564f855ea3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:00.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.982+0000 7f0167e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564f855d3eb0 0x564f855f6310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:36110/0 (socket says 172.21.15.28:36110) 2024-05-12T22:13:00.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.982+0000 7f0167e00640 1 -- 172.21.15.28:0/3626467150 learned_addr learned my addr 172.21.15.28:0/3626467150 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:13:00.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.982+0000 7f016ce00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564f855d3500 0x564f855ea3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36956/0 (socket says 172.21.15.28:36956) 2024-05-12T22:13:00.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.982+0000 7f016d800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564f855de520 0x564f855f6850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:00.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.982+0000 7f0167e00640 1 -- 172.21.15.28:0/3626467150 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564f855de520 msgr2=0x564f855f6850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:13:00.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.982+0000 7f0167e00640 1 --2- 172.21.15.28:0/3626467150 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564f855de520 0x564f855f6850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:00.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.982+0000 7f0167e00640 1 -- 172.21.15.28:0/3626467150 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564f855d3500 msgr2=0x564f855ea3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:13:00.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.982+0000 7f0167e00640 1 --2- 172.21.15.28:0/3626467150 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564f855d3500 0x564f855ea3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:00.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.982+0000 7f0167e00640 1 -- 172.21.15.28:0/3626467150 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564f855d3c50 con 0x564f855d3eb0 2024-05-12T22:13:00.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.983+0000 7f0167e00640 1 --2- 172.21.15.28:0/3626467150 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564f855d3eb0 0x564f855f6310 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f0154009610 tx=0x7f01540cc720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5dd3f5a615fa9c79 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:13:00.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.983+0000 7f0159600640 1 -- 172.21.15.28:0/3626467150 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0154015070 con 0x564f855d3eb0 2024-05-12T22:13:00.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.983+0000 7f0159600640 1 -- 172.21.15.28:0/3626467150 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f015409f550 con 0x564f855d3eb0 2024-05-12T22:13:00.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.983+0000 7f0159600640 1 -- 172.21.15.28:0/3626467150 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0154020040 con 0x564f855d3eb0 2024-05-12T22:13:00.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.984+0000 7f01725a1740 1 -- 172.21.15.28:0/3626467150 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564f855d3eb0 msgr2=0x564f855f6310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:13:00.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.984+0000 7f01725a1740 1 --2- 172.21.15.28:0/3626467150 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564f855d3eb0 0x564f855f6310 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f0154009610 tx=0x7f01540cc720 comp rx=0 tx=0).stop 2024-05-12T22:13:00.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.984+0000 7f01725a1740 1 -- 172.21.15.28:0/3626467150 shutdown_connections 2024-05-12T22:13:00.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.984+0000 7f01725a1740 1 --2- 172.21.15.28:0/3626467150 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564f855de520 0x564f855f6850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:00.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.984+0000 7f01725a1740 1 --2- 172.21.15.28:0/3626467150 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564f855d3eb0 0x564f855f6310 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:00.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.984+0000 7f01725a1740 1 --2- 172.21.15.28:0/3626467150 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564f855d3500 0x564f855ea3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:00.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.984+0000 7f01725a1740 1 -- 172.21.15.28:0/3626467150 >> 172.21.15.28:0/3626467150 conn(0x564f855cf5b0 msgr2=0x564f855d9d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:13:00.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.984+0000 7f01725a1740 1 -- 172.21.15.28:0/3626467150 shutdown_connections 2024-05-12T22:13:00.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.984+0000 7f01725a1740 1 -- 172.21.15.28:0/3626467150 wait complete. 2024-05-12T22:13:00.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.985+0000 7f01725a1740 1 -- start start 2024-05-12T22:13:00.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.985+0000 7f01725a1740 1 Processor -- start 2024-05-12T22:13:00.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.985+0000 7f01725a1740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564f855d3500 0x564f855d8110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:00.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.985+0000 7f01725a1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564f855d3eb0 0x564f855d8650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:00.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.985+0000 7f01725a1740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564f855de520 0x564f855d8b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:00.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.985+0000 7f01725a1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f01681adb80 con 0x564f855d3eb0 2024-05-12T22:13:00.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.985+0000 7f01725a1740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f01681aeff0 con 0x564f855d3500 2024-05-12T22:13:00.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.985+0000 7f01725a1740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f01681ae250 con 0x564f855de520 2024-05-12T22:13:00.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.986+0000 7f01725a1740 10 client.0 _ll_register_callbacks cb 0x7f01540057f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:13:00.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.986+0000 7f016ce00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564f855d3500 0x564f855d8110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:00.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.986+0000 7f016ce00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564f855d3500 0x564f855d8110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:46368/0 (socket says 172.21.15.28:46368) 2024-05-12T22:13:00.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.986+0000 7f016ce00640 1 -- 172.21.15.28:0/3394145252 learned_addr learned my addr 172.21.15.28:0/3394145252 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:13:00.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.986+0000 7f0167e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564f855d3eb0 0x564f855d8650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:00.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.986+0000 7f016d800640 1 --2- 172.21.15.28:0/3394145252 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564f855de520 0x564f855d8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:00.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.986+0000 7f016ce00640 1 -- 172.21.15.28:0/3394145252 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564f855de520 msgr2=0x564f855d8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:13:00.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.986+0000 7f016ce00640 1 --2- 172.21.15.28:0/3394145252 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564f855de520 0x564f855d8b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:00.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.986+0000 7f016ce00640 1 -- 172.21.15.28:0/3394145252 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564f855d3eb0 msgr2=0x564f855d8650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:13:00.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.986+0000 7f016ce00640 1 --2- 172.21.15.28:0/3394145252 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564f855d3eb0 0x564f855d8650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:00.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.986+0000 7f016ce00640 1 -- 172.21.15.28:0/3394145252 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564f8520b170 con 0x564f855d3500 2024-05-12T22:13:00.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.987+0000 7f016ce00640 1 --2- 172.21.15.28:0/3394145252 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564f855d3500 0x564f855d8110 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x564f855f12b0 tx=0x7f015c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=de42f15900ddc333 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:13:00.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.987+0000 7f014ea00640 10 client.0 ms_handle_connect on v2:172.21.15.86:3300/0 2024-05-12T22:13:00.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.987+0000 7f014ea00640 1 -- 172.21.15.28:0/3394145252 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f015c005020 con 0x564f855d3500 2024-05-12T22:13:00.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.987+0000 7f01725a1740 1 -- 172.21.15.28:0/3394145252 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f015409f550 con 0x564f855d3500 2024-05-12T22:13:00.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.987+0000 7f014ea00640 1 -- 172.21.15.28:0/3394145252 <== mon.2 v2:172.21.15.86:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f015c0051c0 con 0x564f855d3500 2024-05-12T22:13:00.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.987+0000 7f014ea00640 1 -- 172.21.15.28:0/3394145252 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f015c03b050 con 0x564f855d3500 2024-05-12T22:13:00.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.987+0000 7f01725a1740 10 client.26222 fetch_fsmap learned FSMap version 25 2024-05-12T22:13:00.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.987+0000 7f01725a1740 1 -- 172.21.15.28:0/3394145252 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x564f85654d20 con 0x564f855d3500 2024-05-12T22:13:00.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.988+0000 7f014ea00640 1 -- 172.21.15.28:0/3394145252 <== mon.2 v2:172.21.15.86:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f015409f550 con 0x564f855d3500 2024-05-12T22:13:00.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.988+0000 7f014ea00640 1 -- 172.21.15.28:0/3394145252 <== mon.2 v2:172.21.15.86:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7f015c02b170 con 0x564f855d3500 2024-05-12T22:13:00.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.988+0000 7f01725a1740 10 client.26222 fetch_fsmap finished waiting for FSMap version 25 2024-05-12T22:13:00.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.988+0000 7f01725a1740 10 client.26222 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:13:00.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.988+0000 7f01725a1740 20 client.26222 populate_metadata read hostname 'smithi028' 2024-05-12T22:13:00.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.988+0000 7f01725a1740 1 --2- 172.21.15.28:0/3394145252 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x564f85655470 0x564f85657890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:00.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.988+0000 7f01725a1740 4 client.26222 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-12T22:13:00.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.988+0000 7f01725a1740 1 -- 172.21.15.28:0/3394145252 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x564f85544ef0 con 0x564f85655470 2024-05-12T22:13:00.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.988+0000 7f0167e00640 1 --2- 172.21.15.28:0/3394145252 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x564f85655470 0x564f85657890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:00.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.989+0000 7f0167e00640 1 --2- 172.21.15.28:0/3394145252 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x564f85655470 0x564f85657890 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9147ccab3c7f4ca4 server_cookie=85eacc29dc5a9567 in_seq=0 out_seq=0 2024-05-12T22:13:00.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.989+0000 7f014ea00640 10 client.26222 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:13:00.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.990+0000 7f014ea00640 1 -- 172.21.15.28:0/3394145252 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x564f85544ef0 con 0x564f85655470 2024-05-12T22:13:00.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.990+0000 7f014ea00640 10 client.26222 handle_command_reply: tid=0 2024-05-12T22:13:00.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.990+0000 7f01725a1740 1 client.26222 shutdown 2024-05-12T22:13:00.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.990+0000 7f01725a1740 1 -- 172.21.15.28:0/3394145252 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564f855d3500 msgr2=0x564f855d8110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:13:00.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.990+0000 7f01725a1740 1 --2- 172.21.15.28:0/3394145252 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564f855d3500 0x564f855d8110 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x564f855f12b0 tx=0x7f015c0046c0 comp rx=0 tx=0).stop 2024-05-12T22:13:00.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.991+0000 7f01725a1740 1 -- 172.21.15.28:0/3394145252 shutdown_connections 2024-05-12T22:13:00.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.991+0000 7f01725a1740 1 --2- 172.21.15.28:0/3394145252 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x564f85655470 0x564f85657890 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:00.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.991+0000 7f01725a1740 1 --2- 172.21.15.28:0/3394145252 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564f855de520 0x564f855d8b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:00.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.991+0000 7f01725a1740 1 --2- 172.21.15.28:0/3394145252 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564f855d3eb0 0x564f855d8650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:00.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.991+0000 7f014ea00640 0 client.26222 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:13:00.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.991+0000 7f01725a1740 1 --2- 172.21.15.28:0/3394145252 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564f855d3500 0x564f855d8110 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:00.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.991+0000 7f01725a1740 1 -- 172.21.15.28:0/3394145252 >> 172.21.15.28:0/3394145252 conn(0x564f855cf5b0 msgr2=0x564f855dd510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:13:00.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.991+0000 7f01725a1740 1 -- 172.21.15.28:0/3394145252 shutdown_connections 2024-05-12T22:13:00.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.991+0000 7f01725a1740 1 -- 172.21.15.28:0/3394145252 wait complete. 2024-05-12T22:13:00.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.991+0000 7f01725a1740 20 client.26222 trim_cache size 0 max 16384 2024-05-12T22:13:00.997 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-05-12T22:13:00.997 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-05-12T22:13:00.998 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "42daa7cf-1cc0-471d-86b2-e26d16f82476", 2024-05-12T22:13:00.998 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-05-12T22:13:00.998 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-05-12T22:13:00.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.992+0000 7f016e200640 1 -- 172.21.15.28:0/1437661447 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f01500a2d80 msgr2=0x7f01500a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:13:00.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.992+0000 7f016e200640 1 --2- 172.21.15.28:0/1437661447 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f01500a2d80 0x7f01500a5240 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x564f855d0830 tx=0x7f015c005f90 comp rx=0 tx=0).stop 2024-05-12T22:13:00.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.992+0000 7f016e200640 1 -- 172.21.15.28:0/1437661447 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f01681020a0 msgr2=0x7f016819a700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:13:00.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.992+0000 7f016e200640 1 --2- 172.21.15.28:0/1437661447 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f01681020a0 0x7f016819a700 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f015400d6d0 tx=0x7f015400db90 comp rx=0 tx=0).stop 2024-05-12T22:13:00.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.992+0000 7f016e200640 1 -- 172.21.15.28:0/1437661447 shutdown_connections 2024-05-12T22:13:00.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.992+0000 7f016e200640 1 --2- 172.21.15.28:0/1437661447 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f01500a2d80 0x7f01500a5240 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:00.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.992+0000 7f016e200640 1 --2- 172.21.15.28:0/1437661447 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f01681076f0 0x7f016819ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:00.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.992+0000 7f016e200640 1 --2- 172.21.15.28:0/1437661447 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f01681020a0 0x7f016819a700 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:00.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.992+0000 7f016e200640 1 --2- 172.21.15.28:0/1437661447 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01681016e0 0x7f016819a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:01.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.992+0000 7f016e200640 1 -- 172.21.15.28:0/1437661447 >> 172.21.15.28:0/1437661447 conn(0x7f01680fd1d0 msgr2=0x7f016810cf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:13:01.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.993+0000 7f016e200640 1 -- 172.21.15.28:0/1437661447 shutdown_connections 2024-05-12T22:13:01.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:00.993+0000 7f016e200640 1 -- 172.21.15.28:0/1437661447 wait complete. 2024-05-12T22:13:01.016 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-12T22:13:01.016 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-12T22:13:01.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.241+0000 7f83ebe00640 1 -- 172.21.15.28:0/2133764890 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f83ec074580 msgr2=0x7f83ec10d9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:13:01.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.241+0000 7f83ebe00640 1 --2- 172.21.15.28:0/2133764890 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f83ec074580 0x7f83ec10d9d0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f83d8009900 tx=0x7f83d802a480 comp rx=0 tx=0).stop 2024-05-12T22:13:01.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.242+0000 7f83ebe00640 1 -- 172.21.15.28:0/2133764890 shutdown_connections 2024-05-12T22:13:01.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.242+0000 7f83ebe00640 1 --2- 172.21.15.28:0/2133764890 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f83ec10df10 0x7f83ec110360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:01.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.242+0000 7f83ebe00640 1 --2- 172.21.15.28:0/2133764890 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f83ec074580 0x7f83ec10d9d0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:01.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.242+0000 7f83ebe00640 1 --2- 172.21.15.28:0/2133764890 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83ec073c60 0x7f83ec074040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:01.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.242+0000 7f83ebe00640 1 -- 172.21.15.28:0/2133764890 >> 172.21.15.28:0/2133764890 conn(0x7f83ec0fd200 msgr2=0x7f83ec0ff620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:13:01.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.242+0000 7f83ebe00640 1 -- 172.21.15.28:0/2133764890 shutdown_connections 2024-05-12T22:13:01.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.242+0000 7f83ebe00640 1 -- 172.21.15.28:0/2133764890 wait complete. 2024-05-12T22:13:01.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.243+0000 7f83ebe00640 1 Processor -- start 2024-05-12T22:13:01.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.243+0000 7f83ebe00640 1 -- start start 2024-05-12T22:13:01.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.243+0000 7f83ebe00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f83ec073c60 0x7f83ec19a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:01.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.243+0000 7f83ebe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83ec074580 0x7f83ec19a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:01.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.243+0000 7f83ebe00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f83ec10df10 0x7f83ec19ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:01.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.243+0000 7f83ebe00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f83ec113d00 con 0x7f83ec074580 2024-05-12T22:13:01.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.243+0000 7f83ebe00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f83ec113b80 con 0x7f83ec10df10 2024-05-12T22:13:01.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.243+0000 7f83ebe00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f83ec113e80 con 0x7f83ec073c60 2024-05-12T22:13:01.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.244+0000 7f83eaa00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f83ec073c60 0x7f83ec19a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:01.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.244+0000 7f83eaa00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f83ec073c60 0x7f83ec19a260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:36144/0 (socket says 172.21.15.28:36144) 2024-05-12T22:13:01.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.244+0000 7f83eaa00640 1 -- 172.21.15.28:0/1432042319 learned_addr learned my addr 172.21.15.28:0/1432042319 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:13:01.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.244+0000 7f83eb400640 1 --2- 172.21.15.28:0/1432042319 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f83ec10df10 0x7f83ec19ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:01.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.244+0000 7f83ea000640 1 --2- 172.21.15.28:0/1432042319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83ec074580 0x7f83ec19a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:01.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.244+0000 7f83eaa00640 1 -- 172.21.15.28:0/1432042319 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f83ec10df10 msgr2=0x7f83ec19ead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:13:01.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.244+0000 7f83eaa00640 1 --2- 172.21.15.28:0/1432042319 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f83ec10df10 0x7f83ec19ead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:01.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.244+0000 7f83eaa00640 1 -- 172.21.15.28:0/1432042319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83ec074580 msgr2=0x7f83ec19a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:13:01.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.244+0000 7f83eaa00640 1 --2- 172.21.15.28:0/1432042319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83ec074580 0x7f83ec19a7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:01.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.244+0000 7f83eaa00640 1 -- 172.21.15.28:0/1432042319 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83ec19f200 con 0x7f83ec073c60 2024-05-12T22:13:01.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.245+0000 7f83eaa00640 1 --2- 172.21.15.28:0/1432042319 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f83ec073c60 0x7f83ec19a260 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f83dc00ea60 tx=0x7f83dc00ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:13:01.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.245+0000 7f83f0a00640 1 -- 172.21.15.28:0/1432042319 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83dc00cde0 con 0x7f83ec073c60 2024-05-12T22:13:01.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.245+0000 7f83f0a00640 1 -- 172.21.15.28:0/1432042319 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f83dc0035f0 con 0x7f83ec073c60 2024-05-12T22:13:01.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.245+0000 7f83f0a00640 1 -- 172.21.15.28:0/1432042319 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83dc010670 con 0x7f83ec073c60 2024-05-12T22:13:01.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.245+0000 7f83ebe00640 1 -- 172.21.15.28:0/1432042319 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83ec19f4f0 con 0x7f83ec073c60 2024-05-12T22:13:01.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.245+0000 7f83ebe00640 1 -- 172.21.15.28:0/1432042319 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83ec1a6d80 con 0x7f83ec073c60 2024-05-12T22:13:01.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.249+0000 7f83f0a00640 1 -- 172.21.15.28:0/1432042319 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7f83dc0037a0 con 0x7f83ec073c60 2024-05-12T22:13:01.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.249+0000 7f83f4dce740 1 Processor -- start 2024-05-12T22:13:01.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.250+0000 7f83f4dce740 1 -- start start 2024-05-12T22:13:01.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.250+0000 7f83f4dce740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56556c86bdd0 0x56556c86c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:01.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.250+0000 7f83f4dce740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56556c86c6f0 0x56556c876880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:01.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.250+0000 7f83f4dce740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56556c876dc0 0x56556c8791b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:01.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.250+0000 7f83f4dce740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56556c8342a0 con 0x56556c876dc0 2024-05-12T22:13:01.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.250+0000 7f83f4dce740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x56556c7d0b90 con 0x56556c86c6f0 2024-05-12T22:13:01.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.250+0000 7f83f4dce740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x56556c7785b0 con 0x56556c86bdd0 2024-05-12T22:13:01.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.250+0000 7f83eaa00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56556c86c6f0 0x56556c876880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:01.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.250+0000 7f83eb400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56556c86bdd0 0x56556c86c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:01.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.250+0000 7f83ea000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56556c876dc0 0x56556c8791b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:01.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.250+0000 7f83eb400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56556c86bdd0 0x56556c86c1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:36156/0 (socket says 172.21.15.28:36156) 2024-05-12T22:13:01.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.250+0000 7f83eb400640 1 -- 172.21.15.28:0/2982728374 learned_addr learned my addr 172.21.15.28:0/2982728374 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:13:01.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.251+0000 7f83eb400640 1 -- 172.21.15.28:0/2982728374 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56556c86c6f0 msgr2=0x56556c876880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:13:01.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.251+0000 7f83eb400640 1 --2- 172.21.15.28:0/2982728374 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56556c86c6f0 0x56556c876880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:01.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.251+0000 7f83eb400640 1 -- 172.21.15.28:0/2982728374 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56556c876dc0 msgr2=0x56556c8791b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:13:01.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.251+0000 7f83eb400640 1 --2- 172.21.15.28:0/2982728374 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56556c876dc0 0x56556c8791b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:01.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.251+0000 7f83eb400640 1 -- 172.21.15.28:0/2982728374 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56556c5ae7d0 con 0x56556c86bdd0 2024-05-12T22:13:01.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.251+0000 7f83ea000640 1 --2- 172.21.15.28:0/2982728374 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56556c876dc0 0x56556c8791b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:13:01.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.251+0000 7f83f0a00640 1 --2- 172.21.15.28:0/1432042319 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f83d00a2d80 0x7f83d00a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:01.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.251+0000 7f83eb400640 1 --2- 172.21.15.28:0/2982728374 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56556c86bdd0 0x56556c86c1b0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f83e4009f70 tx=0x7f83e4010800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c8667cea5f8f7956 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:13:01.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.251+0000 7f83f0a00640 1 -- 172.21.15.28:0/1432042319 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f83dc0cf740 con 0x7f83ec073c60 2024-05-12T22:13:01.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.252+0000 7f83eaa00640 1 --2- 172.21.15.28:0/1432042319 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f83d00a2d80 0x7f83d00a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:01.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.252+0000 7f83d7e00640 1 -- 172.21.15.28:0/2982728374 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83e4016070 con 0x56556c86bdd0 2024-05-12T22:13:01.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.252+0000 7f83d7e00640 1 -- 172.21.15.28:0/2982728374 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f83e4004d20 con 0x56556c86bdd0 2024-05-12T22:13:01.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.252+0000 7f83d7e00640 1 -- 172.21.15.28:0/2982728374 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83e4012950 con 0x56556c86bdd0 2024-05-12T22:13:01.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.253+0000 7f83eaa00640 1 --2- 172.21.15.28:0/1432042319 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f83d00a2d80 0x7f83d00a5240 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f83dc0ccf70 tx=0x7f83dc003f30 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:13:01.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.253+0000 7f83f4dce740 1 -- 172.21.15.28:0/2982728374 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56556c86bdd0 msgr2=0x56556c86c1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:13:01.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.253+0000 7f83f4dce740 1 --2- 172.21.15.28:0/2982728374 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56556c86bdd0 0x56556c86c1b0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f83e4009f70 tx=0x7f83e4010800 comp rx=0 tx=0).stop 2024-05-12T22:13:01.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.253+0000 7f83f4dce740 1 -- 172.21.15.28:0/2982728374 shutdown_connections 2024-05-12T22:13:01.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.253+0000 7f83f4dce740 1 --2- 172.21.15.28:0/2982728374 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56556c876dc0 0x56556c8791b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:01.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.253+0000 7f83f4dce740 1 --2- 172.21.15.28:0/2982728374 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56556c86c6f0 0x56556c876880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:01.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.253+0000 7f83f4dce740 1 --2- 172.21.15.28:0/2982728374 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56556c86bdd0 0x56556c86c1b0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:01.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.253+0000 7f83f4dce740 1 -- 172.21.15.28:0/2982728374 >> 172.21.15.28:0/2982728374 conn(0x56556c8680f0 msgr2=0x56556c8684d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:13:01.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.253+0000 7f83f4dce740 1 -- 172.21.15.28:0/2982728374 shutdown_connections 2024-05-12T22:13:01.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.253+0000 7f83f4dce740 1 -- 172.21.15.28:0/2982728374 wait complete. 2024-05-12T22:13:01.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.254+0000 7f83f4dce740 1 -- start start 2024-05-12T22:13:01.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.254+0000 7f83f4dce740 1 Processor -- start 2024-05-12T22:13:01.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.255+0000 7f83f4dce740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56556c86bdd0 0x56556c876740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:01.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.255+0000 7f83f4dce740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56556c86c6f0 0x56556c873a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:01.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.255+0000 7f83f4dce740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56556c876dc0 0x56556c873fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:01.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.255+0000 7f83f4dce740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56556c78b9f0 con 0x56556c876dc0 2024-05-12T22:13:01.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.255+0000 7f83f4dce740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x56556c7ccb50 con 0x56556c86bdd0 2024-05-12T22:13:01.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.255+0000 7f83f4dce740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x56556c879bb0 con 0x56556c86c6f0 2024-05-12T22:13:01.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.255+0000 7f83f4dce740 10 client.0 _ll_register_callbacks cb 0x56556c4e3be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:13:01.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.255+0000 7f83eb400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56556c86bdd0 0x56556c876740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:01.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.255+0000 7f83eb400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56556c86bdd0 0x56556c876740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:46414/0 (socket says 172.21.15.28:46414) 2024-05-12T22:13:01.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.255+0000 7f83eb400640 1 -- 172.21.15.28:0/1551466179 learned_addr learned my addr 172.21.15.28:0/1551466179 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:13:01.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.255+0000 7f83ea000640 1 --2- 172.21.15.28:0/1551466179 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56556c86c6f0 0x56556c873a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:01.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.255+0000 7f83eb400640 1 --2- 172.21.15.28:0/1551466179 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56556c876dc0 0x56556c873fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:01.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.256+0000 7f83eb400640 1 -- 172.21.15.28:0/1551466179 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56556c86c6f0 msgr2=0x56556c873a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:13:01.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.256+0000 7f83eb400640 1 --2- 172.21.15.28:0/1551466179 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56556c86c6f0 0x56556c873a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:01.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.256+0000 7f83eb400640 1 -- 172.21.15.28:0/1551466179 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56556c876dc0 msgr2=0x56556c873fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:13:01.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.256+0000 7f83eb400640 1 --2- 172.21.15.28:0/1551466179 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56556c876dc0 0x56556c873fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:01.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.256+0000 7f83eb400640 1 -- 172.21.15.28:0/1551466179 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56556c4e43f0 con 0x56556c86bdd0 2024-05-12T22:13:01.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.256+0000 7f83eb400640 1 --2- 172.21.15.28:0/1551466179 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56556c86bdd0 0x56556c876740 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f83e4000f80 tx=0x7f83e40124f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=53fae8772b7d7423 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:13:01.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.256+0000 7f83d4c00640 10 client.0 ms_handle_connect on v2:172.21.15.86:3300/0 2024-05-12T22:13:01.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.256+0000 7f83d4c00640 1 -- 172.21.15.28:0/1551466179 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83e4016040 con 0x56556c86bdd0 2024-05-12T22:13:01.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.256+0000 7f83f4dce740 1 -- 172.21.15.28:0/1551466179 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f83e4004d20 con 0x56556c86bdd0 2024-05-12T22:13:01.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.256+0000 7f83d4c00640 1 -- 172.21.15.28:0/1551466179 <== mon.2 v2:172.21.15.86:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f83e4012d80 con 0x56556c86bdd0 2024-05-12T22:13:01.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.257+0000 7f83d4c00640 1 -- 172.21.15.28:0/1551466179 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f83e4003970 con 0x56556c86bdd0 2024-05-12T22:13:01.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.257+0000 7f83f4dce740 10 client.26234 fetch_fsmap learned FSMap version 25 2024-05-12T22:13:01.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.257+0000 7f83f4dce740 1 -- 172.21.15.28:0/1551466179 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x56556c4a40f0 con 0x56556c86bdd0 2024-05-12T22:13:01.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.257+0000 7f83d4c00640 1 -- 172.21.15.28:0/1551466179 <== mon.2 v2:172.21.15.86:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f83e4004d20 con 0x56556c86bdd0 2024-05-12T22:13:01.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.257+0000 7f83d4c00640 1 -- 172.21.15.28:0/1551466179 <== mon.2 v2:172.21.15.86:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7f83e4028040 con 0x56556c86bdd0 2024-05-12T22:13:01.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.257+0000 7f83f4dce740 10 client.26234 fetch_fsmap finished waiting for FSMap version 25 2024-05-12T22:13:01.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.257+0000 7f83f4dce740 10 client.26234 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:13:01.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.257+0000 7f83f4dce740 20 client.26234 populate_metadata read hostname 'smithi028' 2024-05-12T22:13:01.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.257+0000 7f83f4dce740 1 --2- 172.21.15.28:0/1551466179 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x56556c886810 0x56556c888c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:01.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.257+0000 7f83f4dce740 4 client.26234 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-12T22:13:01.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.257+0000 7f83f4dce740 1 -- 172.21.15.28:0/1551466179 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56556c7de2c0 con 0x56556c886810 2024-05-12T22:13:01.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.258+0000 7f83ea000640 1 --2- 172.21.15.28:0/1551466179 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x56556c886810 0x56556c888c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:01.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.259+0000 7f83ea000640 1 --2- 172.21.15.28:0/1551466179 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x56556c886810 0x56556c888c00 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4044f2285bd9c015 server_cookie=7e9091ca10668e1b in_seq=0 out_seq=0 2024-05-12T22:13:01.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.259+0000 7f83d4c00640 10 client.26234 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:13:01.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.260+0000 7f83d4c00640 1 -- 172.21.15.28:0/1551466179 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56556c7de2c0 con 0x56556c886810 2024-05-12T22:13:01.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.260+0000 7f83d4c00640 10 client.26234 handle_command_reply: tid=0 2024-05-12T22:13:01.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.260+0000 7f83f4dce740 1 client.26234 shutdown 2024-05-12T22:13:01.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.261+0000 7f83f4dce740 1 -- 172.21.15.28:0/1551466179 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56556c86bdd0 msgr2=0x56556c876740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:13:01.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.261+0000 7f83f4dce740 1 --2- 172.21.15.28:0/1551466179 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56556c86bdd0 0x56556c876740 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f83e4000f80 tx=0x7f83e40124f0 comp rx=0 tx=0).stop 2024-05-12T22:13:01.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.261+0000 7f83f4dce740 1 -- 172.21.15.28:0/1551466179 shutdown_connections 2024-05-12T22:13:01.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.261+0000 7f83f4dce740 1 --2- 172.21.15.28:0/1551466179 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x56556c886810 0x56556c888c00 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:01.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.261+0000 7f83f4dce740 1 --2- 172.21.15.28:0/1551466179 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56556c876dc0 0x56556c873fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:01.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.261+0000 7f83f4dce740 1 --2- 172.21.15.28:0/1551466179 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56556c86c6f0 0x56556c873a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:01.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.261+0000 7f83f4dce740 1 --2- 172.21.15.28:0/1551466179 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56556c86bdd0 0x56556c876740 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:01.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.261+0000 7f83f4dce740 1 -- 172.21.15.28:0/1551466179 >> 172.21.15.28:0/1551466179 conn(0x56556c8680f0 msgr2=0x56556c86def0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:13:01.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.261+0000 7f83d4c00640 0 client.26234 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:13:01.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.261+0000 7f83f4dce740 1 -- 172.21.15.28:0/1551466179 shutdown_connections 2024-05-12T22:13:01.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.261+0000 7f83f4dce740 1 -- 172.21.15.28:0/1551466179 wait complete. 2024-05-12T22:13:01.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.262+0000 7f83f4dce740 20 client.26234 trim_cache size 0 max 16384 2024-05-12T22:13:01.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.288+0000 7f83f4dce740 1 Processor -- start 2024-05-12T22:13:01.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.288+0000 7f83f4dce740 1 -- start start 2024-05-12T22:13:01.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.288+0000 7f83f4dce740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56556c86bdd0 0x56556c8828f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:01.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.288+0000 7f83f4dce740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56556c86c6f0 0x56556c876270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:01.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.288+0000 7f83f4dce740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56556c876dc0 0x56556c88e340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:01.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.288+0000 7f83f4dce740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f83ec1b1f10 con 0x56556c86c6f0 2024-05-12T22:13:01.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.288+0000 7f83f4dce740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f83ec1af7e0 con 0x56556c86bdd0 2024-05-12T22:13:01.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.288+0000 7f83f4dce740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f83ec1ad630 con 0x56556c876dc0 2024-05-12T22:13:01.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.289+0000 7f83ea000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56556c86c6f0 0x56556c876270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:01.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.289+0000 7f83eb400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56556c86bdd0 0x56556c8828f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:01.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.289+0000 7f83eb400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56556c86bdd0 0x56556c8828f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:46418/0 (socket says 172.21.15.28:46418) 2024-05-12T22:13:01.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.289+0000 7f83ea000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56556c86c6f0 0x56556c876270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37030/0 (socket says 172.21.15.28:37030) 2024-05-12T22:13:01.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.289+0000 7f83eb400640 1 -- 172.21.15.28:0/3273538529 learned_addr learned my addr 172.21.15.28:0/3273538529 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:13:01.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.289+0000 7f83eb400640 1 --2- 172.21.15.28:0/3273538529 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56556c876dc0 0x56556c88e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:01.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.289+0000 7f83ea000640 1 -- 172.21.15.28:0/3273538529 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56556c876dc0 msgr2=0x56556c88e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:13:01.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.289+0000 7f83ea000640 1 --2- 172.21.15.28:0/3273538529 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56556c876dc0 0x56556c88e340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:01.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.289+0000 7f83ea000640 1 -- 172.21.15.28:0/3273538529 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56556c86bdd0 msgr2=0x56556c8828f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:13:01.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.289+0000 7f83ea000640 1 --2- 172.21.15.28:0/3273538529 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56556c86bdd0 0x56556c8828f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:01.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.289+0000 7f83ea000640 1 -- 172.21.15.28:0/3273538529 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56556c871f80 con 0x56556c86c6f0 2024-05-12T22:13:01.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.289+0000 7f83eb400640 1 --2- 172.21.15.28:0/3273538529 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56556c86bdd0 0x56556c8828f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:13:01.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.289+0000 7f83ea000640 1 --2- 172.21.15.28:0/3273538529 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56556c86c6f0 0x56556c876270 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f83d802aa70 tx=0x7f83d8005b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d81f12b75f92bdb8 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:13:01.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.290+0000 7f83d7400640 1 -- 172.21.15.28:0/3273538529 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83d8035040 con 0x56556c86c6f0 2024-05-12T22:13:01.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.290+0000 7f83d7400640 1 -- 172.21.15.28:0/3273538529 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f83d8045860 con 0x56556c86c6f0 2024-05-12T22:13:01.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.290+0000 7f83d7400640 1 -- 172.21.15.28:0/3273538529 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83d8005460 con 0x56556c86c6f0 2024-05-12T22:13:01.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.291+0000 7f83f4dce740 1 -- 172.21.15.28:0/3273538529 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56556c86c6f0 msgr2=0x56556c876270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:13:01.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.291+0000 7f83f4dce740 1 --2- 172.21.15.28:0/3273538529 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56556c86c6f0 0x56556c876270 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f83d802aa70 tx=0x7f83d8005b50 comp rx=0 tx=0).stop 2024-05-12T22:13:01.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.291+0000 7f83f4dce740 1 -- 172.21.15.28:0/3273538529 shutdown_connections 2024-05-12T22:13:01.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.291+0000 7f83f4dce740 1 --2- 172.21.15.28:0/3273538529 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56556c876dc0 0x56556c88e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:01.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.291+0000 7f83f4dce740 1 --2- 172.21.15.28:0/3273538529 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56556c86c6f0 0x56556c876270 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:01.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.291+0000 7f83f4dce740 1 --2- 172.21.15.28:0/3273538529 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56556c86bdd0 0x56556c8828f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:01.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.291+0000 7f83f4dce740 1 -- 172.21.15.28:0/3273538529 >> 172.21.15.28:0/3273538529 conn(0x56556c8680f0 msgr2=0x56556c867330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:13:01.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.291+0000 7f83f4dce740 1 -- 172.21.15.28:0/3273538529 shutdown_connections 2024-05-12T22:13:01.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.291+0000 7f83f4dce740 1 -- 172.21.15.28:0/3273538529 wait complete. 2024-05-12T22:13:01.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.292+0000 7f83f4dce740 1 -- start start 2024-05-12T22:13:01.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.292+0000 7f83f4dce740 1 Processor -- start 2024-05-12T22:13:01.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.293+0000 7f83f4dce740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56556c86bdd0 0x56556c87f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:01.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.293+0000 7f83f4dce740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56556c86c6f0 0x56556c870fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:01.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.293+0000 7f83f4dce740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56556c876dc0 0x56556c8715d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:01.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.293+0000 7f83f4dce740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f83ec1adca0 con 0x56556c86bdd0 2024-05-12T22:13:01.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.293+0000 7f83f4dce740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f83ec1af110 con 0x56556c86c6f0 2024-05-12T22:13:01.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.293+0000 7f83f4dce740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f83ec1ae370 con 0x56556c876dc0 2024-05-12T22:13:01.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.293+0000 7f83f4dce740 10 client.0 _ll_register_callbacks cb 0x7f83dc005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:13:01.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.293+0000 7f83ea000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56556c86c6f0 0x56556c870fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:01.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.293+0000 7f83ea000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56556c86c6f0 0x56556c870fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:46432/0 (socket says 172.21.15.28:46432) 2024-05-12T22:13:01.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.293+0000 7f83ea000640 1 -- 172.21.15.28:0/2715319887 learned_addr learned my addr 172.21.15.28:0/2715319887 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:13:01.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.293+0000 7f83eb400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56556c86bdd0 0x56556c87f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:01.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.293+0000 7f83eb400640 1 --2- 172.21.15.28:0/2715319887 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56556c876dc0 0x56556c8715d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:01.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.293+0000 7f83ea000640 1 -- 172.21.15.28:0/2715319887 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56556c876dc0 msgr2=0x56556c8715d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:13:01.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.293+0000 7f83ea000640 1 --2- 172.21.15.28:0/2715319887 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56556c876dc0 0x56556c8715d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:01.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.293+0000 7f83ea000640 1 -- 172.21.15.28:0/2715319887 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56556c86bdd0 msgr2=0x56556c87f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:13:01.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.294+0000 7f83ea000640 1 --2- 172.21.15.28:0/2715319887 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56556c86bdd0 0x56556c87f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:01.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.294+0000 7f83ea000640 1 -- 172.21.15.28:0/2715319887 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56556c86fdd0 con 0x56556c86c6f0 2024-05-12T22:13:01.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.294+0000 7f83eb400640 1 --2- 172.21.15.28:0/2715319887 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56556c86bdd0 0x56556c87f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:13:01.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.294+0000 7f83ea000640 1 --2- 172.21.15.28:0/2715319887 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56556c86c6f0 0x56556c870fc0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f83d802a9a0 tx=0x7f83d8005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4bf88736a3094dc7 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:13:01.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.294+0000 7f83d4c00640 10 client.0 ms_handle_connect on v2:172.21.15.86:3300/0 2024-05-12T22:13:01.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.294+0000 7f83d4c00640 1 -- 172.21.15.28:0/2715319887 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83d8052070 con 0x56556c86c6f0 2024-05-12T22:13:01.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.294+0000 7f83d4c00640 1 -- 172.21.15.28:0/2715319887 <== mon.2 v2:172.21.15.86:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f83d8045380 con 0x56556c86c6f0 2024-05-12T22:13:01.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.294+0000 7f83f4dce740 1 -- 172.21.15.28:0/2715319887 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f83d8045860 con 0x56556c86c6f0 2024-05-12T22:13:01.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.295+0000 7f83d4c00640 1 -- 172.21.15.28:0/2715319887 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f83d805e050 con 0x56556c86c6f0 2024-05-12T22:13:01.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.295+0000 7f83f4dce740 10 client.26240 fetch_fsmap learned FSMap version 25 2024-05-12T22:13:01.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.295+0000 7f83f4dce740 1 -- 172.21.15.28:0/2715319887 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x56556c8ec020 con 0x56556c86c6f0 2024-05-12T22:13:01.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.295+0000 7f83d4c00640 1 -- 172.21.15.28:0/2715319887 <== mon.2 v2:172.21.15.86:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f83d8045860 con 0x56556c86c6f0 2024-05-12T22:13:01.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.295+0000 7f83d4c00640 1 -- 172.21.15.28:0/2715319887 <== mon.2 v2:172.21.15.86:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7f83d8004a60 con 0x56556c86c6f0 2024-05-12T22:13:01.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.295+0000 7f83f4dce740 10 client.26240 fetch_fsmap finished waiting for FSMap version 25 2024-05-12T22:13:01.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.295+0000 7f83f4dce740 10 client.26240 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:13:01.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.295+0000 7f83f4dce740 20 client.26240 populate_metadata read hostname 'smithi028' 2024-05-12T22:13:01.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.295+0000 7f83f4dce740 1 --2- 172.21.15.28:0/2715319887 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x56556c8ec750 0x56556c8eeb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:01.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.295+0000 7f83f4dce740 4 client.26240 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-12T22:13:01.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.295+0000 7f83f4dce740 1 -- 172.21.15.28:0/2715319887 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56556c7d8610 con 0x56556c8ec750 2024-05-12T22:13:01.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.296+0000 7f83eb400640 1 --2- 172.21.15.28:0/2715319887 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x56556c8ec750 0x56556c8eeb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:01.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.296+0000 7f83eb400640 1 --2- 172.21.15.28:0/2715319887 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x56556c8ec750 0x56556c8eeb70 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fc52ce724f98198e server_cookie=e3ae5db18a5ca57d in_seq=0 out_seq=0 2024-05-12T22:13:01.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.296+0000 7f83d4c00640 10 client.26240 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:13:01.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.297+0000 7f83d4c00640 1 -- 172.21.15.28:0/2715319887 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+268 (crc 0 0 0) 0x56556c7d8610 con 0x56556c8ec750 2024-05-12T22:13:01.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.297+0000 7f83d4c00640 10 client.26240 handle_command_reply: tid=0 2024-05-12T22:13:01.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.297+0000 7f83f4dce740 1 client.26240 shutdown 2024-05-12T22:13:01.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.297+0000 7f83f4dce740 1 -- 172.21.15.28:0/2715319887 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56556c86c6f0 msgr2=0x56556c870fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:13:01.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.297+0000 7f83f4dce740 1 --2- 172.21.15.28:0/2715319887 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56556c86c6f0 0x56556c870fc0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f83d802a9a0 tx=0x7f83d8005fd0 comp rx=0 tx=0).stop 2024-05-12T22:13:01.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.297+0000 7f83f4dce740 1 -- 172.21.15.28:0/2715319887 shutdown_connections 2024-05-12T22:13:01.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.297+0000 7f83f4dce740 1 --2- 172.21.15.28:0/2715319887 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x56556c8ec750 0x56556c8eeb70 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:01.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.297+0000 7f83f4dce740 1 --2- 172.21.15.28:0/2715319887 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56556c876dc0 0x56556c8715d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:01.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.297+0000 7f83f4dce740 1 --2- 172.21.15.28:0/2715319887 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56556c86c6f0 0x56556c870fc0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:01.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.297+0000 7f83f4dce740 1 --2- 172.21.15.28:0/2715319887 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56556c86bdd0 0x56556c87f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:01.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.297+0000 7f83f4dce740 1 -- 172.21.15.28:0/2715319887 >> 172.21.15.28:0/2715319887 conn(0x56556c8680f0 msgr2=0x56556c86d4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:13:01.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.297+0000 7f83d4c00640 0 client.26240 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:13:01.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.297+0000 7f83f4dce740 1 -- 172.21.15.28:0/2715319887 shutdown_connections 2024-05-12T22:13:01.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.298+0000 7f83f4dce740 1 -- 172.21.15.28:0/2715319887 wait complete. 2024-05-12T22:13:01.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.298+0000 7f83f4dce740 20 client.26240 trim_cache size 0 max 16384 2024-05-12T22:13:01.304 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-05-12T22:13:01.305 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "scrub active (1254 inodes in the stack)", 2024-05-12T22:13:01.305 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-05-12T22:13:01.305 INFO:teuthology.orchestra.run.smithi028.stdout: "42daa7cf-1cc0-471d-86b2-e26d16f82476": { 2024-05-12T22:13:01.305 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-05-12T22:13:01.305 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "42daa7cf-1cc0-471d-86b2-e26d16f82476", 2024-05-12T22:13:01.305 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-05-12T22:13:01.305 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:13:01.305 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:13:01.306 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-05-12T22:13:01.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.298+0000 7f83ebe00640 1 -- 172.21.15.28:0/1432042319 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f83d00a2d80 msgr2=0x7f83d00a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:13:01.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.298+0000 7f83ebe00640 1 --2- 172.21.15.28:0/1432042319 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f83d00a2d80 0x7f83d00a5240 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f83dc0ccf70 tx=0x7f83dc003f30 comp rx=0 tx=0).stop 2024-05-12T22:13:01.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.298+0000 7f83ebe00640 1 -- 172.21.15.28:0/1432042319 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f83ec073c60 msgr2=0x7f83ec19a260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:13:01.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.298+0000 7f83ebe00640 1 --2- 172.21.15.28:0/1432042319 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f83ec073c60 0x7f83ec19a260 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f83dc00ea60 tx=0x7f83dc00ef20 comp rx=0 tx=0).stop 2024-05-12T22:13:01.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.299+0000 7f83ebe00640 1 -- 172.21.15.28:0/1432042319 shutdown_connections 2024-05-12T22:13:01.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.299+0000 7f83ebe00640 1 --2- 172.21.15.28:0/1432042319 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f83d00a2d80 0x7f83d00a5240 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:01.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.299+0000 7f83ebe00640 1 --2- 172.21.15.28:0/1432042319 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f83ec10df10 0x7f83ec19ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:01.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.299+0000 7f83ebe00640 1 --2- 172.21.15.28:0/1432042319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83ec074580 0x7f83ec19a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:01.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.299+0000 7f83ebe00640 1 --2- 172.21.15.28:0/1432042319 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f83ec073c60 0x7f83ec19a260 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:01.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.299+0000 7f83ebe00640 1 -- 172.21.15.28:0/1432042319 >> 172.21.15.28:0/1432042319 conn(0x7f83ec0fd200 msgr2=0x7f83ec0fdfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:13:01.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.299+0000 7f83ebe00640 1 -- 172.21.15.28:0/1432042319 shutdown_connections 2024-05-12T22:13:01.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:01.299+0000 7f83ebe00640 1 -- 172.21.15.28:0/1432042319 wait complete. 2024-05-12T22:13:01.321 INFO:tasks.cephfs.filesystem:scrub status for tag:42daa7cf-1cc0-471d-86b2-e26d16f82476 - {'path': '/', 'tag': '42daa7cf-1cc0-471d-86b2-e26d16f82476', 'options': 'recursive,force'} 2024-05-12T22:13:02.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:13:02 smithi086 ceph-mon[34297]: pgmap v124: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 5.7 KiB/s wr, 8 op/s 2024-05-12T22:13:02.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:13:02 smithi086 ceph-mon[34297]: scrub queued for path: / 2024-05-12T22:13:02.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:13:02 smithi086 ceph-mon[34297]: scrub summary: idle+waiting paths [/] 2024-05-12T22:13:02.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:13:02 smithi086 ceph-mon[34297]: scrub summary: active paths [/] 2024-05-12T22:13:02.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:13:02 smithi086 ceph-mon[34297]: scrub summary: active paths [/] 2024-05-12T22:13:02.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:13:02 smithi086 ceph-mon[34297]: scrub summary: idle+waiting paths [/] 2024-05-12T22:13:02.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:13:02 smithi086 ceph-mon[34297]: scrub summary: active paths [/] 2024-05-12T22:13:02.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:13:02 smithi086 ceph-mon[34297]: scrub summary: idle+waiting paths [/] 2024-05-12T22:13:02.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:13:02 smithi086 ceph-mon[34297]: scrub summary: active paths [/] 2024-05-12T22:13:02.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:13:02 smithi086 ceph-mon[34297]: scrub summary: idle+waiting paths [/] 2024-05-12T22:13:02.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:13:02 smithi086 ceph-mon[34297]: scrub summary: idle+waiting paths [/] 2024-05-12T22:13:02.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:13:02 smithi028 ceph-mon[29847]: pgmap v124: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 5.7 KiB/s wr, 8 op/s 2024-05-12T22:13:02.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:13:02 smithi028 ceph-mon[29847]: scrub queued for path: / 2024-05-12T22:13:02.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:13:02 smithi028 ceph-mon[29847]: scrub summary: idle+waiting paths [/] 2024-05-12T22:13:02.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:13:02 smithi028 ceph-mon[29847]: scrub summary: active paths [/] 2024-05-12T22:13:02.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:13:02 smithi028 ceph-mon[29847]: scrub summary: active paths [/] 2024-05-12T22:13:02.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:13:02 smithi028 ceph-mon[29847]: scrub summary: idle+waiting paths [/] 2024-05-12T22:13:02.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:13:02 smithi028 ceph-mon[29847]: scrub summary: active paths [/] 2024-05-12T22:13:02.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:13:02 smithi028 ceph-mon[29847]: scrub summary: idle+waiting paths [/] 2024-05-12T22:13:02.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:13:02 smithi028 ceph-mon[29847]: scrub summary: active paths [/] 2024-05-12T22:13:02.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:13:02 smithi028 ceph-mon[29847]: scrub summary: idle+waiting paths [/] 2024-05-12T22:13:02.537 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:13:02 smithi028 ceph-mon[29847]: scrub summary: idle+waiting paths [/] 2024-05-12T22:13:02.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:13:02 smithi186 ceph-mon[34785]: pgmap v124: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 5.7 KiB/s wr, 8 op/s 2024-05-12T22:13:02.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:13:02 smithi186 ceph-mon[34785]: scrub queued for path: / 2024-05-12T22:13:02.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:13:02 smithi186 ceph-mon[34785]: scrub summary: idle+waiting paths [/] 2024-05-12T22:13:02.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:13:02 smithi186 ceph-mon[34785]: scrub summary: active paths [/] 2024-05-12T22:13:02.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:13:02 smithi186 ceph-mon[34785]: scrub summary: active paths [/] 2024-05-12T22:13:02.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:13:02 smithi186 ceph-mon[34785]: scrub summary: idle+waiting paths [/] 2024-05-12T22:13:02.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:13:02 smithi186 ceph-mon[34785]: scrub summary: active paths [/] 2024-05-12T22:13:02.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:13:02 smithi186 ceph-mon[34785]: scrub summary: idle+waiting paths [/] 2024-05-12T22:13:02.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:13:02 smithi186 ceph-mon[34785]: scrub summary: active paths [/] 2024-05-12T22:13:02.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:13:02 smithi186 ceph-mon[34785]: scrub summary: idle+waiting paths [/] 2024-05-12T22:13:02.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:13:02 smithi186 ceph-mon[34785]: scrub summary: idle+waiting paths [/] 2024-05-12T22:13:04.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:13:04 smithi086 ceph-mon[34297]: pgmap v125: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 5.7 KiB/s wr, 5 op/s 2024-05-12T22:13:04.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:13:04 smithi028 ceph-mon[29847]: pgmap v125: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 5.7 KiB/s wr, 5 op/s 2024-05-12T22:13:04.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:13:04 smithi186 ceph-mon[34785]: pgmap v125: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 5.7 KiB/s wr, 5 op/s 2024-05-12T22:13:06.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:13:06 smithi028 ceph-mon[29847]: pgmap v126: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 28 KiB/s wr, 23 op/s 2024-05-12T22:13:06.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:13:06 smithi186 ceph-mon[34785]: pgmap v126: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 28 KiB/s wr, 23 op/s 2024-05-12T22:13:06.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:13:06 smithi086 ceph-mon[34297]: pgmap v126: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 28 KiB/s wr, 23 op/s 2024-05-12T22:13:08.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:13:08 smithi028 ceph-mon[29847]: pgmap v127: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 20 op/s 2024-05-12T22:13:08.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:13:08 smithi028 ceph-mon[29847]: scrub summary: idle 2024-05-12T22:13:08.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:13:08 smithi028 ceph-mon[29847]: scrub summary: idle 2024-05-12T22:13:08.544 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:13:08 smithi086 ceph-mon[34297]: pgmap v127: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 20 op/s 2024-05-12T22:13:08.544 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:13:08 smithi086 ceph-mon[34297]: scrub summary: idle 2024-05-12T22:13:08.544 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:13:08 smithi086 ceph-mon[34297]: scrub summary: idle 2024-05-12T22:13:08.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:13:08 smithi186 ceph-mon[34785]: pgmap v127: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 20 op/s 2024-05-12T22:13:08.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:13:08 smithi186 ceph-mon[34785]: scrub summary: idle 2024-05-12T22:13:08.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:13:08 smithi186 ceph-mon[34785]: scrub summary: idle 2024-05-12T22:13:10.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:13:10 smithi028 ceph-mon[29847]: pgmap v128: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 20 op/s 2024-05-12T22:13:10.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:13:10 smithi186 ceph-mon[34785]: pgmap v128: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 20 op/s 2024-05-12T22:13:10.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:13:10 smithi086 ceph-mon[34297]: pgmap v128: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 20 op/s 2024-05-12T22:13:12.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:13:12 smithi028 ceph-mon[29847]: pgmap v129: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 26 KiB/s wr, 23 op/s 2024-05-12T22:13:12.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:13:12 smithi186 ceph-mon[34785]: pgmap v129: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 26 KiB/s wr, 23 op/s 2024-05-12T22:13:12.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:13:12 smithi086 ceph-mon[34297]: pgmap v129: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 26 KiB/s wr, 23 op/s 2024-05-12T22:13:13.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:13:13 smithi028 ceph-mon[29847]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:13:13.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:13:13 smithi186 ceph-mon[34785]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:13:13.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:13:13 smithi086 ceph-mon[34297]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:13:14.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:13:14 smithi028 ceph-mon[29847]: pgmap v130: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 20 op/s 2024-05-12T22:13:14.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:13:14 smithi186 ceph-mon[34785]: pgmap v130: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 20 op/s 2024-05-12T22:13:14.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:13:14 smithi086 ceph-mon[34297]: pgmap v130: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 20 op/s 2024-05-12T22:13:15.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:13:15 smithi028 ceph-mon[29847]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:13:15.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:13:15 smithi028 ceph-mon[29847]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:13:15.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:13:15 smithi028 ceph-mon[29847]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:13:15.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:13:15 smithi186 ceph-mon[34785]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:13:15.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:13:15 smithi186 ceph-mon[34785]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:13:15.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:13:15 smithi186 ceph-mon[34785]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:13:15.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:13:15 smithi086 ceph-mon[34297]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:13:15.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:13:15 smithi086 ceph-mon[34297]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:13:15.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:13:15 smithi086 ceph-mon[34297]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:13:16.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:13:16 smithi028 ceph-mon[29847]: pgmap v131: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 24 KiB/s wr, 22 op/s 2024-05-12T22:13:16.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:13:16 smithi186 ceph-mon[34785]: pgmap v131: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 24 KiB/s wr, 22 op/s 2024-05-12T22:13:16.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:13:16 smithi086 ceph-mon[34297]: pgmap v131: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 24 KiB/s wr, 22 op/s 2024-05-12T22:13:17.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:13:17 smithi186 ceph-mon[34785]: pgmap v132: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 2.0 KiB/s wr, 5 op/s 2024-05-12T22:13:17.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:13:17 smithi086 ceph-mon[34297]: pgmap v132: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 2.0 KiB/s wr, 5 op/s 2024-05-12T22:13:17.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:13:17 smithi028 ceph-mon[29847]: pgmap v132: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 2.0 KiB/s wr, 5 op/s 2024-05-12T22:13:20.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:13:19 smithi186 ceph-mon[34785]: pgmap v133: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 2.0 KiB/s wr, 5 op/s 2024-05-12T22:13:20.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:13:19 smithi086 ceph-mon[34297]: pgmap v133: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 2.0 KiB/s wr, 5 op/s 2024-05-12T22:13:20.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:13:19 smithi028 ceph-mon[29847]: pgmap v133: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 2.0 KiB/s wr, 5 op/s 2024-05-12T22:13:22.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:13:21 smithi186 ceph-mon[34785]: pgmap v134: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 2.0 KiB/s wr, 7 op/s 2024-05-12T22:13:22.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:13:21 smithi086 ceph-mon[34297]: pgmap v134: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 2.0 KiB/s wr, 7 op/s 2024-05-12T22:13:22.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:13:21 smithi028 ceph-mon[29847]: pgmap v134: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 2.0 KiB/s wr, 7 op/s 2024-05-12T22:13:24.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:13:23 smithi186 ceph-mon[34785]: pgmap v135: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:13:24.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:13:23 smithi186 ceph-mon[34785]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=g=up:active} 3 up:standby-replay 6 up:standby 2024-05-12T22:13:24.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:13:23 smithi086 ceph-mon[34297]: pgmap v135: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:13:24.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:13:23 smithi086 ceph-mon[34297]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=g=up:active} 3 up:standby-replay 6 up:standby 2024-05-12T22:13:24.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:13:23 smithi028 ceph-mon[29847]: pgmap v135: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:13:24.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:13:23 smithi028 ceph-mon[29847]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=g=up:active} 3 up:standby-replay 6 up:standby 2024-05-12T22:13:26.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:13:25 smithi186 ceph-mon[34785]: pgmap v136: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-12T22:13:26.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:13:25 smithi086 ceph-mon[34297]: pgmap v136: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-12T22:13:26.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:13:25 smithi028 ceph-mon[29847]: pgmap v136: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-12T22:13:28.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:13:27 smithi186 ceph-mon[34785]: pgmap v137: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:13:28.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:13:27 smithi086 ceph-mon[34297]: pgmap v137: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:13:28.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:13:27 smithi028 ceph-mon[29847]: pgmap v137: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:13:30.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:13:29 smithi186 ceph-mon[34785]: pgmap v138: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:13:30.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:13:29 smithi086 ceph-mon[34297]: pgmap v138: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:13:30.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:13:29 smithi028 ceph-mon[29847]: pgmap v138: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:13:31.322 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-12T22:13:31.338 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-12T22:13:31.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.567+0000 7fda75e00640 1 -- 172.21.15.28:0/1730061694 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fda70109a80 msgr2=0x7fda70109e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:13:31.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.567+0000 7fda75e00640 1 --2- 172.21.15.28:0/1730061694 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fda70109a80 0x7fda70109e60 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fda60009890 tx=0x7fda6002a360 comp rx=0 tx=0).stop 2024-05-12T22:13:31.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.567+0000 7fda75e00640 1 -- 172.21.15.28:0/1730061694 shutdown_connections 2024-05-12T22:13:31.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.567+0000 7fda75e00640 1 --2- 172.21.15.28:0/1730061694 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fda70104430 0x7fda70110940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:31.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.567+0000 7fda75e00640 1 --2- 172.21.15.28:0/1730061694 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda70103a70 0x7fda70103ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:31.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.567+0000 7fda75e00640 1 --2- 172.21.15.28:0/1730061694 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fda70109a80 0x7fda70109e60 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:31.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.567+0000 7fda75e00640 1 -- 172.21.15.28:0/1730061694 >> 172.21.15.28:0/1730061694 conn(0x7fda700ff5c0 msgr2=0x7fda701019e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:13:31.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.567+0000 7fda75e00640 1 -- 172.21.15.28:0/1730061694 shutdown_connections 2024-05-12T22:13:31.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.567+0000 7fda75e00640 1 -- 172.21.15.28:0/1730061694 wait complete. 2024-05-12T22:13:31.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.568+0000 7fda75e00640 1 Processor -- start 2024-05-12T22:13:31.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.569+0000 7fda75e00640 1 -- start start 2024-05-12T22:13:31.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.569+0000 7fda75e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda70103a70 0x7fda7019a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:31.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.569+0000 7fda75e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fda70104430 0x7fda7019a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:31.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.569+0000 7fda75e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fda70109a80 0x7fda7019ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:31.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.569+0000 7fda75e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fda70113e90 con 0x7fda70103a70 2024-05-12T22:13:31.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.569+0000 7fda75e00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fda70113d10 con 0x7fda70109a80 2024-05-12T22:13:31.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.569+0000 7fda75e00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fda70114010 con 0x7fda70104430 2024-05-12T22:13:31.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.569+0000 7fda6ea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda70103a70 0x7fda7019a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:31.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.570+0000 7fda6ea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda70103a70 0x7fda7019a3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50062/0 (socket says 172.21.15.28:50062) 2024-05-12T22:13:31.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.570+0000 7fda6f400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fda70109a80 0x7fda7019ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:31.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.570+0000 7fda6ea00640 1 -- 172.21.15.28:0/2959393856 learned_addr learned my addr 172.21.15.28:0/2959393856 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:13:31.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.570+0000 7fda6e000640 1 --2- 172.21.15.28:0/2959393856 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fda70104430 0x7fda7019a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:31.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.570+0000 7fda6f400640 1 -- 172.21.15.28:0/2959393856 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fda70104430 msgr2=0x7fda7019a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:13:31.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.570+0000 7fda6f400640 1 --2- 172.21.15.28:0/2959393856 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fda70104430 0x7fda7019a910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:31.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.570+0000 7fda6f400640 1 -- 172.21.15.28:0/2959393856 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda70103a70 msgr2=0x7fda7019a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:13:31.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.570+0000 7fda6f400640 1 --2- 172.21.15.28:0/2959393856 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda70103a70 0x7fda7019a3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:31.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.570+0000 7fda6f400640 1 -- 172.21.15.28:0/2959393856 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda7019f3c0 con 0x7fda70109a80 2024-05-12T22:13:31.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.570+0000 7fda6ea00640 1 --2- 172.21.15.28:0/2959393856 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda70103a70 0x7fda7019a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:13:31.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.570+0000 7fda6f400640 1 --2- 172.21.15.28:0/2959393856 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fda70109a80 0x7fda7019ec90 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fda68009fd0 tx=0x7fda6800ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:13:31.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.571+0000 7fda74a00640 1 -- 172.21.15.28:0/2959393856 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda6800ce20 con 0x7fda70109a80 2024-05-12T22:13:31.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.571+0000 7fda74a00640 1 -- 172.21.15.28:0/2959393856 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fda68005020 con 0x7fda70109a80 2024-05-12T22:13:31.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.571+0000 7fda74a00640 1 -- 172.21.15.28:0/2959393856 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda68010720 con 0x7fda70109a80 2024-05-12T22:13:31.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.571+0000 7fda75e00640 1 -- 172.21.15.28:0/2959393856 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda7019f6b0 con 0x7fda70109a80 2024-05-12T22:13:31.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.571+0000 7fda75e00640 1 -- 172.21.15.28:0/2959393856 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda701a6f40 con 0x7fda70109a80 2024-05-12T22:13:31.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.574+0000 7fda78bea740 1 Processor -- start 2024-05-12T22:13:31.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.575+0000 7fda74a00640 1 -- 172.21.15.28:0/2959393856 <== mon.2 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7fda68005540 con 0x7fda70109a80 2024-05-12T22:13:31.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.575+0000 7fda78bea740 1 -- start start 2024-05-12T22:13:31.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.575+0000 7fda78bea740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634bd314bc0 0x5634bd314fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:31.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.575+0000 7fda78bea740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5634bd3154e0 0x5634bd31f670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:31.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.575+0000 7fda78bea740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5634bd31fbb0 0x5634bd321fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:31.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.575+0000 7fda78bea740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5634bd2dd2a0 con 0x5634bd314bc0 2024-05-12T22:13:31.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.575+0000 7fda78bea740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x5634bd279b90 con 0x5634bd3154e0 2024-05-12T22:13:31.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.575+0000 7fda78bea740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5634bd2215b0 con 0x5634bd31fbb0 2024-05-12T22:13:31.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.575+0000 7fda6e000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634bd314bc0 0x5634bd314fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:31.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.576+0000 7fda6ea00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5634bd3154e0 0x5634bd31f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:31.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.576+0000 7fda6ea00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5634bd3154e0 0x5634bd31f670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:43740/0 (socket says 172.21.15.28:43740) 2024-05-12T22:13:31.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.576+0000 7fda6ea00640 1 -- 172.21.15.28:0/3078859861 learned_addr learned my addr 172.21.15.28:0/3078859861 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:13:31.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.576+0000 7fda6e000640 1 --2- 172.21.15.28:0/3078859861 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5634bd31fbb0 0x5634bd321fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:31.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.576+0000 7fda6ea00640 1 -- 172.21.15.28:0/3078859861 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5634bd31fbb0 msgr2=0x5634bd321fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:13:31.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.576+0000 7fda6ea00640 1 --2- 172.21.15.28:0/3078859861 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5634bd31fbb0 0x5634bd321fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:31.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.576+0000 7fda6ea00640 1 -- 172.21.15.28:0/3078859861 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634bd314bc0 msgr2=0x5634bd314fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:13:31.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.576+0000 7fda6ea00640 1 --2- 172.21.15.28:0/3078859861 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634bd314bc0 0x5634bd314fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:31.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.576+0000 7fda6ea00640 1 -- 172.21.15.28:0/3078859861 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5634bd0577d0 con 0x5634bd3154e0 2024-05-12T22:13:31.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.576+0000 7fda6e000640 1 --2- 172.21.15.28:0/3078859861 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634bd314bc0 0x5634bd314fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:13:31.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.576+0000 7fda74a00640 1 --2- 172.21.15.28:0/2959393856 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fda540a2d80 0x7fda540a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:31.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.576+0000 7fda6ea00640 1 --2- 172.21.15.28:0/3078859861 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5634bd3154e0 0x5634bd31f670 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fda60005e00 tx=0x7fda60009520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=30e4937c198bcb4b server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:13:31.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.577+0000 7fda74a00640 1 -- 172.21.15.28:0/2959393856 <== mon.2 v2:172.21.15.86:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7fda680cf940 con 0x7fda70109a80 2024-05-12T22:13:31.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.577+0000 7fda6e000640 1 --2- 172.21.15.28:0/2959393856 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fda540a2d80 0x7fda540a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:31.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.577+0000 7fda5fe00640 1 -- 172.21.15.28:0/3078859861 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda60037040 con 0x5634bd3154e0 2024-05-12T22:13:31.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.577+0000 7fda5fe00640 1 -- 172.21.15.28:0/3078859861 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fda60004230 con 0x5634bd3154e0 2024-05-12T22:13:31.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.577+0000 7fda5fe00640 1 -- 172.21.15.28:0/3078859861 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda60003d90 con 0x5634bd3154e0 2024-05-12T22:13:31.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.578+0000 7fda78bea740 1 -- 172.21.15.28:0/3078859861 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5634bd3154e0 msgr2=0x5634bd31f670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:13:31.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.578+0000 7fda6e000640 1 --2- 172.21.15.28:0/2959393856 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fda540a2d80 0x7fda540a5240 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fda58009fd0 tx=0x7fda58005a00 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:13:31.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.578+0000 7fda78bea740 1 --2- 172.21.15.28:0/3078859861 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5634bd3154e0 0x5634bd31f670 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fda60005e00 tx=0x7fda60009520 comp rx=0 tx=0).stop 2024-05-12T22:13:31.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.578+0000 7fda78bea740 1 -- 172.21.15.28:0/3078859861 shutdown_connections 2024-05-12T22:13:31.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.578+0000 7fda78bea740 1 --2- 172.21.15.28:0/3078859861 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5634bd31fbb0 0x5634bd321fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:31.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.578+0000 7fda78bea740 1 --2- 172.21.15.28:0/3078859861 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5634bd3154e0 0x5634bd31f670 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:31.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.578+0000 7fda78bea740 1 --2- 172.21.15.28:0/3078859861 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634bd314bc0 0x5634bd314fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:31.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.578+0000 7fda78bea740 1 -- 172.21.15.28:0/3078859861 >> 172.21.15.28:0/3078859861 conn(0x5634bd310ee0 msgr2=0x5634bd3112c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:13:31.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.578+0000 7fda78bea740 1 -- 172.21.15.28:0/3078859861 shutdown_connections 2024-05-12T22:13:31.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.578+0000 7fda78bea740 1 -- 172.21.15.28:0/3078859861 wait complete. 2024-05-12T22:13:31.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.578+0000 7fda78bea740 1 -- start start 2024-05-12T22:13:31.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.579+0000 7fda78bea740 1 Processor -- start 2024-05-12T22:13:31.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.579+0000 7fda78bea740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5634bd314bc0 0x5634bd31f420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:31.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.579+0000 7fda78bea740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5634bd3154e0 0x5634bd31c710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:31.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.579+0000 7fda78bea740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634bd31fbb0 0x5634bd31cc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:31.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.579+0000 7fda78bea740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5634bd2349f0 con 0x5634bd31fbb0 2024-05-12T22:13:31.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.579+0000 7fda78bea740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x5634bd275b50 con 0x5634bd314bc0 2024-05-12T22:13:31.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.579+0000 7fda78bea740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5634bd322940 con 0x5634bd3154e0 2024-05-12T22:13:31.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.579+0000 7fda78bea740 10 client.0 _ll_register_callbacks cb 0x5634bcf8cbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:13:31.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.580+0000 7fda6ea00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5634bd314bc0 0x5634bd31f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:31.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.580+0000 7fda6f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634bd31fbb0 0x5634bd31cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:31.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.580+0000 7fda6ea00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5634bd314bc0 0x5634bd31f420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:43752/0 (socket says 172.21.15.28:43752) 2024-05-12T22:13:31.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.580+0000 7fda6ea00640 1 -- 172.21.15.28:0/3764645296 learned_addr learned my addr 172.21.15.28:0/3764645296 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:13:31.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.580+0000 7fda6e000640 1 --2- 172.21.15.28:0/3764645296 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5634bd3154e0 0x5634bd31c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:31.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.580+0000 7fda6ea00640 1 -- 172.21.15.28:0/3764645296 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5634bd3154e0 msgr2=0x5634bd31c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:13:31.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.580+0000 7fda6ea00640 1 --2- 172.21.15.28:0/3764645296 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5634bd3154e0 0x5634bd31c710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:31.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.580+0000 7fda6ea00640 1 -- 172.21.15.28:0/3764645296 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634bd31fbb0 msgr2=0x5634bd31cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:13:31.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.580+0000 7fda6ea00640 1 --2- 172.21.15.28:0/3764645296 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634bd31fbb0 0x5634bd31cc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:31.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.580+0000 7fda6ea00640 1 -- 172.21.15.28:0/3764645296 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5634bcf8d3f0 con 0x5634bd314bc0 2024-05-12T22:13:31.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.580+0000 7fda6f400640 1 --2- 172.21.15.28:0/3764645296 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634bd31fbb0 0x5634bd31cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:13:31.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.580+0000 7fda6ea00640 1 --2- 172.21.15.28:0/3764645296 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5634bd314bc0 0x5634bd31f420 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fda60005d90 tx=0x7fda60003990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3ce4a33a99a12f1 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:13:31.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.580+0000 7fda5cc00640 10 client.0 ms_handle_connect on v2:172.21.15.86:3300/0 2024-05-12T22:13:31.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.581+0000 7fda5cc00640 1 -- 172.21.15.28:0/3764645296 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda60037040 con 0x5634bd314bc0 2024-05-12T22:13:31.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.581+0000 7fda5cc00640 1 -- 172.21.15.28:0/3764645296 <== mon.2 v2:172.21.15.86:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7fda60003d10 con 0x5634bd314bc0 2024-05-12T22:13:31.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.581+0000 7fda78bea740 1 -- 172.21.15.28:0/3764645296 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fda60004230 con 0x5634bd314bc0 2024-05-12T22:13:31.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.581+0000 7fda5cc00640 1 -- 172.21.15.28:0/3764645296 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fda60003f10 con 0x5634bd314bc0 2024-05-12T22:13:31.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.581+0000 7fda78bea740 10 client.26252 fetch_fsmap learned FSMap version 26 2024-05-12T22:13:31.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.581+0000 7fda78bea740 1 -- 172.21.15.28:0/3764645296 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5634bcf4d0f0 con 0x5634bd314bc0 2024-05-12T22:13:31.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.581+0000 7fda5cc00640 1 -- 172.21.15.28:0/3764645296 <== mon.2 v2:172.21.15.86:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fda60004230 con 0x5634bd314bc0 2024-05-12T22:13:31.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.582+0000 7fda5cc00640 1 -- 172.21.15.28:0/3764645296 <== mon.2 v2:172.21.15.86:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fda60057450 con 0x5634bd314bc0 2024-05-12T22:13:31.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.582+0000 7fda78bea740 10 client.26252 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:13:31.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.582+0000 7fda78bea740 10 client.26252 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:13:31.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.582+0000 7fda78bea740 20 client.26252 populate_metadata read hostname 'smithi028' 2024-05-12T22:13:31.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.582+0000 7fda78bea740 1 --2- 172.21.15.28:0/3764645296 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5634bd32f450 0x5634bd331840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:31.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.582+0000 7fda78bea740 4 client.26252 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-12T22:13:31.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.582+0000 7fda78bea740 1 -- 172.21.15.28:0/3764645296 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5634bd2872c0 con 0x5634bd32f450 2024-05-12T22:13:31.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.582+0000 7fda6e000640 1 --2- 172.21.15.28:0/3764645296 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5634bd32f450 0x5634bd331840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:31.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.583+0000 7fda6e000640 1 --2- 172.21.15.28:0/3764645296 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5634bd32f450 0x5634bd331840 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=764e1460270106fe server_cookie=c8dd78dbe4e0e21f in_seq=0 out_seq=0 2024-05-12T22:13:31.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.583+0000 7fda5cc00640 10 client.26252 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:13:31.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.584+0000 7fda5cc00640 1 -- 172.21.15.28:0/3764645296 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5634bd2872c0 con 0x5634bd32f450 2024-05-12T22:13:31.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.584+0000 7fda5cc00640 10 client.26252 handle_command_reply: tid=0 2024-05-12T22:13:31.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.584+0000 7fda78bea740 1 client.26252 shutdown 2024-05-12T22:13:31.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.584+0000 7fda78bea740 1 -- 172.21.15.28:0/3764645296 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5634bd314bc0 msgr2=0x5634bd31f420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:13:31.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.584+0000 7fda78bea740 1 --2- 172.21.15.28:0/3764645296 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5634bd314bc0 0x5634bd31f420 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fda60005d90 tx=0x7fda60003990 comp rx=0 tx=0).stop 2024-05-12T22:13:31.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.584+0000 7fda78bea740 1 -- 172.21.15.28:0/3764645296 shutdown_connections 2024-05-12T22:13:31.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.584+0000 7fda78bea740 1 --2- 172.21.15.28:0/3764645296 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5634bd32f450 0x5634bd331840 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:31.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.584+0000 7fda78bea740 1 --2- 172.21.15.28:0/3764645296 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634bd31fbb0 0x5634bd31cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:31.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.584+0000 7fda78bea740 1 --2- 172.21.15.28:0/3764645296 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5634bd3154e0 0x5634bd31c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:31.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.584+0000 7fda78bea740 1 --2- 172.21.15.28:0/3764645296 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5634bd314bc0 0x5634bd31f420 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:31.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.584+0000 7fda78bea740 1 -- 172.21.15.28:0/3764645296 >> 172.21.15.28:0/3764645296 conn(0x5634bd310ee0 msgr2=0x5634bd3100e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:13:31.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.584+0000 7fda5cc00640 0 client.26252 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:13:31.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.584+0000 7fda78bea740 1 -- 172.21.15.28:0/3764645296 shutdown_connections 2024-05-12T22:13:31.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.584+0000 7fda78bea740 1 -- 172.21.15.28:0/3764645296 wait complete. 2024-05-12T22:13:31.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.585+0000 7fda78bea740 20 client.26252 trim_cache size 0 max 16384 2024-05-12T22:13:31.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.604+0000 7fda78bea740 1 Processor -- start 2024-05-12T22:13:31.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.604+0000 7fda78bea740 1 -- start start 2024-05-12T22:13:31.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.604+0000 7fda78bea740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5634bd314bc0 0x5634bd31c030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:31.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.604+0000 7fda78bea740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634bd3154e0 0x5634bd31ef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:31.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.604+0000 7fda78bea740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5634bd31fbb0 0x5634bd334860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:31.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.604+0000 7fda78bea740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fda701b2020 con 0x5634bd3154e0 2024-05-12T22:13:31.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.604+0000 7fda78bea740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fda701af8f0 con 0x5634bd314bc0 2024-05-12T22:13:31.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.604+0000 7fda78bea740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fda701ad740 con 0x5634bd31fbb0 2024-05-12T22:13:31.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.605+0000 7fda6ea00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5634bd314bc0 0x5634bd31c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:31.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.605+0000 7fda6e000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634bd3154e0 0x5634bd31ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:31.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.605+0000 7fda6ea00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5634bd314bc0 0x5634bd31c030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:43768/0 (socket says 172.21.15.28:43768) 2024-05-12T22:13:31.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.605+0000 7fda6ea00640 1 -- 172.21.15.28:0/3317559408 learned_addr learned my addr 172.21.15.28:0/3317559408 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:13:31.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.605+0000 7fda6f400640 1 --2- 172.21.15.28:0/3317559408 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5634bd31fbb0 0x5634bd334860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:31.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.605+0000 7fda6ea00640 1 -- 172.21.15.28:0/3317559408 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5634bd31fbb0 msgr2=0x5634bd334860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:13:31.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.605+0000 7fda6ea00640 1 --2- 172.21.15.28:0/3317559408 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5634bd31fbb0 0x5634bd334860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:31.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.605+0000 7fda6ea00640 1 -- 172.21.15.28:0/3317559408 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634bd3154e0 msgr2=0x5634bd31ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:13:31.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.605+0000 7fda6ea00640 1 --2- 172.21.15.28:0/3317559408 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634bd3154e0 0x5634bd31ef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:31.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.605+0000 7fda6ea00640 1 -- 172.21.15.28:0/3317559408 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5634bd315310 con 0x5634bd314bc0 2024-05-12T22:13:31.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.605+0000 7fda6f400640 1 --2- 172.21.15.28:0/3317559408 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5634bd31fbb0 0x5634bd334860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:13:31.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.606+0000 7fda6ea00640 1 --2- 172.21.15.28:0/3317559408 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5634bd314bc0 0x5634bd31c030 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fda60000c00 tx=0x7fda600035f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d267950153343f3f server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:13:31.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.606+0000 7fda5f400640 1 -- 172.21.15.28:0/3317559408 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda6005c070 con 0x5634bd314bc0 2024-05-12T22:13:31.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.606+0000 7fda5f400640 1 -- 172.21.15.28:0/3317559408 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fda60049070 con 0x5634bd314bc0 2024-05-12T22:13:31.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.606+0000 7fda5f400640 1 -- 172.21.15.28:0/3317559408 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda60037070 con 0x5634bd314bc0 2024-05-12T22:13:31.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.606+0000 7fda78bea740 1 -- 172.21.15.28:0/3317559408 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5634bd314bc0 msgr2=0x5634bd31c030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:13:31.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.606+0000 7fda78bea740 1 --2- 172.21.15.28:0/3317559408 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5634bd314bc0 0x5634bd31c030 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fda60000c00 tx=0x7fda600035f0 comp rx=0 tx=0).stop 2024-05-12T22:13:31.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.606+0000 7fda78bea740 1 -- 172.21.15.28:0/3317559408 shutdown_connections 2024-05-12T22:13:31.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.606+0000 7fda78bea740 1 --2- 172.21.15.28:0/3317559408 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5634bd31fbb0 0x5634bd334860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:31.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.606+0000 7fda78bea740 1 --2- 172.21.15.28:0/3317559408 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634bd3154e0 0x5634bd31ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:31.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.606+0000 7fda78bea740 1 --2- 172.21.15.28:0/3317559408 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5634bd314bc0 0x5634bd31c030 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:31.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.606+0000 7fda78bea740 1 -- 172.21.15.28:0/3317559408 >> 172.21.15.28:0/3317559408 conn(0x5634bd310ee0 msgr2=0x5634bd3100e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:13:31.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.607+0000 7fda78bea740 1 -- 172.21.15.28:0/3317559408 shutdown_connections 2024-05-12T22:13:31.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.607+0000 7fda78bea740 1 -- 172.21.15.28:0/3317559408 wait complete. 2024-05-12T22:13:31.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.607+0000 7fda78bea740 1 -- start start 2024-05-12T22:13:31.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.607+0000 7fda78bea740 1 Processor -- start 2024-05-12T22:13:31.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.608+0000 7fda78bea740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5634bd314bc0 0x5634bd331a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:31.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.608+0000 7fda78bea740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634bd3154e0 0x5634bd331f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:31.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.608+0000 7fda78bea740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5634bd31fbb0 0x5634bd393620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:31.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.608+0000 7fda78bea740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fda701addb0 con 0x5634bd3154e0 2024-05-12T22:13:31.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.608+0000 7fda78bea740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fda701af220 con 0x5634bd31fbb0 2024-05-12T22:13:31.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.608+0000 7fda78bea740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fda701ae480 con 0x5634bd314bc0 2024-05-12T22:13:31.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.608+0000 7fda78bea740 10 client.0 _ll_register_callbacks cb 0x7fda58009640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:13:31.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.608+0000 7fda6e000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634bd3154e0 0x5634bd331f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:31.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.608+0000 7fda6ea00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5634bd314bc0 0x5634bd331a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:31.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.608+0000 7fda6e000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634bd3154e0 0x5634bd331f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50098/0 (socket says 172.21.15.28:50098) 2024-05-12T22:13:31.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.608+0000 7fda6e000640 1 -- 172.21.15.28:0/486696627 learned_addr learned my addr 172.21.15.28:0/486696627 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:13:31.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.608+0000 7fda6f400640 1 --2- 172.21.15.28:0/486696627 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5634bd31fbb0 0x5634bd393620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:31.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.608+0000 7fda6ea00640 1 -- 172.21.15.28:0/486696627 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5634bd31fbb0 msgr2=0x5634bd393620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:13:31.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.608+0000 7fda6ea00640 1 --2- 172.21.15.28:0/486696627 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5634bd31fbb0 0x5634bd393620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:31.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.608+0000 7fda6ea00640 1 -- 172.21.15.28:0/486696627 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634bd3154e0 msgr2=0x5634bd331f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:13:31.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.608+0000 7fda6ea00640 1 --2- 172.21.15.28:0/486696627 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634bd3154e0 0x5634bd331f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:31.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.608+0000 7fda6ea00640 1 -- 172.21.15.28:0/486696627 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5634bd393b60 con 0x5634bd314bc0 2024-05-12T22:13:31.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.608+0000 7fda6e000640 1 --2- 172.21.15.28:0/486696627 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634bd3154e0 0x5634bd331f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:13:31.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.609+0000 7fda6ea00640 1 --2- 172.21.15.28:0/486696627 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5634bd314bc0 0x5634bd331a30 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fda60009730 tx=0x7fda60005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=780ce3264bf82004 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:13:31.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.609+0000 7fda5cc00640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-12T22:13:31.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.609+0000 7fda5cc00640 1 -- 172.21.15.28:0/486696627 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda6005c020 con 0x5634bd314bc0 2024-05-12T22:13:31.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.609+0000 7fda5cc00640 1 -- 172.21.15.28:0/486696627 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7fda60057940 con 0x5634bd314bc0 2024-05-12T22:13:31.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.609+0000 7fda78bea740 1 -- 172.21.15.28:0/486696627 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fda60049070 con 0x5634bd314bc0 2024-05-12T22:13:31.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.609+0000 7fda5cc00640 1 -- 172.21.15.28:0/486696627 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fda60037070 con 0x5634bd314bc0 2024-05-12T22:13:31.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.609+0000 7fda78bea740 10 client.26218 fetch_fsmap learned FSMap version 26 2024-05-12T22:13:31.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.609+0000 7fda78bea740 1 -- 172.21.15.28:0/486696627 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5634bd394e70 con 0x5634bd314bc0 2024-05-12T22:13:31.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.610+0000 7fda5cc00640 1 -- 172.21.15.28:0/486696627 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fda60049070 con 0x5634bd314bc0 2024-05-12T22:13:31.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.610+0000 7fda5cc00640 1 -- 172.21.15.28:0/486696627 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fda6004e710 con 0x5634bd314bc0 2024-05-12T22:13:31.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.610+0000 7fda78bea740 10 client.26218 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:13:31.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.610+0000 7fda78bea740 10 client.26218 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:13:31.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.610+0000 7fda78bea740 20 client.26218 populate_metadata read hostname 'smithi028' 2024-05-12T22:13:31.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.610+0000 7fda78bea740 1 --2- 172.21.15.28:0/486696627 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5634bd3954d0 0x5634bd3978f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:31.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.610+0000 7fda78bea740 4 client.26218 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-12T22:13:31.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.610+0000 7fda78bea740 1 -- 172.21.15.28:0/486696627 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5634bd281610 con 0x5634bd3954d0 2024-05-12T22:13:31.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.611+0000 7fda6e000640 1 --2- 172.21.15.28:0/486696627 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5634bd3954d0 0x5634bd3978f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:31.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.611+0000 7fda6e000640 1 --2- 172.21.15.28:0/486696627 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5634bd3954d0 0x5634bd3978f0 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f883bc7dbf8cc779 server_cookie=a3f4e636369ae2fa in_seq=0 out_seq=0 2024-05-12T22:13:31.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.611+0000 7fda5cc00640 10 client.26218 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:13:31.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.611+0000 7fda5cc00640 1 -- 172.21.15.28:0/486696627 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5634bd281610 con 0x5634bd3954d0 2024-05-12T22:13:31.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.611+0000 7fda5cc00640 10 client.26218 handle_command_reply: tid=0 2024-05-12T22:13:31.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.611+0000 7fda78bea740 1 client.26218 shutdown 2024-05-12T22:13:31.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.612+0000 7fda78bea740 1 -- 172.21.15.28:0/486696627 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5634bd314bc0 msgr2=0x5634bd331a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:13:31.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.612+0000 7fda78bea740 1 --2- 172.21.15.28:0/486696627 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5634bd314bc0 0x5634bd331a30 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fda60009730 tx=0x7fda60005f90 comp rx=0 tx=0).stop 2024-05-12T22:13:31.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.612+0000 7fda78bea740 1 -- 172.21.15.28:0/486696627 shutdown_connections 2024-05-12T22:13:31.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.612+0000 7fda78bea740 1 --2- 172.21.15.28:0/486696627 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5634bd3954d0 0x5634bd3978f0 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:31.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.612+0000 7fda78bea740 1 --2- 172.21.15.28:0/486696627 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5634bd31fbb0 0x5634bd393620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:31.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.612+0000 7fda78bea740 1 --2- 172.21.15.28:0/486696627 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634bd3154e0 0x5634bd331f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:31.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.612+0000 7fda78bea740 1 --2- 172.21.15.28:0/486696627 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5634bd314bc0 0x5634bd331a30 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:31.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.612+0000 7fda5cc00640 0 client.26218 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:13:31.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.612+0000 7fda78bea740 1 -- 172.21.15.28:0/486696627 >> 172.21.15.28:0/486696627 conn(0x5634bd310ee0 msgr2=0x5634bd31b020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:13:31.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.612+0000 7fda78bea740 1 -- 172.21.15.28:0/486696627 shutdown_connections 2024-05-12T22:13:31.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.612+0000 7fda78bea740 1 -- 172.21.15.28:0/486696627 wait complete. 2024-05-12T22:13:31.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.612+0000 7fda78bea740 20 client.26218 trim_cache size 0 max 16384 2024-05-12T22:13:31.621 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-05-12T22:13:31.621 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-05-12T22:13:31.622 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-05-12T22:13:31.622 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-05-12T22:13:31.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.613+0000 7fda75e00640 1 -- 172.21.15.28:0/2959393856 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fda540a2d80 msgr2=0x7fda540a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:13:31.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.613+0000 7fda75e00640 1 --2- 172.21.15.28:0/2959393856 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fda540a2d80 0x7fda540a5240 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fda58009fd0 tx=0x7fda58005a00 comp rx=0 tx=0).stop 2024-05-12T22:13:31.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.613+0000 7fda75e00640 1 -- 172.21.15.28:0/2959393856 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fda70109a80 msgr2=0x7fda7019ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:13:31.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.613+0000 7fda75e00640 1 --2- 172.21.15.28:0/2959393856 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fda70109a80 0x7fda7019ec90 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fda68009fd0 tx=0x7fda6800ef10 comp rx=0 tx=0).stop 2024-05-12T22:13:31.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.613+0000 7fda75e00640 1 -- 172.21.15.28:0/2959393856 shutdown_connections 2024-05-12T22:13:31.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.613+0000 7fda75e00640 1 --2- 172.21.15.28:0/2959393856 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fda540a2d80 0x7fda540a5240 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:31.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.613+0000 7fda75e00640 1 --2- 172.21.15.28:0/2959393856 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fda70109a80 0x7fda7019ec90 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:31.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.613+0000 7fda75e00640 1 --2- 172.21.15.28:0/2959393856 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fda70104430 0x7fda7019a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:31.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.613+0000 7fda75e00640 1 --2- 172.21.15.28:0/2959393856 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda70103a70 0x7fda7019a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:31.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.613+0000 7fda75e00640 1 -- 172.21.15.28:0/2959393856 >> 172.21.15.28:0/2959393856 conn(0x7fda700ff5c0 msgr2=0x7fda700ffe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:13:31.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.613+0000 7fda75e00640 1 -- 172.21.15.28:0/2959393856 shutdown_connections 2024-05-12T22:13:31.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.613+0000 7fda75e00640 1 -- 172.21.15.28:0/2959393856 wait complete. 2024-05-12T22:13:31.631 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-12T22:13:31.631 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-12T22:13:31.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.835+0000 7f6538200640 1 -- 172.21.15.28:0/3195495468 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6530108a60 msgr2=0x7f653010caa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:13:31.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.835+0000 7f6538200640 1 --2- 172.21.15.28:0/3195495468 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6530108a60 0x7f653010caa0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f651c009900 tx=0x7f651c02a410 comp rx=0 tx=0).stop 2024-05-12T22:13:31.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.835+0000 7f6538200640 1 -- 172.21.15.28:0/3195495468 shutdown_connections 2024-05-12T22:13:31.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.835+0000 7f6538200640 1 --2- 172.21.15.28:0/3195495468 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f653010d180 0x7f6530114d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:31.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.835+0000 7f6538200640 1 --2- 172.21.15.28:0/3195495468 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6530108a60 0x7f653010caa0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:31.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.835+0000 7f6538200640 1 --2- 172.21.15.28:0/3195495468 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f65301080b0 0x7f6530108490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:31.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.835+0000 7f6538200640 1 -- 172.21.15.28:0/3195495468 >> 172.21.15.28:0/3195495468 conn(0x7f6530075030 msgr2=0x7f6530106650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:13:31.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.835+0000 7f6538200640 1 -- 172.21.15.28:0/3195495468 shutdown_connections 2024-05-12T22:13:31.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.835+0000 7f6538200640 1 -- 172.21.15.28:0/3195495468 wait complete. 2024-05-12T22:13:31.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.836+0000 7f6538200640 1 Processor -- start 2024-05-12T22:13:31.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.837+0000 7f6538200640 1 -- start start 2024-05-12T22:13:31.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.837+0000 7f6538200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f65301080b0 0x7f65301a4450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:31.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.837+0000 7f6538200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f653010d180 0x7f65301a4990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:31.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.837+0000 7f6538200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f653019e600 0x7f653019eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:31.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.837+0000 7f6538200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6530118240 con 0x7f653019e600 2024-05-12T22:13:31.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.837+0000 7f6538200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f65301180c0 con 0x7f65301080b0 2024-05-12T22:13:31.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.837+0000 7f6538200640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f65301183c0 con 0x7f653010d180 2024-05-12T22:13:31.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.837+0000 7f6535800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f65301080b0 0x7f65301a4450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:31.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.838+0000 7f6535800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f65301080b0 0x7f65301a4450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:43784/0 (socket says 172.21.15.28:43784) 2024-05-12T22:13:31.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.838+0000 7f6535800640 1 -- 172.21.15.28:0/2918967033 learned_addr learned my addr 172.21.15.28:0/2918967033 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:13:31.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.838+0000 7f6536200640 1 --2- 172.21.15.28:0/2918967033 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f653019e600 0x7f653019eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:31.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.838+0000 7f6534e00640 1 --2- 172.21.15.28:0/2918967033 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f653010d180 0x7f65301a4990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:31.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.838+0000 7f6535800640 1 -- 172.21.15.28:0/2918967033 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f653010d180 msgr2=0x7f65301a4990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:13:31.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.838+0000 7f6535800640 1 --2- 172.21.15.28:0/2918967033 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f653010d180 0x7f65301a4990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:31.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.838+0000 7f6535800640 1 -- 172.21.15.28:0/2918967033 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f653019e600 msgr2=0x7f653019eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:13:31.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.838+0000 7f6535800640 1 --2- 172.21.15.28:0/2918967033 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f653019e600 0x7f653019eab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:31.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.838+0000 7f6535800640 1 -- 172.21.15.28:0/2918967033 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f653019f120 con 0x7f65301080b0 2024-05-12T22:13:31.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.838+0000 7f6535800640 1 --2- 172.21.15.28:0/2918967033 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f65301080b0 0x7f65301a4450 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f652400e9f0 tx=0x7f652400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:13:31.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.839+0000 7f652e000640 1 -- 172.21.15.28:0/2918967033 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f652400cd80 con 0x7f65301080b0 2024-05-12T22:13:31.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.839+0000 7f652e000640 1 -- 172.21.15.28:0/2918967033 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6524010910 con 0x7f65301080b0 2024-05-12T22:13:31.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.839+0000 7f6538200640 1 -- 172.21.15.28:0/2918967033 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f653019f410 con 0x7f65301080b0 2024-05-12T22:13:31.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.839+0000 7f652e000640 1 -- 172.21.15.28:0/2918967033 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6524018990 con 0x7f65301080b0 2024-05-12T22:13:31.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.839+0000 7f6538200640 1 -- 172.21.15.28:0/2918967033 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65301ab660 con 0x7f65301080b0 2024-05-12T22:13:31.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.840+0000 7f6538200640 1 -- 172.21.15.28:0/2918967033 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6530068aa0 con 0x7f65301080b0 2024-05-12T22:13:31.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.842+0000 7f652e000640 1 -- 172.21.15.28:0/2918967033 <== mon.2 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7f6524010ac0 con 0x7f65301080b0 2024-05-12T22:13:31.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.844+0000 7f652e000640 1 --2- 172.21.15.28:0/2918967033 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f65180a2d00 0x7f65180a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:31.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.844+0000 7f652e000640 1 -- 172.21.15.28:0/2918967033 <== mon.2 v2:172.21.15.86:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f65240cfcb0 con 0x7f65301080b0 2024-05-12T22:13:31.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.844+0000 7f6534e00640 1 --2- 172.21.15.28:0/2918967033 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f65180a2d00 0x7f65180a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:31.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.845+0000 7f6534e00640 1 --2- 172.21.15.28:0/2918967033 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f65180a2d00 0x7f65180a51c0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f651c006010 tx=0x7f651c000bc0 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:13:31.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:31.846+0000 7f652e000640 1 -- 172.21.15.28:0/2918967033 <== mon.2 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f652409c250 con 0x7f65301080b0 2024-05-12T22:13:32.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:13:31 smithi028 ceph-mon[29847]: pgmap v139: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-12T22:13:32.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.091+0000 7f6538200640 1 -- 172.21.15.28:0/2918967033 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f653019fb30 con 0x7f65301080b0 2024-05-12T22:13:32.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.092+0000 7f652e000640 1 -- 172.21.15.28:0/2918967033 <== mon.2 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11864 (secure 0 0 0) 0x7f65240a0550 con 0x7f65301080b0 2024-05-12T22:13:32.093 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-12T22:13:32.094 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1844141745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1844141745},{"type":"v1","addr":"172.21.15.186:6833","nonce":1844141745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6833/2960763758","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":2960763758},{"type":"v1","addr":"172.21.15.86:6833","nonce":2960763758}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24470,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/4131134219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":4131134219},{"type":"v1","addr":"172.21.15.28:6835","nonce":4131134219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24481,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/1727869064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1727869064},{"type":"v1","addr":"172.21.15.186:6835","nonce":1727869064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24482,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2938260704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2938260704},{"type":"v1","addr":"172.21.15.28:6837","nonce":2938260704}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24494,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3473700910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3473700910},{"type":"v1","addr":"172.21.15.86:6835","nonce":3473700910}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-12T22:06:33.426986+0000","modified":"2024-05-12T22:13:22.499381+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24581,"mds_1":24550,"mds_2":24532},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24499":{"gid":24499,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6837/3600809235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":3600809235},{"type":"v1","addr":"172.21.15.186:6837","nonce":3600809235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"i","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6839/2399610807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":2399610807},{"type":"v1","addr":"172.21.15.186:6839","nonce":2399610807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24530":{"gid":24530,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.86:6837/3782448884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":3782448884},{"type":"v1","addr":"172.21.15.86:6837","nonce":3782448884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24532":{"gid":24532,"name":"g","rank":2,"incarnation":21,"state":"up:active","state_seq":6,"addr":"172.21.15.28:6839/123892158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":123892158},{"type":"v1","addr":"172.21.15.28:6839","nonce":123892158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24550":{"gid":24550,"name":"a","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.28:6841/761183870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":761183870},{"type":"v1","addr":"172.21.15.28:6841","nonce":761183870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24581":{"gid":24581,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6839/1251999130","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6838","nonce":1251999130},{"type":"v1","addr":"172.21.15.86:6839","nonce":1251999130}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24581,"qdb_cluster":[24550,24581,24532]},"id":1}]} 2024-05-12T22:13:32.094 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 26 2024-05-12T22:13:32.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.096+0000 7f6538200640 1 -- 172.21.15.28:0/2918967033 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f65180a2d00 msgr2=0x7f65180a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:13:32.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.096+0000 7f6538200640 1 --2- 172.21.15.28:0/2918967033 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f65180a2d00 0x7f65180a51c0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f651c006010 tx=0x7f651c000bc0 comp rx=0 tx=0).stop 2024-05-12T22:13:32.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.096+0000 7f6538200640 1 -- 172.21.15.28:0/2918967033 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f65301080b0 msgr2=0x7f65301a4450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:13:32.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.096+0000 7f6538200640 1 --2- 172.21.15.28:0/2918967033 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f65301080b0 0x7f65301a4450 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f652400e9f0 tx=0x7f652400eeb0 comp rx=0 tx=0).stop 2024-05-12T22:13:32.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.097+0000 7f6538200640 1 -- 172.21.15.28:0/2918967033 shutdown_connections 2024-05-12T22:13:32.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.097+0000 7f6538200640 1 --2- 172.21.15.28:0/2918967033 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f65180a2d00 0x7f65180a51c0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:32.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.097+0000 7f6538200640 1 --2- 172.21.15.28:0/2918967033 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f653019e600 0x7f653019eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:32.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.097+0000 7f6538200640 1 --2- 172.21.15.28:0/2918967033 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f653010d180 0x7f65301a4990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:32.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.097+0000 7f6538200640 1 --2- 172.21.15.28:0/2918967033 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f65301080b0 0x7f65301a4450 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:32.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.097+0000 7f6538200640 1 -- 172.21.15.28:0/2918967033 >> 172.21.15.28:0/2918967033 conn(0x7f6530075030 msgr2=0x7f65301074e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:13:32.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.097+0000 7f6538200640 1 -- 172.21.15.28:0/2918967033 shutdown_connections 2024-05-12T22:13:32.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.097+0000 7f6538200640 1 -- 172.21.15.28:0/2918967033 wait complete. 2024-05-12T22:13:32.118 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-05-12T22:13:32.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:13:31 smithi186 ceph-mon[34785]: pgmap v139: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-12T22:13:32.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:13:31 smithi086 ceph-mon[34297]: pgmap v139: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-12T22:13:32.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.343+0000 7f28eae00640 1 -- 172.21.15.28:0/3076617368 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28e410ddf0 msgr2=0x7f28e410e1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:13:32.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.343+0000 7f28eae00640 1 --2- 172.21.15.28:0/3076617368 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28e410ddf0 0x7f28e410e1d0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f28d8009900 tx=0x7f28d802a480 comp rx=0 tx=0).stop 2024-05-12T22:13:32.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.343+0000 7f28eae00640 1 -- 172.21.15.28:0/3076617368 shutdown_connections 2024-05-12T22:13:32.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.343+0000 7f28eae00640 1 --2- 172.21.15.28:0/3076617368 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f28e4108770 0x7f28e4114b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:32.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.343+0000 7f28eae00640 1 --2- 172.21.15.28:0/3076617368 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f28e4107dd0 0x7f28e4108230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:32.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.343+0000 7f28eae00640 1 --2- 172.21.15.28:0/3076617368 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28e410ddf0 0x7f28e410e1d0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:32.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.343+0000 7f28eae00640 1 -- 172.21.15.28:0/3076617368 >> 172.21.15.28:0/3076617368 conn(0x7f28e4077650 msgr2=0x7f28e4106650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:13:32.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.343+0000 7f28eae00640 1 -- 172.21.15.28:0/3076617368 shutdown_connections 2024-05-12T22:13:32.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.343+0000 7f28eae00640 1 -- 172.21.15.28:0/3076617368 wait complete. 2024-05-12T22:13:32.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.344+0000 7f28eae00640 1 Processor -- start 2024-05-12T22:13:32.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.344+0000 7f28eae00640 1 -- start start 2024-05-12T22:13:32.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.345+0000 7f28eae00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f28e4107dd0 0x7f28e419e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:32.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.345+0000 7f28eae00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f28e4108770 0x7f28e419ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:32.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.345+0000 7f28eae00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28e410ddf0 0x7f28e41a2dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:32.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.345+0000 7f28eae00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f28e41180d0 con 0x7f28e410ddf0 2024-05-12T22:13:32.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.345+0000 7f28eae00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f28e4117f50 con 0x7f28e4108770 2024-05-12T22:13:32.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.345+0000 7f28eae00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f28e4118250 con 0x7f28e4107dd0 2024-05-12T22:13:32.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.345+0000 7f28e9a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f28e4107dd0 0x7f28e419e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:32.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.345+0000 7f28ea400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28e410ddf0 0x7f28e41a2dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:32.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.345+0000 7f28e9a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f28e4107dd0 0x7f28e419e550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:37532/0 (socket says 172.21.15.28:37532) 2024-05-12T22:13:32.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.345+0000 7f28e9a00640 1 -- 172.21.15.28:0/709641023 learned_addr learned my addr 172.21.15.28:0/709641023 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:13:32.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.345+0000 7f28e9000640 1 --2- 172.21.15.28:0/709641023 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f28e4108770 0x7f28e419ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:32.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.346+0000 7f28e9a00640 1 -- 172.21.15.28:0/709641023 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f28e4108770 msgr2=0x7f28e419ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:13:32.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.346+0000 7f28e9a00640 1 --2- 172.21.15.28:0/709641023 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f28e4108770 0x7f28e419ea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:32.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.346+0000 7f28e9a00640 1 -- 172.21.15.28:0/709641023 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28e410ddf0 msgr2=0x7f28e41a2dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:13:32.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.346+0000 7f28e9a00640 1 --2- 172.21.15.28:0/709641023 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28e410ddf0 0x7f28e41a2dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:32.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.346+0000 7f28e9a00640 1 -- 172.21.15.28:0/709641023 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28e41a34f0 con 0x7f28e4107dd0 2024-05-12T22:13:32.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.346+0000 7f28ea400640 1 --2- 172.21.15.28:0/709641023 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28e410ddf0 0x7f28e41a2dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:13:32.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.346+0000 7f28e9a00640 1 --2- 172.21.15.28:0/709641023 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f28e4107dd0 0x7f28e419e550 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f28d8006010 tx=0x7f28d8004040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:13:32.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.346+0000 7f28e2000640 1 -- 172.21.15.28:0/709641023 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28d803f030 con 0x7f28e4107dd0 2024-05-12T22:13:32.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.346+0000 7f28e2000640 1 -- 172.21.15.28:0/709641023 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f28d8004440 con 0x7f28e4107dd0 2024-05-12T22:13:32.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.346+0000 7f28eae00640 1 -- 172.21.15.28:0/709641023 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28e41a3780 con 0x7f28e4107dd0 2024-05-12T22:13:32.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.346+0000 7f28e2000640 1 -- 172.21.15.28:0/709641023 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28d8035070 con 0x7f28e4107dd0 2024-05-12T22:13:32.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.346+0000 7f28eae00640 1 -- 172.21.15.28:0/709641023 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28e41ab070 con 0x7f28e4107dd0 2024-05-12T22:13:32.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.350+0000 7f28ef252740 1 Processor -- start 2024-05-12T22:13:32.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.350+0000 7f28e2000640 1 -- 172.21.15.28:0/709641023 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7f28d8042030 con 0x7f28e4107dd0 2024-05-12T22:13:32.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.350+0000 7f28ef252740 1 -- start start 2024-05-12T22:13:32.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.350+0000 7f28ef252740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562a81457b60 0x562a81457f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:32.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.350+0000 7f28ef252740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562a81458480 0x562a81462610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:32.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.350+0000 7f28ef252740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a81462b50 0x562a81464f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:32.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.350+0000 7f28ef252740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562a813bca30 con 0x562a81462b50 2024-05-12T22:13:32.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.350+0000 7f28ef252740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x562a81364700 con 0x562a81457b60 2024-05-12T22:13:32.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.350+0000 7f28ef252740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x562a813b89f0 con 0x562a81458480 2024-05-12T22:13:32.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.351+0000 7f28ea400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562a81457b60 0x562a81457f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:32.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.351+0000 7f28e9000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a81462b50 0x562a81464f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:32.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.351+0000 7f28ea400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562a81457b60 0x562a81457f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:43812/0 (socket says 172.21.15.28:43812) 2024-05-12T22:13:32.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.351+0000 7f28ea400640 1 -- 172.21.15.28:0/147093689 learned_addr learned my addr 172.21.15.28:0/147093689 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:13:32.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.351+0000 7f28e9a00640 1 --2- 172.21.15.28:0/147093689 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562a81458480 0x562a81462610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:32.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.351+0000 7f28ea400640 1 -- 172.21.15.28:0/147093689 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562a81458480 msgr2=0x562a81462610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:13:32.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.351+0000 7f28ea400640 1 --2- 172.21.15.28:0/147093689 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562a81458480 0x562a81462610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:32.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.351+0000 7f28ea400640 1 -- 172.21.15.28:0/147093689 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a81462b50 msgr2=0x562a81464f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:13:32.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.351+0000 7f28ea400640 1 --2- 172.21.15.28:0/147093689 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a81462b50 0x562a81464f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:32.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.351+0000 7f28ea400640 1 -- 172.21.15.28:0/147093689 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562a810d03f0 con 0x562a81457b60 2024-05-12T22:13:32.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.351+0000 7f28e9000640 1 --2- 172.21.15.28:0/147093689 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a81462b50 0x562a81464f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:13:32.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.351+0000 7f28e2000640 1 --2- 172.21.15.28:0/709641023 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f28cc0a2d00 0x7f28cc0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:32.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.352+0000 7f28ea400640 1 --2- 172.21.15.28:0/147093689 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562a81457b60 0x562a81457f40 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f28dc0117f0 tx=0x7f28dc011cb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cbefcd7ea97ed2a server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:13:32.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.352+0000 7f28e2000640 1 -- 172.21.15.28:0/709641023 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f28d80ef7c0 con 0x7f28e4107dd0 2024-05-12T22:13:32.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.352+0000 7f28e9a00640 1 --2- 172.21.15.28:0/709641023 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f28cc0a2d00 0x7f28cc0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:32.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.352+0000 7f28d6a00640 1 -- 172.21.15.28:0/147093689 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28dc0112f0 con 0x562a81457b60 2024-05-12T22:13:32.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.352+0000 7f28d6a00640 1 -- 172.21.15.28:0/147093689 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f28dc003e50 con 0x562a81457b60 2024-05-12T22:13:32.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.352+0000 7f28d6a00640 1 -- 172.21.15.28:0/147093689 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28dc0045f0 con 0x562a81457b60 2024-05-12T22:13:32.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.353+0000 7f28ef252740 1 -- 172.21.15.28:0/147093689 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562a81457b60 msgr2=0x562a81457f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:13:32.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.353+0000 7f28e9a00640 1 --2- 172.21.15.28:0/709641023 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f28cc0a2d00 0x7f28cc0a51c0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x562a81459120 tx=0x7f28d80f5530 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:13:32.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.353+0000 7f28ef252740 1 --2- 172.21.15.28:0/147093689 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562a81457b60 0x562a81457f40 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f28dc0117f0 tx=0x7f28dc011cb0 comp rx=0 tx=0).stop 2024-05-12T22:13:32.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.353+0000 7f28ef252740 1 -- 172.21.15.28:0/147093689 shutdown_connections 2024-05-12T22:13:32.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.353+0000 7f28ef252740 1 --2- 172.21.15.28:0/147093689 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a81462b50 0x562a81464f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:32.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.353+0000 7f28ef252740 1 --2- 172.21.15.28:0/147093689 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562a81458480 0x562a81462610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:32.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.353+0000 7f28ef252740 1 --2- 172.21.15.28:0/147093689 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562a81457b60 0x562a81457f40 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:32.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.353+0000 7f28ef252740 1 -- 172.21.15.28:0/147093689 >> 172.21.15.28:0/147093689 conn(0x562a81453e80 msgr2=0x562a81454260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:13:32.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.353+0000 7f28ef252740 1 -- 172.21.15.28:0/147093689 shutdown_connections 2024-05-12T22:13:32.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.353+0000 7f28ef252740 1 -- 172.21.15.28:0/147093689 wait complete. 2024-05-12T22:13:32.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.353+0000 7f28ef252740 1 -- start start 2024-05-12T22:13:32.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.354+0000 7f28ef252740 1 Processor -- start 2024-05-12T22:13:32.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.354+0000 7f28ef252740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562a81457b60 0x562a8145fb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:32.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.354+0000 7f28ef252740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562a81458480 0x562a81460080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:32.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.354+0000 7f28ef252740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a81462b50 0x562a814605c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:32.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.354+0000 7f28ef252740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562a81465940 con 0x562a81462b50 2024-05-12T22:13:32.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.354+0000 7f28ef252740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x562a81377a70 con 0x562a81458480 2024-05-12T22:13:32.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.354+0000 7f28ef252740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x562a81465ac0 con 0x562a81457b60 2024-05-12T22:13:32.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.354+0000 7f28ef252740 10 client.0 _ll_register_callbacks cb 0x562a8135aba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:13:32.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.355+0000 7f28e9000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562a81458480 0x562a81460080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:32.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.355+0000 7f28e9000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562a81458480 0x562a81460080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:43822/0 (socket says 172.21.15.28:43822) 2024-05-12T22:13:32.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.355+0000 7f28e9000640 1 -- 172.21.15.28:0/1104012095 learned_addr learned my addr 172.21.15.28:0/1104012095 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:13:32.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.355+0000 7f28ea400640 1 --2- 172.21.15.28:0/1104012095 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a81462b50 0x562a814605c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:32.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.355+0000 7f28ea400640 1 --2- 172.21.15.28:0/1104012095 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562a81457b60 0x562a8145fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:32.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.355+0000 7f28e9000640 1 -- 172.21.15.28:0/1104012095 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562a81457b60 msgr2=0x562a8145fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:13:32.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.355+0000 7f28e9000640 1 --2- 172.21.15.28:0/1104012095 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562a81457b60 0x562a8145fb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:32.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.355+0000 7f28e9000640 1 -- 172.21.15.28:0/1104012095 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a81462b50 msgr2=0x562a814605c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:13:32.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.355+0000 7f28e9000640 1 --2- 172.21.15.28:0/1104012095 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a81462b50 0x562a814605c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:32.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.355+0000 7f28e9000640 1 -- 172.21.15.28:0/1104012095 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562a8119a7d0 con 0x562a81458480 2024-05-12T22:13:32.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.355+0000 7f28ea400640 1 --2- 172.21.15.28:0/1104012095 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562a81457b60 0x562a8145fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-12T22:13:32.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.355+0000 7f28e9000640 1 --2- 172.21.15.28:0/1104012095 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562a81458480 0x562a81460080 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f28d00126e0 tx=0x7f28d0012ba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=96daf78da7fef4d server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:13:32.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.355+0000 7f28cb400640 10 client.0 ms_handle_connect on v2:172.21.15.86:3300/0 2024-05-12T22:13:32.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.355+0000 7f28cb400640 1 -- 172.21.15.28:0/1104012095 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28d0004050 con 0x562a81458480 2024-05-12T22:13:32.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.356+0000 7f28cb400640 1 -- 172.21.15.28:0/1104012095 <== mon.2 v2:172.21.15.86:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f28d00041f0 con 0x562a81458480 2024-05-12T22:13:32.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.356+0000 7f28ef252740 1 -- 172.21.15.28:0/1104012095 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f28dc003e50 con 0x562a81458480 2024-05-12T22:13:32.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.356+0000 7f28cb400640 1 -- 172.21.15.28:0/1104012095 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f28d00043f0 con 0x562a81458480 2024-05-12T22:13:32.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.356+0000 7f28ef252740 10 client.26276 fetch_fsmap learned FSMap version 26 2024-05-12T22:13:32.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.356+0000 7f28ef252740 1 -- 172.21.15.28:0/1104012095 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x562a81471cd0 con 0x562a81458480 2024-05-12T22:13:32.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.356+0000 7f28cb400640 1 -- 172.21.15.28:0/1104012095 <== mon.2 v2:172.21.15.86:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f28dc003e50 con 0x562a81458480 2024-05-12T22:13:32.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.356+0000 7f28cb400640 1 -- 172.21.15.28:0/1104012095 <== mon.2 v2:172.21.15.86:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f28d0003c40 con 0x562a81458480 2024-05-12T22:13:32.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.357+0000 7f28ef252740 10 client.26276 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:13:32.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.357+0000 7f28ef252740 10 client.26276 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.g 2024-05-12T22:13:32.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.357+0000 7f28ef252740 20 client.26276 populate_metadata read hostname 'smithi028' 2024-05-12T22:13:32.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.357+0000 7f28ef252740 1 --2- 172.21.15.28:0/1104012095 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x562a81473230 0x562a81475650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:32.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.357+0000 7f28ef252740 4 client.26276 mds_command: new command op to 24532 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-12T22:13:32.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.357+0000 7f28ef252740 1 -- 172.21.15.28:0/1104012095 --> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562a813ca160 con 0x562a81473230 2024-05-12T22:13:32.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.357+0000 7f28ea400640 1 --2- 172.21.15.28:0/1104012095 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x562a81473230 0x562a81475650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:32.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.358+0000 7f28ea400640 1 --2- 172.21.15.28:0/1104012095 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x562a81473230 0x562a81475650 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d82ad163e58a2121 server_cookie=599de6363473d1c7 in_seq=0 out_seq=0 2024-05-12T22:13:32.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.358+0000 7f28cb400640 10 client.26276 ms_handle_connect on v2:172.21.15.28:6838/123892158 2024-05-12T22:13:32.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.358+0000 7f28cb400640 1 -- 172.21.15.28:0/1104012095 <== mds.2 v2:172.21.15.28:6838/123892158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562a813ca160 con 0x562a81473230 2024-05-12T22:13:32.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.358+0000 7f28cb400640 10 client.26276 handle_command_reply: tid=0 2024-05-12T22:13:32.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.358+0000 7f28ef252740 1 client.26276 shutdown 2024-05-12T22:13:32.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.359+0000 7f28ef252740 1 -- 172.21.15.28:0/1104012095 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562a81458480 msgr2=0x562a81460080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:13:32.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.359+0000 7f28ef252740 1 --2- 172.21.15.28:0/1104012095 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562a81458480 0x562a81460080 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f28d00126e0 tx=0x7f28d0012ba0 comp rx=0 tx=0).stop 2024-05-12T22:13:32.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.359+0000 7f28ef252740 1 -- 172.21.15.28:0/1104012095 shutdown_connections 2024-05-12T22:13:32.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.359+0000 7f28ef252740 1 --2- 172.21.15.28:0/1104012095 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a81462b50 0x562a814605c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:32.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.359+0000 7f28ef252740 1 --2- 172.21.15.28:0/1104012095 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x562a81473230 0x562a81475650 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:32.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.359+0000 7f28ef252740 1 --2- 172.21.15.28:0/1104012095 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562a81458480 0x562a81460080 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:32.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.359+0000 7f28ef252740 1 --2- 172.21.15.28:0/1104012095 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562a81457b60 0x562a8145fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:32.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.359+0000 7f28cb400640 0 client.26276 ms_handle_reset on v2:172.21.15.28:6838/123892158 2024-05-12T22:13:32.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.359+0000 7f28ef252740 1 -- 172.21.15.28:0/1104012095 >> 172.21.15.28:0/1104012095 conn(0x562a81453e80 msgr2=0x562a81459c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:13:32.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.359+0000 7f28ef252740 1 -- 172.21.15.28:0/1104012095 shutdown_connections 2024-05-12T22:13:32.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.359+0000 7f28ef252740 1 -- 172.21.15.28:0/1104012095 wait complete. 2024-05-12T22:13:32.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.359+0000 7f28ef252740 20 client.26276 trim_cache size 0 max 16384 2024-05-12T22:13:32.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.374+0000 7f28ef252740 1 Processor -- start 2024-05-12T22:13:32.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.374+0000 7f28ef252740 1 -- start start 2024-05-12T22:13:32.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.374+0000 7f28ef252740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562a81457b60 0x562a8146e830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:32.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.374+0000 7f28ef252740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a81458480 0x562a8147af40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:32.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.374+0000 7f28ef252740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562a81462b50 0x562a8147b480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:32.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.375+0000 7f28ef252740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f28e41b6150 con 0x562a81458480 2024-05-12T22:13:32.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.375+0000 7f28ef252740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f28e41b3a20 con 0x562a81462b50 2024-05-12T22:13:32.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.375+0000 7f28ef252740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f28e41b1870 con 0x562a81457b60 2024-05-12T22:13:32.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.375+0000 7f28e9000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a81458480 0x562a8147af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:32.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.375+0000 7f28e9000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a81458480 0x562a8147af40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50184/0 (socket says 172.21.15.28:50184) 2024-05-12T22:13:32.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.375+0000 7f28e9000640 1 -- 172.21.15.28:0/2659611227 learned_addr learned my addr 172.21.15.28:0/2659611227 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:13:32.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.375+0000 7f28ea400640 1 --2- 172.21.15.28:0/2659611227 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562a81457b60 0x562a8146e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:32.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.375+0000 7f28ea400640 1 --2- 172.21.15.28:0/2659611227 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562a81462b50 0x562a8147b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:32.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.375+0000 7f28e9000640 1 -- 172.21.15.28:0/2659611227 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562a81457b60 msgr2=0x562a8146e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:13:32.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.375+0000 7f28e9000640 1 --2- 172.21.15.28:0/2659611227 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562a81457b60 0x562a8146e830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:32.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.375+0000 7f28e9000640 1 -- 172.21.15.28:0/2659611227 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562a81462b50 msgr2=0x562a8147b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:13:32.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.375+0000 7f28e9000640 1 --2- 172.21.15.28:0/2659611227 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562a81462b50 0x562a8147b480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:32.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.375+0000 7f28e9000640 1 -- 172.21.15.28:0/2659611227 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562a814550b0 con 0x562a81458480 2024-05-12T22:13:32.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.375+0000 7f28ea400640 1 --2- 172.21.15.28:0/2659611227 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562a81462b50 0x562a8147b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-12T22:13:32.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.376+0000 7f28e9000640 1 --2- 172.21.15.28:0/2659611227 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a81458480 0x562a8147af40 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f28d0010430 tx=0x7f28d0010930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e080d8ced44cade server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:13:32.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.376+0000 7f28d6000640 1 -- 172.21.15.28:0/2659611227 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28d0029040 con 0x562a81458480 2024-05-12T22:13:32.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.376+0000 7f28d6000640 1 -- 172.21.15.28:0/2659611227 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f28d0003fb0 con 0x562a81458480 2024-05-12T22:13:32.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.376+0000 7f28d6000640 1 -- 172.21.15.28:0/2659611227 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28d0005040 con 0x562a81458480 2024-05-12T22:13:32.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.377+0000 7f28ef252740 1 -- 172.21.15.28:0/2659611227 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a81458480 msgr2=0x562a8147af40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:13:32.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.377+0000 7f28ef252740 1 --2- 172.21.15.28:0/2659611227 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a81458480 0x562a8147af40 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f28d0010430 tx=0x7f28d0010930 comp rx=0 tx=0).stop 2024-05-12T22:13:32.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.377+0000 7f28ef252740 1 -- 172.21.15.28:0/2659611227 shutdown_connections 2024-05-12T22:13:32.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.377+0000 7f28ef252740 1 --2- 172.21.15.28:0/2659611227 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562a81462b50 0x562a8147b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:32.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.377+0000 7f28ef252740 1 --2- 172.21.15.28:0/2659611227 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a81458480 0x562a8147af40 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:32.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.377+0000 7f28ef252740 1 --2- 172.21.15.28:0/2659611227 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562a81457b60 0x562a8146e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:32.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.377+0000 7f28ef252740 1 -- 172.21.15.28:0/2659611227 >> 172.21.15.28:0/2659611227 conn(0x562a81453e80 msgr2=0x562a81454260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:13:32.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.377+0000 7f28ef252740 1 -- 172.21.15.28:0/2659611227 shutdown_connections 2024-05-12T22:13:32.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.377+0000 7f28ef252740 1 -- 172.21.15.28:0/2659611227 wait complete. 2024-05-12T22:13:32.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.377+0000 7f28ef252740 1 -- start start 2024-05-12T22:13:32.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.378+0000 7f28ef252740 1 Processor -- start 2024-05-12T22:13:32.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.378+0000 7f28ef252740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562a81457b60 0x562a81466940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:32.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.378+0000 7f28ef252740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562a81458480 0x562a81466e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:32.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.378+0000 7f28ef252740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a81462b50 0x562a814673c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:32.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.378+0000 7f28ef252740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f28e41b1ee0 con 0x562a81462b50 2024-05-12T22:13:32.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.378+0000 7f28ef252740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f28e41b3350 con 0x562a81458480 2024-05-12T22:13:32.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.378+0000 7f28ef252740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f28e41b25b0 con 0x562a81457b60 2024-05-12T22:13:32.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.378+0000 7f28ef252740 10 client.0 _ll_register_callbacks cb 0x7f28d0009ea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:13:32.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.379+0000 7f28e9000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562a81458480 0x562a81466e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:32.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.379+0000 7f28e9000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562a81458480 0x562a81466e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:43842/0 (socket says 172.21.15.28:43842) 2024-05-12T22:13:32.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.379+0000 7f28e9000640 1 -- 172.21.15.28:0/1244793227 learned_addr learned my addr 172.21.15.28:0/1244793227 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:13:32.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.379+0000 7f28ea400640 1 --2- 172.21.15.28:0/1244793227 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a81462b50 0x562a814673c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:32.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.379+0000 7f28ea400640 1 --2- 172.21.15.28:0/1244793227 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562a81457b60 0x562a81466940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:32.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.379+0000 7f28e9000640 1 -- 172.21.15.28:0/1244793227 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562a81457b60 msgr2=0x562a81466940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:13:32.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.379+0000 7f28e9000640 1 --2- 172.21.15.28:0/1244793227 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562a81457b60 0x562a81466940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:32.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.379+0000 7f28e9000640 1 -- 172.21.15.28:0/1244793227 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a81462b50 msgr2=0x562a814673c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:13:32.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.379+0000 7f28e9000640 1 --2- 172.21.15.28:0/1244793227 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a81462b50 0x562a814673c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:32.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.379+0000 7f28e9000640 1 -- 172.21.15.28:0/1244793227 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562a814d89b0 con 0x562a81458480 2024-05-12T22:13:32.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.379+0000 7f28e9000640 1 --2- 172.21.15.28:0/1244793227 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562a81458480 0x562a81466e80 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f28d000a010 tx=0x7f28d00055a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6d4b7282b3699993 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:13:32.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.379+0000 7f28cb400640 10 client.0 ms_handle_connect on v2:172.21.15.86:3300/0 2024-05-12T22:13:32.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.380+0000 7f28cb400640 1 -- 172.21.15.28:0/1244793227 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28d0004220 con 0x562a81458480 2024-05-12T22:13:32.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.380+0000 7f28cb400640 1 -- 172.21.15.28:0/1244793227 <== mon.2 v2:172.21.15.86:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f28d000f920 con 0x562a81458480 2024-05-12T22:13:32.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.380+0000 7f28ef252740 1 -- 172.21.15.28:0/1244793227 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f28d0003fb0 con 0x562a81458480 2024-05-12T22:13:32.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.380+0000 7f28cb400640 1 -- 172.21.15.28:0/1244793227 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f28d0040050 con 0x562a81458480 2024-05-12T22:13:32.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.380+0000 7f28ef252740 10 client.26279 fetch_fsmap learned FSMap version 26 2024-05-12T22:13:32.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.380+0000 7f28ef252740 1 -- 172.21.15.28:0/1244793227 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x562a814d9c00 con 0x562a81458480 2024-05-12T22:13:32.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.380+0000 7f28cb400640 1 -- 172.21.15.28:0/1244793227 <== mon.2 v2:172.21.15.86:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f28d0003fb0 con 0x562a81458480 2024-05-12T22:13:32.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.381+0000 7f28cb400640 1 -- 172.21.15.28:0/1244793227 <== mon.2 v2:172.21.15.86:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f28d002e470 con 0x562a81458480 2024-05-12T22:13:32.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.381+0000 7f28ef252740 10 client.26279 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:13:32.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.381+0000 7f28ef252740 10 client.26279 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.g 2024-05-12T22:13:32.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.381+0000 7f28ef252740 20 client.26279 populate_metadata read hostname 'smithi028' 2024-05-12T22:13:32.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.381+0000 7f28ef252740 1 --2- 172.21.15.28:0/1244793227 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x562a814da200 0x562a814dc620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:32.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.381+0000 7f28ef252740 4 client.26279 mds_command: new command op to 24532 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-12T22:13:32.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.381+0000 7f28ef252740 1 -- 172.21.15.28:0/1244793227 --> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562a813c44b0 con 0x562a814da200 2024-05-12T22:13:32.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.381+0000 7f28ea400640 1 --2- 172.21.15.28:0/1244793227 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x562a814da200 0x562a814dc620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:32.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.382+0000 7f28ea400640 1 --2- 172.21.15.28:0/1244793227 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x562a814da200 0x562a814dc620 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=6b91f550c8bad15a server_cookie=d3a0a14bdefbed89 in_seq=0 out_seq=0 2024-05-12T22:13:32.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.382+0000 7f28cb400640 10 client.26279 ms_handle_connect on v2:172.21.15.28:6838/123892158 2024-05-12T22:13:32.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.382+0000 7f28cb400640 1 -- 172.21.15.28:0/1244793227 <== mds.2 v2:172.21.15.28:6838/123892158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562a813c44b0 con 0x562a814da200 2024-05-12T22:13:32.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.382+0000 7f28cb400640 10 client.26279 handle_command_reply: tid=0 2024-05-12T22:13:32.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.382+0000 7f28ef252740 1 client.26279 shutdown 2024-05-12T22:13:32.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.382+0000 7f28ef252740 1 -- 172.21.15.28:0/1244793227 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562a81458480 msgr2=0x562a81466e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:13:32.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.382+0000 7f28ef252740 1 --2- 172.21.15.28:0/1244793227 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562a81458480 0x562a81466e80 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f28d000a010 tx=0x7f28d00055a0 comp rx=0 tx=0).stop 2024-05-12T22:13:32.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.382+0000 7f28ef252740 1 -- 172.21.15.28:0/1244793227 shutdown_connections 2024-05-12T22:13:32.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.382+0000 7f28ef252740 1 --2- 172.21.15.28:0/1244793227 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a81462b50 0x562a814673c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:32.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.382+0000 7f28ef252740 1 --2- 172.21.15.28:0/1244793227 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x562a814da200 0x562a814dc620 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:32.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.382+0000 7f28ef252740 1 --2- 172.21.15.28:0/1244793227 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562a81458480 0x562a81466e80 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:32.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.382+0000 7f28ef252740 1 --2- 172.21.15.28:0/1244793227 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562a81457b60 0x562a81466940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:32.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.382+0000 7f28cb400640 0 client.26279 ms_handle_reset on v2:172.21.15.28:6838/123892158 2024-05-12T22:13:32.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.382+0000 7f28ef252740 1 -- 172.21.15.28:0/1244793227 >> 172.21.15.28:0/1244793227 conn(0x562a81453e80 msgr2=0x562a8145fb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:13:32.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.382+0000 7f28ef252740 1 -- 172.21.15.28:0/1244793227 shutdown_connections 2024-05-12T22:13:32.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.382+0000 7f28ef252740 1 -- 172.21.15.28:0/1244793227 wait complete. 2024-05-12T22:13:32.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.383+0000 7f28ef252740 20 client.26279 trim_cache size 0 max 16384 2024-05-12T22:13:32.390 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-05-12T22:13:32.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.383+0000 7f28eae00640 1 -- 172.21.15.28:0/709641023 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f28cc0a2d00 msgr2=0x7f28cc0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:13:32.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.383+0000 7f28eae00640 1 --2- 172.21.15.28:0/709641023 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f28cc0a2d00 0x7f28cc0a51c0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x562a81459120 tx=0x7f28d80f5530 comp rx=0 tx=0).stop 2024-05-12T22:13:32.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.383+0000 7f28eae00640 1 -- 172.21.15.28:0/709641023 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f28e4107dd0 msgr2=0x7f28e419e550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:13:32.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.383+0000 7f28eae00640 1 --2- 172.21.15.28:0/709641023 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f28e4107dd0 0x7f28e419e550 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f28d8006010 tx=0x7f28d8004040 comp rx=0 tx=0).stop 2024-05-12T22:13:32.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.383+0000 7f28eae00640 1 -- 172.21.15.28:0/709641023 shutdown_connections 2024-05-12T22:13:32.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.383+0000 7f28eae00640 1 --2- 172.21.15.28:0/709641023 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f28cc0a2d00 0x7f28cc0a51c0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:32.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.383+0000 7f28eae00640 1 --2- 172.21.15.28:0/709641023 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28e410ddf0 0x7f28e41a2dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:32.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.383+0000 7f28eae00640 1 --2- 172.21.15.28:0/709641023 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f28e4108770 0x7f28e419ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:32.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.383+0000 7f28eae00640 1 --2- 172.21.15.28:0/709641023 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f28e4107dd0 0x7f28e419e550 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:32.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.383+0000 7f28eae00640 1 -- 172.21.15.28:0/709641023 >> 172.21.15.28:0/709641023 conn(0x7f28e4077650 msgr2=0x7f28e4078660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:13:32.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.383+0000 7f28eae00640 1 -- 172.21.15.28:0/709641023 shutdown_connections 2024-05-12T22:13:32.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.384+0000 7f28eae00640 1 -- 172.21.15.28:0/709641023 wait complete. 2024-05-12T22:13:32.402 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-05-12T22:13:32.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.603+0000 7f877fe00640 1 -- 172.21.15.28:0/3247372472 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8780107dd0 msgr2=0x7f8780108230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:13:32.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.603+0000 7f877fe00640 1 --2- 172.21.15.28:0/3247372472 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8780107dd0 0x7f8780108230 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f876c009900 tx=0x7f876c02a480 comp rx=0 tx=0).stop 2024-05-12T22:13:32.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.603+0000 7f877fe00640 1 -- 172.21.15.28:0/3247372472 shutdown_connections 2024-05-12T22:13:32.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.603+0000 7f877fe00640 1 --2- 172.21.15.28:0/3247372472 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8780108770 0x7f8780114b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:32.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.603+0000 7f877fe00640 1 --2- 172.21.15.28:0/3247372472 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8780107dd0 0x7f8780108230 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:32.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.603+0000 7f877fe00640 1 --2- 172.21.15.28:0/3247372472 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f878010ddf0 0x7f878010e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:32.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.603+0000 7f877fe00640 1 -- 172.21.15.28:0/3247372472 >> 172.21.15.28:0/3247372472 conn(0x7f8780077650 msgr2=0x7f8780106650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:13:32.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.603+0000 7f877fe00640 1 -- 172.21.15.28:0/3247372472 shutdown_connections 2024-05-12T22:13:32.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.603+0000 7f877fe00640 1 -- 172.21.15.28:0/3247372472 wait complete. 2024-05-12T22:13:32.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.604+0000 7f877fe00640 1 Processor -- start 2024-05-12T22:13:32.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.604+0000 7f877fe00640 1 -- start start 2024-05-12T22:13:32.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.605+0000 7f877fe00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8780107dd0 0x7f878019e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:32.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.605+0000 7f877fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8780108770 0x7f878019eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:32.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.605+0000 7f877fe00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f878010ddf0 0x7f87801a2e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:32.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.605+0000 7f877fe00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8780118100 con 0x7f8780108770 2024-05-12T22:13:32.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.605+0000 7f877fe00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f8780117f80 con 0x7f878010ddf0 2024-05-12T22:13:32.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.605+0000 7f877fe00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f8780118280 con 0x7f8780107dd0 2024-05-12T22:13:32.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.605+0000 7f877e000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8780108770 0x7f878019eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:32.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.605+0000 7f877e000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8780108770 0x7f878019eb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50216/0 (socket says 172.21.15.28:50216) 2024-05-12T22:13:32.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.605+0000 7f877e000640 1 -- 172.21.15.28:0/3133628735 learned_addr learned my addr 172.21.15.28:0/3133628735 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:13:32.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.605+0000 7f877ea00640 1 --2- 172.21.15.28:0/3133628735 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8780107dd0 0x7f878019e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:32.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.605+0000 7f877f400640 1 --2- 172.21.15.28:0/3133628735 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f878010ddf0 0x7f87801a2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:32.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.606+0000 7f877e000640 1 -- 172.21.15.28:0/3133628735 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8780107dd0 msgr2=0x7f878019e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:13:32.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.606+0000 7f877e000640 1 --2- 172.21.15.28:0/3133628735 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8780107dd0 0x7f878019e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:32.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.606+0000 7f877e000640 1 -- 172.21.15.28:0/3133628735 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f878010ddf0 msgr2=0x7f87801a2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:13:32.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.606+0000 7f877e000640 1 --2- 172.21.15.28:0/3133628735 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f878010ddf0 0x7f87801a2e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:32.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.606+0000 7f877e000640 1 -- 172.21.15.28:0/3133628735 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87801a35c0 con 0x7f8780108770 2024-05-12T22:13:32.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.606+0000 7f877ea00640 1 --2- 172.21.15.28:0/3133628735 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8780107dd0 0x7f878019e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:13:32.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.606+0000 7f877e000640 1 --2- 172.21.15.28:0/3133628735 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8780108770 0x7f878019eb60 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f876c004770 tx=0x7f876c003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:13:32.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.606+0000 7f8777400640 1 -- 172.21.15.28:0/3133628735 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f876c03f070 con 0x7f8780108770 2024-05-12T22:13:32.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.606+0000 7f8777400640 1 -- 172.21.15.28:0/3133628735 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f876c031ca0 con 0x7f8780108770 2024-05-12T22:13:32.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.606+0000 7f877fe00640 1 -- 172.21.15.28:0/3133628735 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87801a3850 con 0x7f8780108770 2024-05-12T22:13:32.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.607+0000 7f8777400640 1 -- 172.21.15.28:0/3133628735 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f876c035030 con 0x7f8780108770 2024-05-12T22:13:32.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.607+0000 7f877fe00640 1 -- 172.21.15.28:0/3133628735 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87801ab140 con 0x7f8780108770 2024-05-12T22:13:32.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.610+0000 7f87889ef740 1 Processor -- start 2024-05-12T22:13:32.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.610+0000 7f87889ef740 1 -- start start 2024-05-12T22:13:32.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.610+0000 7f87889ef740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558589691b60 0x558589691f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:32.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.610+0000 7f8777400640 1 -- 172.21.15.28:0/3133628735 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7f876c042070 con 0x7f8780108770 2024-05-12T22:13:32.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.610+0000 7f87889ef740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558589692480 0x55858969c610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:32.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.610+0000 7f87889ef740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55858969cb50 0x55858969ef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:32.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.610+0000 7f87889ef740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5585895f6a30 con 0x55858969cb50 2024-05-12T22:13:32.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.610+0000 7f87889ef740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x55858959e700 con 0x558589691b60 2024-05-12T22:13:32.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.610+0000 7f87889ef740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5585895f29f0 con 0x558589692480 2024-05-12T22:13:32.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.611+0000 7f877f400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558589691b60 0x558589691f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:32.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.611+0000 7f877f400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558589691b60 0x558589691f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:43868/0 (socket says 172.21.15.28:43868) 2024-05-12T22:13:32.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.611+0000 7f877f400640 1 -- 172.21.15.28:0/3508502824 learned_addr learned my addr 172.21.15.28:0/3508502824 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:13:32.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.611+0000 7f877ea00640 1 --2- 172.21.15.28:0/3508502824 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558589692480 0x55858969c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:32.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.611+0000 7f877e000640 1 --2- 172.21.15.28:0/3508502824 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55858969cb50 0x55858969ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:32.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.611+0000 7f877f400640 1 -- 172.21.15.28:0/3508502824 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558589692480 msgr2=0x55858969c610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:13:32.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.611+0000 7f877f400640 1 --2- 172.21.15.28:0/3508502824 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558589692480 0x55858969c610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:32.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.611+0000 7f877f400640 1 -- 172.21.15.28:0/3508502824 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55858969cb50 msgr2=0x55858969ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:13:32.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.611+0000 7f877f400640 1 --2- 172.21.15.28:0/3508502824 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55858969cb50 0x55858969ef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:32.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.611+0000 7f877f400640 1 -- 172.21.15.28:0/3508502824 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55858930a3f0 con 0x558589691b60 2024-05-12T22:13:32.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.612+0000 7f877f400640 1 --2- 172.21.15.28:0/3508502824 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558589691b60 0x558589691f40 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f8778009fc0 tx=0x7f87780118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=46a2ea734911ed4b server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:13:32.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.612+0000 7f876be00640 1 -- 172.21.15.28:0/3508502824 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f877801e070 con 0x558589691b60 2024-05-12T22:13:32.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.612+0000 7f876be00640 1 -- 172.21.15.28:0/3508502824 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8778003c30 con 0x558589691b60 2024-05-12T22:13:32.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.612+0000 7f876be00640 1 -- 172.21.15.28:0/3508502824 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f877800fa40 con 0x558589691b60 2024-05-12T22:13:32.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.612+0000 7f8777400640 1 --2- 172.21.15.28:0/3133628735 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f87640a2d80 0x7f87640a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:32.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.612+0000 7f8777400640 1 -- 172.21.15.28:0/3133628735 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f876c0ef630 con 0x7f8780108770 2024-05-12T22:13:32.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.612+0000 7f877ea00640 1 --2- 172.21.15.28:0/3133628735 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f87640a2d80 0x7f87640a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:32.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.613+0000 7f87889ef740 1 -- 172.21.15.28:0/3508502824 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558589691b60 msgr2=0x558589691f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:13:32.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.613+0000 7f87889ef740 1 --2- 172.21.15.28:0/3508502824 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558589691b60 0x558589691f40 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f8778009fc0 tx=0x7f87780118d0 comp rx=0 tx=0).stop 2024-05-12T22:13:32.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.613+0000 7f87889ef740 1 -- 172.21.15.28:0/3508502824 shutdown_connections 2024-05-12T22:13:32.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.613+0000 7f87889ef740 1 --2- 172.21.15.28:0/3508502824 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55858969cb50 0x55858969ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:32.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.613+0000 7f87889ef740 1 --2- 172.21.15.28:0/3508502824 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558589692480 0x55858969c610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:32.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.613+0000 7f87889ef740 1 --2- 172.21.15.28:0/3508502824 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558589691b60 0x558589691f40 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:32.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.613+0000 7f87889ef740 1 -- 172.21.15.28:0/3508502824 >> 172.21.15.28:0/3508502824 conn(0x55858968de80 msgr2=0x55858968e260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:13:32.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.613+0000 7f87889ef740 1 -- 172.21.15.28:0/3508502824 shutdown_connections 2024-05-12T22:13:32.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.613+0000 7f87889ef740 1 -- 172.21.15.28:0/3508502824 wait complete. 2024-05-12T22:13:32.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.613+0000 7f877ea00640 1 --2- 172.21.15.28:0/3133628735 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f87640a2d80 0x7f87640a5240 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x558589693120 tx=0x7f8770005f90 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:13:32.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.614+0000 7f87889ef740 1 -- start start 2024-05-12T22:13:32.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.614+0000 7f87889ef740 1 Processor -- start 2024-05-12T22:13:32.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.614+0000 7f87889ef740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558589691b60 0x558589699ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:32.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.614+0000 7f87889ef740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558589692480 0x55858969a020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:32.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.614+0000 7f87889ef740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55858969cb50 0x55858969a560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:32.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.614+0000 7f87889ef740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55858969f940 con 0x55858969cb50 2024-05-12T22:13:32.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.614+0000 7f87889ef740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x5585895b1a70 con 0x558589691b60 2024-05-12T22:13:32.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.614+0000 7f87889ef740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55858969fac0 con 0x558589692480 2024-05-12T22:13:32.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.614+0000 7f87889ef740 10 client.0 _ll_register_callbacks cb 0x558589594ba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:13:32.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.615+0000 7f877ea00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558589691b60 0x558589699ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:32.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.615+0000 7f877ea00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558589691b60 0x558589699ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:43878/0 (socket says 172.21.15.28:43878) 2024-05-12T22:13:32.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.615+0000 7f877ea00640 1 -- 172.21.15.28:0/369859932 learned_addr learned my addr 172.21.15.28:0/369859932 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:13:32.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.615+0000 7f877e000640 1 --2- 172.21.15.28:0/369859932 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558589692480 0x55858969a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:32.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.615+0000 7f877f400640 1 --2- 172.21.15.28:0/369859932 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55858969cb50 0x55858969a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:32.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.615+0000 7f877ea00640 1 -- 172.21.15.28:0/369859932 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558589692480 msgr2=0x55858969a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:13:32.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.615+0000 7f877ea00640 1 --2- 172.21.15.28:0/369859932 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558589692480 0x55858969a020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:32.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.615+0000 7f877ea00640 1 -- 172.21.15.28:0/369859932 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55858969cb50 msgr2=0x55858969a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:13:32.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.615+0000 7f877ea00640 1 --2- 172.21.15.28:0/369859932 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55858969cb50 0x55858969a560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:32.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.615+0000 7f877ea00640 1 -- 172.21.15.28:0/369859932 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5585893d47d0 con 0x558589691b60 2024-05-12T22:13:32.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.615+0000 7f877ea00640 1 --2- 172.21.15.28:0/369859932 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558589691b60 0x558589699ae0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f8770009c30 tx=0x7f87700046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2df4a48178e9569 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:13:32.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.615+0000 7f8768c00640 10 client.0 ms_handle_connect on v2:172.21.15.86:3300/0 2024-05-12T22:13:32.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.616+0000 7f8768c00640 1 -- 172.21.15.28:0/369859932 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f877000fd90 con 0x558589691b60 2024-05-12T22:13:32.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.616+0000 7f87889ef740 1 -- 172.21.15.28:0/369859932 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8778003c30 con 0x558589691b60 2024-05-12T22:13:32.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.616+0000 7f8768c00640 1 -- 172.21.15.28:0/369859932 <== mon.2 v2:172.21.15.86:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f8770018030 con 0x558589691b60 2024-05-12T22:13:32.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.616+0000 7f8768c00640 1 -- 172.21.15.28:0/369859932 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f877001e920 con 0x558589691b60 2024-05-12T22:13:32.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.616+0000 7f87889ef740 10 client.26291 fetch_fsmap learned FSMap version 26 2024-05-12T22:13:32.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.616+0000 7f87889ef740 1 -- 172.21.15.28:0/369859932 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5585896abcf0 con 0x558589691b60 2024-05-12T22:13:32.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.616+0000 7f8768c00640 1 -- 172.21.15.28:0/369859932 <== mon.2 v2:172.21.15.86:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8778003c30 con 0x558589691b60 2024-05-12T22:13:32.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.616+0000 7f8768c00640 1 -- 172.21.15.28:0/369859932 <== mon.2 v2:172.21.15.86:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f877001c3a0 con 0x558589691b60 2024-05-12T22:13:32.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.617+0000 7f87889ef740 10 client.26291 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:13:32.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.617+0000 7f87889ef740 10 client.26291 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-05-12T22:13:32.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.617+0000 7f87889ef740 20 client.26291 populate_metadata read hostname 'smithi028' 2024-05-12T22:13:32.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.617+0000 7f87889ef740 1 --2- 172.21.15.28:0/369859932 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x5585896ad150 0x5585896af5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:32.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.617+0000 7f87889ef740 4 client.26291 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-12T22:13:32.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.617+0000 7f87889ef740 1 -- 172.21.15.28:0/369859932 --> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558589604160 con 0x5585896ad150 2024-05-12T22:13:32.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.617+0000 7f877e000640 1 --2- 172.21.15.28:0/369859932 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x5585896ad150 0x5585896af5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:32.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.618+0000 7f877e000640 1 --2- 172.21.15.28:0/369859932 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x5585896ad150 0x5585896af5c0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ab038adbbf17328e server_cookie=b7d459684d251b76 in_seq=0 out_seq=0 2024-05-12T22:13:32.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.618+0000 7f8768c00640 10 client.26291 ms_handle_connect on v2:172.21.15.28:6840/761183870 2024-05-12T22:13:32.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.618+0000 7f8768c00640 1 -- 172.21.15.28:0/369859932 <== mds.1 v2:172.21.15.28:6840/761183870 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558589604160 con 0x5585896ad150 2024-05-12T22:13:32.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.618+0000 7f8768c00640 10 client.26291 handle_command_reply: tid=0 2024-05-12T22:13:32.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.618+0000 7f87889ef740 1 client.26291 shutdown 2024-05-12T22:13:32.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.619+0000 7f87889ef740 1 -- 172.21.15.28:0/369859932 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558589691b60 msgr2=0x558589699ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:13:32.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.619+0000 7f87889ef740 1 --2- 172.21.15.28:0/369859932 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558589691b60 0x558589699ae0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f8770009c30 tx=0x7f87700046c0 comp rx=0 tx=0).stop 2024-05-12T22:13:32.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.619+0000 7f87889ef740 1 -- 172.21.15.28:0/369859932 shutdown_connections 2024-05-12T22:13:32.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.619+0000 7f87889ef740 1 --2- 172.21.15.28:0/369859932 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x5585896ad150 0x5585896af5c0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:32.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.619+0000 7f87889ef740 1 --2- 172.21.15.28:0/369859932 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55858969cb50 0x55858969a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:32.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.619+0000 7f87889ef740 1 --2- 172.21.15.28:0/369859932 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558589692480 0x55858969a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:32.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.619+0000 7f8768c00640 0 client.26291 ms_handle_reset on v2:172.21.15.28:6840/761183870 2024-05-12T22:13:32.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.619+0000 7f87889ef740 1 --2- 172.21.15.28:0/369859932 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558589691b60 0x558589699ae0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:32.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.619+0000 7f87889ef740 1 -- 172.21.15.28:0/369859932 >> 172.21.15.28:0/369859932 conn(0x55858968de80 msgr2=0x558589693c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:13:32.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.619+0000 7f87889ef740 1 -- 172.21.15.28:0/369859932 shutdown_connections 2024-05-12T22:13:32.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.619+0000 7f87889ef740 1 -- 172.21.15.28:0/369859932 wait complete. 2024-05-12T22:13:32.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.619+0000 7f87889ef740 20 client.26291 trim_cache size 0 max 16384 2024-05-12T22:13:32.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.634+0000 7f87889ef740 1 Processor -- start 2024-05-12T22:13:32.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.634+0000 7f87889ef740 1 -- start start 2024-05-12T22:13:32.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.635+0000 7f87889ef740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558589691b60 0x5585896a8860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:32.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.635+0000 7f87889ef740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558589692480 0x5585896942c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:32.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.635+0000 7f87889ef740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55858969cb50 0x55858969a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:32.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.635+0000 7f87889ef740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f87801b61d0 con 0x558589692480 2024-05-12T22:13:32.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.635+0000 7f87889ef740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f87801b3aa0 con 0x55858969cb50 2024-05-12T22:13:32.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.635+0000 7f87889ef740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f87801b18f0 con 0x558589691b60 2024-05-12T22:13:32.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.635+0000 7f877e000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558589692480 0x5585896942c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:32.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.635+0000 7f877ea00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558589691b60 0x5585896a8860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:32.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.635+0000 7f877e000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558589692480 0x5585896942c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50240/0 (socket says 172.21.15.28:50240) 2024-05-12T22:13:32.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.635+0000 7f877e000640 1 -- 172.21.15.28:0/1089673090 learned_addr learned my addr 172.21.15.28:0/1089673090 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:13:32.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.635+0000 7f877f400640 1 --2- 172.21.15.28:0/1089673090 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55858969cb50 0x55858969a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:32.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.635+0000 7f877e000640 1 -- 172.21.15.28:0/1089673090 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558589691b60 msgr2=0x5585896a8860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:13:32.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.635+0000 7f877e000640 1 --2- 172.21.15.28:0/1089673090 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558589691b60 0x5585896a8860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:32.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.635+0000 7f877e000640 1 -- 172.21.15.28:0/1089673090 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55858969cb50 msgr2=0x55858969a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:13:32.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.635+0000 7f877e000640 1 --2- 172.21.15.28:0/1089673090 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55858969cb50 0x55858969a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:32.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.635+0000 7f877e000640 1 -- 172.21.15.28:0/1089673090 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5585896922b0 con 0x558589692480 2024-05-12T22:13:32.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.635+0000 7f877ea00640 1 --2- 172.21.15.28:0/1089673090 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558589691b60 0x5585896a8860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:13:32.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.636+0000 7f877e000640 1 --2- 172.21.15.28:0/1089673090 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558589692480 0x5585896942c0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f876c031550 tx=0x7f876c02a480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7b5c06d9c057860 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:13:32.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.636+0000 7f876b400640 1 -- 172.21.15.28:0/1089673090 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f876c049040 con 0x558589692480 2024-05-12T22:13:32.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.636+0000 7f876b400640 1 -- 172.21.15.28:0/1089673090 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f876c031e50 con 0x558589692480 2024-05-12T22:13:32.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.636+0000 7f876b400640 1 -- 172.21.15.28:0/1089673090 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f876c03b4f0 con 0x558589692480 2024-05-12T22:13:32.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.637+0000 7f87889ef740 1 -- 172.21.15.28:0/1089673090 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558589692480 msgr2=0x5585896942c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:13:32.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.637+0000 7f87889ef740 1 --2- 172.21.15.28:0/1089673090 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558589692480 0x5585896942c0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f876c031550 tx=0x7f876c02a480 comp rx=0 tx=0).stop 2024-05-12T22:13:32.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.637+0000 7f87889ef740 1 -- 172.21.15.28:0/1089673090 shutdown_connections 2024-05-12T22:13:32.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.637+0000 7f87889ef740 1 --2- 172.21.15.28:0/1089673090 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55858969cb50 0x55858969a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:32.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.637+0000 7f87889ef740 1 --2- 172.21.15.28:0/1089673090 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558589692480 0x5585896942c0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:32.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.637+0000 7f87889ef740 1 --2- 172.21.15.28:0/1089673090 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558589691b60 0x5585896a8860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:32.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.637+0000 7f87889ef740 1 -- 172.21.15.28:0/1089673090 >> 172.21.15.28:0/1089673090 conn(0x55858968de80 msgr2=0x558589699aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:13:32.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.637+0000 7f87889ef740 1 -- 172.21.15.28:0/1089673090 shutdown_connections 2024-05-12T22:13:32.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.637+0000 7f87889ef740 1 -- 172.21.15.28:0/1089673090 wait complete. 2024-05-12T22:13:32.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.637+0000 7f87889ef740 1 -- start start 2024-05-12T22:13:32.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.638+0000 7f87889ef740 1 Processor -- start 2024-05-12T22:13:32.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.638+0000 7f87889ef740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558589691b60 0x5585896a4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:32.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.638+0000 7f87889ef740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558589692480 0x5585896a0310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:32.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.638+0000 7f87889ef740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55858969cb50 0x5585896a0850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:32.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.638+0000 7f87889ef740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f87801b1f60 con 0x558589691b60 2024-05-12T22:13:32.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.638+0000 7f87889ef740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f87801b33d0 con 0x558589692480 2024-05-12T22:13:32.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.638+0000 7f87889ef740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f87801b2630 con 0x55858969cb50 2024-05-12T22:13:32.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.638+0000 7f87889ef740 10 client.0 _ll_register_callbacks cb 0x7f877000fa70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:13:32.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.638+0000 7f877ea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558589691b60 0x5585896a4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:32.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.638+0000 7f877e000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558589692480 0x5585896a0310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:32.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.638+0000 7f877ea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558589691b60 0x5585896a4ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50248/0 (socket says 172.21.15.28:50248) 2024-05-12T22:13:32.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.638+0000 7f877ea00640 1 -- 172.21.15.28:0/1504643489 learned_addr learned my addr 172.21.15.28:0/1504643489 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:13:32.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.639+0000 7f877f400640 1 --2- 172.21.15.28:0/1504643489 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55858969cb50 0x5585896a0850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:32.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.639+0000 7f877ea00640 1 -- 172.21.15.28:0/1504643489 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55858969cb50 msgr2=0x5585896a0850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:13:32.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.639+0000 7f877ea00640 1 --2- 172.21.15.28:0/1504643489 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55858969cb50 0x5585896a0850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:32.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.639+0000 7f877ea00640 1 -- 172.21.15.28:0/1504643489 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558589692480 msgr2=0x5585896a0310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:13:32.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.639+0000 7f877ea00640 1 --2- 172.21.15.28:0/1504643489 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558589692480 0x5585896a0310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:32.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.639+0000 7f877ea00640 1 -- 172.21.15.28:0/1504643489 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5585896a0dc0 con 0x558589691b60 2024-05-12T22:13:32.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.639+0000 7f877f400640 1 --2- 172.21.15.28:0/1504643489 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55858969cb50 0x5585896a0850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:13:32.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.639+0000 7f877e000640 1 --2- 172.21.15.28:0/1504643489 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558589692480 0x5585896a0310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:13:32.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.639+0000 7f877ea00640 1 --2- 172.21.15.28:0/1504643489 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558589691b60 0x5585896a4ea0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f8770012e60 tx=0x7f8770009f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ed8bb57d28490bb server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:13:32.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.639+0000 7f8768c00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-12T22:13:32.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.639+0000 7f8768c00640 1 -- 172.21.15.28:0/1504643489 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f877001ca00 con 0x558589691b60 2024-05-12T22:13:32.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.639+0000 7f8768c00640 1 -- 172.21.15.28:0/1504643489 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f877001cba0 con 0x558589691b60 2024-05-12T22:13:32.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.639+0000 7f87889ef740 1 -- 172.21.15.28:0/1504643489 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f876c031e50 con 0x558589691b60 2024-05-12T22:13:32.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.640+0000 7f8768c00640 1 -- 172.21.15.28:0/1504643489 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f877001eac0 con 0x558589691b60 2024-05-12T22:13:32.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.640+0000 7f87889ef740 10 client.16170 fetch_fsmap learned FSMap version 26 2024-05-12T22:13:32.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.640+0000 7f87889ef740 1 -- 172.21.15.28:0/1504643489 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x558589713410 con 0x558589691b60 2024-05-12T22:13:32.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.640+0000 7f8768c00640 1 -- 172.21.15.28:0/1504643489 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f876c031e50 con 0x558589691b60 2024-05-12T22:13:32.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.640+0000 7f8768c00640 1 -- 172.21.15.28:0/1504643489 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f8770005160 con 0x558589691b60 2024-05-12T22:13:32.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.640+0000 7f87889ef740 10 client.16170 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:13:32.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.640+0000 7f87889ef740 10 client.16170 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-05-12T22:13:32.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.640+0000 7f87889ef740 20 client.16170 populate_metadata read hostname 'smithi028' 2024-05-12T22:13:32.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.640+0000 7f87889ef740 1 --2- 172.21.15.28:0/1504643489 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x558589713be0 0x558589716000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:32.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.640+0000 7f87889ef740 4 client.16170 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-12T22:13:32.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.640+0000 7f87889ef740 1 -- 172.21.15.28:0/1504643489 --> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5585896956e0 con 0x558589713be0 2024-05-12T22:13:32.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.641+0000 7f877e000640 1 --2- 172.21.15.28:0/1504643489 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x558589713be0 0x558589716000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:32.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.641+0000 7f877e000640 1 --2- 172.21.15.28:0/1504643489 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x558589713be0 0x558589716000 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=98cf4dd640c3abcb server_cookie=268b3fde65effca7 in_seq=0 out_seq=0 2024-05-12T22:13:32.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.641+0000 7f8768c00640 10 client.16170 ms_handle_connect on v2:172.21.15.28:6840/761183870 2024-05-12T22:13:32.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.641+0000 7f8768c00640 1 -- 172.21.15.28:0/1504643489 <== mds.1 v2:172.21.15.28:6840/761183870 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5585896956e0 con 0x558589713be0 2024-05-12T22:13:32.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.641+0000 7f8768c00640 10 client.16170 handle_command_reply: tid=0 2024-05-12T22:13:32.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.641+0000 7f87889ef740 1 client.16170 shutdown 2024-05-12T22:13:32.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.642+0000 7f87889ef740 1 -- 172.21.15.28:0/1504643489 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558589691b60 msgr2=0x5585896a4ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:13:32.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.642+0000 7f87889ef740 1 --2- 172.21.15.28:0/1504643489 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558589691b60 0x5585896a4ea0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f8770012e60 tx=0x7f8770009f60 comp rx=0 tx=0).stop 2024-05-12T22:13:32.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.642+0000 7f87889ef740 1 -- 172.21.15.28:0/1504643489 shutdown_connections 2024-05-12T22:13:32.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.642+0000 7f87889ef740 1 --2- 172.21.15.28:0/1504643489 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x558589713be0 0x558589716000 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:32.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.642+0000 7f87889ef740 1 --2- 172.21.15.28:0/1504643489 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55858969cb50 0x5585896a0850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:32.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.642+0000 7f87889ef740 1 --2- 172.21.15.28:0/1504643489 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558589692480 0x5585896a0310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:32.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.642+0000 7f87889ef740 1 --2- 172.21.15.28:0/1504643489 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558589691b60 0x5585896a4ea0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:32.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.642+0000 7f8768c00640 0 client.16170 ms_handle_reset on v2:172.21.15.28:6840/761183870 2024-05-12T22:13:32.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.642+0000 7f87889ef740 1 -- 172.21.15.28:0/1504643489 >> 172.21.15.28:0/1504643489 conn(0x55858968de80 msgr2=0x55858968d000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:13:32.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.642+0000 7f87889ef740 1 -- 172.21.15.28:0/1504643489 shutdown_connections 2024-05-12T22:13:32.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.642+0000 7f87889ef740 1 -- 172.21.15.28:0/1504643489 wait complete. 2024-05-12T22:13:32.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.642+0000 7f87889ef740 20 client.16170 trim_cache size 0 max 16384 2024-05-12T22:13:32.651 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-05-12T22:13:32.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.643+0000 7f877fe00640 1 -- 172.21.15.28:0/3133628735 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f87640a2d80 msgr2=0x7f87640a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:13:32.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.643+0000 7f877fe00640 1 --2- 172.21.15.28:0/3133628735 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f87640a2d80 0x7f87640a5240 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x558589693120 tx=0x7f8770005f90 comp rx=0 tx=0).stop 2024-05-12T22:13:32.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.643+0000 7f877fe00640 1 -- 172.21.15.28:0/3133628735 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8780108770 msgr2=0x7f878019eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:13:32.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.643+0000 7f877fe00640 1 --2- 172.21.15.28:0/3133628735 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8780108770 0x7f878019eb60 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f876c004770 tx=0x7f876c003c10 comp rx=0 tx=0).stop 2024-05-12T22:13:32.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.643+0000 7f877fe00640 1 -- 172.21.15.28:0/3133628735 shutdown_connections 2024-05-12T22:13:32.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.643+0000 7f877fe00640 1 --2- 172.21.15.28:0/3133628735 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f87640a2d80 0x7f87640a5240 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:32.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.643+0000 7f877fe00640 1 --2- 172.21.15.28:0/3133628735 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f878010ddf0 0x7f87801a2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:32.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.643+0000 7f877fe00640 1 --2- 172.21.15.28:0/3133628735 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8780108770 0x7f878019eb60 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:32.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.643+0000 7f877fe00640 1 --2- 172.21.15.28:0/3133628735 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8780107dd0 0x7f878019e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:32.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.643+0000 7f877fe00640 1 -- 172.21.15.28:0/3133628735 >> 172.21.15.28:0/3133628735 conn(0x7f8780077650 msgr2=0x7f87800786d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:13:32.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.643+0000 7f877fe00640 1 -- 172.21.15.28:0/3133628735 shutdown_connections 2024-05-12T22:13:32.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.643+0000 7f877fe00640 1 -- 172.21.15.28:0/3133628735 wait complete. 2024-05-12T22:13:32.663 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-05-12T22:13:32.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.882+0000 7f7335e00640 1 -- 172.21.15.28:0/2519841849 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7330107810 msgr2=0x7f7330107bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:13:32.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.882+0000 7f7335e00640 1 --2- 172.21.15.28:0/2519841849 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7330107810 0x7f7330107bf0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f7320009930 tx=0x7f732002a420 comp rx=0 tx=0).stop 2024-05-12T22:13:32.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.883+0000 7f7335e00640 1 -- 172.21.15.28:0/2519841849 shutdown_connections 2024-05-12T22:13:32.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.883+0000 7f7335e00640 1 --2- 172.21.15.28:0/2519841849 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f73301021c0 0x7f733010e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:32.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.883+0000 7f7335e00640 1 --2- 172.21.15.28:0/2519841849 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7330101800 0x7f7330101c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:32.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.883+0000 7f7335e00640 1 --2- 172.21.15.28:0/2519841849 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7330107810 0x7f7330107bf0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:32.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.883+0000 7f7335e00640 1 -- 172.21.15.28:0/2519841849 >> 172.21.15.28:0/2519841849 conn(0x7f7330078c80 msgr2=0x7f7330079090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:13:32.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.883+0000 7f7335e00640 1 -- 172.21.15.28:0/2519841849 shutdown_connections 2024-05-12T22:13:32.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.883+0000 7f7335e00640 1 -- 172.21.15.28:0/2519841849 wait complete. 2024-05-12T22:13:32.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.884+0000 7f7335e00640 1 Processor -- start 2024-05-12T22:13:32.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.884+0000 7f7335e00640 1 -- start start 2024-05-12T22:13:32.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.884+0000 7f7335e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7330101800 0x7f7330073b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:32.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.884+0000 7f7335e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f73301021c0 0x7f7330074070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:32.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.884+0000 7f7335e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7330107810 0x7f73300745b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:32.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.884+0000 7f7335e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7330111c30 con 0x7f7330107810 2024-05-12T22:13:32.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.884+0000 7f7335e00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f7330111ab0 con 0x7f73301021c0 2024-05-12T22:13:32.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.884+0000 7f7335e00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f7330111db0 con 0x7f7330101800 2024-05-12T22:13:32.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.885+0000 7f732ea00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7330101800 0x7f7330073b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:32.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.885+0000 7f732f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7330107810 0x7f73300745b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:32.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.885+0000 7f732ea00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7330101800 0x7f7330073b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:37612/0 (socket says 172.21.15.28:37612) 2024-05-12T22:13:32.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.885+0000 7f732ea00640 1 -- 172.21.15.28:0/3930158691 learned_addr learned my addr 172.21.15.28:0/3930158691 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:13:32.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.885+0000 7f7327e00640 1 --2- 172.21.15.28:0/3930158691 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f73301021c0 0x7f7330074070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:32.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.885+0000 7f732ea00640 1 -- 172.21.15.28:0/3930158691 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f73301021c0 msgr2=0x7f7330074070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:13:32.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.885+0000 7f732ea00640 1 --2- 172.21.15.28:0/3930158691 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f73301021c0 0x7f7330074070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:32.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.885+0000 7f732ea00640 1 -- 172.21.15.28:0/3930158691 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7330107810 msgr2=0x7f73300745b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:13:32.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.885+0000 7f732ea00640 1 --2- 172.21.15.28:0/3930158691 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7330107810 0x7f73300745b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:32.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.885+0000 7f732ea00640 1 -- 172.21.15.28:0/3930158691 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73301a6650 con 0x7f7330101800 2024-05-12T22:13:32.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.886+0000 7f732ea00640 1 --2- 172.21.15.28:0/3930158691 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7330101800 0x7f7330073b30 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f732002a940 tx=0x7f7320003b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:13:32.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.886+0000 7f7334a00640 1 -- 172.21.15.28:0/3930158691 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f732003f070 con 0x7f7330101800 2024-05-12T22:13:32.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.886+0000 7f7334a00640 1 -- 172.21.15.28:0/3930158691 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7320031c90 con 0x7f7330101800 2024-05-12T22:13:32.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.886+0000 7f7334a00640 1 -- 172.21.15.28:0/3930158691 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7320035030 con 0x7f7330101800 2024-05-12T22:13:32.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.886+0000 7f7335e00640 1 -- 172.21.15.28:0/3930158691 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73301a68e0 con 0x7f7330101800 2024-05-12T22:13:32.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.886+0000 7f7335e00640 1 -- 172.21.15.28:0/3930158691 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73301a6dd0 con 0x7f7330101800 2024-05-12T22:13:32.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.889+0000 7f7334a00640 1 -- 172.21.15.28:0/3930158691 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7f7320042070 con 0x7f7330101800 2024-05-12T22:13:32.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.889+0000 7f7338c2e740 1 Processor -- start 2024-05-12T22:13:32.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.890+0000 7f7338c2e740 1 -- start start 2024-05-12T22:13:32.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.890+0000 7f7338c2e740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cb820b7980 0x55cb820b7d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:32.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.890+0000 7f7338c2e740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cb820b82a0 0x55cb820c2430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:32.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.890+0000 7f7338c2e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb820c2970 0x55cb820c4d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:32.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.890+0000 7f7338c2e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55cb8201ca30 con 0x55cb820c2970 2024-05-12T22:13:32.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.890+0000 7f7338c2e740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x55cb81fc4700 con 0x55cb820b7980 2024-05-12T22:13:32.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.890+0000 7f7338c2e740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55cb820189f0 con 0x55cb820b82a0 2024-05-12T22:13:32.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.890+0000 7f7327e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb820c2970 0x55cb820c4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:32.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.890+0000 7f7327e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb820c2970 0x55cb820c4d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50276/0 (socket says 172.21.15.28:50276) 2024-05-12T22:13:32.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.890+0000 7f7327e00640 1 -- 172.21.15.28:0/1020264772 learned_addr learned my addr 172.21.15.28:0/1020264772 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:13:32.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.890+0000 7f732f400640 1 --2- 172.21.15.28:0/1020264772 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cb820b7980 0x55cb820b7d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:32.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.890+0000 7f732ea00640 1 --2- 172.21.15.28:0/1020264772 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cb820b82a0 0x55cb820c2430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:32.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.891+0000 7f7327e00640 1 -- 172.21.15.28:0/1020264772 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cb820b82a0 msgr2=0x55cb820c2430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:13:32.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.891+0000 7f7327e00640 1 --2- 172.21.15.28:0/1020264772 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cb820b82a0 0x55cb820c2430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:32.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.891+0000 7f7327e00640 1 -- 172.21.15.28:0/1020264772 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cb820b7980 msgr2=0x55cb820b7d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:13:32.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.891+0000 7f7327e00640 1 --2- 172.21.15.28:0/1020264772 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cb820b7980 0x55cb820b7d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:32.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.891+0000 7f7327e00640 1 -- 172.21.15.28:0/1020264772 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cb81d303f0 con 0x55cb820c2970 2024-05-12T22:13:32.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.891+0000 7f732f400640 1 --2- 172.21.15.28:0/1020264772 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cb820b7980 0x55cb820b7d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:13:32.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.891+0000 7f7334a00640 1 --2- 172.21.15.28:0/3930158691 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f73140a2d00 0x7f73140a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:32.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.891+0000 7f7327e00640 1 --2- 172.21.15.28:0/1020264772 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb820c2970 0x55cb820c4d60 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f7318009c90 tx=0x7f7318011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9137f4b49c677a2d server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:13:32.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.891+0000 7f7334a00640 1 -- 172.21.15.28:0/3930158691 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f73200efab0 con 0x7f7330101800 2024-05-12T22:13:32.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.891+0000 7f732f400640 1 --2- 172.21.15.28:0/3930158691 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f73140a2d00 0x7f73140a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:32.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.892+0000 7f731fe00640 1 -- 172.21.15.28:0/1020264772 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f731801e070 con 0x55cb820c2970 2024-05-12T22:13:32.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.892+0000 7f731fe00640 1 -- 172.21.15.28:0/1020264772 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7318019b10 con 0x55cb820c2970 2024-05-12T22:13:32.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.892+0000 7f731fe00640 1 -- 172.21.15.28:0/1020264772 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f731800eac0 con 0x55cb820c2970 2024-05-12T22:13:32.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.892+0000 7f7338c2e740 1 -- 172.21.15.28:0/1020264772 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb820c2970 msgr2=0x55cb820c4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:13:32.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.892+0000 7f732f400640 1 --2- 172.21.15.28:0/3930158691 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f73140a2d00 0x7f73140a51c0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f7328009fd0 tx=0x7f7328009500 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:13:32.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.892+0000 7f7338c2e740 1 --2- 172.21.15.28:0/1020264772 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb820c2970 0x55cb820c4d60 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f7318009c90 tx=0x7f7318011810 comp rx=0 tx=0).stop 2024-05-12T22:13:32.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.892+0000 7f7338c2e740 1 -- 172.21.15.28:0/1020264772 shutdown_connections 2024-05-12T22:13:32.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.892+0000 7f7338c2e740 1 --2- 172.21.15.28:0/1020264772 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb820c2970 0x55cb820c4d60 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:32.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.893+0000 7f7338c2e740 1 --2- 172.21.15.28:0/1020264772 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cb820b82a0 0x55cb820c2430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:32.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.893+0000 7f7338c2e740 1 --2- 172.21.15.28:0/1020264772 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cb820b7980 0x55cb820b7d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:32.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.893+0000 7f7338c2e740 1 -- 172.21.15.28:0/1020264772 >> 172.21.15.28:0/1020264772 conn(0x55cb820b3ca0 msgr2=0x55cb820b4080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:13:32.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.893+0000 7f7338c2e740 1 -- 172.21.15.28:0/1020264772 shutdown_connections 2024-05-12T22:13:32.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.893+0000 7f7338c2e740 1 -- 172.21.15.28:0/1020264772 wait complete. 2024-05-12T22:13:32.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.893+0000 7f7338c2e740 1 -- start start 2024-05-12T22:13:32.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.893+0000 7f7338c2e740 1 Processor -- start 2024-05-12T22:13:32.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.894+0000 7f7338c2e740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cb820b7980 0x55cb820bf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:32.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.894+0000 7f7338c2e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb820b82a0 0x55cb820bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:32.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.894+0000 7f7338c2e740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cb820c2970 0x55cb820c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:32.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.894+0000 7f7338c2e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55cb820c5760 con 0x55cb820b82a0 2024-05-12T22:13:32.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.894+0000 7f7338c2e740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x55cb81fd7a70 con 0x55cb820c2970 2024-05-12T22:13:32.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.894+0000 7f7338c2e740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55cb820c58e0 con 0x55cb820b7980 2024-05-12T22:13:32.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.894+0000 7f7338c2e740 10 client.0 _ll_register_callbacks cb 0x55cb81fbaba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:13:32.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.894+0000 7f7327e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb820b82a0 0x55cb820bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:32.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.894+0000 7f732ea00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cb820b7980 0x55cb820bf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:32.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.894+0000 7f7327e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb820b82a0 0x55cb820bfec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50288/0 (socket says 172.21.15.28:50288) 2024-05-12T22:13:32.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.894+0000 7f7327e00640 1 -- 172.21.15.28:0/2502793482 learned_addr learned my addr 172.21.15.28:0/2502793482 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:13:32.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.894+0000 7f732f400640 1 --2- 172.21.15.28:0/2502793482 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cb820c2970 0x55cb820c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:32.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.895+0000 7f732ea00640 1 -- 172.21.15.28:0/2502793482 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cb820c2970 msgr2=0x55cb820c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:13:32.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.895+0000 7f732ea00640 1 --2- 172.21.15.28:0/2502793482 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cb820c2970 0x55cb820c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:32.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.895+0000 7f732ea00640 1 -- 172.21.15.28:0/2502793482 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb820b82a0 msgr2=0x55cb820bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:13:32.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.895+0000 7f732ea00640 1 --2- 172.21.15.28:0/2502793482 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb820b82a0 0x55cb820bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:32.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.895+0000 7f732ea00640 1 -- 172.21.15.28:0/2502793482 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cb81dfa7d0 con 0x55cb820b7980 2024-05-12T22:13:32.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.895+0000 7f7327e00640 1 --2- 172.21.15.28:0/2502793482 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb820b82a0 0x55cb820bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:13:32.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.895+0000 7f732ea00640 1 --2- 172.21.15.28:0/2502793482 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cb820b7980 0x55cb820bf980 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f73200ed1d0 tx=0x7f732002a420 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8997383fa7811988 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:13:32.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.895+0000 7f731cc00640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-12T22:13:32.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.895+0000 7f731cc00640 1 -- 172.21.15.28:0/2502793482 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7320049030 con 0x55cb820b7980 2024-05-12T22:13:32.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.895+0000 7f731cc00640 1 -- 172.21.15.28:0/2502793482 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f73200ed450 con 0x55cb820b7980 2024-05-12T22:13:32.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.895+0000 7f7338c2e740 1 -- 172.21.15.28:0/2502793482 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7318019b10 con 0x55cb820b7980 2024-05-12T22:13:32.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.895+0000 7f731cc00640 1 -- 172.21.15.28:0/2502793482 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f73200bbe20 con 0x55cb820b7980 2024-05-12T22:13:32.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.895+0000 7f7338c2e740 10 client.26269 fetch_fsmap learned FSMap version 26 2024-05-12T22:13:32.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.895+0000 7f7338c2e740 1 -- 172.21.15.28:0/2502793482 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55cb820c5810 con 0x55cb820b7980 2024-05-12T22:13:32.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.896+0000 7f731cc00640 1 -- 172.21.15.28:0/2502793482 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7318019b10 con 0x55cb820b7980 2024-05-12T22:13:32.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.896+0000 7f731cc00640 1 -- 172.21.15.28:0/2502793482 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f73200dabe0 con 0x55cb820b7980 2024-05-12T22:13:32.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.896+0000 7f7338c2e740 10 client.26269 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:13:32.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.896+0000 7f7338c2e740 10 client.26269 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:13:32.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.896+0000 7f7338c2e740 20 client.26269 populate_metadata read hostname 'smithi028' 2024-05-12T22:13:32.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.896+0000 7f7338c2e740 1 --2- 172.21.15.28:0/2502793482 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55cb820d2f70 0x55cb820d5400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:32.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.896+0000 7f7338c2e740 4 client.26269 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-12T22:13:32.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.896+0000 7f7338c2e740 1 -- 172.21.15.28:0/2502793482 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cb8202a160 con 0x55cb820d2f70 2024-05-12T22:13:32.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.896+0000 7f7327e00640 1 --2- 172.21.15.28:0/2502793482 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55cb820d2f70 0x55cb820d5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:32.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.897+0000 7f7327e00640 1 --2- 172.21.15.28:0/2502793482 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55cb820d2f70 0x55cb820d5400 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1637e41d85e36178 server_cookie=6364288efbb69488 in_seq=0 out_seq=0 2024-05-12T22:13:32.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.897+0000 7f731cc00640 10 client.26269 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:13:32.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.898+0000 7f731cc00640 1 -- 172.21.15.28:0/2502793482 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55cb8202a160 con 0x55cb820d2f70 2024-05-12T22:13:32.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.898+0000 7f731cc00640 10 client.26269 handle_command_reply: tid=0 2024-05-12T22:13:32.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.898+0000 7f7338c2e740 1 client.26269 shutdown 2024-05-12T22:13:32.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.898+0000 7f7338c2e740 1 -- 172.21.15.28:0/2502793482 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cb820b7980 msgr2=0x55cb820bf980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:13:32.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.898+0000 7f7338c2e740 1 --2- 172.21.15.28:0/2502793482 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cb820b7980 0x55cb820bf980 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f73200ed1d0 tx=0x7f732002a420 comp rx=0 tx=0).stop 2024-05-12T22:13:32.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.898+0000 7f7338c2e740 1 -- 172.21.15.28:0/2502793482 shutdown_connections 2024-05-12T22:13:32.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.898+0000 7f7338c2e740 1 --2- 172.21.15.28:0/2502793482 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55cb820d2f70 0x55cb820d5400 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:32.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.898+0000 7f7338c2e740 1 --2- 172.21.15.28:0/2502793482 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cb820c2970 0x55cb820c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:32.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.898+0000 7f7338c2e740 1 --2- 172.21.15.28:0/2502793482 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb820b82a0 0x55cb820bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:32.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.898+0000 7f7338c2e740 1 --2- 172.21.15.28:0/2502793482 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cb820b7980 0x55cb820bf980 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:32.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.898+0000 7f7338c2e740 1 -- 172.21.15.28:0/2502793482 >> 172.21.15.28:0/2502793482 conn(0x55cb820b3ca0 msgr2=0x55cb820b9aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:13:32.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.898+0000 7f731cc00640 0 client.26269 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:13:32.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.899+0000 7f7338c2e740 1 -- 172.21.15.28:0/2502793482 shutdown_connections 2024-05-12T22:13:32.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.899+0000 7f7338c2e740 1 -- 172.21.15.28:0/2502793482 wait complete. 2024-05-12T22:13:32.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.899+0000 7f7338c2e740 20 client.26269 trim_cache size 0 max 16384 2024-05-12T22:13:32.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.914+0000 7f7338c2e740 1 Processor -- start 2024-05-12T22:13:32.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.914+0000 7f7338c2e740 1 -- start start 2024-05-12T22:13:32.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.914+0000 7f7338c2e740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cb820b7980 0x55cb820ce620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:32.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.914+0000 7f7338c2e740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cb820b82a0 0x55cb820b2c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:32.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.914+0000 7f7338c2e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb820c2970 0x55cb820daca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:32.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.914+0000 7f7338c2e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f73301b1f60 con 0x55cb820c2970 2024-05-12T22:13:32.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.914+0000 7f7338c2e740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f73301af830 con 0x55cb820b7980 2024-05-12T22:13:32.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.914+0000 7f7338c2e740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f73301ad680 con 0x55cb820b82a0 2024-05-12T22:13:32.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.914+0000 7f732f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb820c2970 0x55cb820daca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:32.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.914+0000 7f732f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb820c2970 0x55cb820daca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50302/0 (socket says 172.21.15.28:50302) 2024-05-12T22:13:32.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.914+0000 7f732f400640 1 -- 172.21.15.28:0/2634565396 learned_addr learned my addr 172.21.15.28:0/2634565396 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:13:32.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.915+0000 7f732ea00640 1 --2- 172.21.15.28:0/2634565396 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cb820b7980 0x55cb820ce620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:32.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.915+0000 7f7327e00640 1 --2- 172.21.15.28:0/2634565396 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cb820b82a0 0x55cb820b2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:32.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.915+0000 7f732f400640 1 -- 172.21.15.28:0/2634565396 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cb820b82a0 msgr2=0x55cb820b2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:13:32.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.915+0000 7f732f400640 1 --2- 172.21.15.28:0/2634565396 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cb820b82a0 0x55cb820b2c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:32.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.915+0000 7f732f400640 1 -- 172.21.15.28:0/2634565396 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cb820b7980 msgr2=0x55cb820ce620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:13:32.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.915+0000 7f732f400640 1 --2- 172.21.15.28:0/2634565396 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cb820b7980 0x55cb820ce620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:32.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.915+0000 7f732f400640 1 -- 172.21.15.28:0/2634565396 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cb820c5640 con 0x55cb820c2970 2024-05-12T22:13:32.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.915+0000 7f732f400640 1 --2- 172.21.15.28:0/2634565396 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb820c2970 0x55cb820daca0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f732800f640 tx=0x7f7328019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7d34a60c225ecca server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:13:32.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.916+0000 7f731f400640 1 -- 172.21.15.28:0/2634565396 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7328004b20 con 0x55cb820c2970 2024-05-12T22:13:32.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.916+0000 7f731f400640 1 -- 172.21.15.28:0/2634565396 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7328003e50 con 0x55cb820c2970 2024-05-12T22:13:32.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.916+0000 7f731f400640 1 -- 172.21.15.28:0/2634565396 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73280105b0 con 0x55cb820c2970 2024-05-12T22:13:32.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.916+0000 7f7338c2e740 1 -- 172.21.15.28:0/2634565396 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb820c2970 msgr2=0x55cb820daca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:13:32.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.916+0000 7f7338c2e740 1 --2- 172.21.15.28:0/2634565396 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb820c2970 0x55cb820daca0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f732800f640 tx=0x7f7328019000 comp rx=0 tx=0).stop 2024-05-12T22:13:32.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.916+0000 7f7338c2e740 1 -- 172.21.15.28:0/2634565396 shutdown_connections 2024-05-12T22:13:32.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.916+0000 7f7338c2e740 1 --2- 172.21.15.28:0/2634565396 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb820c2970 0x55cb820daca0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:32.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.916+0000 7f7338c2e740 1 --2- 172.21.15.28:0/2634565396 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cb820b82a0 0x55cb820b2c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:32.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.916+0000 7f7338c2e740 1 --2- 172.21.15.28:0/2634565396 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cb820b7980 0x55cb820ce620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:32.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.916+0000 7f7338c2e740 1 -- 172.21.15.28:0/2634565396 >> 172.21.15.28:0/2634565396 conn(0x55cb820b3ca0 msgr2=0x55cb820b2530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:13:32.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.916+0000 7f7338c2e740 1 -- 172.21.15.28:0/2634565396 shutdown_connections 2024-05-12T22:13:32.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.917+0000 7f7338c2e740 1 -- 172.21.15.28:0/2634565396 wait complete. 2024-05-12T22:13:32.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.917+0000 7f7338c2e740 1 -- start start 2024-05-12T22:13:32.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.917+0000 7f7338c2e740 1 Processor -- start 2024-05-12T22:13:32.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.918+0000 7f7338c2e740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cb820b7980 0x55cb820cab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:32.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.918+0000 7f7338c2e740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cb820b82a0 0x55cb820cb070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:32.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.918+0000 7f7338c2e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb820c2970 0x55cb820c7bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:32.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.918+0000 7f7338c2e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f73301adcf0 con 0x55cb820c2970 2024-05-12T22:13:32.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.918+0000 7f7338c2e740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f73301af160 con 0x55cb820b82a0 2024-05-12T22:13:32.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.918+0000 7f7338c2e740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f73301ae3c0 con 0x55cb820b7980 2024-05-12T22:13:32.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.918+0000 7f7338c2e740 10 client.0 _ll_register_callbacks cb 0x7f7320005de0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:13:32.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.918+0000 7f732f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb820c2970 0x55cb820c7bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:32.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.918+0000 7f732f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb820c2970 0x55cb820c7bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50306/0 (socket says 172.21.15.28:50306) 2024-05-12T22:13:32.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.918+0000 7f732f400640 1 -- 172.21.15.28:0/2909482522 learned_addr learned my addr 172.21.15.28:0/2909482522 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:13:32.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.918+0000 7f732ea00640 1 --2- 172.21.15.28:0/2909482522 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cb820b7980 0x55cb820cab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:32.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.918+0000 7f7327e00640 1 --2- 172.21.15.28:0/2909482522 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cb820b82a0 0x55cb820cb070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:32.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.918+0000 7f732f400640 1 -- 172.21.15.28:0/2909482522 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cb820b7980 msgr2=0x55cb820cab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:13:32.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.918+0000 7f732f400640 1 --2- 172.21.15.28:0/2909482522 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cb820b7980 0x55cb820cab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:32.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.918+0000 7f732f400640 1 -- 172.21.15.28:0/2909482522 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cb820b82a0 msgr2=0x55cb820cb070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:13:32.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.918+0000 7f732f400640 1 --2- 172.21.15.28:0/2909482522 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cb820b82a0 0x55cb820cb070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:32.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.918+0000 7f732f400640 1 -- 172.21.15.28:0/2909482522 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cb820c8420 con 0x55cb820c2970 2024-05-12T22:13:32.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.919+0000 7f732f400640 1 --2- 172.21.15.28:0/2909482522 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb820c2970 0x55cb820c7bf0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f732800f610 tx=0x7f73280042a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d0a31455c2a9718 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:13:32.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.919+0000 7f731cc00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-12T22:13:32.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.919+0000 7f731cc00640 1 -- 172.21.15.28:0/2909482522 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7328037060 con 0x55cb820c2970 2024-05-12T22:13:32.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.919+0000 7f731cc00640 1 -- 172.21.15.28:0/2909482522 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f7328003970 con 0x55cb820c2970 2024-05-12T22:13:32.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.919+0000 7f7338c2e740 1 -- 172.21.15.28:0/2909482522 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7328003e50 con 0x55cb820c2970 2024-05-12T22:13:32.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.919+0000 7f731cc00640 1 -- 172.21.15.28:0/2909482522 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f7328004800 con 0x55cb820c2970 2024-05-12T22:13:32.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.919+0000 7f7338c2e740 10 client.16188 fetch_fsmap learned FSMap version 26 2024-05-12T22:13:32.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.919+0000 7f7338c2e740 1 -- 172.21.15.28:0/2909482522 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55cb82142910 con 0x55cb820c2970 2024-05-12T22:13:32.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.920+0000 7f731cc00640 1 -- 172.21.15.28:0/2909482522 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7328003e50 con 0x55cb820c2970 2024-05-12T22:13:32.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.920+0000 7f731cc00640 1 -- 172.21.15.28:0/2909482522 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f73280053d0 con 0x55cb820c2970 2024-05-12T22:13:32.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.920+0000 7f7338c2e740 10 client.16188 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:13:32.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.920+0000 7f7338c2e740 10 client.16188 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:13:32.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.920+0000 7f7338c2e740 20 client.16188 populate_metadata read hostname 'smithi028' 2024-05-12T22:13:32.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.920+0000 7f7338c2e740 1 --2- 172.21.15.28:0/2909482522 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55cb82142f70 0x55cb82145390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:32.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.920+0000 7f7338c2e740 4 client.16188 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-12T22:13:32.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.920+0000 7f7338c2e740 1 -- 172.21.15.28:0/2909482522 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55cb820244b0 con 0x55cb82142f70 2024-05-12T22:13:32.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.920+0000 7f732ea00640 1 --2- 172.21.15.28:0/2909482522 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55cb82142f70 0x55cb82145390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:32.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.921+0000 7f732ea00640 1 --2- 172.21.15.28:0/2909482522 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55cb82142f70 0x55cb82145390 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cf80838abe798c9 server_cookie=664463290e8455ff in_seq=0 out_seq=0 2024-05-12T22:13:32.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.921+0000 7f731cc00640 10 client.16188 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:13:32.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.921+0000 7f731cc00640 1 -- 172.21.15.28:0/2909482522 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55cb820244b0 con 0x55cb82142f70 2024-05-12T22:13:32.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.921+0000 7f731cc00640 10 client.16188 handle_command_reply: tid=0 2024-05-12T22:13:32.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.921+0000 7f7338c2e740 1 client.16188 shutdown 2024-05-12T22:13:32.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.921+0000 7f7338c2e740 1 -- 172.21.15.28:0/2909482522 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb820c2970 msgr2=0x55cb820c7bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:13:32.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.921+0000 7f7338c2e740 1 --2- 172.21.15.28:0/2909482522 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb820c2970 0x55cb820c7bf0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f732800f610 tx=0x7f73280042a0 comp rx=0 tx=0).stop 2024-05-12T22:13:32.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.922+0000 7f7338c2e740 1 -- 172.21.15.28:0/2909482522 shutdown_connections 2024-05-12T22:13:32.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.922+0000 7f7338c2e740 1 --2- 172.21.15.28:0/2909482522 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55cb82142f70 0x55cb82145390 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:32.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.922+0000 7f7338c2e740 1 --2- 172.21.15.28:0/2909482522 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb820c2970 0x55cb820c7bf0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:32.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.922+0000 7f7338c2e740 1 --2- 172.21.15.28:0/2909482522 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cb820b82a0 0x55cb820cb070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:32.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.922+0000 7f7338c2e740 1 --2- 172.21.15.28:0/2909482522 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cb820b7980 0x55cb820cab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:32.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.922+0000 7f7338c2e740 1 -- 172.21.15.28:0/2909482522 >> 172.21.15.28:0/2909482522 conn(0x55cb820b3ca0 msgr2=0x55cb820b2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:13:32.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.922+0000 7f731cc00640 0 client.16188 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:13:32.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.922+0000 7f7338c2e740 1 -- 172.21.15.28:0/2909482522 shutdown_connections 2024-05-12T22:13:32.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.922+0000 7f7338c2e740 1 -- 172.21.15.28:0/2909482522 wait complete. 2024-05-12T22:13:32.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.922+0000 7f7338c2e740 20 client.16188 trim_cache size 0 max 16384 2024-05-12T22:13:32.931 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-05-12T22:13:32.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.923+0000 7f7335e00640 1 -- 172.21.15.28:0/3930158691 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f73140a2d00 msgr2=0x7f73140a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:13:32.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.923+0000 7f7335e00640 1 --2- 172.21.15.28:0/3930158691 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f73140a2d00 0x7f73140a51c0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f7328009fd0 tx=0x7f7328009500 comp rx=0 tx=0).stop 2024-05-12T22:13:32.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.923+0000 7f7335e00640 1 -- 172.21.15.28:0/3930158691 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7330101800 msgr2=0x7f7330073b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:13:32.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.923+0000 7f7335e00640 1 --2- 172.21.15.28:0/3930158691 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7330101800 0x7f7330073b30 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f732002a940 tx=0x7f7320003b20 comp rx=0 tx=0).stop 2024-05-12T22:13:32.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.923+0000 7f7335e00640 1 -- 172.21.15.28:0/3930158691 shutdown_connections 2024-05-12T22:13:32.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.923+0000 7f7335e00640 1 --2- 172.21.15.28:0/3930158691 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f73140a2d00 0x7f73140a51c0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:32.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.923+0000 7f7335e00640 1 --2- 172.21.15.28:0/3930158691 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7330107810 0x7f73300745b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:32.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.923+0000 7f7335e00640 1 --2- 172.21.15.28:0/3930158691 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f73301021c0 0x7f7330074070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:32.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.923+0000 7f7335e00640 1 --2- 172.21.15.28:0/3930158691 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7330101800 0x7f7330073b30 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:32.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.923+0000 7f7335e00640 1 -- 172.21.15.28:0/3930158691 >> 172.21.15.28:0/3930158691 conn(0x7f7330078c80 msgr2=0x7f73300ff290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:13:32.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.923+0000 7f7335e00640 1 -- 172.21.15.28:0/3930158691 shutdown_connections 2024-05-12T22:13:32.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:32.923+0000 7f7335e00640 1 -- 172.21.15.28:0/3930158691 wait complete. 2024-05-12T22:13:32.940 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:13:32 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/2918967033' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:13:33.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:13:32 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/2918967033' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:13:33.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:13:32 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/2918967033' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:13:33.944 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-12T22:13:33.945 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-12T22:13:33.945 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-05-12T22:13:34.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:13:33 smithi186 ceph-mon[34785]: pgmap v140: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:13:34.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.179+0000 7fe889c00640 1 -- 172.21.15.28:0/544813643 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe884107860 msgr2=0x7fe884107c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:13:34.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.179+0000 7fe889c00640 1 --2- 172.21.15.28:0/544813643 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe884107860 0x7fe884107c40 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fe874009930 tx=0x7fe87402a440 comp rx=0 tx=0).stop 2024-05-12T22:13:34.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.179+0000 7fe889c00640 1 -- 172.21.15.28:0/544813643 shutdown_connections 2024-05-12T22:13:34.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.179+0000 7fe889c00640 1 --2- 172.21.15.28:0/544813643 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe884102210 0x7fe88410e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:34.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.179+0000 7fe889c00640 1 --2- 172.21.15.28:0/544813643 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe884101850 0x7fe884101cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:34.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.179+0000 7fe889c00640 1 --2- 172.21.15.28:0/544813643 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe884107860 0x7fe884107c40 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:34.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.179+0000 7fe889c00640 1 -- 172.21.15.28:0/544813643 >> 172.21.15.28:0/544813643 conn(0x7fe8840fd340 msgr2=0x7fe8840ff760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:13:34.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.179+0000 7fe889c00640 1 -- 172.21.15.28:0/544813643 shutdown_connections 2024-05-12T22:13:34.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.179+0000 7fe889c00640 1 -- 172.21.15.28:0/544813643 wait complete. 2024-05-12T22:13:34.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.180+0000 7fe889c00640 1 Processor -- start 2024-05-12T22:13:34.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.180+0000 7fe889c00640 1 -- start start 2024-05-12T22:13:34.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.180+0000 7fe889c00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe884101850 0x7fe884073b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:34.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.180+0000 7fe889c00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe884102210 0x7fe884074070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:34.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.181+0000 7fe889c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe884107860 0x7fe8840745b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:34.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.181+0000 7fe889c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe884111cc0 con 0x7fe884107860 2024-05-12T22:13:34.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.181+0000 7fe889c00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fe884111b40 con 0x7fe884102210 2024-05-12T22:13:34.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.181+0000 7fe889c00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe884111e40 con 0x7fe884101850 2024-05-12T22:13:34.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.181+0000 7fe883400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe884107860 0x7fe8840745b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:34.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.181+0000 7fe882a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe884101850 0x7fe884073b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:34.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.181+0000 7fe882a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe884101850 0x7fe884073b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:37670/0 (socket says 172.21.15.28:37670) 2024-05-12T22:13:34.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.181+0000 7fe882a00640 1 -- 172.21.15.28:0/3220494230 learned_addr learned my addr 172.21.15.28:0/3220494230 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:13:34.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.181+0000 7fe882000640 1 --2- 172.21.15.28:0/3220494230 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe884102210 0x7fe884074070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:34.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.181+0000 7fe882a00640 1 -- 172.21.15.28:0/3220494230 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe884102210 msgr2=0x7fe884074070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:13:34.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.181+0000 7fe882a00640 1 --2- 172.21.15.28:0/3220494230 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe884102210 0x7fe884074070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:34.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.181+0000 7fe882a00640 1 -- 172.21.15.28:0/3220494230 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe884107860 msgr2=0x7fe8840745b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:13:34.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.181+0000 7fe882a00640 1 --2- 172.21.15.28:0/3220494230 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe884107860 0x7fe8840745b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:34.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.181+0000 7fe882a00640 1 -- 172.21.15.28:0/3220494230 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8841a66a0 con 0x7fe884101850 2024-05-12T22:13:34.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.181+0000 7fe883400640 1 --2- 172.21.15.28:0/3220494230 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe884107860 0x7fe8840745b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:13:34.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.182+0000 7fe882a00640 1 --2- 172.21.15.28:0/3220494230 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe884101850 0x7fe884073b30 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fe87402a960 tx=0x7fe874003b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:13:34.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.182+0000 7fe87b400640 1 -- 172.21.15.28:0/3220494230 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe87403f070 con 0x7fe884101850 2024-05-12T22:13:34.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.182+0000 7fe889c00640 1 -- 172.21.15.28:0/3220494230 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8841a6930 con 0x7fe884101850 2024-05-12T22:13:34.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.182+0000 7fe87b400640 1 -- 172.21.15.28:0/3220494230 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe874031c90 con 0x7fe884101850 2024-05-12T22:13:34.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.182+0000 7fe889c00640 1 -- 172.21.15.28:0/3220494230 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8841a6e70 con 0x7fe884101850 2024-05-12T22:13:34.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.182+0000 7fe87b400640 1 -- 172.21.15.28:0/3220494230 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe874049060 con 0x7fe884101850 2024-05-12T22:13:34.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.186+0000 7fe88ca1e740 1 Processor -- start 2024-05-12T22:13:34.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.186+0000 7fe87b400640 1 -- 172.21.15.28:0/3220494230 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7fe874042070 con 0x7fe884101850 2024-05-12T22:13:34.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.186+0000 7fe88ca1e740 1 -- start start 2024-05-12T22:13:34.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.186+0000 7fe88ca1e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564a661e32c0 0x564a661e36a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:34.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.186+0000 7fe88ca1e740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564a661e3c70 0x564a661edd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:34.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.186+0000 7fe88ca1e740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564a661ee2e0 0x564a661f0730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:34.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.186+0000 7fe88ca1e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x564a661477c0 con 0x564a661e32c0 2024-05-12T22:13:34.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.186+0000 7fe88ca1e740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x564a660ef790 con 0x564a661e3c70 2024-05-12T22:13:34.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.186+0000 7fe88ca1e740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x564a66143780 con 0x564a661ee2e0 2024-05-12T22:13:34.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.187+0000 7fe883400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564a661e32c0 0x564a661e36a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:34.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.187+0000 7fe882a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564a661e3c70 0x564a661edd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:34.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.187+0000 7fe883400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564a661e32c0 0x564a661e36a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50322/0 (socket says 172.21.15.28:50322) 2024-05-12T22:13:34.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.187+0000 7fe883400640 1 -- 172.21.15.28:0/43927622 learned_addr learned my addr 172.21.15.28:0/43927622 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:13:34.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.187+0000 7fe882000640 1 --2- 172.21.15.28:0/43927622 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564a661ee2e0 0x564a661f0730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:34.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.187+0000 7fe883400640 1 -- 172.21.15.28:0/43927622 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564a661ee2e0 msgr2=0x564a661f0730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:13:34.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.187+0000 7fe883400640 1 --2- 172.21.15.28:0/43927622 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564a661ee2e0 0x564a661f0730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:34.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.187+0000 7fe883400640 1 -- 172.21.15.28:0/43927622 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564a661e3c70 msgr2=0x564a661edd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:13:34.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.187+0000 7fe883400640 1 --2- 172.21.15.28:0/43927622 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564a661e3c70 0x564a661edd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:34.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.187+0000 7fe883400640 1 -- 172.21.15.28:0/43927622 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564a6610f6a0 con 0x564a661e32c0 2024-05-12T22:13:34.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.188+0000 7fe87b400640 1 --2- 172.21.15.28:0/3220494230 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fe8680a2d80 0x7fe8680a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:34.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.188+0000 7fe883400640 1 --2- 172.21.15.28:0/43927622 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564a661e32c0 0x564a661e36a0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fe87c010710 tx=0x7fe87c010bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6a92f9f20498da3 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:13:34.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.188+0000 7fe87b400640 1 -- 172.21.15.28:0/3220494230 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7fe8740ef790 con 0x7fe884101850 2024-05-12T22:13:34.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.188+0000 7fe882a00640 1 --2- 172.21.15.28:0/3220494230 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fe8680a2d80 0x7fe8680a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:34.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.188+0000 7fe86fe00640 1 -- 172.21.15.28:0/43927622 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe87c00e8d0 con 0x564a661e32c0 2024-05-12T22:13:34.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.188+0000 7fe86fe00640 1 -- 172.21.15.28:0/43927622 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe87c012450 con 0x564a661e32c0 2024-05-12T22:13:34.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.188+0000 7fe86fe00640 1 -- 172.21.15.28:0/43927622 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe87c01c650 con 0x564a661e32c0 2024-05-12T22:13:34.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.189+0000 7fe88ca1e740 1 -- 172.21.15.28:0/43927622 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564a661e32c0 msgr2=0x564a661e36a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:13:34.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.189+0000 7fe88ca1e740 1 --2- 172.21.15.28:0/43927622 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564a661e32c0 0x564a661e36a0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fe87c010710 tx=0x7fe87c010bd0 comp rx=0 tx=0).stop 2024-05-12T22:13:34.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.189+0000 7fe882a00640 1 --2- 172.21.15.28:0/3220494230 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fe8680a2d80 0x7fe8680a5240 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x564a661e4880 tx=0x7fe8740f5530 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:13:34.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.189+0000 7fe88ca1e740 1 -- 172.21.15.28:0/43927622 shutdown_connections 2024-05-12T22:13:34.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.189+0000 7fe88ca1e740 1 --2- 172.21.15.28:0/43927622 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564a661ee2e0 0x564a661f0730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:34.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.189+0000 7fe88ca1e740 1 --2- 172.21.15.28:0/43927622 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564a661e3c70 0x564a661edd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:34.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.189+0000 7fe88ca1e740 1 --2- 172.21.15.28:0/43927622 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564a661e32c0 0x564a661e36a0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:34.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.189+0000 7fe88ca1e740 1 -- 172.21.15.28:0/43927622 >> 172.21.15.28:0/43927622 conn(0x564a661df370 msgr2=0x564a661df750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:13:34.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.189+0000 7fe88ca1e740 1 -- 172.21.15.28:0/43927622 shutdown_connections 2024-05-12T22:13:34.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.189+0000 7fe88ca1e740 1 -- 172.21.15.28:0/43927622 wait complete. 2024-05-12T22:13:34.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.189+0000 7fe88ca1e740 1 -- start start 2024-05-12T22:13:34.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.190+0000 7fe88ca1e740 1 Processor -- start 2024-05-12T22:13:34.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.190+0000 7fe88ca1e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564a661e32c0 0x564a661eb210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:34.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.190+0000 7fe88ca1e740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564a661e3c70 0x564a661eb750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:34.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.190+0000 7fe88ca1e740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564a661ee2e0 0x564a661ebc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:34.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.190+0000 7fe88ca1e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x564a661f12f0 con 0x564a661e32c0 2024-05-12T22:13:34.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.190+0000 7fe88ca1e740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x564a66102a90 con 0x564a661ee2e0 2024-05-12T22:13:34.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.190+0000 7fe88ca1e740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x564a661f1470 con 0x564a661e3c70 2024-05-12T22:13:34.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.190+0000 7fe88ca1e740 10 client.0 _ll_register_callbacks cb 0x564a6613daf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:13:34.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.190+0000 7fe883400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564a661e32c0 0x564a661eb210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:34.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.190+0000 7fe883400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564a661e32c0 0x564a661eb210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50336/0 (socket says 172.21.15.28:50336) 2024-05-12T22:13:34.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.190+0000 7fe883400640 1 -- 172.21.15.28:0/373927876 learned_addr learned my addr 172.21.15.28:0/373927876 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:13:34.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.190+0000 7fe882000640 1 --2- 172.21.15.28:0/373927876 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564a661e3c70 0x564a661eb750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:34.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.191+0000 7fe883400640 1 --2- 172.21.15.28:0/373927876 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564a661ee2e0 0x564a661ebc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:34.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.191+0000 7fe882000640 1 -- 172.21.15.28:0/373927876 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564a661ee2e0 msgr2=0x564a661ebc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:13:34.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.191+0000 7fe882000640 1 --2- 172.21.15.28:0/373927876 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564a661ee2e0 0x564a661ebc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:34.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.191+0000 7fe882000640 1 -- 172.21.15.28:0/373927876 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564a661e32c0 msgr2=0x564a661eb210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:13:34.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.191+0000 7fe882000640 1 --2- 172.21.15.28:0/373927876 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564a661e32c0 0x564a661eb210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:34.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.191+0000 7fe882000640 1 -- 172.21.15.28:0/373927876 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564a65f237e0 con 0x564a661e3c70 2024-05-12T22:13:34.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.191+0000 7fe882000640 1 --2- 172.21.15.28:0/373927876 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564a661e3c70 0x564a661eb750 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fe870012700 tx=0x7fe870012bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=49f774b50241adca server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:13:34.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.191+0000 7fe86cc00640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-12T22:13:34.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.192+0000 7fe86cc00640 1 -- 172.21.15.28:0/373927876 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe870004050 con 0x564a661e3c70 2024-05-12T22:13:34.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.192+0000 7fe88ca1e740 1 -- 172.21.15.28:0/373927876 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe87c012450 con 0x564a661e3c70 2024-05-12T22:13:34.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.192+0000 7fe86cc00640 1 -- 172.21.15.28:0/373927876 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7fe8700041f0 con 0x564a661e3c70 2024-05-12T22:13:34.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.192+0000 7fe86cc00640 1 -- 172.21.15.28:0/373927876 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fe870004430 con 0x564a661e3c70 2024-05-12T22:13:34.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.192+0000 7fe88ca1e740 10 client.26287 fetch_fsmap learned FSMap version 26 2024-05-12T22:13:34.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.192+0000 7fe88ca1e740 1 -- 172.21.15.28:0/373927876 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x564a65f83f70 con 0x564a661e3c70 2024-05-12T22:13:34.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.192+0000 7fe86cc00640 1 -- 172.21.15.28:0/373927876 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe87c012450 con 0x564a661e3c70 2024-05-12T22:13:34.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.192+0000 7fe86cc00640 1 -- 172.21.15.28:0/373927876 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fe870003be0 con 0x564a661e3c70 2024-05-12T22:13:34.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.193+0000 7fe88ca1e740 10 client.26287 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:13:34.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.193+0000 7fe88ca1e740 10 client.26287 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:13:34.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.193+0000 7fe88ca1e740 20 client.26287 populate_metadata read hostname 'smithi028' 2024-05-12T22:13:34.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.193+0000 7fe88ca1e740 1 --2- 172.21.15.28:0/373927876 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x564a661fe520 0x564a66200910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:34.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.193+0000 7fe88ca1e740 4 client.26287 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-12T22:13:34.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.193+0000 7fe88ca1e740 1 -- 172.21.15.28:0/373927876 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564a6615b3a0 con 0x564a661fe520 2024-05-12T22:13:34.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.193+0000 7fe883400640 1 --2- 172.21.15.28:0/373927876 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x564a661fe520 0x564a66200910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:34.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.194+0000 7fe883400640 1 --2- 172.21.15.28:0/373927876 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x564a661fe520 0x564a66200910 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d9f602c3648a4713 server_cookie=e3f7d418b72f948e in_seq=0 out_seq=0 2024-05-12T22:13:34.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.194+0000 7fe86cc00640 10 client.26287 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:13:34.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.194+0000 7fe86cc00640 1 -- 172.21.15.28:0/373927876 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564a6615b3a0 con 0x564a661fe520 2024-05-12T22:13:34.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.194+0000 7fe86cc00640 10 client.26287 handle_command_reply: tid=0 2024-05-12T22:13:34.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.195+0000 7fe88ca1e740 1 client.26287 shutdown 2024-05-12T22:13:34.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.195+0000 7fe88ca1e740 1 -- 172.21.15.28:0/373927876 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564a661e3c70 msgr2=0x564a661eb750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:13:34.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.195+0000 7fe88ca1e740 1 --2- 172.21.15.28:0/373927876 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564a661e3c70 0x564a661eb750 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fe870012700 tx=0x7fe870012bc0 comp rx=0 tx=0).stop 2024-05-12T22:13:34.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.195+0000 7fe88ca1e740 1 -- 172.21.15.28:0/373927876 shutdown_connections 2024-05-12T22:13:34.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.195+0000 7fe88ca1e740 1 --2- 172.21.15.28:0/373927876 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x564a661fe520 0x564a66200910 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:34.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.195+0000 7fe88ca1e740 1 --2- 172.21.15.28:0/373927876 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564a661ee2e0 0x564a661ebc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:34.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.195+0000 7fe86cc00640 0 client.26287 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:13:34.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.195+0000 7fe88ca1e740 1 --2- 172.21.15.28:0/373927876 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564a661e3c70 0x564a661eb750 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:34.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.195+0000 7fe88ca1e740 1 --2- 172.21.15.28:0/373927876 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564a661e32c0 0x564a661eb210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:34.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.195+0000 7fe88ca1e740 1 -- 172.21.15.28:0/373927876 >> 172.21.15.28:0/373927876 conn(0x564a661df370 msgr2=0x564a661e53f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:13:34.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.195+0000 7fe88ca1e740 1 -- 172.21.15.28:0/373927876 shutdown_connections 2024-05-12T22:13:34.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.195+0000 7fe88ca1e740 1 -- 172.21.15.28:0/373927876 wait complete. 2024-05-12T22:13:34.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.195+0000 7fe88ca1e740 20 client.26287 trim_cache size 0 max 16384 2024-05-12T22:13:34.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.214+0000 7fe88ca1e740 1 Processor -- start 2024-05-12T22:13:34.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.215+0000 7fe88ca1e740 1 -- start start 2024-05-12T22:13:34.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.215+0000 7fe88ca1e740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564a661e32c0 0x564a661fa1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:34.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.215+0000 7fe88ca1e740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564a661e3c70 0x564a661e5200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:34.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.215+0000 7fe88ca1e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564a661ee2e0 0x564a66206220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:34.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.215+0000 7fe88ca1e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe8841b1f30 con 0x564a661ee2e0 2024-05-12T22:13:34.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.215+0000 7fe88ca1e740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fe8841af800 con 0x564a661e3c70 2024-05-12T22:13:34.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.215+0000 7fe88ca1e740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe8841ad650 con 0x564a661e32c0 2024-05-12T22:13:34.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.215+0000 7fe883400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564a661e32c0 0x564a661fa1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:34.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.216+0000 7fe882000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564a661e3c70 0x564a661e5200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:34.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.216+0000 7fe883400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564a661e32c0 0x564a661fa1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:37692/0 (socket says 172.21.15.28:37692) 2024-05-12T22:13:34.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.216+0000 7fe883400640 1 -- 172.21.15.28:0/858596699 learned_addr learned my addr 172.21.15.28:0/858596699 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:13:34.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.216+0000 7fe883400640 1 --2- 172.21.15.28:0/858596699 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564a661ee2e0 0x564a66206220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:34.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.216+0000 7fe882000640 1 -- 172.21.15.28:0/858596699 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564a661e32c0 msgr2=0x564a661fa1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:13:34.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.216+0000 7fe882000640 1 --2- 172.21.15.28:0/858596699 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564a661e32c0 0x564a661fa1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:34.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.216+0000 7fe882000640 1 -- 172.21.15.28:0/858596699 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564a661ee2e0 msgr2=0x564a66206220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:13:34.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.216+0000 7fe882000640 1 --2- 172.21.15.28:0/858596699 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564a661ee2e0 0x564a66206220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:34.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.216+0000 7fe882000640 1 -- 172.21.15.28:0/858596699 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564a65e1b170 con 0x564a661e3c70 2024-05-12T22:13:34.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.216+0000 7fe882000640 1 --2- 172.21.15.28:0/858596699 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564a661e3c70 0x564a661e5200 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fe870025610 tx=0x7fe870009d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2a89d978d62276e9 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:13:34.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.217+0000 7fe86f400640 1 -- 172.21.15.28:0/858596699 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe870025850 con 0x564a661e3c70 2024-05-12T22:13:34.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.217+0000 7fe86f400640 1 -- 172.21.15.28:0/858596699 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe870003fb0 con 0x564a661e3c70 2024-05-12T22:13:34.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.217+0000 7fe86f400640 1 -- 172.21.15.28:0/858596699 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe870005040 con 0x564a661e3c70 2024-05-12T22:13:34.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.217+0000 7fe88ca1e740 1 -- 172.21.15.28:0/858596699 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564a661e3c70 msgr2=0x564a661e5200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:13:34.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.217+0000 7fe88ca1e740 1 --2- 172.21.15.28:0/858596699 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564a661e3c70 0x564a661e5200 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fe870025610 tx=0x7fe870009d10 comp rx=0 tx=0).stop 2024-05-12T22:13:34.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.217+0000 7fe88ca1e740 1 -- 172.21.15.28:0/858596699 shutdown_connections 2024-05-12T22:13:34.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.217+0000 7fe88ca1e740 1 --2- 172.21.15.28:0/858596699 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564a661ee2e0 0x564a66206220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:34.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.217+0000 7fe88ca1e740 1 --2- 172.21.15.28:0/858596699 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564a661e3c70 0x564a661e5200 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:34.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.217+0000 7fe88ca1e740 1 --2- 172.21.15.28:0/858596699 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564a661e32c0 0x564a661fa1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:34.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.217+0000 7fe88ca1e740 1 -- 172.21.15.28:0/858596699 >> 172.21.15.28:0/858596699 conn(0x564a661df370 msgr2=0x564a661e4af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:13:34.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.218+0000 7fe88ca1e740 1 -- 172.21.15.28:0/858596699 shutdown_connections 2024-05-12T22:13:34.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.218+0000 7fe88ca1e740 1 -- 172.21.15.28:0/858596699 wait complete. 2024-05-12T22:13:34.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.218+0000 7fe88ca1e740 1 -- start start 2024-05-12T22:13:34.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.218+0000 7fe88ca1e740 1 Processor -- start 2024-05-12T22:13:34.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.219+0000 7fe88ca1e740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564a661e32c0 0x564a661f6b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:34.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.219+0000 7fe88ca1e740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564a661e3c70 0x564a661f1ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:34.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.219+0000 7fe88ca1e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564a661ee2e0 0x564a661f2460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:34.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.219+0000 7fe88ca1e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe8841adcc0 con 0x564a661ee2e0 2024-05-12T22:13:34.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.219+0000 7fe88ca1e740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fe8841af130 con 0x564a661e3c70 2024-05-12T22:13:34.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.219+0000 7fe88ca1e740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe8841ae390 con 0x564a661e32c0 2024-05-12T22:13:34.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.219+0000 7fe88ca1e740 10 client.0 _ll_register_callbacks cb 0x7fe870009570 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:13:34.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.219+0000 7fe882000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564a661e3c70 0x564a661f1ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:34.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.219+0000 7fe883400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564a661ee2e0 0x564a661f2460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:34.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.219+0000 7fe882000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564a661e3c70 0x564a661f1ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:44006/0 (socket says 172.21.15.28:44006) 2024-05-12T22:13:34.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.219+0000 7fe882000640 1 -- 172.21.15.28:0/3495849217 learned_addr learned my addr 172.21.15.28:0/3495849217 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:13:34.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.219+0000 7fe883400640 1 --2- 172.21.15.28:0/3495849217 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564a661e32c0 0x564a661f6b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:34.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.219+0000 7fe882000640 1 -- 172.21.15.28:0/3495849217 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564a661e32c0 msgr2=0x564a661f6b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:13:34.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.219+0000 7fe882000640 1 --2- 172.21.15.28:0/3495849217 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564a661e32c0 0x564a661f6b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:34.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.220+0000 7fe882000640 1 -- 172.21.15.28:0/3495849217 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564a661ee2e0 msgr2=0x564a661f2460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:13:34.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.220+0000 7fe882000640 1 --2- 172.21.15.28:0/3495849217 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564a661ee2e0 0x564a661f2460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:34.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.220+0000 7fe882000640 1 -- 172.21.15.28:0/3495849217 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564a661f2c90 con 0x564a661e3c70 2024-05-12T22:13:34.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.220+0000 7fe883400640 1 --2- 172.21.15.28:0/3495849217 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564a661ee2e0 0x564a661f2460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:13:34.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.220+0000 7fe882000640 1 --2- 172.21.15.28:0/3495849217 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564a661e3c70 0x564a661f1ee0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fe8700126a0 tx=0x7fe870005da0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a3add7dd45b931f7 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:13:34.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.220+0000 7fe86cc00640 10 client.0 ms_handle_connect on v2:172.21.15.86:3300/0 2024-05-12T22:13:34.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.220+0000 7fe86cc00640 1 -- 172.21.15.28:0/3495849217 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe870033070 con 0x564a661e3c70 2024-05-12T22:13:34.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.220+0000 7fe86cc00640 1 -- 172.21.15.28:0/3495849217 <== mon.2 v2:172.21.15.86:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7fe87000f920 con 0x564a661e3c70 2024-05-12T22:13:34.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.220+0000 7fe88ca1e740 1 -- 172.21.15.28:0/3495849217 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe870003fb0 con 0x564a661e3c70 2024-05-12T22:13:34.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.220+0000 7fe86cc00640 1 -- 172.21.15.28:0/3495849217 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fe87003f050 con 0x564a661e3c70 2024-05-12T22:13:34.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.220+0000 7fe88ca1e740 10 client.26333 fetch_fsmap learned FSMap version 26 2024-05-12T22:13:34.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.220+0000 7fe88ca1e740 1 -- 172.21.15.28:0/3495849217 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x564a66264b70 con 0x564a661e3c70 2024-05-12T22:13:34.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.221+0000 7fe86cc00640 1 -- 172.21.15.28:0/3495849217 <== mon.2 v2:172.21.15.86:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe870003fb0 con 0x564a661e3c70 2024-05-12T22:13:34.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.221+0000 7fe86cc00640 1 -- 172.21.15.28:0/3495849217 <== mon.2 v2:172.21.15.86:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fe870005460 con 0x564a661e3c70 2024-05-12T22:13:34.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.221+0000 7fe88ca1e740 10 client.26333 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:13:34.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.221+0000 7fe88ca1e740 10 client.26333 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:13:34.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.221+0000 7fe88ca1e740 20 client.26333 populate_metadata read hostname 'smithi028' 2024-05-12T22:13:34.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.221+0000 7fe88ca1e740 1 --2- 172.21.15.28:0/3495849217 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x564a662651d0 0x564a662675c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:34.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.221+0000 7fe88ca1e740 4 client.26333 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-12T22:13:34.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.221+0000 7fe88ca1e740 1 -- 172.21.15.28:0/3495849217 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x564a66154ef0 con 0x564a662651d0 2024-05-12T22:13:34.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.221+0000 7fe883400640 1 --2- 172.21.15.28:0/3495849217 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x564a662651d0 0x564a662675c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:34.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.222+0000 7fe883400640 1 --2- 172.21.15.28:0/3495849217 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x564a662651d0 0x564a662675c0 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9d6bd81330c0dd28 server_cookie=4d49f4ac2f961398 in_seq=0 out_seq=0 2024-05-12T22:13:34.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.222+0000 7fe86cc00640 10 client.26333 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:13:34.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.223+0000 7fe86cc00640 1 -- 172.21.15.28:0/3495849217 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x564a66154ef0 con 0x564a662651d0 2024-05-12T22:13:34.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.223+0000 7fe86cc00640 10 client.26333 handle_command_reply: tid=0 2024-05-12T22:13:34.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.223+0000 7fe88ca1e740 1 client.26333 shutdown 2024-05-12T22:13:34.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.223+0000 7fe88ca1e740 1 -- 172.21.15.28:0/3495849217 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564a661e3c70 msgr2=0x564a661f1ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:13:34.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.223+0000 7fe88ca1e740 1 --2- 172.21.15.28:0/3495849217 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564a661e3c70 0x564a661f1ee0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fe8700126a0 tx=0x7fe870005da0 comp rx=0 tx=0).stop 2024-05-12T22:13:34.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.223+0000 7fe88ca1e740 1 -- 172.21.15.28:0/3495849217 shutdown_connections 2024-05-12T22:13:34.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.223+0000 7fe88ca1e740 1 --2- 172.21.15.28:0/3495849217 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x564a662651d0 0x564a662675c0 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:34.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.223+0000 7fe88ca1e740 1 --2- 172.21.15.28:0/3495849217 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564a661ee2e0 0x564a661f2460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:34.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.223+0000 7fe88ca1e740 1 --2- 172.21.15.28:0/3495849217 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564a661e3c70 0x564a661f1ee0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:34.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.223+0000 7fe86cc00640 0 client.26333 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:13:34.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.223+0000 7fe88ca1e740 1 --2- 172.21.15.28:0/3495849217 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564a661e32c0 0x564a661f6b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:34.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.223+0000 7fe88ca1e740 1 -- 172.21.15.28:0/3495849217 >> 172.21.15.28:0/3495849217 conn(0x564a661df370 msgr2=0x564a661e96b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:13:34.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.224+0000 7fe88ca1e740 1 -- 172.21.15.28:0/3495849217 shutdown_connections 2024-05-12T22:13:34.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.224+0000 7fe88ca1e740 1 -- 172.21.15.28:0/3495849217 wait complete. 2024-05-12T22:13:34.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.224+0000 7fe88ca1e740 20 client.26333 trim_cache size 0 max 16384 2024-05-12T22:13:34.231 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-05-12T22:13:34.231 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-05-12T22:13:34.231 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "406ccb73-94f2-43f6-b42d-7e5336642117", 2024-05-12T22:13:34.231 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-05-12T22:13:34.231 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-05-12T22:13:34.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.224+0000 7fe889c00640 1 -- 172.21.15.28:0/3220494230 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fe8680a2d80 msgr2=0x7fe8680a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:13:34.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.224+0000 7fe889c00640 1 --2- 172.21.15.28:0/3220494230 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fe8680a2d80 0x7fe8680a5240 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x564a661e4880 tx=0x7fe8740f5530 comp rx=0 tx=0).stop 2024-05-12T22:13:34.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.224+0000 7fe889c00640 1 -- 172.21.15.28:0/3220494230 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe884101850 msgr2=0x7fe884073b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:13:34.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.224+0000 7fe889c00640 1 --2- 172.21.15.28:0/3220494230 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe884101850 0x7fe884073b30 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fe87402a960 tx=0x7fe874003b90 comp rx=0 tx=0).stop 2024-05-12T22:13:34.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.224+0000 7fe889c00640 1 -- 172.21.15.28:0/3220494230 shutdown_connections 2024-05-12T22:13:34.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.224+0000 7fe889c00640 1 --2- 172.21.15.28:0/3220494230 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fe8680a2d80 0x7fe8680a5240 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:34.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.224+0000 7fe889c00640 1 --2- 172.21.15.28:0/3220494230 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe884107860 0x7fe8840745b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:34.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.224+0000 7fe889c00640 1 --2- 172.21.15.28:0/3220494230 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe884102210 0x7fe884074070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:34.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.224+0000 7fe889c00640 1 --2- 172.21.15.28:0/3220494230 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe884101850 0x7fe884073b30 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:34.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.224+0000 7fe889c00640 1 -- 172.21.15.28:0/3220494230 >> 172.21.15.28:0/3220494230 conn(0x7fe8840fd340 msgr2=0x7fe8840fdb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:13:34.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.225+0000 7fe889c00640 1 -- 172.21.15.28:0/3220494230 shutdown_connections 2024-05-12T22:13:34.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.225+0000 7fe889c00640 1 -- 172.21.15.28:0/3220494230 wait complete. 2024-05-12T22:13:34.241 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:13:33 smithi086 ceph-mon[34297]: pgmap v140: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:13:34.243 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:13:33 smithi028 ceph-mon[29847]: pgmap v140: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:13:34.248 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-12T22:13:34.248 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-12T22:13:34.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.492+0000 7f530aa00640 1 -- 172.21.15.28:0/355145307 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5304101700 msgr2=0x7f5304101b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:13:34.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.492+0000 7f530aa00640 1 --2- 172.21.15.28:0/355145307 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5304101700 0x7f5304101b80 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f52f0009860 tx=0x7f52f002a340 comp rx=0 tx=0).stop 2024-05-12T22:13:34.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.492+0000 7f530aa00640 1 -- 172.21.15.28:0/355145307 shutdown_connections 2024-05-12T22:13:34.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.492+0000 7f530aa00640 1 --2- 172.21.15.28:0/355145307 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f53041020c0 0x7f530410e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:34.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.492+0000 7f530aa00640 1 --2- 172.21.15.28:0/355145307 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5304101700 0x7f5304101b80 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:34.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.492+0000 7f530aa00640 1 --2- 172.21.15.28:0/355145307 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5304107710 0x7f5304107af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:34.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.492+0000 7f530aa00640 1 -- 172.21.15.28:0/355145307 >> 172.21.15.28:0/355145307 conn(0x7f53040fd1f0 msgr2=0x7f53040ff610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:13:34.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.492+0000 7f530aa00640 1 -- 172.21.15.28:0/355145307 shutdown_connections 2024-05-12T22:13:34.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.492+0000 7f530aa00640 1 -- 172.21.15.28:0/355145307 wait complete. 2024-05-12T22:13:34.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.493+0000 7f530aa00640 1 Processor -- start 2024-05-12T22:13:34.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.493+0000 7f530aa00640 1 -- start start 2024-05-12T22:13:34.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.494+0000 7f530aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5304101700 0x7f5304073b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:34.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.494+0000 7f530aa00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f53041020c0 0x7f5304074070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:34.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.494+0000 7f530aa00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5304107710 0x7f53040745b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:34.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.494+0000 7f530aa00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5304111ae0 con 0x7f5304101700 2024-05-12T22:13:34.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.494+0000 7f530aa00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f5304111960 con 0x7f5304107710 2024-05-12T22:13:34.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.494+0000 7f530aa00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f5304111c60 con 0x7f53041020c0 2024-05-12T22:13:34.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.494+0000 7f5308c00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f53041020c0 0x7f5304074070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:34.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.494+0000 7f5308c00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f53041020c0 0x7f5304074070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:37708/0 (socket says 172.21.15.28:37708) 2024-05-12T22:13:34.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.494+0000 7f5308c00640 1 -- 172.21.15.28:0/3458690672 learned_addr learned my addr 172.21.15.28:0/3458690672 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:13:34.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.494+0000 7f5309600640 1 --2- 172.21.15.28:0/3458690672 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5304101700 0x7f5304073b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:34.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.494+0000 7f530a000640 1 --2- 172.21.15.28:0/3458690672 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5304107710 0x7f53040745b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:34.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.495+0000 7f5308c00640 1 -- 172.21.15.28:0/3458690672 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5304107710 msgr2=0x7f53040745b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:13:34.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.495+0000 7f5308c00640 1 --2- 172.21.15.28:0/3458690672 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5304107710 0x7f53040745b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:34.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.495+0000 7f5308c00640 1 -- 172.21.15.28:0/3458690672 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5304101700 msgr2=0x7f5304073b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:13:34.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.495+0000 7f5308c00640 1 --2- 172.21.15.28:0/3458690672 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5304101700 0x7f5304073b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:34.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.495+0000 7f5308c00640 1 -- 172.21.15.28:0/3458690672 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53041a64c0 con 0x7f53041020c0 2024-05-12T22:13:34.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.495+0000 7f5308c00640 1 --2- 172.21.15.28:0/3458690672 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f53041020c0 0x7f5304074070 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f52f0035040 tx=0x7f52f0033b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:13:34.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.496+0000 7f5302000640 1 -- 172.21.15.28:0/3458690672 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52f0037040 con 0x7f53041020c0 2024-05-12T22:13:34.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.496+0000 7f5302000640 1 -- 172.21.15.28:0/3458690672 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f52f0040070 con 0x7f53041020c0 2024-05-12T22:13:34.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.496+0000 7f5302000640 1 -- 172.21.15.28:0/3458690672 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52f0003a10 con 0x7f53041020c0 2024-05-12T22:13:34.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.496+0000 7f530aa00640 1 -- 172.21.15.28:0/3458690672 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53041a6750 con 0x7f53041020c0 2024-05-12T22:13:34.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.496+0000 7f530aa00640 1 -- 172.21.15.28:0/3458690672 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53041a6bf0 con 0x7f53041020c0 2024-05-12T22:13:34.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.500+0000 7f5302000640 1 -- 172.21.15.28:0/3458690672 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7f52f0004150 con 0x7f53041020c0 2024-05-12T22:13:34.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.500+0000 7f530eda2740 1 Processor -- start 2024-05-12T22:13:34.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.500+0000 7f530eda2740 1 -- start start 2024-05-12T22:13:34.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.501+0000 7f530eda2740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56269c588dd0 0x56269c5891b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:34.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.501+0000 7f530eda2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56269c5896f0 0x56269c593880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:34.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.501+0000 7f530eda2740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56269c593dc0 0x56269c5961b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:34.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.501+0000 7f530eda2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56269c5512a0 con 0x56269c5896f0 2024-05-12T22:13:34.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.501+0000 7f530eda2740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x56269c4edb90 con 0x56269c588dd0 2024-05-12T22:13:34.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.501+0000 7f530eda2740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x56269c4955b0 con 0x56269c593dc0 2024-05-12T22:13:34.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.501+0000 7f530a000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56269c588dd0 0x56269c5891b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:34.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.501+0000 7f530a000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56269c588dd0 0x56269c5891b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:44018/0 (socket says 172.21.15.28:44018) 2024-05-12T22:13:34.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.501+0000 7f530a000640 1 -- 172.21.15.28:0/2470898031 learned_addr learned my addr 172.21.15.28:0/2470898031 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:13:34.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.502+0000 7f5309600640 1 --2- 172.21.15.28:0/2470898031 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56269c5896f0 0x56269c593880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:34.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.502+0000 7f5308c00640 1 --2- 172.21.15.28:0/2470898031 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56269c593dc0 0x56269c5961b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:34.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.502+0000 7f530a000640 1 -- 172.21.15.28:0/2470898031 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56269c593dc0 msgr2=0x56269c5961b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:13:34.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.502+0000 7f530a000640 1 --2- 172.21.15.28:0/2470898031 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56269c593dc0 0x56269c5961b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:34.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.502+0000 7f530a000640 1 -- 172.21.15.28:0/2470898031 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56269c5896f0 msgr2=0x56269c593880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:13:34.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.502+0000 7f530a000640 1 --2- 172.21.15.28:0/2470898031 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56269c5896f0 0x56269c593880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:34.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.502+0000 7f530a000640 1 -- 172.21.15.28:0/2470898031 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56269c2cb7d0 con 0x56269c588dd0 2024-05-12T22:13:34.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.502+0000 7f5309600640 1 --2- 172.21.15.28:0/2470898031 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56269c5896f0 0x56269c593880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:13:34.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.502+0000 7f530a000640 1 --2- 172.21.15.28:0/2470898031 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56269c588dd0 0x56269c5891b0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f52fc00e580 tx=0x7f52fc0095c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eb193b28de94d824 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:13:34.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.502+0000 7f5302000640 1 --2- 172.21.15.28:0/3458690672 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f52ec0a2d80 0x7f52ec0a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:34.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.503+0000 7f5302000640 1 -- 172.21.15.28:0/3458690672 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f52f00ef2c0 con 0x7f53041020c0 2024-05-12T22:13:34.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.503+0000 7f5309600640 1 --2- 172.21.15.28:0/3458690672 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f52ec0a2d80 0x7f52ec0a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:34.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.503+0000 7f52f6600640 1 -- 172.21.15.28:0/2470898031 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52fc003970 con 0x56269c588dd0 2024-05-12T22:13:34.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.503+0000 7f52f6600640 1 -- 172.21.15.28:0/2470898031 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f52fc00d830 con 0x56269c588dd0 2024-05-12T22:13:34.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.503+0000 7f52f6600640 1 -- 172.21.15.28:0/2470898031 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52fc0045a0 con 0x56269c588dd0 2024-05-12T22:13:34.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.504+0000 7f530eda2740 1 -- 172.21.15.28:0/2470898031 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56269c588dd0 msgr2=0x56269c5891b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:13:34.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.504+0000 7f5309600640 1 --2- 172.21.15.28:0/3458690672 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f52ec0a2d80 0x7f52ec0a5240 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x56269c58a390 tx=0x7f52f8005f90 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:13:34.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.504+0000 7f530eda2740 1 --2- 172.21.15.28:0/2470898031 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56269c588dd0 0x56269c5891b0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f52fc00e580 tx=0x7f52fc0095c0 comp rx=0 tx=0).stop 2024-05-12T22:13:34.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.504+0000 7f530eda2740 1 -- 172.21.15.28:0/2470898031 shutdown_connections 2024-05-12T22:13:34.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.504+0000 7f530eda2740 1 --2- 172.21.15.28:0/2470898031 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56269c593dc0 0x56269c5961b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:34.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.504+0000 7f530eda2740 1 --2- 172.21.15.28:0/2470898031 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56269c5896f0 0x56269c593880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:34.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.504+0000 7f530eda2740 1 --2- 172.21.15.28:0/2470898031 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56269c588dd0 0x56269c5891b0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:34.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.504+0000 7f530eda2740 1 -- 172.21.15.28:0/2470898031 >> 172.21.15.28:0/2470898031 conn(0x56269c5850f0 msgr2=0x56269c5854d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:13:34.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.504+0000 7f530eda2740 1 -- 172.21.15.28:0/2470898031 shutdown_connections 2024-05-12T22:13:34.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.504+0000 7f530eda2740 1 -- 172.21.15.28:0/2470898031 wait complete. 2024-05-12T22:13:34.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.504+0000 7f530eda2740 1 -- start start 2024-05-12T22:13:34.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.505+0000 7f530eda2740 1 Processor -- start 2024-05-12T22:13:34.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.505+0000 7f530eda2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56269c588dd0 0x56269c5937c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:34.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.505+0000 7f530eda2740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56269c5896f0 0x56269c590ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:34.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.505+0000 7f530eda2740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56269c593dc0 0x56269c5910c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:34.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.505+0000 7f530eda2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56269c4a89f0 con 0x56269c588dd0 2024-05-12T22:13:34.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.505+0000 7f530eda2740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x56269c4e9b50 con 0x56269c5896f0 2024-05-12T22:13:34.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.505+0000 7f530eda2740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x56269c596bb0 con 0x56269c593dc0 2024-05-12T22:13:34.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.505+0000 7f530eda2740 10 client.0 _ll_register_callbacks cb 0x56269c200be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:13:34.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.506+0000 7f530a000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56269c593dc0 0x56269c5910c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:34.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.506+0000 7f530a000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56269c593dc0 0x56269c5910c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:37740/0 (socket says 172.21.15.28:37740) 2024-05-12T22:13:34.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.506+0000 7f5309600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56269c588dd0 0x56269c5937c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:34.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.506+0000 7f530a000640 1 -- 172.21.15.28:0/529040528 learned_addr learned my addr 172.21.15.28:0/529040528 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:13:34.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.506+0000 7f5308c00640 1 --2- 172.21.15.28:0/529040528 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56269c5896f0 0x56269c590ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:34.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.506+0000 7f5308c00640 1 -- 172.21.15.28:0/529040528 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56269c593dc0 msgr2=0x56269c5910c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:13:34.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.506+0000 7f5308c00640 1 --2- 172.21.15.28:0/529040528 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56269c593dc0 0x56269c5910c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:34.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.506+0000 7f5308c00640 1 -- 172.21.15.28:0/529040528 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56269c588dd0 msgr2=0x56269c5937c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:13:34.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.506+0000 7f5308c00640 1 --2- 172.21.15.28:0/529040528 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56269c588dd0 0x56269c5937c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:34.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.506+0000 7f5308c00640 1 -- 172.21.15.28:0/529040528 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56269c2013f0 con 0x56269c5896f0 2024-05-12T22:13:34.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.506+0000 7f5309600640 1 --2- 172.21.15.28:0/529040528 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56269c588dd0 0x56269c5937c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:13:34.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.506+0000 7f530a000640 1 --2- 172.21.15.28:0/529040528 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56269c593dc0 0x56269c5910c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:13:34.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.507+0000 7f5308c00640 1 --2- 172.21.15.28:0/529040528 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56269c5896f0 0x56269c590ab0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f52f00ecaf0 tx=0x7f52f00b7c20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=df5aa030702e951d server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:13:34.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.507+0000 7f52eaa00640 10 client.0 ms_handle_connect on v2:172.21.15.86:3300/0 2024-05-12T22:13:34.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.507+0000 7f52eaa00640 1 -- 172.21.15.28:0/529040528 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52f00bbfc0 con 0x56269c5896f0 2024-05-12T22:13:34.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.507+0000 7f52eaa00640 1 -- 172.21.15.28:0/529040528 <== mon.2 v2:172.21.15.86:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f52f00ecc60 con 0x56269c5896f0 2024-05-12T22:13:34.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.507+0000 7f530eda2740 1 -- 172.21.15.28:0/529040528 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f52fc00d830 con 0x56269c5896f0 2024-05-12T22:13:34.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.507+0000 7f52eaa00640 1 -- 172.21.15.28:0/529040528 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f52f00ece60 con 0x56269c5896f0 2024-05-12T22:13:34.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.507+0000 7f530eda2740 10 client.26345 fetch_fsmap learned FSMap version 26 2024-05-12T22:13:34.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.507+0000 7f530eda2740 1 -- 172.21.15.28:0/529040528 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x56269c1c10f0 con 0x56269c5896f0 2024-05-12T22:13:34.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.508+0000 7f52eaa00640 1 -- 172.21.15.28:0/529040528 <== mon.2 v2:172.21.15.86:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f52fc00d830 con 0x56269c5896f0 2024-05-12T22:13:34.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.508+0000 7f52eaa00640 1 -- 172.21.15.28:0/529040528 <== mon.2 v2:172.21.15.86:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f52f003a070 con 0x56269c5896f0 2024-05-12T22:13:34.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.508+0000 7f530eda2740 10 client.26345 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:13:34.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.508+0000 7f530eda2740 10 client.26345 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:13:34.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.508+0000 7f530eda2740 20 client.26345 populate_metadata read hostname 'smithi028' 2024-05-12T22:13:34.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.508+0000 7f530eda2740 1 --2- 172.21.15.28:0/529040528 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x56269c5a3900 0x56269c5a5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:34.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.508+0000 7f530eda2740 4 client.26345 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-12T22:13:34.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.508+0000 7f530eda2740 1 -- 172.21.15.28:0/529040528 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56269c4fb2c0 con 0x56269c5a3900 2024-05-12T22:13:34.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.509+0000 7f5309600640 1 --2- 172.21.15.28:0/529040528 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x56269c5a3900 0x56269c5a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:34.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.509+0000 7f5309600640 1 --2- 172.21.15.28:0/529040528 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x56269c5a3900 0x56269c5a5d10 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8ed01890c4bb694d server_cookie=db27c96535929921 in_seq=0 out_seq=0 2024-05-12T22:13:34.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.509+0000 7f52eaa00640 10 client.26345 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:13:34.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.511+0000 7f52eaa00640 1 -- 172.21.15.28:0/529040528 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56269c4fb2c0 con 0x56269c5a3900 2024-05-12T22:13:34.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.511+0000 7f52eaa00640 10 client.26345 handle_command_reply: tid=0 2024-05-12T22:13:34.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.511+0000 7f530eda2740 1 client.26345 shutdown 2024-05-12T22:13:34.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.511+0000 7f530eda2740 1 -- 172.21.15.28:0/529040528 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56269c5896f0 msgr2=0x56269c590ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:13:34.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.511+0000 7f530eda2740 1 --2- 172.21.15.28:0/529040528 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56269c5896f0 0x56269c590ab0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f52f00ecaf0 tx=0x7f52f00b7c20 comp rx=0 tx=0).stop 2024-05-12T22:13:34.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.511+0000 7f530eda2740 1 -- 172.21.15.28:0/529040528 shutdown_connections 2024-05-12T22:13:34.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.511+0000 7f530eda2740 1 --2- 172.21.15.28:0/529040528 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x56269c5a3900 0x56269c5a5d10 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:34.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.511+0000 7f530eda2740 1 --2- 172.21.15.28:0/529040528 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56269c593dc0 0x56269c5910c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:34.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.511+0000 7f530eda2740 1 --2- 172.21.15.28:0/529040528 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56269c5896f0 0x56269c590ab0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:34.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.511+0000 7f530eda2740 1 --2- 172.21.15.28:0/529040528 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56269c588dd0 0x56269c5937c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:34.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.511+0000 7f530eda2740 1 -- 172.21.15.28:0/529040528 >> 172.21.15.28:0/529040528 conn(0x56269c5850f0 msgr2=0x56269c584330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:13:34.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.511+0000 7f52eaa00640 0 client.26345 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:13:34.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.512+0000 7f530eda2740 1 -- 172.21.15.28:0/529040528 shutdown_connections 2024-05-12T22:13:34.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.512+0000 7f530eda2740 1 -- 172.21.15.28:0/529040528 wait complete. 2024-05-12T22:13:34.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.512+0000 7f530eda2740 20 client.26345 trim_cache size 0 max 16384 2024-05-12T22:13:34.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.538+0000 7f530eda2740 1 Processor -- start 2024-05-12T22:13:34.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.538+0000 7f530eda2740 1 -- start start 2024-05-12T22:13:34.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.538+0000 7f530eda2740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56269c588dd0 0x56269c5932f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:34.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.538+0000 7f530eda2740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56269c5896f0 0x56269c59f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:34.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.538+0000 7f530eda2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56269c593dc0 0x56269c5ab5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:34.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.538+0000 7f530eda2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f53041b1d90 con 0x56269c593dc0 2024-05-12T22:13:34.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.538+0000 7f530eda2740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f53041af660 con 0x56269c588dd0 2024-05-12T22:13:34.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.539+0000 7f530eda2740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f53041ad4b0 con 0x56269c5896f0 2024-05-12T22:13:34.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.539+0000 7f5309600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56269c588dd0 0x56269c5932f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:34.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.539+0000 7f5309600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56269c588dd0 0x56269c5932f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:44044/0 (socket says 172.21.15.28:44044) 2024-05-12T22:13:34.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.539+0000 7f5308c00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56269c5896f0 0x56269c59f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:34.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.539+0000 7f5309600640 1 -- 172.21.15.28:0/648565746 learned_addr learned my addr 172.21.15.28:0/648565746 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:13:34.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.539+0000 7f530a000640 1 --2- 172.21.15.28:0/648565746 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56269c593dc0 0x56269c5ab5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:34.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.539+0000 7f5309600640 1 -- 172.21.15.28:0/648565746 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56269c5896f0 msgr2=0x56269c59f8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:13:34.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.539+0000 7f5309600640 1 --2- 172.21.15.28:0/648565746 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56269c5896f0 0x56269c59f8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:34.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.539+0000 7f5309600640 1 -- 172.21.15.28:0/648565746 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56269c593dc0 msgr2=0x56269c5ab5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:13:34.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.539+0000 7f5309600640 1 --2- 172.21.15.28:0/648565746 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56269c593dc0 0x56269c5ab5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:34.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.539+0000 7f5309600640 1 -- 172.21.15.28:0/648565746 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56269c58f000 con 0x56269c588dd0 2024-05-12T22:13:34.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.539+0000 7f530a000640 1 --2- 172.21.15.28:0/648565746 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56269c593dc0 0x56269c5ab5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:13:34.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.540+0000 7f5309600640 1 --2- 172.21.15.28:0/648565746 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56269c588dd0 0x56269c5932f0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f52f8009fd0 tx=0x7f52f8017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d8acf2c7223db2ea server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:13:34.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.540+0000 7f52f5c00640 1 -- 172.21.15.28:0/648565746 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52f8031070 con 0x56269c588dd0 2024-05-12T22:13:34.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.540+0000 7f52f5c00640 1 -- 172.21.15.28:0/648565746 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f52f8004390 con 0x56269c588dd0 2024-05-12T22:13:34.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.540+0000 7f52f5c00640 1 -- 172.21.15.28:0/648565746 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52f801f7b0 con 0x56269c588dd0 2024-05-12T22:13:34.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.541+0000 7f530eda2740 1 -- 172.21.15.28:0/648565746 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56269c588dd0 msgr2=0x56269c5932f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:13:34.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.541+0000 7f530eda2740 1 --2- 172.21.15.28:0/648565746 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56269c588dd0 0x56269c5932f0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f52f8009fd0 tx=0x7f52f8017000 comp rx=0 tx=0).stop 2024-05-12T22:13:34.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.541+0000 7f530eda2740 1 -- 172.21.15.28:0/648565746 shutdown_connections 2024-05-12T22:13:34.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.541+0000 7f530eda2740 1 --2- 172.21.15.28:0/648565746 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56269c593dc0 0x56269c5ab5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:34.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.541+0000 7f530eda2740 1 --2- 172.21.15.28:0/648565746 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56269c5896f0 0x56269c59f8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:34.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.541+0000 7f530eda2740 1 --2- 172.21.15.28:0/648565746 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56269c588dd0 0x56269c5932f0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:34.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.541+0000 7f530eda2740 1 -- 172.21.15.28:0/648565746 >> 172.21.15.28:0/648565746 conn(0x56269c5850f0 msgr2=0x56269c58adb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:13:34.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.541+0000 7f530eda2740 1 -- 172.21.15.28:0/648565746 shutdown_connections 2024-05-12T22:13:34.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.541+0000 7f530eda2740 1 -- 172.21.15.28:0/648565746 wait complete. 2024-05-12T22:13:34.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.542+0000 7f530eda2740 1 -- start start 2024-05-12T22:13:34.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.542+0000 7f530eda2740 1 Processor -- start 2024-05-12T22:13:34.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.543+0000 7f530eda2740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56269c588dd0 0x56269c59c010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:34.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.543+0000 7f530eda2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56269c5896f0 0x56269c597420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:34.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.543+0000 7f530eda2740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56269c593dc0 0x56269c597960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:34.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.543+0000 7f530eda2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f53041adb20 con 0x56269c5896f0 2024-05-12T22:13:34.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.543+0000 7f530eda2740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f53041aef90 con 0x56269c588dd0 2024-05-12T22:13:34.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.543+0000 7f530eda2740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f53041ae1f0 con 0x56269c593dc0 2024-05-12T22:13:34.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.543+0000 7f530eda2740 10 client.0 _ll_register_callbacks cb 0x7f52f0004050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:13:34.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.543+0000 7f530a000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56269c593dc0 0x56269c597960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:34.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.543+0000 7f530a000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56269c593dc0 0x56269c597960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:37760/0 (socket says 172.21.15.28:37760) 2024-05-12T22:13:34.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.543+0000 7f530a000640 1 -- 172.21.15.28:0/294016822 learned_addr learned my addr 172.21.15.28:0/294016822 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:13:34.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.543+0000 7f5308c00640 1 --2- 172.21.15.28:0/294016822 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56269c5896f0 0x56269c597420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:34.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.543+0000 7f5309600640 1 --2- 172.21.15.28:0/294016822 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56269c588dd0 0x56269c59c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:34.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.544+0000 7f530a000640 1 -- 172.21.15.28:0/294016822 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56269c588dd0 msgr2=0x56269c59c010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:13:34.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.544+0000 7f530a000640 1 --2- 172.21.15.28:0/294016822 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56269c588dd0 0x56269c59c010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:34.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.544+0000 7f530a000640 1 -- 172.21.15.28:0/294016822 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56269c5896f0 msgr2=0x56269c597420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:13:34.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.544+0000 7f530a000640 1 --2- 172.21.15.28:0/294016822 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56269c5896f0 0x56269c597420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:34.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.544+0000 7f530a000640 1 -- 172.21.15.28:0/294016822 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56269c597fa0 con 0x56269c593dc0 2024-05-12T22:13:34.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.544+0000 7f5308c00640 1 --2- 172.21.15.28:0/294016822 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56269c5896f0 0x56269c597420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:13:34.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.544+0000 7f530a000640 1 --2- 172.21.15.28:0/294016822 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56269c593dc0 0x56269c597960 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f52fc00eed0 tx=0x7f52fc00dad0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=554f232227dbebee server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:13:34.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.544+0000 7f52eaa00640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-12T22:13:34.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.544+0000 7f52eaa00640 1 -- 172.21.15.28:0/294016822 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52fc00ddd0 con 0x56269c593dc0 2024-05-12T22:13:34.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.544+0000 7f52eaa00640 1 -- 172.21.15.28:0/294016822 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f52fc003970 con 0x56269c593dc0 2024-05-12T22:13:34.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.544+0000 7f530eda2740 1 -- 172.21.15.28:0/294016822 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f52f8004390 con 0x56269c593dc0 2024-05-12T22:13:34.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.545+0000 7f52eaa00640 1 -- 172.21.15.28:0/294016822 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f52fc003b70 con 0x56269c593dc0 2024-05-12T22:13:34.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.545+0000 7f530eda2740 10 client.26308 fetch_fsmap learned FSMap version 26 2024-05-12T22:13:34.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.545+0000 7f530eda2740 1 -- 172.21.15.28:0/294016822 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x56269c6124c0 con 0x56269c593dc0 2024-05-12T22:13:34.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.545+0000 7f52eaa00640 1 -- 172.21.15.28:0/294016822 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f52f8004390 con 0x56269c593dc0 2024-05-12T22:13:34.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.545+0000 7f52eaa00640 1 -- 172.21.15.28:0/294016822 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f52fc00b040 con 0x56269c593dc0 2024-05-12T22:13:34.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.545+0000 7f530eda2740 10 client.26308 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:13:34.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.545+0000 7f530eda2740 10 client.26308 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:13:34.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.545+0000 7f530eda2740 20 client.26308 populate_metadata read hostname 'smithi028' 2024-05-12T22:13:34.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.545+0000 7f530eda2740 1 --2- 172.21.15.28:0/294016822 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x56269c612ab0 0x56269c614ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:13:34.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.545+0000 7f530eda2740 4 client.26308 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-12T22:13:34.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.545+0000 7f530eda2740 1 -- 172.21.15.28:0/294016822 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56269c4f5610 con 0x56269c612ab0 2024-05-12T22:13:34.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.546+0000 7f5309600640 1 --2- 172.21.15.28:0/294016822 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x56269c612ab0 0x56269c614ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:13:34.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.546+0000 7f5309600640 1 --2- 172.21.15.28:0/294016822 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x56269c612ab0 0x56269c614ed0 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cc1c444ef7238db1 server_cookie=f3aee665707855ef in_seq=0 out_seq=0 2024-05-12T22:13:34.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.546+0000 7f52eaa00640 10 client.26308 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:13:34.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.547+0000 7f52eaa00640 1 -- 172.21.15.28:0/294016822 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+268 (crc 0 0 0) 0x56269c4f5610 con 0x56269c612ab0 2024-05-12T22:13:34.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.547+0000 7f52eaa00640 10 client.26308 handle_command_reply: tid=0 2024-05-12T22:13:34.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.547+0000 7f530eda2740 1 client.26308 shutdown 2024-05-12T22:13:34.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.547+0000 7f530eda2740 1 -- 172.21.15.28:0/294016822 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56269c593dc0 msgr2=0x56269c597960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:13:34.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.547+0000 7f530eda2740 1 --2- 172.21.15.28:0/294016822 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56269c593dc0 0x56269c597960 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f52fc00eed0 tx=0x7f52fc00dad0 comp rx=0 tx=0).stop 2024-05-12T22:13:34.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.547+0000 7f530eda2740 1 -- 172.21.15.28:0/294016822 shutdown_connections 2024-05-12T22:13:34.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.547+0000 7f530eda2740 1 --2- 172.21.15.28:0/294016822 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x56269c612ab0 0x56269c614ed0 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:34.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.547+0000 7f530eda2740 1 --2- 172.21.15.28:0/294016822 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56269c593dc0 0x56269c597960 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:34.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.547+0000 7f530eda2740 1 --2- 172.21.15.28:0/294016822 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56269c5896f0 0x56269c597420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:34.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.547+0000 7f530eda2740 1 --2- 172.21.15.28:0/294016822 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56269c588dd0 0x56269c59c010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:34.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.547+0000 7f530eda2740 1 -- 172.21.15.28:0/294016822 >> 172.21.15.28:0/294016822 conn(0x56269c5850f0 msgr2=0x56269c58a520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:13:34.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.547+0000 7f52eaa00640 0 client.26308 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:13:34.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.548+0000 7f530eda2740 1 -- 172.21.15.28:0/294016822 shutdown_connections 2024-05-12T22:13:34.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.548+0000 7f530eda2740 1 -- 172.21.15.28:0/294016822 wait complete. 2024-05-12T22:13:34.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.548+0000 7f530eda2740 20 client.26308 trim_cache size 0 max 16384 2024-05-12T22:13:34.554 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-05-12T22:13:34.554 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "scrub active (1112 inodes in the stack)", 2024-05-12T22:13:34.554 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-05-12T22:13:34.555 INFO:teuthology.orchestra.run.smithi028.stdout: "406ccb73-94f2-43f6-b42d-7e5336642117": { 2024-05-12T22:13:34.555 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-05-12T22:13:34.555 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "406ccb73-94f2-43f6-b42d-7e5336642117", 2024-05-12T22:13:34.555 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-05-12T22:13:34.555 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:13:34.555 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:13:34.555 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-05-12T22:13:34.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.549+0000 7f530aa00640 1 -- 172.21.15.28:0/3458690672 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f52ec0a2d80 msgr2=0x7f52ec0a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:13:34.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.549+0000 7f530aa00640 1 --2- 172.21.15.28:0/3458690672 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f52ec0a2d80 0x7f52ec0a5240 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x56269c58a390 tx=0x7f52f8005f90 comp rx=0 tx=0).stop 2024-05-12T22:13:34.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.549+0000 7f530aa00640 1 -- 172.21.15.28:0/3458690672 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f53041020c0 msgr2=0x7f5304074070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:13:34.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.549+0000 7f530aa00640 1 --2- 172.21.15.28:0/3458690672 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f53041020c0 0x7f5304074070 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f52f0035040 tx=0x7f52f0033b60 comp rx=0 tx=0).stop 2024-05-12T22:13:34.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.549+0000 7f530aa00640 1 -- 172.21.15.28:0/3458690672 shutdown_connections 2024-05-12T22:13:34.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.549+0000 7f530aa00640 1 --2- 172.21.15.28:0/3458690672 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f52ec0a2d80 0x7f52ec0a5240 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:34.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.549+0000 7f530aa00640 1 --2- 172.21.15.28:0/3458690672 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5304107710 0x7f53040745b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:34.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.549+0000 7f530aa00640 1 --2- 172.21.15.28:0/3458690672 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f53041020c0 0x7f5304074070 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:34.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.549+0000 7f530aa00640 1 --2- 172.21.15.28:0/3458690672 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5304101700 0x7f5304073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:13:34.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.549+0000 7f530aa00640 1 -- 172.21.15.28:0/3458690672 >> 172.21.15.28:0/3458690672 conn(0x7f53040fd1f0 msgr2=0x7f53040fd7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:13:34.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.549+0000 7f530aa00640 1 -- 172.21.15.28:0/3458690672 shutdown_connections 2024-05-12T22:13:34.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:13:34.549+0000 7f530aa00640 1 -- 172.21.15.28:0/3458690672 wait complete. 2024-05-12T22:13:34.571 INFO:tasks.cephfs.filesystem:scrub status for tag:406ccb73-94f2-43f6-b42d-7e5336642117 - {'path': '/', 'tag': '406ccb73-94f2-43f6-b42d-7e5336642117', 'options': 'recursive,force'} 2024-05-12T22:13:35.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:13:34 smithi186 ceph-mon[34785]: scrub queued for path: / 2024-05-12T22:13:35.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:13:34 smithi186 ceph-mon[34785]: scrub summary: idle+waiting paths [/] 2024-05-12T22:13:35.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:13:34 smithi186 ceph-mon[34785]: scrub summary: active paths [/] 2024-05-12T22:13:35.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:13:34 smithi086 ceph-mon[34297]: scrub queued for path: / 2024-05-12T22:13:35.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:13:34 smithi086 ceph-mon[34297]: scrub summary: idle+waiting paths [/] 2024-05-12T22:13:35.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:13:34 smithi086 ceph-mon[34297]: scrub summary: active paths [/] 2024-05-12T22:13:35.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:13:34 smithi028 ceph-mon[29847]: scrub queued for path: / 2024-05-12T22:13:35.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:13:34 smithi028 ceph-mon[29847]: scrub summary: idle+waiting paths [/] 2024-05-12T22:13:35.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:13:34 smithi028 ceph-mon[29847]: scrub summary: active paths [/] 2024-05-12T22:13:36.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:13:35 smithi186 ceph-mon[34785]: scrub summary: active paths [/] 2024-05-12T22:13:36.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:13:35 smithi186 ceph-mon[34785]: scrub summary: idle+waiting paths [/] 2024-05-12T22:13:36.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:13:35 smithi186 ceph-mon[34785]: scrub summary: active paths [/] 2024-05-12T22:13:36.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:13:35 smithi186 ceph-mon[34785]: scrub summary: idle+waiting paths [/] 2024-05-12T22:13:36.180 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:13:35 smithi186 ceph-mon[34785]: scrub summary: active paths [/] 2024-05-12T22:13:36.180 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:13:35 smithi186 ceph-mon[34785]: scrub summary: idle+waiting paths [/] 2024-05-12T22:13:36.180 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:13:35 smithi186 ceph-mon[34785]: scrub summary: idle+waiting paths [/] 2024-05-12T22:13:36.180 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:13:35 smithi186 ceph-mon[34785]: pgmap v141: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-05-12T22:13:36.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:13:35 smithi086 ceph-mon[34297]: scrub summary: active paths [/] 2024-05-12T22:13:36.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:13:35 smithi086 ceph-mon[34297]: scrub summary: idle+waiting paths [/] 2024-05-12T22:13:36.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:13:35 smithi086 ceph-mon[34297]: scrub summary: active paths [/] 2024-05-12T22:13:36.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:13:35 smithi086 ceph-mon[34297]: scrub summary: idle+waiting paths [/] 2024-05-12T22:13:36.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:13:35 smithi086 ceph-mon[34297]: scrub summary: active paths [/] 2024-05-12T22:13:36.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:13:35 smithi086 ceph-mon[34297]: scrub summary: idle+waiting paths [/] 2024-05-12T22:13:36.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:13:35 smithi086 ceph-mon[34297]: scrub summary: idle+waiting paths [/] 2024-05-12T22:13:36.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:13:35 smithi086 ceph-mon[34297]: pgmap v141: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-05-12T22:13:36.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:13:35 smithi028 ceph-mon[29847]: scrub summary: active paths [/] 2024-05-12T22:13:36.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:13:35 smithi028 ceph-mon[29847]: scrub summary: idle+waiting paths [/] 2024-05-12T22:13:36.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:13:35 smithi028 ceph-mon[29847]: scrub summary: active paths [/] 2024-05-12T22:13:36.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:13:35 smithi028 ceph-mon[29847]: scrub summary: idle+waiting paths [/] 2024-05-12T22:13:36.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:13:35 smithi028 ceph-mon[29847]: scrub summary: active paths [/] 2024-05-12T22:13:36.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:13:35 smithi028 ceph-mon[29847]: scrub summary: idle+waiting paths [/] 2024-05-12T22:13:36.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:13:35 smithi028 ceph-mon[29847]: scrub summary: idle+waiting paths [/] 2024-05-12T22:13:36.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:13:35 smithi028 ceph-mon[29847]: pgmap v141: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-05-12T22:13:38.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:13:37 smithi186 ceph-mon[34785]: pgmap v142: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-05-12T22:13:38.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:13:37 smithi086 ceph-mon[34297]: pgmap v142: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-05-12T22:13:38.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:13:37 smithi028 ceph-mon[29847]: pgmap v142: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-05-12T22:13:40.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:13:39 smithi186 ceph-mon[34785]: pgmap v143: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-05-12T22:13:40.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:13:39 smithi086 ceph-mon[34297]: pgmap v143: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-05-12T22:13:40.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:13:39 smithi028 ceph-mon[29847]: pgmap v143: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-05-12T22:13:42.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:13:41 smithi186 ceph-mon[34785]: pgmap v144: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-05-12T22:13:42.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:13:41 smithi086 ceph-mon[34297]: pgmap v144: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-05-12T22:13:42.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:13:41 smithi028 ceph-mon[29847]: pgmap v144: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-05-12T22:13:43.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:13:42 smithi186 ceph-mon[34785]: scrub summary: idle 2024-05-12T22:13:43.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:13:42 smithi186 ceph-mon[34785]: scrub summary: idle 2024-05-12T22:13:43.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:13:42 smithi086 ceph-mon[34297]: scrub summary: idle 2024-05-12T22:13:43.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:13:42 smithi086 ceph-mon[34297]: scrub summary: idle 2024-05-12T22:13:43.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:13:42 smithi028 ceph-mon[29847]: scrub summary: idle 2024-05-12T22:13:43.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:13:42 smithi028 ceph-mon[29847]: scrub summary: idle 2024-05-12T22:13:44.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:13:43 smithi186 ceph-mon[34785]: pgmap v145: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-05-12T22:13:44.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:13:43 smithi086 ceph-mon[34297]: pgmap v145: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-05-12T22:13:44.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:13:43 smithi028 ceph-mon[29847]: pgmap v145: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-05-12T22:13:46.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:13:45 smithi186 ceph-mon[34785]: pgmap v146: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-05-12T22:13:46.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:13:45 smithi086 ceph-mon[34297]: pgmap v146: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-05-12T22:13:46.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:13:45 smithi028 ceph-mon[29847]: pgmap v146: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-05-12T22:13:48.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:13:47 smithi186 ceph-mon[34785]: pgmap v147: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:13:48.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:13:47 smithi086 ceph-mon[34297]: pgmap v147: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:13:48.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:13:47 smithi028 ceph-mon[29847]: pgmap v147: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:13:50.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:13:49 smithi186 ceph-mon[34785]: pgmap v148: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:13:50.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:13:49 smithi086 ceph-mon[34297]: pgmap v148: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:13:50.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:13:49 smithi028 ceph-mon[29847]: pgmap v148: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:13:52.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:13:51 smithi086 ceph-mon[34297]: pgmap v149: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-12T22:13:52.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:13:51 smithi028 ceph-mon[29847]: pgmap v149: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-12T22:13:52.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:13:51 smithi186 ceph-mon[34785]: pgmap v149: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-12T22:13:54.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:13:53 smithi086 ceph-mon[34297]: pgmap v150: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:13:54.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:13:53 smithi028 ceph-mon[29847]: pgmap v150: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:13:54.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:13:53 smithi186 ceph-mon[34785]: pgmap v150: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:13:56.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:13:55 smithi086 ceph-mon[34297]: pgmap v151: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-12T22:13:56.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:13:55 smithi028 ceph-mon[29847]: pgmap v151: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-12T22:13:56.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:13:55 smithi186 ceph-mon[34785]: pgmap v151: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-12T22:13:58.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:13:57 smithi086 ceph-mon[34297]: pgmap v152: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:13:58.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:13:57 smithi028 ceph-mon[29847]: pgmap v152: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:13:58.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:13:57 smithi186 ceph-mon[34785]: pgmap v152: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:14:00.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:13:59 smithi086 ceph-mon[34297]: pgmap v153: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:14:00.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:13:59 smithi028 ceph-mon[29847]: pgmap v153: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:14:00.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:13:59 smithi186 ceph-mon[34785]: pgmap v153: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:14:02.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:14:01 smithi086 ceph-mon[34297]: pgmap v154: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-12T22:14:02.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:14:01 smithi028 ceph-mon[29847]: pgmap v154: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-12T22:14:02.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:14:01 smithi186 ceph-mon[34785]: pgmap v154: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-12T22:14:04.232 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:14:03 smithi186 ceph-mon[34785]: pgmap v155: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:14:04.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:14:03 smithi086 ceph-mon[34297]: pgmap v155: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:14:04.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:14:03 smithi028 ceph-mon[29847]: pgmap v155: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:14:04.572 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-12T22:14:04.572 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-12T22:14:04.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.800+0000 7fd903200640 1 -- 172.21.15.28:0/2991415077 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd904107dd0 msgr2=0x7fd904108230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:14:04.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.800+0000 7fd903200640 1 --2- 172.21.15.28:0/2991415077 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd904107dd0 0x7fd904108230 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fd8ec009900 tx=0x7fd8ec02a480 comp rx=0 tx=0).stop 2024-05-12T22:14:04.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.800+0000 7fd903200640 1 -- 172.21.15.28:0/2991415077 shutdown_connections 2024-05-12T22:14:04.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.800+0000 7fd903200640 1 --2- 172.21.15.28:0/2991415077 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd904108770 0x7fd904114b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:04.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.800+0000 7fd903200640 1 --2- 172.21.15.28:0/2991415077 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd904107dd0 0x7fd904108230 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:04.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.800+0000 7fd903200640 1 --2- 172.21.15.28:0/2991415077 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd90410ddf0 0x7fd90410e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:04.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.800+0000 7fd903200640 1 -- 172.21.15.28:0/2991415077 >> 172.21.15.28:0/2991415077 conn(0x7fd904077650 msgr2=0x7fd904106650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:14:04.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.800+0000 7fd903200640 1 -- 172.21.15.28:0/2991415077 shutdown_connections 2024-05-12T22:14:04.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.800+0000 7fd903200640 1 -- 172.21.15.28:0/2991415077 wait complete. 2024-05-12T22:14:04.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.801+0000 7fd903200640 1 Processor -- start 2024-05-12T22:14:04.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.801+0000 7fd903200640 1 -- start start 2024-05-12T22:14:04.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.802+0000 7fd903200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd904107dd0 0x7fd90419e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:04.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.802+0000 7fd903200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd904108770 0x7fd90419eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:04.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.802+0000 7fd903200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd90410ddf0 0x7fd9041a2de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:04.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.802+0000 7fd903200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd9041180b0 con 0x7fd904108770 2024-05-12T22:14:04.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.802+0000 7fd903200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fd904117f30 con 0x7fd90410ddf0 2024-05-12T22:14:04.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.802+0000 7fd903200640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd904118230 con 0x7fd904107dd0 2024-05-12T22:14:04.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.802+0000 7fd901400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd904108770 0x7fd90419eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:04.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.802+0000 7fd901400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd904108770 0x7fd90419eab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38192/0 (socket says 172.21.15.28:38192) 2024-05-12T22:14:04.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.802+0000 7fd901400640 1 -- 172.21.15.28:0/18041586 learned_addr learned my addr 172.21.15.28:0/18041586 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:14:04.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.802+0000 7fd901e00640 1 --2- 172.21.15.28:0/18041586 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd904107dd0 0x7fd90419e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:04.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.802+0000 7fd902800640 1 --2- 172.21.15.28:0/18041586 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd90410ddf0 0x7fd9041a2de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:04.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.802+0000 7fd901400640 1 -- 172.21.15.28:0/18041586 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd904107dd0 msgr2=0x7fd90419e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:14:04.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.802+0000 7fd901400640 1 --2- 172.21.15.28:0/18041586 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd904107dd0 0x7fd90419e570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:04.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.802+0000 7fd901400640 1 -- 172.21.15.28:0/18041586 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd90410ddf0 msgr2=0x7fd9041a2de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:14:04.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.802+0000 7fd901400640 1 --2- 172.21.15.28:0/18041586 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd90410ddf0 0x7fd9041a2de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:04.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.802+0000 7fd901400640 1 -- 172.21.15.28:0/18041586 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9041a3510 con 0x7fd904108770 2024-05-12T22:14:04.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.802+0000 7fd902800640 1 --2- 172.21.15.28:0/18041586 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd90410ddf0 0x7fd9041a2de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:14:04.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.803+0000 7fd901e00640 1 --2- 172.21.15.28:0/18041586 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd904107dd0 0x7fd90419e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:14:04.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.803+0000 7fd901400640 1 --2- 172.21.15.28:0/18041586 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd904108770 0x7fd90419eab0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fd8ec006010 tx=0x7fd8ec004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:14:04.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.803+0000 7fd8faa00640 1 -- 172.21.15.28:0/18041586 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8ec03f030 con 0x7fd904108770 2024-05-12T22:14:04.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.803+0000 7fd8faa00640 1 -- 172.21.15.28:0/18041586 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd8ec004440 con 0x7fd904108770 2024-05-12T22:14:04.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.803+0000 7fd8faa00640 1 -- 172.21.15.28:0/18041586 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8ec035070 con 0x7fd904108770 2024-05-12T22:14:04.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.803+0000 7fd903200640 1 -- 172.21.15.28:0/18041586 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9041a37a0 con 0x7fd904108770 2024-05-12T22:14:04.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.803+0000 7fd903200640 1 -- 172.21.15.28:0/18041586 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9041ab090 con 0x7fd904108770 2024-05-12T22:14:04.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.807+0000 7fd90bc1a740 1 Processor -- start 2024-05-12T22:14:04.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.807+0000 7fd8faa00640 1 -- 172.21.15.28:0/18041586 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7fd8ec042030 con 0x7fd904108770 2024-05-12T22:14:04.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.807+0000 7fd90bc1a740 1 -- start start 2024-05-12T22:14:04.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.807+0000 7fd90bc1a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614627e1dd0 0x5614627e21b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:04.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.807+0000 7fd90bc1a740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5614627e26f0 0x5614627ec880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:04.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.807+0000 7fd90bc1a740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5614627ecdc0 0x5614627ef1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:04.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.807+0000 7fd90bc1a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5614627aa2a0 con 0x5614627e1dd0 2024-05-12T22:14:04.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.807+0000 7fd90bc1a740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x561462746b90 con 0x5614627e26f0 2024-05-12T22:14:04.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.807+0000 7fd90bc1a740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5614626ee5b0 con 0x5614627ecdc0 2024-05-12T22:14:04.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.808+0000 7fd902800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614627e1dd0 0x5614627e21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:04.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.808+0000 7fd902800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614627e1dd0 0x5614627e21b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38202/0 (socket says 172.21.15.28:38202) 2024-05-12T22:14:04.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.808+0000 7fd902800640 1 -- 172.21.15.28:0/908381267 learned_addr learned my addr 172.21.15.28:0/908381267 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:14:04.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.808+0000 7fd901e00640 1 --2- 172.21.15.28:0/908381267 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5614627e26f0 0x5614627ec880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:04.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.808+0000 7fd901400640 1 --2- 172.21.15.28:0/908381267 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5614627ecdc0 0x5614627ef1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:04.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.808+0000 7fd902800640 1 -- 172.21.15.28:0/908381267 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5614627ecdc0 msgr2=0x5614627ef1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:14:04.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.808+0000 7fd902800640 1 --2- 172.21.15.28:0/908381267 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5614627ecdc0 0x5614627ef1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:04.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.808+0000 7fd902800640 1 -- 172.21.15.28:0/908381267 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5614627e26f0 msgr2=0x5614627ec880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:14:04.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.808+0000 7fd902800640 1 --2- 172.21.15.28:0/908381267 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5614627e26f0 0x5614627ec880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:04.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.808+0000 7fd902800640 1 -- 172.21.15.28:0/908381267 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5614625247d0 con 0x5614627e1dd0 2024-05-12T22:14:04.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.809+0000 7fd902800640 1 --2- 172.21.15.28:0/908381267 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614627e1dd0 0x5614627e21b0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fd8fc0113e0 tx=0x7fd8fc0118a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78291c10c540936a server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:14:04.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.809+0000 7fd8f3400640 1 -- 172.21.15.28:0/908381267 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8fc012770 con 0x5614627e1dd0 2024-05-12T22:14:04.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.809+0000 7fd8f3400640 1 -- 172.21.15.28:0/908381267 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd8fc012df0 con 0x5614627e1dd0 2024-05-12T22:14:04.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.809+0000 7fd8faa00640 1 --2- 172.21.15.28:0/18041586 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fd8e80a2d80 0x7fd8e80a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:04.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.809+0000 7fd8f3400640 1 -- 172.21.15.28:0/908381267 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8fc01a3f0 con 0x5614627e1dd0 2024-05-12T22:14:04.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.809+0000 7fd8faa00640 1 -- 172.21.15.28:0/18041586 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7fd8ec0ef8a0 con 0x7fd904108770 2024-05-12T22:14:04.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.809+0000 7fd901e00640 1 --2- 172.21.15.28:0/18041586 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fd8e80a2d80 0x7fd8e80a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:04.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.810+0000 7fd90bc1a740 1 -- 172.21.15.28:0/908381267 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614627e1dd0 msgr2=0x5614627e21b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:14:04.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.810+0000 7fd90bc1a740 1 --2- 172.21.15.28:0/908381267 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614627e1dd0 0x5614627e21b0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fd8fc0113e0 tx=0x7fd8fc0118a0 comp rx=0 tx=0).stop 2024-05-12T22:14:04.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.810+0000 7fd90bc1a740 1 -- 172.21.15.28:0/908381267 shutdown_connections 2024-05-12T22:14:04.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.810+0000 7fd90bc1a740 1 --2- 172.21.15.28:0/908381267 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5614627ecdc0 0x5614627ef1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:04.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.810+0000 7fd90bc1a740 1 --2- 172.21.15.28:0/908381267 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5614627e26f0 0x5614627ec880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:04.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.810+0000 7fd90bc1a740 1 --2- 172.21.15.28:0/908381267 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614627e1dd0 0x5614627e21b0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:04.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.810+0000 7fd90bc1a740 1 -- 172.21.15.28:0/908381267 >> 172.21.15.28:0/908381267 conn(0x5614627de0f0 msgr2=0x5614627de4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:14:04.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.810+0000 7fd901e00640 1 --2- 172.21.15.28:0/18041586 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fd8e80a2d80 0x7fd8e80a5240 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x5614627e3390 tx=0x7fd8f4005f90 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:14:04.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.810+0000 7fd90bc1a740 1 -- 172.21.15.28:0/908381267 shutdown_connections 2024-05-12T22:14:04.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.810+0000 7fd90bc1a740 1 -- 172.21.15.28:0/908381267 wait complete. 2024-05-12T22:14:04.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.811+0000 7fd90bc1a740 1 -- start start 2024-05-12T22:14:04.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.811+0000 7fd90bc1a740 1 Processor -- start 2024-05-12T22:14:04.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.811+0000 7fd90bc1a740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5614627e1dd0 0x5614627ec6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:04.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.811+0000 7fd90bc1a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614627e26f0 0x5614627e99a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:04.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.811+0000 7fd90bc1a740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5614627ecdc0 0x5614627e9ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:04.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.811+0000 7fd90bc1a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5614627019f0 con 0x5614627e26f0 2024-05-12T22:14:04.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.811+0000 7fd90bc1a740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x561462742b50 con 0x5614627e1dd0 2024-05-12T22:14:04.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.811+0000 7fd90bc1a740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5614627efbb0 con 0x5614627ecdc0 2024-05-12T22:14:04.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.811+0000 7fd90bc1a740 10 client.0 _ll_register_callbacks cb 0x561462459be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:14:04.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.812+0000 7fd901e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5614627e1dd0 0x5614627ec6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:04.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.812+0000 7fd901400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614627e26f0 0x5614627e99a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:04.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.812+0000 7fd901e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5614627e1dd0 0x5614627ec6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:35284/0 (socket says 172.21.15.28:35284) 2024-05-12T22:14:04.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.812+0000 7fd901e00640 1 -- 172.21.15.28:0/2993239109 learned_addr learned my addr 172.21.15.28:0/2993239109 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:14:04.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.812+0000 7fd902800640 1 --2- 172.21.15.28:0/2993239109 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5614627ecdc0 0x5614627e9ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:04.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.812+0000 7fd901400640 1 -- 172.21.15.28:0/2993239109 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5614627ecdc0 msgr2=0x5614627e9ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:14:04.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.812+0000 7fd901400640 1 --2- 172.21.15.28:0/2993239109 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5614627ecdc0 0x5614627e9ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:04.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.812+0000 7fd901400640 1 -- 172.21.15.28:0/2993239109 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5614627e1dd0 msgr2=0x5614627ec6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:14:04.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.812+0000 7fd901400640 1 --2- 172.21.15.28:0/2993239109 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5614627e1dd0 0x5614627ec6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:04.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.812+0000 7fd901400640 1 -- 172.21.15.28:0/2993239109 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56146245a3f0 con 0x5614627e26f0 2024-05-12T22:14:04.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.812+0000 7fd901400640 1 --2- 172.21.15.28:0/2993239109 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614627e26f0 0x5614627e99a0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fd8ec0ecfc0 tx=0x7fd8ec005b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ab6a731505b1646 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:14:04.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.812+0000 7fd8e7e00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-12T22:14:04.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.813+0000 7fd8e7e00640 1 -- 172.21.15.28:0/2993239109 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8ec049070 con 0x5614627e26f0 2024-05-12T22:14:04.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.813+0000 7fd8e7e00640 1 -- 172.21.15.28:0/2993239109 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7fd8ec0ed240 con 0x5614627e26f0 2024-05-12T22:14:04.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.813+0000 7fd90bc1a740 1 -- 172.21.15.28:0/2993239109 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd8fc012df0 con 0x5614627e26f0 2024-05-12T22:14:04.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.813+0000 7fd8e7e00640 1 -- 172.21.15.28:0/2993239109 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fd8ec0bb8d0 con 0x5614627e26f0 2024-05-12T22:14:04.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.813+0000 7fd90bc1a740 10 client.16236 fetch_fsmap learned FSMap version 26 2024-05-12T22:14:04.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.813+0000 7fd90bc1a740 1 -- 172.21.15.28:0/2993239109 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x56146241a0f0 con 0x5614627e26f0 2024-05-12T22:14:04.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.813+0000 7fd8e7e00640 1 -- 172.21.15.28:0/2993239109 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd8fc012df0 con 0x5614627e26f0 2024-05-12T22:14:04.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.813+0000 7fd8e7e00640 1 -- 172.21.15.28:0/2993239109 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fd8ec0c4570 con 0x5614627e26f0 2024-05-12T22:14:04.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.813+0000 7fd90bc1a740 10 client.16236 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:14:04.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.813+0000 7fd90bc1a740 10 client.16236 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:14:04.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.813+0000 7fd90bc1a740 20 client.16236 populate_metadata read hostname 'smithi028' 2024-05-12T22:14:04.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.814+0000 7fd90bc1a740 1 --2- 172.21.15.28:0/2993239109 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5614627fc8d0 0x5614627fece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:04.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.814+0000 7fd90bc1a740 4 client.16236 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-12T22:14:04.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.814+0000 7fd90bc1a740 1 -- 172.21.15.28:0/2993239109 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5614627542c0 con 0x5614627fc8d0 2024-05-12T22:14:04.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.814+0000 7fd901e00640 1 --2- 172.21.15.28:0/2993239109 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5614627fc8d0 0x5614627fece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:04.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.814+0000 7fd901e00640 1 --2- 172.21.15.28:0/2993239109 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5614627fc8d0 0x5614627fece0 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2ec92c8d77f3fcb1 server_cookie=dd6100f0a18a12e in_seq=0 out_seq=0 2024-05-12T22:14:04.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.814+0000 7fd8e7e00640 10 client.16236 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:14:04.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.815+0000 7fd8e7e00640 1 -- 172.21.15.28:0/2993239109 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5614627542c0 con 0x5614627fc8d0 2024-05-12T22:14:04.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.815+0000 7fd8e7e00640 10 client.16236 handle_command_reply: tid=0 2024-05-12T22:14:04.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.815+0000 7fd90bc1a740 1 client.16236 shutdown 2024-05-12T22:14:04.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.815+0000 7fd90bc1a740 1 -- 172.21.15.28:0/2993239109 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614627e26f0 msgr2=0x5614627e99a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:14:04.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.815+0000 7fd90bc1a740 1 --2- 172.21.15.28:0/2993239109 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614627e26f0 0x5614627e99a0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fd8ec0ecfc0 tx=0x7fd8ec005b50 comp rx=0 tx=0).stop 2024-05-12T22:14:04.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.816+0000 7fd90bc1a740 1 -- 172.21.15.28:0/2993239109 shutdown_connections 2024-05-12T22:14:04.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.816+0000 7fd90bc1a740 1 --2- 172.21.15.28:0/2993239109 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5614627fc8d0 0x5614627fece0 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:04.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.816+0000 7fd90bc1a740 1 --2- 172.21.15.28:0/2993239109 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5614627ecdc0 0x5614627e9ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:04.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.816+0000 7fd90bc1a740 1 --2- 172.21.15.28:0/2993239109 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614627e26f0 0x5614627e99a0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:04.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.816+0000 7fd90bc1a740 1 --2- 172.21.15.28:0/2993239109 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5614627e1dd0 0x5614627ec6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:04.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.816+0000 7fd8e7e00640 0 client.16236 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:14:04.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.816+0000 7fd90bc1a740 1 -- 172.21.15.28:0/2993239109 >> 172.21.15.28:0/2993239109 conn(0x5614627de0f0 msgr2=0x5614627dd330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:14:04.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.816+0000 7fd90bc1a740 1 -- 172.21.15.28:0/2993239109 shutdown_connections 2024-05-12T22:14:04.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.816+0000 7fd90bc1a740 1 -- 172.21.15.28:0/2993239109 wait complete. 2024-05-12T22:14:04.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.816+0000 7fd90bc1a740 20 client.16236 trim_cache size 0 max 16384 2024-05-12T22:14:04.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.835+0000 7fd90bc1a740 1 Processor -- start 2024-05-12T22:14:04.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.836+0000 7fd90bc1a740 1 -- start start 2024-05-12T22:14:04.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.836+0000 7fd90bc1a740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5614627e1dd0 0x5614627f8980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:04.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.836+0000 7fd90bc1a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614627e26f0 0x561462804580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:04.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.836+0000 7fd90bc1a740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5614627ecdc0 0x561462804ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:04.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.836+0000 7fd90bc1a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd9041b6040 con 0x5614627e26f0 2024-05-12T22:14:04.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.836+0000 7fd90bc1a740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fd9041b3910 con 0x5614627e1dd0 2024-05-12T22:14:04.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.836+0000 7fd90bc1a740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd9041b1760 con 0x5614627ecdc0 2024-05-12T22:14:04.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.836+0000 7fd901400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614627e26f0 0x561462804580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:04.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.836+0000 7fd901400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614627e26f0 0x561462804580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38216/0 (socket says 172.21.15.28:38216) 2024-05-12T22:14:04.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.836+0000 7fd901400640 1 -- 172.21.15.28:0/1684152616 learned_addr learned my addr 172.21.15.28:0/1684152616 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:14:04.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.836+0000 7fd901e00640 1 --2- 172.21.15.28:0/1684152616 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5614627e1dd0 0x5614627f8980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:04.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.836+0000 7fd902800640 1 --2- 172.21.15.28:0/1684152616 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5614627ecdc0 0x561462804ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:04.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.837+0000 7fd901400640 1 -- 172.21.15.28:0/1684152616 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5614627ecdc0 msgr2=0x561462804ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:14:04.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.837+0000 7fd901400640 1 --2- 172.21.15.28:0/1684152616 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5614627ecdc0 0x561462804ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:04.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.837+0000 7fd901400640 1 -- 172.21.15.28:0/1684152616 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5614627e1dd0 msgr2=0x5614627f8980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:14:04.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.837+0000 7fd901400640 1 --2- 172.21.15.28:0/1684152616 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5614627e1dd0 0x5614627f8980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:04.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.837+0000 7fd901400640 1 -- 172.21.15.28:0/1684152616 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5614627e7f10 con 0x5614627e26f0 2024-05-12T22:14:04.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.837+0000 7fd901e00640 1 --2- 172.21.15.28:0/1684152616 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5614627e1dd0 0x5614627f8980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:14:04.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.837+0000 7fd901400640 1 --2- 172.21.15.28:0/1684152616 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614627e26f0 0x561462804580 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fd8ec0efc60 tx=0x7fd8ec0d4f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9599712fc348acdf server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:14:04.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.837+0000 7fd8f2a00640 1 -- 172.21.15.28:0/1684152616 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8ec049070 con 0x5614627e26f0 2024-05-12T22:14:04.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.837+0000 7fd8f2a00640 1 -- 172.21.15.28:0/1684152616 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd8ec0d8020 con 0x5614627e26f0 2024-05-12T22:14:04.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.837+0000 7fd8f2a00640 1 -- 172.21.15.28:0/1684152616 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8ec0bb6f0 con 0x5614627e26f0 2024-05-12T22:14:04.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.838+0000 7fd90bc1a740 1 -- 172.21.15.28:0/1684152616 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614627e26f0 msgr2=0x561462804580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:14:04.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.838+0000 7fd90bc1a740 1 --2- 172.21.15.28:0/1684152616 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614627e26f0 0x561462804580 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fd8ec0efc60 tx=0x7fd8ec0d4f90 comp rx=0 tx=0).stop 2024-05-12T22:14:04.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.838+0000 7fd90bc1a740 1 -- 172.21.15.28:0/1684152616 shutdown_connections 2024-05-12T22:14:04.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.838+0000 7fd90bc1a740 1 --2- 172.21.15.28:0/1684152616 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5614627ecdc0 0x561462804ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:04.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.838+0000 7fd90bc1a740 1 --2- 172.21.15.28:0/1684152616 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614627e26f0 0x561462804580 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:04.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.838+0000 7fd90bc1a740 1 --2- 172.21.15.28:0/1684152616 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5614627e1dd0 0x5614627f8980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:04.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.838+0000 7fd90bc1a740 1 -- 172.21.15.28:0/1684152616 >> 172.21.15.28:0/1684152616 conn(0x5614627de0f0 msgr2=0x5614627dd330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:14:04.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.838+0000 7fd90bc1a740 1 -- 172.21.15.28:0/1684152616 shutdown_connections 2024-05-12T22:14:04.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.838+0000 7fd90bc1a740 1 -- 172.21.15.28:0/1684152616 wait complete. 2024-05-12T22:14:04.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.839+0000 7fd90bc1a740 1 -- start start 2024-05-12T22:14:04.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.839+0000 7fd90bc1a740 1 Processor -- start 2024-05-12T22:14:04.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.839+0000 7fd90bc1a740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5614627e1dd0 0x5614627f5120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:04.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.839+0000 7fd90bc1a740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5614627e26f0 0x5614627f2500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:04.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.839+0000 7fd90bc1a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614627ecdc0 0x5614627f2a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:04.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.839+0000 7fd90bc1a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd9041b1dd0 con 0x5614627ecdc0 2024-05-12T22:14:04.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.839+0000 7fd90bc1a740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fd9041b3240 con 0x5614627e1dd0 2024-05-12T22:14:04.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.839+0000 7fd90bc1a740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd9041b24a0 con 0x5614627e26f0 2024-05-12T22:14:04.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.839+0000 7fd90bc1a740 10 client.0 _ll_register_callbacks cb 0x7fd8ec032b60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:14:04.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.840+0000 7fd902800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614627ecdc0 0x5614627f2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:04.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.840+0000 7fd901400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5614627e26f0 0x5614627f2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:04.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.840+0000 7fd901e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5614627e1dd0 0x5614627f5120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:04.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.840+0000 7fd902800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614627ecdc0 0x5614627f2a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38232/0 (socket says 172.21.15.28:38232) 2024-05-12T22:14:04.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.840+0000 7fd902800640 1 -- 172.21.15.28:0/686909230 learned_addr learned my addr 172.21.15.28:0/686909230 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:14:04.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.840+0000 7fd902800640 1 -- 172.21.15.28:0/686909230 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5614627e26f0 msgr2=0x5614627f2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:14:04.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.840+0000 7fd902800640 1 --2- 172.21.15.28:0/686909230 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5614627e26f0 0x5614627f2500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:04.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.840+0000 7fd902800640 1 -- 172.21.15.28:0/686909230 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5614627e1dd0 msgr2=0x5614627f5120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:14:04.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.840+0000 7fd902800640 1 --2- 172.21.15.28:0/686909230 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5614627e1dd0 0x5614627f5120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:04.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.840+0000 7fd902800640 1 -- 172.21.15.28:0/686909230 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5614627f3130 con 0x5614627ecdc0 2024-05-12T22:14:04.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.840+0000 7fd901e00640 1 --2- 172.21.15.28:0/686909230 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5614627e1dd0 0x5614627f5120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:14:04.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.840+0000 7fd902800640 1 --2- 172.21.15.28:0/686909230 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614627ecdc0 0x5614627f2a40 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fd8fc005bf0 tx=0x7fd8fc004bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be32a450e4bbc2b3 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:14:04.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.840+0000 7fd8e7e00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-12T22:14:04.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.841+0000 7fd8e7e00640 1 -- 172.21.15.28:0/686909230 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8fc00f850 con 0x5614627ecdc0 2024-05-12T22:14:04.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.841+0000 7fd8e7e00640 1 -- 172.21.15.28:0/686909230 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7fd8fc012570 con 0x5614627ecdc0 2024-05-12T22:14:04.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.841+0000 7fd90bc1a740 1 -- 172.21.15.28:0/686909230 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd8ec0d8020 con 0x5614627ecdc0 2024-05-12T22:14:04.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.841+0000 7fd8e7e00640 1 -- 172.21.15.28:0/686909230 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fd8fc004e20 con 0x5614627ecdc0 2024-05-12T22:14:04.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.841+0000 7fd90bc1a740 10 client.16242 fetch_fsmap learned FSMap version 26 2024-05-12T22:14:04.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.841+0000 7fd90bc1a740 1 -- 172.21.15.28:0/686909230 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x561462862cd0 con 0x5614627ecdc0 2024-05-12T22:14:04.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.841+0000 7fd8e7e00640 1 -- 172.21.15.28:0/686909230 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd8ec0d8020 con 0x5614627ecdc0 2024-05-12T22:14:04.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.842+0000 7fd8e7e00640 1 -- 172.21.15.28:0/686909230 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fd8fc02f850 con 0x5614627ecdc0 2024-05-12T22:14:04.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.842+0000 7fd90bc1a740 10 client.16242 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:14:04.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.842+0000 7fd90bc1a740 10 client.16242 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:14:04.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.842+0000 7fd90bc1a740 20 client.16242 populate_metadata read hostname 'smithi028' 2024-05-12T22:14:04.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.842+0000 7fd90bc1a740 1 --2- 172.21.15.28:0/686909230 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x561462863190 0x5614628655b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:04.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.842+0000 7fd90bc1a740 4 client.16242 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-12T22:14:04.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.842+0000 7fd90bc1a740 1 -- 172.21.15.28:0/686909230 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56146274e610 con 0x561462863190 2024-05-12T22:14:04.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.842+0000 7fd901e00640 1 --2- 172.21.15.28:0/686909230 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x561462863190 0x5614628655b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:04.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.843+0000 7fd901e00640 1 --2- 172.21.15.28:0/686909230 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x561462863190 0x5614628655b0 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=77e05fe8f7f1b78f server_cookie=e2c6ede2593f2414 in_seq=0 out_seq=0 2024-05-12T22:14:04.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.843+0000 7fd8e7e00640 10 client.16242 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:14:04.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.843+0000 7fd8e7e00640 1 -- 172.21.15.28:0/686909230 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x56146274e610 con 0x561462863190 2024-05-12T22:14:04.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.843+0000 7fd8e7e00640 10 client.16242 handle_command_reply: tid=0 2024-05-12T22:14:04.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.843+0000 7fd90bc1a740 1 client.16242 shutdown 2024-05-12T22:14:04.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.843+0000 7fd90bc1a740 1 -- 172.21.15.28:0/686909230 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614627ecdc0 msgr2=0x5614627f2a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:14:04.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.843+0000 7fd90bc1a740 1 --2- 172.21.15.28:0/686909230 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614627ecdc0 0x5614627f2a40 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fd8fc005bf0 tx=0x7fd8fc004bf0 comp rx=0 tx=0).stop 2024-05-12T22:14:04.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.843+0000 7fd90bc1a740 1 -- 172.21.15.28:0/686909230 shutdown_connections 2024-05-12T22:14:04.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.843+0000 7fd90bc1a740 1 --2- 172.21.15.28:0/686909230 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x561462863190 0x5614628655b0 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:04.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.843+0000 7fd90bc1a740 1 --2- 172.21.15.28:0/686909230 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614627ecdc0 0x5614627f2a40 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:04.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.843+0000 7fd90bc1a740 1 --2- 172.21.15.28:0/686909230 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5614627e26f0 0x5614627f2500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:04.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.843+0000 7fd90bc1a740 1 --2- 172.21.15.28:0/686909230 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5614627e1dd0 0x5614627f5120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:04.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.843+0000 7fd8e7e00640 0 client.16242 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:14:04.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.843+0000 7fd90bc1a740 1 -- 172.21.15.28:0/686909230 >> 172.21.15.28:0/686909230 conn(0x5614627de0f0 msgr2=0x5614627eb1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:14:04.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.843+0000 7fd90bc1a740 1 -- 172.21.15.28:0/686909230 shutdown_connections 2024-05-12T22:14:04.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.843+0000 7fd90bc1a740 1 -- 172.21.15.28:0/686909230 wait complete. 2024-05-12T22:14:04.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.843+0000 7fd90bc1a740 20 client.16242 trim_cache size 0 max 16384 2024-05-12T22:14:04.853 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-05-12T22:14:04.854 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-05-12T22:14:04.854 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-05-12T22:14:04.854 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-05-12T22:14:04.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.844+0000 7fd903200640 1 -- 172.21.15.28:0/18041586 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fd8e80a2d80 msgr2=0x7fd8e80a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:14:04.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.844+0000 7fd903200640 1 --2- 172.21.15.28:0/18041586 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fd8e80a2d80 0x7fd8e80a5240 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x5614627e3390 tx=0x7fd8f4005f90 comp rx=0 tx=0).stop 2024-05-12T22:14:04.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.844+0000 7fd903200640 1 -- 172.21.15.28:0/18041586 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd904108770 msgr2=0x7fd90419eab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:14:04.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.844+0000 7fd903200640 1 --2- 172.21.15.28:0/18041586 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd904108770 0x7fd90419eab0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fd8ec006010 tx=0x7fd8ec004040 comp rx=0 tx=0).stop 2024-05-12T22:14:04.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.844+0000 7fd903200640 1 -- 172.21.15.28:0/18041586 shutdown_connections 2024-05-12T22:14:04.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.844+0000 7fd903200640 1 --2- 172.21.15.28:0/18041586 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fd8e80a2d80 0x7fd8e80a5240 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:04.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.844+0000 7fd903200640 1 --2- 172.21.15.28:0/18041586 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd90410ddf0 0x7fd9041a2de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:04.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.844+0000 7fd903200640 1 --2- 172.21.15.28:0/18041586 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd904108770 0x7fd90419eab0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:04.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.844+0000 7fd903200640 1 --2- 172.21.15.28:0/18041586 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd904107dd0 0x7fd90419e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:04.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.844+0000 7fd903200640 1 -- 172.21.15.28:0/18041586 >> 172.21.15.28:0/18041586 conn(0x7fd904077650 msgr2=0x7fd904078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:14:04.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.844+0000 7fd903200640 1 -- 172.21.15.28:0/18041586 shutdown_connections 2024-05-12T22:14:04.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:04.844+0000 7fd903200640 1 -- 172.21.15.28:0/18041586 wait complete. 2024-05-12T22:14:04.864 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-12T22:14:04.865 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-12T22:14:05.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.076+0000 7fedb5c00640 1 -- 172.21.15.28:0/4039256640 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fedb01027e0 msgr2=0x7fedb0102bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:14:05.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.076+0000 7fedb5c00640 1 --2- 172.21.15.28:0/4039256640 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fedb01027e0 0x7fedb0102bc0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7feda0009ca0 tx=0x7feda002a950 comp rx=0 tx=0).stop 2024-05-12T22:14:05.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.076+0000 7fedb5c00640 1 -- 172.21.15.28:0/4039256640 shutdown_connections 2024-05-12T22:14:05.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.076+0000 7fedb5c00640 1 --2- 172.21.15.28:0/4039256640 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fedb010bc00 0x7fedb010e0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:05.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.076+0000 7fedb5c00640 1 --2- 172.21.15.28:0/4039256640 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fedb0103100 0x7fedb010b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:05.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.076+0000 7fedb5c00640 1 --2- 172.21.15.28:0/4039256640 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fedb01027e0 0x7fedb0102bc0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:05.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.076+0000 7fedb5c00640 1 -- 172.21.15.28:0/4039256640 >> 172.21.15.28:0/4039256640 conn(0x7fedb00fbfe0 msgr2=0x7fedb00fe400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:14:05.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.076+0000 7fedb5c00640 1 -- 172.21.15.28:0/4039256640 shutdown_connections 2024-05-12T22:14:05.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.076+0000 7fedb5c00640 1 -- 172.21.15.28:0/4039256640 wait complete. 2024-05-12T22:14:05.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.077+0000 7fedb5c00640 1 Processor -- start 2024-05-12T22:14:05.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.077+0000 7fedb5c00640 1 -- start start 2024-05-12T22:14:05.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.078+0000 7fedb5c00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fedb01027e0 0x7fedb0197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:05.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.078+0000 7fedb5c00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fedb0103100 0x7fedb01984b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:05.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.078+0000 7fedb5c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fedb010bc00 0x7fedb019c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:05.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.078+0000 7fedb5c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fedb0111a50 con 0x7fedb010bc00 2024-05-12T22:14:05.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.078+0000 7fedb5c00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fedb01118d0 con 0x7fedb0103100 2024-05-12T22:14:05.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.078+0000 7fedb5c00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fedb0111bd0 con 0x7fedb01027e0 2024-05-12T22:14:05.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.078+0000 7fedafe00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fedb01027e0 0x7fedb0197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:05.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.078+0000 7fedafe00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fedb01027e0 0x7fedb0197f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:55216/0 (socket says 172.21.15.28:55216) 2024-05-12T22:14:05.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.078+0000 7fedafe00640 1 -- 172.21.15.28:0/4022679370 learned_addr learned my addr 172.21.15.28:0/4022679370 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:14:05.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.078+0000 7fedb5200640 1 --2- 172.21.15.28:0/4022679370 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fedb010bc00 0x7fedb019c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:05.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.078+0000 7feda7400640 1 --2- 172.21.15.28:0/4022679370 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fedb0103100 0x7fedb01984b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:05.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.078+0000 7fedafe00640 1 -- 172.21.15.28:0/4022679370 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fedb0103100 msgr2=0x7fedb01984b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:14:05.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.078+0000 7fedafe00640 1 --2- 172.21.15.28:0/4022679370 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fedb0103100 0x7fedb01984b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:05.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.078+0000 7fedafe00640 1 -- 172.21.15.28:0/4022679370 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fedb010bc00 msgr2=0x7fedb019c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:14:05.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.078+0000 7fedafe00640 1 --2- 172.21.15.28:0/4022679370 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fedb010bc00 0x7fedb019c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:05.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.078+0000 7fedafe00640 1 -- 172.21.15.28:0/4022679370 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fedb019cf10 con 0x7fedb01027e0 2024-05-12T22:14:05.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.078+0000 7fedb5200640 1 --2- 172.21.15.28:0/4022679370 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fedb010bc00 0x7fedb019c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:14:05.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.078+0000 7feda7400640 1 --2- 172.21.15.28:0/4022679370 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fedb0103100 0x7fedb01984b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:14:05.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.079+0000 7fedafe00640 1 --2- 172.21.15.28:0/4022679370 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fedb01027e0 0x7fedb0197f70 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7feda0005e30 tx=0x7feda0003c90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:14:05.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.079+0000 7fedad600640 1 -- 172.21.15.28:0/4022679370 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feda0034040 con 0x7fedb01027e0 2024-05-12T22:14:05.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.079+0000 7fedb5c00640 1 -- 172.21.15.28:0/4022679370 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fedb019d1a0 con 0x7fedb01027e0 2024-05-12T22:14:05.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.079+0000 7fedad600640 1 -- 172.21.15.28:0/4022679370 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feda002fe20 con 0x7fedb01027e0 2024-05-12T22:14:05.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.079+0000 7fedad600640 1 -- 172.21.15.28:0/4022679370 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feda00047c0 con 0x7fedb01027e0 2024-05-12T22:14:05.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.079+0000 7fedb5c00640 1 -- 172.21.15.28:0/4022679370 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fedb01a4a90 con 0x7fedb01027e0 2024-05-12T22:14:05.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.080+0000 7fedb5c00640 1 -- 172.21.15.28:0/4022679370 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fedb006a6f0 con 0x7fedb01027e0 2024-05-12T22:14:05.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.082+0000 7fedad600640 1 -- 172.21.15.28:0/4022679370 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7feda0037020 con 0x7fedb01027e0 2024-05-12T22:14:05.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.084+0000 7fedad600640 1 --2- 172.21.15.28:0/4022679370 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fed940a2d80 0x7fed940a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:05.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.084+0000 7fedad600640 1 -- 172.21.15.28:0/4022679370 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7feda00f0260 con 0x7fedb01027e0 2024-05-12T22:14:05.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.084+0000 7feda7400640 1 --2- 172.21.15.28:0/4022679370 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fed940a2d80 0x7fed940a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:05.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.085+0000 7feda7400640 1 --2- 172.21.15.28:0/4022679370 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fed940a2d80 0x7fed940a5240 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fedb00fe670 tx=0x7fed98005f90 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:14:05.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.088+0000 7fedad600640 1 -- 172.21.15.28:0/4022679370 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7feda00bcf60 con 0x7fedb01027e0 2024-05-12T22:14:05.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.334+0000 7fedb5c00640 1 -- 172.21.15.28:0/4022679370 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fedb00689c0 con 0x7fedb01027e0 2024-05-12T22:14:05.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.335+0000 7fedad600640 1 -- 172.21.15.28:0/4022679370 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11864 (secure 0 0 0) 0x7feda00bc860 con 0x7fedb01027e0 2024-05-12T22:14:05.336 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-12T22:14:05.336 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1844141745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1844141745},{"type":"v1","addr":"172.21.15.186:6833","nonce":1844141745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6833/2960763758","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":2960763758},{"type":"v1","addr":"172.21.15.86:6833","nonce":2960763758}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24470,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/4131134219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":4131134219},{"type":"v1","addr":"172.21.15.28:6835","nonce":4131134219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24481,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/1727869064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1727869064},{"type":"v1","addr":"172.21.15.186:6835","nonce":1727869064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24482,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2938260704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2938260704},{"type":"v1","addr":"172.21.15.28:6837","nonce":2938260704}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24494,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3473700910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3473700910},{"type":"v1","addr":"172.21.15.86:6835","nonce":3473700910}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-12T22:06:33.426986+0000","modified":"2024-05-12T22:13:22.499381+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24581,"mds_1":24550,"mds_2":24532},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24499":{"gid":24499,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6837/3600809235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":3600809235},{"type":"v1","addr":"172.21.15.186:6837","nonce":3600809235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"i","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6839/2399610807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":2399610807},{"type":"v1","addr":"172.21.15.186:6839","nonce":2399610807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24530":{"gid":24530,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.86:6837/3782448884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":3782448884},{"type":"v1","addr":"172.21.15.86:6837","nonce":3782448884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24532":{"gid":24532,"name":"g","rank":2,"incarnation":21,"state":"up:active","state_seq":6,"addr":"172.21.15.28:6839/123892158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":123892158},{"type":"v1","addr":"172.21.15.28:6839","nonce":123892158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24550":{"gid":24550,"name":"a","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.28:6841/761183870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":761183870},{"type":"v1","addr":"172.21.15.28:6841","nonce":761183870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24581":{"gid":24581,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6839/1251999130","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6838","nonce":1251999130},{"type":"v1","addr":"172.21.15.86:6839","nonce":1251999130}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24581,"qdb_cluster":[24550,24581,24532]},"id":1}]} 2024-05-12T22:14:05.337 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 26 2024-05-12T22:14:05.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.338+0000 7fedb5c00640 1 -- 172.21.15.28:0/4022679370 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fed940a2d80 msgr2=0x7fed940a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:14:05.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.338+0000 7fedb5c00640 1 --2- 172.21.15.28:0/4022679370 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fed940a2d80 0x7fed940a5240 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fedb00fe670 tx=0x7fed98005f90 comp rx=0 tx=0).stop 2024-05-12T22:14:05.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.338+0000 7fedb5c00640 1 -- 172.21.15.28:0/4022679370 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fedb01027e0 msgr2=0x7fedb0197f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:14:05.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.338+0000 7fedb5c00640 1 --2- 172.21.15.28:0/4022679370 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fedb01027e0 0x7fedb0197f70 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7feda0005e30 tx=0x7feda0003c90 comp rx=0 tx=0).stop 2024-05-12T22:14:05.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.338+0000 7fedb5c00640 1 -- 172.21.15.28:0/4022679370 shutdown_connections 2024-05-12T22:14:05.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.338+0000 7fedb5c00640 1 --2- 172.21.15.28:0/4022679370 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fed940a2d80 0x7fed940a5240 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:05.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.338+0000 7fedb5c00640 1 --2- 172.21.15.28:0/4022679370 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fedb010bc00 0x7fedb019c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:05.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.338+0000 7fedb5c00640 1 --2- 172.21.15.28:0/4022679370 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fedb0103100 0x7fedb01984b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:05.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.338+0000 7fedb5c00640 1 --2- 172.21.15.28:0/4022679370 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fedb01027e0 0x7fedb0197f70 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:05.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.338+0000 7fedb5c00640 1 -- 172.21.15.28:0/4022679370 >> 172.21.15.28:0/4022679370 conn(0x7fedb00fbfe0 msgr2=0x7fedb00fd810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:14:05.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.338+0000 7fedb5c00640 1 -- 172.21.15.28:0/4022679370 shutdown_connections 2024-05-12T22:14:05.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.338+0000 7fedb5c00640 1 -- 172.21.15.28:0/4022679370 wait complete. 2024-05-12T22:14:05.359 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-05-12T22:14:05.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.566+0000 7f6c7b800640 1 -- 172.21.15.28:0/518314067 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6c741043d0 msgr2=0x7f6c741108e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:14:05.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.566+0000 7f6c7b800640 1 --2- 172.21.15.28:0/518314067 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6c741043d0 0x7f6c741108e0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f6c6c009cd0 tx=0x7f6c6c02a8f0 comp rx=0 tx=0).stop 2024-05-12T22:14:05.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.566+0000 7f6c7b800640 1 -- 172.21.15.28:0/518314067 shutdown_connections 2024-05-12T22:14:05.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.566+0000 7f6c7b800640 1 --2- 172.21.15.28:0/518314067 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6c741043d0 0x7f6c741108e0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:05.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.566+0000 7f6c7b800640 1 --2- 172.21.15.28:0/518314067 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6c74103a10 0x7f6c74103e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:05.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.566+0000 7f6c7b800640 1 --2- 172.21.15.28:0/518314067 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6c74109a20 0x7f6c74109e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:05.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.566+0000 7f6c7b800640 1 -- 172.21.15.28:0/518314067 >> 172.21.15.28:0/518314067 conn(0x7f6c740ff4a0 msgr2=0x7f6c741018c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:14:05.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.567+0000 7f6c7b800640 1 -- 172.21.15.28:0/518314067 shutdown_connections 2024-05-12T22:14:05.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.567+0000 7f6c7b800640 1 -- 172.21.15.28:0/518314067 wait complete. 2024-05-12T22:14:05.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.568+0000 7f6c7b800640 1 Processor -- start 2024-05-12T22:14:05.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.568+0000 7f6c7b800640 1 -- start start 2024-05-12T22:14:05.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.568+0000 7f6c7b800640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6c74103a10 0x7f6c7419a300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:05.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.568+0000 7f6c7b800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6c741043d0 0x7f6c7419a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:05.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.569+0000 7f6c7b800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6c74109a20 0x7f6c7419ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:05.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.569+0000 7f6c7b800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6c74113e00 con 0x7f6c741043d0 2024-05-12T22:14:05.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.569+0000 7f6c7b800640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f6c74113c80 con 0x7f6c74109a20 2024-05-12T22:14:05.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.569+0000 7f6c7b800640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f6c74113f80 con 0x7f6c74103a10 2024-05-12T22:14:05.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.569+0000 7f6c78e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6c74103a10 0x7f6c7419a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:05.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.569+0000 7f6c6be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6c741043d0 0x7f6c7419a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:05.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.569+0000 7f6c78e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6c74103a10 0x7f6c7419a300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:55224/0 (socket says 172.21.15.28:55224) 2024-05-12T22:14:05.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.569+0000 7f6c78e00640 1 -- 172.21.15.28:0/2124192625 learned_addr learned my addr 172.21.15.28:0/2124192625 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:14:05.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.569+0000 7f6c79800640 1 --2- 172.21.15.28:0/2124192625 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6c74109a20 0x7f6c7419ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:05.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.569+0000 7f6c78e00640 1 -- 172.21.15.28:0/2124192625 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6c74109a20 msgr2=0x7f6c7419ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:14:05.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.569+0000 7f6c78e00640 1 --2- 172.21.15.28:0/2124192625 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6c74109a20 0x7f6c7419ebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:05.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.569+0000 7f6c78e00640 1 -- 172.21.15.28:0/2124192625 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6c741043d0 msgr2=0x7f6c7419a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:14:05.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.569+0000 7f6c78e00640 1 --2- 172.21.15.28:0/2124192625 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6c741043d0 0x7f6c7419a840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:05.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.569+0000 7f6c78e00640 1 -- 172.21.15.28:0/2124192625 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c7419f2f0 con 0x7f6c74103a10 2024-05-12T22:14:05.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.569+0000 7f6c79800640 1 --2- 172.21.15.28:0/2124192625 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6c74109a20 0x7f6c7419ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:14:05.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.570+0000 7f6c78e00640 1 --2- 172.21.15.28:0/2124192625 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6c74103a10 0x7f6c7419a300 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f6c6400e9f0 tx=0x7f6c6400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:14:05.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.570+0000 7f6c72000640 1 -- 172.21.15.28:0/2124192625 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c6400cd80 con 0x7f6c74103a10 2024-05-12T22:14:05.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.570+0000 7f6c7b800640 1 -- 172.21.15.28:0/2124192625 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c7419f5e0 con 0x7f6c74103a10 2024-05-12T22:14:05.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.570+0000 7f6c72000640 1 -- 172.21.15.28:0/2124192625 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6c64010910 con 0x7f6c74103a10 2024-05-12T22:14:05.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.570+0000 7f6c72000640 1 -- 172.21.15.28:0/2124192625 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c640188f0 con 0x7f6c74103a10 2024-05-12T22:14:05.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.570+0000 7f6c7b800640 1 -- 172.21.15.28:0/2124192625 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c741a6e70 con 0x7f6c74103a10 2024-05-12T22:14:05.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.573+0000 7f6c7e7c7740 1 Processor -- start 2024-05-12T22:14:05.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.574+0000 7f6c7e7c7740 1 -- start start 2024-05-12T22:14:05.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.574+0000 7f6c72000640 1 -- 172.21.15.28:0/2124192625 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7f6c64010ac0 con 0x7f6c74103a10 2024-05-12T22:14:05.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.574+0000 7f6c7e7c7740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e8de004980 0x55e8de004d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:05.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.574+0000 7f6c7e7c7740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e8de0052a0 0x55e8de00f430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:05.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.574+0000 7f6c7e7c7740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e8de00f970 0x55e8de011d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:05.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.574+0000 7f6c7e7c7740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e8ddf69a30 con 0x55e8de0052a0 2024-05-12T22:14:05.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.574+0000 7f6c7e7c7740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x55e8ddf11700 con 0x55e8de004980 2024-05-12T22:14:05.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.574+0000 7f6c7e7c7740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55e8ddf659f0 con 0x55e8de00f970 2024-05-12T22:14:05.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.574+0000 7f6c78e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e8de0052a0 0x55e8de00f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:05.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.574+0000 7f6c79800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e8de004980 0x55e8de004d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:05.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.574+0000 7f6c6be00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e8de00f970 0x55e8de011d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:05.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.574+0000 7f6c79800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e8de004980 0x55e8de004d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:35374/0 (socket says 172.21.15.28:35374) 2024-05-12T22:14:05.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.574+0000 7f6c79800640 1 -- 172.21.15.28:0/4050665794 learned_addr learned my addr 172.21.15.28:0/4050665794 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:14:05.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.575+0000 7f6c79800640 1 -- 172.21.15.28:0/4050665794 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e8de00f970 msgr2=0x55e8de011d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:14:05.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.575+0000 7f6c79800640 1 --2- 172.21.15.28:0/4050665794 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e8de00f970 0x55e8de011d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:05.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.575+0000 7f6c79800640 1 -- 172.21.15.28:0/4050665794 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e8de0052a0 msgr2=0x55e8de00f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:14:05.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.575+0000 7f6c79800640 1 --2- 172.21.15.28:0/4050665794 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e8de0052a0 0x55e8de00f430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:05.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.575+0000 7f6c79800640 1 -- 172.21.15.28:0/4050665794 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e8ddc7d3f0 con 0x55e8de004980 2024-05-12T22:14:05.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.575+0000 7f6c79800640 1 --2- 172.21.15.28:0/4050665794 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e8de004980 0x55e8de004d60 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f6c6c0093e0 tx=0x7f6c6c009a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=77f93d58b8720a09 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:14:05.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.576+0000 7f6c6a000640 1 -- 172.21.15.28:0/4050665794 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c6c030620 con 0x55e8de004980 2024-05-12T22:14:05.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.576+0000 7f6c72000640 1 --2- 172.21.15.28:0/2124192625 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f6c580a2d00 0x7f6c580a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:05.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.576+0000 7f6c6a000640 1 -- 172.21.15.28:0/4050665794 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6c6c036040 con 0x55e8de004980 2024-05-12T22:14:05.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.576+0000 7f6c6a000640 1 -- 172.21.15.28:0/4050665794 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c6c0044a0 con 0x55e8de004980 2024-05-12T22:14:05.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.576+0000 7f6c78e00640 1 --2- 172.21.15.28:0/2124192625 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f6c580a2d00 0x7f6c580a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:05.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.576+0000 7f6c72000640 1 -- 172.21.15.28:0/2124192625 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f6c640cf870 con 0x7f6c74103a10 2024-05-12T22:14:05.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.576+0000 7f6c7e7c7740 1 -- 172.21.15.28:0/4050665794 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e8de004980 msgr2=0x55e8de004d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:14:05.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.576+0000 7f6c7e7c7740 1 --2- 172.21.15.28:0/4050665794 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e8de004980 0x55e8de004d60 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f6c6c0093e0 tx=0x7f6c6c009a20 comp rx=0 tx=0).stop 2024-05-12T22:14:05.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.577+0000 7f6c7e7c7740 1 -- 172.21.15.28:0/4050665794 shutdown_connections 2024-05-12T22:14:05.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.577+0000 7f6c7e7c7740 1 --2- 172.21.15.28:0/4050665794 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e8de00f970 0x55e8de011d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:05.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.577+0000 7f6c7e7c7740 1 --2- 172.21.15.28:0/4050665794 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e8de0052a0 0x55e8de00f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:05.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.577+0000 7f6c7e7c7740 1 --2- 172.21.15.28:0/4050665794 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e8de004980 0x55e8de004d60 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:05.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.577+0000 7f6c7e7c7740 1 -- 172.21.15.28:0/4050665794 >> 172.21.15.28:0/4050665794 conn(0x55e8de000ca0 msgr2=0x55e8de001080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:14:05.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.577+0000 7f6c78e00640 1 --2- 172.21.15.28:0/2124192625 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f6c580a2d00 0x7f6c580a51c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x55e8de005f40 tx=0x7f6c640d5530 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:14:05.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.577+0000 7f6c7e7c7740 1 -- 172.21.15.28:0/4050665794 shutdown_connections 2024-05-12T22:14:05.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.577+0000 7f6c7e7c7740 1 -- 172.21.15.28:0/4050665794 wait complete. 2024-05-12T22:14:05.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.577+0000 7f6c7e7c7740 1 -- start start 2024-05-12T22:14:05.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.577+0000 7f6c7e7c7740 1 Processor -- start 2024-05-12T22:14:05.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.578+0000 7f6c7e7c7740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e8de004980 0x55e8de00c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:05.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.578+0000 7f6c7e7c7740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e8de0052a0 0x55e8de00ce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:05.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.578+0000 7f6c7e7c7740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e8de00f970 0x55e8de00d350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:05.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.578+0000 7f6c7e7c7740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e8de012700 con 0x55e8de004980 2024-05-12T22:14:05.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.578+0000 7f6c7e7c7740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x55e8ddf24a70 con 0x55e8de00f970 2024-05-12T22:14:05.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.578+0000 7f6c7e7c7740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55e8de012880 con 0x55e8de0052a0 2024-05-12T22:14:05.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.578+0000 7f6c7e7c7740 10 client.0 _ll_register_callbacks cb 0x55e8ddf07ba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:14:05.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.578+0000 7f6c79800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e8de004980 0x55e8de00c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:05.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.578+0000 7f6c6be00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e8de0052a0 0x55e8de00ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:05.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.578+0000 7f6c79800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e8de004980 0x55e8de00c8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38288/0 (socket says 172.21.15.28:38288) 2024-05-12T22:14:05.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.578+0000 7f6c79800640 1 -- 172.21.15.28:0/622952665 learned_addr learned my addr 172.21.15.28:0/622952665 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:14:05.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.578+0000 7f6c6be00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e8de0052a0 0x55e8de00ce10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:55252/0 (socket says 172.21.15.28:55252) 2024-05-12T22:14:05.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.578+0000 7f6c79800640 1 --2- 172.21.15.28:0/622952665 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e8de00f970 0x55e8de00d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:05.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.578+0000 7f6c6be00640 1 -- 172.21.15.28:0/622952665 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e8de00f970 msgr2=0x55e8de00d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:14:05.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.579+0000 7f6c6be00640 1 --2- 172.21.15.28:0/622952665 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e8de00f970 0x55e8de00d350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:05.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.579+0000 7f6c6be00640 1 -- 172.21.15.28:0/622952665 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e8de004980 msgr2=0x55e8de00c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:14:05.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.579+0000 7f6c6be00640 1 --2- 172.21.15.28:0/622952665 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e8de004980 0x55e8de00c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:05.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.579+0000 7f6c6be00640 1 -- 172.21.15.28:0/622952665 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e8ddd477d0 con 0x55e8de0052a0 2024-05-12T22:14:05.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.579+0000 7f6c79800640 1 --2- 172.21.15.28:0/622952665 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e8de00f970 0x55e8de00d350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:14:05.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.579+0000 7f6c6be00640 1 --2- 172.21.15.28:0/622952665 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e8de0052a0 0x55e8de00ce10 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f6c74101ee0 tx=0x7f6c5c012b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d57dcb8715865580 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:14:05.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.579+0000 7f6c62a00640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-12T22:14:05.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.580+0000 7f6c62a00640 1 -- 172.21.15.28:0/622952665 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c5c00f920 con 0x55e8de0052a0 2024-05-12T22:14:05.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.580+0000 7f6c62a00640 1 -- 172.21.15.28:0/622952665 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f6c5c00fac0 con 0x55e8de0052a0 2024-05-12T22:14:05.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.580+0000 7f6c7e7c7740 1 -- 172.21.15.28:0/622952665 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6c6c036040 con 0x55e8de0052a0 2024-05-12T22:14:05.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.580+0000 7f6c62a00640 1 -- 172.21.15.28:0/622952665 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f6c5c00fcc0 con 0x55e8de0052a0 2024-05-12T22:14:05.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.580+0000 7f6c7e7c7740 10 client.26344 fetch_fsmap learned FSMap version 26 2024-05-12T22:14:05.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.580+0000 7f6c7e7c7740 1 -- 172.21.15.28:0/622952665 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55e8de01e9f0 con 0x55e8de0052a0 2024-05-12T22:14:05.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.580+0000 7f6c62a00640 1 -- 172.21.15.28:0/622952665 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6c6c036040 con 0x55e8de0052a0 2024-05-12T22:14:05.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.580+0000 7f6c62a00640 1 -- 172.21.15.28:0/622952665 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f6c5c01c420 con 0x55e8de0052a0 2024-05-12T22:14:05.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.580+0000 7f6c7e7c7740 10 client.26344 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:14:05.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.580+0000 7f6c7e7c7740 10 client.26344 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.g 2024-05-12T22:14:05.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.581+0000 7f6c7e7c7740 20 client.26344 populate_metadata read hostname 'smithi028' 2024-05-12T22:14:05.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.581+0000 7f6c7e7c7740 1 --2- 172.21.15.28:0/622952665 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x55e8de01feb0 0x55e8de022310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:05.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.581+0000 7f6c7e7c7740 4 client.26344 mds_command: new command op to 24532 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-12T22:14:05.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.581+0000 7f6c7e7c7740 1 -- 172.21.15.28:0/622952665 --> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e8ddf77160 con 0x55e8de01feb0 2024-05-12T22:14:05.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.581+0000 7f6c79800640 1 --2- 172.21.15.28:0/622952665 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x55e8de01feb0 0x55e8de022310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:05.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.581+0000 7f6c79800640 1 --2- 172.21.15.28:0/622952665 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x55e8de01feb0 0x55e8de022310 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=7dbebac4f2b6952c server_cookie=a05a500f11158302 in_seq=0 out_seq=0 2024-05-12T22:14:05.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.581+0000 7f6c62a00640 10 client.26344 ms_handle_connect on v2:172.21.15.28:6838/123892158 2024-05-12T22:14:05.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.582+0000 7f6c62a00640 1 -- 172.21.15.28:0/622952665 <== mds.2 v2:172.21.15.28:6838/123892158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e8ddf77160 con 0x55e8de01feb0 2024-05-12T22:14:05.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.582+0000 7f6c62a00640 10 client.26344 handle_command_reply: tid=0 2024-05-12T22:14:05.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.582+0000 7f6c7e7c7740 1 client.26344 shutdown 2024-05-12T22:14:05.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.582+0000 7f6c7e7c7740 1 -- 172.21.15.28:0/622952665 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e8de0052a0 msgr2=0x55e8de00ce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:14:05.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.582+0000 7f6c7e7c7740 1 --2- 172.21.15.28:0/622952665 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e8de0052a0 0x55e8de00ce10 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f6c74101ee0 tx=0x7f6c5c012b60 comp rx=0 tx=0).stop 2024-05-12T22:14:05.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.582+0000 7f6c7e7c7740 1 -- 172.21.15.28:0/622952665 shutdown_connections 2024-05-12T22:14:05.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.582+0000 7f6c7e7c7740 1 --2- 172.21.15.28:0/622952665 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x55e8de01feb0 0x55e8de022310 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:05.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.582+0000 7f6c7e7c7740 1 --2- 172.21.15.28:0/622952665 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e8de00f970 0x55e8de00d350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:05.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.582+0000 7f6c7e7c7740 1 --2- 172.21.15.28:0/622952665 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e8de0052a0 0x55e8de00ce10 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:05.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.582+0000 7f6c7e7c7740 1 --2- 172.21.15.28:0/622952665 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e8de004980 0x55e8de00c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:05.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.582+0000 7f6c62a00640 0 client.26344 ms_handle_reset on v2:172.21.15.28:6838/123892158 2024-05-12T22:14:05.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.582+0000 7f6c7e7c7740 1 -- 172.21.15.28:0/622952665 >> 172.21.15.28:0/622952665 conn(0x55e8de000ca0 msgr2=0x55e8de006980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:14:05.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.583+0000 7f6c7e7c7740 1 -- 172.21.15.28:0/622952665 shutdown_connections 2024-05-12T22:14:05.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.583+0000 7f6c7e7c7740 1 -- 172.21.15.28:0/622952665 wait complete. 2024-05-12T22:14:05.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.583+0000 7f6c7e7c7740 20 client.26344 trim_cache size 0 max 16384 2024-05-12T22:14:05.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.597+0000 7f6c7e7c7740 1 Processor -- start 2024-05-12T22:14:05.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.598+0000 7f6c7e7c7740 1 -- start start 2024-05-12T22:14:05.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.598+0000 7f6c7e7c7740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e8de004980 0x55e8de01b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:05.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.598+0000 7f6c7e7c7740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e8de0052a0 0x55e8de027be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:05.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.598+0000 7f6c7e7c7740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e8de00f970 0x55e8de028120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:05.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.598+0000 7f6c7e7c7740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6c741b2070 con 0x55e8de00f970 2024-05-12T22:14:05.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.598+0000 7f6c7e7c7740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f6c741af940 con 0x55e8de004980 2024-05-12T22:14:05.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.598+0000 7f6c7e7c7740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f6c741ad790 con 0x55e8de0052a0 2024-05-12T22:14:05.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.598+0000 7f6c79800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e8de00f970 0x55e8de028120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:05.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.598+0000 7f6c6be00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e8de0052a0 0x55e8de027be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:05.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.598+0000 7f6c79800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e8de00f970 0x55e8de028120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38302/0 (socket says 172.21.15.28:38302) 2024-05-12T22:14:05.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.598+0000 7f6c79800640 1 -- 172.21.15.28:0/3946905420 learned_addr learned my addr 172.21.15.28:0/3946905420 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:14:05.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.598+0000 7f6c79800640 1 --2- 172.21.15.28:0/3946905420 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e8de004980 0x55e8de01b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:05.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.599+0000 7f6c79800640 1 -- 172.21.15.28:0/3946905420 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e8de0052a0 msgr2=0x55e8de027be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:14:05.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.599+0000 7f6c79800640 1 --2- 172.21.15.28:0/3946905420 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e8de0052a0 0x55e8de027be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:05.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.599+0000 7f6c79800640 1 -- 172.21.15.28:0/3946905420 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e8de004980 msgr2=0x55e8de01b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:14:05.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.599+0000 7f6c79800640 1 --2- 172.21.15.28:0/3946905420 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e8de004980 0x55e8de01b5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:05.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.599+0000 7f6c79800640 1 -- 172.21.15.28:0/3946905420 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e8de0057a0 con 0x55e8de00f970 2024-05-12T22:14:05.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.599+0000 7f6c79800640 1 --2- 172.21.15.28:0/3946905420 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e8de00f970 0x55e8de028120 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f6c6c003f20 tx=0x7f6c6c02a8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b670d8f87bcb484 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:14:05.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.599+0000 7f6c69600640 1 -- 172.21.15.28:0/3946905420 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c6c030420 con 0x55e8de00f970 2024-05-12T22:14:05.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.599+0000 7f6c69600640 1 -- 172.21.15.28:0/3946905420 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6c6c036040 con 0x55e8de00f970 2024-05-12T22:14:05.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.599+0000 7f6c69600640 1 -- 172.21.15.28:0/3946905420 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c6c058690 con 0x55e8de00f970 2024-05-12T22:14:05.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.600+0000 7f6c7e7c7740 1 -- 172.21.15.28:0/3946905420 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e8de00f970 msgr2=0x55e8de028120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:14:05.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.600+0000 7f6c7e7c7740 1 --2- 172.21.15.28:0/3946905420 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e8de00f970 0x55e8de028120 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f6c6c003f20 tx=0x7f6c6c02a8f0 comp rx=0 tx=0).stop 2024-05-12T22:14:05.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.600+0000 7f6c7e7c7740 1 -- 172.21.15.28:0/3946905420 shutdown_connections 2024-05-12T22:14:05.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.600+0000 7f6c7e7c7740 1 --2- 172.21.15.28:0/3946905420 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e8de00f970 0x55e8de028120 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:05.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.600+0000 7f6c7e7c7740 1 --2- 172.21.15.28:0/3946905420 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e8de0052a0 0x55e8de027be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:05.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.600+0000 7f6c7e7c7740 1 --2- 172.21.15.28:0/3946905420 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e8de004980 0x55e8de01b5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:05.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.600+0000 7f6c7e7c7740 1 -- 172.21.15.28:0/3946905420 >> 172.21.15.28:0/3946905420 conn(0x55e8de000ca0 msgr2=0x55e8de001080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:14:05.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.600+0000 7f6c7e7c7740 1 -- 172.21.15.28:0/3946905420 shutdown_connections 2024-05-12T22:14:05.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.600+0000 7f6c7e7c7740 1 -- 172.21.15.28:0/3946905420 wait complete. 2024-05-12T22:14:05.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.601+0000 7f6c7e7c7740 1 -- start start 2024-05-12T22:14:05.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.601+0000 7f6c7e7c7740 1 Processor -- start 2024-05-12T22:14:05.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.601+0000 7f6c7e7c7740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e8de004980 0x55e8de018020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:05.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.601+0000 7f6c7e7c7740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e8de0052a0 0x55e8de013390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:05.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.601+0000 7f6c7e7c7740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e8de00f970 0x55e8de013ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:05.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.601+0000 7f6c7e7c7740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6c741ade00 con 0x55e8de004980 2024-05-12T22:14:05.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.601+0000 7f6c7e7c7740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f6c741af270 con 0x55e8de00f970 2024-05-12T22:14:05.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.601+0000 7f6c7e7c7740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f6c741ae4d0 con 0x55e8de0052a0 2024-05-12T22:14:05.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.601+0000 7f6c7e7c7740 10 client.0 _ll_register_callbacks cb 0x7f6c5c009ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:14:05.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.602+0000 7f6c79800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e8de004980 0x55e8de018020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:05.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.602+0000 7f6c6be00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e8de0052a0 0x55e8de013390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:05.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.602+0000 7f6c79800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e8de004980 0x55e8de018020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38310/0 (socket says 172.21.15.28:38310) 2024-05-12T22:14:05.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.602+0000 7f6c79800640 1 -- 172.21.15.28:0/2664521261 learned_addr learned my addr 172.21.15.28:0/2664521261 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:14:05.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.602+0000 7f6c79800640 1 --2- 172.21.15.28:0/2664521261 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e8de00f970 0x55e8de013ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:05.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.602+0000 7f6c79800640 1 -- 172.21.15.28:0/2664521261 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e8de0052a0 msgr2=0x55e8de013390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:14:05.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.602+0000 7f6c79800640 1 --2- 172.21.15.28:0/2664521261 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e8de0052a0 0x55e8de013390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:05.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.602+0000 7f6c79800640 1 -- 172.21.15.28:0/2664521261 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e8de00f970 msgr2=0x55e8de013ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:14:05.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.602+0000 7f6c79800640 1 --2- 172.21.15.28:0/2664521261 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e8de00f970 0x55e8de013ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:05.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.602+0000 7f6c79800640 1 -- 172.21.15.28:0/2664521261 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e8de085450 con 0x55e8de004980 2024-05-12T22:14:05.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.602+0000 7f6c6be00640 1 --2- 172.21.15.28:0/2664521261 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e8de0052a0 0x55e8de013390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:14:05.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.602+0000 7f6c79800640 1 --2- 172.21.15.28:0/2664521261 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e8de004980 0x55e8de018020 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f6c6c031040 tx=0x7f6c6c058270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae09cae4d1aa94e8 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:14:05.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.602+0000 7f6c62a00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-12T22:14:05.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.603+0000 7f6c62a00640 1 -- 172.21.15.28:0/2664521261 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c6c05d070 con 0x55e8de004980 2024-05-12T22:14:05.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.603+0000 7f6c62a00640 1 -- 172.21.15.28:0/2664521261 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f6c6c030560 con 0x55e8de004980 2024-05-12T22:14:05.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.603+0000 7f6c7e7c7740 1 -- 172.21.15.28:0/2664521261 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6c6c036040 con 0x55e8de004980 2024-05-12T22:14:05.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.603+0000 7f6c62a00640 1 -- 172.21.15.28:0/2664521261 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f6c6c030930 con 0x55e8de004980 2024-05-12T22:14:05.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.603+0000 7f6c7e7c7740 10 client.16266 fetch_fsmap learned FSMap version 26 2024-05-12T22:14:05.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.603+0000 7f6c7e7c7740 1 -- 172.21.15.28:0/2664521261 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55e8de0865b0 con 0x55e8de004980 2024-05-12T22:14:05.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.603+0000 7f6c62a00640 1 -- 172.21.15.28:0/2664521261 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6c6c036040 con 0x55e8de004980 2024-05-12T22:14:05.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.603+0000 7f6c62a00640 1 -- 172.21.15.28:0/2664521261 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f6c6c060020 con 0x55e8de004980 2024-05-12T22:14:05.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.603+0000 7f6c7e7c7740 10 client.16266 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:14:05.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.604+0000 7f6c7e7c7740 10 client.16266 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.g 2024-05-12T22:14:05.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.604+0000 7f6c7e7c7740 20 client.16266 populate_metadata read hostname 'smithi028' 2024-05-12T22:14:05.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.604+0000 7f6c7e7c7740 1 --2- 172.21.15.28:0/2664521261 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x55e8de086bc0 0x55e8de088fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:05.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.604+0000 7f6c7e7c7740 4 client.16266 mds_command: new command op to 24532 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-12T22:14:05.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.604+0000 7f6c7e7c7740 1 -- 172.21.15.28:0/2664521261 --> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e8ddf714b0 con 0x55e8de086bc0 2024-05-12T22:14:05.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.604+0000 7f6c6be00640 1 --2- 172.21.15.28:0/2664521261 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x55e8de086bc0 0x55e8de088fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:05.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.604+0000 7f6c6be00640 1 --2- 172.21.15.28:0/2664521261 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x55e8de086bc0 0x55e8de088fe0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f9832475bb94be2f server_cookie=b11979ccbab39d76 in_seq=0 out_seq=0 2024-05-12T22:14:05.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.604+0000 7f6c62a00640 10 client.16266 ms_handle_connect on v2:172.21.15.28:6838/123892158 2024-05-12T22:14:05.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.605+0000 7f6c62a00640 1 -- 172.21.15.28:0/2664521261 <== mds.2 v2:172.21.15.28:6838/123892158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e8ddf714b0 con 0x55e8de086bc0 2024-05-12T22:14:05.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.605+0000 7f6c62a00640 10 client.16266 handle_command_reply: tid=0 2024-05-12T22:14:05.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.605+0000 7f6c7e7c7740 1 client.16266 shutdown 2024-05-12T22:14:05.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.605+0000 7f6c7e7c7740 1 -- 172.21.15.28:0/2664521261 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e8de004980 msgr2=0x55e8de018020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:14:05.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.605+0000 7f6c7e7c7740 1 --2- 172.21.15.28:0/2664521261 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e8de004980 0x55e8de018020 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f6c6c031040 tx=0x7f6c6c058270 comp rx=0 tx=0).stop 2024-05-12T22:14:05.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.605+0000 7f6c7e7c7740 1 -- 172.21.15.28:0/2664521261 shutdown_connections 2024-05-12T22:14:05.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.605+0000 7f6c7e7c7740 1 --2- 172.21.15.28:0/2664521261 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x55e8de086bc0 0x55e8de088fe0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:05.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.605+0000 7f6c7e7c7740 1 --2- 172.21.15.28:0/2664521261 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e8de00f970 0x55e8de013ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:05.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.605+0000 7f6c7e7c7740 1 --2- 172.21.15.28:0/2664521261 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e8de0052a0 0x55e8de013390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:05.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.605+0000 7f6c7e7c7740 1 --2- 172.21.15.28:0/2664521261 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e8de004980 0x55e8de018020 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:05.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.605+0000 7f6c62a00640 0 client.16266 ms_handle_reset on v2:172.21.15.28:6838/123892158 2024-05-12T22:14:05.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.605+0000 7f6c7e7c7740 1 -- 172.21.15.28:0/2664521261 >> 172.21.15.28:0/2664521261 conn(0x55e8de000ca0 msgr2=0x55e8ddfff530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:14:05.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.605+0000 7f6c7e7c7740 1 -- 172.21.15.28:0/2664521261 shutdown_connections 2024-05-12T22:14:05.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.605+0000 7f6c7e7c7740 1 -- 172.21.15.28:0/2664521261 wait complete. 2024-05-12T22:14:05.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.605+0000 7f6c7e7c7740 20 client.16266 trim_cache size 0 max 16384 2024-05-12T22:14:05.613 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-05-12T22:14:05.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.606+0000 7f6c7b800640 1 -- 172.21.15.28:0/2124192625 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f6c580a2d00 msgr2=0x7f6c580a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:14:05.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.606+0000 7f6c7b800640 1 --2- 172.21.15.28:0/2124192625 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f6c580a2d00 0x7f6c580a51c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x55e8de005f40 tx=0x7f6c640d5530 comp rx=0 tx=0).stop 2024-05-12T22:14:05.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.606+0000 7f6c7b800640 1 -- 172.21.15.28:0/2124192625 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6c74103a10 msgr2=0x7f6c7419a300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:14:05.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.606+0000 7f6c7b800640 1 --2- 172.21.15.28:0/2124192625 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6c74103a10 0x7f6c7419a300 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f6c6400e9f0 tx=0x7f6c6400eeb0 comp rx=0 tx=0).stop 2024-05-12T22:14:05.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.606+0000 7f6c7b800640 1 -- 172.21.15.28:0/2124192625 shutdown_connections 2024-05-12T22:14:05.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.606+0000 7f6c7b800640 1 --2- 172.21.15.28:0/2124192625 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f6c580a2d00 0x7f6c580a51c0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:05.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.606+0000 7f6c7b800640 1 --2- 172.21.15.28:0/2124192625 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6c74109a20 0x7f6c7419ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:05.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.606+0000 7f6c7b800640 1 --2- 172.21.15.28:0/2124192625 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6c741043d0 0x7f6c7419a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:05.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.606+0000 7f6c7b800640 1 --2- 172.21.15.28:0/2124192625 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6c74103a10 0x7f6c7419a300 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:05.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.606+0000 7f6c7b800640 1 -- 172.21.15.28:0/2124192625 >> 172.21.15.28:0/2124192625 conn(0x7f6c740ff4a0 msgr2=0x7f6c740ffaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:14:05.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.606+0000 7f6c7b800640 1 -- 172.21.15.28:0/2124192625 shutdown_connections 2024-05-12T22:14:05.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.606+0000 7f6c7b800640 1 -- 172.21.15.28:0/2124192625 wait complete. 2024-05-12T22:14:05.624 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-05-12T22:14:05.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.836+0000 7f3ef5000640 1 -- 172.21.15.28:0/381649237 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3ef01038e0 msgr2=0x7f3ef0103d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:14:05.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.836+0000 7f3ef5000640 1 --2- 172.21.15.28:0/381649237 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3ef01038e0 0x7f3ef0103d60 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f3ed8009ca0 tx=0x7f3ed802a8b0 comp rx=0 tx=0).stop 2024-05-12T22:14:05.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.836+0000 7f3ef5000640 1 -- 172.21.15.28:0/381649237 shutdown_connections 2024-05-12T22:14:05.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.836+0000 7f3ef5000640 1 --2- 172.21.15.28:0/381649237 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3ef01042a0 0x7f3ef01107b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:05.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.836+0000 7f3ef5000640 1 --2- 172.21.15.28:0/381649237 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3ef01038e0 0x7f3ef0103d60 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:05.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.836+0000 7f3ef5000640 1 --2- 172.21.15.28:0/381649237 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ef01098f0 0x7f3ef0109cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:05.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.836+0000 7f3ef5000640 1 -- 172.21.15.28:0/381649237 >> 172.21.15.28:0/381649237 conn(0x7f3ef00ff370 msgr2=0x7f3ef0101790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:14:05.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.836+0000 7f3ef5000640 1 -- 172.21.15.28:0/381649237 shutdown_connections 2024-05-12T22:14:05.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.836+0000 7f3ef5000640 1 -- 172.21.15.28:0/381649237 wait complete. 2024-05-12T22:14:05.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.837+0000 7f3ef5000640 1 Processor -- start 2024-05-12T22:14:05.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.837+0000 7f3ef5000640 1 -- start start 2024-05-12T22:14:05.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.837+0000 7f3ef5000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3ef01038e0 0x7f3ef019a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:05.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.837+0000 7f3ef5000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ef01042a0 0x7f3ef019a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:05.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.838+0000 7f3ef5000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3ef01098f0 0x7f3ef019eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:05.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.838+0000 7f3ef5000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3ef0113d50 con 0x7f3ef01042a0 2024-05-12T22:14:05.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.838+0000 7f3ef5000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f3ef0113bd0 con 0x7f3ef01038e0 2024-05-12T22:14:05.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.838+0000 7f3ef5000640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f3ef0113ed0 con 0x7f3ef01098f0 2024-05-12T22:14:05.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.838+0000 7f3eeea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ef01042a0 0x7f3ef019a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:05.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.838+0000 7f3eef400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3ef01038e0 0x7f3ef019a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:05.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.838+0000 7f3eeea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ef01042a0 0x7f3ef019a790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57110/0 (socket says 172.21.15.28:57110) 2024-05-12T22:14:05.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.838+0000 7f3eeea00640 1 -- 172.21.15.28:0/285229588 learned_addr learned my addr 172.21.15.28:0/285229588 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:14:05.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.838+0000 7f3eefe00640 1 --2- 172.21.15.28:0/285229588 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3ef01098f0 0x7f3ef019eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:05.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.838+0000 7f3eeea00640 1 -- 172.21.15.28:0/285229588 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3ef01098f0 msgr2=0x7f3ef019eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:14:05.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.838+0000 7f3eeea00640 1 --2- 172.21.15.28:0/285229588 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3ef01098f0 0x7f3ef019eb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:05.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.838+0000 7f3eeea00640 1 -- 172.21.15.28:0/285229588 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3ef01038e0 msgr2=0x7f3ef019a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:14:05.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.838+0000 7f3eeea00640 1 --2- 172.21.15.28:0/285229588 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3ef01038e0 0x7f3ef019a250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:05.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.838+0000 7f3eeea00640 1 -- 172.21.15.28:0/285229588 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ef019f240 con 0x7f3ef01042a0 2024-05-12T22:14:05.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.838+0000 7f3eefe00640 1 --2- 172.21.15.28:0/285229588 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3ef01098f0 0x7f3ef019eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:14:05.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.839+0000 7f3eeea00640 1 --2- 172.21.15.28:0/285229588 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ef01042a0 0x7f3ef019a790 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f3ed8005e30 tx=0x7f3ed8003f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:14:05.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.839+0000 7f3ee7e00640 1 -- 172.21.15.28:0/285229588 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ed8030e30 con 0x7f3ef01042a0 2024-05-12T22:14:05.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.839+0000 7f3ef5000640 1 -- 172.21.15.28:0/285229588 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ef019f4d0 con 0x7f3ef01042a0 2024-05-12T22:14:05.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.839+0000 7f3ee7e00640 1 -- 172.21.15.28:0/285229588 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3ed8040070 con 0x7f3ef01042a0 2024-05-12T22:14:05.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.839+0000 7f3ef5000640 1 -- 172.21.15.28:0/285229588 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ef01a6dc0 con 0x7f3ef01042a0 2024-05-12T22:14:05.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.839+0000 7f3ee7e00640 1 -- 172.21.15.28:0/285229588 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ed8045ad0 con 0x7f3ef01042a0 2024-05-12T22:14:05.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.843+0000 7f3ef93f7740 1 Processor -- start 2024-05-12T22:14:05.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.844+0000 7f3ef93f7740 1 -- start start 2024-05-12T22:14:05.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.844+0000 7f3ee7e00640 1 -- 172.21.15.28:0/285229588 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7f3ed8045c70 con 0x7f3ef01042a0 2024-05-12T22:14:05.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.844+0000 7f3ef93f7740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db01f1bb60 0x55db01f1bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:05.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.844+0000 7f3ef93f7740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55db01f1c480 0x55db01f26610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:05.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.844+0000 7f3ef93f7740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55db01f26b50 0x55db01f28f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:05.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.844+0000 7f3ef93f7740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55db01e80a30 con 0x55db01f1bb60 2024-05-12T22:14:05.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.844+0000 7f3ef93f7740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x55db01e28700 con 0x55db01f1c480 2024-05-12T22:14:05.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.844+0000 7f3ef93f7740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55db01e7c9f0 con 0x55db01f26b50 2024-05-12T22:14:05.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.844+0000 7f3eef400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55db01f1c480 0x55db01f26610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:05.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.844+0000 7f3eef400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55db01f1c480 0x55db01f26610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:55896/0 (socket says 172.21.15.28:55896) 2024-05-12T22:14:05.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.845+0000 7f3eef400640 1 -- 172.21.15.28:0/2129801539 learned_addr learned my addr 172.21.15.28:0/2129801539 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:14:05.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.845+0000 7f3eeea00640 1 --2- 172.21.15.28:0/2129801539 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55db01f26b50 0x55db01f28f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:05.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.845+0000 7f3eefe00640 1 --2- 172.21.15.28:0/2129801539 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db01f1bb60 0x55db01f1bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:05.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.845+0000 7f3eef400640 1 -- 172.21.15.28:0/2129801539 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55db01f26b50 msgr2=0x55db01f28f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:14:05.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.845+0000 7f3eef400640 1 --2- 172.21.15.28:0/2129801539 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55db01f26b50 0x55db01f28f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:05.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.845+0000 7f3eef400640 1 -- 172.21.15.28:0/2129801539 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db01f1bb60 msgr2=0x55db01f1bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:14:05.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.845+0000 7f3eef400640 1 --2- 172.21.15.28:0/2129801539 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db01f1bb60 0x55db01f1bf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:05.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.845+0000 7f3eef400640 1 -- 172.21.15.28:0/2129801539 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55db01b943f0 con 0x55db01f1c480 2024-05-12T22:14:05.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.845+0000 7f3eef400640 1 --2- 172.21.15.28:0/2129801539 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55db01f1c480 0x55db01f26610 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f3ee0009c70 tx=0x7f3ee0011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b93c09e2c46ccfd0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:14:05.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.845+0000 7f3ee4c00640 1 -- 172.21.15.28:0/2129801539 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ee0004230 con 0x55db01f1c480 2024-05-12T22:14:05.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.845+0000 7f3ee4c00640 1 -- 172.21.15.28:0/2129801539 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3ee00043d0 con 0x55db01f1c480 2024-05-12T22:14:05.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.845+0000 7f3ee4c00640 1 -- 172.21.15.28:0/2129801539 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ee00197c0 con 0x55db01f1c480 2024-05-12T22:14:05.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.846+0000 7f3ee7e00640 1 --2- 172.21.15.28:0/285229588 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f3ed40a2d00 0x7f3ed40a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:05.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.846+0000 7f3ef93f7740 1 -- 172.21.15.28:0/2129801539 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55db01f1c480 msgr2=0x55db01f26610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:14:05.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.846+0000 7f3ef93f7740 1 --2- 172.21.15.28:0/2129801539 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55db01f1c480 0x55db01f26610 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f3ee0009c70 tx=0x7f3ee0011810 comp rx=0 tx=0).stop 2024-05-12T22:14:05.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.846+0000 7f3ee7e00640 1 -- 172.21.15.28:0/285229588 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f3ed80f0760 con 0x7f3ef01042a0 2024-05-12T22:14:05.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.846+0000 7f3eefe00640 1 --2- 172.21.15.28:0/285229588 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f3ed40a2d00 0x7f3ed40a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:05.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.846+0000 7f3ef93f7740 1 -- 172.21.15.28:0/2129801539 shutdown_connections 2024-05-12T22:14:05.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.846+0000 7f3ef93f7740 1 --2- 172.21.15.28:0/2129801539 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55db01f26b50 0x55db01f28f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:05.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.846+0000 7f3ef93f7740 1 --2- 172.21.15.28:0/2129801539 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55db01f1c480 0x55db01f26610 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:05.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.846+0000 7f3ef93f7740 1 --2- 172.21.15.28:0/2129801539 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db01f1bb60 0x55db01f1bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:05.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.846+0000 7f3ef93f7740 1 -- 172.21.15.28:0/2129801539 >> 172.21.15.28:0/2129801539 conn(0x55db01f17e80 msgr2=0x55db01f18260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:14:05.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.846+0000 7f3ef93f7740 1 -- 172.21.15.28:0/2129801539 shutdown_connections 2024-05-12T22:14:05.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.846+0000 7f3ef93f7740 1 -- 172.21.15.28:0/2129801539 wait complete. 2024-05-12T22:14:05.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.847+0000 7f3ef93f7740 1 -- start start 2024-05-12T22:14:05.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.847+0000 7f3eefe00640 1 --2- 172.21.15.28:0/285229588 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f3ed40a2d00 0x7f3ed40a51c0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f3ee8009bc0 tx=0x7f3ee8005f90 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:14:05.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.847+0000 7f3ef93f7740 1 Processor -- start 2024-05-12T22:14:05.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.848+0000 7f3ef93f7740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db01f1bb60 0x55db01f23ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:05.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.848+0000 7f3ef93f7740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55db01f1c480 0x55db01f23ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:05.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.848+0000 7f3ef93f7740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55db01f26b50 0x55db01f24530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:05.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.848+0000 7f3ef93f7740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55db01f29940 con 0x55db01f1bb60 2024-05-12T22:14:05.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.848+0000 7f3ef93f7740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x55db01e3ba70 con 0x55db01f1c480 2024-05-12T22:14:05.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.848+0000 7f3ef93f7740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55db01f29ac0 con 0x55db01f26b50 2024-05-12T22:14:05.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.848+0000 7f3ef93f7740 10 client.0 _ll_register_callbacks cb 0x55db01e1eba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:14:05.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.848+0000 7f3eef400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db01f1bb60 0x55db01f23ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:05.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.848+0000 7f3eeea00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55db01f1c480 0x55db01f23ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:05.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.848+0000 7f3eef400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db01f1bb60 0x55db01f23ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57134/0 (socket says 172.21.15.28:57134) 2024-05-12T22:14:05.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.848+0000 7f3eef400640 1 -- 172.21.15.28:0/1093279877 learned_addr learned my addr 172.21.15.28:0/1093279877 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:14:05.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.848+0000 7f3eefe00640 1 --2- 172.21.15.28:0/1093279877 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55db01f26b50 0x55db01f24530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:05.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.848+0000 7f3eeea00640 1 -- 172.21.15.28:0/1093279877 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55db01f26b50 msgr2=0x55db01f24530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:14:05.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.848+0000 7f3eeea00640 1 --2- 172.21.15.28:0/1093279877 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55db01f26b50 0x55db01f24530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:05.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.848+0000 7f3eeea00640 1 -- 172.21.15.28:0/1093279877 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db01f1bb60 msgr2=0x55db01f23ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:14:05.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.848+0000 7f3eeea00640 1 --2- 172.21.15.28:0/1093279877 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db01f1bb60 0x55db01f23ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:05.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.848+0000 7f3eeea00640 1 -- 172.21.15.28:0/1093279877 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55db01c5e7d0 con 0x55db01f1c480 2024-05-12T22:14:05.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.848+0000 7f3eef400640 1 --2- 172.21.15.28:0/1093279877 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db01f1bb60 0x55db01f23ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:14:05.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.849+0000 7f3eeea00640 1 --2- 172.21.15.28:0/1093279877 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55db01f1c480 0x55db01f23ff0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f3ed80edf90 tx=0x7f3ed8000f40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4c17fbfe44e3262 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:14:05.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.849+0000 7f3edd600640 10 client.0 ms_handle_connect on v2:172.21.15.86:3300/0 2024-05-12T22:14:05.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.849+0000 7f3edd600640 1 -- 172.21.15.28:0/1093279877 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ed80bc750 con 0x55db01f1c480 2024-05-12T22:14:05.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.849+0000 7f3edd600640 1 -- 172.21.15.28:0/1093279877 <== mon.2 v2:172.21.15.86:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f3ed80bd050 con 0x55db01f1c480 2024-05-12T22:14:05.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.849+0000 7f3ef93f7740 1 -- 172.21.15.28:0/1093279877 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3ee00043d0 con 0x55db01f1c480 2024-05-12T22:14:05.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.849+0000 7f3edd600640 1 -- 172.21.15.28:0/1093279877 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f3ed80ee300 con 0x55db01f1c480 2024-05-12T22:14:05.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.849+0000 7f3ef93f7740 10 client.26405 fetch_fsmap learned FSMap version 26 2024-05-12T22:14:05.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.849+0000 7f3ef93f7740 1 -- 172.21.15.28:0/1093279877 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55db01f35ba0 con 0x55db01f1c480 2024-05-12T22:14:05.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.849+0000 7f3edd600640 1 -- 172.21.15.28:0/1093279877 <== mon.2 v2:172.21.15.86:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3ee00043d0 con 0x55db01f1c480 2024-05-12T22:14:05.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.849+0000 7f3edd600640 1 -- 172.21.15.28:0/1093279877 <== mon.2 v2:172.21.15.86:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f3ed804d030 con 0x55db01f1c480 2024-05-12T22:14:05.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.850+0000 7f3ef93f7740 10 client.26405 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:14:05.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.850+0000 7f3ef93f7740 10 client.26405 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-05-12T22:14:05.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.850+0000 7f3ef93f7740 20 client.26405 populate_metadata read hostname 'smithi028' 2024-05-12T22:14:05.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.850+0000 7f3ef93f7740 1 --2- 172.21.15.28:0/1093279877 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x55db01f37000 0x55db01f39490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:05.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.850+0000 7f3ef93f7740 4 client.26405 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-12T22:14:05.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.850+0000 7f3ef93f7740 1 -- 172.21.15.28:0/1093279877 --> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55db01e8e160 con 0x55db01f37000 2024-05-12T22:14:05.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.850+0000 7f3eef400640 1 --2- 172.21.15.28:0/1093279877 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x55db01f37000 0x55db01f39490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:05.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.851+0000 7f3eef400640 1 --2- 172.21.15.28:0/1093279877 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x55db01f37000 0x55db01f39490 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=bf4f80097b82b0ac server_cookie=c1199100669a408f in_seq=0 out_seq=0 2024-05-12T22:14:05.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.851+0000 7f3edd600640 10 client.26405 ms_handle_connect on v2:172.21.15.28:6840/761183870 2024-05-12T22:14:05.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.851+0000 7f3edd600640 1 -- 172.21.15.28:0/1093279877 <== mds.1 v2:172.21.15.28:6840/761183870 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55db01e8e160 con 0x55db01f37000 2024-05-12T22:14:05.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.851+0000 7f3edd600640 10 client.26405 handle_command_reply: tid=0 2024-05-12T22:14:05.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.851+0000 7f3ef93f7740 1 client.26405 shutdown 2024-05-12T22:14:05.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.852+0000 7f3ef93f7740 1 -- 172.21.15.28:0/1093279877 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55db01f1c480 msgr2=0x55db01f23ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:14:05.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.852+0000 7f3ef93f7740 1 --2- 172.21.15.28:0/1093279877 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55db01f1c480 0x55db01f23ff0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f3ed80edf90 tx=0x7f3ed8000f40 comp rx=0 tx=0).stop 2024-05-12T22:14:05.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.852+0000 7f3ef93f7740 1 -- 172.21.15.28:0/1093279877 shutdown_connections 2024-05-12T22:14:05.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.852+0000 7f3ef93f7740 1 --2- 172.21.15.28:0/1093279877 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x55db01f37000 0x55db01f39490 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:05.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.852+0000 7f3ef93f7740 1 --2- 172.21.15.28:0/1093279877 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55db01f26b50 0x55db01f24530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:05.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.852+0000 7f3ef93f7740 1 --2- 172.21.15.28:0/1093279877 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55db01f1c480 0x55db01f23ff0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:05.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.852+0000 7f3ef93f7740 1 --2- 172.21.15.28:0/1093279877 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db01f1bb60 0x55db01f23ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:05.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.852+0000 7f3edd600640 0 client.26405 ms_handle_reset on v2:172.21.15.28:6840/761183870 2024-05-12T22:14:05.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.852+0000 7f3ef93f7740 1 -- 172.21.15.28:0/1093279877 >> 172.21.15.28:0/1093279877 conn(0x55db01f17e80 msgr2=0x55db01f17000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:14:05.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.852+0000 7f3ef93f7740 1 -- 172.21.15.28:0/1093279877 shutdown_connections 2024-05-12T22:14:05.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.852+0000 7f3ef93f7740 1 -- 172.21.15.28:0/1093279877 wait complete. 2024-05-12T22:14:05.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.852+0000 7f3ef93f7740 20 client.26405 trim_cache size 0 max 16384 2024-05-12T22:14:05.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.867+0000 7f3ef93f7740 1 Processor -- start 2024-05-12T22:14:05.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.867+0000 7f3ef93f7740 1 -- start start 2024-05-12T22:14:05.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.867+0000 7f3ef93f7740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db01f1bb60 0x55db01f32800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:05.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.867+0000 7f3ef93f7740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55db01f1c480 0x55db01f16e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:05.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.867+0000 7f3ef93f7740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55db01f26b50 0x55db01f3eda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:05.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.867+0000 7f3ef93f7740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3ef01b1ce0 con 0x55db01f1bb60 2024-05-12T22:14:05.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.867+0000 7f3ef93f7740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f3ef01af5b0 con 0x55db01f26b50 2024-05-12T22:14:05.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.867+0000 7f3ef93f7740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f3ef01ad400 con 0x55db01f1c480 2024-05-12T22:14:05.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.867+0000 7f3eef400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db01f1bb60 0x55db01f32800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:05.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.867+0000 7f3eef400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db01f1bb60 0x55db01f32800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57142/0 (socket says 172.21.15.28:57142) 2024-05-12T22:14:05.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.867+0000 7f3eef400640 1 -- 172.21.15.28:0/2049399438 learned_addr learned my addr 172.21.15.28:0/2049399438 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:14:05.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.867+0000 7f3eefe00640 1 --2- 172.21.15.28:0/2049399438 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55db01f26b50 0x55db01f3eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:05.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.868+0000 7f3eeea00640 1 --2- 172.21.15.28:0/2049399438 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55db01f1c480 0x55db01f16e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:05.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.868+0000 7f3eef400640 1 -- 172.21.15.28:0/2049399438 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55db01f1c480 msgr2=0x55db01f16e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:14:05.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.868+0000 7f3eef400640 1 --2- 172.21.15.28:0/2049399438 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55db01f1c480 0x55db01f16e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:05.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.868+0000 7f3eef400640 1 -- 172.21.15.28:0/2049399438 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55db01f26b50 msgr2=0x55db01f3eda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:14:05.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.868+0000 7f3eef400640 1 --2- 172.21.15.28:0/2049399438 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55db01f26b50 0x55db01f3eda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:05.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.868+0000 7f3eef400640 1 -- 172.21.15.28:0/2049399438 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55db01f1c2b0 con 0x55db01f1bb60 2024-05-12T22:14:05.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.868+0000 7f3eef400640 1 --2- 172.21.15.28:0/2049399438 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db01f1bb60 0x55db01f32800 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f3ee0011d10 tx=0x7f3ee00193e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b354c413b6d9aeec server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:14:05.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.868+0000 7f3edfe00640 1 -- 172.21.15.28:0/2049399438 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ee0033070 con 0x55db01f1bb60 2024-05-12T22:14:05.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.868+0000 7f3edfe00640 1 -- 172.21.15.28:0/2049399438 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3ee002b070 con 0x55db01f1bb60 2024-05-12T22:14:05.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.868+0000 7f3edfe00640 1 -- 172.21.15.28:0/2049399438 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ee00194f0 con 0x55db01f1bb60 2024-05-12T22:14:05.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.869+0000 7f3ef93f7740 1 -- 172.21.15.28:0/2049399438 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db01f1bb60 msgr2=0x55db01f32800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:14:05.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.869+0000 7f3ef93f7740 1 --2- 172.21.15.28:0/2049399438 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db01f1bb60 0x55db01f32800 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f3ee0011d10 tx=0x7f3ee00193e0 comp rx=0 tx=0).stop 2024-05-12T22:14:05.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.869+0000 7f3ef93f7740 1 -- 172.21.15.28:0/2049399438 shutdown_connections 2024-05-12T22:14:05.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.869+0000 7f3ef93f7740 1 --2- 172.21.15.28:0/2049399438 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55db01f26b50 0x55db01f3eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:05.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.869+0000 7f3ef93f7740 1 --2- 172.21.15.28:0/2049399438 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55db01f1c480 0x55db01f16e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:05.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.869+0000 7f3ef93f7740 1 --2- 172.21.15.28:0/2049399438 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db01f1bb60 0x55db01f32800 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:05.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.869+0000 7f3ef93f7740 1 -- 172.21.15.28:0/2049399438 >> 172.21.15.28:0/2049399438 conn(0x55db01f17e80 msgr2=0x55db01f16770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:14:05.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.869+0000 7f3ef93f7740 1 -- 172.21.15.28:0/2049399438 shutdown_connections 2024-05-12T22:14:05.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.869+0000 7f3ef93f7740 1 -- 172.21.15.28:0/2049399438 wait complete. 2024-05-12T22:14:05.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.870+0000 7f3ef93f7740 1 -- start start 2024-05-12T22:14:05.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.870+0000 7f3ef93f7740 1 Processor -- start 2024-05-12T22:14:05.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.870+0000 7f3ef93f7740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55db01f1bb60 0x55db01f2aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:05.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.870+0000 7f3ef93f7740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55db01f1c480 0x55db01f2afe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:05.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.870+0000 7f3ef93f7740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db01f26b50 0x55db01f9c610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:05.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.870+0000 7f3ef93f7740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3ef01ada70 con 0x55db01f26b50 2024-05-12T22:14:05.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.870+0000 7f3ef93f7740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f3ef01aeee0 con 0x55db01f1bb60 2024-05-12T22:14:05.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.870+0000 7f3ef93f7740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f3ef01ae140 con 0x55db01f1c480 2024-05-12T22:14:05.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.870+0000 7f3ef93f7740 10 client.0 _ll_register_callbacks cb 0x7f3ed8009900 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:14:05.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.871+0000 7f3eefe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db01f26b50 0x55db01f9c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:05.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.871+0000 7f3eeea00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55db01f1c480 0x55db01f2afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:05.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.871+0000 7f3eefe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db01f26b50 0x55db01f9c610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57146/0 (socket says 172.21.15.28:57146) 2024-05-12T22:14:05.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.871+0000 7f3eefe00640 1 -- 172.21.15.28:0/2314634145 learned_addr learned my addr 172.21.15.28:0/2314634145 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:14:05.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.871+0000 7f3eef400640 1 --2- 172.21.15.28:0/2314634145 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55db01f1bb60 0x55db01f2aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:05.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.871+0000 7f3eeea00640 1 -- 172.21.15.28:0/2314634145 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55db01f1bb60 msgr2=0x55db01f2aaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:14:05.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.871+0000 7f3eeea00640 1 --2- 172.21.15.28:0/2314634145 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55db01f1bb60 0x55db01f2aaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:05.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.871+0000 7f3eeea00640 1 -- 172.21.15.28:0/2314634145 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db01f26b50 msgr2=0x55db01f9c610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:14:05.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.871+0000 7f3eeea00640 1 --2- 172.21.15.28:0/2314634145 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db01f26b50 0x55db01f9c610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:05.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.871+0000 7f3eeea00640 1 -- 172.21.15.28:0/2314634145 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55db01f2b8e0 con 0x55db01f1c480 2024-05-12T22:14:05.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.871+0000 7f3eefe00640 1 --2- 172.21.15.28:0/2314634145 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db01f26b50 0x55db01f9c610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:14:05.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.871+0000 7f3eeea00640 1 --2- 172.21.15.28:0/2314634145 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55db01f1c480 0x55db01f2afe0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f3ed8005fa0 tx=0x7f3ed80095c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4b93ba2c70f9ea7c server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:14:05.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.871+0000 7f3edd600640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-12T22:14:05.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.872+0000 7f3edd600640 1 -- 172.21.15.28:0/2314634145 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ed80b9090 con 0x55db01f1c480 2024-05-12T22:14:05.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.872+0000 7f3edd600640 1 -- 172.21.15.28:0/2314634145 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f3ed8004b40 con 0x55db01f1c480 2024-05-12T22:14:05.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.872+0000 7f3ef93f7740 1 -- 172.21.15.28:0/2314634145 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3ee002b070 con 0x55db01f1c480 2024-05-12T22:14:05.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.872+0000 7f3edd600640 1 -- 172.21.15.28:0/2314634145 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f3ed80bcdf0 con 0x55db01f1c480 2024-05-12T22:14:05.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.872+0000 7f3ef93f7740 10 client.26368 fetch_fsmap learned FSMap version 26 2024-05-12T22:14:05.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.872+0000 7f3ef93f7740 1 -- 172.21.15.28:0/2314634145 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55db01f9dc80 con 0x55db01f1c480 2024-05-12T22:14:05.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.872+0000 7f3edd600640 1 -- 172.21.15.28:0/2314634145 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3ee002b070 con 0x55db01f1c480 2024-05-12T22:14:05.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.872+0000 7f3edd600640 1 -- 172.21.15.28:0/2314634145 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f3ed8083020 con 0x55db01f1c480 2024-05-12T22:14:05.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.872+0000 7f3ef93f7740 10 client.26368 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:14:05.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.872+0000 7f3ef93f7740 10 client.26368 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-05-12T22:14:05.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.872+0000 7f3ef93f7740 20 client.26368 populate_metadata read hostname 'smithi028' 2024-05-12T22:14:05.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.872+0000 7f3ef93f7740 1 --2- 172.21.15.28:0/2314634145 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x55db01f9e220 0x55db01fa0640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:05.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.872+0000 7f3ef93f7740 4 client.26368 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-12T22:14:05.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.872+0000 7f3ef93f7740 1 -- 172.21.15.28:0/2314634145 --> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55db01e884b0 con 0x55db01f9e220 2024-05-12T22:14:05.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.873+0000 7f3eef400640 1 --2- 172.21.15.28:0/2314634145 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x55db01f9e220 0x55db01fa0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:05.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.873+0000 7f3eef400640 1 --2- 172.21.15.28:0/2314634145 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x55db01f9e220 0x55db01fa0640 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e909c81bde3ded45 server_cookie=927bcd3131cbc4f6 in_seq=0 out_seq=0 2024-05-12T22:14:05.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.873+0000 7f3edd600640 10 client.26368 ms_handle_connect on v2:172.21.15.28:6840/761183870 2024-05-12T22:14:05.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.873+0000 7f3edd600640 1 -- 172.21.15.28:0/2314634145 <== mds.1 v2:172.21.15.28:6840/761183870 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55db01e884b0 con 0x55db01f9e220 2024-05-12T22:14:05.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.873+0000 7f3edd600640 10 client.26368 handle_command_reply: tid=0 2024-05-12T22:14:05.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.873+0000 7f3ef93f7740 1 client.26368 shutdown 2024-05-12T22:14:05.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.874+0000 7f3ef93f7740 1 -- 172.21.15.28:0/2314634145 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55db01f1c480 msgr2=0x55db01f2afe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:14:05.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.874+0000 7f3ef93f7740 1 --2- 172.21.15.28:0/2314634145 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55db01f1c480 0x55db01f2afe0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f3ed8005fa0 tx=0x7f3ed80095c0 comp rx=0 tx=0).stop 2024-05-12T22:14:05.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.874+0000 7f3ef93f7740 1 -- 172.21.15.28:0/2314634145 shutdown_connections 2024-05-12T22:14:05.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.874+0000 7f3ef93f7740 1 --2- 172.21.15.28:0/2314634145 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x55db01f9e220 0x55db01fa0640 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:05.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.874+0000 7f3ef93f7740 1 --2- 172.21.15.28:0/2314634145 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db01f26b50 0x55db01f9c610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:05.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.874+0000 7f3ef93f7740 1 --2- 172.21.15.28:0/2314634145 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55db01f1c480 0x55db01f2afe0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:05.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.874+0000 7f3ef93f7740 1 --2- 172.21.15.28:0/2314634145 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55db01f1bb60 0x55db01f2aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:05.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.874+0000 7f3edd600640 0 client.26368 ms_handle_reset on v2:172.21.15.28:6840/761183870 2024-05-12T22:14:05.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.874+0000 7f3ef93f7740 1 -- 172.21.15.28:0/2314634145 >> 172.21.15.28:0/2314634145 conn(0x55db01f17e80 msgr2=0x55db01f17000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:14:05.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.874+0000 7f3ef93f7740 1 -- 172.21.15.28:0/2314634145 shutdown_connections 2024-05-12T22:14:05.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.874+0000 7f3ef93f7740 1 -- 172.21.15.28:0/2314634145 wait complete. 2024-05-12T22:14:05.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.874+0000 7f3ef93f7740 20 client.26368 trim_cache size 0 max 16384 2024-05-12T22:14:05.883 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-05-12T22:14:05.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.875+0000 7f3ef5000640 1 -- 172.21.15.28:0/285229588 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f3ed40a2d00 msgr2=0x7f3ed40a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:14:05.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.875+0000 7f3ef5000640 1 --2- 172.21.15.28:0/285229588 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f3ed40a2d00 0x7f3ed40a51c0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f3ee8009bc0 tx=0x7f3ee8005f90 comp rx=0 tx=0).stop 2024-05-12T22:14:05.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.875+0000 7f3ef5000640 1 -- 172.21.15.28:0/285229588 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ef01042a0 msgr2=0x7f3ef019a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:14:05.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.875+0000 7f3ef5000640 1 --2- 172.21.15.28:0/285229588 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ef01042a0 0x7f3ef019a790 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f3ed8005e30 tx=0x7f3ed8003f90 comp rx=0 tx=0).stop 2024-05-12T22:14:05.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.875+0000 7f3ef5000640 1 -- 172.21.15.28:0/285229588 shutdown_connections 2024-05-12T22:14:05.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.875+0000 7f3ef5000640 1 --2- 172.21.15.28:0/285229588 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f3ed40a2d00 0x7f3ed40a51c0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:05.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.875+0000 7f3ef5000640 1 --2- 172.21.15.28:0/285229588 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3ef01098f0 0x7f3ef019eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:05.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.875+0000 7f3ef5000640 1 --2- 172.21.15.28:0/285229588 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ef01042a0 0x7f3ef019a790 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:05.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.875+0000 7f3ef5000640 1 --2- 172.21.15.28:0/285229588 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3ef01038e0 0x7f3ef019a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:05.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.875+0000 7f3ef5000640 1 -- 172.21.15.28:0/285229588 >> 172.21.15.28:0/285229588 conn(0x7f3ef00ff370 msgr2=0x7f3ef00ff930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:14:05.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.875+0000 7f3ef5000640 1 -- 172.21.15.28:0/285229588 shutdown_connections 2024-05-12T22:14:05.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:05.875+0000 7f3ef5000640 1 -- 172.21.15.28:0/285229588 wait complete. 2024-05-12T22:14:05.892 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-05-12T22:14:06.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.100+0000 7f98af000640 1 -- 172.21.15.28:0/3288676863 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98a81128e0 msgr2=0x7f98a8114cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:14:06.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.100+0000 7f98af000640 1 --2- 172.21.15.28:0/3288676863 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98a81128e0 0x7f98a8114cd0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f98a0009cd0 tx=0x7f98a002a970 comp rx=0 tx=0).stop 2024-05-12T22:14:06.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.100+0000 7f98af000640 1 -- 172.21.15.28:0/3288676863 shutdown_connections 2024-05-12T22:14:06.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.100+0000 7f98af000640 1 --2- 172.21.15.28:0/3288676863 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98a81128e0 0x7f98a8114cd0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:06.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.100+0000 7f98af000640 1 --2- 172.21.15.28:0/3288676863 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f98a8074420 0x7f98a81123a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:06.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.100+0000 7f98af000640 1 --2- 172.21.15.28:0/3288676863 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f98a8073b00 0x7f98a8073ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:06.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.100+0000 7f98af000640 1 -- 172.21.15.28:0/3288676863 >> 172.21.15.28:0/3288676863 conn(0x7f98a8078c80 msgr2=0x7f98a8079090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:14:06.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.100+0000 7f98af000640 1 -- 172.21.15.28:0/3288676863 shutdown_connections 2024-05-12T22:14:06.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.101+0000 7f98af000640 1 -- 172.21.15.28:0/3288676863 wait complete. 2024-05-12T22:14:06.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.101+0000 7f98af000640 1 Processor -- start 2024-05-12T22:14:06.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.102+0000 7f98af000640 1 -- start start 2024-05-12T22:14:06.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.102+0000 7f98af000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98a8073b00 0x7f98a819e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:06.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.102+0000 7f98af000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f98a8074420 0x7f98a819ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:06.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.102+0000 7f98af000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f98a81128e0 0x7f98a81a3000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:06.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.102+0000 7f98af000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f98a81181c0 con 0x7f98a8073b00 2024-05-12T22:14:06.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.102+0000 7f98af000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f98a8118040 con 0x7f98a8074420 2024-05-12T22:14:06.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.102+0000 7f98af000640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f98a8118340 con 0x7f98a81128e0 2024-05-12T22:14:06.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.102+0000 7f98a7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98a8073b00 0x7f98a819e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:06.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.102+0000 7f98a7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98a8073b00 0x7f98a819e740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57176/0 (socket says 172.21.15.28:57176) 2024-05-12T22:14:06.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.102+0000 7f98a7e00640 1 -- 172.21.15.28:0/705204246 learned_addr learned my addr 172.21.15.28:0/705204246 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:14:06.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.102+0000 7f989fe00640 1 --2- 172.21.15.28:0/705204246 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f98a8074420 0x7f98a819ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:06.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.102+0000 7f98ad000640 1 --2- 172.21.15.28:0/705204246 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f98a81128e0 0x7f98a81a3000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:06.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.103+0000 7f98a7e00640 1 -- 172.21.15.28:0/705204246 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f98a81128e0 msgr2=0x7f98a81a3000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:14:06.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.103+0000 7f98a7e00640 1 --2- 172.21.15.28:0/705204246 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f98a81128e0 0x7f98a81a3000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:06.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.103+0000 7f98a7e00640 1 -- 172.21.15.28:0/705204246 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f98a8074420 msgr2=0x7f98a819ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:14:06.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.103+0000 7f98a7e00640 1 --2- 172.21.15.28:0/705204246 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f98a8074420 0x7f98a819ec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:06.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.103+0000 7f98a7e00640 1 -- 172.21.15.28:0/705204246 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98a81a3730 con 0x7f98a8073b00 2024-05-12T22:14:06.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.103+0000 7f98a7e00640 1 --2- 172.21.15.28:0/705204246 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98a8073b00 0x7f98a819e740 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f989800e9f0 tx=0x7f989800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:14:06.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.103+0000 7f98a5600640 1 -- 172.21.15.28:0/705204246 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f989800cd80 con 0x7f98a8073b00 2024-05-12T22:14:06.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.103+0000 7f98a5600640 1 -- 172.21.15.28:0/705204246 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9898010910 con 0x7f98a8073b00 2024-05-12T22:14:06.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.103+0000 7f98a5600640 1 -- 172.21.15.28:0/705204246 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9898018990 con 0x7f98a8073b00 2024-05-12T22:14:06.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.103+0000 7f98af000640 1 -- 172.21.15.28:0/705204246 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98a81a3a20 con 0x7f98a8073b00 2024-05-12T22:14:06.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.103+0000 7f98af000640 1 -- 172.21.15.28:0/705204246 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98a81ab2b0 con 0x7f98a8073b00 2024-05-12T22:14:06.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.107+0000 7f98a5600640 1 -- 172.21.15.28:0/705204246 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7f9898010ac0 con 0x7f98a8073b00 2024-05-12T22:14:06.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.107+0000 7f98b1d9b740 1 Processor -- start 2024-05-12T22:14:06.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.107+0000 7f98b1d9b740 1 -- start start 2024-05-12T22:14:06.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.107+0000 7f98b1d9b740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561fa0525980 0x561fa0525d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:06.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.107+0000 7f98b1d9b740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561fa05262a0 0x561fa0530430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:06.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.107+0000 7f98b1d9b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561fa0530970 0x561fa0532d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:06.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.107+0000 7f98b1d9b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x561fa048aa30 con 0x561fa0530970 2024-05-12T22:14:06.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.107+0000 7f98b1d9b740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x561fa0432700 con 0x561fa05262a0 2024-05-12T22:14:06.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.107+0000 7f98b1d9b740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x561fa04869f0 con 0x561fa0525980 2024-05-12T22:14:06.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.108+0000 7f98a7e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561fa05262a0 0x561fa0530430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:06.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.108+0000 7f98a7e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561fa05262a0 0x561fa0530430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:55944/0 (socket says 172.21.15.28:55944) 2024-05-12T22:14:06.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.108+0000 7f989fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561fa0530970 0x561fa0532d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:06.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.108+0000 7f98a7e00640 1 -- 172.21.15.28:0/1305929092 learned_addr learned my addr 172.21.15.28:0/1305929092 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:14:06.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.108+0000 7f98ad000640 1 --2- 172.21.15.28:0/1305929092 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561fa0525980 0x561fa0525d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:06.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.108+0000 7f98a7e00640 1 -- 172.21.15.28:0/1305929092 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561fa0525980 msgr2=0x561fa0525d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:14:06.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.108+0000 7f98a7e00640 1 --2- 172.21.15.28:0/1305929092 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561fa0525980 0x561fa0525d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:06.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.108+0000 7f98a7e00640 1 -- 172.21.15.28:0/1305929092 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561fa0530970 msgr2=0x561fa0532d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:14:06.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.108+0000 7f98a7e00640 1 --2- 172.21.15.28:0/1305929092 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561fa0530970 0x561fa0532d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:06.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.108+0000 7f98a7e00640 1 -- 172.21.15.28:0/1305929092 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561fa019e3f0 con 0x561fa05262a0 2024-05-12T22:14:06.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.108+0000 7f98a5600640 1 --2- 172.21.15.28:0/705204246 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f988c0a2d80 0x7f988c0a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:06.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.109+0000 7f98a5600640 1 -- 172.21.15.28:0/705204246 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f98980cf7a0 con 0x7f98a8073b00 2024-05-12T22:14:06.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.109+0000 7f98ad000640 1 --2- 172.21.15.28:0/705204246 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f988c0a2d80 0x7f988c0a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:06.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.109+0000 7f98a7e00640 1 --2- 172.21.15.28:0/1305929092 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561fa05262a0 0x561fa0530430 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f98980cce00 tx=0x7f98980d7670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c70baebf03cf9cdf server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:14:06.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.109+0000 7f989d600640 1 -- 172.21.15.28:0/1305929092 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9898097f30 con 0x561fa05262a0 2024-05-12T22:14:06.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.109+0000 7f989d600640 1 -- 172.21.15.28:0/1305929092 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f989809c390 con 0x561fa05262a0 2024-05-12T22:14:06.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.109+0000 7f989d600640 1 -- 172.21.15.28:0/1305929092 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f989809b5f0 con 0x561fa05262a0 2024-05-12T22:14:06.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.110+0000 7f98b1d9b740 1 -- 172.21.15.28:0/1305929092 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561fa05262a0 msgr2=0x561fa0530430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:14:06.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.110+0000 7f98ad000640 1 --2- 172.21.15.28:0/705204246 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f988c0a2d80 0x7f988c0a5240 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f98a0005e30 tx=0x7f98a0005d80 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:14:06.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.110+0000 7f98b1d9b740 1 --2- 172.21.15.28:0/1305929092 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561fa05262a0 0x561fa0530430 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f98980cce00 tx=0x7f98980d7670 comp rx=0 tx=0).stop 2024-05-12T22:14:06.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.110+0000 7f98b1d9b740 1 -- 172.21.15.28:0/1305929092 shutdown_connections 2024-05-12T22:14:06.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.110+0000 7f98b1d9b740 1 --2- 172.21.15.28:0/1305929092 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561fa0530970 0x561fa0532d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:06.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.110+0000 7f98b1d9b740 1 --2- 172.21.15.28:0/1305929092 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561fa05262a0 0x561fa0530430 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:06.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.110+0000 7f98b1d9b740 1 --2- 172.21.15.28:0/1305929092 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561fa0525980 0x561fa0525d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:06.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.110+0000 7f98b1d9b740 1 -- 172.21.15.28:0/1305929092 >> 172.21.15.28:0/1305929092 conn(0x561fa0521ca0 msgr2=0x561fa0522080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:14:06.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.110+0000 7f98b1d9b740 1 -- 172.21.15.28:0/1305929092 shutdown_connections 2024-05-12T22:14:06.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.110+0000 7f98b1d9b740 1 -- 172.21.15.28:0/1305929092 wait complete. 2024-05-12T22:14:06.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.110+0000 7f98b1d9b740 1 -- start start 2024-05-12T22:14:06.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.111+0000 7f98b1d9b740 1 Processor -- start 2024-05-12T22:14:06.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.111+0000 7f98b1d9b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561fa0525980 0x561fa052d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:06.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.111+0000 7f98b1d9b740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561fa05262a0 0x561fa052de30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:06.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.111+0000 7f98b1d9b740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561fa0530970 0x561fa052e370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:06.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.111+0000 7f98b1d9b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x561fa0533760 con 0x561fa0525980 2024-05-12T22:14:06.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.111+0000 7f98b1d9b740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x561fa0445a70 con 0x561fa05262a0 2024-05-12T22:14:06.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.111+0000 7f98b1d9b740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x561fa05338e0 con 0x561fa0530970 2024-05-12T22:14:06.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.111+0000 7f98b1d9b740 10 client.0 _ll_register_callbacks cb 0x561fa0428ba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:14:06.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.111+0000 7f98a7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561fa0525980 0x561fa052d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:06.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.111+0000 7f98a7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561fa0525980 0x561fa052d8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57202/0 (socket says 172.21.15.28:57202) 2024-05-12T22:14:06.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.111+0000 7f98a7e00640 1 -- 172.21.15.28:0/3369234056 learned_addr learned my addr 172.21.15.28:0/3369234056 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:14:06.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.111+0000 7f989fe00640 1 --2- 172.21.15.28:0/3369234056 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561fa05262a0 0x561fa052de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:06.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.112+0000 7f98ad000640 1 --2- 172.21.15.28:0/3369234056 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561fa0530970 0x561fa052e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:06.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.112+0000 7f98a7e00640 1 -- 172.21.15.28:0/3369234056 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561fa0530970 msgr2=0x561fa052e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:14:06.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.112+0000 7f98a7e00640 1 --2- 172.21.15.28:0/3369234056 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561fa0530970 0x561fa052e370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:06.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.112+0000 7f98a7e00640 1 -- 172.21.15.28:0/3369234056 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561fa05262a0 msgr2=0x561fa052de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:14:06.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.112+0000 7f98a7e00640 1 --2- 172.21.15.28:0/3369234056 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561fa05262a0 0x561fa052de30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:06.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.112+0000 7f98a7e00640 1 -- 172.21.15.28:0/3369234056 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561fa02687d0 con 0x561fa0525980 2024-05-12T22:14:06.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.112+0000 7f98ad000640 1 --2- 172.21.15.28:0/3369234056 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561fa0530970 0x561fa052e370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:14:06.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.112+0000 7f989fe00640 1 --2- 172.21.15.28:0/3369234056 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561fa05262a0 0x561fa052de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:14:06.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.112+0000 7f98a7e00640 1 --2- 172.21.15.28:0/3369234056 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561fa0525980 0x561fa052d8f0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f989800e9c0 tx=0x7f9898009770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18da8b17045954cb server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:14:06.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.112+0000 7f9896000640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-12T22:14:06.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.112+0000 7f9896000640 1 -- 172.21.15.28:0/3369234056 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f989809b7d0 con 0x561fa0525980 2024-05-12T22:14:06.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.112+0000 7f9896000640 1 -- 172.21.15.28:0/3369234056 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f98980d5cb0 con 0x561fa0525980 2024-05-12T22:14:06.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.112+0000 7f98b1d9b740 1 -- 172.21.15.28:0/3369234056 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f989809c390 con 0x561fa0525980 2024-05-12T22:14:06.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.113+0000 7f9896000640 1 -- 172.21.15.28:0/3369234056 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f98980b9050 con 0x561fa0525980 2024-05-12T22:14:06.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.113+0000 7f98b1d9b740 10 client.16296 fetch_fsmap learned FSMap version 26 2024-05-12T22:14:06.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.113+0000 7f98b1d9b740 1 -- 172.21.15.28:0/3369234056 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x561fa053fa10 con 0x561fa0525980 2024-05-12T22:14:06.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.113+0000 7f9896000640 1 -- 172.21.15.28:0/3369234056 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f989809c390 con 0x561fa0525980 2024-05-12T22:14:06.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.113+0000 7f9896000640 1 -- 172.21.15.28:0/3369234056 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f98980b4070 con 0x561fa0525980 2024-05-12T22:14:06.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.113+0000 7f98b1d9b740 10 client.16296 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:14:06.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.113+0000 7f98b1d9b740 10 client.16296 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:14:06.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.113+0000 7f98b1d9b740 20 client.16296 populate_metadata read hostname 'smithi028' 2024-05-12T22:14:06.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.113+0000 7f98b1d9b740 1 --2- 172.21.15.28:0/3369234056 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x561fa0540d60 0x561fa0543150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:06.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.113+0000 7f98b1d9b740 4 client.16296 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-12T22:14:06.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.113+0000 7f98b1d9b740 1 -- 172.21.15.28:0/3369234056 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561fa0498160 con 0x561fa0540d60 2024-05-12T22:14:06.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.114+0000 7f989fe00640 1 --2- 172.21.15.28:0/3369234056 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x561fa0540d60 0x561fa0543150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:06.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.114+0000 7f989fe00640 1 --2- 172.21.15.28:0/3369234056 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x561fa0540d60 0x561fa0543150 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dfe120b76ad4d1ad server_cookie=2e4e504cf5bb500e in_seq=0 out_seq=0 2024-05-12T22:14:06.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.114+0000 7f9896000640 10 client.16296 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:14:06.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.115+0000 7f9896000640 1 -- 172.21.15.28:0/3369234056 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561fa0498160 con 0x561fa0540d60 2024-05-12T22:14:06.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.115+0000 7f9896000640 10 client.16296 handle_command_reply: tid=0 2024-05-12T22:14:06.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.115+0000 7f98b1d9b740 1 client.16296 shutdown 2024-05-12T22:14:06.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.115+0000 7f98b1d9b740 1 -- 172.21.15.28:0/3369234056 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561fa0525980 msgr2=0x561fa052d8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:14:06.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.115+0000 7f98b1d9b740 1 --2- 172.21.15.28:0/3369234056 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561fa0525980 0x561fa052d8f0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f989800e9c0 tx=0x7f9898009770 comp rx=0 tx=0).stop 2024-05-12T22:14:06.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.115+0000 7f98b1d9b740 1 -- 172.21.15.28:0/3369234056 shutdown_connections 2024-05-12T22:14:06.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.115+0000 7f98b1d9b740 1 --2- 172.21.15.28:0/3369234056 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x561fa0540d60 0x561fa0543150 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:06.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.115+0000 7f98b1d9b740 1 --2- 172.21.15.28:0/3369234056 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561fa0530970 0x561fa052e370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:06.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.115+0000 7f98b1d9b740 1 --2- 172.21.15.28:0/3369234056 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561fa05262a0 0x561fa052de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:06.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.115+0000 7f98b1d9b740 1 --2- 172.21.15.28:0/3369234056 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561fa0525980 0x561fa052d8f0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:06.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.116+0000 7f9896000640 0 client.16296 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:14:06.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.116+0000 7f98b1d9b740 1 -- 172.21.15.28:0/3369234056 >> 172.21.15.28:0/3369234056 conn(0x561fa0521ca0 msgr2=0x561fa0527aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:14:06.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.116+0000 7f98b1d9b740 1 -- 172.21.15.28:0/3369234056 shutdown_connections 2024-05-12T22:14:06.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.116+0000 7f98b1d9b740 1 -- 172.21.15.28:0/3369234056 wait complete. 2024-05-12T22:14:06.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.116+0000 7f98b1d9b740 20 client.16296 trim_cache size 0 max 16384 2024-05-12T22:14:06.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.130+0000 7f98b1d9b740 1 Processor -- start 2024-05-12T22:14:06.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.131+0000 7f98b1d9b740 1 -- start start 2024-05-12T22:14:06.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.131+0000 7f98b1d9b740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561fa0525980 0x561fa0530130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:06.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.131+0000 7f98b1d9b740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561fa05262a0 0x561fa05281f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:06.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.131+0000 7f98b1d9b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561fa0530970 0x561fa053c270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:06.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.131+0000 7f98b1d9b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f98a81b64a0 con 0x561fa0530970 2024-05-12T22:14:06.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.131+0000 7f98b1d9b740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f98a81b3d70 con 0x561fa0525980 2024-05-12T22:14:06.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.131+0000 7f98b1d9b740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f98a81b1bc0 con 0x561fa05262a0 2024-05-12T22:14:06.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.131+0000 7f98a7e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561fa0525980 0x561fa0530130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:06.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.131+0000 7f989fe00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561fa05262a0 0x561fa05281f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:06.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.131+0000 7f98a7e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561fa0525980 0x561fa0530130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:55970/0 (socket says 172.21.15.28:55970) 2024-05-12T22:14:06.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.131+0000 7f98a7e00640 1 -- 172.21.15.28:0/359795117 learned_addr learned my addr 172.21.15.28:0/359795117 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:14:06.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.131+0000 7f98ad000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561fa0530970 0x561fa053c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:06.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.132+0000 7f989fe00640 1 -- 172.21.15.28:0/359795117 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561fa0525980 msgr2=0x561fa0530130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:14:06.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.132+0000 7f989fe00640 1 --2- 172.21.15.28:0/359795117 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561fa0525980 0x561fa0530130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:06.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.132+0000 7f989fe00640 1 -- 172.21.15.28:0/359795117 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561fa0530970 msgr2=0x561fa053c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:14:06.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.132+0000 7f989fe00640 1 --2- 172.21.15.28:0/359795117 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561fa0530970 0x561fa053c270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:06.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.132+0000 7f989fe00640 1 -- 172.21.15.28:0/359795117 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561fa0522ed0 con 0x561fa05262a0 2024-05-12T22:14:06.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.132+0000 7f98a7e00640 1 --2- 172.21.15.28:0/359795117 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561fa0525980 0x561fa0530130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:14:06.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.132+0000 7f989fe00640 1 --2- 172.21.15.28:0/359795117 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561fa05262a0 0x561fa05281f0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x561fa052d150 tx=0x7f989000b410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f132549cf9dc95ed server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:14:06.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.132+0000 7f989cc00640 1 -- 172.21.15.28:0/359795117 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9890005220 con 0x561fa05262a0 2024-05-12T22:14:06.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.133+0000 7f989cc00640 1 -- 172.21.15.28:0/359795117 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9890004c80 con 0x561fa05262a0 2024-05-12T22:14:06.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.133+0000 7f989cc00640 1 -- 172.21.15.28:0/359795117 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98900207d0 con 0x561fa05262a0 2024-05-12T22:14:06.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.133+0000 7f98b1d9b740 1 -- 172.21.15.28:0/359795117 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561fa05262a0 msgr2=0x561fa05281f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:14:06.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.133+0000 7f98b1d9b740 1 --2- 172.21.15.28:0/359795117 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561fa05262a0 0x561fa05281f0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x561fa052d150 tx=0x7f989000b410 comp rx=0 tx=0).stop 2024-05-12T22:14:06.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.133+0000 7f98b1d9b740 1 -- 172.21.15.28:0/359795117 shutdown_connections 2024-05-12T22:14:06.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.133+0000 7f98b1d9b740 1 --2- 172.21.15.28:0/359795117 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561fa0530970 0x561fa053c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:06.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.133+0000 7f98b1d9b740 1 --2- 172.21.15.28:0/359795117 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561fa05262a0 0x561fa05281f0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:06.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.133+0000 7f98b1d9b740 1 --2- 172.21.15.28:0/359795117 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561fa0525980 0x561fa0530130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:06.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.133+0000 7f98b1d9b740 1 -- 172.21.15.28:0/359795117 >> 172.21.15.28:0/359795117 conn(0x561fa0521ca0 msgr2=0x561fa053bb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:14:06.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.133+0000 7f98b1d9b740 1 -- 172.21.15.28:0/359795117 shutdown_connections 2024-05-12T22:14:06.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.133+0000 7f98b1d9b740 1 -- 172.21.15.28:0/359795117 wait complete. 2024-05-12T22:14:06.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.134+0000 7f98b1d9b740 1 -- start start 2024-05-12T22:14:06.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.134+0000 7f98b1d9b740 1 Processor -- start 2024-05-12T22:14:06.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.134+0000 7f98b1d9b740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561fa0525980 0x561fa0546590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:06.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.134+0000 7f98b1d9b740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561fa05262a0 0x561fa0543920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:06.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.134+0000 7f98b1d9b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561fa0530970 0x561fa0543e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:06.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.134+0000 7f98b1d9b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f98a81b2230 con 0x561fa0530970 2024-05-12T22:14:06.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.134+0000 7f98b1d9b740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f98a81b36a0 con 0x561fa05262a0 2024-05-12T22:14:06.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.134+0000 7f98b1d9b740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f98a81b2900 con 0x561fa0525980 2024-05-12T22:14:06.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.135+0000 7f98b1d9b740 10 client.0 _ll_register_callbacks cb 0x7f98980d5a70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:14:06.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.135+0000 7f98ad000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561fa0530970 0x561fa0543e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:06.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.135+0000 7f98ad000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561fa0530970 0x561fa0543e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57228/0 (socket says 172.21.15.28:57228) 2024-05-12T22:14:06.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.135+0000 7f98ad000640 1 -- 172.21.15.28:0/3852100405 learned_addr learned my addr 172.21.15.28:0/3852100405 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:14:06.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.135+0000 7f98a7e00640 1 --2- 172.21.15.28:0/3852100405 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561fa0525980 0x561fa0546590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:06.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.135+0000 7f989fe00640 1 --2- 172.21.15.28:0/3852100405 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561fa05262a0 0x561fa0543920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:06.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.135+0000 7f98a7e00640 1 -- 172.21.15.28:0/3852100405 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561fa05262a0 msgr2=0x561fa0543920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:14:06.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.135+0000 7f98a7e00640 1 --2- 172.21.15.28:0/3852100405 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561fa05262a0 0x561fa0543920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:06.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.135+0000 7f98a7e00640 1 -- 172.21.15.28:0/3852100405 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561fa0530970 msgr2=0x561fa0543e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:14:06.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.135+0000 7f98a7e00640 1 --2- 172.21.15.28:0/3852100405 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561fa0530970 0x561fa0543e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:06.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.135+0000 7f98a7e00640 1 -- 172.21.15.28:0/3852100405 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561fa0544690 con 0x561fa0525980 2024-05-12T22:14:06.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.135+0000 7f98ad000640 1 --2- 172.21.15.28:0/3852100405 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561fa0530970 0x561fa0543e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:14:06.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.136+0000 7f98a7e00640 1 --2- 172.21.15.28:0/3852100405 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561fa0525980 0x561fa0546590 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f9898099f40 tx=0x7f9898009700 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1659663e352a658 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:14:06.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.136+0000 7f9896000640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-12T22:14:06.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.136+0000 7f9896000640 1 -- 172.21.15.28:0/3852100405 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f989809ba60 con 0x561fa0525980 2024-05-12T22:14:06.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.136+0000 7f9896000640 1 -- 172.21.15.28:0/3852100405 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f98980a4470 con 0x561fa0525980 2024-05-12T22:14:06.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.136+0000 7f98b1d9b740 1 -- 172.21.15.28:0/3852100405 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9890004c80 con 0x561fa0525980 2024-05-12T22:14:06.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.136+0000 7f9896000640 1 -- 172.21.15.28:0/3852100405 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f989809b5f0 con 0x561fa0525980 2024-05-12T22:14:06.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.136+0000 7f98b1d9b740 10 client.26386 fetch_fsmap learned FSMap version 26 2024-05-12T22:14:06.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.136+0000 7f98b1d9b740 1 -- 172.21.15.28:0/3852100405 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x561fa05a6fa0 con 0x561fa0525980 2024-05-12T22:14:06.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.136+0000 7f9896000640 1 -- 172.21.15.28:0/3852100405 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9890004c80 con 0x561fa0525980 2024-05-12T22:14:06.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.136+0000 7f9896000640 1 -- 172.21.15.28:0/3852100405 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f98980ac080 con 0x561fa0525980 2024-05-12T22:14:06.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.137+0000 7f98b1d9b740 10 client.26386 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:14:06.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.137+0000 7f98b1d9b740 10 client.26386 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:14:06.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.137+0000 7f98b1d9b740 20 client.26386 populate_metadata read hostname 'smithi028' 2024-05-12T22:14:06.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.137+0000 7f98b1d9b740 1 --2- 172.21.15.28:0/3852100405 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x561fa05a74d0 0x561fa05a98c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:06.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.137+0000 7f98b1d9b740 4 client.26386 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-12T22:14:06.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.137+0000 7f98b1d9b740 1 -- 172.21.15.28:0/3852100405 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561fa04924b0 con 0x561fa05a74d0 2024-05-12T22:14:06.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.137+0000 7f989fe00640 1 --2- 172.21.15.28:0/3852100405 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x561fa05a74d0 0x561fa05a98c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:06.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.138+0000 7f989fe00640 1 --2- 172.21.15.28:0/3852100405 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x561fa05a74d0 0x561fa05a98c0 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c5039772c98cb92c server_cookie=a915a8d1f5a8fd55 in_seq=0 out_seq=0 2024-05-12T22:14:06.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.138+0000 7f9896000640 10 client.26386 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:14:06.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.138+0000 7f9896000640 1 -- 172.21.15.28:0/3852100405 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561fa04924b0 con 0x561fa05a74d0 2024-05-12T22:14:06.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.138+0000 7f9896000640 10 client.26386 handle_command_reply: tid=0 2024-05-12T22:14:06.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.138+0000 7f98b1d9b740 1 client.26386 shutdown 2024-05-12T22:14:06.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.138+0000 7f98b1d9b740 1 -- 172.21.15.28:0/3852100405 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561fa0525980 msgr2=0x561fa0546590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:14:06.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.138+0000 7f98b1d9b740 1 --2- 172.21.15.28:0/3852100405 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561fa0525980 0x561fa0546590 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f9898099f40 tx=0x7f9898009700 comp rx=0 tx=0).stop 2024-05-12T22:14:06.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.138+0000 7f98b1d9b740 1 -- 172.21.15.28:0/3852100405 shutdown_connections 2024-05-12T22:14:06.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.138+0000 7f98b1d9b740 1 --2- 172.21.15.28:0/3852100405 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x561fa05a74d0 0x561fa05a98c0 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:06.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.138+0000 7f98b1d9b740 1 --2- 172.21.15.28:0/3852100405 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561fa0530970 0x561fa0543e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:06.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.138+0000 7f98b1d9b740 1 --2- 172.21.15.28:0/3852100405 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561fa05262a0 0x561fa0543920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:06.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.138+0000 7f98b1d9b740 1 --2- 172.21.15.28:0/3852100405 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561fa0525980 0x561fa0546590 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:06.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.138+0000 7f98b1d9b740 1 -- 172.21.15.28:0/3852100405 >> 172.21.15.28:0/3852100405 conn(0x561fa0521ca0 msgr2=0x561fa0520dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:14:06.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.138+0000 7f9896000640 0 client.26386 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:14:06.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.138+0000 7f98b1d9b740 1 -- 172.21.15.28:0/3852100405 shutdown_connections 2024-05-12T22:14:06.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.138+0000 7f98b1d9b740 1 -- 172.21.15.28:0/3852100405 wait complete. 2024-05-12T22:14:06.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.138+0000 7f98b1d9b740 20 client.26386 trim_cache size 0 max 16384 2024-05-12T22:14:06.147 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-05-12T22:14:06.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.139+0000 7f98af000640 1 -- 172.21.15.28:0/705204246 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f988c0a2d80 msgr2=0x7f988c0a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:14:06.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.139+0000 7f98af000640 1 --2- 172.21.15.28:0/705204246 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f988c0a2d80 0x7f988c0a5240 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f98a0005e30 tx=0x7f98a0005d80 comp rx=0 tx=0).stop 2024-05-12T22:14:06.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.139+0000 7f98af000640 1 -- 172.21.15.28:0/705204246 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98a8073b00 msgr2=0x7f98a819e740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:14:06.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.139+0000 7f98af000640 1 --2- 172.21.15.28:0/705204246 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98a8073b00 0x7f98a819e740 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f989800e9f0 tx=0x7f989800eeb0 comp rx=0 tx=0).stop 2024-05-12T22:14:06.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.139+0000 7f98af000640 1 -- 172.21.15.28:0/705204246 shutdown_connections 2024-05-12T22:14:06.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.139+0000 7f98af000640 1 --2- 172.21.15.28:0/705204246 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f988c0a2d80 0x7f988c0a5240 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:06.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.139+0000 7f98af000640 1 --2- 172.21.15.28:0/705204246 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f98a81128e0 0x7f98a81a3000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:06.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.139+0000 7f98af000640 1 --2- 172.21.15.28:0/705204246 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f98a8074420 0x7f98a819ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:06.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.139+0000 7f98af000640 1 --2- 172.21.15.28:0/705204246 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98a8073b00 0x7f98a819e740 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:06.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.139+0000 7f98af000640 1 -- 172.21.15.28:0/705204246 >> 172.21.15.28:0/705204246 conn(0x7f98a8078c80 msgr2=0x7f98a80fe7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:14:06.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.139+0000 7f98af000640 1 -- 172.21.15.28:0/705204246 shutdown_connections 2024-05-12T22:14:06.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:06.139+0000 7f98af000640 1 -- 172.21.15.28:0/705204246 wait complete. 2024-05-12T22:14:06.154 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:14:05 smithi028 ceph-mon[29847]: pgmap v156: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-05-12T22:14:06.154 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:14:05 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/4022679370' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:14:06.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:14:05 smithi086 ceph-mon[34297]: pgmap v156: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-05-12T22:14:06.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:14:05 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/4022679370' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:14:06.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:14:05 smithi186 ceph-mon[34785]: pgmap v156: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-05-12T22:14:06.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:14:05 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/4022679370' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:14:07.157 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-12T22:14:07.157 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-12T22:14:07.157 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-05-12T22:14:07.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.384+0000 7f3c9ae00640 1 -- 172.21.15.28:0/3736772519 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3c94103290 msgr2=0x7f3c9410b770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:14:07.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.384+0000 7f3c9ae00640 1 --2- 172.21.15.28:0/3736772519 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3c94103290 0x7f3c9410b770 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f3c80009900 tx=0x7f3c8002a480 comp rx=0 tx=0).stop 2024-05-12T22:14:07.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.384+0000 7f3c9ae00640 1 -- 172.21.15.28:0/3736772519 shutdown_connections 2024-05-12T22:14:07.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.384+0000 7f3c9ae00640 1 --2- 172.21.15.28:0/3736772519 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c9410bd40 0x7f3c9410e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:07.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.384+0000 7f3c9ae00640 1 --2- 172.21.15.28:0/3736772519 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3c94103290 0x7f3c9410b770 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:07.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.384+0000 7f3c9ae00640 1 --2- 172.21.15.28:0/3736772519 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c94102970 0x7f3c94102d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:07.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.384+0000 7f3c9ae00640 1 -- 172.21.15.28:0/3736772519 >> 172.21.15.28:0/3736772519 conn(0x7f3c940fc0b0 msgr2=0x7f3c940fe4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:14:07.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.384+0000 7f3c9ae00640 1 -- 172.21.15.28:0/3736772519 shutdown_connections 2024-05-12T22:14:07.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.384+0000 7f3c9ae00640 1 -- 172.21.15.28:0/3736772519 wait complete. 2024-05-12T22:14:07.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.385+0000 7f3c9ae00640 1 Processor -- start 2024-05-12T22:14:07.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.385+0000 7f3c9ae00640 1 -- start start 2024-05-12T22:14:07.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.386+0000 7f3c9ae00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c94102970 0x7f3c9419a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:07.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.386+0000 7f3c9ae00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3c94103290 0x7f3c9419a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:07.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.386+0000 7f3c9ae00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c9410bd40 0x7f3c9419eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:07.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.386+0000 7f3c9ae00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3c94111b20 con 0x7f3c9410bd40 2024-05-12T22:14:07.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.386+0000 7f3c9ae00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f3c941119a0 con 0x7f3c94102970 2024-05-12T22:14:07.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.386+0000 7f3c9ae00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f3c94111ca0 con 0x7f3c94103290 2024-05-12T22:14:07.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.386+0000 7f3c9a400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c9410bd40 0x7f3c9419eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:07.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.386+0000 7f3c99a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c94102970 0x7f3c9419a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:07.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.386+0000 7f3c99000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3c94103290 0x7f3c9419a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:07.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.386+0000 7f3c99000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3c94103290 0x7f3c9419a820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:33760/0 (socket says 172.21.15.28:33760) 2024-05-12T22:14:07.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.386+0000 7f3c99000640 1 -- 172.21.15.28:0/4126872931 learned_addr learned my addr 172.21.15.28:0/4126872931 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:14:07.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.386+0000 7f3c99a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c94102970 0x7f3c9419a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:56002/0 (socket says 172.21.15.28:56002) 2024-05-12T22:14:07.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.387+0000 7f3c99a00640 1 -- 172.21.15.28:0/4126872931 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3c94103290 msgr2=0x7f3c9419a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:14:07.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.387+0000 7f3c99a00640 1 --2- 172.21.15.28:0/4126872931 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3c94103290 0x7f3c9419a820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:07.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.387+0000 7f3c99a00640 1 -- 172.21.15.28:0/4126872931 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c9410bd40 msgr2=0x7f3c9419eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:14:07.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.387+0000 7f3c99a00640 1 --2- 172.21.15.28:0/4126872931 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c9410bd40 0x7f3c9419eba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:07.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.387+0000 7f3c99a00640 1 -- 172.21.15.28:0/4126872931 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c9419f2d0 con 0x7f3c94102970 2024-05-12T22:14:07.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.387+0000 7f3c9a400640 1 --2- 172.21.15.28:0/4126872931 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c9410bd40 0x7f3c9419eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:14:07.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.387+0000 7f3c99a00640 1 --2- 172.21.15.28:0/4126872931 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c94102970 0x7f3c9419a2e0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f3c8800eb00 tx=0x7f3c8800c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:14:07.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.387+0000 7f3c92000640 1 -- 172.21.15.28:0/4126872931 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c88010430 con 0x7f3c94102970 2024-05-12T22:14:07.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.388+0000 7f3c9ae00640 1 -- 172.21.15.28:0/4126872931 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c9419f5c0 con 0x7f3c94102970 2024-05-12T22:14:07.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.388+0000 7f3c92000640 1 -- 172.21.15.28:0/4126872931 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3c88010ab0 con 0x7f3c94102970 2024-05-12T22:14:07.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.388+0000 7f3c92000640 1 -- 172.21.15.28:0/4126872931 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c88017bb0 con 0x7f3c94102970 2024-05-12T22:14:07.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.388+0000 7f3c9ae00640 1 -- 172.21.15.28:0/4126872931 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c941a6e50 con 0x7f3c94102970 2024-05-12T22:14:07.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.391+0000 7f3c9f16a740 1 Processor -- start 2024-05-12T22:14:07.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.391+0000 7f3c9f16a740 1 -- start start 2024-05-12T22:14:07.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.391+0000 7f3c9f16a740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5599522d2500 0x5599522d28e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:07.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.391+0000 7f3c9f16a740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5599522d2eb0 0x5599522dcf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:07.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.391+0000 7f3c9f16a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5599522dd520 0x5599522df970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:07.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.391+0000 7f3c92000640 1 -- 172.21.15.28:0/4126872931 <== mon.2 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7f3c88010c60 con 0x7f3c94102970 2024-05-12T22:14:07.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.391+0000 7f3c9f16a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5599522367c0 con 0x5599522dd520 2024-05-12T22:14:07.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.391+0000 7f3c9f16a740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x5599521de790 con 0x5599522d2500 2024-05-12T22:14:07.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.391+0000 7f3c9f16a740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x559952232780 con 0x5599522d2eb0 2024-05-12T22:14:07.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.392+0000 7f3c9a400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5599522d2500 0x5599522d28e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:07.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.392+0000 7f3c9a400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5599522d2500 0x5599522d28e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:56016/0 (socket says 172.21.15.28:56016) 2024-05-12T22:14:07.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.392+0000 7f3c9a400640 1 -- 172.21.15.28:0/436186370 learned_addr learned my addr 172.21.15.28:0/436186370 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:14:07.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.392+0000 7f3c99000640 1 --2- 172.21.15.28:0/436186370 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5599522dd520 0x5599522df970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:07.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.392+0000 7f3c99a00640 1 --2- 172.21.15.28:0/436186370 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5599522d2eb0 0x5599522dcf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:07.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.392+0000 7f3c9a400640 1 -- 172.21.15.28:0/436186370 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5599522d2eb0 msgr2=0x5599522dcf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:14:07.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.392+0000 7f3c9a400640 1 --2- 172.21.15.28:0/436186370 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5599522d2eb0 0x5599522dcf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:07.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.392+0000 7f3c9a400640 1 -- 172.21.15.28:0/436186370 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5599522dd520 msgr2=0x5599522df970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:14:07.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.392+0000 7f3c9a400640 1 --2- 172.21.15.28:0/436186370 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5599522dd520 0x5599522df970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:07.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.392+0000 7f3c9a400640 1 -- 172.21.15.28:0/436186370 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5599521fe6a0 con 0x5599522d2500 2024-05-12T22:14:07.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.393+0000 7f3c9a400640 1 --2- 172.21.15.28:0/436186370 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5599522d2500 0x5599522d28e0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f3c8c009f90 tx=0x7f3c8c0118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3fe67929e8a048fc server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:14:07.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.393+0000 7f3c86a00640 1 -- 172.21.15.28:0/436186370 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c8c01e070 con 0x5599522d2500 2024-05-12T22:14:07.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.393+0000 7f3c86a00640 1 -- 172.21.15.28:0/436186370 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3c8c004310 con 0x5599522d2500 2024-05-12T22:14:07.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.393+0000 7f3c86a00640 1 -- 172.21.15.28:0/436186370 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c8c00f9e0 con 0x5599522d2500 2024-05-12T22:14:07.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.393+0000 7f3c92000640 1 --2- 172.21.15.28:0/4126872931 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f3c7c0a2d00 0x7f3c7c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:07.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.393+0000 7f3c92000640 1 -- 172.21.15.28:0/4126872931 <== mon.2 v2:172.21.15.86:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f3c880cfe30 con 0x7f3c94102970 2024-05-12T22:14:07.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.393+0000 7f3c99a00640 1 --2- 172.21.15.28:0/4126872931 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f3c7c0a2d00 0x7f3c7c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:07.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.394+0000 7f3c9f16a740 1 -- 172.21.15.28:0/436186370 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5599522d2500 msgr2=0x5599522d28e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:14:07.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.394+0000 7f3c9f16a740 1 --2- 172.21.15.28:0/436186370 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5599522d2500 0x5599522d28e0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f3c8c009f90 tx=0x7f3c8c0118d0 comp rx=0 tx=0).stop 2024-05-12T22:14:07.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.394+0000 7f3c9f16a740 1 -- 172.21.15.28:0/436186370 shutdown_connections 2024-05-12T22:14:07.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.394+0000 7f3c9f16a740 1 --2- 172.21.15.28:0/436186370 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5599522dd520 0x5599522df970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:07.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.394+0000 7f3c9f16a740 1 --2- 172.21.15.28:0/436186370 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5599522d2eb0 0x5599522dcf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:07.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.394+0000 7f3c9f16a740 1 --2- 172.21.15.28:0/436186370 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5599522d2500 0x5599522d28e0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:07.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.394+0000 7f3c9f16a740 1 -- 172.21.15.28:0/436186370 >> 172.21.15.28:0/436186370 conn(0x5599522ce5b0 msgr2=0x5599522ce990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:14:07.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.394+0000 7f3c9f16a740 1 -- 172.21.15.28:0/436186370 shutdown_connections 2024-05-12T22:14:07.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.394+0000 7f3c9f16a740 1 -- 172.21.15.28:0/436186370 wait complete. 2024-05-12T22:14:07.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.394+0000 7f3c99a00640 1 --2- 172.21.15.28:0/4126872931 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f3c7c0a2d00 0x7f3c7c0a51c0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x5599522d3ac0 tx=0x7f3c880d38e0 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:14:07.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.395+0000 7f3c9f16a740 1 -- start start 2024-05-12T22:14:07.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.395+0000 7f3c9f16a740 1 Processor -- start 2024-05-12T22:14:07.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.395+0000 7f3c9f16a740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5599522d2500 0x5599522da4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:07.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.395+0000 7f3c9f16a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5599522d2eb0 0x5599522daa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:07.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.395+0000 7f3c9f16a740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5599522dd520 0x5599522daf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:07.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.395+0000 7f3c9f16a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5599522e0530 con 0x5599522d2eb0 2024-05-12T22:14:07.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.395+0000 7f3c9f16a740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x5599521f1a90 con 0x5599522dd520 2024-05-12T22:14:07.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.395+0000 7f3c9f16a740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5599522e06b0 con 0x5599522d2500 2024-05-12T22:14:07.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.396+0000 7f3c9f16a740 10 client.0 _ll_register_callbacks cb 0x55995222caf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:14:07.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.396+0000 7f3c99000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5599522d2eb0 0x5599522daa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:07.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.396+0000 7f3c99000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5599522d2eb0 0x5599522daa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57278/0 (socket says 172.21.15.28:57278) 2024-05-12T22:14:07.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.396+0000 7f3c99000640 1 -- 172.21.15.28:0/1756776463 learned_addr learned my addr 172.21.15.28:0/1756776463 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:14:07.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.396+0000 7f3c9a400640 1 --2- 172.21.15.28:0/1756776463 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5599522d2500 0x5599522da4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:07.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.396+0000 7f3c9a400640 1 --2- 172.21.15.28:0/1756776463 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5599522dd520 0x5599522daf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:07.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.396+0000 7f3c99000640 1 -- 172.21.15.28:0/1756776463 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5599522d2500 msgr2=0x5599522da4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:14:07.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.396+0000 7f3c99000640 1 --2- 172.21.15.28:0/1756776463 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5599522d2500 0x5599522da4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:07.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.396+0000 7f3c99000640 1 -- 172.21.15.28:0/1756776463 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5599522dd520 msgr2=0x5599522daf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:14:07.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.396+0000 7f3c99000640 1 --2- 172.21.15.28:0/1756776463 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5599522dd520 0x5599522daf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:07.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.396+0000 7f3c99000640 1 -- 172.21.15.28:0/1756776463 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5599520127e0 con 0x5599522d2eb0 2024-05-12T22:14:07.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.396+0000 7f3c99000640 1 --2- 172.21.15.28:0/1756776463 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5599522d2eb0 0x5599522daa00 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f3c80006010 tx=0x7f3c800042e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1977bed8a3469181 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:14:07.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.397+0000 7f3c7b400640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-12T22:14:07.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.397+0000 7f3c7b400640 1 -- 172.21.15.28:0/1756776463 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c80035040 con 0x5599522d2eb0 2024-05-12T22:14:07.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.397+0000 7f3c7b400640 1 -- 172.21.15.28:0/1756776463 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f3c80005220 con 0x5599522d2eb0 2024-05-12T22:14:07.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.397+0000 7f3c9f16a740 1 -- 172.21.15.28:0/1756776463 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3c8c004310 con 0x5599522d2eb0 2024-05-12T22:14:07.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.397+0000 7f3c7b400640 1 -- 172.21.15.28:0/1756776463 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f3c80005420 con 0x5599522d2eb0 2024-05-12T22:14:07.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.397+0000 7f3c9f16a740 10 client.16314 fetch_fsmap learned FSMap version 26 2024-05-12T22:14:07.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.397+0000 7f3c9f16a740 1 -- 172.21.15.28:0/1756776463 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x559952072f70 con 0x5599522d2eb0 2024-05-12T22:14:07.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.397+0000 7f3c7b400640 1 -- 172.21.15.28:0/1756776463 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3c8c004310 con 0x5599522d2eb0 2024-05-12T22:14:07.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.398+0000 7f3c7b400640 1 -- 172.21.15.28:0/1756776463 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f3c80003860 con 0x5599522d2eb0 2024-05-12T22:14:07.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.398+0000 7f3c9f16a740 10 client.16314 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:14:07.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.398+0000 7f3c9f16a740 10 client.16314 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:14:07.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.398+0000 7f3c9f16a740 20 client.16314 populate_metadata read hostname 'smithi028' 2024-05-12T22:14:07.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.398+0000 7f3c9f16a740 1 --2- 172.21.15.28:0/1756776463 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5599522ed860 0x5599522efc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:07.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.398+0000 7f3c9f16a740 4 client.16314 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-12T22:14:07.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.398+0000 7f3c9f16a740 1 -- 172.21.15.28:0/1756776463 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55995224a3a0 con 0x5599522ed860 2024-05-12T22:14:07.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.398+0000 7f3c9a400640 1 --2- 172.21.15.28:0/1756776463 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5599522ed860 0x5599522efc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:07.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.399+0000 7f3c9a400640 1 --2- 172.21.15.28:0/1756776463 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5599522ed860 0x5599522efc80 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c83d1e192814dc1c server_cookie=d188c54c0e825363 in_seq=0 out_seq=0 2024-05-12T22:14:07.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.399+0000 7f3c7b400640 10 client.16314 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:14:07.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.400+0000 7f3c7b400640 1 -- 172.21.15.28:0/1756776463 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55995224a3a0 con 0x5599522ed860 2024-05-12T22:14:07.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.400+0000 7f3c7b400640 10 client.16314 handle_command_reply: tid=0 2024-05-12T22:14:07.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.400+0000 7f3c9f16a740 1 client.16314 shutdown 2024-05-12T22:14:07.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.400+0000 7f3c9f16a740 1 -- 172.21.15.28:0/1756776463 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5599522d2eb0 msgr2=0x5599522daa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:14:07.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.400+0000 7f3c9f16a740 1 --2- 172.21.15.28:0/1756776463 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5599522d2eb0 0x5599522daa00 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f3c80006010 tx=0x7f3c800042e0 comp rx=0 tx=0).stop 2024-05-12T22:14:07.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.400+0000 7f3c9f16a740 1 -- 172.21.15.28:0/1756776463 shutdown_connections 2024-05-12T22:14:07.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.400+0000 7f3c9f16a740 1 --2- 172.21.15.28:0/1756776463 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5599522ed860 0x5599522efc80 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:07.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.400+0000 7f3c9f16a740 1 --2- 172.21.15.28:0/1756776463 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5599522dd520 0x5599522daf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:07.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.400+0000 7f3c9f16a740 1 --2- 172.21.15.28:0/1756776463 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5599522d2eb0 0x5599522daa00 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:07.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.400+0000 7f3c7b400640 0 client.16314 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:14:07.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.400+0000 7f3c9f16a740 1 --2- 172.21.15.28:0/1756776463 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5599522d2500 0x5599522da4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:07.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.400+0000 7f3c9f16a740 1 -- 172.21.15.28:0/1756776463 >> 172.21.15.28:0/1756776463 conn(0x5599522ce5b0 msgr2=0x5599522d4630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:14:07.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.400+0000 7f3c9f16a740 1 -- 172.21.15.28:0/1756776463 shutdown_connections 2024-05-12T22:14:07.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.401+0000 7f3c9f16a740 1 -- 172.21.15.28:0/1756776463 wait complete. 2024-05-12T22:14:07.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.401+0000 7f3c9f16a740 20 client.16314 trim_cache size 0 max 16384 2024-05-12T22:14:07.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.420+0000 7f3c9f16a740 1 Processor -- start 2024-05-12T22:14:07.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.420+0000 7f3c9f16a740 1 -- start start 2024-05-12T22:14:07.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.421+0000 7f3c9f16a740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5599522d2500 0x5599522e93f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:07.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.421+0000 7f3c9f16a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5599522d2eb0 0x5599522dad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:07.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.421+0000 7f3c9f16a740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5599522dd520 0x5599522f5550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:07.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.421+0000 7f3c9f16a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3c941b1d90 con 0x5599522d2eb0 2024-05-12T22:14:07.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.421+0000 7f3c9f16a740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f3c941af660 con 0x5599522dd520 2024-05-12T22:14:07.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.421+0000 7f3c9f16a740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f3c941ad4b0 con 0x5599522d2500 2024-05-12T22:14:07.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.421+0000 7f3c99000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5599522d2eb0 0x5599522dad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:07.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.421+0000 7f3c99000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5599522d2eb0 0x5599522dad60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57288/0 (socket says 172.21.15.28:57288) 2024-05-12T22:14:07.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.421+0000 7f3c99000640 1 -- 172.21.15.28:0/3900117932 learned_addr learned my addr 172.21.15.28:0/3900117932 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:14:07.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.421+0000 7f3c9a400640 1 --2- 172.21.15.28:0/3900117932 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5599522dd520 0x5599522f5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:07.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.421+0000 7f3c9a400640 1 --2- 172.21.15.28:0/3900117932 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5599522d2500 0x5599522e93f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:07.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.421+0000 7f3c99000640 1 -- 172.21.15.28:0/3900117932 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5599522d2500 msgr2=0x5599522e93f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:14:07.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.421+0000 7f3c99000640 1 --2- 172.21.15.28:0/3900117932 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5599522d2500 0x5599522e93f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:07.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.421+0000 7f3c99000640 1 -- 172.21.15.28:0/3900117932 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5599522dd520 msgr2=0x5599522f5550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:14:07.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.421+0000 7f3c99000640 1 --2- 172.21.15.28:0/3900117932 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5599522dd520 0x5599522f5550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:07.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.421+0000 7f3c99000640 1 -- 172.21.15.28:0/3900117932 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559951f0a170 con 0x5599522d2eb0 2024-05-12T22:14:07.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.422+0000 7f3c99000640 1 --2- 172.21.15.28:0/3900117932 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5599522d2eb0 0x5599522dad60 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f3c80001eb0 tx=0x7f3c80009520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6bedfcb5fb449e8 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:14:07.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.422+0000 7f3c86000640 1 -- 172.21.15.28:0/3900117932 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c80035030 con 0x5599522d2eb0 2024-05-12T22:14:07.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.422+0000 7f3c86000640 1 -- 172.21.15.28:0/3900117932 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3c800055c0 con 0x5599522d2eb0 2024-05-12T22:14:07.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.422+0000 7f3c86000640 1 -- 172.21.15.28:0/3900117932 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c80052040 con 0x5599522d2eb0 2024-05-12T22:14:07.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.423+0000 7f3c9f16a740 1 -- 172.21.15.28:0/3900117932 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5599522d2eb0 msgr2=0x5599522dad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:14:07.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.423+0000 7f3c9f16a740 1 --2- 172.21.15.28:0/3900117932 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5599522d2eb0 0x5599522dad60 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f3c80001eb0 tx=0x7f3c80009520 comp rx=0 tx=0).stop 2024-05-12T22:14:07.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.423+0000 7f3c9f16a740 1 -- 172.21.15.28:0/3900117932 shutdown_connections 2024-05-12T22:14:07.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.423+0000 7f3c9f16a740 1 --2- 172.21.15.28:0/3900117932 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5599522dd520 0x5599522f5550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:07.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.423+0000 7f3c9f16a740 1 --2- 172.21.15.28:0/3900117932 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5599522d2eb0 0x5599522dad60 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:07.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.423+0000 7f3c9f16a740 1 --2- 172.21.15.28:0/3900117932 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5599522d2500 0x5599522e93f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:07.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.423+0000 7f3c9f16a740 1 -- 172.21.15.28:0/3900117932 >> 172.21.15.28:0/3900117932 conn(0x5599522ce5b0 msgr2=0x5599522dbcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:14:07.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.423+0000 7f3c9f16a740 1 -- 172.21.15.28:0/3900117932 shutdown_connections 2024-05-12T22:14:07.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.423+0000 7f3c9f16a740 1 -- 172.21.15.28:0/3900117932 wait complete. 2024-05-12T22:14:07.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.423+0000 7f3c9f16a740 1 -- start start 2024-05-12T22:14:07.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.423+0000 7f3c9f16a740 1 Processor -- start 2024-05-12T22:14:07.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.424+0000 7f3c9f16a740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5599522d2500 0x5599522e5af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:07.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.424+0000 7f3c9f16a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5599522d2eb0 0x5599522e28d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:07.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.424+0000 7f3c9f16a740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5599522dd520 0x5599522e2e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:07.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.424+0000 7f3c9f16a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3c941adb20 con 0x5599522d2eb0 2024-05-12T22:14:07.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.424+0000 7f3c9f16a740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f3c941aef90 con 0x5599522dd520 2024-05-12T22:14:07.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.424+0000 7f3c9f16a740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f3c941ae1f0 con 0x5599522d2500 2024-05-12T22:14:07.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.424+0000 7f3c9f16a740 10 client.0 _ll_register_callbacks cb 0x7f3c80009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:14:07.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.424+0000 7f3c99000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5599522d2eb0 0x5599522e28d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:07.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.424+0000 7f3c99000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5599522d2eb0 0x5599522e28d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57292/0 (socket says 172.21.15.28:57292) 2024-05-12T22:14:07.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.424+0000 7f3c99000640 1 -- 172.21.15.28:0/1256418389 learned_addr learned my addr 172.21.15.28:0/1256418389 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:14:07.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.424+0000 7f3c9a400640 1 --2- 172.21.15.28:0/1256418389 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5599522d2500 0x5599522e5af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:07.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.425+0000 7f3c9a400640 1 --2- 172.21.15.28:0/1256418389 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5599522dd520 0x5599522e2e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:07.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.425+0000 7f3c99000640 1 -- 172.21.15.28:0/1256418389 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5599522d2500 msgr2=0x5599522e5af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:14:07.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.425+0000 7f3c99000640 1 --2- 172.21.15.28:0/1256418389 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5599522d2500 0x5599522e5af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:07.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.425+0000 7f3c99000640 1 -- 172.21.15.28:0/1256418389 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5599522dd520 msgr2=0x5599522e2e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:14:07.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.425+0000 7f3c99000640 1 --2- 172.21.15.28:0/1256418389 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5599522dd520 0x5599522e2e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:07.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.425+0000 7f3c99000640 1 -- 172.21.15.28:0/1256418389 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5599522e3380 con 0x5599522d2eb0 2024-05-12T22:14:07.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.425+0000 7f3c99000640 1 --2- 172.21.15.28:0/1256418389 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5599522d2eb0 0x5599522e28d0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f3c80009850 tx=0x7f3c8002a480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1711cea0c7e276db server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:14:07.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.425+0000 7f3c7b400640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-12T22:14:07.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.425+0000 7f3c7b400640 1 -- 172.21.15.28:0/1256418389 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c80065060 con 0x5599522d2eb0 2024-05-12T22:14:07.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.425+0000 7f3c7b400640 1 -- 172.21.15.28:0/1256418389 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f3c800042c0 con 0x5599522d2eb0 2024-05-12T22:14:07.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.425+0000 7f3c9f16a740 1 -- 172.21.15.28:0/1256418389 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3c800055c0 con 0x5599522d2eb0 2024-05-12T22:14:07.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.426+0000 7f3c7b400640 1 -- 172.21.15.28:0/1256418389 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f3c80052040 con 0x5599522d2eb0 2024-05-12T22:14:07.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.426+0000 7f3c9f16a740 10 client.16320 fetch_fsmap learned FSMap version 26 2024-05-12T22:14:07.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.426+0000 7f3c9f16a740 1 -- 172.21.15.28:0/1256418389 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x559952354b50 con 0x5599522d2eb0 2024-05-12T22:14:07.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.426+0000 7f3c7b400640 1 -- 172.21.15.28:0/1256418389 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3c800055c0 con 0x5599522d2eb0 2024-05-12T22:14:07.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.426+0000 7f3c7b400640 1 -- 172.21.15.28:0/1256418389 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f3c8006a790 con 0x5599522d2eb0 2024-05-12T22:14:07.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.426+0000 7f3c9f16a740 10 client.16320 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:14:07.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.426+0000 7f3c9f16a740 10 client.16320 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:14:07.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.426+0000 7f3c9f16a740 20 client.16320 populate_metadata read hostname 'smithi028' 2024-05-12T22:14:07.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.426+0000 7f3c9f16a740 1 --2- 172.21.15.28:0/1256418389 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x559952355230 0x559952357650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:07.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.426+0000 7f3c9f16a740 4 client.16320 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-12T22:14:07.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.426+0000 7f3c9f16a740 1 -- 172.21.15.28:0/1256418389 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x559952243ef0 con 0x559952355230 2024-05-12T22:14:07.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.427+0000 7f3c9a400640 1 --2- 172.21.15.28:0/1256418389 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x559952355230 0x559952357650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:07.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.427+0000 7f3c9a400640 1 --2- 172.21.15.28:0/1256418389 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x559952355230 0x559952357650 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9305e209026053f4 server_cookie=8cbbee1f592f4f82 in_seq=0 out_seq=0 2024-05-12T22:14:07.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.427+0000 7f3c7b400640 10 client.16320 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:14:07.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.428+0000 7f3c7b400640 1 -- 172.21.15.28:0/1256418389 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x559952243ef0 con 0x559952355230 2024-05-12T22:14:07.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.428+0000 7f3c7b400640 10 client.16320 handle_command_reply: tid=0 2024-05-12T22:14:07.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.428+0000 7f3c9f16a740 1 client.16320 shutdown 2024-05-12T22:14:07.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.429+0000 7f3c9f16a740 1 -- 172.21.15.28:0/1256418389 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5599522d2eb0 msgr2=0x5599522e28d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:14:07.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.429+0000 7f3c9f16a740 1 --2- 172.21.15.28:0/1256418389 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5599522d2eb0 0x5599522e28d0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f3c80009850 tx=0x7f3c8002a480 comp rx=0 tx=0).stop 2024-05-12T22:14:07.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.429+0000 7f3c9f16a740 1 -- 172.21.15.28:0/1256418389 shutdown_connections 2024-05-12T22:14:07.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.429+0000 7f3c9f16a740 1 --2- 172.21.15.28:0/1256418389 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x559952355230 0x559952357650 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:07.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.429+0000 7f3c9f16a740 1 --2- 172.21.15.28:0/1256418389 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5599522dd520 0x5599522e2e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:07.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.429+0000 7f3c9f16a740 1 --2- 172.21.15.28:0/1256418389 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5599522d2eb0 0x5599522e28d0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:07.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.429+0000 7f3c9f16a740 1 --2- 172.21.15.28:0/1256418389 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5599522d2500 0x5599522e5af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:07.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.429+0000 7f3c7b400640 0 client.16320 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:14:07.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.429+0000 7f3c9f16a740 1 -- 172.21.15.28:0/1256418389 >> 172.21.15.28:0/1256418389 conn(0x5599522ce5b0 msgr2=0x5599522dc580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:14:07.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.429+0000 7f3c9f16a740 1 -- 172.21.15.28:0/1256418389 shutdown_connections 2024-05-12T22:14:07.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.429+0000 7f3c9f16a740 1 -- 172.21.15.28:0/1256418389 wait complete. 2024-05-12T22:14:07.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.429+0000 7f3c9f16a740 20 client.16320 trim_cache size 0 max 16384 2024-05-12T22:14:07.438 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-05-12T22:14:07.439 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-05-12T22:14:07.439 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "2860a46a-25c9-4783-b88a-954434568cec", 2024-05-12T22:14:07.439 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-05-12T22:14:07.439 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-05-12T22:14:07.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.430+0000 7f3c9ae00640 1 -- 172.21.15.28:0/4126872931 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f3c7c0a2d00 msgr2=0x7f3c7c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:14:07.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.430+0000 7f3c9ae00640 1 --2- 172.21.15.28:0/4126872931 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f3c7c0a2d00 0x7f3c7c0a51c0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x5599522d3ac0 tx=0x7f3c880d38e0 comp rx=0 tx=0).stop 2024-05-12T22:14:07.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.430+0000 7f3c9ae00640 1 -- 172.21.15.28:0/4126872931 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c94102970 msgr2=0x7f3c9419a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:14:07.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.430+0000 7f3c9ae00640 1 --2- 172.21.15.28:0/4126872931 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c94102970 0x7f3c9419a2e0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f3c8800eb00 tx=0x7f3c8800c2d0 comp rx=0 tx=0).stop 2024-05-12T22:14:07.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.430+0000 7f3c9ae00640 1 -- 172.21.15.28:0/4126872931 shutdown_connections 2024-05-12T22:14:07.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.430+0000 7f3c9ae00640 1 --2- 172.21.15.28:0/4126872931 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f3c7c0a2d00 0x7f3c7c0a51c0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:07.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.430+0000 7f3c9ae00640 1 --2- 172.21.15.28:0/4126872931 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c9410bd40 0x7f3c9419eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:07.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.430+0000 7f3c9ae00640 1 --2- 172.21.15.28:0/4126872931 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3c94103290 0x7f3c9419a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:07.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.430+0000 7f3c9ae00640 1 --2- 172.21.15.28:0/4126872931 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c94102970 0x7f3c9419a2e0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:07.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.430+0000 7f3c9ae00640 1 -- 172.21.15.28:0/4126872931 >> 172.21.15.28:0/4126872931 conn(0x7f3c940fc0b0 msgr2=0x7f3c94100550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:14:07.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.430+0000 7f3c9ae00640 1 -- 172.21.15.28:0/4126872931 shutdown_connections 2024-05-12T22:14:07.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.430+0000 7f3c9ae00640 1 -- 172.21.15.28:0/4126872931 wait complete. 2024-05-12T22:14:07.450 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-12T22:14:07.451 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-12T22:14:07.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.696+0000 7f078ec00640 1 -- 172.21.15.28:0/2384113773 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f07881010c0 msgr2=0x7f078810d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:14:07.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.696+0000 7f078ec00640 1 --2- 172.21.15.28:0/2384113773 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f07881010c0 0x7f078810d5d0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f0780009c10 tx=0x7f078002a6f0 comp rx=0 tx=0).stop 2024-05-12T22:14:07.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.696+0000 7f078ec00640 1 -- 172.21.15.28:0/2384113773 shutdown_connections 2024-05-12T22:14:07.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.696+0000 7f078ec00640 1 --2- 172.21.15.28:0/2384113773 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f07881010c0 0x7f078810d5d0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:07.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.696+0000 7f078ec00640 1 --2- 172.21.15.28:0/2384113773 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0788100700 0x7f0788100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:07.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.696+0000 7f078ec00640 1 --2- 172.21.15.28:0/2384113773 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0788106710 0x7f0788106af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:07.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.696+0000 7f078ec00640 1 -- 172.21.15.28:0/2384113773 >> 172.21.15.28:0/2384113773 conn(0x7f0788077890 msgr2=0x7f0788077ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:14:07.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.697+0000 7f078ec00640 1 -- 172.21.15.28:0/2384113773 shutdown_connections 2024-05-12T22:14:07.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.697+0000 7f078ec00640 1 -- 172.21.15.28:0/2384113773 wait complete. 2024-05-12T22:14:07.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.698+0000 7f078ec00640 1 Processor -- start 2024-05-12T22:14:07.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.698+0000 7f078ec00640 1 -- start start 2024-05-12T22:14:07.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.698+0000 7f078ec00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0788100700 0x7f078819a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:07.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.698+0000 7f078ec00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f07881010c0 0x7f078819a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:07.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.699+0000 7f078ec00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0788106710 0x7f078819ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:07.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.699+0000 7f078ec00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0788111c20 con 0x7f0788106710 2024-05-12T22:14:07.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.699+0000 7f078ec00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f0788111aa0 con 0x7f07881010c0 2024-05-12T22:14:07.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.699+0000 7f078ec00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f0788111da0 con 0x7f0788100700 2024-05-12T22:14:07.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.699+0000 7f0787e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0788100700 0x7f078819a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:07.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.699+0000 7f0787e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0788100700 0x7f078819a320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:33804/0 (socket says 172.21.15.28:33804) 2024-05-12T22:14:07.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.699+0000 7f0787e00640 1 -- 172.21.15.28:0/88536656 learned_addr learned my addr 172.21.15.28:0/88536656 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:14:07.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.699+0000 7f0787400640 1 --2- 172.21.15.28:0/88536656 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f07881010c0 0x7f078819a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:07.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.699+0000 7f078cc00640 1 --2- 172.21.15.28:0/88536656 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0788106710 0x7f078819ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:07.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.699+0000 7f0787e00640 1 -- 172.21.15.28:0/88536656 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f07881010c0 msgr2=0x7f078819a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:14:07.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.699+0000 7f0787e00640 1 --2- 172.21.15.28:0/88536656 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f07881010c0 0x7f078819a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:07.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.699+0000 7f0787e00640 1 -- 172.21.15.28:0/88536656 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0788106710 msgr2=0x7f078819ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:14:07.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.699+0000 7f0787e00640 1 --2- 172.21.15.28:0/88536656 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0788106710 0x7f078819ebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:07.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.699+0000 7f0787e00640 1 -- 172.21.15.28:0/88536656 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f078819f310 con 0x7f0788100700 2024-05-12T22:14:07.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.700+0000 7f0787e00640 1 --2- 172.21.15.28:0/88536656 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0788100700 0x7f078819a320 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f077800e9f0 tx=0x7f077800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:14:07.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.700+0000 7f0784c00640 1 -- 172.21.15.28:0/88536656 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f077800ccc0 con 0x7f0788100700 2024-05-12T22:14:07.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.700+0000 7f0784c00640 1 -- 172.21.15.28:0/88536656 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0778005500 con 0x7f0788100700 2024-05-12T22:14:07.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.700+0000 7f078ec00640 1 -- 172.21.15.28:0/88536656 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f078819f600 con 0x7f0788100700 2024-05-12T22:14:07.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.700+0000 7f0784c00640 1 -- 172.21.15.28:0/88536656 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0778010690 con 0x7f0788100700 2024-05-12T22:14:07.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.700+0000 7f078ec00640 1 -- 172.21.15.28:0/88536656 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07881a6e90 con 0x7f0788100700 2024-05-12T22:14:07.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.704+0000 7f0784c00640 1 -- 172.21.15.28:0/88536656 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7f07780046c0 con 0x7f0788100700 2024-05-12T22:14:07.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.704+0000 7f07919ff740 1 Processor -- start 2024-05-12T22:14:07.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.705+0000 7f07919ff740 1 -- start start 2024-05-12T22:14:07.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.705+0000 7f07919ff740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b3ac16cbc0 0x55b3ac16cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:07.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.705+0000 7f07919ff740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b3ac16d4e0 0x55b3ac177670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:07.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.705+0000 7f07919ff740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b3ac177bb0 0x55b3ac179fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:07.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.705+0000 7f07919ff740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b3ac1352a0 con 0x55b3ac16d4e0 2024-05-12T22:14:07.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.705+0000 7f07919ff740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x55b3ac0d1b90 con 0x55b3ac16cbc0 2024-05-12T22:14:07.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.705+0000 7f07919ff740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55b3ac0795b0 con 0x55b3ac177bb0 2024-05-12T22:14:07.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.705+0000 7f0787e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b3ac16d4e0 0x55b3ac177670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:07.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.705+0000 7f078cc00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b3ac16cbc0 0x55b3ac16cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:07.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.705+0000 7f0787400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b3ac177bb0 0x55b3ac179fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:07.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.705+0000 7f078cc00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b3ac16cbc0 0x55b3ac16cfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:56070/0 (socket says 172.21.15.28:56070) 2024-05-12T22:14:07.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.705+0000 7f078cc00640 1 -- 172.21.15.28:0/3290378807 learned_addr learned my addr 172.21.15.28:0/3290378807 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:14:07.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.706+0000 7f0787400640 1 -- 172.21.15.28:0/3290378807 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b3ac16cbc0 msgr2=0x55b3ac16cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:14:07.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.706+0000 7f0787400640 1 --2- 172.21.15.28:0/3290378807 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b3ac16cbc0 0x55b3ac16cfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:07.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.706+0000 7f0787400640 1 -- 172.21.15.28:0/3290378807 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b3ac16d4e0 msgr2=0x55b3ac177670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:14:07.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.706+0000 7f0787400640 1 --2- 172.21.15.28:0/3290378807 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b3ac16d4e0 0x55b3ac177670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:07.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.706+0000 7f0787400640 1 -- 172.21.15.28:0/3290378807 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b3abeaf7d0 con 0x55b3ac177bb0 2024-05-12T22:14:07.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.706+0000 7f0787400640 1 --2- 172.21.15.28:0/3290378807 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b3ac177bb0 0x55b3ac179fa0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f0774009bf0 tx=0x7f0774011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c3635c93d31b960b server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:14:07.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.706+0000 7f0784c00640 1 --2- 172.21.15.28:0/88536656 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f076c0a2d80 0x7f076c0a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:07.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.707+0000 7f077d200640 1 -- 172.21.15.28:0/3290378807 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0774004050 con 0x55b3ac177bb0 2024-05-12T22:14:07.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.707+0000 7f0784c00640 1 -- 172.21.15.28:0/88536656 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f07780cf590 con 0x7f0788100700 2024-05-12T22:14:07.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.707+0000 7f077d200640 1 -- 172.21.15.28:0/3290378807 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0774005500 con 0x55b3ac177bb0 2024-05-12T22:14:07.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.707+0000 7f077d200640 1 -- 172.21.15.28:0/3290378807 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0774004e40 con 0x55b3ac177bb0 2024-05-12T22:14:07.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.707+0000 7f078cc00640 1 --2- 172.21.15.28:0/88536656 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f076c0a2d80 0x7f076c0a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:07.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.708+0000 7f07919ff740 1 -- 172.21.15.28:0/3290378807 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b3ac177bb0 msgr2=0x55b3ac179fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:14:07.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.708+0000 7f07919ff740 1 --2- 172.21.15.28:0/3290378807 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b3ac177bb0 0x55b3ac179fa0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f0774009bf0 tx=0x7f0774011810 comp rx=0 tx=0).stop 2024-05-12T22:14:07.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.708+0000 7f078cc00640 1 --2- 172.21.15.28:0/88536656 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f076c0a2d80 0x7f076c0a5240 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f07800099a0 tx=0x7f0780005fd0 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:14:07.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.708+0000 7f07919ff740 1 -- 172.21.15.28:0/3290378807 shutdown_connections 2024-05-12T22:14:07.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.708+0000 7f07919ff740 1 --2- 172.21.15.28:0/3290378807 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b3ac177bb0 0x55b3ac179fa0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:07.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.708+0000 7f07919ff740 1 --2- 172.21.15.28:0/3290378807 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b3ac16d4e0 0x55b3ac177670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:07.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.708+0000 7f07919ff740 1 --2- 172.21.15.28:0/3290378807 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b3ac16cbc0 0x55b3ac16cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:07.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.708+0000 7f07919ff740 1 -- 172.21.15.28:0/3290378807 >> 172.21.15.28:0/3290378807 conn(0x55b3ac168ee0 msgr2=0x55b3ac1692c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:14:07.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.708+0000 7f07919ff740 1 -- 172.21.15.28:0/3290378807 shutdown_connections 2024-05-12T22:14:07.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.708+0000 7f07919ff740 1 -- 172.21.15.28:0/3290378807 wait complete. 2024-05-12T22:14:07.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.709+0000 7f07919ff740 1 -- start start 2024-05-12T22:14:07.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.709+0000 7f07919ff740 1 Processor -- start 2024-05-12T22:14:07.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.710+0000 7f07919ff740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b3ac16cbc0 0x55b3ac177520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:07.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.710+0000 7f07919ff740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b3ac16d4e0 0x55b3ac174810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:07.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.710+0000 7f07919ff740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b3ac177bb0 0x55b3ac174d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:07.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.710+0000 7f07919ff740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b3ac08c9f0 con 0x55b3ac16d4e0 2024-05-12T22:14:07.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.710+0000 7f07919ff740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x55b3ac0cdb50 con 0x55b3ac177bb0 2024-05-12T22:14:07.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.710+0000 7f07919ff740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55b3ac17a9a0 con 0x55b3ac16cbc0 2024-05-12T22:14:07.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.710+0000 7f07919ff740 10 client.0 _ll_register_callbacks cb 0x55b3abde4be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:14:07.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.710+0000 7f0787e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b3ac16cbc0 0x55b3ac177520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:07.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.710+0000 7f0787e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b3ac16cbc0 0x55b3ac177520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:33816/0 (socket says 172.21.15.28:33816) 2024-05-12T22:14:07.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.710+0000 7f0787e00640 1 -- 172.21.15.28:0/682270162 learned_addr learned my addr 172.21.15.28:0/682270162 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:14:07.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.710+0000 7f078cc00640 1 --2- 172.21.15.28:0/682270162 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b3ac177bb0 0x55b3ac174d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:07.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.710+0000 7f0787400640 1 --2- 172.21.15.28:0/682270162 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b3ac16d4e0 0x55b3ac174810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:07.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.710+0000 7f0787e00640 1 -- 172.21.15.28:0/682270162 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b3ac177bb0 msgr2=0x55b3ac174d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:14:07.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.711+0000 7f0787e00640 1 --2- 172.21.15.28:0/682270162 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b3ac177bb0 0x55b3ac174d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:07.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.711+0000 7f0787e00640 1 -- 172.21.15.28:0/682270162 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b3ac16d4e0 msgr2=0x55b3ac174810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:14:07.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.711+0000 7f0787e00640 1 --2- 172.21.15.28:0/682270162 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b3ac16d4e0 0x55b3ac174810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:07.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.711+0000 7f0787e00640 1 -- 172.21.15.28:0/682270162 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b3abde53f0 con 0x55b3ac16cbc0 2024-05-12T22:14:07.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.711+0000 7f0787400640 1 --2- 172.21.15.28:0/682270162 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b3ac16d4e0 0x55b3ac174810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-12T22:14:07.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.711+0000 7f0787e00640 1 --2- 172.21.15.28:0/682270162 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b3ac16cbc0 0x55b3ac177520 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f077800e9c0 tx=0x7f077800cf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=adb800935f7e6485 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:14:07.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.711+0000 7f0771600640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-12T22:14:07.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.711+0000 7f0771600640 1 -- 172.21.15.28:0/682270162 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f077809b440 con 0x55b3ac16cbc0 2024-05-12T22:14:07.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.711+0000 7f0771600640 1 -- 172.21.15.28:0/682270162 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f07780a44c0 con 0x55b3ac16cbc0 2024-05-12T22:14:07.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.711+0000 7f07919ff740 1 -- 172.21.15.28:0/682270162 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0774005500 con 0x55b3ac16cbc0 2024-05-12T22:14:07.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.712+0000 7f0771600640 1 -- 172.21.15.28:0/682270162 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f07780056b0 con 0x55b3ac16cbc0 2024-05-12T22:14:07.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.712+0000 7f07919ff740 10 client.26416 fetch_fsmap learned FSMap version 26 2024-05-12T22:14:07.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.712+0000 7f07919ff740 1 -- 172.21.15.28:0/682270162 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55b3abda50f0 con 0x55b3ac16cbc0 2024-05-12T22:14:07.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.712+0000 7f0771600640 1 -- 172.21.15.28:0/682270162 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0774005500 con 0x55b3ac16cbc0 2024-05-12T22:14:07.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.712+0000 7f0771600640 1 -- 172.21.15.28:0/682270162 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f07780b4ce0 con 0x55b3ac16cbc0 2024-05-12T22:14:07.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.712+0000 7f07919ff740 10 client.26416 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:14:07.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.712+0000 7f07919ff740 10 client.26416 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:14:07.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.712+0000 7f07919ff740 20 client.26416 populate_metadata read hostname 'smithi028' 2024-05-12T22:14:07.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.712+0000 7f07919ff740 1 --2- 172.21.15.28:0/682270162 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55b3ac1877a0 0x55b3ac189bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:07.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.712+0000 7f07919ff740 4 client.26416 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-12T22:14:07.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.712+0000 7f07919ff740 1 -- 172.21.15.28:0/682270162 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b3ac0df2c0 con 0x55b3ac1877a0 2024-05-12T22:14:07.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.713+0000 7f0787400640 1 --2- 172.21.15.28:0/682270162 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55b3ac1877a0 0x55b3ac189bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:07.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.713+0000 7f0787400640 1 --2- 172.21.15.28:0/682270162 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55b3ac1877a0 0x55b3ac189bb0 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=62ef364781edf4e5 server_cookie=94a844fb13f2c595 in_seq=0 out_seq=0 2024-05-12T22:14:07.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.713+0000 7f0771600640 10 client.26416 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:14:07.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.714+0000 7f0771600640 1 -- 172.21.15.28:0/682270162 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b3ac0df2c0 con 0x55b3ac1877a0 2024-05-12T22:14:07.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.714+0000 7f0771600640 10 client.26416 handle_command_reply: tid=0 2024-05-12T22:14:07.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.715+0000 7f07919ff740 1 client.26416 shutdown 2024-05-12T22:14:07.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.715+0000 7f07919ff740 1 -- 172.21.15.28:0/682270162 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b3ac16cbc0 msgr2=0x55b3ac177520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:14:07.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.715+0000 7f07919ff740 1 --2- 172.21.15.28:0/682270162 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b3ac16cbc0 0x55b3ac177520 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f077800e9c0 tx=0x7f077800cf60 comp rx=0 tx=0).stop 2024-05-12T22:14:07.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.715+0000 7f07919ff740 1 -- 172.21.15.28:0/682270162 shutdown_connections 2024-05-12T22:14:07.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.715+0000 7f07919ff740 1 --2- 172.21.15.28:0/682270162 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55b3ac1877a0 0x55b3ac189bb0 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:07.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.715+0000 7f07919ff740 1 --2- 172.21.15.28:0/682270162 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b3ac177bb0 0x55b3ac174d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:07.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.715+0000 7f07919ff740 1 --2- 172.21.15.28:0/682270162 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b3ac16d4e0 0x55b3ac174810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:07.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.715+0000 7f07919ff740 1 --2- 172.21.15.28:0/682270162 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b3ac16cbc0 0x55b3ac177520 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:07.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.715+0000 7f0771600640 0 client.26416 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:14:07.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.715+0000 7f07919ff740 1 -- 172.21.15.28:0/682270162 >> 172.21.15.28:0/682270162 conn(0x55b3ac168ee0 msgr2=0x55b3ac16ece0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:14:07.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.715+0000 7f07919ff740 1 -- 172.21.15.28:0/682270162 shutdown_connections 2024-05-12T22:14:07.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.716+0000 7f07919ff740 1 -- 172.21.15.28:0/682270162 wait complete. 2024-05-12T22:14:07.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.716+0000 7f07919ff740 20 client.26416 trim_cache size 0 max 16384 2024-05-12T22:14:07.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.740+0000 7f07919ff740 1 Processor -- start 2024-05-12T22:14:07.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.740+0000 7f07919ff740 1 -- start start 2024-05-12T22:14:07.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.740+0000 7f07919ff740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b3ac16cbc0 0x55b3ac189b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:07.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.741+0000 7f07919ff740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b3ac16d4e0 0x55b3ac1836e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:07.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.741+0000 7f07919ff740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b3ac177bb0 0x55b3ac18f480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:07.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.741+0000 7f07919ff740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f07881b2060 con 0x55b3ac177bb0 2024-05-12T22:14:07.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.741+0000 7f07919ff740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f07881af930 con 0x55b3ac16cbc0 2024-05-12T22:14:07.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.741+0000 7f07919ff740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f07881ad780 con 0x55b3ac16d4e0 2024-05-12T22:14:07.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.741+0000 7f0787e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b3ac16cbc0 0x55b3ac189b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:07.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.741+0000 7f0787e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b3ac16cbc0 0x55b3ac189b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:56084/0 (socket says 172.21.15.28:56084) 2024-05-12T22:14:07.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.741+0000 7f0787e00640 1 -- 172.21.15.28:0/1630366002 learned_addr learned my addr 172.21.15.28:0/1630366002 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:14:07.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.741+0000 7f0787400640 1 --2- 172.21.15.28:0/1630366002 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b3ac16d4e0 0x55b3ac1836e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:07.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.741+0000 7f078cc00640 1 --2- 172.21.15.28:0/1630366002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b3ac177bb0 0x55b3ac18f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:07.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.741+0000 7f0787400640 1 -- 172.21.15.28:0/1630366002 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b3ac16cbc0 msgr2=0x55b3ac189b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:14:07.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.741+0000 7f0787400640 1 --2- 172.21.15.28:0/1630366002 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b3ac16cbc0 0x55b3ac189b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:07.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.741+0000 7f0787400640 1 -- 172.21.15.28:0/1630366002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b3ac177bb0 msgr2=0x55b3ac18f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:14:07.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.741+0000 7f0787400640 1 --2- 172.21.15.28:0/1630366002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b3ac177bb0 0x55b3ac18f480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:07.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.741+0000 7f0787400640 1 -- 172.21.15.28:0/1630366002 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b3ac163f70 con 0x55b3ac16d4e0 2024-05-12T22:14:07.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.742+0000 7f078cc00640 1 --2- 172.21.15.28:0/1630366002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b3ac177bb0 0x55b3ac18f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:14:07.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.742+0000 7f0787400640 1 --2- 172.21.15.28:0/1630366002 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b3ac16d4e0 0x55b3ac1836e0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x55b3ac174070 tx=0x7f07740097c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3d61bced41eee76a server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:14:07.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.742+0000 7f0773e00640 1 -- 172.21.15.28:0/1630366002 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0774009da0 con 0x55b3ac16d4e0 2024-05-12T22:14:07.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.742+0000 7f0773e00640 1 -- 172.21.15.28:0/1630366002 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0774004530 con 0x55b3ac16d4e0 2024-05-12T22:14:07.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.742+0000 7f0773e00640 1 -- 172.21.15.28:0/1630366002 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f077402a510 con 0x55b3ac16d4e0 2024-05-12T22:14:07.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.743+0000 7f07919ff740 1 -- 172.21.15.28:0/1630366002 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b3ac16d4e0 msgr2=0x55b3ac1836e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:14:07.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.743+0000 7f07919ff740 1 --2- 172.21.15.28:0/1630366002 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b3ac16d4e0 0x55b3ac1836e0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x55b3ac174070 tx=0x7f07740097c0 comp rx=0 tx=0).stop 2024-05-12T22:14:07.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.743+0000 7f07919ff740 1 -- 172.21.15.28:0/1630366002 shutdown_connections 2024-05-12T22:14:07.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.743+0000 7f07919ff740 1 --2- 172.21.15.28:0/1630366002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b3ac177bb0 0x55b3ac18f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:07.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.743+0000 7f07919ff740 1 --2- 172.21.15.28:0/1630366002 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b3ac16d4e0 0x55b3ac1836e0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:07.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.743+0000 7f07919ff740 1 --2- 172.21.15.28:0/1630366002 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b3ac16cbc0 0x55b3ac189b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:07.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.743+0000 7f07919ff740 1 -- 172.21.15.28:0/1630366002 >> 172.21.15.28:0/1630366002 conn(0x55b3ac168ee0 msgr2=0x55b3ac1680e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:14:07.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.743+0000 7f07919ff740 1 -- 172.21.15.28:0/1630366002 shutdown_connections 2024-05-12T22:14:07.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.744+0000 7f07919ff740 1 -- 172.21.15.28:0/1630366002 wait complete. 2024-05-12T22:14:07.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.744+0000 7f07919ff740 1 -- start start 2024-05-12T22:14:07.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.744+0000 7f07919ff740 1 Processor -- start 2024-05-12T22:14:07.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.745+0000 7f07919ff740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b3ac16cbc0 0x55b3ac17d990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:07.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.745+0000 7f07919ff740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b3ac16d4e0 0x55b3ac17ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:07.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.745+0000 7f07919ff740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b3ac177bb0 0x55b3ac1ecd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:07.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.745+0000 7f07919ff740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f07881addf0 con 0x55b3ac16d4e0 2024-05-12T22:14:07.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.745+0000 7f07919ff740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f07881af260 con 0x55b3ac177bb0 2024-05-12T22:14:07.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.745+0000 7f07919ff740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f07881ae4c0 con 0x55b3ac16cbc0 2024-05-12T22:14:07.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.745+0000 7f07919ff740 10 client.0 _ll_register_callbacks cb 0x7f077800ce60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:14:07.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.745+0000 7f0787e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b3ac16cbc0 0x55b3ac17d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:07.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.745+0000 7f0787e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b3ac16cbc0 0x55b3ac17d990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:33840/0 (socket says 172.21.15.28:33840) 2024-05-12T22:14:07.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.745+0000 7f0787e00640 1 -- 172.21.15.28:0/3282623718 learned_addr learned my addr 172.21.15.28:0/3282623718 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:14:07.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.745+0000 7f0787400640 1 --2- 172.21.15.28:0/3282623718 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b3ac16d4e0 0x55b3ac17ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:07.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.745+0000 7f078cc00640 1 --2- 172.21.15.28:0/3282623718 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b3ac177bb0 0x55b3ac1ecd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:07.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.746+0000 7f0787e00640 1 -- 172.21.15.28:0/3282623718 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b3ac177bb0 msgr2=0x55b3ac1ecd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:14:07.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.746+0000 7f0787e00640 1 --2- 172.21.15.28:0/3282623718 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b3ac177bb0 0x55b3ac1ecd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:07.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.746+0000 7f0787e00640 1 -- 172.21.15.28:0/3282623718 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b3ac16d4e0 msgr2=0x55b3ac17ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:14:07.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.746+0000 7f0787e00640 1 --2- 172.21.15.28:0/3282623718 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b3ac16d4e0 0x55b3ac17ded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:07.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.746+0000 7f0787e00640 1 -- 172.21.15.28:0/3282623718 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b3ac17e7a0 con 0x55b3ac16cbc0 2024-05-12T22:14:07.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.746+0000 7f0787400640 1 --2- 172.21.15.28:0/3282623718 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b3ac16d4e0 0x55b3ac17ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:14:07.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.747+0000 7f0787e00640 1 --2- 172.21.15.28:0/3282623718 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b3ac16cbc0 0x55b3ac17d990 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f0778009230 tx=0x7f077800c840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=330f2b33857e8f0f server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:14:07.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.747+0000 7f0771600640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-12T22:14:07.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.747+0000 7f0771600640 1 -- 172.21.15.28:0/3282623718 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07780b9080 con 0x55b3ac16cbc0 2024-05-12T22:14:07.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.747+0000 7f0771600640 1 -- 172.21.15.28:0/3282623718 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f077809af30 con 0x55b3ac16cbc0 2024-05-12T22:14:07.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.747+0000 7f07919ff740 1 -- 172.21.15.28:0/3282623718 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0774004530 con 0x55b3ac16cbc0 2024-05-12T22:14:07.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.747+0000 7f0771600640 1 -- 172.21.15.28:0/3282623718 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f07780aa050 con 0x55b3ac16cbc0 2024-05-12T22:14:07.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.747+0000 7f07919ff740 10 client.26422 fetch_fsmap learned FSMap version 26 2024-05-12T22:14:07.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.747+0000 7f07919ff740 1 -- 172.21.15.28:0/3282623718 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55b3ac1f6b60 con 0x55b3ac16cbc0 2024-05-12T22:14:07.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.748+0000 7f0771600640 1 -- 172.21.15.28:0/3282623718 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0774004530 con 0x55b3ac16cbc0 2024-05-12T22:14:07.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.748+0000 7f0771600640 1 -- 172.21.15.28:0/3282623718 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f07780b4ce0 con 0x55b3ac16cbc0 2024-05-12T22:14:07.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.748+0000 7f07919ff740 10 client.26422 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:14:07.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.748+0000 7f07919ff740 10 client.26422 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:14:07.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.748+0000 7f07919ff740 20 client.26422 populate_metadata read hostname 'smithi028' 2024-05-12T22:14:07.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.748+0000 7f07919ff740 1 --2- 172.21.15.28:0/3282623718 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55b3ac1f7170 0x55b3ac1f9590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:07.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.748+0000 7f07919ff740 4 client.26422 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-12T22:14:07.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.748+0000 7f07919ff740 1 -- 172.21.15.28:0/3282623718 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55b3ac0d9610 con 0x55b3ac1f7170 2024-05-12T22:14:07.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.748+0000 7f0787400640 1 --2- 172.21.15.28:0/3282623718 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55b3ac1f7170 0x55b3ac1f9590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:07.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.749+0000 7f0787400640 1 --2- 172.21.15.28:0/3282623718 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55b3ac1f7170 0x55b3ac1f9590 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5fa8467a42b677a1 server_cookie=1b36396f1d11064c in_seq=0 out_seq=0 2024-05-12T22:14:07.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.749+0000 7f0771600640 10 client.26422 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:14:07.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.749+0000 7f0771600640 1 -- 172.21.15.28:0/3282623718 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+268 (crc 0 0 0) 0x55b3ac0d9610 con 0x55b3ac1f7170 2024-05-12T22:14:07.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.749+0000 7f0771600640 10 client.26422 handle_command_reply: tid=0 2024-05-12T22:14:07.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.749+0000 7f07919ff740 1 client.26422 shutdown 2024-05-12T22:14:07.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.750+0000 7f07919ff740 1 -- 172.21.15.28:0/3282623718 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b3ac16cbc0 msgr2=0x55b3ac17d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:14:07.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.750+0000 7f07919ff740 1 --2- 172.21.15.28:0/3282623718 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b3ac16cbc0 0x55b3ac17d990 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f0778009230 tx=0x7f077800c840 comp rx=0 tx=0).stop 2024-05-12T22:14:07.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.750+0000 7f07919ff740 1 -- 172.21.15.28:0/3282623718 shutdown_connections 2024-05-12T22:14:07.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.750+0000 7f07919ff740 1 --2- 172.21.15.28:0/3282623718 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55b3ac1f7170 0x55b3ac1f9590 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:07.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.750+0000 7f07919ff740 1 --2- 172.21.15.28:0/3282623718 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b3ac177bb0 0x55b3ac1ecd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:07.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.750+0000 7f07919ff740 1 --2- 172.21.15.28:0/3282623718 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b3ac16d4e0 0x55b3ac17ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:07.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.750+0000 7f07919ff740 1 --2- 172.21.15.28:0/3282623718 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b3ac16cbc0 0x55b3ac17d990 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:07.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.750+0000 7f07919ff740 1 -- 172.21.15.28:0/3282623718 >> 172.21.15.28:0/3282623718 conn(0x55b3ac168ee0 msgr2=0x55b3ac176040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:14:07.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.750+0000 7f0771600640 0 client.26422 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:14:07.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.750+0000 7f07919ff740 1 -- 172.21.15.28:0/3282623718 shutdown_connections 2024-05-12T22:14:07.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.750+0000 7f07919ff740 1 -- 172.21.15.28:0/3282623718 wait complete. 2024-05-12T22:14:07.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.750+0000 7f07919ff740 20 client.26422 trim_cache size 0 max 16384 2024-05-12T22:14:07.756 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-05-12T22:14:07.757 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "scrub active (1084 inodes in the stack)", 2024-05-12T22:14:07.757 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-05-12T22:14:07.757 INFO:teuthology.orchestra.run.smithi028.stdout: "2860a46a-25c9-4783-b88a-954434568cec": { 2024-05-12T22:14:07.757 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-05-12T22:14:07.757 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "2860a46a-25c9-4783-b88a-954434568cec", 2024-05-12T22:14:07.757 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-05-12T22:14:07.757 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:14:07.757 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:14:07.758 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-05-12T22:14:07.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.751+0000 7f078ec00640 1 -- 172.21.15.28:0/88536656 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f076c0a2d80 msgr2=0x7f076c0a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:14:07.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.751+0000 7f078ec00640 1 --2- 172.21.15.28:0/88536656 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f076c0a2d80 0x7f076c0a5240 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f07800099a0 tx=0x7f0780005fd0 comp rx=0 tx=0).stop 2024-05-12T22:14:07.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.751+0000 7f078ec00640 1 -- 172.21.15.28:0/88536656 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0788100700 msgr2=0x7f078819a320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:14:07.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.751+0000 7f078ec00640 1 --2- 172.21.15.28:0/88536656 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0788100700 0x7f078819a320 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f077800e9f0 tx=0x7f077800eeb0 comp rx=0 tx=0).stop 2024-05-12T22:14:07.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.751+0000 7f078ec00640 1 -- 172.21.15.28:0/88536656 shutdown_connections 2024-05-12T22:14:07.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.751+0000 7f078ec00640 1 --2- 172.21.15.28:0/88536656 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f076c0a2d80 0x7f076c0a5240 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:07.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.751+0000 7f078ec00640 1 --2- 172.21.15.28:0/88536656 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0788106710 0x7f078819ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:07.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.751+0000 7f078ec00640 1 --2- 172.21.15.28:0/88536656 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f07881010c0 0x7f078819a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:07.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.751+0000 7f078ec00640 1 --2- 172.21.15.28:0/88536656 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0788100700 0x7f078819a320 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:07.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.751+0000 7f078ec00640 1 -- 172.21.15.28:0/88536656 >> 172.21.15.28:0/88536656 conn(0x7f0788077890 msgr2=0x7f078810b970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:14:07.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.751+0000 7f078ec00640 1 -- 172.21.15.28:0/88536656 shutdown_connections 2024-05-12T22:14:07.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:07.751+0000 7f078ec00640 1 -- 172.21.15.28:0/88536656 wait complete. 2024-05-12T22:14:07.773 INFO:tasks.cephfs.filesystem:scrub status for tag:2860a46a-25c9-4783-b88a-954434568cec - {'path': '/', 'tag': '2860a46a-25c9-4783-b88a-954434568cec', 'options': 'recursive,force'} 2024-05-12T22:14:08.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:14:07 smithi086 ceph-mon[34297]: pgmap v157: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-05-12T22:14:08.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:14:07 smithi086 ceph-mon[34297]: scrub queued for path: / 2024-05-12T22:14:08.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:14:07 smithi086 ceph-mon[34297]: scrub summary: idle+waiting paths [/] 2024-05-12T22:14:08.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:14:07 smithi086 ceph-mon[34297]: scrub summary: active paths [/] 2024-05-12T22:14:08.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:14:07 smithi086 ceph-mon[34297]: scrub summary: active paths [/] 2024-05-12T22:14:08.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:14:07 smithi086 ceph-mon[34297]: scrub summary: idle+waiting paths [/] 2024-05-12T22:14:08.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:14:07 smithi086 ceph-mon[34297]: scrub summary: active paths [/] 2024-05-12T22:14:08.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:14:07 smithi086 ceph-mon[34297]: scrub summary: idle+waiting paths [/] 2024-05-12T22:14:08.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:14:07 smithi086 ceph-mon[34297]: scrub summary: active paths [/] 2024-05-12T22:14:08.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:14:07 smithi086 ceph-mon[34297]: scrub summary: idle+waiting paths [/] 2024-05-12T22:14:08.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:14:07 smithi028 ceph-mon[29847]: pgmap v157: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-05-12T22:14:08.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:14:07 smithi028 ceph-mon[29847]: scrub queued for path: / 2024-05-12T22:14:08.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:14:07 smithi028 ceph-mon[29847]: scrub summary: idle+waiting paths [/] 2024-05-12T22:14:08.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:14:07 smithi028 ceph-mon[29847]: scrub summary: active paths [/] 2024-05-12T22:14:08.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:14:07 smithi028 ceph-mon[29847]: scrub summary: active paths [/] 2024-05-12T22:14:08.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:14:07 smithi028 ceph-mon[29847]: scrub summary: idle+waiting paths [/] 2024-05-12T22:14:08.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:14:07 smithi028 ceph-mon[29847]: scrub summary: active paths [/] 2024-05-12T22:14:08.287 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:14:07 smithi028 ceph-mon[29847]: scrub summary: idle+waiting paths [/] 2024-05-12T22:14:08.287 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:14:07 smithi028 ceph-mon[29847]: scrub summary: active paths [/] 2024-05-12T22:14:08.287 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:14:07 smithi028 ceph-mon[29847]: scrub summary: idle+waiting paths [/] 2024-05-12T22:14:08.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:14:07 smithi186 ceph-mon[34785]: pgmap v157: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-05-12T22:14:08.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:14:07 smithi186 ceph-mon[34785]: scrub queued for path: / 2024-05-12T22:14:08.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:14:07 smithi186 ceph-mon[34785]: scrub summary: idle+waiting paths [/] 2024-05-12T22:14:08.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:14:07 smithi186 ceph-mon[34785]: scrub summary: active paths [/] 2024-05-12T22:14:08.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:14:07 smithi186 ceph-mon[34785]: scrub summary: active paths [/] 2024-05-12T22:14:08.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:14:07 smithi186 ceph-mon[34785]: scrub summary: idle+waiting paths [/] 2024-05-12T22:14:08.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:14:07 smithi186 ceph-mon[34785]: scrub summary: active paths [/] 2024-05-12T22:14:08.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:14:07 smithi186 ceph-mon[34785]: scrub summary: idle+waiting paths [/] 2024-05-12T22:14:08.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:14:07 smithi186 ceph-mon[34785]: scrub summary: active paths [/] 2024-05-12T22:14:08.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:14:07 smithi186 ceph-mon[34785]: scrub summary: idle+waiting paths [/] 2024-05-12T22:14:09.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:14:08 smithi086 ceph-mon[34297]: scrub summary: idle+waiting paths [/] 2024-05-12T22:14:09.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:14:08 smithi028 ceph-mon[29847]: scrub summary: idle+waiting paths [/] 2024-05-12T22:14:09.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:14:08 smithi186 ceph-mon[34785]: scrub summary: idle+waiting paths [/] 2024-05-12T22:14:10.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:14:09 smithi086 ceph-mon[34297]: pgmap v158: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-05-12T22:14:10.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:14:09 smithi028 ceph-mon[29847]: pgmap v158: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-05-12T22:14:10.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:14:09 smithi186 ceph-mon[34785]: pgmap v158: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-05-12T22:14:12.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:14:11 smithi028 ceph-mon[29847]: pgmap v159: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-05-12T22:14:12.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:14:11 smithi186 ceph-mon[34785]: pgmap v159: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-05-12T22:14:12.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:14:11 smithi086 ceph-mon[34297]: pgmap v159: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-05-12T22:14:14.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:14:13 smithi028 ceph-mon[29847]: pgmap v160: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-05-12T22:14:14.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:14:13 smithi186 ceph-mon[34785]: pgmap v160: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-05-12T22:14:14.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:14:13 smithi086 ceph-mon[34297]: pgmap v160: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-05-12T22:14:15.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:14:15 smithi028 ceph-mon[29847]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:14:15.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:14:15 smithi186 ceph-mon[34785]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:14:15.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:14:15 smithi086 ceph-mon[34297]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:14:16.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:14:16 smithi028 ceph-mon[29847]: pgmap v161: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-05-12T22:14:16.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:14:16 smithi186 ceph-mon[34785]: pgmap v161: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-05-12T22:14:16.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:14:16 smithi086 ceph-mon[34297]: pgmap v161: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-05-12T22:14:17.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:14:17 smithi028 ceph-mon[29847]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:14:17.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:14:17 smithi028 ceph-mon[29847]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:14:17.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:14:17 smithi028 ceph-mon[29847]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:14:17.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:14:17 smithi186 ceph-mon[34785]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:14:17.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:14:17 smithi186 ceph-mon[34785]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:14:17.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:14:17 smithi186 ceph-mon[34785]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:14:17.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:14:17 smithi086 ceph-mon[34297]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:14:17.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:14:17 smithi086 ceph-mon[34297]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:14:17.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:14:17 smithi086 ceph-mon[34297]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:14:18.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:14:18 smithi086 ceph-mon[34297]: pgmap v162: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 85 B/s wr, 5 op/s 2024-05-12T22:14:18.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:14:18 smithi086 ceph-mon[34297]: scrub summary: idle 2024-05-12T22:14:18.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:14:18 smithi086 ceph-mon[34297]: scrub summary: idle 2024-05-12T22:14:18.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:14:18 smithi028 ceph-mon[29847]: pgmap v162: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 85 B/s wr, 5 op/s 2024-05-12T22:14:18.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:14:18 smithi028 ceph-mon[29847]: scrub summary: idle 2024-05-12T22:14:18.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:14:18 smithi028 ceph-mon[29847]: scrub summary: idle 2024-05-12T22:14:18.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:14:18 smithi186 ceph-mon[34785]: pgmap v162: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 85 B/s wr, 5 op/s 2024-05-12T22:14:18.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:14:18 smithi186 ceph-mon[34785]: scrub summary: idle 2024-05-12T22:14:18.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:14:18 smithi186 ceph-mon[34785]: scrub summary: idle 2024-05-12T22:14:20.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:14:20 smithi086 ceph-mon[34297]: pgmap v163: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 85 B/s wr, 5 op/s 2024-05-12T22:14:20.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:14:20 smithi028 ceph-mon[29847]: pgmap v163: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 85 B/s wr, 5 op/s 2024-05-12T22:14:20.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:14:20 smithi186 ceph-mon[34785]: pgmap v163: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 85 B/s wr, 5 op/s 2024-05-12T22:14:22.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:14:22 smithi086 ceph-mon[34297]: pgmap v164: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 85 B/s wr, 7 op/s 2024-05-12T22:14:22.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:14:22 smithi028 ceph-mon[29847]: pgmap v164: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 85 B/s wr, 7 op/s 2024-05-12T22:14:22.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:14:22 smithi186 ceph-mon[34785]: pgmap v164: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 85 B/s wr, 7 op/s 2024-05-12T22:14:24.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:14:24 smithi086 ceph-mon[34297]: pgmap v165: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:14:24.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:14:24 smithi028 ceph-mon[29847]: pgmap v165: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:14:24.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:14:24 smithi186 ceph-mon[34785]: pgmap v165: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:14:26.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:14:26 smithi086 ceph-mon[34297]: pgmap v166: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-12T22:14:26.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:14:26 smithi028 ceph-mon[29847]: pgmap v166: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-12T22:14:26.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:14:26 smithi186 ceph-mon[34785]: pgmap v166: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-12T22:14:28.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:14:28 smithi086 ceph-mon[34297]: pgmap v167: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:14:28.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:14:28 smithi028 ceph-mon[29847]: pgmap v167: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:14:28.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:14:28 smithi186 ceph-mon[34785]: pgmap v167: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:14:30.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:14:30 smithi028 ceph-mon[29847]: pgmap v168: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:14:30.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:14:30 smithi186 ceph-mon[34785]: pgmap v168: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:14:30.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:14:30 smithi086 ceph-mon[34297]: pgmap v168: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:14:32.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:14:32 smithi028 ceph-mon[29847]: pgmap v169: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-12T22:14:32.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:14:32 smithi186 ceph-mon[34785]: pgmap v169: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-12T22:14:32.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:14:32 smithi086 ceph-mon[34297]: pgmap v169: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-12T22:14:34.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:14:34 smithi028 ceph-mon[29847]: pgmap v170: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:14:34.545 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:14:34 smithi186 ceph-mon[34785]: pgmap v170: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:14:34.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:14:34 smithi086 ceph-mon[34297]: pgmap v170: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:14:36.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:14:36 smithi028 ceph-mon[29847]: pgmap v171: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-12T22:14:36.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:14:36 smithi186 ceph-mon[34785]: pgmap v171: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-12T22:14:36.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:14:36 smithi086 ceph-mon[34297]: pgmap v171: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-12T22:14:37.774 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-12T22:14:37.774 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-12T22:14:38.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.002+0000 7f174e800640 1 -- 172.21.15.28:0/4109194352 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1748107dd0 msgr2=0x7f1748108230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:14:38.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.002+0000 7f174e800640 1 --2- 172.21.15.28:0/4109194352 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1748107dd0 0x7f1748108230 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f1734009900 tx=0x7f173402a480 comp rx=0 tx=0).stop 2024-05-12T22:14:38.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.002+0000 7f174e800640 1 -- 172.21.15.28:0/4109194352 shutdown_connections 2024-05-12T22:14:38.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.002+0000 7f174e800640 1 --2- 172.21.15.28:0/4109194352 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1748108770 0x7f1748114b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:38.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.002+0000 7f174e800640 1 --2- 172.21.15.28:0/4109194352 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1748107dd0 0x7f1748108230 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:38.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.002+0000 7f174e800640 1 --2- 172.21.15.28:0/4109194352 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f174810ddf0 0x7f174810e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:38.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.002+0000 7f174e800640 1 -- 172.21.15.28:0/4109194352 >> 172.21.15.28:0/4109194352 conn(0x7f1748077650 msgr2=0x7f1748106650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:14:38.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.003+0000 7f174e800640 1 -- 172.21.15.28:0/4109194352 shutdown_connections 2024-05-12T22:14:38.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.003+0000 7f174e800640 1 -- 172.21.15.28:0/4109194352 wait complete. 2024-05-12T22:14:38.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.004+0000 7f174e800640 1 Processor -- start 2024-05-12T22:14:38.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.004+0000 7f174e800640 1 -- start start 2024-05-12T22:14:38.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.004+0000 7f174e800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1748107dd0 0x7f174819e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:38.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.004+0000 7f174e800640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1748108770 0x7f174819eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:38.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.004+0000 7f174e800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f174810ddf0 0x7f17481a2e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:38.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.004+0000 7f174e800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1748118120 con 0x7f1748107dd0 2024-05-12T22:14:38.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.004+0000 7f174e800640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f1748117fa0 con 0x7f174810ddf0 2024-05-12T22:14:38.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.004+0000 7f174e800640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f17481182a0 con 0x7f1748108770 2024-05-12T22:14:38.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.004+0000 7f174d400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1748107dd0 0x7f174819e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:38.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.004+0000 7f174de00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f174810ddf0 0x7f17481a2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:38.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.004+0000 7f174d400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1748107dd0 0x7f174819e560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38716/0 (socket says 172.21.15.28:38716) 2024-05-12T22:14:38.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.004+0000 7f174de00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f174810ddf0 0x7f17481a2e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:49586/0 (socket says 172.21.15.28:49586) 2024-05-12T22:14:38.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.004+0000 7f174d400640 1 -- 172.21.15.28:0/2084972441 learned_addr learned my addr 172.21.15.28:0/2084972441 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:14:38.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.004+0000 7f174ca00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1748108770 0x7f174819eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:38.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.005+0000 7f174de00640 1 -- 172.21.15.28:0/2084972441 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1748108770 msgr2=0x7f174819eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:14:38.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.005+0000 7f174de00640 1 --2- 172.21.15.28:0/2084972441 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1748108770 0x7f174819eaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:38.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.005+0000 7f174de00640 1 -- 172.21.15.28:0/2084972441 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1748107dd0 msgr2=0x7f174819e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:14:38.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.005+0000 7f174de00640 1 --2- 172.21.15.28:0/2084972441 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1748107dd0 0x7f174819e560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:38.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.005+0000 7f174de00640 1 -- 172.21.15.28:0/2084972441 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17481a3550 con 0x7f174810ddf0 2024-05-12T22:14:38.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.005+0000 7f174d400640 1 --2- 172.21.15.28:0/2084972441 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1748107dd0 0x7f174819e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:14:38.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.005+0000 7f174de00640 1 --2- 172.21.15.28:0/2084972441 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f174810ddf0 0x7f17481a2e20 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f1740005fd0 tx=0x7f174000ee60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:14:38.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.005+0000 7f1746000640 1 -- 172.21.15.28:0/2084972441 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f174000cd80 con 0x7f174810ddf0 2024-05-12T22:14:38.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.006+0000 7f174e800640 1 -- 172.21.15.28:0/2084972441 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17481a3840 con 0x7f174810ddf0 2024-05-12T22:14:38.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.006+0000 7f1746000640 1 -- 172.21.15.28:0/2084972441 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1740005020 con 0x7f174810ddf0 2024-05-12T22:14:38.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.006+0000 7f1746000640 1 -- 172.21.15.28:0/2084972441 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1740010720 con 0x7f174810ddf0 2024-05-12T22:14:38.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.006+0000 7f174e800640 1 -- 172.21.15.28:0/2084972441 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17481ab0d0 con 0x7f174810ddf0 2024-05-12T22:14:38.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.009+0000 7f1752d98740 1 Processor -- start 2024-05-12T22:14:38.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.009+0000 7f1746000640 1 -- 172.21.15.28:0/2084972441 <== mon.2 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7f17400051d0 con 0x7f174810ddf0 2024-05-12T22:14:38.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.009+0000 7f1752d98740 1 -- start start 2024-05-12T22:14:38.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.010+0000 7f1752d98740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cc79f63dd0 0x55cc79f641b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:38.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.010+0000 7f1752d98740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cc79f646f0 0x55cc79f6e880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:38.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.010+0000 7f1752d98740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cc79f6edc0 0x55cc79f711b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:38.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.010+0000 7f1752d98740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55cc79f2c2a0 con 0x55cc79f63dd0 2024-05-12T22:14:38.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.010+0000 7f1752d98740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x55cc79ec8b90 con 0x55cc79f6edc0 2024-05-12T22:14:38.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.010+0000 7f1752d98740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55cc79e705b0 con 0x55cc79f646f0 2024-05-12T22:14:38.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.010+0000 7f174ca00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cc79f63dd0 0x55cc79f641b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:38.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.010+0000 7f174ca00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cc79f63dd0 0x55cc79f641b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38722/0 (socket says 172.21.15.28:38722) 2024-05-12T22:14:38.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.010+0000 7f174ca00640 1 -- 172.21.15.28:0/63338436 learned_addr learned my addr 172.21.15.28:0/63338436 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:14:38.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.010+0000 7f174d400640 1 --2- 172.21.15.28:0/63338436 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cc79f646f0 0x55cc79f6e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:38.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.010+0000 7f174ca00640 1 --2- 172.21.15.28:0/63338436 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cc79f6edc0 0x55cc79f711b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:38.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.011+0000 7f174d400640 1 -- 172.21.15.28:0/63338436 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cc79f6edc0 msgr2=0x55cc79f711b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:14:38.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.011+0000 7f174d400640 1 --2- 172.21.15.28:0/63338436 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cc79f6edc0 0x55cc79f711b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:38.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.011+0000 7f174d400640 1 -- 172.21.15.28:0/63338436 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cc79f63dd0 msgr2=0x55cc79f641b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:14:38.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.011+0000 7f174d400640 1 --2- 172.21.15.28:0/63338436 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cc79f63dd0 0x55cc79f641b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:38.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.011+0000 7f174d400640 1 -- 172.21.15.28:0/63338436 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cc79ca67d0 con 0x55cc79f646f0 2024-05-12T22:14:38.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.011+0000 7f174ca00640 1 --2- 172.21.15.28:0/63338436 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cc79f63dd0 0x55cc79f641b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:14:38.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.011+0000 7f1746000640 1 --2- 172.21.15.28:0/2084972441 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f17300a2d00 0x7f17300a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:38.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.011+0000 7f174d400640 1 --2- 172.21.15.28:0/63338436 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cc79f646f0 0x55cc79f6e880 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f173c011510 tx=0x7f173c0119d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ed95577318eee614 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:14:38.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.011+0000 7f1746000640 1 -- 172.21.15.28:0/2084972441 <== mon.2 v2:172.21.15.86:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f17400cfa10 con 0x7f174810ddf0 2024-05-12T22:14:38.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.011+0000 7f174ca00640 1 --2- 172.21.15.28:0/2084972441 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f17300a2d00 0x7f17300a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:38.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.011+0000 7f173a600640 1 -- 172.21.15.28:0/63338436 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f173c005020 con 0x55cc79f646f0 2024-05-12T22:14:38.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.011+0000 7f173a600640 1 -- 172.21.15.28:0/63338436 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f173c0047a0 con 0x55cc79f646f0 2024-05-12T22:14:38.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.011+0000 7f173a600640 1 -- 172.21.15.28:0/63338436 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f173c00f920 con 0x55cc79f646f0 2024-05-12T22:14:38.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.012+0000 7f1752d98740 1 -- 172.21.15.28:0/63338436 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cc79f646f0 msgr2=0x55cc79f6e880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:14:38.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.012+0000 7f1752d98740 1 --2- 172.21.15.28:0/63338436 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cc79f646f0 0x55cc79f6e880 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f173c011510 tx=0x7f173c0119d0 comp rx=0 tx=0).stop 2024-05-12T22:14:38.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.012+0000 7f174ca00640 1 --2- 172.21.15.28:0/2084972441 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f17300a2d00 0x7f17300a51c0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f1734031550 tx=0x7f1734005bc0 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:14:38.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.012+0000 7f1752d98740 1 -- 172.21.15.28:0/63338436 shutdown_connections 2024-05-12T22:14:38.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.012+0000 7f1752d98740 1 --2- 172.21.15.28:0/63338436 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cc79f6edc0 0x55cc79f711b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:38.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.012+0000 7f1752d98740 1 --2- 172.21.15.28:0/63338436 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cc79f646f0 0x55cc79f6e880 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:38.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.012+0000 7f1752d98740 1 --2- 172.21.15.28:0/63338436 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cc79f63dd0 0x55cc79f641b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:38.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.012+0000 7f1752d98740 1 -- 172.21.15.28:0/63338436 >> 172.21.15.28:0/63338436 conn(0x55cc79f600f0 msgr2=0x55cc79f604d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:14:38.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.012+0000 7f1752d98740 1 -- 172.21.15.28:0/63338436 shutdown_connections 2024-05-12T22:14:38.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.012+0000 7f1752d98740 1 -- 172.21.15.28:0/63338436 wait complete. 2024-05-12T22:14:38.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.013+0000 7f1752d98740 1 -- start start 2024-05-12T22:14:38.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.013+0000 7f1752d98740 1 Processor -- start 2024-05-12T22:14:38.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.013+0000 7f1752d98740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cc79f63dd0 0x55cc79f6e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:38.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.013+0000 7f1752d98740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cc79f646f0 0x55cc79f6b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:38.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.013+0000 7f1752d98740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cc79f6edc0 0x55cc79f6bed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:38.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.013+0000 7f1752d98740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55cc79e839f0 con 0x55cc79f646f0 2024-05-12T22:14:38.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.013+0000 7f1752d98740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x55cc79ec4b50 con 0x55cc79f63dd0 2024-05-12T22:14:38.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.013+0000 7f1752d98740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55cc79f71bb0 con 0x55cc79f6edc0 2024-05-12T22:14:38.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.014+0000 7f1752d98740 10 client.0 _ll_register_callbacks cb 0x55cc79bdbbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:14:38.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.014+0000 7f174ca00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cc79f646f0 0x55cc79f6b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:38.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.014+0000 7f174ca00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cc79f646f0 0x55cc79f6b990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38724/0 (socket says 172.21.15.28:38724) 2024-05-12T22:14:38.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.014+0000 7f174ca00640 1 -- 172.21.15.28:0/3811401983 learned_addr learned my addr 172.21.15.28:0/3811401983 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:14:38.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.014+0000 7f174de00640 1 --2- 172.21.15.28:0/3811401983 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cc79f6edc0 0x55cc79f6bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:38.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.014+0000 7f174d400640 1 --2- 172.21.15.28:0/3811401983 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cc79f63dd0 0x55cc79f6e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:38.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.014+0000 7f174de00640 1 -- 172.21.15.28:0/3811401983 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cc79f63dd0 msgr2=0x55cc79f6e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:14:38.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.014+0000 7f174de00640 1 --2- 172.21.15.28:0/3811401983 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cc79f63dd0 0x55cc79f6e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:38.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.014+0000 7f174de00640 1 -- 172.21.15.28:0/3811401983 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cc79f646f0 msgr2=0x55cc79f6b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:14:38.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.014+0000 7f174de00640 1 --2- 172.21.15.28:0/3811401983 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cc79f646f0 0x55cc79f6b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:38.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.014+0000 7f174de00640 1 -- 172.21.15.28:0/3811401983 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cc79bdc3f0 con 0x55cc79f6edc0 2024-05-12T22:14:38.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.015+0000 7f174de00640 1 --2- 172.21.15.28:0/3811401983 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cc79f6edc0 0x55cc79f6bed0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f17400cd130 tx=0x7f174009c740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a8f9321b5caf48f7 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:14:38.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.015+0000 7f172ea00640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-12T22:14:38.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.015+0000 7f172ea00640 1 -- 172.21.15.28:0/3811401983 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17400a4b70 con 0x55cc79f6edc0 2024-05-12T22:14:38.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.015+0000 7f172ea00640 1 -- 172.21.15.28:0/3811401983 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f174009c160 con 0x55cc79f6edc0 2024-05-12T22:14:38.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.015+0000 7f1752d98740 1 -- 172.21.15.28:0/3811401983 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f173c0047a0 con 0x55cc79f6edc0 2024-05-12T22:14:38.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.015+0000 7f172ea00640 1 -- 172.21.15.28:0/3811401983 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f174009c360 con 0x55cc79f6edc0 2024-05-12T22:14:38.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.015+0000 7f1752d98740 10 client.26434 fetch_fsmap learned FSMap version 26 2024-05-12T22:14:38.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.015+0000 7f1752d98740 1 -- 172.21.15.28:0/3811401983 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55cc79b9c0f0 con 0x55cc79f6edc0 2024-05-12T22:14:38.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.015+0000 7f172ea00640 1 -- 172.21.15.28:0/3811401983 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f173c0047a0 con 0x55cc79f6edc0 2024-05-12T22:14:38.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.016+0000 7f172ea00640 1 -- 172.21.15.28:0/3811401983 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f174009b350 con 0x55cc79f6edc0 2024-05-12T22:14:38.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.016+0000 7f1752d98740 10 client.26434 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:14:38.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.016+0000 7f1752d98740 10 client.26434 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:14:38.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.016+0000 7f1752d98740 20 client.26434 populate_metadata read hostname 'smithi028' 2024-05-12T22:14:38.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.016+0000 7f1752d98740 1 --2- 172.21.15.28:0/3811401983 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55cc79f7ea10 0x55cc79f80e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:38.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.016+0000 7f1752d98740 4 client.26434 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-12T22:14:38.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.016+0000 7f1752d98740 1 -- 172.21.15.28:0/3811401983 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cc79ed62c0 con 0x55cc79f7ea10 2024-05-12T22:14:38.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.016+0000 7f174d400640 1 --2- 172.21.15.28:0/3811401983 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55cc79f7ea10 0x55cc79f80e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:38.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.017+0000 7f174d400640 1 --2- 172.21.15.28:0/3811401983 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55cc79f7ea10 0x55cc79f80e70 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b8015756e0178e15 server_cookie=e32da74207ccda3e in_seq=0 out_seq=0 2024-05-12T22:14:38.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.017+0000 7f172ea00640 10 client.26434 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:14:38.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.018+0000 7f172ea00640 1 -- 172.21.15.28:0/3811401983 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55cc79ed62c0 con 0x55cc79f7ea10 2024-05-12T22:14:38.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.018+0000 7f172ea00640 10 client.26434 handle_command_reply: tid=0 2024-05-12T22:14:38.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.018+0000 7f1752d98740 1 client.26434 shutdown 2024-05-12T22:14:38.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.018+0000 7f1752d98740 1 -- 172.21.15.28:0/3811401983 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cc79f6edc0 msgr2=0x55cc79f6bed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:14:38.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.018+0000 7f1752d98740 1 --2- 172.21.15.28:0/3811401983 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cc79f6edc0 0x55cc79f6bed0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f17400cd130 tx=0x7f174009c740 comp rx=0 tx=0).stop 2024-05-12T22:14:38.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.018+0000 7f1752d98740 1 -- 172.21.15.28:0/3811401983 shutdown_connections 2024-05-12T22:14:38.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.018+0000 7f1752d98740 1 --2- 172.21.15.28:0/3811401983 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55cc79f7ea10 0x55cc79f80e70 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:38.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.018+0000 7f1752d98740 1 --2- 172.21.15.28:0/3811401983 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cc79f6edc0 0x55cc79f6bed0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:38.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.018+0000 7f1752d98740 1 --2- 172.21.15.28:0/3811401983 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cc79f646f0 0x55cc79f6b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:38.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.018+0000 7f1752d98740 1 --2- 172.21.15.28:0/3811401983 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cc79f63dd0 0x55cc79f6e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:38.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.018+0000 7f1752d98740 1 -- 172.21.15.28:0/3811401983 >> 172.21.15.28:0/3811401983 conn(0x55cc79f600f0 msgr2=0x55cc79f5f330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:14:38.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.018+0000 7f172ea00640 0 client.26434 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:14:38.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.018+0000 7f1752d98740 1 -- 172.21.15.28:0/3811401983 shutdown_connections 2024-05-12T22:14:38.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.018+0000 7f1752d98740 1 -- 172.21.15.28:0/3811401983 wait complete. 2024-05-12T22:14:38.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.018+0000 7f1752d98740 20 client.26434 trim_cache size 0 max 16384 2024-05-12T22:14:38.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.038+0000 7f1752d98740 1 Processor -- start 2024-05-12T22:14:38.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.038+0000 7f1752d98740 1 -- start start 2024-05-12T22:14:38.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.038+0000 7f1752d98740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cc79f63dd0 0x55cc79f65c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:38.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.038+0000 7f1752d98740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cc79f646f0 0x55cc79f6e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:38.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.038+0000 7f1752d98740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cc79f6edc0 0x55cc79f86740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:38.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.038+0000 7f1752d98740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f17481b6070 con 0x55cc79f63dd0 2024-05-12T22:14:38.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.038+0000 7f1752d98740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f17481b3940 con 0x55cc79f6edc0 2024-05-12T22:14:38.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.038+0000 7f1752d98740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f17481b1790 con 0x55cc79f646f0 2024-05-12T22:14:38.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.038+0000 7f174de00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cc79f6edc0 0x55cc79f86740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:38.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.038+0000 7f174de00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cc79f6edc0 0x55cc79f86740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:49622/0 (socket says 172.21.15.28:49622) 2024-05-12T22:14:38.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.038+0000 7f174de00640 1 -- 172.21.15.28:0/2071958765 learned_addr learned my addr 172.21.15.28:0/2071958765 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:14:38.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.039+0000 7f174d400640 1 --2- 172.21.15.28:0/2071958765 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cc79f63dd0 0x55cc79f65c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:38.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.039+0000 7f174ca00640 1 --2- 172.21.15.28:0/2071958765 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cc79f646f0 0x55cc79f6e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:38.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.039+0000 7f174de00640 1 -- 172.21.15.28:0/2071958765 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cc79f646f0 msgr2=0x55cc79f6e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:14:38.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.039+0000 7f174de00640 1 --2- 172.21.15.28:0/2071958765 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cc79f646f0 0x55cc79f6e1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:38.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.039+0000 7f174de00640 1 -- 172.21.15.28:0/2071958765 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cc79f63dd0 msgr2=0x55cc79f65c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:14:38.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.039+0000 7f174de00640 1 --2- 172.21.15.28:0/2071958765 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cc79f63dd0 0x55cc79f65c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:38.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.039+0000 7f174de00640 1 -- 172.21.15.28:0/2071958765 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cc79f5a740 con 0x55cc79f6edc0 2024-05-12T22:14:38.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.039+0000 7f174de00640 1 --2- 172.21.15.28:0/2071958765 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cc79f6edc0 0x55cc79f86740 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f174009be90 tx=0x7f17400bc130 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7e61f730a9d39ff server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:14:38.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.039+0000 7f1739c00640 1 -- 172.21.15.28:0/2071958765 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f174009b350 con 0x55cc79f6edc0 2024-05-12T22:14:38.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.039+0000 7f1739c00640 1 -- 172.21.15.28:0/2071958765 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1740098570 con 0x55cc79f6edc0 2024-05-12T22:14:38.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.039+0000 7f1739c00640 1 -- 172.21.15.28:0/2071958765 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17400a4940 con 0x55cc79f6edc0 2024-05-12T22:14:38.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.040+0000 7f1752d98740 1 -- 172.21.15.28:0/2071958765 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cc79f6edc0 msgr2=0x55cc79f86740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:14:38.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.040+0000 7f1752d98740 1 --2- 172.21.15.28:0/2071958765 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cc79f6edc0 0x55cc79f86740 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f174009be90 tx=0x7f17400bc130 comp rx=0 tx=0).stop 2024-05-12T22:14:38.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.040+0000 7f1752d98740 1 -- 172.21.15.28:0/2071958765 shutdown_connections 2024-05-12T22:14:38.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.040+0000 7f1752d98740 1 --2- 172.21.15.28:0/2071958765 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cc79f6edc0 0x55cc79f86740 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:38.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.040+0000 7f1752d98740 1 --2- 172.21.15.28:0/2071958765 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cc79f646f0 0x55cc79f6e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:38.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.040+0000 7f1752d98740 1 --2- 172.21.15.28:0/2071958765 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cc79f63dd0 0x55cc79f65c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:38.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.040+0000 7f1752d98740 1 -- 172.21.15.28:0/2071958765 >> 172.21.15.28:0/2071958765 conn(0x55cc79f600f0 msgr2=0x55cc79f65520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:14:38.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.040+0000 7f1752d98740 1 -- 172.21.15.28:0/2071958765 shutdown_connections 2024-05-12T22:14:38.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.040+0000 7f1752d98740 1 -- 172.21.15.28:0/2071958765 wait complete. 2024-05-12T22:14:38.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.040+0000 7f1752d98740 1 -- start start 2024-05-12T22:14:38.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.041+0000 7f1752d98740 1 Processor -- start 2024-05-12T22:14:38.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.041+0000 7f1752d98740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cc79f63dd0 0x55cc79f685c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:38.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.041+0000 7f1752d98740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cc79f646f0 0x55cc79f68b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:38.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.041+0000 7f1752d98740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cc79f6edc0 0x55cc79f69040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:38.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.041+0000 7f1752d98740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f17481b1e00 con 0x55cc79f63dd0 2024-05-12T22:14:38.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.041+0000 7f1752d98740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f17481b3270 con 0x55cc79f646f0 2024-05-12T22:14:38.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.041+0000 7f1752d98740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f17481b24d0 con 0x55cc79f6edc0 2024-05-12T22:14:38.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.041+0000 7f1752d98740 10 client.0 _ll_register_callbacks cb 0x7f1740005b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:14:38.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.041+0000 7f174d400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cc79f63dd0 0x55cc79f685c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:38.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.041+0000 7f174d400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cc79f63dd0 0x55cc79f685c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38742/0 (socket says 172.21.15.28:38742) 2024-05-12T22:14:38.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.041+0000 7f174d400640 1 -- 172.21.15.28:0/2726169029 learned_addr learned my addr 172.21.15.28:0/2726169029 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:14:38.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.041+0000 7f174de00640 1 --2- 172.21.15.28:0/2726169029 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cc79f6edc0 0x55cc79f69040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:38.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.042+0000 7f174ca00640 1 --2- 172.21.15.28:0/2726169029 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cc79f646f0 0x55cc79f68b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:38.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.042+0000 7f174d400640 1 -- 172.21.15.28:0/2726169029 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cc79f6edc0 msgr2=0x55cc79f69040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:14:38.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.042+0000 7f174d400640 1 --2- 172.21.15.28:0/2726169029 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cc79f6edc0 0x55cc79f69040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:38.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.042+0000 7f174d400640 1 -- 172.21.15.28:0/2726169029 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cc79f646f0 msgr2=0x55cc79f68b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:14:38.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.042+0000 7f174d400640 1 --2- 172.21.15.28:0/2726169029 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cc79f646f0 0x55cc79f68b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:38.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.042+0000 7f174d400640 1 -- 172.21.15.28:0/2726169029 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cc79fe3fb0 con 0x55cc79f63dd0 2024-05-12T22:14:38.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.042+0000 7f174de00640 1 --2- 172.21.15.28:0/2726169029 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cc79f6edc0 0x55cc79f69040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:14:38.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.042+0000 7f174d400640 1 --2- 172.21.15.28:0/2726169029 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cc79f63dd0 0x55cc79f685c0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f173c004b70 tx=0x7f173c005d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cddbe2bf869aed81 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:14:38.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.042+0000 7f172ea00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-12T22:14:38.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.042+0000 7f172ea00640 1 -- 172.21.15.28:0/2726169029 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f173c02bde0 con 0x55cc79f63dd0 2024-05-12T22:14:38.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.042+0000 7f1752d98740 1 -- 172.21.15.28:0/2726169029 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1740098570 con 0x55cc79f63dd0 2024-05-12T22:14:38.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.043+0000 7f172ea00640 1 -- 172.21.15.28:0/2726169029 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f173c02b2c0 con 0x55cc79f63dd0 2024-05-12T22:14:38.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.043+0000 7f172ea00640 1 -- 172.21.15.28:0/2726169029 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f173c03a050 con 0x55cc79f63dd0 2024-05-12T22:14:38.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.043+0000 7f1752d98740 10 client.16353 fetch_fsmap learned FSMap version 26 2024-05-12T22:14:38.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.043+0000 7f1752d98740 1 -- 172.21.15.28:0/2726169029 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55cc79fe5200 con 0x55cc79f63dd0 2024-05-12T22:14:38.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.043+0000 7f172ea00640 1 -- 172.21.15.28:0/2726169029 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1740098570 con 0x55cc79f63dd0 2024-05-12T22:14:38.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.043+0000 7f172ea00640 1 -- 172.21.15.28:0/2726169029 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f173c0035f0 con 0x55cc79f63dd0 2024-05-12T22:14:38.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.043+0000 7f1752d98740 10 client.16353 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:14:38.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.043+0000 7f1752d98740 10 client.16353 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:14:38.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.043+0000 7f1752d98740 20 client.16353 populate_metadata read hostname 'smithi028' 2024-05-12T22:14:38.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.043+0000 7f1752d98740 1 --2- 172.21.15.28:0/2726169029 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55cc79fe5790 0x55cc79fe7b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:38.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.043+0000 7f1752d98740 4 client.16353 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-12T22:14:38.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.043+0000 7f1752d98740 1 -- 172.21.15.28:0/2726169029 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55cc79ed0610 con 0x55cc79fe5790 2024-05-12T22:14:38.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.044+0000 7f174ca00640 1 --2- 172.21.15.28:0/2726169029 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55cc79fe5790 0x55cc79fe7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:38.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.044+0000 7f174ca00640 1 --2- 172.21.15.28:0/2726169029 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55cc79fe5790 0x55cc79fe7b80 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f8ffe02069ab3c71 server_cookie=8ddb91195cba5aed in_seq=0 out_seq=0 2024-05-12T22:14:38.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.044+0000 7f172ea00640 10 client.16353 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:14:38.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.045+0000 7f172ea00640 1 -- 172.21.15.28:0/2726169029 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55cc79ed0610 con 0x55cc79fe5790 2024-05-12T22:14:38.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.045+0000 7f172ea00640 10 client.16353 handle_command_reply: tid=0 2024-05-12T22:14:38.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.045+0000 7f1752d98740 1 client.16353 shutdown 2024-05-12T22:14:38.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.045+0000 7f1752d98740 1 -- 172.21.15.28:0/2726169029 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cc79f63dd0 msgr2=0x55cc79f685c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:14:38.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.045+0000 7f1752d98740 1 --2- 172.21.15.28:0/2726169029 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cc79f63dd0 0x55cc79f685c0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f173c004b70 tx=0x7f173c005d40 comp rx=0 tx=0).stop 2024-05-12T22:14:38.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.045+0000 7f1752d98740 1 -- 172.21.15.28:0/2726169029 shutdown_connections 2024-05-12T22:14:38.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.045+0000 7f1752d98740 1 --2- 172.21.15.28:0/2726169029 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55cc79fe5790 0x55cc79fe7b80 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:38.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.045+0000 7f1752d98740 1 --2- 172.21.15.28:0/2726169029 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cc79f6edc0 0x55cc79f69040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:38.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.045+0000 7f1752d98740 1 --2- 172.21.15.28:0/2726169029 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cc79f646f0 0x55cc79f68b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:38.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.045+0000 7f172ea00640 0 client.16353 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:14:38.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.045+0000 7f1752d98740 1 --2- 172.21.15.28:0/2726169029 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cc79f63dd0 0x55cc79f685c0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:38.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.045+0000 7f1752d98740 1 -- 172.21.15.28:0/2726169029 >> 172.21.15.28:0/2726169029 conn(0x55cc79f600f0 msgr2=0x55cc79f65db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:14:38.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.045+0000 7f1752d98740 1 -- 172.21.15.28:0/2726169029 shutdown_connections 2024-05-12T22:14:38.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.045+0000 7f1752d98740 1 -- 172.21.15.28:0/2726169029 wait complete. 2024-05-12T22:14:38.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.045+0000 7f1752d98740 20 client.16353 trim_cache size 0 max 16384 2024-05-12T22:14:38.055 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-05-12T22:14:38.055 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-05-12T22:14:38.056 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-05-12T22:14:38.056 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-05-12T22:14:38.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.046+0000 7f174e800640 1 -- 172.21.15.28:0/2084972441 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f17300a2d00 msgr2=0x7f17300a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:14:38.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.046+0000 7f174e800640 1 --2- 172.21.15.28:0/2084972441 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f17300a2d00 0x7f17300a51c0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f1734031550 tx=0x7f1734005bc0 comp rx=0 tx=0).stop 2024-05-12T22:14:38.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.046+0000 7f174e800640 1 -- 172.21.15.28:0/2084972441 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f174810ddf0 msgr2=0x7f17481a2e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:14:38.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.046+0000 7f174e800640 1 --2- 172.21.15.28:0/2084972441 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f174810ddf0 0x7f17481a2e20 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f1740005fd0 tx=0x7f174000ee60 comp rx=0 tx=0).stop 2024-05-12T22:14:38.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.046+0000 7f174e800640 1 -- 172.21.15.28:0/2084972441 shutdown_connections 2024-05-12T22:14:38.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.046+0000 7f174e800640 1 --2- 172.21.15.28:0/2084972441 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f17300a2d00 0x7f17300a51c0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:38.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.046+0000 7f174e800640 1 --2- 172.21.15.28:0/2084972441 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f174810ddf0 0x7f17481a2e20 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:38.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.046+0000 7f174e800640 1 --2- 172.21.15.28:0/2084972441 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1748108770 0x7f174819eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:38.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.046+0000 7f174e800640 1 --2- 172.21.15.28:0/2084972441 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1748107dd0 0x7f174819e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:38.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.046+0000 7f174e800640 1 -- 172.21.15.28:0/2084972441 >> 172.21.15.28:0/2084972441 conn(0x7f1748077650 msgr2=0x7f1748078520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:14:38.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.046+0000 7f174e800640 1 -- 172.21.15.28:0/2084972441 shutdown_connections 2024-05-12T22:14:38.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.046+0000 7f174e800640 1 -- 172.21.15.28:0/2084972441 wait complete. 2024-05-12T22:14:38.065 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-12T22:14:38.066 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-12T22:14:38.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.276+0000 7f3723e00640 1 -- 172.21.15.28:0/1587574657 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f372410dd40 msgr2=0x7f372410e120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:14:38.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.276+0000 7f3723e00640 1 --2- 172.21.15.28:0/1587574657 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f372410dd40 0x7f372410e120 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f3714009860 tx=0x7f371402a3d0 comp rx=0 tx=0).stop 2024-05-12T22:14:38.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.276+0000 7f3723e00640 1 -- 172.21.15.28:0/1587574657 shutdown_connections 2024-05-12T22:14:38.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.276+0000 7f3723e00640 1 --2- 172.21.15.28:0/1587574657 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3724108670 0x7f3724114ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:38.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.276+0000 7f3723e00640 1 --2- 172.21.15.28:0/1587574657 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3724107cd0 0x7f3724108130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:38.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.276+0000 7f3723e00640 1 --2- 172.21.15.28:0/1587574657 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f372410dd40 0x7f372410e120 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:38.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.276+0000 7f3723e00640 1 -- 172.21.15.28:0/1587574657 >> 172.21.15.28:0/1587574657 conn(0x7f37240773b0 msgr2=0x7f37240797d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:14:38.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.276+0000 7f3723e00640 1 -- 172.21.15.28:0/1587574657 shutdown_connections 2024-05-12T22:14:38.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.276+0000 7f3723e00640 1 -- 172.21.15.28:0/1587574657 wait complete. 2024-05-12T22:14:38.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.277+0000 7f3723e00640 1 Processor -- start 2024-05-12T22:14:38.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.277+0000 7f3723e00640 1 -- start start 2024-05-12T22:14:38.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.278+0000 7f3723e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3724107cd0 0x7f372419e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:38.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.278+0000 7f3723e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3724108670 0x7f372419e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:38.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.278+0000 7f3723e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f372410dd40 0x7f37241a2d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:38.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.278+0000 7f3723e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3724118070 con 0x7f372410dd40 2024-05-12T22:14:38.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.278+0000 7f3723e00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f3724117ef0 con 0x7f3724108670 2024-05-12T22:14:38.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.278+0000 7f3723e00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f37241181f0 con 0x7f3724107cd0 2024-05-12T22:14:38.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.278+0000 7f3722a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3724107cd0 0x7f372419e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:38.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.278+0000 7f3723400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f372410dd40 0x7f37241a2d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:38.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.278+0000 7f3722a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3724107cd0 0x7f372419e4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:60308/0 (socket says 172.21.15.28:60308) 2024-05-12T22:14:38.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.278+0000 7f3722a00640 1 -- 172.21.15.28:0/3506153285 learned_addr learned my addr 172.21.15.28:0/3506153285 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:14:38.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.278+0000 7f371be00640 1 --2- 172.21.15.28:0/3506153285 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3724108670 0x7f372419e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:38.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.278+0000 7f3722a00640 1 -- 172.21.15.28:0/3506153285 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3724108670 msgr2=0x7f372419e9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:14:38.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.279+0000 7f3722a00640 1 --2- 172.21.15.28:0/3506153285 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3724108670 0x7f372419e9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:38.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.279+0000 7f3722a00640 1 -- 172.21.15.28:0/3506153285 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f372410dd40 msgr2=0x7f37241a2d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:14:38.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.279+0000 7f3722a00640 1 --2- 172.21.15.28:0/3506153285 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f372410dd40 0x7f37241a2d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:38.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.279+0000 7f3722a00640 1 -- 172.21.15.28:0/3506153285 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37241a34a0 con 0x7f3724107cd0 2024-05-12T22:14:38.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.279+0000 7f3723400640 1 --2- 172.21.15.28:0/3506153285 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f372410dd40 0x7f37241a2d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:14:38.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.279+0000 7f371be00640 1 --2- 172.21.15.28:0/3506153285 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3724108670 0x7f372419e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:14:38.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.279+0000 7f3722a00640 1 --2- 172.21.15.28:0/3506153285 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3724107cd0 0x7f372419e4b0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f3714009980 tx=0x7f37140309a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:14:38.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.279+0000 7f371b400640 1 -- 172.21.15.28:0/3506153285 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3714040070 con 0x7f3724107cd0 2024-05-12T22:14:38.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.279+0000 7f3723e00640 1 -- 172.21.15.28:0/3506153285 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37241a3730 con 0x7f3724107cd0 2024-05-12T22:14:38.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.279+0000 7f371b400640 1 -- 172.21.15.28:0/3506153285 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3714005620 con 0x7f3724107cd0 2024-05-12T22:14:38.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.279+0000 7f371b400640 1 -- 172.21.15.28:0/3506153285 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37140049b0 con 0x7f3724107cd0 2024-05-12T22:14:38.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.280+0000 7f3723e00640 1 -- 172.21.15.28:0/3506153285 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37241ab020 con 0x7f3724107cd0 2024-05-12T22:14:38.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.281+0000 7f3723e00640 1 -- 172.21.15.28:0/3506153285 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f37240689c0 con 0x7f3724107cd0 2024-05-12T22:14:38.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.282+0000 7f371b400640 1 -- 172.21.15.28:0/3506153285 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7f3714043070 con 0x7f3724107cd0 2024-05-12T22:14:38.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.284+0000 7f371b400640 1 --2- 172.21.15.28:0/3506153285 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f37080a2d80 0x7f37080a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:38.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.284+0000 7f371b400640 1 -- 172.21.15.28:0/3506153285 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f37140f0160 con 0x7f3724107cd0 2024-05-12T22:14:38.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.284+0000 7f371be00640 1 --2- 172.21.15.28:0/3506153285 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f37080a2d80 0x7f37080a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:38.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.285+0000 7f371be00640 1 --2- 172.21.15.28:0/3506153285 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f37080a2d80 0x7f37080a5240 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f372410ca40 tx=0x7f370c009260 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:14:38.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.288+0000 7f371b400640 1 -- 172.21.15.28:0/3506153285 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f37140bcd50 con 0x7f3724107cd0 2024-05-12T22:14:38.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:14:38 smithi028 ceph-mon[29847]: pgmap v172: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:14:38.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.535+0000 7f3723e00640 1 -- 172.21.15.28:0/3506153285 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f372419f800 con 0x7f3724107cd0 2024-05-12T22:14:38.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.536+0000 7f371b400640 1 -- 172.21.15.28:0/3506153285 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11864 (secure 0 0 0) 0x7f37140bc650 con 0x7f3724107cd0 2024-05-12T22:14:38.537 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-12T22:14:38.538 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1844141745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1844141745},{"type":"v1","addr":"172.21.15.186:6833","nonce":1844141745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6833/2960763758","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":2960763758},{"type":"v1","addr":"172.21.15.86:6833","nonce":2960763758}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24470,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/4131134219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":4131134219},{"type":"v1","addr":"172.21.15.28:6835","nonce":4131134219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24481,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/1727869064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1727869064},{"type":"v1","addr":"172.21.15.186:6835","nonce":1727869064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24482,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2938260704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2938260704},{"type":"v1","addr":"172.21.15.28:6837","nonce":2938260704}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24494,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3473700910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3473700910},{"type":"v1","addr":"172.21.15.86:6835","nonce":3473700910}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-12T22:06:33.426986+0000","modified":"2024-05-12T22:13:22.499381+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24581,"mds_1":24550,"mds_2":24532},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24499":{"gid":24499,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6837/3600809235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":3600809235},{"type":"v1","addr":"172.21.15.186:6837","nonce":3600809235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"i","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6839/2399610807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":2399610807},{"type":"v1","addr":"172.21.15.186:6839","nonce":2399610807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24530":{"gid":24530,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.86:6837/3782448884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":3782448884},{"type":"v1","addr":"172.21.15.86:6837","nonce":3782448884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24532":{"gid":24532,"name":"g","rank":2,"incarnation":21,"state":"up:active","state_seq":6,"addr":"172.21.15.28:6839/123892158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":123892158},{"type":"v1","addr":"172.21.15.28:6839","nonce":123892158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24550":{"gid":24550,"name":"a","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.28:6841/761183870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":761183870},{"type":"v1","addr":"172.21.15.28:6841","nonce":761183870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24581":{"gid":24581,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6839/1251999130","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6838","nonce":1251999130},{"type":"v1","addr":"172.21.15.86:6839","nonce":1251999130}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24581,"qdb_cluster":[24550,24581,24532]},"id":1}]} 2024-05-12T22:14:38.538 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 26 2024-05-12T22:14:38.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.539+0000 7f3723e00640 1 -- 172.21.15.28:0/3506153285 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f37080a2d80 msgr2=0x7f37080a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:14:38.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.539+0000 7f3723e00640 1 --2- 172.21.15.28:0/3506153285 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f37080a2d80 0x7f37080a5240 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f372410ca40 tx=0x7f370c009260 comp rx=0 tx=0).stop 2024-05-12T22:14:38.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.539+0000 7f3723e00640 1 -- 172.21.15.28:0/3506153285 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3724107cd0 msgr2=0x7f372419e4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:14:38.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.539+0000 7f3723e00640 1 --2- 172.21.15.28:0/3506153285 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3724107cd0 0x7f372419e4b0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f3714009980 tx=0x7f37140309a0 comp rx=0 tx=0).stop 2024-05-12T22:14:38.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.540+0000 7f3723e00640 1 -- 172.21.15.28:0/3506153285 shutdown_connections 2024-05-12T22:14:38.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.540+0000 7f3723e00640 1 --2- 172.21.15.28:0/3506153285 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f37080a2d80 0x7f37080a5240 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:38.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.540+0000 7f3723e00640 1 --2- 172.21.15.28:0/3506153285 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f372410dd40 0x7f37241a2d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:38.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.540+0000 7f3723e00640 1 --2- 172.21.15.28:0/3506153285 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3724108670 0x7f372419e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:38.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.540+0000 7f3723e00640 1 --2- 172.21.15.28:0/3506153285 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3724107cd0 0x7f372419e4b0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:38.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.540+0000 7f3723e00640 1 -- 172.21.15.28:0/3506153285 >> 172.21.15.28:0/3506153285 conn(0x7f37240773b0 msgr2=0x7f3724078b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:14:38.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.540+0000 7f3723e00640 1 -- 172.21.15.28:0/3506153285 shutdown_connections 2024-05-12T22:14:38.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.540+0000 7f3723e00640 1 -- 172.21.15.28:0/3506153285 wait complete. 2024-05-12T22:14:38.543 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:14:38 smithi086 ceph-mon[34297]: pgmap v172: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:14:38.560 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-05-12T22:14:38.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:14:38 smithi186 ceph-mon[34785]: pgmap v172: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:14:38.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.758+0000 7fa5d3e00640 1 -- 172.21.15.28:0/2877770052 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa5d4107dd0 msgr2=0x7fa5d4108230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:14:38.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.758+0000 7fa5d3e00640 1 --2- 172.21.15.28:0/2877770052 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa5d4107dd0 0x7fa5d4108230 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fa5bc009860 tx=0x7fa5bc02a3d0 comp rx=0 tx=0).stop 2024-05-12T22:14:38.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.758+0000 7fa5d3e00640 1 -- 172.21.15.28:0/2877770052 shutdown_connections 2024-05-12T22:14:38.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.758+0000 7fa5d3e00640 1 --2- 172.21.15.28:0/2877770052 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5d4108770 0x7fa5d4114b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:38.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.758+0000 7fa5d3e00640 1 --2- 172.21.15.28:0/2877770052 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa5d4107dd0 0x7fa5d4108230 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:38.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.758+0000 7fa5d3e00640 1 --2- 172.21.15.28:0/2877770052 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa5d410ddf0 0x7fa5d410e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:38.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.758+0000 7fa5d3e00640 1 -- 172.21.15.28:0/2877770052 >> 172.21.15.28:0/2877770052 conn(0x7fa5d4077650 msgr2=0x7fa5d4106650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:14:38.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.758+0000 7fa5d3e00640 1 -- 172.21.15.28:0/2877770052 shutdown_connections 2024-05-12T22:14:38.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.758+0000 7fa5d3e00640 1 -- 172.21.15.28:0/2877770052 wait complete. 2024-05-12T22:14:38.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.759+0000 7fa5d3e00640 1 Processor -- start 2024-05-12T22:14:38.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.759+0000 7fa5d3e00640 1 -- start start 2024-05-12T22:14:38.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.760+0000 7fa5d3e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa5d4107dd0 0x7fa5d419e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:38.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.760+0000 7fa5d3e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa5d4108770 0x7fa5d419ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:38.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.760+0000 7fa5d3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5d410ddf0 0x7fa5d41a2dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:38.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.760+0000 7fa5d3e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa5d4118100 con 0x7fa5d410ddf0 2024-05-12T22:14:38.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.760+0000 7fa5d3e00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fa5d4117f80 con 0x7fa5d4108770 2024-05-12T22:14:38.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.760+0000 7fa5d3e00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa5d4118280 con 0x7fa5d4107dd0 2024-05-12T22:14:38.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.760+0000 7fa5d2a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa5d4107dd0 0x7fa5d419e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:38.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.760+0000 7fa5d3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5d410ddf0 0x7fa5d41a2dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:38.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.760+0000 7fa5d2a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa5d4107dd0 0x7fa5d419e500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:60340/0 (socket says 172.21.15.28:60340) 2024-05-12T22:14:38.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.760+0000 7fa5d2a00640 1 -- 172.21.15.28:0/2170242031 learned_addr learned my addr 172.21.15.28:0/2170242031 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:14:38.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.760+0000 7fa5cbe00640 1 --2- 172.21.15.28:0/2170242031 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa5d4108770 0x7fa5d419ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:38.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.761+0000 7fa5d2a00640 1 -- 172.21.15.28:0/2170242031 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa5d4108770 msgr2=0x7fa5d419ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:14:38.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.761+0000 7fa5d2a00640 1 --2- 172.21.15.28:0/2170242031 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa5d4108770 0x7fa5d419ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:38.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.761+0000 7fa5d2a00640 1 -- 172.21.15.28:0/2170242031 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5d410ddf0 msgr2=0x7fa5d41a2dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:14:38.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.761+0000 7fa5d2a00640 1 --2- 172.21.15.28:0/2170242031 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5d410ddf0 0x7fa5d41a2dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:38.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.761+0000 7fa5d2a00640 1 -- 172.21.15.28:0/2170242031 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5d41a34f0 con 0x7fa5d4107dd0 2024-05-12T22:14:38.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.761+0000 7fa5cbe00640 1 --2- 172.21.15.28:0/2170242031 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa5d4108770 0x7fa5d419ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:14:38.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.761+0000 7fa5d3400640 1 --2- 172.21.15.28:0/2170242031 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5d410ddf0 0x7fa5d41a2dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:14:38.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.761+0000 7fa5d2a00640 1 --2- 172.21.15.28:0/2170242031 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa5d4107dd0 0x7fa5d419e500 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fa5c400e9c0 tx=0x7fa5c400ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:14:38.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.762+0000 7fa5d8a00640 1 -- 172.21.15.28:0/2170242031 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5c400cde0 con 0x7fa5d4107dd0 2024-05-12T22:14:38.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.762+0000 7fa5d8a00640 1 -- 172.21.15.28:0/2170242031 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa5c40035f0 con 0x7fa5d4107dd0 2024-05-12T22:14:38.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.762+0000 7fa5d8a00640 1 -- 172.21.15.28:0/2170242031 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5c4010690 con 0x7fa5d4107dd0 2024-05-12T22:14:38.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.762+0000 7fa5d3e00640 1 -- 172.21.15.28:0/2170242031 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5d41a37e0 con 0x7fa5d4107dd0 2024-05-12T22:14:38.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.762+0000 7fa5d3e00640 1 -- 172.21.15.28:0/2170242031 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5d41ab070 con 0x7fa5d4107dd0 2024-05-12T22:14:38.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.765+0000 7fa5dcbc9740 1 Processor -- start 2024-05-12T22:14:38.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.765+0000 7fa5dcbc9740 1 -- start start 2024-05-12T22:14:38.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.765+0000 7fa5dcbc9740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x559316e84b60 0x559316e84f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:38.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.766+0000 7fa5dcbc9740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559316e85480 0x559316e8f610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:38.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.766+0000 7fa5dcbc9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559316e8fb50 0x559316e91f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:38.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.766+0000 7fa5dcbc9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x559316de9a30 con 0x559316e8fb50 2024-05-12T22:14:38.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.766+0000 7fa5dcbc9740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x559316d91700 con 0x559316e84b60 2024-05-12T22:14:38.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.766+0000 7fa5dcbc9740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x559316de59f0 con 0x559316e85480 2024-05-12T22:14:38.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.766+0000 7fa5d8a00640 1 -- 172.21.15.28:0/2170242031 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7fa5c40037a0 con 0x7fa5d4107dd0 2024-05-12T22:14:38.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.766+0000 7fa5cbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559316e8fb50 0x559316e91f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:38.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.766+0000 7fa5d3400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x559316e84b60 0x559316e84f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:38.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.766+0000 7fa5d3400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x559316e84b60 0x559316e84f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:49684/0 (socket says 172.21.15.28:49684) 2024-05-12T22:14:38.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.766+0000 7fa5d3400640 1 -- 172.21.15.28:0/1246913359 learned_addr learned my addr 172.21.15.28:0/1246913359 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:14:38.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.766+0000 7fa5d2a00640 1 --2- 172.21.15.28:0/1246913359 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559316e85480 0x559316e8f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:38.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.766+0000 7fa5d3400640 1 -- 172.21.15.28:0/1246913359 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559316e85480 msgr2=0x559316e8f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:14:38.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.766+0000 7fa5d3400640 1 --2- 172.21.15.28:0/1246913359 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559316e85480 0x559316e8f610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:38.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.766+0000 7fa5d3400640 1 -- 172.21.15.28:0/1246913359 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559316e8fb50 msgr2=0x559316e91f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:14:38.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.766+0000 7fa5d3400640 1 --2- 172.21.15.28:0/1246913359 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559316e8fb50 0x559316e91f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:38.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.766+0000 7fa5d3400640 1 -- 172.21.15.28:0/1246913359 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559316afd3f0 con 0x559316e84b60 2024-05-12T22:14:38.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.767+0000 7fa5d3400640 1 --2- 172.21.15.28:0/1246913359 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x559316e84b60 0x559316e84f40 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fa5cc010860 tx=0x7fa5cc010d20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ee73269a5a0dfee3 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:14:38.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.767+0000 7fa5c3e00640 1 -- 172.21.15.28:0/1246913359 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5cc003b70 con 0x559316e84b60 2024-05-12T22:14:38.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.767+0000 7fa5c3e00640 1 -- 172.21.15.28:0/1246913359 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa5cc015070 con 0x559316e84b60 2024-05-12T22:14:38.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.767+0000 7fa5c3e00640 1 -- 172.21.15.28:0/1246913359 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5cc0046e0 con 0x559316e84b60 2024-05-12T22:14:38.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.768+0000 7fa5d8a00640 1 --2- 172.21.15.28:0/2170242031 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fa5b80a2d80 0x7fa5b80a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:38.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.768+0000 7fa5dcbc9740 1 -- 172.21.15.28:0/1246913359 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x559316e84b60 msgr2=0x559316e84f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:14:38.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.768+0000 7fa5dcbc9740 1 --2- 172.21.15.28:0/1246913359 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x559316e84b60 0x559316e84f40 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fa5cc010860 tx=0x7fa5cc010d20 comp rx=0 tx=0).stop 2024-05-12T22:14:38.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.768+0000 7fa5dcbc9740 1 -- 172.21.15.28:0/1246913359 shutdown_connections 2024-05-12T22:14:38.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.768+0000 7fa5dcbc9740 1 --2- 172.21.15.28:0/1246913359 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559316e8fb50 0x559316e91f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:38.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.768+0000 7fa5dcbc9740 1 --2- 172.21.15.28:0/1246913359 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559316e85480 0x559316e8f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:38.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.768+0000 7fa5dcbc9740 1 --2- 172.21.15.28:0/1246913359 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x559316e84b60 0x559316e84f40 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:38.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.768+0000 7fa5d8a00640 1 -- 172.21.15.28:0/2170242031 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7fa5c40cf620 con 0x7fa5d4107dd0 2024-05-12T22:14:38.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.768+0000 7fa5dcbc9740 1 -- 172.21.15.28:0/1246913359 >> 172.21.15.28:0/1246913359 conn(0x559316e80e80 msgr2=0x559316e81260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:14:38.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.768+0000 7fa5d2a00640 1 --2- 172.21.15.28:0/2170242031 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fa5b80a2d80 0x7fa5b80a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:38.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.768+0000 7fa5dcbc9740 1 -- 172.21.15.28:0/1246913359 shutdown_connections 2024-05-12T22:14:38.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.768+0000 7fa5dcbc9740 1 -- 172.21.15.28:0/1246913359 wait complete. 2024-05-12T22:14:38.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.769+0000 7fa5dcbc9740 1 -- start start 2024-05-12T22:14:38.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.769+0000 7fa5d2a00640 1 --2- 172.21.15.28:0/2170242031 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fa5b80a2d80 0x7fa5b80a5240 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fa5c40cce50 tx=0x7fa5c4000bc0 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:14:38.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.769+0000 7fa5dcbc9740 1 Processor -- start 2024-05-12T22:14:38.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.769+0000 7fa5dcbc9740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559316e84b60 0x559316e8cb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:38.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.769+0000 7fa5dcbc9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559316e85480 0x559316e8d050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:38.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.769+0000 7fa5dcbc9740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x559316e8fb50 0x559316e8d590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:38.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.769+0000 7fa5dcbc9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x559316e92940 con 0x559316e85480 2024-05-12T22:14:38.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.769+0000 7fa5dcbc9740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x559316da4a70 con 0x559316e8fb50 2024-05-12T22:14:38.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.769+0000 7fa5dcbc9740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x559316e92ac0 con 0x559316e84b60 2024-05-12T22:14:38.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.769+0000 7fa5dcbc9740 10 client.0 _ll_register_callbacks cb 0x559316d87ba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:14:38.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.770+0000 7fa5cbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559316e85480 0x559316e8d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:38.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.770+0000 7fa5cbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559316e85480 0x559316e8d050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38808/0 (socket says 172.21.15.28:38808) 2024-05-12T22:14:38.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.770+0000 7fa5cbe00640 1 -- 172.21.15.28:0/3957499277 learned_addr learned my addr 172.21.15.28:0/3957499277 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:14:38.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.770+0000 7fa5d3400640 1 --2- 172.21.15.28:0/3957499277 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x559316e8fb50 0x559316e8d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:38.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.770+0000 7fa5d3400640 1 --2- 172.21.15.28:0/3957499277 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559316e84b60 0x559316e8cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:38.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.770+0000 7fa5cbe00640 1 -- 172.21.15.28:0/3957499277 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559316e84b60 msgr2=0x559316e8cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:14:38.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.770+0000 7fa5cbe00640 1 --2- 172.21.15.28:0/3957499277 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559316e84b60 0x559316e8cb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:38.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.770+0000 7fa5cbe00640 1 -- 172.21.15.28:0/3957499277 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x559316e8fb50 msgr2=0x559316e8d590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:14:38.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.770+0000 7fa5cbe00640 1 --2- 172.21.15.28:0/3957499277 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x559316e8fb50 0x559316e8d590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:38.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.770+0000 7fa5cbe00640 1 -- 172.21.15.28:0/3957499277 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559316bc77d0 con 0x559316e85480 2024-05-12T22:14:38.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.770+0000 7fa5d3400640 1 --2- 172.21.15.28:0/3957499277 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x559316e8fb50 0x559316e8d590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:14:38.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.771+0000 7fa5cbe00640 1 --2- 172.21.15.28:0/3957499277 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559316e85480 0x559316e8d050 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fa5bc009860 tx=0x7fa5bc005270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf327121347bfb8f server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:14:38.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.771+0000 7fa5c0c00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-12T22:14:38.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.771+0000 7fa5c0c00640 1 -- 172.21.15.28:0/3957499277 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5bc030e60 con 0x559316e85480 2024-05-12T22:14:38.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.771+0000 7fa5c0c00640 1 -- 172.21.15.28:0/3957499277 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7fa5bc0055f0 con 0x559316e85480 2024-05-12T22:14:38.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.771+0000 7fa5dcbc9740 1 -- 172.21.15.28:0/3957499277 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa5cc015070 con 0x559316e85480 2024-05-12T22:14:38.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.771+0000 7fa5c0c00640 1 -- 172.21.15.28:0/3957499277 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fa5bc009450 con 0x559316e85480 2024-05-12T22:14:38.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.771+0000 7fa5dcbc9740 10 client.16371 fetch_fsmap learned FSMap version 26 2024-05-12T22:14:38.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.771+0000 7fa5dcbc9740 1 -- 172.21.15.28:0/3957499277 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x559316e9ec10 con 0x559316e85480 2024-05-12T22:14:38.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.772+0000 7fa5c0c00640 1 -- 172.21.15.28:0/3957499277 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa5cc015070 con 0x559316e85480 2024-05-12T22:14:38.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.772+0000 7fa5c0c00640 1 -- 172.21.15.28:0/3957499277 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fa5bc0035f0 con 0x559316e85480 2024-05-12T22:14:38.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.772+0000 7fa5dcbc9740 10 client.16371 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:14:38.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.772+0000 7fa5dcbc9740 10 client.16371 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.g 2024-05-12T22:14:38.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.772+0000 7fa5dcbc9740 20 client.16371 populate_metadata read hostname 'smithi028' 2024-05-12T22:14:38.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.772+0000 7fa5dcbc9740 1 --2- 172.21.15.28:0/3957499277 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x559316ea0070 0x559316ea2490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:38.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.772+0000 7fa5dcbc9740 4 client.16371 mds_command: new command op to 24532 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-12T22:14:38.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.772+0000 7fa5dcbc9740 1 -- 172.21.15.28:0/3957499277 --> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559316df7160 con 0x559316ea0070 2024-05-12T22:14:38.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.772+0000 7fa5d3400640 1 --2- 172.21.15.28:0/3957499277 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x559316ea0070 0x559316ea2490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:38.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.773+0000 7fa5d3400640 1 --2- 172.21.15.28:0/3957499277 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x559316ea0070 0x559316ea2490 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=801bd83347d69a server_cookie=c64d14a643dc45e1 in_seq=0 out_seq=0 2024-05-12T22:14:38.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.773+0000 7fa5c0c00640 10 client.16371 ms_handle_connect on v2:172.21.15.28:6838/123892158 2024-05-12T22:14:38.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.774+0000 7fa5c0c00640 1 -- 172.21.15.28:0/3957499277 <== mds.2 v2:172.21.15.28:6838/123892158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559316df7160 con 0x559316ea0070 2024-05-12T22:14:38.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.774+0000 7fa5c0c00640 10 client.16371 handle_command_reply: tid=0 2024-05-12T22:14:38.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.774+0000 7fa5dcbc9740 1 client.16371 shutdown 2024-05-12T22:14:38.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.774+0000 7fa5dcbc9740 1 -- 172.21.15.28:0/3957499277 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559316e85480 msgr2=0x559316e8d050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:14:38.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.774+0000 7fa5dcbc9740 1 --2- 172.21.15.28:0/3957499277 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559316e85480 0x559316e8d050 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fa5bc009860 tx=0x7fa5bc005270 comp rx=0 tx=0).stop 2024-05-12T22:14:38.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.774+0000 7fa5dcbc9740 1 -- 172.21.15.28:0/3957499277 shutdown_connections 2024-05-12T22:14:38.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.774+0000 7fa5dcbc9740 1 --2- 172.21.15.28:0/3957499277 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x559316ea0070 0x559316ea2490 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:38.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.774+0000 7fa5dcbc9740 1 --2- 172.21.15.28:0/3957499277 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x559316e8fb50 0x559316e8d590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:38.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.774+0000 7fa5dcbc9740 1 --2- 172.21.15.28:0/3957499277 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559316e85480 0x559316e8d050 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:38.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.774+0000 7fa5dcbc9740 1 --2- 172.21.15.28:0/3957499277 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559316e84b60 0x559316e8cb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:38.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.774+0000 7fa5c0c00640 0 client.16371 ms_handle_reset on v2:172.21.15.28:6838/123892158 2024-05-12T22:14:38.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.774+0000 7fa5dcbc9740 1 -- 172.21.15.28:0/3957499277 >> 172.21.15.28:0/3957499277 conn(0x559316e80e80 msgr2=0x559316e86c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:14:38.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.774+0000 7fa5dcbc9740 1 -- 172.21.15.28:0/3957499277 shutdown_connections 2024-05-12T22:14:38.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.774+0000 7fa5dcbc9740 1 -- 172.21.15.28:0/3957499277 wait complete. 2024-05-12T22:14:38.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.774+0000 7fa5dcbc9740 20 client.16371 trim_cache size 0 max 16384 2024-05-12T22:14:38.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.789+0000 7fa5dcbc9740 1 Processor -- start 2024-05-12T22:14:38.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.789+0000 7fa5dcbc9740 1 -- start start 2024-05-12T22:14:38.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.789+0000 7fa5dcbc9740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559316e84b60 0x559316e8d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:38.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.789+0000 7fa5dcbc9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559316e85480 0x559316ea7d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:38.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.789+0000 7fa5dcbc9740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x559316e8fb50 0x559316ea82a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:38.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.789+0000 7fa5dcbc9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa5d41b6120 con 0x559316e85480 2024-05-12T22:14:38.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.789+0000 7fa5dcbc9740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fa5d41b39f0 con 0x559316e8fb50 2024-05-12T22:14:38.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.789+0000 7fa5dcbc9740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa5d41b1840 con 0x559316e84b60 2024-05-12T22:14:38.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.790+0000 7fa5cbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559316e85480 0x559316ea7d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:38.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.790+0000 7fa5cbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559316e85480 0x559316ea7d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38814/0 (socket says 172.21.15.28:38814) 2024-05-12T22:14:38.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.790+0000 7fa5cbe00640 1 -- 172.21.15.28:0/417708974 learned_addr learned my addr 172.21.15.28:0/417708974 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:14:38.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.790+0000 7fa5d3400640 1 --2- 172.21.15.28:0/417708974 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559316e84b60 0x559316e8d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:38.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.790+0000 7fa5d3400640 1 --2- 172.21.15.28:0/417708974 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x559316e8fb50 0x559316ea82a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:38.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.790+0000 7fa5cbe00640 1 -- 172.21.15.28:0/417708974 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559316e84b60 msgr2=0x559316e8d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:14:38.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.790+0000 7fa5cbe00640 1 --2- 172.21.15.28:0/417708974 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559316e84b60 0x559316e8d3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:38.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.790+0000 7fa5cbe00640 1 -- 172.21.15.28:0/417708974 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x559316e8fb50 msgr2=0x559316ea82a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:14:38.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.790+0000 7fa5cbe00640 1 --2- 172.21.15.28:0/417708974 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x559316e8fb50 0x559316ea82a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:38.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.790+0000 7fa5cbe00640 1 -- 172.21.15.28:0/417708974 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559316e87920 con 0x559316e85480 2024-05-12T22:14:38.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.790+0000 7fa5d3400640 1 --2- 172.21.15.28:0/417708974 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x559316e8fb50 0x559316ea82a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-12T22:14:38.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.790+0000 7fa5cbe00640 1 --2- 172.21.15.28:0/417708974 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559316e85480 0x559316ea7d60 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fa5bc02a8f0 tx=0x7fa5bc001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1ad3e1ee2e149f0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:14:38.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.791+0000 7fa5c3400640 1 -- 172.21.15.28:0/417708974 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5bc0055f0 con 0x559316e85480 2024-05-12T22:14:38.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.791+0000 7fa5c3400640 1 -- 172.21.15.28:0/417708974 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa5bc004490 con 0x559316e85480 2024-05-12T22:14:38.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.791+0000 7fa5c3400640 1 -- 172.21.15.28:0/417708974 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5bc004920 con 0x559316e85480 2024-05-12T22:14:38.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.791+0000 7fa5dcbc9740 1 -- 172.21.15.28:0/417708974 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559316e85480 msgr2=0x559316ea7d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:14:38.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.791+0000 7fa5dcbc9740 1 --2- 172.21.15.28:0/417708974 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559316e85480 0x559316ea7d60 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fa5bc02a8f0 tx=0x7fa5bc001e70 comp rx=0 tx=0).stop 2024-05-12T22:14:38.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.791+0000 7fa5dcbc9740 1 -- 172.21.15.28:0/417708974 shutdown_connections 2024-05-12T22:14:38.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.791+0000 7fa5dcbc9740 1 --2- 172.21.15.28:0/417708974 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x559316e8fb50 0x559316ea82a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:38.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.791+0000 7fa5dcbc9740 1 --2- 172.21.15.28:0/417708974 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559316e85480 0x559316ea7d60 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:38.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.791+0000 7fa5dcbc9740 1 --2- 172.21.15.28:0/417708974 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559316e84b60 0x559316e8d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:38.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.791+0000 7fa5dcbc9740 1 -- 172.21.15.28:0/417708974 >> 172.21.15.28:0/417708974 conn(0x559316e80e80 msgr2=0x559316e7f770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:14:38.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.792+0000 7fa5dcbc9740 1 -- 172.21.15.28:0/417708974 shutdown_connections 2024-05-12T22:14:38.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.792+0000 7fa5dcbc9740 1 -- 172.21.15.28:0/417708974 wait complete. 2024-05-12T22:14:38.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.792+0000 7fa5dcbc9740 1 -- start start 2024-05-12T22:14:38.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.792+0000 7fa5dcbc9740 1 Processor -- start 2024-05-12T22:14:38.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.793+0000 7fa5dcbc9740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x559316e84b60 0x559316e89960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:38.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.793+0000 7fa5dcbc9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559316e85480 0x559316e89ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:38.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.793+0000 7fa5dcbc9740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559316e8fb50 0x559316e8a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:38.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.793+0000 7fa5dcbc9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa5d41b1eb0 con 0x559316e85480 2024-05-12T22:14:38.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.793+0000 7fa5dcbc9740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fa5d41b3320 con 0x559316e84b60 2024-05-12T22:14:38.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.793+0000 7fa5dcbc9740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa5d41b2580 con 0x559316e8fb50 2024-05-12T22:14:38.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.793+0000 7fa5dcbc9740 10 client.0 _ll_register_callbacks cb 0x7fa5bc0307f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:14:38.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.793+0000 7fa5cbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559316e85480 0x559316e89ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:38.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.793+0000 7fa5d3400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x559316e84b60 0x559316e89960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:38.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.793+0000 7fa5d3400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x559316e84b60 0x559316e89960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:49704/0 (socket says 172.21.15.28:49704) 2024-05-12T22:14:38.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.793+0000 7fa5d3400640 1 -- 172.21.15.28:0/698676263 learned_addr learned my addr 172.21.15.28:0/698676263 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:14:38.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.793+0000 7fa5d3400640 1 --2- 172.21.15.28:0/698676263 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559316e8fb50 0x559316e8a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:38.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.794+0000 7fa5d3400640 1 -- 172.21.15.28:0/698676263 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559316e8fb50 msgr2=0x559316e8a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:14:38.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.794+0000 7fa5d3400640 1 --2- 172.21.15.28:0/698676263 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559316e8fb50 0x559316e8a3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:38.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.794+0000 7fa5d3400640 1 -- 172.21.15.28:0/698676263 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559316e85480 msgr2=0x559316e89ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:14:38.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.794+0000 7fa5d3400640 1 --2- 172.21.15.28:0/698676263 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559316e85480 0x559316e89ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:38.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.794+0000 7fa5d3400640 1 -- 172.21.15.28:0/698676263 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559316f05700 con 0x559316e84b60 2024-05-12T22:14:38.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.794+0000 7fa5d3400640 1 --2- 172.21.15.28:0/698676263 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x559316e84b60 0x559316e89960 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fa5cc004450 tx=0x7fa5bc030760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=907c8f3e8150c351 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:14:38.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.794+0000 7fa5c0c00640 10 client.0 ms_handle_connect on v2:172.21.15.86:3300/0 2024-05-12T22:14:38.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.794+0000 7fa5c0c00640 1 -- 172.21.15.28:0/698676263 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5cc023040 con 0x559316e84b60 2024-05-12T22:14:38.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.794+0000 7fa5dcbc9740 1 -- 172.21.15.28:0/698676263 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa5bc004490 con 0x559316e84b60 2024-05-12T22:14:38.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.794+0000 7fa5c0c00640 1 -- 172.21.15.28:0/698676263 <== mon.2 v2:172.21.15.86:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7fa5cc02ee00 con 0x559316e84b60 2024-05-12T22:14:38.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.794+0000 7fa5c0c00640 1 -- 172.21.15.28:0/698676263 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fa5cc003af0 con 0x559316e84b60 2024-05-12T22:14:38.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.794+0000 7fa5dcbc9740 10 client.26504 fetch_fsmap learned FSMap version 26 2024-05-12T22:14:38.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.794+0000 7fa5dcbc9740 1 -- 172.21.15.28:0/698676263 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x559316f06950 con 0x559316e84b60 2024-05-12T22:14:38.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.795+0000 7fa5c0c00640 1 -- 172.21.15.28:0/698676263 <== mon.2 v2:172.21.15.86:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa5bc004490 con 0x559316e84b60 2024-05-12T22:14:38.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.795+0000 7fa5c0c00640 1 -- 172.21.15.28:0/698676263 <== mon.2 v2:172.21.15.86:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fa5cc02d0d0 con 0x559316e84b60 2024-05-12T22:14:38.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.795+0000 7fa5dcbc9740 10 client.26504 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:14:38.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.795+0000 7fa5dcbc9740 10 client.26504 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.g 2024-05-12T22:14:38.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.795+0000 7fa5dcbc9740 20 client.26504 populate_metadata read hostname 'smithi028' 2024-05-12T22:14:38.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.795+0000 7fa5dcbc9740 1 --2- 172.21.15.28:0/698676263 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x559316f06eb0 0x559316f092d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:38.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.795+0000 7fa5dcbc9740 4 client.26504 mds_command: new command op to 24532 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-12T22:14:38.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.795+0000 7fa5dcbc9740 1 -- 172.21.15.28:0/698676263 --> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559316df14b0 con 0x559316f06eb0 2024-05-12T22:14:38.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.795+0000 7fa5cbe00640 1 --2- 172.21.15.28:0/698676263 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x559316f06eb0 0x559316f092d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:38.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.796+0000 7fa5cbe00640 1 --2- 172.21.15.28:0/698676263 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x559316f06eb0 0x559316f092d0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ac852f8d85d7e58e server_cookie=974bb49a794faae5 in_seq=0 out_seq=0 2024-05-12T22:14:38.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.796+0000 7fa5c0c00640 10 client.26504 ms_handle_connect on v2:172.21.15.28:6838/123892158 2024-05-12T22:14:38.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.796+0000 7fa5c0c00640 1 -- 172.21.15.28:0/698676263 <== mds.2 v2:172.21.15.28:6838/123892158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x559316df14b0 con 0x559316f06eb0 2024-05-12T22:14:38.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.796+0000 7fa5c0c00640 10 client.26504 handle_command_reply: tid=0 2024-05-12T22:14:38.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.796+0000 7fa5dcbc9740 1 client.26504 shutdown 2024-05-12T22:14:38.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.796+0000 7fa5dcbc9740 1 -- 172.21.15.28:0/698676263 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x559316e84b60 msgr2=0x559316e89960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:14:38.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.796+0000 7fa5dcbc9740 1 --2- 172.21.15.28:0/698676263 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x559316e84b60 0x559316e89960 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fa5cc004450 tx=0x7fa5bc030760 comp rx=0 tx=0).stop 2024-05-12T22:14:38.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.797+0000 7fa5dcbc9740 1 -- 172.21.15.28:0/698676263 shutdown_connections 2024-05-12T22:14:38.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.797+0000 7fa5dcbc9740 1 --2- 172.21.15.28:0/698676263 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559316e8fb50 0x559316e8a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:38.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.797+0000 7fa5dcbc9740 1 --2- 172.21.15.28:0/698676263 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559316e85480 0x559316e89ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:38.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.797+0000 7fa5dcbc9740 1 --2- 172.21.15.28:0/698676263 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x559316f06eb0 0x559316f092d0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:38.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.797+0000 7fa5dcbc9740 1 --2- 172.21.15.28:0/698676263 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x559316e84b60 0x559316e89960 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:38.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.797+0000 7fa5dcbc9740 1 -- 172.21.15.28:0/698676263 >> 172.21.15.28:0/698676263 conn(0x559316e80e80 msgr2=0x559316e80000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:14:38.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.797+0000 7fa5c0c00640 0 client.26504 ms_handle_reset on v2:172.21.15.28:6838/123892158 2024-05-12T22:14:38.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.797+0000 7fa5dcbc9740 1 -- 172.21.15.28:0/698676263 shutdown_connections 2024-05-12T22:14:38.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.797+0000 7fa5dcbc9740 1 -- 172.21.15.28:0/698676263 wait complete. 2024-05-12T22:14:38.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.797+0000 7fa5dcbc9740 20 client.26504 trim_cache size 0 max 16384 2024-05-12T22:14:38.806 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-05-12T22:14:38.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.797+0000 7fa5d3e00640 1 -- 172.21.15.28:0/2170242031 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fa5b80a2d80 msgr2=0x7fa5b80a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:14:38.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.797+0000 7fa5d3e00640 1 --2- 172.21.15.28:0/2170242031 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fa5b80a2d80 0x7fa5b80a5240 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fa5c40cce50 tx=0x7fa5c4000bc0 comp rx=0 tx=0).stop 2024-05-12T22:14:38.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.797+0000 7fa5d3e00640 1 -- 172.21.15.28:0/2170242031 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa5d4107dd0 msgr2=0x7fa5d419e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:14:38.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.797+0000 7fa5d3e00640 1 --2- 172.21.15.28:0/2170242031 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa5d4107dd0 0x7fa5d419e500 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fa5c400e9c0 tx=0x7fa5c400ee80 comp rx=0 tx=0).stop 2024-05-12T22:14:38.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.798+0000 7fa5d3e00640 1 -- 172.21.15.28:0/2170242031 shutdown_connections 2024-05-12T22:14:38.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.798+0000 7fa5d3e00640 1 --2- 172.21.15.28:0/2170242031 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fa5b80a2d80 0x7fa5b80a5240 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:38.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.798+0000 7fa5d3e00640 1 --2- 172.21.15.28:0/2170242031 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5d410ddf0 0x7fa5d41a2dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:38.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.798+0000 7fa5d3e00640 1 --2- 172.21.15.28:0/2170242031 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa5d4108770 0x7fa5d419ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:38.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.798+0000 7fa5d3e00640 1 --2- 172.21.15.28:0/2170242031 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa5d4107dd0 0x7fa5d419e500 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:38.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.798+0000 7fa5d3e00640 1 -- 172.21.15.28:0/2170242031 >> 172.21.15.28:0/2170242031 conn(0x7fa5d4077650 msgr2=0x7fa5d4078580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:14:38.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.798+0000 7fa5d3e00640 1 -- 172.21.15.28:0/2170242031 shutdown_connections 2024-05-12T22:14:38.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:38.798+0000 7fa5d3e00640 1 -- 172.21.15.28:0/2170242031 wait complete. 2024-05-12T22:14:38.816 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-05-12T22:14:39.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.009+0000 7fa6b0e00640 1 -- 172.21.15.28:0/4262111879 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa6ac073bd0 msgr2=0x7fa6ac073fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:14:39.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.009+0000 7fa6b0e00640 1 --2- 172.21.15.28:0/4262111879 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa6ac073bd0 0x7fa6ac073fb0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fa69c009900 tx=0x7fa69c02a480 comp rx=0 tx=0).stop 2024-05-12T22:14:39.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.009+0000 7fa6b0e00640 1 -- 172.21.15.28:0/4262111879 shutdown_connections 2024-05-12T22:14:39.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.009+0000 7fa6b0e00640 1 --2- 172.21.15.28:0/4262111879 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6ac1126f0 0x7fa6ac114ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:39.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.009+0000 7fa6b0e00640 1 --2- 172.21.15.28:0/4262111879 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa6ac0744f0 0x7fa6ac1121b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:39.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.009+0000 7fa6b0e00640 1 --2- 172.21.15.28:0/4262111879 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa6ac073bd0 0x7fa6ac073fb0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:39.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.009+0000 7fa6b0e00640 1 -- 172.21.15.28:0/4262111879 >> 172.21.15.28:0/4262111879 conn(0x7fa6ac0fd1d0 msgr2=0x7fa6ac0ff5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:14:39.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.009+0000 7fa6b0e00640 1 -- 172.21.15.28:0/4262111879 shutdown_connections 2024-05-12T22:14:39.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.009+0000 7fa6b0e00640 1 -- 172.21.15.28:0/4262111879 wait complete. 2024-05-12T22:14:39.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.010+0000 7fa6b0e00640 1 Processor -- start 2024-05-12T22:14:39.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.010+0000 7fa6b0e00640 1 -- start start 2024-05-12T22:14:39.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.011+0000 7fa6b0e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6ac073bd0 0x7fa6ac19e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:39.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.011+0000 7fa6b0e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa6ac0744f0 0x7fa6ac19ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:39.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.011+0000 7fa6b0e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa6ac1126f0 0x7fa6ac1a2f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:39.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.011+0000 7fa6b0e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa6ac1180e0 con 0x7fa6ac073bd0 2024-05-12T22:14:39.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.011+0000 7fa6b0e00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fa6ac117f60 con 0x7fa6ac0744f0 2024-05-12T22:14:39.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.011+0000 7fa6b0e00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa6ac118260 con 0x7fa6ac1126f0 2024-05-12T22:14:39.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.011+0000 7fa6ab400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6ac073bd0 0x7fa6ac19e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:39.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.011+0000 7fa6ab400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6ac073bd0 0x7fa6ac19e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38842/0 (socket says 172.21.15.28:38842) 2024-05-12T22:14:39.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.011+0000 7fa6ab400640 1 -- 172.21.15.28:0/636208150 learned_addr learned my addr 172.21.15.28:0/636208150 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:14:39.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.011+0000 7fa6abe00640 1 --2- 172.21.15.28:0/636208150 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa6ac1126f0 0x7fa6ac1a2f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:39.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.011+0000 7fa6a3e00640 1 --2- 172.21.15.28:0/636208150 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa6ac0744f0 0x7fa6ac19ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:39.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.012+0000 7fa6abe00640 1 -- 172.21.15.28:0/636208150 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa6ac0744f0 msgr2=0x7fa6ac19ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:14:39.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.012+0000 7fa6abe00640 1 --2- 172.21.15.28:0/636208150 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa6ac0744f0 0x7fa6ac19ebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:39.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.012+0000 7fa6abe00640 1 -- 172.21.15.28:0/636208150 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6ac073bd0 msgr2=0x7fa6ac19e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:14:39.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.012+0000 7fa6abe00640 1 --2- 172.21.15.28:0/636208150 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6ac073bd0 0x7fa6ac19e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:39.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.012+0000 7fa6abe00640 1 -- 172.21.15.28:0/636208150 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6ac1a3640 con 0x7fa6ac1126f0 2024-05-12T22:14:39.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.012+0000 7fa6ab400640 1 --2- 172.21.15.28:0/636208150 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6ac073bd0 0x7fa6ac19e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:14:39.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.012+0000 7fa6a3e00640 1 --2- 172.21.15.28:0/636208150 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa6ac0744f0 0x7fa6ac19ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:14:39.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.012+0000 7fa6abe00640 1 --2- 172.21.15.28:0/636208150 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa6ac1126f0 0x7fa6ac1a2f10 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fa6a4005fd0 tx=0x7fa6a400dea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:14:39.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.012+0000 7fa6a8c00640 1 -- 172.21.15.28:0/636208150 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6a400bd40 con 0x7fa6ac1126f0 2024-05-12T22:14:39.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.012+0000 7fa6a8c00640 1 -- 172.21.15.28:0/636208150 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa6a4003e50 con 0x7fa6ac1126f0 2024-05-12T22:14:39.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.012+0000 7fa6b0e00640 1 -- 172.21.15.28:0/636208150 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6ac1a3930 con 0x7fa6ac1126f0 2024-05-12T22:14:39.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.012+0000 7fa6a8c00640 1 -- 172.21.15.28:0/636208150 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6a40106c0 con 0x7fa6ac1126f0 2024-05-12T22:14:39.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.012+0000 7fa6b0e00640 1 -- 172.21.15.28:0/636208150 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6ac1ab1c0 con 0x7fa6ac1126f0 2024-05-12T22:14:39.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.015+0000 7fa6b5250740 1 Processor -- start 2024-05-12T22:14:39.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.016+0000 7fa6a8c00640 1 -- 172.21.15.28:0/636208150 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7fa6a4003970 con 0x7fa6ac1126f0 2024-05-12T22:14:39.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.016+0000 7fa6b5250740 1 -- start start 2024-05-12T22:14:39.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.016+0000 7fa6b5250740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562144f83b60 0x562144f83f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:39.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.016+0000 7fa6b5250740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562144f84480 0x562144f8e610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:39.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.016+0000 7fa6b5250740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562144f8eb50 0x562144f90f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:39.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.016+0000 7fa6b5250740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562144ee8a30 con 0x562144f83b60 2024-05-12T22:14:39.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.016+0000 7fa6b5250740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x562144e90700 con 0x562144f8eb50 2024-05-12T22:14:39.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.016+0000 7fa6b5250740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x562144ee49f0 con 0x562144f84480 2024-05-12T22:14:39.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.017+0000 7fa6a3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562144f83b60 0x562144f83f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:39.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.017+0000 7fa6ab400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562144f84480 0x562144f8e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:39.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.017+0000 7fa6a3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562144f83b60 0x562144f83f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38854/0 (socket says 172.21.15.28:38854) 2024-05-12T22:14:39.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.017+0000 7fa6a3e00640 1 -- 172.21.15.28:0/4051905103 learned_addr learned my addr 172.21.15.28:0/4051905103 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:14:39.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.017+0000 7fa6ab400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562144f84480 0x562144f8e610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:60386/0 (socket says 172.21.15.28:60386) 2024-05-12T22:14:39.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.017+0000 7fa6a3e00640 1 --2- 172.21.15.28:0/4051905103 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562144f8eb50 0x562144f90f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:39.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.017+0000 7fa6ab400640 1 -- 172.21.15.28:0/4051905103 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562144f8eb50 msgr2=0x562144f90f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:14:39.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.017+0000 7fa6ab400640 1 --2- 172.21.15.28:0/4051905103 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562144f8eb50 0x562144f90f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:39.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.017+0000 7fa6ab400640 1 -- 172.21.15.28:0/4051905103 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562144f83b60 msgr2=0x562144f83f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:14:39.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.017+0000 7fa6ab400640 1 --2- 172.21.15.28:0/4051905103 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562144f83b60 0x562144f83f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:39.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.017+0000 7fa6ab400640 1 -- 172.21.15.28:0/4051905103 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562144bfc3f0 con 0x562144f84480 2024-05-12T22:14:39.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.018+0000 7fa6a8c00640 1 --2- 172.21.15.28:0/636208150 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fa6900a2d00 0x7fa6900a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:39.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.018+0000 7fa6ab400640 1 --2- 172.21.15.28:0/4051905103 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562144f84480 0x562144f8e610 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fa69c009230 tx=0x7fa69c005e80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=993bc762625172b2 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:14:39.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.018+0000 7fa6a8c00640 1 -- 172.21.15.28:0/636208150 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7fa6a40cf730 con 0x7fa6ac1126f0 2024-05-12T22:14:39.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.018+0000 7fa6a3e00640 1 --2- 172.21.15.28:0/636208150 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fa6900a2d00 0x7fa6900a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:39.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.018+0000 7fa6a0c00640 1 -- 172.21.15.28:0/4051905103 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa69c040070 con 0x562144f84480 2024-05-12T22:14:39.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.018+0000 7fa6a0c00640 1 -- 172.21.15.28:0/4051905103 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa69c0319f0 con 0x562144f84480 2024-05-12T22:14:39.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.018+0000 7fa6a0c00640 1 -- 172.21.15.28:0/4051905103 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa69c035040 con 0x562144f84480 2024-05-12T22:14:39.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.019+0000 7fa6b5250740 1 -- 172.21.15.28:0/4051905103 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562144f84480 msgr2=0x562144f8e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:14:39.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.019+0000 7fa6a3e00640 1 --2- 172.21.15.28:0/636208150 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fa6900a2d00 0x7fa6900a51c0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x562144f81010 tx=0x7fa694009b30 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:14:39.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.019+0000 7fa6b5250740 1 --2- 172.21.15.28:0/4051905103 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562144f84480 0x562144f8e610 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fa69c009230 tx=0x7fa69c005e80 comp rx=0 tx=0).stop 2024-05-12T22:14:39.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.019+0000 7fa6b5250740 1 -- 172.21.15.28:0/4051905103 shutdown_connections 2024-05-12T22:14:39.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.019+0000 7fa6b5250740 1 --2- 172.21.15.28:0/4051905103 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562144f8eb50 0x562144f90f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:39.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.019+0000 7fa6b5250740 1 --2- 172.21.15.28:0/4051905103 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562144f84480 0x562144f8e610 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:39.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.019+0000 7fa6b5250740 1 --2- 172.21.15.28:0/4051905103 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562144f83b60 0x562144f83f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:39.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.019+0000 7fa6b5250740 1 -- 172.21.15.28:0/4051905103 >> 172.21.15.28:0/4051905103 conn(0x562144f7fe80 msgr2=0x562144f80260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:14:39.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.019+0000 7fa6b5250740 1 -- 172.21.15.28:0/4051905103 shutdown_connections 2024-05-12T22:14:39.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.019+0000 7fa6b5250740 1 -- 172.21.15.28:0/4051905103 wait complete. 2024-05-12T22:14:39.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.020+0000 7fa6b5250740 1 -- start start 2024-05-12T22:14:39.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.020+0000 7fa6b5250740 1 Processor -- start 2024-05-12T22:14:39.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.020+0000 7fa6b5250740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562144f83b60 0x562144f8bb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:39.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.020+0000 7fa6b5250740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562144f84480 0x562144f8c070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:39.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.020+0000 7fa6b5250740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562144f8eb50 0x562144f8c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:39.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.020+0000 7fa6b5250740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562144f918e0 con 0x562144f8eb50 2024-05-12T22:14:39.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.020+0000 7fa6b5250740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x562144ea3a70 con 0x562144f83b60 2024-05-12T22:14:39.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.020+0000 7fa6b5250740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x562144f91a60 con 0x562144f84480 2024-05-12T22:14:39.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.020+0000 7fa6b5250740 10 client.0 _ll_register_callbacks cb 0x562144e86ba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:14:39.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.021+0000 7fa6a3e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562144f84480 0x562144f8c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:39.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.021+0000 7fa6abe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562144f8eb50 0x562144f8c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:39.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.021+0000 7fa6ab400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562144f83b60 0x562144f8bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:39.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.021+0000 7fa6ab400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562144f83b60 0x562144f8bb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:49736/0 (socket says 172.21.15.28:49736) 2024-05-12T22:14:39.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.021+0000 7fa6ab400640 1 -- 172.21.15.28:0/3316478725 learned_addr learned my addr 172.21.15.28:0/3316478725 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:14:39.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.021+0000 7fa6a3e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562144f84480 0x562144f8c070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:60394/0 (socket says 172.21.15.28:60394) 2024-05-12T22:14:39.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.021+0000 7fa6abe00640 1 -- 172.21.15.28:0/3316478725 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562144f84480 msgr2=0x562144f8c070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:14:39.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.021+0000 7fa6abe00640 1 --2- 172.21.15.28:0/3316478725 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562144f84480 0x562144f8c070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:39.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.021+0000 7fa6abe00640 1 -- 172.21.15.28:0/3316478725 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562144f83b60 msgr2=0x562144f8bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:14:39.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.021+0000 7fa6abe00640 1 --2- 172.21.15.28:0/3316478725 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562144f83b60 0x562144f8bb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:39.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.021+0000 7fa6abe00640 1 -- 172.21.15.28:0/3316478725 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562144cc67d0 con 0x562144f8eb50 2024-05-12T22:14:39.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.021+0000 7fa6ab400640 1 --2- 172.21.15.28:0/3316478725 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562144f83b60 0x562144f8bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:14:39.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.021+0000 7fa6abe00640 1 --2- 172.21.15.28:0/3316478725 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562144f8eb50 0x562144f8c5b0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fa6a40cce50 tx=0x7fa6a409b6b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58ca6a273ed64798 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:14:39.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.021+0000 7fa699600640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-12T22:14:39.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.022+0000 7fa699600640 1 -- 172.21.15.28:0/3316478725 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6a40a4890 con 0x562144f8eb50 2024-05-12T22:14:39.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.022+0000 7fa699600640 1 -- 172.21.15.28:0/3316478725 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7fa6a40cd0d0 con 0x562144f8eb50 2024-05-12T22:14:39.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.022+0000 7fa6b5250740 1 -- 172.21.15.28:0/3316478725 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa69c0319f0 con 0x562144f8eb50 2024-05-12T22:14:39.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.022+0000 7fa699600640 1 -- 172.21.15.28:0/3316478725 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fa6a4003c60 con 0x562144f8eb50 2024-05-12T22:14:39.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.022+0000 7fa6b5250740 10 client.16389 fetch_fsmap learned FSMap version 26 2024-05-12T22:14:39.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.022+0000 7fa6b5250740 1 -- 172.21.15.28:0/3316478725 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x562144f9dc30 con 0x562144f8eb50 2024-05-12T22:14:39.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.022+0000 7fa699600640 1 -- 172.21.15.28:0/3316478725 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa69c0319f0 con 0x562144f8eb50 2024-05-12T22:14:39.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.022+0000 7fa699600640 1 -- 172.21.15.28:0/3316478725 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fa6a4024060 con 0x562144f8eb50 2024-05-12T22:14:39.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.023+0000 7fa6b5250740 10 client.16389 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:14:39.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.023+0000 7fa6b5250740 10 client.16389 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-05-12T22:14:39.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.023+0000 7fa6b5250740 20 client.16389 populate_metadata read hostname 'smithi028' 2024-05-12T22:14:39.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.023+0000 7fa6b5250740 1 --2- 172.21.15.28:0/3316478725 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x562144f9f140 0x562144fa1570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:39.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.023+0000 7fa6b5250740 4 client.16389 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-12T22:14:39.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.023+0000 7fa6b5250740 1 -- 172.21.15.28:0/3316478725 --> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562144ef6160 con 0x562144f9f140 2024-05-12T22:14:39.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.023+0000 7fa6ab400640 1 --2- 172.21.15.28:0/3316478725 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x562144f9f140 0x562144fa1570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:39.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.023+0000 7fa6ab400640 1 --2- 172.21.15.28:0/3316478725 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x562144f9f140 0x562144fa1570 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=9471ec0034b4dd75 server_cookie=6806c433885cc456 in_seq=0 out_seq=0 2024-05-12T22:14:39.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.023+0000 7fa699600640 10 client.16389 ms_handle_connect on v2:172.21.15.28:6840/761183870 2024-05-12T22:14:39.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.024+0000 7fa699600640 1 -- 172.21.15.28:0/3316478725 <== mds.1 v2:172.21.15.28:6840/761183870 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562144ef6160 con 0x562144f9f140 2024-05-12T22:14:39.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.024+0000 7fa699600640 10 client.16389 handle_command_reply: tid=0 2024-05-12T22:14:39.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.024+0000 7fa6b5250740 1 client.16389 shutdown 2024-05-12T22:14:39.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.025+0000 7fa6b5250740 1 -- 172.21.15.28:0/3316478725 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562144f8eb50 msgr2=0x562144f8c5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:14:39.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.025+0000 7fa6b5250740 1 --2- 172.21.15.28:0/3316478725 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562144f8eb50 0x562144f8c5b0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fa6a40cce50 tx=0x7fa6a409b6b0 comp rx=0 tx=0).stop 2024-05-12T22:14:39.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.025+0000 7fa6b5250740 1 -- 172.21.15.28:0/3316478725 shutdown_connections 2024-05-12T22:14:39.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.025+0000 7fa6b5250740 1 --2- 172.21.15.28:0/3316478725 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x562144f9f140 0x562144fa1570 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:39.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.025+0000 7fa6b5250740 1 --2- 172.21.15.28:0/3316478725 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562144f8eb50 0x562144f8c5b0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:39.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.025+0000 7fa6b5250740 1 --2- 172.21.15.28:0/3316478725 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562144f84480 0x562144f8c070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:39.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.025+0000 7fa6b5250740 1 --2- 172.21.15.28:0/3316478725 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562144f83b60 0x562144f8bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:39.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.025+0000 7fa699600640 0 client.16389 ms_handle_reset on v2:172.21.15.28:6840/761183870 2024-05-12T22:14:39.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.025+0000 7fa6b5250740 1 -- 172.21.15.28:0/3316478725 >> 172.21.15.28:0/3316478725 conn(0x562144f7fe80 msgr2=0x562144f85c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:14:39.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.025+0000 7fa6b5250740 1 -- 172.21.15.28:0/3316478725 shutdown_connections 2024-05-12T22:14:39.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.025+0000 7fa6b5250740 1 -- 172.21.15.28:0/3316478725 wait complete. 2024-05-12T22:14:39.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.025+0000 7fa6b5250740 20 client.16389 trim_cache size 0 max 16384 2024-05-12T22:14:39.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.040+0000 7fa6b5250740 1 Processor -- start 2024-05-12T22:14:39.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.040+0000 7fa6b5250740 1 -- start start 2024-05-12T22:14:39.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.040+0000 7fa6b5250740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562144f83b60 0x562144f9a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:39.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.040+0000 7fa6b5250740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562144f84480 0x562144f7ee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:39.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.040+0000 7fa6b5250740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562144f8eb50 0x562144fa6df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:39.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.040+0000 7fa6b5250740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa6ac1b6110 con 0x562144f83b60 2024-05-12T22:14:39.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.040+0000 7fa6b5250740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fa6ac1b39e0 con 0x562144f84480 2024-05-12T22:14:39.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.040+0000 7fa6b5250740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa6ac1b1830 con 0x562144f8eb50 2024-05-12T22:14:39.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.041+0000 7fa6ab400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562144f83b60 0x562144f9a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:39.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.041+0000 7fa6ab400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562144f83b60 0x562144f9a7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38860/0 (socket says 172.21.15.28:38860) 2024-05-12T22:14:39.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.041+0000 7fa6ab400640 1 -- 172.21.15.28:0/2196802912 learned_addr learned my addr 172.21.15.28:0/2196802912 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:14:39.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.041+0000 7fa6a3e00640 1 --2- 172.21.15.28:0/2196802912 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562144f84480 0x562144f7ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:39.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.041+0000 7fa6abe00640 1 --2- 172.21.15.28:0/2196802912 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562144f8eb50 0x562144fa6df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:39.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.041+0000 7fa6ab400640 1 -- 172.21.15.28:0/2196802912 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562144f8eb50 msgr2=0x562144fa6df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:14:39.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.041+0000 7fa6ab400640 1 --2- 172.21.15.28:0/2196802912 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562144f8eb50 0x562144fa6df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:39.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.041+0000 7fa6ab400640 1 -- 172.21.15.28:0/2196802912 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562144f84480 msgr2=0x562144f7ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:14:39.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.041+0000 7fa6ab400640 1 --2- 172.21.15.28:0/2196802912 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562144f84480 0x562144f7ee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:39.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.041+0000 7fa6ab400640 1 -- 172.21.15.28:0/2196802912 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562144f849d0 con 0x562144f83b60 2024-05-12T22:14:39.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.041+0000 7fa6ab400640 1 --2- 172.21.15.28:0/2196802912 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562144f83b60 0x562144f9a7a0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fa69c009a20 tx=0x7fa69c005890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9494a9cd85dfa1da server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:14:39.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.042+0000 7fa69be00640 1 -- 172.21.15.28:0/2196802912 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa69c035040 con 0x562144f83b60 2024-05-12T22:14:39.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.042+0000 7fa69be00640 1 -- 172.21.15.28:0/2196802912 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa69c052600 con 0x562144f83b60 2024-05-12T22:14:39.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.042+0000 7fa69be00640 1 -- 172.21.15.28:0/2196802912 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa69c040040 con 0x562144f83b60 2024-05-12T22:14:39.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.042+0000 7fa6b5250740 1 -- 172.21.15.28:0/2196802912 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562144f83b60 msgr2=0x562144f9a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:14:39.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.042+0000 7fa6b5250740 1 --2- 172.21.15.28:0/2196802912 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562144f83b60 0x562144f9a7a0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fa69c009a20 tx=0x7fa69c005890 comp rx=0 tx=0).stop 2024-05-12T22:14:39.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.043+0000 7fa6b5250740 1 -- 172.21.15.28:0/2196802912 shutdown_connections 2024-05-12T22:14:39.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.043+0000 7fa6b5250740 1 --2- 172.21.15.28:0/2196802912 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562144f8eb50 0x562144fa6df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:39.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.043+0000 7fa6b5250740 1 --2- 172.21.15.28:0/2196802912 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562144f84480 0x562144f7ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:39.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.043+0000 7fa6b5250740 1 --2- 172.21.15.28:0/2196802912 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562144f83b60 0x562144f9a7a0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:39.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.043+0000 7fa6b5250740 1 -- 172.21.15.28:0/2196802912 >> 172.21.15.28:0/2196802912 conn(0x562144f7fe80 msgr2=0x562144f7e770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:14:39.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.043+0000 7fa6b5250740 1 -- 172.21.15.28:0/2196802912 shutdown_connections 2024-05-12T22:14:39.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.043+0000 7fa6b5250740 1 -- 172.21.15.28:0/2196802912 wait complete. 2024-05-12T22:14:39.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.043+0000 7fa6b5250740 1 -- start start 2024-05-12T22:14:39.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.043+0000 7fa6b5250740 1 Processor -- start 2024-05-12T22:14:39.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.044+0000 7fa6b5250740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562144f83b60 0x562144f88a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:39.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.044+0000 7fa6b5250740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562144f84480 0x562144f88fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:39.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.044+0000 7fa6b5250740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562144f8eb50 0x562144f89510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:39.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.044+0000 7fa6b5250740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa6ac1b1ea0 con 0x562144f84480 2024-05-12T22:14:39.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.044+0000 7fa6b5250740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fa6ac1b3310 con 0x562144f83b60 2024-05-12T22:14:39.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.044+0000 7fa6b5250740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa6ac1b2570 con 0x562144f8eb50 2024-05-12T22:14:39.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.044+0000 7fa6b5250740 10 client.0 _ll_register_callbacks cb 0x7fa6a409b7d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:14:39.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.044+0000 7fa6a3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562144f84480 0x562144f88fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:39.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.044+0000 7fa6a3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562144f84480 0x562144f88fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38874/0 (socket says 172.21.15.28:38874) 2024-05-12T22:14:39.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.044+0000 7fa6abe00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562144f8eb50 0x562144f89510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:39.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.044+0000 7fa6a3e00640 1 -- 172.21.15.28:0/3379581195 learned_addr learned my addr 172.21.15.28:0/3379581195 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:14:39.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.044+0000 7fa6ab400640 1 --2- 172.21.15.28:0/3379581195 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562144f83b60 0x562144f88a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:39.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.044+0000 7fa6a3e00640 1 -- 172.21.15.28:0/3379581195 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562144f8eb50 msgr2=0x562144f89510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:14:39.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.044+0000 7fa6a3e00640 1 --2- 172.21.15.28:0/3379581195 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562144f8eb50 0x562144f89510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:39.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.044+0000 7fa6a3e00640 1 -- 172.21.15.28:0/3379581195 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562144f83b60 msgr2=0x562144f88a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:14:39.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.044+0000 7fa6a3e00640 1 --2- 172.21.15.28:0/3379581195 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562144f83b60 0x562144f88a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:39.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.044+0000 7fa6a3e00640 1 -- 172.21.15.28:0/3379581195 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562144f90f40 con 0x562144f84480 2024-05-12T22:14:39.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.044+0000 7fa6abe00640 1 --2- 172.21.15.28:0/3379581195 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562144f8eb50 0x562144f89510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:14:39.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.045+0000 7fa6a3e00640 1 --2- 172.21.15.28:0/3379581195 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562144f84480 0x562144f88fd0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fa6940042b0 tx=0x7fa694005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16c175b31326f7bc server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:14:39.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.045+0000 7fa699600640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-12T22:14:39.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.045+0000 7fa699600640 1 -- 172.21.15.28:0/3379581195 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa694033060 con 0x562144f84480 2024-05-12T22:14:39.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.045+0000 7fa6b5250740 1 -- 172.21.15.28:0/3379581195 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa69c052600 con 0x562144f84480 2024-05-12T22:14:39.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.045+0000 7fa699600640 1 -- 172.21.15.28:0/3379581195 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7fa694009040 con 0x562144f84480 2024-05-12T22:14:39.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.045+0000 7fa699600640 1 -- 172.21.15.28:0/3379581195 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fa694003d80 con 0x562144f84480 2024-05-12T22:14:39.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.045+0000 7fa6b5250740 10 client.16395 fetch_fsmap learned FSMap version 26 2024-05-12T22:14:39.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.046+0000 7fa6b5250740 1 -- 172.21.15.28:0/3379581195 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x562145005a50 con 0x562144f84480 2024-05-12T22:14:39.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.046+0000 7fa699600640 1 -- 172.21.15.28:0/3379581195 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa69c052600 con 0x562144f84480 2024-05-12T22:14:39.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.046+0000 7fa699600640 1 -- 172.21.15.28:0/3379581195 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fa694010430 con 0x562144f84480 2024-05-12T22:14:39.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.046+0000 7fa6b5250740 10 client.16395 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:14:39.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.046+0000 7fa6b5250740 10 client.16395 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-05-12T22:14:39.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.046+0000 7fa6b5250740 20 client.16395 populate_metadata read hostname 'smithi028' 2024-05-12T22:14:39.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.046+0000 7fa6b5250740 1 --2- 172.21.15.28:0/3379581195 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x562145006230 0x562145008650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:39.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.046+0000 7fa6b5250740 4 client.16395 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-12T22:14:39.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.046+0000 7fa6b5250740 1 -- 172.21.15.28:0/3379581195 --> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562144ef04b0 con 0x562145006230 2024-05-12T22:14:39.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.046+0000 7fa6ab400640 1 --2- 172.21.15.28:0/3379581195 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x562145006230 0x562145008650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:39.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.047+0000 7fa6ab400640 1 --2- 172.21.15.28:0/3379581195 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x562145006230 0x562145008650 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=642f1f5812aac672 server_cookie=cac4f30e65520e2c in_seq=0 out_seq=0 2024-05-12T22:14:39.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.047+0000 7fa699600640 10 client.16395 ms_handle_connect on v2:172.21.15.28:6840/761183870 2024-05-12T22:14:39.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.047+0000 7fa699600640 1 -- 172.21.15.28:0/3379581195 <== mds.1 v2:172.21.15.28:6840/761183870 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562144ef04b0 con 0x562145006230 2024-05-12T22:14:39.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.047+0000 7fa699600640 10 client.16395 handle_command_reply: tid=0 2024-05-12T22:14:39.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.047+0000 7fa6b5250740 1 client.16395 shutdown 2024-05-12T22:14:39.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.047+0000 7fa6b5250740 1 -- 172.21.15.28:0/3379581195 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562144f84480 msgr2=0x562144f88fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:14:39.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.047+0000 7fa6b5250740 1 --2- 172.21.15.28:0/3379581195 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562144f84480 0x562144f88fd0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fa6940042b0 tx=0x7fa694005f90 comp rx=0 tx=0).stop 2024-05-12T22:14:39.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.047+0000 7fa6b5250740 1 -- 172.21.15.28:0/3379581195 shutdown_connections 2024-05-12T22:14:39.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.047+0000 7fa6b5250740 1 --2- 172.21.15.28:0/3379581195 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x562145006230 0x562145008650 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:39.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.047+0000 7fa6b5250740 1 --2- 172.21.15.28:0/3379581195 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562144f8eb50 0x562144f89510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:39.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.047+0000 7fa6b5250740 1 --2- 172.21.15.28:0/3379581195 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562144f84480 0x562144f88fd0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:39.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.047+0000 7fa6b5250740 1 --2- 172.21.15.28:0/3379581195 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562144f83b60 0x562144f88a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:39.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.048+0000 7fa699600640 0 client.16395 ms_handle_reset on v2:172.21.15.28:6840/761183870 2024-05-12T22:14:39.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.048+0000 7fa6b5250740 1 -- 172.21.15.28:0/3379581195 >> 172.21.15.28:0/3379581195 conn(0x562144f7fe80 msgr2=0x562144f8a010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:14:39.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.048+0000 7fa6b5250740 1 -- 172.21.15.28:0/3379581195 shutdown_connections 2024-05-12T22:14:39.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.048+0000 7fa6b5250740 1 -- 172.21.15.28:0/3379581195 wait complete. 2024-05-12T22:14:39.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.048+0000 7fa6b5250740 20 client.16395 trim_cache size 0 max 16384 2024-05-12T22:14:39.058 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-05-12T22:14:39.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.048+0000 7fa6b0e00640 1 -- 172.21.15.28:0/636208150 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fa6900a2d00 msgr2=0x7fa6900a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:14:39.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.048+0000 7fa6b0e00640 1 --2- 172.21.15.28:0/636208150 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fa6900a2d00 0x7fa6900a51c0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x562144f81010 tx=0x7fa694009b30 comp rx=0 tx=0).stop 2024-05-12T22:14:39.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.048+0000 7fa6b0e00640 1 -- 172.21.15.28:0/636208150 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa6ac1126f0 msgr2=0x7fa6ac1a2f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:14:39.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.048+0000 7fa6b0e00640 1 --2- 172.21.15.28:0/636208150 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa6ac1126f0 0x7fa6ac1a2f10 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fa6a4005fd0 tx=0x7fa6a400dea0 comp rx=0 tx=0).stop 2024-05-12T22:14:39.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.049+0000 7fa6b0e00640 1 -- 172.21.15.28:0/636208150 shutdown_connections 2024-05-12T22:14:39.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.049+0000 7fa6b0e00640 1 --2- 172.21.15.28:0/636208150 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fa6900a2d00 0x7fa6900a51c0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:39.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.049+0000 7fa6b0e00640 1 --2- 172.21.15.28:0/636208150 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa6ac1126f0 0x7fa6ac1a2f10 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:39.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.049+0000 7fa6b0e00640 1 --2- 172.21.15.28:0/636208150 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa6ac0744f0 0x7fa6ac19ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:39.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.049+0000 7fa6b0e00640 1 --2- 172.21.15.28:0/636208150 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6ac073bd0 0x7fa6ac19e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:39.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.049+0000 7fa6b0e00640 1 -- 172.21.15.28:0/636208150 >> 172.21.15.28:0/636208150 conn(0x7fa6ac0fd1d0 msgr2=0x7fa6ac0fead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:14:39.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.049+0000 7fa6b0e00640 1 -- 172.21.15.28:0/636208150 shutdown_connections 2024-05-12T22:14:39.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.049+0000 7fa6b0e00640 1 -- 172.21.15.28:0/636208150 wait complete. 2024-05-12T22:14:39.068 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-05-12T22:14:39.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.257+0000 7f6883200640 1 -- 172.21.15.28:0/1852788207 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6884107dd0 msgr2=0x7f6884108230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:14:39.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.257+0000 7f6883200640 1 --2- 172.21.15.28:0/1852788207 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6884107dd0 0x7f6884108230 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f686c009860 tx=0x7f686c02a3d0 comp rx=0 tx=0).stop 2024-05-12T22:14:39.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.258+0000 7f6883200640 1 -- 172.21.15.28:0/1852788207 shutdown_connections 2024-05-12T22:14:39.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.258+0000 7f6883200640 1 --2- 172.21.15.28:0/1852788207 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6884108770 0x7f6884114b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:39.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.258+0000 7f6883200640 1 --2- 172.21.15.28:0/1852788207 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6884107dd0 0x7f6884108230 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:39.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.258+0000 7f6883200640 1 --2- 172.21.15.28:0/1852788207 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f688410ddf0 0x7f688410e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:39.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.258+0000 7f6883200640 1 -- 172.21.15.28:0/1852788207 >> 172.21.15.28:0/1852788207 conn(0x7f6884077650 msgr2=0x7f6884106650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:14:39.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.258+0000 7f6883200640 1 -- 172.21.15.28:0/1852788207 shutdown_connections 2024-05-12T22:14:39.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.258+0000 7f6883200640 1 -- 172.21.15.28:0/1852788207 wait complete. 2024-05-12T22:14:39.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.259+0000 7f6883200640 1 Processor -- start 2024-05-12T22:14:39.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.260+0000 7f6883200640 1 -- start start 2024-05-12T22:14:39.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.260+0000 7f6883200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6884107dd0 0x7f688419e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:39.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.260+0000 7f6883200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6884108770 0x7f688419ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:39.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.260+0000 7f6883200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f688410ddf0 0x7f68841a2db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:39.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.260+0000 7f6883200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6884118100 con 0x7f688410ddf0 2024-05-12T22:14:39.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.260+0000 7f6883200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f6884117f80 con 0x7f6884107dd0 2024-05-12T22:14:39.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.260+0000 7f6883200640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f6884118280 con 0x7f6884108770 2024-05-12T22:14:39.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.260+0000 7f6882800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f688410ddf0 0x7f68841a2db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:39.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.260+0000 7f6881e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6884107dd0 0x7f688419e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:39.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.261+0000 7f6881e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6884107dd0 0x7f688419e540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:49774/0 (socket says 172.21.15.28:49774) 2024-05-12T22:14:39.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.261+0000 7f6881e00640 1 -- 172.21.15.28:0/1809420238 learned_addr learned my addr 172.21.15.28:0/1809420238 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:14:39.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.261+0000 7f6879400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6884108770 0x7f688419ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:39.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.261+0000 7f6881e00640 1 -- 172.21.15.28:0/1809420238 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6884108770 msgr2=0x7f688419ea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:14:39.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.261+0000 7f6881e00640 1 --2- 172.21.15.28:0/1809420238 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6884108770 0x7f688419ea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:39.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.261+0000 7f6881e00640 1 -- 172.21.15.28:0/1809420238 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f688410ddf0 msgr2=0x7f68841a2db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:14:39.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.261+0000 7f6881e00640 1 --2- 172.21.15.28:0/1809420238 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f688410ddf0 0x7f68841a2db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:39.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.261+0000 7f6881e00640 1 -- 172.21.15.28:0/1809420238 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68841a34e0 con 0x7f6884107dd0 2024-05-12T22:14:39.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.261+0000 7f6879400640 1 --2- 172.21.15.28:0/1809420238 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6884108770 0x7f688419ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:14:39.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.261+0000 7f6881e00640 1 --2- 172.21.15.28:0/1809420238 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6884107dd0 0x7f688419e540 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f687400ebc0 tx=0x7f687400c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:14:39.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.262+0000 7f687b400640 1 -- 172.21.15.28:0/1809420238 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6874010630 con 0x7f6884107dd0 2024-05-12T22:14:39.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.262+0000 7f6883200640 1 -- 172.21.15.28:0/1809420238 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68841a37d0 con 0x7f6884107dd0 2024-05-12T22:14:39.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.262+0000 7f687b400640 1 -- 172.21.15.28:0/1809420238 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6874010cb0 con 0x7f6884107dd0 2024-05-12T22:14:39.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.262+0000 7f687b400640 1 -- 172.21.15.28:0/1809420238 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6874018bd0 con 0x7f6884107dd0 2024-05-12T22:14:39.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.262+0000 7f6883200640 1 -- 172.21.15.28:0/1809420238 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68841ab060 con 0x7f6884107dd0 2024-05-12T22:14:39.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.265+0000 7f688bbd9740 1 Processor -- start 2024-05-12T22:14:39.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.265+0000 7f688bbd9740 1 -- start start 2024-05-12T22:14:39.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.266+0000 7f688bbd9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e521a2b60 0x558e521a2f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:39.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.266+0000 7f688bbd9740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558e521a3480 0x558e521ad610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:39.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.266+0000 7f688bbd9740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558e521adb50 0x558e521aff40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:39.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.266+0000 7f688bbd9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558e52107a30 con 0x558e521a2b60 2024-05-12T22:14:39.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.266+0000 7f688bbd9740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x558e520af700 con 0x558e521a3480 2024-05-12T22:14:39.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.266+0000 7f688bbd9740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x558e521039f0 con 0x558e521adb50 2024-05-12T22:14:39.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.266+0000 7f6882800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e521a2b60 0x558e521a2f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:39.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.266+0000 7f6882800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e521a2b60 0x558e521a2f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38890/0 (socket says 172.21.15.28:38890) 2024-05-12T22:14:39.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.266+0000 7f6879400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558e521adb50 0x558e521aff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:39.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.266+0000 7f6882800640 1 -- 172.21.15.28:0/2499566583 learned_addr learned my addr 172.21.15.28:0/2499566583 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:14:39.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.267+0000 7f6879400640 1 -- 172.21.15.28:0/2499566583 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558e521a3480 msgr2=0x558e521ad610 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-12T22:14:39.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.267+0000 7f6879400640 1 --2- 172.21.15.28:0/2499566583 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558e521a3480 0x558e521ad610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:39.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.267+0000 7f687b400640 1 -- 172.21.15.28:0/1809420238 <== mon.2 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7f68740107d0 con 0x7f6884107dd0 2024-05-12T22:14:39.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.267+0000 7f6879400640 1 -- 172.21.15.28:0/2499566583 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e521a2b60 msgr2=0x558e521a2f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:14:39.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.267+0000 7f6879400640 1 --2- 172.21.15.28:0/2499566583 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e521a2b60 0x558e521a2f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:39.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.267+0000 7f6879400640 1 -- 172.21.15.28:0/2499566583 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558e51e1b3f0 con 0x558e521adb50 2024-05-12T22:14:39.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.267+0000 7f6882800640 1 --2- 172.21.15.28:0/2499566583 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e521a2b60 0x558e521a2f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:14:39.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.267+0000 7f6879400640 1 --2- 172.21.15.28:0/2499566583 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558e521adb50 0x558e521aff40 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f686c009830 tx=0x7f686c030e60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1ce3b3a85ec1877e server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:14:39.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.268+0000 7f6873000640 1 -- 172.21.15.28:0/2499566583 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f686c004230 con 0x558e521adb50 2024-05-12T22:14:39.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.268+0000 7f6873000640 1 -- 172.21.15.28:0/2499566583 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f686c041030 con 0x558e521adb50 2024-05-12T22:14:39.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.268+0000 7f6873000640 1 -- 172.21.15.28:0/2499566583 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f686c0046c0 con 0x558e521adb50 2024-05-12T22:14:39.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.268+0000 7f687b400640 1 --2- 172.21.15.28:0/1809420238 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f68680a2d80 0x7f68680a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:39.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.269+0000 7f688bbd9740 1 -- 172.21.15.28:0/2499566583 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558e521adb50 msgr2=0x558e521aff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:14:39.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.269+0000 7f688bbd9740 1 --2- 172.21.15.28:0/2499566583 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558e521adb50 0x558e521aff40 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f686c009830 tx=0x7f686c030e60 comp rx=0 tx=0).stop 2024-05-12T22:14:39.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.269+0000 7f687b400640 1 -- 172.21.15.28:0/1809420238 <== mon.2 v2:172.21.15.86:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f68740cf9c0 con 0x7f6884107dd0 2024-05-12T22:14:39.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.269+0000 7f688bbd9740 1 -- 172.21.15.28:0/2499566583 shutdown_connections 2024-05-12T22:14:39.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.269+0000 7f688bbd9740 1 --2- 172.21.15.28:0/2499566583 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558e521adb50 0x558e521aff40 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:39.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.269+0000 7f688bbd9740 1 --2- 172.21.15.28:0/2499566583 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558e521a3480 0x558e521ad610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:39.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.269+0000 7f688bbd9740 1 --2- 172.21.15.28:0/2499566583 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e521a2b60 0x558e521a2f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:39.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.269+0000 7f6882800640 1 --2- 172.21.15.28:0/1809420238 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f68680a2d80 0x7f68680a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:39.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.269+0000 7f688bbd9740 1 -- 172.21.15.28:0/2499566583 >> 172.21.15.28:0/2499566583 conn(0x558e5219ee80 msgr2=0x558e5219f260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:14:39.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.269+0000 7f688bbd9740 1 -- 172.21.15.28:0/2499566583 shutdown_connections 2024-05-12T22:14:39.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.269+0000 7f688bbd9740 1 -- 172.21.15.28:0/2499566583 wait complete. 2024-05-12T22:14:39.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.270+0000 7f688bbd9740 1 -- start start 2024-05-12T22:14:39.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.270+0000 7f6882800640 1 --2- 172.21.15.28:0/1809420238 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f68680a2d80 0x7f68680a5240 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f687c005fd0 tx=0x7f687c009260 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:14:39.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.270+0000 7f688bbd9740 1 Processor -- start 2024-05-12T22:14:39.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.270+0000 7f688bbd9740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558e521a2b60 0x558e521aab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:39.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.270+0000 7f688bbd9740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558e521a3480 0x558e521ab070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:39.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.270+0000 7f688bbd9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e521adb50 0x558e521ab5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:39.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.270+0000 7f688bbd9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558e521b08e0 con 0x558e521adb50 2024-05-12T22:14:39.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.270+0000 7f688bbd9740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x558e520c2a70 con 0x558e521a3480 2024-05-12T22:14:39.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.270+0000 7f688bbd9740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x558e521b0a60 con 0x558e521a2b60 2024-05-12T22:14:39.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.271+0000 7f688bbd9740 10 client.0 _ll_register_callbacks cb 0x558e520a5ba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:14:39.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.271+0000 7f6881e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558e521a2b60 0x558e521aab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:39.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.271+0000 7f6881e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558e521a2b60 0x558e521aab30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:60438/0 (socket says 172.21.15.28:60438) 2024-05-12T22:14:39.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.271+0000 7f6879400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558e521a3480 0x558e521ab070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:39.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.271+0000 7f6881e00640 1 -- 172.21.15.28:0/1588278997 learned_addr learned my addr 172.21.15.28:0/1588278997 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:14:39.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.271+0000 7f6882800640 1 --2- 172.21.15.28:0/1588278997 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e521adb50 0x558e521ab5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:39.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.271+0000 7f6881e00640 1 -- 172.21.15.28:0/1588278997 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558e521a3480 msgr2=0x558e521ab070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:14:39.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.271+0000 7f6881e00640 1 --2- 172.21.15.28:0/1588278997 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558e521a3480 0x558e521ab070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:39.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.271+0000 7f6881e00640 1 -- 172.21.15.28:0/1588278997 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e521adb50 msgr2=0x558e521ab5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:14:39.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.271+0000 7f6881e00640 1 --2- 172.21.15.28:0/1588278997 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e521adb50 0x558e521ab5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:39.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.271+0000 7f6881e00640 1 -- 172.21.15.28:0/1588278997 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558e51ee57d0 con 0x558e521a2b60 2024-05-12T22:14:39.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.272+0000 7f6881e00640 1 --2- 172.21.15.28:0/1588278997 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558e521a2b60 0x558e521aab30 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f687400eb90 tx=0x7f687409bbc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d839c0a88e4b587d server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:14:39.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.272+0000 7f6867400640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-12T22:14:39.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.272+0000 7f6867400640 1 -- 172.21.15.28:0/1588278997 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6874010e60 con 0x558e521a2b60 2024-05-12T22:14:39.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.272+0000 7f6867400640 1 -- 172.21.15.28:0/1588278997 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f68740bab50 con 0x558e521a2b60 2024-05-12T22:14:39.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.272+0000 7f688bbd9740 1 -- 172.21.15.28:0/1588278997 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f686c041030 con 0x558e521a2b60 2024-05-12T22:14:39.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.273+0000 7f6867400640 1 -- 172.21.15.28:0/1588278997 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f68740b7070 con 0x558e521a2b60 2024-05-12T22:14:39.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.273+0000 7f688bbd9740 10 client.26494 fetch_fsmap learned FSMap version 26 2024-05-12T22:14:39.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.273+0000 7f688bbd9740 1 -- 172.21.15.28:0/1588278997 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x558e521bcc00 con 0x558e521a2b60 2024-05-12T22:14:39.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.273+0000 7f6867400640 1 -- 172.21.15.28:0/1588278997 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f686c041030 con 0x558e521a2b60 2024-05-12T22:14:39.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.273+0000 7f6867400640 1 -- 172.21.15.28:0/1588278997 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f6874096b30 con 0x558e521a2b60 2024-05-12T22:14:39.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.273+0000 7f688bbd9740 10 client.26494 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:14:39.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.273+0000 7f688bbd9740 10 client.26494 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:14:39.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.273+0000 7f688bbd9740 20 client.26494 populate_metadata read hostname 'smithi028' 2024-05-12T22:14:39.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.273+0000 7f688bbd9740 1 --2- 172.21.15.28:0/1588278997 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x558e521be010 0x558e521c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:39.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.273+0000 7f688bbd9740 4 client.26494 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-12T22:14:39.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.273+0000 7f688bbd9740 1 -- 172.21.15.28:0/1588278997 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558e52115160 con 0x558e521be010 2024-05-12T22:14:39.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.274+0000 7f6879400640 1 --2- 172.21.15.28:0/1588278997 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x558e521be010 0x558e521c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:39.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.274+0000 7f6879400640 1 --2- 172.21.15.28:0/1588278997 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x558e521be010 0x558e521c0400 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=312ba1610a31fec6 server_cookie=8cff5648eaf8a846 in_seq=0 out_seq=0 2024-05-12T22:14:39.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.274+0000 7f6867400640 10 client.26494 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:14:39.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.275+0000 7f6867400640 1 -- 172.21.15.28:0/1588278997 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558e52115160 con 0x558e521be010 2024-05-12T22:14:39.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.275+0000 7f6867400640 10 client.26494 handle_command_reply: tid=0 2024-05-12T22:14:39.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.275+0000 7f688bbd9740 1 client.26494 shutdown 2024-05-12T22:14:39.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.275+0000 7f688bbd9740 1 -- 172.21.15.28:0/1588278997 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558e521a2b60 msgr2=0x558e521aab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:14:39.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.275+0000 7f688bbd9740 1 --2- 172.21.15.28:0/1588278997 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558e521a2b60 0x558e521aab30 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f687400eb90 tx=0x7f687409bbc0 comp rx=0 tx=0).stop 2024-05-12T22:14:39.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.276+0000 7f688bbd9740 1 -- 172.21.15.28:0/1588278997 shutdown_connections 2024-05-12T22:14:39.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.276+0000 7f688bbd9740 1 --2- 172.21.15.28:0/1588278997 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x558e521be010 0x558e521c0400 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:39.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.276+0000 7f688bbd9740 1 --2- 172.21.15.28:0/1588278997 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e521adb50 0x558e521ab5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:39.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.276+0000 7f6867400640 0 client.26494 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:14:39.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.276+0000 7f688bbd9740 1 --2- 172.21.15.28:0/1588278997 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558e521a3480 0x558e521ab070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:39.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.276+0000 7f688bbd9740 1 --2- 172.21.15.28:0/1588278997 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558e521a2b60 0x558e521aab30 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:39.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.276+0000 7f688bbd9740 1 -- 172.21.15.28:0/1588278997 >> 172.21.15.28:0/1588278997 conn(0x558e5219ee80 msgr2=0x558e5219e000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:14:39.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.276+0000 7f688bbd9740 1 -- 172.21.15.28:0/1588278997 shutdown_connections 2024-05-12T22:14:39.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.276+0000 7f688bbd9740 1 -- 172.21.15.28:0/1588278997 wait complete. 2024-05-12T22:14:39.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.276+0000 7f688bbd9740 20 client.26494 trim_cache size 0 max 16384 2024-05-12T22:14:39.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.291+0000 7f688bbd9740 1 Processor -- start 2024-05-12T22:14:39.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.291+0000 7f688bbd9740 1 -- start start 2024-05-12T22:14:39.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.291+0000 7f688bbd9740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558e521a2b60 0x558e521b97d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:39.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.291+0000 7f688bbd9740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558e521a3480 0x558e521a9c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:39.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.291+0000 7f688bbd9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e521adb50 0x558e5219de80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:39.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.291+0000 7f688bbd9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f68841b6110 con 0x558e521adb50 2024-05-12T22:14:39.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.291+0000 7f688bbd9740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f68841b39e0 con 0x558e521a2b60 2024-05-12T22:14:39.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.291+0000 7f688bbd9740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f68841b1830 con 0x558e521a3480 2024-05-12T22:14:39.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.292+0000 7f6882800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e521adb50 0x558e5219de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:39.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.292+0000 7f6879400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558e521a3480 0x558e521a9c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:39.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.292+0000 7f6881e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558e521a2b60 0x558e521b97d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:39.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.292+0000 7f6879400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558e521a3480 0x558e521a9c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:60446/0 (socket says 172.21.15.28:60446) 2024-05-12T22:14:39.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.292+0000 7f6879400640 1 -- 172.21.15.28:0/4288467404 learned_addr learned my addr 172.21.15.28:0/4288467404 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:14:39.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.292+0000 7f6882800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e521adb50 0x558e5219de80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38908/0 (socket says 172.21.15.28:38908) 2024-05-12T22:14:39.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.292+0000 7f6879400640 1 -- 172.21.15.28:0/4288467404 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558e521a2b60 msgr2=0x558e521b97d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:14:39.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.292+0000 7f6879400640 1 --2- 172.21.15.28:0/4288467404 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558e521a2b60 0x558e521b97d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:39.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.292+0000 7f6879400640 1 -- 172.21.15.28:0/4288467404 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e521adb50 msgr2=0x558e5219de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:14:39.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.292+0000 7f6879400640 1 --2- 172.21.15.28:0/4288467404 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e521adb50 0x558e5219de80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:39.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.292+0000 7f6879400640 1 -- 172.21.15.28:0/4288467404 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558e521c0770 con 0x558e521a3480 2024-05-12T22:14:39.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.292+0000 7f6879400640 1 --2- 172.21.15.28:0/4288467404 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558e521a3480 0x558e521a9c90 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x558e521a0010 tx=0x7f686c03f040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bc01dd89ccea75ee server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:14:39.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.293+0000 7f6872600640 1 -- 172.21.15.28:0/4288467404 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f686c0307e0 con 0x558e521a3480 2024-05-12T22:14:39.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.293+0000 7f6872600640 1 -- 172.21.15.28:0/4288467404 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f686c041040 con 0x558e521a3480 2024-05-12T22:14:39.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.293+0000 7f6872600640 1 -- 172.21.15.28:0/4288467404 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f686c004920 con 0x558e521a3480 2024-05-12T22:14:39.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.293+0000 7f688bbd9740 1 -- 172.21.15.28:0/4288467404 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558e521a3480 msgr2=0x558e521a9c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:14:39.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.293+0000 7f688bbd9740 1 --2- 172.21.15.28:0/4288467404 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558e521a3480 0x558e521a9c90 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x558e521a0010 tx=0x7f686c03f040 comp rx=0 tx=0).stop 2024-05-12T22:14:39.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.293+0000 7f688bbd9740 1 -- 172.21.15.28:0/4288467404 shutdown_connections 2024-05-12T22:14:39.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.293+0000 7f688bbd9740 1 --2- 172.21.15.28:0/4288467404 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e521adb50 0x558e5219de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:39.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.294+0000 7f688bbd9740 1 --2- 172.21.15.28:0/4288467404 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558e521a3480 0x558e521a9c90 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:39.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.294+0000 7f688bbd9740 1 --2- 172.21.15.28:0/4288467404 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558e521a2b60 0x558e521b97d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:39.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.294+0000 7f688bbd9740 1 -- 172.21.15.28:0/4288467404 >> 172.21.15.28:0/4288467404 conn(0x558e5219ee80 msgr2=0x558e5219d770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:14:39.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.294+0000 7f688bbd9740 1 -- 172.21.15.28:0/4288467404 shutdown_connections 2024-05-12T22:14:39.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.294+0000 7f688bbd9740 1 -- 172.21.15.28:0/4288467404 wait complete. 2024-05-12T22:14:39.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.294+0000 7f688bbd9740 1 -- start start 2024-05-12T22:14:39.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.294+0000 7f688bbd9740 1 Processor -- start 2024-05-12T22:14:39.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.295+0000 7f688bbd9740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558e521a2b60 0x558e521b6010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:39.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.295+0000 7f688bbd9740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558e521a3480 0x558e521b1380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:39.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.295+0000 7f688bbd9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e521adb50 0x558e521b18c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:39.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.295+0000 7f688bbd9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f68841b1ea0 con 0x558e521adb50 2024-05-12T22:14:39.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.295+0000 7f688bbd9740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f68841b3310 con 0x558e521a3480 2024-05-12T22:14:39.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.295+0000 7f688bbd9740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f68841b2570 con 0x558e521a2b60 2024-05-12T22:14:39.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.295+0000 7f688bbd9740 10 client.0 _ll_register_callbacks cb 0x7f6874098260 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:14:39.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.295+0000 7f6881e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558e521a2b60 0x558e521b6010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:39.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.295+0000 7f6881e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558e521a2b60 0x558e521b6010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:60452/0 (socket says 172.21.15.28:60452) 2024-05-12T22:14:39.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.295+0000 7f6879400640 1 --2- 172.21.15.28:0/404237802 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558e521a3480 0x558e521b1380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:39.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.295+0000 7f6881e00640 1 -- 172.21.15.28:0/404237802 learned_addr learned my addr 172.21.15.28:0/404237802 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:14:39.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.295+0000 7f6882800640 1 --2- 172.21.15.28:0/404237802 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e521adb50 0x558e521b18c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:39.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.296+0000 7f6881e00640 1 -- 172.21.15.28:0/404237802 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558e521a3480 msgr2=0x558e521b1380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:14:39.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.296+0000 7f6881e00640 1 --2- 172.21.15.28:0/404237802 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558e521a3480 0x558e521b1380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:39.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.296+0000 7f6881e00640 1 -- 172.21.15.28:0/404237802 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e521adb50 msgr2=0x558e521b18c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:14:39.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.296+0000 7f6881e00640 1 --2- 172.21.15.28:0/404237802 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e521adb50 0x558e521b18c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:39.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.296+0000 7f6881e00640 1 -- 172.21.15.28:0/404237802 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558e521b20c0 con 0x558e521a2b60 2024-05-12T22:14:39.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.296+0000 7f6879400640 1 --2- 172.21.15.28:0/404237802 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558e521a3480 0x558e521b1380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:14:39.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.296+0000 7f6881e00640 1 --2- 172.21.15.28:0/404237802 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558e521a2b60 0x558e521b6010 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f68740cfce0 tx=0x7f6874000f40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=320f62d6d8074dcc server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:14:39.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.296+0000 7f6867400640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-12T22:14:39.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.296+0000 7f6867400640 1 -- 172.21.15.28:0/404237802 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6874010e60 con 0x558e521a2b60 2024-05-12T22:14:39.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.296+0000 7f6867400640 1 -- 172.21.15.28:0/404237802 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f68740bab50 con 0x558e521a2b60 2024-05-12T22:14:39.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.296+0000 7f688bbd9740 1 -- 172.21.15.28:0/404237802 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f686c041040 con 0x558e521a2b60 2024-05-12T22:14:39.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.296+0000 7f6867400640 1 -- 172.21.15.28:0/404237802 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f68740a4b80 con 0x558e521a2b60 2024-05-12T22:14:39.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.296+0000 7f688bbd9740 10 client.26500 fetch_fsmap learned FSMap version 26 2024-05-12T22:14:39.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.296+0000 7f688bbd9740 1 -- 172.21.15.28:0/404237802 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x558e522244b0 con 0x558e521a2b60 2024-05-12T22:14:39.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.297+0000 7f6867400640 1 -- 172.21.15.28:0/404237802 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f686c041040 con 0x558e521a2b60 2024-05-12T22:14:39.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.297+0000 7f6867400640 1 -- 172.21.15.28:0/404237802 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f687409ae80 con 0x558e521a2b60 2024-05-12T22:14:39.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.297+0000 7f688bbd9740 10 client.26500 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:14:39.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.297+0000 7f688bbd9740 10 client.26500 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:14:39.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.297+0000 7f688bbd9740 20 client.26500 populate_metadata read hostname 'smithi028' 2024-05-12T22:14:39.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.297+0000 7f688bbd9740 1 --2- 172.21.15.28:0/404237802 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x558e52224c50 0x558e52227070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:39.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.297+0000 7f688bbd9740 4 client.26500 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-12T22:14:39.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.297+0000 7f688bbd9740 1 -- 172.21.15.28:0/404237802 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558e5210f4b0 con 0x558e52224c50 2024-05-12T22:14:39.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.297+0000 7f6879400640 1 --2- 172.21.15.28:0/404237802 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x558e52224c50 0x558e52227070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:39.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.298+0000 7f6879400640 1 --2- 172.21.15.28:0/404237802 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x558e52224c50 0x558e52227070 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5a9e7cd3cbb1f4d5 server_cookie=5da4e10370cf3c2a in_seq=0 out_seq=0 2024-05-12T22:14:39.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.298+0000 7f6867400640 10 client.26500 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:14:39.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.298+0000 7f6867400640 1 -- 172.21.15.28:0/404237802 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558e5210f4b0 con 0x558e52224c50 2024-05-12T22:14:39.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.298+0000 7f6867400640 10 client.26500 handle_command_reply: tid=0 2024-05-12T22:14:39.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.298+0000 7f688bbd9740 1 client.26500 shutdown 2024-05-12T22:14:39.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.298+0000 7f688bbd9740 1 -- 172.21.15.28:0/404237802 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558e521a2b60 msgr2=0x558e521b6010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:14:39.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.298+0000 7f688bbd9740 1 --2- 172.21.15.28:0/404237802 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558e521a2b60 0x558e521b6010 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f68740cfce0 tx=0x7f6874000f40 comp rx=0 tx=0).stop 2024-05-12T22:14:39.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.298+0000 7f688bbd9740 1 -- 172.21.15.28:0/404237802 shutdown_connections 2024-05-12T22:14:39.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.298+0000 7f688bbd9740 1 --2- 172.21.15.28:0/404237802 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x558e52224c50 0x558e52227070 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:39.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.298+0000 7f688bbd9740 1 --2- 172.21.15.28:0/404237802 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e521adb50 0x558e521b18c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:39.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.298+0000 7f688bbd9740 1 --2- 172.21.15.28:0/404237802 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558e521a3480 0x558e521b1380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:39.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.298+0000 7f688bbd9740 1 --2- 172.21.15.28:0/404237802 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558e521a2b60 0x558e521b6010 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:39.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.298+0000 7f688bbd9740 1 -- 172.21.15.28:0/404237802 >> 172.21.15.28:0/404237802 conn(0x558e5219ee80 msgr2=0x558e521a42b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:14:39.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.298+0000 7f6867400640 0 client.26500 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:14:39.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.298+0000 7f688bbd9740 1 -- 172.21.15.28:0/404237802 shutdown_connections 2024-05-12T22:14:39.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.299+0000 7f688bbd9740 1 -- 172.21.15.28:0/404237802 wait complete. 2024-05-12T22:14:39.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.299+0000 7f688bbd9740 20 client.26500 trim_cache size 0 max 16384 2024-05-12T22:14:39.308 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-05-12T22:14:39.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.299+0000 7f6883200640 1 -- 172.21.15.28:0/1809420238 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f68680a2d80 msgr2=0x7f68680a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:14:39.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.299+0000 7f6883200640 1 --2- 172.21.15.28:0/1809420238 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f68680a2d80 0x7f68680a5240 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f687c005fd0 tx=0x7f687c009260 comp rx=0 tx=0).stop 2024-05-12T22:14:39.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.299+0000 7f6883200640 1 -- 172.21.15.28:0/1809420238 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6884107dd0 msgr2=0x7f688419e540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:14:39.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.299+0000 7f6883200640 1 --2- 172.21.15.28:0/1809420238 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6884107dd0 0x7f688419e540 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f687400ebc0 tx=0x7f687400c6a0 comp rx=0 tx=0).stop 2024-05-12T22:14:39.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.299+0000 7f6883200640 1 -- 172.21.15.28:0/1809420238 shutdown_connections 2024-05-12T22:14:39.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.299+0000 7f6883200640 1 --2- 172.21.15.28:0/1809420238 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f68680a2d80 0x7f68680a5240 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:39.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.299+0000 7f6883200640 1 --2- 172.21.15.28:0/1809420238 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f688410ddf0 0x7f68841a2db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:39.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.299+0000 7f6883200640 1 --2- 172.21.15.28:0/1809420238 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6884108770 0x7f688419ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:39.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.299+0000 7f6883200640 1 --2- 172.21.15.28:0/1809420238 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6884107dd0 0x7f688419e540 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:39.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.299+0000 7f6883200640 1 -- 172.21.15.28:0/1809420238 >> 172.21.15.28:0/1809420238 conn(0x7f6884077650 msgr2=0x7f6884078650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:14:39.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.300+0000 7f6883200640 1 -- 172.21.15.28:0/1809420238 shutdown_connections 2024-05-12T22:14:39.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:39.300+0000 7f6883200640 1 -- 172.21.15.28:0/1809420238 wait complete. 2024-05-12T22:14:39.315 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:14:39 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/3506153285' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:14:39.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:14:39 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/3506153285' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:14:39.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:14:39 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/3506153285' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:14:40.319 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-12T22:14:40.319 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-12T22:14:40.320 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-05-12T22:14:40.534 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:14:40 smithi028 ceph-mon[29847]: pgmap v173: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:14:40.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.549+0000 7ff5f3200640 1 -- 172.21.15.28:0/1845303916 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff5f4075d60 msgr2=0x7ff5f4076140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:14:40.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.549+0000 7ff5f3200640 1 --2- 172.21.15.28:0/1845303916 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff5f4075d60 0x7ff5f4076140 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7ff5e4009890 tx=0x7ff5e402a360 comp rx=0 tx=0).stop 2024-05-12T22:14:40.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.550+0000 7ff5f3200640 1 -- 172.21.15.28:0/1845303916 shutdown_connections 2024-05-12T22:14:40.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.550+0000 7ff5f3200640 1 --2- 172.21.15.28:0/1845303916 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff5f4112550 0x7ff5f4114940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:40.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.550+0000 7ff5f3200640 1 --2- 172.21.15.28:0/1845303916 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff5f4076680 0x7ff5f4112010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:40.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.550+0000 7ff5f3200640 1 --2- 172.21.15.28:0/1845303916 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff5f4075d60 0x7ff5f4076140 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:40.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.550+0000 7ff5f3200640 1 -- 172.21.15.28:0/1845303916 >> 172.21.15.28:0/1845303916 conn(0x7ff5f40ff340 msgr2=0x7ff5f4101760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:14:40.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.550+0000 7ff5f3200640 1 -- 172.21.15.28:0/1845303916 shutdown_connections 2024-05-12T22:14:40.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.550+0000 7ff5f3200640 1 -- 172.21.15.28:0/1845303916 wait complete. 2024-05-12T22:14:40.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.551+0000 7ff5f3200640 1 Processor -- start 2024-05-12T22:14:40.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.551+0000 7ff5f3200640 1 -- start start 2024-05-12T22:14:40.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.552+0000 7ff5f3200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff5f4075d60 0x7ff5f419e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:40.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.552+0000 7ff5f3200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff5f4076680 0x7ff5f419eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:40.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.552+0000 7ff5f3200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff5f4112550 0x7ff5f41a2ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:40.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.552+0000 7ff5f3200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff5f41180a0 con 0x7ff5f4075d60 2024-05-12T22:14:40.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.552+0000 7ff5f3200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7ff5f4117f20 con 0x7ff5f4076680 2024-05-12T22:14:40.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.552+0000 7ff5f3200640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff5f4118220 con 0x7ff5f4112550 2024-05-12T22:14:40.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.552+0000 7ff5f1e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff5f4075d60 0x7ff5f419e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:40.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.552+0000 7ff5f1e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff5f4075d60 0x7ff5f419e600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38930/0 (socket says 172.21.15.28:38930) 2024-05-12T22:14:40.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.552+0000 7ff5f1e00640 1 -- 172.21.15.28:0/994939050 learned_addr learned my addr 172.21.15.28:0/994939050 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:14:40.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.552+0000 7ff5f2800640 1 --2- 172.21.15.28:0/994939050 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff5f4112550 0x7ff5f41a2ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:40.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.552+0000 7ff5f1400640 1 --2- 172.21.15.28:0/994939050 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff5f4076680 0x7ff5f419eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:40.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.552+0000 7ff5f1e00640 1 -- 172.21.15.28:0/994939050 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff5f4112550 msgr2=0x7ff5f41a2ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:14:40.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.552+0000 7ff5f1e00640 1 --2- 172.21.15.28:0/994939050 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff5f4112550 0x7ff5f41a2ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:40.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.552+0000 7ff5f1e00640 1 -- 172.21.15.28:0/994939050 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff5f4076680 msgr2=0x7ff5f419eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:14:40.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.552+0000 7ff5f1e00640 1 --2- 172.21.15.28:0/994939050 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff5f4076680 0x7ff5f419eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:40.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.552+0000 7ff5f1e00640 1 -- 172.21.15.28:0/994939050 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5f41a35f0 con 0x7ff5f4075d60 2024-05-12T22:14:40.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.553+0000 7ff5f1e00640 1 --2- 172.21.15.28:0/994939050 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff5f4075d60 0x7ff5f419e600 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7ff5e4033dc0 tx=0x7ff5e4033df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:14:40.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.553+0000 7ff5eaa00640 1 -- 172.21.15.28:0/994939050 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5e4037040 con 0x7ff5f4075d60 2024-05-12T22:14:40.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.553+0000 7ff5eaa00640 1 -- 172.21.15.28:0/994939050 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff5e4040070 con 0x7ff5f4075d60 2024-05-12T22:14:40.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.553+0000 7ff5f3200640 1 -- 172.21.15.28:0/994939050 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5f41a3880 con 0x7ff5f4075d60 2024-05-12T22:14:40.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.553+0000 7ff5eaa00640 1 -- 172.21.15.28:0/994939050 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5e4003920 con 0x7ff5f4075d60 2024-05-12T22:14:40.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.553+0000 7ff5f3200640 1 -- 172.21.15.28:0/994939050 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5f41ab170 con 0x7ff5f4075d60 2024-05-12T22:14:40.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.557+0000 7ff5eaa00640 1 -- 172.21.15.28:0/994939050 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7ff5e4003ae0 con 0x7ff5f4075d60 2024-05-12T22:14:40.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.557+0000 7ff5fbdd9740 1 Processor -- start 2024-05-12T22:14:40.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.557+0000 7ff5fbdd9740 1 -- start start 2024-05-12T22:14:40.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.557+0000 7ff5fbdd9740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558fafd08500 0x558fafd088e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:40.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.557+0000 7ff5fbdd9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558fafd08eb0 0x558fafd12f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:40.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.557+0000 7ff5fbdd9740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558fafd13520 0x558fafd15970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:40.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.557+0000 7ff5fbdd9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558fafc6c7c0 con 0x558fafd08eb0 2024-05-12T22:14:40.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.557+0000 7ff5fbdd9740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x558fafc14790 con 0x558fafd08500 2024-05-12T22:14:40.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.557+0000 7ff5fbdd9740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x558fafc68780 con 0x558fafd13520 2024-05-12T22:14:40.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.558+0000 7ff5f1e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558fafd08eb0 0x558fafd12f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:40.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.558+0000 7ff5f2800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558fafd08500 0x558fafd088e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:40.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.558+0000 7ff5f1e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558fafd08eb0 0x558fafd12f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38934/0 (socket says 172.21.15.28:38934) 2024-05-12T22:14:40.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.558+0000 7ff5f1e00640 1 -- 172.21.15.28:0/3945463271 learned_addr learned my addr 172.21.15.28:0/3945463271 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:14:40.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.558+0000 7ff5f1400640 1 --2- 172.21.15.28:0/3945463271 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558fafd13520 0x558fafd15970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:40.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.558+0000 7ff5f1e00640 1 -- 172.21.15.28:0/3945463271 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558fafd13520 msgr2=0x558fafd15970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:14:40.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.558+0000 7ff5f1e00640 1 --2- 172.21.15.28:0/3945463271 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558fafd13520 0x558fafd15970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:40.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.558+0000 7ff5f1e00640 1 -- 172.21.15.28:0/3945463271 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558fafd08500 msgr2=0x558fafd088e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:14:40.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.558+0000 7ff5f1e00640 1 --2- 172.21.15.28:0/3945463271 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558fafd08500 0x558fafd088e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:40.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.558+0000 7ff5f1e00640 1 -- 172.21.15.28:0/3945463271 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558fafc346a0 con 0x558fafd08eb0 2024-05-12T22:14:40.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.558+0000 7ff5f2800640 1 --2- 172.21.15.28:0/3945463271 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558fafd08500 0x558fafd088e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:14:40.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.558+0000 7ff5eaa00640 1 --2- 172.21.15.28:0/994939050 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7ff5d80a2d00 0x7ff5d80a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:40.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.559+0000 7ff5eaa00640 1 -- 172.21.15.28:0/994939050 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7ff5e40ef3b0 con 0x7ff5f4075d60 2024-05-12T22:14:40.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.559+0000 7ff5f1e00640 1 --2- 172.21.15.28:0/3945463271 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558fafd08eb0 0x558fafd12f50 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7ff5e40ecb50 tx=0x7ff5e40f74b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f524d5b2b619753 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:14:40.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.559+0000 7ff5f2800640 1 --2- 172.21.15.28:0/994939050 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7ff5d80a2d00 0x7ff5d80a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:40.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.559+0000 7ff5df400640 1 -- 172.21.15.28:0/3945463271 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5e40b7c50 con 0x558fafd08eb0 2024-05-12T22:14:40.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.559+0000 7ff5df400640 1 -- 172.21.15.28:0/3945463271 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff5e4004250 con 0x558fafd08eb0 2024-05-12T22:14:40.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.559+0000 7ff5df400640 1 -- 172.21.15.28:0/3945463271 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5e40bb310 con 0x558fafd08eb0 2024-05-12T22:14:40.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.560+0000 7ff5fbdd9740 1 -- 172.21.15.28:0/3945463271 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558fafd08eb0 msgr2=0x558fafd12f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:14:40.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.560+0000 7ff5f2800640 1 --2- 172.21.15.28:0/994939050 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7ff5d80a2d00 0x7ff5d80a51c0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7ff5f406acb0 tx=0x7ff5ec009600 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:14:40.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.560+0000 7ff5fbdd9740 1 --2- 172.21.15.28:0/3945463271 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558fafd08eb0 0x558fafd12f50 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7ff5e40ecb50 tx=0x7ff5e40f74b0 comp rx=0 tx=0).stop 2024-05-12T22:14:40.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.560+0000 7ff5fbdd9740 1 -- 172.21.15.28:0/3945463271 shutdown_connections 2024-05-12T22:14:40.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.560+0000 7ff5fbdd9740 1 --2- 172.21.15.28:0/3945463271 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558fafd13520 0x558fafd15970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:40.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.560+0000 7ff5fbdd9740 1 --2- 172.21.15.28:0/3945463271 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558fafd08eb0 0x558fafd12f50 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:40.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.560+0000 7ff5fbdd9740 1 --2- 172.21.15.28:0/3945463271 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558fafd08500 0x558fafd088e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:40.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.560+0000 7ff5fbdd9740 1 -- 172.21.15.28:0/3945463271 >> 172.21.15.28:0/3945463271 conn(0x558fafd045b0 msgr2=0x558fafd04990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:14:40.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.560+0000 7ff5fbdd9740 1 -- 172.21.15.28:0/3945463271 shutdown_connections 2024-05-12T22:14:40.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.560+0000 7ff5fbdd9740 1 -- 172.21.15.28:0/3945463271 wait complete. 2024-05-12T22:14:40.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.560+0000 7ff5fbdd9740 1 -- start start 2024-05-12T22:14:40.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.561+0000 7ff5fbdd9740 1 Processor -- start 2024-05-12T22:14:40.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.561+0000 7ff5fbdd9740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558fafd08500 0x558fafd10340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:40.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.561+0000 7ff5fbdd9740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558fafd08eb0 0x558fafd10880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:40.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.561+0000 7ff5fbdd9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558fafd13520 0x558fafd10dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:40.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.561+0000 7ff5fbdd9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558fafd16250 con 0x558fafd13520 2024-05-12T22:14:40.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.561+0000 7ff5fbdd9740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x558fafc27a90 con 0x558fafd08500 2024-05-12T22:14:40.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.561+0000 7ff5fbdd9740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x558fafd163d0 con 0x558fafd08eb0 2024-05-12T22:14:40.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.561+0000 7ff5fbdd9740 10 client.0 _ll_register_callbacks cb 0x558fafc62af0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:14:40.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.562+0000 7ff5f2800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558fafd13520 0x558fafd10dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:40.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.562+0000 7ff5f1e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558fafd08500 0x558fafd10340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:40.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.562+0000 7ff5f1e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558fafd08500 0x558fafd10340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:49826/0 (socket says 172.21.15.28:49826) 2024-05-12T22:14:40.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.562+0000 7ff5f1400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558fafd08eb0 0x558fafd10880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:40.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.562+0000 7ff5f1e00640 1 -- 172.21.15.28:0/6312619 learned_addr learned my addr 172.21.15.28:0/6312619 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:14:40.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.562+0000 7ff5f1e00640 1 -- 172.21.15.28:0/6312619 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558fafd08eb0 msgr2=0x558fafd10880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:14:40.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.562+0000 7ff5f1e00640 1 --2- 172.21.15.28:0/6312619 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558fafd08eb0 0x558fafd10880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:40.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.562+0000 7ff5f1e00640 1 -- 172.21.15.28:0/6312619 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558fafd13520 msgr2=0x558fafd10dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:14:40.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.562+0000 7ff5f1e00640 1 --2- 172.21.15.28:0/6312619 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558fafd13520 0x558fafd10dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:40.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.562+0000 7ff5f1e00640 1 -- 172.21.15.28:0/6312619 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558fafa487e0 con 0x558fafd08500 2024-05-12T22:14:40.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.562+0000 7ff5f1e00640 1 --2- 172.21.15.28:0/6312619 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558fafd08500 0x558fafd10340 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7ff5e40b8380 tx=0x7ff5e402a360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6d3e2bbde45c9739 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:14:40.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.562+0000 7ff5d3e00640 10 client.0 ms_handle_connect on v2:172.21.15.86:3300/0 2024-05-12T22:14:40.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.563+0000 7ff5d3e00640 1 -- 172.21.15.28:0/6312619 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5e40bb4f0 con 0x558fafd08500 2024-05-12T22:14:40.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.563+0000 7ff5d3e00640 1 -- 172.21.15.28:0/6312619 <== mon.2 v2:172.21.15.86:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7ff5e40bbc10 con 0x558fafd08500 2024-05-12T22:14:40.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.563+0000 7ff5fbdd9740 1 -- 172.21.15.28:0/6312619 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x558fafc803a0 con 0x558fafd08500 2024-05-12T22:14:40.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.563+0000 7ff5d3e00640 1 -- 172.21.15.28:0/6312619 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7ff5e40bb780 con 0x558fafd08500 2024-05-12T22:14:40.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.563+0000 7ff5fbdd9740 10 client.26549 fetch_fsmap learned FSMap version 26 2024-05-12T22:14:40.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.563+0000 7ff5fbdd9740 1 -- 172.21.15.28:0/6312619 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x558fafaa8f70 con 0x558fafd08500 2024-05-12T22:14:40.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.563+0000 7ff5d3e00640 1 -- 172.21.15.28:0/6312619 <== mon.2 v2:172.21.15.86:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x558fafc803a0 con 0x558fafd08500 2024-05-12T22:14:40.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.563+0000 7ff5d3e00640 1 -- 172.21.15.28:0/6312619 <== mon.2 v2:172.21.15.86:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7ff5e40bcce0 con 0x558fafd08500 2024-05-12T22:14:40.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.563+0000 7ff5fbdd9740 10 client.26549 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:14:40.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.563+0000 7ff5fbdd9740 10 client.26549 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:14:40.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.564+0000 7ff5fbdd9740 20 client.26549 populate_metadata read hostname 'smithi028' 2024-05-12T22:14:40.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.564+0000 7ff5fbdd9740 1 --2- 172.21.15.28:0/6312619 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x558fafd23230 0x558fafd25620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:40.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.564+0000 7ff5fbdd9740 4 client.26549 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-12T22:14:40.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.564+0000 7ff5fbdd9740 1 -- 172.21.15.28:0/6312619 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558fafc79ef0 con 0x558fafd23230 2024-05-12T22:14:40.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.564+0000 7ff5f1400640 1 --2- 172.21.15.28:0/6312619 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x558fafd23230 0x558fafd25620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:40.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.565+0000 7ff5f1400640 1 --2- 172.21.15.28:0/6312619 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x558fafd23230 0x558fafd25620 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3bf1ae4e000f0d55 server_cookie=19a2f89b4e53cb66 in_seq=0 out_seq=0 2024-05-12T22:14:40.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.565+0000 7ff5d3e00640 10 client.26549 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:14:40.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.565+0000 7ff5d3e00640 1 -- 172.21.15.28:0/6312619 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558fafc79ef0 con 0x558fafd23230 2024-05-12T22:14:40.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.565+0000 7ff5d3e00640 10 client.26549 handle_command_reply: tid=0 2024-05-12T22:14:40.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.565+0000 7ff5fbdd9740 1 client.26549 shutdown 2024-05-12T22:14:40.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.566+0000 7ff5fbdd9740 1 -- 172.21.15.28:0/6312619 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558fafd08500 msgr2=0x558fafd10340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:14:40.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.566+0000 7ff5fbdd9740 1 --2- 172.21.15.28:0/6312619 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558fafd08500 0x558fafd10340 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7ff5e40b8380 tx=0x7ff5e402a360 comp rx=0 tx=0).stop 2024-05-12T22:14:40.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.566+0000 7ff5fbdd9740 1 -- 172.21.15.28:0/6312619 shutdown_connections 2024-05-12T22:14:40.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.566+0000 7ff5fbdd9740 1 --2- 172.21.15.28:0/6312619 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x558fafd23230 0x558fafd25620 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:40.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.566+0000 7ff5fbdd9740 1 --2- 172.21.15.28:0/6312619 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558fafd13520 0x558fafd10dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:40.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.566+0000 7ff5fbdd9740 1 --2- 172.21.15.28:0/6312619 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558fafd08eb0 0x558fafd10880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:40.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.566+0000 7ff5fbdd9740 1 --2- 172.21.15.28:0/6312619 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558fafd08500 0x558fafd10340 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:40.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.566+0000 7ff5d3e00640 0 client.26549 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:14:40.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.566+0000 7ff5fbdd9740 1 -- 172.21.15.28:0/6312619 >> 172.21.15.28:0/6312619 conn(0x558fafd045b0 msgr2=0x558fafd0a630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:14:40.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.566+0000 7ff5fbdd9740 1 -- 172.21.15.28:0/6312619 shutdown_connections 2024-05-12T22:14:40.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.566+0000 7ff5fbdd9740 1 -- 172.21.15.28:0/6312619 wait complete. 2024-05-12T22:14:40.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.566+0000 7ff5fbdd9740 20 client.26549 trim_cache size 0 max 16384 2024-05-12T22:14:40.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.585+0000 7ff5fbdd9740 1 Processor -- start 2024-05-12T22:14:40.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.586+0000 7ff5fbdd9740 1 -- start start 2024-05-12T22:14:40.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.586+0000 7ff5fbdd9740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558fafd08500 0x558fafd0a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:40.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.586+0000 7ff5fbdd9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558fafd08eb0 0x558fafd2ac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:40.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.586+0000 7ff5fbdd9740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558fafd13520 0x558fafd2b190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:40.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.586+0000 7ff5fbdd9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff5f41b6230 con 0x558fafd08eb0 2024-05-12T22:14:40.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.586+0000 7ff5fbdd9740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7ff5f41b3b00 con 0x558fafd13520 2024-05-12T22:14:40.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.586+0000 7ff5fbdd9740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff5f41b1950 con 0x558fafd08500 2024-05-12T22:14:40.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.586+0000 7ff5f2800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558fafd13520 0x558fafd2b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:40.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.586+0000 7ff5f2800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558fafd13520 0x558fafd2b190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:49832/0 (socket says 172.21.15.28:49832) 2024-05-12T22:14:40.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.586+0000 7ff5f1400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558fafd08eb0 0x558fafd2ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:40.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.586+0000 7ff5f2800640 1 -- 172.21.15.28:0/1116543527 learned_addr learned my addr 172.21.15.28:0/1116543527 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:14:40.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.586+0000 7ff5f1e00640 1 --2- 172.21.15.28:0/1116543527 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558fafd08500 0x558fafd0a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:40.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.587+0000 7ff5f2800640 1 -- 172.21.15.28:0/1116543527 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558fafd08500 msgr2=0x558fafd0a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:14:40.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.587+0000 7ff5f2800640 1 --2- 172.21.15.28:0/1116543527 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558fafd08500 0x558fafd0a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:40.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.587+0000 7ff5f2800640 1 -- 172.21.15.28:0/1116543527 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558fafd08eb0 msgr2=0x558fafd2ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:14:40.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.587+0000 7ff5f2800640 1 --2- 172.21.15.28:0/1116543527 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558fafd08eb0 0x558fafd2ac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:40.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.587+0000 7ff5f2800640 1 -- 172.21.15.28:0/1116543527 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558fafd08c50 con 0x558fafd13520 2024-05-12T22:14:40.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.587+0000 7ff5f1400640 1 --2- 172.21.15.28:0/1116543527 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558fafd08eb0 0x558fafd2ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:14:40.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.587+0000 7ff5f2800640 1 --2- 172.21.15.28:0/1116543527 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558fafd13520 0x558fafd2b190 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7ff5ec011780 tx=0x7ff5ec016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=75f3c338555e1911 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:14:40.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.587+0000 7ff5dea00640 1 -- 172.21.15.28:0/1116543527 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5ec011e90 con 0x558fafd13520 2024-05-12T22:14:40.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.588+0000 7ff5dea00640 1 -- 172.21.15.28:0/1116543527 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff5ec01d910 con 0x558fafd13520 2024-05-12T22:14:40.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.588+0000 7ff5dea00640 1 -- 172.21.15.28:0/1116543527 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5ec0209f0 con 0x558fafd13520 2024-05-12T22:14:40.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.588+0000 7ff5fbdd9740 1 -- 172.21.15.28:0/1116543527 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558fafd13520 msgr2=0x558fafd2b190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:14:40.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.588+0000 7ff5fbdd9740 1 --2- 172.21.15.28:0/1116543527 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558fafd13520 0x558fafd2b190 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7ff5ec011780 tx=0x7ff5ec016000 comp rx=0 tx=0).stop 2024-05-12T22:14:40.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.588+0000 7ff5fbdd9740 1 -- 172.21.15.28:0/1116543527 shutdown_connections 2024-05-12T22:14:40.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.588+0000 7ff5fbdd9740 1 --2- 172.21.15.28:0/1116543527 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558fafd13520 0x558fafd2b190 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:40.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.588+0000 7ff5fbdd9740 1 --2- 172.21.15.28:0/1116543527 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558fafd08eb0 0x558fafd2ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:40.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.588+0000 7ff5fbdd9740 1 --2- 172.21.15.28:0/1116543527 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558fafd08500 0x558fafd0a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:40.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.588+0000 7ff5fbdd9740 1 -- 172.21.15.28:0/1116543527 >> 172.21.15.28:0/1116543527 conn(0x558fafd045b0 msgr2=0x558fafd02ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:14:40.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.588+0000 7ff5fbdd9740 1 -- 172.21.15.28:0/1116543527 shutdown_connections 2024-05-12T22:14:40.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.588+0000 7ff5fbdd9740 1 -- 172.21.15.28:0/1116543527 wait complete. 2024-05-12T22:14:40.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.589+0000 7ff5fbdd9740 1 -- start start 2024-05-12T22:14:40.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.589+0000 7ff5fbdd9740 1 Processor -- start 2024-05-12T22:14:40.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.589+0000 7ff5fbdd9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558fafd08500 0x558fafd17310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:40.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.589+0000 7ff5fbdd9740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558fafd08eb0 0x558fafd17850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:40.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.589+0000 7ff5fbdd9740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558fafd13520 0x558fafd17d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:40.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.589+0000 7ff5fbdd9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff5f41b1fc0 con 0x558fafd08500 2024-05-12T22:14:40.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.589+0000 7ff5fbdd9740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7ff5f41b3430 con 0x558fafd13520 2024-05-12T22:14:40.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.589+0000 7ff5fbdd9740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff5f41b2690 con 0x558fafd08eb0 2024-05-12T22:14:40.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.589+0000 7ff5fbdd9740 10 client.0 _ll_register_callbacks cb 0x7ff5ec009a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:14:40.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.590+0000 7ff5f2800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558fafd13520 0x558fafd17d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:40.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.590+0000 7ff5f1e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558fafd08500 0x558fafd17310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:40.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.590+0000 7ff5f2800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558fafd13520 0x558fafd17d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:49848/0 (socket says 172.21.15.28:49848) 2024-05-12T22:14:40.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.590+0000 7ff5f2800640 1 -- 172.21.15.28:0/4192753813 learned_addr learned my addr 172.21.15.28:0/4192753813 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:14:40.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.590+0000 7ff5f1400640 1 --2- 172.21.15.28:0/4192753813 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558fafd08eb0 0x558fafd17850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:40.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.590+0000 7ff5f2800640 1 -- 172.21.15.28:0/4192753813 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558fafd08eb0 msgr2=0x558fafd17850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:14:40.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.590+0000 7ff5f2800640 1 --2- 172.21.15.28:0/4192753813 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558fafd08eb0 0x558fafd17850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:40.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.590+0000 7ff5f2800640 1 -- 172.21.15.28:0/4192753813 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558fafd08500 msgr2=0x558fafd17310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:14:40.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.590+0000 7ff5f2800640 1 --2- 172.21.15.28:0/4192753813 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558fafd08500 0x558fafd17310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:40.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.590+0000 7ff5f2800640 1 -- 172.21.15.28:0/4192753813 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558faf940170 con 0x558fafd13520 2024-05-12T22:14:40.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.590+0000 7ff5f1400640 1 --2- 172.21.15.28:0/4192753813 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558fafd08eb0 0x558fafd17850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:14:40.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.590+0000 7ff5f1e00640 1 --2- 172.21.15.28:0/4192753813 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558fafd08500 0x558fafd17310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:14:40.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.590+0000 7ff5f2800640 1 --2- 172.21.15.28:0/4192753813 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558fafd13520 0x558fafd17d90 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7ff5ec0098b0 tx=0x7ff5ec01dcf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=766a6885ae408a72 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:14:40.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.590+0000 7ff5d3e00640 10 client.0 ms_handle_connect on v2:172.21.15.86:3300/0 2024-05-12T22:14:40.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.591+0000 7ff5d3e00640 1 -- 172.21.15.28:0/4192753813 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5ec020420 con 0x558fafd13520 2024-05-12T22:14:40.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.591+0000 7ff5fbdd9740 1 -- 172.21.15.28:0/4192753813 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff5ec01d910 con 0x558fafd13520 2024-05-12T22:14:40.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.591+0000 7ff5d3e00640 1 -- 172.21.15.28:0/4192753813 <== mon.2 v2:172.21.15.86:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7ff5ec02f070 con 0x558fafd13520 2024-05-12T22:14:40.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.591+0000 7ff5d3e00640 1 -- 172.21.15.28:0/4192753813 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7ff5ec004c50 con 0x558fafd13520 2024-05-12T22:14:40.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.591+0000 7ff5fbdd9740 10 client.26555 fetch_fsmap learned FSMap version 26 2024-05-12T22:14:40.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.591+0000 7ff5fbdd9740 1 -- 172.21.15.28:0/4192753813 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x558fafd89920 con 0x558fafd13520 2024-05-12T22:14:40.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.591+0000 7ff5d3e00640 1 -- 172.21.15.28:0/4192753813 <== mon.2 v2:172.21.15.86:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff5ec01d910 con 0x558fafd13520 2024-05-12T22:14:40.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.592+0000 7ff5d3e00640 1 -- 172.21.15.28:0/4192753813 <== mon.2 v2:172.21.15.86:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7ff5ec00f040 con 0x558fafd13520 2024-05-12T22:14:40.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.592+0000 7ff5fbdd9740 10 client.26555 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:14:40.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.592+0000 7ff5fbdd9740 10 client.26555 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:14:40.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.592+0000 7ff5fbdd9740 20 client.26555 populate_metadata read hostname 'smithi028' 2024-05-12T22:14:40.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.592+0000 7ff5fbdd9740 1 --2- 172.21.15.28:0/4192753813 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x558fafd8a040 0x558fafd8c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:40.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.592+0000 7ff5fbdd9740 4 client.26555 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-12T22:14:40.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.592+0000 7ff5fbdd9740 1 -- 172.21.15.28:0/4192753813 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x558fafc74240 con 0x558fafd8a040 2024-05-12T22:14:40.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.592+0000 7ff5f1e00640 1 --2- 172.21.15.28:0/4192753813 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x558fafd8a040 0x558fafd8c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:40.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.592+0000 7ff5f1e00640 1 --2- 172.21.15.28:0/4192753813 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x558fafd8a040 0x558fafd8c430 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fe3c9c9d01ad734d server_cookie=8510904a70a22912 in_seq=0 out_seq=0 2024-05-12T22:14:40.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.592+0000 7ff5d3e00640 10 client.26555 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:14:40.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.594+0000 7ff5d3e00640 1 -- 172.21.15.28:0/4192753813 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x558fafc74240 con 0x558fafd8a040 2024-05-12T22:14:40.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.594+0000 7ff5d3e00640 10 client.26555 handle_command_reply: tid=0 2024-05-12T22:14:40.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.594+0000 7ff5fbdd9740 1 client.26555 shutdown 2024-05-12T22:14:40.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.594+0000 7ff5fbdd9740 1 -- 172.21.15.28:0/4192753813 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558fafd13520 msgr2=0x558fafd17d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:14:40.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.594+0000 7ff5fbdd9740 1 --2- 172.21.15.28:0/4192753813 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558fafd13520 0x558fafd17d90 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7ff5ec0098b0 tx=0x7ff5ec01dcf0 comp rx=0 tx=0).stop 2024-05-12T22:14:40.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.594+0000 7ff5fbdd9740 1 -- 172.21.15.28:0/4192753813 shutdown_connections 2024-05-12T22:14:40.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.594+0000 7ff5fbdd9740 1 --2- 172.21.15.28:0/4192753813 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x558fafd8a040 0x558fafd8c430 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:40.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.594+0000 7ff5fbdd9740 1 --2- 172.21.15.28:0/4192753813 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558fafd13520 0x558fafd17d90 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:40.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.594+0000 7ff5fbdd9740 1 --2- 172.21.15.28:0/4192753813 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558fafd08eb0 0x558fafd17850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:40.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.594+0000 7ff5fbdd9740 1 --2- 172.21.15.28:0/4192753813 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558fafd08500 0x558fafd17310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:40.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.594+0000 7ff5fbdd9740 1 -- 172.21.15.28:0/4192753813 >> 172.21.15.28:0/4192753813 conn(0x558fafd045b0 msgr2=0x558fafd03730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:14:40.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.594+0000 7ff5d3e00640 0 client.26555 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:14:40.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.594+0000 7ff5fbdd9740 1 -- 172.21.15.28:0/4192753813 shutdown_connections 2024-05-12T22:14:40.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.594+0000 7ff5fbdd9740 1 -- 172.21.15.28:0/4192753813 wait complete. 2024-05-12T22:14:40.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.594+0000 7ff5fbdd9740 20 client.26555 trim_cache size 0 max 16384 2024-05-12T22:14:40.606 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-05-12T22:14:40.606 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-05-12T22:14:40.606 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "8c85df85-fd12-43c4-8ea6-298f47ad9a26", 2024-05-12T22:14:40.606 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-05-12T22:14:40.606 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-05-12T22:14:40.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.595+0000 7ff5f3200640 1 -- 172.21.15.28:0/994939050 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7ff5d80a2d00 msgr2=0x7ff5d80a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:14:40.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.595+0000 7ff5f3200640 1 --2- 172.21.15.28:0/994939050 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7ff5d80a2d00 0x7ff5d80a51c0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7ff5f406acb0 tx=0x7ff5ec009600 comp rx=0 tx=0).stop 2024-05-12T22:14:40.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.595+0000 7ff5f3200640 1 -- 172.21.15.28:0/994939050 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff5f4075d60 msgr2=0x7ff5f419e600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:14:40.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.595+0000 7ff5f3200640 1 --2- 172.21.15.28:0/994939050 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff5f4075d60 0x7ff5f419e600 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7ff5e4033dc0 tx=0x7ff5e4033df0 comp rx=0 tx=0).stop 2024-05-12T22:14:40.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.595+0000 7ff5f3200640 1 -- 172.21.15.28:0/994939050 shutdown_connections 2024-05-12T22:14:40.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.595+0000 7ff5f3200640 1 --2- 172.21.15.28:0/994939050 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7ff5d80a2d00 0x7ff5d80a51c0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:40.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.595+0000 7ff5f3200640 1 --2- 172.21.15.28:0/994939050 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff5f4112550 0x7ff5f41a2ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:40.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.595+0000 7ff5f3200640 1 --2- 172.21.15.28:0/994939050 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff5f4076680 0x7ff5f419eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:40.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.595+0000 7ff5f3200640 1 --2- 172.21.15.28:0/994939050 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff5f4075d60 0x7ff5f419e600 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:40.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.595+0000 7ff5f3200640 1 -- 172.21.15.28:0/994939050 >> 172.21.15.28:0/994939050 conn(0x7ff5f40ff340 msgr2=0x7ff5f41140a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:14:40.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.595+0000 7ff5f3200640 1 -- 172.21.15.28:0/994939050 shutdown_connections 2024-05-12T22:14:40.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.596+0000 7ff5f3200640 1 -- 172.21.15.28:0/994939050 wait complete. 2024-05-12T22:14:40.616 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-12T22:14:40.616 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-12T22:14:40.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:14:40 smithi186 ceph-mon[34785]: pgmap v173: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:14:40.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:14:40 smithi086 ceph-mon[34297]: pgmap v173: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:14:40.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.849+0000 7f36ea600640 1 -- 172.21.15.28:0/3196384757 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f36e41098f0 msgr2=0x7f36e4109cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:14:40.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.849+0000 7f36ea600640 1 --2- 172.21.15.28:0/3196384757 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f36e41098f0 0x7f36e4109cd0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f36d8009930 tx=0x7f36d802a420 comp rx=0 tx=0).stop 2024-05-12T22:14:40.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.849+0000 7f36ea600640 1 -- 172.21.15.28:0/3196384757 shutdown_connections 2024-05-12T22:14:40.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.849+0000 7f36ea600640 1 --2- 172.21.15.28:0/3196384757 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f36e41042a0 0x7f36e41107b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:40.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.849+0000 7f36ea600640 1 --2- 172.21.15.28:0/3196384757 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f36e41038e0 0x7f36e4103d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:40.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.849+0000 7f36ea600640 1 --2- 172.21.15.28:0/3196384757 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f36e41098f0 0x7f36e4109cd0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:40.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.849+0000 7f36ea600640 1 -- 172.21.15.28:0/3196384757 >> 172.21.15.28:0/3196384757 conn(0x7f36e40ff370 msgr2=0x7f36e4101790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:14:40.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.849+0000 7f36ea600640 1 -- 172.21.15.28:0/3196384757 shutdown_connections 2024-05-12T22:14:40.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.849+0000 7f36ea600640 1 -- 172.21.15.28:0/3196384757 wait complete. 2024-05-12T22:14:40.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.850+0000 7f36ea600640 1 Processor -- start 2024-05-12T22:14:40.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.850+0000 7f36ea600640 1 -- start start 2024-05-12T22:14:40.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.851+0000 7f36ea600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f36e41038e0 0x7f36e419a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:40.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.851+0000 7f36ea600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f36e41042a0 0x7f36e419a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:40.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.851+0000 7f36ea600640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f36e41098f0 0x7f36e419ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:40.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.851+0000 7f36ea600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f36e4113d20 con 0x7f36e41042a0 2024-05-12T22:14:40.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.851+0000 7f36ea600640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f36e4113ba0 con 0x7f36e41038e0 2024-05-12T22:14:40.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.851+0000 7f36ea600640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f36e4113ea0 con 0x7f36e41098f0 2024-05-12T22:14:40.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.851+0000 7f36e9200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f36e41038e0 0x7f36e419a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:40.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.851+0000 7f36e3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f36e41042a0 0x7f36e419a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:40.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.851+0000 7f36e9200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f36e41038e0 0x7f36e419a220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:49862/0 (socket says 172.21.15.28:49862) 2024-05-12T22:14:40.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.851+0000 7f36e9c00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f36e41098f0 0x7f36e419ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:40.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.851+0000 7f36e9200640 1 -- 172.21.15.28:0/2469298462 learned_addr learned my addr 172.21.15.28:0/2469298462 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:14:40.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.852+0000 7f36e9c00640 1 -- 172.21.15.28:0/2469298462 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f36e41038e0 msgr2=0x7f36e419a220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:14:40.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.852+0000 7f36e9c00640 1 --2- 172.21.15.28:0/2469298462 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f36e41038e0 0x7f36e419a220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:40.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.852+0000 7f36e9c00640 1 -- 172.21.15.28:0/2469298462 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f36e41042a0 msgr2=0x7f36e419a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:14:40.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.852+0000 7f36e9c00640 1 --2- 172.21.15.28:0/2469298462 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f36e41042a0 0x7f36e419a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:40.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.852+0000 7f36e9c00640 1 -- 172.21.15.28:0/2469298462 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36e419f1c0 con 0x7f36e41098f0 2024-05-12T22:14:40.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.852+0000 7f36e9c00640 1 --2- 172.21.15.28:0/2469298462 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f36e41098f0 0x7f36e419ea90 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f36dc009fd0 tx=0x7f36dc00ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:14:40.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.853+0000 7f36e1600640 1 -- 172.21.15.28:0/2469298462 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36dc00ce20 con 0x7f36e41098f0 2024-05-12T22:14:40.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.853+0000 7f36e1600640 1 -- 172.21.15.28:0/2469298462 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f36dc005020 con 0x7f36e41098f0 2024-05-12T22:14:40.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.853+0000 7f36ea600640 1 -- 172.21.15.28:0/2469298462 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36e419f4b0 con 0x7f36e41098f0 2024-05-12T22:14:40.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.853+0000 7f36e1600640 1 -- 172.21.15.28:0/2469298462 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36dc010720 con 0x7f36e41098f0 2024-05-12T22:14:40.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.853+0000 7f36ea600640 1 -- 172.21.15.28:0/2469298462 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36e41a6d40 con 0x7f36e41098f0 2024-05-12T22:14:40.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.857+0000 7f36e1600640 1 -- 172.21.15.28:0/2469298462 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7f36dc005540 con 0x7f36e41098f0 2024-05-12T22:14:40.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.857+0000 7f36ee9fd740 1 Processor -- start 2024-05-12T22:14:40.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.857+0000 7f36ee9fd740 1 -- start start 2024-05-12T22:14:40.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.857+0000 7f36ee9fd740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a427a7fdd0 0x55a427a801b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:40.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.857+0000 7f36ee9fd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a427a806f0 0x55a427a8a880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:40.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.858+0000 7f36ee9fd740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a427a8adc0 0x55a427a8d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:40.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.858+0000 7f36ee9fd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a427a482a0 con 0x55a427a806f0 2024-05-12T22:14:40.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.858+0000 7f36ee9fd740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x55a4279e4b90 con 0x55a427a7fdd0 2024-05-12T22:14:40.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.858+0000 7f36ee9fd740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55a42798c5b0 con 0x55a427a8adc0 2024-05-12T22:14:40.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.858+0000 7f36e3e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a427a7fdd0 0x55a427a801b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:40.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.858+0000 7f36e3e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a427a7fdd0 0x55a427a801b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:49878/0 (socket says 172.21.15.28:49878) 2024-05-12T22:14:40.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.858+0000 7f36e3e00640 1 -- 172.21.15.28:0/2376035072 learned_addr learned my addr 172.21.15.28:0/2376035072 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:14:40.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.858+0000 7f36e3e00640 1 --2- 172.21.15.28:0/2376035072 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a427a8adc0 0x55a427a8d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:40.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.858+0000 7f36e9200640 1 --2- 172.21.15.28:0/2376035072 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a427a806f0 0x55a427a8a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:40.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.858+0000 7f36e3e00640 1 -- 172.21.15.28:0/2376035072 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a427a8adc0 msgr2=0x55a427a8d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:14:40.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.858+0000 7f36e3e00640 1 --2- 172.21.15.28:0/2376035072 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a427a8adc0 0x55a427a8d1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:40.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.858+0000 7f36e3e00640 1 -- 172.21.15.28:0/2376035072 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a427a806f0 msgr2=0x55a427a8a880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:14:40.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.858+0000 7f36e3e00640 1 --2- 172.21.15.28:0/2376035072 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a427a806f0 0x55a427a8a880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:40.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.858+0000 7f36e3e00640 1 -- 172.21.15.28:0/2376035072 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a4277c27d0 con 0x55a427a7fdd0 2024-05-12T22:14:40.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.859+0000 7f36e3e00640 1 --2- 172.21.15.28:0/2376035072 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a427a7fdd0 0x55a427a801b0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f36d0009560 tx=0x7f36d0005850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b285a3527d3731a8 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:14:40.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.859+0000 7f36e1600640 1 --2- 172.21.15.28:0/2469298462 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f36cc0a2d80 0x7f36cc0a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:40.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.859+0000 7f36d6000640 1 -- 172.21.15.28:0/2376035072 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36d0013e50 con 0x55a427a7fdd0 2024-05-12T22:14:40.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.859+0000 7f36d6000640 1 -- 172.21.15.28:0/2376035072 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f36d0013120 con 0x55a427a7fdd0 2024-05-12T22:14:40.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.859+0000 7f36d6000640 1 -- 172.21.15.28:0/2376035072 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36d000fea0 con 0x55a427a7fdd0 2024-05-12T22:14:40.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.859+0000 7f36e1600640 1 -- 172.21.15.28:0/2469298462 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f36dc0cfa40 con 0x7f36e41098f0 2024-05-12T22:14:40.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.859+0000 7f36e9200640 1 --2- 172.21.15.28:0/2469298462 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f36cc0a2d80 0x7f36cc0a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:40.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.860+0000 7f36ee9fd740 1 -- 172.21.15.28:0/2376035072 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a427a7fdd0 msgr2=0x55a427a801b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:14:40.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.860+0000 7f36ee9fd740 1 --2- 172.21.15.28:0/2376035072 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a427a7fdd0 0x55a427a801b0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f36d0009560 tx=0x7f36d0005850 comp rx=0 tx=0).stop 2024-05-12T22:14:40.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.860+0000 7f36ee9fd740 1 -- 172.21.15.28:0/2376035072 shutdown_connections 2024-05-12T22:14:40.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.860+0000 7f36ee9fd740 1 --2- 172.21.15.28:0/2376035072 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a427a8adc0 0x55a427a8d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:40.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.860+0000 7f36ee9fd740 1 --2- 172.21.15.28:0/2376035072 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a427a806f0 0x55a427a8a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:40.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.860+0000 7f36ee9fd740 1 --2- 172.21.15.28:0/2376035072 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a427a7fdd0 0x55a427a801b0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:40.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.860+0000 7f36ee9fd740 1 -- 172.21.15.28:0/2376035072 >> 172.21.15.28:0/2376035072 conn(0x55a427a7c0f0 msgr2=0x55a427a7c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:14:40.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.860+0000 7f36ee9fd740 1 -- 172.21.15.28:0/2376035072 shutdown_connections 2024-05-12T22:14:40.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.861+0000 7f36ee9fd740 1 -- 172.21.15.28:0/2376035072 wait complete. 2024-05-12T22:14:40.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.861+0000 7f36e9200640 1 --2- 172.21.15.28:0/2469298462 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f36cc0a2d80 0x7f36cc0a5240 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f36d8031550 tx=0x7f36d8005b50 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:14:40.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.861+0000 7f36ee9fd740 1 -- start start 2024-05-12T22:14:40.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.861+0000 7f36ee9fd740 1 Processor -- start 2024-05-12T22:14:40.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.862+0000 7f36ee9fd740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a427a7fdd0 0x55a427a8a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:40.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.862+0000 7f36ee9fd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a427a806f0 0x55a427a879a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:40.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.862+0000 7f36ee9fd740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a427a8adc0 0x55a427a87ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:40.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.862+0000 7f36ee9fd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a42799f9f0 con 0x55a427a806f0 2024-05-12T22:14:40.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.862+0000 7f36ee9fd740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x55a4279e0b50 con 0x55a427a7fdd0 2024-05-12T22:14:40.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.862+0000 7f36ee9fd740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55a427a8db50 con 0x55a427a8adc0 2024-05-12T22:14:40.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.862+0000 7f36e9200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a427a7fdd0 0x55a427a8a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:40.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.862+0000 7f36e9200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a427a7fdd0 0x55a427a8a6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:49888/0 (socket says 172.21.15.28:49888) 2024-05-12T22:14:40.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.862+0000 7f36e9200640 1 -- 172.21.15.28:0/3466142405 learned_addr learned my addr 172.21.15.28:0/3466142405 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:14:40.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.862+0000 7f36e3e00640 1 --2- 172.21.15.28:0/3466142405 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a427a806f0 0x55a427a879a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:40.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.862+0000 7f36ee9fd740 10 client.0 _ll_register_callbacks cb 0x55a4276f7be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:14:40.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.863+0000 7f36e9c00640 1 --2- 172.21.15.28:0/3466142405 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a427a8adc0 0x55a427a87ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:40.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.863+0000 7f36e9200640 1 -- 172.21.15.28:0/3466142405 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a427a8adc0 msgr2=0x55a427a87ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:14:40.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.863+0000 7f36e9200640 1 --2- 172.21.15.28:0/3466142405 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a427a8adc0 0x55a427a87ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:40.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.863+0000 7f36e9200640 1 -- 172.21.15.28:0/3466142405 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a427a806f0 msgr2=0x55a427a879a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:14:40.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.863+0000 7f36e9200640 1 --2- 172.21.15.28:0/3466142405 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a427a806f0 0x55a427a879a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:40.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.863+0000 7f36e9200640 1 -- 172.21.15.28:0/3466142405 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a4276f83f0 con 0x55a427a7fdd0 2024-05-12T22:14:40.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.863+0000 7f36ee9fd740 1 -- 172.21.15.28:0/3466142405 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f36d0013120 con 0x55a427a7fdd0 2024-05-12T22:14:40.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.863+0000 7f36e9200640 1 --2- 172.21.15.28:0/3466142405 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a427a7fdd0 0x55a427a8a6b0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f36d802a940 tx=0x7f36d8009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7cb006e19e696b79 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:14:40.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.863+0000 7f36caa00640 10 client.26567 ms_handle_connect on v2:172.21.15.86:3300/0 2024-05-12T22:14:40.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.864+0000 7f36caa00640 1 -- 172.21.15.28:0/3466142405 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36d8057050 con 0x55a427a7fdd0 2024-05-12T22:14:40.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.864+0000 7f36caa00640 1 -- 172.21.15.28:0/3466142405 <== mon.2 v2:172.21.15.86:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f36d8004b90 con 0x55a427a7fdd0 2024-05-12T22:14:40.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.864+0000 7f36caa00640 1 -- 172.21.15.28:0/3466142405 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f36d804d070 con 0x55a427a7fdd0 2024-05-12T22:14:40.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.864+0000 7f36ee9fd740 10 client.26567 fetch_fsmap learned FSMap version 26 2024-05-12T22:14:40.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.864+0000 7f36ee9fd740 1 -- 172.21.15.28:0/3466142405 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({fsmap=26}) -- 0x55a4276b80f0 con 0x55a427a7fdd0 2024-05-12T22:14:40.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.864+0000 7f36caa00640 1 -- 172.21.15.28:0/3466142405 <== mon.2 v2:172.21.15.86:3300/0 4 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f36d80325c0 con 0x55a427a7fdd0 2024-05-12T22:14:40.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.865+0000 7f36ee9fd740 10 client.26567 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:14:40.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.865+0000 7f36ee9fd740 10 client.26567 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:14:40.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.865+0000 7f36ee9fd740 20 client.26567 populate_metadata read hostname 'smithi028' 2024-05-12T22:14:40.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.865+0000 7f36ee9fd740 1 --2- 172.21.15.28:0/3466142405 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55a427aa2fa0 0x55a427aa5390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:40.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.865+0000 7f36ee9fd740 4 client.26567 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-12T22:14:40.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.865+0000 7f36ee9fd740 1 -- 172.21.15.28:0/3466142405 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a4279f22c0 con 0x55a427aa2fa0 2024-05-12T22:14:40.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.865+0000 7f36e3e00640 1 --2- 172.21.15.28:0/3466142405 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55a427aa2fa0 0x55a427aa5390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:40.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.866+0000 7f36e3e00640 1 --2- 172.21.15.28:0/3466142405 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55a427aa2fa0 0x55a427aa5390 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=96bf78f18e3ef07b server_cookie=784c17b3b621b9ac in_seq=0 out_seq=0 2024-05-12T22:14:40.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.866+0000 7f36caa00640 10 client.26567 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:14:40.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.867+0000 7f36caa00640 1 -- 172.21.15.28:0/3466142405 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a4279f22c0 con 0x55a427aa2fa0 2024-05-12T22:14:40.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.867+0000 7f36caa00640 10 client.26567 handle_command_reply: tid=0 2024-05-12T22:14:40.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.867+0000 7f36ee9fd740 1 client.26567 shutdown 2024-05-12T22:14:40.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.867+0000 7f36ee9fd740 1 -- 172.21.15.28:0/3466142405 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a427a7fdd0 msgr2=0x55a427a8a6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:14:40.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.867+0000 7f36ee9fd740 1 --2- 172.21.15.28:0/3466142405 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a427a7fdd0 0x55a427a8a6b0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f36d802a940 tx=0x7f36d8009260 comp rx=0 tx=0).stop 2024-05-12T22:14:40.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.867+0000 7f36ee9fd740 1 -- 172.21.15.28:0/3466142405 shutdown_connections 2024-05-12T22:14:40.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.867+0000 7f36ee9fd740 1 --2- 172.21.15.28:0/3466142405 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55a427aa2fa0 0x55a427aa5390 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:40.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.867+0000 7f36ee9fd740 1 --2- 172.21.15.28:0/3466142405 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a427a8adc0 0x55a427a87ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:40.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.867+0000 7f36ee9fd740 1 --2- 172.21.15.28:0/3466142405 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a427a806f0 0x55a427a879a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:40.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.867+0000 7f36ee9fd740 1 --2- 172.21.15.28:0/3466142405 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a427a7fdd0 0x55a427a8a6b0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:40.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.867+0000 7f36caa00640 0 client.26567 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:14:40.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.867+0000 7f36ee9fd740 1 -- 172.21.15.28:0/3466142405 >> 172.21.15.28:0/3466142405 conn(0x55a427a7c0f0 msgr2=0x55a427a7b330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:14:40.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.867+0000 7f36ee9fd740 1 -- 172.21.15.28:0/3466142405 shutdown_connections 2024-05-12T22:14:40.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.867+0000 7f36ee9fd740 1 -- 172.21.15.28:0/3466142405 wait complete. 2024-05-12T22:14:40.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.868+0000 7f36ee9fd740 20 client.26567 trim_cache size 0 max 16384 2024-05-12T22:14:40.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.893+0000 7f36ee9fd740 1 Processor -- start 2024-05-12T22:14:40.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.893+0000 7f36ee9fd740 1 -- start start 2024-05-12T22:14:40.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.893+0000 7f36ee9fd740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a427a7fdd0 0x55a427aaac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:40.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.893+0000 7f36ee9fd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a427a806f0 0x55a427aab140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:40.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.893+0000 7f36ee9fd740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a427a8adc0 0x55a427a84c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:40.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.893+0000 7f36ee9fd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f36e41ad8b0 con 0x55a427a806f0 2024-05-12T22:14:40.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.893+0000 7f36ee9fd740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f36e41ab180 con 0x55a427a7fdd0 2024-05-12T22:14:40.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.893+0000 7f36ee9fd740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f36e41a8fd0 con 0x55a427a8adc0 2024-05-12T22:14:40.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.894+0000 7f36e9200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a427a7fdd0 0x55a427aaac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:40.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.894+0000 7f36e9c00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a427a8adc0 0x55a427a84c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:40.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.894+0000 7f36e9200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a427a7fdd0 0x55a427aaac00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:49894/0 (socket says 172.21.15.28:49894) 2024-05-12T22:14:40.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.894+0000 7f36e9200640 1 -- 172.21.15.28:0/1318999914 learned_addr learned my addr 172.21.15.28:0/1318999914 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:14:40.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.894+0000 7f36e3e00640 1 --2- 172.21.15.28:0/1318999914 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a427a806f0 0x55a427aab140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:40.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.894+0000 7f36e9c00640 1 -- 172.21.15.28:0/1318999914 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a427a7fdd0 msgr2=0x55a427aaac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:14:40.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.894+0000 7f36e9c00640 1 --2- 172.21.15.28:0/1318999914 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a427a7fdd0 0x55a427aaac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:40.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.894+0000 7f36e9c00640 1 -- 172.21.15.28:0/1318999914 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a427a806f0 msgr2=0x55a427aab140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:14:40.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.894+0000 7f36e9c00640 1 --2- 172.21.15.28:0/1318999914 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a427a806f0 0x55a427aab140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:40.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.894+0000 7f36e9c00640 1 -- 172.21.15.28:0/1318999914 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a427aa5700 con 0x55a427a8adc0 2024-05-12T22:14:40.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.895+0000 7f36e9c00640 1 --2- 172.21.15.28:0/1318999914 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a427a8adc0 0x55a427a84c80 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f36dc0cd160 tx=0x7f36dc09c630 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c888477233f75b23 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:14:40.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.895+0000 7f36d5600640 1 -- 172.21.15.28:0/1318999914 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36dc0b4070 con 0x55a427a8adc0 2024-05-12T22:14:40.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.895+0000 7f36d5600640 1 -- 172.21.15.28:0/1318999914 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f36dc0a4ba0 con 0x55a427a8adc0 2024-05-12T22:14:40.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.895+0000 7f36d5600640 1 -- 172.21.15.28:0/1318999914 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36dc09c3f0 con 0x55a427a8adc0 2024-05-12T22:14:40.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.896+0000 7f36ee9fd740 1 -- 172.21.15.28:0/1318999914 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a427a8adc0 msgr2=0x55a427a84c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:14:40.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.896+0000 7f36ee9fd740 1 --2- 172.21.15.28:0/1318999914 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a427a8adc0 0x55a427a84c80 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f36dc0cd160 tx=0x7f36dc09c630 comp rx=0 tx=0).stop 2024-05-12T22:14:40.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.896+0000 7f36ee9fd740 1 -- 172.21.15.28:0/1318999914 shutdown_connections 2024-05-12T22:14:40.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.896+0000 7f36ee9fd740 1 --2- 172.21.15.28:0/1318999914 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a427a8adc0 0x55a427a84c80 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:40.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.896+0000 7f36ee9fd740 1 --2- 172.21.15.28:0/1318999914 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a427a806f0 0x55a427aab140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:40.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.896+0000 7f36ee9fd740 1 --2- 172.21.15.28:0/1318999914 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a427a7fdd0 0x55a427aaac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:40.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.896+0000 7f36ee9fd740 1 -- 172.21.15.28:0/1318999914 >> 172.21.15.28:0/1318999914 conn(0x55a427a7c0f0 msgr2=0x55a427a9e170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:14:40.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.896+0000 7f36ee9fd740 1 -- 172.21.15.28:0/1318999914 shutdown_connections 2024-05-12T22:14:40.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.896+0000 7f36ee9fd740 1 -- 172.21.15.28:0/1318999914 wait complete. 2024-05-12T22:14:40.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.897+0000 7f36ee9fd740 1 -- start start 2024-05-12T22:14:40.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.897+0000 7f36ee9fd740 1 Processor -- start 2024-05-12T22:14:40.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.897+0000 7f36ee9fd740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a427a7fdd0 0x55a427a99350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:40.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.897+0000 7f36ee9fd740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a427a806f0 0x55a427a99890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:40.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.897+0000 7f36ee9fd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a427a8adc0 0x55a427a99dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:40.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.897+0000 7f36ee9fd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f36e41a9640 con 0x55a427a8adc0 2024-05-12T22:14:40.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.898+0000 7f36ee9fd740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f36e41aaab0 con 0x55a427a806f0 2024-05-12T22:14:40.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.898+0000 7f36ee9fd740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f36e41a9d10 con 0x55a427a7fdd0 2024-05-12T22:14:40.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.898+0000 7f36ee9fd740 10 client.0 _ll_register_callbacks cb 0x7f36d0005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:14:40.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.898+0000 7f36e3e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a427a806f0 0x55a427a99890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:40.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.898+0000 7f36e3e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a427a806f0 0x55a427a99890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:49904/0 (socket says 172.21.15.28:49904) 2024-05-12T22:14:40.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.898+0000 7f36e3e00640 1 -- 172.21.15.28:0/413912923 learned_addr learned my addr 172.21.15.28:0/413912923 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:14:40.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.898+0000 7f36e9c00640 1 --2- 172.21.15.28:0/413912923 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a427a8adc0 0x55a427a99dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:40.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.898+0000 7f36e9200640 1 --2- 172.21.15.28:0/413912923 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a427a7fdd0 0x55a427a99350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:40.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.898+0000 7f36e3e00640 1 -- 172.21.15.28:0/413912923 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a427a7fdd0 msgr2=0x55a427a99350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:14:40.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.898+0000 7f36e3e00640 1 --2- 172.21.15.28:0/413912923 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a427a7fdd0 0x55a427a99350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:40.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.898+0000 7f36e3e00640 1 -- 172.21.15.28:0/413912923 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a427a8adc0 msgr2=0x55a427a99dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:14:40.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.898+0000 7f36e3e00640 1 --2- 172.21.15.28:0/413912923 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a427a8adc0 0x55a427a99dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:40.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.898+0000 7f36e3e00640 1 -- 172.21.15.28:0/413912923 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a427b086d0 con 0x55a427a806f0 2024-05-12T22:14:40.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.899+0000 7f36e3e00640 1 --2- 172.21.15.28:0/413912923 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a427a806f0 0x55a427a99890 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f36d0000c00 tx=0x7f36d0009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=355f3264244e76f3 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:14:40.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.899+0000 7f36caa00640 10 client.0 ms_handle_connect on v2:172.21.15.86:3300/0 2024-05-12T22:14:40.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.899+0000 7f36caa00640 1 -- 172.21.15.28:0/413912923 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36d0029dc0 con 0x55a427a806f0 2024-05-12T22:14:40.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.899+0000 7f36caa00640 1 -- 172.21.15.28:0/413912923 <== mon.2 v2:172.21.15.86:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f36d0029550 con 0x55a427a806f0 2024-05-12T22:14:40.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.899+0000 7f36ee9fd740 1 -- 172.21.15.28:0/413912923 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f36dc0a4ba0 con 0x55a427a806f0 2024-05-12T22:14:40.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.900+0000 7f36caa00640 1 -- 172.21.15.28:0/413912923 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f36d0029750 con 0x55a427a806f0 2024-05-12T22:14:40.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.900+0000 7f36ee9fd740 10 client.26573 fetch_fsmap learned FSMap version 26 2024-05-12T22:14:40.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.900+0000 7f36ee9fd740 1 -- 172.21.15.28:0/413912923 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55a427b099c0 con 0x55a427a806f0 2024-05-12T22:14:40.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.900+0000 7f36caa00640 1 -- 172.21.15.28:0/413912923 <== mon.2 v2:172.21.15.86:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f36dc0a4ba0 con 0x55a427a806f0 2024-05-12T22:14:40.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.900+0000 7f36caa00640 1 -- 172.21.15.28:0/413912923 <== mon.2 v2:172.21.15.86:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f36d000faf0 con 0x55a427a806f0 2024-05-12T22:14:40.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.900+0000 7f36ee9fd740 10 client.26573 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:14:40.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.900+0000 7f36ee9fd740 10 client.26573 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:14:40.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.900+0000 7f36ee9fd740 20 client.26573 populate_metadata read hostname 'smithi028' 2024-05-12T22:14:40.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.900+0000 7f36ee9fd740 1 --2- 172.21.15.28:0/413912923 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55a427b09ea0 0x55a427b0c290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:14:40.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.900+0000 7f36ee9fd740 4 client.26573 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-12T22:14:40.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.900+0000 7f36ee9fd740 1 -- 172.21.15.28:0/413912923 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55a4279ec610 con 0x55a427b09ea0 2024-05-12T22:14:40.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.900+0000 7f36e9200640 1 --2- 172.21.15.28:0/413912923 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55a427b09ea0 0x55a427b0c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:14:40.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.901+0000 7f36e9200640 1 --2- 172.21.15.28:0/413912923 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55a427b09ea0 0x55a427b0c290 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d5ee6a1fd73c792e server_cookie=16c9e8b768738492 in_seq=0 out_seq=0 2024-05-12T22:14:40.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.901+0000 7f36caa00640 10 client.26573 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:14:40.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.902+0000 7f36caa00640 1 -- 172.21.15.28:0/413912923 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+268 (crc 0 0 0) 0x55a4279ec610 con 0x55a427b09ea0 2024-05-12T22:14:40.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.902+0000 7f36caa00640 10 client.26573 handle_command_reply: tid=0 2024-05-12T22:14:40.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.902+0000 7f36ee9fd740 1 client.26573 shutdown 2024-05-12T22:14:40.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.902+0000 7f36ee9fd740 1 -- 172.21.15.28:0/413912923 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a427a806f0 msgr2=0x55a427a99890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:14:40.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.902+0000 7f36ee9fd740 1 --2- 172.21.15.28:0/413912923 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a427a806f0 0x55a427a99890 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f36d0000c00 tx=0x7f36d0009b50 comp rx=0 tx=0).stop 2024-05-12T22:14:40.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.902+0000 7f36ee9fd740 1 -- 172.21.15.28:0/413912923 shutdown_connections 2024-05-12T22:14:40.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.902+0000 7f36ee9fd740 1 --2- 172.21.15.28:0/413912923 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55a427b09ea0 0x55a427b0c290 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:40.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.902+0000 7f36ee9fd740 1 --2- 172.21.15.28:0/413912923 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a427a8adc0 0x55a427a99dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:40.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.902+0000 7f36ee9fd740 1 --2- 172.21.15.28:0/413912923 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a427a806f0 0x55a427a99890 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:40.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.902+0000 7f36ee9fd740 1 --2- 172.21.15.28:0/413912923 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a427a7fdd0 0x55a427a99350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:40.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.902+0000 7f36caa00640 0 client.26573 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:14:40.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.902+0000 7f36ee9fd740 1 -- 172.21.15.28:0/413912923 >> 172.21.15.28:0/413912923 conn(0x55a427a7c0f0 msgr2=0x55a427a89a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:14:40.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.902+0000 7f36ee9fd740 1 -- 172.21.15.28:0/413912923 shutdown_connections 2024-05-12T22:14:40.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.902+0000 7f36ee9fd740 1 -- 172.21.15.28:0/413912923 wait complete. 2024-05-12T22:14:40.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.903+0000 7f36ee9fd740 20 client.26573 trim_cache size 0 max 16384 2024-05-12T22:14:40.909 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-05-12T22:14:40.909 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "scrub active (1158 inodes in the stack)", 2024-05-12T22:14:40.909 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-05-12T22:14:40.909 INFO:teuthology.orchestra.run.smithi028.stdout: "8c85df85-fd12-43c4-8ea6-298f47ad9a26": { 2024-05-12T22:14:40.909 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-05-12T22:14:40.909 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "8c85df85-fd12-43c4-8ea6-298f47ad9a26", 2024-05-12T22:14:40.910 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-05-12T22:14:40.910 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:14:40.910 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:14:40.910 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-05-12T22:14:40.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.903+0000 7f36ea600640 1 -- 172.21.15.28:0/2469298462 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f36cc0a2d80 msgr2=0x7f36cc0a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:14:40.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.903+0000 7f36ea600640 1 --2- 172.21.15.28:0/2469298462 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f36cc0a2d80 0x7f36cc0a5240 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f36d8031550 tx=0x7f36d8005b50 comp rx=0 tx=0).stop 2024-05-12T22:14:40.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.903+0000 7f36ea600640 1 -- 172.21.15.28:0/2469298462 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f36e41098f0 msgr2=0x7f36e419ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:14:40.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.903+0000 7f36ea600640 1 --2- 172.21.15.28:0/2469298462 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f36e41098f0 0x7f36e419ea90 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f36dc009fd0 tx=0x7f36dc00ef10 comp rx=0 tx=0).stop 2024-05-12T22:14:40.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.904+0000 7f36ea600640 1 -- 172.21.15.28:0/2469298462 shutdown_connections 2024-05-12T22:14:40.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.904+0000 7f36ea600640 1 --2- 172.21.15.28:0/2469298462 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f36cc0a2d80 0x7f36cc0a5240 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:40.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.904+0000 7f36ea600640 1 --2- 172.21.15.28:0/2469298462 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f36e41098f0 0x7f36e419ea90 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:40.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.904+0000 7f36ea600640 1 --2- 172.21.15.28:0/2469298462 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f36e41042a0 0x7f36e419a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:40.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.904+0000 7f36ea600640 1 --2- 172.21.15.28:0/2469298462 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f36e41038e0 0x7f36e419a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:14:40.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.904+0000 7f36ea600640 1 -- 172.21.15.28:0/2469298462 >> 172.21.15.28:0/2469298462 conn(0x7f36e40ff370 msgr2=0x7f36e40ffba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:14:40.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.904+0000 7f36ea600640 1 -- 172.21.15.28:0/2469298462 shutdown_connections 2024-05-12T22:14:40.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:14:40.904+0000 7f36ea600640 1 -- 172.21.15.28:0/2469298462 wait complete. 2024-05-12T22:14:40.925 INFO:tasks.cephfs.filesystem:scrub status for tag:8c85df85-fd12-43c4-8ea6-298f47ad9a26 - {'path': '/', 'tag': '8c85df85-fd12-43c4-8ea6-298f47ad9a26', 'options': 'recursive,force'} 2024-05-12T22:14:41.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:14:41 smithi028 ceph-mon[29847]: scrub summary: active paths [/] 2024-05-12T22:14:41.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:14:41 smithi028 ceph-mon[29847]: scrub summary: idle+waiting paths [/] 2024-05-12T22:14:41.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:14:41 smithi028 ceph-mon[29847]: scrub summary: active paths [/] 2024-05-12T22:14:41.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:14:41 smithi028 ceph-mon[29847]: scrub summary: idle+waiting paths [/] 2024-05-12T22:14:41.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:14:41 smithi186 ceph-mon[34785]: scrub summary: active paths [/] 2024-05-12T22:14:41.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:14:41 smithi186 ceph-mon[34785]: scrub summary: idle+waiting paths [/] 2024-05-12T22:14:41.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:14:41 smithi186 ceph-mon[34785]: scrub summary: active paths [/] 2024-05-12T22:14:41.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:14:41 smithi186 ceph-mon[34785]: scrub summary: idle+waiting paths [/] 2024-05-12T22:14:41.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:14:41 smithi086 ceph-mon[34297]: scrub summary: active paths [/] 2024-05-12T22:14:41.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:14:41 smithi086 ceph-mon[34297]: scrub summary: idle+waiting paths [/] 2024-05-12T22:14:41.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:14:41 smithi086 ceph-mon[34297]: scrub summary: active paths [/] 2024-05-12T22:14:41.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:14:41 smithi086 ceph-mon[34297]: scrub summary: idle+waiting paths [/] 2024-05-12T22:14:42.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:14:42 smithi186 ceph-mon[34785]: scrub queued for path: / 2024-05-12T22:14:42.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:14:42 smithi186 ceph-mon[34785]: scrub summary: idle+waiting paths [/] 2024-05-12T22:14:42.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:14:42 smithi186 ceph-mon[34785]: scrub summary: active paths [/] 2024-05-12T22:14:42.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:14:42 smithi186 ceph-mon[34785]: pgmap v174: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-12T22:14:42.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:14:42 smithi186 ceph-mon[34785]: scrub summary: active paths [/] 2024-05-12T22:14:42.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:14:42 smithi186 ceph-mon[34785]: scrub summary: idle+waiting paths [/] 2024-05-12T22:14:42.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:14:42 smithi186 ceph-mon[34785]: scrub summary: idle+waiting paths [/] 2024-05-12T22:14:42.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:14:42 smithi086 ceph-mon[34297]: scrub queued for path: / 2024-05-12T22:14:42.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:14:42 smithi086 ceph-mon[34297]: scrub summary: idle+waiting paths [/] 2024-05-12T22:14:42.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:14:42 smithi086 ceph-mon[34297]: scrub summary: active paths [/] 2024-05-12T22:14:42.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:14:42 smithi086 ceph-mon[34297]: pgmap v174: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-12T22:14:42.743 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:14:42 smithi086 ceph-mon[34297]: scrub summary: active paths [/] 2024-05-12T22:14:42.743 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:14:42 smithi086 ceph-mon[34297]: scrub summary: idle+waiting paths [/] 2024-05-12T22:14:42.743 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:14:42 smithi086 ceph-mon[34297]: scrub summary: idle+waiting paths [/] 2024-05-12T22:14:42.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:14:42 smithi028 ceph-mon[29847]: scrub queued for path: / 2024-05-12T22:14:42.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:14:42 smithi028 ceph-mon[29847]: scrub summary: idle+waiting paths [/] 2024-05-12T22:14:42.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:14:42 smithi028 ceph-mon[29847]: scrub summary: active paths [/] 2024-05-12T22:14:42.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:14:42 smithi028 ceph-mon[29847]: pgmap v174: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-12T22:14:42.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:14:42 smithi028 ceph-mon[29847]: scrub summary: active paths [/] 2024-05-12T22:14:42.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:14:42 smithi028 ceph-mon[29847]: scrub summary: idle+waiting paths [/] 2024-05-12T22:14:42.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:14:42 smithi028 ceph-mon[29847]: scrub summary: idle+waiting paths [/] 2024-05-12T22:14:44.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:14:44 smithi186 ceph-mon[34785]: pgmap v175: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:14:44.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:14:44 smithi086 ceph-mon[34297]: pgmap v175: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:14:44.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:14:44 smithi028 ceph-mon[29847]: pgmap v175: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:14:46.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:14:46 smithi186 ceph-mon[34785]: pgmap v176: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-05-12T22:14:46.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:14:46 smithi086 ceph-mon[34297]: pgmap v176: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-05-12T22:14:46.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:14:46 smithi028 ceph-mon[29847]: pgmap v176: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-05-12T22:14:48.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:14:48 smithi186 ceph-mon[34785]: pgmap v177: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-05-12T22:14:48.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:14:48 smithi186 ceph-mon[34785]: scrub summary: idle 2024-05-12T22:14:48.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:14:48 smithi186 ceph-mon[34785]: scrub summary: idle 2024-05-12T22:14:48.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:14:48 smithi086 ceph-mon[34297]: pgmap v177: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-05-12T22:14:48.743 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:14:48 smithi086 ceph-mon[34297]: scrub summary: idle 2024-05-12T22:14:48.743 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:14:48 smithi086 ceph-mon[34297]: scrub summary: idle 2024-05-12T22:14:48.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:14:48 smithi028 ceph-mon[29847]: pgmap v177: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-05-12T22:14:48.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:14:48 smithi028 ceph-mon[29847]: scrub summary: idle 2024-05-12T22:14:48.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:14:48 smithi028 ceph-mon[29847]: scrub summary: idle 2024-05-12T22:14:50.588 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:14:50 smithi028 ceph-mon[29847]: pgmap v178: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-05-12T22:14:50.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:14:50 smithi186 ceph-mon[34785]: pgmap v178: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-05-12T22:14:50.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:14:50 smithi086 ceph-mon[34297]: pgmap v178: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-05-12T22:14:51.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:14:51 smithi186 ceph-mon[34785]: pgmap v179: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-05-12T22:14:51.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:14:51 smithi086 ceph-mon[34297]: pgmap v179: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-05-12T22:14:51.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:14:51 smithi028 ceph-mon[29847]: pgmap v179: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-05-12T22:14:54.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:14:53 smithi186 ceph-mon[34785]: pgmap v180: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-05-12T22:14:54.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:14:53 smithi086 ceph-mon[34297]: pgmap v180: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-05-12T22:14:54.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:14:53 smithi028 ceph-mon[29847]: pgmap v180: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-05-12T22:14:56.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:14:55 smithi186 ceph-mon[34785]: pgmap v181: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-05-12T22:14:56.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:14:55 smithi086 ceph-mon[34297]: pgmap v181: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-05-12T22:14:56.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:14:55 smithi028 ceph-mon[29847]: pgmap v181: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-05-12T22:14:58.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:14:57 smithi186 ceph-mon[34785]: pgmap v182: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:14:58.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:14:57 smithi086 ceph-mon[34297]: pgmap v182: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:14:58.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:14:57 smithi028 ceph-mon[29847]: pgmap v182: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:15:00.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:14:59 smithi186 ceph-mon[34785]: pgmap v183: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:15:00.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:14:59 smithi086 ceph-mon[34297]: pgmap v183: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:15:00.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:14:59 smithi028 ceph-mon[29847]: pgmap v183: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:15:02.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:15:01 smithi186 ceph-mon[34785]: pgmap v184: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-12T22:15:02.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:15:01 smithi086 ceph-mon[34297]: pgmap v184: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-12T22:15:02.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:15:01 smithi028 ceph-mon[29847]: pgmap v184: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-12T22:15:04.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:15:03 smithi186 ceph-mon[34785]: pgmap v185: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:15:04.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:15:03 smithi086 ceph-mon[34297]: pgmap v185: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:15:04.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:15:03 smithi028 ceph-mon[29847]: pgmap v185: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:15:06.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:15:05 smithi186 ceph-mon[34785]: pgmap v186: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-05-12T22:15:06.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:15:05 smithi086 ceph-mon[34297]: pgmap v186: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-05-12T22:15:06.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:15:05 smithi028 ceph-mon[29847]: pgmap v186: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-05-12T22:15:08.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:15:07 smithi186 ceph-mon[34785]: pgmap v187: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-05-12T22:15:08.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:15:07 smithi086 ceph-mon[34297]: pgmap v187: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-05-12T22:15:08.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:15:07 smithi028 ceph-mon[29847]: pgmap v187: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-05-12T22:15:10.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:15:09 smithi186 ceph-mon[34785]: pgmap v188: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-05-12T22:15:10.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:15:09 smithi086 ceph-mon[34297]: pgmap v188: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-05-12T22:15:10.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:15:09 smithi028 ceph-mon[29847]: pgmap v188: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-05-12T22:15:10.926 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-12T22:15:10.927 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-12T22:15:11.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.155+0000 7f87e7200640 1 -- 172.21.15.28:0/906204650 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f87e80744f0 msgr2=0x7f87e81121a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:15:11.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.155+0000 7f87e7200640 1 --2- 172.21.15.28:0/906204650 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f87e80744f0 0x7f87e81121a0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f87d4009860 tx=0x7f87d402a3d0 comp rx=0 tx=0).stop 2024-05-12T22:15:11.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.155+0000 7f87e7200640 1 -- 172.21.15.28:0/906204650 shutdown_connections 2024-05-12T22:15:11.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.155+0000 7f87e7200640 1 --2- 172.21.15.28:0/906204650 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87e81126e0 0x7f87e8114ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:11.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.155+0000 7f87e7200640 1 --2- 172.21.15.28:0/906204650 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f87e80744f0 0x7f87e81121a0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:11.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.155+0000 7f87e7200640 1 --2- 172.21.15.28:0/906204650 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f87e8073bd0 0x7f87e8073fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:11.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.155+0000 7f87e7200640 1 -- 172.21.15.28:0/906204650 >> 172.21.15.28:0/906204650 conn(0x7f87e80fd200 msgr2=0x7f87e80ff620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:15:11.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.155+0000 7f87e7200640 1 -- 172.21.15.28:0/906204650 shutdown_connections 2024-05-12T22:15:11.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.155+0000 7f87e7200640 1 -- 172.21.15.28:0/906204650 wait complete. 2024-05-12T22:15:11.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.156+0000 7f87e7200640 1 Processor -- start 2024-05-12T22:15:11.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.157+0000 7f87e7200640 1 -- start start 2024-05-12T22:15:11.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.157+0000 7f87e7200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f87e8073bd0 0x7f87e819e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:11.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.157+0000 7f87e7200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87e80744f0 0x7f87e819eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:11.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.157+0000 7f87e7200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f87e81126e0 0x7f87e81a2f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:11.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.157+0000 7f87e7200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f87e8118120 con 0x7f87e80744f0 2024-05-12T22:15:11.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.157+0000 7f87e7200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f87e8117fa0 con 0x7f87e81126e0 2024-05-12T22:15:11.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.157+0000 7f87e7200640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f87e81182a0 con 0x7f87e8073bd0 2024-05-12T22:15:11.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.157+0000 7f87e5e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f87e8073bd0 0x7f87e819e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:11.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.157+0000 7f87e5e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f87e8073bd0 0x7f87e819e640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:36278/0 (socket says 172.21.15.28:36278) 2024-05-12T22:15:11.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.157+0000 7f87e5e00640 1 -- 172.21.15.28:0/4255956323 learned_addr learned my addr 172.21.15.28:0/4255956323 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:15:11.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.157+0000 7f87e5400640 1 --2- 172.21.15.28:0/4255956323 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87e80744f0 0x7f87e819eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:11.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.157+0000 7f87e6800640 1 --2- 172.21.15.28:0/4255956323 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f87e81126e0 0x7f87e81a2f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:11.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.158+0000 7f87e5e00640 1 -- 172.21.15.28:0/4255956323 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f87e81126e0 msgr2=0x7f87e81a2f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:15:11.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.158+0000 7f87e5e00640 1 --2- 172.21.15.28:0/4255956323 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f87e81126e0 0x7f87e81a2f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:11.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.158+0000 7f87e5e00640 1 -- 172.21.15.28:0/4255956323 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87e80744f0 msgr2=0x7f87e819eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:15:11.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.158+0000 7f87e5e00640 1 --2- 172.21.15.28:0/4255956323 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87e80744f0 0x7f87e819eb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:11.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.158+0000 7f87e5e00640 1 -- 172.21.15.28:0/4255956323 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87e81a3630 con 0x7f87e8073bd0 2024-05-12T22:15:11.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.158+0000 7f87e5e00640 1 --2- 172.21.15.28:0/4255956323 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f87e8073bd0 0x7f87e819e640 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f87d800e9f0 tx=0x7f87d800edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:15:11.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.158+0000 7f87dea00640 1 -- 172.21.15.28:0/4255956323 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87d800cd80 con 0x7f87e8073bd0 2024-05-12T22:15:11.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.158+0000 7f87dea00640 1 -- 172.21.15.28:0/4255956323 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f87d8005500 con 0x7f87e8073bd0 2024-05-12T22:15:11.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.158+0000 7f87e7200640 1 -- 172.21.15.28:0/4255956323 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87e81a3920 con 0x7f87e8073bd0 2024-05-12T22:15:11.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.158+0000 7f87dea00640 1 -- 172.21.15.28:0/4255956323 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87d8010690 con 0x7f87e8073bd0 2024-05-12T22:15:11.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.159+0000 7f87e7200640 1 -- 172.21.15.28:0/4255956323 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87e81ab1b0 con 0x7f87e8073bd0 2024-05-12T22:15:11.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.162+0000 7f87ef9db740 1 Processor -- start 2024-05-12T22:15:11.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.162+0000 7f87ef9db740 1 -- start start 2024-05-12T22:15:11.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.162+0000 7f87dea00640 1 -- 172.21.15.28:0/4255956323 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7f87d80046c0 con 0x7f87e8073bd0 2024-05-12T22:15:11.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.162+0000 7f87ef9db740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e202e3ddd0 0x55e202e3e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:11.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.162+0000 7f87ef9db740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e202e3e6f0 0x55e202e48880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:11.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.162+0000 7f87ef9db740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e202e48dc0 0x55e202e4b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:11.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.162+0000 7f87ef9db740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e202e062a0 con 0x55e202e3e6f0 2024-05-12T22:15:11.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.162+0000 7f87ef9db740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x55e202da2b90 con 0x55e202e3ddd0 2024-05-12T22:15:11.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.162+0000 7f87ef9db740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55e202d4a5b0 con 0x55e202e48dc0 2024-05-12T22:15:11.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.163+0000 7f87e6800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e202e3ddd0 0x55e202e3e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:11.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.163+0000 7f87e5400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e202e48dc0 0x55e202e4b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:11.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.163+0000 7f87e6800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e202e3ddd0 0x55e202e3e1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:55092/0 (socket says 172.21.15.28:55092) 2024-05-12T22:15:11.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.163+0000 7f87e6800640 1 -- 172.21.15.28:0/3758943596 learned_addr learned my addr 172.21.15.28:0/3758943596 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:15:11.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.163+0000 7f87e5e00640 1 --2- 172.21.15.28:0/3758943596 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e202e3e6f0 0x55e202e48880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:11.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.163+0000 7f87e6800640 1 -- 172.21.15.28:0/3758943596 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e202e48dc0 msgr2=0x55e202e4b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:15:11.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.163+0000 7f87e6800640 1 --2- 172.21.15.28:0/3758943596 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e202e48dc0 0x55e202e4b1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:11.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.163+0000 7f87e6800640 1 -- 172.21.15.28:0/3758943596 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e202e3e6f0 msgr2=0x55e202e48880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:15:11.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.163+0000 7f87e6800640 1 --2- 172.21.15.28:0/3758943596 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e202e3e6f0 0x55e202e48880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:11.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.163+0000 7f87e6800640 1 -- 172.21.15.28:0/3758943596 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e202b807d0 con 0x55e202e3ddd0 2024-05-12T22:15:11.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.164+0000 7f87e6800640 1 --2- 172.21.15.28:0/3758943596 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e202e3ddd0 0x55e202e3e1b0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f87e000e480 tx=0x7f87e00095c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c09a6d75b2aa1a84 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:15:11.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.164+0000 7f87dea00640 1 --2- 172.21.15.28:0/4255956323 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f87cc0a2d80 0x7f87cc0a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:11.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.164+0000 7f87d3000640 1 -- 172.21.15.28:0/3758943596 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87e0003970 con 0x55e202e3ddd0 2024-05-12T22:15:11.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.164+0000 7f87d3000640 1 -- 172.21.15.28:0/3758943596 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f87e000d7d0 con 0x55e202e3ddd0 2024-05-12T22:15:11.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.164+0000 7f87d3000640 1 -- 172.21.15.28:0/3758943596 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87e0004460 con 0x55e202e3ddd0 2024-05-12T22:15:11.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.164+0000 7f87dea00640 1 -- 172.21.15.28:0/4255956323 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f87d80cf800 con 0x7f87e8073bd0 2024-05-12T22:15:11.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.164+0000 7f87e5e00640 1 --2- 172.21.15.28:0/4255956323 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f87cc0a2d80 0x7f87cc0a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:11.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.165+0000 7f87ef9db740 1 -- 172.21.15.28:0/3758943596 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e202e3ddd0 msgr2=0x55e202e3e1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:15:11.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.165+0000 7f87ef9db740 1 --2- 172.21.15.28:0/3758943596 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e202e3ddd0 0x55e202e3e1b0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f87e000e480 tx=0x7f87e00095c0 comp rx=0 tx=0).stop 2024-05-12T22:15:11.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.165+0000 7f87ef9db740 1 -- 172.21.15.28:0/3758943596 shutdown_connections 2024-05-12T22:15:11.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.165+0000 7f87ef9db740 1 --2- 172.21.15.28:0/3758943596 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e202e48dc0 0x55e202e4b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:11.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.165+0000 7f87ef9db740 1 --2- 172.21.15.28:0/3758943596 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e202e3e6f0 0x55e202e48880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:11.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.165+0000 7f87ef9db740 1 --2- 172.21.15.28:0/3758943596 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e202e3ddd0 0x55e202e3e1b0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:11.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.165+0000 7f87ef9db740 1 -- 172.21.15.28:0/3758943596 >> 172.21.15.28:0/3758943596 conn(0x55e202e3a0f0 msgr2=0x55e202e3a4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:15:11.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.165+0000 7f87e5e00640 1 --2- 172.21.15.28:0/4255956323 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f87cc0a2d80 0x7f87cc0a5240 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x55e202e3f390 tx=0x7f87d8004a40 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:15:11.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.165+0000 7f87ef9db740 1 -- 172.21.15.28:0/3758943596 shutdown_connections 2024-05-12T22:15:11.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.165+0000 7f87ef9db740 1 -- 172.21.15.28:0/3758943596 wait complete. 2024-05-12T22:15:11.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.165+0000 7f87ef9db740 1 -- start start 2024-05-12T22:15:11.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.166+0000 7f87ef9db740 1 Processor -- start 2024-05-12T22:15:11.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.166+0000 7f87ef9db740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e202e3ddd0 0x55e202e48730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:11.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.166+0000 7f87ef9db740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e202e3e6f0 0x55e202e45a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:11.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.166+0000 7f87ef9db740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e202e48dc0 0x55e202e45f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:11.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.166+0000 7f87ef9db740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e202d5d9f0 con 0x55e202e3e6f0 2024-05-12T22:15:11.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.166+0000 7f87ef9db740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x55e202d9eb50 con 0x55e202e48dc0 2024-05-12T22:15:11.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.166+0000 7f87ef9db740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55e202e4bbb0 con 0x55e202e3ddd0 2024-05-12T22:15:11.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.166+0000 7f87ef9db740 10 client.0 _ll_register_callbacks cb 0x55e202ab5be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:15:11.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.166+0000 7f87e5400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e202e3e6f0 0x55e202e45a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:11.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.166+0000 7f87e6800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e202e48dc0 0x55e202e45f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:11.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.167+0000 7f87e5400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e202e3e6f0 0x55e202e45a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34256/0 (socket says 172.21.15.28:34256) 2024-05-12T22:15:11.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.167+0000 7f87e5400640 1 -- 172.21.15.28:0/3654417695 learned_addr learned my addr 172.21.15.28:0/3654417695 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:15:11.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.167+0000 7f87e6800640 1 --2- 172.21.15.28:0/3654417695 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e202e3ddd0 0x55e202e48730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:11.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.167+0000 7f87e5400640 1 -- 172.21.15.28:0/3654417695 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e202e3ddd0 msgr2=0x55e202e48730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:15:11.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.167+0000 7f87e5400640 1 --2- 172.21.15.28:0/3654417695 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e202e3ddd0 0x55e202e48730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:11.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.167+0000 7f87e5400640 1 -- 172.21.15.28:0/3654417695 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e202e48dc0 msgr2=0x55e202e45f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:15:11.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.167+0000 7f87e5400640 1 --2- 172.21.15.28:0/3654417695 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e202e48dc0 0x55e202e45f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:11.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.167+0000 7f87e5400640 1 -- 172.21.15.28:0/3654417695 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e202ab63f0 con 0x55e202e3e6f0 2024-05-12T22:15:11.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.167+0000 7f87e5400640 1 --2- 172.21.15.28:0/3654417695 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e202e3e6f0 0x55e202e45a20 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f87d4009860 tx=0x7f87d40305d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b24dd9c6e0f48c9c server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:15:11.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.167+0000 7f87c7400640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-12T22:15:11.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.168+0000 7f87c7400640 1 -- 172.21.15.28:0/3654417695 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87d4004050 con 0x55e202e3e6f0 2024-05-12T22:15:11.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.168+0000 7f87c7400640 1 -- 172.21.15.28:0/3654417695 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f87d40041f0 con 0x55e202e3e6f0 2024-05-12T22:15:11.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.168+0000 7f87ef9db740 1 -- 172.21.15.28:0/3654417695 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f87e000d7d0 con 0x55e202e3e6f0 2024-05-12T22:15:11.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.168+0000 7f87c7400640 1 -- 172.21.15.28:0/3654417695 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f87d40043f0 con 0x55e202e3e6f0 2024-05-12T22:15:11.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.168+0000 7f87ef9db740 10 client.16461 fetch_fsmap learned FSMap version 26 2024-05-12T22:15:11.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.168+0000 7f87ef9db740 1 -- 172.21.15.28:0/3654417695 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55e202a760f0 con 0x55e202e3e6f0 2024-05-12T22:15:11.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.168+0000 7f87c7400640 1 -- 172.21.15.28:0/3654417695 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f87e000d7d0 con 0x55e202e3e6f0 2024-05-12T22:15:11.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.169+0000 7f87c7400640 1 -- 172.21.15.28:0/3654417695 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f87d4004a30 con 0x55e202e3e6f0 2024-05-12T22:15:11.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.169+0000 7f87ef9db740 10 client.16461 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:15:11.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.169+0000 7f87ef9db740 10 client.16461 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:15:11.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.169+0000 7f87ef9db740 20 client.16461 populate_metadata read hostname 'smithi028' 2024-05-12T22:15:11.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.169+0000 7f87ef9db740 1 --2- 172.21.15.28:0/3654417695 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55e202e58910 0x55e202e5ad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:11.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.169+0000 7f87ef9db740 4 client.16461 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-12T22:15:11.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.169+0000 7f87ef9db740 1 -- 172.21.15.28:0/3654417695 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e202db02c0 con 0x55e202e58910 2024-05-12T22:15:11.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.169+0000 7f87e6800640 1 --2- 172.21.15.28:0/3654417695 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55e202e58910 0x55e202e5ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:11.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.170+0000 7f87e6800640 1 --2- 172.21.15.28:0/3654417695 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55e202e58910 0x55e202e5ad00 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=97a1526361cd2257 server_cookie=561cdf772ab41df9 in_seq=0 out_seq=0 2024-05-12T22:15:11.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.170+0000 7f87c7400640 10 client.16461 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:15:11.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.170+0000 7f87c7400640 1 -- 172.21.15.28:0/3654417695 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e202db02c0 con 0x55e202e58910 2024-05-12T22:15:11.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.170+0000 7f87c7400640 10 client.16461 handle_command_reply: tid=0 2024-05-12T22:15:11.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.171+0000 7f87ef9db740 1 client.16461 shutdown 2024-05-12T22:15:11.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.171+0000 7f87ef9db740 1 -- 172.21.15.28:0/3654417695 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e202e3e6f0 msgr2=0x55e202e45a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:15:11.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.171+0000 7f87ef9db740 1 --2- 172.21.15.28:0/3654417695 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e202e3e6f0 0x55e202e45a20 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f87d4009860 tx=0x7f87d40305d0 comp rx=0 tx=0).stop 2024-05-12T22:15:11.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.171+0000 7f87ef9db740 1 -- 172.21.15.28:0/3654417695 shutdown_connections 2024-05-12T22:15:11.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.171+0000 7f87ef9db740 1 --2- 172.21.15.28:0/3654417695 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55e202e58910 0x55e202e5ad00 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:11.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.171+0000 7f87ef9db740 1 --2- 172.21.15.28:0/3654417695 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e202e48dc0 0x55e202e45f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:11.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.171+0000 7f87ef9db740 1 --2- 172.21.15.28:0/3654417695 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e202e3e6f0 0x55e202e45a20 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:11.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.171+0000 7f87ef9db740 1 --2- 172.21.15.28:0/3654417695 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e202e3ddd0 0x55e202e48730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:11.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.171+0000 7f87c7400640 0 client.16461 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:15:11.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.171+0000 7f87ef9db740 1 -- 172.21.15.28:0/3654417695 >> 172.21.15.28:0/3654417695 conn(0x55e202e3a0f0 msgr2=0x55e202e3fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:15:11.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.171+0000 7f87ef9db740 1 -- 172.21.15.28:0/3654417695 shutdown_connections 2024-05-12T22:15:11.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.171+0000 7f87ef9db740 1 -- 172.21.15.28:0/3654417695 wait complete. 2024-05-12T22:15:11.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.171+0000 7f87ef9db740 20 client.16461 trim_cache size 0 max 16384 2024-05-12T22:15:11.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.191+0000 7f87ef9db740 1 Processor -- start 2024-05-12T22:15:11.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.191+0000 7f87ef9db740 1 -- start start 2024-05-12T22:15:11.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.191+0000 7f87ef9db740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e202e3ddd0 0x55e202e48260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:11.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.191+0000 7f87ef9db740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e202e3e6f0 0x55e202e605b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:11.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.191+0000 7f87ef9db740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e202e48dc0 0x55e202e60af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:11.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.191+0000 7f87ef9db740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f87e81b6250 con 0x55e202e3ddd0 2024-05-12T22:15:11.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.191+0000 7f87ef9db740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f87e81b3b20 con 0x55e202e48dc0 2024-05-12T22:15:11.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.191+0000 7f87ef9db740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f87e81b1970 con 0x55e202e3e6f0 2024-05-12T22:15:11.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.191+0000 7f87e6800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e202e3ddd0 0x55e202e48260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:11.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.191+0000 7f87e5400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e202e3e6f0 0x55e202e605b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:11.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.191+0000 7f87e6800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e202e3ddd0 0x55e202e48260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34266/0 (socket says 172.21.15.28:34266) 2024-05-12T22:15:11.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.191+0000 7f87e6800640 1 -- 172.21.15.28:0/3385440145 learned_addr learned my addr 172.21.15.28:0/3385440145 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:15:11.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.192+0000 7f87e6800640 1 --2- 172.21.15.28:0/3385440145 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e202e48dc0 0x55e202e60af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:11.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.192+0000 7f87e5400640 1 -- 172.21.15.28:0/3385440145 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e202e48dc0 msgr2=0x55e202e60af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:15:11.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.192+0000 7f87e5400640 1 --2- 172.21.15.28:0/3385440145 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e202e48dc0 0x55e202e60af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:11.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.192+0000 7f87e5400640 1 -- 172.21.15.28:0/3385440145 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e202e3ddd0 msgr2=0x55e202e48260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:15:11.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.192+0000 7f87e5400640 1 --2- 172.21.15.28:0/3385440145 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e202e3ddd0 0x55e202e48260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:11.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.192+0000 7f87e5400640 1 -- 172.21.15.28:0/3385440145 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e202e3acc0 con 0x55e202e3e6f0 2024-05-12T22:15:11.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.192+0000 7f87e5400640 1 --2- 172.21.15.28:0/3385440145 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e202e3e6f0 0x55e202e605b0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f87d4004e50 tx=0x7f87d4003820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ce15989f9f5b207c server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:15:11.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.192+0000 7f87d2600640 1 -- 172.21.15.28:0/3385440145 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87d40046c0 con 0x55e202e3e6f0 2024-05-12T22:15:11.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.193+0000 7f87d2600640 1 -- 172.21.15.28:0/3385440145 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f87d4004fc0 con 0x55e202e3e6f0 2024-05-12T22:15:11.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.193+0000 7f87d2600640 1 -- 172.21.15.28:0/3385440145 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87d4003ce0 con 0x55e202e3e6f0 2024-05-12T22:15:11.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.193+0000 7f87ef9db740 1 -- 172.21.15.28:0/3385440145 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e202e3e6f0 msgr2=0x55e202e605b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:15:11.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.193+0000 7f87ef9db740 1 --2- 172.21.15.28:0/3385440145 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e202e3e6f0 0x55e202e605b0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f87d4004e50 tx=0x7f87d4003820 comp rx=0 tx=0).stop 2024-05-12T22:15:11.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.193+0000 7f87ef9db740 1 -- 172.21.15.28:0/3385440145 shutdown_connections 2024-05-12T22:15:11.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.193+0000 7f87ef9db740 1 --2- 172.21.15.28:0/3385440145 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e202e48dc0 0x55e202e60af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:11.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.193+0000 7f87ef9db740 1 --2- 172.21.15.28:0/3385440145 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e202e3e6f0 0x55e202e605b0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:11.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.193+0000 7f87ef9db740 1 --2- 172.21.15.28:0/3385440145 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e202e3ddd0 0x55e202e48260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:11.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.193+0000 7f87ef9db740 1 -- 172.21.15.28:0/3385440145 >> 172.21.15.28:0/3385440145 conn(0x55e202e3a0f0 msgr2=0x55e202e452b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:15:11.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.193+0000 7f87ef9db740 1 -- 172.21.15.28:0/3385440145 shutdown_connections 2024-05-12T22:15:11.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.193+0000 7f87ef9db740 1 -- 172.21.15.28:0/3385440145 wait complete. 2024-05-12T22:15:11.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.194+0000 7f87ef9db740 1 -- start start 2024-05-12T22:15:11.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.194+0000 7f87ef9db740 1 Processor -- start 2024-05-12T22:15:11.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.194+0000 7f87ef9db740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e202e3ddd0 0x55e202e4c860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:11.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.195+0000 7f87ef9db740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e202e48dc0 0x55e202e4cda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:11.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.195+0000 7f87ef9db740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e202e4d2e0 0x55e202ebde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:11.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.195+0000 7f87ef9db740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f87e81b1fe0 con 0x55e202e4d2e0 2024-05-12T22:15:11.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.195+0000 7f87ef9db740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f87e81b3450 con 0x55e202e3ddd0 2024-05-12T22:15:11.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.195+0000 7f87ef9db740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f87e81b26b0 con 0x55e202e48dc0 2024-05-12T22:15:11.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.195+0000 7f87ef9db740 10 client.0 _ll_register_callbacks cb 0x7f87d40096b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:15:11.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.195+0000 7f87e5400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e202e48dc0 0x55e202e4cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:11.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.195+0000 7f87e5400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e202e48dc0 0x55e202e4cda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:36320/0 (socket says 172.21.15.28:36320) 2024-05-12T22:15:11.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.195+0000 7f87e5400640 1 -- 172.21.15.28:0/177589914 learned_addr learned my addr 172.21.15.28:0/177589914 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:15:11.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.195+0000 7f87e6800640 1 --2- 172.21.15.28:0/177589914 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e202e3ddd0 0x55e202e4c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:11.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.195+0000 7f87e6800640 1 --2- 172.21.15.28:0/177589914 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e202e4d2e0 0x55e202ebde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:11.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.195+0000 7f87e5400640 1 -- 172.21.15.28:0/177589914 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e202e3ddd0 msgr2=0x55e202e4c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:15:11.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.195+0000 7f87e5400640 1 --2- 172.21.15.28:0/177589914 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e202e3ddd0 0x55e202e4c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:11.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.195+0000 7f87e5400640 1 -- 172.21.15.28:0/177589914 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e202e4d2e0 msgr2=0x55e202ebde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:15:11.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.195+0000 7f87e5400640 1 --2- 172.21.15.28:0/177589914 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e202e4d2e0 0x55e202ebde20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:11.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.195+0000 7f87e5400640 1 -- 172.21.15.28:0/177589914 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e202ebe360 con 0x55e202e48dc0 2024-05-12T22:15:11.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.195+0000 7f87e6800640 1 --2- 172.21.15.28:0/177589914 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e202e4d2e0 0x55e202ebde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:15:11.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.196+0000 7f87e5400640 1 --2- 172.21.15.28:0/177589914 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e202e48dc0 0x55e202e4cda0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f87d402a8f0 tx=0x7f87d4005a90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e3d618236a409ef4 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:15:11.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.196+0000 7f87c7400640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-12T22:15:11.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.196+0000 7f87c7400640 1 -- 172.21.15.28:0/177589914 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87d40035f0 con 0x55e202e48dc0 2024-05-12T22:15:11.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.196+0000 7f87c7400640 1 -- 172.21.15.28:0/177589914 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f87d4005380 con 0x55e202e48dc0 2024-05-12T22:15:11.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.196+0000 7f87ef9db740 1 -- 172.21.15.28:0/177589914 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f87d4004fc0 con 0x55e202e48dc0 2024-05-12T22:15:11.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.196+0000 7f87c7400640 1 -- 172.21.15.28:0/177589914 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f87d406a050 con 0x55e202e48dc0 2024-05-12T22:15:11.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.197+0000 7f87ef9db740 10 client.26554 fetch_fsmap learned FSMap version 26 2024-05-12T22:15:11.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.197+0000 7f87ef9db740 1 -- 172.21.15.28:0/177589914 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55e202ebf5d0 con 0x55e202e48dc0 2024-05-12T22:15:11.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.197+0000 7f87c7400640 1 -- 172.21.15.28:0/177589914 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f87d4004fc0 con 0x55e202e48dc0 2024-05-12T22:15:11.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.197+0000 7f87c7400640 1 -- 172.21.15.28:0/177589914 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f87d4003a60 con 0x55e202e48dc0 2024-05-12T22:15:11.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.197+0000 7f87ef9db740 10 client.26554 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:15:11.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.197+0000 7f87ef9db740 10 client.26554 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:15:11.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.197+0000 7f87ef9db740 20 client.26554 populate_metadata read hostname 'smithi028' 2024-05-12T22:15:11.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.197+0000 7f87ef9db740 1 --2- 172.21.15.28:0/177589914 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55e202ebfbb0 0x55e202ec1fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:11.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.197+0000 7f87ef9db740 4 client.26554 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-12T22:15:11.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.197+0000 7f87ef9db740 1 -- 172.21.15.28:0/177589914 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55e202daa610 con 0x55e202ebfbb0 2024-05-12T22:15:11.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.197+0000 7f87e6800640 1 --2- 172.21.15.28:0/177589914 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55e202ebfbb0 0x55e202ec1fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:11.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.198+0000 7f87e6800640 1 --2- 172.21.15.28:0/177589914 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55e202ebfbb0 0x55e202ec1fd0 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6e86da6d74c2e5c5 server_cookie=758ac925ff0e57cc in_seq=0 out_seq=0 2024-05-12T22:15:11.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.198+0000 7f87c7400640 10 client.26554 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:15:11.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.198+0000 7f87c7400640 1 -- 172.21.15.28:0/177589914 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55e202daa610 con 0x55e202ebfbb0 2024-05-12T22:15:11.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.198+0000 7f87c7400640 10 client.26554 handle_command_reply: tid=0 2024-05-12T22:15:11.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.198+0000 7f87ef9db740 1 client.26554 shutdown 2024-05-12T22:15:11.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.198+0000 7f87ef9db740 1 -- 172.21.15.28:0/177589914 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e202e48dc0 msgr2=0x55e202e4cda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:15:11.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.198+0000 7f87ef9db740 1 --2- 172.21.15.28:0/177589914 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e202e48dc0 0x55e202e4cda0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f87d402a8f0 tx=0x7f87d4005a90 comp rx=0 tx=0).stop 2024-05-12T22:15:11.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.198+0000 7f87ef9db740 1 -- 172.21.15.28:0/177589914 shutdown_connections 2024-05-12T22:15:11.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.198+0000 7f87ef9db740 1 --2- 172.21.15.28:0/177589914 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55e202ebfbb0 0x55e202ec1fd0 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:11.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.198+0000 7f87ef9db740 1 --2- 172.21.15.28:0/177589914 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e202e4d2e0 0x55e202ebde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:11.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.198+0000 7f87ef9db740 1 --2- 172.21.15.28:0/177589914 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e202e48dc0 0x55e202e4cda0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:11.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.198+0000 7f87c7400640 0 client.26554 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:15:11.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.198+0000 7f87ef9db740 1 --2- 172.21.15.28:0/177589914 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e202e3ddd0 0x55e202e4c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:11.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.198+0000 7f87ef9db740 1 -- 172.21.15.28:0/177589914 >> 172.21.15.28:0/177589914 conn(0x55e202e3a0f0 msgr2=0x55e202e39330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:15:11.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.199+0000 7f87ef9db740 1 -- 172.21.15.28:0/177589914 shutdown_connections 2024-05-12T22:15:11.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.199+0000 7f87ef9db740 1 -- 172.21.15.28:0/177589914 wait complete. 2024-05-12T22:15:11.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.199+0000 7f87ef9db740 20 client.26554 trim_cache size 0 max 16384 2024-05-12T22:15:11.209 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-05-12T22:15:11.209 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-05-12T22:15:11.210 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-05-12T22:15:11.210 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-05-12T22:15:11.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.199+0000 7f87e7200640 1 -- 172.21.15.28:0/4255956323 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f87cc0a2d80 msgr2=0x7f87cc0a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:15:11.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.199+0000 7f87e7200640 1 --2- 172.21.15.28:0/4255956323 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f87cc0a2d80 0x7f87cc0a5240 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x55e202e3f390 tx=0x7f87d8004a40 comp rx=0 tx=0).stop 2024-05-12T22:15:11.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.199+0000 7f87e7200640 1 -- 172.21.15.28:0/4255956323 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f87e8073bd0 msgr2=0x7f87e819e640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:15:11.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.199+0000 7f87e7200640 1 --2- 172.21.15.28:0/4255956323 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f87e8073bd0 0x7f87e819e640 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f87d800e9f0 tx=0x7f87d800edf0 comp rx=0 tx=0).stop 2024-05-12T22:15:11.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.199+0000 7f87e7200640 1 -- 172.21.15.28:0/4255956323 shutdown_connections 2024-05-12T22:15:11.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.199+0000 7f87e7200640 1 --2- 172.21.15.28:0/4255956323 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f87cc0a2d80 0x7f87cc0a5240 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:11.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.199+0000 7f87e7200640 1 --2- 172.21.15.28:0/4255956323 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f87e81126e0 0x7f87e81a2f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:11.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.199+0000 7f87e7200640 1 --2- 172.21.15.28:0/4255956323 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87e80744f0 0x7f87e819eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:11.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.199+0000 7f87e7200640 1 --2- 172.21.15.28:0/4255956323 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f87e8073bd0 0x7f87e819e640 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:11.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.199+0000 7f87e7200640 1 -- 172.21.15.28:0/4255956323 >> 172.21.15.28:0/4255956323 conn(0x7f87e80fd200 msgr2=0x7f87e80fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:15:11.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.200+0000 7f87e7200640 1 -- 172.21.15.28:0/4255956323 shutdown_connections 2024-05-12T22:15:11.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.200+0000 7f87e7200640 1 -- 172.21.15.28:0/4255956323 wait complete. 2024-05-12T22:15:11.217 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-12T22:15:11.217 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-12T22:15:11.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.420+0000 7fb770a00640 1 -- 172.21.15.28:0/3549667016 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb76c073c00 msgr2=0x7fb76c073fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:15:11.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.420+0000 7fb770a00640 1 --2- 172.21.15.28:0/3549667016 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb76c073c00 0x7fb76c073fe0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fb75c009900 tx=0x7fb75c02a480 comp rx=0 tx=0).stop 2024-05-12T22:15:11.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.420+0000 7fb770a00640 1 -- 172.21.15.28:0/3549667016 shutdown_connections 2024-05-12T22:15:11.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.420+0000 7fb770a00640 1 --2- 172.21.15.28:0/3549667016 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb76c10deb0 0x7fb76c1102c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:11.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.420+0000 7fb770a00640 1 --2- 172.21.15.28:0/3549667016 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb76c074520 0x7fb76c10d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:11.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.420+0000 7fb770a00640 1 --2- 172.21.15.28:0/3549667016 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb76c073c00 0x7fb76c073fe0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:11.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.420+0000 7fb770a00640 1 -- 172.21.15.28:0/3549667016 >> 172.21.15.28:0/3549667016 conn(0x7fb76c0fd160 msgr2=0x7fb76c0ff580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:15:11.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.420+0000 7fb770a00640 1 -- 172.21.15.28:0/3549667016 shutdown_connections 2024-05-12T22:15:11.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.420+0000 7fb770a00640 1 -- 172.21.15.28:0/3549667016 wait complete. 2024-05-12T22:15:11.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.421+0000 7fb770a00640 1 Processor -- start 2024-05-12T22:15:11.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.421+0000 7fb770a00640 1 -- start start 2024-05-12T22:15:11.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.421+0000 7fb770a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb76c073c00 0x7fb76c19a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:11.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.422+0000 7fb770a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb76c074520 0x7fb76c19a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:11.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.422+0000 7fb770a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb76c10deb0 0x7fb76c19ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:11.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.422+0000 7fb770a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb76c113c60 con 0x7fb76c073c00 2024-05-12T22:15:11.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.422+0000 7fb770a00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fb76c113ae0 con 0x7fb76c10deb0 2024-05-12T22:15:11.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.422+0000 7fb770a00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb76c113de0 con 0x7fb76c074520 2024-05-12T22:15:11.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.422+0000 7fb76a800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb76c073c00 0x7fb76c19a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:11.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.422+0000 7fb76a800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb76c073c00 0x7fb76c19a160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34292/0 (socket says 172.21.15.28:34292) 2024-05-12T22:15:11.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.422+0000 7fb76a800640 1 -- 172.21.15.28:0/1116145837 learned_addr learned my addr 172.21.15.28:0/1116145837 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:15:11.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.422+0000 7fb769e00640 1 --2- 172.21.15.28:0/1116145837 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb76c074520 0x7fb76c19a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:11.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.422+0000 7fb76b200640 1 --2- 172.21.15.28:0/1116145837 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb76c10deb0 0x7fb76c19ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:11.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.423+0000 7fb76a800640 1 -- 172.21.15.28:0/1116145837 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb76c074520 msgr2=0x7fb76c19a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:15:11.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.423+0000 7fb76a800640 1 --2- 172.21.15.28:0/1116145837 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb76c074520 0x7fb76c19a6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:11.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.423+0000 7fb76a800640 1 -- 172.21.15.28:0/1116145837 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb76c10deb0 msgr2=0x7fb76c19ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:15:11.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.423+0000 7fb76a800640 1 --2- 172.21.15.28:0/1116145837 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb76c10deb0 0x7fb76c19ea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:11.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.423+0000 7fb76a800640 1 -- 172.21.15.28:0/1116145837 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb76c19f150 con 0x7fb76c073c00 2024-05-12T22:15:11.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.423+0000 7fb769e00640 1 --2- 172.21.15.28:0/1116145837 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb76c074520 0x7fb76c19a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:15:11.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.423+0000 7fb76b200640 1 --2- 172.21.15.28:0/1116145837 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb76c10deb0 0x7fb76c19ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:15:11.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.423+0000 7fb76a800640 1 --2- 172.21.15.28:0/1116145837 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb76c073c00 0x7fb76c19a160 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fb75c004770 tx=0x7fb75c003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:15:11.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.423+0000 7fb763400640 1 -- 172.21.15.28:0/1116145837 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb75c03f070 con 0x7fb76c073c00 2024-05-12T22:15:11.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.423+0000 7fb770a00640 1 -- 172.21.15.28:0/1116145837 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb76c19f3e0 con 0x7fb76c073c00 2024-05-12T22:15:11.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.423+0000 7fb763400640 1 -- 172.21.15.28:0/1116145837 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb75c031ca0 con 0x7fb76c073c00 2024-05-12T22:15:11.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.423+0000 7fb763400640 1 -- 172.21.15.28:0/1116145837 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb75c035030 con 0x7fb76c073c00 2024-05-12T22:15:11.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.423+0000 7fb770a00640 1 -- 172.21.15.28:0/1116145837 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb76c1a6cd0 con 0x7fb76c073c00 2024-05-12T22:15:11.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.425+0000 7fb770a00640 1 -- 172.21.15.28:0/1116145837 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb76c0689c0 con 0x7fb76c073c00 2024-05-12T22:15:11.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.427+0000 7fb763400640 1 -- 172.21.15.28:0/1116145837 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7fb75c042070 con 0x7fb76c073c00 2024-05-12T22:15:11.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.428+0000 7fb763400640 1 --2- 172.21.15.28:0/1116145837 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fb7500a2d80 0x7fb7500a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:11.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.428+0000 7fb763400640 1 -- 172.21.15.28:0/1116145837 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7fb75c0efa50 con 0x7fb76c073c00 2024-05-12T22:15:11.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.428+0000 7fb769e00640 1 --2- 172.21.15.28:0/1116145837 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fb7500a2d80 0x7fb7500a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:11.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.429+0000 7fb769e00640 1 --2- 172.21.15.28:0/1116145837 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fb7500a2d80 0x7fb7500a5240 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fb7580095f0 tx=0x7fb758005f90 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:15:11.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.433+0000 7fb763400640 1 -- 172.21.15.28:0/1116145837 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fb75c0bbff0 con 0x7fb76c073c00 2024-05-12T22:15:11.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.677+0000 7fb770a00640 1 -- 172.21.15.28:0/1116145837 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb76c19b4b0 con 0x7fb76c073c00 2024-05-12T22:15:11.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.678+0000 7fb763400640 1 -- 172.21.15.28:0/1116145837 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11864 (secure 0 0 0) 0x7fb75c0c02f0 con 0x7fb76c073c00 2024-05-12T22:15:11.683 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-12T22:15:11.683 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1844141745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1844141745},{"type":"v1","addr":"172.21.15.186:6833","nonce":1844141745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6833/2960763758","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":2960763758},{"type":"v1","addr":"172.21.15.86:6833","nonce":2960763758}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24470,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/4131134219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":4131134219},{"type":"v1","addr":"172.21.15.28:6835","nonce":4131134219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24481,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/1727869064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1727869064},{"type":"v1","addr":"172.21.15.186:6835","nonce":1727869064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24482,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2938260704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2938260704},{"type":"v1","addr":"172.21.15.28:6837","nonce":2938260704}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24494,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3473700910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3473700910},{"type":"v1","addr":"172.21.15.86:6835","nonce":3473700910}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-12T22:06:33.426986+0000","modified":"2024-05-12T22:13:22.499381+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24581,"mds_1":24550,"mds_2":24532},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24499":{"gid":24499,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6837/3600809235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":3600809235},{"type":"v1","addr":"172.21.15.186:6837","nonce":3600809235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"i","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6839/2399610807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":2399610807},{"type":"v1","addr":"172.21.15.186:6839","nonce":2399610807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24530":{"gid":24530,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.86:6837/3782448884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":3782448884},{"type":"v1","addr":"172.21.15.86:6837","nonce":3782448884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24532":{"gid":24532,"name":"g","rank":2,"incarnation":21,"state":"up:active","state_seq":6,"addr":"172.21.15.28:6839/123892158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":123892158},{"type":"v1","addr":"172.21.15.28:6839","nonce":123892158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24550":{"gid":24550,"name":"a","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.28:6841/761183870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":761183870},{"type":"v1","addr":"172.21.15.28:6841","nonce":761183870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24581":{"gid":24581,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6839/1251999130","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6838","nonce":1251999130},{"type":"v1","addr":"172.21.15.86:6839","nonce":1251999130}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24581,"qdb_cluster":[24550,24581,24532]},"id":1}]} 2024-05-12T22:15:11.684 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 26 2024-05-12T22:15:11.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.681+0000 7fb770a00640 1 -- 172.21.15.28:0/1116145837 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fb7500a2d80 msgr2=0x7fb7500a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:15:11.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.681+0000 7fb770a00640 1 --2- 172.21.15.28:0/1116145837 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fb7500a2d80 0x7fb7500a5240 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fb7580095f0 tx=0x7fb758005f90 comp rx=0 tx=0).stop 2024-05-12T22:15:11.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.681+0000 7fb770a00640 1 -- 172.21.15.28:0/1116145837 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb76c073c00 msgr2=0x7fb76c19a160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:15:11.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.681+0000 7fb770a00640 1 --2- 172.21.15.28:0/1116145837 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb76c073c00 0x7fb76c19a160 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fb75c004770 tx=0x7fb75c003c10 comp rx=0 tx=0).stop 2024-05-12T22:15:11.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.682+0000 7fb770a00640 1 -- 172.21.15.28:0/1116145837 shutdown_connections 2024-05-12T22:15:11.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.682+0000 7fb770a00640 1 --2- 172.21.15.28:0/1116145837 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fb7500a2d80 0x7fb7500a5240 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:11.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.682+0000 7fb770a00640 1 --2- 172.21.15.28:0/1116145837 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb76c10deb0 0x7fb76c19ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:11.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.682+0000 7fb770a00640 1 --2- 172.21.15.28:0/1116145837 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb76c074520 0x7fb76c19a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:11.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.682+0000 7fb770a00640 1 --2- 172.21.15.28:0/1116145837 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb76c073c00 0x7fb76c19a160 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:11.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.682+0000 7fb770a00640 1 -- 172.21.15.28:0/1116145837 >> 172.21.15.28:0/1116145837 conn(0x7fb76c0fd160 msgr2=0x7fb76c0fe8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:15:11.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.682+0000 7fb770a00640 1 -- 172.21.15.28:0/1116145837 shutdown_connections 2024-05-12T22:15:11.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.682+0000 7fb770a00640 1 -- 172.21.15.28:0/1116145837 wait complete. 2024-05-12T22:15:11.703 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-05-12T22:15:11.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.922+0000 7f6259800640 1 -- 172.21.15.28:0/270511175 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6254101800 msgr2=0x7f6254101c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:15:11.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.922+0000 7f6259800640 1 --2- 172.21.15.28:0/270511175 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6254101800 0x7f6254101c80 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f623c009900 tx=0x7f623c02a3f0 comp rx=0 tx=0).stop 2024-05-12T22:15:11.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.922+0000 7f6259800640 1 -- 172.21.15.28:0/270511175 shutdown_connections 2024-05-12T22:15:11.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.922+0000 7f6259800640 1 --2- 172.21.15.28:0/270511175 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f62541021c0 0x7f625410e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:11.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.922+0000 7f6259800640 1 --2- 172.21.15.28:0/270511175 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6254101800 0x7f6254101c80 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:11.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.922+0000 7f6259800640 1 --2- 172.21.15.28:0/270511175 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6254107810 0x7f6254107bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:11.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.922+0000 7f6259800640 1 -- 172.21.15.28:0/270511175 >> 172.21.15.28:0/270511175 conn(0x7f6254078c80 msgr2=0x7f6254079090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:15:11.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.922+0000 7f6259800640 1 -- 172.21.15.28:0/270511175 shutdown_connections 2024-05-12T22:15:11.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.922+0000 7f6259800640 1 -- 172.21.15.28:0/270511175 wait complete. 2024-05-12T22:15:11.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.923+0000 7f6259800640 1 Processor -- start 2024-05-12T22:15:11.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.923+0000 7f6259800640 1 -- start start 2024-05-12T22:15:11.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.924+0000 7f6259800640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6254101800 0x7f625419a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:11.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.924+0000 7f6259800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62541021c0 0x7f625419a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:11.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.924+0000 7f6259800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6254107810 0x7f625419ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:11.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.924+0000 7f6259800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6254111c30 con 0x7f62541021c0 2024-05-12T22:15:11.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.924+0000 7f6259800640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f6254111ab0 con 0x7f6254107810 2024-05-12T22:15:11.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.924+0000 7f6259800640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f6254111db0 con 0x7f6254101800 2024-05-12T22:15:11.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.924+0000 7f624be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62541021c0 0x7f625419a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:11.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.924+0000 7f6252a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6254101800 0x7f625419a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:11.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.924+0000 7f6252a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6254101800 0x7f625419a2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:36366/0 (socket says 172.21.15.28:36366) 2024-05-12T22:15:11.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.924+0000 7f6252a00640 1 -- 172.21.15.28:0/4267433983 learned_addr learned my addr 172.21.15.28:0/4267433983 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:15:11.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.924+0000 7f6253400640 1 --2- 172.21.15.28:0/4267433983 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6254107810 0x7f625419ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:11.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.925+0000 7f6252a00640 1 -- 172.21.15.28:0/4267433983 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6254107810 msgr2=0x7f625419ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:15:11.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.925+0000 7f6252a00640 1 --2- 172.21.15.28:0/4267433983 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6254107810 0x7f625419ebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:11.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.925+0000 7f6252a00640 1 -- 172.21.15.28:0/4267433983 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62541021c0 msgr2=0x7f625419a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:15:11.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.925+0000 7f6252a00640 1 --2- 172.21.15.28:0/4267433983 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62541021c0 0x7f625419a830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:11.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.925+0000 7f6252a00640 1 -- 172.21.15.28:0/4267433983 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f625419f2e0 con 0x7f6254101800 2024-05-12T22:15:11.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.925+0000 7f6252a00640 1 --2- 172.21.15.28:0/4267433983 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6254101800 0x7f625419a2f0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f624400ea60 tx=0x7f624400ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:15:11.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.925+0000 7f624b400640 1 -- 172.21.15.28:0/4267433983 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f624400cde0 con 0x7f6254101800 2024-05-12T22:15:11.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.925+0000 7f624b400640 1 -- 172.21.15.28:0/4267433983 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f62440035f0 con 0x7f6254101800 2024-05-12T22:15:11.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.925+0000 7f624b400640 1 -- 172.21.15.28:0/4267433983 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6244010670 con 0x7f6254101800 2024-05-12T22:15:11.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.925+0000 7f6259800640 1 -- 172.21.15.28:0/4267433983 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f625419f5d0 con 0x7f6254101800 2024-05-12T22:15:11.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.926+0000 7f6259800640 1 -- 172.21.15.28:0/4267433983 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62541a6e60 con 0x7f6254101800 2024-05-12T22:15:11.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.929+0000 7f624b400640 1 -- 172.21.15.28:0/4267433983 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7f62440037a0 con 0x7f6254101800 2024-05-12T22:15:11.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.929+0000 7f625c5b0740 1 Processor -- start 2024-05-12T22:15:11.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.929+0000 7f625c5b0740 1 -- start start 2024-05-12T22:15:11.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.929+0000 7f625c5b0740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560d08802980 0x560d08802d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:11.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.929+0000 7f625c5b0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d088032a0 0x560d0880d430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:11.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.929+0000 7f625c5b0740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560d0880d970 0x560d0880fd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:11.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.929+0000 7f625c5b0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x560d08767a30 con 0x560d088032a0 2024-05-12T22:15:11.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.929+0000 7f625c5b0740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x560d0870f700 con 0x560d0880d970 2024-05-12T22:15:11.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.929+0000 7f625c5b0740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x560d087639f0 con 0x560d08802980 2024-05-12T22:15:11.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.930+0000 7f6253400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560d08802980 0x560d08802d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:11.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.930+0000 7f6252a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d088032a0 0x560d0880d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:11.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.930+0000 7f6253400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560d08802980 0x560d08802d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:36378/0 (socket says 172.21.15.28:36378) 2024-05-12T22:15:11.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.930+0000 7f624be00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560d0880d970 0x560d0880fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:11.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.930+0000 7f6253400640 1 -- 172.21.15.28:0/3646393425 learned_addr learned my addr 172.21.15.28:0/3646393425 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:15:11.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.930+0000 7f6253400640 1 -- 172.21.15.28:0/3646393425 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560d0880d970 msgr2=0x560d0880fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:15:11.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.930+0000 7f6253400640 1 --2- 172.21.15.28:0/3646393425 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560d0880d970 0x560d0880fd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:11.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.930+0000 7f6253400640 1 -- 172.21.15.28:0/3646393425 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d088032a0 msgr2=0x560d0880d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:15:11.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.930+0000 7f6253400640 1 --2- 172.21.15.28:0/3646393425 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d088032a0 0x560d0880d430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:11.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.930+0000 7f6253400640 1 -- 172.21.15.28:0/3646393425 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560d0847b3f0 con 0x560d08802980 2024-05-12T22:15:11.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.930+0000 7f624b400640 1 --2- 172.21.15.28:0/4267433983 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f62380a2d00 0x7f62380a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:11.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.931+0000 7f6253400640 1 --2- 172.21.15.28:0/3646393425 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560d08802980 0x560d08802d60 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f624c009fd0 tx=0x7f624c0118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=212e01ac2228250d server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:15:11.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.931+0000 7f624b400640 1 -- 172.21.15.28:0/4267433983 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f62440cf6f0 con 0x7f6254101800 2024-05-12T22:15:11.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.931+0000 7f6252a00640 1 --2- 172.21.15.28:0/4267433983 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f62380a2d00 0x7f62380a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:11.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.931+0000 7f6243e00640 1 -- 172.21.15.28:0/3646393425 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f624c004d20 con 0x560d08802980 2024-05-12T22:15:11.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.931+0000 7f6243e00640 1 -- 172.21.15.28:0/3646393425 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f624c004050 con 0x560d08802980 2024-05-12T22:15:11.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.931+0000 7f6243e00640 1 -- 172.21.15.28:0/3646393425 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f624c005220 con 0x560d08802980 2024-05-12T22:15:11.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.932+0000 7f625c5b0740 1 -- 172.21.15.28:0/3646393425 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560d08802980 msgr2=0x560d08802d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:15:11.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.932+0000 7f625c5b0740 1 --2- 172.21.15.28:0/3646393425 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560d08802980 0x560d08802d60 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f624c009fd0 tx=0x7f624c0118d0 comp rx=0 tx=0).stop 2024-05-12T22:15:11.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.932+0000 7f6252a00640 1 --2- 172.21.15.28:0/4267433983 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f62380a2d00 0x7f62380a51c0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x560d08803f40 tx=0x7f62440d5530 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:15:11.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.932+0000 7f625c5b0740 1 -- 172.21.15.28:0/3646393425 shutdown_connections 2024-05-12T22:15:11.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.932+0000 7f625c5b0740 1 --2- 172.21.15.28:0/3646393425 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560d0880d970 0x560d0880fd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:11.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.932+0000 7f625c5b0740 1 --2- 172.21.15.28:0/3646393425 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d088032a0 0x560d0880d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:11.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.932+0000 7f625c5b0740 1 --2- 172.21.15.28:0/3646393425 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560d08802980 0x560d08802d60 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:11.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.932+0000 7f625c5b0740 1 -- 172.21.15.28:0/3646393425 >> 172.21.15.28:0/3646393425 conn(0x560d087feca0 msgr2=0x560d087ff080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:15:11.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.932+0000 7f625c5b0740 1 -- 172.21.15.28:0/3646393425 shutdown_connections 2024-05-12T22:15:11.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.932+0000 7f625c5b0740 1 -- 172.21.15.28:0/3646393425 wait complete. 2024-05-12T22:15:11.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.932+0000 7f625c5b0740 1 -- start start 2024-05-12T22:15:11.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.933+0000 7f625c5b0740 1 Processor -- start 2024-05-12T22:15:11.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.933+0000 7f625c5b0740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560d08802980 0x560d0880a920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:11.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.933+0000 7f625c5b0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d088032a0 0x560d0880ae60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:11.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.933+0000 7f625c5b0740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560d0880d970 0x560d0880b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:11.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.933+0000 7f625c5b0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x560d08810760 con 0x560d088032a0 2024-05-12T22:15:11.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.933+0000 7f625c5b0740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x560d08722a70 con 0x560d08802980 2024-05-12T22:15:11.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.933+0000 7f625c5b0740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x560d088108e0 con 0x560d0880d970 2024-05-12T22:15:11.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.933+0000 7f625c5b0740 10 client.0 _ll_register_callbacks cb 0x560d08705ba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:15:11.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.933+0000 7f6253400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560d08802980 0x560d0880a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:11.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.933+0000 7f624be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d088032a0 0x560d0880ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:11.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.933+0000 7f6253400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560d08802980 0x560d0880a920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:55160/0 (socket says 172.21.15.28:55160) 2024-05-12T22:15:11.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.933+0000 7f6253400640 1 -- 172.21.15.28:0/1555968810 learned_addr learned my addr 172.21.15.28:0/1555968810 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:15:11.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.933+0000 7f6253400640 1 --2- 172.21.15.28:0/1555968810 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560d0880d970 0x560d0880b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:11.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.934+0000 7f624be00640 1 -- 172.21.15.28:0/1555968810 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560d0880d970 msgr2=0x560d0880b3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:15:11.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.934+0000 7f624be00640 1 --2- 172.21.15.28:0/1555968810 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560d0880d970 0x560d0880b3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:11.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.934+0000 7f624be00640 1 -- 172.21.15.28:0/1555968810 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560d08802980 msgr2=0x560d0880a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:15:11.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.934+0000 7f624be00640 1 --2- 172.21.15.28:0/1555968810 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560d08802980 0x560d0880a920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:11.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.934+0000 7f624be00640 1 -- 172.21.15.28:0/1555968810 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560d085457d0 con 0x560d088032a0 2024-05-12T22:15:11.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.934+0000 7f624be00640 1 --2- 172.21.15.28:0/1555968810 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d088032a0 0x560d0880ae60 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f623c02a910 tx=0x7f623c0043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ce1776d3949ed6e server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:15:11.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.934+0000 7f6240c00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-12T22:15:11.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.934+0000 7f6240c00640 1 -- 172.21.15.28:0/1555968810 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f623c035040 con 0x560d088032a0 2024-05-12T22:15:11.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.935+0000 7f625c5b0740 1 -- 172.21.15.28:0/1555968810 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f624c004050 con 0x560d088032a0 2024-05-12T22:15:11.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.935+0000 7f6240c00640 1 -- 172.21.15.28:0/1555968810 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f623c005450 con 0x560d088032a0 2024-05-12T22:15:11.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.935+0000 7f6240c00640 1 -- 172.21.15.28:0/1555968810 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f623c005650 con 0x560d088032a0 2024-05-12T22:15:11.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.935+0000 7f625c5b0740 10 client.16485 fetch_fsmap learned FSMap version 26 2024-05-12T22:15:11.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.935+0000 7f625c5b0740 1 -- 172.21.15.28:0/1555968810 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x560d0881ca70 con 0x560d088032a0 2024-05-12T22:15:11.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.935+0000 7f6240c00640 1 -- 172.21.15.28:0/1555968810 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f624c004050 con 0x560d088032a0 2024-05-12T22:15:11.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.935+0000 7f6240c00640 1 -- 172.21.15.28:0/1555968810 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f623c0039d0 con 0x560d088032a0 2024-05-12T22:15:11.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.935+0000 7f625c5b0740 10 client.16485 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:15:11.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.935+0000 7f625c5b0740 10 client.16485 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.g 2024-05-12T22:15:11.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.936+0000 7f625c5b0740 20 client.16485 populate_metadata read hostname 'smithi028' 2024-05-12T22:15:11.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.936+0000 7f625c5b0740 1 --2- 172.21.15.28:0/1555968810 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x560d0881de80 0x560d08820270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:11.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.936+0000 7f625c5b0740 4 client.16485 mds_command: new command op to 24532 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-12T22:15:11.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.936+0000 7f625c5b0740 1 -- 172.21.15.28:0/1555968810 --> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560d08775160 con 0x560d0881de80 2024-05-12T22:15:11.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.936+0000 7f6253400640 1 --2- 172.21.15.28:0/1555968810 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x560d0881de80 0x560d08820270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:11.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.936+0000 7f6240c00640 10 client.16485 ms_handle_connect on v2:172.21.15.28:6838/123892158 2024-05-12T22:15:11.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.936+0000 7f6253400640 1 --2- 172.21.15.28:0/1555968810 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x560d0881de80 0x560d08820270 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=52e72ecfc3c953c3 server_cookie=2bca23ee98d21ea8 in_seq=0 out_seq=0 2024-05-12T22:15:11.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.937+0000 7f6240c00640 1 -- 172.21.15.28:0/1555968810 <== mds.2 v2:172.21.15.28:6838/123892158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560d08775160 con 0x560d0881de80 2024-05-12T22:15:11.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.937+0000 7f6240c00640 10 client.16485 handle_command_reply: tid=0 2024-05-12T22:15:11.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.937+0000 7f625c5b0740 1 client.16485 shutdown 2024-05-12T22:15:11.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.937+0000 7f625c5b0740 1 -- 172.21.15.28:0/1555968810 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d088032a0 msgr2=0x560d0880ae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:15:11.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.937+0000 7f625c5b0740 1 --2- 172.21.15.28:0/1555968810 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d088032a0 0x560d0880ae60 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f623c02a910 tx=0x7f623c0043b0 comp rx=0 tx=0).stop 2024-05-12T22:15:11.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.938+0000 7f625c5b0740 1 -- 172.21.15.28:0/1555968810 shutdown_connections 2024-05-12T22:15:11.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.938+0000 7f625c5b0740 1 --2- 172.21.15.28:0/1555968810 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560d0880d970 0x560d0880b3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:11.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.938+0000 7f625c5b0740 1 --2- 172.21.15.28:0/1555968810 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d088032a0 0x560d0880ae60 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:11.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.938+0000 7f625c5b0740 1 --2- 172.21.15.28:0/1555968810 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x560d0881de80 0x560d08820270 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:11.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.938+0000 7f625c5b0740 1 --2- 172.21.15.28:0/1555968810 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560d08802980 0x560d0880a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:11.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.938+0000 7f625c5b0740 1 -- 172.21.15.28:0/1555968810 >> 172.21.15.28:0/1555968810 conn(0x560d087feca0 msgr2=0x560d08804aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:15:11.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.938+0000 7f6240c00640 0 client.16485 ms_handle_reset on v2:172.21.15.28:6838/123892158 2024-05-12T22:15:11.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.938+0000 7f625c5b0740 1 -- 172.21.15.28:0/1555968810 shutdown_connections 2024-05-12T22:15:11.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.938+0000 7f625c5b0740 1 -- 172.21.15.28:0/1555968810 wait complete. 2024-05-12T22:15:11.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.938+0000 7f625c5b0740 20 client.16485 trim_cache size 0 max 16384 2024-05-12T22:15:11.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.953+0000 7f625c5b0740 1 Processor -- start 2024-05-12T22:15:11.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.953+0000 7f625c5b0740 1 -- start start 2024-05-12T22:15:11.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.953+0000 7f625c5b0740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560d08802980 0x560d08819620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:11.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.953+0000 7f625c5b0740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560d088032a0 0x560d088050e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:11.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.953+0000 7f625c5b0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d0880d970 0x560d087fdc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:11.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.953+0000 7f625c5b0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f62541b1fa0 con 0x560d0880d970 2024-05-12T22:15:11.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.953+0000 7f625c5b0740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f62541af870 con 0x560d08802980 2024-05-12T22:15:11.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.953+0000 7f625c5b0740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f62541ad6c0 con 0x560d088032a0 2024-05-12T22:15:11.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.954+0000 7f6253400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560d08802980 0x560d08819620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:11.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.954+0000 7f624be00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560d088032a0 0x560d088050e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:11.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.954+0000 7f624be00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560d088032a0 0x560d088050e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:36402/0 (socket says 172.21.15.28:36402) 2024-05-12T22:15:11.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.954+0000 7f624be00640 1 -- 172.21.15.28:0/1061419127 learned_addr learned my addr 172.21.15.28:0/1061419127 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:15:11.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.954+0000 7f6253400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560d08802980 0x560d08819620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:55168/0 (socket says 172.21.15.28:55168) 2024-05-12T22:15:11.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.954+0000 7f6253400640 1 --2- 172.21.15.28:0/1061419127 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d0880d970 0x560d087fdc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:11.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.954+0000 7f624be00640 1 -- 172.21.15.28:0/1061419127 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560d08802980 msgr2=0x560d08819620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:15:11.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.954+0000 7f624be00640 1 --2- 172.21.15.28:0/1061419127 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560d08802980 0x560d08819620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:11.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.954+0000 7f624be00640 1 -- 172.21.15.28:0/1061419127 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d0880d970 msgr2=0x560d087fdc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:15:11.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.954+0000 7f624be00640 1 --2- 172.21.15.28:0/1061419127 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d0880d970 0x560d087fdc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:11.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.954+0000 7f624be00640 1 -- 172.21.15.28:0/1061419127 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560d08808e70 con 0x560d088032a0 2024-05-12T22:15:11.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.954+0000 7f6253400640 1 --2- 172.21.15.28:0/1061419127 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d0880d970 0x560d087fdc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:15:11.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.954+0000 7f624be00640 1 --2- 172.21.15.28:0/1061419127 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560d088032a0 0x560d088050e0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f623c009850 tx=0x7f623c0316d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=752542f46e0a3895 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:15:11.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.955+0000 7f6243400640 1 -- 172.21.15.28:0/1061419127 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f623c035030 con 0x560d088032a0 2024-05-12T22:15:11.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.955+0000 7f6243400640 1 -- 172.21.15.28:0/1061419127 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f623c054020 con 0x560d088032a0 2024-05-12T22:15:11.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.955+0000 7f6243400640 1 -- 172.21.15.28:0/1061419127 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f623c0035f0 con 0x560d088032a0 2024-05-12T22:15:11.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.955+0000 7f625c5b0740 1 -- 172.21.15.28:0/1061419127 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560d088032a0 msgr2=0x560d088050e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:15:11.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.955+0000 7f625c5b0740 1 --2- 172.21.15.28:0/1061419127 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560d088032a0 0x560d088050e0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f623c009850 tx=0x7f623c0316d0 comp rx=0 tx=0).stop 2024-05-12T22:15:11.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.955+0000 7f625c5b0740 1 -- 172.21.15.28:0/1061419127 shutdown_connections 2024-05-12T22:15:11.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.955+0000 7f625c5b0740 1 --2- 172.21.15.28:0/1061419127 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d0880d970 0x560d087fdc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:11.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.955+0000 7f625c5b0740 1 --2- 172.21.15.28:0/1061419127 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560d088032a0 0x560d088050e0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:11.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.955+0000 7f625c5b0740 1 --2- 172.21.15.28:0/1061419127 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560d08802980 0x560d08819620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:11.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.955+0000 7f625c5b0740 1 -- 172.21.15.28:0/1061419127 >> 172.21.15.28:0/1061419127 conn(0x560d087feca0 msgr2=0x560d087fd530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:15:11.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.955+0000 7f625c5b0740 1 -- 172.21.15.28:0/1061419127 shutdown_connections 2024-05-12T22:15:11.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.955+0000 7f625c5b0740 1 -- 172.21.15.28:0/1061419127 wait complete. 2024-05-12T22:15:11.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.956+0000 7f625c5b0740 1 -- start start 2024-05-12T22:15:11.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.956+0000 7f625c5b0740 1 Processor -- start 2024-05-12T22:15:11.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.956+0000 7f625c5b0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d08802980 0x560d08815e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:11.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.956+0000 7f625c5b0740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560d088032a0 0x560d088071d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:11.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.957+0000 7f625c5b0740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560d0880d970 0x560d08807710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:11.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.957+0000 7f625c5b0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f62541add30 con 0x560d08802980 2024-05-12T22:15:11.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.957+0000 7f625c5b0740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f62541af1a0 con 0x560d088032a0 2024-05-12T22:15:11.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.957+0000 7f625c5b0740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f62541ae400 con 0x560d0880d970 2024-05-12T22:15:11.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.957+0000 7f625c5b0740 10 client.0 _ll_register_callbacks cb 0x7f623c004040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:15:11.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.957+0000 7f6253400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d08802980 0x560d08815e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:11.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.957+0000 7f6253400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d08802980 0x560d08815e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34360/0 (socket says 172.21.15.28:34360) 2024-05-12T22:15:11.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.957+0000 7f6253400640 1 -- 172.21.15.28:0/641787806 learned_addr learned my addr 172.21.15.28:0/641787806 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:15:11.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.957+0000 7f624be00640 1 --2- 172.21.15.28:0/641787806 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560d088032a0 0x560d088071d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:11.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.957+0000 7f6253400640 1 --2- 172.21.15.28:0/641787806 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560d0880d970 0x560d08807710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:11.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.957+0000 7f624be00640 1 -- 172.21.15.28:0/641787806 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560d0880d970 msgr2=0x560d08807710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:15:11.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.957+0000 7f624be00640 1 --2- 172.21.15.28:0/641787806 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560d0880d970 0x560d08807710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:11.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.957+0000 7f624be00640 1 -- 172.21.15.28:0/641787806 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d08802980 msgr2=0x560d08815e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:15:11.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.957+0000 7f624be00640 1 --2- 172.21.15.28:0/641787806 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d08802980 0x560d08815e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:11.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.957+0000 7f624be00640 1 -- 172.21.15.28:0/641787806 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560d08807dd0 con 0x560d088032a0 2024-05-12T22:15:11.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.957+0000 7f6253400640 1 --2- 172.21.15.28:0/641787806 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560d0880d970 0x560d08807710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-12T22:15:11.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.958+0000 7f624be00640 1 --2- 172.21.15.28:0/641787806 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560d088032a0 0x560d088071d0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x560d087fec70 tx=0x7f623c005d70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2f931396838a611a server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:15:11.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.958+0000 7f6240c00640 10 client.0 ms_handle_connect on v2:172.21.15.86:3300/0 2024-05-12T22:15:11.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.958+0000 7f6240c00640 1 -- 172.21.15.28:0/641787806 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f623c05d060 con 0x560d088032a0 2024-05-12T22:15:11.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.958+0000 7f625c5b0740 1 -- 172.21.15.28:0/641787806 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f623c054020 con 0x560d088032a0 2024-05-12T22:15:11.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.958+0000 7f6240c00640 1 -- 172.21.15.28:0/641787806 <== mon.2 v2:172.21.15.86:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f623c031af0 con 0x560d088032a0 2024-05-12T22:15:11.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.958+0000 7f6240c00640 1 -- 172.21.15.28:0/641787806 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f623c053070 con 0x560d088032a0 2024-05-12T22:15:11.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.958+0000 7f625c5b0740 10 client.26615 fetch_fsmap learned FSMap version 26 2024-05-12T22:15:11.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.958+0000 7f625c5b0740 1 -- 172.21.15.28:0/641787806 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x560d08884240 con 0x560d088032a0 2024-05-12T22:15:11.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.958+0000 7f6240c00640 1 -- 172.21.15.28:0/641787806 <== mon.2 v2:172.21.15.86:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f623c054020 con 0x560d088032a0 2024-05-12T22:15:11.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.958+0000 7f6240c00640 1 -- 172.21.15.28:0/641787806 <== mon.2 v2:172.21.15.86:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f623c0046c0 con 0x560d088032a0 2024-05-12T22:15:11.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.959+0000 7f625c5b0740 10 client.26615 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:15:11.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.959+0000 7f625c5b0740 10 client.26615 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.g 2024-05-12T22:15:11.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.959+0000 7f625c5b0740 20 client.26615 populate_metadata read hostname 'smithi028' 2024-05-12T22:15:11.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.959+0000 7f625c5b0740 1 --2- 172.21.15.28:0/641787806 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x560d08884890 0x560d08886cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:11.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.959+0000 7f625c5b0740 4 client.26615 mds_command: new command op to 24532 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-12T22:15:11.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.959+0000 7f625c5b0740 1 -- 172.21.15.28:0/641787806 --> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560d0876f4b0 con 0x560d08884890 2024-05-12T22:15:11.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.959+0000 7f6253400640 1 --2- 172.21.15.28:0/641787806 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x560d08884890 0x560d08886cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:11.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.959+0000 7f6240c00640 10 client.26615 ms_handle_connect on v2:172.21.15.28:6838/123892158 2024-05-12T22:15:11.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.959+0000 7f6253400640 1 --2- 172.21.15.28:0/641787806 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x560d08884890 0x560d08886cb0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d2e61992b59d0650 server_cookie=4fc8511611670acb in_seq=0 out_seq=0 2024-05-12T22:15:11.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.960+0000 7f6240c00640 1 -- 172.21.15.28:0/641787806 <== mds.2 v2:172.21.15.28:6838/123892158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560d0876f4b0 con 0x560d08884890 2024-05-12T22:15:11.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.960+0000 7f6240c00640 10 client.26615 handle_command_reply: tid=0 2024-05-12T22:15:11.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.960+0000 7f625c5b0740 1 client.26615 shutdown 2024-05-12T22:15:11.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.960+0000 7f625c5b0740 1 -- 172.21.15.28:0/641787806 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560d088032a0 msgr2=0x560d088071d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:15:11.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.960+0000 7f625c5b0740 1 --2- 172.21.15.28:0/641787806 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560d088032a0 0x560d088071d0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x560d087fec70 tx=0x7f623c005d70 comp rx=0 tx=0).stop 2024-05-12T22:15:11.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.960+0000 7f625c5b0740 1 -- 172.21.15.28:0/641787806 shutdown_connections 2024-05-12T22:15:11.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.960+0000 7f625c5b0740 1 --2- 172.21.15.28:0/641787806 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560d0880d970 0x560d08807710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:11.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.960+0000 7f625c5b0740 1 --2- 172.21.15.28:0/641787806 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x560d08884890 0x560d08886cb0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:11.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.960+0000 7f625c5b0740 1 --2- 172.21.15.28:0/641787806 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560d088032a0 0x560d088071d0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:11.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.960+0000 7f625c5b0740 1 --2- 172.21.15.28:0/641787806 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d08802980 0x560d08815e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:11.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.960+0000 7f6240c00640 0 client.26615 ms_handle_reset on v2:172.21.15.28:6838/123892158 2024-05-12T22:15:11.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.960+0000 7f625c5b0740 1 -- 172.21.15.28:0/641787806 >> 172.21.15.28:0/641787806 conn(0x560d087feca0 msgr2=0x560d087fddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:15:11.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.960+0000 7f625c5b0740 1 -- 172.21.15.28:0/641787806 shutdown_connections 2024-05-12T22:15:11.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.960+0000 7f625c5b0740 1 -- 172.21.15.28:0/641787806 wait complete. 2024-05-12T22:15:11.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.960+0000 7f625c5b0740 20 client.26615 trim_cache size 0 max 16384 2024-05-12T22:15:11.970 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-05-12T22:15:11.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.961+0000 7f6259800640 1 -- 172.21.15.28:0/4267433983 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f62380a2d00 msgr2=0x7f62380a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:15:11.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.961+0000 7f6259800640 1 --2- 172.21.15.28:0/4267433983 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f62380a2d00 0x7f62380a51c0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x560d08803f40 tx=0x7f62440d5530 comp rx=0 tx=0).stop 2024-05-12T22:15:11.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.961+0000 7f6259800640 1 -- 172.21.15.28:0/4267433983 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6254101800 msgr2=0x7f625419a2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:15:11.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.961+0000 7f6259800640 1 --2- 172.21.15.28:0/4267433983 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6254101800 0x7f625419a2f0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f624400ea60 tx=0x7f624400ef20 comp rx=0 tx=0).stop 2024-05-12T22:15:11.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.961+0000 7f6259800640 1 -- 172.21.15.28:0/4267433983 shutdown_connections 2024-05-12T22:15:11.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.961+0000 7f6259800640 1 --2- 172.21.15.28:0/4267433983 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f62380a2d00 0x7f62380a51c0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:11.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.961+0000 7f6259800640 1 --2- 172.21.15.28:0/4267433983 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6254107810 0x7f625419ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:11.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.961+0000 7f6259800640 1 --2- 172.21.15.28:0/4267433983 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62541021c0 0x7f625419a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:11.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.961+0000 7f6259800640 1 --2- 172.21.15.28:0/4267433983 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6254101800 0x7f625419a2f0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:11.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.961+0000 7f6259800640 1 -- 172.21.15.28:0/4267433983 >> 172.21.15.28:0/4267433983 conn(0x7f6254078c80 msgr2=0x7f625410cfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:15:11.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.961+0000 7f6259800640 1 -- 172.21.15.28:0/4267433983 shutdown_connections 2024-05-12T22:15:11.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:11.961+0000 7f6259800640 1 -- 172.21.15.28:0/4267433983 wait complete. 2024-05-12T22:15:11.976 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:15:11 smithi028 ceph-mon[29847]: pgmap v189: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-05-12T22:15:11.977 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:15:11 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/1116145837' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:15:11.979 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-05-12T22:15:12.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:15:11 smithi186 ceph-mon[34785]: pgmap v189: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-05-12T22:15:12.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:15:11 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/1116145837' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:15:12.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.190+0000 7f8bb3600640 1 -- 172.21.15.28:0/3325166255 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8bac075d60 msgr2=0x7f8bac076140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:15:12.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.190+0000 7f8bb3600640 1 --2- 172.21.15.28:0/3325166255 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8bac075d60 0x7f8bac076140 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f8ba0009930 tx=0x7f8ba002a420 comp rx=0 tx=0).stop 2024-05-12T22:15:12.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.190+0000 7f8bb3600640 1 -- 172.21.15.28:0/3325166255 shutdown_connections 2024-05-12T22:15:12.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.190+0000 7f8bb3600640 1 --2- 172.21.15.28:0/3325166255 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8bac1126b0 0x7f8bac114aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:12.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.190+0000 7f8bb3600640 1 --2- 172.21.15.28:0/3325166255 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bac076680 0x7f8bac112170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:12.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.190+0000 7f8bb3600640 1 --2- 172.21.15.28:0/3325166255 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8bac075d60 0x7f8bac076140 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:12.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.190+0000 7f8bb3600640 1 -- 172.21.15.28:0/3325166255 >> 172.21.15.28:0/3325166255 conn(0x7f8bac0ff4a0 msgr2=0x7f8bac1018c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:15:12.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.190+0000 7f8bb3600640 1 -- 172.21.15.28:0/3325166255 shutdown_connections 2024-05-12T22:15:12.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.190+0000 7f8bb3600640 1 -- 172.21.15.28:0/3325166255 wait complete. 2024-05-12T22:15:12.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.191+0000 7f8bb3600640 1 Processor -- start 2024-05-12T22:15:12.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.191+0000 7f8bb3600640 1 -- start start 2024-05-12T22:15:12.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.192+0000 7f8bb3600640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8bac075d60 0x7f8bac1a4590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:12.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.192+0000 7f8bb3600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bac076680 0x7f8bac1a4ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:12.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.192+0000 7f8bb3600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8bac1126b0 0x7f8bac19e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:12.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.192+0000 7f8bb3600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8bac118270 con 0x7f8bac076680 2024-05-12T22:15:12.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.192+0000 7f8bb3600640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f8bac1180f0 con 0x7f8bac1126b0 2024-05-12T22:15:12.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.192+0000 7f8bb3600640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f8bac1183f0 con 0x7f8bac075d60 2024-05-12T22:15:12.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.192+0000 7f8babe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bac076680 0x7f8bac1a4ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:12.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.192+0000 7f8babe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bac076680 0x7f8bac1a4ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34382/0 (socket says 172.21.15.28:34382) 2024-05-12T22:15:12.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.192+0000 7f8babe00640 1 -- 172.21.15.28:0/2565192959 learned_addr learned my addr 172.21.15.28:0/2565192959 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:15:12.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.192+0000 7f8bb0c00640 1 --2- 172.21.15.28:0/2565192959 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8bac075d60 0x7f8bac1a4590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:12.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.192+0000 7f8bb1600640 1 --2- 172.21.15.28:0/2565192959 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8bac1126b0 0x7f8bac19e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:12.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.192+0000 7f8babe00640 1 -- 172.21.15.28:0/2565192959 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8bac075d60 msgr2=0x7f8bac1a4590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:15:12.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.192+0000 7f8babe00640 1 --2- 172.21.15.28:0/2565192959 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8bac075d60 0x7f8bac1a4590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:12.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.192+0000 7f8babe00640 1 -- 172.21.15.28:0/2565192959 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8bac1126b0 msgr2=0x7f8bac19e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:15:12.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.192+0000 7f8babe00640 1 --2- 172.21.15.28:0/2565192959 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8bac1126b0 0x7f8bac19e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:12.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.192+0000 7f8babe00640 1 -- 172.21.15.28:0/2565192959 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bac19ef10 con 0x7f8bac076680 2024-05-12T22:15:12.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.192+0000 7f8bb1600640 1 --2- 172.21.15.28:0/2565192959 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8bac1126b0 0x7f8bac19e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:15:12.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.192+0000 7f8bb0c00640 1 --2- 172.21.15.28:0/2565192959 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8bac075d60 0x7f8bac1a4590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:15:12.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.193+0000 7f8babe00640 1 --2- 172.21.15.28:0/2565192959 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bac076680 0x7f8bac1a4ad0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f8b9800e9f0 tx=0x7f8b9800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:15:12.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.193+0000 7f8ba9600640 1 -- 172.21.15.28:0/2565192959 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b9800cd80 con 0x7f8bac076680 2024-05-12T22:15:12.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.193+0000 7f8bb3600640 1 -- 172.21.15.28:0/2565192959 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8bac19f200 con 0x7f8bac076680 2024-05-12T22:15:12.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.193+0000 7f8ba9600640 1 -- 172.21.15.28:0/2565192959 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8b98010910 con 0x7f8bac076680 2024-05-12T22:15:12.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.193+0000 7f8bb3600640 1 -- 172.21.15.28:0/2565192959 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8bac1ab510 con 0x7f8bac076680 2024-05-12T22:15:12.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.193+0000 7f8ba9600640 1 -- 172.21.15.28:0/2565192959 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b98018910 con 0x7f8bac076680 2024-05-12T22:15:12.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.197+0000 7f8ba9600640 1 -- 172.21.15.28:0/2565192959 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7f8b98010ac0 con 0x7f8bac076680 2024-05-12T22:15:12.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.197+0000 7f8bb657d740 1 Processor -- start 2024-05-12T22:15:12.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.197+0000 7f8bb657d740 1 -- start start 2024-05-12T22:15:12.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.197+0000 7f8bb657d740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f7d5dbf980 0x55f7d5dbfd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:12.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.197+0000 7f8bb657d740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f7d5dc02a0 0x55f7d5dca430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:12.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.197+0000 7f8bb657d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7d5dca970 0x55f7d5dccd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:12.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.197+0000 7f8bb657d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f7d5d24a30 con 0x55f7d5dca970 2024-05-12T22:15:12.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.197+0000 7f8bb657d740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x55f7d5ccc700 con 0x55f7d5dc02a0 2024-05-12T22:15:12.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.197+0000 7f8bb657d740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55f7d5d209f0 con 0x55f7d5dbf980 2024-05-12T22:15:12.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.198+0000 7f8babe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7d5dca970 0x55f7d5dccd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:12.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.198+0000 7f8bb0c00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f7d5dc02a0 0x55f7d5dca430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:12.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.198+0000 7f8bb1600640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f7d5dbf980 0x55f7d5dbfd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:12.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.198+0000 7f8babe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7d5dca970 0x55f7d5dccd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34398/0 (socket says 172.21.15.28:34398) 2024-05-12T22:15:12.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.198+0000 7f8babe00640 1 -- 172.21.15.28:0/3855581463 learned_addr learned my addr 172.21.15.28:0/3855581463 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:15:12.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.198+0000 7f8bb0c00640 1 -- 172.21.15.28:0/3855581463 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f7d5dbf980 msgr2=0x55f7d5dbfd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:15:12.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.198+0000 7f8bb0c00640 1 --2- 172.21.15.28:0/3855581463 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f7d5dbf980 0x55f7d5dbfd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:12.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.198+0000 7f8bb0c00640 1 -- 172.21.15.28:0/3855581463 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7d5dca970 msgr2=0x55f7d5dccd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:15:12.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.198+0000 7f8bb0c00640 1 --2- 172.21.15.28:0/3855581463 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7d5dca970 0x55f7d5dccd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:12.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.198+0000 7f8bb0c00640 1 -- 172.21.15.28:0/3855581463 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f7d5a383f0 con 0x55f7d5dc02a0 2024-05-12T22:15:12.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.199+0000 7f8bb0c00640 1 --2- 172.21.15.28:0/3855581463 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f7d5dc02a0 0x55f7d5dca430 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f8ba002a940 tx=0x7f8ba0000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b64238e39243a467 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:15:12.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.199+0000 7f8ba9600640 1 --2- 172.21.15.28:0/2565192959 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f8b940a2d00 0x7f8b940a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:12.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.199+0000 7f8ba9600640 1 -- 172.21.15.28:0/2565192959 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f8b980cf8b0 con 0x7f8bac076680 2024-05-12T22:15:12.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.199+0000 7f8b9dc00640 1 -- 172.21.15.28:0/3855581463 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ba0035040 con 0x55f7d5dc02a0 2024-05-12T22:15:12.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.199+0000 7f8bb1600640 1 --2- 172.21.15.28:0/2565192959 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f8b940a2d00 0x7f8b940a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:12.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.199+0000 7f8b9dc00640 1 -- 172.21.15.28:0/3855581463 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8ba0004760 con 0x55f7d5dc02a0 2024-05-12T22:15:12.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.199+0000 7f8b9dc00640 1 -- 172.21.15.28:0/3855581463 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ba00038e0 con 0x55f7d5dc02a0 2024-05-12T22:15:12.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.200+0000 7f8bb657d740 1 -- 172.21.15.28:0/3855581463 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f7d5dc02a0 msgr2=0x55f7d5dca430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:15:12.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.200+0000 7f8bb1600640 1 --2- 172.21.15.28:0/2565192959 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f8b940a2d00 0x7f8b940a51c0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f8bac06ac20 tx=0x7f8ba4009500 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:15:12.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.200+0000 7f8bb657d740 1 --2- 172.21.15.28:0/3855581463 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f7d5dc02a0 0x55f7d5dca430 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f8ba002a940 tx=0x7f8ba0000bc0 comp rx=0 tx=0).stop 2024-05-12T22:15:12.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.200+0000 7f8bb657d740 1 -- 172.21.15.28:0/3855581463 shutdown_connections 2024-05-12T22:15:12.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.200+0000 7f8bb657d740 1 --2- 172.21.15.28:0/3855581463 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7d5dca970 0x55f7d5dccd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:12.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.200+0000 7f8bb657d740 1 --2- 172.21.15.28:0/3855581463 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f7d5dc02a0 0x55f7d5dca430 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:12.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.200+0000 7f8bb657d740 1 --2- 172.21.15.28:0/3855581463 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f7d5dbf980 0x55f7d5dbfd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:12.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.200+0000 7f8bb657d740 1 -- 172.21.15.28:0/3855581463 >> 172.21.15.28:0/3855581463 conn(0x55f7d5dbbca0 msgr2=0x55f7d5dbc080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:15:12.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.200+0000 7f8bb657d740 1 -- 172.21.15.28:0/3855581463 shutdown_connections 2024-05-12T22:15:12.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.200+0000 7f8bb657d740 1 -- 172.21.15.28:0/3855581463 wait complete. 2024-05-12T22:15:12.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.200+0000 7f8bb657d740 1 -- start start 2024-05-12T22:15:12.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.201+0000 7f8bb657d740 1 Processor -- start 2024-05-12T22:15:12.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.201+0000 7f8bb657d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7d5dbf980 0x55f7d5dc7900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:12.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.201+0000 7f8bb657d740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f7d5dc02a0 0x55f7d5dc7e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:12.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.201+0000 7f8bb657d740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f7d5dca970 0x55f7d5dc8380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:12.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.201+0000 7f8bb657d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f7d5dcd700 con 0x55f7d5dbf980 2024-05-12T22:15:12.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.201+0000 7f8bb657d740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x55f7d5cdfa70 con 0x55f7d5dc02a0 2024-05-12T22:15:12.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.201+0000 7f8bb657d740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55f7d5dcd880 con 0x55f7d5dca970 2024-05-12T22:15:12.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.201+0000 7f8bb657d740 10 client.0 _ll_register_callbacks cb 0x55f7d5cc2ba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:15:12.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.202+0000 7f8babe00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f7d5dc02a0 0x55f7d5dc7e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:12.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.202+0000 7f8bb0c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7d5dbf980 0x55f7d5dc7900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:12.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.202+0000 7f8babe00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f7d5dc02a0 0x55f7d5dc7e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:55208/0 (socket says 172.21.15.28:55208) 2024-05-12T22:15:12.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.202+0000 7f8bb1600640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f7d5dca970 0x55f7d5dc8380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:12.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.202+0000 7f8babe00640 1 -- 172.21.15.28:0/189197471 learned_addr learned my addr 172.21.15.28:0/189197471 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:15:12.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.202+0000 7f8babe00640 1 -- 172.21.15.28:0/189197471 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f7d5dca970 msgr2=0x55f7d5dc8380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:15:12.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.202+0000 7f8babe00640 1 --2- 172.21.15.28:0/189197471 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f7d5dca970 0x55f7d5dc8380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:12.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.202+0000 7f8babe00640 1 -- 172.21.15.28:0/189197471 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7d5dbf980 msgr2=0x55f7d5dc7900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:15:12.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.202+0000 7f8babe00640 1 --2- 172.21.15.28:0/189197471 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7d5dbf980 0x55f7d5dc7900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:12.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.202+0000 7f8babe00640 1 -- 172.21.15.28:0/189197471 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f7d5b027d0 con 0x55f7d5dc02a0 2024-05-12T22:15:12.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.202+0000 7f8bb0c00640 1 --2- 172.21.15.28:0/189197471 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7d5dbf980 0x55f7d5dc7900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:15:12.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.202+0000 7f8babe00640 1 --2- 172.21.15.28:0/189197471 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f7d5dc02a0 0x55f7d5dc7e40 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f8b980a44e0 tx=0x7f8b98098130 comp rx=0 tx=0).ready entity=mon.2 client_cookie=32acf5ca27c621ab server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:15:12.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.202+0000 7f8b92000640 10 client.0 ms_handle_connect on v2:172.21.15.86:3300/0 2024-05-12T22:15:12.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.202+0000 7f8b92000640 1 -- 172.21.15.28:0/189197471 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b980cd460 con 0x55f7d5dc02a0 2024-05-12T22:15:12.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.203+0000 7f8b92000640 1 -- 172.21.15.28:0/189197471 <== mon.2 v2:172.21.15.86:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f8b9809b3f0 con 0x55f7d5dc02a0 2024-05-12T22:15:12.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.203+0000 7f8bb657d740 1 -- 172.21.15.28:0/189197471 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8ba0004760 con 0x55f7d5dc02a0 2024-05-12T22:15:12.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.203+0000 7f8b92000640 1 -- 172.21.15.28:0/189197471 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f8b9809c4a0 con 0x55f7d5dc02a0 2024-05-12T22:15:12.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.203+0000 7f8bb657d740 10 client.26627 fetch_fsmap learned FSMap version 26 2024-05-12T22:15:12.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.203+0000 7f8bb657d740 1 -- 172.21.15.28:0/189197471 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55f7d5dd99d0 con 0x55f7d5dc02a0 2024-05-12T22:15:12.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.203+0000 7f8b92000640 1 -- 172.21.15.28:0/189197471 <== mon.2 v2:172.21.15.86:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8ba0004760 con 0x55f7d5dc02a0 2024-05-12T22:15:12.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.203+0000 7f8b92000640 1 -- 172.21.15.28:0/189197471 <== mon.2 v2:172.21.15.86:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f8b98018ab0 con 0x55f7d5dc02a0 2024-05-12T22:15:12.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.203+0000 7f8bb657d740 10 client.26627 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:15:12.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.203+0000 7f8bb657d740 10 client.26627 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-05-12T22:15:12.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.203+0000 7f8bb657d740 20 client.26627 populate_metadata read hostname 'smithi028' 2024-05-12T22:15:12.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.203+0000 7f8bb657d740 1 --2- 172.21.15.28:0/189197471 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x55f7d5ddae90 0x55f7d5ddd2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:12.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.203+0000 7f8bb657d740 4 client.26627 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-12T22:15:12.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.203+0000 7f8bb657d740 1 -- 172.21.15.28:0/189197471 --> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f7d5d32160 con 0x55f7d5ddae90 2024-05-12T22:15:12.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.204+0000 7f8bb0c00640 1 --2- 172.21.15.28:0/189197471 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x55f7d5ddae90 0x55f7d5ddd2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:12.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.204+0000 7f8bb0c00640 1 --2- 172.21.15.28:0/189197471 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x55f7d5ddae90 0x55f7d5ddd2e0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=5184b81efa96702d server_cookie=206a533a47536150 in_seq=0 out_seq=0 2024-05-12T22:15:12.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.204+0000 7f8b92000640 10 client.26627 ms_handle_connect on v2:172.21.15.28:6840/761183870 2024-05-12T22:15:12.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.205+0000 7f8b92000640 1 -- 172.21.15.28:0/189197471 <== mds.1 v2:172.21.15.28:6840/761183870 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f7d5d32160 con 0x55f7d5ddae90 2024-05-12T22:15:12.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.205+0000 7f8b92000640 10 client.26627 handle_command_reply: tid=0 2024-05-12T22:15:12.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.205+0000 7f8bb657d740 1 client.26627 shutdown 2024-05-12T22:15:12.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.205+0000 7f8bb657d740 1 -- 172.21.15.28:0/189197471 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f7d5dc02a0 msgr2=0x55f7d5dc7e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:15:12.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.205+0000 7f8bb657d740 1 --2- 172.21.15.28:0/189197471 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f7d5dc02a0 0x55f7d5dc7e40 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f8b980a44e0 tx=0x7f8b98098130 comp rx=0 tx=0).stop 2024-05-12T22:15:12.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.205+0000 7f8bb657d740 1 -- 172.21.15.28:0/189197471 shutdown_connections 2024-05-12T22:15:12.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.205+0000 7f8bb657d740 1 --2- 172.21.15.28:0/189197471 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x55f7d5ddae90 0x55f7d5ddd2e0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:12.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.205+0000 7f8bb657d740 1 --2- 172.21.15.28:0/189197471 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f7d5dca970 0x55f7d5dc8380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:12.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.205+0000 7f8bb657d740 1 --2- 172.21.15.28:0/189197471 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f7d5dc02a0 0x55f7d5dc7e40 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:12.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.206+0000 7f8bb657d740 1 --2- 172.21.15.28:0/189197471 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7d5dbf980 0x55f7d5dc7900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:12.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.206+0000 7f8b92000640 0 client.26627 ms_handle_reset on v2:172.21.15.28:6840/761183870 2024-05-12T22:15:12.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.206+0000 7f8bb657d740 1 -- 172.21.15.28:0/189197471 >> 172.21.15.28:0/189197471 conn(0x55f7d5dbbca0 msgr2=0x55f7d5dc1aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:15:12.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.206+0000 7f8bb657d740 1 -- 172.21.15.28:0/189197471 shutdown_connections 2024-05-12T22:15:12.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.206+0000 7f8bb657d740 1 -- 172.21.15.28:0/189197471 wait complete. 2024-05-12T22:15:12.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.206+0000 7f8bb657d740 20 client.26627 trim_cache size 0 max 16384 2024-05-12T22:15:12.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.221+0000 7f8bb657d740 1 Processor -- start 2024-05-12T22:15:12.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.221+0000 7f8bb657d740 1 -- start start 2024-05-12T22:15:12.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.221+0000 7f8bb657d740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f7d5dbbca0 0x55f7d5dd65f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:12.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.221+0000 7f8bb657d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7d5dbf980 0x55f7d5dbac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:12.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.221+0000 7f8bb657d740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f7d5dc02a0 0x55f7d5de2bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:12.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.221+0000 7f8bb657d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8bac1b6550 con 0x55f7d5dbf980 2024-05-12T22:15:12.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.221+0000 7f8bb657d740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f8bac1b3e20 con 0x55f7d5dbbca0 2024-05-12T22:15:12.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.221+0000 7f8bb657d740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f8bac1b1c70 con 0x55f7d5dc02a0 2024-05-12T22:15:12.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.221+0000 7f8babe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7d5dbf980 0x55f7d5dbac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:12.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.221+0000 7f8babe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7d5dbf980 0x55f7d5dbac40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34420/0 (socket says 172.21.15.28:34420) 2024-05-12T22:15:12.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.221+0000 7f8babe00640 1 -- 172.21.15.28:0/3268963629 learned_addr learned my addr 172.21.15.28:0/3268963629 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:15:12.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.222+0000 7f8bb0c00640 1 --2- 172.21.15.28:0/3268963629 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f7d5dbbca0 0x55f7d5dd65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:12.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.222+0000 7f8babe00640 1 -- 172.21.15.28:0/3268963629 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f7d5dc02a0 msgr2=0x55f7d5de2bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:15:12.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.222+0000 7f8bb1600640 1 --2- 172.21.15.28:0/3268963629 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f7d5dc02a0 0x55f7d5de2bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:12.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.222+0000 7f8babe00640 1 --2- 172.21.15.28:0/3268963629 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f7d5dc02a0 0x55f7d5de2bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:12.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.222+0000 7f8babe00640 1 -- 172.21.15.28:0/3268963629 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f7d5dbbca0 msgr2=0x55f7d5dd65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:15:12.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.222+0000 7f8babe00640 1 --2- 172.21.15.28:0/3268963629 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f7d5dbbca0 0x55f7d5dd65f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:12.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.222+0000 7f8babe00640 1 -- 172.21.15.28:0/3268963629 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f7d5dc5e50 con 0x55f7d5dbf980 2024-05-12T22:15:12.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.222+0000 7f8bb0c00640 1 --2- 172.21.15.28:0/3268963629 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f7d5dbbca0 0x55f7d5dd65f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:15:12.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.222+0000 7f8babe00640 1 --2- 172.21.15.28:0/3268963629 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7d5dbf980 0x55f7d5dbac40 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f8b9809bd30 tx=0x7f8b98098040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e249710e7e18d3f1 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:15:12.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.222+0000 7f8b9d200640 1 -- 172.21.15.28:0/3268963629 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b9809f020 con 0x55f7d5dbf980 2024-05-12T22:15:12.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.223+0000 7f8b9d200640 1 -- 172.21.15.28:0/3268963629 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8b980a0150 con 0x55f7d5dbf980 2024-05-12T22:15:12.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.223+0000 7f8b9d200640 1 -- 172.21.15.28:0/3268963629 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b9809ae00 con 0x55f7d5dbf980 2024-05-12T22:15:12.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.223+0000 7f8bb657d740 1 -- 172.21.15.28:0/3268963629 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7d5dbf980 msgr2=0x55f7d5dbac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:15:12.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.223+0000 7f8bb657d740 1 --2- 172.21.15.28:0/3268963629 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7d5dbf980 0x55f7d5dbac40 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f8b9809bd30 tx=0x7f8b98098040 comp rx=0 tx=0).stop 2024-05-12T22:15:12.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.223+0000 7f8bb657d740 1 -- 172.21.15.28:0/3268963629 shutdown_connections 2024-05-12T22:15:12.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.223+0000 7f8bb657d740 1 --2- 172.21.15.28:0/3268963629 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f7d5dc02a0 0x55f7d5de2bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:12.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.223+0000 7f8bb657d740 1 --2- 172.21.15.28:0/3268963629 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7d5dbf980 0x55f7d5dbac40 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:12.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.223+0000 7f8bb657d740 1 --2- 172.21.15.28:0/3268963629 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f7d5dbbca0 0x55f7d5dd65f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:12.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.223+0000 7f8bb657d740 1 -- 172.21.15.28:0/3268963629 >> 172.21.15.28:0/3268963629 conn(0x55f7d5dc32d0 msgr2=0x55f7d5dba530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:15:12.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.223+0000 7f8bb657d740 1 -- 172.21.15.28:0/3268963629 shutdown_connections 2024-05-12T22:15:12.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.223+0000 7f8bb657d740 1 -- 172.21.15.28:0/3268963629 wait complete. 2024-05-12T22:15:12.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.224+0000 7f8bb657d740 1 -- start start 2024-05-12T22:15:12.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.224+0000 7f8bb657d740 1 Processor -- start 2024-05-12T22:15:12.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.224+0000 7f8bb657d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7d5dbbca0 0x55f7d5dce460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:12.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.224+0000 7f8bb657d740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f7d5dbf980 0x55f7d5dce9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:12.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.224+0000 7f8bb657d740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f7d5dc02a0 0x55f7d5dceee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:12.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.224+0000 7f8bb657d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8bac1b22e0 con 0x55f7d5dbbca0 2024-05-12T22:15:12.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.224+0000 7f8bb657d740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f8bac1b3750 con 0x55f7d5dc02a0 2024-05-12T22:15:12.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.224+0000 7f8bb657d740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f8bac1b29b0 con 0x55f7d5dbf980 2024-05-12T22:15:12.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.224+0000 7f8bb657d740 10 client.0 _ll_register_callbacks cb 0x7f8b980ccf70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:15:12.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.225+0000 7f8bb0c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7d5dbbca0 0x55f7d5dce460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:12.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.225+0000 7f8bb0c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7d5dbbca0 0x55f7d5dce460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34436/0 (socket says 172.21.15.28:34436) 2024-05-12T22:15:12.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.225+0000 7f8bb0c00640 1 -- 172.21.15.28:0/4010584844 learned_addr learned my addr 172.21.15.28:0/4010584844 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:15:12.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.225+0000 7f8bb1600640 1 --2- 172.21.15.28:0/4010584844 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f7d5dc02a0 0x55f7d5dceee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:12.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.225+0000 7f8babe00640 1 --2- 172.21.15.28:0/4010584844 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f7d5dbf980 0x55f7d5dce9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:12.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.225+0000 7f8bb0c00640 1 -- 172.21.15.28:0/4010584844 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f7d5dbf980 msgr2=0x55f7d5dce9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:15:12.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.225+0000 7f8bb0c00640 1 --2- 172.21.15.28:0/4010584844 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f7d5dbf980 0x55f7d5dce9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:12.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.225+0000 7f8bb0c00640 1 -- 172.21.15.28:0/4010584844 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f7d5dc02a0 msgr2=0x55f7d5dceee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:15:12.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.225+0000 7f8bb0c00640 1 --2- 172.21.15.28:0/4010584844 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f7d5dc02a0 0x55f7d5dceee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:12.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.225+0000 7f8bb0c00640 1 -- 172.21.15.28:0/4010584844 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f7d5e40420 con 0x55f7d5dbbca0 2024-05-12T22:15:12.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.225+0000 7f8bb0c00640 1 --2- 172.21.15.28:0/4010584844 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7d5dbbca0 0x55f7d5dce460 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f8ba002aa10 tx=0x7f8ba00036e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=483d4b663f4e4e2e server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:15:12.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.225+0000 7f8b92000640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-12T22:15:12.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.226+0000 7f8b92000640 1 -- 172.21.15.28:0/4010584844 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ba0059060 con 0x55f7d5dbbca0 2024-05-12T22:15:12.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.226+0000 7f8b92000640 1 -- 172.21.15.28:0/4010584844 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f8ba0003d10 con 0x55f7d5dbbca0 2024-05-12T22:15:12.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.226+0000 7f8bb657d740 1 -- 172.21.15.28:0/4010584844 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8b980a0150 con 0x55f7d5dbbca0 2024-05-12T22:15:12.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.226+0000 7f8b92000640 1 -- 172.21.15.28:0/4010584844 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f8ba0035070 con 0x55f7d5dbbca0 2024-05-12T22:15:12.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.226+0000 7f8bb657d740 10 client.16509 fetch_fsmap learned FSMap version 26 2024-05-12T22:15:12.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.226+0000 7f8bb657d740 1 -- 172.21.15.28:0/4010584844 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55f7d5e416c0 con 0x55f7d5dbbca0 2024-05-12T22:15:12.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.226+0000 7f8b92000640 1 -- 172.21.15.28:0/4010584844 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8b980a0150 con 0x55f7d5dbbca0 2024-05-12T22:15:12.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.226+0000 7f8b92000640 1 -- 172.21.15.28:0/4010584844 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f8ba0053170 con 0x55f7d5dbbca0 2024-05-12T22:15:12.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.226+0000 7f8bb657d740 10 client.16509 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:15:12.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.226+0000 7f8bb657d740 10 client.16509 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-05-12T22:15:12.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.227+0000 7f8bb657d740 20 client.16509 populate_metadata read hostname 'smithi028' 2024-05-12T22:15:12.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.227+0000 7f8bb657d740 1 --2- 172.21.15.28:0/4010584844 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x55f7d5dca970 0x55f7d5e43a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:12.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.227+0000 7f8bb657d740 4 client.16509 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-12T22:15:12.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.227+0000 7f8bb657d740 1 -- 172.21.15.28:0/4010584844 --> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f7d5d2c4b0 con 0x55f7d5dca970 2024-05-12T22:15:12.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.227+0000 7f8babe00640 1 --2- 172.21.15.28:0/4010584844 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x55f7d5dca970 0x55f7d5e43a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:12.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.227+0000 7f8babe00640 1 --2- 172.21.15.28:0/4010584844 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x55f7d5dca970 0x55f7d5e43a60 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=5baf8e91d6b550d9 server_cookie=a1b773331efd18d0 in_seq=0 out_seq=0 2024-05-12T22:15:12.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.227+0000 7f8b92000640 10 client.16509 ms_handle_connect on v2:172.21.15.28:6840/761183870 2024-05-12T22:15:12.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.228+0000 7f8b92000640 1 -- 172.21.15.28:0/4010584844 <== mds.1 v2:172.21.15.28:6840/761183870 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f7d5d2c4b0 con 0x55f7d5dca970 2024-05-12T22:15:12.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.228+0000 7f8b92000640 10 client.16509 handle_command_reply: tid=0 2024-05-12T22:15:12.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.228+0000 7f8bb657d740 1 client.16509 shutdown 2024-05-12T22:15:12.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.228+0000 7f8bb657d740 1 -- 172.21.15.28:0/4010584844 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7d5dbbca0 msgr2=0x55f7d5dce460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:15:12.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.228+0000 7f8bb657d740 1 --2- 172.21.15.28:0/4010584844 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7d5dbbca0 0x55f7d5dce460 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f8ba002aa10 tx=0x7f8ba00036e0 comp rx=0 tx=0).stop 2024-05-12T22:15:12.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.228+0000 7f8bb657d740 1 -- 172.21.15.28:0/4010584844 shutdown_connections 2024-05-12T22:15:12.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.228+0000 7f8bb657d740 1 --2- 172.21.15.28:0/4010584844 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x55f7d5dca970 0x55f7d5e43a60 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:12.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.228+0000 7f8bb657d740 1 --2- 172.21.15.28:0/4010584844 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f7d5dc02a0 0x55f7d5dceee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:12.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.228+0000 7f8bb657d740 1 --2- 172.21.15.28:0/4010584844 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f7d5dbf980 0x55f7d5dce9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:12.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.228+0000 7f8bb657d740 1 --2- 172.21.15.28:0/4010584844 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7d5dbbca0 0x55f7d5dce460 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:12.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.228+0000 7f8b92000640 0 client.16509 ms_handle_reset on v2:172.21.15.28:6840/761183870 2024-05-12T22:15:12.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.228+0000 7f8bb657d740 1 -- 172.21.15.28:0/4010584844 >> 172.21.15.28:0/4010584844 conn(0x55f7d5dc32d0 msgr2=0x55f7d5dc8130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:15:12.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.228+0000 7f8bb657d740 1 -- 172.21.15.28:0/4010584844 shutdown_connections 2024-05-12T22:15:12.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.228+0000 7f8bb657d740 1 -- 172.21.15.28:0/4010584844 wait complete. 2024-05-12T22:15:12.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.228+0000 7f8bb657d740 20 client.16509 trim_cache size 0 max 16384 2024-05-12T22:15:12.237 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-05-12T22:15:12.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.229+0000 7f8bb3600640 1 -- 172.21.15.28:0/2565192959 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f8b940a2d00 msgr2=0x7f8b940a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:15:12.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.229+0000 7f8bb3600640 1 --2- 172.21.15.28:0/2565192959 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f8b940a2d00 0x7f8b940a51c0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f8bac06ac20 tx=0x7f8ba4009500 comp rx=0 tx=0).stop 2024-05-12T22:15:12.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.229+0000 7f8bb3600640 1 -- 172.21.15.28:0/2565192959 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bac076680 msgr2=0x7f8bac1a4ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:15:12.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.229+0000 7f8bb3600640 1 --2- 172.21.15.28:0/2565192959 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bac076680 0x7f8bac1a4ad0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f8b9800e9f0 tx=0x7f8b9800eeb0 comp rx=0 tx=0).stop 2024-05-12T22:15:12.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.229+0000 7f8bb3600640 1 -- 172.21.15.28:0/2565192959 shutdown_connections 2024-05-12T22:15:12.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.229+0000 7f8bb3600640 1 --2- 172.21.15.28:0/2565192959 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f8b940a2d00 0x7f8b940a51c0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:12.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.229+0000 7f8bb3600640 1 --2- 172.21.15.28:0/2565192959 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8bac1126b0 0x7f8bac19e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:12.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.229+0000 7f8bb3600640 1 --2- 172.21.15.28:0/2565192959 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bac076680 0x7f8bac1a4ad0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:12.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.229+0000 7f8bb3600640 1 --2- 172.21.15.28:0/2565192959 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8bac075d60 0x7f8bac1a4590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:12.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.229+0000 7f8bb3600640 1 -- 172.21.15.28:0/2565192959 >> 172.21.15.28:0/2565192959 conn(0x7f8bac0ff4a0 msgr2=0x7f8bac113320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:15:12.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.229+0000 7f8bb3600640 1 -- 172.21.15.28:0/2565192959 shutdown_connections 2024-05-12T22:15:12.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.229+0000 7f8bb3600640 1 -- 172.21.15.28:0/2565192959 wait complete. 2024-05-12T22:15:12.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:15:11 smithi086 ceph-mon[34297]: pgmap v189: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-05-12T22:15:12.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:15:11 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/1116145837' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:15:12.247 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-05-12T22:15:12.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.460+0000 7ff1b1a00640 1 -- 172.21.15.28:0/199664964 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1ac103a10 msgr2=0x7ff1ac103e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:15:12.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.460+0000 7ff1b1a00640 1 --2- 172.21.15.28:0/199664964 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1ac103a10 0x7ff1ac103e90 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7ff19c009930 tx=0x7ff19c02a440 comp rx=0 tx=0).stop 2024-05-12T22:15:12.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.460+0000 7ff1b1a00640 1 -- 172.21.15.28:0/199664964 shutdown_connections 2024-05-12T22:15:12.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.460+0000 7ff1b1a00640 1 --2- 172.21.15.28:0/199664964 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff1ac1043d0 0x7ff1ac1108e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:12.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.460+0000 7ff1b1a00640 1 --2- 172.21.15.28:0/199664964 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1ac103a10 0x7ff1ac103e90 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:12.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.460+0000 7ff1b1a00640 1 --2- 172.21.15.28:0/199664964 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1ac109a20 0x7ff1ac109e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:12.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.460+0000 7ff1b1a00640 1 -- 172.21.15.28:0/199664964 >> 172.21.15.28:0/199664964 conn(0x7ff1ac0ff4a0 msgr2=0x7ff1ac1018c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:15:12.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.460+0000 7ff1b1a00640 1 -- 172.21.15.28:0/199664964 shutdown_connections 2024-05-12T22:15:12.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.460+0000 7ff1b1a00640 1 -- 172.21.15.28:0/199664964 wait complete. 2024-05-12T22:15:12.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.461+0000 7ff1b1a00640 1 Processor -- start 2024-05-12T22:15:12.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.461+0000 7ff1b1a00640 1 -- start start 2024-05-12T22:15:12.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.462+0000 7ff1b1a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1ac103a10 0x7ff1ac19a390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:12.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.462+0000 7ff1b1a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1ac1043d0 0x7ff1ac19a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:12.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.462+0000 7ff1b1a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff1ac109a20 0x7ff1ac19ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:12.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.462+0000 7ff1b1a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff1ac113e50 con 0x7ff1ac1043d0 2024-05-12T22:15:12.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.462+0000 7ff1b1a00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7ff1ac113cd0 con 0x7ff1ac109a20 2024-05-12T22:15:12.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.462+0000 7ff1b1a00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff1ac113fd0 con 0x7ff1ac103a10 2024-05-12T22:15:12.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.462+0000 7ff1aa000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1ac1043d0 0x7ff1ac19a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:12.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.462+0000 7ff1aa000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1ac1043d0 0x7ff1ac19a8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34448/0 (socket says 172.21.15.28:34448) 2024-05-12T22:15:12.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.462+0000 7ff1aa000640 1 -- 172.21.15.28:0/1759946385 learned_addr learned my addr 172.21.15.28:0/1759946385 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:15:12.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.462+0000 7ff1aaa00640 1 --2- 172.21.15.28:0/1759946385 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1ac103a10 0x7ff1ac19a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:12.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.462+0000 7ff1ab400640 1 --2- 172.21.15.28:0/1759946385 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff1ac109a20 0x7ff1ac19ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:12.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.462+0000 7ff1aaa00640 1 -- 172.21.15.28:0/1759946385 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff1ac109a20 msgr2=0x7ff1ac19ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:15:12.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.462+0000 7ff1aaa00640 1 --2- 172.21.15.28:0/1759946385 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff1ac109a20 0x7ff1ac19ec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:12.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.462+0000 7ff1aaa00640 1 -- 172.21.15.28:0/1759946385 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1ac1043d0 msgr2=0x7ff1ac19a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:15:12.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.462+0000 7ff1aaa00640 1 --2- 172.21.15.28:0/1759946385 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1ac1043d0 0x7ff1ac19a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:12.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.462+0000 7ff1aaa00640 1 -- 172.21.15.28:0/1759946385 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1ac19f380 con 0x7ff1ac103a10 2024-05-12T22:15:12.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.462+0000 7ff1aa000640 1 --2- 172.21.15.28:0/1759946385 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1ac1043d0 0x7ff1ac19a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:15:12.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.463+0000 7ff1aaa00640 1 --2- 172.21.15.28:0/1759946385 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1ac103a10 0x7ff1ac19a390 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7ff19800e9f0 tx=0x7ff19800edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:15:12.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.463+0000 7ff1a3400640 1 -- 172.21.15.28:0/1759946385 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff19800ccc0 con 0x7ff1ac103a10 2024-05-12T22:15:12.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.463+0000 7ff1a3400640 1 -- 172.21.15.28:0/1759946385 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff198010910 con 0x7ff1ac103a10 2024-05-12T22:15:12.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.463+0000 7ff1b1a00640 1 -- 172.21.15.28:0/1759946385 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1ac19f670 con 0x7ff1ac103a10 2024-05-12T22:15:12.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.463+0000 7ff1a3400640 1 -- 172.21.15.28:0/1759946385 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff198018800 con 0x7ff1ac103a10 2024-05-12T22:15:12.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.463+0000 7ff1b1a00640 1 -- 172.21.15.28:0/1759946385 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1ac1a6f00 con 0x7ff1ac103a10 2024-05-12T22:15:12.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.468+0000 7ff1b4837740 1 Processor -- start 2024-05-12T22:15:12.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.468+0000 7ff1b4837740 1 -- start start 2024-05-12T22:15:12.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.468+0000 7ff1b4837740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a77b27c980 0x55a77b27cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:12.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.468+0000 7ff1b4837740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a77b27d2a0 0x55a77b287430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:12.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.468+0000 7ff1b4837740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a77b287970 0x55a77b289d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:12.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.468+0000 7ff1b4837740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a77b1e1a30 con 0x55a77b27d2a0 2024-05-12T22:15:12.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.468+0000 7ff1b4837740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x55a77b189700 con 0x55a77b287970 2024-05-12T22:15:12.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.468+0000 7ff1b4837740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55a77b1dd9f0 con 0x55a77b27c980 2024-05-12T22:15:12.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.469+0000 7ff1ab400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a77b27c980 0x55a77b27cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:12.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.469+0000 7ff1ab400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a77b27c980 0x55a77b27cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:36496/0 (socket says 172.21.15.28:36496) 2024-05-12T22:15:12.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.469+0000 7ff1ab400640 1 -- 172.21.15.28:0/120589186 learned_addr learned my addr 172.21.15.28:0/120589186 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:15:12.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.469+0000 7ff1a3400640 1 -- 172.21.15.28:0/1759946385 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7ff198010ac0 con 0x7ff1ac103a10 2024-05-12T22:15:12.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.469+0000 7ff1aa000640 1 --2- 172.21.15.28:0/120589186 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a77b287970 0x55a77b289d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:12.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.469+0000 7ff1ab400640 1 -- 172.21.15.28:0/120589186 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a77b287970 msgr2=0x55a77b289d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:15:12.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.469+0000 7ff1ab400640 1 --2- 172.21.15.28:0/120589186 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a77b287970 0x55a77b289d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:12.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.469+0000 7ff1ab400640 1 -- 172.21.15.28:0/120589186 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a77b27d2a0 msgr2=0x55a77b287430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:15:12.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.469+0000 7ff1ab400640 1 --2- 172.21.15.28:0/120589186 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a77b27d2a0 0x55a77b287430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:12.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.469+0000 7ff1ab400640 1 -- 172.21.15.28:0/120589186 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a77aef53f0 con 0x55a77b27c980 2024-05-12T22:15:12.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.470+0000 7ff1ab400640 1 --2- 172.21.15.28:0/120589186 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a77b27c980 0x55a77b27cd60 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7ff1a4011440 tx=0x7ff1a4011900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fbef57ec4d31446c server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:15:12.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.470+0000 7ff197e00640 1 -- 172.21.15.28:0/120589186 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1a4003c30 con 0x55a77b27c980 2024-05-12T22:15:12.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.470+0000 7ff197e00640 1 -- 172.21.15.28:0/120589186 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff1a4003dd0 con 0x55a77b27c980 2024-05-12T22:15:12.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.470+0000 7ff197e00640 1 -- 172.21.15.28:0/120589186 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1a400faf0 con 0x55a77b27c980 2024-05-12T22:15:12.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.470+0000 7ff1a3400640 1 --2- 172.21.15.28:0/1759946385 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7ff1900a2d80 0x7ff1900a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:12.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.471+0000 7ff1b4837740 1 -- 172.21.15.28:0/120589186 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a77b27c980 msgr2=0x55a77b27cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:15:12.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.471+0000 7ff1b4837740 1 --2- 172.21.15.28:0/120589186 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a77b27c980 0x55a77b27cd60 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7ff1a4011440 tx=0x7ff1a4011900 comp rx=0 tx=0).stop 2024-05-12T22:15:12.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.471+0000 7ff1a3400640 1 -- 172.21.15.28:0/1759946385 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7ff1980cf650 con 0x7ff1ac103a10 2024-05-12T22:15:12.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.471+0000 7ff1b4837740 1 -- 172.21.15.28:0/120589186 shutdown_connections 2024-05-12T22:15:12.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.471+0000 7ff1b4837740 1 --2- 172.21.15.28:0/120589186 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a77b287970 0x55a77b289d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:12.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.471+0000 7ff1b4837740 1 --2- 172.21.15.28:0/120589186 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a77b27d2a0 0x55a77b287430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:12.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.471+0000 7ff1b4837740 1 --2- 172.21.15.28:0/120589186 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a77b27c980 0x55a77b27cd60 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:12.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.471+0000 7ff1aaa00640 1 --2- 172.21.15.28:0/1759946385 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7ff1900a2d80 0x7ff1900a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:12.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.471+0000 7ff1b4837740 1 -- 172.21.15.28:0/120589186 >> 172.21.15.28:0/120589186 conn(0x55a77b278ca0 msgr2=0x55a77b279080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:15:12.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.471+0000 7ff1b4837740 1 -- 172.21.15.28:0/120589186 shutdown_connections 2024-05-12T22:15:12.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.471+0000 7ff1b4837740 1 -- 172.21.15.28:0/120589186 wait complete. 2024-05-12T22:15:12.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.471+0000 7ff1b4837740 1 -- start start 2024-05-12T22:15:12.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.471+0000 7ff1aaa00640 1 --2- 172.21.15.28:0/1759946385 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7ff1900a2d80 0x7ff1900a5240 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x55a77b27df40 tx=0x7ff198009f20 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:15:12.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.472+0000 7ff1b4837740 1 Processor -- start 2024-05-12T22:15:12.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.472+0000 7ff1b4837740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a77b27c980 0x55a77b284910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:12.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.472+0000 7ff1b4837740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a77b27d2a0 0x55a77b284e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:12.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.472+0000 7ff1b4837740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a77b287970 0x55a77b285390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:12.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.472+0000 7ff1b4837740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a77b28a760 con 0x55a77b287970 2024-05-12T22:15:12.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.472+0000 7ff1b4837740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x55a77b19ca70 con 0x55a77b27c980 2024-05-12T22:15:12.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.472+0000 7ff1b4837740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55a77b28a8e0 con 0x55a77b27d2a0 2024-05-12T22:15:12.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.472+0000 7ff1b4837740 10 client.0 _ll_register_callbacks cb 0x55a77b17fba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:15:12.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.473+0000 7ff1ab400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a77b287970 0x55a77b285390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:12.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.473+0000 7ff1ab400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a77b287970 0x55a77b285390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34468/0 (socket says 172.21.15.28:34468) 2024-05-12T22:15:12.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.473+0000 7ff1ab400640 1 -- 172.21.15.28:0/492968074 learned_addr learned my addr 172.21.15.28:0/492968074 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:15:12.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.473+0000 7ff1aa000640 1 --2- 172.21.15.28:0/492968074 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a77b27d2a0 0x55a77b284e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:12.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.473+0000 7ff1ab400640 1 --2- 172.21.15.28:0/492968074 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a77b27c980 0x55a77b284910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:12.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.473+0000 7ff1aa000640 1 -- 172.21.15.28:0/492968074 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a77b27c980 msgr2=0x55a77b284910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:15:12.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.473+0000 7ff1aa000640 1 --2- 172.21.15.28:0/492968074 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a77b27c980 0x55a77b284910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:12.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.473+0000 7ff1aa000640 1 -- 172.21.15.28:0/492968074 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a77b287970 msgr2=0x55a77b285390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:15:12.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.473+0000 7ff1aa000640 1 --2- 172.21.15.28:0/492968074 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a77b287970 0x55a77b285390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:12.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.473+0000 7ff1aa000640 1 -- 172.21.15.28:0/492968074 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a77afbf7d0 con 0x55a77b27d2a0 2024-05-12T22:15:12.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.473+0000 7ff1aa000640 1 --2- 172.21.15.28:0/492968074 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a77b27d2a0 0x55a77b284e50 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7ff19c000c00 tx=0x7ff19c005270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=935f4ebededae2ec server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:15:12.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.473+0000 7ff194c00640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-12T22:15:12.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.474+0000 7ff194c00640 1 -- 172.21.15.28:0/492968074 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff19c035040 con 0x55a77b27d2a0 2024-05-12T22:15:12.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.474+0000 7ff194c00640 1 -- 172.21.15.28:0/492968074 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7ff19c004760 con 0x55a77b27d2a0 2024-05-12T22:15:12.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.474+0000 7ff1b4837740 1 -- 172.21.15.28:0/492968074 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff1a4003dd0 con 0x55a77b27d2a0 2024-05-12T22:15:12.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.474+0000 7ff194c00640 1 -- 172.21.15.28:0/492968074 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7ff19c004960 con 0x55a77b27d2a0 2024-05-12T22:15:12.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.474+0000 7ff1b4837740 10 client.26602 fetch_fsmap learned FSMap version 26 2024-05-12T22:15:12.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.474+0000 7ff1b4837740 1 -- 172.21.15.28:0/492968074 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55a77b296a50 con 0x55a77b27d2a0 2024-05-12T22:15:12.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.475+0000 7ff194c00640 1 -- 172.21.15.28:0/492968074 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff1a4003dd0 con 0x55a77b27d2a0 2024-05-12T22:15:12.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.475+0000 7ff194c00640 1 -- 172.21.15.28:0/492968074 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7ff19c0035f0 con 0x55a77b27d2a0 2024-05-12T22:15:12.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.475+0000 7ff1b4837740 10 client.26602 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:15:12.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.475+0000 7ff1b4837740 10 client.26602 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:15:12.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.475+0000 7ff1b4837740 20 client.26602 populate_metadata read hostname 'smithi028' 2024-05-12T22:15:12.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.475+0000 7ff1b4837740 1 --2- 172.21.15.28:0/492968074 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55a77b297f00 0x55a77b29a320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:12.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.475+0000 7ff1b4837740 4 client.26602 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-12T22:15:12.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.475+0000 7ff1b4837740 1 -- 172.21.15.28:0/492968074 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a77b1ef160 con 0x55a77b297f00 2024-05-12T22:15:12.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.475+0000 7ff1ab400640 1 --2- 172.21.15.28:0/492968074 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55a77b297f00 0x55a77b29a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:12.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.476+0000 7ff1ab400640 1 --2- 172.21.15.28:0/492968074 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55a77b297f00 0x55a77b29a320 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=12a2e6d3bf358a0d server_cookie=768ffb55733d2913 in_seq=0 out_seq=0 2024-05-12T22:15:12.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.476+0000 7ff194c00640 10 client.26602 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:15:12.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.477+0000 7ff194c00640 1 -- 172.21.15.28:0/492968074 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a77b1ef160 con 0x55a77b297f00 2024-05-12T22:15:12.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.477+0000 7ff194c00640 10 client.26602 handle_command_reply: tid=0 2024-05-12T22:15:12.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.477+0000 7ff1b4837740 1 client.26602 shutdown 2024-05-12T22:15:12.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.477+0000 7ff1b4837740 1 -- 172.21.15.28:0/492968074 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a77b27d2a0 msgr2=0x55a77b284e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:15:12.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.477+0000 7ff1b4837740 1 --2- 172.21.15.28:0/492968074 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a77b27d2a0 0x55a77b284e50 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7ff19c000c00 tx=0x7ff19c005270 comp rx=0 tx=0).stop 2024-05-12T22:15:12.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.477+0000 7ff1b4837740 1 -- 172.21.15.28:0/492968074 shutdown_connections 2024-05-12T22:15:12.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.477+0000 7ff1b4837740 1 --2- 172.21.15.28:0/492968074 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55a77b297f00 0x55a77b29a320 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:12.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.477+0000 7ff1b4837740 1 --2- 172.21.15.28:0/492968074 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a77b287970 0x55a77b285390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:12.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.477+0000 7ff1b4837740 1 --2- 172.21.15.28:0/492968074 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a77b27d2a0 0x55a77b284e50 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:12.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.477+0000 7ff1b4837740 1 --2- 172.21.15.28:0/492968074 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a77b27c980 0x55a77b284910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:12.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.477+0000 7ff194c00640 0 client.26602 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:15:12.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.477+0000 7ff1b4837740 1 -- 172.21.15.28:0/492968074 >> 172.21.15.28:0/492968074 conn(0x55a77b278ca0 msgr2=0x55a77b27e980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:15:12.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.477+0000 7ff1b4837740 1 -- 172.21.15.28:0/492968074 shutdown_connections 2024-05-12T22:15:12.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.477+0000 7ff1b4837740 1 -- 172.21.15.28:0/492968074 wait complete. 2024-05-12T22:15:12.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.477+0000 7ff1b4837740 20 client.26602 trim_cache size 0 max 16384 2024-05-12T22:15:12.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.492+0000 7ff1b4837740 1 Processor -- start 2024-05-12T22:15:12.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.492+0000 7ff1b4837740 1 -- start start 2024-05-12T22:15:12.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.493+0000 7ff1b4837740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a77b27c980 0x55a77b293650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:12.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.493+0000 7ff1b4837740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a77b27d2a0 0x55a77b27f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:12.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.493+0000 7ff1b4837740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a77b287970 0x55a77b29fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:12.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.493+0000 7ff1b4837740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff1ac1b2010 con 0x55a77b27d2a0 2024-05-12T22:15:12.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.493+0000 7ff1b4837740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7ff1ac1af8e0 con 0x55a77b287970 2024-05-12T22:15:12.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.493+0000 7ff1b4837740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff1ac1ad730 con 0x55a77b27c980 2024-05-12T22:15:12.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.493+0000 7ff1aa000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a77b27d2a0 0x55a77b27f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:12.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.493+0000 7ff1ab400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a77b27c980 0x55a77b293650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:12.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.493+0000 7ff1aa000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a77b27d2a0 0x55a77b27f0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34472/0 (socket says 172.21.15.28:34472) 2024-05-12T22:15:12.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.493+0000 7ff1aa000640 1 -- 172.21.15.28:0/1306976540 learned_addr learned my addr 172.21.15.28:0/1306976540 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:15:12.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.493+0000 7ff1ab400640 1 --2- 172.21.15.28:0/1306976540 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a77b287970 0x55a77b29fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:12.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.493+0000 7ff1aa000640 1 -- 172.21.15.28:0/1306976540 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a77b27c980 msgr2=0x55a77b293650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:15:12.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.494+0000 7ff1aa000640 1 --2- 172.21.15.28:0/1306976540 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a77b27c980 0x55a77b293650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:12.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.494+0000 7ff1aa000640 1 -- 172.21.15.28:0/1306976540 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a77b287970 msgr2=0x55a77b29fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:15:12.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.494+0000 7ff1aa000640 1 --2- 172.21.15.28:0/1306976540 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a77b287970 0x55a77b29fbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:12.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.494+0000 7ff1aa000640 1 -- 172.21.15.28:0/1306976540 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a77b27d700 con 0x55a77b27d2a0 2024-05-12T22:15:12.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.494+0000 7ff1ab400640 1 --2- 172.21.15.28:0/1306976540 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a77b27c980 0x55a77b293650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:15:12.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.494+0000 7ff1aa000640 1 --2- 172.21.15.28:0/1306976540 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a77b27d2a0 0x55a77b27f0e0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7ff19c02a960 tx=0x7ff19c02a440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5ae68a823fcfcf server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:15:12.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.494+0000 7ff197400640 1 -- 172.21.15.28:0/1306976540 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff19c054030 con 0x55a77b27d2a0 2024-05-12T22:15:12.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.494+0000 7ff197400640 1 -- 172.21.15.28:0/1306976540 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff19c004760 con 0x55a77b27d2a0 2024-05-12T22:15:12.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.494+0000 7ff197400640 1 -- 172.21.15.28:0/1306976540 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff19c035030 con 0x55a77b27d2a0 2024-05-12T22:15:12.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.495+0000 7ff1b4837740 1 -- 172.21.15.28:0/1306976540 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a77b27d2a0 msgr2=0x55a77b27f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:15:12.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.495+0000 7ff1b4837740 1 --2- 172.21.15.28:0/1306976540 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a77b27d2a0 0x55a77b27f0e0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7ff19c02a960 tx=0x7ff19c02a440 comp rx=0 tx=0).stop 2024-05-12T22:15:12.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.495+0000 7ff1b4837740 1 -- 172.21.15.28:0/1306976540 shutdown_connections 2024-05-12T22:15:12.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.495+0000 7ff1b4837740 1 --2- 172.21.15.28:0/1306976540 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a77b287970 0x55a77b29fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:12.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.495+0000 7ff1b4837740 1 --2- 172.21.15.28:0/1306976540 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a77b27d2a0 0x55a77b27f0e0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:12.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.495+0000 7ff1b4837740 1 --2- 172.21.15.28:0/1306976540 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a77b27c980 0x55a77b293650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:12.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.495+0000 7ff1b4837740 1 -- 172.21.15.28:0/1306976540 >> 172.21.15.28:0/1306976540 conn(0x55a77b278ca0 msgr2=0x55a77b285160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:15:12.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.495+0000 7ff1b4837740 1 -- 172.21.15.28:0/1306976540 shutdown_connections 2024-05-12T22:15:12.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.495+0000 7ff1b4837740 1 -- 172.21.15.28:0/1306976540 wait complete. 2024-05-12T22:15:12.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.495+0000 7ff1b4837740 1 -- start start 2024-05-12T22:15:12.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.496+0000 7ff1b4837740 1 Processor -- start 2024-05-12T22:15:12.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.496+0000 7ff1b4837740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a77b27c980 0x55a77b2815c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:12.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.497+0000 7ff1b4837740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a77b27d2a0 0x55a77b281b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:12.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.497+0000 7ff1b4837740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a77b287970 0x55a77b282040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:12.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.497+0000 7ff1b4837740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff1ac1adda0 con 0x55a77b27c980 2024-05-12T22:15:12.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.497+0000 7ff1b4837740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7ff1ac1af210 con 0x55a77b27d2a0 2024-05-12T22:15:12.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.497+0000 7ff1b4837740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff1ac1ae470 con 0x55a77b287970 2024-05-12T22:15:12.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.497+0000 7ff1b4837740 10 client.0 _ll_register_callbacks cb 0x7ff19c0043f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:15:12.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.497+0000 7ff1ab400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a77b27c980 0x55a77b2815c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:12.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.497+0000 7ff1ab400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a77b27c980 0x55a77b2815c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34486/0 (socket says 172.21.15.28:34486) 2024-05-12T22:15:12.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.497+0000 7ff1aa000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a77b27d2a0 0x55a77b281b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:12.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.497+0000 7ff1ab400640 1 -- 172.21.15.28:0/2735829534 learned_addr learned my addr 172.21.15.28:0/2735829534 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:15:12.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.497+0000 7ff1ab400640 1 --2- 172.21.15.28:0/2735829534 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a77b287970 0x55a77b282040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:12.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.497+0000 7ff1aa000640 1 -- 172.21.15.28:0/2735829534 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a77b287970 msgr2=0x55a77b282040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:15:12.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.497+0000 7ff1aa000640 1 --2- 172.21.15.28:0/2735829534 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a77b287970 0x55a77b282040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:12.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.497+0000 7ff1aa000640 1 -- 172.21.15.28:0/2735829534 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a77b27c980 msgr2=0x55a77b2815c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:15:12.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.497+0000 7ff1aa000640 1 --2- 172.21.15.28:0/2735829534 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a77b27c980 0x55a77b2815c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:12.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.497+0000 7ff1aa000640 1 -- 172.21.15.28:0/2735829534 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a77b2fd460 con 0x55a77b27d2a0 2024-05-12T22:15:12.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.497+0000 7ff1ab400640 1 --2- 172.21.15.28:0/2735829534 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a77b27c980 0x55a77b2815c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:15:12.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.497+0000 7ff1aa000640 1 --2- 172.21.15.28:0/2735829534 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a77b27d2a0 0x55a77b281b00 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x55a77b29a920 tx=0x7ff19c005270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5cdceaf9641afe79 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:15:12.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.497+0000 7ff194c00640 10 client.0 ms_handle_connect on v2:172.21.15.86:3300/0 2024-05-12T22:15:12.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.498+0000 7ff194c00640 1 -- 172.21.15.28:0/2735829534 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff19c05d060 con 0x55a77b27d2a0 2024-05-12T22:15:12.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.498+0000 7ff194c00640 1 -- 172.21.15.28:0/2735829534 <== mon.2 v2:172.21.15.86:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7ff19c004b00 con 0x55a77b27d2a0 2024-05-12T22:15:12.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.498+0000 7ff1b4837740 1 -- 172.21.15.28:0/2735829534 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff19c004760 con 0x55a77b27d2a0 2024-05-12T22:15:12.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.498+0000 7ff194c00640 1 -- 172.21.15.28:0/2735829534 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7ff19c035070 con 0x55a77b27d2a0 2024-05-12T22:15:12.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.498+0000 7ff1b4837740 10 client.26651 fetch_fsmap learned FSMap version 26 2024-05-12T22:15:12.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.498+0000 7ff1b4837740 1 -- 172.21.15.28:0/2735829534 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55a77b2fe6a0 con 0x55a77b27d2a0 2024-05-12T22:15:12.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.498+0000 7ff194c00640 1 -- 172.21.15.28:0/2735829534 <== mon.2 v2:172.21.15.86:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff19c004760 con 0x55a77b27d2a0 2024-05-12T22:15:12.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.499+0000 7ff194c00640 1 -- 172.21.15.28:0/2735829534 <== mon.2 v2:172.21.15.86:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7ff19c031a90 con 0x55a77b27d2a0 2024-05-12T22:15:12.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.499+0000 7ff1b4837740 10 client.26651 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:15:12.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.499+0000 7ff1b4837740 10 client.26651 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:15:12.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.499+0000 7ff1b4837740 20 client.26651 populate_metadata read hostname 'smithi028' 2024-05-12T22:15:12.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.499+0000 7ff1b4837740 1 --2- 172.21.15.28:0/2735829534 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55a77b2feca0 0x55a77b301090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:12.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.499+0000 7ff1b4837740 4 client.26651 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-12T22:15:12.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.499+0000 7ff1b4837740 1 -- 172.21.15.28:0/2735829534 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a77b1e94b0 con 0x55a77b2feca0 2024-05-12T22:15:12.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.499+0000 7ff1ab400640 1 --2- 172.21.15.28:0/2735829534 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55a77b2feca0 0x55a77b301090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:12.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.499+0000 7ff1ab400640 1 --2- 172.21.15.28:0/2735829534 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55a77b2feca0 0x55a77b301090 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=439281c380c1d8ec server_cookie=c2694c08145b6a38 in_seq=0 out_seq=0 2024-05-12T22:15:12.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.499+0000 7ff194c00640 10 client.26651 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:15:12.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.500+0000 7ff194c00640 1 -- 172.21.15.28:0/2735829534 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a77b1e94b0 con 0x55a77b2feca0 2024-05-12T22:15:12.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.500+0000 7ff194c00640 10 client.26651 handle_command_reply: tid=0 2024-05-12T22:15:12.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.500+0000 7ff1b4837740 1 client.26651 shutdown 2024-05-12T22:15:12.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.500+0000 7ff1b4837740 1 -- 172.21.15.28:0/2735829534 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a77b27d2a0 msgr2=0x55a77b281b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:15:12.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.500+0000 7ff1b4837740 1 --2- 172.21.15.28:0/2735829534 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a77b27d2a0 0x55a77b281b00 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x55a77b29a920 tx=0x7ff19c005270 comp rx=0 tx=0).stop 2024-05-12T22:15:12.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.500+0000 7ff1b4837740 1 -- 172.21.15.28:0/2735829534 shutdown_connections 2024-05-12T22:15:12.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.500+0000 7ff1b4837740 1 --2- 172.21.15.28:0/2735829534 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55a77b2feca0 0x55a77b301090 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:12.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.500+0000 7ff1b4837740 1 --2- 172.21.15.28:0/2735829534 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a77b287970 0x55a77b282040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:12.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.500+0000 7ff1b4837740 1 --2- 172.21.15.28:0/2735829534 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a77b27d2a0 0x55a77b281b00 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:12.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.500+0000 7ff194c00640 0 client.26651 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:15:12.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.500+0000 7ff1b4837740 1 --2- 172.21.15.28:0/2735829534 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a77b27c980 0x55a77b2815c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:12.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.500+0000 7ff1b4837740 1 -- 172.21.15.28:0/2735829534 >> 172.21.15.28:0/2735829534 conn(0x55a77b278ca0 msgr2=0x55a77b277530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:15:12.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.500+0000 7ff1b4837740 1 -- 172.21.15.28:0/2735829534 shutdown_connections 2024-05-12T22:15:12.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.500+0000 7ff1b4837740 1 -- 172.21.15.28:0/2735829534 wait complete. 2024-05-12T22:15:12.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.500+0000 7ff1b4837740 20 client.26651 trim_cache size 0 max 16384 2024-05-12T22:15:12.512 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-05-12T22:15:12.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.501+0000 7ff1b1a00640 1 -- 172.21.15.28:0/1759946385 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7ff1900a2d80 msgr2=0x7ff1900a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:15:12.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.501+0000 7ff1b1a00640 1 --2- 172.21.15.28:0/1759946385 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7ff1900a2d80 0x7ff1900a5240 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x55a77b27df40 tx=0x7ff198009f20 comp rx=0 tx=0).stop 2024-05-12T22:15:12.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.501+0000 7ff1b1a00640 1 -- 172.21.15.28:0/1759946385 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1ac103a10 msgr2=0x7ff1ac19a390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:15:12.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.501+0000 7ff1b1a00640 1 --2- 172.21.15.28:0/1759946385 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1ac103a10 0x7ff1ac19a390 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7ff19800e9f0 tx=0x7ff19800edf0 comp rx=0 tx=0).stop 2024-05-12T22:15:12.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.501+0000 7ff1b1a00640 1 -- 172.21.15.28:0/1759946385 shutdown_connections 2024-05-12T22:15:12.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.501+0000 7ff1b1a00640 1 --2- 172.21.15.28:0/1759946385 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7ff1900a2d80 0x7ff1900a5240 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:12.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.501+0000 7ff1b1a00640 1 --2- 172.21.15.28:0/1759946385 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff1ac109a20 0x7ff1ac19ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:12.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.501+0000 7ff1b1a00640 1 --2- 172.21.15.28:0/1759946385 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1ac1043d0 0x7ff1ac19a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:12.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.501+0000 7ff1b1a00640 1 --2- 172.21.15.28:0/1759946385 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1ac103a10 0x7ff1ac19a390 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:12.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.501+0000 7ff1b1a00640 1 -- 172.21.15.28:0/1759946385 >> 172.21.15.28:0/1759946385 conn(0x7ff1ac0ff4a0 msgr2=0x7ff1ac0ffc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:15:12.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.501+0000 7ff1b1a00640 1 -- 172.21.15.28:0/1759946385 shutdown_connections 2024-05-12T22:15:12.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:12.501+0000 7ff1b1a00640 1 -- 172.21.15.28:0/1759946385 wait complete. 2024-05-12T22:15:13.520 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-12T22:15:13.520 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-12T22:15:13.521 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-05-12T22:15:13.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.753+0000 7f9f70a00640 1 -- 172.21.15.28:0/1955643200 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f6c074580 msgr2=0x7f9f6c10d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:15:13.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.753+0000 7f9f70a00640 1 --2- 172.21.15.28:0/1955643200 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f6c074580 0x7f9f6c10d9e0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f9f54009900 tx=0x7f9f5402a480 comp rx=0 tx=0).stop 2024-05-12T22:15:13.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.753+0000 7f9f70a00640 1 -- 172.21.15.28:0/1955643200 shutdown_connections 2024-05-12T22:15:13.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.753+0000 7f9f70a00640 1 --2- 172.21.15.28:0/1955643200 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9f6c10df20 0x7f9f6c110370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:13.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.753+0000 7f9f70a00640 1 --2- 172.21.15.28:0/1955643200 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f6c074580 0x7f9f6c10d9e0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:13.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.753+0000 7f9f70a00640 1 --2- 172.21.15.28:0/1955643200 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9f6c073c60 0x7f9f6c074040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:13.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.753+0000 7f9f70a00640 1 -- 172.21.15.28:0/1955643200 >> 172.21.15.28:0/1955643200 conn(0x7f9f6c0fd1d0 msgr2=0x7f9f6c0ff5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:15:13.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.754+0000 7f9f70a00640 1 -- 172.21.15.28:0/1955643200 shutdown_connections 2024-05-12T22:15:13.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.754+0000 7f9f70a00640 1 -- 172.21.15.28:0/1955643200 wait complete. 2024-05-12T22:15:13.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.754+0000 7f9f70a00640 1 Processor -- start 2024-05-12T22:15:13.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.755+0000 7f9f70a00640 1 -- start start 2024-05-12T22:15:13.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.755+0000 7f9f70a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9f6c073c60 0x7f9f6c19a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:13.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.755+0000 7f9f70a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9f6c074580 0x7f9f6c19a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:13.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.755+0000 7f9f70a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f6c10df20 0x7f9f6c19eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:13.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.755+0000 7f9f70a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9f6c113d10 con 0x7f9f6c10df20 2024-05-12T22:15:13.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.755+0000 7f9f70a00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f9f6c113b90 con 0x7f9f6c074580 2024-05-12T22:15:13.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.755+0000 7f9f70a00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f9f6c113e90 con 0x7f9f6c073c60 2024-05-12T22:15:13.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.755+0000 7f9f6b200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f6c10df20 0x7f9f6c19eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:13.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.755+0000 7f9f6a800640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9f6c073c60 0x7f9f6c19a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:13.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.755+0000 7f9f6b200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f6c10df20 0x7f9f6c19eac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34508/0 (socket says 172.21.15.28:34508) 2024-05-12T22:15:13.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.755+0000 7f9f6b200640 1 -- 172.21.15.28:0/1981842361 learned_addr learned my addr 172.21.15.28:0/1981842361 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:15:13.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.755+0000 7f9f63e00640 1 --2- 172.21.15.28:0/1981842361 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9f6c074580 0x7f9f6c19a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:13.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.756+0000 7f9f6b200640 1 -- 172.21.15.28:0/1981842361 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9f6c073c60 msgr2=0x7f9f6c19a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:15:13.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.756+0000 7f9f6b200640 1 --2- 172.21.15.28:0/1981842361 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9f6c073c60 0x7f9f6c19a250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:13.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.756+0000 7f9f6b200640 1 -- 172.21.15.28:0/1981842361 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9f6c074580 msgr2=0x7f9f6c19a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:15:13.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.756+0000 7f9f6b200640 1 --2- 172.21.15.28:0/1981842361 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9f6c074580 0x7f9f6c19a790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:13.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.756+0000 7f9f6b200640 1 -- 172.21.15.28:0/1981842361 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f6c19f1f0 con 0x7f9f6c10df20 2024-05-12T22:15:13.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.756+0000 7f9f6b200640 1 --2- 172.21.15.28:0/1981842361 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f6c10df20 0x7f9f6c19eac0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f9f6400eb10 tx=0x7f9f6400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:15:13.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.756+0000 7f9f63400640 1 -- 172.21.15.28:0/1981842361 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f64004b20 con 0x7f9f6c10df20 2024-05-12T22:15:13.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.757+0000 7f9f63400640 1 -- 172.21.15.28:0/1981842361 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9f64004cc0 con 0x7f9f6c10df20 2024-05-12T22:15:13.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.757+0000 7f9f63400640 1 -- 172.21.15.28:0/1981842361 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f64010760 con 0x7f9f6c10df20 2024-05-12T22:15:13.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.757+0000 7f9f70a00640 1 -- 172.21.15.28:0/1981842361 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f6c19f4e0 con 0x7f9f6c10df20 2024-05-12T22:15:13.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.757+0000 7f9f70a00640 1 -- 172.21.15.28:0/1981842361 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f6c1a6d70 con 0x7f9f6c10df20 2024-05-12T22:15:13.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.760+0000 7f9f74383740 1 Processor -- start 2024-05-12T22:15:13.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.760+0000 7f9f63400640 1 -- 172.21.15.28:0/1981842361 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7f9f64005020 con 0x7f9f6c10df20 2024-05-12T22:15:13.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.760+0000 7f9f74383740 1 -- start start 2024-05-12T22:15:13.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.761+0000 7f9f74383740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558353005500 0x5583530058e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:13.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.761+0000 7f9f74383740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558353005eb0 0x55835300ff50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:13.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.761+0000 7f9f74383740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558353010520 0x558353012970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:13.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.761+0000 7f9f74383740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558352f697c0 con 0x558353005500 2024-05-12T22:15:13.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.761+0000 7f9f74383740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x558352f11790 con 0x558353005eb0 2024-05-12T22:15:13.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.761+0000 7f9f74383740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x558352f65780 con 0x558353010520 2024-05-12T22:15:13.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.761+0000 7f9f63e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558353005500 0x5583530058e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:13.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.761+0000 7f9f63e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558353005500 0x5583530058e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34512/0 (socket says 172.21.15.28:34512) 2024-05-12T22:15:13.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.761+0000 7f9f63e00640 1 -- 172.21.15.28:0/415301382 learned_addr learned my addr 172.21.15.28:0/415301382 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:15:13.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.761+0000 7f9f6a800640 1 --2- 172.21.15.28:0/415301382 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558353005eb0 0x55835300ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:13.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.761+0000 7f9f63e00640 1 --2- 172.21.15.28:0/415301382 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558353010520 0x558353012970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:13.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.761+0000 7f9f63e00640 1 -- 172.21.15.28:0/415301382 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558353010520 msgr2=0x558353012970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:15:13.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.761+0000 7f9f63e00640 1 --2- 172.21.15.28:0/415301382 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558353010520 0x558353012970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:13.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.761+0000 7f9f63e00640 1 -- 172.21.15.28:0/415301382 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558353005eb0 msgr2=0x55835300ff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:15:13.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.761+0000 7f9f63e00640 1 --2- 172.21.15.28:0/415301382 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558353005eb0 0x55835300ff50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:13.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.761+0000 7f9f63e00640 1 -- 172.21.15.28:0/415301382 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558352f316a0 con 0x558353005500 2024-05-12T22:15:13.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.762+0000 7f9f63e00640 1 --2- 172.21.15.28:0/415301382 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558353005500 0x5583530058e0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f9f54009900 tx=0x7f9f54005bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1c5667de685e48e server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:15:13.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.762+0000 7f9f63400640 1 --2- 172.21.15.28:0/1981842361 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f9f500a2d80 0x7f9f500a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:13.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.762+0000 7f9f5b400640 1 -- 172.21.15.28:0/415301382 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f54044020 con 0x558353005500 2024-05-12T22:15:13.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.762+0000 7f9f63400640 1 -- 172.21.15.28:0/1981842361 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f9f640cf8b0 con 0x7f9f6c10df20 2024-05-12T22:15:13.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.762+0000 7f9f5b400640 1 -- 172.21.15.28:0/415301382 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9f54005520 con 0x558353005500 2024-05-12T22:15:13.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.762+0000 7f9f5b400640 1 -- 172.21.15.28:0/415301382 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f54035040 con 0x558353005500 2024-05-12T22:15:13.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.762+0000 7f9f6a800640 1 --2- 172.21.15.28:0/1981842361 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f9f500a2d80 0x7f9f500a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:13.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.763+0000 7f9f74383740 1 -- 172.21.15.28:0/415301382 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558353005500 msgr2=0x5583530058e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:15:13.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.763+0000 7f9f74383740 1 --2- 172.21.15.28:0/415301382 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558353005500 0x5583530058e0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f9f54009900 tx=0x7f9f54005bc0 comp rx=0 tx=0).stop 2024-05-12T22:15:13.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.763+0000 7f9f6a800640 1 --2- 172.21.15.28:0/1981842361 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f9f500a2d80 0x7f9f500a5240 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x558353002830 tx=0x7f9f5c005f90 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:15:13.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.763+0000 7f9f74383740 1 -- 172.21.15.28:0/415301382 shutdown_connections 2024-05-12T22:15:13.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.763+0000 7f9f74383740 1 --2- 172.21.15.28:0/415301382 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558353010520 0x558353012970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:13.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.763+0000 7f9f74383740 1 --2- 172.21.15.28:0/415301382 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558353005eb0 0x55835300ff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:13.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.763+0000 7f9f74383740 1 --2- 172.21.15.28:0/415301382 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558353005500 0x5583530058e0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:13.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.763+0000 7f9f74383740 1 -- 172.21.15.28:0/415301382 >> 172.21.15.28:0/415301382 conn(0x5583530015b0 msgr2=0x558353001990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:15:13.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.763+0000 7f9f74383740 1 -- 172.21.15.28:0/415301382 shutdown_connections 2024-05-12T22:15:13.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.763+0000 7f9f74383740 1 -- 172.21.15.28:0/415301382 wait complete. 2024-05-12T22:15:13.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.764+0000 7f9f74383740 1 -- start start 2024-05-12T22:15:13.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.764+0000 7f9f74383740 1 Processor -- start 2024-05-12T22:15:13.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.764+0000 7f9f74383740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558353005500 0x55835300d470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:13.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.764+0000 7f9f74383740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558353005eb0 0x55835300d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:13.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.765+0000 7f9f74383740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558353010520 0x55835300def0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:13.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.765+0000 7f9f74383740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5583530134d0 con 0x558353005500 2024-05-12T22:15:13.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.765+0000 7f9f74383740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x558352f24a90 con 0x558353005eb0 2024-05-12T22:15:13.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.765+0000 7f9f74383740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x558353013650 con 0x558353010520 2024-05-12T22:15:13.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.765+0000 7f9f74383740 10 client.0 _ll_register_callbacks cb 0x558352f5faf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:15:13.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.765+0000 7f9f6a800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558353005500 0x55835300d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:13.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.765+0000 7f9f6a800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558353005500 0x55835300d470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34518/0 (socket says 172.21.15.28:34518) 2024-05-12T22:15:13.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.765+0000 7f9f6a800640 1 -- 172.21.15.28:0/92217090 learned_addr learned my addr 172.21.15.28:0/92217090 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:15:13.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.765+0000 7f9f63e00640 1 --2- 172.21.15.28:0/92217090 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558353005eb0 0x55835300d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:13.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.765+0000 7f9f6b200640 1 --2- 172.21.15.28:0/92217090 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558353010520 0x55835300def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:13.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.765+0000 7f9f6a800640 1 -- 172.21.15.28:0/92217090 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558353010520 msgr2=0x55835300def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:15:13.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.765+0000 7f9f6a800640 1 --2- 172.21.15.28:0/92217090 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558353010520 0x55835300def0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:13.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.765+0000 7f9f6a800640 1 -- 172.21.15.28:0/92217090 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558353005eb0 msgr2=0x55835300d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:15:13.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.765+0000 7f9f6a800640 1 --2- 172.21.15.28:0/92217090 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558353005eb0 0x55835300d9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:13.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.765+0000 7f9f6a800640 1 -- 172.21.15.28:0/92217090 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558352d457e0 con 0x558353005500 2024-05-12T22:15:13.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.765+0000 7f9f63e00640 1 --2- 172.21.15.28:0/92217090 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558353005eb0 0x55835300d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:15:13.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.766+0000 7f9f6a800640 1 --2- 172.21.15.28:0/92217090 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558353005500 0x55835300d470 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f9f5c004600 tx=0x7f9f5c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7762cc4a111c79b server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:15:13.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.766+0000 7f9f4fe00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-12T22:15:13.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.766+0000 7f9f4fe00640 1 -- 172.21.15.28:0/92217090 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f5c009e60 con 0x558353005500 2024-05-12T22:15:13.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.766+0000 7f9f4fe00640 1 -- 172.21.15.28:0/92217090 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f9f5c005020 con 0x558353005500 2024-05-12T22:15:13.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.766+0000 7f9f74383740 1 -- 172.21.15.28:0/92217090 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9f54005520 con 0x558353005500 2024-05-12T22:15:13.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.766+0000 7f9f4fe00640 1 -- 172.21.15.28:0/92217090 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f9f5c005220 con 0x558353005500 2024-05-12T22:15:13.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.766+0000 7f9f74383740 10 client.16536 fetch_fsmap learned FSMap version 26 2024-05-12T22:15:13.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.766+0000 7f9f74383740 1 -- 172.21.15.28:0/92217090 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x558352da5f70 con 0x558353005500 2024-05-12T22:15:13.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.767+0000 7f9f4fe00640 1 -- 172.21.15.28:0/92217090 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9f54005520 con 0x558353005500 2024-05-12T22:15:13.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.767+0000 7f9f4fe00640 1 -- 172.21.15.28:0/92217090 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f9f5c027070 con 0x558353005500 2024-05-12T22:15:13.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.767+0000 7f9f74383740 10 client.16536 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:15:13.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.767+0000 7f9f74383740 10 client.16536 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:15:13.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.767+0000 7f9f74383740 20 client.16536 populate_metadata read hostname 'smithi028' 2024-05-12T22:15:13.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.767+0000 7f9f74383740 1 --2- 172.21.15.28:0/92217090 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x558353020830 0x558353022ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:13.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.767+0000 7f9f74383740 4 client.16536 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-12T22:15:13.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.767+0000 7f9f74383740 1 -- 172.21.15.28:0/92217090 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558352f7d3a0 con 0x558353020830 2024-05-12T22:15:13.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.767+0000 7f9f63e00640 1 --2- 172.21.15.28:0/92217090 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x558353020830 0x558353022ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:13.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.768+0000 7f9f63e00640 1 --2- 172.21.15.28:0/92217090 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x558353020830 0x558353022ca0 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f98932a0569495fe server_cookie=9a8b251ce038fb73 in_seq=0 out_seq=0 2024-05-12T22:15:13.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.768+0000 7f9f4fe00640 10 client.16536 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:15:13.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.769+0000 7f9f4fe00640 1 -- 172.21.15.28:0/92217090 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558352f7d3a0 con 0x558353020830 2024-05-12T22:15:13.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.769+0000 7f9f4fe00640 10 client.16536 handle_command_reply: tid=0 2024-05-12T22:15:13.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.769+0000 7f9f74383740 1 client.16536 shutdown 2024-05-12T22:15:13.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.769+0000 7f9f74383740 1 -- 172.21.15.28:0/92217090 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558353005500 msgr2=0x55835300d470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:15:13.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.769+0000 7f9f74383740 1 --2- 172.21.15.28:0/92217090 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558353005500 0x55835300d470 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f9f5c004600 tx=0x7f9f5c004730 comp rx=0 tx=0).stop 2024-05-12T22:15:13.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.769+0000 7f9f74383740 1 -- 172.21.15.28:0/92217090 shutdown_connections 2024-05-12T22:15:13.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.769+0000 7f9f74383740 1 --2- 172.21.15.28:0/92217090 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x558353020830 0x558353022ca0 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:13.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.769+0000 7f9f74383740 1 --2- 172.21.15.28:0/92217090 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558353010520 0x55835300def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:13.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.769+0000 7f9f74383740 1 --2- 172.21.15.28:0/92217090 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558353005eb0 0x55835300d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:13.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.769+0000 7f9f74383740 1 --2- 172.21.15.28:0/92217090 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558353005500 0x55835300d470 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:13.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.769+0000 7f9f4fe00640 0 client.16536 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:15:13.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.769+0000 7f9f74383740 1 -- 172.21.15.28:0/92217090 >> 172.21.15.28:0/92217090 conn(0x5583530015b0 msgr2=0x558353007630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:15:13.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.769+0000 7f9f74383740 1 -- 172.21.15.28:0/92217090 shutdown_connections 2024-05-12T22:15:13.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.769+0000 7f9f74383740 1 -- 172.21.15.28:0/92217090 wait complete. 2024-05-12T22:15:13.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.769+0000 7f9f74383740 20 client.16536 trim_cache size 0 max 16384 2024-05-12T22:15:13.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.789+0000 7f9f74383740 1 Processor -- start 2024-05-12T22:15:13.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.789+0000 7f9f74383740 1 -- start start 2024-05-12T22:15:13.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.789+0000 7f9f74383740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558353005eb0 0x558353007d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:13.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.789+0000 7f9f74383740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558353010520 0x55835301c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:13.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.789+0000 7f9f74383740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55835300c3b0 0x558353028570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:13.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.789+0000 7f9f74383740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9f6c1b1d00 con 0x558353010520 2024-05-12T22:15:13.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.789+0000 7f9f74383740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f9f6c1af5d0 con 0x558353005eb0 2024-05-12T22:15:13.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.789+0000 7f9f74383740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f9f6c1ad420 con 0x55835300c3b0 2024-05-12T22:15:13.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.790+0000 7f9f63e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558353010520 0x55835301c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:13.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.790+0000 7f9f6a800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558353005eb0 0x558353007d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:13.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.790+0000 7f9f6a800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558353005eb0 0x558353007d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:55332/0 (socket says 172.21.15.28:55332) 2024-05-12T22:15:13.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.790+0000 7f9f63e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558353010520 0x55835301c3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34520/0 (socket says 172.21.15.28:34520) 2024-05-12T22:15:13.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.790+0000 7f9f6a800640 1 -- 172.21.15.28:0/3755352204 learned_addr learned my addr 172.21.15.28:0/3755352204 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:15:13.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.790+0000 7f9f6b200640 1 --2- 172.21.15.28:0/3755352204 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55835300c3b0 0x558353028570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:13.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.790+0000 7f9f63e00640 1 -- 172.21.15.28:0/3755352204 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55835300c3b0 msgr2=0x558353028570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:15:13.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.790+0000 7f9f63e00640 1 --2- 172.21.15.28:0/3755352204 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55835300c3b0 0x558353028570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:13.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.790+0000 7f9f63e00640 1 -- 172.21.15.28:0/3755352204 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558353005eb0 msgr2=0x558353007d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:15:13.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.790+0000 7f9f63e00640 1 --2- 172.21.15.28:0/3755352204 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558353005eb0 0x558353007d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:13.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.790+0000 7f9f63e00640 1 -- 172.21.15.28:0/3755352204 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558352c3d170 con 0x558353010520 2024-05-12T22:15:13.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.791+0000 7f9f63e00640 1 --2- 172.21.15.28:0/3755352204 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558353010520 0x55835301c3f0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f9f5402aa70 tx=0x7f9f54005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56d84db3ca4af1b6 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:15:13.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.791+0000 7f9f5aa00640 1 -- 172.21.15.28:0/3755352204 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f5404f070 con 0x558353010520 2024-05-12T22:15:13.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.791+0000 7f9f5aa00640 1 -- 172.21.15.28:0/3755352204 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9f54005030 con 0x558353010520 2024-05-12T22:15:13.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.791+0000 7f9f5aa00640 1 -- 172.21.15.28:0/3755352204 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f54035070 con 0x558353010520 2024-05-12T22:15:13.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.791+0000 7f9f74383740 1 -- 172.21.15.28:0/3755352204 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558353010520 msgr2=0x55835301c3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:15:13.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.791+0000 7f9f74383740 1 --2- 172.21.15.28:0/3755352204 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558353010520 0x55835301c3f0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f9f5402aa70 tx=0x7f9f54005fd0 comp rx=0 tx=0).stop 2024-05-12T22:15:13.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.792+0000 7f9f74383740 1 -- 172.21.15.28:0/3755352204 shutdown_connections 2024-05-12T22:15:13.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.792+0000 7f9f74383740 1 --2- 172.21.15.28:0/3755352204 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55835300c3b0 0x558353028570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:13.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.792+0000 7f9f74383740 1 --2- 172.21.15.28:0/3755352204 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558353010520 0x55835301c3f0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:13.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.792+0000 7f9f74383740 1 --2- 172.21.15.28:0/3755352204 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558353005eb0 0x558353007d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:13.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.792+0000 7f9f74383740 1 -- 172.21.15.28:0/3755352204 >> 172.21.15.28:0/3755352204 conn(0x5583530015b0 msgr2=0x558353000730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:15:13.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.792+0000 7f9f74383740 1 -- 172.21.15.28:0/3755352204 shutdown_connections 2024-05-12T22:15:13.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.792+0000 7f9f74383740 1 -- 172.21.15.28:0/3755352204 wait complete. 2024-05-12T22:15:13.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.792+0000 7f9f74383740 1 -- start start 2024-05-12T22:15:13.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.792+0000 7f9f74383740 1 Processor -- start 2024-05-12T22:15:13.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.793+0000 7f9f74383740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558353005eb0 0x558353018e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:13.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.793+0000 7f9f74383740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55835300c3b0 0x5583530141f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:13.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.793+0000 7f9f74383740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558353010520 0x558353014930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:13.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.793+0000 7f9f74383740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9f6c1ada90 con 0x55835300c3b0 2024-05-12T22:15:13.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.793+0000 7f9f74383740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f9f6c1aef00 con 0x558353010520 2024-05-12T22:15:13.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.793+0000 7f9f74383740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f9f6c1ae160 con 0x558353005eb0 2024-05-12T22:15:13.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.793+0000 7f9f74383740 10 client.0 _ll_register_callbacks cb 0x7f9f5c009ab0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:15:13.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.793+0000 7f9f63e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55835300c3b0 0x5583530141f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:13.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.793+0000 7f9f63e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55835300c3b0 0x5583530141f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34534/0 (socket says 172.21.15.28:34534) 2024-05-12T22:15:13.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.793+0000 7f9f63e00640 1 -- 172.21.15.28:0/30935736 learned_addr learned my addr 172.21.15.28:0/30935736 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:15:13.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.793+0000 7f9f6a800640 1 --2- 172.21.15.28:0/30935736 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558353005eb0 0x558353018e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:13.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.793+0000 7f9f6b200640 1 --2- 172.21.15.28:0/30935736 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558353010520 0x558353014930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:13.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.794+0000 7f9f63e00640 1 -- 172.21.15.28:0/30935736 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558353005eb0 msgr2=0x558353018e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:15:13.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.794+0000 7f9f63e00640 1 --2- 172.21.15.28:0/30935736 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558353005eb0 0x558353018e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:13.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.794+0000 7f9f63e00640 1 -- 172.21.15.28:0/30935736 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558353010520 msgr2=0x558353014930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:15:13.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.794+0000 7f9f63e00640 1 --2- 172.21.15.28:0/30935736 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558353010520 0x558353014930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:13.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.794+0000 7f9f63e00640 1 -- 172.21.15.28:0/30935736 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55835308e5f0 con 0x55835300c3b0 2024-05-12T22:15:13.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.794+0000 7f9f6a800640 1 --2- 172.21.15.28:0/30935736 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558353005eb0 0x558353018e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:15:13.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.794+0000 7f9f63e00640 1 --2- 172.21.15.28:0/30935736 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55835300c3b0 0x5583530141f0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f9f5402aa40 tx=0x7f9f540047a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9871a9bebaba3620 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:15:13.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.794+0000 7f9f4fe00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-12T22:15:13.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.794+0000 7f9f4fe00640 1 -- 172.21.15.28:0/30935736 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f54058060 con 0x55835300c3b0 2024-05-12T22:15:13.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.794+0000 7f9f4fe00640 1 -- 172.21.15.28:0/30935736 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f9f540053e0 con 0x55835300c3b0 2024-05-12T22:15:13.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.794+0000 7f9f74383740 1 -- 172.21.15.28:0/30935736 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9f54005030 con 0x55835300c3b0 2024-05-12T22:15:13.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.795+0000 7f9f4fe00640 1 -- 172.21.15.28:0/30935736 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f9f54035070 con 0x55835300c3b0 2024-05-12T22:15:13.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.795+0000 7f9f74383740 10 client.16542 fetch_fsmap learned FSMap version 26 2024-05-12T22:15:13.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.795+0000 7f9f74383740 1 -- 172.21.15.28:0/30935736 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55835308f7f0 con 0x55835300c3b0 2024-05-12T22:15:13.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.795+0000 7f9f4fe00640 1 -- 172.21.15.28:0/30935736 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9f54005030 con 0x55835300c3b0 2024-05-12T22:15:13.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.795+0000 7f9f4fe00640 1 -- 172.21.15.28:0/30935736 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f9f54052070 con 0x55835300c3b0 2024-05-12T22:15:13.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.795+0000 7f9f74383740 10 client.16542 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:15:13.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.795+0000 7f9f74383740 10 client.16542 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:15:13.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.795+0000 7f9f74383740 20 client.16542 populate_metadata read hostname 'smithi028' 2024-05-12T22:15:13.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.795+0000 7f9f74383740 1 --2- 172.21.15.28:0/30935736 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55835308fac0 0x558353091eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:13.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.795+0000 7f9f74383740 4 client.16542 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-12T22:15:13.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.795+0000 7f9f74383740 1 -- 172.21.15.28:0/30935736 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x558352f76ef0 con 0x55835308fac0 2024-05-12T22:15:13.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.795+0000 7f9f6a800640 1 --2- 172.21.15.28:0/30935736 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55835308fac0 0x558353091eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:13.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.796+0000 7f9f6a800640 1 --2- 172.21.15.28:0/30935736 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55835308fac0 0x558353091eb0 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=583675baddfe1a23 server_cookie=ebe5edfe5943cb2d in_seq=0 out_seq=0 2024-05-12T22:15:13.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.796+0000 7f9f4fe00640 10 client.16542 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:15:13.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.797+0000 7f9f4fe00640 1 -- 172.21.15.28:0/30935736 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x558352f76ef0 con 0x55835308fac0 2024-05-12T22:15:13.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.797+0000 7f9f4fe00640 10 client.16542 handle_command_reply: tid=0 2024-05-12T22:15:13.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.797+0000 7f9f74383740 1 client.16542 shutdown 2024-05-12T22:15:13.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.797+0000 7f9f74383740 1 -- 172.21.15.28:0/30935736 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55835300c3b0 msgr2=0x5583530141f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:15:13.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.797+0000 7f9f74383740 1 --2- 172.21.15.28:0/30935736 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55835300c3b0 0x5583530141f0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f9f5402aa40 tx=0x7f9f540047a0 comp rx=0 tx=0).stop 2024-05-12T22:15:13.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.798+0000 7f9f74383740 1 -- 172.21.15.28:0/30935736 shutdown_connections 2024-05-12T22:15:13.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.798+0000 7f9f74383740 1 --2- 172.21.15.28:0/30935736 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55835308fac0 0x558353091eb0 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:13.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.798+0000 7f9f74383740 1 --2- 172.21.15.28:0/30935736 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558353010520 0x558353014930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:13.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.798+0000 7f9f74383740 1 --2- 172.21.15.28:0/30935736 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55835300c3b0 0x5583530141f0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:13.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.798+0000 7f9f74383740 1 --2- 172.21.15.28:0/30935736 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558353005eb0 0x558353018e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:13.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.798+0000 7f9f4fe00640 0 client.16542 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:15:13.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.798+0000 7f9f74383740 1 -- 172.21.15.28:0/30935736 >> 172.21.15.28:0/30935736 conn(0x5583530015b0 msgr2=0x55835300d290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:15:13.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.798+0000 7f9f74383740 1 -- 172.21.15.28:0/30935736 shutdown_connections 2024-05-12T22:15:13.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.798+0000 7f9f74383740 1 -- 172.21.15.28:0/30935736 wait complete. 2024-05-12T22:15:13.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.798+0000 7f9f74383740 20 client.16542 trim_cache size 0 max 16384 2024-05-12T22:15:13.805 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-05-12T22:15:13.805 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-05-12T22:15:13.806 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "69d0a81f-3ff6-4005-a136-63e58f54ec07", 2024-05-12T22:15:13.806 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-05-12T22:15:13.806 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-05-12T22:15:13.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.798+0000 7f9f70a00640 1 -- 172.21.15.28:0/1981842361 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f9f500a2d80 msgr2=0x7f9f500a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:15:13.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.798+0000 7f9f70a00640 1 --2- 172.21.15.28:0/1981842361 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f9f500a2d80 0x7f9f500a5240 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x558353002830 tx=0x7f9f5c005f90 comp rx=0 tx=0).stop 2024-05-12T22:15:13.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.798+0000 7f9f70a00640 1 -- 172.21.15.28:0/1981842361 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f6c10df20 msgr2=0x7f9f6c19eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:15:13.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.798+0000 7f9f70a00640 1 --2- 172.21.15.28:0/1981842361 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f6c10df20 0x7f9f6c19eac0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f9f6400eb10 tx=0x7f9f6400c2d0 comp rx=0 tx=0).stop 2024-05-12T22:15:13.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.799+0000 7f9f70a00640 1 -- 172.21.15.28:0/1981842361 shutdown_connections 2024-05-12T22:15:13.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.799+0000 7f9f70a00640 1 --2- 172.21.15.28:0/1981842361 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f9f500a2d80 0x7f9f500a5240 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:13.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.799+0000 7f9f70a00640 1 --2- 172.21.15.28:0/1981842361 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f6c10df20 0x7f9f6c19eac0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:13.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.799+0000 7f9f70a00640 1 --2- 172.21.15.28:0/1981842361 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9f6c074580 0x7f9f6c19a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:13.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.799+0000 7f9f70a00640 1 --2- 172.21.15.28:0/1981842361 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9f6c073c60 0x7f9f6c19a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:13.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.799+0000 7f9f70a00640 1 -- 172.21.15.28:0/1981842361 >> 172.21.15.28:0/1981842361 conn(0x7f9f6c0fd1d0 msgr2=0x7f9f6c0fea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:15:13.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.799+0000 7f9f70a00640 1 -- 172.21.15.28:0/1981842361 shutdown_connections 2024-05-12T22:15:13.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:13.799+0000 7f9f70a00640 1 -- 172.21.15.28:0/1981842361 wait complete. 2024-05-12T22:15:13.820 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-12T22:15:13.820 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-12T22:15:14.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.060+0000 7f2f7ac00640 1 -- 172.21.15.28:0/191427800 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2f74073be0 msgr2=0x7f2f74073fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:15:14.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.060+0000 7f2f7ac00640 1 --2- 172.21.15.28:0/191427800 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2f74073be0 0x7f2f74073fc0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f2f68009ca0 tx=0x7f2f6802a950 comp rx=0 tx=0).stop 2024-05-12T22:15:14.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.060+0000 7f2f7ac00640 1 -- 172.21.15.28:0/191427800 shutdown_connections 2024-05-12T22:15:14.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.060+0000 7f2f7ac00640 1 --2- 172.21.15.28:0/191427800 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2f7410df90 0x7f2f74110380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:14.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.060+0000 7f2f7ac00640 1 --2- 172.21.15.28:0/191427800 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f74074500 0x7f2f7410da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:14.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.060+0000 7f2f7ac00640 1 --2- 172.21.15.28:0/191427800 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2f74073be0 0x7f2f74073fc0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:14.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.060+0000 7f2f7ac00640 1 -- 172.21.15.28:0/191427800 >> 172.21.15.28:0/191427800 conn(0x7f2f740fd200 msgr2=0x7f2f740ff620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:15:14.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.060+0000 7f2f7ac00640 1 -- 172.21.15.28:0/191427800 shutdown_connections 2024-05-12T22:15:14.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.060+0000 7f2f7ac00640 1 -- 172.21.15.28:0/191427800 wait complete. 2024-05-12T22:15:14.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.062+0000 7f2f7ac00640 1 Processor -- start 2024-05-12T22:15:14.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.062+0000 7f2f7ac00640 1 -- start start 2024-05-12T22:15:14.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.062+0000 7f2f7ac00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2f74073be0 0x7f2f7419a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:14.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.062+0000 7f2f7ac00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f74074500 0x7f2f7419a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:14.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.062+0000 7f2f7ac00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2f7410df90 0x7f2f7419ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:14.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.062+0000 7f2f7ac00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2f74113cb0 con 0x7f2f74074500 2024-05-12T22:15:14.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.062+0000 7f2f7ac00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f2f74113b30 con 0x7f2f7410df90 2024-05-12T22:15:14.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.062+0000 7f2f7ac00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f2f74113e30 con 0x7f2f74073be0 2024-05-12T22:15:14.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.063+0000 7f2f79800640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2f74073be0 0x7f2f7419a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:14.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.063+0000 7f2f79800640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2f74073be0 0x7f2f7419a1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:36560/0 (socket says 172.21.15.28:36560) 2024-05-12T22:15:14.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.063+0000 7f2f79800640 1 -- 172.21.15.28:0/805272255 learned_addr learned my addr 172.21.15.28:0/805272255 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:15:14.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.063+0000 7f2f7a200640 1 --2- 172.21.15.28:0/805272255 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2f7410df90 0x7f2f7419ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:14.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.063+0000 7f2f78e00640 1 --2- 172.21.15.28:0/805272255 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f74074500 0x7f2f7419a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:14.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.063+0000 7f2f79800640 1 -- 172.21.15.28:0/805272255 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2f7410df90 msgr2=0x7f2f7419ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:15:14.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.063+0000 7f2f79800640 1 --2- 172.21.15.28:0/805272255 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2f7410df90 0x7f2f7419ea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:14.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.063+0000 7f2f79800640 1 -- 172.21.15.28:0/805272255 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f74074500 msgr2=0x7f2f7419a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:15:14.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.063+0000 7f2f79800640 1 --2- 172.21.15.28:0/805272255 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f74074500 0x7f2f7419a710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:14.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.063+0000 7f2f79800640 1 -- 172.21.15.28:0/805272255 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f7419f1c0 con 0x7f2f74073be0 2024-05-12T22:15:14.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.064+0000 7f2f79800640 1 --2- 172.21.15.28:0/805272255 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2f74073be0 0x7f2f7419a1d0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f2f68005cc0 tx=0x7f2f6802f420 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:15:14.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.064+0000 7f2f72000640 1 -- 172.21.15.28:0/805272255 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f68034040 con 0x7f2f74073be0 2024-05-12T22:15:14.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.064+0000 7f2f7ac00640 1 -- 172.21.15.28:0/805272255 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f7419f450 con 0x7f2f74073be0 2024-05-12T22:15:14.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.064+0000 7f2f72000640 1 -- 172.21.15.28:0/805272255 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2f68030920 con 0x7f2f74073be0 2024-05-12T22:15:14.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.064+0000 7f2f72000640 1 -- 172.21.15.28:0/805272255 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f68004620 con 0x7f2f74073be0 2024-05-12T22:15:14.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.064+0000 7f2f7ac00640 1 -- 172.21.15.28:0/805272255 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f741a6d40 con 0x7f2f74073be0 2024-05-12T22:15:14.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.068+0000 7f2f7f005740 1 Processor -- start 2024-05-12T22:15:14.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.068+0000 7f2f72000640 1 -- 172.21.15.28:0/805272255 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7f2f68037020 con 0x7f2f74073be0 2024-05-12T22:15:14.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.068+0000 7f2f7f005740 1 -- start start 2024-05-12T22:15:14.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.069+0000 7f2f7f005740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563023160dd0 0x5630231611b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:14.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.069+0000 7f2f7f005740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630231616f0 0x56302316b880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:14.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.069+0000 7f2f7f005740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56302316bdc0 0x56302316e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:14.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.069+0000 7f2f7f005740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5630231292a0 con 0x5630231616f0 2024-05-12T22:15:14.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.069+0000 7f2f7f005740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x5630230c5b90 con 0x563023160dd0 2024-05-12T22:15:14.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.069+0000 7f2f7f005740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x56302306d5b0 con 0x56302316bdc0 2024-05-12T22:15:14.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.069+0000 7f2f7a200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563023160dd0 0x5630231611b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:14.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.069+0000 7f2f79800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630231616f0 0x56302316b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:14.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.069+0000 7f2f7a200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563023160dd0 0x5630231611b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:55378/0 (socket says 172.21.15.28:55378) 2024-05-12T22:15:14.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.069+0000 7f2f7a200640 1 -- 172.21.15.28:0/2075895481 learned_addr learned my addr 172.21.15.28:0/2075895481 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:15:14.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.069+0000 7f2f78e00640 1 --2- 172.21.15.28:0/2075895481 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56302316bdc0 0x56302316e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:14.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.070+0000 7f2f7a200640 1 -- 172.21.15.28:0/2075895481 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56302316bdc0 msgr2=0x56302316e1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:15:14.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.070+0000 7f2f7a200640 1 --2- 172.21.15.28:0/2075895481 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56302316bdc0 0x56302316e1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:14.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.070+0000 7f2f7a200640 1 -- 172.21.15.28:0/2075895481 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630231616f0 msgr2=0x56302316b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:15:14.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.070+0000 7f2f7a200640 1 --2- 172.21.15.28:0/2075895481 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630231616f0 0x56302316b880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:14.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.070+0000 7f2f7a200640 1 -- 172.21.15.28:0/2075895481 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563022ea37d0 con 0x563023160dd0 2024-05-12T22:15:14.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.070+0000 7f2f7a200640 1 --2- 172.21.15.28:0/2075895481 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563023160dd0 0x5630231611b0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f2f6c009c10 tx=0x7f2f6c011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b21dc5a538c1963c server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:15:14.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.070+0000 7f2f72000640 1 --2- 172.21.15.28:0/805272255 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f2f5c0a2d00 0x7f2f5c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:14.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.070+0000 7f2f72000640 1 -- 172.21.15.28:0/805272255 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f2f680f0130 con 0x7f2f74073be0 2024-05-12T22:15:14.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.070+0000 7f2f66a00640 1 -- 172.21.15.28:0/2075895481 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f6c004230 con 0x563023160dd0 2024-05-12T22:15:14.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.071+0000 7f2f66a00640 1 -- 172.21.15.28:0/2075895481 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2f6c0043d0 con 0x563023160dd0 2024-05-12T22:15:14.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.071+0000 7f2f66a00640 1 -- 172.21.15.28:0/2075895481 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f6c0197a0 con 0x563023160dd0 2024-05-12T22:15:14.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.071+0000 7f2f79800640 1 --2- 172.21.15.28:0/805272255 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f2f5c0a2d00 0x7f2f5c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:14.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.072+0000 7f2f7f005740 1 -- 172.21.15.28:0/2075895481 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563023160dd0 msgr2=0x5630231611b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:15:14.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.072+0000 7f2f7f005740 1 --2- 172.21.15.28:0/2075895481 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563023160dd0 0x5630231611b0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f2f6c009c10 tx=0x7f2f6c011810 comp rx=0 tx=0).stop 2024-05-12T22:15:14.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.072+0000 7f2f79800640 1 --2- 172.21.15.28:0/805272255 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f2f5c0a2d00 0x7f2f5c0a51c0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f2f680eda70 tx=0x7f2f680f3c80 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:15:14.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.072+0000 7f2f7f005740 1 -- 172.21.15.28:0/2075895481 shutdown_connections 2024-05-12T22:15:14.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.072+0000 7f2f7f005740 1 --2- 172.21.15.28:0/2075895481 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56302316bdc0 0x56302316e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:14.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.072+0000 7f2f7f005740 1 --2- 172.21.15.28:0/2075895481 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630231616f0 0x56302316b880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:14.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.072+0000 7f2f7f005740 1 --2- 172.21.15.28:0/2075895481 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563023160dd0 0x5630231611b0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:14.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.072+0000 7f2f7f005740 1 -- 172.21.15.28:0/2075895481 >> 172.21.15.28:0/2075895481 conn(0x56302315d0f0 msgr2=0x56302315d4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:15:14.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.072+0000 7f2f7f005740 1 -- 172.21.15.28:0/2075895481 shutdown_connections 2024-05-12T22:15:14.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.072+0000 7f2f7f005740 1 -- 172.21.15.28:0/2075895481 wait complete. 2024-05-12T22:15:14.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.073+0000 7f2f7f005740 1 -- start start 2024-05-12T22:15:14.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.073+0000 7f2f7f005740 1 Processor -- start 2024-05-12T22:15:14.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.073+0000 7f2f7f005740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563023160dd0 0x56302316b760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:14.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.073+0000 7f2f7f005740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5630231616f0 0x563023168a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:14.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.073+0000 7f2f7f005740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56302316bdc0 0x563023168ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:14.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.073+0000 7f2f7f005740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5630230809f0 con 0x56302316bdc0 2024-05-12T22:15:14.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.073+0000 7f2f7f005740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x5630230c1b50 con 0x5630231616f0 2024-05-12T22:15:14.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.073+0000 7f2f7f005740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x56302316ebb0 con 0x563023160dd0 2024-05-12T22:15:14.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.074+0000 7f2f7f005740 10 client.0 _ll_register_callbacks cb 0x563022dd8be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:15:14.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.074+0000 7f2f78e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5630231616f0 0x563023168a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:14.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.074+0000 7f2f78e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5630231616f0 0x563023168a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:55386/0 (socket says 172.21.15.28:55386) 2024-05-12T22:15:14.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.074+0000 7f2f78e00640 1 -- 172.21.15.28:0/923715713 learned_addr learned my addr 172.21.15.28:0/923715713 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:15:14.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.074+0000 7f2f7a200640 1 --2- 172.21.15.28:0/923715713 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56302316bdc0 0x563023168ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:14.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.074+0000 7f2f7a200640 1 --2- 172.21.15.28:0/923715713 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563023160dd0 0x56302316b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:14.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.074+0000 7f2f78e00640 1 -- 172.21.15.28:0/923715713 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563023160dd0 msgr2=0x56302316b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:15:14.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.074+0000 7f2f78e00640 1 --2- 172.21.15.28:0/923715713 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563023160dd0 0x56302316b760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:14.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.074+0000 7f2f78e00640 1 -- 172.21.15.28:0/923715713 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56302316bdc0 msgr2=0x563023168ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:15:14.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.074+0000 7f2f78e00640 1 --2- 172.21.15.28:0/923715713 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56302316bdc0 0x563023168ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:14.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.074+0000 7f2f78e00640 1 -- 172.21.15.28:0/923715713 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563022dd93f0 con 0x5630231616f0 2024-05-12T22:15:14.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.074+0000 7f2f7a200640 1 --2- 172.21.15.28:0/923715713 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563023160dd0 0x56302316b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-12T22:15:14.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.075+0000 7f2f78e00640 1 --2- 172.21.15.28:0/923715713 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5630231616f0 0x563023168a50 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f2f60012700 tx=0x7f2f60012bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=494a061190820755 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:15:14.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.075+0000 7f2f5b400640 10 client.0 ms_handle_connect on v2:172.21.15.86:3300/0 2024-05-12T22:15:14.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.075+0000 7f2f5b400640 1 -- 172.21.15.28:0/923715713 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f6000f8c0 con 0x5630231616f0 2024-05-12T22:15:14.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.075+0000 7f2f5b400640 1 -- 172.21.15.28:0/923715713 <== mon.2 v2:172.21.15.86:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f2f6000fa60 con 0x5630231616f0 2024-05-12T22:15:14.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.075+0000 7f2f7f005740 1 -- 172.21.15.28:0/923715713 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2f6c0043d0 con 0x5630231616f0 2024-05-12T22:15:14.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.076+0000 7f2f5b400640 1 -- 172.21.15.28:0/923715713 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f2f6000fca0 con 0x5630231616f0 2024-05-12T22:15:14.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.076+0000 7f2f7f005740 10 client.26681 fetch_fsmap learned FSMap version 26 2024-05-12T22:15:14.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.076+0000 7f2f7f005740 1 -- 172.21.15.28:0/923715713 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x563022d990f0 con 0x5630231616f0 2024-05-12T22:15:14.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.076+0000 7f2f5b400640 1 -- 172.21.15.28:0/923715713 <== mon.2 v2:172.21.15.86:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2f6c0043d0 con 0x5630231616f0 2024-05-12T22:15:14.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.076+0000 7f2f5b400640 1 -- 172.21.15.28:0/923715713 <== mon.2 v2:172.21.15.86:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f2f60010430 con 0x5630231616f0 2024-05-12T22:15:14.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.076+0000 7f2f7f005740 10 client.26681 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:15:14.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.076+0000 7f2f7f005740 10 client.26681 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:15:14.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.076+0000 7f2f7f005740 20 client.26681 populate_metadata read hostname 'smithi028' 2024-05-12T22:15:14.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.077+0000 7f2f7f005740 1 --2- 172.21.15.28:0/923715713 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x56302317b9f0 0x56302317de30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:14.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.077+0000 7f2f7f005740 4 client.26681 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-12T22:15:14.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.077+0000 7f2f7f005740 1 -- 172.21.15.28:0/923715713 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5630230d32c0 con 0x56302317b9f0 2024-05-12T22:15:14.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.077+0000 7f2f7a200640 1 --2- 172.21.15.28:0/923715713 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x56302317b9f0 0x56302317de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:14.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.077+0000 7f2f7a200640 1 --2- 172.21.15.28:0/923715713 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x56302317b9f0 0x56302317de30 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5ecada01343330c4 server_cookie=8ec652f0c1c9764 in_seq=0 out_seq=0 2024-05-12T22:15:14.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.077+0000 7f2f5b400640 10 client.26681 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:15:14.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.079+0000 7f2f5b400640 1 -- 172.21.15.28:0/923715713 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5630230d32c0 con 0x56302317b9f0 2024-05-12T22:15:14.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.079+0000 7f2f5b400640 10 client.26681 handle_command_reply: tid=0 2024-05-12T22:15:14.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.079+0000 7f2f7f005740 1 client.26681 shutdown 2024-05-12T22:15:14.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.079+0000 7f2f7f005740 1 -- 172.21.15.28:0/923715713 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5630231616f0 msgr2=0x563023168a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:15:14.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.079+0000 7f2f7f005740 1 --2- 172.21.15.28:0/923715713 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5630231616f0 0x563023168a50 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f2f60012700 tx=0x7f2f60012bc0 comp rx=0 tx=0).stop 2024-05-12T22:15:14.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.079+0000 7f2f7f005740 1 -- 172.21.15.28:0/923715713 shutdown_connections 2024-05-12T22:15:14.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.079+0000 7f2f7f005740 1 --2- 172.21.15.28:0/923715713 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x56302317b9f0 0x56302317de30 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:14.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.079+0000 7f2f7f005740 1 --2- 172.21.15.28:0/923715713 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56302316bdc0 0x563023168ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:14.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.079+0000 7f2f7f005740 1 --2- 172.21.15.28:0/923715713 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5630231616f0 0x563023168a50 secure :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f2f60012700 tx=0x7f2f60012bc0 comp rx=0 tx=0).stop 2024-05-12T22:15:14.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.079+0000 7f2f7f005740 1 --2- 172.21.15.28:0/923715713 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563023160dd0 0x56302316b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:14.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.079+0000 7f2f7f005740 1 -- 172.21.15.28:0/923715713 >> 172.21.15.28:0/923715713 conn(0x56302315d0f0 msgr2=0x563023162ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:15:14.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.079+0000 7f2f5b400640 0 client.26681 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:15:14.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.079+0000 7f2f7f005740 1 -- 172.21.15.28:0/923715713 shutdown_connections 2024-05-12T22:15:14.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.080+0000 7f2f7f005740 1 -- 172.21.15.28:0/923715713 wait complete. 2024-05-12T22:15:14.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.080+0000 7f2f7f005740 20 client.26681 trim_cache size 0 max 16384 2024-05-12T22:15:14.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.106+0000 7f2f7f005740 1 Processor -- start 2024-05-12T22:15:14.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.106+0000 7f2f7f005740 1 -- start start 2024-05-12T22:15:14.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.106+0000 7f2f7f005740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563023160dd0 0x563023167990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:14.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.106+0000 7f2f7f005740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56302316bdc0 0x5630231778f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:14.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.106+0000 7f2f7f005740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563023177e30 0x56302316b290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:14.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.106+0000 7f2f7f005740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2f741b1d50 con 0x56302316bdc0 2024-05-12T22:15:14.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.106+0000 7f2f7f005740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f2f741af620 con 0x563023177e30 2024-05-12T22:15:14.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.106+0000 7f2f7f005740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f2f741ad470 con 0x563023160dd0 2024-05-12T22:15:14.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.107+0000 7f2f78e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56302316bdc0 0x5630231778f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:14.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.107+0000 7f2f7a200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563023160dd0 0x563023167990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:14.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.107+0000 7f2f78e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56302316bdc0 0x5630231778f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34574/0 (socket says 172.21.15.28:34574) 2024-05-12T22:15:14.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.107+0000 7f2f78e00640 1 -- 172.21.15.28:0/2442831256 learned_addr learned my addr 172.21.15.28:0/2442831256 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:15:14.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.107+0000 7f2f7a200640 1 --2- 172.21.15.28:0/2442831256 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563023177e30 0x56302316b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:14.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.107+0000 7f2f7a200640 1 -- 172.21.15.28:0/2442831256 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563023177e30 msgr2=0x56302316b290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:15:14.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.107+0000 7f2f7a200640 1 --2- 172.21.15.28:0/2442831256 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563023177e30 0x56302316b290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:14.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.107+0000 7f2f7a200640 1 -- 172.21.15.28:0/2442831256 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56302316bdc0 msgr2=0x5630231778f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:15:14.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.107+0000 7f2f7a200640 1 --2- 172.21.15.28:0/2442831256 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56302316bdc0 0x5630231778f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:14.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.107+0000 7f2f7a200640 1 -- 172.21.15.28:0/2442831256 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563023169360 con 0x563023160dd0 2024-05-12T22:15:14.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.108+0000 7f2f78e00640 1 --2- 172.21.15.28:0/2442831256 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56302316bdc0 0x5630231778f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:15:14.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.108+0000 7f2f7a200640 1 --2- 172.21.15.28:0/2442831256 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563023160dd0 0x563023167990 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x56302315df50 tx=0x7f2f6c000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=460ae9665497f7ed server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:15:14.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.108+0000 7f2f66000640 1 -- 172.21.15.28:0/2442831256 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f6c01f040 con 0x563023160dd0 2024-05-12T22:15:14.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.108+0000 7f2f66000640 1 -- 172.21.15.28:0/2442831256 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2f6c005510 con 0x563023160dd0 2024-05-12T22:15:14.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.108+0000 7f2f66000640 1 -- 172.21.15.28:0/2442831256 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f6c02a3c0 con 0x563023160dd0 2024-05-12T22:15:14.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.109+0000 7f2f7f005740 1 -- 172.21.15.28:0/2442831256 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563023160dd0 msgr2=0x563023167990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:15:14.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.109+0000 7f2f7f005740 1 --2- 172.21.15.28:0/2442831256 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563023160dd0 0x563023167990 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x56302315df50 tx=0x7f2f6c000bc0 comp rx=0 tx=0).stop 2024-05-12T22:15:14.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.109+0000 7f2f7f005740 1 -- 172.21.15.28:0/2442831256 shutdown_connections 2024-05-12T22:15:14.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.109+0000 7f2f7f005740 1 --2- 172.21.15.28:0/2442831256 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563023177e30 0x56302316b290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:14.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.109+0000 7f2f7f005740 1 --2- 172.21.15.28:0/2442831256 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56302316bdc0 0x5630231778f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:14.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.109+0000 7f2f7f005740 1 --2- 172.21.15.28:0/2442831256 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563023160dd0 0x563023167990 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:14.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.109+0000 7f2f7f005740 1 -- 172.21.15.28:0/2442831256 >> 172.21.15.28:0/2442831256 conn(0x56302315d0f0 msgr2=0x563023162ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:15:14.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.109+0000 7f2f7f005740 1 -- 172.21.15.28:0/2442831256 shutdown_connections 2024-05-12T22:15:14.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.109+0000 7f2f7f005740 1 -- 172.21.15.28:0/2442831256 wait complete. 2024-05-12T22:15:14.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.110+0000 7f2f7f005740 1 -- start start 2024-05-12T22:15:14.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.110+0000 7f2f7f005740 1 Processor -- start 2024-05-12T22:15:14.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.110+0000 7f2f7f005740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563023160dd0 0x56302316ff30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:14.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.110+0000 7f2f7f005740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56302316bdc0 0x563023170470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:14.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.110+0000 7f2f7f005740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563023177e30 0x5630231e0f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:14.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.110+0000 7f2f7f005740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2f741adae0 con 0x563023177e30 2024-05-12T22:15:14.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.110+0000 7f2f7f005740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f2f741aef50 con 0x56302316bdc0 2024-05-12T22:15:14.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.110+0000 7f2f7f005740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f2f741ae1b0 con 0x563023160dd0 2024-05-12T22:15:14.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.111+0000 7f2f7f005740 10 client.0 _ll_register_callbacks cb 0x7f2f60009930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:15:14.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.111+0000 7f2f78e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56302316bdc0 0x563023170470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:14.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.111+0000 7f2f78e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56302316bdc0 0x563023170470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:55394/0 (socket says 172.21.15.28:55394) 2024-05-12T22:15:14.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.111+0000 7f2f78e00640 1 -- 172.21.15.28:0/2032204908 learned_addr learned my addr 172.21.15.28:0/2032204908 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:15:14.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.111+0000 7f2f7a200640 1 --2- 172.21.15.28:0/2032204908 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563023160dd0 0x56302316ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:14.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.111+0000 7f2f7a200640 1 --2- 172.21.15.28:0/2032204908 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563023177e30 0x5630231e0f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:14.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.111+0000 7f2f78e00640 1 -- 172.21.15.28:0/2032204908 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563023160dd0 msgr2=0x56302316ff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:15:14.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.111+0000 7f2f78e00640 1 --2- 172.21.15.28:0/2032204908 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563023160dd0 0x56302316ff30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:14.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.111+0000 7f2f78e00640 1 -- 172.21.15.28:0/2032204908 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563023177e30 msgr2=0x5630231e0f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:15:14.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.111+0000 7f2f78e00640 1 --2- 172.21.15.28:0/2032204908 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563023177e30 0x5630231e0f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:14.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.111+0000 7f2f78e00640 1 -- 172.21.15.28:0/2032204908 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5630231e14b0 con 0x56302316bdc0 2024-05-12T22:15:14.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.111+0000 7f2f7a200640 1 --2- 172.21.15.28:0/2032204908 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563023177e30 0x5630231e0f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-12T22:15:14.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.112+0000 7f2f78e00640 1 --2- 172.21.15.28:0/2032204908 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56302316bdc0 0x563023170470 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f2f600108d0 tx=0x7f2f60011fb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d4d775f8c7a9efca server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:15:14.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.112+0000 7f2f5b400640 10 client.0 ms_handle_connect on v2:172.21.15.86:3300/0 2024-05-12T22:15:14.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.112+0000 7f2f5b400640 1 -- 172.21.15.28:0/2032204908 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f60029040 con 0x56302316bdc0 2024-05-12T22:15:14.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.112+0000 7f2f5b400640 1 -- 172.21.15.28:0/2032204908 <== mon.2 v2:172.21.15.86:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f2f60010430 con 0x56302316bdc0 2024-05-12T22:15:14.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.112+0000 7f2f7f005740 1 -- 172.21.15.28:0/2032204908 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2f6c005510 con 0x56302316bdc0 2024-05-12T22:15:14.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.112+0000 7f2f5b400640 1 -- 172.21.15.28:0/2032204908 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f2f60034050 con 0x56302316bdc0 2024-05-12T22:15:14.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.112+0000 7f2f7f005740 10 client.26687 fetch_fsmap learned FSMap version 26 2024-05-12T22:15:14.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.112+0000 7f2f7f005740 1 -- 172.21.15.28:0/2032204908 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5630231e2790 con 0x56302316bdc0 2024-05-12T22:15:14.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.113+0000 7f2f5b400640 1 -- 172.21.15.28:0/2032204908 <== mon.2 v2:172.21.15.86:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2f6c005510 con 0x56302316bdc0 2024-05-12T22:15:14.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.113+0000 7f2f5b400640 1 -- 172.21.15.28:0/2032204908 <== mon.2 v2:172.21.15.86:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f2f600035f0 con 0x56302316bdc0 2024-05-12T22:15:14.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.113+0000 7f2f7f005740 10 client.26687 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:15:14.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.113+0000 7f2f7f005740 10 client.26687 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:15:14.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.113+0000 7f2f7f005740 20 client.26687 populate_metadata read hostname 'smithi028' 2024-05-12T22:15:14.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.113+0000 7f2f7f005740 1 --2- 172.21.15.28:0/2032204908 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5630231e2e50 0x5630231e5270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:14.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.113+0000 7f2f7f005740 4 client.26687 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-12T22:15:14.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.113+0000 7f2f7f005740 1 -- 172.21.15.28:0/2032204908 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5630230cd610 con 0x5630231e2e50 2024-05-12T22:15:14.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.113+0000 7f2f7a200640 1 --2- 172.21.15.28:0/2032204908 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5630231e2e50 0x5630231e5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:14.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.114+0000 7f2f7a200640 1 --2- 172.21.15.28:0/2032204908 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5630231e2e50 0x5630231e5270 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=42566a63656cb442 server_cookie=1010492bfa26fcd9 in_seq=0 out_seq=0 2024-05-12T22:15:14.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.114+0000 7f2f5b400640 10 client.26687 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:15:14.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.114+0000 7f2f5b400640 1 -- 172.21.15.28:0/2032204908 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+268 (crc 0 0 0) 0x5630230cd610 con 0x5630231e2e50 2024-05-12T22:15:14.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.114+0000 7f2f5b400640 10 client.26687 handle_command_reply: tid=0 2024-05-12T22:15:14.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.114+0000 7f2f7f005740 1 client.26687 shutdown 2024-05-12T22:15:14.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.114+0000 7f2f7f005740 1 -- 172.21.15.28:0/2032204908 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56302316bdc0 msgr2=0x563023170470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:15:14.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.114+0000 7f2f7f005740 1 --2- 172.21.15.28:0/2032204908 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56302316bdc0 0x563023170470 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f2f600108d0 tx=0x7f2f60011fb0 comp rx=0 tx=0).stop 2024-05-12T22:15:14.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.115+0000 7f2f7f005740 1 -- 172.21.15.28:0/2032204908 shutdown_connections 2024-05-12T22:15:14.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.115+0000 7f2f7f005740 1 --2- 172.21.15.28:0/2032204908 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5630231e2e50 0x5630231e5270 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:14.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.115+0000 7f2f7f005740 1 --2- 172.21.15.28:0/2032204908 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563023177e30 0x5630231e0f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:14.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.115+0000 7f2f7f005740 1 --2- 172.21.15.28:0/2032204908 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56302316bdc0 0x563023170470 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:14.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.115+0000 7f2f7f005740 1 --2- 172.21.15.28:0/2032204908 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563023160dd0 0x56302316ff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:14.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.115+0000 7f2f7f005740 1 -- 172.21.15.28:0/2032204908 >> 172.21.15.28:0/2032204908 conn(0x56302315d0f0 msgr2=0x563023163700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:15:14.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.115+0000 7f2f5b400640 0 client.26687 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:15:14.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.115+0000 7f2f7f005740 1 -- 172.21.15.28:0/2032204908 shutdown_connections 2024-05-12T22:15:14.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.115+0000 7f2f7f005740 1 -- 172.21.15.28:0/2032204908 wait complete. 2024-05-12T22:15:14.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.115+0000 7f2f7f005740 20 client.26687 trim_cache size 0 max 16384 2024-05-12T22:15:14.125 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-05-12T22:15:14.125 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "scrub active (1168 inodes in the stack)", 2024-05-12T22:15:14.125 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-05-12T22:15:14.126 INFO:teuthology.orchestra.run.smithi028.stdout: "69d0a81f-3ff6-4005-a136-63e58f54ec07": { 2024-05-12T22:15:14.126 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-05-12T22:15:14.126 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "69d0a81f-3ff6-4005-a136-63e58f54ec07", 2024-05-12T22:15:14.126 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-05-12T22:15:14.126 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:15:14.126 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:15:14.127 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-05-12T22:15:14.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.116+0000 7f2f7ac00640 1 -- 172.21.15.28:0/805272255 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f2f5c0a2d00 msgr2=0x7f2f5c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:15:14.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.116+0000 7f2f7ac00640 1 --2- 172.21.15.28:0/805272255 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f2f5c0a2d00 0x7f2f5c0a51c0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f2f680eda70 tx=0x7f2f680f3c80 comp rx=0 tx=0).stop 2024-05-12T22:15:14.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.116+0000 7f2f7ac00640 1 -- 172.21.15.28:0/805272255 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2f74073be0 msgr2=0x7f2f7419a1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:15:14.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.116+0000 7f2f7ac00640 1 --2- 172.21.15.28:0/805272255 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2f74073be0 0x7f2f7419a1d0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f2f68005cc0 tx=0x7f2f6802f420 comp rx=0 tx=0).stop 2024-05-12T22:15:14.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.116+0000 7f2f7ac00640 1 -- 172.21.15.28:0/805272255 shutdown_connections 2024-05-12T22:15:14.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.116+0000 7f2f7ac00640 1 --2- 172.21.15.28:0/805272255 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f2f5c0a2d00 0x7f2f5c0a51c0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:14.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.116+0000 7f2f7ac00640 1 --2- 172.21.15.28:0/805272255 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2f7410df90 0x7f2f7419ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:14.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.116+0000 7f2f7ac00640 1 --2- 172.21.15.28:0/805272255 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f74074500 0x7f2f7419a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:14.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.116+0000 7f2f7ac00640 1 --2- 172.21.15.28:0/805272255 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2f74073be0 0x7f2f7419a1d0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:14.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.116+0000 7f2f7ac00640 1 -- 172.21.15.28:0/805272255 >> 172.21.15.28:0/805272255 conn(0x7f2f740fd200 msgr2=0x7f2f740fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:15:14.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.116+0000 7f2f7ac00640 1 -- 172.21.15.28:0/805272255 shutdown_connections 2024-05-12T22:15:14.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:14.116+0000 7f2f7ac00640 1 -- 172.21.15.28:0/805272255 wait complete. 2024-05-12T22:15:14.138 INFO:tasks.cephfs.filesystem:scrub status for tag:69d0a81f-3ff6-4005-a136-63e58f54ec07 - {'path': '/', 'tag': '69d0a81f-3ff6-4005-a136-63e58f54ec07', 'options': 'recursive,force'} 2024-05-12T22:15:14.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:15:13 smithi186 ceph-mon[34785]: pgmap v190: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-05-12T22:15:14.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:15:13 smithi186 ceph-mon[34785]: scrub summary: active paths [/] 2024-05-12T22:15:14.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:15:13 smithi186 ceph-mon[34785]: scrub summary: idle+waiting paths [/] 2024-05-12T22:15:14.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:15:13 smithi186 ceph-mon[34785]: scrub summary: active paths [/] 2024-05-12T22:15:14.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:15:13 smithi186 ceph-mon[34785]: scrub summary: idle+waiting paths [/] 2024-05-12T22:15:14.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:15:13 smithi086 ceph-mon[34297]: pgmap v190: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-05-12T22:15:14.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:15:13 smithi086 ceph-mon[34297]: scrub summary: active paths [/] 2024-05-12T22:15:14.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:15:13 smithi086 ceph-mon[34297]: scrub summary: idle+waiting paths [/] 2024-05-12T22:15:14.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:15:13 smithi086 ceph-mon[34297]: scrub summary: active paths [/] 2024-05-12T22:15:14.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:15:13 smithi086 ceph-mon[34297]: scrub summary: idle+waiting paths [/] 2024-05-12T22:15:14.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:15:13 smithi028 ceph-mon[29847]: pgmap v190: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-05-12T22:15:14.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:15:13 smithi028 ceph-mon[29847]: scrub summary: active paths [/] 2024-05-12T22:15:14.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:15:13 smithi028 ceph-mon[29847]: scrub summary: idle+waiting paths [/] 2024-05-12T22:15:14.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:15:13 smithi028 ceph-mon[29847]: scrub summary: active paths [/] 2024-05-12T22:15:14.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:15:13 smithi028 ceph-mon[29847]: scrub summary: idle+waiting paths [/] 2024-05-12T22:15:15.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:15:14 smithi086 ceph-mon[34297]: scrub queued for path: / 2024-05-12T22:15:15.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:15:14 smithi086 ceph-mon[34297]: scrub summary: idle+waiting paths [/] 2024-05-12T22:15:15.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:15:14 smithi086 ceph-mon[34297]: scrub summary: active paths [/] 2024-05-12T22:15:15.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:15:14 smithi086 ceph-mon[34297]: scrub summary: active paths [/] 2024-05-12T22:15:15.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:15:14 smithi086 ceph-mon[34297]: scrub summary: idle+waiting paths [/] 2024-05-12T22:15:15.244 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:15:14 smithi086 ceph-mon[34297]: scrub summary: idle+waiting paths [/] 2024-05-12T22:15:15.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:15:14 smithi028 ceph-mon[29847]: scrub queued for path: / 2024-05-12T22:15:15.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:15:14 smithi028 ceph-mon[29847]: scrub summary: idle+waiting paths [/] 2024-05-12T22:15:15.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:15:14 smithi028 ceph-mon[29847]: scrub summary: active paths [/] 2024-05-12T22:15:15.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:15:14 smithi028 ceph-mon[29847]: scrub summary: active paths [/] 2024-05-12T22:15:15.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:15:14 smithi028 ceph-mon[29847]: scrub summary: idle+waiting paths [/] 2024-05-12T22:15:15.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:15:14 smithi028 ceph-mon[29847]: scrub summary: idle+waiting paths [/] 2024-05-12T22:15:15.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:15:14 smithi186 ceph-mon[34785]: scrub queued for path: / 2024-05-12T22:15:15.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:15:14 smithi186 ceph-mon[34785]: scrub summary: idle+waiting paths [/] 2024-05-12T22:15:15.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:15:14 smithi186 ceph-mon[34785]: scrub summary: active paths [/] 2024-05-12T22:15:15.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:15:14 smithi186 ceph-mon[34785]: scrub summary: active paths [/] 2024-05-12T22:15:15.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:15:14 smithi186 ceph-mon[34785]: scrub summary: idle+waiting paths [/] 2024-05-12T22:15:15.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:15:14 smithi186 ceph-mon[34785]: scrub summary: idle+waiting paths [/] 2024-05-12T22:15:16.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:15:15 smithi086 ceph-mon[34297]: pgmap v191: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-05-12T22:15:16.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:15:15 smithi028 ceph-mon[29847]: pgmap v191: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-05-12T22:15:16.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:15:15 smithi186 ceph-mon[34785]: pgmap v191: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-05-12T22:15:17.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:15:16 smithi086 ceph-mon[34297]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:15:17.284 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:15:16 smithi028 ceph-mon[29847]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:15:17.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:15:16 smithi186 ceph-mon[34785]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:15:18.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:15:17 smithi086 ceph-mon[34297]: pgmap v192: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 85 B/s wr, 5 op/s 2024-05-12T22:15:18.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:15:17 smithi086 ceph-mon[34297]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:15:18.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:15:17 smithi086 ceph-mon[34297]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:15:18.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:15:17 smithi086 ceph-mon[34297]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:15:18.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:15:17 smithi028 ceph-mon[29847]: pgmap v192: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 85 B/s wr, 5 op/s 2024-05-12T22:15:18.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:15:17 smithi028 ceph-mon[29847]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:15:18.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:15:17 smithi028 ceph-mon[29847]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:15:18.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:15:17 smithi028 ceph-mon[29847]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:15:18.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:15:17 smithi186 ceph-mon[34785]: pgmap v192: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 85 B/s wr, 5 op/s 2024-05-12T22:15:18.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:15:17 smithi186 ceph-mon[34785]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:15:18.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:15:17 smithi186 ceph-mon[34785]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:15:18.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:15:17 smithi186 ceph-mon[34785]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:15:20.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:15:19 smithi086 ceph-mon[34297]: pgmap v193: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 85 B/s wr, 5 op/s 2024-05-12T22:15:20.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:15:19 smithi028 ceph-mon[29847]: pgmap v193: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 85 B/s wr, 5 op/s 2024-05-12T22:15:20.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:15:19 smithi186 ceph-mon[34785]: pgmap v193: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 85 B/s wr, 5 op/s 2024-05-12T22:15:22.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:15:21 smithi086 ceph-mon[34297]: pgmap v194: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 85 B/s wr, 7 op/s 2024-05-12T22:15:22.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:15:21 smithi028 ceph-mon[29847]: pgmap v194: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 85 B/s wr, 7 op/s 2024-05-12T22:15:22.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:15:21 smithi186 ceph-mon[34785]: pgmap v194: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 85 B/s wr, 7 op/s 2024-05-12T22:15:23.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:15:22 smithi086 ceph-mon[34297]: scrub summary: idle 2024-05-12T22:15:23.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:15:22 smithi086 ceph-mon[34297]: scrub summary: idle 2024-05-12T22:15:23.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:15:22 smithi028 ceph-mon[29847]: scrub summary: idle 2024-05-12T22:15:23.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:15:22 smithi028 ceph-mon[29847]: scrub summary: idle 2024-05-12T22:15:23.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:15:22 smithi186 ceph-mon[34785]: scrub summary: idle 2024-05-12T22:15:23.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:15:22 smithi186 ceph-mon[34785]: scrub summary: idle 2024-05-12T22:15:24.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:15:23 smithi086 ceph-mon[34297]: pgmap v195: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-05-12T22:15:24.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:15:23 smithi028 ceph-mon[29847]: pgmap v195: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-05-12T22:15:24.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:15:23 smithi186 ceph-mon[34785]: pgmap v195: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-05-12T22:15:26.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:15:25 smithi086 ceph-mon[34297]: pgmap v196: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-05-12T22:15:26.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:15:25 smithi028 ceph-mon[29847]: pgmap v196: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-05-12T22:15:26.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:15:25 smithi186 ceph-mon[34785]: pgmap v196: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-05-12T22:15:28.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:15:27 smithi086 ceph-mon[34297]: pgmap v197: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:15:28.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:15:27 smithi028 ceph-mon[29847]: pgmap v197: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:15:28.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:15:27 smithi186 ceph-mon[34785]: pgmap v197: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:15:30.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:15:29 smithi086 ceph-mon[34297]: pgmap v198: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:15:30.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:15:29 smithi028 ceph-mon[29847]: pgmap v198: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:15:30.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:15:29 smithi186 ceph-mon[34785]: pgmap v198: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:15:32.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:15:31 smithi028 ceph-mon[29847]: pgmap v199: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-12T22:15:32.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:15:31 smithi186 ceph-mon[34785]: pgmap v199: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-12T22:15:32.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:15:31 smithi086 ceph-mon[34297]: pgmap v199: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-12T22:15:34.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:15:34 smithi028 ceph-mon[29847]: pgmap v200: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:15:34.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:15:34 smithi186 ceph-mon[34785]: pgmap v200: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:15:34.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:15:34 smithi086 ceph-mon[34297]: pgmap v200: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:15:36.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:15:36 smithi028 ceph-mon[29847]: pgmap v201: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-12T22:15:36.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:15:36 smithi186 ceph-mon[34785]: pgmap v201: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-12T22:15:36.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:15:36 smithi086 ceph-mon[34297]: pgmap v201: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-12T22:15:38.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:15:38 smithi028 ceph-mon[29847]: pgmap v202: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:15:38.294 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:15:38 smithi086 ceph-mon[34297]: pgmap v202: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:15:38.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:15:38 smithi186 ceph-mon[34785]: pgmap v202: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:15:40.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:15:40 smithi028 ceph-mon[29847]: pgmap v203: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:15:40.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:15:40 smithi186 ceph-mon[34785]: pgmap v203: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:15:40.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:15:40 smithi086 ceph-mon[34297]: pgmap v203: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:15:42.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:15:42 smithi028 ceph-mon[29847]: pgmap v204: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-12T22:15:42.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:15:42 smithi186 ceph-mon[34785]: pgmap v204: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-12T22:15:42.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:15:42 smithi086 ceph-mon[34297]: pgmap v204: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-12T22:15:44.139 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-12T22:15:44.140 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-12T22:15:44.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:15:44 smithi028 ceph-mon[29847]: pgmap v205: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:15:44.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.368+0000 7fa1d3200640 1 -- 172.21.15.28:0/2801874371 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa1d4073be0 msgr2=0x7fa1d4073fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:15:44.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.368+0000 7fa1d3200640 1 --2- 172.21.15.28:0/2801874371 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa1d4073be0 0x7fa1d4073fc0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fa1c4009ca0 tx=0x7fa1c402a940 comp rx=0 tx=0).stop 2024-05-12T22:15:44.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.369+0000 7fa1d3200640 1 -- 172.21.15.28:0/2801874371 shutdown_connections 2024-05-12T22:15:44.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.369+0000 7fa1d3200640 1 --2- 172.21.15.28:0/2801874371 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa1d410df90 0x7fa1d4110380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:44.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.369+0000 7fa1d3200640 1 --2- 172.21.15.28:0/2801874371 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1d4074500 0x7fa1d410da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:44.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.369+0000 7fa1d3200640 1 --2- 172.21.15.28:0/2801874371 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa1d4073be0 0x7fa1d4073fc0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:44.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.369+0000 7fa1d3200640 1 -- 172.21.15.28:0/2801874371 >> 172.21.15.28:0/2801874371 conn(0x7fa1d40fd200 msgr2=0x7fa1d40ff620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:15:44.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.369+0000 7fa1d3200640 1 -- 172.21.15.28:0/2801874371 shutdown_connections 2024-05-12T22:15:44.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.369+0000 7fa1d3200640 1 -- 172.21.15.28:0/2801874371 wait complete. 2024-05-12T22:15:44.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.370+0000 7fa1d3200640 1 Processor -- start 2024-05-12T22:15:44.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.370+0000 7fa1d3200640 1 -- start start 2024-05-12T22:15:44.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.370+0000 7fa1d3200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa1d4073be0 0x7fa1d419a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:44.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.371+0000 7fa1d3200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1d4074500 0x7fa1d419a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:44.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.371+0000 7fa1d3200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa1d410df90 0x7fa1d419ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:44.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.371+0000 7fa1d3200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa1d4113d00 con 0x7fa1d4074500 2024-05-12T22:15:44.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.371+0000 7fa1d3200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fa1d4113b80 con 0x7fa1d4073be0 2024-05-12T22:15:44.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.371+0000 7fa1d3200640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa1d4113e80 con 0x7fa1d410df90 2024-05-12T22:15:44.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.371+0000 7fa1c9400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1d4074500 0x7fa1d419a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:44.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.371+0000 7fa1c9400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1d4074500 0x7fa1d419a760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55696/0 (socket says 172.21.15.28:55696) 2024-05-12T22:15:44.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.371+0000 7fa1c9400640 1 -- 172.21.15.28:0/1203071480 learned_addr learned my addr 172.21.15.28:0/1203071480 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:15:44.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.371+0000 7fa1d2800640 1 --2- 172.21.15.28:0/1203071480 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa1d410df90 0x7fa1d419ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:44.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.371+0000 7fa1d1e00640 1 --2- 172.21.15.28:0/1203071480 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa1d4073be0 0x7fa1d419a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:44.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.371+0000 7fa1c9400640 1 -- 172.21.15.28:0/1203071480 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa1d410df90 msgr2=0x7fa1d419ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:15:44.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.371+0000 7fa1c9400640 1 --2- 172.21.15.28:0/1203071480 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa1d410df90 0x7fa1d419ea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:44.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.371+0000 7fa1c9400640 1 -- 172.21.15.28:0/1203071480 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa1d4073be0 msgr2=0x7fa1d419a220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:15:44.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.371+0000 7fa1c9400640 1 --2- 172.21.15.28:0/1203071480 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa1d4073be0 0x7fa1d419a220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:44.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.372+0000 7fa1c9400640 1 -- 172.21.15.28:0/1203071480 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1d419f1c0 con 0x7fa1d4074500 2024-05-12T22:15:44.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.372+0000 7fa1d2800640 1 --2- 172.21.15.28:0/1203071480 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa1d410df90 0x7fa1d419ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:15:44.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.372+0000 7fa1d1e00640 1 --2- 172.21.15.28:0/1203071480 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa1d4073be0 0x7fa1d419a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:15:44.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.372+0000 7fa1c9400640 1 --2- 172.21.15.28:0/1203071480 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1d4074500 0x7fa1d419a760 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fa1bc00ea60 tx=0x7fa1bc00ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:15:44.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.372+0000 7fa1cb400640 1 -- 172.21.15.28:0/1203071480 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1bc00cde0 con 0x7fa1d4074500 2024-05-12T22:15:44.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.372+0000 7fa1cb400640 1 -- 172.21.15.28:0/1203071480 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa1bc0035f0 con 0x7fa1d4074500 2024-05-12T22:15:44.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.372+0000 7fa1cb400640 1 -- 172.21.15.28:0/1203071480 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1bc010670 con 0x7fa1d4074500 2024-05-12T22:15:44.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.372+0000 7fa1d3200640 1 -- 172.21.15.28:0/1203071480 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1d419f4b0 con 0x7fa1d4074500 2024-05-12T22:15:44.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.372+0000 7fa1d3200640 1 -- 172.21.15.28:0/1203071480 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1d41a6d40 con 0x7fa1d4074500 2024-05-12T22:15:44.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.376+0000 7fa1dc058740 1 Processor -- start 2024-05-12T22:15:44.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.376+0000 7fa1dc058740 1 -- start start 2024-05-12T22:15:44.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.376+0000 7fa1cb400640 1 -- 172.21.15.28:0/1203071480 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7fa1bc0037a0 con 0x7fa1d4074500 2024-05-12T22:15:44.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.376+0000 7fa1dc058740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56387bed3dd0 0x56387bed41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:44.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.376+0000 7fa1dc058740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56387bed46f0 0x56387bede880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:44.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.376+0000 7fa1dc058740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56387bededc0 0x56387bee11b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:44.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.376+0000 7fa1dc058740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56387be9c2a0 con 0x56387bed3dd0 2024-05-12T22:15:44.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.376+0000 7fa1dc058740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x56387be38b90 con 0x56387bededc0 2024-05-12T22:15:44.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.376+0000 7fa1dc058740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x56387bde05b0 con 0x56387bed46f0 2024-05-12T22:15:44.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.377+0000 7fa1d2800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56387bed3dd0 0x56387bed41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:44.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.377+0000 7fa1d2800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56387bed3dd0 0x56387bed41b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55710/0 (socket says 172.21.15.28:55710) 2024-05-12T22:15:44.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.377+0000 7fa1d2800640 1 -- 172.21.15.28:0/4102924748 learned_addr learned my addr 172.21.15.28:0/4102924748 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:15:44.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.377+0000 7fa1c9400640 1 --2- 172.21.15.28:0/4102924748 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56387bededc0 0x56387bee11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:44.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.377+0000 7fa1d1e00640 1 --2- 172.21.15.28:0/4102924748 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56387bed46f0 0x56387bede880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:44.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.377+0000 7fa1d2800640 1 -- 172.21.15.28:0/4102924748 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56387bed46f0 msgr2=0x56387bede880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:15:44.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.377+0000 7fa1d2800640 1 --2- 172.21.15.28:0/4102924748 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56387bed46f0 0x56387bede880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:44.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.377+0000 7fa1d2800640 1 -- 172.21.15.28:0/4102924748 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56387bededc0 msgr2=0x56387bee11b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:15:44.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.377+0000 7fa1d2800640 1 --2- 172.21.15.28:0/4102924748 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56387bededc0 0x56387bee11b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:44.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.377+0000 7fa1d2800640 1 -- 172.21.15.28:0/4102924748 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56387bc167d0 con 0x56387bed3dd0 2024-05-12T22:15:44.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.377+0000 7fa1c9400640 1 --2- 172.21.15.28:0/4102924748 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56387bededc0 0x56387bee11b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:15:44.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.377+0000 7fa1d1e00640 1 --2- 172.21.15.28:0/4102924748 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56387bed46f0 0x56387bede880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:15:44.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.377+0000 7fa1d2800640 1 --2- 172.21.15.28:0/4102924748 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56387bed3dd0 0x56387bed41b0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fa1cc010400 tx=0x7fa1cc0108c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=536e61b819eac35d server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:15:44.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.378+0000 7fa1cb400640 1 --2- 172.21.15.28:0/1203071480 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fa1b80a2d80 0x7fa1b80a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:44.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.378+0000 7fa1c3400640 1 -- 172.21.15.28:0/4102924748 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1cc011760 con 0x56387bed3dd0 2024-05-12T22:15:44.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.378+0000 7fa1c3400640 1 -- 172.21.15.28:0/4102924748 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa1cc011de0 con 0x56387bed3dd0 2024-05-12T22:15:44.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.378+0000 7fa1c3400640 1 -- 172.21.15.28:0/4102924748 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1cc018380 con 0x56387bed3dd0 2024-05-12T22:15:44.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.378+0000 7fa1cb400640 1 -- 172.21.15.28:0/1203071480 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7fa1bc0cf690 con 0x7fa1d4074500 2024-05-12T22:15:44.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.378+0000 7fa1d1e00640 1 --2- 172.21.15.28:0/1203071480 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fa1b80a2d80 0x7fa1b80a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:44.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.379+0000 7fa1dc058740 1 -- 172.21.15.28:0/4102924748 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56387bed3dd0 msgr2=0x56387bed41b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:15:44.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.379+0000 7fa1dc058740 1 --2- 172.21.15.28:0/4102924748 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56387bed3dd0 0x56387bed41b0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fa1cc010400 tx=0x7fa1cc0108c0 comp rx=0 tx=0).stop 2024-05-12T22:15:44.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.379+0000 7fa1d1e00640 1 --2- 172.21.15.28:0/1203071480 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fa1b80a2d80 0x7fa1b80a5240 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fa1c4006010 tx=0x7fa1c4005d80 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:15:44.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.379+0000 7fa1dc058740 1 -- 172.21.15.28:0/4102924748 shutdown_connections 2024-05-12T22:15:44.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.379+0000 7fa1dc058740 1 --2- 172.21.15.28:0/4102924748 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56387bededc0 0x56387bee11b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:44.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.379+0000 7fa1dc058740 1 --2- 172.21.15.28:0/4102924748 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56387bed46f0 0x56387bede880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:44.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.379+0000 7fa1dc058740 1 --2- 172.21.15.28:0/4102924748 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56387bed3dd0 0x56387bed41b0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:44.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.379+0000 7fa1dc058740 1 -- 172.21.15.28:0/4102924748 >> 172.21.15.28:0/4102924748 conn(0x56387bed00f0 msgr2=0x56387bed04d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:15:44.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.379+0000 7fa1dc058740 1 -- 172.21.15.28:0/4102924748 shutdown_connections 2024-05-12T22:15:44.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.379+0000 7fa1dc058740 1 -- 172.21.15.28:0/4102924748 wait complete. 2024-05-12T22:15:44.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.379+0000 7fa1dc058740 1 -- start start 2024-05-12T22:15:44.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.380+0000 7fa1dc058740 1 Processor -- start 2024-05-12T22:15:44.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.380+0000 7fa1dc058740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56387bed3dd0 0x56387bede710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:44.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.380+0000 7fa1dc058740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56387bed46f0 0x56387bedba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:44.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.380+0000 7fa1dc058740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56387bededc0 0x56387bedbf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:44.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.380+0000 7fa1dc058740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56387bdf39f0 con 0x56387bed3dd0 2024-05-12T22:15:44.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.380+0000 7fa1dc058740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x56387be34b50 con 0x56387bed46f0 2024-05-12T22:15:44.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.380+0000 7fa1dc058740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x56387bee1bb0 con 0x56387bededc0 2024-05-12T22:15:44.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.380+0000 7fa1dc058740 10 client.0 _ll_register_callbacks cb 0x56387bb4bbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:15:44.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.381+0000 7fa1d1e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56387bed3dd0 0x56387bede710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:44.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.381+0000 7fa1d1e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56387bed3dd0 0x56387bede710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55724/0 (socket says 172.21.15.28:55724) 2024-05-12T22:15:44.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.381+0000 7fa1d1e00640 1 -- 172.21.15.28:0/1229670423 learned_addr learned my addr 172.21.15.28:0/1229670423 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:15:44.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.381+0000 7fa1c9400640 1 --2- 172.21.15.28:0/1229670423 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56387bed46f0 0x56387bedba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:44.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.381+0000 7fa1d2800640 1 --2- 172.21.15.28:0/1229670423 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56387bededc0 0x56387bedbf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:44.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.381+0000 7fa1d1e00640 1 -- 172.21.15.28:0/1229670423 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56387bededc0 msgr2=0x56387bedbf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:15:44.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.381+0000 7fa1d1e00640 1 --2- 172.21.15.28:0/1229670423 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56387bededc0 0x56387bedbf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:44.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.381+0000 7fa1d1e00640 1 -- 172.21.15.28:0/1229670423 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56387bed46f0 msgr2=0x56387bedba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:15:44.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.381+0000 7fa1d1e00640 1 --2- 172.21.15.28:0/1229670423 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56387bed46f0 0x56387bedba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:44.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.381+0000 7fa1d1e00640 1 -- 172.21.15.28:0/1229670423 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56387bb4c3f0 con 0x56387bed3dd0 2024-05-12T22:15:44.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.381+0000 7fa1c9400640 1 --2- 172.21.15.28:0/1229670423 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56387bed46f0 0x56387bedba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:15:44.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.381+0000 7fa1d1e00640 1 --2- 172.21.15.28:0/1229670423 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56387bed3dd0 0x56387bede710 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fa1c4000f80 tx=0x7fa1c40095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=195233e36250c5e3 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:15:44.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.381+0000 7fa1b7e00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-12T22:15:44.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.382+0000 7fa1b7e00640 1 -- 172.21.15.28:0/1229670423 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1c4034040 con 0x56387bed3dd0 2024-05-12T22:15:44.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.382+0000 7fa1b7e00640 1 -- 172.21.15.28:0/1229670423 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7fa1c4003fc0 con 0x56387bed3dd0 2024-05-12T22:15:44.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.382+0000 7fa1dc058740 1 -- 172.21.15.28:0/1229670423 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa1cc011de0 con 0x56387bed3dd0 2024-05-12T22:15:44.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.382+0000 7fa1b7e00640 1 -- 172.21.15.28:0/1229670423 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fa1c4004200 con 0x56387bed3dd0 2024-05-12T22:15:44.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.382+0000 7fa1dc058740 10 client.16569 fetch_fsmap learned FSMap version 26 2024-05-12T22:15:44.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.382+0000 7fa1dc058740 1 -- 172.21.15.28:0/1229670423 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x56387bb0c0f0 con 0x56387bed3dd0 2024-05-12T22:15:44.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.382+0000 7fa1b7e00640 1 -- 172.21.15.28:0/1229670423 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa1cc011de0 con 0x56387bed3dd0 2024-05-12T22:15:44.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.383+0000 7fa1b7e00640 1 -- 172.21.15.28:0/1229670423 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fa1c4050070 con 0x56387bed3dd0 2024-05-12T22:15:44.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.383+0000 7fa1dc058740 10 client.16569 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:15:44.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.383+0000 7fa1dc058740 10 client.16569 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:15:44.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.383+0000 7fa1dc058740 20 client.16569 populate_metadata read hostname 'smithi028' 2024-05-12T22:15:44.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.383+0000 7fa1dc058740 1 --2- 172.21.15.28:0/1229670423 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x56387beee8a0 0x56387bef0d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:44.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.383+0000 7fa1dc058740 4 client.16569 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-12T22:15:44.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.383+0000 7fa1dc058740 1 -- 172.21.15.28:0/1229670423 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56387be462c0 con 0x56387beee8a0 2024-05-12T22:15:44.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.383+0000 7fa1c9400640 1 --2- 172.21.15.28:0/1229670423 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x56387beee8a0 0x56387bef0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:44.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.384+0000 7fa1c9400640 1 --2- 172.21.15.28:0/1229670423 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x56387beee8a0 0x56387bef0d00 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b68096320e09c8d8 server_cookie=19eab09316329fba in_seq=0 out_seq=0 2024-05-12T22:15:44.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.384+0000 7fa1b7e00640 10 client.16569 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:15:44.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.385+0000 7fa1b7e00640 1 -- 172.21.15.28:0/1229670423 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56387be462c0 con 0x56387beee8a0 2024-05-12T22:15:44.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.385+0000 7fa1b7e00640 10 client.16569 handle_command_reply: tid=0 2024-05-12T22:15:44.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.385+0000 7fa1dc058740 1 client.16569 shutdown 2024-05-12T22:15:44.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.385+0000 7fa1dc058740 1 -- 172.21.15.28:0/1229670423 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56387bed3dd0 msgr2=0x56387bede710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:15:44.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.385+0000 7fa1dc058740 1 --2- 172.21.15.28:0/1229670423 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56387bed3dd0 0x56387bede710 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fa1c4000f80 tx=0x7fa1c40095c0 comp rx=0 tx=0).stop 2024-05-12T22:15:44.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.385+0000 7fa1dc058740 1 -- 172.21.15.28:0/1229670423 shutdown_connections 2024-05-12T22:15:44.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.385+0000 7fa1dc058740 1 --2- 172.21.15.28:0/1229670423 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x56387beee8a0 0x56387bef0d00 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:44.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.385+0000 7fa1dc058740 1 --2- 172.21.15.28:0/1229670423 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56387bededc0 0x56387bedbf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:44.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.385+0000 7fa1dc058740 1 --2- 172.21.15.28:0/1229670423 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56387bed46f0 0x56387bedba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:44.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.385+0000 7fa1b7e00640 0 client.16569 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:15:44.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.385+0000 7fa1dc058740 1 --2- 172.21.15.28:0/1229670423 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56387bed3dd0 0x56387bede710 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:44.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.385+0000 7fa1dc058740 1 -- 172.21.15.28:0/1229670423 >> 172.21.15.28:0/1229670423 conn(0x56387bed00f0 msgr2=0x56387bed5ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:15:44.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.385+0000 7fa1dc058740 1 -- 172.21.15.28:0/1229670423 shutdown_connections 2024-05-12T22:15:44.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.385+0000 7fa1dc058740 1 -- 172.21.15.28:0/1229670423 wait complete. 2024-05-12T22:15:44.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.385+0000 7fa1dc058740 20 client.16569 trim_cache size 0 max 16384 2024-05-12T22:15:44.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.405+0000 7fa1dc058740 1 Processor -- start 2024-05-12T22:15:44.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.405+0000 7fa1dc058740 1 -- start start 2024-05-12T22:15:44.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.405+0000 7fa1dc058740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56387bed00f0 0x56387bef65d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:44.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.405+0000 7fa1dc058740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56387bed3dd0 0x56387bef6b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:44.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.405+0000 7fa1dc058740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56387bed46f0 0x56387bed8c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:44.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.405+0000 7fa1dc058740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa1d41b1e00 con 0x56387bed00f0 2024-05-12T22:15:44.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.405+0000 7fa1dc058740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fa1d41af6d0 con 0x56387bed3dd0 2024-05-12T22:15:44.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.405+0000 7fa1dc058740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa1d41ad520 con 0x56387bed46f0 2024-05-12T22:15:44.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.405+0000 7fa1c9400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56387bed3dd0 0x56387bef6b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:44.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.405+0000 7fa1d1e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56387bed00f0 0x56387bef65d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:44.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.405+0000 7fa1c9400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56387bed3dd0 0x56387bef6b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:45582/0 (socket says 172.21.15.28:45582) 2024-05-12T22:15:44.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.405+0000 7fa1c9400640 1 -- 172.21.15.28:0/1355031830 learned_addr learned my addr 172.21.15.28:0/1355031830 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:15:44.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.405+0000 7fa1d2800640 1 --2- 172.21.15.28:0/1355031830 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56387bed46f0 0x56387bed8c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:44.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.406+0000 7fa1c9400640 1 -- 172.21.15.28:0/1355031830 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56387bed46f0 msgr2=0x56387bed8c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:15:44.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.406+0000 7fa1c9400640 1 --2- 172.21.15.28:0/1355031830 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56387bed46f0 0x56387bed8c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:44.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.406+0000 7fa1c9400640 1 -- 172.21.15.28:0/1355031830 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56387bed00f0 msgr2=0x56387bef65d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:15:44.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.406+0000 7fa1c9400640 1 --2- 172.21.15.28:0/1355031830 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56387bed00f0 0x56387bef65d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:44.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.406+0000 7fa1c9400640 1 -- 172.21.15.28:0/1355031830 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56387bee11b0 con 0x56387bed3dd0 2024-05-12T22:15:44.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.406+0000 7fa1d2800640 1 --2- 172.21.15.28:0/1355031830 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56387bed46f0 0x56387bed8c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:15:44.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.406+0000 7fa1c9400640 1 --2- 172.21.15.28:0/1355031830 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56387bed3dd0 0x56387bef6b10 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fa1bc0095f0 tx=0x7fa1bc003f30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=66336cc4db87af60 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:15:44.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.406+0000 7fa1c2a00640 1 -- 172.21.15.28:0/1355031830 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1bc09bbe0 con 0x56387bed3dd0 2024-05-12T22:15:44.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.406+0000 7fa1c2a00640 1 -- 172.21.15.28:0/1355031830 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa1bc0d5030 con 0x56387bed3dd0 2024-05-12T22:15:44.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.406+0000 7fa1c2a00640 1 -- 172.21.15.28:0/1355031830 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1bc0cd090 con 0x56387bed3dd0 2024-05-12T22:15:44.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.407+0000 7fa1dc058740 1 -- 172.21.15.28:0/1355031830 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56387bed3dd0 msgr2=0x56387bef6b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:15:44.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.407+0000 7fa1dc058740 1 --2- 172.21.15.28:0/1355031830 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56387bed3dd0 0x56387bef6b10 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fa1bc0095f0 tx=0x7fa1bc003f30 comp rx=0 tx=0).stop 2024-05-12T22:15:44.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.407+0000 7fa1dc058740 1 -- 172.21.15.28:0/1355031830 shutdown_connections 2024-05-12T22:15:44.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.407+0000 7fa1dc058740 1 --2- 172.21.15.28:0/1355031830 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56387bed46f0 0x56387bed8c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:44.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.407+0000 7fa1dc058740 1 --2- 172.21.15.28:0/1355031830 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56387bed3dd0 0x56387bef6b10 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:44.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.407+0000 7fa1dc058740 1 --2- 172.21.15.28:0/1355031830 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56387bed00f0 0x56387bef65d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:44.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.407+0000 7fa1dc058740 1 -- 172.21.15.28:0/1355031830 >> 172.21.15.28:0/1355031830 conn(0x56387becb530 msgr2=0x56387bee98e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:15:44.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.407+0000 7fa1dc058740 1 -- 172.21.15.28:0/1355031830 shutdown_connections 2024-05-12T22:15:44.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.407+0000 7fa1dc058740 1 -- 172.21.15.28:0/1355031830 wait complete. 2024-05-12T22:15:44.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.407+0000 7fa1dc058740 1 -- start start 2024-05-12T22:15:44.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.408+0000 7fa1dc058740 1 Processor -- start 2024-05-12T22:15:44.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.408+0000 7fa1dc058740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56387bed00f0 0x56387bee4b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:44.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.408+0000 7fa1dc058740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56387bed3dd0 0x56387bee5050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:44.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.408+0000 7fa1dc058740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56387bed46f0 0x56387bee5590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:44.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.408+0000 7fa1dc058740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa1d41adb90 con 0x56387bed3dd0 2024-05-12T22:15:44.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.408+0000 7fa1dc058740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fa1d41af000 con 0x56387bed46f0 2024-05-12T22:15:44.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.408+0000 7fa1dc058740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa1d41ae260 con 0x56387bed00f0 2024-05-12T22:15:44.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.408+0000 7fa1dc058740 10 client.0 _ll_register_callbacks cb 0x7fa1c4009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:15:44.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.408+0000 7fa1c9400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56387bed3dd0 0x56387bee5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:44.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.408+0000 7fa1d2800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56387bed46f0 0x56387bee5590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:44.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.408+0000 7fa1d2800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56387bed46f0 0x56387bee5590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:45594/0 (socket says 172.21.15.28:45594) 2024-05-12T22:15:44.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.408+0000 7fa1c9400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56387bed3dd0 0x56387bee5050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55742/0 (socket says 172.21.15.28:55742) 2024-05-12T22:15:44.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.408+0000 7fa1d2800640 1 -- 172.21.15.28:0/2057382880 learned_addr learned my addr 172.21.15.28:0/2057382880 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:15:44.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.409+0000 7fa1d1e00640 1 --2- 172.21.15.28:0/2057382880 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56387bed00f0 0x56387bee4b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:44.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.409+0000 7fa1c9400640 1 -- 172.21.15.28:0/2057382880 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56387bed00f0 msgr2=0x56387bee4b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:15:44.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.409+0000 7fa1c9400640 1 --2- 172.21.15.28:0/2057382880 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56387bed00f0 0x56387bee4b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:44.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.409+0000 7fa1c9400640 1 -- 172.21.15.28:0/2057382880 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56387bed46f0 msgr2=0x56387bee5590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:15:44.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.409+0000 7fa1c9400640 1 --2- 172.21.15.28:0/2057382880 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56387bed46f0 0x56387bee5590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:44.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.409+0000 7fa1c9400640 1 -- 172.21.15.28:0/2057382880 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56387bf5c900 con 0x56387bed3dd0 2024-05-12T22:15:44.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.409+0000 7fa1d2800640 1 --2- 172.21.15.28:0/2057382880 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56387bed46f0 0x56387bee5590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:15:44.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.409+0000 7fa1c9400640 1 --2- 172.21.15.28:0/2057382880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56387bed3dd0 0x56387bee5050 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fa1bc0a0150 tx=0x7fa1bc0a0000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49d92a5979735cb8 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:15:44.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.409+0000 7fa1b7e00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-12T22:15:44.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.409+0000 7fa1b7e00640 1 -- 172.21.15.28:0/2057382880 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1bc099ee0 con 0x56387bed3dd0 2024-05-12T22:15:44.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.409+0000 7fa1b7e00640 1 -- 172.21.15.28:0/2057382880 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7fa1bc0cd090 con 0x56387bed3dd0 2024-05-12T22:15:44.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.409+0000 7fa1dc058740 1 -- 172.21.15.28:0/2057382880 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa1bc0d5030 con 0x56387bed3dd0 2024-05-12T22:15:44.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.410+0000 7fa1b7e00640 1 -- 172.21.15.28:0/2057382880 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fa1bc0cd290 con 0x56387bed3dd0 2024-05-12T22:15:44.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.410+0000 7fa1dc058740 10 client.16575 fetch_fsmap learned FSMap version 26 2024-05-12T22:15:44.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.410+0000 7fa1dc058740 1 -- 172.21.15.28:0/2057382880 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x56387bf5dbf0 con 0x56387bed3dd0 2024-05-12T22:15:44.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.410+0000 7fa1b7e00640 1 -- 172.21.15.28:0/2057382880 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa1bc0d5030 con 0x56387bed3dd0 2024-05-12T22:15:44.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.410+0000 7fa1b7e00640 1 -- 172.21.15.28:0/2057382880 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fa1bc0d2ce0 con 0x56387bed3dd0 2024-05-12T22:15:44.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.410+0000 7fa1dc058740 10 client.16575 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:15:44.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.410+0000 7fa1dc058740 10 client.16575 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:15:44.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.410+0000 7fa1dc058740 20 client.16575 populate_metadata read hostname 'smithi028' 2024-05-12T22:15:44.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.410+0000 7fa1dc058740 1 --2- 172.21.15.28:0/2057382880 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x56387bededc0 0x56387bf603b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:44.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.410+0000 7fa1dc058740 4 client.16575 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-12T22:15:44.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.410+0000 7fa1dc058740 1 -- 172.21.15.28:0/2057382880 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56387be40610 con 0x56387bededc0 2024-05-12T22:15:44.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.411+0000 7fa1d1e00640 1 --2- 172.21.15.28:0/2057382880 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x56387bededc0 0x56387bf603b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:44.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.411+0000 7fa1d1e00640 1 --2- 172.21.15.28:0/2057382880 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x56387bededc0 0x56387bf603b0 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f54c08945062462d server_cookie=c3987bf79186d52a in_seq=0 out_seq=0 2024-05-12T22:15:44.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.411+0000 7fa1b7e00640 10 client.16575 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:15:44.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.411+0000 7fa1b7e00640 1 -- 172.21.15.28:0/2057382880 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x56387be40610 con 0x56387bededc0 2024-05-12T22:15:44.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.411+0000 7fa1b7e00640 10 client.16575 handle_command_reply: tid=0 2024-05-12T22:15:44.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.411+0000 7fa1dc058740 1 client.16575 shutdown 2024-05-12T22:15:44.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.411+0000 7fa1dc058740 1 -- 172.21.15.28:0/2057382880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56387bed3dd0 msgr2=0x56387bee5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:15:44.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.411+0000 7fa1dc058740 1 --2- 172.21.15.28:0/2057382880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56387bed3dd0 0x56387bee5050 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fa1bc0a0150 tx=0x7fa1bc0a0000 comp rx=0 tx=0).stop 2024-05-12T22:15:44.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.411+0000 7fa1dc058740 1 -- 172.21.15.28:0/2057382880 shutdown_connections 2024-05-12T22:15:44.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.411+0000 7fa1dc058740 1 --2- 172.21.15.28:0/2057382880 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x56387bededc0 0x56387bf603b0 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:44.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.411+0000 7fa1dc058740 1 --2- 172.21.15.28:0/2057382880 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56387bed46f0 0x56387bee5590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:44.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.411+0000 7fa1dc058740 1 --2- 172.21.15.28:0/2057382880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56387bed3dd0 0x56387bee5050 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:44.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.411+0000 7fa1dc058740 1 --2- 172.21.15.28:0/2057382880 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56387bed00f0 0x56387bee4b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:44.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.412+0000 7fa1dc058740 1 -- 172.21.15.28:0/2057382880 >> 172.21.15.28:0/2057382880 conn(0x56387becb530 msgr2=0x56387beddac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:15:44.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.412+0000 7fa1b7e00640 0 client.16575 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:15:44.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.412+0000 7fa1dc058740 1 -- 172.21.15.28:0/2057382880 shutdown_connections 2024-05-12T22:15:44.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.412+0000 7fa1dc058740 1 -- 172.21.15.28:0/2057382880 wait complete. 2024-05-12T22:15:44.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.412+0000 7fa1dc058740 20 client.16575 trim_cache size 0 max 16384 2024-05-12T22:15:44.423 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-05-12T22:15:44.423 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-05-12T22:15:44.423 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-05-12T22:15:44.424 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-05-12T22:15:44.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.413+0000 7fa1d3200640 1 -- 172.21.15.28:0/1203071480 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fa1b80a2d80 msgr2=0x7fa1b80a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:15:44.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.413+0000 7fa1d3200640 1 --2- 172.21.15.28:0/1203071480 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fa1b80a2d80 0x7fa1b80a5240 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fa1c4006010 tx=0x7fa1c4005d80 comp rx=0 tx=0).stop 2024-05-12T22:15:44.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.413+0000 7fa1d3200640 1 -- 172.21.15.28:0/1203071480 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1d4074500 msgr2=0x7fa1d419a760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:15:44.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.413+0000 7fa1d3200640 1 --2- 172.21.15.28:0/1203071480 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1d4074500 0x7fa1d419a760 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fa1bc00ea60 tx=0x7fa1bc00ef20 comp rx=0 tx=0).stop 2024-05-12T22:15:44.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.413+0000 7fa1d3200640 1 -- 172.21.15.28:0/1203071480 shutdown_connections 2024-05-12T22:15:44.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.413+0000 7fa1d3200640 1 --2- 172.21.15.28:0/1203071480 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fa1b80a2d80 0x7fa1b80a5240 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:44.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.413+0000 7fa1d3200640 1 --2- 172.21.15.28:0/1203071480 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa1d410df90 0x7fa1d419ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:44.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.413+0000 7fa1d3200640 1 --2- 172.21.15.28:0/1203071480 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1d4074500 0x7fa1d419a760 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:44.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.413+0000 7fa1d3200640 1 --2- 172.21.15.28:0/1203071480 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa1d4073be0 0x7fa1d419a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:44.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.413+0000 7fa1d3200640 1 -- 172.21.15.28:0/1203071480 >> 172.21.15.28:0/1203071480 conn(0x7fa1d40fd200 msgr2=0x7fa1d40fdfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:15:44.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.413+0000 7fa1d3200640 1 -- 172.21.15.28:0/1203071480 shutdown_connections 2024-05-12T22:15:44.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.413+0000 7fa1d3200640 1 -- 172.21.15.28:0/1203071480 wait complete. 2024-05-12T22:15:44.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:15:44 smithi186 ceph-mon[34785]: pgmap v205: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:15:44.437 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-12T22:15:44.438 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-12T22:15:44.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:15:44 smithi086 ceph-mon[34297]: pgmap v205: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:15:44.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.658+0000 7f48e2200640 1 -- 172.21.15.28:0/1491611614 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48dc074500 msgr2=0x7f48dc10d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:15:44.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.658+0000 7f48e2200640 1 --2- 172.21.15.28:0/1491611614 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48dc074500 0x7f48dc10d990 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f48c8009900 tx=0x7f48c802a480 comp rx=0 tx=0).stop 2024-05-12T22:15:44.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.658+0000 7f48e2200640 1 -- 172.21.15.28:0/1491611614 shutdown_connections 2024-05-12T22:15:44.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.658+0000 7f48e2200640 1 --2- 172.21.15.28:0/1491611614 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f48dc10ded0 0x7f48dc1102c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:44.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.658+0000 7f48e2200640 1 --2- 172.21.15.28:0/1491611614 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48dc074500 0x7f48dc10d990 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:44.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.658+0000 7f48e2200640 1 --2- 172.21.15.28:0/1491611614 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f48dc073be0 0x7f48dc073fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:44.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.658+0000 7f48e2200640 1 -- 172.21.15.28:0/1491611614 >> 172.21.15.28:0/1491611614 conn(0x7f48dc0fd160 msgr2=0x7f48dc0ff580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:15:44.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.658+0000 7f48e2200640 1 -- 172.21.15.28:0/1491611614 shutdown_connections 2024-05-12T22:15:44.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.658+0000 7f48e2200640 1 -- 172.21.15.28:0/1491611614 wait complete. 2024-05-12T22:15:44.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.659+0000 7f48e2200640 1 Processor -- start 2024-05-12T22:15:44.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.660+0000 7f48e2200640 1 -- start start 2024-05-12T22:15:44.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.660+0000 7f48e2200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f48dc073be0 0x7f48dc1015f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:44.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.660+0000 7f48e2200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48dc074500 0x7f48dc101b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:44.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.660+0000 7f48e2200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f48dc10ded0 0x7f48dc102070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:44.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.660+0000 7f48e2200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f48dc113c40 con 0x7f48dc074500 2024-05-12T22:15:44.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.660+0000 7f48e2200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f48dc113ac0 con 0x7f48dc073be0 2024-05-12T22:15:44.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.660+0000 7f48e2200640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f48dc113dc0 con 0x7f48dc10ded0 2024-05-12T22:15:44.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.660+0000 7f48dbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48dc074500 0x7f48dc101b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:44.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.660+0000 7f48e1800640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f48dc10ded0 0x7f48dc102070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:44.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.660+0000 7f48e0e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f48dc073be0 0x7f48dc1015f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:44.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.660+0000 7f48dbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48dc074500 0x7f48dc101b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55766/0 (socket says 172.21.15.28:55766) 2024-05-12T22:15:44.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.660+0000 7f48dbe00640 1 -- 172.21.15.28:0/1401644551 learned_addr learned my addr 172.21.15.28:0/1401644551 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:15:44.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.660+0000 7f48e1800640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f48dc10ded0 0x7f48dc102070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:60002/0 (socket says 172.21.15.28:60002) 2024-05-12T22:15:44.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.661+0000 7f48e0e00640 1 -- 172.21.15.28:0/1401644551 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f48dc10ded0 msgr2=0x7f48dc102070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:15:44.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.661+0000 7f48e0e00640 1 --2- 172.21.15.28:0/1401644551 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f48dc10ded0 0x7f48dc102070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:44.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.661+0000 7f48e0e00640 1 -- 172.21.15.28:0/1401644551 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48dc074500 msgr2=0x7f48dc101b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:15:44.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.661+0000 7f48e0e00640 1 --2- 172.21.15.28:0/1401644551 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48dc074500 0x7f48dc101b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:44.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.661+0000 7f48e0e00640 1 -- 172.21.15.28:0/1401644551 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48dc1aa800 con 0x7f48dc073be0 2024-05-12T22:15:44.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.661+0000 7f48dbe00640 1 --2- 172.21.15.28:0/1401644551 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48dc074500 0x7f48dc101b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:15:44.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.661+0000 7f48e1800640 1 --2- 172.21.15.28:0/1401644551 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f48dc10ded0 0x7f48dc102070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:15:44.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.661+0000 7f48e0e00640 1 --2- 172.21.15.28:0/1401644551 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f48dc073be0 0x7f48dc1015f0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f48d000ea20 tx=0x7f48d000ee20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:15:44.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.661+0000 7f48d9600640 1 -- 172.21.15.28:0/1401644551 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48d000ccc0 con 0x7f48dc073be0 2024-05-12T22:15:44.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.662+0000 7f48e2200640 1 -- 172.21.15.28:0/1401644551 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48dc1aaaf0 con 0x7f48dc073be0 2024-05-12T22:15:44.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.662+0000 7f48d9600640 1 -- 172.21.15.28:0/1401644551 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f48d000ce60 con 0x7f48dc073be0 2024-05-12T22:15:44.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.662+0000 7f48d9600640 1 -- 172.21.15.28:0/1401644551 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48d0018820 con 0x7f48dc073be0 2024-05-12T22:15:44.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.662+0000 7f48e2200640 1 -- 172.21.15.28:0/1401644551 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48dc1ab040 con 0x7f48dc073be0 2024-05-12T22:15:44.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.663+0000 7f48e2200640 1 -- 172.21.15.28:0/1401644551 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f48dc075290 con 0x7f48dc073be0 2024-05-12T22:15:44.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.665+0000 7f48d9600640 1 -- 172.21.15.28:0/1401644551 <== mon.2 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7f48d00189c0 con 0x7f48dc073be0 2024-05-12T22:15:44.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.666+0000 7f48d9600640 1 --2- 172.21.15.28:0/1401644551 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f48c40a2d00 0x7f48c40a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:44.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.666+0000 7f48d9600640 1 -- 172.21.15.28:0/1401644551 <== mon.2 v2:172.21.15.86:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f48d00cf780 con 0x7f48dc073be0 2024-05-12T22:15:44.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.667+0000 7f48dbe00640 1 --2- 172.21.15.28:0/1401644551 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f48c40a2d00 0x7f48c40a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:44.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.667+0000 7f48dbe00640 1 --2- 172.21.15.28:0/1401644551 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f48c40a2d00 0x7f48c40a51c0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f48dc102d60 tx=0x7f48c80046c0 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:15:44.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.671+0000 7f48d9600640 1 -- 172.21.15.28:0/1401644551 <== mon.2 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f48d009c370 con 0x7f48dc073be0 2024-05-12T22:15:44.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.924+0000 7f48e2200640 1 -- 172.21.15.28:0/1401644551 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f48dc102ad0 con 0x7f48dc073be0 2024-05-12T22:15:44.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.926+0000 7f48d9600640 1 -- 172.21.15.28:0/1401644551 <== mon.2 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11864 (secure 0 0 0) 0x7f48d009bc70 con 0x7f48dc073be0 2024-05-12T22:15:44.928 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-12T22:15:44.928 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1844141745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1844141745},{"type":"v1","addr":"172.21.15.186:6833","nonce":1844141745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6833/2960763758","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":2960763758},{"type":"v1","addr":"172.21.15.86:6833","nonce":2960763758}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24470,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/4131134219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":4131134219},{"type":"v1","addr":"172.21.15.28:6835","nonce":4131134219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24481,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/1727869064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1727869064},{"type":"v1","addr":"172.21.15.186:6835","nonce":1727869064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24482,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2938260704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2938260704},{"type":"v1","addr":"172.21.15.28:6837","nonce":2938260704}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24494,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3473700910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3473700910},{"type":"v1","addr":"172.21.15.86:6835","nonce":3473700910}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-12T22:06:33.426986+0000","modified":"2024-05-12T22:13:22.499381+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24581,"mds_1":24550,"mds_2":24532},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24499":{"gid":24499,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6837/3600809235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":3600809235},{"type":"v1","addr":"172.21.15.186:6837","nonce":3600809235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"i","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6839/2399610807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":2399610807},{"type":"v1","addr":"172.21.15.186:6839","nonce":2399610807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24530":{"gid":24530,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.86:6837/3782448884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":3782448884},{"type":"v1","addr":"172.21.15.86:6837","nonce":3782448884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24532":{"gid":24532,"name":"g","rank":2,"incarnation":21,"state":"up:active","state_seq":6,"addr":"172.21.15.28:6839/123892158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":123892158},{"type":"v1","addr":"172.21.15.28:6839","nonce":123892158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24550":{"gid":24550,"name":"a","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.28:6841/761183870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":761183870},{"type":"v1","addr":"172.21.15.28:6841","nonce":761183870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24581":{"gid":24581,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6839/1251999130","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6838","nonce":1251999130},{"type":"v1","addr":"172.21.15.86:6839","nonce":1251999130}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24581,"qdb_cluster":[24550,24581,24532]},"id":1}]} 2024-05-12T22:15:44.930 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 26 2024-05-12T22:15:44.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.929+0000 7f48e2200640 1 -- 172.21.15.28:0/1401644551 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f48c40a2d00 msgr2=0x7f48c40a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:15:44.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.929+0000 7f48e2200640 1 --2- 172.21.15.28:0/1401644551 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f48c40a2d00 0x7f48c40a51c0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f48dc102d60 tx=0x7f48c80046c0 comp rx=0 tx=0).stop 2024-05-12T22:15:44.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.929+0000 7f48e2200640 1 -- 172.21.15.28:0/1401644551 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f48dc073be0 msgr2=0x7f48dc1015f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:15:44.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.929+0000 7f48e2200640 1 --2- 172.21.15.28:0/1401644551 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f48dc073be0 0x7f48dc1015f0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f48d000ea20 tx=0x7f48d000ee20 comp rx=0 tx=0).stop 2024-05-12T22:15:44.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.929+0000 7f48e2200640 1 -- 172.21.15.28:0/1401644551 shutdown_connections 2024-05-12T22:15:44.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.929+0000 7f48e2200640 1 --2- 172.21.15.28:0/1401644551 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f48c40a2d00 0x7f48c40a51c0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:44.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.929+0000 7f48e2200640 1 --2- 172.21.15.28:0/1401644551 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f48dc10ded0 0x7f48dc102070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:44.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.929+0000 7f48e2200640 1 --2- 172.21.15.28:0/1401644551 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48dc074500 0x7f48dc101b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:44.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.929+0000 7f48e2200640 1 --2- 172.21.15.28:0/1401644551 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f48dc073be0 0x7f48dc1015f0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:44.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.929+0000 7f48e2200640 1 -- 172.21.15.28:0/1401644551 >> 172.21.15.28:0/1401644551 conn(0x7f48dc0fd160 msgr2=0x7f48dc0fe990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:15:44.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.929+0000 7f48e2200640 1 -- 172.21.15.28:0/1401644551 shutdown_connections 2024-05-12T22:15:44.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:44.929+0000 7f48e2200640 1 -- 172.21.15.28:0/1401644551 wait complete. 2024-05-12T22:15:44.953 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-05-12T22:15:45.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.168+0000 7f9eb3e00640 1 -- 172.21.15.28:0/539585090 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9eb40744f0 msgr2=0x7f9eb41121a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:15:45.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.168+0000 7f9eb3e00640 1 --2- 172.21.15.28:0/539585090 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9eb40744f0 0x7f9eb41121a0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f9e9c009900 tx=0x7f9e9c02a490 comp rx=0 tx=0).stop 2024-05-12T22:15:45.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.168+0000 7f9eb3e00640 1 -- 172.21.15.28:0/539585090 shutdown_connections 2024-05-12T22:15:45.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.168+0000 7f9eb3e00640 1 --2- 172.21.15.28:0/539585090 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9eb41126e0 0x7f9eb4114ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:45.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.168+0000 7f9eb3e00640 1 --2- 172.21.15.28:0/539585090 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9eb40744f0 0x7f9eb41121a0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:45.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.168+0000 7f9eb3e00640 1 --2- 172.21.15.28:0/539585090 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9eb4073bd0 0x7f9eb4073fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:45.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.168+0000 7f9eb3e00640 1 -- 172.21.15.28:0/539585090 >> 172.21.15.28:0/539585090 conn(0x7f9eb40fd200 msgr2=0x7f9eb40ff620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:15:45.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.168+0000 7f9eb3e00640 1 -- 172.21.15.28:0/539585090 shutdown_connections 2024-05-12T22:15:45.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.168+0000 7f9eb3e00640 1 -- 172.21.15.28:0/539585090 wait complete. 2024-05-12T22:15:45.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.169+0000 7f9eb3e00640 1 Processor -- start 2024-05-12T22:15:45.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.170+0000 7f9eb3e00640 1 -- start start 2024-05-12T22:15:45.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.170+0000 7f9eb3e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9eb4073bd0 0x7f9eb419e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:45.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.170+0000 7f9eb3e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9eb40744f0 0x7f9eb419eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:45.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.170+0000 7f9eb3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9eb41126e0 0x7f9eb41a2eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:45.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.170+0000 7f9eb3e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9eb4118110 con 0x7f9eb41126e0 2024-05-12T22:15:45.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.170+0000 7f9eb3e00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f9eb4117f90 con 0x7f9eb4073bd0 2024-05-12T22:15:45.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.170+0000 7f9eb3e00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f9eb4118290 con 0x7f9eb40744f0 2024-05-12T22:15:45.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.170+0000 7f9eb3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9eb41126e0 0x7f9eb41a2eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:45.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.171+0000 7f9eb3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9eb41126e0 0x7f9eb41a2eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55784/0 (socket says 172.21.15.28:55784) 2024-05-12T22:15:45.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.171+0000 7f9eb3400640 1 -- 172.21.15.28:0/531403567 learned_addr learned my addr 172.21.15.28:0/531403567 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:15:45.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.171+0000 7f9eb2a00640 1 --2- 172.21.15.28:0/531403567 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9eb4073bd0 0x7f9eb419e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:45.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.171+0000 7f9eabe00640 1 --2- 172.21.15.28:0/531403567 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9eb40744f0 0x7f9eb419eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:45.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.171+0000 7f9eb3400640 1 -- 172.21.15.28:0/531403567 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9eb40744f0 msgr2=0x7f9eb419eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:15:45.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.171+0000 7f9eb3400640 1 --2- 172.21.15.28:0/531403567 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9eb40744f0 0x7f9eb419eb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:45.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.171+0000 7f9eb3400640 1 -- 172.21.15.28:0/531403567 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9eb4073bd0 msgr2=0x7f9eb419e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:15:45.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.171+0000 7f9eb3400640 1 --2- 172.21.15.28:0/531403567 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9eb4073bd0 0x7f9eb419e5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:45.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.171+0000 7f9eb3400640 1 -- 172.21.15.28:0/531403567 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9eb41a35e0 con 0x7f9eb41126e0 2024-05-12T22:15:45.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.171+0000 7f9eb2a00640 1 --2- 172.21.15.28:0/531403567 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9eb4073bd0 0x7f9eb419e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:15:45.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.171+0000 7f9eb3400640 1 --2- 172.21.15.28:0/531403567 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9eb41126e0 0x7f9eb41a2eb0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f9eac00eb40 tx=0x7f9eac00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:15:45.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.172+0000 7f9eb8a00640 1 -- 172.21.15.28:0/531403567 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9eac004b20 con 0x7f9eb41126e0 2024-05-12T22:15:45.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.172+0000 7f9eb3e00640 1 -- 172.21.15.28:0/531403567 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9eb41a38d0 con 0x7f9eb41126e0 2024-05-12T22:15:45.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.172+0000 7f9eb8a00640 1 -- 172.21.15.28:0/531403567 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9eac004cc0 con 0x7f9eb41126e0 2024-05-12T22:15:45.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.172+0000 7f9eb8a00640 1 -- 172.21.15.28:0/531403567 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9eac010850 con 0x7f9eb41126e0 2024-05-12T22:15:45.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.172+0000 7f9eb3e00640 1 -- 172.21.15.28:0/531403567 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9eb41ab160 con 0x7f9eb41126e0 2024-05-12T22:15:45.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.175+0000 7f9ebcc05740 1 Processor -- start 2024-05-12T22:15:45.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.175+0000 7f9eb8a00640 1 -- 172.21.15.28:0/531403567 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7f9eac005020 con 0x7f9eb41126e0 2024-05-12T22:15:45.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.176+0000 7f9ebcc05740 1 -- start start 2024-05-12T22:15:45.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.176+0000 7f9ebcc05740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55917f87cb60 0x55917f87cf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:45.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.176+0000 7f9ebcc05740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55917f87d480 0x55917f887610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:45.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.176+0000 7f9ebcc05740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55917f887b50 0x55917f889f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:45.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.176+0000 7f9ebcc05740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55917f7e1a30 con 0x55917f87d480 2024-05-12T22:15:45.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.176+0000 7f9ebcc05740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x55917f789700 con 0x55917f87cb60 2024-05-12T22:15:45.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.176+0000 7f9ebcc05740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55917f7dd9f0 con 0x55917f887b50 2024-05-12T22:15:45.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.177+0000 7f9eb2a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55917f87d480 0x55917f887610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:45.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.177+0000 7f9eabe00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55917f87cb60 0x55917f87cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:45.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.177+0000 7f9eb2a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55917f87d480 0x55917f887610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55800/0 (socket says 172.21.15.28:55800) 2024-05-12T22:15:45.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.177+0000 7f9eb2a00640 1 -- 172.21.15.28:0/2773744024 learned_addr learned my addr 172.21.15.28:0/2773744024 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:15:45.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.177+0000 7f9eabe00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55917f87cb60 0x55917f87cf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:45644/0 (socket says 172.21.15.28:45644) 2024-05-12T22:15:45.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.177+0000 7f9eabe00640 1 --2- 172.21.15.28:0/2773744024 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55917f887b50 0x55917f889f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:45.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.177+0000 7f9eb2a00640 1 -- 172.21.15.28:0/2773744024 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55917f887b50 msgr2=0x55917f889f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:15:45.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.177+0000 7f9eb2a00640 1 --2- 172.21.15.28:0/2773744024 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55917f887b50 0x55917f889f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:45.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.177+0000 7f9eb2a00640 1 -- 172.21.15.28:0/2773744024 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55917f87cb60 msgr2=0x55917f87cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:15:45.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.177+0000 7f9eb2a00640 1 --2- 172.21.15.28:0/2773744024 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55917f87cb60 0x55917f87cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:45.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.177+0000 7f9eb2a00640 1 -- 172.21.15.28:0/2773744024 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55917f4f53f0 con 0x55917f87d480 2024-05-12T22:15:45.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.177+0000 7f9eabe00640 1 --2- 172.21.15.28:0/2773744024 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55917f887b50 0x55917f889f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-12T22:15:45.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.177+0000 7f9eb8a00640 1 --2- 172.21.15.28:0/531403567 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f9e980a2d80 0x7f9e980a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:45.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.177+0000 7f9eb2a00640 1 --2- 172.21.15.28:0/2773744024 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55917f87d480 0x55917f887610 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f9ea40115f0 tx=0x7f9ea4011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=611a5f97108b927f server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:15:45.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.178+0000 7f9ea3e00640 1 -- 172.21.15.28:0/2773744024 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ea401e070 con 0x55917f87d480 2024-05-12T22:15:45.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.178+0000 7f9eb8a00640 1 -- 172.21.15.28:0/531403567 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f9eac0cfb90 con 0x7f9eb41126e0 2024-05-12T22:15:45.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.178+0000 7f9ea3e00640 1 -- 172.21.15.28:0/2773744024 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9ea400fb20 con 0x55917f87d480 2024-05-12T22:15:45.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.178+0000 7f9ea3e00640 1 -- 172.21.15.28:0/2773744024 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ea4019bf0 con 0x55917f87d480 2024-05-12T22:15:45.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.178+0000 7f9eabe00640 1 --2- 172.21.15.28:0/531403567 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f9e980a2d80 0x7f9e980a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:45.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.179+0000 7f9ebcc05740 1 -- 172.21.15.28:0/2773744024 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55917f87d480 msgr2=0x55917f887610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:15:45.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.179+0000 7f9eabe00640 1 --2- 172.21.15.28:0/531403567 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f9e980a2d80 0x7f9e980a5240 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f9e9c000c00 tx=0x7f9e9c005c00 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:15:45.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.179+0000 7f9ebcc05740 1 --2- 172.21.15.28:0/2773744024 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55917f87d480 0x55917f887610 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f9ea40115f0 tx=0x7f9ea4011ab0 comp rx=0 tx=0).stop 2024-05-12T22:15:45.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.179+0000 7f9ebcc05740 1 -- 172.21.15.28:0/2773744024 shutdown_connections 2024-05-12T22:15:45.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.179+0000 7f9ebcc05740 1 --2- 172.21.15.28:0/2773744024 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55917f887b50 0x55917f889f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:45.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.179+0000 7f9ebcc05740 1 --2- 172.21.15.28:0/2773744024 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55917f87d480 0x55917f887610 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:45.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.179+0000 7f9ebcc05740 1 --2- 172.21.15.28:0/2773744024 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55917f87cb60 0x55917f87cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:45.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.179+0000 7f9ebcc05740 1 -- 172.21.15.28:0/2773744024 >> 172.21.15.28:0/2773744024 conn(0x55917f878e80 msgr2=0x55917f879260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:15:45.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.179+0000 7f9ebcc05740 1 -- 172.21.15.28:0/2773744024 shutdown_connections 2024-05-12T22:15:45.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.179+0000 7f9ebcc05740 1 -- 172.21.15.28:0/2773744024 wait complete. 2024-05-12T22:15:45.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.179+0000 7f9ebcc05740 1 -- start start 2024-05-12T22:15:45.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.180+0000 7f9ebcc05740 1 Processor -- start 2024-05-12T22:15:45.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.180+0000 7f9ebcc05740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55917f87cb60 0x55917f884a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:45.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.180+0000 7f9ebcc05740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55917f87d480 0x55917f884fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:45.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.180+0000 7f9ebcc05740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55917f887b50 0x55917f8854f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:45.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.180+0000 7f9ebcc05740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55917f88a940 con 0x55917f87d480 2024-05-12T22:15:45.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.180+0000 7f9ebcc05740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x55917f79ca70 con 0x55917f887b50 2024-05-12T22:15:45.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.180+0000 7f9ebcc05740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55917f88aac0 con 0x55917f87cb60 2024-05-12T22:15:45.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.180+0000 7f9ebcc05740 10 client.0 _ll_register_callbacks cb 0x55917f77fba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:15:45.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.180+0000 7f9eabe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55917f87d480 0x55917f884fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:45.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.181+0000 7f9eabe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55917f87d480 0x55917f884fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55804/0 (socket says 172.21.15.28:55804) 2024-05-12T22:15:45.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.181+0000 7f9eabe00640 1 -- 172.21.15.28:0/1924629399 learned_addr learned my addr 172.21.15.28:0/1924629399 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:15:45.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.181+0000 7f9eb3400640 1 --2- 172.21.15.28:0/1924629399 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55917f887b50 0x55917f8854f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:45.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.181+0000 7f9eb2a00640 1 --2- 172.21.15.28:0/1924629399 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55917f87cb60 0x55917f884a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:45.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.181+0000 7f9eb3400640 1 -- 172.21.15.28:0/1924629399 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55917f87cb60 msgr2=0x55917f884a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:15:45.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.181+0000 7f9eb3400640 1 --2- 172.21.15.28:0/1924629399 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55917f87cb60 0x55917f884a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:45.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.181+0000 7f9eb3400640 1 -- 172.21.15.28:0/1924629399 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55917f87d480 msgr2=0x55917f884fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:15:45.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.181+0000 7f9eb3400640 1 --2- 172.21.15.28:0/1924629399 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55917f87d480 0x55917f884fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:45.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.181+0000 7f9eb3400640 1 -- 172.21.15.28:0/1924629399 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55917f5bf7d0 con 0x55917f887b50 2024-05-12T22:15:45.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.181+0000 7f9eb2a00640 1 --2- 172.21.15.28:0/1924629399 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55917f87cb60 0x55917f884a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:15:45.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.181+0000 7f9eb3400640 1 --2- 172.21.15.28:0/1924629399 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55917f887b50 0x55917f8854f0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f9eac0cd2b0 tx=0x7f9eac09b9e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7bd3887189f94c91 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:15:45.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.181+0000 7f9ea0c00640 10 client.0 ms_handle_connect on v2:172.21.15.86:3300/0 2024-05-12T22:15:45.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.181+0000 7f9ea0c00640 1 -- 172.21.15.28:0/1924629399 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9eac09c080 con 0x55917f887b50 2024-05-12T22:15:45.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.182+0000 7f9ea0c00640 1 -- 172.21.15.28:0/1924629399 <== mon.2 v2:172.21.15.86:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f9eac005310 con 0x55917f887b50 2024-05-12T22:15:45.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.182+0000 7f9ebcc05740 1 -- 172.21.15.28:0/1924629399 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9ea400fb20 con 0x55917f887b50 2024-05-12T22:15:45.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.182+0000 7f9ea0c00640 1 -- 172.21.15.28:0/1924629399 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f9eac0a0430 con 0x55917f887b50 2024-05-12T22:15:45.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.182+0000 7f9ebcc05740 10 client.26723 fetch_fsmap learned FSMap version 26 2024-05-12T22:15:45.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.182+0000 7f9ebcc05740 1 -- 172.21.15.28:0/1924629399 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55917f896b20 con 0x55917f887b50 2024-05-12T22:15:45.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.182+0000 7f9ea0c00640 1 -- 172.21.15.28:0/1924629399 <== mon.2 v2:172.21.15.86:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9ea400fb20 con 0x55917f887b50 2024-05-12T22:15:45.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.182+0000 7f9ea0c00640 1 -- 172.21.15.28:0/1924629399 <== mon.2 v2:172.21.15.86:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f9eac0a4860 con 0x55917f887b50 2024-05-12T22:15:45.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.182+0000 7f9ebcc05740 10 client.26723 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:15:45.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.182+0000 7f9ebcc05740 10 client.26723 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.g 2024-05-12T22:15:45.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.182+0000 7f9ebcc05740 20 client.26723 populate_metadata read hostname 'smithi028' 2024-05-12T22:15:45.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.182+0000 7f9ebcc05740 1 --2- 172.21.15.28:0/1924629399 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x55917f898080 0x55917f89a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:45.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.182+0000 7f9ebcc05740 4 client.26723 mds_command: new command op to 24532 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-12T22:15:45.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.182+0000 7f9ebcc05740 1 -- 172.21.15.28:0/1924629399 --> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55917f7ef160 con 0x55917f898080 2024-05-12T22:15:45.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.183+0000 7f9eb2a00640 1 --2- 172.21.15.28:0/1924629399 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x55917f898080 0x55917f89a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:45.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.183+0000 7f9ea0c00640 10 client.26723 ms_handle_connect on v2:172.21.15.28:6838/123892158 2024-05-12T22:15:45.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.183+0000 7f9eb2a00640 1 --2- 172.21.15.28:0/1924629399 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x55917f898080 0x55917f89a4e0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=eedf5f035a3408ac server_cookie=6bb996479b8dbde9 in_seq=0 out_seq=0 2024-05-12T22:15:45.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.184+0000 7f9ea0c00640 1 -- 172.21.15.28:0/1924629399 <== mds.2 v2:172.21.15.28:6838/123892158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55917f7ef160 con 0x55917f898080 2024-05-12T22:15:45.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.184+0000 7f9ea0c00640 10 client.26723 handle_command_reply: tid=0 2024-05-12T22:15:45.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.184+0000 7f9ebcc05740 1 client.26723 shutdown 2024-05-12T22:15:45.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.184+0000 7f9ebcc05740 1 -- 172.21.15.28:0/1924629399 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55917f887b50 msgr2=0x55917f8854f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:15:45.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.184+0000 7f9ebcc05740 1 --2- 172.21.15.28:0/1924629399 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55917f887b50 0x55917f8854f0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f9eac0cd2b0 tx=0x7f9eac09b9e0 comp rx=0 tx=0).stop 2024-05-12T22:15:45.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.185+0000 7f9ebcc05740 1 -- 172.21.15.28:0/1924629399 shutdown_connections 2024-05-12T22:15:45.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.185+0000 7f9ebcc05740 1 --2- 172.21.15.28:0/1924629399 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x55917f898080 0x55917f89a4e0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:45.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.185+0000 7f9ebcc05740 1 --2- 172.21.15.28:0/1924629399 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55917f887b50 0x55917f8854f0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:45.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.185+0000 7f9ea0c00640 0 client.26723 ms_handle_reset on v2:172.21.15.28:6838/123892158 2024-05-12T22:15:45.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.185+0000 7f9ebcc05740 1 --2- 172.21.15.28:0/1924629399 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55917f87d480 0x55917f884fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:45.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.185+0000 7f9ebcc05740 1 --2- 172.21.15.28:0/1924629399 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55917f87cb60 0x55917f884a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:45.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.185+0000 7f9ebcc05740 1 -- 172.21.15.28:0/1924629399 >> 172.21.15.28:0/1924629399 conn(0x55917f878e80 msgr2=0x55917f87eb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:15:45.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.185+0000 7f9ebcc05740 1 -- 172.21.15.28:0/1924629399 shutdown_connections 2024-05-12T22:15:45.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.185+0000 7f9ebcc05740 1 -- 172.21.15.28:0/1924629399 wait complete. 2024-05-12T22:15:45.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.185+0000 7f9ebcc05740 20 client.26723 trim_cache size 0 max 16384 2024-05-12T22:15:45.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.200+0000 7f9ebcc05740 1 Processor -- start 2024-05-12T22:15:45.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.200+0000 7f9ebcc05740 1 -- start start 2024-05-12T22:15:45.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.200+0000 7f9ebcc05740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55917f87cb60 0x55917f893830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:45.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.200+0000 7f9ebcc05740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55917f87d480 0x55917f89fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:45.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.200+0000 7f9ebcc05740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55917f887b50 0x55917f8a02f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:45.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.200+0000 7f9ebcc05740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9eb41b61c0 con 0x55917f887b50 2024-05-12T22:15:45.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.200+0000 7f9ebcc05740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f9eb41b3a90 con 0x55917f87d480 2024-05-12T22:15:45.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.200+0000 7f9ebcc05740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f9eb41b18e0 con 0x55917f87cb60 2024-05-12T22:15:45.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.200+0000 7f9eb3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55917f887b50 0x55917f8a02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:45.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.200+0000 7f9eb3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55917f887b50 0x55917f8a02f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55808/0 (socket says 172.21.15.28:55808) 2024-05-12T22:15:45.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.201+0000 7f9eb3400640 1 -- 172.21.15.28:0/1773326759 learned_addr learned my addr 172.21.15.28:0/1773326759 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:15:45.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.201+0000 7f9eabe00640 1 --2- 172.21.15.28:0/1773326759 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55917f87d480 0x55917f89fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:45.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.201+0000 7f9eb2a00640 1 --2- 172.21.15.28:0/1773326759 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55917f87cb60 0x55917f893830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:45.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.201+0000 7f9eb3400640 1 -- 172.21.15.28:0/1773326759 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55917f87cb60 msgr2=0x55917f893830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:15:45.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.201+0000 7f9eb3400640 1 --2- 172.21.15.28:0/1773326759 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55917f87cb60 0x55917f893830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:45.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.201+0000 7f9eb3400640 1 -- 172.21.15.28:0/1773326759 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55917f87d480 msgr2=0x55917f89fdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:15:45.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.201+0000 7f9eb3400640 1 --2- 172.21.15.28:0/1773326759 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55917f87d480 0x55917f89fdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:45.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.201+0000 7f9eb3400640 1 -- 172.21.15.28:0/1773326759 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55917f88a280 con 0x55917f887b50 2024-05-12T22:15:45.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.201+0000 7f9eb3400640 1 --2- 172.21.15.28:0/1773326759 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55917f887b50 0x55917f8a02f0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f9eac0058c0 tx=0x7f9eac09bb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d4a4a08fb5c614c server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:15:45.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.201+0000 7f9ea3400640 1 -- 172.21.15.28:0/1773326759 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9eac005620 con 0x55917f887b50 2024-05-12T22:15:45.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.202+0000 7f9ea3400640 1 -- 172.21.15.28:0/1773326759 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9eac0a4cf0 con 0x55917f887b50 2024-05-12T22:15:45.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.202+0000 7f9ea3400640 1 -- 172.21.15.28:0/1773326759 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9eac0986f0 con 0x55917f887b50 2024-05-12T22:15:45.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.202+0000 7f9ebcc05740 1 -- 172.21.15.28:0/1773326759 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55917f887b50 msgr2=0x55917f8a02f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:15:45.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.202+0000 7f9ebcc05740 1 --2- 172.21.15.28:0/1773326759 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55917f887b50 0x55917f8a02f0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f9eac0058c0 tx=0x7f9eac09bb10 comp rx=0 tx=0).stop 2024-05-12T22:15:45.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.202+0000 7f9ebcc05740 1 -- 172.21.15.28:0/1773326759 shutdown_connections 2024-05-12T22:15:45.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.202+0000 7f9ebcc05740 1 --2- 172.21.15.28:0/1773326759 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55917f887b50 0x55917f8a02f0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:45.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.202+0000 7f9ebcc05740 1 --2- 172.21.15.28:0/1773326759 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55917f87d480 0x55917f89fdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:45.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.202+0000 7f9ebcc05740 1 --2- 172.21.15.28:0/1773326759 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55917f87cb60 0x55917f893830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:45.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.202+0000 7f9ebcc05740 1 -- 172.21.15.28:0/1773326759 >> 172.21.15.28:0/1773326759 conn(0x55917f878e80 msgr2=0x55917f87fd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:15:45.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.202+0000 7f9ebcc05740 1 -- 172.21.15.28:0/1773326759 shutdown_connections 2024-05-12T22:15:45.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.202+0000 7f9ebcc05740 1 -- 172.21.15.28:0/1773326759 wait complete. 2024-05-12T22:15:45.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.203+0000 7f9ebcc05740 1 -- start start 2024-05-12T22:15:45.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.203+0000 7f9ebcc05740 1 Processor -- start 2024-05-12T22:15:45.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.203+0000 7f9ebcc05740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55917f87cb60 0x55917f88ff60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:45.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.203+0000 7f9ebcc05740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55917f87d480 0x55917f88b300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:45.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.203+0000 7f9ebcc05740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55917f887b50 0x55917f88b840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:45.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.203+0000 7f9ebcc05740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9eb41b1f50 con 0x55917f87cb60 2024-05-12T22:15:45.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.203+0000 7f9ebcc05740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f9eb41b33c0 con 0x55917f87d480 2024-05-12T22:15:45.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.204+0000 7f9ebcc05740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f9eb41b2620 con 0x55917f887b50 2024-05-12T22:15:45.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.204+0000 7f9ebcc05740 10 client.0 _ll_register_callbacks cb 0x7f9eac09be50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:15:45.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.204+0000 7f9eb2a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55917f87cb60 0x55917f88ff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:45.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.204+0000 7f9eb2a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55917f87cb60 0x55917f88ff60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55812/0 (socket says 172.21.15.28:55812) 2024-05-12T22:15:45.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.204+0000 7f9eb2a00640 1 -- 172.21.15.28:0/3821684443 learned_addr learned my addr 172.21.15.28:0/3821684443 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:15:45.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.204+0000 7f9eb3400640 1 --2- 172.21.15.28:0/3821684443 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55917f887b50 0x55917f88b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:45.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.204+0000 7f9eabe00640 1 --2- 172.21.15.28:0/3821684443 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55917f87d480 0x55917f88b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:45.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.204+0000 7f9eb2a00640 1 -- 172.21.15.28:0/3821684443 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55917f887b50 msgr2=0x55917f88b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:15:45.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.204+0000 7f9eb2a00640 1 --2- 172.21.15.28:0/3821684443 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55917f887b50 0x55917f88b840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:45.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.204+0000 7f9eb2a00640 1 -- 172.21.15.28:0/3821684443 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55917f87d480 msgr2=0x55917f88b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:15:45.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.204+0000 7f9eb2a00640 1 --2- 172.21.15.28:0/3821684443 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55917f87d480 0x55917f88b300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:45.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.204+0000 7f9eb2a00640 1 -- 172.21.15.28:0/3821684443 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55917f88be80 con 0x55917f87cb60 2024-05-12T22:15:45.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.205+0000 7f9eb2a00640 1 --2- 172.21.15.28:0/3821684443 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55917f87cb60 0x55917f88ff60 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f9ea40197a0 tx=0x7f9ea4005870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24b76e767a8d3ab7 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:15:45.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.205+0000 7f9ea0c00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-12T22:15:45.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.205+0000 7f9ea0c00640 1 -- 172.21.15.28:0/3821684443 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ea401e040 con 0x55917f87cb60 2024-05-12T22:15:45.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.205+0000 7f9ea0c00640 1 -- 172.21.15.28:0/3821684443 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f9ea400f740 con 0x55917f87cb60 2024-05-12T22:15:45.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.205+0000 7f9ebcc05740 1 -- 172.21.15.28:0/3821684443 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9eac0a4cf0 con 0x55917f87cb60 2024-05-12T22:15:45.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.205+0000 7f9ea0c00640 1 -- 172.21.15.28:0/3821684443 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f9ea4036050 con 0x55917f87cb60 2024-05-12T22:15:45.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.205+0000 7f9ebcc05740 10 client.16599 fetch_fsmap learned FSMap version 26 2024-05-12T22:15:45.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.205+0000 7f9ebcc05740 1 -- 172.21.15.28:0/3821684443 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55917f8fe490 con 0x55917f87cb60 2024-05-12T22:15:45.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.205+0000 7f9ea0c00640 1 -- 172.21.15.28:0/3821684443 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9eac0a4cf0 con 0x55917f87cb60 2024-05-12T22:15:45.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.206+0000 7f9ea0c00640 1 -- 172.21.15.28:0/3821684443 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f9ea4005160 con 0x55917f87cb60 2024-05-12T22:15:45.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.206+0000 7f9ebcc05740 10 client.16599 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:15:45.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.206+0000 7f9ebcc05740 10 client.16599 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.g 2024-05-12T22:15:45.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.206+0000 7f9ebcc05740 20 client.16599 populate_metadata read hostname 'smithi028' 2024-05-12T22:15:45.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.206+0000 7f9ebcc05740 1 --2- 172.21.15.28:0/3821684443 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x55917f8fea20 0x55917f900e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:45.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.206+0000 7f9ebcc05740 4 client.16599 mds_command: new command op to 24532 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-12T22:15:45.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.206+0000 7f9ebcc05740 1 -- 172.21.15.28:0/3821684443 --> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55917f7e94b0 con 0x55917f8fea20 2024-05-12T22:15:45.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.206+0000 7f9eabe00640 1 --2- 172.21.15.28:0/3821684443 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x55917f8fea20 0x55917f900e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:45.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.207+0000 7f9eabe00640 1 --2- 172.21.15.28:0/3821684443 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x55917f8fea20 0x55917f900e40 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=9259f9f658f0da4 server_cookie=195e23ea819a4d8 in_seq=0 out_seq=0 2024-05-12T22:15:45.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.207+0000 7f9ea0c00640 10 client.16599 ms_handle_connect on v2:172.21.15.28:6838/123892158 2024-05-12T22:15:45.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.207+0000 7f9ea0c00640 1 -- 172.21.15.28:0/3821684443 <== mds.2 v2:172.21.15.28:6838/123892158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55917f7e94b0 con 0x55917f8fea20 2024-05-12T22:15:45.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.207+0000 7f9ea0c00640 10 client.16599 handle_command_reply: tid=0 2024-05-12T22:15:45.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.207+0000 7f9ebcc05740 1 client.16599 shutdown 2024-05-12T22:15:45.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.207+0000 7f9ebcc05740 1 -- 172.21.15.28:0/3821684443 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55917f87cb60 msgr2=0x55917f88ff60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:15:45.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.207+0000 7f9ebcc05740 1 --2- 172.21.15.28:0/3821684443 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55917f87cb60 0x55917f88ff60 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f9ea40197a0 tx=0x7f9ea4005870 comp rx=0 tx=0).stop 2024-05-12T22:15:45.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.207+0000 7f9ebcc05740 1 -- 172.21.15.28:0/3821684443 shutdown_connections 2024-05-12T22:15:45.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.207+0000 7f9ebcc05740 1 --2- 172.21.15.28:0/3821684443 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55917f887b50 0x55917f88b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:45.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.207+0000 7f9ebcc05740 1 --2- 172.21.15.28:0/3821684443 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x55917f8fea20 0x55917f900e40 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:45.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.207+0000 7f9ebcc05740 1 --2- 172.21.15.28:0/3821684443 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55917f87d480 0x55917f88b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:45.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.207+0000 7f9ebcc05740 1 --2- 172.21.15.28:0/3821684443 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55917f87cb60 0x55917f88ff60 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:45.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.207+0000 7f9ebcc05740 1 -- 172.21.15.28:0/3821684443 >> 172.21.15.28:0/3821684443 conn(0x55917f878e80 msgr2=0x55917f87e2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:15:45.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.207+0000 7f9ea0c00640 0 client.16599 ms_handle_reset on v2:172.21.15.28:6838/123892158 2024-05-12T22:15:45.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.208+0000 7f9ebcc05740 1 -- 172.21.15.28:0/3821684443 shutdown_connections 2024-05-12T22:15:45.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.208+0000 7f9ebcc05740 1 -- 172.21.15.28:0/3821684443 wait complete. 2024-05-12T22:15:45.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.208+0000 7f9ebcc05740 20 client.16599 trim_cache size 0 max 16384 2024-05-12T22:15:45.218 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-05-12T22:15:45.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.208+0000 7f9eb3e00640 1 -- 172.21.15.28:0/531403567 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f9e980a2d80 msgr2=0x7f9e980a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:15:45.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.208+0000 7f9eb3e00640 1 --2- 172.21.15.28:0/531403567 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f9e980a2d80 0x7f9e980a5240 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f9e9c000c00 tx=0x7f9e9c005c00 comp rx=0 tx=0).stop 2024-05-12T22:15:45.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.208+0000 7f9eb3e00640 1 -- 172.21.15.28:0/531403567 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9eb41126e0 msgr2=0x7f9eb41a2eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:15:45.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.208+0000 7f9eb3e00640 1 --2- 172.21.15.28:0/531403567 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9eb41126e0 0x7f9eb41a2eb0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f9eac00eb40 tx=0x7f9eac00c2d0 comp rx=0 tx=0).stop 2024-05-12T22:15:45.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.208+0000 7f9eb3e00640 1 -- 172.21.15.28:0/531403567 shutdown_connections 2024-05-12T22:15:45.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.208+0000 7f9eb3e00640 1 --2- 172.21.15.28:0/531403567 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f9e980a2d80 0x7f9e980a5240 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:45.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.208+0000 7f9eb3e00640 1 --2- 172.21.15.28:0/531403567 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9eb41126e0 0x7f9eb41a2eb0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:45.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.208+0000 7f9eb3e00640 1 --2- 172.21.15.28:0/531403567 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9eb40744f0 0x7f9eb419eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:45.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.208+0000 7f9eb3e00640 1 --2- 172.21.15.28:0/531403567 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9eb4073bd0 0x7f9eb419e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:45.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.208+0000 7f9eb3e00640 1 -- 172.21.15.28:0/531403567 >> 172.21.15.28:0/531403567 conn(0x7f9eb40fd200 msgr2=0x7f9eb40fe950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:15:45.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.209+0000 7f9eb3e00640 1 -- 172.21.15.28:0/531403567 shutdown_connections 2024-05-12T22:15:45.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.209+0000 7f9eb3e00640 1 -- 172.21.15.28:0/531403567 wait complete. 2024-05-12T22:15:45.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:15:45 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/1401644551' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:15:45.229 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-05-12T22:15:45.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:15:45 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/1401644551' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:15:45.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.437+0000 7fdafbe00640 1 -- 172.21.15.28:0/36680871 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdafc102890 msgr2=0x7fdafc102c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:15:45.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.437+0000 7fdafbe00640 1 --2- 172.21.15.28:0/36680871 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdafc102890 0x7fdafc102c70 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fdaec009900 tx=0x7fdaec02a490 comp rx=0 tx=0).stop 2024-05-12T22:15:45.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.437+0000 7fdafbe00640 1 -- 172.21.15.28:0/36680871 shutdown_connections 2024-05-12T22:15:45.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.437+0000 7fdafbe00640 1 --2- 172.21.15.28:0/36680871 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdafc10bcb0 0x7fdafc10e150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:45.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.437+0000 7fdafbe00640 1 --2- 172.21.15.28:0/36680871 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdafc1031b0 0x7fdafc10b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:45.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.437+0000 7fdafbe00640 1 --2- 172.21.15.28:0/36680871 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdafc102890 0x7fdafc102c70 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:45.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.437+0000 7fdafbe00640 1 -- 172.21.15.28:0/36680871 >> 172.21.15.28:0/36680871 conn(0x7fdafc0fc0b0 msgr2=0x7fdafc0fe4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:15:45.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.437+0000 7fdafbe00640 1 -- 172.21.15.28:0/36680871 shutdown_connections 2024-05-12T22:15:45.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.437+0000 7fdafbe00640 1 -- 172.21.15.28:0/36680871 wait complete. 2024-05-12T22:15:45.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.438+0000 7fdafbe00640 1 Processor -- start 2024-05-12T22:15:45.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.439+0000 7fdafbe00640 1 -- start start 2024-05-12T22:15:45.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.439+0000 7fdafbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdafc102890 0x7fdafc1004e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:45.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.439+0000 7fdafbe00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdafc1031b0 0x7fdafc100a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:45.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.439+0000 7fdafbe00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdafc10bcb0 0x7fdafc100f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:45.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.439+0000 7fdafbe00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdafc111ab0 con 0x7fdafc102890 2024-05-12T22:15:45.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.439+0000 7fdafbe00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fdafc111930 con 0x7fdafc1031b0 2024-05-12T22:15:45.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.439+0000 7fdafbe00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fdafc111c30 con 0x7fdafc10bcb0 2024-05-12T22:15:45.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.439+0000 7fdafaa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdafc102890 0x7fdafc1004e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:45.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.439+0000 7fdafaa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdafc102890 0x7fdafc1004e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55830/0 (socket says 172.21.15.28:55830) 2024-05-12T22:15:45.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.439+0000 7fdafaa00640 1 -- 172.21.15.28:0/1180904595 learned_addr learned my addr 172.21.15.28:0/1180904595 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:15:45.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.440+0000 7fdafa000640 1 --2- 172.21.15.28:0/1180904595 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdafc1031b0 0x7fdafc100a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:45.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.440+0000 7fdafb400640 1 --2- 172.21.15.28:0/1180904595 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdafc10bcb0 0x7fdafc100f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:45.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.440+0000 7fdafaa00640 1 -- 172.21.15.28:0/1180904595 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdafc10bcb0 msgr2=0x7fdafc100f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:15:45.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.440+0000 7fdafaa00640 1 --2- 172.21.15.28:0/1180904595 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdafc10bcb0 0x7fdafc100f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:45.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.440+0000 7fdafaa00640 1 -- 172.21.15.28:0/1180904595 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdafc1031b0 msgr2=0x7fdafc100a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:15:45.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.440+0000 7fdafaa00640 1 --2- 172.21.15.28:0/1180904595 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdafc1031b0 0x7fdafc100a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:45.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.440+0000 7fdafaa00640 1 -- 172.21.15.28:0/1180904595 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdafc1a63f0 con 0x7fdafc102890 2024-05-12T22:15:45.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.440+0000 7fdafaa00640 1 --2- 172.21.15.28:0/1180904595 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdafc102890 0x7fdafc1004e0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fdaec006010 tx=0x7fdaec0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:15:45.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.440+0000 7fdaf3400640 1 -- 172.21.15.28:0/1180904595 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaec03f030 con 0x7fdafc102890 2024-05-12T22:15:45.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.441+0000 7fdaf3400640 1 -- 172.21.15.28:0/1180904595 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdaec003ce0 con 0x7fdafc102890 2024-05-12T22:15:45.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.441+0000 7fdafbe00640 1 -- 172.21.15.28:0/1180904595 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdafc1a6680 con 0x7fdafc102890 2024-05-12T22:15:45.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.441+0000 7fdaf3400640 1 -- 172.21.15.28:0/1180904595 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaec035070 con 0x7fdafc102890 2024-05-12T22:15:45.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.441+0000 7fdafbe00640 1 -- 172.21.15.28:0/1180904595 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdafc1a6b70 con 0x7fdafc102890 2024-05-12T22:15:45.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.444+0000 7fdb04a20740 1 Processor -- start 2024-05-12T22:15:45.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.444+0000 7fdaf3400640 1 -- 172.21.15.28:0/1180904595 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7fdaec042030 con 0x7fdafc102890 2024-05-12T22:15:45.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.444+0000 7fdb04a20740 1 -- start start 2024-05-12T22:15:45.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.445+0000 7fdb04a20740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56104d2f1b60 0x56104d2f1f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:45.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.445+0000 7fdb04a20740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56104d2f2480 0x56104d2fc610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:45.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.445+0000 7fdb04a20740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56104d2fcb50 0x56104d2fef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:45.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.445+0000 7fdb04a20740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56104d256a30 con 0x56104d2f2480 2024-05-12T22:15:45.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.445+0000 7fdb04a20740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x56104d1fe700 con 0x56104d2f1b60 2024-05-12T22:15:45.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.445+0000 7fdb04a20740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x56104d2529f0 con 0x56104d2fcb50 2024-05-12T22:15:45.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.445+0000 7fdafb400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56104d2f1b60 0x56104d2f1f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:45.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.445+0000 7fdafaa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56104d2f2480 0x56104d2fc610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:45.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.445+0000 7fdafb400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56104d2f1b60 0x56104d2f1f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:45710/0 (socket says 172.21.15.28:45710) 2024-05-12T22:15:45.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.445+0000 7fdafb400640 1 -- 172.21.15.28:0/1248441632 learned_addr learned my addr 172.21.15.28:0/1248441632 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:15:45.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.445+0000 7fdafa000640 1 --2- 172.21.15.28:0/1248441632 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56104d2fcb50 0x56104d2fef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:45.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.445+0000 7fdafb400640 1 -- 172.21.15.28:0/1248441632 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56104d2fcb50 msgr2=0x56104d2fef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:15:45.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.445+0000 7fdafb400640 1 --2- 172.21.15.28:0/1248441632 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56104d2fcb50 0x56104d2fef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:45.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.445+0000 7fdafb400640 1 -- 172.21.15.28:0/1248441632 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56104d2f2480 msgr2=0x56104d2fc610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:15:45.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.445+0000 7fdafb400640 1 --2- 172.21.15.28:0/1248441632 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56104d2f2480 0x56104d2fc610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:45.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.445+0000 7fdafb400640 1 -- 172.21.15.28:0/1248441632 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56104cf6a3f0 con 0x56104d2f1b60 2024-05-12T22:15:45.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.446+0000 7fdafb400640 1 --2- 172.21.15.28:0/1248441632 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56104d2f1b60 0x56104d2f1f40 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fdaf4009e60 tx=0x7fdaf4010740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4880c22958946bd1 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:15:45.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.446+0000 7fdaf3400640 1 --2- 172.21.15.28:0/1180904595 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fdadc0a2d30 0x7fdadc0a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:45.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.446+0000 7fdaebe00640 1 -- 172.21.15.28:0/1248441632 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaf4016070 con 0x56104d2f1b60 2024-05-12T22:15:45.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.446+0000 7fdaf3400640 1 -- 172.21.15.28:0/1180904595 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7fdaec0efa60 con 0x7fdafc102890 2024-05-12T22:15:45.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.446+0000 7fdafaa00640 1 --2- 172.21.15.28:0/1180904595 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fdadc0a2d30 0x7fdadc0a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:45.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.446+0000 7fdaebe00640 1 -- 172.21.15.28:0/1248441632 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdaf4004d20 con 0x56104d2f1b60 2024-05-12T22:15:45.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.446+0000 7fdaebe00640 1 -- 172.21.15.28:0/1248441632 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaf40127d0 con 0x56104d2f1b60 2024-05-12T22:15:45.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.447+0000 7fdb04a20740 1 -- 172.21.15.28:0/1248441632 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56104d2f1b60 msgr2=0x56104d2f1f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:15:45.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.447+0000 7fdb04a20740 1 --2- 172.21.15.28:0/1248441632 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56104d2f1b60 0x56104d2f1f40 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fdaf4009e60 tx=0x7fdaf4010740 comp rx=0 tx=0).stop 2024-05-12T22:15:45.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.447+0000 7fdafaa00640 1 --2- 172.21.15.28:0/1180904595 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fdadc0a2d30 0x7fdadc0a51f0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x56104d2f3120 tx=0x7fdaec0f4040 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:15:45.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.447+0000 7fdb04a20740 1 -- 172.21.15.28:0/1248441632 shutdown_connections 2024-05-12T22:15:45.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.447+0000 7fdb04a20740 1 --2- 172.21.15.28:0/1248441632 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56104d2fcb50 0x56104d2fef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:45.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.447+0000 7fdb04a20740 1 --2- 172.21.15.28:0/1248441632 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56104d2f2480 0x56104d2fc610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:45.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.447+0000 7fdb04a20740 1 --2- 172.21.15.28:0/1248441632 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56104d2f1b60 0x56104d2f1f40 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:45.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.447+0000 7fdb04a20740 1 -- 172.21.15.28:0/1248441632 >> 172.21.15.28:0/1248441632 conn(0x56104d2ede80 msgr2=0x56104d2ee260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:15:45.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.447+0000 7fdb04a20740 1 -- 172.21.15.28:0/1248441632 shutdown_connections 2024-05-12T22:15:45.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.447+0000 7fdb04a20740 1 -- 172.21.15.28:0/1248441632 wait complete. 2024-05-12T22:15:45.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.448+0000 7fdb04a20740 1 -- start start 2024-05-12T22:15:45.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.448+0000 7fdb04a20740 1 Processor -- start 2024-05-12T22:15:45.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.448+0000 7fdb04a20740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56104d2f1b60 0x56104d2f9ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:45.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.449+0000 7fdb04a20740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56104d2f2480 0x56104d2fa020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:45.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.449+0000 7fdb04a20740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56104d2fcb50 0x56104d2fa560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:45.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.449+0000 7fdb04a20740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56104d2ff940 con 0x56104d2fcb50 2024-05-12T22:15:45.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.449+0000 7fdb04a20740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x56104d211a70 con 0x56104d2f1b60 2024-05-12T22:15:45.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.449+0000 7fdb04a20740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x56104d2ffac0 con 0x56104d2f2480 2024-05-12T22:15:45.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.449+0000 7fdb04a20740 10 client.0 _ll_register_callbacks cb 0x56104d1f4ba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:15:45.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.449+0000 7fdafb400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56104d2f1b60 0x56104d2f9ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:45.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.449+0000 7fdafa000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56104d2f2480 0x56104d2fa020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:45.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.449+0000 7fdafb400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56104d2f1b60 0x56104d2f9ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:45712/0 (socket says 172.21.15.28:45712) 2024-05-12T22:15:45.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.449+0000 7fdafb400640 1 -- 172.21.15.28:0/2218350358 learned_addr learned my addr 172.21.15.28:0/2218350358 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:15:45.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.449+0000 7fdafb400640 1 --2- 172.21.15.28:0/2218350358 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56104d2fcb50 0x56104d2fa560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:45.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.449+0000 7fdafa000640 1 -- 172.21.15.28:0/2218350358 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56104d2f1b60 msgr2=0x56104d2f9ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:15:45.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.449+0000 7fdafa000640 1 --2- 172.21.15.28:0/2218350358 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56104d2f1b60 0x56104d2f9ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:45.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.449+0000 7fdafa000640 1 -- 172.21.15.28:0/2218350358 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56104d2fcb50 msgr2=0x56104d2fa560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:15:45.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.449+0000 7fdafa000640 1 --2- 172.21.15.28:0/2218350358 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56104d2fcb50 0x56104d2fa560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:45.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.449+0000 7fdafa000640 1 -- 172.21.15.28:0/2218350358 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56104d0347d0 con 0x56104d2f2480 2024-05-12T22:15:45.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.450+0000 7fdafa000640 1 --2- 172.21.15.28:0/2218350358 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56104d2f2480 0x56104d2fa020 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fdafc101f20 tx=0x7fdae4010c80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ee6215b099775d2d server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:15:45.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.450+0000 7fdae8c00640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-12T22:15:45.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.450+0000 7fdae8c00640 1 -- 172.21.15.28:0/2218350358 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdae400fc50 con 0x56104d2f2480 2024-05-12T22:15:45.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.450+0000 7fdae8c00640 1 -- 172.21.15.28:0/2218350358 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7fdae400fdf0 con 0x56104d2f2480 2024-05-12T22:15:45.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.450+0000 7fdb04a20740 1 -- 172.21.15.28:0/2218350358 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdaf4004d20 con 0x56104d2f2480 2024-05-12T22:15:45.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.450+0000 7fdae8c00640 1 -- 172.21.15.28:0/2218350358 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fdae4005020 con 0x56104d2f2480 2024-05-12T22:15:45.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.451+0000 7fdb04a20740 10 client.26692 fetch_fsmap learned FSMap version 26 2024-05-12T22:15:45.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.451+0000 7fdb04a20740 1 -- 172.21.15.28:0/2218350358 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x56104d30bc60 con 0x56104d2f2480 2024-05-12T22:15:45.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.451+0000 7fdae8c00640 1 -- 172.21.15.28:0/2218350358 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdaf4004d20 con 0x56104d2f2480 2024-05-12T22:15:45.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.451+0000 7fdae8c00640 1 -- 172.21.15.28:0/2218350358 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fdae4019410 con 0x56104d2f2480 2024-05-12T22:15:45.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.451+0000 7fdb04a20740 10 client.26692 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:15:45.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.451+0000 7fdb04a20740 10 client.26692 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-05-12T22:15:45.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.451+0000 7fdb04a20740 20 client.26692 populate_metadata read hostname 'smithi028' 2024-05-12T22:15:45.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.451+0000 7fdb04a20740 1 --2- 172.21.15.28:0/2218350358 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x56104d30d1c0 0x56104d30f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:45.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.451+0000 7fdb04a20740 4 client.26692 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-12T22:15:45.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.451+0000 7fdb04a20740 1 -- 172.21.15.28:0/2218350358 --> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56104d264160 con 0x56104d30d1c0 2024-05-12T22:15:45.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.452+0000 7fdafb400640 1 --2- 172.21.15.28:0/2218350358 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x56104d30d1c0 0x56104d30f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:45.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.452+0000 7fdafb400640 1 --2- 172.21.15.28:0/2218350358 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x56104d30d1c0 0x56104d30f5e0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7ef8e5de1198af7 server_cookie=8c6c42abfd6ba54d in_seq=0 out_seq=0 2024-05-12T22:15:45.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.452+0000 7fdae8c00640 10 client.26692 ms_handle_connect on v2:172.21.15.28:6840/761183870 2024-05-12T22:15:45.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.453+0000 7fdae8c00640 1 -- 172.21.15.28:0/2218350358 <== mds.1 v2:172.21.15.28:6840/761183870 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56104d264160 con 0x56104d30d1c0 2024-05-12T22:15:45.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.453+0000 7fdae8c00640 10 client.26692 handle_command_reply: tid=0 2024-05-12T22:15:45.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.453+0000 7fdb04a20740 1 client.26692 shutdown 2024-05-12T22:15:45.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.453+0000 7fdb04a20740 1 -- 172.21.15.28:0/2218350358 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56104d2f2480 msgr2=0x56104d2fa020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:15:45.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.453+0000 7fdb04a20740 1 --2- 172.21.15.28:0/2218350358 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56104d2f2480 0x56104d2fa020 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fdafc101f20 tx=0x7fdae4010c80 comp rx=0 tx=0).stop 2024-05-12T22:15:45.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.453+0000 7fdb04a20740 1 -- 172.21.15.28:0/2218350358 shutdown_connections 2024-05-12T22:15:45.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.453+0000 7fdb04a20740 1 --2- 172.21.15.28:0/2218350358 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x56104d30d1c0 0x56104d30f5e0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:45.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.453+0000 7fdb04a20740 1 --2- 172.21.15.28:0/2218350358 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56104d2fcb50 0x56104d2fa560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:45.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.453+0000 7fdb04a20740 1 --2- 172.21.15.28:0/2218350358 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56104d2f2480 0x56104d2fa020 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:45.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.453+0000 7fdb04a20740 1 --2- 172.21.15.28:0/2218350358 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56104d2f1b60 0x56104d2f9ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:45.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.453+0000 7fdb04a20740 1 -- 172.21.15.28:0/2218350358 >> 172.21.15.28:0/2218350358 conn(0x56104d2ede80 msgr2=0x56104d2f3b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:15:45.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.453+0000 7fdae8c00640 0 client.26692 ms_handle_reset on v2:172.21.15.28:6840/761183870 2024-05-12T22:15:45.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.453+0000 7fdb04a20740 1 -- 172.21.15.28:0/2218350358 shutdown_connections 2024-05-12T22:15:45.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.453+0000 7fdb04a20740 1 -- 172.21.15.28:0/2218350358 wait complete. 2024-05-12T22:15:45.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.453+0000 7fdb04a20740 20 client.26692 trim_cache size 0 max 16384 2024-05-12T22:15:45.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.468+0000 7fdb04a20740 1 Processor -- start 2024-05-12T22:15:45.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.468+0000 7fdb04a20740 1 -- start start 2024-05-12T22:15:45.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.469+0000 7fdb04a20740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56104d2f1b60 0x56104d308830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:45.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.469+0000 7fdb04a20740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56104d2f2480 0x56104d2f93e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:45.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.469+0000 7fdb04a20740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56104d2fcb50 0x56104d2fb310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:45.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.469+0000 7fdb04a20740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdafc1b1d80 con 0x56104d2f2480 2024-05-12T22:15:45.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.469+0000 7fdb04a20740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fdafc1af650 con 0x56104d2f1b60 2024-05-12T22:15:45.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.469+0000 7fdb04a20740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fdafc1ad4a0 con 0x56104d2fcb50 2024-05-12T22:15:45.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.469+0000 7fdafa000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56104d2f2480 0x56104d2f93e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:45.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.469+0000 7fdafb400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56104d2f1b60 0x56104d308830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:45.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.469+0000 7fdafa000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56104d2f2480 0x56104d2f93e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55848/0 (socket says 172.21.15.28:55848) 2024-05-12T22:15:45.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.469+0000 7fdafb400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56104d2f1b60 0x56104d308830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:45728/0 (socket says 172.21.15.28:45728) 2024-05-12T22:15:45.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.469+0000 7fdafa000640 1 -- 172.21.15.28:0/1570328823 learned_addr learned my addr 172.21.15.28:0/1570328823 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:15:45.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.469+0000 7fdafb400640 1 --2- 172.21.15.28:0/1570328823 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56104d2fcb50 0x56104d2fb310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:45.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.470+0000 7fdafa000640 1 -- 172.21.15.28:0/1570328823 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56104d2fcb50 msgr2=0x56104d2fb310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:15:45.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.470+0000 7fdafa000640 1 --2- 172.21.15.28:0/1570328823 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56104d2fcb50 0x56104d2fb310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:45.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.470+0000 7fdafa000640 1 -- 172.21.15.28:0/1570328823 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56104d2f1b60 msgr2=0x56104d308830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:15:45.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.470+0000 7fdafa000640 1 --2- 172.21.15.28:0/1570328823 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56104d2f1b60 0x56104d308830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:45.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.470+0000 7fdafa000640 1 -- 172.21.15.28:0/1570328823 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56104d2fc1c0 con 0x56104d2f2480 2024-05-12T22:15:45.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.470+0000 7fdafb400640 1 --2- 172.21.15.28:0/1570328823 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56104d2f1b60 0x56104d308830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:15:45.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.470+0000 7fdafa000640 1 --2- 172.21.15.28:0/1570328823 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56104d2f2480 0x56104d2f93e0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fdae400f820 tx=0x7fdae400f850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1942af586d3c4fd server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:15:45.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.470+0000 7fdaeb400640 1 -- 172.21.15.28:0/1570328823 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdae400a2c0 con 0x56104d2f2480 2024-05-12T22:15:45.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.470+0000 7fdaeb400640 1 -- 172.21.15.28:0/1570328823 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdae40193e0 con 0x56104d2f2480 2024-05-12T22:15:45.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.470+0000 7fdaeb400640 1 -- 172.21.15.28:0/1570328823 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdae4005400 con 0x56104d2f2480 2024-05-12T22:15:45.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.471+0000 7fdb04a20740 1 -- 172.21.15.28:0/1570328823 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56104d2f2480 msgr2=0x56104d2f93e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:15:45.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.471+0000 7fdb04a20740 1 --2- 172.21.15.28:0/1570328823 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56104d2f2480 0x56104d2f93e0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fdae400f820 tx=0x7fdae400f850 comp rx=0 tx=0).stop 2024-05-12T22:15:45.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.471+0000 7fdb04a20740 1 -- 172.21.15.28:0/1570328823 shutdown_connections 2024-05-12T22:15:45.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.471+0000 7fdb04a20740 1 --2- 172.21.15.28:0/1570328823 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56104d2fcb50 0x56104d2fb310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:45.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.471+0000 7fdb04a20740 1 --2- 172.21.15.28:0/1570328823 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56104d2f2480 0x56104d2f93e0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:45.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.471+0000 7fdb04a20740 1 --2- 172.21.15.28:0/1570328823 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56104d2f1b60 0x56104d308830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:45.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.471+0000 7fdb04a20740 1 -- 172.21.15.28:0/1570328823 >> 172.21.15.28:0/1570328823 conn(0x56104d2ede80 msgr2=0x56104d2fa330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:15:45.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.471+0000 7fdb04a20740 1 -- 172.21.15.28:0/1570328823 shutdown_connections 2024-05-12T22:15:45.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.471+0000 7fdb04a20740 1 -- 172.21.15.28:0/1570328823 wait complete. 2024-05-12T22:15:45.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.472+0000 7fdb04a20740 1 -- start start 2024-05-12T22:15:45.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.472+0000 7fdb04a20740 1 Processor -- start 2024-05-12T22:15:45.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.472+0000 7fdb04a20740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56104d2f1b60 0x56104d2f6a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:45.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.472+0000 7fdb04a20740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56104d2f2480 0x56104d2f6f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:45.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.472+0000 7fdb04a20740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56104d2fcb50 0x56104d2f74c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:45.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.472+0000 7fdb04a20740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdafc1adb10 con 0x56104d2fcb50 2024-05-12T22:15:45.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.472+0000 7fdb04a20740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fdafc1aef80 con 0x56104d2f1b60 2024-05-12T22:15:45.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.472+0000 7fdb04a20740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fdafc1ae1e0 con 0x56104d2f2480 2024-05-12T22:15:45.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.472+0000 7fdb04a20740 10 client.0 _ll_register_callbacks cb 0x7fdae4010430 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:15:45.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.473+0000 7fdafb400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56104d2f1b60 0x56104d2f6a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:45.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.473+0000 7fdafb400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56104d2f1b60 0x56104d2f6a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:45738/0 (socket says 172.21.15.28:45738) 2024-05-12T22:15:45.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.473+0000 7fdafb400640 1 -- 172.21.15.28:0/3568709445 learned_addr learned my addr 172.21.15.28:0/3568709445 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:15:45.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.473+0000 7fdafa000640 1 --2- 172.21.15.28:0/3568709445 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56104d2f2480 0x56104d2f6f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:45.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.473+0000 7fdafb400640 1 --2- 172.21.15.28:0/3568709445 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56104d2fcb50 0x56104d2f74c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:45.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.473+0000 7fdafb400640 1 -- 172.21.15.28:0/3568709445 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56104d2f2480 msgr2=0x56104d2f6f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:15:45.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.473+0000 7fdafb400640 1 --2- 172.21.15.28:0/3568709445 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56104d2f2480 0x56104d2f6f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:45.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.473+0000 7fdafb400640 1 -- 172.21.15.28:0/3568709445 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56104d2fcb50 msgr2=0x56104d2f74c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:15:45.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.473+0000 7fdafb400640 1 --2- 172.21.15.28:0/3568709445 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56104d2fcb50 0x56104d2f74c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:45.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.473+0000 7fdafb400640 1 -- 172.21.15.28:0/3568709445 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56104d2ff2b0 con 0x56104d2f1b60 2024-05-12T22:15:45.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.473+0000 7fdafa000640 1 --2- 172.21.15.28:0/3568709445 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56104d2f2480 0x56104d2f6f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:15:45.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.473+0000 7fdafb400640 1 --2- 172.21.15.28:0/3568709445 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56104d2f1b60 0x56104d2f6a40 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fdaf4009db0 tx=0x7fdaf4010740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f8b04f3d1b86efdc server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:15:45.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.473+0000 7fdae8c00640 10 client.0 ms_handle_connect on v2:172.21.15.86:3300/0 2024-05-12T22:15:45.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.473+0000 7fdae8c00640 1 -- 172.21.15.28:0/3568709445 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaf4016040 con 0x56104d2f1b60 2024-05-12T22:15:45.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.474+0000 7fdae8c00640 1 -- 172.21.15.28:0/3568709445 <== mon.2 v2:172.21.15.86:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7fdaf40124e0 con 0x56104d2f1b60 2024-05-12T22:15:45.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.474+0000 7fdb04a20740 1 -- 172.21.15.28:0/3568709445 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdae40193e0 con 0x56104d2f1b60 2024-05-12T22:15:45.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.474+0000 7fdae8c00640 1 -- 172.21.15.28:0/3568709445 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fdaf4040050 con 0x56104d2f1b60 2024-05-12T22:15:45.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.474+0000 7fdb04a20740 10 client.26744 fetch_fsmap learned FSMap version 26 2024-05-12T22:15:45.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.474+0000 7fdb04a20740 1 -- 172.21.15.28:0/3568709445 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x56104d373a60 con 0x56104d2f1b60 2024-05-12T22:15:45.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.474+0000 7fdae8c00640 1 -- 172.21.15.28:0/3568709445 <== mon.2 v2:172.21.15.86:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdae40193e0 con 0x56104d2f1b60 2024-05-12T22:15:45.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.474+0000 7fdae8c00640 1 -- 172.21.15.28:0/3568709445 <== mon.2 v2:172.21.15.86:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fdaf400f040 con 0x56104d2f1b60 2024-05-12T22:15:45.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.474+0000 7fdb04a20740 10 client.26744 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:15:45.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.474+0000 7fdb04a20740 10 client.26744 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-05-12T22:15:45.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.474+0000 7fdb04a20740 20 client.26744 populate_metadata read hostname 'smithi028' 2024-05-12T22:15:45.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.474+0000 7fdb04a20740 1 --2- 172.21.15.28:0/3568709445 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x56104d373e60 0x56104d376250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:45.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.474+0000 7fdb04a20740 4 client.26744 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-12T22:15:45.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.474+0000 7fdb04a20740 1 -- 172.21.15.28:0/3568709445 --> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56104d25e4b0 con 0x56104d373e60 2024-05-12T22:15:45.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.475+0000 7fdafa000640 1 --2- 172.21.15.28:0/3568709445 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x56104d373e60 0x56104d376250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:45.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.475+0000 7fdafa000640 1 --2- 172.21.15.28:0/3568709445 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x56104d373e60 0x56104d376250 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c758e56d1d1c757a server_cookie=2a21a32d643a222b in_seq=0 out_seq=0 2024-05-12T22:15:45.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.475+0000 7fdae8c00640 10 client.26744 ms_handle_connect on v2:172.21.15.28:6840/761183870 2024-05-12T22:15:45.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.476+0000 7fdae8c00640 1 -- 172.21.15.28:0/3568709445 <== mds.1 v2:172.21.15.28:6840/761183870 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56104d25e4b0 con 0x56104d373e60 2024-05-12T22:15:45.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.476+0000 7fdae8c00640 10 client.26744 handle_command_reply: tid=0 2024-05-12T22:15:45.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.476+0000 7fdb04a20740 1 client.26744 shutdown 2024-05-12T22:15:45.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.476+0000 7fdb04a20740 1 -- 172.21.15.28:0/3568709445 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56104d2f1b60 msgr2=0x56104d2f6a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:15:45.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.476+0000 7fdb04a20740 1 --2- 172.21.15.28:0/3568709445 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56104d2f1b60 0x56104d2f6a40 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fdaf4009db0 tx=0x7fdaf4010740 comp rx=0 tx=0).stop 2024-05-12T22:15:45.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.476+0000 7fdb04a20740 1 -- 172.21.15.28:0/3568709445 shutdown_connections 2024-05-12T22:15:45.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.476+0000 7fdb04a20740 1 --2- 172.21.15.28:0/3568709445 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x56104d373e60 0x56104d376250 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:45.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.476+0000 7fdb04a20740 1 --2- 172.21.15.28:0/3568709445 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56104d2fcb50 0x56104d2f74c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:45.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.476+0000 7fdb04a20740 1 --2- 172.21.15.28:0/3568709445 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56104d2f2480 0x56104d2f6f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:45.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.476+0000 7fdae8c00640 0 client.26744 ms_handle_reset on v2:172.21.15.28:6840/761183870 2024-05-12T22:15:45.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.476+0000 7fdb04a20740 1 --2- 172.21.15.28:0/3568709445 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56104d2f1b60 0x56104d2f6a40 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:45.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.476+0000 7fdb04a20740 1 -- 172.21.15.28:0/3568709445 >> 172.21.15.28:0/3568709445 conn(0x56104d2ede80 msgr2=0x56104d2f32b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:15:45.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.476+0000 7fdb04a20740 1 -- 172.21.15.28:0/3568709445 shutdown_connections 2024-05-12T22:15:45.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.476+0000 7fdb04a20740 1 -- 172.21.15.28:0/3568709445 wait complete. 2024-05-12T22:15:45.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.476+0000 7fdb04a20740 20 client.26744 trim_cache size 0 max 16384 2024-05-12T22:15:45.486 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-05-12T22:15:45.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.477+0000 7fdafbe00640 1 -- 172.21.15.28:0/1180904595 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fdadc0a2d30 msgr2=0x7fdadc0a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:15:45.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.477+0000 7fdafbe00640 1 --2- 172.21.15.28:0/1180904595 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fdadc0a2d30 0x7fdadc0a51f0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x56104d2f3120 tx=0x7fdaec0f4040 comp rx=0 tx=0).stop 2024-05-12T22:15:45.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.477+0000 7fdafbe00640 1 -- 172.21.15.28:0/1180904595 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdafc102890 msgr2=0x7fdafc1004e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:15:45.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.477+0000 7fdafbe00640 1 --2- 172.21.15.28:0/1180904595 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdafc102890 0x7fdafc1004e0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fdaec006010 tx=0x7fdaec0042f0 comp rx=0 tx=0).stop 2024-05-12T22:15:45.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.477+0000 7fdafbe00640 1 -- 172.21.15.28:0/1180904595 shutdown_connections 2024-05-12T22:15:45.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.477+0000 7fdafbe00640 1 --2- 172.21.15.28:0/1180904595 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fdadc0a2d30 0x7fdadc0a51f0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:45.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.477+0000 7fdafbe00640 1 --2- 172.21.15.28:0/1180904595 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdafc10bcb0 0x7fdafc100f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:45.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.477+0000 7fdafbe00640 1 --2- 172.21.15.28:0/1180904595 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdafc1031b0 0x7fdafc100a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:45.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.477+0000 7fdafbe00640 1 --2- 172.21.15.28:0/1180904595 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdafc102890 0x7fdafc1004e0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:45.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.477+0000 7fdafbe00640 1 -- 172.21.15.28:0/1180904595 >> 172.21.15.28:0/1180904595 conn(0x7fdafc0fc0b0 msgr2=0x7fdafc0fd7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:15:45.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.477+0000 7fdafbe00640 1 -- 172.21.15.28:0/1180904595 shutdown_connections 2024-05-12T22:15:45.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.477+0000 7fdafbe00640 1 -- 172.21.15.28:0/1180904595 wait complete. 2024-05-12T22:15:45.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:15:45 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/1401644551' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:15:45.495 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-05-12T22:15:45.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.701+0000 7fd1a0a00640 1 -- 172.21.15.28:0/2415155797 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd19c108770 msgr2=0x7fd19c114b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:15:45.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.701+0000 7fd1a0a00640 1 --2- 172.21.15.28:0/2415155797 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd19c108770 0x7fd19c114b90 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fd194009860 tx=0x7fd19402a3d0 comp rx=0 tx=0).stop 2024-05-12T22:15:45.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.701+0000 7fd1a0a00640 1 -- 172.21.15.28:0/2415155797 shutdown_connections 2024-05-12T22:15:45.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.701+0000 7fd1a0a00640 1 --2- 172.21.15.28:0/2415155797 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd19c108770 0x7fd19c114b90 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:45.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.701+0000 7fd1a0a00640 1 --2- 172.21.15.28:0/2415155797 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd19c107dd0 0x7fd19c108230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:45.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.701+0000 7fd1a0a00640 1 --2- 172.21.15.28:0/2415155797 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd19c10ddf0 0x7fd19c10e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:45.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.701+0000 7fd1a0a00640 1 -- 172.21.15.28:0/2415155797 >> 172.21.15.28:0/2415155797 conn(0x7fd19c077650 msgr2=0x7fd19c106650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:15:45.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.701+0000 7fd1a0a00640 1 -- 172.21.15.28:0/2415155797 shutdown_connections 2024-05-12T22:15:45.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.701+0000 7fd1a0a00640 1 -- 172.21.15.28:0/2415155797 wait complete. 2024-05-12T22:15:45.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.702+0000 7fd1a0a00640 1 Processor -- start 2024-05-12T22:15:45.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.702+0000 7fd1a0a00640 1 -- start start 2024-05-12T22:15:45.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.703+0000 7fd1a0a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd19c107dd0 0x7fd19c19e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:45.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.703+0000 7fd1a0a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd19c108770 0x7fd19c19eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:45.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.703+0000 7fd1a0a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd19c10ddf0 0x7fd19c1a2e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:45.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.703+0000 7fd1a0a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd19c1180d0 con 0x7fd19c10ddf0 2024-05-12T22:15:45.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.703+0000 7fd1a0a00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fd19c117f50 con 0x7fd19c107dd0 2024-05-12T22:15:45.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.703+0000 7fd1a0a00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd19c118250 con 0x7fd19c108770 2024-05-12T22:15:45.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.703+0000 7fd19b200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd19c10ddf0 0x7fd19c1a2e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:45.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.703+0000 7fd19b200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd19c10ddf0 0x7fd19c1a2e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55884/0 (socket says 172.21.15.28:55884) 2024-05-12T22:15:45.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.703+0000 7fd19b200640 1 -- 172.21.15.28:0/1309439606 learned_addr learned my addr 172.21.15.28:0/1309439606 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:15:45.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.703+0000 7fd191e00640 1 --2- 172.21.15.28:0/1309439606 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd19c108770 0x7fd19c19eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:45.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.703+0000 7fd19a800640 1 --2- 172.21.15.28:0/1309439606 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd19c107dd0 0x7fd19c19e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:45.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.703+0000 7fd19b200640 1 -- 172.21.15.28:0/1309439606 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd19c108770 msgr2=0x7fd19c19eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:15:45.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.703+0000 7fd19b200640 1 --2- 172.21.15.28:0/1309439606 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd19c108770 0x7fd19c19eab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:45.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.704+0000 7fd19b200640 1 -- 172.21.15.28:0/1309439606 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd19c107dd0 msgr2=0x7fd19c19e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:15:45.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.704+0000 7fd19b200640 1 --2- 172.21.15.28:0/1309439606 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd19c107dd0 0x7fd19c19e570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:45.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.704+0000 7fd19b200640 1 -- 172.21.15.28:0/1309439606 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd19c1a3560 con 0x7fd19c10ddf0 2024-05-12T22:15:45.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.704+0000 7fd19b200640 1 --2- 172.21.15.28:0/1309439606 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd19c10ddf0 0x7fd19c1a2e30 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fd194009980 tx=0x7fd194030650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:15:45.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.704+0000 7fd193e00640 1 -- 172.21.15.28:0/1309439606 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd194040070 con 0x7fd19c10ddf0 2024-05-12T22:15:45.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.704+0000 7fd193e00640 1 -- 172.21.15.28:0/1309439606 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd194005550 con 0x7fd19c10ddf0 2024-05-12T22:15:45.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.704+0000 7fd1a0a00640 1 -- 172.21.15.28:0/1309439606 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd19c1a37f0 con 0x7fd19c10ddf0 2024-05-12T22:15:45.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.704+0000 7fd193e00640 1 -- 172.21.15.28:0/1309439606 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd194031a40 con 0x7fd19c10ddf0 2024-05-12T22:15:45.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.704+0000 7fd1a0a00640 1 -- 172.21.15.28:0/1309439606 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd19c1ab0e0 con 0x7fd19c10ddf0 2024-05-12T22:15:45.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.708+0000 7fd193e00640 1 -- 172.21.15.28:0/1309439606 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7fd194039040 con 0x7fd19c10ddf0 2024-05-12T22:15:45.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.708+0000 7fd1a4219740 1 Processor -- start 2024-05-12T22:15:45.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.708+0000 7fd1a4219740 1 -- start start 2024-05-12T22:15:45.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.708+0000 7fd1a4219740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55eff1782b60 0x55eff1782f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:45.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.708+0000 7fd1a4219740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eff1783480 0x55eff178d610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:45.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.709+0000 7fd1a4219740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55eff178db50 0x55eff178ff40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:45.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.709+0000 7fd1a4219740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55eff16e7a30 con 0x55eff1783480 2024-05-12T22:15:45.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.709+0000 7fd1a4219740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x55eff168f700 con 0x55eff178db50 2024-05-12T22:15:45.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.709+0000 7fd1a4219740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55eff16e39f0 con 0x55eff1782b60 2024-05-12T22:15:45.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.709+0000 7fd191e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55eff1782b60 0x55eff1782f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:45.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.709+0000 7fd191e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55eff1782b60 0x55eff1782f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:60168/0 (socket says 172.21.15.28:60168) 2024-05-12T22:15:45.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.709+0000 7fd191e00640 1 -- 172.21.15.28:0/3899743306 learned_addr learned my addr 172.21.15.28:0/3899743306 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:15:45.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.709+0000 7fd19a800640 1 --2- 172.21.15.28:0/3899743306 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eff1783480 0x55eff178d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:45.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.709+0000 7fd191e00640 1 --2- 172.21.15.28:0/3899743306 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55eff178db50 0x55eff178ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:45.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.710+0000 7fd19a800640 1 -- 172.21.15.28:0/3899743306 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55eff1782b60 msgr2=0x55eff1782f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:15:45.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.710+0000 7fd19a800640 1 --2- 172.21.15.28:0/3899743306 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55eff1782b60 0x55eff1782f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:45.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.710+0000 7fd19a800640 1 -- 172.21.15.28:0/3899743306 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55eff178db50 msgr2=0x55eff178ff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:15:45.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.710+0000 7fd19a800640 1 --2- 172.21.15.28:0/3899743306 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55eff178db50 0x55eff178ff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:45.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.710+0000 7fd19a800640 1 -- 172.21.15.28:0/3899743306 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eff13fb3f0 con 0x55eff1783480 2024-05-12T22:15:45.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.710+0000 7fd191e00640 1 --2- 172.21.15.28:0/3899743306 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55eff178db50 0x55eff178ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:15:45.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.710+0000 7fd19a800640 1 --2- 172.21.15.28:0/3899743306 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eff1783480 0x55eff178d610 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fd18c00e450 tx=0x7fd18c0095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=638d20527e573db5 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:15:45.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.710+0000 7fd193e00640 1 --2- 172.21.15.28:0/1309439606 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fd1800a2d80 0x7fd1800a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:45.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.710+0000 7fd18b400640 1 -- 172.21.15.28:0/3899743306 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd18c01d070 con 0x55eff1783480 2024-05-12T22:15:45.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.710+0000 7fd193e00640 1 -- 172.21.15.28:0/1309439606 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7fd1940efd00 con 0x7fd19c10ddf0 2024-05-12T22:15:45.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.710+0000 7fd18b400640 1 -- 172.21.15.28:0/3899743306 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd18c003e40 con 0x55eff1783480 2024-05-12T22:15:45.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.711+0000 7fd191e00640 1 --2- 172.21.15.28:0/1309439606 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fd1800a2d80 0x7fd1800a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:45.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.711+0000 7fd18b400640 1 -- 172.21.15.28:0/3899743306 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd18c00dad0 con 0x55eff1783480 2024-05-12T22:15:45.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.711+0000 7fd1a4219740 1 -- 172.21.15.28:0/3899743306 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eff1783480 msgr2=0x55eff178d610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:15:45.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.711+0000 7fd1a4219740 1 --2- 172.21.15.28:0/3899743306 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eff1783480 0x55eff178d610 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fd18c00e450 tx=0x7fd18c0095c0 comp rx=0 tx=0).stop 2024-05-12T22:15:45.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.711+0000 7fd191e00640 1 --2- 172.21.15.28:0/1309439606 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fd1800a2d80 0x7fd1800a5240 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x55eff1780010 tx=0x7fd184012040 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:15:45.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.711+0000 7fd1a4219740 1 -- 172.21.15.28:0/3899743306 shutdown_connections 2024-05-12T22:15:45.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.711+0000 7fd1a4219740 1 --2- 172.21.15.28:0/3899743306 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55eff178db50 0x55eff178ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:45.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.711+0000 7fd1a4219740 1 --2- 172.21.15.28:0/3899743306 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eff1783480 0x55eff178d610 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:45.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.711+0000 7fd1a4219740 1 --2- 172.21.15.28:0/3899743306 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55eff1782b60 0x55eff1782f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:45.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.711+0000 7fd1a4219740 1 -- 172.21.15.28:0/3899743306 >> 172.21.15.28:0/3899743306 conn(0x55eff177ee80 msgr2=0x55eff177f260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:15:45.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.711+0000 7fd1a4219740 1 -- 172.21.15.28:0/3899743306 shutdown_connections 2024-05-12T22:15:45.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.712+0000 7fd1a4219740 1 -- 172.21.15.28:0/3899743306 wait complete. 2024-05-12T22:15:45.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.712+0000 7fd1a4219740 1 -- start start 2024-05-12T22:15:45.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.712+0000 7fd1a4219740 1 Processor -- start 2024-05-12T22:15:45.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.713+0000 7fd1a4219740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eff1782b60 0x55eff178aa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:45.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.713+0000 7fd1a4219740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55eff1783480 0x55eff178af50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:45.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.713+0000 7fd1a4219740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55eff178db50 0x55eff178b490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:45.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.713+0000 7fd1a4219740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55eff1790990 con 0x55eff1782b60 2024-05-12T22:15:45.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.713+0000 7fd1a4219740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x55eff16a2a70 con 0x55eff1783480 2024-05-12T22:15:45.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.713+0000 7fd1a4219740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55eff1790b10 con 0x55eff178db50 2024-05-12T22:15:45.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.713+0000 7fd1a4219740 10 client.0 _ll_register_callbacks cb 0x55eff1685ba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:15:45.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.713+0000 7fd19a800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eff1782b60 0x55eff178aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:45.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.713+0000 7fd19b200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55eff178db50 0x55eff178b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:45.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.713+0000 7fd19a800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eff1782b60 0x55eff178aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55906/0 (socket says 172.21.15.28:55906) 2024-05-12T22:15:45.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.713+0000 7fd19a800640 1 -- 172.21.15.28:0/3539677012 learned_addr learned my addr 172.21.15.28:0/3539677012 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:15:45.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.713+0000 7fd191e00640 1 --2- 172.21.15.28:0/3539677012 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55eff1783480 0x55eff178af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:45.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.713+0000 7fd19b200640 1 -- 172.21.15.28:0/3539677012 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55eff1783480 msgr2=0x55eff178af50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:15:45.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.713+0000 7fd19b200640 1 --2- 172.21.15.28:0/3539677012 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55eff1783480 0x55eff178af50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:45.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.713+0000 7fd19b200640 1 -- 172.21.15.28:0/3539677012 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eff1782b60 msgr2=0x55eff178aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:15:45.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.713+0000 7fd19b200640 1 --2- 172.21.15.28:0/3539677012 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eff1782b60 0x55eff178aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:45.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.713+0000 7fd19b200640 1 -- 172.21.15.28:0/3539677012 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eff14c57d0 con 0x55eff178db50 2024-05-12T22:15:45.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.713+0000 7fd19a800640 1 --2- 172.21.15.28:0/3539677012 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eff1782b60 0x55eff178aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:15:45.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.714+0000 7fd19b200640 1 --2- 172.21.15.28:0/3539677012 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55eff178db50 0x55eff178b490 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fd1940ed420 tx=0x7fd1940bc170 comp rx=0 tx=0).ready entity=mon.1 client_cookie=40d5770b79a05347 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:15:45.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.714+0000 7fd17fe00640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-12T22:15:45.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.714+0000 7fd17fe00640 1 -- 172.21.15.28:0/3539677012 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd194049040 con 0x55eff178db50 2024-05-12T22:15:45.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.714+0000 7fd17fe00640 1 -- 172.21.15.28:0/3539677012 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7fd1940c0120 con 0x55eff178db50 2024-05-12T22:15:45.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.714+0000 7fd1a4219740 1 -- 172.21.15.28:0/3539677012 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55eff16f5160 con 0x55eff178db50 2024-05-12T22:15:45.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.714+0000 7fd17fe00640 1 -- 172.21.15.28:0/3539677012 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fd1940dc2d0 con 0x55eff178db50 2024-05-12T22:15:45.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.714+0000 7fd1a4219740 10 client.26710 fetch_fsmap learned FSMap version 26 2024-05-12T22:15:45.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.714+0000 7fd1a4219740 1 -- 172.21.15.28:0/3539677012 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55eff1790990 con 0x55eff178db50 2024-05-12T22:15:45.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.715+0000 7fd17fe00640 1 -- 172.21.15.28:0/3539677012 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55eff16f5160 con 0x55eff178db50 2024-05-12T22:15:45.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.715+0000 7fd17fe00640 1 -- 172.21.15.28:0/3539677012 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fd1940c49d0 con 0x55eff178db50 2024-05-12T22:15:45.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.715+0000 7fd1a4219740 10 client.26710 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:15:45.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.715+0000 7fd1a4219740 10 client.26710 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:15:45.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.715+0000 7fd1a4219740 20 client.26710 populate_metadata read hostname 'smithi028' 2024-05-12T22:15:45.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.715+0000 7fd1a4219740 1 --2- 172.21.15.28:0/3539677012 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55eff179de00 0x55eff17a0280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:45.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.715+0000 7fd1a4219740 4 client.26710 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-12T22:15:45.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.715+0000 7fd1a4219740 1 -- 172.21.15.28:0/3539677012 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55eff16ef4b0 con 0x55eff179de00 2024-05-12T22:15:45.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.715+0000 7fd19a800640 1 --2- 172.21.15.28:0/3539677012 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55eff179de00 0x55eff17a0280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:45.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.716+0000 7fd19a800640 1 --2- 172.21.15.28:0/3539677012 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55eff179de00 0x55eff17a0280 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d793d83978655abd server_cookie=7d90d01466586c78 in_seq=0 out_seq=0 2024-05-12T22:15:45.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.716+0000 7fd17fe00640 10 client.26710 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:15:45.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.717+0000 7fd17fe00640 1 -- 172.21.15.28:0/3539677012 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55eff16ef4b0 con 0x55eff179de00 2024-05-12T22:15:45.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.717+0000 7fd17fe00640 10 client.26710 handle_command_reply: tid=0 2024-05-12T22:15:45.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.717+0000 7fd1a4219740 1 client.26710 shutdown 2024-05-12T22:15:45.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.717+0000 7fd1a4219740 1 -- 172.21.15.28:0/3539677012 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55eff178db50 msgr2=0x55eff178b490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:15:45.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.717+0000 7fd1a4219740 1 --2- 172.21.15.28:0/3539677012 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55eff178db50 0x55eff178b490 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fd1940ed420 tx=0x7fd1940bc170 comp rx=0 tx=0).stop 2024-05-12T22:15:45.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.717+0000 7fd1a4219740 1 -- 172.21.15.28:0/3539677012 shutdown_connections 2024-05-12T22:15:45.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.717+0000 7fd1a4219740 1 --2- 172.21.15.28:0/3539677012 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55eff179de00 0x55eff17a0280 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:45.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.717+0000 7fd1a4219740 1 --2- 172.21.15.28:0/3539677012 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55eff178db50 0x55eff178b490 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:45.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.717+0000 7fd1a4219740 1 --2- 172.21.15.28:0/3539677012 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55eff1783480 0x55eff178af50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:45.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.717+0000 7fd17fe00640 0 client.26710 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:15:45.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.717+0000 7fd1a4219740 1 --2- 172.21.15.28:0/3539677012 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eff1782b60 0x55eff178aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:45.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.717+0000 7fd1a4219740 1 -- 172.21.15.28:0/3539677012 >> 172.21.15.28:0/3539677012 conn(0x55eff177ee80 msgr2=0x55eff177e000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:15:45.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.718+0000 7fd1a4219740 1 -- 172.21.15.28:0/3539677012 shutdown_connections 2024-05-12T22:15:45.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.718+0000 7fd1a4219740 1 -- 172.21.15.28:0/3539677012 wait complete. 2024-05-12T22:15:45.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.718+0000 7fd1a4219740 20 client.26710 trim_cache size 0 max 16384 2024-05-12T22:15:45.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.733+0000 7fd1a4219740 1 Processor -- start 2024-05-12T22:15:45.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.733+0000 7fd1a4219740 1 -- start start 2024-05-12T22:15:45.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.733+0000 7fd1a4219740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55eff1782b60 0x55eff17998b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:45.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.733+0000 7fd1a4219740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55eff1783480 0x55eff17a5b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:45.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.733+0000 7fd1a4219740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eff178db50 0x55eff17a6090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:45.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.733+0000 7fd1a4219740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd19c1b6160 con 0x55eff178db50 2024-05-12T22:15:45.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.733+0000 7fd1a4219740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fd19c1b3a30 con 0x55eff1783480 2024-05-12T22:15:45.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.733+0000 7fd1a4219740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd19c1b1880 con 0x55eff1782b60 2024-05-12T22:15:45.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.733+0000 7fd19b200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eff178db50 0x55eff17a6090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:45.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.733+0000 7fd19a800640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55eff1782b60 0x55eff17998b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:45.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.733+0000 7fd19b200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eff178db50 0x55eff17a6090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55908/0 (socket says 172.21.15.28:55908) 2024-05-12T22:15:45.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.733+0000 7fd19b200640 1 -- 172.21.15.28:0/2760012780 learned_addr learned my addr 172.21.15.28:0/2760012780 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:15:45.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.733+0000 7fd191e00640 1 --2- 172.21.15.28:0/2760012780 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55eff1783480 0x55eff17a5b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:45.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.734+0000 7fd19b200640 1 -- 172.21.15.28:0/2760012780 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55eff1782b60 msgr2=0x55eff17998b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:15:45.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.734+0000 7fd19b200640 1 --2- 172.21.15.28:0/2760012780 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55eff1782b60 0x55eff17998b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:45.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.734+0000 7fd19b200640 1 -- 172.21.15.28:0/2760012780 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55eff1783480 msgr2=0x55eff17a5b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:15:45.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.734+0000 7fd19b200640 1 --2- 172.21.15.28:0/2760012780 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55eff1783480 0x55eff17a5b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:45.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.734+0000 7fd19b200640 1 -- 172.21.15.28:0/2760012780 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eff17832b0 con 0x55eff178db50 2024-05-12T22:15:45.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.734+0000 7fd19a800640 1 --2- 172.21.15.28:0/2760012780 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55eff1782b60 0x55eff17998b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:15:45.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.734+0000 7fd191e00640 1 --2- 172.21.15.28:0/2760012780 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55eff1783480 0x55eff17a5b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:15:45.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.734+0000 7fd19b200640 1 --2- 172.21.15.28:0/2760012780 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eff178db50 0x55eff17a6090 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fd1940b8c70 tx=0x7fd19402a3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2a63effb64dfa71 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:15:45.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.734+0000 7fd18aa00640 1 -- 172.21.15.28:0/2760012780 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd194049070 con 0x55eff178db50 2024-05-12T22:15:45.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.735+0000 7fd18aa00640 1 -- 172.21.15.28:0/2760012780 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd1940c4e60 con 0x55eff178db50 2024-05-12T22:15:45.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.735+0000 7fd18aa00640 1 -- 172.21.15.28:0/2760012780 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1940bb6f0 con 0x55eff178db50 2024-05-12T22:15:45.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.735+0000 7fd1a4219740 1 -- 172.21.15.28:0/2760012780 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eff178db50 msgr2=0x55eff17a6090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:15:45.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.735+0000 7fd1a4219740 1 --2- 172.21.15.28:0/2760012780 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eff178db50 0x55eff17a6090 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fd1940b8c70 tx=0x7fd19402a3d0 comp rx=0 tx=0).stop 2024-05-12T22:15:45.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.735+0000 7fd1a4219740 1 -- 172.21.15.28:0/2760012780 shutdown_connections 2024-05-12T22:15:45.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.735+0000 7fd1a4219740 1 --2- 172.21.15.28:0/2760012780 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eff178db50 0x55eff17a6090 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:45.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.735+0000 7fd1a4219740 1 --2- 172.21.15.28:0/2760012780 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55eff1783480 0x55eff17a5b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:45.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.735+0000 7fd1a4219740 1 --2- 172.21.15.28:0/2760012780 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55eff1782b60 0x55eff17998b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:45.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.735+0000 7fd1a4219740 1 -- 172.21.15.28:0/2760012780 >> 172.21.15.28:0/2760012780 conn(0x55eff177ee80 msgr2=0x55eff177f260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:15:45.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.735+0000 7fd1a4219740 1 -- 172.21.15.28:0/2760012780 shutdown_connections 2024-05-12T22:15:45.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.735+0000 7fd1a4219740 1 -- 172.21.15.28:0/2760012780 wait complete. 2024-05-12T22:15:45.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.736+0000 7fd1a4219740 1 -- start start 2024-05-12T22:15:45.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.736+0000 7fd1a4219740 1 Processor -- start 2024-05-12T22:15:45.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.737+0000 7fd1a4219740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55eff1782b60 0x55eff1796070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:45.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.737+0000 7fd1a4219740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55eff1783480 0x55eff17913e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:45.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.737+0000 7fd1a4219740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eff178db50 0x55eff1791920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:45.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.737+0000 7fd1a4219740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd19c1b1ef0 con 0x55eff178db50 2024-05-12T22:15:45.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.737+0000 7fd1a4219740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fd19c1b3360 con 0x55eff1783480 2024-05-12T22:15:45.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.737+0000 7fd1a4219740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd19c1b25c0 con 0x55eff1782b60 2024-05-12T22:15:45.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.737+0000 7fd1a4219740 10 client.0 _ll_register_callbacks cb 0x7fd1940b8540 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:15:45.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.737+0000 7fd191e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55eff1783480 0x55eff17913e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:45.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.737+0000 7fd19b200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eff178db50 0x55eff1791920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:45.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.737+0000 7fd191e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55eff1783480 0x55eff17913e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:45798/0 (socket says 172.21.15.28:45798) 2024-05-12T22:15:45.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.737+0000 7fd191e00640 1 -- 172.21.15.28:0/2640251843 learned_addr learned my addr 172.21.15.28:0/2640251843 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:15:45.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.737+0000 7fd19a800640 1 --2- 172.21.15.28:0/2640251843 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55eff1782b60 0x55eff1796070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:45.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.737+0000 7fd191e00640 1 -- 172.21.15.28:0/2640251843 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55eff1782b60 msgr2=0x55eff1796070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:15:45.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.737+0000 7fd191e00640 1 --2- 172.21.15.28:0/2640251843 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55eff1782b60 0x55eff1796070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:45.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.737+0000 7fd191e00640 1 -- 172.21.15.28:0/2640251843 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eff178db50 msgr2=0x55eff1791920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:15:45.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.737+0000 7fd191e00640 1 --2- 172.21.15.28:0/2640251843 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eff178db50 0x55eff1791920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:45.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.737+0000 7fd191e00640 1 -- 172.21.15.28:0/2640251843 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eff1792010 con 0x55eff1783480 2024-05-12T22:15:45.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.738+0000 7fd19a800640 1 --2- 172.21.15.28:0/2640251843 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55eff1782b60 0x55eff1796070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:15:45.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.738+0000 7fd19b200640 1 --2- 172.21.15.28:0/2640251843 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eff178db50 0x55eff1791920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:15:45.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.738+0000 7fd191e00640 1 --2- 172.21.15.28:0/2640251843 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55eff1783480 0x55eff17913e0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fd184012010 tx=0x7fd1840046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6a88488d888716ec server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:15:45.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.738+0000 7fd17fe00640 10 client.0 ms_handle_connect on v2:172.21.15.86:3300/0 2024-05-12T22:15:45.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.738+0000 7fd17fe00640 1 -- 172.21.15.28:0/2640251843 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd184003a40 con 0x55eff1783480 2024-05-12T22:15:45.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.738+0000 7fd1a4219740 1 -- 172.21.15.28:0/2640251843 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd1940c4e60 con 0x55eff1783480 2024-05-12T22:15:45.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.738+0000 7fd17fe00640 1 -- 172.21.15.28:0/2640251843 <== mon.2 v2:172.21.15.86:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7fd18401c050 con 0x55eff1783480 2024-05-12T22:15:45.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.738+0000 7fd17fe00640 1 -- 172.21.15.28:0/2640251843 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fd184003be0 con 0x55eff1783480 2024-05-12T22:15:45.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.739+0000 7fd1a4219740 10 client.26762 fetch_fsmap learned FSMap version 26 2024-05-12T22:15:45.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.739+0000 7fd1a4219740 1 -- 172.21.15.28:0/2640251843 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55eff1804340 con 0x55eff1783480 2024-05-12T22:15:45.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.739+0000 7fd17fe00640 1 -- 172.21.15.28:0/2640251843 <== mon.2 v2:172.21.15.86:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd1940c4e60 con 0x55eff1783480 2024-05-12T22:15:45.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.739+0000 7fd17fe00640 1 -- 172.21.15.28:0/2640251843 <== mon.2 v2:172.21.15.86:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fd18401e040 con 0x55eff1783480 2024-05-12T22:15:45.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.739+0000 7fd1a4219740 10 client.26762 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:15:45.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.739+0000 7fd1a4219740 10 client.26762 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:15:45.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.739+0000 7fd1a4219740 20 client.26762 populate_metadata read hostname 'smithi028' 2024-05-12T22:15:45.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.739+0000 7fd1a4219740 1 --2- 172.21.15.28:0/2640251843 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55eff1804b20 0x55eff1806f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:45.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.739+0000 7fd1a4219740 4 client.26762 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-12T22:15:45.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.739+0000 7fd1a4219740 1 -- 172.21.15.28:0/2640251843 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55eff166c6e0 con 0x55eff1804b20 2024-05-12T22:15:45.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.740+0000 7fd19a800640 1 --2- 172.21.15.28:0/2640251843 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55eff1804b20 0x55eff1806f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:45.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.740+0000 7fd19a800640 1 --2- 172.21.15.28:0/2640251843 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55eff1804b20 0x55eff1806f40 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2ce656e9590f4d5a server_cookie=e5f792eaa9c2354f in_seq=0 out_seq=0 2024-05-12T22:15:45.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.740+0000 7fd17fe00640 10 client.26762 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:15:45.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.740+0000 7fd17fe00640 1 -- 172.21.15.28:0/2640251843 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55eff166c6e0 con 0x55eff1804b20 2024-05-12T22:15:45.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.740+0000 7fd17fe00640 10 client.26762 handle_command_reply: tid=0 2024-05-12T22:15:45.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.740+0000 7fd1a4219740 1 client.26762 shutdown 2024-05-12T22:15:45.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.741+0000 7fd1a4219740 1 -- 172.21.15.28:0/2640251843 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55eff1783480 msgr2=0x55eff17913e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:15:45.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.741+0000 7fd1a4219740 1 --2- 172.21.15.28:0/2640251843 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55eff1783480 0x55eff17913e0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fd184012010 tx=0x7fd1840046c0 comp rx=0 tx=0).stop 2024-05-12T22:15:45.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.741+0000 7fd1a4219740 1 -- 172.21.15.28:0/2640251843 shutdown_connections 2024-05-12T22:15:45.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.741+0000 7fd1a4219740 1 --2- 172.21.15.28:0/2640251843 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55eff1804b20 0x55eff1806f40 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:45.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.741+0000 7fd1a4219740 1 --2- 172.21.15.28:0/2640251843 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eff178db50 0x55eff1791920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:45.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.741+0000 7fd1a4219740 1 --2- 172.21.15.28:0/2640251843 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55eff1783480 0x55eff17913e0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:45.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.741+0000 7fd17fe00640 0 client.26762 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:15:45.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.741+0000 7fd1a4219740 1 --2- 172.21.15.28:0/2640251843 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55eff1782b60 0x55eff1796070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:45.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.741+0000 7fd1a4219740 1 -- 172.21.15.28:0/2640251843 >> 172.21.15.28:0/2640251843 conn(0x55eff177ee80 msgr2=0x55eff177d770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:15:45.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.741+0000 7fd1a4219740 1 -- 172.21.15.28:0/2640251843 shutdown_connections 2024-05-12T22:15:45.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.741+0000 7fd1a4219740 1 -- 172.21.15.28:0/2640251843 wait complete. 2024-05-12T22:15:45.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.741+0000 7fd1a4219740 20 client.26762 trim_cache size 0 max 16384 2024-05-12T22:15:45.749 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-05-12T22:15:45.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.742+0000 7fd1a0a00640 1 -- 172.21.15.28:0/1309439606 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fd1800a2d80 msgr2=0x7fd1800a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:15:45.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.742+0000 7fd1a0a00640 1 --2- 172.21.15.28:0/1309439606 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fd1800a2d80 0x7fd1800a5240 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x55eff1780010 tx=0x7fd184012040 comp rx=0 tx=0).stop 2024-05-12T22:15:45.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.742+0000 7fd1a0a00640 1 -- 172.21.15.28:0/1309439606 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd19c10ddf0 msgr2=0x7fd19c1a2e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:15:45.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.742+0000 7fd1a0a00640 1 --2- 172.21.15.28:0/1309439606 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd19c10ddf0 0x7fd19c1a2e30 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fd194009980 tx=0x7fd194030650 comp rx=0 tx=0).stop 2024-05-12T22:15:45.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.742+0000 7fd1a0a00640 1 -- 172.21.15.28:0/1309439606 shutdown_connections 2024-05-12T22:15:45.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.742+0000 7fd1a0a00640 1 --2- 172.21.15.28:0/1309439606 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fd1800a2d80 0x7fd1800a5240 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:45.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.742+0000 7fd1a0a00640 1 --2- 172.21.15.28:0/1309439606 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd19c10ddf0 0x7fd19c1a2e30 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:45.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.742+0000 7fd1a0a00640 1 --2- 172.21.15.28:0/1309439606 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd19c108770 0x7fd19c19eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:45.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.742+0000 7fd1a0a00640 1 --2- 172.21.15.28:0/1309439606 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd19c107dd0 0x7fd19c19e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:45.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.742+0000 7fd1a0a00640 1 -- 172.21.15.28:0/1309439606 >> 172.21.15.28:0/1309439606 conn(0x7fd19c077650 msgr2=0x7fd19c078690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:15:45.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.742+0000 7fd1a0a00640 1 -- 172.21.15.28:0/1309439606 shutdown_connections 2024-05-12T22:15:45.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:45.742+0000 7fd1a0a00640 1 -- 172.21.15.28:0/1309439606 wait complete. 2024-05-12T22:15:46.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:15:46 smithi186 ceph-mon[34785]: pgmap v206: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-12T22:15:46.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:15:46 smithi086 ceph-mon[34297]: pgmap v206: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-12T22:15:46.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:15:46 smithi028 ceph-mon[29847]: pgmap v206: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-12T22:15:46.762 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-12T22:15:46.762 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-12T22:15:46.762 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-05-12T22:15:46.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:46.992+0000 7f719f200640 1 -- 172.21.15.28:0/3556442841 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f71a01020a0 msgr2=0x7f71a010e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:15:46.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:46.992+0000 7f719f200640 1 --2- 172.21.15.28:0/3556442841 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f71a01020a0 0x7f71a010e5b0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f7198009cd0 tx=0x7f719802a8f0 comp rx=0 tx=0).stop 2024-05-12T22:15:46.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:46.993+0000 7f719f200640 1 -- 172.21.15.28:0/3556442841 shutdown_connections 2024-05-12T22:15:46.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:46.993+0000 7f719f200640 1 --2- 172.21.15.28:0/3556442841 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f71a01020a0 0x7f71a010e5b0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:46.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:46.993+0000 7f719f200640 1 --2- 172.21.15.28:0/3556442841 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71a01016e0 0x7f71a0101b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:46.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:46.993+0000 7f719f200640 1 --2- 172.21.15.28:0/3556442841 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f71a01076f0 0x7f71a0107ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:46.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:46.993+0000 7f719f200640 1 -- 172.21.15.28:0/3556442841 >> 172.21.15.28:0/3556442841 conn(0x7f71a00fd1d0 msgr2=0x7f71a00ff5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:15:46.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:46.993+0000 7f719f200640 1 -- 172.21.15.28:0/3556442841 shutdown_connections 2024-05-12T22:15:46.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:46.993+0000 7f719f200640 1 -- 172.21.15.28:0/3556442841 wait complete. 2024-05-12T22:15:46.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:46.994+0000 7f719f200640 1 Processor -- start 2024-05-12T22:15:46.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:46.994+0000 7f719f200640 1 -- start start 2024-05-12T22:15:46.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:46.994+0000 7f719f200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f71a01016e0 0x7f71a019a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:46.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:46.994+0000 7f719f200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f71a01020a0 0x7f71a019a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:46.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:46.994+0000 7f719f200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71a01076f0 0x7f71a019ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:46.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:46.994+0000 7f719f200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f71a0111b50 con 0x7f71a01076f0 2024-05-12T22:15:46.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:46.994+0000 7f719f200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f71a01119d0 con 0x7f71a01020a0 2024-05-12T22:15:46.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:46.994+0000 7f719f200640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f71a0111cd0 con 0x7f71a01016e0 2024-05-12T22:15:46.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:46.995+0000 7f719e800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71a01076f0 0x7f71a019ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:46.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:46.995+0000 7f7197e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f71a01020a0 0x7f71a019a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:46.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:46.995+0000 7f7197e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f71a01020a0 0x7f71a019a6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:35054/0 (socket says 172.21.15.28:35054) 2024-05-12T22:15:46.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:46.995+0000 7f7197e00640 1 -- 172.21.15.28:0/2546853564 learned_addr learned my addr 172.21.15.28:0/2546853564 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:15:46.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:46.995+0000 7f719de00640 1 --2- 172.21.15.28:0/2546853564 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f71a01016e0 0x7f71a019a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:46.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:46.995+0000 7f7197e00640 1 -- 172.21.15.28:0/2546853564 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f71a01016e0 msgr2=0x7f71a019a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:15:46.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:46.995+0000 7f7197e00640 1 --2- 172.21.15.28:0/2546853564 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f71a01016e0 0x7f71a019a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:46.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:46.995+0000 7f7197e00640 1 -- 172.21.15.28:0/2546853564 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71a01076f0 msgr2=0x7f71a019ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:15:46.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:46.995+0000 7f7197e00640 1 --2- 172.21.15.28:0/2546853564 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71a01076f0 0x7f71a019ea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:46.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:46.995+0000 7f7197e00640 1 -- 172.21.15.28:0/2546853564 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71a019f1a0 con 0x7f71a01020a0 2024-05-12T22:15:47.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:46.995+0000 7f7197e00640 1 --2- 172.21.15.28:0/2546853564 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f71a01020a0 0x7f71a019a6f0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f718800e9f0 tx=0x7f718800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:15:47.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:46.996+0000 7f7196a00640 1 -- 172.21.15.28:0/2546853564 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f718800cd80 con 0x7f71a01020a0 2024-05-12T22:15:47.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:46.996+0000 7f719f200640 1 -- 172.21.15.28:0/2546853564 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71a019f490 con 0x7f71a01020a0 2024-05-12T22:15:47.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:46.996+0000 7f7196a00640 1 -- 172.21.15.28:0/2546853564 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7188010910 con 0x7f71a01020a0 2024-05-12T22:15:47.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:46.996+0000 7f7196a00640 1 -- 172.21.15.28:0/2546853564 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71880188f0 con 0x7f71a01020a0 2024-05-12T22:15:47.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:46.996+0000 7f719f200640 1 -- 172.21.15.28:0/2546853564 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71a01a6d20 con 0x7f71a01020a0 2024-05-12T22:15:47.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:46.999+0000 7f71a7e34740 1 Processor -- start 2024-05-12T22:15:47.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:46.999+0000 7f71a7e34740 1 -- start start 2024-05-12T22:15:47.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:46.999+0000 7f7196a00640 1 -- 172.21.15.28:0/2546853564 <== mon.2 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7f7188010ac0 con 0x7f71a01020a0 2024-05-12T22:15:47.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:46.999+0000 7f71a7e34740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e16c628500 0x55e16c6288e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:47.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:46.999+0000 7f71a7e34740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e16c628eb0 0x55e16c632f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:47.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.000+0000 7f71a7e34740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e16c633520 0x55e16c635970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:47.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.000+0000 7f71a7e34740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e16c58c7c0 con 0x55e16c628eb0 2024-05-12T22:15:47.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.000+0000 7f71a7e34740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x55e16c534790 con 0x55e16c628500 2024-05-12T22:15:47.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.000+0000 7f71a7e34740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55e16c588780 con 0x55e16c633520 2024-05-12T22:15:47.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.000+0000 7f719de00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e16c628eb0 0x55e16c632f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:47.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.000+0000 7f719de00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e16c628eb0 0x55e16c632f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42462/0 (socket says 172.21.15.28:42462) 2024-05-12T22:15:47.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.000+0000 7f719de00640 1 -- 172.21.15.28:0/3686671144 learned_addr learned my addr 172.21.15.28:0/3686671144 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:15:47.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.000+0000 7f7197e00640 1 --2- 172.21.15.28:0/3686671144 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e16c633520 0x55e16c635970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:47.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.000+0000 7f719e800640 1 --2- 172.21.15.28:0/3686671144 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e16c628500 0x55e16c6288e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:47.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.000+0000 7f719de00640 1 -- 172.21.15.28:0/3686671144 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e16c633520 msgr2=0x55e16c635970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:15:47.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.000+0000 7f719de00640 1 --2- 172.21.15.28:0/3686671144 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e16c633520 0x55e16c635970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:47.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.000+0000 7f719de00640 1 -- 172.21.15.28:0/3686671144 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e16c628500 msgr2=0x55e16c6288e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:15:47.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.000+0000 7f719de00640 1 --2- 172.21.15.28:0/3686671144 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e16c628500 0x55e16c6288e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:47.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.000+0000 7f719de00640 1 -- 172.21.15.28:0/3686671144 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e16c5546a0 con 0x55e16c628eb0 2024-05-12T22:15:47.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.001+0000 7f719de00640 1 --2- 172.21.15.28:0/3686671144 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e16c628eb0 0x55e16c632f50 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f7190009c70 tx=0x7f7190011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5809a268613fd0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:15:47.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.001+0000 7f718f400640 1 -- 172.21.15.28:0/3686671144 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7190004230 con 0x55e16c628eb0 2024-05-12T22:15:47.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.001+0000 7f718f400640 1 -- 172.21.15.28:0/3686671144 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f71900043d0 con 0x55e16c628eb0 2024-05-12T22:15:47.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.001+0000 7f718f400640 1 -- 172.21.15.28:0/3686671144 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7190019820 con 0x55e16c628eb0 2024-05-12T22:15:47.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.001+0000 7f7196a00640 1 --2- 172.21.15.28:0/2546853564 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f71840a2d80 0x7f71840a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:47.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.002+0000 7f7196a00640 1 -- 172.21.15.28:0/2546853564 <== mon.2 v2:172.21.15.86:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f71880cf7b0 con 0x7f71a01020a0 2024-05-12T22:15:47.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.002+0000 7f719e800640 1 --2- 172.21.15.28:0/2546853564 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f71840a2d80 0x7f71840a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:47.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.002+0000 7f71a7e34740 1 -- 172.21.15.28:0/3686671144 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e16c628eb0 msgr2=0x55e16c632f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:15:47.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.002+0000 7f71a7e34740 1 --2- 172.21.15.28:0/3686671144 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e16c628eb0 0x55e16c632f50 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f7190009c70 tx=0x7f7190011810 comp rx=0 tx=0).stop 2024-05-12T22:15:47.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.002+0000 7f71a7e34740 1 -- 172.21.15.28:0/3686671144 shutdown_connections 2024-05-12T22:15:47.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.002+0000 7f71a7e34740 1 --2- 172.21.15.28:0/3686671144 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e16c633520 0x55e16c635970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:47.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.002+0000 7f71a7e34740 1 --2- 172.21.15.28:0/3686671144 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e16c628eb0 0x55e16c632f50 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:47.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.002+0000 7f71a7e34740 1 --2- 172.21.15.28:0/3686671144 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e16c628500 0x55e16c6288e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:47.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.002+0000 7f71a7e34740 1 -- 172.21.15.28:0/3686671144 >> 172.21.15.28:0/3686671144 conn(0x55e16c6245b0 msgr2=0x55e16c624990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:15:47.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.002+0000 7f71a7e34740 1 -- 172.21.15.28:0/3686671144 shutdown_connections 2024-05-12T22:15:47.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.002+0000 7f71a7e34740 1 -- 172.21.15.28:0/3686671144 wait complete. 2024-05-12T22:15:47.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.002+0000 7f719e800640 1 --2- 172.21.15.28:0/2546853564 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f71840a2d80 0x7f71840a5240 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f7198004af0 tx=0x7f7198005d80 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:15:47.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.003+0000 7f71a7e34740 1 -- start start 2024-05-12T22:15:47.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.003+0000 7f71a7e34740 1 Processor -- start 2024-05-12T22:15:47.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.003+0000 7f71a7e34740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e16c628500 0x55e16c630390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:47.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.003+0000 7f71a7e34740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e16c628eb0 0x55e16c6308d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:47.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.003+0000 7f71a7e34740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e16c633520 0x55e16c630e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:47.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.003+0000 7f71a7e34740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e16c636530 con 0x55e16c628eb0 2024-05-12T22:15:47.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.003+0000 7f71a7e34740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x55e16c547a90 con 0x55e16c628500 2024-05-12T22:15:47.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.003+0000 7f71a7e34740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55e16c6366b0 con 0x55e16c633520 2024-05-12T22:15:47.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.003+0000 7f71a7e34740 10 client.0 _ll_register_callbacks cb 0x55e16c582af0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:15:47.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.004+0000 7f7197e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e16c628eb0 0x55e16c6308d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:47.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.004+0000 7f719e800640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e16c633520 0x55e16c630e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:47.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.004+0000 7f7197e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e16c628eb0 0x55e16c6308d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42470/0 (socket says 172.21.15.28:42470) 2024-05-12T22:15:47.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.004+0000 7f7197e00640 1 -- 172.21.15.28:0/3756353067 learned_addr learned my addr 172.21.15.28:0/3756353067 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:15:47.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.004+0000 7f719de00640 1 --2- 172.21.15.28:0/3756353067 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e16c628500 0x55e16c630390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:47.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.004+0000 7f7197e00640 1 -- 172.21.15.28:0/3756353067 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e16c633520 msgr2=0x55e16c630e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:15:47.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.004+0000 7f7197e00640 1 --2- 172.21.15.28:0/3756353067 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e16c633520 0x55e16c630e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:47.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.004+0000 7f7197e00640 1 -- 172.21.15.28:0/3756353067 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e16c628500 msgr2=0x55e16c630390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:15:47.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.004+0000 7f7197e00640 1 --2- 172.21.15.28:0/3756353067 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e16c628500 0x55e16c630390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:47.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.004+0000 7f7197e00640 1 -- 172.21.15.28:0/3756353067 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e16c3687e0 con 0x55e16c628eb0 2024-05-12T22:15:47.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.004+0000 7f719de00640 1 --2- 172.21.15.28:0/3756353067 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e16c628500 0x55e16c630390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:15:47.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.004+0000 7f7197e00640 1 --2- 172.21.15.28:0/3756353067 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e16c628eb0 0x55e16c6308d0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f718800e9c0 tx=0x7f718809b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be258cc62de7151d server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:15:47.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.004+0000 7f7183e00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-12T22:15:47.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.005+0000 7f7183e00640 1 -- 172.21.15.28:0/3756353067 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71880cd360 con 0x55e16c628eb0 2024-05-12T22:15:47.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.005+0000 7f7183e00640 1 -- 172.21.15.28:0/3756353067 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f71880cfda0 con 0x55e16c628eb0 2024-05-12T22:15:47.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.005+0000 7f71a7e34740 1 -- 172.21.15.28:0/3756353067 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f71900043d0 con 0x55e16c628eb0 2024-05-12T22:15:47.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.005+0000 7f7183e00640 1 -- 172.21.15.28:0/3756353067 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f71880a4910 con 0x55e16c628eb0 2024-05-12T22:15:47.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.005+0000 7f71a7e34740 10 client.16647 fetch_fsmap learned FSMap version 26 2024-05-12T22:15:47.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.005+0000 7f71a7e34740 1 -- 172.21.15.28:0/3756353067 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55e16c3c8f70 con 0x55e16c628eb0 2024-05-12T22:15:47.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.005+0000 7f7183e00640 1 -- 172.21.15.28:0/3756353067 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f71900043d0 con 0x55e16c628eb0 2024-05-12T22:15:47.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.005+0000 7f7183e00640 1 -- 172.21.15.28:0/3756353067 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f71880d1ce0 con 0x55e16c628eb0 2024-05-12T22:15:47.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.006+0000 7f71a7e34740 10 client.16647 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:15:47.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.006+0000 7f71a7e34740 10 client.16647 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:15:47.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.006+0000 7f71a7e34740 20 client.16647 populate_metadata read hostname 'smithi028' 2024-05-12T22:15:47.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.006+0000 7f71a7e34740 1 --2- 172.21.15.28:0/3756353067 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55e16c6437a0 0x55e16c645bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:47.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.006+0000 7f71a7e34740 4 client.16647 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-12T22:15:47.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.006+0000 7f71a7e34740 1 -- 172.21.15.28:0/3756353067 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e16c5a03a0 con 0x55e16c6437a0 2024-05-12T22:15:47.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.006+0000 7f719de00640 1 --2- 172.21.15.28:0/3756353067 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55e16c6437a0 0x55e16c645bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:47.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.007+0000 7f719de00640 1 --2- 172.21.15.28:0/3756353067 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55e16c6437a0 0x55e16c645bf0 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b5ba4e5d5df74720 server_cookie=3ce87a2211b3b740 in_seq=0 out_seq=0 2024-05-12T22:15:47.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.007+0000 7f7183e00640 10 client.16647 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:15:47.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.007+0000 7f7183e00640 1 -- 172.21.15.28:0/3756353067 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e16c5a03a0 con 0x55e16c6437a0 2024-05-12T22:15:47.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.007+0000 7f7183e00640 10 client.16647 handle_command_reply: tid=0 2024-05-12T22:15:47.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.008+0000 7f71a7e34740 1 client.16647 shutdown 2024-05-12T22:15:47.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.008+0000 7f71a7e34740 1 -- 172.21.15.28:0/3756353067 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e16c628eb0 msgr2=0x55e16c6308d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:15:47.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.008+0000 7f71a7e34740 1 --2- 172.21.15.28:0/3756353067 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e16c628eb0 0x55e16c6308d0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f718800e9c0 tx=0x7f718809b7a0 comp rx=0 tx=0).stop 2024-05-12T22:15:47.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.008+0000 7f71a7e34740 1 -- 172.21.15.28:0/3756353067 shutdown_connections 2024-05-12T22:15:47.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.008+0000 7f71a7e34740 1 --2- 172.21.15.28:0/3756353067 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55e16c6437a0 0x55e16c645bf0 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:47.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.008+0000 7f71a7e34740 1 --2- 172.21.15.28:0/3756353067 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e16c633520 0x55e16c630e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:47.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.008+0000 7f71a7e34740 1 --2- 172.21.15.28:0/3756353067 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e16c628eb0 0x55e16c6308d0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:47.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.008+0000 7f7183e00640 0 client.16647 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:15:47.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.008+0000 7f71a7e34740 1 --2- 172.21.15.28:0/3756353067 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e16c628500 0x55e16c630390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:47.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.008+0000 7f71a7e34740 1 -- 172.21.15.28:0/3756353067 >> 172.21.15.28:0/3756353067 conn(0x55e16c6245b0 msgr2=0x55e16c62a630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:15:47.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.008+0000 7f71a7e34740 1 -- 172.21.15.28:0/3756353067 shutdown_connections 2024-05-12T22:15:47.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.008+0000 7f71a7e34740 1 -- 172.21.15.28:0/3756353067 wait complete. 2024-05-12T22:15:47.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.008+0000 7f71a7e34740 20 client.16647 trim_cache size 0 max 16384 2024-05-12T22:15:47.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.027+0000 7f71a7e34740 1 Processor -- start 2024-05-12T22:15:47.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.028+0000 7f71a7e34740 1 -- start start 2024-05-12T22:15:47.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.028+0000 7f71a7e34740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e16c628500 0x55e16c62f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:47.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.028+0000 7f71a7e34740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e16c628eb0 0x55e16c63f450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:47.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.028+0000 7f71a7e34740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e16c633520 0x55e16c62abc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:47.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.028+0000 7f71a7e34740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f71a01b1dc0 con 0x55e16c628500 2024-05-12T22:15:47.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.028+0000 7f71a7e34740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f71a01af690 con 0x55e16c633520 2024-05-12T22:15:47.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.028+0000 7f71a7e34740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f71a01ad4e0 con 0x55e16c628eb0 2024-05-12T22:15:47.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.028+0000 7f719de00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e16c628500 0x55e16c62f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:47.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.028+0000 7f719de00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e16c628500 0x55e16c62f2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42480/0 (socket says 172.21.15.28:42480) 2024-05-12T22:15:47.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.028+0000 7f719de00640 1 -- 172.21.15.28:0/3058003108 learned_addr learned my addr 172.21.15.28:0/3058003108 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:15:47.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.028+0000 7f7197e00640 1 --2- 172.21.15.28:0/3058003108 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e16c628eb0 0x55e16c63f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:47.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.028+0000 7f719e800640 1 --2- 172.21.15.28:0/3058003108 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e16c633520 0x55e16c62abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:47.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.029+0000 7f719de00640 1 -- 172.21.15.28:0/3058003108 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e16c628eb0 msgr2=0x55e16c63f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:15:47.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.029+0000 7f719de00640 1 --2- 172.21.15.28:0/3058003108 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e16c628eb0 0x55e16c63f450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:47.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.029+0000 7f719de00640 1 -- 172.21.15.28:0/3058003108 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e16c633520 msgr2=0x55e16c62abc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:15:47.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.029+0000 7f719de00640 1 --2- 172.21.15.28:0/3058003108 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e16c633520 0x55e16c62abc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:47.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.029+0000 7f719de00640 1 -- 172.21.15.28:0/3058003108 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e16c260170 con 0x55e16c628500 2024-05-12T22:15:47.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.029+0000 7f719de00640 1 --2- 172.21.15.28:0/3058003108 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e16c628500 0x55e16c62f2d0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f7190009bc0 tx=0x7f7190003f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35310c65f9868158 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:15:47.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.029+0000 7f718ea00640 1 -- 172.21.15.28:0/3058003108 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71900042f0 con 0x55e16c628500 2024-05-12T22:15:47.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.029+0000 7f718ea00640 1 -- 172.21.15.28:0/3058003108 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7190005520 con 0x55e16c628500 2024-05-12T22:15:47.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.029+0000 7f718ea00640 1 -- 172.21.15.28:0/3058003108 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7190003bb0 con 0x55e16c628500 2024-05-12T22:15:47.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.030+0000 7f71a7e34740 1 -- 172.21.15.28:0/3058003108 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e16c628500 msgr2=0x55e16c62f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:15:47.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.030+0000 7f71a7e34740 1 --2- 172.21.15.28:0/3058003108 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e16c628500 0x55e16c62f2d0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f7190009bc0 tx=0x7f7190003f40 comp rx=0 tx=0).stop 2024-05-12T22:15:47.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.030+0000 7f71a7e34740 1 -- 172.21.15.28:0/3058003108 shutdown_connections 2024-05-12T22:15:47.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.030+0000 7f71a7e34740 1 --2- 172.21.15.28:0/3058003108 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e16c633520 0x55e16c62abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:47.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.030+0000 7f71a7e34740 1 --2- 172.21.15.28:0/3058003108 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e16c628eb0 0x55e16c63f450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:47.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.030+0000 7f71a7e34740 1 --2- 172.21.15.28:0/3058003108 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e16c628500 0x55e16c62f2d0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:47.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.030+0000 7f71a7e34740 1 -- 172.21.15.28:0/3058003108 >> 172.21.15.28:0/3058003108 conn(0x55e16c6245b0 msgr2=0x55e16c630be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:15:47.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.030+0000 7f71a7e34740 1 -- 172.21.15.28:0/3058003108 shutdown_connections 2024-05-12T22:15:47.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.030+0000 7f71a7e34740 1 -- 172.21.15.28:0/3058003108 wait complete. 2024-05-12T22:15:47.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.031+0000 7f71a7e34740 1 -- start start 2024-05-12T22:15:47.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.031+0000 7f71a7e34740 1 Processor -- start 2024-05-12T22:15:47.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.032+0000 7f71a7e34740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e16c628500 0x55e16c63be40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:47.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.032+0000 7f71a7e34740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e16c628eb0 0x55e16c62ce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:47.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.032+0000 7f71a7e34740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e16c633520 0x55e16c62d390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:47.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.032+0000 7f71a7e34740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f71a01adb50 con 0x55e16c628eb0 2024-05-12T22:15:47.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.032+0000 7f71a7e34740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f71a01aefc0 con 0x55e16c628500 2024-05-12T22:15:47.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.032+0000 7f71a7e34740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f71a01ae220 con 0x55e16c633520 2024-05-12T22:15:47.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.032+0000 7f71a7e34740 10 client.0 _ll_register_callbacks cb 0x7f718800cf20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:15:47.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.032+0000 7f7197e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e16c628eb0 0x55e16c62ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:47.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.032+0000 7f7197e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e16c628eb0 0x55e16c62ce10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42494/0 (socket says 172.21.15.28:42494) 2024-05-12T22:15:47.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.032+0000 7f7197e00640 1 -- 172.21.15.28:0/772392234 learned_addr learned my addr 172.21.15.28:0/772392234 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:15:47.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.032+0000 7f719e800640 1 --2- 172.21.15.28:0/772392234 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e16c633520 0x55e16c62d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:47.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.032+0000 7f719de00640 1 --2- 172.21.15.28:0/772392234 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e16c628500 0x55e16c63be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:47.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.032+0000 7f7197e00640 1 -- 172.21.15.28:0/772392234 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e16c633520 msgr2=0x55e16c62d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:15:47.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.032+0000 7f7197e00640 1 --2- 172.21.15.28:0/772392234 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e16c633520 0x55e16c62d390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:47.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.032+0000 7f7197e00640 1 -- 172.21.15.28:0/772392234 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e16c628500 msgr2=0x55e16c63be40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:15:47.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.032+0000 7f7197e00640 1 --2- 172.21.15.28:0/772392234 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e16c628500 0x55e16c63be40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:47.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.033+0000 7f7197e00640 1 -- 172.21.15.28:0/772392234 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e16c62dbc0 con 0x55e16c628eb0 2024-05-12T22:15:47.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.033+0000 7f719de00640 1 --2- 172.21.15.28:0/772392234 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e16c628500 0x55e16c63be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:15:47.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.033+0000 7f7197e00640 1 --2- 172.21.15.28:0/772392234 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e16c628eb0 0x55e16c62ce10 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f718809ba50 tx=0x7f7188097d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=541505390fc10a52 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:15:47.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.033+0000 7f7183e00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-12T22:15:47.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.033+0000 7f7183e00640 1 -- 172.21.15.28:0/772392234 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71880cd360 con 0x55e16c628eb0 2024-05-12T22:15:47.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.033+0000 7f7183e00640 1 -- 172.21.15.28:0/772392234 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7f71880cfb50 con 0x55e16c628eb0 2024-05-12T22:15:47.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.033+0000 7f71a7e34740 1 -- 172.21.15.28:0/772392234 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7190005520 con 0x55e16c628eb0 2024-05-12T22:15:47.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.033+0000 7f7183e00640 1 -- 172.21.15.28:0/772392234 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f718809b890 con 0x55e16c628eb0 2024-05-12T22:15:47.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.033+0000 7f71a7e34740 10 client.16653 fetch_fsmap learned FSMap version 26 2024-05-12T22:15:47.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.033+0000 7f71a7e34740 1 -- 172.21.15.28:0/772392234 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55e16c6a9de0 con 0x55e16c628eb0 2024-05-12T22:15:47.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.034+0000 7f7183e00640 1 -- 172.21.15.28:0/772392234 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7190005520 con 0x55e16c628eb0 2024-05-12T22:15:47.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.034+0000 7f7183e00640 1 -- 172.21.15.28:0/772392234 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f718809ac10 con 0x55e16c628eb0 2024-05-12T22:15:47.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.034+0000 7f71a7e34740 10 client.16653 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:15:47.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.034+0000 7f71a7e34740 10 client.16653 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:15:47.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.034+0000 7f71a7e34740 20 client.16653 populate_metadata read hostname 'smithi028' 2024-05-12T22:15:47.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.034+0000 7f71a7e34740 1 --2- 172.21.15.28:0/772392234 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55e16c6aa560 0x55e16c6ac980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:47.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.034+0000 7f71a7e34740 4 client.16653 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-12T22:15:47.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.034+0000 7f71a7e34740 1 -- 172.21.15.28:0/772392234 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55e16c599ef0 con 0x55e16c6aa560 2024-05-12T22:15:47.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.034+0000 7f719de00640 1 --2- 172.21.15.28:0/772392234 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55e16c6aa560 0x55e16c6ac980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:47.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.035+0000 7f719de00640 1 --2- 172.21.15.28:0/772392234 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55e16c6aa560 0x55e16c6ac980 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=30bad48f30c7539e server_cookie=a085a44bc9b6d22b in_seq=0 out_seq=0 2024-05-12T22:15:47.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.035+0000 7f7183e00640 10 client.16653 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:15:47.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.036+0000 7f7183e00640 1 -- 172.21.15.28:0/772392234 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55e16c599ef0 con 0x55e16c6aa560 2024-05-12T22:15:47.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.036+0000 7f7183e00640 10 client.16653 handle_command_reply: tid=0 2024-05-12T22:15:47.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.036+0000 7f71a7e34740 1 client.16653 shutdown 2024-05-12T22:15:47.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.036+0000 7f71a7e34740 1 -- 172.21.15.28:0/772392234 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e16c628eb0 msgr2=0x55e16c62ce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:15:47.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.036+0000 7f71a7e34740 1 --2- 172.21.15.28:0/772392234 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e16c628eb0 0x55e16c62ce10 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f718809ba50 tx=0x7f7188097d50 comp rx=0 tx=0).stop 2024-05-12T22:15:47.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.036+0000 7f71a7e34740 1 -- 172.21.15.28:0/772392234 shutdown_connections 2024-05-12T22:15:47.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.036+0000 7f71a7e34740 1 --2- 172.21.15.28:0/772392234 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55e16c6aa560 0x55e16c6ac980 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:47.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.036+0000 7f71a7e34740 1 --2- 172.21.15.28:0/772392234 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e16c633520 0x55e16c62d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:47.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.036+0000 7f71a7e34740 1 --2- 172.21.15.28:0/772392234 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e16c628eb0 0x55e16c62ce10 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:47.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.036+0000 7f71a7e34740 1 --2- 172.21.15.28:0/772392234 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e16c628500 0x55e16c63be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:47.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.036+0000 7f71a7e34740 1 -- 172.21.15.28:0/772392234 >> 172.21.15.28:0/772392234 conn(0x55e16c6245b0 msgr2=0x55e16c622ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:15:47.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.036+0000 7f7183e00640 0 client.16653 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:15:47.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.037+0000 7f71a7e34740 1 -- 172.21.15.28:0/772392234 shutdown_connections 2024-05-12T22:15:47.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.037+0000 7f71a7e34740 1 -- 172.21.15.28:0/772392234 wait complete. 2024-05-12T22:15:47.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.037+0000 7f71a7e34740 20 client.16653 trim_cache size 0 max 16384 2024-05-12T22:15:47.046 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-05-12T22:15:47.047 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-05-12T22:15:47.047 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "fd2e5837-fdab-43d4-bdf7-c150ecbcf6ff", 2024-05-12T22:15:47.047 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-05-12T22:15:47.047 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-05-12T22:15:47.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.037+0000 7f719f200640 1 -- 172.21.15.28:0/2546853564 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f71840a2d80 msgr2=0x7f71840a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:15:47.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.037+0000 7f719f200640 1 --2- 172.21.15.28:0/2546853564 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f71840a2d80 0x7f71840a5240 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f7198004af0 tx=0x7f7198005d80 comp rx=0 tx=0).stop 2024-05-12T22:15:47.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.037+0000 7f719f200640 1 -- 172.21.15.28:0/2546853564 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f71a01020a0 msgr2=0x7f71a019a6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:15:47.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.037+0000 7f719f200640 1 --2- 172.21.15.28:0/2546853564 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f71a01020a0 0x7f71a019a6f0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f718800e9f0 tx=0x7f718800eeb0 comp rx=0 tx=0).stop 2024-05-12T22:15:47.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.037+0000 7f719f200640 1 -- 172.21.15.28:0/2546853564 shutdown_connections 2024-05-12T22:15:47.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.037+0000 7f719f200640 1 --2- 172.21.15.28:0/2546853564 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f71840a2d80 0x7f71840a5240 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:47.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.037+0000 7f719f200640 1 --2- 172.21.15.28:0/2546853564 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71a01076f0 0x7f71a019ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:47.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.037+0000 7f719f200640 1 --2- 172.21.15.28:0/2546853564 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f71a01020a0 0x7f71a019a6f0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:47.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.037+0000 7f719f200640 1 --2- 172.21.15.28:0/2546853564 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f71a01016e0 0x7f71a019a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:47.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.037+0000 7f719f200640 1 -- 172.21.15.28:0/2546853564 >> 172.21.15.28:0/2546853564 conn(0x7f71a00fd1d0 msgr2=0x7f71a00fd6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:15:47.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.038+0000 7f719f200640 1 -- 172.21.15.28:0/2546853564 shutdown_connections 2024-05-12T22:15:47.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.038+0000 7f719f200640 1 -- 172.21.15.28:0/2546853564 wait complete. 2024-05-12T22:15:47.058 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-12T22:15:47.058 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-12T22:15:47.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.288+0000 7fae93200640 1 -- 172.21.15.28:0/288126875 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae941038e0 msgr2=0x7fae94103d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:15:47.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.288+0000 7fae93200640 1 --2- 172.21.15.28:0/288126875 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae941038e0 0x7fae94103d60 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fae80009900 tx=0x7fae8002a3f0 comp rx=0 tx=0).stop 2024-05-12T22:15:47.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.288+0000 7fae93200640 1 -- 172.21.15.28:0/288126875 shutdown_connections 2024-05-12T22:15:47.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.288+0000 7fae93200640 1 --2- 172.21.15.28:0/288126875 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fae941042a0 0x7fae941107b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:47.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.288+0000 7fae93200640 1 --2- 172.21.15.28:0/288126875 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae941038e0 0x7fae94103d60 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:47.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.288+0000 7fae93200640 1 --2- 172.21.15.28:0/288126875 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae941098f0 0x7fae94109cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:47.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.288+0000 7fae93200640 1 -- 172.21.15.28:0/288126875 >> 172.21.15.28:0/288126875 conn(0x7fae940ff370 msgr2=0x7fae94101790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:15:47.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.289+0000 7fae93200640 1 -- 172.21.15.28:0/288126875 shutdown_connections 2024-05-12T22:15:47.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.289+0000 7fae93200640 1 -- 172.21.15.28:0/288126875 wait complete. 2024-05-12T22:15:47.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.290+0000 7fae93200640 1 Processor -- start 2024-05-12T22:15:47.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.290+0000 7fae93200640 1 -- start start 2024-05-12T22:15:47.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.291+0000 7fae93200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae941038e0 0x7fae9419a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:47.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.291+0000 7fae93200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae941042a0 0x7fae9419a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:47.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.291+0000 7fae93200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fae941098f0 0x7fae9419ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:47.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.291+0000 7fae93200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fae94113d50 con 0x7fae941042a0 2024-05-12T22:15:47.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.291+0000 7fae93200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fae94113bd0 con 0x7fae941038e0 2024-05-12T22:15:47.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.291+0000 7fae93200640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fae94113ed0 con 0x7fae941098f0 2024-05-12T22:15:47.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.291+0000 7fae91e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae941038e0 0x7fae9419a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:47.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.291+0000 7fae91e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae941038e0 0x7fae9419a1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:35114/0 (socket says 172.21.15.28:35114) 2024-05-12T22:15:47.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.291+0000 7fae91e00640 1 -- 172.21.15.28:0/3618905448 learned_addr learned my addr 172.21.15.28:0/3618905448 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:15:47.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.291+0000 7fae92800640 1 --2- 172.21.15.28:0/3618905448 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fae941098f0 0x7fae9419ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:47.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.291+0000 7fae91400640 1 --2- 172.21.15.28:0/3618905448 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae941042a0 0x7fae9419a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:47.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.291+0000 7fae92800640 1 -- 172.21.15.28:0/3618905448 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae941038e0 msgr2=0x7fae9419a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:15:47.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.291+0000 7fae92800640 1 --2- 172.21.15.28:0/3618905448 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae941038e0 0x7fae9419a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:47.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.291+0000 7fae92800640 1 -- 172.21.15.28:0/3618905448 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae941042a0 msgr2=0x7fae9419a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:15:47.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.291+0000 7fae92800640 1 --2- 172.21.15.28:0/3618905448 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae941042a0 0x7fae9419a6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:47.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.292+0000 7fae92800640 1 -- 172.21.15.28:0/3618905448 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae9419f1a0 con 0x7fae941098f0 2024-05-12T22:15:47.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.292+0000 7fae91e00640 1 --2- 172.21.15.28:0/3618905448 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae941038e0 0x7fae9419a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:15:47.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.292+0000 7fae92800640 1 --2- 172.21.15.28:0/3618905448 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fae941098f0 0x7fae9419ea70 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fae8c00e940 tx=0x7fae8c00ee00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:15:47.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.292+0000 7fae8aa00640 1 -- 172.21.15.28:0/3618905448 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae8c004d20 con 0x7fae941098f0 2024-05-12T22:15:47.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.292+0000 7fae93200640 1 -- 172.21.15.28:0/3618905448 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae9419f490 con 0x7fae941098f0 2024-05-12T22:15:47.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.292+0000 7fae8aa00640 1 -- 172.21.15.28:0/3618905448 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fae8c005020 con 0x7fae941098f0 2024-05-12T22:15:47.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.292+0000 7fae8aa00640 1 -- 172.21.15.28:0/3618905448 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae8c0116b0 con 0x7fae941098f0 2024-05-12T22:15:47.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.292+0000 7fae93200640 1 -- 172.21.15.28:0/3618905448 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae941a6d20 con 0x7fae941098f0 2024-05-12T22:15:47.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.296+0000 7fae8aa00640 1 -- 172.21.15.28:0/3618905448 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7fae8c0054d0 con 0x7fae941098f0 2024-05-12T22:15:47.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.297+0000 7fae9b821740 1 Processor -- start 2024-05-12T22:15:47.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.297+0000 7fae9b821740 1 -- start start 2024-05-12T22:15:47.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.297+0000 7fae9b821740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564a21ef1dd0 0x564a21ef21b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:47.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.297+0000 7fae9b821740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564a21ef26f0 0x564a21efc880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:47.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.298+0000 7fae9b821740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564a21efcdc0 0x564a21eff1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:47.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.298+0000 7fae9b821740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x564a21eba2a0 con 0x564a21efcdc0 2024-05-12T22:15:47.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.298+0000 7fae9b821740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x564a21e56b90 con 0x564a21ef26f0 2024-05-12T22:15:47.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.298+0000 7fae9b821740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x564a21dfe5b0 con 0x564a21ef1dd0 2024-05-12T22:15:47.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.298+0000 7fae91400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564a21ef1dd0 0x564a21ef21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:47.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.298+0000 7fae91400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564a21ef1dd0 0x564a21ef21b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:34276/0 (socket says 172.21.15.28:34276) 2024-05-12T22:15:47.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.298+0000 7fae91400640 1 -- 172.21.15.28:0/3684831857 learned_addr learned my addr 172.21.15.28:0/3684831857 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:15:47.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.298+0000 7fae8aa00640 1 --2- 172.21.15.28:0/3618905448 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fae780a2dc0 0x7fae780a5280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:47.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.298+0000 7fae91e00640 1 --2- 172.21.15.28:0/3684831857 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564a21ef26f0 0x564a21efc880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:47.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.298+0000 7fae91400640 1 --2- 172.21.15.28:0/3684831857 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564a21efcdc0 0x564a21eff1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:47.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.299+0000 7fae92800640 1 --2- 172.21.15.28:0/3618905448 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fae780a2dc0 0x7fae780a5280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:47.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.299+0000 7fae91400640 1 -- 172.21.15.28:0/3684831857 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564a21ef26f0 msgr2=0x564a21efc880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:15:47.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.299+0000 7fae91400640 1 --2- 172.21.15.28:0/3684831857 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564a21ef26f0 0x564a21efc880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:47.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.299+0000 7fae91400640 1 -- 172.21.15.28:0/3684831857 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564a21efcdc0 msgr2=0x564a21eff1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:15:47.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.299+0000 7fae91400640 1 --2- 172.21.15.28:0/3684831857 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564a21efcdc0 0x564a21eff1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:47.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.299+0000 7fae91400640 1 -- 172.21.15.28:0/3684831857 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564a21c347d0 con 0x564a21ef1dd0 2024-05-12T22:15:47.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.299+0000 7fae8aa00640 1 -- 172.21.15.28:0/3618905448 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7fae8c0cf420 con 0x7fae941098f0 2024-05-12T22:15:47.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.299+0000 7fae91400640 1 --2- 172.21.15.28:0/3684831857 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564a21ef1dd0 0x564a21ef21b0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fae80009230 tx=0x7fae80005fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dd7fe0abf6f26969 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:15:47.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.300+0000 7fae92800640 1 --2- 172.21.15.28:0/3618905448 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fae780a2dc0 0x7fae780a5280 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fae8c097e60 tx=0x7fae8c086040 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:15:47.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.300+0000 7fae7f000640 1 -- 172.21.15.28:0/3684831857 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae80044020 con 0x564a21ef1dd0 2024-05-12T22:15:47.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.300+0000 7fae7f000640 1 -- 172.21.15.28:0/3684831857 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fae800046c0 con 0x564a21ef1dd0 2024-05-12T22:15:47.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.300+0000 7fae7f000640 1 -- 172.21.15.28:0/3684831857 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae80035040 con 0x564a21ef1dd0 2024-05-12T22:15:47.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.301+0000 7fae9b821740 1 -- 172.21.15.28:0/3684831857 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564a21ef1dd0 msgr2=0x564a21ef21b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:15:47.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.301+0000 7fae9b821740 1 --2- 172.21.15.28:0/3684831857 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564a21ef1dd0 0x564a21ef21b0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fae80009230 tx=0x7fae80005fd0 comp rx=0 tx=0).stop 2024-05-12T22:15:47.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.301+0000 7fae9b821740 1 -- 172.21.15.28:0/3684831857 shutdown_connections 2024-05-12T22:15:47.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.301+0000 7fae9b821740 1 --2- 172.21.15.28:0/3684831857 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564a21efcdc0 0x564a21eff1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:47.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.301+0000 7fae9b821740 1 --2- 172.21.15.28:0/3684831857 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564a21ef26f0 0x564a21efc880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:47.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.301+0000 7fae9b821740 1 --2- 172.21.15.28:0/3684831857 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564a21ef1dd0 0x564a21ef21b0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:47.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.301+0000 7fae9b821740 1 -- 172.21.15.28:0/3684831857 >> 172.21.15.28:0/3684831857 conn(0x564a21eee0f0 msgr2=0x564a21eee4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:15:47.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.301+0000 7fae9b821740 1 -- 172.21.15.28:0/3684831857 shutdown_connections 2024-05-12T22:15:47.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.301+0000 7fae9b821740 1 -- 172.21.15.28:0/3684831857 wait complete. 2024-05-12T22:15:47.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.302+0000 7fae9b821740 1 -- start start 2024-05-12T22:15:47.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.302+0000 7fae9b821740 1 Processor -- start 2024-05-12T22:15:47.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.302+0000 7fae9b821740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564a21ef1dd0 0x564a21efc6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:47.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.302+0000 7fae9b821740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564a21ef26f0 0x564a21ef99d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:47.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.302+0000 7fae9b821740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564a21efcdc0 0x564a21ef9f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:47.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.302+0000 7fae9b821740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x564a21e119f0 con 0x564a21ef1dd0 2024-05-12T22:15:47.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.302+0000 7fae9b821740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x564a21e52b50 con 0x564a21efcdc0 2024-05-12T22:15:47.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.302+0000 7fae9b821740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x564a21effb50 con 0x564a21ef26f0 2024-05-12T22:15:47.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.302+0000 7fae9b821740 10 client.0 _ll_register_callbacks cb 0x564a21b69be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:15:47.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.303+0000 7fae91e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564a21ef26f0 0x564a21ef99d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:47.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.303+0000 7fae91e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564a21ef26f0 0x564a21ef99d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:34282/0 (socket says 172.21.15.28:34282) 2024-05-12T22:15:47.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.303+0000 7fae91e00640 1 -- 172.21.15.28:0/544967678 learned_addr learned my addr 172.21.15.28:0/544967678 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:15:47.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.303+0000 7fae91400640 1 --2- 172.21.15.28:0/544967678 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564a21ef1dd0 0x564a21efc6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:47.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.303+0000 7fae91400640 1 --2- 172.21.15.28:0/544967678 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564a21efcdc0 0x564a21ef9f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:47.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.303+0000 7fae91e00640 1 -- 172.21.15.28:0/544967678 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564a21efcdc0 msgr2=0x564a21ef9f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:15:47.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.303+0000 7fae91e00640 1 --2- 172.21.15.28:0/544967678 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564a21efcdc0 0x564a21ef9f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:47.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.303+0000 7fae91e00640 1 -- 172.21.15.28:0/544967678 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564a21ef1dd0 msgr2=0x564a21efc6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:15:47.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.303+0000 7fae91e00640 1 --2- 172.21.15.28:0/544967678 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564a21ef1dd0 0x564a21efc6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:47.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.303+0000 7fae91e00640 1 -- 172.21.15.28:0/544967678 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564a21b6a3f0 con 0x564a21ef26f0 2024-05-12T22:15:47.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.303+0000 7fae91400640 1 --2- 172.21.15.28:0/544967678 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564a21ef1dd0 0x564a21efc6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:15:47.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.303+0000 7fae91e00640 1 --2- 172.21.15.28:0/544967678 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564a21ef26f0 0x564a21ef99d0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fae84004570 tx=0x7fae840039c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6d7d21deae85f58 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:15:47.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.303+0000 7fae73400640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-12T22:15:47.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.304+0000 7fae73400640 1 -- 172.21.15.28:0/544967678 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae84011830 con 0x564a21ef26f0 2024-05-12T22:15:47.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.304+0000 7fae73400640 1 -- 172.21.15.28:0/544967678 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7fae840119d0 con 0x564a21ef26f0 2024-05-12T22:15:47.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.304+0000 7fae9b821740 1 -- 172.21.15.28:0/544967678 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fae800046c0 con 0x564a21ef26f0 2024-05-12T22:15:47.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.304+0000 7fae73400640 1 -- 172.21.15.28:0/544967678 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fae84011bd0 con 0x564a21ef26f0 2024-05-12T22:15:47.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.304+0000 7fae9b821740 10 client.26746 fetch_fsmap learned FSMap version 26 2024-05-12T22:15:47.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.304+0000 7fae9b821740 1 -- 172.21.15.28:0/544967678 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x564a21b2a0f0 con 0x564a21ef26f0 2024-05-12T22:15:47.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.305+0000 7fae73400640 1 -- 172.21.15.28:0/544967678 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fae800046c0 con 0x564a21ef26f0 2024-05-12T22:15:47.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.305+0000 7fae73400640 1 -- 172.21.15.28:0/544967678 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fae84025420 con 0x564a21ef26f0 2024-05-12T22:15:47.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.305+0000 7fae9b821740 10 client.26746 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:15:47.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.305+0000 7fae9b821740 10 client.26746 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:15:47.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.305+0000 7fae9b821740 20 client.26746 populate_metadata read hostname 'smithi028' 2024-05-12T22:15:47.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.305+0000 7fae9b821740 1 --2- 172.21.15.28:0/544967678 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x564a21f151b0 0x564a21f175a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:47.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.305+0000 7fae9b821740 4 client.26746 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-12T22:15:47.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.305+0000 7fae9b821740 1 -- 172.21.15.28:0/544967678 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564a21e642c0 con 0x564a21f151b0 2024-05-12T22:15:47.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.306+0000 7fae91400640 1 --2- 172.21.15.28:0/544967678 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x564a21f151b0 0x564a21f175a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:47.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.306+0000 7fae91400640 1 --2- 172.21.15.28:0/544967678 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x564a21f151b0 0x564a21f175a0 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e07198bb8201d10f server_cookie=19ccc473bdb881a6 in_seq=0 out_seq=0 2024-05-12T22:15:47.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.306+0000 7fae73400640 10 client.26746 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:15:47.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.307+0000 7fae73400640 1 -- 172.21.15.28:0/544967678 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564a21e642c0 con 0x564a21f151b0 2024-05-12T22:15:47.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.307+0000 7fae73400640 10 client.26746 handle_command_reply: tid=0 2024-05-12T22:15:47.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.307+0000 7fae9b821740 1 client.26746 shutdown 2024-05-12T22:15:47.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.307+0000 7fae9b821740 1 -- 172.21.15.28:0/544967678 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564a21ef26f0 msgr2=0x564a21ef99d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:15:47.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.307+0000 7fae9b821740 1 --2- 172.21.15.28:0/544967678 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564a21ef26f0 0x564a21ef99d0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fae84004570 tx=0x7fae840039c0 comp rx=0 tx=0).stop 2024-05-12T22:15:47.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.307+0000 7fae9b821740 1 -- 172.21.15.28:0/544967678 shutdown_connections 2024-05-12T22:15:47.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.307+0000 7fae9b821740 1 --2- 172.21.15.28:0/544967678 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x564a21f151b0 0x564a21f175a0 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:47.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.307+0000 7fae9b821740 1 --2- 172.21.15.28:0/544967678 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564a21efcdc0 0x564a21ef9f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:47.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.307+0000 7fae9b821740 1 --2- 172.21.15.28:0/544967678 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564a21ef26f0 0x564a21ef99d0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:47.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.307+0000 7fae9b821740 1 --2- 172.21.15.28:0/544967678 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564a21ef1dd0 0x564a21efc6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:47.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.307+0000 7fae9b821740 1 -- 172.21.15.28:0/544967678 >> 172.21.15.28:0/544967678 conn(0x564a21eee0f0 msgr2=0x564a21ef3ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:15:47.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.307+0000 7fae73400640 0 client.26746 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:15:47.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.308+0000 7fae9b821740 1 -- 172.21.15.28:0/544967678 shutdown_connections 2024-05-12T22:15:47.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.308+0000 7fae9b821740 1 -- 172.21.15.28:0/544967678 wait complete. 2024-05-12T22:15:47.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.308+0000 7fae9b821740 20 client.26746 trim_cache size 0 max 16384 2024-05-12T22:15:47.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.332+0000 7fae9b821740 1 Processor -- start 2024-05-12T22:15:47.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.332+0000 7fae9b821740 1 -- start start 2024-05-12T22:15:47.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.333+0000 7fae9b821740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564a21ef1dd0 0x564a21ef8b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:47.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.333+0000 7fae9b821740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564a21ef26f0 0x564a21f11120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:47.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.333+0000 7fae9b821740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564a21efcdc0 0x564a21ef4e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:47.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.333+0000 7fae9b821740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fae941ad830 con 0x564a21ef1dd0 2024-05-12T22:15:47.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.333+0000 7fae9b821740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fae941ab100 con 0x564a21efcdc0 2024-05-12T22:15:47.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.333+0000 7fae9b821740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fae941a8f70 con 0x564a21ef26f0 2024-05-12T22:15:47.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.333+0000 7fae91e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564a21ef26f0 0x564a21f11120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:47.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.333+0000 7fae91e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564a21ef26f0 0x564a21f11120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:34288/0 (socket says 172.21.15.28:34288) 2024-05-12T22:15:47.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.333+0000 7fae91e00640 1 -- 172.21.15.28:0/1356848610 learned_addr learned my addr 172.21.15.28:0/1356848610 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:15:47.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.334+0000 7fae91400640 1 --2- 172.21.15.28:0/1356848610 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564a21efcdc0 0x564a21ef4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:47.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.334+0000 7fae91400640 1 --2- 172.21.15.28:0/1356848610 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564a21ef1dd0 0x564a21ef8b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:47.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.334+0000 7fae91e00640 1 -- 172.21.15.28:0/1356848610 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564a21efcdc0 msgr2=0x564a21ef4e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:15:47.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.334+0000 7fae91e00640 1 --2- 172.21.15.28:0/1356848610 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564a21efcdc0 0x564a21ef4e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:47.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.334+0000 7fae91e00640 1 -- 172.21.15.28:0/1356848610 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564a21ef1dd0 msgr2=0x564a21ef8b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:15:47.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.334+0000 7fae91e00640 1 --2- 172.21.15.28:0/1356848610 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564a21ef1dd0 0x564a21ef8b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:47.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.334+0000 7fae91e00640 1 -- 172.21.15.28:0/1356848610 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564a21f13cb0 con 0x564a21ef26f0 2024-05-12T22:15:47.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.334+0000 7fae91400640 1 --2- 172.21.15.28:0/1356848610 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564a21efcdc0 0x564a21ef4e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:15:47.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.334+0000 7fae91e00640 1 --2- 172.21.15.28:0/1356848610 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564a21ef26f0 0x564a21f11120 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fae84001eb0 tx=0x7fae84005880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=65c0b497861f6ae9 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:15:47.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.335+0000 7fae7e600640 1 -- 172.21.15.28:0/1356848610 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae8402a070 con 0x564a21ef26f0 2024-05-12T22:15:47.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.335+0000 7fae7e600640 1 -- 172.21.15.28:0/1356848610 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fae84011830 con 0x564a21ef26f0 2024-05-12T22:15:47.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.335+0000 7fae7e600640 1 -- 172.21.15.28:0/1356848610 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae84005290 con 0x564a21ef26f0 2024-05-12T22:15:47.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.335+0000 7fae9b821740 1 -- 172.21.15.28:0/1356848610 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564a21ef26f0 msgr2=0x564a21f11120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:15:47.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.335+0000 7fae9b821740 1 --2- 172.21.15.28:0/1356848610 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564a21ef26f0 0x564a21f11120 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fae84001eb0 tx=0x7fae84005880 comp rx=0 tx=0).stop 2024-05-12T22:15:47.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.336+0000 7fae9b821740 1 -- 172.21.15.28:0/1356848610 shutdown_connections 2024-05-12T22:15:47.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.336+0000 7fae9b821740 1 --2- 172.21.15.28:0/1356848610 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564a21efcdc0 0x564a21ef4e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:47.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.336+0000 7fae9b821740 1 --2- 172.21.15.28:0/1356848610 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564a21ef26f0 0x564a21f11120 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:47.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.336+0000 7fae9b821740 1 --2- 172.21.15.28:0/1356848610 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564a21ef1dd0 0x564a21ef8b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:47.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.336+0000 7fae9b821740 1 -- 172.21.15.28:0/1356848610 >> 172.21.15.28:0/1356848610 conn(0x564a21eee0f0 msgr2=0x564a21ef4700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:15:47.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.336+0000 7fae9b821740 1 -- 172.21.15.28:0/1356848610 shutdown_connections 2024-05-12T22:15:47.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.336+0000 7fae9b821740 1 -- 172.21.15.28:0/1356848610 wait complete. 2024-05-12T22:15:47.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.336+0000 7fae9b821740 1 -- start start 2024-05-12T22:15:47.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.337+0000 7fae9b821740 1 Processor -- start 2024-05-12T22:15:47.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.337+0000 7fae9b821740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564a21ef1dd0 0x564a21f0d630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:47.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.337+0000 7fae9b821740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564a21ef26f0 0x564a21f0db70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:47.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.337+0000 7fae9b821740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564a21efcdc0 0x564a21ef5c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:47.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.337+0000 7fae9b821740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fae941a95c0 con 0x564a21efcdc0 2024-05-12T22:15:47.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.337+0000 7fae9b821740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fae941aaa30 con 0x564a21ef26f0 2024-05-12T22:15:47.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.337+0000 7fae9b821740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fae941a9c90 con 0x564a21ef1dd0 2024-05-12T22:15:47.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.337+0000 7fae9b821740 10 client.0 _ll_register_callbacks cb 0x7fae8400be00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:15:47.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.337+0000 7fae91e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564a21ef26f0 0x564a21f0db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:47.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.337+0000 7fae91400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564a21ef1dd0 0x564a21f0d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:47.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.337+0000 7fae91e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564a21ef26f0 0x564a21f0db70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:35144/0 (socket says 172.21.15.28:35144) 2024-05-12T22:15:47.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.337+0000 7fae91e00640 1 -- 172.21.15.28:0/1451274171 learned_addr learned my addr 172.21.15.28:0/1451274171 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:15:47.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.338+0000 7fae91400640 1 --2- 172.21.15.28:0/1451274171 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564a21efcdc0 0x564a21ef5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:47.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.338+0000 7fae91e00640 1 -- 172.21.15.28:0/1451274171 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564a21ef1dd0 msgr2=0x564a21f0d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:15:47.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.338+0000 7fae91e00640 1 --2- 172.21.15.28:0/1451274171 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564a21ef1dd0 0x564a21f0d630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:47.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.338+0000 7fae91e00640 1 -- 172.21.15.28:0/1451274171 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564a21efcdc0 msgr2=0x564a21ef5c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:15:47.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.338+0000 7fae91e00640 1 --2- 172.21.15.28:0/1451274171 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564a21efcdc0 0x564a21ef5c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:47.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.338+0000 7fae91e00640 1 -- 172.21.15.28:0/1451274171 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564a21ef6180 con 0x564a21ef26f0 2024-05-12T22:15:47.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.338+0000 7fae91400640 1 --2- 172.21.15.28:0/1451274171 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564a21efcdc0 0x564a21ef5c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:15:47.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.338+0000 7fae91e00640 1 --2- 172.21.15.28:0/1451274171 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564a21ef26f0 0x564a21f0db70 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fae8400b770 tx=0x7fae8401b720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=823bd6c1443327eb server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:15:47.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.338+0000 7fae73400640 10 client.0 ms_handle_connect on v2:172.21.15.86:3300/0 2024-05-12T22:15:47.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.339+0000 7fae73400640 1 -- 172.21.15.28:0/1451274171 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae8402a040 con 0x564a21ef26f0 2024-05-12T22:15:47.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.339+0000 7fae73400640 1 -- 172.21.15.28:0/1451274171 <== mon.2 v2:172.21.15.86:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8469+0+0 (secure 0 0 0) 0x7fae8401bb20 con 0x564a21ef26f0 2024-05-12T22:15:47.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.339+0000 7fae9b821740 1 -- 172.21.15.28:0/1451274171 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fae84011830 con 0x564a21ef26f0 2024-05-12T22:15:47.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.339+0000 7fae73400640 1 -- 172.21.15.28:0/1451274171 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fae84025600 con 0x564a21ef26f0 2024-05-12T22:15:47.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.339+0000 7fae9b821740 10 client.26798 fetch_fsmap learned FSMap version 26 2024-05-12T22:15:47.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.339+0000 7fae9b821740 1 -- 172.21.15.28:0/1451274171 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x564a21f7af30 con 0x564a21ef26f0 2024-05-12T22:15:47.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.339+0000 7fae73400640 1 -- 172.21.15.28:0/1451274171 <== mon.2 v2:172.21.15.86:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fae84011830 con 0x564a21ef26f0 2024-05-12T22:15:47.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.340+0000 7fae73400640 1 -- 172.21.15.28:0/1451274171 <== mon.2 v2:172.21.15.86:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fae84037030 con 0x564a21ef26f0 2024-05-12T22:15:47.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.340+0000 7fae9b821740 10 client.26798 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:15:47.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.340+0000 7fae9b821740 10 client.26798 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:15:47.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.340+0000 7fae9b821740 20 client.26798 populate_metadata read hostname 'smithi028' 2024-05-12T22:15:47.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.340+0000 7fae9b821740 1 --2- 172.21.15.28:0/1451274171 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x564a21f7b540 0x564a21f7d930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:15:47.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.340+0000 7fae9b821740 4 client.26798 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-12T22:15:47.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.340+0000 7fae9b821740 1 -- 172.21.15.28:0/1451274171 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "scrub status"}) -- 0x564a21e5e610 con 0x564a21f7b540 2024-05-12T22:15:47.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.340+0000 7fae91400640 1 --2- 172.21.15.28:0/1451274171 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x564a21f7b540 0x564a21f7d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:15:47.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.341+0000 7fae91400640 1 --2- 172.21.15.28:0/1451274171 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x564a21f7b540 0x564a21f7d930 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b67e8f2083d5c947 server_cookie=e556e5a8c2a75996 in_seq=0 out_seq=0 2024-05-12T22:15:47.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.341+0000 7fae73400640 10 client.26798 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:15:47.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.341+0000 7fae73400640 1 -- 172.21.15.28:0/1451274171 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+268 (crc 0 0 0) 0x564a21e5e610 con 0x564a21f7b540 2024-05-12T22:15:47.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.341+0000 7fae73400640 10 client.26798 handle_command_reply: tid=0 2024-05-12T22:15:47.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.341+0000 7fae9b821740 1 client.26798 shutdown 2024-05-12T22:15:47.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.341+0000 7fae9b821740 1 -- 172.21.15.28:0/1451274171 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564a21ef26f0 msgr2=0x564a21f0db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:15:47.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.341+0000 7fae9b821740 1 --2- 172.21.15.28:0/1451274171 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564a21ef26f0 0x564a21f0db70 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fae8400b770 tx=0x7fae8401b720 comp rx=0 tx=0).stop 2024-05-12T22:15:47.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.342+0000 7fae9b821740 1 -- 172.21.15.28:0/1451274171 shutdown_connections 2024-05-12T22:15:47.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.342+0000 7fae9b821740 1 --2- 172.21.15.28:0/1451274171 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x564a21f7b540 0x564a21f7d930 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:47.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.342+0000 7fae9b821740 1 --2- 172.21.15.28:0/1451274171 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564a21efcdc0 0x564a21ef5c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:47.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.342+0000 7fae9b821740 1 --2- 172.21.15.28:0/1451274171 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564a21ef26f0 0x564a21f0db70 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:47.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.342+0000 7fae9b821740 1 --2- 172.21.15.28:0/1451274171 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564a21ef1dd0 0x564a21f0d630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:47.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.342+0000 7fae9b821740 1 -- 172.21.15.28:0/1451274171 >> 172.21.15.28:0/1451274171 conn(0x564a21eee0f0 msgr2=0x564a21ef3600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:15:47.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.342+0000 7fae73400640 0 client.26798 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:15:47.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.342+0000 7fae9b821740 1 -- 172.21.15.28:0/1451274171 shutdown_connections 2024-05-12T22:15:47.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.342+0000 7fae9b821740 1 -- 172.21.15.28:0/1451274171 wait complete. 2024-05-12T22:15:47.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.342+0000 7fae9b821740 20 client.26798 trim_cache size 0 max 16384 2024-05-12T22:15:47.350 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-05-12T22:15:47.350 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "scrub active (1299 inodes in the stack)", 2024-05-12T22:15:47.350 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-05-12T22:15:47.350 INFO:teuthology.orchestra.run.smithi028.stdout: "fd2e5837-fdab-43d4-bdf7-c150ecbcf6ff": { 2024-05-12T22:15:47.350 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-05-12T22:15:47.350 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "fd2e5837-fdab-43d4-bdf7-c150ecbcf6ff", 2024-05-12T22:15:47.351 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-05-12T22:15:47.351 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:15:47.351 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:15:47.351 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-05-12T22:15:47.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.343+0000 7fae93200640 1 -- 172.21.15.28:0/3618905448 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fae780a2dc0 msgr2=0x7fae780a5280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:15:47.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.343+0000 7fae93200640 1 --2- 172.21.15.28:0/3618905448 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fae780a2dc0 0x7fae780a5280 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fae8c097e60 tx=0x7fae8c086040 comp rx=0 tx=0).stop 2024-05-12T22:15:47.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.343+0000 7fae93200640 1 -- 172.21.15.28:0/3618905448 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fae941098f0 msgr2=0x7fae9419ea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:15:47.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.343+0000 7fae93200640 1 --2- 172.21.15.28:0/3618905448 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fae941098f0 0x7fae9419ea70 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fae8c00e940 tx=0x7fae8c00ee00 comp rx=0 tx=0).stop 2024-05-12T22:15:47.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.343+0000 7fae93200640 1 -- 172.21.15.28:0/3618905448 shutdown_connections 2024-05-12T22:15:47.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.343+0000 7fae93200640 1 --2- 172.21.15.28:0/3618905448 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fae780a2dc0 0x7fae780a5280 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:47.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.343+0000 7fae93200640 1 --2- 172.21.15.28:0/3618905448 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fae941098f0 0x7fae9419ea70 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:47.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.343+0000 7fae93200640 1 --2- 172.21.15.28:0/3618905448 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae941042a0 0x7fae9419a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:47.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.343+0000 7fae93200640 1 --2- 172.21.15.28:0/3618905448 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae941038e0 0x7fae9419a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:15:47.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.343+0000 7fae93200640 1 -- 172.21.15.28:0/3618905448 >> 172.21.15.28:0/3618905448 conn(0x7fae940ff370 msgr2=0x7fae940ff950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:15:47.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.343+0000 7fae93200640 1 -- 172.21.15.28:0/3618905448 shutdown_connections 2024-05-12T22:15:47.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:15:47.343+0000 7fae93200640 1 -- 172.21.15.28:0/3618905448 wait complete. 2024-05-12T22:15:47.364 INFO:tasks.cephfs.filesystem:scrub status for tag:fd2e5837-fdab-43d4-bdf7-c150ecbcf6ff - {'path': '/', 'tag': 'fd2e5837-fdab-43d4-bdf7-c150ecbcf6ff', 'options': 'recursive,force'} 2024-05-12T22:15:48.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:15:48 smithi186 ceph-mon[34785]: pgmap v207: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:15:48.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:15:48 smithi186 ceph-mon[34785]: scrub queued for path: / 2024-05-12T22:15:48.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:15:48 smithi186 ceph-mon[34785]: scrub summary: idle+waiting paths [/] 2024-05-12T22:15:48.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:15:48 smithi186 ceph-mon[34785]: scrub summary: active paths [/] 2024-05-12T22:15:48.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:15:48 smithi186 ceph-mon[34785]: scrub summary: active paths [/] 2024-05-12T22:15:48.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:15:48 smithi186 ceph-mon[34785]: scrub summary: idle+waiting paths [/] 2024-05-12T22:15:48.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:15:48 smithi186 ceph-mon[34785]: scrub summary: active paths [/] 2024-05-12T22:15:48.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:15:48 smithi186 ceph-mon[34785]: scrub summary: idle+waiting paths [/] 2024-05-12T22:15:48.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:15:48 smithi186 ceph-mon[34785]: scrub summary: active paths [/] 2024-05-12T22:15:48.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:15:48 smithi186 ceph-mon[34785]: scrub summary: idle+waiting paths [/] 2024-05-12T22:15:48.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:15:48 smithi186 ceph-mon[34785]: scrub summary: idle+waiting paths [/] 2024-05-12T22:15:48.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:15:48 smithi086 ceph-mon[34297]: pgmap v207: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:15:48.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:15:48 smithi086 ceph-mon[34297]: scrub queued for path: / 2024-05-12T22:15:48.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:15:48 smithi086 ceph-mon[34297]: scrub summary: idle+waiting paths [/] 2024-05-12T22:15:48.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:15:48 smithi086 ceph-mon[34297]: scrub summary: active paths [/] 2024-05-12T22:15:48.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:15:48 smithi086 ceph-mon[34297]: scrub summary: active paths [/] 2024-05-12T22:15:48.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:15:48 smithi086 ceph-mon[34297]: scrub summary: idle+waiting paths [/] 2024-05-12T22:15:48.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:15:48 smithi086 ceph-mon[34297]: scrub summary: active paths [/] 2024-05-12T22:15:48.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:15:48 smithi086 ceph-mon[34297]: scrub summary: idle+waiting paths [/] 2024-05-12T22:15:48.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:15:48 smithi086 ceph-mon[34297]: scrub summary: active paths [/] 2024-05-12T22:15:48.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:15:48 smithi086 ceph-mon[34297]: scrub summary: idle+waiting paths [/] 2024-05-12T22:15:48.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:15:48 smithi086 ceph-mon[34297]: scrub summary: idle+waiting paths [/] 2024-05-12T22:15:48.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:15:48 smithi028 ceph-mon[29847]: pgmap v207: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:15:48.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:15:48 smithi028 ceph-mon[29847]: scrub queued for path: / 2024-05-12T22:15:48.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:15:48 smithi028 ceph-mon[29847]: scrub summary: idle+waiting paths [/] 2024-05-12T22:15:48.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:15:48 smithi028 ceph-mon[29847]: scrub summary: active paths [/] 2024-05-12T22:15:48.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:15:48 smithi028 ceph-mon[29847]: scrub summary: active paths [/] 2024-05-12T22:15:48.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:15:48 smithi028 ceph-mon[29847]: scrub summary: idle+waiting paths [/] 2024-05-12T22:15:48.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:15:48 smithi028 ceph-mon[29847]: scrub summary: active paths [/] 2024-05-12T22:15:48.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:15:48 smithi028 ceph-mon[29847]: scrub summary: idle+waiting paths [/] 2024-05-12T22:15:48.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:15:48 smithi028 ceph-mon[29847]: scrub summary: active paths [/] 2024-05-12T22:15:48.537 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:15:48 smithi028 ceph-mon[29847]: scrub summary: idle+waiting paths [/] 2024-05-12T22:15:48.537 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:15:48 smithi028 ceph-mon[29847]: scrub summary: idle+waiting paths [/] 2024-05-12T22:15:50.338 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:15:50 smithi028 ceph-mon[29847]: pgmap v208: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:15:50.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:15:50 smithi186 ceph-mon[34785]: pgmap v208: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:15:50.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:15:50 smithi086 ceph-mon[34297]: pgmap v208: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:15:52.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:15:52 smithi186 ceph-mon[34785]: pgmap v209: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-05-12T22:15:52.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:15:52 smithi086 ceph-mon[34297]: pgmap v209: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-05-12T22:15:52.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:15:52 smithi028 ceph-mon[29847]: pgmap v209: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-05-12T22:15:53.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:15:53 smithi186 ceph-mon[34785]: scrub summary: idle 2024-05-12T22:15:53.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:15:53 smithi186 ceph-mon[34785]: scrub summary: idle 2024-05-12T22:15:53.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:15:53 smithi086 ceph-mon[34297]: scrub summary: idle 2024-05-12T22:15:53.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:15:53 smithi086 ceph-mon[34297]: scrub summary: idle 2024-05-12T22:15:53.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:15:53 smithi028 ceph-mon[29847]: scrub summary: idle 2024-05-12T22:15:53.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:15:53 smithi028 ceph-mon[29847]: scrub summary: idle 2024-05-12T22:15:54.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:15:54 smithi186 ceph-mon[34785]: pgmap v210: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-05-12T22:15:54.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:15:54 smithi086 ceph-mon[34297]: pgmap v210: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-05-12T22:15:54.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:15:54 smithi028 ceph-mon[29847]: pgmap v210: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-05-12T22:15:56.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:15:56 smithi186 ceph-mon[34785]: pgmap v211: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-05-12T22:15:56.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:15:56 smithi086 ceph-mon[34297]: pgmap v211: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-05-12T22:15:56.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:15:56 smithi028 ceph-mon[29847]: pgmap v211: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-05-12T22:15:58.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:15:58 smithi186 ceph-mon[34785]: pgmap v212: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-05-12T22:15:58.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:15:58 smithi086 ceph-mon[34297]: pgmap v212: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-05-12T22:15:58.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:15:58 smithi028 ceph-mon[29847]: pgmap v212: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-05-12T22:16:00.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:16:00 smithi186 ceph-mon[34785]: pgmap v213: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-05-12T22:16:00.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:16:00 smithi086 ceph-mon[34297]: pgmap v213: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-05-12T22:16:00.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:16:00 smithi028 ceph-mon[29847]: pgmap v213: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-05-12T22:16:02.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:16:02 smithi186 ceph-mon[34785]: pgmap v214: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-05-12T22:16:02.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:16:02 smithi086 ceph-mon[34297]: pgmap v214: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-05-12T22:16:02.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:16:02 smithi028 ceph-mon[29847]: pgmap v214: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-05-12T22:16:04.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:16:04 smithi186 ceph-mon[34785]: pgmap v215: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:16:04.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:16:04 smithi086 ceph-mon[34297]: pgmap v215: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:16:04.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:16:04 smithi028 ceph-mon[29847]: pgmap v215: 97 pgs: 97 active+clean; 97 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:16:06.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:16:06 smithi186 ceph-mon[34785]: pgmap v216: 97 pgs: 97 active+clean; 97 MiB data, 754 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 24 op/s 2024-05-12T22:16:06.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:16:06 smithi086 ceph-mon[34297]: pgmap v216: 97 pgs: 97 active+clean; 97 MiB data, 754 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 24 op/s 2024-05-12T22:16:06.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:16:06 smithi028 ceph-mon[29847]: pgmap v216: 97 pgs: 97 active+clean; 97 MiB data, 754 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 24 op/s 2024-05-12T22:16:08.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:16:08 smithi186 ceph-mon[34785]: pgmap v217: 97 pgs: 97 active+clean; 97 MiB data, 754 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 22 op/s 2024-05-12T22:16:08.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:16:08 smithi086 ceph-mon[34297]: pgmap v217: 97 pgs: 97 active+clean; 97 MiB data, 754 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 22 op/s 2024-05-12T22:16:08.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:16:08 smithi028 ceph-mon[29847]: pgmap v217: 97 pgs: 97 active+clean; 97 MiB data, 754 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 22 op/s 2024-05-12T22:16:09.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:16:09 smithi186 ceph-mon[34785]: osdmap e90: 12 total, 12 up, 12 in 2024-05-12T22:16:09.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:16:09 smithi086 ceph-mon[34297]: osdmap e90: 12 total, 12 up, 12 in 2024-05-12T22:16:09.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:16:09 smithi028 ceph-mon[29847]: osdmap e90: 12 total, 12 up, 12 in 2024-05-12T22:16:10.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:16:10 smithi186 ceph-mon[34785]: pgmap v219: 97 pgs: 97 active+clean; 97 MiB data, 754 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-05-12T22:16:10.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:16:10 smithi086 ceph-mon[34297]: pgmap v219: 97 pgs: 97 active+clean; 97 MiB data, 754 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-05-12T22:16:10.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:16:10 smithi028 ceph-mon[29847]: pgmap v219: 97 pgs: 97 active+clean; 97 MiB data, 754 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-05-12T22:16:12.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:16:12 smithi186 ceph-mon[34785]: pgmap v220: 97 pgs: 97 active+clean; 97 MiB data, 757 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-05-12T22:16:12.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:16:12 smithi086 ceph-mon[34297]: pgmap v220: 97 pgs: 97 active+clean; 97 MiB data, 757 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-05-12T22:16:12.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:16:12 smithi028 ceph-mon[29847]: pgmap v220: 97 pgs: 97 active+clean; 97 MiB data, 757 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-05-12T22:16:14.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:16:14 smithi186 ceph-mon[34785]: pgmap v221: 97 pgs: 97 active+clean; 97 MiB data, 757 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-05-12T22:16:14.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:16:14 smithi186 ceph-mon[34785]: osdmap e91: 12 total, 12 up, 12 in 2024-05-12T22:16:14.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:16:14 smithi086 ceph-mon[34297]: pgmap v221: 97 pgs: 97 active+clean; 97 MiB data, 757 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-05-12T22:16:14.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:16:14 smithi086 ceph-mon[34297]: osdmap e91: 12 total, 12 up, 12 in 2024-05-12T22:16:14.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:16:14 smithi028 ceph-mon[29847]: pgmap v221: 97 pgs: 97 active+clean; 97 MiB data, 757 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-05-12T22:16:14.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:16:14 smithi028 ceph-mon[29847]: osdmap e91: 12 total, 12 up, 12 in 2024-05-12T22:16:16.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:16:16 smithi186 ceph-mon[34785]: pgmap v223: 97 pgs: 97 active+clean; 97 MiB data, 757 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2024-05-12T22:16:16.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:16:16 smithi086 ceph-mon[34297]: pgmap v223: 97 pgs: 97 active+clean; 97 MiB data, 757 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2024-05-12T22:16:16.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:16:16 smithi028 ceph-mon[29847]: pgmap v223: 97 pgs: 97 active+clean; 97 MiB data, 757 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2024-05-12T22:16:17.365 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-12T22:16:17.365 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-12T22:16:17.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.583+0000 7f4cf3e00640 1 -- 172.21.15.28:0/487768293 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4cf410ddf0 msgr2=0x7f4cf410e1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:16:17.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.583+0000 7f4cf3e00640 1 --2- 172.21.15.28:0/487768293 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4cf410ddf0 0x7f4cf410e1d0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f4ce4009900 tx=0x7f4ce402a480 comp rx=0 tx=0).stop 2024-05-12T22:16:17.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.583+0000 7f4cf3e00640 1 -- 172.21.15.28:0/487768293 shutdown_connections 2024-05-12T22:16:17.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.583+0000 7f4cf3e00640 1 --2- 172.21.15.28:0/487768293 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4cf4108770 0x7f4cf4114b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:17.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.583+0000 7f4cf3e00640 1 --2- 172.21.15.28:0/487768293 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4cf4107dd0 0x7f4cf4108230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:17.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.583+0000 7f4cf3e00640 1 --2- 172.21.15.28:0/487768293 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4cf410ddf0 0x7f4cf410e1d0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:17.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.583+0000 7f4cf3e00640 1 -- 172.21.15.28:0/487768293 >> 172.21.15.28:0/487768293 conn(0x7f4cf4077650 msgr2=0x7f4cf4106650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:16:17.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.583+0000 7f4cf3e00640 1 -- 172.21.15.28:0/487768293 shutdown_connections 2024-05-12T22:16:17.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.583+0000 7f4cf3e00640 1 -- 172.21.15.28:0/487768293 wait complete. 2024-05-12T22:16:17.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.584+0000 7f4cf3e00640 1 Processor -- start 2024-05-12T22:16:17.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.585+0000 7f4cf3e00640 1 -- start start 2024-05-12T22:16:17.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.585+0000 7f4cf3e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4cf4107dd0 0x7f4cf419e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:17.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.585+0000 7f4cf3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4cf4108770 0x7f4cf419eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:17.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.585+0000 7f4cf3e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4cf410ddf0 0x7f4cf41a2e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:17.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.585+0000 7f4cf3e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4cf41180d0 con 0x7f4cf4108770 2024-05-12T22:16:17.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.585+0000 7f4cf3e00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f4cf4117f50 con 0x7f4cf4107dd0 2024-05-12T22:16:17.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.585+0000 7f4cf3e00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f4cf4118250 con 0x7f4cf410ddf0 2024-05-12T22:16:17.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.585+0000 7f4cebe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4cf4108770 0x7f4cf419eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:17.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.586+0000 7f4cebe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4cf4108770 0x7f4cf419eb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54886/0 (socket says 172.21.15.28:54886) 2024-05-12T22:16:17.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.586+0000 7f4cebe00640 1 -- 172.21.15.28:0/3029583519 learned_addr learned my addr 172.21.15.28:0/3029583519 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:16:17.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.586+0000 7f4cf2a00640 1 --2- 172.21.15.28:0/3029583519 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4cf4107dd0 0x7f4cf419e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:17.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.586+0000 7f4cf3400640 1 --2- 172.21.15.28:0/3029583519 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4cf410ddf0 0x7f4cf41a2e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:17.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.586+0000 7f4cebe00640 1 -- 172.21.15.28:0/3029583519 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4cf410ddf0 msgr2=0x7f4cf41a2e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:16:17.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.586+0000 7f4cebe00640 1 --2- 172.21.15.28:0/3029583519 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4cf410ddf0 0x7f4cf41a2e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:17.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.586+0000 7f4cebe00640 1 -- 172.21.15.28:0/3029583519 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4cf4107dd0 msgr2=0x7f4cf419e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:16:17.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.586+0000 7f4cebe00640 1 --2- 172.21.15.28:0/3029583519 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4cf4107dd0 0x7f4cf419e5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:17.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.586+0000 7f4cebe00640 1 -- 172.21.15.28:0/3029583519 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cf41a3570 con 0x7f4cf4108770 2024-05-12T22:16:17.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.586+0000 7f4cf2a00640 1 --2- 172.21.15.28:0/3029583519 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4cf4107dd0 0x7f4cf419e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:16:17.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.586+0000 7f4cebe00640 1 --2- 172.21.15.28:0/3029583519 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4cf4108770 0x7f4cf419eb10 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f4cdc00e9f0 tx=0x7f4cdc00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:16:17.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.587+0000 7f4ceb400640 1 -- 172.21.15.28:0/3029583519 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cdc00cd80 con 0x7f4cf4108770 2024-05-12T22:16:17.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.587+0000 7f4ceb400640 1 -- 172.21.15.28:0/3029583519 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4cdc005500 con 0x7f4cf4108770 2024-05-12T22:16:17.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.587+0000 7f4ceb400640 1 -- 172.21.15.28:0/3029583519 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cdc010690 con 0x7f4cf4108770 2024-05-12T22:16:17.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.587+0000 7f4cf3e00640 1 -- 172.21.15.28:0/3029583519 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4cf41a3860 con 0x7f4cf4108770 2024-05-12T22:16:17.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.587+0000 7f4cf3e00640 1 -- 172.21.15.28:0/3029583519 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4cf41ab0f0 con 0x7f4cf4108770 2024-05-12T22:16:17.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.590+0000 7f4cfcbe5740 1 Processor -- start 2024-05-12T22:16:17.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.590+0000 7f4ceb400640 1 -- 172.21.15.28:0/3029583519 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7f4cdc0046c0 con 0x7f4cf4108770 2024-05-12T22:16:17.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.591+0000 7f4cfcbe5740 1 -- start start 2024-05-12T22:16:17.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.591+0000 7f4cfcbe5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560169031dd0 0x5601690321b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:17.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.591+0000 7f4cfcbe5740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5601690326f0 0x56016903c880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:17.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.591+0000 7f4cfcbe5740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56016903cdc0 0x56016903f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:17.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.591+0000 7f4cfcbe5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x560168ffa2a0 con 0x560169031dd0 2024-05-12T22:16:17.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.591+0000 7f4cfcbe5740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x560168f96b90 con 0x5601690326f0 2024-05-12T22:16:17.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.591+0000 7f4cfcbe5740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x560168f3e5b0 con 0x56016903cdc0 2024-05-12T22:16:17.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.591+0000 7f4cf3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560169031dd0 0x5601690321b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:17.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.591+0000 7f4cf2a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5601690326f0 0x56016903c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:17.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.591+0000 7f4cf3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560169031dd0 0x5601690321b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54902/0 (socket says 172.21.15.28:54902) 2024-05-12T22:16:17.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.591+0000 7f4cf3400640 1 -- 172.21.15.28:0/919865084 learned_addr learned my addr 172.21.15.28:0/919865084 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:16:17.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.592+0000 7f4cebe00640 1 --2- 172.21.15.28:0/919865084 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56016903cdc0 0x56016903f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:17.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.592+0000 7f4cf2a00640 1 -- 172.21.15.28:0/919865084 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56016903cdc0 msgr2=0x56016903f1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:16:17.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.592+0000 7f4cf2a00640 1 --2- 172.21.15.28:0/919865084 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56016903cdc0 0x56016903f1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:17.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.592+0000 7f4cf2a00640 1 -- 172.21.15.28:0/919865084 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560169031dd0 msgr2=0x5601690321b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:16:17.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.592+0000 7f4cf2a00640 1 --2- 172.21.15.28:0/919865084 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560169031dd0 0x5601690321b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:17.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.592+0000 7f4cf2a00640 1 -- 172.21.15.28:0/919865084 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560168d747d0 con 0x5601690326f0 2024-05-12T22:16:17.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.592+0000 7f4cf3400640 1 --2- 172.21.15.28:0/919865084 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560169031dd0 0x5601690321b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:16:17.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.592+0000 7f4cf2a00640 1 --2- 172.21.15.28:0/919865084 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5601690326f0 0x56016903c880 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f4ce4005940 tx=0x7f4ce4005d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8000fb2000cc4618 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:16:17.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.592+0000 7f4ceb400640 1 --2- 172.21.15.28:0/3029583519 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f4cd80a2d80 0x7f4cd80a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:17.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.593+0000 7f4ceb400640 1 -- 172.21.15.28:0/3029583519 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8513+0+0 (secure 0 0 0) 0x7f4cdc0cf7d0 con 0x7f4cf4108770 2024-05-12T22:16:17.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.593+0000 7f4ce3e00640 1 -- 172.21.15.28:0/919865084 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ce4035040 con 0x5601690326f0 2024-05-12T22:16:17.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.593+0000 7f4cf3400640 1 --2- 172.21.15.28:0/3029583519 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f4cd80a2d80 0x7f4cd80a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:17.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.593+0000 7f4ce3e00640 1 -- 172.21.15.28:0/919865084 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4ce4031830 con 0x5601690326f0 2024-05-12T22:16:17.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.593+0000 7f4ce3e00640 1 -- 172.21.15.28:0/919865084 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ce4004750 con 0x5601690326f0 2024-05-12T22:16:17.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.594+0000 7f4cfcbe5740 1 -- 172.21.15.28:0/919865084 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5601690326f0 msgr2=0x56016903c880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:16:17.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.594+0000 7f4cfcbe5740 1 --2- 172.21.15.28:0/919865084 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5601690326f0 0x56016903c880 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f4ce4005940 tx=0x7f4ce4005d50 comp rx=0 tx=0).stop 2024-05-12T22:16:17.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.594+0000 7f4cf3400640 1 --2- 172.21.15.28:0/3029583519 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f4cd80a2d80 0x7f4cd80a5240 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f4cec011870 tx=0x7f4cec009500 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:16:17.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.594+0000 7f4cfcbe5740 1 -- 172.21.15.28:0/919865084 shutdown_connections 2024-05-12T22:16:17.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.594+0000 7f4cfcbe5740 1 --2- 172.21.15.28:0/919865084 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56016903cdc0 0x56016903f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:17.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.594+0000 7f4cfcbe5740 1 --2- 172.21.15.28:0/919865084 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5601690326f0 0x56016903c880 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:17.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.594+0000 7f4cfcbe5740 1 --2- 172.21.15.28:0/919865084 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560169031dd0 0x5601690321b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:17.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.594+0000 7f4cfcbe5740 1 -- 172.21.15.28:0/919865084 >> 172.21.15.28:0/919865084 conn(0x56016902e0f0 msgr2=0x56016902e4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:16:17.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.594+0000 7f4cfcbe5740 1 -- 172.21.15.28:0/919865084 shutdown_connections 2024-05-12T22:16:17.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.594+0000 7f4cfcbe5740 1 -- 172.21.15.28:0/919865084 wait complete. 2024-05-12T22:16:17.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.594+0000 7f4cfcbe5740 1 -- start start 2024-05-12T22:16:17.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.595+0000 7f4cfcbe5740 1 Processor -- start 2024-05-12T22:16:17.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.595+0000 7f4cfcbe5740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560169031dd0 0x56016903c720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:17.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.595+0000 7f4cfcbe5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5601690326f0 0x560169039a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:17.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.595+0000 7f4cfcbe5740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56016903cdc0 0x560169039f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:17.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.595+0000 7f4cfcbe5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x560168f519f0 con 0x5601690326f0 2024-05-12T22:16:17.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.595+0000 7f4cfcbe5740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x560168f92b50 con 0x560169031dd0 2024-05-12T22:16:17.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.595+0000 7f4cfcbe5740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x56016903fb50 con 0x56016903cdc0 2024-05-12T22:16:17.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.595+0000 7f4cfcbe5740 10 client.0 _ll_register_callbacks cb 0x560168ca9be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:16:17.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.595+0000 7f4cebe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5601690326f0 0x560169039a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:17.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.595+0000 7f4cf2a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560169031dd0 0x56016903c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:17.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.595+0000 7f4cebe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5601690326f0 0x560169039a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54916/0 (socket says 172.21.15.28:54916) 2024-05-12T22:16:17.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.595+0000 7f4cebe00640 1 -- 172.21.15.28:0/1858050799 learned_addr learned my addr 172.21.15.28:0/1858050799 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:16:17.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.595+0000 7f4cf3400640 1 --2- 172.21.15.28:0/1858050799 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56016903cdc0 0x560169039f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:17.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.596+0000 7f4cebe00640 1 -- 172.21.15.28:0/1858050799 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56016903cdc0 msgr2=0x560169039f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:16:17.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.596+0000 7f4cebe00640 1 --2- 172.21.15.28:0/1858050799 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56016903cdc0 0x560169039f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:17.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.596+0000 7f4cebe00640 1 -- 172.21.15.28:0/1858050799 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560169031dd0 msgr2=0x56016903c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:16:17.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.596+0000 7f4cebe00640 1 --2- 172.21.15.28:0/1858050799 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560169031dd0 0x56016903c720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:17.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.596+0000 7f4cebe00640 1 -- 172.21.15.28:0/1858050799 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560168caa3f0 con 0x5601690326f0 2024-05-12T22:16:17.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.596+0000 7f4cf2a00640 1 --2- 172.21.15.28:0/1858050799 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560169031dd0 0x56016903c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:16:17.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.596+0000 7f4cf3400640 1 --2- 172.21.15.28:0/1858050799 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56016903cdc0 0x560169039f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:16:17.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.596+0000 7f4cebe00640 1 --2- 172.21.15.28:0/1858050799 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5601690326f0 0x560169039a10 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f4cdc00e9c0 tx=0x7f4cdc009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85d4dd271cb53718 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:16:17.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.596+0000 7f4ce0c00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-12T22:16:17.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.596+0000 7f4ce0c00640 1 -- 172.21.15.28:0/1858050799 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cdc09b830 con 0x5601690326f0 2024-05-12T22:16:17.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.596+0000 7f4ce0c00640 1 -- 172.21.15.28:0/1858050799 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8513+0+0 (secure 0 0 0) 0x7f4cdc0a46d0 con 0x5601690326f0 2024-05-12T22:16:17.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.596+0000 7f4cfcbe5740 1 -- 172.21.15.28:0/1858050799 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4ce4031830 con 0x5601690326f0 2024-05-12T22:16:17.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.597+0000 7f4ce0c00640 1 -- 172.21.15.28:0/1858050799 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f4cdc09b650 con 0x5601690326f0 2024-05-12T22:16:17.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.597+0000 7f4cfcbe5740 10 client.16683 fetch_fsmap learned FSMap version 26 2024-05-12T22:16:17.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.597+0000 7f4cfcbe5740 1 -- 172.21.15.28:0/1858050799 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x560168c6a0f0 con 0x5601690326f0 2024-05-12T22:16:17.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.597+0000 7f4ce0c00640 1 -- 172.21.15.28:0/1858050799 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4ce4031830 con 0x5601690326f0 2024-05-12T22:16:17.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.597+0000 7f4ce0c00640 1 -- 172.21.15.28:0/1858050799 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f4cdc0b3080 con 0x5601690326f0 2024-05-12T22:16:17.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.597+0000 7f4cfcbe5740 10 client.16683 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:16:17.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.597+0000 7f4cfcbe5740 10 client.16683 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:16:17.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.597+0000 7f4cfcbe5740 20 client.16683 populate_metadata read hostname 'smithi028' 2024-05-12T22:16:17.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.597+0000 7f4cfcbe5740 1 --2- 172.21.15.28:0/1858050799 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x56016904c840 0x56016904ec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:17.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.597+0000 7f4cfcbe5740 4 client.16683 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-12T22:16:17.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.597+0000 7f4cfcbe5740 1 -- 172.21.15.28:0/1858050799 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560168fa42c0 con 0x56016904c840 2024-05-12T22:16:17.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.598+0000 7f4cf2a00640 1 --2- 172.21.15.28:0/1858050799 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x56016904c840 0x56016904ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:17.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.598+0000 7f4cf2a00640 1 --2- 172.21.15.28:0/1858050799 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x56016904c840 0x56016904ec30 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7474a1315834ac82 server_cookie=65a5dcacc1dfbfdf in_seq=0 out_seq=0 2024-05-12T22:16:17.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.598+0000 7f4ce0c00640 10 client.16683 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:16:17.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.599+0000 7f4ce0c00640 1 -- 172.21.15.28:0/1858050799 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560168fa42c0 con 0x56016904c840 2024-05-12T22:16:17.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.599+0000 7f4ce0c00640 10 client.16683 handle_command_reply: tid=0 2024-05-12T22:16:17.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.599+0000 7f4cfcbe5740 1 client.16683 shutdown 2024-05-12T22:16:17.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.600+0000 7f4cfcbe5740 1 -- 172.21.15.28:0/1858050799 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5601690326f0 msgr2=0x560169039a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:16:17.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.600+0000 7f4cfcbe5740 1 --2- 172.21.15.28:0/1858050799 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5601690326f0 0x560169039a10 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f4cdc00e9c0 tx=0x7f4cdc009310 comp rx=0 tx=0).stop 2024-05-12T22:16:17.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.600+0000 7f4cfcbe5740 1 -- 172.21.15.28:0/1858050799 shutdown_connections 2024-05-12T22:16:17.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.600+0000 7f4cfcbe5740 1 --2- 172.21.15.28:0/1858050799 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x56016904c840 0x56016904ec30 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:17.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.600+0000 7f4cfcbe5740 1 --2- 172.21.15.28:0/1858050799 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56016903cdc0 0x560169039f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:17.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.600+0000 7f4cfcbe5740 1 --2- 172.21.15.28:0/1858050799 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5601690326f0 0x560169039a10 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:17.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.600+0000 7f4cfcbe5740 1 --2- 172.21.15.28:0/1858050799 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560169031dd0 0x56016903c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:17.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.600+0000 7f4ce0c00640 0 client.16683 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:16:17.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.600+0000 7f4cfcbe5740 1 -- 172.21.15.28:0/1858050799 >> 172.21.15.28:0/1858050799 conn(0x56016902e0f0 msgr2=0x56016902d330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:16:17.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.600+0000 7f4cfcbe5740 1 -- 172.21.15.28:0/1858050799 shutdown_connections 2024-05-12T22:16:17.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.600+0000 7f4cfcbe5740 1 -- 172.21.15.28:0/1858050799 wait complete. 2024-05-12T22:16:17.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.600+0000 7f4cfcbe5740 20 client.16683 trim_cache size 0 max 16384 2024-05-12T22:16:17.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.619+0000 7f4cfcbe5740 1 Processor -- start 2024-05-12T22:16:17.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.619+0000 7f4cfcbe5740 1 -- start start 2024-05-12T22:16:17.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.620+0000 7f4cfcbe5740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56016902e0f0 0x5601690544b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:17.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.620+0000 7f4cfcbe5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560169031dd0 0x5601690549f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:17.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.620+0000 7f4cfcbe5740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5601690326f0 0x560169036cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:17.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.620+0000 7f4cfcbe5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4cf41b61f0 con 0x560169031dd0 2024-05-12T22:16:17.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.620+0000 7f4cfcbe5740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f4cf41b3ac0 con 0x5601690326f0 2024-05-12T22:16:17.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.620+0000 7f4cfcbe5740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f4cf41b1930 con 0x56016902e0f0 2024-05-12T22:16:17.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.620+0000 7f4cebe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560169031dd0 0x5601690549f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:17.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.620+0000 7f4cebe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560169031dd0 0x5601690549f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54922/0 (socket says 172.21.15.28:54922) 2024-05-12T22:16:17.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.620+0000 7f4cebe00640 1 -- 172.21.15.28:0/1494567862 learned_addr learned my addr 172.21.15.28:0/1494567862 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:16:17.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.620+0000 7f4cf2a00640 1 --2- 172.21.15.28:0/1494567862 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56016902e0f0 0x5601690544b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:17.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.620+0000 7f4cf3400640 1 --2- 172.21.15.28:0/1494567862 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5601690326f0 0x560169036cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:17.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.620+0000 7f4cebe00640 1 -- 172.21.15.28:0/1494567862 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56016902e0f0 msgr2=0x5601690544b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:16:17.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.620+0000 7f4cebe00640 1 --2- 172.21.15.28:0/1494567862 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56016902e0f0 0x5601690544b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:17.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.620+0000 7f4cebe00640 1 -- 172.21.15.28:0/1494567862 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5601690326f0 msgr2=0x560169036cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:16:17.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.620+0000 7f4cebe00640 1 --2- 172.21.15.28:0/1494567862 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5601690326f0 0x560169036cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:17.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.620+0000 7f4cebe00640 1 -- 172.21.15.28:0/1494567862 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56016903a2e0 con 0x560169031dd0 2024-05-12T22:16:17.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.621+0000 7f4cebe00640 1 --2- 172.21.15.28:0/1494567862 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560169031dd0 0x5601690549f0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f4cdc09b9f0 tx=0x7f4cdc0ccfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4efb54f17121c8d server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:16:17.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.621+0000 7f4ce3400640 1 -- 172.21.15.28:0/1494567862 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cdc0a8020 con 0x560169031dd0 2024-05-12T22:16:17.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.621+0000 7f4ce3400640 1 -- 172.21.15.28:0/1494567862 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4cdc0a00a0 con 0x560169031dd0 2024-05-12T22:16:17.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.621+0000 7f4ce3400640 1 -- 172.21.15.28:0/1494567862 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cdc0a4730 con 0x560169031dd0 2024-05-12T22:16:17.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.622+0000 7f4cfcbe5740 1 -- 172.21.15.28:0/1494567862 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560169031dd0 msgr2=0x5601690549f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:16:17.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.622+0000 7f4cfcbe5740 1 --2- 172.21.15.28:0/1494567862 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560169031dd0 0x5601690549f0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f4cdc09b9f0 tx=0x7f4cdc0ccfd0 comp rx=0 tx=0).stop 2024-05-12T22:16:17.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.622+0000 7f4cfcbe5740 1 -- 172.21.15.28:0/1494567862 shutdown_connections 2024-05-12T22:16:17.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.622+0000 7f4cfcbe5740 1 --2- 172.21.15.28:0/1494567862 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5601690326f0 0x560169036cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:17.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.622+0000 7f4cfcbe5740 1 --2- 172.21.15.28:0/1494567862 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560169031dd0 0x5601690549f0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:17.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.622+0000 7f4cfcbe5740 1 --2- 172.21.15.28:0/1494567862 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56016902e0f0 0x5601690544b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:17.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.622+0000 7f4cfcbe5740 1 -- 172.21.15.28:0/1494567862 >> 172.21.15.28:0/1494567862 conn(0x56016902ce10 msgr2=0x560169047880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:16:17.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.622+0000 7f4cfcbe5740 1 -- 172.21.15.28:0/1494567862 shutdown_connections 2024-05-12T22:16:17.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.622+0000 7f4cfcbe5740 1 -- 172.21.15.28:0/1494567862 wait complete. 2024-05-12T22:16:17.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.623+0000 7f4cfcbe5740 1 -- start start 2024-05-12T22:16:17.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.623+0000 7f4cfcbe5740 1 Processor -- start 2024-05-12T22:16:17.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.623+0000 7f4cfcbe5740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56016902e0f0 0x560169042f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:17.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.623+0000 7f4cfcbe5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560169031dd0 0x5601690434d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:17.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.623+0000 7f4cfcbe5740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5601690326f0 0x5601690b1d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:17.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.623+0000 7f4cfcbe5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4cf41b1f80 con 0x560169031dd0 2024-05-12T22:16:17.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.623+0000 7f4cfcbe5740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f4cf41b33f0 con 0x56016902e0f0 2024-05-12T22:16:17.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.623+0000 7f4cfcbe5740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f4cf41b2650 con 0x5601690326f0 2024-05-12T22:16:17.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.623+0000 7f4cfcbe5740 10 client.0 _ll_register_callbacks cb 0x7f4cdc0d5240 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:16:17.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.624+0000 7f4cebe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560169031dd0 0x5601690434d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:17.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.624+0000 7f4cf2a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56016902e0f0 0x560169042f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:17.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.624+0000 7f4cebe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560169031dd0 0x5601690434d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54930/0 (socket says 172.21.15.28:54930) 2024-05-12T22:16:17.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.624+0000 7f4cebe00640 1 -- 172.21.15.28:0/2332130252 learned_addr learned my addr 172.21.15.28:0/2332130252 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:16:17.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.624+0000 7f4cf3400640 1 --2- 172.21.15.28:0/2332130252 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5601690326f0 0x5601690b1d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:17.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.624+0000 7f4cf2a00640 1 -- 172.21.15.28:0/2332130252 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5601690326f0 msgr2=0x5601690b1d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:16:17.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.624+0000 7f4cf2a00640 1 --2- 172.21.15.28:0/2332130252 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5601690326f0 0x5601690b1d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:17.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.624+0000 7f4cf2a00640 1 -- 172.21.15.28:0/2332130252 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560169031dd0 msgr2=0x5601690434d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:16:17.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.624+0000 7f4cf2a00640 1 --2- 172.21.15.28:0/2332130252 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560169031dd0 0x5601690434d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:17.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.624+0000 7f4cf2a00640 1 -- 172.21.15.28:0/2332130252 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5601690b2320 con 0x56016902e0f0 2024-05-12T22:16:17.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.624+0000 7f4cebe00640 1 --2- 172.21.15.28:0/2332130252 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560169031dd0 0x5601690434d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:16:17.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.624+0000 7f4cf2a00640 1 --2- 172.21.15.28:0/2332130252 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56016902e0f0 0x560169042f90 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f4ce4006010 tx=0x7f4ce4005850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=af965950312c826c server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:16:17.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.624+0000 7f4ce0c00640 10 client.0 ms_handle_connect on v2:172.21.15.86:3300/0 2024-05-12T22:16:17.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.625+0000 7f4ce0c00640 1 -- 172.21.15.28:0/2332130252 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ce4058060 con 0x56016902e0f0 2024-05-12T22:16:17.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.625+0000 7f4cfcbe5740 1 -- 172.21.15.28:0/2332130252 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4cdc0a00a0 con 0x56016902e0f0 2024-05-12T22:16:17.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.625+0000 7f4ce0c00640 1 -- 172.21.15.28:0/2332130252 <== mon.2 v2:172.21.15.86:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8513+0+0 (secure 0 0 0) 0x7f4ce4004030 con 0x56016902e0f0 2024-05-12T22:16:17.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.625+0000 7f4ce0c00640 1 -- 172.21.15.28:0/2332130252 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f4ce4035040 con 0x56016902e0f0 2024-05-12T22:16:17.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.625+0000 7f4cfcbe5740 10 client.26816 fetch_fsmap learned FSMap version 26 2024-05-12T22:16:17.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.625+0000 7f4cfcbe5740 1 -- 172.21.15.28:0/2332130252 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5601690b3670 con 0x56016902e0f0 2024-05-12T22:16:17.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.625+0000 7f4ce0c00640 1 -- 172.21.15.28:0/2332130252 <== mon.2 v2:172.21.15.86:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4cdc0a00a0 con 0x56016902e0f0 2024-05-12T22:16:17.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.626+0000 7f4ce0c00640 1 -- 172.21.15.28:0/2332130252 <== mon.2 v2:172.21.15.86:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f4ce40536d0 con 0x56016902e0f0 2024-05-12T22:16:17.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.626+0000 7f4cfcbe5740 10 client.26816 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:16:17.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.626+0000 7f4cfcbe5740 10 client.26816 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:16:17.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.626+0000 7f4cfcbe5740 20 client.26816 populate_metadata read hostname 'smithi028' 2024-05-12T22:16:17.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.626+0000 7f4cfcbe5740 1 --2- 172.21.15.28:0/2332130252 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x56016903cdc0 0x5601690b59c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:17.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.626+0000 7f4cfcbe5740 4 client.26816 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-12T22:16:17.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.626+0000 7f4cfcbe5740 1 -- 172.21.15.28:0/2332130252 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "scrub status"}) -- 0x560168f9e610 con 0x56016903cdc0 2024-05-12T22:16:17.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.626+0000 7f4cebe00640 1 --2- 172.21.15.28:0/2332130252 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x56016903cdc0 0x5601690b59c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:17.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.626+0000 7f4cebe00640 1 --2- 172.21.15.28:0/2332130252 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x56016903cdc0 0x5601690b59c0 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3bad136dd0f9a84f server_cookie=5ea1c251213347ba in_seq=0 out_seq=0 2024-05-12T22:16:17.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.626+0000 7f4ce0c00640 10 client.26816 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:16:17.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.627+0000 7f4ce0c00640 1 -- 172.21.15.28:0/2332130252 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x560168f9e610 con 0x56016903cdc0 2024-05-12T22:16:17.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.627+0000 7f4ce0c00640 10 client.26816 handle_command_reply: tid=0 2024-05-12T22:16:17.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.627+0000 7f4cfcbe5740 1 client.26816 shutdown 2024-05-12T22:16:17.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.627+0000 7f4cfcbe5740 1 -- 172.21.15.28:0/2332130252 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56016902e0f0 msgr2=0x560169042f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:16:17.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.627+0000 7f4cfcbe5740 1 --2- 172.21.15.28:0/2332130252 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56016902e0f0 0x560169042f90 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f4ce4006010 tx=0x7f4ce4005850 comp rx=0 tx=0).stop 2024-05-12T22:16:17.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.627+0000 7f4cfcbe5740 1 -- 172.21.15.28:0/2332130252 shutdown_connections 2024-05-12T22:16:17.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.627+0000 7f4cfcbe5740 1 --2- 172.21.15.28:0/2332130252 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x56016903cdc0 0x5601690b59c0 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:17.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.627+0000 7f4cfcbe5740 1 --2- 172.21.15.28:0/2332130252 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5601690326f0 0x5601690b1d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:17.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.627+0000 7f4cfcbe5740 1 --2- 172.21.15.28:0/2332130252 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560169031dd0 0x5601690434d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:17.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.627+0000 7f4cfcbe5740 1 --2- 172.21.15.28:0/2332130252 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56016902e0f0 0x560169042f90 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:17.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.627+0000 7f4ce0c00640 0 client.26816 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:16:17.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.627+0000 7f4cfcbe5740 1 -- 172.21.15.28:0/2332130252 >> 172.21.15.28:0/2332130252 conn(0x56016902ce10 msgr2=0x56016903bad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:16:17.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.627+0000 7f4cfcbe5740 1 -- 172.21.15.28:0/2332130252 shutdown_connections 2024-05-12T22:16:17.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.627+0000 7f4cfcbe5740 1 -- 172.21.15.28:0/2332130252 wait complete. 2024-05-12T22:16:17.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.627+0000 7f4cfcbe5740 20 client.26816 trim_cache size 0 max 16384 2024-05-12T22:16:17.637 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-05-12T22:16:17.638 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-05-12T22:16:17.638 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-05-12T22:16:17.638 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-05-12T22:16:17.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.628+0000 7f4cf3e00640 1 -- 172.21.15.28:0/3029583519 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f4cd80a2d80 msgr2=0x7f4cd80a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:16:17.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.628+0000 7f4cf3e00640 1 --2- 172.21.15.28:0/3029583519 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f4cd80a2d80 0x7f4cd80a5240 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f4cec011870 tx=0x7f4cec009500 comp rx=0 tx=0).stop 2024-05-12T22:16:17.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.628+0000 7f4cf3e00640 1 -- 172.21.15.28:0/3029583519 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4cf4108770 msgr2=0x7f4cf419eb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:16:17.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.628+0000 7f4cf3e00640 1 --2- 172.21.15.28:0/3029583519 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4cf4108770 0x7f4cf419eb10 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f4cdc00e9f0 tx=0x7f4cdc00edf0 comp rx=0 tx=0).stop 2024-05-12T22:16:17.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.628+0000 7f4cf3e00640 1 -- 172.21.15.28:0/3029583519 shutdown_connections 2024-05-12T22:16:17.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.628+0000 7f4cf3e00640 1 --2- 172.21.15.28:0/3029583519 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f4cd80a2d80 0x7f4cd80a5240 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:17.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.628+0000 7f4cf3e00640 1 --2- 172.21.15.28:0/3029583519 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4cf410ddf0 0x7f4cf41a2e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:17.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.628+0000 7f4cf3e00640 1 --2- 172.21.15.28:0/3029583519 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4cf4108770 0x7f4cf419eb10 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:17.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.628+0000 7f4cf3e00640 1 --2- 172.21.15.28:0/3029583519 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4cf4107dd0 0x7f4cf419e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:17.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.628+0000 7f4cf3e00640 1 -- 172.21.15.28:0/3029583519 >> 172.21.15.28:0/3029583519 conn(0x7f4cf4077650 msgr2=0x7f4cf4078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:16:17.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.628+0000 7f4cf3e00640 1 -- 172.21.15.28:0/3029583519 shutdown_connections 2024-05-12T22:16:17.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.628+0000 7f4cf3e00640 1 -- 172.21.15.28:0/3029583519 wait complete. 2024-05-12T22:16:17.648 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-12T22:16:17.648 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-12T22:16:17.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.858+0000 7f1d87e00640 1 -- 172.21.15.28:0/1604984366 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1d88107cd0 msgr2=0x7f1d88108130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:16:17.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.858+0000 7f1d87e00640 1 --2- 172.21.15.28:0/1604984366 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1d88107cd0 0x7f1d88108130 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f1d74009860 tx=0x7f1d7402a3d0 comp rx=0 tx=0).stop 2024-05-12T22:16:17.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.858+0000 7f1d87e00640 1 -- 172.21.15.28:0/1604984366 shutdown_connections 2024-05-12T22:16:17.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.858+0000 7f1d87e00640 1 --2- 172.21.15.28:0/1604984366 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1d88108670 0x7f1d88114ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:17.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.858+0000 7f1d87e00640 1 --2- 172.21.15.28:0/1604984366 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1d88107cd0 0x7f1d88108130 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:17.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.858+0000 7f1d87e00640 1 --2- 172.21.15.28:0/1604984366 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d8810dd40 0x7f1d8810e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:17.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.858+0000 7f1d87e00640 1 -- 172.21.15.28:0/1604984366 >> 172.21.15.28:0/1604984366 conn(0x7f1d880773b0 msgr2=0x7f1d880797d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:16:17.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.858+0000 7f1d87e00640 1 -- 172.21.15.28:0/1604984366 shutdown_connections 2024-05-12T22:16:17.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.858+0000 7f1d87e00640 1 -- 172.21.15.28:0/1604984366 wait complete. 2024-05-12T22:16:17.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.859+0000 7f1d87e00640 1 Processor -- start 2024-05-12T22:16:17.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.859+0000 7f1d87e00640 1 -- start start 2024-05-12T22:16:17.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.859+0000 7f1d87e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1d88107cd0 0x7f1d8819e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:17.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.859+0000 7f1d87e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d88108670 0x7f1d8819e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:17.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.859+0000 7f1d87e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1d8810dd40 0x7f1d881a2d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:17.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.859+0000 7f1d87e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1d88118070 con 0x7f1d88108670 2024-05-12T22:16:17.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.859+0000 7f1d87e00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f1d88117ef0 con 0x7f1d8810dd40 2024-05-12T22:16:17.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.859+0000 7f1d87e00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f1d881181f0 con 0x7f1d88107cd0 2024-05-12T22:16:17.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.860+0000 7f1d86000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d88108670 0x7f1d8819e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:17.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.860+0000 7f1d86000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d88108670 0x7f1d8819e9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54952/0 (socket says 172.21.15.28:54952) 2024-05-12T22:16:17.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.860+0000 7f1d86000640 1 -- 172.21.15.28:0/930650503 learned_addr learned my addr 172.21.15.28:0/930650503 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:16:17.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.860+0000 7f1d87400640 1 --2- 172.21.15.28:0/930650503 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1d8810dd40 0x7f1d881a2d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:17.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.860+0000 7f1d86a00640 1 --2- 172.21.15.28:0/930650503 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1d88107cd0 0x7f1d8819e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:17.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.860+0000 7f1d86000640 1 -- 172.21.15.28:0/930650503 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1d88107cd0 msgr2=0x7f1d8819e4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:16:17.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.860+0000 7f1d86000640 1 --2- 172.21.15.28:0/930650503 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1d88107cd0 0x7f1d8819e4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:17.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.860+0000 7f1d86000640 1 -- 172.21.15.28:0/930650503 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1d8810dd40 msgr2=0x7f1d881a2d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:16:17.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.860+0000 7f1d86000640 1 --2- 172.21.15.28:0/930650503 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1d8810dd40 0x7f1d881a2d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:17.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.860+0000 7f1d86000640 1 -- 172.21.15.28:0/930650503 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d881a3450 con 0x7f1d88108670 2024-05-12T22:16:17.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.860+0000 7f1d86000640 1 --2- 172.21.15.28:0/930650503 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d88108670 0x7f1d8819e9f0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f1d74009980 tx=0x7f1d740309a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:16:17.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.861+0000 7f1d7f400640 1 -- 172.21.15.28:0/930650503 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d74040070 con 0x7f1d88108670 2024-05-12T22:16:17.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.861+0000 7f1d7f400640 1 -- 172.21.15.28:0/930650503 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1d74005620 con 0x7f1d88108670 2024-05-12T22:16:17.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.861+0000 7f1d87e00640 1 -- 172.21.15.28:0/930650503 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d881a36e0 con 0x7f1d88108670 2024-05-12T22:16:17.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.861+0000 7f1d7f400640 1 -- 172.21.15.28:0/930650503 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d740049b0 con 0x7f1d88108670 2024-05-12T22:16:17.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.861+0000 7f1d87e00640 1 -- 172.21.15.28:0/930650503 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d881aafd0 con 0x7f1d88108670 2024-05-12T22:16:17.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.862+0000 7f1d87e00640 1 -- 172.21.15.28:0/930650503 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1d880689c0 con 0x7f1d88108670 2024-05-12T22:16:17.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.865+0000 7f1d7f400640 1 -- 172.21.15.28:0/930650503 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7f1d74043070 con 0x7f1d88108670 2024-05-12T22:16:17.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.867+0000 7f1d7f400640 1 --2- 172.21.15.28:0/930650503 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f1d6c0a2d00 0x7f1d6c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:17.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.867+0000 7f1d7f400640 1 -- 172.21.15.28:0/930650503 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8513+0+0 (secure 0 0 0) 0x7f1d740f0180 con 0x7f1d88108670 2024-05-12T22:16:17.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.867+0000 7f1d86a00640 1 --2- 172.21.15.28:0/930650503 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f1d6c0a2d00 0x7f1d6c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:17.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.868+0000 7f1d86a00640 1 --2- 172.21.15.28:0/930650503 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f1d6c0a2d00 0x7f1d6c0a51c0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f1d78009530 tx=0x7f1d78009260 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:16:17.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:17.872+0000 7f1d7f400640 1 -- 172.21.15.28:0/930650503 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f1d740bc6f0 con 0x7f1d88108670 2024-05-12T22:16:18.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.125+0000 7f1d87e00640 1 -- 172.21.15.28:0/930650503 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1d8819f7b0 con 0x7f1d88108670 2024-05-12T22:16:18.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.127+0000 7f1d7f400640 1 -- 172.21.15.28:0/930650503 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11864 (secure 0 0 0) 0x7f1d740c09f0 con 0x7f1d88108670 2024-05-12T22:16:18.128 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-12T22:16:18.128 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1844141745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1844141745},{"type":"v1","addr":"172.21.15.186:6833","nonce":1844141745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6833/2960763758","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":2960763758},{"type":"v1","addr":"172.21.15.86:6833","nonce":2960763758}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24470,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/4131134219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":4131134219},{"type":"v1","addr":"172.21.15.28:6835","nonce":4131134219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24481,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/1727869064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1727869064},{"type":"v1","addr":"172.21.15.186:6835","nonce":1727869064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24482,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2938260704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2938260704},{"type":"v1","addr":"172.21.15.28:6837","nonce":2938260704}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24494,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3473700910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3473700910},{"type":"v1","addr":"172.21.15.86:6835","nonce":3473700910}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-12T22:06:33.426986+0000","modified":"2024-05-12T22:13:22.499381+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24581,"mds_1":24550,"mds_2":24532},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24499":{"gid":24499,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6837/3600809235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":3600809235},{"type":"v1","addr":"172.21.15.186:6837","nonce":3600809235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"i","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6839/2399610807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":2399610807},{"type":"v1","addr":"172.21.15.186:6839","nonce":2399610807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24530":{"gid":24530,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.86:6837/3782448884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":3782448884},{"type":"v1","addr":"172.21.15.86:6837","nonce":3782448884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24532":{"gid":24532,"name":"g","rank":2,"incarnation":21,"state":"up:active","state_seq":6,"addr":"172.21.15.28:6839/123892158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":123892158},{"type":"v1","addr":"172.21.15.28:6839","nonce":123892158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24550":{"gid":24550,"name":"a","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.28:6841/761183870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":761183870},{"type":"v1","addr":"172.21.15.28:6841","nonce":761183870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24581":{"gid":24581,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6839/1251999130","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6838","nonce":1251999130},{"type":"v1","addr":"172.21.15.86:6839","nonce":1251999130}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24581,"qdb_cluster":[24550,24581,24532]},"id":1}]} 2024-05-12T22:16:18.128 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 26 2024-05-12T22:16:18.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.130+0000 7f1d87e00640 1 -- 172.21.15.28:0/930650503 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f1d6c0a2d00 msgr2=0x7f1d6c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:16:18.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.130+0000 7f1d87e00640 1 --2- 172.21.15.28:0/930650503 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f1d6c0a2d00 0x7f1d6c0a51c0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f1d78009530 tx=0x7f1d78009260 comp rx=0 tx=0).stop 2024-05-12T22:16:18.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.130+0000 7f1d87e00640 1 -- 172.21.15.28:0/930650503 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d88108670 msgr2=0x7f1d8819e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:16:18.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.130+0000 7f1d87e00640 1 --2- 172.21.15.28:0/930650503 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d88108670 0x7f1d8819e9f0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f1d74009980 tx=0x7f1d740309a0 comp rx=0 tx=0).stop 2024-05-12T22:16:18.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.130+0000 7f1d87e00640 1 -- 172.21.15.28:0/930650503 shutdown_connections 2024-05-12T22:16:18.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.130+0000 7f1d87e00640 1 --2- 172.21.15.28:0/930650503 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f1d6c0a2d00 0x7f1d6c0a51c0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:18.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.130+0000 7f1d87e00640 1 --2- 172.21.15.28:0/930650503 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1d8810dd40 0x7f1d881a2d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:18.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.130+0000 7f1d87e00640 1 --2- 172.21.15.28:0/930650503 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d88108670 0x7f1d8819e9f0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:18.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.130+0000 7f1d87e00640 1 --2- 172.21.15.28:0/930650503 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1d88107cd0 0x7f1d8819e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:18.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.130+0000 7f1d87e00640 1 -- 172.21.15.28:0/930650503 >> 172.21.15.28:0/930650503 conn(0x7f1d880773b0 msgr2=0x7f1d88078a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:16:18.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.130+0000 7f1d87e00640 1 -- 172.21.15.28:0/930650503 shutdown_connections 2024-05-12T22:16:18.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.130+0000 7f1d87e00640 1 -- 172.21.15.28:0/930650503 wait complete. 2024-05-12T22:16:18.150 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-05-12T22:16:18.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.369+0000 7f793f800640 1 -- 172.21.15.28:0/2911087380 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7938073b00 msgr2=0x7f7938073ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:16:18.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.369+0000 7f793f800640 1 --2- 172.21.15.28:0/2911087380 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7938073b00 0x7f7938073ee0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f792c009900 tx=0x7f792c02a480 comp rx=0 tx=0).stop 2024-05-12T22:16:18.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.369+0000 7f793f800640 1 -- 172.21.15.28:0/2911087380 shutdown_connections 2024-05-12T22:16:18.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.369+0000 7f793f800640 1 --2- 172.21.15.28:0/2911087380 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f793810e110 0x7f7938110500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:18.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.369+0000 7f793f800640 1 --2- 172.21.15.28:0/2911087380 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7938074420 0x7f793810dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:18.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.369+0000 7f793f800640 1 --2- 172.21.15.28:0/2911087380 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7938073b00 0x7f7938073ee0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:18.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.369+0000 7f793f800640 1 -- 172.21.15.28:0/2911087380 >> 172.21.15.28:0/2911087380 conn(0x7f7938078c80 msgr2=0x7f7938079090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:16:18.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.369+0000 7f793f800640 1 -- 172.21.15.28:0/2911087380 shutdown_connections 2024-05-12T22:16:18.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.369+0000 7f793f800640 1 -- 172.21.15.28:0/2911087380 wait complete. 2024-05-12T22:16:18.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.370+0000 7f793f800640 1 Processor -- start 2024-05-12T22:16:18.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.370+0000 7f793f800640 1 -- start start 2024-05-12T22:16:18.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.371+0000 7f793f800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7938073b00 0x7f793819a360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:18.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.371+0000 7f793f800640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7938074420 0x7f793819a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:18.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.371+0000 7f793f800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f793810e110 0x7f793819ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:18.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.371+0000 7f793f800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7938113e60 con 0x7f793810e110 2024-05-12T22:16:18.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.371+0000 7f793f800640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f7938113ce0 con 0x7f7938073b00 2024-05-12T22:16:18.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.371+0000 7f793f800640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f7938113fe0 con 0x7f7938074420 2024-05-12T22:16:18.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.371+0000 7f793ce00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7938073b00 0x7f793819a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:18.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.371+0000 7f7937e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7938074420 0x7f793819a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:18.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.371+0000 7f7937e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7938074420 0x7f793819a8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:47162/0 (socket says 172.21.15.28:47162) 2024-05-12T22:16:18.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.371+0000 7f793ce00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7938073b00 0x7f793819a360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:54772/0 (socket says 172.21.15.28:54772) 2024-05-12T22:16:18.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.371+0000 7f7937e00640 1 -- 172.21.15.28:0/2987443072 learned_addr learned my addr 172.21.15.28:0/2987443072 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:16:18.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.371+0000 7f793d800640 1 --2- 172.21.15.28:0/2987443072 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f793810e110 0x7f793819ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:18.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.371+0000 7f7937e00640 1 -- 172.21.15.28:0/2987443072 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7938073b00 msgr2=0x7f793819a360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:16:18.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.371+0000 7f7937e00640 1 --2- 172.21.15.28:0/2987443072 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7938073b00 0x7f793819a360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:18.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.371+0000 7f7937e00640 1 -- 172.21.15.28:0/2987443072 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f793810e110 msgr2=0x7f793819ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:16:18.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.371+0000 7f7937e00640 1 --2- 172.21.15.28:0/2987443072 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f793810e110 0x7f793819ec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:18.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.372+0000 7f7937e00640 1 -- 172.21.15.28:0/2987443072 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f793819f350 con 0x7f7938074420 2024-05-12T22:16:18.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.372+0000 7f7937e00640 1 --2- 172.21.15.28:0/2987443072 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7938074420 0x7f793819a8a0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f792400e9f0 tx=0x7f792400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:16:18.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.372+0000 7f7935600640 1 -- 172.21.15.28:0/2987443072 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f792400cd80 con 0x7f7938074420 2024-05-12T22:16:18.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.372+0000 7f793f800640 1 -- 172.21.15.28:0/2987443072 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f793819f640 con 0x7f7938074420 2024-05-12T22:16:18.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.372+0000 7f7935600640 1 -- 172.21.15.28:0/2987443072 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7924010910 con 0x7f7938074420 2024-05-12T22:16:18.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.372+0000 7f7935600640 1 -- 172.21.15.28:0/2987443072 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79240188f0 con 0x7f7938074420 2024-05-12T22:16:18.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.372+0000 7f793f800640 1 -- 172.21.15.28:0/2987443072 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79381a6ed0 con 0x7f7938074420 2024-05-12T22:16:18.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.376+0000 7f794262e740 1 Processor -- start 2024-05-12T22:16:18.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.376+0000 7f794262e740 1 -- start start 2024-05-12T22:16:18.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.376+0000 7f794262e740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f8e13a6980 0x55f8e13a6d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:18.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.376+0000 7f794262e740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f8e13a72a0 0x55f8e13b1430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:18.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.376+0000 7f794262e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f8e13b1970 0x55f8e13b3d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:18.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.376+0000 7f794262e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f8e130ba30 con 0x55f8e13b1970 2024-05-12T22:16:18.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.376+0000 7f794262e740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x55f8e12b3700 con 0x55f8e13a6980 2024-05-12T22:16:18.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.376+0000 7f794262e740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55f8e13079f0 con 0x55f8e13a72a0 2024-05-12T22:16:18.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.376+0000 7f7935600640 1 -- 172.21.15.28:0/2987443072 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7f7924010ac0 con 0x7f7938074420 2024-05-12T22:16:18.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.377+0000 7f793d800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f8e13a6980 0x55f8e13a6d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:18.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.377+0000 7f793ce00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f8e13a72a0 0x55f8e13b1430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:18.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.377+0000 7f793d800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f8e13a6980 0x55f8e13a6d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:54776/0 (socket says 172.21.15.28:54776) 2024-05-12T22:16:18.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.377+0000 7f793d800640 1 -- 172.21.15.28:0/1360144344 learned_addr learned my addr 172.21.15.28:0/1360144344 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:16:18.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.377+0000 7f7937e00640 1 --2- 172.21.15.28:0/1360144344 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f8e13b1970 0x55f8e13b3d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:18.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.377+0000 7f793ce00640 1 -- 172.21.15.28:0/1360144344 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f8e13a6980 msgr2=0x55f8e13a6d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:16:18.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.377+0000 7f793ce00640 1 --2- 172.21.15.28:0/1360144344 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f8e13a6980 0x55f8e13a6d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:18.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.377+0000 7f793ce00640 1 -- 172.21.15.28:0/1360144344 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f8e13b1970 msgr2=0x55f8e13b3d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:16:18.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.377+0000 7f793ce00640 1 --2- 172.21.15.28:0/1360144344 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f8e13b1970 0x55f8e13b3d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:18.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.377+0000 7f793ce00640 1 -- 172.21.15.28:0/1360144344 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f8e101f3f0 con 0x55f8e13a72a0 2024-05-12T22:16:18.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.378+0000 7f793ce00640 1 --2- 172.21.15.28:0/1360144344 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f8e13a72a0 0x55f8e13b1430 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f792c02aa70 tx=0x7f792c005de0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=420a22d646c2de4 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:16:18.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.378+0000 7f792a000640 1 -- 172.21.15.28:0/1360144344 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f792c035040 con 0x55f8e13a72a0 2024-05-12T22:16:18.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.378+0000 7f792a000640 1 -- 172.21.15.28:0/1360144344 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f792c004400 con 0x55f8e13a72a0 2024-05-12T22:16:18.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.378+0000 7f792a000640 1 -- 172.21.15.28:0/1360144344 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f792c0035f0 con 0x55f8e13a72a0 2024-05-12T22:16:18.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.378+0000 7f7935600640 1 --2- 172.21.15.28:0/2987443072 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f79200a2d80 0x7f79200a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:18.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.378+0000 7f7935600640 1 -- 172.21.15.28:0/2987443072 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8513+0+0 (secure 0 0 0) 0x7f79240cf7e0 con 0x7f7938074420 2024-05-12T22:16:18.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.379+0000 7f793d800640 1 --2- 172.21.15.28:0/2987443072 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f79200a2d80 0x7f79200a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:18.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.379+0000 7f794262e740 1 -- 172.21.15.28:0/1360144344 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f8e13a72a0 msgr2=0x55f8e13b1430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:16:18.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.379+0000 7f794262e740 1 --2- 172.21.15.28:0/1360144344 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f8e13a72a0 0x55f8e13b1430 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f792c02aa70 tx=0x7f792c005de0 comp rx=0 tx=0).stop 2024-05-12T22:16:18.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.379+0000 7f794262e740 1 -- 172.21.15.28:0/1360144344 shutdown_connections 2024-05-12T22:16:18.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.379+0000 7f794262e740 1 --2- 172.21.15.28:0/1360144344 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f8e13b1970 0x55f8e13b3d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:18.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.379+0000 7f794262e740 1 --2- 172.21.15.28:0/1360144344 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f8e13a72a0 0x55f8e13b1430 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:18.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.379+0000 7f794262e740 1 --2- 172.21.15.28:0/1360144344 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f8e13a6980 0x55f8e13a6d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:18.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.379+0000 7f794262e740 1 -- 172.21.15.28:0/1360144344 >> 172.21.15.28:0/1360144344 conn(0x55f8e13a2ca0 msgr2=0x55f8e13a3080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:16:18.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.379+0000 7f794262e740 1 -- 172.21.15.28:0/1360144344 shutdown_connections 2024-05-12T22:16:18.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.379+0000 7f794262e740 1 -- 172.21.15.28:0/1360144344 wait complete. 2024-05-12T22:16:18.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.380+0000 7f793d800640 1 --2- 172.21.15.28:0/2987443072 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f79200a2d80 0x7f79200a5240 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f7930011630 tx=0x7f7930009500 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:16:18.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.380+0000 7f794262e740 1 -- start start 2024-05-12T22:16:18.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.380+0000 7f794262e740 1 Processor -- start 2024-05-12T22:16:18.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.380+0000 7f794262e740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f8e13a6980 0x55f8e13ae900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:18.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.381+0000 7f794262e740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f8e13a72a0 0x55f8e13aee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:18.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.381+0000 7f794262e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f8e13b1970 0x55f8e13af380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:18.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.381+0000 7f794262e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f8e13b4700 con 0x55f8e13b1970 2024-05-12T22:16:18.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.381+0000 7f794262e740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x55f8e12c6a70 con 0x55f8e13a72a0 2024-05-12T22:16:18.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.381+0000 7f794262e740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55f8e13b4880 con 0x55f8e13a6980 2024-05-12T22:16:18.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.381+0000 7f794262e740 10 client.0 _ll_register_callbacks cb 0x55f8e12a9ba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:16:18.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.381+0000 7f793ce00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f8e13a6980 0x55f8e13ae900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:18.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.381+0000 7f793d800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f8e13b1970 0x55f8e13af380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:18.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.381+0000 7f793ce00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f8e13a6980 0x55f8e13ae900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:47178/0 (socket says 172.21.15.28:47178) 2024-05-12T22:16:18.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.381+0000 7f793ce00640 1 -- 172.21.15.28:0/1029555664 learned_addr learned my addr 172.21.15.28:0/1029555664 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:16:18.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.381+0000 7f7937e00640 1 --2- 172.21.15.28:0/1029555664 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f8e13a72a0 0x55f8e13aee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:18.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.381+0000 7f793ce00640 1 -- 172.21.15.28:0/1029555664 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f8e13a72a0 msgr2=0x55f8e13aee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:16:18.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.381+0000 7f793ce00640 1 --2- 172.21.15.28:0/1029555664 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f8e13a72a0 0x55f8e13aee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:18.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.381+0000 7f793ce00640 1 -- 172.21.15.28:0/1029555664 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f8e13b1970 msgr2=0x55f8e13af380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:16:18.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.381+0000 7f793ce00640 1 --2- 172.21.15.28:0/1029555664 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f8e13b1970 0x55f8e13af380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:18.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.381+0000 7f793ce00640 1 -- 172.21.15.28:0/1029555664 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f8e10e97d0 con 0x55f8e13a6980 2024-05-12T22:16:18.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.382+0000 7f793ce00640 1 --2- 172.21.15.28:0/1029555664 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f8e13a6980 0x55f8e13ae900 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f792c0048f0 tx=0x7f792c004920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4b91ce98f4df5757 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:16:18.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.382+0000 7f791ea00640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-12T22:16:18.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.382+0000 7f791ea00640 1 -- 172.21.15.28:0/1029555664 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f792c035040 con 0x55f8e13a6980 2024-05-12T22:16:18.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.382+0000 7f791ea00640 1 -- 172.21.15.28:0/1029555664 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8513+0+0 (secure 0 0 0) 0x7f792c003800 con 0x55f8e13a6980 2024-05-12T22:16:18.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.382+0000 7f794262e740 1 -- 172.21.15.28:0/1029555664 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f792c004400 con 0x55f8e13a6980 2024-05-12T22:16:18.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.382+0000 7f791ea00640 1 -- 172.21.15.28:0/1029555664 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f792c05a050 con 0x55f8e13a6980 2024-05-12T22:16:18.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.382+0000 7f794262e740 10 client.26788 fetch_fsmap learned FSMap version 26 2024-05-12T22:16:18.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.382+0000 7f794262e740 1 -- 172.21.15.28:0/1029555664 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55f8e13b4700 con 0x55f8e13a6980 2024-05-12T22:16:18.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.383+0000 7f791ea00640 1 -- 172.21.15.28:0/1029555664 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f792c004400 con 0x55f8e13a6980 2024-05-12T22:16:18.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.383+0000 7f791ea00640 1 -- 172.21.15.28:0/1029555664 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f792c055030 con 0x55f8e13a6980 2024-05-12T22:16:18.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.383+0000 7f794262e740 10 client.26788 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:16:18.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.383+0000 7f794262e740 10 client.26788 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.g 2024-05-12T22:16:18.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.383+0000 7f794262e740 20 client.26788 populate_metadata read hostname 'smithi028' 2024-05-12T22:16:18.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.383+0000 7f794262e740 1 --2- 172.21.15.28:0/1029555664 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x55f8e13c1b70 0x55f8e13c3f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:18.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.383+0000 7f794262e740 4 client.26788 mds_command: new command op to 24532 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-12T22:16:18.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.383+0000 7f794262e740 1 -- 172.21.15.28:0/1029555664 --> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f8e1319160 con 0x55f8e13c1b70 2024-05-12T22:16:18.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.383+0000 7f7937e00640 1 --2- 172.21.15.28:0/1029555664 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x55f8e13c1b70 0x55f8e13c3f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:18.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.384+0000 7f7937e00640 1 --2- 172.21.15.28:0/1029555664 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x55f8e13c1b70 0x55f8e13c3f60 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c7595a80592705f8 server_cookie=284acb826bc347da in_seq=0 out_seq=0 2024-05-12T22:16:18.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.384+0000 7f791ea00640 10 client.26788 ms_handle_connect on v2:172.21.15.28:6838/123892158 2024-05-12T22:16:18.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.384+0000 7f791ea00640 1 -- 172.21.15.28:0/1029555664 <== mds.2 v2:172.21.15.28:6838/123892158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f8e1319160 con 0x55f8e13c1b70 2024-05-12T22:16:18.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.384+0000 7f791ea00640 10 client.26788 handle_command_reply: tid=0 2024-05-12T22:16:18.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.385+0000 7f794262e740 1 client.26788 shutdown 2024-05-12T22:16:18.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.385+0000 7f794262e740 1 -- 172.21.15.28:0/1029555664 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f8e13a6980 msgr2=0x55f8e13ae900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:16:18.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.385+0000 7f794262e740 1 --2- 172.21.15.28:0/1029555664 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f8e13a6980 0x55f8e13ae900 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f792c0048f0 tx=0x7f792c004920 comp rx=0 tx=0).stop 2024-05-12T22:16:18.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.385+0000 7f794262e740 1 -- 172.21.15.28:0/1029555664 shutdown_connections 2024-05-12T22:16:18.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.385+0000 7f794262e740 1 --2- 172.21.15.28:0/1029555664 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f8e13b1970 0x55f8e13af380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:18.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.385+0000 7f794262e740 1 --2- 172.21.15.28:0/1029555664 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x55f8e13c1b70 0x55f8e13c3f60 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:18.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.385+0000 7f794262e740 1 --2- 172.21.15.28:0/1029555664 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f8e13a72a0 0x55f8e13aee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:18.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.385+0000 7f794262e740 1 --2- 172.21.15.28:0/1029555664 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f8e13a6980 0x55f8e13ae900 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:18.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.385+0000 7f794262e740 1 -- 172.21.15.28:0/1029555664 >> 172.21.15.28:0/1029555664 conn(0x55f8e13a2ca0 msgr2=0x55f8e13a8aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:16:18.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.385+0000 7f791ea00640 0 client.26788 ms_handle_reset on v2:172.21.15.28:6838/123892158 2024-05-12T22:16:18.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.385+0000 7f794262e740 1 -- 172.21.15.28:0/1029555664 shutdown_connections 2024-05-12T22:16:18.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.385+0000 7f794262e740 1 -- 172.21.15.28:0/1029555664 wait complete. 2024-05-12T22:16:18.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.385+0000 7f794262e740 20 client.26788 trim_cache size 0 max 16384 2024-05-12T22:16:18.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.400+0000 7f794262e740 1 Processor -- start 2024-05-12T22:16:18.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.400+0000 7f794262e740 1 -- start start 2024-05-12T22:16:18.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.400+0000 7f794262e740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f8e13a6980 0x55f8e13bd5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:18.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.401+0000 7f794262e740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f8e13a72a0 0x55f8e13a90e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:18.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.401+0000 7f794262e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f8e13b1970 0x55f8e13c9770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:18.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.401+0000 7f794262e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f79381b1fc0 con 0x55f8e13b1970 2024-05-12T22:16:18.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.401+0000 7f794262e740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f79381af890 con 0x55f8e13a6980 2024-05-12T22:16:18.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.401+0000 7f794262e740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f79381ad700 con 0x55f8e13a72a0 2024-05-12T22:16:18.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.401+0000 7f793ce00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f8e13a6980 0x55f8e13bd5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:18.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.401+0000 7f7937e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f8e13a72a0 0x55f8e13a90e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:18.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.401+0000 7f793ce00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f8e13a6980 0x55f8e13bd5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:54798/0 (socket says 172.21.15.28:54798) 2024-05-12T22:16:18.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.401+0000 7f793ce00640 1 -- 172.21.15.28:0/4038045657 learned_addr learned my addr 172.21.15.28:0/4038045657 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:16:18.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.401+0000 7f793d800640 1 --2- 172.21.15.28:0/4038045657 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f8e13b1970 0x55f8e13c9770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:18.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.401+0000 7f7937e00640 1 -- 172.21.15.28:0/4038045657 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f8e13a6980 msgr2=0x55f8e13bd5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:16:18.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.401+0000 7f7937e00640 1 --2- 172.21.15.28:0/4038045657 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f8e13a6980 0x55f8e13bd5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:18.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.401+0000 7f7937e00640 1 -- 172.21.15.28:0/4038045657 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f8e13b1970 msgr2=0x55f8e13c9770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:16:18.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.401+0000 7f7937e00640 1 --2- 172.21.15.28:0/4038045657 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f8e13b1970 0x55f8e13c9770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:18.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.401+0000 7f7937e00640 1 -- 172.21.15.28:0/4038045657 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f8e13a70d0 con 0x55f8e13a72a0 2024-05-12T22:16:18.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.401+0000 7f793ce00640 1 --2- 172.21.15.28:0/4038045657 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f8e13a6980 0x55f8e13bd5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:16:18.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.402+0000 7f7937e00640 1 --2- 172.21.15.28:0/4038045657 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f8e13a72a0 0x55f8e13a90e0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f79240cced0 tx=0x7f79240d1ca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dd243b2d76c4d6fa server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:16:18.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.402+0000 7f7929600640 1 -- 172.21.15.28:0/4038045657 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f792409a050 con 0x55f8e13a72a0 2024-05-12T22:16:18.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.402+0000 7f7929600640 1 -- 172.21.15.28:0/4038045657 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f79240983c0 con 0x55f8e13a72a0 2024-05-12T22:16:18.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.402+0000 7f7929600640 1 -- 172.21.15.28:0/4038045657 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f792409b1a0 con 0x55f8e13a72a0 2024-05-12T22:16:18.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.403+0000 7f794262e740 1 -- 172.21.15.28:0/4038045657 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f8e13a72a0 msgr2=0x55f8e13a90e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:16:18.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.403+0000 7f794262e740 1 --2- 172.21.15.28:0/4038045657 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f8e13a72a0 0x55f8e13a90e0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f79240cced0 tx=0x7f79240d1ca0 comp rx=0 tx=0).stop 2024-05-12T22:16:18.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.403+0000 7f794262e740 1 -- 172.21.15.28:0/4038045657 shutdown_connections 2024-05-12T22:16:18.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.403+0000 7f794262e740 1 --2- 172.21.15.28:0/4038045657 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f8e13b1970 0x55f8e13c9770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:18.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.403+0000 7f794262e740 1 --2- 172.21.15.28:0/4038045657 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f8e13a72a0 0x55f8e13a90e0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:18.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.403+0000 7f794262e740 1 --2- 172.21.15.28:0/4038045657 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f8e13a6980 0x55f8e13bd5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:18.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.403+0000 7f794262e740 1 -- 172.21.15.28:0/4038045657 >> 172.21.15.28:0/4038045657 conn(0x55f8e13a2ca0 msgr2=0x55f8e13ad930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:16:18.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.403+0000 7f794262e740 1 -- 172.21.15.28:0/4038045657 shutdown_connections 2024-05-12T22:16:18.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.403+0000 7f794262e740 1 -- 172.21.15.28:0/4038045657 wait complete. 2024-05-12T22:16:18.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.404+0000 7f794262e740 1 -- start start 2024-05-12T22:16:18.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.404+0000 7f794262e740 1 Processor -- start 2024-05-12T22:16:18.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.404+0000 7f794262e740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f8e13a6980 0x55f8e13b9980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:18.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.404+0000 7f794262e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f8e13a72a0 0x55f8e13b9ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:18.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.404+0000 7f794262e740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f8e13b1970 0x55f8e13b4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:18.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.404+0000 7f794262e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f79381add50 con 0x55f8e13a72a0 2024-05-12T22:16:18.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.404+0000 7f794262e740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f79381af1c0 con 0x55f8e13a6980 2024-05-12T22:16:18.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.404+0000 7f794262e740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f79381ae420 con 0x55f8e13b1970 2024-05-12T22:16:18.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.404+0000 7f794262e740 10 client.0 _ll_register_callbacks cb 0x7f792c0045b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:16:18.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.405+0000 7f7937e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f8e13a72a0 0x55f8e13b9ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:18.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.405+0000 7f793ce00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f8e13a6980 0x55f8e13b9980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:18.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.405+0000 7f793d800640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f8e13b1970 0x55f8e13b4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:18.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.405+0000 7f793ce00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f8e13a6980 0x55f8e13b9980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:54810/0 (socket says 172.21.15.28:54810) 2024-05-12T22:16:18.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.405+0000 7f793ce00640 1 -- 172.21.15.28:0/2543756630 learned_addr learned my addr 172.21.15.28:0/2543756630 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:16:18.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.405+0000 7f793d800640 1 -- 172.21.15.28:0/2543756630 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f8e13a6980 msgr2=0x55f8e13b9980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:16:18.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.405+0000 7f793d800640 1 --2- 172.21.15.28:0/2543756630 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f8e13a6980 0x55f8e13b9980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:18.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.405+0000 7f793d800640 1 -- 172.21.15.28:0/2543756630 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f8e13a72a0 msgr2=0x55f8e13b9ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:16:18.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.405+0000 7f793d800640 1 --2- 172.21.15.28:0/2543756630 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f8e13a72a0 0x55f8e13b9ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:18.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.405+0000 7f793d800640 1 -- 172.21.15.28:0/2543756630 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f8e13b56f0 con 0x55f8e13b1970 2024-05-12T22:16:18.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.405+0000 7f7937e00640 1 --2- 172.21.15.28:0/2543756630 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f8e13a72a0 0x55f8e13b9ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:16:18.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.405+0000 7f793ce00640 1 --2- 172.21.15.28:0/2543756630 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f8e13a6980 0x55f8e13b9980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:16:18.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.406+0000 7f793d800640 1 --2- 172.21.15.28:0/2543756630 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f8e13b1970 0x55f8e13b4ec0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f7930011ec0 tx=0x7f7930004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=da9099522c44fa08 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:16:18.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.406+0000 7f791ea00640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-12T22:16:18.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.406+0000 7f791ea00640 1 -- 172.21.15.28:0/2543756630 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f793002e060 con 0x55f8e13b1970 2024-05-12T22:16:18.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.406+0000 7f794262e740 1 -- 172.21.15.28:0/2543756630 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f79240983c0 con 0x55f8e13b1970 2024-05-12T22:16:18.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.406+0000 7f791ea00640 1 -- 172.21.15.28:0/2543756630 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8513+0+0 (secure 0 0 0) 0x7f793000f9f0 con 0x55f8e13b1970 2024-05-12T22:16:18.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.406+0000 7f791ea00640 1 -- 172.21.15.28:0/2543756630 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f793000fbf0 con 0x55f8e13b1970 2024-05-12T22:16:18.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.406+0000 7f794262e740 10 client.26794 fetch_fsmap learned FSMap version 26 2024-05-12T22:16:18.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.406+0000 7f794262e740 1 -- 172.21.15.28:0/2543756630 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55f8e1430390 con 0x55f8e13b1970 2024-05-12T22:16:18.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.407+0000 7f791ea00640 1 -- 172.21.15.28:0/2543756630 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f79240983c0 con 0x55f8e13b1970 2024-05-12T22:16:18.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.407+0000 7f791ea00640 1 -- 172.21.15.28:0/2543756630 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f793001e400 con 0x55f8e13b1970 2024-05-12T22:16:18.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.407+0000 7f794262e740 10 client.26794 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:16:18.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.407+0000 7f794262e740 10 client.26794 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.g 2024-05-12T22:16:18.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.407+0000 7f794262e740 20 client.26794 populate_metadata read hostname 'smithi028' 2024-05-12T22:16:18.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.407+0000 7f794262e740 1 --2- 172.21.15.28:0/2543756630 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x55f8e1430b70 0x55f8e1432f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:18.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.407+0000 7f794262e740 4 client.26794 mds_command: new command op to 24532 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-12T22:16:18.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.407+0000 7f794262e740 1 -- 172.21.15.28:0/2543756630 --> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f8e13134b0 con 0x55f8e1430b70 2024-05-12T22:16:18.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.407+0000 7f793ce00640 1 --2- 172.21.15.28:0/2543756630 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x55f8e1430b70 0x55f8e1432f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:18.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.408+0000 7f793ce00640 1 --2- 172.21.15.28:0/2543756630 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x55f8e1430b70 0x55f8e1432f90 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=fa8bdc3e7e2578b3 server_cookie=a5193bc5a2e40e21 in_seq=0 out_seq=0 2024-05-12T22:16:18.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.408+0000 7f791ea00640 10 client.26794 ms_handle_connect on v2:172.21.15.28:6838/123892158 2024-05-12T22:16:18.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.408+0000 7f791ea00640 1 -- 172.21.15.28:0/2543756630 <== mds.2 v2:172.21.15.28:6838/123892158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f8e13134b0 con 0x55f8e1430b70 2024-05-12T22:16:18.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.408+0000 7f791ea00640 10 client.26794 handle_command_reply: tid=0 2024-05-12T22:16:18.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.408+0000 7f794262e740 1 client.26794 shutdown 2024-05-12T22:16:18.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.408+0000 7f794262e740 1 -- 172.21.15.28:0/2543756630 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f8e13b1970 msgr2=0x55f8e13b4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:16:18.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.408+0000 7f794262e740 1 --2- 172.21.15.28:0/2543756630 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f8e13b1970 0x55f8e13b4ec0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f7930011ec0 tx=0x7f7930004a40 comp rx=0 tx=0).stop 2024-05-12T22:16:18.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.408+0000 7f794262e740 1 -- 172.21.15.28:0/2543756630 shutdown_connections 2024-05-12T22:16:18.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.408+0000 7f794262e740 1 --2- 172.21.15.28:0/2543756630 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f8e13b1970 0x55f8e13b4ec0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:18.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.408+0000 7f794262e740 1 --2- 172.21.15.28:0/2543756630 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f8e13a72a0 0x55f8e13b9ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:18.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.408+0000 7f794262e740 1 --2- 172.21.15.28:0/2543756630 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x55f8e1430b70 0x55f8e1432f90 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:18.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.408+0000 7f794262e740 1 --2- 172.21.15.28:0/2543756630 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f8e13a6980 0x55f8e13b9980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:18.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.408+0000 7f794262e740 1 -- 172.21.15.28:0/2543756630 >> 172.21.15.28:0/2543756630 conn(0x55f8e13a2ca0 msgr2=0x55f8e13ad0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:16:18.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.408+0000 7f791ea00640 0 client.26794 ms_handle_reset on v2:172.21.15.28:6838/123892158 2024-05-12T22:16:18.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.409+0000 7f794262e740 1 -- 172.21.15.28:0/2543756630 shutdown_connections 2024-05-12T22:16:18.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.409+0000 7f794262e740 1 -- 172.21.15.28:0/2543756630 wait complete. 2024-05-12T22:16:18.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.409+0000 7f794262e740 20 client.26794 trim_cache size 0 max 16384 2024-05-12T22:16:18.417 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-05-12T22:16:18.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.410+0000 7f793f800640 1 -- 172.21.15.28:0/2987443072 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f79200a2d80 msgr2=0x7f79200a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:16:18.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.410+0000 7f793f800640 1 --2- 172.21.15.28:0/2987443072 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f79200a2d80 0x7f79200a5240 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f7930011630 tx=0x7f7930009500 comp rx=0 tx=0).stop 2024-05-12T22:16:18.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.410+0000 7f793f800640 1 -- 172.21.15.28:0/2987443072 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7938074420 msgr2=0x7f793819a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:16:18.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.410+0000 7f793f800640 1 --2- 172.21.15.28:0/2987443072 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7938074420 0x7f793819a8a0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f792400e9f0 tx=0x7f792400eeb0 comp rx=0 tx=0).stop 2024-05-12T22:16:18.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.410+0000 7f793f800640 1 -- 172.21.15.28:0/2987443072 shutdown_connections 2024-05-12T22:16:18.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.410+0000 7f793f800640 1 --2- 172.21.15.28:0/2987443072 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f79200a2d80 0x7f79200a5240 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:18.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.410+0000 7f793f800640 1 --2- 172.21.15.28:0/2987443072 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f793810e110 0x7f793819ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:18.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.410+0000 7f793f800640 1 --2- 172.21.15.28:0/2987443072 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7938074420 0x7f793819a8a0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:18.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.410+0000 7f793f800640 1 --2- 172.21.15.28:0/2987443072 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7938073b00 0x7f793819a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:18.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.410+0000 7f793f800640 1 -- 172.21.15.28:0/2987443072 >> 172.21.15.28:0/2987443072 conn(0x7f7938078c80 msgr2=0x7f79380fe740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:16:18.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.410+0000 7f793f800640 1 -- 172.21.15.28:0/2987443072 shutdown_connections 2024-05-12T22:16:18.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.410+0000 7f793f800640 1 -- 172.21.15.28:0/2987443072 wait complete. 2024-05-12T22:16:18.426 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:16:18 smithi028 ceph-mon[29847]: pgmap v224: 97 pgs: 97 active+clean; 97 MiB data, 757 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 234 B/s wr, 7 op/s 2024-05-12T22:16:18.426 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:16:18 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/930650503' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:16:18.428 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-05-12T22:16:18.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:16:18 smithi186 ceph-mon[34785]: pgmap v224: 97 pgs: 97 active+clean; 97 MiB data, 757 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 234 B/s wr, 7 op/s 2024-05-12T22:16:18.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:16:18 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/930650503' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:16:18.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:16:18 smithi086 ceph-mon[34297]: pgmap v224: 97 pgs: 97 active+clean; 97 MiB data, 757 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 234 B/s wr, 7 op/s 2024-05-12T22:16:18.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:16:18 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/930650503' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:16:18.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.623+0000 7f6d56600640 1 -- 172.21.15.28:0/906423166 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d501010c0 msgr2=0x7f6d5010d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:16:18.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.623+0000 7f6d56600640 1 --2- 172.21.15.28:0/906423166 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d501010c0 0x7f6d5010d5d0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f6d48009900 tx=0x7f6d4802a3f0 comp rx=0 tx=0).stop 2024-05-12T22:16:18.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.623+0000 7f6d56600640 1 -- 172.21.15.28:0/906423166 shutdown_connections 2024-05-12T22:16:18.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.623+0000 7f6d56600640 1 --2- 172.21.15.28:0/906423166 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d501010c0 0x7f6d5010d5d0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:18.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.623+0000 7f6d56600640 1 --2- 172.21.15.28:0/906423166 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d50100700 0x7f6d50100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:18.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.623+0000 7f6d56600640 1 --2- 172.21.15.28:0/906423166 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6d50106710 0x7f6d50106af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:18.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.623+0000 7f6d56600640 1 -- 172.21.15.28:0/906423166 >> 172.21.15.28:0/906423166 conn(0x7f6d50077830 msgr2=0x7f6d50077c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:16:18.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.623+0000 7f6d56600640 1 -- 172.21.15.28:0/906423166 shutdown_connections 2024-05-12T22:16:18.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.623+0000 7f6d56600640 1 -- 172.21.15.28:0/906423166 wait complete. 2024-05-12T22:16:18.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.625+0000 7f6d56600640 1 Processor -- start 2024-05-12T22:16:18.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.625+0000 7f6d56600640 1 -- start start 2024-05-12T22:16:18.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.625+0000 7f6d56600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d50100700 0x7f6d5010e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:18.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.625+0000 7f6d56600640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6d501010c0 0x7f6d5010e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:18.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.625+0000 7f6d56600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d50106710 0x7f6d5010eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:18.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.626+0000 7f6d56600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6d50111c70 con 0x7f6d50100700 2024-05-12T22:16:18.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.626+0000 7f6d56600640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f6d50111af0 con 0x7f6d50106710 2024-05-12T22:16:18.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.626+0000 7f6d56600640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f6d50111df0 con 0x7f6d501010c0 2024-05-12T22:16:18.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.626+0000 7f6d4ea00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6d501010c0 0x7f6d5010e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:18.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.626+0000 7f6d4fe00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d50106710 0x7f6d5010eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:18.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.626+0000 7f6d4f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d50100700 0x7f6d5010e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:18.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.626+0000 7f6d4fe00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d50106710 0x7f6d5010eae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:54826/0 (socket says 172.21.15.28:54826) 2024-05-12T22:16:18.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.626+0000 7f6d4ea00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6d501010c0 0x7f6d5010e5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:47206/0 (socket says 172.21.15.28:47206) 2024-05-12T22:16:18.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.626+0000 7f6d4fe00640 1 -- 172.21.15.28:0/3314522553 learned_addr learned my addr 172.21.15.28:0/3314522553 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:16:18.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.626+0000 7f6d4ea00640 1 -- 172.21.15.28:0/3314522553 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d50106710 msgr2=0x7f6d5010eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:16:18.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.626+0000 7f6d4ea00640 1 --2- 172.21.15.28:0/3314522553 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d50106710 0x7f6d5010eae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:18.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.626+0000 7f6d4ea00640 1 -- 172.21.15.28:0/3314522553 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d50100700 msgr2=0x7f6d5010e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:16:18.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.626+0000 7f6d4ea00640 1 --2- 172.21.15.28:0/3314522553 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d50100700 0x7f6d5010e060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:18.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.626+0000 7f6d4ea00640 1 -- 172.21.15.28:0/3314522553 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d501a65d0 con 0x7f6d501010c0 2024-05-12T22:16:18.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.627+0000 7f6d4ea00640 1 --2- 172.21.15.28:0/3314522553 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6d501010c0 0x7f6d5010e5a0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f6d3c00e9f0 tx=0x7f6d3c00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:16:18.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.627+0000 7f6d47e00640 1 -- 172.21.15.28:0/3314522553 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d3c00ccc0 con 0x7f6d501010c0 2024-05-12T22:16:18.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.627+0000 7f6d47e00640 1 -- 172.21.15.28:0/3314522553 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6d3c010910 con 0x7f6d501010c0 2024-05-12T22:16:18.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.627+0000 7f6d56600640 1 -- 172.21.15.28:0/3314522553 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d501a68c0 con 0x7f6d501010c0 2024-05-12T22:16:18.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.627+0000 7f6d47e00640 1 -- 172.21.15.28:0/3314522553 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d3c018960 con 0x7f6d501010c0 2024-05-12T22:16:18.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.627+0000 7f6d56600640 1 -- 172.21.15.28:0/3314522553 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d501a6e10 con 0x7f6d501010c0 2024-05-12T22:16:18.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.630+0000 7f6d59404740 1 Processor -- start 2024-05-12T22:16:18.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.631+0000 7f6d59404740 1 -- start start 2024-05-12T22:16:18.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.631+0000 7f6d59404740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561755b5a980 0x561755b5ad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:18.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.631+0000 7f6d59404740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561755b5b2a0 0x561755b65430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:18.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.631+0000 7f6d59404740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561755b65970 0x561755b67d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:18.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.631+0000 7f6d59404740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x561755abfa30 con 0x561755b5b2a0 2024-05-12T22:16:18.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.631+0000 7f6d59404740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x561755a67700 con 0x561755b65970 2024-05-12T22:16:18.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.631+0000 7f6d59404740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x561755abb9f0 con 0x561755b5a980 2024-05-12T22:16:18.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.631+0000 7f6d4fe00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561755b5a980 0x561755b5ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:18.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.631+0000 7f6d4fe00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561755b5a980 0x561755b5ad60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:47214/0 (socket says 172.21.15.28:47214) 2024-05-12T22:16:18.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.631+0000 7f6d4fe00640 1 -- 172.21.15.28:0/1420869153 learned_addr learned my addr 172.21.15.28:0/1420869153 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:16:18.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.631+0000 7f6d4f400640 1 --2- 172.21.15.28:0/1420869153 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561755b5b2a0 0x561755b65430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:18.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.632+0000 7f6d4fe00640 1 -- 172.21.15.28:0/1420869153 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561755b65970 msgr2=0x561755b67d60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-12T22:16:18.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.632+0000 7f6d4fe00640 1 --2- 172.21.15.28:0/1420869153 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561755b65970 0x561755b67d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:18.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.632+0000 7f6d4fe00640 1 -- 172.21.15.28:0/1420869153 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561755b5b2a0 msgr2=0x561755b65430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:16:18.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.632+0000 7f6d4fe00640 1 --2- 172.21.15.28:0/1420869153 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561755b5b2a0 0x561755b65430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:18.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.632+0000 7f6d4fe00640 1 -- 172.21.15.28:0/1420869153 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5617557d33f0 con 0x561755b5a980 2024-05-12T22:16:18.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.632+0000 7f6d4fe00640 1 --2- 172.21.15.28:0/1420869153 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561755b5a980 0x561755b5ad60 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f6d48005b50 tx=0x7f6d48005b80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a26501a6a10d6165 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:16:18.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.632+0000 7f6d44c00640 1 -- 172.21.15.28:0/1420869153 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d48040070 con 0x561755b5a980 2024-05-12T22:16:18.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.632+0000 7f6d44c00640 1 -- 172.21.15.28:0/1420869153 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6d48004480 con 0x561755b5a980 2024-05-12T22:16:18.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.632+0000 7f6d44c00640 1 -- 172.21.15.28:0/1420869153 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d48035040 con 0x561755b5a980 2024-05-12T22:16:18.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.633+0000 7f6d47e00640 1 -- 172.21.15.28:0/3314522553 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7f6d3c010ac0 con 0x7f6d501010c0 2024-05-12T22:16:18.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.633+0000 7f6d59404740 1 -- 172.21.15.28:0/1420869153 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561755b5a980 msgr2=0x561755b5ad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:16:18.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.633+0000 7f6d59404740 1 --2- 172.21.15.28:0/1420869153 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561755b5a980 0x561755b5ad60 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f6d48005b50 tx=0x7f6d48005b80 comp rx=0 tx=0).stop 2024-05-12T22:16:18.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.633+0000 7f6d59404740 1 -- 172.21.15.28:0/1420869153 shutdown_connections 2024-05-12T22:16:18.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.633+0000 7f6d59404740 1 --2- 172.21.15.28:0/1420869153 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561755b65970 0x561755b67d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:18.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.633+0000 7f6d59404740 1 --2- 172.21.15.28:0/1420869153 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561755b5b2a0 0x561755b65430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:18.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.633+0000 7f6d59404740 1 --2- 172.21.15.28:0/1420869153 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561755b5a980 0x561755b5ad60 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:18.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.633+0000 7f6d59404740 1 -- 172.21.15.28:0/1420869153 >> 172.21.15.28:0/1420869153 conn(0x561755b56ca0 msgr2=0x561755b57080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:16:18.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.633+0000 7f6d59404740 1 -- 172.21.15.28:0/1420869153 shutdown_connections 2024-05-12T22:16:18.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.634+0000 7f6d59404740 1 -- 172.21.15.28:0/1420869153 wait complete. 2024-05-12T22:16:18.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.634+0000 7f6d59404740 1 -- start start 2024-05-12T22:16:18.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.634+0000 7f6d59404740 1 Processor -- start 2024-05-12T22:16:18.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.635+0000 7f6d59404740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561755b5a980 0x561755b62a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:18.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.635+0000 7f6d59404740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561755b5b2a0 0x561755b62f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:18.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.635+0000 7f6d59404740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561755b65970 0x561755b63490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:18.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.635+0000 7f6d59404740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x561755b68700 con 0x561755b5a980 2024-05-12T22:16:18.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.635+0000 7f6d47e00640 1 --2- 172.21.15.28:0/3314522553 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f6d340a2c10 0x7f6d340a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:18.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.635+0000 7f6d59404740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x561755a7aa70 con 0x561755b65970 2024-05-12T22:16:18.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.635+0000 7f6d59404740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x561755b68880 con 0x561755b5b2a0 2024-05-12T22:16:18.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.635+0000 7f6d59404740 10 client.0 _ll_register_callbacks cb 0x561755a5dba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:16:18.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.635+0000 7f6d47e00640 1 -- 172.21.15.28:0/3314522553 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8513+0+0 (secure 0 0 0) 0x7f6d3c0cf840 con 0x7f6d501010c0 2024-05-12T22:16:18.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.635+0000 7f6d4f400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561755b5b2a0 0x561755b62f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:18.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.635+0000 7f6d4f400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561755b5b2a0 0x561755b62f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:47216/0 (socket says 172.21.15.28:47216) 2024-05-12T22:16:18.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.635+0000 7f6d4ea00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561755b65970 0x561755b63490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:18.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.635+0000 7f6d4f400640 1 -- 172.21.15.28:0/3946839626 learned_addr learned my addr 172.21.15.28:0/3946839626 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:16:18.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.635+0000 7f6d4fe00640 1 --2- 172.21.15.28:0/3946839626 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561755b5a980 0x561755b62a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:18.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.635+0000 7f6d4f400640 1 -- 172.21.15.28:0/3946839626 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561755b65970 msgr2=0x561755b63490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:16:18.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.635+0000 7f6d4f400640 1 --2- 172.21.15.28:0/3946839626 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561755b65970 0x561755b63490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:18.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.635+0000 7f6d4f400640 1 -- 172.21.15.28:0/3946839626 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561755b5a980 msgr2=0x561755b62a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:16:18.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.635+0000 7f6d4f400640 1 --2- 172.21.15.28:0/3946839626 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561755b5a980 0x561755b62a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:18.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.635+0000 7f6d4f400640 1 -- 172.21.15.28:0/3946839626 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56175589d7d0 con 0x561755b5b2a0 2024-05-12T22:16:18.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.636+0000 7f6d4fe00640 1 --2- 172.21.15.28:0/3946839626 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561755b5a980 0x561755b62a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-12T22:16:18.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.636+0000 7f6d4fe00640 1 --2- 172.21.15.28:0/3314522553 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f6d340a2c10 0x7f6d340a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:18.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.636+0000 7f6d4f400640 1 --2- 172.21.15.28:0/3946839626 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561755b5b2a0 0x561755b62f50 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f6d40011d10 tx=0x7f6d4000f790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e6dd4c787a1d8347 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:16:18.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.636+0000 7f6d39600640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-12T22:16:18.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.636+0000 7f6d39600640 1 -- 172.21.15.28:0/3946839626 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d40004b20 con 0x561755b5b2a0 2024-05-12T22:16:18.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.636+0000 7f6d39600640 1 -- 172.21.15.28:0/3946839626 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8513+0+0 (secure 0 0 0) 0x7f6d40017050 con 0x561755b5b2a0 2024-05-12T22:16:18.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.636+0000 7f6d59404740 1 -- 172.21.15.28:0/3946839626 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6d48004480 con 0x561755b5b2a0 2024-05-12T22:16:18.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.636+0000 7f6d4fe00640 1 --2- 172.21.15.28:0/3314522553 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f6d340a2c10 0x7f6d340a50d0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f6d48031660 tx=0x7f6d48005b80 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:16:18.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.636+0000 7f6d39600640 1 -- 172.21.15.28:0/3946839626 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f6d40004cc0 con 0x561755b5b2a0 2024-05-12T22:16:18.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.636+0000 7f6d59404740 10 client.26806 fetch_fsmap learned FSMap version 26 2024-05-12T22:16:18.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.637+0000 7f6d59404740 1 -- 172.21.15.28:0/3946839626 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x561755b7d320 con 0x561755b5b2a0 2024-05-12T22:16:18.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.637+0000 7f6d39600640 1 -- 172.21.15.28:0/3946839626 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6d48004480 con 0x561755b5b2a0 2024-05-12T22:16:18.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.637+0000 7f6d39600640 1 -- 172.21.15.28:0/3946839626 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f6d40019070 con 0x561755b5b2a0 2024-05-12T22:16:18.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.637+0000 7f6d59404740 10 client.26806 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:16:18.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.637+0000 7f6d59404740 10 client.26806 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-05-12T22:16:18.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.637+0000 7f6d59404740 20 client.26806 populate_metadata read hostname 'smithi028' 2024-05-12T22:16:18.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.637+0000 7f6d59404740 1 --2- 172.21.15.28:0/3946839626 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x561755b7e830 0x561755b80c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:18.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.637+0000 7f6d59404740 4 client.26806 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-12T22:16:18.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.637+0000 7f6d59404740 1 -- 172.21.15.28:0/3946839626 --> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561755acd160 con 0x561755b7e830 2024-05-12T22:16:18.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.638+0000 7f6d4ea00640 1 --2- 172.21.15.28:0/3946839626 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x561755b7e830 0x561755b80c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:18.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.638+0000 7f6d4ea00640 1 --2- 172.21.15.28:0/3946839626 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x561755b7e830 0x561755b80c50 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=60ad4f76a1c4472a server_cookie=8639bfc559d5932b in_seq=0 out_seq=0 2024-05-12T22:16:18.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.638+0000 7f6d39600640 10 client.26806 ms_handle_connect on v2:172.21.15.28:6840/761183870 2024-05-12T22:16:18.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.639+0000 7f6d39600640 1 -- 172.21.15.28:0/3946839626 <== mds.1 v2:172.21.15.28:6840/761183870 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561755acd160 con 0x561755b7e830 2024-05-12T22:16:18.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.639+0000 7f6d39600640 10 client.26806 handle_command_reply: tid=0 2024-05-12T22:16:18.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.639+0000 7f6d59404740 1 client.26806 shutdown 2024-05-12T22:16:18.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.639+0000 7f6d59404740 1 -- 172.21.15.28:0/3946839626 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561755b5b2a0 msgr2=0x561755b62f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:16:18.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.639+0000 7f6d59404740 1 --2- 172.21.15.28:0/3946839626 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561755b5b2a0 0x561755b62f50 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f6d40011d10 tx=0x7f6d4000f790 comp rx=0 tx=0).stop 2024-05-12T22:16:18.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.639+0000 7f6d59404740 1 -- 172.21.15.28:0/3946839626 shutdown_connections 2024-05-12T22:16:18.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.639+0000 7f6d59404740 1 --2- 172.21.15.28:0/3946839626 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x561755b7e830 0x561755b80c50 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:18.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.639+0000 7f6d59404740 1 --2- 172.21.15.28:0/3946839626 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561755b65970 0x561755b63490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:18.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.639+0000 7f6d59404740 1 --2- 172.21.15.28:0/3946839626 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561755b5b2a0 0x561755b62f50 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:18.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.639+0000 7f6d59404740 1 --2- 172.21.15.28:0/3946839626 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561755b5a980 0x561755b62a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:18.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.639+0000 7f6d39600640 0 client.26806 ms_handle_reset on v2:172.21.15.28:6840/761183870 2024-05-12T22:16:18.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.639+0000 7f6d59404740 1 -- 172.21.15.28:0/3946839626 >> 172.21.15.28:0/3946839626 conn(0x561755b56ca0 msgr2=0x561755b5caa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:16:18.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.640+0000 7f6d59404740 1 -- 172.21.15.28:0/3946839626 shutdown_connections 2024-05-12T22:16:18.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.640+0000 7f6d59404740 1 -- 172.21.15.28:0/3946839626 wait complete. 2024-05-12T22:16:18.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.640+0000 7f6d59404740 20 client.26806 trim_cache size 0 max 16384 2024-05-12T22:16:18.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.655+0000 7f6d59404740 1 Processor -- start 2024-05-12T22:16:18.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.655+0000 7f6d59404740 1 -- start start 2024-05-12T22:16:18.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.656+0000 7f6d59404740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561755b5a980 0x561755b632b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:18.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.656+0000 7f6d59404740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561755b5b2a0 0x561755b86500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:18.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.656+0000 7f6d59404740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561755b65970 0x561755b86a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:18.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.656+0000 7f6d59404740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6d501adbc0 con 0x561755b5b2a0 2024-05-12T22:16:18.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.656+0000 7f6d59404740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f6d501ab490 con 0x561755b65970 2024-05-12T22:16:18.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.656+0000 7f6d59404740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f6d501a9320 con 0x561755b5a980 2024-05-12T22:16:18.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.656+0000 7f6d4fe00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561755b65970 0x561755b86a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:18.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.656+0000 7f6d4f400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561755b5a980 0x561755b632b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:18.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.656+0000 7f6d4ea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561755b5b2a0 0x561755b86500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:18.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.656+0000 7f6d4fe00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561755b65970 0x561755b86a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:54850/0 (socket says 172.21.15.28:54850) 2024-05-12T22:16:18.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.656+0000 7f6d4f400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561755b5a980 0x561755b632b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:47218/0 (socket says 172.21.15.28:47218) 2024-05-12T22:16:18.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.656+0000 7f6d4fe00640 1 -- 172.21.15.28:0/453144530 learned_addr learned my addr 172.21.15.28:0/453144530 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:16:18.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.657+0000 7f6d4fe00640 1 -- 172.21.15.28:0/453144530 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561755b5a980 msgr2=0x561755b632b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:16:18.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.657+0000 7f6d4fe00640 1 --2- 172.21.15.28:0/453144530 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561755b5a980 0x561755b632b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:18.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.657+0000 7f6d4fe00640 1 -- 172.21.15.28:0/453144530 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561755b5b2a0 msgr2=0x561755b86500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:16:18.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.657+0000 7f6d4fe00640 1 --2- 172.21.15.28:0/453144530 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561755b5b2a0 0x561755b86500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:18.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.657+0000 7f6d4fe00640 1 -- 172.21.15.28:0/453144530 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561755b60f60 con 0x561755b65970 2024-05-12T22:16:18.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.657+0000 7f6d4fe00640 1 --2- 172.21.15.28:0/453144530 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561755b65970 0x561755b86a40 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f6d480096c0 tx=0x7f6d48048040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b5e248d9deb364c6 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:16:18.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.657+0000 7f6d3be00640 1 -- 172.21.15.28:0/453144530 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d48040030 con 0x561755b65970 2024-05-12T22:16:18.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.657+0000 7f6d3be00640 1 -- 172.21.15.28:0/453144530 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6d48005500 con 0x561755b65970 2024-05-12T22:16:18.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.657+0000 7f6d3be00640 1 -- 172.21.15.28:0/453144530 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d48035040 con 0x561755b65970 2024-05-12T22:16:18.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.658+0000 7f6d59404740 1 -- 172.21.15.28:0/453144530 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561755b65970 msgr2=0x561755b86a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:16:18.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.658+0000 7f6d59404740 1 --2- 172.21.15.28:0/453144530 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561755b65970 0x561755b86a40 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f6d480096c0 tx=0x7f6d48048040 comp rx=0 tx=0).stop 2024-05-12T22:16:18.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.658+0000 7f6d59404740 1 -- 172.21.15.28:0/453144530 shutdown_connections 2024-05-12T22:16:18.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.658+0000 7f6d59404740 1 --2- 172.21.15.28:0/453144530 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561755b65970 0x561755b86a40 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:18.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.658+0000 7f6d59404740 1 --2- 172.21.15.28:0/453144530 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561755b5b2a0 0x561755b86500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:18.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.658+0000 7f6d59404740 1 --2- 172.21.15.28:0/453144530 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561755b5a980 0x561755b632b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:18.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.658+0000 7f6d59404740 1 -- 172.21.15.28:0/453144530 >> 172.21.15.28:0/453144530 conn(0x561755b56ca0 msgr2=0x561755b55dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:16:18.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.658+0000 7f6d59404740 1 -- 172.21.15.28:0/453144530 shutdown_connections 2024-05-12T22:16:18.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.658+0000 7f6d59404740 1 -- 172.21.15.28:0/453144530 wait complete. 2024-05-12T22:16:18.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.659+0000 7f6d59404740 1 -- start start 2024-05-12T22:16:18.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.659+0000 7f6d59404740 1 Processor -- start 2024-05-12T22:16:18.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.660+0000 7f6d59404740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561755b5a980 0x561755b72000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:18.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.660+0000 7f6d59404740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561755b5b2a0 0x561755b72540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:18.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.660+0000 7f6d59404740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561755b65970 0x561755b72a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:18.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.660+0000 7f6d59404740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6d501a9970 con 0x561755b65970 2024-05-12T22:16:18.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.660+0000 7f6d59404740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f6d501aadc0 con 0x561755b5b2a0 2024-05-12T22:16:18.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.660+0000 7f6d59404740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f6d501aa020 con 0x561755b5a980 2024-05-12T22:16:18.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.660+0000 7f6d59404740 10 client.0 _ll_register_callbacks cb 0x7f6d40011920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:16:18.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.660+0000 7f6d4f400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561755b5a980 0x561755b72000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:18.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.660+0000 7f6d4fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561755b65970 0x561755b72a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:18.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.660+0000 7f6d4f400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561755b5a980 0x561755b72000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:47228/0 (socket says 172.21.15.28:47228) 2024-05-12T22:16:18.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.660+0000 7f6d4f400640 1 -- 172.21.15.28:0/697397329 learned_addr learned my addr 172.21.15.28:0/697397329 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:16:18.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.660+0000 7f6d4ea00640 1 --2- 172.21.15.28:0/697397329 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561755b5b2a0 0x561755b72540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:18.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.660+0000 7f6d4f400640 1 -- 172.21.15.28:0/697397329 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561755b5b2a0 msgr2=0x561755b72540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:16:18.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.660+0000 7f6d4f400640 1 --2- 172.21.15.28:0/697397329 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561755b5b2a0 0x561755b72540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:18.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.660+0000 7f6d4f400640 1 -- 172.21.15.28:0/697397329 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561755b65970 msgr2=0x561755b72a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:16:18.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.660+0000 7f6d4f400640 1 --2- 172.21.15.28:0/697397329 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561755b65970 0x561755b72a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:18.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.660+0000 7f6d4f400640 1 -- 172.21.15.28:0/697397329 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561755be4010 con 0x561755b5a980 2024-05-12T22:16:18.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.661+0000 7f6d4ea00640 1 --2- 172.21.15.28:0/697397329 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561755b5b2a0 0x561755b72540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:16:18.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.661+0000 7f6d4fe00640 1 --2- 172.21.15.28:0/697397329 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561755b65970 0x561755b72a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:16:18.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.661+0000 7f6d4f400640 1 --2- 172.21.15.28:0/697397329 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561755b5a980 0x561755b72000 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x561755b62270 tx=0x7f6d400093e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=422db35883fd6b90 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:16:18.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.661+0000 7f6d39600640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-12T22:16:18.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.661+0000 7f6d39600640 1 -- 172.21.15.28:0/697397329 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d40030060 con 0x561755b5a980 2024-05-12T22:16:18.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.661+0000 7f6d59404740 1 -- 172.21.15.28:0/697397329 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6d48005500 con 0x561755b5a980 2024-05-12T22:16:18.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.661+0000 7f6d39600640 1 -- 172.21.15.28:0/697397329 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8513+0+0 (secure 0 0 0) 0x7f6d40011e00 con 0x561755b5a980 2024-05-12T22:16:18.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.661+0000 7f6d39600640 1 -- 172.21.15.28:0/697397329 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f6d40022c80 con 0x561755b5a980 2024-05-12T22:16:18.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.661+0000 7f6d59404740 10 client.26812 fetch_fsmap learned FSMap version 26 2024-05-12T22:16:18.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.662+0000 7f6d59404740 1 -- 172.21.15.28:0/697397329 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x561755be52f0 con 0x561755b5a980 2024-05-12T22:16:18.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.662+0000 7f6d39600640 1 -- 172.21.15.28:0/697397329 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6d48005500 con 0x561755b5a980 2024-05-12T22:16:18.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.662+0000 7f6d39600640 1 -- 172.21.15.28:0/697397329 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f6d40019070 con 0x561755b5a980 2024-05-12T22:16:18.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.662+0000 7f6d59404740 10 client.26812 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:16:18.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.662+0000 7f6d59404740 10 client.26812 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-05-12T22:16:18.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.662+0000 7f6d59404740 20 client.26812 populate_metadata read hostname 'smithi028' 2024-05-12T22:16:18.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.662+0000 7f6d59404740 1 --2- 172.21.15.28:0/697397329 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x561755be5a10 0x561755be7e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:18.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.662+0000 7f6d59404740 4 client.26812 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-12T22:16:18.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.662+0000 7f6d59404740 1 -- 172.21.15.28:0/697397329 --> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561755ac74b0 con 0x561755be5a10 2024-05-12T22:16:18.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.662+0000 7f6d4ea00640 1 --2- 172.21.15.28:0/697397329 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x561755be5a10 0x561755be7e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:18.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.663+0000 7f6d4ea00640 1 --2- 172.21.15.28:0/697397329 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x561755be5a10 0x561755be7e30 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=367a21c63e97120f server_cookie=818db298f55bfa03 in_seq=0 out_seq=0 2024-05-12T22:16:18.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.663+0000 7f6d39600640 10 client.26812 ms_handle_connect on v2:172.21.15.28:6840/761183870 2024-05-12T22:16:18.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.663+0000 7f6d39600640 1 -- 172.21.15.28:0/697397329 <== mds.1 v2:172.21.15.28:6840/761183870 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561755ac74b0 con 0x561755be5a10 2024-05-12T22:16:18.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.663+0000 7f6d39600640 10 client.26812 handle_command_reply: tid=0 2024-05-12T22:16:18.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.663+0000 7f6d59404740 1 client.26812 shutdown 2024-05-12T22:16:18.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.663+0000 7f6d59404740 1 -- 172.21.15.28:0/697397329 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561755b5a980 msgr2=0x561755b72000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:16:18.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.663+0000 7f6d59404740 1 --2- 172.21.15.28:0/697397329 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561755b5a980 0x561755b72000 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x561755b62270 tx=0x7f6d400093e0 comp rx=0 tx=0).stop 2024-05-12T22:16:18.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.663+0000 7f6d59404740 1 -- 172.21.15.28:0/697397329 shutdown_connections 2024-05-12T22:16:18.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.663+0000 7f6d59404740 1 --2- 172.21.15.28:0/697397329 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x561755be5a10 0x561755be7e30 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:18.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.663+0000 7f6d59404740 1 --2- 172.21.15.28:0/697397329 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561755b65970 0x561755b72a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:18.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.663+0000 7f6d59404740 1 --2- 172.21.15.28:0/697397329 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561755b5b2a0 0x561755b72540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:18.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.663+0000 7f6d59404740 1 --2- 172.21.15.28:0/697397329 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561755b5a980 0x561755b72000 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:18.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.663+0000 7f6d39600640 0 client.26812 ms_handle_reset on v2:172.21.15.28:6840/761183870 2024-05-12T22:16:18.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.663+0000 7f6d59404740 1 -- 172.21.15.28:0/697397329 >> 172.21.15.28:0/697397329 conn(0x561755b56ca0 msgr2=0x561755b5c0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:16:18.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.664+0000 7f6d59404740 1 -- 172.21.15.28:0/697397329 shutdown_connections 2024-05-12T22:16:18.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.664+0000 7f6d59404740 1 -- 172.21.15.28:0/697397329 wait complete. 2024-05-12T22:16:18.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.664+0000 7f6d59404740 20 client.26812 trim_cache size 0 max 16384 2024-05-12T22:16:18.671 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-05-12T22:16:18.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.664+0000 7f6d56600640 1 -- 172.21.15.28:0/3314522553 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f6d340a2c10 msgr2=0x7f6d340a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:16:18.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.664+0000 7f6d56600640 1 --2- 172.21.15.28:0/3314522553 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f6d340a2c10 0x7f6d340a50d0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f6d48031660 tx=0x7f6d48005b80 comp rx=0 tx=0).stop 2024-05-12T22:16:18.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.664+0000 7f6d56600640 1 -- 172.21.15.28:0/3314522553 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6d501010c0 msgr2=0x7f6d5010e5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:16:18.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.664+0000 7f6d56600640 1 --2- 172.21.15.28:0/3314522553 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6d501010c0 0x7f6d5010e5a0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f6d3c00e9f0 tx=0x7f6d3c00eeb0 comp rx=0 tx=0).stop 2024-05-12T22:16:18.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.665+0000 7f6d56600640 1 -- 172.21.15.28:0/3314522553 shutdown_connections 2024-05-12T22:16:18.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.665+0000 7f6d56600640 1 --2- 172.21.15.28:0/3314522553 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f6d340a2c10 0x7f6d340a50d0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:18.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.665+0000 7f6d56600640 1 --2- 172.21.15.28:0/3314522553 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d50106710 0x7f6d5010eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:18.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.665+0000 7f6d56600640 1 --2- 172.21.15.28:0/3314522553 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6d501010c0 0x7f6d5010e5a0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:18.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.665+0000 7f6d56600640 1 --2- 172.21.15.28:0/3314522553 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d50100700 0x7f6d5010e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:18.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.665+0000 7f6d56600640 1 -- 172.21.15.28:0/3314522553 >> 172.21.15.28:0/3314522553 conn(0x7f6d50077830 msgr2=0x7f6d50102de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:16:18.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.665+0000 7f6d56600640 1 -- 172.21.15.28:0/3314522553 shutdown_connections 2024-05-12T22:16:18.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.665+0000 7f6d56600640 1 -- 172.21.15.28:0/3314522553 wait complete. 2024-05-12T22:16:18.684 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-05-12T22:16:18.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.879+0000 7fea15e00640 1 -- 172.21.15.28:0/1210071619 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fea10102090 msgr2=0x7fea1010e5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:16:18.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.879+0000 7fea15e00640 1 --2- 172.21.15.28:0/1210071619 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fea10102090 0x7fea1010e5a0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fea08009cd0 tx=0x7fea0802a8f0 comp rx=0 tx=0).stop 2024-05-12T22:16:18.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.879+0000 7fea15e00640 1 -- 172.21.15.28:0/1210071619 shutdown_connections 2024-05-12T22:16:18.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.879+0000 7fea15e00640 1 --2- 172.21.15.28:0/1210071619 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fea10102090 0x7fea1010e5a0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:18.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.879+0000 7fea15e00640 1 --2- 172.21.15.28:0/1210071619 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fea101016d0 0x7fea10101b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:18.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.879+0000 7fea15e00640 1 --2- 172.21.15.28:0/1210071619 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fea101076e0 0x7fea10107ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:18.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.879+0000 7fea15e00640 1 -- 172.21.15.28:0/1210071619 >> 172.21.15.28:0/1210071619 conn(0x7fea100fd200 msgr2=0x7fea100ff620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:16:18.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.879+0000 7fea15e00640 1 -- 172.21.15.28:0/1210071619 shutdown_connections 2024-05-12T22:16:18.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.879+0000 7fea15e00640 1 -- 172.21.15.28:0/1210071619 wait complete. 2024-05-12T22:16:18.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.880+0000 7fea15e00640 1 Processor -- start 2024-05-12T22:16:18.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.880+0000 7fea15e00640 1 -- start start 2024-05-12T22:16:18.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.880+0000 7fea15e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fea101016d0 0x7fea1019a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:18.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.880+0000 7fea15e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fea10102090 0x7fea1019a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:18.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.881+0000 7fea15e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fea101076e0 0x7fea1019ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:18.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.881+0000 7fea15e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fea10111b40 con 0x7fea101016d0 2024-05-12T22:16:18.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.881+0000 7fea15e00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fea101119c0 con 0x7fea101076e0 2024-05-12T22:16:18.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.881+0000 7fea15e00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fea10111cc0 con 0x7fea10102090 2024-05-12T22:16:18.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.881+0000 7fea14a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fea101016d0 0x7fea1019a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:18.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.881+0000 7fea14a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fea101016d0 0x7fea1019a180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55066/0 (socket says 172.21.15.28:55066) 2024-05-12T22:16:18.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.881+0000 7fea14a00640 1 -- 172.21.15.28:0/1617296981 learned_addr learned my addr 172.21.15.28:0/1617296981 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:16:18.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.881+0000 7fea0fe00640 1 --2- 172.21.15.28:0/1617296981 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fea10102090 0x7fea1019a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:18.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.881+0000 7fea15400640 1 --2- 172.21.15.28:0/1617296981 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fea101076e0 0x7fea1019ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:18.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.881+0000 7fea14a00640 1 -- 172.21.15.28:0/1617296981 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fea10102090 msgr2=0x7fea1019a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:16:18.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.881+0000 7fea14a00640 1 --2- 172.21.15.28:0/1617296981 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fea10102090 0x7fea1019a6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:18.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.881+0000 7fea14a00640 1 -- 172.21.15.28:0/1617296981 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fea101076e0 msgr2=0x7fea1019ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:16:18.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.881+0000 7fea14a00640 1 --2- 172.21.15.28:0/1617296981 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fea101076e0 0x7fea1019ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:18.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.881+0000 7fea14a00640 1 -- 172.21.15.28:0/1617296981 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea1019f170 con 0x7fea101016d0 2024-05-12T22:16:18.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.881+0000 7fea0fe00640 1 --2- 172.21.15.28:0/1617296981 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fea10102090 0x7fea1019a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:16:18.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.881+0000 7fea15400640 1 --2- 172.21.15.28:0/1617296981 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fea101076e0 0x7fea1019ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:16:18.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.882+0000 7fea14a00640 1 --2- 172.21.15.28:0/1617296981 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fea101016d0 0x7fea1019a180 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fea0400d950 tx=0x7fea0400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:16:18.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.882+0000 7fea0d600640 1 -- 172.21.15.28:0/1617296981 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea04017030 con 0x7fea101016d0 2024-05-12T22:16:18.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.882+0000 7fea15e00640 1 -- 172.21.15.28:0/1617296981 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea1019f460 con 0x7fea101016d0 2024-05-12T22:16:18.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.882+0000 7fea0d600640 1 -- 172.21.15.28:0/1617296981 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fea04004020 con 0x7fea101016d0 2024-05-12T22:16:18.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.882+0000 7fea0d600640 1 -- 172.21.15.28:0/1617296981 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea040047a0 con 0x7fea101016d0 2024-05-12T22:16:18.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.882+0000 7fea15e00640 1 -- 172.21.15.28:0/1617296981 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea101a6cf0 con 0x7fea101016d0 2024-05-12T22:16:18.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.886+0000 7fea1a207740 1 Processor -- start 2024-05-12T22:16:18.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.886+0000 7fea1a207740 1 -- start start 2024-05-12T22:16:18.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.886+0000 7fea0d600640 1 -- 172.21.15.28:0/1617296981 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7fea040041d0 con 0x7fea101016d0 2024-05-12T22:16:18.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.886+0000 7fea1a207740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5592ab9e1b60 0x5592ab9e1f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:18.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.886+0000 7fea1a207740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5592ab9e2480 0x5592ab9ec610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:18.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.886+0000 7fea1a207740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5592ab9ecb50 0x5592ab9eef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:18.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.886+0000 7fea1a207740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5592ab946a30 con 0x5592ab9ecb50 2024-05-12T22:16:18.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.886+0000 7fea1a207740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x5592ab8ee700 con 0x5592ab9e1b60 2024-05-12T22:16:18.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.886+0000 7fea1a207740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5592ab9429f0 con 0x5592ab9e2480 2024-05-12T22:16:18.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.887+0000 7fea0fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5592ab9ecb50 0x5592ab9eef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:18.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.887+0000 7fea0fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5592ab9ecb50 0x5592ab9eef40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55072/0 (socket says 172.21.15.28:55072) 2024-05-12T22:16:18.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.887+0000 7fea0fe00640 1 -- 172.21.15.28:0/2625153918 learned_addr learned my addr 172.21.15.28:0/2625153918 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:16:18.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.887+0000 7fea15400640 1 --2- 172.21.15.28:0/2625153918 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5592ab9e1b60 0x5592ab9e1f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:18.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.887+0000 7fea14a00640 1 --2- 172.21.15.28:0/2625153918 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5592ab9e2480 0x5592ab9ec610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:18.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.887+0000 7fea15400640 1 -- 172.21.15.28:0/2625153918 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5592ab9e2480 msgr2=0x5592ab9ec610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:16:18.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.887+0000 7fea15400640 1 --2- 172.21.15.28:0/2625153918 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5592ab9e2480 0x5592ab9ec610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:18.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.887+0000 7fea15400640 1 -- 172.21.15.28:0/2625153918 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5592ab9ecb50 msgr2=0x5592ab9eef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:16:18.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.888+0000 7fea15400640 1 --2- 172.21.15.28:0/2625153918 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5592ab9ecb50 0x5592ab9eef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:18.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.888+0000 7fea15400640 1 -- 172.21.15.28:0/2625153918 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5592ab65a3f0 con 0x5592ab9e1b60 2024-05-12T22:16:18.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.888+0000 7fea0fe00640 1 --2- 172.21.15.28:0/2625153918 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5592ab9ecb50 0x5592ab9eef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:16:18.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.888+0000 7fea15400640 1 --2- 172.21.15.28:0/2625153918 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5592ab9e1b60 0x5592ab9e1f40 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fea0802ae10 tx=0x7fea080098a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9479555201be038b server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:16:18.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.888+0000 7fea01c00640 1 -- 172.21.15.28:0/2625153918 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea08003b70 con 0x5592ab9e1b60 2024-05-12T22:16:18.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.888+0000 7fea01c00640 1 -- 172.21.15.28:0/2625153918 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fea08036030 con 0x5592ab9e1b60 2024-05-12T22:16:18.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.888+0000 7fea01c00640 1 -- 172.21.15.28:0/2625153918 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea080045b0 con 0x5592ab9e1b60 2024-05-12T22:16:18.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.889+0000 7fea1a207740 1 -- 172.21.15.28:0/2625153918 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5592ab9e1b60 msgr2=0x5592ab9e1f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:16:18.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.889+0000 7fea1a207740 1 --2- 172.21.15.28:0/2625153918 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5592ab9e1b60 0x5592ab9e1f40 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fea0802ae10 tx=0x7fea080098a0 comp rx=0 tx=0).stop 2024-05-12T22:16:18.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.889+0000 7fea1a207740 1 -- 172.21.15.28:0/2625153918 shutdown_connections 2024-05-12T22:16:18.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.889+0000 7fea1a207740 1 --2- 172.21.15.28:0/2625153918 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5592ab9ecb50 0x5592ab9eef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:18.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.889+0000 7fea1a207740 1 --2- 172.21.15.28:0/2625153918 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5592ab9e2480 0x5592ab9ec610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:18.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.889+0000 7fea1a207740 1 --2- 172.21.15.28:0/2625153918 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5592ab9e1b60 0x5592ab9e1f40 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:18.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.889+0000 7fea1a207740 1 -- 172.21.15.28:0/2625153918 >> 172.21.15.28:0/2625153918 conn(0x5592ab9dde80 msgr2=0x5592ab9de260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:16:18.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.889+0000 7fea1a207740 1 -- 172.21.15.28:0/2625153918 shutdown_connections 2024-05-12T22:16:18.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.890+0000 7fea1a207740 1 -- 172.21.15.28:0/2625153918 wait complete. 2024-05-12T22:16:18.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.890+0000 7fea1a207740 1 -- start start 2024-05-12T22:16:18.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.890+0000 7fea0d600640 1 --2- 172.21.15.28:0/1617296981 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fe9f80a2c10 0x7fe9f80a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:18.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.890+0000 7fea1a207740 1 Processor -- start 2024-05-12T22:16:18.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.890+0000 7fea0d600640 1 -- 172.21.15.28:0/1617296981 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8513+0+0 (secure 0 0 0) 0x7fea040ced40 con 0x7fea101016d0 2024-05-12T22:16:18.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.891+0000 7fea15400640 1 --2- 172.21.15.28:0/1617296981 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fe9f80a2c10 0x7fe9f80a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:18.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.891+0000 7fea1a207740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5592ab9e1b60 0x5592ab9e9af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:18.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.891+0000 7fea1a207740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5592ab9e2480 0x5592ab9ea030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:18.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.891+0000 7fea1a207740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5592ab9ecb50 0x5592ab9ea570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:18.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.891+0000 7fea1a207740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5592ab9ef8e0 con 0x5592ab9e1b60 2024-05-12T22:16:18.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.891+0000 7fea1a207740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x5592ab901a70 con 0x5592ab9ecb50 2024-05-12T22:16:18.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.891+0000 7fea1a207740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5592ab9efa60 con 0x5592ab9e2480 2024-05-12T22:16:18.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.891+0000 7fea1a207740 10 client.0 _ll_register_callbacks cb 0x5592ab8e4ba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:16:18.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.892+0000 7fea15400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5592ab9ecb50 0x5592ab9ea570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:18.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.892+0000 7fea0fe00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5592ab9e2480 0x5592ab9ea030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:18.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.892+0000 7fea15400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5592ab9ecb50 0x5592ab9ea570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:54890/0 (socket says 172.21.15.28:54890) 2024-05-12T22:16:18.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.892+0000 7fea15400640 1 -- 172.21.15.28:0/3720959060 learned_addr learned my addr 172.21.15.28:0/3720959060 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:16:18.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.892+0000 7fea14a00640 1 --2- 172.21.15.28:0/3720959060 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5592ab9e1b60 0x5592ab9e9af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:18.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.892+0000 7fea15400640 1 --2- 172.21.15.28:0/1617296981 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fe9f80a2c10 0x7fe9f80a50d0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fea08009600 tx=0x7fea0802a8f0 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:16:18.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.892+0000 7fea0fe00640 1 -- 172.21.15.28:0/3720959060 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5592ab9ecb50 msgr2=0x5592ab9ea570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:16:18.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.892+0000 7fea0fe00640 1 --2- 172.21.15.28:0/3720959060 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5592ab9ecb50 0x5592ab9ea570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:18.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.892+0000 7fea0fe00640 1 -- 172.21.15.28:0/3720959060 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5592ab9e1b60 msgr2=0x5592ab9e9af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:16:18.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.892+0000 7fea0fe00640 1 --2- 172.21.15.28:0/3720959060 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5592ab9e1b60 0x5592ab9e9af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:18.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.892+0000 7fea0fe00640 1 -- 172.21.15.28:0/3720959060 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5592ab7247d0 con 0x5592ab9e2480 2024-05-12T22:16:18.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.892+0000 7fea14a00640 1 --2- 172.21.15.28:0/3720959060 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5592ab9e1b60 0x5592ab9e9af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:16:18.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.892+0000 7fea0fe00640 1 --2- 172.21.15.28:0/3720959060 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5592ab9e2480 0x5592ab9ea030 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fe9fc012d20 tx=0x7fe9fc00f790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7014de6b13dac724 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:16:18.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.892+0000 7fe9f6000640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-12T22:16:18.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.893+0000 7fe9f6000640 1 -- 172.21.15.28:0/3720959060 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9fc00fe60 con 0x5592ab9e2480 2024-05-12T22:16:18.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.893+0000 7fe9f6000640 1 -- 172.21.15.28:0/3720959060 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8513+0+0 (secure 0 0 0) 0x7fe9fc012110 con 0x5592ab9e2480 2024-05-12T22:16:18.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.893+0000 7fea1a207740 1 -- 172.21.15.28:0/3720959060 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fea08036030 con 0x5592ab9e2480 2024-05-12T22:16:18.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.893+0000 7fe9f6000640 1 -- 172.21.15.28:0/3720959060 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fe9fc004050 con 0x5592ab9e2480 2024-05-12T22:16:18.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.893+0000 7fea1a207740 10 client.26824 fetch_fsmap learned FSMap version 26 2024-05-12T22:16:18.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.893+0000 7fea1a207740 1 -- 172.21.15.28:0/3720959060 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5592aba04450 con 0x5592ab9e2480 2024-05-12T22:16:18.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.893+0000 7fe9f6000640 1 -- 172.21.15.28:0/3720959060 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fea08036030 con 0x5592ab9e2480 2024-05-12T22:16:18.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.893+0000 7fe9f6000640 1 -- 172.21.15.28:0/3720959060 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fe9fc025480 con 0x5592ab9e2480 2024-05-12T22:16:18.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.894+0000 7fea1a207740 10 client.26824 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:16:18.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.894+0000 7fea1a207740 10 client.26824 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:16:18.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.894+0000 7fea1a207740 20 client.26824 populate_metadata read hostname 'smithi028' 2024-05-12T22:16:18.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.894+0000 7fea1a207740 1 --2- 172.21.15.28:0/3720959060 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5592aba05860 0x5592aba07c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:18.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.894+0000 7fea1a207740 4 client.26824 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-12T22:16:18.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.894+0000 7fea1a207740 1 -- 172.21.15.28:0/3720959060 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5592ab954160 con 0x5592aba05860 2024-05-12T22:16:18.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.894+0000 7fea14a00640 1 --2- 172.21.15.28:0/3720959060 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5592aba05860 0x5592aba07c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:18.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.895+0000 7fea14a00640 1 --2- 172.21.15.28:0/3720959060 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5592aba05860 0x5592aba07c50 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=10414137173c5139 server_cookie=e6df6f8d12af24b6 in_seq=0 out_seq=0 2024-05-12T22:16:18.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.895+0000 7fe9f6000640 10 client.26824 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:16:18.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.895+0000 7fe9f6000640 1 -- 172.21.15.28:0/3720959060 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5592ab954160 con 0x5592aba05860 2024-05-12T22:16:18.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.895+0000 7fe9f6000640 10 client.26824 handle_command_reply: tid=0 2024-05-12T22:16:18.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.895+0000 7fea1a207740 1 client.26824 shutdown 2024-05-12T22:16:18.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.896+0000 7fea1a207740 1 -- 172.21.15.28:0/3720959060 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5592ab9e2480 msgr2=0x5592ab9ea030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:16:18.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.896+0000 7fea1a207740 1 --2- 172.21.15.28:0/3720959060 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5592ab9e2480 0x5592ab9ea030 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fe9fc012d20 tx=0x7fe9fc00f790 comp rx=0 tx=0).stop 2024-05-12T22:16:18.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.896+0000 7fea1a207740 1 -- 172.21.15.28:0/3720959060 shutdown_connections 2024-05-12T22:16:18.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.896+0000 7fea1a207740 1 --2- 172.21.15.28:0/3720959060 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5592aba05860 0x5592aba07c50 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:18.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.896+0000 7fea1a207740 1 --2- 172.21.15.28:0/3720959060 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5592ab9ecb50 0x5592ab9ea570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:18.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.896+0000 7fea1a207740 1 --2- 172.21.15.28:0/3720959060 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5592ab9e2480 0x5592ab9ea030 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:18.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.896+0000 7fe9f6000640 0 client.26824 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:16:18.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.896+0000 7fea1a207740 1 --2- 172.21.15.28:0/3720959060 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5592ab9e1b60 0x5592ab9e9af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:18.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.896+0000 7fea1a207740 1 -- 172.21.15.28:0/3720959060 >> 172.21.15.28:0/3720959060 conn(0x5592ab9dde80 msgr2=0x5592ab9e3c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:16:18.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.896+0000 7fea1a207740 1 -- 172.21.15.28:0/3720959060 shutdown_connections 2024-05-12T22:16:18.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.896+0000 7fea1a207740 1 -- 172.21.15.28:0/3720959060 wait complete. 2024-05-12T22:16:18.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.896+0000 7fea1a207740 20 client.26824 trim_cache size 0 max 16384 2024-05-12T22:16:18.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.913+0000 7fea1a207740 1 Processor -- start 2024-05-12T22:16:18.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.913+0000 7fea1a207740 1 -- start start 2024-05-12T22:16:18.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.913+0000 7fea1a207740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5592ab9dde80 0x5592aba01060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:18.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.913+0000 7fea1a207740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5592ab9e1b60 0x5592ab9dce80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:18.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.913+0000 7fea1a207740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5592ab9e2480 0x5592aba0d560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:18.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.913+0000 7fea1a207740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fea101ad9c0 con 0x5592ab9e2480 2024-05-12T22:16:18.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.913+0000 7fea1a207740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fea101ab290 con 0x5592ab9dde80 2024-05-12T22:16:18.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.913+0000 7fea1a207740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fea101a9140 con 0x5592ab9e1b60 2024-05-12T22:16:18.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.913+0000 7fea14a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5592ab9dde80 0x5592aba01060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:18.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.914+0000 7fea14a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5592ab9dde80 0x5592aba01060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:54900/0 (socket says 172.21.15.28:54900) 2024-05-12T22:16:18.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.914+0000 7fea14a00640 1 -- 172.21.15.28:0/1131241058 learned_addr learned my addr 172.21.15.28:0/1131241058 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:16:18.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.914+0000 7fea0fe00640 1 --2- 172.21.15.28:0/1131241058 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5592ab9e1b60 0x5592ab9dce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:18.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.914+0000 7fea15400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5592ab9e2480 0x5592aba0d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:18.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.914+0000 7fea0fe00640 1 -- 172.21.15.28:0/1131241058 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5592ab9dde80 msgr2=0x5592aba01060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:16:18.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.914+0000 7fea0fe00640 1 --2- 172.21.15.28:0/1131241058 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5592ab9dde80 0x5592aba01060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:18.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.914+0000 7fea0fe00640 1 -- 172.21.15.28:0/1131241058 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5592ab9e2480 msgr2=0x5592aba0d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:16:18.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.914+0000 7fea0fe00640 1 --2- 172.21.15.28:0/1131241058 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5592ab9e2480 0x5592aba0d560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:18.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.914+0000 7fea0fe00640 1 -- 172.21.15.28:0/1131241058 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5592ab9de7c0 con 0x5592ab9e1b60 2024-05-12T22:16:18.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.914+0000 7fea0fe00640 1 --2- 172.21.15.28:0/1131241058 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5592ab9e1b60 0x5592ab9dce80 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fe9fc025a40 tx=0x7fe9fc0059b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7d909da94c7d188a server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:16:18.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.915+0000 7fea01200640 1 -- 172.21.15.28:0/1131241058 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9fc000e60 con 0x5592ab9e1b60 2024-05-12T22:16:18.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.915+0000 7fea01200640 1 -- 172.21.15.28:0/1131241058 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe9fc005740 con 0x5592ab9e1b60 2024-05-12T22:16:18.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.915+0000 7fea01200640 1 -- 172.21.15.28:0/1131241058 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9fc003850 con 0x5592ab9e1b60 2024-05-12T22:16:18.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.915+0000 7fea1a207740 1 -- 172.21.15.28:0/1131241058 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5592ab9e1b60 msgr2=0x5592ab9dce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:16:18.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.915+0000 7fea1a207740 1 --2- 172.21.15.28:0/1131241058 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5592ab9e1b60 0x5592ab9dce80 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fe9fc025a40 tx=0x7fe9fc0059b0 comp rx=0 tx=0).stop 2024-05-12T22:16:18.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.915+0000 7fea1a207740 1 -- 172.21.15.28:0/1131241058 shutdown_connections 2024-05-12T22:16:18.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.915+0000 7fea1a207740 1 --2- 172.21.15.28:0/1131241058 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5592ab9e2480 0x5592aba0d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:18.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.915+0000 7fea1a207740 1 --2- 172.21.15.28:0/1131241058 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5592ab9e1b60 0x5592ab9dce80 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:18.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.915+0000 7fea1a207740 1 --2- 172.21.15.28:0/1131241058 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5592ab9dde80 0x5592aba01060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:18.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.915+0000 7fea1a207740 1 -- 172.21.15.28:0/1131241058 >> 172.21.15.28:0/1131241058 conn(0x5592ab9e96f0 msgr2=0x5592ab9dc770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:16:18.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.916+0000 7fea1a207740 1 -- 172.21.15.28:0/1131241058 shutdown_connections 2024-05-12T22:16:18.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.916+0000 7fea1a207740 1 -- 172.21.15.28:0/1131241058 wait complete. 2024-05-12T22:16:18.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.916+0000 7fea1a207740 1 -- start start 2024-05-12T22:16:18.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.916+0000 7fea1a207740 1 Processor -- start 2024-05-12T22:16:18.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.917+0000 7fea1a207740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5592ab9dde80 0x5592ab9f9270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:18.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.917+0000 7fea1a207740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5592ab9e1b60 0x5592ab9f97b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:18.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.917+0000 7fea1a207740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5592ab9e2480 0x5592ab9f9d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:18.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.917+0000 7fea1a207740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fea101a9790 con 0x5592ab9dde80 2024-05-12T22:16:18.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.917+0000 7fea1a207740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fea101aabc0 con 0x5592ab9e1b60 2024-05-12T22:16:18.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.917+0000 7fea1a207740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fea101a9e20 con 0x5592ab9e2480 2024-05-12T22:16:18.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.917+0000 7fea1a207740 10 client.0 _ll_register_callbacks cb 0x7fe9fc012930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:16:18.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.917+0000 7fea14a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5592ab9dde80 0x5592ab9f9270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:18.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.917+0000 7fea14a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5592ab9dde80 0x5592ab9f9270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55098/0 (socket says 172.21.15.28:55098) 2024-05-12T22:16:18.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.917+0000 7fea14a00640 1 -- 172.21.15.28:0/1286621076 learned_addr learned my addr 172.21.15.28:0/1286621076 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:16:18.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.917+0000 7fea0fe00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5592ab9e1b60 0x5592ab9f97b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:18.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.917+0000 7fea15400640 1 --2- 172.21.15.28:0/1286621076 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5592ab9e2480 0x5592ab9f9d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:18.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.917+0000 7fea14a00640 1 -- 172.21.15.28:0/1286621076 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5592ab9e2480 msgr2=0x5592ab9f9d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:16:18.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.917+0000 7fea14a00640 1 --2- 172.21.15.28:0/1286621076 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5592ab9e2480 0x5592ab9f9d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:18.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.917+0000 7fea14a00640 1 -- 172.21.15.28:0/1286621076 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5592ab9e1b60 msgr2=0x5592ab9f97b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:16:18.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.917+0000 7fea14a00640 1 --2- 172.21.15.28:0/1286621076 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5592ab9e1b60 0x5592ab9f97b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:18.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.917+0000 7fea14a00640 1 -- 172.21.15.28:0/1286621076 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5592aba6b0c0 con 0x5592ab9dde80 2024-05-12T22:16:18.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.918+0000 7fea15400640 1 --2- 172.21.15.28:0/1286621076 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5592ab9e2480 0x5592ab9f9d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:16:18.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.918+0000 7fea0fe00640 1 --2- 172.21.15.28:0/1286621076 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5592ab9e1b60 0x5592ab9f97b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:16:18.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.918+0000 7fea14a00640 1 --2- 172.21.15.28:0/1286621076 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5592ab9dde80 0x5592ab9f9270 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fea040cefe0 tx=0x7fea04005bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3c67bb348b0061b server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:16:18.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.918+0000 7fe9f6000640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-12T22:16:18.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.918+0000 7fe9f6000640 1 -- 172.21.15.28:0/1286621076 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea0409ab60 con 0x5592ab9dde80 2024-05-12T22:16:18.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.918+0000 7fe9f6000640 1 -- 172.21.15.28:0/1286621076 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8513+0+0 (secure 0 0 0) 0x7fea040cc6b0 con 0x5592ab9dde80 2024-05-12T22:16:18.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.918+0000 7fea1a207740 1 -- 172.21.15.28:0/1286621076 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe9fc005740 con 0x5592ab9dde80 2024-05-12T22:16:18.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.918+0000 7fe9f6000640 1 -- 172.21.15.28:0/1286621076 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fea040d22d0 con 0x5592ab9dde80 2024-05-12T22:16:18.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.918+0000 7fea1a207740 10 client.16746 fetch_fsmap learned FSMap version 26 2024-05-12T22:16:18.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.918+0000 7fea1a207740 1 -- 172.21.15.28:0/1286621076 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5592aba6c420 con 0x5592ab9dde80 2024-05-12T22:16:18.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.919+0000 7fe9f6000640 1 -- 172.21.15.28:0/1286621076 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe9fc005740 con 0x5592ab9dde80 2024-05-12T22:16:18.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.919+0000 7fe9f6000640 1 -- 172.21.15.28:0/1286621076 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fea0409f120 con 0x5592ab9dde80 2024-05-12T22:16:18.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.919+0000 7fea1a207740 10 client.16746 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:16:18.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.919+0000 7fea1a207740 10 client.16746 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:16:18.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.919+0000 7fea1a207740 20 client.16746 populate_metadata read hostname 'smithi028' 2024-05-12T22:16:18.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.919+0000 7fea1a207740 1 --2- 172.21.15.28:0/1286621076 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5592ab9ecb50 0x5592aba6ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:18.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.919+0000 7fea1a207740 4 client.16746 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-12T22:16:18.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.919+0000 7fea1a207740 1 -- 172.21.15.28:0/1286621076 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5592ab94e4b0 con 0x5592ab9ecb50 2024-05-12T22:16:18.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.919+0000 7fea0fe00640 1 --2- 172.21.15.28:0/1286621076 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5592ab9ecb50 0x5592aba6ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:18.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.920+0000 7fea0fe00640 1 --2- 172.21.15.28:0/1286621076 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5592ab9ecb50 0x5592aba6ebd0 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1cc60242a4654f89 server_cookie=32886011649570b1 in_seq=0 out_seq=0 2024-05-12T22:16:18.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.920+0000 7fe9f6000640 10 client.16746 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:16:18.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.920+0000 7fe9f6000640 1 -- 172.21.15.28:0/1286621076 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5592ab94e4b0 con 0x5592ab9ecb50 2024-05-12T22:16:18.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.920+0000 7fe9f6000640 10 client.16746 handle_command_reply: tid=0 2024-05-12T22:16:18.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.920+0000 7fea1a207740 1 client.16746 shutdown 2024-05-12T22:16:18.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.920+0000 7fea1a207740 1 -- 172.21.15.28:0/1286621076 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5592ab9dde80 msgr2=0x5592ab9f9270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:16:18.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.920+0000 7fea1a207740 1 --2- 172.21.15.28:0/1286621076 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5592ab9dde80 0x5592ab9f9270 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fea040cefe0 tx=0x7fea04005bf0 comp rx=0 tx=0).stop 2024-05-12T22:16:18.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.920+0000 7fea1a207740 1 -- 172.21.15.28:0/1286621076 shutdown_connections 2024-05-12T22:16:18.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.920+0000 7fea1a207740 1 --2- 172.21.15.28:0/1286621076 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5592ab9ecb50 0x5592aba6ebd0 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:18.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.920+0000 7fea1a207740 1 --2- 172.21.15.28:0/1286621076 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5592ab9e2480 0x5592ab9f9d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:18.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.920+0000 7fea1a207740 1 --2- 172.21.15.28:0/1286621076 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5592ab9e1b60 0x5592ab9f97b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:18.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.920+0000 7fea1a207740 1 --2- 172.21.15.28:0/1286621076 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5592ab9dde80 0x5592ab9f9270 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:18.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.920+0000 7fe9f6000640 0 client.16746 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:16:18.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.920+0000 7fea1a207740 1 -- 172.21.15.28:0/1286621076 >> 172.21.15.28:0/1286621076 conn(0x5592ab9e96f0 msgr2=0x5592ab9e7f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:16:18.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.920+0000 7fea1a207740 1 -- 172.21.15.28:0/1286621076 shutdown_connections 2024-05-12T22:16:18.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.920+0000 7fea1a207740 1 -- 172.21.15.28:0/1286621076 wait complete. 2024-05-12T22:16:18.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.921+0000 7fea1a207740 20 client.16746 trim_cache size 0 max 16384 2024-05-12T22:16:18.931 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-05-12T22:16:18.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.921+0000 7fea15e00640 1 -- 172.21.15.28:0/1617296981 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fe9f80a2c10 msgr2=0x7fe9f80a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:16:18.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.921+0000 7fea15e00640 1 --2- 172.21.15.28:0/1617296981 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fe9f80a2c10 0x7fe9f80a50d0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fea08009600 tx=0x7fea0802a8f0 comp rx=0 tx=0).stop 2024-05-12T22:16:18.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.921+0000 7fea15e00640 1 -- 172.21.15.28:0/1617296981 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fea101016d0 msgr2=0x7fea1019a180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:16:18.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.921+0000 7fea15e00640 1 --2- 172.21.15.28:0/1617296981 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fea101016d0 0x7fea1019a180 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fea0400d950 tx=0x7fea0400de10 comp rx=0 tx=0).stop 2024-05-12T22:16:18.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.921+0000 7fea15e00640 1 -- 172.21.15.28:0/1617296981 shutdown_connections 2024-05-12T22:16:18.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.921+0000 7fea15e00640 1 --2- 172.21.15.28:0/1617296981 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fe9f80a2c10 0x7fe9f80a50d0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:18.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.921+0000 7fea15e00640 1 --2- 172.21.15.28:0/1617296981 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fea101076e0 0x7fea1019ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:18.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.921+0000 7fea15e00640 1 --2- 172.21.15.28:0/1617296981 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fea10102090 0x7fea1019a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:18.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.921+0000 7fea15e00640 1 --2- 172.21.15.28:0/1617296981 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fea101016d0 0x7fea1019a180 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:18.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.921+0000 7fea15e00640 1 -- 172.21.15.28:0/1617296981 >> 172.21.15.28:0/1617296981 conn(0x7fea100fd200 msgr2=0x7fea1010cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:16:18.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.922+0000 7fea15e00640 1 -- 172.21.15.28:0/1617296981 shutdown_connections 2024-05-12T22:16:18.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:18.922+0000 7fea15e00640 1 -- 172.21.15.28:0/1617296981 wait complete. 2024-05-12T22:16:19.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:16:19 smithi028 ceph-mon[29847]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:16:19.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:16:19 smithi186 ceph-mon[34785]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:16:19.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:16:19 smithi086 ceph-mon[34297]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:16:19.940 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-12T22:16:19.940 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-12T22:16:19.940 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-05-12T22:16:20.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.171+0000 7f2c7fe00640 1 -- 172.21.15.28:0/2248286611 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2c801098e0 msgr2=0x7f2c80109cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:16:20.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.171+0000 7f2c7fe00640 1 --2- 172.21.15.28:0/2248286611 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2c801098e0 0x7f2c80109cc0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f2c70009930 tx=0x7f2c7002a420 comp rx=0 tx=0).stop 2024-05-12T22:16:20.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.171+0000 7f2c7fe00640 1 -- 172.21.15.28:0/2248286611 shutdown_connections 2024-05-12T22:16:20.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.171+0000 7f2c7fe00640 1 --2- 172.21.15.28:0/2248286611 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2c80104290 0x7f2c801107a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:20.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.171+0000 7f2c7fe00640 1 --2- 172.21.15.28:0/2248286611 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2c801038d0 0x7f2c80103d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:20.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.171+0000 7f2c7fe00640 1 --2- 172.21.15.28:0/2248286611 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2c801098e0 0x7f2c80109cc0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:20.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.171+0000 7f2c7fe00640 1 -- 172.21.15.28:0/2248286611 >> 172.21.15.28:0/2248286611 conn(0x7f2c800ff3a0 msgr2=0x7f2c801017c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:16:20.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.171+0000 7f2c7fe00640 1 -- 172.21.15.28:0/2248286611 shutdown_connections 2024-05-12T22:16:20.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.171+0000 7f2c7fe00640 1 -- 172.21.15.28:0/2248286611 wait complete. 2024-05-12T22:16:20.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.172+0000 7f2c7fe00640 1 Processor -- start 2024-05-12T22:16:20.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.173+0000 7f2c7fe00640 1 -- start start 2024-05-12T22:16:20.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.173+0000 7f2c7fe00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2c801038d0 0x7f2c8019a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:20.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.173+0000 7f2c7fe00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2c80104290 0x7f2c8019a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:20.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.173+0000 7f2c7fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2c801098e0 0x7f2c8019eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:20.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.173+0000 7f2c7fe00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2c80113cb0 con 0x7f2c801098e0 2024-05-12T22:16:20.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.173+0000 7f2c7fe00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f2c80113b30 con 0x7f2c801038d0 2024-05-12T22:16:20.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.173+0000 7f2c7fe00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f2c80113e30 con 0x7f2c80104290 2024-05-12T22:16:20.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.173+0000 7f2c7ea00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2c801038d0 0x7f2c8019a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:20.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.173+0000 7f2c7f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2c801098e0 0x7f2c8019eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:20.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.173+0000 7f2c7e000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2c80104290 0x7f2c8019a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:20.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.173+0000 7f2c7ea00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2c801038d0 0x7f2c8019a1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:54934/0 (socket says 172.21.15.28:54934) 2024-05-12T22:16:20.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.174+0000 7f2c7ea00640 1 -- 172.21.15.28:0/3297205849 learned_addr learned my addr 172.21.15.28:0/3297205849 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:16:20.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.174+0000 7f2c7e000640 1 -- 172.21.15.28:0/3297205849 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2c801038d0 msgr2=0x7f2c8019a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:16:20.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.174+0000 7f2c7e000640 1 --2- 172.21.15.28:0/3297205849 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2c801038d0 0x7f2c8019a1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:20.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.174+0000 7f2c7e000640 1 -- 172.21.15.28:0/3297205849 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2c801098e0 msgr2=0x7f2c8019eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:16:20.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.174+0000 7f2c7e000640 1 --2- 172.21.15.28:0/3297205849 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2c801098e0 0x7f2c8019eab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:20.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.174+0000 7f2c7e000640 1 -- 172.21.15.28:0/3297205849 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c8019f1e0 con 0x7f2c80104290 2024-05-12T22:16:20.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.174+0000 7f2c7e000640 1 --2- 172.21.15.28:0/3297205849 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2c80104290 0x7f2c8019a730 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f2c6800e9f0 tx=0x7f2c6800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:16:20.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.175+0000 7f2c77400640 1 -- 172.21.15.28:0/3297205849 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c6800cd80 con 0x7f2c80104290 2024-05-12T22:16:20.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.175+0000 7f2c77400640 1 -- 172.21.15.28:0/3297205849 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2c680035f0 con 0x7f2c80104290 2024-05-12T22:16:20.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.175+0000 7f2c77400640 1 -- 172.21.15.28:0/3297205849 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c68010690 con 0x7f2c80104290 2024-05-12T22:16:20.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.175+0000 7f2c7fe00640 1 -- 172.21.15.28:0/3297205849 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c8019f4d0 con 0x7f2c80104290 2024-05-12T22:16:20.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.175+0000 7f2c7fe00640 1 -- 172.21.15.28:0/3297205849 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c801a6d60 con 0x7f2c80104290 2024-05-12T22:16:20.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.178+0000 7f2c77400640 1 -- 172.21.15.28:0/3297205849 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7f2c68003b30 con 0x7f2c80104290 2024-05-12T22:16:20.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.178+0000 7f2c88870740 1 Processor -- start 2024-05-12T22:16:20.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.179+0000 7f2c88870740 1 -- start start 2024-05-12T22:16:20.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.179+0000 7f2c88870740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560f9339e500 0x560f9339e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:20.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.179+0000 7f2c88870740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560f9339eeb0 0x560f933a8f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:20.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.179+0000 7f2c88870740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f933a9520 0x560f933ab970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:20.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.179+0000 7f2c88870740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x560f933027c0 con 0x560f933a9520 2024-05-12T22:16:20.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.179+0000 7f2c88870740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x560f932aa790 con 0x560f9339e500 2024-05-12T22:16:20.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.179+0000 7f2c88870740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x560f932fe780 con 0x560f9339eeb0 2024-05-12T22:16:20.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.180+0000 7f2c7f400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560f9339e500 0x560f9339e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:20.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.180+0000 7f2c7e000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f933a9520 0x560f933ab970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:20.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.180+0000 7f2c7f400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560f9339e500 0x560f9339e8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:54938/0 (socket says 172.21.15.28:54938) 2024-05-12T22:16:20.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.180+0000 7f2c7f400640 1 -- 172.21.15.28:0/1973810772 learned_addr learned my addr 172.21.15.28:0/1973810772 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:16:20.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.180+0000 7f2c7ea00640 1 --2- 172.21.15.28:0/1973810772 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560f9339eeb0 0x560f933a8f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:20.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.180+0000 7f2c7f400640 1 -- 172.21.15.28:0/1973810772 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560f9339eeb0 msgr2=0x560f933a8f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:16:20.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.180+0000 7f2c7f400640 1 --2- 172.21.15.28:0/1973810772 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560f9339eeb0 0x560f933a8f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:20.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.180+0000 7f2c7f400640 1 -- 172.21.15.28:0/1973810772 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f933a9520 msgr2=0x560f933ab970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:16:20.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.180+0000 7f2c7f400640 1 --2- 172.21.15.28:0/1973810772 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f933a9520 0x560f933ab970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:20.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.180+0000 7f2c7f400640 1 -- 172.21.15.28:0/1973810772 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560f932ca6a0 con 0x560f9339e500 2024-05-12T22:16:20.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.180+0000 7f2c77400640 1 --2- 172.21.15.28:0/3297205849 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f2c640a2d80 0x7f2c640a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:20.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.180+0000 7f2c77400640 1 -- 172.21.15.28:0/3297205849 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8513+0+0 (secure 0 0 0) 0x7f2c680cf700 con 0x7f2c80104290 2024-05-12T22:16:20.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.180+0000 7f2c7f400640 1 --2- 172.21.15.28:0/1973810772 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560f9339e500 0x560f9339e8e0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f2c78009f50 tx=0x7f2c780118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b0094bfb24942950 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:16:20.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.181+0000 7f2c7ea00640 1 --2- 172.21.15.28:0/3297205849 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f2c640a2d80 0x7f2c640a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:20.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.181+0000 7f2c6fe00640 1 -- 172.21.15.28:0/1973810772 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c78004d20 con 0x560f9339e500 2024-05-12T22:16:20.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.181+0000 7f2c6fe00640 1 -- 172.21.15.28:0/1973810772 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2c78004050 con 0x560f9339e500 2024-05-12T22:16:20.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.181+0000 7f2c6fe00640 1 -- 172.21.15.28:0/1973810772 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c78005280 con 0x560f9339e500 2024-05-12T22:16:20.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.181+0000 7f2c88870740 1 -- 172.21.15.28:0/1973810772 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560f9339e500 msgr2=0x560f9339e8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:16:20.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.181+0000 7f2c88870740 1 --2- 172.21.15.28:0/1973810772 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560f9339e500 0x560f9339e8e0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f2c78009f50 tx=0x7f2c780118d0 comp rx=0 tx=0).stop 2024-05-12T22:16:20.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.181+0000 7f2c7ea00640 1 --2- 172.21.15.28:0/3297205849 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f2c640a2d80 0x7f2c640a5240 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x560f9339fac0 tx=0x7f2c70009620 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:16:20.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.182+0000 7f2c88870740 1 -- 172.21.15.28:0/1973810772 shutdown_connections 2024-05-12T22:16:20.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.182+0000 7f2c88870740 1 --2- 172.21.15.28:0/1973810772 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f933a9520 0x560f933ab970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:20.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.182+0000 7f2c88870740 1 --2- 172.21.15.28:0/1973810772 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560f9339eeb0 0x560f933a8f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:20.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.182+0000 7f2c88870740 1 --2- 172.21.15.28:0/1973810772 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560f9339e500 0x560f9339e8e0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:20.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.182+0000 7f2c88870740 1 -- 172.21.15.28:0/1973810772 >> 172.21.15.28:0/1973810772 conn(0x560f9339a5b0 msgr2=0x560f9339a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:16:20.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.182+0000 7f2c88870740 1 -- 172.21.15.28:0/1973810772 shutdown_connections 2024-05-12T22:16:20.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.182+0000 7f2c88870740 1 -- 172.21.15.28:0/1973810772 wait complete. 2024-05-12T22:16:20.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.182+0000 7f2c88870740 1 -- start start 2024-05-12T22:16:20.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.182+0000 7f2c88870740 1 Processor -- start 2024-05-12T22:16:20.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.183+0000 7f2c88870740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560f9339e500 0x560f933a6440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:20.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.183+0000 7f2c88870740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f9339eeb0 0x560f933a6980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:20.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.183+0000 7f2c88870740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560f933a9520 0x560f933a6ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:20.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.183+0000 7f2c88870740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x560f933ac530 con 0x560f9339eeb0 2024-05-12T22:16:20.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.183+0000 7f2c88870740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x560f932bda90 con 0x560f9339e500 2024-05-12T22:16:20.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.183+0000 7f2c88870740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x560f933ac6b0 con 0x560f933a9520 2024-05-12T22:16:20.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.183+0000 7f2c88870740 10 client.0 _ll_register_callbacks cb 0x560f932f8af0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:16:20.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.183+0000 7f2c7e000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f9339eeb0 0x560f933a6980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:20.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.183+0000 7f2c7e000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f9339eeb0 0x560f933a6980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55110/0 (socket says 172.21.15.28:55110) 2024-05-12T22:16:20.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.183+0000 7f2c7e000640 1 -- 172.21.15.28:0/3903860330 learned_addr learned my addr 172.21.15.28:0/3903860330 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:16:20.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.183+0000 7f2c7f400640 1 --2- 172.21.15.28:0/3903860330 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560f933a9520 0x560f933a6ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:20.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.183+0000 7f2c7ea00640 1 --2- 172.21.15.28:0/3903860330 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560f9339e500 0x560f933a6440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:20.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.184+0000 7f2c7e000640 1 -- 172.21.15.28:0/3903860330 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560f933a9520 msgr2=0x560f933a6ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:16:20.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.184+0000 7f2c7e000640 1 --2- 172.21.15.28:0/3903860330 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560f933a9520 0x560f933a6ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:20.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.184+0000 7f2c7e000640 1 -- 172.21.15.28:0/3903860330 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560f9339e500 msgr2=0x560f933a6440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:16:20.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.184+0000 7f2c7e000640 1 --2- 172.21.15.28:0/3903860330 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560f9339e500 0x560f933a6440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:20.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.184+0000 7f2c7e000640 1 -- 172.21.15.28:0/3903860330 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560f930de7e0 con 0x560f9339eeb0 2024-05-12T22:16:20.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.184+0000 7f2c7e000640 1 --2- 172.21.15.28:0/3903860330 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f9339eeb0 0x560f933a6980 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f2c6800c6a0 tx=0x7f2c68097f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32adcde7169b710e server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:16:20.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.184+0000 7f2c6cc00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-12T22:16:20.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.184+0000 7f2c6cc00640 1 -- 172.21.15.28:0/3903860330 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c68009450 con 0x560f9339eeb0 2024-05-12T22:16:20.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.184+0000 7f2c6cc00640 1 -- 172.21.15.28:0/3903860330 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8513+0+0 (secure 0 0 0) 0x7f2c680cd070 con 0x560f9339eeb0 2024-05-12T22:16:20.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.184+0000 7f2c88870740 1 -- 172.21.15.28:0/3903860330 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2c78004050 con 0x560f9339eeb0 2024-05-12T22:16:20.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.184+0000 7f2c6cc00640 1 -- 172.21.15.28:0/3903860330 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f2c680cd270 con 0x560f9339eeb0 2024-05-12T22:16:20.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.184+0000 7f2c88870740 10 client.16758 fetch_fsmap learned FSMap version 26 2024-05-12T22:16:20.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.184+0000 7f2c88870740 1 -- 172.21.15.28:0/3903860330 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x560f9313ef70 con 0x560f9339eeb0 2024-05-12T22:16:20.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.185+0000 7f2c6cc00640 1 -- 172.21.15.28:0/3903860330 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2c78004050 con 0x560f9339eeb0 2024-05-12T22:16:20.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.185+0000 7f2c6cc00640 1 -- 172.21.15.28:0/3903860330 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f2c680d2ce0 con 0x560f9339eeb0 2024-05-12T22:16:20.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.185+0000 7f2c88870740 10 client.16758 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:16:20.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.185+0000 7f2c88870740 10 client.16758 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:16:20.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.185+0000 7f2c88870740 20 client.16758 populate_metadata read hostname 'smithi028' 2024-05-12T22:16:20.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.185+0000 7f2c88870740 1 --2- 172.21.15.28:0/3903860330 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x560f933b97b0 0x560f933bbba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:20.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.185+0000 7f2c88870740 4 client.16758 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-12T22:16:20.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.185+0000 7f2c88870740 1 -- 172.21.15.28:0/3903860330 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560f933163a0 con 0x560f933b97b0 2024-05-12T22:16:20.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.186+0000 7f2c7ea00640 1 --2- 172.21.15.28:0/3903860330 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x560f933b97b0 0x560f933bbba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:20.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.186+0000 7f2c7ea00640 1 --2- 172.21.15.28:0/3903860330 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x560f933b97b0 0x560f933bbba0 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=948639939e46066 server_cookie=8e649691a2225829 in_seq=0 out_seq=0 2024-05-12T22:16:20.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.186+0000 7f2c6cc00640 10 client.16758 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:16:20.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.187+0000 7f2c6cc00640 1 -- 172.21.15.28:0/3903860330 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560f933163a0 con 0x560f933b97b0 2024-05-12T22:16:20.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.187+0000 7f2c6cc00640 10 client.16758 handle_command_reply: tid=0 2024-05-12T22:16:20.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.187+0000 7f2c88870740 1 client.16758 shutdown 2024-05-12T22:16:20.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.187+0000 7f2c88870740 1 -- 172.21.15.28:0/3903860330 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f9339eeb0 msgr2=0x560f933a6980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:16:20.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.187+0000 7f2c88870740 1 --2- 172.21.15.28:0/3903860330 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f9339eeb0 0x560f933a6980 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f2c6800c6a0 tx=0x7f2c68097f90 comp rx=0 tx=0).stop 2024-05-12T22:16:20.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.187+0000 7f2c88870740 1 -- 172.21.15.28:0/3903860330 shutdown_connections 2024-05-12T22:16:20.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.187+0000 7f2c88870740 1 --2- 172.21.15.28:0/3903860330 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x560f933b97b0 0x560f933bbba0 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:20.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.187+0000 7f2c88870740 1 --2- 172.21.15.28:0/3903860330 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560f933a9520 0x560f933a6ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:20.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.188+0000 7f2c88870740 1 --2- 172.21.15.28:0/3903860330 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f9339eeb0 0x560f933a6980 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:20.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.188+0000 7f2c88870740 1 --2- 172.21.15.28:0/3903860330 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560f9339e500 0x560f933a6440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:20.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.188+0000 7f2c6cc00640 0 client.16758 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:16:20.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.188+0000 7f2c88870740 1 -- 172.21.15.28:0/3903860330 >> 172.21.15.28:0/3903860330 conn(0x560f9339a5b0 msgr2=0x560f933a0630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:16:20.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.188+0000 7f2c88870740 1 -- 172.21.15.28:0/3903860330 shutdown_connections 2024-05-12T22:16:20.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.188+0000 7f2c88870740 1 -- 172.21.15.28:0/3903860330 wait complete. 2024-05-12T22:16:20.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.188+0000 7f2c88870740 20 client.16758 trim_cache size 0 max 16384 2024-05-12T22:16:20.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.207+0000 7f2c88870740 1 Processor -- start 2024-05-12T22:16:20.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.207+0000 7f2c88870740 1 -- start start 2024-05-12T22:16:20.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.208+0000 7f2c88870740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560f9339e500 0x560f933b53f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:20.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.208+0000 7f2c88870740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560f9339eeb0 0x560f933995b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:20.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.208+0000 7f2c88870740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f933a9520 0x560f933c1450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:20.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.208+0000 7f2c88870740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2c801b1e80 con 0x560f933a9520 2024-05-12T22:16:20.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.208+0000 7f2c88870740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f2c801af750 con 0x560f9339eeb0 2024-05-12T22:16:20.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.208+0000 7f2c88870740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f2c801ad5c0 con 0x560f9339e500 2024-05-12T22:16:20.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.208+0000 7f2c7f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f933a9520 0x560f933c1450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:20.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.208+0000 7f2c7f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f933a9520 0x560f933c1450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55114/0 (socket says 172.21.15.28:55114) 2024-05-12T22:16:20.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.208+0000 7f2c7f400640 1 -- 172.21.15.28:0/353951441 learned_addr learned my addr 172.21.15.28:0/353951441 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:16:20.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.208+0000 7f2c7e000640 1 --2- 172.21.15.28:0/353951441 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560f9339eeb0 0x560f933995b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:20.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.208+0000 7f2c7ea00640 1 --2- 172.21.15.28:0/353951441 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560f9339e500 0x560f933b53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:20.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.208+0000 7f2c7f400640 1 -- 172.21.15.28:0/353951441 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560f9339e500 msgr2=0x560f933b53f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:16:20.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.208+0000 7f2c7f400640 1 --2- 172.21.15.28:0/353951441 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560f9339e500 0x560f933b53f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:20.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.208+0000 7f2c7f400640 1 -- 172.21.15.28:0/353951441 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560f9339eeb0 msgr2=0x560f933995b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:16:20.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.208+0000 7f2c7f400640 1 --2- 172.21.15.28:0/353951441 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560f9339eeb0 0x560f933995b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:20.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.208+0000 7f2c7f400640 1 -- 172.21.15.28:0/353951441 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560f92fd6170 con 0x560f933a9520 2024-05-12T22:16:20.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.208+0000 7f2c7e000640 1 --2- 172.21.15.28:0/353951441 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560f9339eeb0 0x560f933995b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:16:20.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.209+0000 7f2c7f400640 1 --2- 172.21.15.28:0/353951441 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f933a9520 0x560f933c1450 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f2c78009f50 tx=0x7f2c78005ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c046670b9b28297 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:16:20.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.209+0000 7f2c6f400640 1 -- 172.21.15.28:0/353951441 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c78003c30 con 0x560f933a9520 2024-05-12T22:16:20.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.209+0000 7f2c6f400640 1 -- 172.21.15.28:0/353951441 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2c78003dd0 con 0x560f933a9520 2024-05-12T22:16:20.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.209+0000 7f2c6f400640 1 -- 172.21.15.28:0/353951441 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c7800fa60 con 0x560f933a9520 2024-05-12T22:16:20.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.210+0000 7f2c88870740 1 -- 172.21.15.28:0/353951441 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f933a9520 msgr2=0x560f933c1450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:16:20.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.210+0000 7f2c88870740 1 --2- 172.21.15.28:0/353951441 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f933a9520 0x560f933c1450 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f2c78009f50 tx=0x7f2c78005ad0 comp rx=0 tx=0).stop 2024-05-12T22:16:20.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.210+0000 7f2c88870740 1 -- 172.21.15.28:0/353951441 shutdown_connections 2024-05-12T22:16:20.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.210+0000 7f2c88870740 1 --2- 172.21.15.28:0/353951441 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f933a9520 0x560f933c1450 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:20.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.210+0000 7f2c88870740 1 --2- 172.21.15.28:0/353951441 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560f9339eeb0 0x560f933995b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:20.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.210+0000 7f2c88870740 1 --2- 172.21.15.28:0/353951441 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560f9339e500 0x560f933b53f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:20.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.210+0000 7f2c88870740 1 -- 172.21.15.28:0/353951441 >> 172.21.15.28:0/353951441 conn(0x560f9339a5b0 msgr2=0x560f93398ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:16:20.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.210+0000 7f2c88870740 1 -- 172.21.15.28:0/353951441 shutdown_connections 2024-05-12T22:16:20.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.210+0000 7f2c88870740 1 -- 172.21.15.28:0/353951441 wait complete. 2024-05-12T22:16:20.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.211+0000 7f2c88870740 1 -- start start 2024-05-12T22:16:20.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.211+0000 7f2c88870740 1 Processor -- start 2024-05-12T22:16:20.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.211+0000 7f2c88870740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560f9339e500 0x560f933b1e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:20.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.211+0000 7f2c88870740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f9339eeb0 0x560f933a3000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:20.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.211+0000 7f2c88870740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560f933a9520 0x560f933a3740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:20.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.211+0000 7f2c88870740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2c801adc10 con 0x560f9339eeb0 2024-05-12T22:16:20.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.211+0000 7f2c88870740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f2c801af080 con 0x560f9339e500 2024-05-12T22:16:20.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.211+0000 7f2c88870740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f2c801ae2e0 con 0x560f933a9520 2024-05-12T22:16:20.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.211+0000 7f2c88870740 10 client.0 _ll_register_callbacks cb 0x7f2c680d5220 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:16:20.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.212+0000 7f2c7e000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f9339eeb0 0x560f933a3000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:20.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.212+0000 7f2c7e000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f9339eeb0 0x560f933a3000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55122/0 (socket says 172.21.15.28:55122) 2024-05-12T22:16:20.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.212+0000 7f2c7e000640 1 -- 172.21.15.28:0/1852410746 learned_addr learned my addr 172.21.15.28:0/1852410746 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:16:20.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.212+0000 7f2c7ea00640 1 --2- 172.21.15.28:0/1852410746 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560f9339e500 0x560f933b1e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:20.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.212+0000 7f2c7f400640 1 --2- 172.21.15.28:0/1852410746 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560f933a9520 0x560f933a3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:20.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.212+0000 7f2c7e000640 1 -- 172.21.15.28:0/1852410746 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560f933a9520 msgr2=0x560f933a3740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:16:20.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.212+0000 7f2c7e000640 1 --2- 172.21.15.28:0/1852410746 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560f933a9520 0x560f933a3740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:20.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.212+0000 7f2c7e000640 1 -- 172.21.15.28:0/1852410746 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560f9339e500 msgr2=0x560f933b1e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:16:20.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.212+0000 7f2c7e000640 1 --2- 172.21.15.28:0/1852410746 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560f9339e500 0x560f933b1e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:20.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.212+0000 7f2c7e000640 1 -- 172.21.15.28:0/1852410746 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560f934274d0 con 0x560f9339eeb0 2024-05-12T22:16:20.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.212+0000 7f2c7f400640 1 --2- 172.21.15.28:0/1852410746 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560f933a9520 0x560f933a3740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:16:20.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.212+0000 7f2c7e000640 1 --2- 172.21.15.28:0/1852410746 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f9339eeb0 0x560f933a3000 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f2c6809b8c0 tx=0x7f2c680cd320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6d37cd792072584 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:16:20.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.212+0000 7f2c6cc00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-12T22:16:20.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.213+0000 7f2c6cc00640 1 -- 172.21.15.28:0/1852410746 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c6809b700 con 0x560f9339eeb0 2024-05-12T22:16:20.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.213+0000 7f2c6cc00640 1 -- 172.21.15.28:0/1852410746 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8513+0+0 (secure 0 0 0) 0x7f2c680ba2b0 con 0x560f9339eeb0 2024-05-12T22:16:20.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.213+0000 7f2c88870740 1 -- 172.21.15.28:0/1852410746 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2c78003dd0 con 0x560f9339eeb0 2024-05-12T22:16:20.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.213+0000 7f2c6cc00640 1 -- 172.21.15.28:0/1852410746 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f2c6809c2c0 con 0x560f9339eeb0 2024-05-12T22:16:20.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.213+0000 7f2c88870740 10 client.16764 fetch_fsmap learned FSMap version 26 2024-05-12T22:16:20.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.213+0000 7f2c88870740 1 -- 172.21.15.28:0/1852410746 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x560f934286d0 con 0x560f9339eeb0 2024-05-12T22:16:20.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.213+0000 7f2c6cc00640 1 -- 172.21.15.28:0/1852410746 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2c78003dd0 con 0x560f9339eeb0 2024-05-12T22:16:20.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.213+0000 7f2c6cc00640 1 -- 172.21.15.28:0/1852410746 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f2c680cfaa0 con 0x560f9339eeb0 2024-05-12T22:16:20.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.213+0000 7f2c88870740 10 client.16764 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:16:20.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.213+0000 7f2c88870740 10 client.16764 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:16:20.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.213+0000 7f2c88870740 20 client.16764 populate_metadata read hostname 'smithi028' 2024-05-12T22:16:20.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.213+0000 7f2c88870740 1 --2- 172.21.15.28:0/1852410746 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x560f93428e80 0x560f9342b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:20.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.213+0000 7f2c88870740 4 client.16764 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-12T22:16:20.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.213+0000 7f2c88870740 1 -- 172.21.15.28:0/1852410746 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x560f9330fef0 con 0x560f93428e80 2024-05-12T22:16:20.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.214+0000 7f2c7ea00640 1 --2- 172.21.15.28:0/1852410746 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x560f93428e80 0x560f9342b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:20.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.214+0000 7f2c7ea00640 1 --2- 172.21.15.28:0/1852410746 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x560f93428e80 0x560f9342b2a0 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=30eff6f270bc2f18 server_cookie=b88a854fba45439c in_seq=0 out_seq=0 2024-05-12T22:16:20.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.214+0000 7f2c6cc00640 10 client.16764 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:16:20.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.215+0000 7f2c6cc00640 1 -- 172.21.15.28:0/1852410746 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x560f9330fef0 con 0x560f93428e80 2024-05-12T22:16:20.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.216+0000 7f2c6cc00640 10 client.16764 handle_command_reply: tid=0 2024-05-12T22:16:20.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.216+0000 7f2c88870740 1 client.16764 shutdown 2024-05-12T22:16:20.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.216+0000 7f2c88870740 1 -- 172.21.15.28:0/1852410746 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f9339eeb0 msgr2=0x560f933a3000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:16:20.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.216+0000 7f2c88870740 1 --2- 172.21.15.28:0/1852410746 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f9339eeb0 0x560f933a3000 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f2c6809b8c0 tx=0x7f2c680cd320 comp rx=0 tx=0).stop 2024-05-12T22:16:20.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.216+0000 7f2c88870740 1 -- 172.21.15.28:0/1852410746 shutdown_connections 2024-05-12T22:16:20.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.216+0000 7f2c88870740 1 --2- 172.21.15.28:0/1852410746 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x560f93428e80 0x560f9342b2a0 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:20.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.216+0000 7f2c88870740 1 --2- 172.21.15.28:0/1852410746 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560f933a9520 0x560f933a3740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:20.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.216+0000 7f2c88870740 1 --2- 172.21.15.28:0/1852410746 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f9339eeb0 0x560f933a3000 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:20.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.216+0000 7f2c88870740 1 --2- 172.21.15.28:0/1852410746 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560f9339e500 0x560f933b1e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:20.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.216+0000 7f2c6cc00640 0 client.16764 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:16:20.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.216+0000 7f2c88870740 1 -- 172.21.15.28:0/1852410746 >> 172.21.15.28:0/1852410746 conn(0x560f9339a5b0 msgr2=0x560f933a6c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:16:20.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.216+0000 7f2c88870740 1 -- 172.21.15.28:0/1852410746 shutdown_connections 2024-05-12T22:16:20.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.216+0000 7f2c88870740 1 -- 172.21.15.28:0/1852410746 wait complete. 2024-05-12T22:16:20.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.216+0000 7f2c88870740 20 client.16764 trim_cache size 0 max 16384 2024-05-12T22:16:20.225 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-05-12T22:16:20.225 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-05-12T22:16:20.225 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "201fc886-efac-4914-932d-1f7d30ee7804", 2024-05-12T22:16:20.226 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-05-12T22:16:20.226 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-05-12T22:16:20.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.217+0000 7f2c7fe00640 1 -- 172.21.15.28:0/3297205849 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f2c640a2d80 msgr2=0x7f2c640a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:16:20.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.217+0000 7f2c7fe00640 1 --2- 172.21.15.28:0/3297205849 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f2c640a2d80 0x7f2c640a5240 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x560f9339fac0 tx=0x7f2c70009620 comp rx=0 tx=0).stop 2024-05-12T22:16:20.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.217+0000 7f2c7fe00640 1 -- 172.21.15.28:0/3297205849 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2c80104290 msgr2=0x7f2c8019a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:16:20.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.217+0000 7f2c7fe00640 1 --2- 172.21.15.28:0/3297205849 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2c80104290 0x7f2c8019a730 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f2c6800e9f0 tx=0x7f2c6800eeb0 comp rx=0 tx=0).stop 2024-05-12T22:16:20.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.217+0000 7f2c7fe00640 1 -- 172.21.15.28:0/3297205849 shutdown_connections 2024-05-12T22:16:20.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.217+0000 7f2c7fe00640 1 --2- 172.21.15.28:0/3297205849 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f2c640a2d80 0x7f2c640a5240 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:20.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.217+0000 7f2c7fe00640 1 --2- 172.21.15.28:0/3297205849 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2c801098e0 0x7f2c8019eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:20.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.217+0000 7f2c7fe00640 1 --2- 172.21.15.28:0/3297205849 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2c80104290 0x7f2c8019a730 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:20.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.217+0000 7f2c7fe00640 1 --2- 172.21.15.28:0/3297205849 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2c801038d0 0x7f2c8019a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:20.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.217+0000 7f2c7fe00640 1 -- 172.21.15.28:0/3297205849 >> 172.21.15.28:0/3297205849 conn(0x7f2c800ff3a0 msgr2=0x7f2c800ffc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:16:20.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.217+0000 7f2c7fe00640 1 -- 172.21.15.28:0/3297205849 shutdown_connections 2024-05-12T22:16:20.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.217+0000 7f2c7fe00640 1 -- 172.21.15.28:0/3297205849 wait complete. 2024-05-12T22:16:20.237 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-12T22:16:20.238 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-12T22:16:20.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:16:20 smithi028 ceph-mon[29847]: pgmap v225: 97 pgs: 97 active+clean; 97 MiB data, 757 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-05-12T22:16:20.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:16:20 smithi028 ceph-mon[29847]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:16:20.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:16:20 smithi028 ceph-mon[29847]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:16:20.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:16:20 smithi028 ceph-mon[29847]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:16:20.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:16:20 smithi186 ceph-mon[34785]: pgmap v225: 97 pgs: 97 active+clean; 97 MiB data, 757 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-05-12T22:16:20.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:16:20 smithi186 ceph-mon[34785]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:16:20.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:16:20 smithi186 ceph-mon[34785]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:16:20.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:16:20 smithi186 ceph-mon[34785]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:16:20.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.463+0000 7f2543800640 1 -- 172.21.15.28:0/859517400 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f253c103a70 msgr2=0x7f253c103ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:16:20.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.463+0000 7f2543800640 1 --2- 172.21.15.28:0/859517400 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f253c103a70 0x7f253c103ef0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f252c009900 tx=0x7f252c02a410 comp rx=0 tx=0).stop 2024-05-12T22:16:20.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.463+0000 7f2543800640 1 -- 172.21.15.28:0/859517400 shutdown_connections 2024-05-12T22:16:20.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.463+0000 7f2543800640 1 --2- 172.21.15.28:0/859517400 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f253c104430 0x7f253c110940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:20.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.463+0000 7f2543800640 1 --2- 172.21.15.28:0/859517400 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f253c103a70 0x7f253c103ef0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:20.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.463+0000 7f2543800640 1 --2- 172.21.15.28:0/859517400 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f253c109a80 0x7f253c109e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:20.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.463+0000 7f2543800640 1 -- 172.21.15.28:0/859517400 >> 172.21.15.28:0/859517400 conn(0x7f253c0ff5c0 msgr2=0x7f253c1019e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:16:20.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.463+0000 7f2543800640 1 -- 172.21.15.28:0/859517400 shutdown_connections 2024-05-12T22:16:20.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.463+0000 7f2543800640 1 -- 172.21.15.28:0/859517400 wait complete. 2024-05-12T22:16:20.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.464+0000 7f2543800640 1 Processor -- start 2024-05-12T22:16:20.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.465+0000 7f2543800640 1 -- start start 2024-05-12T22:16:20.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.465+0000 7f2543800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f253c103a70 0x7f253c19a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:20.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.465+0000 7f2543800640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f253c104430 0x7f253c19a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:20.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.465+0000 7f2543800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f253c109a80 0x7f253c19ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:20.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.465+0000 7f2543800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f253c113ee0 con 0x7f253c109a80 2024-05-12T22:16:20.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.465+0000 7f2543800640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f253c113d60 con 0x7f253c103a70 2024-05-12T22:16:20.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.465+0000 7f2543800640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f253c114060 con 0x7f253c104430 2024-05-12T22:16:20.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.465+0000 7f2540e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f253c103a70 0x7f253c19a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:20.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.465+0000 7f253be00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f253c104430 0x7f253c19a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:20.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.465+0000 7f2540e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f253c103a70 0x7f253c19a3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:54988/0 (socket says 172.21.15.28:54988) 2024-05-12T22:16:20.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.465+0000 7f2540e00640 1 -- 172.21.15.28:0/2584779898 learned_addr learned my addr 172.21.15.28:0/2584779898 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:16:20.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.465+0000 7f253be00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f253c104430 0x7f253c19a900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:47352/0 (socket says 172.21.15.28:47352) 2024-05-12T22:16:20.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.466+0000 7f2541800640 1 --2- 172.21.15.28:0/2584779898 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f253c109a80 0x7f253c19ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:20.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.466+0000 7f253be00640 1 -- 172.21.15.28:0/2584779898 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f253c103a70 msgr2=0x7f253c19a3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:16:20.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.466+0000 7f253be00640 1 --2- 172.21.15.28:0/2584779898 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f253c103a70 0x7f253c19a3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:20.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.466+0000 7f253be00640 1 -- 172.21.15.28:0/2584779898 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f253c109a80 msgr2=0x7f253c19ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:16:20.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.466+0000 7f253be00640 1 --2- 172.21.15.28:0/2584779898 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f253c109a80 0x7f253c19ec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:20.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.466+0000 7f253be00640 1 -- 172.21.15.28:0/2584779898 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f253c19f3b0 con 0x7f253c104430 2024-05-12T22:16:20.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.466+0000 7f2540e00640 1 --2- 172.21.15.28:0/2584779898 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f253c103a70 0x7f253c19a3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:16:20.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.467+0000 7f253be00640 1 --2- 172.21.15.28:0/2584779898 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f253c104430 0x7f253c19a900 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f252c02a930 tx=0x7f252c003b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:16:20.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.467+0000 7f2539600640 1 -- 172.21.15.28:0/2584779898 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f252c03f070 con 0x7f253c104430 2024-05-12T22:16:20.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.467+0000 7f2539600640 1 -- 172.21.15.28:0/2584779898 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f252c031c20 con 0x7f253c104430 2024-05-12T22:16:20.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.467+0000 7f2543800640 1 -- 172.21.15.28:0/2584779898 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f253c19f640 con 0x7f253c104430 2024-05-12T22:16:20.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.467+0000 7f2539600640 1 -- 172.21.15.28:0/2584779898 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f252c035030 con 0x7f253c104430 2024-05-12T22:16:20.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.467+0000 7f2543800640 1 -- 172.21.15.28:0/2584779898 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f253c1a6f30 con 0x7f253c104430 2024-05-12T22:16:20.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.471+0000 7f2546643740 1 Processor -- start 2024-05-12T22:16:20.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.471+0000 7f2539600640 1 -- 172.21.15.28:0/2584779898 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7f252c042070 con 0x7f253c104430 2024-05-12T22:16:20.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.471+0000 7f2546643740 1 -- start start 2024-05-12T22:16:20.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.472+0000 7f2546643740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625bc7dcbc0 0x5625bc7dcfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:20.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.472+0000 7f2546643740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5625bc7dd4e0 0x5625bc7e7670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:20.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.472+0000 7f2546643740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5625bc7e7bb0 0x5625bc7e9fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:20.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.472+0000 7f2546643740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5625bc7a52a0 con 0x5625bc7dcbc0 2024-05-12T22:16:20.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.472+0000 7f2546643740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x5625bc741b90 con 0x5625bc7dd4e0 2024-05-12T22:16:20.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.472+0000 7f2546643740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5625bc6e95b0 con 0x5625bc7e7bb0 2024-05-12T22:16:20.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.472+0000 7f2541800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625bc7dcbc0 0x5625bc7dcfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:20.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.472+0000 7f253be00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5625bc7e7bb0 0x5625bc7e9fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:20.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.472+0000 7f253be00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5625bc7e7bb0 0x5625bc7e9fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:47356/0 (socket says 172.21.15.28:47356) 2024-05-12T22:16:20.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.472+0000 7f253be00640 1 -- 172.21.15.28:0/3251937325 learned_addr learned my addr 172.21.15.28:0/3251937325 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:16:20.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.472+0000 7f2540e00640 1 --2- 172.21.15.28:0/3251937325 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5625bc7dd4e0 0x5625bc7e7670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:20.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.473+0000 7f253be00640 1 -- 172.21.15.28:0/3251937325 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5625bc7dd4e0 msgr2=0x5625bc7e7670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:16:20.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.473+0000 7f253be00640 1 --2- 172.21.15.28:0/3251937325 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5625bc7dd4e0 0x5625bc7e7670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:20.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.473+0000 7f253be00640 1 -- 172.21.15.28:0/3251937325 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625bc7dcbc0 msgr2=0x5625bc7dcfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:16:20.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.473+0000 7f253be00640 1 --2- 172.21.15.28:0/3251937325 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625bc7dcbc0 0x5625bc7dcfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:20.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.473+0000 7f253be00640 1 -- 172.21.15.28:0/3251937325 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5625bc51f7d0 con 0x5625bc7e7bb0 2024-05-12T22:16:20.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.473+0000 7f253be00640 1 --2- 172.21.15.28:0/3251937325 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5625bc7e7bb0 0x5625bc7e9fa0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f252c0ed330 tx=0x7f252c0f3f30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=43f2f43577334443 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:16:20.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.473+0000 7f252a000640 1 -- 172.21.15.28:0/3251937325 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f252c0ff070 con 0x5625bc7e7bb0 2024-05-12T22:16:20.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.474+0000 7f2539600640 1 --2- 172.21.15.28:0/2584779898 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f25240a2d80 0x7f25240a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:20.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.474+0000 7f252a000640 1 -- 172.21.15.28:0/3251937325 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f252c0f8b90 con 0x5625bc7e7bb0 2024-05-12T22:16:20.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.474+0000 7f252a000640 1 -- 172.21.15.28:0/3251937325 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f252c0b8460 con 0x5625bc7e7bb0 2024-05-12T22:16:20.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.474+0000 7f2541800640 1 --2- 172.21.15.28:0/2584779898 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f25240a2d80 0x7f25240a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:20.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.474+0000 7f2539600640 1 -- 172.21.15.28:0/2584779898 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8513+0+0 (secure 0 0 0) 0x7f252c0efcc0 con 0x7f253c104430 2024-05-12T22:16:20.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.475+0000 7f2546643740 1 -- 172.21.15.28:0/3251937325 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5625bc7e7bb0 msgr2=0x5625bc7e9fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:16:20.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.475+0000 7f2546643740 1 --2- 172.21.15.28:0/3251937325 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5625bc7e7bb0 0x5625bc7e9fa0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f252c0ed330 tx=0x7f252c0f3f30 comp rx=0 tx=0).stop 2024-05-12T22:16:20.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.475+0000 7f2546643740 1 -- 172.21.15.28:0/3251937325 shutdown_connections 2024-05-12T22:16:20.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.475+0000 7f2546643740 1 --2- 172.21.15.28:0/3251937325 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5625bc7e7bb0 0x5625bc7e9fa0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:20.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.475+0000 7f2546643740 1 --2- 172.21.15.28:0/3251937325 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5625bc7dd4e0 0x5625bc7e7670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:20.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.475+0000 7f2546643740 1 --2- 172.21.15.28:0/3251937325 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625bc7dcbc0 0x5625bc7dcfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:20.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.475+0000 7f2546643740 1 -- 172.21.15.28:0/3251937325 >> 172.21.15.28:0/3251937325 conn(0x5625bc7d8ee0 msgr2=0x5625bc7d92c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:16:20.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.475+0000 7f2541800640 1 --2- 172.21.15.28:0/2584779898 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f25240a2d80 0x7f25240a5240 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f2534005fd0 tx=0x7f2534009500 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:16:20.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.475+0000 7f2546643740 1 -- 172.21.15.28:0/3251937325 shutdown_connections 2024-05-12T22:16:20.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.475+0000 7f2546643740 1 -- 172.21.15.28:0/3251937325 wait complete. 2024-05-12T22:16:20.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.476+0000 7f2546643740 1 -- start start 2024-05-12T22:16:20.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.476+0000 7f2546643740 1 Processor -- start 2024-05-12T22:16:20.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.476+0000 7f2546643740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625bc7dcbc0 0x5625bc7e7550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:20.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.477+0000 7f2546643740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5625bc7dd4e0 0x5625bc7e4840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:20.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.477+0000 7f2546643740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5625bc7e7bb0 0x5625bc7e4de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:20.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.477+0000 7f2546643740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5625bc6fc9f0 con 0x5625bc7dcbc0 2024-05-12T22:16:20.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.477+0000 7f2546643740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x5625bc73db50 con 0x5625bc7dd4e0 2024-05-12T22:16:20.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.477+0000 7f2546643740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5625bc7ea760 con 0x5625bc7e7bb0 2024-05-12T22:16:20.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.477+0000 7f2546643740 10 client.0 _ll_register_callbacks cb 0x5625bc454be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:16:20.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.477+0000 7f2541800640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5625bc7e7bb0 0x5625bc7e4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:20.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.477+0000 7f253be00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5625bc7dd4e0 0x5625bc7e4840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:20.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.477+0000 7f2541800640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5625bc7e7bb0 0x5625bc7e4de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:47362/0 (socket says 172.21.15.28:47362) 2024-05-12T22:16:20.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.477+0000 7f2541800640 1 -- 172.21.15.28:0/3561872775 learned_addr learned my addr 172.21.15.28:0/3561872775 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:16:20.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.477+0000 7f2540e00640 1 --2- 172.21.15.28:0/3561872775 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625bc7dcbc0 0x5625bc7e7550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:20.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.477+0000 7f2541800640 1 -- 172.21.15.28:0/3561872775 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5625bc7dd4e0 msgr2=0x5625bc7e4840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:16:20.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.477+0000 7f2541800640 1 --2- 172.21.15.28:0/3561872775 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5625bc7dd4e0 0x5625bc7e4840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:20.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.477+0000 7f2541800640 1 -- 172.21.15.28:0/3561872775 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625bc7dcbc0 msgr2=0x5625bc7e7550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:16:20.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.477+0000 7f2541800640 1 --2- 172.21.15.28:0/3561872775 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625bc7dcbc0 0x5625bc7e7550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:20.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.478+0000 7f2541800640 1 -- 172.21.15.28:0/3561872775 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5625bc4553f0 con 0x5625bc7e7bb0 2024-05-12T22:16:20.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.478+0000 7f253be00640 1 --2- 172.21.15.28:0/3561872775 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5625bc7dd4e0 0x5625bc7e4840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:16:20.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.478+0000 7f2541800640 1 --2- 172.21.15.28:0/3561872775 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5625bc7e7bb0 0x5625bc7e4de0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f2534009b80 tx=0x7f2534004ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=baee2ad4e8c62d81 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:16:20.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.478+0000 7f251ea00640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-12T22:16:20.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.479+0000 7f251ea00640 1 -- 172.21.15.28:0/3561872775 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f253400f950 con 0x5625bc7e7bb0 2024-05-12T22:16:20.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.479+0000 7f251ea00640 1 -- 172.21.15.28:0/3561872775 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8513+0+0 (secure 0 0 0) 0x7f253400faf0 con 0x5625bc7e7bb0 2024-05-12T22:16:20.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.479+0000 7f2546643740 1 -- 172.21.15.28:0/3561872775 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f252c0f8b90 con 0x5625bc7e7bb0 2024-05-12T22:16:20.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.479+0000 7f251ea00640 1 -- 172.21.15.28:0/3561872775 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f253400fcf0 con 0x5625bc7e7bb0 2024-05-12T22:16:20.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.479+0000 7f2546643740 10 client.26860 fetch_fsmap learned FSMap version 26 2024-05-12T22:16:20.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.480+0000 7f2546643740 1 -- 172.21.15.28:0/3561872775 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5625bc4150f0 con 0x5625bc7e7bb0 2024-05-12T22:16:20.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.480+0000 7f251ea00640 1 -- 172.21.15.28:0/3561872775 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f252c0f8b90 con 0x5625bc7e7bb0 2024-05-12T22:16:20.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.481+0000 7f251ea00640 1 -- 172.21.15.28:0/3561872775 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f2534027070 con 0x5625bc7e7bb0 2024-05-12T22:16:20.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.481+0000 7f2546643740 10 client.26860 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:16:20.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.481+0000 7f2546643740 10 client.26860 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:16:20.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.481+0000 7f2546643740 20 client.26860 populate_metadata read hostname 'smithi028' 2024-05-12T22:16:20.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.481+0000 7f2546643740 1 --2- 172.21.15.28:0/3561872775 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5625bc7f7560 0x5625bc7f99b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:20.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.481+0000 7f2546643740 4 client.26860 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-12T22:16:20.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.481+0000 7f2546643740 1 -- 172.21.15.28:0/3561872775 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5625bc74f2c0 con 0x5625bc7f7560 2024-05-12T22:16:20.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.481+0000 7f2540e00640 1 --2- 172.21.15.28:0/3561872775 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5625bc7f7560 0x5625bc7f99b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:20.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.482+0000 7f2540e00640 1 --2- 172.21.15.28:0/3561872775 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5625bc7f7560 0x5625bc7f99b0 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dec2a41d7e41fe03 server_cookie=d97c276ae6129581 in_seq=0 out_seq=0 2024-05-12T22:16:20.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.482+0000 7f251ea00640 10 client.26860 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:16:20.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.483+0000 7f251ea00640 1 -- 172.21.15.28:0/3561872775 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5625bc74f2c0 con 0x5625bc7f7560 2024-05-12T22:16:20.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.483+0000 7f251ea00640 10 client.26860 handle_command_reply: tid=0 2024-05-12T22:16:20.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.483+0000 7f2546643740 1 client.26860 shutdown 2024-05-12T22:16:20.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.483+0000 7f2546643740 1 -- 172.21.15.28:0/3561872775 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5625bc7e7bb0 msgr2=0x5625bc7e4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:16:20.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.483+0000 7f2546643740 1 --2- 172.21.15.28:0/3561872775 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5625bc7e7bb0 0x5625bc7e4de0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f2534009b80 tx=0x7f2534004ab0 comp rx=0 tx=0).stop 2024-05-12T22:16:20.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.483+0000 7f2546643740 1 -- 172.21.15.28:0/3561872775 shutdown_connections 2024-05-12T22:16:20.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.483+0000 7f2546643740 1 --2- 172.21.15.28:0/3561872775 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5625bc7f7560 0x5625bc7f99b0 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:20.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.483+0000 7f2546643740 1 --2- 172.21.15.28:0/3561872775 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5625bc7e7bb0 0x5625bc7e4de0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:20.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.483+0000 7f2546643740 1 --2- 172.21.15.28:0/3561872775 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5625bc7dd4e0 0x5625bc7e4840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:20.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.483+0000 7f2546643740 1 --2- 172.21.15.28:0/3561872775 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625bc7dcbc0 0x5625bc7e7550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:20.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.483+0000 7f251ea00640 0 client.26860 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:16:20.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.483+0000 7f2546643740 1 -- 172.21.15.28:0/3561872775 >> 172.21.15.28:0/3561872775 conn(0x5625bc7d8ee0 msgr2=0x5625bc7d80e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:16:20.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.483+0000 7f2546643740 1 -- 172.21.15.28:0/3561872775 shutdown_connections 2024-05-12T22:16:20.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.483+0000 7f2546643740 1 -- 172.21.15.28:0/3561872775 wait complete. 2024-05-12T22:16:20.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.484+0000 7f2546643740 20 client.26860 trim_cache size 0 max 16384 2024-05-12T22:16:20.491 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:16:20 smithi086 ceph-mon[34297]: pgmap v225: 97 pgs: 97 active+clean; 97 MiB data, 757 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-05-12T22:16:20.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:16:20 smithi086 ceph-mon[34297]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:16:20.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:16:20 smithi086 ceph-mon[34297]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:16:20.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:16:20 smithi086 ceph-mon[34297]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:16:20.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.508+0000 7f2546643740 1 Processor -- start 2024-05-12T22:16:20.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.509+0000 7f2546643740 1 -- start start 2024-05-12T22:16:20.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.509+0000 7f2546643740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5625bc7dcbc0 0x5625bc7d8860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:20.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.509+0000 7f2546643740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625bc7dd4e0 0x5625bc7e38b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:20.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.509+0000 7f2546643740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5625bc7e7bb0 0x5625bc7e7080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:20.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.509+0000 7f2546643740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f253c1b2080 con 0x5625bc7dd4e0 2024-05-12T22:16:20.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.509+0000 7f2546643740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f253c1af950 con 0x5625bc7e7bb0 2024-05-12T22:16:20.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.509+0000 7f2546643740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f253c1ad7c0 con 0x5625bc7dcbc0 2024-05-12T22:16:20.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.509+0000 7f2540e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5625bc7dcbc0 0x5625bc7d8860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:20.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.509+0000 7f253be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625bc7dd4e0 0x5625bc7e38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:20.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.509+0000 7f2541800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5625bc7e7bb0 0x5625bc7e7080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:20.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.509+0000 7f253be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625bc7dd4e0 0x5625bc7e38b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55166/0 (socket says 172.21.15.28:55166) 2024-05-12T22:16:20.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.509+0000 7f2541800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5625bc7e7bb0 0x5625bc7e7080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:55010/0 (socket says 172.21.15.28:55010) 2024-05-12T22:16:20.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.509+0000 7f253be00640 1 -- 172.21.15.28:0/3183332787 learned_addr learned my addr 172.21.15.28:0/3183332787 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:16:20.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.509+0000 7f2540e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5625bc7dcbc0 0x5625bc7d8860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:47366/0 (socket says 172.21.15.28:47366) 2024-05-12T22:16:20.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.510+0000 7f2540e00640 1 -- 172.21.15.28:0/3183332787 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5625bc7e7bb0 msgr2=0x5625bc7e7080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:16:20.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.510+0000 7f2540e00640 1 --2- 172.21.15.28:0/3183332787 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5625bc7e7bb0 0x5625bc7e7080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:20.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.510+0000 7f2540e00640 1 -- 172.21.15.28:0/3183332787 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625bc7dd4e0 msgr2=0x5625bc7e38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:16:20.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.510+0000 7f2540e00640 1 --2- 172.21.15.28:0/3183332787 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625bc7dd4e0 0x5625bc7e38b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:20.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.510+0000 7f2540e00640 1 -- 172.21.15.28:0/3183332787 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5625bc7e2dd0 con 0x5625bc7dcbc0 2024-05-12T22:16:20.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.510+0000 7f2541800640 1 --2- 172.21.15.28:0/3183332787 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5625bc7e7bb0 0x5625bc7e7080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:16:20.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.510+0000 7f2540e00640 1 --2- 172.21.15.28:0/3183332787 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5625bc7dcbc0 0x5625bc7d8860 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x5625bc7f9fb0 tx=0x7f2530005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=576b1db8952d6119 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:16:20.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.510+0000 7f2529600640 1 -- 172.21.15.28:0/3183332787 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2530004030 con 0x5625bc7dcbc0 2024-05-12T22:16:20.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.510+0000 7f2529600640 1 -- 172.21.15.28:0/3183332787 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2530024030 con 0x5625bc7dcbc0 2024-05-12T22:16:20.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.510+0000 7f2529600640 1 -- 172.21.15.28:0/3183332787 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25300053a0 con 0x5625bc7dcbc0 2024-05-12T22:16:20.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.511+0000 7f2546643740 1 -- 172.21.15.28:0/3183332787 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5625bc7dcbc0 msgr2=0x5625bc7d8860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:16:20.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.511+0000 7f2546643740 1 --2- 172.21.15.28:0/3183332787 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5625bc7dcbc0 0x5625bc7d8860 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x5625bc7f9fb0 tx=0x7f2530005f90 comp rx=0 tx=0).stop 2024-05-12T22:16:20.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.512+0000 7f2546643740 1 -- 172.21.15.28:0/3183332787 shutdown_connections 2024-05-12T22:16:20.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.512+0000 7f2546643740 1 --2- 172.21.15.28:0/3183332787 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5625bc7e7bb0 0x5625bc7e7080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:20.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.512+0000 7f2546643740 1 --2- 172.21.15.28:0/3183332787 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625bc7dd4e0 0x5625bc7e38b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:20.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.512+0000 7f2546643740 1 --2- 172.21.15.28:0/3183332787 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5625bc7dcbc0 0x5625bc7d8860 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:20.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.512+0000 7f2546643740 1 -- 172.21.15.28:0/3183332787 >> 172.21.15.28:0/3183332787 conn(0x5625bc7d8ee0 msgr2=0x5625bc7df410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:16:20.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.512+0000 7f2546643740 1 -- 172.21.15.28:0/3183332787 shutdown_connections 2024-05-12T22:16:20.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.512+0000 7f2546643740 1 -- 172.21.15.28:0/3183332787 wait complete. 2024-05-12T22:16:20.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.512+0000 7f2546643740 1 -- start start 2024-05-12T22:16:20.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.512+0000 7f2546643740 1 Processor -- start 2024-05-12T22:16:20.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.513+0000 7f2546643740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5625bc7dcbc0 0x5625bc7efd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:20.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.513+0000 7f2546643740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5625bc7dd4e0 0x5625bc7e12e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:20.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.513+0000 7f2546643740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625bc7e7bb0 0x5625bc7e1840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:20.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.513+0000 7f2546643740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f253c1ade10 con 0x5625bc7e7bb0 2024-05-12T22:16:20.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.513+0000 7f2546643740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f253c1af280 con 0x5625bc7dd4e0 2024-05-12T22:16:20.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.513+0000 7f2546643740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f253c1ae4e0 con 0x5625bc7dcbc0 2024-05-12T22:16:20.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.513+0000 7f2546643740 10 client.0 _ll_register_callbacks cb 0x7f2534009e70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:16:20.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.513+0000 7f2540e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5625bc7dcbc0 0x5625bc7efd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:20.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.513+0000 7f2540e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5625bc7dcbc0 0x5625bc7efd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:47372/0 (socket says 172.21.15.28:47372) 2024-05-12T22:16:20.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.514+0000 7f2540e00640 1 -- 172.21.15.28:0/2511042998 learned_addr learned my addr 172.21.15.28:0/2511042998 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:16:20.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.514+0000 7f253be00640 1 --2- 172.21.15.28:0/2511042998 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5625bc7dd4e0 0x5625bc7e12e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:20.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.514+0000 7f2541800640 1 --2- 172.21.15.28:0/2511042998 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625bc7e7bb0 0x5625bc7e1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:20.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.514+0000 7f2540e00640 1 -- 172.21.15.28:0/2511042998 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5625bc7dd4e0 msgr2=0x5625bc7e12e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:16:20.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.514+0000 7f2540e00640 1 --2- 172.21.15.28:0/2511042998 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5625bc7dd4e0 0x5625bc7e12e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:20.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.514+0000 7f2540e00640 1 -- 172.21.15.28:0/2511042998 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625bc7e7bb0 msgr2=0x5625bc7e1840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:16:20.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.514+0000 7f2540e00640 1 --2- 172.21.15.28:0/2511042998 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625bc7e7bb0 0x5625bc7e1840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:20.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.514+0000 7f2540e00640 1 -- 172.21.15.28:0/2511042998 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5625bc7e2070 con 0x5625bc7dcbc0 2024-05-12T22:16:20.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.514+0000 7f2541800640 1 --2- 172.21.15.28:0/2511042998 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625bc7e7bb0 0x5625bc7e1840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-12T22:16:20.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.514+0000 7f2540e00640 1 --2- 172.21.15.28:0/2511042998 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5625bc7dcbc0 0x5625bc7efd70 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f2530023040 tx=0x7f25300120f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ae8327a00f331472 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:16:20.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.514+0000 7f251ea00640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-12T22:16:20.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.515+0000 7f251ea00640 1 -- 172.21.15.28:0/2511042998 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2530004230 con 0x5625bc7dcbc0 2024-05-12T22:16:20.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.515+0000 7f251ea00640 1 -- 172.21.15.28:0/2511042998 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8513+0+0 (secure 0 0 0) 0x7f25300043d0 con 0x5625bc7dcbc0 2024-05-12T22:16:20.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.515+0000 7f2546643740 1 -- 172.21.15.28:0/2511042998 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2530024030 con 0x5625bc7dcbc0 2024-05-12T22:16:20.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.515+0000 7f251ea00640 1 -- 172.21.15.28:0/2511042998 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f2530005120 con 0x5625bc7dcbc0 2024-05-12T22:16:20.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.515+0000 7f2546643740 10 client.26866 fetch_fsmap learned FSMap version 26 2024-05-12T22:16:20.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.515+0000 7f2546643740 1 -- 172.21.15.28:0/2511042998 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5625bc85db90 con 0x5625bc7dcbc0 2024-05-12T22:16:20.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.515+0000 7f251ea00640 1 -- 172.21.15.28:0/2511042998 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2530024030 con 0x5625bc7dcbc0 2024-05-12T22:16:20.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.516+0000 7f251ea00640 1 -- 172.21.15.28:0/2511042998 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f2530004880 con 0x5625bc7dcbc0 2024-05-12T22:16:20.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.516+0000 7f2546643740 10 client.26866 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:16:20.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.516+0000 7f2546643740 10 client.26866 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:16:20.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.516+0000 7f2546643740 20 client.26866 populate_metadata read hostname 'smithi028' 2024-05-12T22:16:20.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.516+0000 7f2546643740 1 --2- 172.21.15.28:0/2511042998 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5625bc85e270 0x5625bc860690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:20.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.516+0000 7f2546643740 4 client.26866 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-12T22:16:20.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.516+0000 7f2546643740 1 -- 172.21.15.28:0/2511042998 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5625bc749610 con 0x5625bc85e270 2024-05-12T22:16:20.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.516+0000 7f253be00640 1 --2- 172.21.15.28:0/2511042998 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5625bc85e270 0x5625bc860690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:20.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.517+0000 7f253be00640 1 --2- 172.21.15.28:0/2511042998 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5625bc85e270 0x5625bc860690 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aafad6cfc87351f server_cookie=da21e0ae63c7ae2e in_seq=0 out_seq=0 2024-05-12T22:16:20.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.517+0000 7f251ea00640 10 client.26866 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:16:20.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.517+0000 7f251ea00640 1 -- 172.21.15.28:0/2511042998 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+268 (crc 0 0 0) 0x5625bc749610 con 0x5625bc85e270 2024-05-12T22:16:20.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.517+0000 7f251ea00640 10 client.26866 handle_command_reply: tid=0 2024-05-12T22:16:20.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.517+0000 7f2546643740 1 client.26866 shutdown 2024-05-12T22:16:20.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.517+0000 7f2546643740 1 -- 172.21.15.28:0/2511042998 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5625bc7dcbc0 msgr2=0x5625bc7efd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:16:20.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.517+0000 7f2546643740 1 --2- 172.21.15.28:0/2511042998 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5625bc7dcbc0 0x5625bc7efd70 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f2530023040 tx=0x7f25300120f0 comp rx=0 tx=0).stop 2024-05-12T22:16:20.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.517+0000 7f2546643740 1 -- 172.21.15.28:0/2511042998 shutdown_connections 2024-05-12T22:16:20.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.518+0000 7f2546643740 1 --2- 172.21.15.28:0/2511042998 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5625bc85e270 0x5625bc860690 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:20.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.518+0000 7f2546643740 1 --2- 172.21.15.28:0/2511042998 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625bc7e7bb0 0x5625bc7e1840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:20.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.518+0000 7f2546643740 1 --2- 172.21.15.28:0/2511042998 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5625bc7dd4e0 0x5625bc7e12e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:20.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.518+0000 7f2546643740 1 --2- 172.21.15.28:0/2511042998 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5625bc7dcbc0 0x5625bc7efd70 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:20.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.518+0000 7f251ea00640 0 client.26866 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:16:20.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.518+0000 7f2546643740 1 -- 172.21.15.28:0/2511042998 >> 172.21.15.28:0/2511042998 conn(0x5625bc7d8ee0 msgr2=0x5625bc7dfc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:16:20.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.518+0000 7f2546643740 1 -- 172.21.15.28:0/2511042998 shutdown_connections 2024-05-12T22:16:20.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.518+0000 7f2546643740 1 -- 172.21.15.28:0/2511042998 wait complete. 2024-05-12T22:16:20.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.518+0000 7f2546643740 20 client.26866 trim_cache size 0 max 16384 2024-05-12T22:16:20.531 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-05-12T22:16:20.531 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "scrub active (1465 inodes in the stack)", 2024-05-12T22:16:20.531 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-05-12T22:16:20.532 INFO:teuthology.orchestra.run.smithi028.stdout: "201fc886-efac-4914-932d-1f7d30ee7804": { 2024-05-12T22:16:20.532 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-05-12T22:16:20.532 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "201fc886-efac-4914-932d-1f7d30ee7804", 2024-05-12T22:16:20.532 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-05-12T22:16:20.532 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:16:20.532 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:16:20.533 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-05-12T22:16:20.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.519+0000 7f2543800640 1 -- 172.21.15.28:0/2584779898 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f25240a2d80 msgr2=0x7f25240a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:16:20.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.519+0000 7f2543800640 1 --2- 172.21.15.28:0/2584779898 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f25240a2d80 0x7f25240a5240 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f2534005fd0 tx=0x7f2534009500 comp rx=0 tx=0).stop 2024-05-12T22:16:20.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.519+0000 7f2543800640 1 -- 172.21.15.28:0/2584779898 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f253c104430 msgr2=0x7f253c19a900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:16:20.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.519+0000 7f2543800640 1 --2- 172.21.15.28:0/2584779898 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f253c104430 0x7f253c19a900 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f252c02a930 tx=0x7f252c003b20 comp rx=0 tx=0).stop 2024-05-12T22:16:20.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.519+0000 7f2543800640 1 -- 172.21.15.28:0/2584779898 shutdown_connections 2024-05-12T22:16:20.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.519+0000 7f2543800640 1 --2- 172.21.15.28:0/2584779898 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f25240a2d80 0x7f25240a5240 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:20.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.519+0000 7f2543800640 1 --2- 172.21.15.28:0/2584779898 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f253c109a80 0x7f253c19ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:20.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.519+0000 7f2543800640 1 --2- 172.21.15.28:0/2584779898 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f253c104430 0x7f253c19a900 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:20.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.519+0000 7f2543800640 1 --2- 172.21.15.28:0/2584779898 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f253c103a70 0x7f253c19a3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:20.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.519+0000 7f2543800640 1 -- 172.21.15.28:0/2584779898 >> 172.21.15.28:0/2584779898 conn(0x7f253c0ff5c0 msgr2=0x7f253c0ffd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:16:20.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.519+0000 7f2543800640 1 -- 172.21.15.28:0/2584779898 shutdown_connections 2024-05-12T22:16:20.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:20.519+0000 7f2543800640 1 -- 172.21.15.28:0/2584779898 wait complete. 2024-05-12T22:16:20.544 INFO:tasks.cephfs.filesystem:scrub status for tag:201fc886-efac-4914-932d-1f7d30ee7804 - {'path': '/', 'tag': '201fc886-efac-4914-932d-1f7d30ee7804', 'options': 'recursive,force'} 2024-05-12T22:16:21.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:16:21 smithi186 ceph-mon[34785]: scrub queued for path: / 2024-05-12T22:16:21.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:16:21 smithi186 ceph-mon[34785]: scrub summary: idle+waiting paths [/] 2024-05-12T22:16:21.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:16:21 smithi186 ceph-mon[34785]: scrub summary: active paths [/] 2024-05-12T22:16:21.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:16:21 smithi186 ceph-mon[34785]: scrub summary: active paths [/] 2024-05-12T22:16:21.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:16:21 smithi186 ceph-mon[34785]: scrub summary: idle+waiting paths [/] 2024-05-12T22:16:21.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:16:21 smithi186 ceph-mon[34785]: scrub summary: active paths [/] 2024-05-12T22:16:21.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:16:21 smithi186 ceph-mon[34785]: scrub summary: idle+waiting paths [/] 2024-05-12T22:16:21.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:16:21 smithi186 ceph-mon[34785]: scrub summary: active paths [/] 2024-05-12T22:16:21.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:16:21 smithi186 ceph-mon[34785]: scrub summary: idle+waiting paths [/] 2024-05-12T22:16:21.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:16:21 smithi086 ceph-mon[34297]: scrub queued for path: / 2024-05-12T22:16:21.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:16:21 smithi086 ceph-mon[34297]: scrub summary: idle+waiting paths [/] 2024-05-12T22:16:21.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:16:21 smithi086 ceph-mon[34297]: scrub summary: active paths [/] 2024-05-12T22:16:21.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:16:21 smithi086 ceph-mon[34297]: scrub summary: active paths [/] 2024-05-12T22:16:21.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:16:21 smithi086 ceph-mon[34297]: scrub summary: idle+waiting paths [/] 2024-05-12T22:16:21.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:16:21 smithi086 ceph-mon[34297]: scrub summary: active paths [/] 2024-05-12T22:16:21.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:16:21 smithi086 ceph-mon[34297]: scrub summary: idle+waiting paths [/] 2024-05-12T22:16:21.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:16:21 smithi086 ceph-mon[34297]: scrub summary: active paths [/] 2024-05-12T22:16:21.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:16:21 smithi086 ceph-mon[34297]: scrub summary: idle+waiting paths [/] 2024-05-12T22:16:21.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:16:21 smithi028 ceph-mon[29847]: scrub queued for path: / 2024-05-12T22:16:21.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:16:21 smithi028 ceph-mon[29847]: scrub summary: idle+waiting paths [/] 2024-05-12T22:16:21.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:16:21 smithi028 ceph-mon[29847]: scrub summary: active paths [/] 2024-05-12T22:16:21.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:16:21 smithi028 ceph-mon[29847]: scrub summary: active paths [/] 2024-05-12T22:16:21.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:16:21 smithi028 ceph-mon[29847]: scrub summary: idle+waiting paths [/] 2024-05-12T22:16:21.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:16:21 smithi028 ceph-mon[29847]: scrub summary: active paths [/] 2024-05-12T22:16:21.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:16:21 smithi028 ceph-mon[29847]: scrub summary: idle+waiting paths [/] 2024-05-12T22:16:21.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:16:21 smithi028 ceph-mon[29847]: scrub summary: active paths [/] 2024-05-12T22:16:21.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:16:21 smithi028 ceph-mon[29847]: scrub summary: idle+waiting paths [/] 2024-05-12T22:16:22.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:16:22 smithi186 ceph-mon[34785]: scrub summary: idle+waiting paths [/] 2024-05-12T22:16:22.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:16:22 smithi186 ceph-mon[34785]: pgmap v226: 97 pgs: 97 active+clean; 97 MiB data, 757 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 102 B/s wr, 5 op/s 2024-05-12T22:16:22.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:16:22 smithi086 ceph-mon[34297]: scrub summary: idle+waiting paths [/] 2024-05-12T22:16:22.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:16:22 smithi086 ceph-mon[34297]: pgmap v226: 97 pgs: 97 active+clean; 97 MiB data, 757 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 102 B/s wr, 5 op/s 2024-05-12T22:16:22.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:16:22 smithi028 ceph-mon[29847]: scrub summary: idle+waiting paths [/] 2024-05-12T22:16:22.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:16:22 smithi028 ceph-mon[29847]: pgmap v226: 97 pgs: 97 active+clean; 97 MiB data, 757 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 102 B/s wr, 5 op/s 2024-05-12T22:16:24.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:16:24 smithi186 ceph-mon[34785]: pgmap v227: 97 pgs: 97 active+clean; 97 MiB data, 757 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 102 B/s wr, 5 op/s 2024-05-12T22:16:24.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:16:24 smithi086 ceph-mon[34297]: pgmap v227: 97 pgs: 97 active+clean; 97 MiB data, 757 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 102 B/s wr, 5 op/s 2024-05-12T22:16:24.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:16:24 smithi028 ceph-mon[29847]: pgmap v227: 97 pgs: 97 active+clean; 97 MiB data, 757 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 102 B/s wr, 5 op/s 2024-05-12T22:16:26.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:16:26 smithi086 ceph-mon[34297]: pgmap v228: 97 pgs: 97 active+clean; 97 MiB data, 757 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 93 B/s wr, 8 op/s 2024-05-12T22:16:26.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:16:26 smithi028 ceph-mon[29847]: pgmap v228: 97 pgs: 97 active+clean; 97 MiB data, 757 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 93 B/s wr, 8 op/s 2024-05-12T22:16:26.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:16:26 smithi186 ceph-mon[34785]: pgmap v228: 97 pgs: 97 active+clean; 97 MiB data, 757 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 93 B/s wr, 8 op/s 2024-05-12T22:16:28.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:16:28 smithi086 ceph-mon[34297]: pgmap v229: 97 pgs: 97 active+clean; 97 MiB data, 757 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-05-12T22:16:28.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:16:28 smithi086 ceph-mon[34297]: scrub summary: idle 2024-05-12T22:16:28.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:16:28 smithi086 ceph-mon[34297]: scrub summary: idle 2024-05-12T22:16:28.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:16:28 smithi028 ceph-mon[29847]: pgmap v229: 97 pgs: 97 active+clean; 97 MiB data, 757 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-05-12T22:16:28.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:16:28 smithi028 ceph-mon[29847]: scrub summary: idle 2024-05-12T22:16:28.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:16:28 smithi028 ceph-mon[29847]: scrub summary: idle 2024-05-12T22:16:28.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:16:28 smithi186 ceph-mon[34785]: pgmap v229: 97 pgs: 97 active+clean; 97 MiB data, 757 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-05-12T22:16:28.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:16:28 smithi186 ceph-mon[34785]: scrub summary: idle 2024-05-12T22:16:28.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:16:28 smithi186 ceph-mon[34785]: scrub summary: idle 2024-05-12T22:16:30.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:16:30 smithi086 ceph-mon[34297]: pgmap v230: 97 pgs: 97 active+clean; 97 MiB data, 757 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-05-12T22:16:30.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:16:30 smithi028 ceph-mon[29847]: pgmap v230: 97 pgs: 97 active+clean; 97 MiB data, 757 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-05-12T22:16:30.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:16:30 smithi186 ceph-mon[34785]: pgmap v230: 97 pgs: 97 active+clean; 97 MiB data, 757 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-05-12T22:16:32.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:16:32 smithi086 ceph-mon[34297]: pgmap v231: 97 pgs: 97 active+clean; 97 MiB data, 757 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-05-12T22:16:32.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:16:32 smithi028 ceph-mon[29847]: pgmap v231: 97 pgs: 97 active+clean; 97 MiB data, 757 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-05-12T22:16:32.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:16:32 smithi186 ceph-mon[34785]: pgmap v231: 97 pgs: 97 active+clean; 97 MiB data, 757 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-05-12T22:16:34.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:16:34 smithi086 ceph-mon[34297]: pgmap v232: 97 pgs: 97 active+clean; 97 MiB data, 757 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-05-12T22:16:34.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:16:34 smithi028 ceph-mon[29847]: pgmap v232: 97 pgs: 97 active+clean; 97 MiB data, 757 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-05-12T22:16:34.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:16:34 smithi186 ceph-mon[34785]: pgmap v232: 97 pgs: 97 active+clean; 97 MiB data, 757 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-05-12T22:16:36.481 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:16:36 smithi186 ceph-mon[34785]: pgmap v233: 97 pgs: 97 active+clean; 97 MiB data, 757 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-05-12T22:16:36.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:16:36 smithi086 ceph-mon[34297]: pgmap v233: 97 pgs: 97 active+clean; 97 MiB data, 757 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-05-12T22:16:36.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:16:36 smithi028 ceph-mon[29847]: pgmap v233: 97 pgs: 97 active+clean; 97 MiB data, 757 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-05-12T22:16:38.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:16:38 smithi086 ceph-mon[34297]: pgmap v234: 97 pgs: 97 active+clean; 97 MiB data, 757 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:16:38.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:16:38 smithi028 ceph-mon[29847]: pgmap v234: 97 pgs: 97 active+clean; 97 MiB data, 757 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:16:38.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:16:38 smithi186 ceph-mon[34785]: pgmap v234: 97 pgs: 97 active+clean; 97 MiB data, 757 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:16:40.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:16:40 smithi086 ceph-mon[34297]: pgmap v235: 97 pgs: 97 active+clean; 97 MiB data, 757 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:16:40.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:16:40 smithi028 ceph-mon[29847]: pgmap v235: 97 pgs: 97 active+clean; 97 MiB data, 757 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:16:40.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:16:40 smithi186 ceph-mon[34785]: pgmap v235: 97 pgs: 97 active+clean; 97 MiB data, 757 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:16:42.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:16:42 smithi086 ceph-mon[34297]: pgmap v236: 97 pgs: 97 active+clean; 97 MiB data, 757 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-12T22:16:42.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:16:42 smithi028 ceph-mon[29847]: pgmap v236: 97 pgs: 97 active+clean; 97 MiB data, 757 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-12T22:16:42.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:16:42 smithi186 ceph-mon[34785]: pgmap v236: 97 pgs: 97 active+clean; 97 MiB data, 757 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-12T22:16:44.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:16:44 smithi086 ceph-mon[34297]: pgmap v237: 97 pgs: 97 active+clean; 97 MiB data, 757 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:16:44.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:16:44 smithi028 ceph-mon[29847]: pgmap v237: 97 pgs: 97 active+clean; 97 MiB data, 757 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:16:44.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:16:44 smithi186 ceph-mon[34785]: pgmap v237: 97 pgs: 97 active+clean; 97 MiB data, 757 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:16:46.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:16:46 smithi086 ceph-mon[34297]: pgmap v238: 97 pgs: 97 active+clean; 97 MiB data, 757 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-12T22:16:46.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:16:46 smithi028 ceph-mon[29847]: pgmap v238: 97 pgs: 97 active+clean; 97 MiB data, 757 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-12T22:16:46.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:16:46 smithi186 ceph-mon[34785]: pgmap v238: 97 pgs: 97 active+clean; 97 MiB data, 757 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-12T22:16:48.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:16:48 smithi086 ceph-mon[34297]: pgmap v239: 97 pgs: 97 active+clean; 97 MiB data, 757 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:16:48.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:16:48 smithi028 ceph-mon[29847]: pgmap v239: 97 pgs: 97 active+clean; 97 MiB data, 757 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:16:48.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:16:48 smithi186 ceph-mon[34785]: pgmap v239: 97 pgs: 97 active+clean; 97 MiB data, 757 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:16:50.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:16:50 smithi086 ceph-mon[34297]: pgmap v240: 97 pgs: 97 active+clean; 97 MiB data, 757 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:16:50.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:16:50 smithi028 ceph-mon[29847]: pgmap v240: 97 pgs: 97 active+clean; 97 MiB data, 757 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:16:50.545 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-12T22:16:50.545 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-12T22:16:50.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:16:50 smithi186 ceph-mon[34785]: pgmap v240: 97 pgs: 97 active+clean; 97 MiB data, 757 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:16:50.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.771+0000 7f1371800640 1 -- 172.21.15.28:0/3502057836 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f136c1126b0 msgr2=0x7f136c114aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:16:50.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.771+0000 7f1371800640 1 --2- 172.21.15.28:0/3502057836 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f136c1126b0 0x7f136c114aa0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f1364009c10 tx=0x7f136402a6e0 comp rx=0 tx=0).stop 2024-05-12T22:16:50.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.771+0000 7f1371800640 1 -- 172.21.15.28:0/3502057836 shutdown_connections 2024-05-12T22:16:50.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.771+0000 7f1371800640 1 --2- 172.21.15.28:0/3502057836 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f136c1126b0 0x7f136c114aa0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:50.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.771+0000 7f1371800640 1 --2- 172.21.15.28:0/3502057836 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f136c076680 0x7f136c112170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:50.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.771+0000 7f1371800640 1 --2- 172.21.15.28:0/3502057836 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f136c075d60 0x7f136c076140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:50.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.771+0000 7f1371800640 1 -- 172.21.15.28:0/3502057836 >> 172.21.15.28:0/3502057836 conn(0x7f136c0ff4a0 msgr2=0x7f136c1018c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:16:50.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.771+0000 7f1371800640 1 -- 172.21.15.28:0/3502057836 shutdown_connections 2024-05-12T22:16:50.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.771+0000 7f1371800640 1 -- 172.21.15.28:0/3502057836 wait complete. 2024-05-12T22:16:50.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.772+0000 7f1371800640 1 Processor -- start 2024-05-12T22:16:50.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.772+0000 7f1371800640 1 -- start start 2024-05-12T22:16:50.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.773+0000 7f1371800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f136c075d60 0x7f136c19e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:50.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.773+0000 7f1371800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f136c076680 0x7f136c19eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:50.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.773+0000 7f1371800640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f136c1126b0 0x7f136c1a3020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:50.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.773+0000 7f1371800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f136c1181a0 con 0x7f136c075d60 2024-05-12T22:16:50.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.773+0000 7f1371800640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f136c118020 con 0x7f136c076680 2024-05-12T22:16:50.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.773+0000 7f1371800640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f136c118320 con 0x7f136c1126b0 2024-05-12T22:16:50.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.773+0000 7f136aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f136c075d60 0x7f136c19e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:50.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.773+0000 7f136aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f136c075d60 0x7f136c19e760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35378/0 (socket says 172.21.15.28:35378) 2024-05-12T22:16:50.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.773+0000 7f136aa00640 1 -- 172.21.15.28:0/4209857396 learned_addr learned my addr 172.21.15.28:0/4209857396 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:16:50.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.773+0000 7f1363e00640 1 --2- 172.21.15.28:0/4209857396 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f136c076680 0x7f136c19eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:50.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.773+0000 7f136b400640 1 --2- 172.21.15.28:0/4209857396 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f136c1126b0 0x7f136c1a3020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:50.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.774+0000 7f136aa00640 1 -- 172.21.15.28:0/4209857396 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f136c1126b0 msgr2=0x7f136c1a3020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:16:50.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.774+0000 7f136aa00640 1 --2- 172.21.15.28:0/4209857396 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f136c1126b0 0x7f136c1a3020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:50.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.774+0000 7f136aa00640 1 -- 172.21.15.28:0/4209857396 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f136c076680 msgr2=0x7f136c19eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:16:50.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.774+0000 7f136aa00640 1 --2- 172.21.15.28:0/4209857396 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f136c076680 0x7f136c19eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:50.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.774+0000 7f136aa00640 1 -- 172.21.15.28:0/4209857396 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f136c1a3750 con 0x7f136c075d60 2024-05-12T22:16:50.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.774+0000 7f1363e00640 1 --2- 172.21.15.28:0/4209857396 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f136c076680 0x7f136c19eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:16:50.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.774+0000 7f136aa00640 1 --2- 172.21.15.28:0/4209857396 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f136c075d60 0x7f136c19e760 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f135c00e9c0 tx=0x7f135c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:16:50.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.774+0000 7f1363400640 1 -- 172.21.15.28:0/4209857396 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f135c00cde0 con 0x7f136c075d60 2024-05-12T22:16:50.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.774+0000 7f1371800640 1 -- 172.21.15.28:0/4209857396 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f136c1a3a40 con 0x7f136c075d60 2024-05-12T22:16:50.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.774+0000 7f1363400640 1 -- 172.21.15.28:0/4209857396 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f135c0035f0 con 0x7f136c075d60 2024-05-12T22:16:50.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.774+0000 7f1363400640 1 -- 172.21.15.28:0/4209857396 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f135c010650 con 0x7f136c075d60 2024-05-12T22:16:50.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.775+0000 7f1371800640 1 -- 172.21.15.28:0/4209857396 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f136c1ab2d0 con 0x7f136c075d60 2024-05-12T22:16:50.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.779+0000 7f1374623740 1 Processor -- start 2024-05-12T22:16:50.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.779+0000 7f1374623740 1 -- start start 2024-05-12T22:16:50.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.780+0000 7f1374623740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558a1c7ecbc0 0x558a1c7ecfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:50.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.780+0000 7f1363400640 1 -- 172.21.15.28:0/4209857396 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7f135c0037a0 con 0x7f136c075d60 2024-05-12T22:16:50.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.780+0000 7f1374623740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558a1c7ed4e0 0x558a1c7f7670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:50.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.780+0000 7f1374623740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558a1c7f7bb0 0x558a1c7f9fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:50.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.780+0000 7f1374623740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558a1c7b52a0 con 0x558a1c7ed4e0 2024-05-12T22:16:50.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.780+0000 7f1374623740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x558a1c751b90 con 0x558a1c7f7bb0 2024-05-12T22:16:50.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.780+0000 7f1374623740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x558a1c6f95b0 con 0x558a1c7ecbc0 2024-05-12T22:16:50.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.780+0000 7f136b400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558a1c7ecbc0 0x558a1c7ecfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:50.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.780+0000 7f136aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558a1c7ed4e0 0x558a1c7f7670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:50.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.780+0000 7f136b400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558a1c7ecbc0 0x558a1c7ecfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:59182/0 (socket says 172.21.15.28:59182) 2024-05-12T22:16:50.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.780+0000 7f136b400640 1 -- 172.21.15.28:0/3164372715 learned_addr learned my addr 172.21.15.28:0/3164372715 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:16:50.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.780+0000 7f1363e00640 1 --2- 172.21.15.28:0/3164372715 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558a1c7f7bb0 0x558a1c7f9fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:50.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.781+0000 7f136b400640 1 -- 172.21.15.28:0/3164372715 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558a1c7f7bb0 msgr2=0x558a1c7f9fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:16:50.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.781+0000 7f136b400640 1 --2- 172.21.15.28:0/3164372715 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558a1c7f7bb0 0x558a1c7f9fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:50.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.781+0000 7f136b400640 1 -- 172.21.15.28:0/3164372715 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558a1c7ed4e0 msgr2=0x558a1c7f7670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:16:50.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.781+0000 7f136b400640 1 --2- 172.21.15.28:0/3164372715 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558a1c7ed4e0 0x558a1c7f7670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:50.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.781+0000 7f136b400640 1 -- 172.21.15.28:0/3164372715 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558a1c52f7d0 con 0x558a1c7ecbc0 2024-05-12T22:16:50.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.781+0000 7f136aa00640 1 --2- 172.21.15.28:0/3164372715 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558a1c7ed4e0 0x558a1c7f7670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:16:50.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.781+0000 7f1363e00640 1 --2- 172.21.15.28:0/3164372715 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558a1c7f7bb0 0x558a1c7f9fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:16:50.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.781+0000 7f136b400640 1 --2- 172.21.15.28:0/3164372715 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558a1c7ecbc0 0x558a1c7ecfa0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f1364004af0 tx=0x7f1364005c60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c5696c9e03142b3 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:16:50.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.781+0000 7f1363400640 1 --2- 172.21.15.28:0/4209857396 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f13500a2d00 0x7f13500a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:50.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.781+0000 7f135be00640 1 -- 172.21.15.28:0/3164372715 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1364042030 con 0x558a1c7ecbc0 2024-05-12T22:16:50.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.781+0000 7f135be00640 1 -- 172.21.15.28:0/3164372715 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1364037040 con 0x558a1c7ecbc0 2024-05-12T22:16:50.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.781+0000 7f135be00640 1 -- 172.21.15.28:0/3164372715 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1364036070 con 0x558a1c7ecbc0 2024-05-12T22:16:50.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.782+0000 7f1363400640 1 -- 172.21.15.28:0/4209857396 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8513+0+0 (secure 0 0 0) 0x7f135c0cf770 con 0x7f136c075d60 2024-05-12T22:16:50.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.782+0000 7f136aa00640 1 --2- 172.21.15.28:0/4209857396 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f13500a2d00 0x7f13500a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:50.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.782+0000 7f1374623740 1 -- 172.21.15.28:0/3164372715 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558a1c7ecbc0 msgr2=0x558a1c7ecfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:16:50.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.782+0000 7f1374623740 1 --2- 172.21.15.28:0/3164372715 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558a1c7ecbc0 0x558a1c7ecfa0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f1364004af0 tx=0x7f1364005c60 comp rx=0 tx=0).stop 2024-05-12T22:16:50.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.782+0000 7f136aa00640 1 --2- 172.21.15.28:0/4209857396 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f13500a2d00 0x7f13500a51c0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f135c0ccf70 tx=0x7f135c0d5550 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:16:50.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.782+0000 7f1374623740 1 -- 172.21.15.28:0/3164372715 shutdown_connections 2024-05-12T22:16:50.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.782+0000 7f1374623740 1 --2- 172.21.15.28:0/3164372715 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558a1c7f7bb0 0x558a1c7f9fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:50.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.782+0000 7f1374623740 1 --2- 172.21.15.28:0/3164372715 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558a1c7ed4e0 0x558a1c7f7670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:50.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.782+0000 7f1374623740 1 --2- 172.21.15.28:0/3164372715 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558a1c7ecbc0 0x558a1c7ecfa0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:50.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.782+0000 7f1374623740 1 -- 172.21.15.28:0/3164372715 >> 172.21.15.28:0/3164372715 conn(0x558a1c7e8ee0 msgr2=0x558a1c7e92c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:16:50.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.783+0000 7f1374623740 1 -- 172.21.15.28:0/3164372715 shutdown_connections 2024-05-12T22:16:50.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.783+0000 7f1374623740 1 -- 172.21.15.28:0/3164372715 wait complete. 2024-05-12T22:16:50.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.783+0000 7f1374623740 1 -- start start 2024-05-12T22:16:50.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.783+0000 7f1374623740 1 Processor -- start 2024-05-12T22:16:50.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.784+0000 7f1374623740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558a1c7ecbc0 0x558a1c7f7540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:50.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.784+0000 7f1374623740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558a1c7ed4e0 0x558a1c7f4830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:50.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.784+0000 7f1374623740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558a1c7f7bb0 0x558a1c7f4db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:50.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.784+0000 7f1374623740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558a1c70c9f0 con 0x558a1c7f7bb0 2024-05-12T22:16:50.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.784+0000 7f1374623740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x558a1c74db50 con 0x558a1c7ed4e0 2024-05-12T22:16:50.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.784+0000 7f1374623740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x558a1c7fa940 con 0x558a1c7ecbc0 2024-05-12T22:16:50.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.784+0000 7f1374623740 10 client.0 _ll_register_callbacks cb 0x558a1c464be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:16:50.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.784+0000 7f136b400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558a1c7ecbc0 0x558a1c7f7540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:50.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.784+0000 7f1363e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558a1c7ed4e0 0x558a1c7f4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:50.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.784+0000 7f136b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558a1c7f7bb0 0x558a1c7f4db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:50.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.784+0000 7f1363e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558a1c7ed4e0 0x558a1c7f4830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:49114/0 (socket says 172.21.15.28:49114) 2024-05-12T22:16:50.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.784+0000 7f1363e00640 1 -- 172.21.15.28:0/3282787868 learned_addr learned my addr 172.21.15.28:0/3282787868 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:16:50.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.784+0000 7f1363e00640 1 -- 172.21.15.28:0/3282787868 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558a1c7ecbc0 msgr2=0x558a1c7f7540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:16:50.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.784+0000 7f1363e00640 1 --2- 172.21.15.28:0/3282787868 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558a1c7ecbc0 0x558a1c7f7540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:50.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.784+0000 7f1363e00640 1 -- 172.21.15.28:0/3282787868 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558a1c7f7bb0 msgr2=0x558a1c7f4db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:16:50.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.784+0000 7f1363e00640 1 --2- 172.21.15.28:0/3282787868 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558a1c7f7bb0 0x558a1c7f4db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:50.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.784+0000 7f1363e00640 1 -- 172.21.15.28:0/3282787868 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558a1c4653f0 con 0x558a1c7ed4e0 2024-05-12T22:16:50.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.784+0000 7f136b400640 1 --2- 172.21.15.28:0/3282787868 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558a1c7f7bb0 0x558a1c7f4db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:16:50.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.785+0000 7f1363e00640 1 --2- 172.21.15.28:0/3282787868 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558a1c7ed4e0 0x558a1c7f4830 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f1354012d20 tx=0x7f135400f790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1806b81a6229b20 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:16:50.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.785+0000 7f1358c00640 10 client.0 ms_handle_connect on v2:172.21.15.86:3300/0 2024-05-12T22:16:50.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.785+0000 7f1358c00640 1 -- 172.21.15.28:0/3282787868 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13540120d0 con 0x558a1c7ed4e0 2024-05-12T22:16:50.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.785+0000 7f1358c00640 1 -- 172.21.15.28:0/3282787868 <== mon.2 v2:172.21.15.86:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8513+0+0 (secure 0 0 0) 0x7f1354010430 con 0x558a1c7ed4e0 2024-05-12T22:16:50.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.785+0000 7f1374623740 1 -- 172.21.15.28:0/3282787868 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1364037040 con 0x558a1c7ed4e0 2024-05-12T22:16:50.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.785+0000 7f1358c00640 1 -- 172.21.15.28:0/3282787868 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f1354012270 con 0x558a1c7ed4e0 2024-05-12T22:16:50.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.785+0000 7f1374623740 10 client.26921 fetch_fsmap learned FSMap version 26 2024-05-12T22:16:50.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.785+0000 7f1374623740 1 -- 172.21.15.28:0/3282787868 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x558a1c4250f0 con 0x558a1c7ed4e0 2024-05-12T22:16:50.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.786+0000 7f1358c00640 1 -- 172.21.15.28:0/3282787868 <== mon.2 v2:172.21.15.86:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1364037040 con 0x558a1c7ed4e0 2024-05-12T22:16:50.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.786+0000 7f1358c00640 1 -- 172.21.15.28:0/3282787868 <== mon.2 v2:172.21.15.86:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f135401c750 con 0x558a1c7ed4e0 2024-05-12T22:16:50.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.786+0000 7f1374623740 10 client.26921 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:16:50.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.786+0000 7f1374623740 10 client.26921 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:16:50.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.786+0000 7f1374623740 20 client.26921 populate_metadata read hostname 'smithi028' 2024-05-12T22:16:50.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.786+0000 7f1374623740 1 --2- 172.21.15.28:0/3282787868 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x558a1c8077b0 0x558a1c809bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:50.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.786+0000 7f1374623740 4 client.26921 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-12T22:16:50.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.786+0000 7f1374623740 1 -- 172.21.15.28:0/3282787868 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558a1c75f2c0 con 0x558a1c8077b0 2024-05-12T22:16:50.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.786+0000 7f136b400640 1 --2- 172.21.15.28:0/3282787868 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x558a1c8077b0 0x558a1c809bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:50.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.787+0000 7f136b400640 1 --2- 172.21.15.28:0/3282787868 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x558a1c8077b0 0x558a1c809bd0 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=799417896cdbef33 server_cookie=fed2fda620cd5d27 in_seq=0 out_seq=0 2024-05-12T22:16:50.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.787+0000 7f1358c00640 10 client.26921 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:16:50.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.787+0000 7f1358c00640 1 -- 172.21.15.28:0/3282787868 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558a1c75f2c0 con 0x558a1c8077b0 2024-05-12T22:16:50.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.787+0000 7f1358c00640 10 client.26921 handle_command_reply: tid=0 2024-05-12T22:16:50.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.788+0000 7f1374623740 1 client.26921 shutdown 2024-05-12T22:16:50.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.788+0000 7f1374623740 1 -- 172.21.15.28:0/3282787868 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558a1c7ed4e0 msgr2=0x558a1c7f4830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:16:50.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.788+0000 7f1374623740 1 --2- 172.21.15.28:0/3282787868 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558a1c7ed4e0 0x558a1c7f4830 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f1354012d20 tx=0x7f135400f790 comp rx=0 tx=0).stop 2024-05-12T22:16:50.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.788+0000 7f1374623740 1 -- 172.21.15.28:0/3282787868 shutdown_connections 2024-05-12T22:16:50.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.788+0000 7f1374623740 1 --2- 172.21.15.28:0/3282787868 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x558a1c8077b0 0x558a1c809bd0 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:50.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.788+0000 7f1374623740 1 --2- 172.21.15.28:0/3282787868 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558a1c7f7bb0 0x558a1c7f4db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:50.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.788+0000 7f1374623740 1 --2- 172.21.15.28:0/3282787868 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558a1c7ed4e0 0x558a1c7f4830 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:50.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.788+0000 7f1374623740 1 --2- 172.21.15.28:0/3282787868 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558a1c7ecbc0 0x558a1c7f7540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:50.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.788+0000 7f1374623740 1 -- 172.21.15.28:0/3282787868 >> 172.21.15.28:0/3282787868 conn(0x558a1c7e8ee0 msgr2=0x558a1c7eece0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:16:50.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.788+0000 7f1358c00640 0 client.26921 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:16:50.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.788+0000 7f1374623740 1 -- 172.21.15.28:0/3282787868 shutdown_connections 2024-05-12T22:16:50.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.788+0000 7f1374623740 1 -- 172.21.15.28:0/3282787868 wait complete. 2024-05-12T22:16:50.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.788+0000 7f1374623740 20 client.26921 trim_cache size 0 max 16384 2024-05-12T22:16:50.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.807+0000 7f1374623740 1 Processor -- start 2024-05-12T22:16:50.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.807+0000 7f1374623740 1 -- start start 2024-05-12T22:16:50.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.808+0000 7f1374623740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558a1c7ecbc0 0x558a1c7efea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:50.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.808+0000 7f1374623740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558a1c7ed4e0 0x558a1c7e7f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:50.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.808+0000 7f1374623740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558a1c7f7bb0 0x558a1c7f7070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:50.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.808+0000 7f1374623740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f136c1b6440 con 0x558a1c7f7bb0 2024-05-12T22:16:50.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.808+0000 7f1374623740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f136c1b3d10 con 0x558a1c7ecbc0 2024-05-12T22:16:50.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.808+0000 7f1374623740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f136c1b1b80 con 0x558a1c7ed4e0 2024-05-12T22:16:50.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.808+0000 7f136b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558a1c7f7bb0 0x558a1c7f7070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:50.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.808+0000 7f136b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558a1c7f7bb0 0x558a1c7f7070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35410/0 (socket says 172.21.15.28:35410) 2024-05-12T22:16:50.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.808+0000 7f136b400640 1 -- 172.21.15.28:0/1354680666 learned_addr learned my addr 172.21.15.28:0/1354680666 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:16:50.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.808+0000 7f136b400640 1 --2- 172.21.15.28:0/1354680666 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558a1c7ecbc0 0x558a1c7efea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:50.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.808+0000 7f1363e00640 1 --2- 172.21.15.28:0/1354680666 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558a1c7ed4e0 0x558a1c7e7f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:50.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.809+0000 7f136b400640 1 -- 172.21.15.28:0/1354680666 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558a1c7ed4e0 msgr2=0x558a1c7e7f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:16:50.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.809+0000 7f136b400640 1 --2- 172.21.15.28:0/1354680666 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558a1c7ed4e0 0x558a1c7e7f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:50.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.809+0000 7f136b400640 1 -- 172.21.15.28:0/1354680666 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558a1c7ecbc0 msgr2=0x558a1c7efea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:16:50.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.809+0000 7f136b400640 1 --2- 172.21.15.28:0/1354680666 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558a1c7ecbc0 0x558a1c7efea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:50.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.809+0000 7f136b400640 1 -- 172.21.15.28:0/1354680666 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558a1c80a230 con 0x558a1c7f7bb0 2024-05-12T22:16:50.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.809+0000 7f136b400640 1 --2- 172.21.15.28:0/1354680666 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558a1c7f7bb0 0x558a1c7f7070 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f1364003c10 tx=0x7f1364040040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=368f597e203c13af server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:16:50.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.809+0000 7f135b400640 1 -- 172.21.15.28:0/1354680666 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1364036040 con 0x558a1c7f7bb0 2024-05-12T22:16:50.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.809+0000 7f135b400640 1 -- 172.21.15.28:0/1354680666 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1364030ce0 con 0x558a1c7f7bb0 2024-05-12T22:16:50.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.809+0000 7f135b400640 1 -- 172.21.15.28:0/1354680666 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1364042040 con 0x558a1c7f7bb0 2024-05-12T22:16:50.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.810+0000 7f1374623740 1 -- 172.21.15.28:0/1354680666 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558a1c7f7bb0 msgr2=0x558a1c7f7070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:16:50.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.810+0000 7f1374623740 1 --2- 172.21.15.28:0/1354680666 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558a1c7f7bb0 0x558a1c7f7070 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f1364003c10 tx=0x7f1364040040 comp rx=0 tx=0).stop 2024-05-12T22:16:50.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.810+0000 7f1374623740 1 -- 172.21.15.28:0/1354680666 shutdown_connections 2024-05-12T22:16:50.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.810+0000 7f1374623740 1 --2- 172.21.15.28:0/1354680666 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558a1c7f7bb0 0x558a1c7f7070 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:50.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.810+0000 7f1374623740 1 --2- 172.21.15.28:0/1354680666 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558a1c7ed4e0 0x558a1c7e7f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:50.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.810+0000 7f1374623740 1 --2- 172.21.15.28:0/1354680666 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558a1c7ecbc0 0x558a1c7efea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:50.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.810+0000 7f1374623740 1 -- 172.21.15.28:0/1354680666 >> 172.21.15.28:0/1354680666 conn(0x558a1c7e8ee0 msgr2=0x558a1c7e7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:16:50.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.810+0000 7f1374623740 1 -- 172.21.15.28:0/1354680666 shutdown_connections 2024-05-12T22:16:50.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.810+0000 7f1374623740 1 -- 172.21.15.28:0/1354680666 wait complete. 2024-05-12T22:16:50.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.810+0000 7f1374623740 1 -- start start 2024-05-12T22:16:50.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.811+0000 7f1374623740 1 Processor -- start 2024-05-12T22:16:50.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.811+0000 7f1374623740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558a1c7ecbc0 0x558a1c7ffb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:50.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.811+0000 7f1374623740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558a1c7ed4e0 0x558a1c8000d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:50.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.811+0000 7f1374623740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558a1c7f7bb0 0x558a1c7fcb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:50.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.811+0000 7f1374623740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f136c1b21d0 con 0x558a1c7ecbc0 2024-05-12T22:16:50.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.811+0000 7f1374623740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f136c1b3640 con 0x558a1c7ed4e0 2024-05-12T22:16:50.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.811+0000 7f1374623740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f136c1b28a0 con 0x558a1c7f7bb0 2024-05-12T22:16:50.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.811+0000 7f1374623740 10 client.0 _ll_register_callbacks cb 0x7f1354009cc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:16:50.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.811+0000 7f136b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558a1c7ecbc0 0x558a1c7ffb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:50.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.811+0000 7f136b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558a1c7ecbc0 0x558a1c7ffb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35426/0 (socket says 172.21.15.28:35426) 2024-05-12T22:16:50.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.811+0000 7f136b400640 1 -- 172.21.15.28:0/863708913 learned_addr learned my addr 172.21.15.28:0/863708913 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:16:50.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.811+0000 7f1363e00640 1 --2- 172.21.15.28:0/863708913 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558a1c7ed4e0 0x558a1c8000d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:50.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.812+0000 7f136b400640 1 --2- 172.21.15.28:0/863708913 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558a1c7f7bb0 0x558a1c7fcb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:50.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.812+0000 7f1363e00640 1 -- 172.21.15.28:0/863708913 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558a1c7f7bb0 msgr2=0x558a1c7fcb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:16:50.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.812+0000 7f1363e00640 1 --2- 172.21.15.28:0/863708913 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558a1c7f7bb0 0x558a1c7fcb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:50.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.812+0000 7f1363e00640 1 -- 172.21.15.28:0/863708913 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558a1c7ecbc0 msgr2=0x558a1c7ffb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:16:50.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.812+0000 7f1363e00640 1 --2- 172.21.15.28:0/863708913 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558a1c7ecbc0 0x558a1c7ffb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:50.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.812+0000 7f1363e00640 1 -- 172.21.15.28:0/863708913 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558a1c7f9fa0 con 0x558a1c7ed4e0 2024-05-12T22:16:50.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.812+0000 7f136b400640 1 --2- 172.21.15.28:0/863708913 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558a1c7ecbc0 0x558a1c7ffb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:16:50.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.812+0000 7f1363e00640 1 --2- 172.21.15.28:0/863708913 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558a1c7ed4e0 0x558a1c8000d0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f1354000c00 tx=0x7f1354005270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=71c42397c0eacee1 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:16:50.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.812+0000 7f1358c00640 10 client.0 ms_handle_connect on v2:172.21.15.86:3300/0 2024-05-12T22:16:50.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.812+0000 7f1358c00640 1 -- 172.21.15.28:0/863708913 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13540120d0 con 0x558a1c7ed4e0 2024-05-12T22:16:50.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.812+0000 7f1374623740 1 -- 172.21.15.28:0/863708913 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1364030ce0 con 0x558a1c7ed4e0 2024-05-12T22:16:50.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.812+0000 7f1358c00640 1 -- 172.21.15.28:0/863708913 <== mon.2 v2:172.21.15.86:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8513+0+0 (secure 0 0 0) 0x7f13540055f0 con 0x558a1c7ed4e0 2024-05-12T22:16:50.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.813+0000 7f1358c00640 1 -- 172.21.15.28:0/863708913 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f1354034050 con 0x558a1c7ed4e0 2024-05-12T22:16:50.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.813+0000 7f1374623740 10 client.26927 fetch_fsmap learned FSMap version 26 2024-05-12T22:16:50.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.813+0000 7f1374623740 1 -- 172.21.15.28:0/863708913 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x558a1c7f5390 con 0x558a1c7ed4e0 2024-05-12T22:16:50.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.813+0000 7f1358c00640 1 -- 172.21.15.28:0/863708913 <== mon.2 v2:172.21.15.86:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1364030ce0 con 0x558a1c7ed4e0 2024-05-12T22:16:50.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.813+0000 7f1358c00640 1 -- 172.21.15.28:0/863708913 <== mon.2 v2:172.21.15.86:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f1354010430 con 0x558a1c7ed4e0 2024-05-12T22:16:50.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.813+0000 7f1374623740 10 client.26927 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:16:50.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.813+0000 7f1374623740 10 client.26927 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:16:50.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.813+0000 7f1374623740 20 client.26927 populate_metadata read hostname 'smithi028' 2024-05-12T22:16:50.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.813+0000 7f1374623740 1 --2- 172.21.15.28:0/863708913 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x558a1c86e9f0 0x558a1c870e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:50.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.813+0000 7f1374623740 4 client.26927 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-12T22:16:50.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.813+0000 7f1374623740 1 -- 172.21.15.28:0/863708913 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "scrub status"}) -- 0x558a1c759610 con 0x558a1c86e9f0 2024-05-12T22:16:50.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.814+0000 7f136b400640 1 --2- 172.21.15.28:0/863708913 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x558a1c86e9f0 0x558a1c870e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:50.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.814+0000 7f136b400640 1 --2- 172.21.15.28:0/863708913 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x558a1c86e9f0 0x558a1c870e10 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fb2058bda4c6b4a7 server_cookie=99c87ecc0cd97441 in_seq=0 out_seq=0 2024-05-12T22:16:50.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.814+0000 7f1358c00640 10 client.26927 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:16:50.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.814+0000 7f1358c00640 1 -- 172.21.15.28:0/863708913 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x558a1c759610 con 0x558a1c86e9f0 2024-05-12T22:16:50.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.814+0000 7f1358c00640 10 client.26927 handle_command_reply: tid=0 2024-05-12T22:16:50.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.814+0000 7f1374623740 1 client.26927 shutdown 2024-05-12T22:16:50.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.815+0000 7f1374623740 1 -- 172.21.15.28:0/863708913 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558a1c7ed4e0 msgr2=0x558a1c8000d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:16:50.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.815+0000 7f1374623740 1 --2- 172.21.15.28:0/863708913 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558a1c7ed4e0 0x558a1c8000d0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f1354000c00 tx=0x7f1354005270 comp rx=0 tx=0).stop 2024-05-12T22:16:50.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.815+0000 7f1374623740 1 -- 172.21.15.28:0/863708913 shutdown_connections 2024-05-12T22:16:50.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.815+0000 7f1374623740 1 --2- 172.21.15.28:0/863708913 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x558a1c86e9f0 0x558a1c870e10 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:50.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.815+0000 7f1374623740 1 --2- 172.21.15.28:0/863708913 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558a1c7f7bb0 0x558a1c7fcb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:50.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.815+0000 7f1374623740 1 --2- 172.21.15.28:0/863708913 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558a1c7ed4e0 0x558a1c8000d0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:50.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.815+0000 7f1374623740 1 --2- 172.21.15.28:0/863708913 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558a1c7ecbc0 0x558a1c7ffb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:50.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.815+0000 7f1358c00640 0 client.26927 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:16:50.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.815+0000 7f1374623740 1 -- 172.21.15.28:0/863708913 >> 172.21.15.28:0/863708913 conn(0x558a1c7e8ee0 msgr2=0x558a1c7efc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:16:50.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.815+0000 7f1374623740 1 -- 172.21.15.28:0/863708913 shutdown_connections 2024-05-12T22:16:50.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.815+0000 7f1374623740 1 -- 172.21.15.28:0/863708913 wait complete. 2024-05-12T22:16:50.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.815+0000 7f1374623740 20 client.26927 trim_cache size 0 max 16384 2024-05-12T22:16:50.824 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-05-12T22:16:50.824 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-05-12T22:16:50.824 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-05-12T22:16:50.824 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-05-12T22:16:50.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.816+0000 7f1371800640 1 -- 172.21.15.28:0/4209857396 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f13500a2d00 msgr2=0x7f13500a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:16:50.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.816+0000 7f1371800640 1 --2- 172.21.15.28:0/4209857396 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f13500a2d00 0x7f13500a51c0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f135c0ccf70 tx=0x7f135c0d5550 comp rx=0 tx=0).stop 2024-05-12T22:16:50.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.816+0000 7f1371800640 1 -- 172.21.15.28:0/4209857396 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f136c075d60 msgr2=0x7f136c19e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:16:50.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.816+0000 7f1371800640 1 --2- 172.21.15.28:0/4209857396 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f136c075d60 0x7f136c19e760 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f135c00e9c0 tx=0x7f135c00ee80 comp rx=0 tx=0).stop 2024-05-12T22:16:50.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.816+0000 7f1371800640 1 -- 172.21.15.28:0/4209857396 shutdown_connections 2024-05-12T22:16:50.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.816+0000 7f1371800640 1 --2- 172.21.15.28:0/4209857396 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f13500a2d00 0x7f13500a51c0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:50.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.816+0000 7f1371800640 1 --2- 172.21.15.28:0/4209857396 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f136c1126b0 0x7f136c1a3020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:50.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.816+0000 7f1371800640 1 --2- 172.21.15.28:0/4209857396 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f136c076680 0x7f136c19eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:50.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.816+0000 7f1371800640 1 --2- 172.21.15.28:0/4209857396 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f136c075d60 0x7f136c19e760 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:50.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.816+0000 7f1371800640 1 -- 172.21.15.28:0/4209857396 >> 172.21.15.28:0/4209857396 conn(0x7f136c0ff4a0 msgr2=0x7f136c101890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:16:50.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.816+0000 7f1371800640 1 -- 172.21.15.28:0/4209857396 shutdown_connections 2024-05-12T22:16:50.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:50.816+0000 7f1371800640 1 -- 172.21.15.28:0/4209857396 wait complete. 2024-05-12T22:16:50.834 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-12T22:16:50.834 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-12T22:16:51.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.050+0000 7f7a08e00640 1 -- 172.21.15.28:0/3333421966 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7a04076680 msgr2=0x7f7a04112160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:16:51.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.050+0000 7f7a08e00640 1 --2- 172.21.15.28:0/3333421966 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7a04076680 0x7f7a04112160 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f79ec009930 tx=0x7f79ec02a440 comp rx=0 tx=0).stop 2024-05-12T22:16:51.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.050+0000 7f7a08e00640 1 -- 172.21.15.28:0/3333421966 shutdown_connections 2024-05-12T22:16:51.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.050+0000 7f7a08e00640 1 --2- 172.21.15.28:0/3333421966 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7a041126a0 0x7f7a04114a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:51.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.050+0000 7f7a08e00640 1 --2- 172.21.15.28:0/3333421966 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7a04076680 0x7f7a04112160 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:51.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.050+0000 7f7a08e00640 1 --2- 172.21.15.28:0/3333421966 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a04075d60 0x7f7a04076140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:51.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.050+0000 7f7a08e00640 1 -- 172.21.15.28:0/3333421966 >> 172.21.15.28:0/3333421966 conn(0x7f7a040ff530 msgr2=0x7f7a04101950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:16:51.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.051+0000 7f7a08e00640 1 -- 172.21.15.28:0/3333421966 shutdown_connections 2024-05-12T22:16:51.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.051+0000 7f7a08e00640 1 -- 172.21.15.28:0/3333421966 wait complete. 2024-05-12T22:16:51.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.052+0000 7f7a08e00640 1 Processor -- start 2024-05-12T22:16:51.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.052+0000 7f7a08e00640 1 -- start start 2024-05-12T22:16:51.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.052+0000 7f7a08e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7a04075d60 0x7f7a041a4570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:51.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.052+0000 7f7a08e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7a04076680 0x7f7a041a4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:51.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.052+0000 7f7a08e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a041126a0 0x7f7a0419e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:51.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.052+0000 7f7a08e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7a04118240 con 0x7f7a041126a0 2024-05-12T22:16:51.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.052+0000 7f7a08e00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f7a041180c0 con 0x7f7a04076680 2024-05-12T22:16:51.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.052+0000 7f7a08e00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f7a041183c0 con 0x7f7a04075d60 2024-05-12T22:16:51.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.052+0000 7f7a02800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a041126a0 0x7f7a0419e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:51.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.052+0000 7f7a01e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7a04075d60 0x7f7a041a4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:51.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.052+0000 7f7a01e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7a04075d60 0x7f7a041a4570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:59236/0 (socket says 172.21.15.28:59236) 2024-05-12T22:16:51.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.052+0000 7f7a02800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a041126a0 0x7f7a0419e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35438/0 (socket says 172.21.15.28:35438) 2024-05-12T22:16:51.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.053+0000 7f7a01e00640 1 -- 172.21.15.28:0/2309356405 learned_addr learned my addr 172.21.15.28:0/2309356405 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:16:51.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.053+0000 7f7a01400640 1 --2- 172.21.15.28:0/2309356405 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7a04076680 0x7f7a041a4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:51.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.053+0000 7f7a01e00640 1 -- 172.21.15.28:0/2309356405 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7a04076680 msgr2=0x7f7a041a4ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:16:51.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.053+0000 7f7a01e00640 1 --2- 172.21.15.28:0/2309356405 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7a04076680 0x7f7a041a4ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:51.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.053+0000 7f7a01e00640 1 -- 172.21.15.28:0/2309356405 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a041126a0 msgr2=0x7f7a0419e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:16:51.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.053+0000 7f7a01e00640 1 --2- 172.21.15.28:0/2309356405 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a041126a0 0x7f7a0419e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:51.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.053+0000 7f7a01e00640 1 -- 172.21.15.28:0/2309356405 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a0419ef40 con 0x7f7a04075d60 2024-05-12T22:16:51.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.053+0000 7f7a01e00640 1 --2- 172.21.15.28:0/2309356405 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7a04075d60 0x7f7a041a4570 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f79f4009d50 tx=0x7f79f400c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:16:51.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.054+0000 7f79faa00640 1 -- 172.21.15.28:0/2309356405 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79f4003b70 con 0x7f7a04075d60 2024-05-12T22:16:51.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.054+0000 7f79faa00640 1 -- 172.21.15.28:0/2309356405 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f79f4003d10 con 0x7f7a04075d60 2024-05-12T22:16:51.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.054+0000 7f79faa00640 1 -- 172.21.15.28:0/2309356405 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79f40108f0 con 0x7f7a04075d60 2024-05-12T22:16:51.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.054+0000 7f7a08e00640 1 -- 172.21.15.28:0/2309356405 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a0419f230 con 0x7f7a04075d60 2024-05-12T22:16:51.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.054+0000 7f7a08e00640 1 -- 172.21.15.28:0/2309356405 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a041ab4f0 con 0x7f7a04075d60 2024-05-12T22:16:51.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.055+0000 7f7a08e00640 1 -- 172.21.15.28:0/2309356405 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a0419f570 con 0x7f7a04075d60 2024-05-12T22:16:51.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.057+0000 7f79faa00640 1 -- 172.21.15.28:0/2309356405 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7f79f4004000 con 0x7f7a04075d60 2024-05-12T22:16:51.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.058+0000 7f79faa00640 1 --2- 172.21.15.28:0/2309356405 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f79e80a2d00 0x7f79e80a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:51.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.058+0000 7f79faa00640 1 -- 172.21.15.28:0/2309356405 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8513+0+0 (secure 0 0 0) 0x7f79f40cfac0 con 0x7f7a04075d60 2024-05-12T22:16:51.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.059+0000 7f7a01400640 1 --2- 172.21.15.28:0/2309356405 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f79e80a2d00 0x7f79e80a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:51.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.059+0000 7f7a01400640 1 --2- 172.21.15.28:0/2309356405 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f79e80a2d00 0x7f79e80a51c0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f79ec006010 tx=0x7f79ec000bc0 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:16:51.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.062+0000 7f79faa00640 1 -- 172.21.15.28:0/2309356405 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f79f409c680 con 0x7f7a04075d60 2024-05-12T22:16:51.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.309+0000 7f7a08e00640 1 -- 172.21.15.28:0/2309356405 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7a040774a0 con 0x7f7a04075d60 2024-05-12T22:16:51.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.310+0000 7f79faa00640 1 -- 172.21.15.28:0/2309356405 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11864 (secure 0 0 0) 0x7f79f409bf80 con 0x7f7a04075d60 2024-05-12T22:16:51.311 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-12T22:16:51.312 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1844141745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1844141745},{"type":"v1","addr":"172.21.15.186:6833","nonce":1844141745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6833/2960763758","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":2960763758},{"type":"v1","addr":"172.21.15.86:6833","nonce":2960763758}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24470,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/4131134219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":4131134219},{"type":"v1","addr":"172.21.15.28:6835","nonce":4131134219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24481,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/1727869064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1727869064},{"type":"v1","addr":"172.21.15.186:6835","nonce":1727869064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24482,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2938260704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2938260704},{"type":"v1","addr":"172.21.15.28:6837","nonce":2938260704}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24494,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3473700910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3473700910},{"type":"v1","addr":"172.21.15.86:6835","nonce":3473700910}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-12T22:06:33.426986+0000","modified":"2024-05-12T22:13:22.499381+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24581,"mds_1":24550,"mds_2":24532},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24499":{"gid":24499,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6837/3600809235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":3600809235},{"type":"v1","addr":"172.21.15.186:6837","nonce":3600809235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"i","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6839/2399610807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":2399610807},{"type":"v1","addr":"172.21.15.186:6839","nonce":2399610807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24530":{"gid":24530,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.86:6837/3782448884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":3782448884},{"type":"v1","addr":"172.21.15.86:6837","nonce":3782448884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24532":{"gid":24532,"name":"g","rank":2,"incarnation":21,"state":"up:active","state_seq":6,"addr":"172.21.15.28:6839/123892158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":123892158},{"type":"v1","addr":"172.21.15.28:6839","nonce":123892158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24550":{"gid":24550,"name":"a","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.28:6841/761183870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":761183870},{"type":"v1","addr":"172.21.15.28:6841","nonce":761183870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24581":{"gid":24581,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6839/1251999130","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6838","nonce":1251999130},{"type":"v1","addr":"172.21.15.86:6839","nonce":1251999130}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24581,"qdb_cluster":[24550,24581,24532]},"id":1}]} 2024-05-12T22:16:51.312 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 26 2024-05-12T22:16:51.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.313+0000 7f7a08e00640 1 -- 172.21.15.28:0/2309356405 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f79e80a2d00 msgr2=0x7f79e80a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:16:51.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.313+0000 7f7a08e00640 1 --2- 172.21.15.28:0/2309356405 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f79e80a2d00 0x7f79e80a51c0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f79ec006010 tx=0x7f79ec000bc0 comp rx=0 tx=0).stop 2024-05-12T22:16:51.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.313+0000 7f7a08e00640 1 -- 172.21.15.28:0/2309356405 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7a04075d60 msgr2=0x7f7a041a4570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:16:51.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.313+0000 7f7a08e00640 1 --2- 172.21.15.28:0/2309356405 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7a04075d60 0x7f7a041a4570 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f79f4009d50 tx=0x7f79f400c6a0 comp rx=0 tx=0).stop 2024-05-12T22:16:51.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.313+0000 7f7a08e00640 1 -- 172.21.15.28:0/2309356405 shutdown_connections 2024-05-12T22:16:51.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.313+0000 7f7a08e00640 1 --2- 172.21.15.28:0/2309356405 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f79e80a2d00 0x7f79e80a51c0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:51.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.313+0000 7f7a08e00640 1 --2- 172.21.15.28:0/2309356405 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a041126a0 0x7f7a0419e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:51.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.313+0000 7f7a08e00640 1 --2- 172.21.15.28:0/2309356405 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7a04076680 0x7f7a041a4ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:51.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.313+0000 7f7a08e00640 1 --2- 172.21.15.28:0/2309356405 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7a04075d60 0x7f7a041a4570 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:51.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.313+0000 7f7a08e00640 1 -- 172.21.15.28:0/2309356405 >> 172.21.15.28:0/2309356405 conn(0x7f7a040ff530 msgr2=0x7f7a04113ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:16:51.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.313+0000 7f7a08e00640 1 -- 172.21.15.28:0/2309356405 shutdown_connections 2024-05-12T22:16:51.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.313+0000 7f7a08e00640 1 -- 172.21.15.28:0/2309356405 wait complete. 2024-05-12T22:16:51.333 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-05-12T22:16:51.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.555+0000 7f776b400640 1 -- 172.21.15.28:0/792754150 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7764103a10 msgr2=0x7f7764103e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:16:51.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.555+0000 7f776b400640 1 --2- 172.21.15.28:0/792754150 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7764103a10 0x7f7764103e90 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f7750009900 tx=0x7f775002a400 comp rx=0 tx=0).stop 2024-05-12T22:16:51.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.555+0000 7f776b400640 1 -- 172.21.15.28:0/792754150 shutdown_connections 2024-05-12T22:16:51.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.555+0000 7f776b400640 1 --2- 172.21.15.28:0/792754150 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f77641043d0 0x7f77641108e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:51.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.555+0000 7f776b400640 1 --2- 172.21.15.28:0/792754150 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7764103a10 0x7f7764103e90 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:51.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.555+0000 7f776b400640 1 --2- 172.21.15.28:0/792754150 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7764109a20 0x7f7764109e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:51.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.555+0000 7f776b400640 1 -- 172.21.15.28:0/792754150 >> 172.21.15.28:0/792754150 conn(0x7f77640ff4a0 msgr2=0x7f77641018c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:16:51.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.555+0000 7f776b400640 1 -- 172.21.15.28:0/792754150 shutdown_connections 2024-05-12T22:16:51.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.555+0000 7f776b400640 1 -- 172.21.15.28:0/792754150 wait complete. 2024-05-12T22:16:51.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.556+0000 7f776b400640 1 Processor -- start 2024-05-12T22:16:51.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.556+0000 7f776b400640 1 -- start start 2024-05-12T22:16:51.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.557+0000 7f776b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7764103a10 0x7f776419a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:51.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.557+0000 7f776b400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f77641043d0 0x7f776419a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:51.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.557+0000 7f776b400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7764109a20 0x7f776419ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:51.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.557+0000 7f776b400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7764113e80 con 0x7f7764103a10 2024-05-12T22:16:51.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.557+0000 7f776b400640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f7764113d00 con 0x7f77641043d0 2024-05-12T22:16:51.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.557+0000 7f776b400640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f7764114000 con 0x7f7764109a20 2024-05-12T22:16:51.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.557+0000 7f7768a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7764103a10 0x7f776419a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:51.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.557+0000 7f7763e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f77641043d0 0x7f776419a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:51.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.557+0000 7f7769400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7764109a20 0x7f776419ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:51.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.557+0000 7f7763e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f77641043d0 0x7f776419a900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:49158/0 (socket says 172.21.15.28:49158) 2024-05-12T22:16:51.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.557+0000 7f7763e00640 1 -- 172.21.15.28:0/2478567837 learned_addr learned my addr 172.21.15.28:0/2478567837 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:16:51.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.558+0000 7f7763e00640 1 -- 172.21.15.28:0/2478567837 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7764109a20 msgr2=0x7f776419ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:16:51.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.558+0000 7f7763e00640 1 --2- 172.21.15.28:0/2478567837 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7764109a20 0x7f776419ec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:51.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.558+0000 7f7763e00640 1 -- 172.21.15.28:0/2478567837 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7764103a10 msgr2=0x7f776419a3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:16:51.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.558+0000 7f7763e00640 1 --2- 172.21.15.28:0/2478567837 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7764103a10 0x7f776419a3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:51.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.558+0000 7f7763e00640 1 -- 172.21.15.28:0/2478567837 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f776419f360 con 0x7f77641043d0 2024-05-12T22:16:51.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.558+0000 7f7768a00640 1 --2- 172.21.15.28:0/2478567837 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7764103a10 0x7f776419a3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:16:51.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.558+0000 7f7763e00640 1 --2- 172.21.15.28:0/2478567837 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f77641043d0 0x7f776419a900 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f775002a9f0 tx=0x7f7750003c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:16:51.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.558+0000 7f7761600640 1 -- 172.21.15.28:0/2478567837 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f775003f070 con 0x7f77641043d0 2024-05-12T22:16:51.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.558+0000 7f776b400640 1 -- 172.21.15.28:0/2478567837 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f776419f5f0 con 0x7f77641043d0 2024-05-12T22:16:51.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.558+0000 7f776b400640 1 -- 172.21.15.28:0/2478567837 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77641a6ee0 con 0x7f77641043d0 2024-05-12T22:16:51.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.558+0000 7f7761600640 1 -- 172.21.15.28:0/2478567837 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7750031c30 con 0x7f77641043d0 2024-05-12T22:16:51.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.558+0000 7f7761600640 1 -- 172.21.15.28:0/2478567837 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7750049060 con 0x7f77641043d0 2024-05-12T22:16:51.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.562+0000 7f776e176740 1 Processor -- start 2024-05-12T22:16:51.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.562+0000 7f7761600640 1 -- 172.21.15.28:0/2478567837 <== mon.2 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7f7750042070 con 0x7f77641043d0 2024-05-12T22:16:51.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.562+0000 7f776e176740 1 -- start start 2024-05-12T22:16:51.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.562+0000 7f776e176740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564ffac61980 0x564ffac61d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:51.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.562+0000 7f776e176740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564ffac622a0 0x564ffac6c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:51.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.562+0000 7f776e176740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ffac6c970 0x564ffac6ed60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:51.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.562+0000 7f776e176740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x564ffabc6a30 con 0x564ffac6c970 2024-05-12T22:16:51.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.562+0000 7f776e176740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x564ffab6e700 con 0x564ffac622a0 2024-05-12T22:16:51.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.562+0000 7f776e176740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x564ffabc29f0 con 0x564ffac61980 2024-05-12T22:16:51.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.563+0000 7f7763e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ffac6c970 0x564ffac6ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:51.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.563+0000 7f7768a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564ffac622a0 0x564ffac6c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:51.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.563+0000 7f7769400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564ffac61980 0x564ffac61d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:51.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.563+0000 7f7768a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564ffac622a0 0x564ffac6c430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:49172/0 (socket says 172.21.15.28:49172) 2024-05-12T22:16:51.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.563+0000 7f7768a00640 1 -- 172.21.15.28:0/2602514366 learned_addr learned my addr 172.21.15.28:0/2602514366 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:16:51.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.563+0000 7f7768a00640 1 -- 172.21.15.28:0/2602514366 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564ffac61980 msgr2=0x564ffac61d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:16:51.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.563+0000 7f7768a00640 1 --2- 172.21.15.28:0/2602514366 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564ffac61980 0x564ffac61d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:51.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.563+0000 7f7768a00640 1 -- 172.21.15.28:0/2602514366 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ffac6c970 msgr2=0x564ffac6ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:16:51.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.563+0000 7f7768a00640 1 --2- 172.21.15.28:0/2602514366 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ffac6c970 0x564ffac6ed60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:51.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.563+0000 7f7768a00640 1 -- 172.21.15.28:0/2602514366 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564ffa8da3f0 con 0x564ffac622a0 2024-05-12T22:16:51.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.563+0000 7f7769400640 1 --2- 172.21.15.28:0/2602514366 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564ffac61980 0x564ffac61d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:16:51.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.563+0000 7f7763e00640 1 --2- 172.21.15.28:0/2602514366 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ffac6c970 0x564ffac6ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:16:51.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.564+0000 7f7761600640 1 --2- 172.21.15.28:0/2478567837 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f774c0a2d00 0x7f774c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:51.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.564+0000 7f7768a00640 1 --2- 172.21.15.28:0/2602514366 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564ffac622a0 0x564ffac6c430 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f77580114f0 tx=0x7f77580119b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=62fa9424563eec2d server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:16:51.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.564+0000 7f7761600640 1 -- 172.21.15.28:0/2478567837 <== mon.2 v2:172.21.15.86:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8513+0+0 (secure 0 0 0) 0x7f77500ef820 con 0x7f77641043d0 2024-05-12T22:16:51.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.564+0000 7f7755c00640 1 -- 172.21.15.28:0/2602514366 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f775801e070 con 0x564ffac622a0 2024-05-12T22:16:51.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.564+0000 7f7769400640 1 --2- 172.21.15.28:0/2478567837 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f774c0a2d00 0x7f774c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:51.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.564+0000 7f7755c00640 1 -- 172.21.15.28:0/2602514366 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f775800faa0 con 0x564ffac622a0 2024-05-12T22:16:51.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.564+0000 7f7755c00640 1 -- 172.21.15.28:0/2602514366 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7758019b30 con 0x564ffac622a0 2024-05-12T22:16:51.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.565+0000 7f776e176740 1 -- 172.21.15.28:0/2602514366 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564ffac622a0 msgr2=0x564ffac6c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:16:51.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.565+0000 7f776e176740 1 --2- 172.21.15.28:0/2602514366 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564ffac622a0 0x564ffac6c430 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f77580114f0 tx=0x7f77580119b0 comp rx=0 tx=0).stop 2024-05-12T22:16:51.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.565+0000 7f7769400640 1 --2- 172.21.15.28:0/2478567837 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f774c0a2d00 0x7f774c0a51c0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f775c005fd0 tx=0x7f775c009600 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:16:51.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.565+0000 7f776e176740 1 -- 172.21.15.28:0/2602514366 shutdown_connections 2024-05-12T22:16:51.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.565+0000 7f776e176740 1 --2- 172.21.15.28:0/2602514366 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ffac6c970 0x564ffac6ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:51.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.565+0000 7f776e176740 1 --2- 172.21.15.28:0/2602514366 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564ffac622a0 0x564ffac6c430 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:51.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.565+0000 7f776e176740 1 --2- 172.21.15.28:0/2602514366 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564ffac61980 0x564ffac61d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:51.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.565+0000 7f776e176740 1 -- 172.21.15.28:0/2602514366 >> 172.21.15.28:0/2602514366 conn(0x564ffac5dca0 msgr2=0x564ffac5e080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:16:51.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.565+0000 7f776e176740 1 -- 172.21.15.28:0/2602514366 shutdown_connections 2024-05-12T22:16:51.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.565+0000 7f776e176740 1 -- 172.21.15.28:0/2602514366 wait complete. 2024-05-12T22:16:51.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.565+0000 7f776e176740 1 -- start start 2024-05-12T22:16:51.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.566+0000 7f776e176740 1 Processor -- start 2024-05-12T22:16:51.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.566+0000 7f776e176740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564ffac61980 0x564ffac698e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:51.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.566+0000 7f776e176740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ffac622a0 0x564ffac69e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:51.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.566+0000 7f776e176740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564ffac6c970 0x564ffac6a360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:51.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.566+0000 7f776e176740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x564ffac6f760 con 0x564ffac622a0 2024-05-12T22:16:51.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.566+0000 7f776e176740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x564ffab81a70 con 0x564ffac6c970 2024-05-12T22:16:51.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.566+0000 7f776e176740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x564ffac6f8e0 con 0x564ffac61980 2024-05-12T22:16:51.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.566+0000 7f776e176740 10 client.0 _ll_register_callbacks cb 0x564ffab64ba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:16:51.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.566+0000 7f7763e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ffac622a0 0x564ffac69e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:51.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.566+0000 7f7769400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564ffac6c970 0x564ffac6a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:51.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.566+0000 7f7763e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ffac622a0 0x564ffac69e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35468/0 (socket says 172.21.15.28:35468) 2024-05-12T22:16:51.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.566+0000 7f7763e00640 1 -- 172.21.15.28:0/3183815031 learned_addr learned my addr 172.21.15.28:0/3183815031 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:16:51.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.566+0000 7f7768a00640 1 --2- 172.21.15.28:0/3183815031 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564ffac61980 0x564ffac698e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:51.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.567+0000 7f7763e00640 1 -- 172.21.15.28:0/3183815031 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564ffac61980 msgr2=0x564ffac698e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:16:51.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.567+0000 7f7763e00640 1 --2- 172.21.15.28:0/3183815031 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564ffac61980 0x564ffac698e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:51.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.567+0000 7f7763e00640 1 -- 172.21.15.28:0/3183815031 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564ffac6c970 msgr2=0x564ffac6a360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:16:51.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.567+0000 7f7763e00640 1 --2- 172.21.15.28:0/3183815031 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564ffac6c970 0x564ffac6a360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:51.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.567+0000 7f7763e00640 1 -- 172.21.15.28:0/3183815031 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564ffa9a47d0 con 0x564ffac622a0 2024-05-12T22:16:51.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.567+0000 7f7763e00640 1 --2- 172.21.15.28:0/3183815031 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ffac622a0 0x564ffac69e20 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f7750004650 tx=0x7f775003b800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=542891f4966bab19 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:16:51.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.567+0000 7f774a000640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-12T22:16:51.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.567+0000 7f774a000640 1 -- 172.21.15.28:0/3183815031 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77500bb6f0 con 0x564ffac622a0 2024-05-12T22:16:51.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.567+0000 7f774a000640 1 -- 172.21.15.28:0/3183815031 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8513+0+0 (secure 0 0 0) 0x7f77500b8400 con 0x564ffac622a0 2024-05-12T22:16:51.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.567+0000 7f776e176740 1 -- 172.21.15.28:0/3183815031 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f775800faa0 con 0x564ffac622a0 2024-05-12T22:16:51.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.568+0000 7f774a000640 1 -- 172.21.15.28:0/3183815031 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f77500bbd90 con 0x564ffac622a0 2024-05-12T22:16:51.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.568+0000 7f776e176740 10 client.16815 fetch_fsmap learned FSMap version 26 2024-05-12T22:16:51.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.568+0000 7f776e176740 1 -- 172.21.15.28:0/3183815031 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x564ffac7ba40 con 0x564ffac622a0 2024-05-12T22:16:51.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.568+0000 7f774a000640 1 -- 172.21.15.28:0/3183815031 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f775800faa0 con 0x564ffac622a0 2024-05-12T22:16:51.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.568+0000 7f774a000640 1 -- 172.21.15.28:0/3183815031 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f775004c060 con 0x564ffac622a0 2024-05-12T22:16:51.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.568+0000 7f776e176740 10 client.16815 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:16:51.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.568+0000 7f776e176740 10 client.16815 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.g 2024-05-12T22:16:51.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.568+0000 7f776e176740 20 client.16815 populate_metadata read hostname 'smithi028' 2024-05-12T22:16:51.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.568+0000 7f776e176740 1 --2- 172.21.15.28:0/3183815031 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x564ffac7cf30 0x564ffac7f360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:51.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.568+0000 7f776e176740 4 client.16815 mds_command: new command op to 24532 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-12T22:16:51.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.568+0000 7f776e176740 1 -- 172.21.15.28:0/3183815031 --> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564ffabd4160 con 0x564ffac7cf30 2024-05-12T22:16:51.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.569+0000 7f7768a00640 1 --2- 172.21.15.28:0/3183815031 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x564ffac7cf30 0x564ffac7f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:51.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.569+0000 7f7768a00640 1 --2- 172.21.15.28:0/3183815031 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x564ffac7cf30 0x564ffac7f360 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=fa7fa4613a073bc5 server_cookie=305965a9b8ddd463 in_seq=0 out_seq=0 2024-05-12T22:16:51.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.569+0000 7f774a000640 10 client.16815 ms_handle_connect on v2:172.21.15.28:6838/123892158 2024-05-12T22:16:51.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.570+0000 7f774a000640 1 -- 172.21.15.28:0/3183815031 <== mds.2 v2:172.21.15.28:6838/123892158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564ffabd4160 con 0x564ffac7cf30 2024-05-12T22:16:51.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.570+0000 7f774a000640 10 client.16815 handle_command_reply: tid=0 2024-05-12T22:16:51.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.570+0000 7f776e176740 1 client.16815 shutdown 2024-05-12T22:16:51.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.570+0000 7f776e176740 1 -- 172.21.15.28:0/3183815031 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ffac622a0 msgr2=0x564ffac69e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:16:51.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.570+0000 7f776e176740 1 --2- 172.21.15.28:0/3183815031 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ffac622a0 0x564ffac69e20 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f7750004650 tx=0x7f775003b800 comp rx=0 tx=0).stop 2024-05-12T22:16:51.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.570+0000 7f776e176740 1 -- 172.21.15.28:0/3183815031 shutdown_connections 2024-05-12T22:16:51.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.570+0000 7f776e176740 1 --2- 172.21.15.28:0/3183815031 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x564ffac7cf30 0x564ffac7f360 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:51.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.570+0000 7f776e176740 1 --2- 172.21.15.28:0/3183815031 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564ffac6c970 0x564ffac6a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:51.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.570+0000 7f774a000640 0 client.16815 ms_handle_reset on v2:172.21.15.28:6838/123892158 2024-05-12T22:16:51.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.570+0000 7f776e176740 1 --2- 172.21.15.28:0/3183815031 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ffac622a0 0x564ffac69e20 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:51.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.570+0000 7f776e176740 1 --2- 172.21.15.28:0/3183815031 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564ffac61980 0x564ffac698e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:51.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.570+0000 7f776e176740 1 -- 172.21.15.28:0/3183815031 >> 172.21.15.28:0/3183815031 conn(0x564ffac5dca0 msgr2=0x564ffac63aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:16:51.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.571+0000 7f776e176740 1 -- 172.21.15.28:0/3183815031 shutdown_connections 2024-05-12T22:16:51.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.571+0000 7f776e176740 1 -- 172.21.15.28:0/3183815031 wait complete. 2024-05-12T22:16:51.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.571+0000 7f776e176740 20 client.16815 trim_cache size 0 max 16384 2024-05-12T22:16:51.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.586+0000 7f776e176740 1 Processor -- start 2024-05-12T22:16:51.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.586+0000 7f776e176740 1 -- start start 2024-05-12T22:16:51.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.586+0000 7f776e176740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564ffac61980 0x564ffac78680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:51.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.586+0000 7f776e176740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564ffac622a0 0x564ffac84c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:51.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.586+0000 7f776e176740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ffac6c970 0x564ffac851b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:51.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.586+0000 7f776e176740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f77641b1fa0 con 0x564ffac6c970 2024-05-12T22:16:51.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.586+0000 7f776e176740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f77641af870 con 0x564ffac61980 2024-05-12T22:16:51.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.586+0000 7f776e176740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f77641ad6e0 con 0x564ffac622a0 2024-05-12T22:16:51.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.587+0000 7f7769400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ffac6c970 0x564ffac851b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:51.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.587+0000 7f7769400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ffac6c970 0x564ffac851b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35480/0 (socket says 172.21.15.28:35480) 2024-05-12T22:16:51.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.587+0000 7f7769400640 1 -- 172.21.15.28:0/1711402863 learned_addr learned my addr 172.21.15.28:0/1711402863 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:16:51.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.587+0000 7f7768a00640 1 --2- 172.21.15.28:0/1711402863 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564ffac61980 0x564ffac78680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:51.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.587+0000 7f7763e00640 1 --2- 172.21.15.28:0/1711402863 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564ffac622a0 0x564ffac84c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:51.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.587+0000 7f7769400640 1 -- 172.21.15.28:0/1711402863 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564ffac622a0 msgr2=0x564ffac84c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:16:51.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.587+0000 7f7769400640 1 --2- 172.21.15.28:0/1711402863 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564ffac622a0 0x564ffac84c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:51.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.587+0000 7f7769400640 1 -- 172.21.15.28:0/1711402863 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564ffac61980 msgr2=0x564ffac78680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:16:51.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.587+0000 7f7769400640 1 --2- 172.21.15.28:0/1711402863 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564ffac61980 0x564ffac78680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:51.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.587+0000 7f7769400640 1 -- 172.21.15.28:0/1711402863 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564ffac620d0 con 0x564ffac6c970 2024-05-12T22:16:51.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.587+0000 7f7768a00640 1 --2- 172.21.15.28:0/1711402863 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564ffac61980 0x564ffac78680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:16:51.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.587+0000 7f7769400640 1 --2- 172.21.15.28:0/1711402863 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ffac6c970 0x564ffac851b0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f775c005f40 tx=0x7f775c016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6440e98ab247109d server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:16:51.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.588+0000 7f7755200640 1 -- 172.21.15.28:0/1711402863 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f775c01d430 con 0x564ffac6c970 2024-05-12T22:16:51.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.588+0000 7f7755200640 1 -- 172.21.15.28:0/1711402863 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f775c01dab0 con 0x564ffac6c970 2024-05-12T22:16:51.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.588+0000 7f7755200640 1 -- 172.21.15.28:0/1711402863 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f775c020c30 con 0x564ffac6c970 2024-05-12T22:16:51.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.588+0000 7f776e176740 1 -- 172.21.15.28:0/1711402863 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ffac6c970 msgr2=0x564ffac851b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:16:51.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.588+0000 7f776e176740 1 --2- 172.21.15.28:0/1711402863 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ffac6c970 0x564ffac851b0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f775c005f40 tx=0x7f775c016000 comp rx=0 tx=0).stop 2024-05-12T22:16:51.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.588+0000 7f776e176740 1 -- 172.21.15.28:0/1711402863 shutdown_connections 2024-05-12T22:16:51.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.588+0000 7f776e176740 1 --2- 172.21.15.28:0/1711402863 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ffac6c970 0x564ffac851b0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:51.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.588+0000 7f776e176740 1 --2- 172.21.15.28:0/1711402863 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564ffac622a0 0x564ffac84c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:51.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.588+0000 7f776e176740 1 --2- 172.21.15.28:0/1711402863 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564ffac61980 0x564ffac78680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:51.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.588+0000 7f776e176740 1 -- 172.21.15.28:0/1711402863 >> 172.21.15.28:0/1711402863 conn(0x564ffac5dca0 msgr2=0x564ffac64c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:16:51.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.589+0000 7f776e176740 1 -- 172.21.15.28:0/1711402863 shutdown_connections 2024-05-12T22:16:51.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.589+0000 7f776e176740 1 -- 172.21.15.28:0/1711402863 wait complete. 2024-05-12T22:16:51.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.589+0000 7f776e176740 1 -- start start 2024-05-12T22:16:51.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.589+0000 7f776e176740 1 Processor -- start 2024-05-12T22:16:51.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.590+0000 7f776e176740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564ffac61980 0x564ffac74e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:51.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.590+0000 7f776e176740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ffac622a0 0x564ffac66230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:51.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.590+0000 7f776e176740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564ffac6c970 0x564ffac66770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:51.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.590+0000 7f776e176740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f77641add30 con 0x564ffac622a0 2024-05-12T22:16:51.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.590+0000 7f776e176740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f77641af1a0 con 0x564ffac61980 2024-05-12T22:16:51.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.590+0000 7f776e176740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f77641ae400 con 0x564ffac6c970 2024-05-12T22:16:51.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.590+0000 7f776e176740 10 client.0 _ll_register_callbacks cb 0x7f7750005b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:16:51.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.590+0000 7f7763e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ffac622a0 0x564ffac66230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:51.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.590+0000 7f7763e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ffac622a0 0x564ffac66230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35496/0 (socket says 172.21.15.28:35496) 2024-05-12T22:16:51.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.590+0000 7f7763e00640 1 -- 172.21.15.28:0/362893346 learned_addr learned my addr 172.21.15.28:0/362893346 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:16:51.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.590+0000 7f7769400640 1 --2- 172.21.15.28:0/362893346 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564ffac6c970 0x564ffac66770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:51.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.590+0000 7f7768a00640 1 --2- 172.21.15.28:0/362893346 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564ffac61980 0x564ffac74e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:51.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.590+0000 7f7763e00640 1 -- 172.21.15.28:0/362893346 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564ffac6c970 msgr2=0x564ffac66770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:16:51.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.590+0000 7f7763e00640 1 --2- 172.21.15.28:0/362893346 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564ffac6c970 0x564ffac66770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:51.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.590+0000 7f7763e00640 1 -- 172.21.15.28:0/362893346 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564ffac61980 msgr2=0x564ffac74e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:16:51.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.590+0000 7f7763e00640 1 --2- 172.21.15.28:0/362893346 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564ffac61980 0x564ffac74e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:51.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.590+0000 7f7763e00640 1 -- 172.21.15.28:0/362893346 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564ffac66eb0 con 0x564ffac622a0 2024-05-12T22:16:51.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.591+0000 7f7763e00640 1 --2- 172.21.15.28:0/362893346 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ffac622a0 0x564ffac66230 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f77500c4420 tx=0x7f775003b690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e911b2cf6da329a server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:16:51.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.591+0000 7f774a000640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-12T22:16:51.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.591+0000 7f774a000640 1 -- 172.21.15.28:0/362893346 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77500bb6f0 con 0x564ffac622a0 2024-05-12T22:16:51.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.591+0000 7f774a000640 1 -- 172.21.15.28:0/362893346 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8513+0+0 (secure 0 0 0) 0x7f775009abb0 con 0x564ffac622a0 2024-05-12T22:16:51.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.591+0000 7f776e176740 1 -- 172.21.15.28:0/362893346 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f775c01dab0 con 0x564ffac622a0 2024-05-12T22:16:51.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.591+0000 7f774a000640 1 -- 172.21.15.28:0/362893346 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f77500c4950 con 0x564ffac622a0 2024-05-12T22:16:51.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.591+0000 7f776e176740 10 client.16821 fetch_fsmap learned FSMap version 26 2024-05-12T22:16:51.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.591+0000 7f776e176740 1 -- 172.21.15.28:0/362893346 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x564fface3450 con 0x564ffac622a0 2024-05-12T22:16:51.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.591+0000 7f774a000640 1 -- 172.21.15.28:0/362893346 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f775c01dab0 con 0x564ffac622a0 2024-05-12T22:16:51.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.591+0000 7f774a000640 1 -- 172.21.15.28:0/362893346 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f77500b7ae0 con 0x564ffac622a0 2024-05-12T22:16:51.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.592+0000 7f776e176740 10 client.16821 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:16:51.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.592+0000 7f776e176740 10 client.16821 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.g 2024-05-12T22:16:51.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.592+0000 7f776e176740 20 client.16821 populate_metadata read hostname 'smithi028' 2024-05-12T22:16:51.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.592+0000 7f776e176740 1 --2- 172.21.15.28:0/362893346 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x564fface3a40 0x564fface5e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:51.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.592+0000 7f776e176740 4 client.16821 mds_command: new command op to 24532 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-12T22:16:51.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.592+0000 7f776e176740 1 -- 172.21.15.28:0/362893346 --> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564ffabce4b0 con 0x564fface3a40 2024-05-12T22:16:51.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.592+0000 7f7768a00640 1 --2- 172.21.15.28:0/362893346 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x564fface3a40 0x564fface5e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:51.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.592+0000 7f7768a00640 1 --2- 172.21.15.28:0/362893346 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x564fface3a40 0x564fface5e60 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=26efff831120901b server_cookie=cd7fa8d9ba517ea6 in_seq=0 out_seq=0 2024-05-12T22:16:51.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.592+0000 7f774a000640 10 client.16821 ms_handle_connect on v2:172.21.15.28:6838/123892158 2024-05-12T22:16:51.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.593+0000 7f774a000640 1 -- 172.21.15.28:0/362893346 <== mds.2 v2:172.21.15.28:6838/123892158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564ffabce4b0 con 0x564fface3a40 2024-05-12T22:16:51.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.593+0000 7f774a000640 10 client.16821 handle_command_reply: tid=0 2024-05-12T22:16:51.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.593+0000 7f776e176740 1 client.16821 shutdown 2024-05-12T22:16:51.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.593+0000 7f776e176740 1 -- 172.21.15.28:0/362893346 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ffac622a0 msgr2=0x564ffac66230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:16:51.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.593+0000 7f776e176740 1 --2- 172.21.15.28:0/362893346 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ffac622a0 0x564ffac66230 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f77500c4420 tx=0x7f775003b690 comp rx=0 tx=0).stop 2024-05-12T22:16:51.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.593+0000 7f776e176740 1 -- 172.21.15.28:0/362893346 shutdown_connections 2024-05-12T22:16:51.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.593+0000 7f776e176740 1 --2- 172.21.15.28:0/362893346 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564ffac6c970 0x564ffac66770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:51.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.593+0000 7f776e176740 1 --2- 172.21.15.28:0/362893346 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ffac622a0 0x564ffac66230 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:51.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.593+0000 7f776e176740 1 --2- 172.21.15.28:0/362893346 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x564fface3a40 0x564fface5e60 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:51.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.593+0000 7f776e176740 1 --2- 172.21.15.28:0/362893346 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564ffac61980 0x564ffac74e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:51.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.593+0000 7f776e176740 1 -- 172.21.15.28:0/362893346 >> 172.21.15.28:0/362893346 conn(0x564ffac5dca0 msgr2=0x564ffac5c530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:16:51.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.593+0000 7f774a000640 0 client.16821 ms_handle_reset on v2:172.21.15.28:6838/123892158 2024-05-12T22:16:51.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.593+0000 7f776e176740 1 -- 172.21.15.28:0/362893346 shutdown_connections 2024-05-12T22:16:51.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.593+0000 7f776e176740 1 -- 172.21.15.28:0/362893346 wait complete. 2024-05-12T22:16:51.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.593+0000 7f776e176740 20 client.16821 trim_cache size 0 max 16384 2024-05-12T22:16:51.608 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-05-12T22:16:51.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.594+0000 7f776b400640 1 -- 172.21.15.28:0/2478567837 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f774c0a2d00 msgr2=0x7f774c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:16:51.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.594+0000 7f776b400640 1 --2- 172.21.15.28:0/2478567837 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f774c0a2d00 0x7f774c0a51c0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f775c005fd0 tx=0x7f775c009600 comp rx=0 tx=0).stop 2024-05-12T22:16:51.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.594+0000 7f776b400640 1 -- 172.21.15.28:0/2478567837 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f77641043d0 msgr2=0x7f776419a900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:16:51.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.594+0000 7f776b400640 1 --2- 172.21.15.28:0/2478567837 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f77641043d0 0x7f776419a900 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f775002a9f0 tx=0x7f7750003c10 comp rx=0 tx=0).stop 2024-05-12T22:16:51.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.594+0000 7f776b400640 1 -- 172.21.15.28:0/2478567837 shutdown_connections 2024-05-12T22:16:51.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.594+0000 7f776b400640 1 --2- 172.21.15.28:0/2478567837 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f774c0a2d00 0x7f774c0a51c0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:51.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.594+0000 7f776b400640 1 --2- 172.21.15.28:0/2478567837 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7764109a20 0x7f776419ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:51.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.594+0000 7f776b400640 1 --2- 172.21.15.28:0/2478567837 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f77641043d0 0x7f776419a900 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:51.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.594+0000 7f776b400640 1 --2- 172.21.15.28:0/2478567837 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7764103a10 0x7f776419a3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:51.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.594+0000 7f776b400640 1 -- 172.21.15.28:0/2478567837 >> 172.21.15.28:0/2478567837 conn(0x7f77640ff4a0 msgr2=0x7f77640ffaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:16:51.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.594+0000 7f776b400640 1 -- 172.21.15.28:0/2478567837 shutdown_connections 2024-05-12T22:16:51.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.594+0000 7f776b400640 1 -- 172.21.15.28:0/2478567837 wait complete. 2024-05-12T22:16:51.612 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-05-12T22:16:51.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.828+0000 7fa2b3000640 1 -- 172.21.15.28:0/2851799711 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa2ac102090 msgr2=0x7fa2ac10e5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:16:51.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.828+0000 7fa2b3000640 1 --2- 172.21.15.28:0/2851799711 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa2ac102090 0x7fa2ac10e5a0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fa2a4009be0 tx=0x7fa2a402a6d0 comp rx=0 tx=0).stop 2024-05-12T22:16:51.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.828+0000 7fa2b3000640 1 -- 172.21.15.28:0/2851799711 shutdown_connections 2024-05-12T22:16:51.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.828+0000 7fa2b3000640 1 --2- 172.21.15.28:0/2851799711 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa2ac102090 0x7fa2ac10e5a0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:51.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.828+0000 7fa2b3000640 1 --2- 172.21.15.28:0/2851799711 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa2ac1016d0 0x7fa2ac101b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:51.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.828+0000 7fa2b3000640 1 --2- 172.21.15.28:0/2851799711 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa2ac1076e0 0x7fa2ac107ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:51.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.828+0000 7fa2b3000640 1 -- 172.21.15.28:0/2851799711 >> 172.21.15.28:0/2851799711 conn(0x7fa2ac0fd200 msgr2=0x7fa2ac0ff620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:16:51.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.829+0000 7fa2b3000640 1 -- 172.21.15.28:0/2851799711 shutdown_connections 2024-05-12T22:16:51.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.829+0000 7fa2b3000640 1 -- 172.21.15.28:0/2851799711 wait complete. 2024-05-12T22:16:51.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.830+0000 7fa2b3000640 1 Processor -- start 2024-05-12T22:16:51.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.830+0000 7fa2b3000640 1 -- start start 2024-05-12T22:16:51.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.830+0000 7fa2b3000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa2ac1016d0 0x7fa2ac19a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:51.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.830+0000 7fa2b3000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa2ac102090 0x7fa2ac19a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:51.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.830+0000 7fa2b3000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa2ac1076e0 0x7fa2ac19e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:51.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.830+0000 7fa2b3000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa2ac111af0 con 0x7fa2ac102090 2024-05-12T22:16:51.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.830+0000 7fa2b3000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fa2ac111970 con 0x7fa2ac1016d0 2024-05-12T22:16:51.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.830+0000 7fa2b3000640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa2ac111c70 con 0x7fa2ac1076e0 2024-05-12T22:16:51.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.831+0000 7fa2b1200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa2ac102090 0x7fa2ac19a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:51.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.831+0000 7fa2b1c00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa2ac1016d0 0x7fa2ac19a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:51.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.831+0000 7fa2b1200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa2ac102090 0x7fa2ac19a650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35520/0 (socket says 172.21.15.28:35520) 2024-05-12T22:16:51.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.831+0000 7fa2b1200640 1 -- 172.21.15.28:0/855973630 learned_addr learned my addr 172.21.15.28:0/855973630 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:16:51.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.831+0000 7fa2b2600640 1 --2- 172.21.15.28:0/855973630 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa2ac1076e0 0x7fa2ac19e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:51.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.831+0000 7fa2b1c00640 1 -- 172.21.15.28:0/855973630 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa2ac1076e0 msgr2=0x7fa2ac19e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:16:51.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.831+0000 7fa2b1c00640 1 --2- 172.21.15.28:0/855973630 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa2ac1076e0 0x7fa2ac19e9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:51.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.831+0000 7fa2b1c00640 1 -- 172.21.15.28:0/855973630 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa2ac102090 msgr2=0x7fa2ac19a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:16:51.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.831+0000 7fa2b1c00640 1 --2- 172.21.15.28:0/855973630 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa2ac102090 0x7fa2ac19a650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:51.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.831+0000 7fa2b1c00640 1 -- 172.21.15.28:0/855973630 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2ac19f100 con 0x7fa2ac1016d0 2024-05-12T22:16:51.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.831+0000 7fa2b2600640 1 --2- 172.21.15.28:0/855973630 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa2ac1076e0 0x7fa2ac19e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:16:51.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.831+0000 7fa2b1c00640 1 --2- 172.21.15.28:0/855973630 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa2ac1016d0 0x7fa2ac19a110 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fa2a000e9f0 tx=0x7fa2a000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:16:51.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.832+0000 7fa2aa000640 1 -- 172.21.15.28:0/855973630 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2a000ccc0 con 0x7fa2ac1016d0 2024-05-12T22:16:51.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.832+0000 7fa2b3000640 1 -- 172.21.15.28:0/855973630 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2ac19f3f0 con 0x7fa2ac1016d0 2024-05-12T22:16:51.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.832+0000 7fa2aa000640 1 -- 172.21.15.28:0/855973630 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa2a0005020 con 0x7fa2ac1016d0 2024-05-12T22:16:51.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.832+0000 7fa2aa000640 1 -- 172.21.15.28:0/855973630 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2a00049e0 con 0x7fa2ac1016d0 2024-05-12T22:16:51.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.832+0000 7fa2b3000640 1 -- 172.21.15.28:0/855973630 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2ac1a6c80 con 0x7fa2ac1016d0 2024-05-12T22:16:51.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.836+0000 7fa2b7381740 1 Processor -- start 2024-05-12T22:16:51.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.836+0000 7fa2b7381740 1 -- start start 2024-05-12T22:16:51.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.837+0000 7fa2aa000640 1 -- 172.21.15.28:0/855973630 <== mon.2 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7fa2a0005560 con 0x7fa2ac1016d0 2024-05-12T22:16:51.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.837+0000 7fa2b7381740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a6a5022b60 0x55a6a5022f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:51.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.837+0000 7fa2b7381740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a6a5023480 0x55a6a502d610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:51.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.837+0000 7fa2b7381740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a6a502db50 0x55a6a502ff40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:51.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.837+0000 7fa2b7381740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a6a4f87a30 con 0x55a6a5022b60 2024-05-12T22:16:51.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.837+0000 7fa2b7381740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x55a6a4f2f700 con 0x55a6a5023480 2024-05-12T22:16:51.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.837+0000 7fa2b7381740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55a6a4f839f0 con 0x55a6a502db50 2024-05-12T22:16:51.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.837+0000 7fa2b2600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a6a5022b60 0x55a6a5022f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:51.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.837+0000 7fa2b2600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a6a5022b60 0x55a6a5022f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35530/0 (socket says 172.21.15.28:35530) 2024-05-12T22:16:51.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.837+0000 7fa2b2600640 1 -- 172.21.15.28:0/791077026 learned_addr learned my addr 172.21.15.28:0/791077026 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:16:51.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.837+0000 7fa2b1c00640 1 --2- 172.21.15.28:0/791077026 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a6a5023480 0x55a6a502d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:51.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.837+0000 7fa2b1200640 1 --2- 172.21.15.28:0/791077026 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a6a502db50 0x55a6a502ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:51.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.838+0000 7fa2b2600640 1 -- 172.21.15.28:0/791077026 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a6a502db50 msgr2=0x55a6a502ff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:16:51.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.838+0000 7fa2b2600640 1 --2- 172.21.15.28:0/791077026 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a6a502db50 0x55a6a502ff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:51.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.838+0000 7fa2b2600640 1 -- 172.21.15.28:0/791077026 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a6a5023480 msgr2=0x55a6a502d610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:16:51.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.838+0000 7fa2b2600640 1 --2- 172.21.15.28:0/791077026 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a6a5023480 0x55a6a502d610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:51.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.838+0000 7fa2b2600640 1 -- 172.21.15.28:0/791077026 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a6a4c9b3f0 con 0x55a6a5022b60 2024-05-12T22:16:51.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.838+0000 7fa2b2600640 1 --2- 172.21.15.28:0/791077026 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a6a5022b60 0x55a6a5022f40 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fa2a402acc0 tx=0x7fa2a4030420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ed3289156cd8603 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:16:51.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.838+0000 7fa29ea00640 1 -- 172.21.15.28:0/791077026 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2a4040070 con 0x55a6a5022b60 2024-05-12T22:16:51.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.838+0000 7fa2aa000640 1 --2- 172.21.15.28:0/855973630 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fa2940a2d00 0x7fa2940a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:51.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.839+0000 7fa29ea00640 1 -- 172.21.15.28:0/791077026 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa2a4003970 con 0x55a6a5022b60 2024-05-12T22:16:51.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.839+0000 7fa29ea00640 1 -- 172.21.15.28:0/791077026 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2a4036040 con 0x55a6a5022b60 2024-05-12T22:16:51.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.839+0000 7fa2aa000640 1 -- 172.21.15.28:0/855973630 <== mon.2 v2:172.21.15.86:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8513+0+0 (secure 0 0 0) 0x7fa2a00cf430 con 0x7fa2ac1016d0 2024-05-12T22:16:51.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.839+0000 7fa2b1c00640 1 --2- 172.21.15.28:0/855973630 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fa2940a2d00 0x7fa2940a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:51.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.839+0000 7fa2b7381740 1 -- 172.21.15.28:0/791077026 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a6a5022b60 msgr2=0x55a6a5022f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:16:51.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.839+0000 7fa2b7381740 1 --2- 172.21.15.28:0/791077026 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a6a5022b60 0x55a6a5022f40 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fa2a402acc0 tx=0x7fa2a4030420 comp rx=0 tx=0).stop 2024-05-12T22:16:51.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.839+0000 7fa2b7381740 1 -- 172.21.15.28:0/791077026 shutdown_connections 2024-05-12T22:16:51.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.839+0000 7fa2b7381740 1 --2- 172.21.15.28:0/791077026 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a6a502db50 0x55a6a502ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:51.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.839+0000 7fa2b7381740 1 --2- 172.21.15.28:0/791077026 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a6a5023480 0x55a6a502d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:51.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.839+0000 7fa2b7381740 1 --2- 172.21.15.28:0/791077026 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a6a5022b60 0x55a6a5022f40 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:51.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.839+0000 7fa2b7381740 1 -- 172.21.15.28:0/791077026 >> 172.21.15.28:0/791077026 conn(0x55a6a501ee80 msgr2=0x55a6a501f260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:16:51.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.840+0000 7fa2b1c00640 1 --2- 172.21.15.28:0/855973630 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fa2940a2d00 0x7fa2940a51c0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fa2a00d2fd0 tx=0x7fa2a00d2ee0 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:16:51.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.840+0000 7fa2b7381740 1 -- 172.21.15.28:0/791077026 shutdown_connections 2024-05-12T22:16:51.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.840+0000 7fa2b7381740 1 -- 172.21.15.28:0/791077026 wait complete. 2024-05-12T22:16:51.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.840+0000 7fa2b7381740 1 -- start start 2024-05-12T22:16:51.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.840+0000 7fa2b7381740 1 Processor -- start 2024-05-12T22:16:51.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.841+0000 7fa2b7381740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a6a5022b60 0x55a6a502ab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:51.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.841+0000 7fa2b7381740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a6a5023480 0x55a6a502b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:51.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.841+0000 7fa2b7381740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a6a502db50 0x55a6a502b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:51.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.841+0000 7fa2b7381740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a6a50308e0 con 0x55a6a502db50 2024-05-12T22:16:51.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.841+0000 7fa2b7381740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x55a6a4f42a70 con 0x55a6a5022b60 2024-05-12T22:16:51.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.841+0000 7fa2b7381740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55a6a5030a60 con 0x55a6a5023480 2024-05-12T22:16:51.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.841+0000 7fa2b7381740 10 client.0 _ll_register_callbacks cb 0x55a6a4f25ba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:16:51.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.841+0000 7fa2b2600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a6a5022b60 0x55a6a502ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:51.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.841+0000 7fa2b1200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a6a5023480 0x55a6a502b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:51.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.841+0000 7fa2b1200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a6a5023480 0x55a6a502b080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:59338/0 (socket says 172.21.15.28:59338) 2024-05-12T22:16:51.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.841+0000 7fa2b1200640 1 -- 172.21.15.28:0/1766480167 learned_addr learned my addr 172.21.15.28:0/1766480167 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:16:51.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.841+0000 7fa2b2600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a6a5022b60 0x55a6a502ab40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:49230/0 (socket says 172.21.15.28:49230) 2024-05-12T22:16:51.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.841+0000 7fa2b2600640 1 --2- 172.21.15.28:0/1766480167 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a6a502db50 0x55a6a502b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:51.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.842+0000 7fa2b1200640 1 -- 172.21.15.28:0/1766480167 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a6a5022b60 msgr2=0x55a6a502ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:16:51.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.842+0000 7fa2b1200640 1 --2- 172.21.15.28:0/1766480167 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a6a5022b60 0x55a6a502ab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:51.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.842+0000 7fa2b1200640 1 -- 172.21.15.28:0/1766480167 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a6a502db50 msgr2=0x55a6a502b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:16:51.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.842+0000 7fa2b1200640 1 --2- 172.21.15.28:0/1766480167 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a6a502db50 0x55a6a502b5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:51.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.842+0000 7fa2b1200640 1 -- 172.21.15.28:0/1766480167 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a6a4d657d0 con 0x55a6a5023480 2024-05-12T22:16:51.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.842+0000 7fa2b1200640 1 --2- 172.21.15.28:0/1766480167 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a6a5023480 0x55a6a502b080 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fa298011830 tx=0x7fa298011cf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ea8c7a1f2f1cecaf server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:16:51.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.842+0000 7fa293400640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-12T22:16:51.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.842+0000 7fa293400640 1 -- 172.21.15.28:0/1766480167 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa298004240 con 0x55a6a5023480 2024-05-12T22:16:51.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.843+0000 7fa2b7381740 1 -- 172.21.15.28:0/1766480167 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa2a4003970 con 0x55a6a5023480 2024-05-12T22:16:51.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.843+0000 7fa293400640 1 -- 172.21.15.28:0/1766480167 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8513+0+0 (secure 0 0 0) 0x7fa2980043e0 con 0x55a6a5023480 2024-05-12T22:16:51.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.843+0000 7fa293400640 1 -- 172.21.15.28:0/1766480167 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fa29800f920 con 0x55a6a5023480 2024-05-12T22:16:51.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.843+0000 7fa2b7381740 10 client.26920 fetch_fsmap learned FSMap version 26 2024-05-12T22:16:51.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.843+0000 7fa2b7381740 1 -- 172.21.15.28:0/1766480167 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55a6a503cc90 con 0x55a6a5023480 2024-05-12T22:16:51.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.843+0000 7fa293400640 1 -- 172.21.15.28:0/1766480167 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa2a4003970 con 0x55a6a5023480 2024-05-12T22:16:51.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.843+0000 7fa293400640 1 -- 172.21.15.28:0/1766480167 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fa298019030 con 0x55a6a5023480 2024-05-12T22:16:51.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.844+0000 7fa2b7381740 10 client.26920 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:16:51.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.844+0000 7fa2b7381740 10 client.26920 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-05-12T22:16:51.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.844+0000 7fa2b7381740 20 client.26920 populate_metadata read hostname 'smithi028' 2024-05-12T22:16:51.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.844+0000 7fa2b7381740 1 --2- 172.21.15.28:0/1766480167 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x55a6a503e160 0x55a6a50405a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:51.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.844+0000 7fa2b7381740 4 client.26920 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-12T22:16:51.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.844+0000 7fa2b7381740 1 -- 172.21.15.28:0/1766480167 --> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a6a4f95160 con 0x55a6a503e160 2024-05-12T22:16:51.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.844+0000 7fa2b2600640 1 --2- 172.21.15.28:0/1766480167 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x55a6a503e160 0x55a6a50405a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:51.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.844+0000 7fa293400640 10 client.26920 ms_handle_connect on v2:172.21.15.28:6840/761183870 2024-05-12T22:16:51.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.844+0000 7fa2b2600640 1 --2- 172.21.15.28:0/1766480167 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x55a6a503e160 0x55a6a50405a0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=83df6b5b72872dc3 server_cookie=d48563b230969395 in_seq=0 out_seq=0 2024-05-12T22:16:51.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.845+0000 7fa293400640 1 -- 172.21.15.28:0/1766480167 <== mds.1 v2:172.21.15.28:6840/761183870 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a6a4f95160 con 0x55a6a503e160 2024-05-12T22:16:51.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.845+0000 7fa293400640 10 client.26920 handle_command_reply: tid=0 2024-05-12T22:16:51.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.845+0000 7fa2b7381740 1 client.26920 shutdown 2024-05-12T22:16:51.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.846+0000 7fa2b7381740 1 -- 172.21.15.28:0/1766480167 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a6a5023480 msgr2=0x55a6a502b080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:16:51.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.846+0000 7fa2b7381740 1 --2- 172.21.15.28:0/1766480167 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a6a5023480 0x55a6a502b080 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fa298011830 tx=0x7fa298011cf0 comp rx=0 tx=0).stop 2024-05-12T22:16:51.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.846+0000 7fa2b7381740 1 -- 172.21.15.28:0/1766480167 shutdown_connections 2024-05-12T22:16:51.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.846+0000 7fa2b7381740 1 --2- 172.21.15.28:0/1766480167 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x55a6a503e160 0x55a6a50405a0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:51.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.846+0000 7fa2b7381740 1 --2- 172.21.15.28:0/1766480167 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a6a502db50 0x55a6a502b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:51.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.846+0000 7fa293400640 0 client.26920 ms_handle_reset on v2:172.21.15.28:6840/761183870 2024-05-12T22:16:51.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.846+0000 7fa2b7381740 1 --2- 172.21.15.28:0/1766480167 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a6a5023480 0x55a6a502b080 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:51.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.846+0000 7fa2b7381740 1 --2- 172.21.15.28:0/1766480167 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a6a5022b60 0x55a6a502ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:51.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.846+0000 7fa2b7381740 1 -- 172.21.15.28:0/1766480167 >> 172.21.15.28:0/1766480167 conn(0x55a6a501ee80 msgr2=0x55a6a5024c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:16:51.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.846+0000 7fa2b7381740 1 -- 172.21.15.28:0/1766480167 shutdown_connections 2024-05-12T22:16:51.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.846+0000 7fa2b7381740 1 -- 172.21.15.28:0/1766480167 wait complete. 2024-05-12T22:16:51.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.846+0000 7fa2b7381740 20 client.26920 trim_cache size 0 max 16384 2024-05-12T22:16:51.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.861+0000 7fa2b7381740 1 Processor -- start 2024-05-12T22:16:51.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.861+0000 7fa2b7381740 1 -- start start 2024-05-12T22:16:51.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.861+0000 7fa2b7381740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a6a5022b60 0x55a6a502b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:51.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.861+0000 7fa2b7381740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a6a5023480 0x55a6a5045e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:51.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.861+0000 7fa2b7381740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a6a502db50 0x55a6a50463b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:51.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.861+0000 7fa2b7381740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa2ac1b1e00 con 0x55a6a5023480 2024-05-12T22:16:51.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.861+0000 7fa2b7381740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fa2ac1af6d0 con 0x55a6a5022b60 2024-05-12T22:16:51.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.861+0000 7fa2b7381740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa2ac1ad540 con 0x55a6a502db50 2024-05-12T22:16:51.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.861+0000 7fa2b1200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a6a5023480 0x55a6a5045e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:51.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.861+0000 7fa2b1200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a6a5023480 0x55a6a5045e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35552/0 (socket says 172.21.15.28:35552) 2024-05-12T22:16:51.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.862+0000 7fa2b1200640 1 -- 172.21.15.28:0/865631914 learned_addr learned my addr 172.21.15.28:0/865631914 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:16:51.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.862+0000 7fa2b2600640 1 --2- 172.21.15.28:0/865631914 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a6a5022b60 0x55a6a502b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:51.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.862+0000 7fa2b2600640 1 --2- 172.21.15.28:0/865631914 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a6a502db50 0x55a6a50463b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:51.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.862+0000 7fa2b1200640 1 -- 172.21.15.28:0/865631914 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a6a502db50 msgr2=0x55a6a50463b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:16:51.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.862+0000 7fa2b1200640 1 --2- 172.21.15.28:0/865631914 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a6a502db50 0x55a6a50463b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:51.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.862+0000 7fa2b1200640 1 -- 172.21.15.28:0/865631914 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a6a5022b60 msgr2=0x55a6a502b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:16:51.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.862+0000 7fa2b1200640 1 --2- 172.21.15.28:0/865631914 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a6a5022b60 0x55a6a502b3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:51.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.862+0000 7fa2b1200640 1 -- 172.21.15.28:0/865631914 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a6a50232b0 con 0x55a6a5023480 2024-05-12T22:16:51.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.862+0000 7fa2b2600640 1 --2- 172.21.15.28:0/865631914 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a6a502db50 0x55a6a50463b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-12T22:16:51.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.862+0000 7fa2b1200640 1 --2- 172.21.15.28:0/865631914 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a6a5023480 0x55a6a5045e70 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fa298023620 tx=0x7fa298009e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bf8cbb48629f6ac server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:16:51.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.862+0000 7fa29e000640 1 -- 172.21.15.28:0/865631914 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa29800fac0 con 0x55a6a5023480 2024-05-12T22:16:51.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.863+0000 7fa29e000640 1 -- 172.21.15.28:0/865631914 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa29800fc60 con 0x55a6a5023480 2024-05-12T22:16:51.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.863+0000 7fa29e000640 1 -- 172.21.15.28:0/865631914 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2980035f0 con 0x55a6a5023480 2024-05-12T22:16:51.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.863+0000 7fa2b7381740 1 -- 172.21.15.28:0/865631914 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a6a5023480 msgr2=0x55a6a5045e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:16:51.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.863+0000 7fa2b7381740 1 --2- 172.21.15.28:0/865631914 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a6a5023480 0x55a6a5045e70 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fa298023620 tx=0x7fa298009e70 comp rx=0 tx=0).stop 2024-05-12T22:16:51.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.863+0000 7fa2b7381740 1 -- 172.21.15.28:0/865631914 shutdown_connections 2024-05-12T22:16:51.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.863+0000 7fa2b7381740 1 --2- 172.21.15.28:0/865631914 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a6a502db50 0x55a6a50463b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:51.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.863+0000 7fa2b7381740 1 --2- 172.21.15.28:0/865631914 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a6a5023480 0x55a6a5045e70 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:51.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.863+0000 7fa2b7381740 1 --2- 172.21.15.28:0/865631914 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a6a5022b60 0x55a6a502b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:51.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.863+0000 7fa2b7381740 1 -- 172.21.15.28:0/865631914 >> 172.21.15.28:0/865631914 conn(0x55a6a501ee80 msgr2=0x55a6a501d770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:16:51.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.863+0000 7fa2b7381740 1 -- 172.21.15.28:0/865631914 shutdown_connections 2024-05-12T22:16:51.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.863+0000 7fa2b7381740 1 -- 172.21.15.28:0/865631914 wait complete. 2024-05-12T22:16:51.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.864+0000 7fa2b7381740 1 -- start start 2024-05-12T22:16:51.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.864+0000 7fa2b7381740 1 Processor -- start 2024-05-12T22:16:51.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.864+0000 7fa2b7381740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a6a5022b60 0x55a6a5031b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:51.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.864+0000 7fa2b7381740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a6a5023480 0x55a6a5032080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:51.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.864+0000 7fa2b7381740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a6a502db50 0x55a6a50a36e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:51.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.865+0000 7fa2b7381740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa2ac1adb90 con 0x55a6a502db50 2024-05-12T22:16:51.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.865+0000 7fa2b7381740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fa2ac1af000 con 0x55a6a5023480 2024-05-12T22:16:51.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.865+0000 7fa2b7381740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa2ac1ae260 con 0x55a6a5022b60 2024-05-12T22:16:51.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.865+0000 7fa2b7381740 10 client.0 _ll_register_callbacks cb 0x7fa298011440 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:16:51.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.865+0000 7fa2b2600640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a6a5022b60 0x55a6a5031b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:51.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.865+0000 7fa2b1200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a6a5023480 0x55a6a5032080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:51.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.865+0000 7fa2b2600640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a6a5022b60 0x55a6a5031b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:59358/0 (socket says 172.21.15.28:59358) 2024-05-12T22:16:51.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.865+0000 7fa2b2600640 1 -- 172.21.15.28:0/2664995061 learned_addr learned my addr 172.21.15.28:0/2664995061 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:16:51.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.865+0000 7fa2b2600640 1 --2- 172.21.15.28:0/2664995061 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a6a502db50 0x55a6a50a36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:51.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.865+0000 7fa2b1200640 1 -- 172.21.15.28:0/2664995061 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a6a5022b60 msgr2=0x55a6a5031b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:16:51.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.865+0000 7fa2b1200640 1 --2- 172.21.15.28:0/2664995061 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a6a5022b60 0x55a6a5031b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:51.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.865+0000 7fa2b1200640 1 -- 172.21.15.28:0/2664995061 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a6a502db50 msgr2=0x55a6a50a36e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:16:51.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.865+0000 7fa2b1200640 1 --2- 172.21.15.28:0/2664995061 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a6a502db50 0x55a6a50a36e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:51.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.865+0000 7fa2b1200640 1 -- 172.21.15.28:0/2664995061 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a6a50a3c20 con 0x55a6a5023480 2024-05-12T22:16:51.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.865+0000 7fa2b2600640 1 --2- 172.21.15.28:0/2664995061 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a6a5022b60 0x55a6a5031b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:16:51.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.866+0000 7fa2b1200640 1 --2- 172.21.15.28:0/2664995061 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a6a5023480 0x55a6a5032080 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x55a6a5024160 tx=0x7fa298023510 comp rx=0 tx=0).ready entity=mon.2 client_cookie=19c731bd6208b55f server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:16:51.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.866+0000 7fa293400640 10 client.0 ms_handle_connect on v2:172.21.15.86:3300/0 2024-05-12T22:16:51.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.866+0000 7fa293400640 1 -- 172.21.15.28:0/2664995061 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa298039020 con 0x55a6a5023480 2024-05-12T22:16:51.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.866+0000 7fa293400640 1 -- 172.21.15.28:0/2664995061 <== mon.2 v2:172.21.15.86:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8513+0+0 (secure 0 0 0) 0x7fa2980052c0 con 0x55a6a5023480 2024-05-12T22:16:51.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.866+0000 7fa2b7381740 1 -- 172.21.15.28:0/2664995061 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa29800fc60 con 0x55a6a5023480 2024-05-12T22:16:51.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.866+0000 7fa293400640 1 -- 172.21.15.28:0/2664995061 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fa298023650 con 0x55a6a5023480 2024-05-12T22:16:51.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.866+0000 7fa2b7381740 10 client.26969 fetch_fsmap learned FSMap version 26 2024-05-12T22:16:51.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.866+0000 7fa2b7381740 1 -- 172.21.15.28:0/2664995061 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55a6a50a4d50 con 0x55a6a5023480 2024-05-12T22:16:51.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.867+0000 7fa293400640 1 -- 172.21.15.28:0/2664995061 <== mon.2 v2:172.21.15.86:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa29800fc60 con 0x55a6a5023480 2024-05-12T22:16:51.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.867+0000 7fa293400640 1 -- 172.21.15.28:0/2664995061 <== mon.2 v2:172.21.15.86:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fa298023ce0 con 0x55a6a5023480 2024-05-12T22:16:51.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.867+0000 7fa2b7381740 10 client.26969 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:16:51.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.867+0000 7fa2b7381740 10 client.26969 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-05-12T22:16:51.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.867+0000 7fa2b7381740 20 client.26969 populate_metadata read hostname 'smithi028' 2024-05-12T22:16:51.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.867+0000 7fa2b7381740 1 --2- 172.21.15.28:0/2664995061 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x55a6a50a5350 0x55a6a50a7740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:51.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.867+0000 7fa2b7381740 4 client.26969 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-12T22:16:51.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.867+0000 7fa2b7381740 1 -- 172.21.15.28:0/2664995061 --> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a6a4f8f4b0 con 0x55a6a50a5350 2024-05-12T22:16:51.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.867+0000 7fa2b2600640 1 --2- 172.21.15.28:0/2664995061 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x55a6a50a5350 0x55a6a50a7740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:51.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.868+0000 7fa2b2600640 1 --2- 172.21.15.28:0/2664995061 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x55a6a50a5350 0x55a6a50a7740 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=58ba3141ed0864b0 server_cookie=4b5db865be7491be in_seq=0 out_seq=0 2024-05-12T22:16:51.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.868+0000 7fa293400640 10 client.26969 ms_handle_connect on v2:172.21.15.28:6840/761183870 2024-05-12T22:16:51.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.868+0000 7fa293400640 1 -- 172.21.15.28:0/2664995061 <== mds.1 v2:172.21.15.28:6840/761183870 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a6a4f8f4b0 con 0x55a6a50a5350 2024-05-12T22:16:51.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.868+0000 7fa293400640 10 client.26969 handle_command_reply: tid=0 2024-05-12T22:16:51.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.868+0000 7fa2b7381740 1 client.26969 shutdown 2024-05-12T22:16:51.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.868+0000 7fa2b7381740 1 -- 172.21.15.28:0/2664995061 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a6a5023480 msgr2=0x55a6a5032080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:16:51.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.868+0000 7fa2b7381740 1 --2- 172.21.15.28:0/2664995061 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a6a5023480 0x55a6a5032080 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x55a6a5024160 tx=0x7fa298023510 comp rx=0 tx=0).stop 2024-05-12T22:16:51.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.868+0000 7fa2b7381740 1 -- 172.21.15.28:0/2664995061 shutdown_connections 2024-05-12T22:16:51.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.868+0000 7fa2b7381740 1 --2- 172.21.15.28:0/2664995061 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x55a6a50a5350 0x55a6a50a7740 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:51.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.868+0000 7fa2b7381740 1 --2- 172.21.15.28:0/2664995061 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a6a502db50 0x55a6a50a36e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:51.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.868+0000 7fa2b7381740 1 --2- 172.21.15.28:0/2664995061 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a6a5023480 0x55a6a5032080 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:51.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.868+0000 7fa293400640 0 client.26969 ms_handle_reset on v2:172.21.15.28:6840/761183870 2024-05-12T22:16:51.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.868+0000 7fa2b7381740 1 --2- 172.21.15.28:0/2664995061 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a6a5022b60 0x55a6a5031b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:51.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.868+0000 7fa2b7381740 1 -- 172.21.15.28:0/2664995061 >> 172.21.15.28:0/2664995061 conn(0x55a6a501ee80 msgr2=0x55a6a501e000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:16:51.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.869+0000 7fa2b7381740 1 -- 172.21.15.28:0/2664995061 shutdown_connections 2024-05-12T22:16:51.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.869+0000 7fa2b7381740 1 -- 172.21.15.28:0/2664995061 wait complete. 2024-05-12T22:16:51.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.869+0000 7fa2b7381740 20 client.26969 trim_cache size 0 max 16384 2024-05-12T22:16:51.888 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-05-12T22:16:51.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.869+0000 7fa2b3000640 1 -- 172.21.15.28:0/855973630 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fa2940a2d00 msgr2=0x7fa2940a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:16:51.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.869+0000 7fa2b3000640 1 --2- 172.21.15.28:0/855973630 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fa2940a2d00 0x7fa2940a51c0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fa2a00d2fd0 tx=0x7fa2a00d2ee0 comp rx=0 tx=0).stop 2024-05-12T22:16:51.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.869+0000 7fa2b3000640 1 -- 172.21.15.28:0/855973630 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa2ac1016d0 msgr2=0x7fa2ac19a110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:16:51.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.869+0000 7fa2b3000640 1 --2- 172.21.15.28:0/855973630 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa2ac1016d0 0x7fa2ac19a110 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fa2a000e9f0 tx=0x7fa2a000eeb0 comp rx=0 tx=0).stop 2024-05-12T22:16:51.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.870+0000 7fa2b3000640 1 -- 172.21.15.28:0/855973630 shutdown_connections 2024-05-12T22:16:51.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.870+0000 7fa2b3000640 1 --2- 172.21.15.28:0/855973630 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fa2940a2d00 0x7fa2940a51c0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:51.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.870+0000 7fa2b3000640 1 --2- 172.21.15.28:0/855973630 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa2ac1076e0 0x7fa2ac19e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:51.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.870+0000 7fa2b3000640 1 --2- 172.21.15.28:0/855973630 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa2ac102090 0x7fa2ac19a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:51.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.870+0000 7fa2b3000640 1 --2- 172.21.15.28:0/855973630 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa2ac1016d0 0x7fa2ac19a110 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:51.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.870+0000 7fa2b3000640 1 -- 172.21.15.28:0/855973630 >> 172.21.15.28:0/855973630 conn(0x7fa2ac0fd200 msgr2=0x7fa2ac10ce80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:16:51.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.870+0000 7fa2b3000640 1 -- 172.21.15.28:0/855973630 shutdown_connections 2024-05-12T22:16:51.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:51.870+0000 7fa2b3000640 1 -- 172.21.15.28:0/855973630 wait complete. 2024-05-12T22:16:51.894 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-05-12T22:16:52.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.115+0000 7f6690a00640 1 -- 172.21.15.28:0/2979405642 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f668c102070 msgr2=0x7f668c10e580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:16:52.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.115+0000 7f6690a00640 1 --2- 172.21.15.28:0/2979405642 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f668c102070 0x7f668c10e580 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f6684009cd0 tx=0x7f668402a8f0 comp rx=0 tx=0).stop 2024-05-12T22:16:52.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.115+0000 7f6690a00640 1 -- 172.21.15.28:0/2979405642 shutdown_connections 2024-05-12T22:16:52.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.115+0000 7f6690a00640 1 --2- 172.21.15.28:0/2979405642 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f668c102070 0x7f668c10e580 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:52.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.115+0000 7f6690a00640 1 --2- 172.21.15.28:0/2979405642 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f668c1016b0 0x7f668c101b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:52.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.115+0000 7f6690a00640 1 --2- 172.21.15.28:0/2979405642 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f668c1076c0 0x7f668c107aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:52.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.115+0000 7f6690a00640 1 -- 172.21.15.28:0/2979405642 >> 172.21.15.28:0/2979405642 conn(0x7f668c0fd200 msgr2=0x7f668c0ff620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:16:52.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.115+0000 7f6690a00640 1 -- 172.21.15.28:0/2979405642 shutdown_connections 2024-05-12T22:16:52.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.115+0000 7f6690a00640 1 -- 172.21.15.28:0/2979405642 wait complete. 2024-05-12T22:16:52.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.116+0000 7f6690a00640 1 Processor -- start 2024-05-12T22:16:52.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.117+0000 7f6690a00640 1 -- start start 2024-05-12T22:16:52.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.117+0000 7f6690a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f668c1016b0 0x7f668c19a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:52.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.117+0000 7f6690a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f668c102070 0x7f668c19a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:52.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.117+0000 7f6690a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f668c1076c0 0x7f668c19ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:52.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.117+0000 7f6690a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f668c111ae0 con 0x7f668c1076c0 2024-05-12T22:16:52.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.117+0000 7f6690a00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f668c111960 con 0x7f668c102070 2024-05-12T22:16:52.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.117+0000 7f6690a00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f668c111c60 con 0x7f668c1016b0 2024-05-12T22:16:52.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.117+0000 7f668be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f668c1076c0 0x7f668c19ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:52.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.117+0000 7f668b400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f668c1016b0 0x7f668c19a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:52.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.117+0000 7f668b400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f668c1016b0 0x7f668c19a1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:59366/0 (socket says 172.21.15.28:59366) 2024-05-12T22:16:52.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.117+0000 7f668b400640 1 -- 172.21.15.28:0/3878986519 learned_addr learned my addr 172.21.15.28:0/3878986519 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:16:52.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.117+0000 7f6683e00640 1 --2- 172.21.15.28:0/3878986519 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f668c102070 0x7f668c19a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:52.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.118+0000 7f668b400640 1 -- 172.21.15.28:0/3878986519 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f668c102070 msgr2=0x7f668c19a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:16:52.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.118+0000 7f668b400640 1 --2- 172.21.15.28:0/3878986519 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f668c102070 0x7f668c19a6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:52.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.118+0000 7f668b400640 1 -- 172.21.15.28:0/3878986519 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f668c1076c0 msgr2=0x7f668c19ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:16:52.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.118+0000 7f668b400640 1 --2- 172.21.15.28:0/3878986519 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f668c1076c0 0x7f668c19ea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:52.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.118+0000 7f668b400640 1 -- 172.21.15.28:0/3878986519 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f668c19f190 con 0x7f668c1016b0 2024-05-12T22:16:52.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.118+0000 7f668b400640 1 --2- 172.21.15.28:0/3878986519 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f668c1016b0 0x7f668c19a1a0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f667c00e9f0 tx=0x7f667c00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:16:52.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.118+0000 7f6688c00640 1 -- 172.21.15.28:0/3878986519 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f667c00cd80 con 0x7f668c1016b0 2024-05-12T22:16:52.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.118+0000 7f6688c00640 1 -- 172.21.15.28:0/3878986519 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f667c010910 con 0x7f668c1016b0 2024-05-12T22:16:52.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.118+0000 7f6690a00640 1 -- 172.21.15.28:0/3878986519 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f668c19f480 con 0x7f668c1016b0 2024-05-12T22:16:52.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.118+0000 7f6688c00640 1 -- 172.21.15.28:0/3878986519 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f667c0188f0 con 0x7f668c1016b0 2024-05-12T22:16:52.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.119+0000 7f6690a00640 1 -- 172.21.15.28:0/3878986519 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f668c1a6d10 con 0x7f668c1016b0 2024-05-12T22:16:52.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.122+0000 7f6694e15740 1 Processor -- start 2024-05-12T22:16:52.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.122+0000 7f6694e15740 1 -- start start 2024-05-12T22:16:52.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.122+0000 7f6688c00640 1 -- 172.21.15.28:0/3878986519 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7f667c010ac0 con 0x7f668c1016b0 2024-05-12T22:16:52.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.122+0000 7f6694e15740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558557e1eb60 0x558557e1ef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:52.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.122+0000 7f6694e15740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558557e1f480 0x558557e29610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:52.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.122+0000 7f6694e15740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558557e29b50 0x558557e2bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:52.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.122+0000 7f6694e15740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558557d83a30 con 0x558557e1eb60 2024-05-12T22:16:52.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.122+0000 7f6694e15740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x558557d2b700 con 0x558557e1f480 2024-05-12T22:16:52.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.122+0000 7f6694e15740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x558557d7f9f0 con 0x558557e29b50 2024-05-12T22:16:52.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.123+0000 7f668b400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558557e1f480 0x558557e29610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:52.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.123+0000 7f668be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558557e1eb60 0x558557e1ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:52.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.123+0000 7f668be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558557e1eb60 0x558557e1ef40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35584/0 (socket says 172.21.15.28:35584) 2024-05-12T22:16:52.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.123+0000 7f668b400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558557e1f480 0x558557e29610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:49278/0 (socket says 172.21.15.28:49278) 2024-05-12T22:16:52.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.123+0000 7f668be00640 1 -- 172.21.15.28:0/2005998317 learned_addr learned my addr 172.21.15.28:0/2005998317 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:16:52.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.123+0000 7f6683e00640 1 --2- 172.21.15.28:0/2005998317 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558557e29b50 0x558557e2bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:52.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.123+0000 7f668be00640 1 -- 172.21.15.28:0/2005998317 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558557e29b50 msgr2=0x558557e2bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:16:52.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.123+0000 7f668be00640 1 --2- 172.21.15.28:0/2005998317 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558557e29b50 0x558557e2bf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:52.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.123+0000 7f668be00640 1 -- 172.21.15.28:0/2005998317 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558557e1f480 msgr2=0x558557e29610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:16:52.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.123+0000 7f668be00640 1 --2- 172.21.15.28:0/2005998317 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558557e1f480 0x558557e29610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:52.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.123+0000 7f668be00640 1 -- 172.21.15.28:0/2005998317 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558557a973f0 con 0x558557e1eb60 2024-05-12T22:16:52.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.123+0000 7f668be00640 1 --2- 172.21.15.28:0/2005998317 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558557e1eb60 0x558557e1ef40 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f6684009cd0 tx=0x7f6684005f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9d041a1e7cc3e54 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:16:52.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.124+0000 7f6688c00640 1 --2- 172.21.15.28:0/3878986519 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f66700a2d80 0x7f66700a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:52.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.124+0000 7f667be00640 1 -- 172.21.15.28:0/2005998317 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6684030760 con 0x558557e1eb60 2024-05-12T22:16:52.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.124+0000 7f667be00640 1 -- 172.21.15.28:0/2005998317 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6684036030 con 0x558557e1eb60 2024-05-12T22:16:52.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.124+0000 7f667be00640 1 -- 172.21.15.28:0/2005998317 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66840043c0 con 0x558557e1eb60 2024-05-12T22:16:52.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.124+0000 7f6688c00640 1 -- 172.21.15.28:0/3878986519 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8513+0+0 (secure 0 0 0) 0x7f667c0cf810 con 0x7f668c1016b0 2024-05-12T22:16:52.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.124+0000 7f668b400640 1 --2- 172.21.15.28:0/3878986519 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f66700a2d80 0x7f66700a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:52.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.125+0000 7f6694e15740 1 -- 172.21.15.28:0/2005998317 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558557e1eb60 msgr2=0x558557e1ef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:16:52.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.125+0000 7f6694e15740 1 --2- 172.21.15.28:0/2005998317 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558557e1eb60 0x558557e1ef40 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f6684009cd0 tx=0x7f6684005f80 comp rx=0 tx=0).stop 2024-05-12T22:16:52.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.125+0000 7f668b400640 1 --2- 172.21.15.28:0/3878986519 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f66700a2d80 0x7f66700a5240 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x558557e20120 tx=0x7f667c0d5530 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:16:52.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.125+0000 7f6694e15740 1 -- 172.21.15.28:0/2005998317 shutdown_connections 2024-05-12T22:16:52.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.125+0000 7f6694e15740 1 --2- 172.21.15.28:0/2005998317 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558557e29b50 0x558557e2bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:52.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.125+0000 7f6694e15740 1 --2- 172.21.15.28:0/2005998317 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558557e1f480 0x558557e29610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:52.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.125+0000 7f6694e15740 1 --2- 172.21.15.28:0/2005998317 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558557e1eb60 0x558557e1ef40 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:52.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.125+0000 7f6694e15740 1 -- 172.21.15.28:0/2005998317 >> 172.21.15.28:0/2005998317 conn(0x558557e1ae80 msgr2=0x558557e1b260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:16:52.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.125+0000 7f6694e15740 1 -- 172.21.15.28:0/2005998317 shutdown_connections 2024-05-12T22:16:52.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.125+0000 7f6694e15740 1 -- 172.21.15.28:0/2005998317 wait complete. 2024-05-12T22:16:52.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.125+0000 7f6694e15740 1 -- start start 2024-05-12T22:16:52.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.126+0000 7f6694e15740 1 Processor -- start 2024-05-12T22:16:52.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.126+0000 7f6694e15740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558557e1eb60 0x558557e26b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:52.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.126+0000 7f6694e15740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558557e1f480 0x558557e27090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:52.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.126+0000 7f6694e15740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558557e29b50 0x558557e275d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:52.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.126+0000 7f6694e15740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558557e2c8e0 con 0x558557e1eb60 2024-05-12T22:16:52.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.126+0000 7f6694e15740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x558557d3ea70 con 0x558557e29b50 2024-05-12T22:16:52.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.126+0000 7f6694e15740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x558557e2ca60 con 0x558557e1f480 2024-05-12T22:16:52.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.126+0000 7f6694e15740 10 client.0 _ll_register_callbacks cb 0x558557d21ba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:16:52.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.126+0000 7f668be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558557e1eb60 0x558557e26b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:52.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.126+0000 7f668be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558557e1eb60 0x558557e26b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35600/0 (socket says 172.21.15.28:35600) 2024-05-12T22:16:52.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.126+0000 7f668be00640 1 -- 172.21.15.28:0/2374789352 learned_addr learned my addr 172.21.15.28:0/2374789352 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:16:52.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.127+0000 7f6683e00640 1 --2- 172.21.15.28:0/2374789352 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558557e1f480 0x558557e27090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:52.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.127+0000 7f668be00640 1 --2- 172.21.15.28:0/2374789352 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558557e29b50 0x558557e275d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:52.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.127+0000 7f668be00640 1 -- 172.21.15.28:0/2374789352 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558557e1f480 msgr2=0x558557e27090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:16:52.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.127+0000 7f668be00640 1 --2- 172.21.15.28:0/2374789352 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558557e1f480 0x558557e27090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:52.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.127+0000 7f668be00640 1 -- 172.21.15.28:0/2374789352 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558557e29b50 msgr2=0x558557e275d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:16:52.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.127+0000 7f668be00640 1 --2- 172.21.15.28:0/2374789352 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558557e29b50 0x558557e275d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:52.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.127+0000 7f668be00640 1 -- 172.21.15.28:0/2374789352 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558557b617d0 con 0x558557e1eb60 2024-05-12T22:16:52.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.127+0000 7f668be00640 1 --2- 172.21.15.28:0/2374789352 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558557e1eb60 0x558557e26b50 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f6684009bf0 tx=0x7f6684030fc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcb5dd821cefa494 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:16:52.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.127+0000 7f6678c00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-12T22:16:52.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.127+0000 7f6678c00640 1 -- 172.21.15.28:0/2374789352 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6684058060 con 0x558557e1eb60 2024-05-12T22:16:52.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.127+0000 7f6678c00640 1 -- 172.21.15.28:0/2374789352 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8513+0+0 (secure 0 0 0) 0x7f6684003b80 con 0x558557e1eb60 2024-05-12T22:16:52.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.127+0000 7f6694e15740 1 -- 172.21.15.28:0/2374789352 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6684036030 con 0x558557e1eb60 2024-05-12T22:16:52.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.128+0000 7f6678c00640 1 -- 172.21.15.28:0/2374789352 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f6684003d80 con 0x558557e1eb60 2024-05-12T22:16:52.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.128+0000 7f6694e15740 10 client.16851 fetch_fsmap learned FSMap version 26 2024-05-12T22:16:52.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.128+0000 7f6694e15740 1 -- 172.21.15.28:0/2374789352 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x558557e2c8e0 con 0x558557e1eb60 2024-05-12T22:16:52.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.128+0000 7f6678c00640 1 -- 172.21.15.28:0/2374789352 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6684036030 con 0x558557e1eb60 2024-05-12T22:16:52.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.128+0000 7f6678c00640 1 -- 172.21.15.28:0/2374789352 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f6684050070 con 0x558557e1eb60 2024-05-12T22:16:52.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.128+0000 7f6694e15740 10 client.16851 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:16:52.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.128+0000 7f6694e15740 10 client.16851 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:16:52.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.128+0000 7f6694e15740 20 client.16851 populate_metadata read hostname 'smithi028' 2024-05-12T22:16:52.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.128+0000 7f6694e15740 1 --2- 172.21.15.28:0/2374789352 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x558557e39d60 0x558557e3c150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:52.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.128+0000 7f6694e15740 4 client.16851 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-12T22:16:52.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.128+0000 7f6694e15740 1 -- 172.21.15.28:0/2374789352 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558557d91160 con 0x558557e39d60 2024-05-12T22:16:52.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.129+0000 7f6683e00640 1 --2- 172.21.15.28:0/2374789352 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x558557e39d60 0x558557e3c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:52.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.129+0000 7f6683e00640 1 --2- 172.21.15.28:0/2374789352 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x558557e39d60 0x558557e3c150 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3640d860dd78e9fd server_cookie=3aae734dbb6efe0a in_seq=0 out_seq=0 2024-05-12T22:16:52.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.129+0000 7f6678c00640 10 client.16851 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:16:52.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.130+0000 7f6678c00640 1 -- 172.21.15.28:0/2374789352 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558557d91160 con 0x558557e39d60 2024-05-12T22:16:52.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.130+0000 7f6678c00640 10 client.16851 handle_command_reply: tid=0 2024-05-12T22:16:52.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.130+0000 7f6694e15740 1 client.16851 shutdown 2024-05-12T22:16:52.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.130+0000 7f6694e15740 1 -- 172.21.15.28:0/2374789352 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558557e1eb60 msgr2=0x558557e26b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:16:52.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.130+0000 7f6694e15740 1 --2- 172.21.15.28:0/2374789352 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558557e1eb60 0x558557e26b50 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f6684009bf0 tx=0x7f6684030fc0 comp rx=0 tx=0).stop 2024-05-12T22:16:52.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.130+0000 7f6694e15740 1 -- 172.21.15.28:0/2374789352 shutdown_connections 2024-05-12T22:16:52.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.130+0000 7f6694e15740 1 --2- 172.21.15.28:0/2374789352 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x558557e39d60 0x558557e3c150 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:52.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.130+0000 7f6694e15740 1 --2- 172.21.15.28:0/2374789352 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558557e29b50 0x558557e275d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:52.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.130+0000 7f6694e15740 1 --2- 172.21.15.28:0/2374789352 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558557e1f480 0x558557e27090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:52.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.130+0000 7f6694e15740 1 --2- 172.21.15.28:0/2374789352 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558557e1eb60 0x558557e26b50 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:52.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.130+0000 7f6678c00640 0 client.16851 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:16:52.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.130+0000 7f6694e15740 1 -- 172.21.15.28:0/2374789352 >> 172.21.15.28:0/2374789352 conn(0x558557e1ae80 msgr2=0x558557e20b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:16:52.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.131+0000 7f6694e15740 1 -- 172.21.15.28:0/2374789352 shutdown_connections 2024-05-12T22:16:52.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.131+0000 7f6694e15740 1 -- 172.21.15.28:0/2374789352 wait complete. 2024-05-12T22:16:52.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.131+0000 7f6694e15740 20 client.16851 trim_cache size 0 max 16384 2024-05-12T22:16:52.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.145+0000 7f6694e15740 1 Processor -- start 2024-05-12T22:16:52.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.146+0000 7f6694e15740 1 -- start start 2024-05-12T22:16:52.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.146+0000 7f6694e15740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558557e1eb60 0x558557e357a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:52.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.146+0000 7f6694e15740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558557e1f480 0x558557e1a780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:52.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.146+0000 7f6694e15740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558557e29b50 0x558557e41910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:52.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.146+0000 7f6694e15740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f668c1b1e30 con 0x558557e29b50 2024-05-12T22:16:52.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.146+0000 7f6694e15740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f668c1af700 con 0x558557e1eb60 2024-05-12T22:16:52.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.146+0000 7f6694e15740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f668c1ad570 con 0x558557e1f480 2024-05-12T22:16:52.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.146+0000 7f668be00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558557e1eb60 0x558557e357a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:52.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.146+0000 7f668be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558557e29b50 0x558557e41910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:52.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.146+0000 7f668be00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558557e1eb60 0x558557e357a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:49294/0 (socket says 172.21.15.28:49294) 2024-05-12T22:16:52.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.146+0000 7f668be00640 1 -- 172.21.15.28:0/2044583828 learned_addr learned my addr 172.21.15.28:0/2044583828 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:16:52.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.146+0000 7f6683e00640 1 --2- 172.21.15.28:0/2044583828 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558557e1f480 0x558557e1a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:52.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.147+0000 7f668be00640 1 -- 172.21.15.28:0/2044583828 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558557e1f480 msgr2=0x558557e1a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:16:52.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.147+0000 7f668be00640 1 --2- 172.21.15.28:0/2044583828 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558557e1f480 0x558557e1a780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:52.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.147+0000 7f668be00640 1 -- 172.21.15.28:0/2044583828 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558557e29b50 msgr2=0x558557e41910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:16:52.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.147+0000 7f668be00640 1 --2- 172.21.15.28:0/2044583828 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558557e29b50 0x558557e41910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:52.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.147+0000 7f668be00640 1 -- 172.21.15.28:0/2044583828 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558557e27ab0 con 0x558557e1eb60 2024-05-12T22:16:52.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.147+0000 7f668be00640 1 --2- 172.21.15.28:0/2044583828 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558557e1eb60 0x558557e357a0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f66840316c0 tx=0x7f6684030fc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eb0741fb26e046c5 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:16:52.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.147+0000 7f667b400640 1 -- 172.21.15.28:0/2044583828 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6684058070 con 0x558557e1eb60 2024-05-12T22:16:52.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.147+0000 7f667b400640 1 -- 172.21.15.28:0/2044583828 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6684036040 con 0x558557e1eb60 2024-05-12T22:16:52.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.147+0000 7f667b400640 1 -- 172.21.15.28:0/2044583828 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6684004da0 con 0x558557e1eb60 2024-05-12T22:16:52.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.148+0000 7f6694e15740 1 -- 172.21.15.28:0/2044583828 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558557e1eb60 msgr2=0x558557e357a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:16:52.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.148+0000 7f6694e15740 1 --2- 172.21.15.28:0/2044583828 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558557e1eb60 0x558557e357a0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f66840316c0 tx=0x7f6684030fc0 comp rx=0 tx=0).stop 2024-05-12T22:16:52.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.148+0000 7f6694e15740 1 -- 172.21.15.28:0/2044583828 shutdown_connections 2024-05-12T22:16:52.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.148+0000 7f6694e15740 1 --2- 172.21.15.28:0/2044583828 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558557e29b50 0x558557e41910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:52.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.148+0000 7f6694e15740 1 --2- 172.21.15.28:0/2044583828 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558557e1f480 0x558557e1a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:52.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.148+0000 7f6694e15740 1 --2- 172.21.15.28:0/2044583828 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558557e1eb60 0x558557e357a0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:52.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.148+0000 7f6694e15740 1 -- 172.21.15.28:0/2044583828 >> 172.21.15.28:0/2044583828 conn(0x558557e1ae80 msgr2=0x558557e273a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:16:52.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.148+0000 7f6694e15740 1 -- 172.21.15.28:0/2044583828 shutdown_connections 2024-05-12T22:16:52.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.148+0000 7f6694e15740 1 -- 172.21.15.28:0/2044583828 wait complete. 2024-05-12T22:16:52.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.148+0000 7f6694e15740 1 -- start start 2024-05-12T22:16:52.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.149+0000 7f6694e15740 1 Processor -- start 2024-05-12T22:16:52.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.149+0000 7f6694e15740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558557e1eb60 0x558557e2d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:52.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.149+0000 7f6694e15740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558557e1f480 0x558557e2dde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:52.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.149+0000 7f6694e15740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558557e29b50 0x558557e2e320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:52.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.149+0000 7f6694e15740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f668c1adbc0 con 0x558557e1eb60 2024-05-12T22:16:52.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.149+0000 7f6694e15740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f668c1af030 con 0x558557e1f480 2024-05-12T22:16:52.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.149+0000 7f6694e15740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f668c1ae290 con 0x558557e29b50 2024-05-12T22:16:52.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.149+0000 7f6694e15740 10 client.0 _ll_register_callbacks cb 0x7f6684009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:16:52.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.149+0000 7f668be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558557e1eb60 0x558557e2d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:52.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.149+0000 7f6683e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558557e1f480 0x558557e2dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:52.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.150+0000 7f6683e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558557e1f480 0x558557e2dde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:49296/0 (socket says 172.21.15.28:49296) 2024-05-12T22:16:52.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.150+0000 7f6683e00640 1 -- 172.21.15.28:0/1667767082 learned_addr learned my addr 172.21.15.28:0/1667767082 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:16:52.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.150+0000 7f668be00640 1 --2- 172.21.15.28:0/1667767082 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558557e29b50 0x558557e2e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:52.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.150+0000 7f6683e00640 1 -- 172.21.15.28:0/1667767082 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558557e29b50 msgr2=0x558557e2e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:16:52.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.150+0000 7f6683e00640 1 --2- 172.21.15.28:0/1667767082 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558557e29b50 0x558557e2e320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:52.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.150+0000 7f6683e00640 1 -- 172.21.15.28:0/1667767082 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558557e1eb60 msgr2=0x558557e2d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:16:52.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.150+0000 7f6683e00640 1 --2- 172.21.15.28:0/1667767082 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558557e1eb60 0x558557e2d8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:52.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.150+0000 7f6683e00640 1 -- 172.21.15.28:0/1667767082 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558557e9f380 con 0x558557e1f480 2024-05-12T22:16:52.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.150+0000 7f668be00640 1 --2- 172.21.15.28:0/1667767082 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558557e29b50 0x558557e2e320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:16:52.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.150+0000 7f6683e00640 1 --2- 172.21.15.28:0/1667767082 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558557e1f480 0x558557e2dde0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f6674009dd0 tx=0x7f667401f9f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=975c5fbe85b48f5e server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:16:52.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.150+0000 7f6678c00640 10 client.0 ms_handle_connect on v2:172.21.15.86:3300/0 2024-05-12T22:16:52.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.150+0000 7f6678c00640 1 -- 172.21.15.28:0/1667767082 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6674004250 con 0x558557e1f480 2024-05-12T22:16:52.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.151+0000 7f6678c00640 1 -- 172.21.15.28:0/1667767082 <== mon.2 v2:172.21.15.86:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8513+0+0 (secure 0 0 0) 0x7f66740043f0 con 0x558557e1f480 2024-05-12T22:16:52.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.151+0000 7f6694e15740 1 -- 172.21.15.28:0/1667767082 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6684036040 con 0x558557e1f480 2024-05-12T22:16:52.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.151+0000 7f6678c00640 1 -- 172.21.15.28:0/1667767082 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f667402f050 con 0x558557e1f480 2024-05-12T22:16:52.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.151+0000 7f6694e15740 10 client.26987 fetch_fsmap learned FSMap version 26 2024-05-12T22:16:52.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.151+0000 7f6694e15740 1 -- 172.21.15.28:0/1667767082 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x558557ea06a0 con 0x558557e1f480 2024-05-12T22:16:52.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.151+0000 7f6678c00640 1 -- 172.21.15.28:0/1667767082 <== mon.2 v2:172.21.15.86:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6684036040 con 0x558557e1f480 2024-05-12T22:16:52.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.151+0000 7f6678c00640 1 -- 172.21.15.28:0/1667767082 <== mon.2 v2:172.21.15.86:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f667401e790 con 0x558557e1f480 2024-05-12T22:16:52.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.151+0000 7f6694e15740 10 client.26987 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:16:52.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.151+0000 7f6694e15740 10 client.26987 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:16:52.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.151+0000 7f6694e15740 20 client.26987 populate_metadata read hostname 'smithi028' 2024-05-12T22:16:52.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.151+0000 7f6694e15740 1 --2- 172.21.15.28:0/1667767082 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x558557ea0e50 0x558557ea3270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:52.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.151+0000 7f6694e15740 4 client.26987 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-12T22:16:52.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.151+0000 7f6694e15740 1 -- 172.21.15.28:0/1667767082 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558557d8b4b0 con 0x558557ea0e50 2024-05-12T22:16:52.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.152+0000 7f668be00640 1 --2- 172.21.15.28:0/1667767082 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x558557ea0e50 0x558557ea3270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:52.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.152+0000 7f668be00640 1 --2- 172.21.15.28:0/1667767082 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x558557ea0e50 0x558557ea3270 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c7856ea6f434ce6c server_cookie=ad28dd13f6f3e297 in_seq=0 out_seq=0 2024-05-12T22:16:52.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.152+0000 7f6678c00640 10 client.26987 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:16:52.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.152+0000 7f6678c00640 1 -- 172.21.15.28:0/1667767082 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558557d8b4b0 con 0x558557ea0e50 2024-05-12T22:16:52.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.152+0000 7f6678c00640 10 client.26987 handle_command_reply: tid=0 2024-05-12T22:16:52.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.152+0000 7f6694e15740 1 client.26987 shutdown 2024-05-12T22:16:52.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.153+0000 7f6694e15740 1 -- 172.21.15.28:0/1667767082 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558557e1f480 msgr2=0x558557e2dde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:16:52.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.153+0000 7f6694e15740 1 --2- 172.21.15.28:0/1667767082 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558557e1f480 0x558557e2dde0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f6674009dd0 tx=0x7f667401f9f0 comp rx=0 tx=0).stop 2024-05-12T22:16:52.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.153+0000 7f6694e15740 1 -- 172.21.15.28:0/1667767082 shutdown_connections 2024-05-12T22:16:52.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.153+0000 7f6694e15740 1 --2- 172.21.15.28:0/1667767082 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x558557ea0e50 0x558557ea3270 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:52.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.153+0000 7f6694e15740 1 --2- 172.21.15.28:0/1667767082 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558557e29b50 0x558557e2e320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:52.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.153+0000 7f6694e15740 1 --2- 172.21.15.28:0/1667767082 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558557e1f480 0x558557e2dde0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:52.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.153+0000 7f6678c00640 0 client.26987 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:16:52.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.153+0000 7f6694e15740 1 --2- 172.21.15.28:0/1667767082 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558557e1eb60 0x558557e2d8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:52.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.153+0000 7f6694e15740 1 -- 172.21.15.28:0/1667767082 >> 172.21.15.28:0/1667767082 conn(0x558557e1ae80 msgr2=0x558557e20150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:16:52.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.153+0000 7f6694e15740 1 -- 172.21.15.28:0/1667767082 shutdown_connections 2024-05-12T22:16:52.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.153+0000 7f6694e15740 1 -- 172.21.15.28:0/1667767082 wait complete. 2024-05-12T22:16:52.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.153+0000 7f6694e15740 20 client.26987 trim_cache size 0 max 16384 2024-05-12T22:16:52.161 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-05-12T22:16:52.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.154+0000 7f6690a00640 1 -- 172.21.15.28:0/3878986519 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f66700a2d80 msgr2=0x7f66700a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:16:52.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.154+0000 7f6690a00640 1 --2- 172.21.15.28:0/3878986519 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f66700a2d80 0x7f66700a5240 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x558557e20120 tx=0x7f667c0d5530 comp rx=0 tx=0).stop 2024-05-12T22:16:52.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.154+0000 7f6690a00640 1 -- 172.21.15.28:0/3878986519 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f668c1016b0 msgr2=0x7f668c19a1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:16:52.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.154+0000 7f6690a00640 1 --2- 172.21.15.28:0/3878986519 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f668c1016b0 0x7f668c19a1a0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f667c00e9f0 tx=0x7f667c00eeb0 comp rx=0 tx=0).stop 2024-05-12T22:16:52.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.154+0000 7f6690a00640 1 -- 172.21.15.28:0/3878986519 shutdown_connections 2024-05-12T22:16:52.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.154+0000 7f6690a00640 1 --2- 172.21.15.28:0/3878986519 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f66700a2d80 0x7f66700a5240 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:52.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.154+0000 7f6690a00640 1 --2- 172.21.15.28:0/3878986519 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f668c1076c0 0x7f668c19ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:52.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.154+0000 7f6690a00640 1 --2- 172.21.15.28:0/3878986519 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f668c102070 0x7f668c19a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:52.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.154+0000 7f6690a00640 1 --2- 172.21.15.28:0/3878986519 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f668c1016b0 0x7f668c19a1a0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:52.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.154+0000 7f6690a00640 1 -- 172.21.15.28:0/3878986519 >> 172.21.15.28:0/3878986519 conn(0x7f668c0fd200 msgr2=0x7f668c10cee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:16:52.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.154+0000 7f6690a00640 1 -- 172.21.15.28:0/3878986519 shutdown_connections 2024-05-12T22:16:52.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:52.154+0000 7f6690a00640 1 -- 172.21.15.28:0/3878986519 wait complete. 2024-05-12T22:16:52.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:16:52 smithi028 ceph-mon[29847]: pgmap v241: 97 pgs: 97 active+clean; 97 MiB data, 757 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-12T22:16:52.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:16:52 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/2309356405' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:16:52.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:16:52 smithi186 ceph-mon[34785]: pgmap v241: 97 pgs: 97 active+clean; 97 MiB data, 757 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-12T22:16:52.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:16:52 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/2309356405' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:16:52.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:16:52 smithi086 ceph-mon[34297]: pgmap v241: 97 pgs: 97 active+clean; 97 MiB data, 757 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-12T22:16:52.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:16:52 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/2309356405' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:16:53.174 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-12T22:16:53.175 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-12T22:16:53.175 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-05-12T22:16:53.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.407+0000 7f85e1c00640 1 -- 172.21.15.28:0/2603680287 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85dc1038e0 msgr2=0x7f85dc103d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:16:53.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.407+0000 7f85e1c00640 1 --2- 172.21.15.28:0/2603680287 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85dc1038e0 0x7f85dc103d60 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f85c4009930 tx=0x7f85c402a420 comp rx=0 tx=0).stop 2024-05-12T22:16:53.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.407+0000 7f85e1c00640 1 -- 172.21.15.28:0/2603680287 shutdown_connections 2024-05-12T22:16:53.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.407+0000 7f85e1c00640 1 --2- 172.21.15.28:0/2603680287 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f85dc1042a0 0x7f85dc1107b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:53.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.407+0000 7f85e1c00640 1 --2- 172.21.15.28:0/2603680287 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85dc1038e0 0x7f85dc103d60 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:53.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.407+0000 7f85e1c00640 1 --2- 172.21.15.28:0/2603680287 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f85dc1098f0 0x7f85dc109cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:53.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.407+0000 7f85e1c00640 1 -- 172.21.15.28:0/2603680287 >> 172.21.15.28:0/2603680287 conn(0x7f85dc0ff370 msgr2=0x7f85dc101790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:16:53.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.407+0000 7f85e1c00640 1 -- 172.21.15.28:0/2603680287 shutdown_connections 2024-05-12T22:16:53.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.407+0000 7f85e1c00640 1 -- 172.21.15.28:0/2603680287 wait complete. 2024-05-12T22:16:53.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.408+0000 7f85e1c00640 1 Processor -- start 2024-05-12T22:16:53.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.409+0000 7f85e1c00640 1 -- start start 2024-05-12T22:16:53.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.409+0000 7f85e1c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85dc1038e0 0x7f85dc19a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:53.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.409+0000 7f85e1c00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f85dc1042a0 0x7f85dc19a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:53.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.409+0000 7f85e1c00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f85dc1098f0 0x7f85dc19eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:53.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.409+0000 7f85e1c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f85dc113d00 con 0x7f85dc1038e0 2024-05-12T22:16:53.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.409+0000 7f85e1c00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f85dc113b80 con 0x7f85dc1098f0 2024-05-12T22:16:53.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.409+0000 7f85e1c00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f85dc113e80 con 0x7f85dc1042a0 2024-05-12T22:16:53.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.409+0000 7f85dbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85dc1038e0 0x7f85dc19a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:53.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.409+0000 7f85dbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85dc1038e0 0x7f85dc19a220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35630/0 (socket says 172.21.15.28:35630) 2024-05-12T22:16:53.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.409+0000 7f85dbe00640 1 -- 172.21.15.28:0/1899826432 learned_addr learned my addr 172.21.15.28:0/1899826432 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:16:53.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.409+0000 7f85d3400640 1 --2- 172.21.15.28:0/1899826432 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f85dc1042a0 0x7f85dc19a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:53.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.409+0000 7f85e1200640 1 --2- 172.21.15.28:0/1899826432 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f85dc1098f0 0x7f85dc19eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:53.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.410+0000 7f85dbe00640 1 -- 172.21.15.28:0/1899826432 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f85dc1042a0 msgr2=0x7f85dc19a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:16:53.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.410+0000 7f85dbe00640 1 --2- 172.21.15.28:0/1899826432 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f85dc1042a0 0x7f85dc19a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:53.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.410+0000 7f85dbe00640 1 -- 172.21.15.28:0/1899826432 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f85dc1098f0 msgr2=0x7f85dc19eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:16:53.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.410+0000 7f85dbe00640 1 --2- 172.21.15.28:0/1899826432 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f85dc1098f0 0x7f85dc19eae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:53.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.410+0000 7f85dbe00640 1 -- 172.21.15.28:0/1899826432 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85dc19f210 con 0x7f85dc1038e0 2024-05-12T22:16:53.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.410+0000 7f85dbe00640 1 --2- 172.21.15.28:0/1899826432 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85dc1038e0 0x7f85dc19a220 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f85cc00eb90 tx=0x7f85cc00c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:16:53.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.410+0000 7f85d9600640 1 -- 172.21.15.28:0/1899826432 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85cc004d20 con 0x7f85dc1038e0 2024-05-12T22:16:53.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.410+0000 7f85d9600640 1 -- 172.21.15.28:0/1899826432 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f85cc00ee90 con 0x7f85dc1038e0 2024-05-12T22:16:53.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.410+0000 7f85e1c00640 1 -- 172.21.15.28:0/1899826432 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85dc19f500 con 0x7f85dc1038e0 2024-05-12T22:16:53.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.410+0000 7f85d9600640 1 -- 172.21.15.28:0/1899826432 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85cc010830 con 0x7f85dc1038e0 2024-05-12T22:16:53.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.411+0000 7f85e1c00640 1 -- 172.21.15.28:0/1899826432 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85dc1a6d90 con 0x7f85dc1038e0 2024-05-12T22:16:53.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.414+0000 7f85e6187740 1 Processor -- start 2024-05-12T22:16:53.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.414+0000 7f85e6187740 1 -- start start 2024-05-12T22:16:53.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.414+0000 7f85d9600640 1 -- 172.21.15.28:0/1899826432 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7f85cc005020 con 0x7f85dc1038e0 2024-05-12T22:16:53.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.414+0000 7f85e6187740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d893581500 0x55d8935818e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:53.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.414+0000 7f85e6187740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d893581eb0 0x55d89358bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:53.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.414+0000 7f85e6187740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55d89358c520 0x55d89358e970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:53.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.414+0000 7f85e6187740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d8934e57c0 con 0x55d893581eb0 2024-05-12T22:16:53.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.414+0000 7f85e6187740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x55d89348d790 con 0x55d89358c520 2024-05-12T22:16:53.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.414+0000 7f85e6187740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55d8934e1780 con 0x55d893581500 2024-05-12T22:16:53.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.415+0000 7f85d3400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55d89358c520 0x55d89358e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:53.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.415+0000 7f85d3400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55d89358c520 0x55d89358e970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:49308/0 (socket says 172.21.15.28:49308) 2024-05-12T22:16:53.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.415+0000 7f85dbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d893581eb0 0x55d89358bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:53.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.415+0000 7f85d3400640 1 -- 172.21.15.28:0/1325871214 learned_addr learned my addr 172.21.15.28:0/1325871214 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:16:53.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.415+0000 7f85e1200640 1 --2- 172.21.15.28:0/1325871214 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d893581500 0x55d8935818e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:53.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.415+0000 7f85d3400640 1 -- 172.21.15.28:0/1325871214 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d893581500 msgr2=0x55d8935818e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:16:53.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.415+0000 7f85d3400640 1 --2- 172.21.15.28:0/1325871214 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d893581500 0x55d8935818e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:53.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.415+0000 7f85d3400640 1 -- 172.21.15.28:0/1325871214 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d893581eb0 msgr2=0x55d89358bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:16:53.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.415+0000 7f85d3400640 1 --2- 172.21.15.28:0/1325871214 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d893581eb0 0x55d89358bf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:53.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.415+0000 7f85d3400640 1 -- 172.21.15.28:0/1325871214 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d8934ad6a0 con 0x55d89358c520 2024-05-12T22:16:53.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.415+0000 7f85dbe00640 1 --2- 172.21.15.28:0/1325871214 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d893581eb0 0x55d89358bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:16:53.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.415+0000 7f85d3400640 1 --2- 172.21.15.28:0/1325871214 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55d89358c520 0x55d89358e970 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f85c402a940 tx=0x7f85c4005bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7d532f314b33c19b server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:16:53.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.416+0000 7f85d9600640 1 --2- 172.21.15.28:0/1899826432 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f85c00a2d80 0x7f85c00a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:53.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.416+0000 7f85d1600640 1 -- 172.21.15.28:0/1325871214 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85c4040070 con 0x55d89358c520 2024-05-12T22:16:53.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.416+0000 7f85d1600640 1 -- 172.21.15.28:0/1325871214 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f85c4031870 con 0x55d89358c520 2024-05-12T22:16:53.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.416+0000 7f85d1600640 1 -- 172.21.15.28:0/1325871214 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85c4035040 con 0x55d89358c520 2024-05-12T22:16:53.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.416+0000 7f85d9600640 1 -- 172.21.15.28:0/1899826432 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8513+0+0 (secure 0 0 0) 0x7f85cc0cf9f0 con 0x7f85dc1038e0 2024-05-12T22:16:53.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.416+0000 7f85e1200640 1 --2- 172.21.15.28:0/1899826432 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f85c00a2d80 0x7f85c00a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:53.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.416+0000 7f85e6187740 1 -- 172.21.15.28:0/1325871214 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55d89358c520 msgr2=0x55d89358e970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:16:53.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.416+0000 7f85e6187740 1 --2- 172.21.15.28:0/1325871214 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55d89358c520 0x55d89358e970 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f85c402a940 tx=0x7f85c4005bd0 comp rx=0 tx=0).stop 2024-05-12T22:16:53.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.417+0000 7f85e6187740 1 -- 172.21.15.28:0/1325871214 shutdown_connections 2024-05-12T22:16:53.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.417+0000 7f85e6187740 1 --2- 172.21.15.28:0/1325871214 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55d89358c520 0x55d89358e970 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:53.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.417+0000 7f85e6187740 1 --2- 172.21.15.28:0/1325871214 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d893581eb0 0x55d89358bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:53.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.417+0000 7f85e6187740 1 --2- 172.21.15.28:0/1325871214 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d893581500 0x55d8935818e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:53.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.417+0000 7f85e6187740 1 -- 172.21.15.28:0/1325871214 >> 172.21.15.28:0/1325871214 conn(0x55d89357d5b0 msgr2=0x55d89357d990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:16:53.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.417+0000 7f85e1200640 1 --2- 172.21.15.28:0/1899826432 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f85c00a2d80 0x7f85c00a5240 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f85d4009cf0 tx=0x7f85d4005f90 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:16:53.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.417+0000 7f85e6187740 1 -- 172.21.15.28:0/1325871214 shutdown_connections 2024-05-12T22:16:53.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.417+0000 7f85e6187740 1 -- 172.21.15.28:0/1325871214 wait complete. 2024-05-12T22:16:53.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.417+0000 7f85e6187740 1 -- start start 2024-05-12T22:16:53.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.418+0000 7f85e6187740 1 Processor -- start 2024-05-12T22:16:53.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.418+0000 7f85e6187740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55d893581500 0x55d893589430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:53.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.418+0000 7f85e6187740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d893581eb0 0x55d893589970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:53.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.418+0000 7f85e6187740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d89358c520 0x55d893589eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:53.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.418+0000 7f85e6187740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d89358f4d0 con 0x55d893581eb0 2024-05-12T22:16:53.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.418+0000 7f85e6187740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x55d8934a0a90 con 0x55d893581500 2024-05-12T22:16:53.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.418+0000 7f85e6187740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55d89358f650 con 0x55d89358c520 2024-05-12T22:16:53.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.418+0000 7f85e6187740 10 client.0 _ll_register_callbacks cb 0x55d8934dbaf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:16:53.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.418+0000 7f85d3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d893581eb0 0x55d893589970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:53.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.418+0000 7f85d3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d893581eb0 0x55d893589970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35658/0 (socket says 172.21.15.28:35658) 2024-05-12T22:16:53.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.418+0000 7f85d3400640 1 -- 172.21.15.28:0/2764963856 learned_addr learned my addr 172.21.15.28:0/2764963856 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:16:53.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.418+0000 7f85dbe00640 1 --2- 172.21.15.28:0/2764963856 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55d893581500 0x55d893589430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:53.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.418+0000 7f85e1200640 1 --2- 172.21.15.28:0/2764963856 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d89358c520 0x55d893589eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:53.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.419+0000 7f85dbe00640 1 -- 172.21.15.28:0/2764963856 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d89358c520 msgr2=0x55d893589eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:16:53.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.419+0000 7f85dbe00640 1 --2- 172.21.15.28:0/2764963856 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d89358c520 0x55d893589eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:53.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.419+0000 7f85dbe00640 1 -- 172.21.15.28:0/2764963856 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d893581eb0 msgr2=0x55d893589970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:16:53.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.419+0000 7f85dbe00640 1 --2- 172.21.15.28:0/2764963856 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d893581eb0 0x55d893589970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:53.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.419+0000 7f85dbe00640 1 -- 172.21.15.28:0/2764963856 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d8932c17e0 con 0x55d893581500 2024-05-12T22:16:53.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.419+0000 7f85d3400640 1 --2- 172.21.15.28:0/2764963856 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d893581eb0 0x55d893589970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:16:53.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.419+0000 7f85dbe00640 1 --2- 172.21.15.28:0/2764963856 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55d893581500 0x55d893589430 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f85cc0cd0e0 tx=0x7f85cc098210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=20afeefc7a26adab server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:16:53.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.419+0000 7f85ca000640 10 client.0 ms_handle_connect on v2:172.21.15.86:3300/0 2024-05-12T22:16:53.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.419+0000 7f85ca000640 1 -- 172.21.15.28:0/2764963856 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85cc09c5b0 con 0x55d893581500 2024-05-12T22:16:53.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.419+0000 7f85ca000640 1 -- 172.21.15.28:0/2764963856 <== mon.2 v2:172.21.15.86:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8513+0+0 (secure 0 0 0) 0x7f85cc0cd360 con 0x55d893581500 2024-05-12T22:16:53.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.419+0000 7f85e6187740 1 -- 172.21.15.28:0/2764963856 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f85c4031870 con 0x55d893581500 2024-05-12T22:16:53.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.420+0000 7f85ca000640 1 -- 172.21.15.28:0/2764963856 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f85cc0cd560 con 0x55d893581500 2024-05-12T22:16:53.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.420+0000 7f85e6187740 10 client.26999 fetch_fsmap learned FSMap version 26 2024-05-12T22:16:53.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.420+0000 7f85e6187740 1 -- 172.21.15.28:0/2764963856 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55d893321f70 con 0x55d893581500 2024-05-12T22:16:53.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.420+0000 7f85ca000640 1 -- 172.21.15.28:0/2764963856 <== mon.2 v2:172.21.15.86:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f85c4031870 con 0x55d893581500 2024-05-12T22:16:53.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.420+0000 7f85ca000640 1 -- 172.21.15.28:0/2764963856 <== mon.2 v2:172.21.15.86:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f85cc0a4690 con 0x55d893581500 2024-05-12T22:16:53.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.420+0000 7f85e6187740 10 client.26999 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:16:53.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.420+0000 7f85e6187740 10 client.26999 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:16:53.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.420+0000 7f85e6187740 20 client.26999 populate_metadata read hostname 'smithi028' 2024-05-12T22:16:53.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.420+0000 7f85e6187740 1 --2- 172.21.15.28:0/2764963856 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55d89359c740 0x55d89359ec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:53.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.420+0000 7f85e6187740 4 client.26999 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-12T22:16:53.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.420+0000 7f85e6187740 1 -- 172.21.15.28:0/2764963856 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d8934f93a0 con 0x55d89359c740 2024-05-12T22:16:53.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.420+0000 7f85d3400640 1 --2- 172.21.15.28:0/2764963856 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55d89359c740 0x55d89359ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:53.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.421+0000 7f85d3400640 1 --2- 172.21.15.28:0/2764963856 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55d89359c740 0x55d89359ec00 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bc9cdea2cb49162 server_cookie=cb2ea9651c2f0719 in_seq=0 out_seq=0 2024-05-12T22:16:53.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.421+0000 7f85ca000640 10 client.26999 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:16:53.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.422+0000 7f85ca000640 1 -- 172.21.15.28:0/2764963856 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d8934f93a0 con 0x55d89359c740 2024-05-12T22:16:53.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.422+0000 7f85ca000640 10 client.26999 handle_command_reply: tid=0 2024-05-12T22:16:53.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.422+0000 7f85e6187740 1 client.26999 shutdown 2024-05-12T22:16:53.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.422+0000 7f85e6187740 1 -- 172.21.15.28:0/2764963856 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55d893581500 msgr2=0x55d893589430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:16:53.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.422+0000 7f85e6187740 1 --2- 172.21.15.28:0/2764963856 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55d893581500 0x55d893589430 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f85cc0cd0e0 tx=0x7f85cc098210 comp rx=0 tx=0).stop 2024-05-12T22:16:53.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.422+0000 7f85e6187740 1 -- 172.21.15.28:0/2764963856 shutdown_connections 2024-05-12T22:16:53.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.422+0000 7f85e6187740 1 --2- 172.21.15.28:0/2764963856 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55d89359c740 0x55d89359ec00 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:53.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.422+0000 7f85e6187740 1 --2- 172.21.15.28:0/2764963856 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d89358c520 0x55d893589eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:53.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.422+0000 7f85e6187740 1 --2- 172.21.15.28:0/2764963856 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d893581eb0 0x55d893589970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:53.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.422+0000 7f85ca000640 0 client.26999 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:16:53.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.422+0000 7f85e6187740 1 --2- 172.21.15.28:0/2764963856 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55d893581500 0x55d893589430 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:53.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.422+0000 7f85e6187740 1 -- 172.21.15.28:0/2764963856 >> 172.21.15.28:0/2764963856 conn(0x55d89357d5b0 msgr2=0x55d893583630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:16:53.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.422+0000 7f85e6187740 1 -- 172.21.15.28:0/2764963856 shutdown_connections 2024-05-12T22:16:53.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.422+0000 7f85e6187740 1 -- 172.21.15.28:0/2764963856 wait complete. 2024-05-12T22:16:53.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.423+0000 7f85e6187740 20 client.26999 trim_cache size 0 max 16384 2024-05-12T22:16:53.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.442+0000 7f85e6187740 1 Processor -- start 2024-05-12T22:16:53.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.442+0000 7f85e6187740 1 -- start start 2024-05-12T22:16:53.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.442+0000 7f85e6187740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55d893581500 0x55d893583950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:53.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.442+0000 7f85e6187740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d893581eb0 0x55d893598390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:53.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.442+0000 7f85e6187740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d89358c520 0x55d8935a4480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:53.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.442+0000 7f85e6187740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f85dc1b1e70 con 0x55d893581eb0 2024-05-12T22:16:53.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.442+0000 7f85e6187740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f85dc1af740 con 0x55d893581500 2024-05-12T22:16:53.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.442+0000 7f85e6187740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f85dc1ad5b0 con 0x55d89358c520 2024-05-12T22:16:53.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.443+0000 7f85d3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d893581eb0 0x55d893598390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:53.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.443+0000 7f85d3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d893581eb0 0x55d893598390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35660/0 (socket says 172.21.15.28:35660) 2024-05-12T22:16:53.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.443+0000 7f85d3400640 1 -- 172.21.15.28:0/7914533 learned_addr learned my addr 172.21.15.28:0/7914533 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:16:53.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.443+0000 7f85dbe00640 1 --2- 172.21.15.28:0/7914533 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55d893581500 0x55d893583950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:53.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.443+0000 7f85e1200640 1 --2- 172.21.15.28:0/7914533 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d89358c520 0x55d8935a4480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:53.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.443+0000 7f85d3400640 1 -- 172.21.15.28:0/7914533 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d89358c520 msgr2=0x55d8935a4480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:16:53.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.443+0000 7f85d3400640 1 --2- 172.21.15.28:0/7914533 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d89358c520 0x55d8935a4480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:53.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.443+0000 7f85d3400640 1 -- 172.21.15.28:0/7914533 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55d893581500 msgr2=0x55d893583950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:16:53.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.443+0000 7f85d3400640 1 --2- 172.21.15.28:0/7914533 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55d893581500 0x55d893583950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:53.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.443+0000 7f85d3400640 1 -- 172.21.15.28:0/7914533 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d8931b9170 con 0x55d893581eb0 2024-05-12T22:16:53.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.443+0000 7f85d3400640 1 --2- 172.21.15.28:0/7914533 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d893581eb0 0x55d893598390 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f85c40055d0 tx=0x7f85c40054d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc3cbb741d7ded84 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:16:53.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.444+0000 7f85d0c00640 1 -- 172.21.15.28:0/7914533 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85c4056070 con 0x55d893581eb0 2024-05-12T22:16:53.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.444+0000 7f85d0c00640 1 -- 172.21.15.28:0/7914533 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f85c40500d0 con 0x55d893581eb0 2024-05-12T22:16:53.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.444+0000 7f85d0c00640 1 -- 172.21.15.28:0/7914533 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85c4035040 con 0x55d893581eb0 2024-05-12T22:16:53.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.444+0000 7f85e6187740 1 -- 172.21.15.28:0/7914533 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d893581eb0 msgr2=0x55d893598390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:16:53.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.444+0000 7f85e6187740 1 --2- 172.21.15.28:0/7914533 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d893581eb0 0x55d893598390 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f85c40055d0 tx=0x7f85c40054d0 comp rx=0 tx=0).stop 2024-05-12T22:16:53.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.444+0000 7f85e6187740 1 -- 172.21.15.28:0/7914533 shutdown_connections 2024-05-12T22:16:53.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.444+0000 7f85e6187740 1 --2- 172.21.15.28:0/7914533 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d89358c520 0x55d8935a4480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:53.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.444+0000 7f85e6187740 1 --2- 172.21.15.28:0/7914533 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d893581eb0 0x55d893598390 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:53.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.444+0000 7f85e6187740 1 --2- 172.21.15.28:0/7914533 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55d893581500 0x55d893583950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:53.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.444+0000 7f85e6187740 1 -- 172.21.15.28:0/7914533 >> 172.21.15.28:0/7914533 conn(0x55d89357d5b0 msgr2=0x55d89357c730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:16:53.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.444+0000 7f85e6187740 1 -- 172.21.15.28:0/7914533 shutdown_connections 2024-05-12T22:16:53.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.445+0000 7f85e6187740 1 -- 172.21.15.28:0/7914533 wait complete. 2024-05-12T22:16:53.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.445+0000 7f85e6187740 1 -- start start 2024-05-12T22:16:53.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.445+0000 7f85e6187740 1 Processor -- start 2024-05-12T22:16:53.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.446+0000 7f85e6187740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d893581500 0x55d893590650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:53.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.446+0000 7f85e6187740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55d893581eb0 0x55d893590b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:53.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.446+0000 7f85e6187740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d89358c520 0x55d89360a500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:53.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.446+0000 7f85e6187740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f85dc1adc00 con 0x55d89358c520 2024-05-12T22:16:53.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.446+0000 7f85e6187740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f85dc1af070 con 0x55d893581eb0 2024-05-12T22:16:53.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.446+0000 7f85e6187740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f85dc1ae2d0 con 0x55d893581500 2024-05-12T22:16:53.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.446+0000 7f85e6187740 10 client.0 _ll_register_callbacks cb 0x7f85cc0d5240 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:16:53.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.446+0000 7f85dbe00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d893581500 0x55d893590650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:53.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.446+0000 7f85dbe00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d893581500 0x55d893590650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:59436/0 (socket says 172.21.15.28:59436) 2024-05-12T22:16:53.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.446+0000 7f85dbe00640 1 -- 172.21.15.28:0/3177119621 learned_addr learned my addr 172.21.15.28:0/3177119621 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:16:53.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.446+0000 7f85d3400640 1 --2- 172.21.15.28:0/3177119621 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55d893581eb0 0x55d893590b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:53.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.446+0000 7f85e1200640 1 --2- 172.21.15.28:0/3177119621 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d89358c520 0x55d89360a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:53.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.446+0000 7f85dbe00640 1 -- 172.21.15.28:0/3177119621 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55d893581eb0 msgr2=0x55d893590b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:16:53.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.446+0000 7f85dbe00640 1 --2- 172.21.15.28:0/3177119621 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55d893581eb0 0x55d893590b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:53.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.446+0000 7f85dbe00640 1 -- 172.21.15.28:0/3177119621 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d89358c520 msgr2=0x55d89360a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:16:53.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.446+0000 7f85dbe00640 1 --2- 172.21.15.28:0/3177119621 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d89358c520 0x55d89360a500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:53.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.446+0000 7f85dbe00640 1 -- 172.21.15.28:0/3177119621 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d893591490 con 0x55d893581500 2024-05-12T22:16:53.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.446+0000 7f85d3400640 1 --2- 172.21.15.28:0/3177119621 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55d893581eb0 0x55d893590b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:16:53.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.447+0000 7f85dbe00640 1 --2- 172.21.15.28:0/3177119621 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d893581500 0x55d893590650 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f85cc0049d0 tx=0x7f85cc005b40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ae16f2d090e5a188 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:16:53.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.447+0000 7f85ca000640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-12T22:16:53.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.447+0000 7f85ca000640 1 -- 172.21.15.28:0/3177119621 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85cc09b9f0 con 0x55d893581500 2024-05-12T22:16:53.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.447+0000 7f85e6187740 1 -- 172.21.15.28:0/3177119621 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f85c40500d0 con 0x55d893581500 2024-05-12T22:16:53.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.447+0000 7f85ca000640 1 -- 172.21.15.28:0/3177119621 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8513+0+0 (secure 0 0 0) 0x7f85cc09b3a0 con 0x55d893581500 2024-05-12T22:16:53.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.447+0000 7f85ca000640 1 -- 172.21.15.28:0/3177119621 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f85cc0cd560 con 0x55d893581500 2024-05-12T22:16:53.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.447+0000 7f85e6187740 10 client.26959 fetch_fsmap learned FSMap version 26 2024-05-12T22:16:53.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.447+0000 7f85e6187740 1 -- 172.21.15.28:0/3177119621 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55d89360bb80 con 0x55d893581500 2024-05-12T22:16:53.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.447+0000 7f85ca000640 1 -- 172.21.15.28:0/3177119621 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f85c40500d0 con 0x55d893581500 2024-05-12T22:16:53.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.448+0000 7f85ca000640 1 -- 172.21.15.28:0/3177119621 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f85cc097c00 con 0x55d893581500 2024-05-12T22:16:53.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.448+0000 7f85e6187740 10 client.26959 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:16:53.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.448+0000 7f85e6187740 10 client.26959 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:16:53.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.448+0000 7f85e6187740 20 client.26959 populate_metadata read hostname 'smithi028' 2024-05-12T22:16:53.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.448+0000 7f85e6187740 1 --2- 172.21.15.28:0/3177119621 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55d89360c2a0 0x55d89360e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:53.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.448+0000 7f85e6187740 4 client.26959 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-12T22:16:53.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.448+0000 7f85e6187740 1 -- 172.21.15.28:0/3177119621 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55d8934f2ef0 con 0x55d89360c2a0 2024-05-12T22:16:53.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.448+0000 7f85d3400640 1 --2- 172.21.15.28:0/3177119621 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55d89360c2a0 0x55d89360e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:53.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.449+0000 7f85d3400640 1 --2- 172.21.15.28:0/3177119621 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55d89360c2a0 0x55d89360e6c0 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=54a5b97745d12078 server_cookie=c8ad60f9bac112f4 in_seq=0 out_seq=0 2024-05-12T22:16:53.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.449+0000 7f85ca000640 10 client.26959 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:16:53.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.450+0000 7f85ca000640 1 -- 172.21.15.28:0/3177119621 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55d8934f2ef0 con 0x55d89360c2a0 2024-05-12T22:16:53.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.450+0000 7f85ca000640 10 client.26959 handle_command_reply: tid=0 2024-05-12T22:16:53.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.450+0000 7f85e6187740 1 client.26959 shutdown 2024-05-12T22:16:53.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.450+0000 7f85e6187740 1 -- 172.21.15.28:0/3177119621 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d893581500 msgr2=0x55d893590650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:16:53.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.450+0000 7f85e6187740 1 --2- 172.21.15.28:0/3177119621 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d893581500 0x55d893590650 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f85cc0049d0 tx=0x7f85cc005b40 comp rx=0 tx=0).stop 2024-05-12T22:16:53.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.450+0000 7f85e6187740 1 -- 172.21.15.28:0/3177119621 shutdown_connections 2024-05-12T22:16:53.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.450+0000 7f85e6187740 1 --2- 172.21.15.28:0/3177119621 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55d89360c2a0 0x55d89360e6c0 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:53.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.450+0000 7f85e6187740 1 --2- 172.21.15.28:0/3177119621 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d89358c520 0x55d89360a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:53.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.450+0000 7f85e6187740 1 --2- 172.21.15.28:0/3177119621 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55d893581eb0 0x55d893590b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:53.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.450+0000 7f85e6187740 1 --2- 172.21.15.28:0/3177119621 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d893581500 0x55d893590650 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:53.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.450+0000 7f85ca000640 0 client.26959 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:16:53.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.450+0000 7f85e6187740 1 -- 172.21.15.28:0/3177119621 >> 172.21.15.28:0/3177119621 conn(0x55d89357d5b0 msgr2=0x55d8935893f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:16:53.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.450+0000 7f85e6187740 1 -- 172.21.15.28:0/3177119621 shutdown_connections 2024-05-12T22:16:53.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.450+0000 7f85e6187740 1 -- 172.21.15.28:0/3177119621 wait complete. 2024-05-12T22:16:53.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.451+0000 7f85e6187740 20 client.26959 trim_cache size 0 max 16384 2024-05-12T22:16:53.465 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-05-12T22:16:53.465 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-05-12T22:16:53.466 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "c1467dff-508e-4ec1-a23e-f353c76045ae", 2024-05-12T22:16:53.466 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-05-12T22:16:53.466 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-05-12T22:16:53.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.451+0000 7f85e1c00640 1 -- 172.21.15.28:0/1899826432 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f85c00a2d80 msgr2=0x7f85c00a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:16:53.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.451+0000 7f85e1c00640 1 --2- 172.21.15.28:0/1899826432 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f85c00a2d80 0x7f85c00a5240 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f85d4009cf0 tx=0x7f85d4005f90 comp rx=0 tx=0).stop 2024-05-12T22:16:53.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.451+0000 7f85e1c00640 1 -- 172.21.15.28:0/1899826432 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85dc1038e0 msgr2=0x7f85dc19a220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:16:53.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.451+0000 7f85e1c00640 1 --2- 172.21.15.28:0/1899826432 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85dc1038e0 0x7f85dc19a220 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f85cc00eb90 tx=0x7f85cc00c6a0 comp rx=0 tx=0).stop 2024-05-12T22:16:53.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.451+0000 7f85e1c00640 1 -- 172.21.15.28:0/1899826432 shutdown_connections 2024-05-12T22:16:53.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.451+0000 7f85e1c00640 1 --2- 172.21.15.28:0/1899826432 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f85c00a2d80 0x7f85c00a5240 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:53.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.451+0000 7f85e1c00640 1 --2- 172.21.15.28:0/1899826432 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f85dc1098f0 0x7f85dc19eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:53.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.451+0000 7f85e1c00640 1 --2- 172.21.15.28:0/1899826432 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f85dc1042a0 0x7f85dc19a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:53.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.451+0000 7f85e1c00640 1 --2- 172.21.15.28:0/1899826432 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85dc1038e0 0x7f85dc19a220 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:53.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.451+0000 7f85e1c00640 1 -- 172.21.15.28:0/1899826432 >> 172.21.15.28:0/1899826432 conn(0x7f85dc0ff370 msgr2=0x7f85dc0ffc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:16:53.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.452+0000 7f85e1c00640 1 -- 172.21.15.28:0/1899826432 shutdown_connections 2024-05-12T22:16:53.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.452+0000 7f85e1c00640 1 -- 172.21.15.28:0/1899826432 wait complete. 2024-05-12T22:16:53.472 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-12T22:16:53.472 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-12T22:16:53.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.712+0000 7f011f200640 1 -- 172.21.15.28:0/2519620768 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0120107dd0 msgr2=0x7f0120108230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:16:53.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.712+0000 7f011f200640 1 --2- 172.21.15.28:0/2519620768 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0120107dd0 0x7f0120108230 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f010c009900 tx=0x7f010c02a480 comp rx=0 tx=0).stop 2024-05-12T22:16:53.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.712+0000 7f011f200640 1 -- 172.21.15.28:0/2519620768 shutdown_connections 2024-05-12T22:16:53.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.712+0000 7f011f200640 1 --2- 172.21.15.28:0/2519620768 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0120108770 0x7f0120114b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:53.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.712+0000 7f011f200640 1 --2- 172.21.15.28:0/2519620768 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0120107dd0 0x7f0120108230 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:53.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.712+0000 7f011f200640 1 --2- 172.21.15.28:0/2519620768 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f012010ddf0 0x7f012010e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:53.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.712+0000 7f011f200640 1 -- 172.21.15.28:0/2519620768 >> 172.21.15.28:0/2519620768 conn(0x7f0120077650 msgr2=0x7f0120106650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:16:53.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.712+0000 7f011f200640 1 -- 172.21.15.28:0/2519620768 shutdown_connections 2024-05-12T22:16:53.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.713+0000 7f011f200640 1 -- 172.21.15.28:0/2519620768 wait complete. 2024-05-12T22:16:53.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.714+0000 7f011f200640 1 Processor -- start 2024-05-12T22:16:53.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.714+0000 7f011f200640 1 -- start start 2024-05-12T22:16:53.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.714+0000 7f011f200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0120107dd0 0x7f012019e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:53.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.714+0000 7f011f200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0120108770 0x7f012019eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:53.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.714+0000 7f011f200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f012010ddf0 0x7f01201a2e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:53.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.714+0000 7f011f200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f01201180d0 con 0x7f012010ddf0 2024-05-12T22:16:53.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.714+0000 7f011f200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f0120117f50 con 0x7f0120108770 2024-05-12T22:16:53.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.714+0000 7f011f200640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f0120118250 con 0x7f0120107dd0 2024-05-12T22:16:53.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.715+0000 7f011d400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0120108770 0x7f012019eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:53.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.715+0000 7f011d400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0120108770 0x7f012019eaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:49342/0 (socket says 172.21.15.28:49342) 2024-05-12T22:16:53.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.715+0000 7f011d400640 1 -- 172.21.15.28:0/4133261371 learned_addr learned my addr 172.21.15.28:0/4133261371 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:16:53.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.715+0000 7f011e800640 1 --2- 172.21.15.28:0/4133261371 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f012010ddf0 0x7f01201a2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:53.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.715+0000 7f011de00640 1 --2- 172.21.15.28:0/4133261371 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0120107dd0 0x7f012019e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:53.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.715+0000 7f011d400640 1 -- 172.21.15.28:0/4133261371 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0120107dd0 msgr2=0x7f012019e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:16:53.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.715+0000 7f011d400640 1 --2- 172.21.15.28:0/4133261371 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0120107dd0 0x7f012019e5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:53.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.715+0000 7f011d400640 1 -- 172.21.15.28:0/4133261371 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f012010ddf0 msgr2=0x7f01201a2e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:16:53.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.715+0000 7f011d400640 1 --2- 172.21.15.28:0/4133261371 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f012010ddf0 0x7f01201a2e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:53.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.715+0000 7f011d400640 1 -- 172.21.15.28:0/4133261371 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01201a3550 con 0x7f0120108770 2024-05-12T22:16:53.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.716+0000 7f011d400640 1 --2- 172.21.15.28:0/4133261371 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0120108770 0x7f012019eaf0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f010c006010 tx=0x7f010c0042f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:16:53.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.716+0000 7f0116a00640 1 -- 172.21.15.28:0/4133261371 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f010c03f030 con 0x7f0120108770 2024-05-12T22:16:53.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.716+0000 7f0116a00640 1 -- 172.21.15.28:0/4133261371 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f010c003ce0 con 0x7f0120108770 2024-05-12T22:16:53.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.716+0000 7f011f200640 1 -- 172.21.15.28:0/4133261371 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01201a37e0 con 0x7f0120108770 2024-05-12T22:16:53.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.716+0000 7f0116a00640 1 -- 172.21.15.28:0/4133261371 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f010c035070 con 0x7f0120108770 2024-05-12T22:16:53.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.716+0000 7f011f200640 1 -- 172.21.15.28:0/4133261371 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01201ab0d0 con 0x7f0120108770 2024-05-12T22:16:53.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.720+0000 7f0127a22740 1 Processor -- start 2024-05-12T22:16:53.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.720+0000 7f0127a22740 1 -- start start 2024-05-12T22:16:53.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.720+0000 7f0116a00640 1 -- 172.21.15.28:0/4133261371 <== mon.2 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7f010c042030 con 0x7f0120108770 2024-05-12T22:16:53.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.721+0000 7f0127a22740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557310cc8dd0 0x557310cc91b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:53.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.721+0000 7f0127a22740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557310cc96f0 0x557310cd3880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:53.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.721+0000 7f0127a22740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x557310cd3dc0 0x557310cd61b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:53.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.721+0000 7f0127a22740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x557310c912a0 con 0x557310cc8dd0 2024-05-12T22:16:53.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.721+0000 7f0127a22740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x557310c2db90 con 0x557310cd3dc0 2024-05-12T22:16:53.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.721+0000 7f0127a22740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x557310bd55b0 con 0x557310cc96f0 2024-05-12T22:16:53.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.721+0000 7f011de00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557310cc96f0 0x557310cd3880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:53.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.721+0000 7f011de00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557310cc96f0 0x557310cd3880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:59462/0 (socket says 172.21.15.28:59462) 2024-05-12T22:16:53.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.721+0000 7f011de00640 1 -- 172.21.15.28:0/1699117388 learned_addr learned my addr 172.21.15.28:0/1699117388 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:16:53.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.721+0000 7f011d400640 1 --2- 172.21.15.28:0/1699117388 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x557310cd3dc0 0x557310cd61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:53.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.721+0000 7f011e800640 1 --2- 172.21.15.28:0/1699117388 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557310cc8dd0 0x557310cc91b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:53.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.722+0000 7f011de00640 1 -- 172.21.15.28:0/1699117388 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x557310cd3dc0 msgr2=0x557310cd61b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:16:53.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.722+0000 7f011de00640 1 --2- 172.21.15.28:0/1699117388 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x557310cd3dc0 0x557310cd61b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:53.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.722+0000 7f011de00640 1 -- 172.21.15.28:0/1699117388 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557310cc8dd0 msgr2=0x557310cc91b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:16:53.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.722+0000 7f011de00640 1 --2- 172.21.15.28:0/1699117388 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557310cc8dd0 0x557310cc91b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:53.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.722+0000 7f011de00640 1 -- 172.21.15.28:0/1699117388 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557310a0b7d0 con 0x557310cc96f0 2024-05-12T22:16:53.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.722+0000 7f011de00640 1 --2- 172.21.15.28:0/1699117388 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557310cc96f0 0x557310cd3880 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f0110009c10 tx=0x7f01100117b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f8cec5a1baeb19e server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:16:53.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.723+0000 7f0116a00640 1 --2- 172.21.15.28:0/4133261371 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f01040a2d80 0x7f01040a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:53.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.723+0000 7f010b000640 1 -- 172.21.15.28:0/1699117388 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0110012450 con 0x557310cc96f0 2024-05-12T22:16:53.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.723+0000 7f010b000640 1 -- 172.21.15.28:0/1699117388 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0110012ad0 con 0x557310cc96f0 2024-05-12T22:16:53.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.723+0000 7f010b000640 1 -- 172.21.15.28:0/1699117388 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f011000fbd0 con 0x557310cc96f0 2024-05-12T22:16:53.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.723+0000 7f0116a00640 1 -- 172.21.15.28:0/4133261371 <== mon.2 v2:172.21.15.86:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8513+0+0 (secure 0 0 0) 0x7f010c0ef9f0 con 0x7f0120108770 2024-05-12T22:16:53.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.723+0000 7f011e800640 1 --2- 172.21.15.28:0/4133261371 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f01040a2d80 0x7f01040a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:53.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.724+0000 7f0127a22740 1 -- 172.21.15.28:0/1699117388 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557310cc96f0 msgr2=0x557310cd3880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:16:53.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.724+0000 7f0127a22740 1 --2- 172.21.15.28:0/1699117388 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557310cc96f0 0x557310cd3880 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f0110009c10 tx=0x7f01100117b0 comp rx=0 tx=0).stop 2024-05-12T22:16:53.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.724+0000 7f0127a22740 1 -- 172.21.15.28:0/1699117388 shutdown_connections 2024-05-12T22:16:53.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.724+0000 7f0127a22740 1 --2- 172.21.15.28:0/1699117388 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x557310cd3dc0 0x557310cd61b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:53.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.724+0000 7f0127a22740 1 --2- 172.21.15.28:0/1699117388 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557310cc96f0 0x557310cd3880 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:53.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.724+0000 7f0127a22740 1 --2- 172.21.15.28:0/1699117388 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557310cc8dd0 0x557310cc91b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:53.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.724+0000 7f0127a22740 1 -- 172.21.15.28:0/1699117388 >> 172.21.15.28:0/1699117388 conn(0x557310cc50f0 msgr2=0x557310cc54d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:16:53.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.724+0000 7f0127a22740 1 -- 172.21.15.28:0/1699117388 shutdown_connections 2024-05-12T22:16:53.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.724+0000 7f011e800640 1 --2- 172.21.15.28:0/4133261371 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f01040a2d80 0x7f01040a5240 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f0118005fd0 tx=0x7f01180095f0 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:16:53.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.725+0000 7f0127a22740 1 -- 172.21.15.28:0/1699117388 wait complete. 2024-05-12T22:16:53.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.725+0000 7f0127a22740 1 -- start start 2024-05-12T22:16:53.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.725+0000 7f0127a22740 1 Processor -- start 2024-05-12T22:16:53.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.726+0000 7f0127a22740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x557310cc8dd0 0x557310cd3630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:53.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.726+0000 7f0127a22740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557310cc96f0 0x557310cd0920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:53.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.726+0000 7f0127a22740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557310cd3dc0 0x557310cd0e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:53.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.726+0000 7f0127a22740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x557310be89f0 con 0x557310cd3dc0 2024-05-12T22:16:53.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.726+0000 7f0127a22740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x557310c29b50 con 0x557310cc8dd0 2024-05-12T22:16:53.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.726+0000 7f0127a22740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x557310cd6bb0 con 0x557310cc96f0 2024-05-12T22:16:53.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.726+0000 7f0127a22740 10 client.0 _ll_register_callbacks cb 0x557310940be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:16:53.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.726+0000 7f011de00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x557310cc8dd0 0x557310cd3630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:53.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.726+0000 7f011d400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557310cc96f0 0x557310cd0920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:53.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.726+0000 7f011de00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x557310cc8dd0 0x557310cd3630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:49358/0 (socket says 172.21.15.28:49358) 2024-05-12T22:16:53.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.726+0000 7f011de00640 1 -- 172.21.15.28:0/3926563855 learned_addr learned my addr 172.21.15.28:0/3926563855 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:16:53.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.727+0000 7f011e800640 1 --2- 172.21.15.28:0/3926563855 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557310cd3dc0 0x557310cd0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:53.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.727+0000 7f011d400640 1 -- 172.21.15.28:0/3926563855 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x557310cc8dd0 msgr2=0x557310cd3630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:16:53.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.727+0000 7f011d400640 1 --2- 172.21.15.28:0/3926563855 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x557310cc8dd0 0x557310cd3630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:53.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.727+0000 7f011d400640 1 -- 172.21.15.28:0/3926563855 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557310cd3dc0 msgr2=0x557310cd0e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:16:53.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.727+0000 7f011d400640 1 --2- 172.21.15.28:0/3926563855 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557310cd3dc0 0x557310cd0e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:53.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.727+0000 7f011d400640 1 -- 172.21.15.28:0/3926563855 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5573109413f0 con 0x557310cc96f0 2024-05-12T22:16:53.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.727+0000 7f011de00640 1 --2- 172.21.15.28:0/3926563855 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x557310cc8dd0 0x557310cd3630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:16:53.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.727+0000 7f011d400640 1 --2- 172.21.15.28:0/3926563855 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557310cc96f0 0x557310cd0920 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f010c0ed0e0 tx=0x7f010c0092b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ea8aad4642c4aaf3 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:16:53.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.727+0000 7f00ff400640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-12T22:16:53.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.727+0000 7f00ff400640 1 -- 172.21.15.28:0/3926563855 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f010c049070 con 0x557310cc96f0 2024-05-12T22:16:53.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.727+0000 7f00ff400640 1 -- 172.21.15.28:0/3926563855 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8513+0+0 (secure 0 0 0) 0x7f010c0c4750 con 0x557310cc96f0 2024-05-12T22:16:53.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.727+0000 7f0127a22740 1 -- 172.21.15.28:0/3926563855 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0110012ad0 con 0x557310cc96f0 2024-05-12T22:16:53.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.728+0000 7f00ff400640 1 -- 172.21.15.28:0/3926563855 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f010c0b8210 con 0x557310cc96f0 2024-05-12T22:16:53.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.728+0000 7f0127a22740 10 client.26971 fetch_fsmap learned FSMap version 26 2024-05-12T22:16:53.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.728+0000 7f0127a22740 1 -- 172.21.15.28:0/3926563855 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5573109010f0 con 0x557310cc96f0 2024-05-12T22:16:53.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.728+0000 7f00ff400640 1 -- 172.21.15.28:0/3926563855 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0110012ad0 con 0x557310cc96f0 2024-05-12T22:16:53.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.728+0000 7f00ff400640 1 -- 172.21.15.28:0/3926563855 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f010c03bbb0 con 0x557310cc96f0 2024-05-12T22:16:53.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.728+0000 7f0127a22740 10 client.26971 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:16:53.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.728+0000 7f0127a22740 10 client.26971 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:16:53.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.728+0000 7f0127a22740 20 client.26971 populate_metadata read hostname 'smithi028' 2024-05-12T22:16:53.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.728+0000 7f0127a22740 1 --2- 172.21.15.28:0/3926563855 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x557310ce38c0 0x557310ce5ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:53.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.728+0000 7f0127a22740 4 client.26971 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-12T22:16:53.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.728+0000 7f0127a22740 1 -- 172.21.15.28:0/3926563855 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557310c3b2c0 con 0x557310ce38c0 2024-05-12T22:16:53.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.729+0000 7f011de00640 1 --2- 172.21.15.28:0/3926563855 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x557310ce38c0 0x557310ce5ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:53.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.729+0000 7f011de00640 1 --2- 172.21.15.28:0/3926563855 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x557310ce38c0 0x557310ce5ce0 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dae4609023714c4a server_cookie=35b1b556f52aaa86 in_seq=0 out_seq=0 2024-05-12T22:16:53.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.729+0000 7f00ff400640 10 client.26971 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:16:53.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.730+0000 7f00ff400640 1 -- 172.21.15.28:0/3926563855 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557310c3b2c0 con 0x557310ce38c0 2024-05-12T22:16:53.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.730+0000 7f00ff400640 10 client.26971 handle_command_reply: tid=0 2024-05-12T22:16:53.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.731+0000 7f0127a22740 1 client.26971 shutdown 2024-05-12T22:16:53.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.731+0000 7f0127a22740 1 -- 172.21.15.28:0/3926563855 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557310cc96f0 msgr2=0x557310cd0920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:16:53.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.731+0000 7f0127a22740 1 --2- 172.21.15.28:0/3926563855 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557310cc96f0 0x557310cd0920 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f010c0ed0e0 tx=0x7f010c0092b0 comp rx=0 tx=0).stop 2024-05-12T22:16:53.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.731+0000 7f0127a22740 1 -- 172.21.15.28:0/3926563855 shutdown_connections 2024-05-12T22:16:53.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.731+0000 7f0127a22740 1 --2- 172.21.15.28:0/3926563855 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x557310ce38c0 0x557310ce5ce0 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:53.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.731+0000 7f0127a22740 1 --2- 172.21.15.28:0/3926563855 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557310cd3dc0 0x557310cd0e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:53.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.731+0000 7f0127a22740 1 --2- 172.21.15.28:0/3926563855 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557310cc96f0 0x557310cd0920 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:53.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.731+0000 7f0127a22740 1 --2- 172.21.15.28:0/3926563855 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x557310cc8dd0 0x557310cd3630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:53.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.731+0000 7f0127a22740 1 -- 172.21.15.28:0/3926563855 >> 172.21.15.28:0/3926563855 conn(0x557310cc50f0 msgr2=0x557310cc4330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:16:53.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.731+0000 7f00ff400640 0 client.26971 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:16:53.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.731+0000 7f0127a22740 1 -- 172.21.15.28:0/3926563855 shutdown_connections 2024-05-12T22:16:53.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.731+0000 7f0127a22740 1 -- 172.21.15.28:0/3926563855 wait complete. 2024-05-12T22:16:53.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.731+0000 7f0127a22740 20 client.26971 trim_cache size 0 max 16384 2024-05-12T22:16:53.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.756+0000 7f0127a22740 1 Processor -- start 2024-05-12T22:16:53.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.756+0000 7f0127a22740 1 -- start start 2024-05-12T22:16:53.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.756+0000 7f0127a22740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557310cc8dd0 0x557310cc4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:53.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.756+0000 7f0127a22740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x557310cc96f0 0x557310ccadf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:53.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.756+0000 7f0127a22740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557310cd3dc0 0x557310cd3160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:53.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.756+0000 7f0127a22740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f01201b6100 con 0x557310cd3dc0 2024-05-12T22:16:53.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.756+0000 7f0127a22740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f01201b39d0 con 0x557310cc96f0 2024-05-12T22:16:53.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.756+0000 7f0127a22740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f01201b1840 con 0x557310cc8dd0 2024-05-12T22:16:53.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.756+0000 7f011de00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557310cc8dd0 0x557310cc4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:53.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.756+0000 7f011d400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x557310cc96f0 0x557310ccadf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:53.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.756+0000 7f011de00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557310cc8dd0 0x557310cc4ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:59480/0 (socket says 172.21.15.28:59480) 2024-05-12T22:16:53.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.756+0000 7f011de00640 1 -- 172.21.15.28:0/1475238756 learned_addr learned my addr 172.21.15.28:0/1475238756 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:16:53.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.757+0000 7f011e800640 1 --2- 172.21.15.28:0/1475238756 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557310cd3dc0 0x557310cd3160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:53.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.757+0000 7f011de00640 1 -- 172.21.15.28:0/1475238756 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x557310cc96f0 msgr2=0x557310ccadf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:16:53.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.757+0000 7f011de00640 1 --2- 172.21.15.28:0/1475238756 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x557310cc96f0 0x557310ccadf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:53.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.757+0000 7f011de00640 1 -- 172.21.15.28:0/1475238756 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557310cd3dc0 msgr2=0x557310cd3160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:16:53.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.757+0000 7f011de00640 1 --2- 172.21.15.28:0/1475238756 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557310cd3dc0 0x557310cd3160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:53.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.757+0000 7f011de00640 1 -- 172.21.15.28:0/1475238756 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557310cd1190 con 0x557310cc8dd0 2024-05-12T22:16:53.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.757+0000 7f011e800640 1 --2- 172.21.15.28:0/1475238756 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557310cd3dc0 0x557310cd3160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-12T22:16:53.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.757+0000 7f011de00640 1 --2- 172.21.15.28:0/1475238756 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557310cc8dd0 0x557310cc4ab0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f011000a010 tx=0x7f0110009910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4a1971c1f0927aaf server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:16:53.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.757+0000 7f010a600640 1 -- 172.21.15.28:0/1475238756 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f011000f440 con 0x557310cc8dd0 2024-05-12T22:16:53.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.758+0000 7f010a600640 1 -- 172.21.15.28:0/1475238756 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f011000fd80 con 0x557310cc8dd0 2024-05-12T22:16:53.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.758+0000 7f010a600640 1 -- 172.21.15.28:0/1475238756 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0110003870 con 0x557310cc8dd0 2024-05-12T22:16:53.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.758+0000 7f0127a22740 1 -- 172.21.15.28:0/1475238756 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557310cc8dd0 msgr2=0x557310cc4ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:16:53.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.758+0000 7f0127a22740 1 --2- 172.21.15.28:0/1475238756 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557310cc8dd0 0x557310cc4ab0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f011000a010 tx=0x7f0110009910 comp rx=0 tx=0).stop 2024-05-12T22:16:53.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.758+0000 7f0127a22740 1 -- 172.21.15.28:0/1475238756 shutdown_connections 2024-05-12T22:16:53.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.759+0000 7f0127a22740 1 --2- 172.21.15.28:0/1475238756 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557310cd3dc0 0x557310cd3160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:53.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.759+0000 7f0127a22740 1 --2- 172.21.15.28:0/1475238756 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x557310cc96f0 0x557310ccadf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:53.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.759+0000 7f0127a22740 1 --2- 172.21.15.28:0/1475238756 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557310cc8dd0 0x557310cc4ab0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:53.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.759+0000 7f0127a22740 1 -- 172.21.15.28:0/1475238756 >> 172.21.15.28:0/1475238756 conn(0x557310cc50f0 msgr2=0x557310ccbe90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:16:53.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.759+0000 7f0127a22740 1 -- 172.21.15.28:0/1475238756 shutdown_connections 2024-05-12T22:16:53.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.759+0000 7f0127a22740 1 -- 172.21.15.28:0/1475238756 wait complete. 2024-05-12T22:16:53.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.759+0000 7f0127a22740 1 -- start start 2024-05-12T22:16:53.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.759+0000 7f0127a22740 1 Processor -- start 2024-05-12T22:16:53.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.760+0000 7f0127a22740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x557310cc8dd0 0x557310ccd980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:53.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.760+0000 7f0127a22740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557310cc96f0 0x557310ccdec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:53.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.760+0000 7f0127a22740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557310cd3dc0 0x557310d48e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:53.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.760+0000 7f0127a22740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f01201b1e90 con 0x557310cc96f0 2024-05-12T22:16:53.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.760+0000 7f0127a22740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f01201b3300 con 0x557310cc8dd0 2024-05-12T22:16:53.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.760+0000 7f0127a22740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f01201b2560 con 0x557310cd3dc0 2024-05-12T22:16:53.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.760+0000 7f0127a22740 10 client.0 _ll_register_callbacks cb 0x7f010c0f5220 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:16:53.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.760+0000 7f011de00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x557310cc8dd0 0x557310ccd980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:53.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.760+0000 7f011de00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x557310cc8dd0 0x557310ccd980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:49366/0 (socket says 172.21.15.28:49366) 2024-05-12T22:16:53.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.760+0000 7f011de00640 1 -- 172.21.15.28:0/2690244584 learned_addr learned my addr 172.21.15.28:0/2690244584 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:16:53.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.760+0000 7f011e800640 1 --2- 172.21.15.28:0/2690244584 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557310cd3dc0 0x557310d48e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:53.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.761+0000 7f011d400640 1 --2- 172.21.15.28:0/2690244584 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557310cc96f0 0x557310ccdec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:53.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.761+0000 7f011de00640 1 -- 172.21.15.28:0/2690244584 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557310cd3dc0 msgr2=0x557310d48e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:16:53.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.761+0000 7f011de00640 1 --2- 172.21.15.28:0/2690244584 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557310cd3dc0 0x557310d48e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:53.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.761+0000 7f011de00640 1 -- 172.21.15.28:0/2690244584 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557310cc96f0 msgr2=0x557310ccdec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:16:53.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.761+0000 7f011de00640 1 --2- 172.21.15.28:0/2690244584 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557310cc96f0 0x557310ccdec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:53.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.761+0000 7f011de00640 1 -- 172.21.15.28:0/2690244584 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557310cce790 con 0x557310cc8dd0 2024-05-12T22:16:53.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.761+0000 7f011d400640 1 --2- 172.21.15.28:0/2690244584 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557310cc96f0 0x557310ccdec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:16:53.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.761+0000 7f011de00640 1 --2- 172.21.15.28:0/2690244584 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x557310cc8dd0 0x557310ccd980 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f0110009b30 tx=0x7f0110009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=799351d80778f563 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:16:53.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.761+0000 7f00ff400640 10 client.0 ms_handle_connect on v2:172.21.15.86:3300/0 2024-05-12T22:16:53.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.762+0000 7f00ff400640 1 -- 172.21.15.28:0/2690244584 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01100052c0 con 0x557310cc8dd0 2024-05-12T22:16:53.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.762+0000 7f00ff400640 1 -- 172.21.15.28:0/2690244584 <== mon.2 v2:172.21.15.86:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8513+0+0 (secure 0 0 0) 0x7f011000f770 con 0x557310cc8dd0 2024-05-12T22:16:53.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.762+0000 7f0127a22740 1 -- 172.21.15.28:0/2690244584 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f011000fd80 con 0x557310cc8dd0 2024-05-12T22:16:53.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.762+0000 7f00ff400640 1 -- 172.21.15.28:0/2690244584 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f011000f440 con 0x557310cc8dd0 2024-05-12T22:16:53.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.762+0000 7f0127a22740 10 client.27023 fetch_fsmap learned FSMap version 26 2024-05-12T22:16:53.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.762+0000 7f0127a22740 1 -- 172.21.15.28:0/2690244584 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x557310d4a3e0 con 0x557310cc8dd0 2024-05-12T22:16:53.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.762+0000 7f00ff400640 1 -- 172.21.15.28:0/2690244584 <== mon.2 v2:172.21.15.86:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f011000fd80 con 0x557310cc8dd0 2024-05-12T22:16:53.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.763+0000 7f00ff400640 1 -- 172.21.15.28:0/2690244584 <== mon.2 v2:172.21.15.86:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f0110039510 con 0x557310cc8dd0 2024-05-12T22:16:53.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.763+0000 7f0127a22740 10 client.27023 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:16:53.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.763+0000 7f0127a22740 10 client.27023 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:16:53.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.763+0000 7f0127a22740 20 client.27023 populate_metadata read hostname 'smithi028' 2024-05-12T22:16:53.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.763+0000 7f0127a22740 1 --2- 172.21.15.28:0/2690244584 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x557310d4a960 0x557310d4cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:16:53.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.763+0000 7f0127a22740 4 client.27023 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-12T22:16:53.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.763+0000 7f0127a22740 1 -- 172.21.15.28:0/2690244584 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "scrub status"}) -- 0x557310c35610 con 0x557310d4a960 2024-05-12T22:16:53.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.763+0000 7f011d400640 1 --2- 172.21.15.28:0/2690244584 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x557310d4a960 0x557310d4cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:16:53.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.764+0000 7f011d400640 1 --2- 172.21.15.28:0/2690244584 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x557310d4a960 0x557310d4cd50 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d9099340a32242e8 server_cookie=f3f42c25f730015b in_seq=0 out_seq=0 2024-05-12T22:16:53.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.764+0000 7f00ff400640 10 client.27023 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:16:53.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.764+0000 7f00ff400640 1 -- 172.21.15.28:0/2690244584 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+268 (crc 0 0 0) 0x557310c35610 con 0x557310d4a960 2024-05-12T22:16:53.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.764+0000 7f00ff400640 10 client.27023 handle_command_reply: tid=0 2024-05-12T22:16:53.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.764+0000 7f0127a22740 1 client.27023 shutdown 2024-05-12T22:16:53.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.764+0000 7f0127a22740 1 -- 172.21.15.28:0/2690244584 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x557310cc8dd0 msgr2=0x557310ccd980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:16:53.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.764+0000 7f0127a22740 1 --2- 172.21.15.28:0/2690244584 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x557310cc8dd0 0x557310ccd980 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f0110009b30 tx=0x7f0110009780 comp rx=0 tx=0).stop 2024-05-12T22:16:53.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.765+0000 7f0127a22740 1 -- 172.21.15.28:0/2690244584 shutdown_connections 2024-05-12T22:16:53.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.765+0000 7f0127a22740 1 --2- 172.21.15.28:0/2690244584 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x557310d4a960 0x557310d4cd50 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:53.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.765+0000 7f00ff400640 0 client.27023 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:16:53.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.765+0000 7f0127a22740 1 --2- 172.21.15.28:0/2690244584 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557310cd3dc0 0x557310d48e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:53.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.765+0000 7f0127a22740 1 --2- 172.21.15.28:0/2690244584 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557310cc96f0 0x557310ccdec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:53.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.765+0000 7f0127a22740 1 --2- 172.21.15.28:0/2690244584 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x557310cc8dd0 0x557310ccd980 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:53.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.765+0000 7f0127a22740 1 -- 172.21.15.28:0/2690244584 >> 172.21.15.28:0/2690244584 conn(0x557310cc50f0 msgr2=0x557310ccadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:16:53.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.765+0000 7f0127a22740 1 -- 172.21.15.28:0/2690244584 shutdown_connections 2024-05-12T22:16:53.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.765+0000 7f0127a22740 1 -- 172.21.15.28:0/2690244584 wait complete. 2024-05-12T22:16:53.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.765+0000 7f0127a22740 20 client.27023 trim_cache size 0 max 16384 2024-05-12T22:16:53.774 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-05-12T22:16:53.774 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "scrub active (1364 inodes in the stack)", 2024-05-12T22:16:53.775 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-05-12T22:16:53.775 INFO:teuthology.orchestra.run.smithi028.stdout: "c1467dff-508e-4ec1-a23e-f353c76045ae": { 2024-05-12T22:16:53.775 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-05-12T22:16:53.775 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "c1467dff-508e-4ec1-a23e-f353c76045ae", 2024-05-12T22:16:53.775 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-05-12T22:16:53.775 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:16:53.775 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:16:53.775 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-05-12T22:16:53.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.766+0000 7f011f200640 1 -- 172.21.15.28:0/4133261371 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f01040a2d80 msgr2=0x7f01040a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:16:53.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.766+0000 7f011f200640 1 --2- 172.21.15.28:0/4133261371 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f01040a2d80 0x7f01040a5240 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f0118005fd0 tx=0x7f01180095f0 comp rx=0 tx=0).stop 2024-05-12T22:16:53.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.766+0000 7f011f200640 1 -- 172.21.15.28:0/4133261371 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0120108770 msgr2=0x7f012019eaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:16:53.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.766+0000 7f011f200640 1 --2- 172.21.15.28:0/4133261371 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0120108770 0x7f012019eaf0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f010c006010 tx=0x7f010c0042f0 comp rx=0 tx=0).stop 2024-05-12T22:16:53.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.766+0000 7f011f200640 1 -- 172.21.15.28:0/4133261371 shutdown_connections 2024-05-12T22:16:53.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.766+0000 7f011f200640 1 --2- 172.21.15.28:0/4133261371 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f01040a2d80 0x7f01040a5240 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:53.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.766+0000 7f011f200640 1 --2- 172.21.15.28:0/4133261371 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f012010ddf0 0x7f01201a2e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:53.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.766+0000 7f011f200640 1 --2- 172.21.15.28:0/4133261371 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0120108770 0x7f012019eaf0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:53.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.766+0000 7f011f200640 1 --2- 172.21.15.28:0/4133261371 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0120107dd0 0x7f012019e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:16:53.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.766+0000 7f011f200640 1 -- 172.21.15.28:0/4133261371 >> 172.21.15.28:0/4133261371 conn(0x7f0120077650 msgr2=0x7f0120078760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:16:53.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.766+0000 7f011f200640 1 -- 172.21.15.28:0/4133261371 shutdown_connections 2024-05-12T22:16:53.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:16:53.766+0000 7f011f200640 1 -- 172.21.15.28:0/4133261371 wait complete. 2024-05-12T22:16:53.788 INFO:tasks.cephfs.filesystem:scrub status for tag:c1467dff-508e-4ec1-a23e-f353c76045ae - {'path': '/', 'tag': 'c1467dff-508e-4ec1-a23e-f353c76045ae', 'options': 'recursive,force'} 2024-05-12T22:16:54.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:16:54 smithi028 ceph-mon[29847]: pgmap v242: 97 pgs: 97 active+clean; 97 MiB data, 757 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:16:54.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:16:54 smithi028 ceph-mon[29847]: scrub queued for path: / 2024-05-12T22:16:54.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:16:54 smithi028 ceph-mon[29847]: scrub summary: idle+waiting paths [/] 2024-05-12T22:16:54.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:16:54 smithi028 ceph-mon[29847]: scrub summary: active paths [/] 2024-05-12T22:16:54.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:16:54 smithi028 ceph-mon[29847]: scrub summary: active paths [/] 2024-05-12T22:16:54.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:16:54 smithi028 ceph-mon[29847]: scrub summary: idle+waiting paths [/] 2024-05-12T22:16:54.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:16:54 smithi028 ceph-mon[29847]: scrub summary: active paths [/] 2024-05-12T22:16:54.537 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:16:54 smithi028 ceph-mon[29847]: scrub summary: idle+waiting paths [/] 2024-05-12T22:16:54.537 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:16:54 smithi028 ceph-mon[29847]: scrub summary: active paths [/] 2024-05-12T22:16:54.537 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:16:54 smithi028 ceph-mon[29847]: scrub summary: idle+waiting paths [/] 2024-05-12T22:16:54.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:16:54 smithi186 ceph-mon[34785]: pgmap v242: 97 pgs: 97 active+clean; 97 MiB data, 757 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:16:54.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:16:54 smithi186 ceph-mon[34785]: scrub queued for path: / 2024-05-12T22:16:54.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:16:54 smithi186 ceph-mon[34785]: scrub summary: idle+waiting paths [/] 2024-05-12T22:16:54.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:16:54 smithi186 ceph-mon[34785]: scrub summary: active paths [/] 2024-05-12T22:16:54.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:16:54 smithi186 ceph-mon[34785]: scrub summary: active paths [/] 2024-05-12T22:16:54.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:16:54 smithi186 ceph-mon[34785]: scrub summary: idle+waiting paths [/] 2024-05-12T22:16:54.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:16:54 smithi186 ceph-mon[34785]: scrub summary: active paths [/] 2024-05-12T22:16:54.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:16:54 smithi186 ceph-mon[34785]: scrub summary: idle+waiting paths [/] 2024-05-12T22:16:54.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:16:54 smithi186 ceph-mon[34785]: scrub summary: active paths [/] 2024-05-12T22:16:54.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:16:54 smithi186 ceph-mon[34785]: scrub summary: idle+waiting paths [/] 2024-05-12T22:16:54.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:16:54 smithi086 ceph-mon[34297]: pgmap v242: 97 pgs: 97 active+clean; 97 MiB data, 757 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:16:54.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:16:54 smithi086 ceph-mon[34297]: scrub queued for path: / 2024-05-12T22:16:54.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:16:54 smithi086 ceph-mon[34297]: scrub summary: idle+waiting paths [/] 2024-05-12T22:16:54.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:16:54 smithi086 ceph-mon[34297]: scrub summary: active paths [/] 2024-05-12T22:16:54.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:16:54 smithi086 ceph-mon[34297]: scrub summary: active paths [/] 2024-05-12T22:16:54.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:16:54 smithi086 ceph-mon[34297]: scrub summary: idle+waiting paths [/] 2024-05-12T22:16:54.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:16:54 smithi086 ceph-mon[34297]: scrub summary: active paths [/] 2024-05-12T22:16:54.743 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:16:54 smithi086 ceph-mon[34297]: scrub summary: idle+waiting paths [/] 2024-05-12T22:16:54.743 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:16:54 smithi086 ceph-mon[34297]: scrub summary: active paths [/] 2024-05-12T22:16:54.743 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:16:54 smithi086 ceph-mon[34297]: scrub summary: idle+waiting paths [/] 2024-05-12T22:16:55.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:16:55 smithi028 ceph-mon[29847]: scrub summary: idle+waiting paths [/] 2024-05-12T22:16:55.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:16:55 smithi186 ceph-mon[34785]: scrub summary: idle+waiting paths [/] 2024-05-12T22:16:55.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:16:55 smithi086 ceph-mon[34297]: scrub summary: idle+waiting paths [/] 2024-05-12T22:16:56.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:16:56 smithi028 ceph-mon[29847]: pgmap v243: 97 pgs: 97 active+clean; 97 MiB data, 757 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-05-12T22:16:56.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:16:56 smithi186 ceph-mon[34785]: pgmap v243: 97 pgs: 97 active+clean; 97 MiB data, 757 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-05-12T22:16:56.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:16:56 smithi086 ceph-mon[34297]: pgmap v243: 97 pgs: 97 active+clean; 97 MiB data, 757 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-05-12T22:16:58.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:16:58 smithi028 ceph-mon[29847]: pgmap v244: 97 pgs: 97 active+clean; 97 MiB data, 757 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-05-12T22:16:58.681 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:16:58 smithi186 ceph-mon[34785]: pgmap v244: 97 pgs: 97 active+clean; 97 MiB data, 757 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-05-12T22:16:58.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:16:58 smithi086 ceph-mon[34297]: pgmap v244: 97 pgs: 97 active+clean; 97 MiB data, 757 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-05-12T22:17:00.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:17:00 smithi028 ceph-mon[29847]: pgmap v245: 97 pgs: 97 active+clean; 97 MiB data, 757 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-05-12T22:17:00.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:17:00 smithi186 ceph-mon[34785]: pgmap v245: 97 pgs: 97 active+clean; 97 MiB data, 757 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-05-12T22:17:00.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:17:00 smithi086 ceph-mon[34297]: pgmap v245: 97 pgs: 97 active+clean; 97 MiB data, 757 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-05-12T22:17:02.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:17:02 smithi186 ceph-mon[34785]: pgmap v246: 97 pgs: 97 active+clean; 97 MiB data, 757 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-05-12T22:17:02.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:17:02 smithi086 ceph-mon[34297]: pgmap v246: 97 pgs: 97 active+clean; 97 MiB data, 757 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-05-12T22:17:02.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:17:02 smithi028 ceph-mon[29847]: pgmap v246: 97 pgs: 97 active+clean; 97 MiB data, 757 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-05-12T22:17:03.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:17:03 smithi186 ceph-mon[34785]: scrub summary: idle 2024-05-12T22:17:03.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:17:03 smithi186 ceph-mon[34785]: scrub summary: idle 2024-05-12T22:17:03.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:17:03 smithi086 ceph-mon[34297]: scrub summary: idle 2024-05-12T22:17:03.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:17:03 smithi086 ceph-mon[34297]: scrub summary: idle 2024-05-12T22:17:03.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:17:03 smithi028 ceph-mon[29847]: scrub summary: idle 2024-05-12T22:17:03.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:17:03 smithi028 ceph-mon[29847]: scrub summary: idle 2024-05-12T22:17:04.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:17:04 smithi186 ceph-mon[34785]: pgmap v247: 97 pgs: 97 active+clean; 97 MiB data, 757 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-05-12T22:17:04.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:17:04 smithi086 ceph-mon[34297]: pgmap v247: 97 pgs: 97 active+clean; 97 MiB data, 757 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-05-12T22:17:04.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:17:04 smithi028 ceph-mon[29847]: pgmap v247: 97 pgs: 97 active+clean; 97 MiB data, 757 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-05-12T22:17:06.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:17:06 smithi186 ceph-mon[34785]: pgmap v248: 97 pgs: 97 active+clean; 97 MiB data, 758 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-05-12T22:17:06.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:17:06 smithi086 ceph-mon[34297]: pgmap v248: 97 pgs: 97 active+clean; 97 MiB data, 758 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-05-12T22:17:06.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:17:06 smithi028 ceph-mon[29847]: pgmap v248: 97 pgs: 97 active+clean; 97 MiB data, 758 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-05-12T22:17:08.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:17:08 smithi186 ceph-mon[34785]: pgmap v249: 97 pgs: 97 active+clean; 97 MiB data, 758 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-05-12T22:17:08.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:17:08 smithi086 ceph-mon[34297]: pgmap v249: 97 pgs: 97 active+clean; 97 MiB data, 758 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-05-12T22:17:08.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:17:08 smithi028 ceph-mon[29847]: pgmap v249: 97 pgs: 97 active+clean; 97 MiB data, 758 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-05-12T22:17:09.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:17:09 smithi186 ceph-mon[34785]: osdmap e92: 12 total, 12 up, 12 in 2024-05-12T22:17:09.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:17:09 smithi086 ceph-mon[34297]: osdmap e92: 12 total, 12 up, 12 in 2024-05-12T22:17:09.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:17:09 smithi028 ceph-mon[29847]: osdmap e92: 12 total, 12 up, 12 in 2024-05-12T22:17:10.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:17:10 smithi186 ceph-mon[34785]: pgmap v251: 97 pgs: 97 active+clean; 97 MiB data, 758 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-05-12T22:17:10.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:17:10 smithi086 ceph-mon[34297]: pgmap v251: 97 pgs: 97 active+clean; 97 MiB data, 758 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-05-12T22:17:10.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:17:10 smithi028 ceph-mon[29847]: pgmap v251: 97 pgs: 97 active+clean; 97 MiB data, 758 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-05-12T22:17:12.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:17:12 smithi186 ceph-mon[34785]: pgmap v252: 97 pgs: 97 active+clean; 91 MiB data, 709 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-05-12T22:17:12.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:17:12 smithi086 ceph-mon[34297]: pgmap v252: 97 pgs: 97 active+clean; 91 MiB data, 709 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-05-12T22:17:12.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:17:12 smithi028 ceph-mon[29847]: pgmap v252: 97 pgs: 97 active+clean; 91 MiB data, 709 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-05-12T22:17:13.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:17:13 smithi186 ceph-mon[34785]: pgmap v253: 97 pgs: 97 active+clean; 91 MiB data, 709 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-05-12T22:17:13.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:17:13 smithi086 ceph-mon[34297]: pgmap v253: 97 pgs: 97 active+clean; 91 MiB data, 709 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-05-12T22:17:13.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:17:13 smithi028 ceph-mon[29847]: pgmap v253: 97 pgs: 97 active+clean; 91 MiB data, 709 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-05-12T22:17:15.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:17:14 smithi186 ceph-mon[34785]: osdmap e93: 12 total, 12 up, 12 in 2024-05-12T22:17:15.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:17:14 smithi086 ceph-mon[34297]: osdmap e93: 12 total, 12 up, 12 in 2024-05-12T22:17:15.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:17:14 smithi028 ceph-mon[29847]: osdmap e93: 12 total, 12 up, 12 in 2024-05-12T22:17:16.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:17:15 smithi086 ceph-mon[34297]: pgmap v255: 97 pgs: 97 active+clean; 91 MiB data, 708 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2024-05-12T22:17:16.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:17:15 smithi028 ceph-mon[29847]: pgmap v255: 97 pgs: 97 active+clean; 91 MiB data, 708 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2024-05-12T22:17:16.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:17:15 smithi186 ceph-mon[34785]: pgmap v255: 97 pgs: 97 active+clean; 91 MiB data, 708 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2024-05-12T22:17:18.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:17:17 smithi086 ceph-mon[34297]: pgmap v256: 97 pgs: 97 active+clean; 91 MiB data, 708 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 239 B/s wr, 7 op/s 2024-05-12T22:17:18.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:17:17 smithi028 ceph-mon[29847]: pgmap v256: 97 pgs: 97 active+clean; 91 MiB data, 708 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 239 B/s wr, 7 op/s 2024-05-12T22:17:18.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:17:17 smithi186 ceph-mon[34785]: pgmap v256: 97 pgs: 97 active+clean; 91 MiB data, 708 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 239 B/s wr, 7 op/s 2024-05-12T22:17:20.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:17:19 smithi086 ceph-mon[34297]: pgmap v257: 97 pgs: 97 active+clean; 91 MiB data, 708 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-05-12T22:17:20.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:17:19 smithi086 ceph-mon[34297]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:17:20.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:17:19 smithi028 ceph-mon[29847]: pgmap v257: 97 pgs: 97 active+clean; 91 MiB data, 708 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-05-12T22:17:20.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:17:19 smithi028 ceph-mon[29847]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:17:20.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:17:19 smithi186 ceph-mon[34785]: pgmap v257: 97 pgs: 97 active+clean; 91 MiB data, 708 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-05-12T22:17:20.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:17:19 smithi186 ceph-mon[34785]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:17:22.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:17:21 smithi086 ceph-mon[34297]: pgmap v258: 97 pgs: 97 active+clean; 91 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-05-12T22:17:22.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:17:21 smithi086 ceph-mon[34297]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:17:22.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:17:21 smithi086 ceph-mon[34297]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:17:22.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:17:21 smithi086 ceph-mon[34297]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:17:22.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:17:21 smithi028 ceph-mon[29847]: pgmap v258: 97 pgs: 97 active+clean; 91 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-05-12T22:17:22.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:17:21 smithi028 ceph-mon[29847]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:17:22.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:17:21 smithi028 ceph-mon[29847]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:17:22.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:17:21 smithi028 ceph-mon[29847]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:17:22.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:17:21 smithi186 ceph-mon[34785]: pgmap v258: 97 pgs: 97 active+clean; 91 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-05-12T22:17:22.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:17:21 smithi186 ceph-mon[34785]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:17:22.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:17:21 smithi186 ceph-mon[34785]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:17:22.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:17:21 smithi186 ceph-mon[34785]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:17:23.788 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-12T22:17:23.789 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-12T22:17:24.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.019+0000 7fc1f0c00640 1 -- 172.21.15.28:0/3472408628 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1ec10df50 msgr2=0x7fc1ec110360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:17:24.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.019+0000 7fc1f0c00640 1 --2- 172.21.15.28:0/3472408628 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1ec10df50 0x7fc1ec110360 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fc1e4009ca0 tx=0x7fc1e402a950 comp rx=0 tx=0).stop 2024-05-12T22:17:24.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.019+0000 7fc1f0c00640 1 -- 172.21.15.28:0/3472408628 shutdown_connections 2024-05-12T22:17:24.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.019+0000 7fc1f0c00640 1 --2- 172.21.15.28:0/3472408628 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1ec10df50 0x7fc1ec110360 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:24.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.019+0000 7fc1f0c00640 1 --2- 172.21.15.28:0/3472408628 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc1ec074520 0x7fc1ec10da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:24.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.019+0000 7fc1f0c00640 1 --2- 172.21.15.28:0/3472408628 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc1ec073c00 0x7fc1ec073fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:24.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.019+0000 7fc1f0c00640 1 -- 172.21.15.28:0/3472408628 >> 172.21.15.28:0/3472408628 conn(0x7fc1ec0fd200 msgr2=0x7fc1ec0ff620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:17:24.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.019+0000 7fc1f0c00640 1 -- 172.21.15.28:0/3472408628 shutdown_connections 2024-05-12T22:17:24.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.019+0000 7fc1f0c00640 1 -- 172.21.15.28:0/3472408628 wait complete. 2024-05-12T22:17:24.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.020+0000 7fc1f0c00640 1 Processor -- start 2024-05-12T22:17:24.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.020+0000 7fc1f0c00640 1 -- start start 2024-05-12T22:17:24.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.021+0000 7fc1f0c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1ec073c00 0x7fc1ec19a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:24.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.021+0000 7fc1f0c00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc1ec074520 0x7fc1ec19a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:24.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.021+0000 7fc1f0c00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc1ec10df50 0x7fc1ec19ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:24.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.021+0000 7fc1f0c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc1ec113d00 con 0x7fc1ec073c00 2024-05-12T22:17:24.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.021+0000 7fc1f0c00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fc1ec113b80 con 0x7fc1ec074520 2024-05-12T22:17:24.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.021+0000 7fc1f0c00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc1ec113e80 con 0x7fc1ec10df50 2024-05-12T22:17:24.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.021+0000 7fc1eb400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1ec073c00 0x7fc1ec19a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:24.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.021+0000 7fc1ebe00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc1ec10df50 0x7fc1ec19ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:24.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.021+0000 7fc1eb400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1ec073c00 0x7fc1ec19a1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48838/0 (socket says 172.21.15.28:48838) 2024-05-12T22:17:24.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.021+0000 7fc1eb400640 1 -- 172.21.15.28:0/2331070769 learned_addr learned my addr 172.21.15.28:0/2331070769 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:17:24.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.021+0000 7fc1eaa00640 1 --2- 172.21.15.28:0/2331070769 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc1ec074520 0x7fc1ec19a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:24.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.022+0000 7fc1ebe00640 1 -- 172.21.15.28:0/2331070769 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc1ec074520 msgr2=0x7fc1ec19a700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:17:24.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.022+0000 7fc1ebe00640 1 --2- 172.21.15.28:0/2331070769 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc1ec074520 0x7fc1ec19a700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:24.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.022+0000 7fc1ebe00640 1 -- 172.21.15.28:0/2331070769 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1ec073c00 msgr2=0x7fc1ec19a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:17:24.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.022+0000 7fc1ebe00640 1 --2- 172.21.15.28:0/2331070769 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1ec073c00 0x7fc1ec19a1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:24.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.022+0000 7fc1ebe00640 1 -- 172.21.15.28:0/2331070769 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1ec19f1b0 con 0x7fc1ec10df50 2024-05-12T22:17:24.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.022+0000 7fc1eb400640 1 --2- 172.21.15.28:0/2331070769 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1ec073c00 0x7fc1ec19a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:17:24.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.022+0000 7fc1ebe00640 1 --2- 172.21.15.28:0/2331070769 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc1ec10df50 0x7fc1ec19ea80 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fc1e402ae70 tx=0x7fc1e4003b80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:17:24.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.022+0000 7fc1e3e00640 1 -- 172.21.15.28:0/2331070769 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1e4034040 con 0x7fc1ec10df50 2024-05-12T22:17:24.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.023+0000 7fc1f0c00640 1 -- 172.21.15.28:0/2331070769 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1ec19f440 con 0x7fc1ec10df50 2024-05-12T22:17:24.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.023+0000 7fc1e3e00640 1 -- 172.21.15.28:0/2331070769 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc1e402f940 con 0x7fc1ec10df50 2024-05-12T22:17:24.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.023+0000 7fc1e3e00640 1 -- 172.21.15.28:0/2331070769 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1e4004670 con 0x7fc1ec10df50 2024-05-12T22:17:24.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.023+0000 7fc1f0c00640 1 -- 172.21.15.28:0/2331070769 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1ec1a6d30 con 0x7fc1ec10df50 2024-05-12T22:17:24.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.026+0000 7fc1f5181740 1 Processor -- start 2024-05-12T22:17:24.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.026+0000 7fc1e3e00640 1 -- 172.21.15.28:0/2331070769 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7fc1e4037020 con 0x7fc1ec10df50 2024-05-12T22:17:24.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.026+0000 7fc1f5181740 1 -- start start 2024-05-12T22:17:24.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.026+0000 7fc1f5181740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56021d7c5dd0 0x56021d7c61b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:24.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.026+0000 7fc1f5181740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56021d7c66f0 0x56021d7d0880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:24.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.026+0000 7fc1f5181740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56021d7d0dc0 0x56021d7d31b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:24.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.027+0000 7fc1f5181740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56021d78e2a0 con 0x56021d7c5dd0 2024-05-12T22:17:24.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.027+0000 7fc1f5181740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x56021d72ab90 con 0x56021d7d0dc0 2024-05-12T22:17:24.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.027+0000 7fc1f5181740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x56021d6d25b0 con 0x56021d7c66f0 2024-05-12T22:17:24.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.027+0000 7fc1eaa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56021d7c5dd0 0x56021d7c61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:24.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.027+0000 7fc1eaa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56021d7c5dd0 0x56021d7c61b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48840/0 (socket says 172.21.15.28:48840) 2024-05-12T22:17:24.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.027+0000 7fc1eaa00640 1 -- 172.21.15.28:0/3586695438 learned_addr learned my addr 172.21.15.28:0/3586695438 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:17:24.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.027+0000 7fc1eb400640 1 --2- 172.21.15.28:0/3586695438 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56021d7c66f0 0x56021d7d0880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:24.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.027+0000 7fc1eaa00640 1 --2- 172.21.15.28:0/3586695438 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56021d7d0dc0 0x56021d7d31b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:24.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.027+0000 7fc1eb400640 1 -- 172.21.15.28:0/3586695438 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56021d7d0dc0 msgr2=0x56021d7d31b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:17:24.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.027+0000 7fc1eb400640 1 --2- 172.21.15.28:0/3586695438 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56021d7d0dc0 0x56021d7d31b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:24.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.027+0000 7fc1eb400640 1 -- 172.21.15.28:0/3586695438 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56021d7c5dd0 msgr2=0x56021d7c61b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:17:24.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.027+0000 7fc1eb400640 1 --2- 172.21.15.28:0/3586695438 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56021d7c5dd0 0x56021d7c61b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:24.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.027+0000 7fc1eb400640 1 -- 172.21.15.28:0/3586695438 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56021d5087d0 con 0x56021d7c66f0 2024-05-12T22:17:24.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.027+0000 7fc1eaa00640 1 --2- 172.21.15.28:0/3586695438 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56021d7c5dd0 0x56021d7c61b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:17:24.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.028+0000 7fc1e3e00640 1 --2- 172.21.15.28:0/2331070769 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fc1d00a2d00 0x7fc1d00a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:24.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.028+0000 7fc1eb400640 1 --2- 172.21.15.28:0/3586695438 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56021d7c66f0 0x56021d7d0880 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fc1dc0114f0 tx=0x7fc1dc0119b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b1151610fa65d6bf server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:17:24.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.028+0000 7fc1e3e00640 1 -- 172.21.15.28:0/2331070769 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8513+0+0 (secure 0 0 0) 0x7fc1e40f0190 con 0x7fc1ec10df50 2024-05-12T22:17:24.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.028+0000 7fc1eaa00640 1 --2- 172.21.15.28:0/2331070769 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fc1d00a2d00 0x7fc1d00a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:24.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.028+0000 7fc1d7e00640 1 -- 172.21.15.28:0/3586695438 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1dc01e070 con 0x56021d7c66f0 2024-05-12T22:17:24.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.028+0000 7fc1d7e00640 1 -- 172.21.15.28:0/3586695438 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc1dc0035f0 con 0x56021d7c66f0 2024-05-12T22:17:24.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.028+0000 7fc1d7e00640 1 -- 172.21.15.28:0/3586695438 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1dc00f820 con 0x56021d7c66f0 2024-05-12T22:17:24.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.029+0000 7fc1f5181740 1 -- 172.21.15.28:0/3586695438 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56021d7c66f0 msgr2=0x56021d7d0880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:17:24.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.029+0000 7fc1f5181740 1 --2- 172.21.15.28:0/3586695438 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56021d7c66f0 0x56021d7d0880 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fc1dc0114f0 tx=0x7fc1dc0119b0 comp rx=0 tx=0).stop 2024-05-12T22:17:24.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.029+0000 7fc1eaa00640 1 --2- 172.21.15.28:0/2331070769 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fc1d00a2d00 0x7fc1d00a51c0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fc1d8009fd0 tx=0x7fc1d8005850 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:17:24.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.029+0000 7fc1f5181740 1 -- 172.21.15.28:0/3586695438 shutdown_connections 2024-05-12T22:17:24.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.029+0000 7fc1f5181740 1 --2- 172.21.15.28:0/3586695438 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56021d7d0dc0 0x56021d7d31b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:24.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.029+0000 7fc1f5181740 1 --2- 172.21.15.28:0/3586695438 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56021d7c66f0 0x56021d7d0880 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:24.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.029+0000 7fc1f5181740 1 --2- 172.21.15.28:0/3586695438 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56021d7c5dd0 0x56021d7c61b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:24.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.029+0000 7fc1f5181740 1 -- 172.21.15.28:0/3586695438 >> 172.21.15.28:0/3586695438 conn(0x56021d7c20f0 msgr2=0x56021d7c24d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:17:24.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.029+0000 7fc1f5181740 1 -- 172.21.15.28:0/3586695438 shutdown_connections 2024-05-12T22:17:24.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.029+0000 7fc1f5181740 1 -- 172.21.15.28:0/3586695438 wait complete. 2024-05-12T22:17:24.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.030+0000 7fc1f5181740 1 -- start start 2024-05-12T22:17:24.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.030+0000 7fc1f5181740 1 Processor -- start 2024-05-12T22:17:24.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.030+0000 7fc1f5181740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56021d7c5dd0 0x56021d7d0750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:24.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.030+0000 7fc1f5181740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56021d7c66f0 0x56021d7cda40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:24.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.030+0000 7fc1f5181740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56021d7d0dc0 0x56021d7cdfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:24.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.030+0000 7fc1f5181740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56021d6e59f0 con 0x56021d7d0dc0 2024-05-12T22:17:24.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.030+0000 7fc1f5181740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x56021d726b50 con 0x56021d7c5dd0 2024-05-12T22:17:24.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.030+0000 7fc1f5181740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x56021d7d3bb0 con 0x56021d7c66f0 2024-05-12T22:17:24.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.030+0000 7fc1f5181740 10 client.0 _ll_register_callbacks cb 0x56021d43dbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:17:24.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.031+0000 7fc1eaa00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56021d7c66f0 0x56021d7cda40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:24.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.031+0000 7fc1ebe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56021d7d0dc0 0x56021d7cdfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:24.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.031+0000 7fc1eaa00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56021d7c66f0 0x56021d7cda40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:57760/0 (socket says 172.21.15.28:57760) 2024-05-12T22:17:24.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.031+0000 7fc1eaa00640 1 -- 172.21.15.28:0/3607670257 learned_addr learned my addr 172.21.15.28:0/3607670257 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:17:24.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.031+0000 7fc1eb400640 1 --2- 172.21.15.28:0/3607670257 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56021d7c5dd0 0x56021d7d0750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:24.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.031+0000 7fc1ebe00640 1 -- 172.21.15.28:0/3607670257 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56021d7c66f0 msgr2=0x56021d7cda40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:17:24.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.031+0000 7fc1ebe00640 1 --2- 172.21.15.28:0/3607670257 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56021d7c66f0 0x56021d7cda40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:24.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.031+0000 7fc1ebe00640 1 -- 172.21.15.28:0/3607670257 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56021d7c5dd0 msgr2=0x56021d7d0750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:17:24.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.031+0000 7fc1ebe00640 1 --2- 172.21.15.28:0/3607670257 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56021d7c5dd0 0x56021d7d0750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:24.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.031+0000 7fc1ebe00640 1 -- 172.21.15.28:0/3607670257 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56021d43e3f0 con 0x56021d7d0dc0 2024-05-12T22:17:24.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.031+0000 7fc1ebe00640 1 --2- 172.21.15.28:0/3607670257 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56021d7d0dc0 0x56021d7cdfc0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fc1e40ed990 tx=0x7fc1e40bc180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73770ae3cafd9097 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:17:24.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.031+0000 7fc1d4c00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-12T22:17:24.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.032+0000 7fc1d4c00640 1 -- 172.21.15.28:0/3607670257 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1e402c040 con 0x56021d7d0dc0 2024-05-12T22:17:24.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.032+0000 7fc1d4c00640 1 -- 172.21.15.28:0/3607670257 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8513+0+0 (secure 0 0 0) 0x7fc1e40b8dd0 con 0x56021d7d0dc0 2024-05-12T22:17:24.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.032+0000 7fc1f5181740 1 -- 172.21.15.28:0/3607670257 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc1dc0035f0 con 0x56021d7d0dc0 2024-05-12T22:17:24.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.032+0000 7fc1d4c00640 1 -- 172.21.15.28:0/3607670257 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fc1e40c0b10 con 0x56021d7d0dc0 2024-05-12T22:17:24.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.032+0000 7fc1f5181740 10 client.16902 fetch_fsmap learned FSMap version 26 2024-05-12T22:17:24.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.032+0000 7fc1f5181740 1 -- 172.21.15.28:0/3607670257 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x56021d3fe0f0 con 0x56021d7d0dc0 2024-05-12T22:17:24.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.032+0000 7fc1d4c00640 1 -- 172.21.15.28:0/3607670257 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc1dc0035f0 con 0x56021d7d0dc0 2024-05-12T22:17:24.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.032+0000 7fc1d4c00640 1 -- 172.21.15.28:0/3607670257 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fc1e40bc9c0 con 0x56021d7d0dc0 2024-05-12T22:17:24.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.032+0000 7fc1f5181740 10 client.16902 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:17:24.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.033+0000 7fc1f5181740 10 client.16902 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:17:24.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.033+0000 7fc1f5181740 20 client.16902 populate_metadata read hostname 'smithi028' 2024-05-12T22:17:24.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.033+0000 7fc1f5181740 1 --2- 172.21.15.28:0/3607670257 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x56021d7e0850 0x56021d7e2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:24.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.033+0000 7fc1f5181740 4 client.16902 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-12T22:17:24.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.033+0000 7fc1f5181740 1 -- 172.21.15.28:0/3607670257 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56021d7382c0 con 0x56021d7e0850 2024-05-12T22:17:24.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.033+0000 7fc1eb400640 1 --2- 172.21.15.28:0/3607670257 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x56021d7e0850 0x56021d7e2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:24.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.034+0000 7fc1eb400640 1 --2- 172.21.15.28:0/3607670257 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x56021d7e0850 0x56021d7e2cf0 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c48b3ae00f62fed8 server_cookie=33d8ef8cff4b1e3d in_seq=0 out_seq=0 2024-05-12T22:17:24.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.034+0000 7fc1d4c00640 10 client.16902 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:17:24.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.034+0000 7fc1d4c00640 1 -- 172.21.15.28:0/3607670257 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56021d7382c0 con 0x56021d7e0850 2024-05-12T22:17:24.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.034+0000 7fc1d4c00640 10 client.16902 handle_command_reply: tid=0 2024-05-12T22:17:24.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.035+0000 7fc1f5181740 1 client.16902 shutdown 2024-05-12T22:17:24.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.035+0000 7fc1f5181740 1 -- 172.21.15.28:0/3607670257 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56021d7d0dc0 msgr2=0x56021d7cdfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:17:24.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.035+0000 7fc1f5181740 1 --2- 172.21.15.28:0/3607670257 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56021d7d0dc0 0x56021d7cdfc0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fc1e40ed990 tx=0x7fc1e40bc180 comp rx=0 tx=0).stop 2024-05-12T22:17:24.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.035+0000 7fc1f5181740 1 -- 172.21.15.28:0/3607670257 shutdown_connections 2024-05-12T22:17:24.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.035+0000 7fc1f5181740 1 --2- 172.21.15.28:0/3607670257 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x56021d7e0850 0x56021d7e2cf0 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:24.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.035+0000 7fc1f5181740 1 --2- 172.21.15.28:0/3607670257 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56021d7d0dc0 0x56021d7cdfc0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:24.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.035+0000 7fc1f5181740 1 --2- 172.21.15.28:0/3607670257 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56021d7c66f0 0x56021d7cda40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:24.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.035+0000 7fc1f5181740 1 --2- 172.21.15.28:0/3607670257 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56021d7c5dd0 0x56021d7d0750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:24.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.035+0000 7fc1d4c00640 0 client.16902 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:17:24.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.035+0000 7fc1f5181740 1 -- 172.21.15.28:0/3607670257 >> 172.21.15.28:0/3607670257 conn(0x56021d7c20f0 msgr2=0x56021d7c7ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:17:24.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.035+0000 7fc1f5181740 1 -- 172.21.15.28:0/3607670257 shutdown_connections 2024-05-12T22:17:24.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.035+0000 7fc1f5181740 1 -- 172.21.15.28:0/3607670257 wait complete. 2024-05-12T22:17:24.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.035+0000 7fc1f5181740 20 client.16902 trim_cache size 0 max 16384 2024-05-12T22:17:24.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.054+0000 7fc1f5181740 1 Processor -- start 2024-05-12T22:17:24.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.054+0000 7fc1f5181740 1 -- start start 2024-05-12T22:17:24.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.055+0000 7fc1f5181740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56021d7c5dd0 0x56021d7cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:24.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.055+0000 7fc1f5181740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56021d7c66f0 0x56021d7c8e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:24.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.055+0000 7fc1f5181740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56021d7d0dc0 0x56021d7d0280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:24.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.055+0000 7fc1f5181740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc1ec1b1cf0 con 0x56021d7d0dc0 2024-05-12T22:17:24.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.055+0000 7fc1f5181740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fc1ec1af5c0 con 0x56021d7c5dd0 2024-05-12T22:17:24.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.055+0000 7fc1f5181740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc1ec1ad430 con 0x56021d7c66f0 2024-05-12T22:17:24.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.055+0000 7fc1ebe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56021d7d0dc0 0x56021d7d0280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:24.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.055+0000 7fc1ebe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56021d7d0dc0 0x56021d7d0280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48854/0 (socket says 172.21.15.28:48854) 2024-05-12T22:17:24.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.055+0000 7fc1ebe00640 1 -- 172.21.15.28:0/3392688083 learned_addr learned my addr 172.21.15.28:0/3392688083 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:17:24.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.055+0000 7fc1eaa00640 1 --2- 172.21.15.28:0/3392688083 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56021d7c66f0 0x56021d7c8e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:24.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.055+0000 7fc1eb400640 1 --2- 172.21.15.28:0/3392688083 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56021d7c5dd0 0x56021d7cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:24.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.055+0000 7fc1ebe00640 1 -- 172.21.15.28:0/3392688083 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56021d7c66f0 msgr2=0x56021d7c8e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:17:24.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.055+0000 7fc1ebe00640 1 --2- 172.21.15.28:0/3392688083 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56021d7c66f0 0x56021d7c8e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:24.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.055+0000 7fc1ebe00640 1 -- 172.21.15.28:0/3392688083 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56021d7c5dd0 msgr2=0x56021d7cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:17:24.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.055+0000 7fc1ebe00640 1 --2- 172.21.15.28:0/3392688083 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56021d7c5dd0 0x56021d7cca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:24.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.056+0000 7fc1ebe00640 1 -- 172.21.15.28:0/3392688083 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56021d7cbf90 con 0x56021d7d0dc0 2024-05-12T22:17:24.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.056+0000 7fc1eaa00640 1 --2- 172.21.15.28:0/3392688083 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56021d7c66f0 0x56021d7c8e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:17:24.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.056+0000 7fc1ebe00640 1 --2- 172.21.15.28:0/3392688083 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56021d7d0dc0 0x56021d7d0280 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fc1e40edfd0 tx=0x7fc1e40bc8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d12027fde7e0189 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:17:24.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.056+0000 7fc1d7400640 1 -- 172.21.15.28:0/3392688083 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1e402c040 con 0x56021d7d0dc0 2024-05-12T22:17:24.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.056+0000 7fc1d7400640 1 -- 172.21.15.28:0/3392688083 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc1e4004810 con 0x56021d7d0dc0 2024-05-12T22:17:24.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.056+0000 7fc1d7400640 1 -- 172.21.15.28:0/3392688083 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1e402faf0 con 0x56021d7d0dc0 2024-05-12T22:17:24.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.057+0000 7fc1f5181740 1 -- 172.21.15.28:0/3392688083 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56021d7d0dc0 msgr2=0x56021d7d0280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:17:24.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.057+0000 7fc1f5181740 1 --2- 172.21.15.28:0/3392688083 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56021d7d0dc0 0x56021d7d0280 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fc1e40edfd0 tx=0x7fc1e40bc8e0 comp rx=0 tx=0).stop 2024-05-12T22:17:24.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.057+0000 7fc1f5181740 1 -- 172.21.15.28:0/3392688083 shutdown_connections 2024-05-12T22:17:24.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.057+0000 7fc1f5181740 1 --2- 172.21.15.28:0/3392688083 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56021d7d0dc0 0x56021d7d0280 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:24.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.057+0000 7fc1f5181740 1 --2- 172.21.15.28:0/3392688083 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56021d7c66f0 0x56021d7c8e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:24.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.057+0000 7fc1f5181740 1 --2- 172.21.15.28:0/3392688083 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56021d7c5dd0 0x56021d7cca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:24.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.057+0000 7fc1f5181740 1 -- 172.21.15.28:0/3392688083 >> 172.21.15.28:0/3392688083 conn(0x56021d7c20f0 msgr2=0x56021d7c8700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:17:24.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.057+0000 7fc1f5181740 1 -- 172.21.15.28:0/3392688083 shutdown_connections 2024-05-12T22:17:24.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.057+0000 7fc1f5181740 1 -- 172.21.15.28:0/3392688083 wait complete. 2024-05-12T22:17:24.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.057+0000 7fc1f5181740 1 -- start start 2024-05-12T22:17:24.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.058+0000 7fc1f5181740 1 Processor -- start 2024-05-12T22:17:24.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.058+0000 7fc1f5181740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56021d7c5dd0 0x56021d7d4a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:24.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.058+0000 7fc1f5181740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56021d7c66f0 0x56021d7d4f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:24.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.058+0000 7fc1f5181740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56021d7d0dc0 0x56021d7d5480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:24.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.058+0000 7fc1f5181740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc1ec1ada80 con 0x56021d7c5dd0 2024-05-12T22:17:24.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.058+0000 7fc1f5181740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fc1ec1aeef0 con 0x56021d7c66f0 2024-05-12T22:17:24.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.058+0000 7fc1f5181740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc1ec1ae150 con 0x56021d7d0dc0 2024-05-12T22:17:24.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.058+0000 7fc1f5181740 10 client.0 _ll_register_callbacks cb 0x7fc1e40095c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:17:24.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.058+0000 7fc1eb400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56021d7c5dd0 0x56021d7d4a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:24.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.058+0000 7fc1eb400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56021d7c5dd0 0x56021d7d4a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48868/0 (socket says 172.21.15.28:48868) 2024-05-12T22:17:24.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.058+0000 7fc1eb400640 1 -- 172.21.15.28:0/836637539 learned_addr learned my addr 172.21.15.28:0/836637539 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:17:24.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.059+0000 7fc1ebe00640 1 --2- 172.21.15.28:0/836637539 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56021d7d0dc0 0x56021d7d5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:24.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.059+0000 7fc1eaa00640 1 --2- 172.21.15.28:0/836637539 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56021d7c66f0 0x56021d7d4f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:24.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.059+0000 7fc1eb400640 1 -- 172.21.15.28:0/836637539 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56021d7d0dc0 msgr2=0x56021d7d5480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:17:24.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.059+0000 7fc1eb400640 1 --2- 172.21.15.28:0/836637539 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56021d7d0dc0 0x56021d7d5480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:24.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.059+0000 7fc1eb400640 1 -- 172.21.15.28:0/836637539 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56021d7c66f0 msgr2=0x56021d7d4f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:17:24.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.059+0000 7fc1eb400640 1 --2- 172.21.15.28:0/836637539 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56021d7c66f0 0x56021d7d4f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:24.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.059+0000 7fc1eb400640 1 -- 172.21.15.28:0/836637539 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56021d845f90 con 0x56021d7c5dd0 2024-05-12T22:17:24.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.059+0000 7fc1eb400640 1 --2- 172.21.15.28:0/836637539 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56021d7c5dd0 0x56021d7d4a00 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fc1dc01c040 tx=0x7fc1dc009cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdb0eeba7f51d4e3 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:17:24.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.059+0000 7fc1d4c00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-12T22:17:24.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.059+0000 7fc1d4c00640 1 -- 172.21.15.28:0/836637539 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1dc01e040 con 0x56021d7c5dd0 2024-05-12T22:17:24.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.060+0000 7fc1d4c00640 1 -- 172.21.15.28:0/836637539 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8513+0+0 (secure 0 0 0) 0x7fc1dc005250 con 0x56021d7c5dd0 2024-05-12T22:17:24.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.060+0000 7fc1f5181740 1 -- 172.21.15.28:0/836637539 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc1e4004810 con 0x56021d7c5dd0 2024-05-12T22:17:24.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.060+0000 7fc1d4c00640 1 -- 172.21.15.28:0/836637539 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fc1dc005450 con 0x56021d7c5dd0 2024-05-12T22:17:24.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.060+0000 7fc1f5181740 10 client.16908 fetch_fsmap learned FSMap version 26 2024-05-12T22:17:24.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.060+0000 7fc1f5181740 1 -- 172.21.15.28:0/836637539 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x56021d847280 con 0x56021d7c5dd0 2024-05-12T22:17:24.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.060+0000 7fc1d4c00640 1 -- 172.21.15.28:0/836637539 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc1e4004810 con 0x56021d7c5dd0 2024-05-12T22:17:24.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.060+0000 7fc1d4c00640 1 -- 172.21.15.28:0/836637539 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fc1dc0048a0 con 0x56021d7c5dd0 2024-05-12T22:17:24.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.060+0000 7fc1f5181740 10 client.16908 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:17:24.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.060+0000 7fc1f5181740 10 client.16908 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:17:24.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.060+0000 7fc1f5181740 20 client.16908 populate_metadata read hostname 'smithi028' 2024-05-12T22:17:24.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.060+0000 7fc1f5181740 1 --2- 172.21.15.28:0/836637539 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x56021d847800 0x56021d849bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:24.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.060+0000 7fc1f5181740 4 client.16908 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-12T22:17:24.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.060+0000 7fc1f5181740 1 -- 172.21.15.28:0/836637539 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56021d7cd0f0 con 0x56021d847800 2024-05-12T22:17:24.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.061+0000 7fc1eaa00640 1 --2- 172.21.15.28:0/836637539 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x56021d847800 0x56021d849bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:24.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.061+0000 7fc1eaa00640 1 --2- 172.21.15.28:0/836637539 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x56021d847800 0x56021d849bf0 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a5617ca230f7a3a8 server_cookie=9985912db9e8ea51 in_seq=0 out_seq=0 2024-05-12T22:17:24.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.061+0000 7fc1d4c00640 10 client.16908 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:17:24.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.062+0000 7fc1d4c00640 1 -- 172.21.15.28:0/836637539 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x56021d7cd0f0 con 0x56021d847800 2024-05-12T22:17:24.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.062+0000 7fc1d4c00640 10 client.16908 handle_command_reply: tid=0 2024-05-12T22:17:24.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.062+0000 7fc1f5181740 1 client.16908 shutdown 2024-05-12T22:17:24.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.062+0000 7fc1f5181740 1 -- 172.21.15.28:0/836637539 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56021d7c5dd0 msgr2=0x56021d7d4a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:17:24.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.062+0000 7fc1f5181740 1 --2- 172.21.15.28:0/836637539 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56021d7c5dd0 0x56021d7d4a00 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fc1dc01c040 tx=0x7fc1dc009cd0 comp rx=0 tx=0).stop 2024-05-12T22:17:24.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.062+0000 7fc1f5181740 1 -- 172.21.15.28:0/836637539 shutdown_connections 2024-05-12T22:17:24.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.062+0000 7fc1f5181740 1 --2- 172.21.15.28:0/836637539 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x56021d847800 0x56021d849bf0 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:24.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.062+0000 7fc1f5181740 1 --2- 172.21.15.28:0/836637539 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56021d7d0dc0 0x56021d7d5480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:24.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.062+0000 7fc1d4c00640 0 client.16908 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:17:24.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.062+0000 7fc1f5181740 1 --2- 172.21.15.28:0/836637539 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56021d7c66f0 0x56021d7d4f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:24.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.062+0000 7fc1f5181740 1 --2- 172.21.15.28:0/836637539 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56021d7c5dd0 0x56021d7d4a00 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:24.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.062+0000 7fc1f5181740 1 -- 172.21.15.28:0/836637539 >> 172.21.15.28:0/836637539 conn(0x56021d7c20f0 msgr2=0x56021d7cc8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:17:24.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.062+0000 7fc1f5181740 1 -- 172.21.15.28:0/836637539 shutdown_connections 2024-05-12T22:17:24.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.062+0000 7fc1f5181740 1 -- 172.21.15.28:0/836637539 wait complete. 2024-05-12T22:17:24.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.062+0000 7fc1f5181740 20 client.16908 trim_cache size 0 max 16384 2024-05-12T22:17:24.072 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-05-12T22:17:24.072 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-05-12T22:17:24.072 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-05-12T22:17:24.073 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-05-12T22:17:24.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.063+0000 7fc1f0c00640 1 -- 172.21.15.28:0/2331070769 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fc1d00a2d00 msgr2=0x7fc1d00a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:17:24.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.063+0000 7fc1f0c00640 1 --2- 172.21.15.28:0/2331070769 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fc1d00a2d00 0x7fc1d00a51c0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fc1d8009fd0 tx=0x7fc1d8005850 comp rx=0 tx=0).stop 2024-05-12T22:17:24.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.063+0000 7fc1f0c00640 1 -- 172.21.15.28:0/2331070769 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc1ec10df50 msgr2=0x7fc1ec19ea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:17:24.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.063+0000 7fc1f0c00640 1 --2- 172.21.15.28:0/2331070769 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc1ec10df50 0x7fc1ec19ea80 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fc1e402ae70 tx=0x7fc1e4003b80 comp rx=0 tx=0).stop 2024-05-12T22:17:24.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.063+0000 7fc1f0c00640 1 -- 172.21.15.28:0/2331070769 shutdown_connections 2024-05-12T22:17:24.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.063+0000 7fc1f0c00640 1 --2- 172.21.15.28:0/2331070769 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fc1d00a2d00 0x7fc1d00a51c0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:24.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.063+0000 7fc1f0c00640 1 --2- 172.21.15.28:0/2331070769 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc1ec10df50 0x7fc1ec19ea80 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:24.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.063+0000 7fc1f0c00640 1 --2- 172.21.15.28:0/2331070769 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc1ec074520 0x7fc1ec19a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:24.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.063+0000 7fc1f0c00640 1 --2- 172.21.15.28:0/2331070769 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1ec073c00 0x7fc1ec19a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:24.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.063+0000 7fc1f0c00640 1 -- 172.21.15.28:0/2331070769 >> 172.21.15.28:0/2331070769 conn(0x7fc1ec0fd200 msgr2=0x7fc1ec0fe920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:17:24.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.063+0000 7fc1f0c00640 1 -- 172.21.15.28:0/2331070769 shutdown_connections 2024-05-12T22:17:24.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.063+0000 7fc1f0c00640 1 -- 172.21.15.28:0/2331070769 wait complete. 2024-05-12T22:17:24.080 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-12T22:17:24.080 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-12T22:17:24.150 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:17:23 smithi028 ceph-mon[29847]: pgmap v259: 97 pgs: 97 active+clean; 91 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-05-12T22:17:24.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:17:23 smithi086 ceph-mon[34297]: pgmap v259: 97 pgs: 97 active+clean; 91 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-05-12T22:17:24.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.295+0000 7ffab5800640 1 -- 172.21.15.28:0/3089151546 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffab0075d60 msgr2=0x7ffab0076140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:17:24.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.295+0000 7ffab5800640 1 --2- 172.21.15.28:0/3089151546 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffab0075d60 0x7ffab0076140 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7ffaa0009930 tx=0x7ffaa002a440 comp rx=0 tx=0).stop 2024-05-12T22:17:24.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.295+0000 7ffab5800640 1 -- 172.21.15.28:0/3089151546 shutdown_connections 2024-05-12T22:17:24.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.295+0000 7ffab5800640 1 --2- 172.21.15.28:0/3089151546 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffab01124d0 0x7ffab01148c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:24.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.295+0000 7ffab5800640 1 --2- 172.21.15.28:0/3089151546 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffab0076680 0x7ffab0111f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:24.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.295+0000 7ffab5800640 1 --2- 172.21.15.28:0/3089151546 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffab0075d60 0x7ffab0076140 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:24.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.295+0000 7ffab5800640 1 -- 172.21.15.28:0/3089151546 >> 172.21.15.28:0/3089151546 conn(0x7ffab00ff2c0 msgr2=0x7ffab01016e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:17:24.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.295+0000 7ffab5800640 1 -- 172.21.15.28:0/3089151546 shutdown_connections 2024-05-12T22:17:24.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.295+0000 7ffab5800640 1 -- 172.21.15.28:0/3089151546 wait complete. 2024-05-12T22:17:24.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.297+0000 7ffab5800640 1 Processor -- start 2024-05-12T22:17:24.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.297+0000 7ffab5800640 1 -- start start 2024-05-12T22:17:24.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.297+0000 7ffab5800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffab0075d60 0x7ffab019e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:24.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.297+0000 7ffab5800640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffab0076680 0x7ffab019eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:24.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.297+0000 7ffab5800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffab01124d0 0x7ffab01a2e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:24.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.297+0000 7ffab5800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ffab0118080 con 0x7ffab01124d0 2024-05-12T22:17:24.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.297+0000 7ffab5800640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7ffab0117f00 con 0x7ffab0075d60 2024-05-12T22:17:24.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.297+0000 7ffab5800640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ffab0118200 con 0x7ffab0076680 2024-05-12T22:17:24.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.298+0000 7ffaafe00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffab0075d60 0x7ffab019e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:24.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.298+0000 7ffaafe00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffab0075d60 0x7ffab019e580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:44598/0 (socket says 172.21.15.28:44598) 2024-05-12T22:17:24.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.298+0000 7ffaafe00640 1 -- 172.21.15.28:0/1912409738 learned_addr learned my addr 172.21.15.28:0/1912409738 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:17:24.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.298+0000 7ffab4e00640 1 --2- 172.21.15.28:0/1912409738 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffab01124d0 0x7ffab01a2e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:24.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.298+0000 7ffaaf400640 1 --2- 172.21.15.28:0/1912409738 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffab0076680 0x7ffab019eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:24.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.298+0000 7ffaafe00640 1 -- 172.21.15.28:0/1912409738 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffab0076680 msgr2=0x7ffab019eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:17:24.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.298+0000 7ffaafe00640 1 --2- 172.21.15.28:0/1912409738 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffab0076680 0x7ffab019eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:24.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.298+0000 7ffaafe00640 1 -- 172.21.15.28:0/1912409738 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffab01124d0 msgr2=0x7ffab01a2e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:17:24.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.298+0000 7ffaafe00640 1 --2- 172.21.15.28:0/1912409738 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffab01124d0 0x7ffab01a2e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:24.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.298+0000 7ffaafe00640 1 -- 172.21.15.28:0/1912409738 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffab01a3570 con 0x7ffab0075d60 2024-05-12T22:17:24.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.298+0000 7ffaafe00640 1 --2- 172.21.15.28:0/1912409738 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffab0075d60 0x7ffab019e580 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7ffaa002a960 tx=0x7ffaa0003b20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:17:24.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.299+0000 7ffaacc00640 1 -- 172.21.15.28:0/1912409738 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaa003f070 con 0x7ffab0075d60 2024-05-12T22:17:24.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.299+0000 7ffab5800640 1 -- 172.21.15.28:0/1912409738 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffab01a3800 con 0x7ffab0075d60 2024-05-12T22:17:24.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.299+0000 7ffaacc00640 1 -- 172.21.15.28:0/1912409738 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffaa0031c90 con 0x7ffab0075d60 2024-05-12T22:17:24.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.299+0000 7ffaacc00640 1 -- 172.21.15.28:0/1912409738 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaa0035030 con 0x7ffab0075d60 2024-05-12T22:17:24.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.299+0000 7ffab5800640 1 -- 172.21.15.28:0/1912409738 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffab01ab0f0 con 0x7ffab0075d60 2024-05-12T22:17:24.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.300+0000 7ffab5800640 1 -- 172.21.15.28:0/1912409738 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffab0067ce0 con 0x7ffab0075d60 2024-05-12T22:17:24.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.302+0000 7ffaacc00640 1 -- 172.21.15.28:0/1912409738 <== mon.2 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7ffaa0042070 con 0x7ffab0075d60 2024-05-12T22:17:24.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.304+0000 7ffaacc00640 1 --2- 172.21.15.28:0/1912409738 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7ffa940a2d00 0x7ffa940a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:24.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.304+0000 7ffaacc00640 1 -- 172.21.15.28:0/1912409738 <== mon.2 v2:172.21.15.86:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8513+0+0 (secure 0 0 0) 0x7ffaa00f0260 con 0x7ffab0075d60 2024-05-12T22:17:24.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.304+0000 7ffaaf400640 1 --2- 172.21.15.28:0/1912409738 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7ffa940a2d00 0x7ffa940a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:24.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.305+0000 7ffaaf400640 1 --2- 172.21.15.28:0/1912409738 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7ffa940a2d00 0x7ffa940a51c0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7ffab0078ce0 tx=0x7ffa9c005f90 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:17:24.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.306+0000 7ffaacc00640 1 -- 172.21.15.28:0/1912409738 <== mon.2 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7ffaa00bc7d0 con 0x7ffab0075d60 2024-05-12T22:17:24.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:17:23 smithi186 ceph-mon[34785]: pgmap v259: 97 pgs: 97 active+clean; 91 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-05-12T22:17:24.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.557+0000 7ffab5800640 1 -- 172.21.15.28:0/1912409738 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ffab010d660 con 0x7ffab0075d60 2024-05-12T22:17:24.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.558+0000 7ffaacc00640 1 -- 172.21.15.28:0/1912409738 <== mon.2 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11864 (secure 0 0 0) 0x7ffaa00c0ad0 con 0x7ffab0075d60 2024-05-12T22:17:24.558 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-12T22:17:24.558 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1844141745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1844141745},{"type":"v1","addr":"172.21.15.186:6833","nonce":1844141745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6833/2960763758","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":2960763758},{"type":"v1","addr":"172.21.15.86:6833","nonce":2960763758}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24470,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/4131134219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":4131134219},{"type":"v1","addr":"172.21.15.28:6835","nonce":4131134219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24481,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/1727869064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1727869064},{"type":"v1","addr":"172.21.15.186:6835","nonce":1727869064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24482,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2938260704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2938260704},{"type":"v1","addr":"172.21.15.28:6837","nonce":2938260704}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24494,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3473700910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3473700910},{"type":"v1","addr":"172.21.15.86:6835","nonce":3473700910}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-12T22:06:33.426986+0000","modified":"2024-05-12T22:13:22.499381+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24581,"mds_1":24550,"mds_2":24532},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24499":{"gid":24499,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6837/3600809235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":3600809235},{"type":"v1","addr":"172.21.15.186:6837","nonce":3600809235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"i","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6839/2399610807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":2399610807},{"type":"v1","addr":"172.21.15.186:6839","nonce":2399610807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24530":{"gid":24530,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.86:6837/3782448884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":3782448884},{"type":"v1","addr":"172.21.15.86:6837","nonce":3782448884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24532":{"gid":24532,"name":"g","rank":2,"incarnation":21,"state":"up:active","state_seq":6,"addr":"172.21.15.28:6839/123892158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":123892158},{"type":"v1","addr":"172.21.15.28:6839","nonce":123892158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24550":{"gid":24550,"name":"a","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.28:6841/761183870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":761183870},{"type":"v1","addr":"172.21.15.28:6841","nonce":761183870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24581":{"gid":24581,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6839/1251999130","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6838","nonce":1251999130},{"type":"v1","addr":"172.21.15.86:6839","nonce":1251999130}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24581,"qdb_cluster":[24550,24581,24532]},"id":1}]} 2024-05-12T22:17:24.559 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 26 2024-05-12T22:17:24.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.561+0000 7ffab5800640 1 -- 172.21.15.28:0/1912409738 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7ffa940a2d00 msgr2=0x7ffa940a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:17:24.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.561+0000 7ffab5800640 1 --2- 172.21.15.28:0/1912409738 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7ffa940a2d00 0x7ffa940a51c0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7ffab0078ce0 tx=0x7ffa9c005f90 comp rx=0 tx=0).stop 2024-05-12T22:17:24.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.561+0000 7ffab5800640 1 -- 172.21.15.28:0/1912409738 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffab0075d60 msgr2=0x7ffab019e580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:17:24.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.561+0000 7ffab5800640 1 --2- 172.21.15.28:0/1912409738 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffab0075d60 0x7ffab019e580 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7ffaa002a960 tx=0x7ffaa0003b20 comp rx=0 tx=0).stop 2024-05-12T22:17:24.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.562+0000 7ffab5800640 1 -- 172.21.15.28:0/1912409738 shutdown_connections 2024-05-12T22:17:24.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.562+0000 7ffab5800640 1 --2- 172.21.15.28:0/1912409738 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7ffa940a2d00 0x7ffa940a51c0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:24.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.562+0000 7ffab5800640 1 --2- 172.21.15.28:0/1912409738 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffab01124d0 0x7ffab01a2e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:24.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.562+0000 7ffab5800640 1 --2- 172.21.15.28:0/1912409738 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffab0076680 0x7ffab019eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:24.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.562+0000 7ffab5800640 1 --2- 172.21.15.28:0/1912409738 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffab0075d60 0x7ffab019e580 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:24.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.562+0000 7ffab5800640 1 -- 172.21.15.28:0/1912409738 >> 172.21.15.28:0/1912409738 conn(0x7ffab00ff2c0 msgr2=0x7ffab0113140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:17:24.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.562+0000 7ffab5800640 1 -- 172.21.15.28:0/1912409738 shutdown_connections 2024-05-12T22:17:24.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.562+0000 7ffab5800640 1 -- 172.21.15.28:0/1912409738 wait complete. 2024-05-12T22:17:24.580 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-05-12T22:17:24.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.788+0000 7f8dbde00640 1 -- 172.21.15.28:0/3082397004 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8db8102a90 msgr2=0x7f8db8102e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:17:24.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.788+0000 7f8dbde00640 1 --2- 172.21.15.28:0/3082397004 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8db8102a90 0x7f8db8102e70 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f8da8009900 tx=0x7f8da802a480 comp rx=0 tx=0).stop 2024-05-12T22:17:24.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.789+0000 7f8dbde00640 1 -- 172.21.15.28:0/3082397004 shutdown_connections 2024-05-12T22:17:24.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.789+0000 7f8dbde00640 1 --2- 172.21.15.28:0/3082397004 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8db810be60 0x7f8db810e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:24.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.789+0000 7f8dbde00640 1 --2- 172.21.15.28:0/3082397004 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8db81033b0 0x7f8db810b890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:24.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.789+0000 7f8dbde00640 1 --2- 172.21.15.28:0/3082397004 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8db8102a90 0x7f8db8102e70 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:24.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.789+0000 7f8dbde00640 1 -- 172.21.15.28:0/3082397004 >> 172.21.15.28:0/3082397004 conn(0x7f8db8076ab0 msgr2=0x7f8db8076ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:17:24.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.789+0000 7f8dbde00640 1 -- 172.21.15.28:0/3082397004 shutdown_connections 2024-05-12T22:17:24.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.789+0000 7f8dbde00640 1 -- 172.21.15.28:0/3082397004 wait complete. 2024-05-12T22:17:24.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.790+0000 7f8dbde00640 1 Processor -- start 2024-05-12T22:17:24.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.790+0000 7f8dbde00640 1 -- start start 2024-05-12T22:17:24.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.790+0000 7f8dbde00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8db8102a90 0x7f8db8198140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:24.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.790+0000 7f8dbde00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8db81033b0 0x7f8db8198680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:24.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.790+0000 7f8dbde00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8db810be60 0x7f8db819c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:24.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.790+0000 7f8dbde00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8db8111c40 con 0x7f8db8102a90 2024-05-12T22:17:24.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.790+0000 7f8dbde00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f8db8111ac0 con 0x7f8db810be60 2024-05-12T22:17:24.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.790+0000 7f8dbde00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f8db8111dc0 con 0x7f8db81033b0 2024-05-12T22:17:24.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.790+0000 7f8db6a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8db8102a90 0x7f8db8198140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:24.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.791+0000 7f8db6a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8db8102a90 0x7f8db8198140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48890/0 (socket says 172.21.15.28:48890) 2024-05-12T22:17:24.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.791+0000 7f8db6a00640 1 -- 172.21.15.28:0/109925115 learned_addr learned my addr 172.21.15.28:0/109925115 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:17:24.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.791+0000 7f8dae000640 1 --2- 172.21.15.28:0/109925115 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8db81033b0 0x7f8db8198680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:24.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.791+0000 7f8db7400640 1 --2- 172.21.15.28:0/109925115 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8db810be60 0x7f8db819c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:24.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.791+0000 7f8db6a00640 1 -- 172.21.15.28:0/109925115 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8db81033b0 msgr2=0x7f8db8198680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:17:24.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.791+0000 7f8db6a00640 1 --2- 172.21.15.28:0/109925115 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8db81033b0 0x7f8db8198680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:24.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.791+0000 7f8db6a00640 1 -- 172.21.15.28:0/109925115 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8db810be60 msgr2=0x7f8db819c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:17:24.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.791+0000 7f8db6a00640 1 --2- 172.21.15.28:0/109925115 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8db810be60 0x7f8db819c9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:24.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.791+0000 7f8db6a00640 1 -- 172.21.15.28:0/109925115 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8db819d0e0 con 0x7f8db8102a90 2024-05-12T22:17:24.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.791+0000 7f8db7400640 1 --2- 172.21.15.28:0/109925115 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8db810be60 0x7f8db819c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:17:24.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.791+0000 7f8db6a00640 1 --2- 172.21.15.28:0/109925115 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8db8102a90 0x7f8db8198140 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f8da8004770 tx=0x7f8da8003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:17:24.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.792+0000 7f8dbca00640 1 -- 172.21.15.28:0/109925115 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8da803f070 con 0x7f8db8102a90 2024-05-12T22:17:24.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.792+0000 7f8dbca00640 1 -- 172.21.15.28:0/109925115 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8da8031ca0 con 0x7f8db8102a90 2024-05-12T22:17:24.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.792+0000 7f8dbde00640 1 -- 172.21.15.28:0/109925115 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8db819d370 con 0x7f8db8102a90 2024-05-12T22:17:24.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.792+0000 7f8dbca00640 1 -- 172.21.15.28:0/109925115 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8da8035030 con 0x7f8db8102a90 2024-05-12T22:17:24.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.792+0000 7f8dbde00640 1 -- 172.21.15.28:0/109925115 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8db81a4c60 con 0x7f8db8102a90 2024-05-12T22:17:24.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.796+0000 7f8dc0c6b740 1 Processor -- start 2024-05-12T22:17:24.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.796+0000 7f8dbca00640 1 -- 172.21.15.28:0/109925115 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7f8da8042070 con 0x7f8db8102a90 2024-05-12T22:17:24.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.796+0000 7f8dc0c6b740 1 -- start start 2024-05-12T22:17:24.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.796+0000 7f8dc0c6b740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591cb094980 0x5591cb094d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:24.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.796+0000 7f8dc0c6b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591cb0952a0 0x5591cb09f430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:24.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.796+0000 7f8dc0c6b740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5591cb09f970 0x5591cb0a1d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:24.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.796+0000 7f8dc0c6b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5591caff9a30 con 0x5591cb0952a0 2024-05-12T22:17:24.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.796+0000 7f8dc0c6b740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x5591cafa1700 con 0x5591cb09f970 2024-05-12T22:17:24.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.796+0000 7f8dc0c6b740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5591caff59f0 con 0x5591cb094980 2024-05-12T22:17:24.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.796+0000 7f8db6a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591cb0952a0 0x5591cb09f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:24.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.796+0000 7f8dae000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5591cb09f970 0x5591cb0a1d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:24.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.796+0000 7f8db7400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591cb094980 0x5591cb094d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:24.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.796+0000 7f8dae000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5591cb09f970 0x5591cb0a1d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:44634/0 (socket says 172.21.15.28:44634) 2024-05-12T22:17:24.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.797+0000 7f8dae000640 1 -- 172.21.15.28:0/485320442 learned_addr learned my addr 172.21.15.28:0/485320442 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:17:24.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.797+0000 7f8db7400640 1 -- 172.21.15.28:0/485320442 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5591cb09f970 msgr2=0x5591cb0a1d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:17:24.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.797+0000 7f8db7400640 1 --2- 172.21.15.28:0/485320442 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5591cb09f970 0x5591cb0a1d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:24.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.797+0000 7f8db7400640 1 -- 172.21.15.28:0/485320442 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591cb0952a0 msgr2=0x5591cb09f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:17:24.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.797+0000 7f8db7400640 1 --2- 172.21.15.28:0/485320442 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591cb0952a0 0x5591cb09f430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:24.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.797+0000 7f8db7400640 1 -- 172.21.15.28:0/485320442 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5591cad0d3f0 con 0x5591cb094980 2024-05-12T22:17:24.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.797+0000 7f8db6a00640 1 --2- 172.21.15.28:0/485320442 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591cb0952a0 0x5591cb09f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:17:24.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.797+0000 7f8db7400640 1 --2- 172.21.15.28:0/485320442 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591cb094980 0x5591cb094d60 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f8db806af10 tx=0x7f8db0011bf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2243fd9ab26e049b server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:17:24.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.797+0000 7f8dbca00640 1 --2- 172.21.15.28:0/109925115 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f8d9c0a2d00 0x7f8d9c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:24.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.798+0000 7f8da7e00640 1 -- 172.21.15.28:0/485320442 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8db0004d20 con 0x5591cb094980 2024-05-12T22:17:24.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.798+0000 7f8da7e00640 1 -- 172.21.15.28:0/485320442 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8db000f970 con 0x5591cb094980 2024-05-12T22:17:24.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.798+0000 7f8da7e00640 1 -- 172.21.15.28:0/485320442 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8db0004570 con 0x5591cb094980 2024-05-12T22:17:24.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.798+0000 7f8dbca00640 1 -- 172.21.15.28:0/109925115 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8513+0+0 (secure 0 0 0) 0x7f8da80ef680 con 0x7f8db8102a90 2024-05-12T22:17:24.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.798+0000 7f8db6a00640 1 --2- 172.21.15.28:0/109925115 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f8d9c0a2d00 0x7f8d9c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:24.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.798+0000 7f8dc0c6b740 1 -- 172.21.15.28:0/485320442 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591cb094980 msgr2=0x5591cb094d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:17:24.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.798+0000 7f8dc0c6b740 1 --2- 172.21.15.28:0/485320442 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591cb094980 0x5591cb094d60 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f8db806af10 tx=0x7f8db0011bf0 comp rx=0 tx=0).stop 2024-05-12T22:17:24.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.799+0000 7f8dc0c6b740 1 -- 172.21.15.28:0/485320442 shutdown_connections 2024-05-12T22:17:24.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.799+0000 7f8dc0c6b740 1 --2- 172.21.15.28:0/485320442 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5591cb09f970 0x5591cb0a1d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:24.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.799+0000 7f8db6a00640 1 --2- 172.21.15.28:0/109925115 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f8d9c0a2d00 0x7f8d9c0a51c0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f8da80ece80 tx=0x7f8da80f5550 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:17:24.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.799+0000 7f8dc0c6b740 1 --2- 172.21.15.28:0/485320442 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591cb0952a0 0x5591cb09f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:24.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.799+0000 7f8dc0c6b740 1 --2- 172.21.15.28:0/485320442 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591cb094980 0x5591cb094d60 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:24.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.799+0000 7f8dc0c6b740 1 -- 172.21.15.28:0/485320442 >> 172.21.15.28:0/485320442 conn(0x5591cb090ca0 msgr2=0x5591cb091080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:17:24.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.799+0000 7f8dc0c6b740 1 -- 172.21.15.28:0/485320442 shutdown_connections 2024-05-12T22:17:24.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.799+0000 7f8dc0c6b740 1 -- 172.21.15.28:0/485320442 wait complete. 2024-05-12T22:17:24.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.799+0000 7f8dc0c6b740 1 -- start start 2024-05-12T22:17:24.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.799+0000 7f8dc0c6b740 1 Processor -- start 2024-05-12T22:17:24.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.800+0000 7f8dc0c6b740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5591cb094980 0x5591cb09c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:24.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.800+0000 7f8dc0c6b740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591cb0952a0 0x5591cb09ce50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:24.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.800+0000 7f8dc0c6b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591cb09f970 0x5591cb09d390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:24.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.800+0000 7f8dc0c6b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5591cb0a2760 con 0x5591cb09f970 2024-05-12T22:17:24.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.800+0000 7f8dc0c6b740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x5591cafb4a70 con 0x5591cb094980 2024-05-12T22:17:24.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.800+0000 7f8dc0c6b740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5591cb0a28e0 con 0x5591cb0952a0 2024-05-12T22:17:24.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.800+0000 7f8dc0c6b740 10 client.0 _ll_register_callbacks cb 0x5591caf97ba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:17:24.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.800+0000 7f8db7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591cb09f970 0x5591cb09d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:24.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.800+0000 7f8db7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591cb09f970 0x5591cb09d390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48912/0 (socket says 172.21.15.28:48912) 2024-05-12T22:17:24.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.800+0000 7f8dae000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591cb0952a0 0x5591cb09ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:24.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.800+0000 7f8db7400640 1 -- 172.21.15.28:0/3031418001 learned_addr learned my addr 172.21.15.28:0/3031418001 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:17:24.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.800+0000 7f8db7400640 1 --2- 172.21.15.28:0/3031418001 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5591cb094980 0x5591cb09c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:24.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.801+0000 7f8dae000640 1 -- 172.21.15.28:0/3031418001 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5591cb094980 msgr2=0x5591cb09c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:17:24.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.801+0000 7f8dae000640 1 --2- 172.21.15.28:0/3031418001 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5591cb094980 0x5591cb09c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:24.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.801+0000 7f8dae000640 1 -- 172.21.15.28:0/3031418001 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591cb09f970 msgr2=0x5591cb09d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:17:24.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.801+0000 7f8dae000640 1 --2- 172.21.15.28:0/3031418001 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591cb09f970 0x5591cb09d390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:24.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.801+0000 7f8dae000640 1 -- 172.21.15.28:0/3031418001 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5591cadd77d0 con 0x5591cb0952a0 2024-05-12T22:17:24.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.801+0000 7f8db7400640 1 --2- 172.21.15.28:0/3031418001 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591cb09f970 0x5591cb09d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:17:24.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.801+0000 7f8dae000640 1 --2- 172.21.15.28:0/3031418001 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591cb0952a0 0x5591cb09ce50 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f8da00126d0 tx=0x7f8da0012b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=45e3c2b0c077e6ce server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:17:24.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.801+0000 7f8da4c00640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-12T22:17:24.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.801+0000 7f8da4c00640 1 -- 172.21.15.28:0/3031418001 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8da0004050 con 0x5591cb0952a0 2024-05-12T22:17:24.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.801+0000 7f8da4c00640 1 -- 172.21.15.28:0/3031418001 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8513+0+0 (secure 0 0 0) 0x7f8da00041f0 con 0x5591cb0952a0 2024-05-12T22:17:24.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.802+0000 7f8dc0c6b740 1 -- 172.21.15.28:0/3031418001 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8db000f970 con 0x5591cb0952a0 2024-05-12T22:17:24.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.802+0000 7f8da4c00640 1 -- 172.21.15.28:0/3031418001 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f8da00043f0 con 0x5591cb0952a0 2024-05-12T22:17:24.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.802+0000 7f8dc0c6b740 10 client.27013 fetch_fsmap learned FSMap version 26 2024-05-12T22:17:24.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.802+0000 7f8dc0c6b740 1 -- 172.21.15.28:0/3031418001 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5591cb0b2e90 con 0x5591cb0952a0 2024-05-12T22:17:24.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.802+0000 7f8da4c00640 1 -- 172.21.15.28:0/3031418001 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8db000f970 con 0x5591cb0952a0 2024-05-12T22:17:24.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.803+0000 7f8da4c00640 1 -- 172.21.15.28:0/3031418001 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f8da0003bf0 con 0x5591cb0952a0 2024-05-12T22:17:24.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.803+0000 7f8dc0c6b740 10 client.27013 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:17:24.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.803+0000 7f8dc0c6b740 10 client.27013 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.g 2024-05-12T22:17:24.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.803+0000 7f8dc0c6b740 20 client.27013 populate_metadata read hostname 'smithi028' 2024-05-12T22:17:24.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.803+0000 7f8dc0c6b740 1 --2- 172.21.15.28:0/3031418001 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x5591cb0b42f0 0x5591cb0b6710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:24.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.803+0000 7f8dc0c6b740 4 client.27013 mds_command: new command op to 24532 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-12T22:17:24.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.803+0000 7f8dc0c6b740 1 -- 172.21.15.28:0/3031418001 --> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5591cb007160 con 0x5591cb0b42f0 2024-05-12T22:17:24.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.803+0000 7f8db7400640 1 --2- 172.21.15.28:0/3031418001 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x5591cb0b42f0 0x5591cb0b6710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:24.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.804+0000 7f8db7400640 1 --2- 172.21.15.28:0/3031418001 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x5591cb0b42f0 0x5591cb0b6710 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=423c344f742cd65f server_cookie=78ffc3425adb3d15 in_seq=0 out_seq=0 2024-05-12T22:17:24.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.804+0000 7f8da4c00640 10 client.27013 ms_handle_connect on v2:172.21.15.28:6838/123892158 2024-05-12T22:17:24.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.804+0000 7f8da4c00640 1 -- 172.21.15.28:0/3031418001 <== mds.2 v2:172.21.15.28:6838/123892158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5591cb007160 con 0x5591cb0b42f0 2024-05-12T22:17:24.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.804+0000 7f8da4c00640 10 client.27013 handle_command_reply: tid=0 2024-05-12T22:17:24.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.804+0000 7f8dc0c6b740 1 client.27013 shutdown 2024-05-12T22:17:24.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.805+0000 7f8dc0c6b740 1 -- 172.21.15.28:0/3031418001 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591cb0952a0 msgr2=0x5591cb09ce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:17:24.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.805+0000 7f8dc0c6b740 1 --2- 172.21.15.28:0/3031418001 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591cb0952a0 0x5591cb09ce50 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f8da00126d0 tx=0x7f8da0012b90 comp rx=0 tx=0).stop 2024-05-12T22:17:24.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.805+0000 7f8dc0c6b740 1 -- 172.21.15.28:0/3031418001 shutdown_connections 2024-05-12T22:17:24.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.805+0000 7f8dc0c6b740 1 --2- 172.21.15.28:0/3031418001 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591cb09f970 0x5591cb09d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:24.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.805+0000 7f8dc0c6b740 1 --2- 172.21.15.28:0/3031418001 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591cb0952a0 0x5591cb09ce50 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:24.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.805+0000 7f8dc0c6b740 1 --2- 172.21.15.28:0/3031418001 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x5591cb0b42f0 0x5591cb0b6710 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:24.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.805+0000 7f8dc0c6b740 1 --2- 172.21.15.28:0/3031418001 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5591cb094980 0x5591cb09c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:24.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.805+0000 7f8dc0c6b740 1 -- 172.21.15.28:0/3031418001 >> 172.21.15.28:0/3031418001 conn(0x5591cb090ca0 msgr2=0x5591cb096aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:17:24.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.805+0000 7f8da4c00640 0 client.27013 ms_handle_reset on v2:172.21.15.28:6838/123892158 2024-05-12T22:17:24.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.805+0000 7f8dc0c6b740 1 -- 172.21.15.28:0/3031418001 shutdown_connections 2024-05-12T22:17:24.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.805+0000 7f8dc0c6b740 1 -- 172.21.15.28:0/3031418001 wait complete. 2024-05-12T22:17:24.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.805+0000 7f8dc0c6b740 20 client.27013 trim_cache size 0 max 16384 2024-05-12T22:17:24.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.820+0000 7f8dc0c6b740 1 Processor -- start 2024-05-12T22:17:24.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.820+0000 7f8dc0c6b740 1 -- start start 2024-05-12T22:17:24.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.820+0000 7f8dc0c6b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591cb094980 0x5591cb097150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:24.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.820+0000 7f8dc0c6b740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591cb0952a0 0x5591cb08fc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:24.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.820+0000 7f8dc0c6b740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5591cb09f970 0x5591cb0bbfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:24.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.820+0000 7f8dc0c6b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8db81adc40 con 0x5591cb094980 2024-05-12T22:17:24.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.820+0000 7f8dc0c6b740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f8db81ab510 con 0x5591cb09f970 2024-05-12T22:17:24.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.820+0000 7f8dc0c6b740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f8db81a9380 con 0x5591cb0952a0 2024-05-12T22:17:24.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.820+0000 7f8db7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591cb094980 0x5591cb097150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:24.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.820+0000 7f8db7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591cb094980 0x5591cb097150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48928/0 (socket says 172.21.15.28:48928) 2024-05-12T22:17:24.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.820+0000 7f8db7400640 1 -- 172.21.15.28:0/3811684792 learned_addr learned my addr 172.21.15.28:0/3811684792 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:17:24.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.821+0000 7f8dae000640 1 --2- 172.21.15.28:0/3811684792 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591cb0952a0 0x5591cb08fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:24.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.821+0000 7f8db7400640 1 --2- 172.21.15.28:0/3811684792 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5591cb09f970 0x5591cb0bbfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:24.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.821+0000 7f8db7400640 1 -- 172.21.15.28:0/3811684792 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591cb0952a0 msgr2=0x5591cb08fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:17:24.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.821+0000 7f8db7400640 1 --2- 172.21.15.28:0/3811684792 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591cb0952a0 0x5591cb08fc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:24.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.821+0000 7f8db7400640 1 -- 172.21.15.28:0/3811684792 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5591cb09f970 msgr2=0x5591cb0bbfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:17:24.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.821+0000 7f8db7400640 1 --2- 172.21.15.28:0/3811684792 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5591cb09f970 0x5591cb0bbfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:24.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.821+0000 7f8db7400640 1 -- 172.21.15.28:0/3811684792 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5591cb0915e0 con 0x5591cb094980 2024-05-12T22:17:24.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.821+0000 7f8db7400640 1 --2- 172.21.15.28:0/3811684792 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591cb094980 0x5591cb097150 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f8db0005fd0 tx=0x7f8db0009500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67fc90bf4d15471e server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:17:24.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.822+0000 7f8da7400640 1 -- 172.21.15.28:0/3811684792 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8db002e020 con 0x5591cb094980 2024-05-12T22:17:24.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.822+0000 7f8da7400640 1 -- 172.21.15.28:0/3811684792 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8db000f970 con 0x5591cb094980 2024-05-12T22:17:24.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.822+0000 7f8da7400640 1 -- 172.21.15.28:0/3811684792 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8db0027950 con 0x5591cb094980 2024-05-12T22:17:24.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.822+0000 7f8dc0c6b740 1 -- 172.21.15.28:0/3811684792 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591cb094980 msgr2=0x5591cb097150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:17:24.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.822+0000 7f8dc0c6b740 1 --2- 172.21.15.28:0/3811684792 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591cb094980 0x5591cb097150 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f8db0005fd0 tx=0x7f8db0009500 comp rx=0 tx=0).stop 2024-05-12T22:17:24.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.822+0000 7f8dc0c6b740 1 -- 172.21.15.28:0/3811684792 shutdown_connections 2024-05-12T22:17:24.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.822+0000 7f8dc0c6b740 1 --2- 172.21.15.28:0/3811684792 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5591cb09f970 0x5591cb0bbfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:24.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.822+0000 7f8dc0c6b740 1 --2- 172.21.15.28:0/3811684792 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591cb0952a0 0x5591cb08fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:24.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.822+0000 7f8dc0c6b740 1 --2- 172.21.15.28:0/3811684792 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591cb094980 0x5591cb097150 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:24.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.822+0000 7f8dc0c6b740 1 -- 172.21.15.28:0/3811684792 >> 172.21.15.28:0/3811684792 conn(0x5591cb090ca0 msgr2=0x5591cb08f530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:17:24.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.822+0000 7f8dc0c6b740 1 -- 172.21.15.28:0/3811684792 shutdown_connections 2024-05-12T22:17:24.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.822+0000 7f8dc0c6b740 1 -- 172.21.15.28:0/3811684792 wait complete. 2024-05-12T22:17:24.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.823+0000 7f8dc0c6b740 1 -- start start 2024-05-12T22:17:24.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.823+0000 7f8dc0c6b740 1 Processor -- start 2024-05-12T22:17:24.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.823+0000 7f8dc0c6b740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591cb094980 0x5591cb0a7a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:24.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.823+0000 7f8dc0c6b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591cb0952a0 0x5591cb0a7fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:24.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.823+0000 7f8dc0c6b740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5591cb09f970 0x5591cb0a8500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:24.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.823+0000 7f8dc0c6b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8db81a99d0 con 0x5591cb0952a0 2024-05-12T22:17:24.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.823+0000 7f8dc0c6b740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f8db81aae40 con 0x5591cb09f970 2024-05-12T22:17:24.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.823+0000 7f8dc0c6b740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f8db81aa0a0 con 0x5591cb094980 2024-05-12T22:17:24.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.823+0000 7f8dc0c6b740 10 client.0 _ll_register_callbacks cb 0x7f8da0009e20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:17:24.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.824+0000 7f8dae000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591cb0952a0 0x5591cb0a7fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:24.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.824+0000 7f8dae000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591cb0952a0 0x5591cb0a7fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48936/0 (socket says 172.21.15.28:48936) 2024-05-12T22:17:24.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.824+0000 7f8dae000640 1 -- 172.21.15.28:0/3981128814 learned_addr learned my addr 172.21.15.28:0/3981128814 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:17:24.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.824+0000 7f8db7400640 1 --2- 172.21.15.28:0/3981128814 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591cb094980 0x5591cb0a7a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:24.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.824+0000 7f8db7400640 1 --2- 172.21.15.28:0/3981128814 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5591cb09f970 0x5591cb0a8500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:24.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.824+0000 7f8dae000640 1 -- 172.21.15.28:0/3981128814 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591cb094980 msgr2=0x5591cb0a7a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:17:24.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.824+0000 7f8dae000640 1 --2- 172.21.15.28:0/3981128814 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591cb094980 0x5591cb0a7a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:24.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.824+0000 7f8dae000640 1 -- 172.21.15.28:0/3981128814 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5591cb09f970 msgr2=0x5591cb0a8500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:17:24.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.824+0000 7f8dae000640 1 --2- 172.21.15.28:0/3981128814 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5591cb09f970 0x5591cb0a8500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:24.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.824+0000 7f8dae000640 1 -- 172.21.15.28:0/3981128814 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5591cb119900 con 0x5591cb0952a0 2024-05-12T22:17:24.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.825+0000 7f8dae000640 1 --2- 172.21.15.28:0/3981128814 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591cb0952a0 0x5591cb0a7fc0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f8da00059f0 tx=0x7f8da000fb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1440932ce6328e3 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:17:24.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.825+0000 7f8da4c00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-12T22:17:24.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.825+0000 7f8da4c00640 1 -- 172.21.15.28:0/3981128814 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8da002b040 con 0x5591cb0952a0 2024-05-12T22:17:24.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.825+0000 7f8dc0c6b740 1 -- 172.21.15.28:0/3981128814 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8db000f970 con 0x5591cb0952a0 2024-05-12T22:17:24.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.825+0000 7f8da4c00640 1 -- 172.21.15.28:0/3981128814 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8513+0+0 (secure 0 0 0) 0x7f8da000fcc0 con 0x5591cb0952a0 2024-05-12T22:17:24.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.825+0000 7f8da4c00640 1 -- 172.21.15.28:0/3981128814 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f8da0005040 con 0x5591cb0952a0 2024-05-12T22:17:24.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.825+0000 7f8dc0c6b740 10 client.16932 fetch_fsmap learned FSMap version 26 2024-05-12T22:17:24.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.825+0000 7f8dc0c6b740 1 -- 172.21.15.28:0/3981128814 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5591cb11abe0 con 0x5591cb0952a0 2024-05-12T22:17:24.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.826+0000 7f8da4c00640 1 -- 172.21.15.28:0/3981128814 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8db000f970 con 0x5591cb0952a0 2024-05-12T22:17:24.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.826+0000 7f8da4c00640 1 -- 172.21.15.28:0/3981128814 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f8da0003ae0 con 0x5591cb0952a0 2024-05-12T22:17:24.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.826+0000 7f8dc0c6b740 10 client.16932 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:17:24.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.826+0000 7f8dc0c6b740 10 client.16932 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.g 2024-05-12T22:17:24.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.826+0000 7f8dc0c6b740 20 client.16932 populate_metadata read hostname 'smithi028' 2024-05-12T22:17:24.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.826+0000 7f8dc0c6b740 1 --2- 172.21.15.28:0/3981128814 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x5591cb11b320 0x5591cb11d740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:24.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.826+0000 7f8dc0c6b740 4 client.16932 mds_command: new command op to 24532 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-12T22:17:24.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.826+0000 7f8dc0c6b740 1 -- 172.21.15.28:0/3981128814 --> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5591cb0014b0 con 0x5591cb11b320 2024-05-12T22:17:24.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.826+0000 7f8db7400640 1 --2- 172.21.15.28:0/3981128814 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x5591cb11b320 0x5591cb11d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:24.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.827+0000 7f8db7400640 1 --2- 172.21.15.28:0/3981128814 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x5591cb11b320 0x5591cb11d740 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=44fd2a447dd092aa server_cookie=3b235b26afde0e3c in_seq=0 out_seq=0 2024-05-12T22:17:24.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.827+0000 7f8da4c00640 10 client.16932 ms_handle_connect on v2:172.21.15.28:6838/123892158 2024-05-12T22:17:24.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.827+0000 7f8da4c00640 1 -- 172.21.15.28:0/3981128814 <== mds.2 v2:172.21.15.28:6838/123892158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5591cb0014b0 con 0x5591cb11b320 2024-05-12T22:17:24.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.827+0000 7f8da4c00640 10 client.16932 handle_command_reply: tid=0 2024-05-12T22:17:24.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.827+0000 7f8dc0c6b740 1 client.16932 shutdown 2024-05-12T22:17:24.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.827+0000 7f8dc0c6b740 1 -- 172.21.15.28:0/3981128814 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591cb0952a0 msgr2=0x5591cb0a7fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:17:24.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.827+0000 7f8dc0c6b740 1 --2- 172.21.15.28:0/3981128814 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591cb0952a0 0x5591cb0a7fc0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f8da00059f0 tx=0x7f8da000fb50 comp rx=0 tx=0).stop 2024-05-12T22:17:24.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.827+0000 7f8dc0c6b740 1 -- 172.21.15.28:0/3981128814 shutdown_connections 2024-05-12T22:17:24.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.827+0000 7f8dc0c6b740 1 --2- 172.21.15.28:0/3981128814 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x5591cb11b320 0x5591cb11d740 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:24.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.827+0000 7f8dc0c6b740 1 --2- 172.21.15.28:0/3981128814 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5591cb09f970 0x5591cb0a8500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:24.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.827+0000 7f8dc0c6b740 1 --2- 172.21.15.28:0/3981128814 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591cb0952a0 0x5591cb0a7fc0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:24.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.827+0000 7f8dc0c6b740 1 --2- 172.21.15.28:0/3981128814 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591cb094980 0x5591cb0a7a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:24.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.827+0000 7f8dc0c6b740 1 -- 172.21.15.28:0/3981128814 >> 172.21.15.28:0/3981128814 conn(0x5591cb090ca0 msgr2=0x5591cb08fdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:17:24.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.827+0000 7f8da4c00640 0 client.16932 ms_handle_reset on v2:172.21.15.28:6838/123892158 2024-05-12T22:17:24.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.827+0000 7f8dc0c6b740 1 -- 172.21.15.28:0/3981128814 shutdown_connections 2024-05-12T22:17:24.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.827+0000 7f8dc0c6b740 1 -- 172.21.15.28:0/3981128814 wait complete. 2024-05-12T22:17:24.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.828+0000 7f8dc0c6b740 20 client.16932 trim_cache size 0 max 16384 2024-05-12T22:17:24.835 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-05-12T22:17:24.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.828+0000 7f8dbde00640 1 -- 172.21.15.28:0/109925115 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f8d9c0a2d00 msgr2=0x7f8d9c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:17:24.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.828+0000 7f8dbde00640 1 --2- 172.21.15.28:0/109925115 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f8d9c0a2d00 0x7f8d9c0a51c0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f8da80ece80 tx=0x7f8da80f5550 comp rx=0 tx=0).stop 2024-05-12T22:17:24.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.828+0000 7f8dbde00640 1 -- 172.21.15.28:0/109925115 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8db8102a90 msgr2=0x7f8db8198140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:17:24.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.828+0000 7f8dbde00640 1 --2- 172.21.15.28:0/109925115 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8db8102a90 0x7f8db8198140 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f8da8004770 tx=0x7f8da8003c10 comp rx=0 tx=0).stop 2024-05-12T22:17:24.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.828+0000 7f8dbde00640 1 -- 172.21.15.28:0/109925115 shutdown_connections 2024-05-12T22:17:24.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.828+0000 7f8dbde00640 1 --2- 172.21.15.28:0/109925115 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f8d9c0a2d00 0x7f8d9c0a51c0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:24.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.828+0000 7f8dbde00640 1 --2- 172.21.15.28:0/109925115 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8db810be60 0x7f8db819c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:24.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.828+0000 7f8dbde00640 1 --2- 172.21.15.28:0/109925115 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8db81033b0 0x7f8db8198680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:24.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.828+0000 7f8dbde00640 1 --2- 172.21.15.28:0/109925115 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8db8102a90 0x7f8db8198140 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:24.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.828+0000 7f8dbde00640 1 -- 172.21.15.28:0/109925115 >> 172.21.15.28:0/109925115 conn(0x7f8db8076ab0 msgr2=0x7f8db80fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:17:24.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.829+0000 7f8dbde00640 1 -- 172.21.15.28:0/109925115 shutdown_connections 2024-05-12T22:17:24.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:24.829+0000 7f8dbde00640 1 -- 172.21.15.28:0/109925115 wait complete. 2024-05-12T22:17:24.847 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-05-12T22:17:25.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.069+0000 7f87c8e00640 1 -- 172.21.15.28:0/2436354683 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f87c41031b0 msgr2=0x7f87c410b690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:17:25.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.069+0000 7f87c8e00640 1 --2- 172.21.15.28:0/2436354683 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f87c41031b0 0x7f87c410b690 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f87ac009860 tx=0x7f87ac02a3d0 comp rx=0 tx=0).stop 2024-05-12T22:17:25.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.069+0000 7f87c8e00640 1 -- 172.21.15.28:0/2436354683 shutdown_connections 2024-05-12T22:17:25.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.069+0000 7f87c8e00640 1 --2- 172.21.15.28:0/2436354683 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f87c410bcb0 0x7f87c410e150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:25.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.069+0000 7f87c8e00640 1 --2- 172.21.15.28:0/2436354683 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f87c41031b0 0x7f87c410b690 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:25.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.069+0000 7f87c8e00640 1 --2- 172.21.15.28:0/2436354683 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87c4102890 0x7f87c4102c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:25.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.069+0000 7f87c8e00640 1 -- 172.21.15.28:0/2436354683 >> 172.21.15.28:0/2436354683 conn(0x7f87c40fc0b0 msgr2=0x7f87c40fe4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:17:25.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.069+0000 7f87c8e00640 1 -- 172.21.15.28:0/2436354683 shutdown_connections 2024-05-12T22:17:25.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.069+0000 7f87c8e00640 1 -- 172.21.15.28:0/2436354683 wait complete. 2024-05-12T22:17:25.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.070+0000 7f87c8e00640 1 Processor -- start 2024-05-12T22:17:25.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.070+0000 7f87c8e00640 1 -- start start 2024-05-12T22:17:25.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.071+0000 7f87c8e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87c4102890 0x7f87c419a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:25.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.071+0000 7f87c8e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f87c41031b0 0x7f87c419a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:25.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.071+0000 7f87c8e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f87c410bcb0 0x7f87c419eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:25.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.071+0000 7f87c8e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f87c4111b00 con 0x7f87c4102890 2024-05-12T22:17:25.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.071+0000 7f87c8e00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f87c4111980 con 0x7f87c410bcb0 2024-05-12T22:17:25.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.071+0000 7f87c8e00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f87c4111c80 con 0x7f87c41031b0 2024-05-12T22:17:25.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.071+0000 7f87c3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87c4102890 0x7f87c419a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:25.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.071+0000 7f87baa00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f87c41031b0 0x7f87c419a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:25.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.071+0000 7f87baa00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f87c41031b0 0x7f87c419a720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:57906/0 (socket says 172.21.15.28:57906) 2024-05-12T22:17:25.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.071+0000 7f87baa00640 1 -- 172.21.15.28:0/2019398682 learned_addr learned my addr 172.21.15.28:0/2019398682 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:17:25.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.071+0000 7f87c3e00640 1 --2- 172.21.15.28:0/2019398682 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f87c410bcb0 0x7f87c419eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:25.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.072+0000 7f87baa00640 1 -- 172.21.15.28:0/2019398682 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f87c410bcb0 msgr2=0x7f87c419eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:17:25.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.072+0000 7f87baa00640 1 --2- 172.21.15.28:0/2019398682 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f87c410bcb0 0x7f87c419eaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:25.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.072+0000 7f87baa00640 1 -- 172.21.15.28:0/2019398682 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87c4102890 msgr2=0x7f87c419a1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:17:25.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.072+0000 7f87baa00640 1 --2- 172.21.15.28:0/2019398682 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87c4102890 0x7f87c419a1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:25.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.072+0000 7f87baa00640 1 -- 172.21.15.28:0/2019398682 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87c419f1d0 con 0x7f87c41031b0 2024-05-12T22:17:25.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.072+0000 7f87c3e00640 1 --2- 172.21.15.28:0/2019398682 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f87c410bcb0 0x7f87c419eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:17:25.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.072+0000 7f87baa00640 1 --2- 172.21.15.28:0/2019398682 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f87c41031b0 0x7f87c419a720 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f87ac009980 tx=0x7f87ac0308f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:17:25.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.072+0000 7f87c0c00640 1 -- 172.21.15.28:0/2019398682 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87ac040070 con 0x7f87c41031b0 2024-05-12T22:17:25.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.073+0000 7f87c8e00640 1 -- 172.21.15.28:0/2019398682 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87c419f460 con 0x7f87c41031b0 2024-05-12T22:17:25.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.073+0000 7f87c0c00640 1 -- 172.21.15.28:0/2019398682 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f87ac0055a0 con 0x7f87c41031b0 2024-05-12T22:17:25.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.073+0000 7f87c0c00640 1 -- 172.21.15.28:0/2019398682 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87ac004920 con 0x7f87c41031b0 2024-05-12T22:17:25.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.073+0000 7f87c8e00640 1 -- 172.21.15.28:0/2019398682 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87c41a6d50 con 0x7f87c41031b0 2024-05-12T22:17:25.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.077+0000 7f87cd3ab740 1 Processor -- start 2024-05-12T22:17:25.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.077+0000 7f87cd3ab740 1 -- start start 2024-05-12T22:17:25.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.078+0000 7f87c0c00640 1 -- 172.21.15.28:0/2019398682 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7f87ac043070 con 0x7f87c41031b0 2024-05-12T22:17:25.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.078+0000 7f87cd3ab740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560e2cd63b60 0x560e2cd63f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:25.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.078+0000 7f87cd3ab740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e2cd64480 0x560e2cd6e610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:25.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.078+0000 7f87cd3ab740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560e2cd6eb50 0x560e2cd70f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:25.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.078+0000 7f87cd3ab740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x560e2ccc8a30 con 0x560e2cd64480 2024-05-12T22:17:25.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.078+0000 7f87cd3ab740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x560e2cc70700 con 0x560e2cd6eb50 2024-05-12T22:17:25.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.078+0000 7f87cd3ab740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x560e2ccc49f0 con 0x560e2cd63b60 2024-05-12T22:17:25.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.078+0000 7f87c3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e2cd64480 0x560e2cd6e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:25.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.078+0000 7f87c3e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560e2cd63b60 0x560e2cd63f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:25.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.078+0000 7f87c3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e2cd64480 0x560e2cd6e610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48954/0 (socket says 172.21.15.28:48954) 2024-05-12T22:17:25.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.078+0000 7f87c3400640 1 -- 172.21.15.28:0/1226383867 learned_addr learned my addr 172.21.15.28:0/1226383867 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:17:25.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.078+0000 7f87baa00640 1 --2- 172.21.15.28:0/1226383867 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560e2cd6eb50 0x560e2cd70f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:25.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.079+0000 7f87c3e00640 1 -- 172.21.15.28:0/1226383867 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560e2cd6eb50 msgr2=0x560e2cd70f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:17:25.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.079+0000 7f87c3e00640 1 --2- 172.21.15.28:0/1226383867 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560e2cd6eb50 0x560e2cd70f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:25.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.079+0000 7f87c3e00640 1 -- 172.21.15.28:0/1226383867 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e2cd64480 msgr2=0x560e2cd6e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:17:25.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.079+0000 7f87c3e00640 1 --2- 172.21.15.28:0/1226383867 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e2cd64480 0x560e2cd6e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:25.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.079+0000 7f87c3e00640 1 -- 172.21.15.28:0/1226383867 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560e2c9dc3f0 con 0x560e2cd63b60 2024-05-12T22:17:25.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.079+0000 7f87c3400640 1 --2- 172.21.15.28:0/1226383867 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e2cd64480 0x560e2cd6e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:17:25.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.079+0000 7f87c3e00640 1 --2- 172.21.15.28:0/1226383867 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560e2cd63b60 0x560e2cd63f40 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f87bc005fd0 tx=0x7f87bc011bf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4fcc8fc4beb474c0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:17:25.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.079+0000 7f87c0c00640 1 --2- 172.21.15.28:0/2019398682 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f87a80a2d00 0x7f87a80a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:25.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.079+0000 7f87b8c00640 1 -- 172.21.15.28:0/1226383867 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87bc004d20 con 0x560e2cd63b60 2024-05-12T22:17:25.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.079+0000 7f87b8c00640 1 -- 172.21.15.28:0/1226383867 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f87bc00f9a0 con 0x560e2cd63b60 2024-05-12T22:17:25.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.079+0000 7f87b8c00640 1 -- 172.21.15.28:0/1226383867 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87bc004470 con 0x560e2cd63b60 2024-05-12T22:17:25.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.079+0000 7f87c0c00640 1 -- 172.21.15.28:0/2019398682 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8513+0+0 (secure 0 0 0) 0x7f87ac0effe0 con 0x7f87c41031b0 2024-05-12T22:17:25.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.080+0000 7f87c3400640 1 --2- 172.21.15.28:0/2019398682 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f87a80a2d00 0x7f87a80a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:25.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.080+0000 7f87cd3ab740 1 -- 172.21.15.28:0/1226383867 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560e2cd63b60 msgr2=0x560e2cd63f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:17:25.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.080+0000 7f87cd3ab740 1 --2- 172.21.15.28:0/1226383867 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560e2cd63b60 0x560e2cd63f40 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f87bc005fd0 tx=0x7f87bc011bf0 comp rx=0 tx=0).stop 2024-05-12T22:17:25.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.080+0000 7f87cd3ab740 1 -- 172.21.15.28:0/1226383867 shutdown_connections 2024-05-12T22:17:25.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.080+0000 7f87cd3ab740 1 --2- 172.21.15.28:0/1226383867 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560e2cd6eb50 0x560e2cd70f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:25.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.080+0000 7f87cd3ab740 1 --2- 172.21.15.28:0/1226383867 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e2cd64480 0x560e2cd6e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:25.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.080+0000 7f87cd3ab740 1 --2- 172.21.15.28:0/1226383867 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560e2cd63b60 0x560e2cd63f40 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:25.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.080+0000 7f87cd3ab740 1 -- 172.21.15.28:0/1226383867 >> 172.21.15.28:0/1226383867 conn(0x560e2cd5fe80 msgr2=0x560e2cd60260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:17:25.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.080+0000 7f87c3400640 1 --2- 172.21.15.28:0/2019398682 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f87a80a2d00 0x7f87a80a51c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x560e2cd65120 tx=0x7f87b4009260 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:17:25.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.080+0000 7f87cd3ab740 1 -- 172.21.15.28:0/1226383867 shutdown_connections 2024-05-12T22:17:25.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.080+0000 7f87cd3ab740 1 -- 172.21.15.28:0/1226383867 wait complete. 2024-05-12T22:17:25.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.081+0000 7f87cd3ab740 1 -- start start 2024-05-12T22:17:25.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.081+0000 7f87cd3ab740 1 Processor -- start 2024-05-12T22:17:25.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.082+0000 7f87cd3ab740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e2cd63b60 0x560e2cd6bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:25.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.082+0000 7f87cd3ab740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560e2cd64480 0x560e2cd6c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:25.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.082+0000 7f87cd3ab740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560e2cd6eb50 0x560e2cd6c630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:25.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.082+0000 7f87cd3ab740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x560e2cd71940 con 0x560e2cd63b60 2024-05-12T22:17:25.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.082+0000 7f87cd3ab740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x560e2cc83a70 con 0x560e2cd64480 2024-05-12T22:17:25.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.082+0000 7f87cd3ab740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x560e2cd71ac0 con 0x560e2cd6eb50 2024-05-12T22:17:25.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.082+0000 7f87cd3ab740 10 client.0 _ll_register_callbacks cb 0x560e2cc66ba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:17:25.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.082+0000 7f87c3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e2cd63b60 0x560e2cd6bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:25.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.082+0000 7f87c3e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560e2cd6eb50 0x560e2cd6c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:25.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.082+0000 7f87c3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e2cd63b60 0x560e2cd6bbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48964/0 (socket says 172.21.15.28:48964) 2024-05-12T22:17:25.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.082+0000 7f87c3400640 1 -- 172.21.15.28:0/3013504872 learned_addr learned my addr 172.21.15.28:0/3013504872 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:17:25.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.082+0000 7f87baa00640 1 --2- 172.21.15.28:0/3013504872 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560e2cd64480 0x560e2cd6c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:25.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.082+0000 7f87c3e00640 1 -- 172.21.15.28:0/3013504872 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560e2cd64480 msgr2=0x560e2cd6c0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:17:25.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.082+0000 7f87c3e00640 1 --2- 172.21.15.28:0/3013504872 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560e2cd64480 0x560e2cd6c0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:25.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.082+0000 7f87c3e00640 1 -- 172.21.15.28:0/3013504872 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e2cd63b60 msgr2=0x560e2cd6bbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:17:25.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.082+0000 7f87c3e00640 1 --2- 172.21.15.28:0/3013504872 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e2cd63b60 0x560e2cd6bbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:25.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.082+0000 7f87c3e00640 1 -- 172.21.15.28:0/3013504872 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560e2caa67d0 con 0x560e2cd6eb50 2024-05-12T22:17:25.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.083+0000 7f87c3e00640 1 --2- 172.21.15.28:0/3013504872 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560e2cd6eb50 0x560e2cd6c630 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f87bc00a010 tx=0x7f87bc003c60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f05f617bca786a8b server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:17:25.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.083+0000 7f87b1600640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-12T22:17:25.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.083+0000 7f87b1600640 1 -- 172.21.15.28:0/3013504872 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87bc028020 con 0x560e2cd6eb50 2024-05-12T22:17:25.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.083+0000 7f87cd3ab740 1 -- 172.21.15.28:0/3013504872 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f87bc00f9a0 con 0x560e2cd6eb50 2024-05-12T22:17:25.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.083+0000 7f87b1600640 1 -- 172.21.15.28:0/3013504872 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8513+0+0 (secure 0 0 0) 0x7f87bc005410 con 0x560e2cd6eb50 2024-05-12T22:17:25.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.084+0000 7f87b1600640 1 -- 172.21.15.28:0/3013504872 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f87bc033050 con 0x560e2cd6eb50 2024-05-12T22:17:25.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.084+0000 7f87cd3ab740 10 client.27031 fetch_fsmap learned FSMap version 26 2024-05-12T22:17:25.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.084+0000 7f87cd3ab740 1 -- 172.21.15.28:0/3013504872 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x560e2cd7dd10 con 0x560e2cd6eb50 2024-05-12T22:17:25.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.084+0000 7f87b1600640 1 -- 172.21.15.28:0/3013504872 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f87bc00f9a0 con 0x560e2cd6eb50 2024-05-12T22:17:25.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.084+0000 7f87b1600640 1 -- 172.21.15.28:0/3013504872 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f87bc018070 con 0x560e2cd6eb50 2024-05-12T22:17:25.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.084+0000 7f87cd3ab740 10 client.27031 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:17:25.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.084+0000 7f87cd3ab740 10 client.27031 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-05-12T22:17:25.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.084+0000 7f87cd3ab740 20 client.27031 populate_metadata read hostname 'smithi028' 2024-05-12T22:17:25.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.084+0000 7f87cd3ab740 1 --2- 172.21.15.28:0/3013504872 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x560e2cd7f170 0x560e2cd81560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:25.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.084+0000 7f87cd3ab740 4 client.27031 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-12T22:17:25.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.084+0000 7f87cd3ab740 1 -- 172.21.15.28:0/3013504872 --> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560e2ccd6160 con 0x560e2cd7f170 2024-05-12T22:17:25.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.085+0000 7f87c3400640 1 --2- 172.21.15.28:0/3013504872 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x560e2cd7f170 0x560e2cd81560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:25.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.085+0000 7f87c3400640 1 --2- 172.21.15.28:0/3013504872 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x560e2cd7f170 0x560e2cd81560 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=641dec9840b9dae0 server_cookie=7545c9c99c703148 in_seq=0 out_seq=0 2024-05-12T22:17:25.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.085+0000 7f87b1600640 10 client.27031 ms_handle_connect on v2:172.21.15.28:6840/761183870 2024-05-12T22:17:25.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.086+0000 7f87b1600640 1 -- 172.21.15.28:0/3013504872 <== mds.1 v2:172.21.15.28:6840/761183870 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560e2ccd6160 con 0x560e2cd7f170 2024-05-12T22:17:25.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.086+0000 7f87b1600640 10 client.27031 handle_command_reply: tid=0 2024-05-12T22:17:25.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.086+0000 7f87cd3ab740 1 client.27031 shutdown 2024-05-12T22:17:25.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.086+0000 7f87cd3ab740 1 -- 172.21.15.28:0/3013504872 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560e2cd6eb50 msgr2=0x560e2cd6c630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:17:25.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.086+0000 7f87cd3ab740 1 --2- 172.21.15.28:0/3013504872 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560e2cd6eb50 0x560e2cd6c630 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f87bc00a010 tx=0x7f87bc003c60 comp rx=0 tx=0).stop 2024-05-12T22:17:25.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.086+0000 7f87cd3ab740 1 -- 172.21.15.28:0/3013504872 shutdown_connections 2024-05-12T22:17:25.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.086+0000 7f87cd3ab740 1 --2- 172.21.15.28:0/3013504872 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x560e2cd7f170 0x560e2cd81560 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:25.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.086+0000 7f87cd3ab740 1 --2- 172.21.15.28:0/3013504872 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560e2cd6eb50 0x560e2cd6c630 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:25.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.086+0000 7f87cd3ab740 1 --2- 172.21.15.28:0/3013504872 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560e2cd64480 0x560e2cd6c0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:25.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.086+0000 7f87b1600640 0 client.27031 ms_handle_reset on v2:172.21.15.28:6840/761183870 2024-05-12T22:17:25.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.086+0000 7f87cd3ab740 1 --2- 172.21.15.28:0/3013504872 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e2cd63b60 0x560e2cd6bbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:25.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.086+0000 7f87cd3ab740 1 -- 172.21.15.28:0/3013504872 >> 172.21.15.28:0/3013504872 conn(0x560e2cd5fe80 msgr2=0x560e2cd65c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:17:25.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.086+0000 7f87cd3ab740 1 -- 172.21.15.28:0/3013504872 shutdown_connections 2024-05-12T22:17:25.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.087+0000 7f87cd3ab740 1 -- 172.21.15.28:0/3013504872 wait complete. 2024-05-12T22:17:25.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.087+0000 7f87cd3ab740 20 client.27031 trim_cache size 0 max 16384 2024-05-12T22:17:25.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.101+0000 7f87cd3ab740 1 Processor -- start 2024-05-12T22:17:25.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.102+0000 7f87cd3ab740 1 -- start start 2024-05-12T22:17:25.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.102+0000 7f87cd3ab740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e2cd63b60 0x560e2cd7a800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:25.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.102+0000 7f87cd3ab740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560e2cd64480 0x560e2cd65e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:25.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.102+0000 7f87cd3ab740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560e2cd6eb50 0x560e2cd86d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:25.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.102+0000 7f87cd3ab740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f87c41b1e00 con 0x560e2cd63b60 2024-05-12T22:17:25.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.102+0000 7f87cd3ab740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f87c41af6d0 con 0x560e2cd64480 2024-05-12T22:17:25.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.102+0000 7f87cd3ab740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f87c41ad540 con 0x560e2cd6eb50 2024-05-12T22:17:25.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.102+0000 7f87c3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e2cd63b60 0x560e2cd7a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:25.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.102+0000 7f87c3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e2cd63b60 0x560e2cd7a800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48976/0 (socket says 172.21.15.28:48976) 2024-05-12T22:17:25.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.102+0000 7f87c3400640 1 -- 172.21.15.28:0/976300001 learned_addr learned my addr 172.21.15.28:0/976300001 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:17:25.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.102+0000 7f87c3e00640 1 --2- 172.21.15.28:0/976300001 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560e2cd6eb50 0x560e2cd86d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:25.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.102+0000 7f87baa00640 1 --2- 172.21.15.28:0/976300001 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560e2cd64480 0x560e2cd65e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:25.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.103+0000 7f87c3400640 1 -- 172.21.15.28:0/976300001 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560e2cd6eb50 msgr2=0x560e2cd86d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:17:25.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.103+0000 7f87c3400640 1 --2- 172.21.15.28:0/976300001 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560e2cd6eb50 0x560e2cd86d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:25.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.103+0000 7f87c3400640 1 -- 172.21.15.28:0/976300001 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560e2cd64480 msgr2=0x560e2cd65e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:17:25.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.103+0000 7f87c3400640 1 --2- 172.21.15.28:0/976300001 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560e2cd64480 0x560e2cd65e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:25.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.103+0000 7f87c3400640 1 -- 172.21.15.28:0/976300001 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560e2cd70f40 con 0x560e2cd63b60 2024-05-12T22:17:25.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.103+0000 7f87c3400640 1 --2- 172.21.15.28:0/976300001 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e2cd63b60 0x560e2cd7a800 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x560e2cd81ce0 tx=0x7f87b401b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a6751fac0d1e2c3 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:17:25.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.103+0000 7f87b3e00640 1 -- 172.21.15.28:0/976300001 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87b4004020 con 0x560e2cd63b60 2024-05-12T22:17:25.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.103+0000 7f87b3e00640 1 -- 172.21.15.28:0/976300001 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f87b4031070 con 0x560e2cd63b60 2024-05-12T22:17:25.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.104+0000 7f87b3e00640 1 -- 172.21.15.28:0/976300001 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87b402d330 con 0x560e2cd63b60 2024-05-12T22:17:25.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.104+0000 7f87cd3ab740 1 -- 172.21.15.28:0/976300001 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e2cd63b60 msgr2=0x560e2cd7a800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:17:25.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.104+0000 7f87cd3ab740 1 --2- 172.21.15.28:0/976300001 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e2cd63b60 0x560e2cd7a800 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x560e2cd81ce0 tx=0x7f87b401b000 comp rx=0 tx=0).stop 2024-05-12T22:17:25.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.104+0000 7f87cd3ab740 1 -- 172.21.15.28:0/976300001 shutdown_connections 2024-05-12T22:17:25.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.104+0000 7f87cd3ab740 1 --2- 172.21.15.28:0/976300001 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560e2cd6eb50 0x560e2cd86d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:25.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.104+0000 7f87cd3ab740 1 --2- 172.21.15.28:0/976300001 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560e2cd64480 0x560e2cd65e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:25.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.104+0000 7f87cd3ab740 1 --2- 172.21.15.28:0/976300001 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e2cd63b60 0x560e2cd7a800 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:25.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.104+0000 7f87cd3ab740 1 -- 172.21.15.28:0/976300001 >> 172.21.15.28:0/976300001 conn(0x560e2cd5fe80 msgr2=0x560e2cd6abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:17:25.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.104+0000 7f87cd3ab740 1 -- 172.21.15.28:0/976300001 shutdown_connections 2024-05-12T22:17:25.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.104+0000 7f87cd3ab740 1 -- 172.21.15.28:0/976300001 wait complete. 2024-05-12T22:17:25.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.105+0000 7f87cd3ab740 1 -- start start 2024-05-12T22:17:25.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.105+0000 7f87cd3ab740 1 Processor -- start 2024-05-12T22:17:25.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.105+0000 7f87cd3ab740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e2cd63b60 0x560e2cd77180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:25.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.105+0000 7f87cd3ab740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560e2cd64480 0x560e2cd68600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:25.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.105+0000 7f87cd3ab740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560e2cd6eb50 0x560e2cd68b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:25.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.105+0000 7f87cd3ab740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f87c41adb90 con 0x560e2cd63b60 2024-05-12T22:17:25.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.105+0000 7f87cd3ab740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f87c41af000 con 0x560e2cd6eb50 2024-05-12T22:17:25.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.105+0000 7f87cd3ab740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f87c41ae260 con 0x560e2cd64480 2024-05-12T22:17:25.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.106+0000 7f87cd3ab740 10 client.0 _ll_register_callbacks cb 0x7f87b4009640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:17:25.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.106+0000 7f87c3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e2cd63b60 0x560e2cd77180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:25.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.106+0000 7f87c3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e2cd63b60 0x560e2cd77180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48992/0 (socket says 172.21.15.28:48992) 2024-05-12T22:17:25.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.106+0000 7f87c3400640 1 -- 172.21.15.28:0/2654594211 learned_addr learned my addr 172.21.15.28:0/2654594211 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:17:25.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.106+0000 7f87c3e00640 1 --2- 172.21.15.28:0/2654594211 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560e2cd6eb50 0x560e2cd68b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:25.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.106+0000 7f87baa00640 1 --2- 172.21.15.28:0/2654594211 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560e2cd64480 0x560e2cd68600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:25.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.106+0000 7f87c3400640 1 -- 172.21.15.28:0/2654594211 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560e2cd64480 msgr2=0x560e2cd68600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:17:25.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.106+0000 7f87c3400640 1 --2- 172.21.15.28:0/2654594211 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560e2cd64480 0x560e2cd68600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:25.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.106+0000 7f87c3400640 1 -- 172.21.15.28:0/2654594211 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560e2cd6eb50 msgr2=0x560e2cd68b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:17:25.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.106+0000 7f87c3400640 1 --2- 172.21.15.28:0/2654594211 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560e2cd6eb50 0x560e2cd68b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:25.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.106+0000 7f87c3400640 1 -- 172.21.15.28:0/2654594211 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560e2cd69390 con 0x560e2cd63b60 2024-05-12T22:17:25.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.106+0000 7f87c3400640 1 --2- 172.21.15.28:0/2654594211 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e2cd63b60 0x560e2cd77180 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f87b4003d90 tx=0x7f87b4003680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba77207fdc13b175 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:17:25.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.106+0000 7f87b1600640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-12T22:17:25.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.107+0000 7f87b1600640 1 -- 172.21.15.28:0/2654594211 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87b4039030 con 0x560e2cd63b60 2024-05-12T22:17:25.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.107+0000 7f87b1600640 1 -- 172.21.15.28:0/2654594211 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8513+0+0 (secure 0 0 0) 0x7f87b4003870 con 0x560e2cd63b60 2024-05-12T22:17:25.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.107+0000 7f87cd3ab740 1 -- 172.21.15.28:0/2654594211 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f87b4031070 con 0x560e2cd63b60 2024-05-12T22:17:25.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.107+0000 7f87b1600640 1 -- 172.21.15.28:0/2654594211 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f87b4003a70 con 0x560e2cd63b60 2024-05-12T22:17:25.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.107+0000 7f87cd3ab740 10 client.16950 fetch_fsmap learned FSMap version 26 2024-05-12T22:17:25.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.107+0000 7f87cd3ab740 1 -- 172.21.15.28:0/2654594211 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x560e2cde5640 con 0x560e2cd63b60 2024-05-12T22:17:25.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.107+0000 7f87b1600640 1 -- 172.21.15.28:0/2654594211 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f87b4031070 con 0x560e2cd63b60 2024-05-12T22:17:25.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.108+0000 7f87b1600640 1 -- 172.21.15.28:0/2654594211 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f87b40046c0 con 0x560e2cd63b60 2024-05-12T22:17:25.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.108+0000 7f87cd3ab740 10 client.16950 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:17:25.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.108+0000 7f87cd3ab740 10 client.16950 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-05-12T22:17:25.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.108+0000 7f87cd3ab740 20 client.16950 populate_metadata read hostname 'smithi028' 2024-05-12T22:17:25.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.108+0000 7f87cd3ab740 1 --2- 172.21.15.28:0/2654594211 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x560e2cde5d60 0x560e2cde8180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:25.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.108+0000 7f87cd3ab740 4 client.16950 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-12T22:17:25.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.108+0000 7f87cd3ab740 1 -- 172.21.15.28:0/2654594211 --> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560e2ccd04b0 con 0x560e2cde5d60 2024-05-12T22:17:25.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.108+0000 7f87baa00640 1 --2- 172.21.15.28:0/2654594211 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x560e2cde5d60 0x560e2cde8180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:25.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.108+0000 7f87baa00640 1 --2- 172.21.15.28:0/2654594211 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x560e2cde5d60 0x560e2cde8180 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=605a2dfe7f9c3225 server_cookie=bf37ed9f64c61ff0 in_seq=0 out_seq=0 2024-05-12T22:17:25.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.108+0000 7f87b1600640 10 client.16950 ms_handle_connect on v2:172.21.15.28:6840/761183870 2024-05-12T22:17:25.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.109+0000 7f87b1600640 1 -- 172.21.15.28:0/2654594211 <== mds.1 v2:172.21.15.28:6840/761183870 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560e2ccd04b0 con 0x560e2cde5d60 2024-05-12T22:17:25.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.109+0000 7f87b1600640 10 client.16950 handle_command_reply: tid=0 2024-05-12T22:17:25.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.109+0000 7f87cd3ab740 1 client.16950 shutdown 2024-05-12T22:17:25.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.109+0000 7f87cd3ab740 1 -- 172.21.15.28:0/2654594211 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e2cd63b60 msgr2=0x560e2cd77180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:17:25.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.109+0000 7f87cd3ab740 1 --2- 172.21.15.28:0/2654594211 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e2cd63b60 0x560e2cd77180 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f87b4003d90 tx=0x7f87b4003680 comp rx=0 tx=0).stop 2024-05-12T22:17:25.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.109+0000 7f87cd3ab740 1 -- 172.21.15.28:0/2654594211 shutdown_connections 2024-05-12T22:17:25.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.109+0000 7f87cd3ab740 1 --2- 172.21.15.28:0/2654594211 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x560e2cde5d60 0x560e2cde8180 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:25.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.109+0000 7f87cd3ab740 1 --2- 172.21.15.28:0/2654594211 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560e2cd6eb50 0x560e2cd68b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:25.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.109+0000 7f87cd3ab740 1 --2- 172.21.15.28:0/2654594211 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560e2cd64480 0x560e2cd68600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:25.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.109+0000 7f87cd3ab740 1 --2- 172.21.15.28:0/2654594211 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e2cd63b60 0x560e2cd77180 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:25.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.109+0000 7f87b1600640 0 client.16950 ms_handle_reset on v2:172.21.15.28:6840/761183870 2024-05-12T22:17:25.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.109+0000 7f87cd3ab740 1 -- 172.21.15.28:0/2654594211 >> 172.21.15.28:0/2654594211 conn(0x560e2cd5fe80 msgr2=0x560e2cd6a350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:17:25.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.109+0000 7f87cd3ab740 1 -- 172.21.15.28:0/2654594211 shutdown_connections 2024-05-12T22:17:25.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.109+0000 7f87cd3ab740 1 -- 172.21.15.28:0/2654594211 wait complete. 2024-05-12T22:17:25.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.109+0000 7f87cd3ab740 20 client.16950 trim_cache size 0 max 16384 2024-05-12T22:17:25.118 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-05-12T22:17:25.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.110+0000 7f87c8e00640 1 -- 172.21.15.28:0/2019398682 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f87a80a2d00 msgr2=0x7f87a80a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:17:25.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.110+0000 7f87c8e00640 1 --2- 172.21.15.28:0/2019398682 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f87a80a2d00 0x7f87a80a51c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x560e2cd65120 tx=0x7f87b4009260 comp rx=0 tx=0).stop 2024-05-12T22:17:25.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.110+0000 7f87c8e00640 1 -- 172.21.15.28:0/2019398682 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f87c41031b0 msgr2=0x7f87c419a720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:17:25.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.110+0000 7f87c8e00640 1 --2- 172.21.15.28:0/2019398682 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f87c41031b0 0x7f87c419a720 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f87ac009980 tx=0x7f87ac0308f0 comp rx=0 tx=0).stop 2024-05-12T22:17:25.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.110+0000 7f87c8e00640 1 -- 172.21.15.28:0/2019398682 shutdown_connections 2024-05-12T22:17:25.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.110+0000 7f87c8e00640 1 --2- 172.21.15.28:0/2019398682 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f87a80a2d00 0x7f87a80a51c0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:25.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.110+0000 7f87c8e00640 1 --2- 172.21.15.28:0/2019398682 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f87c410bcb0 0x7f87c419eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:25.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.110+0000 7f87c8e00640 1 --2- 172.21.15.28:0/2019398682 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f87c41031b0 0x7f87c419a720 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:25.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.110+0000 7f87c8e00640 1 --2- 172.21.15.28:0/2019398682 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87c4102890 0x7f87c419a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:25.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.110+0000 7f87c8e00640 1 -- 172.21.15.28:0/2019398682 >> 172.21.15.28:0/2019398682 conn(0x7f87c40fc0b0 msgr2=0x7f87c4100410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:17:25.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.110+0000 7f87c8e00640 1 -- 172.21.15.28:0/2019398682 shutdown_connections 2024-05-12T22:17:25.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.110+0000 7f87c8e00640 1 -- 172.21.15.28:0/2019398682 wait complete. 2024-05-12T22:17:25.125 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:17:24 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/1912409738' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:17:25.127 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-05-12T22:17:25.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:17:24 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/1912409738' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:17:25.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.337+0000 7fa66b200640 1 -- 172.21.15.28:0/1029346221 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa664104290 msgr2=0x7fa6641107a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:17:25.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.337+0000 7fa66b200640 1 --2- 172.21.15.28:0/1029346221 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa664104290 0x7fa6641107a0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fa660009c10 tx=0x7fa66002a6f0 comp rx=0 tx=0).stop 2024-05-12T22:17:25.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.337+0000 7fa66b200640 1 -- 172.21.15.28:0/1029346221 shutdown_connections 2024-05-12T22:17:25.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.337+0000 7fa66b200640 1 --2- 172.21.15.28:0/1029346221 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa664104290 0x7fa6641107a0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:25.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.337+0000 7fa66b200640 1 --2- 172.21.15.28:0/1029346221 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6641038d0 0x7fa664103d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:25.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.337+0000 7fa66b200640 1 --2- 172.21.15.28:0/1029346221 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa6641098e0 0x7fa664109cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:25.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.337+0000 7fa66b200640 1 -- 172.21.15.28:0/1029346221 >> 172.21.15.28:0/1029346221 conn(0x7fa6640ff3a0 msgr2=0x7fa6641017c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:17:25.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.337+0000 7fa66b200640 1 -- 172.21.15.28:0/1029346221 shutdown_connections 2024-05-12T22:17:25.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.337+0000 7fa66b200640 1 -- 172.21.15.28:0/1029346221 wait complete. 2024-05-12T22:17:25.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.338+0000 7fa66b200640 1 Processor -- start 2024-05-12T22:17:25.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.339+0000 7fa66b200640 1 -- start start 2024-05-12T22:17:25.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.339+0000 7fa66b200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa6641038d0 0x7fa66419a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:25.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.339+0000 7fa66b200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa664104290 0x7fa66419a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:25.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.339+0000 7fa66b200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6641098e0 0x7fa66419ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:25.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.339+0000 7fa66b200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa664113cf0 con 0x7fa6641098e0 2024-05-12T22:17:25.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.339+0000 7fa66b200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fa664113b70 con 0x7fa664104290 2024-05-12T22:17:25.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.339+0000 7fa66b200640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa664113e70 con 0x7fa6641038d0 2024-05-12T22:17:25.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.339+0000 7fa66a800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6641098e0 0x7fa66419ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:25.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.339+0000 7fa66a800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6641098e0 0x7fa66419ea20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49016/0 (socket says 172.21.15.28:49016) 2024-05-12T22:17:25.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.339+0000 7fa66a800640 1 -- 172.21.15.28:0/1363362311 learned_addr learned my addr 172.21.15.28:0/1363362311 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:17:25.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.339+0000 7fa669e00640 1 --2- 172.21.15.28:0/1363362311 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa6641038d0 0x7fa66419a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:25.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.339+0000 7fa669400640 1 --2- 172.21.15.28:0/1363362311 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa664104290 0x7fa66419a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:25.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.340+0000 7fa66a800640 1 -- 172.21.15.28:0/1363362311 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa6641038d0 msgr2=0x7fa66419a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:17:25.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.340+0000 7fa66a800640 1 --2- 172.21.15.28:0/1363362311 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa6641038d0 0x7fa66419a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:25.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.340+0000 7fa66a800640 1 -- 172.21.15.28:0/1363362311 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa664104290 msgr2=0x7fa66419a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:17:25.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.340+0000 7fa66a800640 1 --2- 172.21.15.28:0/1363362311 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa664104290 0x7fa66419a6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:25.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.340+0000 7fa66a800640 1 -- 172.21.15.28:0/1363362311 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa66419f150 con 0x7fa6641098e0 2024-05-12T22:17:25.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.340+0000 7fa669e00640 1 --2- 172.21.15.28:0/1363362311 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa6641038d0 0x7fa66419a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:17:25.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.340+0000 7fa669400640 1 --2- 172.21.15.28:0/1363362311 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa664104290 0x7fa66419a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:17:25.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.340+0000 7fa66a800640 1 --2- 172.21.15.28:0/1363362311 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6641098e0 0x7fa66419ea20 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fa66002ac10 tx=0x7fa6600305d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:17:25.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.341+0000 7fa65ea00640 1 -- 172.21.15.28:0/1363362311 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa660041020 con 0x7fa6641098e0 2024-05-12T22:17:25.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.341+0000 7fa65ea00640 1 -- 172.21.15.28:0/1363362311 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa660003e50 con 0x7fa6641098e0 2024-05-12T22:17:25.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.341+0000 7fa66b200640 1 -- 172.21.15.28:0/1363362311 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa66419f3e0 con 0x7fa6641098e0 2024-05-12T22:17:25.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.341+0000 7fa65ea00640 1 -- 172.21.15.28:0/1363362311 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa660036040 con 0x7fa6641098e0 2024-05-12T22:17:25.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.341+0000 7fa66b200640 1 -- 172.21.15.28:0/1363362311 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6641a6cd0 con 0x7fa6641098e0 2024-05-12T22:17:25.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.344+0000 7fa66f673740 1 Processor -- start 2024-05-12T22:17:25.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.344+0000 7fa65ea00640 1 -- 172.21.15.28:0/1363362311 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7fa660044020 con 0x7fa6641098e0 2024-05-12T22:17:25.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.344+0000 7fa66f673740 1 -- start start 2024-05-12T22:17:25.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.345+0000 7fa66f673740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f7706f3b60 0x55f7706f3f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:25.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.345+0000 7fa66f673740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7706f4480 0x55f7706fe610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:25.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.345+0000 7fa66f673740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f7706feb50 0x55f770700f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:25.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.345+0000 7fa66f673740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f770658a30 con 0x55f7706f4480 2024-05-12T22:17:25.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.345+0000 7fa66f673740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x55f770600700 con 0x55f7706feb50 2024-05-12T22:17:25.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.345+0000 7fa66f673740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55f7706549f0 con 0x55f7706f3b60 2024-05-12T22:17:25.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.345+0000 7fa669e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7706f4480 0x55f7706fe610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:25.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.345+0000 7fa669400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f7706f3b60 0x55f7706f3f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:25.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.345+0000 7fa669e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7706f4480 0x55f7706fe610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49024/0 (socket says 172.21.15.28:49024) 2024-05-12T22:17:25.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.345+0000 7fa669400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f7706f3b60 0x55f7706f3f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:57960/0 (socket says 172.21.15.28:57960) 2024-05-12T22:17:25.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.345+0000 7fa669e00640 1 -- 172.21.15.28:0/3992350404 learned_addr learned my addr 172.21.15.28:0/3992350404 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:17:25.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.345+0000 7fa669400640 1 --2- 172.21.15.28:0/3992350404 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f7706feb50 0x55f770700f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:25.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.346+0000 7fa669400640 1 -- 172.21.15.28:0/3992350404 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f7706feb50 msgr2=0x55f770700f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:17:25.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.346+0000 7fa669400640 1 --2- 172.21.15.28:0/3992350404 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f7706feb50 0x55f770700f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:25.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.346+0000 7fa669400640 1 -- 172.21.15.28:0/3992350404 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7706f4480 msgr2=0x55f7706fe610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:17:25.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.346+0000 7fa669400640 1 --2- 172.21.15.28:0/3992350404 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7706f4480 0x55f7706fe610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:25.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.346+0000 7fa669400640 1 -- 172.21.15.28:0/3992350404 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f77036c3f0 con 0x55f7706f3b60 2024-05-12T22:17:25.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.346+0000 7fa669e00640 1 --2- 172.21.15.28:0/3992350404 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7706f4480 0x55f7706fe610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:17:25.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.346+0000 7fa65ea00640 1 --2- 172.21.15.28:0/1363362311 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fa64c0a2d80 0x7fa64c0a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:25.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.346+0000 7fa669400640 1 --2- 172.21.15.28:0/3992350404 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f7706f3b60 0x55f7706f3f40 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fa6641019e0 tx=0x7fa6540095b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fd36a21e26c44e30 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:17:25.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.346+0000 7fa65ea00640 1 -- 172.21.15.28:0/1363362311 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8513+0+0 (secure 0 0 0) 0x7fa6600f0080 con 0x7fa6641098e0 2024-05-12T22:17:25.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.346+0000 7fa669e00640 1 --2- 172.21.15.28:0/1363362311 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fa64c0a2d80 0x7fa64c0a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:25.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.346+0000 7fa653000640 1 -- 172.21.15.28:0/3992350404 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa65400fc20 con 0x55f7706f3b60 2024-05-12T22:17:25.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.346+0000 7fa653000640 1 -- 172.21.15.28:0/3992350404 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa65400fdc0 con 0x55f7706f3b60 2024-05-12T22:17:25.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.346+0000 7fa653000640 1 -- 172.21.15.28:0/3992350404 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa65400e3b0 con 0x55f7706f3b60 2024-05-12T22:17:25.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.347+0000 7fa66f673740 1 -- 172.21.15.28:0/3992350404 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f7706f3b60 msgr2=0x55f7706f3f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:17:25.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.347+0000 7fa669e00640 1 --2- 172.21.15.28:0/1363362311 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fa64c0a2d80 0x7fa64c0a5240 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x55f7706f5120 tx=0x7fa658005f90 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:17:25.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.347+0000 7fa66f673740 1 --2- 172.21.15.28:0/3992350404 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f7706f3b60 0x55f7706f3f40 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fa6641019e0 tx=0x7fa6540095b0 comp rx=0 tx=0).stop 2024-05-12T22:17:25.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.347+0000 7fa66f673740 1 -- 172.21.15.28:0/3992350404 shutdown_connections 2024-05-12T22:17:25.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.347+0000 7fa66f673740 1 --2- 172.21.15.28:0/3992350404 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f7706feb50 0x55f770700f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:25.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.347+0000 7fa66f673740 1 --2- 172.21.15.28:0/3992350404 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7706f4480 0x55f7706fe610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:25.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.347+0000 7fa66f673740 1 --2- 172.21.15.28:0/3992350404 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f7706f3b60 0x55f7706f3f40 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:25.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.347+0000 7fa66f673740 1 -- 172.21.15.28:0/3992350404 >> 172.21.15.28:0/3992350404 conn(0x55f7706efe80 msgr2=0x55f7706f0260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:17:25.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.347+0000 7fa66f673740 1 -- 172.21.15.28:0/3992350404 shutdown_connections 2024-05-12T22:17:25.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.347+0000 7fa66f673740 1 -- 172.21.15.28:0/3992350404 wait complete. 2024-05-12T22:17:25.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.348+0000 7fa66f673740 1 -- start start 2024-05-12T22:17:25.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.348+0000 7fa66f673740 1 Processor -- start 2024-05-12T22:17:25.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.349+0000 7fa66f673740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f7706f3b60 0x55f7706fbae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:25.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.349+0000 7fa66f673740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f7706f4480 0x55f7706fc020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:25.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.349+0000 7fa66f673740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7706feb50 0x55f7706fc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:25.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.349+0000 7fa66f673740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f770701940 con 0x55f7706feb50 2024-05-12T22:17:25.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.349+0000 7fa66f673740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x55f770613a70 con 0x55f7706f4480 2024-05-12T22:17:25.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.349+0000 7fa66f673740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55f770701ac0 con 0x55f7706f3b60 2024-05-12T22:17:25.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.349+0000 7fa66f673740 10 client.0 _ll_register_callbacks cb 0x55f7705f6ba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:17:25.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.349+0000 7fa66a800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7706feb50 0x55f7706fc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:25.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.349+0000 7fa669400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f7706f4480 0x55f7706fc020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:25.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.349+0000 7fa669400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f7706f4480 0x55f7706fc020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:44752/0 (socket says 172.21.15.28:44752) 2024-05-12T22:17:25.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.349+0000 7fa669400640 1 -- 172.21.15.28:0/2759620637 learned_addr learned my addr 172.21.15.28:0/2759620637 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:17:25.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.349+0000 7fa669e00640 1 --2- 172.21.15.28:0/2759620637 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f7706f3b60 0x55f7706fbae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:25.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.349+0000 7fa669400640 1 -- 172.21.15.28:0/2759620637 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f7706f3b60 msgr2=0x55f7706fbae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:17:25.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.349+0000 7fa669400640 1 --2- 172.21.15.28:0/2759620637 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f7706f3b60 0x55f7706fbae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:25.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.349+0000 7fa669400640 1 -- 172.21.15.28:0/2759620637 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7706feb50 msgr2=0x55f7706fc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:17:25.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.349+0000 7fa669400640 1 --2- 172.21.15.28:0/2759620637 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7706feb50 0x55f7706fc560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:25.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.349+0000 7fa669400640 1 -- 172.21.15.28:0/2759620637 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f7704367d0 con 0x55f7706f4480 2024-05-12T22:17:25.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.350+0000 7fa669e00640 1 --2- 172.21.15.28:0/2759620637 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f7706f3b60 0x55f7706fbae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:17:25.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.350+0000 7fa669400640 1 --2- 172.21.15.28:0/2759620637 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f7706f4480 0x55f7706fc020 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fa654011570 tx=0x7fa6540115a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d0ad6f3d1214d90 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:17:25.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.350+0000 7fa647400640 10 client.0 ms_handle_connect on v2:172.21.15.86:3300/0 2024-05-12T22:17:25.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.350+0000 7fa647400640 1 -- 172.21.15.28:0/2759620637 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa654003800 con 0x55f7706f4480 2024-05-12T22:17:25.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.350+0000 7fa647400640 1 -- 172.21.15.28:0/2759620637 <== mon.2 v2:172.21.15.86:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8513+0+0 (secure 0 0 0) 0x7fa654019070 con 0x55f7706f4480 2024-05-12T22:17:25.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.350+0000 7fa66f673740 1 -- 172.21.15.28:0/2759620637 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa65400fdc0 con 0x55f7706f4480 2024-05-12T22:17:25.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.350+0000 7fa647400640 1 -- 172.21.15.28:0/2759620637 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fa654004000 con 0x55f7706f4480 2024-05-12T22:17:25.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.350+0000 7fa66f673740 10 client.27095 fetch_fsmap learned FSMap version 26 2024-05-12T22:17:25.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.350+0000 7fa66f673740 1 -- 172.21.15.28:0/2759620637 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55f77070dc00 con 0x55f7706f4480 2024-05-12T22:17:25.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.351+0000 7fa647400640 1 -- 172.21.15.28:0/2759620637 <== mon.2 v2:172.21.15.86:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa65400fdc0 con 0x55f7706f4480 2024-05-12T22:17:25.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.351+0000 7fa647400640 1 -- 172.21.15.28:0/2759620637 <== mon.2 v2:172.21.15.86:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fa65401d030 con 0x55f7706f4480 2024-05-12T22:17:25.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.351+0000 7fa66f673740 10 client.27095 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:17:25.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.351+0000 7fa66f673740 10 client.27095 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:17:25.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.351+0000 7fa66f673740 20 client.27095 populate_metadata read hostname 'smithi028' 2024-05-12T22:17:25.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.351+0000 7fa66f673740 1 --2- 172.21.15.28:0/2759620637 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55f77070efb0 0x55f7707113a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:25.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.351+0000 7fa66f673740 4 client.27095 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-12T22:17:25.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.351+0000 7fa66f673740 1 -- 172.21.15.28:0/2759620637 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f770666160 con 0x55f77070efb0 2024-05-12T22:17:25.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.351+0000 7fa669e00640 1 --2- 172.21.15.28:0/2759620637 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55f77070efb0 0x55f7707113a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:25.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.352+0000 7fa669e00640 1 --2- 172.21.15.28:0/2759620637 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55f77070efb0 0x55f7707113a0 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7d45d852f3e4c9bc server_cookie=6e284e1d77a16644 in_seq=0 out_seq=0 2024-05-12T22:17:25.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.352+0000 7fa647400640 10 client.27095 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:17:25.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.353+0000 7fa647400640 1 -- 172.21.15.28:0/2759620637 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f770666160 con 0x55f77070efb0 2024-05-12T22:17:25.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.353+0000 7fa647400640 10 client.27095 handle_command_reply: tid=0 2024-05-12T22:17:25.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.353+0000 7fa66f673740 1 client.27095 shutdown 2024-05-12T22:17:25.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.353+0000 7fa66f673740 1 -- 172.21.15.28:0/2759620637 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f7706f4480 msgr2=0x55f7706fc020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:17:25.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.353+0000 7fa66f673740 1 --2- 172.21.15.28:0/2759620637 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f7706f4480 0x55f7706fc020 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fa654011570 tx=0x7fa6540115a0 comp rx=0 tx=0).stop 2024-05-12T22:17:25.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.353+0000 7fa66f673740 1 -- 172.21.15.28:0/2759620637 shutdown_connections 2024-05-12T22:17:25.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.353+0000 7fa66f673740 1 --2- 172.21.15.28:0/2759620637 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55f77070efb0 0x55f7707113a0 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:25.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.353+0000 7fa66f673740 1 --2- 172.21.15.28:0/2759620637 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7706feb50 0x55f7706fc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:25.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.353+0000 7fa66f673740 1 --2- 172.21.15.28:0/2759620637 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f7706f4480 0x55f7706fc020 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:25.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.353+0000 7fa66f673740 1 --2- 172.21.15.28:0/2759620637 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f7706f3b60 0x55f7706fbae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:25.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.353+0000 7fa647400640 0 client.27095 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:17:25.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.353+0000 7fa66f673740 1 -- 172.21.15.28:0/2759620637 >> 172.21.15.28:0/2759620637 conn(0x55f7706efe80 msgr2=0x55f7706f5b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:17:25.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.354+0000 7fa66f673740 1 -- 172.21.15.28:0/2759620637 shutdown_connections 2024-05-12T22:17:25.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.354+0000 7fa66f673740 1 -- 172.21.15.28:0/2759620637 wait complete. 2024-05-12T22:17:25.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.354+0000 7fa66f673740 20 client.27095 trim_cache size 0 max 16384 2024-05-12T22:17:25.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.368+0000 7fa66f673740 1 Processor -- start 2024-05-12T22:17:25.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.369+0000 7fa66f673740 1 -- start start 2024-05-12T22:17:25.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.369+0000 7fa66f673740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7706f3b60 0x55f77070a830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:25.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.369+0000 7fa66f673740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f7706f4480 0x55f770716ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:25.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.369+0000 7fa66f673740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f7706feb50 0x55f770717000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:25.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.369+0000 7fa66f673740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa6641b1e10 con 0x55f7706f3b60 2024-05-12T22:17:25.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.369+0000 7fa66f673740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fa6641af6e0 con 0x55f7706f4480 2024-05-12T22:17:25.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.369+0000 7fa66f673740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa6641ad550 con 0x55f7706feb50 2024-05-12T22:17:25.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.369+0000 7fa669400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f7706f4480 0x55f770716ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:25.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.369+0000 7fa669400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f7706f4480 0x55f770716ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:44754/0 (socket says 172.21.15.28:44754) 2024-05-12T22:17:25.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.369+0000 7fa669400640 1 -- 172.21.15.28:0/525681008 learned_addr learned my addr 172.21.15.28:0/525681008 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:17:25.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.369+0000 7fa669e00640 1 --2- 172.21.15.28:0/525681008 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7706f3b60 0x55f77070a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:25.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.369+0000 7fa66a800640 1 --2- 172.21.15.28:0/525681008 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f7706feb50 0x55f770717000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:25.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.370+0000 7fa669400640 1 -- 172.21.15.28:0/525681008 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f7706feb50 msgr2=0x55f770717000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:17:25.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.370+0000 7fa669400640 1 --2- 172.21.15.28:0/525681008 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f7706feb50 0x55f770717000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:25.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.370+0000 7fa669400640 1 -- 172.21.15.28:0/525681008 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7706f3b60 msgr2=0x55f77070a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:17:25.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.370+0000 7fa669400640 1 --2- 172.21.15.28:0/525681008 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7706f3b60 0x55f77070a830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:25.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.370+0000 7fa669400640 1 -- 172.21.15.28:0/525681008 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f7706fa050 con 0x55f7706f4480 2024-05-12T22:17:25.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.370+0000 7fa669e00640 1 --2- 172.21.15.28:0/525681008 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7706f3b60 0x55f77070a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:17:25.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.370+0000 7fa669400640 1 --2- 172.21.15.28:0/525681008 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f7706f4480 0x55f770716ac0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fa654004ca0 tx=0x7fa654003b40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c6e0fde61ef972f9 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:17:25.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.370+0000 7fa652600640 1 -- 172.21.15.28:0/525681008 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa654036070 con 0x55f7706f4480 2024-05-12T22:17:25.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.370+0000 7fa652600640 1 -- 172.21.15.28:0/525681008 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa654009530 con 0x55f7706f4480 2024-05-12T22:17:25.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.370+0000 7fa652600640 1 -- 172.21.15.28:0/525681008 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6540043e0 con 0x55f7706f4480 2024-05-12T22:17:25.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.371+0000 7fa66f673740 1 -- 172.21.15.28:0/525681008 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f7706f4480 msgr2=0x55f770716ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:17:25.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.371+0000 7fa66f673740 1 --2- 172.21.15.28:0/525681008 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f7706f4480 0x55f770716ac0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fa654004ca0 tx=0x7fa654003b40 comp rx=0 tx=0).stop 2024-05-12T22:17:25.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.371+0000 7fa66f673740 1 -- 172.21.15.28:0/525681008 shutdown_connections 2024-05-12T22:17:25.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.371+0000 7fa66f673740 1 --2- 172.21.15.28:0/525681008 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f7706feb50 0x55f770717000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:25.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.371+0000 7fa66f673740 1 --2- 172.21.15.28:0/525681008 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f7706f4480 0x55f770716ac0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:25.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.371+0000 7fa66f673740 1 --2- 172.21.15.28:0/525681008 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7706f3b60 0x55f77070a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:25.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.371+0000 7fa66f673740 1 -- 172.21.15.28:0/525681008 >> 172.21.15.28:0/525681008 conn(0x55f7706efe80 msgr2=0x55f7706f6e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:17:25.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.371+0000 7fa66f673740 1 -- 172.21.15.28:0/525681008 shutdown_connections 2024-05-12T22:17:25.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.371+0000 7fa66f673740 1 -- 172.21.15.28:0/525681008 wait complete. 2024-05-12T22:17:25.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.371+0000 7fa66f673740 1 -- start start 2024-05-12T22:17:25.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.372+0000 7fa66f673740 1 Processor -- start 2024-05-12T22:17:25.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.372+0000 7fa66f673740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f7706f3b60 0x55f7706f8640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:25.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.372+0000 7fa66f673740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7706f4480 0x55f7706f8b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:25.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.372+0000 7fa66f673740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f7706feb50 0x55f7706f90c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:25.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.372+0000 7fa66f673740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa6641adba0 con 0x55f7706f4480 2024-05-12T22:17:25.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.372+0000 7fa66f673740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fa6641af010 con 0x55f7706f3b60 2024-05-12T22:17:25.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.372+0000 7fa66f673740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa6641ae270 con 0x55f7706feb50 2024-05-12T22:17:25.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.372+0000 7fa66f673740 10 client.0 _ll_register_callbacks cb 0x7fa654011a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:17:25.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.372+0000 7fa669e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f7706f3b60 0x55f7706f8640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:25.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.372+0000 7fa669e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f7706f3b60 0x55f7706f8640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:44770/0 (socket says 172.21.15.28:44770) 2024-05-12T22:17:25.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.372+0000 7fa669e00640 1 -- 172.21.15.28:0/1810705244 learned_addr learned my addr 172.21.15.28:0/1810705244 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:17:25.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.373+0000 7fa66a800640 1 --2- 172.21.15.28:0/1810705244 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f7706feb50 0x55f7706f90c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:25.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.373+0000 7fa669400640 1 --2- 172.21.15.28:0/1810705244 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7706f4480 0x55f7706f8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:25.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.373+0000 7fa669e00640 1 -- 172.21.15.28:0/1810705244 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f7706feb50 msgr2=0x55f7706f90c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:17:25.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.373+0000 7fa669e00640 1 --2- 172.21.15.28:0/1810705244 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f7706feb50 0x55f7706f90c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:25.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.373+0000 7fa669e00640 1 -- 172.21.15.28:0/1810705244 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7706f4480 msgr2=0x55f7706f8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:17:25.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.373+0000 7fa669e00640 1 --2- 172.21.15.28:0/1810705244 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7706f4480 0x55f7706f8b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:25.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.373+0000 7fa669e00640 1 -- 172.21.15.28:0/1810705244 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f77077cb40 con 0x55f7706f3b60 2024-05-12T22:17:25.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.373+0000 7fa669400640 1 --2- 172.21.15.28:0/1810705244 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7706f4480 0x55f7706f8b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:17:25.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.373+0000 7fa66a800640 1 --2- 172.21.15.28:0/1810705244 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f7706feb50 0x55f7706f90c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:17:25.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.373+0000 7fa669e00640 1 --2- 172.21.15.28:0/1810705244 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f7706f3b60 0x55f7706f8640 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x55f770711a60 tx=0x7fa658004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e038ce7ce2191d9e server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:17:25.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.373+0000 7fa647400640 10 client.0 ms_handle_connect on v2:172.21.15.86:3300/0 2024-05-12T22:17:25.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.373+0000 7fa647400640 1 -- 172.21.15.28:0/1810705244 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa65802f070 con 0x55f7706f3b60 2024-05-12T22:17:25.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.373+0000 7fa647400640 1 -- 172.21.15.28:0/1810705244 <== mon.2 v2:172.21.15.86:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8513+0+0 (secure 0 0 0) 0x7fa65800fca0 con 0x55f7706f3b60 2024-05-12T22:17:25.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.373+0000 7fa66f673740 1 -- 172.21.15.28:0/1810705244 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55f7706604b0 con 0x55f7706f3b60 2024-05-12T22:17:25.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.374+0000 7fa647400640 1 -- 172.21.15.28:0/1810705244 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fa658038050 con 0x55f7706f3b60 2024-05-12T22:17:25.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.374+0000 7fa66f673740 10 client.27101 fetch_fsmap learned FSMap version 26 2024-05-12T22:17:25.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.374+0000 7fa66f673740 1 -- 172.21.15.28:0/1810705244 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55f77077dd80 con 0x55f7706f3b60 2024-05-12T22:17:25.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.374+0000 7fa647400640 1 -- 172.21.15.28:0/1810705244 <== mon.2 v2:172.21.15.86:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55f7706604b0 con 0x55f7706f3b60 2024-05-12T22:17:25.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.374+0000 7fa647400640 1 -- 172.21.15.28:0/1810705244 <== mon.2 v2:172.21.15.86:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fa658032020 con 0x55f7706f3b60 2024-05-12T22:17:25.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.374+0000 7fa66f673740 10 client.27101 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:17:25.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.374+0000 7fa66f673740 10 client.27101 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:17:25.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.374+0000 7fa66f673740 20 client.27101 populate_metadata read hostname 'smithi028' 2024-05-12T22:17:25.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.374+0000 7fa66f673740 1 --2- 172.21.15.28:0/1810705244 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55f77077e4e0 0x55f770780900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:25.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.374+0000 7fa66f673740 4 client.27101 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-12T22:17:25.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.374+0000 7fa66f673740 1 -- 172.21.15.28:0/1810705244 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f7705dd6e0 con 0x55f77077e4e0 2024-05-12T22:17:25.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.375+0000 7fa669400640 1 --2- 172.21.15.28:0/1810705244 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55f77077e4e0 0x55f770780900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:25.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.375+0000 7fa669400640 1 --2- 172.21.15.28:0/1810705244 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55f77077e4e0 0x55f770780900 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=43b3b47a074de54f server_cookie=e0c4061247c8afe4 in_seq=0 out_seq=0 2024-05-12T22:17:25.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.375+0000 7fa647400640 10 client.27101 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:17:25.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.375+0000 7fa647400640 1 -- 172.21.15.28:0/1810705244 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f7705dd6e0 con 0x55f77077e4e0 2024-05-12T22:17:25.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.375+0000 7fa647400640 10 client.27101 handle_command_reply: tid=0 2024-05-12T22:17:25.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.375+0000 7fa66f673740 1 client.27101 shutdown 2024-05-12T22:17:25.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.376+0000 7fa66f673740 1 -- 172.21.15.28:0/1810705244 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f7706f3b60 msgr2=0x55f7706f8640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:17:25.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.376+0000 7fa66f673740 1 --2- 172.21.15.28:0/1810705244 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f7706f3b60 0x55f7706f8640 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x55f770711a60 tx=0x7fa658004730 comp rx=0 tx=0).stop 2024-05-12T22:17:25.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.376+0000 7fa66f673740 1 -- 172.21.15.28:0/1810705244 shutdown_connections 2024-05-12T22:17:25.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.376+0000 7fa66f673740 1 --2- 172.21.15.28:0/1810705244 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55f77077e4e0 0x55f770780900 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:25.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.376+0000 7fa66f673740 1 --2- 172.21.15.28:0/1810705244 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f7706feb50 0x55f7706f90c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:25.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.376+0000 7fa66f673740 1 --2- 172.21.15.28:0/1810705244 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7706f4480 0x55f7706f8b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:25.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.376+0000 7fa66f673740 1 --2- 172.21.15.28:0/1810705244 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f7706f3b60 0x55f7706f8640 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:25.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.376+0000 7fa66f673740 1 -- 172.21.15.28:0/1810705244 >> 172.21.15.28:0/1810705244 conn(0x55f7706efe80 msgr2=0x55f7706ee770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:17:25.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.376+0000 7fa647400640 0 client.27101 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:17:25.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.376+0000 7fa66f673740 1 -- 172.21.15.28:0/1810705244 shutdown_connections 2024-05-12T22:17:25.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.376+0000 7fa66f673740 1 -- 172.21.15.28:0/1810705244 wait complete. 2024-05-12T22:17:25.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.376+0000 7fa66f673740 20 client.27101 trim_cache size 0 max 16384 2024-05-12T22:17:25.386 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-05-12T22:17:25.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.377+0000 7fa66b200640 1 -- 172.21.15.28:0/1363362311 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fa64c0a2d80 msgr2=0x7fa64c0a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:17:25.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.377+0000 7fa66b200640 1 --2- 172.21.15.28:0/1363362311 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fa64c0a2d80 0x7fa64c0a5240 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x55f7706f5120 tx=0x7fa658005f90 comp rx=0 tx=0).stop 2024-05-12T22:17:25.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.377+0000 7fa66b200640 1 -- 172.21.15.28:0/1363362311 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6641098e0 msgr2=0x7fa66419ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:17:25.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.377+0000 7fa66b200640 1 --2- 172.21.15.28:0/1363362311 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6641098e0 0x7fa66419ea20 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fa66002ac10 tx=0x7fa6600305d0 comp rx=0 tx=0).stop 2024-05-12T22:17:25.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.377+0000 7fa66b200640 1 -- 172.21.15.28:0/1363362311 shutdown_connections 2024-05-12T22:17:25.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.377+0000 7fa66b200640 1 --2- 172.21.15.28:0/1363362311 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fa64c0a2d80 0x7fa64c0a5240 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:25.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.377+0000 7fa66b200640 1 --2- 172.21.15.28:0/1363362311 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6641098e0 0x7fa66419ea20 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:25.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.377+0000 7fa66b200640 1 --2- 172.21.15.28:0/1363362311 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa664104290 0x7fa66419a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:25.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.377+0000 7fa66b200640 1 --2- 172.21.15.28:0/1363362311 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa6641038d0 0x7fa66419a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:25.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.377+0000 7fa66b200640 1 -- 172.21.15.28:0/1363362311 >> 172.21.15.28:0/1363362311 conn(0x7fa6640ff3a0 msgr2=0x7fa6640ffa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:17:25.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.377+0000 7fa66b200640 1 -- 172.21.15.28:0/1363362311 shutdown_connections 2024-05-12T22:17:25.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:25.377+0000 7fa66b200640 1 -- 172.21.15.28:0/1363362311 wait complete. 2024-05-12T22:17:25.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:17:24 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/1912409738' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:17:26.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:17:25 smithi086 ceph-mon[34297]: pgmap v260: 97 pgs: 97 active+clean; 91 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 93 B/s wr, 8 op/s 2024-05-12T22:17:26.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:17:25 smithi028 ceph-mon[29847]: pgmap v260: 97 pgs: 97 active+clean; 91 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 93 B/s wr, 8 op/s 2024-05-12T22:17:26.396 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-12T22:17:26.397 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-12T22:17:26.397 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-05-12T22:17:26.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:17:25 smithi186 ceph-mon[34785]: pgmap v260: 97 pgs: 97 active+clean; 91 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 93 B/s wr, 8 op/s 2024-05-12T22:17:26.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.627+0000 7fab1d600640 1 -- 172.21.15.28:0/1296313588 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab18107dd0 msgr2=0x7fab18108230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:17:26.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.627+0000 7fab1d600640 1 --2- 172.21.15.28:0/1296313588 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab18107dd0 0x7fab18108230 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fab00009ca0 tx=0x7fab0002a950 comp rx=0 tx=0).stop 2024-05-12T22:17:26.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.627+0000 7fab1d600640 1 -- 172.21.15.28:0/1296313588 shutdown_connections 2024-05-12T22:17:26.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.627+0000 7fab1d600640 1 --2- 172.21.15.28:0/1296313588 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fab18108770 0x7fab18114b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:26.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.627+0000 7fab1d600640 1 --2- 172.21.15.28:0/1296313588 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab18107dd0 0x7fab18108230 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:26.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.627+0000 7fab1d600640 1 --2- 172.21.15.28:0/1296313588 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fab1810ddf0 0x7fab1810e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:26.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.627+0000 7fab1d600640 1 -- 172.21.15.28:0/1296313588 >> 172.21.15.28:0/1296313588 conn(0x7fab18077650 msgr2=0x7fab18106650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:17:26.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.627+0000 7fab1d600640 1 -- 172.21.15.28:0/1296313588 shutdown_connections 2024-05-12T22:17:26.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.627+0000 7fab1d600640 1 -- 172.21.15.28:0/1296313588 wait complete. 2024-05-12T22:17:26.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.628+0000 7fab1d600640 1 Processor -- start 2024-05-12T22:17:26.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.628+0000 7fab1d600640 1 -- start start 2024-05-12T22:17:26.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.629+0000 7fab1d600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fab18107dd0 0x7fab1819e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:26.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.629+0000 7fab1d600640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fab18108770 0x7fab1819eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:26.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.629+0000 7fab1d600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab1810ddf0 0x7fab181a2e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:26.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.629+0000 7fab1d600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fab181180d0 con 0x7fab1810ddf0 2024-05-12T22:17:26.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.629+0000 7fab1d600640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fab18117f50 con 0x7fab18107dd0 2024-05-12T22:17:26.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.629+0000 7fab1d600640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fab18118250 con 0x7fab18108770 2024-05-12T22:17:26.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.629+0000 7fab1cc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab1810ddf0 0x7fab181a2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:26.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.629+0000 7fab1cc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab1810ddf0 0x7fab181a2e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43910/0 (socket says 172.21.15.28:43910) 2024-05-12T22:17:26.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.629+0000 7fab1cc00640 1 -- 172.21.15.28:0/1702756166 learned_addr learned my addr 172.21.15.28:0/1702756166 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:17:26.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.629+0000 7fab17e00640 1 --2- 172.21.15.28:0/1702756166 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fab18107dd0 0x7fab1819e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:26.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.629+0000 7fab17400640 1 --2- 172.21.15.28:0/1702756166 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fab18108770 0x7fab1819eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:26.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.630+0000 7fab17e00640 1 -- 172.21.15.28:0/1702756166 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fab18108770 msgr2=0x7fab1819eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:17:26.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.630+0000 7fab17e00640 1 --2- 172.21.15.28:0/1702756166 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fab18108770 0x7fab1819eb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:26.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.630+0000 7fab17e00640 1 -- 172.21.15.28:0/1702756166 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab1810ddf0 msgr2=0x7fab181a2e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:17:26.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.630+0000 7fab17e00640 1 --2- 172.21.15.28:0/1702756166 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab1810ddf0 0x7fab181a2e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:26.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.630+0000 7fab17e00640 1 -- 172.21.15.28:0/1702756166 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab181a3590 con 0x7fab18107dd0 2024-05-12T22:17:26.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.630+0000 7fab17e00640 1 --2- 172.21.15.28:0/1702756166 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fab18107dd0 0x7fab1819e5f0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fab0800e9f0 tx=0x7fab0800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:17:26.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.630+0000 7fab14c00640 1 -- 172.21.15.28:0/1702756166 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab0800ccc0 con 0x7fab18107dd0 2024-05-12T22:17:26.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.630+0000 7fab14c00640 1 -- 172.21.15.28:0/1702756166 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fab08010910 con 0x7fab18107dd0 2024-05-12T22:17:26.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.630+0000 7fab14c00640 1 -- 172.21.15.28:0/1702756166 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab08018960 con 0x7fab18107dd0 2024-05-12T22:17:26.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.630+0000 7fab1d600640 1 -- 172.21.15.28:0/1702756166 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab181a3880 con 0x7fab18107dd0 2024-05-12T22:17:26.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.630+0000 7fab1d600640 1 -- 172.21.15.28:0/1702756166 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab181ab110 con 0x7fab18107dd0 2024-05-12T22:17:26.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.633+0000 7fab21a13740 1 Processor -- start 2024-05-12T22:17:26.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.634+0000 7fab21a13740 1 -- start start 2024-05-12T22:17:26.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.634+0000 7fab21a13740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5569504cf500 0x5569504cf8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:26.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.634+0000 7fab21a13740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5569504cfeb0 0x5569504d9f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:26.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.634+0000 7fab21a13740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5569504da520 0x5569504dc970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:26.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.634+0000 7fab14c00640 1 -- 172.21.15.28:0/1702756166 <== mon.2 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7fab08010ac0 con 0x7fab18107dd0 2024-05-12T22:17:26.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.634+0000 7fab21a13740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5569504337c0 con 0x5569504cf500 2024-05-12T22:17:26.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.634+0000 7fab21a13740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x5569503db790 con 0x5569504da520 2024-05-12T22:17:26.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.634+0000 7fab21a13740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55695042f780 con 0x5569504cfeb0 2024-05-12T22:17:26.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.634+0000 7fab1cc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5569504cf500 0x5569504cf8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:26.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.635+0000 7fab1cc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5569504cf500 0x5569504cf8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43924/0 (socket says 172.21.15.28:43924) 2024-05-12T22:17:26.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.635+0000 7fab1cc00640 1 -- 172.21.15.28:0/675441830 learned_addr learned my addr 172.21.15.28:0/675441830 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:17:26.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.635+0000 7fab17e00640 1 --2- 172.21.15.28:0/675441830 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5569504cfeb0 0x5569504d9f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:26.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.635+0000 7fab17400640 1 --2- 172.21.15.28:0/675441830 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5569504da520 0x5569504dc970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:26.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.635+0000 7fab1cc00640 1 -- 172.21.15.28:0/675441830 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5569504cfeb0 msgr2=0x5569504d9f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:17:26.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.635+0000 7fab1cc00640 1 --2- 172.21.15.28:0/675441830 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5569504cfeb0 0x5569504d9f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:26.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.635+0000 7fab1cc00640 1 -- 172.21.15.28:0/675441830 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5569504da520 msgr2=0x5569504dc970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:17:26.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.635+0000 7fab1cc00640 1 --2- 172.21.15.28:0/675441830 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5569504da520 0x5569504dc970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:26.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.635+0000 7fab1cc00640 1 -- 172.21.15.28:0/675441830 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5569503fb6a0 con 0x5569504cf500 2024-05-12T22:17:26.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.635+0000 7fab17400640 1 --2- 172.21.15.28:0/675441830 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5569504da520 0x5569504dc970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:17:26.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.635+0000 7fab1cc00640 1 --2- 172.21.15.28:0/675441830 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5569504cf500 0x5569504cf8e0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fab10009d80 tx=0x7fab10011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92dd044a0d177d94 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:17:26.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.636+0000 7fab0d200640 1 -- 172.21.15.28:0/675441830 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab1001e070 con 0x5569504cf500 2024-05-12T22:17:26.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.636+0000 7fab0d200640 1 -- 172.21.15.28:0/675441830 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fab10019b10 con 0x5569504cf500 2024-05-12T22:17:26.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.636+0000 7fab0d200640 1 -- 172.21.15.28:0/675441830 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab1000fb60 con 0x5569504cf500 2024-05-12T22:17:26.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.636+0000 7fab14c00640 1 --2- 172.21.15.28:0/1702756166 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7faafc0a2d80 0x7faafc0a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:26.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.636+0000 7fab21a13740 1 -- 172.21.15.28:0/675441830 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5569504cf500 msgr2=0x5569504cf8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:17:26.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.636+0000 7fab21a13740 1 --2- 172.21.15.28:0/675441830 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5569504cf500 0x5569504cf8e0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fab10009d80 tx=0x7fab10011810 comp rx=0 tx=0).stop 2024-05-12T22:17:26.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.637+0000 7fab14c00640 1 -- 172.21.15.28:0/1702756166 <== mon.2 v2:172.21.15.86:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8513+0+0 (secure 0 0 0) 0x7fab080cf740 con 0x7fab18107dd0 2024-05-12T22:17:26.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.637+0000 7fab17e00640 1 --2- 172.21.15.28:0/1702756166 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7faafc0a2d80 0x7faafc0a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:26.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.637+0000 7fab21a13740 1 -- 172.21.15.28:0/675441830 shutdown_connections 2024-05-12T22:17:26.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.637+0000 7fab21a13740 1 --2- 172.21.15.28:0/675441830 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5569504da520 0x5569504dc970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:26.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.637+0000 7fab21a13740 1 --2- 172.21.15.28:0/675441830 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5569504cfeb0 0x5569504d9f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:26.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.637+0000 7fab21a13740 1 --2- 172.21.15.28:0/675441830 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5569504cf500 0x5569504cf8e0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:26.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.637+0000 7fab21a13740 1 -- 172.21.15.28:0/675441830 >> 172.21.15.28:0/675441830 conn(0x5569504cb5b0 msgr2=0x5569504cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:17:26.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.637+0000 7fab21a13740 1 -- 172.21.15.28:0/675441830 shutdown_connections 2024-05-12T22:17:26.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.637+0000 7fab21a13740 1 -- 172.21.15.28:0/675441830 wait complete. 2024-05-12T22:17:26.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.637+0000 7fab17e00640 1 --2- 172.21.15.28:0/1702756166 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7faafc0a2d80 0x7faafc0a5240 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x5569504d0ac0 tx=0x7fab080d5530 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:17:26.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.637+0000 7fab21a13740 1 -- start start 2024-05-12T22:17:26.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.638+0000 7fab21a13740 1 Processor -- start 2024-05-12T22:17:26.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.638+0000 7fab21a13740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5569504cf500 0x5569504d7440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:26.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.638+0000 7fab21a13740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5569504cfeb0 0x5569504d7980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:26.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.638+0000 7fab21a13740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5569504da520 0x5569504d7ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:26.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.638+0000 7fab21a13740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5569504dd530 con 0x5569504cfeb0 2024-05-12T22:17:26.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.638+0000 7fab21a13740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x5569503eea90 con 0x5569504da520 2024-05-12T22:17:26.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.638+0000 7fab21a13740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5569504dd6b0 con 0x5569504cf500 2024-05-12T22:17:26.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.638+0000 7fab21a13740 10 client.0 _ll_register_callbacks cb 0x556950429af0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:17:26.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.638+0000 7fab17400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5569504cfeb0 0x5569504d7980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:26.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.639+0000 7fab1cc00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5569504cf500 0x5569504d7440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:26.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.639+0000 7fab1cc00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5569504cf500 0x5569504d7440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:38314/0 (socket says 172.21.15.28:38314) 2024-05-12T22:17:26.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.639+0000 7fab17400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5569504cfeb0 0x5569504d7980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43938/0 (socket says 172.21.15.28:43938) 2024-05-12T22:17:26.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.639+0000 7fab1cc00640 1 -- 172.21.15.28:0/3872184140 learned_addr learned my addr 172.21.15.28:0/3872184140 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:17:26.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.639+0000 7fab1cc00640 1 --2- 172.21.15.28:0/3872184140 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5569504da520 0x5569504d7ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:26.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.639+0000 7fab17400640 1 -- 172.21.15.28:0/3872184140 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5569504cf500 msgr2=0x5569504d7440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:17:26.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.639+0000 7fab17400640 1 --2- 172.21.15.28:0/3872184140 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5569504cf500 0x5569504d7440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:26.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.639+0000 7fab17400640 1 -- 172.21.15.28:0/3872184140 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5569504da520 msgr2=0x5569504d7ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:17:26.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.639+0000 7fab17400640 1 --2- 172.21.15.28:0/3872184140 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5569504da520 0x5569504d7ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:26.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.639+0000 7fab17400640 1 -- 172.21.15.28:0/3872184140 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55695020f7e0 con 0x5569504cfeb0 2024-05-12T22:17:26.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.639+0000 7fab17400640 1 --2- 172.21.15.28:0/3872184140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5569504cfeb0 0x5569504d7980 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fab00005e30 tx=0x7fab0002ff50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=caab7aa8555a56e server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:17:26.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.639+0000 7fab05600640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-12T22:17:26.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.640+0000 7fab05600640 1 -- 172.21.15.28:0/3872184140 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab00049060 con 0x5569504cfeb0 2024-05-12T22:17:26.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.640+0000 7fab05600640 1 -- 172.21.15.28:0/3872184140 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8513+0+0 (secure 0 0 0) 0x7fab00003970 con 0x5569504cfeb0 2024-05-12T22:17:26.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.640+0000 7fab21a13740 1 -- 172.21.15.28:0/3872184140 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fab10019b10 con 0x5569504cfeb0 2024-05-12T22:17:26.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.640+0000 7fab05600640 1 -- 172.21.15.28:0/3872184140 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fab00034040 con 0x5569504cfeb0 2024-05-12T22:17:26.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.640+0000 7fab21a13740 10 client.16980 fetch_fsmap learned FSMap version 26 2024-05-12T22:17:26.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.640+0000 7fab21a13740 1 -- 172.21.15.28:0/3872184140 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55695026ff70 con 0x5569504cfeb0 2024-05-12T22:17:26.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.640+0000 7fab05600640 1 -- 172.21.15.28:0/3872184140 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fab10019b10 con 0x5569504cfeb0 2024-05-12T22:17:26.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.640+0000 7fab05600640 1 -- 172.21.15.28:0/3872184140 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fab00030440 con 0x5569504cfeb0 2024-05-12T22:17:26.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.641+0000 7fab21a13740 10 client.16980 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:17:26.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.641+0000 7fab21a13740 10 client.16980 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:17:26.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.641+0000 7fab21a13740 20 client.16980 populate_metadata read hostname 'smithi028' 2024-05-12T22:17:26.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.641+0000 7fab21a13740 1 --2- 172.21.15.28:0/3872184140 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5569504ea750 0x5569504ecb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:26.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.641+0000 7fab21a13740 4 client.16980 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-12T22:17:26.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.641+0000 7fab21a13740 1 -- 172.21.15.28:0/3872184140 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5569504473a0 con 0x5569504ea750 2024-05-12T22:17:26.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.641+0000 7fab1cc00640 1 --2- 172.21.15.28:0/3872184140 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5569504ea750 0x5569504ecb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:26.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.642+0000 7fab1cc00640 1 --2- 172.21.15.28:0/3872184140 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5569504ea750 0x5569504ecb40 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5f65f7d4c14d8e6 server_cookie=cee3003cf6c0ad56 in_seq=0 out_seq=0 2024-05-12T22:17:26.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.642+0000 7fab05600640 10 client.16980 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:17:26.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.642+0000 7fab05600640 1 -- 172.21.15.28:0/3872184140 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5569504473a0 con 0x5569504ea750 2024-05-12T22:17:26.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.642+0000 7fab05600640 10 client.16980 handle_command_reply: tid=0 2024-05-12T22:17:26.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.643+0000 7fab21a13740 1 client.16980 shutdown 2024-05-12T22:17:26.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.643+0000 7fab21a13740 1 -- 172.21.15.28:0/3872184140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5569504cfeb0 msgr2=0x5569504d7980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:17:26.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.643+0000 7fab21a13740 1 --2- 172.21.15.28:0/3872184140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5569504cfeb0 0x5569504d7980 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fab00005e30 tx=0x7fab0002ff50 comp rx=0 tx=0).stop 2024-05-12T22:17:26.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.643+0000 7fab21a13740 1 -- 172.21.15.28:0/3872184140 shutdown_connections 2024-05-12T22:17:26.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.643+0000 7fab21a13740 1 --2- 172.21.15.28:0/3872184140 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5569504ea750 0x5569504ecb40 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:26.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.643+0000 7fab21a13740 1 --2- 172.21.15.28:0/3872184140 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5569504da520 0x5569504d7ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:26.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.643+0000 7fab21a13740 1 --2- 172.21.15.28:0/3872184140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5569504cfeb0 0x5569504d7980 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:26.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.643+0000 7fab21a13740 1 --2- 172.21.15.28:0/3872184140 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5569504cf500 0x5569504d7440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:26.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.643+0000 7fab05600640 0 client.16980 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:17:26.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.643+0000 7fab21a13740 1 -- 172.21.15.28:0/3872184140 >> 172.21.15.28:0/3872184140 conn(0x5569504cb5b0 msgr2=0x5569504d1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:17:26.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.643+0000 7fab21a13740 1 -- 172.21.15.28:0/3872184140 shutdown_connections 2024-05-12T22:17:26.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.643+0000 7fab21a13740 1 -- 172.21.15.28:0/3872184140 wait complete. 2024-05-12T22:17:26.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.643+0000 7fab21a13740 20 client.16980 trim_cache size 0 max 16384 2024-05-12T22:17:26.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.663+0000 7fab21a13740 1 Processor -- start 2024-05-12T22:17:26.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.663+0000 7fab21a13740 1 -- start start 2024-05-12T22:17:26.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.663+0000 7fab21a13740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5569504cf500 0x5569504e63f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:26.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.663+0000 7fab21a13740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5569504cfeb0 0x5569504d7ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:26.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.663+0000 7fab21a13740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5569504da520 0x5569504f23f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:26.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.663+0000 7fab21a13740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fab181b6190 con 0x5569504cfeb0 2024-05-12T22:17:26.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.663+0000 7fab21a13740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fab181b3a60 con 0x5569504cf500 2024-05-12T22:17:26.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.663+0000 7fab21a13740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fab181b18d0 con 0x5569504da520 2024-05-12T22:17:26.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.664+0000 7fab1cc00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5569504cf500 0x5569504e63f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:26.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.664+0000 7fab1cc00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5569504cf500 0x5569504e63f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:51734/0 (socket says 172.21.15.28:51734) 2024-05-12T22:17:26.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.664+0000 7fab1cc00640 1 -- 172.21.15.28:0/3036456830 learned_addr learned my addr 172.21.15.28:0/3036456830 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:17:26.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.664+0000 7fab17400640 1 --2- 172.21.15.28:0/3036456830 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5569504cfeb0 0x5569504d7ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:26.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.664+0000 7fab1cc00640 1 --2- 172.21.15.28:0/3036456830 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5569504da520 0x5569504f23f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:26.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.664+0000 7fab1cc00640 1 -- 172.21.15.28:0/3036456830 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5569504da520 msgr2=0x5569504f23f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:17:26.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.664+0000 7fab1cc00640 1 --2- 172.21.15.28:0/3036456830 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5569504da520 0x5569504f23f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:26.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.664+0000 7fab1cc00640 1 -- 172.21.15.28:0/3036456830 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5569504cfeb0 msgr2=0x5569504d7ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:17:26.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.664+0000 7fab1cc00640 1 --2- 172.21.15.28:0/3036456830 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5569504cfeb0 0x5569504d7ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:26.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.664+0000 7fab1cc00640 1 -- 172.21.15.28:0/3036456830 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5569504d2260 con 0x5569504cf500 2024-05-12T22:17:26.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.664+0000 7fab17400640 1 --2- 172.21.15.28:0/3036456830 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5569504cfeb0 0x5569504d7ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:17:26.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.664+0000 7fab1cc00640 1 --2- 172.21.15.28:0/3036456830 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5569504cf500 0x5569504e63f0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fab1001c040 tx=0x7fab10011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dc306466e9612d1f server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:17:26.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.665+0000 7fab07e00640 1 -- 172.21.15.28:0/3036456830 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab1001e040 con 0x5569504cf500 2024-05-12T22:17:26.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.665+0000 7fab07e00640 1 -- 172.21.15.28:0/3036456830 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fab1000f7f0 con 0x5569504cf500 2024-05-12T22:17:26.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.665+0000 7fab07e00640 1 -- 172.21.15.28:0/3036456830 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab10019690 con 0x5569504cf500 2024-05-12T22:17:26.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.665+0000 7fab21a13740 1 -- 172.21.15.28:0/3036456830 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5569504cf500 msgr2=0x5569504e63f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:17:26.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.665+0000 7fab21a13740 1 --2- 172.21.15.28:0/3036456830 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5569504cf500 0x5569504e63f0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fab1001c040 tx=0x7fab10011810 comp rx=0 tx=0).stop 2024-05-12T22:17:26.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.665+0000 7fab21a13740 1 -- 172.21.15.28:0/3036456830 shutdown_connections 2024-05-12T22:17:26.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.665+0000 7fab21a13740 1 --2- 172.21.15.28:0/3036456830 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5569504da520 0x5569504f23f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:26.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.665+0000 7fab21a13740 1 --2- 172.21.15.28:0/3036456830 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5569504cfeb0 0x5569504d7ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:26.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.665+0000 7fab21a13740 1 --2- 172.21.15.28:0/3036456830 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5569504cf500 0x5569504e63f0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:26.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.665+0000 7fab21a13740 1 -- 172.21.15.28:0/3036456830 >> 172.21.15.28:0/3036456830 conn(0x5569504cb5b0 msgr2=0x5569504d15c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:17:26.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.666+0000 7fab21a13740 1 -- 172.21.15.28:0/3036456830 shutdown_connections 2024-05-12T22:17:26.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.666+0000 7fab21a13740 1 -- 172.21.15.28:0/3036456830 wait complete. 2024-05-12T22:17:26.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.666+0000 7fab21a13740 1 -- start start 2024-05-12T22:17:26.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.666+0000 7fab21a13740 1 Processor -- start 2024-05-12T22:17:26.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.667+0000 7fab21a13740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5569504cf500 0x5569504d41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:26.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.667+0000 7fab21a13740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5569504cfeb0 0x5569504d4730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:26.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.667+0000 7fab21a13740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5569504da520 0x5569504d4c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:26.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.667+0000 7fab21a13740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fab181b1f20 con 0x5569504cfeb0 2024-05-12T22:17:26.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.667+0000 7fab21a13740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fab181b3390 con 0x5569504da520 2024-05-12T22:17:26.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.667+0000 7fab21a13740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fab181b25f0 con 0x5569504cf500 2024-05-12T22:17:26.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.667+0000 7fab21a13740 10 client.0 _ll_register_callbacks cb 0x7fab00009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:17:26.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.667+0000 7fab17400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5569504cfeb0 0x5569504d4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:26.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.667+0000 7fab1cc00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5569504cf500 0x5569504d41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:26.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.667+0000 7fab17400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5569504cfeb0 0x5569504d4730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43958/0 (socket says 172.21.15.28:43958) 2024-05-12T22:17:26.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.667+0000 7fab17400640 1 -- 172.21.15.28:0/2881833000 learned_addr learned my addr 172.21.15.28:0/2881833000 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:17:26.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.667+0000 7fab1cc00640 1 --2- 172.21.15.28:0/2881833000 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5569504da520 0x5569504d4c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:26.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.668+0000 7fab17400640 1 -- 172.21.15.28:0/2881833000 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5569504cf500 msgr2=0x5569504d41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:17:26.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.668+0000 7fab17400640 1 --2- 172.21.15.28:0/2881833000 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5569504cf500 0x5569504d41f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:26.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.668+0000 7fab17400640 1 -- 172.21.15.28:0/2881833000 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5569504da520 msgr2=0x5569504d4c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:17:26.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.668+0000 7fab17400640 1 --2- 172.21.15.28:0/2881833000 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5569504da520 0x5569504d4c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:26.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.668+0000 7fab17400640 1 -- 172.21.15.28:0/2881833000 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556950107170 con 0x5569504cfeb0 2024-05-12T22:17:26.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.668+0000 7fab17400640 1 --2- 172.21.15.28:0/2881833000 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5569504cfeb0 0x5569504d4730 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fab000309d0 tx=0x7fab00005ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a8f216a4b7a39b7 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:17:26.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.668+0000 7fab05600640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-12T22:17:26.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.668+0000 7fab05600640 1 -- 172.21.15.28:0/2881833000 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab00049070 con 0x5569504cfeb0 2024-05-12T22:17:26.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.668+0000 7fab05600640 1 -- 172.21.15.28:0/2881833000 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8513+0+0 (secure 0 0 0) 0x7fab00030c50 con 0x5569504cfeb0 2024-05-12T22:17:26.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.668+0000 7fab21a13740 1 -- 172.21.15.28:0/2881833000 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fab1000f7f0 con 0x5569504cfeb0 2024-05-12T22:17:26.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.669+0000 7fab05600640 1 -- 172.21.15.28:0/2881833000 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fab00009540 con 0x5569504cfeb0 2024-05-12T22:17:26.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.669+0000 7fab21a13740 10 client.16986 fetch_fsmap learned FSMap version 26 2024-05-12T22:17:26.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.669+0000 7fab21a13740 1 -- 172.21.15.28:0/2881833000 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x556950550eb0 con 0x5569504cfeb0 2024-05-12T22:17:26.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.669+0000 7fab05600640 1 -- 172.21.15.28:0/2881833000 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fab1000f7f0 con 0x5569504cfeb0 2024-05-12T22:17:26.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.669+0000 7fab05600640 1 -- 172.21.15.28:0/2881833000 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fab0003a430 con 0x5569504cfeb0 2024-05-12T22:17:26.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.669+0000 7fab21a13740 10 client.16986 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:17:26.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.669+0000 7fab21a13740 10 client.16986 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:17:26.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.669+0000 7fab21a13740 20 client.16986 populate_metadata read hostname 'smithi028' 2024-05-12T22:17:26.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.669+0000 7fab21a13740 1 --2- 172.21.15.28:0/2881833000 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x556950551630 0x556950553a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:26.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.669+0000 7fab21a13740 4 client.16986 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-12T22:17:26.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.669+0000 7fab21a13740 1 -- 172.21.15.28:0/2881833000 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x556950440ef0 con 0x556950551630 2024-05-12T22:17:26.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.670+0000 7fab1cc00640 1 --2- 172.21.15.28:0/2881833000 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x556950551630 0x556950553a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:26.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.670+0000 7fab1cc00640 1 --2- 172.21.15.28:0/2881833000 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x556950551630 0x556950553a50 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b330f4e3d97a3ae7 server_cookie=ae18b009753a7f04 in_seq=0 out_seq=0 2024-05-12T22:17:26.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.670+0000 7fab05600640 10 client.16986 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:17:26.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.671+0000 7fab05600640 1 -- 172.21.15.28:0/2881833000 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x556950440ef0 con 0x556950551630 2024-05-12T22:17:26.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.671+0000 7fab05600640 10 client.16986 handle_command_reply: tid=0 2024-05-12T22:17:26.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.671+0000 7fab21a13740 1 client.16986 shutdown 2024-05-12T22:17:26.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.671+0000 7fab21a13740 1 -- 172.21.15.28:0/2881833000 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5569504cfeb0 msgr2=0x5569504d4730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:17:26.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.671+0000 7fab21a13740 1 --2- 172.21.15.28:0/2881833000 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5569504cfeb0 0x5569504d4730 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fab000309d0 tx=0x7fab00005ef0 comp rx=0 tx=0).stop 2024-05-12T22:17:26.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.672+0000 7fab21a13740 1 -- 172.21.15.28:0/2881833000 shutdown_connections 2024-05-12T22:17:26.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.672+0000 7fab21a13740 1 --2- 172.21.15.28:0/2881833000 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x556950551630 0x556950553a50 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:26.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.672+0000 7fab21a13740 1 --2- 172.21.15.28:0/2881833000 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5569504da520 0x5569504d4c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:26.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.672+0000 7fab21a13740 1 --2- 172.21.15.28:0/2881833000 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5569504cfeb0 0x5569504d4730 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:26.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.672+0000 7fab21a13740 1 --2- 172.21.15.28:0/2881833000 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5569504cf500 0x5569504d41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:26.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.672+0000 7fab05600640 0 client.16986 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:17:26.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.672+0000 7fab21a13740 1 -- 172.21.15.28:0/2881833000 >> 172.21.15.28:0/2881833000 conn(0x5569504cb5b0 msgr2=0x5569504c9ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:17:26.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.672+0000 7fab21a13740 1 -- 172.21.15.28:0/2881833000 shutdown_connections 2024-05-12T22:17:26.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.672+0000 7fab21a13740 1 -- 172.21.15.28:0/2881833000 wait complete. 2024-05-12T22:17:26.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.672+0000 7fab21a13740 20 client.16986 trim_cache size 0 max 16384 2024-05-12T22:17:26.679 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-05-12T22:17:26.680 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-05-12T22:17:26.680 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "0acd9dfa-33df-4433-b28b-45889abc3e13", 2024-05-12T22:17:26.680 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-05-12T22:17:26.680 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-05-12T22:17:26.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.673+0000 7fab1d600640 1 -- 172.21.15.28:0/1702756166 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7faafc0a2d80 msgr2=0x7faafc0a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:17:26.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.673+0000 7fab1d600640 1 --2- 172.21.15.28:0/1702756166 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7faafc0a2d80 0x7faafc0a5240 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x5569504d0ac0 tx=0x7fab080d5530 comp rx=0 tx=0).stop 2024-05-12T22:17:26.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.673+0000 7fab1d600640 1 -- 172.21.15.28:0/1702756166 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fab18107dd0 msgr2=0x7fab1819e5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:17:26.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.673+0000 7fab1d600640 1 --2- 172.21.15.28:0/1702756166 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fab18107dd0 0x7fab1819e5f0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fab0800e9f0 tx=0x7fab0800eeb0 comp rx=0 tx=0).stop 2024-05-12T22:17:26.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.673+0000 7fab1d600640 1 -- 172.21.15.28:0/1702756166 shutdown_connections 2024-05-12T22:17:26.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.673+0000 7fab1d600640 1 --2- 172.21.15.28:0/1702756166 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7faafc0a2d80 0x7faafc0a5240 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:26.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.673+0000 7fab1d600640 1 --2- 172.21.15.28:0/1702756166 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab1810ddf0 0x7fab181a2e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:26.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.673+0000 7fab1d600640 1 --2- 172.21.15.28:0/1702756166 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fab18108770 0x7fab1819eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:26.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.673+0000 7fab1d600640 1 --2- 172.21.15.28:0/1702756166 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fab18107dd0 0x7fab1819e5f0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:26.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.673+0000 7fab1d600640 1 -- 172.21.15.28:0/1702756166 >> 172.21.15.28:0/1702756166 conn(0x7fab18077650 msgr2=0x7fab180786d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:17:26.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.673+0000 7fab1d600640 1 -- 172.21.15.28:0/1702756166 shutdown_connections 2024-05-12T22:17:26.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.673+0000 7fab1d600640 1 -- 172.21.15.28:0/1702756166 wait complete. 2024-05-12T22:17:26.694 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-12T22:17:26.695 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-12T22:17:26.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.939+0000 7f0024e00640 1 -- 172.21.15.28:0/3187498006 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f00201076e0 msgr2=0x7f0020107ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:17:26.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.939+0000 7f0024e00640 1 --2- 172.21.15.28:0/3187498006 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f00201076e0 0x7f0020107ac0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f0010009900 tx=0x7f001002a3f0 comp rx=0 tx=0).stop 2024-05-12T22:17:26.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.939+0000 7f0024e00640 1 -- 172.21.15.28:0/3187498006 shutdown_connections 2024-05-12T22:17:26.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.939+0000 7f0024e00640 1 --2- 172.21.15.28:0/3187498006 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0020102090 0x7f002010e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:26.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.939+0000 7f0024e00640 1 --2- 172.21.15.28:0/3187498006 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00201016d0 0x7f0020101b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:26.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.939+0000 7f0024e00640 1 --2- 172.21.15.28:0/3187498006 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f00201076e0 0x7f0020107ac0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:26.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.939+0000 7f0024e00640 1 -- 172.21.15.28:0/3187498006 >> 172.21.15.28:0/3187498006 conn(0x7f00200fd200 msgr2=0x7f00200ff620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:17:26.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.939+0000 7f0024e00640 1 -- 172.21.15.28:0/3187498006 shutdown_connections 2024-05-12T22:17:26.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.939+0000 7f0024e00640 1 -- 172.21.15.28:0/3187498006 wait complete. 2024-05-12T22:17:26.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.940+0000 7f0024e00640 1 Processor -- start 2024-05-12T22:17:26.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.941+0000 7f0024e00640 1 -- start start 2024-05-12T22:17:26.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.941+0000 7f0024e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f00201016d0 0x7f0020073b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:26.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.941+0000 7f0024e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0020102090 0x7f0020074070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:26.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.941+0000 7f0024e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00201076e0 0x7f00200745b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:26.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.941+0000 7f0024e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0020111b40 con 0x7f00201076e0 2024-05-12T22:17:26.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.941+0000 7f0024e00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f00201119c0 con 0x7f0020102090 2024-05-12T22:17:26.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.941+0000 7f0024e00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f0020111cc0 con 0x7f00201016d0 2024-05-12T22:17:26.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.941+0000 7f001f400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f00201016d0 0x7f0020073b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:26.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.941+0000 7f001f400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f00201016d0 0x7f0020073b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:38352/0 (socket says 172.21.15.28:38352) 2024-05-12T22:17:26.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.941+0000 7f001f400640 1 -- 172.21.15.28:0/1615749708 learned_addr learned my addr 172.21.15.28:0/1615749708 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:17:26.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.941+0000 7f001ea00640 1 --2- 172.21.15.28:0/1615749708 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0020102090 0x7f0020074070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:26.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.942+0000 7f001fe00640 1 --2- 172.21.15.28:0/1615749708 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00201076e0 0x7f00200745b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:26.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.942+0000 7f001f400640 1 -- 172.21.15.28:0/1615749708 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0020102090 msgr2=0x7f0020074070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:17:26.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.942+0000 7f001f400640 1 --2- 172.21.15.28:0/1615749708 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0020102090 0x7f0020074070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:26.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.942+0000 7f001f400640 1 -- 172.21.15.28:0/1615749708 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00201076e0 msgr2=0x7f00200745b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:17:26.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.942+0000 7f001f400640 1 --2- 172.21.15.28:0/1615749708 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00201076e0 0x7f00200745b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:26.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.942+0000 7f001f400640 1 -- 172.21.15.28:0/1615749708 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00201a6480 con 0x7f00201016d0 2024-05-12T22:17:26.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.942+0000 7f001ea00640 1 --2- 172.21.15.28:0/1615749708 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0020102090 0x7f0020074070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:17:26.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.942+0000 7f001fe00640 1 --2- 172.21.15.28:0/1615749708 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00201076e0 0x7f00200745b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:17:26.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.943+0000 7f001f400640 1 --2- 172.21.15.28:0/1615749708 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f00201016d0 0x7f0020073b30 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f001002a9e0 tx=0x7f0010003c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:17:26.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.943+0000 7f0017e00640 1 -- 172.21.15.28:0/1615749708 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f001003f070 con 0x7f00201016d0 2024-05-12T22:17:26.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.943+0000 7f0024e00640 1 -- 172.21.15.28:0/1615749708 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00201a6710 con 0x7f00201016d0 2024-05-12T22:17:26.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.943+0000 7f0017e00640 1 -- 172.21.15.28:0/1615749708 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0010031c30 con 0x7f00201016d0 2024-05-12T22:17:26.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.943+0000 7f0017e00640 1 -- 172.21.15.28:0/1615749708 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0010035030 con 0x7f00201016d0 2024-05-12T22:17:26.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.943+0000 7f0024e00640 1 -- 172.21.15.28:0/1615749708 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00201a6c50 con 0x7f00201016d0 2024-05-12T22:17:26.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.947+0000 7f00291c0740 1 Processor -- start 2024-05-12T22:17:26.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.947+0000 7f0017e00640 1 -- 172.21.15.28:0/1615749708 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7f0010042070 con 0x7f00201016d0 2024-05-12T22:17:26.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.948+0000 7f00291c0740 1 -- start start 2024-05-12T22:17:26.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.948+0000 7f00291c0740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5652710f9dd0 0x5652710fa1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:26.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.948+0000 7f00291c0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5652710fa6f0 0x565271104880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:26.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.948+0000 7f00291c0740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x565271104dc0 0x5652711071b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:26.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.948+0000 7f00291c0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5652710c22a0 con 0x5652710fa6f0 2024-05-12T22:17:26.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.948+0000 7f00291c0740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x56527105eb90 con 0x565271104dc0 2024-05-12T22:17:26.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.948+0000 7f00291c0740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5652710065b0 con 0x5652710f9dd0 2024-05-12T22:17:26.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.949+0000 7f001ea00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x565271104dc0 0x5652711071b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:26.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.949+0000 7f001ea00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x565271104dc0 0x5652711071b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:51770/0 (socket says 172.21.15.28:51770) 2024-05-12T22:17:26.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.949+0000 7f001ea00640 1 -- 172.21.15.28:0/3205609321 learned_addr learned my addr 172.21.15.28:0/3205609321 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:17:26.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.949+0000 7f001f400640 1 --2- 172.21.15.28:0/3205609321 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5652710fa6f0 0x565271104880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:26.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.949+0000 7f001fe00640 1 --2- 172.21.15.28:0/3205609321 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5652710f9dd0 0x5652710fa1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:26.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.949+0000 7f001ea00640 1 -- 172.21.15.28:0/3205609321 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5652710f9dd0 msgr2=0x5652710fa1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:17:26.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.949+0000 7f001ea00640 1 --2- 172.21.15.28:0/3205609321 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5652710f9dd0 0x5652710fa1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:26.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.949+0000 7f001ea00640 1 -- 172.21.15.28:0/3205609321 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5652710fa6f0 msgr2=0x565271104880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:17:26.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.949+0000 7f001ea00640 1 --2- 172.21.15.28:0/3205609321 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5652710fa6f0 0x565271104880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:26.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.949+0000 7f001ea00640 1 -- 172.21.15.28:0/3205609321 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565270e3c7d0 con 0x565271104dc0 2024-05-12T22:17:26.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.949+0000 7f001fe00640 1 --2- 172.21.15.28:0/3205609321 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5652710f9dd0 0x5652710fa1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:17:26.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.950+0000 7f0017e00640 1 --2- 172.21.15.28:0/1615749708 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f00040a2d80 0x7f00040a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:26.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.950+0000 7f001ea00640 1 --2- 172.21.15.28:0/3205609321 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x565271104dc0 0x5652711071b0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f000c00bfd0 tx=0x7f000c0108e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=489550d8e389bac7 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:17:26.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.950+0000 7f0017e00640 1 -- 172.21.15.28:0/1615749708 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8513+0+0 (secure 0 0 0) 0x7f00100efb60 con 0x7f00201016d0 2024-05-12T22:17:26.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.950+0000 7f0014c00640 1 -- 172.21.15.28:0/3205609321 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f000c005220 con 0x565271104dc0 2024-05-12T22:17:26.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.950+0000 7f0014c00640 1 -- 172.21.15.28:0/3205609321 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f000c004c80 con 0x565271104dc0 2024-05-12T22:17:26.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.950+0000 7f0014c00640 1 -- 172.21.15.28:0/3205609321 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f000c011bd0 con 0x565271104dc0 2024-05-12T22:17:26.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.950+0000 7f001fe00640 1 --2- 172.21.15.28:0/1615749708 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f00040a2d80 0x7f00040a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:26.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.951+0000 7f00291c0740 1 -- 172.21.15.28:0/3205609321 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x565271104dc0 msgr2=0x5652711071b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:17:26.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.951+0000 7f00291c0740 1 --2- 172.21.15.28:0/3205609321 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x565271104dc0 0x5652711071b0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f000c00bfd0 tx=0x7f000c0108e0 comp rx=0 tx=0).stop 2024-05-12T22:17:26.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.951+0000 7f001fe00640 1 --2- 172.21.15.28:0/1615749708 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f00040a2d80 0x7f00040a5240 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f0018005fd0 tx=0x7f0018009500 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:17:26.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.951+0000 7f00291c0740 1 -- 172.21.15.28:0/3205609321 shutdown_connections 2024-05-12T22:17:26.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.951+0000 7f00291c0740 1 --2- 172.21.15.28:0/3205609321 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x565271104dc0 0x5652711071b0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:26.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.951+0000 7f00291c0740 1 --2- 172.21.15.28:0/3205609321 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5652710fa6f0 0x565271104880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:26.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.951+0000 7f00291c0740 1 --2- 172.21.15.28:0/3205609321 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5652710f9dd0 0x5652710fa1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:26.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.951+0000 7f00291c0740 1 -- 172.21.15.28:0/3205609321 >> 172.21.15.28:0/3205609321 conn(0x5652710f60f0 msgr2=0x5652710f64d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:17:26.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.951+0000 7f00291c0740 1 -- 172.21.15.28:0/3205609321 shutdown_connections 2024-05-12T22:17:26.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.952+0000 7f00291c0740 1 -- 172.21.15.28:0/3205609321 wait complete. 2024-05-12T22:17:26.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.952+0000 7f00291c0740 1 -- start start 2024-05-12T22:17:26.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.952+0000 7f00291c0740 1 Processor -- start 2024-05-12T22:17:26.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.953+0000 7f00291c0740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5652710f9dd0 0x565271101c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:26.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.953+0000 7f00291c0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5652710fa6f0 0x5652711021b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:26.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.953+0000 7f00291c0740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5652711026f0 0x565271110ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:26.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.953+0000 7f00291c0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5652710199f0 con 0x5652710fa6f0 2024-05-12T22:17:26.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.953+0000 7f00291c0740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x56527105ab50 con 0x5652710f9dd0 2024-05-12T22:17:26.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.953+0000 7f00291c0740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x565271107bb0 con 0x5652711026f0 2024-05-12T22:17:26.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.954+0000 7f00291c0740 10 client.0 _ll_register_callbacks cb 0x565270d71be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:17:26.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.954+0000 7f001f400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5652710f9dd0 0x565271101c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:26.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.954+0000 7f001fe00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5652711026f0 0x565271110ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:26.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.954+0000 7f001f400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5652710f9dd0 0x565271101c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:51774/0 (socket says 172.21.15.28:51774) 2024-05-12T22:17:26.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.954+0000 7f001f400640 1 -- 172.21.15.28:0/1231727615 learned_addr learned my addr 172.21.15.28:0/1231727615 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:17:26.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.954+0000 7f001ea00640 1 --2- 172.21.15.28:0/1231727615 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5652710fa6f0 0x5652711021b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:26.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.954+0000 7f001fe00640 1 -- 172.21.15.28:0/1231727615 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5652710f9dd0 msgr2=0x565271101c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:17:26.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.954+0000 7f001fe00640 1 --2- 172.21.15.28:0/1231727615 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5652710f9dd0 0x565271101c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:26.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.954+0000 7f001fe00640 1 -- 172.21.15.28:0/1231727615 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5652710fa6f0 msgr2=0x5652711021b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:17:26.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.954+0000 7f001fe00640 1 --2- 172.21.15.28:0/1231727615 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5652710fa6f0 0x5652711021b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:26.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.954+0000 7f001fe00640 1 -- 172.21.15.28:0/1231727615 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565270d723f0 con 0x5652711026f0 2024-05-12T22:17:26.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.954+0000 7f001f400640 1 --2- 172.21.15.28:0/1231727615 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5652710f9dd0 0x565271101c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:17:26.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.955+0000 7f001ea00640 1 --2- 172.21.15.28:0/1231727615 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5652710fa6f0 0x5652711021b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:17:26.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.955+0000 7f001fe00640 1 --2- 172.21.15.28:0/1231727615 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5652711026f0 0x565271110ea0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f0018010b70 tx=0x7f0018004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=73d303be75eb4f9e server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:17:26.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.955+0000 7f0009600640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-12T22:17:26.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.955+0000 7f0009600640 1 -- 172.21.15.28:0/1231727615 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f001801ea30 con 0x5652711026f0 2024-05-12T22:17:26.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.955+0000 7f0009600640 1 -- 172.21.15.28:0/1231727615 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8513+0+0 (secure 0 0 0) 0x7f001801ebd0 con 0x5652711026f0 2024-05-12T22:17:26.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.955+0000 7f00291c0740 1 -- 172.21.15.28:0/1231727615 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f000c004c80 con 0x5652711026f0 2024-05-12T22:17:26.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.955+0000 7f0009600640 1 -- 172.21.15.28:0/1231727615 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f001801edd0 con 0x5652711026f0 2024-05-12T22:17:26.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.956+0000 7f00291c0740 10 client.27085 fetch_fsmap learned FSMap version 26 2024-05-12T22:17:26.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.956+0000 7f00291c0740 1 -- 172.21.15.28:0/1231727615 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x565270d320f0 con 0x5652711026f0 2024-05-12T22:17:26.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.956+0000 7f0009600640 1 -- 172.21.15.28:0/1231727615 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f000c004c80 con 0x5652711026f0 2024-05-12T22:17:26.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.956+0000 7f0009600640 1 -- 172.21.15.28:0/1231727615 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f001801c5b0 con 0x5652711026f0 2024-05-12T22:17:26.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.956+0000 7f00291c0740 10 client.27085 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:17:26.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.956+0000 7f00291c0740 10 client.27085 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:17:26.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.956+0000 7f00291c0740 20 client.27085 populate_metadata read hostname 'smithi028' 2024-05-12T22:17:26.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.956+0000 7f00291c0740 1 --2- 172.21.15.28:0/1231727615 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x565271115020 0x565271117470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:26.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.956+0000 7f00291c0740 4 client.27085 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-12T22:17:26.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.956+0000 7f00291c0740 1 -- 172.21.15.28:0/1231727615 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56527106c2c0 con 0x565271115020 2024-05-12T22:17:26.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.957+0000 7f001f400640 1 --2- 172.21.15.28:0/1231727615 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x565271115020 0x565271117470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:26.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.957+0000 7f001f400640 1 --2- 172.21.15.28:0/1231727615 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x565271115020 0x565271117470 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=780e7a421cc0e078 server_cookie=566923ddc7c05768 in_seq=0 out_seq=0 2024-05-12T22:17:26.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.957+0000 7f0009600640 10 client.27085 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:17:26.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.958+0000 7f0009600640 1 -- 172.21.15.28:0/1231727615 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56527106c2c0 con 0x565271115020 2024-05-12T22:17:26.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.958+0000 7f0009600640 10 client.27085 handle_command_reply: tid=0 2024-05-12T22:17:26.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.958+0000 7f00291c0740 1 client.27085 shutdown 2024-05-12T22:17:26.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.958+0000 7f00291c0740 1 -- 172.21.15.28:0/1231727615 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5652711026f0 msgr2=0x565271110ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:17:26.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.958+0000 7f00291c0740 1 --2- 172.21.15.28:0/1231727615 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5652711026f0 0x565271110ea0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f0018010b70 tx=0x7f0018004a40 comp rx=0 tx=0).stop 2024-05-12T22:17:26.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.958+0000 7f00291c0740 1 -- 172.21.15.28:0/1231727615 shutdown_connections 2024-05-12T22:17:26.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.958+0000 7f00291c0740 1 --2- 172.21.15.28:0/1231727615 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x565271115020 0x565271117470 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:26.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.958+0000 7f00291c0740 1 --2- 172.21.15.28:0/1231727615 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5652711026f0 0x565271110ea0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:26.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.958+0000 7f0009600640 0 client.27085 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:17:26.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.958+0000 7f00291c0740 1 --2- 172.21.15.28:0/1231727615 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5652710fa6f0 0x5652711021b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:26.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.958+0000 7f00291c0740 1 --2- 172.21.15.28:0/1231727615 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5652710f9dd0 0x565271101c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:26.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.958+0000 7f00291c0740 1 -- 172.21.15.28:0/1231727615 >> 172.21.15.28:0/1231727615 conn(0x5652710f60f0 msgr2=0x5652710fbef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:17:26.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.959+0000 7f00291c0740 1 -- 172.21.15.28:0/1231727615 shutdown_connections 2024-05-12T22:17:26.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.959+0000 7f00291c0740 1 -- 172.21.15.28:0/1231727615 wait complete. 2024-05-12T22:17:26.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.959+0000 7f00291c0740 20 client.27085 trim_cache size 0 max 16384 2024-05-12T22:17:26.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.986+0000 7f00291c0740 1 Processor -- start 2024-05-12T22:17:26.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.986+0000 7f00291c0740 1 -- start start 2024-05-12T22:17:26.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.986+0000 7f00291c0740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5652710f9dd0 0x565271104440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:26.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.987+0000 7f00291c0740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5652710fa6f0 0x5652710f5ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:26.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.987+0000 7f00291c0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5652711026f0 0x5652710feef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:26.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.987+0000 7f00291c0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f00201b1df0 con 0x5652711026f0 2024-05-12T22:17:26.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.987+0000 7f00291c0740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f00201af6c0 con 0x5652710fa6f0 2024-05-12T22:17:26.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.987+0000 7f00291c0740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f00201ad530 con 0x5652710f9dd0 2024-05-12T22:17:26.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.987+0000 7f001f400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5652710f9dd0 0x565271104440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:26.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.987+0000 7f001f400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5652710f9dd0 0x565271104440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:38394/0 (socket says 172.21.15.28:38394) 2024-05-12T22:17:26.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.987+0000 7f001f400640 1 -- 172.21.15.28:0/1495338845 learned_addr learned my addr 172.21.15.28:0/1495338845 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:17:26.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.987+0000 7f001ea00640 1 --2- 172.21.15.28:0/1495338845 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5652710fa6f0 0x5652710f5ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:26.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.987+0000 7f001fe00640 1 --2- 172.21.15.28:0/1495338845 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5652711026f0 0x5652710feef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:26.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.987+0000 7f001f400640 1 -- 172.21.15.28:0/1495338845 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5652710fa6f0 msgr2=0x5652710f5ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:17:26.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.987+0000 7f001f400640 1 --2- 172.21.15.28:0/1495338845 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5652710fa6f0 0x5652710f5ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:26.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.987+0000 7f001f400640 1 -- 172.21.15.28:0/1495338845 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5652711026f0 msgr2=0x5652710feef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:17:26.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.987+0000 7f001f400640 1 --2- 172.21.15.28:0/1495338845 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5652711026f0 0x5652710feef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:26.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.987+0000 7f001f400640 1 -- 172.21.15.28:0/1495338845 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5652710fa490 con 0x5652710f9dd0 2024-05-12T22:17:26.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.987+0000 7f001ea00640 1 --2- 172.21.15.28:0/1495338845 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5652710fa6f0 0x5652710f5ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:17:26.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.987+0000 7f001fe00640 1 --2- 172.21.15.28:0/1495338845 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5652711026f0 0x5652710feef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:17:26.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.988+0000 7f001f400640 1 --2- 172.21.15.28:0/1495338845 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5652710f9dd0 0x565271104440 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f00100b8480 tx=0x7f00100f3610 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d552731e5ac14d9f server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:17:26.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.988+0000 7f000be00640 1 -- 172.21.15.28:0/1495338845 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0010049040 con 0x5652710f9dd0 2024-05-12T22:17:26.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.988+0000 7f000be00640 1 -- 172.21.15.28:0/1495338845 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0010031de0 con 0x5652710f9dd0 2024-05-12T22:17:26.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.988+0000 7f000be00640 1 -- 172.21.15.28:0/1495338845 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00100c4c90 con 0x5652710f9dd0 2024-05-12T22:17:26.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.989+0000 7f00291c0740 1 -- 172.21.15.28:0/1495338845 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5652710f9dd0 msgr2=0x565271104440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:17:26.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.989+0000 7f00291c0740 1 --2- 172.21.15.28:0/1495338845 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5652710f9dd0 0x565271104440 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f00100b8480 tx=0x7f00100f3610 comp rx=0 tx=0).stop 2024-05-12T22:17:26.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.989+0000 7f00291c0740 1 -- 172.21.15.28:0/1495338845 shutdown_connections 2024-05-12T22:17:26.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.989+0000 7f00291c0740 1 --2- 172.21.15.28:0/1495338845 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5652711026f0 0x5652710feef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:26.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.989+0000 7f00291c0740 1 --2- 172.21.15.28:0/1495338845 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5652710fa6f0 0x5652710f5ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:26.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.989+0000 7f00291c0740 1 --2- 172.21.15.28:0/1495338845 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5652710f9dd0 0x565271104440 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:26.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.989+0000 7f00291c0740 1 -- 172.21.15.28:0/1495338845 >> 172.21.15.28:0/1495338845 conn(0x5652710f60f0 msgr2=0x565271103d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:17:26.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.989+0000 7f00291c0740 1 -- 172.21.15.28:0/1495338845 shutdown_connections 2024-05-12T22:17:26.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.989+0000 7f00291c0740 1 -- 172.21.15.28:0/1495338845 wait complete. 2024-05-12T22:17:26.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.990+0000 7f00291c0740 1 -- start start 2024-05-12T22:17:26.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.990+0000 7f00291c0740 1 Processor -- start 2024-05-12T22:17:26.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.991+0000 7f00291c0740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5652710f9dd0 0x56527110ca60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:26.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.991+0000 7f00291c0740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5652710fa6f0 0x56527110cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:26.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.991+0000 7f00291c0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5652711026f0 0x565271109f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:26.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.991+0000 7f00291c0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f00201adb80 con 0x5652711026f0 2024-05-12T22:17:26.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.991+0000 7f00291c0740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f00201aeff0 con 0x5652710f9dd0 2024-05-12T22:17:26.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.991+0000 7f00291c0740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f00201ae250 con 0x5652710fa6f0 2024-05-12T22:17:26.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.991+0000 7f00291c0740 10 client.0 _ll_register_callbacks cb 0x7f0020075bc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:17:26.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.991+0000 7f001f400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5652710f9dd0 0x56527110ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:26.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.991+0000 7f001ea00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5652710fa6f0 0x56527110cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:26.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.991+0000 7f001ea00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5652710fa6f0 0x56527110cfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:38402/0 (socket says 172.21.15.28:38402) 2024-05-12T22:17:26.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.991+0000 7f001ea00640 1 -- 172.21.15.28:0/929632826 learned_addr learned my addr 172.21.15.28:0/929632826 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:17:26.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.991+0000 7f001f400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5652710f9dd0 0x56527110ca60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:51782/0 (socket says 172.21.15.28:51782) 2024-05-12T22:17:26.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.991+0000 7f001fe00640 1 --2- 172.21.15.28:0/929632826 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5652711026f0 0x565271109f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:26.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.991+0000 7f001ea00640 1 -- 172.21.15.28:0/929632826 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5652710f9dd0 msgr2=0x56527110ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:17:26.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.991+0000 7f001ea00640 1 --2- 172.21.15.28:0/929632826 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5652710f9dd0 0x56527110ca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:26.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.991+0000 7f001ea00640 1 -- 172.21.15.28:0/929632826 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5652711026f0 msgr2=0x565271109f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:17:26.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.992+0000 7f001ea00640 1 --2- 172.21.15.28:0/929632826 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5652711026f0 0x565271109f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:26.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.992+0000 7f001ea00640 1 -- 172.21.15.28:0/929632826 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56527110a680 con 0x5652710fa6f0 2024-05-12T22:17:26.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.992+0000 7f001fe00640 1 --2- 172.21.15.28:0/929632826 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5652711026f0 0x565271109f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:17:26.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.992+0000 7f001ea00640 1 --2- 172.21.15.28:0/929632826 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5652710fa6f0 0x56527110cfa0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f000c000c00 tx=0x7f000c005270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3a52032bd3552dd7 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:17:26.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.992+0000 7f0009600640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-12T22:17:26.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.992+0000 7f0009600640 1 -- 172.21.15.28:0/929632826 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f000c02b060 con 0x5652710fa6f0 2024-05-12T22:17:26.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.992+0000 7f0009600640 1 -- 172.21.15.28:0/929632826 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8513+0+0 (secure 0 0 0) 0x7f000c004660 con 0x5652710fa6f0 2024-05-12T22:17:26.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.992+0000 7f00291c0740 1 -- 172.21.15.28:0/929632826 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0010031de0 con 0x5652710fa6f0 2024-05-12T22:17:26.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.993+0000 7f0009600640 1 -- 172.21.15.28:0/929632826 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f000c004860 con 0x5652710fa6f0 2024-05-12T22:17:26.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.993+0000 7f00291c0740 10 client.27091 fetch_fsmap learned FSMap version 26 2024-05-12T22:17:26.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.993+0000 7f00291c0740 1 -- 172.21.15.28:0/929632826 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x56527117af40 con 0x5652710fa6f0 2024-05-12T22:17:26.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.993+0000 7f0009600640 1 -- 172.21.15.28:0/929632826 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0010031de0 con 0x5652710fa6f0 2024-05-12T22:17:26.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.993+0000 7f0009600640 1 -- 172.21.15.28:0/929632826 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f000c026440 con 0x5652710fa6f0 2024-05-12T22:17:27.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.993+0000 7f00291c0740 10 client.27091 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:17:27.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.993+0000 7f00291c0740 10 client.27091 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:17:27.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.993+0000 7f00291c0740 20 client.27091 populate_metadata read hostname 'smithi028' 2024-05-12T22:17:27.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.993+0000 7f00291c0740 1 --2- 172.21.15.28:0/929632826 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x56527117b330 0x56527117d720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:27.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.993+0000 7f00291c0740 4 client.27091 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-12T22:17:27.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.993+0000 7f00291c0740 1 -- 172.21.15.28:0/929632826 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "scrub status"}) -- 0x565271066610 con 0x56527117b330 2024-05-12T22:17:27.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.994+0000 7f001f400640 1 --2- 172.21.15.28:0/929632826 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x56527117b330 0x56527117d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:27.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.994+0000 7f001f400640 1 --2- 172.21.15.28:0/929632826 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x56527117b330 0x56527117d720 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e200a46028505db5 server_cookie=95f2c2ec91edc337 in_seq=0 out_seq=0 2024-05-12T22:17:27.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.994+0000 7f0009600640 10 client.27091 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:17:27.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.995+0000 7f0009600640 1 -- 172.21.15.28:0/929632826 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+268 (crc 0 0 0) 0x565271066610 con 0x56527117b330 2024-05-12T22:17:27.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.995+0000 7f0009600640 10 client.27091 handle_command_reply: tid=0 2024-05-12T22:17:27.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.995+0000 7f00291c0740 1 client.27091 shutdown 2024-05-12T22:17:27.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.995+0000 7f00291c0740 1 -- 172.21.15.28:0/929632826 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5652710fa6f0 msgr2=0x56527110cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:17:27.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.995+0000 7f00291c0740 1 --2- 172.21.15.28:0/929632826 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5652710fa6f0 0x56527110cfa0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f000c000c00 tx=0x7f000c005270 comp rx=0 tx=0).stop 2024-05-12T22:17:27.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.995+0000 7f00291c0740 1 -- 172.21.15.28:0/929632826 shutdown_connections 2024-05-12T22:17:27.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.995+0000 7f00291c0740 1 --2- 172.21.15.28:0/929632826 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x56527117b330 0x56527117d720 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:27.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.995+0000 7f00291c0740 1 --2- 172.21.15.28:0/929632826 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5652711026f0 0x565271109f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:27.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.995+0000 7f0009600640 0 client.27091 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:17:27.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.995+0000 7f00291c0740 1 --2- 172.21.15.28:0/929632826 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5652710fa6f0 0x56527110cfa0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:27.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.995+0000 7f00291c0740 1 --2- 172.21.15.28:0/929632826 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5652710f9dd0 0x56527110ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:27.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.995+0000 7f00291c0740 1 -- 172.21.15.28:0/929632826 >> 172.21.15.28:0/929632826 conn(0x5652710f60f0 msgr2=0x5652710f4aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:17:27.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.995+0000 7f00291c0740 1 -- 172.21.15.28:0/929632826 shutdown_connections 2024-05-12T22:17:27.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.995+0000 7f00291c0740 1 -- 172.21.15.28:0/929632826 wait complete. 2024-05-12T22:17:27.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.996+0000 7f00291c0740 20 client.27091 trim_cache size 0 max 16384 2024-05-12T22:17:27.004 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-05-12T22:17:27.004 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "scrub active (1174 inodes in the stack)", 2024-05-12T22:17:27.004 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-05-12T22:17:27.004 INFO:teuthology.orchestra.run.smithi028.stdout: "0acd9dfa-33df-4433-b28b-45889abc3e13": { 2024-05-12T22:17:27.004 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-05-12T22:17:27.004 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "0acd9dfa-33df-4433-b28b-45889abc3e13", 2024-05-12T22:17:27.005 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-05-12T22:17:27.005 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:17:27.005 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:17:27.005 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-05-12T22:17:27.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.996+0000 7f0024e00640 1 -- 172.21.15.28:0/1615749708 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f00040a2d80 msgr2=0x7f00040a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:17:27.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.996+0000 7f0024e00640 1 --2- 172.21.15.28:0/1615749708 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f00040a2d80 0x7f00040a5240 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f0018005fd0 tx=0x7f0018009500 comp rx=0 tx=0).stop 2024-05-12T22:17:27.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.996+0000 7f0024e00640 1 -- 172.21.15.28:0/1615749708 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f00201016d0 msgr2=0x7f0020073b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:17:27.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.996+0000 7f0024e00640 1 --2- 172.21.15.28:0/1615749708 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f00201016d0 0x7f0020073b30 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f001002a9e0 tx=0x7f0010003c10 comp rx=0 tx=0).stop 2024-05-12T22:17:27.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.997+0000 7f0024e00640 1 -- 172.21.15.28:0/1615749708 shutdown_connections 2024-05-12T22:17:27.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.997+0000 7f0024e00640 1 --2- 172.21.15.28:0/1615749708 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f00040a2d80 0x7f00040a5240 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:27.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.997+0000 7f0024e00640 1 --2- 172.21.15.28:0/1615749708 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00201076e0 0x7f00200745b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:27.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.997+0000 7f0024e00640 1 --2- 172.21.15.28:0/1615749708 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0020102090 0x7f0020074070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:27.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.997+0000 7f0024e00640 1 --2- 172.21.15.28:0/1615749708 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f00201016d0 0x7f0020073b30 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:27.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.997+0000 7f0024e00640 1 -- 172.21.15.28:0/1615749708 >> 172.21.15.28:0/1615749708 conn(0x7f00200fd200 msgr2=0x7f00200fd7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:17:27.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.997+0000 7f0024e00640 1 -- 172.21.15.28:0/1615749708 shutdown_connections 2024-05-12T22:17:27.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:26.997+0000 7f0024e00640 1 -- 172.21.15.28:0/1615749708 wait complete. 2024-05-12T22:17:27.018 INFO:tasks.cephfs.filesystem:scrub status for tag:0acd9dfa-33df-4433-b28b-45889abc3e13 - {'path': '/', 'tag': '0acd9dfa-33df-4433-b28b-45889abc3e13', 'options': 'recursive,force'} 2024-05-12T22:17:27.034 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:17:26 smithi028 ceph-mon[29847]: scrub summary: active paths [/] 2024-05-12T22:17:27.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:17:26 smithi028 ceph-mon[29847]: scrub summary: idle+waiting paths [/] 2024-05-12T22:17:27.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:17:26 smithi028 ceph-mon[29847]: scrub summary: active paths [/] 2024-05-12T22:17:27.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:17:26 smithi028 ceph-mon[29847]: scrub summary: idle+waiting paths [/] 2024-05-12T22:17:27.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:17:26 smithi086 ceph-mon[34297]: scrub summary: active paths [/] 2024-05-12T22:17:27.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:17:26 smithi086 ceph-mon[34297]: scrub summary: idle+waiting paths [/] 2024-05-12T22:17:27.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:17:26 smithi086 ceph-mon[34297]: scrub summary: active paths [/] 2024-05-12T22:17:27.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:17:26 smithi086 ceph-mon[34297]: scrub summary: idle+waiting paths [/] 2024-05-12T22:17:27.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:17:26 smithi186 ceph-mon[34785]: scrub summary: active paths [/] 2024-05-12T22:17:27.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:17:26 smithi186 ceph-mon[34785]: scrub summary: idle+waiting paths [/] 2024-05-12T22:17:27.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:17:26 smithi186 ceph-mon[34785]: scrub summary: active paths [/] 2024-05-12T22:17:27.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:17:26 smithi186 ceph-mon[34785]: scrub summary: idle+waiting paths [/] 2024-05-12T22:17:28.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:17:27 smithi086 ceph-mon[34297]: scrub queued for path: / 2024-05-12T22:17:28.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:17:27 smithi086 ceph-mon[34297]: scrub summary: idle+waiting paths [/] 2024-05-12T22:17:28.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:17:27 smithi086 ceph-mon[34297]: scrub summary: active paths [/] 2024-05-12T22:17:28.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:17:27 smithi086 ceph-mon[34297]: pgmap v261: 97 pgs: 97 active+clean; 91 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:17:28.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:17:27 smithi086 ceph-mon[34297]: scrub summary: active paths [/] 2024-05-12T22:17:28.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:17:27 smithi086 ceph-mon[34297]: scrub summary: idle+waiting paths [/] 2024-05-12T22:17:28.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:17:27 smithi086 ceph-mon[34297]: scrub summary: idle+waiting paths [/] 2024-05-12T22:17:28.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:17:27 smithi028 ceph-mon[29847]: scrub queued for path: / 2024-05-12T22:17:28.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:17:27 smithi028 ceph-mon[29847]: scrub summary: idle+waiting paths [/] 2024-05-12T22:17:28.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:17:27 smithi028 ceph-mon[29847]: scrub summary: active paths [/] 2024-05-12T22:17:28.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:17:27 smithi028 ceph-mon[29847]: pgmap v261: 97 pgs: 97 active+clean; 91 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:17:28.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:17:27 smithi028 ceph-mon[29847]: scrub summary: active paths [/] 2024-05-12T22:17:28.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:17:27 smithi028 ceph-mon[29847]: scrub summary: idle+waiting paths [/] 2024-05-12T22:17:28.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:17:27 smithi028 ceph-mon[29847]: scrub summary: idle+waiting paths [/] 2024-05-12T22:17:28.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:17:27 smithi186 ceph-mon[34785]: scrub queued for path: / 2024-05-12T22:17:28.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:17:27 smithi186 ceph-mon[34785]: scrub summary: idle+waiting paths [/] 2024-05-12T22:17:28.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:17:27 smithi186 ceph-mon[34785]: scrub summary: active paths [/] 2024-05-12T22:17:28.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:17:27 smithi186 ceph-mon[34785]: pgmap v261: 97 pgs: 97 active+clean; 91 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:17:28.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:17:27 smithi186 ceph-mon[34785]: scrub summary: active paths [/] 2024-05-12T22:17:28.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:17:27 smithi186 ceph-mon[34785]: scrub summary: idle+waiting paths [/] 2024-05-12T22:17:28.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:17:27 smithi186 ceph-mon[34785]: scrub summary: idle+waiting paths [/] 2024-05-12T22:17:30.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:17:29 smithi086 ceph-mon[34297]: pgmap v262: 97 pgs: 97 active+clean; 91 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:17:30.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:17:29 smithi028 ceph-mon[29847]: pgmap v262: 97 pgs: 97 active+clean; 91 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:17:30.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:17:29 smithi186 ceph-mon[34785]: pgmap v262: 97 pgs: 97 active+clean; 91 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:17:32.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:17:31 smithi028 ceph-mon[29847]: pgmap v263: 97 pgs: 97 active+clean; 91 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-05-12T22:17:32.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:17:31 smithi186 ceph-mon[34785]: pgmap v263: 97 pgs: 97 active+clean; 91 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-05-12T22:17:32.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:17:31 smithi086 ceph-mon[34297]: pgmap v263: 97 pgs: 97 active+clean; 91 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-05-12T22:17:33.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:17:33 smithi028 ceph-mon[29847]: scrub summary: idle 2024-05-12T22:17:33.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:17:33 smithi028 ceph-mon[29847]: scrub summary: idle 2024-05-12T22:17:33.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:17:33 smithi186 ceph-mon[34785]: scrub summary: idle 2024-05-12T22:17:33.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:17:33 smithi186 ceph-mon[34785]: scrub summary: idle 2024-05-12T22:17:33.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:17:33 smithi086 ceph-mon[34297]: scrub summary: idle 2024-05-12T22:17:33.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:17:33 smithi086 ceph-mon[34297]: scrub summary: idle 2024-05-12T22:17:34.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:17:34 smithi028 ceph-mon[29847]: pgmap v264: 97 pgs: 97 active+clean; 91 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-05-12T22:17:34.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:17:34 smithi186 ceph-mon[34785]: pgmap v264: 97 pgs: 97 active+clean; 91 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-05-12T22:17:34.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:17:34 smithi086 ceph-mon[34297]: pgmap v264: 97 pgs: 97 active+clean; 91 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-05-12T22:17:36.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:17:36 smithi028 ceph-mon[29847]: pgmap v265: 97 pgs: 97 active+clean; 91 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-05-12T22:17:36.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:17:36 smithi186 ceph-mon[34785]: pgmap v265: 97 pgs: 97 active+clean; 91 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-05-12T22:17:36.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:17:36 smithi086 ceph-mon[34297]: pgmap v265: 97 pgs: 97 active+clean; 91 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-05-12T22:17:38.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:17:38 smithi028 ceph-mon[29847]: pgmap v266: 97 pgs: 97 active+clean; 91 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-05-12T22:17:38.294 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:17:38 smithi086 ceph-mon[34297]: pgmap v266: 97 pgs: 97 active+clean; 91 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-05-12T22:17:38.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:17:38 smithi186 ceph-mon[34785]: pgmap v266: 97 pgs: 97 active+clean; 91 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-05-12T22:17:40.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:17:40 smithi028 ceph-mon[29847]: pgmap v267: 97 pgs: 97 active+clean; 91 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-05-12T22:17:40.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:17:40 smithi186 ceph-mon[34785]: pgmap v267: 97 pgs: 97 active+clean; 91 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-05-12T22:17:40.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:17:40 smithi086 ceph-mon[34297]: pgmap v267: 97 pgs: 97 active+clean; 91 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-05-12T22:17:42.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:17:42 smithi028 ceph-mon[29847]: pgmap v268: 97 pgs: 97 active+clean; 91 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-05-12T22:17:42.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:17:42 smithi186 ceph-mon[34785]: pgmap v268: 97 pgs: 97 active+clean; 91 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-05-12T22:17:42.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:17:42 smithi086 ceph-mon[34297]: pgmap v268: 97 pgs: 97 active+clean; 91 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-05-12T22:17:44.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:17:44 smithi186 ceph-mon[34785]: pgmap v269: 97 pgs: 97 active+clean; 91 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:17:44.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:17:44 smithi086 ceph-mon[34297]: pgmap v269: 97 pgs: 97 active+clean; 91 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:17:44.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:17:44 smithi028 ceph-mon[29847]: pgmap v269: 97 pgs: 97 active+clean; 91 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:17:46.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:17:46 smithi186 ceph-mon[34785]: pgmap v270: 97 pgs: 97 active+clean; 91 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-12T22:17:46.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:17:46 smithi086 ceph-mon[34297]: pgmap v270: 97 pgs: 97 active+clean; 91 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-12T22:17:46.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:17:46 smithi028 ceph-mon[29847]: pgmap v270: 97 pgs: 97 active+clean; 91 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-12T22:17:48.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:17:48 smithi186 ceph-mon[34785]: pgmap v271: 97 pgs: 97 active+clean; 91 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:17:48.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:17:48 smithi086 ceph-mon[34297]: pgmap v271: 97 pgs: 97 active+clean; 91 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:17:48.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:17:48 smithi028 ceph-mon[29847]: pgmap v271: 97 pgs: 97 active+clean; 91 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:17:50.338 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:17:50 smithi028 ceph-mon[29847]: pgmap v272: 97 pgs: 97 active+clean; 91 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:17:50.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:17:50 smithi186 ceph-mon[34785]: pgmap v272: 97 pgs: 97 active+clean; 91 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:17:50.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:17:50 smithi086 ceph-mon[34297]: pgmap v272: 97 pgs: 97 active+clean; 91 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:17:52.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:17:52 smithi186 ceph-mon[34785]: pgmap v273: 97 pgs: 97 active+clean; 91 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-12T22:17:52.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:17:52 smithi086 ceph-mon[34297]: pgmap v273: 97 pgs: 97 active+clean; 91 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-12T22:17:52.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:17:52 smithi028 ceph-mon[29847]: pgmap v273: 97 pgs: 97 active+clean; 91 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-12T22:17:54.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:17:54 smithi186 ceph-mon[34785]: pgmap v274: 97 pgs: 97 active+clean; 91 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:17:54.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:17:54 smithi086 ceph-mon[34297]: pgmap v274: 97 pgs: 97 active+clean; 91 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:17:54.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:17:54 smithi028 ceph-mon[29847]: pgmap v274: 97 pgs: 97 active+clean; 91 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:17:56.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:17:56 smithi186 ceph-mon[34785]: pgmap v275: 97 pgs: 97 active+clean; 91 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-12T22:17:56.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:17:56 smithi086 ceph-mon[34297]: pgmap v275: 97 pgs: 97 active+clean; 91 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-12T22:17:56.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:17:56 smithi028 ceph-mon[29847]: pgmap v275: 97 pgs: 97 active+clean; 91 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-12T22:17:57.019 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-12T22:17:57.020 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-12T22:17:57.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.248+0000 7f898be00640 1 -- 172.21.15.28:0/562177293 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f898c102090 msgr2=0x7f898c10e5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:17:57.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.248+0000 7f898be00640 1 --2- 172.21.15.28:0/562177293 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f898c102090 0x7f898c10e5a0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f8984009be0 tx=0x7f898402a6c0 comp rx=0 tx=0).stop 2024-05-12T22:17:57.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.248+0000 7f898be00640 1 -- 172.21.15.28:0/562177293 shutdown_connections 2024-05-12T22:17:57.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.248+0000 7f898be00640 1 --2- 172.21.15.28:0/562177293 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f898c102090 0x7f898c10e5a0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:57.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.248+0000 7f898be00640 1 --2- 172.21.15.28:0/562177293 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f898c1016d0 0x7f898c101b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:57.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.248+0000 7f898be00640 1 --2- 172.21.15.28:0/562177293 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f898c1076e0 0x7f898c107ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:57.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.248+0000 7f898be00640 1 -- 172.21.15.28:0/562177293 >> 172.21.15.28:0/562177293 conn(0x7f898c0fd200 msgr2=0x7f898c0ff620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:17:57.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.248+0000 7f898be00640 1 -- 172.21.15.28:0/562177293 shutdown_connections 2024-05-12T22:17:57.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.248+0000 7f898be00640 1 -- 172.21.15.28:0/562177293 wait complete. 2024-05-12T22:17:57.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.249+0000 7f898be00640 1 Processor -- start 2024-05-12T22:17:57.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.249+0000 7f898be00640 1 -- start start 2024-05-12T22:17:57.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.250+0000 7f898be00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f898c1016d0 0x7f898c073b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:57.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.250+0000 7f898be00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f898c102090 0x7f898c074070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:57.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.250+0000 7f898be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f898c1076e0 0x7f898c0745b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:57.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.250+0000 7f898be00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f898c111af0 con 0x7f898c1076e0 2024-05-12T22:17:57.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.250+0000 7f898be00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f898c111970 con 0x7f898c102090 2024-05-12T22:17:57.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.250+0000 7f898be00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f898c111c70 con 0x7f898c1016d0 2024-05-12T22:17:57.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.250+0000 7f898b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f898c1076e0 0x7f898c0745b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:57.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.250+0000 7f8983e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f898c102090 0x7f898c074070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:57.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.250+0000 7f898aa00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f898c1016d0 0x7f898c073b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:57.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.250+0000 7f8983e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f898c102090 0x7f898c074070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:33292/0 (socket says 172.21.15.28:33292) 2024-05-12T22:17:57.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.250+0000 7f8983e00640 1 -- 172.21.15.28:0/3833531931 learned_addr learned my addr 172.21.15.28:0/3833531931 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:17:57.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.251+0000 7f8983e00640 1 -- 172.21.15.28:0/3833531931 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f898c1016d0 msgr2=0x7f898c073b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:17:57.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.251+0000 7f8983e00640 1 --2- 172.21.15.28:0/3833531931 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f898c1016d0 0x7f898c073b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:57.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.251+0000 7f8983e00640 1 -- 172.21.15.28:0/3833531931 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f898c1076e0 msgr2=0x7f898c0745b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:17:57.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.251+0000 7f8983e00640 1 --2- 172.21.15.28:0/3833531931 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f898c1076e0 0x7f898c0745b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:57.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.251+0000 7f8983e00640 1 -- 172.21.15.28:0/3833531931 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f898c1a6430 con 0x7f898c102090 2024-05-12T22:17:57.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.251+0000 7f8983e00640 1 --2- 172.21.15.28:0/3833531931 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f898c102090 0x7f898c074070 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f897400e9f0 tx=0x7f897400edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:17:57.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.251+0000 7f8983400640 1 -- 172.21.15.28:0/3833531931 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f897400ccc0 con 0x7f898c102090 2024-05-12T22:17:57.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.251+0000 7f8983400640 1 -- 172.21.15.28:0/3833531931 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f897400ce60 con 0x7f898c102090 2024-05-12T22:17:57.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.251+0000 7f898be00640 1 -- 172.21.15.28:0/3833531931 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f898c1a6720 con 0x7f898c102090 2024-05-12T22:17:57.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.251+0000 7f8983400640 1 -- 172.21.15.28:0/3833531931 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8974010690 con 0x7f898c102090 2024-05-12T22:17:57.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.251+0000 7f898be00640 1 -- 172.21.15.28:0/3833531931 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f898c1a6c70 con 0x7f898c102090 2024-05-12T22:17:57.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.256+0000 7f899482d740 1 Processor -- start 2024-05-12T22:17:57.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.256+0000 7f899482d740 1 -- start start 2024-05-12T22:17:57.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.256+0000 7f899482d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561bae358dd0 0x561bae3591b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:57.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.256+0000 7f899482d740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561bae3596f0 0x561bae363880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:57.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.256+0000 7f899482d740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561bae363dc0 0x561bae3661b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:57.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.256+0000 7f899482d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x561bae3212a0 con 0x561bae358dd0 2024-05-12T22:17:57.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.256+0000 7f899482d740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x561bae2bdb90 con 0x561bae363dc0 2024-05-12T22:17:57.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.256+0000 7f899482d740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x561bae2655b0 con 0x561bae3596f0 2024-05-12T22:17:57.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.257+0000 7f898b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561bae358dd0 0x561bae3591b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:57.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.257+0000 7f8983400640 1 -- 172.21.15.28:0/3833531931 <== mon.2 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7f89740046c0 con 0x7f898c102090 2024-05-12T22:17:57.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.257+0000 7f898b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561bae358dd0 0x561bae3591b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39968/0 (socket says 172.21.15.28:39968) 2024-05-12T22:17:57.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.257+0000 7f898b400640 1 -- 172.21.15.28:0/660647426 learned_addr learned my addr 172.21.15.28:0/660647426 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:17:57.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.257+0000 7f898aa00640 1 --2- 172.21.15.28:0/660647426 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561bae3596f0 0x561bae363880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:57.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.257+0000 7f8983e00640 1 --2- 172.21.15.28:0/660647426 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561bae363dc0 0x561bae3661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:57.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.257+0000 7f898b400640 1 -- 172.21.15.28:0/660647426 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561bae3596f0 msgr2=0x561bae363880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:17:57.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.257+0000 7f898b400640 1 --2- 172.21.15.28:0/660647426 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561bae3596f0 0x561bae363880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:57.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.257+0000 7f898b400640 1 -- 172.21.15.28:0/660647426 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561bae363dc0 msgr2=0x561bae3661b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:17:57.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.257+0000 7f898b400640 1 --2- 172.21.15.28:0/660647426 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561bae363dc0 0x561bae3661b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:57.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.257+0000 7f898b400640 1 -- 172.21.15.28:0/660647426 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561bae09b7d0 con 0x561bae358dd0 2024-05-12T22:17:57.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.258+0000 7f898b400640 1 --2- 172.21.15.28:0/660647426 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561bae358dd0 0x561bae3591b0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f8984009be0 tx=0x7f8984030430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3811a1d2bae87cce server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:17:57.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.258+0000 7f897be00640 1 -- 172.21.15.28:0/660647426 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8984041070 con 0x561bae358dd0 2024-05-12T22:17:57.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.258+0000 7f897be00640 1 -- 172.21.15.28:0/660647426 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89840307a0 con 0x561bae358dd0 2024-05-12T22:17:57.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.258+0000 7f897be00640 1 -- 172.21.15.28:0/660647426 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8984036040 con 0x561bae358dd0 2024-05-12T22:17:57.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.259+0000 7f8983400640 1 --2- 172.21.15.28:0/3833531931 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f89700a2d80 0x7f89700a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:57.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.259+0000 7f899482d740 1 -- 172.21.15.28:0/660647426 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561bae358dd0 msgr2=0x561bae3591b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:17:57.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.259+0000 7f8983400640 1 -- 172.21.15.28:0/3833531931 <== mon.2 v2:172.21.15.86:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8513+0+0 (secure 0 0 0) 0x7f89740cf810 con 0x7f898c102090 2024-05-12T22:17:57.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.259+0000 7f899482d740 1 --2- 172.21.15.28:0/660647426 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561bae358dd0 0x561bae3591b0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f8984009be0 tx=0x7f8984030430 comp rx=0 tx=0).stop 2024-05-12T22:17:57.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.259+0000 7f898aa00640 1 --2- 172.21.15.28:0/3833531931 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f89700a2d80 0x7f89700a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:57.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.259+0000 7f899482d740 1 -- 172.21.15.28:0/660647426 shutdown_connections 2024-05-12T22:17:57.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.259+0000 7f899482d740 1 --2- 172.21.15.28:0/660647426 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561bae363dc0 0x561bae3661b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:57.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.259+0000 7f899482d740 1 --2- 172.21.15.28:0/660647426 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561bae3596f0 0x561bae363880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:57.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.259+0000 7f899482d740 1 --2- 172.21.15.28:0/660647426 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561bae358dd0 0x561bae3591b0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:57.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.259+0000 7f899482d740 1 -- 172.21.15.28:0/660647426 >> 172.21.15.28:0/660647426 conn(0x561bae3550f0 msgr2=0x561bae3554d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:17:57.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.259+0000 7f899482d740 1 -- 172.21.15.28:0/660647426 shutdown_connections 2024-05-12T22:17:57.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.259+0000 7f899482d740 1 -- 172.21.15.28:0/660647426 wait complete. 2024-05-12T22:17:57.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.260+0000 7f899482d740 1 -- start start 2024-05-12T22:17:57.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.260+0000 7f898aa00640 1 --2- 172.21.15.28:0/3833531931 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f89700a2d80 0x7f89700a5240 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x561bae35a390 tx=0x7f897c009260 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:17:57.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.260+0000 7f899482d740 1 Processor -- start 2024-05-12T22:17:57.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.260+0000 7f899482d740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561bae358dd0 0x561bae3636f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:57.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.260+0000 7f899482d740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561bae3596f0 0x561bae3609e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:57.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.261+0000 7f899482d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561bae363dc0 0x561bae360f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:57.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.261+0000 7f899482d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x561bae2789f0 con 0x561bae363dc0 2024-05-12T22:17:57.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.261+0000 7f899482d740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x561bae2b9b50 con 0x561bae358dd0 2024-05-12T22:17:57.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.261+0000 7f899482d740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x561bae366b50 con 0x561bae3596f0 2024-05-12T22:17:57.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.261+0000 7f899482d740 10 client.0 _ll_register_callbacks cb 0x561badfd0be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:17:57.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.261+0000 7f898b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561bae363dc0 0x561bae360f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:57.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.261+0000 7f898aa00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561bae358dd0 0x561bae3636f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:57.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.261+0000 7f898b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561bae363dc0 0x561bae360f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39980/0 (socket says 172.21.15.28:39980) 2024-05-12T22:17:57.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.261+0000 7f898b400640 1 -- 172.21.15.28:0/3535654529 learned_addr learned my addr 172.21.15.28:0/3535654529 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:17:57.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.261+0000 7f8983e00640 1 --2- 172.21.15.28:0/3535654529 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561bae3596f0 0x561bae3609e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:57.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.261+0000 7f898b400640 1 -- 172.21.15.28:0/3535654529 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561bae3596f0 msgr2=0x561bae3609e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:17:57.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.261+0000 7f898b400640 1 --2- 172.21.15.28:0/3535654529 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561bae3596f0 0x561bae3609e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:57.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.261+0000 7f898b400640 1 -- 172.21.15.28:0/3535654529 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561bae358dd0 msgr2=0x561bae3636f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:17:57.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.261+0000 7f898b400640 1 --2- 172.21.15.28:0/3535654529 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561bae358dd0 0x561bae3636f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:57.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.261+0000 7f898b400640 1 -- 172.21.15.28:0/3535654529 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561badfd13f0 con 0x561bae363dc0 2024-05-12T22:17:57.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.261+0000 7f898aa00640 1 --2- 172.21.15.28:0/3535654529 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561bae358dd0 0x561bae3636f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:17:57.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.262+0000 7f898b400640 1 --2- 172.21.15.28:0/3535654529 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561bae363dc0 0x561bae360f20 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f8984009b30 tx=0x7f8984009280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=900d39ac685850ab server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:17:57.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.262+0000 7f8978c00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-12T22:17:57.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.262+0000 7f8978c00640 1 -- 172.21.15.28:0/3535654529 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8984036030 con 0x561bae363dc0 2024-05-12T22:17:57.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.262+0000 7f8978c00640 1 -- 172.21.15.28:0/3535654529 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8513+0+0 (secure 0 0 0) 0x7f8984030a20 con 0x561bae363dc0 2024-05-12T22:17:57.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.262+0000 7f899482d740 1 -- 172.21.15.28:0/3535654529 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f89840307a0 con 0x561bae363dc0 2024-05-12T22:17:57.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.262+0000 7f8978c00640 1 -- 172.21.15.28:0/3535654529 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f8984041040 con 0x561bae363dc0 2024-05-12T22:17:57.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.262+0000 7f899482d740 10 client.17016 fetch_fsmap learned FSMap version 26 2024-05-12T22:17:57.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.263+0000 7f899482d740 1 -- 172.21.15.28:0/3535654529 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x561badf910f0 con 0x561bae363dc0 2024-05-12T22:17:57.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.263+0000 7f8978c00640 1 -- 172.21.15.28:0/3535654529 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89840307a0 con 0x561bae363dc0 2024-05-12T22:17:57.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.263+0000 7f8978c00640 1 -- 172.21.15.28:0/3535654529 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f8984057480 con 0x561bae363dc0 2024-05-12T22:17:57.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.263+0000 7f899482d740 10 client.17016 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:17:57.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.263+0000 7f899482d740 10 client.17016 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:17:57.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.263+0000 7f899482d740 20 client.17016 populate_metadata read hostname 'smithi028' 2024-05-12T22:17:57.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.263+0000 7f899482d740 1 --2- 172.21.15.28:0/3535654529 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x561bae373740 0x561bae375b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:57.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.263+0000 7f899482d740 4 client.17016 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-12T22:17:57.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.263+0000 7f899482d740 1 -- 172.21.15.28:0/3535654529 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561bae2cb2c0 con 0x561bae373740 2024-05-12T22:17:57.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.264+0000 7f898aa00640 1 --2- 172.21.15.28:0/3535654529 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x561bae373740 0x561bae375b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:57.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.264+0000 7f898aa00640 1 --2- 172.21.15.28:0/3535654529 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x561bae373740 0x561bae375b30 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b5f1f48268123d0b server_cookie=c09dbb2de463b9b7 in_seq=0 out_seq=0 2024-05-12T22:17:57.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.264+0000 7f8978c00640 10 client.17016 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:17:57.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.265+0000 7f8978c00640 1 -- 172.21.15.28:0/3535654529 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561bae2cb2c0 con 0x561bae373740 2024-05-12T22:17:57.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.265+0000 7f8978c00640 10 client.17016 handle_command_reply: tid=0 2024-05-12T22:17:57.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.265+0000 7f899482d740 1 client.17016 shutdown 2024-05-12T22:17:57.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.265+0000 7f899482d740 1 -- 172.21.15.28:0/3535654529 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561bae363dc0 msgr2=0x561bae360f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:17:57.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.265+0000 7f899482d740 1 --2- 172.21.15.28:0/3535654529 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561bae363dc0 0x561bae360f20 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f8984009b30 tx=0x7f8984009280 comp rx=0 tx=0).stop 2024-05-12T22:17:57.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.266+0000 7f899482d740 1 -- 172.21.15.28:0/3535654529 shutdown_connections 2024-05-12T22:17:57.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.266+0000 7f899482d740 1 --2- 172.21.15.28:0/3535654529 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x561bae373740 0x561bae375b30 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:57.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.266+0000 7f899482d740 1 --2- 172.21.15.28:0/3535654529 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561bae363dc0 0x561bae360f20 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:57.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.266+0000 7f899482d740 1 --2- 172.21.15.28:0/3535654529 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561bae3596f0 0x561bae3609e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:57.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.266+0000 7f899482d740 1 --2- 172.21.15.28:0/3535654529 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561bae358dd0 0x561bae3636f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:57.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.266+0000 7f8978c00640 0 client.17016 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:17:57.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.266+0000 7f899482d740 1 -- 172.21.15.28:0/3535654529 >> 172.21.15.28:0/3535654529 conn(0x561bae3550f0 msgr2=0x561bae35adf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:17:57.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.266+0000 7f899482d740 1 -- 172.21.15.28:0/3535654529 shutdown_connections 2024-05-12T22:17:57.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.266+0000 7f899482d740 1 -- 172.21.15.28:0/3535654529 wait complete. 2024-05-12T22:17:57.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.266+0000 7f899482d740 20 client.17016 trim_cache size 0 max 16384 2024-05-12T22:17:57.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.285+0000 7f899482d740 1 Processor -- start 2024-05-12T22:17:57.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.285+0000 7f899482d740 1 -- start start 2024-05-12T22:17:57.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.286+0000 7f899482d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561bae358dd0 0x561bae360b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:57.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.286+0000 7f899482d740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561bae3596f0 0x561bae363220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:57.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.286+0000 7f899482d740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561bae363dc0 0x561bae378b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:57.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.286+0000 7f899482d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f898c1b1dc0 con 0x561bae358dd0 2024-05-12T22:17:57.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.286+0000 7f899482d740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f898c1af690 con 0x561bae3596f0 2024-05-12T22:17:57.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.286+0000 7f899482d740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f898c1ad500 con 0x561bae363dc0 2024-05-12T22:17:57.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.286+0000 7f898aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561bae358dd0 0x561bae360b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:57.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.286+0000 7f8983e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561bae3596f0 0x561bae363220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:57.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.286+0000 7f898aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561bae358dd0 0x561bae360b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39992/0 (socket says 172.21.15.28:39992) 2024-05-12T22:17:57.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.286+0000 7f898aa00640 1 -- 172.21.15.28:0/3291989714 learned_addr learned my addr 172.21.15.28:0/3291989714 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:17:57.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.286+0000 7f898b400640 1 --2- 172.21.15.28:0/3291989714 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561bae363dc0 0x561bae378b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:57.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.287+0000 7f8983e00640 1 -- 172.21.15.28:0/3291989714 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561bae363dc0 msgr2=0x561bae378b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:17:57.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.287+0000 7f8983e00640 1 --2- 172.21.15.28:0/3291989714 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561bae363dc0 0x561bae378b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:57.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.287+0000 7f8983e00640 1 -- 172.21.15.28:0/3291989714 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561bae358dd0 msgr2=0x561bae360b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:17:57.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.287+0000 7f8983e00640 1 --2- 172.21.15.28:0/3291989714 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561bae358dd0 0x561bae360b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:57.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.287+0000 7f8983e00640 1 -- 172.21.15.28:0/3291989714 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561bae361500 con 0x561bae3596f0 2024-05-12T22:17:57.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.287+0000 7f8983e00640 1 --2- 172.21.15.28:0/3291989714 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561bae3596f0 0x561bae363220 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f89740ccf00 tx=0x7f897409b810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1104ae9a50b66bd server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:17:57.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.287+0000 7f897b400640 1 -- 172.21.15.28:0/3291989714 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f897409bcd0 con 0x561bae3596f0 2024-05-12T22:17:57.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.287+0000 7f897b400640 1 -- 172.21.15.28:0/3291989714 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89740a0080 con 0x561bae3596f0 2024-05-12T22:17:57.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.287+0000 7f897b400640 1 -- 172.21.15.28:0/3291989714 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89740a44b0 con 0x561bae3596f0 2024-05-12T22:17:57.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.288+0000 7f899482d740 1 -- 172.21.15.28:0/3291989714 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561bae3596f0 msgr2=0x561bae363220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:17:57.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.288+0000 7f899482d740 1 --2- 172.21.15.28:0/3291989714 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561bae3596f0 0x561bae363220 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f89740ccf00 tx=0x7f897409b810 comp rx=0 tx=0).stop 2024-05-12T22:17:57.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.288+0000 7f899482d740 1 -- 172.21.15.28:0/3291989714 shutdown_connections 2024-05-12T22:17:57.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.288+0000 7f899482d740 1 --2- 172.21.15.28:0/3291989714 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561bae363dc0 0x561bae378b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:57.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.288+0000 7f899482d740 1 --2- 172.21.15.28:0/3291989714 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561bae3596f0 0x561bae363220 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:57.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.288+0000 7f899482d740 1 --2- 172.21.15.28:0/3291989714 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561bae358dd0 0x561bae360b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:57.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.288+0000 7f899482d740 1 -- 172.21.15.28:0/3291989714 >> 172.21.15.28:0/3291989714 conn(0x561bae3550f0 msgr2=0x561bae36edd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:17:57.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.288+0000 7f899482d740 1 -- 172.21.15.28:0/3291989714 shutdown_connections 2024-05-12T22:17:57.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.288+0000 7f899482d740 1 -- 172.21.15.28:0/3291989714 wait complete. 2024-05-12T22:17:57.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.289+0000 7f899482d740 1 -- start start 2024-05-12T22:17:57.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.289+0000 7f899482d740 1 Processor -- start 2024-05-12T22:17:57.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.289+0000 7f899482d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561bae358dd0 0x561bae3d60c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:57.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.290+0000 7f899482d740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561bae3596f0 0x561bae3d6600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:57.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.290+0000 7f899482d740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561bae363dc0 0x561bae3d6b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:57.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.290+0000 7f899482d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f898c1adb50 con 0x561bae358dd0 2024-05-12T22:17:57.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.290+0000 7f899482d740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f898c1aefc0 con 0x561bae3596f0 2024-05-12T22:17:57.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.290+0000 7f899482d740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f898c1ae220 con 0x561bae363dc0 2024-05-12T22:17:57.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.290+0000 7f899482d740 10 client.0 _ll_register_callbacks cb 0x7f897c0097e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:17:57.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.290+0000 7f898aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561bae358dd0 0x561bae3d60c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:57.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.290+0000 7f898aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561bae358dd0 0x561bae3d60c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40006/0 (socket says 172.21.15.28:40006) 2024-05-12T22:17:57.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.290+0000 7f898aa00640 1 -- 172.21.15.28:0/1999802864 learned_addr learned my addr 172.21.15.28:0/1999802864 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:17:57.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.290+0000 7f8983e00640 1 --2- 172.21.15.28:0/1999802864 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561bae3596f0 0x561bae3d6600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:57.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.290+0000 7f898b400640 1 --2- 172.21.15.28:0/1999802864 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561bae363dc0 0x561bae3d6b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:57.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.290+0000 7f898aa00640 1 -- 172.21.15.28:0/1999802864 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561bae363dc0 msgr2=0x561bae3d6b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:17:57.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.290+0000 7f898aa00640 1 --2- 172.21.15.28:0/1999802864 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561bae363dc0 0x561bae3d6b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:57.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.290+0000 7f898aa00640 1 -- 172.21.15.28:0/1999802864 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561bae3596f0 msgr2=0x561bae3d6600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:17:57.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.290+0000 7f898aa00640 1 --2- 172.21.15.28:0/1999802864 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561bae3596f0 0x561bae3d6600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:57.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.290+0000 7f898aa00640 1 -- 172.21.15.28:0/1999802864 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561bae3e1390 con 0x561bae358dd0 2024-05-12T22:17:57.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.290+0000 7f8983e00640 1 --2- 172.21.15.28:0/1999802864 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561bae3596f0 0x561bae3d6600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:17:57.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.291+0000 7f898aa00640 1 --2- 172.21.15.28:0/1999802864 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561bae358dd0 0x561bae3d60c0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f897c005c20 tx=0x7f897c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d07b6d7c6cb7c1d4 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:17:57.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.291+0000 7f8978c00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-12T22:17:57.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.291+0000 7f8978c00640 1 -- 172.21.15.28:0/1999802864 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f897c003d30 con 0x561bae358dd0 2024-05-12T22:17:57.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.291+0000 7f8978c00640 1 -- 172.21.15.28:0/1999802864 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8513+0+0 (secure 0 0 0) 0x7f897c02b380 con 0x561bae358dd0 2024-05-12T22:17:57.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.291+0000 7f899482d740 1 -- 172.21.15.28:0/1999802864 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f89740a0080 con 0x561bae358dd0 2024-05-12T22:17:57.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.291+0000 7f8978c00640 1 -- 172.21.15.28:0/1999802864 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f897c02b580 con 0x561bae358dd0 2024-05-12T22:17:57.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.291+0000 7f899482d740 10 client.17022 fetch_fsmap learned FSMap version 26 2024-05-12T22:17:57.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.291+0000 7f899482d740 1 -- 172.21.15.28:0/1999802864 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x561bae3e2700 con 0x561bae358dd0 2024-05-12T22:17:57.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.292+0000 7f8978c00640 1 -- 172.21.15.28:0/1999802864 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89740a0080 con 0x561bae358dd0 2024-05-12T22:17:57.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.292+0000 7f8978c00640 1 -- 172.21.15.28:0/1999802864 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f897c004960 con 0x561bae358dd0 2024-05-12T22:17:57.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.292+0000 7f899482d740 10 client.17022 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:17:57.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.292+0000 7f899482d740 10 client.17022 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:17:57.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.292+0000 7f899482d740 20 client.17022 populate_metadata read hostname 'smithi028' 2024-05-12T22:17:57.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.292+0000 7f899482d740 1 --2- 172.21.15.28:0/1999802864 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x561bae3e2e00 0x561bae3e5220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:57.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.292+0000 7f899482d740 4 client.17022 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-12T22:17:57.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.292+0000 7f899482d740 1 -- 172.21.15.28:0/1999802864 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "scrub status"}) -- 0x561bae2c5610 con 0x561bae3e2e00 2024-05-12T22:17:57.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.292+0000 7f8983e00640 1 --2- 172.21.15.28:0/1999802864 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x561bae3e2e00 0x561bae3e5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:57.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.293+0000 7f8983e00640 1 --2- 172.21.15.28:0/1999802864 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x561bae3e2e00 0x561bae3e5220 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ce3d69c18ae1fa79 server_cookie=521dcdcefcefc5aa in_seq=0 out_seq=0 2024-05-12T22:17:57.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.293+0000 7f8978c00640 10 client.17022 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:17:57.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.293+0000 7f8978c00640 1 -- 172.21.15.28:0/1999802864 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x561bae2c5610 con 0x561bae3e2e00 2024-05-12T22:17:57.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.293+0000 7f8978c00640 10 client.17022 handle_command_reply: tid=0 2024-05-12T22:17:57.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.293+0000 7f899482d740 1 client.17022 shutdown 2024-05-12T22:17:57.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.293+0000 7f899482d740 1 -- 172.21.15.28:0/1999802864 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561bae358dd0 msgr2=0x561bae3d60c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:17:57.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.293+0000 7f899482d740 1 --2- 172.21.15.28:0/1999802864 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561bae358dd0 0x561bae3d60c0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f897c005c20 tx=0x7f897c0046c0 comp rx=0 tx=0).stop 2024-05-12T22:17:57.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.293+0000 7f899482d740 1 -- 172.21.15.28:0/1999802864 shutdown_connections 2024-05-12T22:17:57.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.293+0000 7f899482d740 1 --2- 172.21.15.28:0/1999802864 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x561bae3e2e00 0x561bae3e5220 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:57.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.293+0000 7f899482d740 1 --2- 172.21.15.28:0/1999802864 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561bae363dc0 0x561bae3d6b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:57.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.293+0000 7f899482d740 1 --2- 172.21.15.28:0/1999802864 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561bae3596f0 0x561bae3d6600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:57.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.293+0000 7f899482d740 1 --2- 172.21.15.28:0/1999802864 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561bae358dd0 0x561bae3d60c0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:57.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.293+0000 7f899482d740 1 -- 172.21.15.28:0/1999802864 >> 172.21.15.28:0/1999802864 conn(0x561bae3550f0 msgr2=0x561bae360950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:17:57.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.293+0000 7f8978c00640 0 client.17022 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:17:57.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.294+0000 7f899482d740 1 -- 172.21.15.28:0/1999802864 shutdown_connections 2024-05-12T22:17:57.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.294+0000 7f899482d740 1 -- 172.21.15.28:0/1999802864 wait complete. 2024-05-12T22:17:57.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.294+0000 7f899482d740 20 client.17022 trim_cache size 0 max 16384 2024-05-12T22:17:57.300 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-05-12T22:17:57.301 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-05-12T22:17:57.301 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-05-12T22:17:57.301 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-05-12T22:17:57.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.294+0000 7f898be00640 1 -- 172.21.15.28:0/3833531931 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f89700a2d80 msgr2=0x7f89700a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:17:57.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.294+0000 7f898be00640 1 --2- 172.21.15.28:0/3833531931 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f89700a2d80 0x7f89700a5240 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x561bae35a390 tx=0x7f897c009260 comp rx=0 tx=0).stop 2024-05-12T22:17:57.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.294+0000 7f898be00640 1 -- 172.21.15.28:0/3833531931 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f898c102090 msgr2=0x7f898c074070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:17:57.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.294+0000 7f898be00640 1 --2- 172.21.15.28:0/3833531931 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f898c102090 0x7f898c074070 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f897400e9f0 tx=0x7f897400edf0 comp rx=0 tx=0).stop 2024-05-12T22:17:57.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.294+0000 7f898be00640 1 -- 172.21.15.28:0/3833531931 shutdown_connections 2024-05-12T22:17:57.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.294+0000 7f898be00640 1 --2- 172.21.15.28:0/3833531931 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f89700a2d80 0x7f89700a5240 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:57.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.294+0000 7f898be00640 1 --2- 172.21.15.28:0/3833531931 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f898c1076e0 0x7f898c0745b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:57.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.294+0000 7f898be00640 1 --2- 172.21.15.28:0/3833531931 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f898c102090 0x7f898c074070 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:57.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.294+0000 7f898be00640 1 --2- 172.21.15.28:0/3833531931 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f898c1016d0 0x7f898c073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:57.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.294+0000 7f898be00640 1 -- 172.21.15.28:0/3833531931 >> 172.21.15.28:0/3833531931 conn(0x7f898c0fd200 msgr2=0x7f898c0fd780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:17:57.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.295+0000 7f898be00640 1 -- 172.21.15.28:0/3833531931 shutdown_connections 2024-05-12T22:17:57.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.295+0000 7f898be00640 1 -- 172.21.15.28:0/3833531931 wait complete. 2024-05-12T22:17:57.312 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-12T22:17:57.312 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-12T22:17:57.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.520+0000 7f61ea200640 1 -- 172.21.15.28:0/4014983011 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f61e4103a10 msgr2=0x7f61e4103e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:17:57.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.520+0000 7f61ea200640 1 --2- 172.21.15.28:0/4014983011 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f61e4103a10 0x7f61e4103e90 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f61d0009890 tx=0x7f61d002a360 comp rx=0 tx=0).stop 2024-05-12T22:17:57.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.520+0000 7f61ea200640 1 -- 172.21.15.28:0/4014983011 shutdown_connections 2024-05-12T22:17:57.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.520+0000 7f61ea200640 1 --2- 172.21.15.28:0/4014983011 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f61e41043d0 0x7f61e41108e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:57.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.520+0000 7f61ea200640 1 --2- 172.21.15.28:0/4014983011 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f61e4103a10 0x7f61e4103e90 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:57.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.520+0000 7f61ea200640 1 --2- 172.21.15.28:0/4014983011 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61e4109a20 0x7f61e4109e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:57.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.520+0000 7f61ea200640 1 -- 172.21.15.28:0/4014983011 >> 172.21.15.28:0/4014983011 conn(0x7f61e40ff4e0 msgr2=0x7f61e4101900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:17:57.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.520+0000 7f61ea200640 1 -- 172.21.15.28:0/4014983011 shutdown_connections 2024-05-12T22:17:57.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.520+0000 7f61ea200640 1 -- 172.21.15.28:0/4014983011 wait complete. 2024-05-12T22:17:57.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.521+0000 7f61ea200640 1 Processor -- start 2024-05-12T22:17:57.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.521+0000 7f61ea200640 1 -- start start 2024-05-12T22:17:57.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.522+0000 7f61ea200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f61e4103a10 0x7f61e419a330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:57.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.522+0000 7f61ea200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f61e41043d0 0x7f61e419a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:57.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.522+0000 7f61ea200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61e4109a20 0x7f61e419ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:57.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.522+0000 7f61ea200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f61e4113e30 con 0x7f61e4109a20 2024-05-12T22:17:57.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.522+0000 7f61ea200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f61e4113cb0 con 0x7f61e41043d0 2024-05-12T22:17:57.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.522+0000 7f61ea200640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f61e4113fb0 con 0x7f61e4103a10 2024-05-12T22:17:57.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.522+0000 7f61e2a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f61e41043d0 0x7f61e419a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:57.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.522+0000 7f61e3400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f61e4103a10 0x7f61e419a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:57.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.522+0000 7f61e3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61e4109a20 0x7f61e419ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:57.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.522+0000 7f61e2a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f61e41043d0 0x7f61e419a870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:33336/0 (socket says 172.21.15.28:33336) 2024-05-12T22:17:57.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.522+0000 7f61e2a00640 1 -- 172.21.15.28:0/789036336 learned_addr learned my addr 172.21.15.28:0/789036336 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:17:57.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.522+0000 7f61e2a00640 1 -- 172.21.15.28:0/789036336 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f61e4103a10 msgr2=0x7f61e419a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:17:57.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.522+0000 7f61e2a00640 1 --2- 172.21.15.28:0/789036336 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f61e4103a10 0x7f61e419a330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:57.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.522+0000 7f61e2a00640 1 -- 172.21.15.28:0/789036336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61e4109a20 msgr2=0x7f61e419ebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:17:57.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.523+0000 7f61e2a00640 1 --2- 172.21.15.28:0/789036336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61e4109a20 0x7f61e419ebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:57.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.523+0000 7f61e2a00640 1 -- 172.21.15.28:0/789036336 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61e419f320 con 0x7f61e41043d0 2024-05-12T22:17:57.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.523+0000 7f61e2a00640 1 --2- 172.21.15.28:0/789036336 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f61e41043d0 0x7f61e419a870 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f61d0035040 tx=0x7f61d0033b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:17:57.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.523+0000 7f61dbe00640 1 -- 172.21.15.28:0/789036336 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61d0037040 con 0x7f61e41043d0 2024-05-12T22:17:57.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.523+0000 7f61dbe00640 1 -- 172.21.15.28:0/789036336 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f61d0040070 con 0x7f61e41043d0 2024-05-12T22:17:57.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.523+0000 7f61dbe00640 1 -- 172.21.15.28:0/789036336 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61d0003970 con 0x7f61e41043d0 2024-05-12T22:17:57.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.523+0000 7f61ea200640 1 -- 172.21.15.28:0/789036336 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61e419f5b0 con 0x7f61e41043d0 2024-05-12T22:17:57.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.523+0000 7f61ea200640 1 -- 172.21.15.28:0/789036336 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61e41a6ea0 con 0x7f61e41043d0 2024-05-12T22:17:57.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.524+0000 7f61ea200640 1 -- 172.21.15.28:0/789036336 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61e4068650 con 0x7f61e41043d0 2024-05-12T22:17:57.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.527+0000 7f61dbe00640 1 -- 172.21.15.28:0/789036336 <== mon.2 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7f61d00306a0 con 0x7f61e41043d0 2024-05-12T22:17:57.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.529+0000 7f61dbe00640 1 --2- 172.21.15.28:0/789036336 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f61c80a2d00 0x7f61c80a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:57.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.529+0000 7f61dbe00640 1 -- 172.21.15.28:0/789036336 <== mon.2 v2:172.21.15.86:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8513+0+0 (secure 0 0 0) 0x7f61d00ef900 con 0x7f61e41043d0 2024-05-12T22:17:57.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.529+0000 7f61e3400640 1 --2- 172.21.15.28:0/789036336 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f61c80a2d00 0x7f61c80a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:57.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.530+0000 7f61e3400640 1 --2- 172.21.15.28:0/789036336 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f61c80a2d00 0x7f61c80a51c0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f61d4005fd0 tx=0x7f61d4010040 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:17:57.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.530+0000 7f61dbe00640 1 -- 172.21.15.28:0/789036336 <== mon.2 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f61d00bc5d0 con 0x7f61e41043d0 2024-05-12T22:17:57.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.772+0000 7f61ea200640 1 -- 172.21.15.28:0/789036336 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f61e410bfd0 con 0x7f61e41043d0 2024-05-12T22:17:57.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.774+0000 7f61dbe00640 1 -- 172.21.15.28:0/789036336 <== mon.2 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11864 (secure 0 0 0) 0x7f61d00bbed0 con 0x7f61e41043d0 2024-05-12T22:17:57.774 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-12T22:17:57.774 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1844141745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1844141745},{"type":"v1","addr":"172.21.15.186:6833","nonce":1844141745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6833/2960763758","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":2960763758},{"type":"v1","addr":"172.21.15.86:6833","nonce":2960763758}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24470,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/4131134219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":4131134219},{"type":"v1","addr":"172.21.15.28:6835","nonce":4131134219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24481,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/1727869064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1727869064},{"type":"v1","addr":"172.21.15.186:6835","nonce":1727869064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24482,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2938260704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2938260704},{"type":"v1","addr":"172.21.15.28:6837","nonce":2938260704}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24494,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3473700910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3473700910},{"type":"v1","addr":"172.21.15.86:6835","nonce":3473700910}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-12T22:06:33.426986+0000","modified":"2024-05-12T22:13:22.499381+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24581,"mds_1":24550,"mds_2":24532},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24499":{"gid":24499,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6837/3600809235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":3600809235},{"type":"v1","addr":"172.21.15.186:6837","nonce":3600809235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"i","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6839/2399610807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":2399610807},{"type":"v1","addr":"172.21.15.186:6839","nonce":2399610807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24530":{"gid":24530,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.86:6837/3782448884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":3782448884},{"type":"v1","addr":"172.21.15.86:6837","nonce":3782448884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24532":{"gid":24532,"name":"g","rank":2,"incarnation":21,"state":"up:active","state_seq":6,"addr":"172.21.15.28:6839/123892158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":123892158},{"type":"v1","addr":"172.21.15.28:6839","nonce":123892158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24550":{"gid":24550,"name":"a","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.28:6841/761183870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":761183870},{"type":"v1","addr":"172.21.15.28:6841","nonce":761183870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24581":{"gid":24581,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6839/1251999130","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6838","nonce":1251999130},{"type":"v1","addr":"172.21.15.86:6839","nonce":1251999130}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24581,"qdb_cluster":[24550,24581,24532]},"id":1}]} 2024-05-12T22:17:57.774 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 26 2024-05-12T22:17:57.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.777+0000 7f61ea200640 1 -- 172.21.15.28:0/789036336 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f61c80a2d00 msgr2=0x7f61c80a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:17:57.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.777+0000 7f61ea200640 1 --2- 172.21.15.28:0/789036336 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f61c80a2d00 0x7f61c80a51c0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f61d4005fd0 tx=0x7f61d4010040 comp rx=0 tx=0).stop 2024-05-12T22:17:57.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.777+0000 7f61ea200640 1 -- 172.21.15.28:0/789036336 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f61e41043d0 msgr2=0x7f61e419a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:17:57.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.777+0000 7f61ea200640 1 --2- 172.21.15.28:0/789036336 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f61e41043d0 0x7f61e419a870 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f61d0035040 tx=0x7f61d0033b60 comp rx=0 tx=0).stop 2024-05-12T22:17:57.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.777+0000 7f61ea200640 1 -- 172.21.15.28:0/789036336 shutdown_connections 2024-05-12T22:17:57.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.777+0000 7f61ea200640 1 --2- 172.21.15.28:0/789036336 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f61c80a2d00 0x7f61c80a51c0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:57.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.777+0000 7f61ea200640 1 --2- 172.21.15.28:0/789036336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61e4109a20 0x7f61e419ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:57.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.777+0000 7f61ea200640 1 --2- 172.21.15.28:0/789036336 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f61e41043d0 0x7f61e419a870 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:57.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.777+0000 7f61ea200640 1 --2- 172.21.15.28:0/789036336 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f61e4103a10 0x7f61e419a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:57.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.777+0000 7f61ea200640 1 -- 172.21.15.28:0/789036336 >> 172.21.15.28:0/789036336 conn(0x7f61e40ff4e0 msgr2=0x7f61e40ffca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:17:57.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.777+0000 7f61ea200640 1 -- 172.21.15.28:0/789036336 shutdown_connections 2024-05-12T22:17:57.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:57.777+0000 7f61ea200640 1 -- 172.21.15.28:0/789036336 wait complete. 2024-05-12T22:17:57.797 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-05-12T22:17:58.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.016+0000 7f60e3200640 1 -- 172.21.15.28:0/1926623145 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60e4076680 msgr2=0x7f60e4112030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:17:58.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.016+0000 7f60e3200640 1 --2- 172.21.15.28:0/1926623145 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60e4076680 0x7f60e4112030 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f60cc009900 tx=0x7f60cc02a3f0 comp rx=0 tx=0).stop 2024-05-12T22:17:58.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.016+0000 7f60e3200640 1 -- 172.21.15.28:0/1926623145 shutdown_connections 2024-05-12T22:17:58.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.016+0000 7f60e3200640 1 --2- 172.21.15.28:0/1926623145 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f60e4112570 0x7f60e4114960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:58.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.016+0000 7f60e3200640 1 --2- 172.21.15.28:0/1926623145 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60e4076680 0x7f60e4112030 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:58.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.016+0000 7f60e3200640 1 --2- 172.21.15.28:0/1926623145 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f60e4075d60 0x7f60e4076140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:58.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.016+0000 7f60e3200640 1 -- 172.21.15.28:0/1926623145 >> 172.21.15.28:0/1926623145 conn(0x7f60e40ff340 msgr2=0x7f60e4101760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:17:58.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.016+0000 7f60e3200640 1 -- 172.21.15.28:0/1926623145 shutdown_connections 2024-05-12T22:17:58.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.017+0000 7f60e3200640 1 -- 172.21.15.28:0/1926623145 wait complete. 2024-05-12T22:17:58.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.018+0000 7f60e3200640 1 Processor -- start 2024-05-12T22:17:58.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.018+0000 7f60e3200640 1 -- start start 2024-05-12T22:17:58.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.018+0000 7f60e3200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f60e4075d60 0x7f60e419e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:58.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.018+0000 7f60e3200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f60e4076680 0x7f60e419ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:58.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.018+0000 7f60e3200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60e4112570 0x7f60e41a2f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:58.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.018+0000 7f60e3200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f60e4118110 con 0x7f60e4112570 2024-05-12T22:17:58.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.018+0000 7f60e3200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f60e4117f90 con 0x7f60e4076680 2024-05-12T22:17:58.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.018+0000 7f60e3200640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f60e4118290 con 0x7f60e4075d60 2024-05-12T22:17:58.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.018+0000 7f60e2800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60e4112570 0x7f60e41a2f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:58.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.018+0000 7f60e2800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60e4112570 0x7f60e41a2f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40040/0 (socket says 172.21.15.28:40040) 2024-05-12T22:17:58.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.018+0000 7f60e2800640 1 -- 172.21.15.28:0/2011968764 learned_addr learned my addr 172.21.15.28:0/2011968764 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:17:58.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.018+0000 7f60d9400640 1 --2- 172.21.15.28:0/2011968764 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f60e4076680 0x7f60e419ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:58.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.019+0000 7f60e1e00640 1 --2- 172.21.15.28:0/2011968764 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f60e4075d60 0x7f60e419e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:58.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.019+0000 7f60e2800640 1 -- 172.21.15.28:0/2011968764 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f60e4075d60 msgr2=0x7f60e419e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:17:58.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.019+0000 7f60e2800640 1 --2- 172.21.15.28:0/2011968764 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f60e4075d60 0x7f60e419e670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:58.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.019+0000 7f60e2800640 1 -- 172.21.15.28:0/2011968764 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f60e4076680 msgr2=0x7f60e419ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:17:58.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.019+0000 7f60e2800640 1 --2- 172.21.15.28:0/2011968764 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f60e4076680 0x7f60e419ebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:58.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.019+0000 7f60e2800640 1 -- 172.21.15.28:0/2011968764 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60e41a3660 con 0x7f60e4112570 2024-05-12T22:17:58.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.019+0000 7f60e2800640 1 --2- 172.21.15.28:0/2011968764 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60e4112570 0x7f60e41a2f30 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f60dc00eb40 tx=0x7f60dc00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:17:58.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.020+0000 7f60db400640 1 -- 172.21.15.28:0/2011968764 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60dc004b20 con 0x7f60e4112570 2024-05-12T22:17:58.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.020+0000 7f60e3200640 1 -- 172.21.15.28:0/2011968764 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60e41a3950 con 0x7f60e4112570 2024-05-12T22:17:58.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.020+0000 7f60db400640 1 -- 172.21.15.28:0/2011968764 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f60dc004cc0 con 0x7f60e4112570 2024-05-12T22:17:58.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.020+0000 7f60db400640 1 -- 172.21.15.28:0/2011968764 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60dc010850 con 0x7f60e4112570 2024-05-12T22:17:58.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.020+0000 7f60e3200640 1 -- 172.21.15.28:0/2011968764 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60e41ab1e0 con 0x7f60e4112570 2024-05-12T22:17:58.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.024+0000 7f60ebff6740 1 Processor -- start 2024-05-12T22:17:58.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.024+0000 7f60db400640 1 -- 172.21.15.28:0/2011968764 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7f60dc005020 con 0x7f60e4112570 2024-05-12T22:17:58.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.025+0000 7f60ebff6740 1 -- start start 2024-05-12T22:17:58.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.025+0000 7f60ebff6740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55738d69ab60 0x55738d69af40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:58.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.025+0000 7f60ebff6740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55738d69b480 0x55738d6a5610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:58.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.025+0000 7f60ebff6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55738d6a5b50 0x55738d6a7f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:58.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.025+0000 7f60ebff6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55738d5ffa30 con 0x55738d6a5b50 2024-05-12T22:17:58.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.025+0000 7f60ebff6740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x55738d5a7700 con 0x55738d69ab60 2024-05-12T22:17:58.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.025+0000 7f60ebff6740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55738d5fb9f0 con 0x55738d69b480 2024-05-12T22:17:58.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.026+0000 7f60d9400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55738d69ab60 0x55738d69af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:58.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.026+0000 7f60d9400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55738d69ab60 0x55738d69af40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:33356/0 (socket says 172.21.15.28:33356) 2024-05-12T22:17:58.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.026+0000 7f60d9400640 1 -- 172.21.15.28:0/892333360 learned_addr learned my addr 172.21.15.28:0/892333360 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:17:58.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.026+0000 7f60e1e00640 1 --2- 172.21.15.28:0/892333360 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55738d69b480 0x55738d6a5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:58.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.026+0000 7f60d9400640 1 --2- 172.21.15.28:0/892333360 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55738d6a5b50 0x55738d6a7f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:58.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.026+0000 7f60e1e00640 1 -- 172.21.15.28:0/892333360 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55738d69ab60 msgr2=0x55738d69af40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:17:58.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.026+0000 7f60e1e00640 1 --2- 172.21.15.28:0/892333360 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55738d69ab60 0x55738d69af40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:58.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.026+0000 7f60e1e00640 1 -- 172.21.15.28:0/892333360 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55738d6a5b50 msgr2=0x55738d6a7f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:17:58.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.026+0000 7f60e1e00640 1 --2- 172.21.15.28:0/892333360 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55738d6a5b50 0x55738d6a7f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:58.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.026+0000 7f60e1e00640 1 -- 172.21.15.28:0/892333360 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55738d3133f0 con 0x55738d69b480 2024-05-12T22:17:58.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.026+0000 7f60db400640 1 --2- 172.21.15.28:0/2011968764 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f60c80a2d00 0x7f60c80a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:58.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.027+0000 7f60e1e00640 1 --2- 172.21.15.28:0/892333360 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55738d69b480 0x55738d6a5610 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f60d4009c20 tx=0x7f60d4011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1a7636c524334263 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:17:58.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.027+0000 7f60db400640 1 -- 172.21.15.28:0/2011968764 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8513+0+0 (secure 0 0 0) 0x7f60dc0cfc30 con 0x7f60e4112570 2024-05-12T22:17:58.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.027+0000 7f60d9400640 1 --2- 172.21.15.28:0/2011968764 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f60c80a2d00 0x7f60c80a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:58.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.027+0000 7f60d3400640 1 -- 172.21.15.28:0/892333360 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60d4004050 con 0x55738d69b480 2024-05-12T22:17:58.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.027+0000 7f60d3400640 1 -- 172.21.15.28:0/892333360 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f60d4005500 con 0x55738d69b480 2024-05-12T22:17:58.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.027+0000 7f60d3400640 1 -- 172.21.15.28:0/892333360 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60d4004d50 con 0x55738d69b480 2024-05-12T22:17:58.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.028+0000 7f60ebff6740 1 -- 172.21.15.28:0/892333360 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55738d69b480 msgr2=0x55738d6a5610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:17:58.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.028+0000 7f60d9400640 1 --2- 172.21.15.28:0/2011968764 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f60c80a2d00 0x7f60c80a51c0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x55738d698010 tx=0x7f60cc02a3f0 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:17:58.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.028+0000 7f60ebff6740 1 --2- 172.21.15.28:0/892333360 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55738d69b480 0x55738d6a5610 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f60d4009c20 tx=0x7f60d4011810 comp rx=0 tx=0).stop 2024-05-12T22:17:58.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.028+0000 7f60ebff6740 1 -- 172.21.15.28:0/892333360 shutdown_connections 2024-05-12T22:17:58.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.028+0000 7f60ebff6740 1 --2- 172.21.15.28:0/892333360 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55738d6a5b50 0x55738d6a7f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:58.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.028+0000 7f60ebff6740 1 --2- 172.21.15.28:0/892333360 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55738d69b480 0x55738d6a5610 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:58.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.028+0000 7f60ebff6740 1 --2- 172.21.15.28:0/892333360 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55738d69ab60 0x55738d69af40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:58.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.028+0000 7f60ebff6740 1 -- 172.21.15.28:0/892333360 >> 172.21.15.28:0/892333360 conn(0x55738d696e80 msgr2=0x55738d697260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:17:58.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.028+0000 7f60ebff6740 1 -- 172.21.15.28:0/892333360 shutdown_connections 2024-05-12T22:17:58.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.028+0000 7f60ebff6740 1 -- 172.21.15.28:0/892333360 wait complete. 2024-05-12T22:17:58.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.028+0000 7f60ebff6740 1 -- start start 2024-05-12T22:17:58.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.029+0000 7f60ebff6740 1 Processor -- start 2024-05-12T22:17:58.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.029+0000 7f60ebff6740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55738d69ab60 0x55738d6a2a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:58.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.029+0000 7f60ebff6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55738d69b480 0x55738d6a2fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:58.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.029+0000 7f60ebff6740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55738d6a5b50 0x55738d6a34e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:58.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.029+0000 7f60ebff6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55738d6a8940 con 0x55738d69b480 2024-05-12T22:17:58.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.029+0000 7f60ebff6740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x55738d5baa70 con 0x55738d6a5b50 2024-05-12T22:17:58.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.029+0000 7f60ebff6740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55738d6a8ac0 con 0x55738d69ab60 2024-05-12T22:17:58.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.029+0000 7f60ebff6740 10 client.0 _ll_register_callbacks cb 0x55738d59dba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:17:58.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.029+0000 7f60d9400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55738d69b480 0x55738d6a2fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:58.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.029+0000 7f60e1e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55738d69ab60 0x55738d6a2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:58.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.029+0000 7f60d9400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55738d69b480 0x55738d6a2fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40056/0 (socket says 172.21.15.28:40056) 2024-05-12T22:17:58.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.029+0000 7f60d9400640 1 -- 172.21.15.28:0/976127198 learned_addr learned my addr 172.21.15.28:0/976127198 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:17:58.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.029+0000 7f60e2800640 1 --2- 172.21.15.28:0/976127198 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55738d6a5b50 0x55738d6a34e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:58.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.030+0000 7f60d9400640 1 -- 172.21.15.28:0/976127198 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55738d69ab60 msgr2=0x55738d6a2a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:17:58.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.030+0000 7f60d9400640 1 --2- 172.21.15.28:0/976127198 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55738d69ab60 0x55738d6a2a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:58.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.030+0000 7f60d9400640 1 -- 172.21.15.28:0/976127198 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55738d6a5b50 msgr2=0x55738d6a34e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:17:58.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.030+0000 7f60d9400640 1 --2- 172.21.15.28:0/976127198 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55738d6a5b50 0x55738d6a34e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:58.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.030+0000 7f60d9400640 1 -- 172.21.15.28:0/976127198 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55738d3dd7d0 con 0x55738d69b480 2024-05-12T22:17:58.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.030+0000 7f60d9400640 1 --2- 172.21.15.28:0/976127198 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55738d69b480 0x55738d6a2fa0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f60cc02a910 tx=0x7f60cc005b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48338c4dbfb3878f server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:17:58.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.030+0000 7f60c7e00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-12T22:17:58.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.030+0000 7f60c7e00640 1 -- 172.21.15.28:0/976127198 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60cc004a40 con 0x55738d69b480 2024-05-12T22:17:58.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.031+0000 7f60c7e00640 1 -- 172.21.15.28:0/976127198 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8513+0+0 (secure 0 0 0) 0x7f60cc004be0 con 0x55738d69b480 2024-05-12T22:17:58.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.031+0000 7f60ebff6740 1 -- 172.21.15.28:0/976127198 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f60d4005500 con 0x55738d69b480 2024-05-12T22:17:58.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.031+0000 7f60c7e00640 1 -- 172.21.15.28:0/976127198 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f60cc004de0 con 0x55738d69b480 2024-05-12T22:17:58.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.031+0000 7f60ebff6740 10 client.17040 fetch_fsmap learned FSMap version 26 2024-05-12T22:17:58.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.031+0000 7f60ebff6740 1 -- 172.21.15.28:0/976127198 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55738d6b4b90 con 0x55738d69b480 2024-05-12T22:17:58.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.031+0000 7f60c7e00640 1 -- 172.21.15.28:0/976127198 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f60d4005500 con 0x55738d69b480 2024-05-12T22:17:58.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.031+0000 7f60c7e00640 1 -- 172.21.15.28:0/976127198 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f60cc032880 con 0x55738d69b480 2024-05-12T22:17:58.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.032+0000 7f60ebff6740 10 client.17040 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:17:58.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.032+0000 7f60ebff6740 10 client.17040 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.g 2024-05-12T22:17:58.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.032+0000 7f60ebff6740 20 client.17040 populate_metadata read hostname 'smithi028' 2024-05-12T22:17:58.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.032+0000 7f60ebff6740 1 --2- 172.21.15.28:0/976127198 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x55738d6b6050 0x55738d6b84b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:58.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.032+0000 7f60ebff6740 4 client.17040 mds_command: new command op to 24532 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-12T22:17:58.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.032+0000 7f60ebff6740 1 -- 172.21.15.28:0/976127198 --> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55738d60d160 con 0x55738d6b6050 2024-05-12T22:17:58.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.032+0000 7f60e1e00640 1 --2- 172.21.15.28:0/976127198 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x55738d6b6050 0x55738d6b84b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:58.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.033+0000 7f60e1e00640 1 --2- 172.21.15.28:0/976127198 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x55738d6b6050 0x55738d6b84b0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=4d6ead6e04bec1d2 server_cookie=f87e85a712fae84d in_seq=0 out_seq=0 2024-05-12T22:17:58.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.033+0000 7f60c7e00640 10 client.17040 ms_handle_connect on v2:172.21.15.28:6838/123892158 2024-05-12T22:17:58.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.033+0000 7f60c7e00640 1 -- 172.21.15.28:0/976127198 <== mds.2 v2:172.21.15.28:6838/123892158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55738d60d160 con 0x55738d6b6050 2024-05-12T22:17:58.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.033+0000 7f60c7e00640 10 client.17040 handle_command_reply: tid=0 2024-05-12T22:17:58.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.033+0000 7f60ebff6740 1 client.17040 shutdown 2024-05-12T22:17:58.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.034+0000 7f60ebff6740 1 -- 172.21.15.28:0/976127198 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55738d69b480 msgr2=0x55738d6a2fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:17:58.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.034+0000 7f60ebff6740 1 --2- 172.21.15.28:0/976127198 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55738d69b480 0x55738d6a2fa0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f60cc02a910 tx=0x7f60cc005b50 comp rx=0 tx=0).stop 2024-05-12T22:17:58.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.034+0000 7f60ebff6740 1 -- 172.21.15.28:0/976127198 shutdown_connections 2024-05-12T22:17:58.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.034+0000 7f60ebff6740 1 --2- 172.21.15.28:0/976127198 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x55738d6b6050 0x55738d6b84b0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:58.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.034+0000 7f60ebff6740 1 --2- 172.21.15.28:0/976127198 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55738d6a5b50 0x55738d6a34e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:58.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.034+0000 7f60ebff6740 1 --2- 172.21.15.28:0/976127198 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55738d69b480 0x55738d6a2fa0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:58.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.034+0000 7f60ebff6740 1 --2- 172.21.15.28:0/976127198 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55738d69ab60 0x55738d6a2a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:58.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.034+0000 7f60c7e00640 0 client.17040 ms_handle_reset on v2:172.21.15.28:6838/123892158 2024-05-12T22:17:58.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.034+0000 7f60ebff6740 1 -- 172.21.15.28:0/976127198 >> 172.21.15.28:0/976127198 conn(0x55738d696e80 msgr2=0x55738d69cc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:17:58.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.034+0000 7f60ebff6740 1 -- 172.21.15.28:0/976127198 shutdown_connections 2024-05-12T22:17:58.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.034+0000 7f60ebff6740 1 -- 172.21.15.28:0/976127198 wait complete. 2024-05-12T22:17:58.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.034+0000 7f60ebff6740 20 client.17040 trim_cache size 0 max 16384 2024-05-12T22:17:58.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.049+0000 7f60ebff6740 1 Processor -- start 2024-05-12T22:17:58.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.049+0000 7f60ebff6740 1 -- start start 2024-05-12T22:17:58.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.049+0000 7f60ebff6740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55738d69ab60 0x55738d6b1830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:58.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.049+0000 7f60ebff6740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55738d69b480 0x55738d69d300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:58.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.049+0000 7f60ebff6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55738d6a5b50 0x55738d6bdd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:58.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.049+0000 7f60ebff6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f60e41b62a0 con 0x55738d6a5b50 2024-05-12T22:17:58.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.050+0000 7f60ebff6740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f60e41b3b70 con 0x55738d69b480 2024-05-12T22:17:58.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.050+0000 7f60ebff6740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f60e41b19e0 con 0x55738d69ab60 2024-05-12T22:17:58.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.050+0000 7f60e2800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55738d6a5b50 0x55738d6bdd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:58.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.050+0000 7f60e1e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55738d69ab60 0x55738d6b1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:58.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.050+0000 7f60d9400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55738d69b480 0x55738d69d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:58.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.050+0000 7f60e2800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55738d6a5b50 0x55738d6bdd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40064/0 (socket says 172.21.15.28:40064) 2024-05-12T22:17:58.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.050+0000 7f60e2800640 1 -- 172.21.15.28:0/268984806 learned_addr learned my addr 172.21.15.28:0/268984806 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:17:58.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.050+0000 7f60e2800640 1 -- 172.21.15.28:0/268984806 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55738d69ab60 msgr2=0x55738d6b1830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:17:58.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.050+0000 7f60e2800640 1 --2- 172.21.15.28:0/268984806 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55738d69ab60 0x55738d6b1830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:58.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.050+0000 7f60e2800640 1 -- 172.21.15.28:0/268984806 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55738d69b480 msgr2=0x55738d69d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:17:58.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.050+0000 7f60e2800640 1 --2- 172.21.15.28:0/268984806 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55738d69b480 0x55738d69d300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:58.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.050+0000 7f60e2800640 1 -- 172.21.15.28:0/268984806 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55738d6913e0 con 0x55738d6a5b50 2024-05-12T22:17:58.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.050+0000 7f60e1e00640 1 --2- 172.21.15.28:0/268984806 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55738d69ab60 0x55738d6b1830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:17:58.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.051+0000 7f60e2800640 1 --2- 172.21.15.28:0/268984806 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55738d6a5b50 0x55738d6bdd80 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f60dc00eb10 tx=0x7f60dc09bb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c48835a102d43d9f server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:17:58.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.051+0000 7f60d2a00640 1 -- 172.21.15.28:0/268984806 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60dc0b4020 con 0x55738d6a5b50 2024-05-12T22:17:58.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.051+0000 7f60d2a00640 1 -- 172.21.15.28:0/268984806 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f60dc09c0f0 con 0x55738d6a5b50 2024-05-12T22:17:58.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.051+0000 7f60d2a00640 1 -- 172.21.15.28:0/268984806 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60dc005310 con 0x55738d6a5b50 2024-05-12T22:17:58.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.052+0000 7f60ebff6740 1 -- 172.21.15.28:0/268984806 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55738d6a5b50 msgr2=0x55738d6bdd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:17:58.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.052+0000 7f60ebff6740 1 --2- 172.21.15.28:0/268984806 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55738d6a5b50 0x55738d6bdd80 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f60dc00eb10 tx=0x7f60dc09bb10 comp rx=0 tx=0).stop 2024-05-12T22:17:58.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.052+0000 7f60ebff6740 1 -- 172.21.15.28:0/268984806 shutdown_connections 2024-05-12T22:17:58.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.052+0000 7f60ebff6740 1 --2- 172.21.15.28:0/268984806 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55738d6a5b50 0x55738d6bdd80 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:58.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.052+0000 7f60ebff6740 1 --2- 172.21.15.28:0/268984806 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55738d69b480 0x55738d69d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:58.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.052+0000 7f60ebff6740 1 --2- 172.21.15.28:0/268984806 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55738d69ab60 0x55738d6b1830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:58.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.052+0000 7f60ebff6740 1 -- 172.21.15.28:0/268984806 >> 172.21.15.28:0/268984806 conn(0x55738d696e80 msgr2=0x55738d6a32b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:17:58.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.052+0000 7f60ebff6740 1 -- 172.21.15.28:0/268984806 shutdown_connections 2024-05-12T22:17:58.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.052+0000 7f60ebff6740 1 -- 172.21.15.28:0/268984806 wait complete. 2024-05-12T22:17:58.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.052+0000 7f60ebff6740 1 -- start start 2024-05-12T22:17:58.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.053+0000 7f60ebff6740 1 Processor -- start 2024-05-12T22:17:58.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.053+0000 7f60ebff6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55738d69ab60 0x55738d6a97c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:58.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.053+0000 7f60ebff6740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55738d69b480 0x55738d6a9d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:58.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.053+0000 7f60ebff6740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55738d6a5b50 0x55738d6aa240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:58.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.053+0000 7f60ebff6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f60e41b2030 con 0x55738d69ab60 2024-05-12T22:17:58.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.053+0000 7f60ebff6740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f60e41b34a0 con 0x55738d69b480 2024-05-12T22:17:58.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.053+0000 7f60ebff6740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f60e41b2700 con 0x55738d6a5b50 2024-05-12T22:17:58.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.053+0000 7f60ebff6740 10 client.0 _ll_register_callbacks cb 0x7f60cc005040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:17:58.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.053+0000 7f60d9400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55738d69b480 0x55738d6a9d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:58.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.054+0000 7f60e2800640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55738d6a5b50 0x55738d6aa240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:58.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.054+0000 7f60e1e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55738d69ab60 0x55738d6a97c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:58.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.054+0000 7f60d9400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55738d69b480 0x55738d6a9d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:33380/0 (socket says 172.21.15.28:33380) 2024-05-12T22:17:58.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.054+0000 7f60d9400640 1 -- 172.21.15.28:0/2908161003 learned_addr learned my addr 172.21.15.28:0/2908161003 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:17:58.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.054+0000 7f60e2800640 1 -- 172.21.15.28:0/2908161003 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55738d69b480 msgr2=0x55738d6a9d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:17:58.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.054+0000 7f60e2800640 1 --2- 172.21.15.28:0/2908161003 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55738d69b480 0x55738d6a9d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:58.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.054+0000 7f60e2800640 1 -- 172.21.15.28:0/2908161003 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55738d69ab60 msgr2=0x55738d6a97c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:17:58.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.054+0000 7f60e2800640 1 --2- 172.21.15.28:0/2908161003 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55738d69ab60 0x55738d6a97c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:58.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.054+0000 7f60e2800640 1 -- 172.21.15.28:0/2908161003 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55738d71b620 con 0x55738d6a5b50 2024-05-12T22:17:58.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.054+0000 7f60e2800640 1 --2- 172.21.15.28:0/2908161003 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55738d6a5b50 0x55738d6aa240 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f60dc0bc360 tx=0x7f60dc09bb80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dd0597b340422797 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:17:58.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.054+0000 7f60c7e00640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-12T22:17:58.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.054+0000 7f60c7e00640 1 -- 172.21.15.28:0/2908161003 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60dc07e020 con 0x55738d6a5b50 2024-05-12T22:17:58.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.055+0000 7f60ebff6740 1 -- 172.21.15.28:0/2908161003 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f60dc09c0f0 con 0x55738d6a5b50 2024-05-12T22:17:58.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.055+0000 7f60c7e00640 1 -- 172.21.15.28:0/2908161003 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8513+0+0 (secure 0 0 0) 0x7f60dc005310 con 0x55738d6a5b50 2024-05-12T22:17:58.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.055+0000 7f60c7e00640 1 -- 172.21.15.28:0/2908161003 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f60dc0b4070 con 0x55738d6a5b50 2024-05-12T22:17:58.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.055+0000 7f60ebff6740 10 client.27133 fetch_fsmap learned FSMap version 26 2024-05-12T22:17:58.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.055+0000 7f60ebff6740 1 -- 172.21.15.28:0/2908161003 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55738d71c8e0 con 0x55738d6a5b50 2024-05-12T22:17:58.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.055+0000 7f60c7e00640 1 -- 172.21.15.28:0/2908161003 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f60dc09c0f0 con 0x55738d6a5b50 2024-05-12T22:17:58.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.055+0000 7f60c7e00640 1 -- 172.21.15.28:0/2908161003 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f60dc0b6070 con 0x55738d6a5b50 2024-05-12T22:17:58.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.055+0000 7f60ebff6740 10 client.27133 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:17:58.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.055+0000 7f60ebff6740 10 client.27133 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.g 2024-05-12T22:17:58.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.055+0000 7f60ebff6740 20 client.27133 populate_metadata read hostname 'smithi028' 2024-05-12T22:17:58.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.055+0000 7f60ebff6740 1 --2- 172.21.15.28:0/2908161003 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x55738d71cf80 0x55738d71f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:58.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.055+0000 7f60ebff6740 4 client.27133 mds_command: new command op to 24532 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-12T22:17:58.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.055+0000 7f60ebff6740 1 -- 172.21.15.28:0/2908161003 --> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55738d6074b0 con 0x55738d71cf80 2024-05-12T22:17:58.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.056+0000 7f60e1e00640 1 --2- 172.21.15.28:0/2908161003 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x55738d71cf80 0x55738d71f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:58.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.056+0000 7f60e1e00640 1 --2- 172.21.15.28:0/2908161003 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x55738d71cf80 0x55738d71f3a0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c2c0ae7d64ffbf63 server_cookie=1f222decfd61ef18 in_seq=0 out_seq=0 2024-05-12T22:17:58.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.056+0000 7f60c7e00640 10 client.27133 ms_handle_connect on v2:172.21.15.28:6838/123892158 2024-05-12T22:17:58.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.056+0000 7f60c7e00640 1 -- 172.21.15.28:0/2908161003 <== mds.2 v2:172.21.15.28:6838/123892158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55738d6074b0 con 0x55738d71cf80 2024-05-12T22:17:58.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.056+0000 7f60c7e00640 10 client.27133 handle_command_reply: tid=0 2024-05-12T22:17:58.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.056+0000 7f60ebff6740 1 client.27133 shutdown 2024-05-12T22:17:58.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.057+0000 7f60ebff6740 1 -- 172.21.15.28:0/2908161003 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55738d6a5b50 msgr2=0x55738d6aa240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:17:58.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.057+0000 7f60ebff6740 1 --2- 172.21.15.28:0/2908161003 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55738d6a5b50 0x55738d6aa240 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f60dc0bc360 tx=0x7f60dc09bb80 comp rx=0 tx=0).stop 2024-05-12T22:17:58.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.057+0000 7f60ebff6740 1 -- 172.21.15.28:0/2908161003 shutdown_connections 2024-05-12T22:17:58.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.057+0000 7f60ebff6740 1 --2- 172.21.15.28:0/2908161003 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55738d6a5b50 0x55738d6aa240 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:58.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.057+0000 7f60ebff6740 1 --2- 172.21.15.28:0/2908161003 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x55738d71cf80 0x55738d71f3a0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:58.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.057+0000 7f60ebff6740 1 --2- 172.21.15.28:0/2908161003 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55738d69b480 0x55738d6a9d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:58.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.057+0000 7f60ebff6740 1 --2- 172.21.15.28:0/2908161003 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55738d69ab60 0x55738d6a97c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:58.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.057+0000 7f60ebff6740 1 -- 172.21.15.28:0/2908161003 >> 172.21.15.28:0/2908161003 conn(0x55738d696e80 msgr2=0x55738d695770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:17:58.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.057+0000 7f60c7e00640 0 client.27133 ms_handle_reset on v2:172.21.15.28:6838/123892158 2024-05-12T22:17:58.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.057+0000 7f60ebff6740 1 -- 172.21.15.28:0/2908161003 shutdown_connections 2024-05-12T22:17:58.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.057+0000 7f60ebff6740 1 -- 172.21.15.28:0/2908161003 wait complete. 2024-05-12T22:17:58.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.057+0000 7f60ebff6740 20 client.27133 trim_cache size 0 max 16384 2024-05-12T22:17:58.064 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-05-12T22:17:58.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.058+0000 7f60e3200640 1 -- 172.21.15.28:0/2011968764 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f60c80a2d00 msgr2=0x7f60c80a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:17:58.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.058+0000 7f60e3200640 1 --2- 172.21.15.28:0/2011968764 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f60c80a2d00 0x7f60c80a51c0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x55738d698010 tx=0x7f60cc02a3f0 comp rx=0 tx=0).stop 2024-05-12T22:17:58.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.058+0000 7f60e3200640 1 -- 172.21.15.28:0/2011968764 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60e4112570 msgr2=0x7f60e41a2f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:17:58.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.058+0000 7f60e3200640 1 --2- 172.21.15.28:0/2011968764 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60e4112570 0x7f60e41a2f30 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f60dc00eb40 tx=0x7f60dc00c2d0 comp rx=0 tx=0).stop 2024-05-12T22:17:58.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.058+0000 7f60e3200640 1 -- 172.21.15.28:0/2011968764 shutdown_connections 2024-05-12T22:17:58.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.058+0000 7f60e3200640 1 --2- 172.21.15.28:0/2011968764 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f60c80a2d00 0x7f60c80a51c0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:58.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.058+0000 7f60e3200640 1 --2- 172.21.15.28:0/2011968764 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60e4112570 0x7f60e41a2f30 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:58.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.058+0000 7f60e3200640 1 --2- 172.21.15.28:0/2011968764 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f60e4076680 0x7f60e419ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:58.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.058+0000 7f60e3200640 1 --2- 172.21.15.28:0/2011968764 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f60e4075d60 0x7f60e419e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:58.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.058+0000 7f60e3200640 1 -- 172.21.15.28:0/2011968764 >> 172.21.15.28:0/2011968764 conn(0x7f60e40ff340 msgr2=0x7f60e4113c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:17:58.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.058+0000 7f60e3200640 1 -- 172.21.15.28:0/2011968764 shutdown_connections 2024-05-12T22:17:58.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.058+0000 7f60e3200640 1 -- 172.21.15.28:0/2011968764 wait complete. 2024-05-12T22:17:58.076 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-05-12T22:17:58.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.281+0000 7fa9eec00640 1 -- 172.21.15.28:0/2694720607 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa9e8076680 msgr2=0x7fa9e8112170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:17:58.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.281+0000 7fa9eec00640 1 --2- 172.21.15.28:0/2694720607 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa9e8076680 0x7fa9e8112170 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fa9d4009890 tx=0x7fa9d402a360 comp rx=0 tx=0).stop 2024-05-12T22:17:58.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.281+0000 7fa9eec00640 1 -- 172.21.15.28:0/2694720607 shutdown_connections 2024-05-12T22:17:58.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.281+0000 7fa9eec00640 1 --2- 172.21.15.28:0/2694720607 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa9e81126b0 0x7fa9e8114aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:58.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.281+0000 7fa9eec00640 1 --2- 172.21.15.28:0/2694720607 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa9e8076680 0x7fa9e8112170 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:58.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.281+0000 7fa9eec00640 1 --2- 172.21.15.28:0/2694720607 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9e8075d60 0x7fa9e8076140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:58.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.281+0000 7fa9eec00640 1 -- 172.21.15.28:0/2694720607 >> 172.21.15.28:0/2694720607 conn(0x7fa9e80ff4a0 msgr2=0x7fa9e81018c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:17:58.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.281+0000 7fa9eec00640 1 -- 172.21.15.28:0/2694720607 shutdown_connections 2024-05-12T22:17:58.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.281+0000 7fa9eec00640 1 -- 172.21.15.28:0/2694720607 wait complete. 2024-05-12T22:17:58.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.282+0000 7fa9eec00640 1 Processor -- start 2024-05-12T22:17:58.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.282+0000 7fa9eec00640 1 -- start start 2024-05-12T22:17:58.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.283+0000 7fa9eec00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa9e8075d60 0x7fa9e81a4460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:58.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.283+0000 7fa9eec00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9e8076680 0x7fa9e81a49a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:58.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.283+0000 7fa9eec00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa9e81126b0 0x7fa9e819e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:58.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.283+0000 7fa9eec00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa9e8118200 con 0x7fa9e8076680 2024-05-12T22:17:58.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.283+0000 7fa9eec00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fa9e8118080 con 0x7fa9e8075d60 2024-05-12T22:17:58.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.283+0000 7fa9eec00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa9e8118380 con 0x7fa9e81126b0 2024-05-12T22:17:58.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.283+0000 7fa9e7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9e8076680 0x7fa9e81a49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:58.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.283+0000 7fa9e7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9e8076680 0x7fa9e81a49a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40092/0 (socket says 172.21.15.28:40092) 2024-05-12T22:17:58.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.283+0000 7fa9e7400640 1 -- 172.21.15.28:0/2419990429 learned_addr learned my addr 172.21.15.28:0/2419990429 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:17:58.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.283+0000 7fa9ecc00640 1 --2- 172.21.15.28:0/2419990429 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa9e81126b0 0x7fa9e819e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:58.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.283+0000 7fa9e7e00640 1 --2- 172.21.15.28:0/2419990429 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa9e8075d60 0x7fa9e81a4460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:58.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.284+0000 7fa9e7400640 1 -- 172.21.15.28:0/2419990429 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa9e81126b0 msgr2=0x7fa9e819e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:17:58.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.284+0000 7fa9e7400640 1 --2- 172.21.15.28:0/2419990429 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa9e81126b0 0x7fa9e819e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:58.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.284+0000 7fa9e7400640 1 -- 172.21.15.28:0/2419990429 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa9e8075d60 msgr2=0x7fa9e81a4460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:17:58.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.284+0000 7fa9e7400640 1 --2- 172.21.15.28:0/2419990429 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa9e8075d60 0x7fa9e81a4460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:58.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.284+0000 7fa9e7400640 1 -- 172.21.15.28:0/2419990429 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9e819ed90 con 0x7fa9e8076680 2024-05-12T22:17:58.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.284+0000 7fa9ecc00640 1 --2- 172.21.15.28:0/2419990429 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa9e81126b0 0x7fa9e819e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:17:58.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.284+0000 7fa9e7400640 1 --2- 172.21.15.28:0/2419990429 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9e8076680 0x7fa9e81a49a0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fa9d4033ae0 tx=0x7fa9d4033b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:17:58.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.284+0000 7fa9e4c00640 1 -- 172.21.15.28:0/2419990429 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9d4037040 con 0x7fa9e8076680 2024-05-12T22:17:58.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.284+0000 7fa9eec00640 1 -- 172.21.15.28:0/2419990429 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9e819f020 con 0x7fa9e8076680 2024-05-12T22:17:58.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.284+0000 7fa9e4c00640 1 -- 172.21.15.28:0/2419990429 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa9d4040070 con 0x7fa9e8076680 2024-05-12T22:17:58.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.285+0000 7fa9eec00640 1 -- 172.21.15.28:0/2419990429 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9e81ab320 con 0x7fa9e8076680 2024-05-12T22:17:58.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.285+0000 7fa9e4c00640 1 -- 172.21.15.28:0/2419990429 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9d4032b20 con 0x7fa9e8076680 2024-05-12T22:17:58.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.288+0000 7fa9e4c00640 1 -- 172.21.15.28:0/2419990429 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7fa9d4032cc0 con 0x7fa9e8076680 2024-05-12T22:17:58.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.288+0000 7fa9f1a5c740 1 Processor -- start 2024-05-12T22:17:58.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.288+0000 7fa9f1a5c740 1 -- start start 2024-05-12T22:17:58.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.289+0000 7fa9f1a5c740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56537324b980 0x56537324bd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:58.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.289+0000 7fa9f1a5c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56537324c2a0 0x565373256430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:58.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.289+0000 7fa9f1a5c740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x565373256970 0x565373258d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:58.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.289+0000 7fa9f1a5c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5653731b0a30 con 0x56537324c2a0 2024-05-12T22:17:58.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.289+0000 7fa9f1a5c740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x565373158700 con 0x56537324b980 2024-05-12T22:17:58.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.289+0000 7fa9f1a5c740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5653731ac9f0 con 0x565373256970 2024-05-12T22:17:58.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.289+0000 7fa9e7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56537324c2a0 0x565373256430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:58.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.289+0000 7fa9ecc00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56537324b980 0x56537324bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:58.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.289+0000 7fa9e7400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x565373256970 0x565373258d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:58.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.289+0000 7fa9ecc00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56537324b980 0x56537324bd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:33406/0 (socket says 172.21.15.28:33406) 2024-05-12T22:17:58.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.289+0000 7fa9ecc00640 1 -- 172.21.15.28:0/1417985806 learned_addr learned my addr 172.21.15.28:0/1417985806 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:17:58.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.289+0000 7fa9ecc00640 1 -- 172.21.15.28:0/1417985806 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x565373256970 msgr2=0x565373258d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:17:58.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.289+0000 7fa9ecc00640 1 --2- 172.21.15.28:0/1417985806 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x565373256970 0x565373258d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:58.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.289+0000 7fa9ecc00640 1 -- 172.21.15.28:0/1417985806 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56537324c2a0 msgr2=0x565373256430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:17:58.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.290+0000 7fa9ecc00640 1 --2- 172.21.15.28:0/1417985806 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56537324c2a0 0x565373256430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:58.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.290+0000 7fa9ecc00640 1 -- 172.21.15.28:0/1417985806 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565372ec43f0 con 0x56537324b980 2024-05-12T22:17:58.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.290+0000 7fa9e7e00640 1 --2- 172.21.15.28:0/1417985806 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56537324c2a0 0x565373256430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:17:58.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.290+0000 7fa9ecc00640 1 --2- 172.21.15.28:0/1417985806 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56537324b980 0x56537324bd60 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fa9e0009f90 tx=0x7fa9e00118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=51f56674c45ed77d server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:17:58.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.290+0000 7fa9e4c00640 1 --2- 172.21.15.28:0/2419990429 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fa9cc0a2d00 0x7fa9cc0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:58.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.290+0000 7fa9e4c00640 1 -- 172.21.15.28:0/2419990429 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8513+0+0 (secure 0 0 0) 0x7fa9d40ef0e0 con 0x7fa9e8076680 2024-05-12T22:17:58.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.290+0000 7fa9dd200640 1 -- 172.21.15.28:0/1417985806 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9e001e070 con 0x56537324b980 2024-05-12T22:17:58.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.290+0000 7fa9e7e00640 1 --2- 172.21.15.28:0/2419990429 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fa9cc0a2d00 0x7fa9cc0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:58.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.290+0000 7fa9dd200640 1 -- 172.21.15.28:0/1417985806 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa9e0004310 con 0x56537324b980 2024-05-12T22:17:58.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.290+0000 7fa9dd200640 1 -- 172.21.15.28:0/1417985806 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9e000f980 con 0x56537324b980 2024-05-12T22:17:58.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.291+0000 7fa9f1a5c740 1 -- 172.21.15.28:0/1417985806 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56537324b980 msgr2=0x56537324bd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:17:58.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.291+0000 7fa9e7e00640 1 --2- 172.21.15.28:0/2419990429 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fa9cc0a2d00 0x7fa9cc0a51c0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x56537324cf40 tx=0x7fa9d8010040 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:17:58.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.291+0000 7fa9f1a5c740 1 --2- 172.21.15.28:0/1417985806 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56537324b980 0x56537324bd60 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fa9e0009f90 tx=0x7fa9e00118d0 comp rx=0 tx=0).stop 2024-05-12T22:17:58.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.291+0000 7fa9f1a5c740 1 -- 172.21.15.28:0/1417985806 shutdown_connections 2024-05-12T22:17:58.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.291+0000 7fa9f1a5c740 1 --2- 172.21.15.28:0/1417985806 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x565373256970 0x565373258d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:58.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.291+0000 7fa9f1a5c740 1 --2- 172.21.15.28:0/1417985806 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56537324c2a0 0x565373256430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:58.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.291+0000 7fa9f1a5c740 1 --2- 172.21.15.28:0/1417985806 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56537324b980 0x56537324bd60 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:58.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.291+0000 7fa9f1a5c740 1 -- 172.21.15.28:0/1417985806 >> 172.21.15.28:0/1417985806 conn(0x565373247ca0 msgr2=0x565373248080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:17:58.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.291+0000 7fa9f1a5c740 1 -- 172.21.15.28:0/1417985806 shutdown_connections 2024-05-12T22:17:58.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.291+0000 7fa9f1a5c740 1 -- 172.21.15.28:0/1417985806 wait complete. 2024-05-12T22:17:58.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.292+0000 7fa9f1a5c740 1 -- start start 2024-05-12T22:17:58.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.292+0000 7fa9f1a5c740 1 Processor -- start 2024-05-12T22:17:58.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.293+0000 7fa9f1a5c740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56537324b980 0x565373253950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:58.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.293+0000 7fa9f1a5c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56537324c2a0 0x565373253e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:58.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.293+0000 7fa9f1a5c740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x565373256970 0x5653732543d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:58.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.293+0000 7fa9f1a5c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x565373259760 con 0x56537324c2a0 2024-05-12T22:17:58.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.293+0000 7fa9f1a5c740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x56537316ba70 con 0x56537324b980 2024-05-12T22:17:58.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.293+0000 7fa9f1a5c740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5653732598e0 con 0x565373256970 2024-05-12T22:17:58.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.293+0000 7fa9f1a5c740 10 client.0 _ll_register_callbacks cb 0x56537314eba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:17:58.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.293+0000 7fa9e7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56537324c2a0 0x565373253e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:58.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.293+0000 7fa9e7e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56537324b980 0x565373253950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:58.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.293+0000 7fa9e7e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56537324b980 0x565373253950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:33420/0 (socket says 172.21.15.28:33420) 2024-05-12T22:17:58.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.293+0000 7fa9e7e00640 1 -- 172.21.15.28:0/2694148507 learned_addr learned my addr 172.21.15.28:0/2694148507 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:17:58.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.293+0000 7fa9ecc00640 1 --2- 172.21.15.28:0/2694148507 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x565373256970 0x5653732543d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:58.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.293+0000 7fa9e7400640 1 -- 172.21.15.28:0/2694148507 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x565373256970 msgr2=0x5653732543d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:17:58.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.293+0000 7fa9e7400640 1 --2- 172.21.15.28:0/2694148507 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x565373256970 0x5653732543d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:58.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.293+0000 7fa9e7400640 1 -- 172.21.15.28:0/2694148507 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56537324b980 msgr2=0x565373253950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:17:58.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.293+0000 7fa9e7400640 1 --2- 172.21.15.28:0/2694148507 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56537324b980 0x565373253950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:58.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.293+0000 7fa9e7400640 1 -- 172.21.15.28:0/2694148507 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565372f8e7d0 con 0x56537324c2a0 2024-05-12T22:17:58.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.293+0000 7fa9e7e00640 1 --2- 172.21.15.28:0/2694148507 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56537324b980 0x565373253950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:17:58.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.294+0000 7fa9e7400640 1 --2- 172.21.15.28:0/2694148507 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56537324c2a0 0x565373253e90 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fa9d40ec940 tx=0x7fa9d4009550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d391bb50baff089 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:17:58.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.294+0000 7fa9d1600640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-12T22:17:58.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.294+0000 7fa9d1600640 1 -- 172.21.15.28:0/2694148507 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9d40bb250 con 0x56537324c2a0 2024-05-12T22:17:58.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.294+0000 7fa9d1600640 1 -- 172.21.15.28:0/2694148507 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8513+0+0 (secure 0 0 0) 0x7fa9d40b7d80 con 0x56537324c2a0 2024-05-12T22:17:58.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.294+0000 7fa9f1a5c740 1 -- 172.21.15.28:0/2694148507 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa9e0004310 con 0x56537324c2a0 2024-05-12T22:17:58.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.294+0000 7fa9d1600640 1 -- 172.21.15.28:0/2694148507 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fa9d40bb070 con 0x56537324c2a0 2024-05-12T22:17:58.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.294+0000 7fa9f1a5c740 10 client.17058 fetch_fsmap learned FSMap version 26 2024-05-12T22:17:58.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.294+0000 7fa9f1a5c740 1 -- 172.21.15.28:0/2694148507 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x565373265a50 con 0x56537324c2a0 2024-05-12T22:17:58.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.295+0000 7fa9d1600640 1 -- 172.21.15.28:0/2694148507 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa9e0004310 con 0x56537324c2a0 2024-05-12T22:17:58.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.295+0000 7fa9d1600640 1 -- 172.21.15.28:0/2694148507 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fa9d403a070 con 0x56537324c2a0 2024-05-12T22:17:58.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.295+0000 7fa9f1a5c740 10 client.17058 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:17:58.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.295+0000 7fa9f1a5c740 10 client.17058 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-05-12T22:17:58.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.295+0000 7fa9f1a5c740 20 client.17058 populate_metadata read hostname 'smithi028' 2024-05-12T22:17:58.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.295+0000 7fa9f1a5c740 1 --2- 172.21.15.28:0/2694148507 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x565373266df0 0x565373269240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:58.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.295+0000 7fa9f1a5c740 4 client.17058 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-12T22:17:58.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.295+0000 7fa9f1a5c740 1 -- 172.21.15.28:0/2694148507 --> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5653731be160 con 0x565373266df0 2024-05-12T22:17:58.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.295+0000 7fa9e7e00640 1 --2- 172.21.15.28:0/2694148507 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x565373266df0 0x565373269240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:58.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.296+0000 7fa9e7e00640 1 --2- 172.21.15.28:0/2694148507 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x565373266df0 0x565373269240 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c75a3c33cce71e5c server_cookie=ea6f1ff9217f6b95 in_seq=0 out_seq=0 2024-05-12T22:17:58.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.296+0000 7fa9d1600640 10 client.17058 ms_handle_connect on v2:172.21.15.28:6840/761183870 2024-05-12T22:17:58.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.297+0000 7fa9d1600640 1 -- 172.21.15.28:0/2694148507 <== mds.1 v2:172.21.15.28:6840/761183870 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5653731be160 con 0x565373266df0 2024-05-12T22:17:58.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.297+0000 7fa9d1600640 10 client.17058 handle_command_reply: tid=0 2024-05-12T22:17:58.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.297+0000 7fa9f1a5c740 1 client.17058 shutdown 2024-05-12T22:17:58.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.297+0000 7fa9f1a5c740 1 -- 172.21.15.28:0/2694148507 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56537324c2a0 msgr2=0x565373253e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:17:58.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.297+0000 7fa9f1a5c740 1 --2- 172.21.15.28:0/2694148507 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56537324c2a0 0x565373253e90 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fa9d40ec940 tx=0x7fa9d4009550 comp rx=0 tx=0).stop 2024-05-12T22:17:58.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.297+0000 7fa9f1a5c740 1 -- 172.21.15.28:0/2694148507 shutdown_connections 2024-05-12T22:17:58.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.297+0000 7fa9f1a5c740 1 --2- 172.21.15.28:0/2694148507 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x565373266df0 0x565373269240 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:58.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.297+0000 7fa9f1a5c740 1 --2- 172.21.15.28:0/2694148507 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x565373256970 0x5653732543d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:58.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.297+0000 7fa9f1a5c740 1 --2- 172.21.15.28:0/2694148507 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56537324c2a0 0x565373253e90 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:58.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.297+0000 7fa9f1a5c740 1 --2- 172.21.15.28:0/2694148507 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56537324b980 0x565373253950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:58.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.297+0000 7fa9d1600640 0 client.17058 ms_handle_reset on v2:172.21.15.28:6840/761183870 2024-05-12T22:17:58.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.297+0000 7fa9f1a5c740 1 -- 172.21.15.28:0/2694148507 >> 172.21.15.28:0/2694148507 conn(0x565373247ca0 msgr2=0x56537324d9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:17:58.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.297+0000 7fa9f1a5c740 1 -- 172.21.15.28:0/2694148507 shutdown_connections 2024-05-12T22:17:58.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.297+0000 7fa9f1a5c740 1 -- 172.21.15.28:0/2694148507 wait complete. 2024-05-12T22:17:58.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.297+0000 7fa9f1a5c740 20 client.17058 trim_cache size 0 max 16384 2024-05-12T22:17:58.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.312+0000 7fa9f1a5c740 1 Processor -- start 2024-05-12T22:17:58.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.312+0000 7fa9f1a5c740 1 -- start start 2024-05-12T22:17:58.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.313+0000 7fa9f1a5c740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56537324b980 0x565373262620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:58.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.313+0000 7fa9f1a5c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56537324c2a0 0x56537324e120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:58.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.313+0000 7fa9f1a5c740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x565373256970 0x56537326eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:58.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.313+0000 7fa9f1a5c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa9e81b6520 con 0x56537324c2a0 2024-05-12T22:17:58.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.313+0000 7fa9f1a5c740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fa9e81b3df0 con 0x565373256970 2024-05-12T22:17:58.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.313+0000 7fa9f1a5c740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa9e81b1c60 con 0x56537324b980 2024-05-12T22:17:58.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.313+0000 7fa9e7e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56537324b980 0x565373262620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:58.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.313+0000 7fa9e7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56537324c2a0 0x56537324e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:58.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.313+0000 7fa9e7e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56537324b980 0x565373262620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:56486/0 (socket says 172.21.15.28:56486) 2024-05-12T22:17:58.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.313+0000 7fa9e7e00640 1 -- 172.21.15.28:0/1986074223 learned_addr learned my addr 172.21.15.28:0/1986074223 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:17:58.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.313+0000 7fa9ecc00640 1 --2- 172.21.15.28:0/1986074223 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x565373256970 0x56537326eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:58.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.313+0000 7fa9e7e00640 1 -- 172.21.15.28:0/1986074223 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x565373256970 msgr2=0x56537326eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:17:58.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.313+0000 7fa9e7e00640 1 --2- 172.21.15.28:0/1986074223 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x565373256970 0x56537326eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:58.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.314+0000 7fa9e7e00640 1 -- 172.21.15.28:0/1986074223 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56537324c2a0 msgr2=0x56537324e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:17:58.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.314+0000 7fa9e7e00640 1 --2- 172.21.15.28:0/1986074223 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56537324c2a0 0x56537324e120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:58.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.314+0000 7fa9e7e00640 1 -- 172.21.15.28:0/1986074223 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5653732698c0 con 0x56537324b980 2024-05-12T22:17:58.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.314+0000 7fa9e7400640 1 --2- 172.21.15.28:0/1986074223 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56537324c2a0 0x56537324e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:17:58.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.314+0000 7fa9ecc00640 1 --2- 172.21.15.28:0/1986074223 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x565373256970 0x56537326eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:17:58.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.314+0000 7fa9e7e00640 1 --2- 172.21.15.28:0/1986074223 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56537324b980 0x565373262620 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fa9d8005fd0 tx=0x7fa9d8016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d8c09341b308c5e7 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:17:58.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.314+0000 7fa9d3e00640 1 -- 172.21.15.28:0/1986074223 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9d8020040 con 0x56537324b980 2024-05-12T22:17:58.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.314+0000 7fa9d3e00640 1 -- 172.21.15.28:0/1986074223 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa9d802f030 con 0x56537324b980 2024-05-12T22:17:58.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.314+0000 7fa9d3e00640 1 -- 172.21.15.28:0/1986074223 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9d801a6b0 con 0x56537324b980 2024-05-12T22:17:58.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.315+0000 7fa9f1a5c740 1 -- 172.21.15.28:0/1986074223 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56537324b980 msgr2=0x565373262620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:17:58.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.315+0000 7fa9f1a5c740 1 --2- 172.21.15.28:0/1986074223 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56537324b980 0x565373262620 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fa9d8005fd0 tx=0x7fa9d8016000 comp rx=0 tx=0).stop 2024-05-12T22:17:58.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.315+0000 7fa9f1a5c740 1 -- 172.21.15.28:0/1986074223 shutdown_connections 2024-05-12T22:17:58.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.315+0000 7fa9f1a5c740 1 --2- 172.21.15.28:0/1986074223 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x565373256970 0x56537326eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:58.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.315+0000 7fa9f1a5c740 1 --2- 172.21.15.28:0/1986074223 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56537324c2a0 0x56537324e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:58.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.315+0000 7fa9f1a5c740 1 --2- 172.21.15.28:0/1986074223 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56537324b980 0x565373262620 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:58.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.315+0000 7fa9f1a5c740 1 -- 172.21.15.28:0/1986074223 >> 172.21.15.28:0/1986074223 conn(0x565373247ca0 msgr2=0x565373246dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:17:58.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.315+0000 7fa9f1a5c740 1 -- 172.21.15.28:0/1986074223 shutdown_connections 2024-05-12T22:17:58.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.315+0000 7fa9f1a5c740 1 -- 172.21.15.28:0/1986074223 wait complete. 2024-05-12T22:17:58.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.316+0000 7fa9f1a5c740 1 -- start start 2024-05-12T22:17:58.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.316+0000 7fa9f1a5c740 1 Processor -- start 2024-05-12T22:17:58.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.316+0000 7fa9f1a5c740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56537324b980 0x565373250cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:58.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.316+0000 7fa9f1a5c740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56537324c2a0 0x5653732511f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:58.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.316+0000 7fa9f1a5c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x565373256970 0x5653732cc3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:58.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.316+0000 7fa9f1a5c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa9e81b22b0 con 0x565373256970 2024-05-12T22:17:58.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.316+0000 7fa9f1a5c740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fa9e81b3720 con 0x56537324c2a0 2024-05-12T22:17:58.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.316+0000 7fa9f1a5c740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa9e81b2980 con 0x56537324b980 2024-05-12T22:17:58.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.316+0000 7fa9f1a5c740 10 client.0 _ll_register_callbacks cb 0x7fa9d40b72f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:17:58.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.317+0000 7fa9ecc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x565373256970 0x5653732cc3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:58.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.317+0000 7fa9e7400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56537324c2a0 0x5653732511f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:58.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.317+0000 7fa9ecc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x565373256970 0x5653732cc3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40122/0 (socket says 172.21.15.28:40122) 2024-05-12T22:17:58.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.317+0000 7fa9e7400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56537324c2a0 0x5653732511f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:33432/0 (socket says 172.21.15.28:33432) 2024-05-12T22:17:58.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.317+0000 7fa9ecc00640 1 -- 172.21.15.28:0/3472721693 learned_addr learned my addr 172.21.15.28:0/3472721693 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:17:58.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.317+0000 7fa9e7e00640 1 --2- 172.21.15.28:0/3472721693 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56537324b980 0x565373250cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:58.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.317+0000 7fa9e7400640 1 -- 172.21.15.28:0/3472721693 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56537324b980 msgr2=0x565373250cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:17:58.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.317+0000 7fa9e7400640 1 --2- 172.21.15.28:0/3472721693 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56537324b980 0x565373250cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:58.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.317+0000 7fa9e7400640 1 -- 172.21.15.28:0/3472721693 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x565373256970 msgr2=0x5653732cc3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:17:58.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.317+0000 7fa9e7400640 1 --2- 172.21.15.28:0/3472721693 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x565373256970 0x5653732cc3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:58.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.317+0000 7fa9e7400640 1 -- 172.21.15.28:0/3472721693 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5653732cc900 con 0x56537324c2a0 2024-05-12T22:17:58.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.317+0000 7fa9e7400640 1 --2- 172.21.15.28:0/3472721693 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56537324c2a0 0x5653732511f0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x565373254f20 tx=0x7fa9d40b9c20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=795c681f5dfffc56 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:17:58.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.317+0000 7fa9d1600640 10 client.0 ms_handle_connect on v2:172.21.15.86:3300/0 2024-05-12T22:17:58.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.317+0000 7fa9d1600640 1 -- 172.21.15.28:0/3472721693 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9d40bb250 con 0x56537324c2a0 2024-05-12T22:17:58.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.317+0000 7fa9f1a5c740 1 -- 172.21.15.28:0/3472721693 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa9d802f030 con 0x56537324c2a0 2024-05-12T22:17:58.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.317+0000 7fa9d1600640 1 -- 172.21.15.28:0/3472721693 <== mon.2 v2:172.21.15.86:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8513+0+0 (secure 0 0 0) 0x7fa9d40bad70 con 0x56537324c2a0 2024-05-12T22:17:58.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.318+0000 7fa9d1600640 1 -- 172.21.15.28:0/3472721693 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fa9d40bb710 con 0x56537324c2a0 2024-05-12T22:17:58.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.318+0000 7fa9f1a5c740 10 client.27197 fetch_fsmap learned FSMap version 26 2024-05-12T22:17:58.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.318+0000 7fa9f1a5c740 1 -- 172.21.15.28:0/3472721693 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5653732cc900 con 0x56537324c2a0 2024-05-12T22:17:58.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.318+0000 7fa9d1600640 1 -- 172.21.15.28:0/3472721693 <== mon.2 v2:172.21.15.86:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa9d802f030 con 0x56537324c2a0 2024-05-12T22:17:58.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.318+0000 7fa9d1600640 1 -- 172.21.15.28:0/3472721693 <== mon.2 v2:172.21.15.86:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fa9d403a040 con 0x56537324c2a0 2024-05-12T22:17:58.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.318+0000 7fa9f1a5c740 10 client.27197 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:17:58.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.318+0000 7fa9f1a5c740 10 client.27197 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-05-12T22:17:58.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.318+0000 7fa9f1a5c740 20 client.27197 populate_metadata read hostname 'smithi028' 2024-05-12T22:17:58.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.318+0000 7fa9f1a5c740 1 --2- 172.21.15.28:0/3472721693 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x5653732cdec0 0x5653732d02b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:58.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.318+0000 7fa9f1a5c740 4 client.27197 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-12T22:17:58.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.318+0000 7fa9f1a5c740 1 -- 172.21.15.28:0/3472721693 --> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5653731b84b0 con 0x5653732cdec0 2024-05-12T22:17:58.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.319+0000 7fa9e7e00640 1 --2- 172.21.15.28:0/3472721693 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x5653732cdec0 0x5653732d02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:58.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.319+0000 7fa9e7e00640 1 --2- 172.21.15.28:0/3472721693 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x5653732cdec0 0x5653732d02b0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=3fe7dbd81e861aef server_cookie=a1c25c283759cead in_seq=0 out_seq=0 2024-05-12T22:17:58.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.319+0000 7fa9d1600640 10 client.27197 ms_handle_connect on v2:172.21.15.28:6840/761183870 2024-05-12T22:17:58.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.319+0000 7fa9d1600640 1 -- 172.21.15.28:0/3472721693 <== mds.1 v2:172.21.15.28:6840/761183870 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5653731b84b0 con 0x5653732cdec0 2024-05-12T22:17:58.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.319+0000 7fa9d1600640 10 client.27197 handle_command_reply: tid=0 2024-05-12T22:17:58.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.319+0000 7fa9f1a5c740 1 client.27197 shutdown 2024-05-12T22:17:58.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.320+0000 7fa9f1a5c740 1 -- 172.21.15.28:0/3472721693 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56537324c2a0 msgr2=0x5653732511f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:17:58.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.320+0000 7fa9f1a5c740 1 --2- 172.21.15.28:0/3472721693 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56537324c2a0 0x5653732511f0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x565373254f20 tx=0x7fa9d40b9c20 comp rx=0 tx=0).stop 2024-05-12T22:17:58.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.320+0000 7fa9f1a5c740 1 -- 172.21.15.28:0/3472721693 shutdown_connections 2024-05-12T22:17:58.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.320+0000 7fa9f1a5c740 1 --2- 172.21.15.28:0/3472721693 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x5653732cdec0 0x5653732d02b0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:58.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.320+0000 7fa9f1a5c740 1 --2- 172.21.15.28:0/3472721693 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x565373256970 0x5653732cc3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:58.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.320+0000 7fa9f1a5c740 1 --2- 172.21.15.28:0/3472721693 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56537324c2a0 0x5653732511f0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:58.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.320+0000 7fa9d1600640 0 client.27197 ms_handle_reset on v2:172.21.15.28:6840/761183870 2024-05-12T22:17:58.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.320+0000 7fa9f1a5c740 1 --2- 172.21.15.28:0/3472721693 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56537324b980 0x565373250cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:58.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.320+0000 7fa9f1a5c740 1 -- 172.21.15.28:0/3472721693 >> 172.21.15.28:0/3472721693 conn(0x565373247ca0 msgr2=0x565373253910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:17:58.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.320+0000 7fa9f1a5c740 1 -- 172.21.15.28:0/3472721693 shutdown_connections 2024-05-12T22:17:58.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.320+0000 7fa9f1a5c740 1 -- 172.21.15.28:0/3472721693 wait complete. 2024-05-12T22:17:58.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.320+0000 7fa9f1a5c740 20 client.27197 trim_cache size 0 max 16384 2024-05-12T22:17:58.328 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-05-12T22:17:58.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.321+0000 7fa9eec00640 1 -- 172.21.15.28:0/2419990429 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fa9cc0a2d00 msgr2=0x7fa9cc0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:17:58.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.321+0000 7fa9eec00640 1 --2- 172.21.15.28:0/2419990429 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fa9cc0a2d00 0x7fa9cc0a51c0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x56537324cf40 tx=0x7fa9d8010040 comp rx=0 tx=0).stop 2024-05-12T22:17:58.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.321+0000 7fa9eec00640 1 -- 172.21.15.28:0/2419990429 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9e8076680 msgr2=0x7fa9e81a49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:17:58.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.321+0000 7fa9eec00640 1 --2- 172.21.15.28:0/2419990429 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9e8076680 0x7fa9e81a49a0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fa9d4033ae0 tx=0x7fa9d4033b10 comp rx=0 tx=0).stop 2024-05-12T22:17:58.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.321+0000 7fa9eec00640 1 -- 172.21.15.28:0/2419990429 shutdown_connections 2024-05-12T22:17:58.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.321+0000 7fa9eec00640 1 --2- 172.21.15.28:0/2419990429 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fa9cc0a2d00 0x7fa9cc0a51c0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:58.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.321+0000 7fa9eec00640 1 --2- 172.21.15.28:0/2419990429 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa9e81126b0 0x7fa9e819e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:58.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.321+0000 7fa9eec00640 1 --2- 172.21.15.28:0/2419990429 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9e8076680 0x7fa9e81a49a0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:58.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.321+0000 7fa9eec00640 1 --2- 172.21.15.28:0/2419990429 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa9e8075d60 0x7fa9e81a4460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:58.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.321+0000 7fa9eec00640 1 -- 172.21.15.28:0/2419990429 >> 172.21.15.28:0/2419990429 conn(0x7fa9e80ff4a0 msgr2=0x7fa9e8101890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:17:58.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.321+0000 7fa9eec00640 1 -- 172.21.15.28:0/2419990429 shutdown_connections 2024-05-12T22:17:58.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.321+0000 7fa9eec00640 1 -- 172.21.15.28:0/2419990429 wait complete. 2024-05-12T22:17:58.336 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:17:58 smithi028 ceph-mon[29847]: pgmap v276: 97 pgs: 97 active+clean; 91 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:17:58.336 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:17:58 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/789036336' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:17:58.339 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-05-12T22:17:58.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:17:58 smithi186 ceph-mon[34785]: pgmap v276: 97 pgs: 97 active+clean; 91 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:17:58.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:17:58 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/789036336' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:17:58.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:17:58 smithi086 ceph-mon[34297]: pgmap v276: 97 pgs: 97 active+clean; 91 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:17:58.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:17:58 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/789036336' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:17:58.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.558+0000 7f28eea00640 1 -- 172.21.15.28:0/2883265167 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f28e8102890 msgr2=0x7f28e8102c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:17:58.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.558+0000 7f28eea00640 1 --2- 172.21.15.28:0/2883265167 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f28e8102890 0x7f28e8102c70 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f28dc009900 tx=0x7f28dc02a490 comp rx=0 tx=0).stop 2024-05-12T22:17:58.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.559+0000 7f28eea00640 1 -- 172.21.15.28:0/2883265167 shutdown_connections 2024-05-12T22:17:58.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.559+0000 7f28eea00640 1 --2- 172.21.15.28:0/2883265167 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28e810bcb0 0x7f28e810e150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:58.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.559+0000 7f28eea00640 1 --2- 172.21.15.28:0/2883265167 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f28e81031b0 0x7f28e810b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:58.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.559+0000 7f28eea00640 1 --2- 172.21.15.28:0/2883265167 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f28e8102890 0x7f28e8102c70 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:58.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.559+0000 7f28eea00640 1 -- 172.21.15.28:0/2883265167 >> 172.21.15.28:0/2883265167 conn(0x7f28e80fc0b0 msgr2=0x7f28e80fe4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:17:58.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.559+0000 7f28eea00640 1 -- 172.21.15.28:0/2883265167 shutdown_connections 2024-05-12T22:17:58.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.559+0000 7f28eea00640 1 -- 172.21.15.28:0/2883265167 wait complete. 2024-05-12T22:17:58.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.560+0000 7f28eea00640 1 Processor -- start 2024-05-12T22:17:58.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.560+0000 7f28eea00640 1 -- start start 2024-05-12T22:17:58.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.560+0000 7f28eea00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f28e8102890 0x7f28e8103700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:58.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.560+0000 7f28eea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28e81031b0 0x7f28e8103c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:58.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.560+0000 7f28eea00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f28e810bcb0 0x7f28e8104180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:58.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.560+0000 7f28eea00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f28e8111b00 con 0x7f28e81031b0 2024-05-12T22:17:58.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.560+0000 7f28eea00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f28e8111980 con 0x7f28e810bcb0 2024-05-12T22:17:58.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.560+0000 7f28eea00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f28e8111c80 con 0x7f28e8102890 2024-05-12T22:17:58.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.560+0000 7f28ecc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28e81031b0 0x7f28e8103c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:58.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.560+0000 7f28ee000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f28e810bcb0 0x7f28e8104180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:58.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.561+0000 7f28ee000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f28e810bcb0 0x7f28e8104180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:33444/0 (socket says 172.21.15.28:33444) 2024-05-12T22:17:58.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.561+0000 7f28ee000640 1 -- 172.21.15.28:0/419863186 learned_addr learned my addr 172.21.15.28:0/419863186 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:17:58.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.561+0000 7f28ed600640 1 --2- 172.21.15.28:0/419863186 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f28e8102890 0x7f28e8103700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:58.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.561+0000 7f28ecc00640 1 -- 172.21.15.28:0/419863186 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f28e8102890 msgr2=0x7f28e8103700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:17:58.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.561+0000 7f28ecc00640 1 --2- 172.21.15.28:0/419863186 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f28e8102890 0x7f28e8103700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:58.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.561+0000 7f28ecc00640 1 -- 172.21.15.28:0/419863186 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f28e810bcb0 msgr2=0x7f28e8104180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:17:58.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.561+0000 7f28ecc00640 1 --2- 172.21.15.28:0/419863186 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f28e810bcb0 0x7f28e8104180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:58.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.561+0000 7f28ecc00640 1 -- 172.21.15.28:0/419863186 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28e81aa8d0 con 0x7f28e81031b0 2024-05-12T22:17:58.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.561+0000 7f28ecc00640 1 --2- 172.21.15.28:0/419863186 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28e81031b0 0x7f28e8103c40 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f28d400e7f0 tx=0x7f28d400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:17:58.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.562+0000 7f28e6000640 1 -- 172.21.15.28:0/419863186 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28d4009530 con 0x7f28e81031b0 2024-05-12T22:17:58.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.562+0000 7f28eea00640 1 -- 172.21.15.28:0/419863186 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28e81aabc0 con 0x7f28e81031b0 2024-05-12T22:17:58.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.562+0000 7f28e6000640 1 -- 172.21.15.28:0/419863186 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f28d400cc60 con 0x7f28e81031b0 2024-05-12T22:17:58.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.562+0000 7f28e6000640 1 -- 172.21.15.28:0/419863186 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28d4004a00 con 0x7f28e81031b0 2024-05-12T22:17:58.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.562+0000 7f28eea00640 1 -- 172.21.15.28:0/419863186 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28e81ab110 con 0x7f28e81031b0 2024-05-12T22:17:58.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.566+0000 7f28f2e1c740 1 Processor -- start 2024-05-12T22:17:58.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.566+0000 7f28f2e1c740 1 -- start start 2024-05-12T22:17:58.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.567+0000 7f28e6000640 1 -- 172.21.15.28:0/419863186 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7f28d4004cb0 con 0x7f28e81031b0 2024-05-12T22:17:58.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.567+0000 7f28f2e1c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f25b68b60 0x563f25b68f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:58.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.567+0000 7f28f2e1c740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563f25b69480 0x563f25b73610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:58.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.567+0000 7f28f2e1c740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563f25b73b50 0x563f25b75f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:58.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.567+0000 7f28f2e1c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x563f25acda30 con 0x563f25b68b60 2024-05-12T22:17:58.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.567+0000 7f28f2e1c740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x563f25a75700 con 0x563f25b73b50 2024-05-12T22:17:58.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.567+0000 7f28f2e1c740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x563f25ac99f0 con 0x563f25b69480 2024-05-12T22:17:58.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.567+0000 7f28ee000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f25b68b60 0x563f25b68f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:58.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.567+0000 7f28ee000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f25b68b60 0x563f25b68f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40152/0 (socket says 172.21.15.28:40152) 2024-05-12T22:17:58.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.567+0000 7f28ee000640 1 -- 172.21.15.28:0/4006865763 learned_addr learned my addr 172.21.15.28:0/4006865763 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:17:58.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.567+0000 7f28ed600640 1 --2- 172.21.15.28:0/4006865763 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563f25b69480 0x563f25b73610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:58.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.567+0000 7f28ecc00640 1 --2- 172.21.15.28:0/4006865763 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563f25b73b50 0x563f25b75f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:58.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.568+0000 7f28ee000640 1 -- 172.21.15.28:0/4006865763 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563f25b69480 msgr2=0x563f25b73610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:17:58.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.568+0000 7f28ee000640 1 --2- 172.21.15.28:0/4006865763 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563f25b69480 0x563f25b73610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:58.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.568+0000 7f28ee000640 1 -- 172.21.15.28:0/4006865763 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563f25b73b50 msgr2=0x563f25b75f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:17:58.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.568+0000 7f28ee000640 1 --2- 172.21.15.28:0/4006865763 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563f25b73b50 0x563f25b75f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:58.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.568+0000 7f28ee000640 1 -- 172.21.15.28:0/4006865763 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563f257e13f0 con 0x563f25b68b60 2024-05-12T22:17:58.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.568+0000 7f28ee000640 1 --2- 172.21.15.28:0/4006865763 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f25b68b60 0x563f25b68f40 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f28e0011470 tx=0x7f28e0011930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9395ca9a4ffc0317 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:17:58.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.568+0000 7f28da600640 1 -- 172.21.15.28:0/4006865763 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28e0004b20 con 0x563f25b68b60 2024-05-12T22:17:58.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.568+0000 7f28e6000640 1 --2- 172.21.15.28:0/419863186 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f28d00a2d00 0x7f28d00a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:58.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.568+0000 7f28da600640 1 -- 172.21.15.28:0/4006865763 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f28e0004cc0 con 0x563f25b68b60 2024-05-12T22:17:58.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.568+0000 7f28da600640 1 -- 172.21.15.28:0/4006865763 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28e00053a0 con 0x563f25b68b60 2024-05-12T22:17:58.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.569+0000 7f28e6000640 1 -- 172.21.15.28:0/419863186 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8513+0+0 (secure 0 0 0) 0x7f28d40cf3a0 con 0x7f28e81031b0 2024-05-12T22:17:58.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.569+0000 7f28ed600640 1 --2- 172.21.15.28:0/419863186 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f28d00a2d00 0x7f28d00a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:58.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.569+0000 7f28f2e1c740 1 -- 172.21.15.28:0/4006865763 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f25b68b60 msgr2=0x563f25b68f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:17:58.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.569+0000 7f28f2e1c740 1 --2- 172.21.15.28:0/4006865763 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f25b68b60 0x563f25b68f40 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f28e0011470 tx=0x7f28e0011930 comp rx=0 tx=0).stop 2024-05-12T22:17:58.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.570+0000 7f28f2e1c740 1 -- 172.21.15.28:0/4006865763 shutdown_connections 2024-05-12T22:17:58.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.570+0000 7f28f2e1c740 1 --2- 172.21.15.28:0/4006865763 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563f25b73b50 0x563f25b75f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:58.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.570+0000 7f28f2e1c740 1 --2- 172.21.15.28:0/4006865763 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563f25b69480 0x563f25b73610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:58.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.570+0000 7f28f2e1c740 1 --2- 172.21.15.28:0/4006865763 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f25b68b60 0x563f25b68f40 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:58.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.570+0000 7f28f2e1c740 1 -- 172.21.15.28:0/4006865763 >> 172.21.15.28:0/4006865763 conn(0x563f25b64e80 msgr2=0x563f25b65260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:17:58.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.570+0000 7f28ed600640 1 --2- 172.21.15.28:0/419863186 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f28d00a2d00 0x7f28d00a51c0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f28dc000c00 tx=0x7f28dc005b50 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:17:58.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.570+0000 7f28f2e1c740 1 -- 172.21.15.28:0/4006865763 shutdown_connections 2024-05-12T22:17:58.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.570+0000 7f28f2e1c740 1 -- 172.21.15.28:0/4006865763 wait complete. 2024-05-12T22:17:58.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.570+0000 7f28f2e1c740 1 -- start start 2024-05-12T22:17:58.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.570+0000 7f28f2e1c740 1 Processor -- start 2024-05-12T22:17:58.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.571+0000 7f28f2e1c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f25b68b60 0x563f25b70a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:58.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.571+0000 7f28f2e1c740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563f25b69480 0x563f25b70f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:58.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.571+0000 7f28f2e1c740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563f25b73b50 0x563f25b71490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:58.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.571+0000 7f28f2e1c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x563f25b76940 con 0x563f25b68b60 2024-05-12T22:17:58.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.571+0000 7f28f2e1c740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x563f25a88a70 con 0x563f25b73b50 2024-05-12T22:17:58.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.571+0000 7f28f2e1c740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x563f25b76ac0 con 0x563f25b69480 2024-05-12T22:17:58.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.571+0000 7f28f2e1c740 10 client.0 _ll_register_callbacks cb 0x563f25a6bba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:17:58.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.571+0000 7f28ed600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f25b68b60 0x563f25b70a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:58.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.571+0000 7f28ed600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f25b68b60 0x563f25b70a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40156/0 (socket says 172.21.15.28:40156) 2024-05-12T22:17:58.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.571+0000 7f28ed600640 1 -- 172.21.15.28:0/3863366191 learned_addr learned my addr 172.21.15.28:0/3863366191 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:17:58.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.571+0000 7f28ecc00640 1 --2- 172.21.15.28:0/3863366191 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563f25b69480 0x563f25b70f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:58.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.571+0000 7f28ee000640 1 --2- 172.21.15.28:0/3863366191 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563f25b73b50 0x563f25b71490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:58.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.572+0000 7f28ed600640 1 -- 172.21.15.28:0/3863366191 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563f25b69480 msgr2=0x563f25b70f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:17:58.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.572+0000 7f28ed600640 1 --2- 172.21.15.28:0/3863366191 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563f25b69480 0x563f25b70f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:58.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.572+0000 7f28ed600640 1 -- 172.21.15.28:0/3863366191 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563f25b73b50 msgr2=0x563f25b71490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:17:58.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.572+0000 7f28ed600640 1 --2- 172.21.15.28:0/3863366191 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563f25b73b50 0x563f25b71490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:58.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.572+0000 7f28ed600640 1 -- 172.21.15.28:0/3863366191 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563f258ab7d0 con 0x563f25b68b60 2024-05-12T22:17:58.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.572+0000 7f28ed600640 1 --2- 172.21.15.28:0/3863366191 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f25b68b60 0x563f25b70a10 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f28dc033000 tx=0x7f28dc009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2533047cdc0baaf1 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:17:58.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.572+0000 7f28cea00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-12T22:17:58.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.572+0000 7f28cea00640 1 -- 172.21.15.28:0/3863366191 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28dc04e070 con 0x563f25b68b60 2024-05-12T22:17:58.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.572+0000 7f28cea00640 1 -- 172.21.15.28:0/3863366191 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8513+0+0 (secure 0 0 0) 0x7f28dc004cd0 con 0x563f25b68b60 2024-05-12T22:17:58.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.572+0000 7f28f2e1c740 1 -- 172.21.15.28:0/3863366191 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f28e0004cc0 con 0x563f25b68b60 2024-05-12T22:17:58.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.573+0000 7f28cea00640 1 -- 172.21.15.28:0/3863366191 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f28dc05a050 con 0x563f25b68b60 2024-05-12T22:17:58.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.573+0000 7f28f2e1c740 10 client.17076 fetch_fsmap learned FSMap version 26 2024-05-12T22:17:58.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.573+0000 7f28f2e1c740 1 -- 172.21.15.28:0/3863366191 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x563f25b82b10 con 0x563f25b68b60 2024-05-12T22:17:58.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.573+0000 7f28cea00640 1 -- 172.21.15.28:0/3863366191 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f28e0004cc0 con 0x563f25b68b60 2024-05-12T22:17:58.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.573+0000 7f28cea00640 1 -- 172.21.15.28:0/3863366191 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f28dc003790 con 0x563f25b68b60 2024-05-12T22:17:58.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.573+0000 7f28f2e1c740 10 client.17076 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:17:58.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.573+0000 7f28f2e1c740 10 client.17076 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:17:58.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.573+0000 7f28f2e1c740 20 client.17076 populate_metadata read hostname 'smithi028' 2024-05-12T22:17:58.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.573+0000 7f28f2e1c740 1 --2- 172.21.15.28:0/3863366191 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x563f25b83f70 0x563f25b863a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:58.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.573+0000 7f28f2e1c740 4 client.17076 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-12T22:17:58.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.573+0000 7f28f2e1c740 1 -- 172.21.15.28:0/3863366191 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563f25adb160 con 0x563f25b83f70 2024-05-12T22:17:58.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.574+0000 7f28ecc00640 1 --2- 172.21.15.28:0/3863366191 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x563f25b83f70 0x563f25b863a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:58.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.574+0000 7f28ecc00640 1 --2- 172.21.15.28:0/3863366191 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x563f25b83f70 0x563f25b863a0 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=234eb6fcfc6503a6 server_cookie=9603a5973accc9b7 in_seq=0 out_seq=0 2024-05-12T22:17:58.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.574+0000 7f28cea00640 10 client.17076 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:17:58.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.575+0000 7f28cea00640 1 -- 172.21.15.28:0/3863366191 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563f25adb160 con 0x563f25b83f70 2024-05-12T22:17:58.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.575+0000 7f28cea00640 10 client.17076 handle_command_reply: tid=0 2024-05-12T22:17:58.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.575+0000 7f28f2e1c740 1 client.17076 shutdown 2024-05-12T22:17:58.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.575+0000 7f28f2e1c740 1 -- 172.21.15.28:0/3863366191 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f25b68b60 msgr2=0x563f25b70a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:17:58.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.575+0000 7f28f2e1c740 1 --2- 172.21.15.28:0/3863366191 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f25b68b60 0x563f25b70a10 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f28dc033000 tx=0x7f28dc009260 comp rx=0 tx=0).stop 2024-05-12T22:17:58.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.576+0000 7f28f2e1c740 1 -- 172.21.15.28:0/3863366191 shutdown_connections 2024-05-12T22:17:58.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.576+0000 7f28f2e1c740 1 --2- 172.21.15.28:0/3863366191 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x563f25b83f70 0x563f25b863a0 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:58.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.576+0000 7f28f2e1c740 1 --2- 172.21.15.28:0/3863366191 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563f25b73b50 0x563f25b71490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:58.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.576+0000 7f28f2e1c740 1 --2- 172.21.15.28:0/3863366191 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563f25b69480 0x563f25b70f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:58.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.576+0000 7f28f2e1c740 1 --2- 172.21.15.28:0/3863366191 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f25b68b60 0x563f25b70a10 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:58.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.576+0000 7f28f2e1c740 1 -- 172.21.15.28:0/3863366191 >> 172.21.15.28:0/3863366191 conn(0x563f25b64e80 msgr2=0x563f25b6ac80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:17:58.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.576+0000 7f28cea00640 0 client.17076 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:17:58.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.576+0000 7f28f2e1c740 1 -- 172.21.15.28:0/3863366191 shutdown_connections 2024-05-12T22:17:58.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.576+0000 7f28f2e1c740 1 -- 172.21.15.28:0/3863366191 wait complete. 2024-05-12T22:17:58.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.576+0000 7f28f2e1c740 20 client.17076 trim_cache size 0 max 16384 2024-05-12T22:17:58.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.591+0000 7f28f2e1c740 1 Processor -- start 2024-05-12T22:17:58.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.591+0000 7f28f2e1c740 1 -- start start 2024-05-12T22:17:58.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.591+0000 7f28f2e1c740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563f25b68b60 0x563f25b7f800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:58.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.591+0000 7f28f2e1c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f25b69480 0x563f25b702f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:58.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.591+0000 7f28f2e1c740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563f25b73b50 0x563f25b8bc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:58.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.591+0000 7f28f2e1c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f28e81b61e0 con 0x563f25b69480 2024-05-12T22:17:58.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.591+0000 7f28f2e1c740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f28e81b3ab0 con 0x563f25b73b50 2024-05-12T22:17:58.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.591+0000 7f28f2e1c740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f28e81b1920 con 0x563f25b68b60 2024-05-12T22:17:58.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.592+0000 7f28ecc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f25b69480 0x563f25b702f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:58.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.592+0000 7f28ee000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563f25b73b50 0x563f25b8bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:58.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.592+0000 7f28ed600640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563f25b68b60 0x563f25b7f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:58.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.592+0000 7f28ee000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563f25b73b50 0x563f25b8bc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:33468/0 (socket says 172.21.15.28:33468) 2024-05-12T22:17:58.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.592+0000 7f28ee000640 1 -- 172.21.15.28:0/3093856959 learned_addr learned my addr 172.21.15.28:0/3093856959 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:17:58.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.592+0000 7f28ee000640 1 -- 172.21.15.28:0/3093856959 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563f25b68b60 msgr2=0x563f25b7f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:17:58.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.592+0000 7f28ee000640 1 --2- 172.21.15.28:0/3093856959 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563f25b68b60 0x563f25b7f800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:58.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.592+0000 7f28ee000640 1 -- 172.21.15.28:0/3093856959 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f25b69480 msgr2=0x563f25b702f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:17:58.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.592+0000 7f28ee000640 1 --2- 172.21.15.28:0/3093856959 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f25b69480 0x563f25b702f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:58.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.592+0000 7f28ee000640 1 -- 172.21.15.28:0/3093856959 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563f25b692b0 con 0x563f25b73b50 2024-05-12T22:17:58.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.592+0000 7f28ee000640 1 --2- 172.21.15.28:0/3093856959 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563f25b73b50 0x563f25b8bc70 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f28e0000f80 tx=0x7f28e0004b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=316b830f4d2586f9 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:17:58.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.593+0000 7f28d9c00640 1 -- 172.21.15.28:0/3093856959 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28e0027070 con 0x563f25b73b50 2024-05-12T22:17:58.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.593+0000 7f28d9c00640 1 -- 172.21.15.28:0/3093856959 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f28e000fa20 con 0x563f25b73b50 2024-05-12T22:17:58.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.593+0000 7f28d9c00640 1 -- 172.21.15.28:0/3093856959 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28e0022af0 con 0x563f25b73b50 2024-05-12T22:17:58.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.594+0000 7f28f2e1c740 1 -- 172.21.15.28:0/3093856959 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563f25b73b50 msgr2=0x563f25b8bc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:17:58.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.594+0000 7f28f2e1c740 1 --2- 172.21.15.28:0/3093856959 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563f25b73b50 0x563f25b8bc70 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f28e0000f80 tx=0x7f28e0004b90 comp rx=0 tx=0).stop 2024-05-12T22:17:58.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.594+0000 7f28f2e1c740 1 -- 172.21.15.28:0/3093856959 shutdown_connections 2024-05-12T22:17:58.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.594+0000 7f28f2e1c740 1 --2- 172.21.15.28:0/3093856959 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563f25b73b50 0x563f25b8bc70 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:58.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.594+0000 7f28f2e1c740 1 --2- 172.21.15.28:0/3093856959 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f25b69480 0x563f25b702f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:58.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.594+0000 7f28f2e1c740 1 --2- 172.21.15.28:0/3093856959 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563f25b68b60 0x563f25b7f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:58.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.594+0000 7f28f2e1c740 1 -- 172.21.15.28:0/3093856959 >> 172.21.15.28:0/3093856959 conn(0x563f25b64e80 msgr2=0x563f25b64000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:17:58.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.594+0000 7f28f2e1c740 1 -- 172.21.15.28:0/3093856959 shutdown_connections 2024-05-12T22:17:58.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.594+0000 7f28f2e1c740 1 -- 172.21.15.28:0/3093856959 wait complete. 2024-05-12T22:17:58.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.594+0000 7f28f2e1c740 1 -- start start 2024-05-12T22:17:58.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.595+0000 7f28f2e1c740 1 Processor -- start 2024-05-12T22:17:58.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.595+0000 7f28f2e1c740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563f25b68b60 0x563f25b6d640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:58.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.595+0000 7f28f2e1c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f25b69480 0x563f25b6db80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:58.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.595+0000 7f28f2e1c740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563f25b73b50 0x563f25b6e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:58.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.595+0000 7f28f2e1c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f28e81b1f70 con 0x563f25b69480 2024-05-12T22:17:58.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.595+0000 7f28f2e1c740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f28e81b33e0 con 0x563f25b73b50 2024-05-12T22:17:58.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.595+0000 7f28f2e1c740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f28e81b2640 con 0x563f25b68b60 2024-05-12T22:17:58.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.595+0000 7f28f2e1c740 10 client.0 _ll_register_callbacks cb 0x7f28dc004380 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:17:58.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.595+0000 7f28ed600640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563f25b68b60 0x563f25b6d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:58.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.595+0000 7f28ecc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f25b69480 0x563f25b6db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:58.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.595+0000 7f28ed600640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563f25b68b60 0x563f25b6d640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:56532/0 (socket says 172.21.15.28:56532) 2024-05-12T22:17:58.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.595+0000 7f28ed600640 1 -- 172.21.15.28:0/2236182145 learned_addr learned my addr 172.21.15.28:0/2236182145 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:17:58.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.595+0000 7f28ee000640 1 --2- 172.21.15.28:0/2236182145 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563f25b73b50 0x563f25b6e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:58.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.596+0000 7f28ed600640 1 -- 172.21.15.28:0/2236182145 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563f25b73b50 msgr2=0x563f25b6e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:17:58.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.596+0000 7f28ed600640 1 --2- 172.21.15.28:0/2236182145 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563f25b73b50 0x563f25b6e0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:58.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.596+0000 7f28ed600640 1 -- 172.21.15.28:0/2236182145 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f25b69480 msgr2=0x563f25b6db80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:17:58.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.596+0000 7f28ed600640 1 --2- 172.21.15.28:0/2236182145 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f25b69480 0x563f25b6db80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:58.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.596+0000 7f28ed600640 1 -- 172.21.15.28:0/2236182145 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563f25be94e0 con 0x563f25b68b60 2024-05-12T22:17:58.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.596+0000 7f28ed600640 1 --2- 172.21.15.28:0/2236182145 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563f25b68b60 0x563f25b6d640 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f28dc003b60 tx=0x7f28dc005960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=57824756f62b8482 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:17:58.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.596+0000 7f28cea00640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-12T22:17:58.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.596+0000 7f28cea00640 1 -- 172.21.15.28:0/2236182145 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28dc062070 con 0x563f25b68b60 2024-05-12T22:17:58.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.596+0000 7f28f2e1c740 1 -- 172.21.15.28:0/2236182145 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f28e000fa20 con 0x563f25b68b60 2024-05-12T22:17:58.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.596+0000 7f28cea00640 1 -- 172.21.15.28:0/2236182145 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8513+0+0 (secure 0 0 0) 0x7f28dc0037f0 con 0x563f25b68b60 2024-05-12T22:17:58.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.597+0000 7f28cea00640 1 -- 172.21.15.28:0/2236182145 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f28dc06e050 con 0x563f25b68b60 2024-05-12T22:17:58.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.597+0000 7f28f2e1c740 10 client.27169 fetch_fsmap learned FSMap version 26 2024-05-12T22:17:58.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.597+0000 7f28f2e1c740 1 -- 172.21.15.28:0/2236182145 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x563f25bea740 con 0x563f25b68b60 2024-05-12T22:17:58.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.597+0000 7f28cea00640 1 -- 172.21.15.28:0/2236182145 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f28e000fa20 con 0x563f25b68b60 2024-05-12T22:17:58.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.597+0000 7f28cea00640 1 -- 172.21.15.28:0/2236182145 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f28dc032be0 con 0x563f25b68b60 2024-05-12T22:17:58.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.597+0000 7f28f2e1c740 10 client.27169 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:17:58.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.597+0000 7f28f2e1c740 10 client.27169 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:17:58.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.597+0000 7f28f2e1c740 20 client.27169 populate_metadata read hostname 'smithi028' 2024-05-12T22:17:58.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.597+0000 7f28f2e1c740 1 --2- 172.21.15.28:0/2236182145 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x563f25beae60 0x563f25bed280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:58.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.597+0000 7f28f2e1c740 4 client.27169 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-12T22:17:58.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.597+0000 7f28f2e1c740 1 -- 172.21.15.28:0/2236182145 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563f25ad54b0 con 0x563f25beae60 2024-05-12T22:17:58.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.598+0000 7f28ecc00640 1 --2- 172.21.15.28:0/2236182145 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x563f25beae60 0x563f25bed280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:58.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.598+0000 7f28ecc00640 1 --2- 172.21.15.28:0/2236182145 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x563f25beae60 0x563f25bed280 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d0487ddecd60e599 server_cookie=f82846145787a3f in_seq=0 out_seq=0 2024-05-12T22:17:58.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.598+0000 7f28cea00640 10 client.27169 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:17:58.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.599+0000 7f28cea00640 1 -- 172.21.15.28:0/2236182145 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563f25ad54b0 con 0x563f25beae60 2024-05-12T22:17:58.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.599+0000 7f28cea00640 10 client.27169 handle_command_reply: tid=0 2024-05-12T22:17:58.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.599+0000 7f28f2e1c740 1 client.27169 shutdown 2024-05-12T22:17:58.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.599+0000 7f28f2e1c740 1 -- 172.21.15.28:0/2236182145 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563f25b68b60 msgr2=0x563f25b6d640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:17:58.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.599+0000 7f28f2e1c740 1 --2- 172.21.15.28:0/2236182145 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563f25b68b60 0x563f25b6d640 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f28dc003b60 tx=0x7f28dc005960 comp rx=0 tx=0).stop 2024-05-12T22:17:58.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.599+0000 7f28f2e1c740 1 -- 172.21.15.28:0/2236182145 shutdown_connections 2024-05-12T22:17:58.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.599+0000 7f28f2e1c740 1 --2- 172.21.15.28:0/2236182145 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x563f25beae60 0x563f25bed280 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:58.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.599+0000 7f28f2e1c740 1 --2- 172.21.15.28:0/2236182145 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563f25b73b50 0x563f25b6e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:58.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.599+0000 7f28f2e1c740 1 --2- 172.21.15.28:0/2236182145 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f25b69480 0x563f25b6db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:58.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.599+0000 7f28cea00640 0 client.27169 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:17:58.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.599+0000 7f28f2e1c740 1 --2- 172.21.15.28:0/2236182145 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563f25b68b60 0x563f25b6d640 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:58.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.599+0000 7f28f2e1c740 1 -- 172.21.15.28:0/2236182145 >> 172.21.15.28:0/2236182145 conn(0x563f25b64e80 msgr2=0x563f25b72240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:17:58.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.599+0000 7f28f2e1c740 1 -- 172.21.15.28:0/2236182145 shutdown_connections 2024-05-12T22:17:58.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.599+0000 7f28f2e1c740 1 -- 172.21.15.28:0/2236182145 wait complete. 2024-05-12T22:17:58.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.599+0000 7f28f2e1c740 20 client.27169 trim_cache size 0 max 16384 2024-05-12T22:17:58.605 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-05-12T22:17:58.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.600+0000 7f28eea00640 1 -- 172.21.15.28:0/419863186 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f28d00a2d00 msgr2=0x7f28d00a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:17:58.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.600+0000 7f28eea00640 1 --2- 172.21.15.28:0/419863186 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f28d00a2d00 0x7f28d00a51c0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f28dc000c00 tx=0x7f28dc005b50 comp rx=0 tx=0).stop 2024-05-12T22:17:58.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.600+0000 7f28eea00640 1 -- 172.21.15.28:0/419863186 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28e81031b0 msgr2=0x7f28e8103c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:17:58.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.600+0000 7f28eea00640 1 --2- 172.21.15.28:0/419863186 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28e81031b0 0x7f28e8103c40 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f28d400e7f0 tx=0x7f28d400ecb0 comp rx=0 tx=0).stop 2024-05-12T22:17:58.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.600+0000 7f28eea00640 1 -- 172.21.15.28:0/419863186 shutdown_connections 2024-05-12T22:17:58.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.600+0000 7f28eea00640 1 --2- 172.21.15.28:0/419863186 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f28d00a2d00 0x7f28d00a51c0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:58.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.600+0000 7f28eea00640 1 --2- 172.21.15.28:0/419863186 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f28e810bcb0 0x7f28e8104180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:58.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.600+0000 7f28eea00640 1 --2- 172.21.15.28:0/419863186 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28e81031b0 0x7f28e8103c40 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:58.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.600+0000 7f28eea00640 1 --2- 172.21.15.28:0/419863186 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f28e8102890 0x7f28e8103700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:58.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.600+0000 7f28eea00640 1 -- 172.21.15.28:0/419863186 >> 172.21.15.28:0/419863186 conn(0x7f28e80fc0b0 msgr2=0x7f28e81004b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:17:58.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.600+0000 7f28eea00640 1 -- 172.21.15.28:0/419863186 shutdown_connections 2024-05-12T22:17:58.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:58.600+0000 7f28eea00640 1 -- 172.21.15.28:0/419863186 wait complete. 2024-05-12T22:17:59.619 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-12T22:17:59.620 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-12T22:17:59.620 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-05-12T22:17:59.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.851+0000 7fd6c9000640 1 -- 172.21.15.28:0/3950320772 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd6c4107dd0 msgr2=0x7fd6c4108230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:17:59.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.851+0000 7fd6c9000640 1 --2- 172.21.15.28:0/3950320772 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd6c4107dd0 0x7fd6c4108230 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fd6b0009900 tx=0x7fd6b002a480 comp rx=0 tx=0).stop 2024-05-12T22:17:59.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.852+0000 7fd6c9000640 1 -- 172.21.15.28:0/3950320772 shutdown_connections 2024-05-12T22:17:59.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.852+0000 7fd6c9000640 1 --2- 172.21.15.28:0/3950320772 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd6c4108770 0x7fd6c4114b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:59.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.852+0000 7fd6c9000640 1 --2- 172.21.15.28:0/3950320772 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd6c4107dd0 0x7fd6c4108230 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:59.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.852+0000 7fd6c9000640 1 --2- 172.21.15.28:0/3950320772 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6c410ddf0 0x7fd6c410e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:59.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.852+0000 7fd6c9000640 1 -- 172.21.15.28:0/3950320772 >> 172.21.15.28:0/3950320772 conn(0x7fd6c4077650 msgr2=0x7fd6c4106650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:17:59.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.852+0000 7fd6c9000640 1 -- 172.21.15.28:0/3950320772 shutdown_connections 2024-05-12T22:17:59.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.852+0000 7fd6c9000640 1 -- 172.21.15.28:0/3950320772 wait complete. 2024-05-12T22:17:59.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.853+0000 7fd6c9000640 1 Processor -- start 2024-05-12T22:17:59.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.853+0000 7fd6c9000640 1 -- start start 2024-05-12T22:17:59.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.853+0000 7fd6c9000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd6c4107dd0 0x7fd6c419e510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:59.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.853+0000 7fd6c9000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6c4108770 0x7fd6c419ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:59.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.854+0000 7fd6c9000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd6c410ddf0 0x7fd6c41a2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:59.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.854+0000 7fd6c9000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd6c41180d0 con 0x7fd6c4108770 2024-05-12T22:17:59.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.854+0000 7fd6c9000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fd6c4117f50 con 0x7fd6c4107dd0 2024-05-12T22:17:59.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.854+0000 7fd6c9000640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd6c4118250 con 0x7fd6c410ddf0 2024-05-12T22:17:59.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.854+0000 7fd6c2a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6c4108770 0x7fd6c419ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:59.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.854+0000 7fd6c2a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6c4108770 0x7fd6c419ea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40194/0 (socket says 172.21.15.28:40194) 2024-05-12T22:17:59.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.854+0000 7fd6c2a00640 1 -- 172.21.15.28:0/1456790867 learned_addr learned my addr 172.21.15.28:0/1456790867 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:17:59.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.854+0000 7fd6c3400640 1 --2- 172.21.15.28:0/1456790867 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd6c4107dd0 0x7fd6c419e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:59.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.854+0000 7fd6c3e00640 1 --2- 172.21.15.28:0/1456790867 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd6c410ddf0 0x7fd6c41a2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:59.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.854+0000 7fd6c2a00640 1 -- 172.21.15.28:0/1456790867 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd6c410ddf0 msgr2=0x7fd6c41a2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:17:59.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.854+0000 7fd6c2a00640 1 --2- 172.21.15.28:0/1456790867 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd6c410ddf0 0x7fd6c41a2dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:59.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.854+0000 7fd6c2a00640 1 -- 172.21.15.28:0/1456790867 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd6c4107dd0 msgr2=0x7fd6c419e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:17:59.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.854+0000 7fd6c2a00640 1 --2- 172.21.15.28:0/1456790867 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd6c4107dd0 0x7fd6c419e510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:59.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.854+0000 7fd6c2a00640 1 -- 172.21.15.28:0/1456790867 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6c41a3500 con 0x7fd6c4108770 2024-05-12T22:17:59.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.854+0000 7fd6c3400640 1 --2- 172.21.15.28:0/1456790867 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd6c4107dd0 0x7fd6c419e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:17:59.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.855+0000 7fd6c2a00640 1 --2- 172.21.15.28:0/1456790867 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6c4108770 0x7fd6c419ea50 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fd6b0006010 tx=0x7fd6b0004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:17:59.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.855+0000 7fd6bbe00640 1 -- 172.21.15.28:0/1456790867 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6b003f030 con 0x7fd6c4108770 2024-05-12T22:17:59.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.855+0000 7fd6c9000640 1 -- 172.21.15.28:0/1456790867 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6c41a3790 con 0x7fd6c4108770 2024-05-12T22:17:59.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.855+0000 7fd6bbe00640 1 -- 172.21.15.28:0/1456790867 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd6b0004440 con 0x7fd6c4108770 2024-05-12T22:17:59.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.855+0000 7fd6bbe00640 1 -- 172.21.15.28:0/1456790867 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6b0035070 con 0x7fd6c4108770 2024-05-12T22:17:59.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.855+0000 7fd6c9000640 1 -- 172.21.15.28:0/1456790867 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6c41ab080 con 0x7fd6c4108770 2024-05-12T22:17:59.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.859+0000 7fd6bbe00640 1 -- 172.21.15.28:0/1456790867 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7fd6b0042030 con 0x7fd6c4108770 2024-05-12T22:17:59.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.859+0000 7fd6cd45d740 1 Processor -- start 2024-05-12T22:17:59.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.859+0000 7fd6cd45d740 1 -- start start 2024-05-12T22:17:59.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.859+0000 7fd6cd45d740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e5be9fa500 0x55e5be9fa8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:59.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.859+0000 7fd6cd45d740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e5be9faeb0 0x55e5bea04f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:59.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.859+0000 7fd6cd45d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e5bea05520 0x55e5bea07970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:59.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.859+0000 7fd6cd45d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e5be95e7c0 con 0x55e5bea05520 2024-05-12T22:17:59.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.859+0000 7fd6cd45d740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x55e5be906790 con 0x55e5be9faeb0 2024-05-12T22:17:59.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.859+0000 7fd6cd45d740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55e5be95a780 con 0x55e5be9fa500 2024-05-12T22:17:59.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.860+0000 7fd6c3400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e5be9faeb0 0x55e5bea04f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:59.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.860+0000 7fd6c2a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e5bea05520 0x55e5bea07970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:59.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.860+0000 7fd6c3400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e5be9faeb0 0x55e5bea04f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:33516/0 (socket says 172.21.15.28:33516) 2024-05-12T22:17:59.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.860+0000 7fd6c3e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e5be9fa500 0x55e5be9fa8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:59.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.860+0000 7fd6c3400640 1 -- 172.21.15.28:0/3966841363 learned_addr learned my addr 172.21.15.28:0/3966841363 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:17:59.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.860+0000 7fd6c3e00640 1 -- 172.21.15.28:0/3966841363 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e5be9faeb0 msgr2=0x55e5bea04f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:17:59.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.860+0000 7fd6c3e00640 1 --2- 172.21.15.28:0/3966841363 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e5be9faeb0 0x55e5bea04f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:59.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.860+0000 7fd6c3e00640 1 -- 172.21.15.28:0/3966841363 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e5bea05520 msgr2=0x55e5bea07970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:17:59.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.860+0000 7fd6c3e00640 1 --2- 172.21.15.28:0/3966841363 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e5bea05520 0x55e5bea07970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:59.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.860+0000 7fd6c3e00640 1 -- 172.21.15.28:0/3966841363 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e5be9266a0 con 0x55e5be9fa500 2024-05-12T22:17:59.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.860+0000 7fd6c3400640 1 --2- 172.21.15.28:0/3966841363 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e5be9faeb0 0x55e5bea04f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:17:59.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.861+0000 7fd6c3e00640 1 --2- 172.21.15.28:0/3966841363 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e5be9fa500 0x55e5be9fa8e0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fd6bc009b70 tx=0x7fd6bc011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=67102c75116e6ac8 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:17:59.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.861+0000 7fd6bbe00640 1 --2- 172.21.15.28:0/1456790867 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fd6a80a2d80 0x7fd6a80a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:59.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.861+0000 7fd6bbe00640 1 -- 172.21.15.28:0/1456790867 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8513+0+0 (secure 0 0 0) 0x7fd6b00ef770 con 0x7fd6c4108770 2024-05-12T22:17:59.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.861+0000 7fd6b8c00640 1 -- 172.21.15.28:0/3966841363 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6bc004030 con 0x55e5be9fa500 2024-05-12T22:17:59.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.861+0000 7fd6c3400640 1 --2- 172.21.15.28:0/1456790867 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fd6a80a2d80 0x7fd6a80a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:59.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.861+0000 7fd6b8c00640 1 -- 172.21.15.28:0/3966841363 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd6bc005500 con 0x55e5be9fa500 2024-05-12T22:17:59.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.861+0000 7fd6b8c00640 1 -- 172.21.15.28:0/3966841363 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6bc004cb0 con 0x55e5be9fa500 2024-05-12T22:17:59.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.862+0000 7fd6cd45d740 1 -- 172.21.15.28:0/3966841363 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e5be9fa500 msgr2=0x55e5be9fa8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:17:59.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.862+0000 7fd6cd45d740 1 --2- 172.21.15.28:0/3966841363 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e5be9fa500 0x55e5be9fa8e0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fd6bc009b70 tx=0x7fd6bc011810 comp rx=0 tx=0).stop 2024-05-12T22:17:59.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.862+0000 7fd6c3400640 1 --2- 172.21.15.28:0/1456790867 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fd6a80a2d80 0x7fd6a80a5240 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x55e5be9f7830 tx=0x7fd6b4009500 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:17:59.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.862+0000 7fd6cd45d740 1 -- 172.21.15.28:0/3966841363 shutdown_connections 2024-05-12T22:17:59.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.862+0000 7fd6cd45d740 1 --2- 172.21.15.28:0/3966841363 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e5bea05520 0x55e5bea07970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:59.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.862+0000 7fd6cd45d740 1 --2- 172.21.15.28:0/3966841363 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e5be9faeb0 0x55e5bea04f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:59.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.862+0000 7fd6cd45d740 1 --2- 172.21.15.28:0/3966841363 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e5be9fa500 0x55e5be9fa8e0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:59.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.862+0000 7fd6cd45d740 1 -- 172.21.15.28:0/3966841363 >> 172.21.15.28:0/3966841363 conn(0x55e5be9f65b0 msgr2=0x55e5be9f6990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:17:59.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.862+0000 7fd6cd45d740 1 -- 172.21.15.28:0/3966841363 shutdown_connections 2024-05-12T22:17:59.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.862+0000 7fd6cd45d740 1 -- 172.21.15.28:0/3966841363 wait complete. 2024-05-12T22:17:59.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.862+0000 7fd6cd45d740 1 -- start start 2024-05-12T22:17:59.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.863+0000 7fd6cd45d740 1 Processor -- start 2024-05-12T22:17:59.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.863+0000 7fd6cd45d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e5be9fa500 0x55e5bea02370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:59.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.863+0000 7fd6cd45d740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e5be9faeb0 0x55e5bea028b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:59.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.863+0000 7fd6cd45d740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e5bea05520 0x55e5bea02df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:59.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.863+0000 7fd6cd45d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e5bea08530 con 0x55e5be9fa500 2024-05-12T22:17:59.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.863+0000 7fd6cd45d740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x55e5be919a90 con 0x55e5bea05520 2024-05-12T22:17:59.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.863+0000 7fd6cd45d740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55e5bea086b0 con 0x55e5be9faeb0 2024-05-12T22:17:59.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.863+0000 7fd6cd45d740 10 client.0 _ll_register_callbacks cb 0x55e5be954af0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:17:59.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.863+0000 7fd6c3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e5be9fa500 0x55e5bea02370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:59.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.863+0000 7fd6c3e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e5bea05520 0x55e5bea02df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:59.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.864+0000 7fd6c3e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e5bea05520 0x55e5bea02df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:33524/0 (socket says 172.21.15.28:33524) 2024-05-12T22:17:59.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.864+0000 7fd6c3e00640 1 -- 172.21.15.28:0/255412890 learned_addr learned my addr 172.21.15.28:0/255412890 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:17:59.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.864+0000 7fd6c2a00640 1 --2- 172.21.15.28:0/255412890 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e5be9faeb0 0x55e5bea028b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:59.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.864+0000 7fd6c3e00640 1 -- 172.21.15.28:0/255412890 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e5be9faeb0 msgr2=0x55e5bea028b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:17:59.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.864+0000 7fd6c3e00640 1 --2- 172.21.15.28:0/255412890 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e5be9faeb0 0x55e5bea028b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:59.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.864+0000 7fd6c3e00640 1 -- 172.21.15.28:0/255412890 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e5be9fa500 msgr2=0x55e5bea02370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:17:59.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.864+0000 7fd6c3e00640 1 --2- 172.21.15.28:0/255412890 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e5be9fa500 0x55e5bea02370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:59.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.864+0000 7fd6c3e00640 1 -- 172.21.15.28:0/255412890 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e5be73a7e0 con 0x55e5bea05520 2024-05-12T22:17:59.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.864+0000 7fd6c2a00640 1 --2- 172.21.15.28:0/255412890 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e5be9faeb0 0x55e5bea028b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:17:59.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.864+0000 7fd6c3e00640 1 --2- 172.21.15.28:0/255412890 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e5bea05520 0x55e5bea02df0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fd6bc005090 tx=0x7fd6bc0050c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bdde8efb4582d726 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:17:59.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.864+0000 7fd6ad600640 10 client.0 ms_handle_connect on v2:172.21.15.86:3300/0 2024-05-12T22:17:59.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.865+0000 7fd6ad600640 1 -- 172.21.15.28:0/255412890 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6bc028020 con 0x55e5bea05520 2024-05-12T22:17:59.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.865+0000 7fd6cd45d740 1 -- 172.21.15.28:0/255412890 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd6bc005500 con 0x55e5bea05520 2024-05-12T22:17:59.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.865+0000 7fd6ad600640 1 -- 172.21.15.28:0/255412890 <== mon.2 v2:172.21.15.86:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8513+0+0 (secure 0 0 0) 0x7fd6bc024420 con 0x55e5bea05520 2024-05-12T22:17:59.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.865+0000 7fd6ad600640 1 -- 172.21.15.28:0/255412890 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fd6bc0046c0 con 0x55e5bea05520 2024-05-12T22:17:59.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.865+0000 7fd6cd45d740 10 client.27227 fetch_fsmap learned FSMap version 26 2024-05-12T22:17:59.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.865+0000 7fd6cd45d740 1 -- 172.21.15.28:0/255412890 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55e5be79af70 con 0x55e5bea05520 2024-05-12T22:17:59.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.866+0000 7fd6ad600640 1 -- 172.21.15.28:0/255412890 <== mon.2 v2:172.21.15.86:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd6bc005500 con 0x55e5bea05520 2024-05-12T22:17:59.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.866+0000 7fd6ad600640 1 -- 172.21.15.28:0/255412890 <== mon.2 v2:172.21.15.86:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fd6bc018070 con 0x55e5bea05520 2024-05-12T22:17:59.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.866+0000 7fd6cd45d740 10 client.27227 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:17:59.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.866+0000 7fd6cd45d740 10 client.27227 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:17:59.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.866+0000 7fd6cd45d740 20 client.27227 populate_metadata read hostname 'smithi028' 2024-05-12T22:17:59.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.866+0000 7fd6cd45d740 1 --2- 172.21.15.28:0/255412890 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55e5bea15560 0x55e5bea17950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:59.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.866+0000 7fd6cd45d740 4 client.27227 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-12T22:17:59.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.866+0000 7fd6cd45d740 1 -- 172.21.15.28:0/255412890 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e5be9723a0 con 0x55e5bea15560 2024-05-12T22:17:59.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.866+0000 7fd6c3400640 1 --2- 172.21.15.28:0/255412890 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55e5bea15560 0x55e5bea17950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:59.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.867+0000 7fd6c3400640 1 --2- 172.21.15.28:0/255412890 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55e5bea15560 0x55e5bea17950 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=883692cccce76be0 server_cookie=5e32d949bfe5149d in_seq=0 out_seq=0 2024-05-12T22:17:59.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.867+0000 7fd6ad600640 10 client.27227 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:17:59.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.868+0000 7fd6ad600640 1 -- 172.21.15.28:0/255412890 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e5be9723a0 con 0x55e5bea15560 2024-05-12T22:17:59.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.868+0000 7fd6ad600640 10 client.27227 handle_command_reply: tid=0 2024-05-12T22:17:59.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.868+0000 7fd6cd45d740 1 client.27227 shutdown 2024-05-12T22:17:59.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.868+0000 7fd6cd45d740 1 -- 172.21.15.28:0/255412890 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e5bea05520 msgr2=0x55e5bea02df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:17:59.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.868+0000 7fd6cd45d740 1 --2- 172.21.15.28:0/255412890 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e5bea05520 0x55e5bea02df0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fd6bc005090 tx=0x7fd6bc0050c0 comp rx=0 tx=0).stop 2024-05-12T22:17:59.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.868+0000 7fd6cd45d740 1 -- 172.21.15.28:0/255412890 shutdown_connections 2024-05-12T22:17:59.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.868+0000 7fd6cd45d740 1 --2- 172.21.15.28:0/255412890 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55e5bea15560 0x55e5bea17950 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:59.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.868+0000 7fd6cd45d740 1 --2- 172.21.15.28:0/255412890 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e5bea05520 0x55e5bea02df0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:59.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.868+0000 7fd6cd45d740 1 --2- 172.21.15.28:0/255412890 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e5be9faeb0 0x55e5bea028b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:59.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.868+0000 7fd6ad600640 0 client.27227 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:17:59.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.868+0000 7fd6cd45d740 1 --2- 172.21.15.28:0/255412890 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e5be9fa500 0x55e5bea02370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:59.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.868+0000 7fd6cd45d740 1 -- 172.21.15.28:0/255412890 >> 172.21.15.28:0/255412890 conn(0x55e5be9f65b0 msgr2=0x55e5be9fc630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:17:59.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.868+0000 7fd6cd45d740 1 -- 172.21.15.28:0/255412890 shutdown_connections 2024-05-12T22:17:59.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.868+0000 7fd6cd45d740 1 -- 172.21.15.28:0/255412890 wait complete. 2024-05-12T22:17:59.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.868+0000 7fd6cd45d740 20 client.27227 trim_cache size 0 max 16384 2024-05-12T22:17:59.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.888+0000 7fd6cd45d740 1 Processor -- start 2024-05-12T22:17:59.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.888+0000 7fd6cd45d740 1 -- start start 2024-05-12T22:17:59.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.888+0000 7fd6cd45d740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e5be9fa500 0x55e5bea11450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:59.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.888+0000 7fd6cd45d740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e5be9faeb0 0x55e5be9fcd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:59.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.888+0000 7fd6cd45d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e5bea05520 0x55e5bea1d160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:59.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.888+0000 7fd6cd45d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd6c41b6150 con 0x55e5bea05520 2024-05-12T22:17:59.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.888+0000 7fd6cd45d740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fd6c41b3a20 con 0x55e5be9fa500 2024-05-12T22:17:59.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.888+0000 7fd6cd45d740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd6c41b1890 con 0x55e5be9faeb0 2024-05-12T22:17:59.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.889+0000 7fd6c2a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e5be9faeb0 0x55e5be9fcd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:59.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.889+0000 7fd6c2a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e5be9faeb0 0x55e5be9fcd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:56564/0 (socket says 172.21.15.28:56564) 2024-05-12T22:17:59.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.889+0000 7fd6c2a00640 1 -- 172.21.15.28:0/1090025440 learned_addr learned my addr 172.21.15.28:0/1090025440 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:17:59.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.889+0000 7fd6c3400640 1 --2- 172.21.15.28:0/1090025440 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e5be9fa500 0x55e5bea11450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:59.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.889+0000 7fd6c3e00640 1 --2- 172.21.15.28:0/1090025440 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e5bea05520 0x55e5bea1d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:59.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.889+0000 7fd6c2a00640 1 -- 172.21.15.28:0/1090025440 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e5be9fa500 msgr2=0x55e5bea11450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:17:59.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.889+0000 7fd6c2a00640 1 --2- 172.21.15.28:0/1090025440 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e5be9fa500 0x55e5bea11450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:59.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.889+0000 7fd6c2a00640 1 -- 172.21.15.28:0/1090025440 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e5bea05520 msgr2=0x55e5bea1d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:17:59.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.889+0000 7fd6c2a00640 1 --2- 172.21.15.28:0/1090025440 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e5bea05520 0x55e5bea1d160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:59.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.889+0000 7fd6c2a00640 1 -- 172.21.15.28:0/1090025440 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e5be632170 con 0x55e5be9faeb0 2024-05-12T22:17:59.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.889+0000 7fd6c3400640 1 --2- 172.21.15.28:0/1090025440 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e5be9fa500 0x55e5bea11450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:17:59.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.889+0000 7fd6c2a00640 1 --2- 172.21.15.28:0/1090025440 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e5be9faeb0 0x55e5be9fcd40 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fd6b00b86b0 tx=0x7fd6b00f3040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=47a47b086081f865 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:17:59.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.890+0000 7fd6afe00640 1 -- 172.21.15.28:0/1090025440 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6b0049070 con 0x55e5be9faeb0 2024-05-12T22:17:59.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.890+0000 7fd6afe00640 1 -- 172.21.15.28:0/1090025440 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd6b00bbce0 con 0x55e5be9faeb0 2024-05-12T22:17:59.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.890+0000 7fd6afe00640 1 -- 172.21.15.28:0/1090025440 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6b00b8350 con 0x55e5be9faeb0 2024-05-12T22:17:59.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.890+0000 7fd6cd45d740 1 -- 172.21.15.28:0/1090025440 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e5be9faeb0 msgr2=0x55e5be9fcd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:17:59.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.890+0000 7fd6cd45d740 1 --2- 172.21.15.28:0/1090025440 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e5be9faeb0 0x55e5be9fcd40 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fd6b00b86b0 tx=0x7fd6b00f3040 comp rx=0 tx=0).stop 2024-05-12T22:17:59.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.890+0000 7fd6cd45d740 1 -- 172.21.15.28:0/1090025440 shutdown_connections 2024-05-12T22:17:59.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.890+0000 7fd6cd45d740 1 --2- 172.21.15.28:0/1090025440 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e5bea05520 0x55e5bea1d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:59.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.890+0000 7fd6cd45d740 1 --2- 172.21.15.28:0/1090025440 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e5be9faeb0 0x55e5be9fcd40 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:59.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.890+0000 7fd6cd45d740 1 --2- 172.21.15.28:0/1090025440 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e5be9fa500 0x55e5bea11450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:59.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.890+0000 7fd6cd45d740 1 -- 172.21.15.28:0/1090025440 >> 172.21.15.28:0/1090025440 conn(0x55e5be9f65b0 msgr2=0x55e5be9f4ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:17:59.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.891+0000 7fd6cd45d740 1 -- 172.21.15.28:0/1090025440 shutdown_connections 2024-05-12T22:17:59.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.891+0000 7fd6cd45d740 1 -- 172.21.15.28:0/1090025440 wait complete. 2024-05-12T22:17:59.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.891+0000 7fd6cd45d740 1 -- start start 2024-05-12T22:17:59.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.891+0000 7fd6cd45d740 1 Processor -- start 2024-05-12T22:17:59.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.892+0000 7fd6cd45d740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e5be9fa500 0x55e5bea0d960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:59.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.892+0000 7fd6cd45d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e5be9faeb0 0x55e5bea0dea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:59.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.892+0000 7fd6cd45d740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e5bea05520 0x55e5be9feb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:59.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.892+0000 7fd6cd45d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd6c41b1ee0 con 0x55e5be9faeb0 2024-05-12T22:17:59.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.892+0000 7fd6cd45d740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fd6c41b3350 con 0x55e5be9fa500 2024-05-12T22:17:59.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.892+0000 7fd6cd45d740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd6c41b25b0 con 0x55e5bea05520 2024-05-12T22:17:59.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.892+0000 7fd6cd45d740 10 client.0 _ll_register_callbacks cb 0x7fd6b00f5400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:17:59.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.892+0000 7fd6c2a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e5be9faeb0 0x55e5bea0dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:59.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.892+0000 7fd6c2a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e5be9faeb0 0x55e5bea0dea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40220/0 (socket says 172.21.15.28:40220) 2024-05-12T22:17:59.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.892+0000 7fd6c2a00640 1 -- 172.21.15.28:0/554804391 learned_addr learned my addr 172.21.15.28:0/554804391 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:17:59.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.892+0000 7fd6c3400640 1 --2- 172.21.15.28:0/554804391 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e5be9fa500 0x55e5bea0d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:59.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.892+0000 7fd6c3e00640 1 --2- 172.21.15.28:0/554804391 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e5bea05520 0x55e5be9feb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:59.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.892+0000 7fd6c2a00640 1 -- 172.21.15.28:0/554804391 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e5bea05520 msgr2=0x55e5be9feb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:17:59.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.892+0000 7fd6c2a00640 1 --2- 172.21.15.28:0/554804391 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e5bea05520 0x55e5be9feb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:59.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.892+0000 7fd6c2a00640 1 -- 172.21.15.28:0/554804391 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e5be9fa500 msgr2=0x55e5bea0d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:17:59.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.892+0000 7fd6c2a00640 1 --2- 172.21.15.28:0/554804391 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e5be9fa500 0x55e5bea0d960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:59.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.892+0000 7fd6c2a00640 1 -- 172.21.15.28:0/554804391 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e5be9ff380 con 0x55e5be9faeb0 2024-05-12T22:17:59.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.893+0000 7fd6c2a00640 1 --2- 172.21.15.28:0/554804391 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e5be9faeb0 0x55e5bea0dea0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fd6b0005ab0 tx=0x7fd6b00bb890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a1a226f2b2ec839 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:17:59.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.893+0000 7fd6ad600640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-12T22:17:59.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.893+0000 7fd6ad600640 1 -- 172.21.15.28:0/554804391 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6b0049040 con 0x55e5be9faeb0 2024-05-12T22:17:59.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.893+0000 7fd6ad600640 1 -- 172.21.15.28:0/554804391 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8513+0+0 (secure 0 0 0) 0x7fd6b0031bd0 con 0x55e5be9faeb0 2024-05-12T22:17:59.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.893+0000 7fd6cd45d740 1 -- 172.21.15.28:0/554804391 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd6b00bbce0 con 0x55e5be9faeb0 2024-05-12T22:17:59.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.893+0000 7fd6ad600640 1 -- 172.21.15.28:0/554804391 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fd6b00bbab0 con 0x55e5be9faeb0 2024-05-12T22:17:59.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.893+0000 7fd6cd45d740 10 client.17100 fetch_fsmap learned FSMap version 26 2024-05-12T22:17:59.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.893+0000 7fd6cd45d740 1 -- 172.21.15.28:0/554804391 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55e5bea7b610 con 0x55e5be9faeb0 2024-05-12T22:17:59.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.893+0000 7fd6ad600640 1 -- 172.21.15.28:0/554804391 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd6b00bbce0 con 0x55e5be9faeb0 2024-05-12T22:17:59.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.894+0000 7fd6ad600640 1 -- 172.21.15.28:0/554804391 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fd6b00efb60 con 0x55e5be9faeb0 2024-05-12T22:17:59.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.894+0000 7fd6cd45d740 10 client.17100 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:17:59.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.894+0000 7fd6cd45d740 10 client.17100 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:17:59.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.894+0000 7fd6cd45d740 20 client.17100 populate_metadata read hostname 'smithi028' 2024-05-12T22:17:59.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.894+0000 7fd6cd45d740 1 --2- 172.21.15.28:0/554804391 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55e5bea7bb00 0x55e5bea7df20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:17:59.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.894+0000 7fd6cd45d740 4 client.17100 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-12T22:17:59.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.894+0000 7fd6cd45d740 1 -- 172.21.15.28:0/554804391 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55e5be96bef0 con 0x55e5bea7bb00 2024-05-12T22:17:59.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.894+0000 7fd6c3400640 1 --2- 172.21.15.28:0/554804391 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55e5bea7bb00 0x55e5bea7df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:17:59.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.895+0000 7fd6c3400640 1 --2- 172.21.15.28:0/554804391 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55e5bea7bb00 0x55e5bea7df20 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=833410f611bd1b3c server_cookie=9f9a90473a84ba6d in_seq=0 out_seq=0 2024-05-12T22:17:59.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.895+0000 7fd6ad600640 10 client.17100 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:17:59.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.896+0000 7fd6ad600640 1 -- 172.21.15.28:0/554804391 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55e5be96bef0 con 0x55e5bea7bb00 2024-05-12T22:17:59.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.896+0000 7fd6ad600640 10 client.17100 handle_command_reply: tid=0 2024-05-12T22:17:59.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.896+0000 7fd6cd45d740 1 client.17100 shutdown 2024-05-12T22:17:59.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.896+0000 7fd6cd45d740 1 -- 172.21.15.28:0/554804391 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e5be9faeb0 msgr2=0x55e5bea0dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:17:59.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.896+0000 7fd6cd45d740 1 --2- 172.21.15.28:0/554804391 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e5be9faeb0 0x55e5bea0dea0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fd6b0005ab0 tx=0x7fd6b00bb890 comp rx=0 tx=0).stop 2024-05-12T22:17:59.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.896+0000 7fd6cd45d740 1 -- 172.21.15.28:0/554804391 shutdown_connections 2024-05-12T22:17:59.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.896+0000 7fd6cd45d740 1 --2- 172.21.15.28:0/554804391 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55e5bea7bb00 0x55e5bea7df20 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:59.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.896+0000 7fd6cd45d740 1 --2- 172.21.15.28:0/554804391 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e5bea05520 0x55e5be9feb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:59.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.896+0000 7fd6cd45d740 1 --2- 172.21.15.28:0/554804391 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e5be9faeb0 0x55e5bea0dea0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:59.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.896+0000 7fd6cd45d740 1 --2- 172.21.15.28:0/554804391 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e5be9fa500 0x55e5bea0d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:59.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.896+0000 7fd6cd45d740 1 -- 172.21.15.28:0/554804391 >> 172.21.15.28:0/554804391 conn(0x55e5be9f65b0 msgr2=0x55e5be9f5730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:17:59.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.896+0000 7fd6ad600640 0 client.17100 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:17:59.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.896+0000 7fd6cd45d740 1 -- 172.21.15.28:0/554804391 shutdown_connections 2024-05-12T22:17:59.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.896+0000 7fd6cd45d740 1 -- 172.21.15.28:0/554804391 wait complete. 2024-05-12T22:17:59.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.897+0000 7fd6cd45d740 20 client.17100 trim_cache size 0 max 16384 2024-05-12T22:17:59.905 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-05-12T22:17:59.905 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-05-12T22:17:59.905 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "552e6618-d080-47a8-9b9a-c74098705d4b", 2024-05-12T22:17:59.906 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-05-12T22:17:59.906 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-05-12T22:17:59.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.897+0000 7fd6c9000640 1 -- 172.21.15.28:0/1456790867 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fd6a80a2d80 msgr2=0x7fd6a80a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:17:59.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.897+0000 7fd6c9000640 1 --2- 172.21.15.28:0/1456790867 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fd6a80a2d80 0x7fd6a80a5240 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x55e5be9f7830 tx=0x7fd6b4009500 comp rx=0 tx=0).stop 2024-05-12T22:17:59.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.897+0000 7fd6c9000640 1 -- 172.21.15.28:0/1456790867 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6c4108770 msgr2=0x7fd6c419ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:17:59.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.897+0000 7fd6c9000640 1 --2- 172.21.15.28:0/1456790867 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6c4108770 0x7fd6c419ea50 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fd6b0006010 tx=0x7fd6b0004040 comp rx=0 tx=0).stop 2024-05-12T22:17:59.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.897+0000 7fd6c9000640 1 -- 172.21.15.28:0/1456790867 shutdown_connections 2024-05-12T22:17:59.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.897+0000 7fd6c9000640 1 --2- 172.21.15.28:0/1456790867 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fd6a80a2d80 0x7fd6a80a5240 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:59.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.897+0000 7fd6c9000640 1 --2- 172.21.15.28:0/1456790867 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd6c410ddf0 0x7fd6c41a2dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:59.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.897+0000 7fd6c9000640 1 --2- 172.21.15.28:0/1456790867 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6c4108770 0x7fd6c419ea50 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:59.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.897+0000 7fd6c9000640 1 --2- 172.21.15.28:0/1456790867 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd6c4107dd0 0x7fd6c419e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:17:59.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.897+0000 7fd6c9000640 1 -- 172.21.15.28:0/1456790867 >> 172.21.15.28:0/1456790867 conn(0x7fd6c4077650 msgr2=0x7fd6c4078630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:17:59.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.898+0000 7fd6c9000640 1 -- 172.21.15.28:0/1456790867 shutdown_connections 2024-05-12T22:17:59.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:17:59.898+0000 7fd6c9000640 1 -- 172.21.15.28:0/1456790867 wait complete. 2024-05-12T22:17:59.918 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-12T22:17:59.919 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-12T22:18:00.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.147+0000 7fae8c800640 1 -- 172.21.15.28:0/1789216652 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae84103a10 msgr2=0x7fae84103e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:18:00.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.147+0000 7fae8c800640 1 --2- 172.21.15.28:0/1789216652 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae84103a10 0x7fae84103e90 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fae74009900 tx=0x7fae7402a3f0 comp rx=0 tx=0).stop 2024-05-12T22:18:00.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.148+0000 7fae8c800640 1 -- 172.21.15.28:0/1789216652 shutdown_connections 2024-05-12T22:18:00.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.148+0000 7fae8c800640 1 --2- 172.21.15.28:0/1789216652 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae841043d0 0x7fae841108e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:00.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.148+0000 7fae8c800640 1 --2- 172.21.15.28:0/1789216652 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae84103a10 0x7fae84103e90 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:00.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.148+0000 7fae8c800640 1 --2- 172.21.15.28:0/1789216652 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fae84109a20 0x7fae84109e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:00.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.148+0000 7fae8c800640 1 -- 172.21.15.28:0/1789216652 >> 172.21.15.28:0/1789216652 conn(0x7fae840ff4a0 msgr2=0x7fae841018c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:18:00.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.148+0000 7fae8c800640 1 -- 172.21.15.28:0/1789216652 shutdown_connections 2024-05-12T22:18:00.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.148+0000 7fae8c800640 1 -- 172.21.15.28:0/1789216652 wait complete. 2024-05-12T22:18:00.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.149+0000 7fae8c800640 1 Processor -- start 2024-05-12T22:18:00.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.149+0000 7fae8c800640 1 -- start start 2024-05-12T22:18:00.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.150+0000 7fae8c800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae84103a10 0x7fae8419a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:00.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.150+0000 7fae8c800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae841043d0 0x7fae8419a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:00.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.150+0000 7fae8c800640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fae84109a20 0x7fae8419ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:00.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.150+0000 7fae8c800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fae84113e80 con 0x7fae841043d0 2024-05-12T22:18:00.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.150+0000 7fae8c800640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fae84113d00 con 0x7fae84103a10 2024-05-12T22:18:00.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.150+0000 7fae8c800640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fae84114000 con 0x7fae84109a20 2024-05-12T22:18:00.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.150+0000 7fae89e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae84103a10 0x7fae8419a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:00.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.150+0000 7fae89e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae84103a10 0x7fae8419a3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:33582/0 (socket says 172.21.15.28:33582) 2024-05-12T22:18:00.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.150+0000 7fae89e00640 1 -- 172.21.15.28:0/330829241 learned_addr learned my addr 172.21.15.28:0/330829241 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:18:00.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.150+0000 7fae89400640 1 --2- 172.21.15.28:0/330829241 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae841043d0 0x7fae8419a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:00.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.150+0000 7fae8a800640 1 --2- 172.21.15.28:0/330829241 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fae84109a20 0x7fae8419ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:00.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.151+0000 7fae8a800640 1 -- 172.21.15.28:0/330829241 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae84103a10 msgr2=0x7fae8419a3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:18:00.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.151+0000 7fae8a800640 1 --2- 172.21.15.28:0/330829241 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae84103a10 0x7fae8419a3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:00.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.151+0000 7fae8a800640 1 -- 172.21.15.28:0/330829241 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae841043d0 msgr2=0x7fae8419a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:18:00.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.151+0000 7fae8a800640 1 --2- 172.21.15.28:0/330829241 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae841043d0 0x7fae8419a900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:00.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.151+0000 7fae8a800640 1 -- 172.21.15.28:0/330829241 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae8419f360 con 0x7fae84109a20 2024-05-12T22:18:00.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.151+0000 7fae8a800640 1 --2- 172.21.15.28:0/330829241 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fae84109a20 0x7fae8419ec30 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fae80009fd0 tx=0x7fae8000ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:18:00.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.151+0000 7fae7ea00640 1 -- 172.21.15.28:0/330829241 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae8000ce20 con 0x7fae84109a20 2024-05-12T22:18:00.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.151+0000 7fae8c800640 1 -- 172.21.15.28:0/330829241 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae8419f650 con 0x7fae84109a20 2024-05-12T22:18:00.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.151+0000 7fae7ea00640 1 -- 172.21.15.28:0/330829241 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fae80005020 con 0x7fae84109a20 2024-05-12T22:18:00.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.151+0000 7fae7ea00640 1 -- 172.21.15.28:0/330829241 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae80010720 con 0x7fae84109a20 2024-05-12T22:18:00.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.152+0000 7fae8c800640 1 -- 172.21.15.28:0/330829241 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae841a6ee0 con 0x7fae84109a20 2024-05-12T22:18:00.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.156+0000 7fae8f790740 1 Processor -- start 2024-05-12T22:18:00.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.156+0000 7fae7ea00640 1 -- 172.21.15.28:0/330829241 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7fae80010900 con 0x7fae84109a20 2024-05-12T22:18:00.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.157+0000 7fae8f790740 1 -- start start 2024-05-12T22:18:00.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.157+0000 7fae8f790740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ab127808f0 0x55ab12780cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:00.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.157+0000 7fae8f790740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ab127812a0 0x55ab12793e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:00.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.157+0000 7fae8f790740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ab127943c0 0x55ab127967b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:00.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.157+0000 7fae8f790740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ab127492a0 con 0x55ab127943c0 2024-05-12T22:18:00.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.157+0000 7fae8f790740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x55ab126e5b90 con 0x55ab127808f0 2024-05-12T22:18:00.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.157+0000 7fae8f790740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55ab1268d5b0 con 0x55ab127812a0 2024-05-12T22:18:00.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.157+0000 7fae89400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ab127808f0 0x55ab12780cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:00.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.158+0000 7fae89e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ab127812a0 0x55ab12793e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:00.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.158+0000 7fae89400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ab127808f0 0x55ab12780cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:33588/0 (socket says 172.21.15.28:33588) 2024-05-12T22:18:00.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.158+0000 7fae89400640 1 -- 172.21.15.28:0/3783429939 learned_addr learned my addr 172.21.15.28:0/3783429939 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:18:00.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.158+0000 7fae89400640 1 --2- 172.21.15.28:0/3783429939 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ab127943c0 0x55ab127967b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:00.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.158+0000 7fae89e00640 1 -- 172.21.15.28:0/3783429939 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ab127808f0 msgr2=0x55ab12780cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:18:00.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.158+0000 7fae89e00640 1 --2- 172.21.15.28:0/3783429939 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ab127808f0 0x55ab12780cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:00.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.158+0000 7fae89e00640 1 -- 172.21.15.28:0/3783429939 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ab127943c0 msgr2=0x55ab127967b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:18:00.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.158+0000 7fae89e00640 1 --2- 172.21.15.28:0/3783429939 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ab127943c0 0x55ab127967b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:00.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.158+0000 7fae89e00640 1 -- 172.21.15.28:0/3783429939 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ab124c37d0 con 0x55ab127812a0 2024-05-12T22:18:00.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.158+0000 7fae89400640 1 --2- 172.21.15.28:0/3783429939 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ab127943c0 0x55ab127967b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:18:00.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.159+0000 7fae89e00640 1 --2- 172.21.15.28:0/3783429939 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ab127812a0 0x55ab12793e80 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fae78009c60 tx=0x7fae78011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=edbe8c31bad5a0cf server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:18:00.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.159+0000 7fae7ea00640 1 --2- 172.21.15.28:0/330829241 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fae6c0a2d80 0x7fae6c0a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:00.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.159+0000 7fae73000640 1 -- 172.21.15.28:0/3783429939 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae78004230 con 0x55ab127812a0 2024-05-12T22:18:00.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.159+0000 7fae7ea00640 1 -- 172.21.15.28:0/330829241 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8513+0+0 (secure 0 0 0) 0x7fae800cfb60 con 0x7fae84109a20 2024-05-12T22:18:00.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.159+0000 7fae73000640 1 -- 172.21.15.28:0/3783429939 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fae780043d0 con 0x55ab127812a0 2024-05-12T22:18:00.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.159+0000 7fae73000640 1 -- 172.21.15.28:0/3783429939 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae78004df0 con 0x55ab127812a0 2024-05-12T22:18:00.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.159+0000 7fae89400640 1 --2- 172.21.15.28:0/330829241 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fae6c0a2d80 0x7fae6c0a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:00.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.160+0000 7fae8f790740 1 -- 172.21.15.28:0/3783429939 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ab127812a0 msgr2=0x55ab12793e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:18:00.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.160+0000 7fae8f790740 1 --2- 172.21.15.28:0/3783429939 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ab127812a0 0x55ab12793e80 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fae78009c60 tx=0x7fae78011810 comp rx=0 tx=0).stop 2024-05-12T22:18:00.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.160+0000 7fae8f790740 1 -- 172.21.15.28:0/3783429939 shutdown_connections 2024-05-12T22:18:00.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.160+0000 7fae8f790740 1 --2- 172.21.15.28:0/3783429939 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ab127943c0 0x55ab127967b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:00.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.160+0000 7fae8f790740 1 --2- 172.21.15.28:0/3783429939 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ab127812a0 0x55ab12793e80 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:00.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.160+0000 7fae8f790740 1 --2- 172.21.15.28:0/3783429939 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ab127808f0 0x55ab12780cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:00.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.160+0000 7fae8f790740 1 -- 172.21.15.28:0/3783429939 >> 172.21.15.28:0/3783429939 conn(0x55ab1277cee0 msgr2=0x55ab1277d2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:18:00.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.160+0000 7fae89400640 1 --2- 172.21.15.28:0/330829241 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fae6c0a2d80 0x7fae6c0a5240 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fae740096c0 tx=0x7fae7402a3f0 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:18:00.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.161+0000 7fae8f790740 1 -- 172.21.15.28:0/3783429939 shutdown_connections 2024-05-12T22:18:00.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.161+0000 7fae8f790740 1 -- 172.21.15.28:0/3783429939 wait complete. 2024-05-12T22:18:00.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.161+0000 7fae8f790740 1 -- start start 2024-05-12T22:18:00.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.161+0000 7fae8f790740 1 Processor -- start 2024-05-12T22:18:00.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.162+0000 7fae8f790740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ab127808f0 0x55ab12793d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:00.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.162+0000 7fae8f790740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ab127812a0 0x55ab12791020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:00.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.162+0000 7fae8f790740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ab127943c0 0x55ab12791580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:00.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.162+0000 7fae8f790740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ab126a09f0 con 0x55ab127943c0 2024-05-12T22:18:00.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.162+0000 7fae8f790740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x55ab126e1b50 con 0x55ab127808f0 2024-05-12T22:18:00.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.162+0000 7fae8f790740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55ab12797180 con 0x55ab127812a0 2024-05-12T22:18:00.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.162+0000 7fae8f790740 10 client.0 _ll_register_callbacks cb 0x55ab123f8be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:18:00.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.163+0000 7fae89e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ab127808f0 0x55ab12793d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:00.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.163+0000 7fae89400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ab127812a0 0x55ab12791020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:00.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.163+0000 7fae89400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ab127812a0 0x55ab12791020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:56606/0 (socket says 172.21.15.28:56606) 2024-05-12T22:18:00.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.163+0000 7fae89400640 1 -- 172.21.15.28:0/3707308569 learned_addr learned my addr 172.21.15.28:0/3707308569 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:18:00.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.163+0000 7fae8a800640 1 --2- 172.21.15.28:0/3707308569 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ab127943c0 0x55ab12791580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:00.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.163+0000 7fae89e00640 1 -- 172.21.15.28:0/3707308569 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ab127812a0 msgr2=0x55ab12791020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:18:00.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.163+0000 7fae89e00640 1 --2- 172.21.15.28:0/3707308569 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ab127812a0 0x55ab12791020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:00.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.163+0000 7fae89e00640 1 -- 172.21.15.28:0/3707308569 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ab127943c0 msgr2=0x55ab12791580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:18:00.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.163+0000 7fae89e00640 1 --2- 172.21.15.28:0/3707308569 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ab127943c0 0x55ab12791580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:00.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.164+0000 7fae89e00640 1 -- 172.21.15.28:0/3707308569 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ab123f93f0 con 0x55ab127808f0 2024-05-12T22:18:00.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.164+0000 7fae89e00640 1 --2- 172.21.15.28:0/3707308569 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ab127808f0 0x55ab12793d30 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fae78009620 tx=0x7fae7801abc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=819ccaeab32af7c7 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:18:00.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.164+0000 7fae67400640 10 client.0 ms_handle_connect on v2:172.21.15.86:3300/0 2024-05-12T22:18:00.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.164+0000 7fae67400640 1 -- 172.21.15.28:0/3707308569 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae78028020 con 0x55ab127808f0 2024-05-12T22:18:00.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.164+0000 7fae8f790740 1 -- 172.21.15.28:0/3707308569 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fae780043d0 con 0x55ab127808f0 2024-05-12T22:18:00.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.164+0000 7fae67400640 1 -- 172.21.15.28:0/3707308569 <== mon.2 v2:172.21.15.86:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8513+0+0 (secure 0 0 0) 0x7fae78024620 con 0x55ab127808f0 2024-05-12T22:18:00.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.165+0000 7fae67400640 1 -- 172.21.15.28:0/3707308569 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fae78033050 con 0x55ab127808f0 2024-05-12T22:18:00.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.165+0000 7fae8f790740 10 client.27245 fetch_fsmap learned FSMap version 26 2024-05-12T22:18:00.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.165+0000 7fae8f790740 1 -- 172.21.15.28:0/3707308569 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55ab123b90f0 con 0x55ab127808f0 2024-05-12T22:18:00.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.165+0000 7fae67400640 1 -- 172.21.15.28:0/3707308569 <== mon.2 v2:172.21.15.86:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fae780043d0 con 0x55ab127808f0 2024-05-12T22:18:00.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.165+0000 7fae67400640 1 -- 172.21.15.28:0/3707308569 <== mon.2 v2:172.21.15.86:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fae78018070 con 0x55ab127808f0 2024-05-12T22:18:00.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.165+0000 7fae8f790740 10 client.27245 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:18:00.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.165+0000 7fae8f790740 10 client.27245 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:18:00.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.165+0000 7fae8f790740 20 client.27245 populate_metadata read hostname 'smithi028' 2024-05-12T22:18:00.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.165+0000 7fae8f790740 1 --2- 172.21.15.28:0/3707308569 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55ab127a3e60 0x55ab127a6250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:00.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.166+0000 7fae8f790740 4 client.27245 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-12T22:18:00.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.166+0000 7fae8f790740 1 -- 172.21.15.28:0/3707308569 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ab126f32c0 con 0x55ab127a3e60 2024-05-12T22:18:00.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.166+0000 7fae89400640 1 --2- 172.21.15.28:0/3707308569 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55ab127a3e60 0x55ab127a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:00.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.167+0000 7fae89400640 1 --2- 172.21.15.28:0/3707308569 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55ab127a3e60 0x55ab127a6250 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=81e099b97998ace3 server_cookie=e12ca645f00159b2 in_seq=0 out_seq=0 2024-05-12T22:18:00.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.167+0000 7fae67400640 10 client.27245 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:18:00.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.168+0000 7fae67400640 1 -- 172.21.15.28:0/3707308569 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ab126f32c0 con 0x55ab127a3e60 2024-05-12T22:18:00.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.168+0000 7fae67400640 10 client.27245 handle_command_reply: tid=0 2024-05-12T22:18:00.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.168+0000 7fae8f790740 1 client.27245 shutdown 2024-05-12T22:18:00.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.168+0000 7fae8f790740 1 -- 172.21.15.28:0/3707308569 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ab127808f0 msgr2=0x55ab12793d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:18:00.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.168+0000 7fae8f790740 1 --2- 172.21.15.28:0/3707308569 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ab127808f0 0x55ab12793d30 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fae78009620 tx=0x7fae7801abc0 comp rx=0 tx=0).stop 2024-05-12T22:18:00.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.168+0000 7fae8f790740 1 -- 172.21.15.28:0/3707308569 shutdown_connections 2024-05-12T22:18:00.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.168+0000 7fae8f790740 1 --2- 172.21.15.28:0/3707308569 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55ab127a3e60 0x55ab127a6250 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:00.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.168+0000 7fae8f790740 1 --2- 172.21.15.28:0/3707308569 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ab127943c0 0x55ab12791580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:00.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.168+0000 7fae8f790740 1 --2- 172.21.15.28:0/3707308569 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ab127812a0 0x55ab12791020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:00.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.168+0000 7fae8f790740 1 --2- 172.21.15.28:0/3707308569 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ab127808f0 0x55ab12793d30 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:00.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.168+0000 7fae67400640 0 client.27245 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:18:00.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.168+0000 7fae8f790740 1 -- 172.21.15.28:0/3707308569 >> 172.21.15.28:0/3707308569 conn(0x55ab1277cee0 msgr2=0x55ab12782d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:18:00.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.169+0000 7fae8f790740 1 -- 172.21.15.28:0/3707308569 shutdown_connections 2024-05-12T22:18:00.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.169+0000 7fae8f790740 1 -- 172.21.15.28:0/3707308569 wait complete. 2024-05-12T22:18:00.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.169+0000 7fae8f790740 20 client.27245 trim_cache size 0 max 16384 2024-05-12T22:18:00.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.193+0000 7fae8f790740 1 Processor -- start 2024-05-12T22:18:00.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.193+0000 7fae8f790740 1 -- start start 2024-05-12T22:18:00.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.194+0000 7fae8f790740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ab127808f0 0x55ab1277b850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:00.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.194+0000 7fae8f790740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ab127812a0 0x55ab12793860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:00.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.194+0000 7fae8f790740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ab127943c0 0x55ab127aba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:00.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.194+0000 7fae8f790740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fae841adb80 con 0x55ab127812a0 2024-05-12T22:18:00.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.194+0000 7fae8f790740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fae841ab450 con 0x55ab127943c0 2024-05-12T22:18:00.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.194+0000 7fae8f790740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fae841a92c0 con 0x55ab127808f0 2024-05-12T22:18:00.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.194+0000 7fae89e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ab127808f0 0x55ab1277b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:00.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.194+0000 7fae89400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ab127812a0 0x55ab12793860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:00.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.194+0000 7fae89e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ab127808f0 0x55ab1277b850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:56618/0 (socket says 172.21.15.28:56618) 2024-05-12T22:18:00.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.194+0000 7fae8a800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ab127943c0 0x55ab127aba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:00.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.194+0000 7fae89e00640 1 -- 172.21.15.28:0/1478303491 learned_addr learned my addr 172.21.15.28:0/1478303491 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:18:00.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.194+0000 7fae89e00640 1 -- 172.21.15.28:0/1478303491 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ab127943c0 msgr2=0x55ab127aba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:18:00.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.194+0000 7fae89e00640 1 --2- 172.21.15.28:0/1478303491 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ab127943c0 0x55ab127aba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:00.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.194+0000 7fae89e00640 1 -- 172.21.15.28:0/1478303491 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ab127812a0 msgr2=0x55ab12793860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:18:00.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.194+0000 7fae89e00640 1 --2- 172.21.15.28:0/1478303491 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ab127812a0 0x55ab12793860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:00.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.195+0000 7fae89e00640 1 -- 172.21.15.28:0/1478303491 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ab1277c700 con 0x55ab127808f0 2024-05-12T22:18:00.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.195+0000 7fae89e00640 1 --2- 172.21.15.28:0/1478303491 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ab127808f0 0x55ab1277b850 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fae78009c60 tx=0x7fae7801a710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=af96fb188424dfe2 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:18:00.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.195+0000 7fae72600640 1 -- 172.21.15.28:0/1478303491 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae78028070 con 0x55ab127808f0 2024-05-12T22:18:00.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.195+0000 7fae72600640 1 -- 172.21.15.28:0/1478303491 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fae7801f040 con 0x55ab127808f0 2024-05-12T22:18:00.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.195+0000 7fae72600640 1 -- 172.21.15.28:0/1478303491 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae78024420 con 0x55ab127808f0 2024-05-12T22:18:00.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.196+0000 7fae8f790740 1 -- 172.21.15.28:0/1478303491 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ab127808f0 msgr2=0x55ab1277b850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:18:00.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.196+0000 7fae8f790740 1 --2- 172.21.15.28:0/1478303491 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ab127808f0 0x55ab1277b850 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fae78009c60 tx=0x7fae7801a710 comp rx=0 tx=0).stop 2024-05-12T22:18:00.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.196+0000 7fae8f790740 1 -- 172.21.15.28:0/1478303491 shutdown_connections 2024-05-12T22:18:00.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.196+0000 7fae8f790740 1 --2- 172.21.15.28:0/1478303491 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ab127943c0 0x55ab127aba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:00.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.196+0000 7fae8f790740 1 --2- 172.21.15.28:0/1478303491 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ab127812a0 0x55ab12793860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:00.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.196+0000 7fae8f790740 1 --2- 172.21.15.28:0/1478303491 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ab127808f0 0x55ab1277b850 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:00.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.196+0000 7fae8f790740 1 -- 172.21.15.28:0/1478303491 >> 172.21.15.28:0/1478303491 conn(0x55ab1277cee0 msgr2=0x55ab12790050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:18:00.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.196+0000 7fae8f790740 1 -- 172.21.15.28:0/1478303491 shutdown_connections 2024-05-12T22:18:00.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.196+0000 7fae8f790740 1 -- 172.21.15.28:0/1478303491 wait complete. 2024-05-12T22:18:00.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.197+0000 7fae8f790740 1 -- start start 2024-05-12T22:18:00.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.197+0000 7fae8f790740 1 Processor -- start 2024-05-12T22:18:00.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.198+0000 7fae8f790740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ab127808f0 0x55ab1279c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:00.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.198+0000 7fae8f790740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ab127812a0 0x55ab12797c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:00.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.198+0000 7fae8f790740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ab127943c0 0x55ab127981b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:00.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.198+0000 7fae8f790740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fae841a9910 con 0x55ab127943c0 2024-05-12T22:18:00.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.198+0000 7fae8f790740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fae841aad80 con 0x55ab127812a0 2024-05-12T22:18:00.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.198+0000 7fae8f790740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fae841a9fe0 con 0x55ab127808f0 2024-05-12T22:18:00.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.198+0000 7fae8f790740 10 client.0 _ll_register_callbacks cb 0x7fae78009a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:18:00.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.198+0000 7fae89e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ab127808f0 0x55ab1279c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:00.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.198+0000 7fae89e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ab127808f0 0x55ab1279c8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:56626/0 (socket says 172.21.15.28:56626) 2024-05-12T22:18:00.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.198+0000 7fae89e00640 1 -- 172.21.15.28:0/2379168711 learned_addr learned my addr 172.21.15.28:0/2379168711 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:18:00.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.198+0000 7fae8a800640 1 --2- 172.21.15.28:0/2379168711 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ab127943c0 0x55ab127981b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:00.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.198+0000 7fae89400640 1 --2- 172.21.15.28:0/2379168711 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ab127812a0 0x55ab12797c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:00.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.198+0000 7fae89e00640 1 -- 172.21.15.28:0/2379168711 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ab127812a0 msgr2=0x55ab12797c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:18:00.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.198+0000 7fae89e00640 1 --2- 172.21.15.28:0/2379168711 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ab127812a0 0x55ab12797c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:00.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.198+0000 7fae89e00640 1 -- 172.21.15.28:0/2379168711 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ab127943c0 msgr2=0x55ab127981b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:18:00.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.198+0000 7fae89e00640 1 --2- 172.21.15.28:0/2379168711 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ab127943c0 0x55ab127981b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:00.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.198+0000 7fae89e00640 1 -- 172.21.15.28:0/2379168711 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ab127989e0 con 0x55ab127808f0 2024-05-12T22:18:00.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.199+0000 7fae89e00640 1 --2- 172.21.15.28:0/2379168711 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ab127808f0 0x55ab1279c8c0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fae780092e0 tx=0x7fae7801a8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=66e7e162675e3fc6 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:18:00.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.199+0000 7fae67400640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-12T22:18:00.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.199+0000 7fae67400640 1 -- 172.21.15.28:0/2379168711 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae78044060 con 0x55ab127808f0 2024-05-12T22:18:00.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.199+0000 7fae67400640 1 -- 172.21.15.28:0/2379168711 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8513+0+0 (secure 0 0 0) 0x7fae780035f0 con 0x55ab127808f0 2024-05-12T22:18:00.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.199+0000 7fae8f790740 1 -- 172.21.15.28:0/2379168711 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fae7801f040 con 0x55ab127808f0 2024-05-12T22:18:00.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.199+0000 7fae67400640 1 -- 172.21.15.28:0/2379168711 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fae78028040 con 0x55ab127808f0 2024-05-12T22:18:00.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.200+0000 7fae8f790740 10 client.27205 fetch_fsmap learned FSMap version 26 2024-05-12T22:18:00.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.200+0000 7fae8f790740 1 -- 172.21.15.28:0/2379168711 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55ab1280a330 con 0x55ab127808f0 2024-05-12T22:18:00.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.200+0000 7fae67400640 1 -- 172.21.15.28:0/2379168711 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fae7801f040 con 0x55ab127808f0 2024-05-12T22:18:00.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.200+0000 7fae67400640 1 -- 172.21.15.28:0/2379168711 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fae78018040 con 0x55ab127808f0 2024-05-12T22:18:00.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.200+0000 7fae8f790740 10 client.27205 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:18:00.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.200+0000 7fae8f790740 10 client.27205 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:18:00.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.200+0000 7fae8f790740 20 client.27205 populate_metadata read hostname 'smithi028' 2024-05-12T22:18:00.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.200+0000 7fae8f790740 1 --2- 172.21.15.28:0/2379168711 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55ab1280a930 0x55ab1280cd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:00.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.200+0000 7fae8f790740 4 client.27205 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-12T22:18:00.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.200+0000 7fae8f790740 1 -- 172.21.15.28:0/2379168711 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55ab126ed610 con 0x55ab1280a930 2024-05-12T22:18:00.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.201+0000 7fae89400640 1 --2- 172.21.15.28:0/2379168711 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55ab1280a930 0x55ab1280cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:00.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.201+0000 7fae89400640 1 --2- 172.21.15.28:0/2379168711 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55ab1280a930 0x55ab1280cd20 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bea297154e527d73 server_cookie=c854f42d61853ba2 in_seq=0 out_seq=0 2024-05-12T22:18:00.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.201+0000 7fae67400640 10 client.27205 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:18:00.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.202+0000 7fae67400640 1 -- 172.21.15.28:0/2379168711 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+268 (crc 0 0 0) 0x55ab126ed610 con 0x55ab1280a930 2024-05-12T22:18:00.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.202+0000 7fae67400640 10 client.27205 handle_command_reply: tid=0 2024-05-12T22:18:00.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.202+0000 7fae8f790740 1 client.27205 shutdown 2024-05-12T22:18:00.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.202+0000 7fae8f790740 1 -- 172.21.15.28:0/2379168711 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ab127808f0 msgr2=0x55ab1279c8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:18:00.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.202+0000 7fae8f790740 1 --2- 172.21.15.28:0/2379168711 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ab127808f0 0x55ab1279c8c0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fae780092e0 tx=0x7fae7801a8a0 comp rx=0 tx=0).stop 2024-05-12T22:18:00.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.202+0000 7fae8f790740 1 -- 172.21.15.28:0/2379168711 shutdown_connections 2024-05-12T22:18:00.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.202+0000 7fae8f790740 1 --2- 172.21.15.28:0/2379168711 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55ab1280a930 0x55ab1280cd20 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:00.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.202+0000 7fae8f790740 1 --2- 172.21.15.28:0/2379168711 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ab127943c0 0x55ab127981b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:00.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.202+0000 7fae8f790740 1 --2- 172.21.15.28:0/2379168711 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ab127812a0 0x55ab12797c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:00.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.202+0000 7fae8f790740 1 --2- 172.21.15.28:0/2379168711 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ab127808f0 0x55ab1279c8c0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:00.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.202+0000 7fae67400640 0 client.27205 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:18:00.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.202+0000 7fae8f790740 1 -- 172.21.15.28:0/2379168711 >> 172.21.15.28:0/2379168711 conn(0x55ab1277cee0 msgr2=0x55ab127824a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:18:00.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.202+0000 7fae8f790740 1 -- 172.21.15.28:0/2379168711 shutdown_connections 2024-05-12T22:18:00.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.203+0000 7fae8f790740 1 -- 172.21.15.28:0/2379168711 wait complete. 2024-05-12T22:18:00.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.203+0000 7fae8f790740 20 client.27205 trim_cache size 0 max 16384 2024-05-12T22:18:00.210 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-05-12T22:18:00.211 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "scrub active (1472 inodes in the stack)", 2024-05-12T22:18:00.211 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-05-12T22:18:00.211 INFO:teuthology.orchestra.run.smithi028.stdout: "552e6618-d080-47a8-9b9a-c74098705d4b": { 2024-05-12T22:18:00.211 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-05-12T22:18:00.211 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "552e6618-d080-47a8-9b9a-c74098705d4b", 2024-05-12T22:18:00.211 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-05-12T22:18:00.211 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:18:00.211 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:18:00.212 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-05-12T22:18:00.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.203+0000 7fae8c800640 1 -- 172.21.15.28:0/330829241 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fae6c0a2d80 msgr2=0x7fae6c0a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:18:00.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.203+0000 7fae8c800640 1 --2- 172.21.15.28:0/330829241 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fae6c0a2d80 0x7fae6c0a5240 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fae740096c0 tx=0x7fae7402a3f0 comp rx=0 tx=0).stop 2024-05-12T22:18:00.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.204+0000 7fae8c800640 1 -- 172.21.15.28:0/330829241 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fae84109a20 msgr2=0x7fae8419ec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:18:00.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.204+0000 7fae8c800640 1 --2- 172.21.15.28:0/330829241 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fae84109a20 0x7fae8419ec30 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fae80009fd0 tx=0x7fae8000ef10 comp rx=0 tx=0).stop 2024-05-12T22:18:00.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.204+0000 7fae8c800640 1 -- 172.21.15.28:0/330829241 shutdown_connections 2024-05-12T22:18:00.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.204+0000 7fae8c800640 1 --2- 172.21.15.28:0/330829241 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fae6c0a2d80 0x7fae6c0a5240 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:00.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.204+0000 7fae8c800640 1 --2- 172.21.15.28:0/330829241 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fae84109a20 0x7fae8419ec30 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:00.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.204+0000 7fae8c800640 1 --2- 172.21.15.28:0/330829241 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae841043d0 0x7fae8419a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:00.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.204+0000 7fae8c800640 1 --2- 172.21.15.28:0/330829241 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae84103a10 0x7fae8419a3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:00.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.204+0000 7fae8c800640 1 -- 172.21.15.28:0/330829241 >> 172.21.15.28:0/330829241 conn(0x7fae840ff4a0 msgr2=0x7fae840ffaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:18:00.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.204+0000 7fae8c800640 1 -- 172.21.15.28:0/330829241 shutdown_connections 2024-05-12T22:18:00.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:00.204+0000 7fae8c800640 1 -- 172.21.15.28:0/330829241 wait complete. 2024-05-12T22:18:00.226 INFO:tasks.cephfs.filesystem:scrub status for tag:552e6618-d080-47a8-9b9a-c74098705d4b - {'path': '/', 'tag': '552e6618-d080-47a8-9b9a-c74098705d4b', 'options': 'recursive,force'} 2024-05-12T22:18:00.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:18:00 smithi028 ceph-mon[29847]: pgmap v277: 97 pgs: 97 active+clean; 91 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:18:00.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:18:00 smithi028 ceph-mon[29847]: scrub summary: active paths [/] 2024-05-12T22:18:00.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:18:00 smithi028 ceph-mon[29847]: scrub summary: idle+waiting paths [/] 2024-05-12T22:18:00.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:18:00 smithi186 ceph-mon[34785]: pgmap v277: 97 pgs: 97 active+clean; 91 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:18:00.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:18:00 smithi186 ceph-mon[34785]: scrub summary: active paths [/] 2024-05-12T22:18:00.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:18:00 smithi186 ceph-mon[34785]: scrub summary: idle+waiting paths [/] 2024-05-12T22:18:00.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:18:00 smithi086 ceph-mon[34297]: pgmap v277: 97 pgs: 97 active+clean; 91 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:18:00.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:18:00 smithi086 ceph-mon[34297]: scrub summary: active paths [/] 2024-05-12T22:18:00.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:18:00 smithi086 ceph-mon[34297]: scrub summary: idle+waiting paths [/] 2024-05-12T22:18:01.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:18:01 smithi186 ceph-mon[34785]: scrub queued for path: / 2024-05-12T22:18:01.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:18:01 smithi186 ceph-mon[34785]: scrub summary: idle+waiting paths [/] 2024-05-12T22:18:01.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:18:01 smithi186 ceph-mon[34785]: scrub summary: active paths [/] 2024-05-12T22:18:01.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:18:01 smithi186 ceph-mon[34785]: scrub summary: active paths [/] 2024-05-12T22:18:01.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:18:01 smithi186 ceph-mon[34785]: scrub summary: idle+waiting paths [/] 2024-05-12T22:18:01.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:18:01 smithi186 ceph-mon[34785]: scrub summary: active paths [/] 2024-05-12T22:18:01.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:18:01 smithi186 ceph-mon[34785]: scrub summary: idle+waiting paths [/] 2024-05-12T22:18:01.430 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:18:01 smithi186 ceph-mon[34785]: scrub summary: idle+waiting paths [/] 2024-05-12T22:18:01.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:18:01 smithi086 ceph-mon[34297]: scrub queued for path: / 2024-05-12T22:18:01.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:18:01 smithi086 ceph-mon[34297]: scrub summary: idle+waiting paths [/] 2024-05-12T22:18:01.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:18:01 smithi086 ceph-mon[34297]: scrub summary: active paths [/] 2024-05-12T22:18:01.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:18:01 smithi086 ceph-mon[34297]: scrub summary: active paths [/] 2024-05-12T22:18:01.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:18:01 smithi086 ceph-mon[34297]: scrub summary: idle+waiting paths [/] 2024-05-12T22:18:01.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:18:01 smithi086 ceph-mon[34297]: scrub summary: active paths [/] 2024-05-12T22:18:01.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:18:01 smithi086 ceph-mon[34297]: scrub summary: idle+waiting paths [/] 2024-05-12T22:18:01.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:18:01 smithi086 ceph-mon[34297]: scrub summary: idle+waiting paths [/] 2024-05-12T22:18:01.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:18:01 smithi028 ceph-mon[29847]: scrub queued for path: / 2024-05-12T22:18:01.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:18:01 smithi028 ceph-mon[29847]: scrub summary: idle+waiting paths [/] 2024-05-12T22:18:01.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:18:01 smithi028 ceph-mon[29847]: scrub summary: active paths [/] 2024-05-12T22:18:01.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:18:01 smithi028 ceph-mon[29847]: scrub summary: active paths [/] 2024-05-12T22:18:01.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:18:01 smithi028 ceph-mon[29847]: scrub summary: idle+waiting paths [/] 2024-05-12T22:18:01.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:18:01 smithi028 ceph-mon[29847]: scrub summary: active paths [/] 2024-05-12T22:18:01.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:18:01 smithi028 ceph-mon[29847]: scrub summary: idle+waiting paths [/] 2024-05-12T22:18:01.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:18:01 smithi028 ceph-mon[29847]: scrub summary: idle+waiting paths [/] 2024-05-12T22:18:02.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:18:02 smithi186 ceph-mon[34785]: pgmap v278: 97 pgs: 97 active+clean; 91 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-12T22:18:02.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:18:02 smithi086 ceph-mon[34297]: pgmap v278: 97 pgs: 97 active+clean; 91 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-12T22:18:02.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:18:02 smithi028 ceph-mon[29847]: pgmap v278: 97 pgs: 97 active+clean; 91 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-12T22:18:04.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:18:04 smithi186 ceph-mon[34785]: pgmap v279: 97 pgs: 97 active+clean; 91 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:18:04.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:18:04 smithi086 ceph-mon[34297]: pgmap v279: 97 pgs: 97 active+clean; 91 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:18:04.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:18:04 smithi028 ceph-mon[29847]: pgmap v279: 97 pgs: 97 active+clean; 91 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:18:06.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:18:06 smithi186 ceph-mon[34785]: pgmap v280: 97 pgs: 97 active+clean; 91 MiB data, 717 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-05-12T22:18:06.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:18:06 smithi086 ceph-mon[34297]: pgmap v280: 97 pgs: 97 active+clean; 91 MiB data, 717 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-05-12T22:18:06.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:18:06 smithi028 ceph-mon[29847]: pgmap v280: 97 pgs: 97 active+clean; 91 MiB data, 717 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-05-12T22:18:08.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:18:08 smithi186 ceph-mon[34785]: pgmap v281: 97 pgs: 97 active+clean; 91 MiB data, 717 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-05-12T22:18:08.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:18:08 smithi186 ceph-mon[34785]: scrub summary: idle 2024-05-12T22:18:08.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:18:08 smithi186 ceph-mon[34785]: scrub summary: idle 2024-05-12T22:18:08.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:18:08 smithi086 ceph-mon[34297]: pgmap v281: 97 pgs: 97 active+clean; 91 MiB data, 717 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-05-12T22:18:08.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:18:08 smithi086 ceph-mon[34297]: scrub summary: idle 2024-05-12T22:18:08.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:18:08 smithi086 ceph-mon[34297]: scrub summary: idle 2024-05-12T22:18:08.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:18:08 smithi028 ceph-mon[29847]: pgmap v281: 97 pgs: 97 active+clean; 91 MiB data, 717 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-05-12T22:18:08.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:18:08 smithi028 ceph-mon[29847]: scrub summary: idle 2024-05-12T22:18:08.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:18:08 smithi028 ceph-mon[29847]: scrub summary: idle 2024-05-12T22:18:09.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:18:09 smithi186 ceph-mon[34785]: osdmap e94: 12 total, 12 up, 12 in 2024-05-12T22:18:09.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:18:09 smithi086 ceph-mon[34297]: osdmap e94: 12 total, 12 up, 12 in 2024-05-12T22:18:09.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:18:09 smithi028 ceph-mon[29847]: osdmap e94: 12 total, 12 up, 12 in 2024-05-12T22:18:10.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:18:10 smithi186 ceph-mon[34785]: pgmap v283: 97 pgs: 97 active+clean; 91 MiB data, 717 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-05-12T22:18:10.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:18:10 smithi086 ceph-mon[34297]: pgmap v283: 97 pgs: 97 active+clean; 91 MiB data, 717 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-05-12T22:18:10.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:18:10 smithi028 ceph-mon[29847]: pgmap v283: 97 pgs: 97 active+clean; 91 MiB data, 717 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-05-12T22:18:12.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:18:12 smithi186 ceph-mon[34785]: pgmap v284: 97 pgs: 97 active+clean; 91 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2024-05-12T22:18:12.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:18:12 smithi086 ceph-mon[34297]: pgmap v284: 97 pgs: 97 active+clean; 91 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2024-05-12T22:18:12.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:18:12 smithi028 ceph-mon[29847]: pgmap v284: 97 pgs: 97 active+clean; 91 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2024-05-12T22:18:14.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:18:14 smithi186 ceph-mon[34785]: pgmap v285: 97 pgs: 97 active+clean; 91 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2024-05-12T22:18:14.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:18:14 smithi186 ceph-mon[34785]: osdmap e95: 12 total, 12 up, 12 in 2024-05-12T22:18:14.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:18:14 smithi086 ceph-mon[34297]: pgmap v285: 97 pgs: 97 active+clean; 91 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2024-05-12T22:18:14.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:18:14 smithi086 ceph-mon[34297]: osdmap e95: 12 total, 12 up, 12 in 2024-05-12T22:18:14.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:18:14 smithi028 ceph-mon[29847]: pgmap v285: 97 pgs: 97 active+clean; 91 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2024-05-12T22:18:14.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:18:14 smithi028 ceph-mon[29847]: osdmap e95: 12 total, 12 up, 12 in 2024-05-12T22:18:16.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:18:16 smithi186 ceph-mon[34785]: pgmap v287: 97 pgs: 97 active+clean; 91 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2024-05-12T22:18:16.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:18:16 smithi086 ceph-mon[34297]: pgmap v287: 97 pgs: 97 active+clean; 91 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2024-05-12T22:18:16.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:18:16 smithi028 ceph-mon[29847]: pgmap v287: 97 pgs: 97 active+clean; 91 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2024-05-12T22:18:18.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:18:18 smithi186 ceph-mon[34785]: pgmap v288: 97 pgs: 97 active+clean; 91 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 232 B/s wr, 7 op/s 2024-05-12T22:18:18.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:18:18 smithi086 ceph-mon[34297]: pgmap v288: 97 pgs: 97 active+clean; 91 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 232 B/s wr, 7 op/s 2024-05-12T22:18:18.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:18:18 smithi028 ceph-mon[29847]: pgmap v288: 97 pgs: 97 active+clean; 91 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 232 B/s wr, 7 op/s 2024-05-12T22:18:20.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:18:20 smithi186 ceph-mon[34785]: pgmap v289: 97 pgs: 97 active+clean; 91 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-05-12T22:18:20.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:18:20 smithi086 ceph-mon[34297]: pgmap v289: 97 pgs: 97 active+clean; 91 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-05-12T22:18:20.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:18:20 smithi028 ceph-mon[29847]: pgmap v289: 97 pgs: 97 active+clean; 91 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-05-12T22:18:22.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:18:22 smithi028 ceph-mon[29847]: pgmap v290: 97 pgs: 97 active+clean; 91 MiB data, 725 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-05-12T22:18:22.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:18:22 smithi028 ceph-mon[29847]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:18:22.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:18:22 smithi186 ceph-mon[34785]: pgmap v290: 97 pgs: 97 active+clean; 91 MiB data, 725 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-05-12T22:18:22.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:18:22 smithi186 ceph-mon[34785]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:18:22.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:18:22 smithi086 ceph-mon[34297]: pgmap v290: 97 pgs: 97 active+clean; 91 MiB data, 725 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-05-12T22:18:22.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:18:22 smithi086 ceph-mon[34297]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:18:23.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:18:23 smithi186 ceph-mon[34785]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:18:23.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:18:23 smithi186 ceph-mon[34785]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:18:23.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:18:23 smithi186 ceph-mon[34785]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:18:23.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:18:23 smithi086 ceph-mon[34297]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:18:23.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:18:23 smithi086 ceph-mon[34297]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:18:23.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:18:23 smithi086 ceph-mon[34297]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:18:23.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:18:23 smithi028 ceph-mon[29847]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:18:23.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:18:23 smithi028 ceph-mon[29847]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:18:23.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:18:23 smithi028 ceph-mon[29847]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:18:24.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:18:24 smithi186 ceph-mon[34785]: pgmap v291: 97 pgs: 97 active+clean; 91 MiB data, 725 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-05-12T22:18:24.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:18:24 smithi086 ceph-mon[34297]: pgmap v291: 97 pgs: 97 active+clean; 91 MiB data, 725 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-05-12T22:18:24.613 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:18:24 smithi028 ceph-mon[29847]: pgmap v291: 97 pgs: 97 active+clean; 91 MiB data, 725 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-05-12T22:18:26.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:18:26 smithi186 ceph-mon[34785]: pgmap v292: 97 pgs: 97 active+clean; 91 MiB data, 725 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 93 B/s wr, 8 op/s 2024-05-12T22:18:26.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:18:26 smithi086 ceph-mon[34297]: pgmap v292: 97 pgs: 97 active+clean; 91 MiB data, 725 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 93 B/s wr, 8 op/s 2024-05-12T22:18:26.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:18:26 smithi028 ceph-mon[29847]: pgmap v292: 97 pgs: 97 active+clean; 91 MiB data, 725 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 93 B/s wr, 8 op/s 2024-05-12T22:18:28.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:18:28 smithi186 ceph-mon[34785]: pgmap v293: 97 pgs: 97 active+clean; 91 MiB data, 725 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:18:28.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:18:28 smithi086 ceph-mon[34297]: pgmap v293: 97 pgs: 97 active+clean; 91 MiB data, 725 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:18:28.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:18:28 smithi028 ceph-mon[29847]: pgmap v293: 97 pgs: 97 active+clean; 91 MiB data, 725 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:18:30.227 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-12T22:18:30.228 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-12T22:18:30.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.459+0000 7f4589a00640 1 -- 172.21.15.28:0/614774062 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f458410ddf0 msgr2=0x7f458410e1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:18:30.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.459+0000 7f4589a00640 1 --2- 172.21.15.28:0/614774062 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f458410ddf0 0x7f458410e1d0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f4574009860 tx=0x7f457402a3d0 comp rx=0 tx=0).stop 2024-05-12T22:18:30.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.459+0000 7f4589a00640 1 -- 172.21.15.28:0/614774062 shutdown_connections 2024-05-12T22:18:30.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.459+0000 7f4589a00640 1 --2- 172.21.15.28:0/614774062 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4584108770 0x7f4584114b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:30.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.459+0000 7f4589a00640 1 --2- 172.21.15.28:0/614774062 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4584107dd0 0x7f4584108230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:30.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.459+0000 7f4589a00640 1 --2- 172.21.15.28:0/614774062 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f458410ddf0 0x7f458410e1d0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:30.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.459+0000 7f4589a00640 1 -- 172.21.15.28:0/614774062 >> 172.21.15.28:0/614774062 conn(0x7f4584077650 msgr2=0x7f4584106650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:18:30.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.459+0000 7f4589a00640 1 -- 172.21.15.28:0/614774062 shutdown_connections 2024-05-12T22:18:30.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.459+0000 7f4589a00640 1 -- 172.21.15.28:0/614774062 wait complete. 2024-05-12T22:18:30.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.460+0000 7f4589a00640 1 Processor -- start 2024-05-12T22:18:30.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.460+0000 7f4589a00640 1 -- start start 2024-05-12T22:18:30.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.461+0000 7f4589a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4584107dd0 0x7f458419e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:30.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.461+0000 7f4589a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4584108770 0x7f458419ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:30.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.461+0000 7f4589a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f458410ddf0 0x7f45841a2e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:30.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.461+0000 7f4589a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4584118120 con 0x7f4584108770 2024-05-12T22:18:30.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.461+0000 7f4589a00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f4584117fa0 con 0x7f4584107dd0 2024-05-12T22:18:30.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.461+0000 7f4589a00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f45841182a0 con 0x7f458410ddf0 2024-05-12T22:18:30.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.461+0000 7f4583e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4584107dd0 0x7f458419e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:30.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.461+0000 7f4583e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4584107dd0 0x7f458419e540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:46592/0 (socket says 172.21.15.28:46592) 2024-05-12T22:18:30.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.461+0000 7f4583e00640 1 -- 172.21.15.28:0/3518879835 learned_addr learned my addr 172.21.15.28:0/3518879835 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:18:30.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.461+0000 7f457b400640 1 --2- 172.21.15.28:0/3518879835 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4584108770 0x7f458419ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:30.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.461+0000 7f4589000640 1 --2- 172.21.15.28:0/3518879835 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f458410ddf0 0x7f45841a2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:30.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.462+0000 7f4583e00640 1 -- 172.21.15.28:0/3518879835 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f458410ddf0 msgr2=0x7f45841a2e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:18:30.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.462+0000 7f4583e00640 1 --2- 172.21.15.28:0/3518879835 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f458410ddf0 0x7f45841a2e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:30.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.462+0000 7f4583e00640 1 -- 172.21.15.28:0/3518879835 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4584108770 msgr2=0x7f458419ea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:18:30.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.462+0000 7f4583e00640 1 --2- 172.21.15.28:0/3518879835 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4584108770 0x7f458419ea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:30.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.462+0000 7f4583e00640 1 -- 172.21.15.28:0/3518879835 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45841a3530 con 0x7f4584107dd0 2024-05-12T22:18:30.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.462+0000 7f4583e00640 1 --2- 172.21.15.28:0/3518879835 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4584107dd0 0x7f458419e540 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f4574009980 tx=0x7f45740309a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:18:30.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.462+0000 7f4581600640 1 -- 172.21.15.28:0/3518879835 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4574040070 con 0x7f4584107dd0 2024-05-12T22:18:30.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.462+0000 7f4589a00640 1 -- 172.21.15.28:0/3518879835 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45841a37c0 con 0x7f4584107dd0 2024-05-12T22:18:30.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.462+0000 7f4581600640 1 -- 172.21.15.28:0/3518879835 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4574005620 con 0x7f4584107dd0 2024-05-12T22:18:30.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.462+0000 7f4581600640 1 -- 172.21.15.28:0/3518879835 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4574004920 con 0x7f4584107dd0 2024-05-12T22:18:30.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.463+0000 7f4589a00640 1 -- 172.21.15.28:0/3518879835 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45841ab0b0 con 0x7f4584107dd0 2024-05-12T22:18:30.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.466+0000 7f458de67740 1 Processor -- start 2024-05-12T22:18:30.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.466+0000 7f4581600640 1 -- 172.21.15.28:0/3518879835 <== mon.2 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7f4574043070 con 0x7f4584107dd0 2024-05-12T22:18:30.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.466+0000 7f458de67740 1 -- start start 2024-05-12T22:18:30.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.466+0000 7f458de67740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5638495f7dd0 0x5638495f81b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:30.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.466+0000 7f458de67740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5638495f86f0 0x563849602880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:30.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.466+0000 7f458de67740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563849602dc0 0x5638496051b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:30.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.466+0000 7f458de67740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5638495c02a0 con 0x5638495f86f0 2024-05-12T22:18:30.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.466+0000 7f458de67740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x56384955cb90 con 0x5638495f7dd0 2024-05-12T22:18:30.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.466+0000 7f458de67740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5638495045b0 con 0x563849602dc0 2024-05-12T22:18:30.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.467+0000 7f4583e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5638495f86f0 0x563849602880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:30.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.467+0000 7f4589000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5638495f7dd0 0x5638495f81b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:30.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.467+0000 7f4589000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5638495f7dd0 0x5638495f81b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:46608/0 (socket says 172.21.15.28:46608) 2024-05-12T22:18:30.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.467+0000 7f4589000640 1 -- 172.21.15.28:0/3264851277 learned_addr learned my addr 172.21.15.28:0/3264851277 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:18:30.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.467+0000 7f457b400640 1 --2- 172.21.15.28:0/3264851277 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563849602dc0 0x5638496051b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:30.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.467+0000 7f4589000640 1 -- 172.21.15.28:0/3264851277 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563849602dc0 msgr2=0x5638496051b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:18:30.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.467+0000 7f4589000640 1 --2- 172.21.15.28:0/3264851277 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563849602dc0 0x5638496051b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:30.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.467+0000 7f4589000640 1 -- 172.21.15.28:0/3264851277 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5638495f86f0 msgr2=0x563849602880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:18:30.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.467+0000 7f4589000640 1 --2- 172.21.15.28:0/3264851277 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5638495f86f0 0x563849602880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:30.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.467+0000 7f4589000640 1 -- 172.21.15.28:0/3264851277 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56384933a7d0 con 0x5638495f7dd0 2024-05-12T22:18:30.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.467+0000 7f4583e00640 1 --2- 172.21.15.28:0/3264851277 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5638495f86f0 0x563849602880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:18:30.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.467+0000 7f4589000640 1 --2- 172.21.15.28:0/3264851277 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5638495f7dd0 0x5638495f81b0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f457c009fc0 tx=0x7f457c0118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=be8821a33c933464 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:18:30.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.468+0000 7f4581600640 1 --2- 172.21.15.28:0/3518879835 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f45680a2d80 0x7f45680a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:30.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.468+0000 7f4579600640 1 -- 172.21.15.28:0/3264851277 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f457c004d20 con 0x5638495f7dd0 2024-05-12T22:18:30.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.468+0000 7f4579600640 1 -- 172.21.15.28:0/3264851277 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f457c004050 con 0x5638495f7dd0 2024-05-12T22:18:30.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.468+0000 7f4579600640 1 -- 172.21.15.28:0/3264851277 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f457c005260 con 0x5638495f7dd0 2024-05-12T22:18:30.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.468+0000 7f4581600640 1 -- 172.21.15.28:0/3518879835 <== mon.2 v2:172.21.15.86:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8513+0+0 (secure 0 0 0) 0x7f45740f0000 con 0x7f4584107dd0 2024-05-12T22:18:30.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.468+0000 7f4583e00640 1 --2- 172.21.15.28:0/3518879835 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f45680a2d80 0x7f45680a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:30.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.468+0000 7f458de67740 1 -- 172.21.15.28:0/3264851277 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5638495f7dd0 msgr2=0x5638495f81b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:18:30.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.468+0000 7f458de67740 1 --2- 172.21.15.28:0/3264851277 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5638495f7dd0 0x5638495f81b0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f457c009fc0 tx=0x7f457c0118d0 comp rx=0 tx=0).stop 2024-05-12T22:18:30.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.469+0000 7f458de67740 1 -- 172.21.15.28:0/3264851277 shutdown_connections 2024-05-12T22:18:30.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.469+0000 7f458de67740 1 --2- 172.21.15.28:0/3264851277 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563849602dc0 0x5638496051b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:30.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.469+0000 7f458de67740 1 --2- 172.21.15.28:0/3264851277 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5638495f86f0 0x563849602880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:30.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.469+0000 7f458de67740 1 --2- 172.21.15.28:0/3264851277 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5638495f7dd0 0x5638495f81b0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:30.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.469+0000 7f458de67740 1 -- 172.21.15.28:0/3264851277 >> 172.21.15.28:0/3264851277 conn(0x5638495f40f0 msgr2=0x5638495f44d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:18:30.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.469+0000 7f4583e00640 1 --2- 172.21.15.28:0/3518879835 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f45680a2d80 0x7f45680a5240 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f45740ed800 tx=0x7f45740f3b50 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:18:30.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.469+0000 7f458de67740 1 -- 172.21.15.28:0/3264851277 shutdown_connections 2024-05-12T22:18:30.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.469+0000 7f458de67740 1 -- 172.21.15.28:0/3264851277 wait complete. 2024-05-12T22:18:30.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.469+0000 7f458de67740 1 -- start start 2024-05-12T22:18:30.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.469+0000 7f458de67740 1 Processor -- start 2024-05-12T22:18:30.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.470+0000 7f458de67740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5638495f7dd0 0x5638496026f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:30.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.470+0000 7f458de67740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5638495f86f0 0x5638495ff9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:30.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.470+0000 7f458de67740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563849602dc0 0x5638495fff20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:30.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.470+0000 7f458de67740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5638495179f0 con 0x5638495f7dd0 2024-05-12T22:18:30.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.470+0000 7f458de67740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x563849558b50 con 0x563849602dc0 2024-05-12T22:18:30.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.470+0000 7f458de67740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x563849605bb0 con 0x5638495f86f0 2024-05-12T22:18:30.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.470+0000 7f458de67740 10 client.0 _ll_register_callbacks cb 0x56384926fbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:18:30.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.470+0000 7f457b400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5638495f86f0 0x5638495ff9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:30.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.470+0000 7f457b400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5638495f86f0 0x5638495ff9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:56220/0 (socket says 172.21.15.28:56220) 2024-05-12T22:18:30.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.470+0000 7f457b400640 1 -- 172.21.15.28:0/3741712905 learned_addr learned my addr 172.21.15.28:0/3741712905 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:18:30.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.470+0000 7f4589000640 1 --2- 172.21.15.28:0/3741712905 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5638495f7dd0 0x5638496026f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:30.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.470+0000 7f4589000640 1 --2- 172.21.15.28:0/3741712905 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563849602dc0 0x5638495fff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:30.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.470+0000 7f457b400640 1 -- 172.21.15.28:0/3741712905 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563849602dc0 msgr2=0x5638495fff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:18:30.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.470+0000 7f457b400640 1 --2- 172.21.15.28:0/3741712905 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563849602dc0 0x5638495fff20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:30.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.471+0000 7f457b400640 1 -- 172.21.15.28:0/3741712905 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5638495f7dd0 msgr2=0x5638496026f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:18:30.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.471+0000 7f457b400640 1 --2- 172.21.15.28:0/3741712905 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5638495f7dd0 0x5638496026f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:30.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.471+0000 7f457b400640 1 -- 172.21.15.28:0/3741712905 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5638492703f0 con 0x5638495f86f0 2024-05-12T22:18:30.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.471+0000 7f4589000640 1 --2- 172.21.15.28:0/3741712905 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5638495f7dd0 0x5638496026f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:18:30.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.471+0000 7f457b400640 1 --2- 172.21.15.28:0/3741712905 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5638495f86f0 0x5638495ff9e0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f456c0116f0 tx=0x7f456c011bb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a16033155ff535ca server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:18:30.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.471+0000 7f4572000640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-12T22:18:30.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.471+0000 7f4572000640 1 -- 172.21.15.28:0/3741712905 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f456c00fbf0 con 0x5638495f86f0 2024-05-12T22:18:30.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.471+0000 7f4572000640 1 -- 172.21.15.28:0/3741712905 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8513+0+0 (secure 0 0 0) 0x7f456c017050 con 0x5638495f86f0 2024-05-12T22:18:30.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.471+0000 7f458de67740 1 -- 172.21.15.28:0/3741712905 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f457c004050 con 0x5638495f86f0 2024-05-12T22:18:30.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.472+0000 7f4572000640 1 -- 172.21.15.28:0/3741712905 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f456c027050 con 0x5638495f86f0 2024-05-12T22:18:30.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.472+0000 7f458de67740 10 client.27217 fetch_fsmap learned FSMap version 26 2024-05-12T22:18:30.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.472+0000 7f458de67740 1 -- 172.21.15.28:0/3741712905 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5638492300f0 con 0x5638495f86f0 2024-05-12T22:18:30.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.472+0000 7f4572000640 1 -- 172.21.15.28:0/3741712905 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f457c004050 con 0x5638495f86f0 2024-05-12T22:18:30.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.472+0000 7f4572000640 1 -- 172.21.15.28:0/3741712905 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f456c022030 con 0x5638495f86f0 2024-05-12T22:18:30.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.472+0000 7f458de67740 10 client.27217 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:18:30.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.472+0000 7f458de67740 10 client.27217 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:18:30.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.472+0000 7f458de67740 20 client.27217 populate_metadata read hostname 'smithi028' 2024-05-12T22:18:30.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.472+0000 7f458de67740 1 --2- 172.21.15.28:0/3741712905 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x563849612a30 0x563849614eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:30.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.472+0000 7f458de67740 4 client.27217 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-12T22:18:30.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.472+0000 7f458de67740 1 -- 172.21.15.28:0/3741712905 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56384956a2c0 con 0x563849612a30 2024-05-12T22:18:30.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.473+0000 7f4589000640 1 --2- 172.21.15.28:0/3741712905 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x563849612a30 0x563849614eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:30.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.473+0000 7f4589000640 1 --2- 172.21.15.28:0/3741712905 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x563849612a30 0x563849614eb0 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e6b5121d589fd22a server_cookie=48198c986a55fe23 in_seq=0 out_seq=0 2024-05-12T22:18:30.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.473+0000 7f4572000640 10 client.27217 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:18:30.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.474+0000 7f4572000640 1 -- 172.21.15.28:0/3741712905 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56384956a2c0 con 0x563849612a30 2024-05-12T22:18:30.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.474+0000 7f4572000640 10 client.27217 handle_command_reply: tid=0 2024-05-12T22:18:30.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.474+0000 7f458de67740 1 client.27217 shutdown 2024-05-12T22:18:30.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.474+0000 7f458de67740 1 -- 172.21.15.28:0/3741712905 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5638495f86f0 msgr2=0x5638495ff9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:18:30.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.474+0000 7f458de67740 1 --2- 172.21.15.28:0/3741712905 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5638495f86f0 0x5638495ff9e0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f456c0116f0 tx=0x7f456c011bb0 comp rx=0 tx=0).stop 2024-05-12T22:18:30.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.474+0000 7f458de67740 1 -- 172.21.15.28:0/3741712905 shutdown_connections 2024-05-12T22:18:30.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.474+0000 7f458de67740 1 --2- 172.21.15.28:0/3741712905 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x563849612a30 0x563849614eb0 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:30.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.474+0000 7f458de67740 1 --2- 172.21.15.28:0/3741712905 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563849602dc0 0x5638495fff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:30.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.474+0000 7f458de67740 1 --2- 172.21.15.28:0/3741712905 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5638495f86f0 0x5638495ff9e0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:30.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.474+0000 7f458de67740 1 --2- 172.21.15.28:0/3741712905 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5638495f7dd0 0x5638496026f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:30.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.474+0000 7f458de67740 1 -- 172.21.15.28:0/3741712905 >> 172.21.15.28:0/3741712905 conn(0x5638495f40f0 msgr2=0x5638495f3330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:18:30.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.474+0000 7f4572000640 0 client.27217 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:18:30.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.474+0000 7f458de67740 1 -- 172.21.15.28:0/3741712905 shutdown_connections 2024-05-12T22:18:30.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.474+0000 7f458de67740 1 -- 172.21.15.28:0/3741712905 wait complete. 2024-05-12T22:18:30.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.475+0000 7f458de67740 20 client.27217 trim_cache size 0 max 16384 2024-05-12T22:18:30.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:18:30 smithi086 ceph-mon[34297]: pgmap v294: 97 pgs: 97 active+clean; 91 MiB data, 725 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:18:30.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.494+0000 7f458de67740 1 Processor -- start 2024-05-12T22:18:30.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.494+0000 7f458de67740 1 -- start start 2024-05-12T22:18:30.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.494+0000 7f458de67740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5638495f7dd0 0x5638495faf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:30.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.494+0000 7f458de67740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5638495f86f0 0x56384960e920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:30.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.494+0000 7f458de67740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563849602dc0 0x563849602220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:30.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.494+0000 7f458de67740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f45841b6150 con 0x5638495f7dd0 2024-05-12T22:18:30.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.494+0000 7f458de67740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f45841b3a20 con 0x563849602dc0 2024-05-12T22:18:30.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.494+0000 7f458de67740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f45841b1890 con 0x5638495f86f0 2024-05-12T22:18:30.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.495+0000 7f4589000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5638495f7dd0 0x5638495faf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:30.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.495+0000 7f457b400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5638495f86f0 0x56384960e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:30.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.495+0000 7f4589000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5638495f7dd0 0x5638495faf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48792/0 (socket says 172.21.15.28:48792) 2024-05-12T22:18:30.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.495+0000 7f4589000640 1 -- 172.21.15.28:0/1738944046 learned_addr learned my addr 172.21.15.28:0/1738944046 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:18:30.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.495+0000 7f4589000640 1 --2- 172.21.15.28:0/1738944046 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563849602dc0 0x563849602220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:30.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.495+0000 7f457b400640 1 -- 172.21.15.28:0/1738944046 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563849602dc0 msgr2=0x563849602220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:18:30.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.495+0000 7f457b400640 1 --2- 172.21.15.28:0/1738944046 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563849602dc0 0x563849602220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:30.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.495+0000 7f457b400640 1 -- 172.21.15.28:0/1738944046 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5638495f7dd0 msgr2=0x5638495faf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:18:30.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.495+0000 7f457b400640 1 --2- 172.21.15.28:0/1738944046 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5638495f7dd0 0x5638495faf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:30.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.495+0000 7f457b400640 1 -- 172.21.15.28:0/1738944046 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5638495f8520 con 0x5638495f86f0 2024-05-12T22:18:30.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.495+0000 7f4589000640 1 --2- 172.21.15.28:0/1738944046 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5638495f7dd0 0x5638495faf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:18:30.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.496+0000 7f457b400640 1 --2- 172.21.15.28:0/1738944046 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5638495f86f0 0x56384960e920 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f456c001eb0 tx=0x7f456c0057f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8828f8d3d32e7ed8 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:18:30.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.496+0000 7f4578c00640 1 -- 172.21.15.28:0/1738944046 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f456c000e60 con 0x5638495f86f0 2024-05-12T22:18:30.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.496+0000 7f4578c00640 1 -- 172.21.15.28:0/1738944046 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f456c019620 con 0x5638495f86f0 2024-05-12T22:18:30.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.496+0000 7f4578c00640 1 -- 172.21.15.28:0/1738944046 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f456c0051a0 con 0x5638495f86f0 2024-05-12T22:18:30.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.497+0000 7f458de67740 1 -- 172.21.15.28:0/1738944046 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5638495f86f0 msgr2=0x56384960e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:18:30.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.497+0000 7f458de67740 1 --2- 172.21.15.28:0/1738944046 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5638495f86f0 0x56384960e920 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f456c001eb0 tx=0x7f456c0057f0 comp rx=0 tx=0).stop 2024-05-12T22:18:30.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.497+0000 7f458de67740 1 -- 172.21.15.28:0/1738944046 shutdown_connections 2024-05-12T22:18:30.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.497+0000 7f458de67740 1 --2- 172.21.15.28:0/1738944046 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563849602dc0 0x563849602220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:30.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.497+0000 7f458de67740 1 --2- 172.21.15.28:0/1738944046 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5638495f86f0 0x56384960e920 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:30.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.497+0000 7f458de67740 1 --2- 172.21.15.28:0/1738944046 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5638495f7dd0 0x5638495faf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:30.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.497+0000 7f458de67740 1 -- 172.21.15.28:0/1738944046 >> 172.21.15.28:0/1738944046 conn(0x5638495f40f0 msgr2=0x5638495fe920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:18:30.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.497+0000 7f458de67740 1 -- 172.21.15.28:0/1738944046 shutdown_connections 2024-05-12T22:18:30.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.497+0000 7f458de67740 1 -- 172.21.15.28:0/1738944046 wait complete. 2024-05-12T22:18:30.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.497+0000 7f458de67740 1 -- start start 2024-05-12T22:18:30.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.498+0000 7f458de67740 1 Processor -- start 2024-05-12T22:18:30.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.498+0000 7f458de67740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5638495f7dd0 0x56384960b020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:30.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.498+0000 7f458de67740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5638495f86f0 0x5638496063f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:30.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.498+0000 7f458de67740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563849602dc0 0x563849606930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:30.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.498+0000 7f458de67740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f45841b1ee0 con 0x5638495f86f0 2024-05-12T22:18:30.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.498+0000 7f458de67740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f45841b3350 con 0x563849602dc0 2024-05-12T22:18:30.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.498+0000 7f458de67740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f45841b25b0 con 0x5638495f7dd0 2024-05-12T22:18:30.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.498+0000 7f458de67740 10 client.0 _ll_register_callbacks cb 0x7f456c009890 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:18:30.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.498+0000 7f457b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5638495f86f0 0x5638496063f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:30.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.498+0000 7f4589000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5638495f7dd0 0x56384960b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:30.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.498+0000 7f457b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5638495f86f0 0x5638496063f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48800/0 (socket says 172.21.15.28:48800) 2024-05-12T22:18:30.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.498+0000 7f457b400640 1 -- 172.21.15.28:0/4010914116 learned_addr learned my addr 172.21.15.28:0/4010914116 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:18:30.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.499+0000 7f4589000640 1 --2- 172.21.15.28:0/4010914116 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563849602dc0 0x563849606930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:30.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.499+0000 7f457b400640 1 -- 172.21.15.28:0/4010914116 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5638495f7dd0 msgr2=0x56384960b020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:18:30.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.499+0000 7f457b400640 1 --2- 172.21.15.28:0/4010914116 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5638495f7dd0 0x56384960b020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:30.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.499+0000 7f457b400640 1 -- 172.21.15.28:0/4010914116 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563849602dc0 msgr2=0x563849606930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:18:30.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.499+0000 7f457b400640 1 --2- 172.21.15.28:0/4010914116 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563849602dc0 0x563849606930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:30.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.499+0000 7f457b400640 1 -- 172.21.15.28:0/4010914116 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563849606f70 con 0x5638495f86f0 2024-05-12T22:18:30.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.499+0000 7f457b400640 1 --2- 172.21.15.28:0/4010914116 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5638495f86f0 0x5638496063f0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f456c0194d0 tx=0x7f456c009600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38289f7005552347 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:18:30.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.499+0000 7f4572000640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-12T22:18:30.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.499+0000 7f4572000640 1 -- 172.21.15.28:0/4010914116 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f456c038060 con 0x5638495f86f0 2024-05-12T22:18:30.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.499+0000 7f4572000640 1 -- 172.21.15.28:0/4010914116 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8513+0+0 (secure 0 0 0) 0x7f456c011e00 con 0x5638495f86f0 2024-05-12T22:18:30.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.499+0000 7f458de67740 1 -- 172.21.15.28:0/4010914116 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f456c019620 con 0x5638495f86f0 2024-05-12T22:18:30.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.500+0000 7f4572000640 1 -- 172.21.15.28:0/4010914116 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f456c005570 con 0x5638495f86f0 2024-05-12T22:18:30.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.500+0000 7f458de67740 10 client.17136 fetch_fsmap learned FSMap version 26 2024-05-12T22:18:30.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.500+0000 7f458de67740 1 -- 172.21.15.28:0/4010914116 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x563849678e30 con 0x5638495f86f0 2024-05-12T22:18:30.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.500+0000 7f4572000640 1 -- 172.21.15.28:0/4010914116 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f456c019620 con 0x5638495f86f0 2024-05-12T22:18:30.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.500+0000 7f4572000640 1 -- 172.21.15.28:0/4010914116 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f456c022070 con 0x5638495f86f0 2024-05-12T22:18:30.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.500+0000 7f458de67740 10 client.17136 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:18:30.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.500+0000 7f458de67740 10 client.17136 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:18:30.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.500+0000 7f458de67740 20 client.17136 populate_metadata read hostname 'smithi028' 2024-05-12T22:18:30.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.500+0000 7f458de67740 1 --2- 172.21.15.28:0/4010914116 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x563849679510 0x56384967b930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:30.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.500+0000 7f458de67740 4 client.17136 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-12T22:18:30.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.500+0000 7f458de67740 1 -- 172.21.15.28:0/4010914116 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "scrub status"}) -- 0x563849564610 con 0x563849679510 2024-05-12T22:18:30.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.501+0000 7f4589000640 1 --2- 172.21.15.28:0/4010914116 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x563849679510 0x56384967b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:30.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.501+0000 7f4589000640 1 --2- 172.21.15.28:0/4010914116 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x563849679510 0x56384967b930 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=71c62edccd465fd5 server_cookie=b10c1c48dffc4ae6 in_seq=0 out_seq=0 2024-05-12T22:18:30.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.501+0000 7f4572000640 10 client.17136 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:18:30.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.502+0000 7f4572000640 1 -- 172.21.15.28:0/4010914116 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x563849564610 con 0x563849679510 2024-05-12T22:18:30.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.502+0000 7f4572000640 10 client.17136 handle_command_reply: tid=0 2024-05-12T22:18:30.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.502+0000 7f458de67740 1 client.17136 shutdown 2024-05-12T22:18:30.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.502+0000 7f458de67740 1 -- 172.21.15.28:0/4010914116 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5638495f86f0 msgr2=0x5638496063f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:18:30.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.502+0000 7f458de67740 1 --2- 172.21.15.28:0/4010914116 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5638495f86f0 0x5638496063f0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f456c0194d0 tx=0x7f456c009600 comp rx=0 tx=0).stop 2024-05-12T22:18:30.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.502+0000 7f458de67740 1 -- 172.21.15.28:0/4010914116 shutdown_connections 2024-05-12T22:18:30.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.502+0000 7f458de67740 1 --2- 172.21.15.28:0/4010914116 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x563849679510 0x56384967b930 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:30.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.502+0000 7f458de67740 1 --2- 172.21.15.28:0/4010914116 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563849602dc0 0x563849606930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:30.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.502+0000 7f458de67740 1 --2- 172.21.15.28:0/4010914116 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5638495f86f0 0x5638496063f0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:30.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.502+0000 7f458de67740 1 --2- 172.21.15.28:0/4010914116 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5638495f7dd0 0x56384960b020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:30.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.502+0000 7f458de67740 1 -- 172.21.15.28:0/4010914116 >> 172.21.15.28:0/4010914116 conn(0x5638495f40f0 msgr2=0x5638495f9db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:18:30.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.502+0000 7f4572000640 0 client.17136 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:18:30.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.502+0000 7f458de67740 1 -- 172.21.15.28:0/4010914116 shutdown_connections 2024-05-12T22:18:30.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.502+0000 7f458de67740 1 -- 172.21.15.28:0/4010914116 wait complete. 2024-05-12T22:18:30.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.502+0000 7f458de67740 20 client.17136 trim_cache size 0 max 16384 2024-05-12T22:18:30.512 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-05-12T22:18:30.512 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-05-12T22:18:30.512 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-05-12T22:18:30.512 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-05-12T22:18:30.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.503+0000 7f4589a00640 1 -- 172.21.15.28:0/3518879835 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f45680a2d80 msgr2=0x7f45680a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:18:30.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.503+0000 7f4589a00640 1 --2- 172.21.15.28:0/3518879835 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f45680a2d80 0x7f45680a5240 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f45740ed800 tx=0x7f45740f3b50 comp rx=0 tx=0).stop 2024-05-12T22:18:30.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.503+0000 7f4589a00640 1 -- 172.21.15.28:0/3518879835 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4584107dd0 msgr2=0x7f458419e540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:18:30.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.503+0000 7f4589a00640 1 --2- 172.21.15.28:0/3518879835 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4584107dd0 0x7f458419e540 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f4574009980 tx=0x7f45740309a0 comp rx=0 tx=0).stop 2024-05-12T22:18:30.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.503+0000 7f4589a00640 1 -- 172.21.15.28:0/3518879835 shutdown_connections 2024-05-12T22:18:30.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.503+0000 7f4589a00640 1 --2- 172.21.15.28:0/3518879835 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f45680a2d80 0x7f45680a5240 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:30.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.503+0000 7f4589a00640 1 --2- 172.21.15.28:0/3518879835 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f458410ddf0 0x7f45841a2e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:30.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.503+0000 7f4589a00640 1 --2- 172.21.15.28:0/3518879835 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4584108770 0x7f458419ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:30.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.503+0000 7f4589a00640 1 --2- 172.21.15.28:0/3518879835 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4584107dd0 0x7f458419e540 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:30.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.503+0000 7f4589a00640 1 -- 172.21.15.28:0/3518879835 >> 172.21.15.28:0/3518879835 conn(0x7f4584077650 msgr2=0x7f45840785e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:18:30.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.503+0000 7f4589a00640 1 -- 172.21.15.28:0/3518879835 shutdown_connections 2024-05-12T22:18:30.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.503+0000 7f4589a00640 1 -- 172.21.15.28:0/3518879835 wait complete. 2024-05-12T22:18:30.518 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:18:30 smithi028 ceph-mon[29847]: pgmap v294: 97 pgs: 97 active+clean; 91 MiB data, 725 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:18:30.522 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-12T22:18:30.523 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-12T22:18:30.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:18:30 smithi186 ceph-mon[34785]: pgmap v294: 97 pgs: 97 active+clean; 91 MiB data, 725 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:18:30.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.731+0000 7fb08fe00640 1 -- 172.21.15.28:0/626974741 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb090108670 msgr2=0x7fb090114ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:18:30.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.731+0000 7fb08fe00640 1 --2- 172.21.15.28:0/626974741 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb090108670 0x7fb090114ae0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fb088009ca0 tx=0x7fb08802a950 comp rx=0 tx=0).stop 2024-05-12T22:18:30.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.731+0000 7fb08fe00640 1 -- 172.21.15.28:0/626974741 shutdown_connections 2024-05-12T22:18:30.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.731+0000 7fb08fe00640 1 --2- 172.21.15.28:0/626974741 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb090108670 0x7fb090114ae0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:30.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.731+0000 7fb08fe00640 1 --2- 172.21.15.28:0/626974741 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb090107cd0 0x7fb090108130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:30.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.731+0000 7fb08fe00640 1 --2- 172.21.15.28:0/626974741 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb09010dd40 0x7fb09010e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:30.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.731+0000 7fb08fe00640 1 -- 172.21.15.28:0/626974741 >> 172.21.15.28:0/626974741 conn(0x7fb0900773b0 msgr2=0x7fb0900797d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:18:30.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.731+0000 7fb08fe00640 1 -- 172.21.15.28:0/626974741 shutdown_connections 2024-05-12T22:18:30.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.731+0000 7fb08fe00640 1 -- 172.21.15.28:0/626974741 wait complete. 2024-05-12T22:18:30.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.733+0000 7fb08fe00640 1 Processor -- start 2024-05-12T22:18:30.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.733+0000 7fb08fe00640 1 -- start start 2024-05-12T22:18:30.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.733+0000 7fb08fe00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb090107cd0 0x7fb09019e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:30.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.733+0000 7fb08fe00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb090108670 0x7fb09019e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:30.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.733+0000 7fb08fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb09010dd40 0x7fb0901a2ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:30.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.733+0000 7fb08fe00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb090118020 con 0x7fb09010dd40 2024-05-12T22:18:30.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.733+0000 7fb08fe00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fb090117ea0 con 0x7fb090107cd0 2024-05-12T22:18:30.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.733+0000 7fb08fe00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb0901181a0 con 0x7fb090108670 2024-05-12T22:18:30.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.734+0000 7fb08ea00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb090107cd0 0x7fb09019e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:30.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.734+0000 7fb08f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb09010dd40 0x7fb0901a2ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:30.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.734+0000 7fb08ea00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb090107cd0 0x7fb09019e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:46638/0 (socket says 172.21.15.28:46638) 2024-05-12T22:18:30.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.734+0000 7fb08ea00640 1 -- 172.21.15.28:0/1391184104 learned_addr learned my addr 172.21.15.28:0/1391184104 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:18:30.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.734+0000 7fb08e000640 1 --2- 172.21.15.28:0/1391184104 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb090108670 0x7fb09019e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:30.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.734+0000 7fb08ea00640 1 -- 172.21.15.28:0/1391184104 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb090108670 msgr2=0x7fb09019e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:18:30.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.734+0000 7fb08ea00640 1 --2- 172.21.15.28:0/1391184104 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb090108670 0x7fb09019e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:30.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.734+0000 7fb08ea00640 1 -- 172.21.15.28:0/1391184104 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb09010dd40 msgr2=0x7fb0901a2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:18:30.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.734+0000 7fb08ea00640 1 --2- 172.21.15.28:0/1391184104 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb09010dd40 0x7fb0901a2ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:30.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.734+0000 7fb08ea00640 1 -- 172.21.15.28:0/1391184104 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0901a3410 con 0x7fb090107cd0 2024-05-12T22:18:30.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.734+0000 7fb08ea00640 1 --2- 172.21.15.28:0/1391184104 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb090107cd0 0x7fb09019e420 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fb08000ea60 tx=0x7fb08000ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:18:30.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.735+0000 7fb094a00640 1 -- 172.21.15.28:0/1391184104 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb08000cde0 con 0x7fb090107cd0 2024-05-12T22:18:30.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.735+0000 7fb094a00640 1 -- 172.21.15.28:0/1391184104 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb0800035f0 con 0x7fb090107cd0 2024-05-12T22:18:30.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.735+0000 7fb08fe00640 1 -- 172.21.15.28:0/1391184104 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0901a3700 con 0x7fb090107cd0 2024-05-12T22:18:30.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.735+0000 7fb094a00640 1 -- 172.21.15.28:0/1391184104 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb080010670 con 0x7fb090107cd0 2024-05-12T22:18:30.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.735+0000 7fb08fe00640 1 -- 172.21.15.28:0/1391184104 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0901aaf90 con 0x7fb090107cd0 2024-05-12T22:18:30.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.736+0000 7fb08fe00640 1 -- 172.21.15.28:0/1391184104 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0900689c0 con 0x7fb090107cd0 2024-05-12T22:18:30.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.738+0000 7fb094a00640 1 -- 172.21.15.28:0/1391184104 <== mon.2 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7fb0800037a0 con 0x7fb090107cd0 2024-05-12T22:18:30.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.739+0000 7fb094a00640 1 --2- 172.21.15.28:0/1391184104 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fb0740a2d00 0x7fb0740a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:30.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.740+0000 7fb094a00640 1 -- 172.21.15.28:0/1391184104 <== mon.2 v2:172.21.15.86:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8513+0+0 (secure 0 0 0) 0x7fb0800cf730 con 0x7fb090107cd0 2024-05-12T22:18:30.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.740+0000 7fb08e000640 1 --2- 172.21.15.28:0/1391184104 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fb0740a2d00 0x7fb0740a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:30.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.741+0000 7fb08e000640 1 --2- 172.21.15.28:0/1391184104 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fb0740a2d00 0x7fb0740a51c0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fb09010d370 tx=0x7fb07c005f90 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:18:30.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.742+0000 7fb094a00640 1 -- 172.21.15.28:0/1391184104 <== mon.2 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fb08009c330 con 0x7fb090107cd0 2024-05-12T22:18:30.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.991+0000 7fb08fe00640 1 -- 172.21.15.28:0/1391184104 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb09019f770 con 0x7fb090107cd0 2024-05-12T22:18:30.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.992+0000 7fb094a00640 1 -- 172.21.15.28:0/1391184104 <== mon.2 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11864 (secure 0 0 0) 0x7fb08009bc30 con 0x7fb090107cd0 2024-05-12T22:18:30.992 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-12T22:18:30.993 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1844141745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1844141745},{"type":"v1","addr":"172.21.15.186:6833","nonce":1844141745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6833/2960763758","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":2960763758},{"type":"v1","addr":"172.21.15.86:6833","nonce":2960763758}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24470,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/4131134219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":4131134219},{"type":"v1","addr":"172.21.15.28:6835","nonce":4131134219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24481,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/1727869064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1727869064},{"type":"v1","addr":"172.21.15.186:6835","nonce":1727869064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24482,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2938260704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2938260704},{"type":"v1","addr":"172.21.15.28:6837","nonce":2938260704}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24494,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3473700910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3473700910},{"type":"v1","addr":"172.21.15.86:6835","nonce":3473700910}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-12T22:06:33.426986+0000","modified":"2024-05-12T22:13:22.499381+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24581,"mds_1":24550,"mds_2":24532},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24499":{"gid":24499,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6837/3600809235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":3600809235},{"type":"v1","addr":"172.21.15.186:6837","nonce":3600809235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"i","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6839/2399610807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":2399610807},{"type":"v1","addr":"172.21.15.186:6839","nonce":2399610807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24530":{"gid":24530,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.86:6837/3782448884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":3782448884},{"type":"v1","addr":"172.21.15.86:6837","nonce":3782448884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24532":{"gid":24532,"name":"g","rank":2,"incarnation":21,"state":"up:active","state_seq":6,"addr":"172.21.15.28:6839/123892158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":123892158},{"type":"v1","addr":"172.21.15.28:6839","nonce":123892158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24550":{"gid":24550,"name":"a","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.28:6841/761183870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":761183870},{"type":"v1","addr":"172.21.15.28:6841","nonce":761183870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24581":{"gid":24581,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6839/1251999130","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6838","nonce":1251999130},{"type":"v1","addr":"172.21.15.86:6839","nonce":1251999130}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24581,"qdb_cluster":[24550,24581,24532]},"id":1}]} 2024-05-12T22:18:30.993 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 26 2024-05-12T22:18:30.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.995+0000 7fb08fe00640 1 -- 172.21.15.28:0/1391184104 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fb0740a2d00 msgr2=0x7fb0740a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:18:30.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.995+0000 7fb08fe00640 1 --2- 172.21.15.28:0/1391184104 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fb0740a2d00 0x7fb0740a51c0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fb09010d370 tx=0x7fb07c005f90 comp rx=0 tx=0).stop 2024-05-12T22:18:30.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.995+0000 7fb08fe00640 1 -- 172.21.15.28:0/1391184104 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb090107cd0 msgr2=0x7fb09019e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:18:30.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.995+0000 7fb08fe00640 1 --2- 172.21.15.28:0/1391184104 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb090107cd0 0x7fb09019e420 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fb08000ea60 tx=0x7fb08000ef20 comp rx=0 tx=0).stop 2024-05-12T22:18:30.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.995+0000 7fb08fe00640 1 -- 172.21.15.28:0/1391184104 shutdown_connections 2024-05-12T22:18:30.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.996+0000 7fb08fe00640 1 --2- 172.21.15.28:0/1391184104 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fb0740a2d00 0x7fb0740a51c0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:30.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.996+0000 7fb08fe00640 1 --2- 172.21.15.28:0/1391184104 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb09010dd40 0x7fb0901a2ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:30.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.996+0000 7fb08fe00640 1 --2- 172.21.15.28:0/1391184104 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb090108670 0x7fb09019e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:30.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.996+0000 7fb08fe00640 1 --2- 172.21.15.28:0/1391184104 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb090107cd0 0x7fb09019e420 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:30.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.996+0000 7fb08fe00640 1 -- 172.21.15.28:0/1391184104 >> 172.21.15.28:0/1391184104 conn(0x7fb0900773b0 msgr2=0x7fb090078a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:18:30.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.996+0000 7fb08fe00640 1 -- 172.21.15.28:0/1391184104 shutdown_connections 2024-05-12T22:18:30.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:30.996+0000 7fb08fe00640 1 -- 172.21.15.28:0/1391184104 wait complete. 2024-05-12T22:18:31.014 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-05-12T22:18:31.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.235+0000 7f4a71c00640 1 -- 172.21.15.28:0/4221492474 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4a6c10e0c0 msgr2=0x7f4a6c1104b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:18:31.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.235+0000 7f4a71c00640 1 --2- 172.21.15.28:0/4221492474 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4a6c10e0c0 0x7f4a6c1104b0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f4a64009900 tx=0x7f4a6402a3f0 comp rx=0 tx=0).stop 2024-05-12T22:18:31.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.235+0000 7f4a71c00640 1 -- 172.21.15.28:0/4221492474 shutdown_connections 2024-05-12T22:18:31.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.235+0000 7f4a71c00640 1 --2- 172.21.15.28:0/4221492474 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4a6c10e0c0 0x7f4a6c1104b0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:31.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.235+0000 7f4a71c00640 1 --2- 172.21.15.28:0/4221492474 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4a6c100ec0 0x7f4a6c10daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:31.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.235+0000 7f4a71c00640 1 --2- 172.21.15.28:0/4221492474 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a6c100510 0x7f4a6c1008f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:31.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.235+0000 7f4a71c00640 1 -- 172.21.15.28:0/4221492474 >> 172.21.15.28:0/4221492474 conn(0x7f4a6c0fc120 msgr2=0x7f4a6c0fe540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:18:31.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.235+0000 7f4a71c00640 1 -- 172.21.15.28:0/4221492474 shutdown_connections 2024-05-12T22:18:31.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.236+0000 7f4a71c00640 1 -- 172.21.15.28:0/4221492474 wait complete. 2024-05-12T22:18:31.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.237+0000 7f4a71c00640 1 Processor -- start 2024-05-12T22:18:31.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.237+0000 7f4a71c00640 1 -- start start 2024-05-12T22:18:31.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.237+0000 7f4a71c00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4a6c100510 0x7f4a6c19a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:31.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.237+0000 7f4a71c00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4a6c100ec0 0x7f4a6c19a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:31.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.237+0000 7f4a71c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a6c10e0c0 0x7f4a6c19eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:31.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.237+0000 7f4a71c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4a6c113d80 con 0x7f4a6c10e0c0 2024-05-12T22:18:31.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.237+0000 7f4a71c00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f4a6c113c00 con 0x7f4a6c100510 2024-05-12T22:18:31.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.237+0000 7f4a71c00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f4a6c113f00 con 0x7f4a6c100ec0 2024-05-12T22:18:31.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.237+0000 7f4a71200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a6c10e0c0 0x7f4a6c19eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:31.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.238+0000 7f4a6be00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4a6c100510 0x7f4a6c19a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:31.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.238+0000 7f4a71200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a6c10e0c0 0x7f4a6c19eac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48832/0 (socket says 172.21.15.28:48832) 2024-05-12T22:18:31.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.238+0000 7f4a71200640 1 -- 172.21.15.28:0/1140189034 learned_addr learned my addr 172.21.15.28:0/1140189034 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:18:31.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.238+0000 7f4a6be00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4a6c100510 0x7f4a6c19a200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:46664/0 (socket says 172.21.15.28:46664) 2024-05-12T22:18:31.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.238+0000 7f4a6b400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4a6c100ec0 0x7f4a6c19a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:31.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.238+0000 7f4a6be00640 1 -- 172.21.15.28:0/1140189034 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4a6c100ec0 msgr2=0x7f4a6c19a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:18:31.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.238+0000 7f4a6be00640 1 --2- 172.21.15.28:0/1140189034 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4a6c100ec0 0x7f4a6c19a740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:31.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.238+0000 7f4a6be00640 1 -- 172.21.15.28:0/1140189034 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a6c10e0c0 msgr2=0x7f4a6c19eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:18:31.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.238+0000 7f4a6be00640 1 --2- 172.21.15.28:0/1140189034 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a6c10e0c0 0x7f4a6c19eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:31.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.238+0000 7f4a6be00640 1 -- 172.21.15.28:0/1140189034 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a6c19f1f0 con 0x7f4a6c100510 2024-05-12T22:18:31.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.238+0000 7f4a71200640 1 --2- 172.21.15.28:0/1140189034 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a6c10e0c0 0x7f4a6c19eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:18:31.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.238+0000 7f4a6be00640 1 --2- 172.21.15.28:0/1140189034 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4a6c100510 0x7f4a6c19a200 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f4a5c00eb00 tx=0x7f4a5c00c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:18:31.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.239+0000 7f4a68c00640 1 -- 172.21.15.28:0/1140189034 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a5c010430 con 0x7f4a6c100510 2024-05-12T22:18:31.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.239+0000 7f4a68c00640 1 -- 172.21.15.28:0/1140189034 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4a5c010ab0 con 0x7f4a6c100510 2024-05-12T22:18:31.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.239+0000 7f4a71c00640 1 -- 172.21.15.28:0/1140189034 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a6c19f4e0 con 0x7f4a6c100510 2024-05-12T22:18:31.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.239+0000 7f4a68c00640 1 -- 172.21.15.28:0/1140189034 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a5c017bb0 con 0x7f4a6c100510 2024-05-12T22:18:31.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.239+0000 7f4a71c00640 1 -- 172.21.15.28:0/1140189034 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a6c1a6d70 con 0x7f4a6c100510 2024-05-12T22:18:31.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.242+0000 7f4a68c00640 1 -- 172.21.15.28:0/1140189034 <== mon.2 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7f4a5c010c60 con 0x7f4a6c100510 2024-05-12T22:18:31.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.242+0000 7f4a75fb4740 1 Processor -- start 2024-05-12T22:18:31.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.243+0000 7f4a75fb4740 1 -- start start 2024-05-12T22:18:31.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.243+0000 7f4a75fb4740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f109339b60 0x55f109339f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:31.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.243+0000 7f4a75fb4740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f10933a480 0x55f109344610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:31.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.243+0000 7f4a75fb4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f109344b50 0x55f109346f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:31.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.243+0000 7f4a75fb4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f10929ea30 con 0x55f109344b50 2024-05-12T22:18:31.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.243+0000 7f4a75fb4740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x55f109246700 con 0x55f109339b60 2024-05-12T22:18:31.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.243+0000 7f4a75fb4740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55f10929a9f0 con 0x55f10933a480 2024-05-12T22:18:31.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.243+0000 7f4a71200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f109339b60 0x55f109339f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:31.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.243+0000 7f4a71200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f109339b60 0x55f109339f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:46674/0 (socket says 172.21.15.28:46674) 2024-05-12T22:18:31.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.243+0000 7f4a71200640 1 -- 172.21.15.28:0/1301537677 learned_addr learned my addr 172.21.15.28:0/1301537677 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:18:31.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.243+0000 7f4a6b400640 1 --2- 172.21.15.28:0/1301537677 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f109344b50 0x55f109346f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:31.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.243+0000 7f4a6be00640 1 --2- 172.21.15.28:0/1301537677 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f10933a480 0x55f109344610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:31.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.244+0000 7f4a71200640 1 -- 172.21.15.28:0/1301537677 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f10933a480 msgr2=0x55f109344610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:18:31.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.244+0000 7f4a71200640 1 --2- 172.21.15.28:0/1301537677 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f10933a480 0x55f109344610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:31.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.244+0000 7f4a71200640 1 -- 172.21.15.28:0/1301537677 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f109344b50 msgr2=0x55f109346f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:18:31.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.244+0000 7f4a71200640 1 --2- 172.21.15.28:0/1301537677 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f109344b50 0x55f109346f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:31.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.244+0000 7f4a71200640 1 -- 172.21.15.28:0/1301537677 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f108fb23f0 con 0x55f109339b60 2024-05-12T22:18:31.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.244+0000 7f4a6be00640 1 --2- 172.21.15.28:0/1301537677 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f10933a480 0x55f109344610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:18:31.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.244+0000 7f4a71200640 1 --2- 172.21.15.28:0/1301537677 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f109339b60 0x55f109339f40 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f4a6402a9e0 tx=0x7f4a64005e00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9ffa0d4cb318aee6 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:18:31.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.244+0000 7f4a68c00640 1 --2- 172.21.15.28:0/1140189034 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f4a500a2d00 0x7f4a500a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:31.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.244+0000 7f4a61600640 1 -- 172.21.15.28:0/1301537677 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a64040070 con 0x55f109339b60 2024-05-12T22:18:31.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.244+0000 7f4a61600640 1 -- 172.21.15.28:0/1301537677 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4a64003cd0 con 0x55f109339b60 2024-05-12T22:18:31.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.244+0000 7f4a61600640 1 -- 172.21.15.28:0/1301537677 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a64035040 con 0x55f109339b60 2024-05-12T22:18:31.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.245+0000 7f4a68c00640 1 -- 172.21.15.28:0/1140189034 <== mon.2 v2:172.21.15.86:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8513+0+0 (secure 0 0 0) 0x7f4a5c0cfe60 con 0x7f4a6c100510 2024-05-12T22:18:31.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.245+0000 7f4a6be00640 1 --2- 172.21.15.28:0/1140189034 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f4a500a2d00 0x7f4a500a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:31.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.245+0000 7f4a75fb4740 1 -- 172.21.15.28:0/1301537677 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f109339b60 msgr2=0x55f109339f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:18:31.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.245+0000 7f4a75fb4740 1 --2- 172.21.15.28:0/1301537677 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f109339b60 0x55f109339f40 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f4a6402a9e0 tx=0x7f4a64005e00 comp rx=0 tx=0).stop 2024-05-12T22:18:31.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.245+0000 7f4a75fb4740 1 -- 172.21.15.28:0/1301537677 shutdown_connections 2024-05-12T22:18:31.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.245+0000 7f4a75fb4740 1 --2- 172.21.15.28:0/1301537677 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f109344b50 0x55f109346f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:31.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.245+0000 7f4a75fb4740 1 --2- 172.21.15.28:0/1301537677 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f10933a480 0x55f109344610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:31.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.245+0000 7f4a75fb4740 1 --2- 172.21.15.28:0/1301537677 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f109339b60 0x55f109339f40 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:31.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.245+0000 7f4a75fb4740 1 -- 172.21.15.28:0/1301537677 >> 172.21.15.28:0/1301537677 conn(0x55f109335e80 msgr2=0x55f109336260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:18:31.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.245+0000 7f4a75fb4740 1 -- 172.21.15.28:0/1301537677 shutdown_connections 2024-05-12T22:18:31.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.245+0000 7f4a75fb4740 1 -- 172.21.15.28:0/1301537677 wait complete. 2024-05-12T22:18:31.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.246+0000 7f4a6be00640 1 --2- 172.21.15.28:0/1140189034 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f4a500a2d00 0x7f4a500a51c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x55f10933b120 tx=0x7f4a5c00cf10 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:18:31.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.246+0000 7f4a75fb4740 1 -- start start 2024-05-12T22:18:31.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.246+0000 7f4a75fb4740 1 Processor -- start 2024-05-12T22:18:31.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.247+0000 7f4a75fb4740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f109339b60 0x55f109341b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:31.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.247+0000 7f4a75fb4740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f10933a480 0x55f109342080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:31.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.247+0000 7f4a75fb4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f109344b50 0x55f1093425c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:31.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.247+0000 7f4a75fb4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f1093478e0 con 0x55f109344b50 2024-05-12T22:18:31.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.247+0000 7f4a75fb4740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x55f109259a70 con 0x55f10933a480 2024-05-12T22:18:31.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.247+0000 7f4a75fb4740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55f109347a60 con 0x55f109339b60 2024-05-12T22:18:31.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.247+0000 7f4a75fb4740 10 client.0 _ll_register_callbacks cb 0x55f10923cba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:18:31.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.247+0000 7f4a6b400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f10933a480 0x55f109342080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:31.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.247+0000 7f4a6b400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f10933a480 0x55f109342080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:46686/0 (socket says 172.21.15.28:46686) 2024-05-12T22:18:31.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.247+0000 7f4a6b400640 1 -- 172.21.15.28:0/1050812785 learned_addr learned my addr 172.21.15.28:0/1050812785 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:18:31.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.247+0000 7f4a71200640 1 --2- 172.21.15.28:0/1050812785 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f109344b50 0x55f1093425c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:31.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.247+0000 7f4a71200640 1 --2- 172.21.15.28:0/1050812785 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f109339b60 0x55f109341b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:31.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.247+0000 7f4a6b400640 1 -- 172.21.15.28:0/1050812785 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f109339b60 msgr2=0x55f109341b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:18:31.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.247+0000 7f4a6b400640 1 --2- 172.21.15.28:0/1050812785 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f109339b60 0x55f109341b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:31.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.247+0000 7f4a6b400640 1 -- 172.21.15.28:0/1050812785 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f109344b50 msgr2=0x55f1093425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:18:31.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.247+0000 7f4a6b400640 1 --2- 172.21.15.28:0/1050812785 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f109344b50 0x55f1093425c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:31.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.247+0000 7f4a6b400640 1 -- 172.21.15.28:0/1050812785 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f10907c7d0 con 0x55f10933a480 2024-05-12T22:18:31.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.247+0000 7f4a71200640 1 --2- 172.21.15.28:0/1050812785 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f109344b50 0x55f1093425c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:18:31.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.248+0000 7f4a6b400640 1 --2- 172.21.15.28:0/1050812785 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f10933a480 0x55f109342080 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f4a5400b9f0 tx=0x7f4a5400beb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5acf6649e14cd323 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:18:31.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.248+0000 7f4a5a000640 10 client.0 ms_handle_connect on v2:172.21.15.86:3300/0 2024-05-12T22:18:31.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.248+0000 7f4a5a000640 1 -- 172.21.15.28:0/1050812785 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a54003e20 con 0x55f10933a480 2024-05-12T22:18:31.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.248+0000 7f4a5a000640 1 -- 172.21.15.28:0/1050812785 <== mon.2 v2:172.21.15.86:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8513+0+0 (secure 0 0 0) 0x7f4a54005020 con 0x55f10933a480 2024-05-12T22:18:31.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.248+0000 7f4a75fb4740 1 -- 172.21.15.28:0/1050812785 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4a64003cd0 con 0x55f10933a480 2024-05-12T22:18:31.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.248+0000 7f4a5a000640 1 -- 172.21.15.28:0/1050812785 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f4a54005220 con 0x55f10933a480 2024-05-12T22:18:31.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.248+0000 7f4a75fb4740 10 client.27287 fetch_fsmap learned FSMap version 26 2024-05-12T22:18:31.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.248+0000 7f4a75fb4740 1 -- 172.21.15.28:0/1050812785 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55f109353c70 con 0x55f10933a480 2024-05-12T22:18:31.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.249+0000 7f4a5a000640 1 -- 172.21.15.28:0/1050812785 <== mon.2 v2:172.21.15.86:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4a64003cd0 con 0x55f10933a480 2024-05-12T22:18:31.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.249+0000 7f4a5a000640 1 -- 172.21.15.28:0/1050812785 <== mon.2 v2:172.21.15.86:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f4a5401b5f0 con 0x55f10933a480 2024-05-12T22:18:31.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.249+0000 7f4a75fb4740 10 client.27287 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:18:31.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.249+0000 7f4a75fb4740 10 client.27287 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.g 2024-05-12T22:18:31.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.249+0000 7f4a75fb4740 20 client.27287 populate_metadata read hostname 'smithi028' 2024-05-12T22:18:31.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.249+0000 7f4a75fb4740 1 --2- 172.21.15.28:0/1050812785 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x55f1093551d0 0x55f1093575f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:31.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.249+0000 7f4a75fb4740 4 client.27287 mds_command: new command op to 24532 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-12T22:18:31.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.249+0000 7f4a75fb4740 1 -- 172.21.15.28:0/1050812785 --> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f1092ac160 con 0x55f1093551d0 2024-05-12T22:18:31.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.250+0000 7f4a71200640 1 --2- 172.21.15.28:0/1050812785 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x55f1093551d0 0x55f1093575f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:31.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.250+0000 7f4a71200640 1 --2- 172.21.15.28:0/1050812785 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x55f1093551d0 0x55f1093575f0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c022c88156ab4ddc server_cookie=57214fa93b662378 in_seq=0 out_seq=0 2024-05-12T22:18:31.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.250+0000 7f4a5a000640 10 client.27287 ms_handle_connect on v2:172.21.15.28:6838/123892158 2024-05-12T22:18:31.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.251+0000 7f4a5a000640 1 -- 172.21.15.28:0/1050812785 <== mds.2 v2:172.21.15.28:6838/123892158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f1092ac160 con 0x55f1093551d0 2024-05-12T22:18:31.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.251+0000 7f4a5a000640 10 client.27287 handle_command_reply: tid=0 2024-05-12T22:18:31.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.251+0000 7f4a75fb4740 1 client.27287 shutdown 2024-05-12T22:18:31.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.251+0000 7f4a75fb4740 1 -- 172.21.15.28:0/1050812785 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f10933a480 msgr2=0x55f109342080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:18:31.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.251+0000 7f4a75fb4740 1 --2- 172.21.15.28:0/1050812785 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f10933a480 0x55f109342080 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f4a5400b9f0 tx=0x7f4a5400beb0 comp rx=0 tx=0).stop 2024-05-12T22:18:31.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.251+0000 7f4a75fb4740 1 -- 172.21.15.28:0/1050812785 shutdown_connections 2024-05-12T22:18:31.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.251+0000 7f4a75fb4740 1 --2- 172.21.15.28:0/1050812785 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f109344b50 0x55f1093425c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:31.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.251+0000 7f4a75fb4740 1 --2- 172.21.15.28:0/1050812785 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x55f1093551d0 0x55f1093575f0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:31.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.251+0000 7f4a75fb4740 1 --2- 172.21.15.28:0/1050812785 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f10933a480 0x55f109342080 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:31.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.251+0000 7f4a75fb4740 1 --2- 172.21.15.28:0/1050812785 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f109339b60 0x55f109341b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:31.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.251+0000 7f4a5a000640 0 client.27287 ms_handle_reset on v2:172.21.15.28:6838/123892158 2024-05-12T22:18:31.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.251+0000 7f4a75fb4740 1 -- 172.21.15.28:0/1050812785 >> 172.21.15.28:0/1050812785 conn(0x55f109335e80 msgr2=0x55f10933bc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:18:31.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.251+0000 7f4a75fb4740 1 -- 172.21.15.28:0/1050812785 shutdown_connections 2024-05-12T22:18:31.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.251+0000 7f4a75fb4740 1 -- 172.21.15.28:0/1050812785 wait complete. 2024-05-12T22:18:31.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.252+0000 7f4a75fb4740 20 client.27287 trim_cache size 0 max 16384 2024-05-12T22:18:31.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.266+0000 7f4a75fb4740 1 Processor -- start 2024-05-12T22:18:31.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.267+0000 7f4a75fb4740 1 -- start start 2024-05-12T22:18:31.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.267+0000 7f4a75fb4740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f109339b60 0x55f1093507d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:31.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.267+0000 7f4a75fb4740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f10933a480 0x55f109342210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:31.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.267+0000 7f4a75fb4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f109344b50 0x55f10935cf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:31.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.267+0000 7f4a75fb4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4a6c1b1e80 con 0x55f109344b50 2024-05-12T22:18:31.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.267+0000 7f4a75fb4740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f4a6c1af750 con 0x55f10933a480 2024-05-12T22:18:31.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.267+0000 7f4a75fb4740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f4a6c1ad5c0 con 0x55f109339b60 2024-05-12T22:18:31.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.267+0000 7f4a71200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f109344b50 0x55f10935cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:31.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.267+0000 7f4a71200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f109344b50 0x55f10935cf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48854/0 (socket says 172.21.15.28:48854) 2024-05-12T22:18:31.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.267+0000 7f4a6b400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f10933a480 0x55f109342210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:31.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.267+0000 7f4a71200640 1 -- 172.21.15.28:0/2573884931 learned_addr learned my addr 172.21.15.28:0/2573884931 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:18:31.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.267+0000 7f4a71200640 1 --2- 172.21.15.28:0/2573884931 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f109339b60 0x55f1093507d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:31.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.268+0000 7f4a6b400640 1 -- 172.21.15.28:0/2573884931 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f109339b60 msgr2=0x55f1093507d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:18:31.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.268+0000 7f4a6b400640 1 --2- 172.21.15.28:0/2573884931 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f109339b60 0x55f1093507d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:31.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.268+0000 7f4a6b400640 1 -- 172.21.15.28:0/2573884931 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f109344b50 msgr2=0x55f10935cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:18:31.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.268+0000 7f4a6b400640 1 --2- 172.21.15.28:0/2573884931 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f109344b50 0x55f10935cf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:31.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.268+0000 7f4a6b400640 1 -- 172.21.15.28:0/2573884931 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f10933c920 con 0x55f10933a480 2024-05-12T22:18:31.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.268+0000 7f4a6b400640 1 --2- 172.21.15.28:0/2573884931 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f10933a480 0x55f109342210 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f4a5401ba90 tx=0x7f4a5400b5d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=58cedeae07c0c9fd server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:18:31.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.268+0000 7f4a60c00640 1 -- 172.21.15.28:0/2573884931 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a54000e60 con 0x55f10933a480 2024-05-12T22:18:31.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.268+0000 7f4a60c00640 1 -- 172.21.15.28:0/2573884931 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4a5401cd10 con 0x55f10933a480 2024-05-12T22:18:31.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.268+0000 7f4a60c00640 1 -- 172.21.15.28:0/2573884931 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a54028040 con 0x55f10933a480 2024-05-12T22:18:31.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.269+0000 7f4a75fb4740 1 -- 172.21.15.28:0/2573884931 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f10933a480 msgr2=0x55f109342210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:18:31.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.269+0000 7f4a75fb4740 1 --2- 172.21.15.28:0/2573884931 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f10933a480 0x55f109342210 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f4a5401ba90 tx=0x7f4a5400b5d0 comp rx=0 tx=0).stop 2024-05-12T22:18:31.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.269+0000 7f4a75fb4740 1 -- 172.21.15.28:0/2573884931 shutdown_connections 2024-05-12T22:18:31.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.269+0000 7f4a75fb4740 1 --2- 172.21.15.28:0/2573884931 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f109344b50 0x55f10935cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:31.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.269+0000 7f4a75fb4740 1 --2- 172.21.15.28:0/2573884931 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f10933a480 0x55f109342210 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:31.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.269+0000 7f4a75fb4740 1 --2- 172.21.15.28:0/2573884931 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f109339b60 0x55f1093507d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:31.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.269+0000 7f4a75fb4740 1 -- 172.21.15.28:0/2573884931 >> 172.21.15.28:0/2573884931 conn(0x55f109335e80 msgr2=0x55f109341b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:18:31.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.269+0000 7f4a75fb4740 1 -- 172.21.15.28:0/2573884931 shutdown_connections 2024-05-12T22:18:31.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.269+0000 7f4a75fb4740 1 -- 172.21.15.28:0/2573884931 wait complete. 2024-05-12T22:18:31.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.270+0000 7f4a75fb4740 1 -- start start 2024-05-12T22:18:31.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.270+0000 7f4a75fb4740 1 Processor -- start 2024-05-12T22:18:31.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.270+0000 7f4a75fb4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f109339b60 0x55f10934d200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:31.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.270+0000 7f4a75fb4740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f10933a480 0x55f109348570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:31.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.270+0000 7f4a75fb4740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f109344b50 0x55f109348cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:31.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.270+0000 7f4a75fb4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4a6c1adc10 con 0x55f109339b60 2024-05-12T22:18:31.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.270+0000 7f4a75fb4740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f4a6c1af080 con 0x55f10933a480 2024-05-12T22:18:31.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.270+0000 7f4a75fb4740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f4a6c1ae2e0 con 0x55f109344b50 2024-05-12T22:18:31.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.270+0000 7f4a75fb4740 10 client.0 _ll_register_callbacks cb 0x7f4a540044a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:18:31.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.271+0000 7f4a6b400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f10933a480 0x55f109348570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:31.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.271+0000 7f4a71200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f109339b60 0x55f10934d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:31.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.271+0000 7f4a6b400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f10933a480 0x55f109348570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:46718/0 (socket says 172.21.15.28:46718) 2024-05-12T22:18:31.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.271+0000 7f4a71200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f109339b60 0x55f10934d200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48866/0 (socket says 172.21.15.28:48866) 2024-05-12T22:18:31.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.271+0000 7f4a6b400640 1 -- 172.21.15.28:0/626976349 learned_addr learned my addr 172.21.15.28:0/626976349 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:18:31.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.271+0000 7f4a71200640 1 --2- 172.21.15.28:0/626976349 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f109344b50 0x55f109348cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:31.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.271+0000 7f4a6b400640 1 -- 172.21.15.28:0/626976349 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f109344b50 msgr2=0x55f109348cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:18:31.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.271+0000 7f4a6b400640 1 --2- 172.21.15.28:0/626976349 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f109344b50 0x55f109348cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:31.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.271+0000 7f4a6b400640 1 -- 172.21.15.28:0/626976349 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f109339b60 msgr2=0x55f10934d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:18:31.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.271+0000 7f4a6b400640 1 --2- 172.21.15.28:0/626976349 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f109339b60 0x55f10934d200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:31.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.271+0000 7f4a6b400640 1 -- 172.21.15.28:0/626976349 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f1093ba770 con 0x55f10933a480 2024-05-12T22:18:31.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.271+0000 7f4a71200640 1 --2- 172.21.15.28:0/626976349 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f109344b50 0x55f109348cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-12T22:18:31.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.271+0000 7f4a6b400640 1 --2- 172.21.15.28:0/626976349 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f10933a480 0x55f109348570 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x55f1093368f0 tx=0x7f4a5401cf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=16d1dbb37c16b572 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:18:31.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.271+0000 7f4a5a000640 10 client.0 ms_handle_connect on v2:172.21.15.86:3300/0 2024-05-12T22:18:31.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.272+0000 7f4a5a000640 1 -- 172.21.15.28:0/626976349 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a54028040 con 0x55f10933a480 2024-05-12T22:18:31.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.272+0000 7f4a75fb4740 1 -- 172.21.15.28:0/626976349 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4a5401cd10 con 0x55f10933a480 2024-05-12T22:18:31.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.272+0000 7f4a5a000640 1 -- 172.21.15.28:0/626976349 <== mon.2 v2:172.21.15.86:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8513+0+0 (secure 0 0 0) 0x7f4a5401b680 con 0x55f10933a480 2024-05-12T22:18:31.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.272+0000 7f4a5a000640 1 -- 172.21.15.28:0/626976349 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f4a54000e40 con 0x55f10933a480 2024-05-12T22:18:31.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.272+0000 7f4a75fb4740 10 client.27293 fetch_fsmap learned FSMap version 26 2024-05-12T22:18:31.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.272+0000 7f4a75fb4740 1 -- 172.21.15.28:0/626976349 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55f1093bb8d0 con 0x55f10933a480 2024-05-12T22:18:31.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.272+0000 7f4a5a000640 1 -- 172.21.15.28:0/626976349 <== mon.2 v2:172.21.15.86:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4a5401cd10 con 0x55f10933a480 2024-05-12T22:18:31.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.273+0000 7f4a5a000640 1 -- 172.21.15.28:0/626976349 <== mon.2 v2:172.21.15.86:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f4a5402b030 con 0x55f10933a480 2024-05-12T22:18:31.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.273+0000 7f4a75fb4740 10 client.27293 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:18:31.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.273+0000 7f4a75fb4740 10 client.27293 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.g 2024-05-12T22:18:31.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.273+0000 7f4a75fb4740 20 client.27293 populate_metadata read hostname 'smithi028' 2024-05-12T22:18:31.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.273+0000 7f4a75fb4740 1 --2- 172.21.15.28:0/626976349 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x55f1093bc020 0x55f1093be440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:31.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.273+0000 7f4a75fb4740 4 client.27293 mds_command: new command op to 24532 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-12T22:18:31.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.273+0000 7f4a75fb4740 1 -- 172.21.15.28:0/626976349 --> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f1092a64b0 con 0x55f1093bc020 2024-05-12T22:18:31.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.273+0000 7f4a71200640 1 --2- 172.21.15.28:0/626976349 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x55f1093bc020 0x55f1093be440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:31.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.274+0000 7f4a71200640 1 --2- 172.21.15.28:0/626976349 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x55f1093bc020 0x55f1093be440 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c340681fe5e9905f server_cookie=c2324562dc0ccda8 in_seq=0 out_seq=0 2024-05-12T22:18:31.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.274+0000 7f4a5a000640 10 client.27293 ms_handle_connect on v2:172.21.15.28:6838/123892158 2024-05-12T22:18:31.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.274+0000 7f4a5a000640 1 -- 172.21.15.28:0/626976349 <== mds.2 v2:172.21.15.28:6838/123892158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f1092a64b0 con 0x55f1093bc020 2024-05-12T22:18:31.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.274+0000 7f4a5a000640 10 client.27293 handle_command_reply: tid=0 2024-05-12T22:18:31.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.274+0000 7f4a75fb4740 1 client.27293 shutdown 2024-05-12T22:18:31.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.274+0000 7f4a75fb4740 1 -- 172.21.15.28:0/626976349 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f10933a480 msgr2=0x55f109348570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:18:31.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.274+0000 7f4a75fb4740 1 --2- 172.21.15.28:0/626976349 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f10933a480 0x55f109348570 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x55f1093368f0 tx=0x7f4a5401cf20 comp rx=0 tx=0).stop 2024-05-12T22:18:31.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.274+0000 7f4a75fb4740 1 -- 172.21.15.28:0/626976349 shutdown_connections 2024-05-12T22:18:31.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.274+0000 7f4a75fb4740 1 --2- 172.21.15.28:0/626976349 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f109344b50 0x55f109348cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:31.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.274+0000 7f4a75fb4740 1 --2- 172.21.15.28:0/626976349 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x55f1093bc020 0x55f1093be440 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:31.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.274+0000 7f4a75fb4740 1 --2- 172.21.15.28:0/626976349 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f10933a480 0x55f109348570 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:31.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.274+0000 7f4a75fb4740 1 --2- 172.21.15.28:0/626976349 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f109339b60 0x55f10934d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:31.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.274+0000 7f4a75fb4740 1 -- 172.21.15.28:0/626976349 >> 172.21.15.28:0/626976349 conn(0x55f109335e80 msgr2=0x55f109335000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:18:31.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.274+0000 7f4a5a000640 0 client.27293 ms_handle_reset on v2:172.21.15.28:6838/123892158 2024-05-12T22:18:31.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.274+0000 7f4a75fb4740 1 -- 172.21.15.28:0/626976349 shutdown_connections 2024-05-12T22:18:31.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.274+0000 7f4a75fb4740 1 -- 172.21.15.28:0/626976349 wait complete. 2024-05-12T22:18:31.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.275+0000 7f4a75fb4740 20 client.27293 trim_cache size 0 max 16384 2024-05-12T22:18:31.283 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-05-12T22:18:31.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.275+0000 7f4a71c00640 1 -- 172.21.15.28:0/1140189034 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f4a500a2d00 msgr2=0x7f4a500a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:18:31.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.275+0000 7f4a71c00640 1 --2- 172.21.15.28:0/1140189034 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f4a500a2d00 0x7f4a500a51c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x55f10933b120 tx=0x7f4a5c00cf10 comp rx=0 tx=0).stop 2024-05-12T22:18:31.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.275+0000 7f4a71c00640 1 -- 172.21.15.28:0/1140189034 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4a6c100510 msgr2=0x7f4a6c19a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:18:31.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.275+0000 7f4a71c00640 1 --2- 172.21.15.28:0/1140189034 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4a6c100510 0x7f4a6c19a200 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f4a5c00eb00 tx=0x7f4a5c00c2d0 comp rx=0 tx=0).stop 2024-05-12T22:18:31.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.275+0000 7f4a71c00640 1 -- 172.21.15.28:0/1140189034 shutdown_connections 2024-05-12T22:18:31.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.275+0000 7f4a71c00640 1 --2- 172.21.15.28:0/1140189034 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f4a500a2d00 0x7f4a500a51c0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:31.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.275+0000 7f4a71c00640 1 --2- 172.21.15.28:0/1140189034 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a6c10e0c0 0x7f4a6c19eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:31.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.275+0000 7f4a71c00640 1 --2- 172.21.15.28:0/1140189034 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4a6c100ec0 0x7f4a6c19a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:31.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.275+0000 7f4a71c00640 1 --2- 172.21.15.28:0/1140189034 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4a6c100510 0x7f4a6c19a200 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:31.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.275+0000 7f4a71c00640 1 -- 172.21.15.28:0/1140189034 >> 172.21.15.28:0/1140189034 conn(0x7f4a6c0fc120 msgr2=0x7f4a6c10f350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:18:31.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.275+0000 7f4a71c00640 1 -- 172.21.15.28:0/1140189034 shutdown_connections 2024-05-12T22:18:31.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.275+0000 7f4a71c00640 1 -- 172.21.15.28:0/1140189034 wait complete. 2024-05-12T22:18:31.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:18:31 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/1391184104' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:18:31.293 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-05-12T22:18:31.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:18:31 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/1391184104' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:18:31.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.503+0000 7fe8c1c00640 1 -- 172.21.15.28:0/917947012 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8bc108770 msgr2=0x7fe8bc114b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:18:31.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.503+0000 7fe8c1c00640 1 --2- 172.21.15.28:0/917947012 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8bc108770 0x7fe8bc114b90 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fe8b4009ca0 tx=0x7fe8b402a950 comp rx=0 tx=0).stop 2024-05-12T22:18:31.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.503+0000 7fe8c1c00640 1 -- 172.21.15.28:0/917947012 shutdown_connections 2024-05-12T22:18:31.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.503+0000 7fe8c1c00640 1 --2- 172.21.15.28:0/917947012 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8bc108770 0x7fe8bc114b90 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:31.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.503+0000 7fe8c1c00640 1 --2- 172.21.15.28:0/917947012 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe8bc107dd0 0x7fe8bc108230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:31.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.503+0000 7fe8c1c00640 1 --2- 172.21.15.28:0/917947012 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe8bc10ddf0 0x7fe8bc10e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:31.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.503+0000 7fe8c1c00640 1 -- 172.21.15.28:0/917947012 >> 172.21.15.28:0/917947012 conn(0x7fe8bc077650 msgr2=0x7fe8bc106650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:18:31.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.503+0000 7fe8c1c00640 1 -- 172.21.15.28:0/917947012 shutdown_connections 2024-05-12T22:18:31.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.503+0000 7fe8c1c00640 1 -- 172.21.15.28:0/917947012 wait complete. 2024-05-12T22:18:31.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.504+0000 7fe8c1c00640 1 Processor -- start 2024-05-12T22:18:31.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.504+0000 7fe8c1c00640 1 -- start start 2024-05-12T22:18:31.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.505+0000 7fe8c1c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8bc107dd0 0x7fe8bc19e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:31.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.505+0000 7fe8c1c00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe8bc108770 0x7fe8bc19ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:31.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.505+0000 7fe8c1c00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe8bc10ddf0 0x7fe8bc1a2dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:31.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.505+0000 7fe8c1c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe8bc118120 con 0x7fe8bc107dd0 2024-05-12T22:18:31.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.505+0000 7fe8c1c00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fe8bc117fa0 con 0x7fe8bc10ddf0 2024-05-12T22:18:31.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.505+0000 7fe8c1c00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe8bc1182a0 con 0x7fe8bc108770 2024-05-12T22:18:31.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.505+0000 7fe8bbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8bc107dd0 0x7fe8bc19e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:31.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.505+0000 7fe8bbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8bc107dd0 0x7fe8bc19e500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48884/0 (socket says 172.21.15.28:48884) 2024-05-12T22:18:31.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.505+0000 7fe8bb400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe8bc108770 0x7fe8bc19ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:31.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.505+0000 7fe8bbe00640 1 -- 172.21.15.28:0/485652714 learned_addr learned my addr 172.21.15.28:0/485652714 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:18:31.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.505+0000 7fe8c1200640 1 --2- 172.21.15.28:0/485652714 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe8bc10ddf0 0x7fe8bc1a2dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:31.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.506+0000 7fe8bbe00640 1 -- 172.21.15.28:0/485652714 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe8bc108770 msgr2=0x7fe8bc19ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:18:31.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.506+0000 7fe8bbe00640 1 --2- 172.21.15.28:0/485652714 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe8bc108770 0x7fe8bc19ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:31.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.506+0000 7fe8bbe00640 1 -- 172.21.15.28:0/485652714 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe8bc10ddf0 msgr2=0x7fe8bc1a2dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:18:31.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.506+0000 7fe8bbe00640 1 --2- 172.21.15.28:0/485652714 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe8bc10ddf0 0x7fe8bc1a2dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:31.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.506+0000 7fe8bbe00640 1 -- 172.21.15.28:0/485652714 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8bc1a34f0 con 0x7fe8bc107dd0 2024-05-12T22:18:31.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.506+0000 7fe8bbe00640 1 --2- 172.21.15.28:0/485652714 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8bc107dd0 0x7fe8bc19e500 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fe8ac00e9f0 tx=0x7fe8ac00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:18:31.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.506+0000 7fe8b8c00640 1 -- 172.21.15.28:0/485652714 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8ac00cd80 con 0x7fe8bc107dd0 2024-05-12T22:18:31.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.506+0000 7fe8b8c00640 1 -- 172.21.15.28:0/485652714 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe8ac010910 con 0x7fe8bc107dd0 2024-05-12T22:18:31.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.506+0000 7fe8c1c00640 1 -- 172.21.15.28:0/485652714 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8bc1a37e0 con 0x7fe8bc107dd0 2024-05-12T22:18:31.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.506+0000 7fe8b8c00640 1 -- 172.21.15.28:0/485652714 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8ac018990 con 0x7fe8bc107dd0 2024-05-12T22:18:31.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.506+0000 7fe8c1c00640 1 -- 172.21.15.28:0/485652714 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8bc1ab070 con 0x7fe8bc107dd0 2024-05-12T22:18:31.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.510+0000 7fe8b8c00640 1 -- 172.21.15.28:0/485652714 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7fe8ac010ac0 con 0x7fe8bc107dd0 2024-05-12T22:18:31.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.510+0000 7fe8c5fd8740 1 Processor -- start 2024-05-12T22:18:31.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.510+0000 7fe8c5fd8740 1 -- start start 2024-05-12T22:18:31.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.511+0000 7fe8c5fd8740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562776fc6b60 0x562776fc6f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:31.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.511+0000 7fe8c5fd8740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562776fc7480 0x562776fd1610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:31.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.511+0000 7fe8c5fd8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562776fd1b50 0x562776fd3f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:31.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.511+0000 7fe8c5fd8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562776f2ba30 con 0x562776fd1b50 2024-05-12T22:18:31.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.511+0000 7fe8c5fd8740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x562776ed3700 con 0x562776fc6b60 2024-05-12T22:18:31.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.511+0000 7fe8c5fd8740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x562776f279f0 con 0x562776fc7480 2024-05-12T22:18:31.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.511+0000 7fe8bb400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562776fd1b50 0x562776fd3f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:31.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.511+0000 7fe8c1200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562776fc6b60 0x562776fc6f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:31.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.511+0000 7fe8bb400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562776fd1b50 0x562776fd3f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48898/0 (socket says 172.21.15.28:48898) 2024-05-12T22:18:31.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.511+0000 7fe8c1200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562776fc6b60 0x562776fc6f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:46744/0 (socket says 172.21.15.28:46744) 2024-05-12T22:18:31.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.511+0000 7fe8bb400640 1 -- 172.21.15.28:0/1791444952 learned_addr learned my addr 172.21.15.28:0/1791444952 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:18:31.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.511+0000 7fe8bbe00640 1 --2- 172.21.15.28:0/1791444952 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562776fc7480 0x562776fd1610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:31.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.512+0000 7fe8c1200640 1 -- 172.21.15.28:0/1791444952 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562776fc7480 msgr2=0x562776fd1610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:18:31.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.512+0000 7fe8c1200640 1 --2- 172.21.15.28:0/1791444952 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562776fc7480 0x562776fd1610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:31.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.512+0000 7fe8c1200640 1 -- 172.21.15.28:0/1791444952 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562776fd1b50 msgr2=0x562776fd3f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:18:31.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.512+0000 7fe8c1200640 1 --2- 172.21.15.28:0/1791444952 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562776fd1b50 0x562776fd3f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:31.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.512+0000 7fe8c1200640 1 -- 172.21.15.28:0/1791444952 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562776c3f3f0 con 0x562776fc6b60 2024-05-12T22:18:31.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.512+0000 7fe8c1200640 1 --2- 172.21.15.28:0/1791444952 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562776fc6b60 0x562776fc6f40 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fe8b4005e30 tx=0x7fe8b4005f80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2d517913ca82abeb server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:18:31.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.512+0000 7fe8b8c00640 1 --2- 172.21.15.28:0/485652714 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fe8a00a2d80 0x7fe8a00a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:31.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.512+0000 7fe8b8c00640 1 -- 172.21.15.28:0/485652714 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8513+0+0 (secure 0 0 0) 0x7fe8ac0cf6c0 con 0x7fe8bc107dd0 2024-05-12T22:18:31.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.512+0000 7fe8b1600640 1 -- 172.21.15.28:0/1791444952 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8b4048070 con 0x562776fc6b60 2024-05-12T22:18:31.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.512+0000 7fe8bbe00640 1 --2- 172.21.15.28:0/485652714 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fe8a00a2d80 0x7fe8a00a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:31.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.512+0000 7fe8b1600640 1 -- 172.21.15.28:0/1791444952 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe8b4003e50 con 0x562776fc6b60 2024-05-12T22:18:31.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.513+0000 7fe8b1600640 1 -- 172.21.15.28:0/1791444952 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8b4034040 con 0x562776fc6b60 2024-05-12T22:18:31.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.513+0000 7fe8c5fd8740 1 -- 172.21.15.28:0/1791444952 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562776fc6b60 msgr2=0x562776fc6f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:18:31.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.513+0000 7fe8bbe00640 1 --2- 172.21.15.28:0/485652714 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fe8a00a2d80 0x7fe8a00a5240 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fe8ac0ccec0 tx=0x7fe8ac0d5550 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:18:31.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.513+0000 7fe8c5fd8740 1 --2- 172.21.15.28:0/1791444952 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562776fc6b60 0x562776fc6f40 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fe8b4005e30 tx=0x7fe8b4005f80 comp rx=0 tx=0).stop 2024-05-12T22:18:31.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.513+0000 7fe8c5fd8740 1 -- 172.21.15.28:0/1791444952 shutdown_connections 2024-05-12T22:18:31.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.513+0000 7fe8c5fd8740 1 --2- 172.21.15.28:0/1791444952 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562776fd1b50 0x562776fd3f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:31.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.513+0000 7fe8c5fd8740 1 --2- 172.21.15.28:0/1791444952 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562776fc7480 0x562776fd1610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:31.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.513+0000 7fe8c5fd8740 1 --2- 172.21.15.28:0/1791444952 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562776fc6b60 0x562776fc6f40 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:31.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.513+0000 7fe8c5fd8740 1 -- 172.21.15.28:0/1791444952 >> 172.21.15.28:0/1791444952 conn(0x562776fc2e80 msgr2=0x562776fc3260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:18:31.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.513+0000 7fe8c5fd8740 1 -- 172.21.15.28:0/1791444952 shutdown_connections 2024-05-12T22:18:31.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.513+0000 7fe8c5fd8740 1 -- 172.21.15.28:0/1791444952 wait complete. 2024-05-12T22:18:31.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.514+0000 7fe8c5fd8740 1 -- start start 2024-05-12T22:18:31.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.514+0000 7fe8c5fd8740 1 Processor -- start 2024-05-12T22:18:31.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.514+0000 7fe8c5fd8740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562776fc6b60 0x562776fceb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:31.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.515+0000 7fe8c5fd8740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562776fc7480 0x562776fcf080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:31.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.515+0000 7fe8c5fd8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562776fd1b50 0x562776fcf5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:31.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.515+0000 7fe8c5fd8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562776fd48e0 con 0x562776fd1b50 2024-05-12T22:18:31.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.515+0000 7fe8c5fd8740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x562776ee6a70 con 0x562776fc7480 2024-05-12T22:18:31.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.515+0000 7fe8c5fd8740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x562776fd4a60 con 0x562776fc6b60 2024-05-12T22:18:31.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.515+0000 7fe8c5fd8740 10 client.0 _ll_register_callbacks cb 0x562776ec9ba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:18:31.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.515+0000 7fe8c1200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562776fd1b50 0x562776fcf5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:31.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.515+0000 7fe8c1200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562776fd1b50 0x562776fcf5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48900/0 (socket says 172.21.15.28:48900) 2024-05-12T22:18:31.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.515+0000 7fe8c1200640 1 -- 172.21.15.28:0/1324984323 learned_addr learned my addr 172.21.15.28:0/1324984323 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:18:31.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.515+0000 7fe8bb400640 1 --2- 172.21.15.28:0/1324984323 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562776fc7480 0x562776fcf080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:31.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.515+0000 7fe8c1200640 1 --2- 172.21.15.28:0/1324984323 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562776fc6b60 0x562776fceb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:31.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.515+0000 7fe8bb400640 1 -- 172.21.15.28:0/1324984323 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562776fc6b60 msgr2=0x562776fceb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:18:31.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.515+0000 7fe8bb400640 1 --2- 172.21.15.28:0/1324984323 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562776fc6b60 0x562776fceb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:31.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.515+0000 7fe8bb400640 1 -- 172.21.15.28:0/1324984323 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562776fd1b50 msgr2=0x562776fcf5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:18:31.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.515+0000 7fe8bb400640 1 --2- 172.21.15.28:0/1324984323 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562776fd1b50 0x562776fcf5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:31.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.515+0000 7fe8bb400640 1 -- 172.21.15.28:0/1324984323 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562776d097d0 con 0x562776fc7480 2024-05-12T22:18:31.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.515+0000 7fe8c1200640 1 --2- 172.21.15.28:0/1324984323 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562776fd1b50 0x562776fcf5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:18:31.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.516+0000 7fe8bb400640 1 --2- 172.21.15.28:0/1324984323 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562776fc7480 0x562776fcf080 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fe8a80126d0 tx=0x7fe8a8012b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dce3c6d3e8996859 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:18:31.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.516+0000 7fe8a6000640 10 client.0 ms_handle_connect on v2:172.21.15.86:3300/0 2024-05-12T22:18:31.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.516+0000 7fe8a6000640 1 -- 172.21.15.28:0/1324984323 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8a8004050 con 0x562776fc7480 2024-05-12T22:18:31.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.516+0000 7fe8c5fd8740 1 -- 172.21.15.28:0/1324984323 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe8b4003e50 con 0x562776fc7480 2024-05-12T22:18:31.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.516+0000 7fe8a6000640 1 -- 172.21.15.28:0/1324984323 <== mon.2 v2:172.21.15.86:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8513+0+0 (secure 0 0 0) 0x7fe8a80041f0 con 0x562776fc7480 2024-05-12T22:18:31.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.516+0000 7fe8a6000640 1 -- 172.21.15.28:0/1324984323 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fe8a80043f0 con 0x562776fc7480 2024-05-12T22:18:31.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.516+0000 7fe8c5fd8740 10 client.27305 fetch_fsmap learned FSMap version 26 2024-05-12T22:18:31.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.516+0000 7fe8c5fd8740 1 -- 172.21.15.28:0/1324984323 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x562776fe0c90 con 0x562776fc7480 2024-05-12T22:18:31.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.517+0000 7fe8a6000640 1 -- 172.21.15.28:0/1324984323 <== mon.2 v2:172.21.15.86:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe8b4003e50 con 0x562776fc7480 2024-05-12T22:18:31.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.517+0000 7fe8a6000640 1 -- 172.21.15.28:0/1324984323 <== mon.2 v2:172.21.15.86:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fe8a8003b70 con 0x562776fc7480 2024-05-12T22:18:31.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.517+0000 7fe8c5fd8740 10 client.27305 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:18:31.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.517+0000 7fe8c5fd8740 10 client.27305 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-05-12T22:18:31.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.517+0000 7fe8c5fd8740 20 client.27305 populate_metadata read hostname 'smithi028' 2024-05-12T22:18:31.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.517+0000 7fe8c5fd8740 1 --2- 172.21.15.28:0/1324984323 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x562776fe2220 0x562776fe4640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:31.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.517+0000 7fe8c5fd8740 4 client.27305 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-12T22:18:31.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.517+0000 7fe8c5fd8740 1 -- 172.21.15.28:0/1324984323 --> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562776f39160 con 0x562776fe2220 2024-05-12T22:18:31.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.517+0000 7fe8c1200640 1 --2- 172.21.15.28:0/1324984323 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x562776fe2220 0x562776fe4640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:31.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.518+0000 7fe8c1200640 1 --2- 172.21.15.28:0/1324984323 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x562776fe2220 0x562776fe4640 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ede840861e2afee7 server_cookie=d15b94b66195c77e in_seq=0 out_seq=0 2024-05-12T22:18:31.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.518+0000 7fe8a6000640 10 client.27305 ms_handle_connect on v2:172.21.15.28:6840/761183870 2024-05-12T22:18:31.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.518+0000 7fe8a6000640 1 -- 172.21.15.28:0/1324984323 <== mds.1 v2:172.21.15.28:6840/761183870 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562776f39160 con 0x562776fe2220 2024-05-12T22:18:31.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.518+0000 7fe8a6000640 10 client.27305 handle_command_reply: tid=0 2024-05-12T22:18:31.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.519+0000 7fe8c5fd8740 1 client.27305 shutdown 2024-05-12T22:18:31.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.519+0000 7fe8c5fd8740 1 -- 172.21.15.28:0/1324984323 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562776fc7480 msgr2=0x562776fcf080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:18:31.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.519+0000 7fe8c5fd8740 1 --2- 172.21.15.28:0/1324984323 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562776fc7480 0x562776fcf080 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fe8a80126d0 tx=0x7fe8a8012b90 comp rx=0 tx=0).stop 2024-05-12T22:18:31.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.519+0000 7fe8c5fd8740 1 -- 172.21.15.28:0/1324984323 shutdown_connections 2024-05-12T22:18:31.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.519+0000 7fe8c5fd8740 1 --2- 172.21.15.28:0/1324984323 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x562776fe2220 0x562776fe4640 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:31.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.519+0000 7fe8c5fd8740 1 --2- 172.21.15.28:0/1324984323 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562776fd1b50 0x562776fcf5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:31.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.519+0000 7fe8c5fd8740 1 --2- 172.21.15.28:0/1324984323 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562776fc7480 0x562776fcf080 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:31.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.519+0000 7fe8a6000640 0 client.27305 ms_handle_reset on v2:172.21.15.28:6840/761183870 2024-05-12T22:18:31.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.519+0000 7fe8c5fd8740 1 --2- 172.21.15.28:0/1324984323 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562776fc6b60 0x562776fceb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:31.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.519+0000 7fe8c5fd8740 1 -- 172.21.15.28:0/1324984323 >> 172.21.15.28:0/1324984323 conn(0x562776fc2e80 msgr2=0x562776fc8c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:18:31.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.519+0000 7fe8c5fd8740 1 -- 172.21.15.28:0/1324984323 shutdown_connections 2024-05-12T22:18:31.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.519+0000 7fe8c5fd8740 1 -- 172.21.15.28:0/1324984323 wait complete. 2024-05-12T22:18:31.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.519+0000 7fe8c5fd8740 20 client.27305 trim_cache size 0 max 16384 2024-05-12T22:18:31.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.534+0000 7fe8c5fd8740 1 Processor -- start 2024-05-12T22:18:31.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.534+0000 7fe8c5fd8740 1 -- start start 2024-05-12T22:18:31.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.535+0000 7fe8c5fd8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562776fc6b60 0x562776fdd7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:31.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.535+0000 7fe8c5fd8740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562776fc7480 0x562776fbe120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:31.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.535+0000 7fe8c5fd8740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562776fd1b50 0x562776fc92c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:31.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.535+0000 7fe8c5fd8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe8bc1b6100 con 0x562776fc6b60 2024-05-12T22:18:31.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.535+0000 7fe8c5fd8740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fe8bc1b39d0 con 0x562776fc7480 2024-05-12T22:18:31.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.535+0000 7fe8c5fd8740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe8bc1b1840 con 0x562776fd1b50 2024-05-12T22:18:31.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.535+0000 7fe8c1200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562776fc6b60 0x562776fdd7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:31.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.535+0000 7fe8c1200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562776fc6b60 0x562776fdd7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48912/0 (socket says 172.21.15.28:48912) 2024-05-12T22:18:31.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.535+0000 7fe8c1200640 1 -- 172.21.15.28:0/3710271397 learned_addr learned my addr 172.21.15.28:0/3710271397 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:18:31.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.535+0000 7fe8bb400640 1 --2- 172.21.15.28:0/3710271397 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562776fc7480 0x562776fbe120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:31.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.535+0000 7fe8c1200640 1 --2- 172.21.15.28:0/3710271397 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562776fd1b50 0x562776fc92c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:31.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.535+0000 7fe8c1200640 1 -- 172.21.15.28:0/3710271397 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562776fd1b50 msgr2=0x562776fc92c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:18:31.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.535+0000 7fe8c1200640 1 --2- 172.21.15.28:0/3710271397 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562776fd1b50 0x562776fc92c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:31.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.535+0000 7fe8c1200640 1 -- 172.21.15.28:0/3710271397 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562776fc7480 msgr2=0x562776fbe120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:18:31.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.535+0000 7fe8c1200640 1 --2- 172.21.15.28:0/3710271397 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562776fc7480 0x562776fbe120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:31.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.535+0000 7fe8c1200640 1 -- 172.21.15.28:0/3710271397 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562776fcdaf0 con 0x562776fc6b60 2024-05-12T22:18:31.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.536+0000 7fe8c1200640 1 --2- 172.21.15.28:0/3710271397 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562776fc6b60 0x562776fdd7a0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fe8b402fba0 tx=0x7fe8b4005d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b20031588d5150db server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:18:31.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.536+0000 7fe8b0c00640 1 -- 172.21.15.28:0/3710271397 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8b4048040 con 0x562776fc6b60 2024-05-12T22:18:31.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.536+0000 7fe8b0c00640 1 -- 172.21.15.28:0/3710271397 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe8b4049040 con 0x562776fc6b60 2024-05-12T22:18:31.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.536+0000 7fe8b0c00640 1 -- 172.21.15.28:0/3710271397 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8b4030730 con 0x562776fc6b60 2024-05-12T22:18:31.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.537+0000 7fe8c5fd8740 1 -- 172.21.15.28:0/3710271397 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562776fc6b60 msgr2=0x562776fdd7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:18:31.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.537+0000 7fe8c5fd8740 1 --2- 172.21.15.28:0/3710271397 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562776fc6b60 0x562776fdd7a0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fe8b402fba0 tx=0x7fe8b4005d80 comp rx=0 tx=0).stop 2024-05-12T22:18:31.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.537+0000 7fe8c5fd8740 1 -- 172.21.15.28:0/3710271397 shutdown_connections 2024-05-12T22:18:31.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.537+0000 7fe8c5fd8740 1 --2- 172.21.15.28:0/3710271397 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562776fd1b50 0x562776fc92c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:31.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.537+0000 7fe8c5fd8740 1 --2- 172.21.15.28:0/3710271397 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562776fc7480 0x562776fbe120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:31.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.537+0000 7fe8c5fd8740 1 --2- 172.21.15.28:0/3710271397 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562776fc6b60 0x562776fdd7a0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:31.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.537+0000 7fe8c5fd8740 1 -- 172.21.15.28:0/3710271397 >> 172.21.15.28:0/3710271397 conn(0x562776fc2e80 msgr2=0x562776fcf390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:18:31.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.537+0000 7fe8c5fd8740 1 -- 172.21.15.28:0/3710271397 shutdown_connections 2024-05-12T22:18:31.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.537+0000 7fe8c5fd8740 1 -- 172.21.15.28:0/3710271397 wait complete. 2024-05-12T22:18:31.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.537+0000 7fe8c5fd8740 1 -- start start 2024-05-12T22:18:31.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.537+0000 7fe8c5fd8740 1 Processor -- start 2024-05-12T22:18:31.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.538+0000 7fe8c5fd8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562776fc6b60 0x562776fd9f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:31.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.538+0000 7fe8c5fd8740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562776fc7480 0x562776fd5270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:31.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.538+0000 7fe8c5fd8740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562776fd1b50 0x562776fd57b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:31.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.538+0000 7fe8c5fd8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe8bc1b1e90 con 0x562776fc6b60 2024-05-12T22:18:31.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.538+0000 7fe8c5fd8740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fe8bc1b3300 con 0x562776fc7480 2024-05-12T22:18:31.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.538+0000 7fe8c5fd8740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe8bc1b2560 con 0x562776fd1b50 2024-05-12T22:18:31.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.538+0000 7fe8c5fd8740 10 client.0 _ll_register_callbacks cb 0x7fe8a8009e60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:18:31.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.538+0000 7fe8c1200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562776fc6b60 0x562776fd9f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:31.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.538+0000 7fe8c1200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562776fc6b60 0x562776fd9f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48928/0 (socket says 172.21.15.28:48928) 2024-05-12T22:18:31.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.538+0000 7fe8c1200640 1 -- 172.21.15.28:0/3220600772 learned_addr learned my addr 172.21.15.28:0/3220600772 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:18:31.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.538+0000 7fe8bb400640 1 --2- 172.21.15.28:0/3220600772 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562776fc7480 0x562776fd5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:31.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.538+0000 7fe8c1200640 1 --2- 172.21.15.28:0/3220600772 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562776fd1b50 0x562776fd57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:31.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.539+0000 7fe8c1200640 1 -- 172.21.15.28:0/3220600772 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562776fd1b50 msgr2=0x562776fd57b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:18:31.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.539+0000 7fe8c1200640 1 --2- 172.21.15.28:0/3220600772 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562776fd1b50 0x562776fd57b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:31.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.539+0000 7fe8c1200640 1 -- 172.21.15.28:0/3220600772 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562776fc7480 msgr2=0x562776fd5270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:18:31.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.539+0000 7fe8c1200640 1 --2- 172.21.15.28:0/3220600772 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562776fc7480 0x562776fd5270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:31.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.539+0000 7fe8c1200640 1 -- 172.21.15.28:0/3220600772 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562776fd5e20 con 0x562776fc6b60 2024-05-12T22:18:31.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.539+0000 7fe8bb400640 1 --2- 172.21.15.28:0/3220600772 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562776fc7480 0x562776fd5270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:18:31.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.539+0000 7fe8c1200640 1 --2- 172.21.15.28:0/3220600772 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562776fc6b60 0x562776fd9f00 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fe8b402ae70 tx=0x7fe8b402f940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e753e84a4e22c2cf server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:18:31.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.539+0000 7fe8a6000640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-12T22:18:31.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.539+0000 7fe8a6000640 1 -- 172.21.15.28:0/3220600772 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8b405d060 con 0x562776fc6b60 2024-05-12T22:18:31.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.539+0000 7fe8a6000640 1 -- 172.21.15.28:0/3220600772 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8513+0+0 (secure 0 0 0) 0x7fe8b4004b40 con 0x562776fc6b60 2024-05-12T22:18:31.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.539+0000 7fe8c5fd8740 1 -- 172.21.15.28:0/3220600772 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe8b4049040 con 0x562776fc6b60 2024-05-12T22:18:31.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.539+0000 7fe8a6000640 1 -- 172.21.15.28:0/3220600772 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fe8b4048070 con 0x562776fc6b60 2024-05-12T22:18:31.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.540+0000 7fe8c5fd8740 10 client.17178 fetch_fsmap learned FSMap version 26 2024-05-12T22:18:31.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.540+0000 7fe8c5fd8740 1 -- 172.21.15.28:0/3220600772 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x562777048650 con 0x562776fc6b60 2024-05-12T22:18:31.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.540+0000 7fe8a6000640 1 -- 172.21.15.28:0/3220600772 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe8b4049040 con 0x562776fc6b60 2024-05-12T22:18:31.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.540+0000 7fe8a6000640 1 -- 172.21.15.28:0/3220600772 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fe8b4030690 con 0x562776fc6b60 2024-05-12T22:18:31.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.540+0000 7fe8c5fd8740 10 client.17178 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:18:31.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.540+0000 7fe8c5fd8740 10 client.17178 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-05-12T22:18:31.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.540+0000 7fe8c5fd8740 20 client.17178 populate_metadata read hostname 'smithi028' 2024-05-12T22:18:31.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.540+0000 7fe8c5fd8740 1 --2- 172.21.15.28:0/3220600772 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x562777048880 0x56277704ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:31.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.540+0000 7fe8c5fd8740 4 client.17178 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-12T22:18:31.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.540+0000 7fe8c5fd8740 1 -- 172.21.15.28:0/3220600772 --> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562776f334b0 con 0x562777048880 2024-05-12T22:18:31.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.540+0000 7fe8bb400640 1 --2- 172.21.15.28:0/3220600772 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x562777048880 0x56277704ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:31.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.541+0000 7fe8bb400640 1 --2- 172.21.15.28:0/3220600772 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x562777048880 0x56277704ac70 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=deee3ed25b075582 server_cookie=fd4107e87bdc189e in_seq=0 out_seq=0 2024-05-12T22:18:31.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.541+0000 7fe8a6000640 10 client.17178 ms_handle_connect on v2:172.21.15.28:6840/761183870 2024-05-12T22:18:31.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.541+0000 7fe8a6000640 1 -- 172.21.15.28:0/3220600772 <== mds.1 v2:172.21.15.28:6840/761183870 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562776f334b0 con 0x562777048880 2024-05-12T22:18:31.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.541+0000 7fe8a6000640 10 client.17178 handle_command_reply: tid=0 2024-05-12T22:18:31.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.541+0000 7fe8c5fd8740 1 client.17178 shutdown 2024-05-12T22:18:31.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.542+0000 7fe8c5fd8740 1 -- 172.21.15.28:0/3220600772 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562776fc6b60 msgr2=0x562776fd9f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:18:31.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.542+0000 7fe8c5fd8740 1 --2- 172.21.15.28:0/3220600772 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562776fc6b60 0x562776fd9f00 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fe8b402ae70 tx=0x7fe8b402f940 comp rx=0 tx=0).stop 2024-05-12T22:18:31.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.542+0000 7fe8c5fd8740 1 -- 172.21.15.28:0/3220600772 shutdown_connections 2024-05-12T22:18:31.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.542+0000 7fe8c5fd8740 1 --2- 172.21.15.28:0/3220600772 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x562777048880 0x56277704ac70 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:31.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.542+0000 7fe8c5fd8740 1 --2- 172.21.15.28:0/3220600772 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562776fd1b50 0x562776fd57b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:31.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.542+0000 7fe8c5fd8740 1 --2- 172.21.15.28:0/3220600772 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562776fc7480 0x562776fd5270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:31.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.542+0000 7fe8c5fd8740 1 --2- 172.21.15.28:0/3220600772 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562776fc6b60 0x562776fd9f00 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:31.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.542+0000 7fe8c5fd8740 1 -- 172.21.15.28:0/3220600772 >> 172.21.15.28:0/3220600772 conn(0x562776fc2e80 msgr2=0x562776fc1770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:18:31.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.542+0000 7fe8a6000640 0 client.17178 ms_handle_reset on v2:172.21.15.28:6840/761183870 2024-05-12T22:18:31.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.542+0000 7fe8c5fd8740 1 -- 172.21.15.28:0/3220600772 shutdown_connections 2024-05-12T22:18:31.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.542+0000 7fe8c5fd8740 1 -- 172.21.15.28:0/3220600772 wait complete. 2024-05-12T22:18:31.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.542+0000 7fe8c5fd8740 20 client.17178 trim_cache size 0 max 16384 2024-05-12T22:18:31.549 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-05-12T22:18:31.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.543+0000 7fe8c1c00640 1 -- 172.21.15.28:0/485652714 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fe8a00a2d80 msgr2=0x7fe8a00a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:18:31.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.543+0000 7fe8c1c00640 1 --2- 172.21.15.28:0/485652714 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fe8a00a2d80 0x7fe8a00a5240 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fe8ac0ccec0 tx=0x7fe8ac0d5550 comp rx=0 tx=0).stop 2024-05-12T22:18:31.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.543+0000 7fe8c1c00640 1 -- 172.21.15.28:0/485652714 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8bc107dd0 msgr2=0x7fe8bc19e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:18:31.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.543+0000 7fe8c1c00640 1 --2- 172.21.15.28:0/485652714 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8bc107dd0 0x7fe8bc19e500 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fe8ac00e9f0 tx=0x7fe8ac00eeb0 comp rx=0 tx=0).stop 2024-05-12T22:18:31.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.543+0000 7fe8c1c00640 1 -- 172.21.15.28:0/485652714 shutdown_connections 2024-05-12T22:18:31.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.543+0000 7fe8c1c00640 1 --2- 172.21.15.28:0/485652714 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fe8a00a2d80 0x7fe8a00a5240 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:31.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.543+0000 7fe8c1c00640 1 --2- 172.21.15.28:0/485652714 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe8bc10ddf0 0x7fe8bc1a2dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:31.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.543+0000 7fe8c1c00640 1 --2- 172.21.15.28:0/485652714 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe8bc108770 0x7fe8bc19ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:31.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.543+0000 7fe8c1c00640 1 --2- 172.21.15.28:0/485652714 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8bc107dd0 0x7fe8bc19e500 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:31.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.543+0000 7fe8c1c00640 1 -- 172.21.15.28:0/485652714 >> 172.21.15.28:0/485652714 conn(0x7fe8bc077650 msgr2=0x7fe8bc078480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:18:31.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.543+0000 7fe8c1c00640 1 -- 172.21.15.28:0/485652714 shutdown_connections 2024-05-12T22:18:31.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.543+0000 7fe8c1c00640 1 -- 172.21.15.28:0/485652714 wait complete. 2024-05-12T22:18:31.560 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-05-12T22:18:31.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:18:31 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/1391184104' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:18:31.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.780+0000 7f2fd9a00640 1 -- 172.21.15.28:0/3805337900 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2fd4103270 msgr2=0x7f2fd410b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:18:31.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.780+0000 7f2fd9a00640 1 --2- 172.21.15.28:0/3805337900 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2fd4103270 0x7f2fd410b750 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f2fc4009860 tx=0x7f2fc402a3d0 comp rx=0 tx=0).stop 2024-05-12T22:18:31.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.780+0000 7f2fd9a00640 1 -- 172.21.15.28:0/3805337900 shutdown_connections 2024-05-12T22:18:31.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.780+0000 7f2fd9a00640 1 --2- 172.21.15.28:0/3805337900 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2fd410bd20 0x7f2fd410e150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:31.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.780+0000 7f2fd9a00640 1 --2- 172.21.15.28:0/3805337900 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2fd4103270 0x7f2fd410b750 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:31.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.780+0000 7f2fd9a00640 1 --2- 172.21.15.28:0/3805337900 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2fd4102950 0x7f2fd4102d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:31.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.780+0000 7f2fd9a00640 1 -- 172.21.15.28:0/3805337900 >> 172.21.15.28:0/3805337900 conn(0x7f2fd40fc0b0 msgr2=0x7f2fd40fe4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:18:31.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.780+0000 7f2fd9a00640 1 -- 172.21.15.28:0/3805337900 shutdown_connections 2024-05-12T22:18:31.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.780+0000 7f2fd9a00640 1 -- 172.21.15.28:0/3805337900 wait complete. 2024-05-12T22:18:31.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.781+0000 7f2fd9a00640 1 Processor -- start 2024-05-12T22:18:31.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.782+0000 7f2fd9a00640 1 -- start start 2024-05-12T22:18:31.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.782+0000 7f2fd9a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2fd4102950 0x7f2fd41005a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:31.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.782+0000 7f2fd9a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2fd4103270 0x7f2fd4100ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:31.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.782+0000 7f2fd9a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2fd410bd20 0x7f2fd4101020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:31.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.782+0000 7f2fd9a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2fd4111b00 con 0x7f2fd410bd20 2024-05-12T22:18:31.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.782+0000 7f2fd9a00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f2fd4111980 con 0x7f2fd4102950 2024-05-12T22:18:31.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.782+0000 7f2fd9a00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f2fd4111c80 con 0x7f2fd4103270 2024-05-12T22:18:31.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.782+0000 7f2fd3e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2fd4102950 0x7f2fd41005a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:31.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.782+0000 7f2fd3400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2fd4103270 0x7f2fd4100ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:31.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.782+0000 7f2fd3e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2fd4102950 0x7f2fd41005a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:46788/0 (socket says 172.21.15.28:46788) 2024-05-12T22:18:31.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.783+0000 7f2fd3e00640 1 -- 172.21.15.28:0/4226354861 learned_addr learned my addr 172.21.15.28:0/4226354861 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:18:31.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.783+0000 7f2fd3400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2fd4103270 0x7f2fd4100ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:56386/0 (socket says 172.21.15.28:56386) 2024-05-12T22:18:31.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.783+0000 7f2fd9000640 1 --2- 172.21.15.28:0/4226354861 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2fd410bd20 0x7f2fd4101020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:31.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.783+0000 7f2fd3e00640 1 -- 172.21.15.28:0/4226354861 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2fd4103270 msgr2=0x7f2fd4100ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:18:31.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.783+0000 7f2fd3e00640 1 --2- 172.21.15.28:0/4226354861 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2fd4103270 0x7f2fd4100ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:31.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.783+0000 7f2fd3e00640 1 -- 172.21.15.28:0/4226354861 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2fd410bd20 msgr2=0x7f2fd4101020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:18:31.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.783+0000 7f2fd3e00640 1 --2- 172.21.15.28:0/4226354861 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2fd410bd20 0x7f2fd4101020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:31.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.783+0000 7f2fd3e00640 1 -- 172.21.15.28:0/4226354861 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fd41a64e0 con 0x7f2fd4102950 2024-05-12T22:18:31.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.783+0000 7f2fd3e00640 1 --2- 172.21.15.28:0/4226354861 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2fd4102950 0x7f2fd41005a0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f2fbc00e9f0 tx=0x7f2fbc00edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:18:31.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.783+0000 7f2fd0c00640 1 -- 172.21.15.28:0/4226354861 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fbc00cd80 con 0x7f2fd4102950 2024-05-12T22:18:31.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.784+0000 7f2fd9a00640 1 -- 172.21.15.28:0/4226354861 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2fd41a67d0 con 0x7f2fd4102950 2024-05-12T22:18:31.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.784+0000 7f2fd0c00640 1 -- 172.21.15.28:0/4226354861 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2fbc005500 con 0x7f2fd4102950 2024-05-12T22:18:31.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.784+0000 7f2fd0c00640 1 -- 172.21.15.28:0/4226354861 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fbc010690 con 0x7f2fd4102950 2024-05-12T22:18:31.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.784+0000 7f2fd9a00640 1 -- 172.21.15.28:0/4226354861 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2fd41a6d20 con 0x7f2fd4102950 2024-05-12T22:18:31.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.787+0000 7f2fddd91740 1 Processor -- start 2024-05-12T22:18:31.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.787+0000 7f2fddd91740 1 -- start start 2024-05-12T22:18:31.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.787+0000 7f2fd0c00640 1 -- 172.21.15.28:0/4226354861 <== mon.2 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7f2fbc0046c0 con 0x7f2fd4102950 2024-05-12T22:18:31.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.787+0000 7f2fddd91740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560051f16b60 0x560051f16f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:31.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.788+0000 7f2fddd91740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560051f17480 0x560051f21610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:31.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.788+0000 7f2fddd91740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560051f21b50 0x560051f23f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:31.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.788+0000 7f2fddd91740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x560051e7ba30 con 0x560051f16b60 2024-05-12T22:18:31.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.788+0000 7f2fddd91740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x560051e23700 con 0x560051f21b50 2024-05-12T22:18:31.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.788+0000 7f2fddd91740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x560051e779f0 con 0x560051f17480 2024-05-12T22:18:31.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.788+0000 7f2fd9000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560051f16b60 0x560051f16f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:31.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.788+0000 7f2fd9000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560051f16b60 0x560051f16f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48952/0 (socket says 172.21.15.28:48952) 2024-05-12T22:18:31.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.788+0000 7f2fd9000640 1 -- 172.21.15.28:0/2907206583 learned_addr learned my addr 172.21.15.28:0/2907206583 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:18:31.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.788+0000 7f2fd3e00640 1 --2- 172.21.15.28:0/2907206583 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560051f17480 0x560051f21610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:31.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.788+0000 7f2fd3400640 1 --2- 172.21.15.28:0/2907206583 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560051f21b50 0x560051f23f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:31.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.788+0000 7f2fd9000640 1 -- 172.21.15.28:0/2907206583 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560051f17480 msgr2=0x560051f21610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:18:31.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.788+0000 7f2fd9000640 1 --2- 172.21.15.28:0/2907206583 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560051f17480 0x560051f21610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:31.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.788+0000 7f2fd9000640 1 -- 172.21.15.28:0/2907206583 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560051f21b50 msgr2=0x560051f23f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:18:31.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.788+0000 7f2fd9000640 1 --2- 172.21.15.28:0/2907206583 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560051f21b50 0x560051f23f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:31.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.788+0000 7f2fd9000640 1 -- 172.21.15.28:0/2907206583 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560051b8f3f0 con 0x560051f16b60 2024-05-12T22:18:31.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.789+0000 7f2fd0c00640 1 --2- 172.21.15.28:0/4226354861 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f2fb80a2db0 0x7f2fb80a5270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:31.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.789+0000 7f2fd9000640 1 --2- 172.21.15.28:0/2907206583 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560051f16b60 0x560051f16f40 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f2fcc00e480 tx=0x7f2fcc0095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78b74c8ab1290540 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:18:31.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.789+0000 7f2fd0c00640 1 -- 172.21.15.28:0/4226354861 <== mon.2 v2:172.21.15.86:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8513+0+0 (secure 0 0 0) 0x7f2fbc0cf7b0 con 0x7f2fd4102950 2024-05-12T22:18:31.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.789+0000 7f2fd3e00640 1 --2- 172.21.15.28:0/4226354861 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f2fb80a2db0 0x7f2fb80a5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:31.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.789+0000 7f2fc9600640 1 -- 172.21.15.28:0/2907206583 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fcc003970 con 0x560051f16b60 2024-05-12T22:18:31.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.789+0000 7f2fc9600640 1 -- 172.21.15.28:0/2907206583 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2fcc00d7d0 con 0x560051f16b60 2024-05-12T22:18:31.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.789+0000 7f2fc9600640 1 -- 172.21.15.28:0/2907206583 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fcc004470 con 0x560051f16b60 2024-05-12T22:18:31.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.790+0000 7f2fddd91740 1 -- 172.21.15.28:0/2907206583 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560051f16b60 msgr2=0x560051f16f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:18:31.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.790+0000 7f2fd3e00640 1 --2- 172.21.15.28:0/4226354861 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f2fb80a2db0 0x7f2fb80a5270 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f2fbc0ccfb0 tx=0x7f2fbc0d5550 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:18:31.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.790+0000 7f2fddd91740 1 --2- 172.21.15.28:0/2907206583 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560051f16b60 0x560051f16f40 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f2fcc00e480 tx=0x7f2fcc0095c0 comp rx=0 tx=0).stop 2024-05-12T22:18:31.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.790+0000 7f2fddd91740 1 -- 172.21.15.28:0/2907206583 shutdown_connections 2024-05-12T22:18:31.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.790+0000 7f2fddd91740 1 --2- 172.21.15.28:0/2907206583 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560051f21b50 0x560051f23f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:31.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.790+0000 7f2fddd91740 1 --2- 172.21.15.28:0/2907206583 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560051f17480 0x560051f21610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:31.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.790+0000 7f2fddd91740 1 --2- 172.21.15.28:0/2907206583 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560051f16b60 0x560051f16f40 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:31.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.790+0000 7f2fddd91740 1 -- 172.21.15.28:0/2907206583 >> 172.21.15.28:0/2907206583 conn(0x560051f12e80 msgr2=0x560051f13260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:18:31.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.790+0000 7f2fddd91740 1 -- 172.21.15.28:0/2907206583 shutdown_connections 2024-05-12T22:18:31.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.790+0000 7f2fddd91740 1 -- 172.21.15.28:0/2907206583 wait complete. 2024-05-12T22:18:31.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.791+0000 7f2fddd91740 1 -- start start 2024-05-12T22:18:31.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.791+0000 7f2fddd91740 1 Processor -- start 2024-05-12T22:18:31.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.791+0000 7f2fddd91740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560051f16b60 0x560051f1eb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:31.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.791+0000 7f2fddd91740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560051f17480 0x560051f1f060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:31.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.791+0000 7f2fddd91740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560051f21b50 0x560051f1f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:31.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.791+0000 7f2fddd91740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x560051f24940 con 0x560051f21b50 2024-05-12T22:18:31.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.791+0000 7f2fddd91740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x560051e36a70 con 0x560051f17480 2024-05-12T22:18:31.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.791+0000 7f2fddd91740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x560051f24ac0 con 0x560051f16b60 2024-05-12T22:18:31.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.791+0000 7f2fddd91740 10 client.0 _ll_register_callbacks cb 0x560051e19ba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:18:31.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.792+0000 7f2fd9000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560051f16b60 0x560051f1eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:31.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.792+0000 7f2fd3400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560051f17480 0x560051f1f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:31.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.792+0000 7f2fd9000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560051f16b60 0x560051f1eb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:56396/0 (socket says 172.21.15.28:56396) 2024-05-12T22:18:31.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.792+0000 7f2fd9000640 1 -- 172.21.15.28:0/3223709638 learned_addr learned my addr 172.21.15.28:0/3223709638 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:18:31.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.792+0000 7f2fd9000640 1 --2- 172.21.15.28:0/3223709638 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560051f21b50 0x560051f1f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:31.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.792+0000 7f2fd3400640 1 -- 172.21.15.28:0/3223709638 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560051f16b60 msgr2=0x560051f1eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:18:31.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.792+0000 7f2fd3400640 1 --2- 172.21.15.28:0/3223709638 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560051f16b60 0x560051f1eb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:31.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.792+0000 7f2fd3400640 1 -- 172.21.15.28:0/3223709638 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560051f21b50 msgr2=0x560051f1f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:18:31.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.792+0000 7f2fd3400640 1 --2- 172.21.15.28:0/3223709638 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560051f21b50 0x560051f1f5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:31.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.792+0000 7f2fd3400640 1 -- 172.21.15.28:0/3223709638 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560051c597d0 con 0x560051f17480 2024-05-12T22:18:31.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.792+0000 7f2fd9000640 1 --2- 172.21.15.28:0/3223709638 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560051f16b60 0x560051f1eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:18:31.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.793+0000 7f2fd3400640 1 --2- 172.21.15.28:0/3223709638 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560051f17480 0x560051f1f060 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f2fc4000c00 tx=0x7f2fc40305d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f82a7620f6bcd93b server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:18:31.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.793+0000 7f2fc2000640 10 client.0 ms_handle_connect on v2:172.21.15.86:3300/0 2024-05-12T22:18:31.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.793+0000 7f2fc2000640 1 -- 172.21.15.28:0/3223709638 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fc4004050 con 0x560051f17480 2024-05-12T22:18:31.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.793+0000 7f2fc2000640 1 -- 172.21.15.28:0/3223709638 <== mon.2 v2:172.21.15.86:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8513+0+0 (secure 0 0 0) 0x7f2fc40041f0 con 0x560051f17480 2024-05-12T22:18:31.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.793+0000 7f2fddd91740 1 -- 172.21.15.28:0/3223709638 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2fcc00d7d0 con 0x560051f17480 2024-05-12T22:18:31.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.793+0000 7f2fc2000640 1 -- 172.21.15.28:0/3223709638 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f2fc404e050 con 0x560051f17480 2024-05-12T22:18:31.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.793+0000 7f2fddd91740 10 client.27323 fetch_fsmap learned FSMap version 26 2024-05-12T22:18:31.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.793+0000 7f2fddd91740 1 -- 172.21.15.28:0/3223709638 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x560051f30c30 con 0x560051f17480 2024-05-12T22:18:31.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.793+0000 7f2fc2000640 1 -- 172.21.15.28:0/3223709638 <== mon.2 v2:172.21.15.86:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2fcc00d7d0 con 0x560051f17480 2024-05-12T22:18:31.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.794+0000 7f2fc2000640 1 -- 172.21.15.28:0/3223709638 <== mon.2 v2:172.21.15.86:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f2fc40048f0 con 0x560051f17480 2024-05-12T22:18:31.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.794+0000 7f2fddd91740 10 client.27323 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:18:31.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.794+0000 7f2fddd91740 10 client.27323 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:18:31.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.794+0000 7f2fddd91740 20 client.27323 populate_metadata read hostname 'smithi028' 2024-05-12T22:18:31.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.794+0000 7f2fddd91740 1 --2- 172.21.15.28:0/3223709638 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x560051f32090 0x560051f344c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:31.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.794+0000 7f2fddd91740 4 client.27323 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-12T22:18:31.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.794+0000 7f2fddd91740 1 -- 172.21.15.28:0/3223709638 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560051e89160 con 0x560051f32090 2024-05-12T22:18:31.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.794+0000 7f2fd9000640 1 --2- 172.21.15.28:0/3223709638 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x560051f32090 0x560051f344c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:31.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.794+0000 7f2fd9000640 1 --2- 172.21.15.28:0/3223709638 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x560051f32090 0x560051f344c0 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=44580383cb3e9b36 server_cookie=8d719d95067d28b3 in_seq=0 out_seq=0 2024-05-12T22:18:31.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.794+0000 7f2fc2000640 10 client.27323 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:18:31.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.795+0000 7f2fc2000640 1 -- 172.21.15.28:0/3223709638 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560051e89160 con 0x560051f32090 2024-05-12T22:18:31.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.795+0000 7f2fc2000640 10 client.27323 handle_command_reply: tid=0 2024-05-12T22:18:31.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.795+0000 7f2fddd91740 1 client.27323 shutdown 2024-05-12T22:18:31.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.796+0000 7f2fddd91740 1 -- 172.21.15.28:0/3223709638 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560051f17480 msgr2=0x560051f1f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:18:31.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.796+0000 7f2fddd91740 1 --2- 172.21.15.28:0/3223709638 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560051f17480 0x560051f1f060 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f2fc4000c00 tx=0x7f2fc40305d0 comp rx=0 tx=0).stop 2024-05-12T22:18:31.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.796+0000 7f2fddd91740 1 -- 172.21.15.28:0/3223709638 shutdown_connections 2024-05-12T22:18:31.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.796+0000 7f2fddd91740 1 --2- 172.21.15.28:0/3223709638 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x560051f32090 0x560051f344c0 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:31.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.796+0000 7f2fddd91740 1 --2- 172.21.15.28:0/3223709638 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560051f21b50 0x560051f1f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:31.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.796+0000 7f2fddd91740 1 --2- 172.21.15.28:0/3223709638 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560051f17480 0x560051f1f060 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:31.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.796+0000 7f2fddd91740 1 --2- 172.21.15.28:0/3223709638 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560051f16b60 0x560051f1eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:31.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.796+0000 7f2fddd91740 1 -- 172.21.15.28:0/3223709638 >> 172.21.15.28:0/3223709638 conn(0x560051f12e80 msgr2=0x560051f18c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:18:31.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.796+0000 7f2fc2000640 0 client.27323 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:18:31.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.796+0000 7f2fddd91740 1 -- 172.21.15.28:0/3223709638 shutdown_connections 2024-05-12T22:18:31.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.796+0000 7f2fddd91740 1 -- 172.21.15.28:0/3223709638 wait complete. 2024-05-12T22:18:31.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.796+0000 7f2fddd91740 20 client.27323 trim_cache size 0 max 16384 2024-05-12T22:18:31.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.811+0000 7f2fddd91740 1 Processor -- start 2024-05-12T22:18:31.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.811+0000 7f2fddd91740 1 -- start start 2024-05-12T22:18:31.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.811+0000 7f2fddd91740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560051f16b60 0x560051f2d800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:31.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.811+0000 7f2fddd91740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560051f17480 0x560051f192c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:31.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.811+0000 7f2fddd91740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560051f21b50 0x560051f39d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:31.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.811+0000 7f2fddd91740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2fd41b1e80 con 0x560051f16b60 2024-05-12T22:18:31.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.811+0000 7f2fddd91740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f2fd41af750 con 0x560051f17480 2024-05-12T22:18:31.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.811+0000 7f2fddd91740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f2fd41ad5c0 con 0x560051f21b50 2024-05-12T22:18:31.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.811+0000 7f2fd9000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560051f16b60 0x560051f2d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:31.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.812+0000 7f2fd9000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560051f16b60 0x560051f2d800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48974/0 (socket says 172.21.15.28:48974) 2024-05-12T22:18:31.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.812+0000 7f2fd9000640 1 -- 172.21.15.28:0/3472395297 learned_addr learned my addr 172.21.15.28:0/3472395297 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:18:31.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.812+0000 7f2fd3400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560051f17480 0x560051f192c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:31.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.812+0000 7f2fd9000640 1 --2- 172.21.15.28:0/3472395297 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560051f21b50 0x560051f39d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:31.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.812+0000 7f2fd3400640 1 -- 172.21.15.28:0/3472395297 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560051f21b50 msgr2=0x560051f39d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:18:31.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.812+0000 7f2fd3400640 1 --2- 172.21.15.28:0/3472395297 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560051f21b50 0x560051f39d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:31.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.812+0000 7f2fd3400640 1 -- 172.21.15.28:0/3472395297 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560051f16b60 msgr2=0x560051f2d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:18:31.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.812+0000 7f2fd3400640 1 --2- 172.21.15.28:0/3472395297 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560051f16b60 0x560051f2d800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:31.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.812+0000 7f2fd3400640 1 -- 172.21.15.28:0/3472395297 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560051f23f40 con 0x560051f17480 2024-05-12T22:18:31.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.812+0000 7f2fd9000640 1 --2- 172.21.15.28:0/3472395297 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560051f16b60 0x560051f2d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:18:31.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.812+0000 7f2fd3400640 1 --2- 172.21.15.28:0/3472395297 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560051f17480 0x560051f192c0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f2fc4004de0 tx=0x7f2fc40048d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c031ab66f91454c2 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:18:31.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.813+0000 7f2fc8c00640 1 -- 172.21.15.28:0/3472395297 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fc4057070 con 0x560051f17480 2024-05-12T22:18:31.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.813+0000 7f2fc8c00640 1 -- 172.21.15.28:0/3472395297 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2fc4004ac0 con 0x560051f17480 2024-05-12T22:18:31.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.813+0000 7f2fc8c00640 1 -- 172.21.15.28:0/3472395297 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fc4052430 con 0x560051f17480 2024-05-12T22:18:31.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.813+0000 7f2fddd91740 1 -- 172.21.15.28:0/3472395297 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560051f17480 msgr2=0x560051f192c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:18:31.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.813+0000 7f2fddd91740 1 --2- 172.21.15.28:0/3472395297 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560051f17480 0x560051f192c0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f2fc4004de0 tx=0x7f2fc40048d0 comp rx=0 tx=0).stop 2024-05-12T22:18:31.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.813+0000 7f2fddd91740 1 -- 172.21.15.28:0/3472395297 shutdown_connections 2024-05-12T22:18:31.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.813+0000 7f2fddd91740 1 --2- 172.21.15.28:0/3472395297 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560051f21b50 0x560051f39d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:31.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.813+0000 7f2fddd91740 1 --2- 172.21.15.28:0/3472395297 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560051f17480 0x560051f192c0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:31.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.813+0000 7f2fddd91740 1 --2- 172.21.15.28:0/3472395297 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560051f16b60 0x560051f2d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:31.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.813+0000 7f2fddd91740 1 -- 172.21.15.28:0/3472395297 >> 172.21.15.28:0/3472395297 conn(0x560051f12e80 msgr2=0x560051f1f370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:18:31.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.813+0000 7f2fddd91740 1 -- 172.21.15.28:0/3472395297 shutdown_connections 2024-05-12T22:18:31.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.814+0000 7f2fddd91740 1 -- 172.21.15.28:0/3472395297 wait complete. 2024-05-12T22:18:31.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.814+0000 7f2fddd91740 1 -- start start 2024-05-12T22:18:31.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.814+0000 7f2fddd91740 1 Processor -- start 2024-05-12T22:18:31.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.815+0000 7f2fddd91740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560051f16b60 0x560051f2a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:31.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.815+0000 7f2fddd91740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560051f17480 0x560051f25400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:31.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.815+0000 7f2fddd91740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560051f21b50 0x560051f25940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:31.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.815+0000 7f2fddd91740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2fd41adc10 con 0x560051f16b60 2024-05-12T22:18:31.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.815+0000 7f2fddd91740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f2fd41af080 con 0x560051f17480 2024-05-12T22:18:31.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.815+0000 7f2fddd91740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f2fd41ae2e0 con 0x560051f21b50 2024-05-12T22:18:31.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.815+0000 7f2fddd91740 10 client.0 _ll_register_callbacks cb 0x7f2fc4032ee0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:18:31.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.815+0000 7f2fd9000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560051f16b60 0x560051f2a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:31.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.815+0000 7f2fd3400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560051f17480 0x560051f25400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:31.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.815+0000 7f2fd9000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560051f16b60 0x560051f2a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48982/0 (socket says 172.21.15.28:48982) 2024-05-12T22:18:31.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.815+0000 7f2fd3400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560051f17480 0x560051f25400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:46834/0 (socket says 172.21.15.28:46834) 2024-05-12T22:18:31.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.815+0000 7f2fd9000640 1 -- 172.21.15.28:0/1432355383 learned_addr learned my addr 172.21.15.28:0/1432355383 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:18:31.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.815+0000 7f2fd9000640 1 --2- 172.21.15.28:0/1432355383 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560051f21b50 0x560051f25940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:31.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.815+0000 7f2fd3400640 1 -- 172.21.15.28:0/1432355383 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560051f21b50 msgr2=0x560051f25940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:18:31.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.815+0000 7f2fd3400640 1 --2- 172.21.15.28:0/1432355383 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560051f21b50 0x560051f25940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:31.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.815+0000 7f2fd3400640 1 -- 172.21.15.28:0/1432355383 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560051f16b60 msgr2=0x560051f2a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:18:31.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.815+0000 7f2fd3400640 1 --2- 172.21.15.28:0/1432355383 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560051f16b60 0x560051f2a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:31.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.816+0000 7f2fd3400640 1 -- 172.21.15.28:0/1432355383 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560051f26140 con 0x560051f17480 2024-05-12T22:18:31.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.816+0000 7f2fd9000640 1 --2- 172.21.15.28:0/1432355383 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560051f16b60 0x560051f2a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:18:31.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.816+0000 7f2fd3400640 1 --2- 172.21.15.28:0/1432355383 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560051f17480 0x560051f25400 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x560051f2e560 tx=0x7f2fc402ab50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a3619f4baebe631a server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:18:31.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.816+0000 7f2fc2000640 10 client.0 ms_handle_connect on v2:172.21.15.86:3300/0 2024-05-12T22:18:31.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.816+0000 7f2fc2000640 1 -- 172.21.15.28:0/1432355383 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fc4060060 con 0x560051f17480 2024-05-12T22:18:31.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.816+0000 7f2fddd91740 1 -- 172.21.15.28:0/1432355383 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2fc4004ac0 con 0x560051f17480 2024-05-12T22:18:31.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.816+0000 7f2fc2000640 1 -- 172.21.15.28:0/1432355383 <== mon.2 v2:172.21.15.86:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8513+0+0 (secure 0 0 0) 0x7f2fc4005290 con 0x560051f17480 2024-05-12T22:18:31.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.816+0000 7f2fc2000640 1 -- 172.21.15.28:0/1432355383 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f2fc4057040 con 0x560051f17480 2024-05-12T22:18:31.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.816+0000 7f2fddd91740 10 client.27329 fetch_fsmap learned FSMap version 26 2024-05-12T22:18:31.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.816+0000 7f2fddd91740 1 -- 172.21.15.28:0/1432355383 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x560051f985d0 con 0x560051f17480 2024-05-12T22:18:31.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.817+0000 7f2fc2000640 1 -- 172.21.15.28:0/1432355383 <== mon.2 v2:172.21.15.86:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2fc4004ac0 con 0x560051f17480 2024-05-12T22:18:31.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.817+0000 7f2fc2000640 1 -- 172.21.15.28:0/1432355383 <== mon.2 v2:172.21.15.86:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f2fc4052910 con 0x560051f17480 2024-05-12T22:18:31.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.817+0000 7f2fddd91740 10 client.27329 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:18:31.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.817+0000 7f2fddd91740 10 client.27329 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:18:31.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.817+0000 7f2fddd91740 20 client.27329 populate_metadata read hostname 'smithi028' 2024-05-12T22:18:31.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.817+0000 7f2fddd91740 1 --2- 172.21.15.28:0/1432355383 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x560051f98bd0 0x560051f9afc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:31.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.817+0000 7f2fddd91740 4 client.27329 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-12T22:18:31.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.817+0000 7f2fddd91740 1 -- 172.21.15.28:0/1432355383 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560051e834b0 con 0x560051f98bd0 2024-05-12T22:18:31.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.817+0000 7f2fd9000640 1 --2- 172.21.15.28:0/1432355383 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x560051f98bd0 0x560051f9afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:31.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.818+0000 7f2fd9000640 1 --2- 172.21.15.28:0/1432355383 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x560051f98bd0 0x560051f9afc0 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6ad3e91e518d727e server_cookie=2e5d6f0f36d77a4 in_seq=0 out_seq=0 2024-05-12T22:18:31.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.818+0000 7f2fc2000640 10 client.27329 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:18:31.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.818+0000 7f2fc2000640 1 -- 172.21.15.28:0/1432355383 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560051e834b0 con 0x560051f98bd0 2024-05-12T22:18:31.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.818+0000 7f2fc2000640 10 client.27329 handle_command_reply: tid=0 2024-05-12T22:18:31.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.818+0000 7f2fddd91740 1 client.27329 shutdown 2024-05-12T22:18:31.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.818+0000 7f2fddd91740 1 -- 172.21.15.28:0/1432355383 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560051f17480 msgr2=0x560051f25400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:18:31.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.818+0000 7f2fddd91740 1 --2- 172.21.15.28:0/1432355383 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560051f17480 0x560051f25400 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x560051f2e560 tx=0x7f2fc402ab50 comp rx=0 tx=0).stop 2024-05-12T22:18:31.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.818+0000 7f2fddd91740 1 -- 172.21.15.28:0/1432355383 shutdown_connections 2024-05-12T22:18:31.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.818+0000 7f2fddd91740 1 --2- 172.21.15.28:0/1432355383 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x560051f98bd0 0x560051f9afc0 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:31.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.818+0000 7f2fddd91740 1 --2- 172.21.15.28:0/1432355383 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560051f21b50 0x560051f25940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:31.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.818+0000 7f2fddd91740 1 --2- 172.21.15.28:0/1432355383 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560051f17480 0x560051f25400 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:31.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.818+0000 7f2fc2000640 0 client.27329 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:18:31.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.818+0000 7f2fddd91740 1 --2- 172.21.15.28:0/1432355383 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560051f16b60 0x560051f2a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:31.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.818+0000 7f2fddd91740 1 -- 172.21.15.28:0/1432355383 >> 172.21.15.28:0/1432355383 conn(0x560051f12e80 msgr2=0x560051f20350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:18:31.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.818+0000 7f2fddd91740 1 -- 172.21.15.28:0/1432355383 shutdown_connections 2024-05-12T22:18:31.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.818+0000 7f2fddd91740 1 -- 172.21.15.28:0/1432355383 wait complete. 2024-05-12T22:18:31.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.818+0000 7f2fddd91740 20 client.27329 trim_cache size 0 max 16384 2024-05-12T22:18:31.826 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-05-12T22:18:31.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.819+0000 7f2fd9a00640 1 -- 172.21.15.28:0/4226354861 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f2fb80a2db0 msgr2=0x7f2fb80a5270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:18:31.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.819+0000 7f2fd9a00640 1 --2- 172.21.15.28:0/4226354861 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f2fb80a2db0 0x7f2fb80a5270 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f2fbc0ccfb0 tx=0x7f2fbc0d5550 comp rx=0 tx=0).stop 2024-05-12T22:18:31.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.819+0000 7f2fd9a00640 1 -- 172.21.15.28:0/4226354861 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2fd4102950 msgr2=0x7f2fd41005a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:18:31.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.819+0000 7f2fd9a00640 1 --2- 172.21.15.28:0/4226354861 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2fd4102950 0x7f2fd41005a0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f2fbc00e9f0 tx=0x7f2fbc00edf0 comp rx=0 tx=0).stop 2024-05-12T22:18:31.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.819+0000 7f2fd9a00640 1 -- 172.21.15.28:0/4226354861 shutdown_connections 2024-05-12T22:18:31.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.819+0000 7f2fd9a00640 1 --2- 172.21.15.28:0/4226354861 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f2fb80a2db0 0x7f2fb80a5270 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:31.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.819+0000 7f2fd9a00640 1 --2- 172.21.15.28:0/4226354861 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2fd410bd20 0x7f2fd4101020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:31.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.819+0000 7f2fd9a00640 1 --2- 172.21.15.28:0/4226354861 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2fd4103270 0x7f2fd4100ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:31.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.819+0000 7f2fd9a00640 1 --2- 172.21.15.28:0/4226354861 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2fd4102950 0x7f2fd41005a0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:31.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.819+0000 7f2fd9a00640 1 -- 172.21.15.28:0/4226354861 >> 172.21.15.28:0/4226354861 conn(0x7f2fd40fc0b0 msgr2=0x7f2fd40fd8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:18:31.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.819+0000 7f2fd9a00640 1 -- 172.21.15.28:0/4226354861 shutdown_connections 2024-05-12T22:18:31.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:31.819+0000 7f2fd9a00640 1 -- 172.21.15.28:0/4226354861 wait complete. 2024-05-12T22:18:32.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:18:32 smithi086 ceph-mon[34297]: pgmap v295: 97 pgs: 97 active+clean; 91 MiB data, 725 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-12T22:18:32.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:18:32 smithi028 ceph-mon[29847]: pgmap v295: 97 pgs: 97 active+clean; 91 MiB data, 725 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-12T22:18:32.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:18:32 smithi186 ceph-mon[34785]: pgmap v295: 97 pgs: 97 active+clean; 91 MiB data, 725 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-12T22:18:32.837 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-12T22:18:32.838 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-12T22:18:32.838 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-05-12T22:18:33.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.071+0000 7f72b9000640 1 -- 172.21.15.28:0/1808512742 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f72b4107dd0 msgr2=0x7f72b4108230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:18:33.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.071+0000 7f72b9000640 1 --2- 172.21.15.28:0/1808512742 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f72b4107dd0 0x7f72b4108230 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f729c009860 tx=0x7f729c02a3d0 comp rx=0 tx=0).stop 2024-05-12T22:18:33.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.071+0000 7f72b9000640 1 -- 172.21.15.28:0/1808512742 shutdown_connections 2024-05-12T22:18:33.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.071+0000 7f72b9000640 1 --2- 172.21.15.28:0/1808512742 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f72b4108770 0x7f72b4114b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:33.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.071+0000 7f72b9000640 1 --2- 172.21.15.28:0/1808512742 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f72b4107dd0 0x7f72b4108230 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:33.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.071+0000 7f72b9000640 1 --2- 172.21.15.28:0/1808512742 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72b410ddf0 0x7f72b410e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:33.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.071+0000 7f72b9000640 1 -- 172.21.15.28:0/1808512742 >> 172.21.15.28:0/1808512742 conn(0x7f72b4077650 msgr2=0x7f72b4106650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:18:33.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.071+0000 7f72b9000640 1 -- 172.21.15.28:0/1808512742 shutdown_connections 2024-05-12T22:18:33.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.071+0000 7f72b9000640 1 -- 172.21.15.28:0/1808512742 wait complete. 2024-05-12T22:18:33.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.072+0000 7f72b9000640 1 Processor -- start 2024-05-12T22:18:33.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.073+0000 7f72b9000640 1 -- start start 2024-05-12T22:18:33.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.073+0000 7f72b9000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72b4107dd0 0x7f72b419e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:33.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.073+0000 7f72b9000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f72b4108770 0x7f72b419ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:33.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.073+0000 7f72b9000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f72b410ddf0 0x7f72b41a2dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:33.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.073+0000 7f72b9000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f72b4118100 con 0x7f72b4107dd0 2024-05-12T22:18:33.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.073+0000 7f72b9000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f72b4117f80 con 0x7f72b410ddf0 2024-05-12T22:18:33.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.073+0000 7f72b9000640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f72b4118280 con 0x7f72b4108770 2024-05-12T22:18:33.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.073+0000 7f72b3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72b4107dd0 0x7f72b419e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:33.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.074+0000 7f72b3e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f72b410ddf0 0x7f72b41a2dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:33.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.074+0000 7f72b3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72b4107dd0 0x7f72b419e500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48990/0 (socket says 172.21.15.28:48990) 2024-05-12T22:18:33.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.074+0000 7f72b3400640 1 -- 172.21.15.28:0/2822894466 learned_addr learned my addr 172.21.15.28:0/2822894466 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:18:33.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.074+0000 7f72abe00640 1 --2- 172.21.15.28:0/2822894466 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f72b4108770 0x7f72b419ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:33.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.074+0000 7f72b3400640 1 -- 172.21.15.28:0/2822894466 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f72b4108770 msgr2=0x7f72b419ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:18:33.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.074+0000 7f72b3400640 1 --2- 172.21.15.28:0/2822894466 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f72b4108770 0x7f72b419ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:33.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.074+0000 7f72b3400640 1 -- 172.21.15.28:0/2822894466 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f72b410ddf0 msgr2=0x7f72b41a2dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:18:33.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.074+0000 7f72b3400640 1 --2- 172.21.15.28:0/2822894466 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f72b410ddf0 0x7f72b41a2dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:33.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.074+0000 7f72b3400640 1 -- 172.21.15.28:0/2822894466 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72b41a34f0 con 0x7f72b4107dd0 2024-05-12T22:18:33.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.074+0000 7f72b3e00640 1 --2- 172.21.15.28:0/2822894466 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f72b410ddf0 0x7f72b41a2dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:18:33.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.074+0000 7f72b3400640 1 --2- 172.21.15.28:0/2822894466 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72b4107dd0 0x7f72b419e500 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f72a400e9c0 tx=0x7f72a400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:18:33.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.075+0000 7f72b0c00640 1 -- 172.21.15.28:0/2822894466 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72a400ccc0 con 0x7f72b4107dd0 2024-05-12T22:18:33.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.075+0000 7f72b0c00640 1 -- 172.21.15.28:0/2822894466 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f72a4005500 con 0x7f72b4107dd0 2024-05-12T22:18:33.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.075+0000 7f72b9000640 1 -- 172.21.15.28:0/2822894466 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72b41a37e0 con 0x7f72b4107dd0 2024-05-12T22:18:33.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.075+0000 7f72b0c00640 1 -- 172.21.15.28:0/2822894466 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72a4010430 con 0x7f72b4107dd0 2024-05-12T22:18:33.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.075+0000 7f72b9000640 1 -- 172.21.15.28:0/2822894466 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72b41ab070 con 0x7f72b4107dd0 2024-05-12T22:18:33.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.078+0000 7f72bd3cf740 1 Processor -- start 2024-05-12T22:18:33.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.079+0000 7f72bd3cf740 1 -- start start 2024-05-12T22:18:33.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.079+0000 7f72b0c00640 1 -- 172.21.15.28:0/2822894466 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7f72a40046c0 con 0x7f72b4107dd0 2024-05-12T22:18:33.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.079+0000 7f72bd3cf740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a59cb24500 0x55a59cb248e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:33.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.079+0000 7f72bd3cf740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a59cb24eb0 0x55a59cb2ef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:33.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.079+0000 7f72bd3cf740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a59cb2f520 0x55a59cb31970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:33.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.079+0000 7f72bd3cf740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a59ca887c0 con 0x55a59cb24eb0 2024-05-12T22:18:33.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.079+0000 7f72bd3cf740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x55a59ca30790 con 0x55a59cb2f520 2024-05-12T22:18:33.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.079+0000 7f72bd3cf740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55a59ca84780 con 0x55a59cb24500 2024-05-12T22:18:33.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.079+0000 7f72b3e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a59cb24500 0x55a59cb248e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:33.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.079+0000 7f72b3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a59cb24eb0 0x55a59cb2ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:33.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.079+0000 7f72b3e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a59cb24500 0x55a59cb248e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:56428/0 (socket says 172.21.15.28:56428) 2024-05-12T22:18:33.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.079+0000 7f72abe00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a59cb2f520 0x55a59cb31970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:33.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.079+0000 7f72b3e00640 1 -- 172.21.15.28:0/2443112821 learned_addr learned my addr 172.21.15.28:0/2443112821 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:18:33.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.080+0000 7f72b3e00640 1 -- 172.21.15.28:0/2443112821 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a59cb2f520 msgr2=0x55a59cb31970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:18:33.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.080+0000 7f72b3e00640 1 --2- 172.21.15.28:0/2443112821 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a59cb2f520 0x55a59cb31970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:33.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.080+0000 7f72b3e00640 1 -- 172.21.15.28:0/2443112821 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a59cb24eb0 msgr2=0x55a59cb2ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:18:33.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.080+0000 7f72b3e00640 1 --2- 172.21.15.28:0/2443112821 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a59cb24eb0 0x55a59cb2ef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:33.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.080+0000 7f72b3e00640 1 -- 172.21.15.28:0/2443112821 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a59ca506a0 con 0x55a59cb24500 2024-05-12T22:18:33.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.080+0000 7f72abe00640 1 --2- 172.21.15.28:0/2443112821 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a59cb2f520 0x55a59cb31970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:18:33.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.080+0000 7f72b3e00640 1 --2- 172.21.15.28:0/2443112821 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a59cb24500 0x55a59cb248e0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f72ac0117f0 tx=0x7f72ac011cb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1926b28d928e60da server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:18:33.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.081+0000 7f72b0c00640 1 --2- 172.21.15.28:0/2822894466 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f72980a2d00 0x7f72980a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:33.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.081+0000 7f72a8c00640 1 -- 172.21.15.28:0/2443112821 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72ac0112f0 con 0x55a59cb24500 2024-05-12T22:18:33.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.081+0000 7f72a8c00640 1 -- 172.21.15.28:0/2443112821 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f72ac003e50 con 0x55a59cb24500 2024-05-12T22:18:33.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.081+0000 7f72a8c00640 1 -- 172.21.15.28:0/2443112821 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72ac0045f0 con 0x55a59cb24500 2024-05-12T22:18:33.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.081+0000 7f72b0c00640 1 -- 172.21.15.28:0/2822894466 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8513+0+0 (secure 0 0 0) 0x7f72a40cf7a0 con 0x7f72b4107dd0 2024-05-12T22:18:33.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.081+0000 7f72b3400640 1 --2- 172.21.15.28:0/2822894466 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f72980a2d00 0x7f72980a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:33.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.081+0000 7f72bd3cf740 1 -- 172.21.15.28:0/2443112821 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a59cb24500 msgr2=0x55a59cb248e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:18:33.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.081+0000 7f72b3400640 1 --2- 172.21.15.28:0/2822894466 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f72980a2d00 0x7f72980a51c0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x55a59cb25ac0 tx=0x7f72a40d5530 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:18:33.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.081+0000 7f72bd3cf740 1 --2- 172.21.15.28:0/2443112821 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a59cb24500 0x55a59cb248e0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f72ac0117f0 tx=0x7f72ac011cb0 comp rx=0 tx=0).stop 2024-05-12T22:18:33.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.082+0000 7f72bd3cf740 1 -- 172.21.15.28:0/2443112821 shutdown_connections 2024-05-12T22:18:33.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.082+0000 7f72bd3cf740 1 --2- 172.21.15.28:0/2443112821 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a59cb2f520 0x55a59cb31970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:33.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.082+0000 7f72bd3cf740 1 --2- 172.21.15.28:0/2443112821 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a59cb24eb0 0x55a59cb2ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:33.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.082+0000 7f72bd3cf740 1 --2- 172.21.15.28:0/2443112821 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a59cb24500 0x55a59cb248e0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:33.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.082+0000 7f72bd3cf740 1 -- 172.21.15.28:0/2443112821 >> 172.21.15.28:0/2443112821 conn(0x55a59cb205b0 msgr2=0x55a59cb20990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:18:33.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.082+0000 7f72bd3cf740 1 -- 172.21.15.28:0/2443112821 shutdown_connections 2024-05-12T22:18:33.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.082+0000 7f72bd3cf740 1 -- 172.21.15.28:0/2443112821 wait complete. 2024-05-12T22:18:33.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.082+0000 7f72bd3cf740 1 -- start start 2024-05-12T22:18:33.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.082+0000 7f72bd3cf740 1 Processor -- start 2024-05-12T22:18:33.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.083+0000 7f72bd3cf740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a59cb24500 0x55a59cb2c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:33.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.083+0000 7f72bd3cf740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a59cb24eb0 0x55a59cb2c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:33.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.083+0000 7f72bd3cf740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a59cb2f520 0x55a59cb2ce50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:33.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.083+0000 7f72bd3cf740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a59cb32530 con 0x55a59cb2f520 2024-05-12T22:18:33.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.083+0000 7f72bd3cf740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x55a59ca43a90 con 0x55a59cb24eb0 2024-05-12T22:18:33.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.083+0000 7f72bd3cf740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55a59cb326b0 con 0x55a59cb24500 2024-05-12T22:18:33.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.083+0000 7f72bd3cf740 10 client.0 _ll_register_callbacks cb 0x55a59ca7eaf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:18:33.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.083+0000 7f72b3e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a59cb24500 0x55a59cb2c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:33.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.083+0000 7f72b3e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a59cb24500 0x55a59cb2c3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:56430/0 (socket says 172.21.15.28:56430) 2024-05-12T22:18:33.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.083+0000 7f72b3e00640 1 -- 172.21.15.28:0/2155441320 learned_addr learned my addr 172.21.15.28:0/2155441320 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:18:33.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.083+0000 7f72b3e00640 1 --2- 172.21.15.28:0/2155441320 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a59cb2f520 0x55a59cb2ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:33.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.083+0000 7f72abe00640 1 --2- 172.21.15.28:0/2155441320 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a59cb24eb0 0x55a59cb2c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:33.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.084+0000 7f72b3e00640 1 -- 172.21.15.28:0/2155441320 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a59cb24eb0 msgr2=0x55a59cb2c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:18:33.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.084+0000 7f72b3e00640 1 --2- 172.21.15.28:0/2155441320 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a59cb24eb0 0x55a59cb2c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:33.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.084+0000 7f72b3e00640 1 -- 172.21.15.28:0/2155441320 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a59cb2f520 msgr2=0x55a59cb2ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:18:33.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.084+0000 7f72b3e00640 1 --2- 172.21.15.28:0/2155441320 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a59cb2f520 0x55a59cb2ce50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:33.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.084+0000 7f72b3e00640 1 -- 172.21.15.28:0/2155441320 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a59c8647e0 con 0x55a59cb24500 2024-05-12T22:18:33.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.084+0000 7f72b3e00640 1 --2- 172.21.15.28:0/2155441320 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a59cb24500 0x55a59cb2c3d0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f72ac003d80 tx=0x7f72ac004190 comp rx=0 tx=0).ready entity=mon.1 client_cookie=71c895de88adeac6 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:18:33.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.084+0000 7f72a1600640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-12T22:18:33.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.084+0000 7f72a1600640 1 -- 172.21.15.28:0/2155441320 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72ac01e070 con 0x55a59cb24500 2024-05-12T22:18:33.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.084+0000 7f72a1600640 1 -- 172.21.15.28:0/2155441320 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8513+0+0 (secure 0 0 0) 0x7f72ac004330 con 0x55a59cb24500 2024-05-12T22:18:33.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.084+0000 7f72bd3cf740 1 -- 172.21.15.28:0/2155441320 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f72ac003e50 con 0x55a59cb24500 2024-05-12T22:18:33.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.085+0000 7f72a1600640 1 -- 172.21.15.28:0/2155441320 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f72ac025040 con 0x55a59cb24500 2024-05-12T22:18:33.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.085+0000 7f72bd3cf740 10 client.27292 fetch_fsmap learned FSMap version 26 2024-05-12T22:18:33.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.085+0000 7f72bd3cf740 1 -- 172.21.15.28:0/2155441320 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55a59c8c4f70 con 0x55a59cb24500 2024-05-12T22:18:33.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.085+0000 7f72a1600640 1 -- 172.21.15.28:0/2155441320 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f72ac003e50 con 0x55a59cb24500 2024-05-12T22:18:33.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.085+0000 7f72a1600640 1 -- 172.21.15.28:0/2155441320 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f72ac018020 con 0x55a59cb24500 2024-05-12T22:18:33.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.085+0000 7f72bd3cf740 10 client.27292 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:18:33.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.085+0000 7f72bd3cf740 10 client.27292 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:18:33.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.085+0000 7f72bd3cf740 20 client.27292 populate_metadata read hostname 'smithi028' 2024-05-12T22:18:33.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.085+0000 7f72bd3cf740 1 --2- 172.21.15.28:0/2155441320 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55a59cb3f570 0x55a59cb41960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:33.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.086+0000 7f72bd3cf740 4 client.27292 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-12T22:18:33.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.086+0000 7f72bd3cf740 1 -- 172.21.15.28:0/2155441320 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a59ca9c3a0 con 0x55a59cb3f570 2024-05-12T22:18:33.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.086+0000 7f72abe00640 1 --2- 172.21.15.28:0/2155441320 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55a59cb3f570 0x55a59cb41960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:33.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.087+0000 7f72abe00640 1 --2- 172.21.15.28:0/2155441320 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55a59cb3f570 0x55a59cb41960 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bb6fa818514dee09 server_cookie=29b4a7ff8030fc6f in_seq=0 out_seq=0 2024-05-12T22:18:33.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.087+0000 7f72a1600640 10 client.27292 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:18:33.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.087+0000 7f72a1600640 1 -- 172.21.15.28:0/2155441320 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a59ca9c3a0 con 0x55a59cb3f570 2024-05-12T22:18:33.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.087+0000 7f72a1600640 10 client.27292 handle_command_reply: tid=0 2024-05-12T22:18:33.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.087+0000 7f72bd3cf740 1 client.27292 shutdown 2024-05-12T22:18:33.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.088+0000 7f72bd3cf740 1 -- 172.21.15.28:0/2155441320 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a59cb24500 msgr2=0x55a59cb2c3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:18:33.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.088+0000 7f72bd3cf740 1 --2- 172.21.15.28:0/2155441320 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a59cb24500 0x55a59cb2c3d0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f72ac003d80 tx=0x7f72ac004190 comp rx=0 tx=0).stop 2024-05-12T22:18:33.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.088+0000 7f72bd3cf740 1 -- 172.21.15.28:0/2155441320 shutdown_connections 2024-05-12T22:18:33.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.088+0000 7f72bd3cf740 1 --2- 172.21.15.28:0/2155441320 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55a59cb3f570 0x55a59cb41960 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:33.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.088+0000 7f72bd3cf740 1 --2- 172.21.15.28:0/2155441320 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a59cb2f520 0x55a59cb2ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:33.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.088+0000 7f72bd3cf740 1 --2- 172.21.15.28:0/2155441320 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a59cb24eb0 0x55a59cb2c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:33.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.088+0000 7f72bd3cf740 1 --2- 172.21.15.28:0/2155441320 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a59cb24500 0x55a59cb2c3d0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:33.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.088+0000 7f72bd3cf740 1 -- 172.21.15.28:0/2155441320 >> 172.21.15.28:0/2155441320 conn(0x55a59cb205b0 msgr2=0x55a59cb26630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:18:33.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.088+0000 7f72a1600640 0 client.27292 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:18:33.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.088+0000 7f72bd3cf740 1 -- 172.21.15.28:0/2155441320 shutdown_connections 2024-05-12T22:18:33.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.088+0000 7f72bd3cf740 1 -- 172.21.15.28:0/2155441320 wait complete. 2024-05-12T22:18:33.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.088+0000 7f72bd3cf740 20 client.27292 trim_cache size 0 max 16384 2024-05-12T22:18:33.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.108+0000 7f72bd3cf740 1 Processor -- start 2024-05-12T22:18:33.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.108+0000 7f72bd3cf740 1 -- start start 2024-05-12T22:18:33.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.108+0000 7f72bd3cf740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a59cb24500 0x55a59cb3b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:33.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.108+0000 7f72bd3cf740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a59cb24eb0 0x55a59cb2bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:33.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.108+0000 7f72bd3cf740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a59cb2f520 0x55a59cb2caa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:33.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.108+0000 7f72bd3cf740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f72b41b6280 con 0x55a59cb24500 2024-05-12T22:18:33.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.108+0000 7f72bd3cf740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f72b41b3b50 con 0x55a59cb24eb0 2024-05-12T22:18:33.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.108+0000 7f72bd3cf740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f72b41b19c0 con 0x55a59cb2f520 2024-05-12T22:18:33.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.108+0000 7f72b3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a59cb24500 0x55a59cb3b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:33.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.108+0000 7f72b3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a59cb24500 0x55a59cb3b3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49014/0 (socket says 172.21.15.28:49014) 2024-05-12T22:18:33.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.108+0000 7f72b3e00640 1 -- 172.21.15.28:0/1007021341 learned_addr learned my addr 172.21.15.28:0/1007021341 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:18:33.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.108+0000 7f72abe00640 1 --2- 172.21.15.28:0/1007021341 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a59cb24eb0 0x55a59cb2bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:33.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.108+0000 7f72b3e00640 1 --2- 172.21.15.28:0/1007021341 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a59cb2f520 0x55a59cb2caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:33.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.109+0000 7f72b3e00640 1 -- 172.21.15.28:0/1007021341 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a59cb2f520 msgr2=0x55a59cb2caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:18:33.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.109+0000 7f72b3e00640 1 --2- 172.21.15.28:0/1007021341 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a59cb2f520 0x55a59cb2caa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:33.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.109+0000 7f72b3e00640 1 -- 172.21.15.28:0/1007021341 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a59cb24eb0 msgr2=0x55a59cb2bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:18:33.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.109+0000 7f72b3e00640 1 --2- 172.21.15.28:0/1007021341 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a59cb24eb0 0x55a59cb2bcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:33.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.109+0000 7f72b3e00640 1 -- 172.21.15.28:0/1007021341 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a59c75c170 con 0x55a59cb24500 2024-05-12T22:18:33.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.109+0000 7f72b3e00640 1 --2- 172.21.15.28:0/1007021341 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a59cb24500 0x55a59cb3b3f0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f72ac009f50 tx=0x7f72ac014040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=212ae0d6877366d3 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:18:33.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.109+0000 7f72a3e00640 1 -- 172.21.15.28:0/1007021341 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72ac01e040 con 0x55a59cb24500 2024-05-12T22:18:33.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.109+0000 7f72a3e00640 1 -- 172.21.15.28:0/1007021341 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f72ac03c070 con 0x55a59cb24500 2024-05-12T22:18:33.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.109+0000 7f72a3e00640 1 -- 172.21.15.28:0/1007021341 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72ac02e0d0 con 0x55a59cb24500 2024-05-12T22:18:33.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.110+0000 7f72bd3cf740 1 -- 172.21.15.28:0/1007021341 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a59cb24500 msgr2=0x55a59cb3b3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:18:33.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.110+0000 7f72bd3cf740 1 --2- 172.21.15.28:0/1007021341 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a59cb24500 0x55a59cb3b3f0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f72ac009f50 tx=0x7f72ac014040 comp rx=0 tx=0).stop 2024-05-12T22:18:33.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.110+0000 7f72bd3cf740 1 -- 172.21.15.28:0/1007021341 shutdown_connections 2024-05-12T22:18:33.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.110+0000 7f72bd3cf740 1 --2- 172.21.15.28:0/1007021341 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a59cb2f520 0x55a59cb2caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:33.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.110+0000 7f72bd3cf740 1 --2- 172.21.15.28:0/1007021341 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a59cb24eb0 0x55a59cb2bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:33.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.110+0000 7f72bd3cf740 1 --2- 172.21.15.28:0/1007021341 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a59cb24500 0x55a59cb3b3f0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:33.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.110+0000 7f72bd3cf740 1 -- 172.21.15.28:0/1007021341 >> 172.21.15.28:0/1007021341 conn(0x55a59cb205b0 msgr2=0x55a59cb2c390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:18:33.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.110+0000 7f72bd3cf740 1 -- 172.21.15.28:0/1007021341 shutdown_connections 2024-05-12T22:18:33.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.110+0000 7f72bd3cf740 1 -- 172.21.15.28:0/1007021341 wait complete. 2024-05-12T22:18:33.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.110+0000 7f72bd3cf740 1 -- start start 2024-05-12T22:18:33.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.111+0000 7f72bd3cf740 1 Processor -- start 2024-05-12T22:18:33.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.111+0000 7f72bd3cf740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a59cb24500 0x55a59cb28f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:33.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.111+0000 7f72bd3cf740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a59cb24eb0 0x55a59cb294b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:33.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.111+0000 7f72bd3cf740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a59cb2f520 0x55a59cb299f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:33.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.111+0000 7f72bd3cf740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f72b41b2010 con 0x55a59cb24eb0 2024-05-12T22:18:33.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.111+0000 7f72bd3cf740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f72b41b3480 con 0x55a59cb24500 2024-05-12T22:18:33.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.111+0000 7f72bd3cf740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f72b41b26e0 con 0x55a59cb2f520 2024-05-12T22:18:33.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.111+0000 7f72bd3cf740 10 client.0 _ll_register_callbacks cb 0x7f72a40d5400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:18:33.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.112+0000 7f72abe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a59cb24eb0 0x55a59cb294b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:33.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.112+0000 7f72b3e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a59cb24500 0x55a59cb28f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:33.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.112+0000 7f72b3e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a59cb24500 0x55a59cb28f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:46906/0 (socket says 172.21.15.28:46906) 2024-05-12T22:18:33.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.112+0000 7f72b3e00640 1 -- 172.21.15.28:0/526358794 learned_addr learned my addr 172.21.15.28:0/526358794 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:18:33.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.112+0000 7f72abe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a59cb24eb0 0x55a59cb294b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49028/0 (socket says 172.21.15.28:49028) 2024-05-12T22:18:33.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.112+0000 7f72b3e00640 1 --2- 172.21.15.28:0/526358794 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a59cb2f520 0x55a59cb299f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:33.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.112+0000 7f72abe00640 1 -- 172.21.15.28:0/526358794 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a59cb2f520 msgr2=0x55a59cb299f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:18:33.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.112+0000 7f72abe00640 1 --2- 172.21.15.28:0/526358794 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a59cb2f520 0x55a59cb299f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:33.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.112+0000 7f72abe00640 1 -- 172.21.15.28:0/526358794 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a59cb24500 msgr2=0x55a59cb28f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:18:33.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.112+0000 7f72abe00640 1 --2- 172.21.15.28:0/526358794 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a59cb24500 0x55a59cb28f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:33.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.112+0000 7f72abe00640 1 -- 172.21.15.28:0/526358794 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a59cba4960 con 0x55a59cb24eb0 2024-05-12T22:18:33.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.112+0000 7f72abe00640 1 --2- 172.21.15.28:0/526358794 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a59cb24eb0 0x55a59cb294b0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f729c009830 tx=0x7f729c030ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7376db143278901c server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:18:33.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.112+0000 7f72a1600640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-12T22:18:33.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.113+0000 7f72a1600640 1 -- 172.21.15.28:0/526358794 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f729c004450 con 0x55a59cb24eb0 2024-05-12T22:18:33.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.113+0000 7f72a1600640 1 -- 172.21.15.28:0/526358794 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8513+0+0 (secure 0 0 0) 0x7f729c0303a0 con 0x55a59cb24eb0 2024-05-12T22:18:33.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.113+0000 7f72bd3cf740 1 -- 172.21.15.28:0/526358794 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f72ac03c070 con 0x55a59cb24eb0 2024-05-12T22:18:33.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.113+0000 7f72a1600640 1 -- 172.21.15.28:0/526358794 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f729c056050 con 0x55a59cb24eb0 2024-05-12T22:18:33.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.113+0000 7f72bd3cf740 10 client.17214 fetch_fsmap learned FSMap version 26 2024-05-12T22:18:33.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.113+0000 7f72bd3cf740 1 -- 172.21.15.28:0/526358794 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55a59cba5ae0 con 0x55a59cb24eb0 2024-05-12T22:18:33.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.113+0000 7f72a1600640 1 -- 172.21.15.28:0/526358794 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f72ac03c070 con 0x55a59cb24eb0 2024-05-12T22:18:33.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.114+0000 7f72a1600640 1 -- 172.21.15.28:0/526358794 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f729c047410 con 0x55a59cb24eb0 2024-05-12T22:18:33.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.114+0000 7f72bd3cf740 10 client.17214 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:18:33.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.114+0000 7f72bd3cf740 10 client.17214 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:18:33.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.114+0000 7f72bd3cf740 20 client.17214 populate_metadata read hostname 'smithi028' 2024-05-12T22:18:33.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.114+0000 7f72bd3cf740 1 --2- 172.21.15.28:0/526358794 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55a59cba6230 0x55a59cba8650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:33.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.114+0000 7f72bd3cf740 4 client.17214 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-12T22:18:33.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.114+0000 7f72bd3cf740 1 -- 172.21.15.28:0/526358794 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55a59ca95ef0 con 0x55a59cba6230 2024-05-12T22:18:33.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.114+0000 7f72b3e00640 1 --2- 172.21.15.28:0/526358794 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55a59cba6230 0x55a59cba8650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:33.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.115+0000 7f72b3e00640 1 --2- 172.21.15.28:0/526358794 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55a59cba6230 0x55a59cba8650 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3596bb9c1f93c95c server_cookie=e2e194ff84778b37 in_seq=0 out_seq=0 2024-05-12T22:18:33.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.115+0000 7f72a1600640 10 client.17214 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:18:33.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.116+0000 7f72a1600640 1 -- 172.21.15.28:0/526358794 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55a59ca95ef0 con 0x55a59cba6230 2024-05-12T22:18:33.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.116+0000 7f72a1600640 10 client.17214 handle_command_reply: tid=0 2024-05-12T22:18:33.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.116+0000 7f72bd3cf740 1 client.17214 shutdown 2024-05-12T22:18:33.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.116+0000 7f72bd3cf740 1 -- 172.21.15.28:0/526358794 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a59cb24eb0 msgr2=0x55a59cb294b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:18:33.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.116+0000 7f72bd3cf740 1 --2- 172.21.15.28:0/526358794 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a59cb24eb0 0x55a59cb294b0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f729c009830 tx=0x7f729c030ea0 comp rx=0 tx=0).stop 2024-05-12T22:18:33.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.116+0000 7f72bd3cf740 1 -- 172.21.15.28:0/526358794 shutdown_connections 2024-05-12T22:18:33.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.116+0000 7f72bd3cf740 1 --2- 172.21.15.28:0/526358794 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55a59cba6230 0x55a59cba8650 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:33.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.116+0000 7f72bd3cf740 1 --2- 172.21.15.28:0/526358794 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a59cb2f520 0x55a59cb299f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:33.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.116+0000 7f72bd3cf740 1 --2- 172.21.15.28:0/526358794 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a59cb24eb0 0x55a59cb294b0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:33.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.116+0000 7f72bd3cf740 1 --2- 172.21.15.28:0/526358794 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a59cb24500 0x55a59cb28f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:33.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.116+0000 7f72bd3cf740 1 -- 172.21.15.28:0/526358794 >> 172.21.15.28:0/526358794 conn(0x55a59cb205b0 msgr2=0x55a59cb2cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:18:33.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.116+0000 7f72a1600640 0 client.17214 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:18:33.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.116+0000 7f72bd3cf740 1 -- 172.21.15.28:0/526358794 shutdown_connections 2024-05-12T22:18:33.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.116+0000 7f72bd3cf740 1 -- 172.21.15.28:0/526358794 wait complete. 2024-05-12T22:18:33.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.117+0000 7f72bd3cf740 20 client.17214 trim_cache size 0 max 16384 2024-05-12T22:18:33.124 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-05-12T22:18:33.125 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-05-12T22:18:33.125 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "3c678371-dd8e-4d00-a447-c6edd37cc533", 2024-05-12T22:18:33.125 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-05-12T22:18:33.125 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-05-12T22:18:33.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.117+0000 7f72b9000640 1 -- 172.21.15.28:0/2822894466 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f72980a2d00 msgr2=0x7f72980a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:18:33.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.117+0000 7f72b9000640 1 --2- 172.21.15.28:0/2822894466 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f72980a2d00 0x7f72980a51c0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x55a59cb25ac0 tx=0x7f72a40d5530 comp rx=0 tx=0).stop 2024-05-12T22:18:33.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.117+0000 7f72b9000640 1 -- 172.21.15.28:0/2822894466 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72b4107dd0 msgr2=0x7f72b419e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:18:33.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.117+0000 7f72b9000640 1 --2- 172.21.15.28:0/2822894466 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72b4107dd0 0x7f72b419e500 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f72a400e9c0 tx=0x7f72a400ee80 comp rx=0 tx=0).stop 2024-05-12T22:18:33.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.117+0000 7f72b9000640 1 -- 172.21.15.28:0/2822894466 shutdown_connections 2024-05-12T22:18:33.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.117+0000 7f72b9000640 1 --2- 172.21.15.28:0/2822894466 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f72980a2d00 0x7f72980a51c0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:33.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.117+0000 7f72b9000640 1 --2- 172.21.15.28:0/2822894466 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f72b410ddf0 0x7f72b41a2dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:33.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.117+0000 7f72b9000640 1 --2- 172.21.15.28:0/2822894466 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f72b4108770 0x7f72b419ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:33.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.117+0000 7f72b9000640 1 --2- 172.21.15.28:0/2822894466 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72b4107dd0 0x7f72b419e500 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:33.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.117+0000 7f72b9000640 1 -- 172.21.15.28:0/2822894466 >> 172.21.15.28:0/2822894466 conn(0x7f72b4077650 msgr2=0x7f72b40785f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:18:33.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.118+0000 7f72b9000640 1 -- 172.21.15.28:0/2822894466 shutdown_connections 2024-05-12T22:18:33.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.118+0000 7f72b9000640 1 -- 172.21.15.28:0/2822894466 wait complete. 2024-05-12T22:18:33.140 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-12T22:18:33.140 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-12T22:18:33.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.378+0000 7fe901a00640 1 -- 172.21.15.28:0/2865369679 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe8fc103210 msgr2=0x7fe8fc10b6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:18:33.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.378+0000 7fe901a00640 1 --2- 172.21.15.28:0/2865369679 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe8fc103210 0x7fe8fc10b6f0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fe8e4009900 tx=0x7fe8e402a480 comp rx=0 tx=0).stop 2024-05-12T22:18:33.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.379+0000 7fe901a00640 1 -- 172.21.15.28:0/2865369679 shutdown_connections 2024-05-12T22:18:33.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.379+0000 7fe901a00640 1 --2- 172.21.15.28:0/2865369679 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8fc10bcc0 0x7fe8fc10e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:33.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.379+0000 7fe901a00640 1 --2- 172.21.15.28:0/2865369679 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe8fc103210 0x7fe8fc10b6f0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:33.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.379+0000 7fe901a00640 1 --2- 172.21.15.28:0/2865369679 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe8fc1028f0 0x7fe8fc102cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:33.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.379+0000 7fe901a00640 1 -- 172.21.15.28:0/2865369679 >> 172.21.15.28:0/2865369679 conn(0x7fe8fc0fc0b0 msgr2=0x7fe8fc0fe4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:18:33.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.379+0000 7fe901a00640 1 -- 172.21.15.28:0/2865369679 shutdown_connections 2024-05-12T22:18:33.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.379+0000 7fe901a00640 1 -- 172.21.15.28:0/2865369679 wait complete. 2024-05-12T22:18:33.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.380+0000 7fe901a00640 1 Processor -- start 2024-05-12T22:18:33.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.380+0000 7fe901a00640 1 -- start start 2024-05-12T22:18:33.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.381+0000 7fe901a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe8fc1028f0 0x7fe8fc19a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:33.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.381+0000 7fe901a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe8fc103210 0x7fe8fc19a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:33.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.381+0000 7fe901a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8fc10bcc0 0x7fe8fc19eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:33.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.381+0000 7fe901a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe8fc111af0 con 0x7fe8fc10bcc0 2024-05-12T22:18:33.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.381+0000 7fe901a00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fe8fc111970 con 0x7fe8fc1028f0 2024-05-12T22:18:33.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.381+0000 7fe901a00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe8fc111c70 con 0x7fe8fc103210 2024-05-12T22:18:33.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.381+0000 7fe8f3e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe8fc103210 0x7fe8fc19a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:33.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.381+0000 7fe8fbe00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe8fc1028f0 0x7fe8fc19a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:33.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.381+0000 7fe8f3e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe8fc103210 0x7fe8fc19a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:56474/0 (socket says 172.21.15.28:56474) 2024-05-12T22:18:33.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.381+0000 7fe8fbe00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe8fc1028f0 0x7fe8fc19a1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:46924/0 (socket says 172.21.15.28:46924) 2024-05-12T22:18:33.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.381+0000 7fe8f3e00640 1 -- 172.21.15.28:0/4111046418 learned_addr learned my addr 172.21.15.28:0/4111046418 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:18:33.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.381+0000 7fe901000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8fc10bcc0 0x7fe8fc19eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:33.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.381+0000 7fe8f3e00640 1 -- 172.21.15.28:0/4111046418 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe8fc1028f0 msgr2=0x7fe8fc19a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:18:33.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.382+0000 7fe8f3e00640 1 --2- 172.21.15.28:0/4111046418 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe8fc1028f0 0x7fe8fc19a1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:33.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.382+0000 7fe8f3e00640 1 -- 172.21.15.28:0/4111046418 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8fc10bcc0 msgr2=0x7fe8fc19eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:18:33.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.382+0000 7fe8f3e00640 1 --2- 172.21.15.28:0/4111046418 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8fc10bcc0 0x7fe8fc19eab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:33.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.382+0000 7fe8f3e00640 1 -- 172.21.15.28:0/4111046418 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8fc19f1e0 con 0x7fe8fc103210 2024-05-12T22:18:33.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.382+0000 7fe8fbe00640 1 --2- 172.21.15.28:0/4111046418 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe8fc1028f0 0x7fe8fc19a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:18:33.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.382+0000 7fe8f3e00640 1 --2- 172.21.15.28:0/4111046418 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe8fc103210 0x7fe8fc19a730 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fe8e4004770 tx=0x7fe8e4003c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:18:33.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.382+0000 7fe8f9600640 1 -- 172.21.15.28:0/4111046418 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8e403f070 con 0x7fe8fc103210 2024-05-12T22:18:33.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.382+0000 7fe8f9600640 1 -- 172.21.15.28:0/4111046418 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe8e4031ca0 con 0x7fe8fc103210 2024-05-12T22:18:33.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.382+0000 7fe8f9600640 1 -- 172.21.15.28:0/4111046418 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8e4035030 con 0x7fe8fc103210 2024-05-12T22:18:33.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.382+0000 7fe901a00640 1 -- 172.21.15.28:0/4111046418 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8fc19f470 con 0x7fe8fc103210 2024-05-12T22:18:33.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.383+0000 7fe901a00640 1 -- 172.21.15.28:0/4111046418 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8fc1a6d60 con 0x7fe8fc103210 2024-05-12T22:18:33.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.387+0000 7fe8f9600640 1 -- 172.21.15.28:0/4111046418 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7fe8e4042070 con 0x7fe8fc103210 2024-05-12T22:18:33.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.387+0000 7fe905fd8740 1 Processor -- start 2024-05-12T22:18:33.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.387+0000 7fe905fd8740 1 -- start start 2024-05-12T22:18:33.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.387+0000 7fe905fd8740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5590042a8dd0 0x5590042a91b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:33.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.388+0000 7fe905fd8740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5590042a96f0 0x5590042b3880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:33.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.388+0000 7fe905fd8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5590042b3dc0 0x5590042b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:33.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.388+0000 7fe905fd8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5590042712a0 con 0x5590042b3dc0 2024-05-12T22:18:33.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.388+0000 7fe905fd8740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x55900420db90 con 0x5590042a96f0 2024-05-12T22:18:33.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.388+0000 7fe905fd8740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5590041b55b0 con 0x5590042a8dd0 2024-05-12T22:18:33.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.388+0000 7fe901000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5590042a8dd0 0x5590042a91b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:33.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.388+0000 7fe8fbe00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5590042a96f0 0x5590042b3880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:33.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.388+0000 7fe901000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5590042a8dd0 0x5590042a91b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:56482/0 (socket says 172.21.15.28:56482) 2024-05-12T22:18:33.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.388+0000 7fe901000640 1 -- 172.21.15.28:0/4091610114 learned_addr learned my addr 172.21.15.28:0/4091610114 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:18:33.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.388+0000 7fe8f3e00640 1 --2- 172.21.15.28:0/4091610114 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5590042b3dc0 0x5590042b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:33.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.389+0000 7fe901000640 1 -- 172.21.15.28:0/4091610114 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5590042a96f0 msgr2=0x5590042b3880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:18:33.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.389+0000 7fe901000640 1 --2- 172.21.15.28:0/4091610114 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5590042a96f0 0x5590042b3880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:33.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.389+0000 7fe901000640 1 -- 172.21.15.28:0/4091610114 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5590042b3dc0 msgr2=0x5590042b61b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:18:33.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.389+0000 7fe901000640 1 --2- 172.21.15.28:0/4091610114 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5590042b3dc0 0x5590042b61b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:33.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.389+0000 7fe901000640 1 -- 172.21.15.28:0/4091610114 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559003feb7d0 con 0x5590042a8dd0 2024-05-12T22:18:33.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.389+0000 7fe8fbe00640 1 --2- 172.21.15.28:0/4091610114 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5590042a96f0 0x5590042b3880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:18:33.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.389+0000 7fe8f3e00640 1 --2- 172.21.15.28:0/4091610114 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5590042b3dc0 0x5590042b61b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:18:33.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.389+0000 7fe8f9600640 1 --2- 172.21.15.28:0/4111046418 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fe8e00a2d80 0x7fe8e00a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:33.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.389+0000 7fe8f9600640 1 -- 172.21.15.28:0/4111046418 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8513+0+0 (secure 0 0 0) 0x7fe8e40efa40 con 0x7fe8fc103210 2024-05-12T22:18:33.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.390+0000 7fe901000640 1 --2- 172.21.15.28:0/4091610114 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5590042a8dd0 0x5590042a91b0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fe8f4009f70 tx=0x7fe8f4010800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d30943d4be670388 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:18:33.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.390+0000 7fe8fbe00640 1 --2- 172.21.15.28:0/4111046418 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fe8e00a2d80 0x7fe8e00a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:33.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.390+0000 7fe8f1600640 1 -- 172.21.15.28:0/4091610114 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8f4016070 con 0x5590042a8dd0 2024-05-12T22:18:33.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.390+0000 7fe8f1600640 1 -- 172.21.15.28:0/4091610114 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe8f4004d20 con 0x5590042a8dd0 2024-05-12T22:18:33.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.391+0000 7fe8f1600640 1 -- 172.21.15.28:0/4091610114 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8f4012910 con 0x5590042a8dd0 2024-05-12T22:18:33.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.391+0000 7fe905fd8740 1 -- 172.21.15.28:0/4091610114 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5590042a8dd0 msgr2=0x5590042a91b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:18:33.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.391+0000 7fe8fbe00640 1 --2- 172.21.15.28:0/4111046418 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fe8e00a2d80 0x7fe8e00a5240 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x5590042aa390 tx=0x7fe8ec005f90 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:18:33.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.391+0000 7fe905fd8740 1 --2- 172.21.15.28:0/4091610114 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5590042a8dd0 0x5590042a91b0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fe8f4009f70 tx=0x7fe8f4010800 comp rx=0 tx=0).stop 2024-05-12T22:18:33.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.391+0000 7fe905fd8740 1 -- 172.21.15.28:0/4091610114 shutdown_connections 2024-05-12T22:18:33.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.391+0000 7fe905fd8740 1 --2- 172.21.15.28:0/4091610114 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5590042b3dc0 0x5590042b61b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:33.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.391+0000 7fe905fd8740 1 --2- 172.21.15.28:0/4091610114 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5590042a96f0 0x5590042b3880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:33.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.391+0000 7fe905fd8740 1 --2- 172.21.15.28:0/4091610114 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5590042a8dd0 0x5590042a91b0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:33.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.391+0000 7fe905fd8740 1 -- 172.21.15.28:0/4091610114 >> 172.21.15.28:0/4091610114 conn(0x5590042a50f0 msgr2=0x5590042a54d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:18:33.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.391+0000 7fe905fd8740 1 -- 172.21.15.28:0/4091610114 shutdown_connections 2024-05-12T22:18:33.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.392+0000 7fe905fd8740 1 -- 172.21.15.28:0/4091610114 wait complete. 2024-05-12T22:18:33.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.392+0000 7fe905fd8740 1 -- start start 2024-05-12T22:18:33.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.392+0000 7fe905fd8740 1 Processor -- start 2024-05-12T22:18:33.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.393+0000 7fe905fd8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5590042a8dd0 0x5590042b36e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:33.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.393+0000 7fe905fd8740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5590042a96f0 0x5590042b09d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:33.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.393+0000 7fe905fd8740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5590042b3dc0 0x5590042b0f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:33.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.393+0000 7fe905fd8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5590041c89f0 con 0x5590042a8dd0 2024-05-12T22:18:33.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.393+0000 7fe905fd8740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x559004209b50 con 0x5590042b3dc0 2024-05-12T22:18:33.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.393+0000 7fe905fd8740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5590042b6c50 con 0x5590042a96f0 2024-05-12T22:18:33.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.393+0000 7fe905fd8740 10 client.0 _ll_register_callbacks cb 0x559003f20be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:18:33.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.393+0000 7fe8f3e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5590042a96f0 0x5590042b09d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:33.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.393+0000 7fe8fbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5590042a8dd0 0x5590042b36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:33.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.393+0000 7fe8f3e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5590042a96f0 0x5590042b09d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:56488/0 (socket says 172.21.15.28:56488) 2024-05-12T22:18:33.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.393+0000 7fe8f3e00640 1 -- 172.21.15.28:0/3141780067 learned_addr learned my addr 172.21.15.28:0/3141780067 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:18:33.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.394+0000 7fe901000640 1 --2- 172.21.15.28:0/3141780067 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5590042b3dc0 0x5590042b0f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:33.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.394+0000 7fe8f3e00640 1 -- 172.21.15.28:0/3141780067 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5590042b3dc0 msgr2=0x5590042b0f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:18:33.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.394+0000 7fe8f3e00640 1 --2- 172.21.15.28:0/3141780067 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5590042b3dc0 0x5590042b0f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:33.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.394+0000 7fe8f3e00640 1 -- 172.21.15.28:0/3141780067 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5590042a8dd0 msgr2=0x5590042b36e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:18:33.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.394+0000 7fe8f3e00640 1 --2- 172.21.15.28:0/3141780067 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5590042a8dd0 0x5590042b36e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:33.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.394+0000 7fe8f3e00640 1 -- 172.21.15.28:0/3141780067 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559003f213f0 con 0x5590042a96f0 2024-05-12T22:18:33.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.394+0000 7fe8fbe00640 1 --2- 172.21.15.28:0/3141780067 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5590042a8dd0 0x5590042b36e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:18:33.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.394+0000 7fe8f3e00640 1 --2- 172.21.15.28:0/3141780067 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5590042a96f0 0x5590042b09d0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fe8e4031550 tx=0x7fe8e403b460 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1e08535ff1129a35 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:18:33.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.394+0000 7fe8ea000640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-12T22:18:33.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.395+0000 7fe8ea000640 1 -- 172.21.15.28:0/3141780067 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8e4049050 con 0x5590042a96f0 2024-05-12T22:18:33.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.395+0000 7fe8ea000640 1 -- 172.21.15.28:0/3141780067 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8513+0+0 (secure 0 0 0) 0x7fe8e403b4f0 con 0x5590042a96f0 2024-05-12T22:18:33.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.395+0000 7fe905fd8740 1 -- 172.21.15.28:0/3141780067 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe8f4004d20 con 0x5590042a96f0 2024-05-12T22:18:33.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.395+0000 7fe8ea000640 1 -- 172.21.15.28:0/3141780067 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fe8e4005d50 con 0x5590042a96f0 2024-05-12T22:18:33.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.395+0000 7fe905fd8740 10 client.27310 fetch_fsmap learned FSMap version 26 2024-05-12T22:18:33.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.395+0000 7fe905fd8740 1 -- 172.21.15.28:0/3141780067 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x559003ee10f0 con 0x5590042a96f0 2024-05-12T22:18:33.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.395+0000 7fe8ea000640 1 -- 172.21.15.28:0/3141780067 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe8f4004d20 con 0x5590042a96f0 2024-05-12T22:18:33.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.396+0000 7fe8ea000640 1 -- 172.21.15.28:0/3141780067 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fe8e40ed3f0 con 0x5590042a96f0 2024-05-12T22:18:33.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.396+0000 7fe905fd8740 10 client.27310 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:18:33.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.396+0000 7fe905fd8740 10 client.27310 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:18:33.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.396+0000 7fe905fd8740 20 client.27310 populate_metadata read hostname 'smithi028' 2024-05-12T22:18:33.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.396+0000 7fe905fd8740 1 --2- 172.21.15.28:0/3141780067 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5590042cc0e0 0x5590042ce4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:33.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.396+0000 7fe905fd8740 4 client.27310 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-12T22:18:33.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.396+0000 7fe905fd8740 1 -- 172.21.15.28:0/3141780067 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55900421b2c0 con 0x5590042cc0e0 2024-05-12T22:18:33.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.397+0000 7fe8fbe00640 1 --2- 172.21.15.28:0/3141780067 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5590042cc0e0 0x5590042ce4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:33.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.398+0000 7fe8fbe00640 1 --2- 172.21.15.28:0/3141780067 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5590042cc0e0 0x5590042ce4d0 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f246d524c78f5659 server_cookie=f3586ddd97cbe646 in_seq=0 out_seq=0 2024-05-12T22:18:33.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.398+0000 7fe8ea000640 10 client.27310 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:18:33.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.399+0000 7fe8ea000640 1 -- 172.21.15.28:0/3141780067 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55900421b2c0 con 0x5590042cc0e0 2024-05-12T22:18:33.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.399+0000 7fe8ea000640 10 client.27310 handle_command_reply: tid=0 2024-05-12T22:18:33.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.399+0000 7fe905fd8740 1 client.27310 shutdown 2024-05-12T22:18:33.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.399+0000 7fe905fd8740 1 -- 172.21.15.28:0/3141780067 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5590042a96f0 msgr2=0x5590042b09d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:18:33.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.399+0000 7fe905fd8740 1 --2- 172.21.15.28:0/3141780067 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5590042a96f0 0x5590042b09d0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fe8e4031550 tx=0x7fe8e403b460 comp rx=0 tx=0).stop 2024-05-12T22:18:33.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.400+0000 7fe905fd8740 1 -- 172.21.15.28:0/3141780067 shutdown_connections 2024-05-12T22:18:33.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.400+0000 7fe905fd8740 1 --2- 172.21.15.28:0/3141780067 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5590042cc0e0 0x5590042ce4d0 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:33.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.400+0000 7fe905fd8740 1 --2- 172.21.15.28:0/3141780067 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5590042b3dc0 0x5590042b0f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:33.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.400+0000 7fe905fd8740 1 --2- 172.21.15.28:0/3141780067 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5590042a96f0 0x5590042b09d0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:33.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.400+0000 7fe905fd8740 1 --2- 172.21.15.28:0/3141780067 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5590042a8dd0 0x5590042b36e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:33.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.400+0000 7fe8ea000640 0 client.27310 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:18:33.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.400+0000 7fe905fd8740 1 -- 172.21.15.28:0/3141780067 >> 172.21.15.28:0/3141780067 conn(0x5590042a50f0 msgr2=0x5590042aaef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:18:33.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.400+0000 7fe905fd8740 1 -- 172.21.15.28:0/3141780067 shutdown_connections 2024-05-12T22:18:33.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.400+0000 7fe905fd8740 1 -- 172.21.15.28:0/3141780067 wait complete. 2024-05-12T22:18:33.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.400+0000 7fe905fd8740 20 client.27310 trim_cache size 0 max 16384 2024-05-12T22:18:33.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.425+0000 7fe905fd8740 1 Processor -- start 2024-05-12T22:18:33.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.425+0000 7fe905fd8740 1 -- start start 2024-05-12T22:18:33.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.426+0000 7fe905fd8740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5590042a8dd0 0x5590042b3210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:33.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.426+0000 7fe905fd8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5590042a96f0 0x5590042d3d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:33.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.426+0000 7fe905fd8740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5590042b3dc0 0x5590042d42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:33.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.426+0000 7fe905fd8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe8fc1ada40 con 0x5590042a96f0 2024-05-12T22:18:33.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.426+0000 7fe905fd8740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fe8fc1ab310 con 0x5590042a8dd0 2024-05-12T22:18:33.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.426+0000 7fe905fd8740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe8fc1a91e0 con 0x5590042b3dc0 2024-05-12T22:18:33.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.426+0000 7fe8fbe00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5590042a8dd0 0x5590042b3210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:33.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.426+0000 7fe901000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5590042b3dc0 0x5590042d42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:33.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.426+0000 7fe8fbe00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5590042a8dd0 0x5590042b3210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:46950/0 (socket says 172.21.15.28:46950) 2024-05-12T22:18:33.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.426+0000 7fe8fbe00640 1 -- 172.21.15.28:0/3129717431 learned_addr learned my addr 172.21.15.28:0/3129717431 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:18:33.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.426+0000 7fe8f3e00640 1 --2- 172.21.15.28:0/3129717431 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5590042a96f0 0x5590042d3d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:33.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.427+0000 7fe901000640 1 -- 172.21.15.28:0/3129717431 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5590042a8dd0 msgr2=0x5590042b3210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:18:33.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.427+0000 7fe901000640 1 --2- 172.21.15.28:0/3129717431 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5590042a8dd0 0x5590042b3210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:33.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.427+0000 7fe901000640 1 -- 172.21.15.28:0/3129717431 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5590042a96f0 msgr2=0x5590042d3d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:18:33.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.427+0000 7fe901000640 1 --2- 172.21.15.28:0/3129717431 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5590042a96f0 0x5590042d3d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:33.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.427+0000 7fe901000640 1 -- 172.21.15.28:0/3129717431 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5590042a4f20 con 0x5590042b3dc0 2024-05-12T22:18:33.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.427+0000 7fe8fbe00640 1 --2- 172.21.15.28:0/3129717431 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5590042a8dd0 0x5590042b3210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:18:33.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.427+0000 7fe8f3e00640 1 --2- 172.21.15.28:0/3129717431 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5590042a96f0 0x5590042d3d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:18:33.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.427+0000 7fe901000640 1 --2- 172.21.15.28:0/3129717431 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5590042b3dc0 0x5590042d42a0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fe8f4000f80 tx=0x7fe8f4004a00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ed748690448e5b54 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:18:33.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.427+0000 7fe8f0c00640 1 -- 172.21.15.28:0/3129717431 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8f4016070 con 0x5590042b3dc0 2024-05-12T22:18:33.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.427+0000 7fe8f0c00640 1 -- 172.21.15.28:0/3129717431 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe8f4004d70 con 0x5590042b3dc0 2024-05-12T22:18:33.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.427+0000 7fe8f0c00640 1 -- 172.21.15.28:0/3129717431 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8f40128e0 con 0x5590042b3dc0 2024-05-12T22:18:33.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.428+0000 7fe905fd8740 1 -- 172.21.15.28:0/3129717431 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5590042b3dc0 msgr2=0x5590042d42a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:18:33.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.428+0000 7fe905fd8740 1 --2- 172.21.15.28:0/3129717431 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5590042b3dc0 0x5590042d42a0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fe8f4000f80 tx=0x7fe8f4004a00 comp rx=0 tx=0).stop 2024-05-12T22:18:33.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.428+0000 7fe905fd8740 1 -- 172.21.15.28:0/3129717431 shutdown_connections 2024-05-12T22:18:33.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.428+0000 7fe905fd8740 1 --2- 172.21.15.28:0/3129717431 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5590042b3dc0 0x5590042d42a0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:33.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.428+0000 7fe905fd8740 1 --2- 172.21.15.28:0/3129717431 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5590042a96f0 0x5590042d3d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:33.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.428+0000 7fe905fd8740 1 --2- 172.21.15.28:0/3129717431 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5590042a8dd0 0x5590042b3210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:33.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.428+0000 7fe905fd8740 1 -- 172.21.15.28:0/3129717431 >> 172.21.15.28:0/3129717431 conn(0x5590042a50f0 msgr2=0x5590042aa630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:18:33.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.429+0000 7fe905fd8740 1 -- 172.21.15.28:0/3129717431 shutdown_connections 2024-05-12T22:18:33.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.429+0000 7fe905fd8740 1 -- 172.21.15.28:0/3129717431 wait complete. 2024-05-12T22:18:33.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.429+0000 7fe905fd8740 1 -- start start 2024-05-12T22:18:33.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.429+0000 7fe905fd8740 1 Processor -- start 2024-05-12T22:18:33.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.430+0000 7fe905fd8740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5590042a8dd0 0x5590042ad7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:33.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.430+0000 7fe905fd8740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5590042a96f0 0x5590042adce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:33.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.430+0000 7fe905fd8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5590042b3dc0 0x5590042ae220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:33.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.430+0000 7fe905fd8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe8fc1a9830 con 0x5590042b3dc0 2024-05-12T22:18:33.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.430+0000 7fe905fd8740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fe8fc1aac40 con 0x5590042a96f0 2024-05-12T22:18:33.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.430+0000 7fe905fd8740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe8fc1a9ea0 con 0x5590042a8dd0 2024-05-12T22:18:33.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.430+0000 7fe905fd8740 10 client.0 _ll_register_callbacks cb 0x7fe8e4005b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:18:33.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.430+0000 7fe8fbe00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5590042a8dd0 0x5590042ad7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:33.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.430+0000 7fe8f3e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5590042a96f0 0x5590042adce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:33.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.430+0000 7fe8fbe00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5590042a8dd0 0x5590042ad7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:56514/0 (socket says 172.21.15.28:56514) 2024-05-12T22:18:33.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.430+0000 7fe8fbe00640 1 -- 172.21.15.28:0/3882103657 learned_addr learned my addr 172.21.15.28:0/3882103657 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:18:33.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.431+0000 7fe901000640 1 --2- 172.21.15.28:0/3882103657 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5590042b3dc0 0x5590042ae220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:33.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.431+0000 7fe8f3e00640 1 -- 172.21.15.28:0/3882103657 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5590042a8dd0 msgr2=0x5590042ad7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:18:33.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.431+0000 7fe8f3e00640 1 --2- 172.21.15.28:0/3882103657 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5590042a8dd0 0x5590042ad7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:33.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.431+0000 7fe8f3e00640 1 -- 172.21.15.28:0/3882103657 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5590042b3dc0 msgr2=0x5590042ae220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:18:33.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.431+0000 7fe8f3e00640 1 --2- 172.21.15.28:0/3882103657 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5590042b3dc0 0x5590042ae220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:33.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.431+0000 7fe8f3e00640 1 -- 172.21.15.28:0/3882103657 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559004331700 con 0x5590042a96f0 2024-05-12T22:18:33.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.431+0000 7fe901000640 1 --2- 172.21.15.28:0/3882103657 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5590042b3dc0 0x5590042ae220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-12T22:18:33.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.431+0000 7fe8f3e00640 1 --2- 172.21.15.28:0/3882103657 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5590042a96f0 0x5590042adce0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fe8e40bbf80 tx=0x7fe8e40dc170 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c77cb0a4f872dbcf server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:18:33.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.431+0000 7fe8ea000640 10 client.0 ms_handle_connect on v2:172.21.15.86:3300/0 2024-05-12T22:18:33.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.431+0000 7fe8ea000640 1 -- 172.21.15.28:0/3882103657 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8e4049070 con 0x5590042a96f0 2024-05-12T22:18:33.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.431+0000 7fe8ea000640 1 -- 172.21.15.28:0/3882103657 <== mon.2 v2:172.21.15.86:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8513+0+0 (secure 0 0 0) 0x7fe8e40dabd0 con 0x5590042a96f0 2024-05-12T22:18:33.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.431+0000 7fe905fd8740 1 -- 172.21.15.28:0/3882103657 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe8f4004d70 con 0x5590042a96f0 2024-05-12T22:18:33.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.431+0000 7fe8ea000640 1 -- 172.21.15.28:0/3882103657 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fe8e40c9050 con 0x5590042a96f0 2024-05-12T22:18:33.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.432+0000 7fe905fd8740 10 client.27365 fetch_fsmap learned FSMap version 26 2024-05-12T22:18:33.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.432+0000 7fe905fd8740 1 -- 172.21.15.28:0/3882103657 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5590043329f0 con 0x5590042a96f0 2024-05-12T22:18:33.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.432+0000 7fe8ea000640 1 -- 172.21.15.28:0/3882103657 <== mon.2 v2:172.21.15.86:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe8f4004d70 con 0x5590042a96f0 2024-05-12T22:18:33.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.432+0000 7fe8ea000640 1 -- 172.21.15.28:0/3882103657 <== mon.2 v2:172.21.15.86:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fe8e40b6b60 con 0x5590042a96f0 2024-05-12T22:18:33.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.432+0000 7fe905fd8740 10 client.27365 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:18:33.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.432+0000 7fe905fd8740 10 client.27365 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:18:33.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.432+0000 7fe905fd8740 20 client.27365 populate_metadata read hostname 'smithi028' 2024-05-12T22:18:33.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.432+0000 7fe905fd8740 1 --2- 172.21.15.28:0/3882103657 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x559004333060 0x559004335480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:18:33.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.433+0000 7fe905fd8740 4 client.27365 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-12T22:18:33.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.433+0000 7fe905fd8740 1 -- 172.21.15.28:0/3882103657 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "scrub status"}) -- 0x559004215610 con 0x559004333060 2024-05-12T22:18:33.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.433+0000 7fe8fbe00640 1 --2- 172.21.15.28:0/3882103657 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x559004333060 0x559004335480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:18:33.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.433+0000 7fe8ea000640 10 client.27365 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:18:33.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.433+0000 7fe8fbe00640 1 --2- 172.21.15.28:0/3882103657 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x559004333060 0x559004335480 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f9b0afd0170e1461 server_cookie=14ee2cde6dcf2af0 in_seq=0 out_seq=0 2024-05-12T22:18:33.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.434+0000 7fe8ea000640 1 -- 172.21.15.28:0/3882103657 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+268 (crc 0 0 0) 0x559004215610 con 0x559004333060 2024-05-12T22:18:33.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.434+0000 7fe8ea000640 10 client.27365 handle_command_reply: tid=0 2024-05-12T22:18:33.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.434+0000 7fe905fd8740 1 client.27365 shutdown 2024-05-12T22:18:33.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.434+0000 7fe905fd8740 1 -- 172.21.15.28:0/3882103657 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5590042a96f0 msgr2=0x5590042adce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:18:33.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.434+0000 7fe905fd8740 1 --2- 172.21.15.28:0/3882103657 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5590042a96f0 0x5590042adce0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fe8e40bbf80 tx=0x7fe8e40dc170 comp rx=0 tx=0).stop 2024-05-12T22:18:33.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.434+0000 7fe905fd8740 1 -- 172.21.15.28:0/3882103657 shutdown_connections 2024-05-12T22:18:33.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.434+0000 7fe905fd8740 1 --2- 172.21.15.28:0/3882103657 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x559004333060 0x559004335480 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:33.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.434+0000 7fe905fd8740 1 --2- 172.21.15.28:0/3882103657 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5590042b3dc0 0x5590042ae220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:33.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.434+0000 7fe905fd8740 1 --2- 172.21.15.28:0/3882103657 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5590042a96f0 0x5590042adce0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:33.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.434+0000 7fe905fd8740 1 --2- 172.21.15.28:0/3882103657 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5590042a8dd0 0x5590042ad7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:33.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.434+0000 7fe8ea000640 0 client.27365 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:18:33.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.434+0000 7fe905fd8740 1 -- 172.21.15.28:0/3882103657 >> 172.21.15.28:0/3882103657 conn(0x5590042a50f0 msgr2=0x5590042b0af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:18:33.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.435+0000 7fe905fd8740 1 -- 172.21.15.28:0/3882103657 shutdown_connections 2024-05-12T22:18:33.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.435+0000 7fe905fd8740 1 -- 172.21.15.28:0/3882103657 wait complete. 2024-05-12T22:18:33.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.435+0000 7fe905fd8740 20 client.27365 trim_cache size 0 max 16384 2024-05-12T22:18:33.441 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-05-12T22:18:33.441 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "scrub active (1188 inodes in the stack)", 2024-05-12T22:18:33.441 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-05-12T22:18:33.441 INFO:teuthology.orchestra.run.smithi028.stdout: "3c678371-dd8e-4d00-a447-c6edd37cc533": { 2024-05-12T22:18:33.441 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-05-12T22:18:33.441 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "3c678371-dd8e-4d00-a447-c6edd37cc533", 2024-05-12T22:18:33.441 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-05-12T22:18:33.441 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:18:33.442 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:18:33.442 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-05-12T22:18:33.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.436+0000 7fe901a00640 1 -- 172.21.15.28:0/4111046418 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fe8e00a2d80 msgr2=0x7fe8e00a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:18:33.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.436+0000 7fe901a00640 1 --2- 172.21.15.28:0/4111046418 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fe8e00a2d80 0x7fe8e00a5240 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x5590042aa390 tx=0x7fe8ec005f90 comp rx=0 tx=0).stop 2024-05-12T22:18:33.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.436+0000 7fe901a00640 1 -- 172.21.15.28:0/4111046418 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe8fc103210 msgr2=0x7fe8fc19a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:18:33.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.436+0000 7fe901a00640 1 --2- 172.21.15.28:0/4111046418 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe8fc103210 0x7fe8fc19a730 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fe8e4004770 tx=0x7fe8e4003c10 comp rx=0 tx=0).stop 2024-05-12T22:18:33.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.436+0000 7fe901a00640 1 -- 172.21.15.28:0/4111046418 shutdown_connections 2024-05-12T22:18:33.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.436+0000 7fe901a00640 1 --2- 172.21.15.28:0/4111046418 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fe8e00a2d80 0x7fe8e00a5240 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:33.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.436+0000 7fe901a00640 1 --2- 172.21.15.28:0/4111046418 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8fc10bcc0 0x7fe8fc19eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:33.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.436+0000 7fe901a00640 1 --2- 172.21.15.28:0/4111046418 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe8fc103210 0x7fe8fc19a730 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:33.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.436+0000 7fe901a00640 1 --2- 172.21.15.28:0/4111046418 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe8fc1028f0 0x7fe8fc19a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:18:33.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.436+0000 7fe901a00640 1 -- 172.21.15.28:0/4111046418 >> 172.21.15.28:0/4111046418 conn(0x7fe8fc0fc0b0 msgr2=0x7fe8fc100470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:18:33.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.436+0000 7fe901a00640 1 -- 172.21.15.28:0/4111046418 shutdown_connections 2024-05-12T22:18:33.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:18:33.436+0000 7fe901a00640 1 -- 172.21.15.28:0/4111046418 wait complete. 2024-05-12T22:18:33.456 INFO:tasks.cephfs.filesystem:scrub status for tag:3c678371-dd8e-4d00-a447-c6edd37cc533 - {'path': '/', 'tag': '3c678371-dd8e-4d00-a447-c6edd37cc533', 'options': 'recursive,force'} 2024-05-12T22:18:34.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:18:34 smithi086 ceph-mon[34297]: pgmap v296: 97 pgs: 97 active+clean; 91 MiB data, 725 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:18:34.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:18:34 smithi086 ceph-mon[34297]: scrub queued for path: / 2024-05-12T22:18:34.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:18:34 smithi086 ceph-mon[34297]: scrub summary: idle+waiting paths [/] 2024-05-12T22:18:34.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:18:34 smithi086 ceph-mon[34297]: scrub summary: active paths [/] 2024-05-12T22:18:34.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:18:34 smithi086 ceph-mon[34297]: scrub summary: active paths [/] 2024-05-12T22:18:34.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:18:34 smithi086 ceph-mon[34297]: scrub summary: idle+waiting paths [/] 2024-05-12T22:18:34.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:18:34 smithi086 ceph-mon[34297]: scrub summary: active paths [/] 2024-05-12T22:18:34.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:18:34 smithi086 ceph-mon[34297]: scrub summary: idle+waiting paths [/] 2024-05-12T22:18:34.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:18:34 smithi086 ceph-mon[34297]: scrub summary: active paths [/] 2024-05-12T22:18:34.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:18:34 smithi086 ceph-mon[34297]: scrub summary: idle+waiting paths [/] 2024-05-12T22:18:34.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:18:34 smithi086 ceph-mon[34297]: scrub summary: idle+waiting paths [/] 2024-05-12T22:18:34.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:18:34 smithi028 ceph-mon[29847]: pgmap v296: 97 pgs: 97 active+clean; 91 MiB data, 725 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:18:34.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:18:34 smithi028 ceph-mon[29847]: scrub queued for path: / 2024-05-12T22:18:34.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:18:34 smithi028 ceph-mon[29847]: scrub summary: idle+waiting paths [/] 2024-05-12T22:18:34.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:18:34 smithi028 ceph-mon[29847]: scrub summary: active paths [/] 2024-05-12T22:18:34.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:18:34 smithi028 ceph-mon[29847]: scrub summary: active paths [/] 2024-05-12T22:18:34.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:18:34 smithi028 ceph-mon[29847]: scrub summary: idle+waiting paths [/] 2024-05-12T22:18:34.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:18:34 smithi028 ceph-mon[29847]: scrub summary: active paths [/] 2024-05-12T22:18:34.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:18:34 smithi028 ceph-mon[29847]: scrub summary: idle+waiting paths [/] 2024-05-12T22:18:34.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:18:34 smithi028 ceph-mon[29847]: scrub summary: active paths [/] 2024-05-12T22:18:34.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:18:34 smithi028 ceph-mon[29847]: scrub summary: idle+waiting paths [/] 2024-05-12T22:18:34.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:18:34 smithi028 ceph-mon[29847]: scrub summary: idle+waiting paths [/] 2024-05-12T22:18:34.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:18:34 smithi186 ceph-mon[34785]: pgmap v296: 97 pgs: 97 active+clean; 91 MiB data, 725 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:18:34.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:18:34 smithi186 ceph-mon[34785]: scrub queued for path: / 2024-05-12T22:18:34.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:18:34 smithi186 ceph-mon[34785]: scrub summary: idle+waiting paths [/] 2024-05-12T22:18:34.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:18:34 smithi186 ceph-mon[34785]: scrub summary: active paths [/] 2024-05-12T22:18:34.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:18:34 smithi186 ceph-mon[34785]: scrub summary: active paths [/] 2024-05-12T22:18:34.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:18:34 smithi186 ceph-mon[34785]: scrub summary: idle+waiting paths [/] 2024-05-12T22:18:34.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:18:34 smithi186 ceph-mon[34785]: scrub summary: active paths [/] 2024-05-12T22:18:34.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:18:34 smithi186 ceph-mon[34785]: scrub summary: idle+waiting paths [/] 2024-05-12T22:18:34.681 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:18:34 smithi186 ceph-mon[34785]: scrub summary: active paths [/] 2024-05-12T22:18:34.681 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:18:34 smithi186 ceph-mon[34785]: scrub summary: idle+waiting paths [/] 2024-05-12T22:18:34.681 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:18:34 smithi186 ceph-mon[34785]: scrub summary: idle+waiting paths [/] 2024-05-12T22:18:36.481 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:18:36 smithi186 ceph-mon[34785]: pgmap v297: 97 pgs: 97 active+clean; 91 MiB data, 725 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-05-12T22:18:36.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:18:36 smithi086 ceph-mon[34297]: pgmap v297: 97 pgs: 97 active+clean; 91 MiB data, 725 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-05-12T22:18:36.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:18:36 smithi028 ceph-mon[29847]: pgmap v297: 97 pgs: 97 active+clean; 91 MiB data, 725 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-05-12T22:18:38.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:18:38 smithi086 ceph-mon[34297]: pgmap v298: 97 pgs: 97 active+clean; 91 MiB data, 725 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-05-12T22:18:38.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:18:38 smithi028 ceph-mon[29847]: pgmap v298: 97 pgs: 97 active+clean; 91 MiB data, 725 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-05-12T22:18:38.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:18:38 smithi186 ceph-mon[34785]: pgmap v298: 97 pgs: 97 active+clean; 91 MiB data, 725 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-05-12T22:18:40.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:18:40 smithi086 ceph-mon[34297]: pgmap v299: 97 pgs: 97 active+clean; 91 MiB data, 725 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-05-12T22:18:40.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:18:40 smithi028 ceph-mon[29847]: pgmap v299: 97 pgs: 97 active+clean; 91 MiB data, 725 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-05-12T22:18:40.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:18:40 smithi186 ceph-mon[34785]: pgmap v299: 97 pgs: 97 active+clean; 91 MiB data, 725 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-05-12T22:18:42.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:18:42 smithi086 ceph-mon[34297]: pgmap v300: 97 pgs: 97 active+clean; 91 MiB data, 725 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-05-12T22:18:42.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:18:42 smithi028 ceph-mon[29847]: pgmap v300: 97 pgs: 97 active+clean; 91 MiB data, 725 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-05-12T22:18:42.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:18:42 smithi186 ceph-mon[34785]: pgmap v300: 97 pgs: 97 active+clean; 91 MiB data, 725 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-05-12T22:18:43.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:18:43 smithi086 ceph-mon[34297]: scrub summary: idle 2024-05-12T22:18:43.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:18:43 smithi086 ceph-mon[34297]: scrub summary: idle 2024-05-12T22:18:43.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:18:43 smithi028 ceph-mon[29847]: scrub summary: idle 2024-05-12T22:18:43.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:18:43 smithi028 ceph-mon[29847]: scrub summary: idle 2024-05-12T22:18:43.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:18:43 smithi186 ceph-mon[34785]: scrub summary: idle 2024-05-12T22:18:43.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:18:43 smithi186 ceph-mon[34785]: scrub summary: idle 2024-05-12T22:18:44.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:18:44 smithi086 ceph-mon[34297]: pgmap v301: 97 pgs: 97 active+clean; 91 MiB data, 725 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-05-12T22:18:44.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:18:44 smithi028 ceph-mon[29847]: pgmap v301: 97 pgs: 97 active+clean; 91 MiB data, 725 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-05-12T22:18:44.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:18:44 smithi186 ceph-mon[34785]: pgmap v301: 97 pgs: 97 active+clean; 91 MiB data, 725 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-05-12T22:18:46.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:18:46 smithi086 ceph-mon[34297]: pgmap v302: 97 pgs: 97 active+clean; 91 MiB data, 725 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-05-12T22:18:46.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:18:46 smithi028 ceph-mon[29847]: pgmap v302: 97 pgs: 97 active+clean; 91 MiB data, 725 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-05-12T22:18:46.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:18:46 smithi186 ceph-mon[34785]: pgmap v302: 97 pgs: 97 active+clean; 91 MiB data, 725 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-05-12T22:18:48.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:18:48 smithi086 ceph-mon[34297]: pgmap v303: 97 pgs: 97 active+clean; 91 MiB data, 725 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:18:48.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:18:48 smithi028 ceph-mon[29847]: pgmap v303: 97 pgs: 97 active+clean; 91 MiB data, 725 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:18:48.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:18:48 smithi186 ceph-mon[34785]: pgmap v303: 97 pgs: 97 active+clean; 91 MiB data, 725 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:18:50.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:18:50 smithi028 ceph-mon[29847]: pgmap v304: 97 pgs: 97 active+clean; 91 MiB data, 725 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:18:50.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:18:50 smithi186 ceph-mon[34785]: pgmap v304: 97 pgs: 97 active+clean; 91 MiB data, 725 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:18:50.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:18:50 smithi086 ceph-mon[34297]: pgmap v304: 97 pgs: 97 active+clean; 91 MiB data, 725 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:18:52.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:18:52 smithi028 ceph-mon[29847]: pgmap v305: 97 pgs: 97 active+clean; 91 MiB data, 725 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-12T22:18:52.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:18:52 smithi186 ceph-mon[34785]: pgmap v305: 97 pgs: 97 active+clean; 91 MiB data, 725 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-12T22:18:52.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:18:52 smithi086 ceph-mon[34297]: pgmap v305: 97 pgs: 97 active+clean; 91 MiB data, 725 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-12T22:18:54.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:18:54 smithi028 ceph-mon[29847]: pgmap v306: 97 pgs: 97 active+clean; 91 MiB data, 725 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:18:54.681 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:18:54 smithi186 ceph-mon[34785]: pgmap v306: 97 pgs: 97 active+clean; 91 MiB data, 725 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:18:54.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:18:54 smithi086 ceph-mon[34297]: pgmap v306: 97 pgs: 97 active+clean; 91 MiB data, 725 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:18:56.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:18:56 smithi028 ceph-mon[29847]: pgmap v307: 97 pgs: 97 active+clean; 91 MiB data, 725 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-12T22:18:56.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:18:56 smithi186 ceph-mon[34785]: pgmap v307: 97 pgs: 97 active+clean; 91 MiB data, 725 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-12T22:18:56.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:18:56 smithi086 ceph-mon[34297]: pgmap v307: 97 pgs: 97 active+clean; 91 MiB data, 725 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-12T22:18:58.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:18:58 smithi028 ceph-mon[29847]: pgmap v308: 97 pgs: 97 active+clean; 91 MiB data, 725 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:18:58.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:18:58 smithi186 ceph-mon[34785]: pgmap v308: 97 pgs: 97 active+clean; 91 MiB data, 725 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:18:58.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:18:58 smithi086 ceph-mon[34297]: pgmap v308: 97 pgs: 97 active+clean; 91 MiB data, 725 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:19:00.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:19:00 smithi028 ceph-mon[29847]: pgmap v309: 97 pgs: 97 active+clean; 91 MiB data, 725 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:19:00.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:19:00 smithi186 ceph-mon[34785]: pgmap v309: 97 pgs: 97 active+clean; 91 MiB data, 725 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:19:00.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:19:00 smithi086 ceph-mon[34297]: pgmap v309: 97 pgs: 97 active+clean; 91 MiB data, 725 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:19:02.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:19:02 smithi028 ceph-mon[29847]: pgmap v310: 97 pgs: 97 active+clean; 91 MiB data, 725 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-12T22:19:02.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:19:02 smithi186 ceph-mon[34785]: pgmap v310: 97 pgs: 97 active+clean; 91 MiB data, 725 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-12T22:19:02.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:19:02 smithi086 ceph-mon[34297]: pgmap v310: 97 pgs: 97 active+clean; 91 MiB data, 725 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-12T22:19:03.458 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-12T22:19:03.458 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-12T22:19:03.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.691+0000 7f5c6aa00640 1 -- 172.21.15.28:0/1381810640 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5c64103220 msgr2=0x7f5c6410b700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:03.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.691+0000 7f5c6aa00640 1 --2- 172.21.15.28:0/1381810640 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5c64103220 0x7f5c6410b700 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f5c50009900 tx=0x7f5c5002a480 comp rx=0 tx=0).stop 2024-05-12T22:19:03.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.691+0000 7f5c6aa00640 1 -- 172.21.15.28:0/1381810640 shutdown_connections 2024-05-12T22:19:03.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.691+0000 7f5c6aa00640 1 --2- 172.21.15.28:0/1381810640 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5c6410bd20 0x7f5c6410e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:03.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.691+0000 7f5c6aa00640 1 --2- 172.21.15.28:0/1381810640 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5c64103220 0x7f5c6410b700 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:03.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.691+0000 7f5c6aa00640 1 --2- 172.21.15.28:0/1381810640 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c64102900 0x7f5c64102ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:03.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.691+0000 7f5c6aa00640 1 -- 172.21.15.28:0/1381810640 >> 172.21.15.28:0/1381810640 conn(0x7f5c640fc140 msgr2=0x7f5c640fe560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:19:03.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.691+0000 7f5c6aa00640 1 -- 172.21.15.28:0/1381810640 shutdown_connections 2024-05-12T22:19:03.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.691+0000 7f5c6aa00640 1 -- 172.21.15.28:0/1381810640 wait complete. 2024-05-12T22:19:03.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.692+0000 7f5c6aa00640 1 Processor -- start 2024-05-12T22:19:03.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.693+0000 7f5c6aa00640 1 -- start start 2024-05-12T22:19:03.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.693+0000 7f5c6aa00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5c64102900 0x7f5c6419a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:03.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.693+0000 7f5c6aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c64103220 0x7f5c6419a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:03.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.693+0000 7f5c6aa00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5c6410bd20 0x7f5c6419eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:03.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.693+0000 7f5c6aa00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5c64111b20 con 0x7f5c64103220 2024-05-12T22:19:03.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.693+0000 7f5c6aa00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f5c641119a0 con 0x7f5c6410bd20 2024-05-12T22:19:03.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.693+0000 7f5c6aa00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f5c64111ca0 con 0x7f5c64102900 2024-05-12T22:19:03.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.694+0000 7f5c69600640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5c64102900 0x7f5c6419a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:03.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.694+0000 7f5c6a000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5c6410bd20 0x7f5c6419eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:03.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.694+0000 7f5c68c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c64103220 0x7f5c6419a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:03.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.694+0000 7f5c69600640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5c64102900 0x7f5c6419a2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:46280/0 (socket says 172.21.15.28:46280) 2024-05-12T22:19:03.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.694+0000 7f5c69600640 1 -- 172.21.15.28:0/2253305571 learned_addr learned my addr 172.21.15.28:0/2253305571 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:19:03.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.694+0000 7f5c6a000640 1 -- 172.21.15.28:0/2253305571 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5c64102900 msgr2=0x7f5c6419a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:03.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.694+0000 7f5c6a000640 1 --2- 172.21.15.28:0/2253305571 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5c64102900 0x7f5c6419a2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:03.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.694+0000 7f5c6a000640 1 -- 172.21.15.28:0/2253305571 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c64103220 msgr2=0x7f5c6419a800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:03.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.694+0000 7f5c6a000640 1 --2- 172.21.15.28:0/2253305571 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c64103220 0x7f5c6419a800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:03.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.694+0000 7f5c6a000640 1 -- 172.21.15.28:0/2253305571 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c6419f260 con 0x7f5c6410bd20 2024-05-12T22:19:03.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.694+0000 7f5c6a000640 1 --2- 172.21.15.28:0/2253305571 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5c6410bd20 0x7f5c6419eb30 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f5c5c009fd0 tx=0x7f5c5c00ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:19:03.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.695+0000 7f5c62000640 1 -- 172.21.15.28:0/2253305571 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c5c00ce20 con 0x7f5c6410bd20 2024-05-12T22:19:03.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.695+0000 7f5c62000640 1 -- 172.21.15.28:0/2253305571 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5c5c005020 con 0x7f5c6410bd20 2024-05-12T22:19:03.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.695+0000 7f5c62000640 1 -- 172.21.15.28:0/2253305571 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c5c010720 con 0x7f5c6410bd20 2024-05-12T22:19:03.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.695+0000 7f5c6aa00640 1 -- 172.21.15.28:0/2253305571 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c6419f550 con 0x7f5c6410bd20 2024-05-12T22:19:03.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.695+0000 7f5c6aa00640 1 -- 172.21.15.28:0/2253305571 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c641a6de0 con 0x7f5c6410bd20 2024-05-12T22:19:03.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.698+0000 7f5c62000640 1 -- 172.21.15.28:0/2253305571 <== mon.2 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7f5c5c005540 con 0x7f5c6410bd20 2024-05-12T22:19:03.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.698+0000 7f5c6ed6d740 1 Processor -- start 2024-05-12T22:19:03.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.698+0000 7f5c6ed6d740 1 -- start start 2024-05-12T22:19:03.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.699+0000 7f5c6ed6d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6d3f095e0 0x55b6d3f099c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:03.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.699+0000 7f5c6ed6d740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b6d3f09f00 0x55b6d3f14090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:03.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.699+0000 7f5c6ed6d740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b6d3f145d0 0x55b6d3f169c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:03.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.699+0000 7f5c6ed6d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b6d3ec92a0 con 0x55b6d3f095e0 2024-05-12T22:19:03.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.699+0000 7f5c6ed6d740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x55b6d3e65b90 con 0x55b6d3f09f00 2024-05-12T22:19:03.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.699+0000 7f5c6ed6d740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55b6d3e0d5b0 con 0x55b6d3f145d0 2024-05-12T22:19:03.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.699+0000 7f5c68c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6d3f095e0 0x55b6d3f099c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:03.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.699+0000 7f5c68c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6d3f095e0 0x55b6d3f099c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:32952/0 (socket says 172.21.15.28:32952) 2024-05-12T22:19:03.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.699+0000 7f5c68c00640 1 -- 172.21.15.28:0/2826421750 learned_addr learned my addr 172.21.15.28:0/2826421750 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:19:03.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.699+0000 7f5c69600640 1 --2- 172.21.15.28:0/2826421750 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b6d3f09f00 0x55b6d3f14090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:03.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.699+0000 7f5c68c00640 1 --2- 172.21.15.28:0/2826421750 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b6d3f145d0 0x55b6d3f169c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:03.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.700+0000 7f5c69600640 1 -- 172.21.15.28:0/2826421750 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b6d3f145d0 msgr2=0x55b6d3f169c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:19:03.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.700+0000 7f5c69600640 1 --2- 172.21.15.28:0/2826421750 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b6d3f145d0 0x55b6d3f169c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:03.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.700+0000 7f5c69600640 1 -- 172.21.15.28:0/2826421750 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6d3f095e0 msgr2=0x55b6d3f099c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:19:03.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.700+0000 7f5c69600640 1 --2- 172.21.15.28:0/2826421750 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6d3f095e0 0x55b6d3f099c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:03.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.700+0000 7f5c69600640 1 -- 172.21.15.28:0/2826421750 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b6d3c437d0 con 0x55b6d3f09f00 2024-05-12T22:19:03.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.700+0000 7f5c68c00640 1 --2- 172.21.15.28:0/2826421750 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6d3f095e0 0x55b6d3f099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:19:03.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.700+0000 7f5c69600640 1 --2- 172.21.15.28:0/2826421750 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b6d3f09f00 0x55b6d3f14090 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f5c58009c80 tx=0x7f5c58011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=adac9955065d90e8 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:19:03.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.700+0000 7f5c62000640 1 --2- 172.21.15.28:0/2253305571 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f5c4c0a2d80 0x7f5c4c0a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:03.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.700+0000 7f5c56600640 1 -- 172.21.15.28:0/2826421750 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c58004230 con 0x55b6d3f09f00 2024-05-12T22:19:03.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.700+0000 7f5c56600640 1 -- 172.21.15.28:0/2826421750 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5c580043d0 con 0x55b6d3f09f00 2024-05-12T22:19:03.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.700+0000 7f5c56600640 1 -- 172.21.15.28:0/2826421750 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c58004df0 con 0x55b6d3f09f00 2024-05-12T22:19:03.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.700+0000 7f5c62000640 1 -- 172.21.15.28:0/2253305571 <== mon.2 v2:172.21.15.86:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8513+0+0 (secure 0 0 0) 0x7f5c5c0cf9d0 con 0x7f5c6410bd20 2024-05-12T22:19:03.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.701+0000 7f5c68c00640 1 --2- 172.21.15.28:0/2253305571 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f5c4c0a2d80 0x7f5c4c0a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:03.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.701+0000 7f5c6ed6d740 1 -- 172.21.15.28:0/2826421750 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b6d3f09f00 msgr2=0x55b6d3f14090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:03.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.701+0000 7f5c6ed6d740 1 --2- 172.21.15.28:0/2826421750 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b6d3f09f00 0x55b6d3f14090 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f5c58009c80 tx=0x7f5c58011810 comp rx=0 tx=0).stop 2024-05-12T22:19:03.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.701+0000 7f5c6ed6d740 1 -- 172.21.15.28:0/2826421750 shutdown_connections 2024-05-12T22:19:03.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.701+0000 7f5c6ed6d740 1 --2- 172.21.15.28:0/2826421750 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b6d3f145d0 0x55b6d3f169c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:03.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.701+0000 7f5c6ed6d740 1 --2- 172.21.15.28:0/2826421750 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b6d3f09f00 0x55b6d3f14090 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:03.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.701+0000 7f5c6ed6d740 1 --2- 172.21.15.28:0/2826421750 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6d3f095e0 0x55b6d3f099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:03.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.701+0000 7f5c6ed6d740 1 -- 172.21.15.28:0/2826421750 >> 172.21.15.28:0/2826421750 conn(0x55b6d3f05900 msgr2=0x55b6d3f05ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:19:03.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.701+0000 7f5c68c00640 1 --2- 172.21.15.28:0/2253305571 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f5c4c0a2d80 0x7f5c4c0a5240 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x55b6d3f06a90 tx=0x7f5c50000bc0 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:19:03.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.701+0000 7f5c6ed6d740 1 -- 172.21.15.28:0/2826421750 shutdown_connections 2024-05-12T22:19:03.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.702+0000 7f5c6ed6d740 1 -- 172.21.15.28:0/2826421750 wait complete. 2024-05-12T22:19:03.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.702+0000 7f5c6ed6d740 1 -- start start 2024-05-12T22:19:03.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.702+0000 7f5c6ed6d740 1 Processor -- start 2024-05-12T22:19:03.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.703+0000 7f5c6ed6d740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b6d3f095e0 0x55b6d3f13f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:03.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.703+0000 7f5c6ed6d740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b6d3f09f00 0x55b6d3f11200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:03.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.703+0000 7f5c6ed6d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6d3f145d0 0x55b6d3f11740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:03.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.703+0000 7f5c6ed6d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b6d3e209f0 con 0x55b6d3f145d0 2024-05-12T22:19:03.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.703+0000 7f5c6ed6d740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x55b6d3e61b50 con 0x55b6d3f095e0 2024-05-12T22:19:03.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.703+0000 7f5c6ed6d740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55b6d3f173c0 con 0x55b6d3f09f00 2024-05-12T22:19:03.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.703+0000 7f5c6ed6d740 10 client.0 _ll_register_callbacks cb 0x55b6d3b78be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:19:03.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.703+0000 7f5c6a000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6d3f145d0 0x55b6d3f11740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:03.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.703+0000 7f5c69600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b6d3f095e0 0x55b6d3f13f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:03.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.703+0000 7f5c68c00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b6d3f09f00 0x55b6d3f11200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:03.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.703+0000 7f5c69600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b6d3f095e0 0x55b6d3f13f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:34280/0 (socket says 172.21.15.28:34280) 2024-05-12T22:19:03.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.703+0000 7f5c69600640 1 -- 172.21.15.28:0/1195265809 learned_addr learned my addr 172.21.15.28:0/1195265809 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:19:03.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.703+0000 7f5c6a000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6d3f145d0 0x55b6d3f11740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:32962/0 (socket says 172.21.15.28:32962) 2024-05-12T22:19:03.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.703+0000 7f5c69600640 1 -- 172.21.15.28:0/1195265809 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b6d3f09f00 msgr2=0x55b6d3f11200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:19:03.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.703+0000 7f5c69600640 1 --2- 172.21.15.28:0/1195265809 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b6d3f09f00 0x55b6d3f11200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:03.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.703+0000 7f5c69600640 1 -- 172.21.15.28:0/1195265809 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6d3f145d0 msgr2=0x55b6d3f11740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:19:03.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.703+0000 7f5c69600640 1 --2- 172.21.15.28:0/1195265809 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6d3f145d0 0x55b6d3f11740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:03.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.703+0000 7f5c69600640 1 -- 172.21.15.28:0/1195265809 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b6d3b793f0 con 0x55b6d3f095e0 2024-05-12T22:19:03.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.704+0000 7f5c69600640 1 --2- 172.21.15.28:0/1195265809 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b6d3f095e0 0x55b6d3f13f10 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f5c58000c00 tx=0x7f5c58019b00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e38aa5fd39082dab server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:19:03.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.704+0000 7f5c4aa00640 10 client.0 ms_handle_connect on v2:172.21.15.86:3300/0 2024-05-12T22:19:03.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.704+0000 7f5c4aa00640 1 -- 172.21.15.28:0/1195265809 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c58028020 con 0x55b6d3f095e0 2024-05-12T22:19:03.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.704+0000 7f5c6ed6d740 1 -- 172.21.15.28:0/1195265809 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5c580043d0 con 0x55b6d3f095e0 2024-05-12T22:19:03.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.704+0000 7f5c4aa00640 1 -- 172.21.15.28:0/1195265809 <== mon.2 v2:172.21.15.86:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8513+0+0 (secure 0 0 0) 0x7f5c58024620 con 0x55b6d3f095e0 2024-05-12T22:19:03.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.705+0000 7f5c4aa00640 1 -- 172.21.15.28:0/1195265809 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f5c58033050 con 0x55b6d3f095e0 2024-05-12T22:19:03.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.705+0000 7f5c6ed6d740 10 client.27377 fetch_fsmap learned FSMap version 26 2024-05-12T22:19:03.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.705+0000 7f5c6ed6d740 1 -- 172.21.15.28:0/1195265809 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55b6d3b390f0 con 0x55b6d3f095e0 2024-05-12T22:19:03.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.705+0000 7f5c4aa00640 1 -- 172.21.15.28:0/1195265809 <== mon.2 v2:172.21.15.86:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5c580043d0 con 0x55b6d3f095e0 2024-05-12T22:19:03.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.705+0000 7f5c4aa00640 1 -- 172.21.15.28:0/1195265809 <== mon.2 v2:172.21.15.86:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f5c58018070 con 0x55b6d3f095e0 2024-05-12T22:19:03.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.705+0000 7f5c6ed6d740 10 client.27377 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:19:03.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.705+0000 7f5c6ed6d740 10 client.27377 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:19:03.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.705+0000 7f5c6ed6d740 20 client.27377 populate_metadata read hostname 'smithi028' 2024-05-12T22:19:03.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.705+0000 7f5c6ed6d740 1 --2- 172.21.15.28:0/1195265809 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55b6d3f24060 0x55b6d3f26450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:03.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.705+0000 7f5c6ed6d740 4 client.27377 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-12T22:19:03.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.705+0000 7f5c6ed6d740 1 -- 172.21.15.28:0/1195265809 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b6d3e732c0 con 0x55b6d3f24060 2024-05-12T22:19:03.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.706+0000 7f5c68c00640 1 --2- 172.21.15.28:0/1195265809 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55b6d3f24060 0x55b6d3f26450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:03.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.706+0000 7f5c68c00640 1 --2- 172.21.15.28:0/1195265809 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55b6d3f24060 0x55b6d3f26450 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e39240d61f77fb6d server_cookie=87438771a7d69199 in_seq=0 out_seq=0 2024-05-12T22:19:03.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.706+0000 7f5c4aa00640 10 client.27377 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:19:03.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.707+0000 7f5c4aa00640 1 -- 172.21.15.28:0/1195265809 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b6d3e732c0 con 0x55b6d3f24060 2024-05-12T22:19:03.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.707+0000 7f5c4aa00640 10 client.27377 handle_command_reply: tid=0 2024-05-12T22:19:03.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.707+0000 7f5c6ed6d740 1 client.27377 shutdown 2024-05-12T22:19:03.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.707+0000 7f5c6ed6d740 1 -- 172.21.15.28:0/1195265809 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b6d3f095e0 msgr2=0x55b6d3f13f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:03.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.707+0000 7f5c6ed6d740 1 --2- 172.21.15.28:0/1195265809 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b6d3f095e0 0x55b6d3f13f10 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f5c58000c00 tx=0x7f5c58019b00 comp rx=0 tx=0).stop 2024-05-12T22:19:03.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.707+0000 7f5c6ed6d740 1 -- 172.21.15.28:0/1195265809 shutdown_connections 2024-05-12T22:19:03.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.707+0000 7f5c6ed6d740 1 --2- 172.21.15.28:0/1195265809 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55b6d3f24060 0x55b6d3f26450 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:03.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.707+0000 7f5c6ed6d740 1 --2- 172.21.15.28:0/1195265809 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6d3f145d0 0x55b6d3f11740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:03.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.707+0000 7f5c4aa00640 0 client.27377 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:19:03.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.707+0000 7f5c6ed6d740 1 --2- 172.21.15.28:0/1195265809 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b6d3f09f00 0x55b6d3f11200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:03.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.707+0000 7f5c6ed6d740 1 --2- 172.21.15.28:0/1195265809 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b6d3f095e0 0x55b6d3f13f10 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:03.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.707+0000 7f5c6ed6d740 1 -- 172.21.15.28:0/1195265809 >> 172.21.15.28:0/1195265809 conn(0x55b6d3f05900 msgr2=0x55b6d3efc330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:19:03.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.708+0000 7f5c6ed6d740 1 -- 172.21.15.28:0/1195265809 shutdown_connections 2024-05-12T22:19:03.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.708+0000 7f5c6ed6d740 1 -- 172.21.15.28:0/1195265809 wait complete. 2024-05-12T22:19:03.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.708+0000 7f5c6ed6d740 20 client.27377 trim_cache size 0 max 16384 2024-05-12T22:19:03.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.728+0000 7f5c6ed6d740 1 Processor -- start 2024-05-12T22:19:03.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.728+0000 7f5c6ed6d740 1 -- start start 2024-05-12T22:19:03.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.728+0000 7f5c6ed6d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6d3f095e0 0x55b6d3f0c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:03.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.728+0000 7f5c6ed6d740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b6d3f09f00 0x55b6d3f10ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:03.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.728+0000 7f5c6ed6d740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b6d3f145d0 0x55b6d3f13a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:03.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.728+0000 7f5c6ed6d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5c641adad0 con 0x55b6d3f095e0 2024-05-12T22:19:03.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.728+0000 7f5c6ed6d740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f5c641ab3a0 con 0x55b6d3f09f00 2024-05-12T22:19:03.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.728+0000 7f5c6ed6d740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f5c641a9230 con 0x55b6d3f145d0 2024-05-12T22:19:03.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.728+0000 7f5c68c00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b6d3f09f00 0x55b6d3f10ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:03.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.728+0000 7f5c68c00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b6d3f09f00 0x55b6d3f10ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:34284/0 (socket says 172.21.15.28:34284) 2024-05-12T22:19:03.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.728+0000 7f5c68c00640 1 -- 172.21.15.28:0/2495588236 learned_addr learned my addr 172.21.15.28:0/2495588236 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:19:03.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.729+0000 7f5c6a000640 1 --2- 172.21.15.28:0/2495588236 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b6d3f145d0 0x55b6d3f13a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:03.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.729+0000 7f5c69600640 1 --2- 172.21.15.28:0/2495588236 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6d3f095e0 0x55b6d3f0c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:03.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.729+0000 7f5c68c00640 1 -- 172.21.15.28:0/2495588236 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b6d3f145d0 msgr2=0x55b6d3f13a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:19:03.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.729+0000 7f5c68c00640 1 --2- 172.21.15.28:0/2495588236 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b6d3f145d0 0x55b6d3f13a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:03.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.729+0000 7f5c68c00640 1 -- 172.21.15.28:0/2495588236 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6d3f095e0 msgr2=0x55b6d3f0c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:19:03.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.729+0000 7f5c68c00640 1 --2- 172.21.15.28:0/2495588236 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6d3f095e0 0x55b6d3f0c7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:03.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.729+0000 7f5c68c00640 1 -- 172.21.15.28:0/2495588236 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b6d3f0f750 con 0x55b6d3f09f00 2024-05-12T22:19:03.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.729+0000 7f5c68c00640 1 --2- 172.21.15.28:0/2495588236 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b6d3f09f00 0x55b6d3f10ae0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f5c50005940 tx=0x7f5c5003b000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4e76c47d74e25bdc server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:19:03.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.730+0000 7f5c55c00640 1 -- 172.21.15.28:0/2495588236 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c5005a070 con 0x55b6d3f09f00 2024-05-12T22:19:03.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.730+0000 7f5c55c00640 1 -- 172.21.15.28:0/2495588236 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5c500548f0 con 0x55b6d3f09f00 2024-05-12T22:19:03.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.730+0000 7f5c55c00640 1 -- 172.21.15.28:0/2495588236 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c50035040 con 0x55b6d3f09f00 2024-05-12T22:19:03.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.730+0000 7f5c6ed6d740 1 -- 172.21.15.28:0/2495588236 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b6d3f09f00 msgr2=0x55b6d3f10ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:03.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.730+0000 7f5c6ed6d740 1 --2- 172.21.15.28:0/2495588236 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b6d3f09f00 0x55b6d3f10ae0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f5c50005940 tx=0x7f5c5003b000 comp rx=0 tx=0).stop 2024-05-12T22:19:03.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.730+0000 7f5c6ed6d740 1 -- 172.21.15.28:0/2495588236 shutdown_connections 2024-05-12T22:19:03.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.730+0000 7f5c6ed6d740 1 --2- 172.21.15.28:0/2495588236 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b6d3f145d0 0x55b6d3f13a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:03.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.730+0000 7f5c6ed6d740 1 --2- 172.21.15.28:0/2495588236 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b6d3f09f00 0x55b6d3f10ae0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:03.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.730+0000 7f5c6ed6d740 1 --2- 172.21.15.28:0/2495588236 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6d3f095e0 0x55b6d3f0c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:03.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.730+0000 7f5c6ed6d740 1 -- 172.21.15.28:0/2495588236 >> 172.21.15.28:0/2495588236 conn(0x55b6d3f05900 msgr2=0x55b6d3f0ac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:19:03.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.731+0000 7f5c6ed6d740 1 -- 172.21.15.28:0/2495588236 shutdown_connections 2024-05-12T22:19:03.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.731+0000 7f5c6ed6d740 1 -- 172.21.15.28:0/2495588236 wait complete. 2024-05-12T22:19:03.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.731+0000 7f5c6ed6d740 1 -- start start 2024-05-12T22:19:03.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.731+0000 7f5c6ed6d740 1 Processor -- start 2024-05-12T22:19:03.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.732+0000 7f5c6ed6d740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b6d3f095e0 0x55b6d3f18250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:03.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.732+0000 7f5c6ed6d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6d3f09f00 0x55b6d3f18790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:03.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.732+0000 7f5c6ed6d740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b6d3f145d0 0x55b6d3f18cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:03.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.732+0000 7f5c6ed6d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5c641a9880 con 0x55b6d3f09f00 2024-05-12T22:19:03.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.732+0000 7f5c6ed6d740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f5c641aacd0 con 0x55b6d3f145d0 2024-05-12T22:19:03.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.732+0000 7f5c6ed6d740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f5c641a9f30 con 0x55b6d3f095e0 2024-05-12T22:19:03.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.732+0000 7f5c6ed6d740 10 client.0 _ll_register_callbacks cb 0x7f5c58005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:19:03.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.732+0000 7f5c69600640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b6d3f095e0 0x55b6d3f18250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:03.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.732+0000 7f5c68c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6d3f09f00 0x55b6d3f18790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:03.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.732+0000 7f5c69600640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b6d3f095e0 0x55b6d3f18250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:46316/0 (socket says 172.21.15.28:46316) 2024-05-12T22:19:03.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.732+0000 7f5c69600640 1 -- 172.21.15.28:0/1706894709 learned_addr learned my addr 172.21.15.28:0/1706894709 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:19:03.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.732+0000 7f5c6a000640 1 --2- 172.21.15.28:0/1706894709 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b6d3f145d0 0x55b6d3f18cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:03.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.732+0000 7f5c69600640 1 -- 172.21.15.28:0/1706894709 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b6d3f145d0 msgr2=0x55b6d3f18cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:19:03.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.732+0000 7f5c69600640 1 --2- 172.21.15.28:0/1706894709 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b6d3f145d0 0x55b6d3f18cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:03.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.732+0000 7f5c69600640 1 -- 172.21.15.28:0/1706894709 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6d3f09f00 msgr2=0x55b6d3f18790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:19:03.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.732+0000 7f5c69600640 1 --2- 172.21.15.28:0/1706894709 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6d3f09f00 0x55b6d3f18790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:03.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.732+0000 7f5c69600640 1 -- 172.21.15.28:0/1706894709 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b6d3f89630 con 0x55b6d3f095e0 2024-05-12T22:19:03.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.732+0000 7f5c68c00640 1 --2- 172.21.15.28:0/1706894709 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6d3f09f00 0x55b6d3f18790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:19:03.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.733+0000 7f5c69600640 1 --2- 172.21.15.28:0/1706894709 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b6d3f095e0 0x55b6d3f18250 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f5c5801afd0 tx=0x7f5c58011de0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=47b4dceb47556d14 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:19:03.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.733+0000 7f5c4aa00640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-12T22:19:03.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.733+0000 7f5c4aa00640 1 -- 172.21.15.28:0/1706894709 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c58028070 con 0x55b6d3f095e0 2024-05-12T22:19:03.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.733+0000 7f5c4aa00640 1 -- 172.21.15.28:0/1706894709 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8513+0+0 (secure 0 0 0) 0x7f5c58005320 con 0x55b6d3f095e0 2024-05-12T22:19:03.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.733+0000 7f5c6ed6d740 1 -- 172.21.15.28:0/1706894709 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5c500548f0 con 0x55b6d3f095e0 2024-05-12T22:19:03.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.733+0000 7f5c4aa00640 1 -- 172.21.15.28:0/1706894709 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f5c5803f050 con 0x55b6d3f095e0 2024-05-12T22:19:03.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.733+0000 7f5c6ed6d740 10 client.27334 fetch_fsmap learned FSMap version 26 2024-05-12T22:19:03.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.733+0000 7f5c6ed6d740 1 -- 172.21.15.28:0/1706894709 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55b6d3f8a920 con 0x55b6d3f095e0 2024-05-12T22:19:03.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.734+0000 7f5c4aa00640 1 -- 172.21.15.28:0/1706894709 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5c500548f0 con 0x55b6d3f095e0 2024-05-12T22:19:03.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.734+0000 7f5c4aa00640 1 -- 172.21.15.28:0/1706894709 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f5c58018050 con 0x55b6d3f095e0 2024-05-12T22:19:03.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.734+0000 7f5c6ed6d740 10 client.27334 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:19:03.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.734+0000 7f5c6ed6d740 10 client.27334 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:19:03.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.734+0000 7f5c6ed6d740 20 client.27334 populate_metadata read hostname 'smithi028' 2024-05-12T22:19:03.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.734+0000 7f5c6ed6d740 1 --2- 172.21.15.28:0/1706894709 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55b6d3f8afe0 0x55b6d3f8d400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:03.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.734+0000 7f5c6ed6d740 4 client.27334 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-12T22:19:03.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.734+0000 7f5c6ed6d740 1 -- 172.21.15.28:0/1706894709 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55b6d3f0ce20 con 0x55b6d3f8afe0 2024-05-12T22:19:03.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.734+0000 7f5c68c00640 1 --2- 172.21.15.28:0/1706894709 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55b6d3f8afe0 0x55b6d3f8d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:03.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.735+0000 7f5c68c00640 1 --2- 172.21.15.28:0/1706894709 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55b6d3f8afe0 0x55b6d3f8d400 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8e4945419fa0e6ed server_cookie=22182cf83a008df5 in_seq=0 out_seq=0 2024-05-12T22:19:03.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.735+0000 7f5c4aa00640 10 client.27334 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:19:03.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.735+0000 7f5c4aa00640 1 -- 172.21.15.28:0/1706894709 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55b6d3f0ce20 con 0x55b6d3f8afe0 2024-05-12T22:19:03.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.735+0000 7f5c4aa00640 10 client.27334 handle_command_reply: tid=0 2024-05-12T22:19:03.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.735+0000 7f5c6ed6d740 1 client.27334 shutdown 2024-05-12T22:19:03.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.735+0000 7f5c6ed6d740 1 -- 172.21.15.28:0/1706894709 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b6d3f095e0 msgr2=0x55b6d3f18250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:03.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.735+0000 7f5c6ed6d740 1 --2- 172.21.15.28:0/1706894709 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b6d3f095e0 0x55b6d3f18250 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f5c5801afd0 tx=0x7f5c58011de0 comp rx=0 tx=0).stop 2024-05-12T22:19:03.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.735+0000 7f5c6ed6d740 1 -- 172.21.15.28:0/1706894709 shutdown_connections 2024-05-12T22:19:03.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.735+0000 7f5c6ed6d740 1 --2- 172.21.15.28:0/1706894709 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55b6d3f8afe0 0x55b6d3f8d400 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:03.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.735+0000 7f5c6ed6d740 1 --2- 172.21.15.28:0/1706894709 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b6d3f145d0 0x55b6d3f18cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:03.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.735+0000 7f5c6ed6d740 1 --2- 172.21.15.28:0/1706894709 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6d3f09f00 0x55b6d3f18790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:03.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.735+0000 7f5c6ed6d740 1 --2- 172.21.15.28:0/1706894709 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b6d3f095e0 0x55b6d3f18250 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:03.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.735+0000 7f5c6ed6d740 1 -- 172.21.15.28:0/1706894709 >> 172.21.15.28:0/1706894709 conn(0x55b6d3f05900 msgr2=0x55b6d3f12a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:19:03.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.735+0000 7f5c4aa00640 0 client.27334 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:19:03.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.736+0000 7f5c6ed6d740 1 -- 172.21.15.28:0/1706894709 shutdown_connections 2024-05-12T22:19:03.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.736+0000 7f5c6ed6d740 1 -- 172.21.15.28:0/1706894709 wait complete. 2024-05-12T22:19:03.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.736+0000 7f5c6ed6d740 20 client.27334 trim_cache size 0 max 16384 2024-05-12T22:19:03.742 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-05-12T22:19:03.742 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-05-12T22:19:03.742 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-05-12T22:19:03.742 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-05-12T22:19:03.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.736+0000 7f5c6aa00640 1 -- 172.21.15.28:0/2253305571 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f5c4c0a2d80 msgr2=0x7f5c4c0a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:03.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.736+0000 7f5c6aa00640 1 --2- 172.21.15.28:0/2253305571 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f5c4c0a2d80 0x7f5c4c0a5240 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x55b6d3f06a90 tx=0x7f5c50000bc0 comp rx=0 tx=0).stop 2024-05-12T22:19:03.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.736+0000 7f5c6aa00640 1 -- 172.21.15.28:0/2253305571 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5c6410bd20 msgr2=0x7f5c6419eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:03.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.736+0000 7f5c6aa00640 1 --2- 172.21.15.28:0/2253305571 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5c6410bd20 0x7f5c6419eb30 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f5c5c009fd0 tx=0x7f5c5c00ef10 comp rx=0 tx=0).stop 2024-05-12T22:19:03.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.737+0000 7f5c6aa00640 1 -- 172.21.15.28:0/2253305571 shutdown_connections 2024-05-12T22:19:03.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.737+0000 7f5c6aa00640 1 --2- 172.21.15.28:0/2253305571 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f5c4c0a2d80 0x7f5c4c0a5240 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:03.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.737+0000 7f5c6aa00640 1 --2- 172.21.15.28:0/2253305571 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5c6410bd20 0x7f5c6419eb30 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:03.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.737+0000 7f5c6aa00640 1 --2- 172.21.15.28:0/2253305571 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c64103220 0x7f5c6419a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:03.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.737+0000 7f5c6aa00640 1 --2- 172.21.15.28:0/2253305571 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5c64102900 0x7f5c6419a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:03.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.737+0000 7f5c6aa00640 1 -- 172.21.15.28:0/2253305571 >> 172.21.15.28:0/2253305571 conn(0x7f5c640fc140 msgr2=0x7f5c641015a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:19:03.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.737+0000 7f5c6aa00640 1 -- 172.21.15.28:0/2253305571 shutdown_connections 2024-05-12T22:19:03.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.737+0000 7f5c6aa00640 1 -- 172.21.15.28:0/2253305571 wait complete. 2024-05-12T22:19:03.755 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-12T22:19:03.756 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-12T22:19:03.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.963+0000 7fb188800640 1 -- 172.21.15.28:0/2819983744 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb1800ffa30 msgr2=0x7fb1800ffeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:03.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.963+0000 7fb188800640 1 --2- 172.21.15.28:0/2819983744 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb1800ffa30 0x7fb1800ffeb0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fb170009890 tx=0x7fb17002a370 comp rx=0 tx=0).stop 2024-05-12T22:19:03.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.963+0000 7fb188800640 1 -- 172.21.15.28:0/2819983744 shutdown_connections 2024-05-12T22:19:03.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.963+0000 7fb188800640 1 --2- 172.21.15.28:0/2819983744 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb1801003f0 0x7fb18010c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:03.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.963+0000 7fb188800640 1 --2- 172.21.15.28:0/2819983744 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb1800ffa30 0x7fb1800ffeb0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:03.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.963+0000 7fb188800640 1 --2- 172.21.15.28:0/2819983744 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb180105a40 0x7fb180105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:03.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.963+0000 7fb188800640 1 -- 172.21.15.28:0/2819983744 >> 172.21.15.28:0/2819983744 conn(0x7fb180076890 msgr2=0x7fb1800fdc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:19:03.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.963+0000 7fb188800640 1 -- 172.21.15.28:0/2819983744 shutdown_connections 2024-05-12T22:19:03.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.963+0000 7fb188800640 1 -- 172.21.15.28:0/2819983744 wait complete. 2024-05-12T22:19:03.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.964+0000 7fb188800640 1 Processor -- start 2024-05-12T22:19:03.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.964+0000 7fb188800640 1 -- start start 2024-05-12T22:19:03.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.965+0000 7fb188800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb1800ffa30 0x7fb180198120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:03.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.965+0000 7fb188800640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb1801003f0 0x7fb180198660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:03.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.965+0000 7fb188800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb180105a40 0x7fb18019c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:03.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.965+0000 7fb188800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb18010f9e0 con 0x7fb180105a40 2024-05-12T22:19:03.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.965+0000 7fb188800640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fb18010f860 con 0x7fb1800ffa30 2024-05-12T22:19:03.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.965+0000 7fb188800640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb18010fb60 con 0x7fb1801003f0 2024-05-12T22:19:03.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.965+0000 7fb186800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb180105a40 0x7fb18019c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:03.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.965+0000 7fb185e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb1800ffa30 0x7fb180198120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:03.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.965+0000 7fb186800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb180105a40 0x7fb18019c9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:32998/0 (socket says 172.21.15.28:32998) 2024-05-12T22:19:03.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.965+0000 7fb186800640 1 -- 172.21.15.28:0/2535402623 learned_addr learned my addr 172.21.15.28:0/2535402623 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:19:03.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.965+0000 7fb185400640 1 --2- 172.21.15.28:0/2535402623 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb1801003f0 0x7fb180198660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:03.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.966+0000 7fb186800640 1 -- 172.21.15.28:0/2535402623 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb1801003f0 msgr2=0x7fb180198660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:03.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.966+0000 7fb186800640 1 --2- 172.21.15.28:0/2535402623 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb1801003f0 0x7fb180198660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:03.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.966+0000 7fb186800640 1 -- 172.21.15.28:0/2535402623 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb1800ffa30 msgr2=0x7fb180198120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:03.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.966+0000 7fb186800640 1 --2- 172.21.15.28:0/2535402623 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb1800ffa30 0x7fb180198120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:03.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.966+0000 7fb186800640 1 -- 172.21.15.28:0/2535402623 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb18019d110 con 0x7fb180105a40 2024-05-12T22:19:03.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.966+0000 7fb185e00640 1 --2- 172.21.15.28:0/2535402623 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb1800ffa30 0x7fb180198120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:19:03.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.966+0000 7fb186800640 1 --2- 172.21.15.28:0/2535402623 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb180105a40 0x7fb18019c9e0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fb17c00afd0 tx=0x7fb17c0095b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:19:03.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.966+0000 7fb17aa00640 1 -- 172.21.15.28:0/2535402623 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb17c004b20 con 0x7fb180105a40 2024-05-12T22:19:03.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.966+0000 7fb188800640 1 -- 172.21.15.28:0/2535402623 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb18019d400 con 0x7fb180105a40 2024-05-12T22:19:03.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.966+0000 7fb17aa00640 1 -- 172.21.15.28:0/2535402623 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb17c004cc0 con 0x7fb180105a40 2024-05-12T22:19:03.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.966+0000 7fb17aa00640 1 -- 172.21.15.28:0/2535402623 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb17c015670 con 0x7fb180105a40 2024-05-12T22:19:03.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.967+0000 7fb188800640 1 -- 172.21.15.28:0/2535402623 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1801a4c90 con 0x7fb180105a40 2024-05-12T22:19:03.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.968+0000 7fb188800640 1 -- 172.21.15.28:0/2535402623 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb180068aa0 con 0x7fb180105a40 2024-05-12T22:19:03.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.970+0000 7fb17aa00640 1 -- 172.21.15.28:0/2535402623 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7fb17c01c020 con 0x7fb180105a40 2024-05-12T22:19:03.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.973+0000 7fb17aa00640 1 --2- 172.21.15.28:0/2535402623 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fb1680a2d00 0x7fb1680a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:03.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.973+0000 7fb17aa00640 1 -- 172.21.15.28:0/2535402623 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8513+0+0 (secure 0 0 0) 0x7fb17c0cf780 con 0x7fb180105a40 2024-05-12T22:19:03.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.973+0000 7fb185e00640 1 --2- 172.21.15.28:0/2535402623 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fb1680a2d00 0x7fb1680a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:03.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.974+0000 7fb185e00640 1 --2- 172.21.15.28:0/2535402623 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fb1680a2d00 0x7fb1680a51c0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fb174009f40 tx=0x7fb174005f90 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:19:03.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:03.976+0000 7fb17aa00640 1 -- 172.21.15.28:0/2535402623 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fb17c023030 con 0x7fb180105a40 2024-05-12T22:19:04.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.220+0000 7fb188800640 1 -- 172.21.15.28:0/2535402623 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb18010a030 con 0x7fb180105a40 2024-05-12T22:19:04.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.221+0000 7fb17aa00640 1 -- 172.21.15.28:0/2535402623 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11864 (secure 0 0 0) 0x7fb17c09c340 con 0x7fb180105a40 2024-05-12T22:19:04.221 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-12T22:19:04.222 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1844141745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1844141745},{"type":"v1","addr":"172.21.15.186:6833","nonce":1844141745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6833/2960763758","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":2960763758},{"type":"v1","addr":"172.21.15.86:6833","nonce":2960763758}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24470,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/4131134219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":4131134219},{"type":"v1","addr":"172.21.15.28:6835","nonce":4131134219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24481,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/1727869064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1727869064},{"type":"v1","addr":"172.21.15.186:6835","nonce":1727869064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24482,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2938260704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2938260704},{"type":"v1","addr":"172.21.15.28:6837","nonce":2938260704}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24494,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3473700910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3473700910},{"type":"v1","addr":"172.21.15.86:6835","nonce":3473700910}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-12T22:06:33.426986+0000","modified":"2024-05-12T22:13:22.499381+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24581,"mds_1":24550,"mds_2":24532},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24499":{"gid":24499,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6837/3600809235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":3600809235},{"type":"v1","addr":"172.21.15.186:6837","nonce":3600809235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"i","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6839/2399610807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":2399610807},{"type":"v1","addr":"172.21.15.186:6839","nonce":2399610807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24530":{"gid":24530,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.86:6837/3782448884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":3782448884},{"type":"v1","addr":"172.21.15.86:6837","nonce":3782448884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24532":{"gid":24532,"name":"g","rank":2,"incarnation":21,"state":"up:active","state_seq":6,"addr":"172.21.15.28:6839/123892158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":123892158},{"type":"v1","addr":"172.21.15.28:6839","nonce":123892158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24550":{"gid":24550,"name":"a","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.28:6841/761183870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":761183870},{"type":"v1","addr":"172.21.15.28:6841","nonce":761183870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24581":{"gid":24581,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6839/1251999130","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6838","nonce":1251999130},{"type":"v1","addr":"172.21.15.86:6839","nonce":1251999130}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24581,"qdb_cluster":[24550,24581,24532]},"id":1}]} 2024-05-12T22:19:04.222 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 26 2024-05-12T22:19:04.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.225+0000 7fb188800640 1 -- 172.21.15.28:0/2535402623 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fb1680a2d00 msgr2=0x7fb1680a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:04.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.225+0000 7fb188800640 1 --2- 172.21.15.28:0/2535402623 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fb1680a2d00 0x7fb1680a51c0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fb174009f40 tx=0x7fb174005f90 comp rx=0 tx=0).stop 2024-05-12T22:19:04.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.225+0000 7fb188800640 1 -- 172.21.15.28:0/2535402623 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb180105a40 msgr2=0x7fb18019c9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:04.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.225+0000 7fb188800640 1 --2- 172.21.15.28:0/2535402623 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb180105a40 0x7fb18019c9e0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fb17c00afd0 tx=0x7fb17c0095b0 comp rx=0 tx=0).stop 2024-05-12T22:19:04.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.225+0000 7fb188800640 1 -- 172.21.15.28:0/2535402623 shutdown_connections 2024-05-12T22:19:04.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.225+0000 7fb188800640 1 --2- 172.21.15.28:0/2535402623 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fb1680a2d00 0x7fb1680a51c0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:04.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.225+0000 7fb188800640 1 --2- 172.21.15.28:0/2535402623 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb180105a40 0x7fb18019c9e0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:04.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.225+0000 7fb188800640 1 --2- 172.21.15.28:0/2535402623 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb1801003f0 0x7fb180198660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:04.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.225+0000 7fb188800640 1 --2- 172.21.15.28:0/2535402623 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb1800ffa30 0x7fb180198120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:04.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.225+0000 7fb188800640 1 -- 172.21.15.28:0/2535402623 >> 172.21.15.28:0/2535402623 conn(0x7fb180076890 msgr2=0x7fb1800fe410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:19:04.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.225+0000 7fb188800640 1 -- 172.21.15.28:0/2535402623 shutdown_connections 2024-05-12T22:19:04.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.225+0000 7fb188800640 1 -- 172.21.15.28:0/2535402623 wait complete. 2024-05-12T22:19:04.245 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-05-12T22:19:04.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.461+0000 7fd957200640 1 -- 172.21.15.28:0/43154317 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9581098e0 msgr2=0x7fd958109cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:04.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.461+0000 7fd957200640 1 --2- 172.21.15.28:0/43154317 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9581098e0 0x7fd958109cc0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fd948009930 tx=0x7fd94802a420 comp rx=0 tx=0).stop 2024-05-12T22:19:04.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.461+0000 7fd957200640 1 -- 172.21.15.28:0/43154317 shutdown_connections 2024-05-12T22:19:04.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.461+0000 7fd957200640 1 --2- 172.21.15.28:0/43154317 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd958104290 0x7fd9581107a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:04.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.461+0000 7fd957200640 1 --2- 172.21.15.28:0/43154317 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd9581038d0 0x7fd958103d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:04.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.461+0000 7fd957200640 1 --2- 172.21.15.28:0/43154317 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9581098e0 0x7fd958109cc0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:04.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.461+0000 7fd957200640 1 -- 172.21.15.28:0/43154317 >> 172.21.15.28:0/43154317 conn(0x7fd9580ff360 msgr2=0x7fd958101780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:19:04.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.461+0000 7fd957200640 1 -- 172.21.15.28:0/43154317 shutdown_connections 2024-05-12T22:19:04.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.461+0000 7fd957200640 1 -- 172.21.15.28:0/43154317 wait complete. 2024-05-12T22:19:04.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.462+0000 7fd957200640 1 Processor -- start 2024-05-12T22:19:04.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.463+0000 7fd957200640 1 -- start start 2024-05-12T22:19:04.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.463+0000 7fd957200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd9581038d0 0x7fd95819a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:04.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.463+0000 7fd957200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd958104290 0x7fd95819a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:04.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.463+0000 7fd957200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9581098e0 0x7fd95819ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:04.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.463+0000 7fd957200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd958113d10 con 0x7fd9581098e0 2024-05-12T22:19:04.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.463+0000 7fd957200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fd958113b90 con 0x7fd958104290 2024-05-12T22:19:04.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.463+0000 7fd957200640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd958113e90 con 0x7fd9581038d0 2024-05-12T22:19:04.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.463+0000 7fd956800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9581098e0 0x7fd95819ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:04.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.463+0000 7fd956800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9581098e0 0x7fd95819ea80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33012/0 (socket says 172.21.15.28:33012) 2024-05-12T22:19:04.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.463+0000 7fd956800640 1 -- 172.21.15.28:0/2291922687 learned_addr learned my addr 172.21.15.28:0/2291922687 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:19:04.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.463+0000 7fd955e00640 1 --2- 172.21.15.28:0/2291922687 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd9581038d0 0x7fd95819a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:04.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.463+0000 7fd955400640 1 --2- 172.21.15.28:0/2291922687 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd958104290 0x7fd95819a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:04.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.464+0000 7fd956800640 1 -- 172.21.15.28:0/2291922687 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd9581038d0 msgr2=0x7fd95819a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:04.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.464+0000 7fd956800640 1 --2- 172.21.15.28:0/2291922687 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd9581038d0 0x7fd95819a210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:04.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.464+0000 7fd956800640 1 -- 172.21.15.28:0/2291922687 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd958104290 msgr2=0x7fd95819a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:04.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.464+0000 7fd956800640 1 --2- 172.21.15.28:0/2291922687 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd958104290 0x7fd95819a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:04.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.464+0000 7fd956800640 1 -- 172.21.15.28:0/2291922687 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd95819f1b0 con 0x7fd9581098e0 2024-05-12T22:19:04.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.464+0000 7fd955400640 1 --2- 172.21.15.28:0/2291922687 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd958104290 0x7fd95819a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:19:04.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.464+0000 7fd956800640 1 --2- 172.21.15.28:0/2291922687 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9581098e0 0x7fd95819ea80 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fd95000eb10 tx=0x7fd95000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:19:04.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.464+0000 7fd94ea00640 1 -- 172.21.15.28:0/2291922687 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd950004b20 con 0x7fd9581098e0 2024-05-12T22:19:04.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.464+0000 7fd94ea00640 1 -- 172.21.15.28:0/2291922687 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd950004cc0 con 0x7fd9581098e0 2024-05-12T22:19:04.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.465+0000 7fd957200640 1 -- 172.21.15.28:0/2291922687 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd95819f4a0 con 0x7fd9581098e0 2024-05-12T22:19:04.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.465+0000 7fd94ea00640 1 -- 172.21.15.28:0/2291922687 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd950010760 con 0x7fd9581098e0 2024-05-12T22:19:04.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.465+0000 7fd957200640 1 -- 172.21.15.28:0/2291922687 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9581a6d30 con 0x7fd9581098e0 2024-05-12T22:19:04.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.468+0000 7fd95ffcd740 1 Processor -- start 2024-05-12T22:19:04.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.468+0000 7fd94ea00640 1 -- 172.21.15.28:0/2291922687 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7fd950005020 con 0x7fd9581098e0 2024-05-12T22:19:04.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.468+0000 7fd95ffcd740 1 -- start start 2024-05-12T22:19:04.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.468+0000 7fd95ffcd740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cbfaac2b60 0x55cbfaac2f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:04.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.468+0000 7fd95ffcd740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cbfaac3480 0x55cbfaacd610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:04.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.468+0000 7fd95ffcd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbfaacdb50 0x55cbfaacff40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:04.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.468+0000 7fd95ffcd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55cbfaa27a30 con 0x55cbfaacdb50 2024-05-12T22:19:04.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.468+0000 7fd95ffcd740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x55cbfa9cf700 con 0x55cbfaac2b60 2024-05-12T22:19:04.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.468+0000 7fd95ffcd740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55cbfaa239f0 con 0x55cbfaac3480 2024-05-12T22:19:04.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.469+0000 7fd955e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cbfaac3480 0x55cbfaacd610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:04.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.469+0000 7fd955e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cbfaac3480 0x55cbfaacd610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:46378/0 (socket says 172.21.15.28:46378) 2024-05-12T22:19:04.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.469+0000 7fd955e00640 1 -- 172.21.15.28:0/2350838757 learned_addr learned my addr 172.21.15.28:0/2350838757 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:19:04.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.469+0000 7fd955400640 1 --2- 172.21.15.28:0/2350838757 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cbfaac2b60 0x55cbfaac2f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:04.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.469+0000 7fd955400640 1 --2- 172.21.15.28:0/2350838757 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbfaacdb50 0x55cbfaacff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:04.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.469+0000 7fd955e00640 1 -- 172.21.15.28:0/2350838757 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cbfaac2b60 msgr2=0x55cbfaac2f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:19:04.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.469+0000 7fd955e00640 1 --2- 172.21.15.28:0/2350838757 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cbfaac2b60 0x55cbfaac2f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:04.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.469+0000 7fd955e00640 1 -- 172.21.15.28:0/2350838757 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbfaacdb50 msgr2=0x55cbfaacff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:19:04.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.469+0000 7fd955e00640 1 --2- 172.21.15.28:0/2350838757 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbfaacdb50 0x55cbfaacff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:04.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.469+0000 7fd955e00640 1 -- 172.21.15.28:0/2350838757 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cbfa73b3f0 con 0x55cbfaac3480 2024-05-12T22:19:04.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.469+0000 7fd955400640 1 --2- 172.21.15.28:0/2350838757 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbfaacdb50 0x55cbfaacff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:19:04.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.470+0000 7fd955e00640 1 --2- 172.21.15.28:0/2350838757 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cbfaac3480 0x55cbfaacd610 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fd948009930 tx=0x7fd948005de0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=66156df2f976d233 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:19:04.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.470+0000 7fd94ea00640 1 --2- 172.21.15.28:0/2291922687 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fd93c0a2d80 0x7fd93c0a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:04.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.470+0000 7fd94ea00640 1 -- 172.21.15.28:0/2291922687 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8513+0+0 (secure 0 0 0) 0x7fd9500cf940 con 0x7fd9581098e0 2024-05-12T22:19:04.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.470+0000 7fd955400640 1 --2- 172.21.15.28:0/2291922687 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fd93c0a2d80 0x7fd93c0a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:04.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.470+0000 7fd943400640 1 -- 172.21.15.28:0/2350838757 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd948035040 con 0x55cbfaac3480 2024-05-12T22:19:04.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.470+0000 7fd943400640 1 -- 172.21.15.28:0/2350838757 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd948005520 con 0x55cbfaac3480 2024-05-12T22:19:04.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.470+0000 7fd943400640 1 -- 172.21.15.28:0/2350838757 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9480035f0 con 0x55cbfaac3480 2024-05-12T22:19:04.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.471+0000 7fd95ffcd740 1 -- 172.21.15.28:0/2350838757 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cbfaac3480 msgr2=0x55cbfaacd610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:04.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.471+0000 7fd955400640 1 --2- 172.21.15.28:0/2291922687 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fd93c0a2d80 0x7fd93c0a5240 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x55cbfaac0010 tx=0x7fd944013040 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:19:04.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.471+0000 7fd95ffcd740 1 --2- 172.21.15.28:0/2350838757 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cbfaac3480 0x55cbfaacd610 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fd948009930 tx=0x7fd948005de0 comp rx=0 tx=0).stop 2024-05-12T22:19:04.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.471+0000 7fd95ffcd740 1 -- 172.21.15.28:0/2350838757 shutdown_connections 2024-05-12T22:19:04.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.471+0000 7fd95ffcd740 1 --2- 172.21.15.28:0/2350838757 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbfaacdb50 0x55cbfaacff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:04.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.471+0000 7fd95ffcd740 1 --2- 172.21.15.28:0/2350838757 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cbfaac3480 0x55cbfaacd610 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:04.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.471+0000 7fd95ffcd740 1 --2- 172.21.15.28:0/2350838757 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cbfaac2b60 0x55cbfaac2f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:04.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.471+0000 7fd95ffcd740 1 -- 172.21.15.28:0/2350838757 >> 172.21.15.28:0/2350838757 conn(0x55cbfaabee80 msgr2=0x55cbfaabf260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:19:04.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.471+0000 7fd95ffcd740 1 -- 172.21.15.28:0/2350838757 shutdown_connections 2024-05-12T22:19:04.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.471+0000 7fd95ffcd740 1 -- 172.21.15.28:0/2350838757 wait complete. 2024-05-12T22:19:04.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.472+0000 7fd95ffcd740 1 -- start start 2024-05-12T22:19:04.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.472+0000 7fd95ffcd740 1 Processor -- start 2024-05-12T22:19:04.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.472+0000 7fd95ffcd740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cbfaac2b60 0x55cbfaacaac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:04.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.472+0000 7fd95ffcd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbfaac3480 0x55cbfaacb000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:04.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.472+0000 7fd95ffcd740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cbfaacdb50 0x55cbfaacb540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:04.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.472+0000 7fd95ffcd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55cbfaad08e0 con 0x55cbfaac3480 2024-05-12T22:19:04.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.472+0000 7fd95ffcd740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x55cbfa9e2a70 con 0x55cbfaac2b60 2024-05-12T22:19:04.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.472+0000 7fd95ffcd740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55cbfaad0a60 con 0x55cbfaacdb50 2024-05-12T22:19:04.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.473+0000 7fd95ffcd740 10 client.0 _ll_register_callbacks cb 0x55cbfa9c5ba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:19:04.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.473+0000 7fd955400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbfaac3480 0x55cbfaacb000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:04.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.473+0000 7fd955e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cbfaac2b60 0x55cbfaacaac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:04.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.473+0000 7fd955e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cbfaac2b60 0x55cbfaacaac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:34334/0 (socket says 172.21.15.28:34334) 2024-05-12T22:19:04.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.473+0000 7fd955e00640 1 -- 172.21.15.28:0/1837179137 learned_addr learned my addr 172.21.15.28:0/1837179137 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:19:04.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.473+0000 7fd955400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbfaac3480 0x55cbfaacb000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33020/0 (socket says 172.21.15.28:33020) 2024-05-12T22:19:04.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.473+0000 7fd956800640 1 --2- 172.21.15.28:0/1837179137 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cbfaacdb50 0x55cbfaacb540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:04.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.473+0000 7fd955e00640 1 -- 172.21.15.28:0/1837179137 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cbfaacdb50 msgr2=0x55cbfaacb540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:19:04.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.473+0000 7fd955e00640 1 --2- 172.21.15.28:0/1837179137 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cbfaacdb50 0x55cbfaacb540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:04.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.473+0000 7fd955e00640 1 -- 172.21.15.28:0/1837179137 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbfaac3480 msgr2=0x55cbfaacb000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:19:04.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.473+0000 7fd955e00640 1 --2- 172.21.15.28:0/1837179137 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbfaac3480 0x55cbfaacb000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:04.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.473+0000 7fd955e00640 1 -- 172.21.15.28:0/1837179137 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cbfa8057d0 con 0x55cbfaac2b60 2024-05-12T22:19:04.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.474+0000 7fd955e00640 1 --2- 172.21.15.28:0/1837179137 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cbfaac2b60 0x55cbfaacaac0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fd9480048f0 tx=0x7fd948005b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=847c1e52cf674ec9 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:19:04.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.474+0000 7fd937e00640 10 client.0 ms_handle_connect on v2:172.21.15.86:3300/0 2024-05-12T22:19:04.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.474+0000 7fd937e00640 1 -- 172.21.15.28:0/1837179137 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd948052060 con 0x55cbfaac2b60 2024-05-12T22:19:04.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.474+0000 7fd937e00640 1 -- 172.21.15.28:0/1837179137 <== mon.2 v2:172.21.15.86:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8513+0+0 (secure 0 0 0) 0x7fd948004390 con 0x55cbfaac2b60 2024-05-12T22:19:04.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.474+0000 7fd95ffcd740 1 -- 172.21.15.28:0/1837179137 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd948005520 con 0x55cbfaac2b60 2024-05-12T22:19:04.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.474+0000 7fd937e00640 1 -- 172.21.15.28:0/1837179137 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fd948035070 con 0x55cbfaac2b60 2024-05-12T22:19:04.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.474+0000 7fd95ffcd740 10 client.27401 fetch_fsmap learned FSMap version 26 2024-05-12T22:19:04.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.474+0000 7fd95ffcd740 1 -- 172.21.15.28:0/1837179137 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55cbfaadcbc0 con 0x55cbfaac2b60 2024-05-12T22:19:04.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.474+0000 7fd937e00640 1 -- 172.21.15.28:0/1837179137 <== mon.2 v2:172.21.15.86:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd948005520 con 0x55cbfaac2b60 2024-05-12T22:19:04.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.475+0000 7fd937e00640 1 -- 172.21.15.28:0/1837179137 <== mon.2 v2:172.21.15.86:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fd9480319c0 con 0x55cbfaac2b60 2024-05-12T22:19:04.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.475+0000 7fd95ffcd740 10 client.27401 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:19:04.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.475+0000 7fd95ffcd740 10 client.27401 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.g 2024-05-12T22:19:04.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.475+0000 7fd95ffcd740 20 client.27401 populate_metadata read hostname 'smithi028' 2024-05-12T22:19:04.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.475+0000 7fd95ffcd740 1 --2- 172.21.15.28:0/1837179137 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x55cbfaaddf10 0x55cbfaae0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:04.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.475+0000 7fd95ffcd740 4 client.27401 mds_command: new command op to 24532 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-12T22:19:04.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.475+0000 7fd95ffcd740 1 -- 172.21.15.28:0/1837179137 --> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cbfaa35160 con 0x55cbfaaddf10 2024-05-12T22:19:04.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.475+0000 7fd955400640 1 --2- 172.21.15.28:0/1837179137 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x55cbfaaddf10 0x55cbfaae0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:04.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.476+0000 7fd955400640 1 --2- 172.21.15.28:0/1837179137 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x55cbfaaddf10 0x55cbfaae0300 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=70f2bc545c8b6a07 server_cookie=22f73ce51fa1705c in_seq=0 out_seq=0 2024-05-12T22:19:04.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.476+0000 7fd937e00640 10 client.27401 ms_handle_connect on v2:172.21.15.28:6838/123892158 2024-05-12T22:19:04.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.476+0000 7fd937e00640 1 -- 172.21.15.28:0/1837179137 <== mds.2 v2:172.21.15.28:6838/123892158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55cbfaa35160 con 0x55cbfaaddf10 2024-05-12T22:19:04.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.476+0000 7fd937e00640 10 client.27401 handle_command_reply: tid=0 2024-05-12T22:19:04.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.476+0000 7fd95ffcd740 1 client.27401 shutdown 2024-05-12T22:19:04.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.477+0000 7fd95ffcd740 1 -- 172.21.15.28:0/1837179137 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cbfaac2b60 msgr2=0x55cbfaacaac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:04.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.477+0000 7fd95ffcd740 1 --2- 172.21.15.28:0/1837179137 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cbfaac2b60 0x55cbfaacaac0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fd9480048f0 tx=0x7fd948005b50 comp rx=0 tx=0).stop 2024-05-12T22:19:04.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.477+0000 7fd95ffcd740 1 -- 172.21.15.28:0/1837179137 shutdown_connections 2024-05-12T22:19:04.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.477+0000 7fd95ffcd740 1 --2- 172.21.15.28:0/1837179137 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cbfaacdb50 0x55cbfaacb540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:04.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.477+0000 7fd95ffcd740 1 --2- 172.21.15.28:0/1837179137 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbfaac3480 0x55cbfaacb000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:04.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.477+0000 7fd95ffcd740 1 --2- 172.21.15.28:0/1837179137 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x55cbfaaddf10 0x55cbfaae0300 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:04.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.477+0000 7fd95ffcd740 1 --2- 172.21.15.28:0/1837179137 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cbfaac2b60 0x55cbfaacaac0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:04.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.477+0000 7fd95ffcd740 1 -- 172.21.15.28:0/1837179137 >> 172.21.15.28:0/1837179137 conn(0x55cbfaabee80 msgr2=0x55cbfaac4c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:19:04.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.477+0000 7fd937e00640 0 client.27401 ms_handle_reset on v2:172.21.15.28:6838/123892158 2024-05-12T22:19:04.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.477+0000 7fd95ffcd740 1 -- 172.21.15.28:0/1837179137 shutdown_connections 2024-05-12T22:19:04.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.477+0000 7fd95ffcd740 1 -- 172.21.15.28:0/1837179137 wait complete. 2024-05-12T22:19:04.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.477+0000 7fd95ffcd740 20 client.27401 trim_cache size 0 max 16384 2024-05-12T22:19:04.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.492+0000 7fd95ffcd740 1 Processor -- start 2024-05-12T22:19:04.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.492+0000 7fd95ffcd740 1 -- start start 2024-05-12T22:19:04.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.492+0000 7fd95ffcd740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cbfaac2b60 0x55cbfaad9420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:04.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.492+0000 7fd95ffcd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbfaac3480 0x55cbfaae3320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:04.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.492+0000 7fd95ffcd740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cbfaacdb50 0x55cbfaae3860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:04.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.492+0000 7fd95ffcd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd9581b1e20 con 0x55cbfaac3480 2024-05-12T22:19:04.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.492+0000 7fd95ffcd740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fd9581af6f0 con 0x55cbfaacdb50 2024-05-12T22:19:04.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.492+0000 7fd95ffcd740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd9581ad560 con 0x55cbfaac2b60 2024-05-12T22:19:04.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.493+0000 7fd955400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbfaac3480 0x55cbfaae3320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:04.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.493+0000 7fd955400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbfaac3480 0x55cbfaae3320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33030/0 (socket says 172.21.15.28:33030) 2024-05-12T22:19:04.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.493+0000 7fd955400640 1 -- 172.21.15.28:0/497502062 learned_addr learned my addr 172.21.15.28:0/497502062 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:19:04.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.493+0000 7fd956800640 1 --2- 172.21.15.28:0/497502062 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cbfaacdb50 0x55cbfaae3860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:04.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.493+0000 7fd955e00640 1 --2- 172.21.15.28:0/497502062 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cbfaac2b60 0x55cbfaad9420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:04.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.493+0000 7fd955400640 1 -- 172.21.15.28:0/497502062 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cbfaac2b60 msgr2=0x55cbfaad9420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:19:04.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.493+0000 7fd955400640 1 --2- 172.21.15.28:0/497502062 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cbfaac2b60 0x55cbfaad9420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:04.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.493+0000 7fd955400640 1 -- 172.21.15.28:0/497502062 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cbfaacdb50 msgr2=0x55cbfaae3860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:19:04.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.493+0000 7fd955400640 1 --2- 172.21.15.28:0/497502062 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cbfaacdb50 0x55cbfaae3860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:04.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.493+0000 7fd955400640 1 -- 172.21.15.28:0/497502062 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cbfaae6140 con 0x55cbfaac3480 2024-05-12T22:19:04.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.493+0000 7fd956800640 1 --2- 172.21.15.28:0/497502062 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cbfaacdb50 0x55cbfaae3860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:19:04.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.494+0000 7fd955400640 1 --2- 172.21.15.28:0/497502062 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbfaac3480 0x55cbfaae3320 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fd9440118c0 tx=0x7fd944005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38ca5f98796b6922 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:19:04.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.494+0000 7fd942a00640 1 -- 172.21.15.28:0/497502062 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd94402d470 con 0x55cbfaac3480 2024-05-12T22:19:04.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.494+0000 7fd942a00640 1 -- 172.21.15.28:0/497502062 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd944021d80 con 0x55cbfaac3480 2024-05-12T22:19:04.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.494+0000 7fd942a00640 1 -- 172.21.15.28:0/497502062 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd944005410 con 0x55cbfaac3480 2024-05-12T22:19:04.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.495+0000 7fd95ffcd740 1 -- 172.21.15.28:0/497502062 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbfaac3480 msgr2=0x55cbfaae3320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:04.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.495+0000 7fd95ffcd740 1 --2- 172.21.15.28:0/497502062 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbfaac3480 0x55cbfaae3320 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fd9440118c0 tx=0x7fd944005850 comp rx=0 tx=0).stop 2024-05-12T22:19:04.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.495+0000 7fd95ffcd740 1 -- 172.21.15.28:0/497502062 shutdown_connections 2024-05-12T22:19:04.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.495+0000 7fd95ffcd740 1 --2- 172.21.15.28:0/497502062 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cbfaacdb50 0x55cbfaae3860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:04.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.495+0000 7fd95ffcd740 1 --2- 172.21.15.28:0/497502062 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbfaac3480 0x55cbfaae3320 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:04.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.495+0000 7fd95ffcd740 1 --2- 172.21.15.28:0/497502062 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cbfaac2b60 0x55cbfaad9420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:04.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.495+0000 7fd95ffcd740 1 -- 172.21.15.28:0/497502062 >> 172.21.15.28:0/497502062 conn(0x55cbfaabee80 msgr2=0x55cbfaad8d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:19:04.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.495+0000 7fd95ffcd740 1 -- 172.21.15.28:0/497502062 shutdown_connections 2024-05-12T22:19:04.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.495+0000 7fd95ffcd740 1 -- 172.21.15.28:0/497502062 wait complete. 2024-05-12T22:19:04.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.495+0000 7fd95ffcd740 1 -- start start 2024-05-12T22:19:04.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.495+0000 7fd95ffcd740 1 Processor -- start 2024-05-12T22:19:04.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.496+0000 7fd95ffcd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbfaac2b60 0x55cbfaae0e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:04.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.496+0000 7fd95ffcd740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cbfaac3480 0x55cbfaae1370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:04.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.496+0000 7fd95ffcd740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cbfaacdb50 0x55cbfaae18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:04.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.496+0000 7fd95ffcd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd9581adbb0 con 0x55cbfaac2b60 2024-05-12T22:19:04.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.496+0000 7fd95ffcd740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fd9581af020 con 0x55cbfaacdb50 2024-05-12T22:19:04.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.496+0000 7fd95ffcd740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd9581ae280 con 0x55cbfaac3480 2024-05-12T22:19:04.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.496+0000 7fd95ffcd740 10 client.0 _ll_register_callbacks cb 0x7fd9480056d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:19:04.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.496+0000 7fd955e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbfaac2b60 0x55cbfaae0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:04.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.496+0000 7fd955e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbfaac2b60 0x55cbfaae0e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33040/0 (socket says 172.21.15.28:33040) 2024-05-12T22:19:04.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.496+0000 7fd955e00640 1 -- 172.21.15.28:0/118572679 learned_addr learned my addr 172.21.15.28:0/118572679 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:19:04.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.496+0000 7fd955400640 1 --2- 172.21.15.28:0/118572679 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cbfaac3480 0x55cbfaae1370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:04.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.496+0000 7fd956800640 1 --2- 172.21.15.28:0/118572679 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cbfaacdb50 0x55cbfaae18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:04.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.497+0000 7fd955e00640 1 -- 172.21.15.28:0/118572679 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cbfaac3480 msgr2=0x55cbfaae1370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:19:04.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.497+0000 7fd955e00640 1 --2- 172.21.15.28:0/118572679 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cbfaac3480 0x55cbfaae1370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:04.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.497+0000 7fd955e00640 1 -- 172.21.15.28:0/118572679 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cbfaacdb50 msgr2=0x55cbfaae18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:19:04.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.497+0000 7fd955e00640 1 --2- 172.21.15.28:0/118572679 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cbfaacdb50 0x55cbfaae18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:04.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.497+0000 7fd955e00640 1 -- 172.21.15.28:0/118572679 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cbfab43220 con 0x55cbfaac2b60 2024-05-12T22:19:04.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.497+0000 7fd956800640 1 --2- 172.21.15.28:0/118572679 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cbfaacdb50 0x55cbfaae18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:19:04.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.497+0000 7fd955e00640 1 --2- 172.21.15.28:0/118572679 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbfaac2b60 0x55cbfaae0e30 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fd9480048f0 tx=0x7fd948005de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffaed3695ebbf166 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:19:04.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.497+0000 7fd937e00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-12T22:19:04.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.497+0000 7fd937e00640 1 -- 172.21.15.28:0/118572679 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd948005520 con 0x55cbfaac2b60 2024-05-12T22:19:04.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.497+0000 7fd937e00640 1 -- 172.21.15.28:0/118572679 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8513+0+0 (secure 0 0 0) 0x7fd9480035f0 con 0x55cbfaac2b60 2024-05-12T22:19:04.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.497+0000 7fd95ffcd740 1 -- 172.21.15.28:0/118572679 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd944021d80 con 0x55cbfaac2b60 2024-05-12T22:19:04.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.497+0000 7fd937e00640 1 -- 172.21.15.28:0/118572679 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fd948009520 con 0x55cbfaac2b60 2024-05-12T22:19:04.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.497+0000 7fd95ffcd740 10 client.17271 fetch_fsmap learned FSMap version 26 2024-05-12T22:19:04.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.497+0000 7fd95ffcd740 1 -- 172.21.15.28:0/118572679 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55cbfab44500 con 0x55cbfaac2b60 2024-05-12T22:19:04.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.498+0000 7fd937e00640 1 -- 172.21.15.28:0/118572679 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd944021d80 con 0x55cbfaac2b60 2024-05-12T22:19:04.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.498+0000 7fd937e00640 1 -- 172.21.15.28:0/118572679 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fd94804e850 con 0x55cbfaac2b60 2024-05-12T22:19:04.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.498+0000 7fd95ffcd740 10 client.17271 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:19:04.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.498+0000 7fd95ffcd740 10 client.17271 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.g 2024-05-12T22:19:04.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.498+0000 7fd95ffcd740 20 client.17271 populate_metadata read hostname 'smithi028' 2024-05-12T22:19:04.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.498+0000 7fd95ffcd740 1 --2- 172.21.15.28:0/118572679 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x55cbfab44c20 0x55cbfab47040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:04.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.498+0000 7fd95ffcd740 4 client.17271 mds_command: new command op to 24532 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-12T22:19:04.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.498+0000 7fd95ffcd740 1 -- 172.21.15.28:0/118572679 --> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55cbfaa2f4b0 con 0x55cbfab44c20 2024-05-12T22:19:04.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.498+0000 7fd955400640 1 --2- 172.21.15.28:0/118572679 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x55cbfab44c20 0x55cbfab47040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:04.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.499+0000 7fd955400640 1 --2- 172.21.15.28:0/118572679 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x55cbfab44c20 0x55cbfab47040 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c78bb65d2cb7ab0e server_cookie=50bc43ea2d2db90d in_seq=0 out_seq=0 2024-05-12T22:19:04.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.499+0000 7fd937e00640 10 client.17271 ms_handle_connect on v2:172.21.15.28:6838/123892158 2024-05-12T22:19:04.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.499+0000 7fd937e00640 1 -- 172.21.15.28:0/118572679 <== mds.2 v2:172.21.15.28:6838/123892158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55cbfaa2f4b0 con 0x55cbfab44c20 2024-05-12T22:19:04.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.499+0000 7fd937e00640 10 client.17271 handle_command_reply: tid=0 2024-05-12T22:19:04.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.499+0000 7fd95ffcd740 1 client.17271 shutdown 2024-05-12T22:19:04.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.499+0000 7fd95ffcd740 1 -- 172.21.15.28:0/118572679 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbfaac2b60 msgr2=0x55cbfaae0e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:04.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.499+0000 7fd95ffcd740 1 --2- 172.21.15.28:0/118572679 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbfaac2b60 0x55cbfaae0e30 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fd9480048f0 tx=0x7fd948005de0 comp rx=0 tx=0).stop 2024-05-12T22:19:04.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.499+0000 7fd95ffcd740 1 -- 172.21.15.28:0/118572679 shutdown_connections 2024-05-12T22:19:04.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.499+0000 7fd95ffcd740 1 --2- 172.21.15.28:0/118572679 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x55cbfab44c20 0x55cbfab47040 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:04.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.499+0000 7fd95ffcd740 1 --2- 172.21.15.28:0/118572679 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cbfaacdb50 0x55cbfaae18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:04.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.499+0000 7fd95ffcd740 1 --2- 172.21.15.28:0/118572679 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cbfaac3480 0x55cbfaae1370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:04.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.499+0000 7fd95ffcd740 1 --2- 172.21.15.28:0/118572679 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbfaac2b60 0x55cbfaae0e30 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:04.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.499+0000 7fd937e00640 0 client.17271 ms_handle_reset on v2:172.21.15.28:6838/123892158 2024-05-12T22:19:04.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.499+0000 7fd95ffcd740 1 -- 172.21.15.28:0/118572679 >> 172.21.15.28:0/118572679 conn(0x55cbfaabee80 msgr2=0x55cbfaad95a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:19:04.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.500+0000 7fd95ffcd740 1 -- 172.21.15.28:0/118572679 shutdown_connections 2024-05-12T22:19:04.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.500+0000 7fd95ffcd740 1 -- 172.21.15.28:0/118572679 wait complete. 2024-05-12T22:19:04.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.500+0000 7fd95ffcd740 20 client.17271 trim_cache size 0 max 16384 2024-05-12T22:19:04.508 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-05-12T22:19:04.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.500+0000 7fd957200640 1 -- 172.21.15.28:0/2291922687 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fd93c0a2d80 msgr2=0x7fd93c0a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:04.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.500+0000 7fd957200640 1 --2- 172.21.15.28:0/2291922687 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fd93c0a2d80 0x7fd93c0a5240 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x55cbfaac0010 tx=0x7fd944013040 comp rx=0 tx=0).stop 2024-05-12T22:19:04.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.500+0000 7fd957200640 1 -- 172.21.15.28:0/2291922687 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9581098e0 msgr2=0x7fd95819ea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:04.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.500+0000 7fd957200640 1 --2- 172.21.15.28:0/2291922687 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9581098e0 0x7fd95819ea80 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fd95000eb10 tx=0x7fd95000c2d0 comp rx=0 tx=0).stop 2024-05-12T22:19:04.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.500+0000 7fd957200640 1 -- 172.21.15.28:0/2291922687 shutdown_connections 2024-05-12T22:19:04.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.500+0000 7fd957200640 1 --2- 172.21.15.28:0/2291922687 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fd93c0a2d80 0x7fd93c0a5240 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:04.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.500+0000 7fd957200640 1 --2- 172.21.15.28:0/2291922687 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9581098e0 0x7fd95819ea80 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:04.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.500+0000 7fd957200640 1 --2- 172.21.15.28:0/2291922687 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd958104290 0x7fd95819a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:04.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.500+0000 7fd957200640 1 --2- 172.21.15.28:0/2291922687 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd9581038d0 0x7fd95819a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:04.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.501+0000 7fd957200640 1 -- 172.21.15.28:0/2291922687 >> 172.21.15.28:0/2291922687 conn(0x7fd9580ff360 msgr2=0x7fd9580ffb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:19:04.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.501+0000 7fd957200640 1 -- 172.21.15.28:0/2291922687 shutdown_connections 2024-05-12T22:19:04.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.501+0000 7fd957200640 1 -- 172.21.15.28:0/2291922687 wait complete. 2024-05-12T22:19:04.515 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:19:04 smithi028 ceph-mon[29847]: pgmap v311: 97 pgs: 97 active+clean; 91 MiB data, 725 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:19:04.516 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:19:04 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/2535402623' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:19:04.518 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-05-12T22:19:04.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:19:04 smithi186 ceph-mon[34785]: pgmap v311: 97 pgs: 97 active+clean; 91 MiB data, 725 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:19:04.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:19:04 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/2535402623' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:19:04.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.734+0000 7fc372c00640 1 -- 172.21.15.28:0/4075437493 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc36c074580 msgr2=0x7fc36c10d9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:04.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.734+0000 7fc372c00640 1 --2- 172.21.15.28:0/4075437493 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc36c074580 0x7fc36c10d9d0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fc358009900 tx=0x7fc35802a490 comp rx=0 tx=0).stop 2024-05-12T22:19:04.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.734+0000 7fc372c00640 1 -- 172.21.15.28:0/4075437493 shutdown_connections 2024-05-12T22:19:04.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.734+0000 7fc372c00640 1 --2- 172.21.15.28:0/4075437493 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc36c10df10 0x7fc36c110360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:04.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.734+0000 7fc372c00640 1 --2- 172.21.15.28:0/4075437493 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc36c074580 0x7fc36c10d9d0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:04.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.734+0000 7fc372c00640 1 --2- 172.21.15.28:0/4075437493 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc36c073c60 0x7fc36c074040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:04.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.734+0000 7fc372c00640 1 -- 172.21.15.28:0/4075437493 >> 172.21.15.28:0/4075437493 conn(0x7fc36c0fd200 msgr2=0x7fc36c0ff620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:19:04.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.734+0000 7fc372c00640 1 -- 172.21.15.28:0/4075437493 shutdown_connections 2024-05-12T22:19:04.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.734+0000 7fc372c00640 1 -- 172.21.15.28:0/4075437493 wait complete. 2024-05-12T22:19:04.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.735+0000 7fc372c00640 1 Processor -- start 2024-05-12T22:19:04.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.736+0000 7fc372c00640 1 -- start start 2024-05-12T22:19:04.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.736+0000 7fc372c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc36c073c60 0x7fc36c19a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:04.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.736+0000 7fc372c00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc36c074580 0x7fc36c19a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:04.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.736+0000 7fc372c00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc36c10df10 0x7fc36c19ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:04.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.736+0000 7fc372c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc36c113d00 con 0x7fc36c073c60 2024-05-12T22:19:04.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.736+0000 7fc372c00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fc36c113b80 con 0x7fc36c10df10 2024-05-12T22:19:04.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.736+0000 7fc372c00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc36c113e80 con 0x7fc36c074580 2024-05-12T22:19:04.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.737+0000 7fc371800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc36c073c60 0x7fc36c19a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:04.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.737+0000 7fc371800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc36c073c60 0x7fc36c19a260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33062/0 (socket says 172.21.15.28:33062) 2024-05-12T22:19:04.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.737+0000 7fc371800640 1 -- 172.21.15.28:0/4212422723 learned_addr learned my addr 172.21.15.28:0/4212422723 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:19:04.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.737+0000 7fc370e00640 1 --2- 172.21.15.28:0/4212422723 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc36c074580 0x7fc36c19a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:04.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.737+0000 7fc372200640 1 --2- 172.21.15.28:0/4212422723 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc36c10df10 0x7fc36c19ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:04.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.737+0000 7fc371800640 1 -- 172.21.15.28:0/4212422723 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc36c074580 msgr2=0x7fc36c19a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:04.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.737+0000 7fc371800640 1 --2- 172.21.15.28:0/4212422723 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc36c074580 0x7fc36c19a7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:04.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.737+0000 7fc371800640 1 -- 172.21.15.28:0/4212422723 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc36c10df10 msgr2=0x7fc36c19ead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:04.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.737+0000 7fc371800640 1 --2- 172.21.15.28:0/4212422723 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc36c10df10 0x7fc36c19ead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:04.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.737+0000 7fc371800640 1 -- 172.21.15.28:0/4212422723 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc36c19f200 con 0x7fc36c073c60 2024-05-12T22:19:04.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.737+0000 7fc372200640 1 --2- 172.21.15.28:0/4212422723 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc36c10df10 0x7fc36c19ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:19:04.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.737+0000 7fc371800640 1 --2- 172.21.15.28:0/4212422723 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc36c073c60 0x7fc36c19a260 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fc36000d950 tx=0x7fc36000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:19:04.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.738+0000 7fc36a000640 1 -- 172.21.15.28:0/4212422723 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc360017070 con 0x7fc36c073c60 2024-05-12T22:19:04.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.738+0000 7fc36a000640 1 -- 172.21.15.28:0/4212422723 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc360009c30 con 0x7fc36c073c60 2024-05-12T22:19:04.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.738+0000 7fc36a000640 1 -- 172.21.15.28:0/4212422723 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc36000f6b0 con 0x7fc36c073c60 2024-05-12T22:19:04.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.738+0000 7fc372c00640 1 -- 172.21.15.28:0/4212422723 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc36c19f4f0 con 0x7fc36c073c60 2024-05-12T22:19:04.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.738+0000 7fc372c00640 1 -- 172.21.15.28:0/4212422723 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc36c1a6d80 con 0x7fc36c073c60 2024-05-12T22:19:04.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.741+0000 7fc377025740 1 Processor -- start 2024-05-12T22:19:04.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.742+0000 7fc377025740 1 -- start start 2024-05-12T22:19:04.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.742+0000 7fc377025740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56430cb9eb60 0x56430cb9ef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:04.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.742+0000 7fc377025740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56430cb9f480 0x56430cba9610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:04.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.742+0000 7fc377025740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56430cba9b50 0x56430cbabf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:04.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.742+0000 7fc377025740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56430cb03a30 con 0x56430cb9eb60 2024-05-12T22:19:04.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.742+0000 7fc377025740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x56430caab700 con 0x56430cba9b50 2024-05-12T22:19:04.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.742+0000 7fc377025740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x56430caff9f0 con 0x56430cb9f480 2024-05-12T22:19:04.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.742+0000 7fc36a000640 1 -- 172.21.15.28:0/4212422723 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7fc3600109d0 con 0x7fc36c073c60 2024-05-12T22:19:04.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.742+0000 7fc372200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56430cb9eb60 0x56430cb9ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:04.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.742+0000 7fc372200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56430cb9eb60 0x56430cb9ef40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33066/0 (socket says 172.21.15.28:33066) 2024-05-12T22:19:04.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.742+0000 7fc372200640 1 -- 172.21.15.28:0/1802549636 learned_addr learned my addr 172.21.15.28:0/1802549636 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:19:04.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.742+0000 7fc371800640 1 --2- 172.21.15.28:0/1802549636 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56430cb9f480 0x56430cba9610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:04.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.742+0000 7fc370e00640 1 --2- 172.21.15.28:0/1802549636 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56430cba9b50 0x56430cbabf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:04.744 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:19:04 smithi086 ceph-mon[34297]: pgmap v311: 97 pgs: 97 active+clean; 91 MiB data, 725 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:19:04.744 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:19:04 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/2535402623' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:19:04.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.743+0000 7fc372200640 1 -- 172.21.15.28:0/1802549636 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56430cb9f480 msgr2=0x56430cba9610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:19:04.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.743+0000 7fc372200640 1 --2- 172.21.15.28:0/1802549636 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56430cb9f480 0x56430cba9610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:04.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.743+0000 7fc372200640 1 -- 172.21.15.28:0/1802549636 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56430cba9b50 msgr2=0x56430cbabf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:19:04.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.743+0000 7fc372200640 1 --2- 172.21.15.28:0/1802549636 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56430cba9b50 0x56430cbabf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:04.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.743+0000 7fc372200640 1 -- 172.21.15.28:0/1802549636 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56430c8173f0 con 0x56430cb9eb60 2024-05-12T22:19:04.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.743+0000 7fc372200640 1 --2- 172.21.15.28:0/1802549636 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56430cb9eb60 0x56430cb9ef40 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fc364009f30 tx=0x7fc364010800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=287d88eef15c8eeb server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:19:04.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.743+0000 7fc35ea00640 1 -- 172.21.15.28:0/1802549636 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc364016070 con 0x56430cb9eb60 2024-05-12T22:19:04.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.744+0000 7fc35ea00640 1 -- 172.21.15.28:0/1802549636 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc36401bb40 con 0x56430cb9eb60 2024-05-12T22:19:04.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.744+0000 7fc35ea00640 1 -- 172.21.15.28:0/1802549636 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc364012c50 con 0x56430cb9eb60 2024-05-12T22:19:04.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.744+0000 7fc36a000640 1 --2- 172.21.15.28:0/4212422723 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fc3540a2d80 0x7fc3540a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:04.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.744+0000 7fc36a000640 1 -- 172.21.15.28:0/4212422723 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8513+0+0 (secure 0 0 0) 0x7fc3600cefe0 con 0x7fc36c073c60 2024-05-12T22:19:04.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.744+0000 7fc371800640 1 --2- 172.21.15.28:0/4212422723 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fc3540a2d80 0x7fc3540a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:04.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.744+0000 7fc377025740 1 -- 172.21.15.28:0/1802549636 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56430cb9eb60 msgr2=0x56430cb9ef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:04.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.744+0000 7fc377025740 1 --2- 172.21.15.28:0/1802549636 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56430cb9eb60 0x56430cb9ef40 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fc364009f30 tx=0x7fc364010800 comp rx=0 tx=0).stop 2024-05-12T22:19:04.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.745+0000 7fc377025740 1 -- 172.21.15.28:0/1802549636 shutdown_connections 2024-05-12T22:19:04.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.745+0000 7fc377025740 1 --2- 172.21.15.28:0/1802549636 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56430cba9b50 0x56430cbabf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:04.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.745+0000 7fc377025740 1 --2- 172.21.15.28:0/1802549636 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56430cb9f480 0x56430cba9610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:04.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.745+0000 7fc377025740 1 --2- 172.21.15.28:0/1802549636 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56430cb9eb60 0x56430cb9ef40 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:04.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.745+0000 7fc377025740 1 -- 172.21.15.28:0/1802549636 >> 172.21.15.28:0/1802549636 conn(0x56430cb9ae80 msgr2=0x56430cb9b260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:19:04.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.745+0000 7fc377025740 1 -- 172.21.15.28:0/1802549636 shutdown_connections 2024-05-12T22:19:04.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.745+0000 7fc377025740 1 -- 172.21.15.28:0/1802549636 wait complete. 2024-05-12T22:19:04.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.745+0000 7fc371800640 1 --2- 172.21.15.28:0/4212422723 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fc3540a2d80 0x7fc3540a5240 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fc3600cc7e0 tx=0x7fc3600d2b30 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:19:04.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.745+0000 7fc377025740 1 -- start start 2024-05-12T22:19:04.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.746+0000 7fc377025740 1 Processor -- start 2024-05-12T22:19:04.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.746+0000 7fc377025740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56430cb9eb60 0x56430cba6ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:04.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.746+0000 7fc377025740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56430cb9f480 0x56430cba7010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:04.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.746+0000 7fc377025740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56430cba9b50 0x56430cba7550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:04.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.746+0000 7fc377025740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56430cbac940 con 0x56430cba9b50 2024-05-12T22:19:04.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.746+0000 7fc377025740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x56430cabea70 con 0x56430cb9eb60 2024-05-12T22:19:04.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.746+0000 7fc377025740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x56430cbacac0 con 0x56430cb9f480 2024-05-12T22:19:04.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.746+0000 7fc377025740 10 client.0 _ll_register_callbacks cb 0x56430caa1ba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:19:04.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.747+0000 7fc370e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56430cb9f480 0x56430cba7010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:04.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.747+0000 7fc370e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56430cb9f480 0x56430cba7010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:46442/0 (socket says 172.21.15.28:46442) 2024-05-12T22:19:04.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.747+0000 7fc370e00640 1 -- 172.21.15.28:0/302066229 learned_addr learned my addr 172.21.15.28:0/302066229 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:19:04.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.747+0000 7fc372200640 1 --2- 172.21.15.28:0/302066229 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56430cb9eb60 0x56430cba6ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:04.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.747+0000 7fc372200640 1 --2- 172.21.15.28:0/302066229 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56430cba9b50 0x56430cba7550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:04.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.747+0000 7fc370e00640 1 -- 172.21.15.28:0/302066229 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56430cb9eb60 msgr2=0x56430cba6ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:19:04.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.747+0000 7fc370e00640 1 --2- 172.21.15.28:0/302066229 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56430cb9eb60 0x56430cba6ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:04.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.747+0000 7fc370e00640 1 -- 172.21.15.28:0/302066229 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56430cba9b50 msgr2=0x56430cba7550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:19:04.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.747+0000 7fc370e00640 1 --2- 172.21.15.28:0/302066229 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56430cba9b50 0x56430cba7550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:04.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.747+0000 7fc370e00640 1 -- 172.21.15.28:0/302066229 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56430c8e17d0 con 0x56430cb9f480 2024-05-12T22:19:04.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.747+0000 7fc372200640 1 --2- 172.21.15.28:0/302066229 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56430cba9b50 0x56430cba7550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:19:04.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.747+0000 7fc370e00640 1 --2- 172.21.15.28:0/302066229 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56430cb9f480 0x56430cba7010 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fc358009900 tx=0x7fc358004580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=632df39c2ef2ed12 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:19:04.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.747+0000 7fc353400640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-12T22:19:04.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.748+0000 7fc353400640 1 -- 172.21.15.28:0/302066229 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc358035040 con 0x56430cb9f480 2024-05-12T22:19:04.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.748+0000 7fc353400640 1 -- 172.21.15.28:0/302066229 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8513+0+0 (secure 0 0 0) 0x7fc3580054d0 con 0x56430cb9f480 2024-05-12T22:19:04.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.748+0000 7fc377025740 1 -- 172.21.15.28:0/302066229 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc36401bb40 con 0x56430cb9f480 2024-05-12T22:19:04.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.748+0000 7fc353400640 1 -- 172.21.15.28:0/302066229 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fc3580046c0 con 0x56430cb9f480 2024-05-12T22:19:04.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.748+0000 7fc377025740 10 client.27370 fetch_fsmap learned FSMap version 26 2024-05-12T22:19:04.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.748+0000 7fc377025740 1 -- 172.21.15.28:0/302066229 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x56430cbb8bc0 con 0x56430cb9f480 2024-05-12T22:19:04.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.748+0000 7fc353400640 1 -- 172.21.15.28:0/302066229 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc36401bb40 con 0x56430cb9f480 2024-05-12T22:19:04.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.748+0000 7fc353400640 1 -- 172.21.15.28:0/302066229 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fc358003c00 con 0x56430cb9f480 2024-05-12T22:19:04.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.749+0000 7fc377025740 10 client.27370 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:19:04.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.749+0000 7fc377025740 10 client.27370 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-05-12T22:19:04.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.749+0000 7fc377025740 20 client.27370 populate_metadata read hostname 'smithi028' 2024-05-12T22:19:04.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.749+0000 7fc377025740 1 --2- 172.21.15.28:0/302066229 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x56430cbba080 0x56430cbbc4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:04.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.749+0000 7fc377025740 4 client.27370 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-12T22:19:04.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.749+0000 7fc377025740 1 -- 172.21.15.28:0/302066229 --> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56430cb11160 con 0x56430cbba080 2024-05-12T22:19:04.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.749+0000 7fc372200640 1 --2- 172.21.15.28:0/302066229 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x56430cbba080 0x56430cbbc4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:04.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.750+0000 7fc372200640 1 --2- 172.21.15.28:0/302066229 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x56430cbba080 0x56430cbbc4c0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=6c929447e4416c6d server_cookie=7f1cfa7892906b8 in_seq=0 out_seq=0 2024-05-12T22:19:04.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.750+0000 7fc353400640 10 client.27370 ms_handle_connect on v2:172.21.15.28:6840/761183870 2024-05-12T22:19:04.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.751+0000 7fc353400640 1 -- 172.21.15.28:0/302066229 <== mds.1 v2:172.21.15.28:6840/761183870 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56430cb11160 con 0x56430cbba080 2024-05-12T22:19:04.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.751+0000 7fc353400640 10 client.27370 handle_command_reply: tid=0 2024-05-12T22:19:04.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.751+0000 7fc377025740 1 client.27370 shutdown 2024-05-12T22:19:04.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.751+0000 7fc377025740 1 -- 172.21.15.28:0/302066229 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56430cb9f480 msgr2=0x56430cba7010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:04.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.751+0000 7fc377025740 1 --2- 172.21.15.28:0/302066229 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56430cb9f480 0x56430cba7010 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fc358009900 tx=0x7fc358004580 comp rx=0 tx=0).stop 2024-05-12T22:19:04.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.751+0000 7fc377025740 1 -- 172.21.15.28:0/302066229 shutdown_connections 2024-05-12T22:19:04.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.751+0000 7fc377025740 1 --2- 172.21.15.28:0/302066229 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x56430cbba080 0x56430cbbc4c0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:04.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.751+0000 7fc377025740 1 --2- 172.21.15.28:0/302066229 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56430cba9b50 0x56430cba7550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:04.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.751+0000 7fc377025740 1 --2- 172.21.15.28:0/302066229 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56430cb9f480 0x56430cba7010 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:04.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.751+0000 7fc377025740 1 --2- 172.21.15.28:0/302066229 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56430cb9eb60 0x56430cba6ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:04.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.751+0000 7fc353400640 0 client.27370 ms_handle_reset on v2:172.21.15.28:6840/761183870 2024-05-12T22:19:04.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.751+0000 7fc377025740 1 -- 172.21.15.28:0/302066229 >> 172.21.15.28:0/302066229 conn(0x56430cb9ae80 msgr2=0x56430cba0c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:19:04.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.751+0000 7fc377025740 1 -- 172.21.15.28:0/302066229 shutdown_connections 2024-05-12T22:19:04.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.751+0000 7fc377025740 1 -- 172.21.15.28:0/302066229 wait complete. 2024-05-12T22:19:04.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.752+0000 7fc377025740 20 client.27370 trim_cache size 0 max 16384 2024-05-12T22:19:04.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.766+0000 7fc377025740 1 Processor -- start 2024-05-12T22:19:04.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.767+0000 7fc377025740 1 -- start start 2024-05-12T22:19:04.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.767+0000 7fc377025740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56430cb9eb60 0x56430cbb5800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:04.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.767+0000 7fc377025740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56430cb9f480 0x56430cbc1d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:04.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.767+0000 7fc377025740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56430cba9b50 0x56430cbc22d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:04.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.767+0000 7fc377025740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc36c1b1e80 con 0x56430cb9eb60 2024-05-12T22:19:04.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.767+0000 7fc377025740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fc36c1af750 con 0x56430cba9b50 2024-05-12T22:19:04.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.767+0000 7fc377025740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc36c1ad5c0 con 0x56430cb9f480 2024-05-12T22:19:04.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.767+0000 7fc372200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56430cb9eb60 0x56430cbb5800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:04.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.767+0000 7fc372200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56430cb9eb60 0x56430cbb5800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33088/0 (socket says 172.21.15.28:33088) 2024-05-12T22:19:04.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.767+0000 7fc372200640 1 -- 172.21.15.28:0/3290327991 learned_addr learned my addr 172.21.15.28:0/3290327991 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:19:04.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.767+0000 7fc370e00640 1 --2- 172.21.15.28:0/3290327991 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56430cb9f480 0x56430cbc1d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:04.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.768+0000 7fc372200640 1 --2- 172.21.15.28:0/3290327991 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56430cba9b50 0x56430cbc22d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:04.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.768+0000 7fc372200640 1 -- 172.21.15.28:0/3290327991 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56430cb9f480 msgr2=0x56430cbc1d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:19:04.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.768+0000 7fc372200640 1 --2- 172.21.15.28:0/3290327991 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56430cb9f480 0x56430cbc1d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:04.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.768+0000 7fc372200640 1 -- 172.21.15.28:0/3290327991 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56430cba9b50 msgr2=0x56430cbc22d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:19:04.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.768+0000 7fc372200640 1 --2- 172.21.15.28:0/3290327991 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56430cba9b50 0x56430cbc22d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:04.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.768+0000 7fc372200640 1 -- 172.21.15.28:0/3290327991 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56430cba7a30 con 0x56430cb9eb60 2024-05-12T22:19:04.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.768+0000 7fc370e00640 1 --2- 172.21.15.28:0/3290327991 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56430cb9f480 0x56430cbc1d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:19:04.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.768+0000 7fc372200640 1 --2- 172.21.15.28:0/3290327991 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56430cb9eb60 0x56430cbb5800 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fc364009cc0 tx=0x7fc364009820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c502c13b5bc5f12 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:19:04.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.769+0000 7fc35e000640 1 -- 172.21.15.28:0/3290327991 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc364032070 con 0x56430cb9eb60 2024-05-12T22:19:04.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.769+0000 7fc35e000640 1 -- 172.21.15.28:0/3290327991 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc364023040 con 0x56430cb9eb60 2024-05-12T22:19:04.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.769+0000 7fc35e000640 1 -- 172.21.15.28:0/3290327991 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc364016040 con 0x56430cb9eb60 2024-05-12T22:19:04.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.769+0000 7fc377025740 1 -- 172.21.15.28:0/3290327991 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56430cb9eb60 msgr2=0x56430cbb5800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:04.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.769+0000 7fc377025740 1 --2- 172.21.15.28:0/3290327991 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56430cb9eb60 0x56430cbb5800 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fc364009cc0 tx=0x7fc364009820 comp rx=0 tx=0).stop 2024-05-12T22:19:04.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.769+0000 7fc377025740 1 -- 172.21.15.28:0/3290327991 shutdown_connections 2024-05-12T22:19:04.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.769+0000 7fc377025740 1 --2- 172.21.15.28:0/3290327991 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56430cba9b50 0x56430cbc22d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:04.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.769+0000 7fc377025740 1 --2- 172.21.15.28:0/3290327991 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56430cb9f480 0x56430cbc1d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:04.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.769+0000 7fc377025740 1 --2- 172.21.15.28:0/3290327991 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56430cb9eb60 0x56430cbb5800 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:04.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.769+0000 7fc377025740 1 -- 172.21.15.28:0/3290327991 >> 172.21.15.28:0/3290327991 conn(0x56430cb9ae80 msgr2=0x56430cba7320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:19:04.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.770+0000 7fc377025740 1 -- 172.21.15.28:0/3290327991 shutdown_connections 2024-05-12T22:19:04.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.770+0000 7fc377025740 1 -- 172.21.15.28:0/3290327991 wait complete. 2024-05-12T22:19:04.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.770+0000 7fc377025740 1 -- start start 2024-05-12T22:19:04.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.770+0000 7fc377025740 1 Processor -- start 2024-05-12T22:19:04.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.771+0000 7fc377025740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56430cb9eb60 0x56430cbb1d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:04.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.771+0000 7fc377025740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56430cb9f480 0x56430cbb2250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:04.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.771+0000 7fc377025740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56430cba9b50 0x56430cbaed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:04.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.771+0000 7fc377025740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc36c1adc10 con 0x56430cba9b50 2024-05-12T22:19:04.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.771+0000 7fc377025740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fc36c1af080 con 0x56430cb9eb60 2024-05-12T22:19:04.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.771+0000 7fc377025740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc36c1ae2e0 con 0x56430cb9f480 2024-05-12T22:19:04.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.771+0000 7fc377025740 10 client.0 _ll_register_callbacks cb 0x7fc358004210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:19:04.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.771+0000 7fc372200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56430cba9b50 0x56430cbaed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:04.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.771+0000 7fc372200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56430cb9eb60 0x56430cbb1d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:04.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.771+0000 7fc370e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56430cb9f480 0x56430cbb2250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:04.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.771+0000 7fc372200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56430cb9eb60 0x56430cbb1d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:34398/0 (socket says 172.21.15.28:34398) 2024-05-12T22:19:04.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.771+0000 7fc372200640 1 -- 172.21.15.28:0/3221387450 learned_addr learned my addr 172.21.15.28:0/3221387450 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:19:04.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.771+0000 7fc370e00640 1 -- 172.21.15.28:0/3221387450 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56430cb9eb60 msgr2=0x56430cbb1d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:19:04.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.771+0000 7fc370e00640 1 --2- 172.21.15.28:0/3221387450 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56430cb9eb60 0x56430cbb1d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:04.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.771+0000 7fc370e00640 1 -- 172.21.15.28:0/3221387450 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56430cba9b50 msgr2=0x56430cbaed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:19:04.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.771+0000 7fc370e00640 1 --2- 172.21.15.28:0/3221387450 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56430cba9b50 0x56430cbaed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:04.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.772+0000 7fc370e00640 1 -- 172.21.15.28:0/3221387450 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56430cbaf570 con 0x56430cb9f480 2024-05-12T22:19:04.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.772+0000 7fc372200640 1 --2- 172.21.15.28:0/3221387450 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56430cb9eb60 0x56430cbb1d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:19:04.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.772+0000 7fc370e00640 1 --2- 172.21.15.28:0/3221387450 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56430cb9f480 0x56430cbb2250 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fc358031600 tx=0x7fc358032e90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d816db7a28ee8f0e server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:19:04.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.772+0000 7fc353400640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-12T22:19:04.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.772+0000 7fc353400640 1 -- 172.21.15.28:0/3221387450 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc358035050 con 0x56430cb9f480 2024-05-12T22:19:04.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.772+0000 7fc377025740 1 -- 172.21.15.28:0/3221387450 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc364023040 con 0x56430cb9f480 2024-05-12T22:19:04.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.772+0000 7fc353400640 1 -- 172.21.15.28:0/3221387450 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8513+0+0 (secure 0 0 0) 0x7fc3580054d0 con 0x56430cb9f480 2024-05-12T22:19:04.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.773+0000 7fc353400640 1 -- 172.21.15.28:0/3221387450 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fc358053040 con 0x56430cb9f480 2024-05-12T22:19:04.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.773+0000 7fc377025740 10 client.27376 fetch_fsmap learned FSMap version 26 2024-05-12T22:19:04.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.773+0000 7fc377025740 1 -- 172.21.15.28:0/3221387450 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x56430cc21190 con 0x56430cb9f480 2024-05-12T22:19:04.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.773+0000 7fc353400640 1 -- 172.21.15.28:0/3221387450 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc364023040 con 0x56430cb9f480 2024-05-12T22:19:04.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.773+0000 7fc353400640 1 -- 172.21.15.28:0/3221387450 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fc358003780 con 0x56430cb9f480 2024-05-12T22:19:04.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.773+0000 7fc377025740 10 client.27376 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:19:04.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.773+0000 7fc377025740 10 client.27376 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-05-12T22:19:04.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.773+0000 7fc377025740 20 client.27376 populate_metadata read hostname 'smithi028' 2024-05-12T22:19:04.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.773+0000 7fc377025740 1 --2- 172.21.15.28:0/3221387450 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x56430cc21900 0x56430cc23d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:04.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.773+0000 7fc377025740 4 client.27376 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-12T22:19:04.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.773+0000 7fc377025740 1 -- 172.21.15.28:0/3221387450 --> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56430cb0b4b0 con 0x56430cc21900 2024-05-12T22:19:04.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.774+0000 7fc372200640 1 --2- 172.21.15.28:0/3221387450 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x56430cc21900 0x56430cc23d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:04.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.774+0000 7fc372200640 1 --2- 172.21.15.28:0/3221387450 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x56430cc21900 0x56430cc23d20 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=4d1239c29ad48f60 server_cookie=aa96a791e74ee260 in_seq=0 out_seq=0 2024-05-12T22:19:04.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.774+0000 7fc353400640 10 client.27376 ms_handle_connect on v2:172.21.15.28:6840/761183870 2024-05-12T22:19:04.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.775+0000 7fc353400640 1 -- 172.21.15.28:0/3221387450 <== mds.1 v2:172.21.15.28:6840/761183870 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56430cb0b4b0 con 0x56430cc21900 2024-05-12T22:19:04.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.775+0000 7fc353400640 10 client.27376 handle_command_reply: tid=0 2024-05-12T22:19:04.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.775+0000 7fc377025740 1 client.27376 shutdown 2024-05-12T22:19:04.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.775+0000 7fc377025740 1 -- 172.21.15.28:0/3221387450 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56430cb9f480 msgr2=0x56430cbb2250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:04.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.775+0000 7fc377025740 1 --2- 172.21.15.28:0/3221387450 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56430cb9f480 0x56430cbb2250 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fc358031600 tx=0x7fc358032e90 comp rx=0 tx=0).stop 2024-05-12T22:19:04.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.775+0000 7fc377025740 1 -- 172.21.15.28:0/3221387450 shutdown_connections 2024-05-12T22:19:04.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.775+0000 7fc377025740 1 --2- 172.21.15.28:0/3221387450 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x56430cc21900 0x56430cc23d20 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:04.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.775+0000 7fc377025740 1 --2- 172.21.15.28:0/3221387450 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56430cba9b50 0x56430cbaed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:04.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.775+0000 7fc377025740 1 --2- 172.21.15.28:0/3221387450 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56430cb9f480 0x56430cbb2250 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:04.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.775+0000 7fc353400640 0 client.27376 ms_handle_reset on v2:172.21.15.28:6840/761183870 2024-05-12T22:19:04.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.775+0000 7fc377025740 1 --2- 172.21.15.28:0/3221387450 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56430cb9eb60 0x56430cbb1d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:04.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.775+0000 7fc377025740 1 -- 172.21.15.28:0/3221387450 >> 172.21.15.28:0/3221387450 conn(0x56430cb9ae80 msgr2=0x56430cb99770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:19:04.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.775+0000 7fc377025740 1 -- 172.21.15.28:0/3221387450 shutdown_connections 2024-05-12T22:19:04.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.775+0000 7fc377025740 1 -- 172.21.15.28:0/3221387450 wait complete. 2024-05-12T22:19:04.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.775+0000 7fc377025740 20 client.27376 trim_cache size 0 max 16384 2024-05-12T22:19:04.782 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-05-12T22:19:04.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.776+0000 7fc372c00640 1 -- 172.21.15.28:0/4212422723 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fc3540a2d80 msgr2=0x7fc3540a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:04.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.776+0000 7fc372c00640 1 --2- 172.21.15.28:0/4212422723 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fc3540a2d80 0x7fc3540a5240 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fc3600cc7e0 tx=0x7fc3600d2b30 comp rx=0 tx=0).stop 2024-05-12T22:19:04.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.776+0000 7fc372c00640 1 -- 172.21.15.28:0/4212422723 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc36c073c60 msgr2=0x7fc36c19a260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:04.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.776+0000 7fc372c00640 1 --2- 172.21.15.28:0/4212422723 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc36c073c60 0x7fc36c19a260 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fc36000d950 tx=0x7fc36000de10 comp rx=0 tx=0).stop 2024-05-12T22:19:04.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.776+0000 7fc372c00640 1 -- 172.21.15.28:0/4212422723 shutdown_connections 2024-05-12T22:19:04.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.776+0000 7fc372c00640 1 --2- 172.21.15.28:0/4212422723 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fc3540a2d80 0x7fc3540a5240 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:04.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.776+0000 7fc372c00640 1 --2- 172.21.15.28:0/4212422723 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc36c10df10 0x7fc36c19ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:04.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.776+0000 7fc372c00640 1 --2- 172.21.15.28:0/4212422723 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc36c074580 0x7fc36c19a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:04.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.776+0000 7fc372c00640 1 --2- 172.21.15.28:0/4212422723 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc36c073c60 0x7fc36c19a260 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:04.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.776+0000 7fc372c00640 1 -- 172.21.15.28:0/4212422723 >> 172.21.15.28:0/4212422723 conn(0x7fc36c0fd200 msgr2=0x7fc36c0fdfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:19:04.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.776+0000 7fc372c00640 1 -- 172.21.15.28:0/4212422723 shutdown_connections 2024-05-12T22:19:04.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:04.776+0000 7fc372c00640 1 -- 172.21.15.28:0/4212422723 wait complete. 2024-05-12T22:19:04.796 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-05-12T22:19:05.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.015+0000 7fcb92a00640 1 -- 172.21.15.28:0/1210855059 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcb8c1004a0 msgr2=0x7fcb8c100880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:05.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.015+0000 7fcb92a00640 1 --2- 172.21.15.28:0/1210855059 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcb8c1004a0 0x7fcb8c100880 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fcb80009ca0 tx=0x7fcb8002a8c0 comp rx=0 tx=0).stop 2024-05-12T22:19:05.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.015+0000 7fcb92a00640 1 -- 172.21.15.28:0/1210855059 shutdown_connections 2024-05-12T22:19:05.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.015+0000 7fcb92a00640 1 --2- 172.21.15.28:0/1210855059 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcb8c10e050 0x7fcb8c110440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:05.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.015+0000 7fcb92a00640 1 --2- 172.21.15.28:0/1210855059 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcb8c100e50 0x7fcb8c10da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:05.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.015+0000 7fcb92a00640 1 --2- 172.21.15.28:0/1210855059 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcb8c1004a0 0x7fcb8c100880 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:05.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.015+0000 7fcb92a00640 1 -- 172.21.15.28:0/1210855059 >> 172.21.15.28:0/1210855059 conn(0x7fcb8c0fc0b0 msgr2=0x7fcb8c0fe4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:19:05.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.015+0000 7fcb92a00640 1 -- 172.21.15.28:0/1210855059 shutdown_connections 2024-05-12T22:19:05.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.015+0000 7fcb92a00640 1 -- 172.21.15.28:0/1210855059 wait complete. 2024-05-12T22:19:05.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.016+0000 7fcb92a00640 1 Processor -- start 2024-05-12T22:19:05.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.016+0000 7fcb92a00640 1 -- start start 2024-05-12T22:19:05.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.017+0000 7fcb92a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcb8c1004a0 0x7fcb8c19a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:05.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.017+0000 7fcb92a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcb8c100e50 0x7fcb8c19a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:05.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.017+0000 7fcb92a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcb8c10e050 0x7fcb8c19ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:05.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.017+0000 7fcb92a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcb8c113d30 con 0x7fcb8c10e050 2024-05-12T22:19:05.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.017+0000 7fcb92a00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fcb8c113bb0 con 0x7fcb8c100e50 2024-05-12T22:19:05.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.017+0000 7fcb92a00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fcb8c113eb0 con 0x7fcb8c1004a0 2024-05-12T22:19:05.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.017+0000 7fcb92000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcb8c10e050 0x7fcb8c19ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:05.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.017+0000 7fcb92000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcb8c10e050 0x7fcb8c19ead0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33122/0 (socket says 172.21.15.28:33122) 2024-05-12T22:19:05.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.017+0000 7fcb91600640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcb8c1004a0 0x7fcb8c19a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:05.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.017+0000 7fcb92000640 1 -- 172.21.15.28:0/3073323446 learned_addr learned my addr 172.21.15.28:0/3073323446 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:19:05.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.017+0000 7fcb90c00640 1 --2- 172.21.15.28:0/3073323446 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcb8c100e50 0x7fcb8c19a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:05.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.018+0000 7fcb92000640 1 -- 172.21.15.28:0/3073323446 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcb8c1004a0 msgr2=0x7fcb8c19a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:05.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.018+0000 7fcb92000640 1 --2- 172.21.15.28:0/3073323446 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcb8c1004a0 0x7fcb8c19a210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:05.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.018+0000 7fcb92000640 1 -- 172.21.15.28:0/3073323446 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcb8c100e50 msgr2=0x7fcb8c19a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:05.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.018+0000 7fcb92000640 1 --2- 172.21.15.28:0/3073323446 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcb8c100e50 0x7fcb8c19a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:05.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.018+0000 7fcb92000640 1 -- 172.21.15.28:0/3073323446 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb8c19f200 con 0x7fcb8c10e050 2024-05-12T22:19:05.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.018+0000 7fcb92000640 1 --2- 172.21.15.28:0/3073323446 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcb8c10e050 0x7fcb8c19ead0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fcb8400e940 tx=0x7fcb8400ee00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:19:05.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.018+0000 7fcb8a000640 1 -- 172.21.15.28:0/3073323446 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb8400ccc0 con 0x7fcb8c10e050 2024-05-12T22:19:05.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.018+0000 7fcb92a00640 1 -- 172.21.15.28:0/3073323446 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb8c19f4f0 con 0x7fcb8c10e050 2024-05-12T22:19:05.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.018+0000 7fcb8a000640 1 -- 172.21.15.28:0/3073323446 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcb84005500 con 0x7fcb8c10e050 2024-05-12T22:19:05.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.018+0000 7fcb8a000640 1 -- 172.21.15.28:0/3073323446 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb84010690 con 0x7fcb8c10e050 2024-05-12T22:19:05.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.018+0000 7fcb92a00640 1 -- 172.21.15.28:0/3073323446 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb8c1a6d80 con 0x7fcb8c10e050 2024-05-12T22:19:05.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.022+0000 7fcb96fb9740 1 Processor -- start 2024-05-12T22:19:05.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.022+0000 7fcb8a000640 1 -- 172.21.15.28:0/3073323446 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7fcb840046c0 con 0x7fcb8c10e050 2024-05-12T22:19:05.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.022+0000 7fcb96fb9740 1 -- start start 2024-05-12T22:19:05.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.023+0000 7fcb96fb9740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ff9dc88b60 0x55ff9dc88f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:05.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.023+0000 7fcb96fb9740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ff9dc89480 0x55ff9dc93610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:05.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.023+0000 7fcb96fb9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ff9dc93b50 0x55ff9dc95f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:05.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.023+0000 7fcb96fb9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ff9dbeda30 con 0x55ff9dc93b50 2024-05-12T22:19:05.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.023+0000 7fcb96fb9740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x55ff9db95700 con 0x55ff9dc88b60 2024-05-12T22:19:05.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.023+0000 7fcb96fb9740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55ff9dbe99f0 con 0x55ff9dc89480 2024-05-12T22:19:05.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.023+0000 7fcb91600640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ff9dc89480 0x55ff9dc93610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:05.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.023+0000 7fcb90c00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ff9dc88b60 0x55ff9dc88f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:05.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.023+0000 7fcb91600640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ff9dc89480 0x55ff9dc93610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:46486/0 (socket says 172.21.15.28:46486) 2024-05-12T22:19:05.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.023+0000 7fcb91600640 1 -- 172.21.15.28:0/162158785 learned_addr learned my addr 172.21.15.28:0/162158785 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:19:05.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.023+0000 7fcb90c00640 1 --2- 172.21.15.28:0/162158785 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ff9dc93b50 0x55ff9dc95f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:05.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.024+0000 7fcb91600640 1 -- 172.21.15.28:0/162158785 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ff9dc88b60 msgr2=0x55ff9dc88f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:19:05.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.024+0000 7fcb91600640 1 --2- 172.21.15.28:0/162158785 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ff9dc88b60 0x55ff9dc88f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:05.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.024+0000 7fcb91600640 1 -- 172.21.15.28:0/162158785 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ff9dc93b50 msgr2=0x55ff9dc95f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:19:05.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.024+0000 7fcb91600640 1 --2- 172.21.15.28:0/162158785 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ff9dc93b50 0x55ff9dc95f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:05.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.024+0000 7fcb91600640 1 -- 172.21.15.28:0/162158785 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ff9d9013f0 con 0x55ff9dc89480 2024-05-12T22:19:05.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.024+0000 7fcb90c00640 1 --2- 172.21.15.28:0/162158785 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ff9dc93b50 0x55ff9dc95f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:19:05.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.024+0000 7fcb91600640 1 --2- 172.21.15.28:0/162158785 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ff9dc89480 0x55ff9dc93610 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fcb8002ade0 tx=0x7fcb80005fa0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3603ed4a1f903018 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:19:05.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.024+0000 7fcb8a000640 1 --2- 172.21.15.28:0/3073323446 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fcb740a2d00 0x7fcb740a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:05.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.024+0000 7fcb7e600640 1 -- 172.21.15.28:0/162158785 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb80031db0 con 0x55ff9dc89480 2024-05-12T22:19:05.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.024+0000 7fcb7e600640 1 -- 172.21.15.28:0/162158785 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcb80036030 con 0x55ff9dc89480 2024-05-12T22:19:05.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.024+0000 7fcb7e600640 1 -- 172.21.15.28:0/162158785 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb80004260 con 0x55ff9dc89480 2024-05-12T22:19:05.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.024+0000 7fcb8a000640 1 -- 172.21.15.28:0/3073323446 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8513+0+0 (secure 0 0 0) 0x7fcb840cf740 con 0x7fcb8c10e050 2024-05-12T22:19:05.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.024+0000 7fcb90c00640 1 --2- 172.21.15.28:0/3073323446 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fcb740a2d00 0x7fcb740a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:05.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.025+0000 7fcb96fb9740 1 -- 172.21.15.28:0/162158785 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ff9dc89480 msgr2=0x55ff9dc93610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:05.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.025+0000 7fcb96fb9740 1 --2- 172.21.15.28:0/162158785 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ff9dc89480 0x55ff9dc93610 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fcb8002ade0 tx=0x7fcb80005fa0 comp rx=0 tx=0).stop 2024-05-12T22:19:05.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.025+0000 7fcb96fb9740 1 -- 172.21.15.28:0/162158785 shutdown_connections 2024-05-12T22:19:05.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.025+0000 7fcb96fb9740 1 --2- 172.21.15.28:0/162158785 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ff9dc93b50 0x55ff9dc95f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:05.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.025+0000 7fcb96fb9740 1 --2- 172.21.15.28:0/162158785 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ff9dc89480 0x55ff9dc93610 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:05.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.025+0000 7fcb96fb9740 1 --2- 172.21.15.28:0/162158785 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ff9dc88b60 0x55ff9dc88f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:05.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.025+0000 7fcb96fb9740 1 -- 172.21.15.28:0/162158785 >> 172.21.15.28:0/162158785 conn(0x55ff9dc84e80 msgr2=0x55ff9dc85260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:19:05.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.025+0000 7fcb90c00640 1 --2- 172.21.15.28:0/3073323446 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fcb740a2d00 0x7fcb740a51c0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x55ff9dc86010 tx=0x7fcb78013040 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:19:05.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.025+0000 7fcb96fb9740 1 -- 172.21.15.28:0/162158785 shutdown_connections 2024-05-12T22:19:05.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.025+0000 7fcb96fb9740 1 -- 172.21.15.28:0/162158785 wait complete. 2024-05-12T22:19:05.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.026+0000 7fcb96fb9740 1 -- start start 2024-05-12T22:19:05.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.026+0000 7fcb96fb9740 1 Processor -- start 2024-05-12T22:19:05.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.026+0000 7fcb96fb9740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ff9dc88b60 0x55ff9dc90ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:05.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.026+0000 7fcb96fb9740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ff9dc89480 0x55ff9dc91000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:05.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.026+0000 7fcb96fb9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ff9dc93b50 0x55ff9dc91540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:05.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.026+0000 7fcb96fb9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ff9dc96760 con 0x55ff9dc93b50 2024-05-12T22:19:05.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.026+0000 7fcb96fb9740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x55ff9dba8a70 con 0x55ff9dc88b60 2024-05-12T22:19:05.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.026+0000 7fcb96fb9740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55ff9dc968e0 con 0x55ff9dc89480 2024-05-12T22:19:05.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.027+0000 7fcb96fb9740 10 client.0 _ll_register_callbacks cb 0x55ff9db8bba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:19:05.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.027+0000 7fcb90c00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ff9dc89480 0x55ff9dc91000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:05.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.027+0000 7fcb90c00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ff9dc89480 0x55ff9dc91000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:46502/0 (socket says 172.21.15.28:46502) 2024-05-12T22:19:05.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.027+0000 7fcb90c00640 1 -- 172.21.15.28:0/239785388 learned_addr learned my addr 172.21.15.28:0/239785388 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:19:05.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.027+0000 7fcb92000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ff9dc93b50 0x55ff9dc91540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:05.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.027+0000 7fcb91600640 1 --2- 172.21.15.28:0/239785388 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ff9dc88b60 0x55ff9dc90ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:05.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.027+0000 7fcb90c00640 1 -- 172.21.15.28:0/239785388 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ff9dc88b60 msgr2=0x55ff9dc90ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:19:05.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.027+0000 7fcb90c00640 1 --2- 172.21.15.28:0/239785388 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ff9dc88b60 0x55ff9dc90ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:05.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.027+0000 7fcb90c00640 1 -- 172.21.15.28:0/239785388 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ff9dc93b50 msgr2=0x55ff9dc91540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:19:05.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.027+0000 7fcb90c00640 1 --2- 172.21.15.28:0/239785388 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ff9dc93b50 0x55ff9dc91540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:05.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.027+0000 7fcb90c00640 1 -- 172.21.15.28:0/239785388 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ff9d9cb7d0 con 0x55ff9dc89480 2024-05-12T22:19:05.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.028+0000 7fcb90c00640 1 --2- 172.21.15.28:0/239785388 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ff9dc89480 0x55ff9dc91000 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fcb780038a0 tx=0x7fcb78004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=131b893c9d31803 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:19:05.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.028+0000 7fcb72a00640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-12T22:19:05.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.028+0000 7fcb72a00640 1 -- 172.21.15.28:0/239785388 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb78003d60 con 0x55ff9dc89480 2024-05-12T22:19:05.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.028+0000 7fcb96fb9740 1 -- 172.21.15.28:0/239785388 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcb80036030 con 0x55ff9dc89480 2024-05-12T22:19:05.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.028+0000 7fcb72a00640 1 -- 172.21.15.28:0/239785388 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8513+0+0 (secure 0 0 0) 0x7fcb7800fa80 con 0x55ff9dc89480 2024-05-12T22:19:05.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.028+0000 7fcb72a00640 1 -- 172.21.15.28:0/239785388 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fcb7800fcc0 con 0x55ff9dc89480 2024-05-12T22:19:05.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.028+0000 7fcb96fb9740 10 client.27388 fetch_fsmap learned FSMap version 26 2024-05-12T22:19:05.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.028+0000 7fcb96fb9740 1 -- 172.21.15.28:0/239785388 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55ff9dca2a40 con 0x55ff9dc89480 2024-05-12T22:19:05.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.029+0000 7fcb72a00640 1 -- 172.21.15.28:0/239785388 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcb80036030 con 0x55ff9dc89480 2024-05-12T22:19:05.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.029+0000 7fcb72a00640 1 -- 172.21.15.28:0/239785388 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fcb78021450 con 0x55ff9dc89480 2024-05-12T22:19:05.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.029+0000 7fcb96fb9740 10 client.27388 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:19:05.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.029+0000 7fcb96fb9740 10 client.27388 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:19:05.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.029+0000 7fcb96fb9740 20 client.27388 populate_metadata read hostname 'smithi028' 2024-05-12T22:19:05.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.029+0000 7fcb96fb9740 1 --2- 172.21.15.28:0/239785388 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55ff9dca3ef0 0x55ff9dca6350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:05.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.029+0000 7fcb96fb9740 4 client.27388 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-12T22:19:05.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.029+0000 7fcb96fb9740 1 -- 172.21.15.28:0/239785388 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ff9dbfb160 con 0x55ff9dca3ef0 2024-05-12T22:19:05.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.029+0000 7fcb91600640 1 --2- 172.21.15.28:0/239785388 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55ff9dca3ef0 0x55ff9dca6350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:05.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.030+0000 7fcb91600640 1 --2- 172.21.15.28:0/239785388 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55ff9dca3ef0 0x55ff9dca6350 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dab900fbe4feadb0 server_cookie=25cfad30b3a88b9a in_seq=0 out_seq=0 2024-05-12T22:19:05.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.030+0000 7fcb72a00640 10 client.27388 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:19:05.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.031+0000 7fcb72a00640 1 -- 172.21.15.28:0/239785388 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ff9dbfb160 con 0x55ff9dca3ef0 2024-05-12T22:19:05.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.031+0000 7fcb72a00640 10 client.27388 handle_command_reply: tid=0 2024-05-12T22:19:05.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.031+0000 7fcb96fb9740 1 client.27388 shutdown 2024-05-12T22:19:05.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.031+0000 7fcb96fb9740 1 -- 172.21.15.28:0/239785388 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ff9dc89480 msgr2=0x55ff9dc91000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:05.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.031+0000 7fcb96fb9740 1 --2- 172.21.15.28:0/239785388 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ff9dc89480 0x55ff9dc91000 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fcb780038a0 tx=0x7fcb78004730 comp rx=0 tx=0).stop 2024-05-12T22:19:05.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.031+0000 7fcb96fb9740 1 -- 172.21.15.28:0/239785388 shutdown_connections 2024-05-12T22:19:05.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.031+0000 7fcb96fb9740 1 --2- 172.21.15.28:0/239785388 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55ff9dca3ef0 0x55ff9dca6350 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:05.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.031+0000 7fcb96fb9740 1 --2- 172.21.15.28:0/239785388 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ff9dc93b50 0x55ff9dc91540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:05.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.031+0000 7fcb96fb9740 1 --2- 172.21.15.28:0/239785388 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ff9dc89480 0x55ff9dc91000 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:05.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.031+0000 7fcb96fb9740 1 --2- 172.21.15.28:0/239785388 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ff9dc88b60 0x55ff9dc90ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:05.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.031+0000 7fcb96fb9740 1 -- 172.21.15.28:0/239785388 >> 172.21.15.28:0/239785388 conn(0x55ff9dc84e80 msgr2=0x55ff9dc8ac80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:19:05.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.031+0000 7fcb72a00640 0 client.27388 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:19:05.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.031+0000 7fcb96fb9740 1 -- 172.21.15.28:0/239785388 shutdown_connections 2024-05-12T22:19:05.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.031+0000 7fcb96fb9740 1 -- 172.21.15.28:0/239785388 wait complete. 2024-05-12T22:19:05.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.032+0000 7fcb96fb9740 20 client.27388 trim_cache size 0 max 16384 2024-05-12T22:19:05.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.046+0000 7fcb96fb9740 1 Processor -- start 2024-05-12T22:19:05.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.047+0000 7fcb96fb9740 1 -- start start 2024-05-12T22:19:05.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.047+0000 7fcb96fb9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ff9dc88b60 0x55ff9dc9f620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:05.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.047+0000 7fcb96fb9740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ff9dc89480 0x55ff9dc8b300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:05.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.047+0000 7fcb96fb9740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ff9dc93b50 0x55ff9dcabc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:05.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.047+0000 7fcb96fb9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcb8c1b1e70 con 0x55ff9dc88b60 2024-05-12T22:19:05.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.047+0000 7fcb96fb9740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fcb8c1af740 con 0x55ff9dc93b50 2024-05-12T22:19:05.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.047+0000 7fcb96fb9740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fcb8c1ad5b0 con 0x55ff9dc89480 2024-05-12T22:19:05.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.047+0000 7fcb91600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ff9dc88b60 0x55ff9dc9f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:05.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.047+0000 7fcb91600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ff9dc88b60 0x55ff9dc9f620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33144/0 (socket says 172.21.15.28:33144) 2024-05-12T22:19:05.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.047+0000 7fcb91600640 1 -- 172.21.15.28:0/3343531441 learned_addr learned my addr 172.21.15.28:0/3343531441 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:19:05.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.047+0000 7fcb90c00640 1 --2- 172.21.15.28:0/3343531441 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ff9dc89480 0x55ff9dc8b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:05.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.047+0000 7fcb92000640 1 --2- 172.21.15.28:0/3343531441 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ff9dc93b50 0x55ff9dcabc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:05.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.048+0000 7fcb91600640 1 -- 172.21.15.28:0/3343531441 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ff9dc89480 msgr2=0x55ff9dc8b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:19:05.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.048+0000 7fcb91600640 1 --2- 172.21.15.28:0/3343531441 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ff9dc89480 0x55ff9dc8b300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:05.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.048+0000 7fcb91600640 1 -- 172.21.15.28:0/3343531441 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ff9dc93b50 msgr2=0x55ff9dcabc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:19:05.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.048+0000 7fcb91600640 1 --2- 172.21.15.28:0/3343531441 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ff9dc93b50 0x55ff9dcabc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:05.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.048+0000 7fcb91600640 1 -- 172.21.15.28:0/3343531441 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ff9dc899d0 con 0x55ff9dc88b60 2024-05-12T22:19:05.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.048+0000 7fcb92000640 1 --2- 172.21.15.28:0/3343531441 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ff9dc93b50 0x55ff9dcabc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:19:05.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.048+0000 7fcb91600640 1 --2- 172.21.15.28:0/3343531441 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ff9dc88b60 0x55ff9dc9f620 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fcb8002ade0 tx=0x7fcb800319b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55bd949d4bdaec93 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:19:05.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.048+0000 7fcb7dc00640 1 -- 172.21.15.28:0/3343531441 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb80056e60 con 0x55ff9dc88b60 2024-05-12T22:19:05.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.048+0000 7fcb7dc00640 1 -- 172.21.15.28:0/3343531441 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcb80036040 con 0x55ff9dc88b60 2024-05-12T22:19:05.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.048+0000 7fcb7dc00640 1 -- 172.21.15.28:0/3343531441 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb8004a040 con 0x55ff9dc88b60 2024-05-12T22:19:05.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.049+0000 7fcb96fb9740 1 -- 172.21.15.28:0/3343531441 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ff9dc88b60 msgr2=0x55ff9dc9f620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:05.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.049+0000 7fcb96fb9740 1 --2- 172.21.15.28:0/3343531441 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ff9dc88b60 0x55ff9dc9f620 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fcb8002ade0 tx=0x7fcb800319b0 comp rx=0 tx=0).stop 2024-05-12T22:19:05.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.049+0000 7fcb96fb9740 1 -- 172.21.15.28:0/3343531441 shutdown_connections 2024-05-12T22:19:05.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.049+0000 7fcb96fb9740 1 --2- 172.21.15.28:0/3343531441 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ff9dc93b50 0x55ff9dcabc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:05.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.049+0000 7fcb96fb9740 1 --2- 172.21.15.28:0/3343531441 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ff9dc89480 0x55ff9dc8b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:05.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.049+0000 7fcb96fb9740 1 --2- 172.21.15.28:0/3343531441 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ff9dc88b60 0x55ff9dc9f620 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:05.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.049+0000 7fcb96fb9740 1 -- 172.21.15.28:0/3343531441 >> 172.21.15.28:0/3343531441 conn(0x55ff9dc84e80 msgr2=0x55ff9dc91310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:19:05.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.049+0000 7fcb96fb9740 1 -- 172.21.15.28:0/3343531441 shutdown_connections 2024-05-12T22:19:05.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.049+0000 7fcb96fb9740 1 -- 172.21.15.28:0/3343531441 wait complete. 2024-05-12T22:19:05.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.049+0000 7fcb96fb9740 1 -- start start 2024-05-12T22:19:05.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.050+0000 7fcb96fb9740 1 Processor -- start 2024-05-12T22:19:05.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.050+0000 7fcb96fb9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ff9dc88b60 0x55ff9dc9c030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:05.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.050+0000 7fcb96fb9740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ff9dc89480 0x55ff9dc973a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:05.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.050+0000 7fcb96fb9740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ff9dc93b50 0x55ff9dc97ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:05.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.050+0000 7fcb96fb9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcb8c1adc00 con 0x55ff9dc88b60 2024-05-12T22:19:05.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.050+0000 7fcb96fb9740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fcb8c1af070 con 0x55ff9dc93b50 2024-05-12T22:19:05.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.050+0000 7fcb96fb9740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fcb8c1ae2d0 con 0x55ff9dc89480 2024-05-12T22:19:05.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.050+0000 7fcb96fb9740 10 client.0 _ll_register_callbacks cb 0x7fcb78004420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:19:05.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.051+0000 7fcb91600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ff9dc88b60 0x55ff9dc9c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:05.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.051+0000 7fcb92000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ff9dc93b50 0x55ff9dc97ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:05.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.051+0000 7fcb90c00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ff9dc89480 0x55ff9dc973a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:05.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.051+0000 7fcb92000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ff9dc93b50 0x55ff9dc97ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:34444/0 (socket says 172.21.15.28:34444) 2024-05-12T22:19:05.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.051+0000 7fcb92000640 1 -- 172.21.15.28:0/1029101577 learned_addr learned my addr 172.21.15.28:0/1029101577 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:19:05.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.051+0000 7fcb92000640 1 -- 172.21.15.28:0/1029101577 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ff9dc89480 msgr2=0x55ff9dc973a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:19:05.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.051+0000 7fcb92000640 1 --2- 172.21.15.28:0/1029101577 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ff9dc89480 0x55ff9dc973a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:05.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.051+0000 7fcb92000640 1 -- 172.21.15.28:0/1029101577 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ff9dc88b60 msgr2=0x55ff9dc9c030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:19:05.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.051+0000 7fcb92000640 1 --2- 172.21.15.28:0/1029101577 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ff9dc88b60 0x55ff9dc9c030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:05.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.051+0000 7fcb92000640 1 -- 172.21.15.28:0/1029101577 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ff9dd11ca0 con 0x55ff9dc93b50 2024-05-12T22:19:05.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.051+0000 7fcb92000640 1 --2- 172.21.15.28:0/1029101577 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ff9dc93b50 0x55ff9dc97ae0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fcb8409bb90 tx=0x7fcb8409b690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=59c9cf843988f18 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:19:05.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.051+0000 7fcb72a00640 10 client.0 ms_handle_connect on v2:172.21.15.86:3300/0 2024-05-12T22:19:05.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.051+0000 7fcb72a00640 1 -- 172.21.15.28:0/1029101577 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb840b8020 con 0x55ff9dc93b50 2024-05-12T22:19:05.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.051+0000 7fcb96fb9740 1 -- 172.21.15.28:0/1029101577 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcb80036040 con 0x55ff9dc93b50 2024-05-12T22:19:05.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.051+0000 7fcb72a00640 1 -- 172.21.15.28:0/1029101577 <== mon.2 v2:172.21.15.86:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8513+0+0 (secure 0 0 0) 0x7fcb840a43e0 con 0x55ff9dc93b50 2024-05-12T22:19:05.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.052+0000 7fcb72a00640 1 -- 172.21.15.28:0/1029101577 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fcb8409ffb0 con 0x55ff9dc93b50 2024-05-12T22:19:05.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.052+0000 7fcb96fb9740 10 client.27443 fetch_fsmap learned FSMap version 26 2024-05-12T22:19:05.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.052+0000 7fcb96fb9740 1 -- 172.21.15.28:0/1029101577 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55ff9dd12e30 con 0x55ff9dc93b50 2024-05-12T22:19:05.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.052+0000 7fcb72a00640 1 -- 172.21.15.28:0/1029101577 <== mon.2 v2:172.21.15.86:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcb80036040 con 0x55ff9dc93b50 2024-05-12T22:19:05.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.052+0000 7fcb72a00640 1 -- 172.21.15.28:0/1029101577 <== mon.2 v2:172.21.15.86:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fcb84097950 con 0x55ff9dc93b50 2024-05-12T22:19:05.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.052+0000 7fcb96fb9740 10 client.27443 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:19:05.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.052+0000 7fcb96fb9740 10 client.27443 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:19:05.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.052+0000 7fcb96fb9740 20 client.27443 populate_metadata read hostname 'smithi028' 2024-05-12T22:19:05.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.052+0000 7fcb96fb9740 1 --2- 172.21.15.28:0/1029101577 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55ff9dd13550 0x55ff9dd15970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:05.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.052+0000 7fcb96fb9740 4 client.27443 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-12T22:19:05.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.052+0000 7fcb96fb9740 1 -- 172.21.15.28:0/1029101577 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ff9dbf54b0 con 0x55ff9dd13550 2024-05-12T22:19:05.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.052+0000 7fcb91600640 1 --2- 172.21.15.28:0/1029101577 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55ff9dd13550 0x55ff9dd15970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:05.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.053+0000 7fcb91600640 1 --2- 172.21.15.28:0/1029101577 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55ff9dd13550 0x55ff9dd15970 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6d2eff8b9c5bf910 server_cookie=105b94d317a1506c in_seq=0 out_seq=0 2024-05-12T22:19:05.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.053+0000 7fcb72a00640 10 client.27443 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:19:05.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.053+0000 7fcb72a00640 1 -- 172.21.15.28:0/1029101577 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ff9dbf54b0 con 0x55ff9dd13550 2024-05-12T22:19:05.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.053+0000 7fcb72a00640 10 client.27443 handle_command_reply: tid=0 2024-05-12T22:19:05.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.053+0000 7fcb96fb9740 1 client.27443 shutdown 2024-05-12T22:19:05.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.053+0000 7fcb96fb9740 1 -- 172.21.15.28:0/1029101577 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ff9dc93b50 msgr2=0x55ff9dc97ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:05.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.053+0000 7fcb96fb9740 1 --2- 172.21.15.28:0/1029101577 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ff9dc93b50 0x55ff9dc97ae0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fcb8409bb90 tx=0x7fcb8409b690 comp rx=0 tx=0).stop 2024-05-12T22:19:05.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.054+0000 7fcb96fb9740 1 -- 172.21.15.28:0/1029101577 shutdown_connections 2024-05-12T22:19:05.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.054+0000 7fcb96fb9740 1 --2- 172.21.15.28:0/1029101577 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55ff9dd13550 0x55ff9dd15970 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:05.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.054+0000 7fcb96fb9740 1 --2- 172.21.15.28:0/1029101577 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ff9dc93b50 0x55ff9dc97ae0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:05.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.054+0000 7fcb72a00640 0 client.27443 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:19:05.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.054+0000 7fcb96fb9740 1 --2- 172.21.15.28:0/1029101577 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ff9dc89480 0x55ff9dc973a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:05.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.054+0000 7fcb96fb9740 1 --2- 172.21.15.28:0/1029101577 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ff9dc88b60 0x55ff9dc9c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:05.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.054+0000 7fcb96fb9740 1 -- 172.21.15.28:0/1029101577 >> 172.21.15.28:0/1029101577 conn(0x55ff9dc84e80 msgr2=0x55ff9dc91140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:19:05.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.054+0000 7fcb96fb9740 1 -- 172.21.15.28:0/1029101577 shutdown_connections 2024-05-12T22:19:05.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.054+0000 7fcb96fb9740 1 -- 172.21.15.28:0/1029101577 wait complete. 2024-05-12T22:19:05.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.054+0000 7fcb96fb9740 20 client.27443 trim_cache size 0 max 16384 2024-05-12T22:19:05.062 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-05-12T22:19:05.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.054+0000 7fcb92a00640 1 -- 172.21.15.28:0/3073323446 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fcb740a2d00 msgr2=0x7fcb740a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:05.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.054+0000 7fcb92a00640 1 --2- 172.21.15.28:0/3073323446 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fcb740a2d00 0x7fcb740a51c0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x55ff9dc86010 tx=0x7fcb78013040 comp rx=0 tx=0).stop 2024-05-12T22:19:05.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.054+0000 7fcb92a00640 1 -- 172.21.15.28:0/3073323446 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcb8c10e050 msgr2=0x7fcb8c19ead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:05.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.054+0000 7fcb92a00640 1 --2- 172.21.15.28:0/3073323446 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcb8c10e050 0x7fcb8c19ead0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fcb8400e940 tx=0x7fcb8400ee00 comp rx=0 tx=0).stop 2024-05-12T22:19:05.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.055+0000 7fcb92a00640 1 -- 172.21.15.28:0/3073323446 shutdown_connections 2024-05-12T22:19:05.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.055+0000 7fcb92a00640 1 --2- 172.21.15.28:0/3073323446 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fcb740a2d00 0x7fcb740a51c0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:05.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.055+0000 7fcb92a00640 1 --2- 172.21.15.28:0/3073323446 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcb8c10e050 0x7fcb8c19ead0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:05.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.055+0000 7fcb92a00640 1 --2- 172.21.15.28:0/3073323446 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcb8c100e50 0x7fcb8c19a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:05.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.055+0000 7fcb92a00640 1 --2- 172.21.15.28:0/3073323446 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcb8c1004a0 0x7fcb8c19a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:05.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.055+0000 7fcb92a00640 1 -- 172.21.15.28:0/3073323446 >> 172.21.15.28:0/3073323446 conn(0x7fcb8c0fc0b0 msgr2=0x7fcb8c10f1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:19:05.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.055+0000 7fcb92a00640 1 -- 172.21.15.28:0/3073323446 shutdown_connections 2024-05-12T22:19:05.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:05.055+0000 7fcb92a00640 1 -- 172.21.15.28:0/3073323446 wait complete. 2024-05-12T22:19:06.072 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-12T22:19:06.072 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-12T22:19:06.072 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-05-12T22:19:06.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.314+0000 7f3e96600640 1 -- 172.21.15.28:0/1381408703 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3e90107dd0 msgr2=0x7f3e90108230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:06.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.314+0000 7f3e96600640 1 --2- 172.21.15.28:0/1381408703 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3e90107dd0 0x7f3e90108230 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f3e78009900 tx=0x7f3e7802a480 comp rx=0 tx=0).stop 2024-05-12T22:19:06.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.314+0000 7f3e96600640 1 -- 172.21.15.28:0/1381408703 shutdown_connections 2024-05-12T22:19:06.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.314+0000 7f3e96600640 1 --2- 172.21.15.28:0/1381408703 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3e90108770 0x7f3e90114b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:06.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.314+0000 7f3e96600640 1 --2- 172.21.15.28:0/1381408703 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3e90107dd0 0x7f3e90108230 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:06.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.314+0000 7f3e96600640 1 --2- 172.21.15.28:0/1381408703 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e9010ddf0 0x7f3e9010e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:06.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.314+0000 7f3e96600640 1 -- 172.21.15.28:0/1381408703 >> 172.21.15.28:0/1381408703 conn(0x7f3e90077650 msgr2=0x7f3e90106650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:19:06.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.314+0000 7f3e96600640 1 -- 172.21.15.28:0/1381408703 shutdown_connections 2024-05-12T22:19:06.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.314+0000 7f3e96600640 1 -- 172.21.15.28:0/1381408703 wait complete. 2024-05-12T22:19:06.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.316+0000 7f3e96600640 1 Processor -- start 2024-05-12T22:19:06.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.316+0000 7f3e96600640 1 -- start start 2024-05-12T22:19:06.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.316+0000 7f3e96600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3e90107dd0 0x7f3e9019e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:06.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.316+0000 7f3e96600640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3e90108770 0x7f3e9019e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:06.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.316+0000 7f3e96600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e9010ddf0 0x7f3e901a2d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:06.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.316+0000 7f3e96600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3e90118100 con 0x7f3e9010ddf0 2024-05-12T22:19:06.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.316+0000 7f3e96600640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f3e90117f80 con 0x7f3e90107dd0 2024-05-12T22:19:06.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.316+0000 7f3e96600640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f3e90118280 con 0x7f3e90108770 2024-05-12T22:19:06.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.317+0000 7f3e95200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3e90107dd0 0x7f3e9019e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:06.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.317+0000 7f3e95200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3e90107dd0 0x7f3e9019e4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:41032/0 (socket says 172.21.15.28:41032) 2024-05-12T22:19:06.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.317+0000 7f3e95200640 1 -- 172.21.15.28:0/1196759876 learned_addr learned my addr 172.21.15.28:0/1196759876 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:19:06.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.317+0000 7f3e95c00640 1 --2- 172.21.15.28:0/1196759876 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e9010ddf0 0x7f3e901a2d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:06.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.317+0000 7f3e8fe00640 1 --2- 172.21.15.28:0/1196759876 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3e90108770 0x7f3e9019e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:06.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.317+0000 7f3e95200640 1 -- 172.21.15.28:0/1196759876 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3e90108770 msgr2=0x7f3e9019e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:06.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.317+0000 7f3e95200640 1 --2- 172.21.15.28:0/1196759876 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3e90108770 0x7f3e9019e9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:06.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.317+0000 7f3e95200640 1 -- 172.21.15.28:0/1196759876 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e9010ddf0 msgr2=0x7f3e901a2d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:06.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.317+0000 7f3e95200640 1 --2- 172.21.15.28:0/1196759876 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e9010ddf0 0x7f3e901a2d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:06.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.317+0000 7f3e95200640 1 -- 172.21.15.28:0/1196759876 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e901a3490 con 0x7f3e90107dd0 2024-05-12T22:19:06.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.317+0000 7f3e8fe00640 1 --2- 172.21.15.28:0/1196759876 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3e90108770 0x7f3e9019e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:19:06.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.318+0000 7f3e95200640 1 --2- 172.21.15.28:0/1196759876 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3e90107dd0 0x7f3e9019e4a0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f3e8000eab0 tx=0x7f3e8000ef70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:19:06.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.318+0000 7f3e8d600640 1 -- 172.21.15.28:0/1196759876 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e8000ce40 con 0x7f3e90107dd0 2024-05-12T22:19:06.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.318+0000 7f3e96600640 1 -- 172.21.15.28:0/1196759876 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e901a3780 con 0x7f3e90107dd0 2024-05-12T22:19:06.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.318+0000 7f3e8d600640 1 -- 172.21.15.28:0/1196759876 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e80011950 con 0x7f3e90107dd0 2024-05-12T22:19:06.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.318+0000 7f3e8d600640 1 -- 172.21.15.28:0/1196759876 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e80010ae0 con 0x7f3e90107dd0 2024-05-12T22:19:06.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.318+0000 7f3e96600640 1 -- 172.21.15.28:0/1196759876 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e901ab010 con 0x7f3e90107dd0 2024-05-12T22:19:06.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.321+0000 7f3e8d600640 1 -- 172.21.15.28:0/1196759876 <== mon.2 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7f3e80011b00 con 0x7f3e90107dd0 2024-05-12T22:19:06.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.321+0000 7f3e9aa1b740 1 Processor -- start 2024-05-12T22:19:06.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.322+0000 7f3e9aa1b740 1 -- start start 2024-05-12T22:19:06.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.322+0000 7f3e9aa1b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55686da3e500 0x55686da3e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:06.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.322+0000 7f3e9aa1b740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55686da3eeb0 0x55686da48f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:06.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.322+0000 7f3e9aa1b740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55686da49520 0x55686da4b970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:06.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.322+0000 7f3e9aa1b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55686d9a27c0 con 0x55686da3e500 2024-05-12T22:19:06.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.322+0000 7f3e9aa1b740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x55686d94a790 con 0x55686da3eeb0 2024-05-12T22:19:06.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.322+0000 7f3e9aa1b740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55686d99e780 con 0x55686da49520 2024-05-12T22:19:06.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.322+0000 7f3e95c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55686da3e500 0x55686da3e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:06.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.322+0000 7f3e95c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55686da3e500 0x55686da3e8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55024/0 (socket says 172.21.15.28:55024) 2024-05-12T22:19:06.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.322+0000 7f3e95c00640 1 -- 172.21.15.28:0/2140802665 learned_addr learned my addr 172.21.15.28:0/2140802665 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:19:06.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.322+0000 7f3e8fe00640 1 --2- 172.21.15.28:0/2140802665 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55686da49520 0x55686da4b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:06.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.322+0000 7f3e95200640 1 --2- 172.21.15.28:0/2140802665 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55686da3eeb0 0x55686da48f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:06.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.323+0000 7f3e95c00640 1 -- 172.21.15.28:0/2140802665 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55686da49520 msgr2=0x55686da4b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:19:06.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.323+0000 7f3e95c00640 1 --2- 172.21.15.28:0/2140802665 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55686da49520 0x55686da4b970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:06.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.323+0000 7f3e95c00640 1 -- 172.21.15.28:0/2140802665 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55686da3eeb0 msgr2=0x55686da48f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:19:06.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.323+0000 7f3e95c00640 1 --2- 172.21.15.28:0/2140802665 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55686da3eeb0 0x55686da48f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:06.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.323+0000 7f3e95c00640 1 -- 172.21.15.28:0/2140802665 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55686d96a6a0 con 0x55686da3e500 2024-05-12T22:19:06.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.323+0000 7f3e95200640 1 --2- 172.21.15.28:0/2140802665 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55686da3eeb0 0x55686da48f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:19:06.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.323+0000 7f3e8fe00640 1 --2- 172.21.15.28:0/2140802665 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55686da49520 0x55686da4b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:19:06.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.323+0000 7f3e8d600640 1 --2- 172.21.15.28:0/1196759876 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f3e740a2d00 0x7f3e740a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:06.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.323+0000 7f3e95c00640 1 --2- 172.21.15.28:0/2140802665 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55686da3e500 0x55686da3e8e0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f3e8800ac00 tx=0x7f3e8800f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b32a7b02873d603 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:19:06.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.323+0000 7f3e8d600640 1 -- 172.21.15.28:0/1196759876 <== mon.2 v2:172.21.15.86:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8513+0+0 (secure 0 0 0) 0x7f3e800cfad0 con 0x7f3e90107dd0 2024-05-12T22:19:06.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.324+0000 7f3e95200640 1 --2- 172.21.15.28:0/1196759876 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f3e740a2d00 0x7f3e740a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:06.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.324+0000 7f3e86000640 1 -- 172.21.15.28:0/2140802665 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e88005220 con 0x55686da3e500 2024-05-12T22:19:06.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.324+0000 7f3e86000640 1 -- 172.21.15.28:0/2140802665 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e88004c80 con 0x55686da3e500 2024-05-12T22:19:06.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.324+0000 7f3e86000640 1 -- 172.21.15.28:0/2140802665 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e88018840 con 0x55686da3e500 2024-05-12T22:19:06.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.324+0000 7f3e9aa1b740 1 -- 172.21.15.28:0/2140802665 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55686da3e500 msgr2=0x55686da3e8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:06.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.324+0000 7f3e95200640 1 --2- 172.21.15.28:0/1196759876 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f3e740a2d00 0x7f3e740a51c0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x55686da3fac0 tx=0x7f3e80010540 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:19:06.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.324+0000 7f3e9aa1b740 1 --2- 172.21.15.28:0/2140802665 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55686da3e500 0x55686da3e8e0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f3e8800ac00 tx=0x7f3e8800f790 comp rx=0 tx=0).stop 2024-05-12T22:19:06.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.325+0000 7f3e9aa1b740 1 -- 172.21.15.28:0/2140802665 shutdown_connections 2024-05-12T22:19:06.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.325+0000 7f3e9aa1b740 1 --2- 172.21.15.28:0/2140802665 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55686da49520 0x55686da4b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:06.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.325+0000 7f3e9aa1b740 1 --2- 172.21.15.28:0/2140802665 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55686da3eeb0 0x55686da48f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:06.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.325+0000 7f3e9aa1b740 1 --2- 172.21.15.28:0/2140802665 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55686da3e500 0x55686da3e8e0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:06.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.325+0000 7f3e9aa1b740 1 -- 172.21.15.28:0/2140802665 >> 172.21.15.28:0/2140802665 conn(0x55686da3a5b0 msgr2=0x55686da3a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:19:06.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.325+0000 7f3e9aa1b740 1 -- 172.21.15.28:0/2140802665 shutdown_connections 2024-05-12T22:19:06.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.325+0000 7f3e9aa1b740 1 -- 172.21.15.28:0/2140802665 wait complete. 2024-05-12T22:19:06.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.325+0000 7f3e9aa1b740 1 -- start start 2024-05-12T22:19:06.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.325+0000 7f3e9aa1b740 1 Processor -- start 2024-05-12T22:19:06.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.326+0000 7f3e9aa1b740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55686da3e500 0x55686da46430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:06.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.326+0000 7f3e9aa1b740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55686da3eeb0 0x55686da46970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:06.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.326+0000 7f3e9aa1b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55686da49520 0x55686da46eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:06.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.326+0000 7f3e9aa1b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55686da4c530 con 0x55686da49520 2024-05-12T22:19:06.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.326+0000 7f3e9aa1b740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x55686d95da90 con 0x55686da3e500 2024-05-12T22:19:06.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.326+0000 7f3e9aa1b740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55686da4c6b0 con 0x55686da3eeb0 2024-05-12T22:19:06.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.326+0000 7f3e9aa1b740 10 client.0 _ll_register_callbacks cb 0x55686d998af0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:19:06.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.326+0000 7f3e8fe00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55686da3eeb0 0x55686da46970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:06.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.326+0000 7f3e8fe00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55686da3eeb0 0x55686da46970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:37950/0 (socket says 172.21.15.28:37950) 2024-05-12T22:19:06.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.326+0000 7f3e8fe00640 1 -- 172.21.15.28:0/4101004115 learned_addr learned my addr 172.21.15.28:0/4101004115 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:19:06.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.326+0000 7f3e95c00640 1 --2- 172.21.15.28:0/4101004115 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55686da49520 0x55686da46eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:06.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.326+0000 7f3e95c00640 1 --2- 172.21.15.28:0/4101004115 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55686da3e500 0x55686da46430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:06.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.327+0000 7f3e8fe00640 1 -- 172.21.15.28:0/4101004115 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55686da3e500 msgr2=0x55686da46430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:19:06.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.327+0000 7f3e8fe00640 1 --2- 172.21.15.28:0/4101004115 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55686da3e500 0x55686da46430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:06.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.327+0000 7f3e8fe00640 1 -- 172.21.15.28:0/4101004115 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55686da49520 msgr2=0x55686da46eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:19:06.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.327+0000 7f3e8fe00640 1 --2- 172.21.15.28:0/4101004115 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55686da49520 0x55686da46eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:06.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.327+0000 7f3e8fe00640 1 -- 172.21.15.28:0/4101004115 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55686d77e7e0 con 0x55686da3eeb0 2024-05-12T22:19:06.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.327+0000 7f3e95c00640 1 --2- 172.21.15.28:0/4101004115 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55686da3e500 0x55686da46430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:19:06.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.327+0000 7f3e8fe00640 1 --2- 172.21.15.28:0/4101004115 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55686da3eeb0 0x55686da46970 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f3e78033000 tx=0x7f3e78005440 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c1dbf7d019686977 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:19:06.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.327+0000 7f3e7ea00640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-12T22:19:06.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.327+0000 7f3e7ea00640 1 -- 172.21.15.28:0/4101004115 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e78035040 con 0x55686da3eeb0 2024-05-12T22:19:06.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.327+0000 7f3e7ea00640 1 -- 172.21.15.28:0/4101004115 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8513+0+0 (secure 0 0 0) 0x7f3e78004b70 con 0x55686da3eeb0 2024-05-12T22:19:06.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.327+0000 7f3e9aa1b740 1 -- 172.21.15.28:0/4101004115 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3e88004c80 con 0x55686da3eeb0 2024-05-12T22:19:06.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.328+0000 7f3e7ea00640 1 -- 172.21.15.28:0/4101004115 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f3e78004d70 con 0x55686da3eeb0 2024-05-12T22:19:06.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.328+0000 7f3e9aa1b740 10 client.27406 fetch_fsmap learned FSMap version 26 2024-05-12T22:19:06.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.328+0000 7f3e9aa1b740 1 -- 172.21.15.28:0/4101004115 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55686d7def70 con 0x55686da3eeb0 2024-05-12T22:19:06.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.328+0000 7f3e7ea00640 1 -- 172.21.15.28:0/4101004115 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e88004c80 con 0x55686da3eeb0 2024-05-12T22:19:06.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.328+0000 7f3e7ea00640 1 -- 172.21.15.28:0/4101004115 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f3e780037b0 con 0x55686da3eeb0 2024-05-12T22:19:06.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.328+0000 7f3e9aa1b740 10 client.27406 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:19:06.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.328+0000 7f3e9aa1b740 10 client.27406 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:19:06.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.328+0000 7f3e9aa1b740 20 client.27406 populate_metadata read hostname 'smithi028' 2024-05-12T22:19:06.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.328+0000 7f3e9aa1b740 1 --2- 172.21.15.28:0/4101004115 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55686da59760 0x55686da5bb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:06.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.328+0000 7f3e9aa1b740 4 client.27406 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-12T22:19:06.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.329+0000 7f3e9aa1b740 1 -- 172.21.15.28:0/4101004115 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55686d9b63a0 con 0x55686da59760 2024-05-12T22:19:06.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.329+0000 7f3e95c00640 1 --2- 172.21.15.28:0/4101004115 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55686da59760 0x55686da5bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:06.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.329+0000 7f3e95c00640 1 --2- 172.21.15.28:0/4101004115 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55686da59760 0x55686da5bb80 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d76e0864941d4804 server_cookie=252be9eab76546b6 in_seq=0 out_seq=0 2024-05-12T22:19:06.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.329+0000 7f3e7ea00640 10 client.27406 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:19:06.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.330+0000 7f3e7ea00640 1 -- 172.21.15.28:0/4101004115 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55686d9b63a0 con 0x55686da59760 2024-05-12T22:19:06.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.330+0000 7f3e7ea00640 10 client.27406 handle_command_reply: tid=0 2024-05-12T22:19:06.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.330+0000 7f3e9aa1b740 1 client.27406 shutdown 2024-05-12T22:19:06.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.330+0000 7f3e9aa1b740 1 -- 172.21.15.28:0/4101004115 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55686da3eeb0 msgr2=0x55686da46970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:06.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.330+0000 7f3e9aa1b740 1 --2- 172.21.15.28:0/4101004115 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55686da3eeb0 0x55686da46970 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f3e78033000 tx=0x7f3e78005440 comp rx=0 tx=0).stop 2024-05-12T22:19:06.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.331+0000 7f3e9aa1b740 1 -- 172.21.15.28:0/4101004115 shutdown_connections 2024-05-12T22:19:06.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.331+0000 7f3e9aa1b740 1 --2- 172.21.15.28:0/4101004115 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55686da59760 0x55686da5bb80 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:06.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.331+0000 7f3e9aa1b740 1 --2- 172.21.15.28:0/4101004115 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55686da49520 0x55686da46eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:06.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.331+0000 7f3e9aa1b740 1 --2- 172.21.15.28:0/4101004115 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55686da3eeb0 0x55686da46970 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:06.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.331+0000 7f3e7ea00640 0 client.27406 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:19:06.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.331+0000 7f3e9aa1b740 1 --2- 172.21.15.28:0/4101004115 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55686da3e500 0x55686da46430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:06.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.331+0000 7f3e9aa1b740 1 -- 172.21.15.28:0/4101004115 >> 172.21.15.28:0/4101004115 conn(0x55686da3a5b0 msgr2=0x55686da40630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:19:06.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.331+0000 7f3e9aa1b740 1 -- 172.21.15.28:0/4101004115 shutdown_connections 2024-05-12T22:19:06.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.331+0000 7f3e9aa1b740 1 -- 172.21.15.28:0/4101004115 wait complete. 2024-05-12T22:19:06.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.331+0000 7f3e9aa1b740 20 client.27406 trim_cache size 0 max 16384 2024-05-12T22:19:06.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.350+0000 7f3e9aa1b740 1 Processor -- start 2024-05-12T22:19:06.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.351+0000 7f3e9aa1b740 1 -- start start 2024-05-12T22:19:06.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.351+0000 7f3e9aa1b740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55686da3e500 0x55686da553f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:06.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.351+0000 7f3e9aa1b740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55686da3eeb0 0x55686da46cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:06.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.351+0000 7f3e9aa1b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55686da49520 0x55686da61490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:06.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.351+0000 7f3e9aa1b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3e901b6090 con 0x55686da49520 2024-05-12T22:19:06.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.351+0000 7f3e9aa1b740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f3e901b3960 con 0x55686da3eeb0 2024-05-12T22:19:06.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.351+0000 7f3e9aa1b740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f3e901b17d0 con 0x55686da3e500 2024-05-12T22:19:06.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.351+0000 7f3e95c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55686da49520 0x55686da61490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:06.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.351+0000 7f3e8fe00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55686da3eeb0 0x55686da46cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:06.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.351+0000 7f3e8fe00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55686da3eeb0 0x55686da46cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:41066/0 (socket says 172.21.15.28:41066) 2024-05-12T22:19:06.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.351+0000 7f3e8fe00640 1 -- 172.21.15.28:0/652088274 learned_addr learned my addr 172.21.15.28:0/652088274 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:19:06.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.351+0000 7f3e95c00640 1 --2- 172.21.15.28:0/652088274 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55686da49520 0x55686da61490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55046/0 (socket says 172.21.15.28:55046) 2024-05-12T22:19:06.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.352+0000 7f3e95c00640 1 --2- 172.21.15.28:0/652088274 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55686da3e500 0x55686da553f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:06.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.352+0000 7f3e8fe00640 1 -- 172.21.15.28:0/652088274 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55686da3e500 msgr2=0x55686da553f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:19:06.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.352+0000 7f3e8fe00640 1 --2- 172.21.15.28:0/652088274 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55686da3e500 0x55686da553f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:06.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.352+0000 7f3e8fe00640 1 -- 172.21.15.28:0/652088274 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55686da49520 msgr2=0x55686da61490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:19:06.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.352+0000 7f3e8fe00640 1 --2- 172.21.15.28:0/652088274 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55686da49520 0x55686da61490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:06.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.352+0000 7f3e8fe00640 1 -- 172.21.15.28:0/652088274 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55686d676170 con 0x55686da3eeb0 2024-05-12T22:19:06.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.352+0000 7f3e95c00640 1 --2- 172.21.15.28:0/652088274 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55686da49520 0x55686da61490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:19:06.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.352+0000 7f3e8fe00640 1 --2- 172.21.15.28:0/652088274 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55686da3eeb0 0x55686da46cd0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f3e7802a9a0 tx=0x7f3e78003f50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8a558043bd5bf773 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:19:06.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.353+0000 7f3e85600640 1 -- 172.21.15.28:0/652088274 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e78035030 con 0x55686da3eeb0 2024-05-12T22:19:06.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.353+0000 7f3e85600640 1 -- 172.21.15.28:0/652088274 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e78005850 con 0x55686da3eeb0 2024-05-12T22:19:06.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.353+0000 7f3e85600640 1 -- 172.21.15.28:0/652088274 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e7804f8f0 con 0x55686da3eeb0 2024-05-12T22:19:06.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.353+0000 7f3e9aa1b740 1 -- 172.21.15.28:0/652088274 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55686da3eeb0 msgr2=0x55686da46cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:06.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.353+0000 7f3e9aa1b740 1 --2- 172.21.15.28:0/652088274 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55686da3eeb0 0x55686da46cd0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f3e7802a9a0 tx=0x7f3e78003f50 comp rx=0 tx=0).stop 2024-05-12T22:19:06.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.353+0000 7f3e9aa1b740 1 -- 172.21.15.28:0/652088274 shutdown_connections 2024-05-12T22:19:06.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.353+0000 7f3e9aa1b740 1 --2- 172.21.15.28:0/652088274 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55686da49520 0x55686da61490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:06.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.353+0000 7f3e9aa1b740 1 --2- 172.21.15.28:0/652088274 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55686da3eeb0 0x55686da46cd0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:06.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.353+0000 7f3e9aa1b740 1 --2- 172.21.15.28:0/652088274 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55686da3e500 0x55686da553f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:06.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.353+0000 7f3e9aa1b740 1 -- 172.21.15.28:0/652088274 >> 172.21.15.28:0/652088274 conn(0x55686da3a5b0 msgr2=0x55686da39730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:19:06.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.354+0000 7f3e9aa1b740 1 -- 172.21.15.28:0/652088274 shutdown_connections 2024-05-12T22:19:06.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.354+0000 7f3e9aa1b740 1 -- 172.21.15.28:0/652088274 wait complete. 2024-05-12T22:19:06.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.354+0000 7f3e9aa1b740 1 -- start start 2024-05-12T22:19:06.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.354+0000 7f3e9aa1b740 1 Processor -- start 2024-05-12T22:19:06.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.355+0000 7f3e9aa1b740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55686da3e500 0x55686da4d570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:06.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.355+0000 7f3e9aa1b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55686da3eeb0 0x55686da4dab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:06.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.355+0000 7f3e9aa1b740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55686da49520 0x55686da4dff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:06.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.355+0000 7f3e9aa1b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3e901b1e20 con 0x55686da3eeb0 2024-05-12T22:19:06.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.355+0000 7f3e9aa1b740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f3e901b3290 con 0x55686da3e500 2024-05-12T22:19:06.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.355+0000 7f3e9aa1b740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f3e901b24f0 con 0x55686da49520 2024-05-12T22:19:06.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.355+0000 7f3e9aa1b740 10 client.0 _ll_register_callbacks cb 0x7f3e780043f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:19:06.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.355+0000 7f3e95c00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55686da3e500 0x55686da4d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:06.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.355+0000 7f3e95c00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55686da3e500 0x55686da4d570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:41074/0 (socket says 172.21.15.28:41074) 2024-05-12T22:19:06.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.355+0000 7f3e95c00640 1 -- 172.21.15.28:0/1855774774 learned_addr learned my addr 172.21.15.28:0/1855774774 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:19:06.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.355+0000 7f3e8fe00640 1 --2- 172.21.15.28:0/1855774774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55686da3eeb0 0x55686da4dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:06.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.355+0000 7f3e95c00640 1 --2- 172.21.15.28:0/1855774774 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55686da49520 0x55686da4dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:06.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.356+0000 7f3e8fe00640 1 -- 172.21.15.28:0/1855774774 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55686da49520 msgr2=0x55686da4dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:19:06.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.356+0000 7f3e8fe00640 1 --2- 172.21.15.28:0/1855774774 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55686da49520 0x55686da4dff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:06.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.356+0000 7f3e8fe00640 1 -- 172.21.15.28:0/1855774774 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55686da3e500 msgr2=0x55686da4d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:19:06.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.356+0000 7f3e8fe00640 1 --2- 172.21.15.28:0/1855774774 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55686da3e500 0x55686da4d570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:06.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.356+0000 7f3e8fe00640 1 -- 172.21.15.28:0/1855774774 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55686dabef50 con 0x55686da3eeb0 2024-05-12T22:19:06.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.356+0000 7f3e95c00640 1 --2- 172.21.15.28:0/1855774774 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55686da49520 0x55686da4dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:19:06.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.356+0000 7f3e8fe00640 1 --2- 172.21.15.28:0/1855774774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55686da3eeb0 0x55686da4dab0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f3e78003d40 tx=0x7f3e78031480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45fcc9f9166e1b5d server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:19:06.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.356+0000 7f3e7ea00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-12T22:19:06.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.356+0000 7f3e7ea00640 1 -- 172.21.15.28:0/1855774774 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e78035040 con 0x55686da3eeb0 2024-05-12T22:19:06.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.357+0000 7f3e9aa1b740 1 -- 172.21.15.28:0/1855774774 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3e78005850 con 0x55686da3eeb0 2024-05-12T22:19:06.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.357+0000 7f3e7ea00640 1 -- 172.21.15.28:0/1855774774 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8513+0+0 (secure 0 0 0) 0x7f3e7804f5c0 con 0x55686da3eeb0 2024-05-12T22:19:06.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.357+0000 7f3e7ea00640 1 -- 172.21.15.28:0/1855774774 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f3e78005420 con 0x55686da3eeb0 2024-05-12T22:19:06.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.357+0000 7f3e9aa1b740 10 client.17325 fetch_fsmap learned FSMap version 26 2024-05-12T22:19:06.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.357+0000 7f3e9aa1b740 1 -- 172.21.15.28:0/1855774774 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55686dac0290 con 0x55686da3eeb0 2024-05-12T22:19:06.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.357+0000 7f3e7ea00640 1 -- 172.21.15.28:0/1855774774 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e78005850 con 0x55686da3eeb0 2024-05-12T22:19:06.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.357+0000 7f3e7ea00640 1 -- 172.21.15.28:0/1855774774 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f3e780316d0 con 0x55686da3eeb0 2024-05-12T22:19:06.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.357+0000 7f3e9aa1b740 10 client.17325 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:19:06.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.357+0000 7f3e9aa1b740 10 client.17325 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:19:06.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.357+0000 7f3e9aa1b740 20 client.17325 populate_metadata read hostname 'smithi028' 2024-05-12T22:19:06.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.357+0000 7f3e9aa1b740 1 --2- 172.21.15.28:0/1855774774 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55686dac0900 0x55686dac2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:06.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.358+0000 7f3e9aa1b740 4 client.17325 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-12T22:19:06.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.358+0000 7f3e9aa1b740 1 -- 172.21.15.28:0/1855774774 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55686d9afef0 con 0x55686dac0900 2024-05-12T22:19:06.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.358+0000 7f3e95c00640 1 --2- 172.21.15.28:0/1855774774 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55686dac0900 0x55686dac2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:06.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.358+0000 7f3e95c00640 1 --2- 172.21.15.28:0/1855774774 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55686dac0900 0x55686dac2cf0 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1dab3de2b42b6cb4 server_cookie=8b3b155ffb06a81c in_seq=0 out_seq=0 2024-05-12T22:19:06.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.358+0000 7f3e7ea00640 10 client.17325 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:19:06.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.360+0000 7f3e7ea00640 1 -- 172.21.15.28:0/1855774774 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55686d9afef0 con 0x55686dac0900 2024-05-12T22:19:06.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.360+0000 7f3e7ea00640 10 client.17325 handle_command_reply: tid=0 2024-05-12T22:19:06.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.360+0000 7f3e9aa1b740 1 client.17325 shutdown 2024-05-12T22:19:06.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.360+0000 7f3e9aa1b740 1 -- 172.21.15.28:0/1855774774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55686da3eeb0 msgr2=0x55686da4dab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:06.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.360+0000 7f3e9aa1b740 1 --2- 172.21.15.28:0/1855774774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55686da3eeb0 0x55686da4dab0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f3e78003d40 tx=0x7f3e78031480 comp rx=0 tx=0).stop 2024-05-12T22:19:06.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.360+0000 7f3e9aa1b740 1 -- 172.21.15.28:0/1855774774 shutdown_connections 2024-05-12T22:19:06.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.360+0000 7f3e9aa1b740 1 --2- 172.21.15.28:0/1855774774 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x55686dac0900 0x55686dac2cf0 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:06.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.360+0000 7f3e9aa1b740 1 --2- 172.21.15.28:0/1855774774 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55686da49520 0x55686da4dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:06.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.360+0000 7f3e9aa1b740 1 --2- 172.21.15.28:0/1855774774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55686da3eeb0 0x55686da4dab0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:06.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.360+0000 7f3e9aa1b740 1 --2- 172.21.15.28:0/1855774774 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55686da3e500 0x55686da4d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:06.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.360+0000 7f3e7ea00640 0 client.17325 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:19:06.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.360+0000 7f3e9aa1b740 1 -- 172.21.15.28:0/1855774774 >> 172.21.15.28:0/1855774774 conn(0x55686da3a5b0 msgr2=0x55686da3fbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:19:06.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.360+0000 7f3e9aa1b740 1 -- 172.21.15.28:0/1855774774 shutdown_connections 2024-05-12T22:19:06.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.360+0000 7f3e9aa1b740 1 -- 172.21.15.28:0/1855774774 wait complete. 2024-05-12T22:19:06.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.360+0000 7f3e9aa1b740 20 client.17325 trim_cache size 0 max 16384 2024-05-12T22:19:06.366 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-05-12T22:19:06.366 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-05-12T22:19:06.366 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "41de51df-fccc-4091-a87b-7e718236f742", 2024-05-12T22:19:06.366 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-05-12T22:19:06.366 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-05-12T22:19:06.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.361+0000 7f3e96600640 1 -- 172.21.15.28:0/1196759876 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f3e740a2d00 msgr2=0x7f3e740a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:06.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.361+0000 7f3e96600640 1 --2- 172.21.15.28:0/1196759876 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f3e740a2d00 0x7f3e740a51c0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x55686da3fac0 tx=0x7f3e80010540 comp rx=0 tx=0).stop 2024-05-12T22:19:06.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.361+0000 7f3e96600640 1 -- 172.21.15.28:0/1196759876 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3e90107dd0 msgr2=0x7f3e9019e4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:06.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.361+0000 7f3e96600640 1 --2- 172.21.15.28:0/1196759876 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3e90107dd0 0x7f3e9019e4a0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f3e8000eab0 tx=0x7f3e8000ef70 comp rx=0 tx=0).stop 2024-05-12T22:19:06.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.361+0000 7f3e96600640 1 -- 172.21.15.28:0/1196759876 shutdown_connections 2024-05-12T22:19:06.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.361+0000 7f3e96600640 1 --2- 172.21.15.28:0/1196759876 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f3e740a2d00 0x7f3e740a51c0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:06.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.361+0000 7f3e96600640 1 --2- 172.21.15.28:0/1196759876 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e9010ddf0 0x7f3e901a2d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:06.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.361+0000 7f3e96600640 1 --2- 172.21.15.28:0/1196759876 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3e90108770 0x7f3e9019e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:06.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.361+0000 7f3e96600640 1 --2- 172.21.15.28:0/1196759876 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3e90107dd0 0x7f3e9019e4a0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:06.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.361+0000 7f3e96600640 1 -- 172.21.15.28:0/1196759876 >> 172.21.15.28:0/1196759876 conn(0x7f3e90077650 msgr2=0x7f3e9010ba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:19:06.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.362+0000 7f3e96600640 1 -- 172.21.15.28:0/1196759876 shutdown_connections 2024-05-12T22:19:06.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.362+0000 7f3e96600640 1 -- 172.21.15.28:0/1196759876 wait complete. 2024-05-12T22:19:06.383 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-12T22:19:06.383 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-12T22:19:06.453 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:19:06 smithi028 ceph-mon[29847]: pgmap v312: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-05-12T22:19:06.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.621+0000 7f7475600640 1 -- 172.21.15.28:0/4134608172 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f747010ddf0 msgr2=0x7f747010e1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:06.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.621+0000 7f7475600640 1 --2- 172.21.15.28:0/4134608172 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f747010ddf0 0x7f747010e1d0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f7460009900 tx=0x7f746002a480 comp rx=0 tx=0).stop 2024-05-12T22:19:06.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.621+0000 7f7475600640 1 -- 172.21.15.28:0/4134608172 shutdown_connections 2024-05-12T22:19:06.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.621+0000 7f7475600640 1 --2- 172.21.15.28:0/4134608172 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7470108770 0x7f7470114b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:06.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.621+0000 7f7475600640 1 --2- 172.21.15.28:0/4134608172 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7470107dd0 0x7f7470108230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:06.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.621+0000 7f7475600640 1 --2- 172.21.15.28:0/4134608172 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f747010ddf0 0x7f747010e1d0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:06.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.621+0000 7f7475600640 1 -- 172.21.15.28:0/4134608172 >> 172.21.15.28:0/4134608172 conn(0x7f7470077650 msgr2=0x7f7470106650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:19:06.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.621+0000 7f7475600640 1 -- 172.21.15.28:0/4134608172 shutdown_connections 2024-05-12T22:19:06.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.622+0000 7f7475600640 1 -- 172.21.15.28:0/4134608172 wait complete. 2024-05-12T22:19:06.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.623+0000 7f7475600640 1 Processor -- start 2024-05-12T22:19:06.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.623+0000 7f7475600640 1 -- start start 2024-05-12T22:19:06.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.623+0000 7f7475600640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7470107dd0 0x7f747019e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:06.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.623+0000 7f7475600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7470108770 0x7f747019ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:06.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.623+0000 7f7475600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f747010ddf0 0x7f74701a2e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:06.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.624+0000 7f7475600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7470118100 con 0x7f7470108770 2024-05-12T22:19:06.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.624+0000 7f7475600640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f7470117f80 con 0x7f747010ddf0 2024-05-12T22:19:06.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.624+0000 7f7475600640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f7470118280 con 0x7f7470107dd0 2024-05-12T22:19:06.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.624+0000 7f746fe00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7470107dd0 0x7f747019e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:06.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.624+0000 7f746fe00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7470107dd0 0x7f747019e540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:37988/0 (socket says 172.21.15.28:37988) 2024-05-12T22:19:06.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.624+0000 7f746fe00640 1 -- 172.21.15.28:0/1541419496 learned_addr learned my addr 172.21.15.28:0/1541419496 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:19:06.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.624+0000 7f7474c00640 1 --2- 172.21.15.28:0/1541419496 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f747010ddf0 0x7f74701a2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:06.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.624+0000 7f7467e00640 1 --2- 172.21.15.28:0/1541419496 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7470108770 0x7f747019ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:06.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.624+0000 7f746fe00640 1 -- 172.21.15.28:0/1541419496 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f747010ddf0 msgr2=0x7f74701a2e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:06.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.624+0000 7f746fe00640 1 --2- 172.21.15.28:0/1541419496 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f747010ddf0 0x7f74701a2e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:06.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.624+0000 7f746fe00640 1 -- 172.21.15.28:0/1541419496 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7470108770 msgr2=0x7f747019ea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:06.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.624+0000 7f746fe00640 1 --2- 172.21.15.28:0/1541419496 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7470108770 0x7f747019ea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:06.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.624+0000 7f746fe00640 1 -- 172.21.15.28:0/1541419496 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74701a3530 con 0x7f7470107dd0 2024-05-12T22:19:06.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.624+0000 7f7467e00640 1 --2- 172.21.15.28:0/1541419496 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7470108770 0x7f747019ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:19:06.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.624+0000 7f7474c00640 1 --2- 172.21.15.28:0/1541419496 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f747010ddf0 0x7f74701a2e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:19:06.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.625+0000 7f746fe00640 1 --2- 172.21.15.28:0/1541419496 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7470107dd0 0x7f747019e540 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f7460004770 tx=0x7f7460003c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:19:06.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.625+0000 7f746d600640 1 -- 172.21.15.28:0/1541419496 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f746003f070 con 0x7f7470107dd0 2024-05-12T22:19:06.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.625+0000 7f746d600640 1 -- 172.21.15.28:0/1541419496 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7460031ca0 con 0x7f7470107dd0 2024-05-12T22:19:06.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.625+0000 7f7475600640 1 -- 172.21.15.28:0/1541419496 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74701a37c0 con 0x7f7470107dd0 2024-05-12T22:19:06.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.625+0000 7f746d600640 1 -- 172.21.15.28:0/1541419496 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7460035030 con 0x7f7470107dd0 2024-05-12T22:19:06.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.625+0000 7f7475600640 1 -- 172.21.15.28:0/1541419496 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74701ab0b0 con 0x7f7470107dd0 2024-05-12T22:19:06.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.629+0000 7f746d600640 1 -- 172.21.15.28:0/1541419496 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7f7460042070 con 0x7f7470107dd0 2024-05-12T22:19:06.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.629+0000 7f7479a52740 1 Processor -- start 2024-05-12T22:19:06.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.629+0000 7f7479a52740 1 -- start start 2024-05-12T22:19:06.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.630+0000 7f7479a52740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5616c2a71dd0 0x5616c2a721b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:06.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.630+0000 7f7479a52740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5616c2a726f0 0x5616c2a7c880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:06.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.630+0000 7f7479a52740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5616c2a7cdc0 0x5616c2a7f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:06.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.630+0000 7f7479a52740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5616c2a3a2a0 con 0x5616c2a71dd0 2024-05-12T22:19:06.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.630+0000 7f7479a52740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x5616c29d6b90 con 0x5616c2a726f0 2024-05-12T22:19:06.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.630+0000 7f7479a52740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5616c297e5b0 con 0x5616c2a7cdc0 2024-05-12T22:19:06.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.630+0000 7f746fe00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5616c2a726f0 0x5616c2a7c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:06.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.630+0000 7f7467e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5616c2a7cdc0 0x5616c2a7f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:06.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.630+0000 7f7467e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5616c2a7cdc0 0x5616c2a7f1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:37994/0 (socket says 172.21.15.28:37994) 2024-05-12T22:19:06.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.630+0000 7f7467e00640 1 -- 172.21.15.28:0/1212587595 learned_addr learned my addr 172.21.15.28:0/1212587595 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:19:06.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.630+0000 7f746fe00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5616c2a726f0 0x5616c2a7c880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:41108/0 (socket says 172.21.15.28:41108) 2024-05-12T22:19:06.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.630+0000 7f7474c00640 1 --2- 172.21.15.28:0/1212587595 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5616c2a71dd0 0x5616c2a721b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:06.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.631+0000 7f7467e00640 1 -- 172.21.15.28:0/1212587595 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5616c2a726f0 msgr2=0x5616c2a7c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:19:06.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.631+0000 7f7467e00640 1 --2- 172.21.15.28:0/1212587595 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5616c2a726f0 0x5616c2a7c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:06.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.631+0000 7f7467e00640 1 -- 172.21.15.28:0/1212587595 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5616c2a71dd0 msgr2=0x5616c2a721b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:19:06.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.631+0000 7f7467e00640 1 --2- 172.21.15.28:0/1212587595 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5616c2a71dd0 0x5616c2a721b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:06.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.631+0000 7f7467e00640 1 -- 172.21.15.28:0/1212587595 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5616c27b47d0 con 0x5616c2a7cdc0 2024-05-12T22:19:06.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.631+0000 7f746fe00640 1 --2- 172.21.15.28:0/1212587595 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5616c2a726f0 0x5616c2a7c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:19:06.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.631+0000 7f7467e00640 1 --2- 172.21.15.28:0/1212587595 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5616c2a7cdc0 0x5616c2a7f1b0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f745800bd30 tx=0x7f745800f730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fb1b5ee8977e7e10 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:19:06.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.631+0000 7f746d600640 1 --2- 172.21.15.28:0/1541419496 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f74540a2d00 0x7f74540a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:06.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.632+0000 7f746d600640 1 -- 172.21.15.28:0/1541419496 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8513+0+0 (secure 0 0 0) 0x7f74600efc80 con 0x7f7470107dd0 2024-05-12T22:19:06.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.632+0000 7f7474c00640 1 --2- 172.21.15.28:0/1541419496 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f74540a2d00 0x7f74540a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:06.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.632+0000 7f7465200640 1 -- 172.21.15.28:0/1212587595 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7458015070 con 0x5616c2a7cdc0 2024-05-12T22:19:06.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.632+0000 7f7465200640 1 -- 172.21.15.28:0/1212587595 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f745800be20 con 0x5616c2a7cdc0 2024-05-12T22:19:06.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.632+0000 7f7465200640 1 -- 172.21.15.28:0/1212587595 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7458011aa0 con 0x5616c2a7cdc0 2024-05-12T22:19:06.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.633+0000 7f7479a52740 1 -- 172.21.15.28:0/1212587595 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5616c2a7cdc0 msgr2=0x5616c2a7f1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:06.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.633+0000 7f7474c00640 1 --2- 172.21.15.28:0/1541419496 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f74540a2d00 0x7f74540a51c0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f7468005fd0 tx=0x7f74680095f0 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:19:06.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.633+0000 7f7479a52740 1 --2- 172.21.15.28:0/1212587595 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5616c2a7cdc0 0x5616c2a7f1b0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f745800bd30 tx=0x7f745800f730 comp rx=0 tx=0).stop 2024-05-12T22:19:06.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.633+0000 7f7479a52740 1 -- 172.21.15.28:0/1212587595 shutdown_connections 2024-05-12T22:19:06.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.633+0000 7f7479a52740 1 --2- 172.21.15.28:0/1212587595 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5616c2a7cdc0 0x5616c2a7f1b0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:06.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.633+0000 7f7479a52740 1 --2- 172.21.15.28:0/1212587595 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5616c2a726f0 0x5616c2a7c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:06.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.633+0000 7f7479a52740 1 --2- 172.21.15.28:0/1212587595 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5616c2a71dd0 0x5616c2a721b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:06.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.633+0000 7f7479a52740 1 -- 172.21.15.28:0/1212587595 >> 172.21.15.28:0/1212587595 conn(0x5616c2a6e0f0 msgr2=0x5616c2a6e4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:19:06.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.633+0000 7f7479a52740 1 -- 172.21.15.28:0/1212587595 shutdown_connections 2024-05-12T22:19:06.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.633+0000 7f7479a52740 1 -- 172.21.15.28:0/1212587595 wait complete. 2024-05-12T22:19:06.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.634+0000 7f7479a52740 1 -- start start 2024-05-12T22:19:06.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.634+0000 7f7479a52740 1 Processor -- start 2024-05-12T22:19:06.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.635+0000 7f7479a52740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5616c2a71dd0 0x5616c2a7c710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:06.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.635+0000 7f7479a52740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5616c2a726f0 0x5616c2a79a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:06.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.635+0000 7f7479a52740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5616c2a7cdc0 0x5616c2a79f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:06.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.635+0000 7f7479a52740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5616c29919f0 con 0x5616c2a71dd0 2024-05-12T22:19:06.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.635+0000 7f7479a52740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x5616c29d2b50 con 0x5616c2a726f0 2024-05-12T22:19:06.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.635+0000 7f7479a52740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5616c2a7fbb0 con 0x5616c2a7cdc0 2024-05-12T22:19:06.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.635+0000 7f7479a52740 10 client.0 _ll_register_callbacks cb 0x5616c26e9be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:19:06.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.635+0000 7f7467e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5616c2a726f0 0x5616c2a79a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:06.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.635+0000 7f7474c00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5616c2a7cdc0 0x5616c2a79f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:06.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.635+0000 7f7467e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5616c2a726f0 0x5616c2a79a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:41124/0 (socket says 172.21.15.28:41124) 2024-05-12T22:19:06.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.635+0000 7f7467e00640 1 -- 172.21.15.28:0/3932474113 learned_addr learned my addr 172.21.15.28:0/3932474113 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:19:06.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.635+0000 7f746fe00640 1 --2- 172.21.15.28:0/3932474113 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5616c2a71dd0 0x5616c2a7c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:06.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.635+0000 7f7467e00640 1 -- 172.21.15.28:0/3932474113 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5616c2a7cdc0 msgr2=0x5616c2a79f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:19:06.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.635+0000 7f7467e00640 1 --2- 172.21.15.28:0/3932474113 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5616c2a7cdc0 0x5616c2a79f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:06.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.635+0000 7f7467e00640 1 -- 172.21.15.28:0/3932474113 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5616c2a71dd0 msgr2=0x5616c2a7c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:19:06.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.635+0000 7f7467e00640 1 --2- 172.21.15.28:0/3932474113 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5616c2a71dd0 0x5616c2a7c710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:06.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.636+0000 7f7467e00640 1 -- 172.21.15.28:0/3932474113 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5616c26ea3f0 con 0x5616c2a726f0 2024-05-12T22:19:06.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.636+0000 7f7474c00640 1 --2- 172.21.15.28:0/3932474113 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5616c2a7cdc0 0x5616c2a79f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:19:06.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.636+0000 7f7467e00640 1 --2- 172.21.15.28:0/3932474113 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5616c2a726f0 0x5616c2a79a00 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f745800baa0 tx=0x7f7458011440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=104ec97e07e20acd server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:19:06.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.636+0000 7f745d600640 10 client.0 ms_handle_connect on v2:172.21.15.86:3300/0 2024-05-12T22:19:06.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.636+0000 7f745d600640 1 -- 172.21.15.28:0/3932474113 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7458028060 con 0x5616c2a726f0 2024-05-12T22:19:06.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.636+0000 7f7479a52740 1 -- 172.21.15.28:0/3932474113 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f745800be20 con 0x5616c2a726f0 2024-05-12T22:19:06.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.637+0000 7f745d600640 1 -- 172.21.15.28:0/3932474113 <== mon.2 v2:172.21.15.86:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8513+0+0 (secure 0 0 0) 0x7f74580054d0 con 0x5616c2a726f0 2024-05-12T22:19:06.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.637+0000 7f745d600640 1 -- 172.21.15.28:0/3932474113 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f7458015040 con 0x5616c2a726f0 2024-05-12T22:19:06.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.637+0000 7f7479a52740 10 client.27470 fetch_fsmap learned FSMap version 26 2024-05-12T22:19:06.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.637+0000 7f7479a52740 1 -- 172.21.15.28:0/3932474113 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5616c26aa0f0 con 0x5616c2a726f0 2024-05-12T22:19:06.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.637+0000 7f745d600640 1 -- 172.21.15.28:0/3932474113 <== mon.2 v2:172.21.15.86:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f745800be20 con 0x5616c2a726f0 2024-05-12T22:19:06.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.638+0000 7f745d600640 1 -- 172.21.15.28:0/3932474113 <== mon.2 v2:172.21.15.86:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f7458022030 con 0x5616c2a726f0 2024-05-12T22:19:06.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.638+0000 7f7479a52740 10 client.27470 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:19:06.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.638+0000 7f7479a52740 10 client.27470 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:19:06.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.638+0000 7f7479a52740 20 client.27470 populate_metadata read hostname 'smithi028' 2024-05-12T22:19:06.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.638+0000 7f7479a52740 1 --2- 172.21.15.28:0/3932474113 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5616c2a8c820 0x5616c2a8ec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:06.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.638+0000 7f7479a52740 4 client.27470 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-12T22:19:06.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.638+0000 7f7479a52740 1 -- 172.21.15.28:0/3932474113 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5616c29e42c0 con 0x5616c2a8c820 2024-05-12T22:19:06.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.638+0000 7f746fe00640 1 --2- 172.21.15.28:0/3932474113 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5616c2a8c820 0x5616c2a8ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:06.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.639+0000 7f746fe00640 1 --2- 172.21.15.28:0/3932474113 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5616c2a8c820 0x5616c2a8ec10 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=734f844585ad8b1d server_cookie=bac895d922ac6a28 in_seq=0 out_seq=0 2024-05-12T22:19:06.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.639+0000 7f745d600640 10 client.27470 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:19:06.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.640+0000 7f745d600640 1 -- 172.21.15.28:0/3932474113 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5616c29e42c0 con 0x5616c2a8c820 2024-05-12T22:19:06.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.640+0000 7f745d600640 10 client.27470 handle_command_reply: tid=0 2024-05-12T22:19:06.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.640+0000 7f7479a52740 1 client.27470 shutdown 2024-05-12T22:19:06.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.640+0000 7f7479a52740 1 -- 172.21.15.28:0/3932474113 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5616c2a726f0 msgr2=0x5616c2a79a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:06.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.640+0000 7f7479a52740 1 --2- 172.21.15.28:0/3932474113 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5616c2a726f0 0x5616c2a79a00 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f745800baa0 tx=0x7f7458011440 comp rx=0 tx=0).stop 2024-05-12T22:19:06.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.640+0000 7f7479a52740 1 -- 172.21.15.28:0/3932474113 shutdown_connections 2024-05-12T22:19:06.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.640+0000 7f7479a52740 1 --2- 172.21.15.28:0/3932474113 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5616c2a8c820 0x5616c2a8ec10 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:06.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.640+0000 7f7479a52740 1 --2- 172.21.15.28:0/3932474113 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5616c2a7cdc0 0x5616c2a79f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:06.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.640+0000 7f7479a52740 1 --2- 172.21.15.28:0/3932474113 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5616c2a726f0 0x5616c2a79a00 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:06.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.640+0000 7f7479a52740 1 --2- 172.21.15.28:0/3932474113 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5616c2a71dd0 0x5616c2a7c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:06.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.640+0000 7f7479a52740 1 -- 172.21.15.28:0/3932474113 >> 172.21.15.28:0/3932474113 conn(0x5616c2a6e0f0 msgr2=0x5616c2a6d330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:19:06.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.640+0000 7f745d600640 0 client.27470 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:19:06.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.641+0000 7f7479a52740 1 -- 172.21.15.28:0/3932474113 shutdown_connections 2024-05-12T22:19:06.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.641+0000 7f7479a52740 1 -- 172.21.15.28:0/3932474113 wait complete. 2024-05-12T22:19:06.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.641+0000 7f7479a52740 20 client.27470 trim_cache size 0 max 16384 2024-05-12T22:19:06.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.667+0000 7f7479a52740 1 Processor -- start 2024-05-12T22:19:06.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.667+0000 7f7479a52740 1 -- start start 2024-05-12T22:19:06.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.668+0000 7f7479a52740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5616c2a71dd0 0x5616c2a88920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:06.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.668+0000 7f7479a52740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5616c2a726f0 0x5616c2a94290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:06.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.668+0000 7f7479a52740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5616c2a7cdc0 0x5616c2a947d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:06.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.668+0000 7f7479a52740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f74701b6130 con 0x5616c2a71dd0 2024-05-12T22:19:06.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.668+0000 7f7479a52740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f74701b3a00 con 0x5616c2a726f0 2024-05-12T22:19:06.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.668+0000 7f7479a52740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f74701b1870 con 0x5616c2a7cdc0 2024-05-12T22:19:06.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.668+0000 7f7474c00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5616c2a7cdc0 0x5616c2a947d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:06.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.668+0000 7f7474c00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5616c2a7cdc0 0x5616c2a947d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:38014/0 (socket says 172.21.15.28:38014) 2024-05-12T22:19:06.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.668+0000 7f7474c00640 1 -- 172.21.15.28:0/2334974755 learned_addr learned my addr 172.21.15.28:0/2334974755 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:19:06.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.668+0000 7f7467e00640 1 --2- 172.21.15.28:0/2334974755 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5616c2a726f0 0x5616c2a94290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:06.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.668+0000 7f746fe00640 1 --2- 172.21.15.28:0/2334974755 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5616c2a71dd0 0x5616c2a88920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:06.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.669+0000 7f7474c00640 1 -- 172.21.15.28:0/2334974755 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5616c2a726f0 msgr2=0x5616c2a94290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:19:06.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.669+0000 7f7474c00640 1 --2- 172.21.15.28:0/2334974755 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5616c2a726f0 0x5616c2a94290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:06.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.669+0000 7f7474c00640 1 -- 172.21.15.28:0/2334974755 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5616c2a71dd0 msgr2=0x5616c2a88920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:19:06.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.669+0000 7f7474c00640 1 --2- 172.21.15.28:0/2334974755 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5616c2a71dd0 0x5616c2a88920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:06.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.669+0000 7f7474c00640 1 -- 172.21.15.28:0/2334974755 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5616c2a77fb0 con 0x5616c2a7cdc0 2024-05-12T22:19:06.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.669+0000 7f7467e00640 1 --2- 172.21.15.28:0/2334974755 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5616c2a726f0 0x5616c2a94290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:19:06.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.669+0000 7f7474c00640 1 --2- 172.21.15.28:0/2334974755 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5616c2a7cdc0 0x5616c2a947d0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f7468010e30 tx=0x7f7468019000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=577cc569a317aefb server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:19:06.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.669+0000 7f745fe00640 1 -- 172.21.15.28:0/2334974755 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7468010e60 con 0x5616c2a7cdc0 2024-05-12T22:19:06.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.669+0000 7f745fe00640 1 -- 172.21.15.28:0/2334974755 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7468003e50 con 0x5616c2a7cdc0 2024-05-12T22:19:06.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.669+0000 7f745fe00640 1 -- 172.21.15.28:0/2334974755 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f746801c650 con 0x5616c2a7cdc0 2024-05-12T22:19:06.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.670+0000 7f7479a52740 1 -- 172.21.15.28:0/2334974755 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5616c2a7cdc0 msgr2=0x5616c2a947d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:06.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.670+0000 7f7479a52740 1 --2- 172.21.15.28:0/2334974755 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5616c2a7cdc0 0x5616c2a947d0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f7468010e30 tx=0x7f7468019000 comp rx=0 tx=0).stop 2024-05-12T22:19:06.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.670+0000 7f7479a52740 1 -- 172.21.15.28:0/2334974755 shutdown_connections 2024-05-12T22:19:06.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.670+0000 7f7479a52740 1 --2- 172.21.15.28:0/2334974755 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5616c2a7cdc0 0x5616c2a947d0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:06.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.670+0000 7f7479a52740 1 --2- 172.21.15.28:0/2334974755 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5616c2a726f0 0x5616c2a94290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:06.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.670+0000 7f7479a52740 1 --2- 172.21.15.28:0/2334974755 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5616c2a71dd0 0x5616c2a88920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:06.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.670+0000 7f7479a52740 1 -- 172.21.15.28:0/2334974755 >> 172.21.15.28:0/2334974755 conn(0x5616c2a6e0f0 msgr2=0x5616c2a6d9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:19:06.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.670+0000 7f7479a52740 1 -- 172.21.15.28:0/2334974755 shutdown_connections 2024-05-12T22:19:06.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.671+0000 7f7479a52740 1 -- 172.21.15.28:0/2334974755 wait complete. 2024-05-12T22:19:06.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.671+0000 7f7479a52740 1 -- start start 2024-05-12T22:19:06.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.671+0000 7f7479a52740 1 Processor -- start 2024-05-12T22:19:06.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.672+0000 7f7479a52740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5616c2a71dd0 0x5616c2a827c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:06.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.672+0000 7f7479a52740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5616c2a726f0 0x5616c2a82d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:06.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.672+0000 7f7479a52740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5616c2a7cdc0 0x5616c2a83240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:06.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.672+0000 7f7479a52740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f74701b1ec0 con 0x5616c2a71dd0 2024-05-12T22:19:06.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.672+0000 7f7479a52740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f74701b3330 con 0x5616c2a726f0 2024-05-12T22:19:06.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.672+0000 7f7479a52740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f74701b2590 con 0x5616c2a7cdc0 2024-05-12T22:19:06.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.672+0000 7f7479a52740 10 client.0 _ll_register_callbacks cb 0x7f7468009e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:19:06.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.672+0000 7f7467e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5616c2a726f0 0x5616c2a82d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:06.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.672+0000 7f746fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5616c2a71dd0 0x5616c2a827c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:06.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.672+0000 7f7467e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5616c2a726f0 0x5616c2a82d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:41138/0 (socket says 172.21.15.28:41138) 2024-05-12T22:19:06.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.672+0000 7f7467e00640 1 -- 172.21.15.28:0/3117938466 learned_addr learned my addr 172.21.15.28:0/3117938466 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:19:06.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.672+0000 7f7474c00640 1 --2- 172.21.15.28:0/3117938466 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5616c2a7cdc0 0x5616c2a83240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:06.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.672+0000 7f7467e00640 1 -- 172.21.15.28:0/3117938466 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5616c2a7cdc0 msgr2=0x5616c2a83240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:19:06.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.672+0000 7f7467e00640 1 --2- 172.21.15.28:0/3117938466 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5616c2a7cdc0 0x5616c2a83240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:06.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.672+0000 7f7467e00640 1 -- 172.21.15.28:0/3117938466 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5616c2a71dd0 msgr2=0x5616c2a827c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:19:06.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.672+0000 7f7467e00640 1 --2- 172.21.15.28:0/3117938466 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5616c2a71dd0 0x5616c2a827c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:06.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.672+0000 7f7467e00640 1 -- 172.21.15.28:0/3117938466 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5616c2af1b10 con 0x5616c2a726f0 2024-05-12T22:19:06.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.673+0000 7f7467e00640 1 --2- 172.21.15.28:0/3117938466 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5616c2a726f0 0x5616c2a82d00 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f7458006010 tx=0x7f745800b5a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=440ac937b016dcef server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:19:06.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.673+0000 7f745d600640 10 client.0 ms_handle_connect on v2:172.21.15.86:3300/0 2024-05-12T22:19:06.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.673+0000 7f745d600640 1 -- 172.21.15.28:0/3117938466 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7458028070 con 0x5616c2a726f0 2024-05-12T22:19:06.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.673+0000 7f745d600640 1 -- 172.21.15.28:0/3117938466 <== mon.2 v2:172.21.15.86:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8513+0+0 (secure 0 0 0) 0x7f7458004680 con 0x5616c2a726f0 2024-05-12T22:19:06.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.673+0000 7f7479a52740 1 -- 172.21.15.28:0/3117938466 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7468003e50 con 0x5616c2a726f0 2024-05-12T22:19:06.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.674+0000 7f745d600640 1 -- 172.21.15.28:0/3117938466 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f745800b3a0 con 0x5616c2a726f0 2024-05-12T22:19:06.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.674+0000 7f7479a52740 10 client.27476 fetch_fsmap learned FSMap version 26 2024-05-12T22:19:06.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.674+0000 7f7479a52740 1 -- 172.21.15.28:0/3117938466 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5616c2af2d60 con 0x5616c2a726f0 2024-05-12T22:19:06.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.674+0000 7f745d600640 1 -- 172.21.15.28:0/3117938466 <== mon.2 v2:172.21.15.86:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7468003e50 con 0x5616c2a726f0 2024-05-12T22:19:06.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.674+0000 7f745d600640 1 -- 172.21.15.28:0/3117938466 <== mon.2 v2:172.21.15.86:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f7458022040 con 0x5616c2a726f0 2024-05-12T22:19:06.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.674+0000 7f7479a52740 10 client.27476 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:19:06.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.674+0000 7f7479a52740 10 client.27476 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:19:06.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.674+0000 7f7479a52740 20 client.27476 populate_metadata read hostname 'smithi028' 2024-05-12T22:19:06.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.674+0000 7f7479a52740 1 --2- 172.21.15.28:0/3117938466 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5616c2af3510 0x5616c2af5930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:06.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.674+0000 7f7479a52740 4 client.27476 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-12T22:19:06.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.674+0000 7f7479a52740 1 -- 172.21.15.28:0/3117938466 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5616c29de610 con 0x5616c2af3510 2024-05-12T22:19:06.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.675+0000 7f746fe00640 1 --2- 172.21.15.28:0/3117938466 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5616c2af3510 0x5616c2af5930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:06.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.675+0000 7f746fe00640 1 --2- 172.21.15.28:0/3117938466 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5616c2af3510 0x5616c2af5930 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c73b898d8e9c8841 server_cookie=1730d2282fa7d2a6 in_seq=0 out_seq=0 2024-05-12T22:19:06.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.675+0000 7f745d600640 10 client.27476 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:19:06.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.676+0000 7f745d600640 1 -- 172.21.15.28:0/3117938466 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+268 (crc 0 0 0) 0x5616c29de610 con 0x5616c2af3510 2024-05-12T22:19:06.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.676+0000 7f745d600640 10 client.27476 handle_command_reply: tid=0 2024-05-12T22:19:06.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.676+0000 7f7479a52740 1 client.27476 shutdown 2024-05-12T22:19:06.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.676+0000 7f7479a52740 1 -- 172.21.15.28:0/3117938466 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5616c2a726f0 msgr2=0x5616c2a82d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:06.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.676+0000 7f7479a52740 1 --2- 172.21.15.28:0/3117938466 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5616c2a726f0 0x5616c2a82d00 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f7458006010 tx=0x7f745800b5a0 comp rx=0 tx=0).stop 2024-05-12T22:19:06.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.676+0000 7f7479a52740 1 -- 172.21.15.28:0/3117938466 shutdown_connections 2024-05-12T22:19:06.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.676+0000 7f7479a52740 1 --2- 172.21.15.28:0/3117938466 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5616c2af3510 0x5616c2af5930 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:06.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.676+0000 7f7479a52740 1 --2- 172.21.15.28:0/3117938466 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5616c2a7cdc0 0x5616c2a83240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:06.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.676+0000 7f7479a52740 1 --2- 172.21.15.28:0/3117938466 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5616c2a726f0 0x5616c2a82d00 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:06.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.676+0000 7f7479a52740 1 --2- 172.21.15.28:0/3117938466 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5616c2a71dd0 0x5616c2a827c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:06.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.676+0000 7f745d600640 0 client.27476 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:19:06.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.676+0000 7f7479a52740 1 -- 172.21.15.28:0/3117938466 >> 172.21.15.28:0/3117938466 conn(0x5616c2a6e0f0 msgr2=0x5616c2a7b230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:19:06.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.676+0000 7f7479a52740 1 -- 172.21.15.28:0/3117938466 shutdown_connections 2024-05-12T22:19:06.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.677+0000 7f7479a52740 1 -- 172.21.15.28:0/3117938466 wait complete. 2024-05-12T22:19:06.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.677+0000 7f7479a52740 20 client.27476 trim_cache size 0 max 16384 2024-05-12T22:19:06.683 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-05-12T22:19:06.683 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "scrub active (1177 inodes in the stack)", 2024-05-12T22:19:06.683 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-05-12T22:19:06.684 INFO:teuthology.orchestra.run.smithi028.stdout: "41de51df-fccc-4091-a87b-7e718236f742": { 2024-05-12T22:19:06.684 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-05-12T22:19:06.684 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "41de51df-fccc-4091-a87b-7e718236f742", 2024-05-12T22:19:06.684 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-05-12T22:19:06.684 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:06.684 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:06.684 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-05-12T22:19:06.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.677+0000 7f7475600640 1 -- 172.21.15.28:0/1541419496 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f74540a2d00 msgr2=0x7f74540a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:06.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.677+0000 7f7475600640 1 --2- 172.21.15.28:0/1541419496 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f74540a2d00 0x7f74540a51c0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f7468005fd0 tx=0x7f74680095f0 comp rx=0 tx=0).stop 2024-05-12T22:19:06.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.677+0000 7f7475600640 1 -- 172.21.15.28:0/1541419496 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7470107dd0 msgr2=0x7f747019e540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:06.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.677+0000 7f7475600640 1 --2- 172.21.15.28:0/1541419496 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7470107dd0 0x7f747019e540 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f7460004770 tx=0x7f7460003c10 comp rx=0 tx=0).stop 2024-05-12T22:19:06.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.678+0000 7f7475600640 1 -- 172.21.15.28:0/1541419496 shutdown_connections 2024-05-12T22:19:06.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.678+0000 7f7475600640 1 --2- 172.21.15.28:0/1541419496 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f74540a2d00 0x7f74540a51c0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:06.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.678+0000 7f7475600640 1 --2- 172.21.15.28:0/1541419496 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f747010ddf0 0x7f74701a2e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:06.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.678+0000 7f7475600640 1 --2- 172.21.15.28:0/1541419496 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7470108770 0x7f747019ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:06.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.678+0000 7f7475600640 1 --2- 172.21.15.28:0/1541419496 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7470107dd0 0x7f747019e540 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:06.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.678+0000 7f7475600640 1 -- 172.21.15.28:0/1541419496 >> 172.21.15.28:0/1541419496 conn(0x7f7470077650 msgr2=0x7f7470078660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:19:06.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.678+0000 7f7475600640 1 -- 172.21.15.28:0/1541419496 shutdown_connections 2024-05-12T22:19:06.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:06.678+0000 7f7475600640 1 -- 172.21.15.28:0/1541419496 wait complete. 2024-05-12T22:19:06.687 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:19:06 smithi186 ceph-mon[34785]: pgmap v312: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-05-12T22:19:06.698 INFO:tasks.cephfs.filesystem:scrub status for tag:41de51df-fccc-4091-a87b-7e718236f742 - {'path': '/', 'tag': '41de51df-fccc-4091-a87b-7e718236f742', 'options': 'recursive,force'} 2024-05-12T22:19:06.741 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:19:06 smithi086 ceph-mon[34297]: pgmap v312: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-05-12T22:19:07.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:19:07 smithi186 ceph-mon[34785]: scrub queued for path: / 2024-05-12T22:19:07.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:19:07 smithi186 ceph-mon[34785]: scrub summary: idle+waiting paths [/] 2024-05-12T22:19:07.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:19:07 smithi186 ceph-mon[34785]: scrub summary: active paths [/] 2024-05-12T22:19:07.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:19:07 smithi186 ceph-mon[34785]: scrub summary: active paths [/] 2024-05-12T22:19:07.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:19:07 smithi186 ceph-mon[34785]: scrub summary: idle+waiting paths [/] 2024-05-12T22:19:07.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:19:07 smithi186 ceph-mon[34785]: scrub summary: active paths [/] 2024-05-12T22:19:07.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:19:07 smithi186 ceph-mon[34785]: scrub summary: idle+waiting paths [/] 2024-05-12T22:19:07.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:19:07 smithi186 ceph-mon[34785]: scrub summary: active paths [/] 2024-05-12T22:19:07.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:19:07 smithi186 ceph-mon[34785]: scrub summary: idle+waiting paths [/] 2024-05-12T22:19:07.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:19:07 smithi086 ceph-mon[34297]: scrub queued for path: / 2024-05-12T22:19:07.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:19:07 smithi086 ceph-mon[34297]: scrub summary: idle+waiting paths [/] 2024-05-12T22:19:07.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:19:07 smithi086 ceph-mon[34297]: scrub summary: active paths [/] 2024-05-12T22:19:07.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:19:07 smithi086 ceph-mon[34297]: scrub summary: active paths [/] 2024-05-12T22:19:07.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:19:07 smithi086 ceph-mon[34297]: scrub summary: idle+waiting paths [/] 2024-05-12T22:19:07.743 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:19:07 smithi086 ceph-mon[34297]: scrub summary: active paths [/] 2024-05-12T22:19:07.743 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:19:07 smithi086 ceph-mon[34297]: scrub summary: idle+waiting paths [/] 2024-05-12T22:19:07.743 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:19:07 smithi086 ceph-mon[34297]: scrub summary: active paths [/] 2024-05-12T22:19:07.743 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:19:07 smithi086 ceph-mon[34297]: scrub summary: idle+waiting paths [/] 2024-05-12T22:19:07.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:19:07 smithi028 ceph-mon[29847]: scrub queued for path: / 2024-05-12T22:19:07.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:19:07 smithi028 ceph-mon[29847]: scrub summary: idle+waiting paths [/] 2024-05-12T22:19:07.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:19:07 smithi028 ceph-mon[29847]: scrub summary: active paths [/] 2024-05-12T22:19:07.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:19:07 smithi028 ceph-mon[29847]: scrub summary: active paths [/] 2024-05-12T22:19:07.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:19:07 smithi028 ceph-mon[29847]: scrub summary: idle+waiting paths [/] 2024-05-12T22:19:07.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:19:07 smithi028 ceph-mon[29847]: scrub summary: active paths [/] 2024-05-12T22:19:07.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:19:07 smithi028 ceph-mon[29847]: scrub summary: idle+waiting paths [/] 2024-05-12T22:19:07.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:19:07 smithi028 ceph-mon[29847]: scrub summary: active paths [/] 2024-05-12T22:19:07.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:19:07 smithi028 ceph-mon[29847]: scrub summary: idle+waiting paths [/] 2024-05-12T22:19:08.643 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:19:08 smithi086 ceph-mon[34297]: scrub summary: idle+waiting paths [/] 2024-05-12T22:19:08.643 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:19:08 smithi086 ceph-mon[34297]: pgmap v313: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-05-12T22:19:08.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:19:08 smithi186 ceph-mon[34785]: scrub summary: idle+waiting paths [/] 2024-05-12T22:19:08.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:19:08 smithi186 ceph-mon[34785]: pgmap v313: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-05-12T22:19:08.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:19:08 smithi028 ceph-mon[29847]: scrub summary: idle+waiting paths [/] 2024-05-12T22:19:08.787 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:19:08 smithi028 ceph-mon[29847]: pgmap v313: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-05-12T22:19:09.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:19:09 smithi186 ceph-mon[34785]: osdmap e96: 12 total, 12 up, 12 in 2024-05-12T22:19:09.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:19:09 smithi086 ceph-mon[34297]: osdmap e96: 12 total, 12 up, 12 in 2024-05-12T22:19:09.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:19:09 smithi028 ceph-mon[29847]: osdmap e96: 12 total, 12 up, 12 in 2024-05-12T22:19:10.132 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-05-12T22:19:10.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:10.347+0000 7f3545200640 1 -- 172.21.15.28:0/2621458145 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3540101630 msgr2=0x7f3540101ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:10.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:10.347+0000 7f3545200640 1 --2- 172.21.15.28:0/2621458145 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3540101630 0x7f3540101ab0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f352c009930 tx=0x7f352c02a420 comp rx=0 tx=0).stop 2024-05-12T22:19:10.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:10.347+0000 7f3545200640 1 -- 172.21.15.28:0/2621458145 shutdown_connections 2024-05-12T22:19:10.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:10.347+0000 7f3545200640 1 --2- 172.21.15.28:0/2621458145 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3540101ff0 0x7f354010e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:10.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:10.347+0000 7f3545200640 1 --2- 172.21.15.28:0/2621458145 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3540101630 0x7f3540101ab0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:10.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:10.347+0000 7f3545200640 1 --2- 172.21.15.28:0/2621458145 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3540107640 0x7f3540107a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:10.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:10.347+0000 7f3545200640 1 -- 172.21.15.28:0/2621458145 >> 172.21.15.28:0/2621458145 conn(0x7f35400fd180 msgr2=0x7f35400ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:19:10.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:10.347+0000 7f3545200640 1 -- 172.21.15.28:0/2621458145 shutdown_connections 2024-05-12T22:19:10.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:10.347+0000 7f3545200640 1 -- 172.21.15.28:0/2621458145 wait complete. 2024-05-12T22:19:10.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:10.348+0000 7f3545200640 1 Processor -- start 2024-05-12T22:19:10.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:10.348+0000 7f3545200640 1 -- start start 2024-05-12T22:19:10.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:10.349+0000 7f3545200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3540101630 0x7f3540075730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:10.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:10.349+0000 7f3545200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3540101ff0 0x7f3540073b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:10.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:10.349+0000 7f3545200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3540107640 0x7f3540074170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:10.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:10.349+0000 7f3545200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3540111aa0 con 0x7f3540101630 2024-05-12T22:19:10.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:10.349+0000 7f3545200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f3540111920 con 0x7f3540107640 2024-05-12T22:19:10.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:10.349+0000 7f3545200640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f3540111c20 con 0x7f3540101ff0 2024-05-12T22:19:10.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:10.349+0000 7f353f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3540101630 0x7f3540075730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:10.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:10.349+0000 7f353f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3540101630 0x7f3540075730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55126/0 (socket says 172.21.15.28:55126) 2024-05-12T22:19:10.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:10.349+0000 7f353f400640 1 -- 172.21.15.28:0/4213063533 learned_addr learned my addr 172.21.15.28:0/4213063533 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:19:10.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:10.349+0000 7f353ea00640 1 --2- 172.21.15.28:0/4213063533 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3540101ff0 0x7f3540073b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:10.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:10.349+0000 7f353fe00640 1 --2- 172.21.15.28:0/4213063533 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3540107640 0x7f3540074170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:10.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:10.350+0000 7f353f400640 1 -- 172.21.15.28:0/4213063533 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3540101ff0 msgr2=0x7f3540073b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:10.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:10.350+0000 7f353f400640 1 --2- 172.21.15.28:0/4213063533 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3540101ff0 0x7f3540073b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:10.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:10.350+0000 7f353f400640 1 -- 172.21.15.28:0/4213063533 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3540107640 msgr2=0x7f3540074170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:10.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:10.350+0000 7f353f400640 1 --2- 172.21.15.28:0/4213063533 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3540107640 0x7f3540074170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:10.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:10.350+0000 7f353f400640 1 -- 172.21.15.28:0/4213063533 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35401aa980 con 0x7f3540101630 2024-05-12T22:19:10.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:10.350+0000 7f353fe00640 1 --2- 172.21.15.28:0/4213063533 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3540107640 0x7f3540074170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:19:10.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:10.350+0000 7f353f400640 1 --2- 172.21.15.28:0/4213063533 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3540101630 0x7f3540075730 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f353000e9f0 tx=0x7f353000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:19:10.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:10.350+0000 7f3537e00640 1 -- 172.21.15.28:0/4213063533 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f353000cd80 con 0x7f3540101630 2024-05-12T22:19:10.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:10.350+0000 7f3537e00640 1 -- 172.21.15.28:0/4213063533 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3530005500 con 0x7f3540101630 2024-05-12T22:19:10.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:10.350+0000 7f3537e00640 1 -- 172.21.15.28:0/4213063533 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3530010690 con 0x7f3540101630 2024-05-12T22:19:10.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:10.350+0000 7f3545200640 1 -- 172.21.15.28:0/4213063533 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35401aabb0 con 0x7f3540101630 2024-05-12T22:19:10.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:10.351+0000 7f3545200640 1 -- 172.21.15.28:0/4213063533 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35401ab080 con 0x7f3540101630 2024-05-12T22:19:10.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:10.352+0000 7f3545200640 1 -- 172.21.15.28:0/4213063533 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3540102d70 con 0x7f3540101630 2024-05-12T22:19:10.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:10.354+0000 7f3537e00640 1 -- 172.21.15.28:0/4213063533 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7f35300046c0 con 0x7f3540101630 2024-05-12T22:19:10.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:10.355+0000 7f3537e00640 1 --2- 172.21.15.28:0/4213063533 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f35240a2cf0 0x7f35240a51b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:10.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:10.355+0000 7f3537e00640 1 -- 172.21.15.28:0/4213063533 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8557+0+0 (secure 0 0 0) 0x7f35300cf870 con 0x7f3540101630 2024-05-12T22:19:10.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:10.356+0000 7f353ea00640 1 --2- 172.21.15.28:0/4213063533 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f35240a2cf0 0x7f35240a51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:10.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:10.356+0000 7f353ea00640 1 --2- 172.21.15.28:0/4213063533 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f35240a2cf0 0x7f35240a51b0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f354019c110 tx=0x7f352c000bc0 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:19:10.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:10.360+0000 7f3537e00640 1 -- 172.21.15.28:0/4213063533 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f353009c410 con 0x7f3540101630 2024-05-12T22:19:10.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:19:10 smithi028 ceph-mon[29847]: pgmap v315: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-05-12T22:19:10.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:10.587+0000 7f3545200640 1 -- 172.21.15.28:0/4213063533 --> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7f3540061c70 con 0x7f35240a2cf0 2024-05-12T22:19:10.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:10.601+0000 7f3537e00640 1 -- 172.21.15.28:0/4213063533 <== mgr.25189 v2:172.21.15.28:6800/3177640985 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+350 (secure 0 0 0) 0x7f3540061c70 con 0x7f35240a2cf0 2024-05-12T22:19:10.602 INFO:teuthology.orchestra.run.smithi028.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-05-12T00:00:00", "created": "2024-05-12T22:09:08", "first": "2024-05-12T22:10:00", "last": "2024-05-12T22:19:01", "last_pruned": "2024-05-12T22:19:01", "created_count": 10, "pruned_count": 4, "active": true} 2024-05-12T22:19:10.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:10.604+0000 7f3545200640 1 -- 172.21.15.28:0/4213063533 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f35240a2cf0 msgr2=0x7f35240a51b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:10.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:10.604+0000 7f3545200640 1 --2- 172.21.15.28:0/4213063533 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f35240a2cf0 0x7f35240a51b0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f354019c110 tx=0x7f352c000bc0 comp rx=0 tx=0).stop 2024-05-12T22:19:10.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:10.604+0000 7f3545200640 1 -- 172.21.15.28:0/4213063533 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3540101630 msgr2=0x7f3540075730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:10.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:10.604+0000 7f3545200640 1 --2- 172.21.15.28:0/4213063533 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3540101630 0x7f3540075730 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f353000e9f0 tx=0x7f353000edf0 comp rx=0 tx=0).stop 2024-05-12T22:19:10.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:10.604+0000 7f3545200640 1 -- 172.21.15.28:0/4213063533 shutdown_connections 2024-05-12T22:19:10.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:10.604+0000 7f3545200640 1 --2- 172.21.15.28:0/4213063533 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f35240a2cf0 0x7f35240a51b0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:10.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:10.604+0000 7f3545200640 1 --2- 172.21.15.28:0/4213063533 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3540107640 0x7f3540074170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:10.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:10.604+0000 7f3545200640 1 --2- 172.21.15.28:0/4213063533 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3540101ff0 0x7f3540073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:10.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:10.604+0000 7f3545200640 1 --2- 172.21.15.28:0/4213063533 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3540101630 0x7f3540075730 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:10.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:10.604+0000 7f3545200640 1 -- 172.21.15.28:0/4213063533 >> 172.21.15.28:0/4213063533 conn(0x7f35400fd180 msgr2=0x7f35400fd9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:19:10.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:10.604+0000 7f3545200640 1 -- 172.21.15.28:0/4213063533 shutdown_connections 2024-05-12T22:19:10.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:10.604+0000 7f3545200640 1 -- 172.21.15.28:0/4213063533 wait complete. 2024-05-12T22:19:10.625 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-05-12T22:19:10.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:19:10 smithi186 ceph-mon[34785]: pgmap v315: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-05-12T22:19:10.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:19:10 smithi086 ceph-mon[34297]: pgmap v315: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-05-12T22:19:10.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:10.838+0000 7f57efe00640 1 -- 172.21.15.28:0/2249907375 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57f0100dd0 msgr2=0x7f57f010da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:10.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:10.838+0000 7f57efe00640 1 --2- 172.21.15.28:0/2249907375 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57f0100dd0 0x7f57f010da00 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f57d8009900 tx=0x7f57d802a3f0 comp rx=0 tx=0).stop 2024-05-12T22:19:10.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:10.838+0000 7f57efe00640 1 -- 172.21.15.28:0/2249907375 shutdown_connections 2024-05-12T22:19:10.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:10.838+0000 7f57efe00640 1 --2- 172.21.15.28:0/2249907375 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f57f010dfd0 0x7f57f01103c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:10.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:10.838+0000 7f57efe00640 1 --2- 172.21.15.28:0/2249907375 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57f0100dd0 0x7f57f010da00 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:10.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:10.838+0000 7f57efe00640 1 --2- 172.21.15.28:0/2249907375 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f57f0100420 0x7f57f0100800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:10.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:10.838+0000 7f57efe00640 1 -- 172.21.15.28:0/2249907375 >> 172.21.15.28:0/2249907375 conn(0x7f57f00fc050 msgr2=0x7f57f00fe470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:19:10.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:10.838+0000 7f57efe00640 1 -- 172.21.15.28:0/2249907375 shutdown_connections 2024-05-12T22:19:10.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:10.838+0000 7f57efe00640 1 -- 172.21.15.28:0/2249907375 wait complete. 2024-05-12T22:19:10.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:10.839+0000 7f57efe00640 1 Processor -- start 2024-05-12T22:19:10.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:10.840+0000 7f57efe00640 1 -- start start 2024-05-12T22:19:10.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:10.840+0000 7f57efe00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f57f0100420 0x7f57f019a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:10.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:10.840+0000 7f57efe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57f0100dd0 0x7f57f019a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:10.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:10.840+0000 7f57efe00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f57f010dfd0 0x7f57f019ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:10.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:10.840+0000 7f57efe00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f57f0113cb0 con 0x7f57f0100dd0 2024-05-12T22:19:10.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:10.840+0000 7f57efe00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f57f0113b30 con 0x7f57f0100420 2024-05-12T22:19:10.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:10.840+0000 7f57efe00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f57f0113e30 con 0x7f57f010dfd0 2024-05-12T22:19:10.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:10.840+0000 7f57ee000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57f0100dd0 0x7f57f019a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:10.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:10.840+0000 7f57ee000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57f0100dd0 0x7f57f019a750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55146/0 (socket says 172.21.15.28:55146) 2024-05-12T22:19:10.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:10.840+0000 7f57eea00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f57f0100420 0x7f57f019a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:10.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:10.840+0000 7f57ee000640 1 -- 172.21.15.28:0/3043804539 learned_addr learned my addr 172.21.15.28:0/3043804539 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:19:10.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:10.841+0000 7f57ef400640 1 --2- 172.21.15.28:0/3043804539 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f57f010dfd0 0x7f57f019ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:10.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:10.841+0000 7f57eea00640 1 -- 172.21.15.28:0/3043804539 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f57f010dfd0 msgr2=0x7f57f019ead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:10.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:10.841+0000 7f57eea00640 1 --2- 172.21.15.28:0/3043804539 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f57f010dfd0 0x7f57f019ead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:10.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:10.841+0000 7f57eea00640 1 -- 172.21.15.28:0/3043804539 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57f0100dd0 msgr2=0x7f57f019a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:10.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:10.841+0000 7f57eea00640 1 --2- 172.21.15.28:0/3043804539 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57f0100dd0 0x7f57f019a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:10.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:10.841+0000 7f57eea00640 1 -- 172.21.15.28:0/3043804539 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57f019f200 con 0x7f57f0100420 2024-05-12T22:19:10.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:10.841+0000 7f57ef400640 1 --2- 172.21.15.28:0/3043804539 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f57f010dfd0 0x7f57f019ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:19:10.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:10.841+0000 7f57ee000640 1 --2- 172.21.15.28:0/3043804539 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57f0100dd0 0x7f57f019a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:19:10.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:10.841+0000 7f57eea00640 1 --2- 172.21.15.28:0/3043804539 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f57f0100420 0x7f57f019a210 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f57e000e9f0 tx=0x7f57e000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:19:10.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:10.841+0000 7f57e7400640 1 -- 172.21.15.28:0/3043804539 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57e000ccc0 con 0x7f57f0100420 2024-05-12T22:19:10.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:10.842+0000 7f57e7400640 1 -- 172.21.15.28:0/3043804539 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f57e0005020 con 0x7f57f0100420 2024-05-12T22:19:10.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:10.842+0000 7f57e7400640 1 -- 172.21.15.28:0/3043804539 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57e00049e0 con 0x7f57f0100420 2024-05-12T22:19:10.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:10.842+0000 7f57efe00640 1 -- 172.21.15.28:0/3043804539 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57f019f4f0 con 0x7f57f0100420 2024-05-12T22:19:10.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:10.842+0000 7f57efe00640 1 -- 172.21.15.28:0/3043804539 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57f01a6d80 con 0x7f57f0100420 2024-05-12T22:19:10.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:10.843+0000 7f57efe00640 1 -- 172.21.15.28:0/3043804539 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f57f004fe40 con 0x7f57f0100420 2024-05-12T22:19:10.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:10.845+0000 7f57e7400640 1 -- 172.21.15.28:0/3043804539 <== mon.2 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7f57e0005560 con 0x7f57f0100420 2024-05-12T22:19:10.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:10.846+0000 7f57e7400640 1 --2- 172.21.15.28:0/3043804539 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f57d40a2d70 0x7f57d40a5230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:10.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:10.847+0000 7f57e7400640 1 -- 172.21.15.28:0/3043804539 <== mon.2 v2:172.21.15.86:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8557+0+0 (secure 0 0 0) 0x7f57e00cf4c0 con 0x7f57f0100420 2024-05-12T22:19:10.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:10.847+0000 7f57ee000640 1 --2- 172.21.15.28:0/3043804539 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f57d40a2d70 0x7f57d40a5230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:10.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:10.848+0000 7f57ee000640 1 --2- 172.21.15.28:0/3043804539 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f57d40a2d70 0x7f57d40a5230 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f57d80096c0 tx=0x7f57d80046c0 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:19:10.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:10.849+0000 7f57e7400640 1 -- 172.21.15.28:0/3043804539 <== mon.2 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f57e00051d0 con 0x7f57f0100420 2024-05-12T22:19:11.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:11.078+0000 7f57efe00640 1 -- 172.21.15.28:0/3043804539 --> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7f57f0061c70 con 0x7f57d40a2d70 2024-05-12T22:19:11.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:11.091+0000 7f57e7400640 1 -- 172.21.15.28:0/3043804539 <== mgr.25189 v2:172.21.15.28:6800/3177640985 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7f57f0061c70 con 0x7f57d40a2d70 2024-05-12T22:19:11.092 INFO:teuthology.orchestra.run.smithi028.stdout:/ 1m 6m3h 2024-05-12T22:19:11.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:11.094+0000 7f57efe00640 1 -- 172.21.15.28:0/3043804539 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f57d40a2d70 msgr2=0x7f57d40a5230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:11.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:11.094+0000 7f57efe00640 1 --2- 172.21.15.28:0/3043804539 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f57d40a2d70 0x7f57d40a5230 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f57d80096c0 tx=0x7f57d80046c0 comp rx=0 tx=0).stop 2024-05-12T22:19:11.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:11.094+0000 7f57efe00640 1 -- 172.21.15.28:0/3043804539 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f57f0100420 msgr2=0x7f57f019a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:11.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:11.094+0000 7f57efe00640 1 --2- 172.21.15.28:0/3043804539 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f57f0100420 0x7f57f019a210 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f57e000e9f0 tx=0x7f57e000eeb0 comp rx=0 tx=0).stop 2024-05-12T22:19:11.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:11.094+0000 7f57efe00640 1 -- 172.21.15.28:0/3043804539 shutdown_connections 2024-05-12T22:19:11.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:11.094+0000 7f57efe00640 1 --2- 172.21.15.28:0/3043804539 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f57d40a2d70 0x7f57d40a5230 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:11.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:11.094+0000 7f57efe00640 1 --2- 172.21.15.28:0/3043804539 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f57f010dfd0 0x7f57f019ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:11.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:11.094+0000 7f57efe00640 1 --2- 172.21.15.28:0/3043804539 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57f0100dd0 0x7f57f019a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:11.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:11.094+0000 7f57efe00640 1 --2- 172.21.15.28:0/3043804539 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f57f0100420 0x7f57f019a210 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:11.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:11.094+0000 7f57efe00640 1 -- 172.21.15.28:0/3043804539 >> 172.21.15.28:0/3043804539 conn(0x7f57f00fc050 msgr2=0x7f57f00fe440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:19:11.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:11.095+0000 7f57efe00640 1 -- 172.21.15.28:0/3043804539 shutdown_connections 2024-05-12T22:19:11.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:11.095+0000 7f57efe00640 1 -- 172.21.15.28:0/3043804539 wait complete. 2024-05-12T22:19:11.116 DEBUG:teuthology.run_tasks:Unwinding manager fwd_scrub 2024-05-12T22:19:11.127 INFO:tasks.fwd_scrub:joining ForwardScrubbers 2024-05-12T22:19:12.682 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:19:12 smithi186 ceph-mon[34785]: from='client.17349 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-05-12T22:19:12.682 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:19:12 smithi186 ceph-mon[34785]: pgmap v316: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2024-05-12T22:19:12.682 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:19:12 smithi186 ceph-mon[34785]: from='client.27488 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-12T22:19:12.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:19:12 smithi086 ceph-mon[34297]: from='client.17349 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-05-12T22:19:12.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:19:12 smithi086 ceph-mon[34297]: pgmap v316: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2024-05-12T22:19:12.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:19:12 smithi086 ceph-mon[34297]: from='client.27488 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-12T22:19:12.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:19:12 smithi028 ceph-mon[29847]: from='client.17349 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-05-12T22:19:12.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:19:12 smithi028 ceph-mon[29847]: pgmap v316: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2024-05-12T22:19:12.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:19:12 smithi028 ceph-mon[29847]: from='client.27488 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-12T22:19:13.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:19:13 smithi186 ceph-mon[34785]: scrub summary: idle 2024-05-12T22:19:13.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:19:13 smithi186 ceph-mon[34785]: scrub summary: idle 2024-05-12T22:19:13.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:19:13 smithi086 ceph-mon[34297]: scrub summary: idle 2024-05-12T22:19:13.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:19:13 smithi086 ceph-mon[34297]: scrub summary: idle 2024-05-12T22:19:13.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:19:13 smithi028 ceph-mon[29847]: scrub summary: idle 2024-05-12T22:19:13.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:19:13 smithi028 ceph-mon[29847]: scrub summary: idle 2024-05-12T22:19:14.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:19:14 smithi186 ceph-mon[34785]: pgmap v317: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2024-05-12T22:19:14.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:19:14 smithi186 ceph-mon[34785]: osdmap e97: 12 total, 12 up, 12 in 2024-05-12T22:19:14.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:19:14 smithi086 ceph-mon[34297]: pgmap v317: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2024-05-12T22:19:14.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:19:14 smithi086 ceph-mon[34297]: osdmap e97: 12 total, 12 up, 12 in 2024-05-12T22:19:14.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:19:14 smithi028 ceph-mon[29847]: pgmap v317: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2024-05-12T22:19:14.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:19:14 smithi028 ceph-mon[29847]: osdmap e97: 12 total, 12 up, 12 in 2024-05-12T22:19:16.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:19:16 smithi186 ceph-mon[34785]: pgmap v319: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 255 B/s wr, 11 op/s 2024-05-12T22:19:16.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:19:16 smithi086 ceph-mon[34297]: pgmap v319: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 255 B/s wr, 11 op/s 2024-05-12T22:19:16.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:19:16 smithi028 ceph-mon[29847]: pgmap v319: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 255 B/s wr, 11 op/s 2024-05-12T22:19:18.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:19:18 smithi186 ceph-mon[34785]: pgmap v320: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 237 B/s wr, 11 op/s 2024-05-12T22:19:18.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:19:18 smithi086 ceph-mon[34297]: pgmap v320: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 237 B/s wr, 11 op/s 2024-05-12T22:19:18.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:19:18 smithi028 ceph-mon[29847]: pgmap v320: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 237 B/s wr, 11 op/s 2024-05-12T22:19:20.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:19:20 smithi186 ceph-mon[34785]: pgmap v321: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 204 B/s wr, 9 op/s 2024-05-12T22:19:20.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:19:20 smithi086 ceph-mon[34297]: pgmap v321: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 204 B/s wr, 9 op/s 2024-05-12T22:19:20.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:19:20 smithi028 ceph-mon[29847]: pgmap v321: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 204 B/s wr, 9 op/s 2024-05-12T22:19:22.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:19:22 smithi186 ceph-mon[34785]: pgmap v322: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 9 op/s 2024-05-12T22:19:22.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:19:22 smithi086 ceph-mon[34297]: pgmap v322: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 9 op/s 2024-05-12T22:19:22.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:19:22 smithi028 ceph-mon[29847]: pgmap v322: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 9 op/s 2024-05-12T22:19:23.682 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:19:23 smithi186 ceph-mon[34785]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:19:23.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:19:23 smithi086 ceph-mon[34297]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:19:23.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:19:23 smithi028 ceph-mon[29847]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-12T22:19:24.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:19:24 smithi186 ceph-mon[34785]: pgmap v323: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 9 op/s 2024-05-12T22:19:24.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:19:24 smithi186 ceph-mon[34785]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:19:24.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:19:24 smithi186 ceph-mon[34785]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:19:24.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:19:24 smithi086 ceph-mon[34297]: pgmap v323: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 9 op/s 2024-05-12T22:19:24.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:19:24 smithi086 ceph-mon[34297]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:19:24.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:19:24 smithi086 ceph-mon[34297]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:19:24.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:19:24 smithi028 ceph-mon[29847]: pgmap v323: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 9 op/s 2024-05-12T22:19:24.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:19:24 smithi028 ceph-mon[29847]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:19:24.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:19:24 smithi028 ceph-mon[29847]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:19:26.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:19:25 smithi028 ceph-mon[29847]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:19:26.037 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:19:25 smithi028 ceph-mon[29847]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:19:26.037 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:19:25 smithi028 ceph-mon[29847]: pgmap v324: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 93 B/s wr, 11 op/s 2024-05-12T22:19:26.037 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:19:25 smithi028 ceph-mon[29847]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:19:26.037 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:19:25 smithi028 ceph-mon[29847]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:19:26.037 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:19:25 smithi028 ceph-mon[29847]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:19:26.037 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:19:25 smithi028 ceph-mon[29847]: from='mgr.25189 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:19:26.037 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:19:25 smithi028 ceph-mon[29847]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:19:26.037 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:19:25 smithi028 ceph-mon[29847]: from='mgr.25189 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:19:26.038 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:19:25 smithi028 ceph-mon[29847]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:19:26.038 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:19:25 smithi028 ceph-mon[29847]: from='mgr.25189 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:19:26.038 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:19:25 smithi028 ceph-mon[29847]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:19:26.038 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:19:25 smithi028 ceph-mon[29847]: from='mgr.25189 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:19:26.038 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:19:25 smithi028 ceph-mon[29847]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:19:26.038 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:19:25 smithi028 ceph-mon[29847]: from='mgr.25189 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:19:26.038 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:19:25 smithi028 ceph-mon[29847]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:19:26.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:19:25 smithi028 ceph-mon[29847]: from='mgr.25189 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:19:26.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:19:25 smithi028 ceph-mon[29847]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:19:26.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:19:25 smithi028 ceph-mon[29847]: from='mgr.25189 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:19:26.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:19:25 smithi028 ceph-mon[29847]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:19:26.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:19:25 smithi028 ceph-mon[29847]: from='mgr.25189 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:19:26.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:19:25 smithi186 ceph-mon[34785]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:19:26.180 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:19:25 smithi186 ceph-mon[34785]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:19:26.180 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:19:25 smithi186 ceph-mon[34785]: pgmap v324: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 93 B/s wr, 11 op/s 2024-05-12T22:19:26.180 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:19:25 smithi186 ceph-mon[34785]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:19:26.180 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:19:25 smithi186 ceph-mon[34785]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:19:26.180 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:19:25 smithi186 ceph-mon[34785]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:19:26.180 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:19:25 smithi186 ceph-mon[34785]: from='mgr.25189 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:19:26.180 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:19:25 smithi186 ceph-mon[34785]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:19:26.180 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:19:25 smithi186 ceph-mon[34785]: from='mgr.25189 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:19:26.181 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:19:25 smithi186 ceph-mon[34785]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:19:26.181 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:19:25 smithi186 ceph-mon[34785]: from='mgr.25189 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:19:26.181 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:19:25 smithi186 ceph-mon[34785]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:19:26.181 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:19:25 smithi186 ceph-mon[34785]: from='mgr.25189 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:19:26.181 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:19:25 smithi186 ceph-mon[34785]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:19:26.181 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:19:25 smithi186 ceph-mon[34785]: from='mgr.25189 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:19:26.181 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:19:25 smithi186 ceph-mon[34785]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:19:26.182 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:19:25 smithi186 ceph-mon[34785]: from='mgr.25189 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:19:26.182 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:19:25 smithi186 ceph-mon[34785]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:19:26.182 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:19:25 smithi186 ceph-mon[34785]: from='mgr.25189 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:19:26.182 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:19:25 smithi186 ceph-mon[34785]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:19:26.182 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:19:25 smithi186 ceph-mon[34785]: from='mgr.25189 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:19:26.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:19:25 smithi086 ceph-mon[34297]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:19:26.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:19:25 smithi086 ceph-mon[34297]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:19:26.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:19:25 smithi086 ceph-mon[34297]: pgmap v324: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 93 B/s wr, 11 op/s 2024-05-12T22:19:26.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:19:25 smithi086 ceph-mon[34297]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:19:26.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:19:25 smithi086 ceph-mon[34297]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:19:26.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:19:25 smithi086 ceph-mon[34297]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:19:26.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:19:25 smithi086 ceph-mon[34297]: from='mgr.25189 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:19:26.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:19:25 smithi086 ceph-mon[34297]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:19:26.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:19:25 smithi086 ceph-mon[34297]: from='mgr.25189 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:19:26.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:19:25 smithi086 ceph-mon[34297]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:19:26.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:19:25 smithi086 ceph-mon[34297]: from='mgr.25189 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:19:26.243 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:19:25 smithi086 ceph-mon[34297]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:19:26.244 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:19:25 smithi086 ceph-mon[34297]: from='mgr.25189 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:19:26.244 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:19:25 smithi086 ceph-mon[34297]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:19:26.244 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:19:25 smithi086 ceph-mon[34297]: from='mgr.25189 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:19:26.244 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:19:25 smithi086 ceph-mon[34297]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:19:26.244 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:19:25 smithi086 ceph-mon[34297]: from='mgr.25189 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:19:26.244 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:19:25 smithi086 ceph-mon[34297]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:19:26.244 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:19:25 smithi086 ceph-mon[34297]: from='mgr.25189 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:19:26.244 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:19:25 smithi086 ceph-mon[34297]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:19:26.244 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:19:25 smithi086 ceph-mon[34297]: from='mgr.25189 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:19:27.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:19:26 smithi186 ceph-mon[34785]: Adjusting osd_memory_target on smithi086 to 176.2M 2024-05-12T22:19:27.179 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:19:26 smithi186 ceph-mon[34785]: Unable to set osd_memory_target on smithi086 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-05-12T22:19:27.180 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:19:26 smithi186 ceph-mon[34785]: Adjusting osd_memory_target on smithi186 to 176.2M 2024-05-12T22:19:27.180 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:19:26 smithi186 ceph-mon[34785]: Unable to set osd_memory_target on smithi186 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-05-12T22:19:27.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:19:26 smithi086 ceph-mon[34297]: Adjusting osd_memory_target on smithi086 to 176.2M 2024-05-12T22:19:27.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:19:26 smithi086 ceph-mon[34297]: Unable to set osd_memory_target on smithi086 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-05-12T22:19:27.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:19:26 smithi086 ceph-mon[34297]: Adjusting osd_memory_target on smithi186 to 176.2M 2024-05-12T22:19:27.242 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:19:26 smithi086 ceph-mon[34297]: Unable to set osd_memory_target on smithi186 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-05-12T22:19:27.284 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:19:26 smithi028 ceph-mon[29847]: Adjusting osd_memory_target on smithi086 to 176.2M 2024-05-12T22:19:27.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:19:26 smithi028 ceph-mon[29847]: Unable to set osd_memory_target on smithi086 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-05-12T22:19:27.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:19:26 smithi028 ceph-mon[29847]: Adjusting osd_memory_target on smithi186 to 176.2M 2024-05-12T22:19:27.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:19:26 smithi028 ceph-mon[29847]: Unable to set osd_memory_target on smithi186 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-05-12T22:19:28.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:19:28 smithi086 ceph-mon[34297]: pgmap v325: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:19:28.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:19:28 smithi086 ceph-mon[34297]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:19:28.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:19:28 smithi086 ceph-mon[34297]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:19:28.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:19:28 smithi028 ceph-mon[29847]: pgmap v325: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:19:28.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:19:28 smithi028 ceph-mon[29847]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:19:28.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:19:28 smithi028 ceph-mon[29847]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:19:28.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:19:28 smithi186 ceph-mon[34785]: pgmap v325: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:19:28.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:19:28 smithi186 ceph-mon[34785]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:19:28.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:19:28 smithi186 ceph-mon[34785]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:19:29.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:19:29 smithi086 ceph-mon[34297]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:19:29.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:19:29 smithi086 ceph-mon[34297]: from='mgr.25189 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:19:29.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:19:29 smithi086 ceph-mon[34297]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:19:29.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:19:29 smithi086 ceph-mon[34297]: from='mgr.25189 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:19:29.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:19:29 smithi086 ceph-mon[34297]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:19:29.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:19:29 smithi086 ceph-mon[34297]: from='mgr.25189 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:19:29.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:19:29 smithi086 ceph-mon[34297]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:19:29.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:19:29 smithi086 ceph-mon[34297]: from='mgr.25189 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:19:29.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:19:29 smithi086 ceph-mon[34297]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:19:29.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:19:29 smithi086 ceph-mon[34297]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:19:29.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:19:29 smithi086 ceph-mon[34297]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:19:29.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:19:29 smithi028 ceph-mon[29847]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:19:29.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:19:29 smithi028 ceph-mon[29847]: from='mgr.25189 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:19:29.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:19:29 smithi028 ceph-mon[29847]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:19:29.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:19:29 smithi028 ceph-mon[29847]: from='mgr.25189 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:19:29.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:19:29 smithi028 ceph-mon[29847]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:19:29.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:19:29 smithi028 ceph-mon[29847]: from='mgr.25189 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:19:29.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:19:29 smithi028 ceph-mon[29847]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:19:29.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:19:29 smithi028 ceph-mon[29847]: from='mgr.25189 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:19:29.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:19:29 smithi028 ceph-mon[29847]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:19:29.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:19:29 smithi028 ceph-mon[29847]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:19:29.537 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:19:29 smithi028 ceph-mon[29847]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:19:29.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:19:29 smithi186 ceph-mon[34785]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:19:29.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:19:29 smithi186 ceph-mon[34785]: from='mgr.25189 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:19:29.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:19:29 smithi186 ceph-mon[34785]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:19:29.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:19:29 smithi186 ceph-mon[34785]: from='mgr.25189 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:19:29.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:19:29 smithi186 ceph-mon[34785]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:19:29.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:19:29 smithi186 ceph-mon[34785]: from='mgr.25189 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:19:29.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:19:29 smithi186 ceph-mon[34785]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:19:29.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:19:29 smithi186 ceph-mon[34785]: from='mgr.25189 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-12T22:19:29.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:19:29 smithi186 ceph-mon[34785]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-12T22:19:29.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:19:29 smithi186 ceph-mon[34785]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-12T22:19:29.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:19:29 smithi186 ceph-mon[34785]: from='mgr.25189 ' entity='mgr.x' 2024-05-12T22:19:30.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:19:30 smithi086 ceph-mon[34297]: Adjusting osd_memory_target on smithi028 to 177.5M 2024-05-12T22:19:30.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:19:30 smithi086 ceph-mon[34297]: Unable to set osd_memory_target on smithi028 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-05-12T22:19:30.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:19:30 smithi086 ceph-mon[34297]: pgmap v326: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:19:30.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:19:30 smithi028 ceph-mon[29847]: Adjusting osd_memory_target on smithi028 to 177.5M 2024-05-12T22:19:30.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:19:30 smithi028 ceph-mon[29847]: Unable to set osd_memory_target on smithi028 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-05-12T22:19:30.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:19:30 smithi028 ceph-mon[29847]: pgmap v326: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:19:30.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:19:30 smithi186 ceph-mon[34785]: Adjusting osd_memory_target on smithi028 to 177.5M 2024-05-12T22:19:30.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:19:30 smithi186 ceph-mon[34785]: Unable to set osd_memory_target on smithi028 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-05-12T22:19:30.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:19:30 smithi186 ceph-mon[34785]: pgmap v326: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:19:32.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:19:32 smithi086 ceph-mon[34297]: pgmap v327: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-12T22:19:32.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:19:32 smithi028 ceph-mon[29847]: pgmap v327: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-12T22:19:32.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:19:32 smithi186 ceph-mon[34785]: pgmap v327: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-12T22:19:34.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:19:34 smithi086 ceph-mon[34297]: pgmap v328: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:19:34.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:19:34 smithi028 ceph-mon[29847]: pgmap v328: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:19:34.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:19:34 smithi186 ceph-mon[34785]: pgmap v328: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:19:36.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:19:36 smithi086 ceph-mon[34297]: pgmap v329: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-12T22:19:36.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:19:36 smithi028 ceph-mon[29847]: pgmap v329: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-12T22:19:36.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:19:36 smithi186 ceph-mon[34785]: pgmap v329: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-12T22:19:36.699 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-12T22:19:36.699 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-12T22:19:36.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.930+0000 7f75c5a00640 1 -- 172.21.15.28:0/2373789065 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75c01020a0 msgr2=0x7f75c010e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:36.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.930+0000 7f75c5a00640 1 --2- 172.21.15.28:0/2373789065 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75c01020a0 0x7f75c010e5b0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f75b8009cd0 tx=0x7f75b802a8e0 comp rx=0 tx=0).stop 2024-05-12T22:19:36.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.930+0000 7f75c5a00640 1 -- 172.21.15.28:0/2373789065 shutdown_connections 2024-05-12T22:19:36.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.930+0000 7f75c5a00640 1 --2- 172.21.15.28:0/2373789065 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75c01020a0 0x7f75c010e5b0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:36.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.930+0000 7f75c5a00640 1 --2- 172.21.15.28:0/2373789065 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f75c01016e0 0x7f75c0101b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:36.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.930+0000 7f75c5a00640 1 --2- 172.21.15.28:0/2373789065 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f75c01076f0 0x7f75c0107ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:36.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.930+0000 7f75c5a00640 1 -- 172.21.15.28:0/2373789065 >> 172.21.15.28:0/2373789065 conn(0x7f75c00fd1d0 msgr2=0x7f75c00ff5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:19:36.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.930+0000 7f75c5a00640 1 -- 172.21.15.28:0/2373789065 shutdown_connections 2024-05-12T22:19:36.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.930+0000 7f75c5a00640 1 -- 172.21.15.28:0/2373789065 wait complete. 2024-05-12T22:19:36.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.931+0000 7f75c5a00640 1 Processor -- start 2024-05-12T22:19:36.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.931+0000 7f75c5a00640 1 -- start start 2024-05-12T22:19:36.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.932+0000 7f75c5a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75c01016e0 0x7f75c0197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:36.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.932+0000 7f75c5a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f75c01020a0 0x7f75c01984d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:36.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.932+0000 7f75c5a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f75c01076f0 0x7f75c019c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:36.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.932+0000 7f75c5a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f75c0111b50 con 0x7f75c01016e0 2024-05-12T22:19:36.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.932+0000 7f75c5a00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f75c01119d0 con 0x7f75c01020a0 2024-05-12T22:19:36.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.932+0000 7f75c5a00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f75c0111cd0 con 0x7f75c01076f0 2024-05-12T22:19:36.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.932+0000 7f75bfe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75c01016e0 0x7f75c0197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:36.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.932+0000 7f75bfe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75c01016e0 0x7f75c0197f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42264/0 (socket says 172.21.15.28:42264) 2024-05-12T22:19:36.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.932+0000 7f75bfe00640 1 -- 172.21.15.28:0/3347073345 learned_addr learned my addr 172.21.15.28:0/3347073345 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:19:36.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.932+0000 7f75c5000640 1 --2- 172.21.15.28:0/3347073345 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f75c01076f0 0x7f75c019c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:36.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.932+0000 7f75b7e00640 1 --2- 172.21.15.28:0/3347073345 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f75c01020a0 0x7f75c01984d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:36.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.932+0000 7f75bfe00640 1 -- 172.21.15.28:0/3347073345 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f75c01076f0 msgr2=0x7f75c019c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:36.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.932+0000 7f75bfe00640 1 --2- 172.21.15.28:0/3347073345 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f75c01076f0 0x7f75c019c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:36.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.933+0000 7f75bfe00640 1 -- 172.21.15.28:0/3347073345 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f75c01020a0 msgr2=0x7f75c01984d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:36.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.933+0000 7f75bfe00640 1 --2- 172.21.15.28:0/3347073345 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f75c01020a0 0x7f75c01984d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:36.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.933+0000 7f75bfe00640 1 -- 172.21.15.28:0/3347073345 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75c019cf80 con 0x7f75c01016e0 2024-05-12T22:19:36.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.933+0000 7f75b7e00640 1 --2- 172.21.15.28:0/3347073345 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f75c01020a0 0x7f75c01984d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:19:36.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.933+0000 7f75bfe00640 1 --2- 172.21.15.28:0/3347073345 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75c01016e0 0x7f75c0197f90 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f75b000e9f0 tx=0x7f75b000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:19:36.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.933+0000 7f75bd600640 1 -- 172.21.15.28:0/3347073345 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75b000cd80 con 0x7f75c01016e0 2024-05-12T22:19:36.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.933+0000 7f75bd600640 1 -- 172.21.15.28:0/3347073345 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f75b0005500 con 0x7f75c01016e0 2024-05-12T22:19:36.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.933+0000 7f75bd600640 1 -- 172.21.15.28:0/3347073345 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75b0010690 con 0x7f75c01016e0 2024-05-12T22:19:36.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.933+0000 7f75c5a00640 1 -- 172.21.15.28:0/3347073345 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75c019d270 con 0x7f75c01016e0 2024-05-12T22:19:36.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.933+0000 7f75c5a00640 1 -- 172.21.15.28:0/3347073345 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75c01a4b00 con 0x7f75c01016e0 2024-05-12T22:19:36.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.937+0000 7f75bd600640 1 -- 172.21.15.28:0/3347073345 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7f75b00046c0 con 0x7f75c01016e0 2024-05-12T22:19:36.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.937+0000 7f75c9f8f740 1 Processor -- start 2024-05-12T22:19:36.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.937+0000 7f75c9f8f740 1 -- start start 2024-05-12T22:19:36.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.937+0000 7f75c9f8f740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5650710911e0 0x5650710915c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:36.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.937+0000 7f75c9f8f740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x565071091b00 0x56507109bc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:36.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.937+0000 7f75c9f8f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56507109c1d0 0x56507109e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:36.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.937+0000 7f75c9f8f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5650710552a0 con 0x56507109c1d0 2024-05-12T22:19:36.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.937+0000 7f75c9f8f740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x565070ff1b90 con 0x5650710911e0 2024-05-12T22:19:36.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.937+0000 7f75c9f8f740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x565070f995b0 con 0x565071091b00 2024-05-12T22:19:36.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.938+0000 7f75b7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56507109c1d0 0x56507109e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:36.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.938+0000 7f75b7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56507109c1d0 0x56507109e5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42266/0 (socket says 172.21.15.28:42266) 2024-05-12T22:19:36.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.938+0000 7f75b7e00640 1 -- 172.21.15.28:0/981838231 learned_addr learned my addr 172.21.15.28:0/981838231 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:19:36.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.938+0000 7f75c5000640 1 --2- 172.21.15.28:0/981838231 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5650710911e0 0x5650710915c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:36.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.938+0000 7f75bfe00640 1 --2- 172.21.15.28:0/981838231 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x565071091b00 0x56507109bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:36.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.938+0000 7f75b7e00640 1 -- 172.21.15.28:0/981838231 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x565071091b00 msgr2=0x56507109bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:19:36.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.938+0000 7f75b7e00640 1 --2- 172.21.15.28:0/981838231 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x565071091b00 0x56507109bc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:36.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.938+0000 7f75b7e00640 1 -- 172.21.15.28:0/981838231 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5650710911e0 msgr2=0x5650710915c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:19:36.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.938+0000 7f75b7e00640 1 --2- 172.21.15.28:0/981838231 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5650710911e0 0x5650710915c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:36.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.938+0000 7f75b7e00640 1 -- 172.21.15.28:0/981838231 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565070dcf7d0 con 0x56507109c1d0 2024-05-12T22:19:36.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.938+0000 7f75bd600640 1 --2- 172.21.15.28:0/3347073345 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f75a40a2d00 0x7f75a40a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:36.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.939+0000 7f75b7e00640 1 --2- 172.21.15.28:0/981838231 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56507109c1d0 0x56507109e5c0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f75a8009cf0 tx=0x7f75a80117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e82e2a18bce674e0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:19:36.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.939+0000 7f75bd600640 1 -- 172.21.15.28:0/3347073345 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8513+0+0 (secure 0 0 0) 0x7f75b00cf910 con 0x7f75c01016e0 2024-05-12T22:19:36.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.939+0000 7f75c5000640 1 --2- 172.21.15.28:0/3347073345 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f75a40a2d00 0x7f75a40a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:36.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.939+0000 7f75b5600640 1 -- 172.21.15.28:0/981838231 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75a8012450 con 0x56507109c1d0 2024-05-12T22:19:36.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.939+0000 7f75b5600640 1 -- 172.21.15.28:0/981838231 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f75a8012ad0 con 0x56507109c1d0 2024-05-12T22:19:36.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.939+0000 7f75b5600640 1 -- 172.21.15.28:0/981838231 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75a800fbf0 con 0x56507109c1d0 2024-05-12T22:19:36.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.940+0000 7f75c9f8f740 1 -- 172.21.15.28:0/981838231 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56507109c1d0 msgr2=0x56507109e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:36.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.940+0000 7f75c5000640 1 --2- 172.21.15.28:0/3347073345 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f75a40a2d00 0x7f75a40a51c0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f75b8005e30 tx=0x7f75b803d040 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:19:36.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.940+0000 7f75c9f8f740 1 --2- 172.21.15.28:0/981838231 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56507109c1d0 0x56507109e5c0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f75a8009cf0 tx=0x7f75a80117b0 comp rx=0 tx=0).stop 2024-05-12T22:19:36.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.940+0000 7f75c9f8f740 1 -- 172.21.15.28:0/981838231 shutdown_connections 2024-05-12T22:19:36.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.940+0000 7f75c9f8f740 1 --2- 172.21.15.28:0/981838231 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56507109c1d0 0x56507109e5c0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:36.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.940+0000 7f75c9f8f740 1 --2- 172.21.15.28:0/981838231 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x565071091b00 0x56507109bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:36.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.940+0000 7f75c9f8f740 1 --2- 172.21.15.28:0/981838231 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5650710911e0 0x5650710915c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:36.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.940+0000 7f75c9f8f740 1 -- 172.21.15.28:0/981838231 >> 172.21.15.28:0/981838231 conn(0x56507108d500 msgr2=0x56507108d8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:19:36.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.940+0000 7f75c9f8f740 1 -- 172.21.15.28:0/981838231 shutdown_connections 2024-05-12T22:19:36.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.940+0000 7f75c9f8f740 1 -- 172.21.15.28:0/981838231 wait complete. 2024-05-12T22:19:36.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.940+0000 7f75c9f8f740 1 -- start start 2024-05-12T22:19:36.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.941+0000 7f75c9f8f740 1 Processor -- start 2024-05-12T22:19:36.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.941+0000 7f75c9f8f740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5650710911e0 0x56507109ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:36.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.941+0000 7f75c9f8f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x565071091b00 0x565071098d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:36.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.941+0000 7f75c9f8f740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56507109c1d0 0x565071099290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:36.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.941+0000 7f75c9f8f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x565070fac9f0 con 0x565071091b00 2024-05-12T22:19:36.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.941+0000 7f75c9f8f740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x565070fedb50 con 0x5650710911e0 2024-05-12T22:19:36.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.941+0000 7f75c9f8f740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x56507109efc0 con 0x56507109c1d0 2024-05-12T22:19:36.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.941+0000 7f75c9f8f740 10 client.0 _ll_register_callbacks cb 0x565070d04be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:19:36.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.942+0000 7f75bfe00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5650710911e0 0x56507109ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:36.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.942+0000 7f75c5000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56507109c1d0 0x565071099290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:36.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.942+0000 7f75b7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x565071091b00 0x565071098d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:36.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.942+0000 7f75bfe00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5650710911e0 0x56507109ba60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:55540/0 (socket says 172.21.15.28:55540) 2024-05-12T22:19:36.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.942+0000 7f75bfe00640 1 -- 172.21.15.28:0/3315879318 learned_addr learned my addr 172.21.15.28:0/3315879318 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:19:36.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.942+0000 7f75bfe00640 1 -- 172.21.15.28:0/3315879318 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56507109c1d0 msgr2=0x565071099290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:19:36.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.942+0000 7f75bfe00640 1 --2- 172.21.15.28:0/3315879318 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56507109c1d0 0x565071099290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:36.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.942+0000 7f75bfe00640 1 -- 172.21.15.28:0/3315879318 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x565071091b00 msgr2=0x565071098d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:19:36.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.942+0000 7f75bfe00640 1 --2- 172.21.15.28:0/3315879318 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x565071091b00 0x565071098d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:36.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.942+0000 7f75bfe00640 1 -- 172.21.15.28:0/3315879318 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565070d053f0 con 0x5650710911e0 2024-05-12T22:19:36.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.942+0000 7f75c5000640 1 --2- 172.21.15.28:0/3315879318 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56507109c1d0 0x565071099290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:19:36.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.942+0000 7f75bfe00640 1 --2- 172.21.15.28:0/3315879318 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5650710911e0 0x56507109ba60 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f75b000e9c0 tx=0x7f75b0098070 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f9bd7a6f2670450a server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:19:36.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.942+0000 7f75ae000640 10 client.0 ms_handle_connect on v2:172.21.15.86:3300/0 2024-05-12T22:19:36.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.942+0000 7f75ae000640 1 -- 172.21.15.28:0/3315879318 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75b009b730 con 0x5650710911e0 2024-05-12T22:19:36.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.942+0000 7f75c9f8f740 1 -- 172.21.15.28:0/3315879318 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f75a8012ad0 con 0x5650710911e0 2024-05-12T22:19:36.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.942+0000 7f75ae000640 1 -- 172.21.15.28:0/3315879318 <== mon.2 v2:172.21.15.86:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8513+0+0 (secure 0 0 0) 0x7f75b00a47b0 con 0x5650710911e0 2024-05-12T22:19:36.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.943+0000 7f75ae000640 1 -- 172.21.15.28:0/3315879318 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f75b00cd490 con 0x5650710911e0 2024-05-12T22:19:36.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.943+0000 7f75c9f8f740 10 client.27500 fetch_fsmap learned FSMap version 26 2024-05-12T22:19:36.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.943+0000 7f75c9f8f740 1 -- 172.21.15.28:0/3315879318 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x565070cc50f0 con 0x5650710911e0 2024-05-12T22:19:36.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.943+0000 7f75ae000640 1 -- 172.21.15.28:0/3315879318 <== mon.2 v2:172.21.15.86:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f75a8012ad0 con 0x5650710911e0 2024-05-12T22:19:36.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.943+0000 7f75ae000640 1 -- 172.21.15.28:0/3315879318 <== mon.2 v2:172.21.15.86:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f75b00b1030 con 0x5650710911e0 2024-05-12T22:19:36.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.944+0000 7f75c9f8f740 10 client.27500 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:19:36.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.944+0000 7f75c9f8f740 10 client.27500 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:19:36.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.944+0000 7f75c9f8f740 20 client.27500 populate_metadata read hostname 'smithi028' 2024-05-12T22:19:36.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.944+0000 7f75c9f8f740 1 --2- 172.21.15.28:0/3315879318 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5650710abc60 0x5650710ae070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:36.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.944+0000 7f75c9f8f740 4 client.27500 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-12T22:19:36.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.944+0000 7f75c9f8f740 1 -- 172.21.15.28:0/3315879318 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x565070fff2c0 con 0x5650710abc60 2024-05-12T22:19:36.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.944+0000 7f75b7e00640 1 --2- 172.21.15.28:0/3315879318 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5650710abc60 0x5650710ae070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:36.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.945+0000 7f75b7e00640 1 --2- 172.21.15.28:0/3315879318 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5650710abc60 0x5650710ae070 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d4c7911c20f4b0d7 server_cookie=a26a9577fd4e507a in_seq=0 out_seq=0 2024-05-12T22:19:36.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.945+0000 7f75ae000640 10 client.27500 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:19:36.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.946+0000 7f75ae000640 1 -- 172.21.15.28:0/3315879318 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x565070fff2c0 con 0x5650710abc60 2024-05-12T22:19:36.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.946+0000 7f75ae000640 10 client.27500 handle_command_reply: tid=0 2024-05-12T22:19:36.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.946+0000 7f75c9f8f740 1 client.27500 shutdown 2024-05-12T22:19:36.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.946+0000 7f75c9f8f740 1 -- 172.21.15.28:0/3315879318 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5650710911e0 msgr2=0x56507109ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:36.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.946+0000 7f75c9f8f740 1 --2- 172.21.15.28:0/3315879318 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5650710911e0 0x56507109ba60 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f75b000e9c0 tx=0x7f75b0098070 comp rx=0 tx=0).stop 2024-05-12T22:19:36.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.946+0000 7f75c9f8f740 1 -- 172.21.15.28:0/3315879318 shutdown_connections 2024-05-12T22:19:36.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.946+0000 7f75c9f8f740 1 --2- 172.21.15.28:0/3315879318 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5650710abc60 0x5650710ae070 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:36.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.946+0000 7f75c9f8f740 1 --2- 172.21.15.28:0/3315879318 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56507109c1d0 0x565071099290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:36.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.946+0000 7f75c9f8f740 1 --2- 172.21.15.28:0/3315879318 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x565071091b00 0x565071098d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:36.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.946+0000 7f75c9f8f740 1 --2- 172.21.15.28:0/3315879318 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5650710911e0 0x56507109ba60 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:36.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.946+0000 7f75c9f8f740 1 -- 172.21.15.28:0/3315879318 >> 172.21.15.28:0/3315879318 conn(0x56507108d500 msgr2=0x565071088330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:19:36.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.946+0000 7f75ae000640 0 client.27500 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:19:36.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.947+0000 7f75c9f8f740 1 -- 172.21.15.28:0/3315879318 shutdown_connections 2024-05-12T22:19:36.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.947+0000 7f75c9f8f740 1 -- 172.21.15.28:0/3315879318 wait complete. 2024-05-12T22:19:36.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.947+0000 7f75c9f8f740 20 client.27500 trim_cache size 0 max 16384 2024-05-12T22:19:36.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.967+0000 7f75c9f8f740 1 Processor -- start 2024-05-12T22:19:36.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.967+0000 7f75c9f8f740 1 -- start start 2024-05-12T22:19:36.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.967+0000 7f75c9f8f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5650710911e0 0x565071097eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:36.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.967+0000 7f75c9f8f740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x565071091b00 0x56507109b590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:36.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.967+0000 7f75c9f8f740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56507109c1d0 0x5650710b3940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:36.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.967+0000 7f75c9f8f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f75c01adac0 con 0x5650710911e0 2024-05-12T22:19:36.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.967+0000 7f75c9f8f740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f75c01ab390 con 0x56507109c1d0 2024-05-12T22:19:36.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.967+0000 7f75c9f8f740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f75c01a9200 con 0x565071091b00 2024-05-12T22:19:36.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.968+0000 7f75bfe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5650710911e0 0x565071097eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:36.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.968+0000 7f75c5000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56507109c1d0 0x5650710b3940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:36.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.968+0000 7f75b7e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x565071091b00 0x56507109b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:36.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.968+0000 7f75c5000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56507109c1d0 0x5650710b3940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:55542/0 (socket says 172.21.15.28:55542) 2024-05-12T22:19:36.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.968+0000 7f75c5000640 1 -- 172.21.15.28:0/1892327842 learned_addr learned my addr 172.21.15.28:0/1892327842 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:19:36.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.968+0000 7f75c5000640 1 -- 172.21.15.28:0/1892327842 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x565071091b00 msgr2=0x56507109b590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:19:36.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.968+0000 7f75c5000640 1 --2- 172.21.15.28:0/1892327842 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x565071091b00 0x56507109b590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:36.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.968+0000 7f75c5000640 1 -- 172.21.15.28:0/1892327842 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5650710911e0 msgr2=0x565071097eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:19:36.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.968+0000 7f75c5000640 1 --2- 172.21.15.28:0/1892327842 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5650710911e0 0x565071097eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:36.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.968+0000 7f75c5000640 1 -- 172.21.15.28:0/1892327842 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56507108d330 con 0x56507109c1d0 2024-05-12T22:19:36.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.968+0000 7f75b7e00640 1 --2- 172.21.15.28:0/1892327842 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x565071091b00 0x56507109b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:19:36.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.969+0000 7f75c5000640 1 --2- 172.21.15.28:0/1892327842 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56507109c1d0 0x5650710b3940 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f75b8009cd0 tx=0x7f75b8045000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ee061bc56dfddde1 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:19:36.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.969+0000 7f75b4c00640 1 -- 172.21.15.28:0/1892327842 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75b8004510 con 0x56507109c1d0 2024-05-12T22:19:36.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.969+0000 7f75b4c00640 1 -- 172.21.15.28:0/1892327842 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f75b8056070 con 0x56507109c1d0 2024-05-12T22:19:36.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.969+0000 7f75b4c00640 1 -- 172.21.15.28:0/1892327842 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75b8046700 con 0x56507109c1d0 2024-05-12T22:19:36.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.970+0000 7f75c9f8f740 1 -- 172.21.15.28:0/1892327842 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56507109c1d0 msgr2=0x5650710b3940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:36.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.970+0000 7f75c9f8f740 1 --2- 172.21.15.28:0/1892327842 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56507109c1d0 0x5650710b3940 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f75b8009cd0 tx=0x7f75b8045000 comp rx=0 tx=0).stop 2024-05-12T22:19:36.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.970+0000 7f75c9f8f740 1 -- 172.21.15.28:0/1892327842 shutdown_connections 2024-05-12T22:19:36.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.970+0000 7f75c9f8f740 1 --2- 172.21.15.28:0/1892327842 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56507109c1d0 0x5650710b3940 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:36.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.970+0000 7f75c9f8f740 1 --2- 172.21.15.28:0/1892327842 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x565071091b00 0x56507109b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:36.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.970+0000 7f75c9f8f740 1 --2- 172.21.15.28:0/1892327842 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5650710911e0 0x565071097eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:36.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.970+0000 7f75c9f8f740 1 -- 172.21.15.28:0/1892327842 >> 172.21.15.28:0/1892327842 conn(0x56507108d500 msgr2=0x5650710942a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:19:36.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.970+0000 7f75c9f8f740 1 -- 172.21.15.28:0/1892327842 shutdown_connections 2024-05-12T22:19:36.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.970+0000 7f75c9f8f740 1 -- 172.21.15.28:0/1892327842 wait complete. 2024-05-12T22:19:36.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.970+0000 7f75c9f8f740 1 -- start start 2024-05-12T22:19:36.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.970+0000 7f75c9f8f740 1 Processor -- start 2024-05-12T22:19:36.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.971+0000 7f75c9f8f740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5650710911e0 0x5650710a4310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:36.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.971+0000 7f75c9f8f740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x565071091b00 0x56507109f800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:36.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.971+0000 7f75c9f8f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56507109c1d0 0x56507109fd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:36.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.971+0000 7f75c9f8f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f75c01a9850 con 0x56507109c1d0 2024-05-12T22:19:36.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.971+0000 7f75c9f8f740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f75c01aacc0 con 0x5650710911e0 2024-05-12T22:19:36.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.971+0000 7f75c9f8f740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f75c01a9f20 con 0x565071091b00 2024-05-12T22:19:36.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.971+0000 7f75c9f8f740 10 client.0 _ll_register_callbacks cb 0x7f75b00d5220 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:19:36.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.971+0000 7f75c5000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56507109c1d0 0x56507109fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:36.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.971+0000 7f75c5000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56507109c1d0 0x56507109fd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42290/0 (socket says 172.21.15.28:42290) 2024-05-12T22:19:36.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.971+0000 7f75c5000640 1 -- 172.21.15.28:0/2666812389 learned_addr learned my addr 172.21.15.28:0/2666812389 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:19:36.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.971+0000 7f75bfe00640 1 --2- 172.21.15.28:0/2666812389 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5650710911e0 0x5650710a4310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:36.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.971+0000 7f75b7e00640 1 --2- 172.21.15.28:0/2666812389 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x565071091b00 0x56507109f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:36.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.972+0000 7f75c5000640 1 -- 172.21.15.28:0/2666812389 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x565071091b00 msgr2=0x56507109f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:19:36.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.972+0000 7f75c5000640 1 --2- 172.21.15.28:0/2666812389 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x565071091b00 0x56507109f800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:36.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.972+0000 7f75c5000640 1 -- 172.21.15.28:0/2666812389 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5650710911e0 msgr2=0x5650710a4310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:19:36.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.972+0000 7f75c5000640 1 --2- 172.21.15.28:0/2666812389 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5650710911e0 0x5650710a4310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:36.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.972+0000 7f75c5000640 1 -- 172.21.15.28:0/2666812389 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5650710aa8b0 con 0x56507109c1d0 2024-05-12T22:19:36.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.972+0000 7f75c5000640 1 --2- 172.21.15.28:0/2666812389 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56507109c1d0 0x56507109fd40 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f75b8006010 tx=0x7f75b8009c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5136ba977b4ed032 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:19:36.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.972+0000 7f75ae000640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-12T22:19:36.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.972+0000 7f75ae000640 1 -- 172.21.15.28:0/2666812389 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75b805e030 con 0x56507109c1d0 2024-05-12T22:19:36.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.972+0000 7f75ae000640 1 -- 172.21.15.28:0/2666812389 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8513+0+0 (secure 0 0 0) 0x7f75b80307b0 con 0x56507109c1d0 2024-05-12T22:19:36.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.972+0000 7f75c9f8f740 1 -- 172.21.15.28:0/2666812389 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f75b8056070 con 0x56507109c1d0 2024-05-12T22:19:36.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.973+0000 7f75ae000640 1 -- 172.21.15.28:0/2666812389 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f75b8004860 con 0x56507109c1d0 2024-05-12T22:19:36.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.973+0000 7f75c9f8f740 10 client.17373 fetch_fsmap learned FSMap version 26 2024-05-12T22:19:36.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.973+0000 7f75c9f8f740 1 -- 172.21.15.28:0/2666812389 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x565071111ca0 con 0x56507109c1d0 2024-05-12T22:19:36.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.973+0000 7f75ae000640 1 -- 172.21.15.28:0/2666812389 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f75b8056070 con 0x56507109c1d0 2024-05-12T22:19:36.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.973+0000 7f75ae000640 1 -- 172.21.15.28:0/2666812389 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f75b8030070 con 0x56507109c1d0 2024-05-12T22:19:36.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.973+0000 7f75c9f8f740 10 client.17373 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:19:36.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.973+0000 7f75c9f8f740 10 client.17373 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:19:36.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.973+0000 7f75c9f8f740 20 client.17373 populate_metadata read hostname 'smithi028' 2024-05-12T22:19:36.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.973+0000 7f75c9f8f740 1 --2- 172.21.15.28:0/2666812389 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5650711123c0 0x5650711147e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:36.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.973+0000 7f75c9f8f740 4 client.17373 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-12T22:19:36.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.973+0000 7f75c9f8f740 1 -- 172.21.15.28:0/2666812389 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "scrub status"}) -- 0x565070ff9610 con 0x5650711123c0 2024-05-12T22:19:36.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.974+0000 7f75bfe00640 1 --2- 172.21.15.28:0/2666812389 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5650711123c0 0x5650711147e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:36.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.974+0000 7f75bfe00640 1 --2- 172.21.15.28:0/2666812389 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5650711123c0 0x5650711147e0 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=540acca151d727fd server_cookie=5f2505eb72bca0c4 in_seq=0 out_seq=0 2024-05-12T22:19:36.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.974+0000 7f75ae000640 10 client.17373 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:19:36.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.975+0000 7f75ae000640 1 -- 172.21.15.28:0/2666812389 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x565070ff9610 con 0x5650711123c0 2024-05-12T22:19:36.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.975+0000 7f75ae000640 10 client.17373 handle_command_reply: tid=0 2024-05-12T22:19:36.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.975+0000 7f75c9f8f740 1 client.17373 shutdown 2024-05-12T22:19:36.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.975+0000 7f75c9f8f740 1 -- 172.21.15.28:0/2666812389 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56507109c1d0 msgr2=0x56507109fd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:36.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.975+0000 7f75c9f8f740 1 --2- 172.21.15.28:0/2666812389 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56507109c1d0 0x56507109fd40 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f75b8006010 tx=0x7f75b8009c20 comp rx=0 tx=0).stop 2024-05-12T22:19:36.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.975+0000 7f75c9f8f740 1 -- 172.21.15.28:0/2666812389 shutdown_connections 2024-05-12T22:19:36.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.975+0000 7f75c9f8f740 1 --2- 172.21.15.28:0/2666812389 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x5650711123c0 0x5650711147e0 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:36.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.975+0000 7f75c9f8f740 1 --2- 172.21.15.28:0/2666812389 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56507109c1d0 0x56507109fd40 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:36.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.975+0000 7f75c9f8f740 1 --2- 172.21.15.28:0/2666812389 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x565071091b00 0x56507109f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:36.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.975+0000 7f75c9f8f740 1 --2- 172.21.15.28:0/2666812389 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5650710911e0 0x5650710a4310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:36.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.975+0000 7f75c9f8f740 1 -- 172.21.15.28:0/2666812389 >> 172.21.15.28:0/2666812389 conn(0x56507108d500 msgr2=0x565071097eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:19:36.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.975+0000 7f75ae000640 0 client.17373 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:19:36.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.975+0000 7f75c9f8f740 1 -- 172.21.15.28:0/2666812389 shutdown_connections 2024-05-12T22:19:36.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.975+0000 7f75c9f8f740 1 -- 172.21.15.28:0/2666812389 wait complete. 2024-05-12T22:19:36.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.975+0000 7f75c9f8f740 20 client.17373 trim_cache size 0 max 16384 2024-05-12T22:19:36.992 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-05-12T22:19:36.992 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-05-12T22:19:36.992 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-05-12T22:19:36.992 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-05-12T22:19:36.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.976+0000 7f75c5a00640 1 -- 172.21.15.28:0/3347073345 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f75a40a2d00 msgr2=0x7f75a40a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:36.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.976+0000 7f75c5a00640 1 --2- 172.21.15.28:0/3347073345 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f75a40a2d00 0x7f75a40a51c0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f75b8005e30 tx=0x7f75b803d040 comp rx=0 tx=0).stop 2024-05-12T22:19:36.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.976+0000 7f75c5a00640 1 -- 172.21.15.28:0/3347073345 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75c01016e0 msgr2=0x7f75c0197f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:36.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.976+0000 7f75c5a00640 1 --2- 172.21.15.28:0/3347073345 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75c01016e0 0x7f75c0197f90 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f75b000e9f0 tx=0x7f75b000edf0 comp rx=0 tx=0).stop 2024-05-12T22:19:36.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.976+0000 7f75c5a00640 1 -- 172.21.15.28:0/3347073345 shutdown_connections 2024-05-12T22:19:36.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.976+0000 7f75c5a00640 1 --2- 172.21.15.28:0/3347073345 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f75a40a2d00 0x7f75a40a51c0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:36.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.976+0000 7f75c5a00640 1 --2- 172.21.15.28:0/3347073345 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f75c01076f0 0x7f75c019c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:36.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.976+0000 7f75c5a00640 1 --2- 172.21.15.28:0/3347073345 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f75c01020a0 0x7f75c01984d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:36.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.976+0000 7f75c5a00640 1 --2- 172.21.15.28:0/3347073345 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75c01016e0 0x7f75c0197f90 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:36.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.976+0000 7f75c5a00640 1 -- 172.21.15.28:0/3347073345 >> 172.21.15.28:0/3347073345 conn(0x7f75c00fd1d0 msgr2=0x7f75c00fd910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:19:36.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.976+0000 7f75c5a00640 1 -- 172.21.15.28:0/3347073345 shutdown_connections 2024-05-12T22:19:36.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:36.976+0000 7f75c5a00640 1 -- 172.21.15.28:0/3347073345 wait complete. 2024-05-12T22:19:36.997 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-12T22:19:36.998 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-12T22:19:37.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.214+0000 7f8860c00640 1 -- 172.21.15.28:0/469522619 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f885c107cd0 msgr2=0x7f885c108130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:37.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.214+0000 7f8860c00640 1 --2- 172.21.15.28:0/469522619 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f885c107cd0 0x7f885c108130 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f8844009900 tx=0x7f884402a480 comp rx=0 tx=0).stop 2024-05-12T22:19:37.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.214+0000 7f8860c00640 1 -- 172.21.15.28:0/469522619 shutdown_connections 2024-05-12T22:19:37.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.214+0000 7f8860c00640 1 --2- 172.21.15.28:0/469522619 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f885c108670 0x7f885c114ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:37.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.214+0000 7f8860c00640 1 --2- 172.21.15.28:0/469522619 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f885c107cd0 0x7f885c108130 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:37.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.214+0000 7f8860c00640 1 --2- 172.21.15.28:0/469522619 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f885c10dd40 0x7f885c10e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:37.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.214+0000 7f8860c00640 1 -- 172.21.15.28:0/469522619 >> 172.21.15.28:0/469522619 conn(0x7f885c0773b0 msgr2=0x7f885c0797d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:19:37.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.214+0000 7f8860c00640 1 -- 172.21.15.28:0/469522619 shutdown_connections 2024-05-12T22:19:37.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.214+0000 7f8860c00640 1 -- 172.21.15.28:0/469522619 wait complete. 2024-05-12T22:19:37.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.215+0000 7f8860c00640 1 Processor -- start 2024-05-12T22:19:37.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.215+0000 7f8860c00640 1 -- start start 2024-05-12T22:19:37.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.216+0000 7f8860c00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f885c107cd0 0x7f885c19e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:37.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.216+0000 7f8860c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f885c108670 0x7f885c19ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:37.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.216+0000 7f8860c00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f885c10dd40 0x7f885c1a2de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:37.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.216+0000 7f8860c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f885c118020 con 0x7f885c108670 2024-05-12T22:19:37.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.216+0000 7f8860c00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f885c117ea0 con 0x7f885c10dd40 2024-05-12T22:19:37.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.216+0000 7f8860c00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f885c1181a0 con 0x7f885c107cd0 2024-05-12T22:19:37.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.216+0000 7f885b400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f885c107cd0 0x7f885c19e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:37.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.216+0000 7f885b400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f885c107cd0 0x7f885c19e520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:57506/0 (socket says 172.21.15.28:57506) 2024-05-12T22:19:37.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.216+0000 7f885b400640 1 -- 172.21.15.28:0/2331228972 learned_addr learned my addr 172.21.15.28:0/2331228972 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:19:37.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.216+0000 7f8852a00640 1 --2- 172.21.15.28:0/2331228972 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f885c108670 0x7f885c19ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:37.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.216+0000 7f885be00640 1 --2- 172.21.15.28:0/2331228972 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f885c10dd40 0x7f885c1a2de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:37.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.217+0000 7f885b400640 1 -- 172.21.15.28:0/2331228972 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f885c10dd40 msgr2=0x7f885c1a2de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:37.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.217+0000 7f885b400640 1 --2- 172.21.15.28:0/2331228972 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f885c10dd40 0x7f885c1a2de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:37.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.217+0000 7f885b400640 1 -- 172.21.15.28:0/2331228972 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f885c108670 msgr2=0x7f885c19ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:37.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.217+0000 7f885b400640 1 --2- 172.21.15.28:0/2331228972 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f885c108670 0x7f885c19ea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:37.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.217+0000 7f885b400640 1 -- 172.21.15.28:0/2331228972 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f885c1a3510 con 0x7f885c107cd0 2024-05-12T22:19:37.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.217+0000 7f885b400640 1 --2- 172.21.15.28:0/2331228972 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f885c107cd0 0x7f885c19e520 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f884c00ea60 tx=0x7f884c00ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:19:37.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.217+0000 7f8858c00640 1 -- 172.21.15.28:0/2331228972 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f884c00cde0 con 0x7f885c107cd0 2024-05-12T22:19:37.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.217+0000 7f8860c00640 1 -- 172.21.15.28:0/2331228972 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f885c1a3800 con 0x7f885c107cd0 2024-05-12T22:19:37.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.217+0000 7f8858c00640 1 -- 172.21.15.28:0/2331228972 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f884c0035f0 con 0x7f885c107cd0 2024-05-12T22:19:37.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.217+0000 7f8858c00640 1 -- 172.21.15.28:0/2331228972 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f884c010670 con 0x7f885c107cd0 2024-05-12T22:19:37.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.217+0000 7f8860c00640 1 -- 172.21.15.28:0/2331228972 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f885c1ab090 con 0x7f885c107cd0 2024-05-12T22:19:37.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.219+0000 7f8860c00640 1 -- 172.21.15.28:0/2331228972 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f885c0689c0 con 0x7f885c107cd0 2024-05-12T22:19:37.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.221+0000 7f8858c00640 1 -- 172.21.15.28:0/2331228972 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7f884c0037a0 con 0x7f885c107cd0 2024-05-12T22:19:37.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.222+0000 7f8858c00640 1 --2- 172.21.15.28:0/2331228972 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f88400a2d00 0x7f88400a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:37.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.222+0000 7f8858c00640 1 -- 172.21.15.28:0/2331228972 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8513+0+0 (secure 0 0 0) 0x7f884c0cf720 con 0x7f885c107cd0 2024-05-12T22:19:37.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.222+0000 7f8852a00640 1 --2- 172.21.15.28:0/2331228972 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f88400a2d00 0x7f88400a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:37.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.223+0000 7f8852a00640 1 --2- 172.21.15.28:0/2331228972 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f88400a2d00 0x7f88400a51c0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f8844009230 tx=0x7f88440058d0 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:19:37.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.227+0000 7f8858c00640 1 -- 172.21.15.28:0/2331228972 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f884c09c2e0 con 0x7f885c107cd0 2024-05-12T22:19:37.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.475+0000 7f8860c00640 1 -- 172.21.15.28:0/2331228972 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f885c19f870 con 0x7f885c107cd0 2024-05-12T22:19:37.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.477+0000 7f8858c00640 1 -- 172.21.15.28:0/2331228972 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11864 (secure 0 0 0) 0x7f884c09bbe0 con 0x7f885c107cd0 2024-05-12T22:19:37.476 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-12T22:19:37.477 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1844141745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1844141745},{"type":"v1","addr":"172.21.15.186:6833","nonce":1844141745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6833/2960763758","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":2960763758},{"type":"v1","addr":"172.21.15.86:6833","nonce":2960763758}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24470,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/4131134219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":4131134219},{"type":"v1","addr":"172.21.15.28:6835","nonce":4131134219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24481,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/1727869064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1727869064},{"type":"v1","addr":"172.21.15.186:6835","nonce":1727869064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24482,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2938260704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2938260704},{"type":"v1","addr":"172.21.15.28:6837","nonce":2938260704}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24494,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3473700910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3473700910},{"type":"v1","addr":"172.21.15.86:6835","nonce":3473700910}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-12T22:06:33.426986+0000","modified":"2024-05-12T22:13:22.499381+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24581,"mds_1":24550,"mds_2":24532},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24499":{"gid":24499,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6837/3600809235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":3600809235},{"type":"v1","addr":"172.21.15.186:6837","nonce":3600809235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"i","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6839/2399610807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":2399610807},{"type":"v1","addr":"172.21.15.186:6839","nonce":2399610807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24530":{"gid":24530,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.86:6837/3782448884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":3782448884},{"type":"v1","addr":"172.21.15.86:6837","nonce":3782448884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24532":{"gid":24532,"name":"g","rank":2,"incarnation":21,"state":"up:active","state_seq":6,"addr":"172.21.15.28:6839/123892158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":123892158},{"type":"v1","addr":"172.21.15.28:6839","nonce":123892158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24550":{"gid":24550,"name":"a","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.28:6841/761183870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":761183870},{"type":"v1","addr":"172.21.15.28:6841","nonce":761183870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24581":{"gid":24581,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6839/1251999130","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6838","nonce":1251999130},{"type":"v1","addr":"172.21.15.86:6839","nonce":1251999130}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24581,"qdb_cluster":[24550,24581,24532]},"id":1}]} 2024-05-12T22:19:37.477 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 26 2024-05-12T22:19:37.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.480+0000 7f8860c00640 1 -- 172.21.15.28:0/2331228972 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f88400a2d00 msgr2=0x7f88400a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:37.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.480+0000 7f8860c00640 1 --2- 172.21.15.28:0/2331228972 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f88400a2d00 0x7f88400a51c0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f8844009230 tx=0x7f88440058d0 comp rx=0 tx=0).stop 2024-05-12T22:19:37.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.480+0000 7f8860c00640 1 -- 172.21.15.28:0/2331228972 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f885c107cd0 msgr2=0x7f885c19e520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:37.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.480+0000 7f8860c00640 1 --2- 172.21.15.28:0/2331228972 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f885c107cd0 0x7f885c19e520 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f884c00ea60 tx=0x7f884c00ef20 comp rx=0 tx=0).stop 2024-05-12T22:19:37.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.480+0000 7f8860c00640 1 -- 172.21.15.28:0/2331228972 shutdown_connections 2024-05-12T22:19:37.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.480+0000 7f8860c00640 1 --2- 172.21.15.28:0/2331228972 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f88400a2d00 0x7f88400a51c0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:37.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.480+0000 7f8860c00640 1 --2- 172.21.15.28:0/2331228972 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f885c10dd40 0x7f885c1a2de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:37.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.480+0000 7f8860c00640 1 --2- 172.21.15.28:0/2331228972 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f885c108670 0x7f885c19ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:37.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.480+0000 7f8860c00640 1 --2- 172.21.15.28:0/2331228972 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f885c107cd0 0x7f885c19e520 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:37.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.480+0000 7f8860c00640 1 -- 172.21.15.28:0/2331228972 >> 172.21.15.28:0/2331228972 conn(0x7f885c0773b0 msgr2=0x7f885c078b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:19:37.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.480+0000 7f8860c00640 1 -- 172.21.15.28:0/2331228972 shutdown_connections 2024-05-12T22:19:37.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.480+0000 7f8860c00640 1 -- 172.21.15.28:0/2331228972 wait complete. 2024-05-12T22:19:37.500 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-05-12T22:19:37.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.717+0000 7f3d22400640 1 -- 172.21.15.28:0/1212308336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d1c1038e0 msgr2=0x7f3d1c103d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:37.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.717+0000 7f3d22400640 1 --2- 172.21.15.28:0/1212308336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d1c1038e0 0x7f3d1c103d60 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f3d0c009930 tx=0x7f3d0c02a440 comp rx=0 tx=0).stop 2024-05-12T22:19:37.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.717+0000 7f3d22400640 1 -- 172.21.15.28:0/1212308336 shutdown_connections 2024-05-12T22:19:37.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.717+0000 7f3d22400640 1 --2- 172.21.15.28:0/1212308336 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d1c1042a0 0x7f3d1c1107b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:37.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.717+0000 7f3d22400640 1 --2- 172.21.15.28:0/1212308336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d1c1038e0 0x7f3d1c103d60 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:37.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.717+0000 7f3d22400640 1 --2- 172.21.15.28:0/1212308336 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3d1c1098f0 0x7f3d1c109cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:37.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.717+0000 7f3d22400640 1 -- 172.21.15.28:0/1212308336 >> 172.21.15.28:0/1212308336 conn(0x7f3d1c0ff370 msgr2=0x7f3d1c101790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:19:37.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.717+0000 7f3d22400640 1 -- 172.21.15.28:0/1212308336 shutdown_connections 2024-05-12T22:19:37.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.717+0000 7f3d22400640 1 -- 172.21.15.28:0/1212308336 wait complete. 2024-05-12T22:19:37.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.719+0000 7f3d22400640 1 Processor -- start 2024-05-12T22:19:37.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.719+0000 7f3d22400640 1 -- start start 2024-05-12T22:19:37.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.719+0000 7f3d22400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d1c1038e0 0x7f3d1c19a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:37.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.719+0000 7f3d22400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d1c1042a0 0x7f3d1c19a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:37.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.719+0000 7f3d22400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3d1c1098f0 0x7f3d1c19ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:37.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.719+0000 7f3d22400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3d1c113d50 con 0x7f3d1c1042a0 2024-05-12T22:19:37.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.719+0000 7f3d22400640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f3d1c113bd0 con 0x7f3d1c1038e0 2024-05-12T22:19:37.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.719+0000 7f3d22400640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f3d1c113ed0 con 0x7f3d1c1098f0 2024-05-12T22:19:37.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.719+0000 7f3d1be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d1c1042a0 0x7f3d1c19a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:37.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.720+0000 7f3d1be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d1c1042a0 0x7f3d1c19a6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42326/0 (socket says 172.21.15.28:42326) 2024-05-12T22:19:37.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.720+0000 7f3d1be00640 1 -- 172.21.15.28:0/573273727 learned_addr learned my addr 172.21.15.28:0/573273727 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:19:37.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.720+0000 7f3d21a00640 1 --2- 172.21.15.28:0/573273727 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3d1c1098f0 0x7f3d1c19ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:37.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.720+0000 7f3d21000640 1 --2- 172.21.15.28:0/573273727 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d1c1038e0 0x7f3d1c19a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:37.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.720+0000 7f3d1be00640 1 -- 172.21.15.28:0/573273727 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3d1c1098f0 msgr2=0x7f3d1c19ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:37.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.720+0000 7f3d1be00640 1 --2- 172.21.15.28:0/573273727 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3d1c1098f0 0x7f3d1c19ea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:37.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.720+0000 7f3d1be00640 1 -- 172.21.15.28:0/573273727 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d1c1038e0 msgr2=0x7f3d1c19a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:37.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.720+0000 7f3d1be00640 1 --2- 172.21.15.28:0/573273727 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d1c1038e0 0x7f3d1c19a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:37.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.720+0000 7f3d1be00640 1 -- 172.21.15.28:0/573273727 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d1c19f1a0 con 0x7f3d1c1042a0 2024-05-12T22:19:37.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.720+0000 7f3d1be00640 1 --2- 172.21.15.28:0/573273727 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d1c1042a0 0x7f3d1c19a6f0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f3d0c02a960 tx=0x7f3d0c003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:19:37.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.721+0000 7f3d19600640 1 -- 172.21.15.28:0/573273727 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d0c03f070 con 0x7f3d1c1042a0 2024-05-12T22:19:37.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.721+0000 7f3d19600640 1 -- 172.21.15.28:0/573273727 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3d0c031c90 con 0x7f3d1c1042a0 2024-05-12T22:19:37.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.721+0000 7f3d19600640 1 -- 172.21.15.28:0/573273727 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d0c035030 con 0x7f3d1c1042a0 2024-05-12T22:19:37.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.721+0000 7f3d22400640 1 -- 172.21.15.28:0/573273727 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d1c19f430 con 0x7f3d1c1042a0 2024-05-12T22:19:37.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.721+0000 7f3d22400640 1 -- 172.21.15.28:0/573273727 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d1c1a6d20 con 0x7f3d1c1042a0 2024-05-12T22:19:37.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.724+0000 7f3d267f9740 1 Processor -- start 2024-05-12T22:19:37.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.725+0000 7f3d267f9740 1 -- start start 2024-05-12T22:19:37.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.725+0000 7f3d267f9740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cb72f7eb60 0x55cb72f7ef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:37.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.725+0000 7f3d267f9740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cb72f7f480 0x55cb72f89610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:37.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.725+0000 7f3d267f9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb72f89b50 0x55cb72f8bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:37.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.725+0000 7f3d267f9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55cb72ee3a30 con 0x55cb72f89b50 2024-05-12T22:19:37.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.725+0000 7f3d267f9740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x55cb72e8b700 con 0x55cb72f7f480 2024-05-12T22:19:37.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.725+0000 7f3d267f9740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55cb72edf9f0 con 0x55cb72f7eb60 2024-05-12T22:19:37.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.725+0000 7f3d19600640 1 -- 172.21.15.28:0/573273727 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7f3d0c042070 con 0x7f3d1c1042a0 2024-05-12T22:19:37.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.725+0000 7f3d21000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cb72f7f480 0x55cb72f89610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:37.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.725+0000 7f3d21000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cb72f7f480 0x55cb72f89610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:55592/0 (socket says 172.21.15.28:55592) 2024-05-12T22:19:37.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.725+0000 7f3d21000640 1 -- 172.21.15.28:0/1358843936 learned_addr learned my addr 172.21.15.28:0/1358843936 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:19:37.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.725+0000 7f3d21a00640 1 --2- 172.21.15.28:0/1358843936 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cb72f7eb60 0x55cb72f7ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:37.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.725+0000 7f3d1be00640 1 --2- 172.21.15.28:0/1358843936 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb72f89b50 0x55cb72f8bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:37.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.726+0000 7f3d21000640 1 -- 172.21.15.28:0/1358843936 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cb72f7eb60 msgr2=0x55cb72f7ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:19:37.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.726+0000 7f3d21000640 1 --2- 172.21.15.28:0/1358843936 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cb72f7eb60 0x55cb72f7ef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:37.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.726+0000 7f3d21000640 1 -- 172.21.15.28:0/1358843936 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb72f89b50 msgr2=0x55cb72f8bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:19:37.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.726+0000 7f3d21000640 1 --2- 172.21.15.28:0/1358843936 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb72f89b50 0x55cb72f8bf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:37.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.726+0000 7f3d21000640 1 -- 172.21.15.28:0/1358843936 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cb72bf73f0 con 0x55cb72f7f480 2024-05-12T22:19:37.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.726+0000 7f3d21000640 1 --2- 172.21.15.28:0/1358843936 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cb72f7f480 0x55cb72f89610 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f3d1000b710 tx=0x7f3d10004610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9009a1dc909efe87 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:19:37.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.726+0000 7f3d0a000640 1 -- 172.21.15.28:0/1358843936 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d10003bd0 con 0x55cb72f7f480 2024-05-12T22:19:37.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.726+0000 7f3d0a000640 1 -- 172.21.15.28:0/1358843936 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3d10015070 con 0x55cb72f7f480 2024-05-12T22:19:37.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.726+0000 7f3d0a000640 1 -- 172.21.15.28:0/1358843936 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d10004e40 con 0x55cb72f7f480 2024-05-12T22:19:37.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.727+0000 7f3d19600640 1 --2- 172.21.15.28:0/573273727 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f3d040a2d00 0x7f3d040a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:37.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.727+0000 7f3d267f9740 1 -- 172.21.15.28:0/1358843936 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cb72f7f480 msgr2=0x55cb72f89610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:37.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.727+0000 7f3d267f9740 1 --2- 172.21.15.28:0/1358843936 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cb72f7f480 0x55cb72f89610 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f3d1000b710 tx=0x7f3d10004610 comp rx=0 tx=0).stop 2024-05-12T22:19:37.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.727+0000 7f3d267f9740 1 -- 172.21.15.28:0/1358843936 shutdown_connections 2024-05-12T22:19:37.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.727+0000 7f3d267f9740 1 --2- 172.21.15.28:0/1358843936 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb72f89b50 0x55cb72f8bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:37.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.727+0000 7f3d267f9740 1 --2- 172.21.15.28:0/1358843936 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cb72f7f480 0x55cb72f89610 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:37.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.727+0000 7f3d267f9740 1 --2- 172.21.15.28:0/1358843936 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cb72f7eb60 0x55cb72f7ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:37.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.727+0000 7f3d267f9740 1 -- 172.21.15.28:0/1358843936 >> 172.21.15.28:0/1358843936 conn(0x55cb72f7ae80 msgr2=0x55cb72f7b260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:19:37.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.727+0000 7f3d19600640 1 -- 172.21.15.28:0/573273727 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8513+0+0 (secure 0 0 0) 0x7f3d0c0efba0 con 0x7f3d1c1042a0 2024-05-12T22:19:37.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.727+0000 7f3d21a00640 1 --2- 172.21.15.28:0/573273727 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f3d040a2d00 0x7f3d040a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:37.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.727+0000 7f3d267f9740 1 -- 172.21.15.28:0/1358843936 shutdown_connections 2024-05-12T22:19:37.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.727+0000 7f3d267f9740 1 -- 172.21.15.28:0/1358843936 wait complete. 2024-05-12T22:19:37.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.728+0000 7f3d267f9740 1 -- start start 2024-05-12T22:19:37.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.728+0000 7f3d21a00640 1 --2- 172.21.15.28:0/573273727 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f3d040a2d00 0x7f3d040a51c0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f3d14007f40 tx=0x7f3d14007520 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:19:37.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.728+0000 7f3d267f9740 1 Processor -- start 2024-05-12T22:19:37.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.729+0000 7f3d267f9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb72f7eb60 0x55cb72f86a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:37.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.729+0000 7f3d267f9740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cb72f7f480 0x55cb72f86f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:37.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.729+0000 7f3d267f9740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cb72f89b50 0x55cb72f874a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:37.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.729+0000 7f3d267f9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55cb72f8c940 con 0x55cb72f7eb60 2024-05-12T22:19:37.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.729+0000 7f3d267f9740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x55cb72e9ea70 con 0x55cb72f7f480 2024-05-12T22:19:37.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.729+0000 7f3d267f9740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55cb72f8cac0 con 0x55cb72f89b50 2024-05-12T22:19:37.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.729+0000 7f3d267f9740 10 client.0 _ll_register_callbacks cb 0x55cb72e81ba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:19:37.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.729+0000 7f3d1be00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cb72f7f480 0x55cb72f86f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:37.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.729+0000 7f3d1be00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cb72f7f480 0x55cb72f86f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:55594/0 (socket says 172.21.15.28:55594) 2024-05-12T22:19:37.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.729+0000 7f3d1be00640 1 -- 172.21.15.28:0/2735033483 learned_addr learned my addr 172.21.15.28:0/2735033483 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:19:37.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.729+0000 7f3d21000640 1 --2- 172.21.15.28:0/2735033483 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb72f7eb60 0x55cb72f86a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:37.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.729+0000 7f3d21a00640 1 --2- 172.21.15.28:0/2735033483 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cb72f89b50 0x55cb72f874a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:37.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.729+0000 7f3d1be00640 1 -- 172.21.15.28:0/2735033483 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cb72f89b50 msgr2=0x55cb72f874a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:19:37.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.729+0000 7f3d1be00640 1 --2- 172.21.15.28:0/2735033483 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cb72f89b50 0x55cb72f874a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:37.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.729+0000 7f3d1be00640 1 -- 172.21.15.28:0/2735033483 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb72f7eb60 msgr2=0x55cb72f86a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:19:37.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.729+0000 7f3d1be00640 1 --2- 172.21.15.28:0/2735033483 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb72f7eb60 0x55cb72f86a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:37.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.729+0000 7f3d1be00640 1 -- 172.21.15.28:0/2735033483 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cb72cc17d0 con 0x55cb72f7f480 2024-05-12T22:19:37.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.730+0000 7f3d1be00640 1 --2- 172.21.15.28:0/2735033483 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cb72f7f480 0x55cb72f86f60 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f3d0c009900 tx=0x7f3d0c0b8470 comp rx=0 tx=0).ready entity=mon.2 client_cookie=294013c236a2a9b0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:19:37.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.730+0000 7f3cfea00640 10 client.0 ms_handle_connect on v2:172.21.15.86:3300/0 2024-05-12T22:19:37.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.730+0000 7f3cfea00640 1 -- 172.21.15.28:0/2735033483 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d0c049070 con 0x55cb72f7f480 2024-05-12T22:19:37.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.730+0000 7f3d267f9740 1 -- 172.21.15.28:0/2735033483 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3d10015070 con 0x55cb72f7f480 2024-05-12T22:19:37.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.730+0000 7f3cfea00640 1 -- 172.21.15.28:0/2735033483 <== mon.2 v2:172.21.15.86:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8513+0+0 (secure 0 0 0) 0x7f3d0c03b5f0 con 0x55cb72f7f480 2024-05-12T22:19:37.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.730+0000 7f3cfea00640 1 -- 172.21.15.28:0/2735033483 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f3d0c031e40 con 0x55cb72f7f480 2024-05-12T22:19:37.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.730+0000 7f3d267f9740 10 client.27521 fetch_fsmap learned FSMap version 26 2024-05-12T22:19:37.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.730+0000 7f3d267f9740 1 -- 172.21.15.28:0/2735033483 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55cb72f98b40 con 0x55cb72f7f480 2024-05-12T22:19:37.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.730+0000 7f3cfea00640 1 -- 172.21.15.28:0/2735033483 <== mon.2 v2:172.21.15.86:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3d10015070 con 0x55cb72f7f480 2024-05-12T22:19:37.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.731+0000 7f3cfea00640 1 -- 172.21.15.28:0/2735033483 <== mon.2 v2:172.21.15.86:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f3d0c0c4840 con 0x55cb72f7f480 2024-05-12T22:19:37.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.731+0000 7f3d267f9740 10 client.27521 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:19:37.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.731+0000 7f3d267f9740 10 client.27521 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.g 2024-05-12T22:19:37.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.731+0000 7f3d267f9740 20 client.27521 populate_metadata read hostname 'smithi028' 2024-05-12T22:19:37.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.731+0000 7f3d267f9740 1 --2- 172.21.15.28:0/2735033483 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x55cb72f99f50 0x55cb72f9c410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:37.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.731+0000 7f3d267f9740 4 client.27521 mds_command: new command op to 24532 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-12T22:19:37.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.731+0000 7f3d267f9740 1 -- 172.21.15.28:0/2735033483 --> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cb72ef1160 con 0x55cb72f99f50 2024-05-12T22:19:37.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.731+0000 7f3d21000640 1 --2- 172.21.15.28:0/2735033483 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x55cb72f99f50 0x55cb72f9c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:37.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.732+0000 7f3d21000640 1 --2- 172.21.15.28:0/2735033483 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x55cb72f99f50 0x55cb72f9c410 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=6629090b077eef37 server_cookie=abee8bbdc4791ad3 in_seq=0 out_seq=0 2024-05-12T22:19:37.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.732+0000 7f3cfea00640 10 client.27521 ms_handle_connect on v2:172.21.15.28:6838/123892158 2024-05-12T22:19:37.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.732+0000 7f3cfea00640 1 -- 172.21.15.28:0/2735033483 <== mds.2 v2:172.21.15.28:6838/123892158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55cb72ef1160 con 0x55cb72f99f50 2024-05-12T22:19:37.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.732+0000 7f3cfea00640 10 client.27521 handle_command_reply: tid=0 2024-05-12T22:19:37.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.733+0000 7f3d267f9740 1 client.27521 shutdown 2024-05-12T22:19:37.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.733+0000 7f3d267f9740 1 -- 172.21.15.28:0/2735033483 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cb72f7f480 msgr2=0x55cb72f86f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:37.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.733+0000 7f3d267f9740 1 --2- 172.21.15.28:0/2735033483 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cb72f7f480 0x55cb72f86f60 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f3d0c009900 tx=0x7f3d0c0b8470 comp rx=0 tx=0).stop 2024-05-12T22:19:37.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.733+0000 7f3d267f9740 1 -- 172.21.15.28:0/2735033483 shutdown_connections 2024-05-12T22:19:37.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.733+0000 7f3d267f9740 1 --2- 172.21.15.28:0/2735033483 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cb72f89b50 0x55cb72f874a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:37.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.733+0000 7f3d267f9740 1 --2- 172.21.15.28:0/2735033483 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x55cb72f99f50 0x55cb72f9c410 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:37.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.733+0000 7f3d267f9740 1 --2- 172.21.15.28:0/2735033483 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cb72f7f480 0x55cb72f86f60 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:37.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.733+0000 7f3d267f9740 1 --2- 172.21.15.28:0/2735033483 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb72f7eb60 0x55cb72f86a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:37.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.733+0000 7f3cfea00640 0 client.27521 ms_handle_reset on v2:172.21.15.28:6838/123892158 2024-05-12T22:19:37.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.733+0000 7f3d267f9740 1 -- 172.21.15.28:0/2735033483 >> 172.21.15.28:0/2735033483 conn(0x55cb72f7ae80 msgr2=0x55cb72f80c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:19:37.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.733+0000 7f3d267f9740 1 -- 172.21.15.28:0/2735033483 shutdown_connections 2024-05-12T22:19:37.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.733+0000 7f3d267f9740 1 -- 172.21.15.28:0/2735033483 wait complete. 2024-05-12T22:19:37.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.733+0000 7f3d267f9740 20 client.27521 trim_cache size 0 max 16384 2024-05-12T22:19:37.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.748+0000 7f3d267f9740 1 Processor -- start 2024-05-12T22:19:37.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.748+0000 7f3d267f9740 1 -- start start 2024-05-12T22:19:37.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.748+0000 7f3d267f9740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cb72f7eb60 0x55cb72f95830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:37.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.748+0000 7f3d267f9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb72f7f480 0x55cb72f870f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:37.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.748+0000 7f3d267f9740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cb72f89b50 0x55cb72fa1ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:37.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.749+0000 7f3d267f9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3d1c1b1d00 con 0x55cb72f7f480 2024-05-12T22:19:37.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.749+0000 7f3d267f9740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f3d1c1af5d0 con 0x55cb72f7eb60 2024-05-12T22:19:37.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.749+0000 7f3d267f9740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f3d1c1ad440 con 0x55cb72f89b50 2024-05-12T22:19:37.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.749+0000 7f3d1be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb72f7f480 0x55cb72f870f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:37.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.749+0000 7f3d1be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb72f7f480 0x55cb72f870f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42366/0 (socket says 172.21.15.28:42366) 2024-05-12T22:19:37.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.749+0000 7f3d1be00640 1 -- 172.21.15.28:0/3441958881 learned_addr learned my addr 172.21.15.28:0/3441958881 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:19:37.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.749+0000 7f3d21000640 1 --2- 172.21.15.28:0/3441958881 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cb72f7eb60 0x55cb72f95830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:37.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.749+0000 7f3d21a00640 1 --2- 172.21.15.28:0/3441958881 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cb72f89b50 0x55cb72fa1ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:37.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.749+0000 7f3d21000640 1 -- 172.21.15.28:0/3441958881 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cb72f89b50 msgr2=0x55cb72fa1ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:19:37.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.749+0000 7f3d21000640 1 --2- 172.21.15.28:0/3441958881 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cb72f89b50 0x55cb72fa1ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:37.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.749+0000 7f3d21000640 1 -- 172.21.15.28:0/3441958881 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb72f7f480 msgr2=0x55cb72f870f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:19:37.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.749+0000 7f3d21000640 1 --2- 172.21.15.28:0/3441958881 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb72f7f480 0x55cb72f870f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:37.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.749+0000 7f3d21000640 1 -- 172.21.15.28:0/3441958881 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cb72f8c820 con 0x55cb72f7eb60 2024-05-12T22:19:37.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.749+0000 7f3d1be00640 1 --2- 172.21.15.28:0/3441958881 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb72f7f480 0x55cb72f870f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:19:37.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.750+0000 7f3d21000640 1 --2- 172.21.15.28:0/3441958881 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cb72f7eb60 0x55cb72f95830 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f3d10004a20 tx=0x7f3d1000b390 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d11efd764eac4ec0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:19:37.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.750+0000 7f3d09600640 1 -- 172.21.15.28:0/3441958881 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d10030070 con 0x55cb72f7eb60 2024-05-12T22:19:37.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.750+0000 7f3d09600640 1 -- 172.21.15.28:0/3441958881 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3d10015040 con 0x55cb72f7eb60 2024-05-12T22:19:37.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.750+0000 7f3d09600640 1 -- 172.21.15.28:0/3441958881 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d1002a7f0 con 0x55cb72f7eb60 2024-05-12T22:19:37.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.750+0000 7f3d267f9740 1 -- 172.21.15.28:0/3441958881 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cb72f7eb60 msgr2=0x55cb72f95830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:37.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.750+0000 7f3d267f9740 1 --2- 172.21.15.28:0/3441958881 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cb72f7eb60 0x55cb72f95830 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f3d10004a20 tx=0x7f3d1000b390 comp rx=0 tx=0).stop 2024-05-12T22:19:37.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.751+0000 7f3d267f9740 1 -- 172.21.15.28:0/3441958881 shutdown_connections 2024-05-12T22:19:37.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.751+0000 7f3d267f9740 1 --2- 172.21.15.28:0/3441958881 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cb72f89b50 0x55cb72fa1ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:37.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.751+0000 7f3d267f9740 1 --2- 172.21.15.28:0/3441958881 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb72f7f480 0x55cb72f870f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:37.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.751+0000 7f3d267f9740 1 --2- 172.21.15.28:0/3441958881 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cb72f7eb60 0x55cb72f95830 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:37.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.751+0000 7f3d267f9740 1 -- 172.21.15.28:0/3441958881 >> 172.21.15.28:0/3441958881 conn(0x55cb72f7ae80 msgr2=0x55cb72f869e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:19:37.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.751+0000 7f3d267f9740 1 -- 172.21.15.28:0/3441958881 shutdown_connections 2024-05-12T22:19:37.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.751+0000 7f3d267f9740 1 -- 172.21.15.28:0/3441958881 wait complete. 2024-05-12T22:19:37.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.751+0000 7f3d267f9740 1 -- start start 2024-05-12T22:19:37.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.751+0000 7f3d267f9740 1 Processor -- start 2024-05-12T22:19:37.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.752+0000 7f3d267f9740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cb72f7eb60 0x55cb72f92090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:37.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.752+0000 7f3d267f9740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cb72f7f480 0x55cb72f8d400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:37.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.752+0000 7f3d267f9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb72f89b50 0x55cb72f8d940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:37.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.752+0000 7f3d267f9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3d1c1ada90 con 0x55cb72f89b50 2024-05-12T22:19:37.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.752+0000 7f3d267f9740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f3d1c1aef00 con 0x55cb72f7f480 2024-05-12T22:19:37.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.752+0000 7f3d267f9740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f3d1c1ae160 con 0x55cb72f7eb60 2024-05-12T22:19:37.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.752+0000 7f3d267f9740 10 client.0 _ll_register_callbacks cb 0x7f3d0c03b460 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:19:37.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.752+0000 7f3d21000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cb72f7eb60 0x55cb72f92090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:37.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.752+0000 7f3d21000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cb72f7eb60 0x55cb72f92090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:57576/0 (socket says 172.21.15.28:57576) 2024-05-12T22:19:37.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.752+0000 7f3d21000640 1 -- 172.21.15.28:0/1511845262 learned_addr learned my addr 172.21.15.28:0/1511845262 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:19:37.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.752+0000 7f3d21a00640 1 --2- 172.21.15.28:0/1511845262 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb72f89b50 0x55cb72f8d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:37.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.752+0000 7f3d1be00640 1 --2- 172.21.15.28:0/1511845262 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cb72f7f480 0x55cb72f8d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:37.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.752+0000 7f3d21000640 1 -- 172.21.15.28:0/1511845262 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cb72f7f480 msgr2=0x55cb72f8d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:19:37.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.753+0000 7f3d21000640 1 --2- 172.21.15.28:0/1511845262 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cb72f7f480 0x55cb72f8d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:37.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.753+0000 7f3d21000640 1 -- 172.21.15.28:0/1511845262 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb72f89b50 msgr2=0x55cb72f8d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:19:37.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.753+0000 7f3d21000640 1 --2- 172.21.15.28:0/1511845262 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb72f89b50 0x55cb72f8d940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:37.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.753+0000 7f3d21000640 1 -- 172.21.15.28:0/1511845262 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cb72f8e140 con 0x55cb72f7eb60 2024-05-12T22:19:37.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.753+0000 7f3d1be00640 1 --2- 172.21.15.28:0/1511845262 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cb72f7f480 0x55cb72f8d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:19:37.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.753+0000 7f3d21000640 1 --2- 172.21.15.28:0/1511845262 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cb72f7eb60 0x55cb72f92090 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f3d100039c0 tx=0x7f3d100046d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a4c02dcd77001c16 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:19:37.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.753+0000 7f3cfea00640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-12T22:19:37.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.753+0000 7f3cfea00640 1 -- 172.21.15.28:0/1511845262 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d10030040 con 0x55cb72f7eb60 2024-05-12T22:19:37.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.753+0000 7f3d267f9740 1 -- 172.21.15.28:0/1511845262 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3d10015040 con 0x55cb72f7eb60 2024-05-12T22:19:37.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.753+0000 7f3cfea00640 1 -- 172.21.15.28:0/1511845262 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8513+0+0 (secure 0 0 0) 0x7f3d1002a3f0 con 0x55cb72f7eb60 2024-05-12T22:19:37.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.754+0000 7f3cfea00640 1 -- 172.21.15.28:0/1511845262 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f3d10041050 con 0x55cb72f7eb60 2024-05-12T22:19:37.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.754+0000 7f3d267f9740 10 client.27484 fetch_fsmap learned FSMap version 26 2024-05-12T22:19:37.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.754+0000 7f3d267f9740 1 -- 172.21.15.28:0/1511845262 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55cb730004e0 con 0x55cb72f7eb60 2024-05-12T22:19:37.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.754+0000 7f3cfea00640 1 -- 172.21.15.28:0/1511845262 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3d10015040 con 0x55cb72f7eb60 2024-05-12T22:19:37.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.754+0000 7f3cfea00640 1 -- 172.21.15.28:0/1511845262 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f3d1003c030 con 0x55cb72f7eb60 2024-05-12T22:19:37.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.754+0000 7f3d267f9740 10 client.27484 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:19:37.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.754+0000 7f3d267f9740 10 client.27484 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.g 2024-05-12T22:19:37.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.754+0000 7f3d267f9740 20 client.27484 populate_metadata read hostname 'smithi028' 2024-05-12T22:19:37.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.754+0000 7f3d267f9740 1 --2- 172.21.15.28:0/1511845262 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x55cb73000a90 0x55cb73002eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:37.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.754+0000 7f3d267f9740 4 client.27484 mds_command: new command op to 24532 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-12T22:19:37.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.754+0000 7f3d267f9740 1 -- 172.21.15.28:0/1511845262 --> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55cb72eeb4b0 con 0x55cb73000a90 2024-05-12T22:19:37.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.755+0000 7f3d1be00640 1 --2- 172.21.15.28:0/1511845262 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x55cb73000a90 0x55cb73002eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:37.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.755+0000 7f3cfea00640 10 client.27484 ms_handle_connect on v2:172.21.15.28:6838/123892158 2024-05-12T22:19:37.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.755+0000 7f3d1be00640 1 --2- 172.21.15.28:0/1511845262 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x55cb73000a90 0x55cb73002eb0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=9f5b53844c2560ce server_cookie=fc569907c15b2949 in_seq=0 out_seq=0 2024-05-12T22:19:37.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.755+0000 7f3cfea00640 1 -- 172.21.15.28:0/1511845262 <== mds.2 v2:172.21.15.28:6838/123892158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55cb72eeb4b0 con 0x55cb73000a90 2024-05-12T22:19:37.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.755+0000 7f3cfea00640 10 client.27484 handle_command_reply: tid=0 2024-05-12T22:19:37.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.755+0000 7f3d267f9740 1 client.27484 shutdown 2024-05-12T22:19:37.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.756+0000 7f3d267f9740 1 -- 172.21.15.28:0/1511845262 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cb72f7eb60 msgr2=0x55cb72f92090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:37.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.756+0000 7f3d267f9740 1 --2- 172.21.15.28:0/1511845262 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cb72f7eb60 0x55cb72f92090 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f3d100039c0 tx=0x7f3d100046d0 comp rx=0 tx=0).stop 2024-05-12T22:19:37.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.756+0000 7f3d267f9740 1 -- 172.21.15.28:0/1511845262 shutdown_connections 2024-05-12T22:19:37.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.756+0000 7f3d267f9740 1 --2- 172.21.15.28:0/1511845262 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb72f89b50 0x55cb72f8d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:37.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.756+0000 7f3d267f9740 1 --2- 172.21.15.28:0/1511845262 >> [v2:172.21.15.28:6838/123892158,v1:172.21.15.28:6839/123892158] conn(0x55cb73000a90 0x55cb73002eb0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:37.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.756+0000 7f3d267f9740 1 --2- 172.21.15.28:0/1511845262 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cb72f7f480 0x55cb72f8d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:37.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.756+0000 7f3d267f9740 1 --2- 172.21.15.28:0/1511845262 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cb72f7eb60 0x55cb72f92090 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:37.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.756+0000 7f3d267f9740 1 -- 172.21.15.28:0/1511845262 >> 172.21.15.28:0/1511845262 conn(0x55cb72f7ae80 msgr2=0x55cb72f87270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:19:37.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.756+0000 7f3cfea00640 0 client.27484 ms_handle_reset on v2:172.21.15.28:6838/123892158 2024-05-12T22:19:37.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.756+0000 7f3d267f9740 1 -- 172.21.15.28:0/1511845262 shutdown_connections 2024-05-12T22:19:37.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.756+0000 7f3d267f9740 1 -- 172.21.15.28:0/1511845262 wait complete. 2024-05-12T22:19:37.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.756+0000 7f3d267f9740 20 client.27484 trim_cache size 0 max 16384 2024-05-12T22:19:37.764 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-05-12T22:19:37.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.757+0000 7f3d22400640 1 -- 172.21.15.28:0/573273727 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f3d040a2d00 msgr2=0x7f3d040a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:37.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.757+0000 7f3d22400640 1 --2- 172.21.15.28:0/573273727 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f3d040a2d00 0x7f3d040a51c0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f3d14007f40 tx=0x7f3d14007520 comp rx=0 tx=0).stop 2024-05-12T22:19:37.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.757+0000 7f3d22400640 1 -- 172.21.15.28:0/573273727 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d1c1042a0 msgr2=0x7f3d1c19a6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:37.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.757+0000 7f3d22400640 1 --2- 172.21.15.28:0/573273727 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d1c1042a0 0x7f3d1c19a6f0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f3d0c02a960 tx=0x7f3d0c003b90 comp rx=0 tx=0).stop 2024-05-12T22:19:37.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.757+0000 7f3d22400640 1 -- 172.21.15.28:0/573273727 shutdown_connections 2024-05-12T22:19:37.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.757+0000 7f3d22400640 1 --2- 172.21.15.28:0/573273727 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f3d040a2d00 0x7f3d040a51c0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:37.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.757+0000 7f3d22400640 1 --2- 172.21.15.28:0/573273727 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3d1c1098f0 0x7f3d1c19ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:37.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.757+0000 7f3d22400640 1 --2- 172.21.15.28:0/573273727 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d1c1042a0 0x7f3d1c19a6f0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:37.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.757+0000 7f3d22400640 1 --2- 172.21.15.28:0/573273727 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d1c1038e0 0x7f3d1c19a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:37.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.757+0000 7f3d22400640 1 -- 172.21.15.28:0/573273727 >> 172.21.15.28:0/573273727 conn(0x7f3d1c0ff370 msgr2=0x7f3d1c0ff8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:19:37.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.757+0000 7f3d22400640 1 -- 172.21.15.28:0/573273727 shutdown_connections 2024-05-12T22:19:37.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.757+0000 7f3d22400640 1 -- 172.21.15.28:0/573273727 wait complete. 2024-05-12T22:19:37.776 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-05-12T22:19:37.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.991+0000 7f80b3e00640 1 -- 172.21.15.28:0/568104368 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80b4102890 msgr2=0x7f80b4102c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:37.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.991+0000 7f80b3e00640 1 --2- 172.21.15.28:0/568104368 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80b4102890 0x7f80b4102c70 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f80a4009900 tx=0x7f80a402a480 comp rx=0 tx=0).stop 2024-05-12T22:19:37.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.992+0000 7f80b3e00640 1 -- 172.21.15.28:0/568104368 shutdown_connections 2024-05-12T22:19:37.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.992+0000 7f80b3e00640 1 --2- 172.21.15.28:0/568104368 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f80b410bcb0 0x7f80b410e150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:37.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.992+0000 7f80b3e00640 1 --2- 172.21.15.28:0/568104368 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f80b41031b0 0x7f80b410b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:37.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.992+0000 7f80b3e00640 1 --2- 172.21.15.28:0/568104368 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80b4102890 0x7f80b4102c70 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:37.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.992+0000 7f80b3e00640 1 -- 172.21.15.28:0/568104368 >> 172.21.15.28:0/568104368 conn(0x7f80b40fc0b0 msgr2=0x7f80b40fe4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:19:37.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.992+0000 7f80b3e00640 1 -- 172.21.15.28:0/568104368 shutdown_connections 2024-05-12T22:19:37.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.992+0000 7f80b3e00640 1 -- 172.21.15.28:0/568104368 wait complete. 2024-05-12T22:19:37.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.993+0000 7f80b3e00640 1 Processor -- start 2024-05-12T22:19:37.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.993+0000 7f80b3e00640 1 -- start start 2024-05-12T22:19:37.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.993+0000 7f80b3e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f80b4102890 0x7f80b419a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:37.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.993+0000 7f80b3e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f80b41031b0 0x7f80b419a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:37.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.993+0000 7f80b3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80b410bcb0 0x7f80b419ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:37.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.993+0000 7f80b3e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f80b4111b40 con 0x7f80b410bcb0 2024-05-12T22:19:37.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.993+0000 7f80b3e00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f80b41119c0 con 0x7f80b41031b0 2024-05-12T22:19:37.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.993+0000 7f80b3e00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f80b4111cc0 con 0x7f80b4102890 2024-05-12T22:19:37.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.994+0000 7f80b2a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f80b4102890 0x7f80b419a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:37.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.994+0000 7f80b2a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f80b4102890 0x7f80b419a210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:57592/0 (socket says 172.21.15.28:57592) 2024-05-12T22:19:37.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.994+0000 7f80b2a00640 1 -- 172.21.15.28:0/2673658197 learned_addr learned my addr 172.21.15.28:0/2673658197 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:19:37.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.994+0000 7f80b3400640 1 --2- 172.21.15.28:0/2673658197 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80b410bcb0 0x7f80b419ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:37.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.994+0000 7f80b2000640 1 --2- 172.21.15.28:0/2673658197 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f80b41031b0 0x7f80b419a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:37.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.995+0000 7f80b2a00640 1 -- 172.21.15.28:0/2673658197 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f80b41031b0 msgr2=0x7f80b419a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:37.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.995+0000 7f80b2a00640 1 --2- 172.21.15.28:0/2673658197 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f80b41031b0 0x7f80b419a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:37.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.995+0000 7f80b2a00640 1 -- 172.21.15.28:0/2673658197 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80b410bcb0 msgr2=0x7f80b419ead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:37.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.995+0000 7f80b2a00640 1 --2- 172.21.15.28:0/2673658197 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80b410bcb0 0x7f80b419ead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:37.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.995+0000 7f80b2a00640 1 -- 172.21.15.28:0/2673658197 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80b419f200 con 0x7f80b4102890 2024-05-12T22:19:37.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.995+0000 7f80b2000640 1 --2- 172.21.15.28:0/2673658197 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f80b41031b0 0x7f80b419a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:19:37.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.996+0000 7f80b2a00640 1 --2- 172.21.15.28:0/2673658197 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f80b4102890 0x7f80b419a210 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f80a4004770 tx=0x7f80a4003c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:19:37.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.996+0000 7f80b8a00640 1 -- 172.21.15.28:0/2673658197 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80a403f070 con 0x7f80b4102890 2024-05-12T22:19:37.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.996+0000 7f80b3e00640 1 -- 172.21.15.28:0/2673658197 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80b419f490 con 0x7f80b4102890 2024-05-12T22:19:37.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.996+0000 7f80b8a00640 1 -- 172.21.15.28:0/2673658197 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f80a4031ca0 con 0x7f80b4102890 2024-05-12T22:19:37.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.996+0000 7f80b8a00640 1 -- 172.21.15.28:0/2673658197 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80a4035030 con 0x7f80b4102890 2024-05-12T22:19:37.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:37.997+0000 7f80b3e00640 1 -- 172.21.15.28:0/2673658197 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80b41a6d80 con 0x7f80b4102890 2024-05-12T22:19:37.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.000+0000 7f80bcbfd740 1 Processor -- start 2024-05-12T22:19:38.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.000+0000 7f80bcbfd740 1 -- start start 2024-05-12T22:19:38.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.001+0000 7f80bcbfd740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5570ca0a9b60 0x5570ca0a9f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:38.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.001+0000 7f80bcbfd740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5570ca0aa480 0x5570ca0b4610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:38.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.001+0000 7f80bcbfd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5570ca0b4b50 0x5570ca0b6f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:38.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.001+0000 7f80bcbfd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5570ca00ea30 con 0x5570ca0b4b50 2024-05-12T22:19:38.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.001+0000 7f80bcbfd740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x5570c9fb6700 con 0x5570ca0a9b60 2024-05-12T22:19:38.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.001+0000 7f80bcbfd740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5570ca00a9f0 con 0x5570ca0aa480 2024-05-12T22:19:38.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.001+0000 7f80b8a00640 1 -- 172.21.15.28:0/2673658197 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7f80a4042070 con 0x7f80b4102890 2024-05-12T22:19:38.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.001+0000 7f80b3400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5570ca0a9b60 0x5570ca0a9f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:38.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.001+0000 7f80b3400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5570ca0a9b60 0x5570ca0a9f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:55628/0 (socket says 172.21.15.28:55628) 2024-05-12T22:19:38.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.001+0000 7f80b3400640 1 -- 172.21.15.28:0/1254387475 learned_addr learned my addr 172.21.15.28:0/1254387475 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:19:38.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.001+0000 7f80b2a00640 1 --2- 172.21.15.28:0/1254387475 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5570ca0aa480 0x5570ca0b4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:38.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.001+0000 7f80b2000640 1 --2- 172.21.15.28:0/1254387475 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5570ca0b4b50 0x5570ca0b6f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:38.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.002+0000 7f80b3400640 1 -- 172.21.15.28:0/1254387475 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5570ca0aa480 msgr2=0x5570ca0b4610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:19:38.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.002+0000 7f80b3400640 1 --2- 172.21.15.28:0/1254387475 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5570ca0aa480 0x5570ca0b4610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:38.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.002+0000 7f80b3400640 1 -- 172.21.15.28:0/1254387475 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5570ca0b4b50 msgr2=0x5570ca0b6f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:19:38.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.002+0000 7f80b3400640 1 --2- 172.21.15.28:0/1254387475 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5570ca0b4b50 0x5570ca0b6f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:38.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.002+0000 7f80b3400640 1 -- 172.21.15.28:0/1254387475 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5570c9d223f0 con 0x5570ca0a9b60 2024-05-12T22:19:38.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.002+0000 7f80b2000640 1 --2- 172.21.15.28:0/1254387475 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5570ca0b4b50 0x5570ca0b6f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-12T22:19:38.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.002+0000 7f80b3400640 1 --2- 172.21.15.28:0/1254387475 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5570ca0a9b60 0x5570ca0a9f40 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f80ac00a430 tx=0x7f80ac009650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b1ac263bf35684c8 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:19:38.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.002+0000 7f809fe00640 1 -- 172.21.15.28:0/1254387475 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80ac013790 con 0x5570ca0a9b60 2024-05-12T22:19:38.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.003+0000 7f809fe00640 1 -- 172.21.15.28:0/1254387475 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f80ac019070 con 0x5570ca0a9b60 2024-05-12T22:19:38.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.003+0000 7f809fe00640 1 -- 172.21.15.28:0/1254387475 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80ac014e10 con 0x5570ca0a9b60 2024-05-12T22:19:38.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.003+0000 7f80b8a00640 1 --2- 172.21.15.28:0/2673658197 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f80980a2d80 0x7f80980a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:38.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.003+0000 7f80b8a00640 1 -- 172.21.15.28:0/2673658197 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8513+0+0 (secure 0 0 0) 0x7f80a40efbe0 con 0x7f80b4102890 2024-05-12T22:19:38.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.003+0000 7f80b2a00640 1 --2- 172.21.15.28:0/2673658197 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f80980a2d80 0x7f80980a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:38.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.004+0000 7f80bcbfd740 1 -- 172.21.15.28:0/1254387475 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5570ca0a9b60 msgr2=0x5570ca0a9f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:38.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.004+0000 7f80bcbfd740 1 --2- 172.21.15.28:0/1254387475 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5570ca0a9b60 0x5570ca0a9f40 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f80ac00a430 tx=0x7f80ac009650 comp rx=0 tx=0).stop 2024-05-12T22:19:38.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.004+0000 7f80bcbfd740 1 -- 172.21.15.28:0/1254387475 shutdown_connections 2024-05-12T22:19:38.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.004+0000 7f80bcbfd740 1 --2- 172.21.15.28:0/1254387475 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5570ca0b4b50 0x5570ca0b6f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:38.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.004+0000 7f80bcbfd740 1 --2- 172.21.15.28:0/1254387475 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5570ca0aa480 0x5570ca0b4610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:38.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.004+0000 7f80bcbfd740 1 --2- 172.21.15.28:0/1254387475 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5570ca0a9b60 0x5570ca0a9f40 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:38.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.004+0000 7f80bcbfd740 1 -- 172.21.15.28:0/1254387475 >> 172.21.15.28:0/1254387475 conn(0x5570ca0a5e80 msgr2=0x5570ca0a6260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:19:38.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.004+0000 7f80bcbfd740 1 -- 172.21.15.28:0/1254387475 shutdown_connections 2024-05-12T22:19:38.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.004+0000 7f80bcbfd740 1 -- 172.21.15.28:0/1254387475 wait complete. 2024-05-12T22:19:38.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.004+0000 7f80b2a00640 1 --2- 172.21.15.28:0/2673658197 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f80980a2d80 0x7f80980a5240 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x5570ca0ab120 tx=0x7f80a4009260 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:19:38.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.004+0000 7f80bcbfd740 1 -- start start 2024-05-12T22:19:38.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.005+0000 7f80bcbfd740 1 Processor -- start 2024-05-12T22:19:38.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.005+0000 7f80bcbfd740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5570ca0a9b60 0x5570ca0b1af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:38.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.005+0000 7f80bcbfd740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5570ca0aa480 0x5570ca0b2030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:38.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.005+0000 7f80bcbfd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5570ca0b4b50 0x5570ca0b2570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:38.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.005+0000 7f80bcbfd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5570ca0b7940 con 0x5570ca0b4b50 2024-05-12T22:19:38.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.005+0000 7f80bcbfd740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x5570c9fc9a70 con 0x5570ca0aa480 2024-05-12T22:19:38.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.005+0000 7f80bcbfd740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5570ca0b7ac0 con 0x5570ca0a9b60 2024-05-12T22:19:38.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.005+0000 7f80bcbfd740 10 client.0 _ll_register_callbacks cb 0x5570c9facba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:19:38.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.006+0000 7f80b2000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5570ca0aa480 0x5570ca0b2030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:38.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.006+0000 7f80b2000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5570ca0aa480 0x5570ca0b2030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:55640/0 (socket says 172.21.15.28:55640) 2024-05-12T22:19:38.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.006+0000 7f80b2000640 1 -- 172.21.15.28:0/3950804858 learned_addr learned my addr 172.21.15.28:0/3950804858 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:19:38.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.006+0000 7f80b3400640 1 --2- 172.21.15.28:0/3950804858 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5570ca0a9b60 0x5570ca0b1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:38.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.006+0000 7f80b3400640 1 --2- 172.21.15.28:0/3950804858 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5570ca0b4b50 0x5570ca0b2570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:38.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.006+0000 7f80b2000640 1 -- 172.21.15.28:0/3950804858 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5570ca0a9b60 msgr2=0x5570ca0b1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:19:38.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.006+0000 7f80b2000640 1 --2- 172.21.15.28:0/3950804858 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5570ca0a9b60 0x5570ca0b1af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:38.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.006+0000 7f80b2000640 1 -- 172.21.15.28:0/3950804858 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5570ca0b4b50 msgr2=0x5570ca0b2570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:19:38.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.006+0000 7f80b2000640 1 --2- 172.21.15.28:0/3950804858 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5570ca0b4b50 0x5570ca0b2570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:38.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.006+0000 7f80b2000640 1 -- 172.21.15.28:0/3950804858 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5570c9dec7d0 con 0x5570ca0aa480 2024-05-12T22:19:38.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.006+0000 7f80b3400640 1 --2- 172.21.15.28:0/3950804858 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5570ca0b4b50 0x5570ca0b2570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:19:38.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.006+0000 7f80b2000640 1 --2- 172.21.15.28:0/3950804858 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5570ca0aa480 0x5570ca0b2030 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f80a0012780 tx=0x7f80a0012c40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4baafda6df3b1b7d server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:19:38.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.006+0000 7f809cc00640 10 client.0 ms_handle_connect on v2:172.21.15.86:3300/0 2024-05-12T22:19:38.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.007+0000 7f809cc00640 1 -- 172.21.15.28:0/3950804858 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80a001f060 con 0x5570ca0aa480 2024-05-12T22:19:38.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.007+0000 7f809cc00640 1 -- 172.21.15.28:0/3950804858 <== mon.2 v2:172.21.15.86:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8513+0+0 (secure 0 0 0) 0x7f80a0010dc0 con 0x5570ca0aa480 2024-05-12T22:19:38.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.007+0000 7f80bcbfd740 1 -- 172.21.15.28:0/3950804858 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f80ac019070 con 0x5570ca0aa480 2024-05-12T22:19:38.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.007+0000 7f809cc00640 1 -- 172.21.15.28:0/3950804858 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f80a001a430 con 0x5570ca0aa480 2024-05-12T22:19:38.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.007+0000 7f80bcbfd740 10 client.27539 fetch_fsmap learned FSMap version 26 2024-05-12T22:19:38.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.007+0000 7f80bcbfd740 1 -- 172.21.15.28:0/3950804858 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5570ca0c3bd0 con 0x5570ca0aa480 2024-05-12T22:19:38.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.007+0000 7f809cc00640 1 -- 172.21.15.28:0/3950804858 <== mon.2 v2:172.21.15.86:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f80ac019070 con 0x5570ca0aa480 2024-05-12T22:19:38.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.008+0000 7f809cc00640 1 -- 172.21.15.28:0/3950804858 <== mon.2 v2:172.21.15.86:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f80a000e540 con 0x5570ca0aa480 2024-05-12T22:19:38.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.008+0000 7f80bcbfd740 10 client.27539 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:19:38.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.008+0000 7f80bcbfd740 10 client.27539 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-05-12T22:19:38.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.008+0000 7f80bcbfd740 20 client.27539 populate_metadata read hostname 'smithi028' 2024-05-12T22:19:38.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.008+0000 7f80bcbfd740 1 --2- 172.21.15.28:0/3950804858 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x5570ca0c5130 0x5570ca0c7550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:38.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.008+0000 7f80bcbfd740 4 client.27539 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-12T22:19:38.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.008+0000 7f80bcbfd740 1 -- 172.21.15.28:0/3950804858 --> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5570ca01c160 con 0x5570ca0c5130 2024-05-12T22:19:38.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.008+0000 7f80b3400640 1 --2- 172.21.15.28:0/3950804858 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x5570ca0c5130 0x5570ca0c7550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:38.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.009+0000 7f809cc00640 10 client.27539 ms_handle_connect on v2:172.21.15.28:6840/761183870 2024-05-12T22:19:38.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.009+0000 7f80b3400640 1 --2- 172.21.15.28:0/3950804858 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x5570ca0c5130 0x5570ca0c7550 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=629f7cf91f0f3862 server_cookie=1f6c9d95aa09b13f in_seq=0 out_seq=0 2024-05-12T22:19:38.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.009+0000 7f809cc00640 1 -- 172.21.15.28:0/3950804858 <== mds.1 v2:172.21.15.28:6840/761183870 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5570ca01c160 con 0x5570ca0c5130 2024-05-12T22:19:38.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.009+0000 7f809cc00640 10 client.27539 handle_command_reply: tid=0 2024-05-12T22:19:38.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.010+0000 7f80bcbfd740 1 client.27539 shutdown 2024-05-12T22:19:38.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.010+0000 7f80bcbfd740 1 -- 172.21.15.28:0/3950804858 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5570ca0aa480 msgr2=0x5570ca0b2030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:38.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.010+0000 7f80bcbfd740 1 --2- 172.21.15.28:0/3950804858 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5570ca0aa480 0x5570ca0b2030 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f80a0012780 tx=0x7f80a0012c40 comp rx=0 tx=0).stop 2024-05-12T22:19:38.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.010+0000 7f80bcbfd740 1 -- 172.21.15.28:0/3950804858 shutdown_connections 2024-05-12T22:19:38.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.010+0000 7f80bcbfd740 1 --2- 172.21.15.28:0/3950804858 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x5570ca0c5130 0x5570ca0c7550 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:38.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.010+0000 7f80bcbfd740 1 --2- 172.21.15.28:0/3950804858 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5570ca0b4b50 0x5570ca0b2570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:38.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.010+0000 7f80bcbfd740 1 --2- 172.21.15.28:0/3950804858 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5570ca0aa480 0x5570ca0b2030 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:38.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.010+0000 7f80bcbfd740 1 --2- 172.21.15.28:0/3950804858 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5570ca0a9b60 0x5570ca0b1af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:38.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.010+0000 7f80bcbfd740 1 -- 172.21.15.28:0/3950804858 >> 172.21.15.28:0/3950804858 conn(0x5570ca0a5e80 msgr2=0x5570ca0abc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:19:38.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.010+0000 7f809cc00640 0 client.27539 ms_handle_reset on v2:172.21.15.28:6840/761183870 2024-05-12T22:19:38.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.010+0000 7f80bcbfd740 1 -- 172.21.15.28:0/3950804858 shutdown_connections 2024-05-12T22:19:38.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.010+0000 7f80bcbfd740 1 -- 172.21.15.28:0/3950804858 wait complete. 2024-05-12T22:19:38.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.010+0000 7f80bcbfd740 20 client.27539 trim_cache size 0 max 16384 2024-05-12T22:19:38.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.026+0000 7f80bcbfd740 1 Processor -- start 2024-05-12T22:19:38.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.026+0000 7f80bcbfd740 1 -- start start 2024-05-12T22:19:38.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.027+0000 7f80bcbfd740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5570ca0a9b60 0x5570ca0c0800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:38.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.027+0000 7f80bcbfd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5570ca0aa480 0x5570ca0b2390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:38.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.027+0000 7f80bcbfd740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5570ca0b4b50 0x5570ca0a4e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:38.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.027+0000 7f80bcbfd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f80b41b1e30 con 0x5570ca0aa480 2024-05-12T22:19:38.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.027+0000 7f80bcbfd740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f80b41af700 con 0x5570ca0a9b60 2024-05-12T22:19:38.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.027+0000 7f80bcbfd740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f80b41ad570 con 0x5570ca0b4b50 2024-05-12T22:19:38.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.027+0000 7f80b3400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5570ca0b4b50 0x5570ca0a4e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:38.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.027+0000 7f80b3400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5570ca0b4b50 0x5570ca0a4e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:57626/0 (socket says 172.21.15.28:57626) 2024-05-12T22:19:38.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.027+0000 7f80b3400640 1 -- 172.21.15.28:0/768864992 learned_addr learned my addr 172.21.15.28:0/768864992 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:19:38.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.027+0000 7f80b3400640 1 --2- 172.21.15.28:0/768864992 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5570ca0a9b60 0x5570ca0c0800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:38.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.027+0000 7f80b2000640 1 --2- 172.21.15.28:0/768864992 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5570ca0aa480 0x5570ca0b2390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:38.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.028+0000 7f80b3400640 1 -- 172.21.15.28:0/768864992 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5570ca0a9b60 msgr2=0x5570ca0c0800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:19:38.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.028+0000 7f80b3400640 1 --2- 172.21.15.28:0/768864992 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5570ca0a9b60 0x5570ca0c0800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:38.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.028+0000 7f80b3400640 1 -- 172.21.15.28:0/768864992 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5570ca0aa480 msgr2=0x5570ca0b2390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:19:38.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.028+0000 7f80b3400640 1 --2- 172.21.15.28:0/768864992 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5570ca0aa480 0x5570ca0b2390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:38.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.028+0000 7f80b3400640 1 -- 172.21.15.28:0/768864992 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5570ca0b0040 con 0x5570ca0b4b50 2024-05-12T22:19:38.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.028+0000 7f80b3400640 1 --2- 172.21.15.28:0/768864992 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5570ca0b4b50 0x5570ca0a4e80 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f80ac0149c0 tx=0x7f80ac00af90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ea82ef754a65d981 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:19:38.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.029+0000 7f809f400640 1 -- 172.21.15.28:0/768864992 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80ac029040 con 0x5570ca0b4b50 2024-05-12T22:19:38.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.029+0000 7f809f400640 1 -- 172.21.15.28:0/768864992 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f80ac019070 con 0x5570ca0b4b50 2024-05-12T22:19:38.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.029+0000 7f809f400640 1 -- 172.21.15.28:0/768864992 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80ac0137d0 con 0x5570ca0b4b50 2024-05-12T22:19:38.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.029+0000 7f80bcbfd740 1 -- 172.21.15.28:0/768864992 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5570ca0b4b50 msgr2=0x5570ca0a4e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:38.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.029+0000 7f80bcbfd740 1 --2- 172.21.15.28:0/768864992 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5570ca0b4b50 0x5570ca0a4e80 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f80ac0149c0 tx=0x7f80ac00af90 comp rx=0 tx=0).stop 2024-05-12T22:19:38.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.030+0000 7f80bcbfd740 1 -- 172.21.15.28:0/768864992 shutdown_connections 2024-05-12T22:19:38.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.030+0000 7f80bcbfd740 1 --2- 172.21.15.28:0/768864992 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5570ca0b4b50 0x5570ca0a4e80 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:38.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.030+0000 7f80bcbfd740 1 --2- 172.21.15.28:0/768864992 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5570ca0aa480 0x5570ca0b2390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:38.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.030+0000 7f80bcbfd740 1 --2- 172.21.15.28:0/768864992 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5570ca0a9b60 0x5570ca0c0800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:38.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.030+0000 7f80bcbfd740 1 -- 172.21.15.28:0/768864992 >> 172.21.15.28:0/768864992 conn(0x5570ca0a5e80 msgr2=0x5570ca0a4770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:19:38.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.030+0000 7f80bcbfd740 1 -- 172.21.15.28:0/768864992 shutdown_connections 2024-05-12T22:19:38.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.030+0000 7f80bcbfd740 1 -- 172.21.15.28:0/768864992 wait complete. 2024-05-12T22:19:38.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.030+0000 7f80bcbfd740 1 -- start start 2024-05-12T22:19:38.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.030+0000 7f80bcbfd740 1 Processor -- start 2024-05-12T22:19:38.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.031+0000 7f80bcbfd740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5570ca0a9b60 0x5570ca0b8770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:38.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.031+0000 7f80bcbfd740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5570ca0aa480 0x5570ca0b8cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:38.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.031+0000 7f80bcbfd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5570ca0b4b50 0x5570ca0b91f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:38.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.031+0000 7f80bcbfd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f80b41adbc0 con 0x5570ca0b4b50 2024-05-12T22:19:38.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.031+0000 7f80bcbfd740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f80b41af030 con 0x5570ca0a9b60 2024-05-12T22:19:38.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.031+0000 7f80bcbfd740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f80b41ae290 con 0x5570ca0aa480 2024-05-12T22:19:38.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.031+0000 7f80bcbfd740 10 client.0 _ll_register_callbacks cb 0x7f80a0009d00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:19:38.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.031+0000 7f80b3400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5570ca0a9b60 0x5570ca0b8770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:38.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.031+0000 7f80b2000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5570ca0aa480 0x5570ca0b8cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:38.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.032+0000 7f80b2000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5570ca0aa480 0x5570ca0b8cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:57634/0 (socket says 172.21.15.28:57634) 2024-05-12T22:19:38.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.032+0000 7f80b3400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5570ca0a9b60 0x5570ca0b8770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:55656/0 (socket says 172.21.15.28:55656) 2024-05-12T22:19:38.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.032+0000 7f80b2000640 1 -- 172.21.15.28:0/390353390 learned_addr learned my addr 172.21.15.28:0/390353390 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:19:38.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.032+0000 7f80b3400640 1 --2- 172.21.15.28:0/390353390 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5570ca0b4b50 0x5570ca0b91f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:38.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.032+0000 7f80b2000640 1 -- 172.21.15.28:0/390353390 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5570ca0a9b60 msgr2=0x5570ca0b8770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:19:38.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.032+0000 7f80b2000640 1 --2- 172.21.15.28:0/390353390 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5570ca0a9b60 0x5570ca0b8770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:38.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.032+0000 7f80b2000640 1 -- 172.21.15.28:0/390353390 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5570ca0b4b50 msgr2=0x5570ca0b91f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:19:38.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.032+0000 7f80b2000640 1 --2- 172.21.15.28:0/390353390 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5570ca0b4b50 0x5570ca0b91f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:38.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.032+0000 7f80b2000640 1 -- 172.21.15.28:0/390353390 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5570ca12a6b0 con 0x5570ca0aa480 2024-05-12T22:19:38.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.032+0000 7f80b2000640 1 --2- 172.21.15.28:0/390353390 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5570ca0aa480 0x5570ca0b8cb0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f80a000e9e0 tx=0x7f80a0010e70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=381f8adcdd2df7c0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:19:38.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.032+0000 7f809cc00640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-05-12T22:19:38.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.033+0000 7f809cc00640 1 -- 172.21.15.28:0/390353390 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80a00107a0 con 0x5570ca0aa480 2024-05-12T22:19:38.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.033+0000 7f809cc00640 1 -- 172.21.15.28:0/390353390 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8513+0+0 (secure 0 0 0) 0x7f80a0005040 con 0x5570ca0aa480 2024-05-12T22:19:38.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.033+0000 7f80bcbfd740 1 -- 172.21.15.28:0/390353390 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f80ac019070 con 0x5570ca0aa480 2024-05-12T22:19:38.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.033+0000 7f809cc00640 1 -- 172.21.15.28:0/390353390 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f80a0005260 con 0x5570ca0aa480 2024-05-12T22:19:38.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.033+0000 7f80bcbfd740 10 client.27499 fetch_fsmap learned FSMap version 26 2024-05-12T22:19:38.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.033+0000 7f80bcbfd740 1 -- 172.21.15.28:0/390353390 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5570ca12b990 con 0x5570ca0aa480 2024-05-12T22:19:38.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.033+0000 7f809cc00640 1 -- 172.21.15.28:0/390353390 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f80ac019070 con 0x5570ca0aa480 2024-05-12T22:19:38.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.033+0000 7f809cc00640 1 -- 172.21.15.28:0/390353390 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f80a001a5d0 con 0x5570ca0aa480 2024-05-12T22:19:38.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.033+0000 7f80bcbfd740 10 client.27499 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:19:38.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.033+0000 7f80bcbfd740 10 client.27499 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-05-12T22:19:38.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.033+0000 7f80bcbfd740 20 client.27499 populate_metadata read hostname 'smithi028' 2024-05-12T22:19:38.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.033+0000 7f80bcbfd740 1 --2- 172.21.15.28:0/390353390 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x5570ca12c100 0x5570ca12e520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:38.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.034+0000 7f80bcbfd740 4 client.27499 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-12T22:19:38.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.034+0000 7f80bcbfd740 1 -- 172.21.15.28:0/390353390 --> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5570ca0164b0 con 0x5570ca12c100 2024-05-12T22:19:38.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.034+0000 7f80b3400640 1 --2- 172.21.15.28:0/390353390 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x5570ca12c100 0x5570ca12e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:38.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.034+0000 7f80b3400640 1 --2- 172.21.15.28:0/390353390 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x5570ca12c100 0x5570ca12e520 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=1d12ea011539dc4 server_cookie=4b46c07976f3236f in_seq=0 out_seq=0 2024-05-12T22:19:38.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.034+0000 7f809cc00640 10 client.27499 ms_handle_connect on v2:172.21.15.28:6840/761183870 2024-05-12T22:19:38.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.035+0000 7f809cc00640 1 -- 172.21.15.28:0/390353390 <== mds.1 v2:172.21.15.28:6840/761183870 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5570ca0164b0 con 0x5570ca12c100 2024-05-12T22:19:38.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.035+0000 7f809cc00640 10 client.27499 handle_command_reply: tid=0 2024-05-12T22:19:38.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.035+0000 7f80bcbfd740 1 client.27499 shutdown 2024-05-12T22:19:38.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.035+0000 7f80bcbfd740 1 -- 172.21.15.28:0/390353390 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5570ca0aa480 msgr2=0x5570ca0b8cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:38.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.035+0000 7f80bcbfd740 1 --2- 172.21.15.28:0/390353390 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5570ca0aa480 0x5570ca0b8cb0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f80a000e9e0 tx=0x7f80a0010e70 comp rx=0 tx=0).stop 2024-05-12T22:19:38.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.035+0000 7f80bcbfd740 1 -- 172.21.15.28:0/390353390 shutdown_connections 2024-05-12T22:19:38.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.035+0000 7f80bcbfd740 1 --2- 172.21.15.28:0/390353390 >> [v2:172.21.15.28:6840/761183870,v1:172.21.15.28:6841/761183870] conn(0x5570ca12c100 0x5570ca12e520 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:38.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.035+0000 7f80bcbfd740 1 --2- 172.21.15.28:0/390353390 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5570ca0b4b50 0x5570ca0b91f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:38.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.035+0000 7f80bcbfd740 1 --2- 172.21.15.28:0/390353390 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5570ca0aa480 0x5570ca0b8cb0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:38.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.035+0000 7f80bcbfd740 1 --2- 172.21.15.28:0/390353390 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5570ca0a9b60 0x5570ca0b8770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:38.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.035+0000 7f80bcbfd740 1 -- 172.21.15.28:0/390353390 >> 172.21.15.28:0/390353390 conn(0x5570ca0a5e80 msgr2=0x5570ca0a5000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:19:38.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.035+0000 7f809cc00640 0 client.27499 ms_handle_reset on v2:172.21.15.28:6840/761183870 2024-05-12T22:19:38.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.035+0000 7f80bcbfd740 1 -- 172.21.15.28:0/390353390 shutdown_connections 2024-05-12T22:19:38.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.035+0000 7f80bcbfd740 1 -- 172.21.15.28:0/390353390 wait complete. 2024-05-12T22:19:38.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.035+0000 7f80bcbfd740 20 client.27499 trim_cache size 0 max 16384 2024-05-12T22:19:38.044 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-05-12T22:19:38.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.036+0000 7f80b3e00640 1 -- 172.21.15.28:0/2673658197 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f80980a2d80 msgr2=0x7f80980a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:38.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.036+0000 7f80b3e00640 1 --2- 172.21.15.28:0/2673658197 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f80980a2d80 0x7f80980a5240 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x5570ca0ab120 tx=0x7f80a4009260 comp rx=0 tx=0).stop 2024-05-12T22:19:38.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.036+0000 7f80b3e00640 1 -- 172.21.15.28:0/2673658197 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f80b4102890 msgr2=0x7f80b419a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:38.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.036+0000 7f80b3e00640 1 --2- 172.21.15.28:0/2673658197 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f80b4102890 0x7f80b419a210 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f80a4004770 tx=0x7f80a4003c10 comp rx=0 tx=0).stop 2024-05-12T22:19:38.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.036+0000 7f80b3e00640 1 -- 172.21.15.28:0/2673658197 shutdown_connections 2024-05-12T22:19:38.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.036+0000 7f80b3e00640 1 --2- 172.21.15.28:0/2673658197 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f80980a2d80 0x7f80980a5240 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:38.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.036+0000 7f80b3e00640 1 --2- 172.21.15.28:0/2673658197 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80b410bcb0 0x7f80b419ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:38.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.036+0000 7f80b3e00640 1 --2- 172.21.15.28:0/2673658197 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f80b41031b0 0x7f80b419a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:38.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.036+0000 7f80b3e00640 1 --2- 172.21.15.28:0/2673658197 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f80b4102890 0x7f80b419a210 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:38.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.036+0000 7f80b3e00640 1 -- 172.21.15.28:0/2673658197 >> 172.21.15.28:0/2673658197 conn(0x7f80b40fc0b0 msgr2=0x7f80b4100410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:19:38.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.036+0000 7f80b3e00640 1 -- 172.21.15.28:0/2673658197 shutdown_connections 2024-05-12T22:19:38.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.036+0000 7f80b3e00640 1 -- 172.21.15.28:0/2673658197 wait complete. 2024-05-12T22:19:38.054 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-05-12T22:19:38.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.263+0000 7fa042e00640 1 -- 172.21.15.28:0/552850926 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa03c1043d0 msgr2=0x7fa03c1108e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:38.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.263+0000 7fa042e00640 1 --2- 172.21.15.28:0/552850926 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa03c1043d0 0x7fa03c1108e0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fa034009be0 tx=0x7fa03402a6c0 comp rx=0 tx=0).stop 2024-05-12T22:19:38.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.263+0000 7fa042e00640 1 -- 172.21.15.28:0/552850926 shutdown_connections 2024-05-12T22:19:38.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.263+0000 7fa042e00640 1 --2- 172.21.15.28:0/552850926 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa03c1043d0 0x7fa03c1108e0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:38.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.263+0000 7fa042e00640 1 --2- 172.21.15.28:0/552850926 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa03c103a10 0x7fa03c103e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:38.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.263+0000 7fa042e00640 1 --2- 172.21.15.28:0/552850926 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa03c109a20 0x7fa03c109e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:38.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.263+0000 7fa042e00640 1 -- 172.21.15.28:0/552850926 >> 172.21.15.28:0/552850926 conn(0x7fa03c0ff4a0 msgr2=0x7fa03c1018c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:19:38.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.263+0000 7fa042e00640 1 -- 172.21.15.28:0/552850926 shutdown_connections 2024-05-12T22:19:38.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.263+0000 7fa042e00640 1 -- 172.21.15.28:0/552850926 wait complete. 2024-05-12T22:19:38.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.264+0000 7fa042e00640 1 Processor -- start 2024-05-12T22:19:38.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.265+0000 7fa042e00640 1 -- start start 2024-05-12T22:19:38.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.265+0000 7fa042e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa03c103a10 0x7fa03c19a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:38.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.265+0000 7fa042e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa03c1043d0 0x7fa03c19a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:38.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.265+0000 7fa042e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa03c109a20 0x7fa03c19eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:38.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.265+0000 7fa042e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa03c113e00 con 0x7fa03c103a10 2024-05-12T22:19:38.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.265+0000 7fa042e00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fa03c113c80 con 0x7fa03c109a20 2024-05-12T22:19:38.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.265+0000 7fa042e00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa03c113f80 con 0x7fa03c1043d0 2024-05-12T22:19:38.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.265+0000 7fa03be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa03c103a10 0x7fa03c19a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:38.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.265+0000 7fa03be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa03c103a10 0x7fa03c19a320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42440/0 (socket says 172.21.15.28:42440) 2024-05-12T22:19:38.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.265+0000 7fa03be00640 1 -- 172.21.15.28:0/3242698711 learned_addr learned my addr 172.21.15.28:0/3242698711 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:19:38.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.265+0000 7fa040e00640 1 --2- 172.21.15.28:0/3242698711 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa03c109a20 0x7fa03c19eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:38.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.266+0000 7fa03b400640 1 --2- 172.21.15.28:0/3242698711 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa03c1043d0 0x7fa03c19a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:38.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.266+0000 7fa03be00640 1 -- 172.21.15.28:0/3242698711 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa03c1043d0 msgr2=0x7fa03c19a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:38.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.266+0000 7fa03be00640 1 --2- 172.21.15.28:0/3242698711 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa03c1043d0 0x7fa03c19a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:38.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.266+0000 7fa03be00640 1 -- 172.21.15.28:0/3242698711 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa03c109a20 msgr2=0x7fa03c19eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:38.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.266+0000 7fa03be00640 1 --2- 172.21.15.28:0/3242698711 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa03c109a20 0x7fa03c19eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:38.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.266+0000 7fa03be00640 1 -- 172.21.15.28:0/3242698711 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa03c19f2c0 con 0x7fa03c103a10 2024-05-12T22:19:38.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.266+0000 7fa040e00640 1 --2- 172.21.15.28:0/3242698711 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa03c109a20 0x7fa03c19eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:19:38.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.266+0000 7fa03be00640 1 --2- 172.21.15.28:0/3242698711 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa03c103a10 0x7fa03c19a320 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fa02c00e9c0 tx=0x7fa02c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:19:38.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.266+0000 7fa038c00640 1 -- 172.21.15.28:0/3242698711 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa02c00ccc0 con 0x7fa03c103a10 2024-05-12T22:19:38.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.267+0000 7fa042e00640 1 -- 172.21.15.28:0/3242698711 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa03c19f5b0 con 0x7fa03c103a10 2024-05-12T22:19:38.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.267+0000 7fa038c00640 1 -- 172.21.15.28:0/3242698711 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa02c005500 con 0x7fa03c103a10 2024-05-12T22:19:38.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.267+0000 7fa038c00640 1 -- 172.21.15.28:0/3242698711 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa02c010430 con 0x7fa03c103a10 2024-05-12T22:19:38.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.267+0000 7fa042e00640 1 -- 172.21.15.28:0/3242698711 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa03c1a6e40 con 0x7fa03c103a10 2024-05-12T22:19:38.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.271+0000 7fa045bac740 1 Processor -- start 2024-05-12T22:19:38.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.271+0000 7fa045bac740 1 -- start start 2024-05-12T22:19:38.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.272+0000 7fa038c00640 1 -- 172.21.15.28:0/3242698711 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7fa02c0046c0 con 0x7fa03c103a10 2024-05-12T22:19:38.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.272+0000 7fa045bac740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557db541f980 0x557db541fd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:38.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.272+0000 7fa045bac740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x557db54202a0 0x557db542a430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:38.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.272+0000 7fa045bac740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557db542a970 0x557db542cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:38.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.272+0000 7fa045bac740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x557db5384a30 con 0x557db542a970 2024-05-12T22:19:38.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.272+0000 7fa045bac740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x557db532c700 con 0x557db54202a0 2024-05-12T22:19:38.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.272+0000 7fa045bac740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x557db53809f0 con 0x557db541f980 2024-05-12T22:19:38.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.272+0000 7fa03b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557db542a970 0x557db542cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:38.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.272+0000 7fa03be00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x557db54202a0 0x557db542a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:38.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.272+0000 7fa03b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557db542a970 0x557db542cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42442/0 (socket says 172.21.15.28:42442) 2024-05-12T22:19:38.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.272+0000 7fa03b400640 1 -- 172.21.15.28:0/2483306420 learned_addr learned my addr 172.21.15.28:0/2483306420 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:19:38.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.272+0000 7fa040e00640 1 --2- 172.21.15.28:0/2483306420 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557db541f980 0x557db541fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:38.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.273+0000 7fa03be00640 1 -- 172.21.15.28:0/2483306420 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557db541f980 msgr2=0x557db541fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:19:38.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.273+0000 7fa03be00640 1 --2- 172.21.15.28:0/2483306420 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557db541f980 0x557db541fd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:38.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.273+0000 7fa03be00640 1 -- 172.21.15.28:0/2483306420 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557db542a970 msgr2=0x557db542cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:19:38.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.273+0000 7fa03be00640 1 --2- 172.21.15.28:0/2483306420 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557db542a970 0x557db542cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:38.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.273+0000 7fa03be00640 1 -- 172.21.15.28:0/2483306420 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557db50983f0 con 0x557db54202a0 2024-05-12T22:19:38.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.273+0000 7fa03b400640 1 --2- 172.21.15.28:0/2483306420 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557db542a970 0x557db542cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:19:38.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.273+0000 7fa040e00640 1 --2- 172.21.15.28:0/2483306420 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557db541f980 0x557db541fd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:19:38.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.273+0000 7fa03be00640 1 --2- 172.21.15.28:0/2483306420 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x557db54202a0 0x557db542a430 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fa02c0ccde0 tx=0x7fa02c0d5b40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1aa67ab364e9af60 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:19:38.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.273+0000 7fa031600640 1 -- 172.21.15.28:0/2483306420 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa02c0de070 con 0x557db54202a0 2024-05-12T22:19:38.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.274+0000 7fa031600640 1 -- 172.21.15.28:0/2483306420 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa02c0d7d50 con 0x557db54202a0 2024-05-12T22:19:38.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.274+0000 7fa031600640 1 -- 172.21.15.28:0/2483306420 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa02c0d1960 con 0x557db54202a0 2024-05-12T22:19:38.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.274+0000 7fa038c00640 1 --2- 172.21.15.28:0/3242698711 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fa0200a2d00 0x7fa0200a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:38.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.274+0000 7fa038c00640 1 -- 172.21.15.28:0/3242698711 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8513+0+0 (secure 0 0 0) 0x7fa02c0cf690 con 0x7fa03c103a10 2024-05-12T22:19:38.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.274+0000 7fa040e00640 1 --2- 172.21.15.28:0/3242698711 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fa0200a2d00 0x7fa0200a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:38.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.274+0000 7fa045bac740 1 -- 172.21.15.28:0/2483306420 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x557db54202a0 msgr2=0x557db542a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:38.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.274+0000 7fa045bac740 1 --2- 172.21.15.28:0/2483306420 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x557db54202a0 0x557db542a430 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fa02c0ccde0 tx=0x7fa02c0d5b40 comp rx=0 tx=0).stop 2024-05-12T22:19:38.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.274+0000 7fa045bac740 1 -- 172.21.15.28:0/2483306420 shutdown_connections 2024-05-12T22:19:38.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.274+0000 7fa045bac740 1 --2- 172.21.15.28:0/2483306420 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557db542a970 0x557db542cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:38.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.274+0000 7fa045bac740 1 --2- 172.21.15.28:0/2483306420 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x557db54202a0 0x557db542a430 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:38.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.274+0000 7fa045bac740 1 --2- 172.21.15.28:0/2483306420 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557db541f980 0x557db541fd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:38.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.274+0000 7fa045bac740 1 -- 172.21.15.28:0/2483306420 >> 172.21.15.28:0/2483306420 conn(0x557db541bca0 msgr2=0x557db541c080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:19:38.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.275+0000 7fa045bac740 1 -- 172.21.15.28:0/2483306420 shutdown_connections 2024-05-12T22:19:38.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.275+0000 7fa045bac740 1 -- 172.21.15.28:0/2483306420 wait complete. 2024-05-12T22:19:38.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.275+0000 7fa040e00640 1 --2- 172.21.15.28:0/3242698711 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fa0200a2d00 0x7fa0200a51c0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fa034004af0 tx=0x7fa034005e50 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:19:38.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.275+0000 7fa045bac740 1 -- start start 2024-05-12T22:19:38.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.275+0000 7fa045bac740 1 Processor -- start 2024-05-12T22:19:38.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.276+0000 7fa045bac740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557db541f980 0x557db54278d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:38.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.276+0000 7fa045bac740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x557db54202a0 0x557db5427e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:38.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.276+0000 7fa045bac740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557db542a970 0x557db5428350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:38.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.276+0000 7fa045bac740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x557db542d760 con 0x557db541f980 2024-05-12T22:19:38.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.276+0000 7fa045bac740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x557db533fa70 con 0x557db54202a0 2024-05-12T22:19:38.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.276+0000 7fa045bac740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x557db542d8e0 con 0x557db542a970 2024-05-12T22:19:38.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.276+0000 7fa045bac740 10 client.0 _ll_register_callbacks cb 0x557db5322ba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:19:38.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.276+0000 7fa03be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557db541f980 0x557db54278d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:38.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.276+0000 7fa03be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557db541f980 0x557db54278d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42456/0 (socket says 172.21.15.28:42456) 2024-05-12T22:19:38.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.276+0000 7fa03be00640 1 -- 172.21.15.28:0/2227029212 learned_addr learned my addr 172.21.15.28:0/2227029212 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:19:38.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.276+0000 7fa03b400640 1 --2- 172.21.15.28:0/2227029212 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x557db54202a0 0x557db5427e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:38.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.277+0000 7fa040e00640 1 --2- 172.21.15.28:0/2227029212 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557db542a970 0x557db5428350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:38.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.277+0000 7fa03be00640 1 -- 172.21.15.28:0/2227029212 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557db542a970 msgr2=0x557db5428350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:19:38.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.277+0000 7fa03be00640 1 --2- 172.21.15.28:0/2227029212 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557db542a970 0x557db5428350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:38.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.277+0000 7fa03be00640 1 -- 172.21.15.28:0/2227029212 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x557db54202a0 msgr2=0x557db5427e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:19:38.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.277+0000 7fa03be00640 1 --2- 172.21.15.28:0/2227029212 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x557db54202a0 0x557db5427e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:38.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.277+0000 7fa03be00640 1 -- 172.21.15.28:0/2227029212 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557db51627d0 con 0x557db541f980 2024-05-12T22:19:38.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.277+0000 7fa03be00640 1 --2- 172.21.15.28:0/2227029212 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557db541f980 0x557db54278d0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fa02c09bb40 tx=0x7fa02c0d1560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac9afccd342cb87f server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:19:38.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.277+0000 7fa026000640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-12T22:19:38.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.277+0000 7fa026000640 1 -- 172.21.15.28:0/2227029212 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa02c0de070 con 0x557db541f980 2024-05-12T22:19:38.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.277+0000 7fa026000640 1 -- 172.21.15.28:0/2227029212 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8513+0+0 (secure 0 0 0) 0x7fa02c0cd060 con 0x557db541f980 2024-05-12T22:19:38.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.277+0000 7fa045bac740 1 -- 172.21.15.28:0/2227029212 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa02c0d7d50 con 0x557db541f980 2024-05-12T22:19:38.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.278+0000 7fa026000640 1 -- 172.21.15.28:0/2227029212 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fa02c097e50 con 0x557db541f980 2024-05-12T22:19:38.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.278+0000 7fa045bac740 10 client.17424 fetch_fsmap learned FSMap version 26 2024-05-12T22:19:38.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.278+0000 7fa045bac740 1 -- 172.21.15.28:0/2227029212 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x557db54399f0 con 0x557db541f980 2024-05-12T22:19:38.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.278+0000 7fa026000640 1 -- 172.21.15.28:0/2227029212 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa02c0d7d50 con 0x557db541f980 2024-05-12T22:19:38.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.278+0000 7fa026000640 1 -- 172.21.15.28:0/2227029212 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fa02c098220 con 0x557db541f980 2024-05-12T22:19:38.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.278+0000 7fa045bac740 10 client.17424 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:19:38.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.278+0000 7fa045bac740 10 client.17424 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:19:38.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.278+0000 7fa045bac740 20 client.17424 populate_metadata read hostname 'smithi028' 2024-05-12T22:19:38.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.278+0000 7fa045bac740 1 --2- 172.21.15.28:0/2227029212 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x557db543ad40 0x557db543d130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:38.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.278+0000 7fa045bac740 4 client.17424 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-12T22:19:38.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.278+0000 7fa045bac740 1 -- 172.21.15.28:0/2227029212 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557db5392160 con 0x557db543ad40 2024-05-12T22:19:38.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.278+0000 7fa03b400640 1 --2- 172.21.15.28:0/2227029212 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x557db543ad40 0x557db543d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:38.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.279+0000 7fa03b400640 1 --2- 172.21.15.28:0/2227029212 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x557db543ad40 0x557db543d130 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2a11665b106ae30 server_cookie=5b08642a9bb05968 in_seq=0 out_seq=0 2024-05-12T22:19:38.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.279+0000 7fa026000640 10 client.17424 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:19:38.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.280+0000 7fa026000640 1 -- 172.21.15.28:0/2227029212 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557db5392160 con 0x557db543ad40 2024-05-12T22:19:38.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.280+0000 7fa026000640 10 client.17424 handle_command_reply: tid=0 2024-05-12T22:19:38.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.280+0000 7fa045bac740 1 client.17424 shutdown 2024-05-12T22:19:38.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.280+0000 7fa045bac740 1 -- 172.21.15.28:0/2227029212 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557db541f980 msgr2=0x557db54278d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:38.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.280+0000 7fa045bac740 1 --2- 172.21.15.28:0/2227029212 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557db541f980 0x557db54278d0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fa02c09bb40 tx=0x7fa02c0d1560 comp rx=0 tx=0).stop 2024-05-12T22:19:38.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.280+0000 7fa045bac740 1 -- 172.21.15.28:0/2227029212 shutdown_connections 2024-05-12T22:19:38.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.280+0000 7fa045bac740 1 --2- 172.21.15.28:0/2227029212 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x557db543ad40 0x557db543d130 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:38.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.280+0000 7fa045bac740 1 --2- 172.21.15.28:0/2227029212 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557db542a970 0x557db5428350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:38.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.280+0000 7fa045bac740 1 --2- 172.21.15.28:0/2227029212 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x557db54202a0 0x557db5427e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:38.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.280+0000 7fa045bac740 1 --2- 172.21.15.28:0/2227029212 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557db541f980 0x557db54278d0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:38.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.280+0000 7fa026000640 0 client.17424 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:19:38.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.280+0000 7fa045bac740 1 -- 172.21.15.28:0/2227029212 >> 172.21.15.28:0/2227029212 conn(0x557db541bca0 msgr2=0x557db5421aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:19:38.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.280+0000 7fa045bac740 1 -- 172.21.15.28:0/2227029212 shutdown_connections 2024-05-12T22:19:38.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.280+0000 7fa045bac740 1 -- 172.21.15.28:0/2227029212 wait complete. 2024-05-12T22:19:38.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.281+0000 7fa045bac740 20 client.17424 trim_cache size 0 max 16384 2024-05-12T22:19:38.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.296+0000 7fa045bac740 1 Processor -- start 2024-05-12T22:19:38.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.296+0000 7fa045bac740 1 -- start start 2024-05-12T22:19:38.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.296+0000 7fa045bac740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557db541f980 0x557db5422120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:38.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.296+0000 7fa045bac740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x557db54202a0 0x557db5440150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:38.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.296+0000 7fa045bac740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557db542a970 0x557db5440690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:38.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.297+0000 7fa045bac740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa03c1b20a0 con 0x557db541f980 2024-05-12T22:19:38.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.297+0000 7fa045bac740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fa03c1af970 con 0x557db54202a0 2024-05-12T22:19:38.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.297+0000 7fa045bac740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa03c1ad7e0 con 0x557db542a970 2024-05-12T22:19:38.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.297+0000 7fa03be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557db541f980 0x557db5422120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:38.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.297+0000 7fa03be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557db541f980 0x557db5422120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42460/0 (socket says 172.21.15.28:42460) 2024-05-12T22:19:38.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.297+0000 7fa03be00640 1 -- 172.21.15.28:0/1871145928 learned_addr learned my addr 172.21.15.28:0/1871145928 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:19:38.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.297+0000 7fa040e00640 1 --2- 172.21.15.28:0/1871145928 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557db542a970 0x557db5440690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:38.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.297+0000 7fa03b400640 1 --2- 172.21.15.28:0/1871145928 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x557db54202a0 0x557db5440150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:38.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.297+0000 7fa03be00640 1 -- 172.21.15.28:0/1871145928 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557db542a970 msgr2=0x557db5440690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:19:38.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.297+0000 7fa03be00640 1 --2- 172.21.15.28:0/1871145928 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557db542a970 0x557db5440690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:38.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.297+0000 7fa03be00640 1 -- 172.21.15.28:0/1871145928 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x557db54202a0 msgr2=0x557db5440150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:19:38.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.297+0000 7fa03be00640 1 --2- 172.21.15.28:0/1871145928 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x557db54202a0 0x557db5440150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:38.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.297+0000 7fa03be00640 1 -- 172.21.15.28:0/1871145928 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557db5425e60 con 0x557db541f980 2024-05-12T22:19:38.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.298+0000 7fa03be00640 1 --2- 172.21.15.28:0/1871145928 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557db541f980 0x557db5422120 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fa02c0cd3e0 tx=0x7fa02c09a0c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14ca9248ab69b96f server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:19:38.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.298+0000 7fa030c00640 1 -- 172.21.15.28:0/1871145928 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa02c0b8070 con 0x557db541f980 2024-05-12T22:19:38.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.298+0000 7fa030c00640 1 -- 172.21.15.28:0/1871145928 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa02c0d5320 con 0x557db541f980 2024-05-12T22:19:38.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.298+0000 7fa030c00640 1 -- 172.21.15.28:0/1871145928 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa02c0de070 con 0x557db541f980 2024-05-12T22:19:38.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.298+0000 7fa045bac740 1 -- 172.21.15.28:0/1871145928 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557db541f980 msgr2=0x557db5422120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:38.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.298+0000 7fa045bac740 1 --2- 172.21.15.28:0/1871145928 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557db541f980 0x557db5422120 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fa02c0cd3e0 tx=0x7fa02c09a0c0 comp rx=0 tx=0).stop 2024-05-12T22:19:38.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.299+0000 7fa045bac740 1 -- 172.21.15.28:0/1871145928 shutdown_connections 2024-05-12T22:19:38.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.299+0000 7fa045bac740 1 --2- 172.21.15.28:0/1871145928 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557db542a970 0x557db5440690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:38.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.299+0000 7fa045bac740 1 --2- 172.21.15.28:0/1871145928 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x557db54202a0 0x557db5440150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:38.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.299+0000 7fa045bac740 1 --2- 172.21.15.28:0/1871145928 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557db541f980 0x557db5422120 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:38.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.299+0000 7fa045bac740 1 -- 172.21.15.28:0/1871145928 >> 172.21.15.28:0/1871145928 conn(0x557db541bca0 msgr2=0x557db54363f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:19:38.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.299+0000 7fa045bac740 1 -- 172.21.15.28:0/1871145928 shutdown_connections 2024-05-12T22:19:38.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.299+0000 7fa045bac740 1 -- 172.21.15.28:0/1871145928 wait complete. 2024-05-12T22:19:38.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.299+0000 7fa045bac740 1 -- start start 2024-05-12T22:19:38.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.299+0000 7fa045bac740 1 Processor -- start 2024-05-12T22:19:38.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.300+0000 7fa045bac740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x557db541f980 0x557db543df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:38.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.300+0000 7fa045bac740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557db54202a0 0x557db543e480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:38.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.300+0000 7fa045bac740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557db542a970 0x557db549ff20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:38.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.300+0000 7fa045bac740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa03c1ade30 con 0x557db542a970 2024-05-12T22:19:38.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.300+0000 7fa045bac740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fa03c1af2a0 con 0x557db541f980 2024-05-12T22:19:38.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.300+0000 7fa045bac740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa03c1ae500 con 0x557db54202a0 2024-05-12T22:19:38.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.300+0000 7fa045bac740 10 client.0 _ll_register_callbacks cb 0x7fa02c0d5900 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-12T22:19:38.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.300+0000 7fa040e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557db542a970 0x557db549ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:38.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.300+0000 7fa040e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557db542a970 0x557db549ff20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42464/0 (socket says 172.21.15.28:42464) 2024-05-12T22:19:38.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.300+0000 7fa040e00640 1 -- 172.21.15.28:0/2137068536 learned_addr learned my addr 172.21.15.28:0/2137068536 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:19:38.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.300+0000 7fa03b400640 1 --2- 172.21.15.28:0/2137068536 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557db54202a0 0x557db543e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:38.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.300+0000 7fa03be00640 1 --2- 172.21.15.28:0/2137068536 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x557db541f980 0x557db543df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:38.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.301+0000 7fa040e00640 1 -- 172.21.15.28:0/2137068536 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557db54202a0 msgr2=0x557db543e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:19:38.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.301+0000 7fa040e00640 1 --2- 172.21.15.28:0/2137068536 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557db54202a0 0x557db543e480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:38.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.301+0000 7fa040e00640 1 -- 172.21.15.28:0/2137068536 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x557db541f980 msgr2=0x557db543df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-12T22:19:38.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.301+0000 7fa040e00640 1 --2- 172.21.15.28:0/2137068536 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x557db541f980 0x557db543df40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:38.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.301+0000 7fa040e00640 1 -- 172.21.15.28:0/2137068536 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557db54a0460 con 0x557db542a970 2024-05-12T22:19:38.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.301+0000 7fa03be00640 1 --2- 172.21.15.28:0/2137068536 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x557db541f980 0x557db543df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:19:38.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.301+0000 7fa040e00640 1 --2- 172.21.15.28:0/2137068536 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557db542a970 0x557db549ff20 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fa03402acb0 tx=0x7fa03402a6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2aabb3bab29b127a server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:19:38.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.301+0000 7fa026000640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-05-12T22:19:38.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.301+0000 7fa026000640 1 -- 172.21.15.28:0/2137068536 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa034036040 con 0x557db542a970 2024-05-12T22:19:38.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.301+0000 7fa026000640 1 -- 172.21.15.28:0/2137068536 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8513+0+0 (secure 0 0 0) 0x7fa034003dc0 con 0x557db542a970 2024-05-12T22:19:38.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.301+0000 7fa045bac740 1 -- 172.21.15.28:0/2137068536 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa02c0d5320 con 0x557db542a970 2024-05-12T22:19:38.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.302+0000 7fa026000640 1 -- 172.21.15.28:0/2137068536 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fa034004040 con 0x557db542a970 2024-05-12T22:19:38.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.302+0000 7fa045bac740 10 client.17430 fetch_fsmap learned FSMap version 26 2024-05-12T22:19:38.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.302+0000 7fa045bac740 1 -- 172.21.15.28:0/2137068536 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x557db54a15a0 con 0x557db542a970 2024-05-12T22:19:38.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.302+0000 7fa026000640 1 -- 172.21.15.28:0/2137068536 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa02c0d5320 con 0x557db542a970 2024-05-12T22:19:38.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.302+0000 7fa026000640 1 -- 172.21.15.28:0/2137068536 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fa03405db20 con 0x557db542a970 2024-05-12T22:19:38.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.302+0000 7fa045bac740 10 client.17430 fetch_fsmap finished waiting for FSMap version 26 2024-05-12T22:19:38.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.302+0000 7fa045bac740 10 client.17430 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-12T22:19:38.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.302+0000 7fa045bac740 20 client.17430 populate_metadata read hostname 'smithi028' 2024-05-12T22:19:38.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.302+0000 7fa045bac740 1 --2- 172.21.15.28:0/2137068536 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x557db54a1d20 0x557db54a4140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:38.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.302+0000 7fa045bac740 4 client.17430 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-12T22:19:38.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.302+0000 7fa045bac740 1 -- 172.21.15.28:0/2137068536 --> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557db538c4b0 con 0x557db54a1d20 2024-05-12T22:19:38.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.303+0000 7fa03be00640 1 --2- 172.21.15.28:0/2137068536 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x557db54a1d20 0x557db54a4140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:38.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.303+0000 7fa03be00640 1 --2- 172.21.15.28:0/2137068536 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x557db54a1d20 0x557db54a4140 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2d17e00f408f22e3 server_cookie=ce5f122d4dcbedd4 in_seq=0 out_seq=0 2024-05-12T22:19:38.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.303+0000 7fa026000640 10 client.17430 ms_handle_connect on v2:172.21.15.86:6838/1251999130 2024-05-12T22:19:38.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.303+0000 7fa026000640 1 -- 172.21.15.28:0/2137068536 <== mds.0 v2:172.21.15.86:6838/1251999130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557db538c4b0 con 0x557db54a1d20 2024-05-12T22:19:38.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.303+0000 7fa026000640 10 client.17430 handle_command_reply: tid=0 2024-05-12T22:19:38.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.303+0000 7fa045bac740 1 client.17430 shutdown 2024-05-12T22:19:38.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.303+0000 7fa045bac740 1 -- 172.21.15.28:0/2137068536 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557db542a970 msgr2=0x557db549ff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:38.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.303+0000 7fa045bac740 1 --2- 172.21.15.28:0/2137068536 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557db542a970 0x557db549ff20 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fa03402acb0 tx=0x7fa03402a6c0 comp rx=0 tx=0).stop 2024-05-12T22:19:38.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.303+0000 7fa045bac740 1 -- 172.21.15.28:0/2137068536 shutdown_connections 2024-05-12T22:19:38.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.303+0000 7fa045bac740 1 --2- 172.21.15.28:0/2137068536 >> [v2:172.21.15.86:6838/1251999130,v1:172.21.15.86:6839/1251999130] conn(0x557db54a1d20 0x557db54a4140 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:38.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.304+0000 7fa045bac740 1 --2- 172.21.15.28:0/2137068536 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557db542a970 0x557db549ff20 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:38.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.304+0000 7fa045bac740 1 --2- 172.21.15.28:0/2137068536 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557db54202a0 0x557db543e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:38.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.304+0000 7fa045bac740 1 --2- 172.21.15.28:0/2137068536 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x557db541f980 0x557db543df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:38.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.304+0000 7fa026000640 0 client.17430 ms_handle_reset on v2:172.21.15.86:6838/1251999130 2024-05-12T22:19:38.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.304+0000 7fa045bac740 1 -- 172.21.15.28:0/2137068536 >> 172.21.15.28:0/2137068536 conn(0x557db541bca0 msgr2=0x557db54276c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:19:38.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.304+0000 7fa045bac740 1 -- 172.21.15.28:0/2137068536 shutdown_connections 2024-05-12T22:19:38.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.304+0000 7fa045bac740 1 -- 172.21.15.28:0/2137068536 wait complete. 2024-05-12T22:19:38.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.304+0000 7fa045bac740 20 client.17430 trim_cache size 0 max 16384 2024-05-12T22:19:38.311 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-05-12T22:19:38.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.305+0000 7fa042e00640 1 -- 172.21.15.28:0/3242698711 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fa0200a2d00 msgr2=0x7fa0200a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:38.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.305+0000 7fa042e00640 1 --2- 172.21.15.28:0/3242698711 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fa0200a2d00 0x7fa0200a51c0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fa034004af0 tx=0x7fa034005e50 comp rx=0 tx=0).stop 2024-05-12T22:19:38.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.305+0000 7fa042e00640 1 -- 172.21.15.28:0/3242698711 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa03c103a10 msgr2=0x7fa03c19a320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:38.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.305+0000 7fa042e00640 1 --2- 172.21.15.28:0/3242698711 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa03c103a10 0x7fa03c19a320 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fa02c00e9c0 tx=0x7fa02c00ee80 comp rx=0 tx=0).stop 2024-05-12T22:19:38.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.305+0000 7fa042e00640 1 -- 172.21.15.28:0/3242698711 shutdown_connections 2024-05-12T22:19:38.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.305+0000 7fa042e00640 1 --2- 172.21.15.28:0/3242698711 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fa0200a2d00 0x7fa0200a51c0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:38.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.305+0000 7fa042e00640 1 --2- 172.21.15.28:0/3242698711 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa03c109a20 0x7fa03c19eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:38.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.305+0000 7fa042e00640 1 --2- 172.21.15.28:0/3242698711 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa03c1043d0 0x7fa03c19a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:38.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.305+0000 7fa042e00640 1 --2- 172.21.15.28:0/3242698711 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa03c103a10 0x7fa03c19a320 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:38.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.305+0000 7fa042e00640 1 -- 172.21.15.28:0/3242698711 >> 172.21.15.28:0/3242698711 conn(0x7fa03c0ff4a0 msgr2=0x7fa03c0ffac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:19:38.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.305+0000 7fa042e00640 1 -- 172.21.15.28:0/3242698711 shutdown_connections 2024-05-12T22:19:38.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:38.305+0000 7fa042e00640 1 -- 172.21.15.28:0/3242698711 wait complete. 2024-05-12T22:19:38.320 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:19:38 smithi028 ceph-mon[29847]: pgmap v330: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:19:38.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:19:38 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/2331228972' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:19:38.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:19:38 smithi086 ceph-mon[34297]: pgmap v330: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:19:38.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:19:38 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/2331228972' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:19:38.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:19:38 smithi186 ceph-mon[34785]: pgmap v330: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:19:38.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:19:38 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/2331228972' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:19:39.323 INFO:tasks.fwd_scrub.fs.[cephfs]:end scrubbing fs: cephfs 2024-05-12T22:19:39.323 INFO:tasks.fwd_scrub:done joining 2024-05-12T22:19:39.324 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-05-12T22:19:39.338 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-12T22:19:39.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:39.566+0000 7fead1800640 1 -- 172.21.15.28:0/116169506 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feacc107f80 msgr2=0x7feacc0689e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:39.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:39.566+0000 7fead1800640 1 --2- 172.21.15.28:0/116169506 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feacc107f80 0x7feacc0689e0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7feab8009900 tx=0x7feab802a480 comp rx=0 tx=0).stop 2024-05-12T22:19:39.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:39.567+0000 7fead1800640 1 -- 172.21.15.28:0/116169506 shutdown_connections 2024-05-12T22:19:39.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:39.567+0000 7fead1800640 1 --2- 172.21.15.28:0/116169506 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feacc069110 0x7feacc10e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:39.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:39.567+0000 7fead1800640 1 --2- 172.21.15.28:0/116169506 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feacc107f80 0x7feacc0689e0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:39.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:39.567+0000 7fead1800640 1 --2- 172.21.15.28:0/116169506 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feacc107660 0x7feacc107a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:39.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:39.567+0000 7fead1800640 1 -- 172.21.15.28:0/116169506 >> 172.21.15.28:0/116169506 conn(0x7feacc076070 msgr2=0x7feacc076480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:19:39.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:39.567+0000 7fead1800640 1 -- 172.21.15.28:0/116169506 shutdown_connections 2024-05-12T22:19:39.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:39.567+0000 7fead1800640 1 -- 172.21.15.28:0/116169506 wait complete. 2024-05-12T22:19:39.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:39.568+0000 7fead1800640 1 Processor -- start 2024-05-12T22:19:39.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:39.568+0000 7fead1800640 1 -- start start 2024-05-12T22:19:39.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:39.569+0000 7fead1800640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feacc069110 0x7feacc1008f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:39.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:39.569+0000 7fead1800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feacc107660 0x7feacc100e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:39.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:39.569+0000 7fead1800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feacc107f80 0x7feacc101370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:39.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:39.569+0000 7fead1800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7feacc111a40 con 0x7feacc107660 2024-05-12T22:19:39.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:39.569+0000 7fead1800640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7feacc1118c0 con 0x7feacc107f80 2024-05-12T22:19:39.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:39.569+0000 7fead1800640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7feacc111bc0 con 0x7feacc069110 2024-05-12T22:19:39.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:39.569+0000 7feacb400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feacc107660 0x7feacc100e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:39.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:39.569+0000 7feacb400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feacc107660 0x7feacc100e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42480/0 (socket says 172.21.15.28:42480) 2024-05-12T22:19:39.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:39.569+0000 7feacb400640 1 -- 172.21.15.28:0/3074967529 learned_addr learned my addr 172.21.15.28:0/3074967529 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:19:39.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:39.569+0000 7fead0e00640 1 --2- 172.21.15.28:0/3074967529 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feacc107f80 0x7feacc101370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:39.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:39.569+0000 7feacbe00640 1 --2- 172.21.15.28:0/3074967529 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feacc069110 0x7feacc1008f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:39.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:39.570+0000 7feacb400640 1 -- 172.21.15.28:0/3074967529 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feacc069110 msgr2=0x7feacc1008f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:39.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:39.570+0000 7feacb400640 1 --2- 172.21.15.28:0/3074967529 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feacc069110 0x7feacc1008f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:39.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:39.570+0000 7feacb400640 1 -- 172.21.15.28:0/3074967529 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feacc107f80 msgr2=0x7feacc101370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:39.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:39.570+0000 7feacb400640 1 --2- 172.21.15.28:0/3074967529 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feacc107f80 0x7feacc101370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:39.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:39.570+0000 7feacb400640 1 -- 172.21.15.28:0/3074967529 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feacc1a6440 con 0x7feacc107660 2024-05-12T22:19:39.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:39.570+0000 7feacbe00640 1 --2- 172.21.15.28:0/3074967529 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feacc069110 0x7feacc1008f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:19:39.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:39.570+0000 7fead0e00640 1 --2- 172.21.15.28:0/3074967529 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feacc107f80 0x7feacc101370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:19:39.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:39.570+0000 7feacb400640 1 --2- 172.21.15.28:0/3074967529 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feacc107660 0x7feacc100e30 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7feab802a9a0 tx=0x7feab8003b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:19:39.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:39.570+0000 7feac8c00640 1 -- 172.21.15.28:0/3074967529 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feab803f070 con 0x7feacc107660 2024-05-12T22:19:39.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:39.570+0000 7feac8c00640 1 -- 172.21.15.28:0/3074967529 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feab8031c20 con 0x7feacc107660 2024-05-12T22:19:39.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:39.570+0000 7feac8c00640 1 -- 172.21.15.28:0/3074967529 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feab8035030 con 0x7feacc107660 2024-05-12T22:19:39.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:39.570+0000 7fead1800640 1 -- 172.21.15.28:0/3074967529 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feacc1a66d0 con 0x7feacc107660 2024-05-12T22:19:39.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:39.571+0000 7fead1800640 1 -- 172.21.15.28:0/3074967529 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feacc1a6bc0 con 0x7feacc107660 2024-05-12T22:19:39.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:39.572+0000 7fead1800640 1 -- 172.21.15.28:0/3074967529 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feacc068570 con 0x7feacc107660 2024-05-12T22:19:39.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:39.574+0000 7feac8c00640 1 -- 172.21.15.28:0/3074967529 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7feab8042070 con 0x7feacc107660 2024-05-12T22:19:39.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:39.576+0000 7feac8c00640 1 --2- 172.21.15.28:0/3074967529 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7feab00a2d30 0x7feab00a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:39.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:39.576+0000 7feac8c00640 1 -- 172.21.15.28:0/3074967529 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8513+0+0 (secure 0 0 0) 0x7feab80efb40 con 0x7feacc107660 2024-05-12T22:19:39.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:39.577+0000 7feacbe00640 1 --2- 172.21.15.28:0/3074967529 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7feab00a2d30 0x7feab00a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:39.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:39.577+0000 7feacbe00640 1 --2- 172.21.15.28:0/3074967529 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7feab00a2d30 0x7feab00a51f0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7feabc009bb0 tx=0x7feabc005f90 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:19:39.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:39.581+0000 7feac8c00640 1 -- 172.21.15.28:0/3074967529 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7feab80bc700 con 0x7feacc107660 2024-05-12T22:19:39.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:39.790+0000 7fead1800640 1 -- 172.21.15.28:0/3074967529 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7feacc109b20 con 0x7feacc107660 2024-05-12T22:19:39.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:39.792+0000 7feac8c00640 1 -- 172.21.15.28:0/3074967529 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v97) ==== 74+0+23062 (secure 0 0 0) 0x7feab809ae20 con 0x7feacc107660 2024-05-12T22:19:39.792 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-12T22:19:39.792 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":97,"fsid":"6b92e7b4-10aa-11ef-bc98-c7b262605968","created":"2024-05-12T21:57:37.276782+0000","modified":"2024-05-12T22:19:13.971021+0000","last_up_change":"2024-05-12T22:05:37.929562+0000","last_in_change":"2024-05-12T22:05:18.750768+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-12T22:01:32.292826+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-12T22:06:31.540263+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-12T22:06:32.906118+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"96","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":15,"snap_epoch":96,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6992ce0a-844b-40e3-8ed1-181c8474dccf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":906003882},{"type":"v1","addr":"172.21.15.28:6803","nonce":906003882}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":906003882},{"type":"v1","addr":"172.21.15.28:6805","nonce":906003882}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":906003882},{"type":"v1","addr":"172.21.15.28:6809","nonce":906003882}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":906003882},{"type":"v1","addr":"172.21.15.28:6807","nonce":906003882}]},"public_addr":"172.21.15.28:6803/906003882","cluster_addr":"172.21.15.28:6805/906003882","heartbeat_back_addr":"172.21.15.28:6809/906003882","heartbeat_front_addr":"172.21.15.28:6807/906003882","state":["exists","up"]},{"osd":1,"uuid":"ced20c96-c481-4d48-90d1-6b43228959ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":3354141819},{"type":"v1","addr":"172.21.15.86:6801","nonce":3354141819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":3354141819},{"type":"v1","addr":"172.21.15.86:6803","nonce":3354141819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":3354141819},{"type":"v1","addr":"172.21.15.86:6807","nonce":3354141819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":3354141819},{"type":"v1","addr":"172.21.15.86:6805","nonce":3354141819}]},"public_addr":"172.21.15.86:6801/3354141819","cluster_addr":"172.21.15.86:6803/3354141819","heartbeat_back_addr":"172.21.15.86:6807/3354141819","heartbeat_front_addr":"172.21.15.86:6805/3354141819","state":["exists","up"]},{"osd":2,"uuid":"54a2f221-fa65-4795-9f5e-e660cc7c0098","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":906459365},{"type":"v1","addr":"172.21.15.186:6801","nonce":906459365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":906459365},{"type":"v1","addr":"172.21.15.186:6803","nonce":906459365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":906459365},{"type":"v1","addr":"172.21.15.186:6807","nonce":906459365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":906459365},{"type":"v1","addr":"172.21.15.186:6805","nonce":906459365}]},"public_addr":"172.21.15.186:6801/906459365","cluster_addr":"172.21.15.186:6803/906459365","heartbeat_back_addr":"172.21.15.186:6807/906459365","heartbeat_front_addr":"172.21.15.186:6805/906459365","state":["exists","up"]},{"osd":3,"uuid":"86836737-ac9a-487b-93f2-7fcd3c406960","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":4182884436},{"type":"v1","addr":"172.21.15.28:6811","nonce":4182884436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":4182884436},{"type":"v1","addr":"172.21.15.28:6813","nonce":4182884436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":4182884436},{"type":"v1","addr":"172.21.15.28:6817","nonce":4182884436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":4182884436},{"type":"v1","addr":"172.21.15.28:6815","nonce":4182884436}]},"public_addr":"172.21.15.28:6811/4182884436","cluster_addr":"172.21.15.28:6813/4182884436","heartbeat_back_addr":"172.21.15.28:6817/4182884436","heartbeat_front_addr":"172.21.15.28:6815/4182884436","state":["exists","up"]},{"osd":4,"uuid":"d162ebcd-83c0-4fd5-9925-28b2cecea935","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2596998708},{"type":"v1","addr":"172.21.15.86:6809","nonce":2596998708}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2596998708},{"type":"v1","addr":"172.21.15.86:6811","nonce":2596998708}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2596998708},{"type":"v1","addr":"172.21.15.86:6815","nonce":2596998708}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2596998708},{"type":"v1","addr":"172.21.15.86:6813","nonce":2596998708}]},"public_addr":"172.21.15.86:6809/2596998708","cluster_addr":"172.21.15.86:6811/2596998708","heartbeat_back_addr":"172.21.15.86:6815/2596998708","heartbeat_front_addr":"172.21.15.86:6813/2596998708","state":["exists","up"]},{"osd":5,"uuid":"c794336a-a105-425c-b6c7-a32b90c05bb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":1944205756},{"type":"v1","addr":"172.21.15.186:6809","nonce":1944205756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":1944205756},{"type":"v1","addr":"172.21.15.186:6811","nonce":1944205756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":1944205756},{"type":"v1","addr":"172.21.15.186:6815","nonce":1944205756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":1944205756},{"type":"v1","addr":"172.21.15.186:6813","nonce":1944205756}]},"public_addr":"172.21.15.186:6809/1944205756","cluster_addr":"172.21.15.186:6811/1944205756","heartbeat_back_addr":"172.21.15.186:6815/1944205756","heartbeat_front_addr":"172.21.15.186:6813/1944205756","state":["exists","up"]},{"osd":6,"uuid":"516932ac-af00-45b7-ad3c-d6dec7db442f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":1642308925},{"type":"v1","addr":"172.21.15.28:6819","nonce":1642308925}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":1642308925},{"type":"v1","addr":"172.21.15.28:6821","nonce":1642308925}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1642308925},{"type":"v1","addr":"172.21.15.28:6825","nonce":1642308925}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":1642308925},{"type":"v1","addr":"172.21.15.28:6823","nonce":1642308925}]},"public_addr":"172.21.15.28:6819/1642308925","cluster_addr":"172.21.15.28:6821/1642308925","heartbeat_back_addr":"172.21.15.28:6825/1642308925","heartbeat_front_addr":"172.21.15.28:6823/1642308925","state":["exists","up"]},{"osd":7,"uuid":"ca7a059f-0414-438e-b2d1-a10f348036be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":726062711},{"type":"v1","addr":"172.21.15.86:6817","nonce":726062711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":726062711},{"type":"v1","addr":"172.21.15.86:6819","nonce":726062711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":726062711},{"type":"v1","addr":"172.21.15.86:6823","nonce":726062711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":726062711},{"type":"v1","addr":"172.21.15.86:6821","nonce":726062711}]},"public_addr":"172.21.15.86:6817/726062711","cluster_addr":"172.21.15.86:6819/726062711","heartbeat_back_addr":"172.21.15.86:6823/726062711","heartbeat_front_addr":"172.21.15.86:6821/726062711","state":["exists","up"]},{"osd":8,"uuid":"92e3f41f-aebd-422e-a99e-555f7ba4aa53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":3007687277},{"type":"v1","addr":"172.21.15.186:6817","nonce":3007687277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":3007687277},{"type":"v1","addr":"172.21.15.186:6819","nonce":3007687277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":3007687277},{"type":"v1","addr":"172.21.15.186:6823","nonce":3007687277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":3007687277},{"type":"v1","addr":"172.21.15.186:6821","nonce":3007687277}]},"public_addr":"172.21.15.186:6817/3007687277","cluster_addr":"172.21.15.186:6819/3007687277","heartbeat_back_addr":"172.21.15.186:6823/3007687277","heartbeat_front_addr":"172.21.15.186:6821/3007687277","state":["exists","up"]},{"osd":9,"uuid":"38ff7d9d-ee85-404c-8c59-adc327e1ccfa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":136220472},{"type":"v1","addr":"172.21.15.28:6827","nonce":136220472}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":136220472},{"type":"v1","addr":"172.21.15.28:6829","nonce":136220472}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":136220472},{"type":"v1","addr":"172.21.15.28:6833","nonce":136220472}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":136220472},{"type":"v1","addr":"172.21.15.28:6831","nonce":136220472}]},"public_addr":"172.21.15.28:6827/136220472","cluster_addr":"172.21.15.28:6829/136220472","heartbeat_back_addr":"172.21.15.28:6833/136220472","heartbeat_front_addr":"172.21.15.28:6831/136220472","state":["exists","up"]},{"osd":10,"uuid":"5e9ff657-afa8-4bab-8491-f209038392b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1189968310},{"type":"v1","addr":"172.21.15.86:6825","nonce":1189968310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":1189968310},{"type":"v1","addr":"172.21.15.86:6827","nonce":1189968310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":1189968310},{"type":"v1","addr":"172.21.15.86:6831","nonce":1189968310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":1189968310},{"type":"v1","addr":"172.21.15.86:6829","nonce":1189968310}]},"public_addr":"172.21.15.86:6825/1189968310","cluster_addr":"172.21.15.86:6827/1189968310","heartbeat_back_addr":"172.21.15.86:6831/1189968310","heartbeat_front_addr":"172.21.15.86:6829/1189968310","state":["exists","up"]},{"osd":11,"uuid":"f112e3c6-65b6-4253-a966-62001db7d069","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3801668110},{"type":"v1","addr":"172.21.15.186:6825","nonce":3801668110}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3801668110},{"type":"v1","addr":"172.21.15.186:6827","nonce":3801668110}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3801668110},{"type":"v1","addr":"172.21.15.186:6831","nonce":3801668110}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3801668110},{"type":"v1","addr":"172.21.15.186:6829","nonce":3801668110}]},"public_addr":"172.21.15.186:6825/3801668110","cluster_addr":"172.21.15.186:6827/3801668110","heartbeat_back_addr":"172.21.15.186:6831/3801668110","heartbeat_front_addr":"172.21.15.186:6829/3801668110","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:00:40.483623+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:01:05.284385+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:01:28.645716+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:02:01.295092+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:02:26.486601+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:02:50.807478+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:03:24.756169+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:03:49.975173+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:04:13.772233+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:04:46.758555+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:05:12.331685+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:05:36.462548+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":2,"to":1}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.25","mappings":[{"from":5,"to":9}]},{"pgid":"3.27","mappings":[{"from":4,"to":8}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/2580087728":"2024-05-13T22:08:58.638536+0000","172.21.15.28:6801/3123328651":"2024-05-13T22:08:58.638536+0000","172.21.15.28:0/820158394":"2024-05-13T22:08:58.638536+0000","172.21.15.28:0/1978033727":"2024-05-13T22:08:58.638536+0000","172.21.15.28:0/1177009533":"2024-05-13T21:58:45.550459+0000","172.21.15.28:6800/3760499540":"2024-05-13T21:58:45.550459+0000","172.21.15.28:6801/3783742728":"2024-05-13T21:58:10.421564+0000","172.21.15.28:6800/3783742728":"2024-05-13T21:58:10.421564+0000","172.21.15.28:0/2229156260":"2024-05-13T21:58:45.550459+0000","172.21.15.28:0/1202140963":"2024-05-13T21:58:10.421564+0000","172.21.15.28:6800/3123328651":"2024-05-13T22:08:58.638536+0000","172.21.15.28:0/2924223784":"2024-05-13T21:58:45.550459+0000","172.21.15.28:6801/3760499540":"2024-05-13T21:58:45.550459+0000","172.21.15.28:0/1240027206":"2024-05-13T21:58:10.421564+0000","172.21.15.28:0/1581755795":"2024-05-13T22:08:58.638536+0000","172.21.15.28:0/171417728":"2024-05-13T21:58:10.421564+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":5,"length":1},{"begin":15,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-12T22:19:39.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:39.795+0000 7fead1800640 1 -- 172.21.15.28:0/3074967529 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7feab00a2d30 msgr2=0x7feab00a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:39.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:39.795+0000 7fead1800640 1 --2- 172.21.15.28:0/3074967529 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7feab00a2d30 0x7feab00a51f0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7feabc009bb0 tx=0x7feabc005f90 comp rx=0 tx=0).stop 2024-05-12T22:19:39.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:39.795+0000 7fead1800640 1 -- 172.21.15.28:0/3074967529 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feacc107660 msgr2=0x7feacc100e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:39.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:39.795+0000 7fead1800640 1 --2- 172.21.15.28:0/3074967529 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feacc107660 0x7feacc100e30 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7feab802a9a0 tx=0x7feab8003b20 comp rx=0 tx=0).stop 2024-05-12T22:19:39.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:39.795+0000 7fead1800640 1 -- 172.21.15.28:0/3074967529 shutdown_connections 2024-05-12T22:19:39.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:39.795+0000 7fead1800640 1 --2- 172.21.15.28:0/3074967529 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7feab00a2d30 0x7feab00a51f0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:39.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:39.795+0000 7fead1800640 1 --2- 172.21.15.28:0/3074967529 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feacc107f80 0x7feacc101370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:39.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:39.795+0000 7fead1800640 1 --2- 172.21.15.28:0/3074967529 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feacc107660 0x7feacc100e30 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:39.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:39.795+0000 7fead1800640 1 --2- 172.21.15.28:0/3074967529 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feacc069110 0x7feacc1008f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:39.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:39.795+0000 7fead1800640 1 -- 172.21.15.28:0/3074967529 >> 172.21.15.28:0/3074967529 conn(0x7feacc076070 msgr2=0x7feacc0ff510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:19:39.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:39.795+0000 7fead1800640 1 -- 172.21.15.28:0/3074967529 shutdown_connections 2024-05-12T22:19:39.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:39.795+0000 7fead1800640 1 -- 172.21.15.28:0/3074967529 wait complete. 2024-05-12T22:19:39.815 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-12T22:01:32.292826+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-12T22:06:31.540263+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-12T22:06:32.906118+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '96', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 15, 'snap_epoch': 96, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-12T22:19:39.815 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-12T22:19:40.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.027+0000 7f55e1600640 1 -- 172.21.15.28:0/4007895522 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f55dc073b00 msgr2=0x7f55dc073ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:40.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.027+0000 7f55e1600640 1 --2- 172.21.15.28:0/4007895522 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f55dc073b00 0x7f55dc073ee0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f55cc009900 tx=0x7f55cc02a480 comp rx=0 tx=0).stop 2024-05-12T22:19:40.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.027+0000 7f55e1600640 1 -- 172.21.15.28:0/4007895522 shutdown_connections 2024-05-12T22:19:40.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.027+0000 7f55e1600640 1 --2- 172.21.15.28:0/4007895522 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55dc1128e0 0x7f55dc114cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:40.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.027+0000 7f55e1600640 1 --2- 172.21.15.28:0/4007895522 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f55dc074420 0x7f55dc1123a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:40.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.027+0000 7f55e1600640 1 --2- 172.21.15.28:0/4007895522 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f55dc073b00 0x7f55dc073ee0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:40.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.027+0000 7f55e1600640 1 -- 172.21.15.28:0/4007895522 >> 172.21.15.28:0/4007895522 conn(0x7f55dc078c80 msgr2=0x7f55dc079090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:19:40.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.028+0000 7f55e1600640 1 -- 172.21.15.28:0/4007895522 shutdown_connections 2024-05-12T22:19:40.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.028+0000 7f55e1600640 1 -- 172.21.15.28:0/4007895522 wait complete. 2024-05-12T22:19:40.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.028+0000 7f55e1600640 1 Processor -- start 2024-05-12T22:19:40.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.029+0000 7f55e1600640 1 -- start start 2024-05-12T22:19:40.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.029+0000 7f55e1600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55dc073b00 0x7f55dc19e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:40.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.029+0000 7f55e1600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f55dc074420 0x7f55dc19eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:40.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.029+0000 7f55e1600640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f55dc1128e0 0x7f55dc1a3020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:40.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.029+0000 7f55e1600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f55dc118260 con 0x7f55dc073b00 2024-05-12T22:19:40.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.029+0000 7f55e1600640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f55dc1180e0 con 0x7f55dc074420 2024-05-12T22:19:40.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.029+0000 7f55e1600640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f55dc1183e0 con 0x7f55dc1128e0 2024-05-12T22:19:40.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.029+0000 7f55daa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55dc073b00 0x7f55dc19e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:40.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.029+0000 7f55daa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55dc073b00 0x7f55dc19e760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42510/0 (socket says 172.21.15.28:42510) 2024-05-12T22:19:40.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.029+0000 7f55d2000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f55dc074420 0x7f55dc19eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:40.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.029+0000 7f55daa00640 1 -- 172.21.15.28:0/2057342854 learned_addr learned my addr 172.21.15.28:0/2057342854 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:19:40.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.029+0000 7f55db400640 1 --2- 172.21.15.28:0/2057342854 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f55dc1128e0 0x7f55dc1a3020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:40.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.030+0000 7f55d2000640 1 -- 172.21.15.28:0/2057342854 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f55dc1128e0 msgr2=0x7f55dc1a3020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:40.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.030+0000 7f55d2000640 1 --2- 172.21.15.28:0/2057342854 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f55dc1128e0 0x7f55dc1a3020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:40.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.030+0000 7f55d2000640 1 -- 172.21.15.28:0/2057342854 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55dc073b00 msgr2=0x7f55dc19e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:40.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.030+0000 7f55d2000640 1 --2- 172.21.15.28:0/2057342854 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55dc073b00 0x7f55dc19e760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:40.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.030+0000 7f55d2000640 1 -- 172.21.15.28:0/2057342854 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55dc1a3750 con 0x7f55dc074420 2024-05-12T22:19:40.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.030+0000 7f55d2000640 1 --2- 172.21.15.28:0/2057342854 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f55dc074420 0x7f55dc19eca0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f55c400e9f0 tx=0x7f55c400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:19:40.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.030+0000 7f55d3e00640 1 -- 172.21.15.28:0/2057342854 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55c400ccc0 con 0x7f55dc074420 2024-05-12T22:19:40.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.030+0000 7f55d3e00640 1 -- 172.21.15.28:0/2057342854 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f55c4005020 con 0x7f55dc074420 2024-05-12T22:19:40.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.030+0000 7f55d3e00640 1 -- 172.21.15.28:0/2057342854 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55c40049e0 con 0x7f55dc074420 2024-05-12T22:19:40.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.030+0000 7f55e1600640 1 -- 172.21.15.28:0/2057342854 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55dc1a3a40 con 0x7f55dc074420 2024-05-12T22:19:40.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.031+0000 7f55e1600640 1 -- 172.21.15.28:0/2057342854 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55dc1ab2d0 con 0x7f55dc074420 2024-05-12T22:19:40.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.032+0000 7f55e1600640 1 -- 172.21.15.28:0/2057342854 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f55dc068aa0 con 0x7f55dc074420 2024-05-12T22:19:40.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.033+0000 7f55d3e00640 1 -- 172.21.15.28:0/2057342854 <== mon.2 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7f55c4005560 con 0x7f55dc074420 2024-05-12T22:19:40.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.035+0000 7f55d3e00640 1 --2- 172.21.15.28:0/2057342854 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f55c00a2d00 0x7f55c00a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:40.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.035+0000 7f55d3e00640 1 -- 172.21.15.28:0/2057342854 <== mon.2 v2:172.21.15.86:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8513+0+0 (secure 0 0 0) 0x7f55c40cf400 con 0x7f55dc074420 2024-05-12T22:19:40.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.035+0000 7f55daa00640 1 --2- 172.21.15.28:0/2057342854 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f55c00a2d00 0x7f55c00a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:40.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.036+0000 7f55daa00640 1 --2- 172.21.15.28:0/2057342854 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f55c00a2d00 0x7f55c00a51c0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f55cc000c00 tx=0x7f55cc0046c0 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:19:40.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.039+0000 7f55d3e00640 1 -- 172.21.15.28:0/2057342854 <== mon.2 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f55c40051d0 con 0x7f55dc074420 2024-05-12T22:19:40.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.245+0000 7f55e1600640 1 -- 172.21.15.28:0/2057342854 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f55dc02d1c0 con 0x7f55dc074420 2024-05-12T22:19:40.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.253+0000 7f55d3e00640 1 -- 172.21.15.28:0/2057342854 <== mon.2 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v97) ==== 93+0+10 (secure 0 0 0) 0x7f55c409bfc0 con 0x7f55dc074420 2024-05-12T22:19:40.253 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-05-12T22:19:40.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.256+0000 7f55e1600640 1 -- 172.21.15.28:0/2057342854 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f55c00a2d00 msgr2=0x7f55c00a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:40.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.256+0000 7f55e1600640 1 --2- 172.21.15.28:0/2057342854 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f55c00a2d00 0x7f55c00a51c0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f55cc000c00 tx=0x7f55cc0046c0 comp rx=0 tx=0).stop 2024-05-12T22:19:40.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.256+0000 7f55e1600640 1 -- 172.21.15.28:0/2057342854 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f55dc074420 msgr2=0x7f55dc19eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:40.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.256+0000 7f55e1600640 1 --2- 172.21.15.28:0/2057342854 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f55dc074420 0x7f55dc19eca0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f55c400e9f0 tx=0x7f55c400eeb0 comp rx=0 tx=0).stop 2024-05-12T22:19:40.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.256+0000 7f55e1600640 1 -- 172.21.15.28:0/2057342854 shutdown_connections 2024-05-12T22:19:40.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.256+0000 7f55e1600640 1 --2- 172.21.15.28:0/2057342854 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f55c00a2d00 0x7f55c00a51c0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:40.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.256+0000 7f55e1600640 1 --2- 172.21.15.28:0/2057342854 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f55dc1128e0 0x7f55dc1a3020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:40.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.256+0000 7f55e1600640 1 --2- 172.21.15.28:0/2057342854 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f55dc074420 0x7f55dc19eca0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:40.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.256+0000 7f55e1600640 1 --2- 172.21.15.28:0/2057342854 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55dc073b00 0x7f55dc19e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:40.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.256+0000 7f55e1600640 1 -- 172.21.15.28:0/2057342854 >> 172.21.15.28:0/2057342854 conn(0x7f55dc078c80 msgr2=0x7f55dc0fe700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:19:40.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.257+0000 7f55e1600640 1 -- 172.21.15.28:0/2057342854 shutdown_connections 2024-05-12T22:19:40.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.257+0000 7f55e1600640 1 -- 172.21.15.28:0/2057342854 wait complete. 2024-05-12T22:19:40.276 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-12T22:19:40.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.498+0000 7fa877c00640 1 -- 172.21.15.28:0/2705360623 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa870074420 msgr2=0x7fa87010dbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:40.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.498+0000 7fa877c00640 1 --2- 172.21.15.28:0/2705360623 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa870074420 0x7fa87010dbd0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fa85c009900 tx=0x7fa85c02a480 comp rx=0 tx=0).stop 2024-05-12T22:19:40.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.498+0000 7fa877c00640 1 -- 172.21.15.28:0/2705360623 shutdown_connections 2024-05-12T22:19:40.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.498+0000 7fa877c00640 1 --2- 172.21.15.28:0/2705360623 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa87010e110 0x7fa870110500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:40.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.498+0000 7fa877c00640 1 --2- 172.21.15.28:0/2705360623 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa870074420 0x7fa87010dbd0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:40.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.498+0000 7fa877c00640 1 --2- 172.21.15.28:0/2705360623 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa870073b00 0x7fa870073ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:40.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.498+0000 7fa877c00640 1 -- 172.21.15.28:0/2705360623 >> 172.21.15.28:0/2705360623 conn(0x7fa870078c80 msgr2=0x7fa870079090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:19:40.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.498+0000 7fa877c00640 1 -- 172.21.15.28:0/2705360623 shutdown_connections 2024-05-12T22:19:40.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.498+0000 7fa877c00640 1 -- 172.21.15.28:0/2705360623 wait complete. 2024-05-12T22:19:40.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.499+0000 7fa877c00640 1 Processor -- start 2024-05-12T22:19:40.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.499+0000 7fa877c00640 1 -- start start 2024-05-12T22:19:40.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.500+0000 7fa877c00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa870073b00 0x7fa87019a360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:40.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.500+0000 7fa877c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa870074420 0x7fa87019a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:40.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.500+0000 7fa877c00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa87010e110 0x7fa87019ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:40.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.500+0000 7fa877c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa870113e60 con 0x7fa870074420 2024-05-12T22:19:40.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.500+0000 7fa877c00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fa870113ce0 con 0x7fa870073b00 2024-05-12T22:19:40.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.500+0000 7fa877c00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa870113fe0 con 0x7fa87010e110 2024-05-12T22:19:40.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.500+0000 7fa86fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa870074420 0x7fa87019a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:40.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.500+0000 7fa86fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa870074420 0x7fa87019a8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42536/0 (socket says 172.21.15.28:42536) 2024-05-12T22:19:40.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.500+0000 7fa86fe00640 1 -- 172.21.15.28:0/1994700608 learned_addr learned my addr 172.21.15.28:0/1994700608 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:19:40.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.500+0000 7fa875200640 1 --2- 172.21.15.28:0/1994700608 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa870073b00 0x7fa87019a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:40.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.500+0000 7fa875c00640 1 --2- 172.21.15.28:0/1994700608 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa87010e110 0x7fa87019ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:40.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.500+0000 7fa875200640 1 -- 172.21.15.28:0/1994700608 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa87010e110 msgr2=0x7fa87019ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:40.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.500+0000 7fa875200640 1 --2- 172.21.15.28:0/1994700608 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa87010e110 0x7fa87019ec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:40.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.500+0000 7fa875200640 1 -- 172.21.15.28:0/1994700608 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa870074420 msgr2=0x7fa87019a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:40.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.500+0000 7fa875200640 1 --2- 172.21.15.28:0/1994700608 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa870074420 0x7fa87019a8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:40.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.500+0000 7fa875200640 1 -- 172.21.15.28:0/1994700608 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa87019f350 con 0x7fa870073b00 2024-05-12T22:19:40.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.500+0000 7fa86fe00640 1 --2- 172.21.15.28:0/1994700608 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa870074420 0x7fa87019a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:19:40.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.501+0000 7fa875200640 1 --2- 172.21.15.28:0/1994700608 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa870073b00 0x7fa87019a360 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fa86400d6d0 tx=0x7fa86400db90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:19:40.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.501+0000 7fa86d600640 1 -- 172.21.15.28:0/1994700608 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa864017070 con 0x7fa870073b00 2024-05-12T22:19:40.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.501+0000 7fa86d600640 1 -- 172.21.15.28:0/1994700608 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa864009820 con 0x7fa870073b00 2024-05-12T22:19:40.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.501+0000 7fa877c00640 1 -- 172.21.15.28:0/1994700608 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa87019f640 con 0x7fa870073b00 2024-05-12T22:19:40.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.501+0000 7fa86d600640 1 -- 172.21.15.28:0/1994700608 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa864004cd0 con 0x7fa870073b00 2024-05-12T22:19:40.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.501+0000 7fa877c00640 1 -- 172.21.15.28:0/1994700608 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8701a6ed0 con 0x7fa870073b00 2024-05-12T22:19:40.504 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:19:40 smithi086 ceph-mon[34297]: pgmap v331: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:19:40.504 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:19:40 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/3074967529' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-12T22:19:40.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.502+0000 7fa877c00640 1 -- 172.21.15.28:0/1994700608 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa870068aa0 con 0x7fa870073b00 2024-05-12T22:19:40.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.504+0000 7fa86d600640 1 -- 172.21.15.28:0/1994700608 <== mon.2 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7fa8640099d0 con 0x7fa870073b00 2024-05-12T22:19:40.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.505+0000 7fa86d600640 1 --2- 172.21.15.28:0/1994700608 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fa8580a2d80 0x7fa8580a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:40.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.506+0000 7fa86d600640 1 -- 172.21.15.28:0/1994700608 <== mon.2 v2:172.21.15.86:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8513+0+0 (secure 0 0 0) 0x7fa8640cef20 con 0x7fa870073b00 2024-05-12T22:19:40.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.506+0000 7fa86fe00640 1 --2- 172.21.15.28:0/1994700608 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fa8580a2d80 0x7fa8580a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:40.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.506+0000 7fa86fe00640 1 --2- 172.21.15.28:0/1994700608 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fa8580a2d80 0x7fa8580a5240 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fa85c0096c0 tx=0x7fa85c0046c0 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:19:40.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.510+0000 7fa86d600640 1 -- 172.21.15.28:0/1994700608 <== mon.2 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fa86409b490 con 0x7fa870073b00 2024-05-12T22:19:40.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:19:40 smithi028 ceph-mon[29847]: pgmap v331: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:19:40.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:19:40 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/3074967529' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-12T22:19:40.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:19:40 smithi186 ceph-mon[34785]: pgmap v331: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:19:40.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:19:40 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/3074967529' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-12T22:19:40.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.713+0000 7fa877c00640 1 -- 172.21.15.28:0/1994700608 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fa87002d1c0 con 0x7fa870073b00 2024-05-12T22:19:40.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.714+0000 7fa86d600640 1 -- 172.21.15.28:0/1994700608 <== mon.2 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v97) ==== 104+0+11 (secure 0 0 0) 0x7fa86409f790 con 0x7fa870073b00 2024-05-12T22:19:40.714 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-05-12T22:19:40.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.717+0000 7fa877c00640 1 -- 172.21.15.28:0/1994700608 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fa8580a2d80 msgr2=0x7fa8580a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:40.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.717+0000 7fa877c00640 1 --2- 172.21.15.28:0/1994700608 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fa8580a2d80 0x7fa8580a5240 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fa85c0096c0 tx=0x7fa85c0046c0 comp rx=0 tx=0).stop 2024-05-12T22:19:40.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.717+0000 7fa877c00640 1 -- 172.21.15.28:0/1994700608 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa870073b00 msgr2=0x7fa87019a360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:40.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.717+0000 7fa877c00640 1 --2- 172.21.15.28:0/1994700608 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa870073b00 0x7fa87019a360 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fa86400d6d0 tx=0x7fa86400db90 comp rx=0 tx=0).stop 2024-05-12T22:19:40.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.717+0000 7fa877c00640 1 -- 172.21.15.28:0/1994700608 shutdown_connections 2024-05-12T22:19:40.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.717+0000 7fa877c00640 1 --2- 172.21.15.28:0/1994700608 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fa8580a2d80 0x7fa8580a5240 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:40.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.717+0000 7fa877c00640 1 --2- 172.21.15.28:0/1994700608 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa87010e110 0x7fa87019ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:40.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.717+0000 7fa877c00640 1 --2- 172.21.15.28:0/1994700608 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa870074420 0x7fa87019a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:40.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.717+0000 7fa877c00640 1 --2- 172.21.15.28:0/1994700608 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa870073b00 0x7fa87019a360 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:40.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.717+0000 7fa877c00640 1 -- 172.21.15.28:0/1994700608 >> 172.21.15.28:0/1994700608 conn(0x7fa870078c80 msgr2=0x7fa8700fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:19:40.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.717+0000 7fa877c00640 1 -- 172.21.15.28:0/1994700608 shutdown_connections 2024-05-12T22:19:40.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.717+0000 7fa877c00640 1 -- 172.21.15.28:0/1994700608 wait complete. 2024-05-12T22:19:40.736 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-12T22:19:40.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.961+0000 7f2a61000640 1 -- 172.21.15.28:0/894365648 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a5c1098f0 msgr2=0x7f2a5c109cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:40.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.961+0000 7f2a61000640 1 --2- 172.21.15.28:0/894365648 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a5c1098f0 0x7f2a5c109cd0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f2a4c009930 tx=0x7f2a4c02a420 comp rx=0 tx=0).stop 2024-05-12T22:19:40.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.961+0000 7f2a61000640 1 -- 172.21.15.28:0/894365648 shutdown_connections 2024-05-12T22:19:40.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.961+0000 7f2a61000640 1 --2- 172.21.15.28:0/894365648 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2a5c1042a0 0x7f2a5c1107b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:40.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.961+0000 7f2a61000640 1 --2- 172.21.15.28:0/894365648 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2a5c1038e0 0x7f2a5c103d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:40.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.961+0000 7f2a61000640 1 --2- 172.21.15.28:0/894365648 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a5c1098f0 0x7f2a5c109cd0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:40.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.961+0000 7f2a61000640 1 -- 172.21.15.28:0/894365648 >> 172.21.15.28:0/894365648 conn(0x7f2a5c0ff370 msgr2=0x7f2a5c101790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:19:40.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.961+0000 7f2a61000640 1 -- 172.21.15.28:0/894365648 shutdown_connections 2024-05-12T22:19:40.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.961+0000 7f2a61000640 1 -- 172.21.15.28:0/894365648 wait complete. 2024-05-12T22:19:40.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.962+0000 7f2a61000640 1 Processor -- start 2024-05-12T22:19:40.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.962+0000 7f2a61000640 1 -- start start 2024-05-12T22:19:40.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.963+0000 7f2a61000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a5c1038e0 0x7f2a5c19a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:40.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.963+0000 7f2a61000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2a5c1042a0 0x7f2a5c19a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:40.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.963+0000 7f2a61000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2a5c1098f0 0x7f2a5c19eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:40.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.963+0000 7f2a61000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2a5c113d50 con 0x7f2a5c1038e0 2024-05-12T22:19:40.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.963+0000 7f2a61000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f2a5c113bd0 con 0x7f2a5c1042a0 2024-05-12T22:19:40.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.963+0000 7f2a61000640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f2a5c113ed0 con 0x7f2a5c1098f0 2024-05-12T22:19:40.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.963+0000 7f2a5b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a5c1038e0 0x7f2a5c19a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:40.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.963+0000 7f2a5b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a5c1038e0 0x7f2a5c19a270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42554/0 (socket says 172.21.15.28:42554) 2024-05-12T22:19:40.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.963+0000 7f2a5b400640 1 -- 172.21.15.28:0/3853316862 learned_addr learned my addr 172.21.15.28:0/3853316862 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:19:40.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.963+0000 7f2a5be00640 1 --2- 172.21.15.28:0/3853316862 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2a5c1098f0 0x7f2a5c19eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:40.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.963+0000 7f2a52a00640 1 --2- 172.21.15.28:0/3853316862 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2a5c1042a0 0x7f2a5c19a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:40.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.964+0000 7f2a5b400640 1 -- 172.21.15.28:0/3853316862 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2a5c1098f0 msgr2=0x7f2a5c19eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:40.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.964+0000 7f2a5b400640 1 --2- 172.21.15.28:0/3853316862 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2a5c1098f0 0x7f2a5c19eae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:40.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.964+0000 7f2a5b400640 1 -- 172.21.15.28:0/3853316862 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2a5c1042a0 msgr2=0x7f2a5c19a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:40.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.964+0000 7f2a5b400640 1 --2- 172.21.15.28:0/3853316862 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2a5c1042a0 0x7f2a5c19a7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:40.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.964+0000 7f2a5b400640 1 -- 172.21.15.28:0/3853316862 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a5c19f210 con 0x7f2a5c1038e0 2024-05-12T22:19:40.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.964+0000 7f2a5be00640 1 --2- 172.21.15.28:0/3853316862 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2a5c1098f0 0x7f2a5c19eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:19:40.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.964+0000 7f2a5b400640 1 --2- 172.21.15.28:0/3853316862 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a5c1038e0 0x7f2a5c19a270 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f2a4c02a940 tx=0x7f2a4c003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:19:40.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.964+0000 7f2a58c00640 1 -- 172.21.15.28:0/3853316862 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a4c03f070 con 0x7f2a5c1038e0 2024-05-12T22:19:40.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.964+0000 7f2a58c00640 1 -- 172.21.15.28:0/3853316862 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2a4c031c90 con 0x7f2a5c1038e0 2024-05-12T22:19:40.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.964+0000 7f2a58c00640 1 -- 172.21.15.28:0/3853316862 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a4c035030 con 0x7f2a5c1038e0 2024-05-12T22:19:40.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.964+0000 7f2a61000640 1 -- 172.21.15.28:0/3853316862 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a5c19f4a0 con 0x7f2a5c1038e0 2024-05-12T22:19:40.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.964+0000 7f2a61000640 1 -- 172.21.15.28:0/3853316862 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a5c1a6d90 con 0x7f2a5c1038e0 2024-05-12T22:19:40.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.966+0000 7f2a61000640 1 -- 172.21.15.28:0/3853316862 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a5c0689c0 con 0x7f2a5c1038e0 2024-05-12T22:19:40.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.968+0000 7f2a58c00640 1 -- 172.21.15.28:0/3853316862 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7f2a4c042070 con 0x7f2a5c1038e0 2024-05-12T22:19:40.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.969+0000 7f2a58c00640 1 --2- 172.21.15.28:0/3853316862 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f2a400a2d00 0x7f2a400a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:40.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.969+0000 7f2a58c00640 1 -- 172.21.15.28:0/3853316862 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8513+0+0 (secure 0 0 0) 0x7f2a4c0efc40 con 0x7f2a5c1038e0 2024-05-12T22:19:40.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.969+0000 7f2a52a00640 1 --2- 172.21.15.28:0/3853316862 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f2a400a2d00 0x7f2a400a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:40.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.970+0000 7f2a52a00640 1 --2- 172.21.15.28:0/3853316862 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f2a400a2d00 0x7f2a400a51c0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f2a5c101a00 tx=0x7f2a44009500 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:19:40.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:40.973+0000 7f2a58c00640 1 -- 172.21.15.28:0/3853316862 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f2a4c0bc800 con 0x7f2a5c1038e0 2024-05-12T22:19:41.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:41.179+0000 7f2a61000640 1 -- 172.21.15.28:0/3853316862 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f2a5c10be50 con 0x7f2a5c1038e0 2024-05-12T22:19:41.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:41.180+0000 7f2a58c00640 1 -- 172.21.15.28:0/3853316862 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v97) ==== 100+0+11 (secure 0 0 0) 0x7f2a4c0bc100 con 0x7f2a5c1038e0 2024-05-12T22:19:41.180 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-05-12T22:19:41.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:41.182+0000 7f2a61000640 1 -- 172.21.15.28:0/3853316862 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f2a400a2d00 msgr2=0x7f2a400a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:41.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:41.182+0000 7f2a61000640 1 --2- 172.21.15.28:0/3853316862 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f2a400a2d00 0x7f2a400a51c0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f2a5c101a00 tx=0x7f2a44009500 comp rx=0 tx=0).stop 2024-05-12T22:19:41.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:41.182+0000 7f2a61000640 1 -- 172.21.15.28:0/3853316862 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a5c1038e0 msgr2=0x7f2a5c19a270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:41.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:41.182+0000 7f2a61000640 1 --2- 172.21.15.28:0/3853316862 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a5c1038e0 0x7f2a5c19a270 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f2a4c02a940 tx=0x7f2a4c003b90 comp rx=0 tx=0).stop 2024-05-12T22:19:41.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:41.183+0000 7f2a61000640 1 -- 172.21.15.28:0/3853316862 shutdown_connections 2024-05-12T22:19:41.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:41.183+0000 7f2a61000640 1 --2- 172.21.15.28:0/3853316862 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f2a400a2d00 0x7f2a400a51c0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:41.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:41.183+0000 7f2a61000640 1 --2- 172.21.15.28:0/3853316862 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2a5c1098f0 0x7f2a5c19eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:41.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:41.183+0000 7f2a61000640 1 --2- 172.21.15.28:0/3853316862 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2a5c1042a0 0x7f2a5c19a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:41.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:41.183+0000 7f2a61000640 1 --2- 172.21.15.28:0/3853316862 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a5c1038e0 0x7f2a5c19a270 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:41.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:41.183+0000 7f2a61000640 1 -- 172.21.15.28:0/3853316862 >> 172.21.15.28:0/3853316862 conn(0x7f2a5c0ff370 msgr2=0x7f2a5c0ffb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:19:41.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:41.183+0000 7f2a61000640 1 -- 172.21.15.28:0/3853316862 shutdown_connections 2024-05-12T22:19:41.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:41.183+0000 7f2a61000640 1 -- 172.21.15.28:0/3853316862 wait complete. 2024-05-12T22:19:41.202 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-05-12T22:19:41.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:41.418+0000 7fdd0d600640 1 -- 172.21.15.28:0/1824491489 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdd08109a10 msgr2=0x7fdd08109df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:41.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:41.418+0000 7fdd0d600640 1 --2- 172.21.15.28:0/1824491489 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdd08109a10 0x7fdd08109df0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fdcf8009be0 tx=0x7fdcf802a6c0 comp rx=0 tx=0).stop 2024-05-12T22:19:41.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:41.419+0000 7fdd0d600640 1 -- 172.21.15.28:0/1824491489 shutdown_connections 2024-05-12T22:19:41.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:41.419+0000 7fdd0d600640 1 --2- 172.21.15.28:0/1824491489 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdd081043c0 0x7fdd081108d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:41.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:41.419+0000 7fdd0d600640 1 --2- 172.21.15.28:0/1824491489 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd08103a00 0x7fdd08103e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:41.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:41.419+0000 7fdd0d600640 1 --2- 172.21.15.28:0/1824491489 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdd08109a10 0x7fdd08109df0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:41.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:41.419+0000 7fdd0d600640 1 -- 172.21.15.28:0/1824491489 >> 172.21.15.28:0/1824491489 conn(0x7fdd080ff530 msgr2=0x7fdd08101950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:19:41.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:41.419+0000 7fdd0d600640 1 -- 172.21.15.28:0/1824491489 shutdown_connections 2024-05-12T22:19:41.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:41.419+0000 7fdd0d600640 1 -- 172.21.15.28:0/1824491489 wait complete. 2024-05-12T22:19:41.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:41.420+0000 7fdd0d600640 1 Processor -- start 2024-05-12T22:19:41.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:41.420+0000 7fdd0d600640 1 -- start start 2024-05-12T22:19:41.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:41.421+0000 7fdd0d600640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdd08103a00 0x7fdd0819a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:41.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:41.421+0000 7fdd0d600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdd081043c0 0x7fdd0819a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:41.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:41.421+0000 7fdd0d600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd08109a10 0x7fdd0819ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:41.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:41.421+0000 7fdd0d600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdd08113df0 con 0x7fdd08109a10 2024-05-12T22:19:41.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:41.421+0000 7fdd0d600640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fdd08113c70 con 0x7fdd081043c0 2024-05-12T22:19:41.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:41.421+0000 7fdd0d600640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fdd08113f70 con 0x7fdd08103a00 2024-05-12T22:19:41.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:41.421+0000 7fdd06a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdd08103a00 0x7fdd0819a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:41.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:41.421+0000 7fdd06a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdd08103a00 0x7fdd0819a310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:57784/0 (socket says 172.21.15.28:57784) 2024-05-12T22:19:41.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:41.421+0000 7fdd06a00640 1 -- 172.21.15.28:0/3604000295 learned_addr learned my addr 172.21.15.28:0/3604000295 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:19:41.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:41.421+0000 7fdd07400640 1 --2- 172.21.15.28:0/3604000295 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd08109a10 0x7fdd0819ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:41.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:41.421+0000 7fdd06000640 1 --2- 172.21.15.28:0/3604000295 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdd081043c0 0x7fdd0819a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:41.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:41.422+0000 7fdd06a00640 1 -- 172.21.15.28:0/3604000295 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdd081043c0 msgr2=0x7fdd0819a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:41.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:41.422+0000 7fdd06a00640 1 --2- 172.21.15.28:0/3604000295 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdd081043c0 0x7fdd0819a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:41.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:41.422+0000 7fdd06a00640 1 -- 172.21.15.28:0/3604000295 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd08109a10 msgr2=0x7fdd0819ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:41.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:41.422+0000 7fdd06a00640 1 --2- 172.21.15.28:0/3604000295 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd08109a10 0x7fdd0819ebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:41.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:41.422+0000 7fdd06a00640 1 -- 172.21.15.28:0/3604000295 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd0819f300 con 0x7fdd08103a00 2024-05-12T22:19:41.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:41.422+0000 7fdd06a00640 1 --2- 172.21.15.28:0/3604000295 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdd08103a00 0x7fdd0819a310 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fdcf802acb0 tx=0x7fdcf8030c30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:19:41.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:41.422+0000 7fdcff400640 1 -- 172.21.15.28:0/3604000295 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcf8041020 con 0x7fdd08103a00 2024-05-12T22:19:41.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:41.422+0000 7fdd0d600640 1 -- 172.21.15.28:0/3604000295 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd0819f590 con 0x7fdd08103a00 2024-05-12T22:19:41.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:41.422+0000 7fdcff400640 1 -- 172.21.15.28:0/3604000295 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdcf8003e50 con 0x7fdd08103a00 2024-05-12T22:19:41.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:41.423+0000 7fdcff400640 1 -- 172.21.15.28:0/3604000295 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcf8036040 con 0x7fdd08103a00 2024-05-12T22:19:41.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:41.423+0000 7fdd0d600640 1 -- 172.21.15.28:0/3604000295 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd081a6e80 con 0x7fdd08103a00 2024-05-12T22:19:41.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:41.424+0000 7fdd0d600640 1 -- 172.21.15.28:0/3604000295 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd08068aa0 con 0x7fdd08103a00 2024-05-12T22:19:41.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:41.426+0000 7fdcff400640 1 -- 172.21.15.28:0/3604000295 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7fdcf8044020 con 0x7fdd08103a00 2024-05-12T22:19:41.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:41.427+0000 7fdcff400640 1 --2- 172.21.15.28:0/3604000295 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fdcec0a2d00 0x7fdcec0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:19:41.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:41.427+0000 7fdcff400640 1 -- 172.21.15.28:0/3604000295 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8513+0+0 (secure 0 0 0) 0x7fdcf80f00c0 con 0x7fdd08103a00 2024-05-12T22:19:41.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:41.427+0000 7fdd06000640 1 --2- 172.21.15.28:0/3604000295 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fdcec0a2d00 0x7fdcec0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:19:41.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:41.428+0000 7fdd06000640 1 --2- 172.21.15.28:0/3604000295 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fdcec0a2d00 0x7fdcec0a51c0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fdd08101c10 tx=0x7fdcf0009260 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:19:41.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:41.431+0000 7fdcff400640 1 -- 172.21.15.28:0/3604000295 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fdcf804b020 con 0x7fdd08103a00 2024-05-12T22:19:41.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:19:41 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/2057342854' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-12T22:19:41.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:19:41 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/1994700608' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-12T22:19:41.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:19:41 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/3853316862' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-12T22:19:41.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:19:41 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/2057342854' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-12T22:19:41.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:19:41 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/1994700608' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-12T22:19:41.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:19:41 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/3853316862' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-12T22:19:41.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:41.691+0000 7fdd0d600640 1 -- 172.21.15.28:0/3604000295 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) -- 0x7fdd0810bf70 con 0x7fdd08103a00 2024-05-12T22:19:41.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:41.704+0000 7fdcff400640 1 -- 172.21.15.28:0/3604000295 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v27) ==== 81+0+740902 (secure 0 0 0) 0x7fdcf80bcc80 con 0x7fdd08103a00 2024-05-12T22:19:41.716 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-12T22:19:41.716 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-05-12T22:19:41.716 INFO:teuthology.orchestra.run.smithi028.stdout: "epoch": 27, 2024-05-12T22:19:41.716 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:41.716 INFO:teuthology.orchestra.run.smithi028.stdout: "active_gid": 25189, 2024-05-12T22:19:41.717 INFO:teuthology.orchestra.run.smithi028.stdout: "active_name": "x", 2024-05-12T22:19:41.717 INFO:teuthology.orchestra.run.smithi028.stdout: "active_addrs": { 2024-05-12T22:19:41.717 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-05-12T22:19:41.717 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-12T22:19:41.717 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-05-12T22:19:41.717 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6800", 2024-05-12T22:19:41.718 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3177640985 2024-05-12T22:19:41.718 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.718 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-12T22:19:41.718 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-05-12T22:19:41.718 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6801", 2024-05-12T22:19:41.718 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3177640985 2024-05-12T22:19:41.718 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:41.718 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-05-12T22:19:41.718 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.719 INFO:teuthology.orchestra.run.smithi028.stdout: "active_addr": "172.21.15.28:6801/3177640985", 2024-05-12T22:19:41.719 INFO:teuthology.orchestra.run.smithi028.stdout: "active_change": "2024-05-12T22:08:58.638793+0000", 2024-05-12T22:19:41.719 INFO:teuthology.orchestra.run.smithi028.stdout: "active_mgr_features": 4540701547738038271, 2024-05-12T22:19:41.719 INFO:teuthology.orchestra.run.smithi028.stdout: "available": true, 2024-05-12T22:19:41.719 INFO:teuthology.orchestra.run.smithi028.stdout: "standbys": [ 2024-05-12T22:19:41.719 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-12T22:19:41.719 INFO:teuthology.orchestra.run.smithi028.stdout: "gid": 25183, 2024-05-12T22:19:41.719 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "z", 2024-05-12T22:19:41.719 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_features": 4540701547738038271, 2024-05-12T22:19:41.720 INFO:teuthology.orchestra.run.smithi028.stdout: "available_modules": [ 2024-05-12T22:19:41.720 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-12T22:19:41.720 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "alerts", 2024-05-12T22:19:41.720 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-12T22:19:41.720 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-12T22:19:41.720 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-12T22:19:41.720 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-05-12T22:19:41.720 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-05-12T22:19:41.720 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-05-12T22:19:41.721 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.721 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:41.721 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-05-12T22:19:41.721 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.721 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.721 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.721 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How frequently to reexamine health status", 2024-05-12T22:19:41.721 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.722 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.722 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.722 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.722 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-12T22:19:41.723 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-12T22:19:41.723 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:41.723 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.723 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:41.723 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:41.723 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.723 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.723 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:41.724 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:41.724 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:41.724 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:41.724 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:41.724 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:41.724 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:41.724 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:41.724 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:41.724 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.725 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.725 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.725 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.725 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-12T22:19:41.725 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-12T22:19:41.725 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:41.725 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.725 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:41.725 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:41.726 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.726 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.726 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.726 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:41.726 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.726 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.726 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.727 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.727 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-12T22:19:41.727 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-12T22:19:41.727 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:41.727 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.727 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:41.728 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-12T22:19:41.728 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.728 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.728 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:41.728 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:41.728 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:41.728 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:41.728 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:41.728 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:41.729 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:41.729 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:41.729 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:41.729 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.729 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.729 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.729 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.729 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-12T22:19:41.730 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-12T22:19:41.730 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:41.730 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.730 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:41.730 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:41.730 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.730 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.731 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.731 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:41.731 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.731 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.731 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.731 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.732 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_destination": { 2024-05-12T22:19:41.732 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_destination", 2024-05-12T22:19:41.732 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:41.732 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.732 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:41.732 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:41.733 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.733 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.733 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.733 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Email address to send alerts to", 2024-05-12T22:19:41.733 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.734 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.734 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.734 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.734 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_from_name": { 2024-05-12T22:19:41.734 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_from_name", 2024-05-12T22:19:41.735 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:41.735 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.735 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:41.735 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "Ceph", 2024-05-12T22:19:41.735 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.736 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.736 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.736 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Email From: name", 2024-05-12T22:19:41.736 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.737 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.737 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.737 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.737 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_host": { 2024-05-12T22:19:41.737 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_host", 2024-05-12T22:19:41.738 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:41.738 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.738 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:41.738 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:41.739 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.739 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.739 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.740 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SMTP server", 2024-05-12T22:19:41.740 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.740 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.740 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.740 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.741 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_password": { 2024-05-12T22:19:41.741 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_password", 2024-05-12T22:19:41.741 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:41.741 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.742 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:41.742 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:41.743 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.743 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.743 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.743 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Password to authenticate with", 2024-05-12T22:19:41.744 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.744 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.744 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.744 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.744 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_port": { 2024-05-12T22:19:41.744 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_port", 2024-05-12T22:19:41.744 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-12T22:19:41.745 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.745 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:41.745 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "465", 2024-05-12T22:19:41.745 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.745 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.745 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.745 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SMTP port", 2024-05-12T22:19:41.745 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.745 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.746 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.746 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.746 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_sender": { 2024-05-12T22:19:41.746 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_sender", 2024-05-12T22:19:41.746 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:41.746 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.746 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:41.746 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:41.747 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.747 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.747 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.747 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SMTP envelope sender", 2024-05-12T22:19:41.747 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.747 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.747 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.748 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.748 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_ssl": { 2024-05-12T22:19:41.748 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_ssl", 2024-05-12T22:19:41.748 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:41.748 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.748 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:41.748 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-12T22:19:41.748 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.749 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.749 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.749 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use SSL to connect to SMTP server", 2024-05-12T22:19:41.749 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.749 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.749 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.749 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.750 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_user": { 2024-05-12T22:19:41.750 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_user", 2024-05-12T22:19:41.750 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:41.750 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.750 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:41.750 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:41.750 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.750 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.751 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.751 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "User to authenticate as", 2024-05-12T22:19:41.751 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.751 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.751 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.751 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:41.751 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:41.752 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.752 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-12T22:19:41.752 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "balancer", 2024-05-12T22:19:41.752 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-12T22:19:41.752 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-12T22:19:41.752 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-12T22:19:41.753 INFO:teuthology.orchestra.run.smithi028.stdout: "active": { 2024-05-12T22:19:41.753 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "active", 2024-05-12T22:19:41.753 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:41.753 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.753 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:41.753 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-12T22:19:41.753 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.753 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.753 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.754 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "automatically balance PGs across cluster", 2024-05-12T22:19:41.754 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.754 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.754 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.754 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.754 INFO:teuthology.orchestra.run.smithi028.stdout: "begin_time": { 2024-05-12T22:19:41.755 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "begin_time", 2024-05-12T22:19:41.755 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:41.755 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.755 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:41.755 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0000", 2024-05-12T22:19:41.755 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.755 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.755 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.755 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "beginning time of day to automatically balance", 2024-05-12T22:19:41.756 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-05-12T22:19:41.756 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.756 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.756 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.756 INFO:teuthology.orchestra.run.smithi028.stdout: "begin_weekday": { 2024-05-12T22:19:41.756 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "begin_weekday", 2024-05-12T22:19:41.756 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-05-12T22:19:41.756 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.757 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:41.757 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-05-12T22:19:41.757 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "0", 2024-05-12T22:19:41.757 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "6", 2024-05-12T22:19:41.757 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.757 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-05-12T22:19:41.757 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-05-12T22:19:41.757 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.757 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.758 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.758 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_compat_max_iterations": { 2024-05-12T22:19:41.758 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crush_compat_max_iterations", 2024-05-12T22:19:41.758 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-05-12T22:19:41.758 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.758 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:41.759 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "25", 2024-05-12T22:19:41.759 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-05-12T22:19:41.759 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "250", 2024-05-12T22:19:41.759 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.759 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-05-12T22:19:41.759 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.759 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.759 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.760 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.760 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_compat_metrics": { 2024-05-12T22:19:41.760 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crush_compat_metrics", 2024-05-12T22:19:41.760 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:41.760 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.761 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:41.761 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "pgs,objects,bytes", 2024-05-12T22:19:41.761 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.761 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.761 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.761 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-05-12T22:19:41.761 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-05-12T22:19:41.762 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.762 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.762 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.762 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_compat_step": { 2024-05-12T22:19:41.762 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crush_compat_step", 2024-05-12T22:19:41.762 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-05-12T22:19:41.763 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.763 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:41.763 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0.5", 2024-05-12T22:19:41.763 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "0.001", 2024-05-12T22:19:41.763 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "0.999", 2024-05-12T22:19:41.763 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.763 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "aggressiveness of optimization", 2024-05-12T22:19:41.764 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-05-12T22:19:41.764 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.764 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.764 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.764 INFO:teuthology.orchestra.run.smithi028.stdout: "end_time": { 2024-05-12T22:19:41.764 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "end_time", 2024-05-12T22:19:41.765 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:41.765 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.765 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:41.765 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "2359", 2024-05-12T22:19:41.765 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.765 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.765 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.765 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "ending time of day to automatically balance", 2024-05-12T22:19:41.766 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-05-12T22:19:41.766 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.766 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.766 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.766 INFO:teuthology.orchestra.run.smithi028.stdout: "end_weekday": { 2024-05-12T22:19:41.766 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "end_weekday", 2024-05-12T22:19:41.766 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-05-12T22:19:41.767 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.767 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:41.767 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-05-12T22:19:41.767 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "0", 2024-05-12T22:19:41.767 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "6", 2024-05-12T22:19:41.767 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.767 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-05-12T22:19:41.767 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-05-12T22:19:41.768 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.768 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.768 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.768 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-12T22:19:41.768 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-12T22:19:41.768 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:41.768 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.768 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:41.769 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:41.769 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.769 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.769 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:41.769 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:41.769 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:41.769 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:41.769 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:41.770 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:41.770 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:41.770 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:41.770 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:41.770 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.770 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.770 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.771 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.771 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-12T22:19:41.771 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-12T22:19:41.771 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:41.771 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.772 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:41.772 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:41.772 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.772 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.772 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.772 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:41.772 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.772 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.773 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.773 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.773 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-12T22:19:41.773 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-12T22:19:41.773 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:41.773 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.773 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:41.773 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-12T22:19:41.773 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.774 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.774 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:41.774 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:41.774 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:41.774 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:41.774 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:41.775 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:41.775 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:41.775 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:41.775 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:41.775 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.775 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.775 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.775 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.779 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-12T22:19:41.779 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-12T22:19:41.779 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:41.779 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.779 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:41.780 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:41.780 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.780 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.780 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.780 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:41.780 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.781 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.781 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.781 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.781 INFO:teuthology.orchestra.run.smithi028.stdout: "min_score": { 2024-05-12T22:19:41.781 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "min_score", 2024-05-12T22:19:41.781 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-05-12T22:19:41.781 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.781 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:41.781 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-05-12T22:19:41.782 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.782 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.782 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.782 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-05-12T22:19:41.782 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.782 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.782 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.783 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.783 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": { 2024-05-12T22:19:41.783 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mode", 2024-05-12T22:19:41.783 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:41.783 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.783 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:41.783 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "upmap", 2024-05-12T22:19:41.783 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.784 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.784 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:41.784 INFO:teuthology.orchestra.run.smithi028.stdout: "crush-compat", 2024-05-12T22:19:41.784 INFO:teuthology.orchestra.run.smithi028.stdout: "none", 2024-05-12T22:19:41.784 INFO:teuthology.orchestra.run.smithi028.stdout: "read", 2024-05-12T22:19:41.784 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap", 2024-05-12T22:19:41.784 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap-read" 2024-05-12T22:19:41.784 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:41.785 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Balancer mode", 2024-05-12T22:19:41.785 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.785 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.785 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.785 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.785 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_ids": { 2024-05-12T22:19:41.785 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pool_ids", 2024-05-12T22:19:41.786 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:41.786 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.786 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:41.786 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:41.786 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.786 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.787 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.787 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-05-12T22:19:41.787 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.787 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.788 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.788 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.788 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-05-12T22:19:41.788 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-05-12T22:19:41.788 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-05-12T22:19:41.788 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.788 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:41.788 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-05-12T22:19:41.789 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.789 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.789 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.789 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-05-12T22:19:41.789 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.789 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.789 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.789 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.789 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap_max_deviation": { 2024-05-12T22:19:41.789 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "upmap_max_deviation", 2024-05-12T22:19:41.790 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-12T22:19:41.790 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.790 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:41.790 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-05-12T22:19:41.790 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-05-12T22:19:41.790 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.790 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.791 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "deviation below which no optimization is attempted", 2024-05-12T22:19:41.791 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-05-12T22:19:41.791 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.791 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.791 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.791 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap_max_optimizations": { 2024-05-12T22:19:41.791 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "upmap_max_optimizations", 2024-05-12T22:19:41.791 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-05-12T22:19:41.792 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.792 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:41.792 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-05-12T22:19:41.792 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.792 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.792 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.792 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-05-12T22:19:41.792 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.792 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.793 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.793 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:41.793 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:41.793 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.793 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-12T22:19:41.793 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cephadm", 2024-05-12T22:19:41.793 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-12T22:19:41.793 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-12T22:19:41.793 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-12T22:19:41.794 INFO:teuthology.orchestra.run.smithi028.stdout: "agent_down_multiplier": { 2024-05-12T22:19:41.794 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "agent_down_multiplier", 2024-05-12T22:19:41.794 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-05-12T22:19:41.794 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.794 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:41.794 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "3.0", 2024-05-12T22:19:41.794 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.795 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.795 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.795 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-05-12T22:19:41.795 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.795 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.795 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.795 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.795 INFO:teuthology.orchestra.run.smithi028.stdout: "agent_refresh_rate": { 2024-05-12T22:19:41.795 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "agent_refresh_rate", 2024-05-12T22:19:41.796 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-05-12T22:19:41.796 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.796 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:41.796 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "20", 2024-05-12T22:19:41.796 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.796 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.797 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.797 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-05-12T22:19:41.797 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.797 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.797 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.797 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.798 INFO:teuthology.orchestra.run.smithi028.stdout: "agent_starting_port": { 2024-05-12T22:19:41.798 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "agent_starting_port", 2024-05-12T22:19:41.798 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-12T22:19:41.798 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.798 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:41.798 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "4721", 2024-05-12T22:19:41.798 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.799 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.799 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.799 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-05-12T22:19:41.799 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.799 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.799 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.799 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.800 INFO:teuthology.orchestra.run.smithi028.stdout: "allow_ptrace": { 2024-05-12T22:19:41.800 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "allow_ptrace", 2024-05-12T22:19:41.800 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:41.800 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.800 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:41.800 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:41.800 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.800 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.801 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.801 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-05-12T22:19:41.801 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-05-12T22:19:41.801 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.801 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.801 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.801 INFO:teuthology.orchestra.run.smithi028.stdout: "autotune_interval": { 2024-05-12T22:19:41.801 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "autotune_interval", 2024-05-12T22:19:41.802 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-05-12T22:19:41.802 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.802 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:41.802 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-05-12T22:19:41.802 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.802 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.803 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.803 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to autotune daemon memory", 2024-05-12T22:19:41.803 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.803 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.803 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.803 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.803 INFO:teuthology.orchestra.run.smithi028.stdout: "autotune_memory_target_ratio": { 2024-05-12T22:19:41.803 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "autotune_memory_target_ratio", 2024-05-12T22:19:41.803 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-05-12T22:19:41.804 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.804 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:41.804 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0.7", 2024-05-12T22:19:41.804 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.804 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.804 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.804 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-05-12T22:19:41.804 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.804 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.805 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.805 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.805 INFO:teuthology.orchestra.run.smithi028.stdout: "cephadm_log_destination": { 2024-05-12T22:19:41.805 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cephadm_log_destination", 2024-05-12T22:19:41.805 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:41.805 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.805 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:41.805 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:41.805 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.806 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.806 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:41.806 INFO:teuthology.orchestra.run.smithi028.stdout: "file", 2024-05-12T22:19:41.806 INFO:teuthology.orchestra.run.smithi028.stdout: "file,syslog", 2024-05-12T22:19:41.806 INFO:teuthology.orchestra.run.smithi028.stdout: "syslog" 2024-05-12T22:19:41.806 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:41.806 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-05-12T22:19:41.806 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.807 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.807 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.807 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.807 INFO:teuthology.orchestra.run.smithi028.stdout: "cgroups_split": { 2024-05-12T22:19:41.807 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cgroups_split", 2024-05-12T22:19:41.807 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:41.807 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.807 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:41.807 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-12T22:19:41.808 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.808 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.808 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.808 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-05-12T22:19:41.808 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.808 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.808 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.808 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.809 INFO:teuthology.orchestra.run.smithi028.stdout: "config_checks_enabled": { 2024-05-12T22:19:41.809 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "config_checks_enabled", 2024-05-12T22:19:41.809 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:41.809 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.809 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:41.809 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:41.809 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.809 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.810 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.810 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-05-12T22:19:41.810 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.810 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.810 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.810 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.810 INFO:teuthology.orchestra.run.smithi028.stdout: "config_dashboard": { 2024-05-12T22:19:41.810 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "config_dashboard", 2024-05-12T22:19:41.811 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:41.811 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.811 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:41.811 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-12T22:19:41.811 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.811 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.811 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.811 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-05-12T22:19:41.811 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.812 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.812 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.812 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.812 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_alertmanager": { 2024-05-12T22:19:41.812 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_alertmanager", 2024-05-12T22:19:41.812 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:41.812 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.812 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:41.812 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-05-12T22:19:41.813 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.813 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.813 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.813 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-05-12T22:19:41.813 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.813 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.813 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.814 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.814 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_base": { 2024-05-12T22:19:41.814 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_base", 2024-05-12T22:19:41.814 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:41.814 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.814 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:41.814 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/ceph", 2024-05-12T22:19:41.815 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.815 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.815 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.815 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Container image name, without the tag", 2024-05-12T22:19:41.815 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.815 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.815 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.815 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.815 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_elasticsearch": { 2024-05-12T22:19:41.816 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_elasticsearch", 2024-05-12T22:19:41.816 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:41.816 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.816 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:41.816 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-05-12T22:19:41.816 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.816 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.817 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.817 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "elasticsearch container image", 2024-05-12T22:19:41.817 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.817 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.817 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.817 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.817 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_grafana": { 2024-05-12T22:19:41.817 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_grafana", 2024-05-12T22:19:41.817 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:41.818 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.818 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:41.818 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-05-12T22:19:41.818 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.818 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.818 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.819 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-05-12T22:19:41.819 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.819 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.819 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.819 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.819 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_haproxy": { 2024-05-12T22:19:41.819 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_haproxy", 2024-05-12T22:19:41.819 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:41.819 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.820 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:41.820 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-05-12T22:19:41.820 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.820 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.820 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.820 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "HAproxy container image", 2024-05-12T22:19:41.820 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.821 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.821 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.821 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.821 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_jaeger_agent": { 2024-05-12T22:19:41.821 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_jaeger_agent", 2024-05-12T22:19:41.821 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:41.821 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.821 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:41.821 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-05-12T22:19:41.822 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.822 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.822 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.822 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Jaeger agent container image", 2024-05-12T22:19:41.822 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.825 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.825 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.825 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.825 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_jaeger_collector": { 2024-05-12T22:19:41.826 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_jaeger_collector", 2024-05-12T22:19:41.826 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:41.826 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.826 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:41.826 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-05-12T22:19:41.826 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.826 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.826 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.826 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Jaeger collector container image", 2024-05-12T22:19:41.827 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.827 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.827 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.827 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.827 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_jaeger_query": { 2024-05-12T22:19:41.827 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_jaeger_query", 2024-05-12T22:19:41.827 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:41.827 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.827 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:41.828 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-05-12T22:19:41.828 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.828 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.828 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.828 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Jaeger query container image", 2024-05-12T22:19:41.828 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.828 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.828 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.828 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.829 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_keepalived": { 2024-05-12T22:19:41.829 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_keepalived", 2024-05-12T22:19:41.829 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:41.829 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.829 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:41.829 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-05-12T22:19:41.829 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.829 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.830 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.830 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Keepalived container image", 2024-05-12T22:19:41.830 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.830 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.830 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.830 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.830 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_loki": { 2024-05-12T22:19:41.830 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_loki", 2024-05-12T22:19:41.830 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:41.831 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.831 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:41.831 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-05-12T22:19:41.831 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.831 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.831 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.831 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Loki container image", 2024-05-12T22:19:41.831 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.831 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.832 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.832 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.832 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_node_exporter": { 2024-05-12T22:19:41.832 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_node_exporter", 2024-05-12T22:19:41.832 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:41.832 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.832 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:41.832 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-05-12T22:19:41.832 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.833 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.833 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.833 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-05-12T22:19:41.833 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.833 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.833 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.833 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.833 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_nvmeof": { 2024-05-12T22:19:41.833 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_nvmeof", 2024-05-12T22:19:41.834 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:41.834 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.834 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:41.834 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-05-12T22:19:41.834 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.834 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.834 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.834 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Nvme-of container image", 2024-05-12T22:19:41.835 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.835 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.835 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.835 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.835 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_prometheus": { 2024-05-12T22:19:41.835 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_prometheus", 2024-05-12T22:19:41.835 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:41.835 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.835 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:41.836 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-05-12T22:19:41.836 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.836 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.836 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.836 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-05-12T22:19:41.836 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.836 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.836 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.836 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.837 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_promtail": { 2024-05-12T22:19:41.837 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_promtail", 2024-05-12T22:19:41.837 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:41.837 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.837 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:41.837 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-05-12T22:19:41.837 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.837 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.837 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.838 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Promtail container image", 2024-05-12T22:19:41.838 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.838 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.838 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.838 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.838 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_samba": { 2024-05-12T22:19:41.838 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_samba", 2024-05-12T22:19:41.838 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:41.839 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.839 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:41.839 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-05-12T22:19:41.839 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.839 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.839 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.839 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Samba/SMB container image", 2024-05-12T22:19:41.839 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.839 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.840 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.840 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.840 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_snmp_gateway": { 2024-05-12T22:19:41.840 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_snmp_gateway", 2024-05-12T22:19:41.840 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:41.840 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.840 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:41.840 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-05-12T22:19:41.840 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.841 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.841 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.841 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SNMP Gateway container image", 2024-05-12T22:19:41.841 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.841 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.841 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.841 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.841 INFO:teuthology.orchestra.run.smithi028.stdout: "container_init": { 2024-05-12T22:19:41.841 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_init", 2024-05-12T22:19:41.842 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:41.842 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.842 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:41.842 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-12T22:19:41.842 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.842 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.842 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.843 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Run podman/docker with `--init`", 2024-05-12T22:19:41.843 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.843 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.843 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.843 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.843 INFO:teuthology.orchestra.run.smithi028.stdout: "daemon_cache_timeout": { 2024-05-12T22:19:41.843 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "daemon_cache_timeout", 2024-05-12T22:19:41.843 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-05-12T22:19:41.843 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.844 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:41.844 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-05-12T22:19:41.844 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.844 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.844 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.844 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-05-12T22:19:41.844 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.844 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.844 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.845 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.845 INFO:teuthology.orchestra.run.smithi028.stdout: "default_cephadm_command_timeout": { 2024-05-12T22:19:41.845 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "default_cephadm_command_timeout", 2024-05-12T22:19:41.845 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-12T22:19:41.845 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.845 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:41.845 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "900", 2024-05-12T22:19:41.845 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.845 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.846 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.846 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-05-12T22:19:41.846 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.846 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.846 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.846 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.846 INFO:teuthology.orchestra.run.smithi028.stdout: "default_registry": { 2024-05-12T22:19:41.846 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "default_registry", 2024-05-12T22:19:41.847 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:41.847 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.847 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:41.847 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io", 2024-05-12T22:19:41.847 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.847 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.847 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.847 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-05-12T22:19:41.847 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.848 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.848 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.848 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.848 INFO:teuthology.orchestra.run.smithi028.stdout: "device_cache_timeout": { 2024-05-12T22:19:41.848 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "device_cache_timeout", 2024-05-12T22:19:41.848 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-05-12T22:19:41.848 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.848 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:41.848 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "1800", 2024-05-12T22:19:41.849 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.849 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.849 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.849 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "seconds to cache device inventory", 2024-05-12T22:19:41.849 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.849 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.849 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.849 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.850 INFO:teuthology.orchestra.run.smithi028.stdout: "device_enhanced_scan": { 2024-05-12T22:19:41.850 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "device_enhanced_scan", 2024-05-12T22:19:41.850 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:41.850 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.850 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:41.850 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:41.850 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.850 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.850 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.851 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use libstoragemgmt during device scans", 2024-05-12T22:19:41.851 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.851 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.851 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.851 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.851 INFO:teuthology.orchestra.run.smithi028.stdout: "facts_cache_timeout": { 2024-05-12T22:19:41.851 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "facts_cache_timeout", 2024-05-12T22:19:41.851 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-05-12T22:19:41.851 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.852 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:41.852 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-05-12T22:19:41.852 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.852 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.852 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.852 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "seconds to cache host facts data", 2024-05-12T22:19:41.852 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.852 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.852 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.853 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.853 INFO:teuthology.orchestra.run.smithi028.stdout: "grafana_dashboards_path": { 2024-05-12T22:19:41.853 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "grafana_dashboards_path", 2024-05-12T22:19:41.853 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:41.853 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.853 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:41.853 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-05-12T22:19:41.853 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.853 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.854 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.854 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-05-12T22:19:41.854 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.854 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.854 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.854 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.854 INFO:teuthology.orchestra.run.smithi028.stdout: "host_check_interval": { 2024-05-12T22:19:41.854 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "host_check_interval", 2024-05-12T22:19:41.855 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-05-12T22:19:41.855 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.855 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:41.855 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-05-12T22:19:41.855 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.855 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.855 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.855 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to perform a host check", 2024-05-12T22:19:41.855 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.856 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.856 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.856 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.856 INFO:teuthology.orchestra.run.smithi028.stdout: "hw_monitoring": { 2024-05-12T22:19:41.856 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "hw_monitoring", 2024-05-12T22:19:41.856 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:41.856 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.856 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:41.856 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:41.857 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.857 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.857 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.857 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-05-12T22:19:41.857 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.857 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.857 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.857 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.857 INFO:teuthology.orchestra.run.smithi028.stdout: "inventory_list_all": { 2024-05-12T22:19:41.858 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "inventory_list_all", 2024-05-12T22:19:41.858 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:41.858 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.858 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:41.858 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:41.858 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.858 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.858 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.859 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-05-12T22:19:41.859 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.859 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.859 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.859 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.859 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-12T22:19:41.859 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-12T22:19:41.859 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:41.859 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.860 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:41.860 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:41.860 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.860 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.860 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:41.860 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:41.860 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:41.860 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:41.860 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:41.861 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:41.861 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:41.861 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:41.861 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:41.861 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.861 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.861 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.861 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.861 INFO:teuthology.orchestra.run.smithi028.stdout: "log_refresh_metadata": { 2024-05-12T22:19:41.862 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_refresh_metadata", 2024-05-12T22:19:41.862 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:41.862 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.862 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:41.862 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:41.862 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.862 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.862 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.863 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-05-12T22:19:41.863 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.863 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.863 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.863 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.863 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-12T22:19:41.863 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-12T22:19:41.863 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:41.864 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.864 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:41.864 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-12T22:19:41.864 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.864 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.864 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.864 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-05-12T22:19:41.864 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.864 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.865 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.865 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.865 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-12T22:19:41.865 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-12T22:19:41.865 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:41.865 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.865 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:41.865 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-12T22:19:41.865 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.866 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.866 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:41.866 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:41.866 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:41.866 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:41.866 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:41.866 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:41.866 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:41.866 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:41.867 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:41.867 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.867 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.867 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.867 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.867 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-12T22:19:41.867 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-12T22:19:41.867 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:41.867 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.868 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:41.868 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:41.868 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.868 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.868 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.868 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:41.868 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.868 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.868 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.869 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.869 INFO:teuthology.orchestra.run.smithi028.stdout: "manage_etc_ceph_ceph_conf": { 2024-05-12T22:19:41.869 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-05-12T22:19:41.869 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:41.869 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.869 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:41.869 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:41.869 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.870 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.870 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.870 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-05-12T22:19:41.870 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.870 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.870 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.870 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.870 INFO:teuthology.orchestra.run.smithi028.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-05-12T22:19:41.870 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-05-12T22:19:41.871 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:41.871 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.871 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:41.871 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "*", 2024-05-12T22:19:41.871 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.871 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.871 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.871 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-05-12T22:19:41.871 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.872 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.872 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.872 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.872 INFO:teuthology.orchestra.run.smithi028.stdout: "max_count_per_host": { 2024-05-12T22:19:41.872 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_count_per_host", 2024-05-12T22:19:41.872 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-12T22:19:41.872 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.872 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:41.873 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-05-12T22:19:41.873 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.873 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.873 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.873 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "max number of daemons per service per host", 2024-05-12T22:19:41.873 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.873 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.873 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.873 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.874 INFO:teuthology.orchestra.run.smithi028.stdout: "max_osd_draining_count": { 2024-05-12T22:19:41.874 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_osd_draining_count", 2024-05-12T22:19:41.874 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-12T22:19:41.874 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.874 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:41.874 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-05-12T22:19:41.874 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.874 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.874 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.875 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-05-12T22:19:41.875 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.875 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.875 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.875 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.875 INFO:teuthology.orchestra.run.smithi028.stdout: "migration_current": { 2024-05-12T22:19:41.875 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "migration_current", 2024-05-12T22:19:41.875 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-12T22:19:41.875 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.876 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:41.876 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:41.876 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.876 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.876 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.876 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "internal - do not modify", 2024-05-12T22:19:41.876 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.876 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.877 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.877 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.877 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": { 2024-05-12T22:19:41.877 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mode", 2024-05-12T22:19:41.877 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:41.877 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.877 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:41.877 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "root", 2024-05-12T22:19:41.877 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.878 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.878 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:41.878 INFO:teuthology.orchestra.run.smithi028.stdout: "cephadm-package", 2024-05-12T22:19:41.878 INFO:teuthology.orchestra.run.smithi028.stdout: "root" 2024-05-12T22:19:41.878 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:41.878 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "mode for remote execution of cephadm", 2024-05-12T22:19:41.878 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.878 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.878 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.879 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.879 INFO:teuthology.orchestra.run.smithi028.stdout: "oob_default_addr": { 2024-05-12T22:19:41.879 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "oob_default_addr", 2024-05-12T22:19:41.879 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:41.879 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.879 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:41.879 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "169.254.1.1", 2024-05-12T22:19:41.879 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.879 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.880 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.880 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Default address for RedFish API (oob management).", 2024-05-12T22:19:41.880 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.880 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.880 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.880 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.880 INFO:teuthology.orchestra.run.smithi028.stdout: "prometheus_alerts_path": { 2024-05-12T22:19:41.880 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "prometheus_alerts_path", 2024-05-12T22:19:41.880 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:41.881 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.881 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:41.881 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-05-12T22:19:41.881 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.881 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.881 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.881 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-05-12T22:19:41.881 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.881 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.882 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.882 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.882 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_insecure": { 2024-05-12T22:19:41.882 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_insecure", 2024-05-12T22:19:41.882 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:41.882 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.882 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:41.883 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:41.883 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.883 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.883 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.883 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-05-12T22:19:41.883 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.883 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.883 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.883 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.884 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_password": { 2024-05-12T22:19:41.884 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_password", 2024-05-12T22:19:41.884 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:41.884 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.884 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:41.884 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:41.884 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.884 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.884 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.885 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-05-12T22:19:41.885 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.885 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.885 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.885 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.885 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_url": { 2024-05-12T22:19:41.885 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_url", 2024-05-12T22:19:41.885 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:41.885 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.886 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:41.886 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:41.886 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.886 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.886 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.886 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-05-12T22:19:41.886 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.886 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.887 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.887 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.887 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_username": { 2024-05-12T22:19:41.887 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_username", 2024-05-12T22:19:41.887 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:41.887 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.887 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:41.887 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:41.887 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.888 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.888 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.888 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-05-12T22:19:41.888 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.888 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.888 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.888 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.888 INFO:teuthology.orchestra.run.smithi028.stdout: "secure_monitoring_stack": { 2024-05-12T22:19:41.888 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "secure_monitoring_stack", 2024-05-12T22:19:41.889 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:41.889 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.889 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:41.889 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:41.889 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.889 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.889 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.889 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-05-12T22:19:41.889 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.890 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.890 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.890 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.890 INFO:teuthology.orchestra.run.smithi028.stdout: "service_discovery_port": { 2024-05-12T22:19:41.890 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "service_discovery_port", 2024-05-12T22:19:41.890 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-12T22:19:41.890 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.890 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:41.891 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8765", 2024-05-12T22:19:41.891 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.891 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.891 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.891 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "cephadm service discovery port", 2024-05-12T22:19:41.891 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.891 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.891 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.891 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.892 INFO:teuthology.orchestra.run.smithi028.stdout: "ssh_config_file": { 2024-05-12T22:19:41.892 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssh_config_file", 2024-05-12T22:19:41.892 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:41.892 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.892 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:41.892 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:41.892 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.892 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.892 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.893 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-05-12T22:19:41.893 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.893 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.893 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.893 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.893 INFO:teuthology.orchestra.run.smithi028.stdout: "use_agent": { 2024-05-12T22:19:41.893 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "use_agent", 2024-05-12T22:19:41.893 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:41.893 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.894 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:41.894 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:41.894 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.894 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.894 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.894 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-05-12T22:19:41.894 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.894 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.895 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.895 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.895 INFO:teuthology.orchestra.run.smithi028.stdout: "use_repo_digest": { 2024-05-12T22:19:41.895 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "use_repo_digest", 2024-05-12T22:19:41.895 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:41.895 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.895 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:41.895 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-12T22:19:41.895 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.896 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.896 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.896 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-05-12T22:19:41.896 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.896 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.896 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.896 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.896 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_on_failed_host_check": { 2024-05-12T22:19:41.896 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_on_failed_host_check", 2024-05-12T22:19:41.897 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:41.897 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.897 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:41.897 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-12T22:19:41.897 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.897 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.897 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.897 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise a health warning if the host check fails", 2024-05-12T22:19:41.897 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.898 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.898 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.898 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.898 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_on_stray_daemons": { 2024-05-12T22:19:41.898 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_on_stray_daemons", 2024-05-12T22:19:41.898 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:41.898 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.898 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:41.898 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-12T22:19:41.899 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.899 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.899 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.899 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-05-12T22:19:41.899 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.899 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.899 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.899 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.899 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_on_stray_hosts": { 2024-05-12T22:19:41.900 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_on_stray_hosts", 2024-05-12T22:19:41.900 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:41.900 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.900 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:41.900 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-12T22:19:41.900 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.900 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.900 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.901 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-05-12T22:19:41.901 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.901 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.901 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.901 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:41.901 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:41.901 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.901 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-12T22:19:41.901 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crash", 2024-05-12T22:19:41.901 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-12T22:19:41.902 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-12T22:19:41.902 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-12T22:19:41.902 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-12T22:19:41.902 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-12T22:19:41.902 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:41.902 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.902 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:41.903 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:41.903 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.903 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.903 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:41.903 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:41.903 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:41.903 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:41.903 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:41.903 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:41.904 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:41.904 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:41.904 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:41.904 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.904 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.904 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.904 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.904 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-12T22:19:41.904 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-12T22:19:41.905 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:41.905 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.905 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:41.905 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:41.905 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.905 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.905 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.905 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:41.905 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.906 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.906 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.906 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.906 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-12T22:19:41.906 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-12T22:19:41.906 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:41.906 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.906 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:41.907 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-12T22:19:41.907 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.907 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.907 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:41.907 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:41.907 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:41.907 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:41.907 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:41.907 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:41.908 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:41.908 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:41.908 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:41.908 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.908 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.908 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.908 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.908 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-12T22:19:41.908 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-12T22:19:41.909 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:41.909 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.909 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:41.909 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:41.909 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.909 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.909 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.909 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:41.909 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.910 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.910 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.910 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.910 INFO:teuthology.orchestra.run.smithi028.stdout: "retain_interval": { 2024-05-12T22:19:41.910 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "retain_interval", 2024-05-12T22:19:41.910 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-05-12T22:19:41.910 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.910 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:41.911 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "31536000", 2024-05-12T22:19:41.911 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.911 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.911 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.911 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how long to retain crashes before pruning them", 2024-05-12T22:19:41.911 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.911 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.911 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.911 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.912 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_recent_interval": { 2024-05-12T22:19:41.912 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_recent_interval", 2024-05-12T22:19:41.912 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-05-12T22:19:41.912 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.912 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:41.912 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "1209600", 2024-05-12T22:19:41.912 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.912 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.912 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.913 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "time interval in which to warn about recent crashes", 2024-05-12T22:19:41.913 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.913 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.913 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.913 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:41.913 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:41.913 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.913 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-12T22:19:41.913 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "dashboard", 2024-05-12T22:19:41.914 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-12T22:19:41.914 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-12T22:19:41.914 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-12T22:19:41.914 INFO:teuthology.orchestra.run.smithi028.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-05-12T22:19:41.914 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-05-12T22:19:41.914 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-12T22:19:41.914 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.914 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:41.914 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-05-12T22:19:41.915 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.915 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.915 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.915 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:41.915 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.915 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.915 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.915 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.916 INFO:teuthology.orchestra.run.smithi028.stdout: "ALERTMANAGER_API_HOST": { 2024-05-12T22:19:41.916 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ALERTMANAGER_API_HOST", 2024-05-12T22:19:41.916 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:41.916 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.916 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:41.916 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:41.916 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.916 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.916 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.917 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:41.917 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.917 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.917 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.917 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.917 INFO:teuthology.orchestra.run.smithi028.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-05-12T22:19:41.917 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-05-12T22:19:41.917 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:41.917 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.918 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:41.918 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-12T22:19:41.918 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.918 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.918 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.918 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:41.918 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.918 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.919 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.919 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.919 INFO:teuthology.orchestra.run.smithi028.stdout: "AUDIT_API_ENABLED": { 2024-05-12T22:19:41.919 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "AUDIT_API_ENABLED", 2024-05-12T22:19:41.919 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:41.919 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.919 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:41.919 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:41.919 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.920 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.920 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.920 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:41.920 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.920 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.920 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.920 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.920 INFO:teuthology.orchestra.run.smithi028.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-05-12T22:19:41.920 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-05-12T22:19:41.921 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:41.921 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.921 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:41.921 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-12T22:19:41.921 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.921 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.921 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.921 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:41.922 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.922 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.922 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.922 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.922 INFO:teuthology.orchestra.run.smithi028.stdout: "ENABLE_BROWSABLE_API": { 2024-05-12T22:19:41.922 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ENABLE_BROWSABLE_API", 2024-05-12T22:19:41.922 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:41.922 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.922 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:41.923 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-12T22:19:41.923 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.923 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.923 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.923 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:41.923 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.923 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.923 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.924 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.924 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-05-12T22:19:41.924 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-05-12T22:19:41.924 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:41.924 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.924 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:41.924 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-12T22:19:41.924 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.925 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.925 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.925 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:41.925 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.925 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.925 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.925 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.925 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-05-12T22:19:41.925 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-05-12T22:19:41.926 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:41.926 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.926 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:41.926 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-12T22:19:41.926 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.926 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.926 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.926 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:41.927 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.927 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.927 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.927 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.927 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-05-12T22:19:41.927 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-05-12T22:19:41.927 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:41.927 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.927 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:41.928 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-12T22:19:41.928 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.928 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.928 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.928 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:41.928 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.928 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.928 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.928 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.929 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-05-12T22:19:41.929 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-05-12T22:19:41.929 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:41.929 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.929 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:41.929 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-12T22:19:41.929 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.929 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.929 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.930 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:41.930 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.930 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.930 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.930 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.930 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_NFS": { 2024-05-12T22:19:41.930 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-05-12T22:19:41.930 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:41.930 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.931 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:41.931 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-12T22:19:41.931 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.931 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.931 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.931 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:41.931 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.931 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.931 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.932 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.932 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_RBD": { 2024-05-12T22:19:41.932 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-05-12T22:19:41.932 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:41.932 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.932 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:41.932 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-12T22:19:41.932 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.932 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.933 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.933 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:41.933 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.933 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.933 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.933 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.933 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_RGW": { 2024-05-12T22:19:41.933 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-05-12T22:19:41.934 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:41.934 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.934 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:41.934 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-12T22:19:41.934 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.934 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.934 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.934 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:41.934 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.935 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.935 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.935 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.935 INFO:teuthology.orchestra.run.smithi028.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-05-12T22:19:41.935 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-05-12T22:19:41.935 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:41.935 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.935 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:41.935 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:41.936 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.936 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.936 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.936 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:41.936 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.936 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.936 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.936 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.936 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_PASSWORD": { 2024-05-12T22:19:41.937 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_PASSWORD", 2024-05-12T22:19:41.937 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:41.937 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.937 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:41.937 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "admin", 2024-05-12T22:19:41.937 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.937 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.937 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.937 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:41.938 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.938 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.938 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.938 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.938 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-05-12T22:19:41.938 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-05-12T22:19:41.938 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:41.938 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.939 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:41.939 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-12T22:19:41.939 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.939 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.939 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.939 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:41.939 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.939 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.939 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.940 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.940 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_URL": { 2024-05-12T22:19:41.940 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_URL", 2024-05-12T22:19:41.940 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:41.940 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.940 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:41.940 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:41.940 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.940 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.941 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.941 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:41.941 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.941 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.941 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.941 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.941 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_USERNAME": { 2024-05-12T22:19:41.941 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_USERNAME", 2024-05-12T22:19:41.941 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:41.942 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.942 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:41.942 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "admin", 2024-05-12T22:19:41.942 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.942 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.942 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.942 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:41.942 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.942 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.943 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.943 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.943 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-05-12T22:19:41.943 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-05-12T22:19:41.943 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:41.943 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.943 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:41.943 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:41.943 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.944 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.944 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.944 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:41.944 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.944 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.944 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.944 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.944 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-05-12T22:19:41.945 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-05-12T22:19:41.945 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:41.945 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.945 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:41.945 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:41.945 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.945 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.945 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.946 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:41.946 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.946 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.946 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.946 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.946 INFO:teuthology.orchestra.run.smithi028.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-05-12T22:19:41.946 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-05-12T22:19:41.946 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:41.947 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.947 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:41.947 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-12T22:19:41.947 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.947 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.947 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.947 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:41.947 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.947 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.948 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.948 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.948 INFO:teuthology.orchestra.run.smithi028.stdout: "ISSUE_TRACKER_API_KEY": { 2024-05-12T22:19:41.948 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-05-12T22:19:41.948 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:41.948 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.948 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:41.948 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:41.948 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.949 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.949 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.949 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:41.949 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.949 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.949 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.949 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.949 INFO:teuthology.orchestra.run.smithi028.stdout: "PROMETHEUS_API_HOST": { 2024-05-12T22:19:41.949 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PROMETHEUS_API_HOST", 2024-05-12T22:19:41.950 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:41.950 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.950 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:41.950 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:41.950 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.950 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.950 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.950 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:41.951 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.951 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.951 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.951 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.951 INFO:teuthology.orchestra.run.smithi028.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-05-12T22:19:41.951 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-05-12T22:19:41.951 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:41.951 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.951 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:41.952 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-12T22:19:41.952 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.952 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.952 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.952 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:41.952 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.952 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.952 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.952 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.953 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-05-12T22:19:41.953 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-05-12T22:19:41.953 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:41.953 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.953 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:41.953 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:41.953 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.953 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.954 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.954 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:41.954 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.954 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.954 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.954 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.954 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-05-12T22:19:41.954 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-05-12T22:19:41.954 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:41.955 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.955 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:41.955 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:41.955 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.955 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.955 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.955 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:41.955 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.955 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.956 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.956 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.956 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-05-12T22:19:41.956 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-05-12T22:19:41.956 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:41.956 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.956 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:41.956 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-12T22:19:41.956 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.957 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.957 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.957 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:41.957 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.957 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.957 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.957 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.957 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-05-12T22:19:41.958 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-05-12T22:19:41.958 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:41.958 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.958 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:41.958 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-12T22:19:41.958 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.958 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.958 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.958 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:41.959 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.959 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.959 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.959 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.959 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-05-12T22:19:41.959 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-05-12T22:19:41.959 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:41.959 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.959 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:41.960 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:41.960 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.960 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.960 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.960 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:41.960 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.960 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.960 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.961 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.961 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-05-12T22:19:41.961 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-05-12T22:19:41.961 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:41.961 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.961 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:41.961 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:41.961 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.961 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.962 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.962 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:41.962 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.962 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.962 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.962 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.962 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-05-12T22:19:41.962 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-05-12T22:19:41.962 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:41.963 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.963 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:41.963 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:41.963 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.963 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.963 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.963 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:41.963 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.963 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.964 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.964 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.964 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_ENABLED": { 2024-05-12T22:19:41.964 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_ENABLED", 2024-05-12T22:19:41.964 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:41.964 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.964 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:41.964 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-12T22:19:41.965 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.965 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.965 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.965 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:41.965 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.965 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.965 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.965 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.966 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-05-12T22:19:41.966 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-05-12T22:19:41.966 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:41.966 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.966 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:41.966 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-05-12T22:19:41.966 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.966 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.967 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.967 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:41.967 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.967 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.967 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.967 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.967 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-05-12T22:19:41.967 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-05-12T22:19:41.967 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-12T22:19:41.968 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.968 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:41.968 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-05-12T22:19:41.968 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.968 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.968 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.968 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:41.968 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.968 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.969 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.969 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.969 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-05-12T22:19:41.969 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-05-12T22:19:41.969 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-12T22:19:41.969 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.969 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:41.969 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8", 2024-05-12T22:19:41.969 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.970 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.970 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.970 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:41.970 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.970 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.970 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.970 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.970 INFO:teuthology.orchestra.run.smithi028.stdout: "REST_REQUESTS_TIMEOUT": { 2024-05-12T22:19:41.971 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-05-12T22:19:41.971 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-12T22:19:41.971 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.971 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:41.971 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "45", 2024-05-12T22:19:41.971 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.971 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.971 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.971 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:41.972 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.972 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.972 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.972 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.972 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_ACCESS_KEY": { 2024-05-12T22:19:41.972 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_ACCESS_KEY", 2024-05-12T22:19:41.972 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:41.972 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.972 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:41.973 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:41.973 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.973 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.973 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.973 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:41.973 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.973 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.973 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.973 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.974 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-05-12T22:19:41.974 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-05-12T22:19:41.974 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:41.974 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.974 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:41.974 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "admin", 2024-05-12T22:19:41.974 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.974 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.974 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.975 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:41.975 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.975 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.975 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.975 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.975 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_SECRET_KEY": { 2024-05-12T22:19:41.975 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_SECRET_KEY", 2024-05-12T22:19:41.975 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:41.975 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.976 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:41.976 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:41.976 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.976 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.976 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.976 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:41.976 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.976 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.976 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.977 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.977 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_SSL_VERIFY": { 2024-05-12T22:19:41.977 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_SSL_VERIFY", 2024-05-12T22:19:41.977 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:41.977 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.977 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:41.977 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-12T22:19:41.977 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.978 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.978 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.978 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:41.978 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.978 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.978 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.978 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.978 INFO:teuthology.orchestra.run.smithi028.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-05-12T22:19:41.978 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-05-12T22:19:41.979 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-12T22:19:41.979 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.979 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:41.979 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-05-12T22:19:41.979 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.979 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.979 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.979 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:41.979 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.980 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.980 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.980 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.980 INFO:teuthology.orchestra.run.smithi028.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-05-12T22:19:41.980 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-05-12T22:19:41.980 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-12T22:19:41.980 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.980 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:41.981 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-05-12T22:19:41.981 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.981 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.981 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.981 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:41.981 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.981 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.981 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.981 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.982 INFO:teuthology.orchestra.run.smithi028.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-05-12T22:19:41.982 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-05-12T22:19:41.982 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-12T22:19:41.982 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.982 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:41.982 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-05-12T22:19:41.982 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.982 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.982 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.983 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:41.983 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.983 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.983 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.983 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.983 INFO:teuthology.orchestra.run.smithi028.stdout: "cross_origin_url": { 2024-05-12T22:19:41.983 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cross_origin_url", 2024-05-12T22:19:41.983 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:41.983 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.984 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:41.984 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:41.984 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.984 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.984 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.984 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:41.984 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.984 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.985 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.985 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.985 INFO:teuthology.orchestra.run.smithi028.stdout: "crt_file": { 2024-05-12T22:19:41.985 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crt_file", 2024-05-12T22:19:41.985 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:41.985 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.985 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:41.985 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:41.985 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.985 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.986 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.986 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:41.986 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.986 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.986 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.986 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.986 INFO:teuthology.orchestra.run.smithi028.stdout: "debug": { 2024-05-12T22:19:41.986 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "debug", 2024-05-12T22:19:41.987 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:41.987 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.987 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:41.987 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:41.987 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.987 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.987 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.987 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Enable/disable debug options", 2024-05-12T22:19:41.988 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.988 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.988 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.988 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.988 INFO:teuthology.orchestra.run.smithi028.stdout: "jwt_token_ttl": { 2024-05-12T22:19:41.988 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "jwt_token_ttl", 2024-05-12T22:19:41.988 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-12T22:19:41.988 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.988 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:41.989 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "28800", 2024-05-12T22:19:41.989 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.989 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.989 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.989 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:41.989 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.989 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.989 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.989 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.990 INFO:teuthology.orchestra.run.smithi028.stdout: "key_file": { 2024-05-12T22:19:41.990 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "key_file", 2024-05-12T22:19:41.990 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:41.990 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.990 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:41.990 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:41.990 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.990 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.990 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.991 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:41.991 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.991 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.991 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.991 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.991 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-12T22:19:41.991 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-12T22:19:41.991 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:41.992 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.992 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:41.992 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:41.992 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.992 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.992 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:41.992 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:41.992 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:41.992 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:41.993 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:41.993 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:41.993 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:41.993 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:41.993 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:41.993 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.993 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.993 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.993 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.994 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-12T22:19:41.994 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-12T22:19:41.994 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:41.994 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.994 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:41.994 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:41.994 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.994 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.994 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.995 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:41.995 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.995 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.995 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.995 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.995 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-12T22:19:41.995 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-12T22:19:41.995 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:41.995 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.996 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:41.996 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-12T22:19:41.996 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.996 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.996 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:41.996 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:41.996 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:41.996 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:41.996 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:41.997 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:41.997 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:41.997 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:41.997 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:41.997 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.997 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.997 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.997 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.997 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-12T22:19:41.998 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-12T22:19:41.998 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:41.998 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:41.998 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:41.998 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:41.998 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:41.998 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:41.998 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:41.999 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:41.999 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:41.999 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:41.999 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:41.999 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:41.999 INFO:teuthology.orchestra.run.smithi028.stdout: "motd": { 2024-05-12T22:19:41.999 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "motd", 2024-05-12T22:19:41.999 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:41.999 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.000 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.000 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.000 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.000 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.000 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.000 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "The message of the day", 2024-05-12T22:19:42.000 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.000 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.000 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.001 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.001 INFO:teuthology.orchestra.run.smithi028.stdout: "redirect_resolve_ip_addr": { 2024-05-12T22:19:42.001 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "redirect_resolve_ip_addr", 2024-05-12T22:19:42.001 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.001 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.001 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.001 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.001 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.001 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.002 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.002 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.002 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.002 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.002 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.002 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.002 INFO:teuthology.orchestra.run.smithi028.stdout: "server_addr": { 2024-05-12T22:19:42.002 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_addr", 2024-05-12T22:19:42.003 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.003 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.003 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.003 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "::", 2024-05-12T22:19:42.003 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.003 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.003 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.003 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.003 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.004 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.004 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.004 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.004 INFO:teuthology.orchestra.run.smithi028.stdout: "server_port": { 2024-05-12T22:19:42.004 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_port", 2024-05-12T22:19:42.004 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-12T22:19:42.004 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.004 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.004 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8080", 2024-05-12T22:19:42.005 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.005 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.005 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.005 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.005 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.005 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.005 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.005 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.005 INFO:teuthology.orchestra.run.smithi028.stdout: "ssl": { 2024-05-12T22:19:42.006 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssl", 2024-05-12T22:19:42.006 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.006 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.006 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.006 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-12T22:19:42.006 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.006 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.006 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.006 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.007 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.007 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.007 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.007 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.007 INFO:teuthology.orchestra.run.smithi028.stdout: "ssl_server_port": { 2024-05-12T22:19:42.007 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssl_server_port", 2024-05-12T22:19:42.007 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-12T22:19:42.007 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.007 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.008 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8443", 2024-05-12T22:19:42.008 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.008 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.008 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.008 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.008 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.008 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.008 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.009 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.009 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_behaviour": { 2024-05-12T22:19:42.009 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_behaviour", 2024-05-12T22:19:42.009 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.009 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.009 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.009 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "redirect", 2024-05-12T22:19:42.009 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.009 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.010 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:42.010 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:42.010 INFO:teuthology.orchestra.run.smithi028.stdout: "redirect" 2024-05-12T22:19:42.010 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:42.010 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.010 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.010 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.010 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.011 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.011 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_error_status_code": { 2024-05-12T22:19:42.011 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_error_status_code", 2024-05-12T22:19:42.011 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-12T22:19:42.011 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.011 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.011 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "500", 2024-05-12T22:19:42.011 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "400", 2024-05-12T22:19:42.011 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "599", 2024-05-12T22:19:42.012 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.012 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.012 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.012 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.012 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.012 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.012 INFO:teuthology.orchestra.run.smithi028.stdout: "url_prefix": { 2024-05-12T22:19:42.012 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "url_prefix", 2024-05-12T22:19:42.012 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.013 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.013 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.013 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.013 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.013 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.013 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.013 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.013 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.013 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.014 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.014 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:42.014 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:42.014 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.014 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-12T22:19:42.014 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "devicehealth", 2024-05-12T22:19:42.014 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-12T22:19:42.014 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-12T22:19:42.015 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-12T22:19:42.015 INFO:teuthology.orchestra.run.smithi028.stdout: "enable_monitoring": { 2024-05-12T22:19:42.015 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enable_monitoring", 2024-05-12T22:19:42.015 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.015 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.015 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.015 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-12T22:19:42.015 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.015 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.016 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.016 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "monitor device health metrics", 2024-05-12T22:19:42.016 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.016 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.016 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.016 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.016 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-12T22:19:42.016 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-12T22:19:42.016 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.017 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.017 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.017 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.017 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.017 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.017 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:42.017 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:42.017 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:42.017 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:42.018 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:42.018 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:42.018 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:42.018 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:42.018 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.018 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.018 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.018 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.019 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.019 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-12T22:19:42.019 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-12T22:19:42.019 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.019 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.019 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.019 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.019 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.019 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.020 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.020 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.020 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.020 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.020 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.020 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.020 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-12T22:19:42.020 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-12T22:19:42.020 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.021 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.021 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.021 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-12T22:19:42.021 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.021 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.021 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:42.021 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:42.021 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:42.021 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:42.022 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:42.022 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:42.022 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:42.022 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:42.022 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.022 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.022 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.022 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.022 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.023 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-12T22:19:42.023 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-12T22:19:42.023 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.023 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.023 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.023 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.023 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.023 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.023 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.024 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.024 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.024 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.024 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.024 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.024 INFO:teuthology.orchestra.run.smithi028.stdout: "mark_out_threshold": { 2024-05-12T22:19:42.024 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mark_out_threshold", 2024-05-12T22:19:42.024 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-05-12T22:19:42.024 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.025 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.025 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "2419200", 2024-05-12T22:19:42.025 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.025 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.025 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.025 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-05-12T22:19:42.025 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.025 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.025 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.026 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.026 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": { 2024-05-12T22:19:42.026 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pool_name", 2024-05-12T22:19:42.026 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.026 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.026 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.026 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "device_health_metrics", 2024-05-12T22:19:42.026 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.027 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.027 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.027 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "name of pool in which to store device health metrics", 2024-05-12T22:19:42.027 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.027 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.027 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.027 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.027 INFO:teuthology.orchestra.run.smithi028.stdout: "retention_period": { 2024-05-12T22:19:42.027 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "retention_period", 2024-05-12T22:19:42.028 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-05-12T22:19:42.028 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.028 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.028 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "15552000", 2024-05-12T22:19:42.028 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.028 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.028 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.028 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how long to retain device health metrics", 2024-05-12T22:19:42.028 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.029 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.029 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.029 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.029 INFO:teuthology.orchestra.run.smithi028.stdout: "scrape_frequency": { 2024-05-12T22:19:42.029 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "scrape_frequency", 2024-05-12T22:19:42.029 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-05-12T22:19:42.029 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.029 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.029 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "86400", 2024-05-12T22:19:42.030 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.030 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.030 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.030 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to scrape device health metrics", 2024-05-12T22:19:42.030 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.030 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.030 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.030 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.031 INFO:teuthology.orchestra.run.smithi028.stdout: "self_heal": { 2024-05-12T22:19:42.031 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "self_heal", 2024-05-12T22:19:42.031 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.031 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.031 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.031 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-12T22:19:42.031 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.031 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.032 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.032 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-05-12T22:19:42.032 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.032 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.032 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.032 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.032 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-05-12T22:19:42.032 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-05-12T22:19:42.032 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-05-12T22:19:42.033 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.033 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.033 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-05-12T22:19:42.033 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.033 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.033 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.033 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to wake up and check device health", 2024-05-12T22:19:42.033 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.033 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.034 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.034 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.034 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_threshold": { 2024-05-12T22:19:42.034 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_threshold", 2024-05-12T22:19:42.034 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-05-12T22:19:42.034 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.034 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.034 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "7257600", 2024-05-12T22:19:42.034 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.035 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.035 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.035 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-05-12T22:19:42.035 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.035 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.035 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.035 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:42.035 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:42.036 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.036 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-12T22:19:42.036 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "diskprediction_local", 2024-05-12T22:19:42.036 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-12T22:19:42.036 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-12T22:19:42.036 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-12T22:19:42.036 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-12T22:19:42.036 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-12T22:19:42.036 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.037 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.037 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.037 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.037 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.037 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.037 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:42.037 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:42.037 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:42.037 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:42.038 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:42.038 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:42.038 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:42.038 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:42.038 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.038 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.038 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.038 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.038 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.039 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-12T22:19:42.039 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-12T22:19:42.039 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.039 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.039 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.039 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.039 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.039 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.039 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.040 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.040 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.040 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.040 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.040 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.040 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-12T22:19:42.040 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-12T22:19:42.040 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.040 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.041 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.041 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-12T22:19:42.041 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.041 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.041 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:42.041 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:42.041 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:42.041 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:42.041 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:42.042 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:42.042 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:42.042 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:42.042 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.042 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.042 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.042 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.042 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.043 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-12T22:19:42.043 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-12T22:19:42.043 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.043 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.043 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.043 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.043 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.043 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.043 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.044 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.044 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.044 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.044 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.044 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.044 INFO:teuthology.orchestra.run.smithi028.stdout: "predict_interval": { 2024-05-12T22:19:42.044 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "predict_interval", 2024-05-12T22:19:42.044 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.044 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.045 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.045 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "86400", 2024-05-12T22:19:42.045 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.045 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.045 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.045 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.045 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.045 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.045 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.046 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.046 INFO:teuthology.orchestra.run.smithi028.stdout: "predictor_model": { 2024-05-12T22:19:42.046 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "predictor_model", 2024-05-12T22:19:42.046 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.046 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.046 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.046 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "prophetstor", 2024-05-12T22:19:42.046 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.047 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.047 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.047 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.047 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.047 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.047 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.047 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.047 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-05-12T22:19:42.047 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-05-12T22:19:42.048 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.048 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.048 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.048 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-05-12T22:19:42.048 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.048 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.048 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.048 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.048 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.049 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.049 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.049 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:42.049 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:42.049 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.049 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-12T22:19:42.049 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "influx", 2024-05-12T22:19:42.049 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": false, 2024-05-12T22:19:42.050 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "influxdb python module not found", 2024-05-12T22:19:42.050 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-12T22:19:42.050 INFO:teuthology.orchestra.run.smithi028.stdout: "batch_size": { 2024-05-12T22:19:42.050 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "batch_size", 2024-05-12T22:19:42.050 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-12T22:19:42.050 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.050 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.050 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5000", 2024-05-12T22:19:42.050 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.051 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.051 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.051 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-05-12T22:19:42.051 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.051 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.051 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.051 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.051 INFO:teuthology.orchestra.run.smithi028.stdout: "database": { 2024-05-12T22:19:42.052 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "database", 2024-05-12T22:19:42.052 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.052 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.052 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.052 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "ceph", 2024-05-12T22:19:42.052 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.052 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.052 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.053 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-05-12T22:19:42.053 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.053 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.053 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.053 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.053 INFO:teuthology.orchestra.run.smithi028.stdout: "hostname": { 2024-05-12T22:19:42.053 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "hostname", 2024-05-12T22:19:42.053 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.053 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.054 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.054 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.054 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.054 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.054 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.054 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "InfluxDB server hostname", 2024-05-12T22:19:42.054 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.054 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.055 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.055 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.055 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-05-12T22:19:42.055 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-05-12T22:19:42.055 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-05-12T22:19:42.055 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.055 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.055 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "30", 2024-05-12T22:19:42.055 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "5", 2024-05-12T22:19:42.056 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.056 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.056 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-05-12T22:19:42.056 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.056 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.056 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.056 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.056 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-12T22:19:42.056 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-12T22:19:42.057 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.057 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.057 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.057 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.057 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.057 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.057 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:42.057 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:42.057 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:42.058 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:42.058 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:42.058 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:42.058 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:42.058 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:42.058 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.058 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.058 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.058 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.059 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.059 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-12T22:19:42.059 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-12T22:19:42.059 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.059 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.059 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.059 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.059 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.059 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.060 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.060 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.060 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.060 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.060 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.060 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.060 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-12T22:19:42.060 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-12T22:19:42.060 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.061 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.061 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.061 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-12T22:19:42.061 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.061 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.061 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:42.061 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:42.061 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:42.061 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:42.062 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:42.062 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:42.062 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:42.062 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:42.062 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.062 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.062 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.062 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.063 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.063 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-12T22:19:42.063 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-12T22:19:42.063 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.063 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.063 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.063 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.063 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.063 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.064 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.064 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.064 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.064 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.064 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.064 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.064 INFO:teuthology.orchestra.run.smithi028.stdout: "password": { 2024-05-12T22:19:42.064 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "password", 2024-05-12T22:19:42.064 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.065 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.065 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.065 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.065 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.065 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.065 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.065 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "password of InfluxDB server user", 2024-05-12T22:19:42.065 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.065 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.066 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.066 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.066 INFO:teuthology.orchestra.run.smithi028.stdout: "port": { 2024-05-12T22:19:42.066 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "port", 2024-05-12T22:19:42.066 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-12T22:19:42.066 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.066 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.066 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8086", 2024-05-12T22:19:42.066 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.067 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.067 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.067 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "InfluxDB server port", 2024-05-12T22:19:42.067 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.067 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.067 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.067 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.067 INFO:teuthology.orchestra.run.smithi028.stdout: "ssl": { 2024-05-12T22:19:42.067 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssl", 2024-05-12T22:19:42.068 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.068 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.068 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.068 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "false", 2024-05-12T22:19:42.068 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.068 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.068 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.068 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-05-12T22:19:42.069 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.069 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.069 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.069 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.069 INFO:teuthology.orchestra.run.smithi028.stdout: "threads": { 2024-05-12T22:19:42.069 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "threads", 2024-05-12T22:19:42.069 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-12T22:19:42.069 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.069 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.070 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-05-12T22:19:42.070 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-05-12T22:19:42.070 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "32", 2024-05-12T22:19:42.070 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.070 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-05-12T22:19:42.070 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.070 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.070 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.071 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.071 INFO:teuthology.orchestra.run.smithi028.stdout: "username": { 2024-05-12T22:19:42.071 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "username", 2024-05-12T22:19:42.071 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.071 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.071 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.071 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.071 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.071 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.072 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.072 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "username of InfluxDB server user", 2024-05-12T22:19:42.072 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.072 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.072 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.072 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.072 INFO:teuthology.orchestra.run.smithi028.stdout: "verify_ssl": { 2024-05-12T22:19:42.072 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "verify_ssl", 2024-05-12T22:19:42.073 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.073 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.073 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.073 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "true", 2024-05-12T22:19:42.073 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.073 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.073 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.073 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-05-12T22:19:42.074 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.074 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.074 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.074 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:42.074 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:42.074 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.074 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-12T22:19:42.074 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "insights", 2024-05-12T22:19:42.075 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-12T22:19:42.075 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-12T22:19:42.075 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-12T22:19:42.075 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-12T22:19:42.075 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-12T22:19:42.075 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.075 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.075 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.075 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.076 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.076 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.076 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:42.076 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:42.076 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:42.076 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:42.076 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:42.076 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:42.077 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:42.077 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:42.077 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.077 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.077 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.077 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.077 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.077 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-12T22:19:42.077 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-12T22:19:42.078 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.078 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.078 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.078 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.078 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.078 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.078 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.078 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.079 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.079 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.079 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.079 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.079 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-12T22:19:42.079 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-12T22:19:42.079 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.079 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.079 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.080 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-12T22:19:42.080 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.080 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.080 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:42.080 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:42.080 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:42.080 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:42.080 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:42.081 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:42.081 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:42.081 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:42.081 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.081 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.081 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.081 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.081 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.081 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-12T22:19:42.082 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-12T22:19:42.082 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.082 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.082 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.082 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.082 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.082 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.082 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.082 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.083 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.083 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.083 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.083 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:42.083 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:42.083 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.083 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-12T22:19:42.083 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "iostat", 2024-05-12T22:19:42.083 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-12T22:19:42.084 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-12T22:19:42.084 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-12T22:19:42.084 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-12T22:19:42.084 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-12T22:19:42.084 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.084 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.084 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.084 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.085 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.085 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.085 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:42.085 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:42.085 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:42.085 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:42.085 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:42.085 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:42.085 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:42.086 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:42.086 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.086 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.086 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.086 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.086 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.086 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-12T22:19:42.086 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-12T22:19:42.086 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.087 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.087 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.087 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.087 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.087 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.087 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.087 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.087 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.087 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.088 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.088 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.088 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-12T22:19:42.088 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-12T22:19:42.088 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.088 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.088 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.088 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-12T22:19:42.088 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.089 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.089 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:42.089 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:42.089 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:42.089 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:42.089 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:42.089 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:42.089 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:42.090 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:42.090 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.090 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.090 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.090 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.090 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.090 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-12T22:19:42.090 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-12T22:19:42.090 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.091 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.091 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.091 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.091 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.091 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.091 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.091 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.091 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.091 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.092 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.092 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:42.092 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:42.092 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.092 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-12T22:19:42.092 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "k8sevents", 2024-05-12T22:19:42.092 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-12T22:19:42.092 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-12T22:19:42.092 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-12T22:19:42.093 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_event_retention_days": { 2024-05-12T22:19:42.093 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ceph_event_retention_days", 2024-05-12T22:19:42.093 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-12T22:19:42.093 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.093 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.093 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "7", 2024-05-12T22:19:42.093 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.093 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.093 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.094 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Days to hold ceph event information within local cache", 2024-05-12T22:19:42.094 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.094 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.094 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.094 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.094 INFO:teuthology.orchestra.run.smithi028.stdout: "config_check_secs": { 2024-05-12T22:19:42.094 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "config_check_secs", 2024-05-12T22:19:42.094 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-12T22:19:42.095 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.095 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.095 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-05-12T22:19:42.095 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "10", 2024-05-12T22:19:42.095 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.095 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.095 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-05-12T22:19:42.095 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.096 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.096 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.096 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.096 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-12T22:19:42.096 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-12T22:19:42.096 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.096 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.096 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.096 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.097 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.097 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.097 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:42.097 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:42.097 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:42.097 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:42.097 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:42.097 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:42.098 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:42.098 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:42.098 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.098 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.098 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.098 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.098 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.098 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-12T22:19:42.098 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-12T22:19:42.099 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.099 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.099 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.099 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.099 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.099 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.099 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.099 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.099 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.100 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.100 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.100 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.100 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-12T22:19:42.100 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-12T22:19:42.100 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.100 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.100 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.101 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-12T22:19:42.101 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.101 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.101 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:42.101 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:42.101 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:42.101 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:42.101 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:42.101 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:42.102 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:42.102 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:42.102 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.102 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.102 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.102 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.102 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.102 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-12T22:19:42.103 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-12T22:19:42.103 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.103 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.103 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.103 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.103 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.103 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.103 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.103 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.104 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.104 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.104 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.104 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:42.104 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:42.104 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.104 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-12T22:19:42.104 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "localpool", 2024-05-12T22:19:42.104 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-12T22:19:42.105 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-12T22:19:42.105 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-12T22:19:42.105 INFO:teuthology.orchestra.run.smithi028.stdout: "failure_domain": { 2024-05-12T22:19:42.105 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "failure_domain", 2024-05-12T22:19:42.105 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.105 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.105 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.105 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "host", 2024-05-12T22:19:42.105 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.106 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.106 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.106 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "failure domain for any created local pool", 2024-05-12T22:19:42.106 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-05-12T22:19:42.106 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.106 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.106 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.106 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-12T22:19:42.107 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-12T22:19:42.107 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.107 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.107 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.107 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.107 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.107 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.107 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:42.107 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:42.108 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:42.108 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:42.108 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:42.108 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:42.108 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:42.108 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:42.108 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.108 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.108 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.109 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.109 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.109 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-12T22:19:42.109 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-12T22:19:42.109 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.109 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.109 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.109 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.110 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.110 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.110 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.110 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.110 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.110 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.110 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.110 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.110 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-12T22:19:42.111 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-12T22:19:42.111 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.111 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.111 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.111 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-12T22:19:42.111 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.111 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.111 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:42.111 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:42.112 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:42.112 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:42.112 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:42.112 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:42.112 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:42.112 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:42.112 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.112 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.113 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.113 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.113 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.113 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-12T22:19:42.113 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-12T22:19:42.113 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.113 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.113 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.113 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.114 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.114 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.114 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.114 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.114 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.114 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.114 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.114 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.115 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": { 2024-05-12T22:19:42.115 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "min_size", 2024-05-12T22:19:42.115 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-12T22:19:42.115 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.115 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.115 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.115 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.115 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.115 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.116 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "default min_size for any created local pool", 2024-05-12T22:19:42.116 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-05-12T22:19:42.116 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.116 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.116 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.116 INFO:teuthology.orchestra.run.smithi028.stdout: "num_rep": { 2024-05-12T22:19:42.116 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "num_rep", 2024-05-12T22:19:42.116 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-12T22:19:42.117 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.117 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.117 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "3", 2024-05-12T22:19:42.117 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.117 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.117 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.117 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "default replica count for any created local pool", 2024-05-12T22:19:42.117 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.118 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.118 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.118 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.118 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": { 2024-05-12T22:19:42.118 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pg_num", 2024-05-12T22:19:42.118 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-12T22:19:42.118 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.118 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.118 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "128", 2024-05-12T22:19:42.119 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.119 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.119 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.119 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "default pg_num for any created local pool", 2024-05-12T22:19:42.119 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.119 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.119 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.119 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.119 INFO:teuthology.orchestra.run.smithi028.stdout: "prefix": { 2024-05-12T22:19:42.120 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "prefix", 2024-05-12T22:19:42.120 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.120 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.120 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.120 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.120 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.120 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.120 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.120 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "name prefix for any created local pool", 2024-05-12T22:19:42.121 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.121 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.121 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.121 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.121 INFO:teuthology.orchestra.run.smithi028.stdout: "subtree": { 2024-05-12T22:19:42.121 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "subtree", 2024-05-12T22:19:42.121 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.121 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.122 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.122 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "rack", 2024-05-12T22:19:42.122 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.122 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.122 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.122 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "CRUSH level for which to create a local pool", 2024-05-12T22:19:42.122 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-05-12T22:19:42.122 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.122 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.123 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:42.123 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:42.123 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.123 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-12T22:19:42.123 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mds_autoscaler", 2024-05-12T22:19:42.123 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-12T22:19:42.123 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-12T22:19:42.123 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-12T22:19:42.123 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-12T22:19:42.125 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-12T22:19:42.125 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.125 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.125 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.125 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.126 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.126 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.126 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:42.126 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:42.126 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:42.126 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:42.126 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:42.126 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:42.127 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:42.127 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:42.127 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.127 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.127 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.127 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.127 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.127 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-12T22:19:42.127 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-12T22:19:42.128 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.128 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.128 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.128 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.128 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.128 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.128 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.128 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.128 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.129 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.129 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.129 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.129 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-12T22:19:42.129 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-12T22:19:42.129 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.129 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.129 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.129 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-12T22:19:42.130 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.130 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.130 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:42.130 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:42.130 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:42.130 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:42.130 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:42.130 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:42.131 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:42.131 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:42.131 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.131 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.131 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.131 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.131 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.131 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-12T22:19:42.131 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-12T22:19:42.132 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.132 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.132 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.132 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.132 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.132 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.132 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.132 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.132 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.133 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.133 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.133 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:42.133 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:42.133 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.133 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-12T22:19:42.133 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mirroring", 2024-05-12T22:19:42.133 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-12T22:19:42.133 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-12T22:19:42.134 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-12T22:19:42.134 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-12T22:19:42.134 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-12T22:19:42.134 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.134 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.134 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.134 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.134 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.134 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.135 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:42.135 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:42.135 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:42.135 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:42.135 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:42.135 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:42.135 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:42.135 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:42.135 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.136 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.136 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.136 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.136 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.136 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-12T22:19:42.136 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-12T22:19:42.136 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.136 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.136 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.137 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.137 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.137 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.137 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.137 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.137 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.137 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.137 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.137 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.138 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-12T22:19:42.138 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-12T22:19:42.138 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.138 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.138 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.138 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-12T22:19:42.138 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.138 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.139 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:42.139 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:42.139 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:42.139 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:42.139 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:42.139 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:42.139 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:42.139 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:42.139 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.140 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.140 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.140 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.140 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.140 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-12T22:19:42.140 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-12T22:19:42.140 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.140 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.141 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.141 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.141 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.141 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.141 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.141 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.141 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.141 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.141 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.142 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:42.142 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:42.142 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.142 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-12T22:19:42.142 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "nfs", 2024-05-12T22:19:42.142 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-12T22:19:42.142 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-12T22:19:42.142 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-12T22:19:42.143 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-12T22:19:42.143 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-12T22:19:42.143 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.143 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.143 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.143 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.143 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.143 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.143 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:42.144 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:42.144 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:42.144 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:42.144 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:42.144 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:42.144 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:42.144 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:42.144 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.144 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.145 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.145 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.145 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.145 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-12T22:19:42.145 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-12T22:19:42.145 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.145 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.145 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.145 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.146 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.146 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.146 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.146 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.146 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.146 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.146 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.146 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.147 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-12T22:19:42.147 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-12T22:19:42.147 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.147 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.147 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.147 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-12T22:19:42.147 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.147 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.147 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:42.148 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:42.148 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:42.148 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:42.148 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:42.148 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:42.148 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:42.148 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:42.148 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.149 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.149 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.149 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.149 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.149 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-12T22:19:42.149 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-12T22:19:42.149 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.149 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.149 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.150 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.150 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.150 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.150 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.150 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.150 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.150 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.150 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.151 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:42.151 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:42.151 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.151 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-12T22:19:42.151 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "orchestrator", 2024-05-12T22:19:42.151 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-12T22:19:42.151 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-12T22:19:42.151 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-12T22:19:42.151 INFO:teuthology.orchestra.run.smithi028.stdout: "fail_fs": { 2024-05-12T22:19:42.152 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "fail_fs", 2024-05-12T22:19:42.152 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.152 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.152 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.152 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.152 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.152 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.152 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.153 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-05-12T22:19:42.153 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.153 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.153 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.153 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.153 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-12T22:19:42.153 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-12T22:19:42.153 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.153 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.154 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.154 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.154 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.154 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.154 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:42.154 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:42.154 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:42.154 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:42.154 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:42.155 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:42.155 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:42.155 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:42.155 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.155 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.155 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.155 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.155 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.156 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-12T22:19:42.156 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-12T22:19:42.156 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.156 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.156 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.156 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.156 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.156 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.156 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.157 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.157 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.157 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.157 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.157 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.157 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-12T22:19:42.157 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-12T22:19:42.157 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.157 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.158 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.158 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-12T22:19:42.158 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.158 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.158 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:42.158 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:42.158 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:42.158 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:42.159 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:42.159 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:42.159 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:42.159 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:42.159 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.159 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.159 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.159 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.160 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.160 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-12T22:19:42.160 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-12T22:19:42.160 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.160 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.160 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.160 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.160 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.160 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.161 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.161 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.161 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.161 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.161 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.161 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.161 INFO:teuthology.orchestra.run.smithi028.stdout: "orchestrator": { 2024-05-12T22:19:42.161 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "orchestrator", 2024-05-12T22:19:42.161 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.162 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.162 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.162 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.162 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.162 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.162 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:42.162 INFO:teuthology.orchestra.run.smithi028.stdout: "cephadm", 2024-05-12T22:19:42.162 INFO:teuthology.orchestra.run.smithi028.stdout: "rook", 2024-05-12T22:19:42.162 INFO:teuthology.orchestra.run.smithi028.stdout: "test_orchestrator" 2024-05-12T22:19:42.163 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:42.163 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Orchestrator backend", 2024-05-12T22:19:42.163 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.163 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.163 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.163 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:42.163 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:42.163 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.164 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-12T22:19:42.164 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "osd_perf_query", 2024-05-12T22:19:42.164 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-12T22:19:42.164 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-12T22:19:42.164 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-12T22:19:42.164 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-12T22:19:42.164 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-12T22:19:42.164 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.165 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.165 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.165 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.165 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.165 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.165 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:42.165 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:42.165 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:42.165 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:42.166 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:42.166 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:42.166 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:42.166 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:42.166 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.166 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.166 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.166 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.167 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.167 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-12T22:19:42.167 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-12T22:19:42.167 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.167 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.167 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.167 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.167 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.167 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.168 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.168 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.168 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.168 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.168 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.168 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.168 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-12T22:19:42.168 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-12T22:19:42.168 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.169 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.169 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.169 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-12T22:19:42.169 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.169 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.169 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:42.169 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:42.169 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:42.170 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:42.170 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:42.170 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:42.170 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:42.170 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:42.170 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.170 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.170 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.170 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.171 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.171 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-12T22:19:42.171 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-12T22:19:42.171 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.171 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.171 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.171 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.171 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.172 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.172 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.172 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.172 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.172 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.172 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.172 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:42.172 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:42.172 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.173 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-12T22:19:42.173 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "osd_support", 2024-05-12T22:19:42.173 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-12T22:19:42.173 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-12T22:19:42.173 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-12T22:19:42.173 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-12T22:19:42.173 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-12T22:19:42.173 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.173 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.174 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.174 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.174 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.174 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.174 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:42.174 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:42.174 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:42.174 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:42.175 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:42.175 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:42.175 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:42.175 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:42.175 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.175 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.175 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.175 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.176 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.176 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-12T22:19:42.176 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-12T22:19:42.176 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.176 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.176 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.176 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.176 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.176 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.177 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.177 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.177 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.177 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.177 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.177 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.177 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-12T22:19:42.177 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-12T22:19:42.177 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.178 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.178 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.178 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-12T22:19:42.178 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.178 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.178 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:42.178 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:42.178 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:42.179 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:42.179 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:42.179 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:42.179 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:42.179 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:42.179 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.179 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.179 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.179 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.180 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.180 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-12T22:19:42.180 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-12T22:19:42.180 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.180 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.180 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.180 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.180 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.180 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.181 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.181 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.181 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.181 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.181 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.181 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:42.181 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:42.181 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.181 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-12T22:19:42.182 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pg_autoscaler", 2024-05-12T22:19:42.182 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-12T22:19:42.182 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-12T22:19:42.182 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-12T22:19:42.182 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-12T22:19:42.182 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-12T22:19:42.182 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.182 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.183 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.183 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.183 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.183 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.183 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:42.183 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:42.183 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:42.183 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:42.184 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:42.184 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:42.184 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:42.184 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:42.184 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.184 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.184 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.184 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.184 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.185 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-12T22:19:42.185 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-12T22:19:42.185 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.185 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.185 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.185 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.185 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.185 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.185 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.186 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.186 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.186 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.186 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.186 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.186 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-12T22:19:42.186 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-12T22:19:42.186 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.187 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.187 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.187 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-12T22:19:42.187 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.187 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.187 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:42.187 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:42.187 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:42.188 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:42.188 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:42.188 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:42.188 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:42.188 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:42.188 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.188 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.188 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.188 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.189 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.189 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-12T22:19:42.189 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-12T22:19:42.189 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.189 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.189 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.189 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.189 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.189 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.190 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.190 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.190 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.190 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.190 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.190 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.190 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-05-12T22:19:42.190 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-05-12T22:19:42.190 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-05-12T22:19:42.191 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.191 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.191 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-05-12T22:19:42.191 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.191 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.191 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.191 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.191 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.192 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.192 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.192 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.192 INFO:teuthology.orchestra.run.smithi028.stdout: "threshold": { 2024-05-12T22:19:42.192 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "threshold", 2024-05-12T22:19:42.192 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-05-12T22:19:42.192 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.192 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.193 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "3.0", 2024-05-12T22:19:42.193 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1.0", 2024-05-12T22:19:42.193 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.193 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.193 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "scaling threshold", 2024-05-12T22:19:42.193 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-05-12T22:19:42.193 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.193 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.193 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:42.194 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:42.194 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.194 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-12T22:19:42.194 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "progress", 2024-05-12T22:19:42.194 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-12T22:19:42.194 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-12T22:19:42.194 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-12T22:19:42.194 INFO:teuthology.orchestra.run.smithi028.stdout: "allow_pg_recovery_event": { 2024-05-12T22:19:42.194 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "allow_pg_recovery_event", 2024-05-12T22:19:42.195 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.195 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.195 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.195 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.195 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.195 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.195 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.195 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "allow the module to show pg recovery progress", 2024-05-12T22:19:42.196 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.196 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.196 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.196 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.196 INFO:teuthology.orchestra.run.smithi028.stdout: "enabled": { 2024-05-12T22:19:42.196 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enabled", 2024-05-12T22:19:42.196 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.196 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.196 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.197 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-12T22:19:42.197 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.197 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.197 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.197 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.197 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.197 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.197 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.198 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.198 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-12T22:19:42.198 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-12T22:19:42.198 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.198 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.198 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.198 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.198 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.198 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.199 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:42.199 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:42.199 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:42.199 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:42.199 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:42.199 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:42.199 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:42.199 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:42.200 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.200 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.200 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.200 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.200 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.200 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-12T22:19:42.200 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-12T22:19:42.200 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.200 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.201 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.201 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.201 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.201 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.201 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.201 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.201 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.201 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.201 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.202 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.202 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-12T22:19:42.202 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-12T22:19:42.202 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.202 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.202 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.202 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-12T22:19:42.202 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.203 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.203 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:42.203 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:42.203 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:42.203 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:42.203 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:42.203 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:42.203 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:42.203 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:42.204 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.204 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.204 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.204 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.204 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.204 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-12T22:19:42.204 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-12T22:19:42.204 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.204 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.205 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.205 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.205 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.205 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.205 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.205 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.205 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.205 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.205 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.206 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.206 INFO:teuthology.orchestra.run.smithi028.stdout: "max_completed_events": { 2024-05-12T22:19:42.206 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_completed_events", 2024-05-12T22:19:42.206 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-12T22:19:42.206 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.206 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.206 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "50", 2024-05-12T22:19:42.206 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.207 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.207 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.207 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "number of past completed events to remember", 2024-05-12T22:19:42.207 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.207 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.207 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.207 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.207 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-05-12T22:19:42.208 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-05-12T22:19:42.208 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-05-12T22:19:42.208 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.208 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.208 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-05-12T22:19:42.208 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.208 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.208 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.209 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how long the module is going to sleep", 2024-05-12T22:19:42.209 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.209 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.209 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.209 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:42.209 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:42.209 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.209 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-12T22:19:42.209 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "prometheus", 2024-05-12T22:19:42.210 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-12T22:19:42.210 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-12T22:19:42.210 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-12T22:19:42.210 INFO:teuthology.orchestra.run.smithi028.stdout: "cache": { 2024-05-12T22:19:42.210 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cache", 2024-05-12T22:19:42.210 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.210 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.210 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.210 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-12T22:19:42.211 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.211 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.211 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.211 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.211 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.211 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.211 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.211 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.212 INFO:teuthology.orchestra.run.smithi028.stdout: "exclude_perf_counters": { 2024-05-12T22:19:42.212 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "exclude_perf_counters", 2024-05-12T22:19:42.212 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.212 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.212 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.212 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-12T22:19:42.212 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.212 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.212 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.213 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-05-12T22:19:42.213 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-05-12T22:19:42.213 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.213 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.213 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.213 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-12T22:19:42.213 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-12T22:19:42.213 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.213 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.214 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.214 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.214 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.214 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.214 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:42.214 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:42.214 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:42.214 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:42.214 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:42.215 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:42.215 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:42.215 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:42.215 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.215 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.215 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.215 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.215 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.216 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-12T22:19:42.216 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-12T22:19:42.216 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.216 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.216 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.216 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.216 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.216 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.216 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.217 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.217 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.217 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.217 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.217 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.217 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-12T22:19:42.217 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-12T22:19:42.217 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.217 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.218 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.218 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-12T22:19:42.218 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.218 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.218 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:42.218 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:42.218 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:42.218 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:42.218 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:42.219 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:42.219 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:42.219 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:42.219 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.219 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.219 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.219 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.219 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.219 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-12T22:19:42.220 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-12T22:19:42.220 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.220 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.220 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.220 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.220 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.220 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.220 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.220 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.221 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.221 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.221 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.221 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.221 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_stats_pools": { 2024-05-12T22:19:42.221 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rbd_stats_pools", 2024-05-12T22:19:42.221 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.221 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.221 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.222 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.222 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.222 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.222 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.222 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.222 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.222 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.222 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.222 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.223 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_stats_pools_refresh_interval": { 2024-05-12T22:19:42.223 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-05-12T22:19:42.223 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-12T22:19:42.223 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.223 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.223 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "300", 2024-05-12T22:19:42.223 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.223 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.224 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.224 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.224 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.224 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.224 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.224 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.224 INFO:teuthology.orchestra.run.smithi028.stdout: "scrape_interval": { 2024-05-12T22:19:42.224 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "scrape_interval", 2024-05-12T22:19:42.224 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-05-12T22:19:42.225 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.225 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.225 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "15.0", 2024-05-12T22:19:42.225 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.225 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.225 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.225 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.225 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.225 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.226 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.226 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.226 INFO:teuthology.orchestra.run.smithi028.stdout: "server_addr": { 2024-05-12T22:19:42.226 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_addr", 2024-05-12T22:19:42.226 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.226 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.226 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.226 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "::", 2024-05-12T22:19:42.226 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.227 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.227 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.227 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-05-12T22:19:42.227 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.227 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.227 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.227 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.227 INFO:teuthology.orchestra.run.smithi028.stdout: "server_port": { 2024-05-12T22:19:42.227 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_port", 2024-05-12T22:19:42.228 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-12T22:19:42.228 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.228 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.228 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "9283", 2024-05-12T22:19:42.228 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.228 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.228 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.228 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-05-12T22:19:42.228 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.229 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.229 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.229 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.229 INFO:teuthology.orchestra.run.smithi028.stdout: "stale_cache_strategy": { 2024-05-12T22:19:42.229 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "stale_cache_strategy", 2024-05-12T22:19:42.229 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.229 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.229 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.230 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "log", 2024-05-12T22:19:42.230 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.230 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.230 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.230 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.230 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.230 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.230 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.231 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.231 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_behaviour": { 2024-05-12T22:19:42.231 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_behaviour", 2024-05-12T22:19:42.231 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.231 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.231 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.231 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "default", 2024-05-12T22:19:42.231 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.231 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.232 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:42.232 INFO:teuthology.orchestra.run.smithi028.stdout: "default", 2024-05-12T22:19:42.232 INFO:teuthology.orchestra.run.smithi028.stdout: "error" 2024-05-12T22:19:42.232 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:42.232 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.232 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.232 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.232 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.232 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.233 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_error_status_code": { 2024-05-12T22:19:42.233 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_error_status_code", 2024-05-12T22:19:42.233 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-12T22:19:42.233 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.233 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.233 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "500", 2024-05-12T22:19:42.233 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "400", 2024-05-12T22:19:42.233 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "599", 2024-05-12T22:19:42.234 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.234 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.234 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.234 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.234 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.234 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:42.234 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:42.234 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.234 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-12T22:19:42.235 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rbd_support", 2024-05-12T22:19:42.235 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-12T22:19:42.235 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-12T22:19:42.235 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-12T22:19:42.235 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-12T22:19:42.235 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-12T22:19:42.235 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.235 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.236 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.236 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.236 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.236 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.236 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:42.236 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:42.236 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:42.236 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:42.236 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:42.237 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:42.237 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:42.237 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:42.237 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.237 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.237 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.237 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.237 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.237 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-12T22:19:42.238 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-12T22:19:42.238 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.238 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.238 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.238 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.238 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.238 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.238 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.238 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.239 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.239 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.239 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.239 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.239 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-12T22:19:42.239 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-12T22:19:42.239 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.239 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.240 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.240 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-12T22:19:42.240 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.240 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.240 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:42.240 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:42.240 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:42.240 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:42.240 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:42.241 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:42.241 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:42.241 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:42.241 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.241 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.241 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.241 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.241 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.242 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-12T22:19:42.242 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-12T22:19:42.242 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.242 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.242 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.242 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.242 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.242 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.242 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.243 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.243 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.243 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.243 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.243 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.243 INFO:teuthology.orchestra.run.smithi028.stdout: "max_concurrent_snap_create": { 2024-05-12T22:19:42.243 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_concurrent_snap_create", 2024-05-12T22:19:42.243 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-12T22:19:42.243 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.244 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.244 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-05-12T22:19:42.244 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.244 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.244 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.244 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.244 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.244 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.245 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.245 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.245 INFO:teuthology.orchestra.run.smithi028.stdout: "mirror_snapshot_schedule": { 2024-05-12T22:19:42.245 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mirror_snapshot_schedule", 2024-05-12T22:19:42.245 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.245 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.245 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.245 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.245 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.246 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.246 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.246 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.246 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.246 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.246 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.246 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.246 INFO:teuthology.orchestra.run.smithi028.stdout: "trash_purge_schedule": { 2024-05-12T22:19:42.246 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "trash_purge_schedule", 2024-05-12T22:19:42.247 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.247 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.247 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.247 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.247 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.247 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.247 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.247 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.248 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.248 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.248 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.248 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:42.248 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:42.248 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.248 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-12T22:19:42.248 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "restful", 2024-05-12T22:19:42.248 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-12T22:19:42.249 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-12T22:19:42.249 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-12T22:19:42.249 INFO:teuthology.orchestra.run.smithi028.stdout: "enable_auth": { 2024-05-12T22:19:42.249 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enable_auth", 2024-05-12T22:19:42.249 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.249 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.249 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.249 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-12T22:19:42.249 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.250 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.250 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.250 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.250 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.250 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.250 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.250 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.250 INFO:teuthology.orchestra.run.smithi028.stdout: "key_file": { 2024-05-12T22:19:42.250 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "key_file", 2024-05-12T22:19:42.251 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.251 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.251 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.251 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.251 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.251 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.251 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.251 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.252 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.252 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.252 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.252 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.252 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-12T22:19:42.252 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-12T22:19:42.252 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.253 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.253 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.253 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.253 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.253 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.253 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:42.253 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:42.253 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:42.254 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:42.254 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:42.254 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:42.254 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:42.254 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:42.254 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.254 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.254 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.254 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.255 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.255 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-12T22:19:42.255 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-12T22:19:42.255 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.255 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.255 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.255 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.255 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.256 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.256 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.256 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.256 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.256 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.256 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.256 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.256 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-12T22:19:42.257 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-12T22:19:42.257 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.257 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.257 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.257 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-12T22:19:42.257 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.257 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.257 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:42.257 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:42.258 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:42.258 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:42.258 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:42.258 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:42.258 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:42.258 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:42.258 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.258 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.259 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.259 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.259 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.259 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-12T22:19:42.259 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-12T22:19:42.259 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.259 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.259 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.259 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.260 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.260 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.260 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.260 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.260 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.260 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.260 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.260 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.260 INFO:teuthology.orchestra.run.smithi028.stdout: "server_addr": { 2024-05-12T22:19:42.261 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_addr", 2024-05-12T22:19:42.261 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.261 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.261 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.261 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.261 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.261 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.261 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.262 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.262 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.262 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.262 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.262 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.262 INFO:teuthology.orchestra.run.smithi028.stdout: "server_port": { 2024-05-12T22:19:42.262 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_port", 2024-05-12T22:19:42.262 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.263 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.263 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.263 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.263 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.263 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.263 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.263 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.263 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.263 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.264 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.264 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:42.264 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:42.264 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.264 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-12T22:19:42.264 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rgw", 2024-05-12T22:19:42.264 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-12T22:19:42.264 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-12T22:19:42.264 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-12T22:19:42.265 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-12T22:19:42.265 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-12T22:19:42.265 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.265 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.265 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.265 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.265 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.265 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.266 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:42.266 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:42.266 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:42.266 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:42.266 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:42.266 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:42.266 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:42.266 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:42.266 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.267 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.267 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.267 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.267 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.267 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-12T22:19:42.267 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-12T22:19:42.267 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.267 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.268 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.268 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.268 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.268 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.268 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.268 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.268 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.268 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.268 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.269 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.269 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-12T22:19:42.269 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-12T22:19:42.269 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.269 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.269 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.269 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-12T22:19:42.269 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.270 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.270 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:42.270 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:42.270 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:42.270 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:42.270 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:42.270 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:42.270 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:42.270 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:42.271 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.271 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.271 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.271 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.271 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.271 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-12T22:19:42.271 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-12T22:19:42.271 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.272 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.272 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.272 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.272 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.272 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.272 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.272 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.272 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.272 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.272 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.273 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:42.273 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:42.273 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.273 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-12T22:19:42.273 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rook", 2024-05-12T22:19:42.273 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-12T22:19:42.273 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-12T22:19:42.274 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-12T22:19:42.274 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-12T22:19:42.274 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-12T22:19:42.274 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.274 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.274 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.274 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.274 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.275 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.275 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:42.275 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:42.275 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:42.275 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:42.275 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:42.275 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:42.275 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:42.276 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:42.276 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.276 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.276 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.276 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.276 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.276 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-12T22:19:42.276 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-12T22:19:42.276 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.277 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.277 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.277 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.277 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.277 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.277 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.277 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.277 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.278 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.278 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.278 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.278 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-12T22:19:42.278 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-12T22:19:42.278 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.278 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.278 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.279 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-12T22:19:42.279 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.279 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.279 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:42.279 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:42.279 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:42.279 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:42.279 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:42.280 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:42.280 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:42.280 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:42.280 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.280 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.280 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.280 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.280 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.280 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-12T22:19:42.281 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-12T22:19:42.281 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.281 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.281 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.281 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.281 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.281 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.281 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.281 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.282 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.282 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.282 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.282 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.282 INFO:teuthology.orchestra.run.smithi028.stdout: "storage_class": { 2024-05-12T22:19:42.282 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "storage_class", 2024-05-12T22:19:42.282 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.282 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.283 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.283 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "local", 2024-05-12T22:19:42.283 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.283 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.283 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.283 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-05-12T22:19:42.283 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.283 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.284 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.284 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:42.284 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:42.284 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.284 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-12T22:19:42.284 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "selftest", 2024-05-12T22:19:42.284 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-12T22:19:42.284 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-12T22:19:42.284 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-12T22:19:42.284 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-12T22:19:42.285 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-12T22:19:42.285 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.285 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.285 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.285 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.285 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.285 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.285 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:42.286 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:42.286 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:42.286 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:42.286 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:42.286 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:42.286 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:42.286 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:42.286 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.286 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.287 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.287 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.287 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.287 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-12T22:19:42.287 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-12T22:19:42.287 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.287 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.287 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.287 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.288 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.288 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.288 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.288 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.288 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.288 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.288 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.288 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.288 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-12T22:19:42.289 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-12T22:19:42.289 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.289 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.289 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.289 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-12T22:19:42.289 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.289 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.289 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:42.290 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:42.290 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:42.290 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:42.290 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:42.290 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:42.290 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:42.290 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:42.290 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.290 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.291 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.291 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.291 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.291 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-12T22:19:42.291 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-12T22:19:42.291 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.291 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.291 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.292 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.292 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.292 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.292 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.292 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.292 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.292 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.292 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.292 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.293 INFO:teuthology.orchestra.run.smithi028.stdout: "roption1": { 2024-05-12T22:19:42.293 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "roption1", 2024-05-12T22:19:42.293 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.293 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.293 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.293 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.293 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.293 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.293 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.294 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.294 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.294 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.294 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.294 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.294 INFO:teuthology.orchestra.run.smithi028.stdout: "roption2": { 2024-05-12T22:19:42.294 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "roption2", 2024-05-12T22:19:42.294 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.295 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.295 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.295 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "xyz", 2024-05-12T22:19:42.295 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.295 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.295 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.295 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.295 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.296 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.296 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.296 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.296 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption1": { 2024-05-12T22:19:42.296 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption1", 2024-05-12T22:19:42.296 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.296 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.296 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.296 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.297 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.297 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.297 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.297 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.297 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.297 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.297 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.297 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.298 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption2": { 2024-05-12T22:19:42.298 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption2", 2024-05-12T22:19:42.298 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-12T22:19:42.298 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.298 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.298 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.298 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.298 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.299 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.299 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.299 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.299 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.299 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.299 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.299 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption3": { 2024-05-12T22:19:42.299 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption3", 2024-05-12T22:19:42.300 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-05-12T22:19:42.300 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.300 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.300 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.300 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.300 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.300 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.300 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.300 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.301 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.301 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.301 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.301 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption4": { 2024-05-12T22:19:42.301 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption4", 2024-05-12T22:19:42.301 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.301 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.301 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.302 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.302 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.302 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.302 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.302 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.302 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.302 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.302 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.302 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.303 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption5": { 2024-05-12T22:19:42.303 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption5", 2024-05-12T22:19:42.303 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.303 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.303 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.303 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.303 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.303 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.304 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.304 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.304 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.304 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.304 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.304 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.304 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption6": { 2024-05-12T22:19:42.304 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption6", 2024-05-12T22:19:42.304 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.305 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.305 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.305 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-12T22:19:42.305 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.305 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.305 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.305 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.305 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.306 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.306 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.306 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.306 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption7": { 2024-05-12T22:19:42.306 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption7", 2024-05-12T22:19:42.306 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-12T22:19:42.306 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.306 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.306 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.307 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-05-12T22:19:42.307 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "42", 2024-05-12T22:19:42.307 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.307 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.307 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.307 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.307 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.307 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.308 INFO:teuthology.orchestra.run.smithi028.stdout: "testkey": { 2024-05-12T22:19:42.308 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "testkey", 2024-05-12T22:19:42.308 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.308 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.308 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.308 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.308 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.308 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.308 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.309 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.309 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.309 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.309 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.309 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.309 INFO:teuthology.orchestra.run.smithi028.stdout: "testlkey": { 2024-05-12T22:19:42.309 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "testlkey", 2024-05-12T22:19:42.309 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.310 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.310 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.310 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.310 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.310 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.310 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.310 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.310 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.310 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.311 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.311 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.311 INFO:teuthology.orchestra.run.smithi028.stdout: "testnewline": { 2024-05-12T22:19:42.311 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "testnewline", 2024-05-12T22:19:42.311 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.311 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.311 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.311 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.312 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.312 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.312 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.312 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.312 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.312 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.312 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.312 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:42.312 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:42.313 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.313 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-12T22:19:42.313 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "snap_schedule", 2024-05-12T22:19:42.313 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-12T22:19:42.313 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-12T22:19:42.313 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-12T22:19:42.313 INFO:teuthology.orchestra.run.smithi028.stdout: "allow_m_granularity": { 2024-05-12T22:19:42.313 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "allow_m_granularity", 2024-05-12T22:19:42.314 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.314 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.314 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.314 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.314 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.314 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.314 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.314 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "allow minute scheduled snapshots", 2024-05-12T22:19:42.314 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.315 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.315 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.315 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.315 INFO:teuthology.orchestra.run.smithi028.stdout: "dump_on_update": { 2024-05-12T22:19:42.315 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "dump_on_update", 2024-05-12T22:19:42.315 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.315 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.315 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.315 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.316 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.316 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.316 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.316 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "dump database to debug log on update", 2024-05-12T22:19:42.316 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.316 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.316 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.316 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.316 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-12T22:19:42.317 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-12T22:19:42.317 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.317 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.317 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.317 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.317 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.317 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.317 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:42.318 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:42.318 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:42.318 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:42.318 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:42.318 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:42.318 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:42.318 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:42.318 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.318 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.319 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.319 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.319 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.319 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-12T22:19:42.319 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-12T22:19:42.319 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.319 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.319 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.320 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.320 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.320 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.320 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.320 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.320 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.320 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.320 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.320 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.321 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-12T22:19:42.321 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-12T22:19:42.321 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.321 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.321 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.321 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-12T22:19:42.321 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.322 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.322 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:42.322 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:42.322 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:42.322 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:42.322 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:42.322 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:42.322 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:42.322 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:42.323 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.323 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.323 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.323 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.323 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.323 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-12T22:19:42.323 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-12T22:19:42.323 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.324 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.324 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.324 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.324 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.324 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.324 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.324 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.324 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.324 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.325 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.325 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:42.325 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:42.325 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.325 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-12T22:19:42.325 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "stats", 2024-05-12T22:19:42.325 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-12T22:19:42.325 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-12T22:19:42.326 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-12T22:19:42.326 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-12T22:19:42.326 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-12T22:19:42.326 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.326 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.326 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.326 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.326 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.327 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.327 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:42.327 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:42.327 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:42.329 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:42.329 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:42.329 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:42.329 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:42.329 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:42.329 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.330 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.330 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.330 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.330 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.330 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-12T22:19:42.330 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-12T22:19:42.330 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.330 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.330 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.331 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.331 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.331 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.331 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.331 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.331 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.331 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.331 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.332 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.332 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-12T22:19:42.332 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-12T22:19:42.332 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.332 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.332 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.332 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-12T22:19:42.332 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.332 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.333 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:42.333 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:42.333 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:42.333 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:42.333 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:42.333 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:42.333 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:42.333 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:42.333 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.334 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.334 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.334 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.334 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.334 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-12T22:19:42.334 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-12T22:19:42.334 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.334 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.335 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.335 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.335 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.335 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.335 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.335 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.335 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.335 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.336 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.336 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:42.336 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:42.336 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.336 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-12T22:19:42.336 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "status", 2024-05-12T22:19:42.336 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-12T22:19:42.336 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-12T22:19:42.336 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-12T22:19:42.337 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-12T22:19:42.337 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-12T22:19:42.337 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.337 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.337 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.337 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.337 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.337 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.338 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:42.338 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:42.338 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:42.338 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:42.338 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:42.338 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:42.338 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:42.338 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:42.338 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.339 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.339 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.339 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.339 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.339 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-12T22:19:42.339 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-12T22:19:42.339 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.339 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.340 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.340 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.340 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.340 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.340 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.340 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.340 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.340 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.340 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.341 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.341 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-12T22:19:42.341 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-12T22:19:42.341 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.341 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.341 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.341 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-12T22:19:42.341 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.341 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.342 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:42.342 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:42.342 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:42.342 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:42.342 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:42.342 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:42.342 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:42.342 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:42.342 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.343 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.343 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.343 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.343 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.343 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-12T22:19:42.343 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-12T22:19:42.343 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.343 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.343 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.344 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.344 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.344 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.344 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.344 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.344 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.344 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.344 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.344 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:42.345 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:42.345 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.345 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-12T22:19:42.345 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "telegraf", 2024-05-12T22:19:42.345 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-12T22:19:42.345 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-12T22:19:42.345 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-12T22:19:42.345 INFO:teuthology.orchestra.run.smithi028.stdout: "address": { 2024-05-12T22:19:42.345 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "address", 2024-05-12T22:19:42.346 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.346 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.346 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.346 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-05-12T22:19:42.346 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.346 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.346 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.347 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.347 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.347 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.347 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.347 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.347 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-05-12T22:19:42.347 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-05-12T22:19:42.347 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-05-12T22:19:42.347 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.348 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.348 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "15", 2024-05-12T22:19:42.348 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.348 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.348 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.348 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.348 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.348 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.348 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.349 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.349 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-12T22:19:42.349 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-12T22:19:42.349 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.349 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.349 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.349 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.349 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.349 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.350 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:42.350 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:42.350 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:42.350 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:42.350 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:42.350 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:42.350 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:42.350 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:42.351 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.351 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.351 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.351 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.351 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.351 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-12T22:19:42.351 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-12T22:19:42.351 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.351 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.352 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.352 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.352 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.352 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.352 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.352 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.352 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.352 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.353 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.353 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.353 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-12T22:19:42.353 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-12T22:19:42.353 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.353 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.353 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.353 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-12T22:19:42.353 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.354 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.354 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:42.354 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:42.354 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:42.354 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:42.354 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:42.354 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:42.354 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:42.355 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:42.355 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.355 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.355 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.355 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.355 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.355 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-12T22:19:42.355 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-12T22:19:42.355 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.356 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.356 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.356 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.356 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.356 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.356 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.356 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.356 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.356 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.357 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.357 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:42.357 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:42.357 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.357 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-12T22:19:42.357 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "telemetry", 2024-05-12T22:19:42.357 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-12T22:19:42.357 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-12T22:19:42.357 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-12T22:19:42.358 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_basic": { 2024-05-12T22:19:42.358 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_basic", 2024-05-12T22:19:42.358 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.358 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.358 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.358 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-12T22:19:42.358 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.358 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.359 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.359 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share basic cluster information (size, version)", 2024-05-12T22:19:42.359 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.359 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.359 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.359 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.359 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_crash": { 2024-05-12T22:19:42.359 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_crash", 2024-05-12T22:19:42.359 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.360 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.360 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.360 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-12T22:19:42.360 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.360 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.360 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.360 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-05-12T22:19:42.360 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.361 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.361 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.361 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.361 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_device": { 2024-05-12T22:19:42.361 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_device", 2024-05-12T22:19:42.361 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.361 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.361 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.361 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-12T22:19:42.362 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.362 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.362 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.362 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-05-12T22:19:42.362 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.362 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.362 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.362 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.363 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_ident": { 2024-05-12T22:19:42.363 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_ident", 2024-05-12T22:19:42.363 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.363 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.363 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.363 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.363 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.363 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.363 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.364 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-05-12T22:19:42.364 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.364 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.364 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.364 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.364 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_perf": { 2024-05-12T22:19:42.364 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_perf", 2024-05-12T22:19:42.364 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.365 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.365 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.365 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.365 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.365 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.365 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.365 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share various performance metrics of a cluster", 2024-05-12T22:19:42.365 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.365 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.366 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.366 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.366 INFO:teuthology.orchestra.run.smithi028.stdout: "contact": { 2024-05-12T22:19:42.366 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "contact", 2024-05-12T22:19:42.366 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.366 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.366 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.366 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.367 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.367 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.367 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.367 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.367 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.367 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.367 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.367 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.367 INFO:teuthology.orchestra.run.smithi028.stdout: "description": { 2024-05-12T22:19:42.368 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "description", 2024-05-12T22:19:42.368 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.368 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.368 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.368 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.368 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.368 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.368 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.369 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.369 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.369 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.369 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.369 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.369 INFO:teuthology.orchestra.run.smithi028.stdout: "device_url": { 2024-05-12T22:19:42.369 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "device_url", 2024-05-12T22:19:42.369 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.370 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.370 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.370 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-05-12T22:19:42.370 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.370 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.370 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.370 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.370 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.371 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.371 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.371 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.371 INFO:teuthology.orchestra.run.smithi028.stdout: "enabled": { 2024-05-12T22:19:42.371 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enabled", 2024-05-12T22:19:42.371 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.371 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.371 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.372 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.372 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.372 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.372 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.372 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.372 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.372 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.372 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.372 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.373 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-05-12T22:19:42.373 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-05-12T22:19:42.373 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-12T22:19:42.373 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.373 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.373 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "24", 2024-05-12T22:19:42.373 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "8", 2024-05-12T22:19:42.373 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.373 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.374 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.374 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.374 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.374 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.374 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.374 INFO:teuthology.orchestra.run.smithi028.stdout: "last_opt_revision": { 2024-05-12T22:19:42.374 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "last_opt_revision", 2024-05-12T22:19:42.374 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-12T22:19:42.374 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.375 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.375 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "1", 2024-05-12T22:19:42.375 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.375 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.375 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.375 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.375 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.375 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.376 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.376 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.376 INFO:teuthology.orchestra.run.smithi028.stdout: "leaderboard": { 2024-05-12T22:19:42.376 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "leaderboard", 2024-05-12T22:19:42.376 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.376 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.376 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.376 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.377 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.377 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.377 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.377 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.377 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.377 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.377 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.377 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.378 INFO:teuthology.orchestra.run.smithi028.stdout: "leaderboard_description": { 2024-05-12T22:19:42.378 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "leaderboard_description", 2024-05-12T22:19:42.378 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.378 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.378 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.378 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.378 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.378 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.379 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.379 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.379 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.379 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.379 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.379 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.379 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-12T22:19:42.379 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-12T22:19:42.379 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.380 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.380 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.380 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.380 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.380 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.380 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:42.380 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:42.380 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:42.380 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:42.381 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:42.381 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:42.381 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:42.381 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:42.381 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.381 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.381 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.381 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.381 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.382 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-12T22:19:42.382 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-12T22:19:42.382 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.382 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.382 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.382 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.382 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.382 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.383 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.383 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.383 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.383 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.383 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.383 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.383 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-12T22:19:42.383 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-12T22:19:42.384 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.384 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.384 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.384 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-12T22:19:42.384 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.384 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.384 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:42.384 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:42.385 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:42.385 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:42.385 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:42.385 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:42.385 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:42.385 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:42.385 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.385 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.385 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.386 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.386 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.386 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-12T22:19:42.386 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-12T22:19:42.386 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.386 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.386 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.386 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.387 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.387 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.387 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.387 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.387 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.387 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.387 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.387 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.388 INFO:teuthology.orchestra.run.smithi028.stdout: "organization": { 2024-05-12T22:19:42.388 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "organization", 2024-05-12T22:19:42.388 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.388 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.388 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.388 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.388 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.388 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.389 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.389 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.389 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.389 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.389 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.389 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.389 INFO:teuthology.orchestra.run.smithi028.stdout: "proxy": { 2024-05-12T22:19:42.389 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "proxy", 2024-05-12T22:19:42.389 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.390 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.390 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.390 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.390 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.390 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.390 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.390 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.390 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.391 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.391 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.391 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.391 INFO:teuthology.orchestra.run.smithi028.stdout: "url": { 2024-05-12T22:19:42.391 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "url", 2024-05-12T22:19:42.391 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.391 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.391 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.392 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-05-12T22:19:42.392 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.392 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.392 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.392 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.392 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.392 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.392 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.392 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:42.393 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:42.393 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.393 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-12T22:19:42.393 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "test_orchestrator", 2024-05-12T22:19:42.393 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-12T22:19:42.393 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-12T22:19:42.393 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-12T22:19:42.393 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-12T22:19:42.393 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-12T22:19:42.393 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.394 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.394 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.394 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.394 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.394 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.394 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:42.394 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:42.394 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:42.395 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:42.395 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:42.395 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:42.395 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:42.395 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:42.395 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.395 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.395 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.395 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.396 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.396 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-12T22:19:42.396 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-12T22:19:42.396 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.396 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.396 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.396 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.396 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.396 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.396 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.397 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.397 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.397 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.397 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.397 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.397 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-12T22:19:42.397 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-12T22:19:42.397 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.398 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.398 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.398 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-12T22:19:42.398 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.398 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.398 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:42.398 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:42.398 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:42.399 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:42.399 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:42.399 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:42.399 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:42.399 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:42.399 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.399 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.399 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.400 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.400 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.400 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-12T22:19:42.400 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-12T22:19:42.400 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.400 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.400 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.400 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.400 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.401 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.401 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.401 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.401 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.401 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.401 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.401 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:42.401 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:42.401 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.402 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-12T22:19:42.402 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "volumes", 2024-05-12T22:19:42.402 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-12T22:19:42.402 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-12T22:19:42.402 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-12T22:19:42.402 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-12T22:19:42.402 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-12T22:19:42.402 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.403 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.403 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.403 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.403 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.403 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.403 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:42.403 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:42.403 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:42.404 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:42.404 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:42.404 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:42.404 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:42.404 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:42.404 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.404 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.404 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.404 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.405 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.405 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-12T22:19:42.405 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-12T22:19:42.405 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.405 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.405 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.405 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.405 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.406 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.406 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.406 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.406 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.406 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.406 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.406 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.406 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-12T22:19:42.407 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-12T22:19:42.407 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.407 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.407 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.407 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-12T22:19:42.407 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.407 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.407 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:42.407 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:42.408 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:42.408 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:42.408 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:42.408 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:42.408 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:42.408 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:42.408 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.408 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.409 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.409 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.409 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.409 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-12T22:19:42.409 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-12T22:19:42.409 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.409 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.409 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.409 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.409 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.410 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.410 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.410 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.410 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.410 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.410 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.410 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.410 INFO:teuthology.orchestra.run.smithi028.stdout: "max_concurrent_clones": { 2024-05-12T22:19:42.411 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_concurrent_clones", 2024-05-12T22:19:42.411 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-12T22:19:42.411 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.411 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.411 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "4", 2024-05-12T22:19:42.411 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.411 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.411 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.411 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Number of asynchronous cloner threads", 2024-05-12T22:19:42.412 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.412 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.412 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.412 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.412 INFO:teuthology.orchestra.run.smithi028.stdout: "periodic_async_work": { 2024-05-12T22:19:42.412 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "periodic_async_work", 2024-05-12T22:19:42.412 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.412 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.413 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.413 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.413 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.413 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.413 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.413 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Periodically check for async work", 2024-05-12T22:19:42.413 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.413 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.414 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.414 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.414 INFO:teuthology.orchestra.run.smithi028.stdout: "snapshot_clone_delay": { 2024-05-12T22:19:42.414 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "snapshot_clone_delay", 2024-05-12T22:19:42.414 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-12T22:19:42.414 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.414 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.414 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-05-12T22:19:42.415 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.415 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.415 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.415 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-05-12T22:19:42.415 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.415 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.415 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.415 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.415 INFO:teuthology.orchestra.run.smithi028.stdout: "snapshot_clone_no_wait": { 2024-05-12T22:19:42.416 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "snapshot_clone_no_wait", 2024-05-12T22:19:42.416 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.416 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.416 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.416 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-12T22:19:42.416 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.416 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.416 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.416 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-05-12T22:19:42.416 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.417 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.417 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.417 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:42.417 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:42.417 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.417 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-12T22:19:42.417 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix", 2024-05-12T22:19:42.417 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-12T22:19:42.417 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-12T22:19:42.418 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-12T22:19:42.418 INFO:teuthology.orchestra.run.smithi028.stdout: "discovery_interval": { 2024-05-12T22:19:42.418 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "discovery_interval", 2024-05-12T22:19:42.418 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-05-12T22:19:42.418 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.418 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.418 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "100", 2024-05-12T22:19:42.418 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.419 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.419 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.419 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.419 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.419 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.419 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.419 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.419 INFO:teuthology.orchestra.run.smithi028.stdout: "identifier": { 2024-05-12T22:19:42.419 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "identifier", 2024-05-12T22:19:42.420 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.420 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.420 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.420 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.420 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.420 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.420 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.420 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.421 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.421 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.421 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.421 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.421 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-05-12T22:19:42.421 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-05-12T22:19:42.421 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-05-12T22:19:42.421 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.421 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.422 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-05-12T22:19:42.422 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.422 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.422 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.422 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.422 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.422 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.422 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.423 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.423 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-12T22:19:42.423 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-12T22:19:42.423 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.423 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.423 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.423 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.423 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.423 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.424 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:42.424 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:42.424 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:42.424 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:42.424 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:42.424 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:42.424 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:42.424 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:42.424 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.425 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.425 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.425 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.425 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.425 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-12T22:19:42.425 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-12T22:19:42.425 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.425 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.425 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.426 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.426 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.426 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.426 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.426 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.426 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.426 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.426 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.426 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.427 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-12T22:19:42.427 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-12T22:19:42.427 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.427 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.427 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.427 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-12T22:19:42.427 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.427 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.427 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:42.428 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:42.428 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:42.428 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:42.428 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:42.428 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:42.428 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:42.428 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:42.428 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.429 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.429 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.429 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.429 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.429 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-12T22:19:42.429 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-12T22:19:42.429 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.429 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.429 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.430 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.430 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.430 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.430 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.430 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.430 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.430 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.430 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.431 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.431 INFO:teuthology.orchestra.run.smithi028.stdout: "zabbix_host": { 2024-05-12T22:19:42.431 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix_host", 2024-05-12T22:19:42.431 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.431 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.431 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.431 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.431 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.431 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.431 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.432 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.432 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.432 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.432 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.432 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.432 INFO:teuthology.orchestra.run.smithi028.stdout: "zabbix_port": { 2024-05-12T22:19:42.432 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix_port", 2024-05-12T22:19:42.432 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-12T22:19:42.432 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.433 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.433 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10051", 2024-05-12T22:19:42.433 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.433 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.433 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.433 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.433 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.433 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.433 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.434 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.434 INFO:teuthology.orchestra.run.smithi028.stdout: "zabbix_sender": { 2024-05-12T22:19:42.434 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix_sender", 2024-05-12T22:19:42.434 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.434 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.434 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.434 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-05-12T22:19:42.434 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.435 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.435 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.435 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.435 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.435 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.435 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.435 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:42.435 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:42.436 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:42.436 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-05-12T22:19:42.436 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.436 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-12T22:19:42.436 INFO:teuthology.orchestra.run.smithi028.stdout: "gid": 25202, 2024-05-12T22:19:42.436 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "y", 2024-05-12T22:19:42.436 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_features": 4540701547738038271, 2024-05-12T22:19:42.436 INFO:teuthology.orchestra.run.smithi028.stdout: "available_modules": [ 2024-05-12T22:19:42.436 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-12T22:19:42.437 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "alerts", 2024-05-12T22:19:42.437 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-12T22:19:42.437 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-12T22:19:42.437 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-12T22:19:42.437 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-05-12T22:19:42.437 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-05-12T22:19:42.437 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-05-12T22:19:42.437 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.437 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.438 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-05-12T22:19:42.438 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.438 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.438 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.438 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How frequently to reexamine health status", 2024-05-12T22:19:42.438 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.438 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.438 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.439 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.439 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-12T22:19:42.439 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-12T22:19:42.439 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.439 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.439 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.439 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.439 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.439 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.440 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:42.440 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:42.440 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:42.440 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:42.440 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:42.440 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:42.440 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:42.440 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:42.440 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.441 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.441 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.441 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.441 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.441 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-12T22:19:42.441 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-12T22:19:42.441 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.441 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.442 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.442 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.442 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.442 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.442 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.442 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.442 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.442 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.442 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.443 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.443 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-12T22:19:42.443 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-12T22:19:42.443 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.443 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.443 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.443 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-12T22:19:42.443 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.444 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.444 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:42.444 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:42.444 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:42.444 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:42.444 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:42.444 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:42.444 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:42.444 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:42.445 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.445 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.445 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.445 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.445 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.445 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-12T22:19:42.445 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-12T22:19:42.445 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.445 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.446 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.446 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.446 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.446 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.446 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.446 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.446 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.446 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.447 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.447 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.447 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_destination": { 2024-05-12T22:19:42.447 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_destination", 2024-05-12T22:19:42.447 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.447 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.447 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.447 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.447 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.448 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.448 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.448 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Email address to send alerts to", 2024-05-12T22:19:42.448 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.448 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.448 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.448 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.448 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_from_name": { 2024-05-12T22:19:42.449 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_from_name", 2024-05-12T22:19:42.449 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.449 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.449 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.449 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "Ceph", 2024-05-12T22:19:42.449 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.449 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.449 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.449 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Email From: name", 2024-05-12T22:19:42.450 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.450 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.450 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.450 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.450 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_host": { 2024-05-12T22:19:42.450 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_host", 2024-05-12T22:19:42.450 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.450 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.451 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.451 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.451 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.451 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.451 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.451 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SMTP server", 2024-05-12T22:19:42.451 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.451 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.451 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.452 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.452 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_password": { 2024-05-12T22:19:42.452 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_password", 2024-05-12T22:19:42.452 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.452 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.452 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.452 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.452 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.452 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.453 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.453 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Password to authenticate with", 2024-05-12T22:19:42.453 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.453 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.453 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.453 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.453 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_port": { 2024-05-12T22:19:42.453 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_port", 2024-05-12T22:19:42.453 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-12T22:19:42.454 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.454 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.454 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "465", 2024-05-12T22:19:42.454 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.454 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.454 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.454 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SMTP port", 2024-05-12T22:19:42.454 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.455 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.455 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.455 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.455 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_sender": { 2024-05-12T22:19:42.455 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_sender", 2024-05-12T22:19:42.455 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.455 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.455 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.455 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.456 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.456 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.456 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.456 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SMTP envelope sender", 2024-05-12T22:19:42.456 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.456 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.456 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.456 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.457 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_ssl": { 2024-05-12T22:19:42.457 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_ssl", 2024-05-12T22:19:42.457 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.457 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.457 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.457 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-12T22:19:42.457 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.457 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.457 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.458 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use SSL to connect to SMTP server", 2024-05-12T22:19:42.458 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.458 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.458 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.458 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.458 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_user": { 2024-05-12T22:19:42.458 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_user", 2024-05-12T22:19:42.458 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.459 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.459 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.459 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.459 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.459 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.459 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.459 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "User to authenticate as", 2024-05-12T22:19:42.459 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.459 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.460 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.460 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:42.460 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:42.460 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.460 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-12T22:19:42.460 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "balancer", 2024-05-12T22:19:42.460 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-12T22:19:42.460 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-12T22:19:42.460 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-12T22:19:42.461 INFO:teuthology.orchestra.run.smithi028.stdout: "active": { 2024-05-12T22:19:42.461 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "active", 2024-05-12T22:19:42.461 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.461 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.461 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.461 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-12T22:19:42.461 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.461 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.461 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.462 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "automatically balance PGs across cluster", 2024-05-12T22:19:42.462 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.462 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.462 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.462 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.462 INFO:teuthology.orchestra.run.smithi028.stdout: "begin_time": { 2024-05-12T22:19:42.462 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "begin_time", 2024-05-12T22:19:42.462 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.463 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.463 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.463 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0000", 2024-05-12T22:19:42.463 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.463 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.463 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.463 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "beginning time of day to automatically balance", 2024-05-12T22:19:42.463 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-05-12T22:19:42.463 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.464 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.464 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.464 INFO:teuthology.orchestra.run.smithi028.stdout: "begin_weekday": { 2024-05-12T22:19:42.464 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "begin_weekday", 2024-05-12T22:19:42.464 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-05-12T22:19:42.464 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.464 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.464 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-05-12T22:19:42.465 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "0", 2024-05-12T22:19:42.465 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "6", 2024-05-12T22:19:42.465 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.465 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-05-12T22:19:42.465 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-05-12T22:19:42.465 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.465 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.465 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.465 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_compat_max_iterations": { 2024-05-12T22:19:42.466 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crush_compat_max_iterations", 2024-05-12T22:19:42.466 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-05-12T22:19:42.466 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.466 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.466 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "25", 2024-05-12T22:19:42.466 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-05-12T22:19:42.466 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "250", 2024-05-12T22:19:42.466 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.467 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-05-12T22:19:42.467 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.467 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.467 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.467 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.467 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_compat_metrics": { 2024-05-12T22:19:42.467 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crush_compat_metrics", 2024-05-12T22:19:42.467 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.467 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.467 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.468 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "pgs,objects,bytes", 2024-05-12T22:19:42.468 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.468 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.468 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.468 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-05-12T22:19:42.468 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-05-12T22:19:42.468 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.468 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.468 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.469 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_compat_step": { 2024-05-12T22:19:42.469 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crush_compat_step", 2024-05-12T22:19:42.469 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-05-12T22:19:42.469 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.469 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.469 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0.5", 2024-05-12T22:19:42.469 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "0.001", 2024-05-12T22:19:42.469 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "0.999", 2024-05-12T22:19:42.470 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.470 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "aggressiveness of optimization", 2024-05-12T22:19:42.470 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-05-12T22:19:42.470 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.470 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.470 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.470 INFO:teuthology.orchestra.run.smithi028.stdout: "end_time": { 2024-05-12T22:19:42.470 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "end_time", 2024-05-12T22:19:42.470 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.471 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.471 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.471 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "2359", 2024-05-12T22:19:42.471 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.471 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.471 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.471 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "ending time of day to automatically balance", 2024-05-12T22:19:42.471 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-05-12T22:19:42.472 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.472 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.472 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.472 INFO:teuthology.orchestra.run.smithi028.stdout: "end_weekday": { 2024-05-12T22:19:42.472 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "end_weekday", 2024-05-12T22:19:42.472 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-05-12T22:19:42.472 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.472 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.472 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-05-12T22:19:42.473 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "0", 2024-05-12T22:19:42.473 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "6", 2024-05-12T22:19:42.473 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.473 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-05-12T22:19:42.473 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-05-12T22:19:42.473 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.473 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.473 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.473 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-12T22:19:42.474 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-12T22:19:42.474 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.474 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.474 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.474 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.474 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.474 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.474 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:42.475 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:42.475 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:42.475 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:42.475 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:42.475 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:42.475 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:42.475 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:42.475 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.475 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.476 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.476 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.476 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.476 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-12T22:19:42.476 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-12T22:19:42.476 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.476 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.476 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.477 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.477 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.477 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.477 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.477 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.477 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.477 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.477 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.477 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.478 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-12T22:19:42.478 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-12T22:19:42.478 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.478 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.478 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.478 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-12T22:19:42.478 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.478 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.479 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:42.479 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:42.479 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:42.479 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:42.479 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:42.479 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:42.479 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:42.479 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:42.480 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.480 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.480 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.480 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.480 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.480 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-12T22:19:42.480 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-12T22:19:42.480 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.480 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.481 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.481 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.481 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.481 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.481 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.481 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.481 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.481 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.481 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.482 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.482 INFO:teuthology.orchestra.run.smithi028.stdout: "min_score": { 2024-05-12T22:19:42.482 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "min_score", 2024-05-12T22:19:42.482 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-05-12T22:19:42.482 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.482 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.482 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-05-12T22:19:42.482 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.483 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.483 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.483 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-05-12T22:19:42.483 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.483 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.483 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.483 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.483 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": { 2024-05-12T22:19:42.484 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mode", 2024-05-12T22:19:42.484 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.484 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.484 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.484 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "upmap", 2024-05-12T22:19:42.484 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.484 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.484 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:42.484 INFO:teuthology.orchestra.run.smithi028.stdout: "crush-compat", 2024-05-12T22:19:42.485 INFO:teuthology.orchestra.run.smithi028.stdout: "none", 2024-05-12T22:19:42.485 INFO:teuthology.orchestra.run.smithi028.stdout: "read", 2024-05-12T22:19:42.485 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap", 2024-05-12T22:19:42.485 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap-read" 2024-05-12T22:19:42.485 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:42.485 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Balancer mode", 2024-05-12T22:19:42.485 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.485 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.485 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.486 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.486 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_ids": { 2024-05-12T22:19:42.486 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pool_ids", 2024-05-12T22:19:42.486 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.486 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.486 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.486 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.486 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.486 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.487 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.487 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-05-12T22:19:42.487 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.487 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.487 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.487 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.487 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-05-12T22:19:42.487 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-05-12T22:19:42.488 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-05-12T22:19:42.488 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.488 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.488 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-05-12T22:19:42.488 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.488 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.488 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.488 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-05-12T22:19:42.489 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.489 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.489 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.489 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.489 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap_max_deviation": { 2024-05-12T22:19:42.489 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "upmap_max_deviation", 2024-05-12T22:19:42.489 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-12T22:19:42.489 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.489 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.490 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-05-12T22:19:42.490 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-05-12T22:19:42.490 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.490 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.490 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "deviation below which no optimization is attempted", 2024-05-12T22:19:42.490 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-05-12T22:19:42.490 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.490 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.491 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.491 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap_max_optimizations": { 2024-05-12T22:19:42.491 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "upmap_max_optimizations", 2024-05-12T22:19:42.491 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-05-12T22:19:42.491 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.491 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.491 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-05-12T22:19:42.491 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.491 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.492 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.492 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-05-12T22:19:42.492 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.492 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.492 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.492 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:42.492 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:42.492 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.493 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-12T22:19:42.493 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cephadm", 2024-05-12T22:19:42.493 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-12T22:19:42.493 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-12T22:19:42.493 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-12T22:19:42.493 INFO:teuthology.orchestra.run.smithi028.stdout: "agent_down_multiplier": { 2024-05-12T22:19:42.493 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "agent_down_multiplier", 2024-05-12T22:19:42.493 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-05-12T22:19:42.493 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.494 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.494 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "3.0", 2024-05-12T22:19:42.494 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.494 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.494 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.494 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-05-12T22:19:42.494 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.494 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.495 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.495 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.495 INFO:teuthology.orchestra.run.smithi028.stdout: "agent_refresh_rate": { 2024-05-12T22:19:42.495 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "agent_refresh_rate", 2024-05-12T22:19:42.495 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-05-12T22:19:42.495 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.495 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.495 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "20", 2024-05-12T22:19:42.496 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.496 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.496 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.496 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-05-12T22:19:42.496 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.496 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.496 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.496 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.496 INFO:teuthology.orchestra.run.smithi028.stdout: "agent_starting_port": { 2024-05-12T22:19:42.497 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "agent_starting_port", 2024-05-12T22:19:42.497 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-12T22:19:42.497 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.497 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.497 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "4721", 2024-05-12T22:19:42.497 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.497 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.497 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.497 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-05-12T22:19:42.498 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.498 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.498 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.498 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.498 INFO:teuthology.orchestra.run.smithi028.stdout: "allow_ptrace": { 2024-05-12T22:19:42.498 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "allow_ptrace", 2024-05-12T22:19:42.498 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.498 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.499 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.499 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.499 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.499 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.499 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.499 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-05-12T22:19:42.499 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-05-12T22:19:42.499 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.500 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.500 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.500 INFO:teuthology.orchestra.run.smithi028.stdout: "autotune_interval": { 2024-05-12T22:19:42.500 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "autotune_interval", 2024-05-12T22:19:42.500 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-05-12T22:19:42.500 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.500 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.500 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-05-12T22:19:42.500 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.501 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.501 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.501 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to autotune daemon memory", 2024-05-12T22:19:42.501 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.501 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.501 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.501 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.501 INFO:teuthology.orchestra.run.smithi028.stdout: "autotune_memory_target_ratio": { 2024-05-12T22:19:42.502 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "autotune_memory_target_ratio", 2024-05-12T22:19:42.502 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-05-12T22:19:42.502 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.502 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.502 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0.7", 2024-05-12T22:19:42.502 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.502 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.502 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.503 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-05-12T22:19:42.503 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.503 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.503 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.503 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.503 INFO:teuthology.orchestra.run.smithi028.stdout: "cephadm_log_destination": { 2024-05-12T22:19:42.503 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cephadm_log_destination", 2024-05-12T22:19:42.503 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.503 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.504 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.504 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.504 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.504 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.504 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:42.504 INFO:teuthology.orchestra.run.smithi028.stdout: "file", 2024-05-12T22:19:42.504 INFO:teuthology.orchestra.run.smithi028.stdout: "file,syslog", 2024-05-12T22:19:42.504 INFO:teuthology.orchestra.run.smithi028.stdout: "syslog" 2024-05-12T22:19:42.504 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:42.505 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-05-12T22:19:42.505 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.505 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.505 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.505 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.505 INFO:teuthology.orchestra.run.smithi028.stdout: "cgroups_split": { 2024-05-12T22:19:42.505 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cgroups_split", 2024-05-12T22:19:42.505 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.505 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.506 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.506 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-12T22:19:42.506 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.506 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.506 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.506 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-05-12T22:19:42.506 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.506 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.507 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.507 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.507 INFO:teuthology.orchestra.run.smithi028.stdout: "config_checks_enabled": { 2024-05-12T22:19:42.507 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "config_checks_enabled", 2024-05-12T22:19:42.507 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.507 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.507 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.507 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.507 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.507 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.508 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.508 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-05-12T22:19:42.508 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.508 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.508 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.508 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.508 INFO:teuthology.orchestra.run.smithi028.stdout: "config_dashboard": { 2024-05-12T22:19:42.508 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "config_dashboard", 2024-05-12T22:19:42.509 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.509 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.509 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.509 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-12T22:19:42.509 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.509 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.509 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.509 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-05-12T22:19:42.509 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.510 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.510 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.510 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.510 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_alertmanager": { 2024-05-12T22:19:42.510 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_alertmanager", 2024-05-12T22:19:42.510 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.510 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.510 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.511 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-05-12T22:19:42.511 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.511 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.511 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.511 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-05-12T22:19:42.511 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.511 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.511 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.511 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.512 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_base": { 2024-05-12T22:19:42.512 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_base", 2024-05-12T22:19:42.512 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.512 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.512 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.512 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/ceph", 2024-05-12T22:19:42.512 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.512 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.512 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.513 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Container image name, without the tag", 2024-05-12T22:19:42.513 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.513 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.513 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.513 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.513 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_elasticsearch": { 2024-05-12T22:19:42.513 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_elasticsearch", 2024-05-12T22:19:42.513 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.513 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.514 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.514 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-05-12T22:19:42.514 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.514 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.514 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.514 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "elasticsearch container image", 2024-05-12T22:19:42.514 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.514 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.515 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.515 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.515 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_grafana": { 2024-05-12T22:19:42.515 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_grafana", 2024-05-12T22:19:42.515 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.515 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.515 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.515 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-05-12T22:19:42.515 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.516 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.516 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.516 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-05-12T22:19:42.516 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.516 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.516 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.516 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.516 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_haproxy": { 2024-05-12T22:19:42.516 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_haproxy", 2024-05-12T22:19:42.517 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.517 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.517 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.517 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-05-12T22:19:42.517 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.517 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.517 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.517 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "HAproxy container image", 2024-05-12T22:19:42.518 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.518 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.518 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.518 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.518 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_jaeger_agent": { 2024-05-12T22:19:42.518 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_jaeger_agent", 2024-05-12T22:19:42.518 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.518 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.519 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.519 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-05-12T22:19:42.519 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.519 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.519 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.519 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Jaeger agent container image", 2024-05-12T22:19:42.519 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.519 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.519 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.520 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.520 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_jaeger_collector": { 2024-05-12T22:19:42.520 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_jaeger_collector", 2024-05-12T22:19:42.520 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.520 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.520 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.520 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-05-12T22:19:42.520 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.521 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.521 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.521 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Jaeger collector container image", 2024-05-12T22:19:42.521 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.521 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.521 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.521 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.521 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_jaeger_query": { 2024-05-12T22:19:42.521 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_jaeger_query", 2024-05-12T22:19:42.522 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.522 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.522 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.522 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-05-12T22:19:42.522 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.522 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.522 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.522 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Jaeger query container image", 2024-05-12T22:19:42.523 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.523 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.523 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.523 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.523 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_keepalived": { 2024-05-12T22:19:42.523 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_keepalived", 2024-05-12T22:19:42.523 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.523 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.524 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.524 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-05-12T22:19:42.524 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.524 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.524 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.524 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Keepalived container image", 2024-05-12T22:19:42.524 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.524 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.524 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.525 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.525 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_loki": { 2024-05-12T22:19:42.525 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_loki", 2024-05-12T22:19:42.525 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.525 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.525 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.525 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-05-12T22:19:42.525 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.525 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.526 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.526 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Loki container image", 2024-05-12T22:19:42.526 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.526 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.526 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.526 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.526 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_node_exporter": { 2024-05-12T22:19:42.526 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_node_exporter", 2024-05-12T22:19:42.527 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.527 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.527 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.527 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-05-12T22:19:42.527 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.527 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.527 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.527 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-05-12T22:19:42.527 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.528 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.528 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.528 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.528 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_nvmeof": { 2024-05-12T22:19:42.528 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_nvmeof", 2024-05-12T22:19:42.528 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.528 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.528 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.529 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-05-12T22:19:42.529 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.529 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.529 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.529 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Nvme-of container image", 2024-05-12T22:19:42.529 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.529 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.529 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.529 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.530 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_prometheus": { 2024-05-12T22:19:42.530 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_prometheus", 2024-05-12T22:19:42.530 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.530 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.530 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.530 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-05-12T22:19:42.530 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.530 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.531 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.531 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-05-12T22:19:42.531 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.531 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.531 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.531 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.531 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_promtail": { 2024-05-12T22:19:42.531 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_promtail", 2024-05-12T22:19:42.532 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.532 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.532 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.532 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-05-12T22:19:42.532 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.532 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.532 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.532 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Promtail container image", 2024-05-12T22:19:42.533 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.533 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.533 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.533 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.533 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_samba": { 2024-05-12T22:19:42.533 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_samba", 2024-05-12T22:19:42.533 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.533 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.533 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.534 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-05-12T22:19:42.534 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.534 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.534 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.534 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Samba/SMB container image", 2024-05-12T22:19:42.534 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.534 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.534 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.535 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.535 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_snmp_gateway": { 2024-05-12T22:19:42.535 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_snmp_gateway", 2024-05-12T22:19:42.535 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.535 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.535 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.535 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-05-12T22:19:42.535 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.535 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.536 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.536 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SNMP Gateway container image", 2024-05-12T22:19:42.536 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.536 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.536 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.536 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.536 INFO:teuthology.orchestra.run.smithi028.stdout: "container_init": { 2024-05-12T22:19:42.536 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_init", 2024-05-12T22:19:42.536 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.537 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.537 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.537 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-12T22:19:42.537 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.537 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.537 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.537 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Run podman/docker with `--init`", 2024-05-12T22:19:42.537 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.538 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.538 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.538 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.538 INFO:teuthology.orchestra.run.smithi028.stdout: "daemon_cache_timeout": { 2024-05-12T22:19:42.538 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "daemon_cache_timeout", 2024-05-12T22:19:42.538 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-05-12T22:19:42.538 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.538 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.539 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-05-12T22:19:42.539 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.539 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.539 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.539 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-05-12T22:19:42.539 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.539 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.539 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.540 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.540 INFO:teuthology.orchestra.run.smithi028.stdout: "default_cephadm_command_timeout": { 2024-05-12T22:19:42.540 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "default_cephadm_command_timeout", 2024-05-12T22:19:42.540 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-12T22:19:42.540 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.540 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.540 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "900", 2024-05-12T22:19:42.540 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.540 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.541 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.541 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-05-12T22:19:42.541 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.541 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.541 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.541 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.541 INFO:teuthology.orchestra.run.smithi028.stdout: "default_registry": { 2024-05-12T22:19:42.541 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "default_registry", 2024-05-12T22:19:42.541 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.542 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.542 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.542 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io", 2024-05-12T22:19:42.542 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.542 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.542 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.542 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-05-12T22:19:42.542 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.543 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.543 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.543 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.543 INFO:teuthology.orchestra.run.smithi028.stdout: "device_cache_timeout": { 2024-05-12T22:19:42.543 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "device_cache_timeout", 2024-05-12T22:19:42.543 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-05-12T22:19:42.543 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.543 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.543 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "1800", 2024-05-12T22:19:42.544 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.544 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.544 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.544 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "seconds to cache device inventory", 2024-05-12T22:19:42.544 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.544 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.544 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.544 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.545 INFO:teuthology.orchestra.run.smithi028.stdout: "device_enhanced_scan": { 2024-05-12T22:19:42.545 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "device_enhanced_scan", 2024-05-12T22:19:42.545 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.545 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.545 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.545 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.545 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.545 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.545 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.546 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use libstoragemgmt during device scans", 2024-05-12T22:19:42.546 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.546 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.546 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.546 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.546 INFO:teuthology.orchestra.run.smithi028.stdout: "facts_cache_timeout": { 2024-05-12T22:19:42.546 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "facts_cache_timeout", 2024-05-12T22:19:42.546 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-05-12T22:19:42.547 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.547 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.547 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-05-12T22:19:42.547 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.547 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.547 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.547 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "seconds to cache host facts data", 2024-05-12T22:19:42.547 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.547 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.548 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.548 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.548 INFO:teuthology.orchestra.run.smithi028.stdout: "grafana_dashboards_path": { 2024-05-12T22:19:42.548 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "grafana_dashboards_path", 2024-05-12T22:19:42.548 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.548 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.548 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.548 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-05-12T22:19:42.549 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.549 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.549 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.549 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-05-12T22:19:42.549 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.549 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.549 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.549 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.549 INFO:teuthology.orchestra.run.smithi028.stdout: "host_check_interval": { 2024-05-12T22:19:42.550 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "host_check_interval", 2024-05-12T22:19:42.550 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-05-12T22:19:42.550 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.550 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.550 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-05-12T22:19:42.550 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.550 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.550 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.550 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to perform a host check", 2024-05-12T22:19:42.551 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.551 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.551 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.551 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.551 INFO:teuthology.orchestra.run.smithi028.stdout: "hw_monitoring": { 2024-05-12T22:19:42.551 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "hw_monitoring", 2024-05-12T22:19:42.551 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.551 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.552 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.552 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.552 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.552 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.552 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.552 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-05-12T22:19:42.552 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.552 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.552 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.553 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.553 INFO:teuthology.orchestra.run.smithi028.stdout: "inventory_list_all": { 2024-05-12T22:19:42.553 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "inventory_list_all", 2024-05-12T22:19:42.553 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.553 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.553 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.553 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.553 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.554 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.554 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.554 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-05-12T22:19:42.554 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.554 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.554 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.554 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.554 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-12T22:19:42.555 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-12T22:19:42.555 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.555 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.555 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.555 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.555 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.555 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.555 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:42.555 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:42.556 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:42.556 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:42.556 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:42.556 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:42.556 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:42.556 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:42.556 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.556 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.556 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.557 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.557 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.557 INFO:teuthology.orchestra.run.smithi028.stdout: "log_refresh_metadata": { 2024-05-12T22:19:42.557 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_refresh_metadata", 2024-05-12T22:19:42.557 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.557 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.557 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.557 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.557 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.558 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.558 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.558 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-05-12T22:19:42.558 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.558 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.558 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.558 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.559 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-12T22:19:42.559 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-12T22:19:42.559 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.559 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.559 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.559 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-12T22:19:42.559 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.559 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.559 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.560 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-05-12T22:19:42.560 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.560 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.560 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.560 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.560 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-12T22:19:42.560 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-12T22:19:42.560 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.560 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.561 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.561 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-12T22:19:42.561 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.561 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.561 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:42.561 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:42.561 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:42.561 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:42.561 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:42.562 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:42.562 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:42.562 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:42.562 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.562 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.562 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.562 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.562 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.562 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-12T22:19:42.563 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-12T22:19:42.563 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.563 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.563 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.563 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.563 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.563 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.563 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.564 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.564 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.564 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.564 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.564 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.564 INFO:teuthology.orchestra.run.smithi028.stdout: "manage_etc_ceph_ceph_conf": { 2024-05-12T22:19:42.564 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-05-12T22:19:42.564 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.564 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.565 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.565 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.565 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.565 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.565 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.565 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-05-12T22:19:42.565 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.565 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.565 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.566 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.566 INFO:teuthology.orchestra.run.smithi028.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-05-12T22:19:42.566 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-05-12T22:19:42.566 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.566 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.566 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.566 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "*", 2024-05-12T22:19:42.566 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.567 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.567 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.567 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-05-12T22:19:42.567 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.567 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.567 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.567 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.567 INFO:teuthology.orchestra.run.smithi028.stdout: "max_count_per_host": { 2024-05-12T22:19:42.568 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_count_per_host", 2024-05-12T22:19:42.568 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-12T22:19:42.568 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.568 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.568 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-05-12T22:19:42.568 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.568 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.568 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.568 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "max number of daemons per service per host", 2024-05-12T22:19:42.568 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.569 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.569 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.569 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.569 INFO:teuthology.orchestra.run.smithi028.stdout: "max_osd_draining_count": { 2024-05-12T22:19:42.569 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_osd_draining_count", 2024-05-12T22:19:42.569 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-12T22:19:42.569 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.569 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.570 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-05-12T22:19:42.570 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.570 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.570 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.570 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-05-12T22:19:42.570 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.570 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.570 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.570 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.571 INFO:teuthology.orchestra.run.smithi028.stdout: "migration_current": { 2024-05-12T22:19:42.571 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "migration_current", 2024-05-12T22:19:42.571 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-12T22:19:42.571 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.571 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.571 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.571 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.571 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.572 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.572 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "internal - do not modify", 2024-05-12T22:19:42.572 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.572 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.572 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.572 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.572 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": { 2024-05-12T22:19:42.572 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mode", 2024-05-12T22:19:42.572 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.572 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.573 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.573 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "root", 2024-05-12T22:19:42.573 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.573 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.573 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:42.573 INFO:teuthology.orchestra.run.smithi028.stdout: "cephadm-package", 2024-05-12T22:19:42.573 INFO:teuthology.orchestra.run.smithi028.stdout: "root" 2024-05-12T22:19:42.573 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:42.574 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "mode for remote execution of cephadm", 2024-05-12T22:19:42.574 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.574 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.574 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.574 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.574 INFO:teuthology.orchestra.run.smithi028.stdout: "oob_default_addr": { 2024-05-12T22:19:42.574 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "oob_default_addr", 2024-05-12T22:19:42.574 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.574 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.575 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.575 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "169.254.1.1", 2024-05-12T22:19:42.575 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.575 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.575 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.575 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Default address for RedFish API (oob management).", 2024-05-12T22:19:42.575 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.575 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.575 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.576 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.576 INFO:teuthology.orchestra.run.smithi028.stdout: "prometheus_alerts_path": { 2024-05-12T22:19:42.576 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "prometheus_alerts_path", 2024-05-12T22:19:42.576 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.576 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.576 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.576 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-05-12T22:19:42.576 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.576 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.577 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.577 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-05-12T22:19:42.577 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.577 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.577 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.577 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.577 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_insecure": { 2024-05-12T22:19:42.577 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_insecure", 2024-05-12T22:19:42.578 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.578 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.578 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.578 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.578 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.578 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.578 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.578 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-05-12T22:19:42.578 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.579 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.579 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.579 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.579 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_password": { 2024-05-12T22:19:42.579 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_password", 2024-05-12T22:19:42.579 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.579 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.579 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.580 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.580 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.580 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.580 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.580 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-05-12T22:19:42.580 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.580 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.580 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.580 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.581 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_url": { 2024-05-12T22:19:42.581 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_url", 2024-05-12T22:19:42.581 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.581 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.581 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.581 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.581 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.581 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.582 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.582 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-05-12T22:19:42.582 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.582 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.582 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.582 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.582 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_username": { 2024-05-12T22:19:42.582 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_username", 2024-05-12T22:19:42.582 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.583 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.583 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.583 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.583 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.583 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.583 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.583 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-05-12T22:19:42.583 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.584 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.584 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.584 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.584 INFO:teuthology.orchestra.run.smithi028.stdout: "secure_monitoring_stack": { 2024-05-12T22:19:42.584 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "secure_monitoring_stack", 2024-05-12T22:19:42.584 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.584 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.584 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.584 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.585 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.585 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.585 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.585 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-05-12T22:19:42.585 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.585 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.585 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.585 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.585 INFO:teuthology.orchestra.run.smithi028.stdout: "service_discovery_port": { 2024-05-12T22:19:42.586 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "service_discovery_port", 2024-05-12T22:19:42.586 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-12T22:19:42.586 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.586 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.586 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8765", 2024-05-12T22:19:42.586 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.586 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.586 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.587 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "cephadm service discovery port", 2024-05-12T22:19:42.587 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.587 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.587 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.587 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.587 INFO:teuthology.orchestra.run.smithi028.stdout: "ssh_config_file": { 2024-05-12T22:19:42.587 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssh_config_file", 2024-05-12T22:19:42.587 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.588 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.588 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.588 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.588 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.588 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.588 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.588 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-05-12T22:19:42.588 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.588 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.588 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.589 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.589 INFO:teuthology.orchestra.run.smithi028.stdout: "use_agent": { 2024-05-12T22:19:42.589 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "use_agent", 2024-05-12T22:19:42.589 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.589 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.589 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.589 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.589 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.589 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.590 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.590 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-05-12T22:19:42.590 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.590 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.590 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.590 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.590 INFO:teuthology.orchestra.run.smithi028.stdout: "use_repo_digest": { 2024-05-12T22:19:42.590 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "use_repo_digest", 2024-05-12T22:19:42.591 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.591 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.591 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.591 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-12T22:19:42.591 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.591 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.591 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.591 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-05-12T22:19:42.592 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.592 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.592 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.592 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.592 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_on_failed_host_check": { 2024-05-12T22:19:42.592 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_on_failed_host_check", 2024-05-12T22:19:42.592 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.592 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.592 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.593 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-12T22:19:42.593 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.593 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.593 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.593 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise a health warning if the host check fails", 2024-05-12T22:19:42.593 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.593 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.593 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.594 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.594 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_on_stray_daemons": { 2024-05-12T22:19:42.594 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_on_stray_daemons", 2024-05-12T22:19:42.594 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.594 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.594 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.594 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-12T22:19:42.594 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.594 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.595 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.595 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-05-12T22:19:42.595 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.595 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.595 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.595 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.595 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_on_stray_hosts": { 2024-05-12T22:19:42.595 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_on_stray_hosts", 2024-05-12T22:19:42.596 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.596 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.596 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.596 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-12T22:19:42.596 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.596 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.596 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.596 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-05-12T22:19:42.596 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.597 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.597 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.597 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:42.597 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:42.597 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.597 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-12T22:19:42.597 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crash", 2024-05-12T22:19:42.597 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-12T22:19:42.597 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-12T22:19:42.598 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-12T22:19:42.598 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-12T22:19:42.598 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-12T22:19:42.598 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.598 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.598 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.598 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.598 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.599 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.599 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:42.599 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:42.599 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:42.599 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:42.599 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:42.599 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:42.599 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:42.600 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:42.600 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.600 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.600 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.600 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.600 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.600 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-12T22:19:42.600 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-12T22:19:42.600 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.601 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.601 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.601 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.601 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.601 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.601 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.601 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.601 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.602 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.602 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.602 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.602 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-12T22:19:42.602 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-12T22:19:42.602 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.602 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.602 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.603 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-12T22:19:42.603 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.603 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.603 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:42.603 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:42.603 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:42.603 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:42.603 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:42.603 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:42.604 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:42.604 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:42.604 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.604 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.604 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.604 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.604 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.604 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-12T22:19:42.604 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-12T22:19:42.605 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.605 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.605 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.605 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.605 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.605 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.605 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.605 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.606 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.606 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.606 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.606 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.606 INFO:teuthology.orchestra.run.smithi028.stdout: "retain_interval": { 2024-05-12T22:19:42.606 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "retain_interval", 2024-05-12T22:19:42.606 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-05-12T22:19:42.606 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.606 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.607 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "31536000", 2024-05-12T22:19:42.607 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.607 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.607 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.607 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how long to retain crashes before pruning them", 2024-05-12T22:19:42.607 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.607 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.607 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.607 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.608 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_recent_interval": { 2024-05-12T22:19:42.608 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_recent_interval", 2024-05-12T22:19:42.608 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-05-12T22:19:42.608 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.608 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.608 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "1209600", 2024-05-12T22:19:42.608 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.609 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.609 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.609 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "time interval in which to warn about recent crashes", 2024-05-12T22:19:42.609 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.609 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.609 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.609 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:42.609 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:42.610 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.610 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-12T22:19:42.610 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "dashboard", 2024-05-12T22:19:42.610 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-12T22:19:42.610 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-12T22:19:42.610 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-12T22:19:42.610 INFO:teuthology.orchestra.run.smithi028.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-05-12T22:19:42.610 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-05-12T22:19:42.610 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-12T22:19:42.611 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.611 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.611 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-05-12T22:19:42.611 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.611 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.611 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.611 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.611 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.611 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.612 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.612 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.612 INFO:teuthology.orchestra.run.smithi028.stdout: "ALERTMANAGER_API_HOST": { 2024-05-12T22:19:42.612 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ALERTMANAGER_API_HOST", 2024-05-12T22:19:42.612 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.612 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.612 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.612 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.612 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.613 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.613 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.613 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.613 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.613 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.613 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.613 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.613 INFO:teuthology.orchestra.run.smithi028.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-05-12T22:19:42.613 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-05-12T22:19:42.614 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.614 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.614 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.614 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-12T22:19:42.614 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.614 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.614 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.614 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.615 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.615 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.615 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.615 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.615 INFO:teuthology.orchestra.run.smithi028.stdout: "AUDIT_API_ENABLED": { 2024-05-12T22:19:42.615 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "AUDIT_API_ENABLED", 2024-05-12T22:19:42.615 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.615 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.616 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.616 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.616 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.616 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.616 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.616 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.616 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.616 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.616 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.617 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.617 INFO:teuthology.orchestra.run.smithi028.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-05-12T22:19:42.617 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-05-12T22:19:42.617 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.617 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.617 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.617 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-12T22:19:42.617 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.617 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.618 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.618 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.618 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.618 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.618 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.618 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.618 INFO:teuthology.orchestra.run.smithi028.stdout: "ENABLE_BROWSABLE_API": { 2024-05-12T22:19:42.618 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ENABLE_BROWSABLE_API", 2024-05-12T22:19:42.619 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.619 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.619 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.619 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-12T22:19:42.619 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.619 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.619 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.619 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.619 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.620 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.620 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.620 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.620 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-05-12T22:19:42.620 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-05-12T22:19:42.620 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.620 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.620 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.621 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-12T22:19:42.621 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.621 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.621 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.621 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.621 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.621 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.621 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.622 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.622 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-05-12T22:19:42.622 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-05-12T22:19:42.622 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.622 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.622 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.622 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-12T22:19:42.622 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.622 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.623 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.623 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.623 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.623 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.623 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.623 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.623 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-05-12T22:19:42.623 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-05-12T22:19:42.623 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.624 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.624 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.624 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-12T22:19:42.624 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.624 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.624 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.624 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.624 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.625 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.625 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.625 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.625 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-05-12T22:19:42.625 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-05-12T22:19:42.625 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.625 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.625 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.625 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-12T22:19:42.626 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.626 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.626 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.626 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.626 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.626 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.626 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.626 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.627 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_NFS": { 2024-05-12T22:19:42.627 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-05-12T22:19:42.627 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.627 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.627 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.627 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-12T22:19:42.627 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.627 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.627 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.628 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.628 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.628 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.628 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.628 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.628 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_RBD": { 2024-05-12T22:19:42.628 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-05-12T22:19:42.628 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.628 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.629 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.629 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-12T22:19:42.629 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.629 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.629 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.629 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.629 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.629 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.629 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.629 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.630 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_RGW": { 2024-05-12T22:19:42.630 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-05-12T22:19:42.630 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.630 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.630 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.630 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-12T22:19:42.630 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.630 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.631 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.631 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.631 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.631 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.631 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.631 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.631 INFO:teuthology.orchestra.run.smithi028.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-05-12T22:19:42.631 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-05-12T22:19:42.631 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.632 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.632 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.632 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.632 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.632 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.632 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.632 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.632 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.633 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.633 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.633 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.633 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_PASSWORD": { 2024-05-12T22:19:42.633 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_PASSWORD", 2024-05-12T22:19:42.633 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.633 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.633 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.633 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "admin", 2024-05-12T22:19:42.634 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.634 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.634 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.634 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.634 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.634 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.634 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.634 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.634 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-05-12T22:19:42.635 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-05-12T22:19:42.635 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.635 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.635 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.635 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-12T22:19:42.635 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.635 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.635 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.636 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.636 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.636 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.636 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.636 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.636 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_URL": { 2024-05-12T22:19:42.636 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_URL", 2024-05-12T22:19:42.636 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.637 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.637 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.637 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.637 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.637 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.637 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.637 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.637 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.637 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.638 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.638 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.638 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_USERNAME": { 2024-05-12T22:19:42.638 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_USERNAME", 2024-05-12T22:19:42.638 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.638 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.638 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.638 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "admin", 2024-05-12T22:19:42.638 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.639 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.639 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.639 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.639 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.639 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.639 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.639 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.639 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-05-12T22:19:42.640 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-05-12T22:19:42.640 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.640 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.640 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.640 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.640 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.640 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.640 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.640 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.641 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.641 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.641 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.641 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.641 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-05-12T22:19:42.641 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-05-12T22:19:42.641 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.641 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.642 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.642 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.642 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.642 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.642 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.642 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.642 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.642 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.643 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.643 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.643 INFO:teuthology.orchestra.run.smithi028.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-05-12T22:19:42.643 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-05-12T22:19:42.643 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.643 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.643 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.643 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-12T22:19:42.643 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.644 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.644 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.644 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.644 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.644 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.644 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.644 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.644 INFO:teuthology.orchestra.run.smithi028.stdout: "ISSUE_TRACKER_API_KEY": { 2024-05-12T22:19:42.644 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-05-12T22:19:42.645 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.645 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.645 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.645 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.645 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.645 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.645 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.645 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.646 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.646 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.646 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.646 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.646 INFO:teuthology.orchestra.run.smithi028.stdout: "PROMETHEUS_API_HOST": { 2024-05-12T22:19:42.646 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PROMETHEUS_API_HOST", 2024-05-12T22:19:42.646 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.646 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.647 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.647 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.647 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.647 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.647 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.647 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.647 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.647 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.647 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.648 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.648 INFO:teuthology.orchestra.run.smithi028.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-05-12T22:19:42.648 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-05-12T22:19:42.648 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.648 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.648 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.648 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-12T22:19:42.648 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.648 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.649 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.649 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.649 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.649 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.649 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.649 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.649 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-05-12T22:19:42.649 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-05-12T22:19:42.650 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.650 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.650 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.650 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.650 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.650 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.650 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.650 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.650 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.651 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.651 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.651 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.651 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-05-12T22:19:42.651 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-05-12T22:19:42.651 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.651 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.651 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.651 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.652 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.652 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.652 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.652 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.652 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.652 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.652 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.652 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.653 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-05-12T22:19:42.653 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-05-12T22:19:42.653 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.653 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.653 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.653 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-12T22:19:42.653 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.653 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.654 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.654 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.654 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.654 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.654 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.654 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.654 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-05-12T22:19:42.654 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-05-12T22:19:42.655 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.655 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.655 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.655 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-12T22:19:42.655 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.655 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.655 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.655 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.655 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.656 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.656 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.656 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.656 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-05-12T22:19:42.656 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-05-12T22:19:42.656 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.656 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.656 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.656 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.657 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.657 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.657 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.657 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.657 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.657 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.657 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.657 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.658 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-05-12T22:19:42.658 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-05-12T22:19:42.658 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.658 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.658 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.658 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.658 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.658 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.658 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.659 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.659 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.659 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.659 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.659 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.659 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-05-12T22:19:42.659 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-05-12T22:19:42.659 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.659 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.660 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.660 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.660 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.660 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.660 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.660 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.660 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.660 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.661 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.661 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.661 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_ENABLED": { 2024-05-12T22:19:42.661 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_ENABLED", 2024-05-12T22:19:42.661 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.661 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.661 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.661 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-12T22:19:42.661 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.662 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.662 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.662 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.662 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.662 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.662 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.662 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.662 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-05-12T22:19:42.663 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-05-12T22:19:42.663 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.663 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.663 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.663 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-05-12T22:19:42.663 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.663 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.663 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.663 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.664 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.664 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.664 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.664 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.664 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-05-12T22:19:42.664 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-05-12T22:19:42.664 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-12T22:19:42.664 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.664 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.665 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-05-12T22:19:42.665 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.665 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.665 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.665 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.665 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.665 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.665 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.666 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.666 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-05-12T22:19:42.666 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-05-12T22:19:42.666 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-12T22:19:42.666 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.666 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.666 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8", 2024-05-12T22:19:42.666 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.667 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.667 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.667 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.667 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.667 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.667 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.667 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.667 INFO:teuthology.orchestra.run.smithi028.stdout: "REST_REQUESTS_TIMEOUT": { 2024-05-12T22:19:42.667 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-05-12T22:19:42.668 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-12T22:19:42.668 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.668 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.668 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "45", 2024-05-12T22:19:42.668 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.668 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.668 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.668 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.669 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.669 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.669 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.669 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.669 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_ACCESS_KEY": { 2024-05-12T22:19:42.669 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_ACCESS_KEY", 2024-05-12T22:19:42.669 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.669 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.670 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.670 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.670 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.670 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.670 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.670 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.670 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.670 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.670 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.671 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.671 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-05-12T22:19:42.671 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-05-12T22:19:42.671 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.671 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.671 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.671 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "admin", 2024-05-12T22:19:42.671 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.671 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.672 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.672 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.672 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.672 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.672 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.672 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.672 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_SECRET_KEY": { 2024-05-12T22:19:42.672 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_SECRET_KEY", 2024-05-12T22:19:42.672 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.673 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.673 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.673 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.673 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.673 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.673 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.673 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.673 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.673 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.674 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.674 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.674 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_SSL_VERIFY": { 2024-05-12T22:19:42.674 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_SSL_VERIFY", 2024-05-12T22:19:42.674 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.674 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.674 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.674 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-12T22:19:42.675 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.675 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.675 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.675 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.675 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.675 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.675 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.675 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.675 INFO:teuthology.orchestra.run.smithi028.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-05-12T22:19:42.676 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-05-12T22:19:42.676 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-12T22:19:42.676 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.676 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.676 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-05-12T22:19:42.676 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.676 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.676 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.676 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.677 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.677 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.677 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.677 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.677 INFO:teuthology.orchestra.run.smithi028.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-05-12T22:19:42.677 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-05-12T22:19:42.677 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-12T22:19:42.677 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.678 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.678 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-05-12T22:19:42.678 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.678 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.678 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.678 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.678 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.678 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.678 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.679 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.679 INFO:teuthology.orchestra.run.smithi028.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-05-12T22:19:42.679 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-05-12T22:19:42.679 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-12T22:19:42.679 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.679 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.679 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-05-12T22:19:42.679 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.680 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.680 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.680 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.680 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.680 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.680 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.680 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.680 INFO:teuthology.orchestra.run.smithi028.stdout: "cross_origin_url": { 2024-05-12T22:19:42.680 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cross_origin_url", 2024-05-12T22:19:42.681 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.681 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.681 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.681 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.681 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.681 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.681 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.681 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.681 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.682 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.682 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.682 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.682 INFO:teuthology.orchestra.run.smithi028.stdout: "crt_file": { 2024-05-12T22:19:42.682 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crt_file", 2024-05-12T22:19:42.682 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.682 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.682 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.683 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.683 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.683 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.683 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.683 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.683 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.683 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.683 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.684 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.684 INFO:teuthology.orchestra.run.smithi028.stdout: "debug": { 2024-05-12T22:19:42.684 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "debug", 2024-05-12T22:19:42.684 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.684 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.684 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.684 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.684 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.684 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.685 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.685 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Enable/disable debug options", 2024-05-12T22:19:42.685 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.685 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.685 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.685 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.685 INFO:teuthology.orchestra.run.smithi028.stdout: "jwt_token_ttl": { 2024-05-12T22:19:42.685 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "jwt_token_ttl", 2024-05-12T22:19:42.686 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-12T22:19:42.686 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.686 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.686 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "28800", 2024-05-12T22:19:42.686 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.686 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.686 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.686 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.687 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.687 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.687 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.687 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.687 INFO:teuthology.orchestra.run.smithi028.stdout: "key_file": { 2024-05-12T22:19:42.687 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "key_file", 2024-05-12T22:19:42.687 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.687 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.687 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.688 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.688 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.688 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.688 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.688 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.688 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.688 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.688 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.689 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.689 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-12T22:19:42.689 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-12T22:19:42.689 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.689 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.689 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.689 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.689 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.689 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.690 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:42.690 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:42.690 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:42.690 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:42.690 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:42.690 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:42.690 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:42.690 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:42.691 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.691 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.691 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.691 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.691 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.691 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-12T22:19:42.691 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-12T22:19:42.691 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.691 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.692 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.692 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.692 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.692 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.692 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.692 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.692 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.692 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.692 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.693 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.693 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-12T22:19:42.693 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-12T22:19:42.693 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.693 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.693 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.693 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-12T22:19:42.693 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.694 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.694 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:42.694 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:42.694 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:42.694 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:42.694 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:42.694 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:42.694 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:42.694 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:42.695 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.695 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.695 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.695 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.695 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.695 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-12T22:19:42.695 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-12T22:19:42.695 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.695 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.696 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.696 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.696 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.696 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.696 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.696 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.696 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.696 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.696 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.697 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.697 INFO:teuthology.orchestra.run.smithi028.stdout: "motd": { 2024-05-12T22:19:42.697 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "motd", 2024-05-12T22:19:42.697 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.697 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.697 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.697 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.697 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.697 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.698 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.698 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "The message of the day", 2024-05-12T22:19:42.698 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.698 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.698 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.698 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.698 INFO:teuthology.orchestra.run.smithi028.stdout: "redirect_resolve_ip_addr": { 2024-05-12T22:19:42.698 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "redirect_resolve_ip_addr", 2024-05-12T22:19:42.699 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.699 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.699 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.699 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.699 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.699 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.699 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.699 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.699 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.700 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.700 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.700 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.700 INFO:teuthology.orchestra.run.smithi028.stdout: "server_addr": { 2024-05-12T22:19:42.700 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_addr", 2024-05-12T22:19:42.700 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.700 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.700 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.701 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "::", 2024-05-12T22:19:42.701 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.701 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.701 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.701 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.701 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.701 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.701 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.701 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.702 INFO:teuthology.orchestra.run.smithi028.stdout: "server_port": { 2024-05-12T22:19:42.702 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_port", 2024-05-12T22:19:42.702 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-12T22:19:42.702 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.702 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.702 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8080", 2024-05-12T22:19:42.702 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.702 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.703 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.703 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.703 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.703 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.703 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.703 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.703 INFO:teuthology.orchestra.run.smithi028.stdout: "ssl": { 2024-05-12T22:19:42.703 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssl", 2024-05-12T22:19:42.703 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.704 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.704 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.704 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-12T22:19:42.704 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.704 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.704 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.704 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.704 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.704 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.705 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.705 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.705 INFO:teuthology.orchestra.run.smithi028.stdout: "ssl_server_port": { 2024-05-12T22:19:42.705 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssl_server_port", 2024-05-12T22:19:42.705 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-12T22:19:42.705 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.705 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.705 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8443", 2024-05-12T22:19:42.705 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.705 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.706 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.706 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.706 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.706 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.706 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.706 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.706 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_behaviour": { 2024-05-12T22:19:42.706 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_behaviour", 2024-05-12T22:19:42.707 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.707 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.707 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.707 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "redirect", 2024-05-12T22:19:42.707 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.707 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.707 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:42.707 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:42.708 INFO:teuthology.orchestra.run.smithi028.stdout: "redirect" 2024-05-12T22:19:42.708 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:42.708 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.708 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.708 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.708 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.708 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.708 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_error_status_code": { 2024-05-12T22:19:42.708 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_error_status_code", 2024-05-12T22:19:42.709 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-12T22:19:42.709 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.709 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.709 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "500", 2024-05-12T22:19:42.709 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "400", 2024-05-12T22:19:42.709 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "599", 2024-05-12T22:19:42.709 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.709 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.710 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.710 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.710 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.710 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.710 INFO:teuthology.orchestra.run.smithi028.stdout: "url_prefix": { 2024-05-12T22:19:42.710 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "url_prefix", 2024-05-12T22:19:42.710 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.710 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.710 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.711 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.711 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.711 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.711 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.711 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.711 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.711 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.711 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.711 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:42.712 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:42.712 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.712 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-12T22:19:42.712 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "devicehealth", 2024-05-12T22:19:42.712 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-12T22:19:42.712 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-12T22:19:42.712 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-12T22:19:42.712 INFO:teuthology.orchestra.run.smithi028.stdout: "enable_monitoring": { 2024-05-12T22:19:42.712 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enable_monitoring", 2024-05-12T22:19:42.713 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.713 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.713 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.713 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-12T22:19:42.713 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.713 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.713 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.713 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "monitor device health metrics", 2024-05-12T22:19:42.714 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.714 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.714 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.714 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.714 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-12T22:19:42.714 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-12T22:19:42.714 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.714 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.714 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.715 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.715 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.715 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.715 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:42.715 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:42.715 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:42.715 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:42.715 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:42.715 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:42.716 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:42.716 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:42.716 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.716 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.716 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.716 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.716 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.716 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-12T22:19:42.716 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-12T22:19:42.717 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.717 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.717 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.717 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.717 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.717 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.717 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.717 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.718 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.718 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.718 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.718 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.718 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-12T22:19:42.718 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-12T22:19:42.718 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.718 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.718 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.719 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-12T22:19:42.719 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.719 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.719 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:42.719 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:42.719 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:42.719 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:42.719 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:42.719 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:42.720 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:42.720 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:42.720 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.720 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.720 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.720 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.720 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.720 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-12T22:19:42.720 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-12T22:19:42.721 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.721 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.721 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.721 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.721 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.721 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.721 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.721 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.722 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.722 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.722 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.722 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.722 INFO:teuthology.orchestra.run.smithi028.stdout: "mark_out_threshold": { 2024-05-12T22:19:42.722 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mark_out_threshold", 2024-05-12T22:19:42.722 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-05-12T22:19:42.722 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.722 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.723 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "2419200", 2024-05-12T22:19:42.723 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.723 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.723 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.723 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-05-12T22:19:42.723 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.723 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.723 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.723 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.724 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": { 2024-05-12T22:19:42.724 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pool_name", 2024-05-12T22:19:42.724 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.724 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.724 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.724 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "device_health_metrics", 2024-05-12T22:19:42.724 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.724 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.724 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.725 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "name of pool in which to store device health metrics", 2024-05-12T22:19:42.725 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.725 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.725 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.725 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.725 INFO:teuthology.orchestra.run.smithi028.stdout: "retention_period": { 2024-05-12T22:19:42.725 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "retention_period", 2024-05-12T22:19:42.725 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-05-12T22:19:42.725 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.726 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.726 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "15552000", 2024-05-12T22:19:42.726 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.726 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.726 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.726 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how long to retain device health metrics", 2024-05-12T22:19:42.726 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.726 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.727 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.727 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.727 INFO:teuthology.orchestra.run.smithi028.stdout: "scrape_frequency": { 2024-05-12T22:19:42.727 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "scrape_frequency", 2024-05-12T22:19:42.727 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-05-12T22:19:42.727 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.727 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.727 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "86400", 2024-05-12T22:19:42.727 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.727 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.728 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.728 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to scrape device health metrics", 2024-05-12T22:19:42.728 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.728 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.728 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.728 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.728 INFO:teuthology.orchestra.run.smithi028.stdout: "self_heal": { 2024-05-12T22:19:42.728 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "self_heal", 2024-05-12T22:19:42.729 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.729 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.729 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.729 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-12T22:19:42.729 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.729 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.729 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.729 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-05-12T22:19:42.729 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.730 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.730 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.730 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.730 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-05-12T22:19:42.730 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-05-12T22:19:42.730 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-05-12T22:19:42.730 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.730 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.731 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-05-12T22:19:42.731 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.731 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.731 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.731 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to wake up and check device health", 2024-05-12T22:19:42.731 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.731 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.731 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.731 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.732 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_threshold": { 2024-05-12T22:19:42.732 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_threshold", 2024-05-12T22:19:42.732 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-05-12T22:19:42.732 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.732 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.732 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "7257600", 2024-05-12T22:19:42.732 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.732 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.732 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.733 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-05-12T22:19:42.733 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.733 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.733 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.733 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:42.733 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:42.733 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.733 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-12T22:19:42.733 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "diskprediction_local", 2024-05-12T22:19:42.734 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-12T22:19:42.734 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-12T22:19:42.734 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-12T22:19:42.734 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-12T22:19:42.734 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-12T22:19:42.734 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.734 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.734 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.735 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.735 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.735 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.735 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:42.735 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:42.735 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:42.735 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:42.735 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:42.735 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:42.736 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:42.736 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:42.736 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.736 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.736 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.736 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.736 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.736 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-12T22:19:42.736 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-12T22:19:42.737 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.737 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.737 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.737 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.737 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.737 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.737 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.737 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.737 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.738 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.738 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.738 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.738 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-12T22:19:42.738 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-12T22:19:42.738 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.738 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.738 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.739 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-12T22:19:42.739 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.739 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.739 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:42.739 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:42.739 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:42.739 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:42.739 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:42.739 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:42.740 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:42.740 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:42.740 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.740 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.740 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.740 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.740 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.740 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-12T22:19:42.740 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-12T22:19:42.741 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.741 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.741 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.741 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.741 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.741 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.741 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.741 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.741 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.742 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.742 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.742 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.742 INFO:teuthology.orchestra.run.smithi028.stdout: "predict_interval": { 2024-05-12T22:19:42.742 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "predict_interval", 2024-05-12T22:19:42.742 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.742 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.742 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.742 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "86400", 2024-05-12T22:19:42.743 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.743 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.743 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.743 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.743 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.743 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.743 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.743 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.743 INFO:teuthology.orchestra.run.smithi028.stdout: "predictor_model": { 2024-05-12T22:19:42.744 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "predictor_model", 2024-05-12T22:19:42.744 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.744 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.744 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.744 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "prophetstor", 2024-05-12T22:19:42.744 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.744 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.744 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.745 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.745 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.745 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.745 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.745 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.745 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-05-12T22:19:42.745 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-05-12T22:19:42.745 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.745 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.746 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.746 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-05-12T22:19:42.746 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.746 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.746 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.746 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.746 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.746 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.747 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.747 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:42.747 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:42.747 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.747 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-12T22:19:42.747 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "influx", 2024-05-12T22:19:42.747 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": false, 2024-05-12T22:19:42.747 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "influxdb python module not found", 2024-05-12T22:19:42.747 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-12T22:19:42.748 INFO:teuthology.orchestra.run.smithi028.stdout: "batch_size": { 2024-05-12T22:19:42.748 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "batch_size", 2024-05-12T22:19:42.748 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-12T22:19:42.748 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.748 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.748 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5000", 2024-05-12T22:19:42.748 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.748 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.748 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.749 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-05-12T22:19:42.749 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.749 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.749 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.749 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.749 INFO:teuthology.orchestra.run.smithi028.stdout: "database": { 2024-05-12T22:19:42.749 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "database", 2024-05-12T22:19:42.749 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.750 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.750 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.750 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "ceph", 2024-05-12T22:19:42.750 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.750 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.750 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.750 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-05-12T22:19:42.750 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.751 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.751 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.751 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.751 INFO:teuthology.orchestra.run.smithi028.stdout: "hostname": { 2024-05-12T22:19:42.751 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "hostname", 2024-05-12T22:19:42.751 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.751 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.751 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.751 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.752 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.752 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.752 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.752 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "InfluxDB server hostname", 2024-05-12T22:19:42.752 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.752 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.752 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.752 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.753 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-05-12T22:19:42.753 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-05-12T22:19:42.753 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-05-12T22:19:42.753 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.753 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.753 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "30", 2024-05-12T22:19:42.753 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "5", 2024-05-12T22:19:42.753 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.754 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.754 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-05-12T22:19:42.754 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.754 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.754 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.754 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.754 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-12T22:19:42.754 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-12T22:19:42.754 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.755 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.755 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.755 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.755 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.755 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.755 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:42.755 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:42.755 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:42.755 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:42.756 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:42.756 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:42.756 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:42.756 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:42.756 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.756 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.756 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.756 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.757 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.757 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-12T22:19:42.757 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-12T22:19:42.757 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.757 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.757 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.757 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.757 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.757 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.758 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.758 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.758 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.758 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.758 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.758 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.758 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-12T22:19:42.758 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-12T22:19:42.758 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.759 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.759 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.759 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-12T22:19:42.759 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.759 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.759 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:42.759 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:42.759 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:42.759 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:42.760 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:42.760 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:42.760 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:42.760 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:42.760 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.760 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.760 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.760 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.761 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.761 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-12T22:19:42.761 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-12T22:19:42.761 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.761 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.761 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.761 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.761 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.761 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.762 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.762 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.762 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.762 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.762 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.762 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.762 INFO:teuthology.orchestra.run.smithi028.stdout: "password": { 2024-05-12T22:19:42.762 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "password", 2024-05-12T22:19:42.762 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.763 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.763 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.763 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.763 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.763 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.763 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.763 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "password of InfluxDB server user", 2024-05-12T22:19:42.763 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.763 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.764 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.764 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.764 INFO:teuthology.orchestra.run.smithi028.stdout: "port": { 2024-05-12T22:19:42.764 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "port", 2024-05-12T22:19:42.764 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-12T22:19:42.764 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.764 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.764 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8086", 2024-05-12T22:19:42.765 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.765 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.765 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.765 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "InfluxDB server port", 2024-05-12T22:19:42.765 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.765 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.765 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.765 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.765 INFO:teuthology.orchestra.run.smithi028.stdout: "ssl": { 2024-05-12T22:19:42.766 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssl", 2024-05-12T22:19:42.766 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.766 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.766 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.766 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "false", 2024-05-12T22:19:42.766 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.766 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.766 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.766 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-05-12T22:19:42.767 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.767 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.767 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.767 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.767 INFO:teuthology.orchestra.run.smithi028.stdout: "threads": { 2024-05-12T22:19:42.767 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "threads", 2024-05-12T22:19:42.767 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-12T22:19:42.767 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.768 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.768 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-05-12T22:19:42.768 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-05-12T22:19:42.768 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "32", 2024-05-12T22:19:42.768 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.768 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-05-12T22:19:42.768 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.768 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.768 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.769 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.769 INFO:teuthology.orchestra.run.smithi028.stdout: "username": { 2024-05-12T22:19:42.769 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "username", 2024-05-12T22:19:42.769 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.769 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.769 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.769 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.769 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.770 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.770 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.770 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "username of InfluxDB server user", 2024-05-12T22:19:42.770 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.770 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.770 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.770 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.770 INFO:teuthology.orchestra.run.smithi028.stdout: "verify_ssl": { 2024-05-12T22:19:42.771 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "verify_ssl", 2024-05-12T22:19:42.771 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.771 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.771 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.771 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "true", 2024-05-12T22:19:42.771 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.771 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.771 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.771 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-05-12T22:19:42.772 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.772 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.772 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.772 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:42.772 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:42.772 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.772 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-12T22:19:42.772 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "insights", 2024-05-12T22:19:42.773 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-12T22:19:42.773 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-12T22:19:42.773 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-12T22:19:42.773 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-12T22:19:42.773 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-12T22:19:42.773 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.773 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.773 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.774 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.774 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.774 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.774 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:42.774 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:42.774 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:42.774 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:42.774 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:42.774 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:42.775 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:42.775 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:42.775 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.775 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.775 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.775 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.775 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.775 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-12T22:19:42.776 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-12T22:19:42.776 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.776 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.776 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.776 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.776 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.776 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.776 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.776 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.777 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.777 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.777 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.777 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.777 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-12T22:19:42.777 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-12T22:19:42.777 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.777 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.778 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.778 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-12T22:19:42.778 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.778 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.778 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:42.778 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:42.778 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:42.778 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:42.778 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:42.779 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:42.779 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:42.779 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:42.779 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.779 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.779 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.779 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.779 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.780 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-12T22:19:42.780 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-12T22:19:42.780 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.780 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.780 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.780 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.780 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.780 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.780 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.781 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.781 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.781 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.781 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.781 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:42.781 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:42.781 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.781 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-12T22:19:42.781 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "iostat", 2024-05-12T22:19:42.782 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-12T22:19:42.782 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-12T22:19:42.782 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-12T22:19:42.782 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-12T22:19:42.782 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-12T22:19:42.782 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.782 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.782 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.782 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.783 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.783 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.783 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:42.783 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:42.783 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:42.783 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:42.783 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:42.783 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:42.783 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:42.784 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:42.784 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.784 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.784 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.784 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.784 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.784 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-12T22:19:42.784 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-12T22:19:42.785 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.785 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.785 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.785 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.785 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.785 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.785 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.785 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.785 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.786 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.786 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.786 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.786 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-12T22:19:42.786 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-12T22:19:42.786 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.786 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.786 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.786 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-12T22:19:42.787 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.787 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.787 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:42.787 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:42.787 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:42.787 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:42.787 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:42.787 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:42.787 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:42.788 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:42.788 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.788 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.788 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.788 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.788 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.788 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-12T22:19:42.788 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-12T22:19:42.788 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.789 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.789 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.789 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.789 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.789 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.789 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.789 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.789 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.789 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.790 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.790 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:42.790 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:42.790 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.790 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-12T22:19:42.790 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "k8sevents", 2024-05-12T22:19:42.790 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-12T22:19:42.790 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-12T22:19:42.790 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-12T22:19:42.791 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_event_retention_days": { 2024-05-12T22:19:42.791 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ceph_event_retention_days", 2024-05-12T22:19:42.791 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-12T22:19:42.791 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.791 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.791 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "7", 2024-05-12T22:19:42.791 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.791 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.791 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.792 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Days to hold ceph event information within local cache", 2024-05-12T22:19:42.792 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.792 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.792 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.792 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.792 INFO:teuthology.orchestra.run.smithi028.stdout: "config_check_secs": { 2024-05-12T22:19:42.792 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "config_check_secs", 2024-05-12T22:19:42.792 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-12T22:19:42.793 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.793 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.793 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-05-12T22:19:42.793 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "10", 2024-05-12T22:19:42.793 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.793 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.793 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-05-12T22:19:42.793 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.793 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.794 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.794 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.794 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-12T22:19:42.794 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-12T22:19:42.794 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.794 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.794 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.795 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.795 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.795 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.795 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:42.795 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:42.795 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:42.795 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:42.795 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:42.795 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:42.796 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:42.796 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:42.796 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.796 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.796 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.796 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.796 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.796 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-12T22:19:42.797 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-12T22:19:42.797 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.797 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.797 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.797 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.797 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.797 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.797 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.797 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.798 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.798 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.798 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.798 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.798 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-12T22:19:42.798 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-12T22:19:42.798 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.798 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.799 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.799 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-12T22:19:42.799 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.799 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.799 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:42.799 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:42.799 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:42.799 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:42.799 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:42.800 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:42.800 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:42.800 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:42.800 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.800 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.800 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.800 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.800 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.800 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-12T22:19:42.801 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-12T22:19:42.801 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.801 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.801 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.801 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.801 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.801 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.801 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.801 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.802 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.802 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.802 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.802 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:42.802 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:42.802 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.802 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-12T22:19:42.802 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "localpool", 2024-05-12T22:19:42.803 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-12T22:19:42.803 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-12T22:19:42.803 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-12T22:19:42.803 INFO:teuthology.orchestra.run.smithi028.stdout: "failure_domain": { 2024-05-12T22:19:42.803 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "failure_domain", 2024-05-12T22:19:42.803 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.803 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.803 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.803 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "host", 2024-05-12T22:19:42.804 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.804 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.804 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.804 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "failure domain for any created local pool", 2024-05-12T22:19:42.804 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-05-12T22:19:42.804 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.804 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.804 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.805 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-12T22:19:42.805 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-12T22:19:42.805 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.805 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.805 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.805 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.805 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.805 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.806 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:42.806 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:42.806 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:42.806 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:42.806 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:42.806 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:42.806 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:42.806 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:42.807 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.807 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.807 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.807 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.807 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.807 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-12T22:19:42.807 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-12T22:19:42.807 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.808 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.808 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.808 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.808 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.808 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.808 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.808 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.808 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.808 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.809 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.809 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.809 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-12T22:19:42.809 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-12T22:19:42.809 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.809 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.809 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.809 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-12T22:19:42.809 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.809 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.810 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:42.810 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:42.810 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:42.810 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:42.810 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:42.810 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:42.810 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:42.810 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:42.811 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.811 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.811 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.811 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.811 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.811 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-12T22:19:42.811 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-12T22:19:42.811 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.811 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.812 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.812 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.812 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.812 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.812 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.812 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.812 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.812 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.812 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.813 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.813 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": { 2024-05-12T22:19:42.813 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "min_size", 2024-05-12T22:19:42.813 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-12T22:19:42.813 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.813 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.813 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.813 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.813 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.814 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.814 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "default min_size for any created local pool", 2024-05-12T22:19:42.814 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-05-12T22:19:42.814 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.814 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.814 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.814 INFO:teuthology.orchestra.run.smithi028.stdout: "num_rep": { 2024-05-12T22:19:42.814 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "num_rep", 2024-05-12T22:19:42.814 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-12T22:19:42.815 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.815 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.815 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "3", 2024-05-12T22:19:42.815 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.815 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.815 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.815 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "default replica count for any created local pool", 2024-05-12T22:19:42.815 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.816 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.816 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.816 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.816 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": { 2024-05-12T22:19:42.816 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pg_num", 2024-05-12T22:19:42.816 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-12T22:19:42.816 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.816 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.817 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "128", 2024-05-12T22:19:42.817 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.817 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.817 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.817 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "default pg_num for any created local pool", 2024-05-12T22:19:42.817 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.817 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.818 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.818 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.818 INFO:teuthology.orchestra.run.smithi028.stdout: "prefix": { 2024-05-12T22:19:42.818 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "prefix", 2024-05-12T22:19:42.818 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.818 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.818 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.818 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.819 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.819 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.819 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.819 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "name prefix for any created local pool", 2024-05-12T22:19:42.819 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.819 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.819 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.819 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.819 INFO:teuthology.orchestra.run.smithi028.stdout: "subtree": { 2024-05-12T22:19:42.820 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "subtree", 2024-05-12T22:19:42.820 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.820 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.820 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.820 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "rack", 2024-05-12T22:19:42.820 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.820 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.820 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.820 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "CRUSH level for which to create a local pool", 2024-05-12T22:19:42.821 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-05-12T22:19:42.821 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.821 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.821 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:42.821 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:42.821 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.821 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-12T22:19:42.821 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mds_autoscaler", 2024-05-12T22:19:42.821 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-12T22:19:42.822 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-12T22:19:42.822 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-12T22:19:42.822 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-12T22:19:42.822 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-12T22:19:42.822 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.822 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.822 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.822 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.823 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.823 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.823 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:42.823 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:42.823 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:42.823 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:42.823 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:42.823 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:42.823 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:42.823 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:42.824 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.824 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.824 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.824 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.824 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.824 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-12T22:19:42.824 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-12T22:19:42.824 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.824 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.825 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.825 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.825 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.825 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.825 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.825 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.825 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.825 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.825 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.826 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.826 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-12T22:19:42.826 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-12T22:19:42.826 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.826 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.826 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.826 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-12T22:19:42.826 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.826 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.827 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:42.827 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:42.827 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:42.827 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:42.827 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:42.827 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:42.827 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:42.827 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:42.828 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.828 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.828 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.828 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.828 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.828 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-12T22:19:42.828 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-12T22:19:42.828 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.828 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.829 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.829 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.829 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.829 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.829 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.829 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.829 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.829 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.829 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.830 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:42.830 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:42.830 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.830 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-12T22:19:42.830 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mirroring", 2024-05-12T22:19:42.830 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-12T22:19:42.830 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-12T22:19:42.830 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-12T22:19:42.831 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-12T22:19:42.831 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-12T22:19:42.831 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.831 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.831 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.831 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.831 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.831 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.831 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:42.832 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:42.832 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:42.832 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:42.832 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:42.832 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:42.832 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:42.832 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:42.832 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.832 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.833 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.833 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.833 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.833 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-12T22:19:42.833 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-12T22:19:42.833 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.833 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.833 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.834 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.834 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.834 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.834 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.834 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.834 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.834 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.834 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.835 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.835 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-12T22:19:42.835 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-12T22:19:42.835 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.835 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.835 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.835 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-12T22:19:42.835 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.835 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.836 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:42.836 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:42.836 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:42.836 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:42.836 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:42.836 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:42.836 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:42.836 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:42.836 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.837 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.837 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.837 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.837 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.837 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-12T22:19:42.837 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-12T22:19:42.837 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.837 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.837 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.838 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.838 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.838 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.838 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.838 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.838 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.838 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.838 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.839 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:42.839 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:42.839 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.839 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-12T22:19:42.839 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "nfs", 2024-05-12T22:19:42.839 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-12T22:19:42.839 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-12T22:19:42.839 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-12T22:19:42.839 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-12T22:19:42.840 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-12T22:19:42.840 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.840 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.840 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.840 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.840 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.840 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.840 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:42.841 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:42.841 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:42.841 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:42.841 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:42.841 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:42.841 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:42.841 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:42.841 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.841 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.842 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.842 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.842 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.842 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-12T22:19:42.842 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-12T22:19:42.842 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.842 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.842 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.843 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.843 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.843 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.843 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.843 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.843 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.843 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.843 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.844 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.844 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-12T22:19:42.844 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-12T22:19:42.844 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.844 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.844 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.844 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-12T22:19:42.844 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.844 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.845 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:42.845 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:42.845 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:42.845 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:42.845 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:42.845 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:42.845 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:42.845 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:42.845 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.846 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.846 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.846 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.846 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.846 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-12T22:19:42.846 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-12T22:19:42.846 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.846 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.847 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.847 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.847 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.847 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.847 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.847 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.847 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.847 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.848 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.848 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:42.848 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:42.848 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.848 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-12T22:19:42.848 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "orchestrator", 2024-05-12T22:19:42.848 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-12T22:19:42.848 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-12T22:19:42.848 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-12T22:19:42.849 INFO:teuthology.orchestra.run.smithi028.stdout: "fail_fs": { 2024-05-12T22:19:42.849 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "fail_fs", 2024-05-12T22:19:42.849 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.849 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.849 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.849 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.849 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.849 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.850 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.850 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-05-12T22:19:42.850 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.850 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.850 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.850 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.850 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-12T22:19:42.850 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-12T22:19:42.851 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.851 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.851 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.851 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.851 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.851 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.851 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:42.851 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:42.851 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:42.852 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:42.852 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:42.852 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:42.852 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:42.852 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:42.852 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.852 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.852 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.852 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.853 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.853 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-12T22:19:42.853 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-12T22:19:42.853 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.853 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.853 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.853 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.853 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.854 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.854 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.854 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.854 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.854 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.854 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.854 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.854 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-12T22:19:42.855 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-12T22:19:42.855 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.855 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.855 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.855 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-12T22:19:42.855 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.855 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.855 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:42.855 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:42.856 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:42.856 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:42.856 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:42.856 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:42.856 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:42.856 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:42.856 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.856 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.857 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.857 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.857 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.857 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-12T22:19:42.857 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-12T22:19:42.857 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.857 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.857 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.857 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.858 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.858 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.858 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.858 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.858 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.858 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.858 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.858 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.858 INFO:teuthology.orchestra.run.smithi028.stdout: "orchestrator": { 2024-05-12T22:19:42.859 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "orchestrator", 2024-05-12T22:19:42.859 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.859 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.859 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.859 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.859 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.859 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.859 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:42.859 INFO:teuthology.orchestra.run.smithi028.stdout: "cephadm", 2024-05-12T22:19:42.860 INFO:teuthology.orchestra.run.smithi028.stdout: "rook", 2024-05-12T22:19:42.860 INFO:teuthology.orchestra.run.smithi028.stdout: "test_orchestrator" 2024-05-12T22:19:42.860 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:42.860 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Orchestrator backend", 2024-05-12T22:19:42.860 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.860 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.860 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.860 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:42.860 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:42.861 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.861 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-12T22:19:42.861 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "osd_perf_query", 2024-05-12T22:19:42.861 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-12T22:19:42.861 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-12T22:19:42.861 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-12T22:19:42.861 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-12T22:19:42.861 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-12T22:19:42.862 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.862 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.862 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.862 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.862 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.862 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.862 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:42.862 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:42.863 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:42.863 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:42.863 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:42.863 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:42.863 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:42.863 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:42.863 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.863 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.863 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.864 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.864 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.864 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-12T22:19:42.864 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-12T22:19:42.864 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.864 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.864 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.864 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.864 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.865 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.865 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.865 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.865 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.865 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.865 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.865 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.865 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-12T22:19:42.865 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-12T22:19:42.866 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.866 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.866 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.866 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-12T22:19:42.866 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.866 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.866 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:42.866 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:42.867 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:42.867 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:42.867 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:42.867 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:42.867 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:42.867 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:42.867 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.867 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.867 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.868 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.868 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.868 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-12T22:19:42.868 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-12T22:19:42.868 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.868 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.868 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.868 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.869 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.869 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.869 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.869 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.869 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.869 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.869 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.869 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:42.869 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:42.870 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.870 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-12T22:19:42.870 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "osd_support", 2024-05-12T22:19:42.870 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-12T22:19:42.870 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-12T22:19:42.870 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-12T22:19:42.870 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-12T22:19:42.870 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-12T22:19:42.871 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.871 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.871 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.871 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.871 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.871 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.871 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:42.871 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:42.871 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:42.872 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:42.872 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:42.872 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:42.872 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:42.872 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:42.872 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.872 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.872 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.873 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.873 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.873 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-12T22:19:42.873 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-12T22:19:42.873 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.873 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.873 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.873 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.873 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.874 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.874 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.874 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.874 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.874 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.874 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.874 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.874 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-12T22:19:42.875 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-12T22:19:42.875 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.875 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.875 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.875 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-12T22:19:42.875 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.875 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.875 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:42.875 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:42.876 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:42.876 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:42.876 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:42.876 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:42.876 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:42.876 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:42.876 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.876 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.876 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.877 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.877 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.877 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-12T22:19:42.877 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-12T22:19:42.877 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.877 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.877 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.877 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.877 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.878 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.878 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.878 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.878 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.878 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.878 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.878 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:42.878 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:42.879 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.879 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-12T22:19:42.879 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pg_autoscaler", 2024-05-12T22:19:42.879 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-12T22:19:42.879 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-12T22:19:42.879 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-12T22:19:42.879 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-12T22:19:42.879 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-12T22:19:42.880 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.880 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.880 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.880 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.880 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.880 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.880 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:42.880 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:42.880 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:42.881 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:42.881 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:42.881 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:42.881 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:42.881 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:42.881 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.881 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.881 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.881 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.882 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.882 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-12T22:19:42.882 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-12T22:19:42.882 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.882 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.882 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.882 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.882 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.883 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.883 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.883 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.883 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.883 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.883 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.883 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.883 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-12T22:19:42.883 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-12T22:19:42.884 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.884 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.884 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.884 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-12T22:19:42.884 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.884 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.884 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:42.884 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:42.885 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:42.885 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:42.885 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:42.885 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:42.885 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:42.885 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:42.885 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.885 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.885 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.886 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.886 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.886 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-12T22:19:42.886 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-12T22:19:42.886 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.886 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.886 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.886 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.887 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.887 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.887 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.887 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.887 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.887 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.887 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.887 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.887 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-05-12T22:19:42.888 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-05-12T22:19:42.888 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-05-12T22:19:42.888 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.888 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.888 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-05-12T22:19:42.888 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.888 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.888 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.888 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.889 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.889 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.889 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.889 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.889 INFO:teuthology.orchestra.run.smithi028.stdout: "threshold": { 2024-05-12T22:19:42.889 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "threshold", 2024-05-12T22:19:42.889 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-05-12T22:19:42.889 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.889 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.890 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "3.0", 2024-05-12T22:19:42.890 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1.0", 2024-05-12T22:19:42.890 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.890 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.890 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "scaling threshold", 2024-05-12T22:19:42.890 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-05-12T22:19:42.890 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.890 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.891 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:42.891 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:42.891 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.891 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-12T22:19:42.891 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "progress", 2024-05-12T22:19:42.891 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-12T22:19:42.891 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-12T22:19:42.891 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-12T22:19:42.891 INFO:teuthology.orchestra.run.smithi028.stdout: "allow_pg_recovery_event": { 2024-05-12T22:19:42.892 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "allow_pg_recovery_event", 2024-05-12T22:19:42.892 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.892 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.892 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.892 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.892 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.892 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.892 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.893 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "allow the module to show pg recovery progress", 2024-05-12T22:19:42.893 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.893 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.893 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.893 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.893 INFO:teuthology.orchestra.run.smithi028.stdout: "enabled": { 2024-05-12T22:19:42.893 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enabled", 2024-05-12T22:19:42.893 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.893 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.894 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.894 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-12T22:19:42.894 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.894 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.894 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.894 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.894 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.894 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.894 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.895 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.895 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-12T22:19:42.895 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-12T22:19:42.895 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.895 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.895 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.895 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.895 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.895 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.896 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:42.896 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:42.896 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:42.896 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:42.896 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:42.896 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:42.896 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:42.896 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:42.897 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.897 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.897 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.897 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.897 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.897 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-12T22:19:42.897 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-12T22:19:42.897 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.897 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.897 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.898 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.898 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.898 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.898 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.898 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.898 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.898 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.898 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.899 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.899 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-12T22:19:42.899 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-12T22:19:42.899 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.899 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.899 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.899 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-12T22:19:42.899 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.899 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.900 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:42.900 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:42.900 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:42.900 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:42.900 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:42.900 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:42.900 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:42.900 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:42.900 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.901 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.901 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.901 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.901 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.901 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-12T22:19:42.901 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-12T22:19:42.901 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.901 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.901 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.902 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.902 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.902 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.902 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.902 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.902 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.902 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.902 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.903 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.903 INFO:teuthology.orchestra.run.smithi028.stdout: "max_completed_events": { 2024-05-12T22:19:42.903 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_completed_events", 2024-05-12T22:19:42.903 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-12T22:19:42.903 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.903 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.903 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "50", 2024-05-12T22:19:42.903 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.903 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.904 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.904 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "number of past completed events to remember", 2024-05-12T22:19:42.904 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.904 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.904 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.904 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.904 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-05-12T22:19:42.904 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-05-12T22:19:42.904 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-05-12T22:19:42.905 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.905 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.905 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-05-12T22:19:42.905 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.905 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.905 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.905 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how long the module is going to sleep", 2024-05-12T22:19:42.905 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.906 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.906 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.906 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:42.906 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:42.906 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.906 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-12T22:19:42.906 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "prometheus", 2024-05-12T22:19:42.906 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-12T22:19:42.907 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-12T22:19:42.907 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-12T22:19:42.907 INFO:teuthology.orchestra.run.smithi028.stdout: "cache": { 2024-05-12T22:19:42.907 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cache", 2024-05-12T22:19:42.907 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.907 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.907 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.907 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-12T22:19:42.908 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.908 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.908 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.908 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.908 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.908 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.908 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.908 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.909 INFO:teuthology.orchestra.run.smithi028.stdout: "exclude_perf_counters": { 2024-05-12T22:19:42.909 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "exclude_perf_counters", 2024-05-12T22:19:42.909 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.909 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.909 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.909 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-12T22:19:42.909 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.909 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.910 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.910 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-05-12T22:19:42.910 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-05-12T22:19:42.910 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.910 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.910 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.910 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-12T22:19:42.910 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-12T22:19:42.911 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.911 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.911 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.911 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.911 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.911 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.911 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:42.911 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:42.912 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:42.912 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:42.912 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:42.912 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:42.912 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:42.912 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:42.912 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.912 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.912 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.913 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.913 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.913 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-12T22:19:42.913 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-12T22:19:42.913 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.913 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.913 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.913 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.913 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.914 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.914 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.914 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.914 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.914 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.914 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.914 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.914 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-12T22:19:42.915 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-12T22:19:42.915 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.915 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.915 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.915 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-12T22:19:42.915 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.915 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.915 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:42.915 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:42.916 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:42.916 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:42.916 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:42.916 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:42.916 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:42.916 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:42.916 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.916 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.917 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.917 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.917 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.917 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-12T22:19:42.917 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-12T22:19:42.917 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.917 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.917 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.917 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.918 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.918 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.918 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.918 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.918 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.918 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.918 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.918 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.918 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_stats_pools": { 2024-05-12T22:19:42.919 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rbd_stats_pools", 2024-05-12T22:19:42.919 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.919 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.919 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.919 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.919 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.919 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.919 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.919 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.920 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.920 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.920 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.920 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.920 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_stats_pools_refresh_interval": { 2024-05-12T22:19:42.920 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-05-12T22:19:42.920 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-12T22:19:42.920 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.921 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.921 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "300", 2024-05-12T22:19:42.921 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.921 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.921 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.921 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.921 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.921 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.921 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.922 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.922 INFO:teuthology.orchestra.run.smithi028.stdout: "scrape_interval": { 2024-05-12T22:19:42.922 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "scrape_interval", 2024-05-12T22:19:42.922 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-05-12T22:19:42.922 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.922 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.922 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "15.0", 2024-05-12T22:19:42.922 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.922 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.923 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.923 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.923 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.923 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.923 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.923 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.923 INFO:teuthology.orchestra.run.smithi028.stdout: "server_addr": { 2024-05-12T22:19:42.923 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_addr", 2024-05-12T22:19:42.923 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.924 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.924 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.924 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "::", 2024-05-12T22:19:42.924 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.924 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.924 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.924 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-05-12T22:19:42.924 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.925 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.925 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.925 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.925 INFO:teuthology.orchestra.run.smithi028.stdout: "server_port": { 2024-05-12T22:19:42.925 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_port", 2024-05-12T22:19:42.925 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-12T22:19:42.925 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.925 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.925 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "9283", 2024-05-12T22:19:42.926 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.926 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.926 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.926 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-05-12T22:19:42.926 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.926 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.926 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.926 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.927 INFO:teuthology.orchestra.run.smithi028.stdout: "stale_cache_strategy": { 2024-05-12T22:19:42.927 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "stale_cache_strategy", 2024-05-12T22:19:42.927 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.927 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.927 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.927 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "log", 2024-05-12T22:19:42.927 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.927 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.927 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.928 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.928 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.928 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.928 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.928 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.928 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_behaviour": { 2024-05-12T22:19:42.928 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_behaviour", 2024-05-12T22:19:42.928 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.929 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.929 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.929 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "default", 2024-05-12T22:19:42.929 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.929 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.929 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:42.929 INFO:teuthology.orchestra.run.smithi028.stdout: "default", 2024-05-12T22:19:42.929 INFO:teuthology.orchestra.run.smithi028.stdout: "error" 2024-05-12T22:19:42.929 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:42.930 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.930 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.930 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.930 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.930 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.930 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_error_status_code": { 2024-05-12T22:19:42.930 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_error_status_code", 2024-05-12T22:19:42.930 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-12T22:19:42.930 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.931 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.931 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "500", 2024-05-12T22:19:42.931 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "400", 2024-05-12T22:19:42.931 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "599", 2024-05-12T22:19:42.931 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.931 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.931 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.931 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.931 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.932 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:42.932 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:42.932 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.932 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-12T22:19:42.932 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rbd_support", 2024-05-12T22:19:42.932 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-12T22:19:42.932 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-12T22:19:42.932 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-12T22:19:42.932 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-12T22:19:42.933 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-12T22:19:42.933 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.933 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.933 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.933 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.933 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.933 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.933 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:42.934 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:42.934 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:42.934 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:42.934 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:42.934 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:42.934 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:42.934 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:42.934 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.934 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.935 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.935 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.935 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.935 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-12T22:19:42.935 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-12T22:19:42.935 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.935 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.935 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.936 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.936 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.936 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.936 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.936 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.936 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.936 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.936 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.936 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.937 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-12T22:19:42.937 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-12T22:19:42.937 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.937 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.937 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.937 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-12T22:19:42.937 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.937 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.937 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:42.938 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:42.938 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:42.938 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:42.938 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:42.938 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:42.938 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:42.938 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:42.938 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.939 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.939 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.939 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.939 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.939 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-12T22:19:42.939 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-12T22:19:42.939 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.939 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.939 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.940 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.940 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.940 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.940 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.940 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.940 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.940 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.940 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.941 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.941 INFO:teuthology.orchestra.run.smithi028.stdout: "max_concurrent_snap_create": { 2024-05-12T22:19:42.941 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_concurrent_snap_create", 2024-05-12T22:19:42.941 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-12T22:19:42.941 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.941 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.941 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-05-12T22:19:42.941 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.941 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.942 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.942 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.942 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.942 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.942 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.942 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.945 INFO:teuthology.orchestra.run.smithi028.stdout: "mirror_snapshot_schedule": { 2024-05-12T22:19:42.945 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mirror_snapshot_schedule", 2024-05-12T22:19:42.945 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.945 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.945 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.945 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.946 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.946 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.946 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.946 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.946 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.946 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.946 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.946 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.947 INFO:teuthology.orchestra.run.smithi028.stdout: "trash_purge_schedule": { 2024-05-12T22:19:42.947 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "trash_purge_schedule", 2024-05-12T22:19:42.947 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.947 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.947 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.947 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.947 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.947 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.947 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.948 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.948 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.948 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.948 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.948 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:42.948 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:42.948 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.948 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-12T22:19:42.948 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "restful", 2024-05-12T22:19:42.949 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-12T22:19:42.949 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-12T22:19:42.949 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-12T22:19:42.949 INFO:teuthology.orchestra.run.smithi028.stdout: "enable_auth": { 2024-05-12T22:19:42.949 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enable_auth", 2024-05-12T22:19:42.949 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.949 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.949 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.949 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-12T22:19:42.950 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.950 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.950 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.950 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.950 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.950 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.950 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.950 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.951 INFO:teuthology.orchestra.run.smithi028.stdout: "key_file": { 2024-05-12T22:19:42.951 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "key_file", 2024-05-12T22:19:42.951 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.951 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.951 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.951 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.951 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.951 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.951 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.952 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.952 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.952 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.952 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.952 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.952 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-12T22:19:42.952 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-12T22:19:42.952 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.952 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.953 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.953 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.953 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.953 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.953 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:42.953 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:42.953 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:42.953 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:42.954 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:42.954 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:42.954 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:42.954 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:42.954 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.954 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.954 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.954 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.954 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.955 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-12T22:19:42.955 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-12T22:19:42.955 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.955 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.955 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.955 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.955 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.955 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.955 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.956 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.956 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.956 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.956 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.956 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.956 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-12T22:19:42.956 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-12T22:19:42.956 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.957 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.957 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.957 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-12T22:19:42.957 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.957 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.957 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:42.957 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:42.957 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:42.957 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:42.958 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:42.958 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:42.958 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:42.958 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:42.958 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.958 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.958 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.958 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.958 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.959 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-12T22:19:42.959 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-12T22:19:42.959 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.959 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.959 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.959 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.959 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.959 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.959 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.960 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.960 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.960 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.960 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.960 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.960 INFO:teuthology.orchestra.run.smithi028.stdout: "server_addr": { 2024-05-12T22:19:42.960 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_addr", 2024-05-12T22:19:42.960 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.961 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.961 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.961 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.961 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.961 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.961 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.961 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.961 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.961 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.962 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.962 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.962 INFO:teuthology.orchestra.run.smithi028.stdout: "server_port": { 2024-05-12T22:19:42.962 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_port", 2024-05-12T22:19:42.962 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.962 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.962 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.962 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.962 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.963 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.963 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.963 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.963 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.963 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.963 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.963 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:42.963 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:42.963 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.964 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-12T22:19:42.964 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rgw", 2024-05-12T22:19:42.964 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-12T22:19:42.964 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-12T22:19:42.964 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-12T22:19:42.964 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-12T22:19:42.964 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-12T22:19:42.964 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.965 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.965 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.965 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.965 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.965 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.965 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:42.965 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:42.965 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:42.965 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:42.966 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:42.966 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:42.966 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:42.966 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:42.966 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.966 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.966 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.966 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.966 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.967 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-12T22:19:42.967 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-12T22:19:42.967 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.967 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.967 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.967 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.967 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.967 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.968 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.968 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.968 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.968 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.968 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.968 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.968 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-12T22:19:42.968 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-12T22:19:42.968 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.969 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.969 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.969 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-12T22:19:42.969 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.969 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.969 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:42.969 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:42.969 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:42.969 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:42.970 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:42.970 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:42.970 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:42.970 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:42.970 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.970 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.970 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.970 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.970 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.971 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-12T22:19:42.971 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-12T22:19:42.971 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.971 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.971 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.971 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.971 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.971 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.971 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.972 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.972 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.972 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.972 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.972 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:42.972 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:42.972 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.972 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-12T22:19:42.973 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rook", 2024-05-12T22:19:42.973 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-12T22:19:42.973 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-12T22:19:42.973 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-12T22:19:42.973 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-12T22:19:42.973 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-12T22:19:42.973 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.973 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.973 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.974 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.974 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.974 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.974 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:42.974 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:42.974 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:42.974 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:42.974 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:42.974 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:42.975 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:42.975 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:42.975 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.975 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.975 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.975 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.975 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.975 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-12T22:19:42.975 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-12T22:19:42.976 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.976 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.976 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.976 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.976 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.976 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.976 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.976 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.977 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.977 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.977 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.977 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.977 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-12T22:19:42.977 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-12T22:19:42.977 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.977 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.978 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.978 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-12T22:19:42.978 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.978 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.978 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:42.978 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:42.978 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:42.978 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:42.979 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:42.979 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:42.979 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:42.979 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:42.979 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.979 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.979 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.979 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.980 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.980 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-12T22:19:42.980 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-12T22:19:42.980 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.980 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.980 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.980 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.980 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.980 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.981 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.981 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.981 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.981 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.981 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.981 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.981 INFO:teuthology.orchestra.run.smithi028.stdout: "storage_class": { 2024-05-12T22:19:42.981 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "storage_class", 2024-05-12T22:19:42.981 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.982 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.982 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.982 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "local", 2024-05-12T22:19:42.982 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.982 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.982 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.982 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-05-12T22:19:42.982 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.983 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.983 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.983 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:42.983 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:42.983 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.983 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-12T22:19:42.983 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "selftest", 2024-05-12T22:19:42.983 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-12T22:19:42.983 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-12T22:19:42.984 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-12T22:19:42.984 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-12T22:19:42.984 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-12T22:19:42.984 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.984 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.984 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.984 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.984 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.984 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.985 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:42.985 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:42.985 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:42.985 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:42.985 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:42.985 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:42.985 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:42.985 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:42.985 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.986 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.986 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.986 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.986 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.986 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-12T22:19:42.986 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-12T22:19:42.986 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.986 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.987 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.987 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.987 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.987 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.987 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.987 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.987 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.987 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.987 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.988 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.988 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-12T22:19:42.988 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-12T22:19:42.988 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.988 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.988 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.988 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-12T22:19:42.988 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.988 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.989 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:42.989 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:42.989 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:42.989 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:42.989 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:42.989 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:42.989 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:42.989 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:42.989 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.990 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.990 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.990 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.990 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.990 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-12T22:19:42.990 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-12T22:19:42.990 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:42.990 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.990 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:42.991 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:42.991 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.991 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.991 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.991 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.991 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.991 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.991 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.992 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.992 INFO:teuthology.orchestra.run.smithi028.stdout: "roption1": { 2024-05-12T22:19:42.992 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "roption1", 2024-05-12T22:19:42.992 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.992 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.992 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.992 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.992 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.992 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.993 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.993 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.993 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.993 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.993 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.993 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.993 INFO:teuthology.orchestra.run.smithi028.stdout: "roption2": { 2024-05-12T22:19:42.993 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "roption2", 2024-05-12T22:19:42.994 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.994 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.994 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.994 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "xyz", 2024-05-12T22:19:42.994 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.994 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.994 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.994 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.995 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.995 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.995 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.995 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.995 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption1": { 2024-05-12T22:19:42.995 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption1", 2024-05-12T22:19:42.995 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:42.995 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.995 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.996 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.996 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.996 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.996 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.996 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.996 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.996 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.996 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.996 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.997 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption2": { 2024-05-12T22:19:42.997 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption2", 2024-05-12T22:19:42.997 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-12T22:19:42.997 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.997 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.997 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.997 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.997 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.997 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.998 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.998 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:42.998 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:42.998 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:42.998 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:42.998 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption3": { 2024-05-12T22:19:42.998 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption3", 2024-05-12T22:19:42.998 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-05-12T22:19:42.999 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:42.999 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:42.999 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:42.999 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:42.999 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:42.999 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:42.999 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:42.999 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.000 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.000 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.000 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.000 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption4": { 2024-05-12T22:19:43.000 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption4", 2024-05-12T22:19:43.000 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.000 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.000 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.001 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:43.001 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.001 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.001 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.001 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.001 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.001 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.001 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.001 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.002 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption5": { 2024-05-12T22:19:43.002 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption5", 2024-05-12T22:19:43.002 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.002 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.002 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.002 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:43.002 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.002 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.003 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.003 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.003 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.003 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.003 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.003 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.003 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption6": { 2024-05-12T22:19:43.003 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption6", 2024-05-12T22:19:43.004 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.004 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.004 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.004 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-12T22:19:43.004 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.004 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.004 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.004 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.004 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.005 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.005 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.005 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.005 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption7": { 2024-05-12T22:19:43.005 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption7", 2024-05-12T22:19:43.005 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-12T22:19:43.005 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.005 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.005 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:43.006 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-05-12T22:19:43.006 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "42", 2024-05-12T22:19:43.006 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.006 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.006 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.006 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.006 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.006 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.007 INFO:teuthology.orchestra.run.smithi028.stdout: "testkey": { 2024-05-12T22:19:43.007 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "testkey", 2024-05-12T22:19:43.007 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.007 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.007 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.007 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:43.007 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.007 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.007 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.008 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.008 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.008 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.008 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.008 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.008 INFO:teuthology.orchestra.run.smithi028.stdout: "testlkey": { 2024-05-12T22:19:43.008 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "testlkey", 2024-05-12T22:19:43.008 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.008 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.009 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.009 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:43.009 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.009 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.009 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.009 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.009 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.009 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.009 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.010 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.010 INFO:teuthology.orchestra.run.smithi028.stdout: "testnewline": { 2024-05-12T22:19:43.010 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "testnewline", 2024-05-12T22:19:43.010 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.010 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.010 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.010 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:43.010 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.011 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.011 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.011 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.011 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.011 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.011 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.011 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:43.011 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:43.011 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.012 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-12T22:19:43.012 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "snap_schedule", 2024-05-12T22:19:43.012 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-12T22:19:43.012 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-12T22:19:43.012 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-12T22:19:43.012 INFO:teuthology.orchestra.run.smithi028.stdout: "allow_m_granularity": { 2024-05-12T22:19:43.012 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "allow_m_granularity", 2024-05-12T22:19:43.012 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.013 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.013 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.013 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:43.013 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.013 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.013 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.013 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "allow minute scheduled snapshots", 2024-05-12T22:19:43.013 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.013 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.014 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.014 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.014 INFO:teuthology.orchestra.run.smithi028.stdout: "dump_on_update": { 2024-05-12T22:19:43.014 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "dump_on_update", 2024-05-12T22:19:43.014 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.014 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.014 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.014 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:43.015 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.015 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.015 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.015 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "dump database to debug log on update", 2024-05-12T22:19:43.015 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.015 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.015 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.015 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.015 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-12T22:19:43.016 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-12T22:19:43.016 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.016 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.016 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.016 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:43.016 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.016 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.016 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:43.016 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:43.017 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:43.017 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:43.017 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:43.017 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:43.017 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:43.017 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:43.017 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.017 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.017 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.018 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.018 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.018 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-12T22:19:43.018 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-12T22:19:43.018 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.018 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.018 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.018 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:43.018 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.019 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.019 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.019 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.019 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.019 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.019 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.019 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.019 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-12T22:19:43.019 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-12T22:19:43.020 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.020 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.020 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.020 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-12T22:19:43.020 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.020 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.020 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:43.020 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:43.020 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:43.020 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:43.021 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:43.021 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:43.021 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:43.021 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:43.021 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.021 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.021 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.021 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.022 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.022 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-12T22:19:43.022 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-12T22:19:43.022 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.022 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.022 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.022 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:43.022 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.023 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.023 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.023 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.023 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.023 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.023 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.023 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:43.023 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:43.024 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.024 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-12T22:19:43.024 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "stats", 2024-05-12T22:19:43.024 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-12T22:19:43.024 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-12T22:19:43.024 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-12T22:19:43.024 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-12T22:19:43.024 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-12T22:19:43.024 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.025 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.025 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.025 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:43.025 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.025 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.025 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:43.025 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:43.025 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:43.026 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:43.026 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:43.026 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:43.026 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:43.026 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:43.026 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.026 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.026 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.026 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.027 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.027 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-12T22:19:43.027 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-12T22:19:43.027 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.027 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.027 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.027 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:43.027 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.028 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.028 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.028 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.028 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.028 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.028 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.028 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.028 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-12T22:19:43.028 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-12T22:19:43.029 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.029 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.029 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.029 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-12T22:19:43.029 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.029 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.029 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:43.029 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:43.029 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:43.030 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:43.030 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:43.030 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:43.030 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:43.030 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:43.030 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.030 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.030 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.030 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.031 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.031 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-12T22:19:43.031 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-12T22:19:43.031 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.031 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.031 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.031 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:43.031 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.032 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.032 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.032 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.032 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.032 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.032 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.032 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:43.032 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:43.033 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.033 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-12T22:19:43.033 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "status", 2024-05-12T22:19:43.033 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-12T22:19:43.033 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-12T22:19:43.033 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-12T22:19:43.033 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-12T22:19:43.033 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-12T22:19:43.033 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.034 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.034 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.034 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:43.034 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.034 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.034 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:43.034 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:43.034 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:43.034 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:43.035 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:43.035 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:43.035 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:43.035 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:43.035 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.035 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.035 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.035 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.035 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.036 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-12T22:19:43.036 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-12T22:19:43.036 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.036 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.036 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.036 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:43.036 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.036 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.036 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.037 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.037 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.037 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.037 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.037 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.037 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-12T22:19:43.037 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-12T22:19:43.037 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.038 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.038 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.038 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-12T22:19:43.038 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.038 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.038 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:43.038 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:43.038 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:43.039 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:43.039 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:43.039 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:43.039 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:43.039 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:43.039 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.039 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.039 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.039 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.040 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.040 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-12T22:19:43.040 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-12T22:19:43.040 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.040 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.040 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.040 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:43.040 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.041 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.041 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.041 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.041 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.041 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.041 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.041 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:43.041 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:43.041 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.042 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-12T22:19:43.042 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "telegraf", 2024-05-12T22:19:43.042 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-12T22:19:43.042 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-12T22:19:43.042 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-12T22:19:43.042 INFO:teuthology.orchestra.run.smithi028.stdout: "address": { 2024-05-12T22:19:43.042 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "address", 2024-05-12T22:19:43.042 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.042 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.043 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.043 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-05-12T22:19:43.043 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.043 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.043 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.043 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.043 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.043 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.044 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.044 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.044 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-05-12T22:19:43.044 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-05-12T22:19:43.044 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-05-12T22:19:43.044 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.044 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.044 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "15", 2024-05-12T22:19:43.045 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.045 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.045 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.045 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.045 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.045 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.045 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.045 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.045 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-12T22:19:43.046 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-12T22:19:43.046 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.046 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.046 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.046 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:43.046 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.046 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.046 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:43.046 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:43.047 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:43.047 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:43.047 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:43.047 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:43.047 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:43.047 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:43.047 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.047 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.047 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.048 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.048 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.048 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-12T22:19:43.048 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-12T22:19:43.048 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.048 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.048 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.048 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:43.049 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.049 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.049 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.049 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.049 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.049 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.049 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.049 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.049 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-12T22:19:43.050 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-12T22:19:43.050 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.050 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.050 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.050 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-12T22:19:43.050 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.050 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.050 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:43.051 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:43.051 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:43.051 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:43.051 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:43.051 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:43.051 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:43.051 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:43.051 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.051 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.052 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.052 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.052 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.052 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-12T22:19:43.052 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-12T22:19:43.052 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.052 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.052 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.052 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:43.053 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.053 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.053 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.053 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.053 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.053 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.053 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.053 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:43.054 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:43.054 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.054 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-12T22:19:43.054 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "telemetry", 2024-05-12T22:19:43.054 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-12T22:19:43.054 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-12T22:19:43.054 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-12T22:19:43.054 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_basic": { 2024-05-12T22:19:43.054 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_basic", 2024-05-12T22:19:43.055 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.055 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.055 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.055 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-12T22:19:43.055 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.055 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.055 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.055 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share basic cluster information (size, version)", 2024-05-12T22:19:43.055 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.056 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.056 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.056 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.056 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_crash": { 2024-05-12T22:19:43.056 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_crash", 2024-05-12T22:19:43.056 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.056 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.056 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.057 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-12T22:19:43.057 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.057 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.057 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.057 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-05-12T22:19:43.057 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.057 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.057 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.057 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.058 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_device": { 2024-05-12T22:19:43.058 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_device", 2024-05-12T22:19:43.058 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.058 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.058 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.058 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-12T22:19:43.058 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.058 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.059 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.059 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-05-12T22:19:43.059 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.059 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.059 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.059 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.059 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_ident": { 2024-05-12T22:19:43.059 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_ident", 2024-05-12T22:19:43.060 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.060 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.060 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.060 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:43.060 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.060 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.060 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.060 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-05-12T22:19:43.060 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.061 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.061 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.061 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.061 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_perf": { 2024-05-12T22:19:43.061 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_perf", 2024-05-12T22:19:43.061 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.061 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.061 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.061 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:43.062 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.062 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.062 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.062 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share various performance metrics of a cluster", 2024-05-12T22:19:43.062 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.062 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.062 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.062 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.063 INFO:teuthology.orchestra.run.smithi028.stdout: "contact": { 2024-05-12T22:19:43.063 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "contact", 2024-05-12T22:19:43.063 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.063 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.063 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.063 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:43.063 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.063 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.063 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.064 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.064 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.064 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.064 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.064 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.064 INFO:teuthology.orchestra.run.smithi028.stdout: "description": { 2024-05-12T22:19:43.064 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "description", 2024-05-12T22:19:43.064 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.065 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.065 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.065 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:43.065 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.065 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.065 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.065 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.065 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.065 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.066 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.066 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.066 INFO:teuthology.orchestra.run.smithi028.stdout: "device_url": { 2024-05-12T22:19:43.066 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "device_url", 2024-05-12T22:19:43.066 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.066 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.066 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.067 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-05-12T22:19:43.067 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.067 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.067 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.067 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.067 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.067 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.067 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.067 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.068 INFO:teuthology.orchestra.run.smithi028.stdout: "enabled": { 2024-05-12T22:19:43.068 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enabled", 2024-05-12T22:19:43.068 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.068 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.068 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.068 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:43.068 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.068 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.068 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.069 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.069 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.069 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.069 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.069 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.069 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-05-12T22:19:43.069 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-05-12T22:19:43.069 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-12T22:19:43.069 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.070 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.070 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "24", 2024-05-12T22:19:43.070 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "8", 2024-05-12T22:19:43.070 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.070 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.070 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.070 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.070 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.070 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.071 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.071 INFO:teuthology.orchestra.run.smithi028.stdout: "last_opt_revision": { 2024-05-12T22:19:43.071 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "last_opt_revision", 2024-05-12T22:19:43.071 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-12T22:19:43.071 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.071 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.071 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "1", 2024-05-12T22:19:43.071 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.072 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.072 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.072 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.072 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.072 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.072 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.072 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.072 INFO:teuthology.orchestra.run.smithi028.stdout: "leaderboard": { 2024-05-12T22:19:43.072 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "leaderboard", 2024-05-12T22:19:43.073 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.073 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.073 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.073 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:43.073 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.073 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.073 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.073 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.073 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.074 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.074 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.074 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.074 INFO:teuthology.orchestra.run.smithi028.stdout: "leaderboard_description": { 2024-05-12T22:19:43.074 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "leaderboard_description", 2024-05-12T22:19:43.074 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.074 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.074 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.075 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:43.075 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.075 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.075 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.075 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.075 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.075 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.075 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.075 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.076 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-12T22:19:43.076 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-12T22:19:43.076 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.076 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.076 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.076 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:43.076 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.076 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.076 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:43.077 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:43.077 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:43.077 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:43.077 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:43.077 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:43.077 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:43.077 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:43.077 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.077 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.078 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.078 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.078 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.078 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-12T22:19:43.078 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-12T22:19:43.078 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.078 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.078 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.079 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:43.079 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.079 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.079 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.079 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.079 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.079 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.079 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.079 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.080 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-12T22:19:43.080 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-12T22:19:43.080 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.080 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.080 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.080 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-12T22:19:43.080 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.080 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.080 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:43.081 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:43.081 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:43.081 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:43.081 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:43.081 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:43.081 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:43.081 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:43.081 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.082 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.082 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.082 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.082 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.082 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-12T22:19:43.082 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-12T22:19:43.082 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.082 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.082 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.083 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:43.083 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.083 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.083 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.083 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.083 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.083 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.083 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.084 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.084 INFO:teuthology.orchestra.run.smithi028.stdout: "organization": { 2024-05-12T22:19:43.084 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "organization", 2024-05-12T22:19:43.084 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.084 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.084 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.084 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:43.084 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.084 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.084 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.085 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.085 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.085 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.085 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.085 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.085 INFO:teuthology.orchestra.run.smithi028.stdout: "proxy": { 2024-05-12T22:19:43.085 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "proxy", 2024-05-12T22:19:43.085 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.085 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.086 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.086 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:43.086 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.086 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.086 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.086 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.086 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.086 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.087 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.087 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.087 INFO:teuthology.orchestra.run.smithi028.stdout: "url": { 2024-05-12T22:19:43.087 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "url", 2024-05-12T22:19:43.087 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.087 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.087 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.087 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-05-12T22:19:43.087 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.088 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.088 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.088 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.088 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.088 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.088 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.088 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:43.088 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:43.088 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.088 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-12T22:19:43.089 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "test_orchestrator", 2024-05-12T22:19:43.089 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-12T22:19:43.089 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-12T22:19:43.089 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-12T22:19:43.089 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-12T22:19:43.089 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-12T22:19:43.089 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.089 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.089 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.090 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:43.090 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.090 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.090 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:43.090 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:43.090 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:43.090 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:43.090 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:43.091 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:43.091 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:43.091 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:43.091 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.091 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.091 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.091 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.091 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.092 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-12T22:19:43.092 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-12T22:19:43.092 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.092 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.092 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.092 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:43.092 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.092 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.092 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.093 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.093 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.093 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.093 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.093 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.093 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-12T22:19:43.093 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-12T22:19:43.093 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.093 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.094 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.094 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-12T22:19:43.094 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.094 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.094 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:43.094 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:43.094 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:43.094 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:43.095 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:43.095 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:43.095 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:43.095 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:43.095 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.095 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.095 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.095 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.095 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.096 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-12T22:19:43.096 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-12T22:19:43.096 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.096 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.096 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.096 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:43.096 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.096 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.096 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.096 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.097 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.097 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.097 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.097 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:43.097 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:43.097 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.097 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-12T22:19:43.097 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "volumes", 2024-05-12T22:19:43.097 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-12T22:19:43.098 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-12T22:19:43.098 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-12T22:19:43.098 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-12T22:19:43.098 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-12T22:19:43.098 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.098 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.098 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.098 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:43.099 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.099 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.099 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:43.099 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:43.099 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:43.099 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:43.099 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:43.099 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:43.099 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:43.100 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:43.100 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.100 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.100 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.100 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.100 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.100 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-12T22:19:43.100 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-12T22:19:43.100 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.101 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.101 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.101 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:43.101 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.101 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.101 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.101 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.101 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.101 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.102 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.102 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.102 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-12T22:19:43.102 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-12T22:19:43.102 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.102 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.102 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.102 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-12T22:19:43.103 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.103 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.103 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:43.103 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:43.103 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:43.103 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:43.103 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:43.103 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:43.104 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:43.104 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:43.104 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.104 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.104 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.104 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.104 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.104 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-12T22:19:43.104 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-12T22:19:43.105 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.105 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.105 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.105 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:43.105 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.105 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.105 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.105 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.106 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.106 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.106 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.106 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.106 INFO:teuthology.orchestra.run.smithi028.stdout: "max_concurrent_clones": { 2024-05-12T22:19:43.106 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_concurrent_clones", 2024-05-12T22:19:43.106 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-12T22:19:43.106 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.106 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.107 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "4", 2024-05-12T22:19:43.107 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.107 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.107 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.107 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Number of asynchronous cloner threads", 2024-05-12T22:19:43.107 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.107 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.107 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.107 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.108 INFO:teuthology.orchestra.run.smithi028.stdout: "periodic_async_work": { 2024-05-12T22:19:43.108 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "periodic_async_work", 2024-05-12T22:19:43.108 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.108 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.108 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.108 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:43.108 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.108 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.108 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.109 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Periodically check for async work", 2024-05-12T22:19:43.109 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.109 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.109 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.109 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.109 INFO:teuthology.orchestra.run.smithi028.stdout: "snapshot_clone_delay": { 2024-05-12T22:19:43.109 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "snapshot_clone_delay", 2024-05-12T22:19:43.109 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-12T22:19:43.110 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.110 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.110 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-05-12T22:19:43.110 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.110 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.110 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.110 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-05-12T22:19:43.110 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.111 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.111 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.111 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.111 INFO:teuthology.orchestra.run.smithi028.stdout: "snapshot_clone_no_wait": { 2024-05-12T22:19:43.111 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "snapshot_clone_no_wait", 2024-05-12T22:19:43.111 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.111 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.111 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.111 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-12T22:19:43.112 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.112 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.112 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.112 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-05-12T22:19:43.112 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.112 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.112 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.112 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:43.113 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:43.113 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.113 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-12T22:19:43.113 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix", 2024-05-12T22:19:43.113 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-12T22:19:43.113 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-12T22:19:43.113 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-12T22:19:43.113 INFO:teuthology.orchestra.run.smithi028.stdout: "discovery_interval": { 2024-05-12T22:19:43.113 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "discovery_interval", 2024-05-12T22:19:43.113 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-05-12T22:19:43.114 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.114 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.114 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "100", 2024-05-12T22:19:43.114 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.114 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.114 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.114 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.114 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.115 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.115 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.115 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.115 INFO:teuthology.orchestra.run.smithi028.stdout: "identifier": { 2024-05-12T22:19:43.115 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "identifier", 2024-05-12T22:19:43.115 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.115 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.115 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.115 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:43.116 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.116 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.116 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.116 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.116 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.116 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.116 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.116 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.117 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-05-12T22:19:43.117 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-05-12T22:19:43.117 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-05-12T22:19:43.117 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.117 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.117 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-05-12T22:19:43.117 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.117 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.117 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.118 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.118 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.118 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.118 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.118 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.118 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-12T22:19:43.118 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-12T22:19:43.118 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.119 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.119 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.119 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:43.119 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.119 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.119 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:43.119 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:43.119 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:43.120 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:43.120 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:43.120 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:43.120 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:43.120 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:43.120 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.120 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.120 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.120 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.121 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.121 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-12T22:19:43.121 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-12T22:19:43.121 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.121 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.121 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.121 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:43.121 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.121 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.122 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.122 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.122 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.122 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.122 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.122 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.122 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-12T22:19:43.122 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-12T22:19:43.122 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.123 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.123 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.123 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-12T22:19:43.123 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.123 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.123 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:43.123 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:43.123 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:43.124 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:43.124 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:43.124 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:43.124 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:43.124 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:43.124 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.124 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.124 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.124 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.125 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.125 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-12T22:19:43.125 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-12T22:19:43.125 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.125 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.125 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.125 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:43.125 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.125 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.126 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.126 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.126 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.126 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.126 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.126 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.126 INFO:teuthology.orchestra.run.smithi028.stdout: "zabbix_host": { 2024-05-12T22:19:43.126 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix_host", 2024-05-12T22:19:43.127 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.127 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.127 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.127 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:43.127 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.127 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.127 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.127 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.127 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.128 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.128 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.128 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.128 INFO:teuthology.orchestra.run.smithi028.stdout: "zabbix_port": { 2024-05-12T22:19:43.128 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix_port", 2024-05-12T22:19:43.128 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-12T22:19:43.128 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.128 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.129 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10051", 2024-05-12T22:19:43.129 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.129 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.129 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.129 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.129 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.129 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.129 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.129 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.130 INFO:teuthology.orchestra.run.smithi028.stdout: "zabbix_sender": { 2024-05-12T22:19:43.130 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix_sender", 2024-05-12T22:19:43.130 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.130 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.130 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.130 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-05-12T22:19:43.130 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.130 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.130 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.131 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.131 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.131 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.131 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.131 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:43.131 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:43.131 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:43.131 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-05-12T22:19:43.132 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:43.132 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:43.132 INFO:teuthology.orchestra.run.smithi028.stdout: "modules": [ 2024-05-12T22:19:43.132 INFO:teuthology.orchestra.run.smithi028.stdout: "cephadm", 2024-05-12T22:19:43.132 INFO:teuthology.orchestra.run.smithi028.stdout: "dashboard", 2024-05-12T22:19:43.132 INFO:teuthology.orchestra.run.smithi028.stdout: "iostat", 2024-05-12T22:19:43.132 INFO:teuthology.orchestra.run.smithi028.stdout: "nfs", 2024-05-12T22:19:43.132 INFO:teuthology.orchestra.run.smithi028.stdout: "restful", 2024-05-12T22:19:43.132 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_schedule" 2024-05-12T22:19:43.133 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:43.133 INFO:teuthology.orchestra.run.smithi028.stdout: "available_modules": [ 2024-05-12T22:19:43.133 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-12T22:19:43.133 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "alerts", 2024-05-12T22:19:43.133 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-12T22:19:43.133 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-12T22:19:43.133 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-12T22:19:43.133 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-05-12T22:19:43.133 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-05-12T22:19:43.134 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-05-12T22:19:43.134 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.134 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.134 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-05-12T22:19:43.134 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.134 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.134 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.135 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How frequently to reexamine health status", 2024-05-12T22:19:43.135 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.135 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.135 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.135 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.135 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-12T22:19:43.135 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-12T22:19:43.135 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.135 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.136 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.136 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:43.136 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.136 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.136 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:43.136 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:43.136 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:43.136 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:43.137 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:43.137 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:43.137 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:43.137 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:43.137 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.137 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.137 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.137 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.137 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.138 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-12T22:19:43.138 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-12T22:19:43.138 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.138 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.138 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.138 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:43.138 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.138 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.139 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.139 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.139 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.139 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.139 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.139 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.139 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-12T22:19:43.139 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-12T22:19:43.139 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.140 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.140 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.140 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-12T22:19:43.140 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.140 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.140 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:43.140 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:43.140 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:43.140 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:43.141 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:43.141 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:43.141 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:43.141 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:43.141 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.141 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.141 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.141 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.141 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.142 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-12T22:19:43.142 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-12T22:19:43.142 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.142 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.142 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.142 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:43.142 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.142 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.143 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.143 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.143 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.143 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.143 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.143 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.143 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_destination": { 2024-05-12T22:19:43.143 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_destination", 2024-05-12T22:19:43.143 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.145 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.145 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.145 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:43.145 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.145 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.146 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.146 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Email address to send alerts to", 2024-05-12T22:19:43.146 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.146 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.146 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.146 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.146 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_from_name": { 2024-05-12T22:19:43.146 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_from_name", 2024-05-12T22:19:43.146 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.147 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.147 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.147 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "Ceph", 2024-05-12T22:19:43.147 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.147 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.147 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.147 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Email From: name", 2024-05-12T22:19:43.147 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.147 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.148 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.148 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.148 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_host": { 2024-05-12T22:19:43.148 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_host", 2024-05-12T22:19:43.148 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.148 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.148 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.148 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:43.149 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.149 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.149 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.149 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SMTP server", 2024-05-12T22:19:43.149 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.149 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.149 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.149 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.149 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_password": { 2024-05-12T22:19:43.150 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_password", 2024-05-12T22:19:43.150 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.150 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.150 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.150 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:43.150 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.150 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.150 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.150 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Password to authenticate with", 2024-05-12T22:19:43.151 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.151 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.151 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.151 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.151 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_port": { 2024-05-12T22:19:43.151 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_port", 2024-05-12T22:19:43.151 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-12T22:19:43.151 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.152 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.152 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "465", 2024-05-12T22:19:43.152 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.152 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.152 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.152 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SMTP port", 2024-05-12T22:19:43.152 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.152 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.152 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.153 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.153 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_sender": { 2024-05-12T22:19:43.153 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_sender", 2024-05-12T22:19:43.153 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.153 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.153 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.153 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:43.153 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.153 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.154 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.154 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SMTP envelope sender", 2024-05-12T22:19:43.154 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.154 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.154 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.154 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.154 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_ssl": { 2024-05-12T22:19:43.154 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_ssl", 2024-05-12T22:19:43.155 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.155 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.155 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.155 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-12T22:19:43.155 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.155 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.155 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.155 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use SSL to connect to SMTP server", 2024-05-12T22:19:43.155 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.156 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.156 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.156 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.156 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_user": { 2024-05-12T22:19:43.156 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_user", 2024-05-12T22:19:43.156 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.156 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.156 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.156 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:43.157 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.157 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.157 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.157 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "User to authenticate as", 2024-05-12T22:19:43.157 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.157 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.157 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.157 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:43.158 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:43.158 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.158 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-12T22:19:43.158 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "balancer", 2024-05-12T22:19:43.158 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-12T22:19:43.158 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-12T22:19:43.158 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-12T22:19:43.158 INFO:teuthology.orchestra.run.smithi028.stdout: "active": { 2024-05-12T22:19:43.158 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "active", 2024-05-12T22:19:43.159 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.159 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.159 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.159 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-12T22:19:43.159 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.159 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.159 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.159 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "automatically balance PGs across cluster", 2024-05-12T22:19:43.159 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.160 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.160 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.160 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.160 INFO:teuthology.orchestra.run.smithi028.stdout: "begin_time": { 2024-05-12T22:19:43.160 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "begin_time", 2024-05-12T22:19:43.160 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.160 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.160 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.160 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0000", 2024-05-12T22:19:43.161 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.161 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.161 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.161 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "beginning time of day to automatically balance", 2024-05-12T22:19:43.161 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-05-12T22:19:43.161 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.161 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.161 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.162 INFO:teuthology.orchestra.run.smithi028.stdout: "begin_weekday": { 2024-05-12T22:19:43.162 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "begin_weekday", 2024-05-12T22:19:43.162 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-05-12T22:19:43.162 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.162 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.162 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-05-12T22:19:43.162 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "0", 2024-05-12T22:19:43.162 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "6", 2024-05-12T22:19:43.163 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.163 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-05-12T22:19:43.163 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-05-12T22:19:43.163 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.163 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.163 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.163 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_compat_max_iterations": { 2024-05-12T22:19:43.163 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crush_compat_max_iterations", 2024-05-12T22:19:43.163 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-05-12T22:19:43.164 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.164 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.164 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "25", 2024-05-12T22:19:43.164 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-05-12T22:19:43.164 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "250", 2024-05-12T22:19:43.164 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.164 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-05-12T22:19:43.164 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.164 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.165 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.165 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.165 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_compat_metrics": { 2024-05-12T22:19:43.165 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crush_compat_metrics", 2024-05-12T22:19:43.165 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.165 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.165 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.165 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "pgs,objects,bytes", 2024-05-12T22:19:43.165 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.166 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.166 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.166 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-05-12T22:19:43.166 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-05-12T22:19:43.166 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.166 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.166 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.166 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_compat_step": { 2024-05-12T22:19:43.167 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crush_compat_step", 2024-05-12T22:19:43.167 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-05-12T22:19:43.167 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.167 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.167 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0.5", 2024-05-12T22:19:43.167 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "0.001", 2024-05-12T22:19:43.167 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "0.999", 2024-05-12T22:19:43.167 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.167 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "aggressiveness of optimization", 2024-05-12T22:19:43.167 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-05-12T22:19:43.168 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.168 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.168 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.168 INFO:teuthology.orchestra.run.smithi028.stdout: "end_time": { 2024-05-12T22:19:43.168 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "end_time", 2024-05-12T22:19:43.168 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.168 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.168 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.168 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "2359", 2024-05-12T22:19:43.169 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.169 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.169 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.169 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "ending time of day to automatically balance", 2024-05-12T22:19:43.169 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-05-12T22:19:43.169 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.169 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.169 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.169 INFO:teuthology.orchestra.run.smithi028.stdout: "end_weekday": { 2024-05-12T22:19:43.170 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "end_weekday", 2024-05-12T22:19:43.170 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-05-12T22:19:43.170 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.170 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.170 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-05-12T22:19:43.170 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "0", 2024-05-12T22:19:43.170 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "6", 2024-05-12T22:19:43.170 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.171 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-05-12T22:19:43.171 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-05-12T22:19:43.171 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.171 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.171 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.171 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-12T22:19:43.171 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-12T22:19:43.171 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.171 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.172 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.172 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:43.172 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.172 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.172 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:43.172 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:43.172 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:43.172 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:43.172 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:43.172 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:43.173 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:43.173 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:43.173 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.173 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.173 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.173 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.173 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.173 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-12T22:19:43.173 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-12T22:19:43.174 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.174 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.174 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.174 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:43.174 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.174 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.174 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.174 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.175 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.175 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.175 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.175 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.175 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-12T22:19:43.175 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-12T22:19:43.175 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.175 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.176 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.176 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-12T22:19:43.176 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.176 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.176 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:43.176 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:43.176 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:43.176 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:43.176 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:43.177 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:43.177 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:43.177 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:43.177 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.177 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.177 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.177 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.177 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.178 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-12T22:19:43.178 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-12T22:19:43.178 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.178 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.178 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.178 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:43.178 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.178 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.178 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.179 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.179 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.179 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.179 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.179 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.179 INFO:teuthology.orchestra.run.smithi028.stdout: "min_score": { 2024-05-12T22:19:43.179 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "min_score", 2024-05-12T22:19:43.179 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-05-12T22:19:43.179 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.180 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.180 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-05-12T22:19:43.180 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.180 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.180 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.180 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-05-12T22:19:43.180 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.180 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.180 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.181 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.181 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": { 2024-05-12T22:19:43.181 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mode", 2024-05-12T22:19:43.181 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.181 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.181 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.181 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "upmap", 2024-05-12T22:19:43.181 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.181 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.182 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:43.182 INFO:teuthology.orchestra.run.smithi028.stdout: "crush-compat", 2024-05-12T22:19:43.182 INFO:teuthology.orchestra.run.smithi028.stdout: "none", 2024-05-12T22:19:43.182 INFO:teuthology.orchestra.run.smithi028.stdout: "read", 2024-05-12T22:19:43.182 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap", 2024-05-12T22:19:43.182 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap-read" 2024-05-12T22:19:43.182 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:43.182 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Balancer mode", 2024-05-12T22:19:43.183 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.183 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.183 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.183 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.183 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_ids": { 2024-05-12T22:19:43.183 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pool_ids", 2024-05-12T22:19:43.183 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.183 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.183 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.184 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:43.184 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.184 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.184 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.184 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-05-12T22:19:43.184 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.184 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.184 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.184 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.185 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-05-12T22:19:43.185 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-05-12T22:19:43.185 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-05-12T22:19:43.185 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.185 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.185 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-05-12T22:19:43.185 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.185 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.185 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.186 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-05-12T22:19:43.186 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.186 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.186 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.186 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.186 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap_max_deviation": { 2024-05-12T22:19:43.186 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "upmap_max_deviation", 2024-05-12T22:19:43.186 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-12T22:19:43.187 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.187 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.187 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-05-12T22:19:43.187 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-05-12T22:19:43.187 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.187 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.187 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "deviation below which no optimization is attempted", 2024-05-12T22:19:43.187 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-05-12T22:19:43.187 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.188 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.188 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.188 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap_max_optimizations": { 2024-05-12T22:19:43.188 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "upmap_max_optimizations", 2024-05-12T22:19:43.188 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-05-12T22:19:43.188 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.188 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.188 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-05-12T22:19:43.188 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.189 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.189 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.189 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-05-12T22:19:43.189 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.189 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.189 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.189 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:43.189 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:43.189 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.190 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-12T22:19:43.190 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cephadm", 2024-05-12T22:19:43.190 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-12T22:19:43.190 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-12T22:19:43.190 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-12T22:19:43.190 INFO:teuthology.orchestra.run.smithi028.stdout: "agent_down_multiplier": { 2024-05-12T22:19:43.190 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "agent_down_multiplier", 2024-05-12T22:19:43.190 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-05-12T22:19:43.191 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.191 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.191 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "3.0", 2024-05-12T22:19:43.191 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.191 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.191 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.191 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-05-12T22:19:43.191 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.191 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.192 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.192 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.192 INFO:teuthology.orchestra.run.smithi028.stdout: "agent_refresh_rate": { 2024-05-12T22:19:43.192 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "agent_refresh_rate", 2024-05-12T22:19:43.192 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-05-12T22:19:43.192 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.192 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.192 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "20", 2024-05-12T22:19:43.192 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.193 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.193 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.193 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-05-12T22:19:43.193 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.193 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.193 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.193 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.193 INFO:teuthology.orchestra.run.smithi028.stdout: "agent_starting_port": { 2024-05-12T22:19:43.193 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "agent_starting_port", 2024-05-12T22:19:43.194 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-12T22:19:43.194 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.194 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.194 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "4721", 2024-05-12T22:19:43.194 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.194 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.194 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.194 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-05-12T22:19:43.195 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.195 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.195 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.195 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.195 INFO:teuthology.orchestra.run.smithi028.stdout: "allow_ptrace": { 2024-05-12T22:19:43.195 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "allow_ptrace", 2024-05-12T22:19:43.195 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.195 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.196 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.196 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:43.196 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.196 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.196 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.196 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-05-12T22:19:43.196 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-05-12T22:19:43.196 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.196 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.197 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.197 INFO:teuthology.orchestra.run.smithi028.stdout: "autotune_interval": { 2024-05-12T22:19:43.197 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "autotune_interval", 2024-05-12T22:19:43.197 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-05-12T22:19:43.197 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.197 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.197 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-05-12T22:19:43.197 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.197 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.198 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.198 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to autotune daemon memory", 2024-05-12T22:19:43.198 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.198 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.198 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.198 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.198 INFO:teuthology.orchestra.run.smithi028.stdout: "autotune_memory_target_ratio": { 2024-05-12T22:19:43.198 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "autotune_memory_target_ratio", 2024-05-12T22:19:43.199 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-05-12T22:19:43.199 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.199 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.199 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0.7", 2024-05-12T22:19:43.199 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.199 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.199 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.199 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-05-12T22:19:43.200 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.200 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.200 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.200 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.200 INFO:teuthology.orchestra.run.smithi028.stdout: "cephadm_log_destination": { 2024-05-12T22:19:43.200 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cephadm_log_destination", 2024-05-12T22:19:43.200 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.200 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.200 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.201 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:43.201 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.201 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.201 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:43.201 INFO:teuthology.orchestra.run.smithi028.stdout: "file", 2024-05-12T22:19:43.201 INFO:teuthology.orchestra.run.smithi028.stdout: "file,syslog", 2024-05-12T22:19:43.201 INFO:teuthology.orchestra.run.smithi028.stdout: "syslog" 2024-05-12T22:19:43.201 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:43.202 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-05-12T22:19:43.202 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.202 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.202 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.202 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.202 INFO:teuthology.orchestra.run.smithi028.stdout: "cgroups_split": { 2024-05-12T22:19:43.202 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cgroups_split", 2024-05-12T22:19:43.202 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.202 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.203 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.203 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-12T22:19:43.203 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.203 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.203 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.203 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-05-12T22:19:43.203 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.203 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.204 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.204 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.204 INFO:teuthology.orchestra.run.smithi028.stdout: "config_checks_enabled": { 2024-05-12T22:19:43.204 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "config_checks_enabled", 2024-05-12T22:19:43.204 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.204 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.204 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.204 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:43.204 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.205 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.205 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.205 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-05-12T22:19:43.205 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.205 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.205 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.205 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.205 INFO:teuthology.orchestra.run.smithi028.stdout: "config_dashboard": { 2024-05-12T22:19:43.206 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "config_dashboard", 2024-05-12T22:19:43.206 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.206 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.206 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.206 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-12T22:19:43.206 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.206 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.206 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.206 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-05-12T22:19:43.207 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.207 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.207 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.207 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.207 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_alertmanager": { 2024-05-12T22:19:43.207 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_alertmanager", 2024-05-12T22:19:43.207 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.207 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.208 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.208 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-05-12T22:19:43.208 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.208 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.208 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.208 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-05-12T22:19:43.208 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.208 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.208 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.209 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.209 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_base": { 2024-05-12T22:19:43.209 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_base", 2024-05-12T22:19:43.209 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.209 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.209 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.209 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/ceph", 2024-05-12T22:19:43.209 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.209 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.210 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.210 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Container image name, without the tag", 2024-05-12T22:19:43.210 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.210 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.210 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.210 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.210 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_elasticsearch": { 2024-05-12T22:19:43.210 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_elasticsearch", 2024-05-12T22:19:43.211 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.211 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.211 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.211 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-05-12T22:19:43.211 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.211 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.211 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.211 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "elasticsearch container image", 2024-05-12T22:19:43.211 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.212 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.212 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.212 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.212 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_grafana": { 2024-05-12T22:19:43.212 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_grafana", 2024-05-12T22:19:43.212 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.212 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.212 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.212 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-05-12T22:19:43.213 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.213 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.213 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.213 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-05-12T22:19:43.213 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.213 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.213 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.213 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.213 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_haproxy": { 2024-05-12T22:19:43.214 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_haproxy", 2024-05-12T22:19:43.214 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.214 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.214 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.214 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-05-12T22:19:43.214 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.214 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.215 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.215 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "HAproxy container image", 2024-05-12T22:19:43.215 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.215 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.215 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.216 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.216 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_jaeger_agent": { 2024-05-12T22:19:43.216 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_jaeger_agent", 2024-05-12T22:19:43.216 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.216 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.217 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.217 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-05-12T22:19:43.217 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.217 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.217 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.217 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Jaeger agent container image", 2024-05-12T22:19:43.217 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.218 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.218 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.218 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.218 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_jaeger_collector": { 2024-05-12T22:19:43.218 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_jaeger_collector", 2024-05-12T22:19:43.218 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.218 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.218 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.219 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-05-12T22:19:43.219 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.219 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.219 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.219 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Jaeger collector container image", 2024-05-12T22:19:43.219 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.219 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.219 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.219 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.220 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_jaeger_query": { 2024-05-12T22:19:43.220 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_jaeger_query", 2024-05-12T22:19:43.220 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.220 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.220 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.220 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-05-12T22:19:43.220 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.220 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.220 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.221 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Jaeger query container image", 2024-05-12T22:19:43.221 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.221 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.221 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.221 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.221 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_keepalived": { 2024-05-12T22:19:43.221 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_keepalived", 2024-05-12T22:19:43.221 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.222 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.222 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.222 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-05-12T22:19:43.222 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.222 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.222 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.222 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Keepalived container image", 2024-05-12T22:19:43.222 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.222 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.223 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.223 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.223 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_loki": { 2024-05-12T22:19:43.223 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_loki", 2024-05-12T22:19:43.223 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.223 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.223 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.223 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-05-12T22:19:43.223 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.224 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.224 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.224 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Loki container image", 2024-05-12T22:19:43.224 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.224 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.224 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.224 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.224 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_node_exporter": { 2024-05-12T22:19:43.225 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_node_exporter", 2024-05-12T22:19:43.225 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.225 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.225 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.225 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-05-12T22:19:43.225 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.225 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.225 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.225 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-05-12T22:19:43.226 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.226 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.226 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.226 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.226 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_nvmeof": { 2024-05-12T22:19:43.226 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_nvmeof", 2024-05-12T22:19:43.226 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.226 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.227 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.227 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-05-12T22:19:43.227 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.227 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.227 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.227 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Nvme-of container image", 2024-05-12T22:19:43.227 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.227 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.227 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.228 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.228 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_prometheus": { 2024-05-12T22:19:43.228 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_prometheus", 2024-05-12T22:19:43.228 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.228 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.228 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.228 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-05-12T22:19:43.228 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.228 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.229 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.229 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-05-12T22:19:43.229 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.229 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.229 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.229 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.229 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_promtail": { 2024-05-12T22:19:43.229 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_promtail", 2024-05-12T22:19:43.229 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.230 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.230 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.230 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-05-12T22:19:43.230 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.230 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.230 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.230 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Promtail container image", 2024-05-12T22:19:43.230 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.231 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.231 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.231 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.231 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_samba": { 2024-05-12T22:19:43.231 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_samba", 2024-05-12T22:19:43.231 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.231 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.231 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.232 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-05-12T22:19:43.232 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.232 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.232 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.232 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Samba/SMB container image", 2024-05-12T22:19:43.232 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.232 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.232 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.232 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.233 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_snmp_gateway": { 2024-05-12T22:19:43.233 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_snmp_gateway", 2024-05-12T22:19:43.233 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.233 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.233 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.233 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-05-12T22:19:43.233 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.233 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.233 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.234 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SNMP Gateway container image", 2024-05-12T22:19:43.234 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.234 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.234 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.234 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.234 INFO:teuthology.orchestra.run.smithi028.stdout: "container_init": { 2024-05-12T22:19:43.234 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_init", 2024-05-12T22:19:43.234 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.235 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.235 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.235 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-12T22:19:43.235 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.235 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.235 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.235 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Run podman/docker with `--init`", 2024-05-12T22:19:43.235 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.236 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.236 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.236 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.236 INFO:teuthology.orchestra.run.smithi028.stdout: "daemon_cache_timeout": { 2024-05-12T22:19:43.236 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "daemon_cache_timeout", 2024-05-12T22:19:43.236 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-05-12T22:19:43.236 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.236 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.236 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-05-12T22:19:43.237 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.237 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.237 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.237 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-05-12T22:19:43.237 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.237 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.237 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.237 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.237 INFO:teuthology.orchestra.run.smithi028.stdout: "default_cephadm_command_timeout": { 2024-05-12T22:19:43.238 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "default_cephadm_command_timeout", 2024-05-12T22:19:43.238 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-12T22:19:43.238 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.238 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.238 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "900", 2024-05-12T22:19:43.238 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.238 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.238 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.239 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-05-12T22:19:43.239 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.239 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.239 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.239 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.239 INFO:teuthology.orchestra.run.smithi028.stdout: "default_registry": { 2024-05-12T22:19:43.239 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "default_registry", 2024-05-12T22:19:43.239 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.240 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.240 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.240 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io", 2024-05-12T22:19:43.240 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.240 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.240 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.240 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-05-12T22:19:43.240 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.240 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.241 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.241 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.241 INFO:teuthology.orchestra.run.smithi028.stdout: "device_cache_timeout": { 2024-05-12T22:19:43.241 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "device_cache_timeout", 2024-05-12T22:19:43.241 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-05-12T22:19:43.241 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.241 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.241 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "1800", 2024-05-12T22:19:43.241 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.242 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.242 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.242 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "seconds to cache device inventory", 2024-05-12T22:19:43.242 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.242 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.242 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.242 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.242 INFO:teuthology.orchestra.run.smithi028.stdout: "device_enhanced_scan": { 2024-05-12T22:19:43.243 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "device_enhanced_scan", 2024-05-12T22:19:43.243 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.243 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.243 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.243 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:43.243 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.243 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.243 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.243 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use libstoragemgmt during device scans", 2024-05-12T22:19:43.244 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.244 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.244 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.244 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.244 INFO:teuthology.orchestra.run.smithi028.stdout: "facts_cache_timeout": { 2024-05-12T22:19:43.244 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "facts_cache_timeout", 2024-05-12T22:19:43.244 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-05-12T22:19:43.244 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.245 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.245 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-05-12T22:19:43.245 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.245 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.245 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.245 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "seconds to cache host facts data", 2024-05-12T22:19:43.245 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.245 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.245 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.246 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.246 INFO:teuthology.orchestra.run.smithi028.stdout: "grafana_dashboards_path": { 2024-05-12T22:19:43.246 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "grafana_dashboards_path", 2024-05-12T22:19:43.246 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.246 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.246 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.246 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-05-12T22:19:43.246 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.247 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.247 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.247 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-05-12T22:19:43.247 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.247 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.247 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.247 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.247 INFO:teuthology.orchestra.run.smithi028.stdout: "host_check_interval": { 2024-05-12T22:19:43.247 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "host_check_interval", 2024-05-12T22:19:43.248 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-05-12T22:19:43.248 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.248 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.248 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-05-12T22:19:43.248 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.248 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.248 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.248 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to perform a host check", 2024-05-12T22:19:43.248 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.249 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.249 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.249 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.249 INFO:teuthology.orchestra.run.smithi028.stdout: "hw_monitoring": { 2024-05-12T22:19:43.249 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "hw_monitoring", 2024-05-12T22:19:43.249 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.249 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.249 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.250 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:43.250 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.250 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.250 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.250 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-05-12T22:19:43.250 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.250 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.250 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.250 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.251 INFO:teuthology.orchestra.run.smithi028.stdout: "inventory_list_all": { 2024-05-12T22:19:43.251 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "inventory_list_all", 2024-05-12T22:19:43.251 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.251 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.251 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.251 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:43.251 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.251 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.252 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.252 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-05-12T22:19:43.252 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.252 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.252 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.252 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.252 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-12T22:19:43.252 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-12T22:19:43.252 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.253 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.253 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.253 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:43.253 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.253 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.253 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:43.253 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:43.253 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:43.253 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:43.254 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:43.254 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:43.254 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:43.254 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:43.254 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.254 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.254 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.254 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.255 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.255 INFO:teuthology.orchestra.run.smithi028.stdout: "log_refresh_metadata": { 2024-05-12T22:19:43.255 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_refresh_metadata", 2024-05-12T22:19:43.255 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.255 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.255 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.255 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:43.255 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.255 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.256 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.256 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-05-12T22:19:43.256 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.256 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.256 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.256 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.256 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-12T22:19:43.256 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-12T22:19:43.256 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.257 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.257 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.257 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-12T22:19:43.257 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.257 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.257 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.257 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-05-12T22:19:43.257 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.257 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.258 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.258 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.258 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-12T22:19:43.258 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-12T22:19:43.258 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.258 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.258 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.258 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-12T22:19:43.259 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.259 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.259 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:43.259 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:43.259 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:43.259 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:43.259 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:43.259 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:43.260 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:43.260 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:43.260 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.260 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.260 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.260 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.260 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.260 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-12T22:19:43.260 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-12T22:19:43.261 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.261 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.261 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.261 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:43.261 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.261 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.261 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.261 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.261 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.262 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.262 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.262 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.262 INFO:teuthology.orchestra.run.smithi028.stdout: "manage_etc_ceph_ceph_conf": { 2024-05-12T22:19:43.262 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-05-12T22:19:43.262 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.262 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.262 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.263 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:43.263 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.263 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.263 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.263 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-05-12T22:19:43.263 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.263 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.263 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.263 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.264 INFO:teuthology.orchestra.run.smithi028.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-05-12T22:19:43.264 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-05-12T22:19:43.264 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.264 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.264 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.264 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "*", 2024-05-12T22:19:43.264 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.264 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.265 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.265 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-05-12T22:19:43.265 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.265 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.265 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.265 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.265 INFO:teuthology.orchestra.run.smithi028.stdout: "max_count_per_host": { 2024-05-12T22:19:43.265 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_count_per_host", 2024-05-12T22:19:43.265 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-12T22:19:43.266 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.266 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.266 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-05-12T22:19:43.266 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.266 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.266 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.266 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "max number of daemons per service per host", 2024-05-12T22:19:43.266 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.267 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.267 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.267 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.267 INFO:teuthology.orchestra.run.smithi028.stdout: "max_osd_draining_count": { 2024-05-12T22:19:43.267 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_osd_draining_count", 2024-05-12T22:19:43.267 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-12T22:19:43.267 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.267 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.267 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-05-12T22:19:43.268 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.268 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.268 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.268 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-05-12T22:19:43.268 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.268 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.268 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.268 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.268 INFO:teuthology.orchestra.run.smithi028.stdout: "migration_current": { 2024-05-12T22:19:43.269 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "migration_current", 2024-05-12T22:19:43.269 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-12T22:19:43.269 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.269 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.269 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:43.269 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.269 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.269 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.270 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "internal - do not modify", 2024-05-12T22:19:43.270 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.270 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.270 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.270 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.270 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": { 2024-05-12T22:19:43.270 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mode", 2024-05-12T22:19:43.270 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.270 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.271 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.271 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "root", 2024-05-12T22:19:43.271 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.271 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.271 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:43.271 INFO:teuthology.orchestra.run.smithi028.stdout: "cephadm-package", 2024-05-12T22:19:43.271 INFO:teuthology.orchestra.run.smithi028.stdout: "root" 2024-05-12T22:19:43.271 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:43.271 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "mode for remote execution of cephadm", 2024-05-12T22:19:43.272 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.272 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.272 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.272 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.272 INFO:teuthology.orchestra.run.smithi028.stdout: "oob_default_addr": { 2024-05-12T22:19:43.272 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "oob_default_addr", 2024-05-12T22:19:43.272 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.272 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.273 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.273 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "169.254.1.1", 2024-05-12T22:19:43.273 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.273 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.273 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.273 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Default address for RedFish API (oob management).", 2024-05-12T22:19:43.273 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.273 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.273 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.274 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.274 INFO:teuthology.orchestra.run.smithi028.stdout: "prometheus_alerts_path": { 2024-05-12T22:19:43.274 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "prometheus_alerts_path", 2024-05-12T22:19:43.274 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.274 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.274 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.274 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-05-12T22:19:43.274 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.274 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.275 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.275 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-05-12T22:19:43.275 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.275 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.275 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.275 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.275 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_insecure": { 2024-05-12T22:19:43.275 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_insecure", 2024-05-12T22:19:43.276 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.276 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.276 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.276 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:43.276 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.276 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.276 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.276 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-05-12T22:19:43.276 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.277 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.277 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.277 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.277 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_password": { 2024-05-12T22:19:43.277 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_password", 2024-05-12T22:19:43.277 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.277 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.277 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.278 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:43.278 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.278 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.278 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.278 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-05-12T22:19:43.278 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.278 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.278 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.278 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.279 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_url": { 2024-05-12T22:19:43.279 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_url", 2024-05-12T22:19:43.279 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.279 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.279 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.279 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:43.279 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.279 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.279 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.280 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-05-12T22:19:43.280 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.280 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.280 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.280 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.280 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_username": { 2024-05-12T22:19:43.280 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_username", 2024-05-12T22:19:43.280 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.280 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.281 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.281 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:43.281 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.281 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.281 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.281 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-05-12T22:19:43.281 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.281 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.282 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.282 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.282 INFO:teuthology.orchestra.run.smithi028.stdout: "secure_monitoring_stack": { 2024-05-12T22:19:43.282 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "secure_monitoring_stack", 2024-05-12T22:19:43.282 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.282 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.282 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.282 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:43.282 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.283 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.283 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.283 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-05-12T22:19:43.283 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.283 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.283 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.283 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.283 INFO:teuthology.orchestra.run.smithi028.stdout: "service_discovery_port": { 2024-05-12T22:19:43.284 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "service_discovery_port", 2024-05-12T22:19:43.284 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-12T22:19:43.284 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.284 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.284 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8765", 2024-05-12T22:19:43.284 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.284 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.284 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.284 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "cephadm service discovery port", 2024-05-12T22:19:43.285 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.285 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.285 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.285 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.285 INFO:teuthology.orchestra.run.smithi028.stdout: "ssh_config_file": { 2024-05-12T22:19:43.285 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssh_config_file", 2024-05-12T22:19:43.285 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.285 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.285 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.286 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:43.286 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.286 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.286 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.286 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-05-12T22:19:43.286 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.286 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.286 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.287 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.287 INFO:teuthology.orchestra.run.smithi028.stdout: "use_agent": { 2024-05-12T22:19:43.287 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "use_agent", 2024-05-12T22:19:43.287 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.287 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.287 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.287 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:43.287 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.287 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.288 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.288 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-05-12T22:19:43.288 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.288 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.288 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.288 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.288 INFO:teuthology.orchestra.run.smithi028.stdout: "use_repo_digest": { 2024-05-12T22:19:43.288 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "use_repo_digest", 2024-05-12T22:19:43.289 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.289 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.289 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.289 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-12T22:19:43.289 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.289 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.289 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.289 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-05-12T22:19:43.289 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.290 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.290 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.290 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.290 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_on_failed_host_check": { 2024-05-12T22:19:43.290 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_on_failed_host_check", 2024-05-12T22:19:43.290 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.290 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.290 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.290 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-12T22:19:43.291 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.291 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.291 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.291 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise a health warning if the host check fails", 2024-05-12T22:19:43.291 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.291 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.291 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.291 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.291 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_on_stray_daemons": { 2024-05-12T22:19:43.292 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_on_stray_daemons", 2024-05-12T22:19:43.292 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.292 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.292 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.292 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-12T22:19:43.292 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.292 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.292 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.292 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-05-12T22:19:43.293 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.293 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.293 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.293 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.293 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_on_stray_hosts": { 2024-05-12T22:19:43.293 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_on_stray_hosts", 2024-05-12T22:19:43.293 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.293 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.294 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.294 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-12T22:19:43.294 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.294 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.294 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.294 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-05-12T22:19:43.294 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.294 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.294 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.295 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:43.295 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:43.295 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.295 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-12T22:19:43.295 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crash", 2024-05-12T22:19:43.295 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-12T22:19:43.295 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-12T22:19:43.295 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-12T22:19:43.296 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-12T22:19:43.296 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-12T22:19:43.296 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.296 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.296 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.296 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:43.296 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.296 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.296 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:43.297 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:43.297 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:43.297 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:43.297 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:43.297 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:43.297 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:43.297 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:43.297 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.297 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.297 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.298 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.298 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.298 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-12T22:19:43.298 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-12T22:19:43.298 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.298 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.298 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.298 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:43.299 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.299 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.299 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.299 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.299 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.299 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.299 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.299 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.299 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-12T22:19:43.300 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-12T22:19:43.300 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.300 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.300 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.300 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-12T22:19:43.300 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.300 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.300 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:43.300 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:43.301 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:43.301 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:43.301 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:43.301 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:43.301 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:43.301 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:43.301 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.301 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.302 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.302 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.302 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.302 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-12T22:19:43.302 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-12T22:19:43.302 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.302 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.302 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.302 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:43.303 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.303 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.303 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.303 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.303 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.303 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.303 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.303 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.304 INFO:teuthology.orchestra.run.smithi028.stdout: "retain_interval": { 2024-05-12T22:19:43.304 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "retain_interval", 2024-05-12T22:19:43.304 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-05-12T22:19:43.304 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.304 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.304 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "31536000", 2024-05-12T22:19:43.304 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.304 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.304 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.305 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how long to retain crashes before pruning them", 2024-05-12T22:19:43.305 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.305 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.305 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.305 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.305 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_recent_interval": { 2024-05-12T22:19:43.305 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_recent_interval", 2024-05-12T22:19:43.305 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-05-12T22:19:43.306 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.306 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.306 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "1209600", 2024-05-12T22:19:43.306 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.306 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.306 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.306 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "time interval in which to warn about recent crashes", 2024-05-12T22:19:43.306 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.306 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.307 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.307 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:43.307 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:43.307 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.307 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-12T22:19:43.307 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "dashboard", 2024-05-12T22:19:43.307 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-12T22:19:43.307 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-12T22:19:43.308 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-12T22:19:43.308 INFO:teuthology.orchestra.run.smithi028.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-05-12T22:19:43.308 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-05-12T22:19:43.308 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-12T22:19:43.308 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.308 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.308 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-05-12T22:19:43.308 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.309 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.309 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.309 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.309 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.309 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.309 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.309 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.309 INFO:teuthology.orchestra.run.smithi028.stdout: "ALERTMANAGER_API_HOST": { 2024-05-12T22:19:43.309 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ALERTMANAGER_API_HOST", 2024-05-12T22:19:43.310 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.310 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.310 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.310 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:43.310 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.310 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.310 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.310 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.310 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.311 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.311 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.311 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.311 INFO:teuthology.orchestra.run.smithi028.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-05-12T22:19:43.311 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-05-12T22:19:43.311 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.311 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.311 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.311 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-12T22:19:43.312 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.312 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.312 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.312 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.312 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.312 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.312 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.312 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.313 INFO:teuthology.orchestra.run.smithi028.stdout: "AUDIT_API_ENABLED": { 2024-05-12T22:19:43.313 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "AUDIT_API_ENABLED", 2024-05-12T22:19:43.313 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.313 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.313 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.313 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:43.313 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.313 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.313 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.314 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.314 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.314 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.314 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.314 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.314 INFO:teuthology.orchestra.run.smithi028.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-05-12T22:19:43.314 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-05-12T22:19:43.314 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.315 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.315 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.315 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-12T22:19:43.315 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.315 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.315 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.315 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.315 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.315 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.316 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.316 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.316 INFO:teuthology.orchestra.run.smithi028.stdout: "ENABLE_BROWSABLE_API": { 2024-05-12T22:19:43.316 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ENABLE_BROWSABLE_API", 2024-05-12T22:19:43.316 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.316 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.316 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.316 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-12T22:19:43.316 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.317 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.317 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.317 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.317 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.317 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.317 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.317 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.317 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-05-12T22:19:43.317 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-05-12T22:19:43.318 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.318 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.318 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.318 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-12T22:19:43.318 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.318 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.318 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.318 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.319 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.319 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.319 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.319 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.319 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-05-12T22:19:43.319 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-05-12T22:19:43.319 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.319 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.319 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.319 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-12T22:19:43.320 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.320 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.320 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.320 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.320 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.320 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.320 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.320 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.320 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-05-12T22:19:43.321 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-05-12T22:19:43.321 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.321 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.321 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.321 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-12T22:19:43.321 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.321 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.321 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.321 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.322 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.322 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.322 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.322 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.322 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-05-12T22:19:43.322 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-05-12T22:19:43.322 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.322 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.323 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.323 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-12T22:19:43.323 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.323 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.323 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.323 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.323 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.323 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.323 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.324 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.324 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_NFS": { 2024-05-12T22:19:43.324 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-05-12T22:19:43.324 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.324 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.324 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.324 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-12T22:19:43.324 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.324 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.325 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.325 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.325 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.325 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.325 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.325 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.325 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_RBD": { 2024-05-12T22:19:43.325 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-05-12T22:19:43.325 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.326 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.326 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.326 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-12T22:19:43.326 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.326 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.326 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.326 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.326 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.327 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.327 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.327 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.327 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_RGW": { 2024-05-12T22:19:43.327 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-05-12T22:19:43.327 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.327 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.327 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.327 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-12T22:19:43.328 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.328 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.328 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.328 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.328 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.328 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.328 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.328 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.328 INFO:teuthology.orchestra.run.smithi028.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-05-12T22:19:43.329 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-05-12T22:19:43.329 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.329 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.329 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.329 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:43.329 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.329 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.329 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.329 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.330 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.330 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.330 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.330 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.330 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_PASSWORD": { 2024-05-12T22:19:43.330 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_PASSWORD", 2024-05-12T22:19:43.330 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.330 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.331 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.331 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "admin", 2024-05-12T22:19:43.331 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.331 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.331 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.331 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.331 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.331 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.331 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.332 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.332 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-05-12T22:19:43.332 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-05-12T22:19:43.332 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.332 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.332 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.332 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-12T22:19:43.332 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.332 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.333 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.333 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.333 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.333 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.333 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.333 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.333 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_URL": { 2024-05-12T22:19:43.333 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_URL", 2024-05-12T22:19:43.334 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.334 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.334 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.334 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:43.334 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.334 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.334 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.334 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.334 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.335 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.335 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.335 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.335 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_USERNAME": { 2024-05-12T22:19:43.335 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_USERNAME", 2024-05-12T22:19:43.335 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.335 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.335 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.335 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "admin", 2024-05-12T22:19:43.336 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.336 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.336 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.336 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.336 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.336 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.336 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.336 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.337 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-05-12T22:19:43.337 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-05-12T22:19:43.337 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.337 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.337 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.337 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:43.337 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.337 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.337 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.338 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.338 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.338 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.338 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.338 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.338 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-05-12T22:19:43.338 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-05-12T22:19:43.338 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.338 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.339 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.339 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:43.339 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.339 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.339 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.339 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.339 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.339 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.340 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.340 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.340 INFO:teuthology.orchestra.run.smithi028.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-05-12T22:19:43.340 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-05-12T22:19:43.340 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.340 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.340 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.340 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-12T22:19:43.340 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.341 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.341 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.341 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.341 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.341 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.341 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.341 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.341 INFO:teuthology.orchestra.run.smithi028.stdout: "ISSUE_TRACKER_API_KEY": { 2024-05-12T22:19:43.341 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-05-12T22:19:43.342 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.342 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.342 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.342 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:43.342 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.342 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.342 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.342 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.343 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.343 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.343 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.343 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.343 INFO:teuthology.orchestra.run.smithi028.stdout: "PROMETHEUS_API_HOST": { 2024-05-12T22:19:43.343 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PROMETHEUS_API_HOST", 2024-05-12T22:19:43.343 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.343 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.343 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.344 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:43.344 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.344 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.344 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.344 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.344 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.344 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.344 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.344 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.345 INFO:teuthology.orchestra.run.smithi028.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-05-12T22:19:43.345 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-05-12T22:19:43.345 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.345 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.345 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.345 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-12T22:19:43.345 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.345 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.345 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.346 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.346 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.346 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.346 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.346 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.346 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-05-12T22:19:43.346 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-05-12T22:19:43.346 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.347 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.347 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.347 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:43.347 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.347 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.347 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.347 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.347 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.347 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.348 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.348 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.348 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-05-12T22:19:43.348 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-05-12T22:19:43.348 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.348 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.348 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.348 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:43.348 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.349 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.349 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.349 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.349 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.349 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.349 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.349 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.349 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-05-12T22:19:43.349 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-05-12T22:19:43.350 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.350 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.350 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.350 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-12T22:19:43.350 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.350 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.350 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.350 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.351 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.351 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.351 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.351 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.351 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-05-12T22:19:43.351 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-05-12T22:19:43.351 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.351 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.353 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.353 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-12T22:19:43.353 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.353 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.353 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.353 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.354 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.354 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.354 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.354 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.354 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-05-12T22:19:43.355 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-05-12T22:19:43.355 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.355 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.355 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.355 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:43.355 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.355 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.355 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.355 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.356 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.356 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.356 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.356 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.356 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-05-12T22:19:43.356 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-05-12T22:19:43.356 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.356 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.356 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.357 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:43.357 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.357 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.357 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.357 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.357 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.357 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.357 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.358 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.358 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-05-12T22:19:43.358 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-05-12T22:19:43.358 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.358 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.358 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.358 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:43.358 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.358 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.359 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.359 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.359 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.359 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.359 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.359 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.359 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_ENABLED": { 2024-05-12T22:19:43.359 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_ENABLED", 2024-05-12T22:19:43.359 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.360 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.360 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.360 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-12T22:19:43.360 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.360 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.360 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.360 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.360 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.360 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.361 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.361 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.361 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-05-12T22:19:43.361 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-05-12T22:19:43.361 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.361 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.361 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.361 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-05-12T22:19:43.362 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.362 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.362 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.362 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.362 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.362 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.362 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.362 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.363 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-05-12T22:19:43.363 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-05-12T22:19:43.363 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-12T22:19:43.363 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.363 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.363 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-05-12T22:19:43.363 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.363 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.363 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.364 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.364 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.364 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.364 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.364 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.364 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-05-12T22:19:43.364 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-05-12T22:19:43.364 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-12T22:19:43.364 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.365 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.365 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8", 2024-05-12T22:19:43.365 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.365 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.365 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.365 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.365 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.365 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.365 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.366 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.366 INFO:teuthology.orchestra.run.smithi028.stdout: "REST_REQUESTS_TIMEOUT": { 2024-05-12T22:19:43.366 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-05-12T22:19:43.366 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-12T22:19:43.366 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.366 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.366 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "45", 2024-05-12T22:19:43.366 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.366 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.367 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.367 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.367 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.367 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.367 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.367 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.367 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_ACCESS_KEY": { 2024-05-12T22:19:43.367 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_ACCESS_KEY", 2024-05-12T22:19:43.367 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.368 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.368 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.368 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:43.368 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.368 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.368 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.368 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.368 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.368 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.369 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.369 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.369 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-05-12T22:19:43.369 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-05-12T22:19:43.369 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.369 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.370 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.370 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "admin", 2024-05-12T22:19:43.370 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.370 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.370 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.370 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.370 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.370 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.370 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.371 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.371 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_SECRET_KEY": { 2024-05-12T22:19:43.371 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_SECRET_KEY", 2024-05-12T22:19:43.371 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.371 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.371 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.372 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:43.372 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.372 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.372 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.372 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.372 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.372 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.372 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.373 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.373 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_SSL_VERIFY": { 2024-05-12T22:19:43.373 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_SSL_VERIFY", 2024-05-12T22:19:43.373 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.373 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.373 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.373 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-12T22:19:43.373 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.374 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.374 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.374 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.374 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.374 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.374 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.375 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.375 INFO:teuthology.orchestra.run.smithi028.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-05-12T22:19:43.375 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-05-12T22:19:43.375 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-12T22:19:43.375 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.375 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.375 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-05-12T22:19:43.376 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.376 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.376 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.376 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.376 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.376 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.376 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.377 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.377 INFO:teuthology.orchestra.run.smithi028.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-05-12T22:19:43.377 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-05-12T22:19:43.377 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-12T22:19:43.377 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.377 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.377 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-05-12T22:19:43.378 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.378 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.378 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.378 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.378 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.378 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.378 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.379 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.379 INFO:teuthology.orchestra.run.smithi028.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-05-12T22:19:43.379 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-05-12T22:19:43.379 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-12T22:19:43.379 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.379 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.379 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-05-12T22:19:43.380 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.380 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.380 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.380 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.380 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.380 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.381 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.381 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.381 INFO:teuthology.orchestra.run.smithi028.stdout: "cross_origin_url": { 2024-05-12T22:19:43.381 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cross_origin_url", 2024-05-12T22:19:43.381 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.381 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.381 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.382 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:43.382 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.382 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.382 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.382 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.382 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.383 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.383 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.383 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.383 INFO:teuthology.orchestra.run.smithi028.stdout: "crt_file": { 2024-05-12T22:19:43.383 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crt_file", 2024-05-12T22:19:43.383 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.383 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.383 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.383 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:43.384 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.384 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.384 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.384 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.384 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.384 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.384 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.384 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.385 INFO:teuthology.orchestra.run.smithi028.stdout: "debug": { 2024-05-12T22:19:43.385 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "debug", 2024-05-12T22:19:43.385 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.385 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.385 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.385 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:43.385 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.385 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.386 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.386 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Enable/disable debug options", 2024-05-12T22:19:43.386 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.386 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.386 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.386 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.386 INFO:teuthology.orchestra.run.smithi028.stdout: "jwt_token_ttl": { 2024-05-12T22:19:43.386 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "jwt_token_ttl", 2024-05-12T22:19:43.386 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-12T22:19:43.387 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.387 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.387 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "28800", 2024-05-12T22:19:43.387 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.387 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.387 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.387 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.387 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.388 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.388 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.388 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.388 INFO:teuthology.orchestra.run.smithi028.stdout: "key_file": { 2024-05-12T22:19:43.388 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "key_file", 2024-05-12T22:19:43.388 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.388 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.388 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.389 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:43.389 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.389 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.389 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.389 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.389 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.389 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.389 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.390 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.390 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-12T22:19:43.390 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-12T22:19:43.390 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.390 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.390 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.390 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:43.391 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.391 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.391 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:43.391 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:43.391 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:43.391 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:43.391 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:43.391 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:43.391 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:43.392 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:43.392 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.392 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.392 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.392 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.392 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.392 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-12T22:19:43.392 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-12T22:19:43.392 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.393 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.393 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.393 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:43.393 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.393 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.393 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.393 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.393 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.394 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.394 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.394 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.394 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-12T22:19:43.394 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-12T22:19:43.394 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.394 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.394 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.394 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-12T22:19:43.395 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.395 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.395 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:43.395 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:43.395 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:43.395 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:43.395 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:43.395 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:43.395 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:43.396 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:43.396 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.396 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.396 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.396 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.396 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.396 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-12T22:19:43.396 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-12T22:19:43.396 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.397 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.397 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.397 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:43.397 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.397 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.397 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.397 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.397 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.397 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.398 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.398 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.398 INFO:teuthology.orchestra.run.smithi028.stdout: "motd": { 2024-05-12T22:19:43.398 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "motd", 2024-05-12T22:19:43.398 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.398 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.398 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.398 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:43.398 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.399 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.399 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.399 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "The message of the day", 2024-05-12T22:19:43.399 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.399 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.399 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.399 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.399 INFO:teuthology.orchestra.run.smithi028.stdout: "redirect_resolve_ip_addr": { 2024-05-12T22:19:43.399 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "redirect_resolve_ip_addr", 2024-05-12T22:19:43.400 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.400 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.400 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.400 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:43.400 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.400 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.400 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.400 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.400 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.401 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.401 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.401 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.401 INFO:teuthology.orchestra.run.smithi028.stdout: "server_addr": { 2024-05-12T22:19:43.401 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_addr", 2024-05-12T22:19:43.401 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.401 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.401 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.401 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "::", 2024-05-12T22:19:43.402 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.402 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.402 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.402 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.402 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.402 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.402 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.402 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.402 INFO:teuthology.orchestra.run.smithi028.stdout: "server_port": { 2024-05-12T22:19:43.403 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_port", 2024-05-12T22:19:43.403 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-12T22:19:43.403 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.403 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.403 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8080", 2024-05-12T22:19:43.403 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.403 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.403 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.403 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.404 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.404 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.404 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.404 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.404 INFO:teuthology.orchestra.run.smithi028.stdout: "ssl": { 2024-05-12T22:19:43.404 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssl", 2024-05-12T22:19:43.404 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.404 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.404 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.405 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-12T22:19:43.405 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.405 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.405 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.405 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.405 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.405 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.406 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.406 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.406 INFO:teuthology.orchestra.run.smithi028.stdout: "ssl_server_port": { 2024-05-12T22:19:43.406 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssl_server_port", 2024-05-12T22:19:43.406 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-12T22:19:43.406 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.406 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.407 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8443", 2024-05-12T22:19:43.407 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.407 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.407 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.407 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.407 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.407 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.407 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.407 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.408 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_behaviour": { 2024-05-12T22:19:43.408 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_behaviour", 2024-05-12T22:19:43.408 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.408 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.408 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.408 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "redirect", 2024-05-12T22:19:43.408 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.408 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.408 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:43.409 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:43.409 INFO:teuthology.orchestra.run.smithi028.stdout: "redirect" 2024-05-12T22:19:43.409 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:43.409 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.409 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.409 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.409 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.409 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.409 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_error_status_code": { 2024-05-12T22:19:43.410 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_error_status_code", 2024-05-12T22:19:43.410 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-12T22:19:43.410 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.410 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.410 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "500", 2024-05-12T22:19:43.410 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "400", 2024-05-12T22:19:43.410 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "599", 2024-05-12T22:19:43.411 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.411 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.411 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.411 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.411 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.411 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.411 INFO:teuthology.orchestra.run.smithi028.stdout: "url_prefix": { 2024-05-12T22:19:43.411 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "url_prefix", 2024-05-12T22:19:43.412 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.412 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.412 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.412 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:43.412 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.412 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.412 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.412 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.413 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.413 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.413 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.413 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:43.413 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:43.413 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.413 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-12T22:19:43.414 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "devicehealth", 2024-05-12T22:19:43.414 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-12T22:19:43.414 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-12T22:19:43.414 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-12T22:19:43.414 INFO:teuthology.orchestra.run.smithi028.stdout: "enable_monitoring": { 2024-05-12T22:19:43.414 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enable_monitoring", 2024-05-12T22:19:43.414 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.415 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.415 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.415 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-12T22:19:43.415 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.415 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.415 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.415 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "monitor device health metrics", 2024-05-12T22:19:43.415 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.416 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.416 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.416 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.416 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-12T22:19:43.416 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-12T22:19:43.416 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.417 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.417 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.417 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:43.417 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.417 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.417 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:43.417 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:43.418 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:43.418 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:43.418 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:43.418 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:43.418 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:43.418 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:43.418 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.419 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.419 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.419 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.419 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.419 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-12T22:19:43.419 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-12T22:19:43.420 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.420 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.420 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.420 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:43.420 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.420 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.421 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.421 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.421 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.421 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.421 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.421 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.421 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-12T22:19:43.421 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-12T22:19:43.422 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.422 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.422 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.422 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-12T22:19:43.422 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.422 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.422 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:43.422 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:43.422 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:43.423 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:43.423 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:43.423 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:43.423 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:43.423 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:43.423 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.423 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.423 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.423 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.424 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.424 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-12T22:19:43.424 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-12T22:19:43.424 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.424 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.424 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.424 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:43.424 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.424 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.425 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.425 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.425 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.425 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.425 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.425 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.425 INFO:teuthology.orchestra.run.smithi028.stdout: "mark_out_threshold": { 2024-05-12T22:19:43.425 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mark_out_threshold", 2024-05-12T22:19:43.426 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-05-12T22:19:43.426 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.426 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.426 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "2419200", 2024-05-12T22:19:43.426 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.426 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.426 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.426 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-05-12T22:19:43.426 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.427 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.427 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.427 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.427 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": { 2024-05-12T22:19:43.427 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pool_name", 2024-05-12T22:19:43.427 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.427 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.427 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.427 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "device_health_metrics", 2024-05-12T22:19:43.428 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.428 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.428 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.428 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "name of pool in which to store device health metrics", 2024-05-12T22:19:43.428 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.428 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.428 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.428 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.428 INFO:teuthology.orchestra.run.smithi028.stdout: "retention_period": { 2024-05-12T22:19:43.429 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "retention_period", 2024-05-12T22:19:43.429 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-05-12T22:19:43.429 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.429 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.429 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "15552000", 2024-05-12T22:19:43.429 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.429 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.429 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.429 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how long to retain device health metrics", 2024-05-12T22:19:43.430 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.430 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.430 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.430 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.430 INFO:teuthology.orchestra.run.smithi028.stdout: "scrape_frequency": { 2024-05-12T22:19:43.430 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "scrape_frequency", 2024-05-12T22:19:43.430 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-05-12T22:19:43.430 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.430 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.431 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "86400", 2024-05-12T22:19:43.431 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.431 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.431 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.431 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to scrape device health metrics", 2024-05-12T22:19:43.431 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.431 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.431 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.432 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.432 INFO:teuthology.orchestra.run.smithi028.stdout: "self_heal": { 2024-05-12T22:19:43.432 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "self_heal", 2024-05-12T22:19:43.432 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.432 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.432 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.432 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-12T22:19:43.432 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.432 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.432 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.433 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-05-12T22:19:43.433 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.433 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.433 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.433 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.433 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-05-12T22:19:43.433 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-05-12T22:19:43.433 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-05-12T22:19:43.434 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.434 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.434 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-05-12T22:19:43.434 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.434 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.434 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.434 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to wake up and check device health", 2024-05-12T22:19:43.434 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.434 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.435 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.435 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.435 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_threshold": { 2024-05-12T22:19:43.435 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_threshold", 2024-05-12T22:19:43.435 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-05-12T22:19:43.435 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.435 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.435 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "7257600", 2024-05-12T22:19:43.435 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.436 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.436 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.436 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-05-12T22:19:43.436 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.436 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.436 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.436 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:43.436 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:43.436 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.437 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-12T22:19:43.437 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "diskprediction_local", 2024-05-12T22:19:43.437 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-12T22:19:43.437 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-12T22:19:43.437 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-12T22:19:43.437 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-12T22:19:43.437 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-12T22:19:43.437 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.437 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.438 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.438 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:43.438 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.438 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.438 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:43.438 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:43.438 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:43.438 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:43.438 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:43.439 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:43.439 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:43.439 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:43.439 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.439 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.439 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.439 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.439 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.439 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-12T22:19:43.440 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-12T22:19:43.440 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.440 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.440 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.440 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:43.440 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.440 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.440 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.440 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.441 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.441 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.441 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.441 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.441 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-12T22:19:43.441 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-12T22:19:43.441 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.441 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.441 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.442 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-12T22:19:43.442 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.442 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.442 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:43.442 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:43.442 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:43.442 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:43.442 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:43.442 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:43.443 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:43.443 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:43.443 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.443 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.443 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.443 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.443 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.444 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-12T22:19:43.444 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-12T22:19:43.444 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.444 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.444 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.444 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:43.444 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.444 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.445 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.445 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.445 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.445 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.445 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.445 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.445 INFO:teuthology.orchestra.run.smithi028.stdout: "predict_interval": { 2024-05-12T22:19:43.445 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "predict_interval", 2024-05-12T22:19:43.445 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.446 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.446 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.446 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "86400", 2024-05-12T22:19:43.446 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.447 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.447 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.447 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.447 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.447 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.447 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.447 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.447 INFO:teuthology.orchestra.run.smithi028.stdout: "predictor_model": { 2024-05-12T22:19:43.448 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "predictor_model", 2024-05-12T22:19:43.448 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.448 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.448 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.448 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "prophetstor", 2024-05-12T22:19:43.448 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.448 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.449 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.449 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.449 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.449 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.449 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.449 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.449 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-05-12T22:19:43.449 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-05-12T22:19:43.450 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.450 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.450 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.450 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-05-12T22:19:43.450 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.450 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.451 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.451 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.451 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.451 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.451 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.451 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:43.451 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:43.451 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.452 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-12T22:19:43.452 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "influx", 2024-05-12T22:19:43.452 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": false, 2024-05-12T22:19:43.452 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "influxdb python module not found", 2024-05-12T22:19:43.452 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-12T22:19:43.452 INFO:teuthology.orchestra.run.smithi028.stdout: "batch_size": { 2024-05-12T22:19:43.453 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "batch_size", 2024-05-12T22:19:43.453 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-12T22:19:43.453 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.453 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.453 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5000", 2024-05-12T22:19:43.453 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.453 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.454 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.454 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-05-12T22:19:43.454 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.454 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.454 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.454 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.455 INFO:teuthology.orchestra.run.smithi028.stdout: "database": { 2024-05-12T22:19:43.455 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "database", 2024-05-12T22:19:43.455 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.455 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.455 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.455 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "ceph", 2024-05-12T22:19:43.456 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.456 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.456 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.456 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-05-12T22:19:43.456 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.456 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.456 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.457 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.457 INFO:teuthology.orchestra.run.smithi028.stdout: "hostname": { 2024-05-12T22:19:43.457 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "hostname", 2024-05-12T22:19:43.457 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.457 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.457 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.457 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:43.457 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.457 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.458 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.458 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "InfluxDB server hostname", 2024-05-12T22:19:43.458 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.458 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.458 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.458 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.458 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-05-12T22:19:43.458 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-05-12T22:19:43.458 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-05-12T22:19:43.459 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.459 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.459 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "30", 2024-05-12T22:19:43.459 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "5", 2024-05-12T22:19:43.459 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.459 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.459 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-05-12T22:19:43.459 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.460 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.460 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.460 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.460 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-12T22:19:43.460 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-12T22:19:43.460 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.460 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.460 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.460 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:43.461 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.461 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.461 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:43.461 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:43.461 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:43.461 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:43.461 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:43.461 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:43.461 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:43.462 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:43.462 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.462 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.462 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.462 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.462 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.462 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-12T22:19:43.462 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-12T22:19:43.463 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.463 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.463 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.463 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:43.463 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.463 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.463 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.463 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.463 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.464 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.464 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.464 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.464 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-12T22:19:43.464 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-12T22:19:43.464 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.464 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.464 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.464 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-12T22:19:43.465 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.465 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.465 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:43.465 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:43.465 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:43.465 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:43.465 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:43.465 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:43.465 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:43.466 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:43.466 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.466 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.466 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.466 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.466 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.466 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-12T22:19:43.466 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-12T22:19:43.467 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.467 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.467 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.467 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:43.467 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.467 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.467 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.467 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.467 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.468 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.468 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.468 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.468 INFO:teuthology.orchestra.run.smithi028.stdout: "password": { 2024-05-12T22:19:43.468 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "password", 2024-05-12T22:19:43.468 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.468 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.468 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.468 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:43.469 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.469 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.469 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.469 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "password of InfluxDB server user", 2024-05-12T22:19:43.469 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.469 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.469 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.469 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.470 INFO:teuthology.orchestra.run.smithi028.stdout: "port": { 2024-05-12T22:19:43.470 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "port", 2024-05-12T22:19:43.470 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-12T22:19:43.470 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.470 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.470 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8086", 2024-05-12T22:19:43.470 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.470 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.470 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.471 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "InfluxDB server port", 2024-05-12T22:19:43.471 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.471 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.471 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.471 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.471 INFO:teuthology.orchestra.run.smithi028.stdout: "ssl": { 2024-05-12T22:19:43.471 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssl", 2024-05-12T22:19:43.471 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.471 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.472 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.472 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "false", 2024-05-12T22:19:43.472 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.472 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.472 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.472 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-05-12T22:19:43.472 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.472 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.472 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.473 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.473 INFO:teuthology.orchestra.run.smithi028.stdout: "threads": { 2024-05-12T22:19:43.473 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "threads", 2024-05-12T22:19:43.473 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-12T22:19:43.473 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.473 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.473 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-05-12T22:19:43.473 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-05-12T22:19:43.473 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "32", 2024-05-12T22:19:43.474 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.474 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-05-12T22:19:43.474 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.474 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.474 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.474 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.474 INFO:teuthology.orchestra.run.smithi028.stdout: "username": { 2024-05-12T22:19:43.474 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "username", 2024-05-12T22:19:43.475 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.475 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.475 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.475 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:43.475 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.475 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.475 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.476 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "username of InfluxDB server user", 2024-05-12T22:19:43.476 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.476 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.476 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.476 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.476 INFO:teuthology.orchestra.run.smithi028.stdout: "verify_ssl": { 2024-05-12T22:19:43.476 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "verify_ssl", 2024-05-12T22:19:43.476 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.476 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.477 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.477 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "true", 2024-05-12T22:19:43.477 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.477 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.477 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.477 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-05-12T22:19:43.477 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.477 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.477 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.478 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:43.478 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:43.478 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.478 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-12T22:19:43.478 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "insights", 2024-05-12T22:19:43.478 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-12T22:19:43.478 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-12T22:19:43.478 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-12T22:19:43.479 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-12T22:19:43.479 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-12T22:19:43.479 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.479 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.479 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.479 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:43.480 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.480 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.480 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:43.480 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:43.480 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:43.480 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:43.480 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:43.480 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:43.480 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:43.481 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:43.481 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.481 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.481 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.481 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.481 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.481 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-12T22:19:43.482 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-12T22:19:43.482 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.482 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.482 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.482 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:43.482 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.482 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.483 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.483 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.483 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.483 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.483 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.483 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.483 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-12T22:19:43.483 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-12T22:19:43.484 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.484 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.484 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.484 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-12T22:19:43.484 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.484 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.484 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:43.485 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:43.485 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:43.485 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:43.485 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:43.485 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:43.485 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:43.485 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:43.485 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.485 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.486 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.486 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.486 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.486 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-12T22:19:43.486 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-12T22:19:43.486 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.486 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.486 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.486 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:43.487 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.487 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.487 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.487 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.487 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.487 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.487 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.487 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:43.488 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:43.488 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.488 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-12T22:19:43.488 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "iostat", 2024-05-12T22:19:43.488 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-12T22:19:43.488 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-12T22:19:43.489 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-12T22:19:43.489 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-12T22:19:43.489 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-12T22:19:43.489 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.489 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.489 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.489 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:43.489 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.490 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.490 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:43.490 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:43.490 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:43.490 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:43.490 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:43.491 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:43.491 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:43.491 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:43.491 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.491 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.491 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.492 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.492 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.492 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-12T22:19:43.492 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-12T22:19:43.492 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.492 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.492 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.492 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:43.493 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.493 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.493 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.493 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.493 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.493 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.493 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.494 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.494 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-12T22:19:43.494 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-12T22:19:43.494 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.494 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.494 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.495 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-12T22:19:43.495 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.495 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.495 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:43.495 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:43.495 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:43.495 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:43.495 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:43.496 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:43.496 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:43.496 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:43.496 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.496 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.496 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.496 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.496 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.496 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-12T22:19:43.497 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-12T22:19:43.497 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.497 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.497 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.497 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:43.497 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.497 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.497 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.497 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.498 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.498 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.498 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.498 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:43.498 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:43.498 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.498 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-12T22:19:43.498 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "k8sevents", 2024-05-12T22:19:43.499 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-12T22:19:43.499 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-12T22:19:43.499 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-12T22:19:43.499 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_event_retention_days": { 2024-05-12T22:19:43.499 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ceph_event_retention_days", 2024-05-12T22:19:43.499 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-12T22:19:43.499 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.499 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.499 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "7", 2024-05-12T22:19:43.500 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.500 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.500 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.500 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Days to hold ceph event information within local cache", 2024-05-12T22:19:43.500 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.500 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.500 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.500 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.500 INFO:teuthology.orchestra.run.smithi028.stdout: "config_check_secs": { 2024-05-12T22:19:43.501 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "config_check_secs", 2024-05-12T22:19:43.501 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-12T22:19:43.501 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.501 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.501 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-05-12T22:19:43.501 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "10", 2024-05-12T22:19:43.501 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.501 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.501 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-05-12T22:19:43.502 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.502 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.502 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.502 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.502 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-12T22:19:43.502 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-12T22:19:43.502 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.502 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.502 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.503 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:43.503 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.503 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.503 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:43.503 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:43.503 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:43.503 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:43.503 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:43.503 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:43.504 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:43.504 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:43.504 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.504 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.504 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.504 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.504 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.504 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-12T22:19:43.504 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-12T22:19:43.505 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.505 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.505 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.505 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:43.505 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.505 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.505 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.505 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.505 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.506 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.506 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.506 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.506 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-12T22:19:43.506 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-12T22:19:43.506 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.506 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.506 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.506 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-12T22:19:43.507 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.507 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.507 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:43.507 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:43.507 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:43.507 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:43.507 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:43.507 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:43.507 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:43.508 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:43.508 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.508 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.508 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.508 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.508 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.508 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-12T22:19:43.508 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-12T22:19:43.508 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.509 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.509 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.509 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:43.509 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.509 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.509 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.509 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.509 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.509 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.510 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.510 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:43.510 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:43.510 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.510 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-12T22:19:43.510 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "localpool", 2024-05-12T22:19:43.510 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-12T22:19:43.510 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-12T22:19:43.510 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-12T22:19:43.511 INFO:teuthology.orchestra.run.smithi028.stdout: "failure_domain": { 2024-05-12T22:19:43.511 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "failure_domain", 2024-05-12T22:19:43.511 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.511 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.511 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.511 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "host", 2024-05-12T22:19:43.511 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.511 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.511 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.512 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "failure domain for any created local pool", 2024-05-12T22:19:43.512 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-05-12T22:19:43.512 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.512 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.512 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.512 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-12T22:19:43.512 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-12T22:19:43.512 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.512 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.513 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.513 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:43.513 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.513 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.513 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:43.513 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:43.513 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:43.513 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:43.514 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:43.514 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:43.514 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:43.514 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:43.514 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.514 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.514 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.514 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.515 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.515 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-12T22:19:43.515 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-12T22:19:43.515 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.515 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.515 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.515 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:43.515 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.515 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.515 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.516 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.516 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.516 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.516 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.516 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.516 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-12T22:19:43.516 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-12T22:19:43.516 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.516 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.517 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.517 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-12T22:19:43.517 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.517 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.517 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:43.517 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:43.517 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:43.518 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:43.518 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:43.518 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:43.518 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:43.518 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:43.518 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.518 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.519 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.519 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.519 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.519 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-12T22:19:43.519 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-12T22:19:43.519 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.519 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.519 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.520 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:43.520 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.520 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.520 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.520 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.520 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.521 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.521 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.521 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.521 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": { 2024-05-12T22:19:43.521 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "min_size", 2024-05-12T22:19:43.521 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-12T22:19:43.521 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.521 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.521 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:43.522 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.522 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.522 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.522 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "default min_size for any created local pool", 2024-05-12T22:19:43.522 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-05-12T22:19:43.522 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.523 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.523 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.523 INFO:teuthology.orchestra.run.smithi028.stdout: "num_rep": { 2024-05-12T22:19:43.523 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "num_rep", 2024-05-12T22:19:43.523 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-12T22:19:43.523 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.523 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.523 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "3", 2024-05-12T22:19:43.523 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.524 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.524 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.524 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "default replica count for any created local pool", 2024-05-12T22:19:43.524 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.524 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.524 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.524 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.525 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": { 2024-05-12T22:19:43.525 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pg_num", 2024-05-12T22:19:43.525 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-12T22:19:43.525 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.525 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.525 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "128", 2024-05-12T22:19:43.525 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.525 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.526 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.526 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "default pg_num for any created local pool", 2024-05-12T22:19:43.526 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.526 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.526 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.526 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.527 INFO:teuthology.orchestra.run.smithi028.stdout: "prefix": { 2024-05-12T22:19:43.527 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "prefix", 2024-05-12T22:19:43.527 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.527 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.527 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.527 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:43.527 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.528 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.528 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.528 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "name prefix for any created local pool", 2024-05-12T22:19:43.528 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.528 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.528 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.528 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.529 INFO:teuthology.orchestra.run.smithi028.stdout: "subtree": { 2024-05-12T22:19:43.529 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "subtree", 2024-05-12T22:19:43.529 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.529 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.529 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.529 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "rack", 2024-05-12T22:19:43.530 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.530 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.530 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.530 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "CRUSH level for which to create a local pool", 2024-05-12T22:19:43.530 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-05-12T22:19:43.530 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.530 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.531 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:43.531 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:43.531 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.531 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-12T22:19:43.531 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mds_autoscaler", 2024-05-12T22:19:43.531 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-12T22:19:43.531 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-12T22:19:43.531 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-12T22:19:43.532 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-12T22:19:43.532 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-12T22:19:43.532 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.532 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.532 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.532 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:43.532 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.532 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.532 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:43.532 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:43.533 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:43.533 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:43.533 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:43.533 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:43.533 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:43.533 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:43.533 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.533 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.534 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.534 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.534 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.534 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-12T22:19:43.534 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-12T22:19:43.534 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.534 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.534 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.535 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:43.535 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.535 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.535 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.535 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.535 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.535 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.535 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.535 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.536 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-12T22:19:43.536 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-12T22:19:43.536 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.536 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.536 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.536 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-12T22:19:43.536 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.536 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.536 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:43.537 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:43.537 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:43.537 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:43.537 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:43.537 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:43.537 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:43.537 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:43.537 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.537 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.538 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.538 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.538 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.538 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-12T22:19:43.538 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-12T22:19:43.538 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.538 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.538 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.539 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:43.539 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.539 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.539 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.539 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.539 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.539 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.539 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.539 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:43.540 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:43.540 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.540 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-12T22:19:43.540 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mirroring", 2024-05-12T22:19:43.540 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-12T22:19:43.540 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-12T22:19:43.540 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-12T22:19:43.540 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-12T22:19:43.541 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-12T22:19:43.541 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.541 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.541 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.541 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:43.541 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.541 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.541 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:43.541 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:43.542 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:43.542 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:43.542 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:43.542 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:43.542 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:43.542 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:43.542 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.542 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.543 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.543 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.543 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.543 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-12T22:19:43.543 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-12T22:19:43.543 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.543 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.543 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.544 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:43.544 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.544 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.544 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.544 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.544 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.544 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.544 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.544 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.545 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-12T22:19:43.545 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-12T22:19:43.545 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.545 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.545 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.545 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-12T22:19:43.545 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.545 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.546 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:43.546 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:43.546 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:43.546 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:43.546 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:43.546 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:43.546 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:43.546 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:43.546 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.547 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.547 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.547 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.547 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.547 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-12T22:19:43.547 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-12T22:19:43.547 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.547 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.547 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.548 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:43.548 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.548 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.548 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.548 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.548 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.548 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.548 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.548 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:43.549 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:43.549 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.549 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-12T22:19:43.549 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "nfs", 2024-05-12T22:19:43.549 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-12T22:19:43.549 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-12T22:19:43.549 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-12T22:19:43.549 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-12T22:19:43.549 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-12T22:19:43.550 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.550 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.550 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.550 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:43.550 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.550 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.550 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:43.550 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:43.550 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:43.551 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:43.551 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:43.551 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:43.551 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:43.551 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:43.551 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.551 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.551 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.552 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.552 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.552 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-12T22:19:43.552 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-12T22:19:43.552 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.552 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.552 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.552 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:43.552 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.553 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.553 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.553 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.553 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.553 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.553 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.553 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.554 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-12T22:19:43.554 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-12T22:19:43.554 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.554 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.554 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.554 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-12T22:19:43.554 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.554 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.555 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:43.555 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:43.555 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:43.555 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:43.555 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:43.555 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:43.555 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:43.555 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:43.556 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.556 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.556 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.556 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.556 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.556 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-12T22:19:43.557 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-12T22:19:43.557 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.557 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.557 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.557 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:43.557 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.557 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.557 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.558 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.558 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.558 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.558 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.558 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:43.558 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:43.559 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.559 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-12T22:19:43.561 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "orchestrator", 2024-05-12T22:19:43.561 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-12T22:19:43.561 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-12T22:19:43.562 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-12T22:19:43.562 INFO:teuthology.orchestra.run.smithi028.stdout: "fail_fs": { 2024-05-12T22:19:43.562 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "fail_fs", 2024-05-12T22:19:43.562 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.562 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.562 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.562 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:43.562 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.562 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.563 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.563 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-05-12T22:19:43.563 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.563 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.563 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.563 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.563 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-12T22:19:43.563 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-12T22:19:43.563 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.564 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.564 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.564 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:43.564 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.564 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.564 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:43.564 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:43.564 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:43.565 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:43.565 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:43.565 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:43.565 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:43.565 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:43.565 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.565 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.565 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.566 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.566 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.566 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-12T22:19:43.566 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-12T22:19:43.566 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.566 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.566 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.567 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:43.567 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.567 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.567 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.567 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.567 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.567 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.568 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.568 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.568 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-12T22:19:43.568 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-12T22:19:43.568 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.568 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.568 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.569 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-12T22:19:43.569 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.569 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.569 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:43.569 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:43.569 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:43.570 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:43.570 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:43.570 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:43.570 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:43.570 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:43.570 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.571 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.571 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.571 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.571 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.571 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-12T22:19:43.571 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-12T22:19:43.571 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.572 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.572 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.572 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:43.572 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.572 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.572 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.572 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.572 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.572 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.573 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.573 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.573 INFO:teuthology.orchestra.run.smithi028.stdout: "orchestrator": { 2024-05-12T22:19:43.573 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "orchestrator", 2024-05-12T22:19:43.573 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.573 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.573 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.573 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:43.573 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.574 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.574 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:43.574 INFO:teuthology.orchestra.run.smithi028.stdout: "cephadm", 2024-05-12T22:19:43.574 INFO:teuthology.orchestra.run.smithi028.stdout: "rook", 2024-05-12T22:19:43.574 INFO:teuthology.orchestra.run.smithi028.stdout: "test_orchestrator" 2024-05-12T22:19:43.574 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:43.574 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Orchestrator backend", 2024-05-12T22:19:43.574 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.575 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.575 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.575 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:43.575 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:43.575 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.575 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-12T22:19:43.575 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "osd_perf_query", 2024-05-12T22:19:43.575 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-12T22:19:43.575 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-12T22:19:43.576 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-12T22:19:43.576 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-12T22:19:43.576 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-12T22:19:43.576 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.576 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.576 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.576 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:43.576 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.577 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.577 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:43.577 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:43.577 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:43.577 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:43.577 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:43.577 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:43.577 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:43.577 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:43.578 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.578 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.578 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.578 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.578 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.578 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-12T22:19:43.578 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-12T22:19:43.578 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.578 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.579 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.579 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:43.579 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.579 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.579 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.579 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.579 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.579 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.579 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.580 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.580 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-12T22:19:43.580 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-12T22:19:43.580 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.580 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.580 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.580 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-12T22:19:43.580 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.580 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.581 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:43.581 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:43.581 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:43.581 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:43.581 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:43.581 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:43.581 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:43.581 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:43.581 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.582 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.582 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.582 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.582 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.582 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-12T22:19:43.582 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-12T22:19:43.582 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.582 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.582 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.583 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:43.583 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.583 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.583 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.583 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.583 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.583 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.583 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.584 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:43.584 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:43.584 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.584 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-12T22:19:43.584 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "osd_support", 2024-05-12T22:19:43.584 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-12T22:19:43.584 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-12T22:19:43.584 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-12T22:19:43.584 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-12T22:19:43.585 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-12T22:19:43.585 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.585 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.585 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.585 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:43.585 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.585 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.585 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:43.586 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:43.586 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:43.586 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:43.586 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:43.586 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:43.586 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:43.586 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:43.586 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.586 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.587 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.587 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.587 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.587 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-12T22:19:43.587 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-12T22:19:43.587 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.587 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.587 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.588 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:43.588 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.588 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.588 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.588 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.588 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.588 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.588 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.588 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.589 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-12T22:19:43.589 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-12T22:19:43.589 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.589 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.589 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.589 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-12T22:19:43.589 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.589 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.589 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:43.590 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:43.590 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:43.590 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:43.590 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:43.590 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:43.590 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:43.590 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:43.590 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.590 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.591 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.591 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.591 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.591 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-12T22:19:43.591 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-12T22:19:43.591 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.591 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.591 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.591 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:43.592 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.592 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.592 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.592 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.592 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.592 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.592 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.592 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:43.592 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:43.593 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.593 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-12T22:19:43.593 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pg_autoscaler", 2024-05-12T22:19:43.593 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-12T22:19:43.593 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-12T22:19:43.593 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-12T22:19:43.593 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-12T22:19:43.593 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-12T22:19:43.593 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.594 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.594 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.594 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:43.594 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.594 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.594 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:43.594 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:43.595 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:43.595 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:43.595 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:43.595 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:43.595 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:43.595 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:43.595 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.595 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.596 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.596 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.596 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.596 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-12T22:19:43.596 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-12T22:19:43.596 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.596 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.596 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.597 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:43.597 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.597 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.597 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.597 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.597 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.598 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.598 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.598 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.598 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-12T22:19:43.598 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-12T22:19:43.598 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.598 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.598 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.599 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-12T22:19:43.599 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.599 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.599 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:43.599 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:43.599 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:43.600 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:43.600 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:43.600 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:43.600 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:43.600 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:43.600 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.600 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.600 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.600 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.601 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.601 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-12T22:19:43.601 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-12T22:19:43.601 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.601 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.601 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.601 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:43.602 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.602 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.602 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.602 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.602 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.602 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.602 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.603 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.603 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-05-12T22:19:43.603 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-05-12T22:19:43.603 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-05-12T22:19:43.603 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.603 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.603 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-05-12T22:19:43.604 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.604 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.604 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.604 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.604 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.604 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.604 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.605 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.605 INFO:teuthology.orchestra.run.smithi028.stdout: "threshold": { 2024-05-12T22:19:43.605 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "threshold", 2024-05-12T22:19:43.605 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-05-12T22:19:43.605 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.605 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.605 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "3.0", 2024-05-12T22:19:43.606 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1.0", 2024-05-12T22:19:43.606 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.606 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.606 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "scaling threshold", 2024-05-12T22:19:43.606 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-05-12T22:19:43.606 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.607 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.607 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:43.607 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:43.607 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.607 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-12T22:19:43.607 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "progress", 2024-05-12T22:19:43.608 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-12T22:19:43.608 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-12T22:19:43.608 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-12T22:19:43.608 INFO:teuthology.orchestra.run.smithi028.stdout: "allow_pg_recovery_event": { 2024-05-12T22:19:43.608 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "allow_pg_recovery_event", 2024-05-12T22:19:43.608 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.608 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.608 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.609 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:43.609 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.609 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.609 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.609 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "allow the module to show pg recovery progress", 2024-05-12T22:19:43.609 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.609 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.609 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.609 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.610 INFO:teuthology.orchestra.run.smithi028.stdout: "enabled": { 2024-05-12T22:19:43.610 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enabled", 2024-05-12T22:19:43.610 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.610 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.610 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.610 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-12T22:19:43.610 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.610 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.610 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.611 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.611 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.611 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.611 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.611 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.611 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-12T22:19:43.611 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-12T22:19:43.611 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.612 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.612 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.612 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:43.612 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.612 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.612 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:43.612 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:43.612 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:43.613 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:43.613 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:43.613 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:43.613 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:43.613 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:43.613 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.613 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.614 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.614 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.614 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.614 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-12T22:19:43.614 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-12T22:19:43.614 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.614 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.614 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.615 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:43.615 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.615 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.615 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.615 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.615 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.615 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.616 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.616 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.616 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-12T22:19:43.616 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-12T22:19:43.616 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.616 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.616 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.616 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-12T22:19:43.617 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.617 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.617 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:43.617 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:43.617 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:43.617 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:43.617 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:43.617 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:43.618 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:43.618 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:43.618 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.618 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.618 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.618 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.618 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.619 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-12T22:19:43.619 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-12T22:19:43.619 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.619 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.619 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.619 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:43.619 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.619 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.620 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.620 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.620 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.620 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.620 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.620 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.620 INFO:teuthology.orchestra.run.smithi028.stdout: "max_completed_events": { 2024-05-12T22:19:43.620 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_completed_events", 2024-05-12T22:19:43.620 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-12T22:19:43.621 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.621 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.621 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "50", 2024-05-12T22:19:43.621 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.621 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.621 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.621 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "number of past completed events to remember", 2024-05-12T22:19:43.621 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.622 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.622 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.622 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.622 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-05-12T22:19:43.622 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-05-12T22:19:43.622 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-05-12T22:19:43.622 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.622 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.622 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-05-12T22:19:43.623 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.623 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.623 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.623 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how long the module is going to sleep", 2024-05-12T22:19:43.623 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.623 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.623 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.623 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:43.624 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:43.624 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.624 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-12T22:19:43.624 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "prometheus", 2024-05-12T22:19:43.624 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-12T22:19:43.624 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-12T22:19:43.624 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-12T22:19:43.625 INFO:teuthology.orchestra.run.smithi028.stdout: "cache": { 2024-05-12T22:19:43.625 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cache", 2024-05-12T22:19:43.625 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.625 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.625 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.625 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-12T22:19:43.625 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.625 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.625 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.626 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.626 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.626 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.626 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.626 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.626 INFO:teuthology.orchestra.run.smithi028.stdout: "exclude_perf_counters": { 2024-05-12T22:19:43.626 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "exclude_perf_counters", 2024-05-12T22:19:43.627 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.627 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.627 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.627 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-12T22:19:43.627 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.627 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.627 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.627 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-05-12T22:19:43.627 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-05-12T22:19:43.628 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.628 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.628 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.628 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-12T22:19:43.628 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-12T22:19:43.628 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.628 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.628 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.628 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:43.629 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.629 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.629 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:43.629 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:43.629 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:43.629 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:43.629 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:43.629 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:43.630 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:43.630 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:43.630 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.630 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.630 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.630 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.630 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.630 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-12T22:19:43.630 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-12T22:19:43.631 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.631 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.631 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.631 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:43.631 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.631 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.631 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.631 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.631 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.632 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.632 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.632 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.632 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-12T22:19:43.632 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-12T22:19:43.632 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.632 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.632 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.633 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-12T22:19:43.633 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.633 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.633 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:43.633 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:43.633 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:43.633 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:43.633 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:43.633 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:43.634 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:43.634 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:43.634 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.634 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.634 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.634 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.634 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.634 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-12T22:19:43.634 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-12T22:19:43.635 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.635 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.635 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.635 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:43.635 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.635 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.635 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.635 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.635 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.636 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.636 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.636 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.636 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_stats_pools": { 2024-05-12T22:19:43.636 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rbd_stats_pools", 2024-05-12T22:19:43.636 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.636 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.636 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.636 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:43.637 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.637 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.637 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.637 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.637 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.637 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.637 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.637 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.637 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_stats_pools_refresh_interval": { 2024-05-12T22:19:43.638 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-05-12T22:19:43.638 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-12T22:19:43.638 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.638 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.638 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "300", 2024-05-12T22:19:43.638 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.638 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.638 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.638 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.639 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.639 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.639 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.639 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.639 INFO:teuthology.orchestra.run.smithi028.stdout: "scrape_interval": { 2024-05-12T22:19:43.639 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "scrape_interval", 2024-05-12T22:19:43.639 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-05-12T22:19:43.639 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.640 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.640 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "15.0", 2024-05-12T22:19:43.640 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.640 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.640 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.640 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.640 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.640 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.640 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.641 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.641 INFO:teuthology.orchestra.run.smithi028.stdout: "server_addr": { 2024-05-12T22:19:43.641 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_addr", 2024-05-12T22:19:43.641 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.641 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.641 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.641 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "::", 2024-05-12T22:19:43.641 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.641 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.641 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.642 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-05-12T22:19:43.642 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.642 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.642 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.642 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.642 INFO:teuthology.orchestra.run.smithi028.stdout: "server_port": { 2024-05-12T22:19:43.642 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_port", 2024-05-12T22:19:43.642 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-12T22:19:43.643 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.643 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.643 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "9283", 2024-05-12T22:19:43.643 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.643 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.643 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.643 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-05-12T22:19:43.643 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.644 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.644 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.644 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.644 INFO:teuthology.orchestra.run.smithi028.stdout: "stale_cache_strategy": { 2024-05-12T22:19:43.644 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "stale_cache_strategy", 2024-05-12T22:19:43.644 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.644 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.644 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.644 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "log", 2024-05-12T22:19:43.645 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.645 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.645 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.645 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.645 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.645 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.645 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.645 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.645 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_behaviour": { 2024-05-12T22:19:43.646 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_behaviour", 2024-05-12T22:19:43.646 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.646 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.646 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.646 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "default", 2024-05-12T22:19:43.646 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.646 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.646 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:43.646 INFO:teuthology.orchestra.run.smithi028.stdout: "default", 2024-05-12T22:19:43.647 INFO:teuthology.orchestra.run.smithi028.stdout: "error" 2024-05-12T22:19:43.647 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:43.647 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.647 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.647 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.647 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.647 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.647 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_error_status_code": { 2024-05-12T22:19:43.647 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_error_status_code", 2024-05-12T22:19:43.648 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-12T22:19:43.648 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.648 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.648 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "500", 2024-05-12T22:19:43.648 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "400", 2024-05-12T22:19:43.648 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "599", 2024-05-12T22:19:43.648 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.648 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.649 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.649 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.649 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.649 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:43.649 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:43.649 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.649 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-12T22:19:43.649 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rbd_support", 2024-05-12T22:19:43.650 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-12T22:19:43.650 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-12T22:19:43.650 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-12T22:19:43.650 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-12T22:19:43.650 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-12T22:19:43.650 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.650 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.650 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.651 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:43.651 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.651 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.651 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:43.651 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:43.651 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:43.651 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:43.651 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:43.652 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:43.652 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:43.652 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:43.652 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.652 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.652 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.652 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.652 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.653 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-12T22:19:43.653 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-12T22:19:43.653 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.653 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.653 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.653 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:43.653 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.653 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.654 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.654 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.654 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.654 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.654 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.654 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.654 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-12T22:19:43.654 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-12T22:19:43.655 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.655 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.655 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.655 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-12T22:19:43.655 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.655 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.655 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:43.656 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:43.656 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:43.656 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:43.656 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:43.656 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:43.656 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:43.656 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:43.656 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.656 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.657 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.657 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.657 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.657 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-12T22:19:43.657 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-12T22:19:43.657 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.657 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.657 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.658 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:43.658 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.658 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.658 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.658 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.658 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.658 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.658 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.658 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.659 INFO:teuthology.orchestra.run.smithi028.stdout: "max_concurrent_snap_create": { 2024-05-12T22:19:43.659 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_concurrent_snap_create", 2024-05-12T22:19:43.659 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-12T22:19:43.659 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.659 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.659 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-05-12T22:19:43.659 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.659 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.659 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.660 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.660 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.660 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.660 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.660 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.660 INFO:teuthology.orchestra.run.smithi028.stdout: "mirror_snapshot_schedule": { 2024-05-12T22:19:43.660 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mirror_snapshot_schedule", 2024-05-12T22:19:43.660 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.660 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.661 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.661 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:43.661 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.661 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.661 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.661 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.661 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.661 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.662 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.662 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.662 INFO:teuthology.orchestra.run.smithi028.stdout: "trash_purge_schedule": { 2024-05-12T22:19:43.662 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "trash_purge_schedule", 2024-05-12T22:19:43.662 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.662 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.662 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.662 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:43.663 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.663 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.663 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.663 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.663 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.663 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.663 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.663 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:43.663 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:43.664 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.664 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-12T22:19:43.664 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "restful", 2024-05-12T22:19:43.664 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-12T22:19:43.664 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-12T22:19:43.664 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-12T22:19:43.664 INFO:teuthology.orchestra.run.smithi028.stdout: "enable_auth": { 2024-05-12T22:19:43.664 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enable_auth", 2024-05-12T22:19:43.665 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.665 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.665 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.665 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-12T22:19:43.665 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.665 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.665 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.665 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.665 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.665 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.666 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.666 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.666 INFO:teuthology.orchestra.run.smithi028.stdout: "key_file": { 2024-05-12T22:19:43.666 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "key_file", 2024-05-12T22:19:43.666 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.666 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.666 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.666 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:43.667 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.667 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.667 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.667 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.667 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.667 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.667 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.667 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.667 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-12T22:19:43.668 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-12T22:19:43.668 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.668 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.668 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.668 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:43.668 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.668 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.668 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:43.668 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:43.669 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:43.669 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:43.669 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:43.669 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:43.669 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:43.669 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:43.669 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.669 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.669 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.670 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.670 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.670 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-12T22:19:43.670 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-12T22:19:43.670 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.670 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.670 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.670 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:43.670 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.671 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.671 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.671 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.671 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.671 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.671 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.671 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.671 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-12T22:19:43.671 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-12T22:19:43.672 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.672 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.672 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.672 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-12T22:19:43.672 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.672 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.672 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:43.672 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:43.673 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:43.673 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:43.673 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:43.673 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:43.673 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:43.673 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:43.673 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.673 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.673 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.674 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.674 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.674 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-12T22:19:43.674 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-12T22:19:43.674 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.674 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.674 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.674 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:43.675 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.675 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.675 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.675 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.675 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.675 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.675 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.675 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.675 INFO:teuthology.orchestra.run.smithi028.stdout: "server_addr": { 2024-05-12T22:19:43.676 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_addr", 2024-05-12T22:19:43.676 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.676 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.676 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.676 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:43.676 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.676 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.676 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.676 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.677 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.677 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.677 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.677 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.677 INFO:teuthology.orchestra.run.smithi028.stdout: "server_port": { 2024-05-12T22:19:43.677 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_port", 2024-05-12T22:19:43.677 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.677 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.677 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.678 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:43.678 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.678 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.678 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.678 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.678 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.678 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.678 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.679 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:43.679 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:43.679 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.679 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-12T22:19:43.679 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rgw", 2024-05-12T22:19:43.679 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-12T22:19:43.679 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-12T22:19:43.679 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-12T22:19:43.680 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-12T22:19:43.680 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-12T22:19:43.680 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.680 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.680 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.680 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:43.680 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.680 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.680 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:43.680 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:43.681 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:43.681 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:43.681 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:43.681 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:43.681 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:43.681 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:43.681 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.681 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.682 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.682 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.682 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.682 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-12T22:19:43.682 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-12T22:19:43.682 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.682 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.682 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.682 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:43.683 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.683 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.683 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.683 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.683 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.683 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.683 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.683 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.683 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-12T22:19:43.684 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-12T22:19:43.684 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.684 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.684 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.684 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-12T22:19:43.684 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.684 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.684 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:43.684 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:43.685 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:43.685 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:43.685 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:43.685 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:43.685 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:43.685 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:43.685 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.685 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.685 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.686 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.686 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.686 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-12T22:19:43.686 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-12T22:19:43.686 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.686 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.686 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.686 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:43.686 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.687 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.687 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.687 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.687 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.687 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.687 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.687 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:43.687 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:43.687 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.688 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-12T22:19:43.688 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rook", 2024-05-12T22:19:43.688 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-12T22:19:43.688 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-12T22:19:43.688 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-12T22:19:43.688 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-12T22:19:43.688 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-12T22:19:43.688 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.688 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.689 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.689 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:43.689 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.689 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.689 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:43.689 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:43.689 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:43.689 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:43.689 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:43.690 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:43.690 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:43.690 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:43.690 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.690 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.690 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.690 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.690 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.690 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-12T22:19:43.691 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-12T22:19:43.691 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.691 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.691 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.691 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:43.691 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.691 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.691 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.692 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.692 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.692 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.692 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.692 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.692 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-12T22:19:43.692 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-12T22:19:43.692 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.692 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.693 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.693 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-12T22:19:43.693 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.693 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.693 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:43.693 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:43.693 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:43.693 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:43.693 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:43.694 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:43.694 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:43.694 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:43.694 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.694 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.694 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.694 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.694 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.695 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-12T22:19:43.695 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-12T22:19:43.695 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.695 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.695 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.695 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:43.695 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.695 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.695 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.696 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.696 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.696 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.696 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.696 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.696 INFO:teuthology.orchestra.run.smithi028.stdout: "storage_class": { 2024-05-12T22:19:43.696 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "storage_class", 2024-05-12T22:19:43.696 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.696 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.697 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.697 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "local", 2024-05-12T22:19:43.697 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.697 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.697 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.697 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-05-12T22:19:43.697 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.697 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.697 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.698 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:43.698 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:43.698 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.698 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-12T22:19:43.698 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "selftest", 2024-05-12T22:19:43.698 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-12T22:19:43.698 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-12T22:19:43.698 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-12T22:19:43.699 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-12T22:19:43.699 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-12T22:19:43.699 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.699 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.699 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.699 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:43.699 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.699 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.699 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:43.700 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:43.700 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:43.700 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:43.700 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:43.700 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:43.700 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:43.700 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:43.700 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.700 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.701 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.701 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.701 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.701 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-12T22:19:43.701 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-12T22:19:43.701 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.701 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.701 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.702 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:43.702 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.702 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.702 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.702 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.702 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.702 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.702 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.702 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.703 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-12T22:19:43.703 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-12T22:19:43.703 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.703 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.703 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.703 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-12T22:19:43.703 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.703 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.703 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:43.704 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:43.704 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:43.704 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:43.704 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:43.704 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:43.704 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:43.704 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:43.704 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.704 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.705 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.705 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.705 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.705 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-12T22:19:43.705 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-12T22:19:43.705 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.705 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.705 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.706 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:43.706 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.706 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.706 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.706 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.706 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.706 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.706 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.706 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.707 INFO:teuthology.orchestra.run.smithi028.stdout: "roption1": { 2024-05-12T22:19:43.707 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "roption1", 2024-05-12T22:19:43.707 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.707 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.707 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.707 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:43.707 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.708 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.708 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.708 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.708 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.708 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.708 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.708 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.708 INFO:teuthology.orchestra.run.smithi028.stdout: "roption2": { 2024-05-12T22:19:43.708 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "roption2", 2024-05-12T22:19:43.709 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.709 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.709 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.709 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "xyz", 2024-05-12T22:19:43.709 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.709 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.709 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.709 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.710 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.710 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.710 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.710 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.710 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption1": { 2024-05-12T22:19:43.710 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption1", 2024-05-12T22:19:43.710 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.710 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.710 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.711 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:43.711 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.711 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.711 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.711 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.711 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.711 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.711 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.711 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.712 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption2": { 2024-05-12T22:19:43.712 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption2", 2024-05-12T22:19:43.712 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-12T22:19:43.712 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.712 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.712 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:43.712 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.712 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.712 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.713 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.713 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.713 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.713 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.713 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.713 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption3": { 2024-05-12T22:19:43.713 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption3", 2024-05-12T22:19:43.713 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-05-12T22:19:43.713 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.714 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.714 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:43.714 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.714 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.714 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.714 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.714 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.714 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.714 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.715 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.715 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption4": { 2024-05-12T22:19:43.715 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption4", 2024-05-12T22:19:43.715 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.715 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.715 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.715 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:43.715 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.715 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.716 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.716 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.716 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.716 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.716 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.716 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.716 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption5": { 2024-05-12T22:19:43.716 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption5", 2024-05-12T22:19:43.716 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.717 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.717 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.717 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:43.717 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.717 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.717 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.717 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.717 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.717 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.718 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.718 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.718 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption6": { 2024-05-12T22:19:43.718 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption6", 2024-05-12T22:19:43.718 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.718 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.718 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.718 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-12T22:19:43.718 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.719 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.719 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.719 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.719 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.719 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.719 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.719 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.719 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption7": { 2024-05-12T22:19:43.719 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption7", 2024-05-12T22:19:43.720 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-12T22:19:43.720 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.720 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.720 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:43.720 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-05-12T22:19:43.720 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "42", 2024-05-12T22:19:43.720 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.720 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.720 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.721 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.721 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.721 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.721 INFO:teuthology.orchestra.run.smithi028.stdout: "testkey": { 2024-05-12T22:19:43.721 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "testkey", 2024-05-12T22:19:43.721 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.721 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.721 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.721 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:43.722 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.722 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.722 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.722 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.722 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.722 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.722 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.722 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.723 INFO:teuthology.orchestra.run.smithi028.stdout: "testlkey": { 2024-05-12T22:19:43.723 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "testlkey", 2024-05-12T22:19:43.723 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.723 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.723 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.723 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:43.723 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.723 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.724 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.724 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.724 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.724 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.724 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.724 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.724 INFO:teuthology.orchestra.run.smithi028.stdout: "testnewline": { 2024-05-12T22:19:43.724 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "testnewline", 2024-05-12T22:19:43.724 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.725 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.725 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.725 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:43.725 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.725 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.725 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.725 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.725 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.725 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.726 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.726 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:43.726 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:43.726 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.726 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-12T22:19:43.726 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "snap_schedule", 2024-05-12T22:19:43.726 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-12T22:19:43.726 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-12T22:19:43.726 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-12T22:19:43.727 INFO:teuthology.orchestra.run.smithi028.stdout: "allow_m_granularity": { 2024-05-12T22:19:43.727 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "allow_m_granularity", 2024-05-12T22:19:43.727 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.727 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.727 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.727 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:43.727 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.727 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.727 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.728 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "allow minute scheduled snapshots", 2024-05-12T22:19:43.728 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.728 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.728 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.728 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.728 INFO:teuthology.orchestra.run.smithi028.stdout: "dump_on_update": { 2024-05-12T22:19:43.728 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "dump_on_update", 2024-05-12T22:19:43.728 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.729 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.729 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.729 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:43.729 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.729 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.729 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.729 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "dump database to debug log on update", 2024-05-12T22:19:43.729 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.730 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.730 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.730 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.730 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-12T22:19:43.730 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-12T22:19:43.730 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.730 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.730 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.730 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:43.731 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.731 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.731 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:43.731 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:43.731 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:43.731 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:43.731 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:43.731 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:43.732 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:43.732 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:43.732 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.732 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.732 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.732 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.732 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.732 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-12T22:19:43.732 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-12T22:19:43.733 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.733 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.733 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.733 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:43.733 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.733 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.733 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.733 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.734 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.734 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.734 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.734 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.734 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-12T22:19:43.734 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-12T22:19:43.734 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.735 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.735 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.735 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-12T22:19:43.735 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.735 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.735 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:43.735 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:43.735 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:43.736 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:43.736 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:43.736 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:43.736 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:43.736 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:43.736 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.736 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.736 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.736 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.737 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.737 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-12T22:19:43.737 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-12T22:19:43.737 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.737 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.737 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.737 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:43.737 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.737 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.738 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.738 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.738 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.738 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.738 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.738 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:43.738 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:43.738 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.738 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-12T22:19:43.739 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "stats", 2024-05-12T22:19:43.739 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-12T22:19:43.739 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-12T22:19:43.739 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-12T22:19:43.739 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-12T22:19:43.739 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-12T22:19:43.739 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.739 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.739 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.740 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:43.740 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.740 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.740 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:43.740 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:43.740 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:43.740 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:43.740 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:43.740 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:43.741 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:43.741 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:43.741 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.741 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.741 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.741 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.741 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.741 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-12T22:19:43.741 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-12T22:19:43.742 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.742 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.742 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.742 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:43.742 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.742 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.742 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.742 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.742 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.743 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.743 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.743 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.743 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-12T22:19:43.743 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-12T22:19:43.743 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.743 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.743 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.743 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-12T22:19:43.744 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.744 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.744 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:43.744 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:43.744 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:43.744 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:43.744 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:43.744 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:43.745 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:43.745 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:43.745 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.745 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.745 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.745 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.745 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.745 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-12T22:19:43.745 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-12T22:19:43.746 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.746 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.746 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.746 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:43.746 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.746 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.746 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.746 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.746 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.747 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.747 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.747 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:43.747 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:43.747 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.747 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-12T22:19:43.747 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "status", 2024-05-12T22:19:43.747 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-12T22:19:43.747 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-12T22:19:43.748 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-12T22:19:43.748 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-12T22:19:43.748 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-12T22:19:43.748 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.748 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.748 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.748 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:43.748 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.748 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.749 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:43.749 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:43.749 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:43.749 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:43.749 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:43.749 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:43.749 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:43.749 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:43.749 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.750 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.750 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.750 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.750 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.750 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-12T22:19:43.750 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-12T22:19:43.750 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.750 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.750 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.751 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:43.751 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.751 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.751 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.751 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.751 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.751 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.751 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.752 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.752 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-12T22:19:43.752 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-12T22:19:43.752 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.752 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.752 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.752 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-12T22:19:43.752 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.752 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.753 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:43.753 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:43.753 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:43.753 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:43.753 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:43.753 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:43.753 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:43.753 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:43.753 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.754 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.754 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.754 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.754 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.754 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-12T22:19:43.754 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-12T22:19:43.754 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.754 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.755 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.755 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:43.755 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.755 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.755 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.755 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.755 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.755 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.755 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.756 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:43.756 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:43.756 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.756 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-12T22:19:43.756 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "telegraf", 2024-05-12T22:19:43.756 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-12T22:19:43.756 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-12T22:19:43.756 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-12T22:19:43.756 INFO:teuthology.orchestra.run.smithi028.stdout: "address": { 2024-05-12T22:19:43.757 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "address", 2024-05-12T22:19:43.757 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.757 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.757 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.757 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-05-12T22:19:43.757 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.757 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.757 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.757 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.758 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.758 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.758 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.758 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.758 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-05-12T22:19:43.758 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-05-12T22:19:43.758 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-05-12T22:19:43.758 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.759 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.759 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "15", 2024-05-12T22:19:43.759 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.759 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.759 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.759 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.759 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.759 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.759 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.760 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.760 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-12T22:19:43.760 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-12T22:19:43.760 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.760 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.760 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.760 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:43.760 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.761 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.761 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:43.761 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:43.761 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:43.761 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:43.761 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:43.761 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:43.761 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:43.762 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:43.762 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.762 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.762 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.762 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.762 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.762 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-12T22:19:43.762 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-12T22:19:43.762 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.763 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.763 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.763 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:43.763 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.763 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.763 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.763 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.763 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.763 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.764 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.764 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.764 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-12T22:19:43.764 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-12T22:19:43.764 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.764 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.764 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.764 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-12T22:19:43.765 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.765 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.765 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:43.765 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:43.765 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:43.765 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:43.765 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:43.765 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:43.765 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:43.766 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:43.766 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.766 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.766 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.766 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.766 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.766 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-12T22:19:43.766 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-12T22:19:43.766 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.767 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.767 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.767 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:43.767 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.767 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.767 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.767 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.767 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.767 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.769 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.769 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:43.769 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:43.769 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.769 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-12T22:19:43.770 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "telemetry", 2024-05-12T22:19:43.770 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-12T22:19:43.770 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-12T22:19:43.770 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-12T22:19:43.770 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_basic": { 2024-05-12T22:19:43.770 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_basic", 2024-05-12T22:19:43.770 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.770 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.771 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.771 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-12T22:19:43.771 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.771 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.771 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.771 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share basic cluster information (size, version)", 2024-05-12T22:19:43.771 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.771 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.771 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.772 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.772 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_crash": { 2024-05-12T22:19:43.772 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_crash", 2024-05-12T22:19:43.772 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.772 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.772 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.772 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-12T22:19:43.772 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.773 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.773 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.773 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-05-12T22:19:43.773 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.773 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.773 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.773 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.773 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_device": { 2024-05-12T22:19:43.773 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_device", 2024-05-12T22:19:43.774 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.774 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.774 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.774 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-12T22:19:43.774 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.774 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.774 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.774 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-05-12T22:19:43.775 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.775 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.775 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.775 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.775 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_ident": { 2024-05-12T22:19:43.775 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_ident", 2024-05-12T22:19:43.775 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.775 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.775 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.776 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:43.776 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.776 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.776 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.776 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-05-12T22:19:43.776 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.776 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.776 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.776 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.777 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_perf": { 2024-05-12T22:19:43.777 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_perf", 2024-05-12T22:19:43.777 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.777 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.777 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.777 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:43.777 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.777 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.777 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.778 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share various performance metrics of a cluster", 2024-05-12T22:19:43.778 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.778 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.778 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.778 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.778 INFO:teuthology.orchestra.run.smithi028.stdout: "contact": { 2024-05-12T22:19:43.778 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "contact", 2024-05-12T22:19:43.778 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.778 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.779 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.779 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:43.779 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.779 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.779 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.779 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.779 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.779 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.779 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.780 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.780 INFO:teuthology.orchestra.run.smithi028.stdout: "description": { 2024-05-12T22:19:43.780 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "description", 2024-05-12T22:19:43.780 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.780 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.780 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.780 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:43.780 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.780 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.781 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.781 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.781 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.781 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.781 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.781 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.781 INFO:teuthology.orchestra.run.smithi028.stdout: "device_url": { 2024-05-12T22:19:43.781 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "device_url", 2024-05-12T22:19:43.781 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.782 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.782 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.782 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-05-12T22:19:43.782 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.782 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.782 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.782 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.782 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.782 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.783 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.783 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.783 INFO:teuthology.orchestra.run.smithi028.stdout: "enabled": { 2024-05-12T22:19:43.783 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enabled", 2024-05-12T22:19:43.783 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.783 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.783 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.783 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:43.783 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.784 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.784 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.784 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.784 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.784 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.784 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.784 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.784 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-05-12T22:19:43.784 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-05-12T22:19:43.785 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-12T22:19:43.785 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.785 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.785 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "24", 2024-05-12T22:19:43.785 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "8", 2024-05-12T22:19:43.785 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.785 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.785 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.785 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.786 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.786 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.786 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.786 INFO:teuthology.orchestra.run.smithi028.stdout: "last_opt_revision": { 2024-05-12T22:19:43.786 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "last_opt_revision", 2024-05-12T22:19:43.786 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-12T22:19:43.786 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.786 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.787 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "1", 2024-05-12T22:19:43.787 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.787 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.787 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.787 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.787 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.787 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.787 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.787 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.788 INFO:teuthology.orchestra.run.smithi028.stdout: "leaderboard": { 2024-05-12T22:19:43.788 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "leaderboard", 2024-05-12T22:19:43.788 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.788 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.788 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.788 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:43.788 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.788 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.788 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.789 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.789 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.789 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.789 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.789 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.789 INFO:teuthology.orchestra.run.smithi028.stdout: "leaderboard_description": { 2024-05-12T22:19:43.789 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "leaderboard_description", 2024-05-12T22:19:43.789 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.789 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.790 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.790 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:43.790 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.790 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.790 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.790 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.790 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.790 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.791 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.791 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.791 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-12T22:19:43.791 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-12T22:19:43.791 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.791 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.791 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.791 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:43.791 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.792 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.792 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:43.792 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:43.792 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:43.792 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:43.792 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:43.792 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:43.792 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:43.792 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:43.793 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.793 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.793 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.793 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.793 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.793 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-12T22:19:43.793 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-12T22:19:43.793 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.793 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.794 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.794 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:43.794 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.794 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.794 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.794 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.794 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.794 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.795 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.795 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.795 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-12T22:19:43.795 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-12T22:19:43.795 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.795 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.795 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.795 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-12T22:19:43.795 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.796 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.796 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:43.796 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:43.796 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:43.796 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:43.796 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:43.796 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:43.796 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:43.796 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:43.797 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.797 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.797 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.797 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.797 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.797 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-12T22:19:43.797 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-12T22:19:43.797 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.797 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.798 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.798 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:43.798 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.798 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.798 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.798 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.798 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.798 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.799 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.799 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.799 INFO:teuthology.orchestra.run.smithi028.stdout: "organization": { 2024-05-12T22:19:43.799 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "organization", 2024-05-12T22:19:43.799 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.799 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.799 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.799 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:43.799 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.800 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.800 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.800 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.800 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.800 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.800 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.800 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.800 INFO:teuthology.orchestra.run.smithi028.stdout: "proxy": { 2024-05-12T22:19:43.800 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "proxy", 2024-05-12T22:19:43.801 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.801 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.801 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.801 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:43.801 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.801 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.801 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.801 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.802 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.802 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.802 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.802 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.802 INFO:teuthology.orchestra.run.smithi028.stdout: "url": { 2024-05-12T22:19:43.802 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "url", 2024-05-12T22:19:43.802 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.802 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.803 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.803 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-05-12T22:19:43.803 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.803 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.803 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.803 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.803 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.803 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.803 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.804 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:43.804 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:43.804 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.804 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-12T22:19:43.804 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "test_orchestrator", 2024-05-12T22:19:43.804 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-12T22:19:43.804 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-12T22:19:43.804 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-12T22:19:43.804 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-12T22:19:43.805 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-12T22:19:43.805 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.805 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.805 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.805 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:43.805 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.805 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.805 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:43.805 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:43.806 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:43.806 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:43.806 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:43.806 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:43.806 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:43.806 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:43.806 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.806 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.806 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.807 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.807 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.807 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-12T22:19:43.807 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-12T22:19:43.807 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.807 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.807 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.807 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:43.808 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.808 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.808 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.808 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.808 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.808 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.808 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.808 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.808 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-12T22:19:43.809 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-12T22:19:43.809 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.809 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.809 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.809 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-12T22:19:43.809 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.809 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.809 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:43.809 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:43.810 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:43.810 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:43.810 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:43.810 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:43.810 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:43.810 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:43.810 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.810 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.810 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.811 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.811 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.811 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-12T22:19:43.811 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-12T22:19:43.811 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.811 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.811 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.811 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:43.811 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.812 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.812 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.812 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.812 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.812 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.812 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.812 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:43.812 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:43.812 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.813 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-12T22:19:43.813 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "volumes", 2024-05-12T22:19:43.813 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-12T22:19:43.813 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-12T22:19:43.813 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-12T22:19:43.813 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-12T22:19:43.813 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-12T22:19:43.813 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.813 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.814 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.814 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:43.814 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.814 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.814 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:43.814 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:43.814 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:43.814 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:43.814 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:43.815 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:43.815 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:43.815 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:43.815 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.815 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.815 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.815 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.815 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.815 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-12T22:19:43.816 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-12T22:19:43.816 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.816 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.816 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.816 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:43.816 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.816 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.816 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.816 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.817 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.817 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.817 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.817 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.817 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-12T22:19:43.817 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-12T22:19:43.817 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.817 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.817 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.818 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-12T22:19:43.818 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.818 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.818 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:43.818 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:43.818 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:43.818 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:43.818 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:43.818 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:43.819 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:43.819 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:43.819 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.819 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.819 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.819 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.819 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.819 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-12T22:19:43.819 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-12T22:19:43.820 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.820 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.820 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.820 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:43.820 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.820 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.820 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.820 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.820 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.821 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.821 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.821 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.821 INFO:teuthology.orchestra.run.smithi028.stdout: "max_concurrent_clones": { 2024-05-12T22:19:43.821 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_concurrent_clones", 2024-05-12T22:19:43.821 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-12T22:19:43.821 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.821 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.821 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "4", 2024-05-12T22:19:43.822 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.822 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.822 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.822 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Number of asynchronous cloner threads", 2024-05-12T22:19:43.822 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.822 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.822 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.822 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.822 INFO:teuthology.orchestra.run.smithi028.stdout: "periodic_async_work": { 2024-05-12T22:19:43.823 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "periodic_async_work", 2024-05-12T22:19:43.823 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.823 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.823 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.823 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:43.823 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.823 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.823 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.823 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Periodically check for async work", 2024-05-12T22:19:43.824 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.824 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.824 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.824 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.824 INFO:teuthology.orchestra.run.smithi028.stdout: "snapshot_clone_delay": { 2024-05-12T22:19:43.824 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "snapshot_clone_delay", 2024-05-12T22:19:43.824 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-12T22:19:43.824 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.824 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.825 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-05-12T22:19:43.825 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.825 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.825 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.825 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-05-12T22:19:43.825 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.825 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.825 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.826 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.826 INFO:teuthology.orchestra.run.smithi028.stdout: "snapshot_clone_no_wait": { 2024-05-12T22:19:43.826 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "snapshot_clone_no_wait", 2024-05-12T22:19:43.826 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.826 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.826 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.826 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-05-12T22:19:43.826 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.827 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.827 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.827 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-05-12T22:19:43.827 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.827 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.827 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.827 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:43.827 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:43.827 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.828 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-12T22:19:43.828 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix", 2024-05-12T22:19:43.828 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-05-12T22:19:43.828 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-05-12T22:19:43.828 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-05-12T22:19:43.828 INFO:teuthology.orchestra.run.smithi028.stdout: "discovery_interval": { 2024-05-12T22:19:43.828 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "discovery_interval", 2024-05-12T22:19:43.828 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-05-12T22:19:43.828 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.829 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.829 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "100", 2024-05-12T22:19:43.829 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.829 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.829 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.829 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.829 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.829 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.829 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.830 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.830 INFO:teuthology.orchestra.run.smithi028.stdout: "identifier": { 2024-05-12T22:19:43.830 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "identifier", 2024-05-12T22:19:43.830 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.830 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.830 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.830 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:43.830 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.830 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.831 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.831 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.831 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.831 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.831 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.831 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.831 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-05-12T22:19:43.831 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-05-12T22:19:43.831 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-05-12T22:19:43.832 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.832 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.832 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-05-12T22:19:43.832 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.832 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.832 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.832 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.832 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.833 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.833 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.833 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.833 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-05-12T22:19:43.833 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-05-12T22:19:43.833 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.833 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.833 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.833 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:43.834 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.834 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.834 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:43.834 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:43.834 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:43.834 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:43.834 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:43.834 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:43.834 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:43.835 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:43.835 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.835 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.835 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.835 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.835 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.835 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-05-12T22:19:43.835 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-05-12T22:19:43.835 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.836 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.836 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.836 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:43.836 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.836 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.836 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.836 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.836 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.836 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.837 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.837 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.837 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-05-12T22:19:43.837 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-05-12T22:19:43.837 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.837 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.837 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.837 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-05-12T22:19:43.837 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.838 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.838 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-05-12T22:19:43.838 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-05-12T22:19:43.838 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-05-12T22:19:43.838 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-05-12T22:19:43.838 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-05-12T22:19:43.838 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-05-12T22:19:43.838 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-05-12T22:19:43.838 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:43.839 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.839 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.839 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.839 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.839 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.839 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-05-12T22:19:43.839 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-05-12T22:19:43.839 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-05-12T22:19:43.839 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.840 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-05-12T22:19:43.840 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-05-12T22:19:43.840 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.840 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.840 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.840 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.840 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.840 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.840 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.841 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.841 INFO:teuthology.orchestra.run.smithi028.stdout: "zabbix_host": { 2024-05-12T22:19:43.841 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix_host", 2024-05-12T22:19:43.841 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.841 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.841 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.841 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-05-12T22:19:43.841 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.841 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.842 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.842 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.842 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.842 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.842 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.842 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.842 INFO:teuthology.orchestra.run.smithi028.stdout: "zabbix_port": { 2024-05-12T22:19:43.842 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix_port", 2024-05-12T22:19:43.842 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-05-12T22:19:43.843 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.843 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.843 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10051", 2024-05-12T22:19:43.843 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.843 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.843 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.843 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.843 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.843 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.844 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.844 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.844 INFO:teuthology.orchestra.run.smithi028.stdout: "zabbix_sender": { 2024-05-12T22:19:43.844 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix_sender", 2024-05-12T22:19:43.844 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-05-12T22:19:43.844 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-05-12T22:19:43.844 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-05-12T22:19:43.844 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-05-12T22:19:43.844 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-05-12T22:19:43.845 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-05-12T22:19:43.845 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-05-12T22:19:43.845 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-05-12T22:19:43.845 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-05-12T22:19:43.845 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-05-12T22:19:43.845 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-05-12T22:19:43.845 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:43.845 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:43.845 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:43.846 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:43.846 INFO:teuthology.orchestra.run.smithi028.stdout: "services": { 2024-05-12T22:19:43.846 INFO:teuthology.orchestra.run.smithi028.stdout: "dashboard": "https://172.21.15.28:8443/" 2024-05-12T22:19:43.846 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.846 INFO:teuthology.orchestra.run.smithi028.stdout: "always_on_modules": { 2024-05-12T22:19:43.846 INFO:teuthology.orchestra.run.smithi028.stdout: "octopus": [ 2024-05-12T22:19:43.846 INFO:teuthology.orchestra.run.smithi028.stdout: "balancer", 2024-05-12T22:19:43.846 INFO:teuthology.orchestra.run.smithi028.stdout: "crash", 2024-05-12T22:19:43.847 INFO:teuthology.orchestra.run.smithi028.stdout: "devicehealth", 2024-05-12T22:19:43.847 INFO:teuthology.orchestra.run.smithi028.stdout: "orchestrator", 2024-05-12T22:19:43.847 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscaler", 2024-05-12T22:19:43.847 INFO:teuthology.orchestra.run.smithi028.stdout: "progress", 2024-05-12T22:19:43.847 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_support", 2024-05-12T22:19:43.847 INFO:teuthology.orchestra.run.smithi028.stdout: "status", 2024-05-12T22:19:43.847 INFO:teuthology.orchestra.run.smithi028.stdout: "telemetry", 2024-05-12T22:19:43.847 INFO:teuthology.orchestra.run.smithi028.stdout: "volumes" 2024-05-12T22:19:43.847 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:43.848 INFO:teuthology.orchestra.run.smithi028.stdout: "pacific": [ 2024-05-12T22:19:43.848 INFO:teuthology.orchestra.run.smithi028.stdout: "balancer", 2024-05-12T22:19:43.848 INFO:teuthology.orchestra.run.smithi028.stdout: "crash", 2024-05-12T22:19:43.848 INFO:teuthology.orchestra.run.smithi028.stdout: "devicehealth", 2024-05-12T22:19:43.848 INFO:teuthology.orchestra.run.smithi028.stdout: "orchestrator", 2024-05-12T22:19:43.848 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscaler", 2024-05-12T22:19:43.848 INFO:teuthology.orchestra.run.smithi028.stdout: "progress", 2024-05-12T22:19:43.848 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_support", 2024-05-12T22:19:43.848 INFO:teuthology.orchestra.run.smithi028.stdout: "status", 2024-05-12T22:19:43.849 INFO:teuthology.orchestra.run.smithi028.stdout: "telemetry", 2024-05-12T22:19:43.849 INFO:teuthology.orchestra.run.smithi028.stdout: "volumes" 2024-05-12T22:19:43.849 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:43.849 INFO:teuthology.orchestra.run.smithi028.stdout: "quincy": [ 2024-05-12T22:19:43.849 INFO:teuthology.orchestra.run.smithi028.stdout: "balancer", 2024-05-12T22:19:43.849 INFO:teuthology.orchestra.run.smithi028.stdout: "crash", 2024-05-12T22:19:43.849 INFO:teuthology.orchestra.run.smithi028.stdout: "devicehealth", 2024-05-12T22:19:43.849 INFO:teuthology.orchestra.run.smithi028.stdout: "orchestrator", 2024-05-12T22:19:43.849 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscaler", 2024-05-12T22:19:43.849 INFO:teuthology.orchestra.run.smithi028.stdout: "progress", 2024-05-12T22:19:43.850 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_support", 2024-05-12T22:19:43.850 INFO:teuthology.orchestra.run.smithi028.stdout: "status", 2024-05-12T22:19:43.850 INFO:teuthology.orchestra.run.smithi028.stdout: "telemetry", 2024-05-12T22:19:43.850 INFO:teuthology.orchestra.run.smithi028.stdout: "volumes" 2024-05-12T22:19:43.850 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:43.850 INFO:teuthology.orchestra.run.smithi028.stdout: "reef": [ 2024-05-12T22:19:43.850 INFO:teuthology.orchestra.run.smithi028.stdout: "balancer", 2024-05-12T22:19:43.850 INFO:teuthology.orchestra.run.smithi028.stdout: "crash", 2024-05-12T22:19:43.851 INFO:teuthology.orchestra.run.smithi028.stdout: "devicehealth", 2024-05-12T22:19:43.851 INFO:teuthology.orchestra.run.smithi028.stdout: "orchestrator", 2024-05-12T22:19:43.851 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscaler", 2024-05-12T22:19:43.851 INFO:teuthology.orchestra.run.smithi028.stdout: "progress", 2024-05-12T22:19:43.851 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_support", 2024-05-12T22:19:43.851 INFO:teuthology.orchestra.run.smithi028.stdout: "status", 2024-05-12T22:19:43.851 INFO:teuthology.orchestra.run.smithi028.stdout: "telemetry", 2024-05-12T22:19:43.851 INFO:teuthology.orchestra.run.smithi028.stdout: "volumes" 2024-05-12T22:19:43.851 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-05-12T22:19:43.852 INFO:teuthology.orchestra.run.smithi028.stdout: "squid": [ 2024-05-12T22:19:43.852 INFO:teuthology.orchestra.run.smithi028.stdout: "balancer", 2024-05-12T22:19:43.852 INFO:teuthology.orchestra.run.smithi028.stdout: "crash", 2024-05-12T22:19:43.852 INFO:teuthology.orchestra.run.smithi028.stdout: "devicehealth", 2024-05-12T22:19:43.852 INFO:teuthology.orchestra.run.smithi028.stdout: "orchestrator", 2024-05-12T22:19:43.852 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscaler", 2024-05-12T22:19:43.852 INFO:teuthology.orchestra.run.smithi028.stdout: "progress", 2024-05-12T22:19:43.852 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_support", 2024-05-12T22:19:43.852 INFO:teuthology.orchestra.run.smithi028.stdout: "status", 2024-05-12T22:19:43.853 INFO:teuthology.orchestra.run.smithi028.stdout: "telemetry", 2024-05-12T22:19:43.853 INFO:teuthology.orchestra.run.smithi028.stdout: "volumes" 2024-05-12T22:19:43.853 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-05-12T22:19:43.853 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.853 INFO:teuthology.orchestra.run.smithi028.stdout: "last_failure_osd_epoch": 89, 2024-05-12T22:19:43.853 INFO:teuthology.orchestra.run.smithi028.stdout: "active_clients": [ 2024-05-12T22:19:43.853 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-12T22:19:43.853 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "devicehealth", 2024-05-12T22:19:43.853 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-05-12T22:19:43.853 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-12T22:19:43.854 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-05-12T22:19:43.854 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:0", 2024-05-12T22:19:43.854 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 987264123 2024-05-12T22:19:43.854 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:43.854 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-05-12T22:19:43.854 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.854 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-12T22:19:43.854 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "libcephsqlite", 2024-05-12T22:19:43.855 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-05-12T22:19:43.855 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-12T22:19:43.855 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-05-12T22:19:43.855 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:0", 2024-05-12T22:19:43.855 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 4060636700 2024-05-12T22:19:43.855 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:43.855 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-05-12T22:19:43.855 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.855 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-12T22:19:43.855 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rbd_support", 2024-05-12T22:19:43.856 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-05-12T22:19:43.856 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-12T22:19:43.856 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-05-12T22:19:43.856 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:0", 2024-05-12T22:19:43.856 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 4144955181 2024-05-12T22:19:43.856 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:43.856 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-05-12T22:19:43.856 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.856 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-12T22:19:43.857 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "snap_schedule", 2024-05-12T22:19:43.857 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-05-12T22:19:43.857 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-12T22:19:43.857 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-05-12T22:19:43.857 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:0", 2024-05-12T22:19:43.857 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 4128163034 2024-05-12T22:19:43.857 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:43.857 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-05-12T22:19:43.857 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.858 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-12T22:19:43.858 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "snap_schedule", 2024-05-12T22:19:43.858 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-05-12T22:19:43.858 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-12T22:19:43.858 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-05-12T22:19:43.858 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:0", 2024-05-12T22:19:43.858 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3313133783 2024-05-12T22:19:43.858 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:43.859 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-05-12T22:19:43.859 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:43.859 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-12T22:19:43.859 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "volumes", 2024-05-12T22:19:43.859 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-05-12T22:19:43.859 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-05-12T22:19:43.859 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-05-12T22:19:43.859 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:0", 2024-05-12T22:19:43.859 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3244026428 2024-05-12T22:19:43.860 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:43.860 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-05-12T22:19:43.860 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:43.860 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-05-12T22:19:43.860 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-05-12T22:19:43.861 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:19:42 smithi186 ceph-mon[34785]: pgmap v332: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-12T22:19:43.862 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:19:42 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/3604000295' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-05-12T22:19:43.862 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:19:41 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/2057342854' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-12T22:19:43.862 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:19:41 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/1994700608' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-12T22:19:43.862 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:19:41 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/3853316862' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-12T22:19:43.862 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:19:42 smithi086 ceph-mon[34297]: pgmap v332: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-12T22:19:43.863 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:19:42 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/3604000295' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-05-12T22:19:43.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:41.714+0000 7fdd0d600640 1 -- 172.21.15.28:0/3604000295 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fdcec0a2d00 msgr2=0x7fdcec0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:43.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:41.714+0000 7fdd0d600640 1 --2- 172.21.15.28:0/3604000295 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fdcec0a2d00 0x7fdcec0a51c0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fdd08101c10 tx=0x7fdcf0009260 comp rx=0 tx=0).stop 2024-05-12T22:19:43.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:41.714+0000 7fdd0d600640 1 -- 172.21.15.28:0/3604000295 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdd08103a00 msgr2=0x7fdd0819a310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:19:43.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:41.714+0000 7fdd0d600640 1 --2- 172.21.15.28:0/3604000295 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdd08103a00 0x7fdd0819a310 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fdcf802acb0 tx=0x7fdcf8030c30 comp rx=0 tx=0).stop 2024-05-12T22:19:43.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:41.714+0000 7fdd0d600640 1 -- 172.21.15.28:0/3604000295 shutdown_connections 2024-05-12T22:19:43.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:41.714+0000 7fdd0d600640 1 --2- 172.21.15.28:0/3604000295 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fdcec0a2d00 0x7fdcec0a51c0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:43.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:41.714+0000 7fdd0d600640 1 --2- 172.21.15.28:0/3604000295 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd08109a10 0x7fdd0819ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:43.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:41.714+0000 7fdd0d600640 1 --2- 172.21.15.28:0/3604000295 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdd081043c0 0x7fdd0819a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:43.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:41.715+0000 7fdd0d600640 1 --2- 172.21.15.28:0/3604000295 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdd08103a00 0x7fdd0819a310 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:19:43.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:41.715+0000 7fdd0d600640 1 -- 172.21.15.28:0/3604000295 >> 172.21.15.28:0/3604000295 conn(0x7fdd080ff530 msgr2=0x7fdd080ffd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:19:43.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:41.715+0000 7fdd0d600640 1 -- 172.21.15.28:0/3604000295 shutdown_connections 2024-05-12T22:19:43.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:19:41.715+0000 7fdd0d600640 1 -- 172.21.15.28:0/3604000295 wait complete. 2024-05-12T22:19:43.872 DEBUG:tasks.check_counter:Getting stats from a 2024-05-12T22:19:43.873 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f72fecff68e1d400c4568684327c900485c20d6a shell --fsid 6b92e7b4-10aa-11ef-bc98-c7b262605968 -- ceph daemon mds.a perf dump 2024-05-12T22:19:43.873 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:19:42 smithi028 ceph-mon[29847]: pgmap v332: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-12T22:19:43.873 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:19:42 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/3604000295' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-05-12T22:19:44.346 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/mon.a/config 2024-05-12T22:19:44.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:19:44 smithi028 ceph-mon[29847]: pgmap v333: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:19:44.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:19:44 smithi186 ceph-mon[34785]: pgmap v333: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:19:44.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:19:44 smithi086 ceph-mon[34297]: pgmap v333: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:19:46.393 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:19:46 smithi028 ceph-mon[29847]: pgmap v334: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-12T22:19:46.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:19:46 smithi186 ceph-mon[34785]: pgmap v334: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-12T22:19:46.722 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-05-12T22:19:46.723 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-0": { 2024-05-12T22:19:46.723 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 145, 2024-05-12T22:19:46.723 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 177, 2024-05-12T22:19:46.723 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 35971, 2024-05-12T22:19:46.723 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 568020, 2024-05-12T22:19:46.723 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 51, 2024-05-12T22:19:46.723 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 7, 2024-05-12T22:19:46.723 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.073059026, 2024-05-12T22:19:46.724 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.016165122, 2024-05-12T22:19:46.724 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.034975274, 2024-05-12T22:19:46.724 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.004281620, 2024-05-12T22:19:46.724 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-05-12T22:19:46.724 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 174, 2024-05-12T22:19:46.724 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.055375422, 2024-05-12T22:19:46.724 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000318249 2024-05-12T22:19:46.724 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:46.725 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-05-12T22:19:46.725 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 109, 2024-05-12T22:19:46.725 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000079967, 2024-05-12T22:19:46.725 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000733 2024-05-12T22:19:46.725 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:46.725 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 4992, 2024-05-12T22:19:46.725 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 12464 2024-05-12T22:19:46.725 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:46.725 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-1": { 2024-05-12T22:19:46.726 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 1475, 2024-05-12T22:19:46.726 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 1278, 2024-05-12T22:19:46.726 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 471215, 2024-05-12T22:19:46.726 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 921549, 2024-05-12T22:19:46.726 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 11, 2024-05-12T22:19:46.726 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 6, 2024-05-12T22:19:46.726 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.301804001, 2024-05-12T22:19:46.726 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.142232509, 2024-05-12T22:19:46.726 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.190569476, 2024-05-12T22:19:46.727 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.010329866, 2024-05-12T22:19:46.727 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-05-12T22:19:46.727 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 1277, 2024-05-12T22:19:46.727 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.131272220, 2024-05-12T22:19:46.727 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000102797 2024-05-12T22:19:46.727 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:46.727 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-05-12T22:19:46.727 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 2403, 2024-05-12T22:19:46.727 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.001531996, 2024-05-12T22:19:46.728 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000637 2024-05-12T22:19:46.728 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:46.728 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 3392, 2024-05-12T22:19:46.728 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 456208 2024-05-12T22:19:46.728 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:46.728 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-2": { 2024-05-12T22:19:46.728 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 491, 2024-05-12T22:19:46.728 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 511, 2024-05-12T22:19:46.729 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 1823193, 2024-05-12T22:19:46.729 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 251227, 2024-05-12T22:19:46.729 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 10, 2024-05-12T22:19:46.729 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 6, 2024-05-12T22:19:46.729 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.201613799, 2024-05-12T22:19:46.729 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.081680896, 2024-05-12T22:19:46.729 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.142027760, 2024-05-12T22:19:46.729 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.013798046, 2024-05-12T22:19:46.729 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-05-12T22:19:46.730 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 510, 2024-05-12T22:19:46.730 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.092026334, 2024-05-12T22:19:46.730 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000180443 2024-05-12T22:19:46.730 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:46.730 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-05-12T22:19:46.730 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 138, 2024-05-12T22:19:46.730 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000139750, 2024-05-12T22:19:46.730 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000001012 2024-05-12T22:19:46.730 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:46.731 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 1776160, 2024-05-12T22:19:46.731 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 165776 2024-05-12T22:19:46.731 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:46.731 INFO:teuthology.orchestra.run.smithi028.stdout: "cct": { 2024-05-12T22:19:46.731 INFO:teuthology.orchestra.run.smithi028.stdout: "total_workers": 1, 2024-05-12T22:19:46.731 INFO:teuthology.orchestra.run.smithi028.stdout: "unhealthy_workers": 0 2024-05-12T22:19:46.731 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:46.731 INFO:teuthology.orchestra.run.smithi028.stdout: "finisher-MDSRank": { 2024-05-12T22:19:46.731 INFO:teuthology.orchestra.run.smithi028.stdout: "queue_len": 0, 2024-05-12T22:19:46.732 INFO:teuthology.orchestra.run.smithi028.stdout: "complete_latency": { 2024-05-12T22:19:46.732 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 207, 2024-05-12T22:19:46.732 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.030105044, 2024-05-12T22:19:46.732 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000145434 2024-05-12T22:19:46.732 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:46.732 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:46.732 INFO:teuthology.orchestra.run.smithi028.stdout: "finisher-PurgeQueue": { 2024-05-12T22:19:46.732 INFO:teuthology.orchestra.run.smithi028.stdout: "queue_len": 0, 2024-05-12T22:19:46.733 INFO:teuthology.orchestra.run.smithi028.stdout: "complete_latency": { 2024-05-12T22:19:46.733 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 17, 2024-05-12T22:19:46.733 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000967372, 2024-05-12T22:19:46.733 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000056904 2024-05-12T22:19:46.733 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:46.733 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:46.733 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": { 2024-05-12T22:19:46.733 INFO:teuthology.orchestra.run.smithi028.stdout: "request": 91, 2024-05-12T22:19:46.733 INFO:teuthology.orchestra.run.smithi028.stdout: "reply": 17, 2024-05-12T22:19:46.734 INFO:teuthology.orchestra.run.smithi028.stdout: "reply_latency": { 2024-05-12T22:19:46.734 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 17, 2024-05-12T22:19:46.734 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.345169527, 2024-05-12T22:19:46.734 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.020304089 2024-05-12T22:19:46.734 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:46.734 INFO:teuthology.orchestra.run.smithi028.stdout: "slow_reply": 0, 2024-05-12T22:19:46.734 INFO:teuthology.orchestra.run.smithi028.stdout: "forward": 74, 2024-05-12T22:19:46.734 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_fetch_complete": 0, 2024-05-12T22:19:46.734 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_fetch_keys": 0, 2024-05-12T22:19:46.735 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_commit": 11, 2024-05-12T22:19:46.735 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_split": 0, 2024-05-12T22:19:46.735 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_merge": 0, 2024-05-12T22:19:46.735 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes": 55, 2024-05-12T22:19:46.735 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_top": 10, 2024-05-12T22:19:46.735 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_bottom": 45, 2024-05-12T22:19:46.735 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_pin_tail": 0, 2024-05-12T22:19:46.736 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_pinned": 39, 2024-05-12T22:19:46.736 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_expired": 8, 2024-05-12T22:19:46.736 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_with_caps": 6, 2024-05-12T22:19:46.736 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": 6, 2024-05-12T22:19:46.736 INFO:teuthology.orchestra.run.smithi028.stdout: "subtrees": 6, 2024-05-12T22:19:46.736 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse": 156, 2024-05-12T22:19:46.736 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_hit": 48, 2024-05-12T22:19:46.736 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_forward": 72, 2024-05-12T22:19:46.736 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_discover": 20, 2024-05-12T22:19:46.737 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_dir_fetch": 0, 2024-05-12T22:19:46.737 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_remote_ino": 0, 2024-05-12T22:19:46.737 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_lock": 0, 2024-05-12T22:19:46.737 INFO:teuthology.orchestra.run.smithi028.stdout: "load_cent": 0, 2024-05-12T22:19:46.737 INFO:teuthology.orchestra.run.smithi028.stdout: "q": 0, 2024-05-12T22:19:46.737 INFO:teuthology.orchestra.run.smithi028.stdout: "exported": 0, 2024-05-12T22:19:46.737 INFO:teuthology.orchestra.run.smithi028.stdout: "exported_inodes": 0, 2024-05-12T22:19:46.737 INFO:teuthology.orchestra.run.smithi028.stdout: "imported": 3, 2024-05-12T22:19:46.737 INFO:teuthology.orchestra.run.smithi028.stdout: "imported_inodes": 10, 2024-05-12T22:19:46.738 INFO:teuthology.orchestra.run.smithi028.stdout: "openino_dir_fetch": 0, 2024-05-12T22:19:46.738 INFO:teuthology.orchestra.run.smithi028.stdout: "openino_backtrace_fetch": 123, 2024-05-12T22:19:46.738 INFO:teuthology.orchestra.run.smithi028.stdout: "openino_peer_discover": 0, 2024-05-12T22:19:46.738 INFO:teuthology.orchestra.run.smithi028.stdout: "root_rfiles": 3454, 2024-05-12T22:19:46.738 INFO:teuthology.orchestra.run.smithi028.stdout: "root_rbytes": 26743557, 2024-05-12T22:19:46.738 INFO:teuthology.orchestra.run.smithi028.stdout: "root_rsnaps": 6, 2024-05-12T22:19:46.738 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_backtrace_fetch": 123, 2024-05-12T22:19:46.738 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_set_tag": 0, 2024-05-12T22:19:46.738 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_backtrace_repaired": 0, 2024-05-12T22:19:46.739 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_inotable_repaired": 0, 2024-05-12T22:19:46.739 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_dir_inodes": 0, 2024-05-12T22:19:46.739 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_dir_base_inodes": 0, 2024-05-12T22:19:46.739 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_dirfrag_rstats": 0, 2024-05-12T22:19:46.739 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_file_inodes": 123, 2024-05-12T22:19:46.739 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_inode_file_caps": 0, 2024-05-12T22:19:46.739 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_cap_op_revoke": 0, 2024-05-12T22:19:46.739 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_cap_op_grant": 5, 2024-05-12T22:19:46.739 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_cap_op_trunc": 0, 2024-05-12T22:19:46.740 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-05-12T22:19:46.740 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_cap_op_flush_ack": 0, 2024-05-12T22:19:46.740 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_client_caps": 13, 2024-05-12T22:19:46.740 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_client_caps_dirty": 13, 2024-05-12T22:19:46.740 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_client_cap_release": 4, 2024-05-12T22:19:46.740 INFO:teuthology.orchestra.run.smithi028.stdout: "process_request_cap_release": 5 2024-05-12T22:19:46.740 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:46.740 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_cache": { 2024-05-12T22:19:46.740 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_update": 2, 2024-05-12T22:19:46.741 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_update_receipt": 67, 2024-05-12T22:19:46.741 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_try_discover": 42, 2024-05-12T22:19:46.741 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_send_discover": 46, 2024-05-12T22:19:46.741 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_handle_discover": 2, 2024-05-12T22:19:46.741 INFO:teuthology.orchestra.run.smithi028.stdout: "num_strays": 0, 2024-05-12T22:19:46.741 INFO:teuthology.orchestra.run.smithi028.stdout: "num_strays_delayed": 0, 2024-05-12T22:19:46.741 INFO:teuthology.orchestra.run.smithi028.stdout: "num_strays_enqueuing": 0, 2024-05-12T22:19:46.741 INFO:teuthology.orchestra.run.smithi028.stdout: "strays_created": 2, 2024-05-12T22:19:46.741 INFO:teuthology.orchestra.run.smithi028.stdout: "strays_enqueued": 2, 2024-05-12T22:19:46.742 INFO:teuthology.orchestra.run.smithi028.stdout: "strays_reintegrated": 0, 2024-05-12T22:19:46.742 INFO:teuthology.orchestra.run.smithi028.stdout: "strays_migrated": 0, 2024-05-12T22:19:46.742 INFO:teuthology.orchestra.run.smithi028.stdout: "num_recovering_processing": 0, 2024-05-12T22:19:46.742 INFO:teuthology.orchestra.run.smithi028.stdout: "num_recovering_enqueued": 0, 2024-05-12T22:19:46.742 INFO:teuthology.orchestra.run.smithi028.stdout: "num_recovering_prioritized": 0, 2024-05-12T22:19:46.742 INFO:teuthology.orchestra.run.smithi028.stdout: "recovery_started": 0, 2024-05-12T22:19:46.742 INFO:teuthology.orchestra.run.smithi028.stdout: "recovery_completed": 0, 2024-05-12T22:19:46.742 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_quiesce_path": 0, 2024-05-12T22:19:46.743 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_quiesce_inode": 0, 2024-05-12T22:19:46.743 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_enqueue_scrub": 0, 2024-05-12T22:19:46.743 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_exportdir": 0, 2024-05-12T22:19:46.743 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_flush": 0, 2024-05-12T22:19:46.743 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_fragmentdir": 0, 2024-05-12T22:19:46.743 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_fragstats": 0, 2024-05-12T22:19:46.743 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_inodestats": 0 2024-05-12T22:19:46.743 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:46.743 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_log": { 2024-05-12T22:19:46.744 INFO:teuthology.orchestra.run.smithi028.stdout: "evlrg": 0, 2024-05-12T22:19:46.744 INFO:teuthology.orchestra.run.smithi028.stdout: "evadd": 45, 2024-05-12T22:19:46.744 INFO:teuthology.orchestra.run.smithi028.stdout: "evex": 0, 2024-05-12T22:19:46.744 INFO:teuthology.orchestra.run.smithi028.stdout: "evtrm": 0, 2024-05-12T22:19:46.744 INFO:teuthology.orchestra.run.smithi028.stdout: "ev": 45, 2024-05-12T22:19:46.744 INFO:teuthology.orchestra.run.smithi028.stdout: "evexg": 0, 2024-05-12T22:19:46.744 INFO:teuthology.orchestra.run.smithi028.stdout: "evexd": 0, 2024-05-12T22:19:46.745 INFO:teuthology.orchestra.run.smithi028.stdout: "segadd": 2, 2024-05-12T22:19:46.745 INFO:teuthology.orchestra.run.smithi028.stdout: "segex": 0, 2024-05-12T22:19:46.745 INFO:teuthology.orchestra.run.smithi028.stdout: "segtrm": 0, 2024-05-12T22:19:46.745 INFO:teuthology.orchestra.run.smithi028.stdout: "seg": 2, 2024-05-12T22:19:46.745 INFO:teuthology.orchestra.run.smithi028.stdout: "segmjr": 2, 2024-05-12T22:19:46.745 INFO:teuthology.orchestra.run.smithi028.stdout: "segexg": 0, 2024-05-12T22:19:46.745 INFO:teuthology.orchestra.run.smithi028.stdout: "segexd": 0, 2024-05-12T22:19:46.745 INFO:teuthology.orchestra.run.smithi028.stdout: "expos": 4194304, 2024-05-12T22:19:46.745 INFO:teuthology.orchestra.run.smithi028.stdout: "wrpos": 4268093, 2024-05-12T22:19:46.746 INFO:teuthology.orchestra.run.smithi028.stdout: "rdpos": 0, 2024-05-12T22:19:46.746 INFO:teuthology.orchestra.run.smithi028.stdout: "jlat": { 2024-05-12T22:19:46.746 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 24, 2024-05-12T22:19:46.746 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.144622668, 2024-05-12T22:19:46.746 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.006025944 2024-05-12T22:19:46.746 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:46.746 INFO:teuthology.orchestra.run.smithi028.stdout: "replayed": 0 2024-05-12T22:19:46.746 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:46.746 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_mem": { 2024-05-12T22:19:46.747 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 56, 2024-05-12T22:19:46.747 INFO:teuthology.orchestra.run.smithi028.stdout: "ino+": 63, 2024-05-12T22:19:46.747 INFO:teuthology.orchestra.run.smithi028.stdout: "ino-": 7, 2024-05-12T22:19:46.747 INFO:teuthology.orchestra.run.smithi028.stdout: "dir": 51, 2024-05-12T22:19:46.747 INFO:teuthology.orchestra.run.smithi028.stdout: "dir+": 56, 2024-05-12T22:19:46.747 INFO:teuthology.orchestra.run.smithi028.stdout: "dir-": 5, 2024-05-12T22:19:46.747 INFO:teuthology.orchestra.run.smithi028.stdout: "dn": 55, 2024-05-12T22:19:46.747 INFO:teuthology.orchestra.run.smithi028.stdout: "dn+": 65, 2024-05-12T22:19:46.747 INFO:teuthology.orchestra.run.smithi028.stdout: "dn-": 10, 2024-05-12T22:19:46.748 INFO:teuthology.orchestra.run.smithi028.stdout: "cap": 6, 2024-05-12T22:19:46.748 INFO:teuthology.orchestra.run.smithi028.stdout: "cap+": 13, 2024-05-12T22:19:46.748 INFO:teuthology.orchestra.run.smithi028.stdout: "cap-": 7, 2024-05-12T22:19:46.748 INFO:teuthology.orchestra.run.smithi028.stdout: "rss": 52984, 2024-05-12T22:19:46.748 INFO:teuthology.orchestra.run.smithi028.stdout: "heap": 223612 2024-05-12T22:19:46.748 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:46.748 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_server": { 2024-05-12T22:19:46.748 INFO:teuthology.orchestra.run.smithi028.stdout: "dispatch_client_request": 100, 2024-05-12T22:19:46.748 INFO:teuthology.orchestra.run.smithi028.stdout: "dispatch_server_request": 0, 2024-05-12T22:19:46.749 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_client_request": 91, 2024-05-12T22:19:46.749 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_client_session": 28, 2024-05-12T22:19:46.749 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_peer_request": 5, 2024-05-12T22:19:46.749 INFO:teuthology.orchestra.run.smithi028.stdout: "req_create_latency": { 2024-05-12T22:19:46.749 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 3, 2024-05-12T22:19:46.749 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.007780237, 2024-05-12T22:19:46.749 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.002593412 2024-05-12T22:19:46.749 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:46.749 INFO:teuthology.orchestra.run.smithi028.stdout: "req_getattr_latency": { 2024-05-12T22:19:46.750 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 3, 2024-05-12T22:19:46.750 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000782431, 2024-05-12T22:19:46.750 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000260810 2024-05-12T22:19:46.750 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:46.750 INFO:teuthology.orchestra.run.smithi028.stdout: "req_getfilelock_latency": { 2024-05-12T22:19:46.750 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-12T22:19:46.750 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-12T22:19:46.750 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-12T22:19:46.750 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:46.751 INFO:teuthology.orchestra.run.smithi028.stdout: "req_link_latency": { 2024-05-12T22:19:46.751 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-12T22:19:46.751 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-12T22:19:46.751 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-12T22:19:46.751 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:46.751 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookup_latency": { 2024-05-12T22:19:46.751 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 9, 2024-05-12T22:19:46.751 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.006010285, 2024-05-12T22:19:46.751 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000667809 2024-05-12T22:19:46.752 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:46.752 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookuphash_latency": { 2024-05-12T22:19:46.752 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-12T22:19:46.752 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-12T22:19:46.752 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-12T22:19:46.752 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:46.752 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookupino_latency": { 2024-05-12T22:19:46.752 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-12T22:19:46.752 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-12T22:19:46.753 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-12T22:19:46.753 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:46.753 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookupname_latency": { 2024-05-12T22:19:46.753 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-12T22:19:46.753 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-12T22:19:46.753 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-12T22:19:46.753 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:46.753 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookupparent_latency": { 2024-05-12T22:19:46.753 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-12T22:19:46.754 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-12T22:19:46.754 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-12T22:19:46.754 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:46.754 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookupsnap_latency": { 2024-05-12T22:19:46.754 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-12T22:19:46.754 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-12T22:19:46.754 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-12T22:19:46.754 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:46.754 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lssnap_latency": { 2024-05-12T22:19:46.755 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-12T22:19:46.755 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-12T22:19:46.755 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-12T22:19:46.755 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:46.756 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:19:46 smithi086 ceph-mon[34297]: pgmap v334: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-12T22:19:46.756 INFO:teuthology.orchestra.run.smithi028.stdout: "req_mkdir_latency": { 2024-05-12T22:19:46.756 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-12T22:19:46.756 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-12T22:19:46.756 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-12T22:19:46.756 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:46.757 INFO:teuthology.orchestra.run.smithi028.stdout: "req_mknod_latency": { 2024-05-12T22:19:46.757 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-12T22:19:46.757 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-12T22:19:46.757 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-12T22:19:46.757 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:46.757 INFO:teuthology.orchestra.run.smithi028.stdout: "req_mksnap_latency": { 2024-05-12T22:19:46.757 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-12T22:19:46.757 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-12T22:19:46.757 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-12T22:19:46.758 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:46.758 INFO:teuthology.orchestra.run.smithi028.stdout: "req_open_latency": { 2024-05-12T22:19:46.758 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-12T22:19:46.758 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-12T22:19:46.758 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-12T22:19:46.758 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:46.758 INFO:teuthology.orchestra.run.smithi028.stdout: "req_readdir_latency": { 2024-05-12T22:19:46.758 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-12T22:19:46.759 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-12T22:19:46.759 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-12T22:19:46.759 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:46.759 INFO:teuthology.orchestra.run.smithi028.stdout: "req_rename_latency": { 2024-05-12T22:19:46.759 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-12T22:19:46.759 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-12T22:19:46.759 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-12T22:19:46.759 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:46.759 INFO:teuthology.orchestra.run.smithi028.stdout: "req_renamesnap_latency": { 2024-05-12T22:19:46.760 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-12T22:19:46.760 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-12T22:19:46.760 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-12T22:19:46.760 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:46.760 INFO:teuthology.orchestra.run.smithi028.stdout: "req_snapdiff_latency": { 2024-05-12T22:19:46.760 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-12T22:19:46.760 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-12T22:19:46.760 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-12T22:19:46.760 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:46.761 INFO:teuthology.orchestra.run.smithi028.stdout: "req_rmdir_latency": { 2024-05-12T22:19:46.761 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-12T22:19:46.761 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-12T22:19:46.761 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-12T22:19:46.761 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:46.761 INFO:teuthology.orchestra.run.smithi028.stdout: "req_rmsnap_latency": { 2024-05-12T22:19:46.761 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-12T22:19:46.761 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-12T22:19:46.761 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-12T22:19:46.762 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:46.762 INFO:teuthology.orchestra.run.smithi028.stdout: "req_rmxattr_latency": { 2024-05-12T22:19:46.762 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-12T22:19:46.762 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-12T22:19:46.762 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-12T22:19:46.762 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:46.762 INFO:teuthology.orchestra.run.smithi028.stdout: "req_setattr_latency": { 2024-05-12T22:19:46.762 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-12T22:19:46.762 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-12T22:19:46.763 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-12T22:19:46.763 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:46.763 INFO:teuthology.orchestra.run.smithi028.stdout: "req_setdirlayout_latency": { 2024-05-12T22:19:46.763 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-12T22:19:46.763 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-12T22:19:46.763 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-12T22:19:46.763 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:46.763 INFO:teuthology.orchestra.run.smithi028.stdout: "req_setfilelock_latency": { 2024-05-12T22:19:46.763 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-12T22:19:46.764 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-12T22:19:46.764 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-12T22:19:46.764 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:46.764 INFO:teuthology.orchestra.run.smithi028.stdout: "req_setlayout_latency": { 2024-05-12T22:19:46.764 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-12T22:19:46.764 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-12T22:19:46.764 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-12T22:19:46.764 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:46.764 INFO:teuthology.orchestra.run.smithi028.stdout: "req_setxattr_latency": { 2024-05-12T22:19:46.765 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-12T22:19:46.765 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-12T22:19:46.765 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-12T22:19:46.765 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:46.765 INFO:teuthology.orchestra.run.smithi028.stdout: "req_symlink_latency": { 2024-05-12T22:19:46.765 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-12T22:19:46.765 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-12T22:19:46.765 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-12T22:19:46.765 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:46.766 INFO:teuthology.orchestra.run.smithi028.stdout: "req_unlink_latency": { 2024-05-12T22:19:46.766 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 2, 2024-05-12T22:19:46.766 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.330596574, 2024-05-12T22:19:46.766 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.165298287 2024-05-12T22:19:46.766 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:46.766 INFO:teuthology.orchestra.run.smithi028.stdout: "cap_revoke_eviction": 0, 2024-05-12T22:19:46.767 INFO:teuthology.orchestra.run.smithi028.stdout: "cap_acquisition_throttle": 0, 2024-05-12T22:19:46.767 INFO:teuthology.orchestra.run.smithi028.stdout: "req_getvxattr_latency": { 2024-05-12T22:19:46.767 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-12T22:19:46.767 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-12T22:19:46.767 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-12T22:19:46.767 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:46.767 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:46.767 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_sessions": { 2024-05-12T22:19:46.767 INFO:teuthology.orchestra.run.smithi028.stdout: "session_count": 2, 2024-05-12T22:19:46.768 INFO:teuthology.orchestra.run.smithi028.stdout: "session_add": 5, 2024-05-12T22:19:46.768 INFO:teuthology.orchestra.run.smithi028.stdout: "session_remove": 3, 2024-05-12T22:19:46.768 INFO:teuthology.orchestra.run.smithi028.stdout: "sessions_open": 2, 2024-05-12T22:19:46.768 INFO:teuthology.orchestra.run.smithi028.stdout: "sessions_stale": 0, 2024-05-12T22:19:46.768 INFO:teuthology.orchestra.run.smithi028.stdout: "total_load": 14, 2024-05-12T22:19:46.768 INFO:teuthology.orchestra.run.smithi028.stdout: "average_load": 7, 2024-05-12T22:19:46.768 INFO:teuthology.orchestra.run.smithi028.stdout: "avg_session_uptime": 567, 2024-05-12T22:19:46.768 INFO:teuthology.orchestra.run.smithi028.stdout: "mdthresh_evicted": 0 2024-05-12T22:19:46.768 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:46.769 INFO:teuthology.orchestra.run.smithi028.stdout: "mempool": { 2024-05-12T22:19:46.769 INFO:teuthology.orchestra.run.smithi028.stdout: "bloom_filter_bytes": 0, 2024-05-12T22:19:46.769 INFO:teuthology.orchestra.run.smithi028.stdout: "bloom_filter_items": 0, 2024-05-12T22:19:46.769 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_alloc_bytes": 0, 2024-05-12T22:19:46.769 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_alloc_items": 0, 2024-05-12T22:19:46.769 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_data_bytes": 0, 2024-05-12T22:19:46.769 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_data_items": 0, 2024-05-12T22:19:46.769 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_onode_bytes": 0, 2024-05-12T22:19:46.769 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_onode_items": 0, 2024-05-12T22:19:46.770 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_meta_bytes": 0, 2024-05-12T22:19:46.770 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_meta_items": 0, 2024-05-12T22:19:46.770 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_other_bytes": 0, 2024-05-12T22:19:46.770 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_other_items": 0, 2024-05-12T22:19:46.770 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_buffer_bytes": 0, 2024-05-12T22:19:46.770 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_buffer_items": 0, 2024-05-12T22:19:46.770 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_extent_bytes": 0, 2024-05-12T22:19:46.770 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_extent_items": 0, 2024-05-12T22:19:46.771 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_blob_bytes": 0, 2024-05-12T22:19:46.771 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_blob_items": 0, 2024-05-12T22:19:46.771 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_shared_blob_bytes": 0, 2024-05-12T22:19:46.771 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_shared_blob_items": 0, 2024-05-12T22:19:46.771 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_inline_bl_bytes": 0, 2024-05-12T22:19:46.771 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_inline_bl_items": 0, 2024-05-12T22:19:46.771 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_bytes": 0, 2024-05-12T22:19:46.771 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_items": 0, 2024-05-12T22:19:46.771 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_txc_bytes": 0, 2024-05-12T22:19:46.771 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_txc_items": 0, 2024-05-12T22:19:46.772 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_deferred_bytes": 0, 2024-05-12T22:19:46.772 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_deferred_items": 0, 2024-05-12T22:19:46.772 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_bytes": 0, 2024-05-12T22:19:46.772 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_items": 0, 2024-05-12T22:19:46.772 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_bytes": 0, 2024-05-12T22:19:46.772 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_items": 0, 2024-05-12T22:19:46.772 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_reader_bytes": 0, 2024-05-12T22:19:46.772 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_reader_items": 0, 2024-05-12T22:19:46.772 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_writer_bytes": 0, 2024-05-12T22:19:46.773 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_writer_items": 0, 2024-05-12T22:19:46.773 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_anon_bytes": 294046, 2024-05-12T22:19:46.773 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_anon_items": 170, 2024-05-12T22:19:46.773 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_meta_bytes": 264, 2024-05-12T22:19:46.773 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_meta_items": 3, 2024-05-12T22:19:46.773 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_bytes": 0, 2024-05-12T22:19:46.773 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_items": 0, 2024-05-12T22:19:46.773 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mapbl_bytes": 0, 2024-05-12T22:19:46.773 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mapbl_items": 0, 2024-05-12T22:19:46.774 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pglog_bytes": 0, 2024-05-12T22:19:46.774 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pglog_items": 0, 2024-05-12T22:19:46.774 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_bytes": 8032, 2024-05-12T22:19:46.774 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_items": 167, 2024-05-12T22:19:46.774 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_mapping_bytes": 0, 2024-05-12T22:19:46.774 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_mapping_items": 0, 2024-05-12T22:19:46.774 INFO:teuthology.orchestra.run.smithi028.stdout: "pgmap_bytes": 0, 2024-05-12T22:19:46.774 INFO:teuthology.orchestra.run.smithi028.stdout: "pgmap_items": 0, 2024-05-12T22:19:46.774 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_co_bytes": 250310, 2024-05-12T22:19:46.775 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_co_items": 1703, 2024-05-12T22:19:46.775 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_1_bytes": 0, 2024-05-12T22:19:46.775 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_1_items": 0, 2024-05-12T22:19:46.775 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_2_bytes": 0, 2024-05-12T22:19:46.775 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_2_items": 0 2024-05-12T22:19:46.775 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:46.775 INFO:teuthology.orchestra.run.smithi028.stdout: "objecter": { 2024-05-12T22:19:46.775 INFO:teuthology.orchestra.run.smithi028.stdout: "op_active": 0, 2024-05-12T22:19:46.775 INFO:teuthology.orchestra.run.smithi028.stdout: "op_laggy": 0, 2024-05-12T22:19:46.776 INFO:teuthology.orchestra.run.smithi028.stdout: "op_send": 211, 2024-05-12T22:19:46.776 INFO:teuthology.orchestra.run.smithi028.stdout: "op_send_bytes": 88378, 2024-05-12T22:19:46.776 INFO:teuthology.orchestra.run.smithi028.stdout: "op_resend": 33, 2024-05-12T22:19:46.776 INFO:teuthology.orchestra.run.smithi028.stdout: "op_reply": 211, 2024-05-12T22:19:46.776 INFO:teuthology.orchestra.run.smithi028.stdout: "op_latency": { 2024-05-12T22:19:46.776 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 211, 2024-05-12T22:19:46.776 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.725979188, 2024-05-12T22:19:46.776 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.003440659 2024-05-12T22:19:46.776 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:46.777 INFO:teuthology.orchestra.run.smithi028.stdout: "op_inflight": 0, 2024-05-12T22:19:46.777 INFO:teuthology.orchestra.run.smithi028.stdout: "oplen_avg": { 2024-05-12T22:19:46.777 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 211, 2024-05-12T22:19:46.777 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 244 2024-05-12T22:19:46.777 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:46.777 INFO:teuthology.orchestra.run.smithi028.stdout: "op": 211, 2024-05-12T22:19:46.777 INFO:teuthology.orchestra.run.smithi028.stdout: "op_r": 124, 2024-05-12T22:19:46.777 INFO:teuthology.orchestra.run.smithi028.stdout: "op_w": 87, 2024-05-12T22:19:46.777 INFO:teuthology.orchestra.run.smithi028.stdout: "op_rmw": 0, 2024-05-12T22:19:46.778 INFO:teuthology.orchestra.run.smithi028.stdout: "op_pg": 0, 2024-05-12T22:19:46.778 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_stat": 0, 2024-05-12T22:19:46.778 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_create": 10, 2024-05-12T22:19:46.778 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_read": 1, 2024-05-12T22:19:46.778 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_write": 25, 2024-05-12T22:19:46.778 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_writefull": 15, 2024-05-12T22:19:46.778 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_writesame": 0, 2024-05-12T22:19:46.778 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_append": 0, 2024-05-12T22:19:46.778 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_zero": 0, 2024-05-12T22:19:46.779 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_truncate": 0, 2024-05-12T22:19:46.779 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_delete": 14, 2024-05-12T22:19:46.779 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_mapext": 0, 2024-05-12T22:19:46.779 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_sparse_read": 0, 2024-05-12T22:19:46.779 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_clonerange": 0, 2024-05-12T22:19:46.779 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_getxattr": 123, 2024-05-12T22:19:46.779 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_setxattr": 20, 2024-05-12T22:19:46.779 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_cmpxattr": 0, 2024-05-12T22:19:46.779 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_rmxattr": 0, 2024-05-12T22:19:46.780 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_resetxattrs": 0, 2024-05-12T22:19:46.780 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_call": 0, 2024-05-12T22:19:46.780 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_watch": 0, 2024-05-12T22:19:46.780 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_notify": 0, 2024-05-12T22:19:46.780 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_src_cmpxattr": 0, 2024-05-12T22:19:46.780 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_pgls": 0, 2024-05-12T22:19:46.780 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_pgls_filter": 0, 2024-05-12T22:19:46.780 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_other": 0, 2024-05-12T22:19:46.780 INFO:teuthology.orchestra.run.smithi028.stdout: "linger_active": 0, 2024-05-12T22:19:46.781 INFO:teuthology.orchestra.run.smithi028.stdout: "linger_send": 0, 2024-05-12T22:19:46.781 INFO:teuthology.orchestra.run.smithi028.stdout: "linger_resend": 0, 2024-05-12T22:19:46.781 INFO:teuthology.orchestra.run.smithi028.stdout: "linger_ping": 0, 2024-05-12T22:19:46.781 INFO:teuthology.orchestra.run.smithi028.stdout: "poolop_active": 0, 2024-05-12T22:19:46.781 INFO:teuthology.orchestra.run.smithi028.stdout: "poolop_send": 0, 2024-05-12T22:19:46.781 INFO:teuthology.orchestra.run.smithi028.stdout: "poolop_resend": 0, 2024-05-12T22:19:46.781 INFO:teuthology.orchestra.run.smithi028.stdout: "poolstat_active": 0, 2024-05-12T22:19:46.781 INFO:teuthology.orchestra.run.smithi028.stdout: "poolstat_send": 0, 2024-05-12T22:19:46.781 INFO:teuthology.orchestra.run.smithi028.stdout: "poolstat_resend": 0, 2024-05-12T22:19:46.782 INFO:teuthology.orchestra.run.smithi028.stdout: "statfs_active": 0, 2024-05-12T22:19:46.782 INFO:teuthology.orchestra.run.smithi028.stdout: "statfs_send": 0, 2024-05-12T22:19:46.782 INFO:teuthology.orchestra.run.smithi028.stdout: "statfs_resend": 0, 2024-05-12T22:19:46.782 INFO:teuthology.orchestra.run.smithi028.stdout: "command_active": 0, 2024-05-12T22:19:46.782 INFO:teuthology.orchestra.run.smithi028.stdout: "command_send": 0, 2024-05-12T22:19:46.782 INFO:teuthology.orchestra.run.smithi028.stdout: "command_resend": 0, 2024-05-12T22:19:46.782 INFO:teuthology.orchestra.run.smithi028.stdout: "map_epoch": 97, 2024-05-12T22:19:46.782 INFO:teuthology.orchestra.run.smithi028.stdout: "map_full": 0, 2024-05-12T22:19:46.782 INFO:teuthology.orchestra.run.smithi028.stdout: "map_inc": 11, 2024-05-12T22:19:46.783 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_sessions": 11, 2024-05-12T22:19:46.783 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_session_open": 11, 2024-05-12T22:19:46.783 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_session_close": 0, 2024-05-12T22:19:46.783 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_laggy": 0, 2024-05-12T22:19:46.783 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_wr": 30, 2024-05-12T22:19:46.783 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_rd": 0, 2024-05-12T22:19:46.783 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_del": 6 2024-05-12T22:19:46.783 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:46.784 INFO:teuthology.orchestra.run.smithi028.stdout: "oft": { 2024-05-12T22:19:46.784 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_total_objs": 1, 2024-05-12T22:19:46.784 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_total_kv_pairs": 0, 2024-05-12T22:19:46.784 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_total_updates": 0, 2024-05-12T22:19:46.784 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_total_removes": 0 2024-05-12T22:19:46.784 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:46.784 INFO:teuthology.orchestra.run.smithi028.stdout: "osd-slow-ops": { 2024-05-12T22:19:46.784 INFO:teuthology.orchestra.run.smithi028.stdout: "slow_ops_count": 0 2024-05-12T22:19:46.784 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:46.785 INFO:teuthology.orchestra.run.smithi028.stdout: "purge_queue": { 2024-05-12T22:19:46.785 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executing_ops": 0, 2024-05-12T22:19:46.785 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executing_ops_high_water": 2, 2024-05-12T22:19:46.785 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executing": 0, 2024-05-12T22:19:46.785 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executing_high_water": 2, 2024-05-12T22:19:46.785 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executed_ops": 2, 2024-05-12T22:19:46.785 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executed": 2, 2024-05-12T22:19:46.785 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_item_in_journal": 0 2024-05-12T22:19:46.785 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:46.786 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-05-12T22:19:46.786 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-05-12T22:19:46.786 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 104857600, 2024-05-12T22:19:46.786 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-05-12T22:19:46.786 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 2107, 2024-05-12T22:19:46.786 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 2122554, 2024-05-12T22:19:46.786 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-05-12T22:19:46.786 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 2107, 2024-05-12T22:19:46.786 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 0, 2024-05-12T22:19:46.787 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 0, 2024-05-12T22:19:46.787 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 2107, 2024-05-12T22:19:46.787 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 2122554, 2024-05-12T22:19:46.787 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-05-12T22:19:46.787 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-12T22:19:46.787 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-12T22:19:46.787 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-12T22:19:46.787 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:46.787 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:46.788 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-objecter_bytes": { 2024-05-12T22:19:46.788 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-05-12T22:19:46.788 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 104857600, 2024-05-12T22:19:46.788 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-05-12T22:19:46.788 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 0, 2024-05-12T22:19:46.788 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 0, 2024-05-12T22:19:46.788 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-05-12T22:19:46.788 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 0, 2024-05-12T22:19:46.788 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 211, 2024-05-12T22:19:46.789 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 88596, 2024-05-12T22:19:46.789 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 211, 2024-05-12T22:19:46.789 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 88596, 2024-05-12T22:19:46.789 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-05-12T22:19:46.789 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-12T22:19:46.789 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-12T22:19:46.789 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-12T22:19:46.789 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:46.789 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:46.790 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-objecter_ops": { 2024-05-12T22:19:46.790 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-05-12T22:19:46.790 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 1024, 2024-05-12T22:19:46.790 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-05-12T22:19:46.790 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 0, 2024-05-12T22:19:46.790 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 0, 2024-05-12T22:19:46.790 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-05-12T22:19:46.790 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 0, 2024-05-12T22:19:46.791 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 211, 2024-05-12T22:19:46.791 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 211, 2024-05-12T22:19:46.791 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 211, 2024-05-12T22:19:46.791 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 211, 2024-05-12T22:19:46.791 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-05-12T22:19:46.791 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-12T22:19:46.791 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-12T22:19:46.791 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-12T22:19:46.791 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:46.792 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:46.792 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-write_buf_throttle": { 2024-05-12T22:19:46.792 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-05-12T22:19:46.792 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 3758096384, 2024-05-12T22:19:46.792 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-05-12T22:19:46.792 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 2, 2024-05-12T22:19:46.792 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 218, 2024-05-12T22:19:46.792 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-05-12T22:19:46.792 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 2, 2024-05-12T22:19:46.793 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 0, 2024-05-12T22:19:46.793 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 0, 2024-05-12T22:19:46.793 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 1, 2024-05-12T22:19:46.793 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 218, 2024-05-12T22:19:46.793 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-05-12T22:19:46.793 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-12T22:19:46.793 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-12T22:19:46.793 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-12T22:19:46.793 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:46.794 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:46.794 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-write_buf_throttle-0x56521d39e500": { 2024-05-12T22:19:46.794 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-05-12T22:19:46.794 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 3758096384, 2024-05-12T22:19:46.794 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-05-12T22:19:46.794 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 45, 2024-05-12T22:19:46.794 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 73789, 2024-05-12T22:19:46.794 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-05-12T22:19:46.794 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 45, 2024-05-12T22:19:46.795 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 0, 2024-05-12T22:19:46.795 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 0, 2024-05-12T22:19:46.795 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 24, 2024-05-12T22:19:46.795 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 73789, 2024-05-12T22:19:46.795 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-05-12T22:19:46.795 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-12T22:19:46.795 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-12T22:19:46.795 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-12T22:19:46.795 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:46.796 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:46.796 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-05-12T22:19:47.693 INFO:tasks.check_counter:Daemon mds.a mds.exported=0 2024-05-12T22:19:47.694 INFO:tasks.check_counter:Daemon mds.a mds.imported=3 2024-05-12T22:19:47.694 INFO:tasks.check_counter:Daemon mds.a mds_cache.dir_update=2 2024-05-12T22:19:47.694 INFO:tasks.check_counter:Daemon mds.a mds_cache.dir_update_receipt=67 2024-05-12T22:19:47.694 INFO:tasks.check_counter:Daemon mds.a mds.root_rsnaps=6 2024-05-12T22:19:47.694 INFO:tasks.check_counter:Daemon mds.a mds_server.req_mksnap_latency.avgcount=0 2024-05-12T22:19:47.694 INFO:tasks.check_counter:Daemon mds.a mds_server.req_rmsnap_latency.avgcount=0 2024-05-12T22:19:47.694 INFO:tasks.check_counter:Daemon mds.a mds.dir_split=0 2024-05-12T22:19:47.694 DEBUG:tasks.check_counter:Getting stats from d 2024-05-12T22:19:47.695 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f72fecff68e1d400c4568684327c900485c20d6a shell --fsid 6b92e7b4-10aa-11ef-bc98-c7b262605968 -- ceph daemon mds.d perf dump 2024-05-12T22:19:48.179 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/mon.a/config 2024-05-12T22:19:48.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:19:48 smithi028 ceph-mon[29847]: pgmap v335: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-05-12T22:19:48.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:19:48 smithi186 ceph-mon[34785]: pgmap v335: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-05-12T22:19:48.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:19:48 smithi086 ceph-mon[34297]: pgmap v335: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-05-12T22:19:50.471 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-05-12T22:19:50.472 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-0": { 2024-05-12T22:19:50.472 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 4, 2024-05-12T22:19:50.472 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 4, 2024-05-12T22:19:50.472 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 3072, 2024-05-12T22:19:50.472 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 448, 2024-05-12T22:19:50.473 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 4, 2024-05-12T22:19:50.473 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 0, 2024-05-12T22:19:50.473 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.004917417, 2024-05-12T22:19:50.473 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.000081993, 2024-05-12T22:19:50.473 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.001397905, 2024-05-12T22:19:50.473 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-05-12T22:19:50.473 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-05-12T22:19:50.474 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 3, 2024-05-12T22:19:50.474 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.001595928, 2024-05-12T22:19:50.474 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000531976 2024-05-12T22:19:50.474 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:50.474 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-05-12T22:19:50.474 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-12T22:19:50.475 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-12T22:19:50.475 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-12T22:19:50.475 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:50.475 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 3072, 2024-05-12T22:19:50.475 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 448 2024-05-12T22:19:50.475 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:50.475 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-1": { 2024-05-12T22:19:50.476 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 233, 2024-05-12T22:19:50.476 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 221, 2024-05-12T22:19:50.476 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 1721680, 2024-05-12T22:19:50.476 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 136224, 2024-05-12T22:19:50.476 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 2, 2024-05-12T22:19:50.476 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 1, 2024-05-12T22:19:50.477 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.098867650, 2024-05-12T22:19:50.477 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.025349841, 2024-05-12T22:19:50.477 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.089999881, 2024-05-12T22:19:50.477 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.010763749, 2024-05-12T22:19:50.477 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-05-12T22:19:50.477 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 220, 2024-05-12T22:19:50.477 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.022106709, 2024-05-12T22:19:50.478 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000100485 2024-05-12T22:19:50.478 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:50.478 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-05-12T22:19:50.478 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-12T22:19:50.478 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-12T22:19:50.478 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-12T22:19:50.478 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:50.479 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 1721680, 2024-05-12T22:19:50.479 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 136224 2024-05-12T22:19:50.479 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:50.479 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-2": { 2024-05-12T22:19:50.479 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 2, 2024-05-12T22:19:50.480 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 170, 2024-05-12T22:19:50.480 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 320, 2024-05-12T22:19:50.480 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 185520, 2024-05-12T22:19:50.481 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 5, 2024-05-12T22:19:50.481 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 3, 2024-05-12T22:19:50.481 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.034271041, 2024-05-12T22:19:50.481 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.025425169, 2024-05-12T22:19:50.481 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.003949773, 2024-05-12T22:19:50.481 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-05-12T22:19:50.482 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-05-12T22:19:50.482 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 167, 2024-05-12T22:19:50.482 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.026673775, 2024-05-12T22:19:50.482 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000159723 2024-05-12T22:19:50.482 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:50.483 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-05-12T22:19:50.483 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-12T22:19:50.483 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-12T22:19:50.483 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-12T22:19:50.483 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:50.483 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 320, 2024-05-12T22:19:50.483 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 185520 2024-05-12T22:19:50.484 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:50.484 INFO:teuthology.orchestra.run.smithi028.stdout: "cct": { 2024-05-12T22:19:50.484 INFO:teuthology.orchestra.run.smithi028.stdout: "total_workers": 0, 2024-05-12T22:19:50.484 INFO:teuthology.orchestra.run.smithi028.stdout: "unhealthy_workers": 0 2024-05-12T22:19:50.484 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:50.484 INFO:teuthology.orchestra.run.smithi028.stdout: "mempool": { 2024-05-12T22:19:50.484 INFO:teuthology.orchestra.run.smithi028.stdout: "bloom_filter_bytes": 0, 2024-05-12T22:19:50.484 INFO:teuthology.orchestra.run.smithi028.stdout: "bloom_filter_items": 0, 2024-05-12T22:19:50.484 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_alloc_bytes": 0, 2024-05-12T22:19:50.485 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_alloc_items": 0, 2024-05-12T22:19:50.485 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_data_bytes": 0, 2024-05-12T22:19:50.485 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_data_items": 0, 2024-05-12T22:19:50.485 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_onode_bytes": 0, 2024-05-12T22:19:50.485 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_onode_items": 0, 2024-05-12T22:19:50.485 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_meta_bytes": 0, 2024-05-12T22:19:50.485 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_meta_items": 0, 2024-05-12T22:19:50.485 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_other_bytes": 0, 2024-05-12T22:19:50.485 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_other_items": 0, 2024-05-12T22:19:50.486 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_buffer_bytes": 0, 2024-05-12T22:19:50.486 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_buffer_items": 0, 2024-05-12T22:19:50.486 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_extent_bytes": 0, 2024-05-12T22:19:50.486 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_extent_items": 0, 2024-05-12T22:19:50.486 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_blob_bytes": 0, 2024-05-12T22:19:50.486 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_blob_items": 0, 2024-05-12T22:19:50.486 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_shared_blob_bytes": 0, 2024-05-12T22:19:50.486 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_shared_blob_items": 0, 2024-05-12T22:19:50.487 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_inline_bl_bytes": 0, 2024-05-12T22:19:50.487 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_inline_bl_items": 0, 2024-05-12T22:19:50.487 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_bytes": 0, 2024-05-12T22:19:50.487 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_items": 0, 2024-05-12T22:19:50.487 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_txc_bytes": 0, 2024-05-12T22:19:50.487 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_txc_items": 0, 2024-05-12T22:19:50.487 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_deferred_bytes": 0, 2024-05-12T22:19:50.487 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_deferred_items": 0, 2024-05-12T22:19:50.487 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_bytes": 0, 2024-05-12T22:19:50.488 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_items": 0, 2024-05-12T22:19:50.488 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_bytes": 0, 2024-05-12T22:19:50.488 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_items": 0, 2024-05-12T22:19:50.488 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_reader_bytes": 0, 2024-05-12T22:19:50.488 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_reader_items": 0, 2024-05-12T22:19:50.488 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_writer_bytes": 0, 2024-05-12T22:19:50.488 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_writer_items": 0, 2024-05-12T22:19:50.488 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_anon_bytes": 150008, 2024-05-12T22:19:50.489 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_anon_items": 31, 2024-05-12T22:19:50.489 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_meta_bytes": 0, 2024-05-12T22:19:50.489 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_meta_items": 0, 2024-05-12T22:19:50.489 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_bytes": 0, 2024-05-12T22:19:50.489 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_items": 0, 2024-05-12T22:19:50.489 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mapbl_bytes": 0, 2024-05-12T22:19:50.489 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mapbl_items": 0, 2024-05-12T22:19:50.489 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pglog_bytes": 0, 2024-05-12T22:19:50.489 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pglog_items": 0, 2024-05-12T22:19:50.490 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_bytes": 0, 2024-05-12T22:19:50.490 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_items": 0, 2024-05-12T22:19:50.490 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_mapping_bytes": 0, 2024-05-12T22:19:50.490 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_mapping_items": 0, 2024-05-12T22:19:50.490 INFO:teuthology.orchestra.run.smithi028.stdout: "pgmap_bytes": 0, 2024-05-12T22:19:50.490 INFO:teuthology.orchestra.run.smithi028.stdout: "pgmap_items": 0, 2024-05-12T22:19:50.490 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_co_bytes": 928, 2024-05-12T22:19:50.490 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_co_items": 2, 2024-05-12T22:19:50.491 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_1_bytes": 0, 2024-05-12T22:19:50.491 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_1_items": 0, 2024-05-12T22:19:50.491 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_2_bytes": 0, 2024-05-12T22:19:50.491 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_2_items": 0 2024-05-12T22:19:50.491 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:50.491 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-05-12T22:19:50.491 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-05-12T22:19:50.491 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 104857600, 2024-05-12T22:19:50.491 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-05-12T22:19:50.492 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 235, 2024-05-12T22:19:50.492 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 1688849, 2024-05-12T22:19:50.492 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-05-12T22:19:50.492 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 235, 2024-05-12T22:19:50.492 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 0, 2024-05-12T22:19:50.492 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 0, 2024-05-12T22:19:50.492 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 235, 2024-05-12T22:19:50.493 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 1688849, 2024-05-12T22:19:50.493 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-05-12T22:19:50.493 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-12T22:19:50.493 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-12T22:19:50.493 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-12T22:19:50.493 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:50.494 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:50.494 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-05-12T22:19:50.521 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:19:50 smithi028 ceph-mon[29847]: pgmap v336: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-05-12T22:19:50.563 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:19:50 smithi086 ceph-mon[34297]: pgmap v336: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-05-12T22:19:50.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:19:50 smithi186 ceph-mon[34785]: pgmap v336: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-05-12T22:19:51.405 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.d 2024-05-12T22:19:51.405 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.d 2024-05-12T22:19:51.406 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.d 2024-05-12T22:19:51.406 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.d 2024-05-12T22:19:51.406 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.d 2024-05-12T22:19:51.406 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.d 2024-05-12T22:19:51.406 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.d 2024-05-12T22:19:51.406 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.d 2024-05-12T22:19:51.406 DEBUG:tasks.check_counter:Getting stats from g 2024-05-12T22:19:51.407 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f72fecff68e1d400c4568684327c900485c20d6a shell --fsid 6b92e7b4-10aa-11ef-bc98-c7b262605968 -- ceph daemon mds.g perf dump 2024-05-12T22:19:51.885 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/mon.a/config 2024-05-12T22:19:52.316 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:19:52 smithi028 ceph-mon[29847]: pgmap v337: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-12T22:19:52.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:19:52 smithi186 ceph-mon[34785]: pgmap v337: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-12T22:19:52.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:19:52 smithi086 ceph-mon[34297]: pgmap v337: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-12T22:19:54.058 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-05-12T22:19:54.058 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-0": { 2024-05-12T22:19:54.058 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 147, 2024-05-12T22:19:54.059 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 179, 2024-05-12T22:19:54.059 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 39971, 2024-05-12T22:19:54.059 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 81038, 2024-05-12T22:19:54.059 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 14, 2024-05-12T22:19:54.059 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 3, 2024-05-12T22:19:54.059 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.077533939, 2024-05-12T22:19:54.059 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.041582994, 2024-05-12T22:19:54.059 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.027458328, 2024-05-12T22:19:54.060 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.001025032, 2024-05-12T22:19:54.060 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-05-12T22:19:54.060 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 177, 2024-05-12T22:19:54.060 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.045949822, 2024-05-12T22:19:54.060 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000259603 2024-05-12T22:19:54.060 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:54.060 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-05-12T22:19:54.060 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 147, 2024-05-12T22:19:54.060 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000132088, 2024-05-12T22:19:54.061 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000898 2024-05-12T22:19:54.061 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:54.061 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 3728, 2024-05-12T22:19:54.061 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 4464 2024-05-12T22:19:54.061 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:54.061 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-1": { 2024-05-12T22:19:54.061 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 38, 2024-05-12T22:19:54.061 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 207, 2024-05-12T22:19:54.062 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 13659, 2024-05-12T22:19:54.062 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 548633, 2024-05-12T22:19:54.062 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 17, 2024-05-12T22:19:54.062 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 6, 2024-05-12T22:19:54.062 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.051330231, 2024-05-12T22:19:54.062 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.029615102, 2024-05-12T22:19:54.062 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.014698053, 2024-05-12T22:19:54.062 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.000741254, 2024-05-12T22:19:54.062 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-05-12T22:19:54.063 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 200, 2024-05-12T22:19:54.063 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.057724053, 2024-05-12T22:19:54.063 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000288620 2024-05-12T22:19:54.063 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:54.063 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-05-12T22:19:54.063 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 31, 2024-05-12T22:19:54.063 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000006856, 2024-05-12T22:19:54.063 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000221 2024-05-12T22:19:54.063 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:54.064 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 2496, 2024-05-12T22:19:54.064 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 460112 2024-05-12T22:19:54.064 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:54.064 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-2": { 2024-05-12T22:19:54.064 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 1490, 2024-05-12T22:19:54.064 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 1193, 2024-05-12T22:19:54.064 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 2156870, 2024-05-12T22:19:54.064 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 837709, 2024-05-12T22:19:54.064 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 38, 2024-05-12T22:19:54.065 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 3, 2024-05-12T22:19:54.065 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.354263726, 2024-05-12T22:19:54.065 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.130009123, 2024-05-12T22:19:54.065 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.268142235, 2024-05-12T22:19:54.065 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.018149731, 2024-05-12T22:19:54.065 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-05-12T22:19:54.065 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 1197, 2024-05-12T22:19:54.065 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.117322998, 2024-05-12T22:19:54.065 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000098014 2024-05-12T22:19:54.066 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:54.066 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-05-12T22:19:54.066 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 2080, 2024-05-12T22:19:54.066 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.001091873, 2024-05-12T22:19:54.066 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000524 2024-05-12T22:19:54.066 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:54.066 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 1770512, 2024-05-12T22:19:54.066 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 145472 2024-05-12T22:19:54.066 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:54.067 INFO:teuthology.orchestra.run.smithi028.stdout: "cct": { 2024-05-12T22:19:54.067 INFO:teuthology.orchestra.run.smithi028.stdout: "total_workers": 1, 2024-05-12T22:19:54.067 INFO:teuthology.orchestra.run.smithi028.stdout: "unhealthy_workers": 0 2024-05-12T22:19:54.067 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:54.067 INFO:teuthology.orchestra.run.smithi028.stdout: "finisher-MDSRank": { 2024-05-12T22:19:54.067 INFO:teuthology.orchestra.run.smithi028.stdout: "queue_len": 0, 2024-05-12T22:19:54.067 INFO:teuthology.orchestra.run.smithi028.stdout: "complete_latency": { 2024-05-12T22:19:54.067 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 60, 2024-05-12T22:19:54.068 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.003368781, 2024-05-12T22:19:54.068 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000056146 2024-05-12T22:19:54.068 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:54.068 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:54.068 INFO:teuthology.orchestra.run.smithi028.stdout: "finisher-PurgeQueue": { 2024-05-12T22:19:54.068 INFO:teuthology.orchestra.run.smithi028.stdout: "queue_len": 0, 2024-05-12T22:19:54.068 INFO:teuthology.orchestra.run.smithi028.stdout: "complete_latency": { 2024-05-12T22:19:54.068 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 2, 2024-05-12T22:19:54.068 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000018802, 2024-05-12T22:19:54.068 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000009401 2024-05-12T22:19:54.069 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:54.069 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:54.069 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": { 2024-05-12T22:19:54.069 INFO:teuthology.orchestra.run.smithi028.stdout: "request": 107, 2024-05-12T22:19:54.069 INFO:teuthology.orchestra.run.smithi028.stdout: "reply": 11, 2024-05-12T22:19:54.069 INFO:teuthology.orchestra.run.smithi028.stdout: "reply_latency": { 2024-05-12T22:19:54.069 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 11, 2024-05-12T22:19:54.069 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.005241029, 2024-05-12T22:19:54.069 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000476457 2024-05-12T22:19:54.070 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:54.070 INFO:teuthology.orchestra.run.smithi028.stdout: "slow_reply": 0, 2024-05-12T22:19:54.070 INFO:teuthology.orchestra.run.smithi028.stdout: "forward": 96, 2024-05-12T22:19:54.070 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_fetch_complete": 0, 2024-05-12T22:19:54.070 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_fetch_keys": 0, 2024-05-12T22:19:54.070 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_commit": 11, 2024-05-12T22:19:54.070 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_split": 0, 2024-05-12T22:19:54.070 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_merge": 0, 2024-05-12T22:19:54.071 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes": 37, 2024-05-12T22:19:54.071 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_top": 9, 2024-05-12T22:19:54.071 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_bottom": 28, 2024-05-12T22:19:54.071 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_pin_tail": 0, 2024-05-12T22:19:54.071 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_pinned": 22, 2024-05-12T22:19:54.071 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_expired": 10, 2024-05-12T22:19:54.071 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_with_caps": 1, 2024-05-12T22:19:54.071 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": 1, 2024-05-12T22:19:54.071 INFO:teuthology.orchestra.run.smithi028.stdout: "subtrees": 3, 2024-05-12T22:19:54.072 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse": 164, 2024-05-12T22:19:54.072 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_hit": 43, 2024-05-12T22:19:54.072 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_forward": 93, 2024-05-12T22:19:54.072 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_discover": 23, 2024-05-12T22:19:54.072 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_dir_fetch": 0, 2024-05-12T22:19:54.072 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_remote_ino": 0, 2024-05-12T22:19:54.072 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_lock": 0, 2024-05-12T22:19:54.072 INFO:teuthology.orchestra.run.smithi028.stdout: "load_cent": 0, 2024-05-12T22:19:54.072 INFO:teuthology.orchestra.run.smithi028.stdout: "q": 0, 2024-05-12T22:19:54.073 INFO:teuthology.orchestra.run.smithi028.stdout: "exported": 0, 2024-05-12T22:19:54.073 INFO:teuthology.orchestra.run.smithi028.stdout: "exported_inodes": 0, 2024-05-12T22:19:54.073 INFO:teuthology.orchestra.run.smithi028.stdout: "imported": 0, 2024-05-12T22:19:54.073 INFO:teuthology.orchestra.run.smithi028.stdout: "imported_inodes": 0, 2024-05-12T22:19:54.073 INFO:teuthology.orchestra.run.smithi028.stdout: "openino_dir_fetch": 0, 2024-05-12T22:19:54.073 INFO:teuthology.orchestra.run.smithi028.stdout: "openino_backtrace_fetch": 0, 2024-05-12T22:19:54.073 INFO:teuthology.orchestra.run.smithi028.stdout: "openino_peer_discover": 0, 2024-05-12T22:19:54.074 INFO:teuthology.orchestra.run.smithi028.stdout: "root_rfiles": 3454, 2024-05-12T22:19:54.074 INFO:teuthology.orchestra.run.smithi028.stdout: "root_rbytes": 26743557, 2024-05-12T22:19:54.074 INFO:teuthology.orchestra.run.smithi028.stdout: "root_rsnaps": 6, 2024-05-12T22:19:54.074 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_backtrace_fetch": 0, 2024-05-12T22:19:54.074 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_set_tag": 0, 2024-05-12T22:19:54.074 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_backtrace_repaired": 0, 2024-05-12T22:19:54.074 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_inotable_repaired": 0, 2024-05-12T22:19:54.074 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_dir_inodes": 0, 2024-05-12T22:19:54.075 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_dir_base_inodes": 0, 2024-05-12T22:19:54.075 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_dirfrag_rstats": 0, 2024-05-12T22:19:54.075 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_file_inodes": 0, 2024-05-12T22:19:54.075 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_inode_file_caps": 0, 2024-05-12T22:19:54.075 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_cap_op_revoke": 0, 2024-05-12T22:19:54.075 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_cap_op_grant": 0, 2024-05-12T22:19:54.075 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_cap_op_trunc": 0, 2024-05-12T22:19:54.075 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-05-12T22:19:54.075 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_cap_op_flush_ack": 0, 2024-05-12T22:19:54.076 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_client_caps": 0, 2024-05-12T22:19:54.076 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_client_caps_dirty": 0, 2024-05-12T22:19:54.076 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_client_cap_release": 0, 2024-05-12T22:19:54.076 INFO:teuthology.orchestra.run.smithi028.stdout: "process_request_cap_release": 1 2024-05-12T22:19:54.076 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:54.076 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_cache": { 2024-05-12T22:19:54.076 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_update": 0, 2024-05-12T22:19:54.076 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_update_receipt": 74, 2024-05-12T22:19:54.077 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_try_discover": 47, 2024-05-12T22:19:54.077 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_send_discover": 53, 2024-05-12T22:19:54.077 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_handle_discover": 0, 2024-05-12T22:19:54.077 INFO:teuthology.orchestra.run.smithi028.stdout: "num_strays": 0, 2024-05-12T22:19:54.077 INFO:teuthology.orchestra.run.smithi028.stdout: "num_strays_delayed": 0, 2024-05-12T22:19:54.077 INFO:teuthology.orchestra.run.smithi028.stdout: "num_strays_enqueuing": 0, 2024-05-12T22:19:54.077 INFO:teuthology.orchestra.run.smithi028.stdout: "strays_created": 0, 2024-05-12T22:19:54.077 INFO:teuthology.orchestra.run.smithi028.stdout: "strays_enqueued": 0, 2024-05-12T22:19:54.077 INFO:teuthology.orchestra.run.smithi028.stdout: "strays_reintegrated": 0, 2024-05-12T22:19:54.077 INFO:teuthology.orchestra.run.smithi028.stdout: "strays_migrated": 0, 2024-05-12T22:19:54.078 INFO:teuthology.orchestra.run.smithi028.stdout: "num_recovering_processing": 0, 2024-05-12T22:19:54.078 INFO:teuthology.orchestra.run.smithi028.stdout: "num_recovering_enqueued": 0, 2024-05-12T22:19:54.078 INFO:teuthology.orchestra.run.smithi028.stdout: "num_recovering_prioritized": 0, 2024-05-12T22:19:54.078 INFO:teuthology.orchestra.run.smithi028.stdout: "recovery_started": 0, 2024-05-12T22:19:54.078 INFO:teuthology.orchestra.run.smithi028.stdout: "recovery_completed": 0, 2024-05-12T22:19:54.078 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_quiesce_path": 0, 2024-05-12T22:19:54.078 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_quiesce_inode": 0, 2024-05-12T22:19:54.078 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_enqueue_scrub": 0, 2024-05-12T22:19:54.079 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_exportdir": 0, 2024-05-12T22:19:54.079 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_flush": 0, 2024-05-12T22:19:54.079 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_fragmentdir": 0, 2024-05-12T22:19:54.079 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_fragstats": 0, 2024-05-12T22:19:54.079 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_inodestats": 0 2024-05-12T22:19:54.079 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:54.079 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_log": { 2024-05-12T22:19:54.079 INFO:teuthology.orchestra.run.smithi028.stdout: "evlrg": 0, 2024-05-12T22:19:54.079 INFO:teuthology.orchestra.run.smithi028.stdout: "evadd": 11, 2024-05-12T22:19:54.080 INFO:teuthology.orchestra.run.smithi028.stdout: "evex": 0, 2024-05-12T22:19:54.080 INFO:teuthology.orchestra.run.smithi028.stdout: "evtrm": 0, 2024-05-12T22:19:54.080 INFO:teuthology.orchestra.run.smithi028.stdout: "ev": 11, 2024-05-12T22:19:54.080 INFO:teuthology.orchestra.run.smithi028.stdout: "evexg": 0, 2024-05-12T22:19:54.080 INFO:teuthology.orchestra.run.smithi028.stdout: "evexd": 0, 2024-05-12T22:19:54.080 INFO:teuthology.orchestra.run.smithi028.stdout: "segadd": 2, 2024-05-12T22:19:54.080 INFO:teuthology.orchestra.run.smithi028.stdout: "segex": 0, 2024-05-12T22:19:54.080 INFO:teuthology.orchestra.run.smithi028.stdout: "segtrm": 0, 2024-05-12T22:19:54.080 INFO:teuthology.orchestra.run.smithi028.stdout: "seg": 2, 2024-05-12T22:19:54.081 INFO:teuthology.orchestra.run.smithi028.stdout: "segmjr": 2, 2024-05-12T22:19:54.081 INFO:teuthology.orchestra.run.smithi028.stdout: "segexg": 0, 2024-05-12T22:19:54.081 INFO:teuthology.orchestra.run.smithi028.stdout: "segexd": 0, 2024-05-12T22:19:54.081 INFO:teuthology.orchestra.run.smithi028.stdout: "expos": 4194304, 2024-05-12T22:19:54.081 INFO:teuthology.orchestra.run.smithi028.stdout: "wrpos": 4197330, 2024-05-12T22:19:54.081 INFO:teuthology.orchestra.run.smithi028.stdout: "rdpos": 0, 2024-05-12T22:19:54.081 INFO:teuthology.orchestra.run.smithi028.stdout: "jlat": { 2024-05-12T22:19:54.081 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 10, 2024-05-12T22:19:54.081 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.023669799, 2024-05-12T22:19:54.082 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.002366979 2024-05-12T22:19:54.082 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:54.082 INFO:teuthology.orchestra.run.smithi028.stdout: "replayed": 0 2024-05-12T22:19:54.082 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:54.082 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_mem": { 2024-05-12T22:19:54.082 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 41, 2024-05-12T22:19:54.082 INFO:teuthology.orchestra.run.smithi028.stdout: "ino+": 46, 2024-05-12T22:19:54.082 INFO:teuthology.orchestra.run.smithi028.stdout: "ino-": 5, 2024-05-12T22:19:54.083 INFO:teuthology.orchestra.run.smithi028.stdout: "dir": 45, 2024-05-12T22:19:54.083 INFO:teuthology.orchestra.run.smithi028.stdout: "dir+": 51, 2024-05-12T22:19:54.083 INFO:teuthology.orchestra.run.smithi028.stdout: "dir-": 6, 2024-05-12T22:19:54.083 INFO:teuthology.orchestra.run.smithi028.stdout: "dn": 37, 2024-05-12T22:19:54.083 INFO:teuthology.orchestra.run.smithi028.stdout: "dn+": 47, 2024-05-12T22:19:54.083 INFO:teuthology.orchestra.run.smithi028.stdout: "dn-": 10, 2024-05-12T22:19:54.083 INFO:teuthology.orchestra.run.smithi028.stdout: "cap": 1, 2024-05-12T22:19:54.083 INFO:teuthology.orchestra.run.smithi028.stdout: "cap+": 9, 2024-05-12T22:19:54.083 INFO:teuthology.orchestra.run.smithi028.stdout: "cap-": 8, 2024-05-12T22:19:54.084 INFO:teuthology.orchestra.run.smithi028.stdout: "rss": 51832, 2024-05-12T22:19:54.084 INFO:teuthology.orchestra.run.smithi028.stdout: "heap": 223612 2024-05-12T22:19:54.084 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:54.084 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_server": { 2024-05-12T22:19:54.084 INFO:teuthology.orchestra.run.smithi028.stdout: "dispatch_client_request": 113, 2024-05-12T22:19:54.084 INFO:teuthology.orchestra.run.smithi028.stdout: "dispatch_server_request": 0, 2024-05-12T22:19:54.084 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_client_request": 107, 2024-05-12T22:19:54.084 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_client_session": 25, 2024-05-12T22:19:54.084 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_peer_request": 1, 2024-05-12T22:19:54.085 INFO:teuthology.orchestra.run.smithi028.stdout: "req_create_latency": { 2024-05-12T22:19:54.085 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-12T22:19:54.085 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-12T22:19:54.085 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-12T22:19:54.085 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:54.085 INFO:teuthology.orchestra.run.smithi028.stdout: "req_getattr_latency": { 2024-05-12T22:19:54.085 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 4, 2024-05-12T22:19:54.085 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.002893076, 2024-05-12T22:19:54.085 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000723269 2024-05-12T22:19:54.086 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:54.086 INFO:teuthology.orchestra.run.smithi028.stdout: "req_getfilelock_latency": { 2024-05-12T22:19:54.086 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-12T22:19:54.086 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-12T22:19:54.086 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-12T22:19:54.086 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:54.086 INFO:teuthology.orchestra.run.smithi028.stdout: "req_link_latency": { 2024-05-12T22:19:54.086 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-12T22:19:54.086 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-12T22:19:54.087 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-12T22:19:54.087 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:54.087 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookup_latency": { 2024-05-12T22:19:54.087 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 6, 2024-05-12T22:19:54.087 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.002169974, 2024-05-12T22:19:54.087 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000361662 2024-05-12T22:19:54.087 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:54.087 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookuphash_latency": { 2024-05-12T22:19:54.087 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-12T22:19:54.088 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-12T22:19:54.088 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-12T22:19:54.088 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:54.088 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookupino_latency": { 2024-05-12T22:19:54.088 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 1, 2024-05-12T22:19:54.088 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000177979, 2024-05-12T22:19:54.088 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000177979 2024-05-12T22:19:54.088 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:54.088 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookupname_latency": { 2024-05-12T22:19:54.089 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-12T22:19:54.089 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-12T22:19:54.089 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-12T22:19:54.089 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:54.089 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookupparent_latency": { 2024-05-12T22:19:54.089 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-12T22:19:54.089 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-12T22:19:54.089 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-12T22:19:54.089 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:54.090 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookupsnap_latency": { 2024-05-12T22:19:54.090 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-12T22:19:54.090 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-12T22:19:54.090 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-12T22:19:54.090 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:54.090 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lssnap_latency": { 2024-05-12T22:19:54.090 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-12T22:19:54.090 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-12T22:19:54.090 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-12T22:19:54.091 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:54.091 INFO:teuthology.orchestra.run.smithi028.stdout: "req_mkdir_latency": { 2024-05-12T22:19:54.091 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-12T22:19:54.091 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-12T22:19:54.092 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-12T22:19:54.092 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:54.092 INFO:teuthology.orchestra.run.smithi028.stdout: "req_mknod_latency": { 2024-05-12T22:19:54.092 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-12T22:19:54.092 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-12T22:19:54.092 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-12T22:19:54.092 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:54.092 INFO:teuthology.orchestra.run.smithi028.stdout: "req_mksnap_latency": { 2024-05-12T22:19:54.093 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-12T22:19:54.093 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-12T22:19:54.093 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-12T22:19:54.093 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:54.093 INFO:teuthology.orchestra.run.smithi028.stdout: "req_open_latency": { 2024-05-12T22:19:54.093 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-12T22:19:54.093 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-12T22:19:54.093 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-12T22:19:54.093 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:54.093 INFO:teuthology.orchestra.run.smithi028.stdout: "req_readdir_latency": { 2024-05-12T22:19:54.094 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-12T22:19:54.094 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-12T22:19:54.094 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-12T22:19:54.094 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:54.094 INFO:teuthology.orchestra.run.smithi028.stdout: "req_rename_latency": { 2024-05-12T22:19:54.094 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-12T22:19:54.094 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-12T22:19:54.094 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-12T22:19:54.094 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:54.095 INFO:teuthology.orchestra.run.smithi028.stdout: "req_renamesnap_latency": { 2024-05-12T22:19:54.095 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-12T22:19:54.095 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-12T22:19:54.095 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-12T22:19:54.095 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:54.095 INFO:teuthology.orchestra.run.smithi028.stdout: "req_snapdiff_latency": { 2024-05-12T22:19:54.095 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-12T22:19:54.095 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-12T22:19:54.096 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-12T22:19:54.096 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:54.096 INFO:teuthology.orchestra.run.smithi028.stdout: "req_rmdir_latency": { 2024-05-12T22:19:54.096 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-12T22:19:54.096 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-12T22:19:54.096 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-12T22:19:54.096 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:54.096 INFO:teuthology.orchestra.run.smithi028.stdout: "req_rmsnap_latency": { 2024-05-12T22:19:54.096 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-12T22:19:54.097 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-12T22:19:54.097 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-12T22:19:54.097 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:54.097 INFO:teuthology.orchestra.run.smithi028.stdout: "req_rmxattr_latency": { 2024-05-12T22:19:54.097 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-12T22:19:54.097 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-12T22:19:54.097 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-12T22:19:54.097 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:54.097 INFO:teuthology.orchestra.run.smithi028.stdout: "req_setattr_latency": { 2024-05-12T22:19:54.098 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-12T22:19:54.098 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-12T22:19:54.098 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-12T22:19:54.098 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:54.098 INFO:teuthology.orchestra.run.smithi028.stdout: "req_setdirlayout_latency": { 2024-05-12T22:19:54.098 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-12T22:19:54.098 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-12T22:19:54.098 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-12T22:19:54.098 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:54.099 INFO:teuthology.orchestra.run.smithi028.stdout: "req_setfilelock_latency": { 2024-05-12T22:19:54.099 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-12T22:19:54.099 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-12T22:19:54.099 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-12T22:19:54.099 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:54.099 INFO:teuthology.orchestra.run.smithi028.stdout: "req_setlayout_latency": { 2024-05-12T22:19:54.099 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-12T22:19:54.099 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-12T22:19:54.099 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-12T22:19:54.100 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:54.100 INFO:teuthology.orchestra.run.smithi028.stdout: "req_setxattr_latency": { 2024-05-12T22:19:54.100 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-12T22:19:54.100 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-12T22:19:54.100 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-12T22:19:54.100 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:54.100 INFO:teuthology.orchestra.run.smithi028.stdout: "req_symlink_latency": { 2024-05-12T22:19:54.100 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-12T22:19:54.100 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-12T22:19:54.101 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-12T22:19:54.101 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:54.101 INFO:teuthology.orchestra.run.smithi028.stdout: "req_unlink_latency": { 2024-05-12T22:19:54.101 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-12T22:19:54.101 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-12T22:19:54.101 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-12T22:19:54.101 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:54.101 INFO:teuthology.orchestra.run.smithi028.stdout: "cap_revoke_eviction": 0, 2024-05-12T22:19:54.101 INFO:teuthology.orchestra.run.smithi028.stdout: "cap_acquisition_throttle": 0, 2024-05-12T22:19:54.102 INFO:teuthology.orchestra.run.smithi028.stdout: "req_getvxattr_latency": { 2024-05-12T22:19:54.102 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-12T22:19:54.102 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-12T22:19:54.102 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-12T22:19:54.102 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:54.102 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:54.102 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_sessions": { 2024-05-12T22:19:54.102 INFO:teuthology.orchestra.run.smithi028.stdout: "session_count": 1, 2024-05-12T22:19:54.102 INFO:teuthology.orchestra.run.smithi028.stdout: "session_add": 5, 2024-05-12T22:19:54.103 INFO:teuthology.orchestra.run.smithi028.stdout: "session_remove": 4, 2024-05-12T22:19:54.103 INFO:teuthology.orchestra.run.smithi028.stdout: "sessions_open": 1, 2024-05-12T22:19:54.103 INFO:teuthology.orchestra.run.smithi028.stdout: "sessions_stale": 0, 2024-05-12T22:19:54.103 INFO:teuthology.orchestra.run.smithi028.stdout: "total_load": 9, 2024-05-12T22:19:54.103 INFO:teuthology.orchestra.run.smithi028.stdout: "average_load": 2, 2024-05-12T22:19:54.103 INFO:teuthology.orchestra.run.smithi028.stdout: "avg_session_uptime": 640, 2024-05-12T22:19:54.103 INFO:teuthology.orchestra.run.smithi028.stdout: "mdthresh_evicted": 0 2024-05-12T22:19:54.103 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:54.103 INFO:teuthology.orchestra.run.smithi028.stdout: "mempool": { 2024-05-12T22:19:54.104 INFO:teuthology.orchestra.run.smithi028.stdout: "bloom_filter_bytes": 0, 2024-05-12T22:19:54.104 INFO:teuthology.orchestra.run.smithi028.stdout: "bloom_filter_items": 0, 2024-05-12T22:19:54.104 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_alloc_bytes": 0, 2024-05-12T22:19:54.104 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_alloc_items": 0, 2024-05-12T22:19:54.104 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_data_bytes": 0, 2024-05-12T22:19:54.104 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_data_items": 0, 2024-05-12T22:19:54.104 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_onode_bytes": 0, 2024-05-12T22:19:54.104 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_onode_items": 0, 2024-05-12T22:19:54.104 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_meta_bytes": 0, 2024-05-12T22:19:54.104 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_meta_items": 0, 2024-05-12T22:19:54.105 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_other_bytes": 0, 2024-05-12T22:19:54.105 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_other_items": 0, 2024-05-12T22:19:54.105 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_buffer_bytes": 0, 2024-05-12T22:19:54.105 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_buffer_items": 0, 2024-05-12T22:19:54.105 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_extent_bytes": 0, 2024-05-12T22:19:54.105 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_extent_items": 0, 2024-05-12T22:19:54.105 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_blob_bytes": 0, 2024-05-12T22:19:54.105 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_blob_items": 0, 2024-05-12T22:19:54.105 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_shared_blob_bytes": 0, 2024-05-12T22:19:54.106 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_shared_blob_items": 0, 2024-05-12T22:19:54.106 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_inline_bl_bytes": 0, 2024-05-12T22:19:54.106 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_inline_bl_items": 0, 2024-05-12T22:19:54.106 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_bytes": 0, 2024-05-12T22:19:54.106 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_items": 0, 2024-05-12T22:19:54.106 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_txc_bytes": 0, 2024-05-12T22:19:54.106 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_txc_items": 0, 2024-05-12T22:19:54.106 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_deferred_bytes": 0, 2024-05-12T22:19:54.106 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_deferred_items": 0, 2024-05-12T22:19:54.107 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_bytes": 0, 2024-05-12T22:19:54.107 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_items": 0, 2024-05-12T22:19:54.107 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_bytes": 0, 2024-05-12T22:19:54.107 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_items": 0, 2024-05-12T22:19:54.107 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_reader_bytes": 0, 2024-05-12T22:19:54.107 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_reader_items": 0, 2024-05-12T22:19:54.107 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_writer_bytes": 0, 2024-05-12T22:19:54.107 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_writer_items": 0, 2024-05-12T22:19:54.107 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_anon_bytes": 243379, 2024-05-12T22:19:54.108 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_anon_items": 123, 2024-05-12T22:19:54.108 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_meta_bytes": 0, 2024-05-12T22:19:54.108 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_meta_items": 0, 2024-05-12T22:19:54.108 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_bytes": 0, 2024-05-12T22:19:54.108 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_items": 0, 2024-05-12T22:19:54.108 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mapbl_bytes": 0, 2024-05-12T22:19:54.108 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mapbl_items": 0, 2024-05-12T22:19:54.108 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pglog_bytes": 0, 2024-05-12T22:19:54.108 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pglog_items": 0, 2024-05-12T22:19:54.109 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_bytes": 8032, 2024-05-12T22:19:54.109 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_items": 167, 2024-05-12T22:19:54.109 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_mapping_bytes": 0, 2024-05-12T22:19:54.109 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_mapping_items": 0, 2024-05-12T22:19:54.109 INFO:teuthology.orchestra.run.smithi028.stdout: "pgmap_bytes": 0, 2024-05-12T22:19:54.109 INFO:teuthology.orchestra.run.smithi028.stdout: "pgmap_items": 0, 2024-05-12T22:19:54.109 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_co_bytes": 191489, 2024-05-12T22:19:54.109 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_co_items": 1150, 2024-05-12T22:19:54.109 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_1_bytes": 0, 2024-05-12T22:19:54.110 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_1_items": 0, 2024-05-12T22:19:54.110 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_2_bytes": 0, 2024-05-12T22:19:54.110 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_2_items": 0 2024-05-12T22:19:54.110 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:54.110 INFO:teuthology.orchestra.run.smithi028.stdout: "objecter": { 2024-05-12T22:19:54.110 INFO:teuthology.orchestra.run.smithi028.stdout: "op_active": 0, 2024-05-12T22:19:54.110 INFO:teuthology.orchestra.run.smithi028.stdout: "op_laggy": 0, 2024-05-12T22:19:54.110 INFO:teuthology.orchestra.run.smithi028.stdout: "op_send": 50, 2024-05-12T22:19:54.111 INFO:teuthology.orchestra.run.smithi028.stdout: "op_send_bytes": 13689, 2024-05-12T22:19:54.111 INFO:teuthology.orchestra.run.smithi028.stdout: "op_resend": 28, 2024-05-12T22:19:54.111 INFO:teuthology.orchestra.run.smithi028.stdout: "op_reply": 50, 2024-05-12T22:19:54.111 INFO:teuthology.orchestra.run.smithi028.stdout: "op_latency": { 2024-05-12T22:19:54.111 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 50, 2024-05-12T22:19:54.111 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.414988108, 2024-05-12T22:19:54.111 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.008299762 2024-05-12T22:19:54.111 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:54.111 INFO:teuthology.orchestra.run.smithi028.stdout: "op_inflight": 0, 2024-05-12T22:19:54.112 INFO:teuthology.orchestra.run.smithi028.stdout: "oplen_avg": { 2024-05-12T22:19:54.112 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 50, 2024-05-12T22:19:54.112 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 75 2024-05-12T22:19:54.112 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:54.112 INFO:teuthology.orchestra.run.smithi028.stdout: "op": 50, 2024-05-12T22:19:54.112 INFO:teuthology.orchestra.run.smithi028.stdout: "op_r": 0, 2024-05-12T22:19:54.112 INFO:teuthology.orchestra.run.smithi028.stdout: "op_w": 50, 2024-05-12T22:19:54.112 INFO:teuthology.orchestra.run.smithi028.stdout: "op_rmw": 0, 2024-05-12T22:19:54.112 INFO:teuthology.orchestra.run.smithi028.stdout: "op_pg": 0, 2024-05-12T22:19:54.113 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_stat": 0, 2024-05-12T22:19:54.113 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_create": 10, 2024-05-12T22:19:54.113 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_read": 0, 2024-05-12T22:19:54.113 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_write": 10, 2024-05-12T22:19:54.113 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_writefull": 8, 2024-05-12T22:19:54.113 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_writesame": 0, 2024-05-12T22:19:54.113 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_append": 0, 2024-05-12T22:19:54.113 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_zero": 0, 2024-05-12T22:19:54.113 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_truncate": 0, 2024-05-12T22:19:54.113 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_delete": 6, 2024-05-12T22:19:54.114 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_mapext": 0, 2024-05-12T22:19:54.114 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_sparse_read": 0, 2024-05-12T22:19:54.114 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_clonerange": 0, 2024-05-12T22:19:54.114 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_getxattr": 0, 2024-05-12T22:19:54.114 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_setxattr": 20, 2024-05-12T22:19:54.114 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_cmpxattr": 0, 2024-05-12T22:19:54.114 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_rmxattr": 0, 2024-05-12T22:19:54.114 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_resetxattrs": 0, 2024-05-12T22:19:54.115 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_call": 0, 2024-05-12T22:19:54.115 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_watch": 0, 2024-05-12T22:19:54.115 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_notify": 0, 2024-05-12T22:19:54.115 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_src_cmpxattr": 0, 2024-05-12T22:19:54.115 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_pgls": 0, 2024-05-12T22:19:54.115 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_pgls_filter": 0, 2024-05-12T22:19:54.115 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_other": 0, 2024-05-12T22:19:54.115 INFO:teuthology.orchestra.run.smithi028.stdout: "linger_active": 0, 2024-05-12T22:19:54.115 INFO:teuthology.orchestra.run.smithi028.stdout: "linger_send": 0, 2024-05-12T22:19:54.115 INFO:teuthology.orchestra.run.smithi028.stdout: "linger_resend": 0, 2024-05-12T22:19:54.116 INFO:teuthology.orchestra.run.smithi028.stdout: "linger_ping": 0, 2024-05-12T22:19:54.116 INFO:teuthology.orchestra.run.smithi028.stdout: "poolop_active": 0, 2024-05-12T22:19:54.116 INFO:teuthology.orchestra.run.smithi028.stdout: "poolop_send": 0, 2024-05-12T22:19:54.116 INFO:teuthology.orchestra.run.smithi028.stdout: "poolop_resend": 0, 2024-05-12T22:19:54.116 INFO:teuthology.orchestra.run.smithi028.stdout: "poolstat_active": 0, 2024-05-12T22:19:54.116 INFO:teuthology.orchestra.run.smithi028.stdout: "poolstat_send": 0, 2024-05-12T22:19:54.116 INFO:teuthology.orchestra.run.smithi028.stdout: "poolstat_resend": 0, 2024-05-12T22:19:54.116 INFO:teuthology.orchestra.run.smithi028.stdout: "statfs_active": 0, 2024-05-12T22:19:54.116 INFO:teuthology.orchestra.run.smithi028.stdout: "statfs_send": 0, 2024-05-12T22:19:54.116 INFO:teuthology.orchestra.run.smithi028.stdout: "statfs_resend": 0, 2024-05-12T22:19:54.117 INFO:teuthology.orchestra.run.smithi028.stdout: "command_active": 0, 2024-05-12T22:19:54.117 INFO:teuthology.orchestra.run.smithi028.stdout: "command_send": 0, 2024-05-12T22:19:54.117 INFO:teuthology.orchestra.run.smithi028.stdout: "command_resend": 0, 2024-05-12T22:19:54.117 INFO:teuthology.orchestra.run.smithi028.stdout: "map_epoch": 97, 2024-05-12T22:19:54.117 INFO:teuthology.orchestra.run.smithi028.stdout: "map_full": 0, 2024-05-12T22:19:54.117 INFO:teuthology.orchestra.run.smithi028.stdout: "map_inc": 11, 2024-05-12T22:19:54.117 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_sessions": 8, 2024-05-12T22:19:54.117 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_session_open": 8, 2024-05-12T22:19:54.117 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_session_close": 0, 2024-05-12T22:19:54.118 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_laggy": 0, 2024-05-12T22:19:54.118 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_wr": 19, 2024-05-12T22:19:54.118 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_rd": 0, 2024-05-12T22:19:54.118 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_del": 2 2024-05-12T22:19:54.118 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:54.118 INFO:teuthology.orchestra.run.smithi028.stdout: "oft": { 2024-05-12T22:19:54.118 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_total_objs": 1, 2024-05-12T22:19:54.118 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_total_kv_pairs": 0, 2024-05-12T22:19:54.118 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_total_updates": 0, 2024-05-12T22:19:54.119 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_total_removes": 0 2024-05-12T22:19:54.119 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:54.119 INFO:teuthology.orchestra.run.smithi028.stdout: "osd-slow-ops": { 2024-05-12T22:19:54.119 INFO:teuthology.orchestra.run.smithi028.stdout: "slow_ops_count": 0 2024-05-12T22:19:54.119 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:54.119 INFO:teuthology.orchestra.run.smithi028.stdout: "purge_queue": { 2024-05-12T22:19:54.119 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executing_ops": 0, 2024-05-12T22:19:54.119 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executing_ops_high_water": 0, 2024-05-12T22:19:54.119 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executing": 0, 2024-05-12T22:19:54.120 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executing_high_water": 0, 2024-05-12T22:19:54.120 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executed_ops": 0, 2024-05-12T22:19:54.120 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executed": 0, 2024-05-12T22:19:54.120 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_item_in_journal": 0 2024-05-12T22:19:54.120 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:54.120 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-05-12T22:19:54.120 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-05-12T22:19:54.120 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 104857600, 2024-05-12T22:19:54.120 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-05-12T22:19:54.120 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 1671, 2024-05-12T22:19:54.121 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 2043283, 2024-05-12T22:19:54.121 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-05-12T22:19:54.121 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 1671, 2024-05-12T22:19:54.121 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 0, 2024-05-12T22:19:54.121 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 0, 2024-05-12T22:19:54.121 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 1671, 2024-05-12T22:19:54.121 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 2043283, 2024-05-12T22:19:54.121 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-05-12T22:19:54.122 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-12T22:19:54.122 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-12T22:19:54.122 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-12T22:19:54.122 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:54.122 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:54.122 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-objecter_bytes": { 2024-05-12T22:19:54.122 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-05-12T22:19:54.122 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 104857600, 2024-05-12T22:19:54.122 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-05-12T22:19:54.123 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 0, 2024-05-12T22:19:54.123 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 0, 2024-05-12T22:19:54.123 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-05-12T22:19:54.123 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 0, 2024-05-12T22:19:54.123 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 50, 2024-05-12T22:19:54.123 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 13689, 2024-05-12T22:19:54.123 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 50, 2024-05-12T22:19:54.123 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 13689, 2024-05-12T22:19:54.123 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-05-12T22:19:54.124 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-12T22:19:54.124 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-12T22:19:54.124 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-12T22:19:54.124 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:54.124 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:54.124 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-objecter_ops": { 2024-05-12T22:19:54.124 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-05-12T22:19:54.124 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 1024, 2024-05-12T22:19:54.124 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-05-12T22:19:54.124 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 0, 2024-05-12T22:19:54.125 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 0, 2024-05-12T22:19:54.125 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-05-12T22:19:54.125 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 0, 2024-05-12T22:19:54.125 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 50, 2024-05-12T22:19:54.125 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 50, 2024-05-12T22:19:54.125 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 50, 2024-05-12T22:19:54.125 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 50, 2024-05-12T22:19:54.125 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-05-12T22:19:54.125 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-12T22:19:54.126 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-12T22:19:54.126 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-12T22:19:54.126 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:54.126 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:54.126 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-write_buf_throttle": { 2024-05-12T22:19:54.126 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-05-12T22:19:54.126 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 3758096384, 2024-05-12T22:19:54.126 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-05-12T22:19:54.127 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 0, 2024-05-12T22:19:54.127 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 0, 2024-05-12T22:19:54.127 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-05-12T22:19:54.127 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 0, 2024-05-12T22:19:54.127 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 0, 2024-05-12T22:19:54.127 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 0, 2024-05-12T22:19:54.128 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 0, 2024-05-12T22:19:54.128 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 0, 2024-05-12T22:19:54.128 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-05-12T22:19:54.128 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-12T22:19:54.128 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-12T22:19:54.128 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-12T22:19:54.128 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:54.128 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:54.129 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-write_buf_throttle-0x55f8057e0500": { 2024-05-12T22:19:54.129 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-05-12T22:19:54.129 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 3758096384, 2024-05-12T22:19:54.129 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-05-12T22:19:54.129 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 11, 2024-05-12T22:19:54.129 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 3026, 2024-05-12T22:19:54.129 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-05-12T22:19:54.129 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 11, 2024-05-12T22:19:54.129 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 0, 2024-05-12T22:19:54.130 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 0, 2024-05-12T22:19:54.130 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 10, 2024-05-12T22:19:54.130 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 3026, 2024-05-12T22:19:54.130 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-05-12T22:19:54.130 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-12T22:19:54.130 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-12T22:19:54.130 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-12T22:19:54.130 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:54.131 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:54.131 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-05-12T22:19:54.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:19:54 smithi028 ceph-mon[29847]: pgmap v338: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-05-12T22:19:54.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:19:54 smithi186 ceph-mon[34785]: pgmap v338: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-05-12T22:19:54.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:19:54 smithi086 ceph-mon[34297]: pgmap v338: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-05-12T22:19:55.100 INFO:tasks.check_counter:Daemon mds.g mds.exported=0 2024-05-12T22:19:55.100 INFO:tasks.check_counter:Daemon mds.g mds.imported=0 2024-05-12T22:19:55.100 INFO:tasks.check_counter:Daemon mds.g mds_cache.dir_update=0 2024-05-12T22:19:55.100 INFO:tasks.check_counter:Daemon mds.g mds_cache.dir_update_receipt=74 2024-05-12T22:19:55.101 INFO:tasks.check_counter:Daemon mds.g mds.root_rsnaps=6 2024-05-12T22:19:55.101 INFO:tasks.check_counter:Daemon mds.g mds_server.req_mksnap_latency.avgcount=0 2024-05-12T22:19:55.101 INFO:tasks.check_counter:Daemon mds.g mds_server.req_rmsnap_latency.avgcount=0 2024-05-12T22:19:55.101 INFO:tasks.check_counter:Daemon mds.g mds.dir_split=0 2024-05-12T22:19:55.101 DEBUG:tasks.check_counter:Getting stats from j 2024-05-12T22:19:55.101 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f72fecff68e1d400c4568684327c900485c20d6a shell --fsid 6b92e7b4-10aa-11ef-bc98-c7b262605968 -- ceph daemon mds.j perf dump 2024-05-12T22:19:55.574 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/mon.a/config 2024-05-12T22:19:56.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:19:56 smithi186 ceph-mon[34785]: pgmap v339: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-12T22:19:56.717 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:19:56 smithi028 ceph-mon[29847]: pgmap v339: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-12T22:19:56.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:19:56 smithi086 ceph-mon[34297]: pgmap v339: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-12T22:19:57.569 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-05-12T22:19:57.569 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-0": { 2024-05-12T22:19:57.569 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 236, 2024-05-12T22:19:57.569 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 224, 2024-05-12T22:19:57.569 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 1722496, 2024-05-12T22:19:57.570 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 138144, 2024-05-12T22:19:57.570 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 4, 2024-05-12T22:19:57.570 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 0, 2024-05-12T22:19:57.570 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.106205976, 2024-05-12T22:19:57.570 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.026572484, 2024-05-12T22:19:57.570 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.093991839, 2024-05-12T22:19:57.570 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.010984347, 2024-05-12T22:19:57.571 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-05-12T22:19:57.571 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 223, 2024-05-12T22:19:57.571 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.022881092, 2024-05-12T22:19:57.571 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000102605 2024-05-12T22:19:57.571 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:57.571 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-05-12T22:19:57.571 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-12T22:19:57.572 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-12T22:19:57.572 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-12T22:19:57.572 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:57.572 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 1722496, 2024-05-12T22:19:57.572 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 138144 2024-05-12T22:19:57.572 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:57.572 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-1": { 2024-05-12T22:19:57.572 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 2, 2024-05-12T22:19:57.573 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 171, 2024-05-12T22:19:57.573 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 320, 2024-05-12T22:19:57.573 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 185696, 2024-05-12T22:19:57.573 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 5, 2024-05-12T22:19:57.573 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 3, 2024-05-12T22:19:57.573 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.035998628, 2024-05-12T22:19:57.573 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.026400648, 2024-05-12T22:19:57.573 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.003740403, 2024-05-12T22:19:57.574 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-05-12T22:19:57.574 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-05-12T22:19:57.574 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 168, 2024-05-12T22:19:57.574 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.021104321, 2024-05-12T22:19:57.574 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000125620 2024-05-12T22:19:57.574 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:57.574 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-05-12T22:19:57.575 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-12T22:19:57.575 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-12T22:19:57.575 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-12T22:19:57.575 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:57.575 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 320, 2024-05-12T22:19:57.575 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 185696 2024-05-12T22:19:57.575 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:57.575 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-2": { 2024-05-12T22:19:57.576 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 4, 2024-05-12T22:19:57.576 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 4, 2024-05-12T22:19:57.576 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 3072, 2024-05-12T22:19:57.576 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 448, 2024-05-12T22:19:57.576 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 2, 2024-05-12T22:19:57.576 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 1, 2024-05-12T22:19:57.576 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.001817738, 2024-05-12T22:19:57.577 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.000059784, 2024-05-12T22:19:57.577 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.001631717, 2024-05-12T22:19:57.577 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-05-12T22:19:57.577 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-05-12T22:19:57.577 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 3, 2024-05-12T22:19:57.577 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.001831894, 2024-05-12T22:19:57.577 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000610631 2024-05-12T22:19:57.578 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:57.578 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-05-12T22:19:57.578 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-12T22:19:57.578 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-12T22:19:57.578 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-12T22:19:57.578 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:57.578 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 3072, 2024-05-12T22:19:57.579 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 448 2024-05-12T22:19:57.579 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:57.579 INFO:teuthology.orchestra.run.smithi028.stdout: "cct": { 2024-05-12T22:19:57.579 INFO:teuthology.orchestra.run.smithi028.stdout: "total_workers": 0, 2024-05-12T22:19:57.579 INFO:teuthology.orchestra.run.smithi028.stdout: "unhealthy_workers": 0 2024-05-12T22:19:57.579 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:57.579 INFO:teuthology.orchestra.run.smithi028.stdout: "mempool": { 2024-05-12T22:19:57.579 INFO:teuthology.orchestra.run.smithi028.stdout: "bloom_filter_bytes": 0, 2024-05-12T22:19:57.580 INFO:teuthology.orchestra.run.smithi028.stdout: "bloom_filter_items": 0, 2024-05-12T22:19:57.580 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_alloc_bytes": 0, 2024-05-12T22:19:57.580 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_alloc_items": 0, 2024-05-12T22:19:57.580 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_data_bytes": 0, 2024-05-12T22:19:57.580 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_data_items": 0, 2024-05-12T22:19:57.580 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_onode_bytes": 0, 2024-05-12T22:19:57.580 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_onode_items": 0, 2024-05-12T22:19:57.581 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_meta_bytes": 0, 2024-05-12T22:19:57.581 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_meta_items": 0, 2024-05-12T22:19:57.581 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_other_bytes": 0, 2024-05-12T22:19:57.581 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_other_items": 0, 2024-05-12T22:19:57.581 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_buffer_bytes": 0, 2024-05-12T22:19:57.581 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_buffer_items": 0, 2024-05-12T22:19:57.581 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_extent_bytes": 0, 2024-05-12T22:19:57.581 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_extent_items": 0, 2024-05-12T22:19:57.582 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_blob_bytes": 0, 2024-05-12T22:19:57.582 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_blob_items": 0, 2024-05-12T22:19:57.582 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_shared_blob_bytes": 0, 2024-05-12T22:19:57.582 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_shared_blob_items": 0, 2024-05-12T22:19:57.582 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_inline_bl_bytes": 0, 2024-05-12T22:19:57.582 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_inline_bl_items": 0, 2024-05-12T22:19:57.582 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_bytes": 0, 2024-05-12T22:19:57.583 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_items": 0, 2024-05-12T22:19:57.583 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_txc_bytes": 0, 2024-05-12T22:19:57.583 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_txc_items": 0, 2024-05-12T22:19:57.583 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_deferred_bytes": 0, 2024-05-12T22:19:57.583 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_deferred_items": 0, 2024-05-12T22:19:57.583 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_bytes": 0, 2024-05-12T22:19:57.583 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_items": 0, 2024-05-12T22:19:57.583 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_bytes": 0, 2024-05-12T22:19:57.584 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_items": 0, 2024-05-12T22:19:57.584 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_reader_bytes": 0, 2024-05-12T22:19:57.584 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_reader_items": 0, 2024-05-12T22:19:57.584 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_writer_bytes": 0, 2024-05-12T22:19:57.584 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_writer_items": 0, 2024-05-12T22:19:57.584 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_anon_bytes": 148714, 2024-05-12T22:19:57.584 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_anon_items": 30, 2024-05-12T22:19:57.584 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_meta_bytes": 0, 2024-05-12T22:19:57.585 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_meta_items": 0, 2024-05-12T22:19:57.585 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_bytes": 0, 2024-05-12T22:19:57.585 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_items": 0, 2024-05-12T22:19:57.585 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mapbl_bytes": 0, 2024-05-12T22:19:57.585 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mapbl_items": 0, 2024-05-12T22:19:57.585 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pglog_bytes": 0, 2024-05-12T22:19:57.586 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pglog_items": 0, 2024-05-12T22:19:57.586 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_bytes": 0, 2024-05-12T22:19:57.586 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_items": 0, 2024-05-12T22:19:57.586 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_mapping_bytes": 0, 2024-05-12T22:19:57.586 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_mapping_items": 0, 2024-05-12T22:19:57.586 INFO:teuthology.orchestra.run.smithi028.stdout: "pgmap_bytes": 0, 2024-05-12T22:19:57.586 INFO:teuthology.orchestra.run.smithi028.stdout: "pgmap_items": 0, 2024-05-12T22:19:57.587 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_co_bytes": 928, 2024-05-12T22:19:57.587 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_co_items": 2, 2024-05-12T22:19:57.587 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_1_bytes": 0, 2024-05-12T22:19:57.587 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_1_items": 0, 2024-05-12T22:19:57.587 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_2_bytes": 0, 2024-05-12T22:19:57.587 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_2_items": 0 2024-05-12T22:19:57.587 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-05-12T22:19:57.588 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-05-12T22:19:57.588 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-05-12T22:19:57.588 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 104857600, 2024-05-12T22:19:57.588 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-05-12T22:19:57.588 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 238, 2024-05-12T22:19:57.588 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 1689239, 2024-05-12T22:19:57.588 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-05-12T22:19:57.589 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 238, 2024-05-12T22:19:57.589 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 0, 2024-05-12T22:19:57.589 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 0, 2024-05-12T22:19:57.589 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 238, 2024-05-12T22:19:57.589 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 1689239, 2024-05-12T22:19:57.590 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-05-12T22:19:57.590 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-05-12T22:19:57.590 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-05-12T22:19:57.590 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-05-12T22:19:57.590 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:57.590 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-05-12T22:19:57.591 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-05-12T22:19:58.404 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.j 2024-05-12T22:19:58.404 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.j 2024-05-12T22:19:58.404 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.j 2024-05-12T22:19:58.404 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.j 2024-05-12T22:19:58.404 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.j 2024-05-12T22:19:58.405 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.j 2024-05-12T22:19:58.405 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.j 2024-05-12T22:19:58.405 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.j 2024-05-12T22:19:58.405 DEBUG:tasks.check_counter:Getting stats from b 2024-05-12T22:19:58.405 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f72fecff68e1d400c4568684327c900485c20d6a shell --fsid 6b92e7b4-10aa-11ef-bc98-c7b262605968 -- ceph daemon mds.b perf dump 2024-05-12T22:19:58.658 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/mon.b/config 2024-05-12T22:19:58.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:19:58 smithi186 ceph-mon[34785]: pgmap v340: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:19:58.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:19:58 smithi086 ceph-mon[34297]: pgmap v340: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:19:58.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:19:58 smithi028 ceph-mon[29847]: pgmap v340: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:20:00.439 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:20:00 smithi086 ceph-mon[34297]: pgmap v341: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:20:00.439 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:20:00 smithi086 ceph-mon[34297]: overall HEALTH_OK 2024-05-12T22:20:00.564 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-05-12T22:20:00.565 INFO:teuthology.orchestra.run.smithi086.stdout: "AsyncMessenger::Worker-0": { 2024-05-12T22:20:00.565 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_recv_messages": 22677, 2024-05-12T22:20:00.565 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_send_messages": 22374, 2024-05-12T22:20:00.565 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_recv_bytes": 6078167, 2024-05-12T22:20:00.565 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_send_bytes": 47895151, 2024-05-12T22:20:00.565 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_created_connections": 23, 2024-05-12T22:20:00.565 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_active_connections": 4, 2024-05-12T22:20:00.566 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_running_total_time": 1.434352910, 2024-05-12T22:20:00.566 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_running_send_time": 0.476589480, 2024-05-12T22:20:00.566 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_running_recv_time": 0.320324633, 2024-05-12T22:20:00.566 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_running_fast_dispatch_time": 0.556901137, 2024-05-12T22:20:00.566 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_send_messages_queue_lat": { 2024-05-12T22:20:00.566 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 22376, 2024-05-12T22:20:00.566 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.542698483, 2024-05-12T22:20:00.566 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000024253 2024-05-12T22:20:00.567 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:00.567 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_handle_ack_lat": { 2024-05-12T22:20:00.567 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 3224, 2024-05-12T22:20:00.567 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000623658, 2024-05-12T22:20:00.567 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000193 2024-05-12T22:20:00.567 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:00.567 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_recv_encrypted_bytes": 360496, 2024-05-12T22:20:00.567 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_send_encrypted_bytes": 98112 2024-05-12T22:20:00.567 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:00.568 INFO:teuthology.orchestra.run.smithi086.stdout: "AsyncMessenger::Worker-1": { 2024-05-12T22:20:00.568 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_recv_messages": 23495, 2024-05-12T22:20:00.568 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_send_messages": 23469, 2024-05-12T22:20:00.568 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_recv_bytes": 8040932, 2024-05-12T22:20:00.568 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_send_bytes": 22133550, 2024-05-12T22:20:00.568 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_created_connections": 155, 2024-05-12T22:20:00.568 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_active_connections": 6, 2024-05-12T22:20:00.568 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_running_total_time": 1.239110222, 2024-05-12T22:20:00.568 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_running_send_time": 0.340607695, 2024-05-12T22:20:00.569 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_running_recv_time": 0.327582884, 2024-05-12T22:20:00.569 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_running_fast_dispatch_time": 0.509293279, 2024-05-12T22:20:00.569 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_send_messages_queue_lat": { 2024-05-12T22:20:00.569 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 23468, 2024-05-12T22:20:00.569 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 1.119447370, 2024-05-12T22:20:00.569 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000047701 2024-05-12T22:20:00.569 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:00.569 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_handle_ack_lat": { 2024-05-12T22:20:00.569 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 408, 2024-05-12T22:20:00.570 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000056811, 2024-05-12T22:20:00.570 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000139 2024-05-12T22:20:00.570 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:00.570 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_recv_encrypted_bytes": 1876400, 2024-05-12T22:20:00.570 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_send_encrypted_bytes": 527584 2024-05-12T22:20:00.570 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:00.570 INFO:teuthology.orchestra.run.smithi086.stdout: "AsyncMessenger::Worker-2": { 2024-05-12T22:20:00.570 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_recv_messages": 40516, 2024-05-12T22:20:00.571 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_send_messages": 59018, 2024-05-12T22:20:00.571 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_recv_bytes": 12315521, 2024-05-12T22:20:00.571 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_send_bytes": 32819197, 2024-05-12T22:20:00.571 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_created_connections": 11, 2024-05-12T22:20:00.571 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_active_connections": 7, 2024-05-12T22:20:00.571 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_running_total_time": 2.181782579, 2024-05-12T22:20:00.571 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_running_send_time": 0.918121357, 2024-05-12T22:20:00.571 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_running_recv_time": 1.269233817, 2024-05-12T22:20:00.571 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_running_fast_dispatch_time": 0.457128768, 2024-05-12T22:20:00.572 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_send_messages_queue_lat": { 2024-05-12T22:20:00.572 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 59012, 2024-05-12T22:20:00.572 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 1.333193850, 2024-05-12T22:20:00.572 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000022591 2024-05-12T22:20:00.572 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:00.572 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_handle_ack_lat": { 2024-05-12T22:20:00.572 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 32334, 2024-05-12T22:20:00.572 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.007600232, 2024-05-12T22:20:00.572 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000235 2024-05-12T22:20:00.573 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:00.573 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_recv_encrypted_bytes": 42720, 2024-05-12T22:20:00.573 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_send_encrypted_bytes": 138640 2024-05-12T22:20:00.573 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:00.573 INFO:teuthology.orchestra.run.smithi086.stdout: "cct": { 2024-05-12T22:20:00.573 INFO:teuthology.orchestra.run.smithi086.stdout: "total_workers": 1, 2024-05-12T22:20:00.573 INFO:teuthology.orchestra.run.smithi086.stdout: "unhealthy_workers": 0 2024-05-12T22:20:00.573 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:00.573 INFO:teuthology.orchestra.run.smithi086.stdout: "finisher-MDSRank": { 2024-05-12T22:20:00.574 INFO:teuthology.orchestra.run.smithi086.stdout: "queue_len": 0, 2024-05-12T22:20:00.574 INFO:teuthology.orchestra.run.smithi086.stdout: "complete_latency": { 2024-05-12T22:20:00.574 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 40369, 2024-05-12T22:20:00.574 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 20.305010293, 2024-05-12T22:20:00.574 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000502985 2024-05-12T22:20:00.574 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-05-12T22:20:00.574 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:00.574 INFO:teuthology.orchestra.run.smithi086.stdout: "finisher-PurgeQueue": { 2024-05-12T22:20:00.574 INFO:teuthology.orchestra.run.smithi086.stdout: "queue_len": 0, 2024-05-12T22:20:00.575 INFO:teuthology.orchestra.run.smithi086.stdout: "complete_latency": { 2024-05-12T22:20:00.575 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 5617, 2024-05-12T22:20:00.575 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 1.324409534, 2024-05-12T22:20:00.575 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000235785 2024-05-12T22:20:00.575 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-05-12T22:20:00.575 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:00.575 INFO:teuthology.orchestra.run.smithi086.stdout: "mds": { 2024-05-12T22:20:00.575 INFO:teuthology.orchestra.run.smithi086.stdout: "request": 10828, 2024-05-12T22:20:00.575 INFO:teuthology.orchestra.run.smithi086.stdout: "reply": 10827, 2024-05-12T22:20:00.576 INFO:teuthology.orchestra.run.smithi086.stdout: "reply_latency": { 2024-05-12T22:20:00.576 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 10827, 2024-05-12T22:20:00.576 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 36.326763081, 2024-05-12T22:20:00.576 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.003355201 2024-05-12T22:20:00.576 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:00.576 INFO:teuthology.orchestra.run.smithi086.stdout: "slow_reply": 0, 2024-05-12T22:20:00.576 INFO:teuthology.orchestra.run.smithi086.stdout: "forward": 1, 2024-05-12T22:20:00.576 INFO:teuthology.orchestra.run.smithi086.stdout: "dir_fetch_complete": 0, 2024-05-12T22:20:00.577 INFO:teuthology.orchestra.run.smithi086.stdout: "dir_fetch_keys": 0, 2024-05-12T22:20:00.577 INFO:teuthology.orchestra.run.smithi086.stdout: "dir_commit": 243, 2024-05-12T22:20:00.577 INFO:teuthology.orchestra.run.smithi086.stdout: "dir_split": 28, 2024-05-12T22:20:00.577 INFO:teuthology.orchestra.run.smithi086.stdout: "dir_merge": 1, 2024-05-12T22:20:00.577 INFO:teuthology.orchestra.run.smithi086.stdout: "inodes": 9412, 2024-05-12T22:20:00.577 INFO:teuthology.orchestra.run.smithi086.stdout: "inodes_top": 870, 2024-05-12T22:20:00.577 INFO:teuthology.orchestra.run.smithi086.stdout: "inodes_bottom": 8467, 2024-05-12T22:20:00.577 INFO:teuthology.orchestra.run.smithi086.stdout: "inodes_pin_tail": 75, 2024-05-12T22:20:00.577 INFO:teuthology.orchestra.run.smithi086.stdout: "inodes_pinned": 8169, 2024-05-12T22:20:00.578 INFO:teuthology.orchestra.run.smithi086.stdout: "inodes_expired": 13, 2024-05-12T22:20:00.578 INFO:teuthology.orchestra.run.smithi086.stdout: "inodes_with_caps": 3514, 2024-05-12T22:20:00.578 INFO:teuthology.orchestra.run.smithi086.stdout: "caps": 3514, 2024-05-12T22:20:00.578 INFO:teuthology.orchestra.run.smithi086.stdout: "subtrees": 7, 2024-05-12T22:20:00.578 INFO:teuthology.orchestra.run.smithi086.stdout: "traverse": 11086, 2024-05-12T22:20:00.578 INFO:teuthology.orchestra.run.smithi086.stdout: "traverse_hit": 10540, 2024-05-12T22:20:00.578 INFO:teuthology.orchestra.run.smithi086.stdout: "traverse_forward": 1, 2024-05-12T22:20:00.578 INFO:teuthology.orchestra.run.smithi086.stdout: "traverse_discover": 0, 2024-05-12T22:20:00.578 INFO:teuthology.orchestra.run.smithi086.stdout: "traverse_dir_fetch": 0, 2024-05-12T22:20:00.579 INFO:teuthology.orchestra.run.smithi086.stdout: "traverse_remote_ino": 0, 2024-05-12T22:20:00.579 INFO:teuthology.orchestra.run.smithi086.stdout: "traverse_lock": 3, 2024-05-12T22:20:00.579 INFO:teuthology.orchestra.run.smithi086.stdout: "load_cent": 0, 2024-05-12T22:20:00.579 INFO:teuthology.orchestra.run.smithi086.stdout: "q": 0, 2024-05-12T22:20:00.579 INFO:teuthology.orchestra.run.smithi086.stdout: "exported": 3, 2024-05-12T22:20:00.579 INFO:teuthology.orchestra.run.smithi086.stdout: "exported_inodes": 10, 2024-05-12T22:20:00.579 INFO:teuthology.orchestra.run.smithi086.stdout: "imported": 0, 2024-05-12T22:20:00.579 INFO:teuthology.orchestra.run.smithi086.stdout: "imported_inodes": 0, 2024-05-12T22:20:00.579 INFO:teuthology.orchestra.run.smithi086.stdout: "openino_dir_fetch": 0, 2024-05-12T22:20:00.580 INFO:teuthology.orchestra.run.smithi086.stdout: "openino_backtrace_fetch": 54157, 2024-05-12T22:20:00.580 INFO:teuthology.orchestra.run.smithi086.stdout: "openino_peer_discover": 0, 2024-05-12T22:20:00.580 INFO:teuthology.orchestra.run.smithi086.stdout: "root_rfiles": 3454, 2024-05-12T22:20:00.580 INFO:teuthology.orchestra.run.smithi086.stdout: "root_rbytes": 26743557, 2024-05-12T22:20:00.580 INFO:teuthology.orchestra.run.smithi086.stdout: "root_rsnaps": 6, 2024-05-12T22:20:00.580 INFO:teuthology.orchestra.run.smithi086.stdout: "scrub_backtrace_fetch": 54157, 2024-05-12T22:20:00.580 INFO:teuthology.orchestra.run.smithi086.stdout: "scrub_set_tag": 0, 2024-05-12T22:20:00.580 INFO:teuthology.orchestra.run.smithi086.stdout: "scrub_backtrace_repaired": 0, 2024-05-12T22:20:00.581 INFO:teuthology.orchestra.run.smithi086.stdout: "scrub_inotable_repaired": 0, 2024-05-12T22:20:00.581 INFO:teuthology.orchestra.run.smithi086.stdout: "scrub_dir_inodes": 1211, 2024-05-12T22:20:00.581 INFO:teuthology.orchestra.run.smithi086.stdout: "scrub_dir_base_inodes": 19, 2024-05-12T22:20:00.581 INFO:teuthology.orchestra.run.smithi086.stdout: "scrub_dirfrag_rstats": 1192, 2024-05-12T22:20:00.581 INFO:teuthology.orchestra.run.smithi086.stdout: "scrub_file_inodes": 52946, 2024-05-12T22:20:00.581 INFO:teuthology.orchestra.run.smithi086.stdout: "handle_inode_file_caps": 1, 2024-05-12T22:20:00.581 INFO:teuthology.orchestra.run.smithi086.stdout: "ceph_cap_op_revoke": 332, 2024-05-12T22:20:00.581 INFO:teuthology.orchestra.run.smithi086.stdout: "ceph_cap_op_grant": 11425, 2024-05-12T22:20:00.582 INFO:teuthology.orchestra.run.smithi086.stdout: "ceph_cap_op_trunc": 8, 2024-05-12T22:20:00.582 INFO:teuthology.orchestra.run.smithi086.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-05-12T22:20:00.582 INFO:teuthology.orchestra.run.smithi086.stdout: "ceph_cap_op_flush_ack": 0, 2024-05-12T22:20:00.582 INFO:teuthology.orchestra.run.smithi086.stdout: "handle_client_caps": 12149, 2024-05-12T22:20:00.582 INFO:teuthology.orchestra.run.smithi086.stdout: "handle_client_caps_dirty": 9854, 2024-05-12T22:20:00.582 INFO:teuthology.orchestra.run.smithi086.stdout: "handle_client_cap_release": 1120, 2024-05-12T22:20:00.582 INFO:teuthology.orchestra.run.smithi086.stdout: "process_request_cap_release": 3563 2024-05-12T22:20:00.582 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:00.583 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_cache": { 2024-05-12T22:20:00.583 INFO:teuthology.orchestra.run.smithi086.stdout: "dir_update": 50, 2024-05-12T22:20:00.583 INFO:teuthology.orchestra.run.smithi086.stdout: "dir_update_receipt": 0, 2024-05-12T22:20:00.583 INFO:teuthology.orchestra.run.smithi086.stdout: "dir_try_discover": 0, 2024-05-12T22:20:00.583 INFO:teuthology.orchestra.run.smithi086.stdout: "dir_send_discover": 0, 2024-05-12T22:20:00.583 INFO:teuthology.orchestra.run.smithi086.stdout: "dir_handle_discover": 97, 2024-05-12T22:20:00.583 INFO:teuthology.orchestra.run.smithi086.stdout: "num_strays": 5, 2024-05-12T22:20:00.583 INFO:teuthology.orchestra.run.smithi086.stdout: "num_strays_delayed": 0, 2024-05-12T22:20:00.583 INFO:teuthology.orchestra.run.smithi086.stdout: "num_strays_enqueuing": 0, 2024-05-12T22:20:00.584 INFO:teuthology.orchestra.run.smithi086.stdout: "strays_created": 3267, 2024-05-12T22:20:00.584 INFO:teuthology.orchestra.run.smithi086.stdout: "strays_enqueued": 3259, 2024-05-12T22:20:00.584 INFO:teuthology.orchestra.run.smithi086.stdout: "strays_reintegrated": 3, 2024-05-12T22:20:00.584 INFO:teuthology.orchestra.run.smithi086.stdout: "strays_migrated": 0, 2024-05-12T22:20:00.584 INFO:teuthology.orchestra.run.smithi086.stdout: "num_recovering_processing": 0, 2024-05-12T22:20:00.584 INFO:teuthology.orchestra.run.smithi086.stdout: "num_recovering_enqueued": 0, 2024-05-12T22:20:00.584 INFO:teuthology.orchestra.run.smithi086.stdout: "num_recovering_prioritized": 0, 2024-05-12T22:20:00.584 INFO:teuthology.orchestra.run.smithi086.stdout: "recovery_started": 0, 2024-05-12T22:20:00.585 INFO:teuthology.orchestra.run.smithi086.stdout: "recovery_completed": 0, 2024-05-12T22:20:00.585 INFO:teuthology.orchestra.run.smithi086.stdout: "ireq_quiesce_path": 0, 2024-05-12T22:20:00.585 INFO:teuthology.orchestra.run.smithi086.stdout: "ireq_quiesce_inode": 0, 2024-05-12T22:20:00.585 INFO:teuthology.orchestra.run.smithi086.stdout: "ireq_enqueue_scrub": 19, 2024-05-12T22:20:00.585 INFO:teuthology.orchestra.run.smithi086.stdout: "ireq_exportdir": 3, 2024-05-12T22:20:00.585 INFO:teuthology.orchestra.run.smithi086.stdout: "ireq_flush": 0, 2024-05-12T22:20:00.585 INFO:teuthology.orchestra.run.smithi086.stdout: "ireq_fragmentdir": 29, 2024-05-12T22:20:00.585 INFO:teuthology.orchestra.run.smithi086.stdout: "ireq_fragstats": 0, 2024-05-12T22:20:00.585 INFO:teuthology.orchestra.run.smithi086.stdout: "ireq_inodestats": 0 2024-05-12T22:20:00.586 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:00.586 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_log": { 2024-05-12T22:20:00.586 INFO:teuthology.orchestra.run.smithi086.stdout: "evlrg": 0, 2024-05-12T22:20:00.586 INFO:teuthology.orchestra.run.smithi086.stdout: "evadd": 26730, 2024-05-12T22:20:00.586 INFO:teuthology.orchestra.run.smithi086.stdout: "evex": 0, 2024-05-12T22:20:00.586 INFO:teuthology.orchestra.run.smithi086.stdout: "evtrm": 0, 2024-05-12T22:20:00.586 INFO:teuthology.orchestra.run.smithi086.stdout: "ev": 26730, 2024-05-12T22:20:00.586 INFO:teuthology.orchestra.run.smithi086.stdout: "evexg": 0, 2024-05-12T22:20:00.587 INFO:teuthology.orchestra.run.smithi086.stdout: "evexd": 0, 2024-05-12T22:20:00.587 INFO:teuthology.orchestra.run.smithi086.stdout: "segadd": 37, 2024-05-12T22:20:00.587 INFO:teuthology.orchestra.run.smithi086.stdout: "segex": 0, 2024-05-12T22:20:00.587 INFO:teuthology.orchestra.run.smithi086.stdout: "segtrm": 0, 2024-05-12T22:20:00.587 INFO:teuthology.orchestra.run.smithi086.stdout: "seg": 37, 2024-05-12T22:20:00.587 INFO:teuthology.orchestra.run.smithi086.stdout: "segmjr": 4, 2024-05-12T22:20:00.587 INFO:teuthology.orchestra.run.smithi086.stdout: "segexg": 0, 2024-05-12T22:20:00.587 INFO:teuthology.orchestra.run.smithi086.stdout: "segexd": 0, 2024-05-12T22:20:00.587 INFO:teuthology.orchestra.run.smithi086.stdout: "expos": 4194304, 2024-05-12T22:20:00.588 INFO:teuthology.orchestra.run.smithi086.stdout: "wrpos": 71446379, 2024-05-12T22:20:00.588 INFO:teuthology.orchestra.run.smithi086.stdout: "rdpos": 0, 2024-05-12T22:20:00.588 INFO:teuthology.orchestra.run.smithi086.stdout: "jlat": { 2024-05-12T22:20:00.588 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 452, 2024-05-12T22:20:00.588 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 23.215874656, 2024-05-12T22:20:00.588 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.051362554 2024-05-12T22:20:00.588 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:00.588 INFO:teuthology.orchestra.run.smithi086.stdout: "replayed": 0 2024-05-12T22:20:00.588 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:00.589 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_mem": { 2024-05-12T22:20:00.589 INFO:teuthology.orchestra.run.smithi086.stdout: "ino": 7237, 2024-05-12T22:20:00.589 INFO:teuthology.orchestra.run.smithi086.stdout: "ino+": 10522, 2024-05-12T22:20:00.589 INFO:teuthology.orchestra.run.smithi086.stdout: "ino-": 3285, 2024-05-12T22:20:00.589 INFO:teuthology.orchestra.run.smithi086.stdout: "dir": 281, 2024-05-12T22:20:00.589 INFO:teuthology.orchestra.run.smithi086.stdout: "dir+": 322, 2024-05-12T22:20:00.589 INFO:teuthology.orchestra.run.smithi086.stdout: "dir-": 41, 2024-05-12T22:20:00.589 INFO:teuthology.orchestra.run.smithi086.stdout: "dn": 9412, 2024-05-12T22:20:00.590 INFO:teuthology.orchestra.run.smithi086.stdout: "dn+": 12156, 2024-05-12T22:20:00.590 INFO:teuthology.orchestra.run.smithi086.stdout: "dn-": 2744, 2024-05-12T22:20:00.590 INFO:teuthology.orchestra.run.smithi086.stdout: "cap": 3514, 2024-05-12T22:20:00.590 INFO:teuthology.orchestra.run.smithi086.stdout: "cap+": 6798, 2024-05-12T22:20:00.590 INFO:teuthology.orchestra.run.smithi086.stdout: "cap-": 3284, 2024-05-12T22:20:00.590 INFO:teuthology.orchestra.run.smithi086.stdout: "rss": 113632, 2024-05-12T22:20:00.590 INFO:teuthology.orchestra.run.smithi086.stdout: "heap": 231804 2024-05-12T22:20:00.590 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:00.591 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_server": { 2024-05-12T22:20:00.591 INFO:teuthology.orchestra.run.smithi086.stdout: "dispatch_client_request": 11159, 2024-05-12T22:20:00.591 INFO:teuthology.orchestra.run.smithi086.stdout: "dispatch_server_request": 6, 2024-05-12T22:20:00.591 INFO:teuthology.orchestra.run.smithi086.stdout: "handle_client_request": 10828, 2024-05-12T22:20:00.591 INFO:teuthology.orchestra.run.smithi086.stdout: "handle_client_session": 40, 2024-05-12T22:20:00.591 INFO:teuthology.orchestra.run.smithi086.stdout: "handle_peer_request": 12, 2024-05-12T22:20:00.591 INFO:teuthology.orchestra.run.smithi086.stdout: "req_create_latency": { 2024-05-12T22:20:00.591 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 6700, 2024-05-12T22:20:00.591 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 23.812550706, 2024-05-12T22:20:00.592 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.003554112 2024-05-12T22:20:00.592 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:00.592 INFO:teuthology.orchestra.run.smithi086.stdout: "req_getattr_latency": { 2024-05-12T22:20:00.592 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 112, 2024-05-12T22:20:00.592 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.032149913, 2024-05-12T22:20:00.592 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000287052 2024-05-12T22:20:00.592 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:00.592 INFO:teuthology.orchestra.run.smithi086.stdout: "req_getfilelock_latency": { 2024-05-12T22:20:00.593 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-05-12T22:20:00.593 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-05-12T22:20:00.593 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-05-12T22:20:00.593 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:00.593 INFO:teuthology.orchestra.run.smithi086.stdout: "req_link_latency": { 2024-05-12T22:20:00.593 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 3, 2024-05-12T22:20:00.593 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.003765544, 2024-05-12T22:20:00.593 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.001255181 2024-05-12T22:20:00.593 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:00.594 INFO:teuthology.orchestra.run.smithi086.stdout: "req_lookup_latency": { 2024-05-12T22:20:00.594 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 5, 2024-05-12T22:20:00.594 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.001360879, 2024-05-12T22:20:00.594 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000272175 2024-05-12T22:20:00.594 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:00.594 INFO:teuthology.orchestra.run.smithi086.stdout: "req_lookuphash_latency": { 2024-05-12T22:20:00.594 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-05-12T22:20:00.594 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-05-12T22:20:00.595 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-05-12T22:20:00.595 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:00.595 INFO:teuthology.orchestra.run.smithi086.stdout: "req_lookupino_latency": { 2024-05-12T22:20:00.595 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 1, 2024-05-12T22:20:00.595 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000389145, 2024-05-12T22:20:00.595 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000389145 2024-05-12T22:20:00.595 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:00.595 INFO:teuthology.orchestra.run.smithi086.stdout: "req_lookupname_latency": { 2024-05-12T22:20:00.595 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-05-12T22:20:00.596 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-05-12T22:20:00.596 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-05-12T22:20:00.596 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:00.596 INFO:teuthology.orchestra.run.smithi086.stdout: "req_lookupparent_latency": { 2024-05-12T22:20:00.596 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-05-12T22:20:00.596 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-05-12T22:20:00.596 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-05-12T22:20:00.596 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:00.596 INFO:teuthology.orchestra.run.smithi086.stdout: "req_lookupsnap_latency": { 2024-05-12T22:20:00.597 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-05-12T22:20:00.597 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-05-12T22:20:00.597 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-05-12T22:20:00.597 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:00.598 INFO:teuthology.orchestra.run.smithi086.stdout: "req_lssnap_latency": { 2024-05-12T22:20:00.598 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 11, 2024-05-12T22:20:00.598 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.002583301, 2024-05-12T22:20:00.598 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000234845 2024-05-12T22:20:00.598 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:00.598 INFO:teuthology.orchestra.run.smithi086.stdout: "req_mkdir_latency": { 2024-05-12T22:20:00.599 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 85, 2024-05-12T22:20:00.599 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.166353508, 2024-05-12T22:20:00.599 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.001957100 2024-05-12T22:20:00.599 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:00.599 INFO:teuthology.orchestra.run.smithi086.stdout: "req_mknod_latency": { 2024-05-12T22:20:00.599 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-05-12T22:20:00.599 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-05-12T22:20:00.599 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-05-12T22:20:00.599 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:00.600 INFO:teuthology.orchestra.run.smithi086.stdout: "req_mksnap_latency": { 2024-05-12T22:20:00.600 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 11, 2024-05-12T22:20:00.600 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.112916588, 2024-05-12T22:20:00.600 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.010265144 2024-05-12T22:20:00.600 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:00.600 INFO:teuthology.orchestra.run.smithi086.stdout: "req_open_latency": { 2024-05-12T22:20:00.600 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 442, 2024-05-12T22:20:00.600 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.106866920, 2024-05-12T22:20:00.600 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000241780 2024-05-12T22:20:00.601 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:00.601 INFO:teuthology.orchestra.run.smithi086.stdout: "req_readdir_latency": { 2024-05-12T22:20:00.601 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 108, 2024-05-12T22:20:00.601 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.719881086, 2024-05-12T22:20:00.601 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.006665565 2024-05-12T22:20:00.601 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:00.601 INFO:teuthology.orchestra.run.smithi086.stdout: "req_rename_latency": { 2024-05-12T22:20:00.601 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 82, 2024-05-12T22:20:00.601 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.174115171, 2024-05-12T22:20:00.602 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.002123355 2024-05-12T22:20:00.602 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:00.602 INFO:teuthology.orchestra.run.smithi086.stdout: "req_renamesnap_latency": { 2024-05-12T22:20:00.602 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-05-12T22:20:00.602 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-05-12T22:20:00.602 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-05-12T22:20:00.602 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:00.602 INFO:teuthology.orchestra.run.smithi086.stdout: "req_snapdiff_latency": { 2024-05-12T22:20:00.603 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-05-12T22:20:00.603 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-05-12T22:20:00.603 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-05-12T22:20:00.603 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:00.603 INFO:teuthology.orchestra.run.smithi086.stdout: "req_rmdir_latency": { 2024-05-12T22:20:00.603 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 22, 2024-05-12T22:20:00.603 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.741486757, 2024-05-12T22:20:00.603 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.033703943 2024-05-12T22:20:00.603 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:00.604 INFO:teuthology.orchestra.run.smithi086.stdout: "req_rmsnap_latency": { 2024-05-12T22:20:00.604 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 5, 2024-05-12T22:20:00.604 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.039132290, 2024-05-12T22:20:00.604 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.007826458 2024-05-12T22:20:00.604 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:00.604 INFO:teuthology.orchestra.run.smithi086.stdout: "req_rmxattr_latency": { 2024-05-12T22:20:00.604 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-05-12T22:20:00.604 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-05-12T22:20:00.604 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-05-12T22:20:00.605 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:00.605 INFO:teuthology.orchestra.run.smithi086.stdout: "req_setattr_latency": { 2024-05-12T22:20:00.605 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 9, 2024-05-12T22:20:00.605 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.086940874, 2024-05-12T22:20:00.605 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.009660097 2024-05-12T22:20:00.605 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:00.605 INFO:teuthology.orchestra.run.smithi086.stdout: "req_setdirlayout_latency": { 2024-05-12T22:20:00.605 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-05-12T22:20:00.606 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-05-12T22:20:00.606 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-05-12T22:20:00.606 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:00.606 INFO:teuthology.orchestra.run.smithi086.stdout: "req_setfilelock_latency": { 2024-05-12T22:20:00.606 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 8, 2024-05-12T22:20:00.606 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.002785185, 2024-05-12T22:20:00.606 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000348148 2024-05-12T22:20:00.606 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:00.606 INFO:teuthology.orchestra.run.smithi086.stdout: "req_setlayout_latency": { 2024-05-12T22:20:00.607 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-05-12T22:20:00.607 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-05-12T22:20:00.607 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-05-12T22:20:00.607 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:00.607 INFO:teuthology.orchestra.run.smithi086.stdout: "req_setxattr_latency": { 2024-05-12T22:20:00.607 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 8, 2024-05-12T22:20:00.607 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.020547151, 2024-05-12T22:20:00.607 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.002568393 2024-05-12T22:20:00.607 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:00.608 INFO:teuthology.orchestra.run.smithi086.stdout: "req_symlink_latency": { 2024-05-12T22:20:00.608 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 7, 2024-05-12T22:20:00.608 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.006363835, 2024-05-12T22:20:00.608 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000909119 2024-05-12T22:20:00.608 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:00.608 INFO:teuthology.orchestra.run.smithi086.stdout: "req_unlink_latency": { 2024-05-12T22:20:00.608 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 3208, 2024-05-12T22:20:00.608 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 10.296574228, 2024-05-12T22:20:00.608 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.003209655 2024-05-12T22:20:00.609 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:00.609 INFO:teuthology.orchestra.run.smithi086.stdout: "cap_revoke_eviction": 0, 2024-05-12T22:20:00.609 INFO:teuthology.orchestra.run.smithi086.stdout: "cap_acquisition_throttle": 0, 2024-05-12T22:20:00.609 INFO:teuthology.orchestra.run.smithi086.stdout: "req_getvxattr_latency": { 2024-05-12T22:20:00.609 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-05-12T22:20:00.609 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-05-12T22:20:00.609 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-05-12T22:20:00.609 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-05-12T22:20:00.610 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:00.610 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_sessions": { 2024-05-12T22:20:00.610 INFO:teuthology.orchestra.run.smithi086.stdout: "session_count": 2, 2024-05-12T22:20:00.610 INFO:teuthology.orchestra.run.smithi086.stdout: "session_add": 6, 2024-05-12T22:20:00.610 INFO:teuthology.orchestra.run.smithi086.stdout: "session_remove": 4, 2024-05-12T22:20:00.610 INFO:teuthology.orchestra.run.smithi086.stdout: "sessions_open": 2, 2024-05-12T22:20:00.610 INFO:teuthology.orchestra.run.smithi086.stdout: "sessions_stale": 0, 2024-05-12T22:20:00.610 INFO:teuthology.orchestra.run.smithi086.stdout: "total_load": 49, 2024-05-12T22:20:00.610 INFO:teuthology.orchestra.run.smithi086.stdout: "average_load": 24, 2024-05-12T22:20:00.611 INFO:teuthology.orchestra.run.smithi086.stdout: "avg_session_uptime": 628, 2024-05-12T22:20:00.611 INFO:teuthology.orchestra.run.smithi086.stdout: "mdthresh_evicted": 0 2024-05-12T22:20:00.611 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:00.611 INFO:teuthology.orchestra.run.smithi086.stdout: "mempool": { 2024-05-12T22:20:00.611 INFO:teuthology.orchestra.run.smithi086.stdout: "bloom_filter_bytes": 0, 2024-05-12T22:20:00.611 INFO:teuthology.orchestra.run.smithi086.stdout: "bloom_filter_items": 0, 2024-05-12T22:20:00.611 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_alloc_bytes": 0, 2024-05-12T22:20:00.611 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_alloc_items": 0, 2024-05-12T22:20:00.611 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_cache_data_bytes": 0, 2024-05-12T22:20:00.612 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_cache_data_items": 0, 2024-05-12T22:20:00.612 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_cache_onode_bytes": 0, 2024-05-12T22:20:00.612 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_cache_onode_items": 0, 2024-05-12T22:20:00.612 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_cache_meta_bytes": 0, 2024-05-12T22:20:00.612 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_cache_meta_items": 0, 2024-05-12T22:20:00.612 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_cache_other_bytes": 0, 2024-05-12T22:20:00.612 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_cache_other_items": 0, 2024-05-12T22:20:00.612 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_cache_buffer_bytes": 0, 2024-05-12T22:20:00.612 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_cache_buffer_items": 0, 2024-05-12T22:20:00.613 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_extent_bytes": 0, 2024-05-12T22:20:00.613 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_extent_items": 0, 2024-05-12T22:20:00.613 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_blob_bytes": 0, 2024-05-12T22:20:00.613 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_blob_items": 0, 2024-05-12T22:20:00.613 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_shared_blob_bytes": 0, 2024-05-12T22:20:00.613 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_shared_blob_items": 0, 2024-05-12T22:20:00.613 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_inline_bl_bytes": 0, 2024-05-12T22:20:00.613 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_inline_bl_items": 0, 2024-05-12T22:20:00.613 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_fsck_bytes": 0, 2024-05-12T22:20:00.614 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_fsck_items": 0, 2024-05-12T22:20:00.614 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_txc_bytes": 0, 2024-05-12T22:20:00.614 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_txc_items": 0, 2024-05-12T22:20:00.614 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_writing_deferred_bytes": 0, 2024-05-12T22:20:00.614 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_writing_deferred_items": 0, 2024-05-12T22:20:00.614 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_writing_bytes": 0, 2024-05-12T22:20:00.614 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_writing_items": 0, 2024-05-12T22:20:00.614 INFO:teuthology.orchestra.run.smithi086.stdout: "bluefs_bytes": 0, 2024-05-12T22:20:00.615 INFO:teuthology.orchestra.run.smithi086.stdout: "bluefs_items": 0, 2024-05-12T22:20:00.615 INFO:teuthology.orchestra.run.smithi086.stdout: "bluefs_file_reader_bytes": 0, 2024-05-12T22:20:00.615 INFO:teuthology.orchestra.run.smithi086.stdout: "bluefs_file_reader_items": 0, 2024-05-12T22:20:00.615 INFO:teuthology.orchestra.run.smithi086.stdout: "bluefs_file_writer_bytes": 0, 2024-05-12T22:20:00.615 INFO:teuthology.orchestra.run.smithi086.stdout: "bluefs_file_writer_items": 0, 2024-05-12T22:20:00.615 INFO:teuthology.orchestra.run.smithi086.stdout: "buffer_anon_bytes": 555993, 2024-05-12T22:20:00.615 INFO:teuthology.orchestra.run.smithi086.stdout: "buffer_anon_items": 6215, 2024-05-12T22:20:00.615 INFO:teuthology.orchestra.run.smithi086.stdout: "buffer_meta_bytes": 1408, 2024-05-12T22:20:00.616 INFO:teuthology.orchestra.run.smithi086.stdout: "buffer_meta_items": 16, 2024-05-12T22:20:00.616 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_bytes": 0, 2024-05-12T22:20:00.616 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_items": 0, 2024-05-12T22:20:00.616 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_mapbl_bytes": 0, 2024-05-12T22:20:00.616 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_mapbl_items": 0, 2024-05-12T22:20:00.616 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_pglog_bytes": 0, 2024-05-12T22:20:00.616 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_pglog_items": 0, 2024-05-12T22:20:00.616 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap_bytes": 8032, 2024-05-12T22:20:00.616 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap_items": 167, 2024-05-12T22:20:00.617 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap_mapping_bytes": 0, 2024-05-12T22:20:00.617 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap_mapping_items": 0, 2024-05-12T22:20:00.617 INFO:teuthology.orchestra.run.smithi086.stdout: "pgmap_bytes": 0, 2024-05-12T22:20:00.617 INFO:teuthology.orchestra.run.smithi086.stdout: "pgmap_items": 0, 2024-05-12T22:20:00.617 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_co_bytes": 24993720, 2024-05-12T22:20:00.617 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_co_items": 250827, 2024-05-12T22:20:00.617 INFO:teuthology.orchestra.run.smithi086.stdout: "unittest_1_bytes": 0, 2024-05-12T22:20:00.617 INFO:teuthology.orchestra.run.smithi086.stdout: "unittest_1_items": 0, 2024-05-12T22:20:00.617 INFO:teuthology.orchestra.run.smithi086.stdout: "unittest_2_bytes": 0, 2024-05-12T22:20:00.618 INFO:teuthology.orchestra.run.smithi086.stdout: "unittest_2_items": 0 2024-05-12T22:20:00.618 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:00.618 INFO:teuthology.orchestra.run.smithi086.stdout: "objecter": { 2024-05-12T22:20:00.618 INFO:teuthology.orchestra.run.smithi086.stdout: "op_active": 0, 2024-05-12T22:20:00.618 INFO:teuthology.orchestra.run.smithi086.stdout: "op_laggy": 0, 2024-05-12T22:20:00.618 INFO:teuthology.orchestra.run.smithi086.stdout: "op_send": 58976, 2024-05-12T22:20:00.618 INFO:teuthology.orchestra.run.smithi086.stdout: "op_send_bytes": 71711326, 2024-05-12T22:20:00.618 INFO:teuthology.orchestra.run.smithi086.stdout: "op_resend": 37, 2024-05-12T22:20:00.618 INFO:teuthology.orchestra.run.smithi086.stdout: "op_reply": 58976, 2024-05-12T22:20:00.619 INFO:teuthology.orchestra.run.smithi086.stdout: "op_latency": { 2024-05-12T22:20:00.619 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 58976, 2024-05-12T22:20:00.619 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 63.922625046, 2024-05-12T22:20:00.619 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.001083875 2024-05-12T22:20:00.619 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:00.619 INFO:teuthology.orchestra.run.smithi086.stdout: "op_inflight": 0, 2024-05-12T22:20:00.619 INFO:teuthology.orchestra.run.smithi086.stdout: "oplen_avg": { 2024-05-12T22:20:00.619 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 58976, 2024-05-12T22:20:00.619 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 59338 2024-05-12T22:20:00.620 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:00.620 INFO:teuthology.orchestra.run.smithi086.stdout: "op": 58976, 2024-05-12T22:20:00.620 INFO:teuthology.orchestra.run.smithi086.stdout: "op_r": 54250, 2024-05-12T22:20:00.620 INFO:teuthology.orchestra.run.smithi086.stdout: "op_w": 4726, 2024-05-12T22:20:00.620 INFO:teuthology.orchestra.run.smithi086.stdout: "op_rmw": 0, 2024-05-12T22:20:00.620 INFO:teuthology.orchestra.run.smithi086.stdout: "op_pg": 0, 2024-05-12T22:20:00.620 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_stat": 0, 2024-05-12T22:20:00.620 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_create": 11, 2024-05-12T22:20:00.621 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_read": 74, 2024-05-12T22:20:00.621 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_write": 507, 2024-05-12T22:20:00.621 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_writefull": 56, 2024-05-12T22:20:00.621 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_writesame": 0, 2024-05-12T22:20:00.621 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_append": 0, 2024-05-12T22:20:00.621 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_zero": 0, 2024-05-12T22:20:00.621 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_truncate": 8, 2024-05-12T22:20:00.621 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_delete": 3815, 2024-05-12T22:20:00.621 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_mapext": 0, 2024-05-12T22:20:00.621 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_sparse_read": 0, 2024-05-12T22:20:00.622 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_clonerange": 0, 2024-05-12T22:20:00.622 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_getxattr": 54176, 2024-05-12T22:20:00.622 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_setxattr": 22, 2024-05-12T22:20:00.622 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_cmpxattr": 0, 2024-05-12T22:20:00.622 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_rmxattr": 0, 2024-05-12T22:20:00.622 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_resetxattrs": 0, 2024-05-12T22:20:00.622 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_call": 0, 2024-05-12T22:20:00.622 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_watch": 0, 2024-05-12T22:20:00.623 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_notify": 0, 2024-05-12T22:20:00.623 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_src_cmpxattr": 0, 2024-05-12T22:20:00.623 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_pgls": 0, 2024-05-12T22:20:00.623 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_pgls_filter": 0, 2024-05-12T22:20:00.623 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_other": 8, 2024-05-12T22:20:00.623 INFO:teuthology.orchestra.run.smithi086.stdout: "linger_active": 0, 2024-05-12T22:20:00.623 INFO:teuthology.orchestra.run.smithi086.stdout: "linger_send": 0, 2024-05-12T22:20:00.623 INFO:teuthology.orchestra.run.smithi086.stdout: "linger_resend": 0, 2024-05-12T22:20:00.623 INFO:teuthology.orchestra.run.smithi086.stdout: "linger_ping": 0, 2024-05-12T22:20:00.624 INFO:teuthology.orchestra.run.smithi086.stdout: "poolop_active": 0, 2024-05-12T22:20:00.624 INFO:teuthology.orchestra.run.smithi086.stdout: "poolop_send": 0, 2024-05-12T22:20:00.624 INFO:teuthology.orchestra.run.smithi086.stdout: "poolop_resend": 0, 2024-05-12T22:20:00.624 INFO:teuthology.orchestra.run.smithi086.stdout: "poolstat_active": 0, 2024-05-12T22:20:00.624 INFO:teuthology.orchestra.run.smithi086.stdout: "poolstat_send": 0, 2024-05-12T22:20:00.624 INFO:teuthology.orchestra.run.smithi086.stdout: "poolstat_resend": 0, 2024-05-12T22:20:00.624 INFO:teuthology.orchestra.run.smithi086.stdout: "statfs_active": 0, 2024-05-12T22:20:00.624 INFO:teuthology.orchestra.run.smithi086.stdout: "statfs_send": 0, 2024-05-12T22:20:00.625 INFO:teuthology.orchestra.run.smithi086.stdout: "statfs_resend": 0, 2024-05-12T22:20:00.625 INFO:teuthology.orchestra.run.smithi086.stdout: "command_active": 0, 2024-05-12T22:20:00.625 INFO:teuthology.orchestra.run.smithi086.stdout: "command_send": 0, 2024-05-12T22:20:00.625 INFO:teuthology.orchestra.run.smithi086.stdout: "command_resend": 0, 2024-05-12T22:20:00.625 INFO:teuthology.orchestra.run.smithi086.stdout: "map_epoch": 97, 2024-05-12T22:20:00.625 INFO:teuthology.orchestra.run.smithi086.stdout: "map_full": 0, 2024-05-12T22:20:00.625 INFO:teuthology.orchestra.run.smithi086.stdout: "map_inc": 12, 2024-05-12T22:20:00.625 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_sessions": 12, 2024-05-12T22:20:00.625 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_session_open": 12, 2024-05-12T22:20:00.626 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_session_close": 0, 2024-05-12T22:20:00.626 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_laggy": 0, 2024-05-12T22:20:00.626 INFO:teuthology.orchestra.run.smithi086.stdout: "omap_wr": 616, 2024-05-12T22:20:00.626 INFO:teuthology.orchestra.run.smithi086.stdout: "omap_rd": 0, 2024-05-12T22:20:00.626 INFO:teuthology.orchestra.run.smithi086.stdout: "omap_del": 45 2024-05-12T22:20:00.626 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:00.626 INFO:teuthology.orchestra.run.smithi086.stdout: "oft": { 2024-05-12T22:20:00.626 INFO:teuthology.orchestra.run.smithi086.stdout: "omap_total_objs": 1, 2024-05-12T22:20:00.626 INFO:teuthology.orchestra.run.smithi086.stdout: "omap_total_kv_pairs": 0, 2024-05-12T22:20:00.627 INFO:teuthology.orchestra.run.smithi086.stdout: "omap_total_updates": 0, 2024-05-12T22:20:00.627 INFO:teuthology.orchestra.run.smithi086.stdout: "omap_total_removes": 0 2024-05-12T22:20:00.627 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:00.627 INFO:teuthology.orchestra.run.smithi086.stdout: "osd-slow-ops": { 2024-05-12T22:20:00.627 INFO:teuthology.orchestra.run.smithi086.stdout: "slow_ops_count": 0 2024-05-12T22:20:00.627 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:00.627 INFO:teuthology.orchestra.run.smithi086.stdout: "purge_queue": { 2024-05-12T22:20:00.627 INFO:teuthology.orchestra.run.smithi086.stdout: "pq_executing_ops": 0, 2024-05-12T22:20:00.627 INFO:teuthology.orchestra.run.smithi086.stdout: "pq_executing_ops_high_water": 10, 2024-05-12T22:20:00.628 INFO:teuthology.orchestra.run.smithi086.stdout: "pq_executing": 0, 2024-05-12T22:20:00.628 INFO:teuthology.orchestra.run.smithi086.stdout: "pq_executing_high_water": 10, 2024-05-12T22:20:00.628 INFO:teuthology.orchestra.run.smithi086.stdout: "pq_executed_ops": 3259, 2024-05-12T22:20:00.628 INFO:teuthology.orchestra.run.smithi086.stdout: "pq_executed": 3259, 2024-05-12T22:20:00.628 INFO:teuthology.orchestra.run.smithi086.stdout: "pq_item_in_journal": 0 2024-05-12T22:20:00.628 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:00.628 INFO:teuthology.orchestra.run.smithi086.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-05-12T22:20:00.628 INFO:teuthology.orchestra.run.smithi086.stdout: "val": 0, 2024-05-12T22:20:00.628 INFO:teuthology.orchestra.run.smithi086.stdout: "max": 104857600, 2024-05-12T22:20:00.629 INFO:teuthology.orchestra.run.smithi086.stdout: "get_started": 0, 2024-05-12T22:20:00.629 INFO:teuthology.orchestra.run.smithi086.stdout: "get": 86684, 2024-05-12T22:20:00.629 INFO:teuthology.orchestra.run.smithi086.stdout: "get_sum": 18948112, 2024-05-12T22:20:00.629 INFO:teuthology.orchestra.run.smithi086.stdout: "get_or_fail_fail": 0, 2024-05-12T22:20:00.629 INFO:teuthology.orchestra.run.smithi086.stdout: "get_or_fail_success": 86684, 2024-05-12T22:20:00.629 INFO:teuthology.orchestra.run.smithi086.stdout: "take": 0, 2024-05-12T22:20:00.629 INFO:teuthology.orchestra.run.smithi086.stdout: "take_sum": 0, 2024-05-12T22:20:00.629 INFO:teuthology.orchestra.run.smithi086.stdout: "put": 86684, 2024-05-12T22:20:00.629 INFO:teuthology.orchestra.run.smithi086.stdout: "put_sum": 18948112, 2024-05-12T22:20:00.630 INFO:teuthology.orchestra.run.smithi086.stdout: "wait": { 2024-05-12T22:20:00.630 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-05-12T22:20:00.630 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-05-12T22:20:00.630 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-05-12T22:20:00.630 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-05-12T22:20:00.630 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:00.630 INFO:teuthology.orchestra.run.smithi086.stdout: "throttle-objecter_bytes": { 2024-05-12T22:20:00.630 INFO:teuthology.orchestra.run.smithi086.stdout: "val": 0, 2024-05-12T22:20:00.631 INFO:teuthology.orchestra.run.smithi086.stdout: "max": 104857600, 2024-05-12T22:20:00.631 INFO:teuthology.orchestra.run.smithi086.stdout: "get_started": 0, 2024-05-12T22:20:00.631 INFO:teuthology.orchestra.run.smithi086.stdout: "get": 0, 2024-05-12T22:20:00.631 INFO:teuthology.orchestra.run.smithi086.stdout: "get_sum": 0, 2024-05-12T22:20:00.631 INFO:teuthology.orchestra.run.smithi086.stdout: "get_or_fail_fail": 0, 2024-05-12T22:20:00.631 INFO:teuthology.orchestra.run.smithi086.stdout: "get_or_fail_success": 0, 2024-05-12T22:20:00.631 INFO:teuthology.orchestra.run.smithi086.stdout: "take": 58976, 2024-05-12T22:20:00.631 INFO:teuthology.orchestra.run.smithi086.stdout: "take_sum": 72060957, 2024-05-12T22:20:00.631 INFO:teuthology.orchestra.run.smithi086.stdout: "put": 58976, 2024-05-12T22:20:00.632 INFO:teuthology.orchestra.run.smithi086.stdout: "put_sum": 72060957, 2024-05-12T22:20:00.632 INFO:teuthology.orchestra.run.smithi086.stdout: "wait": { 2024-05-12T22:20:00.632 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-05-12T22:20:00.632 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-05-12T22:20:00.632 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-05-12T22:20:00.632 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-05-12T22:20:00.632 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:00.632 INFO:teuthology.orchestra.run.smithi086.stdout: "throttle-objecter_ops": { 2024-05-12T22:20:00.632 INFO:teuthology.orchestra.run.smithi086.stdout: "val": 0, 2024-05-12T22:20:00.633 INFO:teuthology.orchestra.run.smithi086.stdout: "max": 1024, 2024-05-12T22:20:00.633 INFO:teuthology.orchestra.run.smithi086.stdout: "get_started": 0, 2024-05-12T22:20:00.633 INFO:teuthology.orchestra.run.smithi086.stdout: "get": 0, 2024-05-12T22:20:00.633 INFO:teuthology.orchestra.run.smithi086.stdout: "get_sum": 0, 2024-05-12T22:20:00.633 INFO:teuthology.orchestra.run.smithi086.stdout: "get_or_fail_fail": 0, 2024-05-12T22:20:00.633 INFO:teuthology.orchestra.run.smithi086.stdout: "get_or_fail_success": 0, 2024-05-12T22:20:00.633 INFO:teuthology.orchestra.run.smithi086.stdout: "take": 58976, 2024-05-12T22:20:00.633 INFO:teuthology.orchestra.run.smithi086.stdout: "take_sum": 58976, 2024-05-12T22:20:00.634 INFO:teuthology.orchestra.run.smithi086.stdout: "put": 58976, 2024-05-12T22:20:00.634 INFO:teuthology.orchestra.run.smithi086.stdout: "put_sum": 58976, 2024-05-12T22:20:00.634 INFO:teuthology.orchestra.run.smithi086.stdout: "wait": { 2024-05-12T22:20:00.634 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-05-12T22:20:00.634 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-05-12T22:20:00.634 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-05-12T22:20:00.634 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-05-12T22:20:00.634 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:00.634 INFO:teuthology.orchestra.run.smithi086.stdout: "throttle-write_buf_throttle": { 2024-05-12T22:20:00.635 INFO:teuthology.orchestra.run.smithi086.stdout: "val": 0, 2024-05-12T22:20:00.635 INFO:teuthology.orchestra.run.smithi086.stdout: "max": 3758096384, 2024-05-12T22:20:00.635 INFO:teuthology.orchestra.run.smithi086.stdout: "get_started": 0, 2024-05-12T22:20:00.635 INFO:teuthology.orchestra.run.smithi086.stdout: "get": 3259, 2024-05-12T22:20:00.635 INFO:teuthology.orchestra.run.smithi086.stdout: "get_sum": 349631, 2024-05-12T22:20:00.635 INFO:teuthology.orchestra.run.smithi086.stdout: "get_or_fail_fail": 0, 2024-05-12T22:20:00.635 INFO:teuthology.orchestra.run.smithi086.stdout: "get_or_fail_success": 3259, 2024-05-12T22:20:00.635 INFO:teuthology.orchestra.run.smithi086.stdout: "take": 0, 2024-05-12T22:20:00.635 INFO:teuthology.orchestra.run.smithi086.stdout: "take_sum": 0, 2024-05-12T22:20:00.636 INFO:teuthology.orchestra.run.smithi086.stdout: "put": 55, 2024-05-12T22:20:00.636 INFO:teuthology.orchestra.run.smithi086.stdout: "put_sum": 349631, 2024-05-12T22:20:00.636 INFO:teuthology.orchestra.run.smithi086.stdout: "wait": { 2024-05-12T22:20:00.636 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-05-12T22:20:00.636 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-05-12T22:20:00.636 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-05-12T22:20:00.636 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-05-12T22:20:00.636 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:00.636 INFO:teuthology.orchestra.run.smithi086.stdout: "throttle-write_buf_throttle-0x5607fe2ea500": { 2024-05-12T22:20:00.637 INFO:teuthology.orchestra.run.smithi086.stdout: "val": 84, 2024-05-12T22:20:00.637 INFO:teuthology.orchestra.run.smithi086.stdout: "max": 3758096384, 2024-05-12T22:20:00.637 INFO:teuthology.orchestra.run.smithi086.stdout: "get_started": 0, 2024-05-12T22:20:00.637 INFO:teuthology.orchestra.run.smithi086.stdout: "get": 26730, 2024-05-12T22:20:00.637 INFO:teuthology.orchestra.run.smithi086.stdout: "get_sum": 67252075, 2024-05-12T22:20:00.637 INFO:teuthology.orchestra.run.smithi086.stdout: "get_or_fail_fail": 0, 2024-05-12T22:20:00.637 INFO:teuthology.orchestra.run.smithi086.stdout: "get_or_fail_success": 26730, 2024-05-12T22:20:00.637 INFO:teuthology.orchestra.run.smithi086.stdout: "take": 0, 2024-05-12T22:20:00.638 INFO:teuthology.orchestra.run.smithi086.stdout: "take_sum": 0, 2024-05-12T22:20:00.638 INFO:teuthology.orchestra.run.smithi086.stdout: "put": 452, 2024-05-12T22:20:00.638 INFO:teuthology.orchestra.run.smithi086.stdout: "put_sum": 67251991, 2024-05-12T22:20:00.638 INFO:teuthology.orchestra.run.smithi086.stdout: "wait": { 2024-05-12T22:20:00.638 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-05-12T22:20:00.638 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-05-12T22:20:00.638 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-05-12T22:20:00.638 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-05-12T22:20:00.638 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-05-12T22:20:00.639 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-05-12T22:20:00.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:20:00 smithi186 ceph-mon[34785]: pgmap v341: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:20:00.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:20:00 smithi186 ceph-mon[34785]: overall HEALTH_OK 2024-05-12T22:20:00.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:20:00 smithi028 ceph-mon[29847]: pgmap v341: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:20:00.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:20:00 smithi028 ceph-mon[29847]: overall HEALTH_OK 2024-05-12T22:20:01.235 INFO:tasks.check_counter:Daemon mds.b mds.exported=3 2024-05-12T22:20:01.235 INFO:tasks.check_counter:Daemon mds.b mds.imported=0 2024-05-12T22:20:01.235 INFO:tasks.check_counter:Daemon mds.b mds_cache.dir_update=50 2024-05-12T22:20:01.235 INFO:tasks.check_counter:Daemon mds.b mds_cache.dir_update_receipt=0 2024-05-12T22:20:01.236 INFO:tasks.check_counter:Daemon mds.b mds.root_rsnaps=6 2024-05-12T22:20:01.236 INFO:tasks.check_counter:Daemon mds.b mds_server.req_mksnap_latency.avgcount=11 2024-05-12T22:20:01.236 INFO:tasks.check_counter:Daemon mds.b mds_server.req_rmsnap_latency.avgcount=5 2024-05-12T22:20:01.236 INFO:tasks.check_counter:Daemon mds.b mds.dir_split=28 2024-05-12T22:20:01.236 DEBUG:tasks.check_counter:Getting stats from e 2024-05-12T22:20:01.236 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f72fecff68e1d400c4568684327c900485c20d6a shell --fsid 6b92e7b4-10aa-11ef-bc98-c7b262605968 -- ceph daemon mds.e perf dump 2024-05-12T22:20:01.482 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/mon.b/config 2024-05-12T22:20:02.582 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:20:02 smithi086 ceph-mon[34297]: pgmap v342: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-12T22:20:02.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:20:02 smithi186 ceph-mon[34785]: pgmap v342: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-12T22:20:02.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:20:02 smithi028 ceph-mon[29847]: pgmap v342: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-12T22:20:03.216 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-05-12T22:20:03.217 INFO:teuthology.orchestra.run.smithi086.stdout: "AsyncMessenger::Worker-0": { 2024-05-12T22:20:03.217 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_recv_messages": 0, 2024-05-12T22:20:03.217 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_send_messages": 2, 2024-05-12T22:20:03.217 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_recv_bytes": 0, 2024-05-12T22:20:03.217 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_send_bytes": 0, 2024-05-12T22:20:03.217 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_created_connections": 4, 2024-05-12T22:20:03.217 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_active_connections": 0, 2024-05-12T22:20:03.217 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_running_total_time": 0.001315927, 2024-05-12T22:20:03.218 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_running_send_time": 0.000000000, 2024-05-12T22:20:03.218 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_running_recv_time": 0.000371700, 2024-05-12T22:20:03.218 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-05-12T22:20:03.218 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_send_messages_queue_lat": { 2024-05-12T22:20:03.218 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-05-12T22:20:03.218 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-05-12T22:20:03.218 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-05-12T22:20:03.218 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:03.218 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_handle_ack_lat": { 2024-05-12T22:20:03.219 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-05-12T22:20:03.219 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-05-12T22:20:03.219 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-05-12T22:20:03.219 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:03.219 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_recv_encrypted_bytes": 0, 2024-05-12T22:20:03.219 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_send_encrypted_bytes": 0 2024-05-12T22:20:03.219 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:03.219 INFO:teuthology.orchestra.run.smithi086.stdout: "AsyncMessenger::Worker-1": { 2024-05-12T22:20:03.219 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_recv_messages": 6, 2024-05-12T22:20:03.220 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_send_messages": 173, 2024-05-12T22:20:03.220 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_recv_bytes": 3392, 2024-05-12T22:20:03.220 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_send_bytes": 186144, 2024-05-12T22:20:03.220 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_created_connections": 5, 2024-05-12T22:20:03.220 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_active_connections": 3, 2024-05-12T22:20:03.220 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_running_total_time": 0.011288321, 2024-05-12T22:20:03.220 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_running_send_time": 0.007588468, 2024-05-12T22:20:03.220 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_running_recv_time": 0.001537055, 2024-05-12T22:20:03.221 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-05-12T22:20:03.221 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_send_messages_queue_lat": { 2024-05-12T22:20:03.221 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 171, 2024-05-12T22:20:03.221 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.030110996, 2024-05-12T22:20:03.221 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000176087 2024-05-12T22:20:03.221 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:03.221 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_handle_ack_lat": { 2024-05-12T22:20:03.221 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-05-12T22:20:03.221 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-05-12T22:20:03.221 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-05-12T22:20:03.222 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:03.222 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_recv_encrypted_bytes": 3392, 2024-05-12T22:20:03.222 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_send_encrypted_bytes": 186144 2024-05-12T22:20:03.222 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:03.222 INFO:teuthology.orchestra.run.smithi086.stdout: "AsyncMessenger::Worker-2": { 2024-05-12T22:20:03.222 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_recv_messages": 236, 2024-05-12T22:20:03.222 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_send_messages": 224, 2024-05-12T22:20:03.222 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_recv_bytes": 1722496, 2024-05-12T22:20:03.223 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_send_bytes": 138144, 2024-05-12T22:20:03.223 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_created_connections": 2, 2024-05-12T22:20:03.223 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_active_connections": 1, 2024-05-12T22:20:03.223 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_running_total_time": 0.045790610, 2024-05-12T22:20:03.223 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_running_send_time": 0.014318703, 2024-05-12T22:20:03.223 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_running_recv_time": 0.034210160, 2024-05-12T22:20:03.223 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_running_fast_dispatch_time": 0.005807400, 2024-05-12T22:20:03.223 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_send_messages_queue_lat": { 2024-05-12T22:20:03.223 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 223, 2024-05-12T22:20:03.224 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.011224861, 2024-05-12T22:20:03.224 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000050335 2024-05-12T22:20:03.224 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:03.224 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_handle_ack_lat": { 2024-05-12T22:20:03.224 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-05-12T22:20:03.224 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-05-12T22:20:03.224 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-05-12T22:20:03.224 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:03.225 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_recv_encrypted_bytes": 1722496, 2024-05-12T22:20:03.225 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_send_encrypted_bytes": 138144 2024-05-12T22:20:03.225 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:03.225 INFO:teuthology.orchestra.run.smithi086.stdout: "cct": { 2024-05-12T22:20:03.225 INFO:teuthology.orchestra.run.smithi086.stdout: "total_workers": 0, 2024-05-12T22:20:03.225 INFO:teuthology.orchestra.run.smithi086.stdout: "unhealthy_workers": 0 2024-05-12T22:20:03.225 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:03.225 INFO:teuthology.orchestra.run.smithi086.stdout: "mempool": { 2024-05-12T22:20:03.225 INFO:teuthology.orchestra.run.smithi086.stdout: "bloom_filter_bytes": 0, 2024-05-12T22:20:03.226 INFO:teuthology.orchestra.run.smithi086.stdout: "bloom_filter_items": 0, 2024-05-12T22:20:03.226 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_alloc_bytes": 0, 2024-05-12T22:20:03.226 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_alloc_items": 0, 2024-05-12T22:20:03.226 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_cache_data_bytes": 0, 2024-05-12T22:20:03.226 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_cache_data_items": 0, 2024-05-12T22:20:03.226 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_cache_onode_bytes": 0, 2024-05-12T22:20:03.226 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_cache_onode_items": 0, 2024-05-12T22:20:03.226 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_cache_meta_bytes": 0, 2024-05-12T22:20:03.226 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_cache_meta_items": 0, 2024-05-12T22:20:03.227 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_cache_other_bytes": 0, 2024-05-12T22:20:03.227 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_cache_other_items": 0, 2024-05-12T22:20:03.227 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_cache_buffer_bytes": 0, 2024-05-12T22:20:03.227 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_cache_buffer_items": 0, 2024-05-12T22:20:03.227 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_extent_bytes": 0, 2024-05-12T22:20:03.227 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_extent_items": 0, 2024-05-12T22:20:03.227 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_blob_bytes": 0, 2024-05-12T22:20:03.227 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_blob_items": 0, 2024-05-12T22:20:03.227 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_shared_blob_bytes": 0, 2024-05-12T22:20:03.227 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_shared_blob_items": 0, 2024-05-12T22:20:03.228 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_inline_bl_bytes": 0, 2024-05-12T22:20:03.228 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_inline_bl_items": 0, 2024-05-12T22:20:03.228 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_fsck_bytes": 0, 2024-05-12T22:20:03.228 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_fsck_items": 0, 2024-05-12T22:20:03.228 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_txc_bytes": 0, 2024-05-12T22:20:03.228 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_txc_items": 0, 2024-05-12T22:20:03.228 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_writing_deferred_bytes": 0, 2024-05-12T22:20:03.228 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_writing_deferred_items": 0, 2024-05-12T22:20:03.228 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_writing_bytes": 0, 2024-05-12T22:20:03.229 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_writing_items": 0, 2024-05-12T22:20:03.229 INFO:teuthology.orchestra.run.smithi086.stdout: "bluefs_bytes": 0, 2024-05-12T22:20:03.229 INFO:teuthology.orchestra.run.smithi086.stdout: "bluefs_items": 0, 2024-05-12T22:20:03.229 INFO:teuthology.orchestra.run.smithi086.stdout: "bluefs_file_reader_bytes": 0, 2024-05-12T22:20:03.229 INFO:teuthology.orchestra.run.smithi086.stdout: "bluefs_file_reader_items": 0, 2024-05-12T22:20:03.229 INFO:teuthology.orchestra.run.smithi086.stdout: "bluefs_file_writer_bytes": 0, 2024-05-12T22:20:03.229 INFO:teuthology.orchestra.run.smithi086.stdout: "bluefs_file_writer_items": 0, 2024-05-12T22:20:03.229 INFO:teuthology.orchestra.run.smithi086.stdout: "buffer_anon_bytes": 148995, 2024-05-12T22:20:03.229 INFO:teuthology.orchestra.run.smithi086.stdout: "buffer_anon_items": 31, 2024-05-12T22:20:03.229 INFO:teuthology.orchestra.run.smithi086.stdout: "buffer_meta_bytes": 0, 2024-05-12T22:20:03.230 INFO:teuthology.orchestra.run.smithi086.stdout: "buffer_meta_items": 0, 2024-05-12T22:20:03.230 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_bytes": 0, 2024-05-12T22:20:03.230 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_items": 0, 2024-05-12T22:20:03.230 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_mapbl_bytes": 0, 2024-05-12T22:20:03.230 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_mapbl_items": 0, 2024-05-12T22:20:03.230 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_pglog_bytes": 0, 2024-05-12T22:20:03.230 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_pglog_items": 0, 2024-05-12T22:20:03.230 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap_bytes": 0, 2024-05-12T22:20:03.230 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap_items": 0, 2024-05-12T22:20:03.231 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap_mapping_bytes": 0, 2024-05-12T22:20:03.231 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap_mapping_items": 0, 2024-05-12T22:20:03.231 INFO:teuthology.orchestra.run.smithi086.stdout: "pgmap_bytes": 0, 2024-05-12T22:20:03.231 INFO:teuthology.orchestra.run.smithi086.stdout: "pgmap_items": 0, 2024-05-12T22:20:03.231 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_co_bytes": 928, 2024-05-12T22:20:03.231 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_co_items": 2, 2024-05-12T22:20:03.231 INFO:teuthology.orchestra.run.smithi086.stdout: "unittest_1_bytes": 0, 2024-05-12T22:20:03.231 INFO:teuthology.orchestra.run.smithi086.stdout: "unittest_1_items": 0, 2024-05-12T22:20:03.231 INFO:teuthology.orchestra.run.smithi086.stdout: "unittest_2_bytes": 0, 2024-05-12T22:20:03.232 INFO:teuthology.orchestra.run.smithi086.stdout: "unittest_2_items": 0 2024-05-12T22:20:03.232 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:03.232 INFO:teuthology.orchestra.run.smithi086.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-05-12T22:20:03.232 INFO:teuthology.orchestra.run.smithi086.stdout: "val": 0, 2024-05-12T22:20:03.232 INFO:teuthology.orchestra.run.smithi086.stdout: "max": 104857600, 2024-05-12T22:20:03.232 INFO:teuthology.orchestra.run.smithi086.stdout: "get_started": 0, 2024-05-12T22:20:03.232 INFO:teuthology.orchestra.run.smithi086.stdout: "get": 238, 2024-05-12T22:20:03.232 INFO:teuthology.orchestra.run.smithi086.stdout: "get_sum": 1689239, 2024-05-12T22:20:03.232 INFO:teuthology.orchestra.run.smithi086.stdout: "get_or_fail_fail": 0, 2024-05-12T22:20:03.232 INFO:teuthology.orchestra.run.smithi086.stdout: "get_or_fail_success": 238, 2024-05-12T22:20:03.233 INFO:teuthology.orchestra.run.smithi086.stdout: "take": 0, 2024-05-12T22:20:03.233 INFO:teuthology.orchestra.run.smithi086.stdout: "take_sum": 0, 2024-05-12T22:20:03.233 INFO:teuthology.orchestra.run.smithi086.stdout: "put": 238, 2024-05-12T22:20:03.233 INFO:teuthology.orchestra.run.smithi086.stdout: "put_sum": 1689239, 2024-05-12T22:20:03.233 INFO:teuthology.orchestra.run.smithi086.stdout: "wait": { 2024-05-12T22:20:03.233 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-05-12T22:20:03.233 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-05-12T22:20:03.233 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-05-12T22:20:03.233 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-05-12T22:20:03.234 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-05-12T22:20:03.234 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-05-12T22:20:03.991 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.e 2024-05-12T22:20:03.991 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.e 2024-05-12T22:20:03.991 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.e 2024-05-12T22:20:03.991 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.e 2024-05-12T22:20:03.992 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.e 2024-05-12T22:20:03.992 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.e 2024-05-12T22:20:03.992 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.e 2024-05-12T22:20:03.992 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.e 2024-05-12T22:20:03.992 DEBUG:tasks.check_counter:Getting stats from h 2024-05-12T22:20:03.992 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f72fecff68e1d400c4568684327c900485c20d6a shell --fsid 6b92e7b4-10aa-11ef-bc98-c7b262605968 -- ceph daemon mds.h perf dump 2024-05-12T22:20:04.229 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/mon.b/config 2024-05-12T22:20:04.617 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:20:04 smithi086 ceph-mon[34297]: pgmap v343: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:20:04.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:20:04 smithi186 ceph-mon[34785]: pgmap v343: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:20:04.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:20:04 smithi028 ceph-mon[29847]: pgmap v343: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-12T22:20:06.009 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-05-12T22:20:06.009 INFO:teuthology.orchestra.run.smithi086.stdout: "AsyncMessenger::Worker-0": { 2024-05-12T22:20:06.010 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_recv_messages": 1938, 2024-05-12T22:20:06.010 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_send_messages": 1922, 2024-05-12T22:20:06.010 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_recv_bytes": 23375597, 2024-05-12T22:20:06.010 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_send_bytes": 674990, 2024-05-12T22:20:06.010 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_created_connections": 6, 2024-05-12T22:20:06.010 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_active_connections": 2, 2024-05-12T22:20:06.011 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_running_total_time": 0.205402674, 2024-05-12T22:20:06.011 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_running_send_time": 0.074776328, 2024-05-12T22:20:06.011 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_running_recv_time": 0.076860437, 2024-05-12T22:20:06.011 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_running_fast_dispatch_time": 0.048130676, 2024-05-12T22:20:06.011 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_send_messages_queue_lat": { 2024-05-12T22:20:06.011 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 1921, 2024-05-12T22:20:06.011 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.070018877, 2024-05-12T22:20:06.012 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000036449 2024-05-12T22:20:06.012 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:06.012 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_handle_ack_lat": { 2024-05-12T22:20:06.012 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-05-12T22:20:06.012 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-05-12T22:20:06.012 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-05-12T22:20:06.013 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:06.013 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_recv_encrypted_bytes": 1759584, 2024-05-12T22:20:06.013 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_send_encrypted_bytes": 139792 2024-05-12T22:20:06.013 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:06.013 INFO:teuthology.orchestra.run.smithi086.stdout: "AsyncMessenger::Worker-1": { 2024-05-12T22:20:06.014 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_recv_messages": 60, 2024-05-12T22:20:06.014 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_send_messages": 226, 2024-05-12T22:20:06.014 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_recv_bytes": 20993666, 2024-05-12T22:20:06.014 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_send_bytes": 480326, 2024-05-12T22:20:06.014 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_created_connections": 8, 2024-05-12T22:20:06.015 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_active_connections": 6, 2024-05-12T22:20:06.015 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_running_total_time": 0.024004314, 2024-05-12T22:20:06.015 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_running_send_time": 0.007457776, 2024-05-12T22:20:06.015 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_running_recv_time": 0.013158003, 2024-05-12T22:20:06.018 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_running_fast_dispatch_time": 0.001059158, 2024-05-12T22:20:06.019 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_send_messages_queue_lat": { 2024-05-12T22:20:06.019 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 224, 2024-05-12T22:20:06.019 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.037166069, 2024-05-12T22:20:06.019 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000165919 2024-05-12T22:20:06.019 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:06.019 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_handle_ack_lat": { 2024-05-12T22:20:06.019 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-05-12T22:20:06.019 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-05-12T22:20:06.020 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-05-12T22:20:06.020 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:06.020 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_recv_encrypted_bytes": 3392, 2024-05-12T22:20:06.020 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_send_encrypted_bytes": 463088 2024-05-12T22:20:06.020 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:06.020 INFO:teuthology.orchestra.run.smithi086.stdout: "AsyncMessenger::Worker-2": { 2024-05-12T22:20:06.020 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_recv_messages": 868, 2024-05-12T22:20:06.021 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_send_messages": 870, 2024-05-12T22:20:06.021 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_recv_bytes": 25388066, 2024-05-12T22:20:06.021 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_send_bytes": 276031, 2024-05-12T22:20:06.021 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_created_connections": 5, 2024-05-12T22:20:06.021 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_active_connections": 4, 2024-05-12T22:20:06.021 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_running_total_time": 0.083587804, 2024-05-12T22:20:06.021 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_running_send_time": 0.020174927, 2024-05-12T22:20:06.022 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_running_recv_time": 0.030286203, 2024-05-12T22:20:06.022 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_running_fast_dispatch_time": 0.026679795, 2024-05-12T22:20:06.022 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_send_messages_queue_lat": { 2024-05-12T22:20:06.022 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 868, 2024-05-12T22:20:06.022 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.025842006, 2024-05-12T22:20:06.022 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000029771 2024-05-12T22:20:06.022 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:06.022 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_handle_ack_lat": { 2024-05-12T22:20:06.023 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-05-12T22:20:06.023 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-05-12T22:20:06.023 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-05-12T22:20:06.023 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:06.023 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_recv_encrypted_bytes": 0, 2024-05-12T22:20:06.023 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_send_encrypted_bytes": 0 2024-05-12T22:20:06.023 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:06.023 INFO:teuthology.orchestra.run.smithi086.stdout: "cct": { 2024-05-12T22:20:06.024 INFO:teuthology.orchestra.run.smithi086.stdout: "total_workers": 1, 2024-05-12T22:20:06.024 INFO:teuthology.orchestra.run.smithi086.stdout: "unhealthy_workers": 0 2024-05-12T22:20:06.024 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:06.024 INFO:teuthology.orchestra.run.smithi086.stdout: "finisher-MDSRank": { 2024-05-12T22:20:06.024 INFO:teuthology.orchestra.run.smithi086.stdout: "queue_len": 0, 2024-05-12T22:20:06.024 INFO:teuthology.orchestra.run.smithi086.stdout: "complete_latency": { 2024-05-12T22:20:06.024 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 4320, 2024-05-12T22:20:06.024 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.106332141, 2024-05-12T22:20:06.025 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000024613 2024-05-12T22:20:06.025 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-05-12T22:20:06.025 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:06.025 INFO:teuthology.orchestra.run.smithi086.stdout: "finisher-PurgeQueue": { 2024-05-12T22:20:06.025 INFO:teuthology.orchestra.run.smithi086.stdout: "queue_len": 0, 2024-05-12T22:20:06.025 INFO:teuthology.orchestra.run.smithi086.stdout: "complete_latency": { 2024-05-12T22:20:06.025 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-05-12T22:20:06.025 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-05-12T22:20:06.025 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-05-12T22:20:06.025 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-05-12T22:20:06.026 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:06.026 INFO:teuthology.orchestra.run.smithi086.stdout: "mds": { 2024-05-12T22:20:06.026 INFO:teuthology.orchestra.run.smithi086.stdout: "request": 0, 2024-05-12T22:20:06.026 INFO:teuthology.orchestra.run.smithi086.stdout: "reply": 0, 2024-05-12T22:20:06.026 INFO:teuthology.orchestra.run.smithi086.stdout: "reply_latency": { 2024-05-12T22:20:06.026 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-05-12T22:20:06.026 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-05-12T22:20:06.026 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-05-12T22:20:06.026 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:06.027 INFO:teuthology.orchestra.run.smithi086.stdout: "slow_reply": 0, 2024-05-12T22:20:06.027 INFO:teuthology.orchestra.run.smithi086.stdout: "forward": 0, 2024-05-12T22:20:06.027 INFO:teuthology.orchestra.run.smithi086.stdout: "dir_fetch_complete": 0, 2024-05-12T22:20:06.027 INFO:teuthology.orchestra.run.smithi086.stdout: "dir_fetch_keys": 0, 2024-05-12T22:20:06.027 INFO:teuthology.orchestra.run.smithi086.stdout: "dir_commit": 0, 2024-05-12T22:20:06.027 INFO:teuthology.orchestra.run.smithi086.stdout: "dir_split": 0, 2024-05-12T22:20:06.027 INFO:teuthology.orchestra.run.smithi086.stdout: "dir_merge": 0, 2024-05-12T22:20:06.027 INFO:teuthology.orchestra.run.smithi086.stdout: "inodes": 11702, 2024-05-12T22:20:06.027 INFO:teuthology.orchestra.run.smithi086.stdout: "inodes_top": 0, 2024-05-12T22:20:06.028 INFO:teuthology.orchestra.run.smithi086.stdout: "inodes_bottom": 11702, 2024-05-12T22:20:06.028 INFO:teuthology.orchestra.run.smithi086.stdout: "inodes_pin_tail": 0, 2024-05-12T22:20:06.028 INFO:teuthology.orchestra.run.smithi086.stdout: "inodes_pinned": 11702, 2024-05-12T22:20:06.028 INFO:teuthology.orchestra.run.smithi086.stdout: "inodes_expired": 0, 2024-05-12T22:20:06.028 INFO:teuthology.orchestra.run.smithi086.stdout: "inodes_with_caps": 0, 2024-05-12T22:20:06.028 INFO:teuthology.orchestra.run.smithi086.stdout: "caps": 0, 2024-05-12T22:20:06.028 INFO:teuthology.orchestra.run.smithi086.stdout: "subtrees": 5, 2024-05-12T22:20:06.028 INFO:teuthology.orchestra.run.smithi086.stdout: "traverse": 0, 2024-05-12T22:20:06.028 INFO:teuthology.orchestra.run.smithi086.stdout: "traverse_hit": 0, 2024-05-12T22:20:06.029 INFO:teuthology.orchestra.run.smithi086.stdout: "traverse_forward": 0, 2024-05-12T22:20:06.029 INFO:teuthology.orchestra.run.smithi086.stdout: "traverse_discover": 0, 2024-05-12T22:20:06.029 INFO:teuthology.orchestra.run.smithi086.stdout: "traverse_dir_fetch": 0, 2024-05-12T22:20:06.029 INFO:teuthology.orchestra.run.smithi086.stdout: "traverse_remote_ino": 0, 2024-05-12T22:20:06.029 INFO:teuthology.orchestra.run.smithi086.stdout: "traverse_lock": 0, 2024-05-12T22:20:06.029 INFO:teuthology.orchestra.run.smithi086.stdout: "load_cent": 0, 2024-05-12T22:20:06.029 INFO:teuthology.orchestra.run.smithi086.stdout: "q": 0, 2024-05-12T22:20:06.029 INFO:teuthology.orchestra.run.smithi086.stdout: "exported": 0, 2024-05-12T22:20:06.029 INFO:teuthology.orchestra.run.smithi086.stdout: "exported_inodes": 0, 2024-05-12T22:20:06.029 INFO:teuthology.orchestra.run.smithi086.stdout: "imported": 0, 2024-05-12T22:20:06.030 INFO:teuthology.orchestra.run.smithi086.stdout: "imported_inodes": 0, 2024-05-12T22:20:06.030 INFO:teuthology.orchestra.run.smithi086.stdout: "openino_dir_fetch": 0, 2024-05-12T22:20:06.030 INFO:teuthology.orchestra.run.smithi086.stdout: "openino_backtrace_fetch": 0, 2024-05-12T22:20:06.030 INFO:teuthology.orchestra.run.smithi086.stdout: "openino_peer_discover": 0, 2024-05-12T22:20:06.030 INFO:teuthology.orchestra.run.smithi086.stdout: "root_rfiles": 3454, 2024-05-12T22:20:06.030 INFO:teuthology.orchestra.run.smithi086.stdout: "root_rbytes": 26743557, 2024-05-12T22:20:06.030 INFO:teuthology.orchestra.run.smithi086.stdout: "root_rsnaps": 6, 2024-05-12T22:20:06.030 INFO:teuthology.orchestra.run.smithi086.stdout: "scrub_backtrace_fetch": 0, 2024-05-12T22:20:06.030 INFO:teuthology.orchestra.run.smithi086.stdout: "scrub_set_tag": 0, 2024-05-12T22:20:06.031 INFO:teuthology.orchestra.run.smithi086.stdout: "scrub_backtrace_repaired": 0, 2024-05-12T22:20:06.031 INFO:teuthology.orchestra.run.smithi086.stdout: "scrub_inotable_repaired": 0, 2024-05-12T22:20:06.031 INFO:teuthology.orchestra.run.smithi086.stdout: "scrub_dir_inodes": 0, 2024-05-12T22:20:06.031 INFO:teuthology.orchestra.run.smithi086.stdout: "scrub_dir_base_inodes": 0, 2024-05-12T22:20:06.031 INFO:teuthology.orchestra.run.smithi086.stdout: "scrub_dirfrag_rstats": 0, 2024-05-12T22:20:06.031 INFO:teuthology.orchestra.run.smithi086.stdout: "scrub_file_inodes": 0, 2024-05-12T22:20:06.031 INFO:teuthology.orchestra.run.smithi086.stdout: "handle_inode_file_caps": 0, 2024-05-12T22:20:06.031 INFO:teuthology.orchestra.run.smithi086.stdout: "ceph_cap_op_revoke": 0, 2024-05-12T22:20:06.031 INFO:teuthology.orchestra.run.smithi086.stdout: "ceph_cap_op_grant": 0, 2024-05-12T22:20:06.032 INFO:teuthology.orchestra.run.smithi086.stdout: "ceph_cap_op_trunc": 0, 2024-05-12T22:20:06.032 INFO:teuthology.orchestra.run.smithi086.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-05-12T22:20:06.032 INFO:teuthology.orchestra.run.smithi086.stdout: "ceph_cap_op_flush_ack": 0, 2024-05-12T22:20:06.032 INFO:teuthology.orchestra.run.smithi086.stdout: "handle_client_caps": 0, 2024-05-12T22:20:06.032 INFO:teuthology.orchestra.run.smithi086.stdout: "handle_client_caps_dirty": 0, 2024-05-12T22:20:06.032 INFO:teuthology.orchestra.run.smithi086.stdout: "handle_client_cap_release": 0, 2024-05-12T22:20:06.032 INFO:teuthology.orchestra.run.smithi086.stdout: "process_request_cap_release": 0 2024-05-12T22:20:06.032 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:06.032 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_cache": { 2024-05-12T22:20:06.033 INFO:teuthology.orchestra.run.smithi086.stdout: "dir_update": 0, 2024-05-12T22:20:06.033 INFO:teuthology.orchestra.run.smithi086.stdout: "dir_update_receipt": 0, 2024-05-12T22:20:06.033 INFO:teuthology.orchestra.run.smithi086.stdout: "dir_try_discover": 0, 2024-05-12T22:20:06.033 INFO:teuthology.orchestra.run.smithi086.stdout: "dir_send_discover": 0, 2024-05-12T22:20:06.033 INFO:teuthology.orchestra.run.smithi086.stdout: "dir_handle_discover": 0, 2024-05-12T22:20:06.033 INFO:teuthology.orchestra.run.smithi086.stdout: "num_strays": 0, 2024-05-12T22:20:06.033 INFO:teuthology.orchestra.run.smithi086.stdout: "num_strays_delayed": 0, 2024-05-12T22:20:06.033 INFO:teuthology.orchestra.run.smithi086.stdout: "num_strays_enqueuing": 0, 2024-05-12T22:20:06.033 INFO:teuthology.orchestra.run.smithi086.stdout: "strays_created": 0, 2024-05-12T22:20:06.034 INFO:teuthology.orchestra.run.smithi086.stdout: "strays_enqueued": 0, 2024-05-12T22:20:06.034 INFO:teuthology.orchestra.run.smithi086.stdout: "strays_reintegrated": 0, 2024-05-12T22:20:06.034 INFO:teuthology.orchestra.run.smithi086.stdout: "strays_migrated": 0, 2024-05-12T22:20:06.034 INFO:teuthology.orchestra.run.smithi086.stdout: "num_recovering_processing": 0, 2024-05-12T22:20:06.034 INFO:teuthology.orchestra.run.smithi086.stdout: "num_recovering_enqueued": 0, 2024-05-12T22:20:06.034 INFO:teuthology.orchestra.run.smithi086.stdout: "num_recovering_prioritized": 0, 2024-05-12T22:20:06.034 INFO:teuthology.orchestra.run.smithi086.stdout: "recovery_started": 0, 2024-05-12T22:20:06.034 INFO:teuthology.orchestra.run.smithi086.stdout: "recovery_completed": 0, 2024-05-12T22:20:06.035 INFO:teuthology.orchestra.run.smithi086.stdout: "ireq_quiesce_path": 0, 2024-05-12T22:20:06.035 INFO:teuthology.orchestra.run.smithi086.stdout: "ireq_quiesce_inode": 0, 2024-05-12T22:20:06.035 INFO:teuthology.orchestra.run.smithi086.stdout: "ireq_enqueue_scrub": 0, 2024-05-12T22:20:06.035 INFO:teuthology.orchestra.run.smithi086.stdout: "ireq_exportdir": 0, 2024-05-12T22:20:06.035 INFO:teuthology.orchestra.run.smithi086.stdout: "ireq_flush": 0, 2024-05-12T22:20:06.035 INFO:teuthology.orchestra.run.smithi086.stdout: "ireq_fragmentdir": 0, 2024-05-12T22:20:06.035 INFO:teuthology.orchestra.run.smithi086.stdout: "ireq_fragstats": 0, 2024-05-12T22:20:06.035 INFO:teuthology.orchestra.run.smithi086.stdout: "ireq_inodestats": 0 2024-05-12T22:20:06.035 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:06.036 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_log": { 2024-05-12T22:20:06.036 INFO:teuthology.orchestra.run.smithi086.stdout: "evlrg": 0, 2024-05-12T22:20:06.036 INFO:teuthology.orchestra.run.smithi086.stdout: "evadd": 0, 2024-05-12T22:20:06.036 INFO:teuthology.orchestra.run.smithi086.stdout: "evex": 0, 2024-05-12T22:20:06.036 INFO:teuthology.orchestra.run.smithi086.stdout: "evtrm": 0, 2024-05-12T22:20:06.036 INFO:teuthology.orchestra.run.smithi086.stdout: "ev": 26730, 2024-05-12T22:20:06.036 INFO:teuthology.orchestra.run.smithi086.stdout: "evexg": 0, 2024-05-12T22:20:06.036 INFO:teuthology.orchestra.run.smithi086.stdout: "evexd": 0, 2024-05-12T22:20:06.036 INFO:teuthology.orchestra.run.smithi086.stdout: "segadd": 0, 2024-05-12T22:20:06.037 INFO:teuthology.orchestra.run.smithi086.stdout: "segex": 0, 2024-05-12T22:20:06.037 INFO:teuthology.orchestra.run.smithi086.stdout: "segtrm": 0, 2024-05-12T22:20:06.037 INFO:teuthology.orchestra.run.smithi086.stdout: "seg": 37, 2024-05-12T22:20:06.037 INFO:teuthology.orchestra.run.smithi086.stdout: "segmjr": 4, 2024-05-12T22:20:06.037 INFO:teuthology.orchestra.run.smithi086.stdout: "segexg": 0, 2024-05-12T22:20:06.037 INFO:teuthology.orchestra.run.smithi086.stdout: "segexd": 0, 2024-05-12T22:20:06.037 INFO:teuthology.orchestra.run.smithi086.stdout: "expos": 4194304, 2024-05-12T22:20:06.037 INFO:teuthology.orchestra.run.smithi086.stdout: "wrpos": 71446379, 2024-05-12T22:20:06.038 INFO:teuthology.orchestra.run.smithi086.stdout: "rdpos": 71446295, 2024-05-12T22:20:06.038 INFO:teuthology.orchestra.run.smithi086.stdout: "jlat": { 2024-05-12T22:20:06.038 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-05-12T22:20:06.038 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-05-12T22:20:06.038 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-05-12T22:20:06.038 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:06.038 INFO:teuthology.orchestra.run.smithi086.stdout: "replayed": 26730 2024-05-12T22:20:06.038 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:06.039 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_mem": { 2024-05-12T22:20:06.039 INFO:teuthology.orchestra.run.smithi086.stdout: "ino": 7237, 2024-05-12T22:20:06.039 INFO:teuthology.orchestra.run.smithi086.stdout: "ino+": 10503, 2024-05-12T22:20:06.039 INFO:teuthology.orchestra.run.smithi086.stdout: "ino-": 3266, 2024-05-12T22:20:06.039 INFO:teuthology.orchestra.run.smithi086.stdout: "dir": 281, 2024-05-12T22:20:06.039 INFO:teuthology.orchestra.run.smithi086.stdout: "dir+": 322, 2024-05-12T22:20:06.039 INFO:teuthology.orchestra.run.smithi086.stdout: "dir-": 41, 2024-05-12T22:20:06.039 INFO:teuthology.orchestra.run.smithi086.stdout: "dn": 11702, 2024-05-12T22:20:06.040 INFO:teuthology.orchestra.run.smithi086.stdout: "dn+": 11720, 2024-05-12T22:20:06.040 INFO:teuthology.orchestra.run.smithi086.stdout: "dn-": 18, 2024-05-12T22:20:06.040 INFO:teuthology.orchestra.run.smithi086.stdout: "cap": 0, 2024-05-12T22:20:06.040 INFO:teuthology.orchestra.run.smithi086.stdout: "cap+": 0, 2024-05-12T22:20:06.040 INFO:teuthology.orchestra.run.smithi086.stdout: "cap-": 0, 2024-05-12T22:20:06.040 INFO:teuthology.orchestra.run.smithi086.stdout: "rss": 82916, 2024-05-12T22:20:06.041 INFO:teuthology.orchestra.run.smithi086.stdout: "heap": 215420 2024-05-12T22:20:06.041 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:06.041 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_server": { 2024-05-12T22:20:06.041 INFO:teuthology.orchestra.run.smithi086.stdout: "dispatch_client_request": 0, 2024-05-12T22:20:06.041 INFO:teuthology.orchestra.run.smithi086.stdout: "dispatch_server_request": 0, 2024-05-12T22:20:06.041 INFO:teuthology.orchestra.run.smithi086.stdout: "handle_client_request": 0, 2024-05-12T22:20:06.041 INFO:teuthology.orchestra.run.smithi086.stdout: "handle_client_session": 0, 2024-05-12T22:20:06.041 INFO:teuthology.orchestra.run.smithi086.stdout: "handle_peer_request": 0, 2024-05-12T22:20:06.042 INFO:teuthology.orchestra.run.smithi086.stdout: "req_create_latency": { 2024-05-12T22:20:06.042 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-05-12T22:20:06.042 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-05-12T22:20:06.042 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-05-12T22:20:06.042 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:06.042 INFO:teuthology.orchestra.run.smithi086.stdout: "req_getattr_latency": { 2024-05-12T22:20:06.042 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-05-12T22:20:06.042 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-05-12T22:20:06.043 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-05-12T22:20:06.043 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:06.043 INFO:teuthology.orchestra.run.smithi086.stdout: "req_getfilelock_latency": { 2024-05-12T22:20:06.043 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-05-12T22:20:06.043 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-05-12T22:20:06.043 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-05-12T22:20:06.043 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:06.043 INFO:teuthology.orchestra.run.smithi086.stdout: "req_link_latency": { 2024-05-12T22:20:06.043 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-05-12T22:20:06.044 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-05-12T22:20:06.044 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-05-12T22:20:06.044 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:06.044 INFO:teuthology.orchestra.run.smithi086.stdout: "req_lookup_latency": { 2024-05-12T22:20:06.044 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-05-12T22:20:06.044 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-05-12T22:20:06.044 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-05-12T22:20:06.044 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:06.044 INFO:teuthology.orchestra.run.smithi086.stdout: "req_lookuphash_latency": { 2024-05-12T22:20:06.045 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-05-12T22:20:06.045 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-05-12T22:20:06.045 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-05-12T22:20:06.045 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:06.045 INFO:teuthology.orchestra.run.smithi086.stdout: "req_lookupino_latency": { 2024-05-12T22:20:06.045 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-05-12T22:20:06.045 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-05-12T22:20:06.045 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-05-12T22:20:06.046 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:06.046 INFO:teuthology.orchestra.run.smithi086.stdout: "req_lookupname_latency": { 2024-05-12T22:20:06.046 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-05-12T22:20:06.046 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-05-12T22:20:06.046 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-05-12T22:20:06.046 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:06.046 INFO:teuthology.orchestra.run.smithi086.stdout: "req_lookupparent_latency": { 2024-05-12T22:20:06.046 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-05-12T22:20:06.046 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-05-12T22:20:06.047 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-05-12T22:20:06.047 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:06.047 INFO:teuthology.orchestra.run.smithi086.stdout: "req_lookupsnap_latency": { 2024-05-12T22:20:06.047 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-05-12T22:20:06.047 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-05-12T22:20:06.048 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-05-12T22:20:06.048 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:06.048 INFO:teuthology.orchestra.run.smithi086.stdout: "req_lssnap_latency": { 2024-05-12T22:20:06.048 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-05-12T22:20:06.048 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-05-12T22:20:06.048 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-05-12T22:20:06.048 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:06.049 INFO:teuthology.orchestra.run.smithi086.stdout: "req_mkdir_latency": { 2024-05-12T22:20:06.049 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-05-12T22:20:06.049 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-05-12T22:20:06.050 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-05-12T22:20:06.050 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:06.050 INFO:teuthology.orchestra.run.smithi086.stdout: "req_mknod_latency": { 2024-05-12T22:20:06.050 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-05-12T22:20:06.050 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-05-12T22:20:06.050 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-05-12T22:20:06.051 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:06.051 INFO:teuthology.orchestra.run.smithi086.stdout: "req_mksnap_latency": { 2024-05-12T22:20:06.051 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-05-12T22:20:06.051 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-05-12T22:20:06.051 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-05-12T22:20:06.051 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:06.051 INFO:teuthology.orchestra.run.smithi086.stdout: "req_open_latency": { 2024-05-12T22:20:06.052 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-05-12T22:20:06.052 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-05-12T22:20:06.052 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-05-12T22:20:06.052 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:06.052 INFO:teuthology.orchestra.run.smithi086.stdout: "req_readdir_latency": { 2024-05-12T22:20:06.052 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-05-12T22:20:06.053 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-05-12T22:20:06.053 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-05-12T22:20:06.053 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:06.053 INFO:teuthology.orchestra.run.smithi086.stdout: "req_rename_latency": { 2024-05-12T22:20:06.053 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-05-12T22:20:06.053 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-05-12T22:20:06.053 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-05-12T22:20:06.054 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:06.054 INFO:teuthology.orchestra.run.smithi086.stdout: "req_renamesnap_latency": { 2024-05-12T22:20:06.054 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-05-12T22:20:06.054 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-05-12T22:20:06.054 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-05-12T22:20:06.054 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:06.054 INFO:teuthology.orchestra.run.smithi086.stdout: "req_snapdiff_latency": { 2024-05-12T22:20:06.055 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-05-12T22:20:06.055 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-05-12T22:20:06.055 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-05-12T22:20:06.055 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:06.055 INFO:teuthology.orchestra.run.smithi086.stdout: "req_rmdir_latency": { 2024-05-12T22:20:06.055 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-05-12T22:20:06.055 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-05-12T22:20:06.056 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-05-12T22:20:06.056 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:06.056 INFO:teuthology.orchestra.run.smithi086.stdout: "req_rmsnap_latency": { 2024-05-12T22:20:06.056 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-05-12T22:20:06.056 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-05-12T22:20:06.056 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-05-12T22:20:06.057 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:06.057 INFO:teuthology.orchestra.run.smithi086.stdout: "req_rmxattr_latency": { 2024-05-12T22:20:06.057 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-05-12T22:20:06.057 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-05-12T22:20:06.057 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-05-12T22:20:06.057 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:06.058 INFO:teuthology.orchestra.run.smithi086.stdout: "req_setattr_latency": { 2024-05-12T22:20:06.058 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-05-12T22:20:06.058 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-05-12T22:20:06.058 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-05-12T22:20:06.058 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:06.058 INFO:teuthology.orchestra.run.smithi086.stdout: "req_setdirlayout_latency": { 2024-05-12T22:20:06.058 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-05-12T22:20:06.059 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-05-12T22:20:06.059 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-05-12T22:20:06.059 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:06.059 INFO:teuthology.orchestra.run.smithi086.stdout: "req_setfilelock_latency": { 2024-05-12T22:20:06.059 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-05-12T22:20:06.059 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-05-12T22:20:06.059 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-05-12T22:20:06.059 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:06.059 INFO:teuthology.orchestra.run.smithi086.stdout: "req_setlayout_latency": { 2024-05-12T22:20:06.060 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-05-12T22:20:06.060 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-05-12T22:20:06.060 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-05-12T22:20:06.060 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:06.060 INFO:teuthology.orchestra.run.smithi086.stdout: "req_setxattr_latency": { 2024-05-12T22:20:06.060 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-05-12T22:20:06.060 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-05-12T22:20:06.060 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-05-12T22:20:06.060 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:06.061 INFO:teuthology.orchestra.run.smithi086.stdout: "req_symlink_latency": { 2024-05-12T22:20:06.061 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-05-12T22:20:06.061 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-05-12T22:20:06.061 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-05-12T22:20:06.061 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:06.061 INFO:teuthology.orchestra.run.smithi086.stdout: "req_unlink_latency": { 2024-05-12T22:20:06.061 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-05-12T22:20:06.061 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-05-12T22:20:06.061 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-05-12T22:20:06.062 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:06.062 INFO:teuthology.orchestra.run.smithi086.stdout: "cap_revoke_eviction": 0, 2024-05-12T22:20:06.062 INFO:teuthology.orchestra.run.smithi086.stdout: "cap_acquisition_throttle": 0, 2024-05-12T22:20:06.062 INFO:teuthology.orchestra.run.smithi086.stdout: "req_getvxattr_latency": { 2024-05-12T22:20:06.062 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-05-12T22:20:06.062 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-05-12T22:20:06.062 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-05-12T22:20:06.062 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-05-12T22:20:06.063 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:06.063 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_sessions": { 2024-05-12T22:20:06.063 INFO:teuthology.orchestra.run.smithi086.stdout: "session_count": 2, 2024-05-12T22:20:06.063 INFO:teuthology.orchestra.run.smithi086.stdout: "session_add": 6, 2024-05-12T22:20:06.063 INFO:teuthology.orchestra.run.smithi086.stdout: "session_remove": 4, 2024-05-12T22:20:06.063 INFO:teuthology.orchestra.run.smithi086.stdout: "sessions_open": 5, 2024-05-12T22:20:06.063 INFO:teuthology.orchestra.run.smithi086.stdout: "sessions_stale": 0, 2024-05-12T22:20:06.063 INFO:teuthology.orchestra.run.smithi086.stdout: "total_load": 0, 2024-05-12T22:20:06.063 INFO:teuthology.orchestra.run.smithi086.stdout: "average_load": 0, 2024-05-12T22:20:06.064 INFO:teuthology.orchestra.run.smithi086.stdout: "avg_session_uptime": 4427, 2024-05-12T22:20:06.064 INFO:teuthology.orchestra.run.smithi086.stdout: "mdthresh_evicted": 0 2024-05-12T22:20:06.064 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:06.064 INFO:teuthology.orchestra.run.smithi086.stdout: "mempool": { 2024-05-12T22:20:06.064 INFO:teuthology.orchestra.run.smithi086.stdout: "bloom_filter_bytes": 0, 2024-05-12T22:20:06.064 INFO:teuthology.orchestra.run.smithi086.stdout: "bloom_filter_items": 0, 2024-05-12T22:20:06.064 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_alloc_bytes": 0, 2024-05-12T22:20:06.064 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_alloc_items": 0, 2024-05-12T22:20:06.064 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_cache_data_bytes": 0, 2024-05-12T22:20:06.065 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_cache_data_items": 0, 2024-05-12T22:20:06.065 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_cache_onode_bytes": 0, 2024-05-12T22:20:06.065 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_cache_onode_items": 0, 2024-05-12T22:20:06.065 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_cache_meta_bytes": 0, 2024-05-12T22:20:06.065 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_cache_meta_items": 0, 2024-05-12T22:20:06.065 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_cache_other_bytes": 0, 2024-05-12T22:20:06.065 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_cache_other_items": 0, 2024-05-12T22:20:06.065 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_cache_buffer_bytes": 0, 2024-05-12T22:20:06.065 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_cache_buffer_items": 0, 2024-05-12T22:20:06.066 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_extent_bytes": 0, 2024-05-12T22:20:06.066 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_extent_items": 0, 2024-05-12T22:20:06.066 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_blob_bytes": 0, 2024-05-12T22:20:06.066 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_blob_items": 0, 2024-05-12T22:20:06.066 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_shared_blob_bytes": 0, 2024-05-12T22:20:06.066 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_shared_blob_items": 0, 2024-05-12T22:20:06.066 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_inline_bl_bytes": 0, 2024-05-12T22:20:06.066 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_inline_bl_items": 0, 2024-05-12T22:20:06.066 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_fsck_bytes": 0, 2024-05-12T22:20:06.067 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_fsck_items": 0, 2024-05-12T22:20:06.067 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_txc_bytes": 0, 2024-05-12T22:20:06.067 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_txc_items": 0, 2024-05-12T22:20:06.067 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_writing_deferred_bytes": 0, 2024-05-12T22:20:06.067 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_writing_deferred_items": 0, 2024-05-12T22:20:06.067 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_writing_bytes": 0, 2024-05-12T22:20:06.067 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_writing_items": 0, 2024-05-12T22:20:06.067 INFO:teuthology.orchestra.run.smithi086.stdout: "bluefs_bytes": 0, 2024-05-12T22:20:06.067 INFO:teuthology.orchestra.run.smithi086.stdout: "bluefs_items": 0, 2024-05-12T22:20:06.068 INFO:teuthology.orchestra.run.smithi086.stdout: "bluefs_file_reader_bytes": 0, 2024-05-12T22:20:06.068 INFO:teuthology.orchestra.run.smithi086.stdout: "bluefs_file_reader_items": 0, 2024-05-12T22:20:06.068 INFO:teuthology.orchestra.run.smithi086.stdout: "bluefs_file_writer_bytes": 0, 2024-05-12T22:20:06.068 INFO:teuthology.orchestra.run.smithi086.stdout: "bluefs_file_writer_items": 0, 2024-05-12T22:20:06.068 INFO:teuthology.orchestra.run.smithi086.stdout: "buffer_anon_bytes": 483126, 2024-05-12T22:20:06.068 INFO:teuthology.orchestra.run.smithi086.stdout: "buffer_anon_items": 7328, 2024-05-12T22:20:06.068 INFO:teuthology.orchestra.run.smithi086.stdout: "buffer_meta_bytes": 0, 2024-05-12T22:20:06.068 INFO:teuthology.orchestra.run.smithi086.stdout: "buffer_meta_items": 0, 2024-05-12T22:20:06.068 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_bytes": 0, 2024-05-12T22:20:06.069 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_items": 0, 2024-05-12T22:20:06.069 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_mapbl_bytes": 0, 2024-05-12T22:20:06.069 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_mapbl_items": 0, 2024-05-12T22:20:06.069 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_pglog_bytes": 0, 2024-05-12T22:20:06.069 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_pglog_items": 0, 2024-05-12T22:20:06.069 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap_bytes": 8032, 2024-05-12T22:20:06.069 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap_items": 167, 2024-05-12T22:20:06.069 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap_mapping_bytes": 0, 2024-05-12T22:20:06.069 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap_mapping_items": 0, 2024-05-12T22:20:06.070 INFO:teuthology.orchestra.run.smithi086.stdout: "pgmap_bytes": 0, 2024-05-12T22:20:06.070 INFO:teuthology.orchestra.run.smithi086.stdout: "pgmap_items": 0, 2024-05-12T22:20:06.070 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_co_bytes": 25077633, 2024-05-12T22:20:06.070 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_co_items": 218103, 2024-05-12T22:20:06.070 INFO:teuthology.orchestra.run.smithi086.stdout: "unittest_1_bytes": 0, 2024-05-12T22:20:06.070 INFO:teuthology.orchestra.run.smithi086.stdout: "unittest_1_items": 0, 2024-05-12T22:20:06.070 INFO:teuthology.orchestra.run.smithi086.stdout: "unittest_2_bytes": 0, 2024-05-12T22:20:06.070 INFO:teuthology.orchestra.run.smithi086.stdout: "unittest_2_items": 0 2024-05-12T22:20:06.070 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:06.071 INFO:teuthology.orchestra.run.smithi086.stdout: "objecter": { 2024-05-12T22:20:06.071 INFO:teuthology.orchestra.run.smithi086.stdout: "op_active": 0, 2024-05-12T22:20:06.071 INFO:teuthology.orchestra.run.smithi086.stdout: "op_laggy": 0, 2024-05-12T22:20:06.071 INFO:teuthology.orchestra.run.smithi086.stdout: "op_send": 2605, 2024-05-12T22:20:06.071 INFO:teuthology.orchestra.run.smithi086.stdout: "op_send_bytes": 26, 2024-05-12T22:20:06.071 INFO:teuthology.orchestra.run.smithi086.stdout: "op_resend": 4, 2024-05-12T22:20:06.071 INFO:teuthology.orchestra.run.smithi086.stdout: "op_reply": 2605, 2024-05-12T22:20:06.071 INFO:teuthology.orchestra.run.smithi086.stdout: "op_latency": { 2024-05-12T22:20:06.072 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 2605, 2024-05-12T22:20:06.072 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 1.194974251, 2024-05-12T22:20:06.072 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000458723 2024-05-12T22:20:06.072 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:06.072 INFO:teuthology.orchestra.run.smithi086.stdout: "op_inflight": 0, 2024-05-12T22:20:06.072 INFO:teuthology.orchestra.run.smithi086.stdout: "oplen_avg": { 2024-05-12T22:20:06.072 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 2605, 2024-05-12T22:20:06.072 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 2606 2024-05-12T22:20:06.072 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:06.073 INFO:teuthology.orchestra.run.smithi086.stdout: "op": 2605, 2024-05-12T22:20:06.073 INFO:teuthology.orchestra.run.smithi086.stdout: "op_r": 2605, 2024-05-12T22:20:06.073 INFO:teuthology.orchestra.run.smithi086.stdout: "op_w": 0, 2024-05-12T22:20:06.073 INFO:teuthology.orchestra.run.smithi086.stdout: "op_rmw": 0, 2024-05-12T22:20:06.073 INFO:teuthology.orchestra.run.smithi086.stdout: "op_pg": 0, 2024-05-12T22:20:06.073 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_stat": 1665, 2024-05-12T22:20:06.073 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_create": 0, 2024-05-12T22:20:06.073 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_read": 937, 2024-05-12T22:20:06.073 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_write": 0, 2024-05-12T22:20:06.074 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_writefull": 0, 2024-05-12T22:20:06.074 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_writesame": 0, 2024-05-12T22:20:06.074 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_append": 0, 2024-05-12T22:20:06.074 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_zero": 0, 2024-05-12T22:20:06.074 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_truncate": 0, 2024-05-12T22:20:06.074 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_delete": 0, 2024-05-12T22:20:06.074 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_mapext": 0, 2024-05-12T22:20:06.074 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_sparse_read": 0, 2024-05-12T22:20:06.075 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_clonerange": 0, 2024-05-12T22:20:06.075 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_getxattr": 2, 2024-05-12T22:20:06.075 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_setxattr": 0, 2024-05-12T22:20:06.075 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_cmpxattr": 0, 2024-05-12T22:20:06.075 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_rmxattr": 0, 2024-05-12T22:20:06.075 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_resetxattrs": 0, 2024-05-12T22:20:06.075 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_call": 0, 2024-05-12T22:20:06.075 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_watch": 0, 2024-05-12T22:20:06.075 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_notify": 0, 2024-05-12T22:20:06.076 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_src_cmpxattr": 0, 2024-05-12T22:20:06.076 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_pgls": 0, 2024-05-12T22:20:06.076 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_pgls_filter": 0, 2024-05-12T22:20:06.076 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_other": 0, 2024-05-12T22:20:06.076 INFO:teuthology.orchestra.run.smithi086.stdout: "linger_active": 0, 2024-05-12T22:20:06.076 INFO:teuthology.orchestra.run.smithi086.stdout: "linger_send": 0, 2024-05-12T22:20:06.076 INFO:teuthology.orchestra.run.smithi086.stdout: "linger_resend": 0, 2024-05-12T22:20:06.076 INFO:teuthology.orchestra.run.smithi086.stdout: "linger_ping": 0, 2024-05-12T22:20:06.077 INFO:teuthology.orchestra.run.smithi086.stdout: "poolop_active": 0, 2024-05-12T22:20:06.077 INFO:teuthology.orchestra.run.smithi086.stdout: "poolop_send": 0, 2024-05-12T22:20:06.077 INFO:teuthology.orchestra.run.smithi086.stdout: "poolop_resend": 0, 2024-05-12T22:20:06.077 INFO:teuthology.orchestra.run.smithi086.stdout: "poolstat_active": 0, 2024-05-12T22:20:06.077 INFO:teuthology.orchestra.run.smithi086.stdout: "poolstat_send": 0, 2024-05-12T22:20:06.077 INFO:teuthology.orchestra.run.smithi086.stdout: "poolstat_resend": 0, 2024-05-12T22:20:06.077 INFO:teuthology.orchestra.run.smithi086.stdout: "statfs_active": 0, 2024-05-12T22:20:06.077 INFO:teuthology.orchestra.run.smithi086.stdout: "statfs_send": 0, 2024-05-12T22:20:06.078 INFO:teuthology.orchestra.run.smithi086.stdout: "statfs_resend": 0, 2024-05-12T22:20:06.078 INFO:teuthology.orchestra.run.smithi086.stdout: "command_active": 0, 2024-05-12T22:20:06.078 INFO:teuthology.orchestra.run.smithi086.stdout: "command_send": 0, 2024-05-12T22:20:06.078 INFO:teuthology.orchestra.run.smithi086.stdout: "command_resend": 0, 2024-05-12T22:20:06.078 INFO:teuthology.orchestra.run.smithi086.stdout: "map_epoch": 97, 2024-05-12T22:20:06.078 INFO:teuthology.orchestra.run.smithi086.stdout: "map_full": 0, 2024-05-12T22:20:06.078 INFO:teuthology.orchestra.run.smithi086.stdout: "map_inc": 11, 2024-05-12T22:20:06.078 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_sessions": 8, 2024-05-12T22:20:06.078 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_session_open": 8, 2024-05-12T22:20:06.079 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_session_close": 0, 2024-05-12T22:20:06.079 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_laggy": 0, 2024-05-12T22:20:06.079 INFO:teuthology.orchestra.run.smithi086.stdout: "omap_wr": 0, 2024-05-12T22:20:06.079 INFO:teuthology.orchestra.run.smithi086.stdout: "omap_rd": 2, 2024-05-12T22:20:06.079 INFO:teuthology.orchestra.run.smithi086.stdout: "omap_del": 0 2024-05-12T22:20:06.079 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:06.079 INFO:teuthology.orchestra.run.smithi086.stdout: "oft": { 2024-05-12T22:20:06.079 INFO:teuthology.orchestra.run.smithi086.stdout: "omap_total_objs": 0, 2024-05-12T22:20:06.080 INFO:teuthology.orchestra.run.smithi086.stdout: "omap_total_kv_pairs": 0, 2024-05-12T22:20:06.080 INFO:teuthology.orchestra.run.smithi086.stdout: "omap_total_updates": 0, 2024-05-12T22:20:06.080 INFO:teuthology.orchestra.run.smithi086.stdout: "omap_total_removes": 0 2024-05-12T22:20:06.080 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:06.080 INFO:teuthology.orchestra.run.smithi086.stdout: "osd-slow-ops": { 2024-05-12T22:20:06.080 INFO:teuthology.orchestra.run.smithi086.stdout: "slow_ops_count": 0 2024-05-12T22:20:06.080 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:06.080 INFO:teuthology.orchestra.run.smithi086.stdout: "purge_queue": { 2024-05-12T22:20:06.080 INFO:teuthology.orchestra.run.smithi086.stdout: "pq_executing_ops": 0, 2024-05-12T22:20:06.081 INFO:teuthology.orchestra.run.smithi086.stdout: "pq_executing_ops_high_water": 0, 2024-05-12T22:20:06.081 INFO:teuthology.orchestra.run.smithi086.stdout: "pq_executing": 0, 2024-05-12T22:20:06.081 INFO:teuthology.orchestra.run.smithi086.stdout: "pq_executing_high_water": 0, 2024-05-12T22:20:06.081 INFO:teuthology.orchestra.run.smithi086.stdout: "pq_executed_ops": 0, 2024-05-12T22:20:06.081 INFO:teuthology.orchestra.run.smithi086.stdout: "pq_executed": 0, 2024-05-12T22:20:06.081 INFO:teuthology.orchestra.run.smithi086.stdout: "pq_item_in_journal": 0 2024-05-12T22:20:06.081 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:06.081 INFO:teuthology.orchestra.run.smithi086.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-05-12T22:20:06.081 INFO:teuthology.orchestra.run.smithi086.stdout: "val": 0, 2024-05-12T22:20:06.082 INFO:teuthology.orchestra.run.smithi086.stdout: "max": 104857600, 2024-05-12T22:20:06.082 INFO:teuthology.orchestra.run.smithi086.stdout: "get_started": 0, 2024-05-12T22:20:06.082 INFO:teuthology.orchestra.run.smithi086.stdout: "get": 2862, 2024-05-12T22:20:06.082 INFO:teuthology.orchestra.run.smithi086.stdout: "get_sum": 69483620, 2024-05-12T22:20:06.082 INFO:teuthology.orchestra.run.smithi086.stdout: "get_or_fail_fail": 0, 2024-05-12T22:20:06.082 INFO:teuthology.orchestra.run.smithi086.stdout: "get_or_fail_success": 2862, 2024-05-12T22:20:06.083 INFO:teuthology.orchestra.run.smithi086.stdout: "take": 0, 2024-05-12T22:20:06.083 INFO:teuthology.orchestra.run.smithi086.stdout: "take_sum": 0, 2024-05-12T22:20:06.083 INFO:teuthology.orchestra.run.smithi086.stdout: "put": 2862, 2024-05-12T22:20:06.083 INFO:teuthology.orchestra.run.smithi086.stdout: "put_sum": 69483620, 2024-05-12T22:20:06.083 INFO:teuthology.orchestra.run.smithi086.stdout: "wait": { 2024-05-12T22:20:06.083 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-05-12T22:20:06.083 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-05-12T22:20:06.083 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-05-12T22:20:06.083 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-05-12T22:20:06.084 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:06.084 INFO:teuthology.orchestra.run.smithi086.stdout: "throttle-objecter_bytes": { 2024-05-12T22:20:06.084 INFO:teuthology.orchestra.run.smithi086.stdout: "val": 0, 2024-05-12T22:20:06.084 INFO:teuthology.orchestra.run.smithi086.stdout: "max": 104857600, 2024-05-12T22:20:06.084 INFO:teuthology.orchestra.run.smithi086.stdout: "get_started": 0, 2024-05-12T22:20:06.084 INFO:teuthology.orchestra.run.smithi086.stdout: "get": 0, 2024-05-12T22:20:06.084 INFO:teuthology.orchestra.run.smithi086.stdout: "get_sum": 0, 2024-05-12T22:20:06.085 INFO:teuthology.orchestra.run.smithi086.stdout: "get_or_fail_fail": 0, 2024-05-12T22:20:06.085 INFO:teuthology.orchestra.run.smithi086.stdout: "get_or_fail_success": 0, 2024-05-12T22:20:06.085 INFO:teuthology.orchestra.run.smithi086.stdout: "take": 2605, 2024-05-12T22:20:06.085 INFO:teuthology.orchestra.run.smithi086.stdout: "take_sum": 67264897, 2024-05-12T22:20:06.085 INFO:teuthology.orchestra.run.smithi086.stdout: "put": 2605, 2024-05-12T22:20:06.085 INFO:teuthology.orchestra.run.smithi086.stdout: "put_sum": 67264897, 2024-05-12T22:20:06.085 INFO:teuthology.orchestra.run.smithi086.stdout: "wait": { 2024-05-12T22:20:06.085 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-05-12T22:20:06.086 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-05-12T22:20:06.086 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-05-12T22:20:06.086 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-05-12T22:20:06.086 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:06.086 INFO:teuthology.orchestra.run.smithi086.stdout: "throttle-objecter_ops": { 2024-05-12T22:20:06.086 INFO:teuthology.orchestra.run.smithi086.stdout: "val": 0, 2024-05-12T22:20:06.086 INFO:teuthology.orchestra.run.smithi086.stdout: "max": 1024, 2024-05-12T22:20:06.087 INFO:teuthology.orchestra.run.smithi086.stdout: "get_started": 0, 2024-05-12T22:20:06.087 INFO:teuthology.orchestra.run.smithi086.stdout: "get": 0, 2024-05-12T22:20:06.087 INFO:teuthology.orchestra.run.smithi086.stdout: "get_sum": 0, 2024-05-12T22:20:06.087 INFO:teuthology.orchestra.run.smithi086.stdout: "get_or_fail_fail": 0, 2024-05-12T22:20:06.087 INFO:teuthology.orchestra.run.smithi086.stdout: "get_or_fail_success": 0, 2024-05-12T22:20:06.087 INFO:teuthology.orchestra.run.smithi086.stdout: "take": 2605, 2024-05-12T22:20:06.087 INFO:teuthology.orchestra.run.smithi086.stdout: "take_sum": 2605, 2024-05-12T22:20:06.087 INFO:teuthology.orchestra.run.smithi086.stdout: "put": 2605, 2024-05-12T22:20:06.088 INFO:teuthology.orchestra.run.smithi086.stdout: "put_sum": 2605, 2024-05-12T22:20:06.088 INFO:teuthology.orchestra.run.smithi086.stdout: "wait": { 2024-05-12T22:20:06.088 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-05-12T22:20:06.088 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-05-12T22:20:06.088 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-05-12T22:20:06.088 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-05-12T22:20:06.088 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:06.089 INFO:teuthology.orchestra.run.smithi086.stdout: "throttle-write_buf_throttle": { 2024-05-12T22:20:06.089 INFO:teuthology.orchestra.run.smithi086.stdout: "val": 0, 2024-05-12T22:20:06.089 INFO:teuthology.orchestra.run.smithi086.stdout: "max": 3758096384, 2024-05-12T22:20:06.089 INFO:teuthology.orchestra.run.smithi086.stdout: "get_started": 0, 2024-05-12T22:20:06.089 INFO:teuthology.orchestra.run.smithi086.stdout: "get": 0, 2024-05-12T22:20:06.089 INFO:teuthology.orchestra.run.smithi086.stdout: "get_sum": 0, 2024-05-12T22:20:06.089 INFO:teuthology.orchestra.run.smithi086.stdout: "get_or_fail_fail": 0, 2024-05-12T22:20:06.089 INFO:teuthology.orchestra.run.smithi086.stdout: "get_or_fail_success": 0, 2024-05-12T22:20:06.090 INFO:teuthology.orchestra.run.smithi086.stdout: "take": 0, 2024-05-12T22:20:06.090 INFO:teuthology.orchestra.run.smithi086.stdout: "take_sum": 0, 2024-05-12T22:20:06.090 INFO:teuthology.orchestra.run.smithi086.stdout: "put": 0, 2024-05-12T22:20:06.090 INFO:teuthology.orchestra.run.smithi086.stdout: "put_sum": 0, 2024-05-12T22:20:06.090 INFO:teuthology.orchestra.run.smithi086.stdout: "wait": { 2024-05-12T22:20:06.090 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-05-12T22:20:06.090 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-05-12T22:20:06.091 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-05-12T22:20:06.091 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-05-12T22:20:06.091 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:06.091 INFO:teuthology.orchestra.run.smithi086.stdout: "throttle-write_buf_throttle-0x564e63f6a6e0": { 2024-05-12T22:20:06.091 INFO:teuthology.orchestra.run.smithi086.stdout: "val": 0, 2024-05-12T22:20:06.091 INFO:teuthology.orchestra.run.smithi086.stdout: "max": 3758096384, 2024-05-12T22:20:06.091 INFO:teuthology.orchestra.run.smithi086.stdout: "get_started": 0, 2024-05-12T22:20:06.092 INFO:teuthology.orchestra.run.smithi086.stdout: "get": 0, 2024-05-12T22:20:06.092 INFO:teuthology.orchestra.run.smithi086.stdout: "get_sum": 0, 2024-05-12T22:20:06.092 INFO:teuthology.orchestra.run.smithi086.stdout: "get_or_fail_fail": 0, 2024-05-12T22:20:06.092 INFO:teuthology.orchestra.run.smithi086.stdout: "get_or_fail_success": 0, 2024-05-12T22:20:06.092 INFO:teuthology.orchestra.run.smithi086.stdout: "take": 0, 2024-05-12T22:20:06.092 INFO:teuthology.orchestra.run.smithi086.stdout: "take_sum": 0, 2024-05-12T22:20:06.093 INFO:teuthology.orchestra.run.smithi086.stdout: "put": 0, 2024-05-12T22:20:06.093 INFO:teuthology.orchestra.run.smithi086.stdout: "put_sum": 0, 2024-05-12T22:20:06.093 INFO:teuthology.orchestra.run.smithi086.stdout: "wait": { 2024-05-12T22:20:06.093 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-05-12T22:20:06.093 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-05-12T22:20:06.093 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-05-12T22:20:06.094 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-05-12T22:20:06.094 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-05-12T22:20:06.094 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-05-12T22:20:06.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:20:06 smithi186 ceph-mon[34785]: pgmap v344: 97 pgs: 97 active+clean; 91 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-05-12T22:20:06.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:20:06 smithi086 ceph-mon[34297]: pgmap v344: 97 pgs: 97 active+clean; 91 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-05-12T22:20:06.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:20:06 smithi028 ceph-mon[29847]: pgmap v344: 97 pgs: 97 active+clean; 91 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-05-12T22:20:06.901 INFO:tasks.check_counter:Daemon mds.h mds.exported=0 2024-05-12T22:20:06.901 INFO:tasks.check_counter:Daemon mds.h mds.imported=0 2024-05-12T22:20:06.901 INFO:tasks.check_counter:Daemon mds.h mds_cache.dir_update=0 2024-05-12T22:20:06.901 INFO:tasks.check_counter:Daemon mds.h mds_cache.dir_update_receipt=0 2024-05-12T22:20:06.901 INFO:tasks.check_counter:Daemon mds.h mds.root_rsnaps=6 2024-05-12T22:20:06.902 INFO:tasks.check_counter:Daemon mds.h mds_server.req_mksnap_latency.avgcount=0 2024-05-12T22:20:06.902 INFO:tasks.check_counter:Daemon mds.h mds_server.req_rmsnap_latency.avgcount=0 2024-05-12T22:20:06.902 INFO:tasks.check_counter:Daemon mds.h mds.dir_split=0 2024-05-12T22:20:06.902 DEBUG:tasks.check_counter:Getting stats from k 2024-05-12T22:20:06.902 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f72fecff68e1d400c4568684327c900485c20d6a shell --fsid 6b92e7b4-10aa-11ef-bc98-c7b262605968 -- ceph daemon mds.k perf dump 2024-05-12T22:20:07.143 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/mon.b/config 2024-05-12T22:20:08.632 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:20:08 smithi086 ceph-mon[34297]: pgmap v345: 97 pgs: 97 active+clean; 91 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-05-12T22:20:08.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:20:08 smithi186 ceph-mon[34785]: pgmap v345: 97 pgs: 97 active+clean; 91 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-05-12T22:20:08.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:20:08 smithi028 ceph-mon[29847]: pgmap v345: 97 pgs: 97 active+clean; 91 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-05-12T22:20:08.918 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-05-12T22:20:08.919 INFO:teuthology.orchestra.run.smithi086.stdout: "AsyncMessenger::Worker-0": { 2024-05-12T22:20:08.919 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_recv_messages": 0, 2024-05-12T22:20:08.919 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_send_messages": 2, 2024-05-12T22:20:08.919 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_recv_bytes": 0, 2024-05-12T22:20:08.920 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_send_bytes": 0, 2024-05-12T22:20:08.920 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_created_connections": 4, 2024-05-12T22:20:08.920 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_active_connections": 0, 2024-05-12T22:20:08.920 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_running_total_time": 0.000869438, 2024-05-12T22:20:08.920 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_running_send_time": 0.000000000, 2024-05-12T22:20:08.920 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_running_recv_time": 0.000101349, 2024-05-12T22:20:08.920 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-05-12T22:20:08.921 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_send_messages_queue_lat": { 2024-05-12T22:20:08.921 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-05-12T22:20:08.921 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-05-12T22:20:08.921 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-05-12T22:20:08.921 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:08.921 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_handle_ack_lat": { 2024-05-12T22:20:08.922 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-05-12T22:20:08.922 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-05-12T22:20:08.922 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-05-12T22:20:08.922 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:08.922 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_recv_encrypted_bytes": 0, 2024-05-12T22:20:08.922 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_send_encrypted_bytes": 0 2024-05-12T22:20:08.923 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:08.923 INFO:teuthology.orchestra.run.smithi086.stdout: "AsyncMessenger::Worker-1": { 2024-05-12T22:20:08.923 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_recv_messages": 240, 2024-05-12T22:20:08.923 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_send_messages": 228, 2024-05-12T22:20:08.923 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_recv_bytes": 1723584, 2024-05-12T22:20:08.923 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_send_bytes": 140704, 2024-05-12T22:20:08.923 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_created_connections": 2, 2024-05-12T22:20:08.923 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_active_connections": 1, 2024-05-12T22:20:08.923 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_running_total_time": 0.046055728, 2024-05-12T22:20:08.924 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_running_send_time": 0.014275372, 2024-05-12T22:20:08.924 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_running_recv_time": 0.035016014, 2024-05-12T22:20:08.924 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_running_fast_dispatch_time": 0.005599915, 2024-05-12T22:20:08.924 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_send_messages_queue_lat": { 2024-05-12T22:20:08.924 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 227, 2024-05-12T22:20:08.924 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.012424159, 2024-05-12T22:20:08.924 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000054731 2024-05-12T22:20:08.924 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:08.924 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_handle_ack_lat": { 2024-05-12T22:20:08.925 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-05-12T22:20:08.925 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-05-12T22:20:08.925 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-05-12T22:20:08.925 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:08.925 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_recv_encrypted_bytes": 1723584, 2024-05-12T22:20:08.925 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_send_encrypted_bytes": 140704 2024-05-12T22:20:08.925 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:08.925 INFO:teuthology.orchestra.run.smithi086.stdout: "AsyncMessenger::Worker-2": { 2024-05-12T22:20:08.925 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_recv_messages": 6, 2024-05-12T22:20:08.926 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_send_messages": 176, 2024-05-12T22:20:08.926 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_recv_bytes": 3392, 2024-05-12T22:20:08.926 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_send_bytes": 186672, 2024-05-12T22:20:08.926 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_created_connections": 5, 2024-05-12T22:20:08.926 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_active_connections": 3, 2024-05-12T22:20:08.926 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_running_total_time": 0.011023318, 2024-05-12T22:20:08.926 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_running_send_time": 0.007228468, 2024-05-12T22:20:08.926 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_running_recv_time": 0.001495940, 2024-05-12T22:20:08.926 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-05-12T22:20:08.927 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_send_messages_queue_lat": { 2024-05-12T22:20:08.927 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 174, 2024-05-12T22:20:08.927 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.027994895, 2024-05-12T22:20:08.927 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000160890 2024-05-12T22:20:08.927 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:08.927 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_handle_ack_lat": { 2024-05-12T22:20:08.927 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-05-12T22:20:08.927 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-05-12T22:20:08.928 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-05-12T22:20:08.928 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:08.928 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_recv_encrypted_bytes": 3392, 2024-05-12T22:20:08.928 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_send_encrypted_bytes": 186672 2024-05-12T22:20:08.928 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:08.928 INFO:teuthology.orchestra.run.smithi086.stdout: "cct": { 2024-05-12T22:20:08.928 INFO:teuthology.orchestra.run.smithi086.stdout: "total_workers": 0, 2024-05-12T22:20:08.928 INFO:teuthology.orchestra.run.smithi086.stdout: "unhealthy_workers": 0 2024-05-12T22:20:08.929 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:08.929 INFO:teuthology.orchestra.run.smithi086.stdout: "mempool": { 2024-05-12T22:20:08.929 INFO:teuthology.orchestra.run.smithi086.stdout: "bloom_filter_bytes": 0, 2024-05-12T22:20:08.929 INFO:teuthology.orchestra.run.smithi086.stdout: "bloom_filter_items": 0, 2024-05-12T22:20:08.929 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_alloc_bytes": 0, 2024-05-12T22:20:08.929 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_alloc_items": 0, 2024-05-12T22:20:08.929 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_cache_data_bytes": 0, 2024-05-12T22:20:08.929 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_cache_data_items": 0, 2024-05-12T22:20:08.929 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_cache_onode_bytes": 0, 2024-05-12T22:20:08.930 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_cache_onode_items": 0, 2024-05-12T22:20:08.930 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_cache_meta_bytes": 0, 2024-05-12T22:20:08.930 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_cache_meta_items": 0, 2024-05-12T22:20:08.930 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_cache_other_bytes": 0, 2024-05-12T22:20:08.930 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_cache_other_items": 0, 2024-05-12T22:20:08.930 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_cache_buffer_bytes": 0, 2024-05-12T22:20:08.930 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_cache_buffer_items": 0, 2024-05-12T22:20:08.930 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_extent_bytes": 0, 2024-05-12T22:20:08.931 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_extent_items": 0, 2024-05-12T22:20:08.931 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_blob_bytes": 0, 2024-05-12T22:20:08.931 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_blob_items": 0, 2024-05-12T22:20:08.931 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_shared_blob_bytes": 0, 2024-05-12T22:20:08.931 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_shared_blob_items": 0, 2024-05-12T22:20:08.931 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_inline_bl_bytes": 0, 2024-05-12T22:20:08.931 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_inline_bl_items": 0, 2024-05-12T22:20:08.931 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_fsck_bytes": 0, 2024-05-12T22:20:08.931 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_fsck_items": 0, 2024-05-12T22:20:08.932 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_txc_bytes": 0, 2024-05-12T22:20:08.932 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_txc_items": 0, 2024-05-12T22:20:08.932 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_writing_deferred_bytes": 0, 2024-05-12T22:20:08.932 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_writing_deferred_items": 0, 2024-05-12T22:20:08.932 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_writing_bytes": 0, 2024-05-12T22:20:08.932 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_writing_items": 0, 2024-05-12T22:20:08.932 INFO:teuthology.orchestra.run.smithi086.stdout: "bluefs_bytes": 0, 2024-05-12T22:20:08.932 INFO:teuthology.orchestra.run.smithi086.stdout: "bluefs_items": 0, 2024-05-12T22:20:08.932 INFO:teuthology.orchestra.run.smithi086.stdout: "bluefs_file_reader_bytes": 0, 2024-05-12T22:20:08.933 INFO:teuthology.orchestra.run.smithi086.stdout: "bluefs_file_reader_items": 0, 2024-05-12T22:20:08.933 INFO:teuthology.orchestra.run.smithi086.stdout: "bluefs_file_writer_bytes": 0, 2024-05-12T22:20:08.933 INFO:teuthology.orchestra.run.smithi086.stdout: "bluefs_file_writer_items": 0, 2024-05-12T22:20:08.933 INFO:teuthology.orchestra.run.smithi086.stdout: "buffer_anon_bytes": 148732, 2024-05-12T22:20:08.933 INFO:teuthology.orchestra.run.smithi086.stdout: "buffer_anon_items": 31, 2024-05-12T22:20:08.933 INFO:teuthology.orchestra.run.smithi086.stdout: "buffer_meta_bytes": 0, 2024-05-12T22:20:08.933 INFO:teuthology.orchestra.run.smithi086.stdout: "buffer_meta_items": 0, 2024-05-12T22:20:08.933 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_bytes": 0, 2024-05-12T22:20:08.934 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_items": 0, 2024-05-12T22:20:08.934 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_mapbl_bytes": 0, 2024-05-12T22:20:08.934 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_mapbl_items": 0, 2024-05-12T22:20:08.934 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_pglog_bytes": 0, 2024-05-12T22:20:08.934 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_pglog_items": 0, 2024-05-12T22:20:08.934 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap_bytes": 0, 2024-05-12T22:20:08.934 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap_items": 0, 2024-05-12T22:20:08.934 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap_mapping_bytes": 0, 2024-05-12T22:20:08.934 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap_mapping_items": 0, 2024-05-12T22:20:08.935 INFO:teuthology.orchestra.run.smithi086.stdout: "pgmap_bytes": 0, 2024-05-12T22:20:08.935 INFO:teuthology.orchestra.run.smithi086.stdout: "pgmap_items": 0, 2024-05-12T22:20:08.935 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_co_bytes": 928, 2024-05-12T22:20:08.935 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_co_items": 2, 2024-05-12T22:20:08.935 INFO:teuthology.orchestra.run.smithi086.stdout: "unittest_1_bytes": 0, 2024-05-12T22:20:08.935 INFO:teuthology.orchestra.run.smithi086.stdout: "unittest_1_items": 0, 2024-05-12T22:20:08.935 INFO:teuthology.orchestra.run.smithi086.stdout: "unittest_2_bytes": 0, 2024-05-12T22:20:08.935 INFO:teuthology.orchestra.run.smithi086.stdout: "unittest_2_items": 0 2024-05-12T22:20:08.935 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-05-12T22:20:08.936 INFO:teuthology.orchestra.run.smithi086.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-05-12T22:20:08.936 INFO:teuthology.orchestra.run.smithi086.stdout: "val": 0, 2024-05-12T22:20:08.936 INFO:teuthology.orchestra.run.smithi086.stdout: "max": 104857600, 2024-05-12T22:20:08.936 INFO:teuthology.orchestra.run.smithi086.stdout: "get_started": 0, 2024-05-12T22:20:08.936 INFO:teuthology.orchestra.run.smithi086.stdout: "get": 242, 2024-05-12T22:20:08.936 INFO:teuthology.orchestra.run.smithi086.stdout: "get_sum": 1689759, 2024-05-12T22:20:08.936 INFO:teuthology.orchestra.run.smithi086.stdout: "get_or_fail_fail": 0, 2024-05-12T22:20:08.936 INFO:teuthology.orchestra.run.smithi086.stdout: "get_or_fail_success": 242, 2024-05-12T22:20:08.936 INFO:teuthology.orchestra.run.smithi086.stdout: "take": 0, 2024-05-12T22:20:08.937 INFO:teuthology.orchestra.run.smithi086.stdout: "take_sum": 0, 2024-05-12T22:20:08.937 INFO:teuthology.orchestra.run.smithi086.stdout: "put": 242, 2024-05-12T22:20:08.937 INFO:teuthology.orchestra.run.smithi086.stdout: "put_sum": 1689759, 2024-05-12T22:20:08.937 INFO:teuthology.orchestra.run.smithi086.stdout: "wait": { 2024-05-12T22:20:08.937 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-05-12T22:20:08.937 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-05-12T22:20:08.937 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-05-12T22:20:08.937 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-05-12T22:20:08.937 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-05-12T22:20:08.938 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-05-12T22:20:09.492 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:20:09 smithi086 ceph-mon[34297]: osdmap e98: 12 total, 12 up, 12 in 2024-05-12T22:20:09.570 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.k 2024-05-12T22:20:09.571 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.k 2024-05-12T22:20:09.571 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.k 2024-05-12T22:20:09.571 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.k 2024-05-12T22:20:09.571 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.k 2024-05-12T22:20:09.571 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.k 2024-05-12T22:20:09.571 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.k 2024-05-12T22:20:09.571 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.k 2024-05-12T22:20:09.571 DEBUG:tasks.check_counter:Getting stats from c 2024-05-12T22:20:09.572 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f72fecff68e1d400c4568684327c900485c20d6a shell --fsid 6b92e7b4-10aa-11ef-bc98-c7b262605968 -- ceph daemon mds.c perf dump 2024-05-12T22:20:09.600 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:20:09 smithi186 ceph-mon[34785]: osdmap e98: 12 total, 12 up, 12 in 2024-05-12T22:20:09.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:20:09 smithi028 ceph-mon[29847]: osdmap e98: 12 total, 12 up, 12 in 2024-05-12T22:20:09.827 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/mon.c/config 2024-05-12T22:20:10.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:20:10 smithi186 ceph-mon[34785]: pgmap v347: 97 pgs: 97 active+clean; 91 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-05-12T22:20:10.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:20:10 smithi086 ceph-mon[34297]: pgmap v347: 97 pgs: 97 active+clean; 91 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-05-12T22:20:10.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:20:10 smithi028 ceph-mon[29847]: pgmap v347: 97 pgs: 97 active+clean; 91 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-05-12T22:20:11.574 INFO:teuthology.orchestra.run.smithi186.stdout:{ 2024-05-12T22:20:11.574 INFO:teuthology.orchestra.run.smithi186.stdout: "AsyncMessenger::Worker-0": { 2024-05-12T22:20:11.574 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_messages": 263, 2024-05-12T22:20:11.575 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_messages": 245, 2024-05-12T22:20:11.575 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_bytes": 1765010, 2024-05-12T22:20:11.575 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_bytes": 142500, 2024-05-12T22:20:11.575 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_created_connections": 5, 2024-05-12T22:20:11.575 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_active_connections": 1, 2024-05-12T22:20:11.575 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_total_time": 0.051866554, 2024-05-12T22:20:11.575 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_send_time": 0.014877401, 2024-05-12T22:20:11.575 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_recv_time": 0.041522637, 2024-05-12T22:20:11.576 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_fast_dispatch_time": 0.005866178, 2024-05-12T22:20:11.576 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_messages_queue_lat": { 2024-05-12T22:20:11.576 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 245, 2024-05-12T22:20:11.576 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.014697569, 2024-05-12T22:20:11.576 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000059990 2024-05-12T22:20:11.576 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:11.576 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_handle_ack_lat": { 2024-05-12T22:20:11.576 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-05-12T22:20:11.576 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-05-12T22:20:11.577 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-05-12T22:20:11.577 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:11.577 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_encrypted_bytes": 1764176, 2024-05-12T22:20:11.577 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_encrypted_bytes": 142176 2024-05-12T22:20:11.577 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:11.577 INFO:teuthology.orchestra.run.smithi186.stdout: "AsyncMessenger::Worker-1": { 2024-05-12T22:20:11.577 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_messages": 748, 2024-05-12T22:20:11.577 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_messages": 918, 2024-05-12T22:20:11.578 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_bytes": 183858, 2024-05-12T22:20:11.578 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_bytes": 701841, 2024-05-12T22:20:11.578 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_created_connections": 7, 2024-05-12T22:20:11.578 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_active_connections": 5, 2024-05-12T22:20:11.578 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_total_time": 0.094378613, 2024-05-12T22:20:11.578 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_send_time": 0.029114666, 2024-05-12T22:20:11.578 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_recv_time": 0.022520973, 2024-05-12T22:20:11.578 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_fast_dispatch_time": 0.033592968, 2024-05-12T22:20:11.578 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_messages_queue_lat": { 2024-05-12T22:20:11.579 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 915, 2024-05-12T22:20:11.579 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.059799004, 2024-05-12T22:20:11.579 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000065354 2024-05-12T22:20:11.579 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:11.579 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_handle_ack_lat": { 2024-05-12T22:20:11.579 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-05-12T22:20:11.579 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-05-12T22:20:11.579 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-05-12T22:20:11.580 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:11.580 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_encrypted_bytes": 320, 2024-05-12T22:20:11.580 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_encrypted_bytes": 464608 2024-05-12T22:20:11.580 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:11.580 INFO:teuthology.orchestra.run.smithi186.stdout: "AsyncMessenger::Worker-2": { 2024-05-12T22:20:11.580 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_messages": 1631, 2024-05-12T22:20:11.580 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_messages": 1633, 2024-05-12T22:20:11.580 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_bytes": 490317, 2024-05-12T22:20:11.580 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_bytes": 518716, 2024-05-12T22:20:11.581 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_created_connections": 4, 2024-05-12T22:20:11.581 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_active_connections": 3, 2024-05-12T22:20:11.581 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_total_time": 0.179679464, 2024-05-12T22:20:11.581 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_send_time": 0.054544656, 2024-05-12T22:20:11.581 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_recv_time": 0.049730861, 2024-05-12T22:20:11.581 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_fast_dispatch_time": 0.059770787, 2024-05-12T22:20:11.581 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_messages_queue_lat": { 2024-05-12T22:20:11.581 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 1631, 2024-05-12T22:20:11.581 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.071155618, 2024-05-12T22:20:11.582 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000043626 2024-05-12T22:20:11.582 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:11.582 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_handle_ack_lat": { 2024-05-12T22:20:11.582 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-05-12T22:20:11.582 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-05-12T22:20:11.582 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-05-12T22:20:11.582 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:11.582 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_encrypted_bytes": 0, 2024-05-12T22:20:11.582 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_encrypted_bytes": 0 2024-05-12T22:20:11.583 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:11.583 INFO:teuthology.orchestra.run.smithi186.stdout: "cct": { 2024-05-12T22:20:11.583 INFO:teuthology.orchestra.run.smithi186.stdout: "total_workers": 1, 2024-05-12T22:20:11.583 INFO:teuthology.orchestra.run.smithi186.stdout: "unhealthy_workers": 0 2024-05-12T22:20:11.583 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:11.583 INFO:teuthology.orchestra.run.smithi186.stdout: "finisher-MDSRank": { 2024-05-12T22:20:11.583 INFO:teuthology.orchestra.run.smithi186.stdout: "queue_len": 0, 2024-05-12T22:20:11.583 INFO:teuthology.orchestra.run.smithi186.stdout: "complete_latency": { 2024-05-12T22:20:11.583 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 3963, 2024-05-12T22:20:11.584 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.123995423, 2024-05-12T22:20:11.584 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000031288 2024-05-12T22:20:11.584 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-05-12T22:20:11.584 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:11.584 INFO:teuthology.orchestra.run.smithi186.stdout: "finisher-PurgeQueue": { 2024-05-12T22:20:11.584 INFO:teuthology.orchestra.run.smithi186.stdout: "queue_len": 0, 2024-05-12T22:20:11.584 INFO:teuthology.orchestra.run.smithi186.stdout: "complete_latency": { 2024-05-12T22:20:11.584 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-05-12T22:20:11.584 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-05-12T22:20:11.585 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-05-12T22:20:11.585 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-05-12T22:20:11.585 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:11.585 INFO:teuthology.orchestra.run.smithi186.stdout: "mds": { 2024-05-12T22:20:11.585 INFO:teuthology.orchestra.run.smithi186.stdout: "request": 0, 2024-05-12T22:20:11.585 INFO:teuthology.orchestra.run.smithi186.stdout: "reply": 0, 2024-05-12T22:20:11.585 INFO:teuthology.orchestra.run.smithi186.stdout: "reply_latency": { 2024-05-12T22:20:11.585 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-05-12T22:20:11.585 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-05-12T22:20:11.586 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-05-12T22:20:11.586 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:11.586 INFO:teuthology.orchestra.run.smithi186.stdout: "slow_reply": 0, 2024-05-12T22:20:11.586 INFO:teuthology.orchestra.run.smithi186.stdout: "forward": 0, 2024-05-12T22:20:11.586 INFO:teuthology.orchestra.run.smithi186.stdout: "dir_fetch_complete": 0, 2024-05-12T22:20:11.586 INFO:teuthology.orchestra.run.smithi186.stdout: "dir_fetch_keys": 0, 2024-05-12T22:20:11.586 INFO:teuthology.orchestra.run.smithi186.stdout: "dir_commit": 0, 2024-05-12T22:20:11.586 INFO:teuthology.orchestra.run.smithi186.stdout: "dir_split": 0, 2024-05-12T22:20:11.586 INFO:teuthology.orchestra.run.smithi186.stdout: "dir_merge": 0, 2024-05-12T22:20:11.587 INFO:teuthology.orchestra.run.smithi186.stdout: "inodes": 0, 2024-05-12T22:20:11.587 INFO:teuthology.orchestra.run.smithi186.stdout: "inodes_top": 0, 2024-05-12T22:20:11.587 INFO:teuthology.orchestra.run.smithi186.stdout: "inodes_bottom": 0, 2024-05-12T22:20:11.587 INFO:teuthology.orchestra.run.smithi186.stdout: "inodes_pin_tail": 0, 2024-05-12T22:20:11.587 INFO:teuthology.orchestra.run.smithi186.stdout: "inodes_pinned": 0, 2024-05-12T22:20:11.587 INFO:teuthology.orchestra.run.smithi186.stdout: "inodes_expired": 0, 2024-05-12T22:20:11.587 INFO:teuthology.orchestra.run.smithi186.stdout: "inodes_with_caps": 0, 2024-05-12T22:20:11.587 INFO:teuthology.orchestra.run.smithi186.stdout: "caps": 0, 2024-05-12T22:20:11.588 INFO:teuthology.orchestra.run.smithi186.stdout: "subtrees": 1, 2024-05-12T22:20:11.588 INFO:teuthology.orchestra.run.smithi186.stdout: "traverse": 0, 2024-05-12T22:20:11.588 INFO:teuthology.orchestra.run.smithi186.stdout: "traverse_hit": 0, 2024-05-12T22:20:11.588 INFO:teuthology.orchestra.run.smithi186.stdout: "traverse_forward": 0, 2024-05-12T22:20:11.588 INFO:teuthology.orchestra.run.smithi186.stdout: "traverse_discover": 0, 2024-05-12T22:20:11.588 INFO:teuthology.orchestra.run.smithi186.stdout: "traverse_dir_fetch": 0, 2024-05-12T22:20:11.588 INFO:teuthology.orchestra.run.smithi186.stdout: "traverse_remote_ino": 0, 2024-05-12T22:20:11.588 INFO:teuthology.orchestra.run.smithi186.stdout: "traverse_lock": 0, 2024-05-12T22:20:11.588 INFO:teuthology.orchestra.run.smithi186.stdout: "load_cent": 0, 2024-05-12T22:20:11.589 INFO:teuthology.orchestra.run.smithi186.stdout: "q": 0, 2024-05-12T22:20:11.589 INFO:teuthology.orchestra.run.smithi186.stdout: "exported": 0, 2024-05-12T22:20:11.589 INFO:teuthology.orchestra.run.smithi186.stdout: "exported_inodes": 0, 2024-05-12T22:20:11.589 INFO:teuthology.orchestra.run.smithi186.stdout: "imported": 0, 2024-05-12T22:20:11.589 INFO:teuthology.orchestra.run.smithi186.stdout: "imported_inodes": 0, 2024-05-12T22:20:11.589 INFO:teuthology.orchestra.run.smithi186.stdout: "openino_dir_fetch": 0, 2024-05-12T22:20:11.589 INFO:teuthology.orchestra.run.smithi186.stdout: "openino_backtrace_fetch": 0, 2024-05-12T22:20:11.589 INFO:teuthology.orchestra.run.smithi186.stdout: "openino_peer_discover": 0, 2024-05-12T22:20:11.589 INFO:teuthology.orchestra.run.smithi186.stdout: "root_rfiles": 0, 2024-05-12T22:20:11.590 INFO:teuthology.orchestra.run.smithi186.stdout: "root_rbytes": 0, 2024-05-12T22:20:11.590 INFO:teuthology.orchestra.run.smithi186.stdout: "root_rsnaps": 0, 2024-05-12T22:20:11.590 INFO:teuthology.orchestra.run.smithi186.stdout: "scrub_backtrace_fetch": 0, 2024-05-12T22:20:11.590 INFO:teuthology.orchestra.run.smithi186.stdout: "scrub_set_tag": 0, 2024-05-12T22:20:11.590 INFO:teuthology.orchestra.run.smithi186.stdout: "scrub_backtrace_repaired": 0, 2024-05-12T22:20:11.590 INFO:teuthology.orchestra.run.smithi186.stdout: "scrub_inotable_repaired": 0, 2024-05-12T22:20:11.590 INFO:teuthology.orchestra.run.smithi186.stdout: "scrub_dir_inodes": 0, 2024-05-12T22:20:11.590 INFO:teuthology.orchestra.run.smithi186.stdout: "scrub_dir_base_inodes": 0, 2024-05-12T22:20:11.591 INFO:teuthology.orchestra.run.smithi186.stdout: "scrub_dirfrag_rstats": 0, 2024-05-12T22:20:11.591 INFO:teuthology.orchestra.run.smithi186.stdout: "scrub_file_inodes": 0, 2024-05-12T22:20:11.591 INFO:teuthology.orchestra.run.smithi186.stdout: "handle_inode_file_caps": 0, 2024-05-12T22:20:11.591 INFO:teuthology.orchestra.run.smithi186.stdout: "ceph_cap_op_revoke": 0, 2024-05-12T22:20:11.591 INFO:teuthology.orchestra.run.smithi186.stdout: "ceph_cap_op_grant": 0, 2024-05-12T22:20:11.591 INFO:teuthology.orchestra.run.smithi186.stdout: "ceph_cap_op_trunc": 0, 2024-05-12T22:20:11.591 INFO:teuthology.orchestra.run.smithi186.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-05-12T22:20:11.591 INFO:teuthology.orchestra.run.smithi186.stdout: "ceph_cap_op_flush_ack": 0, 2024-05-12T22:20:11.591 INFO:teuthology.orchestra.run.smithi186.stdout: "handle_client_caps": 0, 2024-05-12T22:20:11.592 INFO:teuthology.orchestra.run.smithi186.stdout: "handle_client_caps_dirty": 0, 2024-05-12T22:20:11.592 INFO:teuthology.orchestra.run.smithi186.stdout: "handle_client_cap_release": 0, 2024-05-12T22:20:11.592 INFO:teuthology.orchestra.run.smithi186.stdout: "process_request_cap_release": 0 2024-05-12T22:20:11.592 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:11.592 INFO:teuthology.orchestra.run.smithi186.stdout: "mds_cache": { 2024-05-12T22:20:11.592 INFO:teuthology.orchestra.run.smithi186.stdout: "dir_update": 0, 2024-05-12T22:20:11.592 INFO:teuthology.orchestra.run.smithi186.stdout: "dir_update_receipt": 0, 2024-05-12T22:20:11.592 INFO:teuthology.orchestra.run.smithi186.stdout: "dir_try_discover": 0, 2024-05-12T22:20:11.592 INFO:teuthology.orchestra.run.smithi186.stdout: "dir_send_discover": 0, 2024-05-12T22:20:11.593 INFO:teuthology.orchestra.run.smithi186.stdout: "dir_handle_discover": 0, 2024-05-12T22:20:11.593 INFO:teuthology.orchestra.run.smithi186.stdout: "num_strays": 0, 2024-05-12T22:20:11.593 INFO:teuthology.orchestra.run.smithi186.stdout: "num_strays_delayed": 0, 2024-05-12T22:20:11.593 INFO:teuthology.orchestra.run.smithi186.stdout: "num_strays_enqueuing": 0, 2024-05-12T22:20:11.593 INFO:teuthology.orchestra.run.smithi186.stdout: "strays_created": 0, 2024-05-12T22:20:11.593 INFO:teuthology.orchestra.run.smithi186.stdout: "strays_enqueued": 0, 2024-05-12T22:20:11.593 INFO:teuthology.orchestra.run.smithi186.stdout: "strays_reintegrated": 0, 2024-05-12T22:20:11.593 INFO:teuthology.orchestra.run.smithi186.stdout: "strays_migrated": 0, 2024-05-12T22:20:11.593 INFO:teuthology.orchestra.run.smithi186.stdout: "num_recovering_processing": 0, 2024-05-12T22:20:11.594 INFO:teuthology.orchestra.run.smithi186.stdout: "num_recovering_enqueued": 0, 2024-05-12T22:20:11.594 INFO:teuthology.orchestra.run.smithi186.stdout: "num_recovering_prioritized": 0, 2024-05-12T22:20:11.594 INFO:teuthology.orchestra.run.smithi186.stdout: "recovery_started": 0, 2024-05-12T22:20:11.594 INFO:teuthology.orchestra.run.smithi186.stdout: "recovery_completed": 0, 2024-05-12T22:20:11.594 INFO:teuthology.orchestra.run.smithi186.stdout: "ireq_quiesce_path": 0, 2024-05-12T22:20:11.594 INFO:teuthology.orchestra.run.smithi186.stdout: "ireq_quiesce_inode": 0, 2024-05-12T22:20:11.594 INFO:teuthology.orchestra.run.smithi186.stdout: "ireq_enqueue_scrub": 0, 2024-05-12T22:20:11.594 INFO:teuthology.orchestra.run.smithi186.stdout: "ireq_exportdir": 0, 2024-05-12T22:20:11.594 INFO:teuthology.orchestra.run.smithi186.stdout: "ireq_flush": 0, 2024-05-12T22:20:11.595 INFO:teuthology.orchestra.run.smithi186.stdout: "ireq_fragmentdir": 0, 2024-05-12T22:20:11.595 INFO:teuthology.orchestra.run.smithi186.stdout: "ireq_fragstats": 0, 2024-05-12T22:20:11.595 INFO:teuthology.orchestra.run.smithi186.stdout: "ireq_inodestats": 0 2024-05-12T22:20:11.595 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:11.595 INFO:teuthology.orchestra.run.smithi186.stdout: "mds_log": { 2024-05-12T22:20:11.595 INFO:teuthology.orchestra.run.smithi186.stdout: "evlrg": 0, 2024-05-12T22:20:11.595 INFO:teuthology.orchestra.run.smithi186.stdout: "evadd": 0, 2024-05-12T22:20:11.595 INFO:teuthology.orchestra.run.smithi186.stdout: "evex": 0, 2024-05-12T22:20:11.596 INFO:teuthology.orchestra.run.smithi186.stdout: "evtrm": 0, 2024-05-12T22:20:11.596 INFO:teuthology.orchestra.run.smithi186.stdout: "ev": 11, 2024-05-12T22:20:11.596 INFO:teuthology.orchestra.run.smithi186.stdout: "evexg": 0, 2024-05-12T22:20:11.596 INFO:teuthology.orchestra.run.smithi186.stdout: "evexd": 0, 2024-05-12T22:20:11.596 INFO:teuthology.orchestra.run.smithi186.stdout: "segadd": 0, 2024-05-12T22:20:11.596 INFO:teuthology.orchestra.run.smithi186.stdout: "segex": 0, 2024-05-12T22:20:11.596 INFO:teuthology.orchestra.run.smithi186.stdout: "segtrm": 0, 2024-05-12T22:20:11.596 INFO:teuthology.orchestra.run.smithi186.stdout: "seg": 2, 2024-05-12T22:20:11.596 INFO:teuthology.orchestra.run.smithi186.stdout: "segmjr": 2, 2024-05-12T22:20:11.597 INFO:teuthology.orchestra.run.smithi186.stdout: "segexg": 0, 2024-05-12T22:20:11.597 INFO:teuthology.orchestra.run.smithi186.stdout: "segexd": 0, 2024-05-12T22:20:11.597 INFO:teuthology.orchestra.run.smithi186.stdout: "expos": 4194304, 2024-05-12T22:20:11.597 INFO:teuthology.orchestra.run.smithi186.stdout: "wrpos": 4197330, 2024-05-12T22:20:11.597 INFO:teuthology.orchestra.run.smithi186.stdout: "rdpos": 4197189, 2024-05-12T22:20:11.597 INFO:teuthology.orchestra.run.smithi186.stdout: "jlat": { 2024-05-12T22:20:11.597 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-05-12T22:20:11.597 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-05-12T22:20:11.597 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-05-12T22:20:11.598 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:11.598 INFO:teuthology.orchestra.run.smithi186.stdout: "replayed": 11 2024-05-12T22:20:11.598 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:11.598 INFO:teuthology.orchestra.run.smithi186.stdout: "mds_mem": { 2024-05-12T22:20:11.598 INFO:teuthology.orchestra.run.smithi186.stdout: "ino": 3, 2024-05-12T22:20:11.598 INFO:teuthology.orchestra.run.smithi186.stdout: "ino+": 3, 2024-05-12T22:20:11.598 INFO:teuthology.orchestra.run.smithi186.stdout: "ino-": 0, 2024-05-12T22:20:11.598 INFO:teuthology.orchestra.run.smithi186.stdout: "dir": 1, 2024-05-12T22:20:11.599 INFO:teuthology.orchestra.run.smithi186.stdout: "dir+": 1, 2024-05-12T22:20:11.599 INFO:teuthology.orchestra.run.smithi186.stdout: "dir-": 0, 2024-05-12T22:20:11.599 INFO:teuthology.orchestra.run.smithi186.stdout: "dn": 0, 2024-05-12T22:20:11.599 INFO:teuthology.orchestra.run.smithi186.stdout: "dn+": 0, 2024-05-12T22:20:11.599 INFO:teuthology.orchestra.run.smithi186.stdout: "dn-": 0, 2024-05-12T22:20:11.599 INFO:teuthology.orchestra.run.smithi186.stdout: "cap": 0, 2024-05-12T22:20:11.599 INFO:teuthology.orchestra.run.smithi186.stdout: "cap+": 0, 2024-05-12T22:20:11.599 INFO:teuthology.orchestra.run.smithi186.stdout: "cap-": 0, 2024-05-12T22:20:11.599 INFO:teuthology.orchestra.run.smithi186.stdout: "rss": 52928, 2024-05-12T22:20:11.600 INFO:teuthology.orchestra.run.smithi186.stdout: "heap": 215420 2024-05-12T22:20:11.600 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:11.600 INFO:teuthology.orchestra.run.smithi186.stdout: "mds_server": { 2024-05-12T22:20:11.600 INFO:teuthology.orchestra.run.smithi186.stdout: "dispatch_client_request": 0, 2024-05-12T22:20:11.600 INFO:teuthology.orchestra.run.smithi186.stdout: "dispatch_server_request": 0, 2024-05-12T22:20:11.600 INFO:teuthology.orchestra.run.smithi186.stdout: "handle_client_request": 0, 2024-05-12T22:20:11.600 INFO:teuthology.orchestra.run.smithi186.stdout: "handle_client_session": 0, 2024-05-12T22:20:11.600 INFO:teuthology.orchestra.run.smithi186.stdout: "handle_peer_request": 0, 2024-05-12T22:20:11.601 INFO:teuthology.orchestra.run.smithi186.stdout: "req_create_latency": { 2024-05-12T22:20:11.601 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-05-12T22:20:11.601 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-05-12T22:20:11.601 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-05-12T22:20:11.601 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:11.601 INFO:teuthology.orchestra.run.smithi186.stdout: "req_getattr_latency": { 2024-05-12T22:20:11.601 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-05-12T22:20:11.601 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-05-12T22:20:11.601 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-05-12T22:20:11.602 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:11.602 INFO:teuthology.orchestra.run.smithi186.stdout: "req_getfilelock_latency": { 2024-05-12T22:20:11.602 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-05-12T22:20:11.602 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-05-12T22:20:11.602 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-05-12T22:20:11.602 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:11.602 INFO:teuthology.orchestra.run.smithi186.stdout: "req_link_latency": { 2024-05-12T22:20:11.602 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-05-12T22:20:11.602 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-05-12T22:20:11.603 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-05-12T22:20:11.603 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:11.603 INFO:teuthology.orchestra.run.smithi186.stdout: "req_lookup_latency": { 2024-05-12T22:20:11.603 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-05-12T22:20:11.603 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-05-12T22:20:11.603 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-05-12T22:20:11.603 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:11.603 INFO:teuthology.orchestra.run.smithi186.stdout: "req_lookuphash_latency": { 2024-05-12T22:20:11.604 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-05-12T22:20:11.604 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-05-12T22:20:11.604 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-05-12T22:20:11.604 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:11.604 INFO:teuthology.orchestra.run.smithi186.stdout: "req_lookupino_latency": { 2024-05-12T22:20:11.604 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-05-12T22:20:11.604 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-05-12T22:20:11.604 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-05-12T22:20:11.604 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:11.605 INFO:teuthology.orchestra.run.smithi186.stdout: "req_lookupname_latency": { 2024-05-12T22:20:11.605 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-05-12T22:20:11.605 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-05-12T22:20:11.605 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-05-12T22:20:11.605 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:11.605 INFO:teuthology.orchestra.run.smithi186.stdout: "req_lookupparent_latency": { 2024-05-12T22:20:11.605 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-05-12T22:20:11.605 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-05-12T22:20:11.605 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-05-12T22:20:11.606 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:11.606 INFO:teuthology.orchestra.run.smithi186.stdout: "req_lookupsnap_latency": { 2024-05-12T22:20:11.606 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-05-12T22:20:11.606 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-05-12T22:20:11.606 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-05-12T22:20:11.606 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:11.606 INFO:teuthology.orchestra.run.smithi186.stdout: "req_lssnap_latency": { 2024-05-12T22:20:11.606 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-05-12T22:20:11.607 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-05-12T22:20:11.607 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-05-12T22:20:11.607 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:11.607 INFO:teuthology.orchestra.run.smithi186.stdout: "req_mkdir_latency": { 2024-05-12T22:20:11.607 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-05-12T22:20:11.608 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-05-12T22:20:11.608 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-05-12T22:20:11.608 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:11.608 INFO:teuthology.orchestra.run.smithi186.stdout: "req_mknod_latency": { 2024-05-12T22:20:11.608 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-05-12T22:20:11.608 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-05-12T22:20:11.609 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-05-12T22:20:11.609 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:11.609 INFO:teuthology.orchestra.run.smithi186.stdout: "req_mksnap_latency": { 2024-05-12T22:20:11.609 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-05-12T22:20:11.609 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-05-12T22:20:11.609 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-05-12T22:20:11.609 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:11.609 INFO:teuthology.orchestra.run.smithi186.stdout: "req_open_latency": { 2024-05-12T22:20:11.609 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-05-12T22:20:11.610 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-05-12T22:20:11.610 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-05-12T22:20:11.610 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:11.610 INFO:teuthology.orchestra.run.smithi186.stdout: "req_readdir_latency": { 2024-05-12T22:20:11.610 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-05-12T22:20:11.610 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-05-12T22:20:11.610 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-05-12T22:20:11.610 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:11.611 INFO:teuthology.orchestra.run.smithi186.stdout: "req_rename_latency": { 2024-05-12T22:20:11.611 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-05-12T22:20:11.611 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-05-12T22:20:11.611 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-05-12T22:20:11.611 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:11.611 INFO:teuthology.orchestra.run.smithi186.stdout: "req_renamesnap_latency": { 2024-05-12T22:20:11.611 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-05-12T22:20:11.611 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-05-12T22:20:11.611 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-05-12T22:20:11.612 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:11.612 INFO:teuthology.orchestra.run.smithi186.stdout: "req_snapdiff_latency": { 2024-05-12T22:20:11.612 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-05-12T22:20:11.612 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-05-12T22:20:11.612 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-05-12T22:20:11.612 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:11.612 INFO:teuthology.orchestra.run.smithi186.stdout: "req_rmdir_latency": { 2024-05-12T22:20:11.612 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-05-12T22:20:11.612 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-05-12T22:20:11.613 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-05-12T22:20:11.613 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:11.613 INFO:teuthology.orchestra.run.smithi186.stdout: "req_rmsnap_latency": { 2024-05-12T22:20:11.613 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-05-12T22:20:11.613 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-05-12T22:20:11.613 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-05-12T22:20:11.613 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:11.613 INFO:teuthology.orchestra.run.smithi186.stdout: "req_rmxattr_latency": { 2024-05-12T22:20:11.613 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-05-12T22:20:11.614 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-05-12T22:20:11.614 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-05-12T22:20:11.614 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:11.614 INFO:teuthology.orchestra.run.smithi186.stdout: "req_setattr_latency": { 2024-05-12T22:20:11.614 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-05-12T22:20:11.614 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-05-12T22:20:11.614 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-05-12T22:20:11.614 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:11.614 INFO:teuthology.orchestra.run.smithi186.stdout: "req_setdirlayout_latency": { 2024-05-12T22:20:11.615 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-05-12T22:20:11.615 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-05-12T22:20:11.615 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-05-12T22:20:11.615 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:11.615 INFO:teuthology.orchestra.run.smithi186.stdout: "req_setfilelock_latency": { 2024-05-12T22:20:11.615 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-05-12T22:20:11.615 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-05-12T22:20:11.615 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-05-12T22:20:11.615 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:11.616 INFO:teuthology.orchestra.run.smithi186.stdout: "req_setlayout_latency": { 2024-05-12T22:20:11.616 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-05-12T22:20:11.616 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-05-12T22:20:11.616 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-05-12T22:20:11.616 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:11.616 INFO:teuthology.orchestra.run.smithi186.stdout: "req_setxattr_latency": { 2024-05-12T22:20:11.616 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-05-12T22:20:11.616 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-05-12T22:20:11.617 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-05-12T22:20:11.617 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:11.617 INFO:teuthology.orchestra.run.smithi186.stdout: "req_symlink_latency": { 2024-05-12T22:20:11.617 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-05-12T22:20:11.617 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-05-12T22:20:11.617 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-05-12T22:20:11.617 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:11.617 INFO:teuthology.orchestra.run.smithi186.stdout: "req_unlink_latency": { 2024-05-12T22:20:11.617 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-05-12T22:20:11.618 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-05-12T22:20:11.618 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-05-12T22:20:11.618 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:11.618 INFO:teuthology.orchestra.run.smithi186.stdout: "cap_revoke_eviction": 0, 2024-05-12T22:20:11.618 INFO:teuthology.orchestra.run.smithi186.stdout: "cap_acquisition_throttle": 0, 2024-05-12T22:20:11.618 INFO:teuthology.orchestra.run.smithi186.stdout: "req_getvxattr_latency": { 2024-05-12T22:20:11.618 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-05-12T22:20:11.618 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-05-12T22:20:11.618 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-05-12T22:20:11.619 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-05-12T22:20:11.619 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:11.619 INFO:teuthology.orchestra.run.smithi186.stdout: "mds_sessions": { 2024-05-12T22:20:11.619 INFO:teuthology.orchestra.run.smithi186.stdout: "session_count": 1, 2024-05-12T22:20:11.619 INFO:teuthology.orchestra.run.smithi186.stdout: "session_add": 5, 2024-05-12T22:20:11.619 INFO:teuthology.orchestra.run.smithi186.stdout: "session_remove": 4, 2024-05-12T22:20:11.619 INFO:teuthology.orchestra.run.smithi186.stdout: "sessions_open": 4, 2024-05-12T22:20:11.619 INFO:teuthology.orchestra.run.smithi186.stdout: "sessions_stale": 0, 2024-05-12T22:20:11.619 INFO:teuthology.orchestra.run.smithi186.stdout: "total_load": 0, 2024-05-12T22:20:11.620 INFO:teuthology.orchestra.run.smithi186.stdout: "average_load": 0, 2024-05-12T22:20:11.620 INFO:teuthology.orchestra.run.smithi186.stdout: "avg_session_uptime": 6614, 2024-05-12T22:20:11.620 INFO:teuthology.orchestra.run.smithi186.stdout: "mdthresh_evicted": 0 2024-05-12T22:20:11.620 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:11.620 INFO:teuthology.orchestra.run.smithi186.stdout: "mempool": { 2024-05-12T22:20:11.620 INFO:teuthology.orchestra.run.smithi186.stdout: "bloom_filter_bytes": 0, 2024-05-12T22:20:11.620 INFO:teuthology.orchestra.run.smithi186.stdout: "bloom_filter_items": 0, 2024-05-12T22:20:11.620 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_alloc_bytes": 0, 2024-05-12T22:20:11.620 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_alloc_items": 0, 2024-05-12T22:20:11.621 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_data_bytes": 0, 2024-05-12T22:20:11.621 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_data_items": 0, 2024-05-12T22:20:11.621 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_onode_bytes": 0, 2024-05-12T22:20:11.621 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_onode_items": 0, 2024-05-12T22:20:11.621 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_meta_bytes": 0, 2024-05-12T22:20:11.621 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_meta_items": 0, 2024-05-12T22:20:11.621 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_other_bytes": 0, 2024-05-12T22:20:11.621 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_other_items": 0, 2024-05-12T22:20:11.621 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_buffer_bytes": 0, 2024-05-12T22:20:11.621 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_buffer_items": 0, 2024-05-12T22:20:11.622 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_extent_bytes": 0, 2024-05-12T22:20:11.622 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_extent_items": 0, 2024-05-12T22:20:11.622 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_blob_bytes": 0, 2024-05-12T22:20:11.622 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_blob_items": 0, 2024-05-12T22:20:11.622 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_shared_blob_bytes": 0, 2024-05-12T22:20:11.622 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_shared_blob_items": 0, 2024-05-12T22:20:11.622 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_inline_bl_bytes": 0, 2024-05-12T22:20:11.622 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_inline_bl_items": 0, 2024-05-12T22:20:11.623 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_fsck_bytes": 0, 2024-05-12T22:20:11.623 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_fsck_items": 0, 2024-05-12T22:20:11.623 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_txc_bytes": 0, 2024-05-12T22:20:11.623 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_txc_items": 0, 2024-05-12T22:20:11.623 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_writing_deferred_bytes": 0, 2024-05-12T22:20:11.623 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_writing_deferred_items": 0, 2024-05-12T22:20:11.623 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_writing_bytes": 0, 2024-05-12T22:20:11.623 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_writing_items": 0, 2024-05-12T22:20:11.623 INFO:teuthology.orchestra.run.smithi186.stdout: "bluefs_bytes": 0, 2024-05-12T22:20:11.624 INFO:teuthology.orchestra.run.smithi186.stdout: "bluefs_items": 0, 2024-05-12T22:20:11.624 INFO:teuthology.orchestra.run.smithi186.stdout: "bluefs_file_reader_bytes": 0, 2024-05-12T22:20:11.624 INFO:teuthology.orchestra.run.smithi186.stdout: "bluefs_file_reader_items": 0, 2024-05-12T22:20:11.624 INFO:teuthology.orchestra.run.smithi186.stdout: "bluefs_file_writer_bytes": 0, 2024-05-12T22:20:11.624 INFO:teuthology.orchestra.run.smithi186.stdout: "bluefs_file_writer_items": 0, 2024-05-12T22:20:11.624 INFO:teuthology.orchestra.run.smithi186.stdout: "buffer_anon_bytes": 189349, 2024-05-12T22:20:11.624 INFO:teuthology.orchestra.run.smithi186.stdout: "buffer_anon_items": 51, 2024-05-12T22:20:11.624 INFO:teuthology.orchestra.run.smithi186.stdout: "buffer_meta_bytes": 0, 2024-05-12T22:20:11.624 INFO:teuthology.orchestra.run.smithi186.stdout: "buffer_meta_items": 0, 2024-05-12T22:20:11.625 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_bytes": 0, 2024-05-12T22:20:11.625 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_items": 0, 2024-05-12T22:20:11.625 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_mapbl_bytes": 0, 2024-05-12T22:20:11.625 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_mapbl_items": 0, 2024-05-12T22:20:11.625 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_pglog_bytes": 0, 2024-05-12T22:20:11.625 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_pglog_items": 0, 2024-05-12T22:20:11.626 INFO:teuthology.orchestra.run.smithi186.stdout: "osdmap_bytes": 8152, 2024-05-12T22:20:11.626 INFO:teuthology.orchestra.run.smithi186.stdout: "osdmap_items": 170, 2024-05-12T22:20:11.626 INFO:teuthology.orchestra.run.smithi186.stdout: "osdmap_mapping_bytes": 0, 2024-05-12T22:20:11.626 INFO:teuthology.orchestra.run.smithi186.stdout: "osdmap_mapping_items": 0, 2024-05-12T22:20:11.626 INFO:teuthology.orchestra.run.smithi186.stdout: "pgmap_bytes": 0, 2024-05-12T22:20:11.626 INFO:teuthology.orchestra.run.smithi186.stdout: "pgmap_items": 0, 2024-05-12T22:20:11.626 INFO:teuthology.orchestra.run.smithi186.stdout: "mds_co_bytes": 8464, 2024-05-12T22:20:11.626 INFO:teuthology.orchestra.run.smithi186.stdout: "mds_co_items": 13, 2024-05-12T22:20:11.626 INFO:teuthology.orchestra.run.smithi186.stdout: "unittest_1_bytes": 0, 2024-05-12T22:20:11.627 INFO:teuthology.orchestra.run.smithi186.stdout: "unittest_1_items": 0, 2024-05-12T22:20:11.627 INFO:teuthology.orchestra.run.smithi186.stdout: "unittest_2_bytes": 0, 2024-05-12T22:20:11.627 INFO:teuthology.orchestra.run.smithi186.stdout: "unittest_2_items": 0 2024-05-12T22:20:11.627 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:11.627 INFO:teuthology.orchestra.run.smithi186.stdout: "objecter": { 2024-05-12T22:20:11.627 INFO:teuthology.orchestra.run.smithi186.stdout: "op_active": 0, 2024-05-12T22:20:11.627 INFO:teuthology.orchestra.run.smithi186.stdout: "op_laggy": 0, 2024-05-12T22:20:11.627 INFO:teuthology.orchestra.run.smithi186.stdout: "op_send": 2378, 2024-05-12T22:20:11.627 INFO:teuthology.orchestra.run.smithi186.stdout: "op_send_bytes": 21, 2024-05-12T22:20:11.628 INFO:teuthology.orchestra.run.smithi186.stdout: "op_resend": 3, 2024-05-12T22:20:11.628 INFO:teuthology.orchestra.run.smithi186.stdout: "op_reply": 2378, 2024-05-12T22:20:11.628 INFO:teuthology.orchestra.run.smithi186.stdout: "op_latency": { 2024-05-12T22:20:11.628 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 2378, 2024-05-12T22:20:11.628 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 2.220959459, 2024-05-12T22:20:11.628 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000933961 2024-05-12T22:20:11.628 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:11.628 INFO:teuthology.orchestra.run.smithi186.stdout: "op_inflight": 0, 2024-05-12T22:20:11.628 INFO:teuthology.orchestra.run.smithi186.stdout: "oplen_avg": { 2024-05-12T22:20:11.629 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 2378, 2024-05-12T22:20:11.629 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 2379 2024-05-12T22:20:11.629 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:11.629 INFO:teuthology.orchestra.run.smithi186.stdout: "op": 2378, 2024-05-12T22:20:11.629 INFO:teuthology.orchestra.run.smithi186.stdout: "op_r": 2378, 2024-05-12T22:20:11.629 INFO:teuthology.orchestra.run.smithi186.stdout: "op_w": 0, 2024-05-12T22:20:11.629 INFO:teuthology.orchestra.run.smithi186.stdout: "op_rmw": 0, 2024-05-12T22:20:11.629 INFO:teuthology.orchestra.run.smithi186.stdout: "op_pg": 0, 2024-05-12T22:20:11.629 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_stat": 1555, 2024-05-12T22:20:11.630 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_create": 0, 2024-05-12T22:20:11.630 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_read": 821, 2024-05-12T22:20:11.630 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_write": 0, 2024-05-12T22:20:11.630 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_writefull": 0, 2024-05-12T22:20:11.630 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_writesame": 0, 2024-05-12T22:20:11.630 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_append": 0, 2024-05-12T22:20:11.630 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_zero": 0, 2024-05-12T22:20:11.630 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_truncate": 0, 2024-05-12T22:20:11.631 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_delete": 0, 2024-05-12T22:20:11.631 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_mapext": 0, 2024-05-12T22:20:11.631 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_sparse_read": 0, 2024-05-12T22:20:11.631 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_clonerange": 0, 2024-05-12T22:20:11.631 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_getxattr": 1, 2024-05-12T22:20:11.631 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_setxattr": 0, 2024-05-12T22:20:11.631 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_cmpxattr": 0, 2024-05-12T22:20:11.631 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_rmxattr": 0, 2024-05-12T22:20:11.631 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_resetxattrs": 0, 2024-05-12T22:20:11.632 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_call": 0, 2024-05-12T22:20:11.632 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_watch": 0, 2024-05-12T22:20:11.632 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_notify": 0, 2024-05-12T22:20:11.632 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_src_cmpxattr": 0, 2024-05-12T22:20:11.632 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_pgls": 0, 2024-05-12T22:20:11.632 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_pgls_filter": 0, 2024-05-12T22:20:11.632 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_other": 0, 2024-05-12T22:20:11.632 INFO:teuthology.orchestra.run.smithi186.stdout: "linger_active": 0, 2024-05-12T22:20:11.633 INFO:teuthology.orchestra.run.smithi186.stdout: "linger_send": 0, 2024-05-12T22:20:11.633 INFO:teuthology.orchestra.run.smithi186.stdout: "linger_resend": 0, 2024-05-12T22:20:11.633 INFO:teuthology.orchestra.run.smithi186.stdout: "linger_ping": 0, 2024-05-12T22:20:11.633 INFO:teuthology.orchestra.run.smithi186.stdout: "poolop_active": 0, 2024-05-12T22:20:11.633 INFO:teuthology.orchestra.run.smithi186.stdout: "poolop_send": 0, 2024-05-12T22:20:11.633 INFO:teuthology.orchestra.run.smithi186.stdout: "poolop_resend": 0, 2024-05-12T22:20:11.633 INFO:teuthology.orchestra.run.smithi186.stdout: "poolstat_active": 0, 2024-05-12T22:20:11.633 INFO:teuthology.orchestra.run.smithi186.stdout: "poolstat_send": 0, 2024-05-12T22:20:11.633 INFO:teuthology.orchestra.run.smithi186.stdout: "poolstat_resend": 0, 2024-05-12T22:20:11.634 INFO:teuthology.orchestra.run.smithi186.stdout: "statfs_active": 0, 2024-05-12T22:20:11.634 INFO:teuthology.orchestra.run.smithi186.stdout: "statfs_send": 0, 2024-05-12T22:20:11.634 INFO:teuthology.orchestra.run.smithi186.stdout: "statfs_resend": 0, 2024-05-12T22:20:11.634 INFO:teuthology.orchestra.run.smithi186.stdout: "command_active": 0, 2024-05-12T22:20:11.634 INFO:teuthology.orchestra.run.smithi186.stdout: "command_send": 0, 2024-05-12T22:20:11.634 INFO:teuthology.orchestra.run.smithi186.stdout: "command_resend": 0, 2024-05-12T22:20:11.634 INFO:teuthology.orchestra.run.smithi186.stdout: "map_epoch": 98, 2024-05-12T22:20:11.634 INFO:teuthology.orchestra.run.smithi186.stdout: "map_full": 0, 2024-05-12T22:20:11.634 INFO:teuthology.orchestra.run.smithi186.stdout: "map_inc": 12, 2024-05-12T22:20:11.635 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_sessions": 5, 2024-05-12T22:20:11.635 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_session_open": 5, 2024-05-12T22:20:11.635 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_session_close": 0, 2024-05-12T22:20:11.635 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_laggy": 0, 2024-05-12T22:20:11.635 INFO:teuthology.orchestra.run.smithi186.stdout: "omap_wr": 0, 2024-05-12T22:20:11.635 INFO:teuthology.orchestra.run.smithi186.stdout: "omap_rd": 2, 2024-05-12T22:20:11.635 INFO:teuthology.orchestra.run.smithi186.stdout: "omap_del": 0 2024-05-12T22:20:11.635 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:11.636 INFO:teuthology.orchestra.run.smithi186.stdout: "oft": { 2024-05-12T22:20:11.636 INFO:teuthology.orchestra.run.smithi186.stdout: "omap_total_objs": 0, 2024-05-12T22:20:11.636 INFO:teuthology.orchestra.run.smithi186.stdout: "omap_total_kv_pairs": 0, 2024-05-12T22:20:11.636 INFO:teuthology.orchestra.run.smithi186.stdout: "omap_total_updates": 0, 2024-05-12T22:20:11.636 INFO:teuthology.orchestra.run.smithi186.stdout: "omap_total_removes": 0 2024-05-12T22:20:11.636 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:11.636 INFO:teuthology.orchestra.run.smithi186.stdout: "osd-slow-ops": { 2024-05-12T22:20:11.636 INFO:teuthology.orchestra.run.smithi186.stdout: "slow_ops_count": 0 2024-05-12T22:20:11.636 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:11.637 INFO:teuthology.orchestra.run.smithi186.stdout: "purge_queue": { 2024-05-12T22:20:11.637 INFO:teuthology.orchestra.run.smithi186.stdout: "pq_executing_ops": 0, 2024-05-12T22:20:11.637 INFO:teuthology.orchestra.run.smithi186.stdout: "pq_executing_ops_high_water": 0, 2024-05-12T22:20:11.637 INFO:teuthology.orchestra.run.smithi186.stdout: "pq_executing": 0, 2024-05-12T22:20:11.637 INFO:teuthology.orchestra.run.smithi186.stdout: "pq_executing_high_water": 0, 2024-05-12T22:20:11.637 INFO:teuthology.orchestra.run.smithi186.stdout: "pq_executed_ops": 0, 2024-05-12T22:20:11.637 INFO:teuthology.orchestra.run.smithi186.stdout: "pq_executed": 0, 2024-05-12T22:20:11.637 INFO:teuthology.orchestra.run.smithi186.stdout: "pq_item_in_journal": 0 2024-05-12T22:20:11.638 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:11.638 INFO:teuthology.orchestra.run.smithi186.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-05-12T22:20:11.638 INFO:teuthology.orchestra.run.smithi186.stdout: "val": 0, 2024-05-12T22:20:11.638 INFO:teuthology.orchestra.run.smithi186.stdout: "max": 104857600, 2024-05-12T22:20:11.638 INFO:teuthology.orchestra.run.smithi186.stdout: "get_started": 0, 2024-05-12T22:20:11.638 INFO:teuthology.orchestra.run.smithi186.stdout: "get": 2638, 2024-05-12T22:20:11.638 INFO:teuthology.orchestra.run.smithi186.stdout: "get_sum": 2185475, 2024-05-12T22:20:11.638 INFO:teuthology.orchestra.run.smithi186.stdout: "get_or_fail_fail": 0, 2024-05-12T22:20:11.638 INFO:teuthology.orchestra.run.smithi186.stdout: "get_or_fail_success": 2638, 2024-05-12T22:20:11.639 INFO:teuthology.orchestra.run.smithi186.stdout: "take": 0, 2024-05-12T22:20:11.639 INFO:teuthology.orchestra.run.smithi186.stdout: "take_sum": 0, 2024-05-12T22:20:11.639 INFO:teuthology.orchestra.run.smithi186.stdout: "put": 2638, 2024-05-12T22:20:11.639 INFO:teuthology.orchestra.run.smithi186.stdout: "put_sum": 2185475, 2024-05-12T22:20:11.639 INFO:teuthology.orchestra.run.smithi186.stdout: "wait": { 2024-05-12T22:20:11.639 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-05-12T22:20:11.639 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-05-12T22:20:11.639 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-05-12T22:20:11.639 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-05-12T22:20:11.640 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:11.640 INFO:teuthology.orchestra.run.smithi186.stdout: "throttle-objecter_bytes": { 2024-05-12T22:20:11.640 INFO:teuthology.orchestra.run.smithi186.stdout: "val": 0, 2024-05-12T22:20:11.640 INFO:teuthology.orchestra.run.smithi186.stdout: "max": 104857600, 2024-05-12T22:20:11.640 INFO:teuthology.orchestra.run.smithi186.stdout: "get_started": 0, 2024-05-12T22:20:11.640 INFO:teuthology.orchestra.run.smithi186.stdout: "get": 0, 2024-05-12T22:20:11.640 INFO:teuthology.orchestra.run.smithi186.stdout: "get_sum": 0, 2024-05-12T22:20:11.640 INFO:teuthology.orchestra.run.smithi186.stdout: "get_or_fail_fail": 0, 2024-05-12T22:20:11.640 INFO:teuthology.orchestra.run.smithi186.stdout: "get_or_fail_success": 0, 2024-05-12T22:20:11.641 INFO:teuthology.orchestra.run.smithi186.stdout: "take": 2378, 2024-05-12T22:20:11.641 INFO:teuthology.orchestra.run.smithi186.stdout: "take_sum": 3031, 2024-05-12T22:20:11.641 INFO:teuthology.orchestra.run.smithi186.stdout: "put": 2378, 2024-05-12T22:20:11.641 INFO:teuthology.orchestra.run.smithi186.stdout: "put_sum": 3031, 2024-05-12T22:20:11.641 INFO:teuthology.orchestra.run.smithi186.stdout: "wait": { 2024-05-12T22:20:11.641 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-05-12T22:20:11.641 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-05-12T22:20:11.641 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-05-12T22:20:11.642 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-05-12T22:20:11.642 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:11.642 INFO:teuthology.orchestra.run.smithi186.stdout: "throttle-objecter_ops": { 2024-05-12T22:20:11.642 INFO:teuthology.orchestra.run.smithi186.stdout: "val": 0, 2024-05-12T22:20:11.642 INFO:teuthology.orchestra.run.smithi186.stdout: "max": 1024, 2024-05-12T22:20:11.642 INFO:teuthology.orchestra.run.smithi186.stdout: "get_started": 0, 2024-05-12T22:20:11.642 INFO:teuthology.orchestra.run.smithi186.stdout: "get": 0, 2024-05-12T22:20:11.642 INFO:teuthology.orchestra.run.smithi186.stdout: "get_sum": 0, 2024-05-12T22:20:11.642 INFO:teuthology.orchestra.run.smithi186.stdout: "get_or_fail_fail": 0, 2024-05-12T22:20:11.643 INFO:teuthology.orchestra.run.smithi186.stdout: "get_or_fail_success": 0, 2024-05-12T22:20:11.643 INFO:teuthology.orchestra.run.smithi186.stdout: "take": 2378, 2024-05-12T22:20:11.643 INFO:teuthology.orchestra.run.smithi186.stdout: "take_sum": 2378, 2024-05-12T22:20:11.643 INFO:teuthology.orchestra.run.smithi186.stdout: "put": 2378, 2024-05-12T22:20:11.643 INFO:teuthology.orchestra.run.smithi186.stdout: "put_sum": 2378, 2024-05-12T22:20:11.643 INFO:teuthology.orchestra.run.smithi186.stdout: "wait": { 2024-05-12T22:20:11.643 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-05-12T22:20:11.643 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-05-12T22:20:11.643 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-05-12T22:20:11.644 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-05-12T22:20:11.644 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:11.644 INFO:teuthology.orchestra.run.smithi186.stdout: "throttle-write_buf_throttle": { 2024-05-12T22:20:11.644 INFO:teuthology.orchestra.run.smithi186.stdout: "val": 0, 2024-05-12T22:20:11.644 INFO:teuthology.orchestra.run.smithi186.stdout: "max": 3758096384, 2024-05-12T22:20:11.644 INFO:teuthology.orchestra.run.smithi186.stdout: "get_started": 0, 2024-05-12T22:20:11.644 INFO:teuthology.orchestra.run.smithi186.stdout: "get": 0, 2024-05-12T22:20:11.644 INFO:teuthology.orchestra.run.smithi186.stdout: "get_sum": 0, 2024-05-12T22:20:11.644 INFO:teuthology.orchestra.run.smithi186.stdout: "get_or_fail_fail": 0, 2024-05-12T22:20:11.645 INFO:teuthology.orchestra.run.smithi186.stdout: "get_or_fail_success": 0, 2024-05-12T22:20:11.645 INFO:teuthology.orchestra.run.smithi186.stdout: "take": 0, 2024-05-12T22:20:11.645 INFO:teuthology.orchestra.run.smithi186.stdout: "take_sum": 0, 2024-05-12T22:20:11.645 INFO:teuthology.orchestra.run.smithi186.stdout: "put": 0, 2024-05-12T22:20:11.645 INFO:teuthology.orchestra.run.smithi186.stdout: "put_sum": 0, 2024-05-12T22:20:11.645 INFO:teuthology.orchestra.run.smithi186.stdout: "wait": { 2024-05-12T22:20:11.645 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-05-12T22:20:11.645 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-05-12T22:20:11.646 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-05-12T22:20:11.646 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-05-12T22:20:11.646 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:11.646 INFO:teuthology.orchestra.run.smithi186.stdout: "throttle-write_buf_throttle-0x564b6550e6e0": { 2024-05-12T22:20:11.646 INFO:teuthology.orchestra.run.smithi186.stdout: "val": 0, 2024-05-12T22:20:11.646 INFO:teuthology.orchestra.run.smithi186.stdout: "max": 3758096384, 2024-05-12T22:20:11.646 INFO:teuthology.orchestra.run.smithi186.stdout: "get_started": 0, 2024-05-12T22:20:11.646 INFO:teuthology.orchestra.run.smithi186.stdout: "get": 0, 2024-05-12T22:20:11.646 INFO:teuthology.orchestra.run.smithi186.stdout: "get_sum": 0, 2024-05-12T22:20:11.647 INFO:teuthology.orchestra.run.smithi186.stdout: "get_or_fail_fail": 0, 2024-05-12T22:20:11.647 INFO:teuthology.orchestra.run.smithi186.stdout: "get_or_fail_success": 0, 2024-05-12T22:20:11.647 INFO:teuthology.orchestra.run.smithi186.stdout: "take": 0, 2024-05-12T22:20:11.647 INFO:teuthology.orchestra.run.smithi186.stdout: "take_sum": 0, 2024-05-12T22:20:11.647 INFO:teuthology.orchestra.run.smithi186.stdout: "put": 0, 2024-05-12T22:20:11.647 INFO:teuthology.orchestra.run.smithi186.stdout: "put_sum": 0, 2024-05-12T22:20:11.647 INFO:teuthology.orchestra.run.smithi186.stdout: "wait": { 2024-05-12T22:20:11.647 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-05-12T22:20:11.647 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-05-12T22:20:11.648 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-05-12T22:20:11.648 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-05-12T22:20:11.648 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-05-12T22:20:11.648 INFO:teuthology.orchestra.run.smithi186.stdout:} 2024-05-12T22:20:12.355 INFO:tasks.check_counter:Daemon mds.c mds.exported=0 2024-05-12T22:20:12.355 INFO:tasks.check_counter:Daemon mds.c mds.imported=0 2024-05-12T22:20:12.355 INFO:tasks.check_counter:Daemon mds.c mds_cache.dir_update=0 2024-05-12T22:20:12.355 INFO:tasks.check_counter:Daemon mds.c mds_cache.dir_update_receipt=0 2024-05-12T22:20:12.356 INFO:tasks.check_counter:Daemon mds.c mds.root_rsnaps=0 2024-05-12T22:20:12.356 INFO:tasks.check_counter:Daemon mds.c mds_server.req_mksnap_latency.avgcount=0 2024-05-12T22:20:12.356 INFO:tasks.check_counter:Daemon mds.c mds_server.req_rmsnap_latency.avgcount=0 2024-05-12T22:20:12.356 INFO:tasks.check_counter:Daemon mds.c mds.dir_split=0 2024-05-12T22:20:12.356 DEBUG:tasks.check_counter:Getting stats from f 2024-05-12T22:20:12.356 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f72fecff68e1d400c4568684327c900485c20d6a shell --fsid 6b92e7b4-10aa-11ef-bc98-c7b262605968 -- ceph daemon mds.f perf dump 2024-05-12T22:20:12.631 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/mon.c/config 2024-05-12T22:20:12.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:20:12 smithi186 ceph-mon[34785]: pgmap v348: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-05-12T22:20:12.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:20:12 smithi086 ceph-mon[34297]: pgmap v348: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-05-12T22:20:12.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:20:12 smithi028 ceph-mon[29847]: pgmap v348: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-05-12T22:20:14.350 INFO:teuthology.orchestra.run.smithi186.stdout:{ 2024-05-12T22:20:14.350 INFO:teuthology.orchestra.run.smithi186.stdout: "AsyncMessenger::Worker-0": { 2024-05-12T22:20:14.350 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_messages": 238, 2024-05-12T22:20:14.350 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_messages": 226, 2024-05-12T22:20:14.350 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_bytes": 1723040, 2024-05-12T22:20:14.351 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_bytes": 139424, 2024-05-12T22:20:14.351 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_created_connections": 4, 2024-05-12T22:20:14.351 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_active_connections": 0, 2024-05-12T22:20:14.351 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_total_time": 0.052712922, 2024-05-12T22:20:14.351 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_send_time": 0.015236894, 2024-05-12T22:20:14.351 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_recv_time": 0.042019950, 2024-05-12T22:20:14.351 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_fast_dispatch_time": 0.005765372, 2024-05-12T22:20:14.352 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_messages_queue_lat": { 2024-05-12T22:20:14.352 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 225, 2024-05-12T22:20:14.352 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.012825304, 2024-05-12T22:20:14.352 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000057001 2024-05-12T22:20:14.352 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:14.352 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_handle_ack_lat": { 2024-05-12T22:20:14.352 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-05-12T22:20:14.352 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-05-12T22:20:14.353 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-05-12T22:20:14.353 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:14.353 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_encrypted_bytes": 1723040, 2024-05-12T22:20:14.353 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_encrypted_bytes": 139424 2024-05-12T22:20:14.353 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:14.353 INFO:teuthology.orchestra.run.smithi186.stdout: "AsyncMessenger::Worker-1": { 2024-05-12T22:20:14.353 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_messages": 2, 2024-05-12T22:20:14.353 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_messages": 173, 2024-05-12T22:20:14.353 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_bytes": 320, 2024-05-12T22:20:14.354 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_bytes": 186048, 2024-05-12T22:20:14.354 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_created_connections": 5, 2024-05-12T22:20:14.354 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_active_connections": 3, 2024-05-12T22:20:14.354 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_total_time": 0.011425398, 2024-05-12T22:20:14.354 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_send_time": 0.007675087, 2024-05-12T22:20:14.354 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_recv_time": 0.001365900, 2024-05-12T22:20:14.354 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-05-12T22:20:14.354 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_messages_queue_lat": { 2024-05-12T22:20:14.354 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 170, 2024-05-12T22:20:14.355 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.031519085, 2024-05-12T22:20:14.355 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000185406 2024-05-12T22:20:14.355 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:14.355 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_handle_ack_lat": { 2024-05-12T22:20:14.355 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-05-12T22:20:14.355 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-05-12T22:20:14.355 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-05-12T22:20:14.355 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:14.355 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_encrypted_bytes": 320, 2024-05-12T22:20:14.356 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_encrypted_bytes": 186048 2024-05-12T22:20:14.356 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:14.356 INFO:teuthology.orchestra.run.smithi186.stdout: "AsyncMessenger::Worker-2": { 2024-05-12T22:20:14.356 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_messages": 4, 2024-05-12T22:20:14.356 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_messages": 4, 2024-05-12T22:20:14.356 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_bytes": 3072, 2024-05-12T22:20:14.356 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_bytes": 448, 2024-05-12T22:20:14.356 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_created_connections": 2, 2024-05-12T22:20:14.357 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_active_connections": 1, 2024-05-12T22:20:14.357 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_total_time": 0.000531359, 2024-05-12T22:20:14.357 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_send_time": 0.000035524, 2024-05-12T22:20:14.357 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_recv_time": 0.000482318, 2024-05-12T22:20:14.357 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-05-12T22:20:14.357 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_messages_queue_lat": { 2024-05-12T22:20:14.357 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 3, 2024-05-12T22:20:14.357 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000723669, 2024-05-12T22:20:14.358 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000241223 2024-05-12T22:20:14.358 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:14.358 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_handle_ack_lat": { 2024-05-12T22:20:14.358 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-05-12T22:20:14.358 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-05-12T22:20:14.358 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-05-12T22:20:14.358 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:14.358 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_encrypted_bytes": 3072, 2024-05-12T22:20:14.359 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_encrypted_bytes": 448 2024-05-12T22:20:14.359 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:14.359 INFO:teuthology.orchestra.run.smithi186.stdout: "cct": { 2024-05-12T22:20:14.359 INFO:teuthology.orchestra.run.smithi186.stdout: "total_workers": 0, 2024-05-12T22:20:14.359 INFO:teuthology.orchestra.run.smithi186.stdout: "unhealthy_workers": 0 2024-05-12T22:20:14.359 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:14.359 INFO:teuthology.orchestra.run.smithi186.stdout: "mempool": { 2024-05-12T22:20:14.359 INFO:teuthology.orchestra.run.smithi186.stdout: "bloom_filter_bytes": 0, 2024-05-12T22:20:14.359 INFO:teuthology.orchestra.run.smithi186.stdout: "bloom_filter_items": 0, 2024-05-12T22:20:14.360 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_alloc_bytes": 0, 2024-05-12T22:20:14.360 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_alloc_items": 0, 2024-05-12T22:20:14.360 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_data_bytes": 0, 2024-05-12T22:20:14.360 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_data_items": 0, 2024-05-12T22:20:14.360 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_onode_bytes": 0, 2024-05-12T22:20:14.360 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_onode_items": 0, 2024-05-12T22:20:14.360 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_meta_bytes": 0, 2024-05-12T22:20:14.360 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_meta_items": 0, 2024-05-12T22:20:14.361 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_other_bytes": 0, 2024-05-12T22:20:14.361 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_other_items": 0, 2024-05-12T22:20:14.361 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_buffer_bytes": 0, 2024-05-12T22:20:14.361 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_buffer_items": 0, 2024-05-12T22:20:14.361 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_extent_bytes": 0, 2024-05-12T22:20:14.361 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_extent_items": 0, 2024-05-12T22:20:14.361 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_blob_bytes": 0, 2024-05-12T22:20:14.361 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_blob_items": 0, 2024-05-12T22:20:14.361 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_shared_blob_bytes": 0, 2024-05-12T22:20:14.362 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_shared_blob_items": 0, 2024-05-12T22:20:14.362 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_inline_bl_bytes": 0, 2024-05-12T22:20:14.362 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_inline_bl_items": 0, 2024-05-12T22:20:14.362 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_fsck_bytes": 0, 2024-05-12T22:20:14.362 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_fsck_items": 0, 2024-05-12T22:20:14.362 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_txc_bytes": 0, 2024-05-12T22:20:14.362 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_txc_items": 0, 2024-05-12T22:20:14.362 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_writing_deferred_bytes": 0, 2024-05-12T22:20:14.363 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_writing_deferred_items": 0, 2024-05-12T22:20:14.363 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_writing_bytes": 0, 2024-05-12T22:20:14.363 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_writing_items": 0, 2024-05-12T22:20:14.363 INFO:teuthology.orchestra.run.smithi186.stdout: "bluefs_bytes": 0, 2024-05-12T22:20:14.363 INFO:teuthology.orchestra.run.smithi186.stdout: "bluefs_items": 0, 2024-05-12T22:20:14.363 INFO:teuthology.orchestra.run.smithi186.stdout: "bluefs_file_reader_bytes": 0, 2024-05-12T22:20:14.363 INFO:teuthology.orchestra.run.smithi186.stdout: "bluefs_file_reader_items": 0, 2024-05-12T22:20:14.363 INFO:teuthology.orchestra.run.smithi186.stdout: "bluefs_file_writer_bytes": 0, 2024-05-12T22:20:14.364 INFO:teuthology.orchestra.run.smithi186.stdout: "bluefs_file_writer_items": 0, 2024-05-12T22:20:14.364 INFO:teuthology.orchestra.run.smithi186.stdout: "buffer_anon_bytes": 148955, 2024-05-12T22:20:14.364 INFO:teuthology.orchestra.run.smithi186.stdout: "buffer_anon_items": 30, 2024-05-12T22:20:14.364 INFO:teuthology.orchestra.run.smithi186.stdout: "buffer_meta_bytes": 0, 2024-05-12T22:20:14.364 INFO:teuthology.orchestra.run.smithi186.stdout: "buffer_meta_items": 0, 2024-05-12T22:20:14.364 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_bytes": 0, 2024-05-12T22:20:14.365 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_items": 0, 2024-05-12T22:20:14.365 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_mapbl_bytes": 0, 2024-05-12T22:20:14.365 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_mapbl_items": 0, 2024-05-12T22:20:14.365 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_pglog_bytes": 0, 2024-05-12T22:20:14.365 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_pglog_items": 0, 2024-05-12T22:20:14.365 INFO:teuthology.orchestra.run.smithi186.stdout: "osdmap_bytes": 0, 2024-05-12T22:20:14.365 INFO:teuthology.orchestra.run.smithi186.stdout: "osdmap_items": 0, 2024-05-12T22:20:14.365 INFO:teuthology.orchestra.run.smithi186.stdout: "osdmap_mapping_bytes": 0, 2024-05-12T22:20:14.366 INFO:teuthology.orchestra.run.smithi186.stdout: "osdmap_mapping_items": 0, 2024-05-12T22:20:14.366 INFO:teuthology.orchestra.run.smithi186.stdout: "pgmap_bytes": 0, 2024-05-12T22:20:14.366 INFO:teuthology.orchestra.run.smithi186.stdout: "pgmap_items": 0, 2024-05-12T22:20:14.366 INFO:teuthology.orchestra.run.smithi186.stdout: "mds_co_bytes": 928, 2024-05-12T22:20:14.366 INFO:teuthology.orchestra.run.smithi186.stdout: "mds_co_items": 2, 2024-05-12T22:20:14.366 INFO:teuthology.orchestra.run.smithi186.stdout: "unittest_1_bytes": 0, 2024-05-12T22:20:14.366 INFO:teuthology.orchestra.run.smithi186.stdout: "unittest_1_items": 0, 2024-05-12T22:20:14.366 INFO:teuthology.orchestra.run.smithi186.stdout: "unittest_2_bytes": 0, 2024-05-12T22:20:14.366 INFO:teuthology.orchestra.run.smithi186.stdout: "unittest_2_items": 0 2024-05-12T22:20:14.367 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:14.367 INFO:teuthology.orchestra.run.smithi186.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-05-12T22:20:14.367 INFO:teuthology.orchestra.run.smithi186.stdout: "val": 0, 2024-05-12T22:20:14.367 INFO:teuthology.orchestra.run.smithi186.stdout: "max": 104857600, 2024-05-12T22:20:14.367 INFO:teuthology.orchestra.run.smithi186.stdout: "get_started": 0, 2024-05-12T22:20:14.367 INFO:teuthology.orchestra.run.smithi186.stdout: "get": 240, 2024-05-12T22:20:14.367 INFO:teuthology.orchestra.run.smithi186.stdout: "get_sum": 1689499, 2024-05-12T22:20:14.367 INFO:teuthology.orchestra.run.smithi186.stdout: "get_or_fail_fail": 0, 2024-05-12T22:20:14.368 INFO:teuthology.orchestra.run.smithi186.stdout: "get_or_fail_success": 240, 2024-05-12T22:20:14.368 INFO:teuthology.orchestra.run.smithi186.stdout: "take": 0, 2024-05-12T22:20:14.368 INFO:teuthology.orchestra.run.smithi186.stdout: "take_sum": 0, 2024-05-12T22:20:14.368 INFO:teuthology.orchestra.run.smithi186.stdout: "put": 240, 2024-05-12T22:20:14.368 INFO:teuthology.orchestra.run.smithi186.stdout: "put_sum": 1689499, 2024-05-12T22:20:14.368 INFO:teuthology.orchestra.run.smithi186.stdout: "wait": { 2024-05-12T22:20:14.368 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-05-12T22:20:14.368 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-05-12T22:20:14.369 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-05-12T22:20:14.369 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-05-12T22:20:14.369 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-05-12T22:20:14.369 INFO:teuthology.orchestra.run.smithi186.stdout:} 2024-05-12T22:20:14.643 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:20:14 smithi186 ceph-mon[34785]: pgmap v349: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-05-12T22:20:14.643 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:20:14 smithi186 ceph-mon[34785]: osdmap e99: 12 total, 12 up, 12 in 2024-05-12T22:20:14.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:20:14 smithi086 ceph-mon[34297]: pgmap v349: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-05-12T22:20:14.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:20:14 smithi086 ceph-mon[34297]: osdmap e99: 12 total, 12 up, 12 in 2024-05-12T22:20:14.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:20:14 smithi028 ceph-mon[29847]: pgmap v349: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-05-12T22:20:14.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:20:14 smithi028 ceph-mon[29847]: osdmap e99: 12 total, 12 up, 12 in 2024-05-12T22:20:15.056 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.f 2024-05-12T22:20:15.056 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.f 2024-05-12T22:20:15.057 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.f 2024-05-12T22:20:15.057 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.f 2024-05-12T22:20:15.057 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.f 2024-05-12T22:20:15.057 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.f 2024-05-12T22:20:15.057 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.f 2024-05-12T22:20:15.057 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.f 2024-05-12T22:20:15.057 DEBUG:tasks.check_counter:Getting stats from i 2024-05-12T22:20:15.057 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f72fecff68e1d400c4568684327c900485c20d6a shell --fsid 6b92e7b4-10aa-11ef-bc98-c7b262605968 -- ceph daemon mds.i perf dump 2024-05-12T22:20:15.294 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/mon.c/config 2024-05-12T22:20:16.621 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:20:16 smithi186 ceph-mon[34785]: pgmap v351: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 255 B/s wr, 7 op/s 2024-05-12T22:20:16.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:20:16 smithi086 ceph-mon[34297]: pgmap v351: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 255 B/s wr, 7 op/s 2024-05-12T22:20:16.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:20:16 smithi028 ceph-mon[29847]: pgmap v351: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 255 B/s wr, 7 op/s 2024-05-12T22:20:16.865 INFO:teuthology.orchestra.run.smithi186.stdout:{ 2024-05-12T22:20:16.865 INFO:teuthology.orchestra.run.smithi186.stdout: "AsyncMessenger::Worker-0": { 2024-05-12T22:20:16.865 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_messages": 2406, 2024-05-12T22:20:16.865 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_messages": 2408, 2024-05-12T22:20:16.866 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_bytes": 752890, 2024-05-12T22:20:16.866 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_bytes": 765115, 2024-05-12T22:20:16.866 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_created_connections": 6, 2024-05-12T22:20:16.866 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_active_connections": 2, 2024-05-12T22:20:16.866 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_total_time": 0.218093216, 2024-05-12T22:20:16.866 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_send_time": 0.072792841, 2024-05-12T22:20:16.866 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_recv_time": 0.057165945, 2024-05-12T22:20:16.866 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_fast_dispatch_time": 0.071406194, 2024-05-12T22:20:16.867 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_messages_queue_lat": { 2024-05-12T22:20:16.867 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 2406, 2024-05-12T22:20:16.867 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.093169304, 2024-05-12T22:20:16.867 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000038723 2024-05-12T22:20:16.867 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:16.867 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_handle_ack_lat": { 2024-05-12T22:20:16.867 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-05-12T22:20:16.867 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-05-12T22:20:16.868 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-05-12T22:20:16.868 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:16.868 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_encrypted_bytes": 0, 2024-05-12T22:20:16.868 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_encrypted_bytes": 0 2024-05-12T22:20:16.868 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:16.868 INFO:teuthology.orchestra.run.smithi186.stdout: "AsyncMessenger::Worker-1": { 2024-05-12T22:20:16.868 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_messages": 9, 2024-05-12T22:20:16.868 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_messages": 178, 2024-05-12T22:20:16.868 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_bytes": 4108, 2024-05-12T22:20:16.869 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_bytes": 545610, 2024-05-12T22:20:16.869 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_created_connections": 7, 2024-05-12T22:20:16.869 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_active_connections": 4, 2024-05-12T22:20:16.869 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_total_time": 0.011469120, 2024-05-12T22:20:16.869 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_send_time": 0.006978404, 2024-05-12T22:20:16.869 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_recv_time": 0.002401944, 2024-05-12T22:20:16.869 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_fast_dispatch_time": 0.000067297, 2024-05-12T22:20:16.869 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_messages_queue_lat": { 2024-05-12T22:20:16.869 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 175, 2024-05-12T22:20:16.870 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.025466111, 2024-05-12T22:20:16.870 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000145520 2024-05-12T22:20:16.870 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:16.870 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_handle_ack_lat": { 2024-05-12T22:20:16.870 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-05-12T22:20:16.870 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-05-12T22:20:16.870 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-05-12T22:20:16.870 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:16.870 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_encrypted_bytes": 3552, 2024-05-12T22:20:16.870 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_encrypted_bytes": 544912 2024-05-12T22:20:16.871 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:16.871 INFO:teuthology.orchestra.run.smithi186.stdout: "AsyncMessenger::Worker-2": { 2024-05-12T22:20:16.871 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_messages": 260, 2024-05-12T22:20:16.871 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_messages": 244, 2024-05-12T22:20:16.871 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_bytes": 1761836, 2024-05-12T22:20:16.871 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_bytes": 142222, 2024-05-12T22:20:16.871 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_created_connections": 3, 2024-05-12T22:20:16.871 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_active_connections": 2, 2024-05-12T22:20:16.871 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_total_time": 0.051172870, 2024-05-12T22:20:16.872 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_send_time": 0.015321593, 2024-05-12T22:20:16.872 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_recv_time": 0.041558259, 2024-05-12T22:20:16.872 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_fast_dispatch_time": 0.005497264, 2024-05-12T22:20:16.872 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_messages_queue_lat": { 2024-05-12T22:20:16.872 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 243, 2024-05-12T22:20:16.872 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.013478156, 2024-05-12T22:20:16.872 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000055465 2024-05-12T22:20:16.872 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:16.872 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_handle_ack_lat": { 2024-05-12T22:20:16.873 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-05-12T22:20:16.873 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-05-12T22:20:16.873 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-05-12T22:20:16.873 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:16.873 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_encrypted_bytes": 1761568, 2024-05-12T22:20:16.873 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_encrypted_bytes": 141904 2024-05-12T22:20:16.873 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:16.873 INFO:teuthology.orchestra.run.smithi186.stdout: "cct": { 2024-05-12T22:20:16.873 INFO:teuthology.orchestra.run.smithi186.stdout: "total_workers": 1, 2024-05-12T22:20:16.874 INFO:teuthology.orchestra.run.smithi186.stdout: "unhealthy_workers": 0 2024-05-12T22:20:16.874 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:16.874 INFO:teuthology.orchestra.run.smithi186.stdout: "finisher-MDSRank": { 2024-05-12T22:20:16.874 INFO:teuthology.orchestra.run.smithi186.stdout: "queue_len": 0, 2024-05-12T22:20:16.874 INFO:teuthology.orchestra.run.smithi186.stdout: "complete_latency": { 2024-05-12T22:20:16.874 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 4061, 2024-05-12T22:20:16.874 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.113431428, 2024-05-12T22:20:16.874 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000027931 2024-05-12T22:20:16.874 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-05-12T22:20:16.875 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:16.875 INFO:teuthology.orchestra.run.smithi186.stdout: "finisher-PurgeQueue": { 2024-05-12T22:20:16.875 INFO:teuthology.orchestra.run.smithi186.stdout: "queue_len": 0, 2024-05-12T22:20:16.875 INFO:teuthology.orchestra.run.smithi186.stdout: "complete_latency": { 2024-05-12T22:20:16.875 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-05-12T22:20:16.875 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-05-12T22:20:16.875 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-05-12T22:20:16.875 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-05-12T22:20:16.875 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:16.876 INFO:teuthology.orchestra.run.smithi186.stdout: "mds": { 2024-05-12T22:20:16.876 INFO:teuthology.orchestra.run.smithi186.stdout: "request": 0, 2024-05-12T22:20:16.876 INFO:teuthology.orchestra.run.smithi186.stdout: "reply": 0, 2024-05-12T22:20:16.876 INFO:teuthology.orchestra.run.smithi186.stdout: "reply_latency": { 2024-05-12T22:20:16.876 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-05-12T22:20:16.876 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-05-12T22:20:16.876 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-05-12T22:20:16.876 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:16.876 INFO:teuthology.orchestra.run.smithi186.stdout: "slow_reply": 0, 2024-05-12T22:20:16.877 INFO:teuthology.orchestra.run.smithi186.stdout: "forward": 0, 2024-05-12T22:20:16.877 INFO:teuthology.orchestra.run.smithi186.stdout: "dir_fetch_complete": 0, 2024-05-12T22:20:16.877 INFO:teuthology.orchestra.run.smithi186.stdout: "dir_fetch_keys": 0, 2024-05-12T22:20:16.877 INFO:teuthology.orchestra.run.smithi186.stdout: "dir_commit": 0, 2024-05-12T22:20:16.877 INFO:teuthology.orchestra.run.smithi186.stdout: "dir_split": 0, 2024-05-12T22:20:16.877 INFO:teuthology.orchestra.run.smithi186.stdout: "dir_merge": 0, 2024-05-12T22:20:16.877 INFO:teuthology.orchestra.run.smithi186.stdout: "inodes": 31, 2024-05-12T22:20:16.877 INFO:teuthology.orchestra.run.smithi186.stdout: "inodes_top": 2, 2024-05-12T22:20:16.877 INFO:teuthology.orchestra.run.smithi186.stdout: "inodes_bottom": 29, 2024-05-12T22:20:16.877 INFO:teuthology.orchestra.run.smithi186.stdout: "inodes_pin_tail": 0, 2024-05-12T22:20:16.878 INFO:teuthology.orchestra.run.smithi186.stdout: "inodes_pinned": 28, 2024-05-12T22:20:16.878 INFO:teuthology.orchestra.run.smithi186.stdout: "inodes_expired": 0, 2024-05-12T22:20:16.878 INFO:teuthology.orchestra.run.smithi186.stdout: "inodes_with_caps": 0, 2024-05-12T22:20:16.878 INFO:teuthology.orchestra.run.smithi186.stdout: "caps": 0, 2024-05-12T22:20:16.878 INFO:teuthology.orchestra.run.smithi186.stdout: "subtrees": 5, 2024-05-12T22:20:16.878 INFO:teuthology.orchestra.run.smithi186.stdout: "traverse": 0, 2024-05-12T22:20:16.878 INFO:teuthology.orchestra.run.smithi186.stdout: "traverse_hit": 0, 2024-05-12T22:20:16.878 INFO:teuthology.orchestra.run.smithi186.stdout: "traverse_forward": 0, 2024-05-12T22:20:16.879 INFO:teuthology.orchestra.run.smithi186.stdout: "traverse_discover": 0, 2024-05-12T22:20:16.879 INFO:teuthology.orchestra.run.smithi186.stdout: "traverse_dir_fetch": 0, 2024-05-12T22:20:16.879 INFO:teuthology.orchestra.run.smithi186.stdout: "traverse_remote_ino": 0, 2024-05-12T22:20:16.879 INFO:teuthology.orchestra.run.smithi186.stdout: "traverse_lock": 0, 2024-05-12T22:20:16.879 INFO:teuthology.orchestra.run.smithi186.stdout: "load_cent": 0, 2024-05-12T22:20:16.879 INFO:teuthology.orchestra.run.smithi186.stdout: "q": 0, 2024-05-12T22:20:16.879 INFO:teuthology.orchestra.run.smithi186.stdout: "exported": 0, 2024-05-12T22:20:16.879 INFO:teuthology.orchestra.run.smithi186.stdout: "exported_inodes": 0, 2024-05-12T22:20:16.879 INFO:teuthology.orchestra.run.smithi186.stdout: "imported": 0, 2024-05-12T22:20:16.879 INFO:teuthology.orchestra.run.smithi186.stdout: "imported_inodes": 0, 2024-05-12T22:20:16.880 INFO:teuthology.orchestra.run.smithi186.stdout: "openino_dir_fetch": 0, 2024-05-12T22:20:16.880 INFO:teuthology.orchestra.run.smithi186.stdout: "openino_backtrace_fetch": 0, 2024-05-12T22:20:16.880 INFO:teuthology.orchestra.run.smithi186.stdout: "openino_peer_discover": 0, 2024-05-12T22:20:16.880 INFO:teuthology.orchestra.run.smithi186.stdout: "root_rfiles": 0, 2024-05-12T22:20:16.880 INFO:teuthology.orchestra.run.smithi186.stdout: "root_rbytes": 0, 2024-05-12T22:20:16.880 INFO:teuthology.orchestra.run.smithi186.stdout: "root_rsnaps": 0, 2024-05-12T22:20:16.880 INFO:teuthology.orchestra.run.smithi186.stdout: "scrub_backtrace_fetch": 0, 2024-05-12T22:20:16.880 INFO:teuthology.orchestra.run.smithi186.stdout: "scrub_set_tag": 0, 2024-05-12T22:20:16.881 INFO:teuthology.orchestra.run.smithi186.stdout: "scrub_backtrace_repaired": 0, 2024-05-12T22:20:16.881 INFO:teuthology.orchestra.run.smithi186.stdout: "scrub_inotable_repaired": 0, 2024-05-12T22:20:16.881 INFO:teuthology.orchestra.run.smithi186.stdout: "scrub_dir_inodes": 0, 2024-05-12T22:20:16.881 INFO:teuthology.orchestra.run.smithi186.stdout: "scrub_dir_base_inodes": 0, 2024-05-12T22:20:16.881 INFO:teuthology.orchestra.run.smithi186.stdout: "scrub_dirfrag_rstats": 0, 2024-05-12T22:20:16.881 INFO:teuthology.orchestra.run.smithi186.stdout: "scrub_file_inodes": 0, 2024-05-12T22:20:16.881 INFO:teuthology.orchestra.run.smithi186.stdout: "handle_inode_file_caps": 0, 2024-05-12T22:20:16.882 INFO:teuthology.orchestra.run.smithi186.stdout: "ceph_cap_op_revoke": 0, 2024-05-12T22:20:16.882 INFO:teuthology.orchestra.run.smithi186.stdout: "ceph_cap_op_grant": 0, 2024-05-12T22:20:16.882 INFO:teuthology.orchestra.run.smithi186.stdout: "ceph_cap_op_trunc": 0, 2024-05-12T22:20:16.882 INFO:teuthology.orchestra.run.smithi186.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-05-12T22:20:16.882 INFO:teuthology.orchestra.run.smithi186.stdout: "ceph_cap_op_flush_ack": 0, 2024-05-12T22:20:16.883 INFO:teuthology.orchestra.run.smithi186.stdout: "handle_client_caps": 0, 2024-05-12T22:20:16.883 INFO:teuthology.orchestra.run.smithi186.stdout: "handle_client_caps_dirty": 0, 2024-05-12T22:20:16.883 INFO:teuthology.orchestra.run.smithi186.stdout: "handle_client_cap_release": 0, 2024-05-12T22:20:16.883 INFO:teuthology.orchestra.run.smithi186.stdout: "process_request_cap_release": 0 2024-05-12T22:20:16.883 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:16.883 INFO:teuthology.orchestra.run.smithi186.stdout: "mds_cache": { 2024-05-12T22:20:16.883 INFO:teuthology.orchestra.run.smithi186.stdout: "dir_update": 0, 2024-05-12T22:20:16.883 INFO:teuthology.orchestra.run.smithi186.stdout: "dir_update_receipt": 0, 2024-05-12T22:20:16.883 INFO:teuthology.orchestra.run.smithi186.stdout: "dir_try_discover": 0, 2024-05-12T22:20:16.884 INFO:teuthology.orchestra.run.smithi186.stdout: "dir_send_discover": 0, 2024-05-12T22:20:16.884 INFO:teuthology.orchestra.run.smithi186.stdout: "dir_handle_discover": 0, 2024-05-12T22:20:16.884 INFO:teuthology.orchestra.run.smithi186.stdout: "num_strays": 0, 2024-05-12T22:20:16.884 INFO:teuthology.orchestra.run.smithi186.stdout: "num_strays_delayed": 0, 2024-05-12T22:20:16.884 INFO:teuthology.orchestra.run.smithi186.stdout: "num_strays_enqueuing": 0, 2024-05-12T22:20:16.884 INFO:teuthology.orchestra.run.smithi186.stdout: "strays_created": 0, 2024-05-12T22:20:16.884 INFO:teuthology.orchestra.run.smithi186.stdout: "strays_enqueued": 0, 2024-05-12T22:20:16.884 INFO:teuthology.orchestra.run.smithi186.stdout: "strays_reintegrated": 0, 2024-05-12T22:20:16.884 INFO:teuthology.orchestra.run.smithi186.stdout: "strays_migrated": 0, 2024-05-12T22:20:16.885 INFO:teuthology.orchestra.run.smithi186.stdout: "num_recovering_processing": 0, 2024-05-12T22:20:16.885 INFO:teuthology.orchestra.run.smithi186.stdout: "num_recovering_enqueued": 0, 2024-05-12T22:20:16.885 INFO:teuthology.orchestra.run.smithi186.stdout: "num_recovering_prioritized": 0, 2024-05-12T22:20:16.885 INFO:teuthology.orchestra.run.smithi186.stdout: "recovery_started": 0, 2024-05-12T22:20:16.885 INFO:teuthology.orchestra.run.smithi186.stdout: "recovery_completed": 0, 2024-05-12T22:20:16.885 INFO:teuthology.orchestra.run.smithi186.stdout: "ireq_quiesce_path": 0, 2024-05-12T22:20:16.885 INFO:teuthology.orchestra.run.smithi186.stdout: "ireq_quiesce_inode": 0, 2024-05-12T22:20:16.885 INFO:teuthology.orchestra.run.smithi186.stdout: "ireq_enqueue_scrub": 0, 2024-05-12T22:20:16.885 INFO:teuthology.orchestra.run.smithi186.stdout: "ireq_exportdir": 0, 2024-05-12T22:20:16.886 INFO:teuthology.orchestra.run.smithi186.stdout: "ireq_flush": 0, 2024-05-12T22:20:16.886 INFO:teuthology.orchestra.run.smithi186.stdout: "ireq_fragmentdir": 0, 2024-05-12T22:20:16.886 INFO:teuthology.orchestra.run.smithi186.stdout: "ireq_fragstats": 0, 2024-05-12T22:20:16.886 INFO:teuthology.orchestra.run.smithi186.stdout: "ireq_inodestats": 0 2024-05-12T22:20:16.886 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:16.886 INFO:teuthology.orchestra.run.smithi186.stdout: "mds_log": { 2024-05-12T22:20:16.886 INFO:teuthology.orchestra.run.smithi186.stdout: "evlrg": 0, 2024-05-12T22:20:16.886 INFO:teuthology.orchestra.run.smithi186.stdout: "evadd": 0, 2024-05-12T22:20:16.886 INFO:teuthology.orchestra.run.smithi186.stdout: "evex": 0, 2024-05-12T22:20:16.887 INFO:teuthology.orchestra.run.smithi186.stdout: "evtrm": 0, 2024-05-12T22:20:16.887 INFO:teuthology.orchestra.run.smithi186.stdout: "ev": 45, 2024-05-12T22:20:16.887 INFO:teuthology.orchestra.run.smithi186.stdout: "evexg": 0, 2024-05-12T22:20:16.887 INFO:teuthology.orchestra.run.smithi186.stdout: "evexd": 0, 2024-05-12T22:20:16.887 INFO:teuthology.orchestra.run.smithi186.stdout: "segadd": 0, 2024-05-12T22:20:16.887 INFO:teuthology.orchestra.run.smithi186.stdout: "segex": 0, 2024-05-12T22:20:16.887 INFO:teuthology.orchestra.run.smithi186.stdout: "segtrm": 0, 2024-05-12T22:20:16.887 INFO:teuthology.orchestra.run.smithi186.stdout: "seg": 2, 2024-05-12T22:20:16.888 INFO:teuthology.orchestra.run.smithi186.stdout: "segmjr": 2, 2024-05-12T22:20:16.888 INFO:teuthology.orchestra.run.smithi186.stdout: "segexg": 0, 2024-05-12T22:20:16.888 INFO:teuthology.orchestra.run.smithi186.stdout: "segexd": 0, 2024-05-12T22:20:16.888 INFO:teuthology.orchestra.run.smithi186.stdout: "expos": 4194304, 2024-05-12T22:20:16.888 INFO:teuthology.orchestra.run.smithi186.stdout: "wrpos": 4268093, 2024-05-12T22:20:16.888 INFO:teuthology.orchestra.run.smithi186.stdout: "rdpos": 4267519, 2024-05-12T22:20:16.888 INFO:teuthology.orchestra.run.smithi186.stdout: "jlat": { 2024-05-12T22:20:16.888 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-05-12T22:20:16.889 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-05-12T22:20:16.889 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-05-12T22:20:16.889 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:16.889 INFO:teuthology.orchestra.run.smithi186.stdout: "replayed": 45 2024-05-12T22:20:16.889 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:16.889 INFO:teuthology.orchestra.run.smithi186.stdout: "mds_mem": { 2024-05-12T22:20:16.889 INFO:teuthology.orchestra.run.smithi186.stdout: "ino": 29, 2024-05-12T22:20:16.889 INFO:teuthology.orchestra.run.smithi186.stdout: "ino+": 31, 2024-05-12T22:20:16.889 INFO:teuthology.orchestra.run.smithi186.stdout: "ino-": 2, 2024-05-12T22:20:16.890 INFO:teuthology.orchestra.run.smithi186.stdout: "dir": 17, 2024-05-12T22:20:16.890 INFO:teuthology.orchestra.run.smithi186.stdout: "dir+": 17, 2024-05-12T22:20:16.890 INFO:teuthology.orchestra.run.smithi186.stdout: "dir-": 0, 2024-05-12T22:20:16.890 INFO:teuthology.orchestra.run.smithi186.stdout: "dn": 31, 2024-05-12T22:20:16.890 INFO:teuthology.orchestra.run.smithi186.stdout: "dn+": 31, 2024-05-12T22:20:16.890 INFO:teuthology.orchestra.run.smithi186.stdout: "dn-": 0, 2024-05-12T22:20:16.890 INFO:teuthology.orchestra.run.smithi186.stdout: "cap": 0, 2024-05-12T22:20:16.890 INFO:teuthology.orchestra.run.smithi186.stdout: "cap+": 0, 2024-05-12T22:20:16.890 INFO:teuthology.orchestra.run.smithi186.stdout: "cap-": 0, 2024-05-12T22:20:16.891 INFO:teuthology.orchestra.run.smithi186.stdout: "rss": 51164, 2024-05-12T22:20:16.891 INFO:teuthology.orchestra.run.smithi186.stdout: "heap": 215420 2024-05-12T22:20:16.891 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:16.891 INFO:teuthology.orchestra.run.smithi186.stdout: "mds_server": { 2024-05-12T22:20:16.891 INFO:teuthology.orchestra.run.smithi186.stdout: "dispatch_client_request": 0, 2024-05-12T22:20:16.891 INFO:teuthology.orchestra.run.smithi186.stdout: "dispatch_server_request": 0, 2024-05-12T22:20:16.891 INFO:teuthology.orchestra.run.smithi186.stdout: "handle_client_request": 0, 2024-05-12T22:20:16.891 INFO:teuthology.orchestra.run.smithi186.stdout: "handle_client_session": 0, 2024-05-12T22:20:16.891 INFO:teuthology.orchestra.run.smithi186.stdout: "handle_peer_request": 0, 2024-05-12T22:20:16.892 INFO:teuthology.orchestra.run.smithi186.stdout: "req_create_latency": { 2024-05-12T22:20:16.892 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-05-12T22:20:16.892 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-05-12T22:20:16.892 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-05-12T22:20:16.892 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:16.892 INFO:teuthology.orchestra.run.smithi186.stdout: "req_getattr_latency": { 2024-05-12T22:20:16.892 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-05-12T22:20:16.892 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-05-12T22:20:16.892 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-05-12T22:20:16.893 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:16.893 INFO:teuthology.orchestra.run.smithi186.stdout: "req_getfilelock_latency": { 2024-05-12T22:20:16.893 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-05-12T22:20:16.893 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-05-12T22:20:16.893 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-05-12T22:20:16.893 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:16.893 INFO:teuthology.orchestra.run.smithi186.stdout: "req_link_latency": { 2024-05-12T22:20:16.893 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-05-12T22:20:16.893 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-05-12T22:20:16.894 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-05-12T22:20:16.894 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:16.894 INFO:teuthology.orchestra.run.smithi186.stdout: "req_lookup_latency": { 2024-05-12T22:20:16.894 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-05-12T22:20:16.894 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-05-12T22:20:16.894 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-05-12T22:20:16.894 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:16.894 INFO:teuthology.orchestra.run.smithi186.stdout: "req_lookuphash_latency": { 2024-05-12T22:20:16.894 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-05-12T22:20:16.895 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-05-12T22:20:16.895 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-05-12T22:20:16.895 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:16.895 INFO:teuthology.orchestra.run.smithi186.stdout: "req_lookupino_latency": { 2024-05-12T22:20:16.895 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-05-12T22:20:16.895 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-05-12T22:20:16.895 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-05-12T22:20:16.895 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:16.896 INFO:teuthology.orchestra.run.smithi186.stdout: "req_lookupname_latency": { 2024-05-12T22:20:16.896 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-05-12T22:20:16.896 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-05-12T22:20:16.896 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-05-12T22:20:16.896 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:16.896 INFO:teuthology.orchestra.run.smithi186.stdout: "req_lookupparent_latency": { 2024-05-12T22:20:16.896 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-05-12T22:20:16.896 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-05-12T22:20:16.896 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-05-12T22:20:16.897 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:16.897 INFO:teuthology.orchestra.run.smithi186.stdout: "req_lookupsnap_latency": { 2024-05-12T22:20:16.897 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-05-12T22:20:16.897 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-05-12T22:20:16.897 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-05-12T22:20:16.897 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:16.897 INFO:teuthology.orchestra.run.smithi186.stdout: "req_lssnap_latency": { 2024-05-12T22:20:16.897 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-05-12T22:20:16.898 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-05-12T22:20:16.898 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-05-12T22:20:16.898 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:16.898 INFO:teuthology.orchestra.run.smithi186.stdout: "req_mkdir_latency": { 2024-05-12T22:20:16.898 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-05-12T22:20:16.899 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-05-12T22:20:16.899 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-05-12T22:20:16.899 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:16.899 INFO:teuthology.orchestra.run.smithi186.stdout: "req_mknod_latency": { 2024-05-12T22:20:16.899 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-05-12T22:20:16.899 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-05-12T22:20:16.900 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-05-12T22:20:16.900 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:16.900 INFO:teuthology.orchestra.run.smithi186.stdout: "req_mksnap_latency": { 2024-05-12T22:20:16.900 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-05-12T22:20:16.900 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-05-12T22:20:16.900 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-05-12T22:20:16.900 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:16.900 INFO:teuthology.orchestra.run.smithi186.stdout: "req_open_latency": { 2024-05-12T22:20:16.901 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-05-12T22:20:16.901 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-05-12T22:20:16.901 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-05-12T22:20:16.901 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:16.901 INFO:teuthology.orchestra.run.smithi186.stdout: "req_readdir_latency": { 2024-05-12T22:20:16.901 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-05-12T22:20:16.901 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-05-12T22:20:16.901 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-05-12T22:20:16.901 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:16.902 INFO:teuthology.orchestra.run.smithi186.stdout: "req_rename_latency": { 2024-05-12T22:20:16.902 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-05-12T22:20:16.902 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-05-12T22:20:16.902 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-05-12T22:20:16.902 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:16.902 INFO:teuthology.orchestra.run.smithi186.stdout: "req_renamesnap_latency": { 2024-05-12T22:20:16.902 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-05-12T22:20:16.902 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-05-12T22:20:16.903 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-05-12T22:20:16.903 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:16.903 INFO:teuthology.orchestra.run.smithi186.stdout: "req_snapdiff_latency": { 2024-05-12T22:20:16.903 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-05-12T22:20:16.903 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-05-12T22:20:16.903 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-05-12T22:20:16.903 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:16.903 INFO:teuthology.orchestra.run.smithi186.stdout: "req_rmdir_latency": { 2024-05-12T22:20:16.903 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-05-12T22:20:16.904 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-05-12T22:20:16.904 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-05-12T22:20:16.904 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:16.904 INFO:teuthology.orchestra.run.smithi186.stdout: "req_rmsnap_latency": { 2024-05-12T22:20:16.904 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-05-12T22:20:16.904 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-05-12T22:20:16.904 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-05-12T22:20:16.904 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:16.904 INFO:teuthology.orchestra.run.smithi186.stdout: "req_rmxattr_latency": { 2024-05-12T22:20:16.905 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-05-12T22:20:16.905 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-05-12T22:20:16.905 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-05-12T22:20:16.905 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:16.905 INFO:teuthology.orchestra.run.smithi186.stdout: "req_setattr_latency": { 2024-05-12T22:20:16.905 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-05-12T22:20:16.905 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-05-12T22:20:16.905 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-05-12T22:20:16.905 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:16.906 INFO:teuthology.orchestra.run.smithi186.stdout: "req_setdirlayout_latency": { 2024-05-12T22:20:16.906 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-05-12T22:20:16.906 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-05-12T22:20:16.906 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-05-12T22:20:16.906 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:16.906 INFO:teuthology.orchestra.run.smithi186.stdout: "req_setfilelock_latency": { 2024-05-12T22:20:16.906 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-05-12T22:20:16.906 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-05-12T22:20:16.907 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-05-12T22:20:16.907 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:16.907 INFO:teuthology.orchestra.run.smithi186.stdout: "req_setlayout_latency": { 2024-05-12T22:20:16.907 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-05-12T22:20:16.907 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-05-12T22:20:16.907 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-05-12T22:20:16.907 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:16.907 INFO:teuthology.orchestra.run.smithi186.stdout: "req_setxattr_latency": { 2024-05-12T22:20:16.907 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-05-12T22:20:16.908 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-05-12T22:20:16.908 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-05-12T22:20:16.908 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:16.908 INFO:teuthology.orchestra.run.smithi186.stdout: "req_symlink_latency": { 2024-05-12T22:20:16.908 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-05-12T22:20:16.908 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-05-12T22:20:16.908 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-05-12T22:20:16.908 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:16.908 INFO:teuthology.orchestra.run.smithi186.stdout: "req_unlink_latency": { 2024-05-12T22:20:16.909 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-05-12T22:20:16.909 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-05-12T22:20:16.909 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-05-12T22:20:16.909 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:16.909 INFO:teuthology.orchestra.run.smithi186.stdout: "cap_revoke_eviction": 0, 2024-05-12T22:20:16.909 INFO:teuthology.orchestra.run.smithi186.stdout: "cap_acquisition_throttle": 0, 2024-05-12T22:20:16.909 INFO:teuthology.orchestra.run.smithi186.stdout: "req_getvxattr_latency": { 2024-05-12T22:20:16.909 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-05-12T22:20:16.909 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-05-12T22:20:16.910 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-05-12T22:20:16.910 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-05-12T22:20:16.910 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:16.910 INFO:teuthology.orchestra.run.smithi186.stdout: "mds_sessions": { 2024-05-12T22:20:16.910 INFO:teuthology.orchestra.run.smithi186.stdout: "session_count": 2, 2024-05-12T22:20:16.910 INFO:teuthology.orchestra.run.smithi186.stdout: "session_add": 5, 2024-05-12T22:20:16.910 INFO:teuthology.orchestra.run.smithi186.stdout: "session_remove": 3, 2024-05-12T22:20:16.910 INFO:teuthology.orchestra.run.smithi186.stdout: "sessions_open": 2, 2024-05-12T22:20:16.911 INFO:teuthology.orchestra.run.smithi186.stdout: "sessions_stale": 0, 2024-05-12T22:20:16.911 INFO:teuthology.orchestra.run.smithi186.stdout: "total_load": 0, 2024-05-12T22:20:16.911 INFO:teuthology.orchestra.run.smithi186.stdout: "average_load": 0, 2024-05-12T22:20:16.911 INFO:teuthology.orchestra.run.smithi186.stdout: "avg_session_uptime": 5135, 2024-05-12T22:20:16.911 INFO:teuthology.orchestra.run.smithi186.stdout: "mdthresh_evicted": 0 2024-05-12T22:20:16.911 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:16.911 INFO:teuthology.orchestra.run.smithi186.stdout: "mempool": { 2024-05-12T22:20:16.911 INFO:teuthology.orchestra.run.smithi186.stdout: "bloom_filter_bytes": 0, 2024-05-12T22:20:16.911 INFO:teuthology.orchestra.run.smithi186.stdout: "bloom_filter_items": 0, 2024-05-12T22:20:16.912 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_alloc_bytes": 0, 2024-05-12T22:20:16.912 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_alloc_items": 0, 2024-05-12T22:20:16.912 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_data_bytes": 0, 2024-05-12T22:20:16.912 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_data_items": 0, 2024-05-12T22:20:16.912 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_onode_bytes": 0, 2024-05-12T22:20:16.912 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_onode_items": 0, 2024-05-12T22:20:16.912 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_meta_bytes": 0, 2024-05-12T22:20:16.912 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_meta_items": 0, 2024-05-12T22:20:16.912 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_other_bytes": 0, 2024-05-12T22:20:16.913 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_other_items": 0, 2024-05-12T22:20:16.913 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_buffer_bytes": 0, 2024-05-12T22:20:16.913 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_buffer_items": 0, 2024-05-12T22:20:16.913 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_extent_bytes": 0, 2024-05-12T22:20:16.913 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_extent_items": 0, 2024-05-12T22:20:16.913 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_blob_bytes": 0, 2024-05-12T22:20:16.913 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_blob_items": 0, 2024-05-12T22:20:16.913 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_shared_blob_bytes": 0, 2024-05-12T22:20:16.913 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_shared_blob_items": 0, 2024-05-12T22:20:16.914 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_inline_bl_bytes": 0, 2024-05-12T22:20:16.914 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_inline_bl_items": 0, 2024-05-12T22:20:16.914 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_fsck_bytes": 0, 2024-05-12T22:20:16.914 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_fsck_items": 0, 2024-05-12T22:20:16.914 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_txc_bytes": 0, 2024-05-12T22:20:16.914 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_txc_items": 0, 2024-05-12T22:20:16.914 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_writing_deferred_bytes": 0, 2024-05-12T22:20:16.914 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_writing_deferred_items": 0, 2024-05-12T22:20:16.915 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_writing_bytes": 0, 2024-05-12T22:20:16.915 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_writing_items": 0, 2024-05-12T22:20:16.915 INFO:teuthology.orchestra.run.smithi186.stdout: "bluefs_bytes": 0, 2024-05-12T22:20:16.915 INFO:teuthology.orchestra.run.smithi186.stdout: "bluefs_items": 0, 2024-05-12T22:20:16.915 INFO:teuthology.orchestra.run.smithi186.stdout: "bluefs_file_reader_bytes": 0, 2024-05-12T22:20:16.915 INFO:teuthology.orchestra.run.smithi186.stdout: "bluefs_file_reader_items": 0, 2024-05-12T22:20:16.915 INFO:teuthology.orchestra.run.smithi186.stdout: "bluefs_file_writer_bytes": 0, 2024-05-12T22:20:16.915 INFO:teuthology.orchestra.run.smithi186.stdout: "bluefs_file_writer_items": 0, 2024-05-12T22:20:16.916 INFO:teuthology.orchestra.run.smithi186.stdout: "buffer_anon_bytes": 182122, 2024-05-12T22:20:16.916 INFO:teuthology.orchestra.run.smithi186.stdout: "buffer_anon_items": 70, 2024-05-12T22:20:16.916 INFO:teuthology.orchestra.run.smithi186.stdout: "buffer_meta_bytes": 0, 2024-05-12T22:20:16.916 INFO:teuthology.orchestra.run.smithi186.stdout: "buffer_meta_items": 0, 2024-05-12T22:20:16.916 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_bytes": 0, 2024-05-12T22:20:16.916 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_items": 0, 2024-05-12T22:20:16.916 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_mapbl_bytes": 0, 2024-05-12T22:20:16.916 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_mapbl_items": 0, 2024-05-12T22:20:16.916 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_pglog_bytes": 0, 2024-05-12T22:20:16.917 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_pglog_items": 0, 2024-05-12T22:20:16.917 INFO:teuthology.orchestra.run.smithi186.stdout: "osdmap_bytes": 8032, 2024-05-12T22:20:16.917 INFO:teuthology.orchestra.run.smithi186.stdout: "osdmap_items": 167, 2024-05-12T22:20:16.917 INFO:teuthology.orchestra.run.smithi186.stdout: "osdmap_mapping_bytes": 0, 2024-05-12T22:20:16.917 INFO:teuthology.orchestra.run.smithi186.stdout: "osdmap_mapping_items": 0, 2024-05-12T22:20:16.917 INFO:teuthology.orchestra.run.smithi186.stdout: "pgmap_bytes": 0, 2024-05-12T22:20:16.917 INFO:teuthology.orchestra.run.smithi186.stdout: "pgmap_items": 0, 2024-05-12T22:20:16.917 INFO:teuthology.orchestra.run.smithi186.stdout: "mds_co_bytes": 114011, 2024-05-12T22:20:16.917 INFO:teuthology.orchestra.run.smithi186.stdout: "mds_co_items": 747, 2024-05-12T22:20:16.918 INFO:teuthology.orchestra.run.smithi186.stdout: "unittest_1_bytes": 0, 2024-05-12T22:20:16.918 INFO:teuthology.orchestra.run.smithi186.stdout: "unittest_1_items": 0, 2024-05-12T22:20:16.918 INFO:teuthology.orchestra.run.smithi186.stdout: "unittest_2_bytes": 0, 2024-05-12T22:20:16.918 INFO:teuthology.orchestra.run.smithi186.stdout: "unittest_2_items": 0 2024-05-12T22:20:16.918 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:16.918 INFO:teuthology.orchestra.run.smithi186.stdout: "objecter": { 2024-05-12T22:20:16.918 INFO:teuthology.orchestra.run.smithi186.stdout: "op_active": 0, 2024-05-12T22:20:16.918 INFO:teuthology.orchestra.run.smithi186.stdout: "op_laggy": 0, 2024-05-12T22:20:16.919 INFO:teuthology.orchestra.run.smithi186.stdout: "op_send": 2409, 2024-05-12T22:20:16.919 INFO:teuthology.orchestra.run.smithi186.stdout: "op_send_bytes": 21, 2024-05-12T22:20:16.919 INFO:teuthology.orchestra.run.smithi186.stdout: "op_resend": 3, 2024-05-12T22:20:16.919 INFO:teuthology.orchestra.run.smithi186.stdout: "op_reply": 2409, 2024-05-12T22:20:16.919 INFO:teuthology.orchestra.run.smithi186.stdout: "op_latency": { 2024-05-12T22:20:16.919 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 2409, 2024-05-12T22:20:16.919 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 1.257977014, 2024-05-12T22:20:16.919 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000522198 2024-05-12T22:20:16.919 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:16.920 INFO:teuthology.orchestra.run.smithi186.stdout: "op_inflight": 0, 2024-05-12T22:20:16.920 INFO:teuthology.orchestra.run.smithi186.stdout: "oplen_avg": { 2024-05-12T22:20:16.920 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 2409, 2024-05-12T22:20:16.920 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 2410 2024-05-12T22:20:16.920 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:16.920 INFO:teuthology.orchestra.run.smithi186.stdout: "op": 2409, 2024-05-12T22:20:16.920 INFO:teuthology.orchestra.run.smithi186.stdout: "op_r": 2409, 2024-05-12T22:20:16.920 INFO:teuthology.orchestra.run.smithi186.stdout: "op_w": 0, 2024-05-12T22:20:16.920 INFO:teuthology.orchestra.run.smithi186.stdout: "op_rmw": 0, 2024-05-12T22:20:16.921 INFO:teuthology.orchestra.run.smithi186.stdout: "op_pg": 0, 2024-05-12T22:20:16.921 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_stat": 1567, 2024-05-12T22:20:16.921 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_create": 0, 2024-05-12T22:20:16.921 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_read": 840, 2024-05-12T22:20:16.921 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_write": 0, 2024-05-12T22:20:16.921 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_writefull": 0, 2024-05-12T22:20:16.921 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_writesame": 0, 2024-05-12T22:20:16.921 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_append": 0, 2024-05-12T22:20:16.921 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_zero": 0, 2024-05-12T22:20:16.922 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_truncate": 0, 2024-05-12T22:20:16.922 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_delete": 0, 2024-05-12T22:20:16.922 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_mapext": 0, 2024-05-12T22:20:16.922 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_sparse_read": 0, 2024-05-12T22:20:16.922 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_clonerange": 0, 2024-05-12T22:20:16.922 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_getxattr": 1, 2024-05-12T22:20:16.922 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_setxattr": 0, 2024-05-12T22:20:16.922 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_cmpxattr": 0, 2024-05-12T22:20:16.922 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_rmxattr": 0, 2024-05-12T22:20:16.923 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_resetxattrs": 0, 2024-05-12T22:20:16.923 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_call": 0, 2024-05-12T22:20:16.923 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_watch": 0, 2024-05-12T22:20:16.923 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_notify": 0, 2024-05-12T22:20:16.923 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_src_cmpxattr": 0, 2024-05-12T22:20:16.923 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_pgls": 0, 2024-05-12T22:20:16.923 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_pgls_filter": 0, 2024-05-12T22:20:16.923 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_other": 0, 2024-05-12T22:20:16.923 INFO:teuthology.orchestra.run.smithi186.stdout: "linger_active": 0, 2024-05-12T22:20:16.924 INFO:teuthology.orchestra.run.smithi186.stdout: "linger_send": 0, 2024-05-12T22:20:16.924 INFO:teuthology.orchestra.run.smithi186.stdout: "linger_resend": 0, 2024-05-12T22:20:16.924 INFO:teuthology.orchestra.run.smithi186.stdout: "linger_ping": 0, 2024-05-12T22:20:16.924 INFO:teuthology.orchestra.run.smithi186.stdout: "poolop_active": 0, 2024-05-12T22:20:16.924 INFO:teuthology.orchestra.run.smithi186.stdout: "poolop_send": 0, 2024-05-12T22:20:16.924 INFO:teuthology.orchestra.run.smithi186.stdout: "poolop_resend": 0, 2024-05-12T22:20:16.924 INFO:teuthology.orchestra.run.smithi186.stdout: "poolstat_active": 0, 2024-05-12T22:20:16.924 INFO:teuthology.orchestra.run.smithi186.stdout: "poolstat_send": 0, 2024-05-12T22:20:16.925 INFO:teuthology.orchestra.run.smithi186.stdout: "poolstat_resend": 0, 2024-05-12T22:20:16.925 INFO:teuthology.orchestra.run.smithi186.stdout: "statfs_active": 0, 2024-05-12T22:20:16.925 INFO:teuthology.orchestra.run.smithi186.stdout: "statfs_send": 0, 2024-05-12T22:20:16.925 INFO:teuthology.orchestra.run.smithi186.stdout: "statfs_resend": 0, 2024-05-12T22:20:16.925 INFO:teuthology.orchestra.run.smithi186.stdout: "command_active": 0, 2024-05-12T22:20:16.925 INFO:teuthology.orchestra.run.smithi186.stdout: "command_send": 0, 2024-05-12T22:20:16.925 INFO:teuthology.orchestra.run.smithi186.stdout: "command_resend": 0, 2024-05-12T22:20:16.925 INFO:teuthology.orchestra.run.smithi186.stdout: "map_epoch": 99, 2024-05-12T22:20:16.925 INFO:teuthology.orchestra.run.smithi186.stdout: "map_full": 0, 2024-05-12T22:20:16.926 INFO:teuthology.orchestra.run.smithi186.stdout: "map_inc": 13, 2024-05-12T22:20:16.926 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_sessions": 4, 2024-05-12T22:20:16.926 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_session_open": 4, 2024-05-12T22:20:16.926 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_session_close": 0, 2024-05-12T22:20:16.926 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_laggy": 0, 2024-05-12T22:20:16.926 INFO:teuthology.orchestra.run.smithi186.stdout: "omap_wr": 0, 2024-05-12T22:20:16.926 INFO:teuthology.orchestra.run.smithi186.stdout: "omap_rd": 2, 2024-05-12T22:20:16.926 INFO:teuthology.orchestra.run.smithi186.stdout: "omap_del": 0 2024-05-12T22:20:16.926 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:16.927 INFO:teuthology.orchestra.run.smithi186.stdout: "oft": { 2024-05-12T22:20:16.927 INFO:teuthology.orchestra.run.smithi186.stdout: "omap_total_objs": 0, 2024-05-12T22:20:16.927 INFO:teuthology.orchestra.run.smithi186.stdout: "omap_total_kv_pairs": 0, 2024-05-12T22:20:16.927 INFO:teuthology.orchestra.run.smithi186.stdout: "omap_total_updates": 0, 2024-05-12T22:20:16.927 INFO:teuthology.orchestra.run.smithi186.stdout: "omap_total_removes": 0 2024-05-12T22:20:16.927 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:16.927 INFO:teuthology.orchestra.run.smithi186.stdout: "osd-slow-ops": { 2024-05-12T22:20:16.927 INFO:teuthology.orchestra.run.smithi186.stdout: "slow_ops_count": 0 2024-05-12T22:20:16.928 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:16.928 INFO:teuthology.orchestra.run.smithi186.stdout: "purge_queue": { 2024-05-12T22:20:16.928 INFO:teuthology.orchestra.run.smithi186.stdout: "pq_executing_ops": 0, 2024-05-12T22:20:16.928 INFO:teuthology.orchestra.run.smithi186.stdout: "pq_executing_ops_high_water": 0, 2024-05-12T22:20:16.928 INFO:teuthology.orchestra.run.smithi186.stdout: "pq_executing": 0, 2024-05-12T22:20:16.928 INFO:teuthology.orchestra.run.smithi186.stdout: "pq_executing_high_water": 0, 2024-05-12T22:20:16.928 INFO:teuthology.orchestra.run.smithi186.stdout: "pq_executed_ops": 0, 2024-05-12T22:20:16.928 INFO:teuthology.orchestra.run.smithi186.stdout: "pq_executed": 0, 2024-05-12T22:20:16.928 INFO:teuthology.orchestra.run.smithi186.stdout: "pq_item_in_journal": 0 2024-05-12T22:20:16.929 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:16.929 INFO:teuthology.orchestra.run.smithi186.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-05-12T22:20:16.929 INFO:teuthology.orchestra.run.smithi186.stdout: "val": 0, 2024-05-12T22:20:16.929 INFO:teuthology.orchestra.run.smithi186.stdout: "max": 104857600, 2024-05-12T22:20:16.929 INFO:teuthology.orchestra.run.smithi186.stdout: "get_started": 0, 2024-05-12T22:20:16.929 INFO:teuthology.orchestra.run.smithi186.stdout: "get": 2671, 2024-05-12T22:20:16.929 INFO:teuthology.orchestra.run.smithi186.stdout: "get_sum": 2262069, 2024-05-12T22:20:16.929 INFO:teuthology.orchestra.run.smithi186.stdout: "get_or_fail_fail": 0, 2024-05-12T22:20:16.929 INFO:teuthology.orchestra.run.smithi186.stdout: "get_or_fail_success": 2671, 2024-05-12T22:20:16.930 INFO:teuthology.orchestra.run.smithi186.stdout: "take": 0, 2024-05-12T22:20:16.930 INFO:teuthology.orchestra.run.smithi186.stdout: "take_sum": 0, 2024-05-12T22:20:16.930 INFO:teuthology.orchestra.run.smithi186.stdout: "put": 2671, 2024-05-12T22:20:16.930 INFO:teuthology.orchestra.run.smithi186.stdout: "put_sum": 2262069, 2024-05-12T22:20:16.930 INFO:teuthology.orchestra.run.smithi186.stdout: "wait": { 2024-05-12T22:20:16.930 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-05-12T22:20:16.930 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-05-12T22:20:16.930 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-05-12T22:20:16.930 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-05-12T22:20:16.931 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:16.931 INFO:teuthology.orchestra.run.smithi186.stdout: "throttle-objecter_bytes": { 2024-05-12T22:20:16.931 INFO:teuthology.orchestra.run.smithi186.stdout: "val": 0, 2024-05-12T22:20:16.931 INFO:teuthology.orchestra.run.smithi186.stdout: "max": 104857600, 2024-05-12T22:20:16.931 INFO:teuthology.orchestra.run.smithi186.stdout: "get_started": 0, 2024-05-12T22:20:16.931 INFO:teuthology.orchestra.run.smithi186.stdout: "get": 0, 2024-05-12T22:20:16.931 INFO:teuthology.orchestra.run.smithi186.stdout: "get_sum": 0, 2024-05-12T22:20:16.931 INFO:teuthology.orchestra.run.smithi186.stdout: "get_or_fail_fail": 0, 2024-05-12T22:20:16.931 INFO:teuthology.orchestra.run.smithi186.stdout: "get_or_fail_success": 0, 2024-05-12T22:20:16.932 INFO:teuthology.orchestra.run.smithi186.stdout: "take": 2409, 2024-05-12T22:20:16.932 INFO:teuthology.orchestra.run.smithi186.stdout: "take_sum": 73794, 2024-05-12T22:20:16.932 INFO:teuthology.orchestra.run.smithi186.stdout: "put": 2409, 2024-05-12T22:20:16.932 INFO:teuthology.orchestra.run.smithi186.stdout: "put_sum": 73794, 2024-05-12T22:20:16.932 INFO:teuthology.orchestra.run.smithi186.stdout: "wait": { 2024-05-12T22:20:16.932 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-05-12T22:20:16.932 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-05-12T22:20:16.932 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-05-12T22:20:16.933 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-05-12T22:20:16.933 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:16.933 INFO:teuthology.orchestra.run.smithi186.stdout: "throttle-objecter_ops": { 2024-05-12T22:20:16.933 INFO:teuthology.orchestra.run.smithi186.stdout: "val": 0, 2024-05-12T22:20:16.933 INFO:teuthology.orchestra.run.smithi186.stdout: "max": 1024, 2024-05-12T22:20:16.933 INFO:teuthology.orchestra.run.smithi186.stdout: "get_started": 0, 2024-05-12T22:20:16.933 INFO:teuthology.orchestra.run.smithi186.stdout: "get": 0, 2024-05-12T22:20:16.933 INFO:teuthology.orchestra.run.smithi186.stdout: "get_sum": 0, 2024-05-12T22:20:16.934 INFO:teuthology.orchestra.run.smithi186.stdout: "get_or_fail_fail": 0, 2024-05-12T22:20:16.934 INFO:teuthology.orchestra.run.smithi186.stdout: "get_or_fail_success": 0, 2024-05-12T22:20:16.934 INFO:teuthology.orchestra.run.smithi186.stdout: "take": 2409, 2024-05-12T22:20:16.934 INFO:teuthology.orchestra.run.smithi186.stdout: "take_sum": 2409, 2024-05-12T22:20:16.934 INFO:teuthology.orchestra.run.smithi186.stdout: "put": 2409, 2024-05-12T22:20:16.934 INFO:teuthology.orchestra.run.smithi186.stdout: "put_sum": 2409, 2024-05-12T22:20:16.934 INFO:teuthology.orchestra.run.smithi186.stdout: "wait": { 2024-05-12T22:20:16.934 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-05-12T22:20:16.934 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-05-12T22:20:16.935 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-05-12T22:20:16.935 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-05-12T22:20:16.935 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:16.935 INFO:teuthology.orchestra.run.smithi186.stdout: "throttle-write_buf_throttle": { 2024-05-12T22:20:16.935 INFO:teuthology.orchestra.run.smithi186.stdout: "val": 0, 2024-05-12T22:20:16.935 INFO:teuthology.orchestra.run.smithi186.stdout: "max": 3758096384, 2024-05-12T22:20:16.935 INFO:teuthology.orchestra.run.smithi186.stdout: "get_started": 0, 2024-05-12T22:20:16.935 INFO:teuthology.orchestra.run.smithi186.stdout: "get": 0, 2024-05-12T22:20:16.936 INFO:teuthology.orchestra.run.smithi186.stdout: "get_sum": 0, 2024-05-12T22:20:16.936 INFO:teuthology.orchestra.run.smithi186.stdout: "get_or_fail_fail": 0, 2024-05-12T22:20:16.936 INFO:teuthology.orchestra.run.smithi186.stdout: "get_or_fail_success": 0, 2024-05-12T22:20:16.936 INFO:teuthology.orchestra.run.smithi186.stdout: "take": 0, 2024-05-12T22:20:16.936 INFO:teuthology.orchestra.run.smithi186.stdout: "take_sum": 0, 2024-05-12T22:20:16.936 INFO:teuthology.orchestra.run.smithi186.stdout: "put": 0, 2024-05-12T22:20:16.936 INFO:teuthology.orchestra.run.smithi186.stdout: "put_sum": 0, 2024-05-12T22:20:16.936 INFO:teuthology.orchestra.run.smithi186.stdout: "wait": { 2024-05-12T22:20:16.936 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-05-12T22:20:16.937 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-05-12T22:20:16.937 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-05-12T22:20:16.937 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-05-12T22:20:16.937 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:16.937 INFO:teuthology.orchestra.run.smithi186.stdout: "throttle-write_buf_throttle-0x561f44ab26e0": { 2024-05-12T22:20:16.937 INFO:teuthology.orchestra.run.smithi186.stdout: "val": 0, 2024-05-12T22:20:16.937 INFO:teuthology.orchestra.run.smithi186.stdout: "max": 3758096384, 2024-05-12T22:20:16.937 INFO:teuthology.orchestra.run.smithi186.stdout: "get_started": 0, 2024-05-12T22:20:16.938 INFO:teuthology.orchestra.run.smithi186.stdout: "get": 0, 2024-05-12T22:20:16.938 INFO:teuthology.orchestra.run.smithi186.stdout: "get_sum": 0, 2024-05-12T22:20:16.938 INFO:teuthology.orchestra.run.smithi186.stdout: "get_or_fail_fail": 0, 2024-05-12T22:20:16.938 INFO:teuthology.orchestra.run.smithi186.stdout: "get_or_fail_success": 0, 2024-05-12T22:20:16.938 INFO:teuthology.orchestra.run.smithi186.stdout: "take": 0, 2024-05-12T22:20:16.938 INFO:teuthology.orchestra.run.smithi186.stdout: "take_sum": 0, 2024-05-12T22:20:16.938 INFO:teuthology.orchestra.run.smithi186.stdout: "put": 0, 2024-05-12T22:20:16.938 INFO:teuthology.orchestra.run.smithi186.stdout: "put_sum": 0, 2024-05-12T22:20:16.938 INFO:teuthology.orchestra.run.smithi186.stdout: "wait": { 2024-05-12T22:20:16.939 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-05-12T22:20:16.939 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-05-12T22:20:16.939 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-05-12T22:20:16.939 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-05-12T22:20:16.939 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-05-12T22:20:16.939 INFO:teuthology.orchestra.run.smithi186.stdout:} 2024-05-12T22:20:17.606 INFO:tasks.check_counter:Daemon mds.i mds.exported=0 2024-05-12T22:20:17.606 INFO:tasks.check_counter:Daemon mds.i mds.imported=0 2024-05-12T22:20:17.606 INFO:tasks.check_counter:Daemon mds.i mds_cache.dir_update=0 2024-05-12T22:20:17.606 INFO:tasks.check_counter:Daemon mds.i mds_cache.dir_update_receipt=0 2024-05-12T22:20:17.606 INFO:tasks.check_counter:Daemon mds.i mds.root_rsnaps=0 2024-05-12T22:20:17.607 INFO:tasks.check_counter:Daemon mds.i mds_server.req_mksnap_latency.avgcount=0 2024-05-12T22:20:17.607 INFO:tasks.check_counter:Daemon mds.i mds_server.req_rmsnap_latency.avgcount=0 2024-05-12T22:20:17.607 INFO:tasks.check_counter:Daemon mds.i mds.dir_split=0 2024-05-12T22:20:17.607 DEBUG:tasks.check_counter:Getting stats from l 2024-05-12T22:20:17.607 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f72fecff68e1d400c4568684327c900485c20d6a shell --fsid 6b92e7b4-10aa-11ef-bc98-c7b262605968 -- ceph daemon mds.l perf dump 2024-05-12T22:20:17.847 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/mon.c/config 2024-05-12T22:20:18.652 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:20:18 smithi186 ceph-mon[34785]: pgmap v352: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 237 B/s wr, 7 op/s 2024-05-12T22:20:18.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:20:18 smithi086 ceph-mon[34297]: pgmap v352: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 237 B/s wr, 7 op/s 2024-05-12T22:20:18.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:20:18 smithi028 ceph-mon[29847]: pgmap v352: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 237 B/s wr, 7 op/s 2024-05-12T22:20:19.498 INFO:teuthology.orchestra.run.smithi186.stdout:{ 2024-05-12T22:20:19.498 INFO:teuthology.orchestra.run.smithi186.stdout: "AsyncMessenger::Worker-0": { 2024-05-12T22:20:19.498 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_messages": 0, 2024-05-12T22:20:19.498 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_messages": 2, 2024-05-12T22:20:19.498 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_bytes": 0, 2024-05-12T22:20:19.498 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_bytes": 0, 2024-05-12T22:20:19.499 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_created_connections": 4, 2024-05-12T22:20:19.499 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_active_connections": 0, 2024-05-12T22:20:19.499 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_total_time": 0.000983961, 2024-05-12T22:20:19.499 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_send_time": 0.000000000, 2024-05-12T22:20:19.499 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_recv_time": 0.000180223, 2024-05-12T22:20:19.499 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-05-12T22:20:19.499 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_messages_queue_lat": { 2024-05-12T22:20:19.499 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-05-12T22:20:19.500 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-05-12T22:20:19.500 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-05-12T22:20:19.500 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:19.500 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_handle_ack_lat": { 2024-05-12T22:20:19.500 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-05-12T22:20:19.500 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-05-12T22:20:19.500 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-05-12T22:20:19.500 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:19.501 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_encrypted_bytes": 0, 2024-05-12T22:20:19.501 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_encrypted_bytes": 0 2024-05-12T22:20:19.501 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:19.501 INFO:teuthology.orchestra.run.smithi186.stdout: "AsyncMessenger::Worker-1": { 2024-05-12T22:20:19.501 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_messages": 243, 2024-05-12T22:20:19.501 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_messages": 231, 2024-05-12T22:20:19.501 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_bytes": 1724400, 2024-05-12T22:20:19.501 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_bytes": 142624, 2024-05-12T22:20:19.501 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_created_connections": 2, 2024-05-12T22:20:19.502 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_active_connections": 1, 2024-05-12T22:20:19.502 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_total_time": 0.049050497, 2024-05-12T22:20:19.502 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_send_time": 0.014794045, 2024-05-12T22:20:19.502 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_recv_time": 0.038645564, 2024-05-12T22:20:19.502 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_fast_dispatch_time": 0.005898933, 2024-05-12T22:20:19.502 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_messages_queue_lat": { 2024-05-12T22:20:19.502 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 230, 2024-05-12T22:20:19.502 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.013694127, 2024-05-12T22:20:19.503 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000059539 2024-05-12T22:20:19.503 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:19.503 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_handle_ack_lat": { 2024-05-12T22:20:19.503 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-05-12T22:20:19.503 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-05-12T22:20:19.503 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-05-12T22:20:19.503 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:19.503 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_encrypted_bytes": 1724400, 2024-05-12T22:20:19.504 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_encrypted_bytes": 142624 2024-05-12T22:20:19.504 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:19.504 INFO:teuthology.orchestra.run.smithi186.stdout: "AsyncMessenger::Worker-2": { 2024-05-12T22:20:19.504 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_messages": 7, 2024-05-12T22:20:19.504 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_messages": 181, 2024-05-12T22:20:19.504 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_bytes": 3552, 2024-05-12T22:20:19.504 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_bytes": 265440, 2024-05-12T22:20:19.504 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_created_connections": 6, 2024-05-12T22:20:19.504 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_active_connections": 3, 2024-05-12T22:20:19.505 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_total_time": 0.012550575, 2024-05-12T22:20:19.505 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_send_time": 0.008263444, 2024-05-12T22:20:19.505 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_recv_time": 0.002016036, 2024-05-12T22:20:19.505 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-05-12T22:20:19.505 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_messages_queue_lat": { 2024-05-12T22:20:19.505 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 178, 2024-05-12T22:20:19.505 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.020111916, 2024-05-12T22:20:19.505 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000112988 2024-05-12T22:20:19.505 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:19.506 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_handle_ack_lat": { 2024-05-12T22:20:19.506 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-05-12T22:20:19.506 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-05-12T22:20:19.506 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-05-12T22:20:19.506 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:19.506 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_encrypted_bytes": 3552, 2024-05-12T22:20:19.506 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_encrypted_bytes": 265440 2024-05-12T22:20:19.507 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:19.507 INFO:teuthology.orchestra.run.smithi186.stdout: "cct": { 2024-05-12T22:20:19.507 INFO:teuthology.orchestra.run.smithi186.stdout: "total_workers": 0, 2024-05-12T22:20:19.507 INFO:teuthology.orchestra.run.smithi186.stdout: "unhealthy_workers": 0 2024-05-12T22:20:19.507 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:19.507 INFO:teuthology.orchestra.run.smithi186.stdout: "mempool": { 2024-05-12T22:20:19.507 INFO:teuthology.orchestra.run.smithi186.stdout: "bloom_filter_bytes": 0, 2024-05-12T22:20:19.507 INFO:teuthology.orchestra.run.smithi186.stdout: "bloom_filter_items": 0, 2024-05-12T22:20:19.507 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_alloc_bytes": 0, 2024-05-12T22:20:19.508 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_alloc_items": 0, 2024-05-12T22:20:19.508 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_data_bytes": 0, 2024-05-12T22:20:19.508 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_data_items": 0, 2024-05-12T22:20:19.508 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_onode_bytes": 0, 2024-05-12T22:20:19.508 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_onode_items": 0, 2024-05-12T22:20:19.508 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_meta_bytes": 0, 2024-05-12T22:20:19.508 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_meta_items": 0, 2024-05-12T22:20:19.509 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_other_bytes": 0, 2024-05-12T22:20:19.509 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_other_items": 0, 2024-05-12T22:20:19.509 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_buffer_bytes": 0, 2024-05-12T22:20:19.509 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_buffer_items": 0, 2024-05-12T22:20:19.509 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_extent_bytes": 0, 2024-05-12T22:20:19.509 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_extent_items": 0, 2024-05-12T22:20:19.509 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_blob_bytes": 0, 2024-05-12T22:20:19.510 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_blob_items": 0, 2024-05-12T22:20:19.510 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_shared_blob_bytes": 0, 2024-05-12T22:20:19.510 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_shared_blob_items": 0, 2024-05-12T22:20:19.510 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_inline_bl_bytes": 0, 2024-05-12T22:20:19.510 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_inline_bl_items": 0, 2024-05-12T22:20:19.510 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_fsck_bytes": 0, 2024-05-12T22:20:19.511 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_fsck_items": 0, 2024-05-12T22:20:19.511 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_txc_bytes": 0, 2024-05-12T22:20:19.511 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_txc_items": 0, 2024-05-12T22:20:19.511 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_writing_deferred_bytes": 0, 2024-05-12T22:20:19.511 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_writing_deferred_items": 0, 2024-05-12T22:20:19.511 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_writing_bytes": 0, 2024-05-12T22:20:19.512 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_writing_items": 0, 2024-05-12T22:20:19.512 INFO:teuthology.orchestra.run.smithi186.stdout: "bluefs_bytes": 0, 2024-05-12T22:20:19.512 INFO:teuthology.orchestra.run.smithi186.stdout: "bluefs_items": 0, 2024-05-12T22:20:19.512 INFO:teuthology.orchestra.run.smithi186.stdout: "bluefs_file_reader_bytes": 0, 2024-05-12T22:20:19.512 INFO:teuthology.orchestra.run.smithi186.stdout: "bluefs_file_reader_items": 0, 2024-05-12T22:20:19.513 INFO:teuthology.orchestra.run.smithi186.stdout: "bluefs_file_writer_bytes": 0, 2024-05-12T22:20:19.513 INFO:teuthology.orchestra.run.smithi186.stdout: "bluefs_file_writer_items": 0, 2024-05-12T22:20:19.513 INFO:teuthology.orchestra.run.smithi186.stdout: "buffer_anon_bytes": 148692, 2024-05-12T22:20:19.513 INFO:teuthology.orchestra.run.smithi186.stdout: "buffer_anon_items": 30, 2024-05-12T22:20:19.513 INFO:teuthology.orchestra.run.smithi186.stdout: "buffer_meta_bytes": 0, 2024-05-12T22:20:19.513 INFO:teuthology.orchestra.run.smithi186.stdout: "buffer_meta_items": 0, 2024-05-12T22:20:19.513 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_bytes": 0, 2024-05-12T22:20:19.514 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_items": 0, 2024-05-12T22:20:19.514 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_mapbl_bytes": 0, 2024-05-12T22:20:19.514 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_mapbl_items": 0, 2024-05-12T22:20:19.514 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_pglog_bytes": 0, 2024-05-12T22:20:19.514 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_pglog_items": 0, 2024-05-12T22:20:19.514 INFO:teuthology.orchestra.run.smithi186.stdout: "osdmap_bytes": 0, 2024-05-12T22:20:19.515 INFO:teuthology.orchestra.run.smithi186.stdout: "osdmap_items": 0, 2024-05-12T22:20:19.515 INFO:teuthology.orchestra.run.smithi186.stdout: "osdmap_mapping_bytes": 0, 2024-05-12T22:20:19.515 INFO:teuthology.orchestra.run.smithi186.stdout: "osdmap_mapping_items": 0, 2024-05-12T22:20:19.515 INFO:teuthology.orchestra.run.smithi186.stdout: "pgmap_bytes": 0, 2024-05-12T22:20:19.515 INFO:teuthology.orchestra.run.smithi186.stdout: "pgmap_items": 0, 2024-05-12T22:20:19.515 INFO:teuthology.orchestra.run.smithi186.stdout: "mds_co_bytes": 928, 2024-05-12T22:20:19.515 INFO:teuthology.orchestra.run.smithi186.stdout: "mds_co_items": 2, 2024-05-12T22:20:19.515 INFO:teuthology.orchestra.run.smithi186.stdout: "unittest_1_bytes": 0, 2024-05-12T22:20:19.515 INFO:teuthology.orchestra.run.smithi186.stdout: "unittest_1_items": 0, 2024-05-12T22:20:19.516 INFO:teuthology.orchestra.run.smithi186.stdout: "unittest_2_bytes": 0, 2024-05-12T22:20:19.516 INFO:teuthology.orchestra.run.smithi186.stdout: "unittest_2_items": 0 2024-05-12T22:20:19.516 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-05-12T22:20:19.516 INFO:teuthology.orchestra.run.smithi186.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-05-12T22:20:19.516 INFO:teuthology.orchestra.run.smithi186.stdout: "val": 0, 2024-05-12T22:20:19.516 INFO:teuthology.orchestra.run.smithi186.stdout: "max": 104857600, 2024-05-12T22:20:19.516 INFO:teuthology.orchestra.run.smithi186.stdout: "get_started": 0, 2024-05-12T22:20:19.517 INFO:teuthology.orchestra.run.smithi186.stdout: "get": 246, 2024-05-12T22:20:19.517 INFO:teuthology.orchestra.run.smithi186.stdout: "get_sum": 1690176, 2024-05-12T22:20:19.517 INFO:teuthology.orchestra.run.smithi186.stdout: "get_or_fail_fail": 0, 2024-05-12T22:20:19.517 INFO:teuthology.orchestra.run.smithi186.stdout: "get_or_fail_success": 246, 2024-05-12T22:20:19.517 INFO:teuthology.orchestra.run.smithi186.stdout: "take": 0, 2024-05-12T22:20:19.517 INFO:teuthology.orchestra.run.smithi186.stdout: "take_sum": 0, 2024-05-12T22:20:19.517 INFO:teuthology.orchestra.run.smithi186.stdout: "put": 246, 2024-05-12T22:20:19.517 INFO:teuthology.orchestra.run.smithi186.stdout: "put_sum": 1690176, 2024-05-12T22:20:19.518 INFO:teuthology.orchestra.run.smithi186.stdout: "wait": { 2024-05-12T22:20:19.518 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-05-12T22:20:19.518 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-05-12T22:20:19.518 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-05-12T22:20:19.518 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-05-12T22:20:19.518 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-05-12T22:20:19.518 INFO:teuthology.orchestra.run.smithi186.stdout:} 2024-05-12T22:20:20.191 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.l 2024-05-12T22:20:20.191 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.l 2024-05-12T22:20:20.191 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.l 2024-05-12T22:20:20.191 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.l 2024-05-12T22:20:20.191 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.l 2024-05-12T22:20:20.191 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.l 2024-05-12T22:20:20.192 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.l 2024-05-12T22:20:20.192 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.l 2024-05-12T22:20:20.192 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-05-12T22:20:20.203 INFO:tasks.kclient:Unmounting kernel clients... 2024-05-12T22:20:20.203 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-05-12T22:20:20.203 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-12T22:20:20.270 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-05-12T22:20:20.270 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-12T22:20:20.328 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-12T22:20:20.429 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:20:20 smithi186 ceph-mon[34785]: pgmap v353: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2024-05-12T22:20:20.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:20.553+0000 7ffa31600640 1 -- 172.21.15.28:0/2093783794 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffa2c107f80 msgr2=0x7ffa2c0689e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:20:20.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:20.553+0000 7ffa31600640 1 --2- 172.21.15.28:0/2093783794 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffa2c107f80 0x7ffa2c0689e0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7ffa18009c10 tx=0x7ffa1802a780 comp rx=0 tx=0).stop 2024-05-12T22:20:20.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:20.553+0000 7ffa31600640 1 -- 172.21.15.28:0/2093783794 shutdown_connections 2024-05-12T22:20:20.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:20.553+0000 7ffa31600640 1 --2- 172.21.15.28:0/2093783794 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffa2c069110 0x7ffa2c10e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:20:20.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:20.553+0000 7ffa31600640 1 --2- 172.21.15.28:0/2093783794 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffa2c107f80 0x7ffa2c0689e0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:20:20.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:20.553+0000 7ffa31600640 1 --2- 172.21.15.28:0/2093783794 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa2c107660 0x7ffa2c107a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:20:20.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:20.553+0000 7ffa31600640 1 -- 172.21.15.28:0/2093783794 >> 172.21.15.28:0/2093783794 conn(0x7ffa2c076070 msgr2=0x7ffa2c076480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:20:20.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:20.553+0000 7ffa31600640 1 -- 172.21.15.28:0/2093783794 shutdown_connections 2024-05-12T22:20:20.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:20.553+0000 7ffa31600640 1 -- 172.21.15.28:0/2093783794 wait complete. 2024-05-12T22:20:20.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:20.555+0000 7ffa31600640 1 Processor -- start 2024-05-12T22:20:20.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:20.555+0000 7ffa31600640 1 -- start start 2024-05-12T22:20:20.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:20.555+0000 7ffa31600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa2c069110 0x7ffa2c19a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:20:20.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:20.555+0000 7ffa31600640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffa2c107660 0x7ffa2c19a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:20:20.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:20.556+0000 7ffa31600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffa2c107f80 0x7ffa2c19eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:20:20.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:20.556+0000 7ffa31600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ffa2c1119f0 con 0x7ffa2c069110 2024-05-12T22:20:20.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:20.556+0000 7ffa31600640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7ffa2c111870 con 0x7ffa2c107f80 2024-05-12T22:20:20.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:20.556+0000 7ffa31600640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ffa2c111b70 con 0x7ffa2c107660 2024-05-12T22:20:20.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:20.556+0000 7ffa2be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa2c069110 0x7ffa2c19a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:20:20.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:20.556+0000 7ffa2be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa2c069110 0x7ffa2c19a220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60118/0 (socket says 172.21.15.28:60118) 2024-05-12T22:20:20.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:20.556+0000 7ffa2be00640 1 -- 172.21.15.28:0/3641119448 learned_addr learned my addr 172.21.15.28:0/3641119448 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:20:20.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:20.556+0000 7ffa30c00640 1 --2- 172.21.15.28:0/3641119448 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffa2c107f80 0x7ffa2c19eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:20:20.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:20.556+0000 7ffa2b400640 1 --2- 172.21.15.28:0/3641119448 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffa2c107660 0x7ffa2c19a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:20:20.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:20.556+0000 7ffa2be00640 1 -- 172.21.15.28:0/3641119448 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffa2c107660 msgr2=0x7ffa2c19a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:20:20.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:20.556+0000 7ffa2be00640 1 --2- 172.21.15.28:0/3641119448 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffa2c107660 0x7ffa2c19a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:20:20.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:20.556+0000 7ffa2be00640 1 -- 172.21.15.28:0/3641119448 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffa2c107f80 msgr2=0x7ffa2c19eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:20:20.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:20.556+0000 7ffa2be00640 1 --2- 172.21.15.28:0/3641119448 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffa2c107f80 0x7ffa2c19eae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:20:20.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:20.556+0000 7ffa2be00640 1 -- 172.21.15.28:0/3641119448 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa2c19f210 con 0x7ffa2c069110 2024-05-12T22:20:20.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:20.556+0000 7ffa30c00640 1 --2- 172.21.15.28:0/3641119448 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffa2c107f80 0x7ffa2c19eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:20:20.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:20.557+0000 7ffa2be00640 1 --2- 172.21.15.28:0/3641119448 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa2c069110 0x7ffa2c19a220 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7ffa1c00e9c0 tx=0x7ffa1c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:20:20.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:20.557+0000 7ffa28c00640 1 -- 172.21.15.28:0/3641119448 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa1c00ccc0 con 0x7ffa2c069110 2024-05-12T22:20:20.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:20.557+0000 7ffa28c00640 1 -- 172.21.15.28:0/3641119448 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa1c005500 con 0x7ffa2c069110 2024-05-12T22:20:20.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:20.557+0000 7ffa28c00640 1 -- 172.21.15.28:0/3641119448 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa1c010430 con 0x7ffa2c069110 2024-05-12T22:20:20.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:20.557+0000 7ffa31600640 1 -- 172.21.15.28:0/3641119448 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa2c19f500 con 0x7ffa2c069110 2024-05-12T22:20:20.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:20.557+0000 7ffa31600640 1 -- 172.21.15.28:0/3641119448 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa2c1a6d90 con 0x7ffa2c069110 2024-05-12T22:20:20.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:20.558+0000 7ffa31600640 1 -- 172.21.15.28:0/3641119448 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa2c068570 con 0x7ffa2c069110 2024-05-12T22:20:20.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:20.561+0000 7ffa28c00640 1 -- 172.21.15.28:0/3641119448 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7ffa1c0046c0 con 0x7ffa2c069110 2024-05-12T22:20:20.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:20.562+0000 7ffa28c00640 1 --2- 172.21.15.28:0/3641119448 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7ffa100a2d00 0x7ffa100a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:20:20.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:20.562+0000 7ffa28c00640 1 -- 172.21.15.28:0/3641119448 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8513+0+0 (secure 0 0 0) 0x7ffa1c0cf6f0 con 0x7ffa2c069110 2024-05-12T22:20:20.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:20.562+0000 7ffa2b400640 1 --2- 172.21.15.28:0/3641119448 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7ffa100a2d00 0x7ffa100a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:20:20.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:20.563+0000 7ffa2b400640 1 --2- 172.21.15.28:0/3641119448 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7ffa100a2d00 0x7ffa100a51c0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7ffa1802aca0 tx=0x7ffa18005fd0 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:20:20.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:20.567+0000 7ffa28c00640 1 -- 172.21.15.28:0/3641119448 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7ffa1c09c2b0 con 0x7ffa2c069110 2024-05-12T22:20:20.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:20:20 smithi086 ceph-mon[34297]: pgmap v353: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2024-05-12T22:20:20.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:20.779+0000 7ffa31600640 1 -- 172.21.15.28:0/3641119448 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ffa2c19b570 con 0x7ffa2c069110 2024-05-12T22:20:20.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:20.782+0000 7ffa28c00640 1 -- 172.21.15.28:0/3641119448 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v99) ==== 74+0+23062 (secure 0 0 0) 0x7ffa1c09bbb0 con 0x7ffa2c069110 2024-05-12T22:20:20.783 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-12T22:20:20.783 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":99,"fsid":"6b92e7b4-10aa-11ef-bc98-c7b262605968","created":"2024-05-12T21:57:37.276782+0000","modified":"2024-05-12T22:20:13.997146+0000","last_up_change":"2024-05-12T22:05:37.929562+0000","last_in_change":"2024-05-12T22:05:18.750768+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-12T22:01:32.292826+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-12T22:06:31.540263+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-12T22:06:32.906118+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"98","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":17,"snap_epoch":98,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6992ce0a-844b-40e3-8ed1-181c8474dccf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":906003882},{"type":"v1","addr":"172.21.15.28:6803","nonce":906003882}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":906003882},{"type":"v1","addr":"172.21.15.28:6805","nonce":906003882}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":906003882},{"type":"v1","addr":"172.21.15.28:6809","nonce":906003882}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":906003882},{"type":"v1","addr":"172.21.15.28:6807","nonce":906003882}]},"public_addr":"172.21.15.28:6803/906003882","cluster_addr":"172.21.15.28:6805/906003882","heartbeat_back_addr":"172.21.15.28:6809/906003882","heartbeat_front_addr":"172.21.15.28:6807/906003882","state":["exists","up"]},{"osd":1,"uuid":"ced20c96-c481-4d48-90d1-6b43228959ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":3354141819},{"type":"v1","addr":"172.21.15.86:6801","nonce":3354141819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":3354141819},{"type":"v1","addr":"172.21.15.86:6803","nonce":3354141819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":3354141819},{"type":"v1","addr":"172.21.15.86:6807","nonce":3354141819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":3354141819},{"type":"v1","addr":"172.21.15.86:6805","nonce":3354141819}]},"public_addr":"172.21.15.86:6801/3354141819","cluster_addr":"172.21.15.86:6803/3354141819","heartbeat_back_addr":"172.21.15.86:6807/3354141819","heartbeat_front_addr":"172.21.15.86:6805/3354141819","state":["exists","up"]},{"osd":2,"uuid":"54a2f221-fa65-4795-9f5e-e660cc7c0098","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":906459365},{"type":"v1","addr":"172.21.15.186:6801","nonce":906459365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":906459365},{"type":"v1","addr":"172.21.15.186:6803","nonce":906459365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":906459365},{"type":"v1","addr":"172.21.15.186:6807","nonce":906459365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":906459365},{"type":"v1","addr":"172.21.15.186:6805","nonce":906459365}]},"public_addr":"172.21.15.186:6801/906459365","cluster_addr":"172.21.15.186:6803/906459365","heartbeat_back_addr":"172.21.15.186:6807/906459365","heartbeat_front_addr":"172.21.15.186:6805/906459365","state":["exists","up"]},{"osd":3,"uuid":"86836737-ac9a-487b-93f2-7fcd3c406960","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":4182884436},{"type":"v1","addr":"172.21.15.28:6811","nonce":4182884436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":4182884436},{"type":"v1","addr":"172.21.15.28:6813","nonce":4182884436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":4182884436},{"type":"v1","addr":"172.21.15.28:6817","nonce":4182884436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":4182884436},{"type":"v1","addr":"172.21.15.28:6815","nonce":4182884436}]},"public_addr":"172.21.15.28:6811/4182884436","cluster_addr":"172.21.15.28:6813/4182884436","heartbeat_back_addr":"172.21.15.28:6817/4182884436","heartbeat_front_addr":"172.21.15.28:6815/4182884436","state":["exists","up"]},{"osd":4,"uuid":"d162ebcd-83c0-4fd5-9925-28b2cecea935","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2596998708},{"type":"v1","addr":"172.21.15.86:6809","nonce":2596998708}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2596998708},{"type":"v1","addr":"172.21.15.86:6811","nonce":2596998708}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2596998708},{"type":"v1","addr":"172.21.15.86:6815","nonce":2596998708}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2596998708},{"type":"v1","addr":"172.21.15.86:6813","nonce":2596998708}]},"public_addr":"172.21.15.86:6809/2596998708","cluster_addr":"172.21.15.86:6811/2596998708","heartbeat_back_addr":"172.21.15.86:6815/2596998708","heartbeat_front_addr":"172.21.15.86:6813/2596998708","state":["exists","up"]},{"osd":5,"uuid":"c794336a-a105-425c-b6c7-a32b90c05bb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":1944205756},{"type":"v1","addr":"172.21.15.186:6809","nonce":1944205756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":1944205756},{"type":"v1","addr":"172.21.15.186:6811","nonce":1944205756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":1944205756},{"type":"v1","addr":"172.21.15.186:6815","nonce":1944205756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":1944205756},{"type":"v1","addr":"172.21.15.186:6813","nonce":1944205756}]},"public_addr":"172.21.15.186:6809/1944205756","cluster_addr":"172.21.15.186:6811/1944205756","heartbeat_back_addr":"172.21.15.186:6815/1944205756","heartbeat_front_addr":"172.21.15.186:6813/1944205756","state":["exists","up"]},{"osd":6,"uuid":"516932ac-af00-45b7-ad3c-d6dec7db442f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":1642308925},{"type":"v1","addr":"172.21.15.28:6819","nonce":1642308925}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":1642308925},{"type":"v1","addr":"172.21.15.28:6821","nonce":1642308925}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1642308925},{"type":"v1","addr":"172.21.15.28:6825","nonce":1642308925}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":1642308925},{"type":"v1","addr":"172.21.15.28:6823","nonce":1642308925}]},"public_addr":"172.21.15.28:6819/1642308925","cluster_addr":"172.21.15.28:6821/1642308925","heartbeat_back_addr":"172.21.15.28:6825/1642308925","heartbeat_front_addr":"172.21.15.28:6823/1642308925","state":["exists","up"]},{"osd":7,"uuid":"ca7a059f-0414-438e-b2d1-a10f348036be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":726062711},{"type":"v1","addr":"172.21.15.86:6817","nonce":726062711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":726062711},{"type":"v1","addr":"172.21.15.86:6819","nonce":726062711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":726062711},{"type":"v1","addr":"172.21.15.86:6823","nonce":726062711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":726062711},{"type":"v1","addr":"172.21.15.86:6821","nonce":726062711}]},"public_addr":"172.21.15.86:6817/726062711","cluster_addr":"172.21.15.86:6819/726062711","heartbeat_back_addr":"172.21.15.86:6823/726062711","heartbeat_front_addr":"172.21.15.86:6821/726062711","state":["exists","up"]},{"osd":8,"uuid":"92e3f41f-aebd-422e-a99e-555f7ba4aa53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":3007687277},{"type":"v1","addr":"172.21.15.186:6817","nonce":3007687277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":3007687277},{"type":"v1","addr":"172.21.15.186:6819","nonce":3007687277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":3007687277},{"type":"v1","addr":"172.21.15.186:6823","nonce":3007687277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":3007687277},{"type":"v1","addr":"172.21.15.186:6821","nonce":3007687277}]},"public_addr":"172.21.15.186:6817/3007687277","cluster_addr":"172.21.15.186:6819/3007687277","heartbeat_back_addr":"172.21.15.186:6823/3007687277","heartbeat_front_addr":"172.21.15.186:6821/3007687277","state":["exists","up"]},{"osd":9,"uuid":"38ff7d9d-ee85-404c-8c59-adc327e1ccfa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":136220472},{"type":"v1","addr":"172.21.15.28:6827","nonce":136220472}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":136220472},{"type":"v1","addr":"172.21.15.28:6829","nonce":136220472}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":136220472},{"type":"v1","addr":"172.21.15.28:6833","nonce":136220472}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":136220472},{"type":"v1","addr":"172.21.15.28:6831","nonce":136220472}]},"public_addr":"172.21.15.28:6827/136220472","cluster_addr":"172.21.15.28:6829/136220472","heartbeat_back_addr":"172.21.15.28:6833/136220472","heartbeat_front_addr":"172.21.15.28:6831/136220472","state":["exists","up"]},{"osd":10,"uuid":"5e9ff657-afa8-4bab-8491-f209038392b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1189968310},{"type":"v1","addr":"172.21.15.86:6825","nonce":1189968310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":1189968310},{"type":"v1","addr":"172.21.15.86:6827","nonce":1189968310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":1189968310},{"type":"v1","addr":"172.21.15.86:6831","nonce":1189968310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":1189968310},{"type":"v1","addr":"172.21.15.86:6829","nonce":1189968310}]},"public_addr":"172.21.15.86:6825/1189968310","cluster_addr":"172.21.15.86:6827/1189968310","heartbeat_back_addr":"172.21.15.86:6831/1189968310","heartbeat_front_addr":"172.21.15.86:6829/1189968310","state":["exists","up"]},{"osd":11,"uuid":"f112e3c6-65b6-4253-a966-62001db7d069","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3801668110},{"type":"v1","addr":"172.21.15.186:6825","nonce":3801668110}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3801668110},{"type":"v1","addr":"172.21.15.186:6827","nonce":3801668110}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3801668110},{"type":"v1","addr":"172.21.15.186:6831","nonce":3801668110}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3801668110},{"type":"v1","addr":"172.21.15.186:6829","nonce":3801668110}]},"public_addr":"172.21.15.186:6825/3801668110","cluster_addr":"172.21.15.186:6827/3801668110","heartbeat_back_addr":"172.21.15.186:6831/3801668110","heartbeat_front_addr":"172.21.15.186:6829/3801668110","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:00:40.483623+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:01:05.284385+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:01:28.645716+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:02:01.295092+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:02:26.486601+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:02:50.807478+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:03:24.756169+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:03:49.975173+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:04:13.772233+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:04:46.758555+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:05:12.331685+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:05:36.462548+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":2,"to":1}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.25","mappings":[{"from":5,"to":9}]},{"pgid":"3.27","mappings":[{"from":4,"to":8}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/2580087728":"2024-05-13T22:08:58.638536+0000","172.21.15.28:6801/3123328651":"2024-05-13T22:08:58.638536+0000","172.21.15.28:0/820158394":"2024-05-13T22:08:58.638536+0000","172.21.15.28:0/1978033727":"2024-05-13T22:08:58.638536+0000","172.21.15.28:0/1177009533":"2024-05-13T21:58:45.550459+0000","172.21.15.28:6800/3760499540":"2024-05-13T21:58:45.550459+0000","172.21.15.28:6801/3783742728":"2024-05-13T21:58:10.421564+0000","172.21.15.28:6800/3783742728":"2024-05-13T21:58:10.421564+0000","172.21.15.28:0/2229156260":"2024-05-13T21:58:45.550459+0000","172.21.15.28:0/1202140963":"2024-05-13T21:58:10.421564+0000","172.21.15.28:6800/3123328651":"2024-05-13T22:08:58.638536+0000","172.21.15.28:0/2924223784":"2024-05-13T21:58:45.550459+0000","172.21.15.28:6801/3760499540":"2024-05-13T21:58:45.550459+0000","172.21.15.28:0/1240027206":"2024-05-13T21:58:10.421564+0000","172.21.15.28:0/1581755795":"2024-05-13T22:08:58.638536+0000","172.21.15.28:0/171417728":"2024-05-13T21:58:10.421564+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-12T22:20:20.784 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:20:20 smithi028 ceph-mon[29847]: pgmap v353: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2024-05-12T22:20:20.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:20.785+0000 7ffa31600640 1 -- 172.21.15.28:0/3641119448 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7ffa100a2d00 msgr2=0x7ffa100a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:20:20.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:20.785+0000 7ffa31600640 1 --2- 172.21.15.28:0/3641119448 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7ffa100a2d00 0x7ffa100a51c0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7ffa1802aca0 tx=0x7ffa18005fd0 comp rx=0 tx=0).stop 2024-05-12T22:20:20.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:20.785+0000 7ffa31600640 1 -- 172.21.15.28:0/3641119448 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa2c069110 msgr2=0x7ffa2c19a220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:20:20.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:20.785+0000 7ffa31600640 1 --2- 172.21.15.28:0/3641119448 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa2c069110 0x7ffa2c19a220 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7ffa1c00e9c0 tx=0x7ffa1c00ee80 comp rx=0 tx=0).stop 2024-05-12T22:20:20.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:20.785+0000 7ffa31600640 1 -- 172.21.15.28:0/3641119448 shutdown_connections 2024-05-12T22:20:20.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:20.785+0000 7ffa31600640 1 --2- 172.21.15.28:0/3641119448 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7ffa100a2d00 0x7ffa100a51c0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:20:20.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:20.785+0000 7ffa31600640 1 --2- 172.21.15.28:0/3641119448 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffa2c107f80 0x7ffa2c19eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:20:20.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:20.785+0000 7ffa31600640 1 --2- 172.21.15.28:0/3641119448 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffa2c107660 0x7ffa2c19a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:20:20.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:20.785+0000 7ffa31600640 1 --2- 172.21.15.28:0/3641119448 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa2c069110 0x7ffa2c19a220 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:20:20.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:20.785+0000 7ffa31600640 1 -- 172.21.15.28:0/3641119448 >> 172.21.15.28:0/3641119448 conn(0x7ffa2c076070 msgr2=0x7ffa2c10c820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:20:20.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:20.786+0000 7ffa31600640 1 -- 172.21.15.28:0/3641119448 shutdown_connections 2024-05-12T22:20:20.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:20.786+0000 7ffa31600640 1 -- 172.21.15.28:0/3641119448 wait complete. 2024-05-12T22:20:20.805 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-12T22:01:32.292826+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-12T22:06:31.540263+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-12T22:06:32.906118+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '98', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 17, 'snap_epoch': 98, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-12T22:20:20.806 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-12T22:20:21.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:21.023+0000 7f2f2c200640 1 -- 172.21.15.28:0/2016536196 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2f24075d60 msgr2=0x7f2f24076140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:20:21.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:21.023+0000 7f2f2c200640 1 --2- 172.21.15.28:0/2016536196 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2f24075d60 0x7f2f24076140 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f2f18009890 tx=0x7f2f1802a360 comp rx=0 tx=0).stop 2024-05-12T22:20:21.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:21.023+0000 7f2f2c200640 1 -- 172.21.15.28:0/2016536196 shutdown_connections 2024-05-12T22:20:21.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:21.023+0000 7f2f2c200640 1 --2- 172.21.15.28:0/2016536196 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f24112710 0x7f2f24114b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:20:21.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:21.023+0000 7f2f2c200640 1 --2- 172.21.15.28:0/2016536196 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2f24076680 0x7f2f241121d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:20:21.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:21.023+0000 7f2f2c200640 1 --2- 172.21.15.28:0/2016536196 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2f24075d60 0x7f2f24076140 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:20:21.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:21.023+0000 7f2f2c200640 1 -- 172.21.15.28:0/2016536196 >> 172.21.15.28:0/2016536196 conn(0x7f2f240ff5c0 msgr2=0x7f2f241019e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:20:21.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:21.023+0000 7f2f2c200640 1 -- 172.21.15.28:0/2016536196 shutdown_connections 2024-05-12T22:20:21.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:21.023+0000 7f2f2c200640 1 -- 172.21.15.28:0/2016536196 wait complete. 2024-05-12T22:20:21.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:21.024+0000 7f2f2c200640 1 Processor -- start 2024-05-12T22:20:21.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:21.024+0000 7f2f2c200640 1 -- start start 2024-05-12T22:20:21.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:21.025+0000 7f2f2c200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f24075d60 0x7f2f241a4460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:20:21.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:21.025+0000 7f2f2c200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2f24076680 0x7f2f241a49a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:20:21.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:21.025+0000 7f2f2c200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2f24112710 0x7f2f2419e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:20:21.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:21.025+0000 7f2f2c200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2f24118260 con 0x7f2f24075d60 2024-05-12T22:20:21.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:21.025+0000 7f2f2c200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f2f241180e0 con 0x7f2f24112710 2024-05-12T22:20:21.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:21.025+0000 7f2f2c200640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f2f241183e0 con 0x7f2f24076680 2024-05-12T22:20:21.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:21.025+0000 7f2f29800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f24075d60 0x7f2f241a4460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:20:21.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:21.025+0000 7f2f28e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2f24076680 0x7f2f241a49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:20:21.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:21.025+0000 7f2f29800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f24075d60 0x7f2f241a4460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60128/0 (socket says 172.21.15.28:60128) 2024-05-12T22:20:21.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:21.025+0000 7f2f29800640 1 -- 172.21.15.28:0/3784825778 learned_addr learned my addr 172.21.15.28:0/3784825778 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:20:21.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:21.025+0000 7f2f2a200640 1 --2- 172.21.15.28:0/3784825778 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2f24112710 0x7f2f2419e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:20:21.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:21.026+0000 7f2f29800640 1 -- 172.21.15.28:0/3784825778 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2f24076680 msgr2=0x7f2f241a49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:20:21.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:21.026+0000 7f2f29800640 1 --2- 172.21.15.28:0/3784825778 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2f24076680 0x7f2f241a49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:20:21.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:21.026+0000 7f2f29800640 1 -- 172.21.15.28:0/3784825778 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2f24112710 msgr2=0x7f2f2419e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:20:21.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:21.026+0000 7f2f29800640 1 --2- 172.21.15.28:0/3784825778 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2f24112710 0x7f2f2419e660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:20:21.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:21.026+0000 7f2f29800640 1 -- 172.21.15.28:0/3784825778 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f2419ede0 con 0x7f2f24075d60 2024-05-12T22:20:21.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:21.026+0000 7f2f28e00640 1 --2- 172.21.15.28:0/3784825778 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2f24076680 0x7f2f241a49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:20:21.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:21.026+0000 7f2f29800640 1 --2- 172.21.15.28:0/3784825778 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f24075d60 0x7f2f241a4460 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f2f18009860 tx=0x7f2f18033d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:20:21.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:21.026+0000 7f2f22000640 1 -- 172.21.15.28:0/3784825778 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f1803f020 con 0x7f2f24075d60 2024-05-12T22:20:21.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:21.026+0000 7f2f2c200640 1 -- 172.21.15.28:0/3784825778 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f2419f070 con 0x7f2f24075d60 2024-05-12T22:20:21.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:21.026+0000 7f2f22000640 1 -- 172.21.15.28:0/3784825778 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2f18032770 con 0x7f2f24075d60 2024-05-12T22:20:21.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:21.026+0000 7f2f22000640 1 -- 172.21.15.28:0/3784825778 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f18037040 con 0x7f2f24075d60 2024-05-12T22:20:21.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:21.027+0000 7f2f2c200640 1 -- 172.21.15.28:0/3784825778 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f241ab320 con 0x7f2f24075d60 2024-05-12T22:20:21.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:21.028+0000 7f2f2c200640 1 -- 172.21.15.28:0/3784825778 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f24068aa0 con 0x7f2f24075d60 2024-05-12T22:20:21.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:21.030+0000 7f2f22000640 1 -- 172.21.15.28:0/3784825778 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7f2f18042020 con 0x7f2f24075d60 2024-05-12T22:20:21.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:21.031+0000 7f2f22000640 1 --2- 172.21.15.28:0/3784825778 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f2f0c0a2d00 0x7f2f0c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:20:21.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:21.032+0000 7f2f22000640 1 -- 172.21.15.28:0/3784825778 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8513+0+0 (secure 0 0 0) 0x7f2f180ef380 con 0x7f2f24075d60 2024-05-12T22:20:21.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:21.032+0000 7f2f28e00640 1 --2- 172.21.15.28:0/3784825778 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f2f0c0a2d00 0x7f2f0c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:20:21.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:21.033+0000 7f2f28e00640 1 --2- 172.21.15.28:0/3784825778 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f2f0c0a2d00 0x7f2f0c0a51c0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f2f24101a90 tx=0x7f2f10005f90 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:20:21.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:21.036+0000 7f2f22000640 1 -- 172.21.15.28:0/3784825778 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f2f18049020 con 0x7f2f24075d60 2024-05-12T22:20:21.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:21.246+0000 7f2f2c200640 1 -- 172.21.15.28:0/3784825778 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f2f2410d8a0 con 0x7f2f24075d60 2024-05-12T22:20:21.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:21.247+0000 7f2f22000640 1 -- 172.21.15.28:0/3784825778 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v99) ==== 93+0+10 (secure 0 0 0) 0x7f2f180bc050 con 0x7f2f24075d60 2024-05-12T22:20:21.612 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-05-12T22:20:21.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:21.250+0000 7f2f2c200640 1 -- 172.21.15.28:0/3784825778 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f2f0c0a2d00 msgr2=0x7f2f0c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:20:21.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:21.250+0000 7f2f2c200640 1 --2- 172.21.15.28:0/3784825778 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f2f0c0a2d00 0x7f2f0c0a51c0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f2f24101a90 tx=0x7f2f10005f90 comp rx=0 tx=0).stop 2024-05-12T22:20:21.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:21.250+0000 7f2f2c200640 1 -- 172.21.15.28:0/3784825778 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f24075d60 msgr2=0x7f2f241a4460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:20:21.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:21.250+0000 7f2f2c200640 1 --2- 172.21.15.28:0/3784825778 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f24075d60 0x7f2f241a4460 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f2f18009860 tx=0x7f2f18033d10 comp rx=0 tx=0).stop 2024-05-12T22:20:21.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:21.250+0000 7f2f2c200640 1 -- 172.21.15.28:0/3784825778 shutdown_connections 2024-05-12T22:20:21.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:21.250+0000 7f2f2c200640 1 --2- 172.21.15.28:0/3784825778 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f2f0c0a2d00 0x7f2f0c0a51c0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:20:21.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:21.250+0000 7f2f2c200640 1 --2- 172.21.15.28:0/3784825778 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2f24112710 0x7f2f2419e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:20:21.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:21.250+0000 7f2f2c200640 1 --2- 172.21.15.28:0/3784825778 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2f24076680 0x7f2f241a49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:20:21.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:21.250+0000 7f2f2c200640 1 --2- 172.21.15.28:0/3784825778 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f24075d60 0x7f2f241a4460 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:20:21.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:21.250+0000 7f2f2c200640 1 -- 172.21.15.28:0/3784825778 >> 172.21.15.28:0/3784825778 conn(0x7f2f240ff5c0 msgr2=0x7f2f24113dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:20:21.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:21.250+0000 7f2f2c200640 1 -- 172.21.15.28:0/3784825778 shutdown_connections 2024-05-12T22:20:21.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:21.250+0000 7f2f2c200640 1 -- 172.21.15.28:0/3784825778 wait complete. 2024-05-12T22:20:21.619 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-12T22:20:21.619 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:20:21 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/3641119448' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-12T22:20:21.619 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:20:21 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/3784825778' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-12T22:20:21.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:20:21 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/3641119448' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-12T22:20:21.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:20:21 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/3784825778' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-12T22:20:21.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:20:21 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/3641119448' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-12T22:20:21.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:20:21 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/3784825778' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-12T22:20:21.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:21.828+0000 7f8207e00640 1 -- 172.21.15.28:0/3035753 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82081016d0 msgr2=0x7f8208101b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:20:21.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:21.828+0000 7f8207e00640 1 --2- 172.21.15.28:0/3035753 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82081016d0 0x7f8208101b50 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f81f4009930 tx=0x7f81f402a440 comp rx=0 tx=0).stop 2024-05-12T22:20:21.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:21.828+0000 7f8207e00640 1 -- 172.21.15.28:0/3035753 shutdown_connections 2024-05-12T22:20:21.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:21.828+0000 7f8207e00640 1 --2- 172.21.15.28:0/3035753 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8208102090 0x7f820810e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:20:21.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:21.828+0000 7f8207e00640 1 --2- 172.21.15.28:0/3035753 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82081016d0 0x7f8208101b50 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:20:21.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:21.828+0000 7f8207e00640 1 --2- 172.21.15.28:0/3035753 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f82081076e0 0x7f8208107ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:20:21.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:21.828+0000 7f8207e00640 1 -- 172.21.15.28:0/3035753 >> 172.21.15.28:0/3035753 conn(0x7f82080fd200 msgr2=0x7f82080ff620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:20:21.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:21.828+0000 7f8207e00640 1 -- 172.21.15.28:0/3035753 shutdown_connections 2024-05-12T22:20:21.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:21.828+0000 7f8207e00640 1 -- 172.21.15.28:0/3035753 wait complete. 2024-05-12T22:20:21.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:21.829+0000 7f8207e00640 1 Processor -- start 2024-05-12T22:20:21.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:21.829+0000 7f8207e00640 1 -- start start 2024-05-12T22:20:21.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:21.830+0000 7f8207e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82081016d0 0x7f820819a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:20:21.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:21.830+0000 7f8207e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8208102090 0x7f820819a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:20:21.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:21.830+0000 7f8207e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f82081076e0 0x7f820819eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:20:21.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:21.830+0000 7f8207e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8208111b00 con 0x7f82081016d0 2024-05-12T22:20:21.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:21.830+0000 7f8207e00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f8208111980 con 0x7f8208102090 2024-05-12T22:20:21.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:21.830+0000 7f8207e00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f8208111c80 con 0x7f82081076e0 2024-05-12T22:20:21.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:21.830+0000 7f8206a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82081016d0 0x7f820819a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:20:21.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:21.830+0000 7f8206a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82081016d0 0x7f820819a280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60148/0 (socket says 172.21.15.28:60148) 2024-05-12T22:20:21.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:21.830+0000 7f8206a00640 1 -- 172.21.15.28:0/694113185 learned_addr learned my addr 172.21.15.28:0/694113185 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:20:21.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:21.830+0000 7f8207400640 1 --2- 172.21.15.28:0/694113185 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f82081076e0 0x7f820819eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:20:21.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:21.830+0000 7f8206000640 1 --2- 172.21.15.28:0/694113185 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8208102090 0x7f820819a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:20:21.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:21.831+0000 7f8206a00640 1 -- 172.21.15.28:0/694113185 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f82081076e0 msgr2=0x7f820819eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:20:21.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:21.831+0000 7f8206a00640 1 --2- 172.21.15.28:0/694113185 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f82081076e0 0x7f820819eaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:20:21.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:21.831+0000 7f8206a00640 1 -- 172.21.15.28:0/694113185 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8208102090 msgr2=0x7f820819a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:20:21.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:21.831+0000 7f8206a00640 1 --2- 172.21.15.28:0/694113185 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8208102090 0x7f820819a7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:20:21.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:21.831+0000 7f8206a00640 1 -- 172.21.15.28:0/694113185 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f820819f220 con 0x7f82081016d0 2024-05-12T22:20:21.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:21.831+0000 7f8206000640 1 --2- 172.21.15.28:0/694113185 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8208102090 0x7f820819a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:20:21.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:21.831+0000 7f8206a00640 1 --2- 172.21.15.28:0/694113185 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82081016d0 0x7f820819a280 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f81f800ea60 tx=0x7f81f800ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:20:21.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:21.831+0000 7f81ff400640 1 -- 172.21.15.28:0/694113185 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81f800cde0 con 0x7f82081016d0 2024-05-12T22:20:21.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:21.831+0000 7f81ff400640 1 -- 172.21.15.28:0/694113185 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f81f80035f0 con 0x7f82081016d0 2024-05-12T22:20:21.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:21.831+0000 7f8207e00640 1 -- 172.21.15.28:0/694113185 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f820819f510 con 0x7f82081016d0 2024-05-12T22:20:21.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:21.831+0000 7f81ff400640 1 -- 172.21.15.28:0/694113185 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81f8010670 con 0x7f82081016d0 2024-05-12T22:20:21.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:21.831+0000 7f8207e00640 1 -- 172.21.15.28:0/694113185 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82081a6da0 con 0x7f82081016d0 2024-05-12T22:20:21.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:21.833+0000 7f8207e00640 1 -- 172.21.15.28:0/694113185 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8208068570 con 0x7f82081016d0 2024-05-12T22:20:21.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:21.835+0000 7f81ff400640 1 -- 172.21.15.28:0/694113185 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7f81f80037a0 con 0x7f82081016d0 2024-05-12T22:20:21.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:21.837+0000 7f81ff400640 1 --2- 172.21.15.28:0/694113185 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f81ec0a2d00 0x7f81ec0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:20:21.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:21.837+0000 7f81ff400640 1 -- 172.21.15.28:0/694113185 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8513+0+0 (secure 0 0 0) 0x7f81f80cf720 con 0x7f82081016d0 2024-05-12T22:20:21.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:21.837+0000 7f8206000640 1 --2- 172.21.15.28:0/694113185 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f81ec0a2d00 0x7f81ec0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:20:21.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:21.838+0000 7f8206000640 1 --2- 172.21.15.28:0/694113185 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f81ec0a2d00 0x7f81ec0a51c0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f81f4006010 tx=0x7f81f4031510 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:20:21.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:21.841+0000 7f81ff400640 1 -- 172.21.15.28:0/694113185 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f81f809c2e0 con 0x7f82081016d0 2024-05-12T22:20:22.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:22.059+0000 7f8207e00640 1 -- 172.21.15.28:0/694113185 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f82081a6f70 con 0x7f82081016d0 2024-05-12T22:20:22.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:22.060+0000 7f81ff400640 1 -- 172.21.15.28:0/694113185 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v99) ==== 104+0+11 (secure 0 0 0) 0x7f81f809bbe0 con 0x7f82081016d0 2024-05-12T22:20:22.061 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-05-12T22:20:22.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:22.063+0000 7f8207e00640 1 -- 172.21.15.28:0/694113185 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f81ec0a2d00 msgr2=0x7f81ec0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:20:22.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:22.063+0000 7f8207e00640 1 --2- 172.21.15.28:0/694113185 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f81ec0a2d00 0x7f81ec0a51c0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f81f4006010 tx=0x7f81f4031510 comp rx=0 tx=0).stop 2024-05-12T22:20:22.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:22.063+0000 7f8207e00640 1 -- 172.21.15.28:0/694113185 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82081016d0 msgr2=0x7f820819a280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:20:22.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:22.063+0000 7f8207e00640 1 --2- 172.21.15.28:0/694113185 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82081016d0 0x7f820819a280 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f81f800ea60 tx=0x7f81f800ef20 comp rx=0 tx=0).stop 2024-05-12T22:20:22.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:22.063+0000 7f8207e00640 1 -- 172.21.15.28:0/694113185 shutdown_connections 2024-05-12T22:20:22.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:22.063+0000 7f8207e00640 1 --2- 172.21.15.28:0/694113185 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f81ec0a2d00 0x7f81ec0a51c0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:20:22.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:22.063+0000 7f8207e00640 1 --2- 172.21.15.28:0/694113185 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f82081076e0 0x7f820819eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:20:22.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:22.063+0000 7f8207e00640 1 --2- 172.21.15.28:0/694113185 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8208102090 0x7f820819a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:20:22.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:22.063+0000 7f8207e00640 1 --2- 172.21.15.28:0/694113185 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82081016d0 0x7f820819a280 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:20:22.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:22.063+0000 7f8207e00640 1 -- 172.21.15.28:0/694113185 >> 172.21.15.28:0/694113185 conn(0x7f82080fd200 msgr2=0x7f82080fd7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:20:22.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:22.063+0000 7f8207e00640 1 -- 172.21.15.28:0/694113185 shutdown_connections 2024-05-12T22:20:22.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:22.063+0000 7f8207e00640 1 -- 172.21.15.28:0/694113185 wait complete. 2024-05-12T22:20:22.083 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-12T22:20:22.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:22.297+0000 7f0fd3200640 1 -- 172.21.15.28:0/2583369526 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0fd41076e0 msgr2=0x7f0fd4107ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:20:22.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:22.297+0000 7f0fd3200640 1 --2- 172.21.15.28:0/2583369526 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0fd41076e0 0x7f0fd4107ac0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f0fc4009cd0 tx=0x7f0fc402a8f0 comp rx=0 tx=0).stop 2024-05-12T22:20:22.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:22.298+0000 7f0fd3200640 1 -- 172.21.15.28:0/2583369526 shutdown_connections 2024-05-12T22:20:22.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:22.298+0000 7f0fd3200640 1 --2- 172.21.15.28:0/2583369526 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0fd4102090 0x7f0fd410e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:20:22.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:22.298+0000 7f0fd3200640 1 --2- 172.21.15.28:0/2583369526 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0fd41016d0 0x7f0fd4101b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:20:22.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:22.298+0000 7f0fd3200640 1 --2- 172.21.15.28:0/2583369526 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0fd41076e0 0x7f0fd4107ac0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:20:22.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:22.298+0000 7f0fd3200640 1 -- 172.21.15.28:0/2583369526 >> 172.21.15.28:0/2583369526 conn(0x7f0fd40fd200 msgr2=0x7f0fd40ff620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:20:22.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:22.298+0000 7f0fd3200640 1 -- 172.21.15.28:0/2583369526 shutdown_connections 2024-05-12T22:20:22.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:22.298+0000 7f0fd3200640 1 -- 172.21.15.28:0/2583369526 wait complete. 2024-05-12T22:20:22.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:22.299+0000 7f0fd3200640 1 Processor -- start 2024-05-12T22:20:22.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:22.299+0000 7f0fd3200640 1 -- start start 2024-05-12T22:20:22.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:22.299+0000 7f0fd3200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0fd41016d0 0x7f0fd419a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:20:22.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:22.299+0000 7f0fd3200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0fd4102090 0x7f0fd419a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:20:22.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:22.299+0000 7f0fd3200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0fd41076e0 0x7f0fd419eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:20:22.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:22.299+0000 7f0fd3200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0fd4111b00 con 0x7f0fd4102090 2024-05-12T22:20:22.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:22.299+0000 7f0fd3200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f0fd4111980 con 0x7f0fd41076e0 2024-05-12T22:20:22.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:22.299+0000 7f0fd3200640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f0fd4111c80 con 0x7f0fd41016d0 2024-05-12T22:20:22.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:22.300+0000 7f0fd1e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0fd41016d0 0x7f0fd419a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:20:22.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:22.300+0000 7f0fd1400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0fd4102090 0x7f0fd419a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:20:22.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:22.300+0000 7f0fd1e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0fd41016d0 0x7f0fd419a1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:60520/0 (socket says 172.21.15.28:60520) 2024-05-12T22:20:22.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:22.300+0000 7f0fd1e00640 1 -- 172.21.15.28:0/4240842993 learned_addr learned my addr 172.21.15.28:0/4240842993 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:20:22.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:22.300+0000 7f0fd2800640 1 --2- 172.21.15.28:0/4240842993 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0fd41076e0 0x7f0fd419eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:20:22.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:22.300+0000 7f0fd1e00640 1 -- 172.21.15.28:0/4240842993 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0fd41076e0 msgr2=0x7f0fd419eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:20:22.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:22.300+0000 7f0fd1e00640 1 --2- 172.21.15.28:0/4240842993 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0fd41076e0 0x7f0fd419eab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:20:22.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:22.300+0000 7f0fd1e00640 1 -- 172.21.15.28:0/4240842993 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0fd4102090 msgr2=0x7f0fd419a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:20:22.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:22.300+0000 7f0fd1e00640 1 --2- 172.21.15.28:0/4240842993 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0fd4102090 0x7f0fd419a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:20:22.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:22.300+0000 7f0fd1e00640 1 -- 172.21.15.28:0/4240842993 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fd419f1e0 con 0x7f0fd41016d0 2024-05-12T22:20:22.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:22.300+0000 7f0fd1400640 1 --2- 172.21.15.28:0/4240842993 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0fd4102090 0x7f0fd419a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:20:22.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:22.300+0000 7f0fd2800640 1 --2- 172.21.15.28:0/4240842993 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0fd41076e0 0x7f0fd419eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:20:22.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:22.301+0000 7f0fd1e00640 1 --2- 172.21.15.28:0/4240842993 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0fd41016d0 0x7f0fd419a1f0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f0fc4000f80 tx=0x7f0fc4003ea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:20:22.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:22.301+0000 7f0fcaa00640 1 -- 172.21.15.28:0/4240842993 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fc4030e40 con 0x7f0fd41016d0 2024-05-12T22:20:22.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:22.301+0000 7f0fd3200640 1 -- 172.21.15.28:0/4240842993 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0fd419f470 con 0x7f0fd41016d0 2024-05-12T22:20:22.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:22.301+0000 7f0fcaa00640 1 -- 172.21.15.28:0/4240842993 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0fc4040070 con 0x7f0fd41016d0 2024-05-12T22:20:22.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:22.301+0000 7f0fcaa00640 1 -- 172.21.15.28:0/4240842993 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fc4045ae0 con 0x7f0fd41016d0 2024-05-12T22:20:22.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:22.301+0000 7f0fd3200640 1 -- 172.21.15.28:0/4240842993 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0fd41a6d60 con 0x7f0fd41016d0 2024-05-12T22:20:22.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:22.303+0000 7f0fd3200640 1 -- 172.21.15.28:0/4240842993 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0fd4068570 con 0x7f0fd41016d0 2024-05-12T22:20:22.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:22.304+0000 7f0fcaa00640 1 -- 172.21.15.28:0/4240842993 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7f0fc4045c80 con 0x7f0fd41016d0 2024-05-12T22:20:22.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:22.306+0000 7f0fcaa00640 1 --2- 172.21.15.28:0/4240842993 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f0fb80a2d00 0x7f0fb80a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:20:22.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:22.306+0000 7f0fcaa00640 1 -- 172.21.15.28:0/4240842993 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8513+0+0 (secure 0 0 0) 0x7f0fc40f0390 con 0x7f0fd41016d0 2024-05-12T22:20:22.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:22.306+0000 7f0fd1400640 1 --2- 172.21.15.28:0/4240842993 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f0fb80a2d00 0x7f0fb80a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:20:22.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:22.307+0000 7f0fd1400640 1 --2- 172.21.15.28:0/4240842993 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f0fb80a2d00 0x7f0fb80a51c0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f0fc00095f0 tx=0x7f0fc0005f90 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:20:22.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:22.310+0000 7f0fcaa00640 1 -- 172.21.15.28:0/4240842993 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f0fc40bd060 con 0x7f0fd41016d0 2024-05-12T22:20:22.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:22.510+0000 7f0fd3200640 1 -- 172.21.15.28:0/4240842993 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f0fd41a6f30 con 0x7f0fd41016d0 2024-05-12T22:20:22.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:22.511+0000 7f0fcaa00640 1 -- 172.21.15.28:0/4240842993 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v99) ==== 100+0+11 (secure 0 0 0) 0x7f0fc40bc960 con 0x7f0fd41016d0 2024-05-12T22:20:22.512 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-05-12T22:20:22.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:22.513+0000 7f0fd3200640 1 -- 172.21.15.28:0/4240842993 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f0fb80a2d00 msgr2=0x7f0fb80a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:20:22.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:22.513+0000 7f0fd3200640 1 --2- 172.21.15.28:0/4240842993 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f0fb80a2d00 0x7f0fb80a51c0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f0fc00095f0 tx=0x7f0fc0005f90 comp rx=0 tx=0).stop 2024-05-12T22:20:22.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:22.513+0000 7f0fd3200640 1 -- 172.21.15.28:0/4240842993 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0fd41016d0 msgr2=0x7f0fd419a1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:20:22.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:22.513+0000 7f0fd3200640 1 --2- 172.21.15.28:0/4240842993 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0fd41016d0 0x7f0fd419a1f0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f0fc4000f80 tx=0x7f0fc4003ea0 comp rx=0 tx=0).stop 2024-05-12T22:20:22.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:22.513+0000 7f0fd3200640 1 -- 172.21.15.28:0/4240842993 shutdown_connections 2024-05-12T22:20:22.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:22.513+0000 7f0fd3200640 1 --2- 172.21.15.28:0/4240842993 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f0fb80a2d00 0x7f0fb80a51c0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:20:22.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:22.513+0000 7f0fd3200640 1 --2- 172.21.15.28:0/4240842993 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0fd41076e0 0x7f0fd419eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:20:22.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:22.513+0000 7f0fd3200640 1 --2- 172.21.15.28:0/4240842993 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0fd4102090 0x7f0fd419a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:20:22.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:22.513+0000 7f0fd3200640 1 --2- 172.21.15.28:0/4240842993 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0fd41016d0 0x7f0fd419a1f0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:20:22.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:22.513+0000 7f0fd3200640 1 -- 172.21.15.28:0/4240842993 >> 172.21.15.28:0/4240842993 conn(0x7f0fd40fd200 msgr2=0x7f0fd410ce40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:20:22.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:22.514+0000 7f0fd3200640 1 -- 172.21.15.28:0/4240842993 shutdown_connections 2024-05-12T22:20:22.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:22.514+0000 7f0fd3200640 1 -- 172.21.15.28:0/4240842993 wait complete. 2024-05-12T22:20:22.534 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-05-12T22:20:22.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:20:22 smithi186 ceph-mon[34785]: pgmap v354: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-05-12T22:20:22.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:20:22 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/694113185' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-12T22:20:22.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:20:22 smithi086 ceph-mon[34297]: pgmap v354: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-05-12T22:20:22.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:20:22 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/694113185' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-12T22:20:22.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:22.745+0000 7f36ee200640 1 -- 172.21.15.28:0/1790802896 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f36e81098f0 msgr2=0x7f36e8109cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:20:22.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:22.745+0000 7f36ee200640 1 --2- 172.21.15.28:0/1790802896 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f36e81098f0 0x7f36e8109cd0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f36dc009900 tx=0x7f36dc02a3f0 comp rx=0 tx=0).stop 2024-05-12T22:20:22.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:22.745+0000 7f36ee200640 1 -- 172.21.15.28:0/1790802896 shutdown_connections 2024-05-12T22:20:22.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:22.745+0000 7f36ee200640 1 --2- 172.21.15.28:0/1790802896 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f36e81042a0 0x7f36e81107b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:20:22.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:22.745+0000 7f36ee200640 1 --2- 172.21.15.28:0/1790802896 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f36e81038e0 0x7f36e8103d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:20:22.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:22.745+0000 7f36ee200640 1 --2- 172.21.15.28:0/1790802896 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f36e81098f0 0x7f36e8109cd0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:20:22.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:22.745+0000 7f36ee200640 1 -- 172.21.15.28:0/1790802896 >> 172.21.15.28:0/1790802896 conn(0x7f36e80ff370 msgr2=0x7f36e8101790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:20:22.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:22.745+0000 7f36ee200640 1 -- 172.21.15.28:0/1790802896 shutdown_connections 2024-05-12T22:20:22.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:22.745+0000 7f36ee200640 1 -- 172.21.15.28:0/1790802896 wait complete. 2024-05-12T22:20:22.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:22.746+0000 7f36ee200640 1 Processor -- start 2024-05-12T22:20:22.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:22.747+0000 7f36ee200640 1 -- start start 2024-05-12T22:20:22.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:22.747+0000 7f36ee200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f36e81038e0 0x7f36e819a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:20:22.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:22.747+0000 7f36ee200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f36e81042a0 0x7f36e819a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:20:22.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:22.747+0000 7f36ee200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f36e81098f0 0x7f36e819ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:20:22.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:22.747+0000 7f36ee200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f36e8113d20 con 0x7f36e81038e0 2024-05-12T22:20:22.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:22.747+0000 7f36ee200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f36e8113ba0 con 0x7f36e81098f0 2024-05-12T22:20:22.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:22.747+0000 7f36ee200640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f36e8113ea0 con 0x7f36e81042a0 2024-05-12T22:20:22.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:22.747+0000 7f36ece00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f36e81038e0 0x7f36e819a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:20:22.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:22.747+0000 7f36ece00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f36e81038e0 0x7f36e819a1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60190/0 (socket says 172.21.15.28:60190) 2024-05-12T22:20:22.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:22.747+0000 7f36ece00640 1 -- 172.21.15.28:0/3621187102 learned_addr learned my addr 172.21.15.28:0/3621187102 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:20:22.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:22.747+0000 7f36e7e00640 1 --2- 172.21.15.28:0/3621187102 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f36e81042a0 0x7f36e819a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:20:22.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:22.747+0000 7f36ed800640 1 --2- 172.21.15.28:0/3621187102 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f36e81098f0 0x7f36e819ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:20:22.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:22.748+0000 7f36e7e00640 1 -- 172.21.15.28:0/3621187102 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f36e81098f0 msgr2=0x7f36e819ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:20:22.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:22.748+0000 7f36e7e00640 1 --2- 172.21.15.28:0/3621187102 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f36e81098f0 0x7f36e819ea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:20:22.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:22.748+0000 7f36e7e00640 1 -- 172.21.15.28:0/3621187102 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f36e81038e0 msgr2=0x7f36e819a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:20:22.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:22.748+0000 7f36e7e00640 1 --2- 172.21.15.28:0/3621187102 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f36e81038e0 0x7f36e819a1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:20:22.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:22.748+0000 7f36e7e00640 1 -- 172.21.15.28:0/3621187102 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36e819f190 con 0x7f36e81042a0 2024-05-12T22:20:22.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:22.748+0000 7f36e7e00640 1 --2- 172.21.15.28:0/3621187102 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f36e81042a0 0x7f36e819a6e0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f36d400e9f0 tx=0x7f36d400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:20:22.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:22.748+0000 7f36e5600640 1 -- 172.21.15.28:0/3621187102 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36d400ccc0 con 0x7f36e81042a0 2024-05-12T22:20:22.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:22.749+0000 7f36ee200640 1 -- 172.21.15.28:0/3621187102 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36e819f480 con 0x7f36e81042a0 2024-05-12T22:20:22.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:22.749+0000 7f36e5600640 1 -- 172.21.15.28:0/3621187102 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f36d4010910 con 0x7f36e81042a0 2024-05-12T22:20:22.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:22.749+0000 7f36e5600640 1 -- 172.21.15.28:0/3621187102 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36d4018960 con 0x7f36e81042a0 2024-05-12T22:20:22.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:22.749+0000 7f36ee200640 1 -- 172.21.15.28:0/3621187102 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36e81a6d10 con 0x7f36e81042a0 2024-05-12T22:20:22.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:22.750+0000 7f36ee200640 1 -- 172.21.15.28:0/3621187102 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f36e80689c0 con 0x7f36e81042a0 2024-05-12T22:20:22.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:22.752+0000 7f36e5600640 1 -- 172.21.15.28:0/3621187102 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7f36d4010ac0 con 0x7f36e81042a0 2024-05-12T22:20:22.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:22.753+0000 7f36e5600640 1 --2- 172.21.15.28:0/3621187102 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f36d00a2d00 0x7f36d00a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:20:22.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:22.753+0000 7f36e5600640 1 -- 172.21.15.28:0/3621187102 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8513+0+0 (secure 0 0 0) 0x7f36d40cfb40 con 0x7f36e81042a0 2024-05-12T22:20:22.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:22.753+0000 7f36ece00640 1 --2- 172.21.15.28:0/3621187102 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f36d00a2d00 0x7f36d00a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:20:22.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:22.754+0000 7f36ece00640 1 --2- 172.21.15.28:0/3621187102 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f36d00a2d00 0x7f36d00a51c0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f36dc000c00 tx=0x7f36dc0046c0 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:20:22.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:22.755+0000 7f36e5600640 1 -- 172.21.15.28:0/3621187102 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f36d409c0b0 con 0x7f36e81042a0 2024-05-12T22:20:22.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:20:22 smithi028 ceph-mon[29847]: pgmap v354: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-05-12T22:20:22.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:20:22 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/694113185' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-12T22:20:22.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:22.975+0000 7f36ee200640 1 -- 172.21.15.28:0/3621187102 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f36e810be50 con 0x7f36e81042a0 2024-05-12T22:20:22.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:22.976+0000 7f36e5600640 1 -- 172.21.15.28:0/3621187102 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 16 entries v99) ==== 81+0+944 (secure 0 0 0) 0x7f36d40a03b0 con 0x7f36e81042a0 2024-05-12T22:20:22.977 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/2580087728 2024-05-13T22:08:58.638536+0000 2024-05-12T22:20:22.977 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:6801/3123328651 2024-05-13T22:08:58.638536+0000 2024-05-12T22:20:22.977 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/820158394 2024-05-13T22:08:58.638536+0000 2024-05-12T22:20:22.977 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/1978033727 2024-05-13T22:08:58.638536+0000 2024-05-12T22:20:22.977 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/1177009533 2024-05-13T21:58:45.550459+0000 2024-05-12T22:20:22.977 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:6800/3760499540 2024-05-13T21:58:45.550459+0000 2024-05-12T22:20:22.977 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:6801/3783742728 2024-05-13T21:58:10.421564+0000 2024-05-12T22:20:22.978 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:6800/3783742728 2024-05-13T21:58:10.421564+0000 2024-05-12T22:20:22.978 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/2229156260 2024-05-13T21:58:45.550459+0000 2024-05-12T22:20:22.978 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/1202140963 2024-05-13T21:58:10.421564+0000 2024-05-12T22:20:22.978 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:6800/3123328651 2024-05-13T22:08:58.638536+0000 2024-05-12T22:20:22.978 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/2924223784 2024-05-13T21:58:45.550459+0000 2024-05-12T22:20:22.978 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:6801/3760499540 2024-05-13T21:58:45.550459+0000 2024-05-12T22:20:22.979 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/1240027206 2024-05-13T21:58:10.421564+0000 2024-05-12T22:20:22.979 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/1581755795 2024-05-13T22:08:58.638536+0000 2024-05-12T22:20:22.979 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/171417728 2024-05-13T21:58:10.421564+0000 2024-05-12T22:20:22.983 INFO:teuthology.orchestra.run.smithi028.stderr:listed 16 entries 2024-05-12T22:20:22.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:22.978+0000 7f36ee200640 1 -- 172.21.15.28:0/3621187102 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f36d00a2d00 msgr2=0x7f36d00a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:20:22.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:22.978+0000 7f36ee200640 1 --2- 172.21.15.28:0/3621187102 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f36d00a2d00 0x7f36d00a51c0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f36dc000c00 tx=0x7f36dc0046c0 comp rx=0 tx=0).stop 2024-05-12T22:20:22.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:22.978+0000 7f36ee200640 1 -- 172.21.15.28:0/3621187102 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f36e81042a0 msgr2=0x7f36e819a6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:20:22.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:22.978+0000 7f36ee200640 1 --2- 172.21.15.28:0/3621187102 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f36e81042a0 0x7f36e819a6e0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f36d400e9f0 tx=0x7f36d400eeb0 comp rx=0 tx=0).stop 2024-05-12T22:20:22.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:22.978+0000 7f36ee200640 1 -- 172.21.15.28:0/3621187102 shutdown_connections 2024-05-12T22:20:22.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:22.978+0000 7f36ee200640 1 --2- 172.21.15.28:0/3621187102 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f36d00a2d00 0x7f36d00a51c0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:20:22.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:22.978+0000 7f36ee200640 1 --2- 172.21.15.28:0/3621187102 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f36e81098f0 0x7f36e819ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:20:22.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:22.979+0000 7f36ee200640 1 --2- 172.21.15.28:0/3621187102 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f36e81042a0 0x7f36e819a6e0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:20:22.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:22.979+0000 7f36ee200640 1 --2- 172.21.15.28:0/3621187102 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f36e81038e0 0x7f36e819a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:20:22.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:22.979+0000 7f36ee200640 1 -- 172.21.15.28:0/3621187102 >> 172.21.15.28:0/3621187102 conn(0x7f36e80ff370 msgr2=0x7f36e80ff980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:20:22.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:22.979+0000 7f36ee200640 1 -- 172.21.15.28:0/3621187102 shutdown_connections 2024-05-12T22:20:22.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:22.979+0000 7f36ee200640 1 -- 172.21.15.28:0/3621187102 wait complete. 2024-05-12T22:20:22.999 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-05-12T22:20:22.999 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-12T22:20:22.999 DEBUG:teuthology.orchestra.run.smithi028:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-05-12T22:20:23.249 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi028.front.sepia.ceph.com 2024-05-12T22:20:23.249 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-12T22:20:23.250 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-05-12T22:20:23.296 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-12T22:20:23.296 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list 2024-05-12T22:20:23.378 INFO:teuthology.orchestra.run.smithi028.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-05-12T22:20:23.378 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-12T22:20:23.378 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-12T22:20:23.502 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-12T22:20:23.503 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link delete ceph-brx 2024-05-12T22:20:23.618 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-05-12T22:20:23.629 INFO:tasks.cephadm:Teardown begin 2024-05-12T22:20:23.629 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-05-12T22:20:23.629 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-12T22:20:23.631 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:20:23 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/4240842993' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-12T22:20:23.631 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:20:23 smithi028 ceph-mon[29847]: pgmap v355: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-05-12T22:20:23.631 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:20:23 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/3621187102' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-12T22:20:23.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:20:23 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/4240842993' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-12T22:20:23.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:20:23 smithi186 ceph-mon[34785]: pgmap v355: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-05-12T22:20:23.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:20:23 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/3621187102' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-12T22:20:23.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:20:23 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/4240842993' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-12T22:20:23.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:20:23 smithi086 ceph-mon[34297]: pgmap v355: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-05-12T22:20:23.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:20:23 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/3621187102' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-12T22:20:23.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:23.824+0000 7f55fe000640 1 -- 172.21.15.28:0/223213643 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f55f81041f0 msgr2=0x7f55f8110700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:20:23.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:23.824+0000 7f55fe000640 1 --2- 172.21.15.28:0/223213643 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f55f81041f0 0x7f55f8110700 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f55f0009cd0 tx=0x7f55f002a8f0 comp rx=0 tx=0).stop 2024-05-12T22:20:23.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:23.824+0000 7f55fe000640 1 -- 172.21.15.28:0/223213643 shutdown_connections 2024-05-12T22:20:23.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:23.825+0000 7f55fe000640 1 --2- 172.21.15.28:0/223213643 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f55f81041f0 0x7f55f8110700 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:20:23.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:23.825+0000 7f55fe000640 1 --2- 172.21.15.28:0/223213643 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f55f8103830 0x7f55f8103cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:20:23.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:23.825+0000 7f55fe000640 1 --2- 172.21.15.28:0/223213643 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55f8109840 0x7f55f8109c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:20:23.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:23.825+0000 7f55fe000640 1 -- 172.21.15.28:0/223213643 >> 172.21.15.28:0/223213643 conn(0x7f55f80ff300 msgr2=0x7f55f8101720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:20:23.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:23.825+0000 7f55fe000640 1 -- 172.21.15.28:0/223213643 shutdown_connections 2024-05-12T22:20:23.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:23.825+0000 7f55fe000640 1 -- 172.21.15.28:0/223213643 wait complete. 2024-05-12T22:20:23.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:23.826+0000 7f55fe000640 1 Processor -- start 2024-05-12T22:20:23.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:23.826+0000 7f55fe000640 1 -- start start 2024-05-12T22:20:23.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:23.826+0000 7f55fe000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f55f8103830 0x7f55f819a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:20:23.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:23.826+0000 7f55fe000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55f81041f0 0x7f55f819a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:20:23.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:23.826+0000 7f55fe000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f55f8109840 0x7f55f819e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:20:23.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:23.826+0000 7f55fe000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f55f8113ca0 con 0x7f55f81041f0 2024-05-12T22:20:23.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:23.826+0000 7f55fe000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f55f8113b20 con 0x7f55f8103830 2024-05-12T22:20:23.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:23.826+0000 7f55fe000640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f55f8113e20 con 0x7f55f8109840 2024-05-12T22:20:23.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:23.827+0000 7f55fcc00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f55f8103830 0x7f55f819a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:20:23.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:23.827+0000 7f55fcc00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f55f8103830 0x7f55f819a100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:54760/0 (socket says 172.21.15.28:54760) 2024-05-12T22:20:23.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:23.827+0000 7f55fcc00640 1 -- 172.21.15.28:0/3008729710 learned_addr learned my addr 172.21.15.28:0/3008729710 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:20:23.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:23.827+0000 7f55fd600640 1 --2- 172.21.15.28:0/3008729710 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f55f8109840 0x7f55f819e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:20:23.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:23.827+0000 7f55f7e00640 1 --2- 172.21.15.28:0/3008729710 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55f81041f0 0x7f55f819a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:20:23.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:23.827+0000 7f55fcc00640 1 -- 172.21.15.28:0/3008729710 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f55f8109840 msgr2=0x7f55f819e9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:20:23.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:23.827+0000 7f55fcc00640 1 --2- 172.21.15.28:0/3008729710 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f55f8109840 0x7f55f819e9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:20:23.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:23.827+0000 7f55fcc00640 1 -- 172.21.15.28:0/3008729710 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55f81041f0 msgr2=0x7f55f819a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:20:23.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:23.827+0000 7f55fcc00640 1 --2- 172.21.15.28:0/3008729710 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55f81041f0 0x7f55f819a640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:20:23.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:23.827+0000 7f55fcc00640 1 -- 172.21.15.28:0/3008729710 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55f819f0f0 con 0x7f55f8103830 2024-05-12T22:20:23.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:23.827+0000 7f55fcc00640 1 --2- 172.21.15.28:0/3008729710 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f55f8103830 0x7f55f819a100 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f55ec00e9f0 tx=0x7f55ec00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:20:23.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:23.828+0000 7f55f5600640 1 -- 172.21.15.28:0/3008729710 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55ec00cd80 con 0x7f55f8103830 2024-05-12T22:20:23.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:23.828+0000 7f55fe000640 1 -- 172.21.15.28:0/3008729710 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55f819f3e0 con 0x7f55f8103830 2024-05-12T22:20:23.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:23.828+0000 7f55f5600640 1 -- 172.21.15.28:0/3008729710 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f55ec010910 con 0x7f55f8103830 2024-05-12T22:20:23.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:23.828+0000 7f55f5600640 1 -- 172.21.15.28:0/3008729710 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55ec0188f0 con 0x7f55f8103830 2024-05-12T22:20:23.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:23.828+0000 7f55fe000640 1 -- 172.21.15.28:0/3008729710 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55f81a6c70 con 0x7f55f8103830 2024-05-12T22:20:23.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:23.829+0000 7f55fe000640 1 -- 172.21.15.28:0/3008729710 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f55f80689c0 con 0x7f55f8103830 2024-05-12T22:20:23.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:23.832+0000 7f55f5600640 1 -- 172.21.15.28:0/3008729710 <== mon.2 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7f55ec010ac0 con 0x7f55f8103830 2024-05-12T22:20:23.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:23.833+0000 7f55f5600640 1 --2- 172.21.15.28:0/3008729710 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f55e00a2d00 0x7f55e00a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:20:23.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:23.833+0000 7f55f5600640 1 -- 172.21.15.28:0/3008729710 <== mon.2 v2:172.21.15.86:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8513+0+0 (secure 0 0 0) 0x7f55ec0cf8c0 con 0x7f55f8103830 2024-05-12T22:20:23.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:23.833+0000 7f55f7e00640 1 --2- 172.21.15.28:0/3008729710 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f55e00a2d00 0x7f55e00a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:20:23.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:23.834+0000 7f55f7e00640 1 --2- 172.21.15.28:0/3008729710 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f55e00a2d00 0x7f55e00a51c0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f55f81018a0 tx=0x7f55e4005f90 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:20:23.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:23.837+0000 7f55f5600640 1 -- 172.21.15.28:0/3008729710 <== mon.2 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f55ec09c480 con 0x7f55f8103830 2024-05-12T22:20:24.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:24.084+0000 7f55fe000640 1 -- 172.21.15.28:0/3008729710 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f55f819b450 con 0x7f55f8103830 2024-05-12T22:20:24.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:24.086+0000 7f55f5600640 1 -- 172.21.15.28:0/3008729710 <== mon.2 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11864 (secure 0 0 0) 0x7f55ec09bd80 con 0x7f55f8103830 2024-05-12T22:20:24.087 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-12T22:20:24.087 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1844141745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1844141745},{"type":"v1","addr":"172.21.15.186:6833","nonce":1844141745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6833/2960763758","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":2960763758},{"type":"v1","addr":"172.21.15.86:6833","nonce":2960763758}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24470,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/4131134219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":4131134219},{"type":"v1","addr":"172.21.15.28:6835","nonce":4131134219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24481,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/1727869064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1727869064},{"type":"v1","addr":"172.21.15.186:6835","nonce":1727869064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24482,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2938260704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2938260704},{"type":"v1","addr":"172.21.15.28:6837","nonce":2938260704}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24494,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3473700910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3473700910},{"type":"v1","addr":"172.21.15.86:6835","nonce":3473700910}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-12T22:06:33.426986+0000","modified":"2024-05-12T22:13:22.499381+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24581,"mds_1":24550,"mds_2":24532},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24499":{"gid":24499,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6837/3600809235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":3600809235},{"type":"v1","addr":"172.21.15.186:6837","nonce":3600809235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"i","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6839/2399610807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":2399610807},{"type":"v1","addr":"172.21.15.186:6839","nonce":2399610807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24530":{"gid":24530,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.86:6837/3782448884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":3782448884},{"type":"v1","addr":"172.21.15.86:6837","nonce":3782448884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24532":{"gid":24532,"name":"g","rank":2,"incarnation":21,"state":"up:active","state_seq":6,"addr":"172.21.15.28:6839/123892158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":123892158},{"type":"v1","addr":"172.21.15.28:6839","nonce":123892158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24550":{"gid":24550,"name":"a","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.28:6841/761183870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":761183870},{"type":"v1","addr":"172.21.15.28:6841","nonce":761183870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24581":{"gid":24581,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6839/1251999130","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6838","nonce":1251999130},{"type":"v1","addr":"172.21.15.86:6839","nonce":1251999130}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24581,"qdb_cluster":[24550,24581,24532]},"id":1}]} 2024-05-12T22:20:24.087 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 26 2024-05-12T22:20:24.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:24.088+0000 7f55fe000640 1 -- 172.21.15.28:0/3008729710 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f55e00a2d00 msgr2=0x7f55e00a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:20:24.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:24.088+0000 7f55fe000640 1 --2- 172.21.15.28:0/3008729710 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f55e00a2d00 0x7f55e00a51c0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f55f81018a0 tx=0x7f55e4005f90 comp rx=0 tx=0).stop 2024-05-12T22:20:24.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:24.088+0000 7f55fe000640 1 -- 172.21.15.28:0/3008729710 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f55f8103830 msgr2=0x7f55f819a100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:20:24.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:24.088+0000 7f55fe000640 1 --2- 172.21.15.28:0/3008729710 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f55f8103830 0x7f55f819a100 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f55ec00e9f0 tx=0x7f55ec00eeb0 comp rx=0 tx=0).stop 2024-05-12T22:20:24.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:24.089+0000 7f55fe000640 1 -- 172.21.15.28:0/3008729710 shutdown_connections 2024-05-12T22:20:24.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:24.089+0000 7f55fe000640 1 --2- 172.21.15.28:0/3008729710 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f55e00a2d00 0x7f55e00a51c0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:20:24.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:24.089+0000 7f55fe000640 1 --2- 172.21.15.28:0/3008729710 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f55f8109840 0x7f55f819e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:20:24.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:24.089+0000 7f55fe000640 1 --2- 172.21.15.28:0/3008729710 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55f81041f0 0x7f55f819a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:20:24.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:24.089+0000 7f55fe000640 1 --2- 172.21.15.28:0/3008729710 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f55f8103830 0x7f55f819a100 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:20:24.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:24.089+0000 7f55fe000640 1 -- 172.21.15.28:0/3008729710 >> 172.21.15.28:0/3008729710 conn(0x7f55f80ff300 msgr2=0x7f55f80ff8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:20:24.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:24.089+0000 7f55fe000640 1 -- 172.21.15.28:0/3008729710 shutdown_connections 2024-05-12T22:20:24.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:24.089+0000 7f55fe000640 1 -- 172.21.15.28:0/3008729710 wait complete. 2024-05-12T22:20:24.110 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-12T22:20:24.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:24.327+0000 7fc2d5200640 1 -- 172.21.15.28:0/1306752183 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc2d0109830 msgr2=0x7fc2d0109c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:20:24.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:24.327+0000 7fc2d5200640 1 --2- 172.21.15.28:0/1306752183 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc2d0109830 0x7fc2d0109c10 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fc2c0009890 tx=0x7fc2c002a380 comp rx=0 tx=0).stop 2024-05-12T22:20:24.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:24.328+0000 7fc2d5200640 1 -- 172.21.15.28:0/1306752183 shutdown_connections 2024-05-12T22:20:24.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:24.328+0000 7fc2d5200640 1 --2- 172.21.15.28:0/1306752183 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc2d01041e0 0x7fc2d01106f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:20:24.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:24.328+0000 7fc2d5200640 1 --2- 172.21.15.28:0/1306752183 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2d0103820 0x7fc2d0103ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:20:24.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:24.328+0000 7fc2d5200640 1 --2- 172.21.15.28:0/1306752183 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc2d0109830 0x7fc2d0109c10 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:20:24.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:24.328+0000 7fc2d5200640 1 -- 172.21.15.28:0/1306752183 >> 172.21.15.28:0/1306752183 conn(0x7fc2d00ff2d0 msgr2=0x7fc2d01016f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:20:24.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:24.328+0000 7fc2d5200640 1 -- 172.21.15.28:0/1306752183 shutdown_connections 2024-05-12T22:20:24.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:24.328+0000 7fc2d5200640 1 -- 172.21.15.28:0/1306752183 wait complete. 2024-05-12T22:20:24.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:24.329+0000 7fc2d5200640 1 Processor -- start 2024-05-12T22:20:24.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:24.329+0000 7fc2d5200640 1 -- start start 2024-05-12T22:20:24.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:24.329+0000 7fc2d5200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc2d0103820 0x7fc2d019a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:20:24.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:24.329+0000 7fc2d5200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2d01041e0 0x7fc2d019a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:20:24.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:24.329+0000 7fc2d5200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc2d0109830 0x7fc2d019ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:20:24.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:24.329+0000 7fc2d5200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc2d0113c00 con 0x7fc2d01041e0 2024-05-12T22:20:24.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:24.329+0000 7fc2d5200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fc2d0113a80 con 0x7fc2d0103820 2024-05-12T22:20:24.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:24.329+0000 7fc2d5200640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc2d0113d80 con 0x7fc2d0109830 2024-05-12T22:20:24.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:24.329+0000 7fc2cea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2d01041e0 0x7fc2d019a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:20:24.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:24.329+0000 7fc2cea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2d01041e0 0x7fc2d019a6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60238/0 (socket says 172.21.15.28:60238) 2024-05-12T22:20:24.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:24.329+0000 7fc2cea00640 1 -- 172.21.15.28:0/2046808506 learned_addr learned my addr 172.21.15.28:0/2046808506 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:20:24.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:24.330+0000 7fc2cfe00640 1 --2- 172.21.15.28:0/2046808506 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc2d0109830 0x7fc2d019ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:20:24.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:24.329+0000 7fc2cf400640 1 --2- 172.21.15.28:0/2046808506 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc2d0103820 0x7fc2d019a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:20:24.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:24.330+0000 7fc2cea00640 1 -- 172.21.15.28:0/2046808506 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc2d0109830 msgr2=0x7fc2d019ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:20:24.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:24.330+0000 7fc2cea00640 1 --2- 172.21.15.28:0/2046808506 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc2d0109830 0x7fc2d019ea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:20:24.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:24.330+0000 7fc2cea00640 1 -- 172.21.15.28:0/2046808506 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc2d0103820 msgr2=0x7fc2d019a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:20:24.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:24.330+0000 7fc2cea00640 1 --2- 172.21.15.28:0/2046808506 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc2d0103820 0x7fc2d019a160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:20:24.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:24.330+0000 7fc2cea00640 1 -- 172.21.15.28:0/2046808506 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2d019f150 con 0x7fc2d01041e0 2024-05-12T22:20:24.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:24.330+0000 7fc2cea00640 1 --2- 172.21.15.28:0/2046808506 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2d01041e0 0x7fc2d019a6a0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fc2bc00e9c0 tx=0x7fc2bc00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:20:24.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:24.330+0000 7fc2c7e00640 1 -- 172.21.15.28:0/2046808506 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2bc00cde0 con 0x7fc2d01041e0 2024-05-12T22:20:24.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:24.331+0000 7fc2d5200640 1 -- 172.21.15.28:0/2046808506 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2d019f440 con 0x7fc2d01041e0 2024-05-12T22:20:24.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:24.331+0000 7fc2c7e00640 1 -- 172.21.15.28:0/2046808506 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc2bc0035f0 con 0x7fc2d01041e0 2024-05-12T22:20:24.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:24.331+0000 7fc2c7e00640 1 -- 172.21.15.28:0/2046808506 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2bc018630 con 0x7fc2d01041e0 2024-05-12T22:20:24.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:24.331+0000 7fc2d5200640 1 -- 172.21.15.28:0/2046808506 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2d01a6cd0 con 0x7fc2d01041e0 2024-05-12T22:20:24.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:24.332+0000 7fc2d5200640 1 -- 172.21.15.28:0/2046808506 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2d00689c0 con 0x7fc2d01041e0 2024-05-12T22:20:24.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:24.335+0000 7fc2c7e00640 1 -- 172.21.15.28:0/2046808506 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7fc2bc0037a0 con 0x7fc2d01041e0 2024-05-12T22:20:24.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:24.337+0000 7fc2c7e00640 1 --2- 172.21.15.28:0/2046808506 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fc2b40a2d00 0x7fc2b40a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:20:24.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:24.337+0000 7fc2c7e00640 1 -- 172.21.15.28:0/2046808506 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8513+0+0 (secure 0 0 0) 0x7fc2bc0cf730 con 0x7fc2d01041e0 2024-05-12T22:20:24.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:24.337+0000 7fc2cf400640 1 --2- 172.21.15.28:0/2046808506 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fc2b40a2d00 0x7fc2b40a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:20:24.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:24.338+0000 7fc2cf400640 1 --2- 172.21.15.28:0/2046808506 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fc2b40a2d00 0x7fc2b40a51c0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fc2c00097b0 tx=0x7fc2c00058d0 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:20:24.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:24.342+0000 7fc2c7e00640 1 -- 172.21.15.28:0/2046808506 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fc2bc09c2f0 con 0x7fc2d01041e0 2024-05-12T22:20:24.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:24.588+0000 7fc2d5200640 1 -- 172.21.15.28:0/2046808506 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc2d002d1c0 con 0x7fc2d01041e0 2024-05-12T22:20:24.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:24.590+0000 7fc2c7e00640 1 -- 172.21.15.28:0/2046808506 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11864 (secure 0 0 0) 0x7fc2bc09bbf0 con 0x7fc2d01041e0 2024-05-12T22:20:24.591 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-12T22:20:24.592 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1844141745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1844141745},{"type":"v1","addr":"172.21.15.186:6833","nonce":1844141745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24452,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6833/2960763758","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":2960763758},{"type":"v1","addr":"172.21.15.86:6833","nonce":2960763758}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24470,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/4131134219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":4131134219},{"type":"v1","addr":"172.21.15.28:6835","nonce":4131134219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24481,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/1727869064","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1727869064},{"type":"v1","addr":"172.21.15.186:6835","nonce":1727869064}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24482,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2938260704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2938260704},{"type":"v1","addr":"172.21.15.28:6837","nonce":2938260704}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24494,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3473700910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3473700910},{"type":"v1","addr":"172.21.15.86:6835","nonce":3473700910}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-12T22:06:33.426986+0000","modified":"2024-05-12T22:13:22.499381+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24581,"mds_1":24550,"mds_2":24532},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24499":{"gid":24499,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6837/3600809235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":3600809235},{"type":"v1","addr":"172.21.15.186:6837","nonce":3600809235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"i","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6839/2399610807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":2399610807},{"type":"v1","addr":"172.21.15.186:6839","nonce":2399610807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24530":{"gid":24530,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.86:6837/3782448884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":3782448884},{"type":"v1","addr":"172.21.15.86:6837","nonce":3782448884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24532":{"gid":24532,"name":"g","rank":2,"incarnation":21,"state":"up:active","state_seq":6,"addr":"172.21.15.28:6839/123892158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":123892158},{"type":"v1","addr":"172.21.15.28:6839","nonce":123892158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24550":{"gid":24550,"name":"a","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.28:6841/761183870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":761183870},{"type":"v1","addr":"172.21.15.28:6841","nonce":761183870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24581":{"gid":24581,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6839/1251999130","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6838","nonce":1251999130},{"type":"v1","addr":"172.21.15.86:6839","nonce":1251999130}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24581,"qdb_cluster":[24550,24581,24532]},"id":1}]} 2024-05-12T22:20:24.592 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 26 2024-05-12T22:20:24.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:24.593+0000 7fc2d5200640 1 -- 172.21.15.28:0/2046808506 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fc2b40a2d00 msgr2=0x7fc2b40a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:20:24.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:24.593+0000 7fc2d5200640 1 --2- 172.21.15.28:0/2046808506 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fc2b40a2d00 0x7fc2b40a51c0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fc2c00097b0 tx=0x7fc2c00058d0 comp rx=0 tx=0).stop 2024-05-12T22:20:24.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:24.593+0000 7fc2d5200640 1 -- 172.21.15.28:0/2046808506 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2d01041e0 msgr2=0x7fc2d019a6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:20:24.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:24.593+0000 7fc2d5200640 1 --2- 172.21.15.28:0/2046808506 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2d01041e0 0x7fc2d019a6a0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fc2bc00e9c0 tx=0x7fc2bc00ee80 comp rx=0 tx=0).stop 2024-05-12T22:20:24.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:24.593+0000 7fc2d5200640 1 -- 172.21.15.28:0/2046808506 shutdown_connections 2024-05-12T22:20:24.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:24.593+0000 7fc2d5200640 1 --2- 172.21.15.28:0/2046808506 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7fc2b40a2d00 0x7fc2b40a51c0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:20:24.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:24.593+0000 7fc2d5200640 1 --2- 172.21.15.28:0/2046808506 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc2d0109830 0x7fc2d019ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:20:24.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:24.593+0000 7fc2d5200640 1 --2- 172.21.15.28:0/2046808506 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2d01041e0 0x7fc2d019a6a0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:20:24.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:24.593+0000 7fc2d5200640 1 --2- 172.21.15.28:0/2046808506 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc2d0103820 0x7fc2d019a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:20:24.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:24.593+0000 7fc2d5200640 1 -- 172.21.15.28:0/2046808506 >> 172.21.15.28:0/2046808506 conn(0x7fc2d00ff2d0 msgr2=0x7fc2d00ffb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:20:24.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:24.594+0000 7fc2d5200640 1 -- 172.21.15.28:0/2046808506 shutdown_connections 2024-05-12T22:20:24.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:24.594+0000 7fc2d5200640 1 -- 172.21.15.28:0/2046808506 wait complete. 2024-05-12T22:20:24.613 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-12T22:20:24.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:20:24 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/3008729710' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:20:24.690 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:20:24 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/3008729710' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:20:24.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:20:24 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/3008729710' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:20:24.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:24.832+0000 7f7168a00640 1 -- 172.21.15.28:0/1433298208 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7164106710 msgr2=0x7f7164106af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:20:24.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:24.832+0000 7f7168a00640 1 --2- 172.21.15.28:0/1433298208 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7164106710 0x7f7164106af0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f7154009930 tx=0x7f715402a420 comp rx=0 tx=0).stop 2024-05-12T22:20:24.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:24.832+0000 7f7168a00640 1 -- 172.21.15.28:0/1433298208 shutdown_connections 2024-05-12T22:20:24.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:24.832+0000 7f7168a00640 1 --2- 172.21.15.28:0/1433298208 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f71641010c0 0x7f716410d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:20:24.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:24.832+0000 7f7168a00640 1 --2- 172.21.15.28:0/1433298208 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7164100700 0x7f7164100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:20:24.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:24.832+0000 7f7168a00640 1 --2- 172.21.15.28:0/1433298208 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7164106710 0x7f7164106af0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:20:24.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:24.832+0000 7f7168a00640 1 -- 172.21.15.28:0/1433298208 >> 172.21.15.28:0/1433298208 conn(0x7f7164076ab0 msgr2=0x7f7164076ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:20:24.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:24.832+0000 7f7168a00640 1 -- 172.21.15.28:0/1433298208 shutdown_connections 2024-05-12T22:20:24.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:24.832+0000 7f7168a00640 1 -- 172.21.15.28:0/1433298208 wait complete. 2024-05-12T22:20:24.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:24.833+0000 7f7168a00640 1 Processor -- start 2024-05-12T22:20:24.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:24.833+0000 7f7168a00640 1 -- start start 2024-05-12T22:20:24.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:24.834+0000 7f7168a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7164100700 0x7f71640fdfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:20:24.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:24.834+0000 7f7168a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f71641010c0 0x7f71640fe4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:20:24.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:24.834+0000 7f7168a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7164106710 0x7f71640fea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:20:24.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:24.834+0000 7f7168a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7164110ae0 con 0x7f7164100700 2024-05-12T22:20:24.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:24.834+0000 7f7168a00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f7164110960 con 0x7f7164106710 2024-05-12T22:20:24.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:24.834+0000 7f7168a00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f7164110c60 con 0x7f71641010c0 2024-05-12T22:20:24.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:24.834+0000 7f7161e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7164100700 0x7f71640fdfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:20:24.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:24.834+0000 7f7161400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f71641010c0 0x7f71640fe4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:20:24.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:24.834+0000 7f7161400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f71641010c0 0x7f71640fe4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:60598/0 (socket says 172.21.15.28:60598) 2024-05-12T22:20:24.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:24.834+0000 7f7161400640 1 -- 172.21.15.28:0/3653622450 learned_addr learned my addr 172.21.15.28:0/3653622450 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:20:24.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:24.834+0000 7f7162800640 1 --2- 172.21.15.28:0/3653622450 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7164106710 0x7f71640fea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:20:24.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:24.835+0000 7f7161e00640 1 -- 172.21.15.28:0/3653622450 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f71641010c0 msgr2=0x7f71640fe4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:20:24.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:24.835+0000 7f7161e00640 1 --2- 172.21.15.28:0/3653622450 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f71641010c0 0x7f71640fe4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:20:24.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:24.835+0000 7f7161e00640 1 -- 172.21.15.28:0/3653622450 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7164106710 msgr2=0x7f71640fea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:20:24.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:24.835+0000 7f7161e00640 1 --2- 172.21.15.28:0/3653622450 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7164106710 0x7f71640fea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:20:24.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:24.835+0000 7f7161e00640 1 -- 172.21.15.28:0/3653622450 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71641a6610 con 0x7f7164100700 2024-05-12T22:20:24.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:24.835+0000 7f7161400640 1 --2- 172.21.15.28:0/3653622450 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f71641010c0 0x7f71640fe4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:20:24.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:24.835+0000 7f7161e00640 1 --2- 172.21.15.28:0/3653622450 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7164100700 0x7f71640fdfb0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f7154005940 tx=0x7f71540042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:20:24.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:24.835+0000 7f715aa00640 1 -- 172.21.15.28:0/3653622450 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f715403f070 con 0x7f7164100700 2024-05-12T22:20:24.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:24.835+0000 7f715aa00640 1 -- 172.21.15.28:0/3653622450 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f715403b460 con 0x7f7164100700 2024-05-12T22:20:24.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:24.835+0000 7f715aa00640 1 -- 172.21.15.28:0/3653622450 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7154035070 con 0x7f7164100700 2024-05-12T22:20:24.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:24.835+0000 7f7168a00640 1 -- 172.21.15.28:0/3653622450 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71641a68a0 con 0x7f7164100700 2024-05-12T22:20:24.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:24.835+0000 7f7168a00640 1 -- 172.21.15.28:0/3653622450 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71641a6d40 con 0x7f7164100700 2024-05-12T22:20:24.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:24.837+0000 7f7168a00640 1 -- 172.21.15.28:0/3653622450 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7164101e40 con 0x7f7164100700 2024-05-12T22:20:24.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:24.839+0000 7f715aa00640 1 -- 172.21.15.28:0/3653622450 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7f715403b610 con 0x7f7164100700 2024-05-12T22:20:24.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:24.840+0000 7f715aa00640 1 --2- 172.21.15.28:0/3653622450 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f71480a2d80 0x7f71480a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:20:24.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:24.840+0000 7f715aa00640 1 -- 172.21.15.28:0/3653622450 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8513+0+0 (secure 0 0 0) 0x7f71540ef8d0 con 0x7f7164100700 2024-05-12T22:20:24.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:24.841+0000 7f7161400640 1 --2- 172.21.15.28:0/3653622450 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f71480a2d80 0x7f71480a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:20:24.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:24.841+0000 7f7161400640 1 --2- 172.21.15.28:0/3653622450 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f71480a2d80 0x7f71480a5240 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f71640ffa40 tx=0x7f715000b480 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:20:24.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:24.845+0000 7f715aa00640 1 -- 172.21.15.28:0/3653622450 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f71540bbe40 con 0x7f7164100700 2024-05-12T22:20:25.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:25.056+0000 7f7168a00640 1 -- 172.21.15.28:0/3653622450 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7164109c80 con 0x7f7164100700 2024-05-12T22:20:25.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:25.060+0000 7f715aa00640 1 -- 172.21.15.28:0/3653622450 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v99) ==== 74+0+23062 (secure 0 0 0) 0x7f71540c0140 con 0x7f7164100700 2024-05-12T22:20:25.061 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-05-12T22:20:25.062 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":99,"fsid":"6b92e7b4-10aa-11ef-bc98-c7b262605968","created":"2024-05-12T21:57:37.276782+0000","modified":"2024-05-12T22:20:13.997146+0000","last_up_change":"2024-05-12T22:05:37.929562+0000","last_in_change":"2024-05-12T22:05:18.750768+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-12T22:01:32.292826+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-12T22:06:31.540263+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-12T22:06:32.906118+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"98","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":17,"snap_epoch":98,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6992ce0a-844b-40e3-8ed1-181c8474dccf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":906003882},{"type":"v1","addr":"172.21.15.28:6803","nonce":906003882}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":906003882},{"type":"v1","addr":"172.21.15.28:6805","nonce":906003882}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":906003882},{"type":"v1","addr":"172.21.15.28:6809","nonce":906003882}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":906003882},{"type":"v1","addr":"172.21.15.28:6807","nonce":906003882}]},"public_addr":"172.21.15.28:6803/906003882","cluster_addr":"172.21.15.28:6805/906003882","heartbeat_back_addr":"172.21.15.28:6809/906003882","heartbeat_front_addr":"172.21.15.28:6807/906003882","state":["exists","up"]},{"osd":1,"uuid":"ced20c96-c481-4d48-90d1-6b43228959ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":3354141819},{"type":"v1","addr":"172.21.15.86:6801","nonce":3354141819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":3354141819},{"type":"v1","addr":"172.21.15.86:6803","nonce":3354141819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":3354141819},{"type":"v1","addr":"172.21.15.86:6807","nonce":3354141819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":3354141819},{"type":"v1","addr":"172.21.15.86:6805","nonce":3354141819}]},"public_addr":"172.21.15.86:6801/3354141819","cluster_addr":"172.21.15.86:6803/3354141819","heartbeat_back_addr":"172.21.15.86:6807/3354141819","heartbeat_front_addr":"172.21.15.86:6805/3354141819","state":["exists","up"]},{"osd":2,"uuid":"54a2f221-fa65-4795-9f5e-e660cc7c0098","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":906459365},{"type":"v1","addr":"172.21.15.186:6801","nonce":906459365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":906459365},{"type":"v1","addr":"172.21.15.186:6803","nonce":906459365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":906459365},{"type":"v1","addr":"172.21.15.186:6807","nonce":906459365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":906459365},{"type":"v1","addr":"172.21.15.186:6805","nonce":906459365}]},"public_addr":"172.21.15.186:6801/906459365","cluster_addr":"172.21.15.186:6803/906459365","heartbeat_back_addr":"172.21.15.186:6807/906459365","heartbeat_front_addr":"172.21.15.186:6805/906459365","state":["exists","up"]},{"osd":3,"uuid":"86836737-ac9a-487b-93f2-7fcd3c406960","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":4182884436},{"type":"v1","addr":"172.21.15.28:6811","nonce":4182884436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":4182884436},{"type":"v1","addr":"172.21.15.28:6813","nonce":4182884436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":4182884436},{"type":"v1","addr":"172.21.15.28:6817","nonce":4182884436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":4182884436},{"type":"v1","addr":"172.21.15.28:6815","nonce":4182884436}]},"public_addr":"172.21.15.28:6811/4182884436","cluster_addr":"172.21.15.28:6813/4182884436","heartbeat_back_addr":"172.21.15.28:6817/4182884436","heartbeat_front_addr":"172.21.15.28:6815/4182884436","state":["exists","up"]},{"osd":4,"uuid":"d162ebcd-83c0-4fd5-9925-28b2cecea935","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2596998708},{"type":"v1","addr":"172.21.15.86:6809","nonce":2596998708}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2596998708},{"type":"v1","addr":"172.21.15.86:6811","nonce":2596998708}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2596998708},{"type":"v1","addr":"172.21.15.86:6815","nonce":2596998708}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2596998708},{"type":"v1","addr":"172.21.15.86:6813","nonce":2596998708}]},"public_addr":"172.21.15.86:6809/2596998708","cluster_addr":"172.21.15.86:6811/2596998708","heartbeat_back_addr":"172.21.15.86:6815/2596998708","heartbeat_front_addr":"172.21.15.86:6813/2596998708","state":["exists","up"]},{"osd":5,"uuid":"c794336a-a105-425c-b6c7-a32b90c05bb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":1944205756},{"type":"v1","addr":"172.21.15.186:6809","nonce":1944205756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":1944205756},{"type":"v1","addr":"172.21.15.186:6811","nonce":1944205756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":1944205756},{"type":"v1","addr":"172.21.15.186:6815","nonce":1944205756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":1944205756},{"type":"v1","addr":"172.21.15.186:6813","nonce":1944205756}]},"public_addr":"172.21.15.186:6809/1944205756","cluster_addr":"172.21.15.186:6811/1944205756","heartbeat_back_addr":"172.21.15.186:6815/1944205756","heartbeat_front_addr":"172.21.15.186:6813/1944205756","state":["exists","up"]},{"osd":6,"uuid":"516932ac-af00-45b7-ad3c-d6dec7db442f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":1642308925},{"type":"v1","addr":"172.21.15.28:6819","nonce":1642308925}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":1642308925},{"type":"v1","addr":"172.21.15.28:6821","nonce":1642308925}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1642308925},{"type":"v1","addr":"172.21.15.28:6825","nonce":1642308925}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":1642308925},{"type":"v1","addr":"172.21.15.28:6823","nonce":1642308925}]},"public_addr":"172.21.15.28:6819/1642308925","cluster_addr":"172.21.15.28:6821/1642308925","heartbeat_back_addr":"172.21.15.28:6825/1642308925","heartbeat_front_addr":"172.21.15.28:6823/1642308925","state":["exists","up"]},{"osd":7,"uuid":"ca7a059f-0414-438e-b2d1-a10f348036be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":726062711},{"type":"v1","addr":"172.21.15.86:6817","nonce":726062711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":726062711},{"type":"v1","addr":"172.21.15.86:6819","nonce":726062711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":726062711},{"type":"v1","addr":"172.21.15.86:6823","nonce":726062711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":726062711},{"type":"v1","addr":"172.21.15.86:6821","nonce":726062711}]},"public_addr":"172.21.15.86:6817/726062711","cluster_addr":"172.21.15.86:6819/726062711","heartbeat_back_addr":"172.21.15.86:6823/726062711","heartbeat_front_addr":"172.21.15.86:6821/726062711","state":["exists","up"]},{"osd":8,"uuid":"92e3f41f-aebd-422e-a99e-555f7ba4aa53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":3007687277},{"type":"v1","addr":"172.21.15.186:6817","nonce":3007687277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":3007687277},{"type":"v1","addr":"172.21.15.186:6819","nonce":3007687277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":3007687277},{"type":"v1","addr":"172.21.15.186:6823","nonce":3007687277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":3007687277},{"type":"v1","addr":"172.21.15.186:6821","nonce":3007687277}]},"public_addr":"172.21.15.186:6817/3007687277","cluster_addr":"172.21.15.186:6819/3007687277","heartbeat_back_addr":"172.21.15.186:6823/3007687277","heartbeat_front_addr":"172.21.15.186:6821/3007687277","state":["exists","up"]},{"osd":9,"uuid":"38ff7d9d-ee85-404c-8c59-adc327e1ccfa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":136220472},{"type":"v1","addr":"172.21.15.28:6827","nonce":136220472}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":136220472},{"type":"v1","addr":"172.21.15.28:6829","nonce":136220472}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":136220472},{"type":"v1","addr":"172.21.15.28:6833","nonce":136220472}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":136220472},{"type":"v1","addr":"172.21.15.28:6831","nonce":136220472}]},"public_addr":"172.21.15.28:6827/136220472","cluster_addr":"172.21.15.28:6829/136220472","heartbeat_back_addr":"172.21.15.28:6833/136220472","heartbeat_front_addr":"172.21.15.28:6831/136220472","state":["exists","up"]},{"osd":10,"uuid":"5e9ff657-afa8-4bab-8491-f209038392b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1189968310},{"type":"v1","addr":"172.21.15.86:6825","nonce":1189968310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":1189968310},{"type":"v1","addr":"172.21.15.86:6827","nonce":1189968310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":1189968310},{"type":"v1","addr":"172.21.15.86:6831","nonce":1189968310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":1189968310},{"type":"v1","addr":"172.21.15.86:6829","nonce":1189968310}]},"public_addr":"172.21.15.86:6825/1189968310","cluster_addr":"172.21.15.86:6827/1189968310","heartbeat_back_addr":"172.21.15.86:6831/1189968310","heartbeat_front_addr":"172.21.15.86:6829/1189968310","state":["exists","up"]},{"osd":11,"uuid":"f112e3c6-65b6-4253-a966-62001db7d069","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3801668110},{"type":"v1","addr":"172.21.15.186:6825","nonce":3801668110}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3801668110},{"type":"v1","addr":"172.21.15.186:6827","nonce":3801668110}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3801668110},{"type":"v1","addr":"172.21.15.186:6831","nonce":3801668110}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3801668110},{"type":"v1","addr":"172.21.15.186:6829","nonce":3801668110}]},"public_addr":"172.21.15.186:6825/3801668110","cluster_addr":"172.21.15.186:6827/3801668110","heartbeat_back_addr":"172.21.15.186:6831/3801668110","heartbeat_front_addr":"172.21.15.186:6829/3801668110","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:00:40.483623+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:01:05.284385+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:01:28.645716+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:02:01.295092+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:02:26.486601+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:02:50.807478+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:03:24.756169+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:03:49.975173+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:04:13.772233+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:04:46.758555+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:05:12.331685+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T22:05:36.462548+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":2,"to":1}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.25","mappings":[{"from":5,"to":9}]},{"pgid":"3.27","mappings":[{"from":4,"to":8}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/2580087728":"2024-05-13T22:08:58.638536+0000","172.21.15.28:6801/3123328651":"2024-05-13T22:08:58.638536+0000","172.21.15.28:0/820158394":"2024-05-13T22:08:58.638536+0000","172.21.15.28:0/1978033727":"2024-05-13T22:08:58.638536+0000","172.21.15.28:0/1177009533":"2024-05-13T21:58:45.550459+0000","172.21.15.28:6800/3760499540":"2024-05-13T21:58:45.550459+0000","172.21.15.28:6801/3783742728":"2024-05-13T21:58:10.421564+0000","172.21.15.28:6800/3783742728":"2024-05-13T21:58:10.421564+0000","172.21.15.28:0/2229156260":"2024-05-13T21:58:45.550459+0000","172.21.15.28:0/1202140963":"2024-05-13T21:58:10.421564+0000","172.21.15.28:6800/3123328651":"2024-05-13T22:08:58.638536+0000","172.21.15.28:0/2924223784":"2024-05-13T21:58:45.550459+0000","172.21.15.28:6801/3760499540":"2024-05-13T21:58:45.550459+0000","172.21.15.28:0/1240027206":"2024-05-13T21:58:10.421564+0000","172.21.15.28:0/1581755795":"2024-05-13T22:08:58.638536+0000","172.21.15.28:0/171417728":"2024-05-13T21:58:10.421564+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-12T22:20:25.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:25.063+0000 7f7168a00640 1 -- 172.21.15.28:0/3653622450 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f71480a2d80 msgr2=0x7f71480a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:20:25.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:25.063+0000 7f7168a00640 1 --2- 172.21.15.28:0/3653622450 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f71480a2d80 0x7f71480a5240 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f71640ffa40 tx=0x7f715000b480 comp rx=0 tx=0).stop 2024-05-12T22:20:25.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:25.063+0000 7f7168a00640 1 -- 172.21.15.28:0/3653622450 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7164100700 msgr2=0x7f71640fdfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:20:25.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:25.063+0000 7f7168a00640 1 --2- 172.21.15.28:0/3653622450 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7164100700 0x7f71640fdfb0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f7154005940 tx=0x7f71540042f0 comp rx=0 tx=0).stop 2024-05-12T22:20:25.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:25.063+0000 7f7168a00640 1 -- 172.21.15.28:0/3653622450 shutdown_connections 2024-05-12T22:20:25.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:25.063+0000 7f7168a00640 1 --2- 172.21.15.28:0/3653622450 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f71480a2d80 0x7f71480a5240 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:20:25.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:25.063+0000 7f7168a00640 1 --2- 172.21.15.28:0/3653622450 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7164106710 0x7f71640fea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:20:25.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:25.063+0000 7f7168a00640 1 --2- 172.21.15.28:0/3653622450 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f71641010c0 0x7f71640fe4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:20:25.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:25.063+0000 7f7168a00640 1 --2- 172.21.15.28:0/3653622450 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7164100700 0x7f71640fdfb0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:20:25.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:25.063+0000 7f7168a00640 1 -- 172.21.15.28:0/3653622450 >> 172.21.15.28:0/3653622450 conn(0x7f7164076ab0 msgr2=0x7f716406a1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:20:25.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:25.063+0000 7f7168a00640 1 -- 172.21.15.28:0/3653622450 shutdown_connections 2024-05-12T22:20:25.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:25.064+0000 7f7168a00640 1 -- 172.21.15.28:0/3653622450 wait complete. 2024-05-12T22:20:25.085 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-05-12T22:20:25.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:25.291+0000 7f392be00640 1 -- 172.21.15.28:0/2412940593 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f392c107dd0 msgr2=0x7f392c108230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:20:25.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:25.291+0000 7f392be00640 1 --2- 172.21.15.28:0/2412940593 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f392c107dd0 0x7f392c108230 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f3914009900 tx=0x7f391402a490 comp rx=0 tx=0).stop 2024-05-12T22:20:25.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:25.292+0000 7f392be00640 1 -- 172.21.15.28:0/2412940593 shutdown_connections 2024-05-12T22:20:25.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:25.292+0000 7f392be00640 1 --2- 172.21.15.28:0/2412940593 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f392c108770 0x7f392c114b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:20:25.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:25.292+0000 7f392be00640 1 --2- 172.21.15.28:0/2412940593 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f392c107dd0 0x7f392c108230 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:20:25.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:25.292+0000 7f392be00640 1 --2- 172.21.15.28:0/2412940593 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f392c10ddf0 0x7f392c10e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:20:25.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:25.292+0000 7f392be00640 1 -- 172.21.15.28:0/2412940593 >> 172.21.15.28:0/2412940593 conn(0x7f392c077650 msgr2=0x7f392c106650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:20:25.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:25.292+0000 7f392be00640 1 -- 172.21.15.28:0/2412940593 shutdown_connections 2024-05-12T22:20:25.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:25.292+0000 7f392be00640 1 -- 172.21.15.28:0/2412940593 wait complete. 2024-05-12T22:20:25.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:25.293+0000 7f392be00640 1 Processor -- start 2024-05-12T22:20:25.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:25.293+0000 7f392be00640 1 -- start start 2024-05-12T22:20:25.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:25.293+0000 7f392be00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f392c107dd0 0x7f392c19e490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:20:25.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:25.293+0000 7f392be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f392c108770 0x7f392c19e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:20:25.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:25.293+0000 7f392be00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f392c10ddf0 0x7f392c1a2d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:20:25.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:25.293+0000 7f392be00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f392c1180b0 con 0x7f392c108770 2024-05-12T22:20:25.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:25.293+0000 7f392be00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f392c117f30 con 0x7f392c107dd0 2024-05-12T22:20:25.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:25.293+0000 7f392be00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f392c118230 con 0x7f392c10ddf0 2024-05-12T22:20:25.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:25.293+0000 7f3922000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f392c108770 0x7f392c19e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:20:25.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:25.293+0000 7f392aa00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f392c107dd0 0x7f392c19e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:20:25.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:25.294+0000 7f3922000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f392c108770 0x7f392c19e9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60288/0 (socket says 172.21.15.28:60288) 2024-05-12T22:20:25.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:25.294+0000 7f392aa00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f392c107dd0 0x7f392c19e490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:54804/0 (socket says 172.21.15.28:54804) 2024-05-12T22:20:25.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:25.294+0000 7f3922000640 1 -- 172.21.15.28:0/1358852549 learned_addr learned my addr 172.21.15.28:0/1358852549 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:20:25.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:25.294+0000 7f392b400640 1 --2- 172.21.15.28:0/1358852549 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f392c10ddf0 0x7f392c1a2d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:20:25.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:25.294+0000 7f392aa00640 1 -- 172.21.15.28:0/1358852549 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f392c10ddf0 msgr2=0x7f392c1a2d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:20:25.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:25.294+0000 7f392aa00640 1 --2- 172.21.15.28:0/1358852549 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f392c10ddf0 0x7f392c1a2d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:20:25.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:25.294+0000 7f392aa00640 1 -- 172.21.15.28:0/1358852549 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f392c108770 msgr2=0x7f392c19e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:20:25.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:25.294+0000 7f392aa00640 1 --2- 172.21.15.28:0/1358852549 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f392c108770 0x7f392c19e9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:20:25.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:25.294+0000 7f392aa00640 1 -- 172.21.15.28:0/1358852549 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f392c1a3480 con 0x7f392c107dd0 2024-05-12T22:20:25.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:25.294+0000 7f3922000640 1 --2- 172.21.15.28:0/1358852549 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f392c108770 0x7f392c19e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:20:25.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:25.294+0000 7f392aa00640 1 --2- 172.21.15.28:0/1358852549 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f392c107dd0 0x7f392c19e490 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f391c005fd0 tx=0x7f391c00dde0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:20:25.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:25.295+0000 7f3923e00640 1 -- 172.21.15.28:0/1358852549 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f391c009c30 con 0x7f392c107dd0 2024-05-12T22:20:25.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:25.295+0000 7f3923e00640 1 -- 172.21.15.28:0/1358852549 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f391c009dd0 con 0x7f392c107dd0 2024-05-12T22:20:25.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:25.295+0000 7f3923e00640 1 -- 172.21.15.28:0/1358852549 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f391c010460 con 0x7f392c107dd0 2024-05-12T22:20:25.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:25.295+0000 7f392be00640 1 -- 172.21.15.28:0/1358852549 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f392c1a3770 con 0x7f392c107dd0 2024-05-12T22:20:25.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:25.295+0000 7f392be00640 1 -- 172.21.15.28:0/1358852549 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f392c1ab000 con 0x7f392c107dd0 2024-05-12T22:20:25.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:25.296+0000 7f392be00640 1 -- 172.21.15.28:0/1358852549 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f392c0689c0 con 0x7f392c107dd0 2024-05-12T22:20:25.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:25.298+0000 7f3923e00640 1 -- 172.21.15.28:0/1358852549 <== mon.2 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7f391c003f50 con 0x7f392c107dd0 2024-05-12T22:20:25.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:25.300+0000 7f3923e00640 1 --2- 172.21.15.28:0/1358852549 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f39100a2d00 0x7f39100a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:20:25.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:25.300+0000 7f3923e00640 1 -- 172.21.15.28:0/1358852549 <== mon.2 v2:172.21.15.86:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8513+0+0 (secure 0 0 0) 0x7f391c0cf340 con 0x7f392c107dd0 2024-05-12T22:20:25.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:25.300+0000 7f3922000640 1 --2- 172.21.15.28:0/1358852549 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f39100a2d00 0x7f39100a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:20:25.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:25.301+0000 7f3922000640 1 --2- 172.21.15.28:0/1358852549 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f39100a2d00 0x7f39100a51c0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f3914031550 tx=0x7f3914005a20 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:20:25.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:25.302+0000 7f3923e00640 1 -- 172.21.15.28:0/1358852549 <== mon.2 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f391c017020 con 0x7f392c107dd0 2024-05-12T22:20:25.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:25.551+0000 7f392be00640 1 -- 172.21.15.28:0/1358852549 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f392c1101e0 con 0x7f392c107dd0 2024-05-12T22:20:25.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:25.563+0000 7f3923e00640 1 -- 172.21.15.28:0/1358852549 <== mon.2 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v27) ==== 161+0+0 (secure 0 0 0) 0x7f391c09b8b0 con 0x7f392c107dd0 2024-05-12T22:20:25.566 INFO:teuthology.orchestra.run.smithi028.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-05-12T22:20:25.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:25.569+0000 7f392be00640 1 -- 172.21.15.28:0/1358852549 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f39100a2d00 msgr2=0x7f39100a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:20:25.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:25.569+0000 7f392be00640 1 --2- 172.21.15.28:0/1358852549 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f39100a2d00 0x7f39100a51c0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f3914031550 tx=0x7f3914005a20 comp rx=0 tx=0).stop 2024-05-12T22:20:25.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:25.569+0000 7f392be00640 1 -- 172.21.15.28:0/1358852549 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f392c107dd0 msgr2=0x7f392c19e490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:20:25.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:25.569+0000 7f392be00640 1 --2- 172.21.15.28:0/1358852549 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f392c107dd0 0x7f392c19e490 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f391c005fd0 tx=0x7f391c00dde0 comp rx=0 tx=0).stop 2024-05-12T22:20:25.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:25.569+0000 7f392be00640 1 -- 172.21.15.28:0/1358852549 shutdown_connections 2024-05-12T22:20:25.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:25.569+0000 7f392be00640 1 --2- 172.21.15.28:0/1358852549 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f39100a2d00 0x7f39100a51c0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:20:25.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:25.569+0000 7f392be00640 1 --2- 172.21.15.28:0/1358852549 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f392c10ddf0 0x7f392c1a2d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:20:25.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:25.569+0000 7f392be00640 1 --2- 172.21.15.28:0/1358852549 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f392c108770 0x7f392c19e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:20:25.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:25.569+0000 7f392be00640 1 --2- 172.21.15.28:0/1358852549 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f392c107dd0 0x7f392c19e490 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:20:25.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:25.569+0000 7f392be00640 1 -- 172.21.15.28:0/1358852549 >> 172.21.15.28:0/1358852549 conn(0x7f392c077650 msgr2=0x7f392c078580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:20:25.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:25.569+0000 7f392be00640 1 -- 172.21.15.28:0/1358852549 shutdown_connections 2024-05-12T22:20:25.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:25.569+0000 7f392be00640 1 -- 172.21.15.28:0/1358852549 wait complete. 2024-05-12T22:20:25.591 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-05-12T22:20:25.645 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:20:25 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/2046808506' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:20:25.645 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:20:25 smithi028 ceph-mon[29847]: pgmap v356: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 840 B/s wr, 9 op/s 2024-05-12T22:20:25.646 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:20:25 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/3653622450' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-12T22:20:25.646 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:20:25 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mon-a[29843]: 2024-05-12T22:20:25.554+0000 7ff5c0e00700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-12T22:20:25.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:20:25 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/2046808506' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:20:25.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:20:25 smithi186 ceph-mon[34785]: pgmap v356: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 840 B/s wr, 9 op/s 2024-05-12T22:20:25.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:20:25 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/3653622450' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-12T22:20:25.680 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 12 22:20:25 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-c[65626]: ignoring --setuser ceph since I am not root 2024-05-12T22:20:25.680 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 12 22:20:25 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-c[65626]: ignoring --setgroup ceph since I am not root 2024-05-12T22:20:25.680 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 12 22:20:25 smithi186 ceph-mds[65630]: -- 172.21.15.186:0/3475693025 <== mon.1 v2:172.21.15.186:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55bdb6729a40 con 0x55bdb66fc800 2024-05-12T22:20:25.680 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 12 22:20:25 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-c[65626]: starting mds.c at 2024-05-12T22:20:25.681 INFO:journalctl@ceph.mds.i.smithi186.stdout:May 12 22:20:25 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-i[65930]: ignoring --setuser ceph since I am not root 2024-05-12T22:20:25.681 INFO:journalctl@ceph.mds.i.smithi186.stdout:May 12 22:20:25 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-i[65930]: ignoring --setgroup ceph since I am not root 2024-05-12T22:20:25.681 INFO:journalctl@ceph.mds.i.smithi186.stdout:May 12 22:20:25 smithi186 ceph-mds[65934]: -- 172.21.15.186:0/686669517 <== mon.1 v2:172.21.15.186:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55689e7c1a40 con 0x55689e794c00 2024-05-12T22:20:25.681 INFO:journalctl@ceph.mds.i.smithi186.stdout:May 12 22:20:25 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-i[65930]: starting mds.i at 2024-05-12T22:20:25.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:20:25 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/2046808506' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-12T22:20:25.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:20:25 smithi086 ceph-mon[34297]: pgmap v356: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 840 B/s wr, 9 op/s 2024-05-12T22:20:25.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:20:25 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/3653622450' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-12T22:20:25.743 INFO:journalctl@ceph.mds.b.smithi086.stdout:May 12 22:20:25 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-b[64802]: ignoring --setuser ceph since I am not root 2024-05-12T22:20:25.743 INFO:journalctl@ceph.mds.b.smithi086.stdout:May 12 22:20:25 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-b[64802]: ignoring --setgroup ceph since I am not root 2024-05-12T22:20:25.743 INFO:journalctl@ceph.mds.b.smithi086.stdout:May 12 22:20:25 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-b[64802]: starting mds.b at 2024-05-12T22:20:25.743 INFO:journalctl@ceph.mds.h.smithi086.stdout:May 12 22:20:25 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-h[64496]: ignoring --setuser ceph since I am not root 2024-05-12T22:20:25.744 INFO:journalctl@ceph.mds.h.smithi086.stdout:May 12 22:20:25 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-h[64496]: ignoring --setgroup ceph since I am not root 2024-05-12T22:20:25.744 INFO:journalctl@ceph.mds.h.smithi086.stdout:May 12 22:20:25 smithi086 ceph-mds[64500]: -- 172.21.15.86:0/1247065088 <== mon.2 v2:172.21.15.86:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x557055683a40 con 0x557055657000 2024-05-12T22:20:25.744 INFO:journalctl@ceph.mds.h.smithi086.stdout:May 12 22:20:25 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-h[64496]: starting mds.h at 2024-05-12T22:20:25.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:25.788+0000 7f7f2dc00640 1 -- 172.21.15.28:0/4052089286 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7f28076680 msgr2=0x7f7f28112030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:20:25.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:25.788+0000 7f7f2dc00640 1 --2- 172.21.15.28:0/4052089286 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7f28076680 0x7f7f28112030 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f7f14009cd0 tx=0x7f7f1402a8f0 comp rx=0 tx=0).stop 2024-05-12T22:20:25.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:25.788+0000 7f7f2dc00640 1 -- 172.21.15.28:0/4052089286 shutdown_connections 2024-05-12T22:20:25.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:25.788+0000 7f7f2dc00640 1 --2- 172.21.15.28:0/4052089286 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7f28112570 0x7f7f28114960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:20:25.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:25.788+0000 7f7f2dc00640 1 --2- 172.21.15.28:0/4052089286 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7f28076680 0x7f7f28112030 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:20:25.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:25.788+0000 7f7f2dc00640 1 --2- 172.21.15.28:0/4052089286 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f28075d60 0x7f7f28076140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:20:25.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:25.788+0000 7f7f2dc00640 1 -- 172.21.15.28:0/4052089286 >> 172.21.15.28:0/4052089286 conn(0x7f7f280ff3a0 msgr2=0x7f7f281017c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:20:25.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:25.788+0000 7f7f2dc00640 1 -- 172.21.15.28:0/4052089286 shutdown_connections 2024-05-12T22:20:25.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:25.789+0000 7f7f2dc00640 1 -- 172.21.15.28:0/4052089286 wait complete. 2024-05-12T22:20:25.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:25.790+0000 7f7f2dc00640 1 Processor -- start 2024-05-12T22:20:25.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:25.790+0000 7f7f2dc00640 1 -- start start 2024-05-12T22:20:25.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:25.790+0000 7f7f2dc00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7f28075d60 0x7f7f2819e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:20:25.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:25.790+0000 7f7f2dc00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7f28076680 0x7f7f2819ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:20:25.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:25.790+0000 7f7f2dc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f28112570 0x7f7f281a2f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:20:25.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:25.790+0000 7f7f2dc00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7f281180d0 con 0x7f7f28112570 2024-05-12T22:20:25.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:25.790+0000 7f7f2dc00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f7f28117f50 con 0x7f7f28075d60 2024-05-12T22:20:25.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:25.790+0000 7f7f2dc00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f7f28118250 con 0x7f7f28076680 2024-05-12T22:20:25.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:25.791+0000 7f7f27e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7f28075d60 0x7f7f2819e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:20:25.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:25.791+0000 7f7f27e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7f28075d60 0x7f7f2819e700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.28:54858/0 (socket says 172.21.15.28:54858) 2024-05-12T22:20:25.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:25.791+0000 7f7f27e00640 1 -- 172.21.15.28:0/3100554677 learned_addr learned my addr 172.21.15.28:0/3100554677 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:20:25.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:25.791+0000 7f7f2d200640 1 --2- 172.21.15.28:0/3100554677 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f28112570 0x7f7f281a2f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:20:25.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:25.791+0000 7f7f27400640 1 --2- 172.21.15.28:0/3100554677 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7f28076680 0x7f7f2819ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:20:25.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:25.791+0000 7f7f27e00640 1 -- 172.21.15.28:0/3100554677 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7f28076680 msgr2=0x7f7f2819ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:20:25.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:25.791+0000 7f7f27e00640 1 --2- 172.21.15.28:0/3100554677 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7f28076680 0x7f7f2819ec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:20:25.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:25.791+0000 7f7f27e00640 1 -- 172.21.15.28:0/3100554677 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f28112570 msgr2=0x7f7f281a2f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:20:25.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:25.791+0000 7f7f27e00640 1 --2- 172.21.15.28:0/3100554677 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f28112570 0x7f7f281a2f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:20:25.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:25.791+0000 7f7f27e00640 1 -- 172.21.15.28:0/3100554677 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f281a36a0 con 0x7f7f28075d60 2024-05-12T22:20:25.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:25.791+0000 7f7f27400640 1 --2- 172.21.15.28:0/3100554677 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7f28076680 0x7f7f2819ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-12T22:20:25.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:25.791+0000 7f7f27e00640 1 --2- 172.21.15.28:0/3100554677 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7f28075d60 0x7f7f2819e700 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f7f1800d950 tx=0x7f7f1800de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:20:25.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:25.792+0000 7f7f24c00640 1 -- 172.21.15.28:0/3100554677 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f18017070 con 0x7f7f28075d60 2024-05-12T22:20:25.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:25.792+0000 7f7f24c00640 1 -- 172.21.15.28:0/3100554677 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f18009c30 con 0x7f7f28075d60 2024-05-12T22:20:25.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:25.792+0000 7f7f2dc00640 1 -- 172.21.15.28:0/3100554677 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f281a3990 con 0x7f7f28075d60 2024-05-12T22:20:25.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:25.792+0000 7f7f24c00640 1 -- 172.21.15.28:0/3100554677 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f1800f6b0 con 0x7f7f28075d60 2024-05-12T22:20:25.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:25.792+0000 7f7f2dc00640 1 -- 172.21.15.28:0/3100554677 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f281ab220 con 0x7f7f28075d60 2024-05-12T22:20:25.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:25.793+0000 7f7f2dc00640 1 -- 172.21.15.28:0/3100554677 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f280689c0 con 0x7f7f28075d60 2024-05-12T22:20:25.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:25.795+0000 7f7f24c00640 1 -- 172.21.15.28:0/3100554677 <== mon.2 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7f7f180109d0 con 0x7f7f28075d60 2024-05-12T22:20:25.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:25.796+0000 7f7f24c00640 1 --2- 172.21.15.28:0/3100554677 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f7f0c0a2d80 0x7f7f0c0a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:20:25.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:25.796+0000 7f7f24c00640 1 -- 172.21.15.28:0/3100554677 <== mon.2 v2:172.21.15.86:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 8727+0+0 (secure 0 0 0) 0x7f7f180cf110 con 0x7f7f28075d60 2024-05-12T22:20:25.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:25.797+0000 7f7f27400640 1 --2- 172.21.15.28:0/3100554677 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f7f0c0a2d80 0x7f7f0c0a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:20:25.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:25.797+0000 7f7f27400640 1 --2- 172.21.15.28:0/3100554677 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f7f0c0a2d80 0x7f7f0c0a5240 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f7f280789f0 tx=0x7f7f14005ef0 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:20:25.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:25.800+0000 7f7f24c00640 1 -- 172.21.15.28:0/3100554677 <== mon.2 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f7f18009de0 con 0x7f7f28075d60 2024-05-12T22:20:26.036 INFO:journalctl@ceph.mds.g.smithi028.stdout:May 12 22:20:25 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-g[67758]: ignoring --setuser ceph since I am not root 2024-05-12T22:20:26.036 INFO:journalctl@ceph.mds.g.smithi028.stdout:May 12 22:20:25 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-g[67758]: ignoring --setgroup ceph since I am not root 2024-05-12T22:20:26.036 INFO:journalctl@ceph.mds.g.smithi028.stdout:May 12 22:20:25 smithi028 ceph-mds[67762]: -- 172.21.15.28:0/4259908565 <== mon.2 v2:172.21.15.86:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x555635d6fa40 con 0x555635d42800 2024-05-12T22:20:26.036 INFO:journalctl@ceph.mds.g.smithi028.stdout:May 12 22:20:25 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-g[67758]: starting mds.g at 2024-05-12T22:20:26.037 INFO:journalctl@ceph.mds.a.smithi028.stdout:May 12 22:20:25 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-a[68142]: ignoring --setuser ceph since I am not root 2024-05-12T22:20:26.037 INFO:journalctl@ceph.mds.a.smithi028.stdout:May 12 22:20:25 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-a[68142]: ignoring --setgroup ceph since I am not root 2024-05-12T22:20:26.037 INFO:journalctl@ceph.mds.a.smithi028.stdout:May 12 22:20:25 smithi028 ceph-mds[68146]: -- 172.21.15.28:0/3986605690 <== mon.2 v2:172.21.15.86:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x558c0c47ba40 con 0x558c0c44f000 2024-05-12T22:20:26.037 INFO:journalctl@ceph.mds.a.smithi028.stdout:May 12 22:20:25 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-a[68142]: starting mds.a at 2024-05-12T22:20:26.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:26.059+0000 7f7f2dc00640 1 -- 172.21.15.28:0/3100554677 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f7f2810d700 con 0x7f7f28075d60 2024-05-12T22:20:26.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:26.563+0000 7f7f24c00640 1 -- 172.21.15.28:0/3100554677 <== mon.2 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v28) ==== 104+0+0 (secure 0 0 0) 0x7f7f1809bc00 con 0x7f7f28075d60 2024-05-12T22:20:26.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:26.568+0000 7f7f2dc00640 1 -- 172.21.15.28:0/3100554677 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f7f0c0a2d80 msgr2=0x7f7f0c0a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:20:26.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:26.568+0000 7f7f2dc00640 1 --2- 172.21.15.28:0/3100554677 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f7f0c0a2d80 0x7f7f0c0a5240 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f7f280789f0 tx=0x7f7f14005ef0 comp rx=0 tx=0).stop 2024-05-12T22:20:26.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:26.568+0000 7f7f2dc00640 1 -- 172.21.15.28:0/3100554677 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7f28075d60 msgr2=0x7f7f2819e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:20:26.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:26.568+0000 7f7f2dc00640 1 --2- 172.21.15.28:0/3100554677 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7f28075d60 0x7f7f2819e700 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f7f1800d950 tx=0x7f7f1800de10 comp rx=0 tx=0).stop 2024-05-12T22:20:26.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:26.568+0000 7f7f2dc00640 1 -- 172.21.15.28:0/3100554677 shutdown_connections 2024-05-12T22:20:26.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:26.568+0000 7f7f2dc00640 1 --2- 172.21.15.28:0/3100554677 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f7f0c0a2d80 0x7f7f0c0a5240 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:20:26.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:26.568+0000 7f7f2dc00640 1 --2- 172.21.15.28:0/3100554677 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f28112570 0x7f7f281a2f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:20:26.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:26.568+0000 7f7f2dc00640 1 --2- 172.21.15.28:0/3100554677 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7f28076680 0x7f7f2819ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:20:26.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:26.568+0000 7f7f2dc00640 1 --2- 172.21.15.28:0/3100554677 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7f28075d60 0x7f7f2819e700 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:20:26.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:26.568+0000 7f7f2dc00640 1 -- 172.21.15.28:0/3100554677 >> 172.21.15.28:0/3100554677 conn(0x7f7f280ff3a0 msgr2=0x7f7f28101790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:20:26.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:26.569+0000 7f7f2dc00640 1 -- 172.21.15.28:0/3100554677 shutdown_connections 2024-05-12T22:20:26.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:26.569+0000 7f7f2dc00640 1 -- 172.21.15.28:0/3100554677 wait complete. 2024-05-12T22:20:26.593 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-05-12T22:20:26.663 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:20:26 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/1358852549' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-05-12T22:20:26.663 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:20:26 smithi028 ceph-mon[29847]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-05-12T22:20:26.664 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:20:26 smithi028 ceph-mon[29847]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-05-12T22:20:26.664 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:20:26 smithi028 ceph-mon[29847]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-12T22:20:26.664 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:20:26 smithi028 ceph-mon[29847]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-05-12T22:20:26.664 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:20:26 smithi028 ceph-mon[29847]: osdmap e100: 12 total, 12 up, 12 in 2024-05-12T22:20:26.664 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:20:26 smithi028 ceph-mon[29847]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-05-12T22:20:26.664 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:20:26 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/3100554677' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-05-12T22:20:26.664 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:20:26 smithi028 ceph-mon[29847]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-05-12T22:20:26.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:20:26 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/1358852549' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-05-12T22:20:26.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:20:26 smithi186 ceph-mon[34785]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-05-12T22:20:26.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:20:26 smithi186 ceph-mon[34785]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-05-12T22:20:26.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:20:26 smithi186 ceph-mon[34785]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-12T22:20:26.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:20:26 smithi186 ceph-mon[34785]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-05-12T22:20:26.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:20:26 smithi186 ceph-mon[34785]: osdmap e100: 12 total, 12 up, 12 in 2024-05-12T22:20:26.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:20:26 smithi186 ceph-mon[34785]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-05-12T22:20:26.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:20:26 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/3100554677' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-05-12T22:20:26.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:20:26 smithi186 ceph-mon[34785]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-05-12T22:20:26.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:20:26 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/1358852549' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-05-12T22:20:26.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:20:26 smithi086 ceph-mon[34297]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-05-12T22:20:26.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:20:26 smithi086 ceph-mon[34297]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-05-12T22:20:26.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:20:26 smithi086 ceph-mon[34297]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-12T22:20:26.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:20:26 smithi086 ceph-mon[34297]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-05-12T22:20:26.743 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:20:26 smithi086 ceph-mon[34297]: osdmap e100: 12 total, 12 up, 12 in 2024-05-12T22:20:26.743 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:20:26 smithi086 ceph-mon[34297]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-05-12T22:20:26.743 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:20:26 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/3100554677' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-05-12T22:20:26.743 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:20:26 smithi086 ceph-mon[34297]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-05-12T22:20:26.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:26.806+0000 7f9d9e800640 1 -- 172.21.15.28:0/2582083315 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9d981038e0 msgr2=0x7f9d98103d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:20:26.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:26.806+0000 7f9d9e800640 1 --2- 172.21.15.28:0/2582083315 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9d981038e0 0x7f9d98103d60 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f9d84009860 tx=0x7f9d8402a340 comp rx=0 tx=0).stop 2024-05-12T22:20:26.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:26.806+0000 7f9d9e800640 1 -- 172.21.15.28:0/2582083315 shutdown_connections 2024-05-12T22:20:26.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:26.806+0000 7f9d9e800640 1 --2- 172.21.15.28:0/2582083315 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d981042a0 0x7f9d981107b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:20:26.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:26.806+0000 7f9d9e800640 1 --2- 172.21.15.28:0/2582083315 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9d981038e0 0x7f9d98103d60 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:20:26.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:26.806+0000 7f9d9e800640 1 --2- 172.21.15.28:0/2582083315 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9d981098f0 0x7f9d98109cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:20:26.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:26.806+0000 7f9d9e800640 1 -- 172.21.15.28:0/2582083315 >> 172.21.15.28:0/2582083315 conn(0x7f9d980ff370 msgr2=0x7f9d98101790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:20:26.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:26.807+0000 7f9d9e800640 1 -- 172.21.15.28:0/2582083315 shutdown_connections 2024-05-12T22:20:26.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:26.807+0000 7f9d9e800640 1 -- 172.21.15.28:0/2582083315 wait complete. 2024-05-12T22:20:26.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:26.808+0000 7f9d9e800640 1 Processor -- start 2024-05-12T22:20:26.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:26.808+0000 7f9d9e800640 1 -- start start 2024-05-12T22:20:26.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:26.808+0000 7f9d9e800640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9d981038e0 0x7f9d9819a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:20:26.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:26.808+0000 7f9d9e800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9d981042a0 0x7f9d9819a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:20:26.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:26.808+0000 7f9d9e800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d981098f0 0x7f9d9819eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:20:26.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:26.808+0000 7f9d9e800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9d98113d00 con 0x7f9d981098f0 2024-05-12T22:20:26.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:26.808+0000 7f9d9e800640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f9d98113b80 con 0x7f9d981042a0 2024-05-12T22:20:26.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:26.808+0000 7f9d9e800640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f9d98113e80 con 0x7f9d981038e0 2024-05-12T22:20:26.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:26.808+0000 7f9d9d400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9d981038e0 0x7f9d9819a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:20:26.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:26.808+0000 7f9d9de00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d981098f0 0x7f9d9819eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:20:26.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:26.809+0000 7f9d9d400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9d981038e0 0x7f9d9819a200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.28:36552/0 (socket says 172.21.15.28:36552) 2024-05-12T22:20:26.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:26.809+0000 7f9d9ca00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9d981042a0 0x7f9d9819a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:20:26.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:26.809+0000 7f9d9d400640 1 -- 172.21.15.28:0/1832524965 learned_addr learned my addr 172.21.15.28:0/1832524965 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:20:26.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:26.809+0000 7f9d9ca00640 1 -- 172.21.15.28:0/1832524965 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9d981038e0 msgr2=0x7f9d9819a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:20:26.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:26.809+0000 7f9d9ca00640 1 --2- 172.21.15.28:0/1832524965 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9d981038e0 0x7f9d9819a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:20:26.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:26.809+0000 7f9d9ca00640 1 -- 172.21.15.28:0/1832524965 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d981098f0 msgr2=0x7f9d9819eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:20:26.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:26.809+0000 7f9d9ca00640 1 --2- 172.21.15.28:0/1832524965 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d981098f0 0x7f9d9819eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:20:26.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:26.809+0000 7f9d9ca00640 1 -- 172.21.15.28:0/1832524965 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d9819f1f0 con 0x7f9d981042a0 2024-05-12T22:20:26.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:26.809+0000 7f9d9ca00640 1 --2- 172.21.15.28:0/1832524965 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9d981042a0 0x7f9d9819a740 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f9d8402a860 tx=0x7f9d84033b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:20:26.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:26.810+0000 7f9d96000640 1 -- 172.21.15.28:0/1832524965 <== mon.2 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d84037040 con 0x7f9d981042a0 2024-05-12T22:20:26.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:26.810+0000 7f9d96000640 1 -- 172.21.15.28:0/1832524965 <== mon.2 v2:172.21.15.86:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9d84033d50 con 0x7f9d981042a0 2024-05-12T22:20:26.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:26.810+0000 7f9d9e800640 1 -- 172.21.15.28:0/1832524965 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d9819f480 con 0x7f9d981042a0 2024-05-12T22:20:26.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:26.810+0000 7f9d96000640 1 -- 172.21.15.28:0/1832524965 <== mon.2 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d840039e0 con 0x7f9d981042a0 2024-05-12T22:20:26.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:26.810+0000 7f9d9e800640 1 -- 172.21.15.28:0/1832524965 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d981a6d70 con 0x7f9d981042a0 2024-05-12T22:20:26.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:26.811+0000 7f9d9e800640 1 -- 172.21.15.28:0/1832524965 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d98068570 con 0x7f9d981042a0 2024-05-12T22:20:26.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:26.813+0000 7f9d96000640 1 -- 172.21.15.28:0/1832524965 <== mon.2 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7f9d84042020 con 0x7f9d981042a0 2024-05-12T22:20:26.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:26.814+0000 7f9d96000640 1 --2- 172.21.15.28:0/1832524965 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f9d800a2d80 0x7f9d800a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:20:26.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:26.815+0000 7f9d96000640 1 -- 172.21.15.28:0/1832524965 <== mon.2 v2:172.21.15.86:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 8727+0+0 (secure 0 0 0) 0x7f9d840ee950 con 0x7f9d981042a0 2024-05-12T22:20:26.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:26.815+0000 7f9d9d400640 1 --2- 172.21.15.28:0/1832524965 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f9d800a2d80 0x7f9d800a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:20:26.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:26.815+0000 7f9d9d400640 1 --2- 172.21.15.28:0/1832524965 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f9d800a2d80 0x7f9d800a5240 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f9d8c0096c0 tx=0x7f9d8c009260 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:20:26.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:26.819+0000 7f9d96000640 1 -- 172.21.15.28:0/1832524965 <== mon.2 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f9d84049020 con 0x7f9d981042a0 2024-05-12T22:20:27.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:27.025+0000 7f9d9e800640 1 -- 172.21.15.28:0/1832524965 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f9d9819b550 con 0x7f9d981042a0 2024-05-12T22:20:27.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:27.574+0000 7f9d96000640 1 -- 172.21.15.28:0/1832524965 <== mon.2 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v101) ==== 181+0+0 (secure 0 0 0) 0x7f9d840bb550 con 0x7f9d981042a0 2024-05-12T22:20:27.575 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_metadata' removed 2024-05-12T22:20:27.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:27.579+0000 7f9d9e800640 1 -- 172.21.15.28:0/1832524965 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f9d800a2d80 msgr2=0x7f9d800a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:20:27.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:27.579+0000 7f9d9e800640 1 --2- 172.21.15.28:0/1832524965 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f9d800a2d80 0x7f9d800a5240 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f9d8c0096c0 tx=0x7f9d8c009260 comp rx=0 tx=0).stop 2024-05-12T22:20:27.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:27.579+0000 7f9d9e800640 1 -- 172.21.15.28:0/1832524965 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9d981042a0 msgr2=0x7f9d9819a740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:20:27.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:27.579+0000 7f9d9e800640 1 --2- 172.21.15.28:0/1832524965 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9d981042a0 0x7f9d9819a740 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f9d8402a860 tx=0x7f9d84033b60 comp rx=0 tx=0).stop 2024-05-12T22:20:27.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:27.580+0000 7f9d9e800640 1 -- 172.21.15.28:0/1832524965 shutdown_connections 2024-05-12T22:20:27.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:27.580+0000 7f9d9e800640 1 --2- 172.21.15.28:0/1832524965 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f9d800a2d80 0x7f9d800a5240 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:20:27.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:27.580+0000 7f9d9e800640 1 --2- 172.21.15.28:0/1832524965 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d981098f0 0x7f9d9819eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:20:27.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:27.580+0000 7f9d9e800640 1 --2- 172.21.15.28:0/1832524965 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9d981042a0 0x7f9d9819a740 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:20:27.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:27.580+0000 7f9d9e800640 1 --2- 172.21.15.28:0/1832524965 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9d981038e0 0x7f9d9819a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:20:27.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:27.580+0000 7f9d9e800640 1 -- 172.21.15.28:0/1832524965 >> 172.21.15.28:0/1832524965 conn(0x7f9d980ff370 msgr2=0x7f9d980ffb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:20:27.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:27.580+0000 7f9d9e800640 1 -- 172.21.15.28:0/1832524965 shutdown_connections 2024-05-12T22:20:27.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:27.580+0000 7f9d9e800640 1 -- 172.21.15.28:0/1832524965 wait complete. 2024-05-12T22:20:27.603 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-05-12T22:20:27.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:20:27 smithi028 ceph-mon[29847]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-05-12T22:20:27.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:20:27 smithi028 ceph-mon[29847]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-05-12T22:20:27.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:20:27 smithi028 ceph-mon[29847]: Cluster is now healthy 2024-05-12T22:20:27.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:20:27 smithi028 ceph-mon[29847]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-05-12T22:20:27.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:20:27 smithi028 ceph-mon[29847]: mds.? [v2:172.21.15.186:6837/3590753392,v1:172.21.15.186:6839/3590753392] up:boot 2024-05-12T22:20:27.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:20:27 smithi028 ceph-mon[29847]: mds.? [v2:172.21.15.186:6836/216167677,v1:172.21.15.186:6838/216167677] up:boot 2024-05-12T22:20:27.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:20:27 smithi028 ceph-mon[29847]: mds.? [v2:172.21.15.86:6836/2895199623,v1:172.21.15.86:6837/2895199623] up:boot 2024-05-12T22:20:27.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:20:27 smithi028 ceph-mon[29847]: mds.? [v2:172.21.15.86:6838/2748656364,v1:172.21.15.86:6839/2748656364] up:boot 2024-05-12T22:20:27.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:20:27 smithi028 ceph-mon[29847]: mds.? [v2:172.21.15.28:6838/1695951355,v1:172.21.15.28:6839/1695951355] up:boot 2024-05-12T22:20:27.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:20:27 smithi028 ceph-mon[29847]: mds.? [v2:172.21.15.28:6840/1098203132,v1:172.21.15.28:6841/1098203132] up:boot 2024-05-12T22:20:27.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:20:27 smithi028 ceph-mon[29847]: fsmap 12 up:standby 2024-05-12T22:20:27.672 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:20:27 smithi028 ceph-mon[29847]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-12T22:20:27.673 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:20:27 smithi028 ceph-mon[29847]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-12T22:20:27.673 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:20:27 smithi028 ceph-mon[29847]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-12T22:20:27.673 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:20:27 smithi028 ceph-mon[29847]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-12T22:20:27.673 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:20:27 smithi028 ceph-mon[29847]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-12T22:20:27.673 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:20:27 smithi028 ceph-mon[29847]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-12T22:20:27.673 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:20:27 smithi028 ceph-mon[29847]: pgmap v358: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 818 B/s wr, 7 op/s 2024-05-12T22:20:27.673 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:20:27 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/1832524965' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-12T22:20:27.673 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:20:27 smithi028 ceph-mon[29847]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-12T22:20:27.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:20:27 smithi186 ceph-mon[34785]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-05-12T22:20:27.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:20:27 smithi186 ceph-mon[34785]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-05-12T22:20:27.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:20:27 smithi186 ceph-mon[34785]: Cluster is now healthy 2024-05-12T22:20:27.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:20:27 smithi186 ceph-mon[34785]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-05-12T22:20:27.679 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:20:27 smithi186 ceph-mon[34785]: mds.? [v2:172.21.15.186:6837/3590753392,v1:172.21.15.186:6839/3590753392] up:boot 2024-05-12T22:20:27.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:20:27 smithi186 ceph-mon[34785]: mds.? [v2:172.21.15.186:6836/216167677,v1:172.21.15.186:6838/216167677] up:boot 2024-05-12T22:20:27.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:20:27 smithi186 ceph-mon[34785]: mds.? [v2:172.21.15.86:6836/2895199623,v1:172.21.15.86:6837/2895199623] up:boot 2024-05-12T22:20:27.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:20:27 smithi186 ceph-mon[34785]: mds.? [v2:172.21.15.86:6838/2748656364,v1:172.21.15.86:6839/2748656364] up:boot 2024-05-12T22:20:27.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:20:27 smithi186 ceph-mon[34785]: mds.? [v2:172.21.15.28:6838/1695951355,v1:172.21.15.28:6839/1695951355] up:boot 2024-05-12T22:20:27.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:20:27 smithi186 ceph-mon[34785]: mds.? [v2:172.21.15.28:6840/1098203132,v1:172.21.15.28:6841/1098203132] up:boot 2024-05-12T22:20:27.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:20:27 smithi186 ceph-mon[34785]: fsmap 12 up:standby 2024-05-12T22:20:27.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:20:27 smithi186 ceph-mon[34785]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-12T22:20:27.680 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:20:27 smithi186 ceph-mon[34785]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-12T22:20:27.681 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:20:27 smithi186 ceph-mon[34785]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-12T22:20:27.681 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:20:27 smithi186 ceph-mon[34785]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-12T22:20:27.681 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:20:27 smithi186 ceph-mon[34785]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-12T22:20:27.681 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:20:27 smithi186 ceph-mon[34785]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-12T22:20:27.681 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:20:27 smithi186 ceph-mon[34785]: pgmap v358: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 818 B/s wr, 7 op/s 2024-05-12T22:20:27.681 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:20:27 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/1832524965' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-12T22:20:27.681 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:20:27 smithi186 ceph-mon[34785]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-12T22:20:27.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:20:27 smithi086 ceph-mon[34297]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-05-12T22:20:27.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:20:27 smithi086 ceph-mon[34297]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-05-12T22:20:27.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:20:27 smithi086 ceph-mon[34297]: Cluster is now healthy 2024-05-12T22:20:27.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:20:27 smithi086 ceph-mon[34297]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-05-12T22:20:27.742 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:20:27 smithi086 ceph-mon[34297]: mds.? [v2:172.21.15.186:6837/3590753392,v1:172.21.15.186:6839/3590753392] up:boot 2024-05-12T22:20:27.743 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:20:27 smithi086 ceph-mon[34297]: mds.? [v2:172.21.15.186:6836/216167677,v1:172.21.15.186:6838/216167677] up:boot 2024-05-12T22:20:27.743 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:20:27 smithi086 ceph-mon[34297]: mds.? [v2:172.21.15.86:6836/2895199623,v1:172.21.15.86:6837/2895199623] up:boot 2024-05-12T22:20:27.743 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:20:27 smithi086 ceph-mon[34297]: mds.? [v2:172.21.15.86:6838/2748656364,v1:172.21.15.86:6839/2748656364] up:boot 2024-05-12T22:20:27.743 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:20:27 smithi086 ceph-mon[34297]: mds.? [v2:172.21.15.28:6838/1695951355,v1:172.21.15.28:6839/1695951355] up:boot 2024-05-12T22:20:27.743 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:20:27 smithi086 ceph-mon[34297]: mds.? [v2:172.21.15.28:6840/1098203132,v1:172.21.15.28:6841/1098203132] up:boot 2024-05-12T22:20:27.743 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:20:27 smithi086 ceph-mon[34297]: fsmap 12 up:standby 2024-05-12T22:20:27.743 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:20:27 smithi086 ceph-mon[34297]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-12T22:20:27.743 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:20:27 smithi086 ceph-mon[34297]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-12T22:20:27.743 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:20:27 smithi086 ceph-mon[34297]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-12T22:20:27.744 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:20:27 smithi086 ceph-mon[34297]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-12T22:20:27.744 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:20:27 smithi086 ceph-mon[34297]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-12T22:20:27.744 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:20:27 smithi086 ceph-mon[34297]: from='mgr.25189 172.21.15.28:0/2702788964' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-12T22:20:27.744 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:20:27 smithi086 ceph-mon[34297]: pgmap v358: 97 pgs: 97 active+clean; 91 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 818 B/s wr, 7 op/s 2024-05-12T22:20:27.744 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:20:27 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/1832524965' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-12T22:20:27.744 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:20:27 smithi086 ceph-mon[34297]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-12T22:20:27.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:27.808+0000 7f99d7e00640 1 -- 172.21.15.28:0/3342376968 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f99d8108850 msgr2=0x7f99d810c880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:20:27.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:27.808+0000 7f99d7e00640 1 --2- 172.21.15.28:0/3342376968 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f99d8108850 0x7f99d810c880 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f99c0009860 tx=0x7f99c002a340 comp rx=0 tx=0).stop 2024-05-12T22:20:27.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:27.809+0000 7f99d7e00640 1 -- 172.21.15.28:0/3342376968 shutdown_connections 2024-05-12T22:20:27.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:27.809+0000 7f99d7e00640 1 --2- 172.21.15.28:0/3342376968 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99d810cf60 0x7f99d8114b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:20:27.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:27.809+0000 7f99d7e00640 1 --2- 172.21.15.28:0/3342376968 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f99d8108850 0x7f99d810c880 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:20:27.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:27.809+0000 7f99d7e00640 1 --2- 172.21.15.28:0/3342376968 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f99d8107ea0 0x7f99d8108280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:20:27.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:27.809+0000 7f99d7e00640 1 -- 172.21.15.28:0/3342376968 >> 172.21.15.28:0/3342376968 conn(0x7f99d8077440 msgr2=0x7f99d8079860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:20:27.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:27.809+0000 7f99d7e00640 1 -- 172.21.15.28:0/3342376968 shutdown_connections 2024-05-12T22:20:27.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:27.809+0000 7f99d7e00640 1 -- 172.21.15.28:0/3342376968 wait complete. 2024-05-12T22:20:27.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:27.810+0000 7f99d7e00640 1 Processor -- start 2024-05-12T22:20:27.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:27.810+0000 7f99d7e00640 1 -- start start 2024-05-12T22:20:27.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:27.811+0000 7f99d7e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f99d8107ea0 0x7f99d819e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:20:27.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:27.811+0000 7f99d7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99d810cf60 0x7f99d819ea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:20:27.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:27.811+0000 7f99d7e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f99d81a2db0 0x7f99d81a3260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:20:27.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:27.811+0000 7f99d7e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f99d8117fd0 con 0x7f99d810cf60 2024-05-12T22:20:27.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:27.811+0000 7f99d7e00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f99d8117e50 con 0x7f99d81a2db0 2024-05-12T22:20:27.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:27.811+0000 7f99d7e00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f99d8118150 con 0x7f99d8107ea0 2024-05-12T22:20:27.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:27.811+0000 7f99d6000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99d810cf60 0x7f99d819ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:20:27.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:27.811+0000 7f99d6a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f99d8107ea0 0x7f99d819e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:20:27.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:27.811+0000 7f99d6000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99d810cf60 0x7f99d819ea30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42106/0 (socket says 172.21.15.28:42106) 2024-05-12T22:20:27.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:27.811+0000 7f99d6000640 1 -- 172.21.15.28:0/3202412912 learned_addr learned my addr 172.21.15.28:0/3202412912 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-05-12T22:20:27.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:27.811+0000 7f99d7400640 1 --2- 172.21.15.28:0/3202412912 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f99d81a2db0 0x7f99d81a3260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:20:27.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:27.812+0000 7f99d6a00640 1 -- 172.21.15.28:0/3202412912 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f99d81a2db0 msgr2=0x7f99d81a3260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:20:27.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:27.812+0000 7f99d6a00640 1 --2- 172.21.15.28:0/3202412912 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f99d81a2db0 0x7f99d81a3260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:20:27.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:27.812+0000 7f99d6a00640 1 -- 172.21.15.28:0/3202412912 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99d810cf60 msgr2=0x7f99d819ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:20:27.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:27.812+0000 7f99d6a00640 1 --2- 172.21.15.28:0/3202412912 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99d810cf60 0x7f99d819ea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:20:27.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:27.812+0000 7f99d6a00640 1 -- 172.21.15.28:0/3202412912 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99d81a38a0 con 0x7f99d8107ea0 2024-05-12T22:20:27.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:27.812+0000 7f99d6000640 1 --2- 172.21.15.28:0/3202412912 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99d810cf60 0x7f99d819ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-12T22:20:27.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:27.812+0000 7f99d6a00640 1 --2- 172.21.15.28:0/3202412912 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f99d8107ea0 0x7f99d819e4f0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f99c800e9f0 tx=0x7f99c800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:20:27.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:27.812+0000 7f99cf400640 1 -- 172.21.15.28:0/3202412912 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99c800ccc0 con 0x7f99d8107ea0 2024-05-12T22:20:27.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:27.812+0000 7f99cf400640 1 -- 172.21.15.28:0/3202412912 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f99c8010910 con 0x7f99d8107ea0 2024-05-12T22:20:27.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:27.812+0000 7f99d7e00640 1 -- 172.21.15.28:0/3202412912 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99d81a3b90 con 0x7f99d8107ea0 2024-05-12T22:20:27.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:27.813+0000 7f99cf400640 1 -- 172.21.15.28:0/3202412912 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99c8018960 con 0x7f99d8107ea0 2024-05-12T22:20:27.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:27.813+0000 7f99d7e00640 1 -- 172.21.15.28:0/3202412912 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99d81ab380 con 0x7f99d8107ea0 2024-05-12T22:20:27.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:27.814+0000 7f99d7e00640 1 -- 172.21.15.28:0/3202412912 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99d80689c0 con 0x7f99d8107ea0 2024-05-12T22:20:27.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:27.816+0000 7f99cf400640 1 -- 172.21.15.28:0/3202412912 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 27) ==== 137964+0+0 (secure 0 0 0) 0x7f99c8010ac0 con 0x7f99d8107ea0 2024-05-12T22:20:27.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:27.817+0000 7f99cf400640 1 --2- 172.21.15.28:0/3202412912 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f99bc0a2d00 0x7f99bc0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-12T22:20:27.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:27.818+0000 7f99cf400640 1 -- 172.21.15.28:0/3202412912 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 8173+0+0 (secure 0 0 0) 0x7f99c80cf9d0 con 0x7f99d8107ea0 2024-05-12T22:20:27.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:27.818+0000 7f99d6000640 1 --2- 172.21.15.28:0/3202412912 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f99bc0a2d00 0x7f99bc0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-12T22:20:27.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:27.819+0000 7f99d6000640 1 --2- 172.21.15.28:0/3202412912 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f99bc0a2d00 0x7f99bc0a51c0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f99c0005e00 tx=0x7f99c00094f0 comp rx=0 tx=0).ready entity=mgr.25189 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-12T22:20:27.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:27.820+0000 7f99cf400640 1 -- 172.21.15.28:0/3202412912 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f99c809c0b0 con 0x7f99d8107ea0 2024-05-12T22:20:28.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:28.037+0000 7f99d7e00640 1 -- 172.21.15.28:0/3202412912 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f99d819f840 con 0x7f99d8107ea0 2024-05-12T22:20:28.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:28.588+0000 7f99cf400640 1 -- 172.21.15.28:0/3202412912 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v102) ==== 169+0+0 (secure 0 0 0) 0x7f99c80a03b0 con 0x7f99d8107ea0 2024-05-12T22:20:28.590 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data' removed 2024-05-12T22:20:28.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:28.597+0000 7f99d7e00640 1 -- 172.21.15.28:0/3202412912 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f99bc0a2d00 msgr2=0x7f99bc0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:20:28.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:28.597+0000 7f99d7e00640 1 --2- 172.21.15.28:0/3202412912 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f99bc0a2d00 0x7f99bc0a51c0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f99c0005e00 tx=0x7f99c00094f0 comp rx=0 tx=0).stop 2024-05-12T22:20:28.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:28.598+0000 7f99d7e00640 1 -- 172.21.15.28:0/3202412912 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f99d8107ea0 msgr2=0x7f99d819e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-12T22:20:28.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:28.598+0000 7f99d7e00640 1 --2- 172.21.15.28:0/3202412912 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f99d8107ea0 0x7f99d819e4f0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f99c800e9f0 tx=0x7f99c800eeb0 comp rx=0 tx=0).stop 2024-05-12T22:20:28.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:28.598+0000 7f99d7e00640 1 -- 172.21.15.28:0/3202412912 shutdown_connections 2024-05-12T22:20:28.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:28.598+0000 7f99d7e00640 1 --2- 172.21.15.28:0/3202412912 >> [v2:172.21.15.28:6800/3177640985,v1:172.21.15.28:6801/3177640985] conn(0x7f99bc0a2d00 0x7f99bc0a51c0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:20:28.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:28.598+0000 7f99d7e00640 1 --2- 172.21.15.28:0/3202412912 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f99d81a2db0 0x7f99d81a3260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:20:28.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:28.598+0000 7f99d7e00640 1 --2- 172.21.15.28:0/3202412912 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99d810cf60 0x7f99d819ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:20:28.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:28.598+0000 7f99d7e00640 1 --2- 172.21.15.28:0/3202412912 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f99d8107ea0 0x7f99d819e4f0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-12T22:20:28.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:28.598+0000 7f99d7e00640 1 -- 172.21.15.28:0/3202412912 >> 172.21.15.28:0/3202412912 conn(0x7f99d8077440 msgr2=0x7f99d8079870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-12T22:20:28.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:28.598+0000 7f99d7e00640 1 -- 172.21.15.28:0/3202412912 shutdown_connections 2024-05-12T22:20:28.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-05-12T22:20:28.598+0000 7f99d7e00640 1 -- 172.21.15.28:0/3202412912 wait complete. 2024-05-12T22:20:28.631 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-12T22:20:28.697 DEBUG:teuthology.orchestra.run.smithi086:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-12T22:20:28.730 DEBUG:teuthology.orchestra.run.smithi186:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-12T22:20:28.760 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-05-12T22:20:28.760 DEBUG:teuthology.orchestra.run.smithi028:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-05-12T22:20:28.817 DEBUG:teuthology.orchestra.run.smithi086:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-05-12T22:20:28.834 DEBUG:teuthology.orchestra.run.smithi186:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-05-12T22:20:28.850 INFO:tasks.cephadm:Stopping all daemons... 2024-05-12T22:20:28.850 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-05-12T22:20:28.851 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968@mon.a 2024-05-12T22:20:28.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:20:28 smithi186 ceph-mon[34785]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-05-12T22:20:28.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:20:28 smithi186 ceph-mon[34785]: osdmap e101: 12 total, 12 up, 12 in 2024-05-12T22:20:28.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:20:28 smithi186 ceph-mon[34785]: from='client.? 172.21.15.28:0/3202412912' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-12T22:20:28.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:20:28 smithi186 ceph-mon[34785]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-12T22:20:28.930 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:20:28 smithi028 ceph-mon[29847]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-05-12T22:20:28.930 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:20:28 smithi028 ceph-mon[29847]: osdmap e101: 12 total, 12 up, 12 in 2024-05-12T22:20:28.930 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:20:28 smithi028 ceph-mon[29847]: from='client.? 172.21.15.28:0/3202412912' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-12T22:20:28.930 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:20:28 smithi028 ceph-mon[29847]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-12T22:20:28.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:20:28 smithi086 ceph-mon[34297]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-05-12T22:20:28.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:20:28 smithi086 ceph-mon[34297]: osdmap e101: 12 total, 12 up, 12 in 2024-05-12T22:20:28.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:20:28 smithi086 ceph-mon[34297]: from='client.? 172.21.15.28:0/3202412912' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-12T22:20:28.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:20:28 smithi086 ceph-mon[34297]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-12T22:20:29.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:20:28 smithi028 systemd[1]: Stopping Ceph mon.a for 6b92e7b4-10aa-11ef-bc98-c7b262605968... 2024-05-12T22:20:29.637 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:20:29 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mon-a[29843]: 2024-05-12T22:20:29.368+0000 7ff5c7c00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-05-12T22:20:29.638 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:20:29 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mon-a[29843]: 2024-05-12T22:20:29.368+0000 7ff5c7c00700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-05-12T22:20:29.638 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:20:29 smithi028 podman[99243]: 2024-05-12 22:20:29.451809129 +0000 UTC m=+0.227991377 container died d414f473c6992788c451a5cddeb3df439654f5c4c6067e5b77723e23aab44fc9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f72fecff68e1d400c4568684327c900485c20d6a, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mon-a, org.label-schema.build-date=20240507, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=squid-f72fecf, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-12T22:20:29.895 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:20:29 smithi028 podman[99243]: 2024-05-12 22:20:29.881155838 +0000 UTC m=+0.657338109 container cleanup d414f473c6992788c451a5cddeb3df439654f5c4c6067e5b77723e23aab44fc9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f72fecff68e1d400c4568684327c900485c20d6a, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mon-a, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, CEPH_POINT_RELEASE=, RELEASE=squid-f72fecf, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240507, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-12T22:20:29.895 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:20:29 smithi028 bash[99243]: ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mon-a 2024-05-12T22:20:30.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:May 12 22:20:30 smithi028 podman[99256]: 2024-05-12 22:20:30.090559001 +0000 UTC m=+0.621905868 container remove d414f473c6992788c451a5cddeb3df439654f5c4c6067e5b77723e23aab44fc9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f72fecff68e1d400c4568684327c900485c20d6a, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mon-a, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240507, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, RELEASE=squid-f72fecf, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-05-12T22:20:30.580 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968@mon.a.service' 2024-05-12T22:20:30.718 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-12T22:20:30.719 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-05-12T22:20:30.719 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-05-12T22:20:30.719 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl stop ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968@mon.b 2024-05-12T22:20:31.006 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:20:30 smithi086 systemd[1]: Stopping Ceph mon.b for 6b92e7b4-10aa-11ef-bc98-c7b262605968... 2024-05-12T22:20:31.323 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:20:31 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mon-b[34293]: 2024-05-12T22:20:31.005+0000 7ff1a0834700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-05-12T22:20:31.323 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:20:31 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mon-b[34293]: 2024-05-12T22:20:31.005+0000 7ff1a0834700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-05-12T22:20:31.323 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:20:31 smithi086 podman[70298]: 2024-05-12 22:20:31.026755879 +0000 UTC m=+0.116604513 container died d12e520b6c5ccd7d5e17136e3f52defceaf31c757111ebdcdffcb1bff67132fc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mon-b, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=squid-f72fecf, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240507, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-05-12T22:20:31.607 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:20:31 smithi086 podman[70298]: 2024-05-12 22:20:31.40621617 +0000 UTC m=+0.496064811 container cleanup d12e520b6c5ccd7d5e17136e3f52defceaf31c757111ebdcdffcb1bff67132fc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mon-b, ceph=True, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240507, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=squid-f72fecf, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-05-12T22:20:31.607 INFO:journalctl@ceph.mon.b.smithi086.stdout:May 12 22:20:31 smithi086 bash[70298]: ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mon-b 2024-05-12T22:20:31.745 DEBUG:teuthology.orchestra.run.smithi086:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968@mon.b.service' 2024-05-12T22:20:31.795 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-12T22:20:31.796 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-05-12T22:20:31.796 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-05-12T22:20:31.796 DEBUG:teuthology.orchestra.run.smithi186:> sudo systemctl stop ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968@mon.c 2024-05-12T22:20:32.115 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:20:31 smithi186 systemd[1]: Stopping Ceph mon.c for 6b92e7b4-10aa-11ef-bc98-c7b262605968... 2024-05-12T22:20:32.381 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:20:32 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mon-c[34767]: 2024-05-12T22:20:32.114+0000 7fa1cde10700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-05-12T22:20:32.381 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:20:32 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mon-c[34767]: 2024-05-12T22:20:32.114+0000 7fa1cde10700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-05-12T22:20:32.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:20:32 smithi186 podman[71359]: 2024-05-12 22:20:32.381347782 +0000 UTC m=+0.386946860 container died 5a2c2778badf3aa40a49a55c171003adbc3e3f4884e15380caa84eba0cd00c82 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mon-c, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, ceph=True, org.label-schema.build-date=20240507, RELEASE=squid-f72fecf, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-05-12T22:20:32.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:20:32 smithi186 podman[71359]: 2024-05-12 22:20:32.731955466 +0000 UTC m=+0.737554534 container cleanup 5a2c2778badf3aa40a49a55c171003adbc3e3f4884e15380caa84eba0cd00c82 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mon-c, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=squid-f72fecf, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240507) 2024-05-12T22:20:32.929 INFO:journalctl@ceph.mon.c.smithi186.stdout:May 12 22:20:32 smithi186 bash[71359]: ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mon-c 2024-05-12T22:20:33.149 DEBUG:teuthology.orchestra.run.smithi186:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968@mon.c.service' 2024-05-12T22:20:33.215 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-12T22:20:33.215 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-05-12T22:20:33.215 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-05-12T22:20:33.215 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968@mgr.x 2024-05-12T22:20:33.676 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 22:20:33 smithi028 systemd[1]: Stopping Ceph mgr.x for 6b92e7b4-10aa-11ef-bc98-c7b262605968... 2024-05-12T22:20:34.285 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 22:20:33 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T22:20:33.977+0000 7f3d29600700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-12T22:20:34.286 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 22:20:33 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x[30078]: 2024-05-12T22:20:33.977+0000 7f3d29600700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-05-12T22:20:34.286 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 22:20:34 smithi028 podman[99393]: 2024-05-12 22:20:34.073064669 +0000 UTC m=+0.351834101 container died 1cf9bf8977739000e8e79d4d9f12daf54a215193bfc5820e3a728e0af49d0def (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f72fecff68e1d400c4568684327c900485c20d6a, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.license=GPLv2, org.label-schema.build-date=20240507, GIT_BRANCH=HEAD, RELEASE=squid-f72fecf, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-05-12T22:20:35.036 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 22:20:34 smithi028 podman[99393]: 2024-05-12 22:20:34.843727087 +0000 UTC m=+1.122496533 container cleanup 1cf9bf8977739000e8e79d4d9f12daf54a215193bfc5820e3a728e0af49d0def (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f72fecff68e1d400c4568684327c900485c20d6a, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240507, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, RELEASE=squid-f72fecf, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-12T22:20:35.037 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 22:20:34 smithi028 bash[99393]: ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x 2024-05-12T22:20:35.535 INFO:journalctl@ceph.mgr.x.smithi028.stdout:May 12 22:20:35 smithi028 podman[99407]: 2024-05-12 22:20:35.04496613 +0000 UTC m=+0.955539925 container remove 1cf9bf8977739000e8e79d4d9f12daf54a215193bfc5820e3a728e0af49d0def (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:f72fecff68e1d400c4568684327c900485c20d6a, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-x, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-f72fecf, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.build-date=20240507, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-05-12T22:20:35.631 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968@mgr.x.service' 2024-05-12T22:20:35.749 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-12T22:20:35.749 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-05-12T22:20:35.749 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-05-12T22:20:35.749 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl stop ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968@mgr.y 2024-05-12T22:20:36.064 INFO:journalctl@ceph.mgr.y.smithi086.stdout:May 12 22:20:35 smithi086 systemd[1]: Stopping Ceph mgr.y for 6b92e7b4-10aa-11ef-bc98-c7b262605968... 2024-05-12T22:20:36.318 INFO:journalctl@ceph.mgr.y.smithi086.stdout:May 12 22:20:36 smithi086 podman[70420]: 2024-05-12 22:20:36.064449865 +0000 UTC m=+0.124705609 container died 529d155079c6b0e621975eb64a9629b69079d706d81014410167dec2b7046e37 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-y, RELEASE=squid-f72fecf, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240507, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True) 2024-05-12T22:20:36.685 INFO:journalctl@ceph.mgr.y.smithi086.stdout:May 12 22:20:36 smithi086 podman[70420]: 2024-05-12 22:20:36.435110199 +0000 UTC m=+0.495365931 container cleanup 529d155079c6b0e621975eb64a9629b69079d706d81014410167dec2b7046e37 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-y, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.build-date=20240507, RELEASE=squid-f72fecf, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-05-12T22:20:36.686 INFO:journalctl@ceph.mgr.y.smithi086.stdout:May 12 22:20:36 smithi086 bash[70420]: ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-y 2024-05-12T22:20:36.810 DEBUG:teuthology.orchestra.run.smithi086:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968@mgr.y.service' 2024-05-12T22:20:36.848 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-12T22:20:36.849 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-05-12T22:20:36.849 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-05-12T22:20:36.849 DEBUG:teuthology.orchestra.run.smithi186:> sudo systemctl stop ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968@mgr.z 2024-05-12T22:20:37.132 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 12 22:20:36 smithi186 systemd[1]: Stopping Ceph mgr.z for 6b92e7b4-10aa-11ef-bc98-c7b262605968... 2024-05-12T22:20:37.429 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 12 22:20:37 smithi186 podman[71482]: 2024-05-12 22:20:37.17425708 +0000 UTC m=+0.125436407 container died 893dc82d49b9126afe509635c6e6b3b1352b943e8cf19a8b03126128249e2a54 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-z, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240507, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-f72fecf, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-05-12T22:20:37.688 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 12 22:20:37 smithi186 podman[71482]: 2024-05-12 22:20:37.511699457 +0000 UTC m=+0.462878776 container cleanup 893dc82d49b9126afe509635c6e6b3b1352b943e8cf19a8b03126128249e2a54 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-z, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240507, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, GIT_BRANCH=HEAD, RELEASE=squid-f72fecf, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-05-12T22:20:37.689 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 12 22:20:37 smithi186 bash[71482]: ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-z 2024-05-12T22:20:37.943 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 12 22:20:37 smithi186 podman[71496]: 2024-05-12 22:20:37.688096435 +0000 UTC m=+0.509577663 container remove 893dc82d49b9126afe509635c6e6b3b1352b943e8cf19a8b03126128249e2a54 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mgr-z, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240507, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=squid-f72fecf, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-05-12T22:20:37.943 INFO:journalctl@ceph.mgr.z.smithi186.stdout:May 12 22:20:37 smithi186 systemd[1]: ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968@mgr.z.service: Main process exited, code=exited, status=143/n/a 2024-05-12T22:20:37.955 DEBUG:teuthology.orchestra.run.smithi186:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968@mgr.z.service' 2024-05-12T22:20:37.991 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-12T22:20:37.991 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-05-12T22:20:37.991 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-05-12T22:20:37.991 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968@osd.0 2024-05-12T22:20:38.535 INFO:journalctl@ceph.osd.0.smithi028.stdout:May 12 22:20:38 smithi028 systemd[1]: Stopping Ceph osd.0 for 6b92e7b4-10aa-11ef-bc98-c7b262605968... 2024-05-12T22:20:39.035 INFO:journalctl@ceph.osd.0.smithi028.stdout:May 12 22:20:38 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-0[41956]: 2024-05-12T22:20:38.623+0000 7f9629000700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-12T22:20:39.036 INFO:journalctl@ceph.osd.0.smithi028.stdout:May 12 22:20:38 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-0[41956]: 2024-05-12T22:20:38.623+0000 7f9629000700 -1 osd.0 102 *** Got signal Terminated *** 2024-05-12T22:20:39.036 INFO:journalctl@ceph.osd.0.smithi028.stdout:May 12 22:20:38 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-0[41956]: 2024-05-12T22:20:38.623+0000 7f9629000700 -1 osd.0 102 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-12T22:20:44.275 INFO:journalctl@ceph.osd.0.smithi028.stdout:May 12 22:20:44 smithi028 podman[99514]: 2024-05-12 22:20:44.005449478 +0000 UTC m=+5.545460832 container died f675be533a36b54111473e65f421a76fdf0962d529f829e2cb97e7597bb9932d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-0, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=squid-f72fecf, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240507, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-05-12T22:20:44.535 INFO:journalctl@ceph.osd.0.smithi028.stdout:May 12 22:20:44 smithi028 podman[99514]: 2024-05-12 22:20:44.389844373 +0000 UTC m=+5.929855735 container cleanup f675be533a36b54111473e65f421a76fdf0962d529f829e2cb97e7597bb9932d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-0, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, ceph=True, RELEASE=squid-f72fecf, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240507) 2024-05-12T22:20:44.536 INFO:journalctl@ceph.osd.0.smithi028.stdout:May 12 22:20:44 smithi028 bash[99514]: ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-0 2024-05-12T22:20:45.035 INFO:journalctl@ceph.osd.0.smithi028.stdout:May 12 22:20:44 smithi028 podman[99532]: 2024-05-12 22:20:44.591253058 +0000 UTC m=+0.568835873 container remove f675be533a36b54111473e65f421a76fdf0962d529f829e2cb97e7597bb9932d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-0, io.buildah.version=1.33.5, RELEASE=squid-f72fecf, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240507, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-12T22:20:45.785 INFO:journalctl@ceph.osd.0.smithi028.stdout:May 12 22:20:45 smithi028 podman[99621]: 2024-05-12 22:20:45.508825994 +0000 UTC m=+0.125891534 container create 7837c5fd88ef27ee80e544f42cb3094f6c55be751774d5da2df31a06832483d5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-0-deactivate, RELEASE=squid-f72fecf, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.build-date=20240507, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-05-12T22:20:45.785 INFO:journalctl@ceph.osd.0.smithi028.stdout:May 12 22:20:45 smithi028 podman[99621]: 2024-05-12 22:20:45.425050753 +0000 UTC m=+0.042116278 image pull 8252b7944d3b5c188c0ff632647ba6cc4e9a19588ee097b8a3dba27b42779c25 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc 2024-05-12T22:20:46.285 INFO:journalctl@ceph.osd.0.smithi028.stdout:May 12 22:20:45 smithi028 podman[99621]: 2024-05-12 22:20:45.812976413 +0000 UTC m=+0.430042027 container init 7837c5fd88ef27ee80e544f42cb3094f6c55be751774d5da2df31a06832483d5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-0-deactivate, maintainer=Guillaume Abrioux , GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-f72fecf, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240507) 2024-05-12T22:20:46.285 INFO:journalctl@ceph.osd.0.smithi028.stdout:May 12 22:20:45 smithi028 podman[99621]: 2024-05-12 22:20:45.824787375 +0000 UTC m=+0.441852939 container start 7837c5fd88ef27ee80e544f42cb3094f6c55be751774d5da2df31a06832483d5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-0-deactivate, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240507, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=squid-f72fecf, ceph=True) 2024-05-12T22:20:46.286 INFO:journalctl@ceph.osd.0.smithi028.stdout:May 12 22:20:45 smithi028 podman[99621]: 2024-05-12 22:20:45.927891172 +0000 UTC m=+0.544956779 container attach 7837c5fd88ef27ee80e544f42cb3094f6c55be751774d5da2df31a06832483d5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-0-deactivate, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240507, RELEASE=squid-f72fecf, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-05-12T22:20:46.568 INFO:journalctl@ceph.osd.0.smithi028.stdout:May 12 22:20:46 smithi028 podman[99621]: 2024-05-12 22:20:46.291469492 +0000 UTC m=+0.908535089 container died 7837c5fd88ef27ee80e544f42cb3094f6c55be751774d5da2df31a06832483d5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-0-deactivate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240507, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=squid-f72fecf, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-12T22:20:46.955 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968@osd.0.service' 2024-05-12T22:20:47.082 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-12T22:20:47.082 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-05-12T22:20:47.082 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-05-12T22:20:47.082 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl stop ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968@osd.1 2024-05-12T22:20:47.381 INFO:journalctl@ceph.osd.1.smithi086.stdout:May 12 22:20:47 smithi086 systemd[1]: Stopping Ceph osd.1 for 6b92e7b4-10aa-11ef-bc98-c7b262605968... 2024-05-12T22:20:47.742 INFO:journalctl@ceph.osd.1.smithi086.stdout:May 12 22:20:47 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-1[38925]: 2024-05-12T22:20:47.380+0000 7efc66c90700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-12T22:20:47.742 INFO:journalctl@ceph.osd.1.smithi086.stdout:May 12 22:20:47 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-1[38925]: 2024-05-12T22:20:47.380+0000 7efc66c90700 -1 osd.1 102 *** Got signal Terminated *** 2024-05-12T22:20:47.742 INFO:journalctl@ceph.osd.1.smithi086.stdout:May 12 22:20:47 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-1[38925]: 2024-05-12T22:20:47.380+0000 7efc66c90700 -1 osd.1 102 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-12T22:20:53.188 INFO:journalctl@ceph.osd.1.smithi086.stdout:May 12 22:20:52 smithi086 podman[70555]: 2024-05-12 22:20:52.889905903 +0000 UTC m=+5.604264963 container died 801b9a5181ff5a5084fe0caa0935db9d32bcfe8dd3760b665d2ac25ed5fd3158 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-1, org.label-schema.license=GPLv2, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=squid-f72fecf, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240507, GIT_BRANCH=HEAD) 2024-05-12T22:20:53.492 INFO:journalctl@ceph.osd.1.smithi086.stdout:May 12 22:20:53 smithi086 podman[70555]: 2024-05-12 22:20:53.189275235 +0000 UTC m=+5.903634292 container cleanup 801b9a5181ff5a5084fe0caa0935db9d32bcfe8dd3760b665d2ac25ed5fd3158 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-1, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=squid-f72fecf, GIT_CLEAN=True, org.label-schema.build-date=20240507, org.label-schema.license=GPLv2, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-05-12T22:20:53.492 INFO:journalctl@ceph.osd.1.smithi086.stdout:May 12 22:20:53 smithi086 bash[70555]: ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-1 2024-05-12T22:20:53.492 INFO:journalctl@ceph.osd.1.smithi086.stdout:May 12 22:20:53 smithi086 podman[70570]: 2024-05-12 22:20:53.423410662 +0000 UTC m=+0.530577651 container remove 801b9a5181ff5a5084fe0caa0935db9d32bcfe8dd3760b665d2ac25ed5fd3158 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-1, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-f72fecf, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240507, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-05-12T22:20:53.992 INFO:journalctl@ceph.osd.1.smithi086.stdout:May 12 22:20:53 smithi086 podman[70666]: 2024-05-12 22:20:53.740941078 +0000 UTC m=+0.104978665 container create 64fc78602b7968364db15d10d05d3030a326f32e58debc0b0015bd7e3e81577e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-1-deactivate, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240507, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=squid-f72fecf, ceph=True, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e) 2024-05-12T22:20:53.992 INFO:journalctl@ceph.osd.1.smithi086.stdout:May 12 22:20:53 smithi086 podman[70666]: 2024-05-12 22:20:53.653989306 +0000 UTC m=+0.018026859 image pull 8252b7944d3b5c188c0ff632647ba6cc4e9a19588ee097b8a3dba27b42779c25 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc 2024-05-12T22:20:53.992 INFO:journalctl@ceph.osd.1.smithi086.stdout:May 12 22:20:53 smithi086 podman[70666]: 2024-05-12 22:20:53.944041098 +0000 UTC m=+0.308078642 container init 64fc78602b7968364db15d10d05d3030a326f32e58debc0b0015bd7e3e81577e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-1-deactivate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240507, RELEASE=squid-f72fecf, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-05-12T22:20:53.992 INFO:journalctl@ceph.osd.1.smithi086.stdout:May 12 22:20:53 smithi086 podman[70666]: 2024-05-12 22:20:53.947068811 +0000 UTC m=+0.311106353 container start 64fc78602b7968364db15d10d05d3030a326f32e58debc0b0015bd7e3e81577e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-1-deactivate, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-f72fecf, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240507) 2024-05-12T22:20:54.492 INFO:journalctl@ceph.osd.1.smithi086.stdout:May 12 22:20:54 smithi086 podman[70666]: 2024-05-12 22:20:54.010640232 +0000 UTC m=+0.374677790 container attach 64fc78602b7968364db15d10d05d3030a326f32e58debc0b0015bd7e3e81577e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-1-deactivate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=squid-f72fecf, GIT_BRANCH=HEAD, org.label-schema.build-date=20240507, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True) 2024-05-12T22:20:54.492 INFO:journalctl@ceph.osd.1.smithi086.stdout:May 12 22:20:54 smithi086 podman[70666]: 2024-05-12 22:20:54.239365188 +0000 UTC m=+0.603402735 container died 64fc78602b7968364db15d10d05d3030a326f32e58debc0b0015bd7e3e81577e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-1-deactivate, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=squid-f72fecf, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240507) 2024-05-12T22:20:54.845 DEBUG:teuthology.orchestra.run.smithi086:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968@osd.1.service' 2024-05-12T22:20:54.883 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-12T22:20:54.884 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-05-12T22:20:54.884 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-05-12T22:20:54.884 DEBUG:teuthology.orchestra.run.smithi186:> sudo systemctl stop ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968@osd.2 2024-05-12T22:20:55.174 INFO:journalctl@ceph.osd.2.smithi186.stdout:May 12 22:20:54 smithi186 systemd[1]: Stopping Ceph osd.2 for 6b92e7b4-10aa-11ef-bc98-c7b262605968... 2024-05-12T22:20:55.429 INFO:journalctl@ceph.osd.2.smithi186.stdout:May 12 22:20:55 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-2[39983]: 2024-05-12T22:20:55.173+0000 7fee1858b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-12T22:20:55.429 INFO:journalctl@ceph.osd.2.smithi186.stdout:May 12 22:20:55 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-2[39983]: 2024-05-12T22:20:55.173+0000 7fee1858b700 -1 osd.2 102 *** Got signal Terminated *** 2024-05-12T22:20:55.429 INFO:journalctl@ceph.osd.2.smithi186.stdout:May 12 22:20:55 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-2[39983]: 2024-05-12T22:20:55.173+0000 7fee1858b700 -1 osd.2 102 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-12T22:21:01.042 INFO:journalctl@ceph.osd.2.smithi186.stdout:May 12 22:21:00 smithi186 podman[71618]: 2024-05-12 22:21:00.711219292 +0000 UTC m=+5.638803703 container died 20fe3c8a9449708ca5429db74f666fba5ab5b673bb10e2d4460d3a710208fd4b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-2, ceph=True, org.label-schema.build-date=20240507, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=squid-f72fecf, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-05-12T22:21:01.349 INFO:journalctl@ceph.osd.2.smithi186.stdout:May 12 22:21:01 smithi186 podman[71618]: 2024-05-12 22:21:01.115694115 +0000 UTC m=+6.043278576 container cleanup 20fe3c8a9449708ca5429db74f666fba5ab5b673bb10e2d4460d3a710208fd4b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-2, org.label-schema.build-date=20240507, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, RELEASE=squid-f72fecf) 2024-05-12T22:21:01.350 INFO:journalctl@ceph.osd.2.smithi186.stdout:May 12 22:21:01 smithi186 bash[71618]: ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-2 2024-05-12T22:21:01.663 INFO:journalctl@ceph.osd.2.smithi186.stdout:May 12 22:21:01 smithi186 podman[71636]: 2024-05-12 22:21:01.35046443 +0000 UTC m=+0.634830716 container remove 20fe3c8a9449708ca5429db74f666fba5ab5b673bb10e2d4460d3a710208fd4b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-2, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=squid-f72fecf, org.label-schema.build-date=20240507, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5) 2024-05-12T22:21:01.929 INFO:journalctl@ceph.osd.2.smithi186.stdout:May 12 22:21:01 smithi186 podman[71724]: 2024-05-12 22:21:01.566651652 +0000 UTC m=+0.017968383 image pull 8252b7944d3b5c188c0ff632647ba6cc4e9a19588ee097b8a3dba27b42779c25 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc 2024-05-12T22:21:01.929 INFO:journalctl@ceph.osd.2.smithi186.stdout:May 12 22:21:01 smithi186 podman[71724]: 2024-05-12 22:21:01.692828587 +0000 UTC m=+0.144145306 container create c2c6425b6c4e6730fdc5549e617b4258479b6846063658e9ca73b16781f25fd1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-2-deactivate, RELEASE=squid-f72fecf, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240507, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-05-12T22:21:01.929 INFO:journalctl@ceph.osd.2.smithi186.stdout:May 12 22:21:01 smithi186 podman[71724]: 2024-05-12 22:21:01.903610518 +0000 UTC m=+0.354927241 container init c2c6425b6c4e6730fdc5549e617b4258479b6846063658e9ca73b16781f25fd1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-2-deactivate, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-f72fecf, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240507, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-05-12T22:21:01.930 INFO:journalctl@ceph.osd.2.smithi186.stdout:May 12 22:21:01 smithi186 podman[71724]: 2024-05-12 22:21:01.906733997 +0000 UTC m=+0.358050721 container start c2c6425b6c4e6730fdc5549e617b4258479b6846063658e9ca73b16781f25fd1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-2-deactivate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=squid-f72fecf, CEPH_POINT_RELEASE=, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240507, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-05-12T22:21:02.203 INFO:journalctl@ceph.osd.2.smithi186.stdout:May 12 22:21:02 smithi186 podman[71724]: 2024-05-12 22:21:02.020596776 +0000 UTC m=+0.471913511 container attach c2c6425b6c4e6730fdc5549e617b4258479b6846063658e9ca73b16781f25fd1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-2-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=squid-f72fecf, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240507, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, ceph=True, GIT_CLEAN=True) 2024-05-12T22:21:02.477 INFO:journalctl@ceph.osd.2.smithi186.stdout:May 12 22:21:02 smithi186 podman[71724]: 2024-05-12 22:21:02.204566915 +0000 UTC m=+0.655883644 container died c2c6425b6c4e6730fdc5549e617b4258479b6846063658e9ca73b16781f25fd1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-2-deactivate, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240507, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=squid-f72fecf, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-05-12T22:21:02.804 DEBUG:teuthology.orchestra.run.smithi186:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968@osd.2.service' 2024-05-12T22:21:02.843 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-12T22:21:02.843 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-05-12T22:21:02.843 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-05-12T22:21:02.843 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968@osd.3 2024-05-12T22:21:03.286 INFO:journalctl@ceph.osd.3.smithi028.stdout:May 12 22:21:02 smithi028 systemd[1]: Stopping Ceph osd.3 for 6b92e7b4-10aa-11ef-bc98-c7b262605968... 2024-05-12T22:21:03.656 INFO:journalctl@ceph.osd.3.smithi028.stdout:May 12 22:21:03 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-3[48501]: 2024-05-12T22:21:03.465+0000 7f74a6400700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-12T22:21:03.657 INFO:journalctl@ceph.osd.3.smithi028.stdout:May 12 22:21:03 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-3[48501]: 2024-05-12T22:21:03.465+0000 7f74a6400700 -1 osd.3 102 *** Got signal Terminated *** 2024-05-12T22:21:03.657 INFO:journalctl@ceph.osd.3.smithi028.stdout:May 12 22:21:03 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-3[48501]: 2024-05-12T22:21:03.465+0000 7f74a6400700 -1 osd.3 102 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-12T22:21:06.035 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:05 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:05.778+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.490538+0000 front 2024-05-12T22:20:41.490674+0000 (oldest deadline 2024-05-12T22:21:04.990968+0000) 2024-05-12T22:21:06.492 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:06 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:06.146+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:43.519531+0000 front 2024-05-12T22:20:43.519638+0000 (oldest deadline 2024-05-12T22:21:05.219288+0000) 2024-05-12T22:21:07.035 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:06 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:06.763+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.490538+0000 front 2024-05-12T22:20:41.490674+0000 (oldest deadline 2024-05-12T22:21:04.990968+0000) 2024-05-12T22:21:07.179 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:06 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:06.684+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.561281+0000 front 2024-05-12T22:20:41.561023+0000 (oldest deadline 2024-05-12T22:21:06.261036+0000) 2024-05-12T22:21:07.492 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:07 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:07.162+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:43.519531+0000 front 2024-05-12T22:20:43.519638+0000 (oldest deadline 2024-05-12T22:21:05.219288+0000) 2024-05-12T22:21:07.657 INFO:journalctl@ceph.osd.5.smithi186.stdout:May 12 22:21:07 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-5[46962]: 2024-05-12T22:21:07.276+0000 7f82e356f700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.993114+0000 front 2024-05-12T22:20:41.993039+0000 (oldest deadline 2024-05-12T22:21:07.193555+0000) 2024-05-12T22:21:07.929 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:07 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:07.657+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.561281+0000 front 2024-05-12T22:20:41.561023+0000 (oldest deadline 2024-05-12T22:21:06.261036+0000) 2024-05-12T22:21:08.035 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:07 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:07.714+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.490538+0000 front 2024-05-12T22:20:41.490674+0000 (oldest deadline 2024-05-12T22:21:04.990968+0000) 2024-05-12T22:21:08.492 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:08 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:08.158+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:43.519531+0000 front 2024-05-12T22:20:43.519638+0000 (oldest deadline 2024-05-12T22:21:05.219288+0000) 2024-05-12T22:21:08.672 INFO:journalctl@ceph.osd.5.smithi186.stdout:May 12 22:21:08 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-5[46962]: 2024-05-12T22:21:08.265+0000 7f82e356f700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.993114+0000 front 2024-05-12T22:20:41.993039+0000 (oldest deadline 2024-05-12T22:21:07.193555+0000) 2024-05-12T22:21:08.929 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:08 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:08.672+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.561281+0000 front 2024-05-12T22:20:41.561023+0000 (oldest deadline 2024-05-12T22:21:06.261036+0000) 2024-05-12T22:21:08.983 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:08 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:08.700+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.490538+0000 front 2024-05-12T22:20:41.490674+0000 (oldest deadline 2024-05-12T22:21:04.990968+0000) 2024-05-12T22:21:09.285 INFO:journalctl@ceph.osd.3.smithi028.stdout:May 12 22:21:08 smithi028 podman[99724]: 2024-05-12 22:21:08.983562017 +0000 UTC m=+5.654084303 container died 613a90bed9f68faa3bb26673a9cee2861205b8f77aed4f220ac5de097a577cd6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-3, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=squid-f72fecf, org.label-schema.build-date=20240507, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-12T22:21:09.285 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 12 22:21:09 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-6[55080]: 2024-05-12T22:21:09.217+0000 7f2fefe00700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:43.392022+0000 front 2024-05-12T22:20:43.392513+0000 (oldest deadline 2024-05-12T22:21:08.692004+0000) 2024-05-12T22:21:09.492 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:09 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:09.144+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:43.519531+0000 front 2024-05-12T22:20:43.519638+0000 (oldest deadline 2024-05-12T22:21:05.219288+0000) 2024-05-12T22:21:09.657 INFO:journalctl@ceph.osd.5.smithi186.stdout:May 12 22:21:09 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-5[46962]: 2024-05-12T22:21:09.266+0000 7f82e356f700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.993114+0000 front 2024-05-12T22:20:41.993039+0000 (oldest deadline 2024-05-12T22:21:07.193555+0000) 2024-05-12T22:21:09.742 INFO:journalctl@ceph.osd.3.smithi028.stdout:May 12 22:21:09 smithi028 podman[99724]: 2024-05-12 22:21:09.467638633 +0000 UTC m=+6.138160969 container cleanup 613a90bed9f68faa3bb26673a9cee2861205b8f77aed4f220ac5de097a577cd6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-3, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.build-date=20240507, RELEASE=squid-f72fecf, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-05-12T22:21:09.742 INFO:journalctl@ceph.osd.3.smithi028.stdout:May 12 22:21:09 smithi028 bash[99724]: ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-3 2024-05-12T22:21:09.929 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:09 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:09.657+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.561281+0000 front 2024-05-12T22:20:41.561023+0000 (oldest deadline 2024-05-12T22:21:06.261036+0000) 2024-05-12T22:21:10.034 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:09 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:09.741+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.490538+0000 front 2024-05-12T22:20:41.490674+0000 (oldest deadline 2024-05-12T22:21:04.990968+0000) 2024-05-12T22:21:10.035 INFO:journalctl@ceph.osd.3.smithi028.stdout:May 12 22:21:09 smithi028 podman[99753]: 2024-05-12 22:21:09.772685436 +0000 UTC m=+0.771652995 container remove 613a90bed9f68faa3bb26673a9cee2861205b8f77aed4f220ac5de097a577cd6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-3, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240507, RELEASE=squid-f72fecf, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e) 2024-05-12T22:21:10.492 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:10 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:10.100+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:43.519531+0000 front 2024-05-12T22:20:43.519638+0000 (oldest deadline 2024-05-12T22:21:05.219288+0000) 2024-05-12T22:21:10.535 INFO:journalctl@ceph.osd.3.smithi028.stdout:May 12 22:21:10 smithi028 podman[99842]: 2024-05-12 22:21:10.274510182 +0000 UTC m=+0.034999295 image pull 8252b7944d3b5c188c0ff632647ba6cc4e9a19588ee097b8a3dba27b42779c25 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc 2024-05-12T22:21:10.536 INFO:journalctl@ceph.osd.3.smithi028.stdout:May 12 22:21:10 smithi028 podman[99842]: 2024-05-12 22:21:10.411547363 +0000 UTC m=+0.172036509 container create 4e94270033606b346eb2f69405e8e3a96f61815a1454606d98d77e12f29882f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-3-deactivate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, RELEASE=squid-f72fecf, org.label-schema.build-date=20240507, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-12T22:21:10.536 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 12 22:21:10 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-6[55080]: 2024-05-12T22:21:10.203+0000 7f2fefe00700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:43.392022+0000 front 2024-05-12T22:20:43.392513+0000 (oldest deadline 2024-05-12T22:21:08.692004+0000) 2024-05-12T22:21:10.622 INFO:journalctl@ceph.osd.5.smithi186.stdout:May 12 22:21:10 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-5[46962]: 2024-05-12T22:21:10.256+0000 7f82e356f700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.993114+0000 front 2024-05-12T22:20:41.993039+0000 (oldest deadline 2024-05-12T22:21:07.193555+0000) 2024-05-12T22:21:10.929 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:10 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:10.622+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.561281+0000 front 2024-05-12T22:20:41.561023+0000 (oldest deadline 2024-05-12T22:21:06.261036+0000) 2024-05-12T22:21:11.035 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:10 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:10.713+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.490538+0000 front 2024-05-12T22:20:41.490674+0000 (oldest deadline 2024-05-12T22:21:04.990968+0000) 2024-05-12T22:21:11.035 INFO:journalctl@ceph.osd.3.smithi028.stdout:May 12 22:21:10 smithi028 podman[99842]: 2024-05-12 22:21:10.699483406 +0000 UTC m=+0.459972559 container init 4e94270033606b346eb2f69405e8e3a96f61815a1454606d98d77e12f29882f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-3-deactivate, RELEASE=squid-f72fecf, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240507, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-05-12T22:21:11.035 INFO:journalctl@ceph.osd.3.smithi028.stdout:May 12 22:21:10 smithi028 podman[99842]: 2024-05-12 22:21:10.712967855 +0000 UTC m=+0.473456987 container start 4e94270033606b346eb2f69405e8e3a96f61815a1454606d98d77e12f29882f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-3-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-f72fecf, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240507, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-05-12T22:21:11.036 INFO:journalctl@ceph.osd.3.smithi028.stdout:May 12 22:21:10 smithi028 podman[99842]: 2024-05-12 22:21:10.813925076 +0000 UTC m=+0.574414235 container attach 4e94270033606b346eb2f69405e8e3a96f61815a1454606d98d77e12f29882f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-3-deactivate, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.build-date=20240507, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, RELEASE=squid-f72fecf, CEPH_POINT_RELEASE=) 2024-05-12T22:21:11.492 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:11 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:11.061+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:43.519531+0000 front 2024-05-12T22:20:43.519638+0000 (oldest deadline 2024-05-12T22:21:05.219288+0000) 2024-05-12T22:21:11.521 INFO:journalctl@ceph.osd.3.smithi028.stdout:May 12 22:21:11 smithi028 podman[99842]: 2024-05-12 22:21:11.181052314 +0000 UTC m=+0.941541454 container died 4e94270033606b346eb2f69405e8e3a96f61815a1454606d98d77e12f29882f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-3-deactivate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=squid-f72fecf, org.label-schema.build-date=20240507, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-05-12T22:21:11.521 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 12 22:21:11 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-6[55080]: 2024-05-12T22:21:11.178+0000 7f2fefe00700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:43.392022+0000 front 2024-05-12T22:20:43.392513+0000 (oldest deadline 2024-05-12T22:21:08.692004+0000) 2024-05-12T22:21:11.576 INFO:journalctl@ceph.osd.5.smithi186.stdout:May 12 22:21:11 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-5[46962]: 2024-05-12T22:21:11.289+0000 7f82e356f700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.993114+0000 front 2024-05-12T22:20:41.993039+0000 (oldest deadline 2024-05-12T22:21:07.193555+0000) 2024-05-12T22:21:11.785 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:11 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:11.728+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.490538+0000 front 2024-05-12T22:20:41.490674+0000 (oldest deadline 2024-05-12T22:21:04.990968+0000) 2024-05-12T22:21:11.886 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968@osd.3.service' 2024-05-12T22:21:11.929 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:11 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:11.577+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.561281+0000 front 2024-05-12T22:20:41.561023+0000 (oldest deadline 2024-05-12T22:21:06.261036+0000) 2024-05-12T22:21:12.010 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-12T22:21:12.011 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-05-12T22:21:12.011 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-05-12T22:21:12.011 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl stop ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968@osd.4 2024-05-12T22:21:12.325 INFO:journalctl@ceph.osd.4.smithi086.stdout:May 12 22:21:12 smithi086 systemd[1]: Stopping Ceph osd.4 for 6b92e7b4-10aa-11ef-bc98-c7b262605968... 2024-05-12T22:21:12.326 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:12 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:12.105+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:43.519531+0000 front 2024-05-12T22:20:43.519638+0000 (oldest deadline 2024-05-12T22:21:05.219288+0000) 2024-05-12T22:21:12.535 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 12 22:21:12 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-6[55080]: 2024-05-12T22:21:12.183+0000 7f2fefe00700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:43.392022+0000 front 2024-05-12T22:20:43.392513+0000 (oldest deadline 2024-05-12T22:21:08.692004+0000) 2024-05-12T22:21:12.604 INFO:journalctl@ceph.osd.5.smithi186.stdout:May 12 22:21:12 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-5[46962]: 2024-05-12T22:21:12.298+0000 7f82e356f700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.993114+0000 front 2024-05-12T22:20:41.993039+0000 (oldest deadline 2024-05-12T22:21:07.193555+0000) 2024-05-12T22:21:12.742 INFO:journalctl@ceph.osd.4.smithi086.stdout:May 12 22:21:12 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-4[45634]: 2024-05-12T22:21:12.325+0000 7f74ed308700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-12T22:21:12.742 INFO:journalctl@ceph.osd.4.smithi086.stdout:May 12 22:21:12 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-4[45634]: 2024-05-12T22:21:12.325+0000 7f74ed308700 -1 osd.4 102 *** Got signal Terminated *** 2024-05-12T22:21:12.742 INFO:journalctl@ceph.osd.4.smithi086.stdout:May 12 22:21:12 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-4[45634]: 2024-05-12T22:21:12.325+0000 7f74ed308700 -1 osd.4 102 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-12T22:21:12.929 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:12 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:12.605+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.561281+0000 front 2024-05-12T22:20:41.561023+0000 (oldest deadline 2024-05-12T22:21:06.261036+0000) 2024-05-12T22:21:13.035 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:12 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:12.710+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.490538+0000 front 2024-05-12T22:20:41.490674+0000 (oldest deadline 2024-05-12T22:21:04.990968+0000) 2024-05-12T22:21:13.492 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:13 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:13.142+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:43.519531+0000 front 2024-05-12T22:20:43.519638+0000 (oldest deadline 2024-05-12T22:21:05.219288+0000) 2024-05-12T22:21:13.536 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 12 22:21:13 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-6[55080]: 2024-05-12T22:21:13.178+0000 7f2fefe00700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:43.392022+0000 front 2024-05-12T22:20:43.392513+0000 (oldest deadline 2024-05-12T22:21:08.692004+0000) 2024-05-12T22:21:13.603 INFO:journalctl@ceph.osd.5.smithi186.stdout:May 12 22:21:13 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-5[46962]: 2024-05-12T22:21:13.250+0000 7f82e356f700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.993114+0000 front 2024-05-12T22:20:41.993039+0000 (oldest deadline 2024-05-12T22:21:07.193555+0000) 2024-05-12T22:21:13.603 INFO:journalctl@ceph.osd.5.smithi186.stdout:May 12 22:21:13 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-5[46962]: 2024-05-12T22:21:13.250+0000 7f82e356f700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:47.194145+0000 front 2024-05-12T22:20:47.194079+0000 (oldest deadline 2024-05-12T22:21:13.094071+0000) 2024-05-12T22:21:13.930 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:13 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:13.601+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.561281+0000 front 2024-05-12T22:20:41.561023+0000 (oldest deadline 2024-05-12T22:21:06.261036+0000) 2024-05-12T22:21:13.931 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:13 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:13.601+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:50.163666+0000 front 2024-05-12T22:20:50.163649+0000 (oldest deadline 2024-05-12T22:21:13.063447+0000) 2024-05-12T22:21:14.035 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:13 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:13.759+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.490538+0000 front 2024-05-12T22:20:41.490674+0000 (oldest deadline 2024-05-12T22:21:04.990968+0000) 2024-05-12T22:21:14.242 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 12 22:21:13 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-7[52481]: 2024-05-12T22:21:13.911+0000 7f1b778cf700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:51.904377+0000 front 2024-05-12T22:20:51.904452+0000 (oldest deadline 2024-05-12T22:21:13.604466+0000) 2024-05-12T22:21:14.242 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:14 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:14.150+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:43.519531+0000 front 2024-05-12T22:20:43.519638+0000 (oldest deadline 2024-05-12T22:21:05.219288+0000) 2024-05-12T22:21:14.535 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 12 22:21:14 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-6[55080]: 2024-05-12T22:21:14.201+0000 7f2fefe00700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:43.392022+0000 front 2024-05-12T22:20:43.392513+0000 (oldest deadline 2024-05-12T22:21:08.692004+0000) 2024-05-12T22:21:14.606 INFO:journalctl@ceph.osd.5.smithi186.stdout:May 12 22:21:14 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-5[46962]: 2024-05-12T22:21:14.241+0000 7f82e356f700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.993114+0000 front 2024-05-12T22:20:41.993039+0000 (oldest deadline 2024-05-12T22:21:07.193555+0000) 2024-05-12T22:21:14.607 INFO:journalctl@ceph.osd.5.smithi186.stdout:May 12 22:21:14 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-5[46962]: 2024-05-12T22:21:14.241+0000 7f82e356f700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:47.194145+0000 front 2024-05-12T22:20:47.194079+0000 (oldest deadline 2024-05-12T22:21:13.094071+0000) 2024-05-12T22:21:14.742 INFO:journalctl@ceph.osd.4.smithi086.stdout:May 12 22:21:14 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-4[45634]: 2024-05-12T22:21:14.398+0000 7f74ea0fd700 -1 osd.4 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:48.896694+0000 front 2024-05-12T22:20:48.896490+0000 (oldest deadline 2024-05-12T22:21:13.596640+0000) 2024-05-12T22:21:14.929 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:14 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:14.606+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.561281+0000 front 2024-05-12T22:20:41.561023+0000 (oldest deadline 2024-05-12T22:21:06.261036+0000) 2024-05-12T22:21:14.929 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:14 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:14.606+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:50.163666+0000 front 2024-05-12T22:20:50.163649+0000 (oldest deadline 2024-05-12T22:21:13.063447+0000) 2024-05-12T22:21:15.035 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:14 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:14.725+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.490538+0000 front 2024-05-12T22:20:41.490674+0000 (oldest deadline 2024-05-12T22:21:04.990968+0000) 2024-05-12T22:21:15.035 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:14 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:14.725+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:48.493555+0000 front 2024-05-12T22:20:48.493616+0000 (oldest deadline 2024-05-12T22:21:14.393780+0000) 2024-05-12T22:21:15.138 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 12 22:21:14 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-7[52481]: 2024-05-12T22:21:14.861+0000 7f1b778cf700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:51.904377+0000 front 2024-05-12T22:20:51.904452+0000 (oldest deadline 2024-05-12T22:21:13.604466+0000) 2024-05-12T22:21:15.492 INFO:journalctl@ceph.osd.4.smithi086.stdout:May 12 22:21:15 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-4[45634]: 2024-05-12T22:21:15.353+0000 7f74ea0fd700 -1 osd.4 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:48.896694+0000 front 2024-05-12T22:20:48.896490+0000 (oldest deadline 2024-05-12T22:21:13.596640+0000) 2024-05-12T22:21:15.492 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:15 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:15.137+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:43.519531+0000 front 2024-05-12T22:20:43.519638+0000 (oldest deadline 2024-05-12T22:21:05.219288+0000) 2024-05-12T22:21:15.535 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 12 22:21:15 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-6[55080]: 2024-05-12T22:21:15.217+0000 7f2fefe00700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:43.392022+0000 front 2024-05-12T22:20:43.392513+0000 (oldest deadline 2024-05-12T22:21:08.692004+0000) 2024-05-12T22:21:15.596 INFO:journalctl@ceph.osd.5.smithi186.stdout:May 12 22:21:15 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-5[46962]: 2024-05-12T22:21:15.243+0000 7f82e356f700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.993114+0000 front 2024-05-12T22:20:41.993039+0000 (oldest deadline 2024-05-12T22:21:07.193555+0000) 2024-05-12T22:21:15.597 INFO:journalctl@ceph.osd.5.smithi186.stdout:May 12 22:21:15 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-5[46962]: 2024-05-12T22:21:15.243+0000 7f82e356f700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:47.194145+0000 front 2024-05-12T22:20:47.194079+0000 (oldest deadline 2024-05-12T22:21:13.094071+0000) 2024-05-12T22:21:15.929 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:15 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:15.595+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.561281+0000 front 2024-05-12T22:20:41.561023+0000 (oldest deadline 2024-05-12T22:21:06.261036+0000) 2024-05-12T22:21:15.929 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:15 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:15.595+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:50.163666+0000 front 2024-05-12T22:20:50.163649+0000 (oldest deadline 2024-05-12T22:21:13.063447+0000) 2024-05-12T22:21:16.035 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:15 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:15.753+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.490538+0000 front 2024-05-12T22:20:41.490674+0000 (oldest deadline 2024-05-12T22:21:04.990968+0000) 2024-05-12T22:21:16.035 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:15 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:15.753+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:48.493555+0000 front 2024-05-12T22:20:48.493616+0000 (oldest deadline 2024-05-12T22:21:14.393780+0000) 2024-05-12T22:21:16.148 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 12 22:21:15 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-7[52481]: 2024-05-12T22:21:15.851+0000 7f1b778cf700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:51.904377+0000 front 2024-05-12T22:20:51.904452+0000 (oldest deadline 2024-05-12T22:21:13.604466+0000) 2024-05-12T22:21:16.398 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:16 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:16.148+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:43.519531+0000 front 2024-05-12T22:20:43.519638+0000 (oldest deadline 2024-05-12T22:21:05.219288+0000) 2024-05-12T22:21:16.535 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 12 22:21:16 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-6[55080]: 2024-05-12T22:21:16.239+0000 7f2fefe00700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:43.392022+0000 front 2024-05-12T22:20:43.392513+0000 (oldest deadline 2024-05-12T22:21:08.692004+0000) 2024-05-12T22:21:16.632 INFO:journalctl@ceph.osd.5.smithi186.stdout:May 12 22:21:16 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-5[46962]: 2024-05-12T22:21:16.275+0000 7f82e356f700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.993114+0000 front 2024-05-12T22:20:41.993039+0000 (oldest deadline 2024-05-12T22:21:07.193555+0000) 2024-05-12T22:21:16.633 INFO:journalctl@ceph.osd.5.smithi186.stdout:May 12 22:21:16 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-5[46962]: 2024-05-12T22:21:16.275+0000 7f82e356f700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:47.194145+0000 front 2024-05-12T22:20:47.194079+0000 (oldest deadline 2024-05-12T22:21:13.094071+0000) 2024-05-12T22:21:16.742 INFO:journalctl@ceph.osd.4.smithi086.stdout:May 12 22:21:16 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-4[45634]: 2024-05-12T22:21:16.398+0000 7f74ea0fd700 -1 osd.4 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:48.896694+0000 front 2024-05-12T22:20:48.896490+0000 (oldest deadline 2024-05-12T22:21:13.596640+0000) 2024-05-12T22:21:16.893 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:16 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:16.632+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.561281+0000 front 2024-05-12T22:20:41.561023+0000 (oldest deadline 2024-05-12T22:21:06.261036+0000) 2024-05-12T22:21:16.894 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:16 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:16.632+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:50.163666+0000 front 2024-05-12T22:20:50.163649+0000 (oldest deadline 2024-05-12T22:21:13.063447+0000) 2024-05-12T22:21:17.120 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 12 22:21:16 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-7[52481]: 2024-05-12T22:21:16.808+0000 7f1b778cf700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:51.904377+0000 front 2024-05-12T22:20:51.904452+0000 (oldest deadline 2024-05-12T22:21:13.604466+0000) 2024-05-12T22:21:17.179 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 12 22:21:16 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-8[53963]: 2024-05-12T22:21:16.892+0000 7f4ff7d79700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:50.580039+0000 front 2024-05-12T22:20:50.580181+0000 (oldest deadline 2024-05-12T22:21:16.480068+0000) 2024-05-12T22:21:17.269 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:16 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:16.785+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.490538+0000 front 2024-05-12T22:20:41.490674+0000 (oldest deadline 2024-05-12T22:21:04.990968+0000) 2024-05-12T22:21:17.269 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:16 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:16.785+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:48.493555+0000 front 2024-05-12T22:20:48.493616+0000 (oldest deadline 2024-05-12T22:21:14.393780+0000) 2024-05-12T22:21:17.492 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:17 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:17.120+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:43.519531+0000 front 2024-05-12T22:20:43.519638+0000 (oldest deadline 2024-05-12T22:21:05.219288+0000) 2024-05-12T22:21:17.492 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:17 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:17.120+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:51.620499+0000 front 2024-05-12T22:20:51.620463+0000 (oldest deadline 2024-05-12T22:21:16.221349+0000) 2024-05-12T22:21:17.535 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 12 22:21:17 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-6[55080]: 2024-05-12T22:21:17.269+0000 7f2fefe00700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:43.392022+0000 front 2024-05-12T22:20:43.392513+0000 (oldest deadline 2024-05-12T22:21:08.692004+0000) 2024-05-12T22:21:17.535 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 12 22:21:17 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-6[55080]: 2024-05-12T22:21:17.269+0000 7f2fefe00700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:50.994527+0000 front 2024-05-12T22:20:50.994611+0000 (oldest deadline 2024-05-12T22:21:16.894684+0000) 2024-05-12T22:21:17.636 INFO:journalctl@ceph.osd.5.smithi186.stdout:May 12 22:21:17 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-5[46962]: 2024-05-12T22:21:17.227+0000 7f82e356f700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.993114+0000 front 2024-05-12T22:20:41.993039+0000 (oldest deadline 2024-05-12T22:21:07.193555+0000) 2024-05-12T22:21:17.637 INFO:journalctl@ceph.osd.5.smithi186.stdout:May 12 22:21:17 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-5[46962]: 2024-05-12T22:21:17.227+0000 7f82e356f700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:47.194145+0000 front 2024-05-12T22:20:47.194079+0000 (oldest deadline 2024-05-12T22:21:13.094071+0000) 2024-05-12T22:21:17.929 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:17 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:17.635+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.561281+0000 front 2024-05-12T22:20:41.561023+0000 (oldest deadline 2024-05-12T22:21:06.261036+0000) 2024-05-12T22:21:17.929 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:17 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:17.635+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:50.163666+0000 front 2024-05-12T22:20:50.163649+0000 (oldest deadline 2024-05-12T22:21:13.063447+0000) 2024-05-12T22:21:18.076 INFO:journalctl@ceph.osd.4.smithi086.stdout:May 12 22:21:17 smithi086 podman[70776]: 2024-05-12 22:21:17.841631992 +0000 UTC m=+5.630995002 container died db131f4bd6a14c8203573f20b6e6ed21e4355765209175bf0efe3aec7ecc0646 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-4, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-f72fecf, io.buildah.version=1.33.5, org.label-schema.build-date=20240507, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-05-12T22:21:18.076 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 12 22:21:17 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-7[52481]: 2024-05-12T22:21:17.823+0000 7f1b778cf700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:51.904377+0000 front 2024-05-12T22:20:51.904452+0000 (oldest deadline 2024-05-12T22:21:13.604466+0000) 2024-05-12T22:21:18.232 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 12 22:21:17 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-8[53963]: 2024-05-12T22:21:17.931+0000 7f4ff7d79700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:50.580039+0000 front 2024-05-12T22:20:50.580181+0000 (oldest deadline 2024-05-12T22:21:16.480068+0000) 2024-05-12T22:21:18.285 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:17 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:17.806+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.490538+0000 front 2024-05-12T22:20:41.490674+0000 (oldest deadline 2024-05-12T22:21:04.990968+0000) 2024-05-12T22:21:18.285 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:17 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:17.806+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:48.493555+0000 front 2024-05-12T22:20:48.493616+0000 (oldest deadline 2024-05-12T22:21:14.393780+0000) 2024-05-12T22:21:18.384 INFO:journalctl@ceph.osd.4.smithi086.stdout:May 12 22:21:18 smithi086 podman[70776]: 2024-05-12 22:21:18.134145329 +0000 UTC m=+5.923508326 container cleanup db131f4bd6a14c8203573f20b6e6ed21e4355765209175bf0efe3aec7ecc0646 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-4, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=squid-f72fecf, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240507) 2024-05-12T22:21:18.384 INFO:journalctl@ceph.osd.4.smithi086.stdout:May 12 22:21:18 smithi086 bash[70776]: ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-4 2024-05-12T22:21:18.385 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:18 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:18.103+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:43.519531+0000 front 2024-05-12T22:20:43.519638+0000 (oldest deadline 2024-05-12T22:21:05.219288+0000) 2024-05-12T22:21:18.385 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:18 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:18.103+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:51.620499+0000 front 2024-05-12T22:20:51.620463+0000 (oldest deadline 2024-05-12T22:21:16.221349+0000) 2024-05-12T22:21:18.618 INFO:journalctl@ceph.osd.5.smithi186.stdout:May 12 22:21:18 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-5[46962]: 2024-05-12T22:21:18.231+0000 7f82e356f700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.993114+0000 front 2024-05-12T22:20:41.993039+0000 (oldest deadline 2024-05-12T22:21:07.193555+0000) 2024-05-12T22:21:18.618 INFO:journalctl@ceph.osd.5.smithi186.stdout:May 12 22:21:18 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-5[46962]: 2024-05-12T22:21:18.231+0000 7f82e356f700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:47.194145+0000 front 2024-05-12T22:20:47.194079+0000 (oldest deadline 2024-05-12T22:21:13.094071+0000) 2024-05-12T22:21:18.742 INFO:journalctl@ceph.osd.4.smithi086.stdout:May 12 22:21:18 smithi086 podman[70793]: 2024-05-12 22:21:18.385175309 +0000 UTC m=+0.538143139 container remove db131f4bd6a14c8203573f20b6e6ed21e4355765209175bf0efe3aec7ecc0646 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-4, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240507, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=squid-f72fecf, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-05-12T22:21:18.786 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 12 22:21:18 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-6[55080]: 2024-05-12T22:21:18.293+0000 7f2fefe00700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:43.392022+0000 front 2024-05-12T22:20:43.392513+0000 (oldest deadline 2024-05-12T22:21:08.692004+0000) 2024-05-12T22:21:18.786 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 12 22:21:18 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-6[55080]: 2024-05-12T22:21:18.293+0000 7f2fefe00700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:50.994527+0000 front 2024-05-12T22:20:50.994611+0000 (oldest deadline 2024-05-12T22:21:16.894684+0000) 2024-05-12T22:21:18.907 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:18 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:18.618+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.561281+0000 front 2024-05-12T22:20:41.561023+0000 (oldest deadline 2024-05-12T22:21:06.261036+0000) 2024-05-12T22:21:18.907 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:18 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:18.618+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:50.163666+0000 front 2024-05-12T22:20:50.163649+0000 (oldest deadline 2024-05-12T22:21:13.063447+0000) 2024-05-12T22:21:19.062 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 12 22:21:18 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-7[52481]: 2024-05-12T22:21:18.838+0000 7f1b778cf700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:51.904377+0000 front 2024-05-12T22:20:51.904452+0000 (oldest deadline 2024-05-12T22:21:13.604466+0000) 2024-05-12T22:21:19.063 INFO:journalctl@ceph.osd.4.smithi086.stdout:May 12 22:21:18 smithi086 podman[70887]: 2024-05-12 22:21:18.650383204 +0000 UTC m=+0.018358542 image pull 8252b7944d3b5c188c0ff632647ba6cc4e9a19588ee097b8a3dba27b42779c25 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc 2024-05-12T22:21:19.063 INFO:journalctl@ceph.osd.4.smithi086.stdout:May 12 22:21:18 smithi086 podman[70887]: 2024-05-12 22:21:18.769663639 +0000 UTC m=+0.137638963 container create 2a8cbd48316765bf4e85b717e36c8203a122dc6fb74fdf6c2a04fbe0ff07da4b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-4-deactivate, org.label-schema.build-date=20240507, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, RELEASE=squid-f72fecf, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-05-12T22:21:19.063 INFO:journalctl@ceph.osd.4.smithi086.stdout:May 12 22:21:18 smithi086 podman[70887]: 2024-05-12 22:21:18.988508851 +0000 UTC m=+0.356484176 container init 2a8cbd48316765bf4e85b717e36c8203a122dc6fb74fdf6c2a04fbe0ff07da4b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-4-deactivate, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240507, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=squid-f72fecf, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, io.buildah.version=1.33.5) 2024-05-12T22:21:19.063 INFO:journalctl@ceph.osd.4.smithi086.stdout:May 12 22:21:18 smithi086 podman[70887]: 2024-05-12 22:21:18.991592015 +0000 UTC m=+0.359567341 container start 2a8cbd48316765bf4e85b717e36c8203a122dc6fb74fdf6c2a04fbe0ff07da4b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-4-deactivate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240507, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=squid-f72fecf, org.label-schema.vendor=CentOS) 2024-05-12T22:21:19.064 INFO:journalctl@ceph.osd.4.smithi086.stdout:May 12 22:21:19 smithi086 podman[70887]: 2024-05-12 22:21:19.063407013 +0000 UTC m=+0.431382357 container attach 2a8cbd48316765bf4e85b717e36c8203a122dc6fb74fdf6c2a04fbe0ff07da4b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-4-deactivate, org.label-schema.build-date=20240507, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=squid-f72fecf, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-12T22:21:19.179 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 12 22:21:18 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-8[53963]: 2024-05-12T22:21:18.906+0000 7f4ff7d79700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:50.580039+0000 front 2024-05-12T22:20:50.580181+0000 (oldest deadline 2024-05-12T22:21:16.480068+0000) 2024-05-12T22:21:19.285 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:18 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:18.854+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.490538+0000 front 2024-05-12T22:20:41.490674+0000 (oldest deadline 2024-05-12T22:21:04.990968+0000) 2024-05-12T22:21:19.285 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:18 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:18.854+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:48.493555+0000 front 2024-05-12T22:20:48.493616+0000 (oldest deadline 2024-05-12T22:21:14.393780+0000) 2024-05-12T22:21:19.492 INFO:journalctl@ceph.osd.4.smithi086.stdout:May 12 22:21:19 smithi086 podman[70887]: 2024-05-12 22:21:19.282482028 +0000 UTC m=+0.650457354 container died 2a8cbd48316765bf4e85b717e36c8203a122dc6fb74fdf6c2a04fbe0ff07da4b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-4-deactivate, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-f72fecf, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240507, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e) 2024-05-12T22:21:19.493 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:19 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:19.066+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:43.519531+0000 front 2024-05-12T22:20:43.519638+0000 (oldest deadline 2024-05-12T22:21:05.219288+0000) 2024-05-12T22:21:19.493 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:19 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:19.066+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:51.620499+0000 front 2024-05-12T22:20:51.620463+0000 (oldest deadline 2024-05-12T22:21:16.221349+0000) 2024-05-12T22:21:19.624 INFO:journalctl@ceph.osd.5.smithi186.stdout:May 12 22:21:19 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-5[46962]: 2024-05-12T22:21:19.208+0000 7f82e356f700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.993114+0000 front 2024-05-12T22:20:41.993039+0000 (oldest deadline 2024-05-12T22:21:07.193555+0000) 2024-05-12T22:21:19.625 INFO:journalctl@ceph.osd.5.smithi186.stdout:May 12 22:21:19 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-5[46962]: 2024-05-12T22:21:19.208+0000 7f82e356f700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:47.194145+0000 front 2024-05-12T22:20:47.194079+0000 (oldest deadline 2024-05-12T22:21:13.094071+0000) 2024-05-12T22:21:19.785 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 12 22:21:19 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-6[55080]: 2024-05-12T22:21:19.302+0000 7f2fefe00700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:43.392022+0000 front 2024-05-12T22:20:43.392513+0000 (oldest deadline 2024-05-12T22:21:08.692004+0000) 2024-05-12T22:21:19.785 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 12 22:21:19 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-6[55080]: 2024-05-12T22:21:19.302+0000 7f2fefe00700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:50.994527+0000 front 2024-05-12T22:20:50.994611+0000 (oldest deadline 2024-05-12T22:21:16.894684+0000) 2024-05-12T22:21:19.838 INFO:journalctl@ceph.osd.4.smithi086.stdout:May 12 22:21:19 smithi086 podman[70913]: 2024-05-12 22:21:19.8252348 +0000 UTC m=+0.536734930 container remove 2a8cbd48316765bf4e85b717e36c8203a122dc6fb74fdf6c2a04fbe0ff07da4b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-4-deactivate, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, RELEASE=squid-f72fecf, org.label-schema.build-date=20240507) 2024-05-12T22:21:19.844 DEBUG:teuthology.orchestra.run.smithi086:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968@osd.4.service' 2024-05-12T22:21:19.883 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-12T22:21:19.883 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-05-12T22:21:19.883 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-05-12T22:21:19.883 DEBUG:teuthology.orchestra.run.smithi186:> sudo systemctl stop ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968@osd.5 2024-05-12T22:21:19.889 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:19 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:19.624+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.561281+0000 front 2024-05-12T22:20:41.561023+0000 (oldest deadline 2024-05-12T22:21:06.261036+0000) 2024-05-12T22:21:19.889 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:19 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:19.624+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:50.163666+0000 front 2024-05-12T22:20:50.163649+0000 (oldest deadline 2024-05-12T22:21:13.063447+0000) 2024-05-12T22:21:20.107 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 12 22:21:19 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-7[52481]: 2024-05-12T22:21:19.860+0000 7f1b778cf700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:51.904377+0000 front 2024-05-12T22:20:51.904452+0000 (oldest deadline 2024-05-12T22:21:13.604466+0000) 2024-05-12T22:21:20.176 INFO:journalctl@ceph.osd.5.smithi186.stdout:May 12 22:21:19 smithi186 systemd[1]: Stopping Ceph osd.5 for 6b92e7b4-10aa-11ef-bc98-c7b262605968... 2024-05-12T22:21:20.177 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 12 22:21:19 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-8[53963]: 2024-05-12T22:21:19.888+0000 7f4ff7d79700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:50.580039+0000 front 2024-05-12T22:20:50.580181+0000 (oldest deadline 2024-05-12T22:21:16.480068+0000) 2024-05-12T22:21:20.285 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:19 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:19.875+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.490538+0000 front 2024-05-12T22:20:41.490674+0000 (oldest deadline 2024-05-12T22:21:04.990968+0000) 2024-05-12T22:21:20.285 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:19 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:19.875+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:48.493555+0000 front 2024-05-12T22:20:48.493616+0000 (oldest deadline 2024-05-12T22:21:14.393780+0000) 2024-05-12T22:21:20.429 INFO:journalctl@ceph.osd.5.smithi186.stdout:May 12 22:21:20 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-5[46962]: 2024-05-12T22:21:20.175+0000 7f82e677a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-12T22:21:20.430 INFO:journalctl@ceph.osd.5.smithi186.stdout:May 12 22:21:20 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-5[46962]: 2024-05-12T22:21:20.175+0000 7f82e677a700 -1 osd.5 102 *** Got signal Terminated *** 2024-05-12T22:21:20.430 INFO:journalctl@ceph.osd.5.smithi186.stdout:May 12 22:21:20 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-5[46962]: 2024-05-12T22:21:20.175+0000 7f82e677a700 -1 osd.5 102 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-12T22:21:20.430 INFO:journalctl@ceph.osd.5.smithi186.stdout:May 12 22:21:20 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-5[46962]: 2024-05-12T22:21:20.197+0000 7f82e356f700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.993114+0000 front 2024-05-12T22:20:41.993039+0000 (oldest deadline 2024-05-12T22:21:07.193555+0000) 2024-05-12T22:21:20.430 INFO:journalctl@ceph.osd.5.smithi186.stdout:May 12 22:21:20 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-5[46962]: 2024-05-12T22:21:20.197+0000 7f82e356f700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:47.194145+0000 front 2024-05-12T22:20:47.194079+0000 (oldest deadline 2024-05-12T22:21:13.094071+0000) 2024-05-12T22:21:20.492 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:20 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:20.107+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:43.519531+0000 front 2024-05-12T22:20:43.519638+0000 (oldest deadline 2024-05-12T22:21:05.219288+0000) 2024-05-12T22:21:20.492 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:20 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:20.107+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:51.620499+0000 front 2024-05-12T22:20:51.620463+0000 (oldest deadline 2024-05-12T22:21:16.221349+0000) 2024-05-12T22:21:20.785 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 12 22:21:20 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-6[55080]: 2024-05-12T22:21:20.302+0000 7f2fefe00700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:43.392022+0000 front 2024-05-12T22:20:43.392513+0000 (oldest deadline 2024-05-12T22:21:08.692004+0000) 2024-05-12T22:21:20.785 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 12 22:21:20 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-6[55080]: 2024-05-12T22:21:20.302+0000 7f2fefe00700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:50.994527+0000 front 2024-05-12T22:20:50.994611+0000 (oldest deadline 2024-05-12T22:21:16.894684+0000) 2024-05-12T22:21:20.876 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:20 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:20.613+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.561281+0000 front 2024-05-12T22:20:41.561023+0000 (oldest deadline 2024-05-12T22:21:06.261036+0000) 2024-05-12T22:21:20.876 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:20 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:20.613+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:50.163666+0000 front 2024-05-12T22:20:50.163649+0000 (oldest deadline 2024-05-12T22:21:13.063447+0000) 2024-05-12T22:21:21.151 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 12 22:21:20 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-7[52481]: 2024-05-12T22:21:20.819+0000 7f1b778cf700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:51.904377+0000 front 2024-05-12T22:20:51.904452+0000 (oldest deadline 2024-05-12T22:21:13.604466+0000) 2024-05-12T22:21:21.179 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 12 22:21:20 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-8[53963]: 2024-05-12T22:21:20.875+0000 7f4ff7d79700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:50.580039+0000 front 2024-05-12T22:20:50.580181+0000 (oldest deadline 2024-05-12T22:21:16.480068+0000) 2024-05-12T22:21:21.270 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:20 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:20.829+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.490538+0000 front 2024-05-12T22:20:41.490674+0000 (oldest deadline 2024-05-12T22:21:04.990968+0000) 2024-05-12T22:21:21.270 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:20 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:20.829+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:48.493555+0000 front 2024-05-12T22:20:48.493616+0000 (oldest deadline 2024-05-12T22:21:14.393780+0000) 2024-05-12T22:21:21.492 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:21 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:21.151+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:43.519531+0000 front 2024-05-12T22:20:43.519638+0000 (oldest deadline 2024-05-12T22:21:05.219288+0000) 2024-05-12T22:21:21.492 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:21 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:21.151+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:51.620499+0000 front 2024-05-12T22:20:51.620463+0000 (oldest deadline 2024-05-12T22:21:16.221349+0000) 2024-05-12T22:21:21.535 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 12 22:21:21 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-6[55080]: 2024-05-12T22:21:21.278+0000 7f2fefe00700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:43.392022+0000 front 2024-05-12T22:20:43.392513+0000 (oldest deadline 2024-05-12T22:21:08.692004+0000) 2024-05-12T22:21:21.535 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 12 22:21:21 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-6[55080]: 2024-05-12T22:21:21.278+0000 7f2fefe00700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:50.994527+0000 front 2024-05-12T22:20:50.994611+0000 (oldest deadline 2024-05-12T22:21:16.894684+0000) 2024-05-12T22:21:21.567 INFO:journalctl@ceph.osd.5.smithi186.stdout:May 12 22:21:21 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-5[46962]: 2024-05-12T22:21:21.220+0000 7f82e356f700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.993114+0000 front 2024-05-12T22:20:41.993039+0000 (oldest deadline 2024-05-12T22:21:07.193555+0000) 2024-05-12T22:21:21.568 INFO:journalctl@ceph.osd.5.smithi186.stdout:May 12 22:21:21 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-5[46962]: 2024-05-12T22:21:21.220+0000 7f82e356f700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:47.194145+0000 front 2024-05-12T22:20:47.194079+0000 (oldest deadline 2024-05-12T22:21:13.094071+0000) 2024-05-12T22:21:21.833 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:21 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:21.567+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.561281+0000 front 2024-05-12T22:20:41.561023+0000 (oldest deadline 2024-05-12T22:21:06.261036+0000) 2024-05-12T22:21:21.833 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:21 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:21.567+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:50.163666+0000 front 2024-05-12T22:20:50.163649+0000 (oldest deadline 2024-05-12T22:21:13.063447+0000) 2024-05-12T22:21:22.115 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 12 22:21:21 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-7[52481]: 2024-05-12T22:21:21.773+0000 7f1b778cf700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:51.904377+0000 front 2024-05-12T22:20:51.904452+0000 (oldest deadline 2024-05-12T22:21:13.604466+0000) 2024-05-12T22:21:22.179 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 12 22:21:21 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-8[53963]: 2024-05-12T22:21:21.833+0000 7f4ff7d79700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:50.580039+0000 front 2024-05-12T22:20:50.580181+0000 (oldest deadline 2024-05-12T22:21:16.480068+0000) 2024-05-12T22:21:22.179 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 12 22:21:21 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-8[53963]: 2024-05-12T22:21:21.833+0000 7f4ff7d79700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:56.480473+0000 front 2024-05-12T22:20:56.480615+0000 (oldest deadline 2024-05-12T22:21:21.780524+0000) 2024-05-12T22:21:22.262 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:21 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:21.830+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.490538+0000 front 2024-05-12T22:20:41.490674+0000 (oldest deadline 2024-05-12T22:21:04.990968+0000) 2024-05-12T22:21:22.262 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:21 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:21.830+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:48.493555+0000 front 2024-05-12T22:20:48.493616+0000 (oldest deadline 2024-05-12T22:21:14.393780+0000) 2024-05-12T22:21:22.262 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:21 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:21.830+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:56.095925+0000 front 2024-05-12T22:20:56.096412+0000 (oldest deadline 2024-05-12T22:21:21.396155+0000) 2024-05-12T22:21:22.492 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:22 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:22.114+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:43.519531+0000 front 2024-05-12T22:20:43.519638+0000 (oldest deadline 2024-05-12T22:21:05.219288+0000) 2024-05-12T22:21:22.492 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:22 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:22.114+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:51.620499+0000 front 2024-05-12T22:20:51.620463+0000 (oldest deadline 2024-05-12T22:21:16.221349+0000) 2024-05-12T22:21:22.492 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:22 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:22.114+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:59.722447+0000 front 2024-05-12T22:20:59.722655+0000 (oldest deadline 2024-05-12T22:21:21.422316+0000) 2024-05-12T22:21:22.535 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 12 22:21:22 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-6[55080]: 2024-05-12T22:21:22.262+0000 7f2fefe00700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:43.392022+0000 front 2024-05-12T22:20:43.392513+0000 (oldest deadline 2024-05-12T22:21:08.692004+0000) 2024-05-12T22:21:22.535 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 12 22:21:22 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-6[55080]: 2024-05-12T22:21:22.262+0000 7f2fefe00700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:50.994527+0000 front 2024-05-12T22:20:50.994611+0000 (oldest deadline 2024-05-12T22:21:16.894684+0000) 2024-05-12T22:21:22.564 INFO:journalctl@ceph.osd.5.smithi186.stdout:May 12 22:21:22 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-5[46962]: 2024-05-12T22:21:22.243+0000 7f82e356f700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.993114+0000 front 2024-05-12T22:20:41.993039+0000 (oldest deadline 2024-05-12T22:21:07.193555+0000) 2024-05-12T22:21:22.564 INFO:journalctl@ceph.osd.5.smithi186.stdout:May 12 22:21:22 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-5[46962]: 2024-05-12T22:21:22.243+0000 7f82e356f700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:47.194145+0000 front 2024-05-12T22:20:47.194079+0000 (oldest deadline 2024-05-12T22:21:13.094071+0000) 2024-05-12T22:21:22.564 INFO:journalctl@ceph.osd.5.smithi186.stdout:May 12 22:21:22 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-5[46962]: 2024-05-12T22:21:22.243+0000 7f82e356f700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:58.796131+0000 front 2024-05-12T22:20:58.796209+0000 (oldest deadline 2024-05-12T22:21:21.696164+0000) 2024-05-12T22:21:22.843 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:22 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:22.563+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.561281+0000 front 2024-05-12T22:20:41.561023+0000 (oldest deadline 2024-05-12T22:21:06.261036+0000) 2024-05-12T22:21:22.844 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:22 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:22.563+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:50.163666+0000 front 2024-05-12T22:20:50.163649+0000 (oldest deadline 2024-05-12T22:21:13.063447+0000) 2024-05-12T22:21:23.130 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 12 22:21:22 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-7[52481]: 2024-05-12T22:21:22.786+0000 7f1b778cf700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:51.904377+0000 front 2024-05-12T22:20:51.904452+0000 (oldest deadline 2024-05-12T22:21:13.604466+0000) 2024-05-12T22:21:23.179 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 12 22:21:22 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-8[53963]: 2024-05-12T22:21:22.842+0000 7f4ff7d79700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:50.580039+0000 front 2024-05-12T22:20:50.580181+0000 (oldest deadline 2024-05-12T22:21:16.480068+0000) 2024-05-12T22:21:23.179 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 12 22:21:22 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-8[53963]: 2024-05-12T22:21:22.842+0000 7f4ff7d79700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:56.480473+0000 front 2024-05-12T22:20:56.480615+0000 (oldest deadline 2024-05-12T22:21:21.780524+0000) 2024-05-12T22:21:23.265 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:22 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:22.810+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.490538+0000 front 2024-05-12T22:20:41.490674+0000 (oldest deadline 2024-05-12T22:21:04.990968+0000) 2024-05-12T22:21:23.266 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:22 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:22.810+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:48.493555+0000 front 2024-05-12T22:20:48.493616+0000 (oldest deadline 2024-05-12T22:21:14.393780+0000) 2024-05-12T22:21:23.266 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:22 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:22.810+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:56.095925+0000 front 2024-05-12T22:20:56.096412+0000 (oldest deadline 2024-05-12T22:21:21.396155+0000) 2024-05-12T22:21:23.492 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:23 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:23.129+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:43.519531+0000 front 2024-05-12T22:20:43.519638+0000 (oldest deadline 2024-05-12T22:21:05.219288+0000) 2024-05-12T22:21:23.492 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:23 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:23.129+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:51.620499+0000 front 2024-05-12T22:20:51.620463+0000 (oldest deadline 2024-05-12T22:21:16.221349+0000) 2024-05-12T22:21:23.492 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:23 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:23.129+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:59.722447+0000 front 2024-05-12T22:20:59.722655+0000 (oldest deadline 2024-05-12T22:21:21.422316+0000) 2024-05-12T22:21:23.535 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 12 22:21:23 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-6[55080]: 2024-05-12T22:21:23.265+0000 7f2fefe00700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:43.392022+0000 front 2024-05-12T22:20:43.392513+0000 (oldest deadline 2024-05-12T22:21:08.692004+0000) 2024-05-12T22:21:23.535 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 12 22:21:23 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-6[55080]: 2024-05-12T22:21:23.265+0000 7f2fefe00700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:50.994527+0000 front 2024-05-12T22:20:50.994611+0000 (oldest deadline 2024-05-12T22:21:16.894684+0000) 2024-05-12T22:21:23.583 INFO:journalctl@ceph.osd.5.smithi186.stdout:May 12 22:21:23 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-5[46962]: 2024-05-12T22:21:23.248+0000 7f82e356f700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.993114+0000 front 2024-05-12T22:20:41.993039+0000 (oldest deadline 2024-05-12T22:21:07.193555+0000) 2024-05-12T22:21:23.583 INFO:journalctl@ceph.osd.5.smithi186.stdout:May 12 22:21:23 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-5[46962]: 2024-05-12T22:21:23.248+0000 7f82e356f700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:47.194145+0000 front 2024-05-12T22:20:47.194079+0000 (oldest deadline 2024-05-12T22:21:13.094071+0000) 2024-05-12T22:21:23.583 INFO:journalctl@ceph.osd.5.smithi186.stdout:May 12 22:21:23 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-5[46962]: 2024-05-12T22:21:23.248+0000 7f82e356f700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:58.796131+0000 front 2024-05-12T22:20:58.796209+0000 (oldest deadline 2024-05-12T22:21:21.696164+0000) 2024-05-12T22:21:23.761 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 12 22:21:23 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-7[52481]: 2024-05-12T22:21:23.759+0000 7f1b778cf700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:51.904377+0000 front 2024-05-12T22:20:51.904452+0000 (oldest deadline 2024-05-12T22:21:13.604466+0000) 2024-05-12T22:21:23.761 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 12 22:21:23 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-7[52481]: 2024-05-12T22:21:23.760+0000 7f1b778cf700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:59.306433+0000 front 2024-05-12T22:20:59.306362+0000 (oldest deadline 2024-05-12T22:21:22.806281+0000) 2024-05-12T22:21:23.842 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:23 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:23.582+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.561281+0000 front 2024-05-12T22:20:41.561023+0000 (oldest deadline 2024-05-12T22:21:06.261036+0000) 2024-05-12T22:21:23.843 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:23 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:23.582+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:50.163666+0000 front 2024-05-12T22:20:50.163649+0000 (oldest deadline 2024-05-12T22:21:13.063447+0000) 2024-05-12T22:21:23.843 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:23 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:23.582+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:59.365658+0000 front 2024-05-12T22:20:59.365719+0000 (oldest deadline 2024-05-12T22:21:23.465741+0000) 2024-05-12T22:21:24.183 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 12 22:21:23 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-8[53963]: 2024-05-12T22:21:23.842+0000 7f4ff7d79700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:50.580039+0000 front 2024-05-12T22:20:50.580181+0000 (oldest deadline 2024-05-12T22:21:16.480068+0000) 2024-05-12T22:21:24.183 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 12 22:21:23 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-8[53963]: 2024-05-12T22:21:23.842+0000 7f4ff7d79700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:56.480473+0000 front 2024-05-12T22:20:56.480615+0000 (oldest deadline 2024-05-12T22:21:21.780524+0000) 2024-05-12T22:21:24.274 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:23 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:23.815+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.490538+0000 front 2024-05-12T22:20:41.490674+0000 (oldest deadline 2024-05-12T22:21:04.990968+0000) 2024-05-12T22:21:24.275 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:23 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:23.815+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:48.493555+0000 front 2024-05-12T22:20:48.493616+0000 (oldest deadline 2024-05-12T22:21:14.393780+0000) 2024-05-12T22:21:24.275 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:23 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:23.815+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:56.095925+0000 front 2024-05-12T22:20:56.096412+0000 (oldest deadline 2024-05-12T22:21:21.396155+0000) 2024-05-12T22:21:24.492 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:24 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:24.104+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:43.519531+0000 front 2024-05-12T22:20:43.519638+0000 (oldest deadline 2024-05-12T22:21:05.219288+0000) 2024-05-12T22:21:24.492 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:24 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:24.104+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:51.620499+0000 front 2024-05-12T22:20:51.620463+0000 (oldest deadline 2024-05-12T22:21:16.221349+0000) 2024-05-12T22:21:24.492 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:24 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:24.104+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:59.722447+0000 front 2024-05-12T22:20:59.722655+0000 (oldest deadline 2024-05-12T22:21:21.422316+0000) 2024-05-12T22:21:24.535 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 12 22:21:24 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-6[55080]: 2024-05-12T22:21:24.274+0000 7f2fefe00700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:43.392022+0000 front 2024-05-12T22:20:43.392513+0000 (oldest deadline 2024-05-12T22:21:08.692004+0000) 2024-05-12T22:21:24.535 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 12 22:21:24 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-6[55080]: 2024-05-12T22:21:24.274+0000 7f2fefe00700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:50.994527+0000 front 2024-05-12T22:20:50.994611+0000 (oldest deadline 2024-05-12T22:21:16.894684+0000) 2024-05-12T22:21:24.535 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 12 22:21:24 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-6[55080]: 2024-05-12T22:21:24.274+0000 7f2fefe00700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:59.797212+0000 front 2024-05-12T22:20:59.797306+0000 (oldest deadline 2024-05-12T22:21:23.297380+0000) 2024-05-12T22:21:24.621 INFO:journalctl@ceph.osd.5.smithi186.stdout:May 12 22:21:24 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-5[46962]: 2024-05-12T22:21:24.235+0000 7f82e356f700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.993114+0000 front 2024-05-12T22:20:41.993039+0000 (oldest deadline 2024-05-12T22:21:07.193555+0000) 2024-05-12T22:21:24.621 INFO:journalctl@ceph.osd.5.smithi186.stdout:May 12 22:21:24 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-5[46962]: 2024-05-12T22:21:24.235+0000 7f82e356f700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:47.194145+0000 front 2024-05-12T22:20:47.194079+0000 (oldest deadline 2024-05-12T22:21:13.094071+0000) 2024-05-12T22:21:24.621 INFO:journalctl@ceph.osd.5.smithi186.stdout:May 12 22:21:24 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-5[46962]: 2024-05-12T22:21:24.235+0000 7f82e356f700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:58.796131+0000 front 2024-05-12T22:20:58.796209+0000 (oldest deadline 2024-05-12T22:21:21.696164+0000) 2024-05-12T22:21:24.929 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 12 22:21:24 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-8[53963]: 2024-05-12T22:21:24.802+0000 7f4ff7d79700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:50.580039+0000 front 2024-05-12T22:20:50.580181+0000 (oldest deadline 2024-05-12T22:21:16.480068+0000) 2024-05-12T22:21:24.929 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 12 22:21:24 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-8[53963]: 2024-05-12T22:21:24.802+0000 7f4ff7d79700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:56.480473+0000 front 2024-05-12T22:20:56.480615+0000 (oldest deadline 2024-05-12T22:21:21.780524+0000) 2024-05-12T22:21:24.929 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:24 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:24.620+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.561281+0000 front 2024-05-12T22:20:41.561023+0000 (oldest deadline 2024-05-12T22:21:06.261036+0000) 2024-05-12T22:21:24.930 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:24 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:24.620+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:50.163666+0000 front 2024-05-12T22:20:50.163649+0000 (oldest deadline 2024-05-12T22:21:13.063447+0000) 2024-05-12T22:21:24.930 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:24 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:24.620+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:59.365658+0000 front 2024-05-12T22:20:59.365719+0000 (oldest deadline 2024-05-12T22:21:23.465741+0000) 2024-05-12T22:21:25.113 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 12 22:21:24 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-7[52481]: 2024-05-12T22:21:24.788+0000 7f1b778cf700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:51.904377+0000 front 2024-05-12T22:20:51.904452+0000 (oldest deadline 2024-05-12T22:21:13.604466+0000) 2024-05-12T22:21:25.113 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 12 22:21:24 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-7[52481]: 2024-05-12T22:21:24.788+0000 7f1b778cf700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:59.306433+0000 front 2024-05-12T22:20:59.306362+0000 (oldest deadline 2024-05-12T22:21:22.806281+0000) 2024-05-12T22:21:25.285 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:24 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:24.803+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.490538+0000 front 2024-05-12T22:20:41.490674+0000 (oldest deadline 2024-05-12T22:21:04.990968+0000) 2024-05-12T22:21:25.285 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:24 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:24.803+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:48.493555+0000 front 2024-05-12T22:20:48.493616+0000 (oldest deadline 2024-05-12T22:21:14.393780+0000) 2024-05-12T22:21:25.285 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:24 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:24.803+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:56.095925+0000 front 2024-05-12T22:20:56.096412+0000 (oldest deadline 2024-05-12T22:21:21.396155+0000) 2024-05-12T22:21:25.492 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:25 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:25.112+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:43.519531+0000 front 2024-05-12T22:20:43.519638+0000 (oldest deadline 2024-05-12T22:21:05.219288+0000) 2024-05-12T22:21:25.492 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:25 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:25.112+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:51.620499+0000 front 2024-05-12T22:20:51.620463+0000 (oldest deadline 2024-05-12T22:21:16.221349+0000) 2024-05-12T22:21:25.492 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:25 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:25.112+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:59.722447+0000 front 2024-05-12T22:20:59.722655+0000 (oldest deadline 2024-05-12T22:21:21.422316+0000) 2024-05-12T22:21:25.754 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 12 22:21:25 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-6[55080]: 2024-05-12T22:21:25.301+0000 7f2fefe00700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:43.392022+0000 front 2024-05-12T22:20:43.392513+0000 (oldest deadline 2024-05-12T22:21:08.692004+0000) 2024-05-12T22:21:25.755 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 12 22:21:25 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-6[55080]: 2024-05-12T22:21:25.301+0000 7f2fefe00700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:50.994527+0000 front 2024-05-12T22:20:50.994611+0000 (oldest deadline 2024-05-12T22:21:16.894684+0000) 2024-05-12T22:21:25.755 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 12 22:21:25 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-6[55080]: 2024-05-12T22:21:25.301+0000 7f2fefe00700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:59.797212+0000 front 2024-05-12T22:20:59.797306+0000 (oldest deadline 2024-05-12T22:21:23.297380+0000) 2024-05-12T22:21:25.884 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 12 22:21:25 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-8[53963]: 2024-05-12T22:21:25.797+0000 7f4ff7d79700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:50.580039+0000 front 2024-05-12T22:20:50.580181+0000 (oldest deadline 2024-05-12T22:21:16.480068+0000) 2024-05-12T22:21:25.885 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 12 22:21:25 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-8[53963]: 2024-05-12T22:21:25.797+0000 7f4ff7d79700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:56.480473+0000 front 2024-05-12T22:20:56.480615+0000 (oldest deadline 2024-05-12T22:21:21.780524+0000) 2024-05-12T22:21:25.885 INFO:journalctl@ceph.osd.5.smithi186.stdout:May 12 22:21:25 smithi186 podman[71842]: 2024-05-12 22:21:25.560723768 +0000 UTC m=+5.478974284 container died e3ead61e84dfe38e83acabed6cbea775b86ec730af75eae220d7b6c63ef68419 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=squid-f72fecf, org.label-schema.build-date=20240507, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-05-12T22:21:25.885 INFO:journalctl@ceph.osd.5.smithi186.stdout:May 12 22:21:25 smithi186 podman[71842]: 2024-05-12 22:21:25.885068071 +0000 UTC m=+5.803318586 container cleanup e3ead61e84dfe38e83acabed6cbea775b86ec730af75eae220d7b6c63ef68419 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-5, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240507, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, RELEASE=squid-f72fecf, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.vendor=CentOS) 2024-05-12T22:21:25.885 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:25 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:25.660+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.561281+0000 front 2024-05-12T22:20:41.561023+0000 (oldest deadline 2024-05-12T22:21:06.261036+0000) 2024-05-12T22:21:25.886 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:25 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:25.660+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:50.163666+0000 front 2024-05-12T22:20:50.163649+0000 (oldest deadline 2024-05-12T22:21:13.063447+0000) 2024-05-12T22:21:25.886 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:25 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:25.660+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:59.365658+0000 front 2024-05-12T22:20:59.365719+0000 (oldest deadline 2024-05-12T22:21:23.465741+0000) 2024-05-12T22:21:26.035 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:25 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:25.754+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.490538+0000 front 2024-05-12T22:20:41.490674+0000 (oldest deadline 2024-05-12T22:21:04.990968+0000) 2024-05-12T22:21:26.035 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:25 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:25.754+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:48.493555+0000 front 2024-05-12T22:20:48.493616+0000 (oldest deadline 2024-05-12T22:21:14.393780+0000) 2024-05-12T22:21:26.035 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:25 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:25.754+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:56.095925+0000 front 2024-05-12T22:20:56.096412+0000 (oldest deadline 2024-05-12T22:21:21.396155+0000) 2024-05-12T22:21:26.156 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 12 22:21:25 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-7[52481]: 2024-05-12T22:21:25.830+0000 7f1b778cf700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:51.904377+0000 front 2024-05-12T22:20:51.904452+0000 (oldest deadline 2024-05-12T22:21:13.604466+0000) 2024-05-12T22:21:26.157 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 12 22:21:25 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-7[52481]: 2024-05-12T22:21:25.831+0000 7f1b778cf700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:59.306433+0000 front 2024-05-12T22:20:59.306362+0000 (oldest deadline 2024-05-12T22:21:22.806281+0000) 2024-05-12T22:21:26.179 INFO:journalctl@ceph.osd.5.smithi186.stdout:May 12 22:21:25 smithi186 bash[71842]: ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-5 2024-05-12T22:21:26.179 INFO:journalctl@ceph.osd.5.smithi186.stdout:May 12 22:21:26 smithi186 podman[71859]: 2024-05-12 22:21:26.0864245 +0000 UTC m=+0.516691449 container remove e3ead61e84dfe38e83acabed6cbea775b86ec730af75eae220d7b6c63ef68419 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240507, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-f72fecf, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.license=GPLv2) 2024-05-12T22:21:26.492 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:26 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:26.156+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:43.519531+0000 front 2024-05-12T22:20:43.519638+0000 (oldest deadline 2024-05-12T22:21:05.219288+0000) 2024-05-12T22:21:26.492 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:26 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:26.156+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:51.620499+0000 front 2024-05-12T22:20:51.620463+0000 (oldest deadline 2024-05-12T22:21:16.221349+0000) 2024-05-12T22:21:26.492 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:26 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:26.156+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:59.722447+0000 front 2024-05-12T22:20:59.722655+0000 (oldest deadline 2024-05-12T22:21:21.422316+0000) 2024-05-12T22:21:26.535 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 12 22:21:26 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-6[55080]: 2024-05-12T22:21:26.276+0000 7f2fefe00700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:43.392022+0000 front 2024-05-12T22:20:43.392513+0000 (oldest deadline 2024-05-12T22:21:08.692004+0000) 2024-05-12T22:21:26.535 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 12 22:21:26 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-6[55080]: 2024-05-12T22:21:26.276+0000 7f2fefe00700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:50.994527+0000 front 2024-05-12T22:20:50.994611+0000 (oldest deadline 2024-05-12T22:21:16.894684+0000) 2024-05-12T22:21:26.535 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 12 22:21:26 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-6[55080]: 2024-05-12T22:21:26.276+0000 7f2fefe00700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:59.797212+0000 front 2024-05-12T22:20:59.797306+0000 (oldest deadline 2024-05-12T22:21:23.297380+0000) 2024-05-12T22:21:26.797 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:26 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:26.628+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.561281+0000 front 2024-05-12T22:20:41.561023+0000 (oldest deadline 2024-05-12T22:21:06.261036+0000) 2024-05-12T22:21:26.798 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:26 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:26.628+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:50.163666+0000 front 2024-05-12T22:20:50.163649+0000 (oldest deadline 2024-05-12T22:21:13.063447+0000) 2024-05-12T22:21:26.798 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:26 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:26.629+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:59.365658+0000 front 2024-05-12T22:20:59.365719+0000 (oldest deadline 2024-05-12T22:21:23.465741+0000) 2024-05-12T22:21:26.798 INFO:journalctl@ceph.osd.5.smithi186.stdout:May 12 22:21:26 smithi186 podman[71946]: 2024-05-12 22:21:26.504132721 +0000 UTC m=+0.112712214 container create d1f2186a39f2814e8916b3d86e4f5ef6c15d92e50e49c2b58ff3689e9e6d3bf9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-5-deactivate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240507, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=squid-f72fecf, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e) 2024-05-12T22:21:26.798 INFO:journalctl@ceph.osd.5.smithi186.stdout:May 12 22:21:26 smithi186 podman[71946]: 2024-05-12 22:21:26.409754767 +0000 UTC m=+0.018334292 image pull 8252b7944d3b5c188c0ff632647ba6cc4e9a19588ee097b8a3dba27b42779c25 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc 2024-05-12T22:21:26.798 INFO:journalctl@ceph.osd.5.smithi186.stdout:May 12 22:21:26 smithi186 podman[71946]: 2024-05-12 22:21:26.71555882 +0000 UTC m=+0.324138326 container init d1f2186a39f2814e8916b3d86e4f5ef6c15d92e50e49c2b58ff3689e9e6d3bf9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-5-deactivate, org.label-schema.build-date=20240507, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=squid-f72fecf, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-05-12T22:21:26.798 INFO:journalctl@ceph.osd.5.smithi186.stdout:May 12 22:21:26 smithi186 podman[71946]: 2024-05-12 22:21:26.718766438 +0000 UTC m=+0.327345936 container start d1f2186a39f2814e8916b3d86e4f5ef6c15d92e50e49c2b58ff3689e9e6d3bf9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-5-deactivate, RELEASE=squid-f72fecf, org.label-schema.license=GPLv2, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240507, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-05-12T22:21:27.039 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:26 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:26.772+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.490538+0000 front 2024-05-12T22:20:41.490674+0000 (oldest deadline 2024-05-12T22:21:04.990968+0000) 2024-05-12T22:21:27.040 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:26 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:26.772+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:48.493555+0000 front 2024-05-12T22:20:48.493616+0000 (oldest deadline 2024-05-12T22:21:14.393780+0000) 2024-05-12T22:21:27.040 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:26 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:26.772+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:56.095925+0000 front 2024-05-12T22:20:56.096412+0000 (oldest deadline 2024-05-12T22:21:21.396155+0000) 2024-05-12T22:21:27.134 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 12 22:21:26 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-7[52481]: 2024-05-12T22:21:26.870+0000 7f1b778cf700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:51.904377+0000 front 2024-05-12T22:20:51.904452+0000 (oldest deadline 2024-05-12T22:21:13.604466+0000) 2024-05-12T22:21:27.135 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 12 22:21:26 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-7[52481]: 2024-05-12T22:21:26.870+0000 7f1b778cf700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:59.306433+0000 front 2024-05-12T22:20:59.306362+0000 (oldest deadline 2024-05-12T22:21:22.806281+0000) 2024-05-12T22:21:27.179 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 12 22:21:26 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-8[53963]: 2024-05-12T22:21:26.810+0000 7f4ff7d79700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:50.580039+0000 front 2024-05-12T22:20:50.580181+0000 (oldest deadline 2024-05-12T22:21:16.480068+0000) 2024-05-12T22:21:27.179 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 12 22:21:26 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-8[53963]: 2024-05-12T22:21:26.810+0000 7f4ff7d79700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:56.480473+0000 front 2024-05-12T22:20:56.480615+0000 (oldest deadline 2024-05-12T22:21:21.780524+0000) 2024-05-12T22:21:27.180 INFO:journalctl@ceph.osd.5.smithi186.stdout:May 12 22:21:26 smithi186 podman[71946]: 2024-05-12 22:21:26.798219681 +0000 UTC m=+0.406799185 container attach d1f2186a39f2814e8916b3d86e4f5ef6c15d92e50e49c2b58ff3689e9e6d3bf9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-5-deactivate, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240507, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=squid-f72fecf, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-05-12T22:21:27.180 INFO:journalctl@ceph.osd.5.smithi186.stdout:May 12 22:21:27 smithi186 podman[71946]: 2024-05-12 22:21:27.01541717 +0000 UTC m=+0.623996677 container died d1f2186a39f2814e8916b3d86e4f5ef6c15d92e50e49c2b58ff3689e9e6d3bf9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-5-deactivate, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=squid-f72fecf, org.label-schema.build-date=20240507) 2024-05-12T22:21:27.492 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:27 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:27.134+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:43.519531+0000 front 2024-05-12T22:20:43.519638+0000 (oldest deadline 2024-05-12T22:21:05.219288+0000) 2024-05-12T22:21:27.492 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:27 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:27.134+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:51.620499+0000 front 2024-05-12T22:20:51.620463+0000 (oldest deadline 2024-05-12T22:21:16.221349+0000) 2024-05-12T22:21:27.492 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:27 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:27.134+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:59.722447+0000 front 2024-05-12T22:20:59.722655+0000 (oldest deadline 2024-05-12T22:21:21.422316+0000) 2024-05-12T22:21:27.573 DEBUG:teuthology.orchestra.run.smithi186:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968@osd.5.service' 2024-05-12T22:21:27.651 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-12T22:21:27.654 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-05-12T22:21:27.654 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-05-12T22:21:27.654 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968@osd.6 2024-05-12T22:21:27.735 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 12 22:21:27 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-6[55080]: 2024-05-12T22:21:27.288+0000 7f2fefe00700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:43.392022+0000 front 2024-05-12T22:20:43.392513+0000 (oldest deadline 2024-05-12T22:21:08.692004+0000) 2024-05-12T22:21:27.736 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 12 22:21:27 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-6[55080]: 2024-05-12T22:21:27.288+0000 7f2fefe00700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:50.994527+0000 front 2024-05-12T22:20:50.994611+0000 (oldest deadline 2024-05-12T22:21:16.894684+0000) 2024-05-12T22:21:27.736 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 12 22:21:27 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-6[55080]: 2024-05-12T22:21:27.288+0000 7f2fefe00700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:59.797212+0000 front 2024-05-12T22:20:59.797306+0000 (oldest deadline 2024-05-12T22:21:23.297380+0000) 2024-05-12T22:21:27.846 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:27 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:27.597+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.561281+0000 front 2024-05-12T22:20:41.561023+0000 (oldest deadline 2024-05-12T22:21:06.261036+0000) 2024-05-12T22:21:27.847 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:27 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:27.597+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:50.163666+0000 front 2024-05-12T22:20:50.163649+0000 (oldest deadline 2024-05-12T22:21:13.063447+0000) 2024-05-12T22:21:27.847 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:27 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:27.597+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:59.365658+0000 front 2024-05-12T22:20:59.365719+0000 (oldest deadline 2024-05-12T22:21:23.465741+0000) 2024-05-12T22:21:28.035 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 12 22:21:27 smithi028 systemd[1]: Stopping Ceph osd.6 for 6b92e7b4-10aa-11ef-bc98-c7b262605968... 2024-05-12T22:21:28.035 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:27 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:27.763+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.490538+0000 front 2024-05-12T22:20:41.490674+0000 (oldest deadline 2024-05-12T22:21:04.990968+0000) 2024-05-12T22:21:28.035 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:27 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:27.763+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:48.493555+0000 front 2024-05-12T22:20:48.493616+0000 (oldest deadline 2024-05-12T22:21:14.393780+0000) 2024-05-12T22:21:28.035 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:27 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:27.763+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:56.095925+0000 front 2024-05-12T22:20:56.096412+0000 (oldest deadline 2024-05-12T22:21:21.396155+0000) 2024-05-12T22:21:28.179 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 12 22:21:27 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-8[53963]: 2024-05-12T22:21:27.845+0000 7f4ff7d79700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:50.580039+0000 front 2024-05-12T22:20:50.580181+0000 (oldest deadline 2024-05-12T22:21:16.480068+0000) 2024-05-12T22:21:28.179 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 12 22:21:27 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-8[53963]: 2024-05-12T22:21:27.845+0000 7f4ff7d79700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:56.480473+0000 front 2024-05-12T22:20:56.480615+0000 (oldest deadline 2024-05-12T22:21:21.780524+0000) 2024-05-12T22:21:28.242 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 12 22:21:27 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-7[52481]: 2024-05-12T22:21:27.843+0000 7f1b778cf700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:51.904377+0000 front 2024-05-12T22:20:51.904452+0000 (oldest deadline 2024-05-12T22:21:13.604466+0000) 2024-05-12T22:21:28.242 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 12 22:21:27 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-7[52481]: 2024-05-12T22:21:27.843+0000 7f1b778cf700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:59.306433+0000 front 2024-05-12T22:20:59.306362+0000 (oldest deadline 2024-05-12T22:21:22.806281+0000) 2024-05-12T22:21:28.242 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:28 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:28.085+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:43.519531+0000 front 2024-05-12T22:20:43.519638+0000 (oldest deadline 2024-05-12T22:21:05.219288+0000) 2024-05-12T22:21:28.243 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:28 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:28.085+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:51.620499+0000 front 2024-05-12T22:20:51.620463+0000 (oldest deadline 2024-05-12T22:21:16.221349+0000) 2024-05-12T22:21:28.243 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:28 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:28.085+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:59.722447+0000 front 2024-05-12T22:20:59.722655+0000 (oldest deadline 2024-05-12T22:21:21.422316+0000) 2024-05-12T22:21:28.535 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 12 22:21:28 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-6[55080]: 2024-05-12T22:21:28.212+0000 7f2ff3c00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-12T22:21:28.536 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 12 22:21:28 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-6[55080]: 2024-05-12T22:21:28.212+0000 7f2ff3c00700 -1 osd.6 102 *** Got signal Terminated *** 2024-05-12T22:21:28.536 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 12 22:21:28 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-6[55080]: 2024-05-12T22:21:28.212+0000 7f2ff3c00700 -1 osd.6 102 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-12T22:21:28.536 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 12 22:21:28 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-6[55080]: 2024-05-12T22:21:28.321+0000 7f2fefe00700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:43.392022+0000 front 2024-05-12T22:20:43.392513+0000 (oldest deadline 2024-05-12T22:21:08.692004+0000) 2024-05-12T22:21:28.536 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 12 22:21:28 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-6[55080]: 2024-05-12T22:21:28.321+0000 7f2fefe00700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:50.994527+0000 front 2024-05-12T22:20:50.994611+0000 (oldest deadline 2024-05-12T22:21:16.894684+0000) 2024-05-12T22:21:28.536 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 12 22:21:28 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-6[55080]: 2024-05-12T22:21:28.321+0000 7f2fefe00700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:59.797212+0000 front 2024-05-12T22:20:59.797306+0000 (oldest deadline 2024-05-12T22:21:23.297380+0000) 2024-05-12T22:21:28.929 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 12 22:21:28 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-8[53963]: 2024-05-12T22:21:28.849+0000 7f4ff7d79700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:50.580039+0000 front 2024-05-12T22:20:50.580181+0000 (oldest deadline 2024-05-12T22:21:16.480068+0000) 2024-05-12T22:21:28.929 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 12 22:21:28 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-8[53963]: 2024-05-12T22:21:28.849+0000 7f4ff7d79700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:56.480473+0000 front 2024-05-12T22:20:56.480615+0000 (oldest deadline 2024-05-12T22:21:21.780524+0000) 2024-05-12T22:21:28.929 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:28 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:28.610+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.561281+0000 front 2024-05-12T22:20:41.561023+0000 (oldest deadline 2024-05-12T22:21:06.261036+0000) 2024-05-12T22:21:28.930 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:28 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:28.610+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:50.163666+0000 front 2024-05-12T22:20:50.163649+0000 (oldest deadline 2024-05-12T22:21:13.063447+0000) 2024-05-12T22:21:28.930 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:28 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:28.610+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:59.365658+0000 front 2024-05-12T22:20:59.365719+0000 (oldest deadline 2024-05-12T22:21:23.465741+0000) 2024-05-12T22:21:29.035 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:28 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:28.783+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.490538+0000 front 2024-05-12T22:20:41.490674+0000 (oldest deadline 2024-05-12T22:21:04.990968+0000) 2024-05-12T22:21:29.035 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:28 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:28.783+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:48.493555+0000 front 2024-05-12T22:20:48.493616+0000 (oldest deadline 2024-05-12T22:21:14.393780+0000) 2024-05-12T22:21:29.035 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:28 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:28.783+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:56.095925+0000 front 2024-05-12T22:20:56.096412+0000 (oldest deadline 2024-05-12T22:21:21.396155+0000) 2024-05-12T22:21:29.242 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 12 22:21:28 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-7[52481]: 2024-05-12T22:21:28.818+0000 7f1b778cf700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:51.904377+0000 front 2024-05-12T22:20:51.904452+0000 (oldest deadline 2024-05-12T22:21:13.604466+0000) 2024-05-12T22:21:29.242 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 12 22:21:28 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-7[52481]: 2024-05-12T22:21:28.818+0000 7f1b778cf700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:59.306433+0000 front 2024-05-12T22:20:59.306362+0000 (oldest deadline 2024-05-12T22:21:22.806281+0000) 2024-05-12T22:21:29.242 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:29 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:29.039+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:43.519531+0000 front 2024-05-12T22:20:43.519638+0000 (oldest deadline 2024-05-12T22:21:05.219288+0000) 2024-05-12T22:21:29.243 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:29 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:29.039+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:51.620499+0000 front 2024-05-12T22:20:51.620463+0000 (oldest deadline 2024-05-12T22:21:16.221349+0000) 2024-05-12T22:21:29.243 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:29 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:29.039+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:59.722447+0000 front 2024-05-12T22:20:59.722655+0000 (oldest deadline 2024-05-12T22:21:21.422316+0000) 2024-05-12T22:21:29.535 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 12 22:21:29 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-6[55080]: 2024-05-12T22:21:29.272+0000 7f2fefe00700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:43.392022+0000 front 2024-05-12T22:20:43.392513+0000 (oldest deadline 2024-05-12T22:21:08.692004+0000) 2024-05-12T22:21:29.535 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 12 22:21:29 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-6[55080]: 2024-05-12T22:21:29.272+0000 7f2fefe00700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:50.994527+0000 front 2024-05-12T22:20:50.994611+0000 (oldest deadline 2024-05-12T22:21:16.894684+0000) 2024-05-12T22:21:29.535 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 12 22:21:29 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-6[55080]: 2024-05-12T22:21:29.272+0000 7f2fefe00700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:59.797212+0000 front 2024-05-12T22:20:59.797306+0000 (oldest deadline 2024-05-12T22:21:23.297380+0000) 2024-05-12T22:21:29.929 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 12 22:21:29 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-8[53963]: 2024-05-12T22:21:29.830+0000 7f4ff7d79700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:50.580039+0000 front 2024-05-12T22:20:50.580181+0000 (oldest deadline 2024-05-12T22:21:16.480068+0000) 2024-05-12T22:21:29.929 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 12 22:21:29 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-8[53963]: 2024-05-12T22:21:29.830+0000 7f4ff7d79700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:56.480473+0000 front 2024-05-12T22:20:56.480615+0000 (oldest deadline 2024-05-12T22:21:21.780524+0000) 2024-05-12T22:21:29.929 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:29 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:29.610+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.561281+0000 front 2024-05-12T22:20:41.561023+0000 (oldest deadline 2024-05-12T22:21:06.261036+0000) 2024-05-12T22:21:29.930 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:29 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:29.610+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:50.163666+0000 front 2024-05-12T22:20:50.163649+0000 (oldest deadline 2024-05-12T22:21:13.063447+0000) 2024-05-12T22:21:29.930 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:29 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:29.610+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:59.365658+0000 front 2024-05-12T22:20:59.365719+0000 (oldest deadline 2024-05-12T22:21:23.465741+0000) 2024-05-12T22:21:30.035 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:29 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:29.776+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.490538+0000 front 2024-05-12T22:20:41.490674+0000 (oldest deadline 2024-05-12T22:21:04.990968+0000) 2024-05-12T22:21:30.035 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:29 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:29.776+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:48.493555+0000 front 2024-05-12T22:20:48.493616+0000 (oldest deadline 2024-05-12T22:21:14.393780+0000) 2024-05-12T22:21:30.035 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:29 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:29.776+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:56.095925+0000 front 2024-05-12T22:20:56.096412+0000 (oldest deadline 2024-05-12T22:21:21.396155+0000) 2024-05-12T22:21:30.242 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 12 22:21:29 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-7[52481]: 2024-05-12T22:21:29.842+0000 7f1b778cf700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:51.904377+0000 front 2024-05-12T22:20:51.904452+0000 (oldest deadline 2024-05-12T22:21:13.604466+0000) 2024-05-12T22:21:30.242 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 12 22:21:29 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-7[52481]: 2024-05-12T22:21:29.842+0000 7f1b778cf700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:59.306433+0000 front 2024-05-12T22:20:59.306362+0000 (oldest deadline 2024-05-12T22:21:22.806281+0000) 2024-05-12T22:21:30.242 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:30 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:30.008+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:43.519531+0000 front 2024-05-12T22:20:43.519638+0000 (oldest deadline 2024-05-12T22:21:05.219288+0000) 2024-05-12T22:21:30.243 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:30 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:30.008+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:51.620499+0000 front 2024-05-12T22:20:51.620463+0000 (oldest deadline 2024-05-12T22:21:16.221349+0000) 2024-05-12T22:21:30.243 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:30 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:30.008+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:59.722447+0000 front 2024-05-12T22:20:59.722655+0000 (oldest deadline 2024-05-12T22:21:21.422316+0000) 2024-05-12T22:21:30.785 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 12 22:21:30 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-6[55080]: 2024-05-12T22:21:30.314+0000 7f2fefe00700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:43.392022+0000 front 2024-05-12T22:20:43.392513+0000 (oldest deadline 2024-05-12T22:21:08.692004+0000) 2024-05-12T22:21:30.785 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 12 22:21:30 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-6[55080]: 2024-05-12T22:21:30.314+0000 7f2fefe00700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:50.994527+0000 front 2024-05-12T22:20:50.994611+0000 (oldest deadline 2024-05-12T22:21:16.894684+0000) 2024-05-12T22:21:30.785 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 12 22:21:30 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-6[55080]: 2024-05-12T22:21:30.314+0000 7f2fefe00700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:59.797212+0000 front 2024-05-12T22:20:59.797306+0000 (oldest deadline 2024-05-12T22:21:23.297380+0000) 2024-05-12T22:21:30.785 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 12 22:21:30 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-6[55080]: 2024-05-12T22:21:30.314+0000 7f2fefe00700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-12T22:21:07.999251+0000 front 2024-05-12T22:21:07.999589+0000 (oldest deadline 2024-05-12T22:21:30.200050+0000) 2024-05-12T22:21:30.929 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 12 22:21:30 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-8[53963]: 2024-05-12T22:21:30.842+0000 7f4ff7d79700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:50.580039+0000 front 2024-05-12T22:20:50.580181+0000 (oldest deadline 2024-05-12T22:21:16.480068+0000) 2024-05-12T22:21:30.929 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 12 22:21:30 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-8[53963]: 2024-05-12T22:21:30.842+0000 7f4ff7d79700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:56.480473+0000 front 2024-05-12T22:20:56.480615+0000 (oldest deadline 2024-05-12T22:21:21.780524+0000) 2024-05-12T22:21:30.929 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 12 22:21:30 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-8[53963]: 2024-05-12T22:21:30.842+0000 7f4ff7d79700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-12T22:21:05.281742+0000 front 2024-05-12T22:21:05.281856+0000 (oldest deadline 2024-05-12T22:21:29.981483+0000) 2024-05-12T22:21:30.930 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:30 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:30.624+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.561281+0000 front 2024-05-12T22:20:41.561023+0000 (oldest deadline 2024-05-12T22:21:06.261036+0000) 2024-05-12T22:21:30.930 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:30 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:30.624+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:50.163666+0000 front 2024-05-12T22:20:50.163649+0000 (oldest deadline 2024-05-12T22:21:13.063447+0000) 2024-05-12T22:21:30.930 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:30 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:30.624+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:59.365658+0000 front 2024-05-12T22:20:59.365719+0000 (oldest deadline 2024-05-12T22:21:23.465741+0000) 2024-05-12T22:21:31.242 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 12 22:21:30 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-7[52481]: 2024-05-12T22:21:30.801+0000 7f1b778cf700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:51.904377+0000 front 2024-05-12T22:20:51.904452+0000 (oldest deadline 2024-05-12T22:21:13.604466+0000) 2024-05-12T22:21:31.242 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 12 22:21:30 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-7[52481]: 2024-05-12T22:21:30.801+0000 7f1b778cf700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:59.306433+0000 front 2024-05-12T22:20:59.306362+0000 (oldest deadline 2024-05-12T22:21:22.806281+0000) 2024-05-12T22:21:31.242 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 12 22:21:30 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-7[52481]: 2024-05-12T22:21:30.801+0000 7f1b778cf700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-12T22:21:05.107730+0000 front 2024-05-12T22:21:05.107621+0000 (oldest deadline 2024-05-12T22:21:30.407516+0000) 2024-05-12T22:21:31.242 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:30 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:30.993+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:43.519531+0000 front 2024-05-12T22:20:43.519638+0000 (oldest deadline 2024-05-12T22:21:05.219288+0000) 2024-05-12T22:21:31.242 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:30 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:30.993+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:51.620499+0000 front 2024-05-12T22:20:51.620463+0000 (oldest deadline 2024-05-12T22:21:16.221349+0000) 2024-05-12T22:21:31.243 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:30 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:30.993+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:59.722447+0000 front 2024-05-12T22:20:59.722655+0000 (oldest deadline 2024-05-12T22:21:21.422316+0000) 2024-05-12T22:21:31.285 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:30 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:30.812+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.490538+0000 front 2024-05-12T22:20:41.490674+0000 (oldest deadline 2024-05-12T22:21:04.990968+0000) 2024-05-12T22:21:31.285 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:30 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:30.812+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:48.493555+0000 front 2024-05-12T22:20:48.493616+0000 (oldest deadline 2024-05-12T22:21:14.393780+0000) 2024-05-12T22:21:31.285 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:30 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:30.812+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:56.095925+0000 front 2024-05-12T22:20:56.096412+0000 (oldest deadline 2024-05-12T22:21:21.396155+0000) 2024-05-12T22:21:31.285 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:30 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:30.812+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-12T22:21:08.399553+0000 front 2024-05-12T22:21:08.399957+0000 (oldest deadline 2024-05-12T22:21:30.099451+0000) 2024-05-12T22:21:31.785 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 12 22:21:31 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-6[55080]: 2024-05-12T22:21:31.333+0000 7f2fefe00700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:43.392022+0000 front 2024-05-12T22:20:43.392513+0000 (oldest deadline 2024-05-12T22:21:08.692004+0000) 2024-05-12T22:21:31.785 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 12 22:21:31 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-6[55080]: 2024-05-12T22:21:31.334+0000 7f2fefe00700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:50.994527+0000 front 2024-05-12T22:20:50.994611+0000 (oldest deadline 2024-05-12T22:21:16.894684+0000) 2024-05-12T22:21:31.785 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 12 22:21:31 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-6[55080]: 2024-05-12T22:21:31.334+0000 7f2fefe00700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:59.797212+0000 front 2024-05-12T22:20:59.797306+0000 (oldest deadline 2024-05-12T22:21:23.297380+0000) 2024-05-12T22:21:31.785 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 12 22:21:31 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-6[55080]: 2024-05-12T22:21:31.334+0000 7f2fefe00700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-12T22:21:07.999251+0000 front 2024-05-12T22:21:07.999589+0000 (oldest deadline 2024-05-12T22:21:30.200050+0000) 2024-05-12T22:21:31.929 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 12 22:21:31 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-8[53963]: 2024-05-12T22:21:31.837+0000 7f4ff7d79700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:50.580039+0000 front 2024-05-12T22:20:50.580181+0000 (oldest deadline 2024-05-12T22:21:16.480068+0000) 2024-05-12T22:21:31.929 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 12 22:21:31 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-8[53963]: 2024-05-12T22:21:31.837+0000 7f4ff7d79700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:56.480473+0000 front 2024-05-12T22:20:56.480615+0000 (oldest deadline 2024-05-12T22:21:21.780524+0000) 2024-05-12T22:21:31.929 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 12 22:21:31 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-8[53963]: 2024-05-12T22:21:31.837+0000 7f4ff7d79700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-12T22:21:05.281742+0000 front 2024-05-12T22:21:05.281856+0000 (oldest deadline 2024-05-12T22:21:29.981483+0000) 2024-05-12T22:21:31.929 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:31 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:31.647+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.561281+0000 front 2024-05-12T22:20:41.561023+0000 (oldest deadline 2024-05-12T22:21:06.261036+0000) 2024-05-12T22:21:31.929 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:31 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:31.647+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:50.163666+0000 front 2024-05-12T22:20:50.163649+0000 (oldest deadline 2024-05-12T22:21:13.063447+0000) 2024-05-12T22:21:31.930 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:31 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:31.647+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:59.365658+0000 front 2024-05-12T22:20:59.365719+0000 (oldest deadline 2024-05-12T22:21:23.465741+0000) 2024-05-12T22:21:32.242 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:31 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:31.952+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:43.519531+0000 front 2024-05-12T22:20:43.519638+0000 (oldest deadline 2024-05-12T22:21:05.219288+0000) 2024-05-12T22:21:32.242 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:31 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:31.952+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:51.620499+0000 front 2024-05-12T22:20:51.620463+0000 (oldest deadline 2024-05-12T22:21:16.221349+0000) 2024-05-12T22:21:32.242 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:31 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:31.952+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:59.722447+0000 front 2024-05-12T22:20:59.722655+0000 (oldest deadline 2024-05-12T22:21:21.422316+0000) 2024-05-12T22:21:32.242 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 12 22:21:31 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-7[52481]: 2024-05-12T22:21:31.828+0000 7f1b778cf700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:51.904377+0000 front 2024-05-12T22:20:51.904452+0000 (oldest deadline 2024-05-12T22:21:13.604466+0000) 2024-05-12T22:21:32.243 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 12 22:21:31 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-7[52481]: 2024-05-12T22:21:31.828+0000 7f1b778cf700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:59.306433+0000 front 2024-05-12T22:20:59.306362+0000 (oldest deadline 2024-05-12T22:21:22.806281+0000) 2024-05-12T22:21:32.243 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 12 22:21:31 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-7[52481]: 2024-05-12T22:21:31.828+0000 7f1b778cf700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-12T22:21:05.107730+0000 front 2024-05-12T22:21:05.107621+0000 (oldest deadline 2024-05-12T22:21:30.407516+0000) 2024-05-12T22:21:32.285 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:31 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:31.827+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.490538+0000 front 2024-05-12T22:20:41.490674+0000 (oldest deadline 2024-05-12T22:21:04.990968+0000) 2024-05-12T22:21:32.285 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:31 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:31.827+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:48.493555+0000 front 2024-05-12T22:20:48.493616+0000 (oldest deadline 2024-05-12T22:21:14.393780+0000) 2024-05-12T22:21:32.285 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:31 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:31.827+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:56.095925+0000 front 2024-05-12T22:20:56.096412+0000 (oldest deadline 2024-05-12T22:21:21.396155+0000) 2024-05-12T22:21:32.286 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:31 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:31.827+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-12T22:21:08.399553+0000 front 2024-05-12T22:21:08.399957+0000 (oldest deadline 2024-05-12T22:21:30.099451+0000) 2024-05-12T22:21:32.781 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 12 22:21:32 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-6[55080]: 2024-05-12T22:21:32.353+0000 7f2fefe00700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:43.392022+0000 front 2024-05-12T22:20:43.392513+0000 (oldest deadline 2024-05-12T22:21:08.692004+0000) 2024-05-12T22:21:32.781 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 12 22:21:32 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-6[55080]: 2024-05-12T22:21:32.353+0000 7f2fefe00700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:50.994527+0000 front 2024-05-12T22:20:50.994611+0000 (oldest deadline 2024-05-12T22:21:16.894684+0000) 2024-05-12T22:21:32.781 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 12 22:21:32 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-6[55080]: 2024-05-12T22:21:32.353+0000 7f2fefe00700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:59.797212+0000 front 2024-05-12T22:20:59.797306+0000 (oldest deadline 2024-05-12T22:21:23.297380+0000) 2024-05-12T22:21:32.781 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 12 22:21:32 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-6[55080]: 2024-05-12T22:21:32.353+0000 7f2fefe00700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-12T22:21:07.999251+0000 front 2024-05-12T22:21:07.999589+0000 (oldest deadline 2024-05-12T22:21:30.200050+0000) 2024-05-12T22:21:32.929 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:32 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:32.616+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.561281+0000 front 2024-05-12T22:20:41.561023+0000 (oldest deadline 2024-05-12T22:21:06.261036+0000) 2024-05-12T22:21:32.929 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:32 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:32.616+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:50.163666+0000 front 2024-05-12T22:20:50.163649+0000 (oldest deadline 2024-05-12T22:21:13.063447+0000) 2024-05-12T22:21:32.929 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:32 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:32.616+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:59.365658+0000 front 2024-05-12T22:20:59.365719+0000 (oldest deadline 2024-05-12T22:21:23.465741+0000) 2024-05-12T22:21:32.930 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 12 22:21:32 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-8[53963]: 2024-05-12T22:21:32.810+0000 7f4ff7d79700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:50.580039+0000 front 2024-05-12T22:20:50.580181+0000 (oldest deadline 2024-05-12T22:21:16.480068+0000) 2024-05-12T22:21:32.930 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 12 22:21:32 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-8[53963]: 2024-05-12T22:21:32.810+0000 7f4ff7d79700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:56.480473+0000 front 2024-05-12T22:20:56.480615+0000 (oldest deadline 2024-05-12T22:21:21.780524+0000) 2024-05-12T22:21:32.930 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 12 22:21:32 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-8[53963]: 2024-05-12T22:21:32.810+0000 7f4ff7d79700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-12T22:21:05.281742+0000 front 2024-05-12T22:21:05.281856+0000 (oldest deadline 2024-05-12T22:21:29.981483+0000) 2024-05-12T22:21:33.035 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:32 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:32.779+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.490538+0000 front 2024-05-12T22:20:41.490674+0000 (oldest deadline 2024-05-12T22:21:04.990968+0000) 2024-05-12T22:21:33.035 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:32 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:32.779+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:48.493555+0000 front 2024-05-12T22:20:48.493616+0000 (oldest deadline 2024-05-12T22:21:14.393780+0000) 2024-05-12T22:21:33.035 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:32 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:32.779+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:56.095925+0000 front 2024-05-12T22:20:56.096412+0000 (oldest deadline 2024-05-12T22:21:21.396155+0000) 2024-05-12T22:21:33.035 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:32 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:32.779+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-12T22:21:08.399553+0000 front 2024-05-12T22:21:08.399957+0000 (oldest deadline 2024-05-12T22:21:30.099451+0000) 2024-05-12T22:21:33.242 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 12 22:21:32 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-7[52481]: 2024-05-12T22:21:32.852+0000 7f1b778cf700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:51.904377+0000 front 2024-05-12T22:20:51.904452+0000 (oldest deadline 2024-05-12T22:21:13.604466+0000) 2024-05-12T22:21:33.242 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 12 22:21:32 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-7[52481]: 2024-05-12T22:21:32.852+0000 7f1b778cf700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:59.306433+0000 front 2024-05-12T22:20:59.306362+0000 (oldest deadline 2024-05-12T22:21:22.806281+0000) 2024-05-12T22:21:33.242 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 12 22:21:32 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-7[52481]: 2024-05-12T22:21:32.852+0000 7f1b778cf700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-12T22:21:05.107730+0000 front 2024-05-12T22:21:05.107621+0000 (oldest deadline 2024-05-12T22:21:30.407516+0000) 2024-05-12T22:21:33.243 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:32 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:32.972+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:43.519531+0000 front 2024-05-12T22:20:43.519638+0000 (oldest deadline 2024-05-12T22:21:05.219288+0000) 2024-05-12T22:21:33.243 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:32 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:32.973+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:51.620499+0000 front 2024-05-12T22:20:51.620463+0000 (oldest deadline 2024-05-12T22:21:16.221349+0000) 2024-05-12T22:21:33.243 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:32 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:32.973+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:59.722447+0000 front 2024-05-12T22:20:59.722655+0000 (oldest deadline 2024-05-12T22:21:21.422316+0000) 2024-05-12T22:21:33.886 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 12 22:21:33 smithi028 podman[99945]: 2024-05-12 22:21:33.587148712 +0000 UTC m=+5.505089919 container died be82f07b03d72704c4ab211d26bd3677bb5cd3a859476b058dc76eab6ad11864 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-6, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240507, RELEASE=squid-f72fecf, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, GIT_CLEAN=True) 2024-05-12T22:21:33.886 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:33 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:33.774+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.490538+0000 front 2024-05-12T22:20:41.490674+0000 (oldest deadline 2024-05-12T22:21:04.990968+0000) 2024-05-12T22:21:33.886 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:33 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:33.774+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:48.493555+0000 front 2024-05-12T22:20:48.493616+0000 (oldest deadline 2024-05-12T22:21:14.393780+0000) 2024-05-12T22:21:33.886 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:33 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:33.774+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:56.095925+0000 front 2024-05-12T22:20:56.096412+0000 (oldest deadline 2024-05-12T22:21:21.396155+0000) 2024-05-12T22:21:33.886 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:33 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:33.774+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-12T22:21:08.399553+0000 front 2024-05-12T22:21:08.399957+0000 (oldest deadline 2024-05-12T22:21:30.099451+0000) 2024-05-12T22:21:33.929 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 12 22:21:33 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-8[53963]: 2024-05-12T22:21:33.794+0000 7f4ff7d79700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:50.580039+0000 front 2024-05-12T22:20:50.580181+0000 (oldest deadline 2024-05-12T22:21:16.480068+0000) 2024-05-12T22:21:33.929 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 12 22:21:33 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-8[53963]: 2024-05-12T22:21:33.794+0000 7f4ff7d79700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:56.480473+0000 front 2024-05-12T22:20:56.480615+0000 (oldest deadline 2024-05-12T22:21:21.780524+0000) 2024-05-12T22:21:33.929 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 12 22:21:33 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-8[53963]: 2024-05-12T22:21:33.794+0000 7f4ff7d79700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-12T22:21:05.281742+0000 front 2024-05-12T22:21:05.281856+0000 (oldest deadline 2024-05-12T22:21:29.981483+0000) 2024-05-12T22:21:33.929 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:33 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:33.622+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.561281+0000 front 2024-05-12T22:20:41.561023+0000 (oldest deadline 2024-05-12T22:21:06.261036+0000) 2024-05-12T22:21:33.930 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:33 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:33.622+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:50.163666+0000 front 2024-05-12T22:20:50.163649+0000 (oldest deadline 2024-05-12T22:21:13.063447+0000) 2024-05-12T22:21:33.930 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:33 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:33.622+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:59.365658+0000 front 2024-05-12T22:20:59.365719+0000 (oldest deadline 2024-05-12T22:21:23.465741+0000) 2024-05-12T22:21:34.243 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 12 22:21:33 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-7[52481]: 2024-05-12T22:21:33.831+0000 7f1b778cf700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:51.904377+0000 front 2024-05-12T22:20:51.904452+0000 (oldest deadline 2024-05-12T22:21:13.604466+0000) 2024-05-12T22:21:34.244 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 12 22:21:33 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-7[52481]: 2024-05-12T22:21:33.831+0000 7f1b778cf700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:59.306433+0000 front 2024-05-12T22:20:59.306362+0000 (oldest deadline 2024-05-12T22:21:22.806281+0000) 2024-05-12T22:21:34.244 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 12 22:21:33 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-7[52481]: 2024-05-12T22:21:33.831+0000 7f1b778cf700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-12T22:21:05.107730+0000 front 2024-05-12T22:21:05.107621+0000 (oldest deadline 2024-05-12T22:21:30.407516+0000) 2024-05-12T22:21:34.244 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:33 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:33.962+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:43.519531+0000 front 2024-05-12T22:20:43.519638+0000 (oldest deadline 2024-05-12T22:21:05.219288+0000) 2024-05-12T22:21:34.244 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:33 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:33.962+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:51.620499+0000 front 2024-05-12T22:20:51.620463+0000 (oldest deadline 2024-05-12T22:21:16.221349+0000) 2024-05-12T22:21:34.244 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:33 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:33.962+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:59.722447+0000 front 2024-05-12T22:20:59.722655+0000 (oldest deadline 2024-05-12T22:21:21.422316+0000) 2024-05-12T22:21:34.244 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:33 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:33.962+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-12T22:21:05.224800+0000 front 2024-05-12T22:21:05.225010+0000 (oldest deadline 2024-05-12T22:21:33.425092+0000) 2024-05-12T22:21:34.281 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 12 22:21:34 smithi028 podman[99945]: 2024-05-12 22:21:34.038207876 +0000 UTC m=+5.956149039 container cleanup be82f07b03d72704c4ab211d26bd3677bb5cd3a859476b058dc76eab6ad11864 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-6, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=squid-f72fecf, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240507) 2024-05-12T22:21:34.282 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 12 22:21:34 smithi028 bash[99945]: ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-6 2024-05-12T22:21:34.535 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 12 22:21:34 smithi028 podman[99964]: 2024-05-12 22:21:34.281009242 +0000 UTC m=+0.682779851 container remove be82f07b03d72704c4ab211d26bd3677bb5cd3a859476b058dc76eab6ad11864 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-6, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=squid-f72fecf, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240507, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-12T22:21:34.929 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 12 22:21:34 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-8[53963]: 2024-05-12T22:21:34.748+0000 7f4ff7d79700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:50.580039+0000 front 2024-05-12T22:20:50.580181+0000 (oldest deadline 2024-05-12T22:21:16.480068+0000) 2024-05-12T22:21:34.929 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 12 22:21:34 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-8[53963]: 2024-05-12T22:21:34.748+0000 7f4ff7d79700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:56.480473+0000 front 2024-05-12T22:20:56.480615+0000 (oldest deadline 2024-05-12T22:21:21.780524+0000) 2024-05-12T22:21:34.929 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 12 22:21:34 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-8[53963]: 2024-05-12T22:21:34.748+0000 7f4ff7d79700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-12T22:21:05.281742+0000 front 2024-05-12T22:21:05.281856+0000 (oldest deadline 2024-05-12T22:21:29.981483+0000) 2024-05-12T22:21:34.930 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:34 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:34.607+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.561281+0000 front 2024-05-12T22:20:41.561023+0000 (oldest deadline 2024-05-12T22:21:06.261036+0000) 2024-05-12T22:21:34.930 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:34 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:34.607+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:50.163666+0000 front 2024-05-12T22:20:50.163649+0000 (oldest deadline 2024-05-12T22:21:13.063447+0000) 2024-05-12T22:21:34.930 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:34 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:34.607+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:59.365658+0000 front 2024-05-12T22:20:59.365719+0000 (oldest deadline 2024-05-12T22:21:23.465741+0000) 2024-05-12T22:21:35.035 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:34 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:34.733+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.490538+0000 front 2024-05-12T22:20:41.490674+0000 (oldest deadline 2024-05-12T22:21:04.990968+0000) 2024-05-12T22:21:35.035 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:34 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:34.733+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:48.493555+0000 front 2024-05-12T22:20:48.493616+0000 (oldest deadline 2024-05-12T22:21:14.393780+0000) 2024-05-12T22:21:35.035 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:34 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:34.733+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:56.095925+0000 front 2024-05-12T22:20:56.096412+0000 (oldest deadline 2024-05-12T22:21:21.396155+0000) 2024-05-12T22:21:35.035 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:34 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:34.733+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-12T22:21:08.399553+0000 front 2024-05-12T22:21:08.399957+0000 (oldest deadline 2024-05-12T22:21:30.099451+0000) 2024-05-12T22:21:35.242 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 12 22:21:34 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-7[52481]: 2024-05-12T22:21:34.784+0000 7f1b778cf700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:51.904377+0000 front 2024-05-12T22:20:51.904452+0000 (oldest deadline 2024-05-12T22:21:13.604466+0000) 2024-05-12T22:21:35.242 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 12 22:21:34 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-7[52481]: 2024-05-12T22:21:34.784+0000 7f1b778cf700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:59.306433+0000 front 2024-05-12T22:20:59.306362+0000 (oldest deadline 2024-05-12T22:21:22.806281+0000) 2024-05-12T22:21:35.242 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 12 22:21:34 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-7[52481]: 2024-05-12T22:21:34.784+0000 7f1b778cf700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-12T22:21:05.107730+0000 front 2024-05-12T22:21:05.107621+0000 (oldest deadline 2024-05-12T22:21:30.407516+0000) 2024-05-12T22:21:35.242 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:34 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:34.991+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:43.519531+0000 front 2024-05-12T22:20:43.519638+0000 (oldest deadline 2024-05-12T22:21:05.219288+0000) 2024-05-12T22:21:35.243 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:34 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:34.991+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:51.620499+0000 front 2024-05-12T22:20:51.620463+0000 (oldest deadline 2024-05-12T22:21:16.221349+0000) 2024-05-12T22:21:35.243 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:34 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:34.991+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:59.722447+0000 front 2024-05-12T22:20:59.722655+0000 (oldest deadline 2024-05-12T22:21:21.422316+0000) 2024-05-12T22:21:35.243 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:34 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:34.991+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-12T22:21:05.224800+0000 front 2024-05-12T22:21:05.225010+0000 (oldest deadline 2024-05-12T22:21:33.425092+0000) 2024-05-12T22:21:35.403 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 12 22:21:35 smithi028 podman[100052]: 2024-05-12 22:21:35.000186692 +0000 UTC m=+0.040785154 image pull 8252b7944d3b5c188c0ff632647ba6cc4e9a19588ee097b8a3dba27b42779c25 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc 2024-05-12T22:21:35.404 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 12 22:21:35 smithi028 podman[100052]: 2024-05-12 22:21:35.132776232 +0000 UTC m=+0.173374734 container create a71adaa0a656ad4a186d8bf03ea48b28c0d0db2d32f2219d495278d1a6fcc5dc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-6-deactivate, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240507, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, RELEASE=squid-f72fecf) 2024-05-12T22:21:35.665 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 12 22:21:35 smithi028 podman[100052]: 2024-05-12 22:21:35.402907031 +0000 UTC m=+0.443505494 container init a71adaa0a656ad4a186d8bf03ea48b28c0d0db2d32f2219d495278d1a6fcc5dc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-6-deactivate, GIT_CLEAN=True, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240507, RELEASE=squid-f72fecf, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-05-12T22:21:35.665 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 12 22:21:35 smithi028 podman[100052]: 2024-05-12 22:21:35.414557561 +0000 UTC m=+0.455156013 container start a71adaa0a656ad4a186d8bf03ea48b28c0d0db2d32f2219d495278d1a6fcc5dc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-6-deactivate, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240507, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=squid-f72fecf, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-05-12T22:21:35.665 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 12 22:21:35 smithi028 podman[100052]: 2024-05-12 22:21:35.485083818 +0000 UTC m=+0.525682286 container attach a71adaa0a656ad4a186d8bf03ea48b28c0d0db2d32f2219d495278d1a6fcc5dc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-6-deactivate, GIT_CLEAN=True, org.label-schema.build-date=20240507, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=squid-f72fecf) 2024-05-12T22:21:35.929 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 12 22:21:35 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-8[53963]: 2024-05-12T22:21:35.719+0000 7f4ff7d79700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:50.580039+0000 front 2024-05-12T22:20:50.580181+0000 (oldest deadline 2024-05-12T22:21:16.480068+0000) 2024-05-12T22:21:35.929 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 12 22:21:35 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-8[53963]: 2024-05-12T22:21:35.719+0000 7f4ff7d79700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:56.480473+0000 front 2024-05-12T22:20:56.480615+0000 (oldest deadline 2024-05-12T22:21:21.780524+0000) 2024-05-12T22:21:35.929 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 12 22:21:35 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-8[53963]: 2024-05-12T22:21:35.719+0000 7f4ff7d79700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-12T22:21:05.281742+0000 front 2024-05-12T22:21:05.281856+0000 (oldest deadline 2024-05-12T22:21:29.981483+0000) 2024-05-12T22:21:35.930 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:35 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:35.590+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.561281+0000 front 2024-05-12T22:20:41.561023+0000 (oldest deadline 2024-05-12T22:21:06.261036+0000) 2024-05-12T22:21:35.930 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:35 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:35.590+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:50.163666+0000 front 2024-05-12T22:20:50.163649+0000 (oldest deadline 2024-05-12T22:21:13.063447+0000) 2024-05-12T22:21:35.930 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:35 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:35.590+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:59.365658+0000 front 2024-05-12T22:20:59.365719+0000 (oldest deadline 2024-05-12T22:21:23.465741+0000) 2024-05-12T22:21:35.981 INFO:journalctl@ceph.osd.6.smithi028.stdout:May 12 22:21:35 smithi028 podman[100052]: 2024-05-12 22:21:35.904591256 +0000 UTC m=+0.945189728 container died a71adaa0a656ad4a186d8bf03ea48b28c0d0db2d32f2219d495278d1a6fcc5dc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-6-deactivate, ceph=True, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, RELEASE=squid-f72fecf, org.label-schema.build-date=20240507, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-05-12T22:21:35.981 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:35 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:35.744+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.490538+0000 front 2024-05-12T22:20:41.490674+0000 (oldest deadline 2024-05-12T22:21:04.990968+0000) 2024-05-12T22:21:35.981 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:35 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:35.744+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:48.493555+0000 front 2024-05-12T22:20:48.493616+0000 (oldest deadline 2024-05-12T22:21:14.393780+0000) 2024-05-12T22:21:35.981 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:35 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:35.744+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:56.095925+0000 front 2024-05-12T22:20:56.096412+0000 (oldest deadline 2024-05-12T22:21:21.396155+0000) 2024-05-12T22:21:35.981 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:35 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:35.744+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-12T22:21:08.399553+0000 front 2024-05-12T22:21:08.399957+0000 (oldest deadline 2024-05-12T22:21:30.099451+0000) 2024-05-12T22:21:36.242 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 12 22:21:35 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-7[52481]: 2024-05-12T22:21:35.791+0000 7f1b778cf700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:51.904377+0000 front 2024-05-12T22:20:51.904452+0000 (oldest deadline 2024-05-12T22:21:13.604466+0000) 2024-05-12T22:21:36.242 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 12 22:21:35 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-7[52481]: 2024-05-12T22:21:35.791+0000 7f1b778cf700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:59.306433+0000 front 2024-05-12T22:20:59.306362+0000 (oldest deadline 2024-05-12T22:21:22.806281+0000) 2024-05-12T22:21:36.242 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 12 22:21:35 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-7[52481]: 2024-05-12T22:21:35.791+0000 7f1b778cf700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-12T22:21:05.107730+0000 front 2024-05-12T22:21:05.107621+0000 (oldest deadline 2024-05-12T22:21:30.407516+0000) 2024-05-12T22:21:36.243 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:35 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:35.948+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:43.519531+0000 front 2024-05-12T22:20:43.519638+0000 (oldest deadline 2024-05-12T22:21:05.219288+0000) 2024-05-12T22:21:36.243 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:35 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:35.948+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:51.620499+0000 front 2024-05-12T22:20:51.620463+0000 (oldest deadline 2024-05-12T22:21:16.221349+0000) 2024-05-12T22:21:36.243 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:35 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:35.948+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:59.722447+0000 front 2024-05-12T22:20:59.722655+0000 (oldest deadline 2024-05-12T22:21:21.422316+0000) 2024-05-12T22:21:36.243 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:35 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:35.948+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-12T22:21:05.224800+0000 front 2024-05-12T22:21:05.225010+0000 (oldest deadline 2024-05-12T22:21:33.425092+0000) 2024-05-12T22:21:36.616 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968@osd.6.service' 2024-05-12T22:21:36.727 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-12T22:21:36.727 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-05-12T22:21:36.728 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-05-12T22:21:36.728 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl stop ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968@osd.7 2024-05-12T22:21:36.929 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 12 22:21:36 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-8[53963]: 2024-05-12T22:21:36.699+0000 7f4ff7d79700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:50.580039+0000 front 2024-05-12T22:20:50.580181+0000 (oldest deadline 2024-05-12T22:21:16.480068+0000) 2024-05-12T22:21:36.929 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 12 22:21:36 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-8[53963]: 2024-05-12T22:21:36.699+0000 7f4ff7d79700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:56.480473+0000 front 2024-05-12T22:20:56.480615+0000 (oldest deadline 2024-05-12T22:21:21.780524+0000) 2024-05-12T22:21:36.930 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 12 22:21:36 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-8[53963]: 2024-05-12T22:21:36.699+0000 7f4ff7d79700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-12T22:21:05.281742+0000 front 2024-05-12T22:21:05.281856+0000 (oldest deadline 2024-05-12T22:21:29.981483+0000) 2024-05-12T22:21:36.930 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:36 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:36.559+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.561281+0000 front 2024-05-12T22:20:41.561023+0000 (oldest deadline 2024-05-12T22:21:06.261036+0000) 2024-05-12T22:21:36.930 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:36 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:36.559+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:50.163666+0000 front 2024-05-12T22:20:50.163649+0000 (oldest deadline 2024-05-12T22:21:13.063447+0000) 2024-05-12T22:21:36.930 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:36 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:36.559+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:59.365658+0000 front 2024-05-12T22:20:59.365719+0000 (oldest deadline 2024-05-12T22:21:23.465741+0000) 2024-05-12T22:21:36.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:36 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:36.705+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.490538+0000 front 2024-05-12T22:20:41.490674+0000 (oldest deadline 2024-05-12T22:21:04.990968+0000) 2024-05-12T22:21:36.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:36 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:36.705+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:48.493555+0000 front 2024-05-12T22:20:48.493616+0000 (oldest deadline 2024-05-12T22:21:14.393780+0000) 2024-05-12T22:21:36.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:36 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:36.705+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:56.095925+0000 front 2024-05-12T22:20:56.096412+0000 (oldest deadline 2024-05-12T22:21:21.396155+0000) 2024-05-12T22:21:36.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:36 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:36.705+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-12T22:21:08.399553+0000 front 2024-05-12T22:21:08.399957+0000 (oldest deadline 2024-05-12T22:21:30.099451+0000) 2024-05-12T22:21:37.020 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:36 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:36.935+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:43.519531+0000 front 2024-05-12T22:20:43.519638+0000 (oldest deadline 2024-05-12T22:21:05.219288+0000) 2024-05-12T22:21:37.020 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:36 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:36.935+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:51.620499+0000 front 2024-05-12T22:20:51.620463+0000 (oldest deadline 2024-05-12T22:21:16.221349+0000) 2024-05-12T22:21:37.020 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:36 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:36.935+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:59.722447+0000 front 2024-05-12T22:20:59.722655+0000 (oldest deadline 2024-05-12T22:21:21.422316+0000) 2024-05-12T22:21:37.020 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:36 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:36.935+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-12T22:21:05.224800+0000 front 2024-05-12T22:21:05.225010+0000 (oldest deadline 2024-05-12T22:21:33.425092+0000) 2024-05-12T22:21:37.020 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 12 22:21:36 smithi086 systemd[1]: Stopping Ceph osd.7 for 6b92e7b4-10aa-11ef-bc98-c7b262605968... 2024-05-12T22:21:37.021 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 12 22:21:36 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-7[52481]: 2024-05-12T22:21:36.817+0000 7f1b778cf700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:51.904377+0000 front 2024-05-12T22:20:51.904452+0000 (oldest deadline 2024-05-12T22:21:13.604466+0000) 2024-05-12T22:21:37.021 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 12 22:21:36 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-7[52481]: 2024-05-12T22:21:36.817+0000 7f1b778cf700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:59.306433+0000 front 2024-05-12T22:20:59.306362+0000 (oldest deadline 2024-05-12T22:21:22.806281+0000) 2024-05-12T22:21:37.021 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 12 22:21:36 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-7[52481]: 2024-05-12T22:21:36.817+0000 7f1b778cf700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-12T22:21:05.107730+0000 front 2024-05-12T22:21:05.107621+0000 (oldest deadline 2024-05-12T22:21:30.407516+0000) 2024-05-12T22:21:37.021 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 12 22:21:37 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-7[52481]: 2024-05-12T22:21:37.019+0000 7f1b7aada700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-12T22:21:37.021 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 12 22:21:37 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-7[52481]: 2024-05-12T22:21:37.019+0000 7f1b7aada700 -1 osd.7 102 *** Got signal Terminated *** 2024-05-12T22:21:37.021 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 12 22:21:37 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-7[52481]: 2024-05-12T22:21:37.019+0000 7f1b7aada700 -1 osd.7 102 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-12T22:21:37.831 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 12 22:21:37 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-8[53963]: 2024-05-12T22:21:37.715+0000 7f4ff7d79700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:50.580039+0000 front 2024-05-12T22:20:50.580181+0000 (oldest deadline 2024-05-12T22:21:16.480068+0000) 2024-05-12T22:21:37.831 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 12 22:21:37 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-8[53963]: 2024-05-12T22:21:37.715+0000 7f4ff7d79700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:56.480473+0000 front 2024-05-12T22:20:56.480615+0000 (oldest deadline 2024-05-12T22:21:21.780524+0000) 2024-05-12T22:21:37.831 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 12 22:21:37 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-8[53963]: 2024-05-12T22:21:37.715+0000 7f4ff7d79700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-12T22:21:05.281742+0000 front 2024-05-12T22:21:05.281856+0000 (oldest deadline 2024-05-12T22:21:29.981483+0000) 2024-05-12T22:21:37.831 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:37 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:37.547+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.561281+0000 front 2024-05-12T22:20:41.561023+0000 (oldest deadline 2024-05-12T22:21:06.261036+0000) 2024-05-12T22:21:37.831 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:37 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:37.547+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:50.163666+0000 front 2024-05-12T22:20:50.163649+0000 (oldest deadline 2024-05-12T22:21:13.063447+0000) 2024-05-12T22:21:37.832 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:37 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:37.547+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:59.365658+0000 front 2024-05-12T22:20:59.365719+0000 (oldest deadline 2024-05-12T22:21:23.465741+0000) 2024-05-12T22:21:38.035 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:37 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:37.723+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.490538+0000 front 2024-05-12T22:20:41.490674+0000 (oldest deadline 2024-05-12T22:21:04.990968+0000) 2024-05-12T22:21:38.035 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:37 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:37.723+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:48.493555+0000 front 2024-05-12T22:20:48.493616+0000 (oldest deadline 2024-05-12T22:21:14.393780+0000) 2024-05-12T22:21:38.035 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:37 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:37.723+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:56.095925+0000 front 2024-05-12T22:20:56.096412+0000 (oldest deadline 2024-05-12T22:21:21.396155+0000) 2024-05-12T22:21:38.035 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:37 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:37.723+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-12T22:21:08.399553+0000 front 2024-05-12T22:21:08.399957+0000 (oldest deadline 2024-05-12T22:21:30.099451+0000) 2024-05-12T22:21:38.242 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 12 22:21:37 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-7[52481]: 2024-05-12T22:21:37.799+0000 7f1b778cf700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:51.904377+0000 front 2024-05-12T22:20:51.904452+0000 (oldest deadline 2024-05-12T22:21:13.604466+0000) 2024-05-12T22:21:38.242 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 12 22:21:37 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-7[52481]: 2024-05-12T22:21:37.799+0000 7f1b778cf700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:59.306433+0000 front 2024-05-12T22:20:59.306362+0000 (oldest deadline 2024-05-12T22:21:22.806281+0000) 2024-05-12T22:21:38.242 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 12 22:21:37 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-7[52481]: 2024-05-12T22:21:37.799+0000 7f1b778cf700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-12T22:21:05.107730+0000 front 2024-05-12T22:21:05.107621+0000 (oldest deadline 2024-05-12T22:21:30.407516+0000) 2024-05-12T22:21:38.243 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:37 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:37.893+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:43.519531+0000 front 2024-05-12T22:20:43.519638+0000 (oldest deadline 2024-05-12T22:21:05.219288+0000) 2024-05-12T22:21:38.243 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:37 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:37.893+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:51.620499+0000 front 2024-05-12T22:20:51.620463+0000 (oldest deadline 2024-05-12T22:21:16.221349+0000) 2024-05-12T22:21:38.243 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:37 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:37.893+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:59.722447+0000 front 2024-05-12T22:20:59.722655+0000 (oldest deadline 2024-05-12T22:21:21.422316+0000) 2024-05-12T22:21:38.243 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:37 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:37.893+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-12T22:21:05.224800+0000 front 2024-05-12T22:21:05.225010+0000 (oldest deadline 2024-05-12T22:21:33.425092+0000) 2024-05-12T22:21:38.929 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:38 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:38.517+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.561281+0000 front 2024-05-12T22:20:41.561023+0000 (oldest deadline 2024-05-12T22:21:06.261036+0000) 2024-05-12T22:21:38.930 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:38 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:38.517+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:50.163666+0000 front 2024-05-12T22:20:50.163649+0000 (oldest deadline 2024-05-12T22:21:13.063447+0000) 2024-05-12T22:21:38.930 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:38 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:38.517+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:59.365658+0000 front 2024-05-12T22:20:59.365719+0000 (oldest deadline 2024-05-12T22:21:23.465741+0000) 2024-05-12T22:21:38.930 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:38 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:38.517+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-05-12T22:21:14.568000+0000 front 2024-05-12T22:21:14.568013+0000 (oldest deadline 2024-05-12T22:21:38.067911+0000) 2024-05-12T22:21:38.930 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 12 22:21:38 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-8[53963]: 2024-05-12T22:21:38.753+0000 7f4ff7d79700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:50.580039+0000 front 2024-05-12T22:20:50.580181+0000 (oldest deadline 2024-05-12T22:21:16.480068+0000) 2024-05-12T22:21:38.930 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 12 22:21:38 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-8[53963]: 2024-05-12T22:21:38.753+0000 7f4ff7d79700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:56.480473+0000 front 2024-05-12T22:20:56.480615+0000 (oldest deadline 2024-05-12T22:21:21.780524+0000) 2024-05-12T22:21:38.930 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 12 22:21:38 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-8[53963]: 2024-05-12T22:21:38.753+0000 7f4ff7d79700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-12T22:21:05.281742+0000 front 2024-05-12T22:21:05.281856+0000 (oldest deadline 2024-05-12T22:21:29.981483+0000) 2024-05-12T22:21:38.931 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 12 22:21:38 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-8[53963]: 2024-05-12T22:21:38.753+0000 7f4ff7d79700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-05-12T22:21:14.082707+0000 front 2024-05-12T22:21:14.082684+0000 (oldest deadline 2024-05-12T22:21:38.182601+0000) 2024-05-12T22:21:39.035 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:38 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:38.762+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.490538+0000 front 2024-05-12T22:20:41.490674+0000 (oldest deadline 2024-05-12T22:21:04.990968+0000) 2024-05-12T22:21:39.035 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:38 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:38.762+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:48.493555+0000 front 2024-05-12T22:20:48.493616+0000 (oldest deadline 2024-05-12T22:21:14.393780+0000) 2024-05-12T22:21:39.035 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:38 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:38.762+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:56.095925+0000 front 2024-05-12T22:20:56.096412+0000 (oldest deadline 2024-05-12T22:21:21.396155+0000) 2024-05-12T22:21:39.035 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:38 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:38.762+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-12T22:21:08.399553+0000 front 2024-05-12T22:21:08.399957+0000 (oldest deadline 2024-05-12T22:21:30.099451+0000) 2024-05-12T22:21:39.242 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 12 22:21:38 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-7[52481]: 2024-05-12T22:21:38.791+0000 7f1b778cf700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:51.904377+0000 front 2024-05-12T22:20:51.904452+0000 (oldest deadline 2024-05-12T22:21:13.604466+0000) 2024-05-12T22:21:39.242 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 12 22:21:38 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-7[52481]: 2024-05-12T22:21:38.791+0000 7f1b778cf700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:59.306433+0000 front 2024-05-12T22:20:59.306362+0000 (oldest deadline 2024-05-12T22:21:22.806281+0000) 2024-05-12T22:21:39.242 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 12 22:21:38 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-7[52481]: 2024-05-12T22:21:38.791+0000 7f1b778cf700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-12T22:21:05.107730+0000 front 2024-05-12T22:21:05.107621+0000 (oldest deadline 2024-05-12T22:21:30.407516+0000) 2024-05-12T22:21:39.242 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 12 22:21:38 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-7[52481]: 2024-05-12T22:21:38.791+0000 7f1b778cf700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-05-12T22:21:16.808980+0000 front 2024-05-12T22:21:16.809053+0000 (oldest deadline 2024-05-12T22:21:37.909247+0000) 2024-05-12T22:21:39.243 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:38 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:38.941+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:43.519531+0000 front 2024-05-12T22:20:43.519638+0000 (oldest deadline 2024-05-12T22:21:05.219288+0000) 2024-05-12T22:21:39.243 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:38 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:38.941+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:51.620499+0000 front 2024-05-12T22:20:51.620463+0000 (oldest deadline 2024-05-12T22:21:16.221349+0000) 2024-05-12T22:21:39.243 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:38 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:38.941+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:59.722447+0000 front 2024-05-12T22:20:59.722655+0000 (oldest deadline 2024-05-12T22:21:21.422316+0000) 2024-05-12T22:21:39.243 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:38 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:38.941+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-12T22:21:05.224800+0000 front 2024-05-12T22:21:05.225010+0000 (oldest deadline 2024-05-12T22:21:33.425092+0000) 2024-05-12T22:21:39.243 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:38 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:38.941+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-05-12T22:21:16.826477+0000 front 2024-05-12T22:21:16.826438+0000 (oldest deadline 2024-05-12T22:21:37.926536+0000) 2024-05-12T22:21:39.929 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:39 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:39.543+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.561281+0000 front 2024-05-12T22:20:41.561023+0000 (oldest deadline 2024-05-12T22:21:06.261036+0000) 2024-05-12T22:21:39.929 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:39 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:39.543+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:50.163666+0000 front 2024-05-12T22:20:50.163649+0000 (oldest deadline 2024-05-12T22:21:13.063447+0000) 2024-05-12T22:21:39.930 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:39 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:39.543+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:59.365658+0000 front 2024-05-12T22:20:59.365719+0000 (oldest deadline 2024-05-12T22:21:23.465741+0000) 2024-05-12T22:21:39.930 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:39 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:39.543+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-05-12T22:21:14.568000+0000 front 2024-05-12T22:21:14.568013+0000 (oldest deadline 2024-05-12T22:21:38.067911+0000) 2024-05-12T22:21:39.930 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 12 22:21:39 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-8[53963]: 2024-05-12T22:21:39.719+0000 7f4ff7d79700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:50.580039+0000 front 2024-05-12T22:20:50.580181+0000 (oldest deadline 2024-05-12T22:21:16.480068+0000) 2024-05-12T22:21:39.930 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 12 22:21:39 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-8[53963]: 2024-05-12T22:21:39.719+0000 7f4ff7d79700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:56.480473+0000 front 2024-05-12T22:20:56.480615+0000 (oldest deadline 2024-05-12T22:21:21.780524+0000) 2024-05-12T22:21:39.930 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 12 22:21:39 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-8[53963]: 2024-05-12T22:21:39.719+0000 7f4ff7d79700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-12T22:21:05.281742+0000 front 2024-05-12T22:21:05.281856+0000 (oldest deadline 2024-05-12T22:21:29.981483+0000) 2024-05-12T22:21:39.930 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 12 22:21:39 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-8[53963]: 2024-05-12T22:21:39.719+0000 7f4ff7d79700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-05-12T22:21:14.082707+0000 front 2024-05-12T22:21:14.082684+0000 (oldest deadline 2024-05-12T22:21:38.182601+0000) 2024-05-12T22:21:40.035 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:39 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:39.724+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.490538+0000 front 2024-05-12T22:20:41.490674+0000 (oldest deadline 2024-05-12T22:21:04.990968+0000) 2024-05-12T22:21:40.035 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:39 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:39.724+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:48.493555+0000 front 2024-05-12T22:20:48.493616+0000 (oldest deadline 2024-05-12T22:21:14.393780+0000) 2024-05-12T22:21:40.035 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:39 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:39.724+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:56.095925+0000 front 2024-05-12T22:20:56.096412+0000 (oldest deadline 2024-05-12T22:21:21.396155+0000) 2024-05-12T22:21:40.035 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:39 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:39.724+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-12T22:21:08.399553+0000 front 2024-05-12T22:21:08.399957+0000 (oldest deadline 2024-05-12T22:21:30.099451+0000) 2024-05-12T22:21:40.242 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 12 22:21:39 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-7[52481]: 2024-05-12T22:21:39.800+0000 7f1b778cf700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:51.904377+0000 front 2024-05-12T22:20:51.904452+0000 (oldest deadline 2024-05-12T22:21:13.604466+0000) 2024-05-12T22:21:40.242 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 12 22:21:39 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-7[52481]: 2024-05-12T22:21:39.800+0000 7f1b778cf700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:59.306433+0000 front 2024-05-12T22:20:59.306362+0000 (oldest deadline 2024-05-12T22:21:22.806281+0000) 2024-05-12T22:21:40.242 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 12 22:21:39 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-7[52481]: 2024-05-12T22:21:39.800+0000 7f1b778cf700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-12T22:21:05.107730+0000 front 2024-05-12T22:21:05.107621+0000 (oldest deadline 2024-05-12T22:21:30.407516+0000) 2024-05-12T22:21:40.243 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 12 22:21:39 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-7[52481]: 2024-05-12T22:21:39.800+0000 7f1b778cf700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-05-12T22:21:16.808980+0000 front 2024-05-12T22:21:16.809053+0000 (oldest deadline 2024-05-12T22:21:37.909247+0000) 2024-05-12T22:21:40.243 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:39 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:39.988+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:43.519531+0000 front 2024-05-12T22:20:43.519638+0000 (oldest deadline 2024-05-12T22:21:05.219288+0000) 2024-05-12T22:21:40.243 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:39 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:39.988+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:51.620499+0000 front 2024-05-12T22:20:51.620463+0000 (oldest deadline 2024-05-12T22:21:16.221349+0000) 2024-05-12T22:21:40.243 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:39 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:39.988+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:59.722447+0000 front 2024-05-12T22:20:59.722655+0000 (oldest deadline 2024-05-12T22:21:21.422316+0000) 2024-05-12T22:21:40.243 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:39 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:39.988+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-12T22:21:05.224800+0000 front 2024-05-12T22:21:05.225010+0000 (oldest deadline 2024-05-12T22:21:33.425092+0000) 2024-05-12T22:21:40.243 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:39 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:39.988+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-05-12T22:21:16.826477+0000 front 2024-05-12T22:21:16.826438+0000 (oldest deadline 2024-05-12T22:21:37.926536+0000) 2024-05-12T22:21:40.929 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:40 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:40.529+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.561281+0000 front 2024-05-12T22:20:41.561023+0000 (oldest deadline 2024-05-12T22:21:06.261036+0000) 2024-05-12T22:21:40.929 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:40 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:40.529+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:50.163666+0000 front 2024-05-12T22:20:50.163649+0000 (oldest deadline 2024-05-12T22:21:13.063447+0000) 2024-05-12T22:21:40.929 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:40 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:40.529+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:59.365658+0000 front 2024-05-12T22:20:59.365719+0000 (oldest deadline 2024-05-12T22:21:23.465741+0000) 2024-05-12T22:21:40.929 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:40 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:40.529+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-05-12T22:21:14.568000+0000 front 2024-05-12T22:21:14.568013+0000 (oldest deadline 2024-05-12T22:21:38.067911+0000) 2024-05-12T22:21:40.930 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 12 22:21:40 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-8[53963]: 2024-05-12T22:21:40.746+0000 7f4ff7d79700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:50.580039+0000 front 2024-05-12T22:20:50.580181+0000 (oldest deadline 2024-05-12T22:21:16.480068+0000) 2024-05-12T22:21:40.930 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 12 22:21:40 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-8[53963]: 2024-05-12T22:21:40.746+0000 7f4ff7d79700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:56.480473+0000 front 2024-05-12T22:20:56.480615+0000 (oldest deadline 2024-05-12T22:21:21.780524+0000) 2024-05-12T22:21:40.930 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 12 22:21:40 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-8[53963]: 2024-05-12T22:21:40.746+0000 7f4ff7d79700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-12T22:21:05.281742+0000 front 2024-05-12T22:21:05.281856+0000 (oldest deadline 2024-05-12T22:21:29.981483+0000) 2024-05-12T22:21:40.930 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 12 22:21:40 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-8[53963]: 2024-05-12T22:21:40.746+0000 7f4ff7d79700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-05-12T22:21:14.082707+0000 front 2024-05-12T22:21:14.082684+0000 (oldest deadline 2024-05-12T22:21:38.182601+0000) 2024-05-12T22:21:41.035 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:40 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:40.719+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.490538+0000 front 2024-05-12T22:20:41.490674+0000 (oldest deadline 2024-05-12T22:21:04.990968+0000) 2024-05-12T22:21:41.035 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:40 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:40.719+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:48.493555+0000 front 2024-05-12T22:20:48.493616+0000 (oldest deadline 2024-05-12T22:21:14.393780+0000) 2024-05-12T22:21:41.035 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:40 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:40.719+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:56.095925+0000 front 2024-05-12T22:20:56.096412+0000 (oldest deadline 2024-05-12T22:21:21.396155+0000) 2024-05-12T22:21:41.036 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:40 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:40.719+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-12T22:21:08.399553+0000 front 2024-05-12T22:21:08.399957+0000 (oldest deadline 2024-05-12T22:21:30.099451+0000) 2024-05-12T22:21:41.036 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:40 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:40.719+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-05-12T22:21:16.001427+0000 front 2024-05-12T22:21:16.001354+0000 (oldest deadline 2024-05-12T22:21:40.701607+0000) 2024-05-12T22:21:41.242 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 12 22:21:40 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-7[52481]: 2024-05-12T22:21:40.843+0000 7f1b778cf700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:51.904377+0000 front 2024-05-12T22:20:51.904452+0000 (oldest deadline 2024-05-12T22:21:13.604466+0000) 2024-05-12T22:21:41.242 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 12 22:21:40 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-7[52481]: 2024-05-12T22:21:40.843+0000 7f1b778cf700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:59.306433+0000 front 2024-05-12T22:20:59.306362+0000 (oldest deadline 2024-05-12T22:21:22.806281+0000) 2024-05-12T22:21:41.242 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 12 22:21:40 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-7[52481]: 2024-05-12T22:21:40.843+0000 7f1b778cf700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-12T22:21:05.107730+0000 front 2024-05-12T22:21:05.107621+0000 (oldest deadline 2024-05-12T22:21:30.407516+0000) 2024-05-12T22:21:41.243 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 12 22:21:40 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-7[52481]: 2024-05-12T22:21:40.843+0000 7f1b778cf700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-05-12T22:21:16.808980+0000 front 2024-05-12T22:21:16.809053+0000 (oldest deadline 2024-05-12T22:21:37.909247+0000) 2024-05-12T22:21:41.243 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:41 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:41.006+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:43.519531+0000 front 2024-05-12T22:20:43.519638+0000 (oldest deadline 2024-05-12T22:21:05.219288+0000) 2024-05-12T22:21:41.243 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:41 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:41.006+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:51.620499+0000 front 2024-05-12T22:20:51.620463+0000 (oldest deadline 2024-05-12T22:21:16.221349+0000) 2024-05-12T22:21:41.243 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:41 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:41.006+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:59.722447+0000 front 2024-05-12T22:20:59.722655+0000 (oldest deadline 2024-05-12T22:21:21.422316+0000) 2024-05-12T22:21:41.243 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:41 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:41.006+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-12T22:21:05.224800+0000 front 2024-05-12T22:21:05.225010+0000 (oldest deadline 2024-05-12T22:21:33.425092+0000) 2024-05-12T22:21:41.243 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:41 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:41.006+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-05-12T22:21:16.826477+0000 front 2024-05-12T22:21:16.826438+0000 (oldest deadline 2024-05-12T22:21:37.926536+0000) 2024-05-12T22:21:41.768 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:41 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:41.495+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.561281+0000 front 2024-05-12T22:20:41.561023+0000 (oldest deadline 2024-05-12T22:21:06.261036+0000) 2024-05-12T22:21:41.768 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:41 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:41.495+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:50.163666+0000 front 2024-05-12T22:20:50.163649+0000 (oldest deadline 2024-05-12T22:21:13.063447+0000) 2024-05-12T22:21:41.768 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:41 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:41.495+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:59.365658+0000 front 2024-05-12T22:20:59.365719+0000 (oldest deadline 2024-05-12T22:21:23.465741+0000) 2024-05-12T22:21:41.768 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:41 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:41.495+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-05-12T22:21:14.568000+0000 front 2024-05-12T22:21:14.568013+0000 (oldest deadline 2024-05-12T22:21:38.067911+0000) 2024-05-12T22:21:42.035 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:41 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:41.733+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.490538+0000 front 2024-05-12T22:20:41.490674+0000 (oldest deadline 2024-05-12T22:21:04.990968+0000) 2024-05-12T22:21:42.035 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:41 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:41.733+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:48.493555+0000 front 2024-05-12T22:20:48.493616+0000 (oldest deadline 2024-05-12T22:21:14.393780+0000) 2024-05-12T22:21:42.036 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:41 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:41.733+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:56.095925+0000 front 2024-05-12T22:20:56.096412+0000 (oldest deadline 2024-05-12T22:21:21.396155+0000) 2024-05-12T22:21:42.036 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:41 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:41.733+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-12T22:21:08.399553+0000 front 2024-05-12T22:21:08.399957+0000 (oldest deadline 2024-05-12T22:21:30.099451+0000) 2024-05-12T22:21:42.036 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:41 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:41.733+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-05-12T22:21:16.001427+0000 front 2024-05-12T22:21:16.001354+0000 (oldest deadline 2024-05-12T22:21:40.701607+0000) 2024-05-12T22:21:42.179 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 12 22:21:41 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-8[53963]: 2024-05-12T22:21:41.768+0000 7f4ff7d79700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:50.580039+0000 front 2024-05-12T22:20:50.580181+0000 (oldest deadline 2024-05-12T22:21:16.480068+0000) 2024-05-12T22:21:42.179 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 12 22:21:41 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-8[53963]: 2024-05-12T22:21:41.768+0000 7f4ff7d79700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:56.480473+0000 front 2024-05-12T22:20:56.480615+0000 (oldest deadline 2024-05-12T22:21:21.780524+0000) 2024-05-12T22:21:42.179 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 12 22:21:41 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-8[53963]: 2024-05-12T22:21:41.768+0000 7f4ff7d79700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-12T22:21:05.281742+0000 front 2024-05-12T22:21:05.281856+0000 (oldest deadline 2024-05-12T22:21:29.981483+0000) 2024-05-12T22:21:42.180 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 12 22:21:41 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-8[53963]: 2024-05-12T22:21:41.768+0000 7f4ff7d79700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-05-12T22:21:14.082707+0000 front 2024-05-12T22:21:14.082684+0000 (oldest deadline 2024-05-12T22:21:38.182601+0000) 2024-05-12T22:21:42.242 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 12 22:21:41 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-7[52481]: 2024-05-12T22:21:41.879+0000 7f1b778cf700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:51.904377+0000 front 2024-05-12T22:20:51.904452+0000 (oldest deadline 2024-05-12T22:21:13.604466+0000) 2024-05-12T22:21:42.242 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 12 22:21:41 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-7[52481]: 2024-05-12T22:21:41.879+0000 7f1b778cf700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:59.306433+0000 front 2024-05-12T22:20:59.306362+0000 (oldest deadline 2024-05-12T22:21:22.806281+0000) 2024-05-12T22:21:42.242 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 12 22:21:41 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-7[52481]: 2024-05-12T22:21:41.879+0000 7f1b778cf700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-12T22:21:05.107730+0000 front 2024-05-12T22:21:05.107621+0000 (oldest deadline 2024-05-12T22:21:30.407516+0000) 2024-05-12T22:21:42.243 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 12 22:21:41 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-7[52481]: 2024-05-12T22:21:41.879+0000 7f1b778cf700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-05-12T22:21:16.808980+0000 front 2024-05-12T22:21:16.809053+0000 (oldest deadline 2024-05-12T22:21:37.909247+0000) 2024-05-12T22:21:42.243 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:42 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:42.029+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:43.519531+0000 front 2024-05-12T22:20:43.519638+0000 (oldest deadline 2024-05-12T22:21:05.219288+0000) 2024-05-12T22:21:42.243 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:42 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:42.029+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:51.620499+0000 front 2024-05-12T22:20:51.620463+0000 (oldest deadline 2024-05-12T22:21:16.221349+0000) 2024-05-12T22:21:42.243 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:42 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:42.029+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:59.722447+0000 front 2024-05-12T22:20:59.722655+0000 (oldest deadline 2024-05-12T22:21:21.422316+0000) 2024-05-12T22:21:42.243 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:42 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:42.029+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-12T22:21:05.224800+0000 front 2024-05-12T22:21:05.225010+0000 (oldest deadline 2024-05-12T22:21:33.425092+0000) 2024-05-12T22:21:42.243 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:42 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:42.029+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-05-12T22:21:16.826477+0000 front 2024-05-12T22:21:16.826438+0000 (oldest deadline 2024-05-12T22:21:37.926536+0000) 2024-05-12T22:21:42.670 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 12 22:21:42 smithi086 podman[70992]: 2024-05-12 22:21:42.374501261 +0000 UTC m=+5.452386905 container died 455895dd3a6cc83ab9c4970b3e4fb0b9207644487db9287362b06ea4c56757c0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-7, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240507, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, RELEASE=squid-f72fecf, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-05-12T22:21:42.802 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:42 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:42.482+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.561281+0000 front 2024-05-12T22:20:41.561023+0000 (oldest deadline 2024-05-12T22:21:06.261036+0000) 2024-05-12T22:21:42.802 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:42 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:42.482+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:50.163666+0000 front 2024-05-12T22:20:50.163649+0000 (oldest deadline 2024-05-12T22:21:13.063447+0000) 2024-05-12T22:21:42.802 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:42 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:42.482+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:59.365658+0000 front 2024-05-12T22:20:59.365719+0000 (oldest deadline 2024-05-12T22:21:23.465741+0000) 2024-05-12T22:21:42.802 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:42 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:42.482+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-05-12T22:21:14.568000+0000 front 2024-05-12T22:21:14.568013+0000 (oldest deadline 2024-05-12T22:21:38.067911+0000) 2024-05-12T22:21:42.978 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 12 22:21:42 smithi086 podman[70992]: 2024-05-12 22:21:42.76152643 +0000 UTC m=+5.839412075 container cleanup 455895dd3a6cc83ab9c4970b3e4fb0b9207644487db9287362b06ea4c56757c0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-7, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240507, org.label-schema.vendor=CentOS, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, ceph=True, org.label-schema.schema-version=1.0, RELEASE=squid-f72fecf, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-05-12T22:21:42.979 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 12 22:21:42 smithi086 bash[70992]: ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-7 2024-05-12T22:21:43.035 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:42 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:42.707+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.490538+0000 front 2024-05-12T22:20:41.490674+0000 (oldest deadline 2024-05-12T22:21:04.990968+0000) 2024-05-12T22:21:43.035 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:42 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:42.707+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:48.493555+0000 front 2024-05-12T22:20:48.493616+0000 (oldest deadline 2024-05-12T22:21:14.393780+0000) 2024-05-12T22:21:43.035 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:42 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:42.707+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:56.095925+0000 front 2024-05-12T22:20:56.096412+0000 (oldest deadline 2024-05-12T22:21:21.396155+0000) 2024-05-12T22:21:43.035 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:42 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:42.707+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-12T22:21:08.399553+0000 front 2024-05-12T22:21:08.399957+0000 (oldest deadline 2024-05-12T22:21:30.099451+0000) 2024-05-12T22:21:43.035 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:42 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:42.707+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-05-12T22:21:16.001427+0000 front 2024-05-12T22:21:16.001354+0000 (oldest deadline 2024-05-12T22:21:40.701607+0000) 2024-05-12T22:21:43.179 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 12 22:21:42 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-8[53963]: 2024-05-12T22:21:42.802+0000 7f4ff7d79700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:50.580039+0000 front 2024-05-12T22:20:50.580181+0000 (oldest deadline 2024-05-12T22:21:16.480068+0000) 2024-05-12T22:21:43.179 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 12 22:21:42 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-8[53963]: 2024-05-12T22:21:42.802+0000 7f4ff7d79700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:56.480473+0000 front 2024-05-12T22:20:56.480615+0000 (oldest deadline 2024-05-12T22:21:21.780524+0000) 2024-05-12T22:21:43.179 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 12 22:21:42 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-8[53963]: 2024-05-12T22:21:42.802+0000 7f4ff7d79700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-12T22:21:05.281742+0000 front 2024-05-12T22:21:05.281856+0000 (oldest deadline 2024-05-12T22:21:29.981483+0000) 2024-05-12T22:21:43.179 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 12 22:21:42 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-8[53963]: 2024-05-12T22:21:42.802+0000 7f4ff7d79700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-05-12T22:21:14.082707+0000 front 2024-05-12T22:21:14.082684+0000 (oldest deadline 2024-05-12T22:21:38.182601+0000) 2024-05-12T22:21:43.242 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 12 22:21:42 smithi086 podman[71010]: 2024-05-12 22:21:42.979304206 +0000 UTC m=+0.603096475 container remove 455895dd3a6cc83ab9c4970b3e4fb0b9207644487db9287362b06ea4c56757c0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-7, org.label-schema.license=GPLv2, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=squid-f72fecf, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240507, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-12T22:21:43.242 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:43 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:43.045+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:43.519531+0000 front 2024-05-12T22:20:43.519638+0000 (oldest deadline 2024-05-12T22:21:05.219288+0000) 2024-05-12T22:21:43.242 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:43 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:43.045+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:51.620499+0000 front 2024-05-12T22:20:51.620463+0000 (oldest deadline 2024-05-12T22:21:16.221349+0000) 2024-05-12T22:21:43.242 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:43 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:43.045+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:59.722447+0000 front 2024-05-12T22:20:59.722655+0000 (oldest deadline 2024-05-12T22:21:21.422316+0000) 2024-05-12T22:21:43.242 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:43 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:43.045+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-12T22:21:05.224800+0000 front 2024-05-12T22:21:05.225010+0000 (oldest deadline 2024-05-12T22:21:33.425092+0000) 2024-05-12T22:21:43.243 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:43 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:43.045+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-05-12T22:21:16.826477+0000 front 2024-05-12T22:21:16.826438+0000 (oldest deadline 2024-05-12T22:21:37.926536+0000) 2024-05-12T22:21:43.623 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 12 22:21:43 smithi086 podman[71103]: 2024-05-12 22:21:43.204759458 +0000 UTC m=+0.018134522 image pull 8252b7944d3b5c188c0ff632647ba6cc4e9a19588ee097b8a3dba27b42779c25 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc 2024-05-12T22:21:43.623 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 12 22:21:43 smithi086 podman[71103]: 2024-05-12 22:21:43.322029221 +0000 UTC m=+0.135404275 container create b07451f9e6fa42da3762ff6746de10b4221f474a242466abb0630ff08307d735 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-7-deactivate, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.build-date=20240507, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=squid-f72fecf, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-05-12T22:21:43.624 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 12 22:21:43 smithi086 podman[71103]: 2024-05-12 22:21:43.541203738 +0000 UTC m=+0.354578799 container init b07451f9e6fa42da3762ff6746de10b4221f474a242466abb0630ff08307d735 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-7-deactivate, GIT_CLEAN=True, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, RELEASE=squid-f72fecf, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240507, io.buildah.version=1.33.5) 2024-05-12T22:21:43.624 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 12 22:21:43 smithi086 podman[71103]: 2024-05-12 22:21:43.544344984 +0000 UTC m=+0.357720044 container start b07451f9e6fa42da3762ff6746de10b4221f474a242466abb0630ff08307d735 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-7-deactivate, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, RELEASE=squid-f72fecf, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240507, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-12T22:21:43.624 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 12 22:21:43 smithi086 podman[71103]: 2024-05-12 22:21:43.623994976 +0000 UTC m=+0.437370031 container attach b07451f9e6fa42da3762ff6746de10b4221f474a242466abb0630ff08307d735 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-7-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=squid-f72fecf, org.label-schema.build-date=20240507, ceph=True, maintainer=Guillaume Abrioux ) 2024-05-12T22:21:43.802 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:43 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:43.524+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.561281+0000 front 2024-05-12T22:20:41.561023+0000 (oldest deadline 2024-05-12T22:21:06.261036+0000) 2024-05-12T22:21:43.802 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:43 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:43.524+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:50.163666+0000 front 2024-05-12T22:20:50.163649+0000 (oldest deadline 2024-05-12T22:21:13.063447+0000) 2024-05-12T22:21:43.802 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:43 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:43.524+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:59.365658+0000 front 2024-05-12T22:20:59.365719+0000 (oldest deadline 2024-05-12T22:21:23.465741+0000) 2024-05-12T22:21:43.802 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:43 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:43.524+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-05-12T22:21:14.568000+0000 front 2024-05-12T22:21:14.568013+0000 (oldest deadline 2024-05-12T22:21:38.067911+0000) 2024-05-12T22:21:43.992 INFO:journalctl@ceph.osd.7.smithi086.stdout:May 12 22:21:43 smithi086 podman[71103]: 2024-05-12 22:21:43.831824306 +0000 UTC m=+0.645199367 container died b07451f9e6fa42da3762ff6746de10b4221f474a242466abb0630ff08307d735 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-7-deactivate, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, RELEASE=squid-f72fecf, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240507, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-12T22:21:44.035 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:43 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:43.677+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.490538+0000 front 2024-05-12T22:20:41.490674+0000 (oldest deadline 2024-05-12T22:21:04.990968+0000) 2024-05-12T22:21:44.035 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:43 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:43.677+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:48.493555+0000 front 2024-05-12T22:20:48.493616+0000 (oldest deadline 2024-05-12T22:21:14.393780+0000) 2024-05-12T22:21:44.035 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:43 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:43.677+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:56.095925+0000 front 2024-05-12T22:20:56.096412+0000 (oldest deadline 2024-05-12T22:21:21.396155+0000) 2024-05-12T22:21:44.036 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:43 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:43.677+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-12T22:21:08.399553+0000 front 2024-05-12T22:21:08.399957+0000 (oldest deadline 2024-05-12T22:21:30.099451+0000) 2024-05-12T22:21:44.036 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:43 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:43.677+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-05-12T22:21:16.001427+0000 front 2024-05-12T22:21:16.001354+0000 (oldest deadline 2024-05-12T22:21:40.701607+0000) 2024-05-12T22:21:44.179 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 12 22:21:43 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-8[53963]: 2024-05-12T22:21:43.802+0000 7f4ff7d79700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:50.580039+0000 front 2024-05-12T22:20:50.580181+0000 (oldest deadline 2024-05-12T22:21:16.480068+0000) 2024-05-12T22:21:44.179 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 12 22:21:43 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-8[53963]: 2024-05-12T22:21:43.802+0000 7f4ff7d79700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:56.480473+0000 front 2024-05-12T22:20:56.480615+0000 (oldest deadline 2024-05-12T22:21:21.780524+0000) 2024-05-12T22:21:44.179 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 12 22:21:43 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-8[53963]: 2024-05-12T22:21:43.802+0000 7f4ff7d79700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-12T22:21:05.281742+0000 front 2024-05-12T22:21:05.281856+0000 (oldest deadline 2024-05-12T22:21:29.981483+0000) 2024-05-12T22:21:44.179 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 12 22:21:43 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-8[53963]: 2024-05-12T22:21:43.802+0000 7f4ff7d79700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-05-12T22:21:14.082707+0000 front 2024-05-12T22:21:14.082684+0000 (oldest deadline 2024-05-12T22:21:38.182601+0000) 2024-05-12T22:21:44.419 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:44 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:44.017+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:43.519531+0000 front 2024-05-12T22:20:43.519638+0000 (oldest deadline 2024-05-12T22:21:05.219288+0000) 2024-05-12T22:21:44.419 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:44 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:44.017+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:51.620499+0000 front 2024-05-12T22:20:51.620463+0000 (oldest deadline 2024-05-12T22:21:16.221349+0000) 2024-05-12T22:21:44.419 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:44 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:44.017+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:59.722447+0000 front 2024-05-12T22:20:59.722655+0000 (oldest deadline 2024-05-12T22:21:21.422316+0000) 2024-05-12T22:21:44.419 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:44 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:44.017+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-12T22:21:05.224800+0000 front 2024-05-12T22:21:05.225010+0000 (oldest deadline 2024-05-12T22:21:33.425092+0000) 2024-05-12T22:21:44.419 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:44 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:44.017+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-05-12T22:21:16.826477+0000 front 2024-05-12T22:21:16.826438+0000 (oldest deadline 2024-05-12T22:21:37.926536+0000) 2024-05-12T22:21:44.442 DEBUG:teuthology.orchestra.run.smithi086:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968@osd.7.service' 2024-05-12T22:21:44.480 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-12T22:21:44.481 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-05-12T22:21:44.481 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-05-12T22:21:44.481 DEBUG:teuthology.orchestra.run.smithi186:> sudo systemctl stop ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968@osd.8 2024-05-12T22:21:44.770 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 12 22:21:44 smithi186 systemd[1]: Stopping Ceph osd.8 for 6b92e7b4-10aa-11ef-bc98-c7b262605968... 2024-05-12T22:21:44.771 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:44 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:44.536+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.561281+0000 front 2024-05-12T22:20:41.561023+0000 (oldest deadline 2024-05-12T22:21:06.261036+0000) 2024-05-12T22:21:44.771 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:44 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:44.536+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:50.163666+0000 front 2024-05-12T22:20:50.163649+0000 (oldest deadline 2024-05-12T22:21:13.063447+0000) 2024-05-12T22:21:44.771 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:44 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:44.536+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:59.365658+0000 front 2024-05-12T22:20:59.365719+0000 (oldest deadline 2024-05-12T22:21:23.465741+0000) 2024-05-12T22:21:44.771 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:44 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:44.536+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-05-12T22:21:14.568000+0000 front 2024-05-12T22:21:14.568013+0000 (oldest deadline 2024-05-12T22:21:38.067911+0000) 2024-05-12T22:21:45.035 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:44 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:44.633+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.490538+0000 front 2024-05-12T22:20:41.490674+0000 (oldest deadline 2024-05-12T22:21:04.990968+0000) 2024-05-12T22:21:45.035 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:44 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:44.633+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:48.493555+0000 front 2024-05-12T22:20:48.493616+0000 (oldest deadline 2024-05-12T22:21:14.393780+0000) 2024-05-12T22:21:45.036 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:44 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:44.633+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:56.095925+0000 front 2024-05-12T22:20:56.096412+0000 (oldest deadline 2024-05-12T22:21:21.396155+0000) 2024-05-12T22:21:45.036 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:44 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:44.633+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-12T22:21:08.399553+0000 front 2024-05-12T22:21:08.399957+0000 (oldest deadline 2024-05-12T22:21:30.099451+0000) 2024-05-12T22:21:45.036 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:44 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:44.633+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-05-12T22:21:16.001427+0000 front 2024-05-12T22:21:16.001354+0000 (oldest deadline 2024-05-12T22:21:40.701607+0000) 2024-05-12T22:21:45.179 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 12 22:21:44 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-8[53963]: 2024-05-12T22:21:44.770+0000 7f4ffef8c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-12T22:21:45.179 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 12 22:21:44 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-8[53963]: 2024-05-12T22:21:44.770+0000 7f4ffef8c700 -1 osd.8 102 *** Got signal Terminated *** 2024-05-12T22:21:45.180 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 12 22:21:44 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-8[53963]: 2024-05-12T22:21:44.770+0000 7f4ffef8c700 -1 osd.8 102 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-12T22:21:45.180 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 12 22:21:44 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-8[53963]: 2024-05-12T22:21:44.818+0000 7f4ff7d79700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:50.580039+0000 front 2024-05-12T22:20:50.580181+0000 (oldest deadline 2024-05-12T22:21:16.480068+0000) 2024-05-12T22:21:45.180 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 12 22:21:44 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-8[53963]: 2024-05-12T22:21:44.818+0000 7f4ff7d79700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:56.480473+0000 front 2024-05-12T22:20:56.480615+0000 (oldest deadline 2024-05-12T22:21:21.780524+0000) 2024-05-12T22:21:45.180 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 12 22:21:44 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-8[53963]: 2024-05-12T22:21:44.818+0000 7f4ff7d79700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-12T22:21:05.281742+0000 front 2024-05-12T22:21:05.281856+0000 (oldest deadline 2024-05-12T22:21:29.981483+0000) 2024-05-12T22:21:45.180 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 12 22:21:44 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-8[53963]: 2024-05-12T22:21:44.818+0000 7f4ff7d79700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-05-12T22:21:14.082707+0000 front 2024-05-12T22:21:14.082684+0000 (oldest deadline 2024-05-12T22:21:38.182601+0000) 2024-05-12T22:21:45.492 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:45 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:45.006+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:43.519531+0000 front 2024-05-12T22:20:43.519638+0000 (oldest deadline 2024-05-12T22:21:05.219288+0000) 2024-05-12T22:21:45.492 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:45 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:45.006+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:51.620499+0000 front 2024-05-12T22:20:51.620463+0000 (oldest deadline 2024-05-12T22:21:16.221349+0000) 2024-05-12T22:21:45.492 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:45 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:45.006+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:59.722447+0000 front 2024-05-12T22:20:59.722655+0000 (oldest deadline 2024-05-12T22:21:21.422316+0000) 2024-05-12T22:21:45.492 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:45 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:45.006+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-12T22:21:05.224800+0000 front 2024-05-12T22:21:05.225010+0000 (oldest deadline 2024-05-12T22:21:33.425092+0000) 2024-05-12T22:21:45.492 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:45 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:45.006+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-05-12T22:21:16.826477+0000 front 2024-05-12T22:21:16.826438+0000 (oldest deadline 2024-05-12T22:21:37.926536+0000) 2024-05-12T22:21:45.821 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:45 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:45.556+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.561281+0000 front 2024-05-12T22:20:41.561023+0000 (oldest deadline 2024-05-12T22:21:06.261036+0000) 2024-05-12T22:21:45.821 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:45 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:45.556+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:50.163666+0000 front 2024-05-12T22:20:50.163649+0000 (oldest deadline 2024-05-12T22:21:13.063447+0000) 2024-05-12T22:21:45.821 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:45 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:45.556+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:59.365658+0000 front 2024-05-12T22:20:59.365719+0000 (oldest deadline 2024-05-12T22:21:23.465741+0000) 2024-05-12T22:21:45.821 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:45 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:45.556+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-05-12T22:21:14.568000+0000 front 2024-05-12T22:21:14.568013+0000 (oldest deadline 2024-05-12T22:21:38.067911+0000) 2024-05-12T22:21:46.035 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:45 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:45.661+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.490538+0000 front 2024-05-12T22:20:41.490674+0000 (oldest deadline 2024-05-12T22:21:04.990968+0000) 2024-05-12T22:21:46.035 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:45 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:45.661+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:48.493555+0000 front 2024-05-12T22:20:48.493616+0000 (oldest deadline 2024-05-12T22:21:14.393780+0000) 2024-05-12T22:21:46.036 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:45 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:45.661+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:56.095925+0000 front 2024-05-12T22:20:56.096412+0000 (oldest deadline 2024-05-12T22:21:21.396155+0000) 2024-05-12T22:21:46.036 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:45 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:45.661+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-12T22:21:08.399553+0000 front 2024-05-12T22:21:08.399957+0000 (oldest deadline 2024-05-12T22:21:30.099451+0000) 2024-05-12T22:21:46.036 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:45 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:45.661+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-05-12T22:21:16.001427+0000 front 2024-05-12T22:21:16.001354+0000 (oldest deadline 2024-05-12T22:21:40.701607+0000) 2024-05-12T22:21:46.179 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 12 22:21:45 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-8[53963]: 2024-05-12T22:21:45.821+0000 7f4ff7d79700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:50.580039+0000 front 2024-05-12T22:20:50.580181+0000 (oldest deadline 2024-05-12T22:21:16.480068+0000) 2024-05-12T22:21:46.179 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 12 22:21:45 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-8[53963]: 2024-05-12T22:21:45.821+0000 7f4ff7d79700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:56.480473+0000 front 2024-05-12T22:20:56.480615+0000 (oldest deadline 2024-05-12T22:21:21.780524+0000) 2024-05-12T22:21:46.179 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 12 22:21:45 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-8[53963]: 2024-05-12T22:21:45.821+0000 7f4ff7d79700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-12T22:21:05.281742+0000 front 2024-05-12T22:21:05.281856+0000 (oldest deadline 2024-05-12T22:21:29.981483+0000) 2024-05-12T22:21:46.180 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 12 22:21:45 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-8[53963]: 2024-05-12T22:21:45.821+0000 7f4ff7d79700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-05-12T22:21:14.082707+0000 front 2024-05-12T22:21:14.082684+0000 (oldest deadline 2024-05-12T22:21:38.182601+0000) 2024-05-12T22:21:46.242 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:45 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:45.973+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:43.519531+0000 front 2024-05-12T22:20:43.519638+0000 (oldest deadline 2024-05-12T22:21:05.219288+0000) 2024-05-12T22:21:46.242 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:45 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:45.973+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:51.620499+0000 front 2024-05-12T22:20:51.620463+0000 (oldest deadline 2024-05-12T22:21:16.221349+0000) 2024-05-12T22:21:46.242 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:45 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:45.973+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:59.722447+0000 front 2024-05-12T22:20:59.722655+0000 (oldest deadline 2024-05-12T22:21:21.422316+0000) 2024-05-12T22:21:46.242 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:45 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:45.973+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-12T22:21:05.224800+0000 front 2024-05-12T22:21:05.225010+0000 (oldest deadline 2024-05-12T22:21:33.425092+0000) 2024-05-12T22:21:46.242 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:45 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:45.973+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-05-12T22:21:16.826477+0000 front 2024-05-12T22:21:16.826438+0000 (oldest deadline 2024-05-12T22:21:37.926536+0000) 2024-05-12T22:21:46.850 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:46 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:46.553+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.561281+0000 front 2024-05-12T22:20:41.561023+0000 (oldest deadline 2024-05-12T22:21:06.261036+0000) 2024-05-12T22:21:46.851 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:46 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:46.553+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:50.163666+0000 front 2024-05-12T22:20:50.163649+0000 (oldest deadline 2024-05-12T22:21:13.063447+0000) 2024-05-12T22:21:46.851 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:46 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:46.553+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:59.365658+0000 front 2024-05-12T22:20:59.365719+0000 (oldest deadline 2024-05-12T22:21:23.465741+0000) 2024-05-12T22:21:46.851 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:46 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:46.553+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-05-12T22:21:14.568000+0000 front 2024-05-12T22:21:14.568013+0000 (oldest deadline 2024-05-12T22:21:38.067911+0000) 2024-05-12T22:21:47.034 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:46 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:46.705+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.490538+0000 front 2024-05-12T22:20:41.490674+0000 (oldest deadline 2024-05-12T22:21:04.990968+0000) 2024-05-12T22:21:47.034 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:46 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:46.705+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:48.493555+0000 front 2024-05-12T22:20:48.493616+0000 (oldest deadline 2024-05-12T22:21:14.393780+0000) 2024-05-12T22:21:47.034 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:46 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:46.705+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:56.095925+0000 front 2024-05-12T22:20:56.096412+0000 (oldest deadline 2024-05-12T22:21:21.396155+0000) 2024-05-12T22:21:47.034 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:46 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:46.705+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-12T22:21:08.399553+0000 front 2024-05-12T22:21:08.399957+0000 (oldest deadline 2024-05-12T22:21:30.099451+0000) 2024-05-12T22:21:47.034 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:46 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:46.705+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-05-12T22:21:16.001427+0000 front 2024-05-12T22:21:16.001354+0000 (oldest deadline 2024-05-12T22:21:40.701607+0000) 2024-05-12T22:21:47.179 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 12 22:21:46 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-8[53963]: 2024-05-12T22:21:46.851+0000 7f4ff7d79700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:50.580039+0000 front 2024-05-12T22:20:50.580181+0000 (oldest deadline 2024-05-12T22:21:16.480068+0000) 2024-05-12T22:21:47.179 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 12 22:21:46 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-8[53963]: 2024-05-12T22:21:46.851+0000 7f4ff7d79700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:56.480473+0000 front 2024-05-12T22:20:56.480615+0000 (oldest deadline 2024-05-12T22:21:21.780524+0000) 2024-05-12T22:21:47.179 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 12 22:21:46 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-8[53963]: 2024-05-12T22:21:46.851+0000 7f4ff7d79700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-12T22:21:05.281742+0000 front 2024-05-12T22:21:05.281856+0000 (oldest deadline 2024-05-12T22:21:29.981483+0000) 2024-05-12T22:21:47.179 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 12 22:21:46 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-8[53963]: 2024-05-12T22:21:46.851+0000 7f4ff7d79700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-05-12T22:21:14.082707+0000 front 2024-05-12T22:21:14.082684+0000 (oldest deadline 2024-05-12T22:21:38.182601+0000) 2024-05-12T22:21:47.242 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:46 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:46.990+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:43.519531+0000 front 2024-05-12T22:20:43.519638+0000 (oldest deadline 2024-05-12T22:21:05.219288+0000) 2024-05-12T22:21:47.242 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:46 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:46.990+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:51.620499+0000 front 2024-05-12T22:20:51.620463+0000 (oldest deadline 2024-05-12T22:21:16.221349+0000) 2024-05-12T22:21:47.242 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:46 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:46.990+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:59.722447+0000 front 2024-05-12T22:20:59.722655+0000 (oldest deadline 2024-05-12T22:21:21.422316+0000) 2024-05-12T22:21:47.242 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:46 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:46.990+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-12T22:21:05.224800+0000 front 2024-05-12T22:21:05.225010+0000 (oldest deadline 2024-05-12T22:21:33.425092+0000) 2024-05-12T22:21:47.242 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:46 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:46.990+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-05-12T22:21:16.826477+0000 front 2024-05-12T22:21:16.826438+0000 (oldest deadline 2024-05-12T22:21:37.926536+0000) 2024-05-12T22:21:47.880 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:47 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:47.576+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.561281+0000 front 2024-05-12T22:20:41.561023+0000 (oldest deadline 2024-05-12T22:21:06.261036+0000) 2024-05-12T22:21:47.880 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:47 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:47.576+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:50.163666+0000 front 2024-05-12T22:20:50.163649+0000 (oldest deadline 2024-05-12T22:21:13.063447+0000) 2024-05-12T22:21:47.880 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:47 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:47.576+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:59.365658+0000 front 2024-05-12T22:20:59.365719+0000 (oldest deadline 2024-05-12T22:21:23.465741+0000) 2024-05-12T22:21:47.880 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:47 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:47.576+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-05-12T22:21:14.568000+0000 front 2024-05-12T22:21:14.568013+0000 (oldest deadline 2024-05-12T22:21:38.067911+0000) 2024-05-12T22:21:47.880 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:47 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:47.576+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-05-12T22:21:23.968898+0000 front 2024-05-12T22:21:23.968952+0000 (oldest deadline 2024-05-12T22:21:46.868939+0000) 2024-05-12T22:21:48.035 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:47 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:47.748+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.490538+0000 front 2024-05-12T22:20:41.490674+0000 (oldest deadline 2024-05-12T22:21:04.990968+0000) 2024-05-12T22:21:48.035 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:47 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:47.748+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:48.493555+0000 front 2024-05-12T22:20:48.493616+0000 (oldest deadline 2024-05-12T22:21:14.393780+0000) 2024-05-12T22:21:48.036 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:47 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:47.748+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:56.095925+0000 front 2024-05-12T22:20:56.096412+0000 (oldest deadline 2024-05-12T22:21:21.396155+0000) 2024-05-12T22:21:48.036 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:47 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:47.748+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-12T22:21:08.399553+0000 front 2024-05-12T22:21:08.399957+0000 (oldest deadline 2024-05-12T22:21:30.099451+0000) 2024-05-12T22:21:48.036 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:47 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:47.748+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-05-12T22:21:16.001427+0000 front 2024-05-12T22:21:16.001354+0000 (oldest deadline 2024-05-12T22:21:40.701607+0000) 2024-05-12T22:21:48.179 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 12 22:21:47 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-8[53963]: 2024-05-12T22:21:47.879+0000 7f4ff7d79700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:50.580039+0000 front 2024-05-12T22:20:50.580181+0000 (oldest deadline 2024-05-12T22:21:16.480068+0000) 2024-05-12T22:21:48.179 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 12 22:21:47 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-8[53963]: 2024-05-12T22:21:47.879+0000 7f4ff7d79700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:56.480473+0000 front 2024-05-12T22:20:56.480615+0000 (oldest deadline 2024-05-12T22:21:21.780524+0000) 2024-05-12T22:21:48.179 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 12 22:21:47 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-8[53963]: 2024-05-12T22:21:47.879+0000 7f4ff7d79700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-12T22:21:05.281742+0000 front 2024-05-12T22:21:05.281856+0000 (oldest deadline 2024-05-12T22:21:29.981483+0000) 2024-05-12T22:21:48.179 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 12 22:21:47 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-8[53963]: 2024-05-12T22:21:47.879+0000 7f4ff7d79700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-05-12T22:21:14.082707+0000 front 2024-05-12T22:21:14.082684+0000 (oldest deadline 2024-05-12T22:21:38.182601+0000) 2024-05-12T22:21:48.492 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:48 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:48.038+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:43.519531+0000 front 2024-05-12T22:20:43.519638+0000 (oldest deadline 2024-05-12T22:21:05.219288+0000) 2024-05-12T22:21:48.492 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:48 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:48.038+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:51.620499+0000 front 2024-05-12T22:20:51.620463+0000 (oldest deadline 2024-05-12T22:21:16.221349+0000) 2024-05-12T22:21:48.492 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:48 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:48.038+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:59.722447+0000 front 2024-05-12T22:20:59.722655+0000 (oldest deadline 2024-05-12T22:21:21.422316+0000) 2024-05-12T22:21:48.492 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:48 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:48.038+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-12T22:21:05.224800+0000 front 2024-05-12T22:21:05.225010+0000 (oldest deadline 2024-05-12T22:21:33.425092+0000) 2024-05-12T22:21:48.492 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:48 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:48.038+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-05-12T22:21:16.826477+0000 front 2024-05-12T22:21:16.826438+0000 (oldest deadline 2024-05-12T22:21:37.926536+0000) 2024-05-12T22:21:48.873 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:48 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:48.531+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.561281+0000 front 2024-05-12T22:20:41.561023+0000 (oldest deadline 2024-05-12T22:21:06.261036+0000) 2024-05-12T22:21:48.873 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:48 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:48.531+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:50.163666+0000 front 2024-05-12T22:20:50.163649+0000 (oldest deadline 2024-05-12T22:21:13.063447+0000) 2024-05-12T22:21:48.873 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:48 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:48.531+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:59.365658+0000 front 2024-05-12T22:20:59.365719+0000 (oldest deadline 2024-05-12T22:21:23.465741+0000) 2024-05-12T22:21:48.873 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:48 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:48.531+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-05-12T22:21:14.568000+0000 front 2024-05-12T22:21:14.568013+0000 (oldest deadline 2024-05-12T22:21:38.067911+0000) 2024-05-12T22:21:48.873 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:48 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:48.531+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-05-12T22:21:23.968898+0000 front 2024-05-12T22:21:23.968952+0000 (oldest deadline 2024-05-12T22:21:46.868939+0000) 2024-05-12T22:21:49.035 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:48 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:48.754+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.490538+0000 front 2024-05-12T22:20:41.490674+0000 (oldest deadline 2024-05-12T22:21:04.990968+0000) 2024-05-12T22:21:49.035 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:48 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:48.754+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:48.493555+0000 front 2024-05-12T22:20:48.493616+0000 (oldest deadline 2024-05-12T22:21:14.393780+0000) 2024-05-12T22:21:49.035 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:48 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:48.754+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:56.095925+0000 front 2024-05-12T22:20:56.096412+0000 (oldest deadline 2024-05-12T22:21:21.396155+0000) 2024-05-12T22:21:49.036 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:48 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:48.754+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-12T22:21:08.399553+0000 front 2024-05-12T22:21:08.399957+0000 (oldest deadline 2024-05-12T22:21:30.099451+0000) 2024-05-12T22:21:49.036 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:48 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:48.754+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-05-12T22:21:16.001427+0000 front 2024-05-12T22:21:16.001354+0000 (oldest deadline 2024-05-12T22:21:40.701607+0000) 2024-05-12T22:21:49.179 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 12 22:21:48 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-8[53963]: 2024-05-12T22:21:48.872+0000 7f4ff7d79700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:50.580039+0000 front 2024-05-12T22:20:50.580181+0000 (oldest deadline 2024-05-12T22:21:16.480068+0000) 2024-05-12T22:21:49.179 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 12 22:21:48 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-8[53963]: 2024-05-12T22:21:48.872+0000 7f4ff7d79700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:56.480473+0000 front 2024-05-12T22:20:56.480615+0000 (oldest deadline 2024-05-12T22:21:21.780524+0000) 2024-05-12T22:21:49.179 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 12 22:21:48 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-8[53963]: 2024-05-12T22:21:48.872+0000 7f4ff7d79700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-12T22:21:05.281742+0000 front 2024-05-12T22:21:05.281856+0000 (oldest deadline 2024-05-12T22:21:29.981483+0000) 2024-05-12T22:21:49.179 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 12 22:21:48 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-8[53963]: 2024-05-12T22:21:48.872+0000 7f4ff7d79700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-05-12T22:21:14.082707+0000 front 2024-05-12T22:21:14.082684+0000 (oldest deadline 2024-05-12T22:21:38.182601+0000) 2024-05-12T22:21:49.180 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 12 22:21:48 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-8[53963]: 2024-05-12T22:21:48.872+0000 7f4ff7d79700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-05-12T22:21:25.084407+0000 front 2024-05-12T22:21:25.084385+0000 (oldest deadline 2024-05-12T22:21:47.984473+0000) 2024-05-12T22:21:49.492 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:49 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:49.022+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:43.519531+0000 front 2024-05-12T22:20:43.519638+0000 (oldest deadline 2024-05-12T22:21:05.219288+0000) 2024-05-12T22:21:49.492 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:49 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:49.022+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:51.620499+0000 front 2024-05-12T22:20:51.620463+0000 (oldest deadline 2024-05-12T22:21:16.221349+0000) 2024-05-12T22:21:49.492 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:49 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:49.022+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:59.722447+0000 front 2024-05-12T22:20:59.722655+0000 (oldest deadline 2024-05-12T22:21:21.422316+0000) 2024-05-12T22:21:49.492 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:49 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:49.022+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-12T22:21:05.224800+0000 front 2024-05-12T22:21:05.225010+0000 (oldest deadline 2024-05-12T22:21:33.425092+0000) 2024-05-12T22:21:49.493 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:49 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:49.022+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-05-12T22:21:16.826477+0000 front 2024-05-12T22:21:16.826438+0000 (oldest deadline 2024-05-12T22:21:37.926536+0000) 2024-05-12T22:21:49.929 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:49 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:49.491+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.561281+0000 front 2024-05-12T22:20:41.561023+0000 (oldest deadline 2024-05-12T22:21:06.261036+0000) 2024-05-12T22:21:49.929 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:49 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:49.491+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:50.163666+0000 front 2024-05-12T22:20:50.163649+0000 (oldest deadline 2024-05-12T22:21:13.063447+0000) 2024-05-12T22:21:49.929 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:49 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:49.491+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:59.365658+0000 front 2024-05-12T22:20:59.365719+0000 (oldest deadline 2024-05-12T22:21:23.465741+0000) 2024-05-12T22:21:49.930 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:49 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:49.491+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-05-12T22:21:14.568000+0000 front 2024-05-12T22:21:14.568013+0000 (oldest deadline 2024-05-12T22:21:38.067911+0000) 2024-05-12T22:21:49.930 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:49 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:49.491+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-05-12T22:21:23.968898+0000 front 2024-05-12T22:21:23.968952+0000 (oldest deadline 2024-05-12T22:21:46.868939+0000) 2024-05-12T22:21:50.242 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:49 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:49.977+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:43.519531+0000 front 2024-05-12T22:20:43.519638+0000 (oldest deadline 2024-05-12T22:21:05.219288+0000) 2024-05-12T22:21:50.242 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:49 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:49.977+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:51.620499+0000 front 2024-05-12T22:20:51.620463+0000 (oldest deadline 2024-05-12T22:21:16.221349+0000) 2024-05-12T22:21:50.243 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:49 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:49.977+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:59.722447+0000 front 2024-05-12T22:20:59.722655+0000 (oldest deadline 2024-05-12T22:21:21.422316+0000) 2024-05-12T22:21:50.243 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:49 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:49.977+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-12T22:21:05.224800+0000 front 2024-05-12T22:21:05.225010+0000 (oldest deadline 2024-05-12T22:21:33.425092+0000) 2024-05-12T22:21:50.243 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:49 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:49.977+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-05-12T22:21:16.826477+0000 front 2024-05-12T22:21:16.826438+0000 (oldest deadline 2024-05-12T22:21:37.926536+0000) 2024-05-12T22:21:50.243 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:49 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:49.977+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-05-12T22:21:24.828681+0000 front 2024-05-12T22:21:24.828551+0000 (oldest deadline 2024-05-12T22:21:49.428752+0000) 2024-05-12T22:21:50.285 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:49 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:49.788+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.490538+0000 front 2024-05-12T22:20:41.490674+0000 (oldest deadline 2024-05-12T22:21:04.990968+0000) 2024-05-12T22:21:50.285 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:49 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:49.788+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:48.493555+0000 front 2024-05-12T22:20:48.493616+0000 (oldest deadline 2024-05-12T22:21:14.393780+0000) 2024-05-12T22:21:50.285 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:49 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:49.788+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:56.095925+0000 front 2024-05-12T22:20:56.096412+0000 (oldest deadline 2024-05-12T22:21:21.396155+0000) 2024-05-12T22:21:50.285 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:49 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:49.788+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-12T22:21:08.399553+0000 front 2024-05-12T22:21:08.399957+0000 (oldest deadline 2024-05-12T22:21:30.099451+0000) 2024-05-12T22:21:50.286 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:49 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:49.788+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-05-12T22:21:16.001427+0000 front 2024-05-12T22:21:16.001354+0000 (oldest deadline 2024-05-12T22:21:40.701607+0000) 2024-05-12T22:21:50.372 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 12 22:21:50 smithi186 podman[72063]: 2024-05-12 22:21:50.087033531 +0000 UTC m=+5.412670578 container died fa294356d5fb55fdbc78995f4b7d6485e62a06490405fb26a274b36ab5abd1e2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-8, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240507, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=squid-f72fecf, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e) 2024-05-12T22:21:50.679 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 12 22:21:50 smithi186 podman[72063]: 2024-05-12 22:21:50.462679593 +0000 UTC m=+5.788316632 container cleanup fa294356d5fb55fdbc78995f4b7d6485e62a06490405fb26a274b36ab5abd1e2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-8, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=squid-f72fecf, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240507, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e) 2024-05-12T22:21:50.679 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 12 22:21:50 smithi186 bash[72063]: ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-8 2024-05-12T22:21:50.680 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:50 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:50.493+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.561281+0000 front 2024-05-12T22:20:41.561023+0000 (oldest deadline 2024-05-12T22:21:06.261036+0000) 2024-05-12T22:21:50.680 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:50 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:50.493+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:50.163666+0000 front 2024-05-12T22:20:50.163649+0000 (oldest deadline 2024-05-12T22:21:13.063447+0000) 2024-05-12T22:21:50.680 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:50 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:50.493+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:59.365658+0000 front 2024-05-12T22:20:59.365719+0000 (oldest deadline 2024-05-12T22:21:23.465741+0000) 2024-05-12T22:21:50.680 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:50 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:50.493+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-05-12T22:21:14.568000+0000 front 2024-05-12T22:21:14.568013+0000 (oldest deadline 2024-05-12T22:21:38.067911+0000) 2024-05-12T22:21:50.680 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:50 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:50.493+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-05-12T22:21:23.968898+0000 front 2024-05-12T22:21:23.968952+0000 (oldest deadline 2024-05-12T22:21:46.868939+0000) 2024-05-12T22:21:51.035 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:50 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:50.746+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.490538+0000 front 2024-05-12T22:20:41.490674+0000 (oldest deadline 2024-05-12T22:21:04.990968+0000) 2024-05-12T22:21:51.036 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:50 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:50.746+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:48.493555+0000 front 2024-05-12T22:20:48.493616+0000 (oldest deadline 2024-05-12T22:21:14.393780+0000) 2024-05-12T22:21:51.036 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:50 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:50.746+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:56.095925+0000 front 2024-05-12T22:20:56.096412+0000 (oldest deadline 2024-05-12T22:21:21.396155+0000) 2024-05-12T22:21:51.036 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:50 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:50.746+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-12T22:21:08.399553+0000 front 2024-05-12T22:21:08.399957+0000 (oldest deadline 2024-05-12T22:21:30.099451+0000) 2024-05-12T22:21:51.036 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:50 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:50.746+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-05-12T22:21:16.001427+0000 front 2024-05-12T22:21:16.001354+0000 (oldest deadline 2024-05-12T22:21:40.701607+0000) 2024-05-12T22:21:51.072 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 12 22:21:50 smithi186 podman[72084]: 2024-05-12 22:21:50.763878866 +0000 UTC m=+0.668227026 container remove fa294356d5fb55fdbc78995f4b7d6485e62a06490405fb26a274b36ab5abd1e2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-8, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=squid-f72fecf, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240507, io.buildah.version=1.33.5) 2024-05-12T22:21:51.242 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:50 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:50.946+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:43.519531+0000 front 2024-05-12T22:20:43.519638+0000 (oldest deadline 2024-05-12T22:21:05.219288+0000) 2024-05-12T22:21:51.242 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:50 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:50.946+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:51.620499+0000 front 2024-05-12T22:20:51.620463+0000 (oldest deadline 2024-05-12T22:21:16.221349+0000) 2024-05-12T22:21:51.243 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:50 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:50.946+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:59.722447+0000 front 2024-05-12T22:20:59.722655+0000 (oldest deadline 2024-05-12T22:21:21.422316+0000) 2024-05-12T22:21:51.243 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:50 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:50.946+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-12T22:21:05.224800+0000 front 2024-05-12T22:21:05.225010+0000 (oldest deadline 2024-05-12T22:21:33.425092+0000) 2024-05-12T22:21:51.243 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:50 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:50.946+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-05-12T22:21:16.826477+0000 front 2024-05-12T22:21:16.826438+0000 (oldest deadline 2024-05-12T22:21:37.926536+0000) 2024-05-12T22:21:51.243 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:50 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:50.946+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-05-12T22:21:24.828681+0000 front 2024-05-12T22:21:24.828551+0000 (oldest deadline 2024-05-12T22:21:49.428752+0000) 2024-05-12T22:21:51.361 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 12 22:21:51 smithi186 podman[72175]: 2024-05-12 22:21:51.072954067 +0000 UTC m=+0.104977713 container create 65f287790a753cf9866c27a91fe39e0627cb8f07dd9c95dd16559ba687b82dfe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-8-deactivate, RELEASE=squid-f72fecf, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240507, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-05-12T22:21:51.361 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 12 22:21:51 smithi186 podman[72175]: 2024-05-12 22:21:50.985845779 +0000 UTC m=+0.017869439 image pull 8252b7944d3b5c188c0ff632647ba6cc4e9a19588ee097b8a3dba27b42779c25 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc 2024-05-12T22:21:51.361 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 12 22:21:51 smithi186 podman[72175]: 2024-05-12 22:21:51.286378224 +0000 UTC m=+0.318401866 container init 65f287790a753cf9866c27a91fe39e0627cb8f07dd9c95dd16559ba687b82dfe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-8-deactivate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240507, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=squid-f72fecf, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-12T22:21:51.361 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 12 22:21:51 smithi186 podman[72175]: 2024-05-12 22:21:51.289546442 +0000 UTC m=+0.321570085 container start 65f287790a753cf9866c27a91fe39e0627cb8f07dd9c95dd16559ba687b82dfe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-8-deactivate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=squid-f72fecf, io.buildah.version=1.33.5, org.label-schema.build-date=20240507) 2024-05-12T22:21:51.679 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 12 22:21:51 smithi186 podman[72175]: 2024-05-12 22:21:51.361146796 +0000 UTC m=+0.393170462 container attach 65f287790a753cf9866c27a91fe39e0627cb8f07dd9c95dd16559ba687b82dfe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-8-deactivate, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=squid-f72fecf, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240507, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e) 2024-05-12T22:21:51.680 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 12 22:21:51 smithi186 podman[72175]: 2024-05-12 22:21:51.583359252 +0000 UTC m=+0.615382895 container died 65f287790a753cf9866c27a91fe39e0627cb8f07dd9c95dd16559ba687b82dfe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-8-deactivate, org.label-schema.build-date=20240507, RELEASE=squid-f72fecf, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True) 2024-05-12T22:21:51.680 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:51 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:51.470+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.561281+0000 front 2024-05-12T22:20:41.561023+0000 (oldest deadline 2024-05-12T22:21:06.261036+0000) 2024-05-12T22:21:51.680 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:51 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:51.470+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:50.163666+0000 front 2024-05-12T22:20:50.163649+0000 (oldest deadline 2024-05-12T22:21:13.063447+0000) 2024-05-12T22:21:51.680 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:51 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:51.470+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:59.365658+0000 front 2024-05-12T22:20:59.365719+0000 (oldest deadline 2024-05-12T22:21:23.465741+0000) 2024-05-12T22:21:51.680 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:51 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:51.470+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-05-12T22:21:14.568000+0000 front 2024-05-12T22:21:14.568013+0000 (oldest deadline 2024-05-12T22:21:38.067911+0000) 2024-05-12T22:21:51.681 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:51 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:51.470+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-05-12T22:21:23.968898+0000 front 2024-05-12T22:21:23.968952+0000 (oldest deadline 2024-05-12T22:21:46.868939+0000) 2024-05-12T22:21:52.085 DEBUG:teuthology.orchestra.run.smithi186:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968@osd.8.service' 2024-05-12T22:21:52.108 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 12 22:21:52 smithi186 podman[72201]: 2024-05-12 22:21:52.064830069 +0000 UTC m=+0.474730421 container remove 65f287790a753cf9866c27a91fe39e0627cb8f07dd9c95dd16559ba687b82dfe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-8-deactivate, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, GIT_CLEAN=True, org.label-schema.build-date=20240507, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-f72fecf, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True) 2024-05-12T22:21:52.109 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 12 22:21:52 smithi186 systemd[1]: ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968@osd.8.service: Deactivated successfully. 2024-05-12T22:21:52.109 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 12 22:21:52 smithi186 systemd[1]: Stopped Ceph osd.8 for 6b92e7b4-10aa-11ef-bc98-c7b262605968. 2024-05-12T22:21:52.109 INFO:journalctl@ceph.osd.8.smithi186.stdout:May 12 22:21:52 smithi186 systemd[1]: ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968@osd.8.service: Consumed 14.686s CPU time. 2024-05-12T22:21:52.126 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-12T22:21:52.126 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-05-12T22:21:52.127 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-05-12T22:21:52.127 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968@osd.9 2024-05-12T22:21:52.209 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:51 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:51.796+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.490538+0000 front 2024-05-12T22:20:41.490674+0000 (oldest deadline 2024-05-12T22:21:04.990968+0000) 2024-05-12T22:21:52.209 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:51 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:51.796+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:48.493555+0000 front 2024-05-12T22:20:48.493616+0000 (oldest deadline 2024-05-12T22:21:14.393780+0000) 2024-05-12T22:21:52.209 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:51 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:51.796+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:56.095925+0000 front 2024-05-12T22:20:56.096412+0000 (oldest deadline 2024-05-12T22:21:21.396155+0000) 2024-05-12T22:21:52.209 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:51 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:51.796+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-12T22:21:08.399553+0000 front 2024-05-12T22:21:08.399957+0000 (oldest deadline 2024-05-12T22:21:30.099451+0000) 2024-05-12T22:21:52.209 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:51 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:51.796+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-05-12T22:21:16.001427+0000 front 2024-05-12T22:21:16.001354+0000 (oldest deadline 2024-05-12T22:21:40.701607+0000) 2024-05-12T22:21:52.242 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:51 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:51.916+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:43.519531+0000 front 2024-05-12T22:20:43.519638+0000 (oldest deadline 2024-05-12T22:21:05.219288+0000) 2024-05-12T22:21:52.242 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:51 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:51.916+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:51.620499+0000 front 2024-05-12T22:20:51.620463+0000 (oldest deadline 2024-05-12T22:21:16.221349+0000) 2024-05-12T22:21:52.242 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:51 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:51.916+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:59.722447+0000 front 2024-05-12T22:20:59.722655+0000 (oldest deadline 2024-05-12T22:21:21.422316+0000) 2024-05-12T22:21:52.242 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:51 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:51.916+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-12T22:21:05.224800+0000 front 2024-05-12T22:21:05.225010+0000 (oldest deadline 2024-05-12T22:21:33.425092+0000) 2024-05-12T22:21:52.242 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:51 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:51.916+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-05-12T22:21:16.826477+0000 front 2024-05-12T22:21:16.826438+0000 (oldest deadline 2024-05-12T22:21:37.926536+0000) 2024-05-12T22:21:52.242 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:51 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:51.916+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-05-12T22:21:24.828681+0000 front 2024-05-12T22:21:24.828551+0000 (oldest deadline 2024-05-12T22:21:49.428752+0000) 2024-05-12T22:21:52.534 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:52 smithi028 systemd[1]: Stopping Ceph osd.9 for 6b92e7b4-10aa-11ef-bc98-c7b262605968... 2024-05-12T22:21:52.929 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:52 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:52.436+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.561281+0000 front 2024-05-12T22:20:41.561023+0000 (oldest deadline 2024-05-12T22:21:06.261036+0000) 2024-05-12T22:21:52.929 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:52 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:52.437+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:50.163666+0000 front 2024-05-12T22:20:50.163649+0000 (oldest deadline 2024-05-12T22:21:13.063447+0000) 2024-05-12T22:21:52.929 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:52 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:52.437+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:59.365658+0000 front 2024-05-12T22:20:59.365719+0000 (oldest deadline 2024-05-12T22:21:23.465741+0000) 2024-05-12T22:21:52.930 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:52 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:52.437+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-05-12T22:21:14.568000+0000 front 2024-05-12T22:21:14.568013+0000 (oldest deadline 2024-05-12T22:21:38.067911+0000) 2024-05-12T22:21:52.930 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:52 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:52.437+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-05-12T22:21:23.968898+0000 front 2024-05-12T22:21:23.968952+0000 (oldest deadline 2024-05-12T22:21:46.868939+0000) 2024-05-12T22:21:53.036 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:52 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:52.707+0000 7f087aa00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-12T22:21:53.036 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:52 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:52.707+0000 7f087aa00700 -1 osd.9 102 *** Got signal Terminated *** 2024-05-12T22:21:53.036 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:52 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:52.707+0000 7f087aa00700 -1 osd.9 102 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-12T22:21:53.036 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:52 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:52.792+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.490538+0000 front 2024-05-12T22:20:41.490674+0000 (oldest deadline 2024-05-12T22:21:04.990968+0000) 2024-05-12T22:21:53.036 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:52 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:52.792+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:48.493555+0000 front 2024-05-12T22:20:48.493616+0000 (oldest deadline 2024-05-12T22:21:14.393780+0000) 2024-05-12T22:21:53.036 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:52 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:52.792+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:56.095925+0000 front 2024-05-12T22:20:56.096412+0000 (oldest deadline 2024-05-12T22:21:21.396155+0000) 2024-05-12T22:21:53.037 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:52 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:52.792+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-12T22:21:08.399553+0000 front 2024-05-12T22:21:08.399957+0000 (oldest deadline 2024-05-12T22:21:30.099451+0000) 2024-05-12T22:21:53.037 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:52 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:52.792+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-05-12T22:21:16.001427+0000 front 2024-05-12T22:21:16.001354+0000 (oldest deadline 2024-05-12T22:21:40.701607+0000) 2024-05-12T22:21:53.242 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:52 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:52.911+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:43.519531+0000 front 2024-05-12T22:20:43.519638+0000 (oldest deadline 2024-05-12T22:21:05.219288+0000) 2024-05-12T22:21:53.242 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:52 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:52.911+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:51.620499+0000 front 2024-05-12T22:20:51.620463+0000 (oldest deadline 2024-05-12T22:21:16.221349+0000) 2024-05-12T22:21:53.242 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:52 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:52.911+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:59.722447+0000 front 2024-05-12T22:20:59.722655+0000 (oldest deadline 2024-05-12T22:21:21.422316+0000) 2024-05-12T22:21:53.242 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:52 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:52.911+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-12T22:21:05.224800+0000 front 2024-05-12T22:21:05.225010+0000 (oldest deadline 2024-05-12T22:21:33.425092+0000) 2024-05-12T22:21:53.243 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:52 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:52.911+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-05-12T22:21:16.826477+0000 front 2024-05-12T22:21:16.826438+0000 (oldest deadline 2024-05-12T22:21:37.926536+0000) 2024-05-12T22:21:53.243 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:52 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:52.911+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-05-12T22:21:24.828681+0000 front 2024-05-12T22:21:24.828551+0000 (oldest deadline 2024-05-12T22:21:49.428752+0000) 2024-05-12T22:21:53.680 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:53 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:53.456+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.561281+0000 front 2024-05-12T22:20:41.561023+0000 (oldest deadline 2024-05-12T22:21:06.261036+0000) 2024-05-12T22:21:53.680 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:53 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:53.456+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:50.163666+0000 front 2024-05-12T22:20:50.163649+0000 (oldest deadline 2024-05-12T22:21:13.063447+0000) 2024-05-12T22:21:53.680 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:53 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:53.456+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:59.365658+0000 front 2024-05-12T22:20:59.365719+0000 (oldest deadline 2024-05-12T22:21:23.465741+0000) 2024-05-12T22:21:53.680 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:53 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:53.456+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-05-12T22:21:14.568000+0000 front 2024-05-12T22:21:14.568013+0000 (oldest deadline 2024-05-12T22:21:38.067911+0000) 2024-05-12T22:21:53.680 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:53 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:53.456+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-05-12T22:21:23.968898+0000 front 2024-05-12T22:21:23.968952+0000 (oldest deadline 2024-05-12T22:21:46.868939+0000) 2024-05-12T22:21:54.035 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:53 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:53.784+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.490538+0000 front 2024-05-12T22:20:41.490674+0000 (oldest deadline 2024-05-12T22:21:04.990968+0000) 2024-05-12T22:21:54.035 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:53 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:53.784+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:48.493555+0000 front 2024-05-12T22:20:48.493616+0000 (oldest deadline 2024-05-12T22:21:14.393780+0000) 2024-05-12T22:21:54.036 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:53 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:53.784+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:56.095925+0000 front 2024-05-12T22:20:56.096412+0000 (oldest deadline 2024-05-12T22:21:21.396155+0000) 2024-05-12T22:21:54.036 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:53 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:53.784+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-12T22:21:08.399553+0000 front 2024-05-12T22:21:08.399957+0000 (oldest deadline 2024-05-12T22:21:30.099451+0000) 2024-05-12T22:21:54.036 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:53 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:53.784+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-05-12T22:21:16.001427+0000 front 2024-05-12T22:21:16.001354+0000 (oldest deadline 2024-05-12T22:21:40.701607+0000) 2024-05-12T22:21:54.242 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:53 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:53.939+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:43.519531+0000 front 2024-05-12T22:20:43.519638+0000 (oldest deadline 2024-05-12T22:21:05.219288+0000) 2024-05-12T22:21:54.242 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:53 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:53.939+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:51.620499+0000 front 2024-05-12T22:20:51.620463+0000 (oldest deadline 2024-05-12T22:21:16.221349+0000) 2024-05-12T22:21:54.242 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:53 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:53.939+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:59.722447+0000 front 2024-05-12T22:20:59.722655+0000 (oldest deadline 2024-05-12T22:21:21.422316+0000) 2024-05-12T22:21:54.242 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:53 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:53.939+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-12T22:21:05.224800+0000 front 2024-05-12T22:21:05.225010+0000 (oldest deadline 2024-05-12T22:21:33.425092+0000) 2024-05-12T22:21:54.242 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:53 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:53.939+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-05-12T22:21:16.826477+0000 front 2024-05-12T22:21:16.826438+0000 (oldest deadline 2024-05-12T22:21:37.926536+0000) 2024-05-12T22:21:54.243 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:53 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:53.939+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-05-12T22:21:24.828681+0000 front 2024-05-12T22:21:24.828551+0000 (oldest deadline 2024-05-12T22:21:49.428752+0000) 2024-05-12T22:21:54.929 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:54 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:54.464+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.561281+0000 front 2024-05-12T22:20:41.561023+0000 (oldest deadline 2024-05-12T22:21:06.261036+0000) 2024-05-12T22:21:54.929 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:54 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:54.464+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:50.163666+0000 front 2024-05-12T22:20:50.163649+0000 (oldest deadline 2024-05-12T22:21:13.063447+0000) 2024-05-12T22:21:54.929 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:54 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:54.464+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:59.365658+0000 front 2024-05-12T22:20:59.365719+0000 (oldest deadline 2024-05-12T22:21:23.465741+0000) 2024-05-12T22:21:54.930 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:54 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:54.464+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-05-12T22:21:14.568000+0000 front 2024-05-12T22:21:14.568013+0000 (oldest deadline 2024-05-12T22:21:38.067911+0000) 2024-05-12T22:21:54.930 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:54 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:54.464+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-05-12T22:21:23.968898+0000 front 2024-05-12T22:21:23.968952+0000 (oldest deadline 2024-05-12T22:21:46.868939+0000) 2024-05-12T22:21:55.035 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:54 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:54.761+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.490538+0000 front 2024-05-12T22:20:41.490674+0000 (oldest deadline 2024-05-12T22:21:04.990968+0000) 2024-05-12T22:21:55.035 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:54 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:54.761+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:48.493555+0000 front 2024-05-12T22:20:48.493616+0000 (oldest deadline 2024-05-12T22:21:14.393780+0000) 2024-05-12T22:21:55.036 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:54 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:54.761+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:56.095925+0000 front 2024-05-12T22:20:56.096412+0000 (oldest deadline 2024-05-12T22:21:21.396155+0000) 2024-05-12T22:21:55.036 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:54 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:54.761+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-12T22:21:08.399553+0000 front 2024-05-12T22:21:08.399957+0000 (oldest deadline 2024-05-12T22:21:30.099451+0000) 2024-05-12T22:21:55.036 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:54 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:54.761+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-05-12T22:21:16.001427+0000 front 2024-05-12T22:21:16.001354+0000 (oldest deadline 2024-05-12T22:21:40.701607+0000) 2024-05-12T22:21:55.036 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:54 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:54.761+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-05-12T22:21:32.405674+0000 front 2024-05-12T22:21:32.405720+0000 (oldest deadline 2024-05-12T22:21:54.105397+0000) 2024-05-12T22:21:55.242 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:54 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:54.958+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:43.519531+0000 front 2024-05-12T22:20:43.519638+0000 (oldest deadline 2024-05-12T22:21:05.219288+0000) 2024-05-12T22:21:55.242 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:54 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:54.958+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:51.620499+0000 front 2024-05-12T22:20:51.620463+0000 (oldest deadline 2024-05-12T22:21:16.221349+0000) 2024-05-12T22:21:55.242 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:54 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:54.958+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:59.722447+0000 front 2024-05-12T22:20:59.722655+0000 (oldest deadline 2024-05-12T22:21:21.422316+0000) 2024-05-12T22:21:55.243 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:54 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:54.958+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-12T22:21:05.224800+0000 front 2024-05-12T22:21:05.225010+0000 (oldest deadline 2024-05-12T22:21:33.425092+0000) 2024-05-12T22:21:55.243 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:54 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:54.958+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-05-12T22:21:16.826477+0000 front 2024-05-12T22:21:16.826438+0000 (oldest deadline 2024-05-12T22:21:37.926536+0000) 2024-05-12T22:21:55.243 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:54 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:54.958+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-05-12T22:21:24.828681+0000 front 2024-05-12T22:21:24.828551+0000 (oldest deadline 2024-05-12T22:21:49.428752+0000) 2024-05-12T22:21:55.929 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:55 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:55.494+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.561281+0000 front 2024-05-12T22:20:41.561023+0000 (oldest deadline 2024-05-12T22:21:06.261036+0000) 2024-05-12T22:21:55.929 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:55 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:55.494+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:50.163666+0000 front 2024-05-12T22:20:50.163649+0000 (oldest deadline 2024-05-12T22:21:13.063447+0000) 2024-05-12T22:21:55.930 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:55 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:55.494+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:59.365658+0000 front 2024-05-12T22:20:59.365719+0000 (oldest deadline 2024-05-12T22:21:23.465741+0000) 2024-05-12T22:21:55.930 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:55 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:55.494+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-05-12T22:21:14.568000+0000 front 2024-05-12T22:21:14.568013+0000 (oldest deadline 2024-05-12T22:21:38.067911+0000) 2024-05-12T22:21:55.930 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:55 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:55.494+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-05-12T22:21:23.968898+0000 front 2024-05-12T22:21:23.968952+0000 (oldest deadline 2024-05-12T22:21:46.868939+0000) 2024-05-12T22:21:55.992 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:55 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:55.930+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:43.519531+0000 front 2024-05-12T22:20:43.519638+0000 (oldest deadline 2024-05-12T22:21:05.219288+0000) 2024-05-12T22:21:55.992 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:55 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:55.930+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:51.620499+0000 front 2024-05-12T22:20:51.620463+0000 (oldest deadline 2024-05-12T22:21:16.221349+0000) 2024-05-12T22:21:55.992 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:55 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:55.930+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:59.722447+0000 front 2024-05-12T22:20:59.722655+0000 (oldest deadline 2024-05-12T22:21:21.422316+0000) 2024-05-12T22:21:55.993 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:55 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:55.930+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-12T22:21:05.224800+0000 front 2024-05-12T22:21:05.225010+0000 (oldest deadline 2024-05-12T22:21:33.425092+0000) 2024-05-12T22:21:55.993 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:55 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:55.930+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-05-12T22:21:16.826477+0000 front 2024-05-12T22:21:16.826438+0000 (oldest deadline 2024-05-12T22:21:37.926536+0000) 2024-05-12T22:21:55.993 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:55 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:55.930+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-05-12T22:21:24.828681+0000 front 2024-05-12T22:21:24.828551+0000 (oldest deadline 2024-05-12T22:21:49.428752+0000) 2024-05-12T22:21:55.993 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:55 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:55.930+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-05-12T22:21:29.429618+0000 front 2024-05-12T22:21:29.429553+0000 (oldest deadline 2024-05-12T22:21:55.229662+0000) 2024-05-12T22:21:56.035 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:55 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:55.717+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.490538+0000 front 2024-05-12T22:20:41.490674+0000 (oldest deadline 2024-05-12T22:21:04.990968+0000) 2024-05-12T22:21:56.036 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:55 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:55.717+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:48.493555+0000 front 2024-05-12T22:20:48.493616+0000 (oldest deadline 2024-05-12T22:21:14.393780+0000) 2024-05-12T22:21:56.036 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:55 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:55.717+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:56.095925+0000 front 2024-05-12T22:20:56.096412+0000 (oldest deadline 2024-05-12T22:21:21.396155+0000) 2024-05-12T22:21:56.036 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:55 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:55.717+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-12T22:21:08.399553+0000 front 2024-05-12T22:21:08.399957+0000 (oldest deadline 2024-05-12T22:21:30.099451+0000) 2024-05-12T22:21:56.036 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:55 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:55.717+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-05-12T22:21:16.001427+0000 front 2024-05-12T22:21:16.001354+0000 (oldest deadline 2024-05-12T22:21:40.701607+0000) 2024-05-12T22:21:56.036 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:55 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:55.717+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-05-12T22:21:32.405674+0000 front 2024-05-12T22:21:32.405720+0000 (oldest deadline 2024-05-12T22:21:54.105397+0000) 2024-05-12T22:21:56.929 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:56 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:56.508+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.561281+0000 front 2024-05-12T22:20:41.561023+0000 (oldest deadline 2024-05-12T22:21:06.261036+0000) 2024-05-12T22:21:56.929 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:56 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:56.508+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:50.163666+0000 front 2024-05-12T22:20:50.163649+0000 (oldest deadline 2024-05-12T22:21:13.063447+0000) 2024-05-12T22:21:56.929 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:56 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:56.508+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:59.365658+0000 front 2024-05-12T22:20:59.365719+0000 (oldest deadline 2024-05-12T22:21:23.465741+0000) 2024-05-12T22:21:56.930 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:56 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:56.508+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-05-12T22:21:14.568000+0000 front 2024-05-12T22:21:14.568013+0000 (oldest deadline 2024-05-12T22:21:38.067911+0000) 2024-05-12T22:21:56.930 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:56 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:56.508+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-05-12T22:21:23.968898+0000 front 2024-05-12T22:21:23.968952+0000 (oldest deadline 2024-05-12T22:21:46.868939+0000) 2024-05-12T22:21:56.930 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:56 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:56.508+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-05-12T22:21:32.170175+0000 front 2024-05-12T22:21:32.170133+0000 (oldest deadline 2024-05-12T22:21:55.669945+0000) 2024-05-12T22:21:57.035 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:56 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:56.693+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.490538+0000 front 2024-05-12T22:20:41.490674+0000 (oldest deadline 2024-05-12T22:21:04.990968+0000) 2024-05-12T22:21:57.035 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:56 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:56.693+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:48.493555+0000 front 2024-05-12T22:20:48.493616+0000 (oldest deadline 2024-05-12T22:21:14.393780+0000) 2024-05-12T22:21:57.036 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:56 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:56.693+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:56.095925+0000 front 2024-05-12T22:20:56.096412+0000 (oldest deadline 2024-05-12T22:21:21.396155+0000) 2024-05-12T22:21:57.036 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:56 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:56.693+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-12T22:21:08.399553+0000 front 2024-05-12T22:21:08.399957+0000 (oldest deadline 2024-05-12T22:21:30.099451+0000) 2024-05-12T22:21:57.036 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:56 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:56.693+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-05-12T22:21:16.001427+0000 front 2024-05-12T22:21:16.001354+0000 (oldest deadline 2024-05-12T22:21:40.701607+0000) 2024-05-12T22:21:57.036 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:56 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:56.693+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-05-12T22:21:32.405674+0000 front 2024-05-12T22:21:32.405720+0000 (oldest deadline 2024-05-12T22:21:54.105397+0000) 2024-05-12T22:21:57.242 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:56 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:56.948+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:43.519531+0000 front 2024-05-12T22:20:43.519638+0000 (oldest deadline 2024-05-12T22:21:05.219288+0000) 2024-05-12T22:21:57.242 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:56 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:56.948+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:51.620499+0000 front 2024-05-12T22:20:51.620463+0000 (oldest deadline 2024-05-12T22:21:16.221349+0000) 2024-05-12T22:21:57.242 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:56 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:56.948+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:59.722447+0000 front 2024-05-12T22:20:59.722655+0000 (oldest deadline 2024-05-12T22:21:21.422316+0000) 2024-05-12T22:21:57.242 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:56 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:56.948+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-12T22:21:05.224800+0000 front 2024-05-12T22:21:05.225010+0000 (oldest deadline 2024-05-12T22:21:33.425092+0000) 2024-05-12T22:21:57.242 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:56 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:56.948+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-05-12T22:21:16.826477+0000 front 2024-05-12T22:21:16.826438+0000 (oldest deadline 2024-05-12T22:21:37.926536+0000) 2024-05-12T22:21:57.243 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:56 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:56.948+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-05-12T22:21:24.828681+0000 front 2024-05-12T22:21:24.828551+0000 (oldest deadline 2024-05-12T22:21:49.428752+0000) 2024-05-12T22:21:57.243 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:56 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:56.948+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-05-12T22:21:29.429618+0000 front 2024-05-12T22:21:29.429553+0000 (oldest deadline 2024-05-12T22:21:55.229662+0000) 2024-05-12T22:21:57.929 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:57 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:57.473+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.561281+0000 front 2024-05-12T22:20:41.561023+0000 (oldest deadline 2024-05-12T22:21:06.261036+0000) 2024-05-12T22:21:57.930 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:57 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:57.473+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:50.163666+0000 front 2024-05-12T22:20:50.163649+0000 (oldest deadline 2024-05-12T22:21:13.063447+0000) 2024-05-12T22:21:57.930 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:57 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:57.473+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:59.365658+0000 front 2024-05-12T22:20:59.365719+0000 (oldest deadline 2024-05-12T22:21:23.465741+0000) 2024-05-12T22:21:57.930 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:57 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:57.473+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-05-12T22:21:14.568000+0000 front 2024-05-12T22:21:14.568013+0000 (oldest deadline 2024-05-12T22:21:38.067911+0000) 2024-05-12T22:21:57.930 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:57 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:57.473+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-05-12T22:21:23.968898+0000 front 2024-05-12T22:21:23.968952+0000 (oldest deadline 2024-05-12T22:21:46.868939+0000) 2024-05-12T22:21:57.930 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:57 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:57.473+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-05-12T22:21:32.170175+0000 front 2024-05-12T22:21:32.170133+0000 (oldest deadline 2024-05-12T22:21:55.669945+0000) 2024-05-12T22:21:58.035 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:57 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:57.649+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.490538+0000 front 2024-05-12T22:20:41.490674+0000 (oldest deadline 2024-05-12T22:21:04.990968+0000) 2024-05-12T22:21:58.035 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:57 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:57.649+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:48.493555+0000 front 2024-05-12T22:20:48.493616+0000 (oldest deadline 2024-05-12T22:21:14.393780+0000) 2024-05-12T22:21:58.035 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:57 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:57.649+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:56.095925+0000 front 2024-05-12T22:20:56.096412+0000 (oldest deadline 2024-05-12T22:21:21.396155+0000) 2024-05-12T22:21:58.035 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:57 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:57.649+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-12T22:21:08.399553+0000 front 2024-05-12T22:21:08.399957+0000 (oldest deadline 2024-05-12T22:21:30.099451+0000) 2024-05-12T22:21:58.036 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:57 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:57.649+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-05-12T22:21:16.001427+0000 front 2024-05-12T22:21:16.001354+0000 (oldest deadline 2024-05-12T22:21:40.701607+0000) 2024-05-12T22:21:58.036 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:57 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9[61665]: 2024-05-12T22:21:57.649+0000 7f0876c00700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-05-12T22:21:32.405674+0000 front 2024-05-12T22:21:32.405720+0000 (oldest deadline 2024-05-12T22:21:54.105397+0000) 2024-05-12T22:21:58.242 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:57 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:57.970+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:43.519531+0000 front 2024-05-12T22:20:43.519638+0000 (oldest deadline 2024-05-12T22:21:05.219288+0000) 2024-05-12T22:21:58.242 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:57 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:57.970+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:51.620499+0000 front 2024-05-12T22:20:51.620463+0000 (oldest deadline 2024-05-12T22:21:16.221349+0000) 2024-05-12T22:21:58.242 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:57 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:57.970+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:59.722447+0000 front 2024-05-12T22:20:59.722655+0000 (oldest deadline 2024-05-12T22:21:21.422316+0000) 2024-05-12T22:21:58.242 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:57 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:57.970+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-12T22:21:05.224800+0000 front 2024-05-12T22:21:05.225010+0000 (oldest deadline 2024-05-12T22:21:33.425092+0000) 2024-05-12T22:21:58.243 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:57 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:57.970+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-05-12T22:21:16.826477+0000 front 2024-05-12T22:21:16.826438+0000 (oldest deadline 2024-05-12T22:21:37.926536+0000) 2024-05-12T22:21:58.243 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:57 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:57.970+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-05-12T22:21:24.828681+0000 front 2024-05-12T22:21:24.828551+0000 (oldest deadline 2024-05-12T22:21:49.428752+0000) 2024-05-12T22:21:58.243 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:57 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:57.970+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-05-12T22:21:29.429618+0000 front 2024-05-12T22:21:29.429553+0000 (oldest deadline 2024-05-12T22:21:55.229662+0000) 2024-05-12T22:21:58.462 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:58 smithi028 podman[100169]: 2024-05-12 22:21:58.186792455 +0000 UTC m=+5.614992558 container died d4a26aca41c71a98f16a311f39bc6b221790ea72348b93903663bab4bec7418c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240507, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, RELEASE=squid-f72fecf, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-05-12T22:21:58.785 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:58 smithi028 podman[100169]: 2024-05-12 22:21:58.617533364 +0000 UTC m=+6.045733524 container cleanup d4a26aca41c71a98f16a311f39bc6b221790ea72348b93903663bab4bec7418c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240507, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=squid-f72fecf, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-12T22:21:58.785 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:58 smithi028 bash[100169]: ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9 2024-05-12T22:21:58.929 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:58 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:58.441+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.561281+0000 front 2024-05-12T22:20:41.561023+0000 (oldest deadline 2024-05-12T22:21:06.261036+0000) 2024-05-12T22:21:58.929 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:58 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:58.441+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:50.163666+0000 front 2024-05-12T22:20:50.163649+0000 (oldest deadline 2024-05-12T22:21:13.063447+0000) 2024-05-12T22:21:58.929 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:58 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:58.441+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:59.365658+0000 front 2024-05-12T22:20:59.365719+0000 (oldest deadline 2024-05-12T22:21:23.465741+0000) 2024-05-12T22:21:58.929 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:58 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:58.441+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-05-12T22:21:14.568000+0000 front 2024-05-12T22:21:14.568013+0000 (oldest deadline 2024-05-12T22:21:38.067911+0000) 2024-05-12T22:21:58.930 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:58 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:58.441+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-05-12T22:21:23.968898+0000 front 2024-05-12T22:21:23.968952+0000 (oldest deadline 2024-05-12T22:21:46.868939+0000) 2024-05-12T22:21:58.930 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:58 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:58.441+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-05-12T22:21:32.170175+0000 front 2024-05-12T22:21:32.170133+0000 (oldest deadline 2024-05-12T22:21:55.669945+0000) 2024-05-12T22:21:59.242 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:58 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:58.930+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:43.519531+0000 front 2024-05-12T22:20:43.519638+0000 (oldest deadline 2024-05-12T22:21:05.219288+0000) 2024-05-12T22:21:59.242 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:58 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:58.930+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:51.620499+0000 front 2024-05-12T22:20:51.620463+0000 (oldest deadline 2024-05-12T22:21:16.221349+0000) 2024-05-12T22:21:59.242 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:58 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:58.930+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:59.722447+0000 front 2024-05-12T22:20:59.722655+0000 (oldest deadline 2024-05-12T22:21:21.422316+0000) 2024-05-12T22:21:59.243 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:58 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:58.930+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-12T22:21:05.224800+0000 front 2024-05-12T22:21:05.225010+0000 (oldest deadline 2024-05-12T22:21:33.425092+0000) 2024-05-12T22:21:59.243 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:58 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:58.930+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-05-12T22:21:16.826477+0000 front 2024-05-12T22:21:16.826438+0000 (oldest deadline 2024-05-12T22:21:37.926536+0000) 2024-05-12T22:21:59.243 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:58 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:58.930+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-05-12T22:21:24.828681+0000 front 2024-05-12T22:21:24.828551+0000 (oldest deadline 2024-05-12T22:21:49.428752+0000) 2024-05-12T22:21:59.243 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:58 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:58.930+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-05-12T22:21:29.429618+0000 front 2024-05-12T22:21:29.429553+0000 (oldest deadline 2024-05-12T22:21:55.229662+0000) 2024-05-12T22:21:59.285 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:58 smithi028 podman[100187]: 2024-05-12 22:21:58.825133637 +0000 UTC m=+0.617985041 container remove d4a26aca41c71a98f16a311f39bc6b221790ea72348b93903663bab4bec7418c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9, org.label-schema.build-date=20240507, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, RELEASE=squid-f72fecf, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True) 2024-05-12T22:21:59.679 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:59 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:59.403+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.561281+0000 front 2024-05-12T22:20:41.561023+0000 (oldest deadline 2024-05-12T22:21:06.261036+0000) 2024-05-12T22:21:59.679 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:59 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:59.403+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:50.163666+0000 front 2024-05-12T22:20:50.163649+0000 (oldest deadline 2024-05-12T22:21:13.063447+0000) 2024-05-12T22:21:59.680 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:59 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:59.403+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:59.365658+0000 front 2024-05-12T22:20:59.365719+0000 (oldest deadline 2024-05-12T22:21:23.465741+0000) 2024-05-12T22:21:59.680 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:59 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:59.403+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-05-12T22:21:14.568000+0000 front 2024-05-12T22:21:14.568013+0000 (oldest deadline 2024-05-12T22:21:38.067911+0000) 2024-05-12T22:21:59.680 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:59 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:59.403+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-05-12T22:21:23.968898+0000 front 2024-05-12T22:21:23.968952+0000 (oldest deadline 2024-05-12T22:21:46.868939+0000) 2024-05-12T22:21:59.680 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:21:59 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:21:59.403+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-05-12T22:21:32.170175+0000 front 2024-05-12T22:21:32.170133+0000 (oldest deadline 2024-05-12T22:21:55.669945+0000) 2024-05-12T22:21:59.980 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:59 smithi028 podman[100276]: 2024-05-12 22:21:59.567717691 +0000 UTC m=+0.041118565 image pull 8252b7944d3b5c188c0ff632647ba6cc4e9a19588ee097b8a3dba27b42779c25 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc 2024-05-12T22:21:59.980 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:59 smithi028 podman[100276]: 2024-05-12 22:21:59.717699113 +0000 UTC m=+0.191099993 container create b9152030cb6e0f6e24f13db22152a71986719d3212b2bb8a1bb11a89da4286a8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9-deactivate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240507, GIT_CLEAN=True, RELEASE=squid-f72fecf) 2024-05-12T22:21:59.980 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:59 smithi028 podman[100276]: 2024-05-12 22:21:59.979789153 +0000 UTC m=+0.453190078 container init b9152030cb6e0f6e24f13db22152a71986719d3212b2bb8a1bb11a89da4286a8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=squid-f72fecf, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240507, CEPH_POINT_RELEASE=, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e) 2024-05-12T22:22:00.242 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:59 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:59.925+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:43.519531+0000 front 2024-05-12T22:20:43.519638+0000 (oldest deadline 2024-05-12T22:21:05.219288+0000) 2024-05-12T22:22:00.242 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:59 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:59.925+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:51.620499+0000 front 2024-05-12T22:20:51.620463+0000 (oldest deadline 2024-05-12T22:21:16.221349+0000) 2024-05-12T22:22:00.242 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:59 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:59.925+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:59.722447+0000 front 2024-05-12T22:20:59.722655+0000 (oldest deadline 2024-05-12T22:21:21.422316+0000) 2024-05-12T22:22:00.243 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:59 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:59.926+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-12T22:21:05.224800+0000 front 2024-05-12T22:21:05.225010+0000 (oldest deadline 2024-05-12T22:21:33.425092+0000) 2024-05-12T22:22:00.243 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:59 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:59.926+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-05-12T22:21:16.826477+0000 front 2024-05-12T22:21:16.826438+0000 (oldest deadline 2024-05-12T22:21:37.926536+0000) 2024-05-12T22:22:00.243 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:59 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:59.926+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-05-12T22:21:24.828681+0000 front 2024-05-12T22:21:24.828551+0000 (oldest deadline 2024-05-12T22:21:49.428752+0000) 2024-05-12T22:22:00.243 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:21:59 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:21:59.926+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-05-12T22:21:29.429618+0000 front 2024-05-12T22:21:29.429553+0000 (oldest deadline 2024-05-12T22:21:55.229662+0000) 2024-05-12T22:22:00.285 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:21:59 smithi028 podman[100276]: 2024-05-12 22:21:59.991109924 +0000 UTC m=+0.464510800 container start b9152030cb6e0f6e24f13db22152a71986719d3212b2bb8a1bb11a89da4286a8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240507, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=squid-f72fecf, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-05-12T22:22:00.285 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:22:00 smithi028 podman[100276]: 2024-05-12 22:22:00.086987406 +0000 UTC m=+0.560388295 container attach b9152030cb6e0f6e24f13db22152a71986719d3212b2bb8a1bb11a89da4286a8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9-deactivate, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240507, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, RELEASE=squid-f72fecf, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-05-12T22:22:00.679 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:22:00 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:22:00.379+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.561281+0000 front 2024-05-12T22:20:41.561023+0000 (oldest deadline 2024-05-12T22:21:06.261036+0000) 2024-05-12T22:22:00.679 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:22:00 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:22:00.379+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:50.163666+0000 front 2024-05-12T22:20:50.163649+0000 (oldest deadline 2024-05-12T22:21:13.063447+0000) 2024-05-12T22:22:00.679 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:22:00 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:22:00.379+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:59.365658+0000 front 2024-05-12T22:20:59.365719+0000 (oldest deadline 2024-05-12T22:21:23.465741+0000) 2024-05-12T22:22:00.680 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:22:00 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:22:00.379+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-05-12T22:21:14.568000+0000 front 2024-05-12T22:21:14.568013+0000 (oldest deadline 2024-05-12T22:21:38.067911+0000) 2024-05-12T22:22:00.680 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:22:00 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:22:00.379+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-05-12T22:21:23.968898+0000 front 2024-05-12T22:21:23.968952+0000 (oldest deadline 2024-05-12T22:21:46.868939+0000) 2024-05-12T22:22:00.680 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:22:00 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:22:00.379+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-05-12T22:21:32.170175+0000 front 2024-05-12T22:21:32.170133+0000 (oldest deadline 2024-05-12T22:21:55.669945+0000) 2024-05-12T22:22:00.728 INFO:journalctl@ceph.osd.9.smithi028.stdout:May 12 22:22:00 smithi028 podman[100276]: 2024-05-12 22:22:00.463151638 +0000 UTC m=+0.936552527 container died b9152030cb6e0f6e24f13db22152a71986719d3212b2bb8a1bb11a89da4286a8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-9-deactivate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240507, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, RELEASE=squid-f72fecf, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-05-12T22:22:01.071 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968@osd.9.service' 2024-05-12T22:22:01.171 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-12T22:22:01.172 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-05-12T22:22:01.172 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-05-12T22:22:01.172 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl stop ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968@osd.10 2024-05-12T22:22:01.199 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:22:00 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:22:00.945+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:43.519531+0000 front 2024-05-12T22:20:43.519638+0000 (oldest deadline 2024-05-12T22:21:05.219288+0000) 2024-05-12T22:22:01.199 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:22:00 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:22:00.945+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:51.620499+0000 front 2024-05-12T22:20:51.620463+0000 (oldest deadline 2024-05-12T22:21:16.221349+0000) 2024-05-12T22:22:01.199 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:22:00 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:22:00.945+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:59.722447+0000 front 2024-05-12T22:20:59.722655+0000 (oldest deadline 2024-05-12T22:21:21.422316+0000) 2024-05-12T22:22:01.199 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:22:00 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:22:00.945+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-12T22:21:05.224800+0000 front 2024-05-12T22:21:05.225010+0000 (oldest deadline 2024-05-12T22:21:33.425092+0000) 2024-05-12T22:22:01.199 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:22:00 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:22:00.945+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-05-12T22:21:16.826477+0000 front 2024-05-12T22:21:16.826438+0000 (oldest deadline 2024-05-12T22:21:37.926536+0000) 2024-05-12T22:22:01.199 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:22:00 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:22:00.945+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-05-12T22:21:24.828681+0000 front 2024-05-12T22:21:24.828551+0000 (oldest deadline 2024-05-12T22:21:49.428752+0000) 2024-05-12T22:22:01.200 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:22:00 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:22:00.945+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-05-12T22:21:29.429618+0000 front 2024-05-12T22:21:29.429553+0000 (oldest deadline 2024-05-12T22:21:55.229662+0000) 2024-05-12T22:22:01.492 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:22:01 smithi086 systemd[1]: Stopping Ceph osd.10 for 6b92e7b4-10aa-11ef-bc98-c7b262605968... 2024-05-12T22:22:01.492 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:22:01 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:22:01.446+0000 7f113e8df700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-12T22:22:01.492 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:22:01 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:22:01.446+0000 7f113e8df700 -1 osd.10 102 *** Got signal Terminated *** 2024-05-12T22:22:01.493 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:22:01 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:22:01.446+0000 7f113e8df700 -1 osd.10 102 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-12T22:22:01.679 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:22:01 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:22:01.336+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.561281+0000 front 2024-05-12T22:20:41.561023+0000 (oldest deadline 2024-05-12T22:21:06.261036+0000) 2024-05-12T22:22:01.679 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:22:01 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:22:01.336+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:50.163666+0000 front 2024-05-12T22:20:50.163649+0000 (oldest deadline 2024-05-12T22:21:13.063447+0000) 2024-05-12T22:22:01.679 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:22:01 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:22:01.336+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:59.365658+0000 front 2024-05-12T22:20:59.365719+0000 (oldest deadline 2024-05-12T22:21:23.465741+0000) 2024-05-12T22:22:01.679 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:22:01 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:22:01.336+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-05-12T22:21:14.568000+0000 front 2024-05-12T22:21:14.568013+0000 (oldest deadline 2024-05-12T22:21:38.067911+0000) 2024-05-12T22:22:01.680 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:22:01 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:22:01.336+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-05-12T22:21:23.968898+0000 front 2024-05-12T22:21:23.968952+0000 (oldest deadline 2024-05-12T22:21:46.868939+0000) 2024-05-12T22:22:01.680 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:22:01 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:22:01.336+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-05-12T22:21:32.170175+0000 front 2024-05-12T22:21:32.170133+0000 (oldest deadline 2024-05-12T22:21:55.669945+0000) 2024-05-12T22:22:02.242 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:22:01 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:22:01.953+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:43.519531+0000 front 2024-05-12T22:20:43.519638+0000 (oldest deadline 2024-05-12T22:21:05.219288+0000) 2024-05-12T22:22:02.242 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:22:01 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:22:01.953+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:51.620499+0000 front 2024-05-12T22:20:51.620463+0000 (oldest deadline 2024-05-12T22:21:16.221349+0000) 2024-05-12T22:22:02.242 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:22:01 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:22:01.953+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:59.722447+0000 front 2024-05-12T22:20:59.722655+0000 (oldest deadline 2024-05-12T22:21:21.422316+0000) 2024-05-12T22:22:02.242 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:22:01 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:22:01.953+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-12T22:21:05.224800+0000 front 2024-05-12T22:21:05.225010+0000 (oldest deadline 2024-05-12T22:21:33.425092+0000) 2024-05-12T22:22:02.242 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:22:01 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:22:01.953+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-05-12T22:21:16.826477+0000 front 2024-05-12T22:21:16.826438+0000 (oldest deadline 2024-05-12T22:21:37.926536+0000) 2024-05-12T22:22:02.243 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:22:01 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:22:01.953+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-05-12T22:21:24.828681+0000 front 2024-05-12T22:21:24.828551+0000 (oldest deadline 2024-05-12T22:21:49.428752+0000) 2024-05-12T22:22:02.243 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:22:01 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:22:01.953+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-05-12T22:21:29.429618+0000 front 2024-05-12T22:21:29.429553+0000 (oldest deadline 2024-05-12T22:21:55.229662+0000) 2024-05-12T22:22:02.679 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:22:02 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:22:02.329+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.561281+0000 front 2024-05-12T22:20:41.561023+0000 (oldest deadline 2024-05-12T22:21:06.261036+0000) 2024-05-12T22:22:02.679 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:22:02 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:22:02.329+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:50.163666+0000 front 2024-05-12T22:20:50.163649+0000 (oldest deadline 2024-05-12T22:21:13.063447+0000) 2024-05-12T22:22:02.679 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:22:02 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:22:02.329+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:59.365658+0000 front 2024-05-12T22:20:59.365719+0000 (oldest deadline 2024-05-12T22:21:23.465741+0000) 2024-05-12T22:22:02.679 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:22:02 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:22:02.329+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-05-12T22:21:14.568000+0000 front 2024-05-12T22:21:14.568013+0000 (oldest deadline 2024-05-12T22:21:38.067911+0000) 2024-05-12T22:22:02.679 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:22:02 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:22:02.329+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-05-12T22:21:23.968898+0000 front 2024-05-12T22:21:23.968952+0000 (oldest deadline 2024-05-12T22:21:46.868939+0000) 2024-05-12T22:22:02.680 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:22:02 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:22:02.329+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-05-12T22:21:32.170175+0000 front 2024-05-12T22:21:32.170133+0000 (oldest deadline 2024-05-12T22:21:55.669945+0000) 2024-05-12T22:22:03.242 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:22:02 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:22:02.991+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:43.519531+0000 front 2024-05-12T22:20:43.519638+0000 (oldest deadline 2024-05-12T22:21:05.219288+0000) 2024-05-12T22:22:03.242 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:22:02 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:22:02.991+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:51.620499+0000 front 2024-05-12T22:20:51.620463+0000 (oldest deadline 2024-05-12T22:21:16.221349+0000) 2024-05-12T22:22:03.243 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:22:02 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:22:02.991+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:59.722447+0000 front 2024-05-12T22:20:59.722655+0000 (oldest deadline 2024-05-12T22:21:21.422316+0000) 2024-05-12T22:22:03.243 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:22:02 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:22:02.991+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-12T22:21:05.224800+0000 front 2024-05-12T22:21:05.225010+0000 (oldest deadline 2024-05-12T22:21:33.425092+0000) 2024-05-12T22:22:03.243 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:22:02 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:22:02.991+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-05-12T22:21:16.826477+0000 front 2024-05-12T22:21:16.826438+0000 (oldest deadline 2024-05-12T22:21:37.926536+0000) 2024-05-12T22:22:03.243 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:22:02 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:22:02.991+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-05-12T22:21:24.828681+0000 front 2024-05-12T22:21:24.828551+0000 (oldest deadline 2024-05-12T22:21:49.428752+0000) 2024-05-12T22:22:03.243 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:22:02 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:22:02.991+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-05-12T22:21:29.429618+0000 front 2024-05-12T22:21:29.429553+0000 (oldest deadline 2024-05-12T22:21:55.229662+0000) 2024-05-12T22:22:03.679 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:22:03 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:22:03.330+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.561281+0000 front 2024-05-12T22:20:41.561023+0000 (oldest deadline 2024-05-12T22:21:06.261036+0000) 2024-05-12T22:22:03.679 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:22:03 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:22:03.330+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:50.163666+0000 front 2024-05-12T22:20:50.163649+0000 (oldest deadline 2024-05-12T22:21:13.063447+0000) 2024-05-12T22:22:03.680 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:22:03 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:22:03.330+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:59.365658+0000 front 2024-05-12T22:20:59.365719+0000 (oldest deadline 2024-05-12T22:21:23.465741+0000) 2024-05-12T22:22:03.680 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:22:03 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:22:03.330+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-05-12T22:21:14.568000+0000 front 2024-05-12T22:21:14.568013+0000 (oldest deadline 2024-05-12T22:21:38.067911+0000) 2024-05-12T22:22:03.680 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:22:03 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:22:03.330+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-05-12T22:21:23.968898+0000 front 2024-05-12T22:21:23.968952+0000 (oldest deadline 2024-05-12T22:21:46.868939+0000) 2024-05-12T22:22:03.680 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:22:03 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:22:03.330+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-05-12T22:21:32.170175+0000 front 2024-05-12T22:21:32.170133+0000 (oldest deadline 2024-05-12T22:21:55.669945+0000) 2024-05-12T22:22:03.680 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:22:03 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:22:03.330+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.86:6820 osd.7 since back 2024-05-12T22:21:41.571241+0000 front 2024-05-12T22:21:41.571165+0000 (oldest deadline 2024-05-12T22:22:03.270950+0000) 2024-05-12T22:22:04.242 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:22:03 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:22:03.942+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:43.519531+0000 front 2024-05-12T22:20:43.519638+0000 (oldest deadline 2024-05-12T22:21:05.219288+0000) 2024-05-12T22:22:04.242 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:22:03 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:22:03.942+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:51.620499+0000 front 2024-05-12T22:20:51.620463+0000 (oldest deadline 2024-05-12T22:21:16.221349+0000) 2024-05-12T22:22:04.242 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:22:03 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:22:03.942+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:59.722447+0000 front 2024-05-12T22:20:59.722655+0000 (oldest deadline 2024-05-12T22:21:21.422316+0000) 2024-05-12T22:22:04.243 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:22:03 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:22:03.942+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-12T22:21:05.224800+0000 front 2024-05-12T22:21:05.225010+0000 (oldest deadline 2024-05-12T22:21:33.425092+0000) 2024-05-12T22:22:04.243 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:22:03 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:22:03.942+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-05-12T22:21:16.826477+0000 front 2024-05-12T22:21:16.826438+0000 (oldest deadline 2024-05-12T22:21:37.926536+0000) 2024-05-12T22:22:04.243 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:22:03 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:22:03.942+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-05-12T22:21:24.828681+0000 front 2024-05-12T22:21:24.828551+0000 (oldest deadline 2024-05-12T22:21:49.428752+0000) 2024-05-12T22:22:04.243 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:22:03 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:22:03.942+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-05-12T22:21:29.429618+0000 front 2024-05-12T22:21:29.429553+0000 (oldest deadline 2024-05-12T22:21:55.229662+0000) 2024-05-12T22:22:04.679 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:22:04 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:22:04.320+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.561281+0000 front 2024-05-12T22:20:41.561023+0000 (oldest deadline 2024-05-12T22:21:06.261036+0000) 2024-05-12T22:22:04.679 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:22:04 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:22:04.320+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:50.163666+0000 front 2024-05-12T22:20:50.163649+0000 (oldest deadline 2024-05-12T22:21:13.063447+0000) 2024-05-12T22:22:04.679 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:22:04 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:22:04.320+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:59.365658+0000 front 2024-05-12T22:20:59.365719+0000 (oldest deadline 2024-05-12T22:21:23.465741+0000) 2024-05-12T22:22:04.680 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:22:04 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:22:04.320+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-05-12T22:21:14.568000+0000 front 2024-05-12T22:21:14.568013+0000 (oldest deadline 2024-05-12T22:21:38.067911+0000) 2024-05-12T22:22:04.680 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:22:04 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:22:04.320+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-05-12T22:21:23.968898+0000 front 2024-05-12T22:21:23.968952+0000 (oldest deadline 2024-05-12T22:21:46.868939+0000) 2024-05-12T22:22:04.680 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:22:04 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:22:04.320+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-05-12T22:21:32.170175+0000 front 2024-05-12T22:21:32.170133+0000 (oldest deadline 2024-05-12T22:21:55.669945+0000) 2024-05-12T22:22:04.680 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:22:04 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:22:04.320+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.86:6820 osd.7 since back 2024-05-12T22:21:41.571241+0000 front 2024-05-12T22:21:41.571165+0000 (oldest deadline 2024-05-12T22:22:03.270950+0000) 2024-05-12T22:22:05.242 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:22:04 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:22:04.966+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:43.519531+0000 front 2024-05-12T22:20:43.519638+0000 (oldest deadline 2024-05-12T22:21:05.219288+0000) 2024-05-12T22:22:05.242 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:22:04 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:22:04.966+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:51.620499+0000 front 2024-05-12T22:20:51.620463+0000 (oldest deadline 2024-05-12T22:21:16.221349+0000) 2024-05-12T22:22:05.242 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:22:04 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:22:04.966+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:59.722447+0000 front 2024-05-12T22:20:59.722655+0000 (oldest deadline 2024-05-12T22:21:21.422316+0000) 2024-05-12T22:22:05.242 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:22:04 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:22:04.966+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-12T22:21:05.224800+0000 front 2024-05-12T22:21:05.225010+0000 (oldest deadline 2024-05-12T22:21:33.425092+0000) 2024-05-12T22:22:05.242 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:22:04 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:22:04.966+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-05-12T22:21:16.826477+0000 front 2024-05-12T22:21:16.826438+0000 (oldest deadline 2024-05-12T22:21:37.926536+0000) 2024-05-12T22:22:05.243 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:22:04 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:22:04.966+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-05-12T22:21:24.828681+0000 front 2024-05-12T22:21:24.828551+0000 (oldest deadline 2024-05-12T22:21:49.428752+0000) 2024-05-12T22:22:05.243 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:22:04 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:22:04.966+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-05-12T22:21:29.429618+0000 front 2024-05-12T22:21:29.429553+0000 (oldest deadline 2024-05-12T22:21:55.229662+0000) 2024-05-12T22:22:05.679 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:22:05 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:22:05.294+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.561281+0000 front 2024-05-12T22:20:41.561023+0000 (oldest deadline 2024-05-12T22:21:06.261036+0000) 2024-05-12T22:22:05.679 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:22:05 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:22:05.294+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:50.163666+0000 front 2024-05-12T22:20:50.163649+0000 (oldest deadline 2024-05-12T22:21:13.063447+0000) 2024-05-12T22:22:05.679 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:22:05 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:22:05.294+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:59.365658+0000 front 2024-05-12T22:20:59.365719+0000 (oldest deadline 2024-05-12T22:21:23.465741+0000) 2024-05-12T22:22:05.679 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:22:05 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:22:05.294+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-05-12T22:21:14.568000+0000 front 2024-05-12T22:21:14.568013+0000 (oldest deadline 2024-05-12T22:21:38.067911+0000) 2024-05-12T22:22:05.680 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:22:05 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:22:05.294+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-05-12T22:21:23.968898+0000 front 2024-05-12T22:21:23.968952+0000 (oldest deadline 2024-05-12T22:21:46.868939+0000) 2024-05-12T22:22:05.680 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:22:05 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:22:05.294+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-05-12T22:21:32.170175+0000 front 2024-05-12T22:21:32.170133+0000 (oldest deadline 2024-05-12T22:21:55.669945+0000) 2024-05-12T22:22:05.680 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:22:05 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:22:05.294+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.86:6820 osd.7 since back 2024-05-12T22:21:41.571241+0000 front 2024-05-12T22:21:41.571165+0000 (oldest deadline 2024-05-12T22:22:03.270950+0000) 2024-05-12T22:22:06.242 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:22:05 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:22:05.959+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:43.519531+0000 front 2024-05-12T22:20:43.519638+0000 (oldest deadline 2024-05-12T22:21:05.219288+0000) 2024-05-12T22:22:06.242 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:22:05 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:22:05.959+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:51.620499+0000 front 2024-05-12T22:20:51.620463+0000 (oldest deadline 2024-05-12T22:21:16.221349+0000) 2024-05-12T22:22:06.242 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:22:05 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:22:05.959+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:59.722447+0000 front 2024-05-12T22:20:59.722655+0000 (oldest deadline 2024-05-12T22:21:21.422316+0000) 2024-05-12T22:22:06.243 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:22:05 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:22:05.959+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-05-12T22:21:05.224800+0000 front 2024-05-12T22:21:05.225010+0000 (oldest deadline 2024-05-12T22:21:33.425092+0000) 2024-05-12T22:22:06.243 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:22:05 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:22:05.959+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-05-12T22:21:16.826477+0000 front 2024-05-12T22:21:16.826438+0000 (oldest deadline 2024-05-12T22:21:37.926536+0000) 2024-05-12T22:22:06.243 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:22:05 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:22:05.959+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-05-12T22:21:24.828681+0000 front 2024-05-12T22:21:24.828551+0000 (oldest deadline 2024-05-12T22:21:49.428752+0000) 2024-05-12T22:22:06.243 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:22:05 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10[59562]: 2024-05-12T22:22:05.959+0000 7f113b6d4700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-05-12T22:21:29.429618+0000 front 2024-05-12T22:21:29.429553+0000 (oldest deadline 2024-05-12T22:21:55.229662+0000) 2024-05-12T22:22:06.679 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:22:06 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:22:06.313+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.561281+0000 front 2024-05-12T22:20:41.561023+0000 (oldest deadline 2024-05-12T22:21:06.261036+0000) 2024-05-12T22:22:06.679 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:22:06 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:22:06.313+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:50.163666+0000 front 2024-05-12T22:20:50.163649+0000 (oldest deadline 2024-05-12T22:21:13.063447+0000) 2024-05-12T22:22:06.679 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:22:06 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:22:06.313+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:59.365658+0000 front 2024-05-12T22:20:59.365719+0000 (oldest deadline 2024-05-12T22:21:23.465741+0000) 2024-05-12T22:22:06.680 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:22:06 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:22:06.313+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-05-12T22:21:14.568000+0000 front 2024-05-12T22:21:14.568013+0000 (oldest deadline 2024-05-12T22:21:38.067911+0000) 2024-05-12T22:22:06.680 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:22:06 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:22:06.313+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-05-12T22:21:23.968898+0000 front 2024-05-12T22:21:23.968952+0000 (oldest deadline 2024-05-12T22:21:46.868939+0000) 2024-05-12T22:22:06.680 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:22:06 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:22:06.313+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-05-12T22:21:32.170175+0000 front 2024-05-12T22:21:32.170133+0000 (oldest deadline 2024-05-12T22:21:55.669945+0000) 2024-05-12T22:22:06.680 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:22:06 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:22:06.313+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.86:6820 osd.7 since back 2024-05-12T22:21:41.571241+0000 front 2024-05-12T22:21:41.571165+0000 (oldest deadline 2024-05-12T22:22:03.270950+0000) 2024-05-12T22:22:07.047 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:22:06 smithi086 podman[71220]: 2024-05-12 22:22:06.772789387 +0000 UTC m=+5.413941910 container died 989788f62f1c9826c7d616d30ca8239afa170ec6768258f5018e604cc6d8936e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, CEPH_POINT_RELEASE=, RELEASE=squid-f72fecf, ceph=True, org.label-schema.build-date=20240507, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-12T22:22:07.047 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:22:07 smithi086 podman[71220]: 2024-05-12 22:22:07.047070875 +0000 UTC m=+5.688223398 container cleanup 989788f62f1c9826c7d616d30ca8239afa170ec6768258f5018e604cc6d8936e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, RELEASE=squid-f72fecf, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240507, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-05-12T22:22:07.297 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:22:07 smithi086 bash[71220]: ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10 2024-05-12T22:22:07.581 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:22:07 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:22:07.320+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.561281+0000 front 2024-05-12T22:20:41.561023+0000 (oldest deadline 2024-05-12T22:21:06.261036+0000) 2024-05-12T22:22:07.581 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:22:07 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:22:07.320+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:50.163666+0000 front 2024-05-12T22:20:50.163649+0000 (oldest deadline 2024-05-12T22:21:13.063447+0000) 2024-05-12T22:22:07.581 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:22:07 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:22:07.320+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:59.365658+0000 front 2024-05-12T22:20:59.365719+0000 (oldest deadline 2024-05-12T22:21:23.465741+0000) 2024-05-12T22:22:07.581 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:22:07 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:22:07.320+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-05-12T22:21:14.568000+0000 front 2024-05-12T22:21:14.568013+0000 (oldest deadline 2024-05-12T22:21:38.067911+0000) 2024-05-12T22:22:07.581 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:22:07 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:22:07.320+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-05-12T22:21:23.968898+0000 front 2024-05-12T22:21:23.968952+0000 (oldest deadline 2024-05-12T22:21:46.868939+0000) 2024-05-12T22:22:07.581 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:22:07 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:22:07.320+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-05-12T22:21:32.170175+0000 front 2024-05-12T22:21:32.170133+0000 (oldest deadline 2024-05-12T22:21:55.669945+0000) 2024-05-12T22:22:07.582 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:22:07 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:22:07.320+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.86:6820 osd.7 since back 2024-05-12T22:21:41.571241+0000 front 2024-05-12T22:21:41.571165+0000 (oldest deadline 2024-05-12T22:22:03.270950+0000) 2024-05-12T22:22:07.588 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:22:07 smithi086 podman[71241]: 2024-05-12 22:22:07.298230372 +0000 UTC m=+0.524444122 container remove 989788f62f1c9826c7d616d30ca8239afa170ec6768258f5018e604cc6d8936e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240507, ceph=True, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=squid-f72fecf) 2024-05-12T22:22:07.876 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:22:07 smithi086 podman[71335]: 2024-05-12 22:22:07.491055013 +0000 UTC m=+0.017517930 image pull 8252b7944d3b5c188c0ff632647ba6cc4e9a19588ee097b8a3dba27b42779c25 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc 2024-05-12T22:22:07.876 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:22:07 smithi086 podman[71335]: 2024-05-12 22:22:07.599031827 +0000 UTC m=+0.125494734 container create 4b003d770ed1023d1d15a1f4148dfd6f15e5adb193b83d91a7b7e11488ce4a11 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10-deactivate, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240507, RELEASE=squid-f72fecf) 2024-05-12T22:22:07.876 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:22:07 smithi086 podman[71335]: 2024-05-12 22:22:07.801526419 +0000 UTC m=+0.327989323 container init 4b003d770ed1023d1d15a1f4148dfd6f15e5adb193b83d91a7b7e11488ce4a11 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10-deactivate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240507, GIT_BRANCH=HEAD, RELEASE=squid-f72fecf, org.label-schema.license=GPLv2, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-05-12T22:22:07.876 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:22:07 smithi086 podman[71335]: 2024-05-12 22:22:07.804590187 +0000 UTC m=+0.331053094 container start 4b003d770ed1023d1d15a1f4148dfd6f15e5adb193b83d91a7b7e11488ce4a11 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10-deactivate, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240507, GIT_CLEAN=True, RELEASE=squid-f72fecf) 2024-05-12T22:22:07.876 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:22:07 smithi086 podman[71335]: 2024-05-12 22:22:07.876349141 +0000 UTC m=+0.402812054 container attach 4b003d770ed1023d1d15a1f4148dfd6f15e5adb193b83d91a7b7e11488ce4a11 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10-deactivate, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, RELEASE=squid-f72fecf, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240507) 2024-05-12T22:22:08.242 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:22:08 smithi086 podman[71335]: 2024-05-12 22:22:08.094642015 +0000 UTC m=+0.621104920 container died 4b003d770ed1023d1d15a1f4148dfd6f15e5adb193b83d91a7b7e11488ce4a11 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240507, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=squid-f72fecf, org.label-schema.license=GPLv2, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, GIT_CLEAN=True, ceph=True) 2024-05-12T22:22:08.597 DEBUG:teuthology.orchestra.run.smithi086:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968@osd.10.service' 2024-05-12T22:22:08.659 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:22:08 smithi086 podman[71361]: 2024-05-12 22:22:08.570759935 +0000 UTC m=+0.471414291 container remove 4b003d770ed1023d1d15a1f4148dfd6f15e5adb193b83d91a7b7e11488ce4a11 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-10-deactivate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=squid-f72fecf, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240507, ceph=True) 2024-05-12T22:22:08.659 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:22:08 smithi086 systemd[1]: ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968@osd.10.service: Deactivated successfully. 2024-05-12T22:22:08.660 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:22:08 smithi086 systemd[1]: Stopped Ceph osd.10 for 6b92e7b4-10aa-11ef-bc98-c7b262605968. 2024-05-12T22:22:08.660 INFO:journalctl@ceph.osd.10.smithi086.stdout:May 12 22:22:08 smithi086 systemd[1]: ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968@osd.10.service: Consumed 14.292s CPU time. 2024-05-12T22:22:08.674 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-12T22:22:08.675 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-05-12T22:22:08.675 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-05-12T22:22:08.675 DEBUG:teuthology.orchestra.run.smithi186:> sudo systemctl stop ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968@osd.11 2024-05-12T22:22:08.679 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:22:08 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:22:08.334+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.561281+0000 front 2024-05-12T22:20:41.561023+0000 (oldest deadline 2024-05-12T22:21:06.261036+0000) 2024-05-12T22:22:08.679 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:22:08 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:22:08.334+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:50.163666+0000 front 2024-05-12T22:20:50.163649+0000 (oldest deadline 2024-05-12T22:21:13.063447+0000) 2024-05-12T22:22:08.679 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:22:08 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:22:08.334+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:59.365658+0000 front 2024-05-12T22:20:59.365719+0000 (oldest deadline 2024-05-12T22:21:23.465741+0000) 2024-05-12T22:22:08.679 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:22:08 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:22:08.334+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-05-12T22:21:14.568000+0000 front 2024-05-12T22:21:14.568013+0000 (oldest deadline 2024-05-12T22:21:38.067911+0000) 2024-05-12T22:22:08.679 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:22:08 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:22:08.334+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-05-12T22:21:23.968898+0000 front 2024-05-12T22:21:23.968952+0000 (oldest deadline 2024-05-12T22:21:46.868939+0000) 2024-05-12T22:22:08.679 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:22:08 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:22:08.334+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-05-12T22:21:32.170175+0000 front 2024-05-12T22:21:32.170133+0000 (oldest deadline 2024-05-12T22:21:55.669945+0000) 2024-05-12T22:22:08.680 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:22:08 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:22:08.334+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.86:6820 osd.7 since back 2024-05-12T22:21:41.571241+0000 front 2024-05-12T22:21:41.571165+0000 (oldest deadline 2024-05-12T22:22:03.270950+0000) 2024-05-12T22:22:08.975 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:22:08 smithi186 systemd[1]: Stopping Ceph osd.11 for 6b92e7b4-10aa-11ef-bc98-c7b262605968... 2024-05-12T22:22:09.306 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:22:08 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:22:08.974+0000 7f9706c91700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-12T22:22:09.307 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:22:08 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:22:08.974+0000 7f9706c91700 -1 osd.11 102 *** Got signal Terminated *** 2024-05-12T22:22:09.307 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:22:08 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:22:08.974+0000 7f9706c91700 -1 osd.11 102 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-12T22:22:09.307 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:22:09 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:22:09.305+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.561281+0000 front 2024-05-12T22:20:41.561023+0000 (oldest deadline 2024-05-12T22:21:06.261036+0000) 2024-05-12T22:22:09.307 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:22:09 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:22:09.305+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:50.163666+0000 front 2024-05-12T22:20:50.163649+0000 (oldest deadline 2024-05-12T22:21:13.063447+0000) 2024-05-12T22:22:09.307 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:22:09 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:22:09.305+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:59.365658+0000 front 2024-05-12T22:20:59.365719+0000 (oldest deadline 2024-05-12T22:21:23.465741+0000) 2024-05-12T22:22:09.307 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:22:09 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:22:09.305+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-05-12T22:21:14.568000+0000 front 2024-05-12T22:21:14.568013+0000 (oldest deadline 2024-05-12T22:21:38.067911+0000) 2024-05-12T22:22:09.308 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:22:09 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:22:09.305+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-05-12T22:21:23.968898+0000 front 2024-05-12T22:21:23.968952+0000 (oldest deadline 2024-05-12T22:21:46.868939+0000) 2024-05-12T22:22:09.308 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:22:09 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:22:09.305+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-05-12T22:21:32.170175+0000 front 2024-05-12T22:21:32.170133+0000 (oldest deadline 2024-05-12T22:21:55.669945+0000) 2024-05-12T22:22:09.308 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:22:09 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:22:09.305+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.86:6820 osd.7 since back 2024-05-12T22:21:41.571241+0000 front 2024-05-12T22:21:41.571165+0000 (oldest deadline 2024-05-12T22:22:03.270950+0000) 2024-05-12T22:22:10.679 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:22:10 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:22:10.287+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.561281+0000 front 2024-05-12T22:20:41.561023+0000 (oldest deadline 2024-05-12T22:21:06.261036+0000) 2024-05-12T22:22:10.680 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:22:10 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:22:10.287+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:50.163666+0000 front 2024-05-12T22:20:50.163649+0000 (oldest deadline 2024-05-12T22:21:13.063447+0000) 2024-05-12T22:22:10.680 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:22:10 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:22:10.287+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:59.365658+0000 front 2024-05-12T22:20:59.365719+0000 (oldest deadline 2024-05-12T22:21:23.465741+0000) 2024-05-12T22:22:10.680 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:22:10 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:22:10.287+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-05-12T22:21:14.568000+0000 front 2024-05-12T22:21:14.568013+0000 (oldest deadline 2024-05-12T22:21:38.067911+0000) 2024-05-12T22:22:10.680 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:22:10 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:22:10.287+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-05-12T22:21:23.968898+0000 front 2024-05-12T22:21:23.968952+0000 (oldest deadline 2024-05-12T22:21:46.868939+0000) 2024-05-12T22:22:10.680 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:22:10 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:22:10.287+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-05-12T22:21:32.170175+0000 front 2024-05-12T22:21:32.170133+0000 (oldest deadline 2024-05-12T22:21:55.669945+0000) 2024-05-12T22:22:10.680 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:22:10 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:22:10.287+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.86:6820 osd.7 since back 2024-05-12T22:21:41.571241+0000 front 2024-05-12T22:21:41.571165+0000 (oldest deadline 2024-05-12T22:22:03.270950+0000) 2024-05-12T22:22:11.679 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:22:11 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:22:11.275+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.561281+0000 front 2024-05-12T22:20:41.561023+0000 (oldest deadline 2024-05-12T22:21:06.261036+0000) 2024-05-12T22:22:11.679 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:22:11 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:22:11.275+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:50.163666+0000 front 2024-05-12T22:20:50.163649+0000 (oldest deadline 2024-05-12T22:21:13.063447+0000) 2024-05-12T22:22:11.679 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:22:11 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:22:11.275+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:59.365658+0000 front 2024-05-12T22:20:59.365719+0000 (oldest deadline 2024-05-12T22:21:23.465741+0000) 2024-05-12T22:22:11.680 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:22:11 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:22:11.275+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-05-12T22:21:14.568000+0000 front 2024-05-12T22:21:14.568013+0000 (oldest deadline 2024-05-12T22:21:38.067911+0000) 2024-05-12T22:22:11.680 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:22:11 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:22:11.275+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-05-12T22:21:23.968898+0000 front 2024-05-12T22:21:23.968952+0000 (oldest deadline 2024-05-12T22:21:46.868939+0000) 2024-05-12T22:22:11.680 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:22:11 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:22:11.275+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-05-12T22:21:32.170175+0000 front 2024-05-12T22:21:32.170133+0000 (oldest deadline 2024-05-12T22:21:55.669945+0000) 2024-05-12T22:22:11.680 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:22:11 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:22:11.275+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.86:6820 osd.7 since back 2024-05-12T22:21:41.571241+0000 front 2024-05-12T22:21:41.571165+0000 (oldest deadline 2024-05-12T22:22:03.270950+0000) 2024-05-12T22:22:12.679 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:22:12 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:22:12.273+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.561281+0000 front 2024-05-12T22:20:41.561023+0000 (oldest deadline 2024-05-12T22:21:06.261036+0000) 2024-05-12T22:22:12.679 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:22:12 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:22:12.273+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:50.163666+0000 front 2024-05-12T22:20:50.163649+0000 (oldest deadline 2024-05-12T22:21:13.063447+0000) 2024-05-12T22:22:12.679 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:22:12 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:22:12.273+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:59.365658+0000 front 2024-05-12T22:20:59.365719+0000 (oldest deadline 2024-05-12T22:21:23.465741+0000) 2024-05-12T22:22:12.680 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:22:12 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:22:12.273+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-05-12T22:21:14.568000+0000 front 2024-05-12T22:21:14.568013+0000 (oldest deadline 2024-05-12T22:21:38.067911+0000) 2024-05-12T22:22:12.680 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:22:12 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:22:12.273+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-05-12T22:21:23.968898+0000 front 2024-05-12T22:21:23.968952+0000 (oldest deadline 2024-05-12T22:21:46.868939+0000) 2024-05-12T22:22:12.680 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:22:12 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:22:12.273+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-05-12T22:21:32.170175+0000 front 2024-05-12T22:21:32.170133+0000 (oldest deadline 2024-05-12T22:21:55.669945+0000) 2024-05-12T22:22:12.680 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:22:12 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:22:12.273+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.86:6820 osd.7 since back 2024-05-12T22:21:41.571241+0000 front 2024-05-12T22:21:41.571165+0000 (oldest deadline 2024-05-12T22:22:03.270950+0000) 2024-05-12T22:22:13.679 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:22:13 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:22:13.309+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-05-12T22:20:41.561281+0000 front 2024-05-12T22:20:41.561023+0000 (oldest deadline 2024-05-12T22:21:06.261036+0000) 2024-05-12T22:22:13.679 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:22:13 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:22:13.309+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.86:6804 osd.1 since back 2024-05-12T22:20:50.163666+0000 front 2024-05-12T22:20:50.163649+0000 (oldest deadline 2024-05-12T22:21:13.063447+0000) 2024-05-12T22:22:13.680 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:22:13 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:22:13.309+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-05-12T22:20:59.365658+0000 front 2024-05-12T22:20:59.365719+0000 (oldest deadline 2024-05-12T22:21:23.465741+0000) 2024-05-12T22:22:13.680 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:22:13 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:22:13.309+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.86:6812 osd.4 since back 2024-05-12T22:21:14.568000+0000 front 2024-05-12T22:21:14.568013+0000 (oldest deadline 2024-05-12T22:21:38.067911+0000) 2024-05-12T22:22:13.680 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:22:13 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:22:13.309+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-05-12T22:21:23.968898+0000 front 2024-05-12T22:21:23.968952+0000 (oldest deadline 2024-05-12T22:21:46.868939+0000) 2024-05-12T22:22:13.680 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:22:13 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:22:13.309+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-05-12T22:21:32.170175+0000 front 2024-05-12T22:21:32.170133+0000 (oldest deadline 2024-05-12T22:21:55.669945+0000) 2024-05-12T22:22:13.680 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:22:13 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:22:13.309+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.86:6820 osd.7 since back 2024-05-12T22:21:41.571241+0000 front 2024-05-12T22:21:41.571165+0000 (oldest deadline 2024-05-12T22:22:03.270950+0000) 2024-05-12T22:22:13.680 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:22:13 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11[60636]: 2024-05-12T22:22:13.309+0000 7f9703a86700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.186:6820 osd.8 since back 2024-05-12T22:21:48.571819+0000 front 2024-05-12T22:21:48.571798+0000 (oldest deadline 2024-05-12T22:22:12.671744+0000) 2024-05-12T22:22:14.569 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:22:14 smithi186 podman[72292]: 2024-05-12 22:22:14.300955533 +0000 UTC m=+5.428508792 container died 8f5f3e5c471e598650dc9492ccebeae6cae4478a314e4949dcedd49041b21ee0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11, org.label-schema.build-date=20240507, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=squid-f72fecf, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-05-12T22:22:14.826 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:22:14 smithi186 podman[72292]: 2024-05-12 22:22:14.567447934 +0000 UTC m=+5.695001185 container cleanup 8f5f3e5c471e598650dc9492ccebeae6cae4478a314e4949dcedd49041b21ee0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11, RELEASE=squid-f72fecf, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240507, org.label-schema.schema-version=1.0) 2024-05-12T22:22:14.826 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:22:14 smithi186 bash[72292]: ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11 2024-05-12T22:22:15.127 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:22:14 smithi186 podman[72310]: 2024-05-12 22:22:14.826978404 +0000 UTC m=+0.527110836 container remove 8f5f3e5c471e598650dc9492ccebeae6cae4478a314e4949dcedd49041b21ee0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240507, io.buildah.version=1.33.5, RELEASE=squid-f72fecf, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2) 2024-05-12T22:22:15.429 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:22:15 smithi186 podman[72406]: 2024-05-12 22:22:15.12765675 +0000 UTC m=+0.102590502 container create 325a24873ab1465771ac49d3f818c603207a328b37314a7b28bb472e917b0901 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11-deactivate, org.label-schema.vendor=CentOS, RELEASE=squid-f72fecf, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.build-date=20240507, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-12T22:22:15.429 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:22:15 smithi186 podman[72406]: 2024-05-12 22:22:15.043401296 +0000 UTC m=+0.018335072 image pull 8252b7944d3b5c188c0ff632647ba6cc4e9a19588ee097b8a3dba27b42779c25 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc 2024-05-12T22:22:15.430 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:22:15 smithi186 podman[72406]: 2024-05-12 22:22:15.363835428 +0000 UTC m=+0.338769181 container init 325a24873ab1465771ac49d3f818c603207a328b37314a7b28bb472e917b0901 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11-deactivate, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=squid-f72fecf, org.label-schema.build-date=20240507, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2) 2024-05-12T22:22:15.430 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:22:15 smithi186 podman[72406]: 2024-05-12 22:22:15.367394207 +0000 UTC m=+0.342327960 container start 325a24873ab1465771ac49d3f818c603207a328b37314a7b28bb472e917b0901 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=squid-f72fecf, org.label-schema.build-date=20240507, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-12T22:22:15.929 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:22:15 smithi186 podman[72406]: 2024-05-12 22:22:15.448322676 +0000 UTC m=+0.423256429 container attach 325a24873ab1465771ac49d3f818c603207a328b37314a7b28bb472e917b0901 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11-deactivate, io.buildah.version=1.33.5, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-f72fecf, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240507, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-05-12T22:22:15.929 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:22:15 smithi186 podman[72406]: 2024-05-12 22:22:15.660019399 +0000 UTC m=+0.634953161 container died 325a24873ab1465771ac49d3f818c603207a328b37314a7b28bb472e917b0901 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11-deactivate, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240507, org.label-schema.vendor=CentOS, RELEASE=squid-f72fecf, io.buildah.version=1.33.5) 2024-05-12T22:22:16.154 DEBUG:teuthology.orchestra.run.smithi186:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968@osd.11.service' 2024-05-12T22:22:16.217 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:22:16 smithi186 podman[72432]: 2024-05-12 22:22:16.135430346 +0000 UTC m=+0.468846706 container remove 325a24873ab1465771ac49d3f818c603207a328b37314a7b28bb472e917b0901 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-osd-11-deactivate, GIT_BRANCH=HEAD, RELEASE=squid-f72fecf, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240507, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e) 2024-05-12T22:22:16.217 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:22:16 smithi186 systemd[1]: ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968@osd.11.service: Deactivated successfully. 2024-05-12T22:22:16.217 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:22:16 smithi186 systemd[1]: Stopped Ceph osd.11 for 6b92e7b4-10aa-11ef-bc98-c7b262605968. 2024-05-12T22:22:16.217 INFO:journalctl@ceph.osd.11.smithi186.stdout:May 12 22:22:16 smithi186 systemd[1]: ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968@osd.11.service: Consumed 13.306s CPU time. 2024-05-12T22:22:16.233 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-12T22:22:16.233 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-05-12T22:22:16.233 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-05-12T22:22:16.234 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968@mds.a 2024-05-12T22:22:16.785 INFO:journalctl@ceph.mds.a.smithi028.stdout:May 12 22:22:16 smithi028 systemd[1]: Stopping Ceph mds.a for 6b92e7b4-10aa-11ef-bc98-c7b262605968... 2024-05-12T22:22:17.285 INFO:journalctl@ceph.mds.a.smithi028.stdout:May 12 22:22:16 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-a[68142]: 2024-05-12T22:22:16.848+0000 7f0d32e00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-12T22:22:17.285 INFO:journalctl@ceph.mds.a.smithi028.stdout:May 12 22:22:16 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-a[68142]: 2024-05-12T22:22:16.848+0000 7f0d32e00700 -1 mds.a *** got signal Terminated *** 2024-05-12T22:22:21.138 INFO:journalctl@ceph.mds.a.smithi028.stdout:May 12 22:22:20 smithi028 podman[100398]: 2024-05-12 22:22:20.863557353 +0000 UTC m=+4.161802174 container died 8d25c544008053ac556b084585611198c114f05ad1d14ca0a241d8e0e03af57a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-a, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240507, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=squid-f72fecf, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-05-12T22:22:21.509 INFO:journalctl@ceph.mds.a.smithi028.stdout:May 12 22:22:21 smithi028 podman[100398]: 2024-05-12 22:22:21.258475517 +0000 UTC m=+4.556720338 container cleanup 8d25c544008053ac556b084585611198c114f05ad1d14ca0a241d8e0e03af57a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-a, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-f72fecf, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240507, GIT_BRANCH=HEAD, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-05-12T22:22:21.510 INFO:journalctl@ceph.mds.a.smithi028.stdout:May 12 22:22:21 smithi028 bash[100398]: ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-a 2024-05-12T22:22:21.784 INFO:journalctl@ceph.mds.a.smithi028.stdout:May 12 22:22:21 smithi028 podman[100412]: 2024-05-12 22:22:21.509563537 +0000 UTC m=+0.628273053 container remove 8d25c544008053ac556b084585611198c114f05ad1d14ca0a241d8e0e03af57a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-a, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-f72fecf, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240507, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-05-12T22:22:22.043 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968@mds.a.service' 2024-05-12T22:22:22.161 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-12T22:22:22.161 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-05-12T22:22:22.161 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-05-12T22:22:22.162 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968@mds.d 2024-05-12T22:22:22.535 INFO:journalctl@ceph.mds.d.smithi028.stdout:May 12 22:22:22 smithi028 systemd[1]: Stopping Ceph mds.d for 6b92e7b4-10aa-11ef-bc98-c7b262605968... 2024-05-12T22:22:23.035 INFO:journalctl@ceph.mds.d.smithi028.stdout:May 12 22:22:22 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-d[67158]: 2024-05-12T22:22:22.752+0000 7fd339a00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-12T22:22:23.035 INFO:journalctl@ceph.mds.d.smithi028.stdout:May 12 22:22:22 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-d[67158]: 2024-05-12T22:22:22.752+0000 7fd339a00700 -1 mds.d *** got signal Terminated *** 2024-05-12T22:22:27.029 INFO:journalctl@ceph.mds.d.smithi028.stdout:May 12 22:22:26 smithi028 podman[100517]: 2024-05-12 22:22:26.770489176 +0000 UTC m=+4.149346455 container died a971d49e34e43d0c0b3bfaff3d4abeaa13df5ea39822fda4351d28b45665d568 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-d, RELEASE=squid-f72fecf, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, GIT_BRANCH=HEAD, org.label-schema.build-date=20240507, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-12T22:22:27.372 INFO:journalctl@ceph.mds.d.smithi028.stdout:May 12 22:22:27 smithi028 podman[100517]: 2024-05-12 22:22:27.136818621 +0000 UTC m=+4.515675893 container cleanup a971d49e34e43d0c0b3bfaff3d4abeaa13df5ea39822fda4351d28b45665d568 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-d, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240507, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=squid-f72fecf) 2024-05-12T22:22:27.373 INFO:journalctl@ceph.mds.d.smithi028.stdout:May 12 22:22:27 smithi028 bash[100517]: ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-d 2024-05-12T22:22:27.785 INFO:journalctl@ceph.mds.d.smithi028.stdout:May 12 22:22:27 smithi028 podman[100531]: 2024-05-12 22:22:27.370852724 +0000 UTC m=+0.584342867 container remove a971d49e34e43d0c0b3bfaff3d4abeaa13df5ea39822fda4351d28b45665d568 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-d, org.label-schema.build-date=20240507, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=squid-f72fecf, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-12T22:22:27.855 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968@mds.d.service' 2024-05-12T22:22:27.970 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-12T22:22:27.970 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-05-12T22:22:27.971 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-05-12T22:22:27.971 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968@mds.g 2024-05-12T22:22:28.535 INFO:journalctl@ceph.mds.g.smithi028.stdout:May 12 22:22:28 smithi028 systemd[1]: Stopping Ceph mds.g for 6b92e7b4-10aa-11ef-bc98-c7b262605968... 2024-05-12T22:22:29.035 INFO:journalctl@ceph.mds.g.smithi028.stdout:May 12 22:22:28 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-g[67758]: 2024-05-12T22:22:28.588+0000 7f2471600700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-12T22:22:29.035 INFO:journalctl@ceph.mds.g.smithi028.stdout:May 12 22:22:28 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-g[67758]: 2024-05-12T22:22:28.588+0000 7f2471600700 -1 mds.g *** got signal Terminated *** 2024-05-12T22:22:29.978 INFO:journalctl@ceph.mds.g.smithi028.stdout:May 12 22:22:29 smithi028 podman[100636]: 2024-05-12 22:22:29.690406942 +0000 UTC m=+1.261462537 container died 82c3e6745e1fcc4a660b4369bfb61045ce21d45d1662f2c141c8002762f74fd1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-g, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240507, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=squid-f72fecf, org.label-schema.schema-version=1.0, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-05-12T22:22:30.285 INFO:journalctl@ceph.mds.g.smithi028.stdout:May 12 22:22:30 smithi028 podman[100636]: 2024-05-12 22:22:30.0985757 +0000 UTC m=+1.669631288 container cleanup 82c3e6745e1fcc4a660b4369bfb61045ce21d45d1662f2c141c8002762f74fd1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-g, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240507, GIT_CLEAN=True, RELEASE=squid-f72fecf, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-05-12T22:22:30.285 INFO:journalctl@ceph.mds.g.smithi028.stdout:May 12 22:22:30 smithi028 bash[100636]: ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-g 2024-05-12T22:22:30.785 INFO:journalctl@ceph.mds.g.smithi028.stdout:May 12 22:22:30 smithi028 podman[100650]: 2024-05-12 22:22:30.329152165 +0000 UTC m=+0.621334754 container remove 82c3e6745e1fcc4a660b4369bfb61045ce21d45d1662f2c141c8002762f74fd1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-g, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-f72fecf, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240507, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-12T22:22:30.826 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968@mds.g.service' 2024-05-12T22:22:30.934 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-12T22:22:30.935 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-05-12T22:22:30.935 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-05-12T22:22:30.935 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968@mds.j 2024-05-12T22:22:31.467 INFO:journalctl@ceph.mds.j.smithi028.stdout:May 12 22:22:31 smithi028 systemd[1]: Stopping Ceph mds.j for 6b92e7b4-10aa-11ef-bc98-c7b262605968... 2024-05-12T22:22:31.785 INFO:journalctl@ceph.mds.j.smithi028.stdout:May 12 22:22:31 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-j[66813]: 2024-05-12T22:22:31.467+0000 7f94fc200700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-12T22:22:31.785 INFO:journalctl@ceph.mds.j.smithi028.stdout:May 12 22:22:31 smithi028 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-j[66813]: 2024-05-12T22:22:31.467+0000 7f94fc200700 -1 mds.j *** got signal Terminated *** 2024-05-12T22:22:35.501 INFO:journalctl@ceph.mds.j.smithi028.stdout:May 12 22:22:35 smithi028 podman[100755]: 2024-05-12 22:22:35.22381754 +0000 UTC m=+3.859599266 container died 78eeedeae0c598c11d1db2c528edc9b8ffbf55983b0af8330fc7ee2d4f4200af (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-j, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240507, RELEASE=squid-f72fecf, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0) 2024-05-12T22:22:35.785 INFO:journalctl@ceph.mds.j.smithi028.stdout:May 12 22:22:35 smithi028 podman[100755]: 2024-05-12 22:22:35.610486795 +0000 UTC m=+4.246268530 container cleanup 78eeedeae0c598c11d1db2c528edc9b8ffbf55983b0af8330fc7ee2d4f4200af (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-j, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=squid-f72fecf, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240507, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-05-12T22:22:35.785 INFO:journalctl@ceph.mds.j.smithi028.stdout:May 12 22:22:35 smithi028 bash[100755]: ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-j 2024-05-12T22:22:36.270 INFO:journalctl@ceph.mds.j.smithi028.stdout:May 12 22:22:35 smithi028 podman[100780]: 2024-05-12 22:22:35.844891133 +0000 UTC m=+0.601270373 container remove 78eeedeae0c598c11d1db2c528edc9b8ffbf55983b0af8330fc7ee2d4f4200af (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-j, org.label-schema.build-date=20240507, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=squid-f72fecf, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-05-12T22:22:36.305 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968@mds.j.service' 2024-05-12T22:22:36.413 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-12T22:22:36.413 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-05-12T22:22:36.413 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-05-12T22:22:36.414 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl stop ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968@mds.b 2024-05-12T22:22:36.737 INFO:journalctl@ceph.mds.b.smithi086.stdout:May 12 22:22:36 smithi086 systemd[1]: Stopping Ceph mds.b for 6b92e7b4-10aa-11ef-bc98-c7b262605968... 2024-05-12T22:22:36.992 INFO:journalctl@ceph.mds.b.smithi086.stdout:May 12 22:22:36 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-b[64802]: 2024-05-12T22:22:36.737+0000 7f3eddbb2700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-12T22:22:36.992 INFO:journalctl@ceph.mds.b.smithi086.stdout:May 12 22:22:36 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-b[64802]: 2024-05-12T22:22:36.737+0000 7f3eddbb2700 -1 mds.b *** got signal Terminated *** 2024-05-12T22:22:41.072 INFO:journalctl@ceph.mds.b.smithi086.stdout:May 12 22:22:40 smithi086 podman[71439]: 2024-05-12 22:22:40.745920694 +0000 UTC m=+4.143048299 container died c6618c6944eb2b460403c7fa3458d4db60d8b0a046be4fac7c80d55607c7fb2c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-b, org.label-schema.license=GPLv2, org.label-schema.build-date=20240507, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=squid-f72fecf, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-05-12T22:22:41.387 INFO:journalctl@ceph.mds.b.smithi086.stdout:May 12 22:22:41 smithi086 podman[71439]: 2024-05-12 22:22:41.14581762 +0000 UTC m=+4.542945230 container cleanup c6618c6944eb2b460403c7fa3458d4db60d8b0a046be4fac7c80d55607c7fb2c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-b, RELEASE=squid-f72fecf, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240507, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-05-12T22:22:41.388 INFO:journalctl@ceph.mds.b.smithi086.stdout:May 12 22:22:41 smithi086 bash[71439]: ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-b 2024-05-12T22:22:41.535 DEBUG:teuthology.orchestra.run.smithi086:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968@mds.b.service' 2024-05-12T22:22:41.569 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-12T22:22:41.569 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-05-12T22:22:41.569 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-05-12T22:22:41.569 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl stop ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968@mds.e 2024-05-12T22:22:41.992 INFO:journalctl@ceph.mds.e.smithi086.stdout:May 12 22:22:41 smithi086 systemd[1]: Stopping Ceph mds.e for 6b92e7b4-10aa-11ef-bc98-c7b262605968... 2024-05-12T22:22:41.992 INFO:journalctl@ceph.mds.e.smithi086.stdout:May 12 22:22:41 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-e[64174]: 2024-05-12T22:22:41.882+0000 7fdd5e5fa700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-12T22:22:41.992 INFO:journalctl@ceph.mds.e.smithi086.stdout:May 12 22:22:41 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-e[64174]: 2024-05-12T22:22:41.882+0000 7fdd5e5fa700 -1 mds.e *** got signal Terminated *** 2024-05-12T22:22:46.190 INFO:journalctl@ceph.mds.e.smithi086.stdout:May 12 22:22:45 smithi086 podman[71572]: 2024-05-12 22:22:45.890377946 +0000 UTC m=+4.106696941 container died d1d854c7ed0bdecf637489fdc37ec7118d6d395acf840a27ea55c05a62ca1305 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-e, org.label-schema.schema-version=1.0, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240507, RELEASE=squid-f72fecf, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-05-12T22:22:46.190 INFO:journalctl@ceph.mds.e.smithi086.stdout:May 12 22:22:46 smithi086 podman[71572]: 2024-05-12 22:22:46.190643862 +0000 UTC m=+4.406962857 container cleanup d1d854c7ed0bdecf637489fdc37ec7118d6d395acf840a27ea55c05a62ca1305 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-e, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=squid-f72fecf, ceph=True, org.label-schema.build-date=20240507, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-05-12T22:22:46.492 INFO:journalctl@ceph.mds.e.smithi086.stdout:May 12 22:22:46 smithi086 bash[71572]: ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-e 2024-05-12T22:22:46.492 INFO:journalctl@ceph.mds.e.smithi086.stdout:May 12 22:22:46 smithi086 podman[71586]: 2024-05-12 22:22:46.391449533 +0000 UTC m=+0.498321224 container remove d1d854c7ed0bdecf637489fdc37ec7118d6d395acf840a27ea55c05a62ca1305 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-e, CEPH_POINT_RELEASE=, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-f72fecf, org.label-schema.build-date=20240507, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5) 2024-05-12T22:22:46.516 DEBUG:teuthology.orchestra.run.smithi086:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968@mds.e.service' 2024-05-12T22:22:46.548 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-12T22:22:46.548 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-05-12T22:22:46.548 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-05-12T22:22:46.549 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl stop ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968@mds.h 2024-05-12T22:22:46.860 INFO:journalctl@ceph.mds.h.smithi086.stdout:May 12 22:22:46 smithi086 systemd[1]: Stopping Ceph mds.h for 6b92e7b4-10aa-11ef-bc98-c7b262605968... 2024-05-12T22:22:47.242 INFO:journalctl@ceph.mds.h.smithi086.stdout:May 12 22:22:46 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-h[64496]: 2024-05-12T22:22:46.860+0000 7f37ebf8c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-12T22:22:47.242 INFO:journalctl@ceph.mds.h.smithi086.stdout:May 12 22:22:46 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-h[64496]: 2024-05-12T22:22:46.860+0000 7f37ebf8c700 -1 mds.h *** got signal Terminated *** 2024-05-12T22:22:49.886 INFO:journalctl@ceph.mds.h.smithi086.stdout:May 12 22:22:49 smithi086 podman[71692]: 2024-05-12 22:22:49.624678117 +0000 UTC m=+2.857396924 container died 8e70c957f69d5998c4d23321e0389c5f741ed4027daa456b71c01cdeb6435ea0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-h, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.build-date=20240507, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-f72fecf, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-05-12T22:22:50.161 INFO:journalctl@ceph.mds.h.smithi086.stdout:May 12 22:22:49 smithi086 podman[71692]: 2024-05-12 22:22:49.97720879 +0000 UTC m=+3.209927593 container cleanup 8e70c957f69d5998c4d23321e0389c5f741ed4027daa456b71c01cdeb6435ea0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-h, org.label-schema.build-date=20240507, RELEASE=squid-f72fecf, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-05-12T22:22:50.161 INFO:journalctl@ceph.mds.h.smithi086.stdout:May 12 22:22:49 smithi086 bash[71692]: ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-h 2024-05-12T22:22:50.373 DEBUG:teuthology.orchestra.run.smithi086:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968@mds.h.service' 2024-05-12T22:22:50.405 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-12T22:22:50.406 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-05-12T22:22:50.406 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-05-12T22:22:50.406 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl stop ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968@mds.k 2024-05-12T22:22:50.742 INFO:journalctl@ceph.mds.k.smithi086.stdout:May 12 22:22:50 smithi086 systemd[1]: Stopping Ceph mds.k for 6b92e7b4-10aa-11ef-bc98-c7b262605968... 2024-05-12T22:22:50.742 INFO:journalctl@ceph.mds.k.smithi086.stdout:May 12 22:22:50 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-k[63855]: 2024-05-12T22:22:50.713+0000 7f042775f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-12T22:22:50.742 INFO:journalctl@ceph.mds.k.smithi086.stdout:May 12 22:22:50 smithi086 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-k[63855]: 2024-05-12T22:22:50.713+0000 7f042775f700 -1 mds.k *** got signal Terminated *** 2024-05-12T22:22:54.980 INFO:journalctl@ceph.mds.k.smithi086.stdout:May 12 22:22:54 smithi086 podman[71811]: 2024-05-12 22:22:54.720787875 +0000 UTC m=+4.104846623 container died 5c29b95217929e50456847bf58a8c8ed942f33ca1399c25e306d35cc776f6eaf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-k, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240507, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, io.buildah.version=1.33.5, RELEASE=squid-f72fecf, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-05-12T22:22:55.242 INFO:journalctl@ceph.mds.k.smithi086.stdout:May 12 22:22:55 smithi086 podman[71811]: 2024-05-12 22:22:55.097330778 +0000 UTC m=+4.481389532 container cleanup 5c29b95217929e50456847bf58a8c8ed942f33ca1399c25e306d35cc776f6eaf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-k, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240507, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, RELEASE=squid-f72fecf, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-05-12T22:22:55.242 INFO:journalctl@ceph.mds.k.smithi086.stdout:May 12 22:22:55 smithi086 bash[71811]: ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-k 2024-05-12T22:22:55.439 DEBUG:teuthology.orchestra.run.smithi086:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968@mds.k.service' 2024-05-12T22:22:55.469 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-12T22:22:55.470 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-05-12T22:22:55.470 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-05-12T22:22:55.470 DEBUG:teuthology.orchestra.run.smithi186:> sudo systemctl stop ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968@mds.c 2024-05-12T22:22:55.759 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 12 22:22:55 smithi186 systemd[1]: Stopping Ceph mds.c for 6b92e7b4-10aa-11ef-bc98-c7b262605968... 2024-05-12T22:22:56.021 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 12 22:22:55 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-c[65626]: 2024-05-12T22:22:55.759+0000 7f45f7be2700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-12T22:22:56.021 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 12 22:22:55 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-c[65626]: 2024-05-12T22:22:55.759+0000 7f45f7be2700 -1 mds.c *** got signal Terminated *** 2024-05-12T22:22:57.887 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 12 22:22:57 smithi186 podman[72512]: 2024-05-12 22:22:57.626176413 +0000 UTC m=+1.964007368 container died bfbd5c1aad9994d6b1f812db50681402c28a6f8e213766acd4baffe5c077565d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-c, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240507, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-f72fecf, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e) 2024-05-12T22:22:58.179 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 12 22:22:57 smithi186 podman[72512]: 2024-05-12 22:22:57.97773994 +0000 UTC m=+2.315570895 container cleanup bfbd5c1aad9994d6b1f812db50681402c28a6f8e213766acd4baffe5c077565d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-c, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240507, ceph=True, CEPH_POINT_RELEASE=, RELEASE=squid-f72fecf, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-12T22:22:58.179 INFO:journalctl@ceph.mds.c.smithi186.stdout:May 12 22:22:57 smithi186 bash[72512]: ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-c 2024-05-12T22:22:58.403 DEBUG:teuthology.orchestra.run.smithi186:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968@mds.c.service' 2024-05-12T22:22:58.437 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-12T22:22:58.437 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-05-12T22:22:58.438 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-05-12T22:22:58.438 DEBUG:teuthology.orchestra.run.smithi186:> sudo systemctl stop ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968@mds.f 2024-05-12T22:22:58.679 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 12 22:22:58 smithi186 systemd[1]: Stopping Ceph mds.f for 6b92e7b4-10aa-11ef-bc98-c7b262605968... 2024-05-12T22:22:59.179 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 12 22:22:58 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-f[65318]: 2024-05-12T22:22:58.765+0000 7fb55dfa8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-12T22:22:59.179 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 12 22:22:58 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-f[65318]: 2024-05-12T22:22:58.765+0000 7fb55dfa8700 -1 mds.f *** got signal Terminated *** 2024-05-12T22:23:00.929 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 12 22:23:00 smithi186 podman[72647]: 2024-05-12 22:23:00.650123038 +0000 UTC m=+1.999367202 container died 24c6a9b76babfc0e4f1a04c0488d4a710839519ce5481eef28d7db9f6af9516b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-f, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240507, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=squid-f72fecf) 2024-05-12T22:23:01.284 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 12 22:23:01 smithi186 podman[72647]: 2024-05-12 22:23:01.064945608 +0000 UTC m=+2.414189767 container cleanup 24c6a9b76babfc0e4f1a04c0488d4a710839519ce5481eef28d7db9f6af9516b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-f, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.build-date=20240507, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=squid-f72fecf, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-12T22:23:01.285 INFO:journalctl@ceph.mds.f.smithi186.stdout:May 12 22:23:01 smithi186 bash[72647]: ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-f 2024-05-12T22:23:01.493 DEBUG:teuthology.orchestra.run.smithi186:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968@mds.f.service' 2024-05-12T22:23:01.525 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-12T22:23:01.525 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-05-12T22:23:01.526 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-05-12T22:23:01.526 DEBUG:teuthology.orchestra.run.smithi186:> sudo systemctl stop ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968@mds.i 2024-05-12T22:23:01.852 INFO:journalctl@ceph.mds.i.smithi186.stdout:May 12 22:23:01 smithi186 systemd[1]: Stopping Ceph mds.i for 6b92e7b4-10aa-11ef-bc98-c7b262605968... 2024-05-12T22:23:02.179 INFO:journalctl@ceph.mds.i.smithi186.stdout:May 12 22:23:01 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-i[65930]: 2024-05-12T22:23:01.851+0000 7f6e4ef68700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-12T22:23:02.179 INFO:journalctl@ceph.mds.i.smithi186.stdout:May 12 22:23:01 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-i[65930]: 2024-05-12T22:23:01.851+0000 7f6e4ef68700 -1 mds.i *** got signal Terminated *** 2024-05-12T22:23:05.929 INFO:journalctl@ceph.mds.i.smithi186.stdout:May 12 22:23:05 smithi186 podman[72768]: 2024-05-12 22:23:05.624037694 +0000 UTC m=+3.882247693 container died ce0d3c9e0bae4476845de5f52188e87dd13cc52bbb60a0b9773bb701f8736c34 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-i, org.label-schema.schema-version=1.0, RELEASE=squid-f72fecf, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240507, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-05-12T22:23:06.429 INFO:journalctl@ceph.mds.i.smithi186.stdout:May 12 22:23:06 smithi186 podman[72768]: 2024-05-12 22:23:06.218763992 +0000 UTC m=+4.476973984 container cleanup ce0d3c9e0bae4476845de5f52188e87dd13cc52bbb60a0b9773bb701f8736c34 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-i, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240507, io.buildah.version=1.33.5, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=squid-f72fecf, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-05-12T22:23:06.429 INFO:journalctl@ceph.mds.i.smithi186.stdout:May 12 22:23:06 smithi186 bash[72768]: ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-i 2024-05-12T22:23:06.561 DEBUG:teuthology.orchestra.run.smithi186:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968@mds.i.service' 2024-05-12T22:23:06.592 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-12T22:23:06.593 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-05-12T22:23:06.593 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-05-12T22:23:06.593 DEBUG:teuthology.orchestra.run.smithi186:> sudo systemctl stop ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968@mds.l 2024-05-12T22:23:06.880 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 12 22:23:06 smithi186 systemd[1]: Stopping Ceph mds.l for 6b92e7b4-10aa-11ef-bc98-c7b262605968... 2024-05-12T22:23:07.179 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 12 22:23:06 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-l[65014]: 2024-05-12T22:23:06.879+0000 7fc65d6f1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-12T22:23:07.179 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 12 22:23:06 smithi186 ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-l[65014]: 2024-05-12T22:23:06.879+0000 7fc65d6f1700 -1 mds.l *** got signal Terminated *** 2024-05-12T22:23:08.748 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 12 22:23:08 smithi186 podman[72890]: 2024-05-12 22:23:08.489161017 +0000 UTC m=+1.681407864 container died c639841ebac2e8ba315fd23434a49a83d9354149f555fa3a8ccd3bb2e49e8ac4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-l, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240507, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-f72fecf, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-05-12T22:23:09.106 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 12 22:23:08 smithi186 podman[72890]: 2024-05-12 22:23:08.847642982 +0000 UTC m=+2.039889830 container cleanup c639841ebac2e8ba315fd23434a49a83d9354149f555fa3a8ccd3bb2e49e8ac4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:74e5ef5504e0304d38bdf19dfa72b49cf09cb02e13f017d98eaff2c84200b0bc, name=ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-l, org.label-schema.schema-version=1.0, GIT_COMMIT=26c185c6730a227e8afae88cfa8b303d6e5c470e, GIT_CLEAN=True, RELEASE=squid-f72fecf, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240507) 2024-05-12T22:23:09.106 INFO:journalctl@ceph.mds.l.smithi186.stdout:May 12 22:23:08 smithi186 bash[72890]: ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968-mds-l 2024-05-12T22:23:09.268 DEBUG:teuthology.orchestra.run.smithi186:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6b92e7b4-10aa-11ef-bc98-c7b262605968@mds.l.service' 2024-05-12T22:23:09.299 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-12T22:23:09.300 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-05-12T22:23:09.300 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 6b92e7b4-10aa-11ef-bc98-c7b262605968 --force --keep-logs 2024-05-12T22:23:09.639 INFO:teuthology.orchestra.run.smithi028.stdout:Deleting cluster with fsid: 6b92e7b4-10aa-11ef-bc98-c7b262605968 2024-05-12T22:23:14.709 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 6b92e7b4-10aa-11ef-bc98-c7b262605968 --force --keep-logs 2024-05-12T22:23:14.900 INFO:teuthology.orchestra.run.smithi086.stdout:Deleting cluster with fsid: 6b92e7b4-10aa-11ef-bc98-c7b262605968 2024-05-12T22:23:17.329 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 6b92e7b4-10aa-11ef-bc98-c7b262605968 --force --keep-logs 2024-05-12T22:23:17.517 INFO:teuthology.orchestra.run.smithi186.stdout:Deleting cluster with fsid: 6b92e7b4-10aa-11ef-bc98-c7b262605968 2024-05-12T22:23:19.833 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-12T22:23:19.932 DEBUG:teuthology.orchestra.run.smithi086:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-12T22:23:19.964 DEBUG:teuthology.orchestra.run.smithi186:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-12T22:23:19.990 INFO:tasks.cephadm:Archiving crash dumps... 2024-05-12T22:23:19.992 DEBUG:teuthology.misc:Transferring archived files from smithi028:/var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-05-12_21:24:02-fs-squid-distro-default-smithi/7703941/remote/smithi028/crash 2024-05-12T22:23:19.993 DEBUG:teuthology.orchestra.run.smithi028:> sudo tar c -f - -C /var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/crash -- . 2024-05-12T22:23:20.092 INFO:teuthology.orchestra.run.smithi028.stderr:tar: /var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/crash: Cannot open: No such file or directory 2024-05-12T22:23:20.093 INFO:teuthology.orchestra.run.smithi028.stderr:tar: Error is not recoverable: exiting now 2024-05-12T22:23:20.097 DEBUG:teuthology.misc:Transferring archived files from smithi086:/var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-05-12_21:24:02-fs-squid-distro-default-smithi/7703941/remote/smithi086/crash 2024-05-12T22:23:20.098 DEBUG:teuthology.orchestra.run.smithi086:> sudo tar c -f - -C /var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/crash -- . 2024-05-12T22:23:20.139 INFO:teuthology.orchestra.run.smithi086.stderr:tar: /var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/crash: Cannot open: No such file or directory 2024-05-12T22:23:20.139 INFO:teuthology.orchestra.run.smithi086.stderr:tar: Error is not recoverable: exiting now 2024-05-12T22:23:20.141 DEBUG:teuthology.misc:Transferring archived files from smithi186:/var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-05-12_21:24:02-fs-squid-distro-default-smithi/7703941/remote/smithi186/crash 2024-05-12T22:23:20.142 DEBUG:teuthology.orchestra.run.smithi186:> sudo tar c -f - -C /var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/crash -- . 2024-05-12T22:23:20.188 INFO:teuthology.orchestra.run.smithi186.stderr:tar: /var/lib/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/crash: Cannot open: No such file or directory 2024-05-12T22:23:20.188 INFO:teuthology.orchestra.run.smithi186.stderr:tar: Error is not recoverable: exiting now 2024-05-12T22:23:20.189 INFO:tasks.cephadm:Checking cluster log for badness... 2024-05-12T22:23:20.189 DEBUG:teuthology.orchestra.run.smithi028:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-05-12T22:23:20.300 INFO:tasks.cephadm:Compressing logs... 2024-05-12T22:23:20.301 DEBUG:teuthology.orchestra.run.smithi028:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-05-12T22:23:20.305 DEBUG:teuthology.orchestra.run.smithi086:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-05-12T22:23:20.308 DEBUG:teuthology.orchestra.run.smithi186:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-05-12T22:23:20.331 INFO:teuthology.orchestra.run.smithi186.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-05-12T22:23:20.332 INFO:teuthology.orchestra.run.smithi186.stderr:: No such file or directory 2024-05-12T22:23:20.332 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph.log 2024-05-12T22:23:20.332 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph-osd.2.log 2024-05-12T22:23:20.332 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph-mds.f.log 2024-05-12T22:23:20.339 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph.log: /var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph.audit.log 2024-05-12T22:23:20.340 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph-mds.f.log: gzip -5 --verbose -- /var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph-osd.5.log 2024-05-12T22:23:20.340 INFO:teuthology.orchestra.run.smithi186.stderr: 92.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-05-12T22:23:20.340 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph-mds.c.log 2024-05-12T22:23:20.340 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph-mon.c.log 2024-05-12T22:23:20.340 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph-mgr.z.log 2024-05-12T22:23:20.340 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph-mon.c.log: gzip -5 --verbose 87.4% -- replaced with /var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph.log.gz 2024-05-12T22:23:20.340 INFO:teuthology.orchestra.run.smithi186.stderr: -- /var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph-mds.i.log 2024-05-12T22:23:20.341 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph-mgr.z.log: gzip -5 --verbose -- /var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph.cephadm.log 2024-05-12T22:23:20.341 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph-mds.i.log: 90.2% -- replaced with /var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph.audit.log.gz 2024-05-12T22:23:20.341 INFO:teuthology.orchestra.run.smithi186.stderr: 94.5% -- replaced with /var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph-mds.f.log.gz 2024-05-12T22:23:20.343 INFO:teuthology.orchestra.run.smithi086.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-05-12T22:23:20.344 INFO:teuthology.orchestra.run.smithi086.stderr:: No such file or directory 2024-05-12T22:23:20.344 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph.log 2024-05-12T22:23:20.344 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph.audit.log 2024-05-12T22:23:20.344 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph-osd.7.log 2024-05-12T22:23:20.344 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph-mds.b.log 2024-05-12T22:23:20.344 INFO:teuthology.orchestra.run.smithi086.stderr: 92.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-05-12T22:23:20.344 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph-osd.4.log 2024-05-12T22:23:20.345 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph-osd.1.log 2024-05-12T22:23:20.345 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph-mds.b.log: /var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph-mgr.y.log 2024-05-12T22:23:20.345 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph-mds.h.log 2024-05-12T22:23:20.345 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph-mgr.y.log: gzip -5 --verbose 87.4% -- replaced with /var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph.log.gz 2024-05-12T22:23:20.345 INFO:teuthology.orchestra.run.smithi086.stderr: -- /var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph-osd.10.log 2024-05-12T22:23:20.345 INFO:teuthology.orchestra.run.smithi086.stderr: 90.2% -- replaced with /var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph.audit.log.gz 2024-05-12T22:23:20.345 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph.cephadm.log 2024-05-12T22:23:20.345 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph-mds.h.log: /var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph-mds.e.log 2024-05-12T22:23:20.345 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph.cephadm.log: 83.5% -- replaced with /var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph.cephadm.log.gz 2024-05-12T22:23:20.346 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph-volume.log 2024-05-12T22:23:20.346 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph-mds.e.log: 94.4% -- replaced with /var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph-mds.e.log.gz 2024-05-12T22:23:20.348 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph-mon.b.log 2024-05-12T22:23:20.348 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph-volume.log: 92.0% -- replaced with /var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph-mgr.y.log.gz 2024-05-12T22:23:20.350 INFO:teuthology.orchestra.run.smithi186.stderr: 92.0% -- replaced with /var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph-mgr.z.log.gz 2024-05-12T22:23:20.350 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph-osd.11.log 2024-05-12T22:23:20.350 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph-osd.8.log 2024-05-12T22:23:20.350 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph-osd.11.log: 83.5% -- replaced with /var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph.cephadm.log.gz 2024-05-12T22:23:20.350 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph-volume.log 2024-05-12T22:23:20.350 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph-mds.l.log 2024-05-12T22:23:20.357 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph-mds.k.log 2024-05-12T22:23:20.358 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph-mon.b.log: 91.0% -- replaced with /var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph-volume.log.gz 2024-05-12T22:23:20.364 INFO:teuthology.orchestra.run.smithi028.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose: No such file or directory -- 2024-05-12T22:23:20.364 INFO:teuthology.orchestra.run.smithi028.stderr: /var/log/ceph/cephadm.log 2024-05-12T22:23:20.365 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph-mds.k.log: 94.3% -- replaced with /var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph-mds.k.log.gz 2024-05-12T22:23:20.366 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph-volume.log: /var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph-mds.l.log: 91.0% -- replaced with /var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph-volume.log.gz 2024-05-12T22:23:20.366 INFO:teuthology.orchestra.run.smithi186.stderr: 94.5% -- replaced with /var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph-mds.l.log.gz 2024-05-12T22:23:20.367 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph.log 2024-05-12T22:23:20.367 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph.audit.log 2024-05-12T22:23:20.367 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph-mds.g.log 2024-05-12T22:23:20.367 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph-mds.j.log 2024-05-12T22:23:20.368 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph-osd.3.log 2024-05-12T22:23:20.369 INFO:teuthology.orchestra.run.smithi028.stderr: 87.4% -- replaced with /var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph.log.gz 2024-05-12T22:23:20.370 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph-mds.j.log: gzip -5 --verbose -- /var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph-osd.0.log 2024-05-12T22:23:20.371 INFO:teuthology.orchestra.run.smithi028.stderr: 90.0% -- replaced with /var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph.audit.log.gz 2024-05-12T22:23:20.371 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph-osd.3.log: 93.2% -- replaced with /var/log/ceph/cephadm.log.gz 2024-05-12T22:23:20.371 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph-mgr.x.log 2024-05-12T22:23:20.373 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph-mon.a.log 2024-05-12T22:23:20.374 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph-osd.0.log: /var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph-osd.6.log 2024-05-12T22:23:20.375 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph.cephadm.log 2024-05-12T22:23:20.376 INFO:teuthology.orchestra.run.smithi028.stderr: 94.2% -- replaced with /var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph-mds.j.log.gz 2024-05-12T22:23:20.377 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph-mds.a.log 2024-05-12T22:23:20.379 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph.cephadm.log: 83.9% -- replaced with /var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph.cephadm.log.gz 2024-05-12T22:23:20.379 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph-volume.log 2024-05-12T22:23:20.381 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph-mds.d.log 2024-05-12T22:23:20.390 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph-volume.log: /var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph-mds.d.log: gzip -5 --verbose -- /var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph-osd.9.log 2024-05-12T22:23:20.392 INFO:teuthology.orchestra.run.smithi186.stderr: 94.3% -- replaced with /var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph-mds.i.log.gz 2024-05-12T22:23:20.397 INFO:teuthology.orchestra.run.smithi186.stderr: 94.4% -- replaced with /var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph-mds.c.log.gz 2024-05-12T22:23:20.398 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph-osd.9.log: 94.1% -- replaced with /var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph-mds.d.log.gz 2024-05-12T22:23:20.400 INFO:teuthology.orchestra.run.smithi028.stderr: 90.9% -- replaced with /var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph-volume.log.gz 2024-05-12T22:23:20.492 INFO:teuthology.orchestra.run.smithi028.stderr: 93.2% -- replaced with /var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph-mds.g.log.gz 2024-05-12T22:23:20.547 INFO:teuthology.orchestra.run.smithi028.stderr: 92.9% -- replaced with /var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph-mds.a.log.gz 2024-05-12T22:23:20.786 INFO:teuthology.orchestra.run.smithi028.stderr: 90.1% -- replaced with /var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph-mgr.x.log.gz 2024-05-12T22:23:20.863 INFO:teuthology.orchestra.run.smithi186.stderr: 91.9% -- replaced with /var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph-mon.c.log.gz 2024-05-12T22:23:20.971 INFO:teuthology.orchestra.run.smithi086.stderr: 91.7% -- replaced with /var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph-mon.b.log.gz 2024-05-12T22:23:21.368 INFO:teuthology.orchestra.run.smithi086.stderr: 93.4% -- replaced with /var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph-osd.4.log.gz 2024-05-12T22:23:21.411 INFO:teuthology.orchestra.run.smithi186.stderr: 93.7% -- replaced with /var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph-osd.8.log.gz 2024-05-12T22:23:21.570 INFO:teuthology.orchestra.run.smithi186.stderr: 94.1% -- replaced with /var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph-osd.11.log.gz 2024-05-12T22:23:21.601 INFO:teuthology.orchestra.run.smithi086.stderr: 95.4% -- replaced with /var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph-mds.h.log.gz 2024-05-12T22:23:21.620 INFO:teuthology.orchestra.run.smithi186.stderr: 93.8% -- replaced with /var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph-osd.2.log.gz 2024-05-12T22:23:21.733 INFO:teuthology.orchestra.run.smithi028.stderr: 90.5% -- replaced with /var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph-mon.a.log.gz 2024-05-12T22:23:21.814 INFO:teuthology.orchestra.run.smithi186.stderr: 93.8% -- replaced with /var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph-osd.5.log.gz 2024-05-12T22:23:21.817 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-05-12T22:23:21.817 INFO:teuthology.orchestra.run.smithi186.stderr:real 0m1.495s 2024-05-12T22:23:21.817 INFO:teuthology.orchestra.run.smithi186.stderr:user 0m5.558s 2024-05-12T22:23:21.817 INFO:teuthology.orchestra.run.smithi186.stderr:sys 0m0.216s 2024-05-12T22:23:21.828 INFO:teuthology.orchestra.run.smithi028.stderr: 93.7% -- replaced with /var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph-osd.6.log.gz 2024-05-12T22:23:21.918 INFO:teuthology.orchestra.run.smithi086.stderr: 93.7% -- replaced with /var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph-osd.10.log.gz 2024-05-12T22:23:21.936 INFO:teuthology.orchestra.run.smithi028.stderr: 94.0% -- replaced with /var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph-osd.0.log.gz 2024-05-12T22:23:21.938 INFO:teuthology.orchestra.run.smithi028.stderr: 93.6% -- replaced with /var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph-osd.9.log.gz 2024-05-12T22:23:22.004 INFO:teuthology.orchestra.run.smithi086.stderr: 93.9% -- replaced with /var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph-osd.1.log.gz 2024-05-12T22:23:22.181 INFO:teuthology.orchestra.run.smithi028.stderr: 93.8% -- replaced with /var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph-osd.3.log.gz 2024-05-12T22:23:22.185 INFO:teuthology.orchestra.run.smithi028.stderr: 2024-05-12T22:23:22.185 INFO:teuthology.orchestra.run.smithi028.stderr:real 0m1.847s 2024-05-12T22:23:22.185 INFO:teuthology.orchestra.run.smithi028.stderr:user 0m7.829s 2024-05-12T22:23:22.185 INFO:teuthology.orchestra.run.smithi028.stderr:sys 0m0.720s 2024-05-12T22:23:22.245 INFO:teuthology.orchestra.run.smithi086.stderr: 94.0% -- replaced with /var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph-osd.7.log.gz 2024-05-12T22:23:39.468 INFO:teuthology.orchestra.run.smithi086.stderr: 93.6% -- replaced with /var/log/ceph/6b92e7b4-10aa-11ef-bc98-c7b262605968/ceph-mds.b.log.gz 2024-05-12T22:23:39.469 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-05-12T22:23:39.469 INFO:teuthology.orchestra.run.smithi086.stderr:real 0m19.147s 2024-05-12T22:23:39.469 INFO:teuthology.orchestra.run.smithi086.stderr:user 0m26.393s 2024-05-12T22:23:39.469 INFO:teuthology.orchestra.run.smithi086.stderr:sys 0m0.836s 2024-05-12T22:23:39.472 INFO:tasks.cephadm:Archiving logs... 2024-05-12T22:23:39.472 DEBUG:teuthology.misc:Transferring archived files from smithi028:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-05-12_21:24:02-fs-squid-distro-default-smithi/7703941/remote/smithi028/log 2024-05-12T22:23:39.479 DEBUG:teuthology.orchestra.run.smithi028:> sudo tar c -f - -C /var/log/ceph -- . 2024-05-12T22:23:40.877 DEBUG:teuthology.misc:Transferring archived files from smithi086:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-05-12_21:24:02-fs-squid-distro-default-smithi/7703941/remote/smithi086/log 2024-05-12T22:23:40.878 DEBUG:teuthology.orchestra.run.smithi086:> sudo tar c -f - -C /var/log/ceph -- . 2024-05-12T22:23:44.724 DEBUG:teuthology.misc:Transferring archived files from smithi186:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-05-12_21:24:02-fs-squid-distro-default-smithi/7703941/remote/smithi186/log 2024-05-12T22:23:44.725 DEBUG:teuthology.orchestra.run.smithi186:> sudo tar c -f - -C /var/log/ceph -- . 2024-05-12T22:23:45.542 INFO:tasks.cephadm:Removing cluster... 2024-05-12T22:23:45.543 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 6b92e7b4-10aa-11ef-bc98-c7b262605968 --force 2024-05-12T22:23:45.879 INFO:teuthology.orchestra.run.smithi028.stdout:Deleting cluster with fsid: 6b92e7b4-10aa-11ef-bc98-c7b262605968 2024-05-12T22:23:46.517 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 6b92e7b4-10aa-11ef-bc98-c7b262605968 --force 2024-05-12T22:23:46.696 INFO:teuthology.orchestra.run.smithi086.stdout:Deleting cluster with fsid: 6b92e7b4-10aa-11ef-bc98-c7b262605968 2024-05-12T22:23:46.972 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 6b92e7b4-10aa-11ef-bc98-c7b262605968 --force 2024-05-12T22:23:47.154 INFO:teuthology.orchestra.run.smithi186.stdout:Deleting cluster with fsid: 6b92e7b4-10aa-11ef-bc98-c7b262605968 2024-05-12T22:23:47.403 INFO:tasks.cephadm:Removing cephadm ... 2024-05-12T22:23:47.403 DEBUG:teuthology.orchestra.run.smithi028:> rm -rf /home/ubuntu/cephtest/cephadm 2024-05-12T22:23:47.465 DEBUG:teuthology.orchestra.run.smithi086:> rm -rf /home/ubuntu/cephtest/cephadm 2024-05-12T22:23:47.481 DEBUG:teuthology.orchestra.run.smithi186:> rm -rf /home/ubuntu/cephtest/cephadm 2024-05-12T22:23:47.496 INFO:tasks.cephadm:Teardown complete 2024-05-12T22:23:47.496 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-05-12T22:23:47.511 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-05-12T22:23:47.511 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-05-12T22:23:47.519 DEBUG:teuthology.orchestra.run.smithi086:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-05-12T22:23:47.523 DEBUG:teuthology.orchestra.run.smithi186:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-05-12T22:23:47.684 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-05-12T22:23:47.694 INFO:teuthology.task.clock:Checking final clock skew... 2024-05-12T22:23:47.694 DEBUG:teuthology.orchestra.run.smithi028:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-12T22:23:47.702 DEBUG:teuthology.orchestra.run.smithi086:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-12T22:23:47.704 DEBUG:teuthology.orchestra.run.smithi186:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-12T22:23:47.716 INFO:teuthology.orchestra.run.smithi086.stderr:bash: line 1: ntpq: command not found 2024-05-12T22:23:47.718 INFO:teuthology.orchestra.run.smithi186.stderr:bash: line 1: ntpq: command not found 2024-05-12T22:23:47.720 INFO:teuthology.orchestra.run.smithi086.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-12T22:23:47.720 INFO:teuthology.orchestra.run.smithi086.stdout:=============================================================================== 2024-05-12T22:23:47.720 INFO:teuthology.orchestra.run.smithi086.stdout:^* hv01.front.sepia.ceph.com 3 8 377 175 +919us[ +996us] +/- 40ms 2024-05-12T22:23:47.720 INFO:teuthology.orchestra.run.smithi086.stdout:^+ hv02.front.sepia.ceph.com 3 7 377 106 -748us[ -748us] +/- 71ms 2024-05-12T22:23:47.720 INFO:teuthology.orchestra.run.smithi086.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 42 -625us[ -625us] +/- 90ms 2024-05-12T22:23:47.721 INFO:teuthology.orchestra.run.smithi086.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-05-12T22:23:47.722 INFO:teuthology.orchestra.run.smithi186.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-12T22:23:47.722 INFO:teuthology.orchestra.run.smithi186.stdout:=============================================================================== 2024-05-12T22:23:47.722 INFO:teuthology.orchestra.run.smithi186.stdout:^* hv01.front.sepia.ceph.com 3 6 377 42 +813us[ +808us] +/- 42ms 2024-05-12T22:23:47.722 INFO:teuthology.orchestra.run.smithi186.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 41 -790us[ -790us] +/- 72ms 2024-05-12T22:23:47.722 INFO:teuthology.orchestra.run.smithi186.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 41 -703us[ -703us] +/- 90ms 2024-05-12T22:23:47.723 INFO:teuthology.orchestra.run.smithi186.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-05-12T22:23:47.751 INFO:teuthology.orchestra.run.smithi028.stderr:bash: line 1: ntpq: command not found 2024-05-12T22:23:47.762 INFO:teuthology.orchestra.run.smithi028.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-12T22:23:47.762 INFO:teuthology.orchestra.run.smithi028.stdout:=============================================================================== 2024-05-12T22:23:47.762 INFO:teuthology.orchestra.run.smithi028.stdout:^* hv01.front.sepia.ceph.com 3 7 377 44 +897us[ +926us] +/- 42ms 2024-05-12T22:23:47.762 INFO:teuthology.orchestra.run.smithi028.stdout:^+ hv02.front.sepia.ceph.com 3 7 377 39 -711us[ -711us] +/- 72ms 2024-05-12T22:23:47.763 INFO:teuthology.orchestra.run.smithi028.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 41 -640us[ -640us] +/- 90ms 2024-05-12T22:23:47.763 INFO:teuthology.orchestra.run.smithi028.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-05-12T22:23:47.764 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-05-12T22:23:47.775 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-05-12T22:23:47.776 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-05-12T22:23:47.786 DEBUG:teuthology.orchestra.run.smithi028:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-05-12T22:23:48.007 DEBUG:teuthology.orchestra.run.smithi086:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-05-12T22:23:48.073 DEBUG:teuthology.orchestra.run.smithi186:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-05-12T22:23:48.138 DEBUG:teuthology.orchestra.run.smithi028:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-12T22:23:48.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-12T22:23:48.242 DEBUG:teuthology.orchestra.run.smithi086:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-12T22:23:48.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-12T22:23:48.269 DEBUG:teuthology.orchestra.run.smithi186:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-12T22:23:48.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-12T22:23:48.296 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-05-12T22:23:48.307 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-05-12T22:23:48.319 INFO:teuthology.task.internal:Duration was 2142.940596 seconds 2024-05-12T22:23:48.319 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-05-12T22:23:48.334 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-05-12T22:23:48.335 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-05-12T22:23:48.343 DEBUG:teuthology.orchestra.run.smithi086:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-05-12T22:23:48.346 DEBUG:teuthology.orchestra.run.smithi186:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-05-12T22:23:48.380 INFO:teuthology.orchestra.run.smithi086.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-12T22:23:48.382 INFO:teuthology.orchestra.run.smithi186.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-12T22:23:48.492 INFO:teuthology.orchestra.run.smithi028.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-12T22:23:48.782 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-05-12T22:23:48.782 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi028.front.sepia.ceph.com 2024-05-12T22:23:48.783 DEBUG:teuthology.orchestra.run.smithi028:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-05-12T22:23:48.863 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi086.front.sepia.ceph.com 2024-05-12T22:23:48.864 DEBUG:teuthology.orchestra.run.smithi086:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-05-12T22:23:48.887 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi186.front.sepia.ceph.com 2024-05-12T22:23:48.888 DEBUG:teuthology.orchestra.run.smithi186:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-05-12T22:23:48.912 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-05-12T22:23:48.913 DEBUG:teuthology.orchestra.run.smithi028:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-05-12T22:23:48.920 DEBUG:teuthology.orchestra.run.smithi086:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-05-12T22:23:48.931 DEBUG:teuthology.orchestra.run.smithi186:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-05-12T22:23:49.010 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-05-12T22:23:49.011 DEBUG:teuthology.orchestra.run.smithi028:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-05-12T22:23:49.016 DEBUG:teuthology.orchestra.run.smithi086:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-05-12T22:23:49.019 DEBUG:teuthology.orchestra.run.smithi186:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-05-12T22:23:49.532 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-05-12T22:23:49.546 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-05-12T22:23:49.546 DEBUG:teuthology.orchestra.run.smithi028:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-05-12T22:23:49.636 DEBUG:teuthology.orchestra.run.smithi086:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-05-12T22:23:49.659 DEBUG:teuthology.orchestra.run.smithi186:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-05-12T22:23:49.684 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-05-12T22:23:49.697 DEBUG:teuthology.orchestra.run.smithi028:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-05-12T22:23:49.705 DEBUG:teuthology.orchestra.run.smithi086:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-05-12T22:23:49.708 DEBUG:teuthology.orchestra.run.smithi186:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-05-12T22:23:49.728 INFO:teuthology.orchestra.run.smithi086.stdout:kernel.core_pattern = core 2024-05-12T22:23:49.747 INFO:teuthology.orchestra.run.smithi186.stdout:kernel.core_pattern = core 2024-05-12T22:23:49.808 INFO:teuthology.orchestra.run.smithi028.stdout:kernel.core_pattern = core 2024-05-12T22:23:49.852 DEBUG:teuthology.orchestra.run.smithi028:> test -e /home/ubuntu/cephtest/archive/coredump 2024-05-12T22:23:49.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-12T22:23:49.905 DEBUG:teuthology.orchestra.run.smithi086:> test -e /home/ubuntu/cephtest/archive/coredump 2024-05-12T22:23:49.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-12T22:23:49.919 DEBUG:teuthology.orchestra.run.smithi186:> test -e /home/ubuntu/cephtest/archive/coredump 2024-05-12T22:23:49.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-12T22:23:49.934 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-05-12T22:23:49.946 INFO:teuthology.task.internal:Transferring archived files... 2024-05-12T22:23:49.947 DEBUG:teuthology.misc:Transferring archived files from smithi028:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-05-12_21:24:02-fs-squid-distro-default-smithi/7703941/remote/smithi028 2024-05-12T22:23:49.947 DEBUG:teuthology.orchestra.run.smithi028:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-05-12T22:23:50.054 DEBUG:teuthology.misc:Transferring archived files from smithi086:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-05-12_21:24:02-fs-squid-distro-default-smithi/7703941/remote/smithi086 2024-05-12T22:23:50.054 DEBUG:teuthology.orchestra.run.smithi086:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-05-12T22:23:50.090 DEBUG:teuthology.misc:Transferring archived files from smithi186:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-05-12_21:24:02-fs-squid-distro-default-smithi/7703941/remote/smithi186 2024-05-12T22:23:50.090 DEBUG:teuthology.orchestra.run.smithi186:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-05-12T22:23:50.129 INFO:teuthology.task.internal:Removing archive directory... 2024-05-12T22:23:50.129 DEBUG:teuthology.orchestra.run.smithi028:> rm -rf -- /home/ubuntu/cephtest/archive 2024-05-12T22:23:50.137 DEBUG:teuthology.orchestra.run.smithi086:> rm -rf -- /home/ubuntu/cephtest/archive 2024-05-12T22:23:50.139 DEBUG:teuthology.orchestra.run.smithi186:> rm -rf -- /home/ubuntu/cephtest/archive 2024-05-12T22:23:50.190 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-05-12T22:23:50.204 INFO:teuthology.task.internal:Not uploading archives. 2024-05-12T22:23:50.204 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-05-12T22:23:50.216 INFO:teuthology.task.internal:Tidying up after the test... 2024-05-12T22:23:50.217 DEBUG:teuthology.orchestra.run.smithi028:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-05-12T22:23:50.222 DEBUG:teuthology.orchestra.run.smithi086:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-05-12T22:23:50.225 DEBUG:teuthology.orchestra.run.smithi186:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-05-12T22:23:50.237 INFO:teuthology.orchestra.run.smithi086.stdout: 262161 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 12 22:23 /home/ubuntu/cephtest 2024-05-12T22:23:50.240 INFO:teuthology.orchestra.run.smithi186.stdout: 262161 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 12 22:23 /home/ubuntu/cephtest 2024-05-12T22:23:50.274 INFO:teuthology.orchestra.run.smithi028.stdout: 393243 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 12 22:23 /home/ubuntu/cephtest 2024-05-12T22:23:50.278 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-05-12T22:23:50.291 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-05-12T22:23:50.315 INFO:teuthology.nuke:Checking targets against current locks 2024-05-12T22:23:50.344 DEBUG:teuthology.nuke:shortname: smithi028 2024-05-12T22:23:50.344 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-05-12T22:23:50.370 DEBUG:teuthology.nuke:shortname: smithi086 2024-05-12T22:23:50.370 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-05-12T22:23:50.376 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi028.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-05-12_21:24:02-fs-squid-distro-default-smithi/7703941', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-12 21:39:47.310876', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDo+zcEat3FoejyP1Xh22qYVt0nyVtxgcdLMqfh03Mdzze/iDt4FpewLerXH3e4my7vCG+Ao6QZBuJsQF5Kt6HA='} 2024-05-12T22:23:50.395 DEBUG:teuthology.nuke:shortname: smithi186 2024-05-12T22:23:50.396 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-05-12T22:23:50.400 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi086.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-05-12_21:24:02-fs-squid-distro-default-smithi/7703941', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-12 21:39:47.312120', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKe91EP00mtpT0bhWgWPA1pmY+P/QDBNmgZRLVNpx6BW4OgCPc/hZK6pSgc/C5+VBSOnxwa99MLSwzPW2Y6ng8I='} 2024-05-12T22:23:50.422 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi186.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-05-12_21:24:02-fs-squid-distro-default-smithi/7703941', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-12 21:39:47.309124', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBX7vLlnMm7oYwlDmyuB3GJBNIVtLzrjtokbCrB+aW/HHlnpxPaROl2qRZfyN7/XEmH6eknXHzThFehR4kxyjWA='} 2024-05-12T22:23:50.435 INFO:teuthology.orchestra.console.smithi028:Power off 2024-05-12T22:23:50.436 DEBUG:teuthology.orchestra.console.smithi028:pexpect command: ipmitool -H smithi028.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-05-12T22:23:50.462 DEBUG:teuthology.orchestra.console.smithi028:power off output: Chassis Power Control: Down/Off 2024-05-12T22:23:50.462 DEBUG:teuthology.orchestra.console.smithi028:pexpect command: ipmitool -H smithi028.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-12T22:23:50.479 INFO:teuthology.orchestra.console.smithi086:Power off 2024-05-12T22:23:50.480 DEBUG:teuthology.orchestra.console.smithi086:pexpect command: ipmitool -H smithi086.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-05-12T22:23:50.499 INFO:teuthology.orchestra.console.smithi186:Power off 2024-05-12T22:23:50.499 DEBUG:teuthology.orchestra.console.smithi186:pexpect command: ipmitool -H smithi186.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-05-12T22:23:50.508 DEBUG:teuthology.orchestra.console.smithi028:check power output: Chassis Power is on 2024-05-12T22:23:50.509 DEBUG:teuthology.orchestra.console.smithi086:power off output: Chassis Power Control: Down/Off 2024-05-12T22:23:50.509 DEBUG:teuthology.orchestra.console.smithi086:pexpect command: ipmitool -H smithi086.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-12T22:23:50.523 DEBUG:teuthology.orchestra.console.smithi186:power off output: Chassis Power Control: Down/Off 2024-05-12T22:23:50.524 DEBUG:teuthology.orchestra.console.smithi186:pexpect command: ipmitool -H smithi186.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-12T22:23:50.534 DEBUG:teuthology.orchestra.console.smithi086:check power output: Chassis Power is on 2024-05-12T22:23:50.549 DEBUG:teuthology.orchestra.console.smithi186:check power output: Chassis Power is on 2024-05-12T22:23:54.509 DEBUG:teuthology.orchestra.console.smithi028:pexpect command: ipmitool -H smithi028.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-12T22:23:54.535 DEBUG:teuthology.orchestra.console.smithi086:pexpect command: ipmitool -H smithi086.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-12T22:23:54.550 DEBUG:teuthology.orchestra.console.smithi186:pexpect command: ipmitool -H smithi186.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-12T22:23:54.621 DEBUG:teuthology.orchestra.console.smithi028:check power output: Chassis Power is on 2024-05-12T22:23:54.646 DEBUG:teuthology.orchestra.console.smithi086:check power output: Chassis Power is on 2024-05-12T22:23:54.660 DEBUG:teuthology.orchestra.console.smithi186:check power output: Chassis Power is on 2024-05-12T22:23:58.626 DEBUG:teuthology.orchestra.console.smithi028:pexpect command: ipmitool -H smithi028.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-12T22:23:58.647 DEBUG:teuthology.orchestra.console.smithi086:pexpect command: ipmitool -H smithi086.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-12T22:23:58.661 DEBUG:teuthology.orchestra.console.smithi186:pexpect command: ipmitool -H smithi186.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-12T22:23:58.738 DEBUG:teuthology.orchestra.console.smithi028:check power output: Chassis Power is off 2024-05-12T22:23:58.759 DEBUG:teuthology.orchestra.console.smithi086:check power output: Chassis Power is off 2024-05-12T22:23:58.773 DEBUG:teuthology.orchestra.console.smithi186:check power output: Chassis Power is on 2024-05-12T22:23:58.840 INFO:teuthology.orchestra.console.smithi028:Power off completed 2024-05-12T22:23:58.860 INFO:teuthology.orchestra.console.smithi086:Power off completed 2024-05-12T22:24:02.778 DEBUG:teuthology.orchestra.console.smithi186:pexpect command: ipmitool -H smithi186.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-12T22:24:02.890 DEBUG:teuthology.orchestra.console.smithi186:check power output: Chassis Power is off 2024-05-12T22:24:02.991 INFO:teuthology.orchestra.console.smithi186:Power off completed 2024-05-12T22:24:03.102 INFO:teuthology.run:Summary data: client.0-kernel-sha1: fb9a6a44107e0b9f9ca2d9a78f2308f5dd49911e description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/fsx}} duration: 2142.940596103668 failure_reason: 'Command failed (workunit test suites/fsx.sh) on smithi028 with status 2: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c50db29d375ad9ca1a200449f9347b89e25fb278 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsx.sh''' flavor: default owner: scheduled_teuthology@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=e81098b5d28a49e2bee829231cd48db9 status: fail success: false 2024-05-12T22:24:03.103 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-05-12T22:24:03.199 INFO:teuthology.run:FAIL